From 2ccbda828bb3c15f2936362540540b2f3c1b64f3 Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 01/38] [None][feat] MiniMax-M3: load the compressed-tensors NVFP4 MoE packaging MXFP8 non-expert projections with NVFP4 routed experts on the existing model: model_config expands the packaging's expert recipe into the mixed per-layer config, the weight mapper renames the packed expert tensors and inverts their global scales, MXFP8 index projections and fused MXFP8 shards load their own block-scale rows, the sparse cache manager takes the layer schedule from the checkpoint and counts index-K bytes, the router GEMM runs in FP32 cuBLAS, and the VL input processor loads the HF processor only for image requests. Signed-off-by: Pranav Shrestha --- ...ployment-guide-for-minimax-m3-on-trtllm.md | 26 +- .../sparse/minimax_m3/cache_manager.py | 128 ++- .../backends/sparse/minimax_m3/common.py | 39 +- tensorrt_llm/_torch/model_config.py | 158 +++- .../checkpoints/hf/minimaxm3_weight_mapper.py | 96 ++- .../_torch/models/modeling_minimaxm3.py | 102 ++- .../_torch/models/modeling_minimaxm3_vl.py | 79 +- tensorrt_llm/_torch/modules/linear.py | 46 +- .../integration/test_lists/test-db/l0_cpu.yml | 6 + .../test-db/l0_gb300_multi_gpus.yml | 4 + .../test_minimax_m3_shared_draft_layers.py | 107 +++ .../hf/test_minimaxm3_weight_mapper.py | 213 +++++ .../unittest/_torch/models/test_minimax_m3.py | 747 +++++++++++++++++- .../_torch/models/test_minimax_m3_vl.py | 142 ++++ .../_torch/modules/test_mxfp8_linear.py | 93 +++ 15 files changed, 1839 insertions(+), 147 deletions(-) diff --git a/docs/source/deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md b/docs/source/deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md index 93e4bf3fe471..c93507fd6b61 100644 --- a/docs/source/deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md +++ b/docs/source/deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md @@ -56,8 +56,32 @@ The `MiniMax-M3-NVFP4` checkpoint uses MXFP8 base layers and NVFP4 routed expert (`MIXED_PRECISION`). Its name describes weight precision; set `kv_cache_config.dtype` explicitly to select FP8 or NVFP4 KV cache. +The compressed-tensors NVFP4 MoE packaging of the MiniMax M3 architecture is +also loaded as mixed MXFP8/NVFP4. Such a checkpoint declares +`quant_method: mxfp8` together with `moe_quant_algo: NVFP4`, +`moe_quant_format: nvfp4-pack-quantized` and `group_size: 16`, and stores each +routed expert projection as `weight_packed`, `weight_scale`, +`weight_global_scale` and `input_global_scale`. TensorRT LLM keeps the packed +FP4 bytes and E4M3 block scales as stored and converts the global scales at +load time. A global scale that is zero, negative or not finite fails the load +with an error naming the checkpoint tensor. Use `moe_config.backend: CUTLASS` +for these routed experts. They follow the rounding steps of the reference +implementation of this packaging: every token's FC1 input is quantized with +its own scale (2688 / amax of the token), the inverse of that scale times the +FC1 weight scale multiplies the FC1 accumulators before their BF16 rounding, +and the routing weight multiplies the activation before it is requantized to +NVFP4 for FC2 (from FP32, with the stored static FC2 input scale), so the +finalize step sums the expert outputs without weights. Only the CUTLASS +backend implements this, so loading such a checkpoint with another MoE backend +fails; the finalize step runs unfused, and the stored w1/w3 input scales must +be 1.0. + Use the checkpoint's chat template (`chat_template.jinja`) when launching the server (see [Launch the TensorRT LLM Server](#launch-the-tensorrt-llm-server)). +Evaluate a checkpoint in this packaging through the chat template as well, for +example `trtllm-eval ... gsm8k --apply_chat_template --chat_template_kwargs +'{"thinking_mode": "disabled"}'`; the raw eight-shot prompts of the `gsm8k` +default bypass the template the model was trained with. ## Feature Support Notes @@ -67,7 +91,7 @@ server (see [Launch the TensorRT LLM Server](#launch-the-tensorrt-llm-server)). * **Eagle3.** The MSA path supports one shared GQA draft layer with linear Eagle3 drafting. Other MTP and tree-drafting configurations are unsupported. Keep NVFP4 KV and piecewise CUDA graph qualification separate: the existing PCG accuracy cases use FP8 KV. * **`max_seq_len` must be capped for CUDA graphs.** The dense GQA expansion in the first few attention layers and the per-Q FP32 expansion in the sparse decode kernel allocate temporary tensors whose size grows linearly with the warmup decode's `max_k`. If `max_seq_len` is left at the checkpoint default, that `max_k` follows `max_position_embeddings` (1M for MXFP8, 512K for BF16) and the resulting gigabyte-scale single-allocation request exceeds the caching allocator's CUDA-graph-safe path, so capture fails with `cudaErrorStreamCaptureUnsupported` / OOM. The curated YAML therefore sets `max_seq_len` to a small value just above ISL+OSL (`2068` for the 1k/1k benchmark) so CUDA graphs can capture cleanly. Raise it for longer-context workloads, but expect a corresponding cut in `max_batch_size`. * **Parallelism.** MoE experts run with expert parallelism. The attention layers support both Tensor-Expert Parallelism (TEP) and Data-Expert Parallelism (DEP, via `enable_attention_dp: true`). The overlap scheduler is enabled by default. -* **Multimodal.** `MiniMaxM3SparseForConditionalGeneration` supports text, image, and video inputs. The text decoder is also usable standalone (text-only) via the `MiniMaxM3SparseForCausalLM` architecture. +* **Multimodal.** `MiniMaxM3SparseForConditionalGeneration` supports text, image, and video inputs. The text decoder is also usable standalone (text-only) via the `MiniMaxM3SparseForCausalLM` architecture. Text requests need only the tokenizer. The checkpoint's Hugging Face processor is loaded on the first image or video request. If `AutoProcessor` cannot build a processor with an image processor for the checkpoint directory (for example, it returns only a tokenizer because no loadable processor code is shipped), that request fails with a `ValueError` saying that MiniMax-M3 image and video inputs need an HF processor with an image processor. Text requests keep working. To serve images, point the model at a directory whose processor loads with `AutoProcessor`. ## Deployment Steps diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py index 841bb0366237..1b5304d25b85 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py @@ -53,6 +53,8 @@ PageIndexMode, ) +from .common import get_sparse_disable_index_value_layer_ids, get_sparse_layer_ids + if TYPE_CHECKING: from tensorrt_llm.llmapi.llm_args import DecodingBaseConfig @@ -159,6 +161,47 @@ def shared_draft_layer_count(spec_config, layer_mask) -> int: return int(get_num_spec_layers(spec_config)) +def resolve_sparse_layer_schedule( + sparse_layer_ids: Optional[Sequence[int]], + disable_index_value_layer_ids: Optional[Sequence[int]], + pretrained_config, + num_target_layers: Optional[int], +) -> Tuple[List[int], List[int]]: + """Return ``(sparse_layer_ids, disable_index_value_layer_ids)`` for the cache. + + Explicit ids win. Otherwise the checkpoint's attention schedule + (``pretrained_config.sparse_attention_config``) decides, which is the + schedule the model builds its sparse layers from; it may make every layer + sparse, independently of the dense-MLP layers. Without either, the public + M3 convention applies: layers 0..2 dense, 3..N-1 sparse and K-only. + """ + if ( + sparse_layer_ids is None + and getattr(pretrained_config, "sparse_attention_config", None) is not None + ): + _, sparse_layer_ids = get_sparse_layer_ids(pretrained_config) + if disable_index_value_layer_ids is None: + disable_index_value_layer_ids = get_sparse_disable_index_value_layer_ids( + pretrained_config + ) + if num_target_layers is not None and any( + not 0 <= int(i) < num_target_layers + for i in (*sparse_layer_ids, *disable_index_value_layer_ids) + ): + raise ValueError( + "MiniMax M3 checkpoint attention schedule names layers outside the " + f"{num_target_layers} target layers: sparse={sparse_layer_ids}, " + f"disable_index_value={disable_index_value_layer_ids}." + ) + if sparse_layer_ids is None: + sparse_layer_ids = ( + list(range(3, num_target_layers)) if num_target_layers is not None else [] + ) + if disable_index_value_layer_ids is None: + disable_index_value_layer_ids = list(sparse_layer_ids) + return list(sparse_layer_ids), list(disable_index_value_layer_ids) + + def derive_shared_draft_layout( num_layers: Optional[int], num_kv_heads, @@ -279,13 +322,16 @@ def __init__( **kwargs, ): # Resolve M3 sparse-layer metadata from explicit kwargs first, then - # from the executor's ``sparse_attention_config`` keyword, then from - # the M3 checkpoint convention (layers 0..2 dense, 3..N-1 sparse, + # from the executor's ``sparse_attention_config`` keyword and the + # checkpoint's attention schedule (``pretrained_config``, the same + # schedule the model builds its sparse layers from), then from the + # public M3 convention (layers 0..2 dense, 3..N-1 sparse, # disable_index_value=True, sparse_index_dim=128). Honoring the # executor keyword also makes non-default sparse_index_dim values # authoritative for the cache layout instead of falling back to 128. # Peeked (not popped) so the base __init__ still receives them. sparse_attention_config = kwargs.get("sparse_attention_config") + pretrained_config = kwargs.get("pretrained_config") num_layers = kwargs.get("num_layers") implementation = getattr(sparse_attention_config, "implementation", "triton") self._main_kv_mapper_kind = MapperKind.HND if implementation == "msa" else MapperKind.NHD @@ -306,13 +352,9 @@ def __init__( kwargs.get("num_kv_heads"), shared_draft_layer_count(kwargs.get("spec_config"), kwargs.get("layer_mask")), ) - if sparse_layer_ids is None: - if num_target_layers is not None: - sparse_layer_ids = list(range(3, num_target_layers)) - else: - sparse_layer_ids = [] - if disable_index_value_layer_ids is None: - disable_index_value_layer_ids = list(sparse_layer_ids) + sparse_layer_ids, disable_index_value_layer_ids = resolve_sparse_layer_schedule( + sparse_layer_ids, disable_index_value_layer_ids, pretrained_config, num_target_layers + ) # Must be set BEFORE super().__init__ — the base # ``_build_base_config`` invokes ``_extra_buffers_per_layer`` @@ -417,25 +459,36 @@ def _build_cache_config(self, config): def get_layer_bytes_per_token(self, local_layer_idx: int, data_role: Role): """Report the hybrid sparse-NVFP4 / dense-FP8 storage footprint.""" - if self.dtype != DataType.NVFP4: - return super().get_layer_bytes_per_token(local_layer_idx, data_role) global_layer_idx = int(self.pp_layers[int(local_layer_idx)]) - if global_layer_idx in self.sparse_layer_ids: - return super().get_layer_bytes_per_token(local_layer_idx, data_role) - - if data_role in (Role.KEY_BLOCK_SCALE, Role.VALUE_BLOCK_SCALE): - return 0 - if data_role == Role.ALL: - kv_factor = self.kv_factor - elif data_role in (Role.KEY, Role.VALUE): - kv_factor = 1 + has_index_k = global_layer_idx in self.sparse_layer_ids + if data_role == Role.INDEX_KEY: + if not has_index_k: + return 0 + index_dtype = self._torch_dtype_for_index_cache() + index_elem_bytes = torch.empty(0, dtype=index_dtype, device="cpu").element_size() + return self.sparse_index_dim * index_elem_bytes + + if self.dtype != DataType.NVFP4 or has_index_k: + cache_bytes = super().get_layer_bytes_per_token(local_layer_idx, data_role) else: - return super().get_layer_bytes_per_token(local_layer_idx, data_role) - return ( - kv_factor - * self.num_kv_heads_per_layer[int(local_layer_idx)] - * self.head_dim_per_layer[int(local_layer_idx)] - ) + # Existing NVFP4 dense-target / shared-Eagle FP8 cache calculation. + if data_role in (Role.KEY_BLOCK_SCALE, Role.VALUE_BLOCK_SCALE): + return 0 + if data_role == Role.ALL: + kv_factor = self.kv_factor + elif data_role in (Role.KEY, Role.VALUE): + kv_factor = 1 + else: + return super().get_layer_bytes_per_token(local_layer_idx, data_role) + cache_bytes = ( + kv_factor + * self.num_kv_heads_per_layer[int(local_layer_idx)] + * self.head_dim_per_layer[int(local_layer_idx)] + ) + + if data_role == Role.ALL and has_index_k: + cache_bytes += self.get_layer_bytes_per_token(local_layer_idx, Role.INDEX_KEY) + return cache_bytes def is_nvfp4_layer(self, layer_idx: int) -> bool: """Whether ``layer_idx`` stores packed NVFP4 data and block scales.""" @@ -589,20 +642,17 @@ def update_resources(self, scheduled_batch, attn_metadata=None, kv_cache_dtype_b super().update_resources(scheduled_batch, attn_metadata, kv_cache_dtype_byte_size) def _extra_buffers_per_layer(self, *, tokens_per_block): - """Register a per-sparse-layer ``Role.INDEX_KEY`` :class:`BufferConfig`. - - ``size`` is bytes per **block**: ``1 * sparse_index_dim * - elem_bytes * tokens_per_block``. Keyed by **local** layer id — - the base ``_build_base_config`` iterates local ids, so keying - by global ids would silently skip registration on non-trivial - PP ranks. - """ - torch_dtype = self._torch_dtype_for_index_cache() - elem_bytes = torch.tensor([], dtype=torch_dtype).element_size() - bytes_per_token = 1 * self.sparse_index_dim * elem_bytes - size_per_block = bytes_per_token * tokens_per_block + """Register the paged index-K buffer for each sparse local layer.""" return { - self.layer_offsets[layer_id]: [BufferConfig(role=Role.INDEX_KEY, size=size_per_block)] + self.layer_offsets[layer_id]: [ + BufferConfig( + role=Role.INDEX_KEY, + size=self.get_layer_bytes_per_token( + self.layer_offsets[layer_id], Role.INDEX_KEY + ) + * tokens_per_block, + ) + ] for layer_id in self.sparse_layer_ids if layer_id in self.layer_offsets } diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py index f288495eeff1..93df670a4137 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py @@ -11,7 +11,7 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import TYPE_CHECKING, Literal, NamedTuple, Optional, Tuple +from typing import TYPE_CHECKING, Any, List, Literal, NamedTuple, Optional, Tuple import torch @@ -32,6 +32,43 @@ _LOCAL_SCORE = 1e29 +def get_sparse_layer_ids(text_config: Any) -> Tuple[List[int], List[int]]: + """Return ``(dense_layer_ids, sparse_layer_ids)`` for the M3 text model. + + The checkpoint's ``sparse_attention_config.sparse_attention_freq`` is the + attention schedule; it is independent of the dense-MLP/MoE schedule. + Shared by the model, which builds sparse attention on these layers, and + the KV-cache manager, which allocates their index-K cache. + """ + sparse_cfg = getattr(text_config, "sparse_attention_config", None) + n_layers = int(text_config.num_hidden_layers) + if sparse_cfg is None: + return list(range(n_layers)), [] + if not sparse_cfg.get("use_sparse_attention", True): + return list(range(n_layers)), [] + freq = sparse_cfg.get("sparse_attention_freq") + if freq is None: + return list(range(n_layers)), [] + if len(freq) != n_layers: + raise ValueError( + f"sparse_attention_freq length {len(freq)} does not match num_hidden_layers {n_layers}" + ) + dense = [i for i, f in enumerate(freq) if int(f) == 0] + sparse = [i for i, f in enumerate(freq) if int(f) != 0] + return dense, sparse + + +def get_sparse_disable_index_value_layer_ids(text_config: Any) -> List[int]: + """Return layer ids where the sparse index-value branch is disabled.""" + sparse_cfg = getattr(text_config, "sparse_attention_config", None) + if sparse_cfg is None: + return [] + flags = sparse_cfg.get("sparse_disable_index_value") + if flags is None: + return [] + return [i for i, f in enumerate(flags) if int(f) != 0] + + def index_head_range( num_index_heads: int, num_kv_heads: int, mapping: Optional["Mapping"] = None ) -> Tuple[int, int]: diff --git a/tensorrt_llm/_torch/model_config.py b/tensorrt_llm/_torch/model_config.py index 13534230bad9..0fc3433fdbf2 100644 --- a/tensorrt_llm/_torch/model_config.py +++ b/tensorrt_llm/_torch/model_config.py @@ -21,6 +21,7 @@ import tempfile from dataclasses import dataclass, field from pathlib import Path +from types import SimpleNamespace from typing import (TYPE_CHECKING, Any, Dict, Generic, Iterator, List, Optional, TypeVar) @@ -936,11 +937,147 @@ def _set_deepseek_v4_routed_moe_quant_config(pretrained_config, return layer_quant_config @staticmethod - def _set_minimax_m3_layer_quant_config(pretrained_config, + def _get_mxfp8_moe_quant_config( + hf_quant_config: Dict[str, Any]) -> Optional[QuantConfig]: + """Return the routed-expert recipe declared next to an MXFP8 recipe. + + The compressed-tensors NVFP4 MoE packaging of the MiniMax M3 + architecture keeps ``quant_method: mxfp8`` for the base layers and + declares the routed experts beside it (``moe_quant_algo: NVFP4``, + ``moe_quant_format: nvfp4-pack-quantized``, ``group_size: 16``). + Returns None when no separate expert recipe is declared; raises for a + declared combination TensorRT-LLM cannot load. + """ + moe_quant_algo = hf_quant_config.get("moe_quant_algo") + moe_quant_format = hf_quant_config.get("moe_quant_format") + if moe_quant_algo is None and moe_quant_format is None: + return None + group_size = hf_quant_config.get("group_size") + if (moe_quant_algo, moe_quant_format, + group_size) != ("NVFP4", "nvfp4-pack-quantized", + NVFP4_SF_VEC_SIZE): + raise ValueError( + "Unsupported routed-expert packaging in an MXFP8 checkpoint: " + f"moe_quant_algo={moe_quant_algo!r}, moe_quant_format=" + f"{moe_quant_format!r}, group_size={group_size!r}. Supported: " + "moe_quant_algo='NVFP4', moe_quant_format=" + f"'nvfp4-pack-quantized', group_size={NVFP4_SF_VEC_SIZE}.") + return QuantConfig(quant_algo=QuantAlgo.NVFP4, + group_size=NVFP4_SF_VEC_SIZE) + + @staticmethod + def _build_minimax_m3_mxfp8_moe_quant_config(pretrained_config, + quant_config): + """Expand MXFP8 base + NVFP4 routed experts into MIXED_PRECISION. + + Used for the compressed-tensors NVFP4 MoE packaging, which has no + per-layer map. Every quantized text linear gets an explicit MXFP8 + entry, because an unmatched module would inherit the global + MIXED_PRECISION config and load unquantized; the routed experts of + each MoE layer get the coarse NVFP4 entry + ``MiniMaxM3MoE._get_experts_quant_config`` reads. Modules listed in + ``ignored_layers`` get no entry, and those names lose their + ``language_model.`` prefix so they match the TensorRT-LLM module tree. + + Returns ``(quant_config, layer_quant_config)``; a checkpoint without the + expert recipe is returned unchanged with no per-layer map. + """ + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.common import \ + get_sparse_layer_ids + hf_quant_config = getattr(pretrained_config, "quantization_config", + None) + if (quant_config is None or quant_config.quant_algo != QuantAlgo.MXFP8 + or not isinstance(hf_quant_config, dict)): + return quant_config, None + experts_quant_config = ModelConfig._get_mxfp8_moe_quant_config( + hf_quant_config) + if experts_quant_config is None: + return quant_config, None + + text_config = getattr(pretrained_config, "text_config", + None) or pretrained_config + if isinstance(text_config, dict): + text_config = SimpleNamespace(**text_config) + num_layers = int(text_config.num_hidden_layers) + moe_layer_freq = getattr(text_config, "moe_layer_freq", None) + if moe_layer_freq is None: + moe_layer_freq = [1] * num_layers + if len(moe_layer_freq) != num_layers: + raise ValueError( + f"moe_layer_freq length {len(moe_layer_freq)} does not match " + f"num_hidden_layers {num_layers}") + _, sparse_layer_ids = get_sparse_layer_ids(text_config) + has_shared_experts = int( + getattr(text_config, "n_shared_experts", 0) or 0) > 0 + + lm_prefix = "language_model." + exclude_modules = None + if quant_config.exclude_modules is not None: + exclude_modules = list( + dict.fromkeys( + m[len(lm_prefix):] if m.startswith(lm_prefix) else m + for m in quant_config.exclude_modules)) + mixed_quant_config = QuantConfig( + quant_algo=QuantAlgo.MIXED_PRECISION, + kv_cache_quant_algo=quant_config.kv_cache_quant_algo, + group_size=quant_config.group_size, + exclude_modules=exclude_modules) + base_quant_config = QuantConfig( + quant_algo=QuantAlgo.MXFP8, + kv_cache_quant_algo=quant_config.kv_cache_quant_algo, + group_size=quant_config.group_size) + experts_quant_config.kv_cache_quant_algo = quant_config.kv_cache_quant_algo + + layer_quant_config = {} + for layer_idx in range(num_layers): + prefix = f"model.layers.{layer_idx}" + entries = { + f"{prefix}.self_attn.{proj}": base_quant_config + for proj in ("q_proj", "k_proj", "v_proj", "o_proj") + } + if layer_idx in sparse_layer_ids: + entries.update({ + f"{prefix}.self_attn.{proj}": base_quant_config + for proj in ("index_q_proj", "index_k_proj") + }) + if int(moe_layer_freq[layer_idx]) != 0: + if has_shared_experts: + entries.update({ + f"{prefix}.block_sparse_moe.shared_experts.{proj}": + base_quant_config + for proj in ("gate_proj", "up_proj", "down_proj") + }) + entries[ + f"{prefix}.block_sparse_moe.experts"] = experts_quant_config + else: + entries.update({ + f"{prefix}.mlp.{proj}": base_quant_config + for proj in ("gate_proj", "up_proj", "down_proj") + }) + excluded = mixed_quant_config.is_module_excluded_from_quantization + layer_quant_config.update({ + name: config + for name, config in entries.items() if not excluded(name) + }) + + logger.info( + "Detected the compressed-tensors NVFP4 MoE packaging of MiniMax M3: " + "MXFP8 base linears with NVFP4 routed experts " + f"({len(layer_quant_config)} per-layer quant entries).") + return mixed_quant_config, layer_quant_config + + @staticmethod + def _set_minimax_m3_layer_quant_config(pretrained_config, quant_config, layer_quant_config): - """Normalize the Minimax M3 MIXED_PRECISION per-layer quant config. + """Normalize the Minimax M3 MIXED_PRECISION quant configs. + + Returns ``(quant_config, layer_quant_config)``. A checkpoint without a + per-layer map goes through + ``_build_minimax_m3_mxfp8_moe_quant_config``, which builds one for the + compressed-tensors NVFP4 MoE packaging and leaves uniform MXFP8 or BF16 + checkpoints unchanged. - Two fix-ups are applied: + For a per-layer map (ModelOpt MIXED_PRECISION), two fix-ups are applied: 1. Strip the ``language_model.`` prefix from every per-layer key. The M3 VL checkpoint stores keys like @@ -960,13 +1097,11 @@ def _set_minimax_m3_layer_quant_config(pretrained_config, entry per MoE layer so ``MiniMaxM3MoE._get_experts_quant_config`` can select the NVFP4 backend for the routed experts (the fine-grained per-linear NVFP4 expert keys can't be used directly). - - Does nothing when there is no per-layer config (e.g. the uniform MXFP8 - or a BF16 checkpoint). """ from tensorrt_llm.models.modeling_utils import QuantAlgo if layer_quant_config is None: - return layer_quant_config + return ModelConfig._build_minimax_m3_mxfp8_moe_quant_config( + pretrained_config, quant_config) # (1) Strip the ``language_model.`` prefix so exact-match per-layer # quant assignment works for standalone base Linears. @@ -983,7 +1118,7 @@ def _set_minimax_m3_layer_quant_config(pretrained_config, and v.quant_algo == QuantAlgo.NVFP4 for k, v in layer_quant_config.items()) if not has_nvfp4_experts: - return layer_quant_config + return quant_config, layer_quant_config experts_quant_config = QuantConfig() experts_quant_config.quant_algo = QuantAlgo.NVFP4 @@ -1008,7 +1143,7 @@ def _set_minimax_m3_layer_quant_config(pretrained_config, logger.info( "Detected Minimax M3 NVFP4 routed MoE checkpoint; using NVFP4 " "for routed experts.") - return layer_quant_config + return quant_config, layer_quant_config @staticmethod def load_quant_config_from_dtypes_json(dtypes_json_file, moe_backend: str): @@ -1500,9 +1635,8 @@ def _recursive_update_config(config: transformers.PretrainedConfig, require_layout=require_deepseek_v4_routed_moe_layout) if architecture in _MINIMAX_M3_ARCHITECTURES: - layer_quant_config = cls._set_minimax_m3_layer_quant_config( - pretrained_config, layer_quant_config) - + quant_config, layer_quant_config = cls._set_minimax_m3_layer_quant_config( + pretrained_config, quant_config, layer_quant_config) model_config = cls(pretrained_config=pretrained_config, quant_config=quant_config, quant_config_dict=layer_quant_config, diff --git a/tensorrt_llm/_torch/models/checkpoints/hf/minimaxm3_weight_mapper.py b/tensorrt_llm/_torch/models/checkpoints/hf/minimaxm3_weight_mapper.py index bb6d58957fd3..261559f186d4 100644 --- a/tensorrt_llm/_torch/models/checkpoints/hf/minimaxm3_weight_mapper.py +++ b/tensorrt_llm/_torch/models/checkpoints/hf/minimaxm3_weight_mapper.py @@ -13,8 +13,13 @@ # See the License for the specific language governing permissions and # limitations under the License. +import re +from collections.abc import Mapping + +import torch from torch import Tensor, nn +from tensorrt_llm._torch.models.checkpoints.base_weight_loader import ConsumableWeightsDict from tensorrt_llm._torch.models.checkpoints.hf.weight_mapper import HfWeightMapper from tensorrt_llm._torch.models.modeling_utils import register_mapper @@ -22,16 +27,105 @@ r"^(.*\.block_sparse_moe)\.e_score_correction_bias$": r"\1.gate.e_score_correction_bias", } +# Routed-expert tensors of the compressed-tensors NVFP4 MoE packaging of the +# MiniMax M3 architecture. Per projection: ``weight_packed`` (uint8, two E2M1 +# values per byte), ``weight_scale`` (E4M3 per 16 elements) and the FP32 +# scalars ``weight_global_scale`` / ``input_global_scale``, stored as the +# multiplier 2688 / amax (2688 = E4M3 max 448 * E2M1 max 6). +_PACKED_NVFP4_EXPERT_KEY = re.compile( + r"^(?P.*\.block_sparse_moe\.experts\.\d+\.w[123])\." + r"(?Pweight_packed|weight_global_scale|input_global_scale)$" +) +_NVFP4_GLOBAL_SCALE_RANGE = 448.0 * 6.0 + + +def nvfp4_weight_scale_2(weight_global_scale: Tensor) -> Tensor: + """Return the NVFP4 MoE ``weight_scale_2`` of a stored global scale. + + compressed-tensors stores ``weight_global_scale = 2688 / amax`` (E4M3 max + 448 times E2M1 max 6); the dequantization scale is its reciprocal, + ``amax / 2688``, computed in FP32. + """ + return torch.reciprocal(weight_global_scale.float()) + + +def _read_global_scale(name: str, value) -> Tensor: + """Materialize one global scale, rejecting values that would disable an expert.""" + scale = (value if isinstance(value, Tensor) else value[...]).to(torch.float32) + if scale.numel() != 1 or not bool(torch.isfinite(scale).all()) or not bool((scale > 0).all()): + raise ValueError( + f"Checkpoint tensor {name} must hold one finite positive global scale, " + f"got {scale.flatten()[:4].tolist()}." + ) + return scale + + +def translate_packed_nvfp4_experts(weights: Mapping[str, Tensor]) -> Mapping[str, Tensor]: + """Rename packed NVFP4 routed-expert tensors to the NVFP4 MoE contract. + + ``weight_packed`` becomes ``weight`` with the same bytes, and the global + scales become the ``weight_scale_2`` / ``input_scale`` divisors + (amax / 2688) the NVFP4 MoE loader reads; ``weight_scale`` keeps its name + and bytes. Only the scalar scales are materialized. Every other key passes + through, including already canonical ``weight_scale_2`` / ``input_scale``, + so a second call is a no-op. + + Raises: + ValueError: naming the checkpoint tensor, when a global scale is not + finite and positive, or when an expert's gate (w1) and up (w3) + weight scales differ: the fused gate/up GEMM applies one weight + scale per expert. + """ + matches = {} + for name in weights.keys(): + match = _PACKED_NVFP4_EXPERT_KEY.match(name) + if match is not None: + matches[name] = match + if not matches: + return weights + + renamed = {} + gate_up_scales: dict[str, dict[str, tuple[str, Tensor]]] = {} + for name in weights.keys(): + value = weights[name] + match = matches.get(name) + if match is None: + renamed[name] = value + continue + proj, kind = match["proj"], match["kind"] + if kind == "weight_packed": + renamed[f"{proj}.weight"] = value + elif kind == "weight_global_scale": + scale_2 = nvfp4_weight_scale_2(_read_global_scale(name, value)) + renamed[f"{proj}.weight_scale_2"] = scale_2 + expert, w = proj.rsplit(".", 1) + if w != "w2": + gate_up_scales.setdefault(expert, {})[w] = (name, scale_2) + else: + renamed[f"{proj}.input_scale"] = torch.reciprocal(_read_global_scale(name, value)) + + for scales in gate_up_scales.values(): + if len(scales) == 2 and not torch.equal(scales["w1"][1], scales["w3"][1]): + raise ValueError( + f"Checkpoint tensors {scales['w1'][0]} and {scales['w3'][0]} decode to " + "different weight scales; the fused gate/up expert GEMM needs one." + ) + return ConsumableWeightsDict.take_ownership(weights, renamed) + @register_mapper("HF", "MiniMaxM3SparseForCausalLM") @register_mapper("HF", "MiniMaxM3SparseForConditionalGeneration") class MiniMaxM3HfWeightMapper(HfWeightMapper): - """Handle M3 gate naming and MXFP8 GQA duplication for loader v2.""" + """Handle M3 gate naming, MXFP8 GQA duplication and packed NVFP4 experts.""" def __init__(self) -> None: super().__init__() self.params_map = MINIMAX_M3_PARAMS_MAP + def preprocess_weights(self, weights: Mapping[str, Tensor]) -> Mapping[str, Tensor]: + """See :func:`translate_packed_nvfp4_experts`.""" + return translate_packed_nvfp4_experts(weights) + def _duplicate_kv_weights( self, module: nn.Module, new_name: str, weights: dict[str, Tensor] ) -> dict[str, Tensor]: diff --git a/tensorrt_llm/_torch/models/modeling_minimaxm3.py b/tensorrt_llm/_torch/models/modeling_minimaxm3.py index f17b64016231..a13b1a9d13a8 100644 --- a/tensorrt_llm/_torch/models/modeling_minimaxm3.py +++ b/tensorrt_llm/_torch/models/modeling_minimaxm3.py @@ -50,7 +50,11 @@ _gather_paged_batched, _write_main_kv_slots_to_pool, ) -from ..attention.backends.sparse.minimax_m3.common import index_head_range +from ..attention.backends.sparse.minimax_m3.common import ( + get_sparse_disable_index_value_layer_ids, + get_sparse_layer_ids, + index_head_range, +) from ..attention.backends.sparse.params import SparseBackendForwardArgs from ..distributed import AllReduce, AllReduceFusionOp, AllReduceParams, MiniMaxAllReduceRMS from ..modules.decoder_layer import DecoderLayer @@ -72,7 +76,11 @@ from ..speculative import SpecMetadata from ..utils import AuxStreamType, EventType, get_model_extra_attrs, is_torch_compiling from .checkpoints.base_weight_mapper import BaseWeightMapper -from .checkpoints.hf.minimaxm3_weight_mapper import MINIMAX_M3_PARAMS_MAP, MiniMaxM3HfWeightMapper +from .checkpoints.hf.minimaxm3_weight_mapper import ( + MINIMAX_M3_PARAMS_MAP, + MiniMaxM3HfWeightMapper, + translate_packed_nvfp4_experts, +) from .modeling_speculative import SpecDecOneEngineForCausalLM from .modeling_utils import DecoderModel, ModelConfig, filter_weights, register_auto_model @@ -426,39 +434,6 @@ def _validate_sparse_attention_runtime_config( ) -def get_sparse_layer_ids(text_config: PretrainedConfig) -> Tuple[List[int], List[int]]: - """Return ``(dense_layer_ids, sparse_layer_ids)`` for the M3 text model.""" - sparse_cfg = getattr(text_config, "sparse_attention_config", None) - n_layers = int(text_config.num_hidden_layers) - if sparse_cfg is None: - return list(range(n_layers)), [] - if not sparse_cfg.get("use_sparse_attention", True): - return list(range(n_layers)), [] - freq = sparse_cfg.get("sparse_attention_freq") - if freq is None: - return list(range(n_layers)), [] - if len(freq) != n_layers: - raise ValueError( - f"sparse_attention_freq length {len(freq)} does not match num_hidden_layers {n_layers}" - ) - dense = [i for i, f in enumerate(freq) if int(f) == 0] - sparse = [i for i, f in enumerate(freq) if int(f) != 0] - return dense, sparse - - -def get_sparse_disable_index_value_layer_ids( - text_config: PretrainedConfig, -) -> List[int]: - """Return layer ids where the sparse index-value branch is disabled.""" - sparse_cfg = getattr(text_config, "sparse_attention_config", None) - if sparse_cfg is None: - return [] - flags = sparse_cfg.get("sparse_disable_index_value") - if flags is None: - return [] - return [i for i, f in enumerate(flags) if int(f) != 0] - - def get_moe_layer_ids(text_config: PretrainedConfig) -> Tuple[List[int], List[int]]: """Return ``(dense_mlp_layer_ids, moe_layer_ids)`` for the M3 text model.""" freq = getattr(text_config, "moe_layer_freq", None) @@ -474,6 +449,36 @@ def get_moe_layer_ids(text_config: PretrainedConfig) -> Tuple[List[int], List[in return dense, moe +def _get_index_projection_quant_config( + model_config: "ModelConfig[PretrainedConfig]", layer_idx: Optional[int] +) -> Optional[QuantConfig]: + """Return the quant config the checkpoint declares for the index projections. + + ``index_q_proj`` and ``index_k_proj`` load into one fused ``index_qk_proj`` + that ``apply_layerwise_quant_config`` cannot match by name, so their + per-layer entries are read here. Without such entries the projection + stays unquantized, as before. + """ + quant_config_dict = getattr(model_config, "quant_config_dict", None) + if quant_config_dict is None or layer_idx is None: + return None + prefix = f"model.layers.{layer_idx}.self_attn" + q_config = quant_config_dict.get(f"{prefix}.index_q_proj") + k_config = quant_config_dict.get(f"{prefix}.index_k_proj") + if q_config is None and k_config is None: + return None + if ( + q_config is None + or k_config is None + or (q_config.quant_algo, q_config.group_size) != (k_config.quant_algo, k_config.group_size) + ): + raise ValueError( + f"{prefix}.index_q_proj and {prefix}.index_k_proj load into one fused " + f"projection and need the same quantization, got {q_config} and {k_config}." + ) + return q_config + + # Outside ``language_model.*``: deferred multimodal branches. _VL_PREFIXES = ("vision_tower.", "multi_modal_projector.", "patch_merge_mlp.") @@ -583,8 +588,16 @@ def __init__( ) def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: - # Router runs in fp32 to match SGLang. - return torch.nn.functional.linear(hidden_states.to(torch.float32), self.weight) + # Router runs in fp32 to match SGLang. cublas_mm always computes with + # CUBLAS_COMPUTE_32F; F.linear follows torch's fp32 matmul precision, + # which is TF32 under TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 (set in the + # NGC PyTorch based containers) and then flips near-tied top-k choices. + return torch.ops.trtllm.cublas_mm( + hidden_states.to(torch.float32).contiguous(), + self.weight.t(), + bias=None, + out_dtype=torch.float32, + ) def load_weights(self, weights: List[Dict]): """Load the router weight and the e_score_correction_bias. @@ -1176,7 +1189,9 @@ def __init__( dtype=config.torch_dtype, mapping=self.qkv_proj.mapping, tensor_parallel_mode=TensorParallelMode.COLUMN, - quant_config=None, + # Quantized when the checkpoint declares per-layer entries + # for the index projections; None keeps them unquantized. + quant_config=_get_index_projection_quant_config(model_config, layer_idx), weights_loading_config=WeightsLoadingConfig( weight_mode=WeightMode.FUSED_GATE_UP_LINEAR ), @@ -2897,6 +2912,9 @@ def load_weights( params_map: Optional[Dict[str, str]] = None, allow_partial_loading: bool = False, ) -> None: + # Rename packed NVFP4 routed-expert tensors to the NVFP4 MoE contract + # (a no-op for other packagings and for already rewritten weights). + weights = translate_packed_nvfp4_experts(weights) # Pack the opt-in five-way projection with MiniMax-specific TP # sharding before the generic mapper handles the compatibility path. packed_projection_modules = _load_qkv_index_proj_weights(self, weights) @@ -3097,6 +3115,8 @@ def load_weights( # set of unknown top-level keys it surfaces as errors. from .modeling_minimaxm3_vl import load_minimax_m3_vl_state_dict, split_multimodal_weights + # Rewrite before the prefix strip so errors name the checkpoint tensor. + weights = translate_packed_nvfp4_experts(weights) text_or_other_weights, vision_weights = split_multimodal_weights(weights) text_weights, ignored = _strip_language_model_prefix(text_or_other_weights) @@ -3108,6 +3128,12 @@ def load_weights( ) self.last_loaded_vision_keys = loaded self.last_missing_vision_keys = missing + logger.info( + f"MiniMax-M3 VL weights: {len(self.last_loaded_vision_keys)} vision tensors " + f"loaded, {len(self.last_missing_vision_keys)} vision parameters missing, " + f"{len(ignored)} checkpoint tensors skipped " + f"({sorted(set(ignored.values()))})." + ) super().load_weights( weights=text_weights, diff --git a/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py b/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py index 024e7ae68b45..5e56482ca385 100644 --- a/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py +++ b/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py @@ -1760,7 +1760,7 @@ def __init__( **kwargs, ) - from transformers import AutoProcessor, AutoTokenizer + from transformers import AutoTokenizer self._tokenizer = ( tokenizer @@ -1769,20 +1769,18 @@ def __init__( ) self._model_path = model_path self._config = config - self._processor = AutoProcessor.from_pretrained( - model_path, - use_fast=self._use_fast, - trust_remote_code=trust_remote_code, - ) + # Loaded on the first image/video request; text requests need only + # the tokenizer (see ``processor``). + self._processor = None # The HF processor tokenizes with add_special_tokens=True and the cache with False, # so their ids match only if the tokenizer adds no special tokens. - if enable_tokenization_cache and self._processor.tokenizer.num_special_tokens_to_add() != 0: + if enable_tokenization_cache and self._tokenizer.num_special_tokens_to_add() != 0: logger.warning( "enable_tokenization_cache is ignored: the MiniMax-M3 tokenizer adds " "special tokens, so cached ids would differ from the HF processor's." ) enable_tokenization_cache = False - self._init_tokenization_cache(enable_tokenization_cache, self._processor.tokenizer) + self._init_tokenization_cache(enable_tokenization_cache, self._tokenizer) text_cfg = getattr(config, "text_config", None) if isinstance(text_cfg, dict): self._dtype = getattr(text_cfg, "torch_dtype", torch.bfloat16) @@ -1830,6 +1828,32 @@ def _resolve_token_id(self, token: str) -> int: # ----- registry contract properties ----------------------------------- @property def processor(self): + """HF processor for image/video requests, loaded on first use. + + ``AutoProcessor`` returns a plain tokenizer when it cannot build the + checkpoint's processor (no processor code the installed transformers + can load). That cannot preprocess images, so it is rejected here with + an actionable error rather than failing inside the request. + """ + if self._processor is None: + from transformers import AutoProcessor + + processor = AutoProcessor.from_pretrained( + self._model_path, + use_fast=self._use_fast, + trust_remote_code=self._trust_remote_code, + ) + if getattr(processor, "image_processor", None) is None: + raise ValueError( + "MiniMax-M3 image and video inputs need an HF processor with an " + f"image processor, but AutoProcessor.from_pretrained({self._model_path!r}) " + f"returned {type(processor).__name__}, which has none: the checkpoint " + "ships no processor this transformers version can load. Text-only " + "requests are unaffected. For image or video requests, serve a model " + "directory whose processor loads with AutoProcessor (for example, " + "processor code registered through an auto_map entry)." + ) + self._processor = processor return self._processor @property @@ -1896,7 +1920,7 @@ def _hw(media: Any) -> Tuple[int, int]: def get_num_tokens_per_image(self, *, image: Any, **kwargs: Any) -> int: """Token count per image: ``(grid_h*grid_w)/merge^2 + 2`` (+VS/VE framing).""" - ip = self._processor.image_processor + ip = self.processor.image_processor h, w = self._hw(image) merge = int(ip.merge_size) return ip.get_number_of_image_patches(h, w) // (merge * merge) + 2 @@ -1922,7 +1946,7 @@ def get_num_tokens_per_video( grid = ( video_grid_thw.tolist() if isinstance(video_grid_thw, torch.Tensor) else video_grid_thw ) - merge = int(self._processor.image_processor.merge_size) + merge = int(self.processor.image_processor.merge_size) grid_t, grid_h, grid_w = int(grid[0]), int(grid[1]), int(grid[2]) return grid_t * (grid_h // merge) * (grid_w // merge) + 2 @@ -2002,19 +2026,37 @@ def call_with_text_prompt( 1. Pull ``prompt`` (text) and ``multi_modal_data`` (dict with optional ``image`` / ``video`` lists) from the ``inputs`` mapping. - 2. Drive the checkpoint's HF processor (loaded via AutoProcessor; - backed by ``processing_minimax.MiniMaxVLProcessor`` on the - real checkpoint) to expand placeholders and produce - ``pixel_values`` / ``image_grid_thw`` (and the video - equivalents). + 2. Drive the checkpoint's HF processor (see ``processor``: loaded via + AutoProcessor on the first image or video request) to expand + placeholders and produce ``pixel_values`` / ``image_grid_thw`` + (and the video equivalents). 3. Return ``(input_ids_list, {"multimodal_data": {...}})`` in the shape TRT-LLM's runtime expects. + + Text-only requests are tokenized like the default input processor and + never load the HF processor. """ text_prompt = inputs.get("prompt") if isinstance(inputs, Mapping) else None mm_data = (inputs.get("multi_modal_data") if isinstance(inputs, Mapping) else None) or {} images = self._coerce_to_list(mm_data.get("image")) video_inputs = self._coerce_to_list(mm_data.get("video")) + if not images and not video_inputs: + ids = self._encode_with_tokenization_cache(text_prompt or "", sampling_params) + if ids is not None: + return ids, None + truncate_prompt_tokens = getattr(sampling_params, "truncate_prompt_tokens", None) + kwargs = ( + {} + if truncate_prompt_tokens is None + else dict(truncation=True, max_length=truncate_prompt_tokens) + ) + token_ids = self._tokenizer.encode( + text_prompt or "", + add_special_tokens=getattr(sampling_params, "add_special_tokens", True), + **kwargs, + ) + return list(token_ids), None # The HF MiniMaxVLProcessor expects video frames as nested lists # of frames (one list per video). Accept either pre-extracted @@ -2061,15 +2103,10 @@ def call_with_text_prompt( explicit.extend(["]<]image[>["] * len(images)) explicit.extend(["]<]video[>["] * len(videos)) templated_text = "\n".join(explicit) - else: - templated_text = text_prompt or "" - ids = self._encode_with_tokenization_cache(templated_text, sampling_params) - if ids is not None: - return ids, {"multimodal_data": {}} # Run the HF processor. ``return_tensors='pt'`` yields tensors # in the BatchFeature output. - processed = self._processor( + processed = self.processor( text=[templated_text], images=images if images else None, videos=videos if videos else None, diff --git a/tensorrt_llm/_torch/modules/linear.py b/tensorrt_llm/_torch/modules/linear.py index 26d09566eba2..0b229654a59c 100644 --- a/tensorrt_llm/_torch/modules/linear.py +++ b/tensorrt_llm/_torch/modules/linear.py @@ -3679,19 +3679,40 @@ def load_weights_vanilla(self, module: Linear, weights: List[Dict]) -> None: self._store_scale(module, scales[0]) self.load_kv_cache_scales(module, weights) + def _load_fused_weight_scales(self, module: Linear, weights: List[Dict], + shard_names: tuple[str, ...]) -> torch.Tensor: + """Load the scales of fused shards with the rows of their weights. + + Scale rows share the out_features dim with weight rows, so each + shard's scale is sliced by ``Linear.load_shard`` exactly like its + weight, including uneven or replicated ``override_tp_sharding`` + ranges. + """ + scale_name = self._get_scale_name(weights) + scales = [] + for shard_weights, shard_name in zip(weights, shard_names): + scale = module.load_shard(shard_weights, + scale_name, + device=torch.device("cuda"), + name=shard_name) + if scale is None: + raise ValueError( + f"MXFP8 fused load is missing {scale_name} for shard " + f"'{shard_name}'") + assert scale.dtype == torch.uint8, ( + f"MXFP8 weight_scale must be uint8 (UE8M0), got {scale.dtype}") + scales.append(scale.contiguous()) + # Concatenate along dim 0 (out_features), the axis of the weights. + return torch.cat(scales, dim=0) + def load_weights_fused_qkv_linear(self, module: Linear, weights: List[Dict]) -> None: q_weight, k_weight, v_weight = load_weights_fused_qkv_helper( module, weights) copy_weight(module.weight, torch.cat((q_weight, k_weight, v_weight))) - - scales = self.load_weight_scales(weights, - tp_size=module.tp_size, - tp_rank=module.tp_rank, - tp_mode=module.tp_mode) - # Scales share the out_features dim with weights; concatenate along - # dim 0 (out_features), same axis the weights are concatenated on. - self._store_scale(module, torch.cat(scales, dim=0)) + self._store_scale( + module, + self._load_fused_weight_scales(module, weights, ("q", "k", "v"))) self.load_kv_cache_scales(module, weights) @@ -3712,12 +3733,9 @@ def load_weights_fused_gate_up_linear(self, module: Linear, gate_weight, up_weight = load_weights_fused_gate_up_helper( module, weights) copy_weight(module.weight, torch.cat((gate_weight, up_weight))) - - scales = self.load_weight_scales(weights, - tp_size=module.tp_size, - tp_rank=module.tp_rank, - tp_mode=module.tp_mode) - self._store_scale(module, torch.cat(scales, dim=0)) + self._store_scale( + module, + self._load_fused_weight_scales(module, weights, ("gate", "up"))) def get_quant_method(quant_config: Optional[QuantConfig] = None): diff --git a/tests/integration/test_lists/test-db/l0_cpu.yml b/tests/integration/test_lists/test-db/l0_cpu.yml index 7e911ce59cdd..d96d4b294820 100644 --- a/tests/integration/test_lists/test-db/l0_cpu.yml +++ b/tests/integration/test_lists/test-db/l0_cpu.yml @@ -83,6 +83,12 @@ l0_cpu: - unittest/_torch/models/test_minimax_m3.py::test_piecewise_attention_boundary_preserves_indexer_cache_contract - unittest/_torch/models/test_minimax_m3.py::test_minimax_m3_five_way_projection_shard_geometry - unittest/_torch/models/test_minimax_m3.py::test_minimax_m3_five_way_loader_returns_exact_generic_skip + # MiniMax-M3 packaged-checkpoint loading and tokenizer-only CPU contracts. + - unittest/_torch/models/test_minimax_m3.py::test_minimax_m3_fold_gemma_boundary_weights_accepts_partial_lazy_load + - unittest/_torch/models/test_minimax_m3.py::test_minimax_m3_index_projection_quantization_follows_checkpoint_entries + - unittest/_torch/models/test_minimax_m3_vl.py::test_text_request_needs_no_hf_processor + - unittest/_torch/models/test_minimax_m3_vl.py::test_tokenizer_from_auto_processor_is_not_image_support + - unittest/_torch/models/test_minimax_m3_vl.py::test_image_request_uses_a_valid_processor - unittest/_torch/models/checkpoints - unittest/_torch/modules - unittest/_torch/moe diff --git a/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml b/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml index c335bf4ac973..8bfb1d52ee4b 100644 --- a/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml +++ b/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml @@ -71,6 +71,10 @@ l0_gb300_multi_gpus: # Kimi K3 MoE routing shape on the TRTLLM backend (random weights, no # checkpoint; the case moved out of the skipped TestMoeFp4 class). - unittest/_torch/moe/test_moe_backend.py -k "kimi_k3_shape" + # MiniMax-M3 compressed-tensors NVFP4 MoE packaging: expert loading and + # eager/graph numerics on CUTLASS, MXFP8 index projections per TP rank. + - unittest/_torch/models/test_minimax_m3.py::test_packaged_nvfp4_experts_load_and_run_on_cutlass + - unittest/_torch/models/test_minimax_m3.py::test_packaged_index_projection_keeps_mxfp8_scales_per_tp4_rank # Kimi K3 truncated-model (first 4 layers) e2e SA spec-dec logits parity; # resolves /Kimi-K3 and FAILS if the checkpoint is not # staged (deliberate: a skipped post-merge entry is indistinguishable diff --git a/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_shared_draft_layers.py b/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_shared_draft_layers.py index 3364da7089a5..54a13032dd59 100644 --- a/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_shared_draft_layers.py +++ b/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_shared_draft_layers.py @@ -449,3 +449,110 @@ class CacheSetupReached(Exception): setup.assert_not_called() else: setup.assert_called_once() + + +# Bytes per token for local layers 0/1/2 (dense global 11, sparse global 27, +# shared Eagle global 60), each with 4 KV heads x head_dim 128 (K + V), plus a +# 128-wide index-K on the sparse layer: 256 B in BF16, 128 B in FP8. +# BF16 K/V is 2048 B per layer and FP8 K/V is 1024 B. With NVFP4 only the +# sparse layer is NVFP4 (512 B data + 64 B FP8 block scales); the dense and +# Eagle layers keep FP8 K/V (1024 B), the Eagle layer on P32 physical pages. +_M3_LAYER_BYTES_PER_TOKEN = { + (DataType.BF16, "bf16"): [2048, 2048 + 256, 2048], + (DataType.BF16, "fp8"): [2048, 2048 + 128, 2048], + (DataType.FP8, "bf16"): [1024, 1024 + 256, 1024], + (DataType.FP8, "fp8"): [1024, 1024 + 128, 1024], + (DataType.NVFP4, "bf16"): [1024, 576 + 256, 1024], + (DataType.NVFP4, "fp8"): [1024, 576 + 128, 1024], +} + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("main_dtype", [DataType.BF16, DataType.FP8, DataType.NVFP4]) +@pytest.mark.parametrize("index_dtype", ["bf16", "fp8"]) +def test_m3_budget_estimate_covers_actual_paged_buffers(main_dtype, index_dtype): + manager = MiniMaxM3KVCacheManagerV2.__new__(MiniMaxM3KVCacheManagerV2) + manager.dtype = main_dtype + manager.kv_cache_type = CacheType.SELF + manager.kv_factor = 2 + manager.num_kv_heads_per_layer = [4, 4, 4] + manager.head_dim_per_layer = [128, 128, 128] + manager.num_local_layers = 3 + # Nonidentity PP mapping: local layers 0/1/2 are global layers 11/27/60. + manager.pp_layers = [11, 27, 60] + manager.layer_offsets = {11: 0, 27: 1, 60: 2} + manager.sparse_layer_ids = {27} + manager.disable_index_value_layer_ids = {27} + manager._shared_draft_layer_ids = [60] + manager.sparse_index_dim = 128 + manager.indexer_kv_dtype = index_dtype + manager.tokens_per_block = 128 + manager._ledger_tokens_per_block = 128 + manager._generation_kv_capacity_headroom = 1 + manager.max_num_tokens = 8192 + manager.max_seq_len = 8192 + manager.max_batch_size = 1 + manager.max_beam_width = 1 + manager.enable_partial_reuse = False + manager.max_draft_len = 0 + manager.max_attention_window_vec = [None, None, None] + manager.max_cuda_graph_batch_size = None + manager.num_extra_kv_tokens = 0 + manager.enable_swa_scratch_reuse = False + manager.enable_block_reuse = False + manager.block_reuse_policy = kv_cache_manager_v2.BlockReusePolicy.ALL_REUSABLE + manager.reuse_match_backoff = 0 + manager.enable_stats = False + manager._has_cp_helix = False + + config = manager._build_base_config( + SimpleNamespace( + pool_ratio=[1.0], + enable_block_reuse=False, + max_util_for_resume=0.9, + enable_partial_reuse=False, + ), + tokens_per_block=manager.tokens_per_block, + cache_tiers=[kv_cache_manager_v2.GpuCacheTierConfig(quota=1 << 20)], + ) + config = manager._build_cache_config(config) + config = manager._remove_zero_size_buffers(config) + + # Independently total final allocator descriptors. Eagle P32 buffers use + # their physical-page override; all other buffers use the ledger P128. + actual_bytes_per_token = [] + for layer in config.layers: + layer_bytes = 0 + for buffer in layer.buffers: + tokens_per_page = buffer.tokens_per_block_override or manager.tokens_per_block + assert buffer.size % tokens_per_page == 0 + layer_bytes += buffer.size // tokens_per_page + actual_bytes_per_token.append(layer_bytes) + expected_layer_bytes = _M3_LAYER_BYTES_PER_TOKEN[(main_dtype, index_dtype)] + assert actual_bytes_per_token == expected_layer_bytes + estimated_layer_bytes, _ = manager._get_runtime_cache_size_layer_components() + assert estimated_layer_bytes == expected_layer_bytes + assert [ + sum(buffer.role == Role.INDEX_KEY for buffer in layer.buffers) for layer in config.layers + ] == [0, 1, 0] + + expected_index_bytes = 128 * (2 if index_dtype == "bf16" else 1) + assert manager.get_layer_bytes_per_token(1, Role.INDEX_KEY) == expected_index_bytes + assert manager.get_layer_bytes_per_token(0, Role.INDEX_KEY) == 0 + assert manager.get_layer_bytes_per_token(2, Role.INDEX_KEY) == 0 + for role in (Role.KEY, Role.VALUE): + assert manager.get_layer_bytes_per_token(1, role) == ( + KVCacheManagerV2.get_layer_bytes_per_token(manager, 1, role) + ) + if main_dtype == DataType.NVFP4: + for role in (Role.KEY_BLOCK_SCALE, Role.VALUE_BLOCK_SCALE): + assert manager.get_layer_bytes_per_token(1, role) == ( + KVCacheManagerV2.get_layer_bytes_per_token(manager, 1, role) + ) + scale_roles = {Role.KEY_BLOCK_SCALE, Role.VALUE_BLOCK_SCALE} + assert all(buffer.role not in scale_roles for buffer in config.layers[0].buffers) + assert all(buffer.role not in scale_roles for buffer in config.layers[2].buffers) + + configured_footprint = sum(actual_bytes_per_token) * manager.max_num_tokens + estimated_quota = manager._get_quota_from_max_tokens(manager.max_num_tokens) + assert estimated_quota >= configured_footprint diff --git a/tests/unittest/_torch/models/checkpoints/hf/test_minimaxm3_weight_mapper.py b/tests/unittest/_torch/models/checkpoints/hf/test_minimaxm3_weight_mapper.py index 160bb485d5f8..d4ac9bee2469 100644 --- a/tests/unittest/_torch/models/checkpoints/hf/test_minimaxm3_weight_mapper.py +++ b/tests/unittest/_torch/models/checkpoints/hf/test_minimaxm3_weight_mapper.py @@ -14,6 +14,9 @@ # limitations under the License. import inspect +import math +import re +import struct from types import SimpleNamespace from unittest.mock import patch @@ -22,8 +25,11 @@ from tensorrt_llm._torch.model_config import ModelConfig from tensorrt_llm._torch.models.checkpoints.auto_mapper import AutoCheckpointMapper +from tensorrt_llm._torch.models.checkpoints.base_weight_loader import ConsumableWeightsDict from tensorrt_llm._torch.models.checkpoints.hf.minimaxm3_weight_mapper import ( + MINIMAX_M3_PARAMS_MAP, MiniMaxM3HfWeightMapper, + nvfp4_weight_scale_2, ) from tensorrt_llm._torch.models.checkpoints.hf.weight_mapper import HfWeightMapper from tensorrt_llm._torch.models.modeling_minimaxm3 import ( @@ -201,3 +207,210 @@ def test_load_weights_accepts_base_mapper_without_params_map() -> None: assert call_kwargs["weight_mapper"] is mapper renamed = mapper.rename_by_params_map(call_kwargs["params_map"], {source_name: bias}) assert renamed[target_name] is bias + + +# --------------------------------------------------------------------------- +# Compressed-tensors NVFP4 MoE packaging of the MiniMax M3 architecture +# --------------------------------------------------------------------------- + +# Checkpoint naming of one routed expert in the multimodal checkpoint layout. +_EXPERT = "language_model.model.layers.3.block_sparse_moe.experts.7" + +# weight_scale_2 = fp32(fp32(bf16(2688 / gs) * 2688) * fp32(1 / 2688**2)) as +# float32 bit patterns. Computed outside PyTorch: Python float arithmetic +# rounded to float32 after every operation (exact for these operations), BF16 +# by round-to-nearest-even on the float32 bits. +# FP32 reciprocals of the stored weight_global_scale values. +_WEIGHT_SCALE_2_GOLDENS = { + 2688.0: 0x39C30C31, + 3000.0: 0x39AEC33E, + 2913.75: 0x39B3EF92, + 16999.5: 0x3876BB24, + 10000.0: 0x38D1B717, +} +_RECIPROCAL_GOLDENS = {1.0: 0x3F800000, 0.5: 0x40000000, 3.0: 0x3EAAAAAB} + + +def _f32_bits(value) -> int: + return struct.unpack(" dict[str, torch.Tensor]: + """Synthetic tensors of one packed NVFP4 expert, shaped like the checkpoint's.""" + generator = torch.Generator().manual_seed(0) + weights = {} + # (rows, packed bytes per row) for hidden 64, intermediate 32: w1/w3 are + # [inter, hidden], w2 is [hidden, inter]; two E2M1 values per byte. + for proj, (rows, packed_cols) in {"w1": (32, 32), "w2": (64, 16), "w3": (32, 32)}.items(): + prefix = f"{_EXPERT}.{proj}" + weights[f"{prefix}.weight_packed"] = torch.randint( + 0, 256, (rows, packed_cols), dtype=torch.uint8, generator=generator + ) + weights[f"{prefix}.weight_scale"] = ( + torch.rand(rows, packed_cols * 2 // 16, generator=generator) * 4 + ).to(torch.float8_e4m3fn) + weights[f"{prefix}.weight_global_scale"] = torch.tensor( + down_global_scale if proj == "w2" else gate_up_global_scale + ) + weights[f"{prefix}.input_global_scale"] = torch.tensor(input_global_scale) + return weights + + +@pytest.mark.parametrize("global_scale", sorted(_WEIGHT_SCALE_2_GOLDENS)) +def test_nvfp4_weight_scale_2_matches_independent_goldens(global_scale: float) -> None: + scale_2 = nvfp4_weight_scale_2(torch.tensor(global_scale)) + assert scale_2.dtype == torch.float32 + assert _f32_bits(scale_2) == _WEIGHT_SCALE_2_GOLDENS[global_scale] + + +def test_packed_nvfp4_expert_translates_to_nvfp4_moe_contract() -> None: + source = _packed_expert() + unrelated = { + "language_model.model.layers.3.self_attn.q_proj.weight": torch.ones(4, 4), + "language_model.model.layers.3.block_sparse_moe.gate.weight": torch.ones(4, 4), + } + out = MiniMaxM3HfWeightMapper().preprocess_weights({**source, **unrelated}) + + for name, value in unrelated.items(): + assert out[name] is value + for proj in ("w1", "w2", "w3"): + prefix = f"{_EXPERT}.{proj}" + for stale in ("weight_packed", "weight_global_scale", "input_global_scale"): + assert f"{prefix}.{stale}" not in out + # Packed FP4 bytes and E4M3 group scales are passed through, not copied or cast. + assert out[f"{prefix}.weight"] is source[f"{prefix}.weight_packed"] + assert out[f"{prefix}.weight"].dtype == torch.uint8 + assert out[f"{prefix}.weight_scale"] is source[f"{prefix}.weight_scale"] + expected = _WEIGHT_SCALE_2_GOLDENS[2913.75 if proj == "w2" else 3000.0] + assert _f32_bits(out[f"{prefix}.weight_scale_2"]) == expected + assert out[f"{prefix}.weight_scale_2"].shape == () + assert _f32_bits(out[f"{prefix}.input_scale"]) == _RECIPROCAL_GOLDENS[1.0] + assert len(out) == len(source) + len(unrelated) + + +@pytest.mark.parametrize("input_global_scale", sorted(_RECIPROCAL_GOLDENS)) +def test_packed_nvfp4_input_scale_is_the_reciprocal(input_global_scale: float) -> None: + out = MiniMaxM3HfWeightMapper().preprocess_weights( + _packed_expert(input_global_scale=input_global_scale) + ) + for proj in ("w1", "w2", "w3"): + assert ( + _f32_bits(out[f"{_EXPERT}.{proj}.input_scale"]) + == _RECIPROCAL_GOLDENS[input_global_scale] + ) + + +@pytest.mark.parametrize("kind", ["weight_global_scale", "input_global_scale"]) +@pytest.mark.parametrize("bad", [0.0, -1.0, math.nan, math.inf, -math.inf]) +def test_packed_nvfp4_rejects_invalid_global_scale(kind: str, bad: float) -> None: + weights = _packed_expert() + name = f"{_EXPERT}.w2.{kind}" + weights[name] = torch.tensor(bad) + with pytest.raises(ValueError, match=re.escape(name)): + MiniMaxM3HfWeightMapper().preprocess_weights(weights) + + +def test_packed_nvfp4_rejects_different_gate_up_weight_scales() -> None: + weights = _packed_expert() + weights[f"{_EXPERT}.w3.weight_global_scale"] = torch.tensor(3100.0) + with pytest.raises(ValueError, match=re.escape(f"{_EXPERT}.w3.weight_global_scale")): + MiniMaxM3HfWeightMapper().preprocess_weights(weights) + + +def test_canonical_nvfp4_expert_tensors_are_not_inverted_again() -> None: + canonical = {} + for proj in ("w1", "w2", "w3"): + prefix = f"model.layers.3.block_sparse_moe.experts.0.{proj}" + canonical[f"{prefix}.weight"] = torch.zeros(8, 32, dtype=torch.uint8) + canonical[f"{prefix}.weight_scale"] = torch.ones(8, 4).to(torch.float8_e4m3fn) + canonical[f"{prefix}.weight_scale_2"] = torch.tensor(1.0 / 3000.0) + canonical[f"{prefix}.input_scale"] = torch.tensor(0.25) + out = MiniMaxM3HfWeightMapper().preprocess_weights(canonical) + assert out is canonical + + +def test_packed_nvfp4_translation_is_idempotent() -> None: + mapper = MiniMaxM3HfWeightMapper() + once = mapper.preprocess_weights(_packed_expert()) + assert mapper.preprocess_weights(once) is once + + +def test_packed_nvfp4_translation_takes_over_consumable_weights() -> None: + source = _packed_expert() + packed = source[f"{_EXPERT}.w1.weight_packed"] + consumable = ConsumableWeightsDict(dict(source)) + out = MiniMaxM3HfWeightMapper().preprocess_weights(consumable) + assert isinstance(out, ConsumableWeightsDict) + # The translated dict now owns the tensors, so consuming them frees memory. + assert len(consumable) == 0 + assert out[f"{_EXPERT}.w1.weight"] is packed + assert out.mark_consumed(_EXPERT) == len(source) + + +def _bare_model(model_class: type) -> torch.nn.Module: + config = SimpleNamespace(num_key_value_heads=_NUM_KV_HEADS, num_attention_heads=8) + model = object.__new__(model_class) + torch.nn.Module.__init__(model) + model.model_config = ModelConfig(pretrained_config=config, mapping=Mapping()) + model.model = torch.nn.Module() + model.model.layers = torch.nn.ModuleList() + model.spec_config = None + if model_class is MiniMaxM3VLForConditionalGeneration: + model.vision_tower = None + model.last_loaded_vision_keys = [] + model.last_missing_vision_keys = [] + return model + + +def test_vl_load_weights_translates_packed_experts_before_loading() -> None: + model = _bare_model(MiniMaxM3VLForConditionalGeneration) + with patch.object(DecoderModelForCausalLM, "load_weights") as base_load_weights: + model.load_weights(_packed_expert(), weight_mapper=MiniMaxM3HfWeightMapper()) + + loaded = base_load_weights.call_args.kwargs["weights"] + expert = _EXPERT[len("language_model.") :] + assert sorted(loaded.keys()) == sorted( + f"{expert}.{proj}.{kind}" + for proj in ("w1", "w2", "w3") + for kind in ("weight", "weight_scale", "weight_scale_2", "input_scale") + ) + assert _f32_bits(loaded[f"{expert}.w1.weight_scale_2"]) == _WEIGHT_SCALE_2_GOLDENS[3000.0] + assert _f32_bits(loaded[f"{expert}.w2.weight_scale_2"]) == _WEIGHT_SCALE_2_GOLDENS[2913.75] + + +def test_vl_load_weights_names_the_checkpoint_tensor_of_an_invalid_scale() -> None: + model = _bare_model(MiniMaxM3VLForConditionalGeneration) + weights = _packed_expert() + name = f"{_EXPERT}.w2.input_global_scale" + weights[name] = torch.tensor(0.0) + with patch.object(DecoderModelForCausalLM, "load_weights") as base_load_weights: + with pytest.raises(ValueError, match=re.escape(name)): + model.load_weights(weights, weight_mapper=HfWeightMapper()) + base_load_weights.assert_not_called() + + +def test_text_load_weights_translates_packed_experts_with_any_mapper() -> None: + # The translation does not depend on the mapper driving the load, and the + # M3 params map still reaches the base loader. + model = _bare_model(MiniMaxM3ForCausalLM) + expert = _EXPERT[len("language_model.") :] + weights = {k[len("language_model.") :]: v for k, v in _packed_expert().items()} + mapper = HfWeightMapper() + with patch.object(DecoderModelForCausalLM, "load_weights") as base_load_weights: + model.load_weights(weights, weight_mapper=mapper) + + call_kwargs = base_load_weights.call_args.kwargs + assert call_kwargs["weight_mapper"] is mapper + assert call_kwargs["params_map"] == MINIMAX_M3_PARAMS_MAP + loaded = call_kwargs["weights"] + assert sorted(loaded.keys()) == sorted( + f"{expert}.{proj}.{kind}" + for proj in ("w1", "w2", "w3") + for kind in ("weight", "weight_scale", "weight_scale_2", "input_scale") + ) + assert loaded[f"{expert}.w3.weight"] is weights[f"{expert}.w3.weight_packed"] + assert _f32_bits(loaded[f"{expert}.w3.weight_scale_2"]) == _WEIGHT_SCALE_2_GOLDENS[3000.0] diff --git a/tests/unittest/_torch/models/test_minimax_m3.py b/tests/unittest/_torch/models/test_minimax_m3.py index 0638b5ba3685..9a8d704ccf14 100644 --- a/tests/unittest/_torch/models/test_minimax_m3.py +++ b/tests/unittest/_torch/models/test_minimax_m3.py @@ -46,6 +46,7 @@ ) from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.cache_manager import ( MiniMaxM3KVCacheManagerV2, + resolve_sparse_layer_schedule, ) from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.common import ( MiniMaxM3SparseConfig, @@ -57,6 +58,7 @@ MiniMaxM3MsaSparseAttentionMetadata, ) from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.msa_indexer import _group_max_reduce +from tensorrt_llm._torch.autotuner import autotune from tensorrt_llm._torch.compilation.remove_copy_pass import remove_copy_for_mutates_args from tensorrt_llm._torch.model_config import ModelConfig from tensorrt_llm._torch.models.checkpoints.hf.minimaxm3_weight_mapper import ( @@ -71,6 +73,9 @@ MiniMaxM3QKVIndexerLinear, _build_swiglu_oai_dense_mlp, _dispatch_attention_over_live_tokens, + _fold_gemma_boundary_norm_weights, + _get_index_projection_quant_config, + _load_index_qk_proj_weights, _load_qkv_index_proj_weights, _minimax_m3_swiglu_oai, _moe_routed_output_is_global, @@ -295,21 +300,15 @@ def test_minimax_m3_moe_reduces_only_local_terms( assert moe.allreduce.params == [None] -@pytest.mark.cpu_only -@pytest.mark.parametrize( - "scheduler_kind, expected_post_fusion", - [ - (MoESchedulerKind.EXTERNAL_COMM, True), - (MoESchedulerKind.FUSED_COMM, False), - ], -) -def test_minimax_m3_decoder_layer_sets_post_fusion_from_moe_scheduler( +def _build_decoder_layer_with_stubs( monkeypatch: pytest.MonkeyPatch, scheduler_kind: MoESchedulerKind, - expected_post_fusion: bool, -) -> None: - """A fused-communication MoE must not schedule a second boundary reduction.""" - + use_gemma_norm: bool, + *, + dtype: torch.dtype = torch.bfloat16, + hidden_size: int = 6144, +) -> MiniMaxM3DecoderLayer: + """A TP2 MoE decoder layer with real norms and stub attention, MoE and AllReduce.""" monkeypatch.setenv("TRTLLM_MINIMAX_M3_EAGER_FUSION_DISABLED", "0") class _DecoderComponent(nn.Module): @@ -331,20 +330,19 @@ def __init__(self, *args: object, **kwargs: object) -> None: "tensorrt_llm._torch.models.modeling_minimaxm3.MiniMaxM3MoE", _DecoderMoE, ) - monkeypatch.setattr( - "tensorrt_llm._torch.models.modeling_minimaxm3.RMSNorm", - _DecoderComponent, - ) monkeypatch.setattr( "tensorrt_llm._torch.models.modeling_minimaxm3.AllReduce", _DecoderComponent, ) + config = _make_text_config() + config.use_gemma_norm = use_gemma_norm + config.torch_dtype = dtype + config.hidden_size = hidden_size model_config = ModelConfig( - pretrained_config=_make_text_config(), + pretrained_config=config, mapping=Mapping(world_size=2, rank=0, tp_size=2), ) - - layer = MiniMaxM3DecoderLayer( + return MiniMaxM3DecoderLayer( model_config=model_config, layer_idx=3, aux_stream_dict={ @@ -353,6 +351,25 @@ def __init__(self, *args: object, **kwargs: object) -> None: }, ) + +@pytest.mark.cpu_only +@pytest.mark.parametrize( + "scheduler_kind, expected_post_fusion", + [ + (MoESchedulerKind.EXTERNAL_COMM, True), + (MoESchedulerKind.FUSED_COMM, False), + ], +) +@pytest.mark.parametrize("use_gemma_norm", [False, True]) +def test_minimax_m3_decoder_layer_sets_post_fusion_from_moe_scheduler( + monkeypatch: pytest.MonkeyPatch, + scheduler_kind: MoESchedulerKind, + expected_post_fusion: bool, + use_gemma_norm: bool, +) -> None: + """Plain and folded-Gemma norms preserve fusion without a second routed reduction.""" + layer = _build_decoder_layer_with_stubs(monkeypatch, scheduler_kind, use_gemma_norm) + assert layer.enable_fusion assert layer.pre_feed_forward_fusion assert layer.post_feed_forward_fusion is expected_post_fusion @@ -1303,6 +1320,206 @@ def test_get_moe_layer_ids_length_mismatch_raises(): get_moe_layer_ids(cfg) +# --------------------------------------------------------------------------- +# Compressed-tensors NVFP4 MoE packaging of the MiniMax M3 architecture +# --------------------------------------------------------------------------- + + +def _make_packaged_text_config(): + """The trimmed text config as the compressed-tensors package declares it. + + Every attention layer is sparse and K-only while layers 0-2 keep a dense + MLP; rope_theta 1e7, 1M positions, no MTP module and no text model_type. + """ + cfg = _make_text_config() + cfg.sparse_attention_config["sparse_attention_freq"] = [1] * _NUM_HIDDEN_LAYERS + cfg.sparse_attention_config["sparse_disable_index_value"] = [1] * _NUM_HIDDEN_LAYERS + cfg.rope_theta = 10000000 + cfg.max_position_embeddings = 1048576 + cfg.num_mtp_modules = 0 + del cfg.model_type + return cfg + + +def _packaged_vl_config(text_config_as_dict: bool = False, **quant_overrides): + quantization_config = { + "quant_method": "mxfp8", + "activation_scheme": "dynamic", + "weight_block_size": [1, 32], + "group_size": 16, + "moe_quant_algo": "NVFP4", + "moe_quant_format": "nvfp4-pack-quantized", + "ignored_layers": [ + "lm_head", + "model.embed_tokens", + "vision_tower", + "multi_modal_projector", + "patch_merge_mlp", + ] + + [ + f"language_model.model.layers.{i}.block_sparse_moe.gate" + for i in range(3, _NUM_HIDDEN_LAYERS) + ], + } + quantization_config.update(quant_overrides) + text_config = _make_packaged_text_config() + cfg = _make_vl_config() + cfg.text_config = dict(vars(text_config)) if text_config_as_dict else text_config + cfg.quantization_config = quantization_config + return cfg + + +def _expand_packaged_quant_config(pretrained_config): + """Parse the inline config as from_pretrained does, then apply the M3 hook.""" + quant_config, layer_quant_config = ModelConfig.load_hf_quant_config( + pretrained_config.quantization_config, "CUTLASS" + ) + assert quant_config.quant_algo == QuantAlgo.MXFP8 + assert layer_quant_config is None + return ModelConfig._set_minimax_m3_layer_quant_config( + pretrained_config, quant_config, layer_quant_config + ) + + +@pytest.mark.parametrize("text_config_as_dict", [False, True]) +def test_packaged_moe_config_expands_to_mixed_precision(text_config_as_dict: bool) -> None: + quant_config, layer_quant_config = _expand_packaged_quant_config( + _packaged_vl_config(text_config_as_dict) + ) + + assert quant_config.quant_algo == QuantAlgo.MIXED_PRECISION + assert "model.layers.3.block_sparse_moe.gate" in quant_config.exclude_modules + assert not any(name.startswith("language_model.") for name in quant_config.exclude_modules) + assert quant_config.is_module_excluded_from_quantization("lm_head") + + expected = {} + for layer in range(_NUM_HIDDEN_LAYERS): + prefix = f"model.layers.{layer}" + names = [ + f"{prefix}.self_attn.{proj}" + for proj in ("q_proj", "k_proj", "v_proj", "o_proj", "index_q_proj", "index_k_proj") + ] + if _MOE_LAYER_FREQ[layer]: + names += [ + f"{prefix}.block_sparse_moe.shared_experts.{proj}" + for proj in ("gate_proj", "up_proj", "down_proj") + ] + expected[f"{prefix}.block_sparse_moe.experts"] = (QuantAlgo.NVFP4, 16) + else: + names += [f"{prefix}.mlp.{proj}" for proj in ("gate_proj", "up_proj", "down_proj")] + expected.update({name: (QuantAlgo.MXFP8, 32) for name in names}) + assert {k: (v.quant_algo, v.group_size) for k, v in layer_quant_config.items()} == expected + + +def test_packaged_moe_config_honors_ignored_layers() -> None: + cfg = _packaged_vl_config() + cfg.quantization_config["ignored_layers"].append( + "language_model.model.layers.4.self_attn.o_proj" + ) + quant_config, layer_quant_config = _expand_packaged_quant_config(cfg) + assert "model.layers.4.self_attn.o_proj" not in layer_quant_config + assert quant_config.is_module_excluded_from_quantization("model.layers.4.self_attn.o_proj") + assert "model.layers.5.self_attn.o_proj" in layer_quant_config + + +@pytest.mark.parametrize( + "overrides", + [ + {"moe_quant_algo": "MXFP4"}, + {"moe_quant_format": "float-quantized"}, + {"group_size": 32}, + {"group_size": None}, + {"moe_quant_format": None}, + {"moe_quant_algo": None}, + ], +) +def test_packaged_moe_config_rejects_unsupported_expert_recipes(overrides) -> None: + with pytest.raises(ValueError, match="routed-expert packaging"): + _expand_packaged_quant_config(_packaged_vl_config(**overrides)) + + +def test_uniform_mxfp8_and_bf16_configs_are_unchanged() -> None: + cfg = _packaged_vl_config() + for key in ("moe_quant_algo", "moe_quant_format", "group_size"): + del cfg.quantization_config[key] + quant_config, _ = ModelConfig.load_hf_quant_config(cfg.quantization_config, "CUTLASS") + result = ModelConfig._set_minimax_m3_layer_quant_config(cfg, quant_config, None) + assert result[0] is quant_config and result[1] is None + + no_quant = QuantConfig() + result = ModelConfig._set_minimax_m3_layer_quant_config(_make_vl_config(), no_quant, None) + assert result[0] is no_quant and result[1] is None + + +def test_modelopt_mixed_precision_map_is_normalized_as_before() -> None: + base_config = QuantConfig(quant_algo=QuantAlgo.MXFP8, group_size=32) + expert_config = QuantConfig(quant_algo=QuantAlgo.NVFP4, group_size=16) + global_config = QuantConfig(quant_algo=QuantAlgo.MIXED_PRECISION) + layer_map = { + "language_model.model.layers.0.self_attn.o_proj": base_config, + "language_model.model.layers.3.block_sparse_moe.experts.0.w1": expert_config, + } + quant_config, out = ModelConfig._set_minimax_m3_layer_quant_config( + _make_vl_config(), global_config, layer_map + ) + assert quant_config is global_config + assert out["model.layers.0.self_attn.o_proj"] is base_config + assert out["model.layers.3.block_sparse_moe.experts.0.w1"] is expert_config + assert not any(key.startswith("language_model.") for key in out) + for layer in range(_NUM_HIDDEN_LAYERS): + key = f"model.layers.{layer}.block_sparse_moe.experts" + if _MOE_LAYER_FREQ[layer]: + assert (out[key].quant_algo, out[key].group_size) == (QuantAlgo.NVFP4, 16) + else: + assert key not in out + + +def test_index_projection_quant_config_follows_declared_entries() -> None: + _, layer_quant_config = _expand_packaged_quant_config(_packaged_vl_config()) + declared = _get_index_projection_quant_config( + SimpleNamespace(quant_config_dict=layer_quant_config), 4 + ) + assert (declared.quant_algo, declared.group_size) == (QuantAlgo.MXFP8, 32) + # Checkpoints without per-layer entries keep the unquantized index projection. + for quant_config_dict in (None, {}): + assert ( + _get_index_projection_quant_config( + SimpleNamespace(quant_config_dict=quant_config_dict), 4 + ) + is None + ) + + mismatched = dict(layer_quant_config) + mismatched["model.layers.4.self_attn.index_k_proj"] = QuantConfig( + quant_algo=QuantAlgo.NVFP4, group_size=16 + ) + with pytest.raises(ValueError, match="same quantization"): + _get_index_projection_quant_config(SimpleNamespace(quant_config_dict=mismatched), 4) + del mismatched["model.layers.4.self_attn.index_k_proj"] + with pytest.raises(ValueError, match="same quantization"): + _get_index_projection_quant_config(SimpleNamespace(quant_config_dict=mismatched), 4) + + +def test_kv_cache_schedule_follows_the_checkpoint_attention_schedule() -> None: + all_layers = list(range(_NUM_HIDDEN_LAYERS)) + assert resolve_sparse_layer_schedule( + None, None, _make_packaged_text_config(), _NUM_HIDDEN_LAYERS + ) == (all_layers, all_layers) + assert resolve_sparse_layer_schedule(None, None, _make_text_config(), _NUM_HIDDEN_LAYERS) == ( + [3, 4, 5, 6], + [3, 4, 5, 6], + ) + # Explicit ids win; without a checkpoint schedule the public convention holds. + assert resolve_sparse_layer_schedule([5], None, _make_packaged_text_config(), 7) == ([5], [5]) + for config in (None, SimpleNamespace(num_hidden_layers=_NUM_HIDDEN_LAYERS)): + assert resolve_sparse_layer_schedule(None, None, config, _NUM_HIDDEN_LAYERS) == ( + [3, 4, 5, 6], + [3, 4, 5, 6], + ) + with pytest.raises(ValueError, match="outside the 4 target layers"): + resolve_sparse_layer_schedule(None, None, _make_packaged_text_config(), 4) + + # --------------------------------------------------------------------------- # attention module transforms # --------------------------------------------------------------------------- @@ -2436,6 +2653,50 @@ def test_minimax_m3_fused_routing_cuda_graph_replay_tracks_inputs() -> None: torch.testing.assert_close(graph_weights, reference_weights, rtol=0, atol=0) +@pytest.mark.gpu +@pytest.mark.skipif(not _has_cuda(), reason="MiniMax-M3 router gate needs CUDA") +@pytest.mark.parametrize("num_tokens", [1, 64, 2048]) +def test_minimax_m3_gate_stays_fp32_when_tf32_is_allowed(num_tokens: int) -> None: + """The router GEMM is FP32 (as in SGLang) even when torch runs fp32 matmuls as TF32.""" + hidden_size, num_experts = 6144, 128 + torch.manual_seed(0) + gate = modeling_minimaxm3.MiniMaxM3Gate( + hidden_size, num_experts, top_k=4, routed_scaling_factor=2.0 + ).cuda() + gate.weight.data.normal_(std=0.02) + hidden = torch.randn(num_tokens, hidden_size, device="cuda").bfloat16() + replay_hidden = torch.randn(num_tokens, hidden_size, device="cuda").bfloat16() + reference = hidden.double() @ gate.weight.double().t() + scale = reference.abs().max() + + previous = torch.backends.cuda.matmul.allow_tf32 + torch.backends.cuda.matmul.allow_tf32 = True + try: + logits = gate(hidden) + tf32_logits = torch.nn.functional.linear(hidden.float(), gate.weight) + static_hidden = hidden.clone() + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + gate(static_hidden) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + graph_logits = gate(static_hidden) + static_hidden.copy_(replay_hidden) + graph.replay() + replay_reference = gate(replay_hidden) + finally: + torch.backends.cuda.matmul.allow_tf32 = previous + + assert logits.dtype == torch.float32 and logits.shape == (num_tokens, num_experts) + assert float((logits.double() - reference).abs().max() / scale) < 1e-5 + if num_tokens == 2048: + # The bound above only means something if F.linear really runs as TF32 here. + assert float((tf32_logits.double() - reference).abs().max() / scale) > 1e-4 + torch.testing.assert_close(graph_logits, replay_reference, rtol=0, atol=0) + + @pytest.mark.gpu @pytest.mark.skipif(not _has_cuda(), reason="MiniMax-M3 needs CUDA") def test_text_norm_weights_real_loader_smoke(monkeypatch: pytest.MonkeyPatch): @@ -2691,3 +2952,449 @@ def test_minimax_m3_swiglu_oai_fused_matches_reference(dtype): plain_ref = torch.nn.functional.silu(gate_c) * up_c plain = swiglu(gate_up, swiglu_limit=limit) torch.testing.assert_close(plain.float(), plain_ref.float(), atol=atol, rtol=1e-2) + + +# --------------------------------------------------------------------------- +# Compressed-tensors NVFP4 MoE packaging: CUDA loading and forward checks +# --------------------------------------------------------------------------- +# +# The references below are written from the checkpoint format (E4M3 / UE8M0 / +# E2M1 definitions) without calling TensorRT-LLM quantizers. Each forward is +# checked eagerly and again through CUDA graph capture and replay. + +_E2M1_GRID = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0) + + +def _relative_l2(out: torch.Tensor, ref: torch.Tensor) -> float: + return float((out.float() - ref.float()).norm() / ref.float().norm().clamp_min(1e-6)) + + +def _cosine(out: torch.Tensor, ref: torch.Tensor) -> float: + return float( + torch.nn.functional.cosine_similarity(out.float().flatten(), ref.float().flatten(), dim=0) + ) + + +def _graph_replay(fn, *inputs: torch.Tensor): + """Capture ``fn(*inputs)`` in a CUDA graph; return a replay(new_inputs) helper.""" + static_inputs = [t.clone() for t in inputs] + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + fn(*static_inputs) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + static_output = fn(*static_inputs) + + def replay(*new_inputs: torch.Tensor) -> torch.Tensor: + for static, new in zip(static_inputs, new_inputs): + static.copy_(new) + graph.replay() + torch.cuda.synchronize() + return static_output.clone() + + return replay + + +def _mxfp8_dequant(weight: torch.Tensor, scale_inv: torch.Tensor) -> torch.Tensor: + """E4M3 weight times 2**(UE8M0 - 127) per 32 input channels.""" + exponent = scale_inv.float().repeat_interleave(32, dim=1) - 127.0 + return weight.float() * torch.exp2(exponent) + + +@pytest.mark.gpu +@pytest.mark.skipif(not _has_cuda(), reason="MXFP8 index projection needs CUDA") +@pytest.mark.parametrize("use_cuda_graph", [False, True]) +def test_packaged_index_projection_keeps_mxfp8_scales_per_tp4_rank( + use_cuda_graph: bool, monkeypatch: pytest.MonkeyPatch +) -> None: + """index_q/k_proj load as MXFP8 with each rank's scale rows next to its weight rows. + + Every index-Q head gets its own scale exponent, so a scale row that lands + on another head's weights changes the output by powers of two; index-K is + replicated on every rank with all of its scale rows. + """ + import tensorrt_llm._torch.distributed as distributed + + # Simulate each rank on one GPU; COLUMN projections need no collective. + monkeypatch.setattr(distributed, "AllReduce", lambda **_kwargs: nn.Identity()) + torch.manual_seed(0) + hidden, index_dim, index_heads, tp_size = 256, 32, 4, 4 + head_exponents = torch.tensor([127, 123, 130, 125], dtype=torch.int32) + q_weight = (torch.randn(index_heads * index_dim, hidden) * 2).to(torch.float8_e4m3fn) + q_scale = ( + head_exponents.repeat_interleave(index_dim)[:, None] + + torch.randint(-1, 2, (index_heads * index_dim, hidden // 32)) + ).to(torch.uint8) + k_weight = (torch.randn(index_dim, hidden) * 2).to(torch.float8_e4m3fn) + k_scale = torch.randint(121, 129, (index_dim, hidden // 32)).to(torch.uint8) + x = torch.randn(48, hidden, device="cuda", dtype=torch.bfloat16) + x_next = torch.randn_like(x) + ref_q = x.float() @ _mxfp8_dequant(q_weight, q_scale).cuda().T + ref_k = x.float() @ _mxfp8_dequant(k_weight, k_scale).cuda().T + mxfp8 = QuantConfig(quant_algo=QuantAlgo.MXFP8, group_size=32) + + for rank in range(tp_size): + text_cfg, model_cfg = _make_attention_test_config() + text_cfg.hidden_size = hidden + text_cfg.num_attention_heads = 8 + text_cfg.num_key_value_heads = 4 + text_cfg.sparse_attention_config["sparse_num_index_heads"] = index_heads + model_cfg.mapping = Mapping(world_size=tp_size, rank=rank, tp_size=tp_size) + model_cfg.quant_config_dict = { + "model.layers.3.self_attn.index_q_proj": mxfp8, + "model.layers.3.self_attn.index_k_proj": mxfp8, + } + attn = MiniMaxM3Attention( + model_config=model_cfg, + layer_idx=3, + is_sparse_attention_layer=True, + disable_index_value=True, + ) + proj = attn.index_qk_proj + assert proj.quant_config.quant_algo == QuantAlgo.MXFP8 + proj.create_weights() + assert type(proj.quant_method).__name__ == "MXFP8LinearMethod" + proj.cuda() + + layer = nn.Module() + layer.self_attn = attn + _load_index_qk_proj_weights( + layer, + { + "self_attn.index_q_proj.weight": q_weight, + "self_attn.index_q_proj.weight_scale_inv": q_scale, + "self_attn.index_k_proj.weight": k_weight, + "self_attn.index_k_proj.weight_scale_inv": k_scale, + }, + ) + rows = slice(rank * index_dim, (rank + 1) * index_dim) + expected_weight = torch.cat([q_weight[rows], k_weight]).view(torch.uint8).cuda() + assert torch.equal(proj.weight.view(torch.uint8), expected_weight) + expected_scale = torch.cat([q_scale[rows], k_scale]).cuda() + if proj.quant_method.use_cutlass: + expected_scale = torch.ops.trtllm.block_scale_interleave(expected_scale) + assert torch.equal(proj.weight_scale.view(torch.uint8).flatten(), expected_scale.flatten()) + + def forward(inp: torch.Tensor) -> torch.Tensor: + return proj(inp) + + if use_cuda_graph: + replay = _graph_replay(forward, x) + out, out_next = replay(x), replay(x_next) + assert torch.equal(out_next, forward(x_next)) + else: + out = forward(x) + ref = torch.cat([ref_q[:, rows], ref_k], dim=1) + # Dynamic per-token MXFP8 activations cost ~1% relative error; a scale + # row on the wrong head is off by at least a factor of two. + assert _relative_l2(out, ref) < 0.05, rank + assert _cosine(out, ref) > 0.998, rank + + +def _e2m1_values(packed: torch.Tensor) -> torch.Tensor: + """Unpack [rows, K/2] bytes (even element in the low nibble) to float32 [rows, K].""" + codes = torch.stack((packed & 0xF, packed >> 4), dim=-1).reshape(packed.shape[0], -1).long() + magnitude = torch.tensor(_E2M1_GRID, device=packed.device)[codes & 0x7] + return torch.where(codes >= 8, -magnitude, magnitude) + + +def _round_to_e2m1(x: torch.Tensor) -> torch.Tensor: + """Round to the E2M1 grid, ties to the even code, saturating at 6.""" + grid = torch.tensor(_E2M1_GRID, device=x.device) + bounds = torch.tensor([0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0], device=x.device) + magnitude = x.abs() + index = torch.bucketize(magnitude, bounds) + index = index + torch.isin(magnitude, bounds[[1, 3, 5]]).long() + return torch.copysign(grid[index], x) + + +def _nvfp4_activation(x: torch.Tensor, global_scale: float = 1.0) -> torch.Tensor: + """Quantize-dequantize with 16-element E4M3 block scales and a static global scale.""" + blocks = x.float().reshape(*x.shape[:-1], -1, 16) + scale = blocks.abs().amax(-1, keepdim=True) * (1.0 / 6.0) * global_scale + scale = scale.to(torch.float8_e4m3fn).float() + scaled = torch.where(scale > 0, blocks * (global_scale / scale), torch.zeros_like(blocks)) + return (_round_to_e2m1(scaled) * scale / global_scale).reshape(x.shape) + + +@pytest.mark.gpu +@pytest.mark.skipif(not _has_cuda(), reason="CUTLASS NVFP4 MoE needs CUDA") +@pytest.mark.parametrize("use_cuda_graph", [False, True]) +@pytest.mark.parametrize("tune", [False, True]) +def test_packaged_nvfp4_experts_load_and_run_on_cutlass(use_cuda_graph: bool, tune: bool) -> None: + """Packed experts reach the CUTLASS NVFP4 path intact and compute the static recipe. + + Each expert has its own global scales, so a dropped scale or a wrong + expert assignment is visible in the output. The FC1 input is quantized + with the static input scale, the FC1 results round to BF16, the FC2 input + is quantized from that BF16 activation and the routing weights apply in + the finalize. + """ + from tensorrt_llm._torch.moe.fused_moe import SwigluBiasActivation, create_moe + from tensorrt_llm.quantization.utils.fp4_utils import float4_sf_dtype + + torch.manual_seed(0) + num_experts, top_k, hidden, inter, tokens = 8, 2, 256, 256, 24 + prefix = "language_model.model.layers.3.block_sparse_moe.experts" + generator = torch.Generator().manual_seed(1) + checkpoint = {} + for e in range(num_experts): + for proj, rows, cols, global_scale in ( + ("w1", inter, hidden, 50.0 + 7.0 * e), + ("w3", inter, hidden, 50.0 + 7.0 * e), + ("w2", hidden, inter, 60.0 + 5.0 * e), + ): + name = f"{prefix}.{e}.{proj}" + checkpoint[f"{name}.weight_packed"] = torch.randint( + 0, 256, (rows, cols // 2), dtype=torch.uint8, generator=generator + ) + checkpoint[f"{name}.weight_scale"] = ( + torch.rand(rows, cols // 16, generator=generator) * 2 + 0.25 + ).to(torch.float8_e4m3fn) + checkpoint[f"{name}.weight_global_scale"] = torch.tensor(global_scale) + checkpoint[f"{name}.input_global_scale"] = torch.tensor(1.0) + + _, layer_quant_config = _expand_packaged_quant_config(_packaged_vl_config()) + pretrained = _make_packaged_text_config() + pretrained.num_local_experts = num_experts + pretrained.hidden_size = hidden + pretrained.intermediate_size = inter + pretrained.torch_dtype = torch.bfloat16 + model_config = ModelConfig( + pretrained_config=pretrained, + quant_config=QuantConfig(quant_algo=QuantAlgo.MIXED_PRECISION), + quant_config_dict=layer_quant_config, + moe_backend="CUTLASS", + max_num_tokens=256, + ) + bias = torch.randn(num_experts, device="cuda") * 0.1 + routing = MiniMaxM3MoeRoutingMethod( + top_k=top_k, + num_experts=num_experts, + callable_e_score_correction_bias=lambda: bias, + routed_scaling_factor=2.0, + ) + moe = create_moe( + routing_method=routing, + num_experts=num_experts, + model_config=model_config, + layer_idx=3, + override_quant_config=MiniMaxM3MoE._get_experts_quant_config(model_config, 3), + activation=SwigluBiasActivation(gate_sigmoid_scale=1.702, linear_offset=1.0, clamp=7.0), + allow_backend_degradation=False, + ) + backend = moe.backend + assert type(backend).__name__ == "CutlassFusedMoE" + translated = MiniMaxM3HfWeightMapper().preprocess_weights(dict(checkpoint)) + module_weights = { + key[len(prefix) + 1 :]: value + for key, value in translated.items() + if key.startswith(prefix + ".") + } + with torch.device("cuda"): + moe.create_weights() + assert type(backend.quant_method).__name__ == "NVFP4CutlassFusedMoEMethod" + moe.load_weights([module_weights]) + moe.post_load_weights() + + # Static input scales of 1.0 stay 1.0; alphas are the reciprocal weight global scales. + assert backend.fc31_input_scale.item() == 1.0 and backend.fc2_input_scale.item() == 1.0 + experts = [] + for e in range(num_experts): + source = { + proj: ( + checkpoint[f"{prefix}.{e}.{proj}.weight_packed"], + checkpoint[f"{prefix}.{e}.{proj}.weight_scale"], + torch.reciprocal(checkpoint[f"{prefix}.{e}.{proj}.weight_global_scale"]).item(), + ) + for proj in ("w1", "w2", "w3") + } + assert torch.equal( + backend.w3_w1_weight[e].view(torch.uint8), + torch.cat([source["w3"][0], source["w1"][0]]).cuda(), + ) + assert torch.equal(backend.w2_weight[e].view(torch.uint8), source["w2"][0].cuda()) + expected_fc31_scale = torch.ops.trtllm.block_scale_interleave( + torch.cat([source["w3"][1], source["w1"][1]]).cuda().view(float4_sf_dtype) + ) + assert torch.equal( + backend.w3_w1_weight_scale[e].view(torch.uint8).flatten(), + expected_fc31_scale.view(torch.uint8).flatten(), + ) + expected_fc2_scale = torch.ops.trtllm.block_scale_interleave( + source["w2"][1].cuda().view(float4_sf_dtype) + ) + assert torch.equal( + backend.w2_weight_scale[e].view(torch.uint8).flatten(), + expected_fc2_scale.view(torch.uint8).flatten(), + ) + alphas = (source["w1"][2], source["w3"][2], source["w2"][2]) + # The alpha is 1 / (input_scale * (1 / weight_scale_2)): within one ulp. + for alpha, golden in ( + (backend.fc31_alpha[e].item(), source["w1"][2]), + (backend.fc2_alpha[e].item(), source["w2"][2]), + ): + assert abs(alpha - golden) <= 2**-23 * golden, (e, alpha, golden) + # Dequantized FP4 weights without the global scale (exact in FP32), and that scale. + experts.append( + ( + tuple( + _e2m1_values(w).cuda() * s.float().cuda().repeat_interleave(16, dim=1) + for w, s, _ in (source["w1"], source["w3"], source["w2"]) + ), + alphas, + ) + ) + + def gemm(a: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + """FP32 rounding of the FP64 sum of exactly representable products.""" + return (a.double() @ w.double().T).float() + + def reference(x: torch.Tensor, logits: torch.Tensor) -> torch.Tensor: + """Static unit FC1 scale; BF16 FC1 results; FC2 input quantized from its + BF16 cast; routing weights in the finalize.""" + scores = torch.sigmoid(logits.float()) + ids = torch.topk(scores + bias, top_k, dim=-1).indices + weights = scores.gather(1, ids) + weights = weights / (weights.sum(-1, keepdim=True) + 1e-20) * 2.0 + x_q = _nvfp4_activation(x) + slots = torch.zeros(x.shape[0], top_k, hidden, device="cuda") + for e, ((w1, w3, w2), (a1, a3, a2)) in enumerate(experts): + rows, slot = (ids == e).nonzero(as_tuple=True) + if rows.numel() == 0: + continue + acc1, acc3 = gemm(x_q[rows], w1), gemm(x_q[rows], w3) + w = weights[rows, slot, None] + gate = (acc1 * a1).bfloat16().float().clamp(max=7.0) + up = (acc3 * a3).bfloat16().float().clamp(-7.0, 7.0) + act = (gate * torch.sigmoid(1.702 * gate) * (up + 1.0)).bfloat16().float() + y = (gemm(_nvfp4_activation(act), w2) * a2).bfloat16().float() + slots[rows, slot] = y * w + out = torch.zeros(x.shape[0], hidden, device="cuda") + for j in range(top_k): + out = out + slots[:, j] + return out + + def forward(x: torch.Tensor, logits: torch.Tensor) -> torch.Tensor: + return moe(x, logits, all_rank_num_tokens=[tokens], use_dp_padding=False) + + x = torch.randn(tokens, hidden, device="cuda", dtype=torch.bfloat16) + logits = torch.randn(tokens, num_experts, device="cuda") + x_next, logits_next = torch.randn_like(x), torch.randn_like(logits) + with torch.inference_mode(): + # Tune the actual shape, then verify the selected tactic eagerly and in a graph. + if tune: + with autotune(tune_mode=True, skip_dynamic_tuning_buckets=True): + forward(x, logits) + if use_cuda_graph: + replay = _graph_replay(forward, x, logits) + outputs = [ + (replay(x, logits), x, logits), + (replay(x_next, logits_next), x_next, logits_next), + ] + # The fused top-k combine may accumulate in a different order per launch. + assert _relative_l2(outputs[1][0], forward(x_next, logits_next)) < 1e-3 + else: + outputs = [(forward(x, logits), x, logits)] + for out, inp, lg in outputs: + ref = reference(inp, lg) + assert torch.isfinite(out).all() + # Same recipe on both sides; the remainder is accumulation order + # and the BF16 rounding of the GEMM outputs. + assert _relative_l2(out, ref) < 0.02 + assert _cosine(out, ref) > 0.999 + + +@pytest.mark.cpu_only +def test_minimax_m3_fold_gemma_boundary_weights_accepts_partial_lazy_load(): + stored = torch.tensor([0.25, -0.25], dtype=torch.bfloat16) + + class LazyWeight: + def __getitem__(self, index): + assert index == slice(None) + return stored + + untouched = LazyWeight() + weights = { + "model.layers.2.input_layernorm.weight": LazyWeight(), + "model.layers.2.self_attn.q_norm.weight": untouched, + } + result = _fold_gemma_boundary_norm_weights(weights) + assert result is weights + torch.testing.assert_close( + result["model.layers.2.input_layernorm.weight"], stored + 1.0, rtol=0, atol=0 + ) + assert result["model.layers.2.self_attn.q_norm.weight"] is untouched + + +def _stub_minimax_m3_attention_construction(monkeypatch, *, is_neox=True): + """Keep the M3 constructor real while removing generic GPU setup and projection allocation.""" + + def initialize_attention(self, **kwargs): + nn.Module.__init__(self) + config = kwargs["config"] + self.pos_embd_params = SimpleNamespace(rope=kwargs["pos_embd_params"].rope, is_neox=is_neox) + self.rotary_emb = object() + self.head_dim = config.pretrained_config.head_dim + self.qkv_proj = nn.Module() + self.qkv_proj.mapping = config.mapping + + def make_linear(*args, **kwargs): + projection = nn.Module() + projection.quant_config = kwargs["quant_config"] + return projection + + monkeypatch.setattr(modeling_minimaxm3.Attention, "__init__", initialize_attention) + monkeypatch.setattr(torch.cuda, "Stream", lambda: None) + monkeypatch.setattr(torch.cuda, "Event", lambda: None) + monkeypatch.setattr(torch.cuda, "current_device", lambda: 0) + projection_factory = Mock(side_effect=make_linear) + monkeypatch.setattr(modeling_minimaxm3, "Linear", projection_factory) + return projection_factory + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("entries", ["matching", "absent", "mismatched", "missing_k"]) +def test_minimax_m3_index_projection_quantization_follows_checkpoint_entries(monkeypatch, entries): + """index_qk_proj is quantized exactly when the checkpoint declares matching per-layer entries.""" + _, model_config = _make_fused_qk_norm_rope_test_config() + # Other-layer entries must not affect the projection being constructed. + declared = QuantConfig(quant_algo=QuantAlgo.MXFP8, group_size=32) + other = QuantConfig(quant_algo=QuantAlgo.NVFP4, group_size=16) + layer_quant_config = { + "model.layers.2.self_attn.index_q_proj": other, + "model.layers.2.self_attn.index_k_proj": other, + } + if entries != "absent": + layer_quant_config["model.layers.3.self_attn.index_q_proj"] = declared + if entries in ("matching", "mismatched"): + layer_quant_config["model.layers.3.self_attn.index_k_proj"] = ( + declared if entries == "matching" else other + ) + model_config.quant_config_dict = layer_quant_config + projection_factory = _stub_minimax_m3_attention_construction(monkeypatch) + lookup = Mock(wraps=_get_index_projection_quant_config) + monkeypatch.setattr(modeling_minimaxm3, "_get_index_projection_quant_config", lookup) + if entries in ("mismatched", "missing_k"): + with pytest.raises(ValueError, match="same quantization"): + MiniMaxM3Attention( + model_config=model_config, + layer_idx=3, + is_sparse_attention_layer=True, + disable_index_value=True, + ) + projection_factory.assert_not_called() + else: + layer = MiniMaxM3Attention( + model_config=model_config, + layer_idx=3, + is_sparse_attention_layer=True, + disable_index_value=True, + ) + projection_factory.assert_called_once() + expected = declared if entries == "matching" else None + assert layer.index_qk_proj.quant_config is expected + assert projection_factory.call_args.kwargs["quant_config"] is expected + lookup.assert_called_once_with(model_config, 3) diff --git a/tests/unittest/_torch/models/test_minimax_m3_vl.py b/tests/unittest/_torch/models/test_minimax_m3_vl.py index 93cb2ffff332..b3cdffcd1a5e 100644 --- a/tests/unittest/_torch/models/test_minimax_m3_vl.py +++ b/tests/unittest/_torch/models/test_minimax_m3_vl.py @@ -2195,3 +2195,145 @@ def test_fast_path_hooks_are_registered_on_processor_class(): assert hasattr(cls, "expand_prompt_token_ids_for_mm") assert hasattr(cls, "get_num_tokens_per_image") assert hasattr(cls, "get_num_tokens_per_video") + + +class _WordTokenizer: + """Tokenizer stand-in: one id per whitespace-separated word, BOS id 1.""" + + unk_token_id = 0 + + def __init__(self): + self.template_calls = 0 + + def convert_tokens_to_ids(self, token: str) -> int: + return {"]<]start of image[>[": _VS_TOK, "]<]end of image[>[": _VE_TOK}.get(token, 0) + + def encode(self, text, add_special_tokens=True, truncation=False, max_length=None): + ids = [100 + i for i, _ in enumerate(text.split())] + if add_special_tokens: + ids = [1] + ids + return ids[:max_length] if truncation and max_length is not None else ids + + def apply_chat_template( + self, messages, tokenize=False, add_generation_prompt=True, tools=None, **kwargs + ): + # Content is either the flattened text trtllm-serve renders (STRING + # content format) or the content list the input processor builds. + self.template_calls += 1 + content = messages[0]["content"] + if isinstance(content, str): + return "user: " + content + " assistant:" + markers = {"image": "]<]image[>[", "video": "]<]video[>["} + parts = [ + item["text"] if item["type"] == "text" else markers[item["type"]] for item in content + ] + return "user: " + " ".join(parts) + " assistant:" + + +class _ValidImageProcessor: + """Processor stand-in with an image processor: one placeholder per item. + + Patch 14 / merge 2 / temporal 2 like the checkpoint; ``calls`` records + the text of every call and ``call_kwargs`` the processor kwargs. + """ + + image_processor = SimpleNamespace( + merge_size=2, get_number_of_image_patches=lambda h, w: (h // 14) * (w // 14) + ) + video_processor = SimpleNamespace(merge_size=2, temporal_patch_size=2) + + def __init__(self): + self.calls = [] + self.call_kwargs = [] + + def __call__(self, text, images=None, videos=None, return_tensors=None, **kwargs): + self.calls.append(text) + self.call_kwargs.append(kwargs) + num_images = len(images or []) + num_videos = len(videos or []) + input_ids = ( + [7] + + [_VS_TOK, _IMG_TOK, _VE_TOK] * num_images + + [_VS_TOK, _VID_TOK, _VE_TOK] * num_videos + ) + out = {"input_ids": torch.tensor([input_ids])} + if num_images: + out["pixel_values"] = torch.ones(4 * num_images, 8) + out["image_grid_thw"] = torch.tensor([[1, 2, 2]] * num_images) + if num_videos: + out["pixel_values_videos"] = torch.ones(4 * num_videos, 8) + out["video_grid_thw"] = torch.tensor([[1, 2, 2]] * num_videos) + return out + + +def _tokenizer_only_processor(monkeypatch, auto_processor_result): + """Construct the input processor while recording AutoProcessor loads.""" + loads = [] + + def fake_from_pretrained(*args, **kwargs): + loads.append(args) + return auto_processor_result + + monkeypatch.setattr(AutoProcessor, "from_pretrained", fake_from_pretrained) + cls = get_minimax_m3_vl_input_processor_cls() + config = SimpleNamespace( + image_token_index=_IMG_TOK, + video_token_index=_VID_TOK, + vocab_size=200_064, + text_config=SimpleNamespace(torch_dtype=torch.bfloat16), + ) + proc = cls(model_path="/m3-model-dir", config=config, tokenizer=_WordTokenizer()) + return proc, loads + + +@pytest.mark.cpu_only +def test_text_request_needs_no_hf_processor(monkeypatch): + proc, loads = _tokenizer_only_processor(monkeypatch, _WordTokenizer()) + assert loads == [] and proc._processor is None + + ids, extra = proc.call_with_text_prompt( + {"prompt": "why is the sky blue"}, SimpleNamespace(add_special_tokens=True) + ) + assert ids == [1, 100, 101, 102, 103, 104] and extra is None + ids, _ = proc.call_with_text_prompt( + {"prompt": "why is the sky blue"}, + SimpleNamespace(add_special_tokens=False, truncate_prompt_tokens=2), + ) + assert ids == [100, 101] + assert loads == [] and proc._processor is None + + +@pytest.mark.cpu_only +def test_tokenizer_from_auto_processor_is_not_image_support(monkeypatch): + proc, loads = _tokenizer_only_processor(monkeypatch, _WordTokenizer()) + image = Image.new("RGB", (32, 32)) + with pytest.raises(ValueError, match="need an HF processor with an image processor") as error: + proc.call_with_text_prompt( + {"prompt": "describe", "multi_modal_data": {"image": [image]}}, SimpleNamespace() + ) + assert "Text-only requests are unaffected" in str(error.value) + assert "_WordTokenizer" in str(error.value) + assert len(loads) == 1 + # Text keeps working after the rejected image request. + ids, extra = proc.call_with_text_prompt({"prompt": "hello"}, SimpleNamespace()) + assert ids == [1, 100] and extra is None + + +@pytest.mark.cpu_only +def test_image_request_uses_a_valid_processor(monkeypatch): + valid = _ValidImageProcessor() + proc, loads = _tokenizer_only_processor(monkeypatch, valid) + image = Image.new("RGB", (32, 32)) + ids, extra = proc.call_with_text_prompt( + {"prompt": "describe", "multi_modal_data": {"image": [image]}}, SimpleNamespace() + ) + assert ids == [7, _VS_TOK, _IMG_TOK, _VE_TOK] + assert valid.calls == [["user: describe ]<]image[>[ assistant:"]] + image_data = extra["multimodal_data"]["image"] + assert image_data["pixel_values"].dtype == torch.bfloat16 + assert image_data["image_grid_thw"].tolist() == [[1, 2, 2]] + # Loaded once, then reused; token counting uses the same processor. + proc.call_with_text_prompt( + {"prompt": "again", "multi_modal_data": {"image": [image]}}, SimpleNamespace() + ) + assert len(loads) == 1 and proc.processor is valid diff --git a/tests/unittest/_torch/modules/test_mxfp8_linear.py b/tests/unittest/_torch/modules/test_mxfp8_linear.py index 7dc5995fd080..27162d44f6ef 100644 --- a/tests/unittest/_torch/modules/test_mxfp8_linear.py +++ b/tests/unittest/_torch/modules/test_mxfp8_linear.py @@ -35,6 +35,7 @@ from tensorrt_llm._torch.modules.linear import ( Linear, MXFP8LinearMethod, + TensorParallelMode, WeightMode, WeightsLoadingConfig, _load_kv_cache_scales, @@ -43,6 +44,7 @@ get_quant_method, ) from tensorrt_llm._torch.modules.mxfp8_utils import dequant_mxfp8_weight, quant_bf16_to_mxfp8 +from tensorrt_llm.mapping import Mapping from tensorrt_llm.models.modeling_utils import QuantConfig from tensorrt_llm.quantization.mode import QuantAlgo @@ -793,3 +795,94 @@ def record_tactic(*args, **kwargs): static_x.copy_(replay_x) graph.replay() torch.testing.assert_close(graph_output, native(replay_x), rtol=2e-2, atol=2e-2) + + +def _even_rows(rows: int, rank: int, tp_size: int) -> tuple[int, int]: + return rank * rows // tp_size, (rank + 1) * rows // tp_size + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="MXFP8 Linear load path requires CUDA") +@pytest.mark.parametrize( + "weight_mode,shard_rows,override", + [ + # Eight Q heads and four K/V heads of 32 rows, evenly split over TP4. + (WeightMode.FUSED_QKV_LINEAR, {"q": 256, "k": 128, "v": 128}, None), + # K/V heads duplicated for TP4 (two heads repeated), then split evenly. + (WeightMode.FUSED_QKV_LINEAR, {"q": 256, "k": 128, "v": 128}, "duplicated_kv"), + # One K/V head replicated whole on every rank, Q split by override. + (WeightMode.FUSED_QKV_LINEAR, {"q": 256, "k": 32, "v": 32}, "replicated_kv"), + (WeightMode.FUSED_GATE_UP_LINEAR, {"gate": 256, "up": 256}, None), + # Uneven gate rows with an up projection replicated on every rank. + (WeightMode.FUSED_GATE_UP_LINEAR, {"gate": 256, "up": 32}, "uneven"), + ], +) +def test_mxfp8_fused_scales_follow_weight_rows_per_tp_rank(weight_mode, shard_rows, override): + """Each rank keeps the UE8M0 scale rows of exactly the weight rows it loads. + + Scale exponents differ per row, so a scale row paired with another row's + weights changes the output by powers of two. + """ + torch.manual_seed(0) + tp_size, in_f = 4, 256 + names = list(shard_rows) + # FP8 weights travel as uint8 views: byte-level slicing and comparison. + sources = {} + for name, rows in shard_rows.items(): + w = (torch.randn(rows, in_f) * 2).to(torch.float8_e4m3fn).view(torch.uint8) + exponents = torch.randint(121, 134, (rows, 1)) + torch.randint(-1, 2, (rows, in_f // 32)) + sources[name] = (w, exponents.to(torch.uint8)) + if override == "duplicated_kv": + # What the HF weight mappers hand the loader for two KV heads on TP4. + for name in ("k", "v"): + w, s = sources[name] + sources[name] = tuple( + torch.cat([t[head * 32 : (head + 1) * 32] for head in (0, 0, 1, 1)]) for t in (w, s) + ) + uneven_gate = [(0, 96), (96, 160), (160, 224), (224, 256)] + x = torch.randn(16, in_f, dtype=torch.bfloat16, device="cuda") + qc = QuantConfig(quant_algo=QuantAlgo.MXFP8, group_size=32) + + for rank in range(tp_size): + if override == "replicated_kv": + ranges = {"q": _even_rows(256, rank, tp_size), "k": (0, 32), "v": (0, 32)} + elif override == "uneven": + ranges = {"gate": uneven_gate[rank], "up": (0, 32)} + else: + ranges = {name: _even_rows(sources[name][0].shape[0], rank, tp_size) for name in names} + lin = Linear( + in_features=in_f, + out_features=sum(w.shape[0] for w, _ in sources.values()), + bias=False, + dtype=torch.bfloat16, + mapping=Mapping(world_size=tp_size, rank=rank, tp_size=tp_size), + tensor_parallel_mode=TensorParallelMode.COLUMN, + quant_config=qc, + weights_loading_config=WeightsLoadingConfig(weight_mode=weight_mode), + reduce_output=False, + override_tp_sharding=ranges if override in ("replicated_kv", "uneven") else None, + ).cuda() + lin.load_weights( + [ + { + "weight": sources[name][0].view(torch.float8_e4m3fn), + "weight_scale_inv": sources[name][1], + } + for name in names + ] + ) + + expected_w = torch.cat([sources[n][0][slice(*ranges[n])] for n in names]) + expected_s = torch.cat([sources[n][1][slice(*ranges[n])] for n in names]) + assert torch.equal(lin.weight.view(torch.uint8).cpu(), expected_w), rank + stored = expected_s.cuda() + if lin.quant_method.use_cutlass: + stored = torch.ops.trtllm.block_scale_interleave(stored) + assert torch.equal(lin.weight_scale.view(torch.uint8).flatten(), stored.flatten()), rank + + dequant = dequant_mxfp8_weight(expected_w.view(torch.float8_e4m3fn), expected_s, 32) + ref = x.float() @ dequant.cuda().t() + out = lin(x).float() + rel = (out - ref).norm() / ref.norm().clamp_min(1e-6) + # Dynamic MXFP8 activations cost about 1% relative error; a scale row + # on the wrong weight row is off by at least a factor of two. + assert rel < 0.05, (rank, float(rel)) From f4e8e70944bd0e212692a83dc883da57b4dfe523 Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 02/38] [None][perf] MiniMax-M3: BF16-split tensor-core router GEMM MiniMaxM3Gate keeps a BF16 split of its FP32 router weight (three terms along K, built at load_weights, the multi-term emulation of CUTLASS's 3xTF32 GEMMs). BF16 inputs run one cuBLAS BF16 GEMM with FP32 accumulation over the replicated input instead of an FP32 cast plus a SIMT SGEMM; FP32 inputs keep the FP32 cuBLAS GEMM. Top-k parity against an FP64 reference is tested. Signed-off-by: Pranav Shrestha --- .../_torch/models/modeling_minimaxm3.py | 53 ++++++- .../unittest/_torch/models/test_minimax_m3.py | 141 ++++++++++++++++++ 2 files changed, 190 insertions(+), 4 deletions(-) diff --git a/tensorrt_llm/_torch/models/modeling_minimaxm3.py b/tensorrt_llm/_torch/models/modeling_minimaxm3.py index a13b1a9d13a8..4446afbb3cf5 100644 --- a/tensorrt_llm/_torch/models/modeling_minimaxm3.py +++ b/tensorrt_llm/_torch/models/modeling_minimaxm3.py @@ -555,6 +555,33 @@ def _build_swiglu_oai_dense_mlp( ) +ROUTER_BF16_SPLIT_TERMS = 3 +"""int: BF16 terms the FP32 router weight is split into for the tensor-core router GEMM.""" + + +def split_router_weight_bf16( + weight: torch.Tensor, terms: int = ROUTER_BF16_SPLIT_TERMS +) -> torch.Tensor: + """Split the FP32 router weight into BF16 terms laid side by side along K. + + ``weight`` [num_experts, hidden] FP32 becomes [num_experts, terms * hidden] BF16 with + ``weight ~= sum_t result[:, t * hidden:(t + 1) * hidden]``; each term is the BF16 + rounding of what the previous terms left over (three terms carry 24 mantissa bits). A + BF16 activation ``y`` then gets FP32-grade logits from one tensor-core GEMM with FP32 + accumulation, ``cat([y] * terms, -1) @ result.t()``, since BF16 x BF16 products are + exact in FP32. + """ + if weight.dim() != 2 or weight.dtype != torch.float32: + raise ValueError("router weight must be a 2-D FP32 tensor") + pieces = [] + remainder = weight + for _ in range(terms): + piece = remainder.to(torch.bfloat16) + pieces.append(piece) + remainder = remainder - piece.float() + return torch.cat(pieces, dim=1).contiguous() + + class MiniMaxM3Gate(nn.Module): """MiniMax-M3 router gate: float32 sigmoid scoring + per-expert bias correction. @@ -586,12 +613,29 @@ def __init__( torch.empty((num_experts,), dtype=torch.float32), requires_grad=False, ) + # ``weight`` as BF16 terms along K (split_router_weight_bf16) for the + # tensor-core GEMM on BF16 inputs; built by ``load_weights``, else None. + self.register_buffer("split_weight", None, persistent=False) def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: - # Router runs in fp32 to match SGLang. cublas_mm always computes with - # CUBLAS_COMPUTE_32F; F.linear follows torch's fp32 matmul precision, - # which is TF32 under TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 (set in the - # NGC PyTorch based containers) and then flips near-tied top-k choices. + """FP32 router logits ``hidden_states @ weight.t()``. + + BF16 inputs with a split weight run one BF16 tensor-core GEMM over the + replicated input and the BF16 terms of the weight, with FP32 accumulation + and output. Otherwise the router runs in fp32 to match SGLang: cublas_mm + always computes with CUBLAS_COMPUTE_32F, whereas F.linear follows torch's + fp32 matmul precision, which is TF32 under TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 + (set in the NGC PyTorch based containers) and then flips near-tied top-k + choices. + """ + if self.split_weight is not None and hidden_states.dtype == torch.bfloat16: + terms = self.split_weight.shape[1] // self.hidden_size + return torch.ops.trtllm.cublas_mm( + torch.cat([hidden_states] * terms, dim=-1), + self.split_weight.t(), + bias=None, + out_dtype=torch.float32, + ) return torch.ops.trtllm.cublas_mm( hidden_states.to(torch.float32).contiguous(), self.weight.t(), @@ -615,6 +659,7 @@ def load_weights(self, weights: List[Dict]): w = weights[0] if "weight" in w: self.weight.copy_(w["weight"][:].to(self.weight.dtype)) + self.split_weight = split_router_weight_bf16(self.weight) if "e_score_correction_bias" in w: self.e_score_correction_bias.copy_( w["e_score_correction_bias"][:].to(self.e_score_correction_bias.dtype) diff --git a/tests/unittest/_torch/models/test_minimax_m3.py b/tests/unittest/_torch/models/test_minimax_m3.py index 9a8d704ccf14..f7ad75c73a91 100644 --- a/tests/unittest/_torch/models/test_minimax_m3.py +++ b/tests/unittest/_torch/models/test_minimax_m3.py @@ -22,6 +22,7 @@ import json import os +import struct from types import SimpleNamespace from unittest.mock import Mock, create_autospec @@ -2697,6 +2698,142 @@ def test_minimax_m3_gate_stays_fp32_when_tf32_is_allowed(num_tokens: int) -> Non torch.testing.assert_close(graph_logits, replay_reference, rtol=0, atol=0) +def _router_topk_mismatches( + logits: torch.Tensor, reference: torch.Tensor, bias: torch.Tensor, top_k: int, tie: float +) -> int: + """Rows whose sigmoid + bias top-k expert set differs from the FP64 reference. + + Rows whose reference k-th and (k+1)-th scores are within ``tie`` are not counted. + """ + scores = torch.sigmoid(logits.double()) + bias.double() + ref_scores = torch.sigmoid(reference.double()) + bias.double() + got = torch.sort(torch.topk(scores, top_k, dim=-1).indices, dim=-1).values + ref_values, ref_indices = torch.topk(ref_scores, top_k + 1, dim=-1) + want = torch.sort(ref_indices[:, :top_k], dim=-1).values + clear = (ref_values[:, top_k - 1] - ref_values[:, top_k]) > tie + return int(((got != want).any(dim=-1) & clear).sum()) + + +@pytest.mark.cpu_only +def test_minimax_m3_split_router_weight_reconstructs_fp32() -> None: + """The three BF16 terms of the router weight sum back to the FP32 weight exactly.""" + torch.manual_seed(0) + weight = torch.randn(128, 6144) * 0.02 + split = modeling_minimaxm3.split_router_weight_bf16(weight) + assert split.dtype == torch.bfloat16 and split.shape == (128, 3 * 6144) + total = sum(term.double() for term in split.split(6144, dim=1)) + torch.testing.assert_close(total, weight.double(), rtol=0, atol=0) + with pytest.raises(ValueError): + modeling_minimaxm3.split_router_weight_bf16(weight.bfloat16()) + + +@pytest.mark.skipif(not _has_cuda(), reason="MiniMax-M3 router gate needs CUDA") +@pytest.mark.parametrize("rows", [33, 96, 256]) +def test_minimax_m3_gate_bf16_split_matches_fp32_router(rows: int) -> None: + """The BF16-split GEMM matches the FP32 router in logits and top-k. + + Inputs are BF16 norm outputs, as on the serving path. Logits are compared with an + FP64 reference and with the FP32 cuBLAS router; the top-4 sets of sigmoid + bias must + match the FP64 reference on every row that is not a near-tie. + """ + hidden_size, num_experts, top_k = 6144, 128, 4 + torch.manual_seed(rows) + gate = modeling_minimaxm3.MiniMaxM3Gate( + hidden_size, num_experts, top_k=top_k, routed_scaling_factor=1.0 + ).cuda() + weight = torch.randn(num_experts, hidden_size, device="cuda") * 0.02 + bias = torch.randn(num_experts, device="cuda") * 0.05 + gate.load_weights([{"weight": weight, "e_score_correction_bias": bias}]) + assert gate.split_weight is not None + + mismatches, fp32_mismatches, total = 0, 0, 0 + for _ in range(8): + y = torch.randn(rows, hidden_size, device="cuda").bfloat16() + logits = gate(y) + assert logits.dtype == torch.float32 and logits.shape == (rows, num_experts) + reference = y.double() @ weight.double().t() + fp32_logits = torch.ops.trtllm.cublas_mm( + y.float(), weight.t(), bias=None, out_dtype=torch.float32 + ) + scale = float(reference.abs().max()) + assert float((logits.double() - reference).abs().max()) / scale < 1e-5 + assert float((fp32_logits.double() - reference).abs().max()) / scale < 1e-5 + mismatches += _router_topk_mismatches( + logits, reference, gate.e_score_correction_bias, top_k, 1e-5 + ) + fp32_mismatches += _router_topk_mismatches( + fp32_logits, reference, gate.e_score_correction_bias, top_k, 1e-5 + ) + total += rows + assert mismatches == 0, ( + f"{mismatches}/{total} top-{top_k} flips (FP32 router: {fp32_mismatches})" + ) + + # CUDA graph capture and replay of the split path at this row count. + static_y = torch.randn(rows, hidden_size, device="cuda").bfloat16() + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + gate(static_y.clone()) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + graph_logits = gate(static_y) + new_y = torch.randn(rows, hidden_size, device="cuda").bfloat16() + static_y.copy_(new_y) + graph.replay() + torch.testing.assert_close(graph_logits, gate(new_y), rtol=0, atol=0) + + +@pytest.mark.skipif(not _has_cuda(), reason="MiniMax-M3 router gate needs CUDA") +@pytest.mark.skipif( + not os.environ.get("MINIMAX_M3_ROUTER_DUMP_DIR"), + reason="set MINIMAX_M3_ROUTER_DUMP_DIR to dumped router inputs (router_dump_*.pt: y, w, b)", +) +@pytest.mark.parametrize("rows", [33, 96, 256]) +def test_minimax_m3_gate_bf16_split_on_dumped_router_inputs(rows: int) -> None: + """The BF16-split router keeps every top-4 choice of the FP64 reference on real inputs. + + Each dump holds a decode step's BF16 router input ``y`` and the layer's FP32 gate + weight ``w`` and bias ``b``; ``rows`` rows are taken from consecutive dumps. + """ + import glob + + files = sorted( + glob.glob(os.path.join(os.environ["MINIMAX_M3_ROUTER_DUMP_DIR"], "router_dump_*.pt")) + ) + assert files, "no router_dump_*.pt files" + files = files[: int(os.environ.get("MINIMAX_M3_ROUTER_DUMP_MAX", "96"))] + hidden_size, top_k = 6144, 4 + mismatches, fp32_mismatches, total, worst = 0, 0, 0, 0.0 + for start in range(0, len(files), 3): + dumps = [torch.load(f, map_location="cuda") for f in files[start : start + 3]] + y = torch.cat([d["y"] for d in dumps])[:rows].contiguous() + if y.shape[0] < rows: + continue + weight, bias = dumps[0]["w"].float(), dumps[0]["b"].float() + gate = modeling_minimaxm3.MiniMaxM3Gate( + hidden_size, weight.shape[0], top_k=top_k, routed_scaling_factor=1.0 + ).cuda() + gate.load_weights([{"weight": weight, "e_score_correction_bias": bias}]) + logits = gate(y) + reference = y.double() @ weight.double().t() + fp32_logits = torch.ops.trtllm.cublas_mm( + y.float(), weight.t(), bias=None, out_dtype=torch.float32 + ) + worst = max(worst, float((logits.double() - reference).abs().max())) + mismatches += _router_topk_mismatches(logits, reference, bias, top_k, 0.0) + fp32_mismatches += _router_topk_mismatches(fp32_logits, reference, bias, top_k, 0.0) + total += rows + assert total > 0 + print( + f"rows={rows}: {total} rows, top-{top_k} flips split {mismatches} fp32 {fp32_mismatches}, " + f"max |logit err| {worst:.2e}" + ) + assert mismatches == 0 + assert worst < 1e-4 + + @pytest.mark.gpu @pytest.mark.skipif(not _has_cuda(), reason="MiniMax-M3 needs CUDA") def test_text_norm_weights_real_loader_smoke(monkeypatch: pytest.MonkeyPatch): @@ -3119,6 +3256,10 @@ def _nvfp4_activation(x: torch.Tensor, global_scale: float = 1.0) -> torch.Tenso return (_round_to_e2m1(scaled) * scale / global_scale).reshape(x.shape) +def _fp32(v: float) -> float: + return struct.unpack(" Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 03/38] [None][perf] MiniMax-M3 sparse attention: trim per-step host work KV pool and block-scale views are built once per layer and layout, the fused QKV/index producer caches its per-layer-invariant arguments, block-id rows are copied from the page-index buffers, the MSA metadata builders do their batch-sized index arithmetic in numpy, and the NVFP4 decode and CSR prefill wrappers resolve the device capability and the fmha_sm100 API once. Same kernels, same arguments; adds prefix-reuse and BF16-cache decode tests. Signed-off-by: Pranav Shrestha --- .../attention/backends/fmha/msa_prefill.py | 38 +- .../sparse/minimax_m3/cache_manager.py | 46 +- .../backends/sparse/minimax_m3/common.py | 29 +- .../kernels/triton_sparse_decode.py | 33 +- .../backends/sparse/minimax_m3/msa_backend.py | 25 +- .../_torch/models/modeling_minimaxm3.py | 136 ++-- .../msa/test_minimax_m3_prefix_reuse.py | 641 ++++++++++++++++++ .../msa/test_minimax_m3_sparse_attn_decode.py | 60 ++ 8 files changed, 925 insertions(+), 83 deletions(-) create mode 100644 tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_prefix_reuse.py diff --git a/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py b/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py index 120feaafeeee..a560980ad860 100644 --- a/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py +++ b/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py @@ -21,6 +21,7 @@ from __future__ import annotations +import functools from typing import TYPE_CHECKING, Optional import torch @@ -155,6 +156,28 @@ def _aligned_nvfp4_dequant_scales( return k_global_scale, v_global_scale +@functools.lru_cache(maxsize=1) +def _msa_sparse_api(): + """Resolve fmha_sm100's sparse API once; the package cannot change under a run.""" + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + require_msa_module, + ) + + fmha_sm100 = require_msa_module() + sparse = getattr(fmha_sm100, "sparse", None) + if sparse is None: + try: + from fmha_sm100 import sparse + except ImportError as exc: + raise RuntimeError("MiniMax-M3 NVFP4 KV cache requires the Fan MSA sparse API") from exc + if not hasattr(sparse, "build_k2q_csr") or not hasattr(sparse, "sparse_atten_nvfp4_kv_func"): + raise RuntimeError( + "The loaded MSA build lacks build_k2q_csr or " + "sparse_atten_nvfp4_kv_func; use the NVFP4-capable Fan revision" + ) + return sparse + + def run_msa_nvfp4_sparse_gqa( q: torch.Tensor, k_paged: torch.Tensor, @@ -178,22 +201,9 @@ def run_msa_nvfp4_sparse_gqa( """ from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( MSA_REQUIRED_TOPK, - require_msa_module, ) - fmha_sm100 = require_msa_module() - sparse = getattr(fmha_sm100, "sparse", None) - if sparse is None: - try: - from fmha_sm100 import sparse - except ImportError as exc: - raise RuntimeError("MiniMax-M3 NVFP4 KV cache requires the Fan MSA sparse API") from exc - if not hasattr(sparse, "build_k2q_csr") or not hasattr(sparse, "sparse_atten_nvfp4_kv_func"): - raise RuntimeError( - "The loaded MSA build lacks build_k2q_csr or " - "sparse_atten_nvfp4_kv_func; use the NVFP4-capable Fan revision" - ) - + sparse = _msa_sparse_api() for name, scale in (("K", k_global_scale), ("V", v_global_scale)): if scale.dtype != torch.float32 or scale.numel() != 1: raise ValueError(f"MiniMax-M3 NVFP4 {name} dequantization scale must be one FP32 value") diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py index 1b5304d25b85..4563331e9754 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py @@ -30,6 +30,7 @@ from typing import TYPE_CHECKING, List, Optional, Sequence, Tuple +import numpy as np import torch from tensorrt_llm._torch.disaggregation.resource.page import MapperKind @@ -772,6 +773,29 @@ def _kv_slot_geometry( page_shape = [num_kv_heads, self.tokens_per_block, containers] return addr_key, torch_dtype, num_slots, scale, page_shape + def _cached_pool_view(self, kind: str, layer_idx: int, kv_layout: Optional[str], build): + """Return the pool view ``build`` makes, resolved once per layer and layout. + + The pools are allocated for the manager's lifetime, so a view's address + and strides never change; building one costs several native accessor + calls plus a tensor-from-pointer conversion, and the MiniMax-M3 layers + ask for the same views several times per step. + """ + cache = self.__dict__.get("_pool_view_cache") + if cache is None: + cache = self.__dict__["_pool_view_cache"] = {} + key = (kind, int(layer_idx), kv_layout) + view = cache.get(key) + if view is None: + # A view created under inference mode would be an inference tensor + # and could not be updated in place outside it (the cache scrub + # and the eager writers do both); build it as a plain tensor. + with torch.inference_mode(False): + view = build(layer_idx, kv_layout) + if view is not None: + cache[key] = view + return view + def get_buffers( self, layer_idx: int, kv_layout: Optional[str] = None ) -> Optional[torch.Tensor]: @@ -791,6 +815,11 @@ def get_buffers( preserves the dim-0 stride (``scale * page_stride``), so ``view[s, 0/1, ...]`` lands on this layer's K/V at slot ``s``. """ + return self._cached_pool_view("kv", layer_idx, kv_layout, self._build_kv_buffers) + + def _build_kv_buffers( + self, layer_idx: int, kv_layout: Optional[str] = None + ) -> Optional[torch.Tensor]: addr_key, torch_dtype, num_slots, scale, page_shape = self._kv_slot_geometry( layer_idx, kv_layout ) @@ -861,6 +890,13 @@ def get_block_scale_buffers( self, layer_idx: int, kv_layout: Optional[str] = None ) -> torch.Tensor: """Return paged NVFP4 K+V E4M3 scale-byte views for ``layer_idx``.""" + return self._cached_pool_view( + "scale", layer_idx, kv_layout, self._build_block_scale_buffers + ) + + def _build_block_scale_buffers( + self, layer_idx: int, kv_layout: Optional[str] = None + ) -> torch.Tensor: addr_key, torch_dtype, num_slots, scale, page_shape = self._kv_scale_slot_geometry( layer_idx, kv_layout ) @@ -975,9 +1011,15 @@ def get_block_ids_per_seq(self, request_ids): The rows are written through a numpy view of a single zero-filled, pinned result, so the attention metadata builders ship it to the device - in one asynchronous copy. + in one asynchronous copy. Each row is copied from the request's pool-0 + page-index buffer as an array: the buffer spans the whole + ``max_seq_len`` capacity, so going through the per-element Python + lists of :meth:`get_batch_cache_indices` costs host time on every + decode step in proportion to ``max_seq_len``, not to the live context. """ - block_ids_per_seq = self.get_batch_cache_indices(request_ids) + block_ids_per_seq = [ + np.asarray(self.kv_cache_map[req_id].get_base_page_indices(0)) for req_id in request_ids + ] batch = len(block_ids_per_seq) max_blocks = max((len(block_ids) for block_ids in block_ids_per_seq), default=0) padded_tensor = torch.zeros( diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py index 93df670a4137..20e4fcff808a 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py @@ -13,6 +13,7 @@ from dataclasses import dataclass, field from typing import TYPE_CHECKING, Any, List, Literal, NamedTuple, Optional, Tuple +import numpy as np import torch from tensorrt_llm._utils import async_tensor_h2d, maybe_pin_memory @@ -299,25 +300,31 @@ def build_paged_kv_slot_mapping( # Expanding the per-request lengths on the host reproduces the same slot # ids as indexing req_to_token, without copying that [batch, max_kv_len] # grid back from the device. - qo = qo_lens_cpu.to(torch.long) + # Host arithmetic on batch-sized vectors in numpy: the torch CPU ops this + # replaced (repeat_interleave, cumsum, gather) each cost 100-200 us of + # dispatch and OpenMP fork/join, every step, for a few dozen elements. + qo = qo_lens_cpu.numpy().astype(np.int64, copy=False) total_q = int(qo.sum()) if total_q == 0: out_cache_loc_cpu = torch.empty(0, dtype=torch.int32) else: - row = torch.repeat_interleave(torch.arange(batch, dtype=torch.long), qo) - starts = torch.cumsum(qo, 0) - qo - pos = qo_offset_cpu.to(torch.long)[row] + ( - torch.arange(total_q, dtype=torch.long) - starts[row] + row = np.repeat(np.arange(batch, dtype=np.int64), qo) + starts = np.cumsum(qo) - qo + pos = qo_offset_cpu.numpy().astype(np.int64, copy=False)[row] + ( + np.arange(total_q, dtype=np.int64) - starts[row] ) # A zero-length CUDA-graph padding row offsets to -1. Its slot is a # placeholder no forward reads, so keep it in the row instead of # indexing off the table. - pos.clamp_(min=0, max=max(max_kv_len - 1, 0)) - block_col = torch.div(pos, tokens_per_block, rounding_mode="floor") - out_cache_loc_cpu = ( - block_ids[row, block_col].to(torch.long) * tokens_per_block - + (pos - block_col * tokens_per_block) - ).to(torch.int32) + np.clip(pos, 0, max(max_kv_len - 1, 0), out=pos) + block_col = pos // tokens_per_block + block_ids_np = block_ids.numpy() + out_cache_loc_cpu = torch.from_numpy( + ( + block_ids_np[row, block_col].astype(np.int64) * tokens_per_block + + (pos - block_col * tokens_per_block) + ).astype(np.int32) + ) out_cache_loc = async_tensor_h2d(out_cache_loc_cpu, torch.int32, device) return PagedKvSlotMapping(req_to_token, slot_ids, out_cache_loc, block_ids) diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/triton_sparse_decode.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/triton_sparse_decode.py index 805cccc28a41..72f3627eeb72 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/triton_sparse_decode.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/triton_sparse_decode.py @@ -801,6 +801,26 @@ def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> i # NVFP4 instructions are supported on both datacenter Blackwell targets. _SM100F_CAPABILITIES = ((10, 0), (10, 3)) + + +# Devices already found to support the NVFP4 decode kernels, by capability. +_NVFP4_DEVICE_CAPABILITIES: dict[torch.device, tuple[int, int]] = {} + + +def _nvfp4_device_capability(device: torch.device) -> tuple[int, int]: + """Return the device's capability, checking NVFP4 support once per device.""" + capability = _NVFP4_DEVICE_CAPABILITIES.get(device) + if capability is None: + capability = torch.cuda.get_device_capability(device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _NVFP4_DEVICE_CAPABILITIES[device] = capability + return capability + + # Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. _NVFP4_TUNED_CAPABILITIES = ((10, 3),) @@ -1069,13 +1089,9 @@ def minimax_m3_sparse_attn_decode( nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) kv_nvfp4 = any(arg is not None for arg in nvfp4_args) scale_cols = head_dim // NVFP4_SF_VEC_SIZE + # Checked once per device; the shape heuristics below consult it per call. + capability = _nvfp4_device_capability(q.device) if kv_nvfp4 else None if kv_nvfp4: - capability = torch.cuda.get_device_capability(q.device) - if capability not in _SM100F_CAPABILITIES: - raise NotImplementedError( - "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103; " - f"got SM{capability[0] * 10 + capability[1]}." - ) _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) # Two E2M1 elements per byte; Triton must see them unsigned so that the # high-nibble shift does not sign-extend. @@ -1113,6 +1129,7 @@ def minimax_m3_sparse_attn_decode( gqa_group_size=gqa_group_size, max_topk=max_topk, decode_query_len=decode_query_len, + capability=capability, ) if kv_nvfp4 else 1 @@ -1128,6 +1145,7 @@ def minimax_m3_sparse_attn_decode( gqa_group_size=gqa_group_size, max_topk=max_topk, decode_query_len=decode_query_len, + capability=capability, ) if kv_nvfp4 else None @@ -1177,6 +1195,7 @@ def minimax_m3_sparse_attn_decode( gqa_group_size=gqa_group_size, max_topk=max_topk, decode_query_len=decode_query_len, + capability=capability, ) if query_group_size == 2: launch_options = {"num_warps": 4, "num_stages": 1} @@ -1188,6 +1207,7 @@ def minimax_m3_sparse_attn_decode( gqa_group_size=gqa_group_size, max_topk=max_topk, decode_query_len=decode_query_len, + capability=capability, ) if kv_nvfp4 else {} @@ -1276,6 +1296,7 @@ def minimax_m3_sparse_attn_decode( gqa_group_size=gqa_group_size, max_topk=max_topk, decode_query_len=decode_query_len, + capability=capability, ) if kv_nvfp4 else {} diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py index d871bc9d8d91..b5022eb77256 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py @@ -34,6 +34,7 @@ from dataclasses import dataclass from typing import NamedTuple, Optional, Tuple +import numpy as np import torch from tensorrt_llm._torch.attention.backends.interface import AttentionForwardArgs @@ -1158,19 +1159,21 @@ def _build_msa_fields(self) -> None: # Inputs for on_update_kv_lens: each query token's (request row, offset # in request). Pinned and non-blocking so the copies do not synchronize # the stream. The slots themselves come from msa_block_table above. - qo_long = qo_lens_cpu.to(torch.long) - batch_row_cpu = torch.repeat_interleave( - torch.arange(batch_size, dtype=torch.int32), qo_long - ) - starts = torch.cumsum(qo_long, 0) - qo_long - intra_cpu = ( - torch.arange(total_new_tokens, dtype=torch.int64) - - torch.repeat_interleave(starts, qo_long) - ).to(torch.int32) + # Host arithmetic on a batch-sized vector: numpy, not torch. Each CPU + # torch op here (repeat_interleave above all) costs 100-200 us of + # dispatch and OpenMP fork/join per call, and this runs every step. + qo_np = qo_lens_cpu.numpy().astype(np.int64, copy=False) + batch_row_np = np.repeat(np.arange(batch_size, dtype=np.int32), qo_np) + starts_np = np.cumsum(qo_np) - qo_np + intra_np = ( + np.arange(total_new_tokens, dtype=np.int64) - np.repeat(starts_np, qo_np) + ).astype(np.int32) self.msa_q_batch_row[:total_new_tokens].copy_( - maybe_pin_memory(batch_row_cpu), non_blocking=True + maybe_pin_memory(torch.from_numpy(batch_row_np)), non_blocking=True + ) + self.msa_q_intra[:total_new_tokens].copy_( + maybe_pin_memory(torch.from_numpy(intra_np)), non_blocking=True ) - self.msa_q_intra[:total_new_tokens].copy_(maybe_pin_memory(intra_cpu), non_blocking=True) # The staged lens are the upper bound on_update_kv_lens clamps to: the # same attended lengths msa_seq_lens_cuda was just filled from, kept # apart so the clamp stays fixed however often the hook runs. diff --git a/tensorrt_llm/_torch/models/modeling_minimaxm3.py b/tensorrt_llm/_torch/models/modeling_minimaxm3.py index 4446afbb3cf5..f78d6171d379 100644 --- a/tensorrt_llm/_torch/models/modeling_minimaxm3.py +++ b/tensorrt_llm/_torch/models/modeling_minimaxm3.py @@ -1514,20 +1514,110 @@ def _fused_fp8_qkv_indexer_norm_rope_kv_insert( or not isinstance(self.attn, MiniMaxM3MsaSparseAttention) or not (self.main_kv_is_fp8 or self.main_kv_is_nvfp4) or self.attn.indexer_kv_dtype != "fp8" + or packed.dtype != torch.bfloat16 + or position_ids is None ): return None + kv_cache_manager = attn_metadata.kv_cache_manager + if cache_tensors is None: + if kv_cache_manager is None: + return None + buffers = kv_cache_manager.get_buffers(self.layer_idx, kv_layout="HND") + index_k_cache = attn_metadata.msa_idx_k_cache(self.layer_idx) + out_cache_loc = getattr(attn_metadata, "msa_out_cache_loc", None) + else: + buffers, index_k_cache, out_cache_loc = cache_tensors + if out_cache_loc is None: + return None + plan = self._fused_producer_plan(kv_cache_manager, buffers, index_k_cache, out_cache_loc) + if plan is None: + return None + num_tokens = int(packed.shape[0]) + if out_cache_loc.numel() < num_tokens: + return None + layer_uses_nvfp4, main_cache, scale_buffers, kv_quant_scale, static_args = plan + positions = position_ids.reshape(-1).contiguous().to(torch.int32) + if layer_uses_nvfp4: + q, index_q = torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed.contiguous(), + main_cache, + scale_buffers, + index_k_cache, + out_cache_loc[:num_tokens], + kv_quant_scale, + *static_args, + positions, + ) + else: + q, index_q = torch.ops.trtllm.minimax_m3_fp8_qkv_indexer_norm_rope_kv_insert( + packed.contiguous(), + main_cache, + index_k_cache, + out_cache_loc[:num_tokens], + *static_args, + positions, + ) + return q.flatten(1), index_q.flatten(1) + + def _fused_producer_plan( + self, + kv_cache_manager, + buffers: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + ) -> Optional[tuple]: + """Resolve the horizontal producer's per-layer-invariant inputs once. + + Everything the producer validates besides the step's tokens (the + geometry and strides of the paged caches, the RoPE table, the norm + weights and epsilons) is fixed for a given set of cache views and + parameters, so the checks run once per distinct set and the resolved + arguments are reused. The key is the storage identity of every tensor + involved: a reloaded parameter or a new cache manager starts a fresh + entry. Returns ``None`` when the fused producer does not apply. + """ + rotary_cos_sin = self.rotary_emb.rotary_cos_sin if self.rotary_emb is not None else None + key = ( + id(kv_cache_manager), + None if buffers is None else buffers.data_ptr(), + index_k_cache.data_ptr(), + out_cache_loc.data_ptr(), + out_cache_loc.dtype, + None if rotary_cos_sin is None else rotary_cos_sin.data_ptr(), + tuple( + module.weight.data_ptr() + for module in (self.q_norm, self.k_norm, self.index_q_norm, self.index_k_norm) + ), + ) + cache = self.__dict__.get("_fused_producer_plans") + if cache is None: + cache = self.__dict__["_fused_producer_plans"] = {} + if key in cache: + return cache[key] + plan = self._resolve_fused_producer_plan( + kv_cache_manager, buffers, index_k_cache, out_cache_loc, rotary_cos_sin + ) + cache[key] = plan + return plan + + def _resolve_fused_producer_plan( + self, + kv_cache_manager, + buffers: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + rotary_cos_sin: Optional[torch.Tensor], + ) -> Optional[tuple]: rope = self.pos_embd_params.rope if self.pos_embd_params is not None else None rotary_dim = int(rope.dim) if rope is not None else 0 if ( - packed.dtype != torch.bfloat16 - or position_ids is None - or self.head_dim != 128 + self.head_dim != 128 or self.sparse_index_dim != 128 or self.sparse_num_index_heads != self.num_key_value_heads or not self.use_gemma_norm or self.pos_embd_params is None or not self.pos_embd_params.is_neox - or self.rotary_emb is None + or rotary_cos_sin is None or rope is None or rotary_dim != 64 ): @@ -1547,16 +1637,6 @@ def _fused_fp8_qkv_indexer_norm_rope_kv_insert( if any(weight.dtype != torch.bfloat16 or not weight.is_cuda for weight in norm_weights): return None - kv_cache_manager = attn_metadata.kv_cache_manager - if cache_tensors is None: - if kv_cache_manager is None: - return None - buffers = kv_cache_manager.get_buffers(self.layer_idx, kv_layout="HND") - index_k_cache = attn_metadata.msa_idx_k_cache(self.layer_idx) - out_cache_loc = getattr(attn_metadata, "msa_out_cache_loc", None) - else: - buffers, index_k_cache, out_cache_loc = cache_tensors - num_tokens = int(packed.shape[0]) layer_uses_nvfp4 = bool( getattr(kv_cache_manager, "is_nvfp4_layer", lambda _layer_idx: False)(self.layer_idx) ) @@ -1614,7 +1694,6 @@ def _fused_fp8_qkv_indexer_norm_rope_kv_insert( and buffers is not None and index_k_cache.shape[0] == buffers.shape[0] ) - rotary_cos_sin = self.rotary_emb.rotary_cos_sin supported_rope_cache = ( rotary_cos_sin.is_cuda and rotary_cos_sin.dtype == torch.float32 @@ -1626,16 +1705,12 @@ def _fused_fp8_qkv_indexer_norm_rope_kv_insert( not supported_main_cache or not supported_index_cache or not supported_rope_cache - or out_cache_loc is None or not out_cache_loc.is_cuda or out_cache_loc.dtype != torch.int32 or not out_cache_loc.is_contiguous() - or out_cache_loc.numel() < num_tokens ): return None - - common_args = ( - out_cache_loc[:num_tokens], + static_args = ( self.num_heads, self.num_key_value_heads, self.sparse_num_index_heads, @@ -1647,26 +1722,9 @@ def _fused_fp8_qkv_indexer_norm_rope_kv_insert( self.index_q_norm.weight, self.index_k_norm.weight, rotary_cos_sin, - position_ids.reshape(-1).contiguous().to(torch.int32), ) - if layer_uses_nvfp4: - q, index_q = torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( - packed.contiguous(), - buffers.view(torch.uint8), - scale_buffers, - index_k_cache, - common_args[0], - kv_quant_scale, - *common_args[1:], - ) - else: - q, index_q = torch.ops.trtllm.minimax_m3_fp8_qkv_indexer_norm_rope_kv_insert( - packed.contiguous(), - buffers, - index_k_cache, - *common_args, - ) - return q.flatten(1), index_q.flatten(1) + main_cache = buffers.view(torch.uint8) if layer_uses_nvfp4 else buffers + return layer_uses_nvfp4, main_cache, scale_buffers, kv_quant_scale, static_args def _expect_fused_qk_norm_rope(self, position_ids: Optional[torch.Tensor]) -> bool: """Whether the fused kernel is expected to run instead of the fallback. diff --git a/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_prefix_reuse.py b/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_prefix_reuse.py new file mode 100644 index 000000000000..7289cb33e128 --- /dev/null +++ b/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_prefix_reuse.py @@ -0,0 +1,641 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 prefix reuse on the MSA path: one mixed hit/miss case with graph refresh. + +Component-level: the real ``MiniMaxM3KVCacheManagerV2`` with block reuse enabled and +the MSA attention metadata, at the per-rank M3 geometry (60 sparse layers, 1 KV head, +head and index dim 128, 128-token pages). No checkpoint is loaded. K, V and index-K are +written with the backend's fused cache writer from sentinel tensors that encode +(writer, layer, role, page ordinal, offset, step), so every cache read can be checked +for exact equality. + +Flow, mirroring ``KVCacheV2Scheduler`` and ``PyExecutor`` call order: + +1. A 2311-token publisher prefills, commits and stays live in generation. +2. Its 2433-token extension and an unrelated 2177-token miss are scheduled together. + Expected exact reuse: 2311 for the extension, 0 for the miss. +3. Sentinels at all 60 layers verify the page tables (shared full pages, disjoint miss + pages), suffix placement, that the publisher's pages are untouched by the extension + and the miss, and that the publisher's first decode write does not reach the + extension's copy of the partial page. +4. Two real decode rows are prepared in a CUDA-graph metadata, the write kernel is + captured and replayed, then the rows are reversed with new tags and replayed again. +5. The extension is released; the graph replays with the survivor plus the graph's + padding row. The released pages stay unchanged and the pool's storage counts return + to their initial values once every request is freed. +""" + +from __future__ import annotations + +import gc +from dataclasses import dataclass, field + +import pytest +import torch + +import tensorrt_llm.bindings +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.cache_manager import ( + MiniMaxM3KVCacheManagerV2, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_scatter import ( + fused_write_layer_caches, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.msa_backend import ( + MiniMaxM3MsaSparseAttentionMetadata, +) +from tensorrt_llm._torch.metadata import KVCacheParams +from tensorrt_llm._torch.pyexecutor.llm_request import LlmRequest, SamplingConfig +from tensorrt_llm._torch.pyexecutor.scheduler import ScheduledRequests +from tensorrt_llm.bindings import DataType +from tensorrt_llm.bindings.internal.batch_manager import CacheType +from tensorrt_llm.llmapi.llm_args import KvCacheConfig, MiniMaxM3SparseAttentionConfig +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.runtime.kv_cache_manager_v2 import BAD_PAGE_INDEX, GPU_LEVEL + +LlmRequestState = tensorrt_llm.bindings.LlmRequestState + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA cache pools") + +# Per-rank M3 geometry (TP4/EP4 deployment): every target layer is sparse and carries +# K, V and index-K, which is what makes the coalesced pool page 60 * 3 * 128 * 128 * 2 +# = 5,898,240 bytes. +NUM_LAYERS = 60 +NUM_KV_HEADS = 1 +HEAD_DIM = 128 +INDEX_DIM = 128 +TOKENS_PER_BLOCK = 128 +PAGE_BYTES = NUM_LAYERS * 3 * TOKENS_PER_BLOCK * HEAD_DIM * 2 +POOL_PAGES = 72 +MAX_SEQ_LEN = 2560 +MAX_BATCH_SIZE = 4 +GRAPH_BATCH = 2 + +PUBLISHER_LEN = 2311 +EXTENSION_LEN = 2433 +MISS_LEN = 2177 + +PUBLISHER_ID, EXTENSION_ID, MISS_ID, PADDING_ID = 11, 12, 13, 99 +# Writer codes stored in the sentinel's first element. +PUBLISHER_CODE, EXTENSION_CODE, MISS_CODE, PADDING_CODE = 1, 2, 3, 4 +ROLE_K, ROLE_V, ROLE_INDEX_K = 0, 1, 2 + +# Step tags stored in the sentinel's sixth element. +TAG_PUBLISHER_PREFILL = 1 +TAG_MIXED_PREFILL = 2 +TAG_PUBLISHER_DECODE = 3 +TAG_GRAPH_FORWARD = 4 +TAG_GRAPH_REVERSED = 5 +TAG_GRAPH_SURVIVOR = 6 + + +def _sentinel(code: int, layer: int, role: int, positions: list[int], tag: int) -> torch.Tensor: + """One ``[len(positions), HEAD_DIM]`` BF16 row per position; all entries are small + integers, which BF16 represents exactly.""" + rows = torch.zeros((len(positions), HEAD_DIM), dtype=torch.float32) + pos = torch.tensor(positions, dtype=torch.float32) + rows[:, 0] = code + rows[:, 1] = layer + rows[:, 2] = role + rows[:, 3] = pos % TOKENS_PER_BLOCK + rows[:, 4] = pos // TOKENS_PER_BLOCK + rows[:, 5] = tag + return rows.to(torch.bfloat16).cuda() + + +def _make_manager() -> MiniMaxM3KVCacheManagerV2: + return MiniMaxM3KVCacheManagerV2( + KvCacheConfig( + max_gpu_total_bytes=POOL_PAGES * PAGE_BYTES, + host_cache_size=0, + enable_block_reuse=True, + enable_partial_reuse=True, + copy_on_partial_reuse=True, + use_kv_cache_manager_v2=True, + enable_swa_scratch_reuse=False, + ), + CacheType.SELF, + num_layers=NUM_LAYERS, + num_kv_heads=NUM_KV_HEADS, + head_dim=HEAD_DIM, + tokens_per_block=TOKENS_PER_BLOCK, + max_seq_len=MAX_SEQ_LEN, + max_batch_size=MAX_BATCH_SIZE, + max_num_tokens=MAX_SEQ_LEN, + mapping=Mapping(world_size=1, rank=0, tp_size=1, pp_size=1), + dtype=DataType.BF16, + vocab_size=65536, + sparse_attention_config=MiniMaxM3SparseAttentionConfig( + implementation="msa", indexer_kv_dtype="bf16", fuse_qkv_index_projection=False + ), + sparse_layer_ids=list(range(NUM_LAYERS)), + sparse_index_dim=INDEX_DIM, + ) + + +def _make_request(request_id: int, tokens: list[int]) -> LlmRequest: + return LlmRequest( + request_id=request_id, + max_new_tokens=8, + input_tokens=tokens, + sampling_config=SamplingConfig(1), + is_streaming=False, + ) + + +def _page_table(manager: MiniMaxM3KVCacheManagerV2, request_id: int) -> list[int]: + """Slot ids of the request's pages, in block order, without the padding entries.""" + return [ + slot for slot in manager.get_batch_cache_indices([request_id])[0] if slot != BAD_PAGE_INDEX + ] + + +def _layer_views( + manager: MiniMaxM3KVCacheManagerV2, layer: int +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """``[num_slots, 1, TOKENS_PER_BLOCK, dim]`` HND views of K, V and index-K, as the + backend's ``write_layer_caches`` resolves them.""" + kv = manager.get_buffers(layer, kv_layout="HND") + idx = manager.get_index_k_buffer(layer) + assert idx is not None, f"layer {layer} has no index-K cache" + return kv[:, 0], kv[:, 1], idx + + +def _gather(view: torch.Tensor, page_table: list[int], positions: list[int]) -> torch.Tensor: + pages = torch.tensor([page_table[p // TOKENS_PER_BLOCK] for p in positions], device="cuda") + within = torch.tensor([p % TOKENS_PER_BLOCK for p in positions], device="cuda") + return view[pages, 0, within, :] + + +@dataclass +class _Ledger: + """What each request's cache must read back: position -> (writer code, step tag).""" + + entries: dict[int, dict[int, tuple[int, int]]] = field(default_factory=dict) + + def record(self, request_id: int, code: int, positions: list[int], tag: int) -> None: + self.entries.setdefault(request_id, {}).update({p: (code, tag) for p in positions}) + + def share(self, request_id: int, from_request_id: int, upto: int) -> None: + """The reused prefix reads back the publisher's writes.""" + source = self.entries[from_request_id] + self.entries.setdefault(request_id, {}).update( + {p: source[p] for p in range(upto) if p in source} + ) + + +def _check_request( + manager: MiniMaxM3KVCacheManagerV2, request_id: int, ledger: _Ledger, label: str +) -> None: + """Every recorded position of the request reads back its sentinel at all layers.""" + expected_by_pos = ledger.entries[request_id] + positions = sorted(expected_by_pos) + page_table = _page_table(manager, request_id) + assert len(page_table) * TOKENS_PER_BLOCK > positions[-1], ( + f"{label}: request {request_id} has {len(page_table)} pages for position {positions[-1]}" + ) + groups: dict[tuple[int, int], list[int]] = {} + for p in positions: + groups.setdefault(expected_by_pos[p], []).append(p) + for layer in range(NUM_LAYERS): + views = _layer_views(manager, layer) + for role, view in enumerate(views): + got = _gather(view, page_table, positions) + expected = torch.cat( + [_sentinel(code, layer, role, group, tag) for (code, tag), group in groups.items()] + ) + order = torch.tensor( + [p for group in groups.values() for p in group], device="cuda" + ).argsort() + expected = expected[order] + if not torch.equal(got, expected): + bad = (got != expected).any(dim=1).nonzero().flatten().tolist() + first = bad[0] + raise AssertionError( + f"{label}: request {request_id} layer {layer} role {role}: " + f"{len(bad)} of {len(positions)} positions differ; position " + f"{positions[first]} read {got[first, :6].float().tolist()} expected " + f"{expected[first, :6].float().tolist()}" + ) + + +def _storage_counts(manager: MiniMaxM3KVCacheManagerV2) -> list[tuple[int, int, int, int, int]]: + return [ + (int(s.total), int(s.free), int(s.evictable), int(s.available), int(s.unavailable)) + for s in manager.impl.get_storage_statistics(GPU_LEVEL) + ] + + +def _prepare( + metadata: MiniMaxM3MsaSparseAttentionMetadata, + rows: list[tuple[LlmRequest, int, int]], + num_contexts: int, +) -> None: + """Stage one step: ``rows`` are (request, query tokens, cached tokens) in batch order.""" + metadata.request_ids = [req.py_request_id for req, _, _ in rows] + metadata.prompt_lens = [req.prompt_len for req, _, _ in rows] + metadata.num_contexts = num_contexts + metadata.kv_cache_params = KVCacheParams( + use_cache=True, num_cached_tokens_per_seq=[cached for _, _, cached in rows] + ) + metadata.seq_lens = torch.tensor([q for _, q, _ in rows], dtype=torch.int) + metadata.prepare() + + +def _expected_slots( + manager: MiniMaxM3KVCacheManagerV2, rows: list[tuple[int, list[int]]] +) -> list[int]: + """The slot per new token the staged ``msa_out_cache_loc`` must hold, from the page tables.""" + slots = [] + for request_id, positions in rows: + table = _page_table(manager, request_id) + slots.extend( + table[p // TOKENS_PER_BLOCK] * TOKENS_PER_BLOCK + p % TOKENS_PER_BLOCK + for p in positions + ) + return slots + + +def _write_eager( + manager: MiniMaxM3KVCacheManagerV2, + metadata: MiniMaxM3MsaSparseAttentionMetadata, + rows: list[tuple[int, list[int]]], + tag: int, +) -> None: + """Write this step's sentinels through the staged slots, as ``write_layer_caches`` does.""" + num_tokens = sum(len(positions) for _, positions in rows) + out_cache_loc = metadata.msa_out_cache_loc[:num_tokens] + for layer in range(NUM_LAYERS): + k_view, v_view, idx_view = _layer_views(manager, layer) + sources = [ + torch.cat([_sentinel(code, layer, role, positions, tag) for code, positions in rows]) + for role in (ROLE_K, ROLE_V, ROLE_INDEX_K) + ] + assert fused_write_layer_caches(k_view, v_view, idx_view, out_cache_loc, *sources), ( + f"fused writer declined layer {layer}" + ) + + +def _schedule_context(manager: MiniMaxM3KVCacheManagerV2, request: LlmRequest) -> int: + """One scheduling pass for a context request; returns the reused token count.""" + assert manager.prepare_context(request), f"prepare_context failed for {request.py_request_id}" + reused = manager.kv_cache_map[request.py_request_id].num_committed_tokens + assert manager.resize_context(request, request.context_remaining_length), ( + f"resize_context failed for {request.py_request_id}" + ) + return reused + + +def _finish_context(manager: MiniMaxM3KVCacheManagerV2, requests: list[LlmRequest]) -> None: + """Post-forward bookkeeping: advance the chunk, commit the blocks, enter generation.""" + batch = ScheduledRequests() + for request in requests: + batch.append_context_request(request) + request.move_to_next_context_chunk() + assert request.context_remaining_length == 0 + request.state = LlmRequestState.GENERATION_IN_PROGRESS + manager.update_context_resources(batch) + for request in requests: + request.add_new_token(7, 0) + + +def _finish_generation(manager: MiniMaxM3KVCacheManagerV2, requests: list[LlmRequest]) -> None: + batch = ScheduledRequests() + for request in requests: + request.add_new_token(7, 0) + batch.generation_requests.append(request) + manager.update_resources(batch) + + +class _GraphWriter: + """The per-layer fused cache write for ``GRAPH_BATCH`` decode rows, captured once. + + Sources are static buffers refilled before each replay; the slots are read from the + metadata's graph-stable ``msa_out_cache_loc``, which ``prepare()`` refreshes in place. + """ + + def __init__( + self, manager: MiniMaxM3KVCacheManagerV2, metadata: MiniMaxM3MsaSparseAttentionMetadata + ) -> None: + self.manager = manager + self.metadata = metadata + self.sources = [ + [ + torch.zeros((GRAPH_BATCH, HEAD_DIM), dtype=torch.bfloat16, device="cuda") + for _ in range(3) + ] + for _ in range(NUM_LAYERS) + ] + self.graph = torch.cuda.CUDAGraph() + self.slot_ptr = metadata.msa_out_cache_loc.data_ptr() + + def _launch(self) -> None: + out_cache_loc = self.metadata.msa_out_cache_loc[:GRAPH_BATCH] + for layer in range(NUM_LAYERS): + k_view, v_view, idx_view = _layer_views(self.manager, layer) + assert fused_write_layer_caches( + k_view, v_view, idx_view, out_cache_loc, *self.sources[layer] + ) + + def capture(self) -> None: + # Warm up with every slot negative so the compile pass writes nothing. + staged = self.metadata.msa_out_cache_loc[:GRAPH_BATCH].clone() + self.metadata.msa_out_cache_loc[:GRAPH_BATCH].fill_(-1) + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + self._launch() + torch.cuda.current_stream().wait_stream(side) + torch.cuda.synchronize() + with torch.cuda.graph(self.graph): + self._launch() + self.metadata.msa_out_cache_loc[:GRAPH_BATCH].copy_(staged) + + def replay(self, rows: list[tuple[int, list[int]]], tag: int) -> None: + """``rows`` are (writer code, positions) per graph row, padding rows excluded.""" + assert self.metadata.msa_out_cache_loc.data_ptr() == self.slot_ptr, "slot buffer moved" + for layer in range(NUM_LAYERS): + for role in (ROLE_K, ROLE_V, ROLE_INDEX_K): + values = torch.cat( + [_sentinel(code, layer, role, positions, tag) for code, positions in rows] + ) + self.sources[layer][role][: values.shape[0]].copy_(values) + self.graph.replay() + torch.cuda.synchronize() + + +def test_mixed_hit_miss_prefix_reuse_with_graph_refresh() -> None: + torch.cuda.init() + gc.collect() + torch.cuda.empty_cache() + manager = _make_manager() + ledger = _Ledger() + try: + assert manager.enable_block_reuse + assert manager.tokens_per_block == TOKENS_PER_BLOCK + assert sorted(manager.sparse_layer_ids) == list(range(NUM_LAYERS)) + initial_free_blocks = manager.get_num_free_blocks() + initial_counts = _storage_counts(manager) + + # The graph's padding row, allocated once up front as the CUDA-graph runner does, + # so its page is fixed before the extension exists. + padding = manager.add_dummy_requests([PADDING_ID], is_gen=True)[0] + padding_table = _page_table(manager, PADDING_ID) + assert len(padding_table) == 1 + + eager = MiniMaxM3MsaSparseAttentionMetadata( + max_num_requests=MAX_BATCH_SIZE, + max_num_tokens=MAX_SEQ_LEN, + kv_cache_manager=manager, + mapping=manager.mapping, + ) + + # 1. Publisher prefill, commit, live in generation. + publisher_tokens = [1000 + i for i in range(PUBLISHER_LEN)] + publisher = _make_request(PUBLISHER_ID, publisher_tokens) + assert _schedule_context(manager, publisher) == 0 + assert publisher.context_current_position == 0 + publisher_positions = list(range(PUBLISHER_LEN)) + _prepare(eager, [(publisher, PUBLISHER_LEN, 0)], num_contexts=1) + assert eager.msa_out_cache_loc[:PUBLISHER_LEN].tolist() == _expected_slots( + manager, [(PUBLISHER_ID, publisher_positions)] + ) + _write_eager(manager, eager, [(PUBLISHER_CODE, publisher_positions)], TAG_PUBLISHER_PREFILL) + _finish_context(manager, [publisher]) + ledger.record(PUBLISHER_ID, PUBLISHER_CODE, publisher_positions, TAG_PUBLISHER_PREFILL) + publisher_table = _page_table(manager, PUBLISHER_ID) + assert len(publisher_table) == -(-PUBLISHER_LEN // TOKENS_PER_BLOCK) == 19 + assert manager.kv_cache_map[PUBLISHER_ID].num_committed_tokens == PUBLISHER_LEN + _check_request(manager, PUBLISHER_ID, ledger, "after publisher prefill") + + # 2. Extension (hit) and miss scheduled together while the publisher is live. + extension = _make_request( + EXTENSION_ID, + publisher_tokens + [3000 + i for i in range(EXTENSION_LEN - PUBLISHER_LEN)], + ) + miss = _make_request(MISS_ID, [50000 + i for i in range(MISS_LEN)]) + reused_extension = _schedule_context(manager, extension) + reused_miss = _schedule_context(manager, miss) + assert (reused_extension, reused_miss) == (PUBLISHER_LEN, 0), ( + f"exact reuse expected (2311, 0), got ({reused_extension}, {reused_miss})" + ) + print(f"reuse: extension={reused_extension} miss={reused_miss}") + assert extension.context_current_position == PUBLISHER_LEN + assert extension.context_chunk_size == EXTENSION_LEN - PUBLISHER_LEN + assert miss.context_current_position == 0 + + extension_table = _page_table(manager, EXTENSION_ID) + miss_table = _page_table(manager, MISS_ID) + full_pages = PUBLISHER_LEN // TOKENS_PER_BLOCK # 18 + assert extension_table[:full_pages] == publisher_table[:full_pages], ( + "the reused full pages must be the publisher's pages" + ) + assert len(extension_table) == -(-EXTENSION_LEN // TOKENS_PER_BLOCK) == 20 + assert len(miss_table) == -(-MISS_LEN // TOKENS_PER_BLOCK) == 18 + assert not set(miss_table) & (set(publisher_table) | set(extension_table)), ( + "the miss must not share pages with the publisher or the extension" + ) + assert not set(miss_table) & set(padding_table) + assert not set(extension_table) & set(padding_table) + + # 3. Mixed prefill: extension suffix plus the whole miss, one step. + suffix_positions = list(range(PUBLISHER_LEN, EXTENSION_LEN)) + miss_positions = list(range(MISS_LEN)) + _prepare( + eager, + [(extension, len(suffix_positions), PUBLISHER_LEN), (miss, MISS_LEN, 0)], + num_contexts=2, + ) + assert eager.msa_out_cache_loc[ + : len(suffix_positions) + MISS_LEN + ].tolist() == _expected_slots( + manager, [(EXTENSION_ID, suffix_positions), (MISS_ID, miss_positions)] + ) + _write_eager( + manager, + eager, + [(EXTENSION_CODE, suffix_positions), (MISS_CODE, miss_positions)], + TAG_MIXED_PREFILL, + ) + _finish_context(manager, [extension, miss]) + ledger.share(EXTENSION_ID, PUBLISHER_ID, PUBLISHER_LEN) + ledger.record(EXTENSION_ID, EXTENSION_CODE, suffix_positions, TAG_MIXED_PREFILL) + ledger.record(MISS_ID, MISS_CODE, miss_positions, TAG_MIXED_PREFILL) + _check_request(manager, EXTENSION_ID, ledger, "after mixed prefill") + _check_request(manager, MISS_ID, ledger, "after mixed prefill") + _check_request(manager, PUBLISHER_ID, ledger, "publisher after mixed prefill") + + # Publisher's first decode writes position 2311 into its own partial page. If that + # page were shared with the extension, the extension's position 2311 would change. + assert manager.try_allocate_generation(publisher) + _prepare(eager, [(publisher, 1, PUBLISHER_LEN)], num_contexts=0) + assert eager.msa_out_cache_loc[:1].tolist() == _expected_slots( + manager, [(PUBLISHER_ID, [PUBLISHER_LEN])] + ) + _write_eager(manager, eager, [(PUBLISHER_CODE, [PUBLISHER_LEN])], TAG_PUBLISHER_DECODE) + _finish_generation(manager, [publisher]) + ledger.record(PUBLISHER_ID, PUBLISHER_CODE, [PUBLISHER_LEN], TAG_PUBLISHER_DECODE) + _check_request(manager, PUBLISHER_ID, ledger, "publisher after its decode") + _check_request(manager, EXTENSION_ID, ledger, "extension after publisher decode") + _check_request(manager, MISS_ID, ledger, "miss after publisher decode") + print( + f"page tables: publisher={publisher_table} extension={extension_table} " + f"miss={miss_table} padding={padding_table}" + ) + + # Checker sensitivity: one corrupted element must be reported at its position + # (page 5, offset 9 -> position 649) and nowhere else. + k_view, _, _ = _layer_views(manager, 17) + probe = (publisher_table[5], 0, 9, 5) + saved = k_view[probe].clone() + k_view[probe] = 0 + with pytest.raises( + AssertionError, match=r"layer 17 role 0: 1 of 2312 positions differ; position 649 " + ): + _check_request(manager, PUBLISHER_ID, ledger, "sensitivity") + k_view[probe] = saved + _check_request(manager, PUBLISHER_ID, ledger, "after sensitivity restore") + + # 4. Two real decode rows under a CUDA graph: capture, replay, reverse the rows. + graph_meta = MiniMaxM3MsaSparseAttentionMetadata( + max_num_requests=GRAPH_BATCH, + max_num_tokens=GRAPH_BATCH, + kv_cache_manager=manager, + mapping=manager.mapping, + is_cuda_graph=True, + ) + writer = _GraphWriter(manager, graph_meta) + + publisher_pos = publisher.max_beam_num_tokens - 1 # 2312 + extension_pos = extension.max_beam_num_tokens - 1 # 2433 + assert (publisher_pos, extension_pos) == (PUBLISHER_LEN + 1, EXTENSION_LEN) + assert manager.try_allocate_generation(publisher) + assert manager.try_allocate_generation(extension) + _prepare( + graph_meta, + [(publisher, 1, publisher_pos), (extension, 1, extension_pos)], + num_contexts=0, + ) + writer.capture() + block_table_ptr = graph_meta.msa_block_table.data_ptr() + assert ( + graph_meta.msa_block_table[0, :19].tolist() == _page_table(manager, PUBLISHER_ID)[:19] + ) + assert ( + graph_meta.msa_block_table[1, :20].tolist() == _page_table(manager, EXTENSION_ID)[:20] + ) + assert graph_meta.msa_out_cache_loc[:GRAPH_BATCH].tolist() == _expected_slots( + manager, [(PUBLISHER_ID, [publisher_pos]), (EXTENSION_ID, [extension_pos])] + ) + writer.replay( + [(PUBLISHER_CODE, [publisher_pos]), (EXTENSION_CODE, [extension_pos])], + TAG_GRAPH_FORWARD, + ) + _finish_generation(manager, [publisher, extension]) + ledger.record(PUBLISHER_ID, PUBLISHER_CODE, [publisher_pos], TAG_GRAPH_FORWARD) + ledger.record(EXTENSION_ID, EXTENSION_CODE, [extension_pos], TAG_GRAPH_FORWARD) + for request_id in (PUBLISHER_ID, EXTENSION_ID, MISS_ID): + _check_request(manager, request_id, ledger, "after graph replay") + + # Reversed row order, new tags: the refreshed staging must route each row's write. + publisher_pos += 1 + extension_pos += 1 + assert manager.try_allocate_generation(extension) + assert manager.try_allocate_generation(publisher) + _prepare( + graph_meta, + [(extension, 1, extension_pos), (publisher, 1, publisher_pos)], + num_contexts=0, + ) + assert graph_meta.msa_block_table.data_ptr() == block_table_ptr + assert ( + graph_meta.msa_block_table[0, :20].tolist() == _page_table(manager, EXTENSION_ID)[:20] + ) + assert ( + graph_meta.msa_block_table[1, :19].tolist() == _page_table(manager, PUBLISHER_ID)[:19] + ) + writer.replay( + [(EXTENSION_CODE, [extension_pos]), (PUBLISHER_CODE, [publisher_pos])], + TAG_GRAPH_REVERSED, + ) + _finish_generation(manager, [extension, publisher]) + ledger.record(PUBLISHER_ID, PUBLISHER_CODE, [publisher_pos], TAG_GRAPH_REVERSED) + ledger.record(EXTENSION_ID, EXTENSION_CODE, [extension_pos], TAG_GRAPH_REVERSED) + for request_id in (PUBLISHER_ID, EXTENSION_ID, MISS_ID): + _check_request(manager, request_id, ledger, "after reversed graph replay") + + # 5. Release the extension; replay with the survivor and the padding row. + extension_table = _page_table(manager, EXTENSION_ID) + released = sorted(set(extension_table) - set(_page_table(manager, PUBLISHER_ID))) + assert released, "the extension must own pages beyond the shared prefix" + counts_before_release = _storage_counts(manager) + manager.free_resources(extension) + assert EXTENSION_ID not in manager.kv_cache_map + counts_after_release = _storage_counts(manager) + print( + f"released pages={released} storage (total, free, evictable, available, " + f"unavailable): initial={initial_counts} before_release={counts_before_release} " + f"after_release={counts_after_release}" + ) + assert sum(c[4] for c in counts_before_release) - sum( + c[4] for c in counts_after_release + ) == len(released), ( + f"unavailable pages before {counts_before_release} after {counts_after_release}" + ) + snapshot = [ + [view[released].clone() for view in _layer_views(manager, layer)] + for layer in range(NUM_LAYERS) + ] + + publisher_pos += 1 + assert manager.try_allocate_generation(publisher) + _prepare(graph_meta, [(publisher, 1, publisher_pos), (padding, 1, 0)], num_contexts=0) + assert graph_meta.msa_out_cache_loc[:GRAPH_BATCH].tolist() == _expected_slots( + manager, [(PUBLISHER_ID, [publisher_pos]), (PADDING_ID, [0])] + ) + writer.replay([(PUBLISHER_CODE, [publisher_pos]), (PADDING_CODE, [0])], TAG_GRAPH_SURVIVOR) + _finish_generation(manager, [publisher]) + ledger.record(PUBLISHER_ID, PUBLISHER_CODE, [publisher_pos], TAG_GRAPH_SURVIVOR) + ledger.record(PADDING_ID, PADDING_CODE, [0], TAG_GRAPH_SURVIVOR) + _check_request(manager, PUBLISHER_ID, ledger, "survivor after release replay") + _check_request(manager, MISS_ID, ledger, "miss after release replay") + _check_request(manager, PADDING_ID, ledger, "padding row after release replay") + for layer in range(NUM_LAYERS): + for role, view in enumerate(_layer_views(manager, layer)): + # Compare bit patterns: BF16 equality would treat NaN bytes as + # changed and +0/-0 as unchanged. + assert torch.equal( + view[released].view(torch.int16), snapshot[layer][role].view(torch.int16) + ), f"released pages {released} changed at layer {layer} role {role}" + + # Free-block accounting returns to its initial value. + for request in (publisher, miss, padding): + manager.free_resources(request) + assert not manager.kv_cache_map.keys() - {k for k in manager.kv_cache_map if k < 0} + assert manager.get_num_free_blocks() == initial_free_blocks + final_counts = _storage_counts(manager) + print( + f"free blocks: initial={initial_free_blocks} final={manager.get_num_free_blocks()} " + f"storage final={final_counts}" + ) + assert [c[4] for c in final_counts] == [c[4] for c in initial_counts], ( + f"unavailable pages: initial {initial_counts}, final {final_counts}" + ) + assert [c[3] for c in final_counts] == [c[3] for c in initial_counts] + manager.impl.clear_reusable_blocks() + assert _storage_counts(manager) == initial_counts + finally: + manager.shutdown() + gc.collect() + torch.cuda.empty_cache() diff --git a/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_sparse_attn_decode.py b/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_sparse_attn_decode.py index 88760a79452f..2a4a94e7f7d5 100644 --- a/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_sparse_attn_decode.py +++ b/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_sparse_attn_decode.py @@ -531,6 +531,66 @@ def run(): torch.testing.assert_close(out.float(), expected, rtol=3e-2, atol=3e-2) +def test_sparse_decode_bf16_cache_zero_length_rows_are_zero_not_nan(): + """CUDA-graph padding rows of a BF16 cache emit zeros, not NaN.""" + seq_lens = [1024, 0, 512, 0] + inputs = _make_inputs(seq_lens, kv_dtype=torch.bfloat16, seed=75) + for num_topk_chunks in (None, 1): + kwargs = {} if num_topk_chunks is None else {"num_topk_chunks": num_topk_chunks} + out = _run(*inputs, 1, **kwargs) + assert torch.isfinite(out).all() + assert torch.equal(out[1], torch.zeros_like(out[1])) + assert torch.equal(out[3], torch.zeros_like(out[3])) + assert out[0].abs().sum() > 0 + + +@pytest.mark.parametrize( + "num_topk_chunks", [None, 1, 4], ids=["default", "direct-output", "split-merge"] +) +def test_sparse_decode_bf16_cache_cuda_graph_replay_equals_eager(num_topk_chunks): + """A captured BF16-cache decode replays bit-identically to eager on new inputs.""" + seq_lens = [174, 4097, 900] + q, k_paged, v_paged, topk_idx, block_table, seq_lens_dev = _make_inputs( + seq_lens, kv_dtype=torch.bfloat16, num_kv_heads=1, group=16, seed=77 + ) + kwargs = {} + if num_topk_chunks is not None: + kwargs["num_topk_chunks"] = num_topk_chunks + out = torch.empty_like(q) + + def run(target): + minimax_m3_sparse_attn_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens_dev, + sm_scale=HEAD_DIM**-0.5, + output=target, + decode_query_len=1, + **kwargs, + ) + + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + run(out) + torch.cuda.current_stream().wait_stream(side) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + run(out) + + generator = torch.Generator(device="cuda").manual_seed(101) + q.copy_(torch.randn(q.shape, device="cuda", generator=generator, dtype=torch.float32)) + graph.replay() + torch.cuda.synchronize() + eager = torch.empty_like(q) + run(eager) + torch.cuda.synchronize() + assert torch.equal(out, eager) + + @skip_not_sm100 @pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float8_e4m3fn], ids=["bf16", "fp8"]) def test_msa_prefill_head_major_selector_cuda_graph(dtype: torch.dtype) -> None: From d63fe42a3933ef2c39253a4a7cc90d39d9d0c77a Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 04/38] [None][feat] Allow NVFP4 KV cache for MiniMax-M3 sparse attention on SM107 The MiniMax-M3 Triton NVFP4 sparse decode kernels run on SM107, so keep NVFP4 KV for that path instead of falling back to FP8 and include SM107 in the decode capability list and test guards. Signed-off-by: Pranav Shrestha --- .../minimax_m3/kernels/triton_sparse_decode.py | 6 +++--- tensorrt_llm/_torch/pyexecutor/model_loader.py | 5 ++++- .../msa/test_minimax_m3_sparse_attn_decode.py | 4 ++-- .../attention/sparse/msa/test_msa_backend.py | 6 +++--- tests/unittest/_torch/test_model_config.py | 14 ++++++++++++++ 5 files changed, 26 insertions(+), 9 deletions(-) diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/triton_sparse_decode.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/triton_sparse_decode.py index 72f3627eeb72..0ac44cb6e05c 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/triton_sparse_decode.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/triton_sparse_decode.py @@ -799,8 +799,8 @@ def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> i return 1 << (target.bit_length() - 1) -# NVFP4 instructions are supported on both datacenter Blackwell targets. -_SM100F_CAPABILITIES = ((10, 0), (10, 3)) +# NVFP4 instructions are supported on the datacenter Blackwell and Rubin targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3), (10, 7)) # Devices already found to support the NVFP4 decode kernels, by capability. @@ -814,7 +814,7 @@ def _nvfp4_device_capability(device: torch.device) -> tuple[int, int]: capability = torch.cuda.get_device_capability(device) if capability not in _SM100F_CAPABILITIES: raise NotImplementedError( - "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103; " + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103/SM107; " f"got SM{capability[0] * 10 + capability[1]}." ) _NVFP4_DEVICE_CAPABILITIES[device] = capability diff --git a/tensorrt_llm/_torch/pyexecutor/model_loader.py b/tensorrt_llm/_torch/pyexecutor/model_loader.py index 9f2f4e12a855..8f30c61376ff 100644 --- a/tensorrt_llm/_torch/pyexecutor/model_loader.py +++ b/tensorrt_llm/_torch/pyexecutor/model_loader.py @@ -167,7 +167,10 @@ def validate_and_set_kv_cache_quant(model_config: ModelConfig, else mapped_pyt_quant) if (effective_kv_cache_quant in (QuantAlgo.NVFP4, QuantAlgo.NVFP4.value) and not supports_fp4_mla_attention(model_config) - and torch.cuda.is_available() and get_sm_version() == 107): + # MiniMax-M3 sparse attention decodes NVFP4 KV with its own kernels. + and getattr(model_config.sparse_attention_config, "algorithm", + None) != "minimax_m3" and + torch.cuda.is_available() and get_sm_version() == 107): logger.warning( "NVFP4 KV cache is not supported by trtllm-gen on SM107; " "using FP8 KV cache instead.") diff --git a/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_sparse_attn_decode.py b/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_sparse_attn_decode.py index 2a4a94e7f7d5..eb0e9dc8d983 100644 --- a/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_sparse_attn_decode.py +++ b/tests/unittest/_torch/attention/sparse/msa/test_minimax_m3_sparse_attn_decode.py @@ -46,8 +46,8 @@ pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required") skip_not_sm100 = pytest.mark.skipif( - get_sm_version() not in (100, 103), - reason="MiniMax-M3 NVFP4 and fmha_sm100 require SM100/SM103.", + get_sm_version() not in (100, 103, 107), + reason="MiniMax-M3 NVFP4 and fmha_sm100 require SM100/SM103/SM107.", ) diff --git a/tests/unittest/_torch/attention/sparse/msa/test_msa_backend.py b/tests/unittest/_torch/attention/sparse/msa/test_msa_backend.py index a006e6e87fcc..aa11f448c72e 100644 --- a/tests/unittest/_torch/attention/sparse/msa/test_msa_backend.py +++ b/tests/unittest/_torch/attention/sparse/msa/test_msa_backend.py @@ -1650,8 +1650,8 @@ def test_mixed_batch_generation_span_matches_the_whole_batch_msa_path(): if not msa_package_available(): pytest.skip("fmha_sm100 (MSA submodule) required") - if get_sm_version() not in (100, 103): - pytest.skip("fmha_sm100 requires SM100/SM103") + if get_sm_version() not in (100, 103, 107): + pytest.skip("fmha_sm100 requires SM100/SM103/SM107") page_size, head_dim = 128, 128 attention, fields, q, kv_block_indexes, num_ctx_tokens = _mixed_batch_sparse_gqa_case( @@ -2076,7 +2076,7 @@ def forward(self, q, k, v, metadata, forward_args=None): def test_nvfp4_scatter_writes_physical_p32_data_and_scale_layouts(): from tensorrt_llm._utils import get_sm_version - if get_sm_version() not in (100, 103): + if get_sm_version() not in (100, 103, 107): pytest.skip("NVFP4 quantization requires Blackwell") torch.manual_seed(13) num_slots, pages_per_role, num_heads = 2, 4, 1 diff --git a/tests/unittest/_torch/test_model_config.py b/tests/unittest/_torch/test_model_config.py index 0ed31615b378..7322b3affe24 100644 --- a/tests/unittest/_torch/test_model_config.py +++ b/tests/unittest/_torch/test_model_config.py @@ -15,6 +15,7 @@ validate_and_set_kv_cache_quant, validate_encoder_decoder_kv_cache_config, ) +from tensorrt_llm.llmapi.llm_args import MiniMaxM3SparseAttentionConfig from tensorrt_llm.mapping import Mapping from tensorrt_llm.models.modeling_utils import QuantAlgo, QuantConfig @@ -159,6 +160,19 @@ def test_validate_and_set_kv_cache_quant_downgrades_nvfp4_on_sm107( assert model_config.quant_config.kv_cache_quant_algo == QuantAlgo.FP8 +def test_validate_and_set_kv_cache_quant_keeps_nvfp4_for_minimax_m3_on_sm107( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """MiniMax-M3 sparse attention has its own SM107 NVFP4 KV decode kernels.""" + _mock_sm107(monkeypatch) + model_config = ModelConfig( + quant_config=QuantConfig(kv_cache_quant_algo=QuantAlgo.FP8), + sparse_attention_config=MiniMaxM3SparseAttentionConfig(), + ) + validate_and_set_kv_cache_quant(model_config, "nvfp4") + assert model_config.quant_config.kv_cache_quant_algo == QuantAlgo.NVFP4 + + def _make_mixed_precision_model_config(): """MIXED_PRECISION checkpoint shape: global config plus per-layer entries whose kv_cache_quant_algo comes only from hf_quant_config.json (None here).""" From 186d513f98d76987b184ef6c39fb7e24771447cf Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 05/38] [None][fix] MiniMax-M3 VL: use the serve-rendered prompt and count video tokens per patch trtllm-serve and the multimodal prompt builders mark their rendered TextPrompt with chat_template_applied; the input processor uses it verbatim instead of wrapping it in a second user turn, normalises a framed marker, and rejects a marker/media count mismatch before the HF processor runs. Media arrive as uint8 numpy, and video tokens count one start/end bracket per temporal patch. Signed-off-by: Pranav Shrestha --- .../_torch/models/modeling_minimaxm3.py | 6 +- .../_torch/models/modeling_minimaxm3_vl.py | 169 +++++++-- tensorrt_llm/inputs/data.py | 9 + tensorrt_llm/inputs/utils.py | 2 + tensorrt_llm/serve/openai_server.py | 7 + .../integration/test_lists/test-db/l0_cpu.yml | 11 + .../_torch/models/test_minimax_m3_vl.py | 345 +++++++++++++++++- 7 files changed, 503 insertions(+), 46 deletions(-) diff --git a/tensorrt_llm/_torch/models/modeling_minimaxm3.py b/tensorrt_llm/_torch/models/modeling_minimaxm3.py index f78d6171d379..be17af7d6435 100644 --- a/tensorrt_llm/_torch/models/modeling_minimaxm3.py +++ b/tensorrt_llm/_torch/models/modeling_minimaxm3.py @@ -3116,8 +3116,10 @@ def _build_minimax_m3_vl_input_processor_registration(): model_type="minimax_m3_vl", placeholder_metadata=MultimodalPlaceholderMetadata( placeholder_map={ - "image": "]<]start of image[>[]<]image[>[]<]end of image[>[", - "video": "]<]start of image[>[]<]video[>[]<]end of image[>[", + # The bare per-item markers the M3 chat template renders; the + # HF processor adds the start/end-of-image framing. + "image": "]<]image[>[", + "video": "]<]video[>[", }, placeholder_placement=MultimodalPlaceholderPlacement.BEFORE_TEXT, content_format=ContentFormat.STRING, diff --git a/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py b/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py index 5e56482ca385..e05b6b53b676 100644 --- a/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py +++ b/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py @@ -16,6 +16,7 @@ from dataclasses import dataclass, field from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence, Tuple +import numpy as np import torch import torch.nn.functional as F from torch import nn @@ -37,6 +38,10 @@ MINIMAX_M3_VL_VISION_START_TOKEN = "]<]start of image[>[" # nosec B105 - vision delimiter token, not a password MINIMAX_M3_VL_VISION_END_TOKEN = "]<]end of image[>[" # nosec B105 - vision delimiter token, not a password +# Per-item markers the chat template renders for ``{"type": "image"}`` / +# ``{"type": "video"}`` content entries; the HF processor expands each one. +MINIMAX_M3_VL_IMAGE_TOKEN = "]<]image[>[" # nosec B105 - image marker token, not a password +MINIMAX_M3_VL_VIDEO_TOKEN = "]<]video[>[" # nosec B105 - video marker token, not a password # --------------------------------------------------------------------------- @@ -1916,6 +1921,10 @@ def get_mm_special_token_ids(self) -> torch.Tensor: def _hw(media: Any) -> Tuple[int, int]: if isinstance(media, torch.Tensor): return int(media.shape[-2]), int(media.shape[-1]) + if isinstance(media, np.ndarray): + # HWC uint8 from ImageMediaIO's "np" format, the format + # ``get_preferred_media_io_kwargs`` asks for. + return int(media.shape[0]), int(media.shape[1]) return int(media.height), int(media.width) def get_num_tokens_per_image(self, *, image: Any, **kwargs: Any) -> int: @@ -1932,23 +1941,56 @@ def get_num_tokens_per_video( video_grid_thw: Optional[Any] = None, **kwargs: Any, ) -> int: - """Token count per video block (+ VS/VE framing). - - ``video_grid_thw`` is populated by the fast-path's dummy-text pass - in :func:`find_mm_token_lengths`, so we don't duplicate the video - resize / temporal-patch math here. + """Token count per video item, framing included. + + The HF processor expands one ``]<]video[>[`` marker into one bracket + per temporal patch, ``VISION_START + VIDEO_TOKEN * (grid_h*grid_w / + merge^2) + VISION_END`` repeated ``grid_t`` times, so the count is + ``grid_t * (grid_h*grid_w / merge^2 + 2)``. The brackets are + adjacent: the processor prefixes a timestamp only when the video + carries fps metadata, and the frame lists handed over here carry + none. + + ``video_grid_thw`` is the processor's grid for this video, + computed by the dummy-text pass of :func:`find_mm_token_lengths` + and passed as a ``[1, 3]`` row; a bare ``(t, h, w)`` triple is + accepted too. """ if video_grid_thw is None: raise RuntimeError( "MiniMaxM3VL fast path: get_num_tokens_per_video requires " "precomputed video_grid_thw from the dummy-text pass." ) - grid = ( - video_grid_thw.tolist() if isinstance(video_grid_thw, torch.Tensor) else video_grid_thw - ) - merge = int(self.processor.image_processor.merge_size) - grid_t, grid_h, grid_w = int(grid[0]), int(grid[1]), int(grid[2]) - return grid_t * (grid_h // merge) * (grid_w // merge) + 2 + grid = torch.as_tensor(video_grid_thw).reshape(-1, 3) + merge = int(self.processor.video_processor.merge_size) + per_patch = (grid[:, 1] * grid[:, 2]) // (merge * merge) + 2 + return int((grid[:, 0] * per_patch).sum().item()) + + def get_preferred_media_io_kwargs(self) -> Dict[str, Dict[str, Any]]: + # uint8 HWC arrays for both modalities. The checkpoint's image and + # video processors rescale by 1/255 themselves (``do_rescale``); + # media_io's default "pt" format delivers float [0, 1] tensors, which + # they rescale a second time, and the vision tower then sees an + # almost black input. trtllm-serve seeds its media_io defaults from + # this method, so no --media_io_kwargs flag is needed; an explicit + # flag still wins. + return {"image": {"format": "np"}, "video": {"format": "np"}} + + @staticmethod + def _video_frames(video: Any) -> List[Any]: + """Frames of one raw video item (``VideoData`` or a frame list).""" + if hasattr(video, "frames"): + return list(video.frames) + return list(video) if isinstance(video, (list, tuple)) else [video] + + def _video_bracket_count(self, video: Any) -> int: + """Temporal patches of a video item: one framed bracket each. + + Mirrors the video processor, which pads the frame count to a + multiple of ``temporal_patch_size`` and emits ``grid_t`` brackets. + """ + temporal = int(self.processor.video_processor.temporal_patch_size) + return -(-len(self._video_frames(video)) // temporal) def get_text_with_mm_placeholders(self, mm_counts: Dict[str, int]) -> str: # Return ``""``: ``__call__``'s chat-template branch already @@ -1970,7 +2012,10 @@ def expand_prompt_token_ids_for_mm( ``video_token_id`` per item with no VISION_START / VISION_END framing; the HF MiniMaxVLProcessor would normally add it on the slow path. We do the same here from ``num_mm_tokens_per_placeholder`` - (which already accounts for the +2 framing). + (which already accounts for the +2 framing per bracket). An image is + one bracket; a video is one bracket per temporal patch, as the + processor lays it out (see :meth:`get_num_tokens_per_video`), so the + placeholder runs line up with the vision rows at merge time. """ mm_data = mm_data or {} has_image = bool(mm_data.get("image")) @@ -1982,6 +2027,7 @@ def expand_prompt_token_ids_for_mm( if not (has_image or has_video): return list(prompt_token_ids), None placeholder_id = self._image_token_id if has_image else self._video_token_id + items = self._coerce_to_list(mm_data.get("video")) if has_video else [] expected = len(num_mm_tokens_per_placeholder) expanded: List[int] = [] consumed = 0 @@ -1994,15 +2040,18 @@ def expand_prompt_token_ids_for_mm( "MiniMaxM3VL fast path: prompt has more placeholders than " f"num_mm_tokens entries ({expected})." ) - inner = int(num_mm_tokens_per_placeholder[consumed]) - 2 - if inner < 1: + brackets = self._video_bracket_count(items[consumed]) if has_video else 1 + total = int(num_mm_tokens_per_placeholder[consumed]) + inner, remainder = divmod(total - 2 * brackets, brackets) + if inner < 1 or remainder: raise ValueError( - f"MiniMaxM3VL fast path: num_mm_tokens[{consumed}]={inner + 2} " - "must be >= 3 (VISION_START + >=1 placeholder + VISION_END)." + f"MiniMaxM3VL fast path: num_mm_tokens[{consumed}]={total} must be " + f"{brackets} x (VISION_START + >=1 placeholder + VISION_END)." ) - expanded.append(self._vision_start_token_id) - expanded.extend([placeholder_id] * inner) - expanded.append(self._vision_end_token_id) + for _ in range(brackets): + expanded.append(self._vision_start_token_id) + expanded.extend([placeholder_id] * inner) + expanded.append(self._vision_end_token_id) consumed += 1 if consumed != expected: raise ValueError( @@ -2061,24 +2110,26 @@ def call_with_text_prompt( # The HF MiniMaxVLProcessor expects video frames as nested lists # of frames (one list per video). Accept either pre-extracted # frames or wrapper objects with a ``frames`` attribute. - videos: List[Any] = [] - for v in video_inputs: - if hasattr(v, "frames"): - videos.append(list(v.frames)) - else: - videos.append(list(v) if isinstance(v, (list, tuple)) else [v]) - - # Apply the M3 chat template before invoking the processor. The - # M3 ``chat_template.jinja`` inserts the canonical - # ``IMAGE_TOKEN`` / ``VIDEO_TOKEN`` markers around each image - # / video item in the user's content list. The - # ``MiniMaxVLProcessor`` then expands those markers in place - # using ``VISION_START_TOKEN + IMAGE_TOKEN*N + VISION_END_TOKEN`` - # before tokenization. Without the chat template, the raw user - # prompt doesn't carry any image marker and the processor - # emits no multimodal token positions in the final input_ids. - templated_text: str - if images or videos: + videos: List[Any] = [self._video_frames(v) for v in video_inputs] + + # The ``MiniMaxVLProcessor`` expands each ``IMAGE_TOKEN`` / + # ``VIDEO_TOKEN`` marker in place into + # ``VISION_START_TOKEN + IMAGE_TOKEN*N + VISION_END_TOKEN`` before + # tokenization. + # + # ``chat_template_applied`` (``TextPrompt``) says the prompt is the + # chat template's output with one registry marker per media item + # already in it; trtllm-serve and the prompt builders in + # ``tensorrt_llm.inputs.utils`` set it. Such a prompt is used + # verbatim. Without it the prompt is raw LLM-API text: it becomes + # the text of one user turn with one ``{"type": "image"}`` / + # ``{"type": "video"}`` entry per item, and the M3 chat template + # inserts the markers. Marker text inside a raw prompt is counted + # on top of the template's markers and rejected by the count check + # below: the markers are control tokens, not literal text. + if inputs.get("chat_template_applied"): + templated_text = text_prompt if isinstance(text_prompt, str) else "" + else: content: List[Dict[str, Any]] = [] if isinstance(text_prompt, str) and text_prompt: content.append({"type": "text", "text": text_prompt}) @@ -2100,17 +2151,57 @@ def call_with_text_prompt( explicit: List[str] = [] if isinstance(text_prompt, str) and text_prompt: explicit.append(text_prompt) - explicit.extend(["]<]image[>["] * len(images)) - explicit.extend(["]<]video[>["] * len(videos)) + explicit.extend([MINIMAX_M3_VL_IMAGE_TOKEN] * len(images)) + explicit.extend([MINIMAX_M3_VL_VIDEO_TOKEN] * len(videos)) templated_text = "\n".join(explicit) + # Callers may frame a marker with the vision start/end tokens + # (``VISION_START_TOKEN + IMAGE_TOKEN + VISION_END_TOKEN``). The + # processor adds exactly that framing when it expands a bare marker, + # so a framed marker can only mean it is framed already: reduce it to + # the bare marker rather than nesting a second frame around it. + # Accepted instead of rejected so a prompt that embeds the framed form + # to position an image keeps working. + for marker in (MINIMAX_M3_VL_IMAGE_TOKEN, MINIMAX_M3_VL_VIDEO_TOKEN): + templated_text = templated_text.replace( + MINIMAX_M3_VL_VISION_START_TOKEN + marker + MINIMAX_M3_VL_VISION_END_TOKEN, + marker, + ) + + # Each marker becomes a placeholder run that the model fills with + # vision-tower rows, so marker and item counts must agree. Reject a + # mismatch here as a request error; at forward time it fails the + # whole batch. + for marker, items in ( + (MINIMAX_M3_VL_IMAGE_TOKEN, images), + (MINIMAX_M3_VL_VIDEO_TOKEN, videos), + ): + if templated_text.count(marker) != len(items): + raise ValueError( + f"MiniMax-M3 VL prompt has {templated_text.count(marker)} {marker!r} " + f"markers but the request carries {len(items)} matching media items. " + "A prompt rendered outside TRT-LLM needs 'chat_template_applied' set; " + "a raw prompt must not contain marker text." + ) + # Run the HF processor. ``return_tensors='pt'`` yields tensors # in the BatchFeature output. + # + # The processor's video default is ``do_resize=False`` (it expects + # frames already resized by the caller, as SGLang does); the decoded + # frames arrive here as is, so let the processor resize them: frame + # sides must be multiples of ``patch_size * merge_size`` for its + # patch view, and its ``max_pixels`` bounds the token count. Images + # are resized by default. + processor_kwargs: Dict[str, Any] = {} + if videos: + processor_kwargs["videos_kwargs"] = {"do_resize": True} processed = self.processor( text=[templated_text], images=images if images else None, videos=videos if videos else None, return_tensors="pt", + **processor_kwargs, ) multimodal_data: Dict[str, Any] = {} diff --git a/tensorrt_llm/inputs/data.py b/tensorrt_llm/inputs/data.py index fe16de2b941b..c28a2b0cbeb2 100644 --- a/tensorrt_llm/inputs/data.py +++ b/tensorrt_llm/inputs/data.py @@ -32,6 +32,15 @@ class TextPrompt(TypedDict): input processor for mm input processing. """ + chat_template_applied: NotRequired[bool] + """ + True when ``prompt`` is the chat template's output with one registered + multimodal placeholder per media item already in it. Set by trtllm-serve + and by the prompt builders in ``tensorrt_llm.inputs.utils``; input + processors that otherwise render a raw ``prompt`` through the model's chat + template use it verbatim when this is set. + """ + class TokensPrompt(TypedDict): """Schema for a tokenized prompt.""" diff --git a/tensorrt_llm/inputs/utils.py b/tensorrt_llm/inputs/utils.py index 4d81366928e0..5d1e503b6d44 100644 --- a/tensorrt_llm/inputs/utils.py +++ b/tensorrt_llm/inputs/utils.py @@ -1002,6 +1002,7 @@ async def async_build_multimodal_prompt( inputs = prompt_inputs(rendered) if mm_data: inputs["multi_modal_data"] = mm_data + inputs["chat_template_applied"] = True item_order = mm_data_tracker.item_order() if item_order: inputs["mm_item_order"] = item_order @@ -1185,6 +1186,7 @@ def convert_to_conversation_message( input = {"prompt": prompt} if mm_placeholder_counts: + input["chat_template_applied"] = True if mm_embeddings is not None: _, input[ "multi_modal_embeddings"] = mm_data_tracker.retrieve_all_sync( diff --git a/tensorrt_llm/serve/openai_server.py b/tensorrt_llm/serve/openai_server.py index b2b83e4a54e4..7b1b5feb59b8 100644 --- a/tensorrt_llm/serve/openai_server.py +++ b/tensorrt_llm/serve/openai_server.py @@ -2270,6 +2270,11 @@ async def chat_stream_generator( mm_data, mm_embeddings = await mm_coroutines if mm_data: prompt["multi_modal_data"] = mm_data + if request.prompt_token_ids is None: + # `prompt` is the rendered chat template with the + # placeholders in place; the input processor must not + # render it again. + prompt["chat_template_applied"] = True if mm_embeddings: prompt["multi_modal_embeddings"] = mm_embeddings if mm_data and mm_embeddings: @@ -2785,6 +2790,8 @@ async def create_mm_embedding_response(promise: RequestOutput): raise ValueError("Cannot use multimodal embeddings as input") if mm_data is not None: prompt["multi_modal_data"] = mm_data + if request.prompt_token_ids is None: + prompt["chat_template_applied"] = True if mm_item_order: prompt["mm_item_order"] = mm_item_order diff --git a/tests/integration/test_lists/test-db/l0_cpu.yml b/tests/integration/test_lists/test-db/l0_cpu.yml index d96d4b294820..21d535999a93 100644 --- a/tests/integration/test_lists/test-db/l0_cpu.yml +++ b/tests/integration/test_lists/test-db/l0_cpu.yml @@ -89,6 +89,17 @@ l0_cpu: - unittest/_torch/models/test_minimax_m3_vl.py::test_text_request_needs_no_hf_processor - unittest/_torch/models/test_minimax_m3_vl.py::test_tokenizer_from_auto_processor_is_not_image_support - unittest/_torch/models/test_minimax_m3_vl.py::test_image_request_uses_a_valid_processor + - unittest/_torch/models/test_minimax_m3_vl.py::test_serve_rendered_image_prompt_is_used_verbatim + - unittest/_torch/models/test_minimax_m3_vl.py::test_serve_rendered_mixed_image_and_video_prompt + - unittest/_torch/models/test_minimax_m3_vl.py::test_framed_marker_is_reduced_to_the_bare_marker + - unittest/_torch/models/test_minimax_m3_vl.py::test_marker_and_media_count_mismatch_is_rejected + - unittest/_torch/models/test_minimax_m3_vl.py::test_raw_prompt_marker_text_is_rejected + - unittest/_torch/models/test_minimax_m3_vl.py::test_openai_chat_marks_the_rendered_prompt_for_the_input_processor + - unittest/_torch/models/test_minimax_m3_vl.py::test_image_token_count_accepts_every_media_io_format + - unittest/_torch/models/test_minimax_m3_vl.py::test_preferred_media_io_format_is_uint8_numpy + - unittest/_torch/models/test_minimax_m3_vl.py::test_video_token_count_is_one_bracket_per_temporal_patch + - unittest/_torch/models/test_minimax_m3_vl.py::test_fast_path_expand_video_one_bracket_per_temporal_patch + - unittest/_torch/models/test_minimax_m3_vl.py::test_video_request_lets_the_processor_resize_frames - unittest/_torch/models/checkpoints - unittest/_torch/modules - unittest/_torch/moe diff --git a/tests/unittest/_torch/models/test_minimax_m3_vl.py b/tests/unittest/_torch/models/test_minimax_m3_vl.py index b3cdffcd1a5e..131d93a23313 100644 --- a/tests/unittest/_torch/models/test_minimax_m3_vl.py +++ b/tests/unittest/_torch/models/test_minimax_m3_vl.py @@ -15,6 +15,9 @@ from __future__ import annotations +import asyncio +import base64 +import io import json import os from types import SimpleNamespace @@ -30,6 +33,8 @@ from tensorrt_llm._torch.models.modeling_minimaxm3 import get_text_config from tensorrt_llm._torch.models.modeling_minimaxm3_vl import ( + MINIMAX_M3_VL_VISION_END_TOKEN, + MINIMAX_M3_VL_VISION_START_TOKEN, CLIPVisionConfig, MiniMaxVLPatchEmbedding, MiniMaxVLPatchMerger, @@ -2037,11 +2042,14 @@ def _make_minimax_m3_vl_fast_path_processor(): obj._video_token_id = _VID_TOK obj._vision_start_token_id = _VS_TOK obj._vision_end_token_id = _VE_TOK - # Fake processor exposing the merge_size that the per-image / per-video - # token-count helpers consult. We don't need the real image_processor - # here because the tests below feed precomputed image_grid_thw values. - fake_image_processor = SimpleNamespace(merge_size=2) - obj._processor = SimpleNamespace(image_processor=fake_image_processor) + # Fake processor exposing the merge_size / temporal_patch_size that the + # per-image / per-video token-count and bracket helpers consult. We don't + # need the real processors here because the tests below feed precomputed + # grid values. + obj._processor = SimpleNamespace( + image_processor=SimpleNamespace(merge_size=2), + video_processor=SimpleNamespace(merge_size=2, temporal_patch_size=2), + ) return obj @@ -2135,6 +2143,69 @@ def test_fast_path_expand_video_placeholders_use_video_token_id(): assert expanded.count(vid) == 2 +def test_fast_path_expand_video_one_bracket_per_temporal_patch(): + """A video spans ``grid_t`` framed brackets, as the HF processor lays it out. + + Three frames pad to two temporal patches; a count of 8 is two brackets of + VISION_START + 2 placeholders + VISION_END. + """ + proc = _make_minimax_m3_vl_fast_path_processor() + frames = [object()] * 3 + expanded, _ = proc.expand_prompt_token_ids_for_mm( + [100, _VID_TOK, 200], + num_mm_tokens_per_placeholder=[8], + mm_data={"video": [SimpleNamespace(frames=frames)]}, + ) + bracket = [_VS_TOK, _VID_TOK, _VID_TOK, _VE_TOK] + assert expanded == [100, *bracket, *bracket, 200] + # A plain frame list works like the ``frames`` wrapper. + expanded_list, _ = proc.expand_prompt_token_ids_for_mm( + [100, _VID_TOK, 200], num_mm_tokens_per_placeholder=[8], mm_data={"video": [frames]} + ) + assert expanded_list == expanded + # A count that does not split into whole brackets is rejected. + with pytest.raises(ValueError, match="2 x \\(VISION_START"): + proc.expand_prompt_token_ids_for_mm( + [100, _VID_TOK, 200], num_mm_tokens_per_placeholder=[7], mm_data={"video": [frames]} + ) + + +def test_video_token_count_is_one_bracket_per_temporal_patch(): + """``grid_t * (grid_h*grid_w/merge^2 + 2)`` for the processor's grid. + + ``find_mm_token_lengths`` passes the grid as a ``[1, 3]`` row; a 1-D + tensor and a plain triple are accepted too. 10 frames at 224x224 give + grid [5, 16, 16]: 5 brackets of 64 video tokens plus framing, 330. + """ + proc = _make_minimax_m3_vl_fast_path_processor() + assert ( + proc.get_num_tokens_per_video(video=[], video_grid_thw=torch.tensor([[5, 16, 16]])) == 330 + ) + assert proc.get_num_tokens_per_video(video=[], video_grid_thw=torch.tensor([5, 16, 16])) == 330 + assert proc.get_num_tokens_per_video(video=[], video_grid_thw=(5, 16, 16)) == 330 + assert proc.get_num_tokens_per_video(video=[], video_grid_thw=[[1, 4, 6]]) == 8 + with pytest.raises(RuntimeError, match="requires precomputed video_grid_thw"): + proc.get_num_tokens_per_video(video=[]) + + +def test_preferred_media_io_format_is_uint8_numpy(): + """The registered processor asks media_io for uint8 HWC arrays. + + The HF processors rescale by 1/255 themselves, so float tensors (the + media_io default) would be rescaled twice; ``OpenAIServer`` seeds its + media_io defaults from this method for ``BaseMultimodalInputProcessor`` + instances. + """ + from tensorrt_llm.inputs.registry import BaseMultimodalInputProcessor + + proc = _make_minimax_m3_vl_fast_path_processor() + assert isinstance(proc, BaseMultimodalInputProcessor) + assert proc.get_preferred_media_io_kwargs() == { + "image": {"format": "np"}, + "video": {"format": "np"}, + } + + def test_fast_path_expand_rejects_mixed_image_and_video(): """Single-modality only: the registry's ``_get_single_mm_token_lengths`` returns only one modality's counts, so mixing image + video in one @@ -2197,6 +2268,12 @@ def test_fast_path_hooks_are_registered_on_processor_class(): assert hasattr(cls, "get_num_tokens_per_video") +# --------------------------------------------------------------------------- +# Input contract without an image processor: text needs only the tokenizer; +# image requests load the HF processor on first use and reject a tokenizer. +# --------------------------------------------------------------------------- + + class _WordTokenizer: """Tokenizer stand-in: one id per whitespace-separated word, BOS id 1.""" @@ -2266,6 +2343,34 @@ def __call__(self, text, images=None, videos=None, return_tensors=None, **kwargs return out +def _serve_style_prompt(tokenizer, text: str, media: Dict[str, list]) -> dict: + """Build the request the way trtllm-serve does for a chat completion. + + Same steps as `parse_chat_messages_coroutines` + `apply_chat_template` + (STRING content format) + the prompt assembly in `openai_server.py`: + the registry supplies the placeholder per item, `apply_mm_placeholders` + writes it into the message text, the tokenizer renders the chat + template, and the rendered text is handed over with the media, + `mm_item_order` and `chat_template_applied`. + """ + from tensorrt_llm.inputs.utils import MultimodalDataTracker, apply_mm_placeholders + + tracker = MultimodalDataTracker("minimax_m3_vl") + for modality, items in media.items(): + for item in items: + tracker.add_data(modality, item) + message = {"role": "user", "content": text, "media": [], "content_parts": []} + apply_mm_placeholders("minimax_m3_vl", message, tracker.placeholder_counts(), tracker) + rendered = tokenizer.apply_chat_template([message], tokenize=False, add_generation_prompt=True) + mm_data, _ = tracker.retrieve_all_sync() + return { + "prompt": rendered, + "multi_modal_data": mm_data, + "mm_item_order": tracker.item_order(), + "chat_template_applied": True, + } + + def _tokenizer_only_processor(monkeypatch, auto_processor_result): """Construct the input processor while recording AutoProcessor loads.""" loads = [] @@ -2337,3 +2442,233 @@ def test_image_request_uses_a_valid_processor(monkeypatch): {"prompt": "again", "multi_modal_data": {"image": [image]}}, SimpleNamespace() ) assert len(loads) == 1 and proc.processor is valid + + +@pytest.mark.cpu_only +def test_serve_rendered_image_prompt_is_used_verbatim(monkeypatch): + valid = _ValidImageProcessor() + proc, _ = _tokenizer_only_processor(monkeypatch, valid) + request = _serve_style_prompt( + proc.tokenizer, "describe", {"image": [Image.new("RGB", (32, 32))]} + ) + rendered = request["prompt"] + # The registry placeholder is the bare marker: the HF processor adds the + # start/end framing, so the rendered prompt must not carry it. + assert rendered == "user: ]<]image[>[\ndescribe assistant:" + assert MINIMAX_M3_VL_VISION_START_TOKEN not in rendered + assert MINIMAX_M3_VL_VISION_END_TOKEN not in rendered + assert proc.tokenizer.template_calls == 1 + + ids, extra = proc.call_with_text_prompt(request, SimpleNamespace()) + # Not rendered a second time; the processor sees the serve output as is. + assert proc.tokenizer.template_calls == 1 + assert valid.calls == [[rendered]] + assert ids == [7, _VS_TOK, _IMG_TOK, _VE_TOK] + assert extra["multimodal_data"]["image"]["image_grid_thw"].tolist() == [[1, 2, 2]] + + +@pytest.mark.cpu_only +def test_serve_rendered_mixed_image_and_video_prompt(monkeypatch): + valid = _ValidImageProcessor() + proc, _ = _tokenizer_only_processor(monkeypatch, valid) + video = SimpleNamespace(frames=[Image.new("RGB", (32, 32))] * 2) + request = _serve_style_prompt( + proc.tokenizer, "compare", {"image": [Image.new("RGB", (32, 32))], "video": [video]} + ) + rendered = request["prompt"] + assert rendered == "user: ]<]image[>[\n]<]video[>[\ncompare assistant:" + + ids, extra = proc.call_with_text_prompt(request, SimpleNamespace()) + assert proc.tokenizer.template_calls == 1 + assert valid.calls == [[rendered]] + assert ids == [7, _VS_TOK, _IMG_TOK, _VE_TOK, _VS_TOK, _VID_TOK, _VE_TOK] + assert extra["multimodal_data"]["image"]["image_grid_thw"].tolist() == [[1, 2, 2]] + assert extra["multimodal_data"]["video"]["video_grid_thw"].tolist() == [[1, 2, 2]] + + +@pytest.mark.cpu_only +def test_framed_marker_is_reduced_to_the_bare_marker(monkeypatch): + # A caller that frames the marker itself gets one frame, from the processor. + valid = _ValidImageProcessor() + proc, _ = _tokenizer_only_processor(monkeypatch, valid) + framed = MINIMAX_M3_VL_VISION_START_TOKEN + "]<]image[>[" + MINIMAX_M3_VL_VISION_END_TOKEN + ids, _ = proc.call_with_text_prompt( + { + "prompt": f"user: {framed}\ndescribe assistant:", + "multi_modal_data": {"image": [Image.new("RGB", (32, 32))]}, + "chat_template_applied": True, + }, + SimpleNamespace(), + ) + assert valid.calls == [["user: ]<]image[>[\ndescribe assistant:"]] + assert ids == [7, _VS_TOK, _IMG_TOK, _VE_TOK] + + +@pytest.mark.cpu_only +def test_marker_and_media_count_mismatch_is_rejected(monkeypatch): + valid = _ValidImageProcessor() + proc, _ = _tokenizer_only_processor(monkeypatch, valid) + image = Image.new("RGB", (32, 32)) + with pytest.raises(ValueError, match="2 ']<]image\\[>\\[' markers but the request carries 1"): + proc.call_with_text_prompt( + { + "prompt": "]<]image[>[ ]<]image[>[ compare", + "multi_modal_data": {"image": [image]}, + "chat_template_applied": True, + }, + SimpleNamespace(), + ) + assert valid.calls == [] + + +@pytest.mark.cpu_only +def test_raw_prompt_marker_text_is_rejected(monkeypatch): + # Without chat_template_applied the template inserts the markers, so + # marker text in the raw prompt is one marker too many. + valid = _ValidImageProcessor() + proc, _ = _tokenizer_only_processor(monkeypatch, valid) + with pytest.raises(ValueError, match="2 ']<]image\\[>\\[' markers but the request carries 1"): + proc.call_with_text_prompt( + { + "prompt": "Explain ]<]image[>[", + "multi_modal_data": {"image": [Image.new("RGB", (32, 32))]}, + }, + SimpleNamespace(), + ) + assert proc.tokenizer.template_calls == 1 + assert valid.calls == [] + # With no media the prompt is plain text and is tokenized as written. + ids, extra = proc.call_with_text_prompt({"prompt": "Explain ]<]image[>["}, SimpleNamespace()) + assert ids == [1, 100, 101] and extra is None + assert proc._processor is None + + +@pytest.mark.cpu_only +def test_image_token_count_accepts_every_media_io_format(monkeypatch): + """``get_num_tokens_per_image`` reads the size of PIL, CHW tensor and HWC array input. + + The hashing path (``find_mm_token_lengths``) hands over whatever + media_io decoded; with the preferred "np" format that is a uint8 HWC + array. 56x84 is a 4x6 patch grid: 24 patches / 4 + 2 framing = 8. + """ + proc, _ = _tokenizer_only_processor(monkeypatch, _ValidImageProcessor()) + pil = Image.new("RGB", (84, 56)) + tensor = torch.zeros(3, 56, 84) + array = np.zeros((56, 84, 3), dtype=np.uint8) + assert [proc.get_num_tokens_per_image(image=m) for m in (pil, tensor, array)] == [8, 8, 8] + + +@pytest.mark.cpu_only +def test_video_request_lets_the_processor_resize_frames(monkeypatch): + """Videos go to the HF processor with ``videos_kwargs={"do_resize": True}``. + + The processor's video default is ``do_resize=False`` and its patch + view needs frame sides that are multiples of 28; the decoded frames + are not pre-resized here. Image-only requests pass no such kwargs. + """ + valid = _ValidImageProcessor() + proc, _ = _tokenizer_only_processor(monkeypatch, valid) + frames = [np.zeros((230, 230, 3), dtype=np.uint8)] * 3 + proc.call_with_text_prompt( + {"prompt": "watch", "multi_modal_data": {"video": [SimpleNamespace(frames=frames)]}}, + SimpleNamespace(), + ) + proc.call_with_text_prompt( + {"prompt": "look", "multi_modal_data": {"image": [Image.new("RGB", (32, 32))]}}, + SimpleNamespace(), + ) + assert valid.call_kwargs == [{"videos_kwargs": {"do_resize": True}}, {}] + + +@pytest.mark.cpu_only +def test_openai_chat_marks_the_rendered_prompt_for_the_input_processor(monkeypatch): + """The prompt ``openai_chat`` assembles is accepted by the input processor as rendered. + + Runs ``OpenAIServer.openai_chat`` on a one-image chat request with the + generator replaced by a recorder, then feeds the captured ``TextPrompt`` + to the M3 input processor. The server must set ``chat_template_applied``: + without it the processor renders the chat template a second time and + rejects the request for having one marker more than images. + """ + from tensorrt_llm.serve.openai_protocol import ChatCompletionRequest + from tensorrt_llm.serve.openai_server import OpenAIServer + + valid = _ValidImageProcessor() + proc, _ = _tokenizer_only_processor(monkeypatch, valid) + tokenizer = proc.tokenizer + # The server reads the vocabulary size off the wrapped HF tokenizer. + tokenizer.tokenizer = tokenizer + + class _M3VLConfig: + model_type = "minimax_m3_vl" + vocab_size = 200_064 + + captured = {} + + def generate_async(**kwargs): + captured.update(kwargs) + return SimpleNamespace(prompt_token_ids=[7, _VS_TOK, _IMG_TOK, _VE_TOK], finished=True) + + async def create_chat_response(*args, **kwargs): + return SimpleNamespace(model_dump=lambda: {"captured": True}) + + server = object.__new__(OpenAIServer) + server.generator = SimpleNamespace( + args=SimpleNamespace( + reasoning_parser=None, + backend="pytorch", + gather_generation_logits=False, + guided_decoding_backend=None, + num_postprocess_workers=0, + cache_transceiver_config=None, + ), + generate_async=generate_async, + ) + server.model_config = _M3VLConfig() + server.tokenizer = tokenizer + server.processor = None + server.chat_template = None + server.tool_parser = None + server.tool_call_id_type = "random" + server.multimodal_server_config = None + server.allow_request_chat_template = False + server._create_chat_response = create_chat_response + + png = io.BytesIO() + Image.new("RGB", (32, 32), "red").save(png, format="PNG") + request = ChatCompletionRequest( + model="minimax-m3", + messages=[ + { + "role": "user", + "content": [ + {"type": "text", "text": "describe"}, + { + "type": "image_url", + "image_url": { + "url": "data:image/png;base64," + + base64.b64encode(png.getvalue()).decode() + }, + }, + ], + } + ], + ) + response = asyncio.run(server.openai_chat(request, None)) + assert response.status_code == 200, response.body + + prompt = captured["inputs"] + assert prompt["prompt"] == "user: ]<]image[>[\ndescribe assistant:" + assert prompt["chat_template_applied"] is True + assert prompt["mm_item_order"] == [ + {"modality": "image", "index": 0, "placeholder": "]<]image[>["} + ] + assert len(prompt["multi_modal_data"]["image"]) == 1 + assert tokenizer.template_calls == 1 + + ids, extra = proc.call_with_text_prompt(prompt, SimpleNamespace()) + # The server's rendering is the only one; the processor saw it verbatim. + assert tokenizer.template_calls == 1 + assert valid.calls == [[prompt["prompt"]]] + assert ids == [7, _VS_TOK, _IMG_TOK, _VE_TOK] + assert extra["multimodal_data"]["image"]["image_grid_thw"].tolist() == [[1, 2, 2]] From 50cee82c76c511bc56a7e73387d52e16acbc6b9d Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 06/38] [None][fix] MiniMax-M3 VL: keep speculative decoding and KV reuse working on multimodal requests Forward the pre-fusion token ids as orig_input_ids so the spec worker (DSpark/DFlash, MTP, Eagle) sees them after fuse_input_embeds drops input_ids, and slice the vision embeddings with find_input_mm_embeds so a cached prefix or a chunk boundary inside a placeholder span no longer trips the token-count check. Signed-off-by: Pranav Shrestha --- .../_torch/models/modeling_minimaxm3.py | 35 ++- .../integration/test_lists/test-db/l0_cpu.yml | 8 + .../models/test_minimax_m3_vl_spec_reuse.py | 281 ++++++++++++++++++ 3 files changed, 314 insertions(+), 10 deletions(-) create mode 100644 tests/unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py diff --git a/tensorrt_llm/_torch/models/modeling_minimaxm3.py b/tensorrt_llm/_torch/models/modeling_minimaxm3.py index be17af7d6435..dfe772f4dc04 100644 --- a/tensorrt_llm/_torch/models/modeling_minimaxm3.py +++ b/tensorrt_llm/_torch/models/modeling_minimaxm3.py @@ -3318,7 +3318,7 @@ def forward( # ``input_ids`` for ``mm_token_ids`` — exactly the right # semantics for visual-input requests. from .modeling_minimaxm3_vl import extract_multimodal_items_in_request_order - from .modeling_multimodal_utils import fuse_input_embeds + from .modeling_multimodal_utils import find_input_mm_embeds, fuse_input_embeds # Walk ``multimodal_params`` in left-to-right request order # and produce ``mm_embeds`` per-item so the concatenated rows @@ -3343,15 +3343,30 @@ def forward( # device for the fuse path (registry buffer device follows # ``model.to(device)`` automatically, but the standalone # unit-test path may use a different device). - fused_ids, fused_embeds = fuse_input_embeds( - embedding_layer=embed_tokens, - input_ids=flat_ids, - mm_embeds=mm_embeds, - mm_token_ids=self.mm_token_ids.to(flat_ids.device), - **kwargs, - ) - input_ids = fused_ids - inputs_embeds = fused_embeds + # With KV-cache reuse or chunked prefill, part of a request's + # placeholder span may already be cached or lie beyond the + # current chunk; keep only the embedding rows whose tokens are in + # this forward (``multimodal_runtime`` carries the counts). The + # list is pre-concatenated in request order so one cursor covers + # multi-image requests. + mm_embeds = find_input_mm_embeds([torch.cat(mm_embeds, dim=0)], mm_params_with_data) + if mm_embeds: + fused_ids, fused_embeds = fuse_input_embeds( + embedding_layer=embed_tokens, + input_ids=flat_ids, + mm_embeds=mm_embeds, + mm_token_ids=self.mm_token_ids.to(flat_ids.device), + **kwargs, + ) + input_ids = fused_ids + inputs_embeds = fused_embeds + # ``fuse_input_embeds`` returns ``input_ids=None`` once the + # embeddings are materialised. The speculative-decoding + # workers (DSpark/DFlash, MTP, Eagle) still need the + # per-position token ids of the fused batch; + # ``SpecDecOneEngineForCausalLM`` reads them from + # ``orig_input_ids`` when ``input_ids`` is None. + kwargs["orig_input_ids"] = flat_ids return super().forward( attn_metadata=attn_metadata, diff --git a/tests/integration/test_lists/test-db/l0_cpu.yml b/tests/integration/test_lists/test-db/l0_cpu.yml index 21d535999a93..fd01805c4ea7 100644 --- a/tests/integration/test_lists/test-db/l0_cpu.yml +++ b/tests/integration/test_lists/test-db/l0_cpu.yml @@ -100,6 +100,14 @@ l0_cpu: - unittest/_torch/models/test_minimax_m3_vl.py::test_video_token_count_is_one_bracket_per_temporal_patch - unittest/_torch/models/test_minimax_m3_vl.py::test_fast_path_expand_video_one_bracket_per_temporal_patch - unittest/_torch/models/test_minimax_m3_vl.py::test_video_request_lets_the_processor_resize_frames + # MiniMax-M3 VL forward under speculative decoding, KV reuse and chunked prefill. + - unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py::test_speculative_multimodal_prefill_forwards_orig_input_ids + - unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py::test_full_image_prefix_reuse_hit_skips_fusion + - unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py::test_partial_image_prefix_reuse_slices_the_uncached_rows + - unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py::test_chunk_boundary_inside_image_span_keeps_only_the_chunk_rows + - unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py::test_batched_requests_advance_the_row_cursor_per_request + - unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py::test_two_images_in_one_request_fuse_in_prompt_order + - unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py::test_text_only_batch_passes_through - unittest/_torch/models/checkpoints - unittest/_torch/modules - unittest/_torch/moe diff --git a/tests/unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py b/tests/unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py new file mode 100644 index 000000000000..9a9073a40921 --- /dev/null +++ b/tests/unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py @@ -0,0 +1,281 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""CPU tests for the MiniMax-M3 VL forward under speculative decoding and KV reuse. + +``MiniMaxM3VLForConditionalGeneration.forward`` owes the executor and the +speculative-decoding workers this contract: + +* a plain multimodal prefill fuses the vision rows into ``inputs_embeds`` and + forwards the pre-fusion token ids as ``orig_input_ids`` (``input_ids`` is None + after fusion, and ``SpecDecOneEngineForCausalLM.forward`` reads + ``orig_input_ids`` for the DSpark/DFlash, MTP and Eagle workers); +* a full image-prefix KV-reuse hit (``multimodal_runtime`` says every embedding + row is cached) skips fusion: ``input_ids`` passes through unchanged, + ``inputs_embeds`` stays None and no ``orig_input_ids`` is set; +* a partial reuse hit or a chunk boundary inside the placeholder span keeps + exactly the rows ``find_input_mm_embeds`` selects + (``[num_cached : num_cached + in_chunk]``) and scatters them at the + placeholder positions of the current chunk; +* a batch of two requests with one image each, the second partially cached, + advances the pre-concatenated row cursor by ``total_embeds_in_request``; +* one request carrying two images fuses both in prompt order. + +The model is built with ``object.__new__`` plus ``nn.Module.__init__`` so no +checkpoint, config or CUDA is needed; the vision tower is a fake that maps each +``merge_size**2`` block of ``pixel_values`` rows to one embedding row whose value +is the block's tag, so a wrong slice or offset shows up as a wrong value. +""" + +from __future__ import annotations + +from typing import Any, Dict, List, Optional, Tuple + +import pytest +import torch +from torch import nn + +from tensorrt_llm._torch.models import modeling_speculative +from tensorrt_llm._torch.models.modeling_minimaxm3 import MiniMaxM3VLForConditionalGeneration +from tensorrt_llm.inputs.multimodal import MultimodalParams, MultimodalRuntimeData + +pytestmark = pytest.mark.cpu_only + +_VOCAB = 64 +_HIDDEN = 8 +_IMG = 50 # image placeholder id (in-vocab, as in the checkpoint) +_VID = 51 # video placeholder id +_TXT = 7 # an ordinary text token id +_MERGE = 2 # fake tower: merge_size**2 = 4 pixel rows per embedding row +_PATCH_FEATURES = 6 + +_VLModel = Tuple[MiniMaxM3VLForConditionalGeneration, Dict[str, Any]] + + +class _FakeVisionTower(nn.Module): + """Returns one row per ``merge**2`` pixel rows, valued by the block's tag.""" + + def __init__(self) -> None: + super().__init__() + self.dtype = torch.float32 + self.calls: List[int] = [] + + def forward(self, pixel_values: torch.Tensor, grid_thw: Any) -> torch.Tensor: + tags = pixel_values[:: _MERGE * _MERGE, 0] + self.calls.append(int(tags.numel())) + return tags.unsqueeze(1).expand(-1, _HIDDEN).to(torch.float32).clone() + + +def _pixel_values(tags: List[float]) -> torch.Tensor: + """``len(tags) * merge**2`` pixel rows; every row of a block carries its tag.""" + t = torch.tensor(tags, dtype=torch.float32).repeat_interleave(_MERGE * _MERGE) + return t.unsqueeze(1).expand(-1, _PATCH_FEATURES).clone() + + +def _grid(n_tokens: int) -> torch.Tensor: + """``[1, 3]`` grid: t=1, h*w = ``n_tokens * merge**2`` patches.""" + return torch.tensor([[1, _MERGE, n_tokens * _MERGE]], dtype=torch.int64) + + +def _runtime(full_ids: List[int], past_seen: int, chunk_end: int) -> MultimodalRuntimeData: + mask = torch.tensor([1 if t in (_IMG, _VID) else 0 for t in full_ids], dtype=torch.int64) + return MultimodalRuntimeData( + past_seen_token_num=past_seen, + chunk_end_pos=chunk_end, + embed_mask_cumsum=torch.cumsum(mask, dim=0), + ) + + +def _params( + image_tags: List[List[float]], + runtime: Optional[MultimodalRuntimeData] = None, +) -> MultimodalParams: + """One request carrying ``len(image_tags)`` images (rows concatenated in order).""" + pv = torch.cat([_pixel_values(tags) for tags in image_tags], dim=0) + grid = torch.cat([_grid(len(tags)) for tags in image_tags], dim=0) + return MultimodalParams( + multimodal_data={"image": {"pixel_values": pv, "image_grid_thw": grid}}, + multimodal_runtime=runtime, + ) + + +@pytest.fixture +def vl_model(monkeypatch: pytest.MonkeyPatch) -> _VLModel: + """A VL wrapper with fake tower/embedding and a captured text forward.""" + model = object.__new__(MiniMaxM3VLForConditionalGeneration) + nn.Module.__init__(model) + embed = nn.Embedding(_VOCAB, _HIDDEN) + with torch.no_grad(): + # Text embedding of token t is the constant -(t + 1): distinct from any + # (non-negative) vision tag. + embed.weight.copy_( + -(torch.arange(_VOCAB, dtype=torch.float32) + 1).unsqueeze(1).expand(-1, _HIDDEN) + ) + model.model = nn.Module() + model.model.embed_tokens = embed + model.vision_tower = _FakeVisionTower() + model.register_buffer( + "mm_token_ids", torch.tensor([_IMG, _VID], dtype=torch.int32), persistent=False + ) + + captured: Dict[str, Any] = {} + + def fake_text_forward( + self: Any, + attn_metadata: Any = None, + input_ids: Optional[torch.Tensor] = None, + position_ids: Optional[torch.Tensor] = None, + inputs_embeds: Optional[torch.Tensor] = None, + return_context_logits: bool = False, + **kwargs: Any, + ) -> torch.Tensor: + captured.update( + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + kwargs=kwargs, + ) + return torch.zeros(1) + + # ``MiniMaxM3ForCausalLM`` does not override ``forward``; the wrapper's + # ``super().forward`` resolves to ``SpecDecOneEngineForCausalLM.forward``. + monkeypatch.setattr( + modeling_speculative.SpecDecOneEngineForCausalLM, "forward", fake_text_forward + ) + return model, captured + + +def _run( + model: MiniMaxM3VLForConditionalGeneration, + ids: List[int], + params: List[Optional[MultimodalParams]], + **extra: Any, +) -> torch.Tensor: + input_ids = torch.tensor(ids, dtype=torch.int32) + position_ids = torch.arange(len(ids), dtype=torch.int32).unsqueeze(0) + model.forward( + attn_metadata=object(), + input_ids=input_ids, + position_ids=position_ids, + multimodal_params=params, + **extra, + ) + return input_ids + + +def _fused_rows(captured: Dict[str, Any], ids: List[int]) -> List[float]: + """First feature of every row of the fused embeddings (one value per position).""" + embeds = captured["inputs_embeds"] + assert embeds is not None and embeds.shape == (len(ids), _HIDDEN) + return embeds[:, 0].tolist() + + +def test_speculative_multimodal_prefill_forwards_orig_input_ids(vl_model: _VLModel) -> None: + """Fusion drops ``input_ids``; the pre-fusion ids reach the spec worker as ``orig_input_ids``.""" + model, captured = vl_model + ids = [_TXT, _TXT, _IMG, _IMG, _IMG, _IMG, _TXT] + spec_metadata = object() + input_ids = _run(model, ids, [_params([[10.0, 11.0, 12.0, 13.0]])], spec_metadata=spec_metadata) + + assert captured["input_ids"] is None + assert torch.equal(captured["kwargs"]["orig_input_ids"], input_ids.flatten()) + assert captured["kwargs"]["spec_metadata"] is spec_metadata + assert "multimodal_params" not in captured["kwargs"] + assert _fused_rows(captured, ids) == [-8.0, -8.0, 10.0, 11.0, 12.0, 13.0, -8.0] + + +def test_full_image_prefix_reuse_hit_skips_fusion(vl_model: _VLModel) -> None: + """Every placeholder row is cached: no fusion, ``input_ids`` passes through unchanged.""" + model, captured = vl_model + full = [_TXT, _IMG, _IMG, _IMG, _IMG, _TXT, _TXT] + # The cached prefix covers the whole image span; this forward computes the tail only. + runtime = _runtime(full, past_seen=5, chunk_end=len(full)) + assert runtime.num_cached_mm_tokens == 4 and runtime.num_mm_tokens_in_chunk == 0 + chunk_ids = full[5:] + input_ids = _run(model, chunk_ids, [_params([[1.0, 2.0, 3.0, 4.0]], runtime)]) + + assert captured["input_ids"] is not None + assert torch.equal(captured["input_ids"], input_ids) + assert captured["inputs_embeds"] is None + assert "orig_input_ids" not in captured["kwargs"] + + +def test_partial_image_prefix_reuse_slices_the_uncached_rows(vl_model: _VLModel) -> None: + """Two of four rows cached: rows ``[2:4]`` land on the two placeholders left in this forward.""" + model, captured = vl_model + full = [_TXT, _IMG, _IMG, _IMG, _IMG, _TXT] + runtime = _runtime(full, past_seen=3, chunk_end=len(full)) + assert (runtime.num_cached_mm_tokens, runtime.num_mm_tokens_in_chunk) == (2, 2) + chunk_ids = full[3:] + input_ids = _run(model, chunk_ids, [_params([[1.0, 2.0, 3.0, 4.0]], runtime)]) + + assert captured["input_ids"] is None + assert torch.equal(captured["kwargs"]["orig_input_ids"], input_ids) + assert _fused_rows(captured, chunk_ids) == [3.0, 4.0, -8.0] + + +def test_chunk_boundary_inside_image_span_keeps_only_the_chunk_rows(vl_model: _VLModel) -> None: + """Chunked prefill: the chunk ends inside the span; cached rows and rows beyond the chunk are dropped.""" + model, captured = vl_model + full = [_TXT, _IMG, _IMG, _IMG, _IMG, _IMG, _IMG, _TXT] + runtime = _runtime(full, past_seen=2, chunk_end=5) + assert (runtime.num_cached_mm_tokens, runtime.num_mm_tokens_in_chunk) == (1, 3) + chunk_ids = full[2:5] + _run(model, chunk_ids, [_params([[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]], runtime)]) + + assert _fused_rows(captured, chunk_ids) == [2.0, 3.0, 4.0] + + +def test_batched_requests_advance_the_row_cursor_per_request(vl_model: _VLModel) -> None: + """Two requests, one image each; the second has two cached rows. Rows are taken per request.""" + model, captured = vl_model + full_a = [_TXT, _IMG, _IMG, _IMG, _IMG] + full_b = [_TXT, _IMG, _IMG, _IMG, _IMG, _IMG, _IMG, _TXT] + runtime_a = _runtime(full_a, past_seen=0, chunk_end=len(full_a)) + runtime_b = _runtime(full_b, past_seen=3, chunk_end=len(full_b)) + ids = full_a + full_b[3:] + params: List[Optional[MultimodalParams]] = [ + _params([[10.0, 11.0, 12.0, 13.0]], runtime_a), + _params([[20.0, 21.0, 22.0, 23.0, 24.0, 25.0]], runtime_b), + ] + input_ids = _run(model, ids, params) + + assert torch.equal(captured["kwargs"]["orig_input_ids"], input_ids) + assert _fused_rows(captured, ids) == [ + -8.0, 10.0, 11.0, 12.0, 13.0, # request A, nothing cached + 22.0, 23.0, 24.0, 25.0, -8.0, # request B, rows [2:6] + ] # fmt: skip + + +def test_two_images_in_one_request_fuse_in_prompt_order(vl_model: _VLModel) -> None: + """One request with two images (4 + 2 tokens) and no runtime data fuses both in order.""" + model, captured = vl_model + ids = [_TXT, _IMG, _IMG, _IMG, _IMG, _TXT, _IMG, _IMG, _TXT] + input_ids = _run(model, ids, [_params([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0]])]) + + assert torch.equal(captured["kwargs"]["orig_input_ids"], input_ids) + assert _fused_rows(captured, ids) == [-8.0, 1.0, 2.0, 3.0, 4.0, -8.0, 5.0, 6.0, -8.0] + + +def test_text_only_batch_passes_through(vl_model: _VLModel) -> None: + """No vision data: ``input_ids`` and ``multimodal_params`` are untouched, no ``orig_input_ids``.""" + model, captured = vl_model + ids = [_TXT, _TXT, _TXT] + text_only = MultimodalParams(multimodal_data={}) + input_ids = _run(model, ids, [text_only, None]) + + assert torch.equal(captured["input_ids"], input_ids) + assert captured["inputs_embeds"] is None + assert "orig_input_ids" not in captured["kwargs"] + assert model.vision_tower.calls == [] From 6975733261295d6c3278e84e031362701d8016a5 Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 07/38] [None][fix] Take the token-ID fast path for MiniMax-M3 VL image requests The input processor implements the token-ID expansion hooks but never set the opt-in flag, so a prompt_token_ids + image request was detokenized and re-templated, doubling the image marker and failing the marker count check. Set the flag and test the dispatch end to end. Signed-off-by: Pranav Shrestha --- .../_torch/models/modeling_minimaxm3_vl.py | 10 ++- .../integration/test_lists/test-db/l0_cpu.yml | 1 + .../_torch/models/test_minimax_m3_vl.py | 62 +++++++++++++------ 3 files changed, 54 insertions(+), 19 deletions(-) diff --git a/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py b/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py index e05b6b53b676..ebb5ea0c58a6 100644 --- a/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py +++ b/tensorrt_llm/_torch/models/modeling_minimaxm3_vl.py @@ -14,7 +14,7 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence, Tuple +from typing import Any, ClassVar, Dict, Iterable, List, Mapping, Optional, Sequence, Tuple import numpy as np import torch @@ -1744,6 +1744,14 @@ class MiniMaxM3VLInputProcessor: supports_tokenization_cache = True + # Opt into ``BaseMultimodalInputProcessor.call_with_token_ids`` for + # ``prompt_token_ids`` + ``multi_modal_data`` requests (a pre-templated, + # pre-tokenized prompt from a front end such as Dynamo). The hooks below + # expand the placeholder token in place; without this flag the base class + # detokenizes the ids and re-applies the chat template, which doubles the + # image marker and rejects the request. + supports_token_id_mm_expansion: ClassVar[bool] = True + def __init__( self, model_path: str, diff --git a/tests/integration/test_lists/test-db/l0_cpu.yml b/tests/integration/test_lists/test-db/l0_cpu.yml index fd01805c4ea7..8ff779ac101d 100644 --- a/tests/integration/test_lists/test-db/l0_cpu.yml +++ b/tests/integration/test_lists/test-db/l0_cpu.yml @@ -99,6 +99,7 @@ l0_cpu: - unittest/_torch/models/test_minimax_m3_vl.py::test_preferred_media_io_format_is_uint8_numpy - unittest/_torch/models/test_minimax_m3_vl.py::test_video_token_count_is_one_bracket_per_temporal_patch - unittest/_torch/models/test_minimax_m3_vl.py::test_fast_path_expand_video_one_bracket_per_temporal_patch + - unittest/_torch/models/test_minimax_m3_vl.py::test_token_id_request_takes_the_fast_path_without_detokenizing - unittest/_torch/models/test_minimax_m3_vl.py::test_video_request_lets_the_processor_resize_frames # MiniMax-M3 VL forward under speculative decoding, KV reuse and chunked prefill. - unittest/_torch/models/test_minimax_m3_vl_spec_reuse.py::test_speculative_multimodal_prefill_forwards_orig_input_ids diff --git a/tests/unittest/_torch/models/test_minimax_m3_vl.py b/tests/unittest/_torch/models/test_minimax_m3_vl.py index 131d93a23313..52da35076935 100644 --- a/tests/unittest/_torch/models/test_minimax_m3_vl.py +++ b/tests/unittest/_torch/models/test_minimax_m3_vl.py @@ -2246,26 +2246,52 @@ def test_fast_path_expand_rejects_placeholder_count_mismatch(): ) -def test_fast_path_hooks_are_registered_on_processor_class(): - """The runtime's tokenized-prompt branch in - ``create_input_processor_with_hash.input_processor_wrapper`` gates - on ``hasattr(input_processor, 'get_text_with_mm_placeholders')`` and - ``hasattr(input_processor, 'expand_prompt_token_ids_for_mm')``. If - these go missing, the runtime detokenizes Dynamo's prompt_token_ids - and the M3 chat template gets re-applied, producing the duplicate - image marker that crashed the HF processor. +def test_token_id_request_takes_the_fast_path_without_detokenizing(): + """A ``prompt_token_ids`` + image request is expanded in token space. + + ``BaseMultimodalInputProcessor.__call__`` dispatches such a request to + ``call_with_token_ids`` only when ``supports_token_id_mm_expansion`` is + set; otherwise it detokenizes the ids and re-applies the chat template, + which doubles the image marker and rejects the request. A pre-templated + request from a front end such as Dynamo must therefore never reach the + tokenizer's ``decode`` and must come back with the placeholder framed as + ``VISION_START + IMAGE_TOKEN * N + VISION_END``. """ + from tensorrt_llm.sampling_params import SamplingParams + cls = get_minimax_m3_vl_input_processor_cls() - assert hasattr(cls, "get_text_with_mm_placeholders"), ( - "MiniMaxM3VLInputProcessor must expose get_text_with_mm_placeholders " - "for the tokenized + MM fast path; without it the runtime " - "detokenizes Dynamo's prompt_token_ids and re-runs the chat " - "template, doubling image markers and crashing at " - "image_grid_thw[index].prod()." - ) - assert hasattr(cls, "expand_prompt_token_ids_for_mm") - assert hasattr(cls, "get_num_tokens_per_image") - assert hasattr(cls, "get_num_tokens_per_video") + assert cls.supports_token_id_mm_expansion is True + + proc = _make_minimax_m3_vl_fast_path_processor() + + class _NoDecodeTokenizer: + def decode(self, *args, **kwargs): + raise AssertionError("token-id request was detokenized") + + proc._tokenizer = _NoDecodeTokenizer() + image = object() + grid = torch.tensor([[1, 4, 4]]) + text_calls: List[Dict] = [] + + def fake_call_with_text_prompt(inputs, sampling_params=None): + text_calls.append(dict(inputs)) + return [], {"multimodal_data": {"image": {"image_grid_thw": grid}}} + + proc.call_with_text_prompt = fake_call_with_text_prompt + # grid [1, 4, 4] at merge 2 -> 4 placeholders + the two framing tokens. + proc.get_num_tokens_per_image = lambda *, image, **kwargs: 6 + + ids, extra = proc( + {"prompt_token_ids": [1, _IMG_TOK, 2], "multi_modal_data": {"image": [image]}}, + SamplingParams(), + ) + + assert ids == [1, _VS_TOK] + [_IMG_TOK] * 4 + [_VE_TOK, 2] + assert torch.equal(extra["multimodal_data"]["image"]["image_grid_thw"], grid) + # The vision tower ran once, on the dummy placeholder prompt with the + # real image, not on a detokenized copy of the request. + assert [c.get("prompt") for c in text_calls] == [""] + assert text_calls[0]["multi_modal_data"] == {"image": [image]} # --------------------------------------------------------------------------- From 02f75fc9c19bca8199fc593737d0480d82ace077 Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 08/38] [None][feat] Serve MiniMax-M3 with a standalone DSpark drafter The drafter's nested HF config is flattened, its checkpoint tensors renamed, MXFP8 linears dequantized and Gemma norm weights folded at load. Its KV pool uses its own page size, is funded for its attention window plus the chunk in flight, and the unpaired draft mirror is sized from the target request and keeps the chunk start as history; the block decode reads a rebased table. Signed-off-by: Pranav Shrestha --- tensorrt_llm/_torch/models/_arch_index.py | 2 + tensorrt_llm/_torch/models/modeling_dspark.py | 257 ++++++++- tensorrt_llm/_torch/pyexecutor/_util.py | 221 ++++++- .../_torch/pyexecutor/config_utils.py | 76 +++ .../kv_cache/kv_cache_manager_v2.py | 122 +++- tensorrt_llm/_torch/speculative/dflash.py | 144 ++++- .../kv_cache/test_kv_cache_estimation.py | 427 +++++++++++++- .../kv_cache/test_kvcm2_integration.py | 543 ++++++++++++++++++ .../hw_agnostic/test_dflash_draft_kv.py | 104 ++++ .../hw_agnostic/test_minimax_dspark_draft.py | 241 ++++++++ 10 files changed, 2070 insertions(+), 67 deletions(-) create mode 100644 tests/unittest/_torch/speculative/hw_agnostic/test_minimax_dspark_draft.py diff --git a/tensorrt_llm/_torch/models/_arch_index.py b/tensorrt_llm/_torch/models/_arch_index.py index 64f8b80b2edc..f9a2006f3678 100644 --- a/tensorrt_llm/_torch/models/_arch_index.py +++ b/tensorrt_llm/_torch/models/_arch_index.py @@ -43,6 +43,7 @@ def is_builtin_zoo_module(module_name: str) -> bool: "Cohere2ForCausalLM": "modeling_cohere2", "Cosmos3ForConditionalGeneration": "modeling_cosmos3", "DeciLMForCausalLM": "modeling_nemotron_nas", + "DSparkMiniMaxDraftModel": "modeling_dspark", "DeepseekV32ForCausalLM": "modeling_deepseekv3", "DeepseekV3ForCausalLM": "modeling_deepseekv3", "DeepseekV41ForCausalLM": "modeling_deepseekv41", @@ -137,6 +138,7 @@ def is_builtin_zoo_module(module_name: str) -> bool: "DeepseekV41ForCausalLM": "modeling_deepseekv41", "DeepseekV41ForConditionalGeneration": "modeling_deepseekv41_multimodal", "DeepseekV4ForCausalLM": "modeling_deepseekv4", + "MiniMaxDSparkBackbone": "modeling_dspark", "Exaone4ForCausalLM": "modeling_exaone4", "Exaone4_5_ForConditionalGeneration": "modeling_exaone4_5", "ExaoneMoeForCausalLM": "modeling_exaone_moe", diff --git a/tensorrt_llm/_torch/models/modeling_dspark.py b/tensorrt_llm/_torch/models/modeling_dspark.py index ccb4609d09c8..1e2edea4800d 100644 --- a/tensorrt_llm/_torch/models/modeling_dspark.py +++ b/tensorrt_llm/_torch/models/modeling_dspark.py @@ -90,13 +90,15 @@ class or the edge would become a cycle. from torch import nn from transformers import PretrainedConfig -from tensorrt_llm.functional import RopeEmbeddingUtils +from tensorrt_llm.functional import PositionEmbeddingType, RopeEmbeddingUtils from tensorrt_llm.logger import logger from tensorrt_llm.mapping import Mapping from tensorrt_llm.quantization.mode import QuantAlgo from ..._utils import get_sm_version, is_sm_100f from ...models.modeling_utils import QuantConfig +from ..attention.attention import Attention +from ..attention.backends.interface import PositionalEmbeddingParams, RopeParams from ..cute_dsl_utils import IS_CUTLASS_DSL_AVAILABLE from ..distributed import AllReduceParams from ..model_config import ModelConfig @@ -106,7 +108,7 @@ class or the edge would become a cycle. from ..modules.linear import Linear, TensorParallelMode from ..modules.mhc.hyper_connection import HCHead from ..modules.rms_norm import RMSNorm -from ..pyexecutor.config_utils import is_mla +from ..pyexecutor.config_utils import MINIMAX_DSPARK_DRAFT_ARCH, is_mla from ..speculative.interface import SpeculativeDecodingMode from ..utils import AuxStreamType from .modeling_deepseekv4 import ( @@ -125,7 +127,12 @@ class or the edge would become a cycle. build_markov_head, dspark_markov_chain_logits, ) -from .modeling_utils import DecoderModel, register_auto_model, register_draft_model +from .modeling_utils import ( + DecoderModel, + DecoderModelForCausalLM, + register_auto_model, + register_draft_model, +) if IS_CUTLASS_DSL_AVAILABLE: from ..custom_ops.dspark_attention_custom_op import ( @@ -3503,6 +3510,230 @@ def load_weights_from_target_model(self, target_model: torch.nn.Module) -> None: self.lm_head = target_model.lm_head +# ---------------------------------------------------------------------------- +# MiniMax-M3 standalone DSpark drafter (architecture ``DSparkMiniMaxDraftModel``). +# +# A GQA backbone (64 heads / 4 KV heads / head_dim 128 at hidden 6144), per-head +# Gemma Q/K norm, NeoX RoPE over the first ``rotary_dim`` channels of each head, +# a dense swiglu-oai MLP and Gemma boundary norms: the dense-layer block of the +# MiniMax-M3 target. Weights ship MXFP8 (e4m3 + UE8M0 1x32 block scales) under +# ``language_model.model.dspark.layers..decoder_layer.*``; the heads are +# ``language_model.model.dspark.{fc,hidden_norm,final_norm,markov_head.*, +# confidence_head.proj.*}``. +# ---------------------------------------------------------------------------- + +MINIMAX_DSPARK_WEIGHT_PREFIX = "language_model.model.dspark." +_MINIMAX_DSPARK_DECODER_LAYER_INFIX = ".decoder_layer." +_MINIMAX_DSPARK_RENAMES = {"final_norm.weight": "norm.weight"} +# Every RMSNorm in this drafter is a Gemma norm, (1 + weight) * x. The backbone +# and the wrapper-owned ``hidden_norm`` are plain norms, so the stored weights +# are pre-incremented at load (the same fold the MiniMax-M3 target applies to +# its layer-boundary norms). +_MINIMAX_DSPARK_GEMMA_NORM_SUFFIXES = ( + "input_layernorm.weight", + "post_attention_layernorm.weight", + "q_norm.weight", + "k_norm.weight", + "hidden_norm.weight", +) +_MINIMAX_DSPARK_GEMMA_NORM_KEYS = ("norm.weight",) +_MXFP8_BLOCK_SIZE = 32 +_MXFP8_SCALE_SUFFIX = ".weight_scale_inv" + + +def remap_minimax_dspark_weight_name(name: str) -> str: + """Checkpoint tensor name -> the name the DFlash/DSpark loaders expect. + + Strips the ``language_model.model.dspark.`` namespace and the + ``.decoder_layer.`` level, and renames ``final_norm`` to ``norm``. Names + outside the namespace are returned unchanged. + """ + if not name.startswith(MINIMAX_DSPARK_WEIGHT_PREFIX): + return name + name = name[len(MINIMAX_DSPARK_WEIGHT_PREFIX) :] + name = name.replace(_MINIMAX_DSPARK_DECODER_LAYER_INFIX, ".") + return _MINIMAX_DSPARK_RENAMES.get(name, name) + + +def dequantize_mxfp8_weight( + weight: torch.Tensor, scale_inv: torch.Tensor, dtype: torch.dtype +) -> torch.Tensor: + """``weight`` (e4m3, ``[out, in]``) times ``2 ** (scale_inv - 127)`` per + ``[1, 32]`` block (``scale_inv``: UE8M0 as uint8, ``[out, in / 32]``), in + ``dtype``.""" + w = weight.to(torch.float32) + exponent = scale_inv.to(torch.int32) - 127 + scale = torch.exp2(exponent.to(torch.float32)) + scale = scale.repeat_interleave(_MXFP8_BLOCK_SIZE, dim=1)[:, : w.shape[1]] + return (w * scale).to(dtype) + + +def prepare_minimax_dspark_weights(weights: Dict, dtype: torch.dtype) -> Dict: + """Rename, dequantize and norm-fold a MiniMax DSpark drafter checkpoint. + + Returns a new dict keyed by the DFlash/DSpark loader names, with every + MXFP8 ``weight`` / ``weight_scale_inv`` pair replaced by one ``dtype`` + tensor and every Gemma norm weight incremented by one. + """ + renamed = {remap_minimax_dspark_weight_name(k): v for k, v in weights.items()} + out = {} + for name, value in renamed.items(): + if name.endswith(_MXFP8_SCALE_SUFFIX): + continue + tensor = value[:] if hasattr(value, "__getitem__") else value + scale_name = name[: -len(".weight")] + _MXFP8_SCALE_SUFFIX + if name.endswith(".weight") and scale_name in renamed: + scale = renamed[scale_name] + scale = scale[:] if hasattr(scale, "__getitem__") else scale + tensor = dequantize_mxfp8_weight(tensor, scale, dtype) + elif name.endswith(_MINIMAX_DSPARK_GEMMA_NORM_SUFFIXES) or name in ( + _MINIMAX_DSPARK_GEMMA_NORM_KEYS + ): + tensor = tensor + 1.0 + out[name] = tensor + return out + + +class MiniMaxDSparkAttention(Attention): + """GQA attention of the MiniMax DSpark drafter: fused ``qkv_proj`` + + ``o_proj`` with per-head ``q_norm`` / ``k_norm`` and partial NeoX RoPE. + + Only the projections, norms and RoPE parameters are read; the block decode + is ``DFlashForCausalLM.dflash_forward``. ``self.pretrained_config`` is what + that decode reads ``partial_rotary_factor`` from. + """ + + def __init__(self, model_config: ModelConfig, layer_idx: int): + config = model_config.pretrained_config + self.pretrained_config = config + head_dim = int( + getattr(config, "head_dim", config.hidden_size // config.num_attention_heads) + ) + super().__init__( + hidden_size=config.hidden_size, + num_attention_heads=config.num_attention_heads, + num_key_value_heads=config.num_key_value_heads, + max_position_embeddings=config.max_position_embeddings, + bias=False, + pos_embd_params=PositionalEmbeddingParams( + type=PositionEmbeddingType.rope_gpt_neox, + rope=RopeParams.from_config(config), + ), + rope_fusion=False, + layer_idx=layer_idx, + dtype=config.torch_dtype, + dense_bias=False, + config=model_config, + head_dim=head_dim, + ) + self.q_norm = RMSNorm( + hidden_size=head_dim, eps=config.rms_norm_eps, dtype=config.torch_dtype + ) + self.k_norm = RMSNorm( + hidden_size=head_dim, eps=config.rms_norm_eps, dtype=config.torch_dtype + ) + + +class MiniMaxDSparkDecoderLayer(DecoderLayer): + """One drafter layer: GQA projections + dense swiglu-oai MLP + the two norms.""" + + def __init__(self, model_config: ModelConfig, layer_idx: int): + super().__init__() + config = model_config.pretrained_config + self.layer_idx = layer_idx + self.self_attn = MiniMaxDSparkAttention(model_config, layer_idx) + intermediate_size = int( + getattr(config, "dspark_ffn_hidden_size", None) + or getattr(config, "dense_intermediate_size", config.intermediate_size) + ) + self.mlp = GatedMLP( + hidden_size=config.hidden_size, + intermediate_size=intermediate_size, + bias=False, + activation=torch.nn.functional.silu, + swiglu_alpha=float(getattr(config, "swiglu_alpha", 1.702)), + swiglu_beta=1.0, + swiglu_limit=float(getattr(config, "swiglu_limit", 7.0)), + dtype=config.torch_dtype, + config=model_config, + overridden_tp_size=1 if model_config.mapping.enable_attention_dp else None, + layer_idx=layer_idx, + ) + self.input_layernorm = RMSNorm( + hidden_size=config.hidden_size, eps=config.rms_norm_eps, dtype=config.torch_dtype + ) + self.post_attention_layernorm = RMSNorm( + hidden_size=config.hidden_size, eps=config.rms_norm_eps, dtype=config.torch_dtype + ) + + def forward(self, *args, **kwargs): + raise NotImplementedError( + "MiniMaxDSparkDecoderLayer is driven by DFlashForCausalLM.dflash_forward, " + "not by the generic decoder loop." + ) + + +class MiniMaxDSparkModel(DecoderModel): + """Weight container for the MiniMax DSpark block decode.""" + + def __init__(self, model_config: ModelConfig): + super().__init__(model_config) + config = model_config.pretrained_config + # Replaced by the target's embedding in load_weights_from_target_model; + # the checkpoint ships none. + self.embed_tokens = Embedding( + config.vocab_size, + config.hidden_size, + dtype=config.torch_dtype, + mapping=model_config.mapping, + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=True, + ) + self.layers = nn.ModuleList( + [ + MiniMaxDSparkDecoderLayer(model_config, layer_idx) + for layer_idx in range(config.num_hidden_layers) + ] + ) + self.norm = RMSNorm( + hidden_size=config.hidden_size, eps=config.rms_norm_eps, dtype=config.torch_dtype + ) + + def forward(self, *args, **kwargs): + raise NotImplementedError( + "MiniMaxDSparkModel is a weight container for the DSpark block decode; " + "see DFlashForCausalLM.dflash_forward." + ) + + +@register_auto_model(MINIMAX_DSPARK_DRAFT_ARCH) +class MiniMaxDSparkBackbone(DecoderModelForCausalLM[MiniMaxDSparkModel, PretrainedConfig]): + """Registry entry for the MiniMax DSpark drafter checkpoint + (``config.architectures == ["DSparkMiniMaxDraftModel"]``).""" + + def __init__(self, model_config: ModelConfig): + config = model_config.pretrained_config + super().__init__( + MiniMaxDSparkModel(model_config), + config=model_config, + hidden_size=config.hidden_size, + vocab_size=config.vocab_size, + ) + + +class MiniMaxDSparkForCausalLM(GQADSparkForCausalLM): + """DSpark drafter on the MiniMax-M3 GQA backbone, from a standalone checkpoint. + + The GQA block decode and head set are inherited; this class only translates + the checkpoint (namespace, MXFP8 dequantization, Gemma norm fold) before + the DFlash loader sees it. See :func:`prepare_minimax_dspark_weights`. + """ + + def load_weights(self, weights: Dict, weight_mapper=None, **kwargs): + weights = prepare_minimax_dspark_weights(weights, self.config.torch_dtype) + return super().load_weights(weights, weight_mapper=weight_mapper, **kwargs) + + def draft_is_embedded_in_target(model_config) -> bool: """True when the DSpark draft weights live inside the target checkpoint. @@ -3560,14 +3791,21 @@ def _build_dspark_draft(model_config, draft_config, lm_head, model): # what differs between the two wrappers, and everything below it # (``DFlashForCausalLM.__init__``) already resolves the backbone through the # model registry, so keying on model_type a second time would force a new - # entry for every GQA family that already works. - if is_mla(draft_config.pretrained_config): + # entry for every GQA family that already works. The MiniMax drafter is the + # one architecture-keyed exception: its checkpoint needs a translation step + # (namespace, MXFP8, Gemma norms) that the generic GQA loader has no hook for. + draft_arches = getattr(draft_config.pretrained_config, "architectures", None) or [] + if draft_arches and draft_arches[0] == MINIMAX_DSPARK_DRAFT_ARCH: + drafter_cls = MiniMaxDSparkForCausalLM + elif is_mla(draft_config.pretrained_config): return MLADSparkForCausalLM( draft_config, dflash_attention_backend=model_config.spec_config.attention_backend, ) + else: + drafter_cls = GQADSparkForCausalLM # The drafter's own ModelConfig has no spec_config, so the worker's mask id is handed over here. - return GQADSparkForCausalLM( + return drafter_cls( draft_config, dflash_attention_backend=model_config.spec_config.attention_backend, spec_mask_token_id=model_config.spec_config.mask_token_id, @@ -3582,6 +3820,13 @@ def _build_dspark_draft(model_config, draft_config, lm_head, model): # Standalone flavour. "GQADSparkForCausalLM", "MLADSparkForCausalLM", + # MiniMax-M3 GQA drafter and its checkpoint translation. + "MiniMaxDSparkForCausalLM", + "MiniMaxDSparkBackbone", + "MiniMaxDSparkModel", + "prepare_minimax_dspark_weights", + "remap_minimax_dspark_weight_name", + "dequantize_mxfp8_weight", # MLA drafter backbone (a weight container for the block decode). "K3DsparkForCausalLM", "K3DsparkModel", diff --git a/tensorrt_llm/_torch/pyexecutor/_util.py b/tensorrt_llm/_torch/pyexecutor/_util.py index 7aedaf8a81e6..b177b8c50894 100644 --- a/tensorrt_llm/_torch/pyexecutor/_util.py +++ b/tensorrt_llm/_torch/pyexecutor/_util.py @@ -14,9 +14,11 @@ import copy import dataclasses +import math import os from typing import Any, Dict, List, Optional, Sequence, Union +import numpy as np import torch import tensorrt_llm @@ -1027,9 +1029,12 @@ def _per_manager_cache_cost(self, is_draft: bool = False, mapping=None, include_request_cost: bool = False, + tokens_per_block: Optional[int] = None, **extra_kwargs) -> CacheCost: kv_cache_config = (kv_cache_config if kv_cache_config is not None else self._kv_cache_config) + tokens_per_block = (tokens_per_block if tokens_per_block is not None + else self._tokens_per_block) if not is_draft: derived_windows = _derive_v2_layer_type_attention_windows( kv_cache_config, manager_cls, model_config, self._max_seq_len) @@ -1044,7 +1049,7 @@ def _per_manager_cache_cost(self, manager_cls.get_cache_size_per_token( model_config, mapping if mapping is not None else self._mapping, - tokens_per_block=self._tokens_per_block, + tokens_per_block=tokens_per_block, max_seq_len=self._max_seq_len, max_batch_size=self._max_batch_size, max_num_tokens=self._max_num_tokens, @@ -1221,13 +1226,25 @@ def scaled(cost: CacheCost) -> CacheCost: draft_kv_cache_manager_cls, effective_draft_config, draft_kv_cache_config) if self._speculative_config.spec_dec_mode.is_external_drafter(): - # External drafter: layers start from 0, normal PP distribution + # External drafter: layers start from 0, normal PP distribution. + # Costed at the page size its pool is carved with. + draft_tokens_per_block = self._draft_tokens_per_block() + if self._is_standalone_draft_pool(): + per_token_cost = self._standalone_draft_per_token_cost( + draft_kv_cache_manager_cls, effective_draft_config, + draft_kv_cache_config, draft_mapping, + draft_tokens_per_block) + return scaled( + self._standalone_draft_pool_cost( + per_token_cost, kv_cache_config, + draft_tokens_per_block)) return scaled( self._per_manager_cache_cost( draft_kv_cache_manager_cls, effective_draft_config, draft_kv_cache_config, mapping=draft_mapping, + tokens_per_block=draft_tokens_per_block, is_draft=True, include_request_cost=include_request_cost)) elif self._mapping.is_last_pp_rank(): @@ -2034,6 +2051,10 @@ def _joint_reuse_supported(self) -> bool: """ if not self._is_kv_cache_manager_v2: return False + if self._draft_tokens_per_block() != self._tokens_per_block: + # KVCacheV2Scheduler._try_allocate_generation pairs the two pools + # block for block, which needs one page size. + return False lookahead = draft_prompt_lookahead(self._speculative_config) if lookahead is None: return False @@ -2047,6 +2068,49 @@ def _joint_reuse_supported(self) -> bool: return False return True + def _draft_tokens_per_block(self) -> int: + """Page size of the separate one-model draft KV pool. + + The target's, unless the drafter reading the pool cannot address pages + of that size. A standalone DFlash/DSpark drafter on the TRTLLM backend + runs the shared trtllm-gen op set, which accepts only + ``FlashInferTrtllmGenFmha.SUPPORTED_TOKENS_PER_BLOCK``, while the + target's page size follows the target (MiniMax-M3 MSA and FP4 MLA pin + 128). The drafter validates the page size of the pool it is handed + (``validate_dflash_trtllm_gen_runtime``) once that pool is carved, so + the pool is carved at the largest size it accepts instead. + + The two page sizes are independent: the draft manager mirrors the + target per token (``KVCacheManagerV2._prepare_draft_resources``) and + the drafter builds its block tables from its own pool + (``DFlashDrafter._init_ctx_block_tables``). Joint block reuse is the + one pairing that assumes equal page sizes; ``_joint_reuse_supported`` + declines it when they differ. + """ + tokens_per_block = self._tokens_per_block + draft_model = getattr(self._model_engine.model, "draft_model", None) + if draft_model is None: + return tokens_per_block + # Only drafters running the shared DFlash op sets read pages with + # trtllm-gen; the MLA DSpark drafter names its own kernels "TRTLLM". + if not getattr(draft_model, "_uses_worker_attention_backend", False): + return tokens_per_block + if getattr(draft_model, "dflash_attention_backend", None) != "TRTLLM": + return tokens_per_block + from ..attention.backends.fmha.flashinfer_trtllm_gen import \ + FlashInferTrtllmGenFmha + supported = FlashInferTrtllmGenFmha.SUPPORTED_TOKENS_PER_BLOCK + if tokens_per_block in supported: + return tokens_per_block + smaller = [size for size in supported if size < tokens_per_block] + draft_tokens_per_block = max(smaller) if smaller else min(supported) + logger.info_once( + f"Separate draft KV cache uses tokens_per_block=" + f"{draft_tokens_per_block}: the DFlash TRTLLM attention backend " + f"supports {sorted(supported)}, not the target's {tokens_per_block}.", + key="draft_tokens_per_block") + return draft_tokens_per_block + def _get_effective_draft_config(self) -> ModelConfig: """ Return the ModelConfig to use for draft KV cache creation. @@ -2135,6 +2199,149 @@ def _get_draft_max_attention_window( self._get_num_draft_layers(), ) + def _is_standalone_draft_pool(self) -> bool: + """Whether the separate draft pool belongs to a standalone drafter. + + True for the DFlash/DSpark drafter. Its pool mirrors the target per + request (``KVCacheManagerV2._prepare_draft_resources``) and is never + shared between requests: block reuse lets the target hold a prompt + prefix once for every request that matches it, while each mirror + stores the drafter's K/V for that prefix privately. The target's + per-token funding therefore does not bound the mirrors, and the pool + is funded as a fixed cost for ``max_batch_size`` of them + (``_standalone_draft_pool_cost``). + """ + spec_dec_mode = self._speculative_config.spec_dec_mode + return spec_dec_mode.is_dflash() or spec_dec_mode.is_dspark() + + def _standalone_draft_mirror_window( + self, kv_cache_config: KvCacheConfig) -> Optional[int]: + """The largest attention window the standalone drafter's pool keeps, + as the manager will hold it, or None when a layer attends to the whole + context. + + The manager extends sliding windows by the reuse-match backoff under + block reuse (``KVCacheManagerV2.__init__``), so the funded window + does the same. + """ + from .kv_cache.kv_cache_manager_v2 import _extend_swa_windows_for_reuse + windows = self._get_draft_max_attention_window(self._max_seq_len, + kv_cache_config) + if windows is None: + return None + if kv_cache_config.enable_block_reuse: + windows = _extend_swa_windows_for_reuse( + windows, + draft_prompt_lookahead(self._speculative_config) or 0, + self._max_seq_len) + if any(window is None or window >= self._max_seq_len + for window in windows): + return None + return max(windows) + + def _standalone_draft_per_token_cost(self, manager_cls, model_config, + draft_kv_cache_config: KvCacheConfig, + mapping, + tokens_per_block: int) -> CacheCost: + """The standalone drafter's KV bytes per token, as a pure slope. + + Costed with its windows, the drafter's pool is a window-sized fixed + cost with no per-token slope (``get_cache_size_per_token`` charges a + sliding-window layer per request, not per token), which is not what + ``_standalone_draft_pool_cost`` multiplies by. The same layers costed + without windows give the bytes one token of drafter K/V occupies, on + the same manager class and dtype path the pool is allocated with. + """ + return self._per_manager_cache_cost( + manager_cls, + model_config, + draft_kv_cache_config.model_copy( + update={"max_attention_window": None}), + mapping=mapping, + tokens_per_block=tokens_per_block, + is_draft=True) + + def _standalone_draft_pool_cost(self, per_token_cost: CacheCost, + kv_cache_config: KvCacheConfig, + tokens_per_block: int) -> CacheCost: + """Funding for the standalone drafter's pool. + + ``per_token_cost`` is the drafter's KV cost per token + (``_standalone_draft_per_token_cost``): a slope and no fixed part. + + The pool carries the drafter's attention windows, so a mirror's live + pages are bounded by what its window keeps plus what it is computing: + the manager releases pages below ``history + 1 - window`` on every + resize (``AttnLifeCycle::getStaleRange``), and the mirror carries the + chunk start as its history on each context resize and the generated + length on each generation update. The chunk is at most + ``max_num_tokens``; the generation headroom is counted twice because + the first generation step grows the mirror by another step before + ``update_resources`` advances its history; one page covers the live + interval straddling page boundaries at both ends. The number of such + mirrors is the number of requests that can hold KV at once + (``max_admissible_sequences``): ``max_batch_size`` per pipeline + stage, doubled under the overlap scheduler or disaggregated serving, + where requests finishing in one iteration keep their pages until the + next iteration's batch has been prepared. The same bound holds with + or without block reuse. + + Without a window a mirror keeps its whole context. Then the target's + per-token funding bounds the mirrors as long as the target holds every + token itself, and under block reuse the pool is funded for that many + full-length mirrors instead. + """ + from .kv_cache.kv_cache_manager_v2 import (_get_generation_kv_capacity, + max_admissible_sequences) + if per_token_cost.slope <= 0 or per_token_cost.intercept != 0: + # A windowed cost (slope 0, fixed part) would fund the pool with + # zero bytes and the executor would fail to create the KV cache. + raise ValueError( + "Standalone draft pool funding needs the drafter's bytes per " + f"token as a pure slope, got {per_token_cost}.") + _, headroom = _get_generation_kv_capacity(self._speculative_config, + is_draft=True) + num_mirrors = max_admissible_sequences( + self._max_batch_size, + self._mapping, + is_disagg=self._is_disagg, + disable_overlap_scheduler=self._disable_overlap_scheduler) + window = self._standalone_draft_mirror_window(kv_cache_config) + if window is None: + if not kv_cache_config.enable_block_reuse: + return per_token_cost + tokens_per_mirror = self._max_seq_len + headroom + reason = ("block reuse lets the target share prompt tokens the " + "drafter stores per request") + else: + tokens_per_mirror = (window + self._max_num_tokens + 2 * headroom + + tokens_per_block) + reason = (f"the drafter's {window}-token window bounds what a " + "mirror keeps beyond the chunk it is computing") + cost = self.standalone_draft_pool_worst_case_cost( + per_token_cost.slope, tokens_per_block, tokens_per_mirror, + num_mirrors, kv_cache_config.max_util_for_resume) + logger.info_once( + "Separate draft KV cache of the standalone drafter is funded for " + f"{num_mirrors} mirrors of {tokens_per_mirror} tokens " + f"({cost.intercept / GB:.2f} GiB): {reason}.", + key="draft_pool_funding") + return cost + + @staticmethod + def standalone_draft_pool_worst_case_cost( + bytes_per_token: int, tokens_per_block: int, + tokens_per_request: int, num_mirrors: int, + max_util_for_resume: float) -> CacheCost: + """``num_mirrors`` mirrors of ``tokens_per_request`` tokens, page + rounded per request and scaled to the resume watermark like the + manager's fixed costs (``KVCacheManagerV2.get_cache_size_per_token``).""" + pages = num_mirrors * math.ceil(tokens_per_request / tokens_per_block) + resume_util = float(np.float32(max_util_for_resume)) + return CacheCost(slope=0, + intercept=math.ceil(pages / resume_util) * + tokens_per_block * bytes_per_token) + def _get_one_model_draft_kv_cache_config( self, kv_cache_config: KvCacheConfig, @@ -2146,9 +2353,11 @@ def _get_one_model_draft_kv_cache_config( # Estimation uses a small max_tokens-sized temporary draft cache before # the measured GPU budget is available to split. Applying VSWA there # would size every window pool from the unsplit free-memory budget. - max_attention_window = (None if estimating_kv_cache else - self._get_draft_max_attention_window( - max_seq_len, kv_cache_config)) + if estimating_kv_cache: + max_attention_window = None + else: + max_attention_window = self._get_draft_max_attention_window( + max_seq_len, kv_cache_config) return kv_cache_config.model_copy( update={"max_attention_window": max_attention_window}) @@ -2231,7 +2440,7 @@ def _create_one_model_draft_kv_cache_manager( kv_cache_manager_cls=draft_kv_cache_manager_cls, mapping=draft_mapping, kv_cache_config=draft_kv_config, - tokens_per_block=self._tokens_per_block, + tokens_per_block=self._draft_tokens_per_block(), max_seq_len=max_seq_len, max_batch_size=self._max_batch_size, spec_config=self._speculative_config, diff --git a/tensorrt_llm/_torch/pyexecutor/config_utils.py b/tensorrt_llm/_torch/pyexecutor/config_utils.py index a9b2b16bfbe3..657bc9c260b3 100644 --- a/tensorrt_llm/_torch/pyexecutor/config_utils.py +++ b/tensorrt_llm/_torch/pyexecutor/config_utils.py @@ -973,6 +973,78 @@ def match_nemotron_h_layer_types(model_config, layer_types): return [_NEMOTRON_H_LEGACY_LAYER_TYPES.get(t, t) for t in layer_types] +MINIMAX_DSPARK_DRAFT_ARCH = "DSparkMiniMaxDraftModel" + +# Keys of the standalone MiniMax DSpark drafter config.json that describe the +# target checkpoint it was exported next to, not the drafter itself. The +# drafter's MXFP8 linears are dequantized to the compute dtype at load, so the +# quantization block is dropped and the backbone is built unquantized. +_MINIMAX_DSPARK_DROPPED_KEYS = frozenset({ + "text_config", + "vision_config", + "quantization_config", + "architectures", + "model_type", + "_name_or_path", +}) + + +def build_minimax_dspark_draft_config( + config_dict: dict) -> transformers.PretrainedConfig: + """Flat drafter config for a standalone MiniMax-M3 DSpark checkpoint. + + The drafter ships a MiniMax VL-shaped config.json: ``model_type`` + ``minimax_m3_vl`` with the backbone geometry nested under ``text_config``, + no ``auto_map`` and no modeling code, so ``AutoConfig.from_pretrained`` + cannot resolve it. The drafter's own fields (``dspark_*``, + ``dflash_config``, ``hidden_size``, ``vocab_size``, ...) are duplicated + at the top level and inside ``text_config``; the text side wins where both + are present. The result is a plain ``PretrainedConfig`` with every field as + an attribute, which is what ``DFlashForCausalLM`` and the registered + ``DSparkMiniMaxDraftModel`` backbone read. + + ``dflash_config`` gets ``use_swa`` / ``swa_window_size`` when the drafter + declares ``sliding_attention`` layers with a ``sliding_window`` but not the + DFlash spelling: the block decode keys its per-layer window on the former, + and the HF reference these drafters are distilled with applies the window + on exactly those layers. + """ + flat = { + k: v + for k, v in config_dict.items() if k not in _MINIMAX_DSPARK_DROPPED_KEYS + } + text = config_dict.get("text_config") or {} + flat.update({ + k: v + for k, v in text.items() if k not in _MINIMAX_DSPARK_DROPPED_KEYS + }) + dflash = dict(flat.get("dflash_config") or {}) + layer_types = dflash.get("layer_types") or flat.get("layer_types") or [] + sliding_window = dflash.get("sliding_window") or flat.get("sliding_window") + if ("use_swa" not in dflash and sliding_window + and "sliding_attention" in layer_types): + dflash["use_swa"] = True + dflash["swa_window_size"] = int(sliding_window) + dflash.setdefault("layer_types", list(layer_types)) + flat["dflash_config"] = dflash + flat["architectures"] = [MINIMAX_DSPARK_DRAFT_ARCH] + # Not ``minimax_m3_vl``: that model_type selects the VL target's + # text_config unwrapping and input-processor registration, neither of + # which applies to the flattened drafter. + flat["model_type"] = "minimax_dspark_draft" + dtype = flat.pop("dtype", None) or flat.pop("torch_dtype", + None) or "bfloat16" + flat.pop("torch_dtype", None) + if isinstance(dtype, str): + dtype = str_dtype_to_torch(dtype) + config = transformers.PretrainedConfig() + for key, value in flat.items(): + setattr(config, key, value) + config.dtype = dtype + config.torch_dtype = dtype + return config + + def load_pretrained_config(model_name_or_path: str, trust_remote_code: bool = False, checkpoint_format: Optional[str] = None, @@ -1114,6 +1186,10 @@ def load_pretrained_config(model_name_or_path: str, vision_config = getattr(model_config, "vision_config", None) if getattr(vision_config, "num_hidden_layers", 0) > 0: model_config.architectures = ["DeepseekV41ForConditionalGeneration"] + elif architectures and architectures[0] == MINIMAX_DSPARK_DRAFT_ARCH: + # Standalone MiniMax-M3 DSpark drafter: VL-shaped config.json with + # no auto_map, flattened here (see build_minimax_dspark_draft_config). + model_config = build_minimax_dspark_draft_config(config_dict) elif model_type in _CONFIG_REGISTRY: config_class = _CONFIG_REGISTRY[model_type] model_config = config_class.from_pretrained(model_name_or_path, diff --git a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py index 5f3d0eb64f1c..552bfc609ea9 100644 --- a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py +++ b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py @@ -533,6 +533,31 @@ def _estimate_swa_cache_size( return size_per_token, size_per_request +def max_admissible_sequences( + max_batch_size: int, mapping: Mapping, *, is_disagg: bool, disable_overlap_scheduler: bool +) -> int: + """Most requests that can hold KV in the manager at the same time. + + With pipeline parallelism, multiple microbatches can be in flight + simultaneously, so every concurrent sequence needs room. In disaggregated + mode the coefficient is 2: at any moment up to ``max_num_sequences`` + requests can be actively generating while another up to + ``max_num_sequences`` are still in KV transfer (TRANS_IN_PROGRESS) and + keep their index slots and pages. With the overlap scheduler on (non-PP) a + retiring request needs the same coefficient: ``GENERATION_TO_COMPLETE`` + requests leave the scheduler's budget (``KVCacheV2Scheduler`` schedules + states below that one), so the next iteration admits new requests in + their place, while their KV is released only by + ``PyExecutor._process_previous_batch`` after that iteration's + ``prepare_resources``. The IndexMapper is sized from this count, and so is + any pool funded as a fixed cost per resident request + (``KvCacheCreator._standalone_draft_pool_cost``). + """ + max_num_sequences = max_batch_size * mapping.pp_size + needs_extra_leases = is_disagg or (not disable_overlap_scheduler and not mapping.has_pp()) + return max_num_sequences * (2 if needs_extra_leases else 1) + + def _get_generation_kv_capacity(spec_config, *, is_draft: bool) -> tuple[int, int]: """Return draft-token reserve and total capacity headroom over history.""" from tensorrt_llm._torch.speculative import get_num_extra_kv_tokens @@ -1412,6 +1437,14 @@ def __init__( from tensorrt_llm._torch.speculative import draft_prompt_lookahead, get_num_extra_kv_tokens self.num_extra_kv_tokens = get_num_extra_kv_tokens(spec_config) + # The draft request view (LlmRequest.use_draft_model) reads the context + # position and chunk a separate draft engine advances for its own + # prefill; a one-engine external drafter (DFlash, DSpark) has no such + # engine, so those fields stay at position 0 / chunk = prompt length + # and its mirror follows the target's chunking instead. + self._mirror_uses_draft_view = not ( + spec_config is not None and spec_config.spec_dec_mode.is_external_drafter() + ) # Mirror V1: expose max_draft_len so the native disagg AuxBuffer # (_make_aux_buffer's getattr fallback) is sized for MTP/spec decoding. self.max_draft_len = spec_config.max_draft_len if spec_config is not None else 0 @@ -1904,20 +1937,9 @@ def create_cold_page_codec(cache_config: object) -> Optional[object]: else None ) - # With pipeline parallelism, multiple microbatches can be in-flight - # simultaneously, so we need slots for all concurrent sequences. # Reserve stable slots for persistent request IDs such as CUDA-graph # padding requests. The default preserves the main-branch allocation. - # In disaggregated mode, use a coefficient of 2: at any moment up to - # `max_num_sequences` requests can be actively generating while another - # up to `max_num_sequences` requests are still in KV transfer - # (TRANS_IN_PROGRESS) and continue to hold their index slots. The 2x - # capacity lets the next batch of active requests acquire slots without - # waiting for the previous batch's transfers to finish. With the overlap - # scheduler on (non-PP), a retiring request holds its lease one extra - # iteration and needs the same coefficient. max_num_sequences = max_batch_size * mapping.pp_size - needs_extra_leases = is_disagg or (not disable_overlap_scheduler and not mapping.has_pp()) assert num_reserved_index_slots >= 0, "num_reserved_index_slots must be non-negative" # Both diagnostics below are off unless their environment variable is # set; with neither set nothing here changes any allocation, any page @@ -1936,7 +1958,12 @@ def create_cold_page_codec(cache_config: object) -> Optional[object]: self._fresh_pages_filled: Dict[int, Dict[int, np.ndarray]] = {} self._fresh_fill_announced = False self._fresh_fill_unavailable_announced = False - self.max_admissible_sequences = max_num_sequences * (2 if needs_extra_leases else 1) + self.max_admissible_sequences = max_admissible_sequences( + max_batch_size, + mapping, + is_disagg=is_disagg, + disable_overlap_scheduler=disable_overlap_scheduler, + ) # The guard page is held by a permanent sequence, so it needs an index # slot of its own. Taking one of the scheduler's would change which # requests get admitted, so a run with the diagnostic on would no @@ -4656,16 +4683,22 @@ def _draft_pool_diagnostic(self) -> str: byte budget, and the capacity scheduler admits on the TARGET pool alone (`scheduler_v2` touches `draft_kv_cache_manager` only to suspend/free). A draft pool smaller in tokens than the target cannot backpressure -- it can - only raise, and the raise kills every rank. + only raise (see _mirror_resize_error), and the raise kills every rank. The first question is always how big the draft pool was and how full, so the message answers it rather than leaving post-hoc arithmetic over the split log. """ - live = sum(c.capacity for c in self.kv_cache_map.values()) + capacity = sum(c.capacity for c in self.kv_cache_map.values()) + blocks = sum(int(c.num_blocks) for c in self.kv_cache_map.values()) + per_cache = ", ".join( + f"{req_id}: capacity={c.capacity} history={c.history_length} blocks={c.num_blocks}" + for req_id, c in self.kv_cache_map.items() + ) return ( - f" [draft pool: {len(self.kv_cache_map)} live caches holding " - f"{live} tokens, gpu_max_tokens={self._gpu_max_tokens}]" + f" [draft pool: {len(self.kv_cache_map)} live caches, capacity {capacity} tokens " + f"in {blocks} allocated blocks ({per_cache}), " + f"windows={self.max_attention_window_vec}, gpu_max_tokens={self._gpu_max_tokens}]" ) def _prepare_draft_resources(self, scheduled_batch: ScheduledRequests): @@ -4677,8 +4710,11 @@ def _prepare_draft_resources(self, scheduled_batch: ScheduledRequests): scheduler pre-creates context caches to coordinate the reusable prefix. """ # Joint V2 reuse deliberately shares the target request progress. A - # separate draft engine still uses the legacy draft request view. - with request_context(not self.enable_joint_kv_cache_reuse, scheduled_batch): + # separate draft engine still uses the legacy draft request view; a + # one-engine external drafter never advances that view, so its mirror + # reads the target's position and chunk (see _mirror_uses_draft_view). + use_draft_view = self._mirror_uses_draft_view and not self.enable_joint_kv_cache_reuse + with request_context(use_draft_view, scheduled_batch): for req in scheduled_batch.context_requests: kv_cache = self._mirror_draft_kv_cache(req) if kv_cache is None: @@ -4711,12 +4747,18 @@ def _prepare_draft_resources(self, scheduled_batch: ScheduledRequests): + draft_len + self.num_extra_kv_tokens ) - if not kv_cache.resize(capacity): - raise RuntimeError( - f"Draft KV cache context resize failed for request " - f"{req.py_request_id}: could not resize to {capacity} tokens" - f"{self._draft_pool_diagnostic()}" - ) + # The chunk start is this cache's history: sliding-window layers + # keep pages only from history + 1 - window (AttnLifeCycle:: + # getStaleRange), so a mirror holds its window plus the chunk + # it is computing, which is what the standalone drafter's pool + # is funded for (KvCacheCreator._standalone_draft_pool_cost). + # The executor advances the draft history through + # update_context_resources only under joint reuse, and a reused + # target prefix starts the mirror at position > 0 with no pages + # of its own, so the resize carries the history itself. + history = max(kv_cache.history_length, req.context_current_position) + if not kv_cache.resize(capacity, history): + raise self._mirror_resize_error(req, "context", capacity) self._after_shrink(req.py_request_id, kv_cache) for req in scheduled_batch.generation_requests: @@ -4740,11 +4782,26 @@ def _prepare_draft_resources(self, scheduled_batch: ScheduledRequests): continue new_cap = self._required_gen_capacity(req, kv_cache.capacity) if not kv_cache.resize(new_cap): - raise RuntimeError( - f"Draft KV cache generation resize failed for request " - f"{req.py_request_id}: could not resize to {new_cap} tokens" - f"{self._draft_pool_diagnostic()}" - ) + raise self._mirror_resize_error(req, "generation", new_cap) + + def _mirror_resize_error(self, req: LlmRequest, phase: str, capacity: int) -> RuntimeError: + """The error for a draft mirror that could not grow to ``capacity``. + + `KvCache.resize` leaves the cache as it was when the pool has no page to + give, and the cache as it was cannot hold what the drafter writes next: + the DFlash block decode appends the block's own K/V past the stored + context through the mirror's block table, so a mirror that did not grow + (a fresh one has no page at all) hands that write a table entry with no + page behind it (``BAD_PAGE_INDEX`` on the python V2 backend, another + request's block 0 on the cpp one). There is no per-request way to skip + drafting, so the failure stays explicit; the pool funding + (``KvCacheCreator._standalone_draft_pool_cost``) is what keeps it + from happening. The message carries the pool diagnostic. + """ + return RuntimeError( + f"Draft KV cache {phase} resize failed for request {req.py_request_id}: " + f"could not resize to {capacity} tokens{self._draft_pool_diagnostic()}" + ) def _reuse_token_source(self, req: LlmRequest) -> Sequence[int]: """Beam-0 tokens for block reuse, in the form the active backend consumes. @@ -5575,7 +5632,12 @@ def release_resources( release_resources(req, free_draft_resources=True) return None draft_kv_cache.stop_committing() - success = draft_kv_cache.resize(dummy_capacity) + if is_gen and not materialize_history: + draft_kv_cache.enable_swa_scratch_reuse = False + # Same history hint as the target cache above: a sliding-window + # draft pool then backs only the dummy's window, not its whole + # length, which is what the standalone drafter's pool is funded for. + success = draft_kv_cache.resize(dummy_capacity, history_length=history_hint) if not success: release_resources(req, free_draft_resources=True) return None diff --git a/tensorrt_llm/_torch/speculative/dflash.py b/tensorrt_llm/_torch/speculative/dflash.py index 88da4911cca0..27a28dd9961a 100644 --- a/tensorrt_llm/_torch/speculative/dflash.py +++ b/tensorrt_llm/_torch/speculative/dflash.py @@ -519,6 +519,21 @@ def dflash_allocated_ctx_limit( return (block_counts * page_size - block_size).clamp(min=0) +def dflash_rebased_ctx_rows(rows: torch.Tensor, base_pages: torch.Tensor) -> torch.Tensor: + """Block-table rows shifted left by ``base_pages`` entries each. + + Entry ``j`` of the result names the page entry ``j + base`` of the input + row named, so a reader that numbers positions from ``base * page_size`` + reaches the same physical pages as one numbering from 0 and never + addresses an entry below ``base``. Entries shifted in from past the row's + end repeat the last entry; a reader never dereferences them because its + sequence length is shortened by the same ``base * page_size``. + """ + width = rows.size(1) + index = torch.arange(width, device=rows.device).unsqueeze(0) + base_pages.view(-1, 1) + return rows.gather(1, index.clamp_(max=width - 1)) + + @dataclass class DFlashSpecMetadata(SpecMetadata): """Metadata for DFlash speculative decoding. @@ -746,6 +761,14 @@ def __init__( # manager it is replaced by _ctx_block_tables, refreshed per iteration. self._ctx_page_table = None self._ctx_block_tables = None # [max_batch+1, max_blocks_per_seq] int32 + # The block table the block decode reads: each row of _ctx_block_tables + # shifted left by that slot's _ctx_base_pages, so a prefix the target + # reused (no drafter K/V stored for it) is not addressable. + self._ctx_block_tables_read = None + self._ctx_base_pages = None # [max_batch+1] pages below the first stored position + self._ctx_base_host = None + self._saved_ctx_base = None + self._saved_ctx_base_host = None self._ctx_block_indptr = None # Pages the manager handed this request, per batch row. Saturates at # the table width on the default backend -- dflash_allocated_ctx_limit. @@ -990,6 +1013,7 @@ def _init_ctx_block_tables( self._ctx_block_tables = torch.zeros( (num_slots, max_blocks), dtype=torch.int32, device="cuda" ) + self._ctx_block_tables_read = torch.zeros_like(self._ctx_block_tables) self._ctx_block_indptr = torch.arange( 0, (num_slots + 1) * max_blocks, max_blocks, dtype=torch.int32, device="cuda" ) @@ -1059,6 +1083,7 @@ def _lazy_init_ctx_buffers( # pass falling back to the arena still rebinds once the real pool lands. self._ctx_kv_manager = draft_kv_cache_manager self._ctx_block_tables = None + self._ctx_block_tables_read = None self._ctx_block_counts = None max_batch = spec_metadata.max_num_requests @@ -1114,6 +1139,8 @@ def _lazy_init_ctx_buffers( self._ctx_len = torch.zeros(num_slots, dtype=torch.long, device="cuda") self._ctx_len_host = [0] * num_slots + self._ctx_base_pages = torch.zeros(num_slots, dtype=torch.long, device="cuda") + self._ctx_base_host = [0] * num_slots self._batch_to_slot = torch.zeros(max_batch, dtype=torch.long, device="cuda") self._free_slots = deque(range(max_batch)) @@ -1285,6 +1312,37 @@ def _ctx_paged_index_args(self): indptr = self._ctx_kv_indptr return table.flatten().contiguous(), indptr + def _ctx_store_start(self, cur: int, first_pos: int) -> int: + """Position at which a context chunk's drafter K/V is stored. + + ``cur`` is the slot's stored length, ``first_pos`` the chunk's first + target position. They differ only when block reuse let the target skip + [0, first_pos), so just the tail reaches the capture buffer. + + On the managed pool the block table is indexed by target position, so + the tail is stored at ``first_pos``. The private arena numbers every + request from 0, where the tail is all the context there is. + """ + if cur == 0 and first_pos > 0 and self._ctx_block_tables is not None: + return first_pos + return cur + + def _ctx_base_for_store(self, cur: int, start: int) -> Optional[int]: + """Pages the block decode must not read for a slot whose first stored + position is ``start``, or None when the slot numbers from ``cur``. + + A pool paired with the target's reuse holds the prefix's drafter K/V + below ``start``, so it stays readable. An unpaired pool holds no drafter + K/V there: the block decode is handed the slot's block-table row shifted + past those pages (``dflash_rebased_ctx_rows``) and a context length + shortened by the same amount, so a tail shorter than the drafter's + window attends only to itself. ``start`` is rounded up to a page + boundary; the tail tokens inside that page stay stored but unread. + """ + if start == cur or self._ctx_reuse_addressable: + return None + return -(-start // self._ctx_page_size) + def _assign_slot(self, req_id: int, reset: bool = False, updates: Optional[dict] = None): """Get (or refresh) this request's context slot; None when none is free. @@ -1417,30 +1475,51 @@ def _advance_ctx_len(self, slots: torch.Tensor, num_accepted: torch.Tensor) -> N self._ctx_len.clamp_(max=self._max_ctx) self._ctx_len[self._dummy_slot].zero_() - def _write_ctx_len(self, updates: dict[int, int]) -> None: - """Apply a slot-to-length mapping to _ctx_len in one async scatter.""" - if not updates: + def _write_ctx_len( + self, updates: dict[int, int], base_updates: Optional[dict[int, int]] = None + ) -> None: + """Apply slot-to-length (and slot-to-base-page) mappings in one async + scatter each. A slot reset to length 0 also drops its base pages, and + that reset wins over a base recorded for the same slot in + ``base_updates`` (a chunk's base is recorded before the overflow check + that may reset the slot).""" + if not updates and not base_updates: return + bases = dict(base_updates or {}) + bases.update({slot: 0 for slot, value in updates.items() if value == 0}) for slot, value in updates.items(): # Update host side mirror self._ctx_len_host[slot] = value + for slot, value in bases.items(): + self._ctx_base_host[slot] = value - slots, values = zip(*updates.items()) pinned = prefer_pinned() - slots_pinned = torch.tensor(slots, dtype=torch.long, pin_memory=pinned) - vals_pinned = torch.tensor(values, dtype=torch.long, pin_memory=pinned) - self._ctx_len.index_copy_( - 0, - slots_pinned.to("cuda", non_blocking=True), - vals_pinned.to("cuda", non_blocking=True), - ) + for target, mapping in ((self._ctx_len, updates), (self._ctx_base_pages, bases)): + if not mapping: + continue + slots, values = zip(*mapping.items()) + slots_pinned = torch.tensor(slots, dtype=torch.long, pin_memory=pinned) + vals_pinned = torch.tensor(values, dtype=torch.long, pin_memory=pinned) + target.index_copy_( + 0, + slots_pinned.to(target.device, non_blocking=True), + vals_pinned.to(target.device, non_blocking=True), + ) def _restore_ctx_len_host(self) -> None: """Roll the host shadows back to their pre-forward snapshot.""" if self._saved_ctx_len_host is not None: self._ctx_len_host = list(self._saved_ctx_len_host) + if self._saved_ctx_base_host is not None: + self._ctx_base_host = list(self._saved_ctx_base_host) if self._saved_req_ctx_pos is not None: self._req_ctx_pos = dict(self._saved_req_ctx_pos) + def _restore_ctx_len_device(self) -> None: + """Roll _ctx_len and _ctx_base_pages back to their pre-forward snapshot.""" + self._ctx_len.copy_(self._saved_ctx_len) + if self._saved_ctx_base is not None: + self._ctx_base_pages.copy_(self._saved_ctx_base) + def _store_prefill_context( self, draft_model, @@ -1480,6 +1559,7 @@ def _store_prefill_context( # the same values, but reading it costs a D2H sync per request. host_pos = spec_metadata.host_position_ids ctx_len_updates = {} + ctx_base_updates = {} offset = 0 num_contexts = attn_metadata.num_contexts # A context write addresses pages through row i of the block table, so @@ -1518,11 +1598,11 @@ def _store_prefill_context( slot = self._req_to_slot[req_id] cur = ctx_len_updates.get(slot, self._ctx_len_host[slot]) cap = self._max_ctx if ctx_alloc is None else min(self._max_ctx, ctx_alloc[i]) - if cur == 0 and first_pos > 0 and self._ctx_reuse_addressable: - # Reuse let the target skip [0, first_pos), so only the tail - # reaches the capture buffer. The prefix K/V are still in the - # paired pool: precompute_context_kv is per-token in (hidden, p). - cur = first_pos + start = self._ctx_store_start(cur, first_pos) + base = self._ctx_base_for_store(cur, start) + if base is not None: + ctx_base_updates[slot] = base + cur = start if cur + slen > cap: # Request-level, like the no-free-slots path above: truncating # would silently draft from a stale prefix, but killing the @@ -1581,7 +1661,7 @@ def _store_prefill_context( self._ctx_v_buf[slot, :, cur:end] = chunk_v.permute(1, 0, 2, 3) offset += slen - self._write_ctx_len(ctx_len_updates) + self._write_ctx_len(ctx_len_updates, ctx_base_updates) def _ensure_spec_dec_state_restored(self, attn_metadata, spec_metadata): # Restore first (in warmup mode kv_lens_cuda was saved and comes back @@ -1601,7 +1681,7 @@ def _ensure_spec_dec_state_restored(self, attn_metadata, spec_metadata): ): # A failed forward must not keep this iteration's in-place # _ctx_len updates: roll back to the pre-forward snapshot. - self._ctx_len.copy_(self._saved_ctx_len) + self._restore_ctx_len_device() self._restore_ctx_len_host() self._ctx_len_restore_pending = False @@ -1657,6 +1737,8 @@ def _forward_impl( # mutate _ctx_len until replay. self._saved_ctx_len = self._ctx_len.clone() self._saved_ctx_len_host = list(self._ctx_len_host) + self._saved_ctx_base = self._ctx_base_pages.clone() + self._saved_ctx_base_host = list(self._ctx_base_host) self._saved_req_ctx_pos = dict(self._req_ctx_pos) self._ctx_len_restore_pending = True @@ -1854,7 +1936,7 @@ def _forward_impl( # Restore context lengths after warmup; real runs keep the updates. if is_warmup: - self._ctx_len.copy_(self._saved_ctx_len) + self._restore_ctx_len_device() self._restore_ctx_len_host() self._ctx_len_restore_pending = False @@ -2188,6 +2270,7 @@ def prepare_1st_drafter_inputs( self._advance_ctx_len(slots, gen_num_accepted_long) num_ctx_per_req_t = self._ctx_len[slots] + ctx_page_table = self._ctx_page_table if self._ctx_block_tables is not None: # The write above clamps columns to the request's allocation, so # a context that outruns it has its tail written over the last @@ -2197,6 +2280,17 @@ def prepare_1st_drafter_inputs( self._ctx_block_counts, self._ctx_page_size, block_size ) num_ctx_per_req_t = torch.minimum(num_ctx_per_req_t, allocated[gen_rows_out]) + # Pages below a slot's base hold no drafter K/V (a prefix the + # target reused): the read numbers from the base instead, see + # _ctx_base_for_store. Writes above keep the absolute table. + base_pages = self._ctx_base_pages[slots] + num_ctx_per_req_t = (num_ctx_per_req_t - base_pages * self._ctx_page_size).clamp_( + min=0 + ) + self._ctx_block_tables_read[gen_rows_out] = dflash_rebased_ctx_rows( + self._ctx_block_tables[gen_rows_out], base_pages + ) + ctx_page_table = self._ctx_block_tables_read noise_embedding = noise_embed_2d query_positions = query_position_ids.long() @@ -2213,6 +2307,11 @@ def prepare_1st_drafter_inputs( slots = torch.empty(0, dtype=torch.long, device="cuda") gen_rows_out = slots bonus = torch.empty(0, dtype=torch.long, device="cuda") + ctx_page_table = ( + self._ctx_block_tables_read + if self._ctx_block_tables is not None + else self._ctx_page_table + ) return { "lilicorr_anchor": lilicorr_anchor, @@ -2228,9 +2327,6 @@ def prepare_1st_drafter_inputs( # token. The dspark Markov chain conditions its first step on it. "first_prev_tokens": bonus, "ctx_kv_cache": self._ctx_kv_buf, - "ctx_page_table": ( - self._ctx_block_tables - if self._ctx_block_tables is not None - else self._ctx_page_table - ), + # The manager-bound table is the read view rebased per slot above. + "ctx_page_table": ctx_page_table, } diff --git a/tests/unittest/_torch/executor/kv_cache/test_kv_cache_estimation.py b/tests/unittest/_torch/executor/kv_cache/test_kv_cache_estimation.py index d706fb826687..24cd7cc8833d 100644 --- a/tests/unittest/_torch/executor/kv_cache/test_kv_cache_estimation.py +++ b/tests/unittest/_torch/executor/kv_cache/test_kv_cache_estimation.py @@ -10,10 +10,12 @@ share, not all copies. """ +import math from dataclasses import dataclass from types import SimpleNamespace from unittest.mock import Mock, patch +import numpy as np import pytest import torch @@ -30,12 +32,18 @@ get_mla_context_workspace_kv_len_cap, ) from tensorrt_llm._torch.pyexecutor.config_utils import get_layer_attention_window -from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 +from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import ( + KVCacheManagerV2, + _get_generation_kv_capacity, + max_admissible_sequences, +) from tensorrt_llm._torch.pyexecutor.model_engine import PyTorchModelEngine from tensorrt_llm._torch.pyexecutor.resource_manager import KVCacheManager from tensorrt_llm._torch.speculative.interface import SpeculativeDecodingMode from tensorrt_llm.inputs.multimodal import MultimodalParams from tensorrt_llm.llmapi.llm_args import ( + DFlashDecodingConfig, + DSparkDecodingConfig, KvCacheConfig, MTPDecodingConfig, MultimodalConfig, @@ -1471,6 +1479,423 @@ def test_separate_one_model_draft_normalizes_target_pool_ratio() -> None: assert creator._kv_cache_config.pool_ratio == target_pool_ratio +def _dflash_draft_model(uses_worker_backend: bool, backend: str) -> SimpleNamespace: + """The two attributes KvCacheCreator._draft_tokens_per_block reads off the + built draft model: whether it runs the shared DFlash op sets, and the + resolved attention backend.""" + return SimpleNamespace( + _uses_worker_attention_backend=uses_worker_backend, + dflash_attention_backend=backend, + ) + + +def _page_size_creator(tokens_per_block: int, draft_model) -> KvCacheCreator: + creator = object.__new__(KvCacheCreator) + creator._tokens_per_block = tokens_per_block + creator._model_engine = Mock(_max_cuda_graph_batch_size=4) + creator._model_engine.model = SimpleNamespace(draft_model=draft_model) + return creator + + +class TestDraftPoolPageSize: + """A standalone DFlash/DSpark drafter on the TRTLLM backend reads its pool + with trtllm-gen, which accepts fewer page sizes than a target may pin + (MiniMax-M3 MSA: 128), so the separate draft pool is carved at the largest + size the drafter accepts and joint block reuse is declined when the two + pools differ.""" + + def test_premise_trtllm_gen_page_sizes(self): + from tensorrt_llm._torch.attention.backends.fmha.flashinfer_trtllm_gen import ( + FlashInferTrtllmGenFmha, + ) + + supported = FlashInferTrtllmGenFmha.SUPPORTED_TOKENS_PER_BLOCK + assert 128 not in supported + assert max(size for size in supported if size < 128) == 64 + + @pytest.mark.parametrize( + "tokens_per_block, draft_model, expected", + [ + (128, _dflash_draft_model(True, "TRTLLM"), 64), + (128, _dflash_draft_model(True, "VANILLA"), 128), + (128, _dflash_draft_model(True, "FA4"), 128), + # The MLA DSpark drafter brings its own kernels under the same name. + (128, _dflash_draft_model(False, "TRTLLM"), 128), + # No drafter at all (MTP / Eagle3 draft layers). + (128, None, 128), + (64, _dflash_draft_model(True, "TRTLLM"), 64), + (32, _dflash_draft_model(True, "TRTLLM"), 32), + ], + ) + def test_draft_tokens_per_block(self, tokens_per_block, draft_model, expected): + creator = _page_size_creator(tokens_per_block, draft_model) + assert creator._draft_tokens_per_block() == expected + + def test_separate_draft_manager_is_carved_at_the_draft_page_size(self): + creator = _page_size_creator(128, _dflash_draft_model(True, "TRTLLM")) + creator._kv_cache_config = KvCacheConfig(max_attention_window=None) + creator._max_seq_len = 9472 + creator._max_batch_size = 8 + creator._max_num_tokens = 9472 + creator._max_beam_width = 1 + creator._kv_connector_manager = None + creator._skip_est = False + creator._execution_stream = None + creator._is_disagg = False + creator._cache_transceiver_config = None + creator._mapping = Mock() + creator._mapping.has_cp_helix.return_value = False + creator._speculative_config = Mock() + creator._disable_overlap_scheduler = False + + effective_draft_config = Mock() + effective_draft_config.pretrained_config.torch_dtype = "bfloat16" + effective_draft_config.sparse_attention_config = None + + with ( + patch.object(creator, "_get_num_draft_layers", return_value=5), + patch.object(creator, "_get_one_model_draft_layer_mask", return_value=[True] * 5), + patch.object( + creator, + "_get_effective_draft_config", + return_value=effective_draft_config, + ), + patch.object(creator, "_enable_kv_cache_stats", return_value=False), + patch.object( + creator, + "_validate_or_fallback_kv_cache_manager_v2", + return_value=KVCacheManagerV2, + ), + patch( + "tensorrt_llm._torch.pyexecutor._util._derive_draft_max_attention_window", + return_value=None, + ), + patch( + "tensorrt_llm._torch.pyexecutor._util.get_kv_cache_manager_cls", + return_value=KVCacheManagerV2, + ), + patch( + "tensorrt_llm._torch.pyexecutor._util._create_kv_cache_manager", + return_value=Mock(), + ) as create_manager, + ): + creator._create_one_model_draft_kv_cache_manager( + creator._max_seq_len, estimating_kv_cache=False + ) + + assert create_manager.call_args.kwargs["tokens_per_block"] == 64 + assert create_manager.call_args.kwargs["is_draft"] is True + # The target's own page size is untouched. + assert creator._tokens_per_block == 128 + + @pytest.mark.parametrize("tokens_per_block, expected", [(128, False), (64, True)]) + def test_joint_reuse_needs_one_page_size(self, tokens_per_block, expected): + creator = _page_size_creator(tokens_per_block, _dflash_draft_model(True, "TRTLLM")) + creator._is_kv_cache_manager_v2 = True + creator._kv_cache_manager_cls = KVCacheManagerV2 + # Standalone DSpark: lookahead 0, so only the page sizes decide. + creator._speculative_config = DSparkDecodingConfig(max_draft_len=4) + assert creator._joint_reuse_supported() is expected + + +class _StandaloneDrafterConfig: + """KV geometry of a five-layer GQA drafter with one KV head per TP rank + (4 KV heads over TP4, head_dim 128, bf16): 512 bytes per layer and token, + 2560 per token.""" + + quant_config = None + pretrained_config = SimpleNamespace( + hidden_size=6144, + num_attention_heads=64, + num_key_value_heads=4, + head_dim=128, + layer_types=["sliding_attention"] * 5, + sliding_window=4096, + ) + + def get_num_attention_layers(self) -> int: + return 5 + + +def _standalone_drafter_cost(max_attention_window) -> CacheCost: + mapping = Mock(enable_attention_dp=False, tp_size=4) + mapping.pp_layers.return_value = [0, 1, 2, 3, 4] + return CacheCost.from_raw( + KVCacheManagerV2.get_cache_size_per_token( + _StandaloneDrafterConfig(), + mapping, + tokens_per_block=64, + max_seq_len=1048576, + max_batch_size=8, + max_num_tokens=16384, + kv_cache_config=KvCacheConfig( + max_attention_window=max_attention_window, max_util_for_resume=0.95 + ), + spec_config=DSparkDecodingConfig(max_draft_len=7), + is_draft=True, + ) + ) + + +class TestStandaloneDraftPoolFunding: + """The standalone DFlash/DSpark drafter's pool keeps the drafter's attention + windows, so a mirror holds its window plus the chunk it is computing and the + pool is funded for max_batch_size such mirrors as a fixed cost, with or + without block reuse. A drafter without a window keeps whole contexts: the + target's per-token funding bounds the mirrors without block reuse, and with + it the pool is funded for max_batch_size full-length mirrors.""" + + @staticmethod + def _creator( + spec_config, + *, + max_seq_len, + max_batch_size, + max_num_tokens=16384, + disable_overlap_scheduler=False, + is_disagg=False, + mapping=None, + ): + creator = object.__new__(KvCacheCreator) + creator._speculative_config = spec_config + creator._max_seq_len = max_seq_len + creator._max_batch_size = max_batch_size + creator._max_num_tokens = max_num_tokens + creator._disable_overlap_scheduler = disable_overlap_scheduler + creator._is_disagg = is_disagg + creator._mapping = mapping if mapping is not None else Mapping() + return creator + + @staticmethod + def _fixed_cost(bytes_per_token, tokens_per_mirror, num_mirrors, tokens_per_block=64): + pages = num_mirrors * math.ceil(tokens_per_mirror / tokens_per_block) + slots = math.ceil(pages / float(np.float32(0.95))) + return CacheCost(slope=0, intercept=slots * tokens_per_block * bytes_per_token) + + @pytest.mark.parametrize( + "disable_overlap_scheduler, is_disagg, mapping, expected", + [ + # Overlap scheduler: requests retiring from iteration n keep their + # pages while iteration n+1 admits new ones in their place. + (False, False, Mapping(), 16), + (True, False, Mapping(), 8), + (True, True, Mapping(), 16), + # Pipeline parallelism: one batch per stage, no retiring overlap. + (False, False, Mapping(world_size=2, pp_size=2), 16), + ], + ) + def test_mirrors_follow_the_manager_admissible_sequences( + self, disable_overlap_scheduler, is_disagg, mapping, expected + ): + assert ( + max_admissible_sequences( + 8, mapping, is_disagg=is_disagg, disable_overlap_scheduler=disable_overlap_scheduler + ) + == expected + ) + spec = DSparkDecodingConfig(max_draft_len=7) + creator = self._creator( + spec, + max_seq_len=65536, + max_batch_size=8, + disable_overlap_scheduler=disable_overlap_scheduler, + is_disagg=is_disagg, + mapping=mapping, + ) + _, headroom = _get_generation_kv_capacity(spec, is_draft=True) + with patch.object(creator, "_get_draft_max_attention_window", return_value=[4096] * 5): + cost = creator._standalone_draft_pool_cost( + CacheCost(slope=2560), KvCacheConfig(max_util_for_resume=0.95), 64 + ) + assert cost == self._fixed_cost(2560, 4096 + 16384 + 2 * headroom + 64, expected) + + def test_per_token_cost_comes_from_the_windowless_manager_cost(self): + # Through the real KVCacheManagerV2.get_cache_size_per_token: costed + # with its windows the drafter's pool is a fixed cost with slope 0 + # (a funding that multiplies by it funds the pool with 0 bytes); the + # same layers costed without windows give the + # 2560 bytes one token of drafter K/V occupies at TP4. + mapping = Mock(enable_attention_dp=False, tp_size=4) + mapping.pp_layers.return_value = [0, 1, 2, 3, 4] + creator = self._creator( + DSparkDecodingConfig(max_draft_len=7), max_seq_len=1048576, max_batch_size=8 + ) + creator._kv_cache_config = KvCacheConfig(max_util_for_resume=0.95) + creator._tokens_per_block = 128 + windowed = KvCacheConfig(max_attention_window=[4096] * 5, max_util_for_resume=0.95) + windowed_cost = creator._per_manager_cache_cost( + KVCacheManagerV2, + _StandaloneDrafterConfig(), + windowed, + mapping=mapping, + tokens_per_block=64, + is_draft=True, + ) + assert windowed_cost.slope == 0 and windowed_cost.intercept > 0 + per_token = creator._standalone_draft_per_token_cost( + KVCacheManagerV2, _StandaloneDrafterConfig(), windowed, mapping, 64 + ) + assert per_token == CacheCost(slope=2560, intercept=0) + with patch.object(creator, "_get_draft_max_attention_window", return_value=[4096] * 5): + cost = creator._standalone_draft_pool_cost( + per_token, KvCacheConfig(enable_block_reuse=True, max_util_for_resume=0.95), 64 + ) + assert cost.slope == 0 and cost.intercept > 2560 * 64 * 16 * 4096 // 64 + + def test_pool_cost_rejects_a_cost_that_is_not_a_per_token_slope(self): + creator = self._creator( + DSparkDecodingConfig(max_draft_len=7), max_seq_len=1048576, max_batch_size=8 + ) + for bad in ( + CacheCost(slope=0, intercept=826 * 64 * 2560), + CacheCost(slope=2560, intercept=1), + ): + with pytest.raises(ValueError, match="pure slope"): + creator._standalone_draft_pool_cost(bad, KvCacheConfig(), 64) + + @pytest.mark.parametrize("enable_block_reuse", [True, False]) + def test_windowed_mirrors_are_funded_for_window_plus_chunk(self, enable_block_reuse): + # TP2: 5 layers x 4 KV heads x 128 x bf16 K+V = 5120 B/token/rank, + # max_batch_size 30 at 1M max_seq_len and 16384-token chunks, overlap + # scheduler on (60 admissible mirrors). 30 full-length mirrors alone + # would cost ~160 GiB per rank, more than the whole KV budget; 60 + # window-plus-chunk mirrors cost a few GiB. + spec = DSparkDecodingConfig(max_draft_len=3) + creator = self._creator(spec, max_seq_len=1048576, max_batch_size=30) + _, headroom = _get_generation_kv_capacity(spec, is_draft=True) + config = KvCacheConfig(enable_block_reuse=enable_block_reuse, max_util_for_resume=0.95) + with patch.object(creator, "_get_draft_max_attention_window", return_value=[4096] * 5): + cost = creator._standalone_draft_pool_cost(CacheCost(slope=5120), config, 64) + tokens_per_mirror = 4096 + 16384 + 2 * headroom + 64 + assert cost == self._fixed_cost(5120, tokens_per_mirror, 60) + assert 6 < cost.intercept / 2**30 < 8 + # The window is the drafter's, read from its own derived layout, so + # the cost does not move with the context length. + creator._max_seq_len = 65536 + with patch.object(creator, "_get_draft_max_attention_window", return_value=[4096] * 5): + assert creator._standalone_draft_pool_cost(CacheCost(slope=5120), config, 64) == cost + + def test_windowed_funding_covers_the_live_page_geometry(self): + # TP4, batch 8, 1M context, K=7, 16384-token chunks, overlap scheduler + # on: the windowed pool holds up to 841 live pages, against the 826 the + # manager's own SWA cost funds + # (test_premise_windowed_funding_is_a_fixed_cost). + spec = DSparkDecodingConfig(max_draft_len=7) + creator = self._creator(spec, max_seq_len=1048576, max_batch_size=8) + _, headroom = _get_generation_kv_capacity(spec, is_draft=True) + with patch.object(creator, "_get_draft_max_attention_window", return_value=[4096] * 5): + cost = creator._standalone_draft_pool_cost( + CacheCost(slope=2560), KvCacheConfig(enable_block_reuse=True), 64 + ) + assert cost == self._fixed_cost(2560, 4096 + 16384 + 2 * headroom + 64, 16) + assert cost.intercept // (64 * 2560) > 841 + assert cost.intercept / 2**30 < 1 + + def test_windowless_drafter_under_reuse_funds_max_batch_size_full_length_mirrors(self): + spec = DSparkDecodingConfig(max_draft_len=7) + creator = self._creator(spec, max_seq_len=1048576, max_batch_size=8) + _, headroom = _get_generation_kv_capacity(spec, is_draft=True) + with patch.object(creator, "_get_draft_max_attention_window", return_value=None): + cost = creator._standalone_draft_pool_cost( + CacheCost(slope=2560), + KvCacheConfig(enable_block_reuse=True, max_util_for_resume=0.95), + 64, + ) + # 16 admissible mirrors (overlap scheduler) x 1M tokens at 2560 B: + # about 42 GiB, which is why a windowless drafter cannot serve long + # context under block reuse at any useful batch size. + assert cost == self._fixed_cost(2560, 1048576 + headroom, 16) + assert 41 < cost.intercept / 2**30 < 43 + + def test_windowless_drafter_without_reuse_keeps_the_per_token_funding(self): + creator = self._creator( + DSparkDecodingConfig(max_draft_len=7), max_seq_len=1048576, max_batch_size=8 + ) + per_token = CacheCost(slope=2560) + with patch.object(creator, "_get_draft_max_attention_window", return_value=None): + cost = creator._standalone_draft_pool_cost( + per_token, KvCacheConfig(enable_block_reuse=False), 64 + ) + assert cost == per_token + + @pytest.mark.parametrize( + "windows, max_seq_len, expected", + [ + ([4096] * 5, 1048576, 4096), + ([4096], 1048576, 4096), + # A layer attending to the whole context makes the mirror full-length. + ([4096, 1048576], 1048576, None), + ([4096] * 5, 4096, None), + (None, 1048576, None), + ], + ) + def test_standalone_draft_mirror_window(self, windows, max_seq_len, expected): + creator = self._creator( + DSparkDecodingConfig(max_draft_len=7), max_seq_len=max_seq_len, max_batch_size=8 + ) + with patch.object(creator, "_get_draft_max_attention_window", return_value=windows): + window = creator._standalone_draft_mirror_window(KvCacheConfig(enable_block_reuse=True)) + assert window == expected + + def test_shared_prefix_mirrors_fit_the_reuse_funding_not_the_per_token_one(self): + # Two requests sharing a 32768-token prefix, 128 new tokens and 7 + # drafts each: the target holds the prefix once, the unpaired mirrors + # hold it twice. + prefix, tail, draft = 32768, 128, 7 + target_tokens = prefix + 2 * (tail + draft) + mirror_tokens = 2 * (prefix + tail + draft) + per_token = CacheCost(slope=2560) + assert ( + per_token.tokens_for_budget(per_token.bytes_for_tokens(target_tokens)) < mirror_tokens + ) + bound = KvCacheCreator.standalone_draft_pool_worst_case_cost( + 2560, 64, prefix + tail + draft, 2, 1.0 + ) + assert bound.intercept // 2560 >= mirror_tokens + + def test_premise_windowed_funding_is_a_fixed_cost(self): + # Five 4096-token windows: 66 pages per request x 8 requests plus one + # max_num_tokens of context, rounded to 64-token slots at the resume + # watermark. 826 slots = 0.13 GiB, independent of max_seq_len. + assert _standalone_drafter_cost([4096] * 5) == CacheCost(slope=0, intercept=826 * 2560 * 64) + + def test_full_context_funding_is_per_token(self): + assert _standalone_drafter_cost(None) == CacheCost(slope=2560, intercept=0) + + @pytest.mark.parametrize( + "spec_config", + [ + DSparkDecodingConfig(max_draft_len=7), + DFlashDecodingConfig(max_draft_len=7), + MTPDecodingConfig(max_draft_len=1), + ], + ) + def test_draft_pool_keeps_the_drafter_windows(self, spec_config): + # The standalone drafter's pool trims below the window like any other; + # only the estimation pass carves without windows. + creator = object.__new__(KvCacheCreator) + creator._speculative_config = spec_config + with patch.object(creator, "_get_draft_max_attention_window", return_value=[4096] * 5): + draft_kv_config = creator._get_one_model_draft_kv_cache_config(KvCacheConfig(), 1048576) + estimation_config = creator._get_one_model_draft_kv_cache_config( + KvCacheConfig(), 1048576, estimating_kv_cache=True + ) + assert draft_kv_config.max_attention_window == [4096] * 5 + assert estimation_config.max_attention_window is None + + def test_full_context_share_of_the_budget(self): + # The price of the per-token funding: 2560 of every 8640 + 2560 bytes + # per token, 22.9 % of a 181 GiB GPU budget (41.4 GiB). + creator = object.__new__(KvCacheCreator) + total = 181 * 2**30 + target_budget, draft_budget = creator._compute_draft_budget_shares( + total, CacheCost(slope=8640), CacheCost(slope=2560) + ) + assert draft_budget == total * 2560 // 11200 + assert target_budget + draft_budget == total + assert 41.3 < draft_budget / 2**30 < 41.5 + + # --------------------------------------------------------------------------- # Per-layer attention windows derived from a mixed layer_types schedule # --------------------------------------------------------------------------- diff --git a/tests/unittest/_torch/executor/kv_cache/test_kvcm2_integration.py b/tests/unittest/_torch/executor/kv_cache/test_kvcm2_integration.py index 75e32a232825..4aa22534de38 100644 --- a/tests/unittest/_torch/executor/kv_cache/test_kvcm2_integration.py +++ b/tests/unittest/_torch/executor/kv_cache/test_kvcm2_integration.py @@ -24,6 +24,7 @@ import torch from tensorrt_llm._torch.distributed.communicator import Distributed, ReduceOp +from tensorrt_llm._torch.pyexecutor._util import KvCacheCreator from tensorrt_llm._torch.pyexecutor.engine.runners.decoder import DecoderRunner from tensorrt_llm._torch.pyexecutor.kv_cache import kv_cache_manager_v2 as kv_cache_v2_module from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import ( @@ -31,6 +32,7 @@ KVCacheManagerV2, Role, _extend_swa_windows_for_reuse, + _get_generation_kv_capacity, _KVCacheManagerInitStatus, _swa_endpoint_priority, _sync_kv_cache_manager_init_status, @@ -1465,6 +1467,7 @@ def test_draft_manager_keeps_shared_progress_across_context_and_generation() -> manager.is_draft = True manager.enable_block_reuse = True manager.enable_joint_kv_cache_reuse = True + manager._mirror_uses_draft_view = True manager.kv_cache_map = {request.py_request_id: kv_cache} observed_progress_views = [] @@ -1490,6 +1493,546 @@ def resume_and_restore(request_id, current_cache): manager._required_gen_capacity.assert_not_called() +@pytest.mark.parametrize( + ("mirror_history", "expected_history"), + [ + # A fresh mirror for a request whose target prefix was reused: the + # history jumps to the chunk start so no page below the window is + # allocated for [0, position). + (0, 8192), + # A later chunk of the same request: the history follows the chunk. + (4096, 8192), + # Joint reuse already advanced the history past the chunk start; a + # resize may not decrease it. + (8704, 8704), + ], +) +def test_unpaired_one_engine_draft_mirror_follows_the_target_chunk( + mirror_history: int, expected_history: int +) -> None: + position, chunk, draft_len = 8192, 1024, 3 + request = LlmRequest( + request_id=41, + max_new_tokens=4, + input_tokens=[0] * (position + chunk), + sampling_config=SamplingConfig(1), + is_streaming=False, + draft_tokens=[1] * draft_len, + ) + request.state = LlmRequestState.CONTEXT_INIT + request.context_current_position = position + request.context_chunk_size = chunk + assert request.is_last_context_chunk + + kv_cache = Mock(is_active=True, capacity=mirror_history, history_length=mirror_history) + kv_cache.resize.return_value = True + manager = object.__new__(KVCacheManagerV2) + manager.is_draft = True + manager.enable_joint_kv_cache_reuse = False + manager._mirror_uses_draft_view = False + manager.num_extra_kv_tokens = 0 + manager.kv_cache_map = {request.py_request_id: kv_cache} + manager._resume_and_restore = Mock(return_value=True) + + batch = ScheduledRequests() + batch.context_requests_last_chunk.append(request) + manager._prepare_draft_resources(batch) + + kv_cache.resize.assert_called_once_with(position + chunk + draft_len, expected_history) + assert not request.use_draft_model + + +def test_unpaired_two_engine_draft_mirror_reads_the_draft_request_view() -> None: + """The draft view starts at position 0 with the whole prompt as its chunk + until a separate draft engine advances it, so a two-engine mirror sizes + its cache for the prompt at once.""" + position, chunk, draft_len = 8192, 1024, 3 + request = LlmRequest( + request_id=42, + max_new_tokens=4, + input_tokens=[0] * (position + chunk), + sampling_config=SamplingConfig(1), + is_streaming=False, + draft_tokens=[1] * draft_len, + ) + request.state = LlmRequestState.CONTEXT_INIT + request.context_current_position = position + request.context_chunk_size = chunk + + kv_cache = Mock(is_active=True, capacity=0, history_length=0) + kv_cache.resize.return_value = True + manager = object.__new__(KVCacheManagerV2) + manager.is_draft = True + manager.enable_joint_kv_cache_reuse = False + manager._mirror_uses_draft_view = True + manager.num_extra_kv_tokens = 0 + manager.kv_cache_map = {request.py_request_id: kv_cache} + manager._resume_and_restore = Mock(return_value=True) + + batch = ScheduledRequests() + batch.context_requests_last_chunk.append(request) + manager._prepare_draft_resources(batch) + + kv_cache.resize.assert_called_once_with(position + chunk + draft_len, 0) + assert not request.use_draft_model + + +@pytest.mark.parametrize("phase", ["context", "generation"]) +def test_unpaired_draft_mirror_that_cannot_grow_raises_with_the_pool_diagnostic(phase: str) -> None: + """Two requests whose target prefixes are shared need two full mirrors in + the unpaired draft pool. When the second one finds no page, the iteration + fails explicitly: the mirror left at its size has no storage for the K/V + the drafter writes next, and there is no per-request way to skip drafting. + The error carries the pool diagnostic.""" + position, chunk, draft_len = 8192, 1024, 3 + + def make_request(request_id: int) -> LlmRequest: + request = LlmRequest( + request_id=request_id, + max_new_tokens=4, + input_tokens=[0] * (position + chunk), + sampling_config=SamplingConfig(1), + is_streaming=False, + draft_tokens=[1] * draft_len, + ) + if phase == "context": + request.state = LlmRequestState.CONTEXT_INIT + request.context_current_position = position + request.context_chunk_size = chunk + else: + request.state = LlmRequestState.GENERATION_IN_PROGRESS + return request + + requests = [make_request(51), make_request(52)] + caches = { + 51: Mock(is_active=True, capacity=position, history_length=position, beam_width=1), + 52: Mock(is_active=True, capacity=position, history_length=position, beam_width=1), + } + caches[51].resize.return_value = True + caches[52].resize.return_value = False + manager = object.__new__(KVCacheManagerV2) + manager.is_draft = True + manager.enable_joint_kv_cache_reuse = False + manager._mirror_uses_draft_view = False + manager.num_extra_kv_tokens = 0 + manager.kv_cache_map = dict(caches) + manager._allocated_draft_lens = set() + manager._resume_and_restore = Mock(return_value=True) + manager._required_gen_capacity = Mock(return_value=position + 1 + draft_len) + manager._draft_pool_diagnostic = Mock(return_value=" [draft pool: full]") + manager.kv_cache_type = CacheType.SELF + manager.max_beam_width = 1 + + batch = ScheduledRequests() + if phase == "context": + batch.context_requests_last_chunk.extend(requests) + expected_resize = call(position + chunk + draft_len, position) + expected_capacity = position + chunk + draft_len + else: + batch.generation_requests.extend(requests) + expected_resize = call(position + 1 + draft_len) + expected_capacity = position + 1 + draft_len + with pytest.raises(RuntimeError) as excinfo: + manager._prepare_draft_resources(batch) + + assert caches[51].resize.call_args_list == [expected_resize] + assert caches[52].resize.call_args_list == [expected_resize] + message = str(excinfo.value) + assert f"{phase} resize failed for request 52" in message + assert f"could not resize to {expected_capacity} tokens" in message + assert "[draft pool: full]" in message + + +def test_exhausted_pool_fails_explicitly_on_an_empty_fresh_mirror() -> None: + """A fresh mirror whose first resize fails holds no page at all. Nothing + the drafter writes for that request has storage, so the manager raises + before the forward, with a diagnostic that shows the empty mirror and how + full the pool was.""" + prompt, draft_len = 1024, 3 + request = LlmRequest( + request_id=61, + max_new_tokens=4, + input_tokens=[0] * prompt, + sampling_config=SamplingConfig(1), + is_streaming=False, + draft_tokens=[1] * draft_len, + ) + request.state = LlmRequestState.CONTEXT_INIT + request.context_current_position = 0 + request.context_chunk_size = prompt + + occupant = Mock(is_active=True, capacity=1216, history_length=0, num_blocks=38) + fresh = Mock(is_active=True, capacity=0, history_length=0, num_blocks=0) + fresh.resize.return_value = False + manager = object.__new__(KVCacheManagerV2) + manager.is_draft = True + manager.enable_joint_kv_cache_reuse = False + manager._mirror_uses_draft_view = False + manager.num_extra_kv_tokens = 0 + manager.kv_cache_map = {60: occupant, 61: fresh} + manager.max_attention_window_vec = [2048] + manager._gpu_max_tokens = 1216 + manager._resume_and_restore = Mock(return_value=True) + + batch = ScheduledRequests() + batch.context_requests_last_chunk.append(request) + with pytest.raises(RuntimeError) as excinfo: + manager._prepare_draft_resources(batch) + + fresh.resize.assert_called_once_with(prompt + draft_len, 0) + message = str(excinfo.value) + assert ( + f"context resize failed for request 61: could not resize to {prompt + draft_len} tokens" + in message + ) + assert "2 live caches, capacity 1216 tokens in 38 allocated blocks" in message + assert "61: capacity=0 history=0 blocks=0" in message + assert "gpu_max_tokens=1216" in message + + +def test_exhausted_real_pool_raises_before_an_empty_mirror_reaches_the_drafter() -> None: + """On a real draft pool with room for one mirror, the second request's + fresh mirror cannot grow. _prepare_draft_resources raises instead of + handing the drafter a mirror without pages: the context length such a + mirror could advertise is zero, and the K/V the block decode appends + beyond it would have no page to land on.""" + if not torch.cuda.is_available(): + pytest.skip("requires CUDA") + torch.cuda.init() + from tensorrt_llm._torch.speculative.dflash import dflash_allocated_ctx_limit + + spec = DFlashDecodingConfig(max_draft_len=4, speculative_model="draft") + # 32 KiB per token (8 layers, 8 KV heads of 128, fp16), 1 MiB pages: a 40 MiB + # budget holds one 1024-token mirror (33 pages with the generation headroom) + # and not two, with margin on both sides of the GPU tier's 2 MiB rounding. + num_layers, num_kv_heads, head_dim = 8, 8, 128 + tokens_per_block, max_seq_len, batch_size = 32, 2048, 2 + prompt, draft_len = 1024, spec.max_draft_len + budget = 40 << 20 + config = KvCacheConfig(enable_block_reuse=False, max_gpu_total_bytes=budget) + manager = KVCacheManagerV2( + config, + CacheType.SELF, + num_layers=num_layers, + num_kv_heads=num_kv_heads, + head_dim=head_dim, + tokens_per_block=tokens_per_block, + max_seq_len=max_seq_len, + max_batch_size=batch_size, + max_num_tokens=max_seq_len, + mapping=Mapping(), + dtype=DataType.HALF, + spec_config=spec, + is_draft=True, + ) + + def make_request(request_id: int) -> LlmRequest: + request = LlmRequest( + request_id=request_id, + max_new_tokens=4, + input_tokens=[0] * prompt, + sampling_config=SamplingConfig(1), + is_streaming=False, + draft_tokens=[1] * draft_len, + ) + request.state = LlmRequestState.CONTEXT_INIT + request.context_current_position = 0 + request.context_chunk_size = prompt + return request + + stream = torch.cuda.current_stream().cuda_stream + caches = {} + try: + for request_id in (71, 72): + cache = manager.impl.create_kv_cache() + caches[request_id] = cache + assert cache.resume(stream) + manager.kv_cache_map.update(caches) + + batch = ScheduledRequests() + batch.context_requests_last_chunk.extend([make_request(71), make_request(72)]) + with pytest.raises(RuntimeError) as excinfo: + manager._prepare_draft_resources(batch) + + message = str(excinfo.value) + assert "context resize failed for request 72" in message + assert "72: capacity=0 history=0 blocks=0" in message + assert int(caches[71].num_blocks) > 0, "the first mirror is what exhausted the pool" + assert int(caches[72].num_blocks) == 0 + # What the drafter would have been handed: no readable context, and + # not even a one-token block has a page to be appended to. + limit = dflash_allocated_ctx_limit( + torch.tensor([int(caches[72].num_blocks)]), tokens_per_block, block_size=1 + ) + assert limit.item() == 0 + finally: + for cache in caches.values(): + cache.close() + manager.shutdown() + + +def test_reuse_funding_holds_max_batch_size_full_mirrors_where_per_token_funding_does_not() -> None: + """Two requests sharing a prefix on a real draft pool. Funded for the + target's token count (prefix once) the second full mirror does not fit; + funded for max_batch_size full-length mirrors both do.""" + if not torch.cuda.is_available(): + pytest.skip("requires CUDA") + torch.cuda.init() + spec = DFlashDecodingConfig(max_draft_len=4, speculative_model="draft") + # 32 KiB per token (8 layers, 8 KV heads of 128, fp16) so that one prefix is 32 MiB: the + # GPU tier rounds its quota up to 2 MiB chunks, which must stay small next to the shortfall. + num_layers, num_kv_heads, head_dim = 8, 8, 128 + model_config = SimpleNamespace( + quant_config=None, + pretrained_config=SimpleNamespace( + hidden_size=num_kv_heads * head_dim, + num_attention_heads=num_kv_heads, + num_key_value_heads=num_kv_heads, + layer_types=["full_attention"] * num_layers, + ), + get_num_attention_layers=lambda: num_layers, + ) + tokens_per_block, max_seq_len, batch_size = 32, 2048, 2 + prefix, tail = 1024, 64 + config = KvCacheConfig(enable_block_reuse=False) + slope, fixed = KVCacheManagerV2.get_cache_size_per_token( + model_config, + Mapping(), + tokens_per_block=tokens_per_block, + max_seq_len=max_seq_len, + max_batch_size=batch_size, + max_num_tokens=max_seq_len, + kv_cache_config=config, + spec_config=spec, + is_draft=True, + ) + assert fixed == 0, "a windowless pool has no fixed cost" + assert slope == num_layers * 2 * num_kv_heads * head_dim * 2 + _, headroom = _get_generation_kv_capacity(spec, is_draft=True) + mirror = prefix + tail + headroom + # What the target holds when the two requests share the prefix. + per_token_budget = slope * (prefix + 2 * (tail + headroom)) + reuse_budget = KvCacheCreator.standalone_draft_pool_worst_case_cost( + slope, tokens_per_block, max_seq_len + headroom, batch_size, config.max_util_for_resume + ).intercept + stream = torch.cuda.current_stream().cuda_stream + + def both_mirrors_fit(budget: int) -> bool: + manager = KVCacheManagerV2( + config.model_copy(update={"max_gpu_total_bytes": budget}), + CacheType.SELF, + num_layers=num_layers, + num_kv_heads=num_kv_heads, + head_dim=head_dim, + tokens_per_block=tokens_per_block, + max_seq_len=max_seq_len, + max_batch_size=batch_size, + max_num_tokens=max_seq_len, + mapping=Mapping(), + dtype=DataType.HALF, + spec_config=spec, + is_draft=True, + ) + caches = [] + try: + for _ in range(batch_size): + cache = manager.impl.create_kv_cache() + caches.append(cache) + assert cache.resume(stream) + # As _prepare_draft_resources sizes a fresh mirror on a reused + # prefix: the whole prompt as capacity, the prefix as history. + return all(cache.resize(mirror, prefix) for cache in caches) + finally: + for cache in caches: + cache.close() + manager.shutdown() + + assert not both_mirrors_fit(per_token_budget) + assert both_mirrors_fit(reuse_budget) + + +def test_windowed_draft_mirrors_release_pages_below_the_window_as_chunks_land() -> None: + """A standalone drafter's mirror carries the chunk start as its history on + every context resize, so the windowed draft pool releases the pages below + ``history + 1 - window`` and a mirror holds at most its window plus the + chunk it is computing. Funded for max_batch_size such mirrors + (``KvCacheCreator._standalone_draft_pool_cost``), the pool prefills contexts + many times longer than it could hold in full.""" + if not torch.cuda.is_available(): + pytest.skip("requires CUDA") + torch.cuda.init() + spec = DFlashDecodingConfig(max_draft_len=4, speculative_model="draft") + # 32 KiB per token (8 layers, 8 KV heads of 128, fp16): a 256-token window + # plus a 512-token chunk is ~26 MiB per mirror, two full 8192-token + # mirrors would be 512 MiB, and the GPU tier's 2 MiB quota rounding + # cannot blur the two. + num_layers, num_kv_heads, head_dim = 8, 8, 128 + tokens_per_block, window, chunk, max_seq_len, batch_size = 32, 256, 512, 8192, 2 + config = KvCacheConfig(enable_block_reuse=False, max_attention_window=[window] * num_layers) + bytes_per_token = num_layers * 2 * num_kv_heads * head_dim * 2 + _, headroom = _get_generation_kv_capacity(spec, is_draft=True) + tokens_per_mirror = window + chunk + 2 * headroom + tokens_per_block + pages_per_mirror = -(-tokens_per_mirror // tokens_per_block) + budget = KvCacheCreator.standalone_draft_pool_worst_case_cost( + bytes_per_token, tokens_per_block, tokens_per_mirror, batch_size, config.max_util_for_resume + ).intercept + assert budget * 8 < bytes_per_token * batch_size * max_seq_len + manager = KVCacheManagerV2( + config.model_copy(update={"max_gpu_total_bytes": budget}), + CacheType.SELF, + num_layers=num_layers, + num_kv_heads=num_kv_heads, + head_dim=head_dim, + tokens_per_block=tokens_per_block, + max_seq_len=max_seq_len, + max_batch_size=batch_size, + max_num_tokens=chunk, + mapping=Mapping(), + dtype=DataType.HALF, + spec_config=spec, + is_draft=True, + ) + stream = torch.cuda.current_stream().cuda_stream + caches = [] + + def live_pages(cache) -> int: + return len(cache.get_aggregated_page_indices(0, valid_only=True)) + + try: + for _ in range(batch_size): + cache = manager.impl.create_kv_cache() + caches.append(cache) + assert cache.resume(stream) + # Draft mirrors never commit (_mirror_draft_kv_cache); a cache that + # may still commit keeps its stale blocks held for reuse instead of + # releasing them below the window. + cache.stop_committing() + prompt_len = max_seq_len - chunk + for position in range(0, prompt_len, chunk): + for cache in caches: + # As _prepare_draft_resources sizes a context chunk: the chunk + # end plus the generation headroom as capacity, the chunk start + # as history. + assert cache.resize(position + chunk + headroom, position), ( + position, + live_pages(cache), + ) + assert live_pages(cache) <= pages_per_mirror + # The first generation step grows the mirror once more before + # update_resources advances its history past the prompt. + for cache in caches: + assert cache.resize(cache.capacity + 1 + spec.max_draft_len) + assert live_pages(cache) <= pages_per_mirror + assert cache.capacity > prompt_len + # Held in full, the two prompts would need several times the pool. + assert batch_size * prompt_len * bytes_per_token > 4 * budget + finally: + for cache in caches: + cache.close() + manager.shutdown() + + +def test_windowed_draft_pool_backs_only_the_window_of_a_generation_dummy() -> None: + """A CUDA-graph warmup dummy is a generation request at the longest length + the target pool can hold. Its draft mirror carries the same history hint + as the target cache, so a windowed draft pool funded for max_batch_size + mirrors of window + chunk (``KvCacheCreator._standalone_draft_pool_cost``) + backs only the dummy's window; without the hint the mirror would need the + dummy's whole length, which such a pool cannot hold.""" + if not torch.cuda.is_available(): + pytest.skip("requires CUDA") + torch.cuda.init() + spec = DFlashDecodingConfig(max_draft_len=4, speculative_model="draft") + # Draft pool as in the test above: 32 KiB per token, funded for two + # mirrors of a 256-token window plus a 512-token chunk (~52 MiB). A + # 60k-token dummy held in full would need ~1.8 GiB of it. + num_layers, num_kv_heads, head_dim = 8, 8, 128 + tokens_per_block, window, chunk, max_seq_len, batch_size = 32, 256, 512, 65536, 2 + token_num = 60_000 + draft_config = KvCacheConfig( + enable_block_reuse=False, max_attention_window=[window] * num_layers + ) + bytes_per_token = num_layers * 2 * num_kv_heads * head_dim * 2 + _, headroom = _get_generation_kv_capacity(spec, is_draft=True) + tokens_per_mirror = window + chunk + 2 * headroom + tokens_per_block + pages_per_mirror = -(-tokens_per_mirror // tokens_per_block) + draft_budget = KvCacheCreator.standalone_draft_pool_worst_case_cost( + bytes_per_token, + tokens_per_block, + tokens_per_mirror, + batch_size, + draft_config.max_util_for_resume, + ).intercept + assert token_num // tokens_per_block > 4 * pages_per_mirror + draft = KVCacheManagerV2( + draft_config.model_copy(update={"max_gpu_total_bytes": draft_budget}), + CacheType.SELF, + num_layers=num_layers, + num_kv_heads=num_kv_heads, + head_dim=head_dim, + tokens_per_block=tokens_per_block, + max_seq_len=max_seq_len, + max_batch_size=batch_size, + max_num_tokens=chunk, + mapping=Mapping(), + dtype=DataType.HALF, + spec_config=spec, + is_draft=True, + ) + # A small windowless target pool (512 B per token) that holds the whole + # dummy, so the target side of add_dummy_requests cannot be what fails. + target = KVCacheManagerV2( + KvCacheConfig(enable_block_reuse=False, max_gpu_total_bytes=128 << 20), + CacheType.SELF, + num_layers=1, + num_kv_heads=1, + head_dim=head_dim, + tokens_per_block=tokens_per_block, + max_seq_len=max_seq_len, + max_batch_size=batch_size, + max_num_tokens=chunk, + mapping=Mapping(), + dtype=DataType.HALF, + spec_config=spec, + ) + stream = torch.cuda.current_stream().cuda_stream + + def live_pages(cache) -> int: + return len(cache.get_aggregated_page_indices(0, valid_only=True)) + + requests = None + try: + # The mirror resize without a history hint: the whole dummy on the + # windowed pool, which is more than the pool has. + probe = draft.impl.create_kv_cache() + try: + assert probe.resume(stream) + assert not probe.resize(token_num + draft.num_extra_kv_tokens) + finally: + probe.close() + requests = target.add_dummy_requests( + [7], + token_nums=[token_num], + is_gen=True, + max_num_draft_tokens=spec.max_draft_len, + draft_kv_cache_manager=draft, + ) + assert requests is not None, "the dummy's draft mirror did not fit the windowed pool" + (request,) = requests + target_cache = target.kv_cache_map[request.py_request_id] + mirror = draft.kv_cache_map[request.py_request_id] + assert live_pages(target_cache) >= token_num // tokens_per_block + assert mirror.capacity == target_cache.capacity + assert mirror.history_length == token_num - 1 + assert live_pages(mirror) <= pages_per_mirror + finally: + for request in requests or (): + draft.free_resources(request) + target.free_resources(request) + draft.shutdown() + target.shutdown() + + def _make_publishing_manager(policy: BlockReusePolicy) -> KVCacheManagerV2: """A manager wired just far enough to run the publish/history bookkeeping.""" manager = object.__new__(KVCacheManagerV2) diff --git a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_draft_kv.py b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_draft_kv.py index 53958cbab57c..4b13968ae4bb 100644 --- a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_draft_kv.py +++ b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_draft_kv.py @@ -47,3 +47,107 @@ def test_eagle3_one_model_still_gets_a_separate_draft_kv_cache(): config = EagleDecodingConfig(max_draft_len=3, speculative_model="/tmp/eagle3-draft") assert config.spec_dec_mode.is_eagle3_one_model() assert should_use_separate_draft_kv_cache(config) is True + + +@pytest.mark.parametrize( + ("bound_to_pool", "cur", "first_pos", "expected"), + [ + # Managed pool: the block table is indexed by target position, so a + # tail that reuse let the target start at first_pos is stored there. + (True, 0, 8192, 8192), + # A continuing chunk keeps the slot's stored length. + (True, 4096, 4096, 4096), + # Private arena: every request numbers from 0. + (False, 0, 8192, 0), + (False, 0, 0, 0), + ], +) +def test_dflash_ctx_store_start_follows_the_pool_numbering( + bound_to_pool: bool, cur: int, first_pos: int, expected: int +) -> None: + from tensorrt_llm._torch.speculative.dflash import DFlashWorker + + worker = object.__new__(DFlashWorker) + worker._ctx_block_tables = object() if bound_to_pool else None + assert worker._ctx_store_start(cur, first_pos) == expected + + +@pytest.mark.parametrize( + ("cur", "start", "reuse_addressable", "expected"), + [ + # Unpaired pool, tail stored at a reused prefix's end: the pages below + # it hold no drafter K/V and are hidden from the block decode. + (0, 8192, False, 128), + # A start inside a page rounds the base up; the tail tokens in that + # page stay stored but unread. + (0, 8200, False, 129), + # A continuing chunk numbers from the stored length: no base change. + (4096, 4096, False, None), + # A pool paired with the target's reuse holds the prefix's drafter K/V. + (0, 8192, True, None), + ], +) +def test_dflash_ctx_base_for_store( + cur: int, start: int, reuse_addressable: bool, expected: int | None +) -> None: + from tensorrt_llm._torch.speculative.dflash import DFlashWorker + + worker = object.__new__(DFlashWorker) + worker._ctx_page_size = 64 + worker._ctx_reuse_addressable = reuse_addressable + assert worker._ctx_base_for_store(cur, start) == expected + + +def test_dflash_rebased_ctx_rows_hide_the_reused_prefix() -> None: + """A tail stored after a reused prefix on an unpaired pool. The block decode + reads the slot's row rebased to the tail and a context length shortened by + the prefix, so the never-written prefix pages (poisoned here) are not within + reach, and every tail position still resolves to the page and offset the + store wrote it to.""" + import torch + + from tensorrt_llm._torch.speculative.dflash import dflash_rebased_ctx_rows + + page_size, first_pos, tail = 4, 24, 5 + row = torch.tensor([[7, 3, 11, 0, 5, 9, 13, 2, 6, 1, 15, 4, 8, 10, 12, 14]], dtype=torch.int32) + pool = torch.zeros(16, page_size) + pool[row[0, : first_pos // page_size].long()] = float("nan") + for pos in range(first_pos, first_pos + tail): + pool[row[0, pos // page_size], pos % page_size] = pos + + base_pages = torch.tensor([first_pos // page_size]) + view = dflash_rebased_ctx_rows(row, base_pages) + num_ctx = first_pos + tail - base_pages.item() * page_size + assert view.shape == row.shape + readable = view[0, : -(-num_ctx // page_size)].long() + assert torch.isfinite(pool[readable]).all() + for pos in range(num_ctx): + assert pool[view[0, pos // page_size], pos % page_size] == first_pos + pos + # A slot without a base reads its row unchanged. + assert torch.equal(dflash_rebased_ctx_rows(row, torch.zeros(1, dtype=torch.long)), row) + + +def test_dflash_write_ctx_len_zero_length_reset_wins_over_a_pending_base() -> None: + """_store_prefill_context records a chunk's base pages before the overflow + check that may reset the same slot to length 0 in the same call. The reset + takes precedence: the slot ends with length 0 and base 0 on the host + mirrors and the device tensors alike, while a slot that was not reset keeps + its new length and base.""" + import torch + + from tensorrt_llm._torch.speculative.dflash import DFlashWorker + + worker = object.__new__(DFlashWorker) + worker._ctx_len_host = [0, 512, 64] + worker._ctx_base_host = [0, 8, 1] + worker._ctx_len = torch.tensor(worker._ctx_len_host) + worker._ctx_base_pages = torch.tensor(worker._ctx_base_host) + + # Slot 1: base recorded, then reset by the overflow check. Slot 2: a + # continuing chunk with a new base. + worker._write_ctx_len({1: 0, 2: 96}, {1: 128, 2: 2}) + + assert worker._ctx_len_host == [0, 0, 96] + assert worker._ctx_base_host == [0, 0, 2] + assert worker._ctx_len.tolist() == [0, 0, 96] + assert worker._ctx_base_pages.tolist() == [0, 0, 2] diff --git a/tests/unittest/_torch/speculative/hw_agnostic/test_minimax_dspark_draft.py b/tests/unittest/_torch/speculative/hw_agnostic/test_minimax_dspark_draft.py new file mode 100644 index 000000000000..132f47295ae6 --- /dev/null +++ b/tests/unittest/_torch/speculative/hw_agnostic/test_minimax_dspark_draft.py @@ -0,0 +1,241 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 standalone DSpark drafter: config flattening and checkpoint +translation (CPU only; no drafter module is built here).""" + +import json + +import pytest +import torch + +from tensorrt_llm._torch.models.modeling_dspark import ( + GQADSparkForCausalLM, + MiniMaxDSparkBackbone, + MiniMaxDSparkForCausalLM, + dequantize_mxfp8_weight, + prepare_minimax_dspark_weights, + remap_minimax_dspark_weight_name, +) +from tensorrt_llm._torch.models.modeling_utils import get_registered_model_class +from tensorrt_llm._torch.pyexecutor.config_utils import ( + MINIMAX_DSPARK_DRAFT_ARCH, + build_minimax_dspark_draft_config, + load_pretrained_config, +) + +# The shipped drafter config.json, reduced to the fields the loaders read. The +# backbone geometry sits under text_config; the dspark_* / dflash_config knobs +# are duplicated at both levels, as in the checkpoint. +_TEXT = { + "architectures": ["DSparkMiniMaxDraftModel"], + "dtype": "bfloat16", + "hidden_size": 6144, + "intermediate_size": 3072, + "num_hidden_layers": 5, + "num_attention_heads": 64, + "num_key_value_heads": 4, + "head_dim": 128, + "vocab_size": 200064, + "max_position_embeddings": 1048576, + "rms_norm_eps": 1e-06, + "use_gemma_norm": True, + "rope_theta": 10000000, + "rotary_dim": 64, + "partial_rotary_factor": 0.5, + "hidden_act": "swigluoai", + "use_qk_norm": True, + "dense_intermediate_size": 12288, + "qk_norm_type": "per_head", + "swiglu_alpha": 1.702, + "swiglu_limit": 7.0, + "sparse_attention_config": None, + "dspark_block_size": 7, + "dspark_markov_rank": 256, + "dspark_markov_head_type": "vanilla", + "dspark_noise_token_id": 200058, + "dspark_use_dense_ffn": True, + "dspark_ffn_hidden_size": 12288, + "dspark_target_layer_ids": [3, 17, 31, 45, 59], + "dspark_config": { + "markov_rank": 256, + "markov_head_type": "vanilla", + "mask_token_id": 200058, + "use_dense_ffn": True, + "ffn_hidden_size": 12288, + }, + "dflash_config": { + "num_hidden_layers": 5, + "num_target_layers": 60, + "block_size": 7, + "mask_token_id": 200058, + "target_layer_ids": [3, 17, 31, 45, 59], + "layer_types": ["sliding_attention"] * 5, + "sliding_window": 4096, + }, + "enable_confidence_head": True, + "layer_types": ["sliding_attention"] * 5, + "sliding_window": 4096, + "model_type": "", +} + +_CONFIG = { + "transformers_version": "5.12.1", + "architectures": ["DSparkMiniMaxDraftModel"], + "dtype": "bfloat16", + "model_type": "minimax_m3_vl", + "text_config": _TEXT, + "quantization_config": { + "quant_method": "mxfp8", + "weight_block_size": [1, 32], + "ignored_layers": ["lm_head"], + }, + "tie_word_embeddings": False, + "dspark_block_size": 7, + "dspark_target_layer_ids": [3, 17, 31, 45, 59], + "dflash_config": dict(_TEXT["dflash_config"]), + "hidden_size": 6144, + "vocab_size": 200064, + "num_hidden_layers": 5, +} + + +def test_flattened_config_exposes_backbone_and_drafter_fields(): + cfg = build_minimax_dspark_draft_config(_CONFIG) + assert cfg.architectures == [MINIMAX_DSPARK_DRAFT_ARCH] + assert cfg.model_type != "minimax_m3_vl" + assert cfg.num_attention_heads == 64 + assert cfg.num_key_value_heads == 4 + assert cfg.head_dim == 128 + assert cfg.rotary_dim == 64 and cfg.partial_rotary_factor == 0.5 + assert cfg.dspark_target_layer_ids == [3, 17, 31, 45, 59] + assert cfg.dspark_markov_rank == 256 + assert cfg.dflash_config["mask_token_id"] == 200058 + assert cfg.dflash_config["block_size"] == 7 + # The HF sliding window on 'sliding_attention' layers is translated to the + # DFlash block-decode spelling. + assert cfg.dflash_config["use_swa"] is True + assert cfg.dflash_config["swa_window_size"] == 4096 + # Dequantized at load: the backbone is built unquantized. + assert getattr(cfg, "quantization_config", None) is None + assert not hasattr(cfg, "text_config") + assert cfg.torch_dtype == torch.bfloat16 + + +def test_flattened_config_keeps_existing_dflash_window_spelling(): + config = json.loads(json.dumps(_CONFIG)) + config["text_config"]["dflash_config"]["use_swa"] = False + cfg = build_minimax_dspark_draft_config(config) + assert cfg.dflash_config["use_swa"] is False + assert "swa_window_size" not in cfg.dflash_config + + +def test_load_pretrained_config_routes_by_architecture(tmp_path): + (tmp_path / "config.json").write_text(json.dumps(_CONFIG)) + cfg = load_pretrained_config(str(tmp_path), trust_remote_code=False) + assert cfg.architectures == [MINIMAX_DSPARK_DRAFT_ARCH] + assert cfg.hidden_size == 6144 and cfg.num_hidden_layers == 5 + + +def test_backbone_is_registered_under_the_checkpoint_architecture(): + assert get_registered_model_class(MINIMAX_DSPARK_DRAFT_ARCH) is MiniMaxDSparkBackbone + assert issubclass(MiniMaxDSparkForCausalLM, GQADSparkForCausalLM) + + +@pytest.mark.parametrize( + "name, expected", + [ + ( + "language_model.model.dspark.layers.3.decoder_layer.self_attn.q_proj.weight", + "layers.3.self_attn.q_proj.weight", + ), + ( + "language_model.model.dspark.layers.0.decoder_layer.mlp.down_proj.weight_scale_inv", + "layers.0.mlp.down_proj.weight_scale_inv", + ), + ("language_model.model.dspark.fc.weight", "fc.weight"), + ("language_model.model.dspark.hidden_norm.weight", "hidden_norm.weight"), + ("language_model.model.dspark.final_norm.weight", "norm.weight"), + ( + "language_model.model.dspark.markov_head.markov_w1.weight", + "markov_head.markov_w1.weight", + ), + ( + "language_model.model.dspark.confidence_head.proj.bias", + "confidence_head.proj.bias", + ), + ("lm_head.weight", "lm_head.weight"), + ], +) +def test_weight_name_remap(name, expected): + assert remap_minimax_dspark_weight_name(name) == expected + + +def test_mxfp8_dequantization_matches_per_block_reference(): + torch.manual_seed(0) + out_features, in_features = 4, 96 # three 32-wide K blocks + # Build finite E4M3 codes directly: dividing bf16-range samples by block scales as small + # as 2**-13 would overflow E4M3 (max 448) into NaN and make the comparison meaningless. + quantized = (torch.randn(out_features, in_features) * 3).to(torch.float8_e4m3fn) + assert torch.isfinite(quantized.to(torch.float32)).all() + # UE8M0 exponents around 127 (2**0); 114..118 is what the shipped shard holds. + exponents = torch.randint(114, 131, (out_features, in_features // 32), dtype=torch.uint8) + scale = torch.exp2(exponents.to(torch.float32) - 127).repeat_interleave(32, dim=1) + expected = (quantized.to(torch.float32) * scale).to(torch.bfloat16) + + got = dequantize_mxfp8_weight(quantized, exponents, torch.bfloat16) + assert got.dtype == torch.bfloat16 + torch.testing.assert_close(got, expected, rtol=0, atol=0) + + +def test_prepare_weights_dequantizes_and_folds_gemma_norms(): + prefix = "language_model.model.dspark." + quantized = torch.ones(2, 64, dtype=torch.float8_e4m3fn) + scale = torch.full((2, 2), 128, dtype=torch.uint8) # 2**1 per block + norm = torch.full((8,), 0.25, dtype=torch.bfloat16) + weights = { + prefix + "layers.0.decoder_layer.self_attn.k_proj.weight": quantized, + prefix + "layers.0.decoder_layer.self_attn.k_proj.weight_scale_inv": scale, + prefix + "layers.0.decoder_layer.self_attn.k_norm.weight": norm.clone(), + prefix + "layers.0.decoder_layer.input_layernorm.weight": norm.clone(), + prefix + "hidden_norm.weight": norm.clone(), + prefix + "final_norm.weight": norm.clone(), + prefix + "fc.weight": torch.zeros(8, 16, dtype=torch.bfloat16), + prefix + "markov_head.markov_w1.weight": norm.clone(), + } + out = prepare_minimax_dspark_weights(weights, torch.bfloat16) + + assert set(out) == { + "layers.0.self_attn.k_proj.weight", + "layers.0.self_attn.k_norm.weight", + "layers.0.input_layernorm.weight", + "hidden_norm.weight", + "norm.weight", + "fc.weight", + "markov_head.markov_w1.weight", + } + assert out["layers.0.self_attn.k_proj.weight"].dtype == torch.bfloat16 + torch.testing.assert_close( + out["layers.0.self_attn.k_proj.weight"], + torch.full((2, 64), 2.0, dtype=torch.bfloat16), + ) + for key in ( + "layers.0.self_attn.k_norm.weight", + "layers.0.input_layernorm.weight", + "hidden_norm.weight", + "norm.weight", + ): + torch.testing.assert_close(out[key], torch.full((8,), 1.25, dtype=torch.bfloat16)) + # Head tensors are not norms and are passed through untouched. + torch.testing.assert_close(out["markov_head.markov_w1.weight"], norm) From 493ec2cf6c3bce00212535bb80141e2cf015997d Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 09/38] [None][fix] Size the standalone drafter's KV mirror for disaggregated generation-init requests A generation worker receives the prompt KV instead of prefilling it and starts the drafter on a fresh slot, so the draft mirror never holds K/V for the transferred prompt. Treat that prompt as history on gen-init so the drafter's window trims the mirror to its tail plus headroom, instead of asking for a prompt-sized chunk that exceeds the pool funding. Signed-off-by: Pranav Shrestha --- .../kv_cache/kv_cache_manager_v2.py | 10 +++++ .../test_draft_kv_resume_pressure.py | 37 +++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py index 552bfc609ea9..8ee3b7dff9e8 100644 --- a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py +++ b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py @@ -4757,6 +4757,16 @@ def _prepare_draft_resources(self, scheduled_batch: ScheduledRequests): # target prefix starts the mirror at position > 0 with no pages # of its own, so the resize carries the history itself. history = max(kv_cache.history_length, req.context_current_position) + if req.is_disagg_generation_init_state: + # A disaggregated generation worker receives the prompt KV + # instead of prefilling it, and the standalone drafter + # starts this request on a fresh slot (DFlash/DSpark + # prepare): the mirror never holds drafter K/V for the + # transferred prompt. The whole prompt is therefore + # history, so the window trims the mirror to its tail + # plus the generation headroom the pool is funded for, + # instead of one prompt-sized chunk that exceeds it. + history = max(history, req.context_current_position + req.context_chunk_size) if not kv_cache.resize(capacity, history): raise self._mirror_resize_error(req, "context", capacity) self._after_shrink(req.py_request_id, kv_cache) diff --git a/tests/unittest/_torch/speculative/hw_agnostic/test_draft_kv_resume_pressure.py b/tests/unittest/_torch/speculative/hw_agnostic/test_draft_kv_resume_pressure.py index dc90385bf0cb..e691f5757b81 100644 --- a/tests/unittest/_torch/speculative/hw_agnostic/test_draft_kv_resume_pressure.py +++ b/tests/unittest/_torch/speculative/hw_agnostic/test_draft_kv_resume_pressure.py @@ -54,6 +54,7 @@ def __init__(self, req_id): self.context_current_position = 0 self.context_chunk_size = 8 self.py_draft_tokens = None + self.is_disagg_generation_init_state = False class _Batch: @@ -78,10 +79,12 @@ def _manager(resume_ok): manager.is_draft = True manager.enable_joint_kv_cache_reuse = False manager.num_extra_kv_tokens = 0 + manager._mirror_uses_draft_view = False # one-engine external drafter: target view manager._allocated_draft_lens = set() cache = MagicMock(name="kv_cache") cache.resize.return_value = True cache.capacity = 0 + cache.history_length = 0 manager._mirror_draft_kv_cache = MagicMock(return_value=cache) manager._resume_and_restore = MagicMock(return_value=resume_ok) manager._required_gen_capacity = MagicMock(return_value=1) @@ -125,6 +128,40 @@ def test_context_request_is_prepared_normally_when_resume_succeeds(): cache.resize.assert_called_once() +def test_disagg_generation_init_sizes_the_mirror_from_history_and_chunk() -> None: + """A transferred prompt is history for the drafter's mirror, not a chunk to hold. + + On a disaggregated generation worker the prompt KV arrives over the wire + and the standalone drafter starts the request on a fresh slot, so the + mirror never holds drafter K/V for that prompt. The resize must therefore + carry the whole prompt as history, which lets the attention window trim + the mirror to its tail plus the generation headroom the draft pool is + funded for; a history of 0 asks for one prompt-sized chunk instead and + exceeds that funding for long prompts. + """ + manager, cache = _manager(resume_ok=True) + req = _Req(9) + req.is_disagg_generation_init_state = True + req.context_current_position = 0 + req.context_chunk_size = 200_000 + + manager._prepare_draft_resources(_Batch(context_requests=[req])) + + cache.resize.assert_called_once_with(200_000, 200_000) + + +def test_ordinary_context_chunk_keeps_its_start_as_history() -> None: + """The agg chunked path is untouched: history is the chunk start, not its end.""" + manager, cache = _manager(resume_ok=True) + req = _Req(10) + req.context_current_position = 64 + req.context_chunk_size = 8 + + manager._prepare_draft_resources(_Batch(context_requests=[req])) + + cache.resize.assert_called_once_with(72, 64) + + def test_generation_request_still_raises_on_a_resume_refusal(): """Skipping a generation request asserts later in C++, so it must raise here.""" manager, _ = _manager(resume_ok=False) From eae7bb88432117b28a55774ac0f501735df466ec Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 10/38] [None][fix] Keep the DFlash block write on an owned page after an unaligned reused prefix On an unpaired draft pool the block decode reads a view rebased by the rounded-up base of a reused prefix. While the stored tail is shorter than that base, entry 0 of the view lies past the request's allocation and resolves to page 0, so the block's own K/V landed in another request's page and the drafter stayed blind until the context caught up. The view is now shifted only to the page holding the context length on those steps; the shortened read length is unchanged, so nothing below the base is read. Adds a CPU test that walks every context length of that window. Signed-off-by: Pranav Shrestha --- tensorrt_llm/_torch/speculative/dflash.py | 40 +++++++++-- .../hw_agnostic/test_dflash_draft_kv.py | 68 +++++++++++++++++++ .../hw_agnostic/test_dflash_dummy_slot.py | 2 + .../test_dflash_guided_rollback.py | 2 + 4 files changed, 105 insertions(+), 7 deletions(-) diff --git a/tensorrt_llm/_torch/speculative/dflash.py b/tensorrt_llm/_torch/speculative/dflash.py index 27a28dd9961a..0db5892872bd 100644 --- a/tensorrt_llm/_torch/speculative/dflash.py +++ b/tensorrt_llm/_torch/speculative/dflash.py @@ -16,7 +16,7 @@ import os from collections import deque from dataclasses import dataclass -from typing import TYPE_CHECKING, Callable, List, Optional +from typing import TYPE_CHECKING, Callable, List, Optional, Tuple import torch from torch import nn @@ -534,6 +534,29 @@ def dflash_rebased_ctx_rows(rows: torch.Tensor, base_pages: torch.Tensor) -> tor return rows.gather(1, index.clamp_(max=width - 1)) +def dflash_rebased_ctx_view( + rows: torch.Tensor, base_pages: torch.Tensor, num_ctx: torch.Tensor, page_size: int +) -> Tuple[torch.Tensor, torch.Tensor]: + """Block-table rows and context lengths the block decode reads for slots + whose first ``base_pages`` pages hold no drafter K/V (a prefix the target + reused, see ``_ctx_base_for_store``). + + The context length is shortened by the base, so the decode never reads + below it. The decode also writes the block's own K/V through the returned + rows, at rebased positions ``num_ctx .. num_ctx + block_size``, so entry 0 + of a row has to name a page the request holds. While the context is + shorter than the base (a tail stored inside the rounded-up page) it reads + nothing and the row is shifted only to the page holding the context + length: entry ``base`` lies past the allocation and resolves to another + request's page. The offsets written in that page belong to the reused + prefix, which no reader addresses. From ``base * page_size`` on the shift + is the base itself. + """ + rebase_pages = torch.minimum(base_pages, num_ctx // page_size) + num_ctx = (num_ctx - base_pages * page_size).clamp(min=0) + return dflash_rebased_ctx_rows(rows, rebase_pages), num_ctx + + @dataclass class DFlashSpecMetadata(SpecMetadata): """Metadata for DFlash speculative decoding. @@ -2282,13 +2305,16 @@ def prepare_1st_drafter_inputs( num_ctx_per_req_t = torch.minimum(num_ctx_per_req_t, allocated[gen_rows_out]) # Pages below a slot's base hold no drafter K/V (a prefix the # target reused): the read numbers from the base instead, see - # _ctx_base_for_store. Writes above keep the absolute table. + # _ctx_base_for_store and dflash_rebased_ctx_view. Writes above + # keep the absolute table. base_pages = self._ctx_base_pages[slots] - num_ctx_per_req_t = (num_ctx_per_req_t - base_pages * self._ctx_page_size).clamp_( - min=0 - ) - self._ctx_block_tables_read[gen_rows_out] = dflash_rebased_ctx_rows( - self._ctx_block_tables[gen_rows_out], base_pages + self._ctx_block_tables_read[gen_rows_out], num_ctx_per_req_t = ( + dflash_rebased_ctx_view( + self._ctx_block_tables[gen_rows_out], + base_pages, + num_ctx_per_req_t, + self._ctx_page_size, + ) ) ctx_page_table = self._ctx_block_tables_read noise_embedding = noise_embed_2d diff --git a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_draft_kv.py b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_draft_kv.py index 4b13968ae4bb..06ee944acd3d 100644 --- a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_draft_kv.py +++ b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_draft_kv.py @@ -127,6 +127,74 @@ def test_dflash_rebased_ctx_rows_hide_the_reused_prefix() -> None: assert torch.equal(dflash_rebased_ctx_rows(row, torch.zeros(1, dtype=torch.long)), row) +@pytest.mark.parametrize( + ("first_pos", "tail"), + [ + # A reused prefix ending inside a 64-token page with a one-token tail: + # the base rounds up to the next page and the context stays shorter + # than it for 55 steps. + (8200, 1), + # One token into a page: the first write offsets overlap the stored + # tail, which sits below the base and is never read. + (8193, 1), + # A page-aligned prefix end is never short of its base. + (8192, 1), + ], +) +def test_dflash_rebased_ctx_view_writes_the_block_into_an_owned_page( + first_pos: int, tail: int +) -> None: + """A tail stored after a reused prefix on an unpaired pool, with the mirror + funded the way the manager funds it: pages up to the one holding the + context plus the draft tokens, the window below it, nothing past it. The + block decode writes the block's own K/V at rebased positions + num_ctx .. num_ctx + block_size through the view it reads, so at every + context length those positions must resolve to a funded entry, including + the steps on which the context is still shorter than the rounded-up base + and the decode reads nothing. Entry ``base`` is unfunded on those steps and + holds 0, the pool page another request owns.""" + import torch + + from tensorrt_llm._torch.speculative.dflash import DFlashWorker, dflash_rebased_ctx_view + + page_size, block_size, draft_len, window_pages = 64, 3, 3, 64 + worker = object.__new__(DFlashWorker) + worker._ctx_page_size = page_size + worker._ctx_reuse_addressable = False + base = worker._ctx_base_for_store(0, first_pos) + assert base == -(-first_pos // page_size) + base_pages = torch.tensor([base]) + + def funded_row(ctx_len: int) -> torch.Tensor: + last = (ctx_len + draft_len - 1) // page_size + row = torch.zeros(1, base + 8, dtype=torch.int32) + row[0, last + 1 - window_pages : last + 1] = ( + torch.arange(last + 1 - window_pages, last + 1, dtype=torch.int32) + 1000 + ) + return row + + blind_steps = 0 + for ctx_len in range(first_pos + tail, (base + 1) * page_size): + row = funded_row(ctx_len) + view, num_ctx = dflash_rebased_ctx_view(row, base_pages, torch.tensor([ctx_len]), page_size) + assert view.shape == row.shape + blind = ctx_len < base * page_size + blind_steps += blind + assert num_ctx.item() == max(0, ctx_len - base * page_size) + if blind: + assert view[0, 0] == row[0, ctx_len // page_size] + if (ctx_len + draft_len - 1) // page_size < base: + # Entry ``base`` is funded only once the draft tokens reach it. + assert row[0, base] == 0 + for pos in range(num_ctx.item() + block_size): + entry = view[0, pos // page_size] + assert entry >= 1000, (ctx_len, pos) + if pos < num_ctx.item(): + # A read position p is absolute position base * page_size + p. + assert entry == row[0, (base * page_size + pos) // page_size] + assert blind_steps == max(0, base * page_size - first_pos - tail) + + def test_dflash_write_ctx_len_zero_length_reset_wins_over_a_pending_base() -> None: """_store_prefill_context records a chunk's base pages before the overflow check that may reset the same slot to length 0 in the same call. The reset diff --git a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_dummy_slot.py b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_dummy_slot.py index ace488b3fad5..cf2a2478974e 100644 --- a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_dummy_slot.py +++ b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_dummy_slot.py @@ -47,6 +47,8 @@ def _worker(lengths): _max_ctx=1000, _ctx_len=torch.tensor(lengths, dtype=torch.long, device="cuda"), _ctx_len_host=list(lengths), + _ctx_base_pages=torch.zeros(len(lengths), dtype=torch.long, device="cuda"), + _ctx_base_host=[0] * len(lengths), _batch_to_slot=torch.zeros(8, dtype=torch.long, device="cuda"), ) w._write_ctx_len = partial(DFlashWorker._write_ctx_len, w) diff --git a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_guided_rollback.py b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_guided_rollback.py index 2e29fd57a5cc..7960f49ba0cd 100644 --- a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_guided_rollback.py +++ b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_guided_rollback.py @@ -93,6 +93,8 @@ def test_dflash_worker_rolls_back_after_verification(monkeypatch): worker._ctx_len = torch.zeros(batch_size, dtype=torch.long, device="cuda") # _lazy_init_ctx_buffers (stubbed here) normally builds the host mirror. worker._ctx_len_host = [0] * batch_size + worker._ctx_base_pages = torch.zeros(batch_size, dtype=torch.long, device="cuda") + worker._ctx_base_host = [0] * batch_size accepted = torch.zeros(batch_size, K + 1, dtype=torch.int32, device="cuda") # Distinct per-row counts: one row accepts the whole draft, the other only # the golden token. Uniform values would let a broken forward that drops or From 29880cc4497e09779f6730f5f1c9ba1c2cc8d7bb Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:03:41 -0700 Subject: [PATCH 11/38] [None][fix] Keep MiniMax-M3 streaming and non-streaming reasoning parsing in step Resolve the parser's mode from thinking_mode or the prompt-resolved relay instead of the bare thinking flag, so a prefilled block is read as reasoning. Report truncated reasoning as reasoning in the non-streaming parse and make the streaming parser emit the same split: unopened is a delimiter, text before an opening tag is content, partial tags are withheld at chunk boundaries. Signed-off-by: Pranav Shrestha --- tensorrt_llm/llmapi/reasoning_parser.py | 112 ++++++++++++--- .../unittest/llmapi/test_reasoning_parser.py | 133 ++++++++++++++++++ 2 files changed, 227 insertions(+), 18 deletions(-) diff --git a/tensorrt_llm/llmapi/reasoning_parser.py b/tensorrt_llm/llmapi/reasoning_parser.py index 594b32bf8857..9863181705da 100644 --- a/tensorrt_llm/llmapi/reasoning_parser.py +++ b/tensorrt_llm/llmapi/reasoning_parser.py @@ -530,40 +530,116 @@ class MiniMaxM3ReasoningParser(DeepSeekR1Parser): {reasoning}{content} - * Without reasoning (the template still emits a bare ```` - as a sentinel so the model knows where the visible content starts):: + * Without reasoning (a bare ```` sentinel marks where the + visible content starts; the template prefills it when thinking is + disabled, and in adaptive mode the model generates it):: {content} - M3 is therefore not strictly ``reasoning_at_start`` — the leading - ```` may or may not be present — so we partition on the - closing tag first and then strip an optional leading opening tag - from the reasoning portion. This keeps streaming behavior identical - to :class:`DeepSeekR1Parser` for the common (``...``) case - while also handling the bare-sentinel form. + Which shape the model produces depends on how the prompt was rendered: + with thinking enabled the template prefills ```` into the + prompt, so the output opens inside the reasoning block with no opening + tag; with thinking disabled it prefills ````, so the output is + all content; in adaptive mode (the template default) it prefills nothing + and the model emits ```` or the bare sentinel first. The parser + therefore takes its mode from an explicit ``thinking_mode`` or from the + mode the server resolved off the rendered prompt (relayed as + ``enable_thinking``), and otherwise starts outside reasoning. Both + tags are delimiters and never appear in the result; text before an + opening tag is content; an opening tag without a closing tag is + reasoning cut off by the token limit. ``parse`` and ``parse_delta`` + agree on every input for every chunking. """ + resolves_thinking_from_prompt = True + reasoning_start = "" + reasoning_end = "" + def __init__(self, *, chat_template_kwargs: Optional[dict[str, Any]] = None) -> None: - super().__init__(reasoning_at_start=False, + kwargs = chat_template_kwargs or {} + # The template's own knob is `thinking_mode`; only "enabled" prefills + # the opening tag. Without it, trust the mode the server resolved + # from the rendered prompt, relayed as `enable_thinking` (a caller + # cannot pass that key: the template never reads it, so the + # unused-kwargs guard rejects it). A bare `thinking` is what the + # Responses path injects for `reasoning.effort`; this template does + # not act on it, so neither does the parser. + thinking_mode = kwargs.get("thinking_mode") + if thinking_mode is not None: + prefilled_open = thinking_mode == "enabled" + else: + prefilled_open = kwargs.get("enable_thinking") is True + super().__init__(reasoning_at_start=prefilled_open, chat_template_kwargs=chat_template_kwargs) self.reasoning_start = "" self.reasoning_end = "" def parse(self, text: str) -> ReasoningParserResult: + if self.reasoning_at_start: + # The prompt ends with the opening tag, so the output begins + # inside the reasoning block. + text = self.reasoning_start + text + start_idx = text.find(self.reasoning_start) end_idx = text.find(self.reasoning_end) + if start_idx != -1 and (end_idx == -1 or start_idx < end_idx): + prefix = text[:start_idx] + body = text[start_idx + len(self.reasoning_start):] + if end_idx == -1: + # An opening tag with no closing tag is reasoning cut off + # (e.g. by max_tokens); streaming reports it as reasoning. + return self._create_reasoning_end_result(prefix, body) + reasoning_content, _, content = body.partition(self.reasoning_end) + return self._create_reasoning_end_result(prefix + content, + reasoning_content) if end_idx == -1: - # No closing tag → no reasoning block in this response. + # No tags at all: no reasoning block in this response. return ReasoningParserResult(content=text) - reasoning_content = text[:end_idx] - content = text[end_idx + len(self.reasoning_end):] - # Strip an optional leading from the reasoning portion - # so the sentinel-only shape (no opening tag) reduces to - # reasoning_content="". - if reasoning_content.startswith(self.reasoning_start): - reasoning_content = reasoning_content[len(self.reasoning_start):] - return self._create_reasoning_end_result(content, reasoning_content) + # A closing tag with no opening tag is the empty-reasoning sentinel: + # a delimiter, dropped; what follows is parsed by the same rules. + rest = self.parse(text[end_idx + len(self.reasoning_end):]) + return self._create_reasoning_end_result(text[:end_idx] + rest.content, + rest.reasoning_content) + + def parse_delta(self, delta_text: str) -> ReasoningParserResult: + # Outside a reasoning block the inherited parser only looks for the + # opening tag, so a bare ```` sentinel would leak into the + # content; it drops text that precedes an opening tag in the same + # delta although it reports that text as content when it arrives on + # its own; and it only withholds a partial tag when the whole delta + # is one. Resolve the three here so streaming matches ``parse`` for + # every chunking; inside a reasoning block the inherited behaviour + # already does. + if self.in_reasoning: + return super().parse_delta(delta_text) + pending = self._buffer + delta_text + self._buffer = "" + start_idx = pending.find(self.reasoning_start) + end_idx = pending.find(self.reasoning_end) + if end_idx != -1 and (start_idx == -1 or end_idx < start_idx): + # Sentinel: a delimiter, dropped; the rest is parsed as usual. + before = pending[:end_idx] + after = pending[end_idx + len(self.reasoning_end):] + elif start_idx != -1: + # Text before the opening tag is content; the inherited parser + # takes over at the tag. + rest = super().parse_delta(pending[start_idx:]) + return ReasoningParserResult( + content=pending[:start_idx] + rest.content, + reasoning_content=rest.reasoning_content) + else: + # Withhold a trailing fragment that could still grow into a tag. + tag_start = pending.rfind("<") + if tag_start != -1 and ( + self.reasoning_start.startswith(pending[tag_start:]) + or self.reasoning_end.startswith(pending[tag_start:])): + self._buffer = pending[tag_start:] + pending = pending[:tag_start] + return ReasoningParserResult(content=pending) + rest = (self.parse_delta(after) if after else ReasoningParserResult()) + return ReasoningParserResult(content=before + rest.content, + reasoning_content=rest.reasoning_content) MODEL_TYPE_TO_REASONING_PARSER: dict[str, str] = { diff --git a/tests/unittest/llmapi/test_reasoning_parser.py b/tests/unittest/llmapi/test_reasoning_parser.py index 0e18c8f175f4..07c2f547f3d3 100644 --- a/tests/unittest/llmapi/test_reasoning_parser.py +++ b/tests/unittest/llmapi/test_reasoning_parser.py @@ -1412,3 +1412,136 @@ def test_deepseek_r1_keeps_a_second_close_as_content() -> None: result = ReasoningParserFactory.create_reasoning_parser( "deepseek-r1").parse("abf") assert result.content == "bf" + + +M3_START, M3_END = "", "" +# How the server relays the mode it resolved from the rendered prompt: +# thinking enabled (`` prefilled), disabled (`` +# prefilled) and adaptive / unresolved (nothing prefilled). +M3_ENABLED = {"enable_thinking": True} +M3_DISABLED = {"enable_thinking": False} +M3_ADAPTIVE = None +# The template's own knob, when the caller passes it. +M3_MODE_ENABLED = {"thinking_mode": "enabled"} +M3_MODE_ADAPTIVE = {"thinking_mode": "adaptive"} +# What the Responses path hands the parser for `reasoning.effort`: an injected +# `thinking` the M3 template never reads, here next to a resolved mode of off. +M3_EFFORT_INJECTED = {"enable_thinking": False, "thinking": True} + + +def _m3_parser(kwargs: dict | None): + return ReasoningParserFactory.create_reasoning_parser("minimax_m3", kwargs) + + +@pytest.mark.parametrize( + ("kwargs", "text", "content", "reasoning_content"), + [ + # Adaptive: the model opens with a tag or with the bare sentinel. + (M3_ADAPTIVE, f"{M3_START}a{M3_END}b", "b", "a"), + (M3_ADAPTIVE, f"{M3_END}b", "b", ""), + (M3_ADAPTIVE, "b", "b", ""), + # Reasoning truncated before the closing tag (e.g. by max_tokens). + (M3_ADAPTIVE, f"{M3_START}a", "", "a"), + (M3_ADAPTIVE, M3_START, "", ""), + # Text before an opening tag is content, as in streaming. + (M3_ADAPTIVE, f"x{M3_START}a{M3_END}b", "xb", "a"), + (M3_ADAPTIVE, f"x{M3_START}a", "x", "a"), + (M3_ADAPTIVE, f"x{M3_END}b", "xb", ""), + # Enabled: the prompt ends with ``, so the output starts + # inside the reasoning block and carries no opening tag. + (M3_ENABLED, f"a{M3_END}b", "b", "a"), + (M3_ENABLED, "a", "", "a"), + (M3_ENABLED, f"{M3_END}b", "b", ""), + # Disabled: the prompt ends with ``; the output is content. + (M3_DISABLED, "b", "b", ""), + (M3_DISABLED, f"{M3_START}a{M3_END}b", "b", "a"), + # The template's knob decides when present; an injected `thinking` + # (Responses path, `reasoning.effort`) does not open the block. + (M3_MODE_ENABLED, f"a{M3_END}b", "b", "a"), + (M3_MODE_ADAPTIVE, f"{M3_START}a{M3_END}b", "b", "a"), + (M3_EFFORT_INJECTED, "b", "b", ""), + (M3_EFFORT_INJECTED, f"{M3_START}a{M3_END}b", "b", "a"), + ({ + "thinking": True + }, f"{M3_START}a{M3_END}b", "b", "a"), + ]) +def test_minimax_m3_reasoning_parser(kwargs: dict | None, text: str, + content: str, + reasoning_content: str) -> None: + result = _m3_parser(kwargs).parse(text) + assert result.content == content + assert result.reasoning_content == reasoning_content + + +def _m3_stream(kwargs: dict | None, chunks: list[str]) -> tuple[str, str]: + reasoning_parser = _m3_parser(kwargs) + content, reasoning_content = "", "" + for chunk in chunks: + result = reasoning_parser.parse_delta(chunk) + content += result.content + reasoning_content += result.reasoning_content + result = reasoning_parser.finish() + content += result.content + reasoning_content += result.reasoning_content + return content, reasoning_content + + +@pytest.mark.parametrize( + "kwargs", [M3_ADAPTIVE, M3_ENABLED, M3_DISABLED, M3_EFFORT_INJECTED], + ids=["adaptive", "enabled", "disabled", "effort-injected"]) +@pytest.mark.parametrize("text", [ + f"{M3_START}a{M3_END}b", + f"{M3_END}b", + f"{M3_END}", + f"{M3_START}a", + M3_START, + f"a{M3_END}b", + "a", + f"x{M3_START}a{M3_END}b", + f"x{M3_END}b", +]) +def test_minimax_m3_reasoning_parser_stream_matches_non_stream( + kwargs: dict | None, text: str) -> None: + """Streaming in any chunking then finishing must match a non-streaming parse.""" + if kwargs == M3_ENABLED and text.endswith(M3_START): + # Inside a prefilled block a trailing second opening tag is withheld + # by streaming as a possible delimiter and dropped at the end; the + # shape is outside the contract. + pytest.skip("second opening tag inside a prefilled block") + whole = _m3_parser(kwargs).parse(text) + expected = (whole.content, whole.reasoning_content) + assert _m3_stream(kwargs, [text]) == expected + assert _m3_stream(kwargs, list(text)) == expected + for i, j in itertools.combinations_with_replacement(range(len(text) + 1), + 2): + chunks = [text[:i], text[i:j], text[j:]] + assert _m3_stream(kwargs, chunks) == expected, chunks + + +@pytest.mark.parametrize("kwargs", [M3_ADAPTIVE, M3_ENABLED, M3_DISABLED], + ids=["adaptive", "enabled", "disabled"]) +def test_minimax_m3_streaming_matches_parse_on_random_text( + kwargs: dict | None) -> None: + """Parity on random tag soup with at most one reasoning block. + + A second ```` after the close is outside the parsers' contract. + """ + pieces = [ + M3_START, M3_END, "<", "", "a", " ", "\n" + ] + rng = random.Random(0) + for _ in range(3000): + text = "".join(rng.choices(pieces, k=rng.randint(0, 8))) + if text.count(M3_START) > 1 or text.count(M3_END) > 1: + continue + if kwargs == M3_ENABLED and ( + text.endswith(M3_START) or + (M3_END in text and M3_START in text[text.index(M3_END):])): + # The prefilled block is the one block; an opening tag after its + # close (or trailing, see above) would start a second one. + continue + whole = _m3_parser(kwargs).parse(text) + cuts = [i for i in range(1, len(text)) if rng.random() < 0.3] + chunks = [text[i:j] for i, j in zip([0] + cuts, cuts + [len(text)])] + expected = (whole.content, whole.reasoning_content) + assert _m3_stream(kwargs, chunks) == expected, chunks From 0280b23a659f5fdaaee719ebcdc94c5793f5c1cb Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:17:14 -0700 Subject: [PATCH 12/38] [None][doc] Document MiniMax-M3.1-Flash serving in the MiniMax-M3 guide The deployment guide gains a linkable "Serving MiniMax-M3.1-Flash" section (bundled DSpark drafter, KV cache reuse, NVFP4 KV cache on SM107, thinking_mode and reasoning parsing) and states that both MoE backends load the NVFP4 expert packaging with the same static recipe. The supported-models matrices and the sparse-attention page now say block reuse is supported and mixed MXFP8/NVFP4 loads natively. Signed-off-by: Pranav Shrestha --- ...ployment-guide-for-minimax-m3-on-trtllm.md | 137 ++++++++++++++---- docs/source/features/sparse-attention.md | 8 +- docs/source/models/supported-models.md | 8 +- 3 files changed, 118 insertions(+), 35 deletions(-) diff --git a/docs/source/deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md b/docs/source/deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md index c93507fd6b61..7232f70febd0 100644 --- a/docs/source/deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md +++ b/docs/source/deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md @@ -12,7 +12,7 @@ TensorRT LLM supports three checkpoint weight formats for MiniMax-M3: * **MXFP8** — weights are stored in MXFP8. The curated configuration uses BF16 KV cache and is the default for this guide. * **Mixed MXFP8/NVFP4** — MXFP8 base layers with NVFP4 routed experts. FP8 or NVFP4 KV cache is selected separately at runtime. -The block-sparse attention path requires KV cache reuse to be disabled. The MSA backend supports linear Eagle3 with one shared GQA draft layer; see [Feature Support Notes](#feature-support-notes) for its constraints. +The MSA backend supports KV cache block reuse, linear Eagle3 with one shared GQA draft layer, and a standalone DSpark drafter; see [Feature Support Notes](#feature-support-notes) for the constraints and [Serving MiniMax-M3.1-Flash](#serving-minimax-m31-flash) for the MiniMax-M3.1-Flash checkpoint. This guide deploys MiniMax-M3 on **8x NVIDIA GB200 GPUs across 2 nodes** (4 GPUs per node) using Slurm and the `trtllm-llmapi-launch` multi-node launcher, with the MoE experts distributed via expert parallelism. The attention layers can run with either Tensor-Expert Parallelism (TEP) or Data-Expert Parallelism (DEP); see [Recommended Performance Settings](#recommended-performance-settings). @@ -64,45 +64,127 @@ routed expert projection as `weight_packed`, `weight_scale`, `weight_global_scale` and `input_global_scale`. TensorRT LLM keeps the packed FP4 bytes and E4M3 block scales as stored and converts the global scales at load time. A global scale that is zero, negative or not finite fails the load -with an error naming the checkpoint tensor. Use `moe_config.backend: CUTLASS` -for these routed experts. They follow the rounding steps of the reference -implementation of this packaging: every token's FC1 input is quantized with -its own scale (2688 / amax of the token), the inverse of that scale times the -FC1 weight scale multiplies the FC1 accumulators before their BF16 rounding, -and the routing weight multiplies the activation before it is requantized to -NVFP4 for FC2 (from FP32, with the stored static FC2 input scale), so the -finalize step sums the expert outputs without weights. Only the CUTLASS -backend implements this, so loading such a checkpoint with another MoE backend -fails; the finalize step runs unfused, and the stored w1/w3 input scales must -be 1.0. +with an error naming the checkpoint tensor. +Both `moe_config.backend: TRTLLM` (TRTLLM-Gen, the backend the serving configs +use on SM100 and SM103) and `moe_config.backend: CUTLASS` (the portable +grouped-GEMM path, which `AUTO` also resolves to) load this packaging and apply +the same static recipe: the stored `input_global_scale` becomes the layer's +per-tensor FC1 input scale (1.0 in the shipped checkpoints, so each 16-element +block of the FC1 input is quantized with only its own E4M3 block scale), +`weight_global_scale` is inverted into the per-expert FC1 and FC2 dequantization +alphas, and the gated intermediate is requantized to NVFP4 with the static FC2 +input scale before FC2, with the routing weights applied in the fused finalize. +Neither backend quantizes the FC1 input per token; they differ only in where the +FC1 dequantization and FC2 requantization happen (TRTLLM-Gen folds them into the +GEMM1 epilogue, CUTLASS applies them through its quantization scales), so the +two are numerically equivalent up to kernel rounding. Use the checkpoint's chat template (`chat_template.jinja`) when launching the server (see [Launch the TensorRT LLM Server](#launch-the-tensorrt-llm-server)). Evaluate a checkpoint in this packaging through the chat template as well, for -example `trtllm-eval ... gsm8k --apply_chat_template --chat_template_kwargs -'{"thinking_mode": "disabled"}'`; the raw eight-shot prompts of the `gsm8k` -default bypass the template the model was trained with. +example `trtllm-eval ... gsm8k --apply_chat_template`; the raw eight-shot prompts +of the `gsm8k` default bypass the template the model was trained with. + +### MiniMax-M3.1-Flash + +* [MiniMaxAI/MiniMax-M3.1-Flash](https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash) — MiniMaxAI-published checkpoint in this mixed MXFP8/NVFP4 packaging, with its DSpark drafter bundled in the `dspark/` subdirectory. See [Serving MiniMax-M3.1-Flash](#serving-minimax-m31-flash). + +```bash +git lfs install +git clone https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash /models/MiniMax-M3.1-Flash +``` ## Feature Support Notes * **Block-sparse attention is required.** MiniMax-M3 runs on the block-sparse attention backend, which must be selected via `sparse_attention_config.algorithm: minimax_m3` in the YAML configuration. There is no dense fallback for the sparse layers. -* **Weight and KV precision.** BF16, MXFP8, and mixed MXFP8/NVFP4 checkpoints are supported. On SM100/SM103, the MSA backend supports FP8 and NVFP4 KV cache. NVFP4 storage is used in sparse target layers; dense target layers and the shared Eagle3 draft layer use FP8. The curated BF16-KV YAML remains a separate starting configuration. -* **KV cache reuse must be disabled.** KV cache reuse is not supported on the sparse-attention path, so set `kv_cache_config.enable_block_reuse: false`. -* **Eagle3.** The MSA path supports one shared GQA draft layer with linear Eagle3 drafting. Other MTP and tree-drafting configurations are unsupported. Keep NVFP4 KV and piecewise CUDA graph qualification separate: the existing PCG accuracy cases use FP8 KV. +* **Weight and KV precision.** BF16, MXFP8, and mixed MXFP8/NVFP4 checkpoints are supported. On SM100/SM103/SM107, the MSA backend supports FP8 and NVFP4 KV cache. NVFP4 storage is used in sparse target layers; dense target layers and the shared Eagle3 draft layer use FP8. The curated BF16-KV YAML remains a separate starting configuration. +* **KV cache reuse.** KV cache block reuse is supported on the sparse-attention path for text and multimodal requests; enable it with `kv_cache_config.enable_block_reuse: true`. The curated throughput YAML leaves it disabled. See [KV cache reuse](#kv-cache-reuse). +* **Speculative decoding.** The MSA path supports one shared GQA draft layer with linear Eagle3 drafting and the standalone DSpark drafter bundled with MiniMax-M3.1-Flash (see [Speculative decoding with the DSpark drafter](#speculative-decoding-with-the-dspark-drafter)). MTP and tree-drafting configurations are unsupported. Keep NVFP4 KV and piecewise CUDA graph qualification separate: the existing PCG accuracy cases use FP8 KV. * **`max_seq_len` must be capped for CUDA graphs.** The dense GQA expansion in the first few attention layers and the per-Q FP32 expansion in the sparse decode kernel allocate temporary tensors whose size grows linearly with the warmup decode's `max_k`. If `max_seq_len` is left at the checkpoint default, that `max_k` follows `max_position_embeddings` (1M for MXFP8, 512K for BF16) and the resulting gigabyte-scale single-allocation request exceeds the caching allocator's CUDA-graph-safe path, so capture fails with `cudaErrorStreamCaptureUnsupported` / OOM. The curated YAML therefore sets `max_seq_len` to a small value just above ISL+OSL (`2068` for the 1k/1k benchmark) so CUDA graphs can capture cleanly. Raise it for longer-context workloads, but expect a corresponding cut in `max_batch_size`. * **Parallelism.** MoE experts run with expert parallelism. The attention layers support both Tensor-Expert Parallelism (TEP) and Data-Expert Parallelism (DEP, via `enable_attention_dp: true`). The overlap scheduler is enabled by default. * **Multimodal.** `MiniMaxM3SparseForConditionalGeneration` supports text, image, and video inputs. The text decoder is also usable standalone (text-only) via the `MiniMaxM3SparseForCausalLM` architecture. Text requests need only the tokenizer. The checkpoint's Hugging Face processor is loaded on the first image or video request. If `AutoProcessor` cannot build a processor with an image processor for the checkpoint directory (for example, it returns only a tokenizer because no loadable processor code is shipped), that request fails with a `ValueError` saying that MiniMax-M3 image and video inputs need an HF processor with an image processor. Text requests keep working. To serve images, point the model at a directory whose processor loads with `AutoProcessor`. +## Serving MiniMax-M3.1-Flash + +MiniMax-M3.1-Flash is served with the same `MiniMaxM3SparseForConditionalGeneration` architecture, the same CLI flags, and the same YAML options as MiniMax-M3; everything above applies to it, except that all 60 of its attention layers are sparse (its `sparse_attention_freq` has no dense entries), so there are no dense target layers. The examples use a local copy of the checkpoint at `/models/MiniMax-M3.1-Flash`. + +### Checkpoint + +The checkpoint ships MXFP8 base layers with NVFP4 routed experts in the compressed-tensors packaging described in [Mixed MXFP8/NVFP4](#mixed-mxfp8nvfp4); the target loads natively in that mixed precision. `moe_config.backend: TRTLLM` is the serving choice on SM100/SM103/SM107 and `CUTLASS` the portable reference. The DSpark drafter is bundled in the `dspark/` subdirectory of the checkpoint, is loaded only when `speculative_config` selects it, and runs in BF16 (its MXFP8 linears are dequantized at load). The checkpoint ships its own Hugging Face processor, which the server loads on the first image or video request (see **Multimodal** under [Feature Support Notes](#feature-support-notes)). + +### KV cache reuse + +KV cache block reuse is supported on the sparse-attention path for text and multimodal requests, with or without speculative decoding. Enable it with `kv_cache_config.enable_block_reuse: true`; the MSA backend matches reused prefixes at token granularity on its 128-token pages, and a reuse hit or a chunk boundary inside an image or video placeholder span is handled. With the DSpark drafter enabled, the target's cache is reused as usual while the drafter keeps its own per-request context, so after a partial-prefix hit the drafter can run without context for the rest of its current 64-token page; this lowers the acceptance length for a few steps and does not change outputs. + +### NVFP4 KV cache on SM107 + +The [NVFP4 KV-cache configuration](#nvfp4-kv-cache-configuration) above now also runs on SM107: the MSA backend and the Triton NVFP4 sparse decode kernels accept SM100, SM103, and SM107, and the generic SM107 fallback from NVFP4 to FP8 KV cache does not apply to MiniMax-M3. Dense target layers and the shared Eagle3 draft layer keep FP8 caches as before. + +### Speculative decoding with the DSpark drafter + +DSpark is a target-dependent block drafter: it reads hidden states from five target layers and proposes a block of up to 7 draft tokens per step, which the target verifies in one forward. Select it with `decoding_type: DSpark` and point `speculative_model` at the bundled drafter. The target attention backend must be `TRTLLM` (`FLASHINFER` is rejected for one-engine speculation), and the drafter's KV cache is a separate pool sized from `max_batch_size` and the drafter's sliding attention window rather than from `max_seq_len`. The following YAML runs on 4 SM100/SM103/SM107 GPUs with NVFP4 KV cache and block reuse enabled at the checkpoint's 1M context; set `max_seq_len` to the longest prompt plus output you serve: + +```yaml +tensor_parallel_size: 4 +moe_expert_parallel_size: 4 +trust_remote_code: true +attn_backend: TRTLLM +moe_config: + backend: TRTLLM +max_batch_size: 8 +max_num_tokens: 16384 +max_seq_len: 1048576 +enable_chunked_prefill: true +cuda_graph_config: + enable_padding: true + max_batch_size: 8 +kv_cache_config: + dtype: nvfp4 + enable_block_reuse: true + free_gpu_memory_fraction: 0.9 +sparse_attention_config: + algorithm: minimax_m3 + implementation: msa + indexer_kv_dtype: fp8 + fuse_qkv_index_projection: true +speculative_config: + decoding_type: DSpark + max_draft_len: 7 + speculative_model: /models/MiniMax-M3.1-Flash/dspark +``` + +`max_draft_len` is the number of draft tokens per step, up to the drafter's native block of 7; smaller values trade acceptance length for cheaper drafting at high concurrency. `block_size` defaults to the drafter's value and must equal `max_draft_len`, so to draft 3 tokens set both `max_draft_len: 3` and `block_size: 3`. Launch the server with the same command as in [Launch the TensorRT LLM Server](#launch-the-tensorrt-llm-server), pointing `$MODEL` at `/models/MiniMax-M3.1-Flash` and `--extra_llm_api_options` at this file, and adding `--served_model_name MiniMax-M3.1-Flash` so requests can name the model as in the examples below; a single-node deployment uses `-N 1 --ntasks 4 --ntasks-per-node 4 --gres=gpu:4`. The startup log reports the drafter's cache as `Separate draft KV cache of the standalone drafter is funded for ...`. + +In disaggregated serving the drafter's cache is not transferred with the prompt, so a generation worker starts drafting without prompt context and draft acceptance is reduced until its attention window fills; compare against generation workers without speculative decoding before enabling it there. + +### Thinking mode and reasoning parsing + +The MiniMax-M3.1-Flash chat template has no thinking-mode switch. It always renders the adaptive instructions (`Current thinking mode: adaptive`) and ends the prompt at the assistant header without prefilling `` or ``, so the model decides per turn whether to emit a reasoning block or a bare `` before its answer. A `thinking_mode` key in `chat_template_kwargs` is ignored by this template; the `enabled` / `disabled` / `adaptive` switch belongs to the upstream MiniMax-M3 checkpoints' template. The only key this template reads is `reasoning_effort` (alias `effort`), one of `low`, `medium`, `high`, `xhigh`, or `max`, rendered into an `` tag of the system prompt. MiniMaxAI recommends sampling with `temperature: 1.0` and `top_p: 0.95`. A plain request needs no template keys; this one also asks for high reasoning effort: + +```bash +curl http://localhost:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "MiniMax-M3.1-Flash", + "messages": [{"role": "user", "content": "What is the capital of France?"}], + "chat_template_kwargs": {"reasoning_effort": "high"}, + "temperature": 1.0, + "top_p": 0.95, + "max_tokens": 1024 + }' +``` + +With `--reasoning_parser minimax_m3` the server resolves the reasoning state from the rendered prompt: nothing is prefilled, so parsing starts outside reasoning and the model's own `...` block is returned in `reasoning_content` with the answer in `content`; the tags themselves are never returned, and a turn without a reasoning block has an empty `reasoning_content`. Reasoning cut off by `max_tokens` is returned as `reasoning_content` with empty `content`. Streaming responses split the deltas the same way (`reasoning_content` deltas, then `content` deltas, with no tag text leaking into `content`), so a client sees the same fields whether or not it streams. `trtllm-eval` applies the template with `--apply_chat_template` and needs no `--chat_template_kwargs`. The `/v1/completions` endpoint bypasses the chat template and returns the raw tags. + ## Deployment Steps ### NVFP4 KV-cache configuration -For an SM100/SM103 deployment with a compatible checkpoint, use these overrides: +For an SM100/SM103/SM107 deployment with a compatible checkpoint, use these overrides: ```yaml kv_cache_config: dtype: nvfp4 - enable_block_reuse: false sparse_attention_config: algorithm: minimax_m3 implementation: msa @@ -113,12 +195,14 @@ sparse_attention_config: NVFP4 prefill uses MSA kernels, while decode uses TensorRT LLM's Triton sparse kernels inside the MSA backend. The legacy `implementation: triton` backend rejects NVFP4 KV and recommends MSA for its supported configurations too. -Both SM100 and SM103 support NVFP4; GB300-measured decode tuning applies only -to SM103. MSA requires 128-token sparse pages and the packaged `fmha_sm100` +SM100, SM103, and SM107 support NVFP4; GB300-measured decode tuning applies +only to SM103. MSA requires 128-token sparse pages and the packaged `fmha_sm100` dependency. The indexer stays FP8 in this configuration. -For mixed MXFP8/NVFP4 weights, select `moe_config.backend: CUTLASS` as in the -accuracy tests. KV-cache quantization does not choose the MoE weight backend. +For mixed MXFP8/NVFP4 weights, `moe_config.backend: TRTLLM` is the serving +choice on SM100 and SM103 and `CUTLASS` the portable reference; both apply the +same static expert recipe. KV-cache quantization does not choose the MoE weight +backend. Full weight reload refreshes the quantization and aligned prefill scale buffers in place so CUDA-graph replay retains valid addresses. @@ -214,7 +298,7 @@ The parallelism, batch, KV-cache, sparse-attention, and CUDA-graph settings all #### `--extra_llm_api_options` YAML options -The curated config sets all the YAML knobs; this section only documents the two MiniMax-M3-specific constraints. Every other field in the YAML is a standard TensorRT LLM throughput knob — see the [`TorchLlmArgs` class](https://nvidia.github.io/TensorRT-LLM/llm-api/reference.html#tensorrt_llm.llmapi.TorchLlmArgs) for the full reference. +The curated config sets all the YAML knobs; this section only documents the MiniMax-M3-specific settings. Every other field in the YAML is a standard TensorRT LLM throughput knob — see the [`TorchLlmArgs` class](https://nvidia.github.io/TensorRT-LLM/llm-api/reference.html#tensorrt_llm.llmapi.TorchLlmArgs) for the full reference. ##### `sparse_attention_config.algorithm` @@ -222,7 +306,7 @@ Must be set to `minimax_m3`. There is no dense fallback for the MiniMax-M3 spars ##### `kv_cache_config.enable_block_reuse` -Must be `false`. The sparse-attention path does not support KV cache block reuse across requests with shared prefixes. +Optional. KV cache block reuse is supported on the sparse-attention path; the curated throughput YAML leaves it disabled. Set it to `true` for workloads with shared prefixes (multi-turn chat, agent loops). See [KV cache reuse](#kv-cache-reuse). ## Testing API Endpoint @@ -286,7 +370,6 @@ The `message` object contains the visible answer in `content` and, when the mode ## Troubleshooting Tips * **Sparse attention not applied:** Ensure `sparse_attention_config.algorithm` is set to `minimax_m3` in the YAML file passed to `--extra_llm_api_options`. -* **KV cache reuse errors:** Confirm `kv_cache_config.enable_block_reuse` is `false`; KV cache reuse is not supported on the sparse-attention path. * **Model fails to load:** Make sure `--trust_remote_code` is set and that the checkpoint path is correct and fully downloaded (Git LFS). * **Multi-node startup hangs or ranks can't find each other:** Verify that the model weights, the curated YAML, and all mounted paths resolve identically on both nodes (shared filesystem), that the total Slurm task count equals `tensor_parallel_size` (`8`), and that the inter-node interconnect (e.g., InfiniBand) is healthy. * **Reasoning/tool output not parsed:** Verify `--reasoning_parser minimax_m3`, `--tool_parser minimax_m3`, and `--chat_template $MODEL/chat_template.jinja` are all passed. diff --git a/docs/source/features/sparse-attention.md b/docs/source/features/sparse-attention.md index b075a127e819..5c22a30d3eca 100644 --- a/docs/source/features/sparse-attention.md +++ b/docs/source/features/sparse-attention.md @@ -408,7 +408,6 @@ For NVFP4 KV cache, select MSA explicitly: ```yaml kv_cache_config: dtype: nvfp4 - enable_block_reuse: false sparse_attention_config: algorithm: minimax_m3 implementation: msa @@ -422,9 +421,10 @@ within the MSA backend. This does not select `implementation: triton`. The optional `fuse_qkv_index_projection: true` supports FP8 and NVFP4 main caches with an FP8 indexer. -The sparse path has no dense fallback and does not support KV-cache reuse. -Eagle3 supports one shared draft layer and linear drafting; other MTP/tree -configurations are unsupported. NVFP4 KV with piecewise CUDA graphs is not +The sparse path has no dense fallback; KV-cache block reuse is supported. +Eagle3 supports one shared draft layer and linear drafting, and a standalone +DSpark drafter is supported; MTP and tree configurations are unsupported. +NVFP4 KV with piecewise CUDA graphs is not qualified by the existing FP8-KV PCG tests. See the [MiniMax-M3 deployment guide](../deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md) for supported checkpoints and parallel deployment settings. diff --git a/docs/source/models/supported-models.md b/docs/source/models/supported-models.md index d6943819348c..5bba21b65692 100644 --- a/docs/source/models/supported-models.md +++ b/docs/source/models/supported-models.md @@ -35,7 +35,7 @@ The following is a table of supported models for the PyTorch backend: | `Llama4ForConditionalGeneration` | Llama 4 | `meta-llama/Llama-4-Scout-17B-16E-Instruct` | | `MBartForConditionalGeneration` | mBART | `facebook/mbart-large-50-many-to-one-mmt` | | `MiniCPMV4_6ForConditionalGeneration` [^14]| MiniCPM-V 4.6 | `openbmb/MiniCPM-V-4.6` | -| `MiniMaxM3SparseForConditionalGeneration` [^12]| MiniMax-M3 | `MiniMaxAI/MiniMax-M3` | +| `MiniMaxM3SparseForConditionalGeneration` [^12]| MiniMax-M3, MiniMax-M3.1-Flash | `MiniMaxAI/MiniMax-M3` | | `MistralForCausalLM` | Mistral | `mistralai/Mistral-7B-v0.1` | | `MixtralForCausalLM` | Mixtral | `mistralai/Mixtral-8x7B-v0.1` | | `MllamaForConditionalGeneration` | Llama 3.2 | `meta-llama/Llama-3.2-11B-Vision` | @@ -83,7 +83,7 @@ statuses for the same architecture in the two matrices. | `Gemma4ForConditionalGeneration` | Yes | Yes | Yes | No | Yes | MTP | Yes | No | Yes | Yes | Yes | | `Gemma4UnifiedForConditionalGeneration` | Yes | Yes | Yes | No | Yes | No | Yes | No | Yes | Yes | Yes | | `Step3p7ForConditionalGeneration`| Yes | Yes | Yes | Untested | Untested | MTP | Yes | No | Yes | Yes | Yes | -| `MiniMaxM3SparseForConditionalGeneration` [^12] | Yes | Yes | Yes | Untested | Untested | EAGLE-3 (Linear) | Yes | No | N/A | Yes | Yes | +| `MiniMaxM3SparseForConditionalGeneration` [^12] | Yes | Yes | Yes | Untested | Yes | EAGLE-3 (Linear), DSpark | Yes | Yes | N/A | Yes | Yes | [^1]: Chunked Prefill for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120. [^2]: KV cache reuse for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120/SM121 and in BF16/FP8 KV cache dtype. @@ -92,7 +92,7 @@ statuses for the same architecture in the two matrices. [^9]: Audio modality only supported on E2B/E4B variants. [^10]: Audio requires a checkpoint with a `sound_config` and is supported only on the full (non-disaggregated) model path, not the EPD disaggregated path. [^11]: DeepSeek-V4 is only supported on Blackwell GPUs (`SM100+`). See the [DeepSeek-V4 example README](source:examples/models/core/deepseek_v4/README.md) for setup and parallelism. -[^12]: Supports text, image, and video inputs over the block-sparse attention path. The published MXFP8 checkpoint is dequantized on load so the runtime sees an effectively BF16 model. The text decoder is also usable standalone (text-only) via the `MiniMaxM3SparseForCausalLM` architecture. KV cache reuse and MTP are not supported on the sparse-attention path in this release. +[^12]: Supports text, image, and video inputs over the block-sparse attention path. BF16, MXFP8, and mixed MXFP8/NVFP4 (NVFP4 routed experts) checkpoints load natively; the MiniMax-M3.1-Flash checkpoint uses the mixed packaging and bundles a DSpark drafter. The text decoder is also usable standalone (text-only) via the `MiniMaxM3SparseForCausalLM` architecture. KV cache reuse is supported on the sparse-attention path, including multimodal requests and with speculative decoding; MTP is not supported. See the [MiniMax-M3 deployment guide](../deployment-guide/deployment-guide-for-minimax-m3-on-trtllm.md#serving-minimax-m31-flash). [^13]: The Cosmos 3 family also supports visual generation through the VisualGen API. See [Visual Generation Models](#visual-generation-models). [^14]: Requires `transformers>=5.7.0`: MiniCPM-V 4.6 was upstreamed into transformers as a native model type (`minicpmv4_6`) and the checkpoint ships no remote code (`auto_map`) to fall back on. The Qwen3.5-hybrid text tower runs in BF16. Image, video, and text inputs are supported in this release (video reuses the same NaViT-packed vision path as image via `MiniCPMV4_6InputProcessor`). [^15]: Kimi K3 is only supported on NVIDIA Blackwell GPUs (`SM100` family); see the [Kimi K3 deployment guide](../deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md). Which recipes fit is set by the attention layout rather than by the GPU count: DEP16 (`enable_attention_dp: true`) replicates the BF16 non-expert weights on every rank (114 GB) on top of the MXFP4 routed experts at 16-way expert parallelism (90 GB), needing 210 GB per rank, so it requires GB300-class per-GPU memory. TEP16 (`enable_attention_dp: false`) shards those non-expert weights instead and needs 115 GB per rank; it is validated end-to-end on GB200 (`SM100`) at 16 GPUs. On B200 (`SM100`), kernel and module support is functional and covered by CI. Note that the FP8 weight-read path (TRTLLM-14765) does not relax the DEP16 requirement: the conversion runs after the weights are already resident in BF16, so it lowers the steady-state footprint but not the load-time peak. @@ -144,7 +144,7 @@ not used to mark a feature as supported here. | `Qwen3VLForConditionalGeneration` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | L + I + V | | `Qwen3VLMoeForConditionalGeneration` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | L + I + V | | `Step3p7ForConditionalGeneration` | Yes | Yes | Untested | Yes | Untested | Yes | Untested | L + I | -| `MiniMaxM3SparseForConditionalGeneration` [^12] | Yes | Yes | Untested | Yes | No | Yes | Untested | L + I + V | +| `MiniMaxM3SparseForConditionalGeneration` [^12] | Yes | Yes | Yes | Yes | Yes | Yes | Untested | L + I + V | | `Cosmos3ForConditionalGeneration` [^13] | Yes | Yes | Yes | Yes | Yes | Untested | Untested | L + I + V | | `Qwen3_5ForConditionalGeneration` | Yes | Yes | Untested | Yes | No | Untested | Yes | L + I + V | | `Qwen3_5MoeForConditionalGeneration` | Yes | Yes | Untested | Yes | No | Untested | Yes | L + I + V | From 7193d64e3d6e4ccef589a972968254142dfe96fd Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:17:14 -0700 Subject: [PATCH 13/38] [None][fix] Serve MiniMax-M3 chat with the tokenizer's template and tokenization cache `TransformersTokenizer` delegates `num_special_tokens_to_add` to the wrapped tokenizer (the base class raises), so the MiniMax-M3 input processor's tokenization-cache check runs. `resolve_hf_chat_template` falls through to the tokenizer's template when the processor's `chat_template` is None. Signed-off-by: Pranav Shrestha --- tensorrt_llm/inputs/utils.py | 4 ++- tensorrt_llm/tokenizer/tokenizer.py | 4 +++ .../inputs/test_chat_template_dispatch.py | 33 +++++++++++++++++++ .../unittest/llmapi/test_tokenizer_decode.py | 32 ++++++++++++++++++ 4 files changed, 72 insertions(+), 1 deletion(-) diff --git a/tensorrt_llm/inputs/utils.py b/tensorrt_llm/inputs/utils.py index 5d1e503b6d44..9ac9b8415602 100644 --- a/tensorrt_llm/inputs/utils.py +++ b/tensorrt_llm/inputs/utils.py @@ -658,7 +658,9 @@ def resolve_hf_chat_template( return chat_template # 2. If tool is not provided, use the processor's default chat template - if not tools and processor and hasattr(processor, 'chat_template'): + # (a processor whose chat_template is None has none: fall through to the tokenizer's) + if not tools and processor and getattr(processor, 'chat_template', + None) is not None: return processor.chat_template # 3. If tool is provided, use the tool diff --git a/tensorrt_llm/tokenizer/tokenizer.py b/tensorrt_llm/tokenizer/tokenizer.py index 84b9d588cc21..e15ed3d8fa82 100644 --- a/tensorrt_llm/tokenizer/tokenizer.py +++ b/tensorrt_llm/tokenizer/tokenizer.py @@ -293,6 +293,10 @@ def name_or_path(self) -> str: def encode(self, text: str, *args, **kwargs) -> List[int]: return self.tokenizer.encode(text, *args, **kwargs) + def num_special_tokens_to_add(self, pair: bool = False) -> int: + # PreTrainedTokenizerBase.num_special_tokens_to_add raises NotImplementedError. + return self.tokenizer.num_special_tokens_to_add(pair) + def decode(self, token_ids: List[int], *args, **kwargs) -> str: return self.tokenizer.decode(token_ids, *args, **kwargs) diff --git a/tests/unittest/inputs/test_chat_template_dispatch.py b/tests/unittest/inputs/test_chat_template_dispatch.py index 78d77b571e1e..3bbc838cfc1a 100644 --- a/tests/unittest/inputs/test_chat_template_dispatch.py +++ b/tests/unittest/inputs/test_chat_template_dispatch.py @@ -21,6 +21,7 @@ apply_chat_template, async_apply_chat_template, interleave_mm_placeholders, + resolve_hf_chat_template, ) from tensorrt_llm.tokenizer import TransformersTokenizer @@ -556,3 +557,35 @@ async def fake_async_apply(**kwargs): assert token_ids == [1, 2, 3] # mm_data is the data dict, not the (mm_data, mm_embeddings) tuple. assert mm_data == {"image": ["data"]} + + +class TestResolveHfChatTemplate: + def test_processor_without_a_template_falls_back_to_the_tokenizer(self): + """A processor with ``chat_template`` None has no template of its own. + + The MiniMax-M3 VL processor loads that way; the tokenizer's template (``chat_template.jinja``) is + used instead of None. + """ + + class Tokenizer: + def get_chat_template(self, chat_template=None, tools=None): + return "tokenizer-template" + + class Processor: + chat_template = None + + assert ( + resolve_hf_chat_template(Tokenizer(), Processor(), None, None) == "tokenizer-template" + ) + + def test_processor_template_wins_without_tools(self): + class Tokenizer: + def get_chat_template(self, chat_template=None, tools=None): + raise AssertionError("the processor's template should be used") + + class Processor: + chat_template = "processor-template" + + assert ( + resolve_hf_chat_template(Tokenizer(), Processor(), None, None) == "processor-template" + ) diff --git a/tests/unittest/llmapi/test_tokenizer_decode.py b/tests/unittest/llmapi/test_tokenizer_decode.py index b1df835b29b6..1a40b28eb103 100644 --- a/tests/unittest/llmapi/test_tokenizer_decode.py +++ b/tests/unittest/llmapi/test_tokenizer_decode.py @@ -42,3 +42,35 @@ def test_hf_decode_incrementally_recovers_from_invalid_prefix() -> None: text, states = tokenizer.hf_decode_incrementally([token_id], text, states) assert text == "\N{NO-BREAK SPACE}»" + + +def _byte_fallback_tokenizer() -> TransformersTokenizer: + """A tokenizer with one added special token, wrapped the way the LLM API wraps it.""" + backend = Tokenizer( + WordLevel( + { + "<0xC2>": 0, + "<0xA0>": 1, + "»": 2, + "": 3, + "": 4, + }, + unk_token="", + ) + ) + backend.decoder = Sequence([ByteFallback(), Fuse()]) + backend.add_special_tokens([""]) + return TransformersTokenizer(PreTrainedTokenizerFast(tokenizer_object=backend)) + + +def test_num_special_tokens_to_add_delegates_to_the_wrapped_tokenizer() -> None: + """The wrapper answers from the tokenizer it wraps. + + ``PreTrainedTokenizerBase.num_special_tokens_to_add`` raises NotImplementedError; the MiniMax-M3 input + processor's tokenization-cache check calls it on the wrapper. + """ + tokenizer = _byte_fallback_tokenizer() + assert tokenizer.num_special_tokens_to_add() == tokenizer.tokenizer.num_special_tokens_to_add() + assert tokenizer.num_special_tokens_to_add( + pair=True + ) == tokenizer.tokenizer.num_special_tokens_to_add(True) From 59256ca18a3dcfeea96d270a19dc5ab347b9af49 Mon Sep 17 00:00:00 2001 From: Pranav Shrestha Date: Thu, 22 Oct 2026 18:17:14 -0700 Subject: [PATCH 14/38] [None][chore] Set MiniMax-M3.1-Flash side-branch release version to 1.4.0a1 The side-branch release job derives its wheel and container version from tensorrt_llm/version.py, so the branch gets its own alpha base instead of sharing the main nightlies' 1.4.0rc1 line. Signed-off-by: Pranav Shrestha --- README.md | 2 +- examples/constraints.txt | 2 +- tensorrt_llm/version.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9d53b2d7e71c..3a3408812654 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ TensorRT LLM [![python](https://img.shields.io/badge/python-3.10-green)](https://www.python.org/downloads/release/python-31012/) [![cuda](https://img.shields.io/badge/cuda-13.4.1-green)](https://developer.nvidia.com/cuda-downloads) [![torch](https://img.shields.io/badge/torch-2.14.0-green)](https://pytorch.org) -[![version](https://img.shields.io/badge/release-1.4.0rc1-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py) +[![version](https://img.shields.io/badge/release-1.4.0a1-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py) [![license](https://img.shields.io/badge/license-Apache%202-blue)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/LICENSE) [Architecture](https://nvidia.github.io/TensorRT-LLM/developer-guide/overview.html)   |   [Performance](https://nvidia.github.io/TensorRT-LLM/developer-guide/perf-overview.html)   |   [Examples](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html)   |   [Documentation](https://nvidia.github.io/TensorRT-LLM/)   |   [Roadmap](https://github.com/NVIDIA/TensorRT-LLM/issues?q=is%3Aissue%20state%3Aopen%20label%3Aroadmap) diff --git a/examples/constraints.txt b/examples/constraints.txt index ed3b1ce13c2f..7f2454f023f5 100644 --- a/examples/constraints.txt +++ b/examples/constraints.txt @@ -1,3 +1,3 @@ -tensorrt_llm==1.4.0rc1 +tensorrt_llm==1.4.0a1 evaluate~=0.4.1 rouge_score~=0.1.2 diff --git a/tensorrt_llm/version.py b/tensorrt_llm/version.py index 5b50500a5053..37d3447dc1c4 100644 --- a/tensorrt_llm/version.py +++ b/tensorrt_llm/version.py @@ -12,4 +12,4 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -__version__ = "1.4.0rc1" +__version__ = "1.4.0a1" From 5d72b49999f738b881d76c1b3ae060bf1b6ba0a9 Mon Sep 17 00:00:00 2001 From: Yanchao Lu Date: Tue, 20 Oct 2026 17:21:36 +0800 Subject: [PATCH 15/38] [None][fix] Enforce container wheel version and source commit (#20564) Signed-off-by: Yanchao Lu (cherry picked from commit ee7c59aa78cf4f47c46528b79c8714fc4c526a54) Signed-off-by: Yanchao Lu --- docker/Dockerfile.multi | 14 ++- docker/common/validate_wheel_metadata.py | 60 ++++++++++ jenkins/BuildDockerImage.groovy | 2 +- jenkins/L0_MergeRequest.groovy | 17 +-- .../scripts/test_docker_wheel_version.py | 103 ++++++++++++++++ .../scripts/test_validate_wheel_metadata.py | 112 ++++++++++++++++++ 6 files changed, 294 insertions(+), 14 deletions(-) create mode 100644 docker/common/validate_wheel_metadata.py create mode 100644 tests/unittest/scripts/test_docker_wheel_version.py create mode 100644 tests/unittest/scripts/test_validate_wheel_metadata.py diff --git a/docker/Dockerfile.multi b/docker/Dockerfile.multi index c6712f3c8b4e..59883b84fe44 100644 --- a/docker/Dockerfile.multi +++ b/docker/Dockerfile.multi @@ -156,6 +156,7 @@ COPY setup.py requirements.txt requirements-dev.txt requirements-grpc-smg.txt co ENV CCACHE_DIR=/root/.cache/ccache # Build the TRT-LLM wheel ARG GIT_COMMIT +ARG TRT_LLM_VER ARG GITHUB_MIRROR="" ARG BUILD_WHEEL_ARGS="--clean" ARG BUILD_WHEEL_SCRIPT="scripts/build_wheel.py" @@ -167,9 +168,13 @@ RUN --mount=type=cache,target=/root/.cache/pip --mount=type=cache,target=${CCACH # Authenticates `git submodule update` and the CMake FetchContent clones # (3rdparty/fetch_content.json) that this build pulls from github.com. . docker/common/github_auth.sh && \ + set -- ${BUILD_WHEEL_ARGS} && \ + if [ "${BUILD_WHEEL_SCRIPT}" = "scripts/build_wheel.py" ] && [ -n "${TRT_LLM_VER}" ]; then \ + set -- "$@" --version-override "${TRT_LLM_VER}"; \ + fi && \ TRTLLM_BUILD_SOURCE_COMMIT=${GIT_COMMIT} \ GITHUB_MIRROR=${GITHUB_MIRROR} \ - python3 ${BUILD_WHEEL_SCRIPT} ${BUILD_WHEEL_ARGS} + python3 ${BUILD_WHEEL_SCRIPT} "$@" FROM ${DEVEL_IMAGE} AS release @@ -215,6 +220,13 @@ ARG TARGETARCH ENV TRT_LLM_GIT_COMMIT=${GIT_COMMIT} \ TRT_LLM_VERSION=${TRT_LLM_VER} +# Check installed metadata without importing CUDA-linked modules. +RUN --mount=type=bind,source=docker/common/validate_wheel_metadata.py,target=/tmp/validate_wheel_metadata.py \ + --mount=type=bind,source=tensorrt_llm/version.py,target=/tmp/trtllm_source_version.py \ + python3 /tmp/validate_wheel_metadata.py \ + --expected-version "${TRT_LLM_VER}" --expected-commit "${GIT_COMMIT}" \ + --source-version-file /tmp/trtllm_source_version.py + # Generate OSS attribution file for release image RUN --mount=type=bind,source=scripts/generate_container_oss_attribution.sh,target=/mnt/gen_attribution.sh \ bash /mnt/gen_attribution.sh "release" "${TRT_LLM_VER}" "${TARGETARCH}" diff --git a/docker/common/validate_wheel_metadata.py b/docker/common/validate_wheel_metadata.py new file mode 100644 index 000000000000..167a18649224 --- /dev/null +++ b/docker/common/validate_wheel_metadata.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Check installed wheel provenance without importing CUDA-linked modules.""" + +import re +from argparse import ArgumentParser +from importlib.metadata import distribution +from pathlib import Path + +from packaging.version import Version + + +def validate_wheel_metadata( + expected_version: str, expected_commit: str, source_version_file: Path | None = None +) -> None: + """Check installed metadata, resolving relative versions from the build source file.""" + if not expected_version and not expected_commit: + return + if expected_version.startswith((".", "+")): + if source_version_file is None: + raise SystemExit("Relative versions require --source-version-file") + match = re.search(r'(?m)^__version__ = "([^"]+)"$', source_version_file.read_text()) + if not match: + raise SystemExit(f"Unable to read __version__ from {source_version_file}") + base_version = match.group(1) + expected_version = ( + base_version + if base_version.endswith(expected_version) + else base_version + expected_version + ) + wheel = distribution("tensorrt_llm") + if expected_version and Version(wheel.version) != Version(expected_version): + raise SystemExit( + f"TRT-LLM version mismatch: installed={wheel.version}, expected={expected_version}" + ) + if expected_commit: + if not re.fullmatch(r"[0-9a-fA-F]{40}", expected_commit): + raise SystemExit(f"Expected a full 40-character source commit SHA: {expected_commit}") + source_urls = [] + for entry in wheel.metadata.get_all("Project-URL", []): + label, _, url = entry.partition(",") + if label.strip() == "Source Commit": + source_urls.append(url.strip().lower()) + expected_url = f"https://github.com/NVIDIA/TensorRT-LLM/commit/{expected_commit}" + if source_urls != [expected_url.lower()]: + raise SystemExit( + f"TRT-LLM source commit mismatch: installed={source_urls or ''}, " + f"expected={expected_url}" + ) + + +if __name__ == "__main__": + parser = ArgumentParser(description=__doc__) + parser.add_argument("--expected-version", default="") + parser.add_argument("--expected-commit", default="") + parser.add_argument( + "--source-version-file", type=Path, help="Build-context version.py for relative overrides" + ) + args = parser.parse_args() + validate_wheel_metadata(args.expected_version, args.expected_commit, args.source_version_file) diff --git a/jenkins/BuildDockerImage.groovy b/jenkins/BuildDockerImage.groovy index b1313321c155..5f39c903dd71 100644 --- a/jenkins/BuildDockerImage.groovy +++ b/jenkins/BuildDockerImage.groovy @@ -976,7 +976,7 @@ pipeline { booleanParam( name: "useWheelFromBuildStage", defaultValue: false, - description: "Install the wheel from the build-stage tarball instead of compiling one inside the image. Read since Aug 2025 but never DECLARED, so params.useWheelFromBuildStage was always null and prepareWheelFromBuildStage() returned early on every run -- see nvbug 5433581, whose temporary kill switch was never reverted. Declared here so the flag is at least capable of being set; it stays false by default, and nothing turns it on. boltOptimizeWheel does NOT depend on it: a BOLT request carries itself past this gate for the SBSA release image only." + description: "Install the build-stage wheel instead of compiling inside the image. The declared job default is false; the nightly release caller explicitly enables this option. SBSA release images with boltOptimizeWheel enabled require the build-stage wheel independently of this flag." ) booleanParam( name: "boltOptimizeWheel", diff --git a/jenkins/L0_MergeRequest.groovy b/jenkins/L0_MergeRequest.groovy index ae9f1471dfe9..4916f68029ac 100644 --- a/jenkins/L0_MergeRequest.groovy +++ b/jenkins/L0_MergeRequest.groovy @@ -2803,6 +2803,9 @@ def launchStages(pipeline, reuseBuild, testFilter, enableFailFast, globalVars) script { def testStageName = "[Build-Docker-Images] Remote Run" stage(testStageName) { + if (runMode == "nightly_release") { + error "Build-Docker-Images does not support nightly_release; use Build-Release-Docker-Images" + } try { def branch = globalVars[BUILD_BRANCH] @@ -2815,9 +2818,7 @@ def launchStages(pipeline, reuseBuild, testFilter, enableFailFast, globalVars) def additionalParameters = [ 'branch': branch, 'action': "push", - 'triggerType': runMode == "nightly_release" ? - "nightly-release" : - (env.JOB_NAME ==~ /.*PostMerge.*/ ? "post-merge" : "pre-merge"), + 'triggerType': env.JOB_NAME ==~ /.*PostMerge.*/ ? "post-merge" : "pre-merge", 'runSanityCheck': env.JOB_NAME ==~ /.*PostMerge.*/, 'defaultTag': defaultTag, 'program_version_name': env.NSPECT_RELEASE_VERSION, @@ -2842,15 +2843,6 @@ def launchStages(pipeline, reuseBuild, testFilter, enableFailFast, globalVars) // built from source rather than downloaded. 'boltOptimizeWheel': true, ] - if (runMode == "nightly_release") { - additionalParameters += [ - 'buildInternalRelease': false, - 'buildCiImage': false, - 'buildNgcRelease': true, - 'useWheelFromBuildStage': false, - 'wait_success_seconds': "", - ] - } if (params.release_type_id) { additionalParameters['release_type_id'] = params.release_type_id } @@ -2910,6 +2902,7 @@ def launchStages(pipeline, reuseBuild, testFilter, enableFailFast, globalVars) if (runMode == "nightly_release") { additionalParameters += [ 'buildNgcRelease': true, + 'useWheelFromBuildStage': true, 'wait_success_seconds': "", ] additionalParameters['program_version_name'] = env.NSPECT_RELEASE_VERSION diff --git a/tests/unittest/scripts/test_docker_wheel_version.py b/tests/unittest/scripts/test_docker_wheel_version.py new file mode 100644 index 000000000000..41a5722ebcf1 --- /dev/null +++ b/tests/unittest/scripts/test_docker_wheel_version.py @@ -0,0 +1,103 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CPU-only checks for Docker source and downloaded wheel arguments.""" + +import importlib.util +import os +import re +import shlex +import subprocess +from argparse import ArgumentParser +from pathlib import Path + +import pytest + +pytestmark = pytest.mark.cpu_only +ROOT = Path(__file__).resolve().parents[3] + + +@pytest.fixture(scope="module") +def wheel_command() -> str: + """Extract the wheel-stage shell command, excluding BuildKit mounts and comments.""" + dockerfile = (ROOT / "docker/Dockerfile.multi").read_text() + stage = dockerfile.split(" AS wheel\n", 1)[1].split("\nFROM ", 1)[0] + stage = "\n".join(line for line in stage.splitlines() if not line.lstrip().startswith("#")) + run = next(line for line in stage.replace("\\\n", "").splitlines() if line.startswith("RUN ")) + return re.sub(r"^RUN (?:--mount=\S+\s+)*", "", run) + + +@pytest.mark.parametrize( + "script,version,extra_args,expected", + [ + ("build_wheel.py", "", "--clean", "1.4.0a0"), + ("build_wheel.py", "1.5.0", "--clean -G Ninja -j 2", "1.5.0"), + ("build_wheel.py", "1.4.0alpha0.dev202610190100", "", "1.4.0a0.dev202610190100"), + ("build_wheel.py", ".dev202610190100", "--clean", "1.4.0a0.dev202610190100"), + ("build_wheel.py", "+local", "--clean", "1.4.0a0+local"), + ("build_wheel.py", "1.5.0", "--version-override 1.3.0", "1.5.0"), + ("build_wheel.py", "", "--version-override 1.3.0", "1.3.0"), + ("get_wheel_from_package.py", "1.5.0", "--arch x86_64 --artifact_path artifacts", None), + ( + "get_wheel_from_package.py", + "1.5.0", + "--arch sbsa --artifact_path artifacts --bolt-branch main --bolt-profile-ref pinned", + None, + ), + ], +) +def test_docker_wheel_arguments( + wheel_command: str, + tmp_path: Path, + script: str, + version: str, + extra_args: str, + expected: str | None, +) -> None: + """Check Docker argument forwarding against the real builder and download parsers.""" + auth = tmp_path / "docker/common/github_auth.sh" + auth.parent.mkdir(parents=True) + auth.write_text("") + commit = "abcdef01" * 5 + result = subprocess.run( + [ + "sh", + "-eu", + "-c", + 'python3() { printf "%s\\n" "$TRTLLM_BUILD_SOURCE_COMMIT" "$@"; }; ' + wheel_command, + ], + cwd=tmp_path, + env={ + **os.environ, + "BUILD_WHEEL_SCRIPT": f"scripts/{script}", + "BUILD_WHEEL_ARGS": extra_args, + "TRT_LLM_VER": version, + "GIT_COMMIT": commit, + "GITHUB_MIRROR": "", + }, + capture_output=True, + text=True, + check=True, + timeout=60, + ) + source_commit, invoked_script, *arguments = result.stdout.splitlines() + assert source_commit == commit + assert invoked_script == f"scripts/{script}" + original = shlex.split(extra_args) + assert arguments[: len(original)] == original + + spec = importlib.util.spec_from_file_location( + script.removesuffix(".py"), ROOT / "scripts" / script + ) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + parser = ArgumentParser() + module.add_arguments(parser) + parsed = parser.parse_args(arguments) + if expected is None: + assert arguments == original + else: + version_file = tmp_path / "tensorrt_llm/version.py" + version_file.parent.mkdir() + version_file.write_text('__version__ = "1.4.0a0"\n') + module.apply_version_override(tmp_path, parsed.version_override) + assert version_file.read_text() == f'__version__ = "{expected}"\n' diff --git a/tests/unittest/scripts/test_validate_wheel_metadata.py b/tests/unittest/scripts/test_validate_wheel_metadata.py new file mode 100644 index 000000000000..d58e23b2086c --- /dev/null +++ b/tests/unittest/scripts/test_validate_wheel_metadata.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CPU-only checks against installed wheel metadata fixtures.""" + +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +pytestmark = pytest.mark.cpu_only + +SCRIPT = Path(__file__).resolve().parents[3] / "docker/common/validate_wheel_metadata.py" +COMMIT = "abcdef01" * 5 +OTHER_COMMIT = "12345678" * 5 +SOURCE_URL = f"Source Commit, https://github.com/NVIDIA/TensorRT-LLM/commit/{COMMIT}" + + +@pytest.mark.parametrize( + "installed_version,project_urls,expected_version,expected_commit,error", + [ + ("1.4.0a0", [SOURCE_URL], "1.4.0a0", COMMIT, None), + ("1.4.0a0", [SOURCE_URL], "1.4.0alpha0", COMMIT.upper(), None), + ("1.4.0a0", [SOURCE_URL], "1.4.0a0", OTHER_COMMIT, "source commit mismatch"), + ("1.4.0a0", [SOURCE_URL], "1.4.0a0.dev202610190100", COMMIT, "version mismatch"), + ("1.4.0a0", [], "1.4.0a0", COMMIT, "source commit mismatch"), + ("1.4.0a0", ["Homepage, https://example.com"], "", COMMIT, "source commit mismatch"), + ("1.4.0a0", [SOURCE_URL, SOURCE_URL], "", COMMIT, "source commit mismatch"), + ("1.4.0a0", [SOURCE_URL], "", COMMIT[:7], "full 40-character"), + ("1.4.0a0", [SOURCE_URL], "", COMMIT, None), + ("1.4.0a0", [], "1.4.0a0", "", None), + (None, [], "", "", None), + ], +) +def test_installed_metadata( + tmp_path, installed_version, project_urls, expected_version, expected_commit, error +): + """Accept matching metadata and reject version or source commit mismatches.""" + if installed_version: + metadata = tmp_path / "tensorrt_llm-1.4.0a0.dist-info" / "METADATA" + metadata.parent.mkdir() + metadata.write_text( + f"Metadata-Version: 2.1\nName: tensorrt_llm\nVersion: {installed_version}\n" + + "".join(f"Project-URL: {url}\n" for url in project_urls) + ) + # Any accidental package import must fail even on a GPU-capable test runner. + package = tmp_path / "tensorrt_llm" + package.mkdir() + (package / "__init__.py").write_text('raise RuntimeError("must not import TRT-LLM")\n') + result = subprocess.run( + [ + sys.executable, + str(SCRIPT), + "--expected-version", + expected_version, + "--expected-commit", + expected_commit, + ], + env={**os.environ, "PYTHONPATH": str(tmp_path)}, + capture_output=True, + text=True, + check=False, + timeout=60, + ) + if error: + assert result.returncode != 0 + assert error in result.stderr + else: + assert result.returncode == 0, result.stderr + + +@pytest.mark.parametrize( + "base_version,override,installed_version,error", + [ + ("1.4.0a0", ".dev202610190100", "1.4.0a0.dev202610190100", None), + ("1.4.0a0", "+local", "1.4.0a0+local", None), + ("1.4.0a0", ".dev202610190100", "1.3.0a0.dev202610190100", "version mismatch"), + ("1.4.0a0", "+local", "1.3.0a0+local", "version mismatch"), + ("1.4.0a0.dev202610190100", ".dev202610190100", "1.4.0a0.dev202610190100", None), + ("1.4.0a0+local", "+local", "1.4.0a0+local", None), + ], +) +def test_relative_version(tmp_path, base_version, override, installed_version, error): + """Resolve suffixes from the source base, independently of installed metadata.""" + metadata = tmp_path / "tensorrt_llm-1.4.0a0.dist-info" / "METADATA" + metadata.parent.mkdir() + metadata.write_text( + f"Metadata-Version: 2.1\nName: tensorrt_llm\nVersion: {installed_version}\n" + ) + source_version = tmp_path / "version.py" + source_version.write_text(f'__version__ = "{base_version}"\n') + result = subprocess.run( + [ + sys.executable, + str(SCRIPT), + "--expected-version", + override, + "--source-version-file", + str(source_version), + ], + env={**os.environ, "PYTHONPATH": str(tmp_path)}, + capture_output=True, + text=True, + check=False, + timeout=60, + ) + if error: + assert result.returncode != 0 + assert error in result.stderr + else: + assert result.returncode == 0, result.stderr From 47ecfa7d2eea9a7e6cdc4b6546c0551ee338a39f Mon Sep 17 00:00:00 2001 From: Yanchao Lu Date: Fri, 23 Oct 2026 16:14:19 +0800 Subject: [PATCH 16/38] [None][chore] Set MiniMax-M3.1-Flash side-branch version to 1.4.0a0 Use the 1.4.0a0 base required by the side-branch release convention and keep the README badge and example dependency constraint aligned. Signed-off-by: Yanchao Lu --- README.md | 2 +- examples/constraints.txt | 2 +- tensorrt_llm/version.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 3a3408812654..336a08c8c5d4 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ TensorRT LLM [![python](https://img.shields.io/badge/python-3.10-green)](https://www.python.org/downloads/release/python-31012/) [![cuda](https://img.shields.io/badge/cuda-13.4.1-green)](https://developer.nvidia.com/cuda-downloads) [![torch](https://img.shields.io/badge/torch-2.14.0-green)](https://pytorch.org) -[![version](https://img.shields.io/badge/release-1.4.0a1-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py) +[![version](https://img.shields.io/badge/release-1.4.0a0-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py) [![license](https://img.shields.io/badge/license-Apache%202-blue)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/LICENSE) [Architecture](https://nvidia.github.io/TensorRT-LLM/developer-guide/overview.html)   |   [Performance](https://nvidia.github.io/TensorRT-LLM/developer-guide/perf-overview.html)   |   [Examples](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html)   |   [Documentation](https://nvidia.github.io/TensorRT-LLM/)   |   [Roadmap](https://github.com/NVIDIA/TensorRT-LLM/issues?q=is%3Aissue%20state%3Aopen%20label%3Aroadmap) diff --git a/examples/constraints.txt b/examples/constraints.txt index 7f2454f023f5..4da6e63f2a53 100644 --- a/examples/constraints.txt +++ b/examples/constraints.txt @@ -1,3 +1,3 @@ -tensorrt_llm==1.4.0a1 +tensorrt_llm==1.4.0a0 evaluate~=0.4.1 rouge_score~=0.1.2 diff --git a/tensorrt_llm/version.py b/tensorrt_llm/version.py index 37d3447dc1c4..25ad2d6ac807 100644 --- a/tensorrt_llm/version.py +++ b/tensorrt_llm/version.py @@ -12,4 +12,4 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -__version__ = "1.4.0a1" +__version__ = "1.4.0a0" From 98660c55de3aa9a13625084148825efc8a9e0fef Mon Sep 17 00:00:00 2001 From: tburt-nv <195370667+tburt-nv@users.noreply.github.com> Date: Fri, 16 Oct 2026 20:10:59 -0400 Subject: [PATCH 17/38] [None][fix] disable bolt (#20496) Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com> (cherry picked from commit c962eddaa4a079c315773451860ea6518548b5c2) Signed-off-by: Yanchao Lu --- jenkins/L0_MergeRequest.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jenkins/L0_MergeRequest.groovy b/jenkins/L0_MergeRequest.groovy index 4916f68029ac..0c725f5d2352 100644 --- a/jenkins/L0_MergeRequest.groovy +++ b/jenkins/L0_MergeRequest.groovy @@ -216,7 +216,7 @@ def BOLT_CONSUME = "bolt_consume" // `"bolt_consume": true` opts in a single run without one. Either way // resolveBoltConsume() still applies the post-merge and branch restrictions. @Field -def ENABLE_BOLT_PREMERGE_CONSUME = true +def ENABLE_BOLT_PREMERGE_CONSUME = false // Version-controlled rollout switch for post-merge BOLT, same idiom as above. // // Post-merge cannot use the pre-merge shape, where the BOLTed build REPLACES the From 40ca75b4b215a3c8d6093596f20741936aa0057b Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Fri, 23 Oct 2026 12:40:00 -0700 Subject: [PATCH 18/38] [None][perf] Add GB300 MiniMax-M3.1 low-latency decode targets Add SM103 TP8 single-request decode through eight rows and TP4 decode through 32 rows, with shared native kernels and guarded DSpark runtime integration. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- cpp/tensorrt_llm/kernels/CMakeLists.txt | 23 +- cpp/tensorrt_llm/kernels/minimaxM3/attnC1.cu | 3846 +++++++++++ cpp/tensorrt_llm/kernels/minimaxM3/attnC1.h | 110 + .../kernels/minimaxM3/attnMega.cu | 3673 +++++++++++ cpp/tensorrt_llm/kernels/minimaxM3/attnMega.h | 112 + .../kernels/minimaxM3/draftStep.cu | 576 ++ .../kernels/minimaxM3/draftStep.h | 99 + .../kernels/minimaxM3/fusedDecodeAttn.cu | 3679 +++++++++++ .../kernels/minimaxM3/fusedDecodeAttn.h | 105 + .../kernels/minimaxM3/mcAllreduceNorm.cu | 563 ++ .../kernels/minimaxM3/mcAllreduceNorm.h | 76 + .../kernels/minimaxM3/msaKvUpdate.cu | 70 + .../kernels/minimaxM3/msaKvUpdate.h | 45 + .../kernels/minimaxM3/mxfp8LowMGemm.cu | 820 +++ .../kernels/minimaxM3/mxfp8LowMGemm.h | 46 + .../kernels/minimaxM3/producerPw.cu | 324 + .../kernels/minimaxM3/producerPw.h | 69 + .../kernels/minimaxM3/routerGemv.cu | 606 ++ .../kernels/minimaxM3/routerGemv.h | 46 + .../kernels/minimaxM3/sampledAccept.cu | 525 ++ .../kernels/minimaxM3/sampledAccept.h | 79 + .../kernels/minimaxM3/selectLong.cu | 235 + .../kernels/minimaxM3/selectLong.h | 45 + .../kernels/minimaxM3/shardAccept.cu | 204 + .../kernels/minimaxM3/shardAccept.h | 67 + .../kernels/minimaxM3/sparseDecodeLc.cu | 561 ++ .../kernels/minimaxM3/sparseDecodeLc.h | 70 + .../kernels/minimaxM3/specTail.cu | 425 ++ cpp/tensorrt_llm/kernels/minimaxM3/specTail.h | 80 + .../kernels/minimaxM3/stepInputsScatter.cu | 69 + .../kernels/minimaxM3/stepInputsScatter.h | 40 + .../kernels/minimaxM3/targetHead.cu | 372 ++ .../kernels/minimaxM3/targetHead.h | 68 + .../minimaxM31/chainSpecSamplingMcSm103.cu | 418 ++ .../minimaxM31/chainSpecSamplingMcSm103.h | 84 + .../ctxResidualRmsNormMxfp8Sm103.cu | 232 + .../minimaxM31/ctxResidualRmsNormMxfp8Sm103.h | 43 + .../minimaxM31/selectPrefillTileSm103.cu | 545 ++ .../minimaxM31/selectPrefillTileSm103.h | 51 + .../kernels/minimaxM31/softmaxMcSm103.cu | 295 + .../kernels/minimaxM31/softmaxMcSm103.h | 71 + .../kernels/minimaxM31/sparseDecodeLc8Tp8.cu | 621 ++ .../kernels/minimaxM31/sparseDecodeLc8Tp8.h | 76 + .../kernels/minimaxM31/sparseDecodeLcTp4.cu | 635 ++ .../kernels/minimaxM31/sparseDecodeLcTp4.h | 77 + .../kernels/minimaxM31/toppRenormDetSm103.cu | 535 ++ .../kernels/minimaxM31/toppRenormDetSm103.h | 80 + .../minimaxM31/toppRenormDetV3Sm103.cu | 650 ++ .../kernels/minimaxM31/toppRenormDetV3Sm103.h | 58 + .../kernels/minimaxM3Tp8/fusedScoreSelect.cu | 770 +++ .../kernels/minimaxM3Tp8/fusedScoreSelect.h | 77 + .../kernels/minimaxM3Tp8/sparseDecodeLc8.cu | 610 ++ .../kernels/minimaxM3Tp8/sparseDecodeLc8.h | 75 + .../chainSpeculativeSamplingKernels.cu | 283 + .../chainSpeculativeSamplingKernels.h | 65 + cpp/tensorrt_llm/thop/CMakeLists.txt | 32 + .../thop/chainSpeculativeSamplingOp.cpp | 91 + .../thop/minimaxM3/minimaxM3AttnC1Op.cpp | 191 + .../thop/minimaxM3/minimaxM3AttnMegaOp.cpp | 207 + .../thop/minimaxM3/minimaxM3DraftStepOp.cpp | 198 + .../minimaxM3/minimaxM3FusedDecodeAttnOp.cpp | 175 + .../minimaxM3/minimaxM3McAllreduceNormOp.cpp | 159 + .../thop/minimaxM3/minimaxM3MsaKvUpdateOp.cpp | 110 + .../minimaxM3/minimaxM3Mxfp8LowMGemmOp.cpp | 83 + .../thop/minimaxM3/minimaxM3ProducerPwOp.cpp | 181 + .../thop/minimaxM3/minimaxM3RouterGemvOp.cpp | 149 + .../minimaxM3/minimaxM3SampledAcceptOp.cpp | 105 + .../thop/minimaxM3/minimaxM3SelectLongOp.cpp | 80 + .../thop/minimaxM3/minimaxM3ShardAcceptOp.cpp | 118 + .../minimaxM3/minimaxM3SparseDecodeLcOp.cpp | 139 + .../thop/minimaxM3/minimaxM3SpecTailOp.cpp | 202 + .../thop/minimaxM3/minimaxM3TargetHeadOp.cpp | 123 + .../minimaxM3Tp8FusedScoreSelectOp.cpp | 173 + .../minimaxM3Tp8SparseDecodeLc8Op.cpp | 139 + .../thop/minimaxM3/stepInputsScatterOp.cpp | 57 + .../minimaxM31ChainSpecSamplingMcOp.cpp | 140 + .../minimaxM31Sm103CtxNormMxfp8Op.cpp | 100 + .../minimaxM31Sm103SelectPrefillTileOp.cpp | 86 + .../thop/minimaxM31/minimaxM31SoftmaxMcOp.cpp | 95 + .../minimaxM31/minimaxM31ToppRenormDetOp.cpp | 98 + .../minimaxM31ToppRenormDetV3Op.cpp | 98 + .../minimaxM31Tp4SparseDecodeLcOp.cpp | 144 + .../minimaxM31Tp8SparseDecodeLc8Op.cpp | 143 + .../modeling_v2/_router_index.py | 70 +- .../catalog/attention/fused_qk_norm_rope.py | 360 ++ .../catalog/comm/allgather_cast.py | 540 ++ .../catalog/comm/allgather_cast_kernel.py | 327 + .../modeling_v2/catalog/fakes/__init__.py | 3 + .../modeling_v2/catalog/fakes/m31_fake_ops.py | 161 + .../modeling_v2/catalog/moe/__init__.py | 3 + .../catalog/moe/fp4_block_scale_moe_runner.py | 450 ++ .../catalog/quantization/__init__.py | 3 + .../catalog/quantization/fp4_quantize.py | 229 + .../catalog/spec_decode/__init__.py | 3 + .../spec_decode/dspark_block_tables.py | 173 + .../catalog/spec_decode/dspark_ctx_advance.py | 137 + .../catalog/spec_decode/dspark_ctx_store.py | 309 + .../spec_decode/dspark_drafter_inputs.py | 143 + .../spec_decode/dspark_forward_indices.py | 134 + .../catalog/spec_decode/dspark_markov_bias.py | 128 + .../spec_decode/dspark_markov_chain.py | 651 ++ .../spec_decode/dspark_markov_chain_kernel.py | 583 ++ .../dspark_markov_chain_kernel_b1.py | 616 ++ .../dspark_markov_chain_kernel_b1_ll.py | 624 ++ .../spec_decode/dspark_qk_norm_rope.py | 309 + .../catalog/spec_decode/strict_accept.py | 482 ++ .../spec_decode/strict_accept_kernel.py | 174 + .../modeling_v2/models/minimax_m3/__init__.py | 3 + .../modeling_v2/models/minimax_m3/fake_ops.py | 124 + .../__init__.py | 8 + .../minimax_m31_nvfp4__sm_103__tp4/dspark.py | 631 ++ .../dspark_s4.py | 491 ++ .../m31_switches.py | 61 + .../modeling.py | 5647 +++++++++++++++++ .../minimax_m31_nvfp4__sm_103__tp4/mxdraft.py | 112 + .../ops/__init__.py | 3 + .../ops/activation/__init__.py | 3 + .../ops/activation/m3_silu_and_mul.py | 83 + .../ops/activation/m3_swq.py | 159 + .../ops/attention/__init__.py | 3 + .../ops/attention/m3_attn_c1.py | 280 + .../ops/attention/m3_attn_mega.py | 340 + .../ops/attention/m3_dense_decode.py | 187 + .../ops/attention/m3_draft_attention.py | 111 + .../attention/m3_fp8_indexer_qk_norm_rope.py | 114 + .../ops/attention/m3_fused_decode_attn.py | 643 ++ .../ops/attention/m3_index_score.py | 167 + .../ops/attention/m3_msa_attention.py | 224 + .../ops/attention/m3_producer.py | 202 + .../ops/attention/m3_qk_norm_rope_fp8.py | 137 + .../ops/attention/m3_qkv_insert.py | 134 + .../ops/attention/m3_select_blocks.py | 82 + .../ops/attention/m3_select_long.py | 172 + .../ops/attention/m3_sparse_decode.py | 215 + .../ops/attention/m3_sparse_decode_lc.py | 487 ++ .../ops/attention/msa_kv_update.py | 200 + .../ops/comm/__init__.py | 3 + .../ops/comm/m3_allreduce.py | 61 + .../ops/comm/m3_mc_allreduce_norm.py | 277 + .../ops/comm/m3_mc_finalize_allreduce_norm.py | 197 + .../ops/embedding/__init__.py | 3 + .../ops/embedding/m3_embedding.py | 48 + .../ops/embedding_firstnorm/__init__.py | 3 + .../m3_embedding_firstnorm_kernel.py | 128 + .../ops/embedding_firstnorm/op.py | 142 + .../ops/gemm/__init__.py | 3 + .../ops/gemm/m3_bf16_linear.py | 41 + .../ops/gemm/m3_fp32_linear.py | 46 + .../ops/gemm/m3_mx_gemv.py | 166 + .../ops/gemm/m3_mx_gemv_bf16.py | 106 + .../ops/gemm/m3_mxfp8_gemm.py | 76 + .../ops/gemm/m3_mxfp8_linear.py | 137 + .../ops/gemm/m3_mxfp8_pinned_quantize.py | 47 + .../ops/gemm/m3_target_head.py | 346 + .../ops/gemm/mxfp8_low_m_gemm.py | 176 + .../ops/gqamix_sparse_decode.py | 1311 ++++ .../ops/long_select.py | 62 + .../ops/m3_attn_lc.py | 113 + .../ops/m3_attn_mega.py | 206 + .../ops/m3_attn_mega_c1.py | 365 ++ .../ops/m3_bp_tail.py | 192 + .../ops/m3_draft_step.py | 178 + .../ops/m3_force_vec.py | 35 + .../ops/m3_fused_attn.py | 352 + .../ops/m3_index_score.py | 645 ++ .../ops/m3_mc_allreduce.py | 376 ++ .../ops/m3_moe/__init__.py | 16 + .../ops/m3_moe/m3_moe_kernel.py | 3719 +++++++++++ .../ops/m3_moe/op.py | 622 ++ .../ops/m3_moe/op16.py | 288 + .../ops/m3_prequant.py | 111 + .../ops/m3_router_gemv.py | 107 + .../ops/m3_sampled_accept.py | 159 + .../ops/m3_shard_accept.py | 176 + .../ops/m3_spec_tail.py | 116 + .../ops/m3_target_head.py | 113 + .../ops/moe/__init__.py | 3 + .../ops/moe/m3_moe16_engine.py | 67 + .../ops/moe/m3_moe_engine.py | 326 + .../ops/moe/m3_router_gemv.py | 101 + .../ops/moe/m3_router_gemv_fp4.py | 63 + .../ops/moe/m3_router_tc.py | 106 + .../ops/msa_kv_update.py | 89 + .../ops/mx_gemv/__init__.py | 3 + .../ops/mx_gemv/m3_mx_gemv_kernel.py | 1238 ++++ .../ops/mx_gemv/op.py | 474 ++ .../ops/mxfp8_low_m_gemm.py | 109 + .../ops/norm/__init__.py | 3 + .../ops/norm/m3_embedding_firstnorm.py | 87 + .../ops/sampling/__init__.py | 4 + .../ops/sampling/m31_chain_spec_mc.py | 266 + .../ops/sampling/m31_softmax_mc.py | 199 + .../ops/sampling/m31_topp_det.py | 248 + .../ops/spec/__init__.py | 4 + .../ops/spec/m3_capture_norm.py | 80 + .../ops/spec/m3_chunk_norm.py | 70 + .../ops/spec/m3_draft_input.py | 140 + .../ops/spec/m3_draft_step.py | 367 ++ .../ops/spec/m3_force_vec.py | 86 + .../ops/spec/m3_sampled_accept.py | 269 + .../ops/spec/m3_sampled_accept_ws_words.py | 43 + .../ops/spec/m3_shard_accept.py | 284 + .../ops/spec/m3_slot_scatter.py | 72 + .../ops/spec/m3_step_gather.py | 152 + .../ops/triton_sparse_decode.py | 1651 +++++ .../ops/triton_sparse_decode_mq.py | 237 + .../minimax_m31_nvfp4__sm_103__tp4/weights.py | 910 +++ .../__init__.py | 3 + .../minimax_m31_nvfp4__sm_103__tp8/dspark.py | 829 +++ .../dspark_s4.py | 477 ++ .../m31_switches.py | 61 + .../modeling.py | 5107 +++++++++++++++ .../minimax_m31_nvfp4__sm_103__tp8/mxdraft.py | 112 + .../ops/__init__.py | 3 + .../ops/activation/__init__.py | 3 + .../ops/activation/m3_silu_and_mul.py | 83 + .../ops/activation/m3_swq.py | 162 + .../ops/attention/__init__.py | 3 + .../ops/attention/m3_dense_decode.py | 187 + .../ops/attention/m3_draft_attention.py | 111 + .../attention/m3_fp8_indexer_qk_norm_rope.py | 114 + .../ops/attention/m3_fused_score_select.py | 189 + .../ops/attention/m3_index_score.py | 165 + .../ops/attention/m3_msa_attention.py | 224 + .../ops/attention/m3_producer.py | 202 + .../ops/attention/m3_qk_norm_rope_fp8.py | 137 + .../ops/attention/m3_qkv_insert.py | 134 + .../ops/attention/m3_select_blocks.py | 82 + .../ops/attention/m3_select_long.py | 172 + .../ops/attention/m3_select_long_dsl.py | 126 + .../ops/attention/m3_sparse_decode.py | 215 + .../ops/attention/m3_sparse_decode_lc8.py | 499 ++ .../ops/attention/msa_kv_update.py | 200 + .../ops/comm/__init__.py | 3 + .../ops/comm/m3_allreduce.py | 60 + .../ops/comm/m3_mc_allreduce_norm.py | 277 + .../ops/comm/m3_mc_finalize_allreduce_norm.py | 197 + .../ops/embedding/__init__.py | 3 + .../ops/embedding/m3_embedding.py | 48 + .../ops/embedding_firstnorm/__init__.py | 3 + .../m3_embedding_firstnorm_kernel.py | 128 + .../ops/embedding_firstnorm/op.py | 142 + .../ops/gemm/__init__.py | 3 + .../ops/gemm/m3_bf16_linear.py | 41 + .../ops/gemm/m3_draft_lowm.py | 150 + .../ops/gemm/m3_fp32_linear.py | 46 + .../ops/gemm/m3_mx_gemv.py | 160 + .../ops/gemm/m3_mx_gemv_bf16.py | 106 + .../ops/gemm/m3_mxfp8_gemm.py | 58 + .../ops/gemm/m3_mxfp8_linear.py | 137 + .../ops/gqamix_sparse_decode.py | 1311 ++++ .../ops/long_select.py | 77 + .../ops/m3_attn_lc8.py | 116 + .../ops/m3_bp_tail.py | 192 + .../ops/m3_draft_step.py | 149 + .../ops/m3_force_vec.py | 35 + .../ops/m3_fused_score_select.py | 110 + .../ops/m3_index_score.py | 700 ++ .../ops/m3_layer/__init__.py | 14 + .../ops/m3_layer/phase_producer.py | 489 ++ .../ops/m3_mc_allreduce.py | 376 ++ .../ops/m3_moe/__init__.py | 16 + .../ops/m3_moe/m3_moe_kernel.py | 3673 +++++++++++ .../ops/m3_moe/op.py | 670 ++ .../ops/m3_prequant.py | 111 + .../ops/m3_router_gemv.py | 107 + .../ops/m3_sampled_accept.py | 159 + .../ops/m3_select_long_dsl.py | 367 ++ .../ops/m3_shard_accept.py | 176 + .../ops/m3_spec_tail.py | 116 + .../ops/moe/__init__.py | 3 + .../ops/moe/m3_moe_engine.py | 325 + .../ops/moe/m3_router_gemv.py | 101 + .../ops/moe/m3_router_gemv_fp4.py | 63 + .../ops/moe/m3_router_tc.py | 106 + .../ops/msa_kv_update.py | 89 + .../ops/mx_gemv/__init__.py | 4 + .../ops/mx_gemv/m3_mx_gemv_kernel.py | 1472 +++++ .../ops/mx_gemv/op.py | 649 ++ .../ops/norm/__init__.py | 3 + .../ops/norm/m3_embedding_firstnorm.py | 87 + .../ops/sampling/__init__.py | 4 + .../ops/sampling/m31_chain_spec_mc.py | 271 + .../ops/sampling/m31_softmax_mc.py | 199 + .../ops/sampling/m31_topp_det.py | 248 + .../ops/spec/__init__.py | 4 + .../ops/spec/m3_capture_norm.py | 80 + .../ops/spec/m3_chunk_norm.py | 70 + .../ops/spec/m3_draft_input.py | 140 + .../ops/spec/m3_draft_step.py | 359 ++ .../ops/spec/m3_force_vec.py | 86 + .../ops/spec/m3_sampled_accept.py | 269 + .../ops/spec/m3_sampled_accept_ws_words.py | 43 + .../ops/spec/m3_shard_accept.py | 284 + .../ops/spec/m3_slot_scatter.py | 72 + .../ops/spec/m3_step_gather.py | 152 + .../ops/triton_sparse_decode.py | 1662 +++++ .../ops/triton_sparse_decode_mq.py | 237 + .../minimax_m31_nvfp4__sm_103__tp8/weights.py | 928 +++ .../modeling_v2/models/minimax_m3/routing.py | 206 + .../sparse/minimax_m3/cache_manager.py | 78 + .../backends/sparse/minimax_m3/msa_backend.py | 212 +- .../_torch/attention/backends/trtllm.py | 13 +- .../_torch/custom_ops/torch_custom_ops.py | 59 + .../checkpoints/hf/minimaxm3_weight_mapper.py | 5 + tensorrt_llm/_torch/models/modeling_dflash.py | 202 +- tensorrt_llm/_torch/modules/linear.py | 10 + tensorrt_llm/_torch/pyexecutor/_util.py | 15 +- .../_torch/pyexecutor/cuda_graph_runner.py | 83 +- .../engine/runners/decoder/runner.py | 352 +- tensorrt_llm/_torch/pyexecutor/h2d_cache.py | 204 + .../kv_cache/kv_cache_manager_v2.py | 68 + .../_torch/pyexecutor/model_engine.py | 5 + tensorrt_llm/_torch/pyexecutor/py_executor.py | 56 +- .../_torch/pyexecutor/py_executor_creator.py | 1 + .../_torch/pyexecutor/runtime_hooks.py | 255 + .../pyexecutor/sampler/ops/flashinfer.py | 58 +- .../_torch/pyexecutor/sampler/sampler.py | 64 +- tensorrt_llm/_torch/pyexecutor/step_inputs.py | 471 ++ tensorrt_llm/_torch/speculative/dflash.py | 511 +- tensorrt_llm/_torch/speculative/eagle3.py | 7 +- tensorrt_llm/_torch/speculative/interface.py | 298 +- .../_torch/speculative/spec_sampler_base.py | 191 +- .../_torch/executor/test_h2d_cache.py | 580 ++ .../test_instantiate_sampler_hooks.py | 112 + .../test_kv_cache_v2_block_offset_guard.py | 201 + .../_torch/executor/test_runtime_hooks.py | 159 + .../_torch/executor/test_step_inputs.py | 1202 ++++ .../comm/_allgather_cast_op_matrix.py | 266 + .../_torch/modeling_v2/comm/_rank_job.py | 5 +- ...st_modeling_v2_allgather_cast_op_matrix.py | 32 + .../test_modeling_v2_m31_tp4_swq.py | 99 + .../attention/test_modeling_v2_m31_gqamix.py | 189 + .../attention/test_modeling_v2_m31_lc4.py | 170 + .../attention/test_modeling_v2_m31_tsel.py | 141 + ...deling_v2_m31_tp4_mxfp8_pinned_quantize.py | 33 + .../test_modeling_v2_m31_tp4_shdown_pin.py | 238 + .../test_modeling_v2_m31_unfused_index.py | 112 + .../test_modeling_v2_m31_tp4_moe_early0n.py | 29 + .../moe/test_modeling_v2_m31_tp4_moe_m4cfg.py | 29 + .../norm/test_modeling_v2_m31_ctxnormq.py | 104 + .../test_modeling_v2_m31_chain_spec_mc.py | 286 + .../test_modeling_v2_m31_softmax_mc.py | 256 + .../sampling/test_modeling_v2_m31_topp_det.py | 271 + .../spec/test_modeling_v2_m31_dspark_n3a.py | 466 ++ .../spec/test_modeling_v2_m31_dspark_s4.py | 347 + .../spec/test_modeling_v2_m31_dspark_w0.py | 171 + .../test_modeling_v2_m3_exact_numerics.py | 289 + .../test_modeling_v2_m31_ar_det.py | 43 + .../test_modeling_v2_m31_fmha_sm100_fixes.py | 44 + .../test_modeling_v2_m31_hosttail.py | 173 + .../test_modeling_v2_m31_switches.py | 156 + .../test_modeling_v2_m31_tp4_ar64auto.py | 79 + ...ing_v2_m31_tp4_fused_projection_default.py | 37 + ...test_modeling_v2_m31_tp4_weights_layout.py | 68 + .../_dspark_markov_chain_ll_op_matrix.py | 671 ++ .../_dspark_markov_chain_op_matrix.py | 287 + .../modeling_v2/spec_decode/_rank_job.py | 17 + .../test_modeling_v2_dspark_block_tables.py | 104 + .../test_modeling_v2_dspark_ctx_advance.py | 141 + .../test_modeling_v2_dspark_ctx_store.py | 266 + .../test_modeling_v2_dspark_drafter_inputs.py | 105 + ...test_modeling_v2_dspark_forward_indices.py | 97 + .../test_modeling_v2_dspark_markov_bias.py | 140 + ...ing_v2_dspark_markov_chain_ll_op_matrix.py | 32 + ...deling_v2_dspark_markov_chain_op_matrix.py | 32 + .../test_modeling_v2_dspark_qk_norm_rope.py | 244 + .../test_modeling_v2_spec_strict_accept.py | 158 + .../test_m31_decode_scope_source.py | 96 + .../test_m31_tp4_routing_source.py | 325 + .../test_m31_tp8_routing_source.py | 253 + .../speculative/test_rng_upload_pack.py | 150 + .../test_spec_sampler_step_fold.py | 648 ++ .../_torch/speculative/test_tail_bookkeep.py | 590 ++ 374 files changed, 106304 insertions(+), 348 deletions(-) create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/attnC1.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/attnC1.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/attnMega.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/attnMega.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/draftStep.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/draftStep.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/msaKvUpdate.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/msaKvUpdate.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/producerPw.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/producerPw.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/selectLong.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/selectLong.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/shardAccept.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/shardAccept.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/specTail.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/targetHead.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3/targetHead.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.h create mode 100644 cpp/tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.cu create mode 100644 cpp/tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.h create mode 100644 cpp/tensorrt_llm/thop/chainSpeculativeSamplingOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3AttnC1Op.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3AttnMegaOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3DraftStepOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3FusedDecodeAttnOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3McAllreduceNormOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3MsaKvUpdateOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Mxfp8LowMGemmOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3ProducerPwOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3RouterGemvOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SampledAcceptOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SelectLongOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3ShardAcceptOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SparseDecodeLcOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SpecTailOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3TargetHeadOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Tp8FusedScoreSelectOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Tp8SparseDecodeLc8Op.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/stepInputsScatterOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Sm103CtxNormMxfp8Op.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Sm103SelectPrefillTileOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SoftmaxMcOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetV3Op.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Tp4SparseDecodeLcOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Tp8SparseDecodeLc8Op.cpp create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/attention/fused_qk_norm_rope.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/m31_fake_ops.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/moe/fp4_block_scale_moe_runner.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/quantization/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/quantization/fp4_quantize.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_block_tables.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_advance.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_store.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_drafter_inputs.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_forward_indices.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_bias.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1_ll.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_qk_norm_rope.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/fake_ops.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/dspark.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/dspark_s4.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/m31_switches.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/modeling.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/mxdraft.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/m3_silu_and_mul.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/m3_swq.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_attn_c1.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_attn_mega.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_dense_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_draft_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_fp8_indexer_qk_norm_rope.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_fused_decode_attn.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_msa_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_producer.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_qk_norm_rope_fp8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_qkv_insert.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_select_blocks.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_select_long.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_sparse_decode_lc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_mc_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_mc_finalize_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding/m3_embedding.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/m3_embedding_firstnorm_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_bf16_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_fp32_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mx_gemv.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mx_gemv_bf16.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_pinned_quantize.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_target_head.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/mxfp8_low_m_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gqamix_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/long_select.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_lc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_mega.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_mega_c1.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_bp_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_fused_attn.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_mc_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/m3_moe_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/op16.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_prequant.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_router_gemv.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_spec_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_target_head.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_moe16_engine.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_moe_engine.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_gemv.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_gemv_fp4.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_tc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/m3_mx_gemv_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mxfp8_low_m_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/norm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/norm/m3_embedding_firstnorm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_chain_spec_mc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_softmax_mc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_topp_det.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_capture_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_chunk_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_draft_input.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_sampled_accept_ws_words.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_slot_scatter.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_step_gather.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/triton_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/triton_sparse_decode_mq.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/weights.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/dspark.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/dspark_s4.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/m31_switches.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/modeling.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/mxdraft.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/m3_silu_and_mul.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/m3_swq.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_dense_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_draft_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_fp8_indexer_qk_norm_rope.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_fused_score_select.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_msa_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_producer.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_qk_norm_rope_fp8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_qkv_insert.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_blocks.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_long.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_long_dsl.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_sparse_decode_lc8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_mc_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_mc_finalize_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding/m3_embedding.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/m3_embedding_firstnorm_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_bf16_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_draft_lowm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_fp32_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mx_gemv.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mx_gemv_bf16.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mxfp8_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mxfp8_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gqamix_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/long_select.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_attn_lc8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_bp_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_fused_score_select.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_layer/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_layer/phase_producer.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_mc_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/m3_moe_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_prequant.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_router_gemv.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_select_long_dsl.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_spec_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_moe_engine.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_gemv.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_gemv_fp4.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_tc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/m3_mx_gemv_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/norm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/norm/m3_embedding_firstnorm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_chain_spec_mc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_softmax_mc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_topp_det.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_capture_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_chunk_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_draft_input.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_sampled_accept_ws_words.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_slot_scatter.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_step_gather.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/triton_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/triton_sparse_decode_mq.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/weights.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py create mode 100644 tensorrt_llm/_torch/pyexecutor/h2d_cache.py create mode 100644 tensorrt_llm/_torch/pyexecutor/runtime_hooks.py create mode 100644 tensorrt_llm/_torch/pyexecutor/step_inputs.py create mode 100644 tests/unittest/_torch/executor/test_h2d_cache.py create mode 100644 tests/unittest/_torch/executor/test_instantiate_sampler_hooks.py create mode 100644 tests/unittest/_torch/executor/test_kv_cache_v2_block_offset_guard.py create mode 100644 tests/unittest/_torch/executor/test_runtime_hooks.py create mode 100644 tests/unittest/_torch/executor/test_step_inputs.py create mode 100644 tests/unittest/_torch/modeling_v2/comm/_allgather_cast_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_allgather_cast_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/activation/test_modeling_v2_m31_tp4_swq.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_gqamix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_lc4.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_tsel.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_tp4_mxfp8_pinned_quantize.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_tp4_shdown_pin.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_unfused_index.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/moe/test_modeling_v2_m31_tp4_moe_early0n.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/moe/test_modeling_v2_m31_tp4_moe_m4cfg.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/norm/test_modeling_v2_m31_ctxnormq.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_chain_spec_mc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_softmax_mc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_topp_det.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_n3a.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_s4.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_w0.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m3_exact_numerics.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_ar_det.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_fmha_sm100_fixes.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_hosttail.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_switches.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_ar64auto.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_fused_projection_default.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_weights_layout.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/_dspark_markov_chain_ll_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/_dspark_markov_chain_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/_rank_job.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_block_tables.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_ctx_advance.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_ctx_store.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_drafter_inputs.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_forward_indices.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_bias.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_chain_ll_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_chain_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_qk_norm_rope.py create mode 100644 tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_spec_strict_accept.py create mode 100644 tests/unittest/_torch/modeling_v2/test_m31_decode_scope_source.py create mode 100644 tests/unittest/_torch/modeling_v2/test_m31_tp4_routing_source.py create mode 100644 tests/unittest/_torch/modeling_v2/test_m31_tp8_routing_source.py create mode 100644 tests/unittest/_torch/speculative/test_rng_upload_pack.py create mode 100644 tests/unittest/_torch/speculative/test_spec_sampler_step_fold.py create mode 100644 tests/unittest/_torch/speculative/test_tail_bookkeep.py diff --git a/cpp/tensorrt_llm/kernels/CMakeLists.txt b/cpp/tensorrt_llm/kernels/CMakeLists.txt index 0e7fc9390dc0..54a477030e66 100644 --- a/cpp/tensorrt_llm/kernels/CMakeLists.txt +++ b/cpp/tensorrt_llm/kernels/CMakeLists.txt @@ -78,6 +78,10 @@ list(FILTER SRC_CU EXCLUDE REGEX "fusedSampling/.*") list(FILTER SRC_CPP EXCLUDE REGEX "marlin/.*") list(FILTER SRC_CU EXCLUDE REGEX "marlin/.*") +# Scope tcgen05 attention sources to their supported architectures. +list(FILTER SRC_CU EXCLUDE REGEX + "minimaxM3/(fusedDecodeAttn|attnMega|attnC1)\\.cu$") + if(NOT ENABLE_MULTI_DEVICE) list(FILTER SRC_CU EXCLUDE REGEX "customAllReduceKernels*.*cu$") endif() @@ -111,10 +115,27 @@ if(MARLIN_SRC) set_cuda_architectures(marlin_src 89 90 120f) endif() +# Preserve the FlashInfer-matched compilation for sampling only. Do not enable +# fast math globally: attention exactness depends on its flags. +set_source_files_properties( + ${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/toppRenormDetSm103.cu + ${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/toppRenormDetV3Sm103.cu + ${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/softmaxMcSm103.cu + PROPERTIES COMPILE_OPTIONS "$<$:--use_fast_math>") + +# Preserve the historical architecture-scoped attention object library. +add_library( + minimax_m3_attn_src OBJECT minimaxM3/fusedDecodeAttn.cu minimaxM3/attnMega.cu + minimaxM3/attnC1.cu) +set_property(TARGET minimax_m3_attn_src PROPERTY POSITION_INDEPENDENT_CODE ON) +set_property(TARGET minimax_m3_attn_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON) +set_cuda_architectures(minimax_m3_attn_src 100 103) + add_library( kernels_src STATIC ${SRC_CPP} ${SRC_CU} ${MOE_KERNELS_SRC} - $<$:$>) + $<$:$> + $) set_property(TARGET kernels_src PROPERTY POSITION_INDEPENDENT_CODE ON) set_property(TARGET kernels_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON) target_include_directories( diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/attnC1.cu b/cpp/tensorrt_llm/kernels/minimaxM3/attnC1.cu new file mode 100644 index 000000000000..e375b6c994fd --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/attnC1.cu @@ -0,0 +1,3846 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3 attention + o_proj kernel (c1): fused decode attention and o_proj GEMV in one 96-CTA launch. + +#include "tensorrt_llm/kernels/minimaxM3/attnC1.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// The kernel uses tcgen05/TMEM (including block-scaled MMAs, which family targets lack), cluster DSMEM and E2M1 +// conversions: its body is compiled for the arch-specific sm_100a / sm_103a targets only (see the minimax_m3_attn_src +// library in kernels/CMakeLists.txt). Any other target compiles a kernel that traps, so a launch on a build without +// those targets fails loudly instead of returning untouched outputs. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ == 1000 || __CUDA_ARCH__ == 1030) \ + && (defined(__CUDA_ARCH_SPECIFIC__) || defined(__CUDA_ARCH_FEAT_SM100_ALL) || defined(__CUDA_ARCH_FEAT_SM103_ALL)) +#define MINIMAX_M3_ATTN_DEVICE_ENABLED 1 +#else +#define MINIMAX_M3_ATTN_DEVICE_ENABLED 0 +#endif + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kHeadDim = 128; +constexpr int kRotDim = 64; +constexpr int kPage = 128; +constexpr int kHq = 16; +constexpr int kColK = kHq; // packed head slots: [Q x16 | K | V | idxQ | idxK] +constexpr int kColV = kHq + 1; +constexpr int kColIdxK = kHq + 3; +constexpr int kBlocks = 16; // KV blocks per request == topk == cluster size +constexpr int kComputeWarps = 16; // QK / softmax / PV and the dequant (one 16-byte chunk per thread) +constexpr int kWarps = kComputeWarps + 4; // plus producer-only warps, so new K/V/index-K never queue behind a Q head +constexpr int kThreads = kWarps * 32; +constexpr int kComputeThreads = kComputeWarps * 32; +constexpr int kOPitch = 132; // floats per partial-O row +constexpr float kNegInf = -__builtin_huge_valf(); + +using Params = MinimaxM3AttnC1Params; + +// Optional per-CTA clock64 probes for development (-DM3FA_PROFILE and a prof tensor): [CTA][64] timestamps. +// -DM3FA_PROFILE_GT records %globaltimer (ns, comparable across SMs and kernels) instead of clock64. +#ifdef M3FA_PROFILE_GT +#define M3FA_PROFILE +#define M3FA_NOW(v) asm volatile("mov.u64 %0, %%globaltimer;" : "=l"(v)) +#else +#define M3FA_NOW(v) v = clock64() +#endif +#ifdef M3FA_PROFILE +#define M3FA_PROF_T(i, thread) \ + do \ + { \ + if (p.prof != nullptr && threadIdx.x == (thread)) \ + { \ + unsigned long long now_; \ + M3FA_NOW(now_); \ + p.prof[blockIdx.x * 64 + (i)] = now_; \ + } \ + } while (0) +#else +#define M3FA_PROF_T(i, thread) \ + do \ + { \ + } while (0) +#endif +#define M3FA_PROF(i) M3FA_PROF_T(i, 0) +// clock64 probes (development): slot i of this CTA, written by every calling thread. +#ifdef M3FA_PROFILE +#define M3FA_CLK(i) \ + do \ + { \ + if (p.prof != nullptr) \ + { \ + p.prof[blockIdx.x * 64 + (i)] = clock64(); \ + } \ + } while (0) +#else +#define M3FA_CLK(i) \ + do \ + { \ + } while (0) +#endif + +#ifndef M3FA_MEGA_EARLYCLW +#define M3FA_MEGA_EARLYCLW 1 // pass the cluster barrier (peers' merge mbarriers initialized) early, not before the push +#endif +#ifndef M3FA_V15_EARLYV +#define M3FA_V15_EARLYV 0 // packed path: dequantize V before the PDL wait (switch; off by default) +#endif +#ifndef M3FA_MEGA_TRIGGER +// Megakernel PDL release: 0 o_proj CTAs after their activation (attention CTAs at exit); 1 as 0 with the attention +// CTAs after their KV reads; 2 every CTA right after its griddepcontrol.wait; 3 o_proj CTAs after their epilogue. +#define M3FA_MEGA_TRIGGER 3 +#endif +#ifndef M3FA_TRIGGER_AT +#define M3FA_TRIGGER_AT 2 // where the PDL dependents are released: 0 entry, 1 after the wait, 2 after the KV reads +#endif + +// Generic-proxy shared-memory writes (fp16 tiles, Q, P) become visible to the tensor core (async proxy). +#define M3FA_FENCE_PROXY_ASYNC() asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory") + +__device__ __forceinline__ void cp_async16(void* dst, void const* src) +{ + uint32_t const d = static_cast(__cvta_generic_to_shared(dst)); + asm volatile("cp.async.cg.shared.global [%0], [%1], 16;\n" ::"r"(d), "l"(src) : "memory"); +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +// One packed E2M1 byte -> half2 (low nibble in the low half). +__device__ __forceinline__ uint32_t e2m1x2_to_h2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 src;\n mov.b32 {src, _, _, _}, %1;\n cvt.rn.f16x2.e2m1x2 %0, src;\n}\n" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t hmul2_u(uint32_t a, uint32_t b) +{ + __half2 r = __hmul2(*reinterpret_cast<__half2*>(&a), *reinterpret_cast<__half2*>(&b)); + return *reinterpret_cast(&r); +} + +__device__ __forceinline__ uint32_t e4m3_to_h2_splat(uint32_t byte) +{ + __half_raw const h = __nv_cvt_fp8_to_halfraw(static_cast<__nv_fp8_storage_t>(byte), __NV_E4M3); + return static_cast(h.x) | (static_cast(h.x) << 16); +} + +__device__ __forceinline__ uint32_t e4m3x2_to_h2(uint32_t two_bytes) +{ + __half2_raw const h = __nv_cvt_fp8x2_to_halfraw2(static_cast<__nv_fp8x2_storage_t>(two_bytes), __NV_E4M3); + return static_cast(h.x) | (static_cast(h.y) << 16); +} + +// ---- Exact copies of the C++ producer's helpers (fusedQKNormRopeKernel.cu / quantization.cuh) ---- +template +__device__ __forceinline__ uint32_t cvt_sf_group_shfl_mask() +{ + constexpr uint32_t groupSize = static_cast(NUM_THREADS_PER_SF); + constexpr uint32_t groupMask = 0xFFFFFFFFU >> (32U - groupSize); + uint32_t laneId = 0; + asm("mov.u32 %0, %%laneid;" : "=r"(laneId)); + return groupMask << (laneId & ~(groupSize - 1U)); +} + +__device__ __forceinline__ uint32_t fp32_vec_to_e2m1(float (&array)[8]) +{ + uint32_t val; + asm volatile( + "{\n" + ".reg .b8 byte0;\n" + ".reg .b8 byte1;\n" + ".reg .b8 byte2;\n" + ".reg .b8 byte3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte0, %2, %1;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte1, %4, %3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte2, %6, %5;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte3, %8, %7;\n" + "mov.b32 %0, {byte0, byte1, byte2, byte3};\n" + "}" + : "=r"(val) + : "f"(array[0]), "f"(array[1]), "f"(array[2]), "f"(array[3]), "f"(array[4]), "f"(array[5]), "f"(array[6]), + "f"(array[7])); + return val; +} + +__device__ __forceinline__ uint16_t quantizeMinimaxM3Fp4x4( + float const (&elements)[4], float secondLevelScale, uint8_t* scaleOut) +{ + __nv_bfloat16 rounded[4]; + float localMax = 0.0F; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + rounded[i] = __float2bfloat16_rn(elements[i]); + localMax = fmaxf(localMax, fabsf(__bfloat162float(rounded[i]))); + } + uint32_t const groupMask = cvt_sf_group_shfl_mask<4>(); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 1)); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 2)); + float const scaleValue = secondLevelScale * (localMax * rcp_approx_ftz(6.0F)); + __nv_fp8_e4m3 const encodedScale(scaleValue); + if (scaleOut != nullptr) + { + *scaleOut = encodedScale.__x; + } + float const narrowedScale = static_cast(encodedScale); + float const outputScale + = localMax != 0.0F ? rcp_approx_ftz(narrowedScale * rcp_approx_ftz(secondLevelScale)) : 0.0F; + float quantValues[8] = {}; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + quantValues[i] = __bfloat162float(rounded[i]) * outputScale; + } + return static_cast(fp32_vec_to_e2m1(quantValues)); +} + +// RMSNorm (gemma 1+w) + NeoX RoPE over the first 64 dims, exactly as the producer: lane owns 4 dims. +__device__ __forceinline__ void norm_rope( + float (&e)[4], float const (&w)[4], float eps, float const (&cs)[4], float const (&sn)[4], int lane) +{ + float ss = 0.0F; + ss += e[0] * e[0] + e[1] * e[1]; + ss += e[2] * e[2] + e[3] * e[3]; +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + { + ss = ss + __shfl_xor_sync(0xffffffffu, ss, mask, 32); + } + float const rmsReciprocal = rsqrtf(ss / static_cast(kHeadDim) + eps); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + e[i] *= rmsReciprocal * (1.0F + w[i]); + } + __syncwarp(); + constexpr int kPairOffset = (kRotDim / 2) / 4; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const dim = lane * 4 + i; + float paired = __shfl_xor_sync(0xffffffffu, e[i], kPairOffset); + if (dim < kRotDim) + { + bool const firstHalf = dim < kRotDim / 2; + if (firstHalf) + { + paired = -paired; + } + e[i] = e[i] * cs[i] + paired * sn[i]; + } + } + __syncwarp(); +} + +__device__ __forceinline__ void load_w4(float (&w)[4], __nv_bfloat16 const* src, int lane) +{ + uint2 const raw = *reinterpret_cast(src + lane * 4); + float2 const a = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw.x)); + float2 const b = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw.y)); + w[0] = a.x; + w[1] = a.y; + w[2] = b.x; + w[3] = b.y; +} + +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t mapa_shared(uint32_t addr, uint32_t rank) +{ + uint32_t r; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;\n" : "=r"(r) : "r"(addr), "r"(rank)); + return r; +} + +// QLEN query tokens per request; QG of them per cluster (QLEN / QG clusters per request); BPC KV blocks per CTA +// (cluster of kBlocks / BPC CTAs). +// Byte offset of element (r, k) in a UMMA no-swizzle K-major operand tile of 128 k-elements per row: 8x8 core +// matrices of 16-byte rows, 128 bytes apart along k (LBO) and 2048 bytes apart per 8 rows (SBO). The same formula +// with r = token, k = dim stores V so that it is the MN-major (dims contiguous) A operand of O^T = V^T P^T. +__device__ __forceinline__ int kmaj_off(int r, int k) +{ + return (r >> 3) * 2048 + (k >> 3) * 128 + (r & 7) * 16 + (k & 7) * 2; +} + +__device__ __forceinline__ uint64_t umma_desc(uint32_t addr, uint32_t lbo, uint32_t sbo) +{ + return static_cast((addr >> 4) & 0x3FFFu) | (static_cast((lbo >> 4) & 0x3FFFu) << 16) + | (static_cast((sbo >> 4) & 0x3FFFu) << 32) | (1ull << 46); +} + +// kind::f16 instruction descriptor: F32 accumulate, F16 operands, B K-major, M = 128. +__host__ __device__ constexpr uint32_t umma_idesc(int n, int a_mn_major, int b_mn_major) +{ + return (1u << 4) | (static_cast(a_mn_major) << 15) | (static_cast(b_mn_major) << 16) + | (static_cast(n >> 3) << 17) | (static_cast(128 >> 4) << 24); +} + +// 8-row version for the split softmax: the same lane-bit order (16, 8, 4, then 2 and 1 in full) as +// warp_reduce_scatter16, so every row is combined in the same association (fp add is commutative, not associative). +// After it lanes with bits 4..2 = (b4, b3, b2) hold row 4*b4 + 2*b3 + b2. +template +__device__ __forceinline__ float warp_reduce_scatter8(float (&a)[8], int lane) +{ +#pragma unroll + for (int step = 0; step < 3; ++step) + { + int const mask = 16 >> step; + int const half = 4 >> step; + bool const upper = (lane & mask) != 0; +#pragma unroll + for (int i = 0; i < half; ++i) + { + float const send = upper ? a[i] : a[i + half]; + float const keep = upper ? a[i + half] : a[i]; + float const recv = __shfl_xor_sync(0xffffffffu, send, mask); + a[i] = kMax ? fmaxf(keep, recv) : keep + recv; + } + } + float x = a[0]; + float const o2 = __shfl_xor_sync(0xffffffffu, x, 2); + x = kMax ? fmaxf(x, o2) : x + o2; + float const o1 = __shfl_xor_sync(0xffffffffu, x, 1); + return kMax ? fmaxf(x, o1) : x + o1; +} + +// Reduce 16 per-lane values over the warp, one value per row: after the reduce-scatter lane l holds row +// 8*b4 + 4*b3 + 2*b2 + b1 (b_k = bit k of l) reduced over all 32 lanes. 16 shuffles instead of 80. +template +__device__ __forceinline__ float warp_reduce_scatter16(float (&a)[16], int lane) +{ +#pragma unroll + for (int step = 0; step < 4; ++step) + { + int const mask = 16 >> step; + int const half = 8 >> step; + bool const upper = (lane & mask) != 0; +#pragma unroll + for (int i = 0; i < half; ++i) + { + float const send = upper ? a[i] : a[i + half]; + float const keep = upper ? a[i + half] : a[i]; + float const recv = __shfl_xor_sync(0xffffffffu, send, mask); + a[i] = kMax ? fmaxf(keep, recv) : keep + recv; + } + } + float const other = __shfl_xor_sync(0xffffffffu, a[0], 1); + return kMax ? fmaxf(a[0], other) : a[0] + other; +} + +__device__ __forceinline__ void umma_f16( + uint32_t tmem_d, uint64_t adesc, uint64_t bdesc, uint32_t idesc, int accumulate) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, %4, 0;\n tcgen05.mma.cta_group::1.kind::f16 [%0], %1, %2, %3, p;\n}\n" :: + "r"(tmem_d), + "l"(adesc), "l"(bdesc), "r"(idesc), "r"(accumulate) + : "memory"); +} + +__device__ __forceinline__ void umma_commit(uint32_t bar) +{ + asm volatile("tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.b64 [%0];\n" ::"r"(bar) : "memory"); +} + +__device__ __forceinline__ void mbar_wait_cta(uint32_t bar, uint32_t parity) +{ + uint32_t ready = 0; +#ifdef M3FA_DBG_HANG + long long spins = 0; +#endif + while (!ready) + { +#ifdef M3FA_DBG_HANG + if (++spins == (1LL << 24)) + { + printf("M3FA hang: block %d thread %d mbar 0x%x parity %u\n", blockIdx.x, threadIdx.x, bar, parity); + } +#endif + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.shared::cta.b64 p, [%1], %2;\n selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(bar), "r"(parity) + : "memory"); + } +} + +// Spin without suspending (v52's acc_done wait): a suspended try_wait wakes late on a tcgen05.commit arrival. +#ifndef M3FA_EPI_SPIN +#define M3FA_EPI_SPIN 1 +#endif +__device__ __forceinline__ void mbar_wait_epi(uint32_t bar, uint32_t parity) +{ + if (!M3FA_EPI_SPIN) + { + mbar_wait_cta(bar, parity); + return; + } + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.shared::cta.b64 p, [%1], %2;\n selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(bar), "r"(parity) + : "memory"); + } +} + +__device__ __forceinline__ void tmem_ld8(uint32_t addr, float (&v)[8]) +{ + uint32_t r[8]; + asm volatile("tcgen05.ld.sync.aligned.32x32b.x8.b32 {%0, %1, %2, %3, %4, %5, %6, %7}, [%8];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]), "=r"(r[4]), "=r"(r[5]), "=r"(r[6]), "=r"(r[7]) + : "r"(addr)); + asm volatile("tcgen05.wait::ld.sync.aligned;\n" ::: "memory"); +#pragma unroll + for (int i = 0; i < 8; ++i) + { + v[i] = __uint_as_float(r[i]); + } +} + +__device__ __forceinline__ void tmem_ld16(uint32_t addr, float (&v)[16]) +{ + uint32_t r[16]; + asm volatile( + "tcgen05.ld.sync.aligned.32x32b.x16.b32 {%0, %1, %2, %3, %4, %5, %6, %7, %8, %9, %10, %11, %12, %13, %14, " + "%15}, [%16];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]), "=r"(r[4]), "=r"(r[5]), "=r"(r[6]), "=r"(r[7]), "=r"(r[8]), + "=r"(r[9]), "=r"(r[10]), "=r"(r[11]), "=r"(r[12]), "=r"(r[13]), "=r"(r[14]), "=r"(r[15]) + : "r"(addr)); + asm volatile("tcgen05.wait::ld.sync.aligned;\n" ::: "memory"); +#pragma unroll + for (int i = 0; i < 16; ++i) + { + v[i] = __uint_as_float(r[i]); + } +} + +// Warp-level fp16 MMA (m16n8k16, fp32 accumulate) and its shared-memory fragment loads. +__device__ __forceinline__ void ldsm_x4(uint32_t addr, uint32_t (&r)[4]) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0, %1, %2, %3}, [%4];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x2(uint32_t addr, uint32_t (&r)[2]) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x2.shared.b16 {%0, %1}, [%2];\n" : "=r"(r[0]), "=r"(r[1]) : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x2_trans(uint32_t addr, uint32_t (&r)[2]) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x2.trans.shared.b16 {%0, %1}, [%2];\n" + : "=r"(r[0]), "=r"(r[1]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma_f16_16816(float (&d)[4], uint32_t const (&a)[4], uint32_t const (&b)[2]) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0, %1, %2, %3}, {%4, %5, %6, %7}, {%8, %9}, " + "{%0, %1, %2, %3};\n" + : "+f"(d[0]), "+f"(d[1]), "+f"(d[2]), "+f"(d[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b[0]), "r"(b[1])); +} + +// Register attention core (warp MMAs, no TMEM) for up to two tokens per cluster; the tcgen05 core above that. +// Off by default: on GB300 it measured ~0.25 us slower than the tcgen05 core at M=1 (GEMV + attention chain). +#ifndef M3FA_REG_CORE +#define M3FA_REG_CORE 0 +#endif +template +struct RegCore +{ + static constexpr bool kOn = M3FA_REG_CORE != 0 && QG <= 2; +}; + +template +struct Smem +{ + static constexpr int kRows = 16 * QG; // query rows (token, head) of this cluster = UMMA N + static constexpr int kTok = BPC * kPage; // block tokens per CTA + static constexpr int kTileBytes = kPage * kHeadDim * 2; // one fp16 K or V block tile in UMMA layout + static constexpr int kK16 = 0; + static constexpr int kV16 = kK16 + BPC * kTileBytes; + static constexpr int kQ = kV16 + BPC * kTileBytes; // Q rows, UMMA K-major [row][dim] + // P^T per block, UMMA MN-major [token][row], over the block's K tile: P is written after the QK MMAs (which + // read every K tile) have completed. + static constexpr int kP = kK16; + // Packed V tiles as copied from the cache: [block][128 rows x 64 data bytes | 1 KB of 4x4-quad scales]. + static constexpr int kRawV = kQ + kRows * kHeadDim * 2; + static constexpr int kRawVBlock = kPage * 64 + kPage * 8; + static constexpr int kRowScale = kRawV + BPC * kRawVBlock; + static constexpr int kRowLse = kRowScale + BPC * kRows * 4; + // Per-warp softmax partials [block][row][4 warps]: max, then sum. + static constexpr int kRedM = (kRowLse + BPC * kRows * 4 + 15) / 16 * 16; + static constexpr int kRedL = kRedM + BPC * kRows * 16; + // Partials pushed by every CTA of the cluster: [chunk][token][local head][kOPitch] floats, lse at column 128. + // When they do not fit beside the tiles they overlay the K/V tiles (and P, Q): peers then push only once every + // CTA of the cluster has completed its PV MMAs (a cluster barrier right before the push). + // Register core: per-warp row maxima and sums [block][row][warp of the block], two arrays. + static constexpr int kRedX = kRedL + BPC * kRows * 16; + static constexpr int kRedXBytes = RegCore::kOn ? 2 * BPC * kRows * (16 / BPC) * 4 : 0; + static constexpr int kStatsEnd = kRedX + kRedXBytes; + // Register core: P (fp16, [row][token] core matrices) at the start of its block's K tile, the scaled-free O + // partial ([row][kOPitch] fp32) behind it. + static constexpr int kObufOff = 8192; + static_assert(!RegCore::kOn || (kRows * kPage * 2 <= kObufOff && kObufOff + kRows * kOPitch * 4 <= kTileBytes), + "register-core P and O staging fit in a K tile"); + static constexpr int kRecvBytes = kBlocks * QG * BPC * kOPitch * 4; + static constexpr bool kRecvAlias = kStatsEnd + kRecvBytes + 32 > 227 * 1024; + static constexpr int kRecv = kRecvAlias ? 0 : kStatsEnd; + static constexpr int kMbar = kRecvAlias ? kStatsEnd : kStatsEnd + kRecvBytes; // merge, MMA mbarriers, TMEM addr + static constexpr int kBytes = kMbar + 32; + static constexpr int kTmemCols = BPC * 2 * kRows <= 32 ? 32 + : BPC * 2 * kRows <= 64 ? 64 + : BPC * 2 * kRows <= 128 ? 128 + : BPC * 2 * kRows <= 256 ? 256 + : 512; + static_assert(BPC * 2 * kRows <= 512, "TMEM columns"); + static_assert(!kRecvAlias || kRecvBytes <= kRawV, "aliased partials must fit in the tiles, Q and P"); + static_assert(kRows * kPage * 2 <= kTileBytes, "P^T fits in the K tiles"); + static_assert(kBytes <= 227 * 1024, "shared memory budget"); + static_assert(QLEN % QG == 0 && (BPC == 1 || BPC == 2), "unsupported geometry"); +}; + +// The packed NVFP4 bytes of one 16-byte chunk (32 dims of one block row) and its two block scales. +struct RawChunk +{ + uint4 data; + uint32_t scales; // byte 0: dims [0, 16), byte 1: dims [16, 32) of the chunk +}; + +// Chunk q of a block tile: lanes 8j..8j+7 of a warp take 8 consecutive rows of one 32-dim chunk, so every 16-byte +// fp16 store phase writes 8 distinct core-matrix rows (conflict-free in the UMMA layout). +__device__ __forceinline__ int chunk_row(int q) +{ + return (q >> 5) * 8 + (q & 7); +} + +__device__ __forceinline__ int chunk_col(int q) +{ + return (q >> 3) & 3; +} + +__device__ __forceinline__ RawChunk load_chunk( + uint8_t const* data_page, uint8_t const* scale_page, int row, int cc, bool is_v) +{ + RawChunk c; + c.data = *reinterpret_cast(data_page + row * 64 + cc * 16); + if (is_v) + { + // V scales: 4x4 token-quad order [row / 4][scale column][row % 4]. + uint8_t const* sc = scale_page + (row >> 2) * 32 + (row & 3); + c.scales = static_cast(sc[(cc * 2) * 4]) | (static_cast(sc[(cc * 2 + 1) * 4]) << 8); + } + else + { + c.scales = *reinterpret_cast(scale_page + row * 8 + cc * 2); + } + return c; +} + +__device__ __forceinline__ void store_chunk_fp16(uint8_t* tile, int row, int cc, RawChunk const& c, bool valid) +{ + uint32_t const s0 = e4m3_to_h2_splat(c.scales & 0xFFu); + uint32_t const s1 = e4m3_to_h2_splat((c.scales >> 8) & 0xFFu); + uint32_t const wd[4] = {c.data.x, c.data.y, c.data.z, c.data.w}; + uint32_t out[16]; +#pragma unroll + for (int i = 0; i < 16; ++i) + { + uint32_t const b = (wd[i >> 2] >> (8 * (i & 3))) & 0xFFu; + out[i] = valid ? hmul2_u(e2m1x2_to_h2(b), i < 8 ? s0 : s1) : 0u; + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + *reinterpret_cast(tile + kmaj_off(row, cc * 32 + j * 8)) + = make_uint4(out[4 * j], out[4 * j + 1], out[4 * j + 2], out[4 * j + 3]); + } +} + +// ---------------- Fused QKV projection: block-scaled MXFP8 on tcgen05 (N=2560, K=6144) ---------------- +// out[M, 2560] = mxfp8(x)[M, 6144] @ W^T, the computation of the CUTLASS MXFP8 GEMM: W is the A operand (128 output +// rows per MMA), the quantized activation the B operand (8 token columns), FP32 accumulation in TMEM. A unit is one +// (128-row tile, K split) pair; the 20 x kQkvSplits units run on the last CTAs of the grid. Before the PDL wait a +// CTA streams its unit's weight tiles and scale atoms through a shared-memory ring into TMEM; after it, it quantizes +// its x slice (mxfp8_quantize math), runs the MMAs and stores its FP32 partial rows. +// +// Hand-off without a flag: the partial buffers are double buffered by call parity and hold a sentinel (an FP32 NaN +// the MMA never produces) at rest, so a consumer polls the partials themselves. The producers of call n reset the +// other parity's buffers after their PDL wait; call n - 1, the last reader of those, is complete by then. +constexpr int kQkvN = 2560; +constexpr int kQkvK = 6144; +constexpr int kQkvKTiles = kQkvK / 128; +constexpr int kQkvRowTiles = kQkvN / 128; +#ifndef M3FA_QKV_SPLITS +#define M3FA_QKV_SPLITS 4 +#endif +#ifndef M3FA_QKV_RING +#define M3FA_QKV_RING kQkvMaxTiles // every tile of the unit in flight at once +#endif +#ifndef M3FA_QKV_NACC +#define M3FA_QKV_NACC 1 +#endif +#ifndef M3FA_QKV_SCALED_MMA +#define M3FA_QKV_SCALED_MMA 1 // 0: unscaled per-block MMAs with the scales applied in the epilogue (see below) +#endif +#ifndef M3FA_QKV_PIPE +// 1: the weight TMA is issued first at entry; the activation has its own region, copied right after the PDL wait; +// each weight tile's MMAs issue as soon as it lands (v52's order). 0: tiles into TMEM first, activation over the ring. +#define M3FA_QKV_PIPE 1 +#endif +#ifndef M3FA_QKV_A_SMEM +#define M3FA_QKV_A_SMEM 1 // scaled MMAs only; 1: they read the weight tiles from shared memory (no copy into TMEM) +#endif +// Default: block-scaled kind::mxf8f6f4 MMAs reading A (the weight tiles) in place from the ring, issued per tile as +// it lands (M3FA_QKV_PIPE), one accumulator; M3FA_QKV_A_SMEM=0 copies each tile into TMEM first. Alternative +// (M3FA_QKV_SCALED_MMA=0): one unscaled kind::f8f6f4 MMA per 32-wide K block into its own accumulator, A read in place +// from shared memory, the epilogue applying both UE8M0 scales. On GB300 an M128 x N8 x K32 MMA occupies the tensor pipe +// ~105 cycles block-scaled and ~60 unscaled, independent of N up to 64; the unscaled form's epilogue (320 accumulator +// columns) costs more than it saves. +constexpr bool kQkvScaled = M3FA_QKV_SCALED_MMA != 0; +constexpr bool kQkvASmem = !kQkvScaled || M3FA_QKV_A_SMEM != 0; +constexpr int kQkvSplits = M3FA_QKV_SPLITS; +constexpr int kQkvUnits = kQkvRowTiles * kQkvSplits; +constexpr int kQkvMaxTiles = (kQkvKTiles + kQkvSplits - 1) / kQkvSplits; +constexpr int kQkvRing = M3FA_QKV_RING; +constexpr int kQkvNacc = M3FA_QKV_NACC; +constexpr int kQkvGrid = 112; // 7 clusters of 16 (or 14 of 8): one wave with the attention +constexpr int kMaxTok = 8; +constexpr int kQkvWTile = 128 * 128; // E4M3 bytes of one 128 x 128 weight tile +constexpr int kQkvSfAtom = 512; // one 128-row x 4-block UE8M0 scale atom +constexpr uint32_t kQkvSentinel = 0xFFFFFFFFu; +constexpr int64_t kQkvPartStride = static_cast(kQkvSplits) * kMaxTok * kQkvN; // one parity +// TMEM columns of the one 512-column allocation of a CTA: attention [0, 64), then the GEMM's accumulators, scale +// atoms (A per tile, B per tile) and the A tiles (32 columns of 128 x 128 E4M3 each). +constexpr int kTmemAll = 512; +// Megakernel: the attention CTAs run no projection unit, so the GEMM starts at column 0 of a projection CTA. +constexpr int kTAcc = 0; +constexpr int kTSfa = kTAcc + 16; +constexpr int kTSfb = kTSfa + 4 * kQkvMaxTiles; +constexpr int kTA = (kTSfb + 4 * kQkvMaxTiles + 31) / 32 * 32; +static_assert(!kQkvScaled || kTA + 32 * kQkvMaxTiles <= kTmemAll, "TMEM columns"); +static_assert(kQkvScaled || kTAcc + 32 * kQkvMaxTiles <= kTmemAll, "TMEM columns (one accumulator per K block)"); +static_assert(!kQkvASmem || kQkvRing == kQkvMaxTiles, "shared-memory A needs every tile resident"); +static_assert(kQkvUnits <= kQkvGrid, "at most one unit per CTA"); +static_assert(kQkvNacc == 1 || kQkvNacc == 2 || kQkvNacc == 4, "accumulator count"); +static_assert(!M3FA_QKV_PIPE || kQkvScaled, "the pipelined projection uses the block-scaled MMAs"); +constexpr int kQkvOwnRows = 128 / kQkvSplits; // rows of a tile whose split-K sum a CTA owns +// Megakernel o_proj (v52 m3_mx_gemv for [6144, 2048]: split 2, contiguous k-tiles, one accumulator): unit +// o = CTA - 16 (the last 96 CTAs), row tile o / 2, split o % 2. +constexpr int kOpN = 6144; +constexpr int kOpK = 2048; +constexpr int kOpKTiles = kOpK / 128; +constexpr int kOpSplits = 2; +constexpr int kOpTiles = kOpKTiles / kOpSplits; +constexpr int kOpOwnRows = 128 / kOpSplits; +constexpr int kOpFirstCta = kQkvGrid - kOpN / 128 * kOpSplits; // 16 +constexpr int kOpSfBytes = kOpKTiles * 512; // one parity of the MXFP8 hand-off's scales +#ifndef M3FA_OP_L2PF +#define M3FA_OP_L2PF \ + 0 // 1: QKV CTAs prefetch their o_proj weight slice into L2 before the PDL wait (competes + // with the QKV weight stream for HBM behind the MoE) +#endif +// o_proj TMEM columns (after the QKV unit's MMAs completed): accumulator, A / B scale atoms, A (8 tiles x 32) +// c1 (attention + o_proj kernel): the attention holds columns [0, 32) while the o_proj weights stream in, so the +// o_proj accumulator and scale atoms sit above them. +constexpr int kOAcc = 32; +constexpr int kOSfa = 48; +constexpr int kOSfb = kOSfa + 4 * kOpTiles; +constexpr int kOA = 128; +static_assert(kOA + 32 * kOpTiles <= kTmemAll && kOSfb + 4 * kOpTiles <= kOA, "o_proj TMEM columns"); +constexpr int kQkvMbStride = kMaxTok + 4; // mailbox row pitch (fp32 words) + +__host__ __device__ constexpr int qkv_tile0(int s) +{ + return s * kQkvKTiles / kQkvSplits; +} + +__device__ __forceinline__ int qkv_unit() +{ + int const u = kQkvGrid - 1 - static_cast(blockIdx.x); + return u < kQkvUnits ? u : -1; +} + +// Attention layout plus, for the fused projection, the weight ring in front (the attention tiles reuse that space +// once the ring has drained into TMEM); the activation tiles, the pre-wait V copy and the barriers sit behind. +template +struct SmemX : Smem +{ + using B = Smem; + static constexpr int kRingW = 0; // [ring][128 rows][128 B], 128-byte swizzle + static constexpr int kRingSf = kRingW + kQkvRing * kQkvWTile; // [ring][512 B] + static constexpr int kRingEnd = kRingSf + kQkvRing * kQkvSfAtom; + // The activation tiles overlay the ring: written after the PDL wait, once every weight tile went on into TMEM. + static constexpr int kXt = kRingW; // [tile][8][128 B] + static constexpr int kXsf = kXt + kQkvMaxTiles * 1024; // [tile][512 B] + static constexpr int kXsc = (kRingEnd + 127) / 128 * 128; + // Peers push into recv at any time (an empty CTA right away), so it stays clear of the ring. + static constexpr int kRecv = FQKV ? kXsc : B::kRecv; + static constexpr int kRawV = FQKV ? (kRecv + kBlocks * QG * BPC * kOPitch * 4 + 127) / 128 * 128 : B::kRawV; + // Pipelined projection: the activation tiles and scale atoms in their own region (a projection CTA runs no + // attention, so it overlays the attention's partial-O and V regions), 1024-byte aligned for the swizzle. + static constexpr int kXA = FQKV ? (kRecv + 1023) / 1024 * 1024 : 0; + static constexpr int kXASf = kXA + kQkvMaxTiles * 1024; + static constexpr int kXAEnd = kXASf + kQkvMaxTiles * kQkvSfAtom; + static constexpr int kMbarA = FQKV ? kRawV + BPC * B::kRawVBlock : B::kMbar; + static constexpr int kMbar = FQKV && M3FA_QKV_PIPE && kXAEnd > kMbarA ? (kXAEnd + 127) / 128 * 128 : kMbarA; + static constexpr int kGBar = kMbar + 32; // full[ring], empty[ring], a_ready, acc_done, tile_done[tiles] + static constexpr int kRedBar = kGBar + 8 * (2 * kQkvRing + 2 + kQkvMaxTiles); // split-K partials in + static constexpr int kOBar = kRedBar + 8; // o_proj: full[8], a_ready, acc_done, reduce + static constexpr int kMailbox = (kOBar + 8 * (kOpTiles + 3) + 15) / 16 * 16; // [split][own row][pitch] fp32 + // o_proj weight ring = ring tiles [0, 8); its activation tiles, scale atoms and mailbox behind it. + static constexpr int kOXt = kRingW + kOpTiles * kQkvWTile; + static constexpr int kOXsf = kOXt + kOpTiles * 1024; + static constexpr int kOMailbox = kOXsf + kOpTiles * kQkvSfAtom; + static_assert(!FQKV || kOMailbox + kOpSplits * kOpOwnRows * kQkvMbStride * 4 <= kRingW + kQkvRing * kQkvWTile, + "o_proj activation and mailbox fit behind its ring"); + static constexpr int kBytes = FQKV ? kMailbox + kQkvSplits * kQkvOwnRows * kQkvMbStride * 4 : B::kBytes; + static constexpr int kTmemCols = FQKV ? kTmemAll : B::kTmemCols; + static constexpr bool kRecvAlias = !FQKV && B::kRecvAlias; + // The epilogue reads the weight scale atoms while the attention already writes its tiles. + static_assert(!FQKV || kQkvScaled || B::kRecv <= kRingSf, "attention regions overlap the scale atoms"); + // The other attention regions overlay the ring: a unit's CTA writes them only once the ring has drained. + static_assert(kBytes <= 227 * 1024, "shared memory budget"); +}; + +// c1: the packed-input attention plus the o_proj unit blockIdx.x (tile bx / 2, split bx % 2). The o_proj weight ring +// and its barriers sit behind the attention layout; the activation tiles, scale atoms and the pair's split-K mailbox +// overlay the K tiles, free once this CTA has pushed its partials (a peer's mailbox push follows that push: its +// activation needs the merges of this cluster, which need this CTA's partials). +template +struct SmemC1 : SmemX +{ + using B = Smem; + static constexpr int kRingW = (B::kBytes + 1023) / 1024 * 1024; + static constexpr int kRingSf = kRingW + kOpTiles * kQkvWTile; + static constexpr int kOBar = kRingSf + kOpTiles * kQkvSfAtom; // full[8], a_ready, acc_done, reduce + static constexpr int kOXt = B::kK16; + static constexpr int kOXsf = kOXt + kOpTiles * 1024; + static constexpr int kOMailbox = kOXsf + kOpTiles * kQkvSfAtom; + static constexpr int kBytes = kOBar + 8 * (kOpTiles + 3); + static constexpr int kTmemCols = kTmemAll; + static_assert(!B::kRecvAlias, "c1: the partials have their own region"); + static_assert( + kOMailbox + kOpSplits * kOpOwnRows * kQkvMbStride * 4 <= B::kV16, "c1: o_proj overlay in the K tiles"); + static_assert(BPC * 2 * B::kRows <= kOAcc, "c1: attention TMEM columns below the o_proj's"); + static_assert(kBytes <= 227 * 1024, "c1: shared memory budget"); +}; + +__device__ __forceinline__ void mbar_init(uint32_t bar, uint32_t count) +{ + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;\n" ::"r"(bar), "r"(count) : "memory"); +} + +__device__ __forceinline__ void mbar_expect_tx(uint32_t bar, uint32_t bytes) +{ + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(bar), "r"(bytes) : "memory"); +} + +__device__ __forceinline__ void bulk_g2s(void* dst, void const* src, uint32_t bytes, uint32_t bar) +{ + asm volatile( + "cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes.L2::cache_hint [%0], [%1], %2, [%3], " + "%4;\n" ::"r"(smem_u32(dst)), + "l"(src), "r"(bytes), "r"(bar), "l"(0x12F0000000000000ull) // EVICT_FIRST: weights are read once per layer + : "memory"); +} + +__device__ __forceinline__ void tma_load_2d(uint32_t dst, void const* map, int c0, int c1, uint32_t bar) +{ + asm volatile( + "cp.async.bulk.tensor.2d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.L2::cache_hint" + " [%0], [%1, {%2, %3}], [%4], %5;\n" ::"r"(dst), + "l"(map), "r"(c0), "r"(c1), "r"(bar), "l"(0x12F0000000000000ull) + : "memory"); +} + +// K-major operand in the 128-byte swizzle: rows of 128 bytes, 8-row groups 1024 bytes apart. +__device__ __forceinline__ uint64_t umma_desc_sw128(uint32_t addr) +{ + return umma_desc(addr, 16, 1024) | (2ull << 61); +} + +__device__ __forceinline__ void tmem_cp_128x256b(uint32_t taddr, uint64_t desc) +{ + asm volatile("tcgen05.cp.cta_group::1.128x256b [%0], %1;\n" ::"r"(taddr), "l"(desc)); +} + +__device__ __forceinline__ void tmem_cp_sf(uint32_t taddr, uint32_t saddr) +{ + asm volatile("tcgen05.cp.cta_group::1.32x128b.warpx4 [%0], %1;\n" ::"r"(taddr), "l"(umma_desc(saddr, 16, 128))); +} + +// Block-scaled MXFP8 instruction descriptor: E4M3 x E4M3, UE8M0 scales, M = 128, N = 8, scale byte kb. +__host__ __device__ constexpr uint32_t qkv_idesc(int kb) +{ + return (static_cast(kb) << 4) | ((8u >> 3) << 17) | (1u << 23) | ((128u >> 7) << 27) + | (static_cast(kb) << 29); +} + +__device__ __forceinline__ void umma_mx( + uint32_t tmem_d, uint32_t tmem_a, uint64_t bdesc, uint32_t idesc, uint32_t sfa, uint32_t sfb, int accumulate) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, %6, 0;\n" + " tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X [%0], [%1], %2, %3, [%4], [%5], p;\n}\n" :: + "r"(tmem_d), + "r"(tmem_a), "l"(bdesc), "r"(idesc), "r"(sfa), "r"(sfb), "r"(accumulate)); +} + +__device__ __forceinline__ float pow2_from_ue8m0(uint32_t e) +{ + return __uint_as_float(e == 0 ? 0x00400000u : (e << 23)); +} + +// Unscaled E4M3 x E4M3 MMA, FP32 accumulator: M = 128, N = 8, K = 32, a fresh accumulator. +constexpr uint32_t kQkvIdescF8 = (1u << 4) | ((8u >> 3) << 17) | ((128u >> 4) << 24); + +__device__ __forceinline__ void umma_f8_ss(uint32_t tmem_d, uint64_t adesc, uint64_t bdesc) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, 0, 0;\n tcgen05.mma.cta_group::1.kind::f8f6f4 [%0], %1, %2, %3, p;\n}\n" :: + "r"(tmem_d), + "l"(adesc), "l"(bdesc), "r"(kQkvIdescF8)); +} + +__device__ __forceinline__ void tmem_ld16_nowait(uint32_t addr, float* v) +{ + uint32_t r[16]; + asm volatile( + "tcgen05.ld.sync.aligned.32x32b.x16.b32 {%0, %1, %2, %3, %4, %5, %6, %7, %8, %9, %10, %11, %12, %13, %14, " + "%15}, [%16];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]), "=r"(r[4]), "=r"(r[5]), "=r"(r[6]), "=r"(r[7]), "=r"(r[8]), + "=r"(r[9]), "=r"(r[10]), "=r"(r[11]), "=r"(r[12]), "=r"(r[13]), "=r"(r[14]), "=r"(r[15]) + : "r"(addr)); +#pragma unroll + for (int i = 0; i < 16; ++i) + { + v[i] = __uint_as_float(r[i]); + } +} + +__device__ __forceinline__ void umma_mx_ss( + uint32_t tmem_d, uint64_t adesc, uint64_t bdesc, uint32_t idesc, uint32_t sfa, uint32_t sfb, int accumulate) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, %6, 0;\n" + " tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X [%0], %1, %2, %3, [%4], [%5], p;\n}\n" :: + "r"(tmem_d), + "l"(adesc), "l"(bdesc), "r"(idesc), "r"(sfa), "r"(sfb), "r"(accumulate)); +} + +__device__ __forceinline__ bool elect_one() +{ + uint32_t pred; + asm volatile("{\n .reg .pred P;\n elect.sync _|P, 0xffffffff;\n selp.u32 %0, 1, 0, P;\n}\n" : "=r"(pred)); + return pred != 0; +} + +__device__ __forceinline__ int qkv_parity(Params const& p) +{ + int v; + asm volatile("ld.relaxed.gpu.global.s32 %0, [%1];\n" : "=r"(v) : "l"(p.counters) : "memory"); + return v & 1; +} + +// Barriers and TMEM (warp 1), then the first ring stages (warp 0). Warps 0 and 1 of every CTA call it. +template +__device__ __forceinline__ void qkv_setup( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int warp, int lane, int u) +{ + constexpr bool kPipe = M3FA_QKV_PIPE != 0; + if (kPipe && warp == 0 && lane == 0 && u >= 0) + { + // First thing at entry: this unit's whole weight slice. The issuing thread initializes the ring's full + // barriers itself; every waiter on them passes bar.sync 8 below, which this thread reaches after the init. + asm volatile("prefetch.tensormap [%0];\n" ::"l"(&p.wmap) : "memory"); + int const rt = u / kQkvSplits; + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + for (int i = 0; i < kQkvRing; ++i) + { + mbar_init(gbar + 8 * i, 1); + } + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + for (int i = 0; i < kQkvRing && i < nt; ++i) + { + uint32_t const full = gbar + 8 * i; + mbar_expect_tx(full, kQkvWTile + kQkvSfAtom); + if (p.wpre != nullptr) + { + // Pre-tiled weight: the tile's 16 KB shared-memory image is contiguous (one bulk copy). + bulk_g2s(smem + SX::kRingW + i * kQkvWTile, + p.wpre + (static_cast(rt) * kQkvKTiles + kt0 + i) * kQkvWTile, kQkvWTile, full); + } + else + { + tma_load_2d(smem_u32(smem + SX::kRingW + i * kQkvWTile), &p.wmap, (kt0 + i) * 128, rt * 128, full); + } + bulk_g2s(smem + SX::kRingSf + i * kQkvSfAtom, + p.wsf + (static_cast(rt) * kQkvKTiles + kt0 + i) * kQkvSfAtom, kQkvSfAtom, full); + } + M3FA_PROF_T(34, 0); + } + if (warp == 1) + { + asm volatile("tcgen05.alloc.cta_group::1.sync.aligned.shared::cta.b32 [%0], %1;\n" ::"r"(smem_u32(tmem_slot)), + "r"(kTmemAll) + : "memory"); + asm volatile("tcgen05.relinquish_alloc_permit.cta_group::1.sync.aligned;\n" ::: "memory"); + M3FA_PROF_T(35, 32); + if (lane == 0) + { + for (int i = (kPipe && u >= 0) ? kQkvRing : 0; i < 2 * kQkvRing + 2 + kQkvMaxTiles; ++i) + { + // PIPE 2: tile_done[0] (unused by the scaled MMAs) is act_ready, one arrival per thread outside warp 1 + mbar_init(gbar + 8 * i, M3FA_QKV_PIPE == 2 && i == 2 * kQkvRing + 2 ? kThreads - 32 : 1); + } + // The split-K reduce barrier: this CTA's own arrival carries the peers' partial bytes. + uint32_t const red = gbar + 8 * (2 * kQkvRing + 2 + kQkvMaxTiles); + mbar_init(red, 1); + mbar_expect_tx(red, (kQkvSplits - 1) * kQkvOwnRows * kMaxTok * 4); + uint32_t const ob = smem_u32(smem + SX::kOBar); + for (int i = 0; i < kOpTiles + 3; ++i) + { + mbar_init(ob + 8 * i, 1); + } + mbar_expect_tx(ob + 8 * (kOpTiles + 2), (kOpSplits - 1) * kOpOwnRows * kMaxTok * 4); + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + } + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + asm volatile("bar.sync 8, 64;\n" ::: "memory"); + if (!kPipe && warp == 0 && lane == 0 && u >= 0) + { + int const rt = u / kQkvSplits; + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + for (int i = 0; i < kQkvRing && i < nt; ++i) + { + uint32_t const full = gbar + 8 * i; + mbar_expect_tx(full, kQkvWTile + kQkvSfAtom); + tma_load_2d(smem_u32(smem + SX::kRingW + i * kQkvWTile), &p.wmap, (kt0 + i) * 128, rt * 128, full); + bulk_g2s(smem + SX::kRingSf + i * kQkvSfAtom, + p.wsf + (static_cast(rt) * kQkvKTiles + kt0 + i) * kQkvSfAtom, kQkvSfAtom, full); + } + M3FA_PROF_T(34, 0); + } +} + +// The rest of the ring (warp 0, lane 0) and the copies into TMEM (warp 1, lane 0); before the PDL wait. +template +__device__ __forceinline__ void qkv_stream_lane( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int warp, int u) +{ + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + uint32_t const full = gbar; + uint32_t const a_ready = gbar + 16 * kQkvRing; + if (warp == 0 || !kQkvScaled) + { + return; // every tile was issued in qkv_setup; the unscaled MMAs read them in place + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm = *tmem_slot; + // Fully unrolled with compile-time offsets: tcgen05 operands stay in uniform registers (a per-instruction + // conversion of computed operands costs ~150 cycles per MMA or copy). + uint64_t const w0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + uint32_t const sf0 = smem_u32(smem + SX::kRingSf); +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + mbar_wait_cta(full + 8 * i, 0); + if (i == 0) + { + M3FA_PROF_T(30, 32); + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if constexpr (!kQkvASmem) + { +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b(tm + kTA + (i * 4 + kb) * 8, w0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + } + tmem_cp_sf(tm + kTSfa + i * 4, sf0 + i * kQkvSfAtom); + } + } + M3FA_PROF_T(31, 32); + umma_commit(a_ready); +} + +__device__ __forceinline__ bool mbar_test(uint32_t bar, uint32_t parity) +{ + uint32_t ready; + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.shared::cta.b64 p, [%1], %2;\n selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(bar), "r"(parity) + : "memory"); + return ready != 0; +} + +// PIPE 2, warp 1's elected thread from before the PDL wait to the last MMA. Tiles that land before the activation +// is ready go on into TMEM (A and scale atom), off the critical path; once the activation is in (act_ready), the +// MMAs issue in tile order: the copied tiles with A in TMEM, every later tile as it lands with A read in place from +// the ring. The MMA sequence (and so the accumulation order) is the same either way. +template +__device__ __forceinline__ void qkv_hybrid(Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int u) +{ + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + uint32_t const act_ready = gbar + 8 * (2 * kQkvRing + 2); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm = *tmem_slot; + uint64_t const w0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + uint64_t const b0 = umma_desc_sw128(smem_u32(smem + SX::kXA)); + uint32_t const wsf0 = smem_u32(smem + SX::kRingSf); + uint32_t const xsf0 = smem_u32(smem + SX::kXASf); + int copied = 0; // tiles [0, copied) are in TMEM + bool act = false; +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt && !act) + { + while (!mbar_test(gbar + 8 * i, 0)) + { + if (mbar_test(act_ready, 0)) + { + act = true; + break; + } + } + if (!act) + { + if (i == 0) + { + M3FA_PROF_T(30, 32); + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b(tm + kTA + (i * 4 + kb) * 8, w0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + tmem_cp_sf(tm + kTSfa + i * 4, wsf0 + i * kQkvSfAtom); + copied = i + 1; + } + } + } + while (!act) + { + act = mbar_test(act_ready, 0); + } + M3FA_PROF_T(21, 32); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + bool const in_tmem = i < copied; + if (!in_tmem) + { + mbar_wait_epi(gbar + 8 * i, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + tmem_cp_sf(tm + kTSfa + i * 4, wsf0 + i * kQkvSfAtom); + } + uint32_t const sfb = tm + kTSfb + i * 4; + tmem_cp_sf(sfb, xsf0 + i * kQkvSfAtom); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + int const j = i * 4 + kb; + uint64_t const bd = b0 + ((i * 1024 + kb * 32) >> 4); + if (in_tmem) + { + umma_mx(tm + kTAcc, tm + kTA + j * 8, bd, qkv_idesc(kb), tm + kTSfa + i * 4, sfb, j > 0); + } + else + { + umma_mx_ss(tm + kTAcc, w0 + ((i * kQkvWTile + kb * 32) >> 4), bd, qkv_idesc(kb), tm + kTSfa + i * 4, + sfb, j > 0); + } + } + } + } + M3FA_PROF_T(22, 32); + M3FA_PROF_T(31, 32); + umma_commit(gbar + 16 * kQkvRing); // a_ready (for its waiters) + umma_commit(gbar + 16 * kQkvRing + 8); // acc_done +} + +// The other lanes of warps 0 and 1 hold at __syncwarp rather than at griddepcontrol.wait, which would stall the +// whole warp (lane 0's loop included) until the producer grid completes. +template +__device__ __forceinline__ void qkv_stream( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int warp, int lane, int u) +{ + if (M3FA_QKV_PIPE == 2 && u >= 0 && warp == 1) + { + if (elect_one()) + { + qkv_hybrid(p, smem, gbar, tmem_slot, u); + } + __syncwarp(); + return; + } + if (u < 0 || warp > 1 || M3FA_QKV_PIPE) + { + return; // pipelined: the MMA thread takes each tile as it lands + } + if (lane == 0) + { + qkv_stream_lane(p, smem, gbar, tmem_slot, warp, u); + } + __syncwarp(); +} + +// After the PDL wait. Every thread quantizes a piece of the x slice into the B tiles (named barrier 9); warp 1 then +// issues the scale copies and the MMAs. +template +__device__ __forceinline__ void qkv_mma( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int tid, int u) +{ + int const warp = tid >> 5; + int const s = u % kQkvSplits; + int const kt0 = qkv_tile0(s); + int const nt = qkv_tile0(s + 1) - kt0; + { + // The MXFP8 activation slice (E4M3 rows of the unit's k-tiles in 128-byte-swizzled B tiles, and their scale + // atoms), copied with plain 16-byte loads over the overlaid ring once its tiles went on into TMEM: every load + // first, then the stores (one L2 round trip). + constexpr int kXDst = M3FA_QKV_PIPE ? SX::kXA : SX::kXt; + constexpr int kXSfDst = M3FA_QKV_PIPE ? SX::kXASf : SX::kXsf; + if (!M3FA_QKV_PIPE) + { + mbar_wait_cta(gbar + 16 * kQkvRing, 0); // a_ready: the ring's weight tiles are in TMEM + } + constexpr int kXItems = kQkvMaxTiles * 8 * kMaxTok; // (tile, 16-byte chunk, token) + constexpr int kSfItems = kQkvMaxTiles * (kQkvSfAtom / 16); + constexpr bool kHyb = M3FA_QKV_PIPE == 2; + constexpr int kCopiers = kHyb ? kThreads - 32 : kThreads; // PIPE 2: warp 1 is issuing the tiles + if (kHyb && warp == 1) + { + return; + } + int const ctid = kHyb && tid >= 64 ? tid - 32 : tid; + constexpr int kPer = (kXItems + kSfItems + kCopiers - 1) / kCopiers; + uint4 v[kPer]; + int dst[kPer]; +#pragma unroll + for (int k = 0; k < kPer; ++k) + { + int const q = ctid + kCopiers * k; + dst[k] = -1; + if (q < kXItems) + { + int const t = q % kMaxTok; + int const c = (q / kMaxTok) % 8; + int const i = q / (kMaxTok * 8); + if (i < nt && t < p.num_tokens) + { + v[k] = *reinterpret_cast( + p.xq + static_cast(t) * p.ldxq + (kt0 + i) * 128 + c * 16); + dst[k] = kXDst + i * 1024 + t * 128 + ((c ^ t) * 16); + } + } + else if (q < kXItems + kSfItems) + { + int const j = q - kXItems; + int const i = j / (kQkvSfAtom / 16); + int const c = j % (kQkvSfAtom / 16); + if (i < nt) + { + v[k] = *reinterpret_cast( + p.xsf_act + static_cast(kt0 + i) * kQkvSfAtom + c * 16); + dst[k] = kXSfDst + i * kQkvSfAtom + c * 16; + } + } + } +#pragma unroll + for (int k = 0; k < kPer; ++k) + { + if (dst[k] >= 0) + { + *reinterpret_cast(smem + dst[k]) = v[k]; + } + } + asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory"); + if (kHyb) + { + asm volatile("mbarrier.arrive.release.cta.shared::cta.b64 _, [%0];\n" ::"r"(gbar + 8 * (2 * kQkvRing + 2)) + : "memory"); + return; + } + if (warp != 1) + { + asm volatile("bar.arrive 9, %0;\n" ::"n"(kThreads) : "memory"); + } + } + if (warp == 1) + { + M3FA_PROF_T(32, 32); + asm volatile("bar.sync 9, %0;\n" ::"n"(kThreads) : "memory"); + M3FA_PROF_T(21, 32); + // Operands computed by the whole warp (warp-uniform, kept in uniform registers); one elected lane issues. + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm = *tmem_slot; + uint64_t const b0 = umma_desc_sw128(smem_u32(smem + (M3FA_QKV_PIPE ? SX::kXA : SX::kXt))); + uint64_t const a0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + if (elect_one()) + { + M3FA_CLK(41); + if constexpr (!kQkvScaled) + { +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + mbar_wait_cta(gbar + 8 * i, 0); // weight tile i landed + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + umma_f8_ss(tm + kTAcc + (i * 4 + kb) * 8, a0 + ((i * kQkvWTile + kb * 32) >> 4), + b0 + ((i * 1024 + kb * 32) >> 4)); + } + umma_commit(gbar + 8 * (2 * kQkvRing + 2 + i)); // tile i's blocks are in TMEM + } + } + } + else + { + uint32_t const xsf0 = smem_u32(smem + (M3FA_QKV_PIPE ? SX::kXASf : SX::kXsf)); + uint32_t const wsf0 = smem_u32(smem + SX::kRingSf); +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + if constexpr (M3FA_QKV_PIPE != 0) + { + // Tile i as it lands: its A copy and scale atom into TMEM, then its MMAs (issue order). + mbar_wait_cta(gbar + 8 * i, 0); + if (i == 0) + { + M3FA_PROF_T(30, 32); + } + if (i == nt - 1) + { + M3FA_CLK(47); // the unit's last weight tile landed (clock64 builds) + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if constexpr (!kQkvASmem) + { +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b( + tm + kTA + (i * 4 + kb) * 8, a0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + } + tmem_cp_sf(tm + kTSfa + i * 4, wsf0 + i * kQkvSfAtom); + } + uint32_t const sfb = tm + kTSfb + i * 4; + tmem_cp_sf(sfb, xsf0 + i * kQkvSfAtom); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + int const j = i * 4 + kb; + uint64_t const bd = b0 + ((i * 1024 + kb * 32) >> 4); + if constexpr (kQkvASmem) + { + umma_mx_ss(tm + kTAcc + (j % kQkvNacc) * 8, a0 + ((i * kQkvWTile + kb * 32) >> 4), bd, + qkv_idesc(kb), tm + kTSfa + i * 4, sfb, j >= kQkvNacc); + } + else + { + umma_mx(tm + kTAcc + (j % kQkvNacc) * 8, tm + kTA + j * 8, bd, qkv_idesc(kb), + tm + kTSfa + i * 4, sfb, j >= kQkvNacc); + } + } + } + } + } + M3FA_CLK(42); + if (M3FA_QKV_PIPE) + { + M3FA_PROF_T(31, 32); + umma_commit(gbar + 16 * kQkvRing); // a_ready (kept for its waiters: every tile is consumed) + } + umma_commit(gbar + 16 * kQkvRing + 8); // acc_done + } + M3FA_PROF_T(22, 32); + } +} + +// Warps 16..19 (TMEM lanes 32 (warp % 4) + lane = the unit's rows): partial rows out, the other parity reset. +template +__device__ __forceinline__ void qkv_epilogue( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int tid, int u, int par, unsigned int calls) +{ + int const warp = tid >> 5; + int const lane = tid & 31; + if constexpr (kQkvScaled) + { + mbar_wait_epi(gbar + 16 * kQkvRing + 8, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + } + M3FA_PROF_T(24, 512); + if (tid == 512) + { + M3FA_CLK(43); + } + int const r = (warp & 3) * 32 + lane; // row within the unit's tile + uint32_t const taddr = *tmem_slot + (((warp & 3) * 32) << 16) + kTAcc; + float val[kMaxTok]; + if constexpr (kQkvScaled) + { + float v[32]; + tmem_ld16(taddr, *reinterpret_cast(v)); + if constexpr (kQkvNacc == 4) + { + tmem_ld16(taddr + 16, *reinterpret_cast(v + 16)); + } +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + val[t] = kQkvNacc == 4 ? (v[t] + v[8 + t]) + (v[16 + t] + v[24 + t]) + : kQkvNacc == 2 ? v[t] + v[8 + t] + : v[t]; + } + } + else + { + // Sum over K blocks in order: block result x (weight scale x activation scale). + int const nt = qkv_tile0(u % kQkvSplits + 1) - qkv_tile0(u % kQkvSplits); + float const* xsc = reinterpret_cast(smem + SX::kXsc); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + val[t] = 0.0f; + } +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + // Tile i's four blocks as soon as its MMAs complete: the epilogue trails the MMAs by one tile. + mbar_wait_cta(gbar + 8 * (2 * kQkvRing + 2 + i), 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + float d[32]; + tmem_ld16_nowait(taddr + i * 32, d); + tmem_ld16_nowait(taddr + i * 32 + 16, d + 16); + uint32_t const wsw = *reinterpret_cast( + smem + SX::kRingSf + i * kQkvSfAtom + (r % 32) * 16 + (r / 32) * 4); + asm volatile("tcgen05.wait::ld.sync.aligned;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + float const ws = pow2_from_ue8m0((wsw >> (8 * kb)) & 0xFFu); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + val[t] = fmaf(d[kb * 8 + t], ws * xsc[(i * 4 + kb) * kMaxTok + t], val[t]); + } + } + } + } + } + // v52's split-K reduce: split s owns rows [32 s, 32 s + 32) of the tile; the other splits push their fp32 partials + // of those rows into its mailbox (st.async, completing its reduce barrier's bytes); the owner adds the splits in + // rank order from 0.0 and stores bf16. + int const s = u % kQkvSplits; + int const owner = r / kQkvOwnRows; + int const local = r % kQkvOwnRows; + uint32_t const red = gbar + 8 * (2 * kQkvRing + 2 + kQkvMaxTiles); + uint32_t cta_rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(cta_rank)); + // Units are numbered down from the last CTA: split q of this tile is CTA rank cta_rank - (q - s). + if (owner != s) + { + uint32_t const peer = cta_rank + s - owner; + uint32_t const dst + = mapa_shared(smem_u32(smem + SX::kMailbox) + ((s * kQkvOwnRows + local) * kQkvMbStride) * 4, peer); + uint32_t const pbar = mapa_shared(red, peer); +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + asm volatile( + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [%0], {%1, %2, %3, %4}, [%5];\n" ::"r"( + dst + 16 * g), + "r"(__float_as_uint(val[4 * g])), "r"(__float_as_uint(val[4 * g + 1])), + "r"(__float_as_uint(val[4 * g + 2])), "r"(__float_as_uint(val[4 * g + 3])), "r"(pbar) + : "memory"); + } + } + else + { + M3FA_PROF_T(26, tid - lane); + // test_wait (no suspend): completions by peers' st.async bytes wake a suspended try_wait late. + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(red) + : "memory"); + } + M3FA_PROF_T(27, tid - lane); + float const* mb = reinterpret_cast(smem + SX::kMailbox); + int const row = (u / kQkvSplits) * 128 + r; + // Lamport hand-off: this call's rows go to buffer (calls & 1) with relaxed stores, the consumers poll the + // values themselves (0xFFFF, a NaN the rounding of a finite sum never produces, marks "not yet"); the other + // buffer, last read by the previous call (complete: this grid passed its PDL wait), is reset to the marker. + uint16_t* cur = reinterpret_cast(p.qkv_out) + (calls & 1u) * kMaxTok * kQkvN; + uint16_t* nxt = reinterpret_cast(p.qkv_out) + ((calls & 1u) ^ 1u) * kMaxTok * kQkvN; +#pragma unroll + float qsum[kMaxTok]; +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + float4 part[kQkvSplits]; +#pragma unroll + for (int q = 0; q < kQkvSplits; ++q) + { + part[q] = *reinterpret_cast(mb + (q * kQkvOwnRows + local) * kQkvMbStride + 4 * g); + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float sum = 0.0F; +#pragma unroll + for (int q = 0; q < kQkvSplits; ++q) + { + float const pj = j == 0 ? part[q].x : j == 1 ? part[q].y : j == 2 ? part[q].z : part[q].w; + sum = sum + (q == s ? val[4 * g + j] : pj); + } + qsum[4 * g + j] = sum; + } + } +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + float const sum = qsum[t]; + if (t < p.num_tokens) + { + __nv_bfloat16 const b = __float2bfloat16_rn(sum); + uint16_t bits = *reinterpret_cast(&b); + // Never the "not yet" marker: a 0xFFFF NaN sum is stored as the NaN 0x7FFF. Only a NaN projection row + // changes (its K/V cache bytes then differ from v52 + p6 in the NaN's encoding); finite rows are exact. + bits = bits == 0xFFFFu ? static_cast(0x7FFFu) : bits; +#ifndef M3FA_QKV_REL +#define M3FA_QKV_REL 0 // 1: the row store is a release (tried: whether it becomes visible to the pollers sooner) +#endif + if (M3FA_QKV_REL) + { + asm volatile( + "st.release.gpu.global.u16 [%0], %1;\n" ::"l"(cur + static_cast(t) * kQkvN + row), + "h"(bits) + : "memory"); + } + else + { + asm volatile( + "st.relaxed.gpu.global.u16 [%0], %1;\n" ::"l"(cur + static_cast(t) * kQkvN + row), + "h"(bits) + : "memory"); + } + } + asm volatile("st.relaxed.gpu.global.u16 [%0], %1;\n" ::"l"(nxt + static_cast(t) * kQkvN + row), + "h"(static_cast(0xFFFFu)) + : "memory"); + } + M3FA_PROF_T(28, tid - lane); + } + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + M3FA_PROF_T(25, 512); + if (tid == 512) + { + M3FA_CLK(46); + } +} + +// ---------------- Megakernel o_proj (v52 m3_mx_gemv bit for bit: split 2, contiguous k-tiles, nacc 1) ---------------- +// The unit's weight tiles and scale atoms into the ring (warp 0, lane 0). From L2 when the CTA prefetched them there. +template +__device__ __forceinline__ void op_issue(Params const& p, uint8_t* smem, int ou) +{ + int const rt = ou / kOpSplits; + int const kt0 = (ou % kOpSplits) * kOpTiles; + uint32_t const ob = smem_u32(smem + SX::kOBar); + for (int i = 0; i < kOpTiles; ++i) + { + mbar_expect_tx(ob + 8 * i, kQkvWTile + kQkvSfAtom); + tma_load_2d(smem_u32(smem + SX::kRingW + i * kQkvWTile), &p.wmap_o, (kt0 + i) * 128, rt * 128, ob + 8 * i); + bulk_g2s(smem + SX::kRingSf + i * kQkvSfAtom, + p.wsf_o + (static_cast(rt) * kOpKTiles + kt0 + i) * kQkvSfAtom, kQkvSfAtom, ob + 8 * i); + } +} + +// L2 prefetch of the unit's weight tiles and scale atoms (one lane; nothing waits for it). +// The QKV units' share of the window-1 L2 prefetch (the next consumer's weights), issued once their own QKV weight +// stream has landed (acc_done): the attention core reads little from HBM. +__device__ __forceinline__ void window_prefetch_l2(Params const& p, int u) +{ + constexpr long long kChunk = 32768; + long long const n0 = (p.pf_b0 + kChunk - 1) / kChunk, n1 = (p.pf_b1 + kChunk - 1) / kChunk; + uint64_t pol = 0; + if (p.pf_hint) + { + asm volatile("createpolicy.fractional.L2::evict_last.b64 %0, 1.0;\n" : "=l"(pol)); + } + for (long long c = u; c < n0 + n1; c += kQkvUnits) + { + long long const a = c < n0 ? p.pf_a0 + c * kChunk : p.pf_a1 + (c - n0) * kChunk; + long long const rem = c < n0 ? p.pf_b0 - c * kChunk : p.pf_b1 - (c - n0) * kChunk; + unsigned int const n = static_cast(rem < kChunk ? rem : kChunk) & ~15u; + if (n == 0) + { + continue; + } + if (p.pf_hint) + { + asm volatile("cp.async.bulk.prefetch.L2.global.L2::cache_hint [%0], %1, %2;\n" ::"l"(a), "r"(n), "l"(pol) + : "memory"); + } + else + { + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;\n" ::"l"(a), "r"(n) : "memory"); + } + } +} + +__device__ __forceinline__ void op_prefetch_l2(Params const& p, int ou) +{ + int const rt = ou / kOpSplits; + int const kt0 = (ou % kOpSplits) * kOpTiles; + for (int i = 0; i < kOpTiles; ++i) + { + asm volatile("cp.async.bulk.prefetch.tensor.2d.L2.global [%0, {%1, %2}];\n" ::"l"(&p.wmap_o), + "r"((kt0 + i) * 128), "r"(rt * 128) + : "memory"); + } + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;\n" ::"l"( + p.wsf_o + (static_cast(rt) * kOpKTiles + kt0) * kQkvSfAtom), + "r"(kOpTiles * kQkvSfAtom) + : "memory"); +} + +// The weight tiles into TMEM as they land (warp 1, elected lane, operands warp-uniform). +template +__device__ __forceinline__ void op_stream(uint8_t* smem, uint32_t tm) +{ + uint32_t const ob = smem_u32(smem + SX::kOBar); + uint64_t const w0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + uint32_t const sf0 = smem_u32(smem + SX::kRingSf); +#pragma unroll + for (int i = 0; i < kOpTiles; ++i) + { + mbar_wait_cta(ob + 8 * i, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b(tm + kOA + (i * 4 + kb) * 8, w0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + tmem_cp_sf(tm + kOSfa + i * 4, sf0 + i * kQkvSfAtom); + } + umma_commit(ob + 8 * kOpTiles); // a_ready +} + +// The attention's MXFP8 rows of this unit's k-tiles (the heads [kt0, kt0 + 8)), Lamport-polled value by value: 0xFF +// is NaN in both E4M3 and UE8M0, which the merge never writes (see the merge). Every thread of the CTA, one item each. +template +__device__ __forceinline__ void op_act(Params const& p, uint8_t* smem, int ou, int tid, unsigned int calls) +{ + int const kt0 = (ou % kOpSplits) * kOpTiles; + uint8_t const* q8 = p.q8 + (calls & 1u) * kMaxTok * kOpK; + uint8_t const* sf = p.sf + (calls & 1u) * kOpSfBytes; + constexpr int kXItems = kOpTiles * 8 * kMaxTok; + constexpr int kSfItems = kOpTiles * (kQkvSfAtom / 16); + for (int q = tid; q < kXItems + kSfItems; q += kThreads) + { + if (q < kXItems) + { + int const t = q % kMaxTok; + int const c = (q / kMaxTok) % 8; + int const i = q / (kMaxTok * 8); + if (t < p.num_tokens) + { + uint4 v; + bool pending; + do + { + asm volatile("ld.relaxed.gpu.global.v4.b32 {%0, %1, %2, %3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(q8 + static_cast(t) * kOpK + (kt0 + i) * 128 + c * 16) + : "memory"); + uint32_t const w[4] = {v.x, v.y, v.z, v.w}; + pending = false; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + // a byte equal to 0xFF: (x & 0x7F7F7F7F) + 0x01010101 sets bit 7 of exactly the 0x7F/0xFF bytes + uint32_t const hit = (((w[j] & 0x7F7F7F7Fu) + 0x01010101u) & w[j] & 0x80808080u); + pending |= hit != 0u; + } + } while (pending); + *reinterpret_cast(smem + SX::kOXt + i * 1024 + t * 128 + ((c ^ t) * 16)) = v; + } + } + else if (q < kXItems + kSfItems) + { + int const j = q - kXItems; + int const i = j / (kQkvSfAtom / 16); + int const c = j % (kQkvSfAtom / 16); + uint8_t const* src = sf + (kt0 + i) * kQkvSfAtom + c * 16; + uint4 v; + if (c < p.num_tokens) + { + // Row c of the atom: its 4 block scales in bytes [16 c, 16 c + 4). + do + { + asm volatile("ld.relaxed.gpu.global.v4.b32 {%0, %1, %2, %3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(src) + : "memory"); + } while ((((v.x & 0x7F7F7F7Fu) + 0x01010101u) & v.x & 0x80808080u) != 0u); + } + else + { + v = *reinterpret_cast(src); + } + *reinterpret_cast(smem + SX::kOXsf + i * kQkvSfAtom + c * 16) = v; + } + } +} + +// The MMAs (warp 1, elected lane): v52's order (per k-tile: B scale copy, 4 MMAs; one accumulator). +template +__device__ __forceinline__ void op_mma(uint8_t* smem, uint32_t tm) +{ + uint32_t const ob = smem_u32(smem + SX::kOBar); + mbar_wait_cta(ob + 8 * kOpTiles, 0); // a_ready + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint64_t const b0 = umma_desc_sw128(smem_u32(smem + SX::kOXt)); + uint32_t const xsf0 = smem_u32(smem + SX::kOXsf); +#pragma unroll + for (int i = 0; i < kOpTiles; ++i) + { + uint32_t const sfb = tm + kOSfb + i * 4; + tmem_cp_sf(sfb, xsf0 + i * kQkvSfAtom); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + int const j = i * 4 + kb; + umma_mx(tm + kOAcc, tm + kOA + j * 8, b0 + ((i * 1024 + kb * 32) >> 4), qkv_idesc(kb), tm + kOSfa + i * 4, + sfb, j > 0); + } + } + umma_commit(ob + 8 * (kOpTiles + 1)); // acc_done +} + +// Warps 16..19: the pair's split-K reduce (owner rows [64 s, 64 s + 64)) and the bf16 rows of y. +template +__device__ __forceinline__ void op_epilogue(Params const& p, uint8_t* smem, uint32_t tm, int tid, int ou) +{ + int const warp = tid >> 5; + int const lane = tid & 31; + uint32_t const ob = smem_u32(smem + SX::kOBar); + // The store's kernel parameters, fetched before the wait: a first touch of their constant-bank line costs a miss. + __nv_bfloat16* y_out = p.y_out; + int num_tokens = p.num_tokens; + int ar_mode = p.ar_mode; + asm volatile("" : "+l"(y_out), "+r"(num_tokens), "+r"(ar_mode)); +#ifndef M3FA_OP_PARPRE +#define M3FA_OP_PARPRE 1 // read the all-reduce slot parities before the accumulator wait (a global load off the tail) +#endif + // The parity flags are stable here: the all-reduce that flips them last completed before this grid's PDL wait, + // and the next one reads them only after this kernel's pushes. + int par_pre[kMaxTok]; + { + int const row_pre = (ou / kOpSplits) * 128 + ((tid >> 5) & 3) * 32 + (tid & 31); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + par_pre[t] = 0; + if (M3FA_OP_PARPRE && ar_mode != 0 && t < num_tokens) + { + asm volatile("ld.volatile.global.s32 %0, [%1];\n" + : "=r"(par_pre[t]) + : "l"(p.ar_flags + t * 8 + row_pre / (kOpN / 8)) + : "memory"); + } + } + } + mbar_wait_epi(ob + 8 * (kOpTiles + 1), 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + M3FA_PROF_T(51, kComputeThreads); + int const r = (warp & 3) * 32 + lane; + float v[16]; + tmem_ld16(tm + (((warp & 3) * 32) << 16) + kOAcc, v); + int const s = ou % kOpSplits; + int const owner = r / kOpOwnRows; + int const local = r % kOpOwnRows; + uint32_t const red = ob + 8 * (kOpTiles + 2); + uint32_t cta_rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(cta_rank)); + if (owner != s) + { + uint32_t const peer = cta_rank + owner - s; // o_proj units count up with the CTA index + uint32_t const dst + = mapa_shared(smem_u32(smem + SX::kOMailbox) + ((s * kOpOwnRows + local) * kQkvMbStride) * 4, peer); + uint32_t const pbar = mapa_shared(red, peer); +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + asm volatile( + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [%0], {%1, %2, %3, %4}, [%5];\n" ::"r"( + dst + 16 * g), + "r"(__float_as_uint(v[4 * g])), "r"(__float_as_uint(v[4 * g + 1])), "r"(__float_as_uint(v[4 * g + 2])), + "r"(__float_as_uint(v[4 * g + 3])), "r"(pbar) + : "memory"); + } + } + else + { + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(red) + : "memory"); + } + M3FA_PROF_T(52, tid - lane); + float const* mb = reinterpret_cast(smem + SX::kOMailbox); + int const row = (ou / kOpSplits) * 128 + r; + float sums[kMaxTok]; +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + float4 part[kOpSplits]; +#pragma unroll + for (int q = 0; q < kOpSplits; ++q) + { + part[q] = *reinterpret_cast(mb + (q * kOpOwnRows + local) * kQkvMbStride + 4 * g); + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float sum = 0.0F; +#pragma unroll + for (int q = 0; q < kOpSplits; ++q) + { + float const pj = j == 0 ? part[q].x : j == 1 ? part[q].y : j == 2 ? part[q].z : part[q].w; + sum = sum + (q == s ? v[4 * g + j] : pj); + } + sums[4 * g + j] = sum; + } + } + M3FA_PROF_T(54, tid - lane); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + float const sum = sums[t]; +#ifndef M3FA_EXP +#define M3FA_EXP 0 +#endif + if (t < num_tokens) + { + if (M3FA_EXP == 1) + { + asm volatile( + "st.relaxed.gpu.global.u16 [%0], %1;\n" ::"l"(y_out + static_cast(t) * kOpN + row), + "h"(__bfloat16_as_ushort(__float2bfloat16_rn(sum))) + : "memory"); + } + else + { + y_out[static_cast(t) * kOpN + row] = __float2bfloat16_rn(sum); + } + } + if (ar_mode != 0) + { + // m3_mx_gemv's push into the all-reduce exchange: lane pairs hold row pairs; the even lane writes the + // bf16x2 word into this rank's slot of the (token, column slice) parity the AR's flags select. + float const hi = __shfl_xor_sync(0xffffffffu, sum, 1); + if ((lane & 1) == 0 && t < num_tokens) + { + int par; + if (M3FA_OP_PARPRE) + { + par = par_pre[t]; + } + else + { + asm volatile("ld.volatile.global.s32 %0, [%1];\n" + : "=r"(par) + : "l"(p.ar_flags + t * 8 + row / (kOpN / 8)) + : "memory"); + } + int64_t const off = static_cast(par & 1) * p.ar_maxt * p.ar_world * (kOpN / 2) + + (static_cast(t) * p.ar_world + p.ar_rank) * (kOpN / 2) + row / 2; + __nv_bfloat162 w2 = __floats2bfloat162_rn(sum, hi); + uint32_t word = *reinterpret_cast(&w2); + // -0.0 halves to +0.0: 0x80000000 marks an empty exchange word (as m3_mx_gemv's _bf16x2_word) + word = (word & 0xFFFFu) == 0x8000u ? (word & 0xFFFF0000u) : word; + word = (word >> 16) == 0x8000u ? (word & 0x0000FFFFu) : word; + asm volatile("st.global.b32 [%0], %1;\n" ::"l"(p.ar_ws + off), "r"(word) : "memory"); + } + } + } + M3FA_PROF_T(53, tid - lane); + } + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); +} + +// Packed-QKV element group [token][col * 128 + 4 * lane, +4): the split partials, polled until every one is stored, +// summed in split order and rounded to BF16. +__device__ __forceinline__ void qkv_load(Params const& p, int par, int tok, int col, int lane, float (&e)[4]) +{ + float const* base = p.part + par * kQkvPartStride + static_cast(tok) * kQkvN + col * kHeadDim + lane * 4; + uint4 pr[kQkvSplits]; + long long spins = 0; + bool ready; + do + { + ready = true; +#pragma unroll + for (int s = 0; s < kQkvSplits; ++s) + { + asm volatile("ld.relaxed.gpu.global.v4.b32 {%0, %1, %2, %3}, [%4];\n" + : "=r"(pr[s].x), "=r"(pr[s].y), "=r"(pr[s].z), "=r"(pr[s].w) + : "l"(base + static_cast(s) * kMaxTok * kQkvN) + : "memory"); + ready &= pr[s].x != kQkvSentinel && pr[s].y != kQkvSentinel && pr[s].z != kQkvSentinel + && pr[s].w != kQkvSentinel; + } + if (++spins > (1LL << 26)) + { + __trap(); + } + } while (!ready); + float s0 = __uint_as_float(pr[0].x), s1 = __uint_as_float(pr[0].y), s2 = __uint_as_float(pr[0].z), + s3 = __uint_as_float(pr[0].w); +#pragma unroll + for (int s = 1; s < kQkvSplits; ++s) + { + s0 += __uint_as_float(pr[s].x); + s1 += __uint_as_float(pr[s].y); + s2 += __uint_as_float(pr[s].z); + s3 += __uint_as_float(pr[s].w); + } + e[0] = __bfloat162float(__float2bfloat16_rn(s0)); + e[1] = __bfloat162float(__float2bfloat16_rn(s1)); + e[2] = __bfloat162float(__float2bfloat16_rn(s2)); + e[3] = __bfloat162float(__float2bfloat16_rn(s3)); +} + +// Every CTA once it has read the call parity and the partials it needs: the last one flips the parity. +__device__ __forceinline__ void qkv_done(Params const& p) +{ + if (atomicAdd(p.counters + 1, 1) == kQkvGrid - 1) + { + atomicExch(p.counters + 1, 0); + atomicXor(p.counters, 1); + } +} + +} // namespace + +namespace +{ + +// FQKV: the kernel also computes the QKV(+index) projection from the layer input (see qkv_mma) and runs on a fixed +// kQkvGrid-CTA grid; the first attn_clusters clusters then run the attention on the projection's output. +#ifndef M3FA_C1_OPISSUE +#define M3FA_C1_OPISSUE \ + 4 // c1: o_proj weight TMA issued 0: at entry (before the PDL wait), 1: right after the PDL + // wait, 2: once the attention's K/V reads landed, 3: after the merge, 4: before the push, 5: after it +#endif +#ifndef M3FA_C1_TRIGGER +#define M3FA_C1_TRIGGER 3 // c1: dependents released 2: at the attention's usual point, 3: after the o_proj pushes +#endif +template +__global__ void __launch_bounds__(kThreads, 1) m3_fused_decode_attn_kernel(__grid_constant__ Params const p) +{ +#if MINIMAX_M3_ATTN_DEVICE_ENABLED + static_assert(!OPJ || (!FQKV && MXFP8), "c1: packed input, MXFP8 hand-off"); + using S = std::conditional_t, SmemX>; + constexpr bool kAttnTrigger = !FQKV && !(OPJ && M3FA_C1_TRIGGER == 3); + constexpr int kRows = S::kRows; + constexpr int kTok = S::kTok; + constexpr int kCs = kBlocks / BPC; // cluster size + constexpr int kGroups = QLEN / QG; // clusters per request + constexpr uint32_t kPushBytes = (kHeadDim + 1) * 4; // one partial row: 128 outputs and the LSE + constexpr int kTasks = kHq * QG + 3 * QLEN; // producer tasks: 16 Q heads x QG, [K, V, index-K] x QLEN + constexpr int kMaxT = (kTasks + kWarps - 1) / kWarps; + constexpr int kChunks = BPC * kPage * 4; // 16-byte packed chunks per K (or V) tile set + constexpr int kKPer = (kChunks + kComputeThreads - 1) / kComputeThreads; // K chunks per thread [0, 512) + constexpr bool kRegCore = RegCore::kOn; // warp-MMA attention core (see RegCore) + // V tiles (raw copy before the PDL wait, dequantized during QK and the softmax): on the packed path every warp + // outside the softmax groups takes part (at M=1, warps 4..19), on the fused-projection path the producer warps. +#ifndef M3FA_RS2 +#define M3FA_RS2 1 // megakernel at one 16-row chunk: the softmax and push rows split over two warp groups (8 each) +#endif + // Split softmax: the megakernel and multi-row (spec verify) steps; at one-row decode on the packed path it + // measured slower (it takes four warps from the V dequantization there). + constexpr bool kRs2 = M3FA_RS2 && BPC * QG == 1 && !RegCore::kOn && (FQKV || QLEN > 1); +#ifndef M3FA_MEGA_WARPMMA +#define M3FA_MEGA_WARPMMA 1 +#endif + // Warp-issued MMAs and the early cluster barrier: measured gains in the megakernel and on multi-row / paired-block + // steps, but not at one-row decode on 16-CTA clusters (the packed path's (1, 1, 1)), where they stay off. + constexpr bool kTuned = FQKV || QLEN > 1 || BPC > 1; + constexpr bool kWarpMma = M3FA_MEGA_WARPMMA && kTuned; + constexpr bool kEarlyClw = M3FA_MEGA_EARLYCLW && kTuned; + constexpr int kSoftmaxGroups = kRs2 ? 2 : (BPC * QG) < 4 ? (BPC * QG) : 4; +#ifndef M3FA_MEGA_VWIDE +#define M3FA_MEGA_VWIDE \ + 1 // megakernel attention CTAs (no projection unit): every warp outside the softmax copies and + // dequantizes V, as on the packed path (not only the four producer warps) +#endif + constexpr int kVStart = (kRegCore || (FQKV && !M3FA_MEGA_VWIDE)) ? kComputeThreads : 128 * kSoftmaxGroups; + constexpr int kVThreads = kThreads - kVStart; // V chunks: the warps outside the softmax groups + constexpr int kVPer = (kChunks + kVThreads - 1) / kVThreads; + constexpr int kPairs = BPC * QG; // (block, 16-row chunk) pairs of the softmax and push + constexpr int kPairsPerGroup = (kPairs + 3) / 4; // per group of four compute warps + // Named barrier 6: the softmax warps (P written) and the producer warps (V written) before the PV issue. + constexpr int kPvBarThreads = 128 * kSoftmaxGroups + kVThreads; + extern __shared__ __align__(128) uint8_t smem[]; + float* row_scale = reinterpret_cast(smem + S::kRowScale); + float* row_lse = reinterpret_cast(smem + S::kRowLse); + float* red_m = reinterpret_cast(smem + S::kRedM); + float* red_l = reinterpret_cast(smem + S::kRedL); + float* recv = reinterpret_cast(smem + S::kRecv); + uint32_t const mbar = smem_u32(smem + S::kMbar); + uint32_t const mma_bar = mbar + 8; + uint32_t* tmem_slot = reinterpret_cast(smem + S::kMbar + 16); + + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const cr = blockIdx.x % kCs; // cluster rank; owns blocks [cr * BPC, cr * BPC + BPC) + int const ci = blockIdx.x / kCs; + int const req = ci / kGroups; + int const tfirst = (ci % kGroups) * QG; // this cluster's query tokens [tfirst, tfirst + QG) of the request + int const tok0 = req * QLEN; + uint32_t const gbar = smem_u32(smem + S::kGBar); // FQKV: weight ring, A ready, accumulators done + int const qkv_u = FQKV ? qkv_unit() : -1; // FQKV: this CTA's projection unit, or -1 + + // ---------------- Prologue: nothing here depends on the producer kernel ---------------- +#if M3FA_TRIGGER_AT == 0 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (!FQKV) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + M3FA_PROF(0); +#ifdef M3FA_PROFILE + if (p.prof != nullptr && tid == 0) + { + uint32_t smid; + asm volatile("mov.u32 %0, %%smid;" : "=r"(smid)); + p.prof[blockIdx.x * 64 + 40] = smid; + } +#endif + if constexpr (FQKV) + { + // The projection weights stream into TMEM while the producer (the all-reduce) still runs. + if (warp < 2) + { + qkv_setup(p, smem, gbar, tmem_slot, warp, lane, qkv_u); + } + if (ci >= p.attn_clusters) + { + // A projection-only CTA. Every reduce barrier of the cluster is initialized before any peer pushes. + asm volatile("barrier.cluster.arrive.release.aligned;\nbarrier.cluster.wait.acquire.aligned;\n" :: + : "memory"); + int const op_u = static_cast(blockIdx.x) - kOpFirstCta; // this CTA's o_proj unit (>= 0 here) +#ifndef M3FA_OP_EARLY +#define M3FA_OP_EARLY 0 // 1: o_proj-only CTAs stream their weights from launch (0: after the PDL wait) +#endif + if (M3FA_OP_EARLY && qkv_u < 0 && warp == 0 && lane == 0) + { + op_issue(p, smem, op_u); // an o_proj-only CTA: its weights stream from launch on + } + if (M3FA_OP_L2PF && qkv_u >= 0 && warp == 0 && lane == 0) + { + op_prefetch_l2(p, op_u); // into L2 during the all-reduce, behind this CTA's QKV ring + } + qkv_stream(p, smem, gbar, tmem_slot, warp, lane, qkv_u); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); + if (M3FA_MEGA_TRIGGER == 2) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + // This CTA's call counter: read after the PDL wait (the previous call's CTA increments it at its end), and + // early, so its latency stays off the epilogue's critical path. + unsigned int const calls = p.flags[32 + blockIdx.x]; + M3FA_PROF(1); + if (!M3FA_OP_EARLY && qkv_u < 0 && warp == 0 && lane == 0) + { + op_issue(p, smem, op_u); + } + if (qkv_u >= 0) + { + int const par = 0; + qkv_mma(p, smem, gbar, tmem_slot, tid, qkv_u); + if (warp >= kComputeWarps) + { + qkv_epilogue(p, smem, gbar, tmem_slot, tid, qkv_u, par, calls); + } + if (warp == 0 && lane == 0) + { + // The ring (and the activation over it) is free once the QKV MMAs completed. + mbar_wait_cta(gbar + 16 * kQkvRing + 8, 0); + op_issue(p, smem, op_u); + if (p.pf_b0 > 0 || p.pf_b1 > 0) + { + window_prefetch_l2(p, qkv_u); + } + } + } + __syncthreads(); // the QKV accumulator reads are done: TMEM columns [0, 512) are the o_proj's + M3FA_PROF(2); + uint32_t const tm = *tmem_slot; + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if (elect_one()) + { + op_stream(smem, tm); + } + __syncwarp(); + } + op_act(p, smem, op_u, tid, calls); + asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory"); + M3FA_PROF_T(36, 0); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + // Late release: the dependents launch once every unit has its attention input (the rel_v4 rule). + if (M3FA_MEGA_TRIGGER < 2) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + __syncthreads(); + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if (elect_one()) + { + op_mma(smem, tm); + M3FA_PROF_T(50, threadIdx.x); + } + __syncwarp(); + } + if (warp >= kComputeWarps) + { + op_epilogue(p, smem, tm, tid, op_u); + } + M3FA_PROF_T(37, kComputeThreads); + __syncthreads(); + M3FA_PROF_T(55, 0); + if (M3FA_MEGA_TRIGGER == 3) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); // after this CTA's y rows / pushes + } + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + asm volatile("tcgen05.dealloc.cta_group::1.sync.aligned.b32 %0, %1;\n" ::"r"(*tmem_slot), "r"(kTmemAll) + : "memory"); + M3FA_PROF_T(56, 32); + } + if (tid == 0) + { + p.flags[32 + blockIdx.x] = calls + 1u; // a plain store: no load on the exit path + } + M3FA_PROF(39); +#ifndef M3FA_FINAL_CLBAR +#define M3FA_FINAL_CLBAR 0 +#endif + if (M3FA_FINAL_CLBAR) + { + // No peer may exit while its partials can still be pushed into it (DSMEM lifetime). + asm volatile("barrier.cluster.arrive.release.aligned;\nbarrier.cluster.wait.acquire.aligned;\n" :: + : "memory"); + } + M3FA_PROF(38); + return; + } + } +#ifndef M3FA_MEGA_ATTN_WAIT_FIRST +#define M3FA_MEGA_ATTN_WAIT_FIRST 0 // 1: the attention CTAs read nothing (metadata, caches) before their PDL wait +#endif +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (FQKV && M3FA_MEGA_ATTN_WAIT_FIRST) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + } +#endif + if (!FQKV && warp == kWarps - 1) // a producer warp: it has nothing else to do before the PDL wait + { + asm volatile("tcgen05.alloc.cta_group::1.sync.aligned.shared::cta.b32 [%0], %1;\n" ::"r"(smem_u32(tmem_slot)), + "r"(S::kTmemCols) + : "memory"); + asm volatile("tcgen05.relinquish_alloc_permit.cta_group::1.sync.aligned;\n" ::: "memory"); + } + int const seq_len = p.seq_lens[req]; + int page[BPC]; +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + int const b = cr * BPC + h; + page[h] = b < p.bt_cols ? p.block_table[static_cast(req) * p.bt_stride + b] : 0; + } + // New-token cache slots, for the tokens up to this cluster's last one. + int slot[QLEN]; +#pragma unroll + for (int t = 0; t < QLEN; ++t) + { + slot[t] = t < tfirst + QG ? p.out_cache_loc[tok0 + t] : -1; + } + // This warp's producer tasks and, for every lane that applies RoPE, the task token's cos/sin. + float cs[kMaxT][4], sn[kMaxT][4]; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + int const i = warp + kWarps * k; + int const t = i < kHq * QG ? tfirst + i / kHq : (i - kHq * QG) / 3; + if (i < kTasks && lane < 16 && t < QLEN) + { + int64_t const row = static_cast(p.position_ids[tok0 + t]) * kRotDim; +#pragma unroll + for (int e = 0; e < 4; ++e) + { + int const coefficient = (lane * 4 + e) & (kRotDim / 2 - 1); + cs[k][e] = p.rope[row + coefficient]; + sn[k][e] = p.rope[row + kRotDim / 2 + coefficient]; + } + } + else + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + cs[k][e] = 1.0F; + sn[k][e] = 0.0F; + } + } + } + float qw[4], kw[4], ikw[4]; + load_w4(qw, p.q_w, lane); + load_w4(kw, p.k_w, lane); + load_w4(ikw, p.ik_w, lane); + float const kqs = p.kv_quant_scale[1]; + float const vqs = p.kv_quant_scale[2]; + float const sl2e = (p.sm_scale * 1.4426950409f) * p.k_gs[0]; + float const vgs = p.v_gs[0]; + // Longest attended length among this cluster's tokens, and each block's rows within it. + int const kv_max = max(seq_len - QLEN + tfirst + QG, 0); + int nvh[BPC]; +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + nvh[h] = min(max(kv_max - (cr * BPC + h) * kPage, 0), kPage); + } + bool const any_valid = nvh[0] > 0; + if (tid == 32) + { + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;\n" ::"r"(mbar), "r"(1)); + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;\n" ::"r"(mma_bar), "r"(1)); + asm volatile( + "mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(mbar), "r"(kBlocks * QG * BPC * kPushBytes) + : "memory"); + if constexpr (OPJ) + { + uint32_t const ob = smem_u32(smem + S::kOBar); + for (int i = 0; i < kOpTiles + 3; ++i) + { + mbar_init(ob + 8 * i, 1); + } + mbar_expect_tx(ob + 8 * (kOpTiles + 2), (kOpSplits - 1) * kOpOwnRows * kMaxTok * 4); + } + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + if constexpr (OPJ) + { + asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory"); + if (M3FA_C1_OPISSUE == 0) + { + op_issue(p, smem, static_cast(blockIdx.x)); + } + } + } + // This thread's packed K / V chunks, held in registers until they are dequantized after the PDL wait. + RawChunk kraw[kKPer]; +#pragma unroll + for (int k = 0; k < kKPer; ++k) + { + int const q = tid + kComputeThreads * k; + kraw[k] = RawChunk{make_uint4(0u, 0u, 0u, 0u), 0u}; + int const h = q / (kPage * 4); + if (q < kChunks && tid < kComputeThreads && nvh[h] > 0) + { + kraw[k] = load_chunk(p.kv_data + static_cast(page[h]) * p.d_page, + p.kv_scale + static_cast(page[h]) * p.s_page, chunk_row(q % (kPage * 4)), + chunk_col(q % (kPage * 4)), false); + } + } + // V goes through cp.async into shared memory: an asynchronous copy does not hold the producer warps up the way + // an outstanding register load does, and they need V only after QK has been issued. + if (tid >= kVStart) + { +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + if (nvh[h] > 0) + { + uint8_t* dst = smem + S::kRawV + h * S::kRawVBlock; + uint8_t const* data = p.kv_data + static_cast(page[h]) * p.d_page + p.d_plane; + uint8_t const* scales = p.kv_scale + static_cast(page[h]) * p.s_page + p.s_plane; + for (int c = tid - kVStart; c < S::kRawVBlock / 16; c += kVThreads) + { + cp_async16(dst + c * 16, c < kPage * 4 ? data + c * 16 : scales + (c - kPage * 4) * 16); + } + } + } + } + asm volatile("cp.async.commit_group;\n" ::: "memory"); + if (warp == kWarps - 1) + { + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + // Every CTA's mbarrier is initialized before any peer pushes into it; the matching wait sits right before the + // push, so this barrier's latency hides behind the attention (with aliased partials it moves there). + if constexpr (!S::kRecvAlias) + { + asm volatile("barrier.cluster.arrive.relaxed.aligned;\n" ::: "memory"); + } + int kv_len[QG]; +#pragma unroll + for (int tl = 0; tl < QG; ++tl) + { + kv_len[tl] = max(seq_len - QLEN + tfirst + tl + 1, 0); + } + int const cta_tok0 = cr * kTok; // request position of this CTA's first block row + // New token t's row in this CTA's tiles (block * 128 + row), or -1: the CTA owns the block holding the token's + // position, and the token is attended by this cluster (t <= its last token). Every thread derives the same. + int new_row[QLEN]; +#pragma unroll + for (int t = 0; t < QLEN; ++t) + { + int const pos = seq_len - QLEN + t; + int const blk = min(pos >> 7, kBlocks - 1); + int const spage = slot[t] / kPage; + bool const mine = t < tfirst + QG && pos >= 0 && blk / BPC == cr && slot[t] >= 0 && spage < p.num_pages; + new_row[t] = mine && spage == page[blk % BPC] ? (blk % BPC) * kPage + (slot[t] & (kPage - 1)) : -1; + } + auto is_new_row = [&](int r) + { + bool hit = false; +#pragma unroll + for (int t = 0; t < QLEN; ++t) + { + hit |= new_row[t] == r; + } + return hit; + }; + // Cache writes of the new K/V/index-K are issued after this CTA has pushed its partials: only the fp16 tile rows + // are on the attention's path. kind 0: none, 1: NVFP4 data + scale, 2: FP8 index-K. + int wr_kind[kMaxT]; + uint8_t* wr_ptr[kMaxT]; + uint8_t* wr_sptr[kMaxT]; + uint32_t wr_val[kMaxT]; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + wr_kind[k] = 0; + wr_ptr[k] = nullptr; + wr_sptr[k] = nullptr; + wr_val[k] = 0u; + } + uint32_t tmem = 0; + auto dequant_k = [&]() + { + if (tid < kComputeThreads) + { +#pragma unroll + for (int k = 0; k < kKPer; ++k) + { + int const q = tid + kComputeThreads * k; + if (q < kChunks) + { + int const h = q / (kPage * 4); + int const row = chunk_row(q % (kPage * 4)); + if (!is_new_row(h * kPage + row)) + { + store_chunk_fp16( + smem + S::kK16 + h * S::kTileBytes, row, chunk_col(q % (kPage * 4)), kraw[k], row < nvh[h]); + } + } + } + } + }; + // The producer warps' V tiles (raw tiles from cp.async, dequantized to fp16), then their PV-barrier arrival. + auto dequant_v = [&]() + { +#ifdef M3FA_DBG_NO_VDQ + if constexpr (!kRegCore) + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + return; +#endif + asm volatile("cp.async.wait_all;\n" ::: "memory"); + asm volatile("bar.sync 4, %0;\n" ::"n"(kVThreads) : "memory"); +#pragma unroll + for (int k = 0; k < kVPer; ++k) + { + int const q = tid - kVStart + kVThreads * k; + if (q < kChunks) + { + int const h = q / (kPage * 4); + int const row = chunk_row(q % (kPage * 4)); + int const cc = chunk_col(q % (kPage * 4)); + if (!is_new_row(h * kPage + row)) + { + uint8_t const* rawv = smem + S::kRawV + h * S::kRawVBlock; + uint8_t const* sc = rawv + kPage * 64 + (row >> 2) * 32 + (row & 3); + RawChunk c; + c.data = *reinterpret_cast(rawv + row * 64 + cc * 16); + c.scales + = static_cast(sc[(cc * 2) * 4]) | (static_cast(sc[(cc * 2 + 1) * 4]) << 8); + store_chunk_fp16(smem + S::kV16 + h * S::kTileBytes, row, cc, c, row < nvh[h]); + } + } + } + M3FA_FENCE_PROXY_ASYNC(); // V tiles -> tensor core reads (ordered by the barrier before the PV issue) + if constexpr (!kRegCore) + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + M3FA_PROF_T(16, kComputeThreads); + M3FA_PROF_T(17, kThreads - 1); + }; + if constexpr (FQKV) + { + qkv_stream(p, smem, gbar, tmem_slot, warp, lane, qkv_u); + } + if constexpr (!FQKV) + { + if (M3FA_V15_EARLYV && any_valid && tid >= kVStart) + { + dequant_v(); + } + if constexpr (kEarlyClw && !S::kRecvAlias) + { + asm volatile("barrier.cluster.wait.aligned;\n" ::: "memory"); + } + } + M3FA_PROF(11); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); + if (FQKV && M3FA_MEGA_TRIGGER == 2) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#if M3FA_TRIGGER_AT == 1 + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); +#endif +#endif + // FQKV (megakernel): this call's QKV rows sit in buffer (calls & 1), polled value by value below. + unsigned int const mega_calls = (FQKV || OPJ) ? p.flags[32 + blockIdx.x] : 0u; + if (OPJ && M3FA_C1_OPISSUE == 1 && tid == 32) + { + op_issue(p, smem, static_cast(blockIdx.x)); + } + __nv_bfloat16 const* qkv_src = FQKV ? p.packed + (mega_calls & 1u) * kMaxTok * kQkvN : p.packed; + M3FA_PROF(1); + int const par = 0; + if constexpr (FQKV) + { + if (qkv_u >= 0) + { + qkv_mma(p, smem, gbar, tmem_slot, tid, qkv_u); + } + } + + if constexpr (FQKV) + { + if (qkv_u >= 0) + { + if (warp >= kComputeWarps) + { + qkv_epilogue(p, smem, gbar, tmem_slot, tid, qkv_u, par, mega_calls); + } + else + { + // The attention regions overlay the weight ring: every weight byte has gone on into TMEM (or, with + // the MMAs reading the ring, every MMA has completed) first. + mbar_wait_cta(gbar + 16 * kQkvRing + (kQkvASmem ? 8 : 0), 0); + } + } + if (any_valid) + { + dequant_k(); + } +#ifndef M3FA_MEGA_EARLYV +#define M3FA_MEGA_EARLYV 1 // attention CTAs dequantize V here, while the QKV rows are still on their way +#endif +#ifndef M3FA_MEGA_EARLYCLW +#define M3FA_MEGA_EARLYCLW 1 // and pass the cluster barrier (peers' merge mbarriers initialized) here too +#endif + if (M3FA_MEGA_EARLYV && any_valid && tid >= kVStart) + { + dequant_v(); + } + if constexpr (kEarlyClw && !S::kRecvAlias) + { + asm volatile("barrier.cluster.wait.aligned;\n" ::: "memory"); + } + M3FA_PROF(14); + } + + if (any_valid) // CTA-uniform + { + // ---------------- Producer ---------------- + // Q rows of this cluster's tokens in every CTA. The new K/V of every token up to this cluster's last one + // whose position falls in this CTA's blocks go straight into the fp16 tiles; the CTA writes the caches, and + // the index-K, only for its own tokens. + uint2 raw[kMaxT]; + float fq[FQKV ? kMaxT : 1][4]; + bool act[kMaxT]; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + int const i = warp + kWarps * k; + raw[k] = make_uint2(0u, 0u); + act[k] = false; + if (i < kTasks) + { + int t, col; + bool active = true; + if (i >= kHq * QG) + { + int const kv = i - kHq * QG; + t = kv / 3; + int const kind = kv % 3; + col = kind == 0 ? kColK : (kind == 1 ? kColV : kColIdxK); + int const pos = seq_len - QLEN + t; + active = t < tfirst + QG && pos >= 0 && min(pos >> 7, kBlocks - 1) / BPC == cr && slot[t] >= 0 + && slot[t] / kPage < p.num_pages && (kind != 2 || t >= tfirst); + } + else + { + t = tfirst + i / kHq; + col = i % kHq; + } + act[k] = active; + if (active) + { + uint2 const* src = reinterpret_cast( + qkv_src + static_cast(tok0 + t) * p.packed_stride + col * kHeadDim + lane * 4); + if constexpr (FQKV) + { + uint2 v; + bool pending; + do + { + asm volatile("ld.relaxed.gpu.global.v2.b32 {%0, %1}, [%2];\n" + : "=r"(v.x), "=r"(v.y) + : "l"(src) + : "memory"); + pending = (v.x & 0xFFFFu) == 0xFFFFu || (v.x >> 16) == 0xFFFFu || (v.y & 0xFFFFu) == 0xFFFFu + || (v.y >> 16) == 0xFFFFu; + } while (pending); + raw[k] = v; + } + else + { + raw[k] = *src; + } + } + } + } + M3FA_PROF(15); + // While the packed-QKV loads are in flight: dequantize this thread's K chunks to fp16 (exact: e2m1 x e4m3 + // fits fp16). Rows past the longest attended length are zeroed, as the Triton producer masks its loads; the + // new tokens' rows are written by their producer tasks instead. (Building the tiles before the PDL wait was + // measured slower: the cache loads issued at entry are still in flight there, so the wait moves later.) + if constexpr (!FQKV) + { + dequant_k(); + } +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + int const i = warp + kWarps * k; + if (i >= kTasks || !act[k]) + { + continue; + } + float e[4]; + { + float2 const a = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw[k].x)); + float2 const b = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw[k].y)); + e[0] = a.x; + e[1] = a.y; + e[2] = b.x; + e[3] = b.y; + } + if (i < kHq * QG) + { + norm_rope(e, qw, p.eps, cs[k], sn[k], lane); + __nv_fp8x2_e4m3 const lo(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const hi(make_float2(e[2], e[3])); + uint2 hv; + hv.x = e4m3x2_to_h2(lo.__x); + hv.y = e4m3x2_to_h2(hi.__x); + *reinterpret_cast(smem + S::kQ + kmaj_off(i, lane * 4)) = hv; // row i = token * 16 + head + continue; + } + int const t = (i - kHq * QG) / 3; + int const kind = (i - kHq * QG) % 3; + int const spage = slot[t] / kPage; + int const within = slot[t] & (kPage - 1); + if (kind == 2) + { + norm_rope(e, ikw, p.eps, cs[k], sn[k], lane); + __nv_fp8x2_e4m3 const lo(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const hi(make_float2(e[2], e[3])); + wr_kind[k] = 2; + wr_val[k] = static_cast(lo.__x) | (static_cast(hi.__x) << 16); + wr_ptr[k] = p.idx_k + static_cast(spage) * p.i_page + static_cast(within) * p.i_tok + + lane * 4; + continue; + } + bool const isV = kind == 1; + if (!isV) + { + norm_rope(e, kw, p.eps, cs[k], sn[k], lane); + } + uint8_t sbyte = 0; + uint16_t const fp4 = quantizeMinimaxM3Fp4x4(e, isV ? vqs : kqs, &sbyte); + if (t >= tfirst) + { + int const scale_col = lane / 4; + int const scale_within = isV ? (within / 4) * 32 + scale_col * 4 + within % 4 : within * 8 + scale_col; + wr_kind[k] = 1; + wr_val[k] = static_cast(fp4) | (static_cast(sbyte) << 16); + wr_ptr[k] = p.kv_data + static_cast(spage) * p.d_page + (isV ? p.d_plane : 0) + within * 64 + + lane * 2; + wr_sptr[k] = p.kv_scale + static_cast(spage) * p.s_page + (isV ? p.s_plane : 0) + scale_within; + } + if (new_row[t] >= 0) + { + // The attention reads this page: its fp16 tile row is the freshly quantized row, dequantized like + // the rest of the tile. + int const h = new_row[t] / kPage; + int const row = new_row[t] % kPage; + uint32_t const sc2 = e4m3_to_h2_splat(sbyte); + uint2 hv; + hv.x = hmul2_u(e2m1x2_to_h2(fp4 & 0xFFu), sc2); + hv.y = hmul2_u(e2m1x2_to_h2(fp4 >> 8), sc2); + *reinterpret_cast( + smem + (isV ? S::kV16 : S::kK16) + h * S::kTileBytes + kmaj_off(row, lane * 4)) + = hv; + } + } + // K tiles, Q rows and patched rows -> tensor core reads. Only threads that wrote fence: the fence also + // waits for the thread's outstanding loads, and the producer warps still have their V chunks in flight. + bool wrote = tid < kComputeThreads; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + wrote |= act[k]; + } + if (wrote) + { + M3FA_FENCE_PROXY_ASYNC(); + } + M3FA_PROF(13); + __syncthreads(); +#if M3FA_TRIGGER_AT == 2 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + // Every K/V byte this CTA reads has arrived (the V chunks sit in registers): release the dependents now so + // their weight streams do not queue ahead of the KV reads (the megakernel releases late, from its o_proj). + if constexpr (kAttnTrigger || (FQKV && M3FA_MEGA_TRIGGER == 1)) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + if (OPJ && M3FA_C1_OPISSUE == 2 && tid == 32) + { + op_issue(p, smem, static_cast(blockIdx.x)); + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + tmem = *tmem_slot; + M3FA_PROF(2); + // ---------------- S^T = K Q^T per block on the tensor core: TMEM lane = token, column = query row ---------- +#ifndef M3FA_MEGA_WARPMMA +#define M3FA_MEGA_WARPMMA 1 // attention MMAs issued by warp 0's elected lane with warp-uniform operands (not tid 0) +#endif + if (!kRegCore && (kWarpMma ? warp == 0 : tid == 0)) + { + // Operands computed by the whole warp stay in uniform registers; one elected lane issues. + uint32_t const tm_u = tmem; + bool const issue = kWarpMma ? elect_one() : true; + if (issue) + { +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + uint32_t const a0 = smem_u32(smem + S::kK16 + h * S::kTileBytes); + uint32_t const b0 = smem_u32(smem + S::kQ); +#pragma unroll + for (int ks = 0; ks < kHeadDim / 16; ++ks) + { + umma_f16(tm_u + h * kRows, umma_desc(a0 + ks * 256, 128, 2048), + umma_desc(b0 + ks * 256, 128, 2048), umma_idesc(kRows, 0, 0), ks > 0); + } + } + umma_commit(mma_bar); + } + if (kWarpMma) + { + __syncwarp(); + } + } + if (tid >= kVStart && !(FQKV && M3FA_MEGA_EARLYV) && !(!FQKV && M3FA_V15_EARLYV)) + { + // The producer warps dequantize V while QK and the softmax run. + dequant_v(); + } + if constexpr (kRegCore) + { + // ---------------- Register core: warp w of block h = w / kWpb owns 8 * kNt tokens for S and 8 * kNt + // output dims for O; m16n8k16 fp16 MMAs with fp32 accumulators, the softmax in the S fragments ---------- + constexpr int kWpb = kComputeWarps / BPC; // warps per block + constexpr int kNt = 16 / kWpb; // 8-token (S) or 8-dim (O) tiles per warp + float* redx_m = reinterpret_cast(smem + S::kRedX); + float* redx_l = redx_m + BPC * kRows * kWpb; + if (warp < kComputeWarps) + { + int const h = warp / kWpb; + int const wib = warp % kWpb; + int const g = lane >> 2; + int const q4 = lane & 3; + uint32_t const ktile = smem_u32(smem + S::kK16 + h * S::kTileBytes); + uint32_t const vtile = smem_u32(smem + S::kV16 + h * S::kTileBytes); + uint32_t const qrows = smem_u32(smem + S::kQ); + uint8_t* ptile = smem + S::kK16 + h * S::kTileBytes; // P over this block's K tile, after QK + auto block_bar = [&]() + { + if (h == 0) + { + asm volatile("bar.sync 14, %0;\n" ::"n"(kWpb * 32) : "memory"); + } + else + { + asm volatile("bar.sync 15, %0;\n" ::"n"(kWpb * 32) : "memory"); + } + }; + // S = Q K^T: rows (c, g / g + 8), tokens 8 tb + 2 q4 (+1). + float sv[QG][kNt][4]; +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + sv[c][nt][e] = 0.0F; + } + } + } +#pragma unroll + for (int ks = 0; ks < kHeadDim / 16; ++ks) + { + uint32_t bk[kNt][2]; +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + int const tb = wib * kNt + nt; + ldsm_x2(ktile + tb * 2048 + (ks * 2 + ((lane >> 3) & 1)) * 128 + (lane & 7) * 16, bk[nt]); + } +#pragma unroll + for (int c = 0; c < QG; ++c) + { + uint32_t aq[4]; + ldsm_x4( + qrows + (c * 2 + ((lane >> 3) & 1)) * 2048 + (ks * 2 + (lane >> 4)) * 128 + (lane & 7) * 16, + aq); +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + mma_f16_16816(sv[c][nt], aq, bk[nt]); + } + } + } + M3FA_PROF(7); + // Row maxima (log2 domain, masked as the producer masks), quad then across the block's warps. + float mrow[QG][2]; +#pragma unroll + for (int c = 0; c < QG; ++c) + { + float mx0 = kNegInf, mx1 = kNegInf; +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + int const tok = cta_tok0 + h * kPage + (wib * kNt + nt) * 8 + 2 * q4 + (e & 1); + bool const valid = tok < kv_max && tok < kv_len[c]; + float const v = valid ? sv[c][nt][e] * sl2e : kNegInf; + sv[c][nt][e] = v; + if (e < 2) + { + mx0 = fmaxf(mx0, v); + } + else + { + mx1 = fmaxf(mx1, v); + } + } + } + mx0 = fmaxf(mx0, __shfl_xor_sync(0xffffffffu, mx0, 1)); + mx0 = fmaxf(mx0, __shfl_xor_sync(0xffffffffu, mx0, 2)); + mx1 = fmaxf(mx1, __shfl_xor_sync(0xffffffffu, mx1, 1)); + mx1 = fmaxf(mx1, __shfl_xor_sync(0xffffffffu, mx1, 2)); + if (q4 == 0) + { + redx_m[(h * kRows + c * 16 + g) * kWpb + wib] = mx0; + redx_m[(h * kRows + c * 16 + g + 8) * kWpb + wib] = mx1; + } + } + block_bar(); // every warp of the block is past QK (P may overwrite the K tile) and wrote its maxima + M3FA_PROF(8); +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int half = 0; half < 2; ++half) + { + float const* rm = redx_m + (h * kRows + c * 16 + g + 8 * half) * kWpb; + float m = kNegInf; +#pragma unroll + for (int w4 = 0; w4 < kWpb; w4 += 4) + { + float4 const v = *reinterpret_cast(rm + w4); + m = fmaxf(m, fmaxf(fmaxf(v.x, v.y), fmaxf(v.z, v.w))); + } + mrow[c][half] = m; + } + // P = exp2(s - m) as fp16 into [row][token] core matrices; the fp32 values feed the row sums. + float l0 = 0.0F, l1 = 0.0F; +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + float a[4]; +#pragma unroll + for (int e = 0; e < 4; ++e) + { + float const m = mrow[c][e >> 1]; + a[e] = m == kNegInf ? 0.0F : ex2(sv[c][nt][e] - m); + } + l0 += a[0] + a[1]; + l1 += a[2] + a[3]; + int const t0 = (wib * kNt + nt) * 8 + 2 * q4; + int const r0 = c * 16 + g; + __half2 const p0 = __floats2half2_rn(a[0], a[1]); + __half2 const p1 = __floats2half2_rn(a[2], a[3]); + *reinterpret_cast<__half2*>( + ptile + (r0 >> 3) * 2048 + (t0 >> 3) * 128 + (r0 & 7) * 16 + (t0 & 7) * 2) + = p0; + *reinterpret_cast<__half2*>( + ptile + ((r0 + 8) >> 3) * 2048 + (t0 >> 3) * 128 + (r0 & 7) * 16 + (t0 & 7) * 2) + = p1; + } + l0 += __shfl_xor_sync(0xffffffffu, l0, 1); + l0 += __shfl_xor_sync(0xffffffffu, l0, 2); + l1 += __shfl_xor_sync(0xffffffffu, l1, 1); + l1 += __shfl_xor_sync(0xffffffffu, l1, 2); + if (q4 == 0) + { + redx_l[(h * kRows + c * 16 + g) * kWpb + wib] = l0; + redx_l[(h * kRows + c * 16 + g + 8) * kWpb + wib] = l1; + } + } + M3FA_PROF_T(19, 0); + } + __syncthreads(); // P, the V tiles (producer warps) and the row sums visible + M3FA_PROF(9); + if (warp < kComputeWarps) + { + int const h = warp / kWpb; + int const wib = warp % kWpb; + int const g = lane >> 2; + int const q4 = lane & 3; + uint32_t const vtile = smem_u32(smem + S::kV16 + h * S::kTileBytes); + uint32_t const ptile = smem_u32(smem + S::kK16 + h * S::kTileBytes); + float* obuf = reinterpret_cast(smem + S::kK16 + h * S::kTileBytes + S::kObufOff); + // O = P V: rows (c, g / g + 8), dims 8 dt + 2 q4 (+1). + float ov[QG][kNt][4]; +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + ov[c][nt][e] = 0.0F; + } + } + } +#pragma unroll + for (int ks = 0; ks < kPage / 16; ++ks) + { + uint32_t bv[kNt][2]; + int const tok = ks * 16 + ((lane >> 3) & 1) * 8 + (lane & 7); +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + int const dt = wib * kNt + nt; + ldsm_x2_trans(vtile + (tok >> 3) * 2048 + dt * 128 + (tok & 7) * 16, bv[nt]); + } +#pragma unroll + for (int c = 0; c < QG; ++c) + { + uint32_t ap[4]; + ldsm_x4( + ptile + (c * 2 + ((lane >> 3) & 1)) * 2048 + (ks * 2 + (lane >> 4)) * 128 + (lane & 7) * 16, + ap); +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + mma_f16_16816(ov[c][nt], ap, bv[nt]); + } + } + } +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + int const d = (wib * kNt + nt) * 8 + 2 * q4; + int const r0 = c * 16 + g; + *reinterpret_cast(obuf + r0 * kOPitch + d) = make_float2(ov[c][nt][0], ov[c][nt][1]); + *reinterpret_cast(obuf + (r0 + 8) * kOPitch + d) + = make_float2(ov[c][nt][2], ov[c][nt][3]); + } + } + } + else if (tid - kComputeThreads < BPC * kRows) + { + // Row statistics of every (block, row): LSE and the PV scale, as the producer derives them. + int const r = tid - kComputeThreads; + float const* rm = redx_m + r * kWpb; + float const* rl = redx_l + r * kWpb; + float m = kNegInf; + float l = 0.0F; +#pragma unroll + for (int w4 = 0; w4 < kWpb; w4 += 4) + { + float4 const vm = *reinterpret_cast(rm + w4); + float4 const vl = *reinterpret_cast(rl + w4); + m = fmaxf(m, fmaxf(fmaxf(vm.x, vm.y), fmaxf(vm.z, vm.w))); + l += (vl.x + vl.y) + (vl.z + vl.w); + } + float const lse = l > 0.0F ? m + lg2(l) : kNegInf; + float const scale = lse > kNegInf ? ex2(m - lse) : 0.0F; + row_scale[r] = scale * vgs; + row_lse[r] = lse; + } + __syncthreads(); // O partials and row statistics visible to the push + } + else + { + if constexpr (kRs2) + { + // ---------------- Split softmax: warps 0..3 rows 0..7, warps 4..7 rows 8..15 (thread = block token) + // -------- + if (warp < 8) + { + int const rg = warp / 4; + int const wq = warp % 4; + int const tk = wq * 32 + lane; + auto rg_bar = [&]() + { + if (rg == 0) + { + asm volatile("bar.sync 10, 128;\n" ::: "memory"); + } + else + { + asm volatile("bar.sync 11, 128;\n" ::: "memory"); + } + }; + int const rs8 = ((lane >> 4) & 1) * 4 + ((lane >> 3) & 1) * 2 + ((lane >> 2) & 1); + mbar_wait_cta(mma_bar, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + M3FA_PROF(7); + float v[8]; + tmem_ld8(tmem + ((wq * 32) << 16) + 8 * rg, v); + bool const valid = cta_tok0 + tk < kv_max && cta_tok0 + tk < kv_len[0]; + float a[8]; +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + v[j2] = valid ? v[j2] * sl2e : kNegInf; + a[j2] = v[j2]; + } + float const mw = warp_reduce_scatter8(a, lane); + if ((lane & 3) == 0) + { + red_m[(8 * rg + rs8) * 4 + wq] = mw; + } + rg_bar(); + M3FA_PROF(8); + uint8_t* pdst = smem + S::kP + (tk >> 3) * 128 + (tk & 7) * 16; + float mrow = kNegInf; + if (lane < 8) + { + float4 const mq = *reinterpret_cast(red_m + (8 * rg + lane) * 4); + mrow = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + } + float pa[8]; +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + float const m = __shfl_sync(0xffffffffu, mrow, j2); + pa[j2] = m == kNegInf ? 0.0F : ex2(v[j2] - m); + } + if (tid == 0) + { + M3FA_CLK(41); + } + uint32_t pw[4]; +#pragma unroll + for (int j2 = 0; j2 < 4; ++j2) + { + __half2 const hp = __floats2half2_rn(pa[2 * j2], pa[2 * j2 + 1]); + pw[j2] = *reinterpret_cast(&hp); + } + *reinterpret_cast(pdst + rg * 2048) = make_uint4(pw[0], pw[1], pw[2], pw[3]); + M3FA_PROF_T(18, 0); + if (tid == 0) + { + M3FA_CLK(42); + } + M3FA_FENCE_PROXY_ASYNC(); // P -> tensor core reads + if (tid == 0) + { + M3FA_CLK(43); + } + if (warp == 0) + { + asm volatile("bar.sync 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + if (tid == 0) + { + M3FA_CLK(44); + } + } + else + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + if (kWarpMma ? warp == 0 : tid == 0) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm_u = tmem; + uint32_t const a0 = smem_u32(smem + S::kV16); + uint32_t const b0 = smem_u32(smem + S::kP); + if (kWarpMma ? elect_one() : true) + { +#pragma unroll + for (int ks = 0; ks < kPage / 16; ++ks) + { + umma_f16(tm_u + BPC * kRows, umma_desc(a0 + ks * 4096, 2048, 128), + umma_desc(b0 + ks * 256, 128, 2048), umma_idesc(kRows, 1, 1), ks > 0); + } + umma_commit(mma_bar); + } + if (kWarpMma) + { + __syncwarp(); + } + } + M3FA_PROF_T(19, 0); + float const lw = warp_reduce_scatter8(pa, lane); + if ((lane & 3) == 0) + { + red_l[(8 * rg + rs8) * 4 + wq] = lw; + } + rg_bar(); + M3FA_PROF_T(20, 0); + if (wq == 0 && lane < 8) + { + int const r = 8 * rg + lane; + float4 const mq = *reinterpret_cast(red_m + r * 4); + float4 const lq = *reinterpret_cast(red_l + r * 4); + float const m = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + float const l = (lq.x + lq.y) + (lq.z + lq.w); + float const lse = l > 0.0F ? m + lg2(l) : kNegInf; + float const scale = lse > kNegInf ? ex2(m - lse) : 0.0F; + row_scale[r] = scale * vgs; + row_lse[r] = lse; + } + } + __syncthreads(); // row scales and LSEs visible (PV was issued above) + M3FA_PROF(9); + if (warp < 8) + { + mbar_wait_cta(mma_bar, 1); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + } + } + else + { + // ---------------- Softmax: thread = block token; the (block, 16-row chunk) pairs are spread over the + // four groups of four compute warps (each group covers the 128 TMEM lanes = block tokens) + // ---------------- + if (warp < kComputeWarps && warp / 4 < kPairs) + { + int const g = warp / 4; + int const wq = warp % 4; + int const tk = wq * 32 + lane; + auto group_bar = [&]() + { + if (g == 0) + { + asm volatile("bar.sync 10, 128;\n" ::: "memory"); + } + else if (g == 1) + { + asm volatile("bar.sync 11, 128;\n" ::: "memory"); + } + else if (g == 2) + { + asm volatile("bar.sync 12, 128;\n" ::: "memory"); + } + else + { + asm volatile("bar.sync 13, 128;\n" ::: "memory"); + } + }; + // Row r(lane) this lane holds after the reduce-scatter: bits 4..1 of the lane. + int const rs_row + = ((lane >> 4) & 1) * 8 + ((lane >> 3) & 1) * 4 + ((lane >> 2) & 1) * 2 + ((lane >> 1) & 1); + mbar_wait_cta(mma_bar, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + M3FA_PROF(7); + float v[kPairsPerGroup][16]; +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + tmem_ld16(tmem + ((wq * 32) << 16) + h * kRows + c * 16, v[k]); + bool const valid + = cta_tok0 + h * kPage + tk < kv_max && cta_tok0 + h * kPage + tk < kv_len[c]; + float a[16]; +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[k][j] = valid ? v[k][j] * sl2e : kNegInf; + a[j] = v[k][j]; + } + float const mw = warp_reduce_scatter16(a, lane); + if ((lane & 1) == 0) + { + red_m[((h * QG + c) * 16 + rs_row) * 4 + wq] = mw; + } + } + } + group_bar(); + M3FA_PROF(8); + float pa[kPairsPerGroup][16]; +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + uint8_t* pdst = smem + S::kP + h * kRows * kPage * 2 + (tk >> 3) * 128 + (tk & 7) * 16; + // Row maxima: lane j (< 16) reduces row j's four warp partials, then broadcasts it. + float mrow = kNegInf; + if (lane < 16) + { + float4 const mq + = *reinterpret_cast(red_m + ((h * QG + c) * 16 + lane) * 4); + mrow = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + } +#pragma unroll + for (int j = 0; j < 16; ++j) + { + float const m = __shfl_sync(0xffffffffu, mrow, j); + pa[k][j] = m == kNegInf ? 0.0F : ex2(v[k][j] - m); + } + if (tid == 0) + { + M3FA_CLK(41); + } + uint32_t pw[8]; +#pragma unroll + for (int j = 0; j < 8; ++j) + { + __half2 const hp = __floats2half2_rn(pa[k][2 * j], pa[k][2 * j + 1]); + pw[j] = *reinterpret_cast(&hp); + } + // P^T as the MN-major B operand: 8 query rows of one token are one 16-byte core-matrix row. + *reinterpret_cast(pdst + (2 * c) * 2048) = make_uint4(pw[0], pw[1], pw[2], pw[3]); + *reinterpret_cast(pdst + (2 * c + 1) * 2048) + = make_uint4(pw[4], pw[5], pw[6], pw[7]); + M3FA_PROF_T(18, 0); + } + } + if (tid == 0) + { + M3FA_CLK(42); + } + M3FA_FENCE_PROXY_ASYNC(); // P -> tensor core reads + if (tid == 0) + { + M3FA_CLK(43); + } + // PV starts as soon as P and the V tiles are in: thread 0 waits for the softmax warps and the + // producer warps (V); the row sums and statistics below overlap the MMAs. + if (warp == 0) + { + asm volatile("bar.sync 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + else + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + if (kWarpMma ? warp == 0 : tid == 0) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm_u = tmem; + if (kWarpMma ? elect_one() : true) + { +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + uint32_t const a0 = smem_u32(smem + S::kV16 + h * S::kTileBytes); + uint32_t const b0 = smem_u32(smem + S::kP + h * kRows * kPage * 2); +#pragma unroll + for (int ks = 0; ks < kPage / 16; ++ks) + { + umma_f16(tm_u + (BPC + h) * kRows, umma_desc(a0 + ks * 4096, 2048, 128), + umma_desc(b0 + ks * 256, 128, 2048), umma_idesc(kRows, 1, 1), ks > 0); + } + } + umma_commit(mma_bar); + } + if (kWarpMma) + { + __syncwarp(); + } + } + M3FA_PROF_T(19, 0); +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + float const lw = warp_reduce_scatter16(pa[k], lane); + if ((lane & 1) == 0) + { + red_l[((h * QG + c) * 16 + rs_row) * 4 + wq] = lw; + } + } + } + group_bar(); + M3FA_PROF_T(20, 0); + if (wq == 0 && lane < 16) + { +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + int const r = (h * QG + c) * 16 + lane; + float4 const mq = *reinterpret_cast(red_m + r * 4); + float4 const lq = *reinterpret_cast(red_l + r * 4); + float const m = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + float const l = (lq.x + lq.y) + (lq.z + lq.w); + float const lse = l > 0.0F ? m + lg2(l) : kNegInf; + float const scale = lse > kNegInf ? ex2(m - lse) : 0.0F; + row_scale[h * kRows + c * 16 + lane] = scale * vgs; + row_lse[h * kRows + c * 16 + lane] = lse; + } + } + } + } + __syncthreads(); // row scales and LSEs visible (PV was issued above) + M3FA_PROF(9); + if (warp < kComputeWarps && warp / 4 < kPairs) + { + mbar_wait_cta(mma_bar, 1); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + } + } + } + M3FA_PROF(3); + } + else + { + asm volatile("cp.async.wait_all;\n" ::: "memory"); + if (OPJ && M3FA_C1_OPISSUE == 2 && tid == 32) + { + op_issue(p, smem, static_cast(blockIdx.x)); + } +#if M3FA_TRIGGER_AT == 2 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (kAttnTrigger || (FQKV && M3FA_MEGA_TRIGGER == 1)) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + for (int r = tid; r < BPC * kRows; r += kThreads) + { + row_lse[r] = kNegInf; + row_scale[r] = 0.0F; + } + __syncthreads(); + } + + // ---------------- Push every partial row to the CTA that merges its head (CTA r merges heads r*BPC..) -------- + if constexpr (S::kRecvAlias) + { + // Peers' partials overlay this CTA's tiles: every CTA's PV MMAs have completed. + asm volatile("barrier.cluster.arrive.aligned;\n" ::: "memory"); + } + if (!(kEarlyClw && !S::kRecvAlias)) + { + asm volatile("barrier.cluster.wait.aligned;\n" ::: "memory"); // peers' mbarriers are initialized + } + M3FA_PROF(4); + if (OPJ && M3FA_C1_OPISSUE == 4 && tid == 32) + { + op_issue(p, smem, static_cast(blockIdx.x)); + } + uint32_t const recv_base = smem_u32(recv); + if (kRs2 && warp < 8) + { + // Split push: warp group rg pushes rows 8 rg .. 8 rg + 7 (row j goes to CTA j, which merges head j). + int const rg = warp / 4; + int const d = (warp % 4) * 32 + lane; + float v[8]; + if (any_valid) + { + tmem_ld8(tmem + ((warp % 4) * 32 << 16) + BPC * kRows + 8 * rg, v); + } + else + { +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + v[j2] = 0.0F; + } + } +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + v[j2] *= row_scale[8 * rg + j2]; + } +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + uint32_t const dst_cta = 8 * rg + j2; + uint32_t const row_addr = recv_base + static_cast((cr * kOPitch + d) * 4); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_shared(row_addr, dst_cta)), + "r"(__float_as_uint(v[j2])), "r"(mapa_shared(mbar, dst_cta)) + : "memory"); + } + } + else if (!kRs2 && warp < kComputeWarps && warp / 4 < kPairs) + { + int const g = warp / 4; + int const d = (warp % 4) * 32 + lane; // output dim +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + int const chunk = cr * BPC + h; + float v[16]; + if (any_valid && kRegCore) + { + float const* obuf + = reinterpret_cast(smem + S::kK16 + h * S::kTileBytes + S::kObufOff); +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[j] = obuf[(c * 16 + j) * kOPitch + d]; + } + } + else if (any_valid) + { + tmem_ld16(tmem + ((warp % 4) * 32 << 16) + (BPC + h) * kRows + c * 16, v); + } + else + { +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[j] = 0.0F; + } + } +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[j] *= row_scale[h * kRows + c * 16 + j]; + } + // One scalar store per row: a vector store's lanes would address several destination CTAs at once, + // which measured several times slower. +#pragma unroll + for (int j = 0; j < 16; ++j) + { + uint32_t const dst_cta = j / BPC; + uint32_t const row_addr + = recv_base + static_cast((((chunk * QG + c) * BPC + j % BPC) * kOPitch + d) * 4); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_shared(row_addr, dst_cta)), + "r"(__float_as_uint(v[j])), "r"(mapa_shared(mbar, dst_cta)) + : "memory"); + } + } + } + } + else if (tid >= kComputeThreads) + { + // Every partial row's LSE (BPC * kRows of them; more than the producer threads for the widest geometries). + for (int r = tid - kComputeThreads; r < BPC * kRows; r += kThreads - kComputeThreads) + { + int const h = r / kRows; + int const c = (r % kRows) / 16; + int const j = r % 16; + uint32_t const dst_cta = j / BPC; + uint32_t const row_addr + = recv_base + static_cast((((cr * BPC + h) * QG + c) * BPC + j % BPC) * kOPitch * 4); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_shared(row_addr + kHeadDim * 4, dst_cta)), + "r"(__float_as_uint(row_lse[r])), "r"(mapa_shared(mbar, dst_cta)) + : "memory"); + } + } + M3FA_PROF(10); + if (OPJ && M3FA_C1_OPISSUE == 5 && tid == 32) + { + op_issue(p, smem, static_cast(blockIdx.x)); + } +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + if (wr_kind[k] == 1) + { + *reinterpret_cast(wr_ptr[k]) = static_cast(wr_val[k] & 0xFFFFu); + if (lane % 4 == 0) + { + *wr_sptr[k] = static_cast(wr_val[k] >> 16); + } + } + else if (wr_kind[k] == 2) + { + *reinterpret_cast(wr_ptr[k]) = wr_val[k]; + } + } + { + uint32_t ready = 0; +#ifdef M3FA_DBG_HANG + long long spins = 0; +#endif + while (!ready) + { +#ifdef M3FA_DBG_HANG + if (++spins == (1LL << 24)) + { + printf("M3FA hang: block %d thread %d merge recv\n", blockIdx.x, threadIdx.x); + } +#endif + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(mbar) + : "memory"); + } + } + M3FA_PROF(5); + + // ---------------- Merge heads [cr * BPC, cr * BPC + BPC) (split-K merge kernel's LSE weighting) ---------------- + constexpr int kMergeRows = QG * BPC; // (token, local head) pairs, 16 chunk weights each + // Wide geometries: chunk weights once per merged row (into the softmax scratch, free by now), then the weighted + // sums. Narrow ones recompute the weights per output element: no extra barrier on the critical path. + constexpr bool kRowWeights = kMergeRows >= 4; + float* wmerge = red_m; // [kMergeRows][kBlocks]; red_m holds BPC * kRows * 4 = kMergeRows * 64 floats + [[maybe_unused]] auto chunk_weights = [&](int mr, float(&lc)[kBlocks]) + { + float lmax = kNegInf; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + lc[c] = recv[(c * kMergeRows + mr) * kOPitch + kHeadDim]; + lmax = fmaxf(lmax, lc[c]); + } + lmax = lmax == kNegInf ? 0.0F : lmax; + float denom = 0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + lc[c] = ex2(lc[c] - lmax); // now the chunk weight; 0 for an empty chunk + denom += lc[c]; + } + denom = denom > 0.0F ? denom : 1.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + // -0.0 marks an empty chunk (skipped below, as the merge kernel skips zero weights). + lc[c] = lc[c] != 0.0F ? __fdividef(lc[c], denom) : -0.0F; + } + }; + if constexpr (kRowWeights) + { + for (int mr = tid; mr < kMergeRows; mr += kThreads) + { + float lc[kBlocks]; + chunk_weights(mr, lc); +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + wmerge[mr * kBlocks + c] = lc[c]; + } + } + __syncthreads(); + } +#pragma unroll + for (int idx = tid; idx < kMergeRows * kHeadDim; idx += kThreads) + { + int const mr = idx >> 7; + int const d = idx & (kHeadDim - 1); + float wn[kBlocks]; + if constexpr (kRowWeights) + { +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + wn[c] = wmerge[mr * kBlocks + c]; + } + } + else + { +#ifndef M3FA_MEGA_MERGE_LOCAL +#define M3FA_MEGA_MERGE_LOCAL \ + 0 // 1 (measured +0.6 us slower): every merging thread derives the 16 chunk weights itself (broadcast shared loads, + // independent exps): no shuffle chain; the same arithmetic as chunk_weights +#endif + if constexpr (M3FA_MEGA_MERGE_LOCAL != 0) + { + chunk_weights(mr, wn); + } + else + { + // Narrow geometries: the warp (32 dims of one merged row) derives the 16 chunk weights together, lane c + // holding chunk c's, and broadcasts them. + float lc = (lane < kBlocks) ? recv[(lane * kMergeRows + mr) * kOPitch + kHeadDim] : kNegInf; + float lmax = lc; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + lmax = fmaxf(lmax, __shfl_xor_sync(0xffffffffu, lmax, o)); + } + lmax = lmax == kNegInf ? 0.0F : lmax; + float const wc = lane < kBlocks ? ex2(lc - lmax) : 0.0F; + // The denominator in chunk order from 0.0, as chunk_weights (and the merge kernel) add it: a butterfly + // reduction rounds differently in the last bit. + float denom = 0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + denom += __shfl_sync(0xffffffffu, wc, c); + } + denom = denom > 0.0F ? denom : 1.0F; + float const wl = wc != 0.0F ? __fdividef(wc, denom) : -0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + wn[c] = __shfl_sync(0xffffffffu, wl, c); + } + } + } + if (tid == 0) + { + M3FA_CLK(45); + } + float acc = 0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + float const prod = recv[(c * kMergeRows + mr) * kOPitch + d] * wn[c]; + acc = __float_as_uint(wn[c]) != 0x80000000u ? acc + prod : acc; + } + if (tid == 0) + { + M3FA_CLK(46); + } + int const tl = mr / BPC; + int const head = cr * BPC + mr % BPC; + __nv_bfloat16 const ob = __float2bfloat16_rn(acc); + int const out_row = tok0 + tfirst + tl; + p.out[static_cast(out_row) * p.out_stride + head * kHeadDim + d] = ob; + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + { + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, mask)); + } + float const sfv = amax * rcp_approx_ftz(448.0F); + int const bits = __float_as_int(sfv); + int ex = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0 ? 1 : 0); + ex = min(ex, 254); + float scale = __int_as_float((254 - ex) << 23); + scale = amax != 0.0F ? scale : 0.0F; + __nv_fp8_e4m3 const q8v(x * scale); + int64_t const q8off = static_cast(out_row) * (p.num_sf_k_tiles * 128) + head * kHeadDim + d; + if constexpr (FQKV || OPJ) + { + // Megakernel hand-off to the o_proj units (Lamport): this call's parity buffer, the other one reset to + // 0xFF. 0xFF is never written: a NaN code is canonicalized to 0x7F; the UE8M0 code is at most 254. + uint8_t const code = q8v.__x == 0xFFu ? static_cast(0x7Fu) : q8v.__x; + int64_t const cur = (mega_calls & 1u) * kMaxTok * kOpK; + int64_t const nxt = ((mega_calls & 1u) ^ 1u) * kMaxTok * kOpK; + asm volatile( + "st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.q8 + cur + q8off), "h"(static_cast(code)) + : "memory"); + asm volatile( + "st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.q8 + nxt + q8off), "h"(static_cast(0xFFu)) + : "memory"); + } + else + { + p.q8[q8off] = q8v.__x; + } + if (lane == 0) + { + int const k_sf = head * (kHeadDim / 32) + (d >> 5); + int64_t const off = static_cast(out_row / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (out_row % 32) * 16 + ((out_row % 128) / 32) * 4 + k_sf % 4; + if constexpr (FQKV || OPJ) + { + int64_t const cur = (mega_calls & 1u) * kOpSfBytes; + int64_t const nxt = ((mega_calls & 1u) ^ 1u) * kOpSfBytes; + asm volatile( + "st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.sf + cur + off), "h"(static_cast(ex)) + : "memory"); + asm volatile("st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.sf + nxt + off), + "h"(static_cast(0xFFu)) + : "memory"); + } + else + { + p.sf[off] = static_cast(ex); + } + } + } + } + M3FA_PROF(60); + if constexpr (OPJ) + { + // ---------------- c1: o_proj unit blockIdx.x (v52 m3_mx_gemv bit for bit, mega_v7's phase) ---------------- + if (any_valid) + { + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + __syncthreads(); // the merge's reads done; the K tiles are free for the activation overlay + int const op_u = static_cast(blockIdx.x); + if (M3FA_C1_OPISSUE == 3 && tid == 32) + { + op_issue(p, smem, op_u); + } + uint32_t const tm = *tmem_slot; + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if (elect_one()) + { + op_stream(smem, tm); + } + __syncwarp(); + } + op_act(p, smem, op_u, tid, mega_calls); + asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory"); + M3FA_PROF_T(36, 0); + __syncthreads(); + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if (elect_one()) + { + op_mma(smem, tm); + M3FA_PROF_T(50, threadIdx.x); + } + __syncwarp(); + } + if (warp >= kComputeWarps) + { + op_epilogue(p, smem, tm, tid, op_u); + } + M3FA_PROF_T(37, kComputeThreads); + __syncthreads(); +#if M3FA_C1_TRIGGER == 3 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); // after this CTA's y rows / pushes +#endif + if (warp == kWarps - 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + asm volatile("tcgen05.dealloc.cta_group::1.sync.aligned.b32 %0, %1;\n" ::"r"(tm), "r"(S::kTmemCols) + : "memory"); + } + if (tid == 0) + { + p.flags[32 + blockIdx.x] = mega_calls + 1u; + } + M3FA_PROF(6); + return; + } + if (any_valid) + { + // Every TMEM read completed above (tcgen05.wait::ld); release the columns. + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + __syncthreads(); + if (FQKV && tid == 0) + { + p.flags[32 + blockIdx.x] = mega_calls + 1u; + } + M3FA_PROF(61); + if (warp == kWarps - 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + asm volatile("tcgen05.dealloc.cta_group::1.sync.aligned.b32 %0, %1;\n" ::"r"(*tmem_slot), "r"(S::kTmemCols) + : "memory"); + } + M3FA_PROF(6); +#else + __trap(); +#endif // MINIMAX_M3_ATTN_DEVICE_ENABLED +} + +template +cudaError_t launch_impl(Params const& prm, int num_reqs, bool pdl, cudaStream_t stream, int* max_clusters) +{ + auto kernel = m3_fused_decode_attn_kernel; + constexpr int smem = std::conditional_t, SmemX>::kBytes; + constexpr int kCs = kBlocks / BPC; + static bool const configured = [&]() + { + bool ok = cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, smem) == cudaSuccess; + ok = ok + && (kCs <= 8 + || cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1) == cudaSuccess); + if (!ok) + { + cudaGetLastError(); + } + return ok; + }(); + if (!configured) + { + return cudaErrorInvalidConfiguration; + } + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(FQKV ? kQkvGrid : kCs * (QLEN / QG) * num_reqs); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeClusterDimension; + attr[0].val.clusterDim.x = kCs; + attr[0].val.clusterDim.y = 1; + attr[0].val.clusterDim.z = 1; + attr[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[1].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + if (max_clusters != nullptr) + { + cfg.numAttrs = 1; + cudaError_t const err = cudaOccupancyMaxActiveClusters(max_clusters, kernel, &cfg); + if (err != cudaSuccess) + { + cudaGetLastError(); + } + return err; + } + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +// The one geometry the target launches: one request verifying 6 query tokens, 6 clusters of 16 CTAs (96 o_proj +// units). +cudaError_t dispatch(Params const& prm, int qlen, int qg, int bpc, int num_reqs, bool pdl, cudaStream_t stream) +{ + if (qlen == 6 && qg == 1 && bpc == 1) + { + return launch_impl<6, 1, 1, true, false, true>(prm, num_reqs, pdl, stream, nullptr); + } + return cudaErrorInvalidValue; +} + +static_assert(kMaxTok == minimax_m3_attn_c1::kMaxTokens && kOpN == minimax_m3_attn_c1::kOpN + && kOpK == minimax_m3_attn_c1::kOpK && kOpSfBytes == minimax_m3_attn_c1::kOpSfBytes + && kOpN / 128 * kOpSplits == minimax_m3_attn_c1::kCtas, + "header dimensions"); + +} // namespace + +namespace +{ +// TMA descriptor of a [n, k] E4M3 weight (128 x 128 boxes, 128-byte swizzle), cached per weight address. +bool weight_map(CUtensorMap& map, void const* w, int k, int n) +{ + static std::mutex mu; + static std::unordered_map cache; + std::lock_guard lock(mu); + auto it = cache.find(w); + if (it != cache.end()) + { + map = it->second; + return true; + } + static PFN_cuTensorMapEncodeTiled_v12000 encode = []() + { + void* fn = nullptr; + cudaDriverEntryPointQueryResult q; + cudaGetDriverEntryPointByVersion("cuTensorMapEncodeTiled", &fn, 12000, cudaEnableDefault, &q); + return reinterpret_cast(fn); + }(); + if (encode == nullptr) + { + return false; + } + cuuint64_t dims[2] = {static_cast(k), static_cast(n)}; + cuuint64_t strides[1] = {static_cast(k)}; + cuuint32_t box[2] = {128, 128}; + cuuint32_t elem[2] = {1, 1}; + CUresult const r = encode(&map, CU_TENSOR_MAP_DATA_TYPE_UINT8, 2, const_cast(w), dims, strides, box, elem, + CU_TENSOR_MAP_INTERLEAVE_NONE, CU_TENSOR_MAP_SWIZZLE_128B, CU_TENSOR_MAP_L2_PROMOTION_L2_256B, + CU_TENSOR_MAP_FLOAT_OOB_FILL_NONE); + if (r != CUDA_SUCCESS) + { + return false; + } + cache.emplace(w, map); + return true; +} +} // namespace + +cudaError_t invokeMinimaxM3AttnC1(MinimaxM3AttnC1Params& params, void const* oWeight, int qlen, int qg, int bpc, + int numReqs, bool pdl, cudaStream_t stream) +{ + if (!weight_map(params.wmap_o, oWeight, kOpK, kOpN)) + { + return cudaErrorInvalidValue; + } + return dispatch(params, qlen, qg, bpc, numReqs, pdl, stream); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/attnC1.h b/cpp/tensorrt_llm/kernels/minimaxM3/attnC1.h new file mode 100644 index 000000000000..f13de72e14fb --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/attnC1.h @@ -0,0 +1,110 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace minimax_m3_attn_c1 +{ +constexpr int kCtas = 96; // attention CTAs == o_proj units (48 row tiles x 2 K splits) +constexpr int kMaxTokens = 8; // rows of each call-parity buffer of the MXFP8 hand-off +constexpr int kOpN = 6144; // o_proj output width +constexpr int kOpK = 2048; // o_proj input width (16 heads x 128) +constexpr int kOpSfBytes = 16 * 512; // one parity of the MXFP8 hand-off's swizzled scales +} // namespace minimax_m3_attn_c1 + +//! Kernel parameters of the attention + o_proj (c1) kernel (the device code's Params). +struct MinimaxM3AttnC1Params +{ + __nv_bfloat16 const* packed; + int64_t packed_stride; + uint8_t* kv_data; + int64_t d_page, d_plane; + uint8_t* kv_scale; + int64_t s_page, s_plane; + uint8_t* idx_k; + int64_t i_page, i_tok; + int64_t num_pages; + int const* out_cache_loc; + int const* position_ids; + int const* block_table; + int64_t bt_stride; + int bt_cols; + int const* seq_lens; + float const* kv_quant_scale; + float const* k_gs; + float const* v_gs; + __nv_bfloat16 const* q_w; + __nv_bfloat16 const* k_w; + __nv_bfloat16 const* ik_w; + float const* rope; + float eps; + float sm_scale; + __nv_bfloat16* out; + int64_t out_stride; + uint8_t* q8; + uint8_t* sf; + int num_sf_k_tiles; + unsigned long long* prof; // optional per-CTA clock64 probes (-DM3FA_PROFILE builds only) + // Fused QKV projection (FQKV kernels only): out[M, 2560] = mxfp8(x)[M, 6144] @ W^T. + __nv_bfloat16 const* x; // [M, 6144] layer input (the all-reduce output) + int ldx; + int num_tokens; // M + uint8_t const* wsf; // UE8M0 weight scales, 128x4-swizzled + float* part; // [2 parities][splits][8 tokens][2560] partial sums, sentinel at rest + int* counters; // [0]: call parity, [1]: CTAs done with this call (zero at rest) + int attn_clusters; // leading clusters that run the attention + uint8_t const* xq; // mega: the layer input's MXFP8 values [M, 6144] (row stride ldxq) ... + uint8_t const* xsf_act; // ... and its 128x4-swizzled UE8M0 scale atoms (atom k at byte 512 k) + int ldxq; + __nv_bfloat16* qkv_out; // mega: the packed QKV rows [2 parities][8][2560] the attention reads + uint8_t const* wsf_o; // mega: the o_proj weight's scale atoms + __nv_bfloat16* y_out; // mega: the o_proj output [M, 6144] + int* ar_ws; // mega, ar_mode: the next m3_mc_allreduce's multicast exchange (int32 words)... + int const* ar_flags; // ... its call flags (parity of a (token, column slice) slot = flags[t * 8 + slice] & 1) + int ar_rank; + int ar_world; + int ar_maxt; + int ar_mode; // 0: none; 2: push the o_proj rows into the exchange (as m3_mx_gemv's ar mode 2) + unsigned int* flags; // mega: [0, 20) tile counters (4 per call), [32 + CTA] the attention CTAs' targets + long long pf_a0, pf_b0, pf_a1, pf_b1; // mega: L2 prefetch regions issued in the attention window (bytes 0: none) + int pf_hint; // 1: evict_last policy on the prefetch + uint8_t const* wpre; // mega: optional pre-tiled QKV weight [N/128][K/128][16 KB swizzled smem image], or null + CUtensorMap wmap; // the [2560, 6144] E4M3 weight, 128 x 128 boxes, 128-byte swizzle + CUtensorMap wmap_o; // mega: the [6144, 2048] E4M3 o_proj weight, same boxes +}; + +//! Launch the c1 kernel on `stream`. The caller fills `params` except `wmap_o`, the o_proj weight's TMA +//! descriptor (cached per address). Only (qlen, qg, bpc) = (6, 1, 1) with one request is compiled; anything else, or +//! a failed TMA descriptor encode, returns cudaErrorInvalidValue. +cudaError_t invokeMinimaxM3AttnC1(MinimaxM3AttnC1Params& params, void const* oWeight, int qlen, int qg, int bpc, + int numReqs, bool pdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/attnMega.cu b/cpp/tensorrt_llm/kernels/minimaxM3/attnMega.cu new file mode 100644 index 000000000000..a4c8dd923f6f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/attnMega.cu @@ -0,0 +1,3673 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3 attention megakernel: QKV GEMV + fused decode attention + o_proj GEMV in one launch. + +#include "tensorrt_llm/kernels/minimaxM3/attnMega.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +// The kernel uses tcgen05/TMEM (including block-scaled MMAs, which family targets lack), cluster DSMEM and E2M1 +// conversions: its body is compiled for the arch-specific sm_100a / sm_103a targets only (see the minimax_m3_attn_src +// library in kernels/CMakeLists.txt). Any other target compiles a kernel that traps, so a launch on a build without +// those targets fails loudly instead of returning untouched outputs. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ == 1000 || __CUDA_ARCH__ == 1030) \ + && (defined(__CUDA_ARCH_SPECIFIC__) || defined(__CUDA_ARCH_FEAT_SM100_ALL) || defined(__CUDA_ARCH_FEAT_SM103_ALL)) +#define MINIMAX_M3_ATTN_DEVICE_ENABLED 1 +#else +#define MINIMAX_M3_ATTN_DEVICE_ENABLED 0 +#endif + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kHeadDim = 128; +constexpr int kRotDim = 64; +constexpr int kPage = 128; +constexpr int kHq = 16; +constexpr int kColK = kHq; // packed head slots: [Q x16 | K | V | idxQ | idxK] +constexpr int kColV = kHq + 1; +constexpr int kColIdxK = kHq + 3; +constexpr int kBlocks = 16; // KV blocks per request == topk == cluster size +constexpr int kComputeWarps = 16; // QK / softmax / PV and the dequant (one 16-byte chunk per thread) +constexpr int kWarps = kComputeWarps + 4; // plus producer-only warps, so new K/V/index-K never queue behind a Q head +constexpr int kThreads = kWarps * 32; +constexpr int kComputeThreads = kComputeWarps * 32; +constexpr int kOPitch = 132; // floats per partial-O row +constexpr float kNegInf = -__builtin_huge_valf(); + +using Params = MinimaxM3AttnMegaParams; + +// Optional per-CTA clock64 probes for development (-DM3FA_PROFILE and a prof tensor): [CTA][64] timestamps. +// -DM3FA_PROFILE_GT records %globaltimer (ns, comparable across SMs and kernels) instead of clock64. +#ifdef M3FA_PROFILE_GT +#define M3FA_PROFILE +#define M3FA_NOW(v) asm volatile("mov.u64 %0, %%globaltimer;" : "=l"(v)) +#else +#define M3FA_NOW(v) v = clock64() +#endif +#ifdef M3FA_PROFILE +#define M3FA_PROF_T(i, thread) \ + do \ + { \ + if (p.prof != nullptr && threadIdx.x == (thread)) \ + { \ + unsigned long long now_; \ + M3FA_NOW(now_); \ + p.prof[blockIdx.x * 64 + (i)] = now_; \ + } \ + } while (0) +#else +#define M3FA_PROF_T(i, thread) \ + do \ + { \ + } while (0) +#endif +#define M3FA_PROF(i) M3FA_PROF_T(i, 0) +// clock64 probes (development): slot i of this CTA, written by every calling thread. +#ifdef M3FA_PROFILE +#define M3FA_CLK(i) \ + do \ + { \ + if (p.prof != nullptr) \ + { \ + p.prof[blockIdx.x * 64 + (i)] = clock64(); \ + } \ + } while (0) +#else +#define M3FA_CLK(i) \ + do \ + { \ + } while (0) +#endif + +#ifndef M3FA_MEGA_TRIGGER +// Megakernel PDL release: 0 o_proj CTAs after their activation (attention CTAs at exit); 1 as 0 with the attention +// CTAs after their KV reads; 2 every CTA right after its griddepcontrol.wait; 3 o_proj CTAs after their epilogue. +#define M3FA_MEGA_TRIGGER 3 +#endif +#ifndef M3FA_TRIGGER_AT +#define M3FA_TRIGGER_AT 2 // where the PDL dependents are released: 0 entry, 1 after the wait, 2 after the KV reads +#endif + +// Generic-proxy shared-memory writes (fp16 tiles, Q, P) become visible to the tensor core (async proxy). +#define M3FA_FENCE_PROXY_ASYNC() asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory") + +__device__ __forceinline__ void cp_async16(void* dst, void const* src) +{ + uint32_t const d = static_cast(__cvta_generic_to_shared(dst)); + asm volatile("cp.async.cg.shared.global [%0], [%1], 16;\n" ::"r"(d), "l"(src) : "memory"); +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +// One packed E2M1 byte -> half2 (low nibble in the low half). +__device__ __forceinline__ uint32_t e2m1x2_to_h2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 src;\n mov.b32 {src, _, _, _}, %1;\n cvt.rn.f16x2.e2m1x2 %0, src;\n}\n" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t hmul2_u(uint32_t a, uint32_t b) +{ + __half2 r = __hmul2(*reinterpret_cast<__half2*>(&a), *reinterpret_cast<__half2*>(&b)); + return *reinterpret_cast(&r); +} + +__device__ __forceinline__ uint32_t e4m3_to_h2_splat(uint32_t byte) +{ + __half_raw const h = __nv_cvt_fp8_to_halfraw(static_cast<__nv_fp8_storage_t>(byte), __NV_E4M3); + return static_cast(h.x) | (static_cast(h.x) << 16); +} + +__device__ __forceinline__ uint32_t e4m3x2_to_h2(uint32_t two_bytes) +{ + __half2_raw const h = __nv_cvt_fp8x2_to_halfraw2(static_cast<__nv_fp8x2_storage_t>(two_bytes), __NV_E4M3); + return static_cast(h.x) | (static_cast(h.y) << 16); +} + +// ---- Exact copies of the C++ producer's helpers (fusedQKNormRopeKernel.cu / quantization.cuh) ---- +template +__device__ __forceinline__ uint32_t cvt_sf_group_shfl_mask() +{ + constexpr uint32_t groupSize = static_cast(NUM_THREADS_PER_SF); + constexpr uint32_t groupMask = 0xFFFFFFFFU >> (32U - groupSize); + uint32_t laneId = 0; + asm("mov.u32 %0, %%laneid;" : "=r"(laneId)); + return groupMask << (laneId & ~(groupSize - 1U)); +} + +__device__ __forceinline__ uint32_t fp32_vec_to_e2m1(float (&array)[8]) +{ + uint32_t val; + asm volatile( + "{\n" + ".reg .b8 byte0;\n" + ".reg .b8 byte1;\n" + ".reg .b8 byte2;\n" + ".reg .b8 byte3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte0, %2, %1;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte1, %4, %3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte2, %6, %5;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte3, %8, %7;\n" + "mov.b32 %0, {byte0, byte1, byte2, byte3};\n" + "}" + : "=r"(val) + : "f"(array[0]), "f"(array[1]), "f"(array[2]), "f"(array[3]), "f"(array[4]), "f"(array[5]), "f"(array[6]), + "f"(array[7])); + return val; +} + +__device__ __forceinline__ uint16_t quantizeMinimaxM3Fp4x4( + float const (&elements)[4], float secondLevelScale, uint8_t* scaleOut) +{ + __nv_bfloat16 rounded[4]; + float localMax = 0.0F; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + rounded[i] = __float2bfloat16_rn(elements[i]); + localMax = fmaxf(localMax, fabsf(__bfloat162float(rounded[i]))); + } + uint32_t const groupMask = cvt_sf_group_shfl_mask<4>(); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 1)); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 2)); + float const scaleValue = secondLevelScale * (localMax * rcp_approx_ftz(6.0F)); + __nv_fp8_e4m3 const encodedScale(scaleValue); + if (scaleOut != nullptr) + { + *scaleOut = encodedScale.__x; + } + float const narrowedScale = static_cast(encodedScale); + float const outputScale + = localMax != 0.0F ? rcp_approx_ftz(narrowedScale * rcp_approx_ftz(secondLevelScale)) : 0.0F; + float quantValues[8] = {}; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + quantValues[i] = __bfloat162float(rounded[i]) * outputScale; + } + return static_cast(fp32_vec_to_e2m1(quantValues)); +} + +// RMSNorm (gemma 1+w) + NeoX RoPE over the first 64 dims, exactly as the producer: lane owns 4 dims. +__device__ __forceinline__ void norm_rope( + float (&e)[4], float const (&w)[4], float eps, float const (&cs)[4], float const (&sn)[4], int lane) +{ + float ss = 0.0F; + ss += e[0] * e[0] + e[1] * e[1]; + ss += e[2] * e[2] + e[3] * e[3]; +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + { + ss = ss + __shfl_xor_sync(0xffffffffu, ss, mask, 32); + } + float const rmsReciprocal = rsqrtf(ss / static_cast(kHeadDim) + eps); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + e[i] *= rmsReciprocal * (1.0F + w[i]); + } + __syncwarp(); + constexpr int kPairOffset = (kRotDim / 2) / 4; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const dim = lane * 4 + i; + float paired = __shfl_xor_sync(0xffffffffu, e[i], kPairOffset); + if (dim < kRotDim) + { + bool const firstHalf = dim < kRotDim / 2; + if (firstHalf) + { + paired = -paired; + } + e[i] = e[i] * cs[i] + paired * sn[i]; + } + } + __syncwarp(); +} + +__device__ __forceinline__ void load_w4(float (&w)[4], __nv_bfloat16 const* src, int lane) +{ + uint2 const raw = *reinterpret_cast(src + lane * 4); + float2 const a = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw.x)); + float2 const b = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw.y)); + w[0] = a.x; + w[1] = a.y; + w[2] = b.x; + w[3] = b.y; +} + +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t mapa_shared(uint32_t addr, uint32_t rank) +{ + uint32_t r; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;\n" : "=r"(r) : "r"(addr), "r"(rank)); + return r; +} + +// QLEN query tokens per request; QG of them per cluster (QLEN / QG clusters per request); BPC KV blocks per CTA +// (cluster of kBlocks / BPC CTAs). +// Byte offset of element (r, k) in a UMMA no-swizzle K-major operand tile of 128 k-elements per row: 8x8 core +// matrices of 16-byte rows, 128 bytes apart along k (LBO) and 2048 bytes apart per 8 rows (SBO). The same formula +// with r = token, k = dim stores V so that it is the MN-major (dims contiguous) A operand of O^T = V^T P^T. +__device__ __forceinline__ int kmaj_off(int r, int k) +{ + return (r >> 3) * 2048 + (k >> 3) * 128 + (r & 7) * 16 + (k & 7) * 2; +} + +__device__ __forceinline__ uint64_t umma_desc(uint32_t addr, uint32_t lbo, uint32_t sbo) +{ + return static_cast((addr >> 4) & 0x3FFFu) | (static_cast((lbo >> 4) & 0x3FFFu) << 16) + | (static_cast((sbo >> 4) & 0x3FFFu) << 32) | (1ull << 46); +} + +// kind::f16 instruction descriptor: F32 accumulate, F16 operands, B K-major, M = 128. +__host__ __device__ constexpr uint32_t umma_idesc(int n, int a_mn_major, int b_mn_major) +{ + return (1u << 4) | (static_cast(a_mn_major) << 15) | (static_cast(b_mn_major) << 16) + | (static_cast(n >> 3) << 17) | (static_cast(128 >> 4) << 24); +} + +// 8-row version for the split softmax: the same lane-bit order (16, 8, 4, then 2 and 1 in full) as +// warp_reduce_scatter16, so every row is combined in the same association (fp add is commutative, not associative). +// After it lanes with bits 4..2 = (b4, b3, b2) hold row 4*b4 + 2*b3 + b2. +template +__device__ __forceinline__ float warp_reduce_scatter8(float (&a)[8], int lane) +{ +#pragma unroll + for (int step = 0; step < 3; ++step) + { + int const mask = 16 >> step; + int const half = 4 >> step; + bool const upper = (lane & mask) != 0; +#pragma unroll + for (int i = 0; i < half; ++i) + { + float const send = upper ? a[i] : a[i + half]; + float const keep = upper ? a[i + half] : a[i]; + float const recv = __shfl_xor_sync(0xffffffffu, send, mask); + a[i] = kMax ? fmaxf(keep, recv) : keep + recv; + } + } + float x = a[0]; + float const o2 = __shfl_xor_sync(0xffffffffu, x, 2); + x = kMax ? fmaxf(x, o2) : x + o2; + float const o1 = __shfl_xor_sync(0xffffffffu, x, 1); + return kMax ? fmaxf(x, o1) : x + o1; +} + +// Reduce 16 per-lane values over the warp, one value per row: after the reduce-scatter lane l holds row +// 8*b4 + 4*b3 + 2*b2 + b1 (b_k = bit k of l) reduced over all 32 lanes. 16 shuffles instead of 80. +template +__device__ __forceinline__ float warp_reduce_scatter16(float (&a)[16], int lane) +{ +#pragma unroll + for (int step = 0; step < 4; ++step) + { + int const mask = 16 >> step; + int const half = 8 >> step; + bool const upper = (lane & mask) != 0; +#pragma unroll + for (int i = 0; i < half; ++i) + { + float const send = upper ? a[i] : a[i + half]; + float const keep = upper ? a[i + half] : a[i]; + float const recv = __shfl_xor_sync(0xffffffffu, send, mask); + a[i] = kMax ? fmaxf(keep, recv) : keep + recv; + } + } + float const other = __shfl_xor_sync(0xffffffffu, a[0], 1); + return kMax ? fmaxf(a[0], other) : a[0] + other; +} + +__device__ __forceinline__ void umma_f16( + uint32_t tmem_d, uint64_t adesc, uint64_t bdesc, uint32_t idesc, int accumulate) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, %4, 0;\n tcgen05.mma.cta_group::1.kind::f16 [%0], %1, %2, %3, p;\n}\n" :: + "r"(tmem_d), + "l"(adesc), "l"(bdesc), "r"(idesc), "r"(accumulate) + : "memory"); +} + +__device__ __forceinline__ void umma_commit(uint32_t bar) +{ + asm volatile("tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.b64 [%0];\n" ::"r"(bar) : "memory"); +} + +__device__ __forceinline__ void mbar_wait_cta(uint32_t bar, uint32_t parity) +{ + uint32_t ready = 0; +#ifdef M3FA_DBG_HANG + long long spins = 0; +#endif + while (!ready) + { +#ifdef M3FA_DBG_HANG + if (++spins == (1LL << 24)) + { + printf("M3FA hang: block %d thread %d mbar 0x%x parity %u\n", blockIdx.x, threadIdx.x, bar, parity); + } +#endif + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.shared::cta.b64 p, [%1], %2;\n selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(bar), "r"(parity) + : "memory"); + } +} + +// Spin without suspending (v52's acc_done wait): a suspended try_wait wakes late on a tcgen05.commit arrival. +#ifndef M3FA_EPI_SPIN +#define M3FA_EPI_SPIN 1 +#endif +__device__ __forceinline__ void mbar_wait_epi(uint32_t bar, uint32_t parity) +{ + if (!M3FA_EPI_SPIN) + { + mbar_wait_cta(bar, parity); + return; + } + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.shared::cta.b64 p, [%1], %2;\n selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(bar), "r"(parity) + : "memory"); + } +} + +__device__ __forceinline__ void tmem_ld8(uint32_t addr, float (&v)[8]) +{ + uint32_t r[8]; + asm volatile("tcgen05.ld.sync.aligned.32x32b.x8.b32 {%0, %1, %2, %3, %4, %5, %6, %7}, [%8];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]), "=r"(r[4]), "=r"(r[5]), "=r"(r[6]), "=r"(r[7]) + : "r"(addr)); + asm volatile("tcgen05.wait::ld.sync.aligned;\n" ::: "memory"); +#pragma unroll + for (int i = 0; i < 8; ++i) + { + v[i] = __uint_as_float(r[i]); + } +} + +__device__ __forceinline__ void tmem_ld16(uint32_t addr, float (&v)[16]) +{ + uint32_t r[16]; + asm volatile( + "tcgen05.ld.sync.aligned.32x32b.x16.b32 {%0, %1, %2, %3, %4, %5, %6, %7, %8, %9, %10, %11, %12, %13, %14, " + "%15}, [%16];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]), "=r"(r[4]), "=r"(r[5]), "=r"(r[6]), "=r"(r[7]), "=r"(r[8]), + "=r"(r[9]), "=r"(r[10]), "=r"(r[11]), "=r"(r[12]), "=r"(r[13]), "=r"(r[14]), "=r"(r[15]) + : "r"(addr)); + asm volatile("tcgen05.wait::ld.sync.aligned;\n" ::: "memory"); +#pragma unroll + for (int i = 0; i < 16; ++i) + { + v[i] = __uint_as_float(r[i]); + } +} + +// Warp-level fp16 MMA (m16n8k16, fp32 accumulate) and its shared-memory fragment loads. +__device__ __forceinline__ void ldsm_x4(uint32_t addr, uint32_t (&r)[4]) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0, %1, %2, %3}, [%4];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x2(uint32_t addr, uint32_t (&r)[2]) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x2.shared.b16 {%0, %1}, [%2];\n" : "=r"(r[0]), "=r"(r[1]) : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x2_trans(uint32_t addr, uint32_t (&r)[2]) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x2.trans.shared.b16 {%0, %1}, [%2];\n" + : "=r"(r[0]), "=r"(r[1]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma_f16_16816(float (&d)[4], uint32_t const (&a)[4], uint32_t const (&b)[2]) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0, %1, %2, %3}, {%4, %5, %6, %7}, {%8, %9}, " + "{%0, %1, %2, %3};\n" + : "+f"(d[0]), "+f"(d[1]), "+f"(d[2]), "+f"(d[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b[0]), "r"(b[1])); +} + +// Register attention core (warp MMAs, no TMEM) for up to two tokens per cluster; the tcgen05 core above that. +// Off by default: on GB300 it measured ~0.25 us slower than the tcgen05 core at M=1 (GEMV + attention chain). +#ifndef M3FA_REG_CORE +#define M3FA_REG_CORE 0 +#endif +template +struct RegCore +{ + static constexpr bool kOn = M3FA_REG_CORE != 0 && QG <= 2; +}; + +template +struct Smem +{ + static constexpr int kRows = 16 * QG; // query rows (token, head) of this cluster = UMMA N + static constexpr int kTok = BPC * kPage; // block tokens per CTA + static constexpr int kTileBytes = kPage * kHeadDim * 2; // one fp16 K or V block tile in UMMA layout + static constexpr int kK16 = 0; + static constexpr int kV16 = kK16 + BPC * kTileBytes; + static constexpr int kQ = kV16 + BPC * kTileBytes; // Q rows, UMMA K-major [row][dim] + // P^T per block, UMMA MN-major [token][row], over the block's K tile: P is written after the QK MMAs (which + // read every K tile) have completed. + static constexpr int kP = kK16; + // Packed V tiles as copied from the cache: [block][128 rows x 64 data bytes | 1 KB of 4x4-quad scales]. + static constexpr int kRawV = kQ + kRows * kHeadDim * 2; + static constexpr int kRawVBlock = kPage * 64 + kPage * 8; + static constexpr int kRowScale = kRawV + BPC * kRawVBlock; + static constexpr int kRowLse = kRowScale + BPC * kRows * 4; + // Per-warp softmax partials [block][row][4 warps]: max, then sum. + static constexpr int kRedM = (kRowLse + BPC * kRows * 4 + 15) / 16 * 16; + static constexpr int kRedL = kRedM + BPC * kRows * 16; + // Partials pushed by every CTA of the cluster: [chunk][token][local head][kOPitch] floats, lse at column 128. + // When they do not fit beside the tiles they overlay the K/V tiles (and P, Q): peers then push only once every + // CTA of the cluster has completed its PV MMAs (a cluster barrier right before the push). + // Register core: per-warp row maxima and sums [block][row][warp of the block], two arrays. + static constexpr int kRedX = kRedL + BPC * kRows * 16; + static constexpr int kRedXBytes = RegCore::kOn ? 2 * BPC * kRows * (16 / BPC) * 4 : 0; + static constexpr int kStatsEnd = kRedX + kRedXBytes; + // Register core: P (fp16, [row][token] core matrices) at the start of its block's K tile, the scaled-free O + // partial ([row][kOPitch] fp32) behind it. + static constexpr int kObufOff = 8192; + static_assert(!RegCore::kOn || (kRows * kPage * 2 <= kObufOff && kObufOff + kRows * kOPitch * 4 <= kTileBytes), + "register-core P and O staging fit in a K tile"); + static constexpr int kRecvBytes = kBlocks * QG * BPC * kOPitch * 4; + static constexpr bool kRecvAlias = kStatsEnd + kRecvBytes + 32 > 227 * 1024; + static constexpr int kRecv = kRecvAlias ? 0 : kStatsEnd; + static constexpr int kMbar = kRecvAlias ? kStatsEnd : kStatsEnd + kRecvBytes; // merge, MMA mbarriers, TMEM addr + static constexpr int kBytes = kMbar + 32; + static constexpr int kTmemCols = BPC * 2 * kRows <= 32 ? 32 + : BPC * 2 * kRows <= 64 ? 64 + : BPC * 2 * kRows <= 128 ? 128 + : BPC * 2 * kRows <= 256 ? 256 + : 512; + static_assert(BPC * 2 * kRows <= 512, "TMEM columns"); + static_assert(!kRecvAlias || kRecvBytes <= kRawV, "aliased partials must fit in the tiles, Q and P"); + static_assert(kRows * kPage * 2 <= kTileBytes, "P^T fits in the K tiles"); + static_assert(kBytes <= 227 * 1024, "shared memory budget"); + static_assert(QLEN % QG == 0 && (BPC == 1 || BPC == 2), "unsupported geometry"); +}; + +// The packed NVFP4 bytes of one 16-byte chunk (32 dims of one block row) and its two block scales. +struct RawChunk +{ + uint4 data; + uint32_t scales; // byte 0: dims [0, 16), byte 1: dims [16, 32) of the chunk +}; + +// Chunk q of a block tile: lanes 8j..8j+7 of a warp take 8 consecutive rows of one 32-dim chunk, so every 16-byte +// fp16 store phase writes 8 distinct core-matrix rows (conflict-free in the UMMA layout). +__device__ __forceinline__ int chunk_row(int q) +{ + return (q >> 5) * 8 + (q & 7); +} + +__device__ __forceinline__ int chunk_col(int q) +{ + return (q >> 3) & 3; +} + +__device__ __forceinline__ RawChunk load_chunk( + uint8_t const* data_page, uint8_t const* scale_page, int row, int cc, bool is_v) +{ + RawChunk c; + c.data = *reinterpret_cast(data_page + row * 64 + cc * 16); + if (is_v) + { + // V scales: 4x4 token-quad order [row / 4][scale column][row % 4]. + uint8_t const* sc = scale_page + (row >> 2) * 32 + (row & 3); + c.scales = static_cast(sc[(cc * 2) * 4]) | (static_cast(sc[(cc * 2 + 1) * 4]) << 8); + } + else + { + c.scales = *reinterpret_cast(scale_page + row * 8 + cc * 2); + } + return c; +} + +__device__ __forceinline__ void store_chunk_fp16(uint8_t* tile, int row, int cc, RawChunk const& c, bool valid) +{ + uint32_t const s0 = e4m3_to_h2_splat(c.scales & 0xFFu); + uint32_t const s1 = e4m3_to_h2_splat((c.scales >> 8) & 0xFFu); + uint32_t const wd[4] = {c.data.x, c.data.y, c.data.z, c.data.w}; + uint32_t out[16]; +#pragma unroll + for (int i = 0; i < 16; ++i) + { + uint32_t const b = (wd[i >> 2] >> (8 * (i & 3))) & 0xFFu; + out[i] = valid ? hmul2_u(e2m1x2_to_h2(b), i < 8 ? s0 : s1) : 0u; + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + *reinterpret_cast(tile + kmaj_off(row, cc * 32 + j * 8)) + = make_uint4(out[4 * j], out[4 * j + 1], out[4 * j + 2], out[4 * j + 3]); + } +} + +// ---------------- Fused QKV projection: block-scaled MXFP8 on tcgen05 (N=2560, K=6144) ---------------- +// out[M, 2560] = mxfp8(x)[M, 6144] @ W^T, the computation of the CUTLASS MXFP8 GEMM: W is the A operand (128 output +// rows per MMA), the quantized activation the B operand (8 token columns), FP32 accumulation in TMEM. A unit is one +// (128-row tile, K split) pair; the 20 x kQkvSplits units run on the last CTAs of the grid. Before the PDL wait a +// CTA streams its unit's weight tiles and scale atoms through a shared-memory ring into TMEM; after it, it quantizes +// its x slice (mxfp8_quantize math), runs the MMAs and stores its FP32 partial rows. +// +// Hand-off without a flag: the partial buffers are double buffered by call parity and hold a sentinel (an FP32 NaN +// the MMA never produces) at rest, so a consumer polls the partials themselves. The producers of call n reset the +// other parity's buffers after their PDL wait; call n - 1, the last reader of those, is complete by then. +constexpr int kQkvN = 2560; +constexpr int kQkvK = 6144; +constexpr int kQkvKTiles = kQkvK / 128; +constexpr int kQkvRowTiles = kQkvN / 128; +#ifndef M3FA_QKV_SPLITS +#define M3FA_QKV_SPLITS 4 +#endif +#ifndef M3FA_QKV_RING +#define M3FA_QKV_RING kQkvMaxTiles // every tile of the unit in flight at once +#endif +#ifndef M3FA_QKV_NACC +#define M3FA_QKV_NACC 1 +#endif +#ifndef M3FA_QKV_SCALED_MMA +#define M3FA_QKV_SCALED_MMA 1 // 0: unscaled per-block MMAs with the scales applied in the epilogue (see below) +#endif +#ifndef M3FA_QKV_PIPE +// 1: the weight TMA is issued first at entry; the activation has its own region, copied right after the PDL wait; +// each weight tile's MMAs issue as soon as it lands (v52's order). 0: tiles into TMEM first, activation over the ring. +#define M3FA_QKV_PIPE 1 +#endif +#ifndef M3FA_QKV_A_SMEM +#define M3FA_QKV_A_SMEM 1 // scaled MMAs only; 1: they read the weight tiles from shared memory (no copy into TMEM) +#endif +// Default: block-scaled kind::mxf8f6f4 MMAs reading A (the weight tiles) in place from the ring, issued per tile as +// it lands (M3FA_QKV_PIPE), one accumulator; M3FA_QKV_A_SMEM=0 copies each tile into TMEM first. Alternative +// (M3FA_QKV_SCALED_MMA=0): one unscaled kind::f8f6f4 MMA per 32-wide K block into its own accumulator, A read in place +// from shared memory, the epilogue applying both UE8M0 scales. On GB300 an M128 x N8 x K32 MMA occupies the tensor pipe +// ~105 cycles block-scaled and ~60 unscaled, independent of N up to 64; the unscaled form's epilogue (320 accumulator +// columns) costs more than it saves. +constexpr bool kQkvScaled = M3FA_QKV_SCALED_MMA != 0; +constexpr bool kQkvASmem = !kQkvScaled || M3FA_QKV_A_SMEM != 0; +constexpr int kQkvSplits = M3FA_QKV_SPLITS; +constexpr int kQkvUnits = kQkvRowTiles * kQkvSplits; +constexpr int kQkvMaxTiles = (kQkvKTiles + kQkvSplits - 1) / kQkvSplits; +constexpr int kQkvRing = M3FA_QKV_RING; +constexpr int kQkvNacc = M3FA_QKV_NACC; +constexpr int kQkvGrid = 112; // 7 clusters of 16 (or 14 of 8): one wave with the attention +constexpr int kMaxTok = 8; +constexpr int kQkvWTile = 128 * 128; // E4M3 bytes of one 128 x 128 weight tile +constexpr int kQkvSfAtom = 512; // one 128-row x 4-block UE8M0 scale atom +constexpr uint32_t kQkvSentinel = 0xFFFFFFFFu; +constexpr int64_t kQkvPartStride = static_cast(kQkvSplits) * kMaxTok * kQkvN; // one parity +// TMEM columns of the one 512-column allocation of a CTA: attention [0, 64), then the GEMM's accumulators, scale +// atoms (A per tile, B per tile) and the A tiles (32 columns of 128 x 128 E4M3 each). +constexpr int kTmemAll = 512; +// Megakernel: the attention CTAs run no projection unit, so the GEMM starts at column 0 of a projection CTA. +constexpr int kTAcc = 0; +constexpr int kTSfa = kTAcc + 16; +constexpr int kTSfb = kTSfa + 4 * kQkvMaxTiles; +constexpr int kTA = (kTSfb + 4 * kQkvMaxTiles + 31) / 32 * 32; +static_assert(!kQkvScaled || kTA + 32 * kQkvMaxTiles <= kTmemAll, "TMEM columns"); +static_assert(kQkvScaled || kTAcc + 32 * kQkvMaxTiles <= kTmemAll, "TMEM columns (one accumulator per K block)"); +static_assert(!kQkvASmem || kQkvRing == kQkvMaxTiles, "shared-memory A needs every tile resident"); +static_assert(kQkvUnits <= kQkvGrid, "at most one unit per CTA"); +static_assert(kQkvNacc == 1 || kQkvNacc == 2 || kQkvNacc == 4, "accumulator count"); +static_assert(!M3FA_QKV_PIPE || kQkvScaled, "the pipelined projection uses the block-scaled MMAs"); +constexpr int kQkvOwnRows = 128 / kQkvSplits; // rows of a tile whose split-K sum a CTA owns +// Megakernel o_proj (v52 m3_mx_gemv for [6144, 2048]: split 2, contiguous k-tiles, one accumulator): unit +// o = CTA - 16 (the last 96 CTAs), row tile o / 2, split o % 2. +constexpr int kOpN = 6144; +constexpr int kOpK = 2048; +constexpr int kOpKTiles = kOpK / 128; +constexpr int kOpSplits = 2; +constexpr int kOpTiles = kOpKTiles / kOpSplits; +constexpr int kOpOwnRows = 128 / kOpSplits; +constexpr int kOpFirstCta = kQkvGrid - kOpN / 128 * kOpSplits; // 16 +constexpr int kOpSfBytes = kOpKTiles * 512; // one parity of the MXFP8 hand-off's scales +#ifndef M3FA_OP_L2PF +#define M3FA_OP_L2PF \ + 0 // 1: QKV CTAs prefetch their o_proj weight slice into L2 before the PDL wait (competes + // with the QKV weight stream for HBM behind the MoE) +#endif +// o_proj TMEM columns (after the QKV unit's MMAs completed): accumulator, A / B scale atoms, A (8 tiles x 32) +constexpr int kOAcc = 0; +constexpr int kOSfa = 16; +constexpr int kOSfb = kOSfa + 4 * kOpTiles; +constexpr int kOA = 128; +static_assert(kOA + 32 * kOpTiles <= kTmemAll && kOSfb + 4 * kOpTiles <= kOA, "o_proj TMEM columns"); +constexpr int kQkvMbStride = kMaxTok + 4; // mailbox row pitch (fp32 words) + +__host__ __device__ constexpr int qkv_tile0(int s) +{ + return s * kQkvKTiles / kQkvSplits; +} + +__device__ __forceinline__ int qkv_unit() +{ + int const u = kQkvGrid - 1 - static_cast(blockIdx.x); + return u < kQkvUnits ? u : -1; +} + +// Attention layout plus, for the fused projection, the weight ring in front (the attention tiles reuse that space +// once the ring has drained into TMEM); the activation tiles, the pre-wait V copy and the barriers sit behind. +template +struct SmemX : Smem +{ + using B = Smem; + static constexpr int kRingW = 0; // [ring][128 rows][128 B], 128-byte swizzle + static constexpr int kRingSf = kRingW + kQkvRing * kQkvWTile; // [ring][512 B] + static constexpr int kRingEnd = kRingSf + kQkvRing * kQkvSfAtom; + // The activation tiles overlay the ring: written after the PDL wait, once every weight tile went on into TMEM. + static constexpr int kXt = kRingW; // [tile][8][128 B] + static constexpr int kXsf = kXt + kQkvMaxTiles * 1024; // [tile][512 B] + static constexpr int kXsc = (kRingEnd + 127) / 128 * 128; + // Peers push into recv at any time (an empty CTA right away), so it stays clear of the ring. + static constexpr int kRecv = FQKV ? kXsc : B::kRecv; + static constexpr int kRawV = FQKV ? (kRecv + kBlocks * QG * BPC * kOPitch * 4 + 127) / 128 * 128 : B::kRawV; + // Pipelined projection: the activation tiles and scale atoms in their own region (a projection CTA runs no + // attention, so it overlays the attention's partial-O and V regions), 1024-byte aligned for the swizzle. + static constexpr int kXA = FQKV ? (kRecv + 1023) / 1024 * 1024 : 0; + static constexpr int kXASf = kXA + kQkvMaxTiles * 1024; + static constexpr int kXAEnd = kXASf + kQkvMaxTiles * kQkvSfAtom; + static constexpr int kMbarA = FQKV ? kRawV + BPC * B::kRawVBlock : B::kMbar; + static constexpr int kMbar = FQKV && M3FA_QKV_PIPE && kXAEnd > kMbarA ? (kXAEnd + 127) / 128 * 128 : kMbarA; + static constexpr int kGBar = kMbar + 32; // full[ring], empty[ring], a_ready, acc_done, tile_done[tiles] + static constexpr int kRedBar = kGBar + 8 * (2 * kQkvRing + 2 + kQkvMaxTiles); // split-K partials in + static constexpr int kOBar = kRedBar + 8; // o_proj: full[8], a_ready, acc_done, reduce + static constexpr int kMailbox = (kOBar + 8 * (kOpTiles + 3) + 15) / 16 * 16; // [split][own row][pitch] fp32 + // o_proj weight ring = ring tiles [0, 8); its activation tiles, scale atoms and mailbox behind it. + static constexpr int kOXt = kRingW + kOpTiles * kQkvWTile; + static constexpr int kOXsf = kOXt + kOpTiles * 1024; + static constexpr int kOMailbox = kOXsf + kOpTiles * kQkvSfAtom; + static_assert(!FQKV || kOMailbox + kOpSplits * kOpOwnRows * kQkvMbStride * 4 <= kRingW + kQkvRing * kQkvWTile, + "o_proj activation and mailbox fit behind its ring"); + static constexpr int kBytes = FQKV ? kMailbox + kQkvSplits * kQkvOwnRows * kQkvMbStride * 4 : B::kBytes; + static constexpr int kTmemCols = FQKV ? kTmemAll : B::kTmemCols; + static constexpr bool kRecvAlias = !FQKV && B::kRecvAlias; + // The epilogue reads the weight scale atoms while the attention already writes its tiles. + static_assert(!FQKV || kQkvScaled || B::kRecv <= kRingSf, "attention regions overlap the scale atoms"); + // The other attention regions overlay the ring: a unit's CTA writes them only once the ring has drained. + static_assert(kBytes <= 227 * 1024, "shared memory budget"); +}; + +__device__ __forceinline__ void mbar_init(uint32_t bar, uint32_t count) +{ + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;\n" ::"r"(bar), "r"(count) : "memory"); +} + +__device__ __forceinline__ void mbar_expect_tx(uint32_t bar, uint32_t bytes) +{ + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(bar), "r"(bytes) : "memory"); +} + +__device__ __forceinline__ void bulk_g2s(void* dst, void const* src, uint32_t bytes, uint32_t bar) +{ + asm volatile( + "cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes.L2::cache_hint [%0], [%1], %2, [%3], " + "%4;\n" ::"r"(smem_u32(dst)), + "l"(src), "r"(bytes), "r"(bar), "l"(0x12F0000000000000ull) // EVICT_FIRST: weights are read once per layer + : "memory"); +} + +__device__ __forceinline__ void tma_load_2d(uint32_t dst, void const* map, int c0, int c1, uint32_t bar) +{ + asm volatile( + "cp.async.bulk.tensor.2d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.L2::cache_hint" + " [%0], [%1, {%2, %3}], [%4], %5;\n" ::"r"(dst), + "l"(map), "r"(c0), "r"(c1), "r"(bar), "l"(0x12F0000000000000ull) + : "memory"); +} + +// K-major operand in the 128-byte swizzle: rows of 128 bytes, 8-row groups 1024 bytes apart. +__device__ __forceinline__ uint64_t umma_desc_sw128(uint32_t addr) +{ + return umma_desc(addr, 16, 1024) | (2ull << 61); +} + +__device__ __forceinline__ void tmem_cp_128x256b(uint32_t taddr, uint64_t desc) +{ + asm volatile("tcgen05.cp.cta_group::1.128x256b [%0], %1;\n" ::"r"(taddr), "l"(desc)); +} + +__device__ __forceinline__ void tmem_cp_sf(uint32_t taddr, uint32_t saddr) +{ + asm volatile("tcgen05.cp.cta_group::1.32x128b.warpx4 [%0], %1;\n" ::"r"(taddr), "l"(umma_desc(saddr, 16, 128))); +} + +// Block-scaled MXFP8 instruction descriptor: E4M3 x E4M3, UE8M0 scales, M = 128, N = 8, scale byte kb. +__host__ __device__ constexpr uint32_t qkv_idesc(int kb) +{ + return (static_cast(kb) << 4) | ((8u >> 3) << 17) | (1u << 23) | ((128u >> 7) << 27) + | (static_cast(kb) << 29); +} + +__device__ __forceinline__ void umma_mx( + uint32_t tmem_d, uint32_t tmem_a, uint64_t bdesc, uint32_t idesc, uint32_t sfa, uint32_t sfb, int accumulate) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, %6, 0;\n" + " tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X [%0], [%1], %2, %3, [%4], [%5], p;\n}\n" :: + "r"(tmem_d), + "r"(tmem_a), "l"(bdesc), "r"(idesc), "r"(sfa), "r"(sfb), "r"(accumulate)); +} + +__device__ __forceinline__ float pow2_from_ue8m0(uint32_t e) +{ + return __uint_as_float(e == 0 ? 0x00400000u : (e << 23)); +} + +// Unscaled E4M3 x E4M3 MMA, FP32 accumulator: M = 128, N = 8, K = 32, a fresh accumulator. +constexpr uint32_t kQkvIdescF8 = (1u << 4) | ((8u >> 3) << 17) | ((128u >> 4) << 24); + +__device__ __forceinline__ void umma_f8_ss(uint32_t tmem_d, uint64_t adesc, uint64_t bdesc) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, 0, 0;\n tcgen05.mma.cta_group::1.kind::f8f6f4 [%0], %1, %2, %3, p;\n}\n" :: + "r"(tmem_d), + "l"(adesc), "l"(bdesc), "r"(kQkvIdescF8)); +} + +__device__ __forceinline__ void tmem_ld16_nowait(uint32_t addr, float* v) +{ + uint32_t r[16]; + asm volatile( + "tcgen05.ld.sync.aligned.32x32b.x16.b32 {%0, %1, %2, %3, %4, %5, %6, %7, %8, %9, %10, %11, %12, %13, %14, " + "%15}, [%16];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]), "=r"(r[4]), "=r"(r[5]), "=r"(r[6]), "=r"(r[7]), "=r"(r[8]), + "=r"(r[9]), "=r"(r[10]), "=r"(r[11]), "=r"(r[12]), "=r"(r[13]), "=r"(r[14]), "=r"(r[15]) + : "r"(addr)); +#pragma unroll + for (int i = 0; i < 16; ++i) + { + v[i] = __uint_as_float(r[i]); + } +} + +__device__ __forceinline__ void umma_mx_ss( + uint32_t tmem_d, uint64_t adesc, uint64_t bdesc, uint32_t idesc, uint32_t sfa, uint32_t sfb, int accumulate) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, %6, 0;\n" + " tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X [%0], %1, %2, %3, [%4], [%5], p;\n}\n" :: + "r"(tmem_d), + "l"(adesc), "l"(bdesc), "r"(idesc), "r"(sfa), "r"(sfb), "r"(accumulate)); +} + +__device__ __forceinline__ bool elect_one() +{ + uint32_t pred; + asm volatile("{\n .reg .pred P;\n elect.sync _|P, 0xffffffff;\n selp.u32 %0, 1, 0, P;\n}\n" : "=r"(pred)); + return pred != 0; +} + +__device__ __forceinline__ int qkv_parity(Params const& p) +{ + int v; + asm volatile("ld.relaxed.gpu.global.s32 %0, [%1];\n" : "=r"(v) : "l"(p.counters) : "memory"); + return v & 1; +} + +// Barriers and TMEM (warp 1), then the first ring stages (warp 0). Warps 0 and 1 of every CTA call it. +template +__device__ __forceinline__ void qkv_setup( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int warp, int lane, int u) +{ + constexpr bool kPipe = M3FA_QKV_PIPE != 0; + if (kPipe && warp == 0 && lane == 0 && u >= 0) + { + // First thing at entry: this unit's whole weight slice. The issuing thread initializes the ring's full + // barriers itself; every waiter on them passes bar.sync 8 below, which this thread reaches after the init. + asm volatile("prefetch.tensormap [%0];\n" ::"l"(&p.wmap) : "memory"); + int const rt = u / kQkvSplits; + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + for (int i = 0; i < kQkvRing; ++i) + { + mbar_init(gbar + 8 * i, 1); + } + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + for (int i = 0; i < kQkvRing && i < nt; ++i) + { + uint32_t const full = gbar + 8 * i; + mbar_expect_tx(full, kQkvWTile + kQkvSfAtom); + tma_load_2d(smem_u32(smem + SX::kRingW + i * kQkvWTile), &p.wmap, (kt0 + i) * 128, rt * 128, full); + bulk_g2s(smem + SX::kRingSf + i * kQkvSfAtom, + p.wsf + (static_cast(rt) * kQkvKTiles + kt0 + i) * kQkvSfAtom, kQkvSfAtom, full); + } + M3FA_PROF_T(34, 0); + } + if (warp == 1) + { + asm volatile("tcgen05.alloc.cta_group::1.sync.aligned.shared::cta.b32 [%0], %1;\n" ::"r"(smem_u32(tmem_slot)), + "r"(kTmemAll) + : "memory"); + asm volatile("tcgen05.relinquish_alloc_permit.cta_group::1.sync.aligned;\n" ::: "memory"); + M3FA_PROF_T(35, 32); + if (lane == 0) + { + for (int i = (kPipe && u >= 0) ? kQkvRing : 0; i < 2 * kQkvRing + 2 + kQkvMaxTiles; ++i) + { + // PIPE 2: tile_done[0] (unused by the scaled MMAs) is act_ready, one arrival per thread outside warp 1 + mbar_init(gbar + 8 * i, M3FA_QKV_PIPE == 2 && i == 2 * kQkvRing + 2 ? kThreads - 32 : 1); + } + // The split-K reduce barrier: this CTA's own arrival carries the peers' partial bytes. + uint32_t const red = gbar + 8 * (2 * kQkvRing + 2 + kQkvMaxTiles); + mbar_init(red, 1); + mbar_expect_tx(red, (kQkvSplits - 1) * kQkvOwnRows * kMaxTok * 4); + uint32_t const ob = smem_u32(smem + SX::kOBar); + for (int i = 0; i < kOpTiles + 3; ++i) + { + mbar_init(ob + 8 * i, 1); + } + mbar_expect_tx(ob + 8 * (kOpTiles + 2), (kOpSplits - 1) * kOpOwnRows * kMaxTok * 4); + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + } + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + asm volatile("bar.sync 8, 64;\n" ::: "memory"); + if (!kPipe && warp == 0 && lane == 0 && u >= 0) + { + int const rt = u / kQkvSplits; + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + for (int i = 0; i < kQkvRing && i < nt; ++i) + { + uint32_t const full = gbar + 8 * i; + mbar_expect_tx(full, kQkvWTile + kQkvSfAtom); + tma_load_2d(smem_u32(smem + SX::kRingW + i * kQkvWTile), &p.wmap, (kt0 + i) * 128, rt * 128, full); + bulk_g2s(smem + SX::kRingSf + i * kQkvSfAtom, + p.wsf + (static_cast(rt) * kQkvKTiles + kt0 + i) * kQkvSfAtom, kQkvSfAtom, full); + } + M3FA_PROF_T(34, 0); + } +} + +// The rest of the ring (warp 0, lane 0) and the copies into TMEM (warp 1, lane 0); before the PDL wait. +template +__device__ __forceinline__ void qkv_stream_lane( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int warp, int u) +{ + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + uint32_t const full = gbar; + uint32_t const a_ready = gbar + 16 * kQkvRing; + if (warp == 0 || !kQkvScaled) + { + return; // every tile was issued in qkv_setup; the unscaled MMAs read them in place + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm = *tmem_slot; + // Fully unrolled with compile-time offsets: tcgen05 operands stay in uniform registers (a per-instruction + // conversion of computed operands costs ~150 cycles per MMA or copy). + uint64_t const w0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + uint32_t const sf0 = smem_u32(smem + SX::kRingSf); +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + mbar_wait_cta(full + 8 * i, 0); + if (i == 0) + { + M3FA_PROF_T(30, 32); + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if constexpr (!kQkvASmem) + { +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b(tm + kTA + (i * 4 + kb) * 8, w0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + } + tmem_cp_sf(tm + kTSfa + i * 4, sf0 + i * kQkvSfAtom); + } + } + M3FA_PROF_T(31, 32); + umma_commit(a_ready); +} + +__device__ __forceinline__ bool mbar_test(uint32_t bar, uint32_t parity) +{ + uint32_t ready; + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.shared::cta.b64 p, [%1], %2;\n selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(bar), "r"(parity) + : "memory"); + return ready != 0; +} + +// PIPE 2, warp 1's elected thread from before the PDL wait to the last MMA. Tiles that land before the activation +// is ready go on into TMEM (A and scale atom), off the critical path; once the activation is in (act_ready), the +// MMAs issue in tile order: the copied tiles with A in TMEM, every later tile as it lands with A read in place from +// the ring. The MMA sequence (and so the accumulation order) is the same either way. +template +__device__ __forceinline__ void qkv_hybrid(Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int u) +{ + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + uint32_t const act_ready = gbar + 8 * (2 * kQkvRing + 2); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm = *tmem_slot; + uint64_t const w0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + uint64_t const b0 = umma_desc_sw128(smem_u32(smem + SX::kXA)); + uint32_t const wsf0 = smem_u32(smem + SX::kRingSf); + uint32_t const xsf0 = smem_u32(smem + SX::kXASf); + int copied = 0; // tiles [0, copied) are in TMEM + bool act = false; +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt && !act) + { + while (!mbar_test(gbar + 8 * i, 0)) + { + if (mbar_test(act_ready, 0)) + { + act = true; + break; + } + } + if (!act) + { + if (i == 0) + { + M3FA_PROF_T(30, 32); + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b(tm + kTA + (i * 4 + kb) * 8, w0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + tmem_cp_sf(tm + kTSfa + i * 4, wsf0 + i * kQkvSfAtom); + copied = i + 1; + } + } + } + while (!act) + { + act = mbar_test(act_ready, 0); + } + M3FA_PROF_T(21, 32); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + bool const in_tmem = i < copied; + if (!in_tmem) + { + mbar_wait_epi(gbar + 8 * i, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + tmem_cp_sf(tm + kTSfa + i * 4, wsf0 + i * kQkvSfAtom); + } + uint32_t const sfb = tm + kTSfb + i * 4; + tmem_cp_sf(sfb, xsf0 + i * kQkvSfAtom); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + int const j = i * 4 + kb; + uint64_t const bd = b0 + ((i * 1024 + kb * 32) >> 4); + if (in_tmem) + { + umma_mx(tm + kTAcc, tm + kTA + j * 8, bd, qkv_idesc(kb), tm + kTSfa + i * 4, sfb, j > 0); + } + else + { + umma_mx_ss(tm + kTAcc, w0 + ((i * kQkvWTile + kb * 32) >> 4), bd, qkv_idesc(kb), tm + kTSfa + i * 4, + sfb, j > 0); + } + } + } + } + M3FA_PROF_T(22, 32); + M3FA_PROF_T(31, 32); + umma_commit(gbar + 16 * kQkvRing); // a_ready (for its waiters) + umma_commit(gbar + 16 * kQkvRing + 8); // acc_done +} + +// The other lanes of warps 0 and 1 hold at __syncwarp rather than at griddepcontrol.wait, which would stall the +// whole warp (lane 0's loop included) until the producer grid completes. +template +__device__ __forceinline__ void qkv_stream( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int warp, int lane, int u) +{ + if (M3FA_QKV_PIPE == 2 && u >= 0 && warp == 1) + { + if (elect_one()) + { + qkv_hybrid(p, smem, gbar, tmem_slot, u); + } + __syncwarp(); + return; + } + if (u < 0 || warp > 1 || M3FA_QKV_PIPE) + { + return; // pipelined: the MMA thread takes each tile as it lands + } + if (lane == 0) + { + qkv_stream_lane(p, smem, gbar, tmem_slot, warp, u); + } + __syncwarp(); +} + +// After the PDL wait. Every thread quantizes a piece of the x slice into the B tiles (named barrier 9); warp 1 then +// issues the scale copies and the MMAs. +template +__device__ __forceinline__ void qkv_mma( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int tid, int u) +{ + int const warp = tid >> 5; + int const s = u % kQkvSplits; + int const kt0 = qkv_tile0(s); + int const nt = qkv_tile0(s + 1) - kt0; + { + // The MXFP8 activation slice (E4M3 rows of the unit's k-tiles in 128-byte-swizzled B tiles, and their scale + // atoms), copied with plain 16-byte loads over the overlaid ring once its tiles went on into TMEM: every load + // first, then the stores (one L2 round trip). + constexpr int kXDst = M3FA_QKV_PIPE ? SX::kXA : SX::kXt; + constexpr int kXSfDst = M3FA_QKV_PIPE ? SX::kXASf : SX::kXsf; + if (!M3FA_QKV_PIPE) + { + mbar_wait_cta(gbar + 16 * kQkvRing, 0); // a_ready: the ring's weight tiles are in TMEM + } + constexpr int kXItems = kQkvMaxTiles * 8 * kMaxTok; // (tile, 16-byte chunk, token) + constexpr int kSfItems = kQkvMaxTiles * (kQkvSfAtom / 16); + constexpr bool kHyb = M3FA_QKV_PIPE == 2; + constexpr int kCopiers = kHyb ? kThreads - 32 : kThreads; // PIPE 2: warp 1 is issuing the tiles + if (kHyb && warp == 1) + { + return; + } + int const ctid = kHyb && tid >= 64 ? tid - 32 : tid; + constexpr int kPer = (kXItems + kSfItems + kCopiers - 1) / kCopiers; + uint4 v[kPer]; + int dst[kPer]; +#pragma unroll + for (int k = 0; k < kPer; ++k) + { + int const q = ctid + kCopiers * k; + dst[k] = -1; + if (q < kXItems) + { + int const t = q % kMaxTok; + int const c = (q / kMaxTok) % 8; + int const i = q / (kMaxTok * 8); + if (i < nt && t < p.num_tokens) + { + v[k] = *reinterpret_cast( + p.xq + static_cast(t) * p.ldxq + (kt0 + i) * 128 + c * 16); + dst[k] = kXDst + i * 1024 + t * 128 + ((c ^ t) * 16); + } + } + else if (q < kXItems + kSfItems) + { + int const j = q - kXItems; + int const i = j / (kQkvSfAtom / 16); + int const c = j % (kQkvSfAtom / 16); + if (i < nt) + { + v[k] = *reinterpret_cast( + p.xsf_act + static_cast(kt0 + i) * kQkvSfAtom + c * 16); + dst[k] = kXSfDst + i * kQkvSfAtom + c * 16; + } + } + } +#pragma unroll + for (int k = 0; k < kPer; ++k) + { + if (dst[k] >= 0) + { + *reinterpret_cast(smem + dst[k]) = v[k]; + } + } + asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory"); + if (kHyb) + { + asm volatile("mbarrier.arrive.release.cta.shared::cta.b64 _, [%0];\n" ::"r"(gbar + 8 * (2 * kQkvRing + 2)) + : "memory"); + return; + } + if (warp != 1) + { + asm volatile("bar.arrive 9, %0;\n" ::"n"(kThreads) : "memory"); + } + } + if (warp == 1) + { + M3FA_PROF_T(32, 32); + asm volatile("bar.sync 9, %0;\n" ::"n"(kThreads) : "memory"); + M3FA_PROF_T(21, 32); + // Operands computed by the whole warp (warp-uniform, kept in uniform registers); one elected lane issues. + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm = *tmem_slot; + uint64_t const b0 = umma_desc_sw128(smem_u32(smem + (M3FA_QKV_PIPE ? SX::kXA : SX::kXt))); + uint64_t const a0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + if (elect_one()) + { + M3FA_CLK(41); + if constexpr (!kQkvScaled) + { +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + mbar_wait_cta(gbar + 8 * i, 0); // weight tile i landed + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + umma_f8_ss(tm + kTAcc + (i * 4 + kb) * 8, a0 + ((i * kQkvWTile + kb * 32) >> 4), + b0 + ((i * 1024 + kb * 32) >> 4)); + } + umma_commit(gbar + 8 * (2 * kQkvRing + 2 + i)); // tile i's blocks are in TMEM + } + } + } + else + { + uint32_t const xsf0 = smem_u32(smem + (M3FA_QKV_PIPE ? SX::kXASf : SX::kXsf)); + uint32_t const wsf0 = smem_u32(smem + SX::kRingSf); +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + if constexpr (M3FA_QKV_PIPE != 0) + { + // Tile i as it lands: its A copy and scale atom into TMEM, then its MMAs (issue order). + mbar_wait_cta(gbar + 8 * i, 0); + if (i == 0) + { + M3FA_PROF_T(30, 32); + } + if (i == nt - 1) + { + M3FA_CLK(47); // the unit's last weight tile landed (clock64 builds) + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if constexpr (!kQkvASmem) + { +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b( + tm + kTA + (i * 4 + kb) * 8, a0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + } + tmem_cp_sf(tm + kTSfa + i * 4, wsf0 + i * kQkvSfAtom); + } + uint32_t const sfb = tm + kTSfb + i * 4; + tmem_cp_sf(sfb, xsf0 + i * kQkvSfAtom); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + int const j = i * 4 + kb; + uint64_t const bd = b0 + ((i * 1024 + kb * 32) >> 4); + if constexpr (kQkvASmem) + { + umma_mx_ss(tm + kTAcc + (j % kQkvNacc) * 8, a0 + ((i * kQkvWTile + kb * 32) >> 4), bd, + qkv_idesc(kb), tm + kTSfa + i * 4, sfb, j >= kQkvNacc); + } + else + { + umma_mx(tm + kTAcc + (j % kQkvNacc) * 8, tm + kTA + j * 8, bd, qkv_idesc(kb), + tm + kTSfa + i * 4, sfb, j >= kQkvNacc); + } + } + } + } + } + M3FA_CLK(42); + if (M3FA_QKV_PIPE) + { + M3FA_PROF_T(31, 32); + umma_commit(gbar + 16 * kQkvRing); // a_ready (kept for its waiters: every tile is consumed) + } + umma_commit(gbar + 16 * kQkvRing + 8); // acc_done + } + M3FA_PROF_T(22, 32); + } +} + +// Warps 16..19 (TMEM lanes 32 (warp % 4) + lane = the unit's rows): partial rows out, the other parity reset. +template +__device__ __forceinline__ void qkv_epilogue( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int tid, int u, int par, unsigned int calls) +{ + int const warp = tid >> 5; + int const lane = tid & 31; + if constexpr (kQkvScaled) + { + mbar_wait_epi(gbar + 16 * kQkvRing + 8, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + } + M3FA_PROF_T(24, 512); + if (tid == 512) + { + M3FA_CLK(43); + } + int const r = (warp & 3) * 32 + lane; // row within the unit's tile + uint32_t const taddr = *tmem_slot + (((warp & 3) * 32) << 16) + kTAcc; + float val[kMaxTok]; + if constexpr (kQkvScaled) + { + float v[32]; + tmem_ld16(taddr, *reinterpret_cast(v)); + if constexpr (kQkvNacc == 4) + { + tmem_ld16(taddr + 16, *reinterpret_cast(v + 16)); + } +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + val[t] = kQkvNacc == 4 ? (v[t] + v[8 + t]) + (v[16 + t] + v[24 + t]) + : kQkvNacc == 2 ? v[t] + v[8 + t] + : v[t]; + } + } + else + { + // Sum over K blocks in order: block result x (weight scale x activation scale). + int const nt = qkv_tile0(u % kQkvSplits + 1) - qkv_tile0(u % kQkvSplits); + float const* xsc = reinterpret_cast(smem + SX::kXsc); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + val[t] = 0.0f; + } +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + // Tile i's four blocks as soon as its MMAs complete: the epilogue trails the MMAs by one tile. + mbar_wait_cta(gbar + 8 * (2 * kQkvRing + 2 + i), 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + float d[32]; + tmem_ld16_nowait(taddr + i * 32, d); + tmem_ld16_nowait(taddr + i * 32 + 16, d + 16); + uint32_t const wsw = *reinterpret_cast( + smem + SX::kRingSf + i * kQkvSfAtom + (r % 32) * 16 + (r / 32) * 4); + asm volatile("tcgen05.wait::ld.sync.aligned;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + float const ws = pow2_from_ue8m0((wsw >> (8 * kb)) & 0xFFu); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + val[t] = fmaf(d[kb * 8 + t], ws * xsc[(i * 4 + kb) * kMaxTok + t], val[t]); + } + } + } + } + } + // v52's split-K reduce: split s owns rows [32 s, 32 s + 32) of the tile; the other splits push their fp32 partials + // of those rows into its mailbox (st.async, completing its reduce barrier's bytes); the owner adds the splits in + // rank order from 0.0 and stores bf16. + int const s = u % kQkvSplits; + int const owner = r / kQkvOwnRows; + int const local = r % kQkvOwnRows; + uint32_t const red = gbar + 8 * (2 * kQkvRing + 2 + kQkvMaxTiles); + uint32_t cta_rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(cta_rank)); + // Units are numbered down from the last CTA: split q of this tile is CTA rank cta_rank - (q - s). + if (owner != s) + { + uint32_t const peer = cta_rank + s - owner; + uint32_t const dst + = mapa_shared(smem_u32(smem + SX::kMailbox) + ((s * kQkvOwnRows + local) * kQkvMbStride) * 4, peer); + uint32_t const pbar = mapa_shared(red, peer); +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + asm volatile( + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [%0], {%1, %2, %3, %4}, [%5];\n" ::"r"( + dst + 16 * g), + "r"(__float_as_uint(val[4 * g])), "r"(__float_as_uint(val[4 * g + 1])), + "r"(__float_as_uint(val[4 * g + 2])), "r"(__float_as_uint(val[4 * g + 3])), "r"(pbar) + : "memory"); + } + } + else + { + M3FA_PROF_T(26, tid - lane); + // test_wait (no suspend): completions by peers' st.async bytes wake a suspended try_wait late. + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(red) + : "memory"); + } + M3FA_PROF_T(27, tid - lane); + float const* mb = reinterpret_cast(smem + SX::kMailbox); + int const row = (u / kQkvSplits) * 128 + r; + // Lamport hand-off: this call's rows go to buffer (calls & 1) with relaxed stores, the consumers poll the + // values themselves (0xFFFF, a NaN the rounding of a finite sum never produces, marks "not yet"); the other + // buffer, last read by the previous call (complete: this grid passed its PDL wait), is reset to the marker. + uint16_t* cur = reinterpret_cast(p.qkv_out) + (calls & 1u) * kMaxTok * kQkvN; + uint16_t* nxt = reinterpret_cast(p.qkv_out) + ((calls & 1u) ^ 1u) * kMaxTok * kQkvN; +#pragma unroll + float qsum[kMaxTok]; +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + float4 part[kQkvSplits]; +#pragma unroll + for (int q = 0; q < kQkvSplits; ++q) + { + part[q] = *reinterpret_cast(mb + (q * kQkvOwnRows + local) * kQkvMbStride + 4 * g); + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float sum = 0.0F; +#pragma unroll + for (int q = 0; q < kQkvSplits; ++q) + { + float const pj = j == 0 ? part[q].x : j == 1 ? part[q].y : j == 2 ? part[q].z : part[q].w; + sum = sum + (q == s ? val[4 * g + j] : pj); + } + qsum[4 * g + j] = sum; + } + } +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + float const sum = qsum[t]; + if (t < p.num_tokens) + { + __nv_bfloat16 const b = __float2bfloat16_rn(sum); + uint16_t bits = *reinterpret_cast(&b); + // Never the "not yet" marker: a 0xFFFF NaN sum is stored as the NaN 0x7FFF. Only a NaN projection row + // changes (its K/V cache bytes then differ from v52 + p6 in the NaN's encoding); finite rows are exact. + bits = bits == 0xFFFFu ? static_cast(0x7FFFu) : bits; +#ifndef M3FA_QKV_REL +#define M3FA_QKV_REL 0 // 1: the row store is a release (tried: whether it becomes visible to the pollers sooner) +#endif + if (M3FA_QKV_REL) + { + asm volatile( + "st.release.gpu.global.u16 [%0], %1;\n" ::"l"(cur + static_cast(t) * kQkvN + row), + "h"(bits) + : "memory"); + } + else + { + asm volatile( + "st.relaxed.gpu.global.u16 [%0], %1;\n" ::"l"(cur + static_cast(t) * kQkvN + row), + "h"(bits) + : "memory"); + } + } + asm volatile("st.relaxed.gpu.global.u16 [%0], %1;\n" ::"l"(nxt + static_cast(t) * kQkvN + row), + "h"(static_cast(0xFFFFu)) + : "memory"); + } + M3FA_PROF_T(28, tid - lane); + } + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + M3FA_PROF_T(25, 512); + if (tid == 512) + { + M3FA_CLK(46); + } +} + +// ---------------- Megakernel o_proj (v52 m3_mx_gemv bit for bit: split 2, contiguous k-tiles, nacc 1) ---------------- +// The unit's weight tiles and scale atoms into the ring (warp 0, lane 0). From L2 when the CTA prefetched them there. +template +__device__ __forceinline__ void op_issue(Params const& p, uint8_t* smem, int ou) +{ + int const rt = ou / kOpSplits; + int const kt0 = (ou % kOpSplits) * kOpTiles; + uint32_t const ob = smem_u32(smem + SX::kOBar); + for (int i = 0; i < kOpTiles; ++i) + { + mbar_expect_tx(ob + 8 * i, kQkvWTile + kQkvSfAtom); + tma_load_2d(smem_u32(smem + SX::kRingW + i * kQkvWTile), &p.wmap_o, (kt0 + i) * 128, rt * 128, ob + 8 * i); + bulk_g2s(smem + SX::kRingSf + i * kQkvSfAtom, + p.wsf_o + (static_cast(rt) * kOpKTiles + kt0 + i) * kQkvSfAtom, kQkvSfAtom, ob + 8 * i); + } +} + +// L2 prefetch of the unit's weight tiles and scale atoms (one lane; nothing waits for it). +// The QKV units' share of the window-1 L2 prefetch (the next consumer's weights), issued once their own QKV weight +// stream has landed (acc_done): the attention core reads little from HBM. +__device__ __forceinline__ void window_prefetch_l2(Params const& p, int u) +{ + constexpr long long kChunk = 32768; + long long const n0 = (p.pf_b0 + kChunk - 1) / kChunk, n1 = (p.pf_b1 + kChunk - 1) / kChunk; + uint64_t pol = 0; + if (p.pf_hint) + { + asm volatile("createpolicy.fractional.L2::evict_last.b64 %0, 1.0;\n" : "=l"(pol)); + } + for (long long c = u; c < n0 + n1; c += kQkvUnits) + { + long long const a = c < n0 ? p.pf_a0 + c * kChunk : p.pf_a1 + (c - n0) * kChunk; + long long const rem = c < n0 ? p.pf_b0 - c * kChunk : p.pf_b1 - (c - n0) * kChunk; + unsigned int const n = static_cast(rem < kChunk ? rem : kChunk) & ~15u; + if (n == 0) + { + continue; + } + if (p.pf_hint) + { + asm volatile("cp.async.bulk.prefetch.L2.global.L2::cache_hint [%0], %1, %2;\n" ::"l"(a), "r"(n), "l"(pol) + : "memory"); + } + else + { + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;\n" ::"l"(a), "r"(n) : "memory"); + } + } +} + +__device__ __forceinline__ void op_prefetch_l2(Params const& p, int ou) +{ + int const rt = ou / kOpSplits; + int const kt0 = (ou % kOpSplits) * kOpTiles; + for (int i = 0; i < kOpTiles; ++i) + { + asm volatile("cp.async.bulk.prefetch.tensor.2d.L2.global [%0, {%1, %2}];\n" ::"l"(&p.wmap_o), + "r"((kt0 + i) * 128), "r"(rt * 128) + : "memory"); + } + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;\n" ::"l"( + p.wsf_o + (static_cast(rt) * kOpKTiles + kt0) * kQkvSfAtom), + "r"(kOpTiles * kQkvSfAtom) + : "memory"); +} + +// The weight tiles into TMEM as they land (warp 1, elected lane, operands warp-uniform). +template +__device__ __forceinline__ void op_stream(uint8_t* smem, uint32_t tm) +{ + uint32_t const ob = smem_u32(smem + SX::kOBar); + uint64_t const w0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + uint32_t const sf0 = smem_u32(smem + SX::kRingSf); +#pragma unroll + for (int i = 0; i < kOpTiles; ++i) + { + mbar_wait_cta(ob + 8 * i, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b(tm + kOA + (i * 4 + kb) * 8, w0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + tmem_cp_sf(tm + kOSfa + i * 4, sf0 + i * kQkvSfAtom); + } + umma_commit(ob + 8 * kOpTiles); // a_ready +} + +// The attention's MXFP8 rows of this unit's k-tiles (the heads [kt0, kt0 + 8)), Lamport-polled value by value: 0xFF +// is NaN in both E4M3 and UE8M0, which the merge never writes (see the merge). Every thread of the CTA, one item each. +template +__device__ __forceinline__ void op_act(Params const& p, uint8_t* smem, int ou, int tid, unsigned int calls) +{ + int const kt0 = (ou % kOpSplits) * kOpTiles; + uint8_t const* q8 = p.q8 + (calls & 1u) * kMaxTok * kOpK; + uint8_t const* sf = p.sf + (calls & 1u) * kOpSfBytes; + constexpr int kXItems = kOpTiles * 8 * kMaxTok; + constexpr int kSfItems = kOpTiles * (kQkvSfAtom / 16); + for (int q = tid; q < kXItems + kSfItems; q += kThreads) + { + if (q < kXItems) + { + int const t = q % kMaxTok; + int const c = (q / kMaxTok) % 8; + int const i = q / (kMaxTok * 8); + if (t < p.num_tokens) + { + uint4 v; + bool pending; + do + { + asm volatile("ld.relaxed.gpu.global.v4.b32 {%0, %1, %2, %3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(q8 + static_cast(t) * kOpK + (kt0 + i) * 128 + c * 16) + : "memory"); + uint32_t const w[4] = {v.x, v.y, v.z, v.w}; + pending = false; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + // a byte equal to 0xFF: (x & 0x7F7F7F7F) + 0x01010101 sets bit 7 of exactly the 0x7F/0xFF bytes + uint32_t const hit = (((w[j] & 0x7F7F7F7Fu) + 0x01010101u) & w[j] & 0x80808080u); + pending |= hit != 0u; + } + } while (pending); + *reinterpret_cast(smem + SX::kOXt + i * 1024 + t * 128 + ((c ^ t) * 16)) = v; + } + } + else if (q < kXItems + kSfItems) + { + int const j = q - kXItems; + int const i = j / (kQkvSfAtom / 16); + int const c = j % (kQkvSfAtom / 16); + uint8_t const* src = sf + (kt0 + i) * kQkvSfAtom + c * 16; + uint4 v; + if (c < p.num_tokens) + { + // Row c of the atom: its 4 block scales in bytes [16 c, 16 c + 4). + do + { + asm volatile("ld.relaxed.gpu.global.v4.b32 {%0, %1, %2, %3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(src) + : "memory"); + } while ((((v.x & 0x7F7F7F7Fu) + 0x01010101u) & v.x & 0x80808080u) != 0u); + } + else + { + v = *reinterpret_cast(src); + } + *reinterpret_cast(smem + SX::kOXsf + i * kQkvSfAtom + c * 16) = v; + } + } +} + +// The MMAs (warp 1, elected lane): v52's order (per k-tile: B scale copy, 4 MMAs; one accumulator). +template +__device__ __forceinline__ void op_mma(uint8_t* smem, uint32_t tm) +{ + uint32_t const ob = smem_u32(smem + SX::kOBar); + mbar_wait_cta(ob + 8 * kOpTiles, 0); // a_ready + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint64_t const b0 = umma_desc_sw128(smem_u32(smem + SX::kOXt)); + uint32_t const xsf0 = smem_u32(smem + SX::kOXsf); +#pragma unroll + for (int i = 0; i < kOpTiles; ++i) + { + uint32_t const sfb = tm + kOSfb + i * 4; + tmem_cp_sf(sfb, xsf0 + i * kQkvSfAtom); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + int const j = i * 4 + kb; + umma_mx(tm + kOAcc, tm + kOA + j * 8, b0 + ((i * 1024 + kb * 32) >> 4), qkv_idesc(kb), tm + kOSfa + i * 4, + sfb, j > 0); + } + } + umma_commit(ob + 8 * (kOpTiles + 1)); // acc_done +} + +// Warps 16..19: the pair's split-K reduce (owner rows [64 s, 64 s + 64)) and the bf16 rows of y. +template +__device__ __forceinline__ void op_epilogue(Params const& p, uint8_t* smem, uint32_t tm, int tid, int ou) +{ + int const warp = tid >> 5; + int const lane = tid & 31; + uint32_t const ob = smem_u32(smem + SX::kOBar); + // The store's kernel parameters, fetched before the wait: a first touch of their constant-bank line costs a miss. + __nv_bfloat16* y_out = p.y_out; + int num_tokens = p.num_tokens; + int ar_mode = p.ar_mode; + asm volatile("" : "+l"(y_out), "+r"(num_tokens), "+r"(ar_mode)); +#ifndef M3FA_OP_PARPRE +#define M3FA_OP_PARPRE 1 // read the all-reduce slot parities before the accumulator wait (a global load off the tail) +#endif + // The parity flags are stable here: the all-reduce that flips them last completed before this grid's PDL wait, + // and the next one reads them only after this kernel's pushes. + int par_pre[kMaxTok]; + { + int const row_pre = (ou / kOpSplits) * 128 + ((tid >> 5) & 3) * 32 + (tid & 31); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + par_pre[t] = 0; + if (M3FA_OP_PARPRE && ar_mode != 0 && t < num_tokens) + { + asm volatile("ld.volatile.global.s32 %0, [%1];\n" + : "=r"(par_pre[t]) + : "l"(p.ar_flags + t * 8 + row_pre / (kOpN / 8)) + : "memory"); + } + } + } + mbar_wait_epi(ob + 8 * (kOpTiles + 1), 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + M3FA_PROF_T(51, kComputeThreads); + int const r = (warp & 3) * 32 + lane; + float v[16]; + tmem_ld16(tm + (((warp & 3) * 32) << 16) + kOAcc, v); + int const s = ou % kOpSplits; + int const owner = r / kOpOwnRows; + int const local = r % kOpOwnRows; + uint32_t const red = ob + 8 * (kOpTiles + 2); + uint32_t cta_rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(cta_rank)); + if (owner != s) + { + uint32_t const peer = cta_rank + owner - s; // o_proj units count up with the CTA index + uint32_t const dst + = mapa_shared(smem_u32(smem + SX::kOMailbox) + ((s * kOpOwnRows + local) * kQkvMbStride) * 4, peer); + uint32_t const pbar = mapa_shared(red, peer); +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + asm volatile( + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [%0], {%1, %2, %3, %4}, [%5];\n" ::"r"( + dst + 16 * g), + "r"(__float_as_uint(v[4 * g])), "r"(__float_as_uint(v[4 * g + 1])), "r"(__float_as_uint(v[4 * g + 2])), + "r"(__float_as_uint(v[4 * g + 3])), "r"(pbar) + : "memory"); + } + } + else + { + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(red) + : "memory"); + } + M3FA_PROF_T(52, tid - lane); + float const* mb = reinterpret_cast(smem + SX::kOMailbox); + int const row = (ou / kOpSplits) * 128 + r; + float sums[kMaxTok]; +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + float4 part[kOpSplits]; +#pragma unroll + for (int q = 0; q < kOpSplits; ++q) + { + part[q] = *reinterpret_cast(mb + (q * kOpOwnRows + local) * kQkvMbStride + 4 * g); + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float sum = 0.0F; +#pragma unroll + for (int q = 0; q < kOpSplits; ++q) + { + float const pj = j == 0 ? part[q].x : j == 1 ? part[q].y : j == 2 ? part[q].z : part[q].w; + sum = sum + (q == s ? v[4 * g + j] : pj); + } + sums[4 * g + j] = sum; + } + } + M3FA_PROF_T(54, tid - lane); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + float const sum = sums[t]; +#ifndef M3FA_EXP +#define M3FA_EXP 0 +#endif + if (t < num_tokens) + { + if (M3FA_EXP == 1) + { + asm volatile( + "st.relaxed.gpu.global.u16 [%0], %1;\n" ::"l"(y_out + static_cast(t) * kOpN + row), + "h"(__bfloat16_as_ushort(__float2bfloat16_rn(sum))) + : "memory"); + } + else + { + y_out[static_cast(t) * kOpN + row] = __float2bfloat16_rn(sum); + } + } + if (ar_mode != 0) + { + // m3_mx_gemv's push into the all-reduce exchange: lane pairs hold row pairs; the even lane writes the + // bf16x2 word into this rank's slot of the (token, column slice) parity the AR's flags select. + float const hi = __shfl_xor_sync(0xffffffffu, sum, 1); + if ((lane & 1) == 0 && t < num_tokens) + { + int par; + if (M3FA_OP_PARPRE) + { + par = par_pre[t]; + } + else + { + asm volatile("ld.volatile.global.s32 %0, [%1];\n" + : "=r"(par) + : "l"(p.ar_flags + t * 8 + row / (kOpN / 8)) + : "memory"); + } + int64_t const off = static_cast(par & 1) * p.ar_maxt * p.ar_world * (kOpN / 2) + + (static_cast(t) * p.ar_world + p.ar_rank) * (kOpN / 2) + row / 2; + __nv_bfloat162 w2 = __floats2bfloat162_rn(sum, hi); + uint32_t word = *reinterpret_cast(&w2); + // -0.0 halves to +0.0: 0x80000000 marks an empty exchange word (as m3_mx_gemv's _bf16x2_word) + word = (word & 0xFFFFu) == 0x8000u ? (word & 0xFFFF0000u) : word; + word = (word >> 16) == 0x8000u ? (word & 0x0000FFFFu) : word; + asm volatile("st.global.b32 [%0], %1;\n" ::"l"(p.ar_ws + off), "r"(word) : "memory"); + } + } + } + M3FA_PROF_T(53, tid - lane); + } + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); +} + +// Packed-QKV element group [token][col * 128 + 4 * lane, +4): the split partials, polled until every one is stored, +// summed in split order and rounded to BF16. +__device__ __forceinline__ void qkv_load(Params const& p, int par, int tok, int col, int lane, float (&e)[4]) +{ + float const* base = p.part + par * kQkvPartStride + static_cast(tok) * kQkvN + col * kHeadDim + lane * 4; + uint4 pr[kQkvSplits]; + long long spins = 0; + bool ready; + do + { + ready = true; +#pragma unroll + for (int s = 0; s < kQkvSplits; ++s) + { + asm volatile("ld.relaxed.gpu.global.v4.b32 {%0, %1, %2, %3}, [%4];\n" + : "=r"(pr[s].x), "=r"(pr[s].y), "=r"(pr[s].z), "=r"(pr[s].w) + : "l"(base + static_cast(s) * kMaxTok * kQkvN) + : "memory"); + ready &= pr[s].x != kQkvSentinel && pr[s].y != kQkvSentinel && pr[s].z != kQkvSentinel + && pr[s].w != kQkvSentinel; + } + if (++spins > (1LL << 26)) + { + __trap(); + } + } while (!ready); + float s0 = __uint_as_float(pr[0].x), s1 = __uint_as_float(pr[0].y), s2 = __uint_as_float(pr[0].z), + s3 = __uint_as_float(pr[0].w); +#pragma unroll + for (int s = 1; s < kQkvSplits; ++s) + { + s0 += __uint_as_float(pr[s].x); + s1 += __uint_as_float(pr[s].y); + s2 += __uint_as_float(pr[s].z); + s3 += __uint_as_float(pr[s].w); + } + e[0] = __bfloat162float(__float2bfloat16_rn(s0)); + e[1] = __bfloat162float(__float2bfloat16_rn(s1)); + e[2] = __bfloat162float(__float2bfloat16_rn(s2)); + e[3] = __bfloat162float(__float2bfloat16_rn(s3)); +} + +// Every CTA once it has read the call parity and the partials it needs: the last one flips the parity. +__device__ __forceinline__ void qkv_done(Params const& p) +{ + if (atomicAdd(p.counters + 1, 1) == kQkvGrid - 1) + { + atomicExch(p.counters + 1, 0); + atomicXor(p.counters, 1); + } +} + +} // namespace + +namespace +{ + +// FQKV: the kernel also computes the QKV(+index) projection from the layer input (see qkv_mma) and runs on a fixed +// kQkvGrid-CTA grid; the first attn_clusters clusters then run the attention on the projection's output. +template +__global__ void __launch_bounds__(kThreads, 1) m3_fused_decode_attn_kernel(__grid_constant__ Params const p) +{ +#if MINIMAX_M3_ATTN_DEVICE_ENABLED + using S = SmemX; + constexpr int kRows = S::kRows; + constexpr int kTok = S::kTok; + constexpr int kCs = kBlocks / BPC; // cluster size + constexpr int kGroups = QLEN / QG; // clusters per request + constexpr uint32_t kPushBytes = (kHeadDim + 1) * 4; // one partial row: 128 outputs and the LSE + constexpr int kTasks = kHq * QG + 3 * QLEN; // producer tasks: 16 Q heads x QG, [K, V, index-K] x QLEN + constexpr int kMaxT = (kTasks + kWarps - 1) / kWarps; + constexpr int kChunks = BPC * kPage * 4; // 16-byte packed chunks per K (or V) tile set + constexpr int kKPer = (kChunks + kComputeThreads - 1) / kComputeThreads; // K chunks per thread [0, 512) + constexpr bool kRegCore = RegCore::kOn; // warp-MMA attention core (see RegCore) + // V tiles (raw copy before the PDL wait, dequantized during QK and the softmax): on the packed path every warp + // outside the softmax groups takes part (at M=1, warps 4..19), on the fused-projection path the producer warps. +#ifndef M3FA_RS2 +#define M3FA_RS2 1 // megakernel at one 16-row chunk: the softmax and push rows split over two warp groups (8 each) +#endif + constexpr bool kRs2 = FQKV && M3FA_RS2 && BPC * QG == 1 && !RegCore::kOn; + constexpr int kSoftmaxGroups = kRs2 ? 2 : (BPC * QG) < 4 ? (BPC * QG) : 4; +#ifndef M3FA_MEGA_VWIDE +#define M3FA_MEGA_VWIDE \ + 1 // megakernel attention CTAs (no projection unit): every warp outside the softmax copies and + // dequantizes V, as on the packed path (not only the four producer warps) +#endif + constexpr int kVStart = (kRegCore || (FQKV && !M3FA_MEGA_VWIDE)) ? kComputeThreads : 128 * kSoftmaxGroups; + constexpr int kVThreads = kThreads - kVStart; // V chunks: the warps outside the softmax groups + constexpr int kVPer = (kChunks + kVThreads - 1) / kVThreads; + constexpr int kPairs = BPC * QG; // (block, 16-row chunk) pairs of the softmax and push + constexpr int kPairsPerGroup = (kPairs + 3) / 4; // per group of four compute warps + // Named barrier 6: the softmax warps (P written) and the producer warps (V written) before the PV issue. + constexpr int kPvBarThreads = 128 * kSoftmaxGroups + kVThreads; + extern __shared__ __align__(128) uint8_t smem[]; + float* row_scale = reinterpret_cast(smem + S::kRowScale); + float* row_lse = reinterpret_cast(smem + S::kRowLse); + float* red_m = reinterpret_cast(smem + S::kRedM); + float* red_l = reinterpret_cast(smem + S::kRedL); + float* recv = reinterpret_cast(smem + S::kRecv); + uint32_t const mbar = smem_u32(smem + S::kMbar); + uint32_t const mma_bar = mbar + 8; + uint32_t* tmem_slot = reinterpret_cast(smem + S::kMbar + 16); + + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const cr = blockIdx.x % kCs; // cluster rank; owns blocks [cr * BPC, cr * BPC + BPC) + int const ci = blockIdx.x / kCs; + int const req = ci / kGroups; + int const tfirst = (ci % kGroups) * QG; // this cluster's query tokens [tfirst, tfirst + QG) of the request + int const tok0 = req * QLEN; + uint32_t const gbar = smem_u32(smem + S::kGBar); // FQKV: weight ring, A ready, accumulators done + int const qkv_u = FQKV ? qkv_unit() : -1; // FQKV: this CTA's projection unit, or -1 + + // ---------------- Prologue: nothing here depends on the producer kernel ---------------- +#if M3FA_TRIGGER_AT == 0 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (!FQKV) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + M3FA_PROF(0); +#ifdef M3FA_PROFILE + if (p.prof != nullptr && tid == 0) + { + uint32_t smid; + asm volatile("mov.u32 %0, %%smid;" : "=r"(smid)); + p.prof[blockIdx.x * 64 + 40] = smid; + } +#endif + if constexpr (FQKV) + { + // The projection weights stream into TMEM while the producer (the all-reduce) still runs. + if (warp < 2) + { + qkv_setup(p, smem, gbar, tmem_slot, warp, lane, qkv_u); + } + if (ci >= p.attn_clusters) + { + // A projection-only CTA. Every reduce barrier of the cluster is initialized before any peer pushes. + asm volatile("barrier.cluster.arrive.release.aligned;\nbarrier.cluster.wait.acquire.aligned;\n" :: + : "memory"); + int const op_u = static_cast(blockIdx.x) - kOpFirstCta; // this CTA's o_proj unit (>= 0 here) +#ifndef M3FA_OP_EARLY +#define M3FA_OP_EARLY 0 // 1: o_proj-only CTAs stream their weights from launch (0: after the PDL wait) +#endif + if (M3FA_OP_EARLY && qkv_u < 0 && warp == 0 && lane == 0) + { + op_issue(p, smem, op_u); // an o_proj-only CTA: its weights stream from launch on + } + if (M3FA_OP_L2PF && qkv_u >= 0 && warp == 0 && lane == 0) + { + op_prefetch_l2(p, op_u); // into L2 during the all-reduce, behind this CTA's QKV ring + } + qkv_stream(p, smem, gbar, tmem_slot, warp, lane, qkv_u); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); + if (M3FA_MEGA_TRIGGER == 2) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + // This CTA's call counter: read after the PDL wait (the previous call's CTA increments it at its end), and + // early, so its latency stays off the epilogue's critical path. + unsigned int const calls = p.flags[32 + blockIdx.x]; + M3FA_PROF(1); + if (!M3FA_OP_EARLY && qkv_u < 0 && warp == 0 && lane == 0) + { + op_issue(p, smem, op_u); + } + if (qkv_u >= 0) + { + int const par = 0; + qkv_mma(p, smem, gbar, tmem_slot, tid, qkv_u); + if (warp >= kComputeWarps) + { + qkv_epilogue(p, smem, gbar, tmem_slot, tid, qkv_u, par, calls); + } + if (warp == 0 && lane == 0) + { + // The ring (and the activation over it) is free once the QKV MMAs completed. + mbar_wait_cta(gbar + 16 * kQkvRing + 8, 0); + op_issue(p, smem, op_u); + if (p.pf_b0 > 0 || p.pf_b1 > 0) + { + window_prefetch_l2(p, qkv_u); + } + } + } + __syncthreads(); // the QKV accumulator reads are done: TMEM columns [0, 512) are the o_proj's + M3FA_PROF(2); + uint32_t const tm = *tmem_slot; + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if (elect_one()) + { + op_stream(smem, tm); + } + __syncwarp(); + } + op_act(p, smem, op_u, tid, calls); + asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory"); + M3FA_PROF_T(36, 0); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + // Late release: the dependents launch once every unit has its attention input (the rel_v4 rule). + if (M3FA_MEGA_TRIGGER < 2) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + __syncthreads(); + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if (elect_one()) + { + op_mma(smem, tm); + M3FA_PROF_T(50, threadIdx.x); + } + __syncwarp(); + } + if (warp >= kComputeWarps) + { + op_epilogue(p, smem, tm, tid, op_u); + } + M3FA_PROF_T(37, kComputeThreads); + __syncthreads(); + M3FA_PROF_T(55, 0); + if (M3FA_MEGA_TRIGGER == 3) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); // after this CTA's y rows / pushes + } + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + asm volatile("tcgen05.dealloc.cta_group::1.sync.aligned.b32 %0, %1;\n" ::"r"(*tmem_slot), "r"(kTmemAll) + : "memory"); + M3FA_PROF_T(56, 32); + } + if (tid == 0) + { + p.flags[32 + blockIdx.x] = calls + 1u; // a plain store: no load on the exit path + } + M3FA_PROF(39); +#ifndef M3FA_FINAL_CLBAR +#define M3FA_FINAL_CLBAR 0 +#endif + if (M3FA_FINAL_CLBAR) + { + // No peer may exit while its partials can still be pushed into it (DSMEM lifetime). + asm volatile("barrier.cluster.arrive.release.aligned;\nbarrier.cluster.wait.acquire.aligned;\n" :: + : "memory"); + } + M3FA_PROF(38); + return; + } + } +#ifndef M3FA_MEGA_ATTN_WAIT_FIRST +#define M3FA_MEGA_ATTN_WAIT_FIRST 0 // 1: the attention CTAs read nothing (metadata, caches) before their PDL wait +#endif +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (FQKV && M3FA_MEGA_ATTN_WAIT_FIRST) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + } +#endif + if (!FQKV && warp == kWarps - 1) // a producer warp: it has nothing else to do before the PDL wait + { + asm volatile("tcgen05.alloc.cta_group::1.sync.aligned.shared::cta.b32 [%0], %1;\n" ::"r"(smem_u32(tmem_slot)), + "r"(S::kTmemCols) + : "memory"); + asm volatile("tcgen05.relinquish_alloc_permit.cta_group::1.sync.aligned;\n" ::: "memory"); + } + int const seq_len = p.seq_lens[req]; + int page[BPC]; +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + int const b = cr * BPC + h; + page[h] = b < p.bt_cols ? p.block_table[static_cast(req) * p.bt_stride + b] : 0; + } + // New-token cache slots, for the tokens up to this cluster's last one. + int slot[QLEN]; +#pragma unroll + for (int t = 0; t < QLEN; ++t) + { + slot[t] = t < tfirst + QG ? p.out_cache_loc[tok0 + t] : -1; + } + // This warp's producer tasks and, for every lane that applies RoPE, the task token's cos/sin. + float cs[kMaxT][4], sn[kMaxT][4]; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + int const i = warp + kWarps * k; + int const t = i < kHq * QG ? tfirst + i / kHq : (i - kHq * QG) / 3; + if (i < kTasks && lane < 16 && t < QLEN) + { + int64_t const row = static_cast(p.position_ids[tok0 + t]) * kRotDim; +#pragma unroll + for (int e = 0; e < 4; ++e) + { + int const coefficient = (lane * 4 + e) & (kRotDim / 2 - 1); + cs[k][e] = p.rope[row + coefficient]; + sn[k][e] = p.rope[row + kRotDim / 2 + coefficient]; + } + } + else + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + cs[k][e] = 1.0F; + sn[k][e] = 0.0F; + } + } + } + float qw[4], kw[4], ikw[4]; + load_w4(qw, p.q_w, lane); + load_w4(kw, p.k_w, lane); + load_w4(ikw, p.ik_w, lane); + float const kqs = p.kv_quant_scale[1]; + float const vqs = p.kv_quant_scale[2]; + float const sl2e = (p.sm_scale * 1.4426950409f) * p.k_gs[0]; + float const vgs = p.v_gs[0]; + // Longest attended length among this cluster's tokens, and each block's rows within it. + int const kv_max = max(seq_len - QLEN + tfirst + QG, 0); + int nvh[BPC]; +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + nvh[h] = min(max(kv_max - (cr * BPC + h) * kPage, 0), kPage); + } + bool const any_valid = nvh[0] > 0; + if (tid == 32) + { + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;\n" ::"r"(mbar), "r"(1)); + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;\n" ::"r"(mma_bar), "r"(1)); + asm volatile( + "mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(mbar), "r"(kBlocks * QG * BPC * kPushBytes) + : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + } + // This thread's packed K / V chunks, held in registers until they are dequantized after the PDL wait. + RawChunk kraw[kKPer]; +#pragma unroll + for (int k = 0; k < kKPer; ++k) + { + int const q = tid + kComputeThreads * k; + kraw[k] = RawChunk{make_uint4(0u, 0u, 0u, 0u), 0u}; + int const h = q / (kPage * 4); + if (q < kChunks && tid < kComputeThreads && nvh[h] > 0) + { + kraw[k] = load_chunk(p.kv_data + static_cast(page[h]) * p.d_page, + p.kv_scale + static_cast(page[h]) * p.s_page, chunk_row(q % (kPage * 4)), + chunk_col(q % (kPage * 4)), false); + } + } + // V goes through cp.async into shared memory: an asynchronous copy does not hold the producer warps up the way + // an outstanding register load does, and they need V only after QK has been issued. + if (tid >= kVStart) + { +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + if (nvh[h] > 0) + { + uint8_t* dst = smem + S::kRawV + h * S::kRawVBlock; + uint8_t const* data = p.kv_data + static_cast(page[h]) * p.d_page + p.d_plane; + uint8_t const* scales = p.kv_scale + static_cast(page[h]) * p.s_page + p.s_plane; + for (int c = tid - kVStart; c < S::kRawVBlock / 16; c += kVThreads) + { + cp_async16(dst + c * 16, c < kPage * 4 ? data + c * 16 : scales + (c - kPage * 4) * 16); + } + } + } + } + asm volatile("cp.async.commit_group;\n" ::: "memory"); + if (warp == kWarps - 1) + { + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + // Every CTA's mbarrier is initialized before any peer pushes into it; the matching wait sits right before the + // push, so this barrier's latency hides behind the attention (with aliased partials it moves there). + if constexpr (!S::kRecvAlias) + { + asm volatile("barrier.cluster.arrive.relaxed.aligned;\n" ::: "memory"); + } + int kv_len[QG]; +#pragma unroll + for (int tl = 0; tl < QG; ++tl) + { + kv_len[tl] = max(seq_len - QLEN + tfirst + tl + 1, 0); + } + int const cta_tok0 = cr * kTok; // request position of this CTA's first block row + // New token t's row in this CTA's tiles (block * 128 + row), or -1: the CTA owns the block holding the token's + // position, and the token is attended by this cluster (t <= its last token). Every thread derives the same. + int new_row[QLEN]; +#pragma unroll + for (int t = 0; t < QLEN; ++t) + { + int const pos = seq_len - QLEN + t; + int const blk = min(pos >> 7, kBlocks - 1); + int const spage = slot[t] / kPage; + bool const mine = t < tfirst + QG && pos >= 0 && blk / BPC == cr && slot[t] >= 0 && spage < p.num_pages; + new_row[t] = mine && spage == page[blk % BPC] ? (blk % BPC) * kPage + (slot[t] & (kPage - 1)) : -1; + } + auto is_new_row = [&](int r) + { + bool hit = false; +#pragma unroll + for (int t = 0; t < QLEN; ++t) + { + hit |= new_row[t] == r; + } + return hit; + }; + // Cache writes of the new K/V/index-K are issued after this CTA has pushed its partials: only the fp16 tile rows + // are on the attention's path. kind 0: none, 1: NVFP4 data + scale, 2: FP8 index-K. + int wr_kind[kMaxT]; + uint8_t* wr_ptr[kMaxT]; + uint8_t* wr_sptr[kMaxT]; + uint32_t wr_val[kMaxT]; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + wr_kind[k] = 0; + wr_ptr[k] = nullptr; + wr_sptr[k] = nullptr; + wr_val[k] = 0u; + } + uint32_t tmem = 0; + auto dequant_k = [&]() + { + if (tid < kComputeThreads) + { +#pragma unroll + for (int k = 0; k < kKPer; ++k) + { + int const q = tid + kComputeThreads * k; + if (q < kChunks) + { + int const h = q / (kPage * 4); + int const row = chunk_row(q % (kPage * 4)); + if (!is_new_row(h * kPage + row)) + { + store_chunk_fp16( + smem + S::kK16 + h * S::kTileBytes, row, chunk_col(q % (kPage * 4)), kraw[k], row < nvh[h]); + } + } + } + } + }; + // The producer warps' V tiles (raw tiles from cp.async, dequantized to fp16), then their PV-barrier arrival. + auto dequant_v = [&]() + { +#ifdef M3FA_DBG_NO_VDQ + if constexpr (!kRegCore) + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + return; +#endif + asm volatile("cp.async.wait_all;\n" ::: "memory"); + asm volatile("bar.sync 4, %0;\n" ::"n"(kVThreads) : "memory"); +#pragma unroll + for (int k = 0; k < kVPer; ++k) + { + int const q = tid - kVStart + kVThreads * k; + if (q < kChunks) + { + int const h = q / (kPage * 4); + int const row = chunk_row(q % (kPage * 4)); + int const cc = chunk_col(q % (kPage * 4)); + if (!is_new_row(h * kPage + row)) + { + uint8_t const* rawv = smem + S::kRawV + h * S::kRawVBlock; + uint8_t const* sc = rawv + kPage * 64 + (row >> 2) * 32 + (row & 3); + RawChunk c; + c.data = *reinterpret_cast(rawv + row * 64 + cc * 16); + c.scales + = static_cast(sc[(cc * 2) * 4]) | (static_cast(sc[(cc * 2 + 1) * 4]) << 8); + store_chunk_fp16(smem + S::kV16 + h * S::kTileBytes, row, cc, c, row < nvh[h]); + } + } + } + M3FA_FENCE_PROXY_ASYNC(); // V tiles -> tensor core reads (ordered by the barrier before the PV issue) + if constexpr (!kRegCore) + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + M3FA_PROF_T(16, kComputeThreads); + M3FA_PROF_T(17, kThreads - 1); + }; + if constexpr (FQKV) + { + qkv_stream(p, smem, gbar, tmem_slot, warp, lane, qkv_u); + } + M3FA_PROF(11); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); + if (FQKV && M3FA_MEGA_TRIGGER == 2) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#if M3FA_TRIGGER_AT == 1 + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); +#endif +#endif + // FQKV (megakernel): this call's QKV rows sit in buffer (calls & 1), polled value by value below. + unsigned int const mega_calls = FQKV ? p.flags[32 + blockIdx.x] : 0u; + __nv_bfloat16 const* qkv_src = FQKV ? p.packed + (mega_calls & 1u) * kMaxTok * kQkvN : p.packed; + M3FA_PROF(1); + int const par = 0; + if constexpr (FQKV) + { + if (qkv_u >= 0) + { + qkv_mma(p, smem, gbar, tmem_slot, tid, qkv_u); + } + } + + if constexpr (FQKV) + { + if (qkv_u >= 0) + { + if (warp >= kComputeWarps) + { + qkv_epilogue(p, smem, gbar, tmem_slot, tid, qkv_u, par, mega_calls); + } + else + { + // The attention regions overlay the weight ring: every weight byte has gone on into TMEM (or, with + // the MMAs reading the ring, every MMA has completed) first. + mbar_wait_cta(gbar + 16 * kQkvRing + (kQkvASmem ? 8 : 0), 0); + } + } + if (any_valid) + { + dequant_k(); + } +#ifndef M3FA_MEGA_EARLYV +#define M3FA_MEGA_EARLYV 1 // attention CTAs dequantize V here, while the QKV rows are still on their way +#endif +#ifndef M3FA_MEGA_EARLYCLW +#define M3FA_MEGA_EARLYCLW 1 // and pass the cluster barrier (peers' merge mbarriers initialized) here too +#endif + if (M3FA_MEGA_EARLYV && any_valid && tid >= kVStart) + { + dequant_v(); + } + if constexpr (M3FA_MEGA_EARLYCLW && !S::kRecvAlias) + { + asm volatile("barrier.cluster.wait.aligned;\n" ::: "memory"); + } + M3FA_PROF(14); + } + + if (any_valid) // CTA-uniform + { + // ---------------- Producer ---------------- + // Q rows of this cluster's tokens in every CTA. The new K/V of every token up to this cluster's last one + // whose position falls in this CTA's blocks go straight into the fp16 tiles; the CTA writes the caches, and + // the index-K, only for its own tokens. + uint2 raw[kMaxT]; + float fq[FQKV ? kMaxT : 1][4]; + bool act[kMaxT]; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + int const i = warp + kWarps * k; + raw[k] = make_uint2(0u, 0u); + act[k] = false; + if (i < kTasks) + { + int t, col; + bool active = true; + if (i >= kHq * QG) + { + int const kv = i - kHq * QG; + t = kv / 3; + int const kind = kv % 3; + col = kind == 0 ? kColK : (kind == 1 ? kColV : kColIdxK); + int const pos = seq_len - QLEN + t; + active = t < tfirst + QG && pos >= 0 && min(pos >> 7, kBlocks - 1) / BPC == cr && slot[t] >= 0 + && slot[t] / kPage < p.num_pages && (kind != 2 || t >= tfirst); + } + else + { + t = tfirst + i / kHq; + col = i % kHq; + } + act[k] = active; + if (active) + { + uint2 const* src = reinterpret_cast( + qkv_src + static_cast(tok0 + t) * p.packed_stride + col * kHeadDim + lane * 4); + if constexpr (FQKV) + { + uint2 v; + bool pending; + do + { + asm volatile("ld.relaxed.gpu.global.v2.b32 {%0, %1}, [%2];\n" + : "=r"(v.x), "=r"(v.y) + : "l"(src) + : "memory"); + pending = (v.x & 0xFFFFu) == 0xFFFFu || (v.x >> 16) == 0xFFFFu || (v.y & 0xFFFFu) == 0xFFFFu + || (v.y >> 16) == 0xFFFFu; + } while (pending); + raw[k] = v; + } + else + { + raw[k] = *src; + } + } + } + } + M3FA_PROF(15); + // While the packed-QKV loads are in flight: dequantize this thread's K chunks to fp16 (exact: e2m1 x e4m3 + // fits fp16). Rows past the longest attended length are zeroed, as the Triton producer masks its loads; the + // new tokens' rows are written by their producer tasks instead. (Building the tiles before the PDL wait was + // measured slower: the cache loads issued at entry are still in flight there, so the wait moves later.) + if constexpr (!FQKV) + { + dequant_k(); + } +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + int const i = warp + kWarps * k; + if (i >= kTasks || !act[k]) + { + continue; + } + float e[4]; + { + float2 const a = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw[k].x)); + float2 const b = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw[k].y)); + e[0] = a.x; + e[1] = a.y; + e[2] = b.x; + e[3] = b.y; + } + if (i < kHq * QG) + { + norm_rope(e, qw, p.eps, cs[k], sn[k], lane); + __nv_fp8x2_e4m3 const lo(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const hi(make_float2(e[2], e[3])); + uint2 hv; + hv.x = e4m3x2_to_h2(lo.__x); + hv.y = e4m3x2_to_h2(hi.__x); + *reinterpret_cast(smem + S::kQ + kmaj_off(i, lane * 4)) = hv; // row i = token * 16 + head + continue; + } + int const t = (i - kHq * QG) / 3; + int const kind = (i - kHq * QG) % 3; + int const spage = slot[t] / kPage; + int const within = slot[t] & (kPage - 1); + if (kind == 2) + { + norm_rope(e, ikw, p.eps, cs[k], sn[k], lane); + __nv_fp8x2_e4m3 const lo(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const hi(make_float2(e[2], e[3])); + wr_kind[k] = 2; + wr_val[k] = static_cast(lo.__x) | (static_cast(hi.__x) << 16); + wr_ptr[k] = p.idx_k + static_cast(spage) * p.i_page + static_cast(within) * p.i_tok + + lane * 4; + continue; + } + bool const isV = kind == 1; + if (!isV) + { + norm_rope(e, kw, p.eps, cs[k], sn[k], lane); + } + uint8_t sbyte = 0; + uint16_t const fp4 = quantizeMinimaxM3Fp4x4(e, isV ? vqs : kqs, &sbyte); + if (t >= tfirst) + { + int const scale_col = lane / 4; + int const scale_within = isV ? (within / 4) * 32 + scale_col * 4 + within % 4 : within * 8 + scale_col; + wr_kind[k] = 1; + wr_val[k] = static_cast(fp4) | (static_cast(sbyte) << 16); + wr_ptr[k] = p.kv_data + static_cast(spage) * p.d_page + (isV ? p.d_plane : 0) + within * 64 + + lane * 2; + wr_sptr[k] = p.kv_scale + static_cast(spage) * p.s_page + (isV ? p.s_plane : 0) + scale_within; + } + if (new_row[t] >= 0) + { + // The attention reads this page: its fp16 tile row is the freshly quantized row, dequantized like + // the rest of the tile. + int const h = new_row[t] / kPage; + int const row = new_row[t] % kPage; + uint32_t const sc2 = e4m3_to_h2_splat(sbyte); + uint2 hv; + hv.x = hmul2_u(e2m1x2_to_h2(fp4 & 0xFFu), sc2); + hv.y = hmul2_u(e2m1x2_to_h2(fp4 >> 8), sc2); + *reinterpret_cast( + smem + (isV ? S::kV16 : S::kK16) + h * S::kTileBytes + kmaj_off(row, lane * 4)) + = hv; + } + } + // K tiles, Q rows and patched rows -> tensor core reads. Only threads that wrote fence: the fence also + // waits for the thread's outstanding loads, and the producer warps still have their V chunks in flight. + bool wrote = tid < kComputeThreads; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + wrote |= act[k]; + } + if (wrote) + { + M3FA_FENCE_PROXY_ASYNC(); + } + M3FA_PROF(13); + __syncthreads(); +#if M3FA_TRIGGER_AT == 2 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + // Every K/V byte this CTA reads has arrived (the V chunks sit in registers): release the dependents now so + // their weight streams do not queue ahead of the KV reads (the megakernel releases late, from its o_proj). + if constexpr (!FQKV || M3FA_MEGA_TRIGGER == 1) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + tmem = *tmem_slot; + M3FA_PROF(2); + // ---------------- S^T = K Q^T per block on the tensor core: TMEM lane = token, column = query row ---------- +#ifndef M3FA_MEGA_WARPMMA +#define M3FA_MEGA_WARPMMA 1 // attention MMAs issued by warp 0's elected lane with warp-uniform operands (not tid 0) +#endif + if (!kRegCore && (FQKV && M3FA_MEGA_WARPMMA ? warp == 0 : tid == 0)) + { + // Operands computed by the whole warp stay in uniform registers; one elected lane issues. + uint32_t const tm_u = tmem; + bool const issue = FQKV && M3FA_MEGA_WARPMMA ? elect_one() : true; + if (issue) + { +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + uint32_t const a0 = smem_u32(smem + S::kK16 + h * S::kTileBytes); + uint32_t const b0 = smem_u32(smem + S::kQ); +#pragma unroll + for (int ks = 0; ks < kHeadDim / 16; ++ks) + { + umma_f16(tm_u + h * kRows, umma_desc(a0 + ks * 256, 128, 2048), + umma_desc(b0 + ks * 256, 128, 2048), umma_idesc(kRows, 0, 0), ks > 0); + } + } + umma_commit(mma_bar); + } + if (FQKV && M3FA_MEGA_WARPMMA) + { + __syncwarp(); + } + } + if (tid >= kVStart && !(FQKV && M3FA_MEGA_EARLYV)) + { + // The producer warps dequantize V while QK and the softmax run. + dequant_v(); + } + if constexpr (kRegCore) + { + // ---------------- Register core: warp w of block h = w / kWpb owns 8 * kNt tokens for S and 8 * kNt + // output dims for O; m16n8k16 fp16 MMAs with fp32 accumulators, the softmax in the S fragments ---------- + constexpr int kWpb = kComputeWarps / BPC; // warps per block + constexpr int kNt = 16 / kWpb; // 8-token (S) or 8-dim (O) tiles per warp + float* redx_m = reinterpret_cast(smem + S::kRedX); + float* redx_l = redx_m + BPC * kRows * kWpb; + if (warp < kComputeWarps) + { + int const h = warp / kWpb; + int const wib = warp % kWpb; + int const g = lane >> 2; + int const q4 = lane & 3; + uint32_t const ktile = smem_u32(smem + S::kK16 + h * S::kTileBytes); + uint32_t const vtile = smem_u32(smem + S::kV16 + h * S::kTileBytes); + uint32_t const qrows = smem_u32(smem + S::kQ); + uint8_t* ptile = smem + S::kK16 + h * S::kTileBytes; // P over this block's K tile, after QK + auto block_bar = [&]() + { + if (h == 0) + { + asm volatile("bar.sync 14, %0;\n" ::"n"(kWpb * 32) : "memory"); + } + else + { + asm volatile("bar.sync 15, %0;\n" ::"n"(kWpb * 32) : "memory"); + } + }; + // S = Q K^T: rows (c, g / g + 8), tokens 8 tb + 2 q4 (+1). + float sv[QG][kNt][4]; +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + sv[c][nt][e] = 0.0F; + } + } + } +#pragma unroll + for (int ks = 0; ks < kHeadDim / 16; ++ks) + { + uint32_t bk[kNt][2]; +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + int const tb = wib * kNt + nt; + ldsm_x2(ktile + tb * 2048 + (ks * 2 + ((lane >> 3) & 1)) * 128 + (lane & 7) * 16, bk[nt]); + } +#pragma unroll + for (int c = 0; c < QG; ++c) + { + uint32_t aq[4]; + ldsm_x4( + qrows + (c * 2 + ((lane >> 3) & 1)) * 2048 + (ks * 2 + (lane >> 4)) * 128 + (lane & 7) * 16, + aq); +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + mma_f16_16816(sv[c][nt], aq, bk[nt]); + } + } + } + M3FA_PROF(7); + // Row maxima (log2 domain, masked as the producer masks), quad then across the block's warps. + float mrow[QG][2]; +#pragma unroll + for (int c = 0; c < QG; ++c) + { + float mx0 = kNegInf, mx1 = kNegInf; +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + int const tok = cta_tok0 + h * kPage + (wib * kNt + nt) * 8 + 2 * q4 + (e & 1); + bool const valid = tok < kv_max && tok < kv_len[c]; + float const v = valid ? sv[c][nt][e] * sl2e : kNegInf; + sv[c][nt][e] = v; + if (e < 2) + { + mx0 = fmaxf(mx0, v); + } + else + { + mx1 = fmaxf(mx1, v); + } + } + } + mx0 = fmaxf(mx0, __shfl_xor_sync(0xffffffffu, mx0, 1)); + mx0 = fmaxf(mx0, __shfl_xor_sync(0xffffffffu, mx0, 2)); + mx1 = fmaxf(mx1, __shfl_xor_sync(0xffffffffu, mx1, 1)); + mx1 = fmaxf(mx1, __shfl_xor_sync(0xffffffffu, mx1, 2)); + if (q4 == 0) + { + redx_m[(h * kRows + c * 16 + g) * kWpb + wib] = mx0; + redx_m[(h * kRows + c * 16 + g + 8) * kWpb + wib] = mx1; + } + } + block_bar(); // every warp of the block is past QK (P may overwrite the K tile) and wrote its maxima + M3FA_PROF(8); +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int half = 0; half < 2; ++half) + { + float const* rm = redx_m + (h * kRows + c * 16 + g + 8 * half) * kWpb; + float m = kNegInf; +#pragma unroll + for (int w4 = 0; w4 < kWpb; w4 += 4) + { + float4 const v = *reinterpret_cast(rm + w4); + m = fmaxf(m, fmaxf(fmaxf(v.x, v.y), fmaxf(v.z, v.w))); + } + mrow[c][half] = m; + } + // P = exp2(s - m) as fp16 into [row][token] core matrices; the fp32 values feed the row sums. + float l0 = 0.0F, l1 = 0.0F; +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + float a[4]; +#pragma unroll + for (int e = 0; e < 4; ++e) + { + float const m = mrow[c][e >> 1]; + a[e] = m == kNegInf ? 0.0F : ex2(sv[c][nt][e] - m); + } + l0 += a[0] + a[1]; + l1 += a[2] + a[3]; + int const t0 = (wib * kNt + nt) * 8 + 2 * q4; + int const r0 = c * 16 + g; + __half2 const p0 = __floats2half2_rn(a[0], a[1]); + __half2 const p1 = __floats2half2_rn(a[2], a[3]); + *reinterpret_cast<__half2*>( + ptile + (r0 >> 3) * 2048 + (t0 >> 3) * 128 + (r0 & 7) * 16 + (t0 & 7) * 2) + = p0; + *reinterpret_cast<__half2*>( + ptile + ((r0 + 8) >> 3) * 2048 + (t0 >> 3) * 128 + (r0 & 7) * 16 + (t0 & 7) * 2) + = p1; + } + l0 += __shfl_xor_sync(0xffffffffu, l0, 1); + l0 += __shfl_xor_sync(0xffffffffu, l0, 2); + l1 += __shfl_xor_sync(0xffffffffu, l1, 1); + l1 += __shfl_xor_sync(0xffffffffu, l1, 2); + if (q4 == 0) + { + redx_l[(h * kRows + c * 16 + g) * kWpb + wib] = l0; + redx_l[(h * kRows + c * 16 + g + 8) * kWpb + wib] = l1; + } + } + M3FA_PROF_T(19, 0); + } + __syncthreads(); // P, the V tiles (producer warps) and the row sums visible + M3FA_PROF(9); + if (warp < kComputeWarps) + { + int const h = warp / kWpb; + int const wib = warp % kWpb; + int const g = lane >> 2; + int const q4 = lane & 3; + uint32_t const vtile = smem_u32(smem + S::kV16 + h * S::kTileBytes); + uint32_t const ptile = smem_u32(smem + S::kK16 + h * S::kTileBytes); + float* obuf = reinterpret_cast(smem + S::kK16 + h * S::kTileBytes + S::kObufOff); + // O = P V: rows (c, g / g + 8), dims 8 dt + 2 q4 (+1). + float ov[QG][kNt][4]; +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + ov[c][nt][e] = 0.0F; + } + } + } +#pragma unroll + for (int ks = 0; ks < kPage / 16; ++ks) + { + uint32_t bv[kNt][2]; + int const tok = ks * 16 + ((lane >> 3) & 1) * 8 + (lane & 7); +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + int const dt = wib * kNt + nt; + ldsm_x2_trans(vtile + (tok >> 3) * 2048 + dt * 128 + (tok & 7) * 16, bv[nt]); + } +#pragma unroll + for (int c = 0; c < QG; ++c) + { + uint32_t ap[4]; + ldsm_x4( + ptile + (c * 2 + ((lane >> 3) & 1)) * 2048 + (ks * 2 + (lane >> 4)) * 128 + (lane & 7) * 16, + ap); +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + mma_f16_16816(ov[c][nt], ap, bv[nt]); + } + } + } +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + int const d = (wib * kNt + nt) * 8 + 2 * q4; + int const r0 = c * 16 + g; + *reinterpret_cast(obuf + r0 * kOPitch + d) = make_float2(ov[c][nt][0], ov[c][nt][1]); + *reinterpret_cast(obuf + (r0 + 8) * kOPitch + d) + = make_float2(ov[c][nt][2], ov[c][nt][3]); + } + } + } + else if (tid - kComputeThreads < BPC * kRows) + { + // Row statistics of every (block, row): LSE and the PV scale, as the producer derives them. + int const r = tid - kComputeThreads; + float const* rm = redx_m + r * kWpb; + float const* rl = redx_l + r * kWpb; + float m = kNegInf; + float l = 0.0F; +#pragma unroll + for (int w4 = 0; w4 < kWpb; w4 += 4) + { + float4 const vm = *reinterpret_cast(rm + w4); + float4 const vl = *reinterpret_cast(rl + w4); + m = fmaxf(m, fmaxf(fmaxf(vm.x, vm.y), fmaxf(vm.z, vm.w))); + l += (vl.x + vl.y) + (vl.z + vl.w); + } + float const lse = l > 0.0F ? m + lg2(l) : kNegInf; + float const scale = lse > kNegInf ? ex2(m - lse) : 0.0F; + row_scale[r] = scale * vgs; + row_lse[r] = lse; + } + __syncthreads(); // O partials and row statistics visible to the push + } + else + { + if constexpr (kRs2) + { + // ---------------- Split softmax: warps 0..3 rows 0..7, warps 4..7 rows 8..15 (thread = block token) + // -------- + if (warp < 8) + { + int const rg = warp / 4; + int const wq = warp % 4; + int const tk = wq * 32 + lane; + auto rg_bar = [&]() + { + if (rg == 0) + { + asm volatile("bar.sync 10, 128;\n" ::: "memory"); + } + else + { + asm volatile("bar.sync 11, 128;\n" ::: "memory"); + } + }; + int const rs8 = ((lane >> 4) & 1) * 4 + ((lane >> 3) & 1) * 2 + ((lane >> 2) & 1); + mbar_wait_cta(mma_bar, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + M3FA_PROF(7); + float v[8]; + tmem_ld8(tmem + ((wq * 32) << 16) + 8 * rg, v); + bool const valid = cta_tok0 + tk < kv_max && cta_tok0 + tk < kv_len[0]; + float a[8]; +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + v[j2] = valid ? v[j2] * sl2e : kNegInf; + a[j2] = v[j2]; + } + float const mw = warp_reduce_scatter8(a, lane); + if ((lane & 3) == 0) + { + red_m[(8 * rg + rs8) * 4 + wq] = mw; + } + rg_bar(); + M3FA_PROF(8); + uint8_t* pdst = smem + S::kP + (tk >> 3) * 128 + (tk & 7) * 16; + float mrow = kNegInf; + if (lane < 8) + { + float4 const mq = *reinterpret_cast(red_m + (8 * rg + lane) * 4); + mrow = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + } + float pa[8]; +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + float const m = __shfl_sync(0xffffffffu, mrow, j2); + pa[j2] = m == kNegInf ? 0.0F : ex2(v[j2] - m); + } + if (tid == 0) + { + M3FA_CLK(41); + } + uint32_t pw[4]; +#pragma unroll + for (int j2 = 0; j2 < 4; ++j2) + { + __half2 const hp = __floats2half2_rn(pa[2 * j2], pa[2 * j2 + 1]); + pw[j2] = *reinterpret_cast(&hp); + } + *reinterpret_cast(pdst + rg * 2048) = make_uint4(pw[0], pw[1], pw[2], pw[3]); + M3FA_PROF_T(18, 0); + if (tid == 0) + { + M3FA_CLK(42); + } + M3FA_FENCE_PROXY_ASYNC(); // P -> tensor core reads + if (tid == 0) + { + M3FA_CLK(43); + } + if (warp == 0) + { + asm volatile("bar.sync 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + if (tid == 0) + { + M3FA_CLK(44); + } + } + else + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + if (M3FA_MEGA_WARPMMA ? warp == 0 : tid == 0) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm_u = tmem; + uint32_t const a0 = smem_u32(smem + S::kV16); + uint32_t const b0 = smem_u32(smem + S::kP); + if (M3FA_MEGA_WARPMMA ? elect_one() : true) + { +#pragma unroll + for (int ks = 0; ks < kPage / 16; ++ks) + { + umma_f16(tm_u + BPC * kRows, umma_desc(a0 + ks * 4096, 2048, 128), + umma_desc(b0 + ks * 256, 128, 2048), umma_idesc(kRows, 1, 1), ks > 0); + } + umma_commit(mma_bar); + } + if (M3FA_MEGA_WARPMMA) + { + __syncwarp(); + } + } + M3FA_PROF_T(19, 0); + float const lw = warp_reduce_scatter8(pa, lane); + if ((lane & 3) == 0) + { + red_l[(8 * rg + rs8) * 4 + wq] = lw; + } + rg_bar(); + M3FA_PROF_T(20, 0); + if (wq == 0 && lane < 8) + { + int const r = 8 * rg + lane; + float4 const mq = *reinterpret_cast(red_m + r * 4); + float4 const lq = *reinterpret_cast(red_l + r * 4); + float const m = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + float const l = (lq.x + lq.y) + (lq.z + lq.w); + float const lse = l > 0.0F ? m + lg2(l) : kNegInf; + float const scale = lse > kNegInf ? ex2(m - lse) : 0.0F; + row_scale[r] = scale * vgs; + row_lse[r] = lse; + } + } + __syncthreads(); // row scales and LSEs visible (PV was issued above) + M3FA_PROF(9); + if (warp < 8) + { + mbar_wait_cta(mma_bar, 1); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + } + } + else + { + // ---------------- Softmax: thread = block token; the (block, 16-row chunk) pairs are spread over the + // four groups of four compute warps (each group covers the 128 TMEM lanes = block tokens) + // ---------------- + if (warp < kComputeWarps && warp / 4 < kPairs) + { + int const g = warp / 4; + int const wq = warp % 4; + int const tk = wq * 32 + lane; + auto group_bar = [&]() + { + if (g == 0) + { + asm volatile("bar.sync 10, 128;\n" ::: "memory"); + } + else if (g == 1) + { + asm volatile("bar.sync 11, 128;\n" ::: "memory"); + } + else if (g == 2) + { + asm volatile("bar.sync 12, 128;\n" ::: "memory"); + } + else + { + asm volatile("bar.sync 13, 128;\n" ::: "memory"); + } + }; + // Row r(lane) this lane holds after the reduce-scatter: bits 4..1 of the lane. + int const rs_row + = ((lane >> 4) & 1) * 8 + ((lane >> 3) & 1) * 4 + ((lane >> 2) & 1) * 2 + ((lane >> 1) & 1); + mbar_wait_cta(mma_bar, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + M3FA_PROF(7); + float v[kPairsPerGroup][16]; +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + tmem_ld16(tmem + ((wq * 32) << 16) + h * kRows + c * 16, v[k]); + bool const valid + = cta_tok0 + h * kPage + tk < kv_max && cta_tok0 + h * kPage + tk < kv_len[c]; + float a[16]; +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[k][j] = valid ? v[k][j] * sl2e : kNegInf; + a[j] = v[k][j]; + } + float const mw = warp_reduce_scatter16(a, lane); + if ((lane & 1) == 0) + { + red_m[((h * QG + c) * 16 + rs_row) * 4 + wq] = mw; + } + } + } + group_bar(); + M3FA_PROF(8); + float pa[kPairsPerGroup][16]; +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + uint8_t* pdst = smem + S::kP + h * kRows * kPage * 2 + (tk >> 3) * 128 + (tk & 7) * 16; + // Row maxima: lane j (< 16) reduces row j's four warp partials, then broadcasts it. + float mrow = kNegInf; + if (lane < 16) + { + float4 const mq + = *reinterpret_cast(red_m + ((h * QG + c) * 16 + lane) * 4); + mrow = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + } +#pragma unroll + for (int j = 0; j < 16; ++j) + { + float const m = __shfl_sync(0xffffffffu, mrow, j); + pa[k][j] = m == kNegInf ? 0.0F : ex2(v[k][j] - m); + } + if (tid == 0) + { + M3FA_CLK(41); + } + uint32_t pw[8]; +#pragma unroll + for (int j = 0; j < 8; ++j) + { + __half2 const hp = __floats2half2_rn(pa[k][2 * j], pa[k][2 * j + 1]); + pw[j] = *reinterpret_cast(&hp); + } + // P^T as the MN-major B operand: 8 query rows of one token are one 16-byte core-matrix row. + *reinterpret_cast(pdst + (2 * c) * 2048) = make_uint4(pw[0], pw[1], pw[2], pw[3]); + *reinterpret_cast(pdst + (2 * c + 1) * 2048) + = make_uint4(pw[4], pw[5], pw[6], pw[7]); + M3FA_PROF_T(18, 0); + } + } + if (tid == 0) + { + M3FA_CLK(42); + } + M3FA_FENCE_PROXY_ASYNC(); // P -> tensor core reads + if (tid == 0) + { + M3FA_CLK(43); + } + // PV starts as soon as P and the V tiles are in: thread 0 waits for the softmax warps and the + // producer warps (V); the row sums and statistics below overlap the MMAs. + if (warp == 0) + { + asm volatile("bar.sync 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + else + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + if (tid == 0) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + uint32_t const a0 = smem_u32(smem + S::kV16 + h * S::kTileBytes); + uint32_t const b0 = smem_u32(smem + S::kP + h * kRows * kPage * 2); +#pragma unroll + for (int ks = 0; ks < kPage / 16; ++ks) + { + umma_f16(tmem + (BPC + h) * kRows, umma_desc(a0 + ks * 4096, 2048, 128), + umma_desc(b0 + ks * 256, 128, 2048), umma_idesc(kRows, 1, 1), ks > 0); + } + } + umma_commit(mma_bar); + } + M3FA_PROF_T(19, 0); +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + float const lw = warp_reduce_scatter16(pa[k], lane); + if ((lane & 1) == 0) + { + red_l[((h * QG + c) * 16 + rs_row) * 4 + wq] = lw; + } + } + } + group_bar(); + M3FA_PROF_T(20, 0); + if (wq == 0 && lane < 16) + { +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + int const r = (h * QG + c) * 16 + lane; + float4 const mq = *reinterpret_cast(red_m + r * 4); + float4 const lq = *reinterpret_cast(red_l + r * 4); + float const m = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + float const l = (lq.x + lq.y) + (lq.z + lq.w); + float const lse = l > 0.0F ? m + lg2(l) : kNegInf; + float const scale = lse > kNegInf ? ex2(m - lse) : 0.0F; + row_scale[h * kRows + c * 16 + lane] = scale * vgs; + row_lse[h * kRows + c * 16 + lane] = lse; + } + } + } + } + __syncthreads(); // row scales and LSEs visible (PV was issued above) + M3FA_PROF(9); + if (warp < kComputeWarps && warp / 4 < kPairs) + { + mbar_wait_cta(mma_bar, 1); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + } + } + } + M3FA_PROF(3); + } + else + { + asm volatile("cp.async.wait_all;\n" ::: "memory"); +#if M3FA_TRIGGER_AT == 2 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (!FQKV || M3FA_MEGA_TRIGGER == 1) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + for (int r = tid; r < BPC * kRows; r += kThreads) + { + row_lse[r] = kNegInf; + row_scale[r] = 0.0F; + } + __syncthreads(); + } + + // ---------------- Push every partial row to the CTA that merges its head (CTA r merges heads r*BPC..) -------- + if constexpr (S::kRecvAlias) + { + // Peers' partials overlay this CTA's tiles: every CTA's PV MMAs have completed. + asm volatile("barrier.cluster.arrive.aligned;\n" ::: "memory"); + } + if (!(FQKV && M3FA_MEGA_EARLYCLW && !S::kRecvAlias)) + { + asm volatile("barrier.cluster.wait.aligned;\n" ::: "memory"); // peers' mbarriers are initialized + } + M3FA_PROF(4); + uint32_t const recv_base = smem_u32(recv); + if (kRs2 && warp < 8) + { + // Split push: warp group rg pushes rows 8 rg .. 8 rg + 7 (row j goes to CTA j, which merges head j). + int const rg = warp / 4; + int const d = (warp % 4) * 32 + lane; + float v[8]; + if (any_valid) + { + tmem_ld8(tmem + ((warp % 4) * 32 << 16) + BPC * kRows + 8 * rg, v); + } + else + { +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + v[j2] = 0.0F; + } + } +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + v[j2] *= row_scale[8 * rg + j2]; + } +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + uint32_t const dst_cta = 8 * rg + j2; + uint32_t const row_addr = recv_base + static_cast((cr * kOPitch + d) * 4); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_shared(row_addr, dst_cta)), + "r"(__float_as_uint(v[j2])), "r"(mapa_shared(mbar, dst_cta)) + : "memory"); + } + } + else if (!kRs2 && warp < kComputeWarps && warp / 4 < kPairs) + { + int const g = warp / 4; + int const d = (warp % 4) * 32 + lane; // output dim +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + int const chunk = cr * BPC + h; + float v[16]; + if (any_valid && kRegCore) + { + float const* obuf + = reinterpret_cast(smem + S::kK16 + h * S::kTileBytes + S::kObufOff); +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[j] = obuf[(c * 16 + j) * kOPitch + d]; + } + } + else if (any_valid) + { + tmem_ld16(tmem + ((warp % 4) * 32 << 16) + (BPC + h) * kRows + c * 16, v); + } + else + { +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[j] = 0.0F; + } + } +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[j] *= row_scale[h * kRows + c * 16 + j]; + } + // One scalar store per row: a vector store's lanes would address several destination CTAs at once, + // which measured several times slower. +#pragma unroll + for (int j = 0; j < 16; ++j) + { + uint32_t const dst_cta = j / BPC; + uint32_t const row_addr + = recv_base + static_cast((((chunk * QG + c) * BPC + j % BPC) * kOPitch + d) * 4); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_shared(row_addr, dst_cta)), + "r"(__float_as_uint(v[j])), "r"(mapa_shared(mbar, dst_cta)) + : "memory"); + } + } + } + } + else if (tid >= kComputeThreads) + { + // Every partial row's LSE (BPC * kRows of them; more than the producer threads for the widest geometries). + for (int r = tid - kComputeThreads; r < BPC * kRows; r += kThreads - kComputeThreads) + { + int const h = r / kRows; + int const c = (r % kRows) / 16; + int const j = r % 16; + uint32_t const dst_cta = j / BPC; + uint32_t const row_addr + = recv_base + static_cast((((cr * BPC + h) * QG + c) * BPC + j % BPC) * kOPitch * 4); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_shared(row_addr + kHeadDim * 4, dst_cta)), + "r"(__float_as_uint(row_lse[r])), "r"(mapa_shared(mbar, dst_cta)) + : "memory"); + } + } + M3FA_PROF(10); +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + if (wr_kind[k] == 1) + { + *reinterpret_cast(wr_ptr[k]) = static_cast(wr_val[k] & 0xFFFFu); + if (lane % 4 == 0) + { + *wr_sptr[k] = static_cast(wr_val[k] >> 16); + } + } + else if (wr_kind[k] == 2) + { + *reinterpret_cast(wr_ptr[k]) = wr_val[k]; + } + } + { + uint32_t ready = 0; +#ifdef M3FA_DBG_HANG + long long spins = 0; +#endif + while (!ready) + { +#ifdef M3FA_DBG_HANG + if (++spins == (1LL << 24)) + { + printf("M3FA hang: block %d thread %d merge recv\n", blockIdx.x, threadIdx.x); + } +#endif + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(mbar) + : "memory"); + } + } + M3FA_PROF(5); + + // ---------------- Merge heads [cr * BPC, cr * BPC + BPC) (split-K merge kernel's LSE weighting) ---------------- + constexpr int kMergeRows = QG * BPC; // (token, local head) pairs, 16 chunk weights each + // Wide geometries: chunk weights once per merged row (into the softmax scratch, free by now), then the weighted + // sums. Narrow ones recompute the weights per output element: no extra barrier on the critical path. + constexpr bool kRowWeights = kMergeRows >= 4; + float* wmerge = red_m; // [kMergeRows][kBlocks]; red_m holds BPC * kRows * 4 = kMergeRows * 64 floats + [[maybe_unused]] auto chunk_weights = [&](int mr, float(&lc)[kBlocks]) + { + float lmax = kNegInf; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + lc[c] = recv[(c * kMergeRows + mr) * kOPitch + kHeadDim]; + lmax = fmaxf(lmax, lc[c]); + } + lmax = lmax == kNegInf ? 0.0F : lmax; + float denom = 0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + lc[c] = ex2(lc[c] - lmax); // now the chunk weight; 0 for an empty chunk + denom += lc[c]; + } + denom = denom > 0.0F ? denom : 1.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + // -0.0 marks an empty chunk (skipped below, as the merge kernel skips zero weights). + lc[c] = lc[c] != 0.0F ? __fdividef(lc[c], denom) : -0.0F; + } + }; + if constexpr (kRowWeights) + { + for (int mr = tid; mr < kMergeRows; mr += kThreads) + { + float lc[kBlocks]; + chunk_weights(mr, lc); +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + wmerge[mr * kBlocks + c] = lc[c]; + } + } + __syncthreads(); + } +#pragma unroll + for (int idx = tid; idx < kMergeRows * kHeadDim; idx += kThreads) + { + int const mr = idx >> 7; + int const d = idx & (kHeadDim - 1); + float wn[kBlocks]; + if constexpr (kRowWeights) + { +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + wn[c] = wmerge[mr * kBlocks + c]; + } + } + else + { +#ifndef M3FA_MEGA_MERGE_LOCAL +#define M3FA_MEGA_MERGE_LOCAL \ + 0 // 1 (measured +0.6 us slower): every merging thread derives the 16 chunk weights itself (broadcast shared loads, + // independent exps): no shuffle chain; the same arithmetic as chunk_weights +#endif + if constexpr (M3FA_MEGA_MERGE_LOCAL != 0) + { + chunk_weights(mr, wn); + } + else + { + // Narrow geometries: the warp (32 dims of one merged row) derives the 16 chunk weights together, lane c + // holding chunk c's, and broadcasts them. + float lc = (lane < kBlocks) ? recv[(lane * kMergeRows + mr) * kOPitch + kHeadDim] : kNegInf; + float lmax = lc; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + lmax = fmaxf(lmax, __shfl_xor_sync(0xffffffffu, lmax, o)); + } + lmax = lmax == kNegInf ? 0.0F : lmax; + float const wc = lane < kBlocks ? ex2(lc - lmax) : 0.0F; + // The denominator in chunk order from 0.0, as chunk_weights (and the merge kernel) add it: a butterfly + // reduction rounds differently in the last bit. + float denom = 0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + denom += __shfl_sync(0xffffffffu, wc, c); + } + denom = denom > 0.0F ? denom : 1.0F; + float const wl = wc != 0.0F ? __fdividef(wc, denom) : -0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + wn[c] = __shfl_sync(0xffffffffu, wl, c); + } + } + } + if (tid == 0) + { + M3FA_CLK(45); + } + float acc = 0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + float const prod = recv[(c * kMergeRows + mr) * kOPitch + d] * wn[c]; + acc = __float_as_uint(wn[c]) != 0x80000000u ? acc + prod : acc; + } + if (tid == 0) + { + M3FA_CLK(46); + } + int const tl = mr / BPC; + int const head = cr * BPC + mr % BPC; + __nv_bfloat16 const ob = __float2bfloat16_rn(acc); + int const out_row = tok0 + tfirst + tl; + p.out[static_cast(out_row) * p.out_stride + head * kHeadDim + d] = ob; + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + { + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, mask)); + } + float const sfv = amax * rcp_approx_ftz(448.0F); + int const bits = __float_as_int(sfv); + int ex = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0 ? 1 : 0); + ex = min(ex, 254); + float scale = __int_as_float((254 - ex) << 23); + scale = amax != 0.0F ? scale : 0.0F; + __nv_fp8_e4m3 const q8v(x * scale); + int64_t const q8off = static_cast(out_row) * (p.num_sf_k_tiles * 128) + head * kHeadDim + d; + if constexpr (FQKV) + { + // Megakernel hand-off to the o_proj units (Lamport): this call's parity buffer, the other one reset to + // 0xFF. 0xFF is never written: a NaN code is canonicalized to 0x7F; the UE8M0 code is at most 254. + uint8_t const code = q8v.__x == 0xFFu ? static_cast(0x7Fu) : q8v.__x; + int64_t const cur = (mega_calls & 1u) * kMaxTok * kOpK; + int64_t const nxt = ((mega_calls & 1u) ^ 1u) * kMaxTok * kOpK; + asm volatile( + "st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.q8 + cur + q8off), "h"(static_cast(code)) + : "memory"); + asm volatile( + "st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.q8 + nxt + q8off), "h"(static_cast(0xFFu)) + : "memory"); + } + else + { + p.q8[q8off] = q8v.__x; + } + if (lane == 0) + { + int const k_sf = head * (kHeadDim / 32) + (d >> 5); + int64_t const off = static_cast(out_row / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (out_row % 32) * 16 + ((out_row % 128) / 32) * 4 + k_sf % 4; + if constexpr (FQKV) + { + int64_t const cur = (mega_calls & 1u) * kOpSfBytes; + int64_t const nxt = ((mega_calls & 1u) ^ 1u) * kOpSfBytes; + asm volatile( + "st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.sf + cur + off), "h"(static_cast(ex)) + : "memory"); + asm volatile("st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.sf + nxt + off), + "h"(static_cast(0xFFu)) + : "memory"); + } + else + { + p.sf[off] = static_cast(ex); + } + } + } + } + M3FA_PROF(60); + if (any_valid) + { + // Every TMEM read completed above (tcgen05.wait::ld); release the columns. + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + __syncthreads(); + if (FQKV && tid == 0) + { + p.flags[32 + blockIdx.x] = mega_calls + 1u; + } + M3FA_PROF(61); + if (warp == kWarps - 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + asm volatile("tcgen05.dealloc.cta_group::1.sync.aligned.b32 %0, %1;\n" ::"r"(*tmem_slot), "r"(S::kTmemCols) + : "memory"); + } + M3FA_PROF(6); +#else + __trap(); +#endif // MINIMAX_M3_ATTN_DEVICE_ENABLED +} + +template +cudaError_t launch_impl(Params const& prm, int num_reqs, bool pdl, cudaStream_t stream, int* max_clusters) +{ + auto kernel = m3_fused_decode_attn_kernel; + constexpr int smem = SmemX::kBytes; + constexpr int kCs = kBlocks / BPC; + static bool const configured = [&]() + { + bool ok = cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, smem) == cudaSuccess; + ok = ok + && (kCs <= 8 + || cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1) == cudaSuccess); + if (!ok) + { + cudaGetLastError(); + } + return ok; + }(); + if (!configured) + { + return cudaErrorInvalidConfiguration; + } + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(FQKV ? kQkvGrid : kCs * (QLEN / QG) * num_reqs); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeClusterDimension; + attr[0].val.clusterDim.x = kCs; + attr[0].val.clusterDim.y = 1; + attr[0].val.clusterDim.z = 1; + attr[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[1].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + if (max_clusters != nullptr) + { + cfg.numAttrs = 1; + cudaError_t const err = cudaOccupancyMaxActiveClusters(max_clusters, kernel, &cfg); + if (err != cudaSuccess) + { + cudaGetLastError(); + } + return err; + } + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +// The one geometry the target launches: one request, one query token, one 16-CTA attention cluster. +cudaError_t dispatch(Params const& prm, int qlen, int qg, int bpc, int num_reqs, bool pdl, cudaStream_t stream) +{ + if (qlen == 1 && qg == 1 && bpc == 1) + { + return launch_impl<1, 1, 1, true, true>(prm, num_reqs, pdl, stream, nullptr); + } + return cudaErrorInvalidValue; +} + +static_assert(kQkvGrid == minimax_m3_attn_mega::kGrid && kMaxTok == minimax_m3_attn_mega::kMaxTokens + && kQkvN == minimax_m3_attn_mega::kQkvN && kQkvK == minimax_m3_attn_mega::kQkvK + && kOpN == minimax_m3_attn_mega::kOpN && kOpK == minimax_m3_attn_mega::kOpK + && kOpSfBytes == minimax_m3_attn_mega::kOpSfBytes, + "header dimensions"); + +} // namespace + +namespace +{ +// TMA descriptor of a [n, k] E4M3 weight (128 x 128 boxes, 128-byte swizzle), cached per weight address. +bool weight_map(CUtensorMap& map, void const* w, int k, int n) +{ + static std::mutex mu; + static std::unordered_map cache; + std::lock_guard lock(mu); + auto it = cache.find(w); + if (it != cache.end()) + { + map = it->second; + return true; + } + static PFN_cuTensorMapEncodeTiled_v12000 encode = []() + { + void* fn = nullptr; + cudaDriverEntryPointQueryResult q; + cudaGetDriverEntryPointByVersion("cuTensorMapEncodeTiled", &fn, 12000, cudaEnableDefault, &q); + return reinterpret_cast(fn); + }(); + if (encode == nullptr) + { + return false; + } + cuuint64_t dims[2] = {static_cast(k), static_cast(n)}; + cuuint64_t strides[1] = {static_cast(k)}; + cuuint32_t box[2] = {128, 128}; + cuuint32_t elem[2] = {1, 1}; + CUresult const r = encode(&map, CU_TENSOR_MAP_DATA_TYPE_UINT8, 2, const_cast(w), dims, strides, box, elem, + CU_TENSOR_MAP_INTERLEAVE_NONE, CU_TENSOR_MAP_SWIZZLE_128B, CU_TENSOR_MAP_L2_PROMOTION_L2_256B, + CU_TENSOR_MAP_FLOAT_OOB_FILL_NONE); + if (r != CUDA_SUCCESS) + { + return false; + } + cache.emplace(w, map); + return true; +} +} // namespace + +cudaError_t invokeMinimaxM3AttnMega(MinimaxM3AttnMegaParams& params, void const* qkvWeight, void const* oWeight, + int qlen, int qg, int bpc, int numReqs, bool pdl, cudaStream_t stream) +{ + if (!weight_map(params.wmap, qkvWeight, kQkvK, kQkvN) || !weight_map(params.wmap_o, oWeight, kOpK, kOpN)) + { + return cudaErrorInvalidValue; + } + params.packed = params.qkv_out; + params.packed_stride = kQkvN; + params.attn_clusters = numReqs * (qlen / qg); + return dispatch(params, qlen, qg, bpc, numReqs, pdl, stream); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/attnMega.h b/cpp/tensorrt_llm/kernels/minimaxM3/attnMega.h new file mode 100644 index 000000000000..cc751911b3c0 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/attnMega.h @@ -0,0 +1,112 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace minimax_m3_attn_mega +{ +constexpr int kGrid = 112; // CTAs of the persistent grid (7 clusters of 16) +constexpr int kMaxTokens = 8; // rows of each call-parity buffer +constexpr int kQkvN = 2560; // packed QKV(+index) width +constexpr int kQkvK = 6144; // hidden size +constexpr int kOpN = 6144; // o_proj output width +constexpr int kOpK = 2048; // o_proj input width (16 heads x 128) +constexpr int kOpSfBytes = 16 * 512; // one parity of the MXFP8 hand-off's swizzled scales +} // namespace minimax_m3_attn_mega + +//! Kernel parameters of the attention megakernel (the device code's Params). +struct MinimaxM3AttnMegaParams +{ + __nv_bfloat16 const* packed; + int64_t packed_stride; + uint8_t* kv_data; + int64_t d_page, d_plane; + uint8_t* kv_scale; + int64_t s_page, s_plane; + uint8_t* idx_k; + int64_t i_page, i_tok; + int64_t num_pages; + int const* out_cache_loc; + int const* position_ids; + int const* block_table; + int64_t bt_stride; + int bt_cols; + int const* seq_lens; + float const* kv_quant_scale; + float const* k_gs; + float const* v_gs; + __nv_bfloat16 const* q_w; + __nv_bfloat16 const* k_w; + __nv_bfloat16 const* ik_w; + float const* rope; + float eps; + float sm_scale; + __nv_bfloat16* out; + int64_t out_stride; + uint8_t* q8; + uint8_t* sf; + int num_sf_k_tiles; + unsigned long long* prof; // optional per-CTA clock64 probes (-DM3FA_PROFILE builds only) + // Fused QKV projection (FQKV kernels only): out[M, 2560] = mxfp8(x)[M, 6144] @ W^T. + __nv_bfloat16 const* x; // [M, 6144] layer input (the all-reduce output) + int ldx; + int num_tokens; // M + uint8_t const* wsf; // UE8M0 weight scales, 128x4-swizzled + float* part; // [2 parities][splits][8 tokens][2560] partial sums, sentinel at rest + int* counters; // [0]: call parity, [1]: CTAs done with this call (zero at rest) + int attn_clusters; // leading clusters that run the attention + uint8_t const* xq; // mega: the layer input's MXFP8 values [M, 6144] (row stride ldxq) ... + uint8_t const* xsf_act; // ... and its 128x4-swizzled UE8M0 scale atoms (atom k at byte 512 k) + int ldxq; + __nv_bfloat16* qkv_out; // mega: the packed QKV rows [2 parities][8][2560] the attention reads + uint8_t const* wsf_o; // mega: the o_proj weight's scale atoms + __nv_bfloat16* y_out; // mega: the o_proj output [M, 6144] + int* ar_ws; // mega, ar_mode: the next m3_mc_allreduce's multicast exchange (int32 words)... + int const* ar_flags; // ... its call flags (parity of a (token, column slice) slot = flags[t * 8 + slice] & 1) + int ar_rank; + int ar_world; + int ar_maxt; + int ar_mode; // 0: none; 2: push the o_proj rows into the exchange (as m3_mx_gemv's ar mode 2) + unsigned int* flags; // mega: [0, 20) tile counters (4 per call), [32 + CTA] the attention CTAs' targets + long long pf_a0, pf_b0, pf_a1, pf_b1; // mega: L2 prefetch regions issued in the attention window (bytes 0: none) + int pf_hint; // 1: evict_last policy on the prefetch + CUtensorMap wmap; // the [2560, 6144] E4M3 weight, 128 x 128 boxes, 128-byte swizzle + CUtensorMap wmap_o; // mega: the [6144, 2048] E4M3 o_proj weight, same boxes +}; + +//! Launch the megakernel on `stream`. The caller fills `params` except `wmap`, `wmap_o`, `packed`, +//! `packed_stride` and `attn_clusters`, which this derives (the TMA descriptors of the two weights, cached per +//! address). Only (qlen, qg, bpc) = (1, 1, 1) with the MXFP8 hand-off is compiled; anything else, or a failed TMA +//! descriptor encode, returns cudaErrorInvalidValue. +cudaError_t invokeMinimaxM3AttnMega(MinimaxM3AttnMegaParams& params, void const* qkvWeight, void const* oWeight, + int qlen, int qg, int bpc, int numReqs, bool pdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/draftStep.cu b/cpp/tensorrt_llm/kernels/minimaxM3/draftStep.cu new file mode 100644 index 000000000000..4df471b7a78f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/draftStep.cu @@ -0,0 +1,576 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/draftStep.h" + +#include +#include +#include + +// One pass of the fused greedy Eagle3 draft loop of MiniMax-M3 (GB300, TP4, <= 8 requests per pass). +// +// One kernel replaces the draft lm_head GEMM, the logits all-gather / float cast / argmax of the in-tree greedy draft +// sampling and the draft loop's per-pass bookkeeping, at the in-tree precision: +// +// * The vocab-sharded BF16 lm_head weight is stored in mma.m16n8k16 A-fragment order (the same values): per 16-row +// tile and k16 step, 512 bytes; lane (g, j) holds 16 bytes = {row g k 2j..2j+1, row g+8 k 2j.., row g k 2j+8.., +// row g+8 k 2j+8..}. +// * Or, by default, in a lossless 12-bit coding of those fragments (75% of the bytes; the target head's coding, see +// targetHead.cu): per 16-row tile and unit of 4 k16 steps a lane reads three 16-byte words, the sign/mantissa bytes +// of steps 0-1, of steps 2-3, then 4 code words (exponent - base, 4 bits; 15 = an exception). The decoded values +// are the checkpoint's BF16 bits; an exception decodes to +0 and the segment's flush adds the exact product x * w +// from the tile's list (products of BF16 values are exact in FP32). +// * A persistent grid (one 16-warp CTA per SM) streams whole tiles over the full K with BF16 MMAs and FP32 +// accumulation; the BF16 head input rows sit in shared memory in B-fragment order, unchanged. +// * Each logit is rounded to BF16 (the in-tree logits are the BF16 GEMM output) and becomes a 64-bit key +// ordered(value) << 32 | ~global_index, with -0 taken as +0: the larger value wins and ties pick the lowest index, as +// torch.max / torch.argmax over the gathered logits do. Keys are max-reduced per row: warp shuffles, then shared +// memory, then atomicMax. +// * The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping +// (Lamport: 0 = empty, a key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the +// slots, and runs the pass's bookkeeping. Call n uses slot half n % 2. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +namespace draftStep +{ +namespace +{ +constexpr int kUnroll = 8; // k16 chunks in flight per warp +constexpr int kChunkBytes = 512; // one tile's k16 step: 32 lanes x 16 bytes + +// Coded units (4 k16 steps, 3 x 16 bytes per lane) in flight per warp: 3 up to 4 rows, 2 at 8 rows. +template +constexpr int codedUnroll() +{ + return MP >= 8 ? 2 : 3; +} + +__device__ __forceinline__ void mma_bf16( + float (&d)[4], uint32_t a0, uint32_t a1, uint32_t a2, uint32_t a3, uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.bf16.bf16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};\n" + : "+f"(d[0]), "+f"(d[1]), "+f"(d[2]), "+f"(d[3]) + : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(b0), "r"(b1)); +} + +// BF16 pair p (values 2p, 2p + 1) of a lane's 8 coded values: sign/mantissa bytes sm (4 per word), codes word cw. +__device__ __forceinline__ uint32_t coded_decode_pair(uint32_t sm_word, int byte0, uint32_t cw, int p, uint32_t base) +{ + uint32_t const b = __byte_perm(sm_word, 0u, byte0 == 0 ? 0x4140 : 0x4342); // {b0, 0, b1, 0} + uint32_t const q = (cw >> (4 * p)) & 0x000f000fu; + uint32_t const e = q * 128u + base; // exponent field of both halves + // Code 15 (an exception, sign/mantissa byte 0) decodes to +0. + return ((b & 0x007f007fu) | ((b << 8) & 0x80008000u) | e) & __vcmpne2(q, 0x000f000fu); +} + +// One coded unit (4 k16 steps) against the BF16 B fragments: steps alternate between the two accumulators, as the +// k16 steps of the plain BF16 head do. +template +__device__ __forceinline__ void chunk_mma_coded(uint4 const& sm01, uint4 const& sm23, uint4 const& cw, uint32_t base, + uint2 const* xs, int unit, int j, int g, bool b_active, float (&da)[4], float (&db)[4]) +{ + uint32_t const smw[8] = {sm01.x, sm01.y, sm01.z, sm01.w, sm23.x, sm23.y, sm23.z, sm23.w}; + uint32_t const cws[4] = {cw.x, cw.y, cw.z, cw.w}; +#pragma unroll + for (int ci = 0; ci < 4; ++ci) + { + uint2 b = make_uint2(0u, 0u); + if (b_active) + b = xs[((unit * 4 + ci) * 4 + j) * MP + g]; + uint32_t const w0 = coded_decode_pair(smw[2 * ci], 0, cws[ci], 0, base); + uint32_t const w1 = coded_decode_pair(smw[2 * ci], 2, cws[ci], 1, base); + uint32_t const w2 = coded_decode_pair(smw[2 * ci + 1], 0, cws[ci], 2, base); + uint32_t const w3 = coded_decode_pair(smw[2 * ci + 1], 2, cws[ci], 3, base); + if (ci & 1) + mma_bf16(db, w0, w1, w2, w3, b.x, b.y); + else + mma_bf16(da, w0, w1, w2, w3, b.x, b.y); + } +} + +// Exceptions of `tile` with k in [kb, ke): v += x * w (exact) for the owner lanes (row g or g + 8; tokens 2j, +// 2j + 1 < M), x from the staged B fragments. Warp-uniform (shuffles). +template +__device__ __forceinline__ void coded_exceptions( + CodedArgs const& cd, int tile, int kb, int ke, uint2 const* xs, int M, int lane, float (&v)[4]) +{ + int const g = lane >> 2, j = lane & 3; + int const e0 = __ldg(cd.exc_off + tile), e1 = __ldg(cd.exc_off + tile + 1); + for (int b0 = e0; b0 < e1; b0 += 32) + { + uint32_t pos = 0xffffffffu; + float2 wv = make_float2(0.f, 0.f); + if (b0 + lane < e1) + { + pos = __ldg(cd.exc_pos + b0 + lane); + wv = __ldg(cd.exc_w + b0 + lane); + } + int const cnt = min(32, e1 - b0); + for (int q = 0; q < cnt; ++q) + { + uint32_t const pq = __shfl_sync(0xffffffffu, pos, q); + float const w0 = __shfl_sync(0xffffffffu, wv.x, q); + int const r = static_cast(pq >> 16), k = static_cast(pq & 0xffffu); + if (k >= kb && k < ke && (r & 7) == g) + { + int const st = k >> 4, rr = k & 15; + bool const hi = r >= 8; +#pragma unroll + for (int q2 = 0; q2 < 2; ++q2) + { + int const n = 2 * j + q2; + if (n < M) + { + uint2 const bw = xs[(st * 4 + ((rr & 7) >> 1)) * MP + n]; + uint32_t const word = rr >= 8 ? bw.y : bw.x; + float const xv = __uint_as_float((rr & 1) ? (word & 0xffff0000u) : (word << 16)); + float const d = __fmul_rn(xv, w0); + if (hi) + v[2 + q2] += d; + else + v[q2] += d; + } + } + } + __syncwarp(); + } + } +} + +__device__ __forceinline__ uint4 ldg_stream(uint4 const* p) +{ + uint4 r; + asm volatile("ld.global.nc.L1::no_allocate.L2::256B.v4.u32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(r.x), "=r"(r.y), "=r"(r.z), "=r"(r.w) + : "l"(p)); + return r; +} + +// The argmax value of a logit: its BF16 rounding (the in-tree logits' dtype), -0 as +0. +__device__ __forceinline__ float key_value(float v) +{ + float const r = __bfloat162float(__float2bfloat16(v)); + return r == 0.f ? 0.f : r; +} + +__device__ __forceinline__ unsigned long long make_key(float v, uint32_t index) +{ + uint32_t u = __float_as_uint(v); + u = (u & 0x80000000u) ? ~u : (u | 0x80000000u); + return (static_cast(u) << 32) | static_cast(0xFFFFFFFFu - index); +} + +__device__ __forceinline__ unsigned long long shfl_max(unsigned long long k, int o) +{ + unsigned long long const ok = __shfl_xor_sync(0xffffffffu, k, o); + return ok > k ? ok : k; +} + +// One k16 step of a tile: the lane's A fragment w against the activation B fragment of step `chunk` (rows g < MP). +// Even and odd steps accumulate separately (two independent MMA chains); the tile's sum is da + db. +template +__device__ __forceinline__ void chunk_mma( + uint4 const& w, uint2 const* xs, int chunk, int j, int g, bool b_active, float (&da)[4], float (&db)[4]) +{ + uint2 b = make_uint2(0u, 0u); + if (b_active) + b = xs[(chunk * 4 + j) * MP + g]; + if (chunk & 1) + mma_bf16(db, w.x, w.y, w.z, w.w, b.x, b.y); + else + mma_bf16(da, w.x, w.y, w.z, w.w, b.x, b.y); +} + +// Add a warp's segment of a tile into the CTA's accumulator (with a coded head, the segment's exceptions first). +template +__device__ __forceinline__ void flush_tile( + float4* acc, float (&da)[4], float (&db)[4], Params const& p, int tile, int kb, int ke, uint2 const* xs, int lane) +{ + float v[4] = {da[0] + db[0], da[1] + db[1], da[2] + db[2], da[3] + db[3]}; + if constexpr (CODED) + coded_exceptions(p.cd, tile, kb, ke, xs, p.M, lane, v); + float* a = reinterpret_cast(acc); + atomicAdd(a + 0, v[0]); + atomicAdd(a + 1, v[1]); + atomicAdd(a + 2, v[2]); + atomicAdd(a + 3, v[3]); +#pragma unroll + for (int e = 0; e < 4; ++e) + { + da[e] = 0.f; + db[e] = 0.f; + } +} + +// Balanced streaming GEMV over the fragment-ordered weight, persistent grid of 16-warp CTAs. CTA c owns tiles +// [t_begin, t_end); its (tile, chunk) space is split into 16 equal contiguous ranges, one per warp, so every warp +// streams the same number of bytes. A tile straddles at most two warps; their partial sums are added into shared +// memory (two addends: the order does not change the sum). MP = rows padded to 1 / 2 / 4 / 8. A chunk is one k16 +// step of the BF16 head, or one coded unit (4 k16 steps) of the coded head (CODED: no shared-memory prologue). +template +__global__ void __launch_bounds__(kWarps * 32, 1) + draft_step_kernel(Params const p, int kchunks, int pf_chunks, int pf_off) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(16) uint2 xs[]; // [K/16 steps][4 (j)][MP (g)], then the weight prologue + __shared__ float4 acc_s[kMaxTilesPerCta][32]; + __shared__ unsigned long long red[kWarps][kMaxTokens]; + __shared__ __align__(8) unsigned long long pf_bar[kWarps]; + __shared__ bool s_last; + __shared__ int s_tok[kMaxTokens]; + + int const warp = threadIdx.x >> 5; + int const lane = threadIdx.x & 31; + int const g = lane >> 2; + int const j = lane & 3; + constexpr int kChunkK = CODED ? 64 : 16; // k per chunk + constexpr int kWords = CODED ? 3 : 1; // 16-byte words per lane and chunk + constexpr int KU = CODED ? codedUnroll() : kUnroll; + int const K = kchunks * kChunkK; + int const M = p.M; + int const t_begin = static_cast((static_cast(blockIdx.x) * p.n_tiles) / gridDim.x); + int const t_end = static_cast((static_cast(blockIdx.x + 1) * p.n_tiles) / gridDim.x); + int const n_local = t_end - t_begin; + int const span = n_local * kchunks; + int const q_begin = warp * span / kWarps; + int const q_end = (warp + 1) * span / kWarps; + long long const base = static_cast(t_begin) * kchunks; // first chunk of this CTA + uint4 const* const wf = p.wf; + + // Weights are static: start this warp's stream before waiting on the producer of x. The warp's first pf_chunks + // chunks come in by one bulk copy into shared memory (dynamic offset pf_off), which fills the grid-dependency + // wait and the staging; the register ring starts after them. + int const n_pf = CODED ? 0 : min(pf_chunks, q_end - q_begin); + uint4* const wpf = reinterpret_cast(reinterpret_cast(xs) + pf_off) + warp * pf_chunks * 32; + if (n_pf > 0 && lane == 0) + { + uint32_t const bar = static_cast(__cvta_generic_to_shared(&pf_bar[warp])); + uint32_t const bytes = static_cast(n_pf) * kChunkBytes; + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(bar) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(bar), "r"(bytes) : "memory"); + asm volatile("cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes [%0], [%1], %2, [%3];" ::"r"( + static_cast(__cvta_generic_to_shared(wpf))), + "l"(wf + (base + q_begin) * 32), "r"(bytes), "r"(bar) + : "memory"); + } + __syncwarp(); + int const q_reg = q_begin + n_pf; + uint4 buf[KU][kWords]; +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q_reg + u < q_end) + { +#pragma unroll + for (int l = 0; l < kWords; ++l) + buf[u][l] = ldg_stream(wf + ((base + q_reg + u) * kWords + l) * 32 + lane); + } + } + for (int i = threadIdx.x; i < n_local * 32; i += blockDim.x) + acc_s[i / 32][i % 32] = make_float4(0.f, 0.f, 0.f, 0.f); + + asm volatile("griddepcontrol.wait;" ::: "memory"); + + // Stage the head input rows in B-fragment order: item (k16 step S, row gg) = two 16-byte loads of x, four + // fragment words {x[2jj], x[2jj + 1]}, {x[2jj + 8], x[2jj + 9]} (BF16 pairs, moved unchanged). Rows >= M are 0. + constexpr int kSU = MP >= 4 ? 4 : 2; + int const n_items = (K / 16) * MP; + for (int i0 = threadIdx.x; i0 < n_items; i0 += kSU * blockDim.x) + { + uint4 xv[kSU][2]; +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + int const gg = i % MP; + int const S = i / MP; + xv[u][0] = make_uint4(0u, 0u, 0u, 0u); + xv[u][1] = make_uint4(0u, 0u, 0u, 0u); + if (i < n_items && gg < M) + { + long long const xr = p.x_rows != nullptr ? p.x_rows[gg] : gg; + uint4 const* xp = reinterpret_cast(p.x + xr * K + 16 * S); + xv[u][0] = xp[0]; + xv[u][1] = xp[1]; + } + } +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + if (i >= n_items) + continue; + int const gg = i % MP; + int const S = i / MP; + uint32_t const lo[4] = {xv[u][0].x, xv[u][0].y, xv[u][0].z, xv[u][0].w}; + uint32_t const hi[4] = {xv[u][1].x, xv[u][1].y, xv[u][1].z, xv[u][1].w}; +#pragma unroll + for (int jj = 0; jj < 4; ++jj) + xs[(S * 4 + jj) * MP + gg] = make_uint2(lo[jj], hi[jj]); + } + } + __syncthreads(); + + bool const b_active = g < MP; + float da[4] = {0.f, 0.f, 0.f, 0.f}; + float db[4] = {0.f, 0.f, 0.f, 0.f}; + int tl = q_begin / kchunks; // local tile of the current accumulator + int c = q_begin - tl * kchunks; // chunk within that tile + int cs = c; // first chunk of the warp's segment of that tile + if (n_pf > 0) + { + uint32_t const bar = static_cast(__cvta_generic_to_shared(&pf_bar[warp])); + asm volatile( + "{\n .reg .pred p;\n PF_WAIT%=:\n mbarrier.try_wait.parity.shared::cta.b64 p, [%0], 0;\n" + " @!p bra PF_WAIT%=;\n}\n" ::"r"(bar) + : "memory"); + for (int q = q_begin; q < q_reg; ++q) + { + chunk_mma(wpf[(q - q_begin) * 32 + lane], xs, c, j, g, b_active, da, db); + if (++c == kchunks) + { + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, K, xs, lane); + c = 0; + cs = 0; + ++tl; + } + } + } + for (int q = q_reg; q < q_end; q += KU) + { + uint4 nxt[KU][kWords]; +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q + KU + u < q_end) + { +#pragma unroll + for (int l = 0; l < kWords; ++l) + nxt[u][l] = ldg_stream(wf + ((base + q + KU + u) * kWords + l) * 32 + lane); + } + } +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q + u < q_end) + { + if constexpr (CODED) + chunk_mma_coded(buf[u][0], buf[u][kWords > 1 ? 1 : 0], buf[u][kWords > 2 ? 2 : 0], p.cd.base, + xs, c, j, g, b_active, da, db); + else + chunk_mma(buf[u][0], xs, c, j, g, b_active, da, db); + if (++c == kchunks) + { + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, K, xs, lane); + c = 0; + cs = 0; + ++tl; + } + } + } +#pragma unroll + for (int u = 0; u < KU; ++u) +#pragma unroll + for (int l = 0; l < kWords; ++l) + buf[u][l] = nxt[u][l]; + } + if (c != 0) + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, c * kChunkK, xs, lane); + __syncthreads(); + + // Accumulator lane (g, j) of a tile: rows g / g + 8 (x, y / z, w) of tokens 2j / 2j + 1 (x, z / y, w). + unsigned long long best0 = 0ull, best1 = 0ull; // tokens 2j, 2j + 1 + for (int t = warp; t < n_local; t += kWarps) + { + float4 const v = acc_s[t][lane]; + int const r0 = (t_begin + t) * 16 + g; + uint32_t const i0 = static_cast(p.vocab_offset + r0); + unsigned long long k0 = make_key(key_value(v.x), i0); + unsigned long long const k0b = make_key(key_value(v.z), i0 + 8); + unsigned long long k1 = make_key(key_value(v.y), i0); + unsigned long long const k1b = make_key(key_value(v.w), i0 + 8); + k0 = k0b > k0 ? k0b : k0; + k1 = k1b > k1 ? k1b : k1; + best0 = k0 > best0 ? k0 : best0; + best1 = k1 > best1 ? k1 : best1; + } +#pragma unroll + for (int o = 4; o < 32; o <<= 1) + { + best0 = shfl_max(best0, o); + best1 = shfl_max(best1, o); + } + asm volatile("griddepcontrol.launch_dependents;"); + if (lane < 4) + { + red[warp][2 * lane] = best0; + red[warp][2 * lane + 1] = best1; + } + __syncthreads(); + if (threadIdx.x < M) + { + unsigned long long b = 0ull; +#pragma unroll + for (int w = 0; w < kWarps; ++w) + b = red[w][threadIdx.x] > b ? red[w][threadIdx.x] : b; + if (b != 0ull) + atomicMax(p.keys + threadIdx.x, b); + } + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + unsigned int const prev = atomicAdd(p.counters, 1u); + s_last = prev == gridDim.x - 1; + } + __syncthreads(); + if (!s_last) + return; + + __threadfence(); + unsigned int const call = *reinterpret_cast(p.counters + 1); + int const half = static_cast(call & 1u); + if (threadIdx.x < M) + { + int const t = threadIdx.x; + unsigned long long const mine = atomicExch(p.keys + t, 0ull); + unsigned long long* slot_mc = p.ws_mc + (half * kMaxTokens + t) * p.world; + unsigned long long* slot_uc = p.ws_uc + (half * kMaxTokens + t) * p.world; + slot_mc[p.rank] = mine; + unsigned long long best = 0ull; + for (int r = 0; r < p.world; ++r) + { + unsigned long long v; + do + { + v = *reinterpret_cast(slot_uc + r); + } while (v == 0ull); + best = v > best ? v : best; + slot_uc[r] = 0ull; + } + s_tok[t] = static_cast(0xFFFFFFFFu - static_cast(best & 0xFFFFFFFFull)); + if (p.out_keys != nullptr) + p.out_keys[t] = best; + } + __syncthreads(); + if (threadIdx.x == 0) + { + p.counters[0] = 0u; + p.counters[1] = call + 1u; + } + if (threadIdx.x < M) + { + StepArgs const& st = p.st; + int const b = threadIdx.x; + int const tok = s_tok[b]; + st.draft_out[b * st.draft_stride + st.step] = tok; + st.ids_next[b] = tok; + long long const r = p.x_rows != nullptr ? p.x_rows[b] : b; + st.pos_next[b] = st.pos_src[r] + 1; + if (st.step == 0 && st.num_acc != nullptr) + { + st.kv_lens[b] -= st.draft_len - st.num_acc[b]; + st.seq_lens[b] = 1; + } + else if (st.step > 0) + { + st.kv_lens[b] += 1; + } + } +#endif +} + +template +cudaError_t launch(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + auto kern = draft_step_kernel; + size_t const xs_bytes = static_cast(K / 16) * 4 * MP * sizeof(uint2); + // Shared-memory weight prologue: chunks per warp that fit next to the kernel's static + dynamic shared memory. + static int const sStaticSmem = [] + { + cudaFuncAttributes fa{}; + return cudaFuncGetAttributes(&fa, draft_step_kernel) == cudaSuccess + ? static_cast(fa.sharedSizeBytes) + : -1; + }(); + static int const sMaxSmemPerBlock = [] + { + int dev = 0, maxSmem = 0; + cudaGetDevice(&dev); + cudaDeviceGetAttribute(&maxSmem, cudaDevAttrMaxSharedMemoryPerBlockOptin, dev); + return maxSmem; + }(); + int pf = 0; + if (!CODED && sStaticSmem >= 0) + { + long long const room + = static_cast(sMaxSmemPerBlock) - sStaticSmem - static_cast(xs_bytes) - 1024; + long long const want = static_cast(kPrefetchKb) * 1024; + pf = static_cast(std::max(0ll, std::min(room, want)) / (kWarps * kChunkBytes)); + } + size_t const smem = xs_bytes + static_cast(pf) * kWarps * kChunkBytes; + cudaError_t err = cudaFuncSetAttribute(kern, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smem)); + if (err != cudaSuccess) + return err; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(static_cast(grid)); + cfg.blockDim = dim3(kWarps * 32); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, kern, p, K / (CODED ? 64 : 16), pf, static_cast(xs_bytes)); +} + +template +cudaError_t dispatchRows(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + if (p.M <= 1) + return launch<1, CODED>(p, K, grid, pdl, stream); + if (p.M <= 2) + return launch<2, CODED>(p, K, grid, pdl, stream); + if (p.M <= 4) + return launch<4, CODED>(p, K, grid, pdl, stream); + return launch<8, CODED>(p, K, grid, pdl, stream); +} + +} // namespace + +cudaError_t invokeDraftStep(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + bool const coded = p.cd.exc_off != nullptr; + if (p.M < 1 || p.M > kMaxTokens || K % (coded ? 64 : 16) != 0 || grid < 1) + return cudaErrorInvalidValue; + return coded ? dispatchRows(p, K, grid, pdl, stream) : dispatchRows(p, K, grid, pdl, stream); +} + +} // namespace draftStep +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/draftStep.h b/cpp/tensorrt_llm/kernels/minimaxM3/draftStep.h new file mode 100644 index 000000000000..d6f4c1dd67d9 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/draftStep.h @@ -0,0 +1,99 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +//! One pass of the fused greedy Eagle3 draft loop of the MiniMax-M3 NVFP4 TP4 decode path (sm_103a): the BF16 +//! vocab-sharded draft head, the cross-rank argmax and the draft loop's per-pass bookkeeping in one kernel. See +//! draftStep.cu. +namespace draftStep +{ + +constexpr int kMaxTokens = 8; // head rows (requests) of one launch +constexpr int kWarps = 16; // warps per CTA +constexpr int kMaxTilesPerCta = 32; // 16-row weight tiles one CTA can own +//! Shared-memory weight prologue per CTA, in KB (capped by the free shared memory). +constexpr int kPrefetchKb = 112; + +// Draft-pass epilogue (last CTA, one thread per request row b): stores the token as draft token `step`, stages the +// next pass's input id and position, and advances the draft KV lengths the way the in-tree draft loop does between +// passes (pass 0 with num_acc: drop the rejected verify rows and switch to one token per request; later passes: +1). +struct StepArgs +{ + int* draft_out; // [nreq, draft_stride]: column `step` receives the token + int draft_stride; + int step; + int* ids_next; // [nreq] + int const* pos_src; // positions of the rows the head read (indexed like x) + int* pos_next; // [nreq] = pos_src[x_rows[b] or b] + 1 + int* kv_lens; // [nreq] draft KV lengths, updated in place + int const* num_acc; // pass 0: accepted counts (nullptr: no rollback) + int* seq_lens; // pass 0: set to 1 + int draft_len; +}; + +// The 12-bit coded head (see draftStep.cu): the per-tile exception lists and the exponent base. exc_off == nullptr: +// the head is the plain BF16 fragment-ordered copy. +struct CodedArgs +{ + int const* exc_off; // [n_tiles + 1] exception list offsets + uint32_t const* exc_pos; // row in tile << 16 | k, per exception + float2 const* exc_w; // {w, 0} per exception: the exact weight + uint32_t base; // exponent base << 7 in both BF16 halves (base << 7 | base << 23) +}; + +// Everything a draft-step launch takes (see invokeDraftStep). +struct Params +{ + __nv_bfloat16 const* x; // [*, K] head input rows + long long const* x_rows; // optional [M]: head input row of request b (pass 0: the last accepted verify row) + int M; // requests, <= kMaxTokens + uint4 const* wf; // [n_tiles][K / 16][32 lanes] 16-byte BF16 A fragments, or the 12-bit coded stream + // [n_tiles][K / 64][3][32 lanes] when cd.exc_off is set (see draftStep.cu) + int n_tiles; + int vocab_offset; // global id of this rank's first head row + unsigned long long* keys; + unsigned int* counters; + unsigned long long* ws_uc; // this rank's exchange slots [2][kMaxTokens][world] + unsigned long long* ws_mc; // their multicast address + int rank; + int world; + unsigned long long* out_keys; // optional [M]: the winning global key of each row + StepArgs st; + CodedArgs cd; +}; + +//! Launch the draft step over `grid` 512-thread CTAs (K = the hidden size, a multiple of 16; of 64 for a coded head). +cudaError_t invokeDraftStep(Params const& p, int K, int grid, bool pdl, cudaStream_t stream); + +} // namespace draftStep +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.cu b/cpp/tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.cu new file mode 100644 index 000000000000..4383bb0fb2e6 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.cu @@ -0,0 +1,3679 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Fused short-sequence decode attention of the MiniMax-M3 MSA sparse layers (packed QKV input). + +#include "tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +// The kernel uses tcgen05/TMEM (including block-scaled MMAs, which family targets lack), cluster DSMEM and E2M1 +// conversions: its body is compiled for the arch-specific sm_100a / sm_103a targets only (see the minimax_m3_attn_src +// library in kernels/CMakeLists.txt). Any other target compiles a kernel that traps, so a launch on a build without +// those targets fails loudly instead of returning untouched outputs. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ == 1000 || __CUDA_ARCH__ == 1030) \ + && (defined(__CUDA_ARCH_SPECIFIC__) || defined(__CUDA_ARCH_FEAT_SM100_ALL) || defined(__CUDA_ARCH_FEAT_SM103_ALL)) +#define MINIMAX_M3_ATTN_DEVICE_ENABLED 1 +#else +#define MINIMAX_M3_ATTN_DEVICE_ENABLED 0 +#endif + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kHeadDim = 128; +constexpr int kRotDim = 64; +constexpr int kPage = 128; +constexpr int kHq = 16; +constexpr int kColK = kHq; // packed head slots: [Q x16 | K | V | idxQ | idxK] +constexpr int kColV = kHq + 1; +constexpr int kColIdxK = kHq + 3; +constexpr int kBlocks = 16; // KV blocks per request == topk == cluster size +constexpr int kComputeWarps = 16; // QK / softmax / PV and the dequant (one 16-byte chunk per thread) +constexpr int kWarps = kComputeWarps + 4; // plus producer-only warps, so new K/V/index-K never queue behind a Q head +constexpr int kThreads = kWarps * 32; +constexpr int kComputeThreads = kComputeWarps * 32; +constexpr int kOPitch = 132; // floats per partial-O row +constexpr float kNegInf = -__builtin_huge_valf(); + +using Params = MinimaxM3FusedDecodeAttnParams; + +// Optional per-CTA clock64 probes for development (-DM3FA_PROFILE and a prof tensor): [CTA][64] timestamps. +// -DM3FA_PROFILE_GT records %globaltimer (ns, comparable across SMs and kernels) instead of clock64. +#ifdef M3FA_PROFILE_GT +#define M3FA_PROFILE +#define M3FA_NOW(v) asm volatile("mov.u64 %0, %%globaltimer;" : "=l"(v)) +#else +#define M3FA_NOW(v) v = clock64() +#endif +#ifdef M3FA_PROFILE +#define M3FA_PROF_T(i, thread) \ + do \ + { \ + if (p.prof != nullptr && threadIdx.x == (thread)) \ + { \ + unsigned long long now_; \ + M3FA_NOW(now_); \ + p.prof[blockIdx.x * 64 + (i)] = now_; \ + } \ + } while (0) +#else +#define M3FA_PROF_T(i, thread) \ + do \ + { \ + } while (0) +#endif +#define M3FA_PROF(i) M3FA_PROF_T(i, 0) +// clock64 probes (development): slot i of this CTA, written by every calling thread. +#ifdef M3FA_PROFILE +#define M3FA_CLK(i) \ + do \ + { \ + if (p.prof != nullptr) \ + { \ + p.prof[blockIdx.x * 64 + (i)] = clock64(); \ + } \ + } while (0) +#else +#define M3FA_CLK(i) \ + do \ + { \ + } while (0) +#endif + +#ifndef M3FA_MEGA_EARLYCLW +#define M3FA_MEGA_EARLYCLW 1 // pass the cluster barrier (peers' merge mbarriers initialized) early, not before the push +#endif +#ifndef M3FA_V15_EARLYV +#define M3FA_V15_EARLYV 0 // packed path: dequantize V before the PDL wait (switch; off by default) +#endif +#ifndef M3FA_MEGA_TRIGGER +// Megakernel PDL release: 0 o_proj CTAs after their activation (attention CTAs at exit); 1 as 0 with the attention +// CTAs after their KV reads; 2 every CTA right after its griddepcontrol.wait; 3 o_proj CTAs after their epilogue. +#define M3FA_MEGA_TRIGGER 3 +#endif +#ifndef M3FA_TRIGGER_AT +#define M3FA_TRIGGER_AT 2 // where the PDL dependents are released: 0 entry, 1 after the wait, 2 after the KV reads +#endif + +// Generic-proxy shared-memory writes (fp16 tiles, Q, P) become visible to the tensor core (async proxy). +#define M3FA_FENCE_PROXY_ASYNC() asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory") + +__device__ __forceinline__ void cp_async16(void* dst, void const* src) +{ + uint32_t const d = static_cast(__cvta_generic_to_shared(dst)); + asm volatile("cp.async.cg.shared.global [%0], [%1], 16;\n" ::"r"(d), "l"(src) : "memory"); +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +// One packed E2M1 byte -> half2 (low nibble in the low half). +__device__ __forceinline__ uint32_t e2m1x2_to_h2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 src;\n mov.b32 {src, _, _, _}, %1;\n cvt.rn.f16x2.e2m1x2 %0, src;\n}\n" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t hmul2_u(uint32_t a, uint32_t b) +{ + __half2 r = __hmul2(*reinterpret_cast<__half2*>(&a), *reinterpret_cast<__half2*>(&b)); + return *reinterpret_cast(&r); +} + +__device__ __forceinline__ uint32_t e4m3_to_h2_splat(uint32_t byte) +{ + __half_raw const h = __nv_cvt_fp8_to_halfraw(static_cast<__nv_fp8_storage_t>(byte), __NV_E4M3); + return static_cast(h.x) | (static_cast(h.x) << 16); +} + +__device__ __forceinline__ uint32_t e4m3x2_to_h2(uint32_t two_bytes) +{ + __half2_raw const h = __nv_cvt_fp8x2_to_halfraw2(static_cast<__nv_fp8x2_storage_t>(two_bytes), __NV_E4M3); + return static_cast(h.x) | (static_cast(h.y) << 16); +} + +// ---- Exact copies of the C++ producer's helpers (fusedQKNormRopeKernel.cu / quantization.cuh) ---- +template +__device__ __forceinline__ uint32_t cvt_sf_group_shfl_mask() +{ + constexpr uint32_t groupSize = static_cast(NUM_THREADS_PER_SF); + constexpr uint32_t groupMask = 0xFFFFFFFFU >> (32U - groupSize); + uint32_t laneId = 0; + asm("mov.u32 %0, %%laneid;" : "=r"(laneId)); + return groupMask << (laneId & ~(groupSize - 1U)); +} + +__device__ __forceinline__ uint32_t fp32_vec_to_e2m1(float (&array)[8]) +{ + uint32_t val; + asm volatile( + "{\n" + ".reg .b8 byte0;\n" + ".reg .b8 byte1;\n" + ".reg .b8 byte2;\n" + ".reg .b8 byte3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte0, %2, %1;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte1, %4, %3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte2, %6, %5;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte3, %8, %7;\n" + "mov.b32 %0, {byte0, byte1, byte2, byte3};\n" + "}" + : "=r"(val) + : "f"(array[0]), "f"(array[1]), "f"(array[2]), "f"(array[3]), "f"(array[4]), "f"(array[5]), "f"(array[6]), + "f"(array[7])); + return val; +} + +__device__ __forceinline__ uint16_t quantizeMinimaxM3Fp4x4( + float const (&elements)[4], float secondLevelScale, uint8_t* scaleOut) +{ + __nv_bfloat16 rounded[4]; + float localMax = 0.0F; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + rounded[i] = __float2bfloat16_rn(elements[i]); + localMax = fmaxf(localMax, fabsf(__bfloat162float(rounded[i]))); + } + uint32_t const groupMask = cvt_sf_group_shfl_mask<4>(); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 1)); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 2)); + float const scaleValue = secondLevelScale * (localMax * rcp_approx_ftz(6.0F)); + __nv_fp8_e4m3 const encodedScale(scaleValue); + if (scaleOut != nullptr) + { + *scaleOut = encodedScale.__x; + } + float const narrowedScale = static_cast(encodedScale); + float const outputScale + = localMax != 0.0F ? rcp_approx_ftz(narrowedScale * rcp_approx_ftz(secondLevelScale)) : 0.0F; + float quantValues[8] = {}; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + quantValues[i] = __bfloat162float(rounded[i]) * outputScale; + } + return static_cast(fp32_vec_to_e2m1(quantValues)); +} + +// RMSNorm (gemma 1+w) + NeoX RoPE over the first 64 dims, exactly as the producer: lane owns 4 dims. +__device__ __forceinline__ void norm_rope( + float (&e)[4], float const (&w)[4], float eps, float const (&cs)[4], float const (&sn)[4], int lane) +{ + float ss = 0.0F; + ss += e[0] * e[0] + e[1] * e[1]; + ss += e[2] * e[2] + e[3] * e[3]; +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + { + ss = ss + __shfl_xor_sync(0xffffffffu, ss, mask, 32); + } + float const rmsReciprocal = rsqrtf(ss / static_cast(kHeadDim) + eps); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + e[i] *= rmsReciprocal * (1.0F + w[i]); + } + __syncwarp(); + constexpr int kPairOffset = (kRotDim / 2) / 4; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const dim = lane * 4 + i; + float paired = __shfl_xor_sync(0xffffffffu, e[i], kPairOffset); + if (dim < kRotDim) + { + bool const firstHalf = dim < kRotDim / 2; + if (firstHalf) + { + paired = -paired; + } + e[i] = e[i] * cs[i] + paired * sn[i]; + } + } + __syncwarp(); +} + +__device__ __forceinline__ void load_w4(float (&w)[4], __nv_bfloat16 const* src, int lane) +{ + uint2 const raw = *reinterpret_cast(src + lane * 4); + float2 const a = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw.x)); + float2 const b = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw.y)); + w[0] = a.x; + w[1] = a.y; + w[2] = b.x; + w[3] = b.y; +} + +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t mapa_shared(uint32_t addr, uint32_t rank) +{ + uint32_t r; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;\n" : "=r"(r) : "r"(addr), "r"(rank)); + return r; +} + +// QLEN query tokens per request; QG of them per cluster (QLEN / QG clusters per request); BPC KV blocks per CTA +// (cluster of kBlocks / BPC CTAs). +// Byte offset of element (r, k) in a UMMA no-swizzle K-major operand tile of 128 k-elements per row: 8x8 core +// matrices of 16-byte rows, 128 bytes apart along k (LBO) and 2048 bytes apart per 8 rows (SBO). The same formula +// with r = token, k = dim stores V so that it is the MN-major (dims contiguous) A operand of O^T = V^T P^T. +__device__ __forceinline__ int kmaj_off(int r, int k) +{ + return (r >> 3) * 2048 + (k >> 3) * 128 + (r & 7) * 16 + (k & 7) * 2; +} + +__device__ __forceinline__ uint64_t umma_desc(uint32_t addr, uint32_t lbo, uint32_t sbo) +{ + return static_cast((addr >> 4) & 0x3FFFu) | (static_cast((lbo >> 4) & 0x3FFFu) << 16) + | (static_cast((sbo >> 4) & 0x3FFFu) << 32) | (1ull << 46); +} + +// kind::f16 instruction descriptor: F32 accumulate, F16 operands, B K-major, M = 128. +__host__ __device__ constexpr uint32_t umma_idesc(int n, int a_mn_major, int b_mn_major) +{ + return (1u << 4) | (static_cast(a_mn_major) << 15) | (static_cast(b_mn_major) << 16) + | (static_cast(n >> 3) << 17) | (static_cast(128 >> 4) << 24); +} + +// 8-row version for the split softmax: the same lane-bit order (16, 8, 4, then 2 and 1 in full) as +// warp_reduce_scatter16, so every row is combined in the same association (fp add is commutative, not associative). +// After it lanes with bits 4..2 = (b4, b3, b2) hold row 4*b4 + 2*b3 + b2. +template +__device__ __forceinline__ float warp_reduce_scatter8(float (&a)[8], int lane) +{ +#pragma unroll + for (int step = 0; step < 3; ++step) + { + int const mask = 16 >> step; + int const half = 4 >> step; + bool const upper = (lane & mask) != 0; +#pragma unroll + for (int i = 0; i < half; ++i) + { + float const send = upper ? a[i] : a[i + half]; + float const keep = upper ? a[i + half] : a[i]; + float const recv = __shfl_xor_sync(0xffffffffu, send, mask); + a[i] = kMax ? fmaxf(keep, recv) : keep + recv; + } + } + float x = a[0]; + float const o2 = __shfl_xor_sync(0xffffffffu, x, 2); + x = kMax ? fmaxf(x, o2) : x + o2; + float const o1 = __shfl_xor_sync(0xffffffffu, x, 1); + return kMax ? fmaxf(x, o1) : x + o1; +} + +// Reduce 16 per-lane values over the warp, one value per row: after the reduce-scatter lane l holds row +// 8*b4 + 4*b3 + 2*b2 + b1 (b_k = bit k of l) reduced over all 32 lanes. 16 shuffles instead of 80. +template +__device__ __forceinline__ float warp_reduce_scatter16(float (&a)[16], int lane) +{ +#pragma unroll + for (int step = 0; step < 4; ++step) + { + int const mask = 16 >> step; + int const half = 8 >> step; + bool const upper = (lane & mask) != 0; +#pragma unroll + for (int i = 0; i < half; ++i) + { + float const send = upper ? a[i] : a[i + half]; + float const keep = upper ? a[i + half] : a[i]; + float const recv = __shfl_xor_sync(0xffffffffu, send, mask); + a[i] = kMax ? fmaxf(keep, recv) : keep + recv; + } + } + float const other = __shfl_xor_sync(0xffffffffu, a[0], 1); + return kMax ? fmaxf(a[0], other) : a[0] + other; +} + +__device__ __forceinline__ void umma_f16( + uint32_t tmem_d, uint64_t adesc, uint64_t bdesc, uint32_t idesc, int accumulate) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, %4, 0;\n tcgen05.mma.cta_group::1.kind::f16 [%0], %1, %2, %3, p;\n}\n" :: + "r"(tmem_d), + "l"(adesc), "l"(bdesc), "r"(idesc), "r"(accumulate) + : "memory"); +} + +__device__ __forceinline__ void umma_commit(uint32_t bar) +{ + asm volatile("tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.b64 [%0];\n" ::"r"(bar) : "memory"); +} + +__device__ __forceinline__ void mbar_wait_cta(uint32_t bar, uint32_t parity) +{ + uint32_t ready = 0; +#ifdef M3FA_DBG_HANG + long long spins = 0; +#endif + while (!ready) + { +#ifdef M3FA_DBG_HANG + if (++spins == (1LL << 24)) + { + printf("M3FA hang: block %d thread %d mbar 0x%x parity %u\n", blockIdx.x, threadIdx.x, bar, parity); + } +#endif + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.shared::cta.b64 p, [%1], %2;\n selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(bar), "r"(parity) + : "memory"); + } +} + +// Spin without suspending (v52's acc_done wait): a suspended try_wait wakes late on a tcgen05.commit arrival. +#ifndef M3FA_EPI_SPIN +#define M3FA_EPI_SPIN 1 +#endif +__device__ __forceinline__ void mbar_wait_epi(uint32_t bar, uint32_t parity) +{ + if (!M3FA_EPI_SPIN) + { + mbar_wait_cta(bar, parity); + return; + } + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.shared::cta.b64 p, [%1], %2;\n selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(bar), "r"(parity) + : "memory"); + } +} + +__device__ __forceinline__ void tmem_ld8(uint32_t addr, float (&v)[8]) +{ + uint32_t r[8]; + asm volatile("tcgen05.ld.sync.aligned.32x32b.x8.b32 {%0, %1, %2, %3, %4, %5, %6, %7}, [%8];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]), "=r"(r[4]), "=r"(r[5]), "=r"(r[6]), "=r"(r[7]) + : "r"(addr)); + asm volatile("tcgen05.wait::ld.sync.aligned;\n" ::: "memory"); +#pragma unroll + for (int i = 0; i < 8; ++i) + { + v[i] = __uint_as_float(r[i]); + } +} + +__device__ __forceinline__ void tmem_ld16(uint32_t addr, float (&v)[16]) +{ + uint32_t r[16]; + asm volatile( + "tcgen05.ld.sync.aligned.32x32b.x16.b32 {%0, %1, %2, %3, %4, %5, %6, %7, %8, %9, %10, %11, %12, %13, %14, " + "%15}, [%16];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]), "=r"(r[4]), "=r"(r[5]), "=r"(r[6]), "=r"(r[7]), "=r"(r[8]), + "=r"(r[9]), "=r"(r[10]), "=r"(r[11]), "=r"(r[12]), "=r"(r[13]), "=r"(r[14]), "=r"(r[15]) + : "r"(addr)); + asm volatile("tcgen05.wait::ld.sync.aligned;\n" ::: "memory"); +#pragma unroll + for (int i = 0; i < 16; ++i) + { + v[i] = __uint_as_float(r[i]); + } +} + +// Warp-level fp16 MMA (m16n8k16, fp32 accumulate) and its shared-memory fragment loads. +__device__ __forceinline__ void ldsm_x4(uint32_t addr, uint32_t (&r)[4]) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0, %1, %2, %3}, [%4];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x2(uint32_t addr, uint32_t (&r)[2]) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x2.shared.b16 {%0, %1}, [%2];\n" : "=r"(r[0]), "=r"(r[1]) : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x2_trans(uint32_t addr, uint32_t (&r)[2]) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x2.trans.shared.b16 {%0, %1}, [%2];\n" + : "=r"(r[0]), "=r"(r[1]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma_f16_16816(float (&d)[4], uint32_t const (&a)[4], uint32_t const (&b)[2]) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0, %1, %2, %3}, {%4, %5, %6, %7}, {%8, %9}, " + "{%0, %1, %2, %3};\n" + : "+f"(d[0]), "+f"(d[1]), "+f"(d[2]), "+f"(d[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b[0]), "r"(b[1])); +} + +// Register attention core (warp MMAs, no TMEM) for up to two tokens per cluster; the tcgen05 core above that. +// Off by default: on GB300 it measured ~0.25 us slower than the tcgen05 core at M=1 (GEMV + attention chain). +#ifndef M3FA_REG_CORE +#define M3FA_REG_CORE 0 +#endif +template +struct RegCore +{ + static constexpr bool kOn = M3FA_REG_CORE != 0 && QG <= 2; +}; + +template +struct Smem +{ + static constexpr int kRows = 16 * QG; // query rows (token, head) of this cluster = UMMA N + static constexpr int kTok = BPC * kPage; // block tokens per CTA + static constexpr int kTileBytes = kPage * kHeadDim * 2; // one fp16 K or V block tile in UMMA layout + static constexpr int kK16 = 0; + static constexpr int kV16 = kK16 + BPC * kTileBytes; + static constexpr int kQ = kV16 + BPC * kTileBytes; // Q rows, UMMA K-major [row][dim] + // P^T per block, UMMA MN-major [token][row], over the block's K tile: P is written after the QK MMAs (which + // read every K tile) have completed. + static constexpr int kP = kK16; + // Packed V tiles as copied from the cache: [block][128 rows x 64 data bytes | 1 KB of 4x4-quad scales]. + static constexpr int kRawV = kQ + kRows * kHeadDim * 2; + static constexpr int kRawVBlock = kPage * 64 + kPage * 8; + static constexpr int kRowScale = kRawV + BPC * kRawVBlock; + static constexpr int kRowLse = kRowScale + BPC * kRows * 4; + // Per-warp softmax partials [block][row][4 warps]: max, then sum. + static constexpr int kRedM = (kRowLse + BPC * kRows * 4 + 15) / 16 * 16; + static constexpr int kRedL = kRedM + BPC * kRows * 16; + // Partials pushed by every CTA of the cluster: [chunk][token][local head][kOPitch] floats, lse at column 128. + // When they do not fit beside the tiles they overlay the K/V tiles (and P, Q): peers then push only once every + // CTA of the cluster has completed its PV MMAs (a cluster barrier right before the push). + // Register core: per-warp row maxima and sums [block][row][warp of the block], two arrays. + static constexpr int kRedX = kRedL + BPC * kRows * 16; + static constexpr int kRedXBytes = RegCore::kOn ? 2 * BPC * kRows * (16 / BPC) * 4 : 0; + static constexpr int kStatsEnd = kRedX + kRedXBytes; + // Register core: P (fp16, [row][token] core matrices) at the start of its block's K tile, the scaled-free O + // partial ([row][kOPitch] fp32) behind it. + static constexpr int kObufOff = 8192; + static_assert(!RegCore::kOn || (kRows * kPage * 2 <= kObufOff && kObufOff + kRows * kOPitch * 4 <= kTileBytes), + "register-core P and O staging fit in a K tile"); + static constexpr int kRecvBytes = kBlocks * QG * BPC * kOPitch * 4; + static constexpr bool kRecvAlias = kStatsEnd + kRecvBytes + 32 > 227 * 1024; + static constexpr int kRecv = kRecvAlias ? 0 : kStatsEnd; + static constexpr int kMbar = kRecvAlias ? kStatsEnd : kStatsEnd + kRecvBytes; // merge, MMA mbarriers, TMEM addr + static constexpr int kBytes = kMbar + 32; + static constexpr int kTmemCols = BPC * 2 * kRows <= 32 ? 32 + : BPC * 2 * kRows <= 64 ? 64 + : BPC * 2 * kRows <= 128 ? 128 + : BPC * 2 * kRows <= 256 ? 256 + : 512; + static_assert(BPC * 2 * kRows <= 512, "TMEM columns"); + static_assert(!kRecvAlias || kRecvBytes <= kRawV, "aliased partials must fit in the tiles, Q and P"); + static_assert(kRows * kPage * 2 <= kTileBytes, "P^T fits in the K tiles"); + static_assert(kBytes <= 227 * 1024, "shared memory budget"); + static_assert(QLEN % QG == 0 && (BPC == 1 || BPC == 2), "unsupported geometry"); +}; + +// The packed NVFP4 bytes of one 16-byte chunk (32 dims of one block row) and its two block scales. +struct RawChunk +{ + uint4 data; + uint32_t scales; // byte 0: dims [0, 16), byte 1: dims [16, 32) of the chunk +}; + +// Chunk q of a block tile: lanes 8j..8j+7 of a warp take 8 consecutive rows of one 32-dim chunk, so every 16-byte +// fp16 store phase writes 8 distinct core-matrix rows (conflict-free in the UMMA layout). +__device__ __forceinline__ int chunk_row(int q) +{ + return (q >> 5) * 8 + (q & 7); +} + +__device__ __forceinline__ int chunk_col(int q) +{ + return (q >> 3) & 3; +} + +__device__ __forceinline__ RawChunk load_chunk( + uint8_t const* data_page, uint8_t const* scale_page, int row, int cc, bool is_v) +{ + RawChunk c; + c.data = *reinterpret_cast(data_page + row * 64 + cc * 16); + if (is_v) + { + // V scales: 4x4 token-quad order [row / 4][scale column][row % 4]. + uint8_t const* sc = scale_page + (row >> 2) * 32 + (row & 3); + c.scales = static_cast(sc[(cc * 2) * 4]) | (static_cast(sc[(cc * 2 + 1) * 4]) << 8); + } + else + { + c.scales = *reinterpret_cast(scale_page + row * 8 + cc * 2); + } + return c; +} + +__device__ __forceinline__ void store_chunk_fp16(uint8_t* tile, int row, int cc, RawChunk const& c, bool valid) +{ + uint32_t const s0 = e4m3_to_h2_splat(c.scales & 0xFFu); + uint32_t const s1 = e4m3_to_h2_splat((c.scales >> 8) & 0xFFu); + uint32_t const wd[4] = {c.data.x, c.data.y, c.data.z, c.data.w}; + uint32_t out[16]; +#pragma unroll + for (int i = 0; i < 16; ++i) + { + uint32_t const b = (wd[i >> 2] >> (8 * (i & 3))) & 0xFFu; + out[i] = valid ? hmul2_u(e2m1x2_to_h2(b), i < 8 ? s0 : s1) : 0u; + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + *reinterpret_cast(tile + kmaj_off(row, cc * 32 + j * 8)) + = make_uint4(out[4 * j], out[4 * j + 1], out[4 * j + 2], out[4 * j + 3]); + } +} + +// ---------------- Fused QKV projection: block-scaled MXFP8 on tcgen05 (N=2560, K=6144) ---------------- +// out[M, 2560] = mxfp8(x)[M, 6144] @ W^T, the computation of the CUTLASS MXFP8 GEMM: W is the A operand (128 output +// rows per MMA), the quantized activation the B operand (8 token columns), FP32 accumulation in TMEM. A unit is one +// (128-row tile, K split) pair; the 20 x kQkvSplits units run on the last CTAs of the grid. Before the PDL wait a +// CTA streams its unit's weight tiles and scale atoms through a shared-memory ring into TMEM; after it, it quantizes +// its x slice (mxfp8_quantize math), runs the MMAs and stores its FP32 partial rows. +// +// Hand-off without a flag: the partial buffers are double buffered by call parity and hold a sentinel (an FP32 NaN +// the MMA never produces) at rest, so a consumer polls the partials themselves. The producers of call n reset the +// other parity's buffers after their PDL wait; call n - 1, the last reader of those, is complete by then. +constexpr int kQkvN = 2560; +constexpr int kQkvK = 6144; +constexpr int kQkvKTiles = kQkvK / 128; +constexpr int kQkvRowTiles = kQkvN / 128; +#ifndef M3FA_QKV_SPLITS +#define M3FA_QKV_SPLITS 4 +#endif +#ifndef M3FA_QKV_RING +#define M3FA_QKV_RING kQkvMaxTiles // every tile of the unit in flight at once +#endif +#ifndef M3FA_QKV_NACC +#define M3FA_QKV_NACC 1 +#endif +#ifndef M3FA_QKV_SCALED_MMA +#define M3FA_QKV_SCALED_MMA 1 // 0: unscaled per-block MMAs with the scales applied in the epilogue (see below) +#endif +#ifndef M3FA_QKV_PIPE +// 1: the weight TMA is issued first at entry; the activation has its own region, copied right after the PDL wait; +// each weight tile's MMAs issue as soon as it lands (v52's order). 0: tiles into TMEM first, activation over the ring. +#define M3FA_QKV_PIPE 1 +#endif +#ifndef M3FA_QKV_A_SMEM +#define M3FA_QKV_A_SMEM 1 // scaled MMAs only; 1: they read the weight tiles from shared memory (no copy into TMEM) +#endif +// Default: block-scaled kind::mxf8f6f4 MMAs reading A (the weight tiles) in place from the ring, issued per tile as +// it lands (M3FA_QKV_PIPE), one accumulator; M3FA_QKV_A_SMEM=0 copies each tile into TMEM first. Alternative +// (M3FA_QKV_SCALED_MMA=0): one unscaled kind::f8f6f4 MMA per 32-wide K block into its own accumulator, A read in place +// from shared memory, the epilogue applying both UE8M0 scales. On GB300 an M128 x N8 x K32 MMA occupies the tensor pipe +// ~105 cycles block-scaled and ~60 unscaled, independent of N up to 64; the unscaled form's epilogue (320 accumulator +// columns) costs more than it saves. +constexpr bool kQkvScaled = M3FA_QKV_SCALED_MMA != 0; +constexpr bool kQkvASmem = !kQkvScaled || M3FA_QKV_A_SMEM != 0; +constexpr int kQkvSplits = M3FA_QKV_SPLITS; +constexpr int kQkvUnits = kQkvRowTiles * kQkvSplits; +constexpr int kQkvMaxTiles = (kQkvKTiles + kQkvSplits - 1) / kQkvSplits; +constexpr int kQkvRing = M3FA_QKV_RING; +constexpr int kQkvNacc = M3FA_QKV_NACC; +constexpr int kQkvGrid = 112; // 7 clusters of 16 (or 14 of 8): one wave with the attention +constexpr int kMaxTok = 8; +constexpr int kQkvWTile = 128 * 128; // E4M3 bytes of one 128 x 128 weight tile +constexpr int kQkvSfAtom = 512; // one 128-row x 4-block UE8M0 scale atom +constexpr uint32_t kQkvSentinel = 0xFFFFFFFFu; +constexpr int64_t kQkvPartStride = static_cast(kQkvSplits) * kMaxTok * kQkvN; // one parity +// TMEM columns of the one 512-column allocation of a CTA: attention [0, 64), then the GEMM's accumulators, scale +// atoms (A per tile, B per tile) and the A tiles (32 columns of 128 x 128 E4M3 each). +constexpr int kTmemAll = 512; +// Megakernel: the attention CTAs run no projection unit, so the GEMM starts at column 0 of a projection CTA. +constexpr int kTAcc = 0; +constexpr int kTSfa = kTAcc + 16; +constexpr int kTSfb = kTSfa + 4 * kQkvMaxTiles; +constexpr int kTA = (kTSfb + 4 * kQkvMaxTiles + 31) / 32 * 32; +static_assert(!kQkvScaled || kTA + 32 * kQkvMaxTiles <= kTmemAll, "TMEM columns"); +static_assert(kQkvScaled || kTAcc + 32 * kQkvMaxTiles <= kTmemAll, "TMEM columns (one accumulator per K block)"); +static_assert(!kQkvASmem || kQkvRing == kQkvMaxTiles, "shared-memory A needs every tile resident"); +static_assert(kQkvUnits <= kQkvGrid, "at most one unit per CTA"); +static_assert(kQkvNacc == 1 || kQkvNacc == 2 || kQkvNacc == 4, "accumulator count"); +static_assert(!M3FA_QKV_PIPE || kQkvScaled, "the pipelined projection uses the block-scaled MMAs"); +constexpr int kQkvOwnRows = 128 / kQkvSplits; // rows of a tile whose split-K sum a CTA owns +// Megakernel o_proj (v52 m3_mx_gemv for [6144, 2048]: split 2, contiguous k-tiles, one accumulator): unit +// o = CTA - 16 (the last 96 CTAs), row tile o / 2, split o % 2. +constexpr int kOpN = 6144; +constexpr int kOpK = 2048; +constexpr int kOpKTiles = kOpK / 128; +constexpr int kOpSplits = 2; +constexpr int kOpTiles = kOpKTiles / kOpSplits; +constexpr int kOpOwnRows = 128 / kOpSplits; +constexpr int kOpFirstCta = kQkvGrid - kOpN / 128 * kOpSplits; // 16 +constexpr int kOpSfBytes = kOpKTiles * 512; // one parity of the MXFP8 hand-off's scales +#ifndef M3FA_OP_L2PF +#define M3FA_OP_L2PF \ + 0 // 1: QKV CTAs prefetch their o_proj weight slice into L2 before the PDL wait (competes + // with the QKV weight stream for HBM behind the MoE) +#endif +// o_proj TMEM columns (after the QKV unit's MMAs completed): accumulator, A / B scale atoms, A (8 tiles x 32) +constexpr int kOAcc = 0; +constexpr int kOSfa = 16; +constexpr int kOSfb = kOSfa + 4 * kOpTiles; +constexpr int kOA = 128; +static_assert(kOA + 32 * kOpTiles <= kTmemAll && kOSfb + 4 * kOpTiles <= kOA, "o_proj TMEM columns"); +constexpr int kQkvMbStride = kMaxTok + 4; // mailbox row pitch (fp32 words) + +__host__ __device__ constexpr int qkv_tile0(int s) +{ + return s * kQkvKTiles / kQkvSplits; +} + +__device__ __forceinline__ int qkv_unit() +{ + int const u = kQkvGrid - 1 - static_cast(blockIdx.x); + return u < kQkvUnits ? u : -1; +} + +// Attention layout plus, for the fused projection, the weight ring in front (the attention tiles reuse that space +// once the ring has drained into TMEM); the activation tiles, the pre-wait V copy and the barriers sit behind. +template +struct SmemX : Smem +{ + using B = Smem; + static constexpr int kRingW = 0; // [ring][128 rows][128 B], 128-byte swizzle + static constexpr int kRingSf = kRingW + kQkvRing * kQkvWTile; // [ring][512 B] + static constexpr int kRingEnd = kRingSf + kQkvRing * kQkvSfAtom; + // The activation tiles overlay the ring: written after the PDL wait, once every weight tile went on into TMEM. + static constexpr int kXt = kRingW; // [tile][8][128 B] + static constexpr int kXsf = kXt + kQkvMaxTiles * 1024; // [tile][512 B] + static constexpr int kXsc = (kRingEnd + 127) / 128 * 128; + // Peers push into recv at any time (an empty CTA right away), so it stays clear of the ring. + static constexpr int kRecv = FQKV ? kXsc : B::kRecv; + static constexpr int kRawV = FQKV ? (kRecv + kBlocks * QG * BPC * kOPitch * 4 + 127) / 128 * 128 : B::kRawV; + // Pipelined projection: the activation tiles and scale atoms in their own region (a projection CTA runs no + // attention, so it overlays the attention's partial-O and V regions), 1024-byte aligned for the swizzle. + static constexpr int kXA = FQKV ? (kRecv + 1023) / 1024 * 1024 : 0; + static constexpr int kXASf = kXA + kQkvMaxTiles * 1024; + static constexpr int kXAEnd = kXASf + kQkvMaxTiles * kQkvSfAtom; + static constexpr int kMbarA = FQKV ? kRawV + BPC * B::kRawVBlock : B::kMbar; + static constexpr int kMbar = FQKV && M3FA_QKV_PIPE && kXAEnd > kMbarA ? (kXAEnd + 127) / 128 * 128 : kMbarA; + static constexpr int kGBar = kMbar + 32; // full[ring], empty[ring], a_ready, acc_done, tile_done[tiles] + static constexpr int kRedBar = kGBar + 8 * (2 * kQkvRing + 2 + kQkvMaxTiles); // split-K partials in + static constexpr int kOBar = kRedBar + 8; // o_proj: full[8], a_ready, acc_done, reduce + static constexpr int kMailbox = (kOBar + 8 * (kOpTiles + 3) + 15) / 16 * 16; // [split][own row][pitch] fp32 + // o_proj weight ring = ring tiles [0, 8); its activation tiles, scale atoms and mailbox behind it. + static constexpr int kOXt = kRingW + kOpTiles * kQkvWTile; + static constexpr int kOXsf = kOXt + kOpTiles * 1024; + static constexpr int kOMailbox = kOXsf + kOpTiles * kQkvSfAtom; + static_assert(!FQKV || kOMailbox + kOpSplits * kOpOwnRows * kQkvMbStride * 4 <= kRingW + kQkvRing * kQkvWTile, + "o_proj activation and mailbox fit behind its ring"); + static constexpr int kBytes = FQKV ? kMailbox + kQkvSplits * kQkvOwnRows * kQkvMbStride * 4 : B::kBytes; + static constexpr int kTmemCols = FQKV ? kTmemAll : B::kTmemCols; + static constexpr bool kRecvAlias = !FQKV && B::kRecvAlias; + // The epilogue reads the weight scale atoms while the attention already writes its tiles. + static_assert(!FQKV || kQkvScaled || B::kRecv <= kRingSf, "attention regions overlap the scale atoms"); + // The other attention regions overlay the ring: a unit's CTA writes them only once the ring has drained. + static_assert(kBytes <= 227 * 1024, "shared memory budget"); +}; + +__device__ __forceinline__ void mbar_init(uint32_t bar, uint32_t count) +{ + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;\n" ::"r"(bar), "r"(count) : "memory"); +} + +__device__ __forceinline__ void mbar_expect_tx(uint32_t bar, uint32_t bytes) +{ + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(bar), "r"(bytes) : "memory"); +} + +__device__ __forceinline__ void bulk_g2s(void* dst, void const* src, uint32_t bytes, uint32_t bar) +{ + asm volatile( + "cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes.L2::cache_hint [%0], [%1], %2, [%3], " + "%4;\n" ::"r"(smem_u32(dst)), + "l"(src), "r"(bytes), "r"(bar), "l"(0x12F0000000000000ull) // EVICT_FIRST: weights are read once per layer + : "memory"); +} + +__device__ __forceinline__ void tma_load_2d(uint32_t dst, void const* map, int c0, int c1, uint32_t bar) +{ + asm volatile( + "cp.async.bulk.tensor.2d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.L2::cache_hint" + " [%0], [%1, {%2, %3}], [%4], %5;\n" ::"r"(dst), + "l"(map), "r"(c0), "r"(c1), "r"(bar), "l"(0x12F0000000000000ull) + : "memory"); +} + +// K-major operand in the 128-byte swizzle: rows of 128 bytes, 8-row groups 1024 bytes apart. +__device__ __forceinline__ uint64_t umma_desc_sw128(uint32_t addr) +{ + return umma_desc(addr, 16, 1024) | (2ull << 61); +} + +__device__ __forceinline__ void tmem_cp_128x256b(uint32_t taddr, uint64_t desc) +{ + asm volatile("tcgen05.cp.cta_group::1.128x256b [%0], %1;\n" ::"r"(taddr), "l"(desc)); +} + +__device__ __forceinline__ void tmem_cp_sf(uint32_t taddr, uint32_t saddr) +{ + asm volatile("tcgen05.cp.cta_group::1.32x128b.warpx4 [%0], %1;\n" ::"r"(taddr), "l"(umma_desc(saddr, 16, 128))); +} + +// Block-scaled MXFP8 instruction descriptor: E4M3 x E4M3, UE8M0 scales, M = 128, N = 8, scale byte kb. +__host__ __device__ constexpr uint32_t qkv_idesc(int kb) +{ + return (static_cast(kb) << 4) | ((8u >> 3) << 17) | (1u << 23) | ((128u >> 7) << 27) + | (static_cast(kb) << 29); +} + +__device__ __forceinline__ void umma_mx( + uint32_t tmem_d, uint32_t tmem_a, uint64_t bdesc, uint32_t idesc, uint32_t sfa, uint32_t sfb, int accumulate) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, %6, 0;\n" + " tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X [%0], [%1], %2, %3, [%4], [%5], p;\n}\n" :: + "r"(tmem_d), + "r"(tmem_a), "l"(bdesc), "r"(idesc), "r"(sfa), "r"(sfb), "r"(accumulate)); +} + +__device__ __forceinline__ float pow2_from_ue8m0(uint32_t e) +{ + return __uint_as_float(e == 0 ? 0x00400000u : (e << 23)); +} + +// Unscaled E4M3 x E4M3 MMA, FP32 accumulator: M = 128, N = 8, K = 32, a fresh accumulator. +constexpr uint32_t kQkvIdescF8 = (1u << 4) | ((8u >> 3) << 17) | ((128u >> 4) << 24); + +__device__ __forceinline__ void umma_f8_ss(uint32_t tmem_d, uint64_t adesc, uint64_t bdesc) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, 0, 0;\n tcgen05.mma.cta_group::1.kind::f8f6f4 [%0], %1, %2, %3, p;\n}\n" :: + "r"(tmem_d), + "l"(adesc), "l"(bdesc), "r"(kQkvIdescF8)); +} + +__device__ __forceinline__ void tmem_ld16_nowait(uint32_t addr, float* v) +{ + uint32_t r[16]; + asm volatile( + "tcgen05.ld.sync.aligned.32x32b.x16.b32 {%0, %1, %2, %3, %4, %5, %6, %7, %8, %9, %10, %11, %12, %13, %14, " + "%15}, [%16];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]), "=r"(r[4]), "=r"(r[5]), "=r"(r[6]), "=r"(r[7]), "=r"(r[8]), + "=r"(r[9]), "=r"(r[10]), "=r"(r[11]), "=r"(r[12]), "=r"(r[13]), "=r"(r[14]), "=r"(r[15]) + : "r"(addr)); +#pragma unroll + for (int i = 0; i < 16; ++i) + { + v[i] = __uint_as_float(r[i]); + } +} + +__device__ __forceinline__ void umma_mx_ss( + uint32_t tmem_d, uint64_t adesc, uint64_t bdesc, uint32_t idesc, uint32_t sfa, uint32_t sfb, int accumulate) +{ + asm volatile( + "{\n .reg .pred p;\n setp.ne.b32 p, %6, 0;\n" + " tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X [%0], %1, %2, %3, [%4], [%5], p;\n}\n" :: + "r"(tmem_d), + "l"(adesc), "l"(bdesc), "r"(idesc), "r"(sfa), "r"(sfb), "r"(accumulate)); +} + +__device__ __forceinline__ bool elect_one() +{ + uint32_t pred; + asm volatile("{\n .reg .pred P;\n elect.sync _|P, 0xffffffff;\n selp.u32 %0, 1, 0, P;\n}\n" : "=r"(pred)); + return pred != 0; +} + +__device__ __forceinline__ int qkv_parity(Params const& p) +{ + int v; + asm volatile("ld.relaxed.gpu.global.s32 %0, [%1];\n" : "=r"(v) : "l"(p.counters) : "memory"); + return v & 1; +} + +// Barriers and TMEM (warp 1), then the first ring stages (warp 0). Warps 0 and 1 of every CTA call it. +template +__device__ __forceinline__ void qkv_setup( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int warp, int lane, int u) +{ + constexpr bool kPipe = M3FA_QKV_PIPE != 0; + if (kPipe && warp == 0 && lane == 0 && u >= 0) + { + // First thing at entry: this unit's whole weight slice. The issuing thread initializes the ring's full + // barriers itself; every waiter on them passes bar.sync 8 below, which this thread reaches after the init. + asm volatile("prefetch.tensormap [%0];\n" ::"l"(&p.wmap) : "memory"); + int const rt = u / kQkvSplits; + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + for (int i = 0; i < kQkvRing; ++i) + { + mbar_init(gbar + 8 * i, 1); + } + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + for (int i = 0; i < kQkvRing && i < nt; ++i) + { + uint32_t const full = gbar + 8 * i; + mbar_expect_tx(full, kQkvWTile + kQkvSfAtom); + if (p.wpre != nullptr) + { + // Pre-tiled weight: the tile's 16 KB shared-memory image is contiguous (one bulk copy). + bulk_g2s(smem + SX::kRingW + i * kQkvWTile, + p.wpre + (static_cast(rt) * kQkvKTiles + kt0 + i) * kQkvWTile, kQkvWTile, full); + } + else + { + tma_load_2d(smem_u32(smem + SX::kRingW + i * kQkvWTile), &p.wmap, (kt0 + i) * 128, rt * 128, full); + } + bulk_g2s(smem + SX::kRingSf + i * kQkvSfAtom, + p.wsf + (static_cast(rt) * kQkvKTiles + kt0 + i) * kQkvSfAtom, kQkvSfAtom, full); + } + M3FA_PROF_T(34, 0); + } + if (warp == 1) + { + asm volatile("tcgen05.alloc.cta_group::1.sync.aligned.shared::cta.b32 [%0], %1;\n" ::"r"(smem_u32(tmem_slot)), + "r"(kTmemAll) + : "memory"); + asm volatile("tcgen05.relinquish_alloc_permit.cta_group::1.sync.aligned;\n" ::: "memory"); + M3FA_PROF_T(35, 32); + if (lane == 0) + { + for (int i = (kPipe && u >= 0) ? kQkvRing : 0; i < 2 * kQkvRing + 2 + kQkvMaxTiles; ++i) + { + // PIPE 2: tile_done[0] (unused by the scaled MMAs) is act_ready, one arrival per thread outside warp 1 + mbar_init(gbar + 8 * i, M3FA_QKV_PIPE == 2 && i == 2 * kQkvRing + 2 ? kThreads - 32 : 1); + } + // The split-K reduce barrier: this CTA's own arrival carries the peers' partial bytes. + uint32_t const red = gbar + 8 * (2 * kQkvRing + 2 + kQkvMaxTiles); + mbar_init(red, 1); + mbar_expect_tx(red, (kQkvSplits - 1) * kQkvOwnRows * kMaxTok * 4); + uint32_t const ob = smem_u32(smem + SX::kOBar); + for (int i = 0; i < kOpTiles + 3; ++i) + { + mbar_init(ob + 8 * i, 1); + } + mbar_expect_tx(ob + 8 * (kOpTiles + 2), (kOpSplits - 1) * kOpOwnRows * kMaxTok * 4); + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + } + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + asm volatile("bar.sync 8, 64;\n" ::: "memory"); + if (!kPipe && warp == 0 && lane == 0 && u >= 0) + { + int const rt = u / kQkvSplits; + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + for (int i = 0; i < kQkvRing && i < nt; ++i) + { + uint32_t const full = gbar + 8 * i; + mbar_expect_tx(full, kQkvWTile + kQkvSfAtom); + tma_load_2d(smem_u32(smem + SX::kRingW + i * kQkvWTile), &p.wmap, (kt0 + i) * 128, rt * 128, full); + bulk_g2s(smem + SX::kRingSf + i * kQkvSfAtom, + p.wsf + (static_cast(rt) * kQkvKTiles + kt0 + i) * kQkvSfAtom, kQkvSfAtom, full); + } + M3FA_PROF_T(34, 0); + } +} + +// The rest of the ring (warp 0, lane 0) and the copies into TMEM (warp 1, lane 0); before the PDL wait. +template +__device__ __forceinline__ void qkv_stream_lane( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int warp, int u) +{ + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + uint32_t const full = gbar; + uint32_t const a_ready = gbar + 16 * kQkvRing; + if (warp == 0 || !kQkvScaled) + { + return; // every tile was issued in qkv_setup; the unscaled MMAs read them in place + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm = *tmem_slot; + // Fully unrolled with compile-time offsets: tcgen05 operands stay in uniform registers (a per-instruction + // conversion of computed operands costs ~150 cycles per MMA or copy). + uint64_t const w0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + uint32_t const sf0 = smem_u32(smem + SX::kRingSf); +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + mbar_wait_cta(full + 8 * i, 0); + if (i == 0) + { + M3FA_PROF_T(30, 32); + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if constexpr (!kQkvASmem) + { +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b(tm + kTA + (i * 4 + kb) * 8, w0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + } + tmem_cp_sf(tm + kTSfa + i * 4, sf0 + i * kQkvSfAtom); + } + } + M3FA_PROF_T(31, 32); + umma_commit(a_ready); +} + +__device__ __forceinline__ bool mbar_test(uint32_t bar, uint32_t parity) +{ + uint32_t ready; + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.shared::cta.b64 p, [%1], %2;\n selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(bar), "r"(parity) + : "memory"); + return ready != 0; +} + +// PIPE 2, warp 1's elected thread from before the PDL wait to the last MMA. Tiles that land before the activation +// is ready go on into TMEM (A and scale atom), off the critical path; once the activation is in (act_ready), the +// MMAs issue in tile order: the copied tiles with A in TMEM, every later tile as it lands with A read in place from +// the ring. The MMA sequence (and so the accumulation order) is the same either way. +template +__device__ __forceinline__ void qkv_hybrid(Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int u) +{ + int const kt0 = qkv_tile0(u % kQkvSplits); + int const nt = qkv_tile0(u % kQkvSplits + 1) - kt0; + uint32_t const act_ready = gbar + 8 * (2 * kQkvRing + 2); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm = *tmem_slot; + uint64_t const w0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + uint64_t const b0 = umma_desc_sw128(smem_u32(smem + SX::kXA)); + uint32_t const wsf0 = smem_u32(smem + SX::kRingSf); + uint32_t const xsf0 = smem_u32(smem + SX::kXASf); + int copied = 0; // tiles [0, copied) are in TMEM + bool act = false; +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt && !act) + { + while (!mbar_test(gbar + 8 * i, 0)) + { + if (mbar_test(act_ready, 0)) + { + act = true; + break; + } + } + if (!act) + { + if (i == 0) + { + M3FA_PROF_T(30, 32); + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b(tm + kTA + (i * 4 + kb) * 8, w0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + tmem_cp_sf(tm + kTSfa + i * 4, wsf0 + i * kQkvSfAtom); + copied = i + 1; + } + } + } + while (!act) + { + act = mbar_test(act_ready, 0); + } + M3FA_PROF_T(21, 32); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + bool const in_tmem = i < copied; + if (!in_tmem) + { + mbar_wait_epi(gbar + 8 * i, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + tmem_cp_sf(tm + kTSfa + i * 4, wsf0 + i * kQkvSfAtom); + } + uint32_t const sfb = tm + kTSfb + i * 4; + tmem_cp_sf(sfb, xsf0 + i * kQkvSfAtom); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + int const j = i * 4 + kb; + uint64_t const bd = b0 + ((i * 1024 + kb * 32) >> 4); + if (in_tmem) + { + umma_mx(tm + kTAcc, tm + kTA + j * 8, bd, qkv_idesc(kb), tm + kTSfa + i * 4, sfb, j > 0); + } + else + { + umma_mx_ss(tm + kTAcc, w0 + ((i * kQkvWTile + kb * 32) >> 4), bd, qkv_idesc(kb), tm + kTSfa + i * 4, + sfb, j > 0); + } + } + } + } + M3FA_PROF_T(22, 32); + M3FA_PROF_T(31, 32); + umma_commit(gbar + 16 * kQkvRing); // a_ready (for its waiters) + umma_commit(gbar + 16 * kQkvRing + 8); // acc_done +} + +// The other lanes of warps 0 and 1 hold at __syncwarp rather than at griddepcontrol.wait, which would stall the +// whole warp (lane 0's loop included) until the producer grid completes. +template +__device__ __forceinline__ void qkv_stream( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int warp, int lane, int u) +{ + if (M3FA_QKV_PIPE == 2 && u >= 0 && warp == 1) + { + if (elect_one()) + { + qkv_hybrid(p, smem, gbar, tmem_slot, u); + } + __syncwarp(); + return; + } + if (u < 0 || warp > 1 || M3FA_QKV_PIPE) + { + return; // pipelined: the MMA thread takes each tile as it lands + } + if (lane == 0) + { + qkv_stream_lane(p, smem, gbar, tmem_slot, warp, u); + } + __syncwarp(); +} + +// After the PDL wait. Every thread quantizes a piece of the x slice into the B tiles (named barrier 9); warp 1 then +// issues the scale copies and the MMAs. +template +__device__ __forceinline__ void qkv_mma( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int tid, int u) +{ + int const warp = tid >> 5; + int const s = u % kQkvSplits; + int const kt0 = qkv_tile0(s); + int const nt = qkv_tile0(s + 1) - kt0; + { + // The MXFP8 activation slice (E4M3 rows of the unit's k-tiles in 128-byte-swizzled B tiles, and their scale + // atoms), copied with plain 16-byte loads over the overlaid ring once its tiles went on into TMEM: every load + // first, then the stores (one L2 round trip). + constexpr int kXDst = M3FA_QKV_PIPE ? SX::kXA : SX::kXt; + constexpr int kXSfDst = M3FA_QKV_PIPE ? SX::kXASf : SX::kXsf; + if (!M3FA_QKV_PIPE) + { + mbar_wait_cta(gbar + 16 * kQkvRing, 0); // a_ready: the ring's weight tiles are in TMEM + } + constexpr int kXItems = kQkvMaxTiles * 8 * kMaxTok; // (tile, 16-byte chunk, token) + constexpr int kSfItems = kQkvMaxTiles * (kQkvSfAtom / 16); + constexpr bool kHyb = M3FA_QKV_PIPE == 2; + constexpr int kCopiers = kHyb ? kThreads - 32 : kThreads; // PIPE 2: warp 1 is issuing the tiles + if (kHyb && warp == 1) + { + return; + } + int const ctid = kHyb && tid >= 64 ? tid - 32 : tid; + constexpr int kPer = (kXItems + kSfItems + kCopiers - 1) / kCopiers; + uint4 v[kPer]; + int dst[kPer]; +#pragma unroll + for (int k = 0; k < kPer; ++k) + { + int const q = ctid + kCopiers * k; + dst[k] = -1; + if (q < kXItems) + { + int const t = q % kMaxTok; + int const c = (q / kMaxTok) % 8; + int const i = q / (kMaxTok * 8); + if (i < nt && t < p.num_tokens) + { + v[k] = *reinterpret_cast( + p.xq + static_cast(t) * p.ldxq + (kt0 + i) * 128 + c * 16); + dst[k] = kXDst + i * 1024 + t * 128 + ((c ^ t) * 16); + } + } + else if (q < kXItems + kSfItems) + { + int const j = q - kXItems; + int const i = j / (kQkvSfAtom / 16); + int const c = j % (kQkvSfAtom / 16); + if (i < nt) + { + v[k] = *reinterpret_cast( + p.xsf_act + static_cast(kt0 + i) * kQkvSfAtom + c * 16); + dst[k] = kXSfDst + i * kQkvSfAtom + c * 16; + } + } + } +#pragma unroll + for (int k = 0; k < kPer; ++k) + { + if (dst[k] >= 0) + { + *reinterpret_cast(smem + dst[k]) = v[k]; + } + } + asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory"); + if (kHyb) + { + asm volatile("mbarrier.arrive.release.cta.shared::cta.b64 _, [%0];\n" ::"r"(gbar + 8 * (2 * kQkvRing + 2)) + : "memory"); + return; + } + if (warp != 1) + { + asm volatile("bar.arrive 9, %0;\n" ::"n"(kThreads) : "memory"); + } + } + if (warp == 1) + { + M3FA_PROF_T(32, 32); + asm volatile("bar.sync 9, %0;\n" ::"n"(kThreads) : "memory"); + M3FA_PROF_T(21, 32); + // Operands computed by the whole warp (warp-uniform, kept in uniform registers); one elected lane issues. + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm = *tmem_slot; + uint64_t const b0 = umma_desc_sw128(smem_u32(smem + (M3FA_QKV_PIPE ? SX::kXA : SX::kXt))); + uint64_t const a0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + if (elect_one()) + { + M3FA_CLK(41); + if constexpr (!kQkvScaled) + { +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + mbar_wait_cta(gbar + 8 * i, 0); // weight tile i landed + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + umma_f8_ss(tm + kTAcc + (i * 4 + kb) * 8, a0 + ((i * kQkvWTile + kb * 32) >> 4), + b0 + ((i * 1024 + kb * 32) >> 4)); + } + umma_commit(gbar + 8 * (2 * kQkvRing + 2 + i)); // tile i's blocks are in TMEM + } + } + } + else + { + uint32_t const xsf0 = smem_u32(smem + (M3FA_QKV_PIPE ? SX::kXASf : SX::kXsf)); + uint32_t const wsf0 = smem_u32(smem + SX::kRingSf); +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + if constexpr (M3FA_QKV_PIPE != 0) + { + // Tile i as it lands: its A copy and scale atom into TMEM, then its MMAs (issue order). + mbar_wait_cta(gbar + 8 * i, 0); + if (i == 0) + { + M3FA_PROF_T(30, 32); + } + if (i == nt - 1) + { + M3FA_CLK(47); // the unit's last weight tile landed (clock64 builds) + } + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if constexpr (!kQkvASmem) + { +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b( + tm + kTA + (i * 4 + kb) * 8, a0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + } + tmem_cp_sf(tm + kTSfa + i * 4, wsf0 + i * kQkvSfAtom); + } + uint32_t const sfb = tm + kTSfb + i * 4; + tmem_cp_sf(sfb, xsf0 + i * kQkvSfAtom); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + int const j = i * 4 + kb; + uint64_t const bd = b0 + ((i * 1024 + kb * 32) >> 4); + if constexpr (kQkvASmem) + { + umma_mx_ss(tm + kTAcc + (j % kQkvNacc) * 8, a0 + ((i * kQkvWTile + kb * 32) >> 4), bd, + qkv_idesc(kb), tm + kTSfa + i * 4, sfb, j >= kQkvNacc); + } + else + { + umma_mx(tm + kTAcc + (j % kQkvNacc) * 8, tm + kTA + j * 8, bd, qkv_idesc(kb), + tm + kTSfa + i * 4, sfb, j >= kQkvNacc); + } + } + } + } + } + M3FA_CLK(42); + if (M3FA_QKV_PIPE) + { + M3FA_PROF_T(31, 32); + umma_commit(gbar + 16 * kQkvRing); // a_ready (kept for its waiters: every tile is consumed) + } + umma_commit(gbar + 16 * kQkvRing + 8); // acc_done + } + M3FA_PROF_T(22, 32); + } +} + +// Warps 16..19 (TMEM lanes 32 (warp % 4) + lane = the unit's rows): partial rows out, the other parity reset. +template +__device__ __forceinline__ void qkv_epilogue( + Params const& p, uint8_t* smem, uint32_t gbar, uint32_t* tmem_slot, int tid, int u, int par, unsigned int calls) +{ + int const warp = tid >> 5; + int const lane = tid & 31; + if constexpr (kQkvScaled) + { + mbar_wait_epi(gbar + 16 * kQkvRing + 8, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + } + M3FA_PROF_T(24, 512); + if (tid == 512) + { + M3FA_CLK(43); + } + int const r = (warp & 3) * 32 + lane; // row within the unit's tile + uint32_t const taddr = *tmem_slot + (((warp & 3) * 32) << 16) + kTAcc; + float val[kMaxTok]; + if constexpr (kQkvScaled) + { + float v[32]; + tmem_ld16(taddr, *reinterpret_cast(v)); + if constexpr (kQkvNacc == 4) + { + tmem_ld16(taddr + 16, *reinterpret_cast(v + 16)); + } +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + val[t] = kQkvNacc == 4 ? (v[t] + v[8 + t]) + (v[16 + t] + v[24 + t]) + : kQkvNacc == 2 ? v[t] + v[8 + t] + : v[t]; + } + } + else + { + // Sum over K blocks in order: block result x (weight scale x activation scale). + int const nt = qkv_tile0(u % kQkvSplits + 1) - qkv_tile0(u % kQkvSplits); + float const* xsc = reinterpret_cast(smem + SX::kXsc); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + val[t] = 0.0f; + } +#pragma unroll + for (int i = 0; i < kQkvMaxTiles; ++i) + { + if (i < nt) + { + // Tile i's four blocks as soon as its MMAs complete: the epilogue trails the MMAs by one tile. + mbar_wait_cta(gbar + 8 * (2 * kQkvRing + 2 + i), 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + float d[32]; + tmem_ld16_nowait(taddr + i * 32, d); + tmem_ld16_nowait(taddr + i * 32 + 16, d + 16); + uint32_t const wsw = *reinterpret_cast( + smem + SX::kRingSf + i * kQkvSfAtom + (r % 32) * 16 + (r / 32) * 4); + asm volatile("tcgen05.wait::ld.sync.aligned;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + float const ws = pow2_from_ue8m0((wsw >> (8 * kb)) & 0xFFu); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + val[t] = fmaf(d[kb * 8 + t], ws * xsc[(i * 4 + kb) * kMaxTok + t], val[t]); + } + } + } + } + } + // v52's split-K reduce: split s owns rows [32 s, 32 s + 32) of the tile; the other splits push their fp32 partials + // of those rows into its mailbox (st.async, completing its reduce barrier's bytes); the owner adds the splits in + // rank order from 0.0 and stores bf16. + int const s = u % kQkvSplits; + int const owner = r / kQkvOwnRows; + int const local = r % kQkvOwnRows; + uint32_t const red = gbar + 8 * (2 * kQkvRing + 2 + kQkvMaxTiles); + uint32_t cta_rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(cta_rank)); + // Units are numbered down from the last CTA: split q of this tile is CTA rank cta_rank - (q - s). + if (owner != s) + { + uint32_t const peer = cta_rank + s - owner; + uint32_t const dst + = mapa_shared(smem_u32(smem + SX::kMailbox) + ((s * kQkvOwnRows + local) * kQkvMbStride) * 4, peer); + uint32_t const pbar = mapa_shared(red, peer); +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + asm volatile( + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [%0], {%1, %2, %3, %4}, [%5];\n" ::"r"( + dst + 16 * g), + "r"(__float_as_uint(val[4 * g])), "r"(__float_as_uint(val[4 * g + 1])), + "r"(__float_as_uint(val[4 * g + 2])), "r"(__float_as_uint(val[4 * g + 3])), "r"(pbar) + : "memory"); + } + } + else + { + M3FA_PROF_T(26, tid - lane); + // test_wait (no suspend): completions by peers' st.async bytes wake a suspended try_wait late. + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(red) + : "memory"); + } + M3FA_PROF_T(27, tid - lane); + float const* mb = reinterpret_cast(smem + SX::kMailbox); + int const row = (u / kQkvSplits) * 128 + r; + // Lamport hand-off: this call's rows go to buffer (calls & 1) with relaxed stores, the consumers poll the + // values themselves (0xFFFF, a NaN the rounding of a finite sum never produces, marks "not yet"); the other + // buffer, last read by the previous call (complete: this grid passed its PDL wait), is reset to the marker. + uint16_t* cur = reinterpret_cast(p.qkv_out) + (calls & 1u) * kMaxTok * kQkvN; + uint16_t* nxt = reinterpret_cast(p.qkv_out) + ((calls & 1u) ^ 1u) * kMaxTok * kQkvN; +#pragma unroll + float qsum[kMaxTok]; +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + float4 part[kQkvSplits]; +#pragma unroll + for (int q = 0; q < kQkvSplits; ++q) + { + part[q] = *reinterpret_cast(mb + (q * kQkvOwnRows + local) * kQkvMbStride + 4 * g); + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float sum = 0.0F; +#pragma unroll + for (int q = 0; q < kQkvSplits; ++q) + { + float const pj = j == 0 ? part[q].x : j == 1 ? part[q].y : j == 2 ? part[q].z : part[q].w; + sum = sum + (q == s ? val[4 * g + j] : pj); + } + qsum[4 * g + j] = sum; + } + } +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + float const sum = qsum[t]; + if (t < p.num_tokens) + { + __nv_bfloat16 const b = __float2bfloat16_rn(sum); + uint16_t bits = *reinterpret_cast(&b); + // Never the "not yet" marker: a 0xFFFF NaN sum is stored as the NaN 0x7FFF. Only a NaN projection row + // changes (its K/V cache bytes then differ from v52 + p6 in the NaN's encoding); finite rows are exact. + bits = bits == 0xFFFFu ? static_cast(0x7FFFu) : bits; +#ifndef M3FA_QKV_REL +#define M3FA_QKV_REL 0 // 1: the row store is a release (tried: whether it becomes visible to the pollers sooner) +#endif + if (M3FA_QKV_REL) + { + asm volatile( + "st.release.gpu.global.u16 [%0], %1;\n" ::"l"(cur + static_cast(t) * kQkvN + row), + "h"(bits) + : "memory"); + } + else + { + asm volatile( + "st.relaxed.gpu.global.u16 [%0], %1;\n" ::"l"(cur + static_cast(t) * kQkvN + row), + "h"(bits) + : "memory"); + } + } + asm volatile("st.relaxed.gpu.global.u16 [%0], %1;\n" ::"l"(nxt + static_cast(t) * kQkvN + row), + "h"(static_cast(0xFFFFu)) + : "memory"); + } + M3FA_PROF_T(28, tid - lane); + } + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + M3FA_PROF_T(25, 512); + if (tid == 512) + { + M3FA_CLK(46); + } +} + +// ---------------- Megakernel o_proj (v52 m3_mx_gemv bit for bit: split 2, contiguous k-tiles, nacc 1) ---------------- +// The unit's weight tiles and scale atoms into the ring (warp 0, lane 0). From L2 when the CTA prefetched them there. +template +__device__ __forceinline__ void op_issue(Params const& p, uint8_t* smem, int ou) +{ + int const rt = ou / kOpSplits; + int const kt0 = (ou % kOpSplits) * kOpTiles; + uint32_t const ob = smem_u32(smem + SX::kOBar); + for (int i = 0; i < kOpTiles; ++i) + { + mbar_expect_tx(ob + 8 * i, kQkvWTile + kQkvSfAtom); + tma_load_2d(smem_u32(smem + SX::kRingW + i * kQkvWTile), &p.wmap_o, (kt0 + i) * 128, rt * 128, ob + 8 * i); + bulk_g2s(smem + SX::kRingSf + i * kQkvSfAtom, + p.wsf_o + (static_cast(rt) * kOpKTiles + kt0 + i) * kQkvSfAtom, kQkvSfAtom, ob + 8 * i); + } +} + +// L2 prefetch of the unit's weight tiles and scale atoms (one lane; nothing waits for it). +// The QKV units' share of the window-1 L2 prefetch (the next consumer's weights), issued once their own QKV weight +// stream has landed (acc_done): the attention core reads little from HBM. +__device__ __forceinline__ void window_prefetch_l2(Params const& p, int u) +{ + constexpr long long kChunk = 32768; + long long const n0 = (p.pf_b0 + kChunk - 1) / kChunk, n1 = (p.pf_b1 + kChunk - 1) / kChunk; + uint64_t pol = 0; + if (p.pf_hint) + { + asm volatile("createpolicy.fractional.L2::evict_last.b64 %0, 1.0;\n" : "=l"(pol)); + } + for (long long c = u; c < n0 + n1; c += kQkvUnits) + { + long long const a = c < n0 ? p.pf_a0 + c * kChunk : p.pf_a1 + (c - n0) * kChunk; + long long const rem = c < n0 ? p.pf_b0 - c * kChunk : p.pf_b1 - (c - n0) * kChunk; + unsigned int const n = static_cast(rem < kChunk ? rem : kChunk) & ~15u; + if (n == 0) + { + continue; + } + if (p.pf_hint) + { + asm volatile("cp.async.bulk.prefetch.L2.global.L2::cache_hint [%0], %1, %2;\n" ::"l"(a), "r"(n), "l"(pol) + : "memory"); + } + else + { + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;\n" ::"l"(a), "r"(n) : "memory"); + } + } +} + +__device__ __forceinline__ void op_prefetch_l2(Params const& p, int ou) +{ + int const rt = ou / kOpSplits; + int const kt0 = (ou % kOpSplits) * kOpTiles; + for (int i = 0; i < kOpTiles; ++i) + { + asm volatile("cp.async.bulk.prefetch.tensor.2d.L2.global [%0, {%1, %2}];\n" ::"l"(&p.wmap_o), + "r"((kt0 + i) * 128), "r"(rt * 128) + : "memory"); + } + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;\n" ::"l"( + p.wsf_o + (static_cast(rt) * kOpKTiles + kt0) * kQkvSfAtom), + "r"(kOpTiles * kQkvSfAtom) + : "memory"); +} + +// The weight tiles into TMEM as they land (warp 1, elected lane, operands warp-uniform). +template +__device__ __forceinline__ void op_stream(uint8_t* smem, uint32_t tm) +{ + uint32_t const ob = smem_u32(smem + SX::kOBar); + uint64_t const w0 = umma_desc_sw128(smem_u32(smem + SX::kRingW)); + uint32_t const sf0 = smem_u32(smem + SX::kRingSf); +#pragma unroll + for (int i = 0; i < kOpTiles; ++i) + { + mbar_wait_cta(ob + 8 * i, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + tmem_cp_128x256b(tm + kOA + (i * 4 + kb) * 8, w0 + ((i * kQkvWTile + kb * 32) >> 4)); + } + tmem_cp_sf(tm + kOSfa + i * 4, sf0 + i * kQkvSfAtom); + } + umma_commit(ob + 8 * kOpTiles); // a_ready +} + +// The attention's MXFP8 rows of this unit's k-tiles (the heads [kt0, kt0 + 8)), Lamport-polled value by value: 0xFF +// is NaN in both E4M3 and UE8M0, which the merge never writes (see the merge). Every thread of the CTA, one item each. +template +__device__ __forceinline__ void op_act(Params const& p, uint8_t* smem, int ou, int tid, unsigned int calls) +{ + int const kt0 = (ou % kOpSplits) * kOpTiles; + uint8_t const* q8 = p.q8 + (calls & 1u) * kMaxTok * kOpK; + uint8_t const* sf = p.sf + (calls & 1u) * kOpSfBytes; + constexpr int kXItems = kOpTiles * 8 * kMaxTok; + constexpr int kSfItems = kOpTiles * (kQkvSfAtom / 16); + for (int q = tid; q < kXItems + kSfItems; q += kThreads) + { + if (q < kXItems) + { + int const t = q % kMaxTok; + int const c = (q / kMaxTok) % 8; + int const i = q / (kMaxTok * 8); + if (t < p.num_tokens) + { + uint4 v; + bool pending; + do + { + asm volatile("ld.relaxed.gpu.global.v4.b32 {%0, %1, %2, %3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(q8 + static_cast(t) * kOpK + (kt0 + i) * 128 + c * 16) + : "memory"); + uint32_t const w[4] = {v.x, v.y, v.z, v.w}; + pending = false; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + // a byte equal to 0xFF: (x & 0x7F7F7F7F) + 0x01010101 sets bit 7 of exactly the 0x7F/0xFF bytes + uint32_t const hit = (((w[j] & 0x7F7F7F7Fu) + 0x01010101u) & w[j] & 0x80808080u); + pending |= hit != 0u; + } + } while (pending); + *reinterpret_cast(smem + SX::kOXt + i * 1024 + t * 128 + ((c ^ t) * 16)) = v; + } + } + else if (q < kXItems + kSfItems) + { + int const j = q - kXItems; + int const i = j / (kQkvSfAtom / 16); + int const c = j % (kQkvSfAtom / 16); + uint8_t const* src = sf + (kt0 + i) * kQkvSfAtom + c * 16; + uint4 v; + if (c < p.num_tokens) + { + // Row c of the atom: its 4 block scales in bytes [16 c, 16 c + 4). + do + { + asm volatile("ld.relaxed.gpu.global.v4.b32 {%0, %1, %2, %3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(src) + : "memory"); + } while ((((v.x & 0x7F7F7F7Fu) + 0x01010101u) & v.x & 0x80808080u) != 0u); + } + else + { + v = *reinterpret_cast(src); + } + *reinterpret_cast(smem + SX::kOXsf + i * kQkvSfAtom + c * 16) = v; + } + } +} + +// The MMAs (warp 1, elected lane): v52's order (per k-tile: B scale copy, 4 MMAs; one accumulator). +template +__device__ __forceinline__ void op_mma(uint8_t* smem, uint32_t tm) +{ + uint32_t const ob = smem_u32(smem + SX::kOBar); + mbar_wait_cta(ob + 8 * kOpTiles, 0); // a_ready + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint64_t const b0 = umma_desc_sw128(smem_u32(smem + SX::kOXt)); + uint32_t const xsf0 = smem_u32(smem + SX::kOXsf); +#pragma unroll + for (int i = 0; i < kOpTiles; ++i) + { + uint32_t const sfb = tm + kOSfb + i * 4; + tmem_cp_sf(sfb, xsf0 + i * kQkvSfAtom); +#pragma unroll + for (int kb = 0; kb < 4; ++kb) + { + int const j = i * 4 + kb; + umma_mx(tm + kOAcc, tm + kOA + j * 8, b0 + ((i * 1024 + kb * 32) >> 4), qkv_idesc(kb), tm + kOSfa + i * 4, + sfb, j > 0); + } + } + umma_commit(ob + 8 * (kOpTiles + 1)); // acc_done +} + +// Warps 16..19: the pair's split-K reduce (owner rows [64 s, 64 s + 64)) and the bf16 rows of y. +template +__device__ __forceinline__ void op_epilogue(Params const& p, uint8_t* smem, uint32_t tm, int tid, int ou) +{ + int const warp = tid >> 5; + int const lane = tid & 31; + uint32_t const ob = smem_u32(smem + SX::kOBar); + // The store's kernel parameters, fetched before the wait: a first touch of their constant-bank line costs a miss. + __nv_bfloat16* y_out = p.y_out; + int num_tokens = p.num_tokens; + int ar_mode = p.ar_mode; + asm volatile("" : "+l"(y_out), "+r"(num_tokens), "+r"(ar_mode)); +#ifndef M3FA_OP_PARPRE +#define M3FA_OP_PARPRE 1 // read the all-reduce slot parities before the accumulator wait (a global load off the tail) +#endif + // The parity flags are stable here: the all-reduce that flips them last completed before this grid's PDL wait, + // and the next one reads them only after this kernel's pushes. + int par_pre[kMaxTok]; + { + int const row_pre = (ou / kOpSplits) * 128 + ((tid >> 5) & 3) * 32 + (tid & 31); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + par_pre[t] = 0; + if (M3FA_OP_PARPRE && ar_mode != 0 && t < num_tokens) + { + asm volatile("ld.volatile.global.s32 %0, [%1];\n" + : "=r"(par_pre[t]) + : "l"(p.ar_flags + t * 8 + row_pre / (kOpN / 8)) + : "memory"); + } + } + } + mbar_wait_epi(ob + 8 * (kOpTiles + 1), 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + M3FA_PROF_T(51, kComputeThreads); + int const r = (warp & 3) * 32 + lane; + float v[16]; + tmem_ld16(tm + (((warp & 3) * 32) << 16) + kOAcc, v); + int const s = ou % kOpSplits; + int const owner = r / kOpOwnRows; + int const local = r % kOpOwnRows; + uint32_t const red = ob + 8 * (kOpTiles + 2); + uint32_t cta_rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(cta_rank)); + if (owner != s) + { + uint32_t const peer = cta_rank + owner - s; // o_proj units count up with the CTA index + uint32_t const dst + = mapa_shared(smem_u32(smem + SX::kOMailbox) + ((s * kOpOwnRows + local) * kQkvMbStride) * 4, peer); + uint32_t const pbar = mapa_shared(red, peer); +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + asm volatile( + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [%0], {%1, %2, %3, %4}, [%5];\n" ::"r"( + dst + 16 * g), + "r"(__float_as_uint(v[4 * g])), "r"(__float_as_uint(v[4 * g + 1])), "r"(__float_as_uint(v[4 * g + 2])), + "r"(__float_as_uint(v[4 * g + 3])), "r"(pbar) + : "memory"); + } + } + else + { + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(red) + : "memory"); + } + M3FA_PROF_T(52, tid - lane); + float const* mb = reinterpret_cast(smem + SX::kOMailbox); + int const row = (ou / kOpSplits) * 128 + r; + float sums[kMaxTok]; +#pragma unroll + for (int g = 0; g < kMaxTok / 4; ++g) + { + float4 part[kOpSplits]; +#pragma unroll + for (int q = 0; q < kOpSplits; ++q) + { + part[q] = *reinterpret_cast(mb + (q * kOpOwnRows + local) * kQkvMbStride + 4 * g); + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float sum = 0.0F; +#pragma unroll + for (int q = 0; q < kOpSplits; ++q) + { + float const pj = j == 0 ? part[q].x : j == 1 ? part[q].y : j == 2 ? part[q].z : part[q].w; + sum = sum + (q == s ? v[4 * g + j] : pj); + } + sums[4 * g + j] = sum; + } + } + M3FA_PROF_T(54, tid - lane); +#pragma unroll + for (int t = 0; t < kMaxTok; ++t) + { + float const sum = sums[t]; +#ifndef M3FA_EXP +#define M3FA_EXP 0 +#endif + if (t < num_tokens) + { + if (M3FA_EXP == 1) + { + asm volatile( + "st.relaxed.gpu.global.u16 [%0], %1;\n" ::"l"(y_out + static_cast(t) * kOpN + row), + "h"(__bfloat16_as_ushort(__float2bfloat16_rn(sum))) + : "memory"); + } + else + { + y_out[static_cast(t) * kOpN + row] = __float2bfloat16_rn(sum); + } + } + if (ar_mode != 0) + { + // m3_mx_gemv's push into the all-reduce exchange: lane pairs hold row pairs; the even lane writes the + // bf16x2 word into this rank's slot of the (token, column slice) parity the AR's flags select. + float const hi = __shfl_xor_sync(0xffffffffu, sum, 1); + if ((lane & 1) == 0 && t < num_tokens) + { + int par; + if (M3FA_OP_PARPRE) + { + par = par_pre[t]; + } + else + { + asm volatile("ld.volatile.global.s32 %0, [%1];\n" + : "=r"(par) + : "l"(p.ar_flags + t * 8 + row / (kOpN / 8)) + : "memory"); + } + int64_t const off = static_cast(par & 1) * p.ar_maxt * p.ar_world * (kOpN / 2) + + (static_cast(t) * p.ar_world + p.ar_rank) * (kOpN / 2) + row / 2; + __nv_bfloat162 w2 = __floats2bfloat162_rn(sum, hi); + uint32_t word = *reinterpret_cast(&w2); + // -0.0 halves to +0.0: 0x80000000 marks an empty exchange word (as m3_mx_gemv's _bf16x2_word) + word = (word & 0xFFFFu) == 0x8000u ? (word & 0xFFFF0000u) : word; + word = (word >> 16) == 0x8000u ? (word & 0x0000FFFFu) : word; + asm volatile("st.global.b32 [%0], %1;\n" ::"l"(p.ar_ws + off), "r"(word) : "memory"); + } + } + } + M3FA_PROF_T(53, tid - lane); + } + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); +} + +// Packed-QKV element group [token][col * 128 + 4 * lane, +4): the split partials, polled until every one is stored, +// summed in split order and rounded to BF16. +__device__ __forceinline__ void qkv_load(Params const& p, int par, int tok, int col, int lane, float (&e)[4]) +{ + float const* base = p.part + par * kQkvPartStride + static_cast(tok) * kQkvN + col * kHeadDim + lane * 4; + uint4 pr[kQkvSplits]; + long long spins = 0; + bool ready; + do + { + ready = true; +#pragma unroll + for (int s = 0; s < kQkvSplits; ++s) + { + asm volatile("ld.relaxed.gpu.global.v4.b32 {%0, %1, %2, %3}, [%4];\n" + : "=r"(pr[s].x), "=r"(pr[s].y), "=r"(pr[s].z), "=r"(pr[s].w) + : "l"(base + static_cast(s) * kMaxTok * kQkvN) + : "memory"); + ready &= pr[s].x != kQkvSentinel && pr[s].y != kQkvSentinel && pr[s].z != kQkvSentinel + && pr[s].w != kQkvSentinel; + } + if (++spins > (1LL << 26)) + { + __trap(); + } + } while (!ready); + float s0 = __uint_as_float(pr[0].x), s1 = __uint_as_float(pr[0].y), s2 = __uint_as_float(pr[0].z), + s3 = __uint_as_float(pr[0].w); +#pragma unroll + for (int s = 1; s < kQkvSplits; ++s) + { + s0 += __uint_as_float(pr[s].x); + s1 += __uint_as_float(pr[s].y); + s2 += __uint_as_float(pr[s].z); + s3 += __uint_as_float(pr[s].w); + } + e[0] = __bfloat162float(__float2bfloat16_rn(s0)); + e[1] = __bfloat162float(__float2bfloat16_rn(s1)); + e[2] = __bfloat162float(__float2bfloat16_rn(s2)); + e[3] = __bfloat162float(__float2bfloat16_rn(s3)); +} + +// Every CTA once it has read the call parity and the partials it needs: the last one flips the parity. +__device__ __forceinline__ void qkv_done(Params const& p) +{ + if (atomicAdd(p.counters + 1, 1) == kQkvGrid - 1) + { + atomicExch(p.counters + 1, 0); + atomicXor(p.counters, 1); + } +} + +} // namespace + +namespace +{ + +// FQKV: the kernel also computes the QKV(+index) projection from the layer input (see qkv_mma) and runs on a fixed +// kQkvGrid-CTA grid; the first attn_clusters clusters then run the attention on the projection's output. +template +__global__ void __launch_bounds__(kThreads, 1) m3_fused_decode_attn_kernel(__grid_constant__ Params const p) +{ +#if MINIMAX_M3_ATTN_DEVICE_ENABLED + using S = SmemX; + constexpr int kRows = S::kRows; + constexpr int kTok = S::kTok; + constexpr int kCs = kBlocks / BPC; // cluster size + constexpr int kGroups = QLEN / QG; // clusters per request + constexpr uint32_t kPushBytes = (kHeadDim + 1) * 4; // one partial row: 128 outputs and the LSE + constexpr int kTasks = kHq * QG + 3 * QLEN; // producer tasks: 16 Q heads x QG, [K, V, index-K] x QLEN + constexpr int kMaxT = (kTasks + kWarps - 1) / kWarps; + constexpr int kChunks = BPC * kPage * 4; // 16-byte packed chunks per K (or V) tile set + constexpr int kKPer = (kChunks + kComputeThreads - 1) / kComputeThreads; // K chunks per thread [0, 512) + constexpr bool kRegCore = RegCore::kOn; // warp-MMA attention core (see RegCore) + // V tiles (raw copy before the PDL wait, dequantized during QK and the softmax): on the packed path every warp + // outside the softmax groups takes part (at M=1, warps 4..19), on the fused-projection path the producer warps. +#ifndef M3FA_RS2 +#define M3FA_RS2 1 // megakernel at one 16-row chunk: the softmax and push rows split over two warp groups (8 each) +#endif + // Split softmax: the megakernel and multi-row (spec verify) steps; at one-row decode on the packed path it + // measured slower (it takes four warps from the V dequantization there). + constexpr bool kRs2 = M3FA_RS2 && BPC * QG == 1 && !RegCore::kOn && (FQKV || QLEN > 1); +#ifndef M3FA_MEGA_WARPMMA +#define M3FA_MEGA_WARPMMA 1 +#endif + // Warp-issued MMAs and the early cluster barrier: measured gains in the megakernel and on multi-row / paired-block + // steps, but not at one-row decode on 16-CTA clusters (the packed path's (1, 1, 1)), where they stay off. + constexpr bool kTuned = FQKV || QLEN > 1 || BPC > 1; + constexpr bool kWarpMma = M3FA_MEGA_WARPMMA && kTuned; + constexpr bool kEarlyClw = M3FA_MEGA_EARLYCLW && kTuned; + constexpr int kSoftmaxGroups = kRs2 ? 2 : (BPC * QG) < 4 ? (BPC * QG) : 4; +#ifndef M3FA_MEGA_VWIDE +#define M3FA_MEGA_VWIDE \ + 1 // megakernel attention CTAs (no projection unit): every warp outside the softmax copies and + // dequantizes V, as on the packed path (not only the four producer warps) +#endif + constexpr int kVStart = (kRegCore || (FQKV && !M3FA_MEGA_VWIDE)) ? kComputeThreads : 128 * kSoftmaxGroups; + constexpr int kVThreads = kThreads - kVStart; // V chunks: the warps outside the softmax groups + constexpr int kVPer = (kChunks + kVThreads - 1) / kVThreads; + constexpr int kPairs = BPC * QG; // (block, 16-row chunk) pairs of the softmax and push + constexpr int kPairsPerGroup = (kPairs + 3) / 4; // per group of four compute warps + // Named barrier 6: the softmax warps (P written) and the producer warps (V written) before the PV issue. + constexpr int kPvBarThreads = 128 * kSoftmaxGroups + kVThreads; + extern __shared__ __align__(128) uint8_t smem[]; + float* row_scale = reinterpret_cast(smem + S::kRowScale); + float* row_lse = reinterpret_cast(smem + S::kRowLse); + float* red_m = reinterpret_cast(smem + S::kRedM); + float* red_l = reinterpret_cast(smem + S::kRedL); + float* recv = reinterpret_cast(smem + S::kRecv); + uint32_t const mbar = smem_u32(smem + S::kMbar); + uint32_t const mma_bar = mbar + 8; + uint32_t* tmem_slot = reinterpret_cast(smem + S::kMbar + 16); + + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const cr = blockIdx.x % kCs; // cluster rank; owns blocks [cr * BPC, cr * BPC + BPC) + int const ci = blockIdx.x / kCs; + int const req = ci / kGroups; + int const tfirst = (ci % kGroups) * QG; // this cluster's query tokens [tfirst, tfirst + QG) of the request + int const tok0 = req * QLEN; + uint32_t const gbar = smem_u32(smem + S::kGBar); // FQKV: weight ring, A ready, accumulators done + int const qkv_u = FQKV ? qkv_unit() : -1; // FQKV: this CTA's projection unit, or -1 + + // ---------------- Prologue: nothing here depends on the producer kernel ---------------- +#if M3FA_TRIGGER_AT == 0 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (!FQKV) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + M3FA_PROF(0); +#ifdef M3FA_PROFILE + if (p.prof != nullptr && tid == 0) + { + uint32_t smid; + asm volatile("mov.u32 %0, %%smid;" : "=r"(smid)); + p.prof[blockIdx.x * 64 + 40] = smid; + } +#endif + if constexpr (FQKV) + { + // The projection weights stream into TMEM while the producer (the all-reduce) still runs. + if (warp < 2) + { + qkv_setup(p, smem, gbar, tmem_slot, warp, lane, qkv_u); + } + if (ci >= p.attn_clusters) + { + // A projection-only CTA. Every reduce barrier of the cluster is initialized before any peer pushes. + asm volatile("barrier.cluster.arrive.release.aligned;\nbarrier.cluster.wait.acquire.aligned;\n" :: + : "memory"); + int const op_u = static_cast(blockIdx.x) - kOpFirstCta; // this CTA's o_proj unit (>= 0 here) +#ifndef M3FA_OP_EARLY +#define M3FA_OP_EARLY 0 // 1: o_proj-only CTAs stream their weights from launch (0: after the PDL wait) +#endif + if (M3FA_OP_EARLY && qkv_u < 0 && warp == 0 && lane == 0) + { + op_issue(p, smem, op_u); // an o_proj-only CTA: its weights stream from launch on + } + if (M3FA_OP_L2PF && qkv_u >= 0 && warp == 0 && lane == 0) + { + op_prefetch_l2(p, op_u); // into L2 during the all-reduce, behind this CTA's QKV ring + } + qkv_stream(p, smem, gbar, tmem_slot, warp, lane, qkv_u); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); + if (M3FA_MEGA_TRIGGER == 2) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + // This CTA's call counter: read after the PDL wait (the previous call's CTA increments it at its end), and + // early, so its latency stays off the epilogue's critical path. + unsigned int const calls = p.flags[32 + blockIdx.x]; + M3FA_PROF(1); + if (!M3FA_OP_EARLY && qkv_u < 0 && warp == 0 && lane == 0) + { + op_issue(p, smem, op_u); + } + if (qkv_u >= 0) + { + int const par = 0; + qkv_mma(p, smem, gbar, tmem_slot, tid, qkv_u); + if (warp >= kComputeWarps) + { + qkv_epilogue(p, smem, gbar, tmem_slot, tid, qkv_u, par, calls); + } + if (warp == 0 && lane == 0) + { + // The ring (and the activation over it) is free once the QKV MMAs completed. + mbar_wait_cta(gbar + 16 * kQkvRing + 8, 0); + op_issue(p, smem, op_u); + if (p.pf_b0 > 0 || p.pf_b1 > 0) + { + window_prefetch_l2(p, qkv_u); + } + } + } + __syncthreads(); // the QKV accumulator reads are done: TMEM columns [0, 512) are the o_proj's + M3FA_PROF(2); + uint32_t const tm = *tmem_slot; + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if (elect_one()) + { + op_stream(smem, tm); + } + __syncwarp(); + } + op_act(p, smem, op_u, tid, calls); + asm volatile("fence.proxy.async.shared::cta;\n" ::: "memory"); + M3FA_PROF_T(36, 0); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + // Late release: the dependents launch once every unit has its attention input (the rel_v4 rule). + if (M3FA_MEGA_TRIGGER < 2) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + __syncthreads(); + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + if (elect_one()) + { + op_mma(smem, tm); + M3FA_PROF_T(50, threadIdx.x); + } + __syncwarp(); + } + if (warp >= kComputeWarps) + { + op_epilogue(p, smem, tm, tid, op_u); + } + M3FA_PROF_T(37, kComputeThreads); + __syncthreads(); + M3FA_PROF_T(55, 0); + if (M3FA_MEGA_TRIGGER == 3) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); // after this CTA's y rows / pushes + } + if (warp == 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + asm volatile("tcgen05.dealloc.cta_group::1.sync.aligned.b32 %0, %1;\n" ::"r"(*tmem_slot), "r"(kTmemAll) + : "memory"); + M3FA_PROF_T(56, 32); + } + if (tid == 0) + { + p.flags[32 + blockIdx.x] = calls + 1u; // a plain store: no load on the exit path + } + M3FA_PROF(39); +#ifndef M3FA_FINAL_CLBAR +#define M3FA_FINAL_CLBAR 0 +#endif + if (M3FA_FINAL_CLBAR) + { + // No peer may exit while its partials can still be pushed into it (DSMEM lifetime). + asm volatile("barrier.cluster.arrive.release.aligned;\nbarrier.cluster.wait.acquire.aligned;\n" :: + : "memory"); + } + M3FA_PROF(38); + return; + } + } +#ifndef M3FA_MEGA_ATTN_WAIT_FIRST +#define M3FA_MEGA_ATTN_WAIT_FIRST 0 // 1: the attention CTAs read nothing (metadata, caches) before their PDL wait +#endif +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (FQKV && M3FA_MEGA_ATTN_WAIT_FIRST) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + } +#endif + if (!FQKV && warp == kWarps - 1) // a producer warp: it has nothing else to do before the PDL wait + { + asm volatile("tcgen05.alloc.cta_group::1.sync.aligned.shared::cta.b32 [%0], %1;\n" ::"r"(smem_u32(tmem_slot)), + "r"(S::kTmemCols) + : "memory"); + asm volatile("tcgen05.relinquish_alloc_permit.cta_group::1.sync.aligned;\n" ::: "memory"); + } + int const seq_len = p.seq_lens[req]; + int page[BPC]; +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + int const b = cr * BPC + h; + page[h] = b < p.bt_cols ? p.block_table[static_cast(req) * p.bt_stride + b] : 0; + } + // New-token cache slots, for the tokens up to this cluster's last one. + int slot[QLEN]; +#pragma unroll + for (int t = 0; t < QLEN; ++t) + { + slot[t] = t < tfirst + QG ? p.out_cache_loc[tok0 + t] : -1; + } + // This warp's producer tasks and, for every lane that applies RoPE, the task token's cos/sin. + float cs[kMaxT][4], sn[kMaxT][4]; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + int const i = warp + kWarps * k; + int const t = i < kHq * QG ? tfirst + i / kHq : (i - kHq * QG) / 3; + if (i < kTasks && lane < 16 && t < QLEN) + { + int64_t const row = static_cast(p.position_ids[tok0 + t]) * kRotDim; +#pragma unroll + for (int e = 0; e < 4; ++e) + { + int const coefficient = (lane * 4 + e) & (kRotDim / 2 - 1); + cs[k][e] = p.rope[row + coefficient]; + sn[k][e] = p.rope[row + kRotDim / 2 + coefficient]; + } + } + else + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + cs[k][e] = 1.0F; + sn[k][e] = 0.0F; + } + } + } + float qw[4], kw[4], ikw[4]; + load_w4(qw, p.q_w, lane); + load_w4(kw, p.k_w, lane); + load_w4(ikw, p.ik_w, lane); + float const kqs = p.kv_quant_scale[1]; + float const vqs = p.kv_quant_scale[2]; + float const sl2e = (p.sm_scale * 1.4426950409f) * p.k_gs[0]; + float const vgs = p.v_gs[0]; + // Longest attended length among this cluster's tokens, and each block's rows within it. + int const kv_max = max(seq_len - QLEN + tfirst + QG, 0); + int nvh[BPC]; +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + nvh[h] = min(max(kv_max - (cr * BPC + h) * kPage, 0), kPage); + } + bool const any_valid = nvh[0] > 0; + if (tid == 32) + { + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;\n" ::"r"(mbar), "r"(1)); + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;\n" ::"r"(mma_bar), "r"(1)); + asm volatile( + "mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(mbar), "r"(kBlocks * QG * BPC * kPushBytes) + : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + } + // This thread's packed K / V chunks, held in registers until they are dequantized after the PDL wait. + RawChunk kraw[kKPer]; +#pragma unroll + for (int k = 0; k < kKPer; ++k) + { + int const q = tid + kComputeThreads * k; + kraw[k] = RawChunk{make_uint4(0u, 0u, 0u, 0u), 0u}; + int const h = q / (kPage * 4); + if (q < kChunks && tid < kComputeThreads && nvh[h] > 0) + { + kraw[k] = load_chunk(p.kv_data + static_cast(page[h]) * p.d_page, + p.kv_scale + static_cast(page[h]) * p.s_page, chunk_row(q % (kPage * 4)), + chunk_col(q % (kPage * 4)), false); + } + } + // V goes through cp.async into shared memory: an asynchronous copy does not hold the producer warps up the way + // an outstanding register load does, and they need V only after QK has been issued. + if (tid >= kVStart) + { +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + if (nvh[h] > 0) + { + uint8_t* dst = smem + S::kRawV + h * S::kRawVBlock; + uint8_t const* data = p.kv_data + static_cast(page[h]) * p.d_page + p.d_plane; + uint8_t const* scales = p.kv_scale + static_cast(page[h]) * p.s_page + p.s_plane; + for (int c = tid - kVStart; c < S::kRawVBlock / 16; c += kVThreads) + { + cp_async16(dst + c * 16, c < kPage * 4 ? data + c * 16 : scales + (c - kPage * 4) * 16); + } + } + } + } + asm volatile("cp.async.commit_group;\n" ::: "memory"); + if (warp == kWarps - 1) + { + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + // Every CTA's mbarrier is initialized before any peer pushes into it; the matching wait sits right before the + // push, so this barrier's latency hides behind the attention (with aliased partials it moves there). + if constexpr (!S::kRecvAlias) + { + asm volatile("barrier.cluster.arrive.relaxed.aligned;\n" ::: "memory"); + } + int kv_len[QG]; +#pragma unroll + for (int tl = 0; tl < QG; ++tl) + { + kv_len[tl] = max(seq_len - QLEN + tfirst + tl + 1, 0); + } + int const cta_tok0 = cr * kTok; // request position of this CTA's first block row + // New token t's row in this CTA's tiles (block * 128 + row), or -1: the CTA owns the block holding the token's + // position, and the token is attended by this cluster (t <= its last token). Every thread derives the same. + int new_row[QLEN]; +#pragma unroll + for (int t = 0; t < QLEN; ++t) + { + int const pos = seq_len - QLEN + t; + int const blk = min(pos >> 7, kBlocks - 1); + int const spage = slot[t] / kPage; + bool const mine = t < tfirst + QG && pos >= 0 && blk / BPC == cr && slot[t] >= 0 && spage < p.num_pages; + new_row[t] = mine && spage == page[blk % BPC] ? (blk % BPC) * kPage + (slot[t] & (kPage - 1)) : -1; + } + auto is_new_row = [&](int r) + { + bool hit = false; +#pragma unroll + for (int t = 0; t < QLEN; ++t) + { + hit |= new_row[t] == r; + } + return hit; + }; + // Cache writes of the new K/V/index-K are issued after this CTA has pushed its partials: only the fp16 tile rows + // are on the attention's path. kind 0: none, 1: NVFP4 data + scale, 2: FP8 index-K. + int wr_kind[kMaxT]; + uint8_t* wr_ptr[kMaxT]; + uint8_t* wr_sptr[kMaxT]; + uint32_t wr_val[kMaxT]; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + wr_kind[k] = 0; + wr_ptr[k] = nullptr; + wr_sptr[k] = nullptr; + wr_val[k] = 0u; + } + uint32_t tmem = 0; + auto dequant_k = [&]() + { + if (tid < kComputeThreads) + { +#pragma unroll + for (int k = 0; k < kKPer; ++k) + { + int const q = tid + kComputeThreads * k; + if (q < kChunks) + { + int const h = q / (kPage * 4); + int const row = chunk_row(q % (kPage * 4)); + if (!is_new_row(h * kPage + row)) + { + store_chunk_fp16( + smem + S::kK16 + h * S::kTileBytes, row, chunk_col(q % (kPage * 4)), kraw[k], row < nvh[h]); + } + } + } + } + }; + // The producer warps' V tiles (raw tiles from cp.async, dequantized to fp16), then their PV-barrier arrival. + auto dequant_v = [&]() + { +#ifdef M3FA_DBG_NO_VDQ + if constexpr (!kRegCore) + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + return; +#endif + asm volatile("cp.async.wait_all;\n" ::: "memory"); + asm volatile("bar.sync 4, %0;\n" ::"n"(kVThreads) : "memory"); +#pragma unroll + for (int k = 0; k < kVPer; ++k) + { + int const q = tid - kVStart + kVThreads * k; + if (q < kChunks) + { + int const h = q / (kPage * 4); + int const row = chunk_row(q % (kPage * 4)); + int const cc = chunk_col(q % (kPage * 4)); + if (!is_new_row(h * kPage + row)) + { + uint8_t const* rawv = smem + S::kRawV + h * S::kRawVBlock; + uint8_t const* sc = rawv + kPage * 64 + (row >> 2) * 32 + (row & 3); + RawChunk c; + c.data = *reinterpret_cast(rawv + row * 64 + cc * 16); + c.scales + = static_cast(sc[(cc * 2) * 4]) | (static_cast(sc[(cc * 2 + 1) * 4]) << 8); + store_chunk_fp16(smem + S::kV16 + h * S::kTileBytes, row, cc, c, row < nvh[h]); + } + } + } + M3FA_FENCE_PROXY_ASYNC(); // V tiles -> tensor core reads (ordered by the barrier before the PV issue) + if constexpr (!kRegCore) + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + M3FA_PROF_T(16, kComputeThreads); + M3FA_PROF_T(17, kThreads - 1); + }; + if constexpr (FQKV) + { + qkv_stream(p, smem, gbar, tmem_slot, warp, lane, qkv_u); + } + if constexpr (!FQKV) + { + if (M3FA_V15_EARLYV && any_valid && tid >= kVStart) + { + dequant_v(); + } + if constexpr (kEarlyClw && !S::kRecvAlias) + { + asm volatile("barrier.cluster.wait.aligned;\n" ::: "memory"); + } + } + M3FA_PROF(11); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); + if (FQKV && M3FA_MEGA_TRIGGER == 2) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#if M3FA_TRIGGER_AT == 1 + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); +#endif +#endif + // FQKV (megakernel): this call's QKV rows sit in buffer (calls & 1), polled value by value below. + unsigned int const mega_calls = FQKV ? p.flags[32 + blockIdx.x] : 0u; + __nv_bfloat16 const* qkv_src = FQKV ? p.packed + (mega_calls & 1u) * kMaxTok * kQkvN : p.packed; + M3FA_PROF(1); + int const par = 0; + if constexpr (FQKV) + { + if (qkv_u >= 0) + { + qkv_mma(p, smem, gbar, tmem_slot, tid, qkv_u); + } + } + + if constexpr (FQKV) + { + if (qkv_u >= 0) + { + if (warp >= kComputeWarps) + { + qkv_epilogue(p, smem, gbar, tmem_slot, tid, qkv_u, par, mega_calls); + } + else + { + // The attention regions overlay the weight ring: every weight byte has gone on into TMEM (or, with + // the MMAs reading the ring, every MMA has completed) first. + mbar_wait_cta(gbar + 16 * kQkvRing + (kQkvASmem ? 8 : 0), 0); + } + } + if (any_valid) + { + dequant_k(); + } +#ifndef M3FA_MEGA_EARLYV +#define M3FA_MEGA_EARLYV 1 // attention CTAs dequantize V here, while the QKV rows are still on their way +#endif +#ifndef M3FA_MEGA_EARLYCLW +#define M3FA_MEGA_EARLYCLW 1 // and pass the cluster barrier (peers' merge mbarriers initialized) here too +#endif + if (M3FA_MEGA_EARLYV && any_valid && tid >= kVStart) + { + dequant_v(); + } + if constexpr (kEarlyClw && !S::kRecvAlias) + { + asm volatile("barrier.cluster.wait.aligned;\n" ::: "memory"); + } + M3FA_PROF(14); + } + + if (any_valid) // CTA-uniform + { + // ---------------- Producer ---------------- + // Q rows of this cluster's tokens in every CTA. The new K/V of every token up to this cluster's last one + // whose position falls in this CTA's blocks go straight into the fp16 tiles; the CTA writes the caches, and + // the index-K, only for its own tokens. + uint2 raw[kMaxT]; + float fq[FQKV ? kMaxT : 1][4]; + bool act[kMaxT]; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + int const i = warp + kWarps * k; + raw[k] = make_uint2(0u, 0u); + act[k] = false; + if (i < kTasks) + { + int t, col; + bool active = true; + if (i >= kHq * QG) + { + int const kv = i - kHq * QG; + t = kv / 3; + int const kind = kv % 3; + col = kind == 0 ? kColK : (kind == 1 ? kColV : kColIdxK); + int const pos = seq_len - QLEN + t; + active = t < tfirst + QG && pos >= 0 && min(pos >> 7, kBlocks - 1) / BPC == cr && slot[t] >= 0 + && slot[t] / kPage < p.num_pages && (kind != 2 || t >= tfirst); + } + else + { + t = tfirst + i / kHq; + col = i % kHq; + } + act[k] = active; + if (active) + { + uint2 const* src = reinterpret_cast( + qkv_src + static_cast(tok0 + t) * p.packed_stride + col * kHeadDim + lane * 4); + if constexpr (FQKV) + { + uint2 v; + bool pending; + do + { + asm volatile("ld.relaxed.gpu.global.v2.b32 {%0, %1}, [%2];\n" + : "=r"(v.x), "=r"(v.y) + : "l"(src) + : "memory"); + pending = (v.x & 0xFFFFu) == 0xFFFFu || (v.x >> 16) == 0xFFFFu || (v.y & 0xFFFFu) == 0xFFFFu + || (v.y >> 16) == 0xFFFFu; + } while (pending); + raw[k] = v; + } + else + { + raw[k] = *src; + } + } + } + } + M3FA_PROF(15); + // While the packed-QKV loads are in flight: dequantize this thread's K chunks to fp16 (exact: e2m1 x e4m3 + // fits fp16). Rows past the longest attended length are zeroed, as the Triton producer masks its loads; the + // new tokens' rows are written by their producer tasks instead. (Building the tiles before the PDL wait was + // measured slower: the cache loads issued at entry are still in flight there, so the wait moves later.) + if constexpr (!FQKV) + { + dequant_k(); + } +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + int const i = warp + kWarps * k; + if (i >= kTasks || !act[k]) + { + continue; + } + float e[4]; + { + float2 const a = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw[k].x)); + float2 const b = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&raw[k].y)); + e[0] = a.x; + e[1] = a.y; + e[2] = b.x; + e[3] = b.y; + } + if (i < kHq * QG) + { + norm_rope(e, qw, p.eps, cs[k], sn[k], lane); + __nv_fp8x2_e4m3 const lo(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const hi(make_float2(e[2], e[3])); + uint2 hv; + hv.x = e4m3x2_to_h2(lo.__x); + hv.y = e4m3x2_to_h2(hi.__x); + *reinterpret_cast(smem + S::kQ + kmaj_off(i, lane * 4)) = hv; // row i = token * 16 + head + continue; + } + int const t = (i - kHq * QG) / 3; + int const kind = (i - kHq * QG) % 3; + int const spage = slot[t] / kPage; + int const within = slot[t] & (kPage - 1); + if (kind == 2) + { + norm_rope(e, ikw, p.eps, cs[k], sn[k], lane); + __nv_fp8x2_e4m3 const lo(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const hi(make_float2(e[2], e[3])); + wr_kind[k] = 2; + wr_val[k] = static_cast(lo.__x) | (static_cast(hi.__x) << 16); + wr_ptr[k] = p.idx_k + static_cast(spage) * p.i_page + static_cast(within) * p.i_tok + + lane * 4; + continue; + } + bool const isV = kind == 1; + if (!isV) + { + norm_rope(e, kw, p.eps, cs[k], sn[k], lane); + } + uint8_t sbyte = 0; + uint16_t const fp4 = quantizeMinimaxM3Fp4x4(e, isV ? vqs : kqs, &sbyte); + if (t >= tfirst) + { + int const scale_col = lane / 4; + int const scale_within = isV ? (within / 4) * 32 + scale_col * 4 + within % 4 : within * 8 + scale_col; + wr_kind[k] = 1; + wr_val[k] = static_cast(fp4) | (static_cast(sbyte) << 16); + wr_ptr[k] = p.kv_data + static_cast(spage) * p.d_page + (isV ? p.d_plane : 0) + within * 64 + + lane * 2; + wr_sptr[k] = p.kv_scale + static_cast(spage) * p.s_page + (isV ? p.s_plane : 0) + scale_within; + } + if (new_row[t] >= 0) + { + // The attention reads this page: its fp16 tile row is the freshly quantized row, dequantized like + // the rest of the tile. + int const h = new_row[t] / kPage; + int const row = new_row[t] % kPage; + uint32_t const sc2 = e4m3_to_h2_splat(sbyte); + uint2 hv; + hv.x = hmul2_u(e2m1x2_to_h2(fp4 & 0xFFu), sc2); + hv.y = hmul2_u(e2m1x2_to_h2(fp4 >> 8), sc2); + *reinterpret_cast( + smem + (isV ? S::kV16 : S::kK16) + h * S::kTileBytes + kmaj_off(row, lane * 4)) + = hv; + } + } + // K tiles, Q rows and patched rows -> tensor core reads. Only threads that wrote fence: the fence also + // waits for the thread's outstanding loads, and the producer warps still have their V chunks in flight. + bool wrote = tid < kComputeThreads; +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + wrote |= act[k]; + } + if (wrote) + { + M3FA_FENCE_PROXY_ASYNC(); + } + M3FA_PROF(13); + __syncthreads(); +#if M3FA_TRIGGER_AT == 2 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + // Every K/V byte this CTA reads has arrived (the V chunks sit in registers): release the dependents now so + // their weight streams do not queue ahead of the KV reads (the megakernel releases late, from its o_proj). + if constexpr (!FQKV || M3FA_MEGA_TRIGGER == 1) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + tmem = *tmem_slot; + M3FA_PROF(2); + // ---------------- S^T = K Q^T per block on the tensor core: TMEM lane = token, column = query row ---------- +#ifndef M3FA_MEGA_WARPMMA +#define M3FA_MEGA_WARPMMA 1 // attention MMAs issued by warp 0's elected lane with warp-uniform operands (not tid 0) +#endif + if (!kRegCore && (kWarpMma ? warp == 0 : tid == 0)) + { + // Operands computed by the whole warp stay in uniform registers; one elected lane issues. + uint32_t const tm_u = tmem; + bool const issue = kWarpMma ? elect_one() : true; + if (issue) + { +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + uint32_t const a0 = smem_u32(smem + S::kK16 + h * S::kTileBytes); + uint32_t const b0 = smem_u32(smem + S::kQ); +#pragma unroll + for (int ks = 0; ks < kHeadDim / 16; ++ks) + { + umma_f16(tm_u + h * kRows, umma_desc(a0 + ks * 256, 128, 2048), + umma_desc(b0 + ks * 256, 128, 2048), umma_idesc(kRows, 0, 0), ks > 0); + } + } + umma_commit(mma_bar); + } + if (kWarpMma) + { + __syncwarp(); + } + } + if (tid >= kVStart && !(FQKV && M3FA_MEGA_EARLYV) && !(!FQKV && M3FA_V15_EARLYV)) + { + // The producer warps dequantize V while QK and the softmax run. + dequant_v(); + } + if constexpr (kRegCore) + { + // ---------------- Register core: warp w of block h = w / kWpb owns 8 * kNt tokens for S and 8 * kNt + // output dims for O; m16n8k16 fp16 MMAs with fp32 accumulators, the softmax in the S fragments ---------- + constexpr int kWpb = kComputeWarps / BPC; // warps per block + constexpr int kNt = 16 / kWpb; // 8-token (S) or 8-dim (O) tiles per warp + float* redx_m = reinterpret_cast(smem + S::kRedX); + float* redx_l = redx_m + BPC * kRows * kWpb; + if (warp < kComputeWarps) + { + int const h = warp / kWpb; + int const wib = warp % kWpb; + int const g = lane >> 2; + int const q4 = lane & 3; + uint32_t const ktile = smem_u32(smem + S::kK16 + h * S::kTileBytes); + uint32_t const vtile = smem_u32(smem + S::kV16 + h * S::kTileBytes); + uint32_t const qrows = smem_u32(smem + S::kQ); + uint8_t* ptile = smem + S::kK16 + h * S::kTileBytes; // P over this block's K tile, after QK + auto block_bar = [&]() + { + if (h == 0) + { + asm volatile("bar.sync 14, %0;\n" ::"n"(kWpb * 32) : "memory"); + } + else + { + asm volatile("bar.sync 15, %0;\n" ::"n"(kWpb * 32) : "memory"); + } + }; + // S = Q K^T: rows (c, g / g + 8), tokens 8 tb + 2 q4 (+1). + float sv[QG][kNt][4]; +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + sv[c][nt][e] = 0.0F; + } + } + } +#pragma unroll + for (int ks = 0; ks < kHeadDim / 16; ++ks) + { + uint32_t bk[kNt][2]; +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + int const tb = wib * kNt + nt; + ldsm_x2(ktile + tb * 2048 + (ks * 2 + ((lane >> 3) & 1)) * 128 + (lane & 7) * 16, bk[nt]); + } +#pragma unroll + for (int c = 0; c < QG; ++c) + { + uint32_t aq[4]; + ldsm_x4( + qrows + (c * 2 + ((lane >> 3) & 1)) * 2048 + (ks * 2 + (lane >> 4)) * 128 + (lane & 7) * 16, + aq); +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + mma_f16_16816(sv[c][nt], aq, bk[nt]); + } + } + } + M3FA_PROF(7); + // Row maxima (log2 domain, masked as the producer masks), quad then across the block's warps. + float mrow[QG][2]; +#pragma unroll + for (int c = 0; c < QG; ++c) + { + float mx0 = kNegInf, mx1 = kNegInf; +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + int const tok = cta_tok0 + h * kPage + (wib * kNt + nt) * 8 + 2 * q4 + (e & 1); + bool const valid = tok < kv_max && tok < kv_len[c]; + float const v = valid ? sv[c][nt][e] * sl2e : kNegInf; + sv[c][nt][e] = v; + if (e < 2) + { + mx0 = fmaxf(mx0, v); + } + else + { + mx1 = fmaxf(mx1, v); + } + } + } + mx0 = fmaxf(mx0, __shfl_xor_sync(0xffffffffu, mx0, 1)); + mx0 = fmaxf(mx0, __shfl_xor_sync(0xffffffffu, mx0, 2)); + mx1 = fmaxf(mx1, __shfl_xor_sync(0xffffffffu, mx1, 1)); + mx1 = fmaxf(mx1, __shfl_xor_sync(0xffffffffu, mx1, 2)); + if (q4 == 0) + { + redx_m[(h * kRows + c * 16 + g) * kWpb + wib] = mx0; + redx_m[(h * kRows + c * 16 + g + 8) * kWpb + wib] = mx1; + } + } + block_bar(); // every warp of the block is past QK (P may overwrite the K tile) and wrote its maxima + M3FA_PROF(8); +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int half = 0; half < 2; ++half) + { + float const* rm = redx_m + (h * kRows + c * 16 + g + 8 * half) * kWpb; + float m = kNegInf; +#pragma unroll + for (int w4 = 0; w4 < kWpb; w4 += 4) + { + float4 const v = *reinterpret_cast(rm + w4); + m = fmaxf(m, fmaxf(fmaxf(v.x, v.y), fmaxf(v.z, v.w))); + } + mrow[c][half] = m; + } + // P = exp2(s - m) as fp16 into [row][token] core matrices; the fp32 values feed the row sums. + float l0 = 0.0F, l1 = 0.0F; +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + float a[4]; +#pragma unroll + for (int e = 0; e < 4; ++e) + { + float const m = mrow[c][e >> 1]; + a[e] = m == kNegInf ? 0.0F : ex2(sv[c][nt][e] - m); + } + l0 += a[0] + a[1]; + l1 += a[2] + a[3]; + int const t0 = (wib * kNt + nt) * 8 + 2 * q4; + int const r0 = c * 16 + g; + __half2 const p0 = __floats2half2_rn(a[0], a[1]); + __half2 const p1 = __floats2half2_rn(a[2], a[3]); + *reinterpret_cast<__half2*>( + ptile + (r0 >> 3) * 2048 + (t0 >> 3) * 128 + (r0 & 7) * 16 + (t0 & 7) * 2) + = p0; + *reinterpret_cast<__half2*>( + ptile + ((r0 + 8) >> 3) * 2048 + (t0 >> 3) * 128 + (r0 & 7) * 16 + (t0 & 7) * 2) + = p1; + } + l0 += __shfl_xor_sync(0xffffffffu, l0, 1); + l0 += __shfl_xor_sync(0xffffffffu, l0, 2); + l1 += __shfl_xor_sync(0xffffffffu, l1, 1); + l1 += __shfl_xor_sync(0xffffffffu, l1, 2); + if (q4 == 0) + { + redx_l[(h * kRows + c * 16 + g) * kWpb + wib] = l0; + redx_l[(h * kRows + c * 16 + g + 8) * kWpb + wib] = l1; + } + } + M3FA_PROF_T(19, 0); + } + __syncthreads(); // P, the V tiles (producer warps) and the row sums visible + M3FA_PROF(9); + if (warp < kComputeWarps) + { + int const h = warp / kWpb; + int const wib = warp % kWpb; + int const g = lane >> 2; + int const q4 = lane & 3; + uint32_t const vtile = smem_u32(smem + S::kV16 + h * S::kTileBytes); + uint32_t const ptile = smem_u32(smem + S::kK16 + h * S::kTileBytes); + float* obuf = reinterpret_cast(smem + S::kK16 + h * S::kTileBytes + S::kObufOff); + // O = P V: rows (c, g / g + 8), dims 8 dt + 2 q4 (+1). + float ov[QG][kNt][4]; +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { +#pragma unroll + for (int e = 0; e < 4; ++e) + { + ov[c][nt][e] = 0.0F; + } + } + } +#pragma unroll + for (int ks = 0; ks < kPage / 16; ++ks) + { + uint32_t bv[kNt][2]; + int const tok = ks * 16 + ((lane >> 3) & 1) * 8 + (lane & 7); +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + int const dt = wib * kNt + nt; + ldsm_x2_trans(vtile + (tok >> 3) * 2048 + dt * 128 + (tok & 7) * 16, bv[nt]); + } +#pragma unroll + for (int c = 0; c < QG; ++c) + { + uint32_t ap[4]; + ldsm_x4( + ptile + (c * 2 + ((lane >> 3) & 1)) * 2048 + (ks * 2 + (lane >> 4)) * 128 + (lane & 7) * 16, + ap); +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + mma_f16_16816(ov[c][nt], ap, bv[nt]); + } + } + } +#pragma unroll + for (int c = 0; c < QG; ++c) + { +#pragma unroll + for (int nt = 0; nt < kNt; ++nt) + { + int const d = (wib * kNt + nt) * 8 + 2 * q4; + int const r0 = c * 16 + g; + *reinterpret_cast(obuf + r0 * kOPitch + d) = make_float2(ov[c][nt][0], ov[c][nt][1]); + *reinterpret_cast(obuf + (r0 + 8) * kOPitch + d) + = make_float2(ov[c][nt][2], ov[c][nt][3]); + } + } + } + else if (tid - kComputeThreads < BPC * kRows) + { + // Row statistics of every (block, row): LSE and the PV scale, as the producer derives them. + int const r = tid - kComputeThreads; + float const* rm = redx_m + r * kWpb; + float const* rl = redx_l + r * kWpb; + float m = kNegInf; + float l = 0.0F; +#pragma unroll + for (int w4 = 0; w4 < kWpb; w4 += 4) + { + float4 const vm = *reinterpret_cast(rm + w4); + float4 const vl = *reinterpret_cast(rl + w4); + m = fmaxf(m, fmaxf(fmaxf(vm.x, vm.y), fmaxf(vm.z, vm.w))); + l += (vl.x + vl.y) + (vl.z + vl.w); + } + float const lse = l > 0.0F ? m + lg2(l) : kNegInf; + float const scale = lse > kNegInf ? ex2(m - lse) : 0.0F; + row_scale[r] = scale * vgs; + row_lse[r] = lse; + } + __syncthreads(); // O partials and row statistics visible to the push + } + else + { + if constexpr (kRs2) + { + // ---------------- Split softmax: warps 0..3 rows 0..7, warps 4..7 rows 8..15 (thread = block token) + // -------- + if (warp < 8) + { + int const rg = warp / 4; + int const wq = warp % 4; + int const tk = wq * 32 + lane; + auto rg_bar = [&]() + { + if (rg == 0) + { + asm volatile("bar.sync 10, 128;\n" ::: "memory"); + } + else + { + asm volatile("bar.sync 11, 128;\n" ::: "memory"); + } + }; + int const rs8 = ((lane >> 4) & 1) * 4 + ((lane >> 3) & 1) * 2 + ((lane >> 2) & 1); + mbar_wait_cta(mma_bar, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + M3FA_PROF(7); + float v[8]; + tmem_ld8(tmem + ((wq * 32) << 16) + 8 * rg, v); + bool const valid = cta_tok0 + tk < kv_max && cta_tok0 + tk < kv_len[0]; + float a[8]; +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + v[j2] = valid ? v[j2] * sl2e : kNegInf; + a[j2] = v[j2]; + } + float const mw = warp_reduce_scatter8(a, lane); + if ((lane & 3) == 0) + { + red_m[(8 * rg + rs8) * 4 + wq] = mw; + } + rg_bar(); + M3FA_PROF(8); + uint8_t* pdst = smem + S::kP + (tk >> 3) * 128 + (tk & 7) * 16; + float mrow = kNegInf; + if (lane < 8) + { + float4 const mq = *reinterpret_cast(red_m + (8 * rg + lane) * 4); + mrow = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + } + float pa[8]; +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + float const m = __shfl_sync(0xffffffffu, mrow, j2); + pa[j2] = m == kNegInf ? 0.0F : ex2(v[j2] - m); + } + if (tid == 0) + { + M3FA_CLK(41); + } + uint32_t pw[4]; +#pragma unroll + for (int j2 = 0; j2 < 4; ++j2) + { + __half2 const hp = __floats2half2_rn(pa[2 * j2], pa[2 * j2 + 1]); + pw[j2] = *reinterpret_cast(&hp); + } + *reinterpret_cast(pdst + rg * 2048) = make_uint4(pw[0], pw[1], pw[2], pw[3]); + M3FA_PROF_T(18, 0); + if (tid == 0) + { + M3FA_CLK(42); + } + M3FA_FENCE_PROXY_ASYNC(); // P -> tensor core reads + if (tid == 0) + { + M3FA_CLK(43); + } + if (warp == 0) + { + asm volatile("bar.sync 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + if (tid == 0) + { + M3FA_CLK(44); + } + } + else + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + if (kWarpMma ? warp == 0 : tid == 0) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm_u = tmem; + uint32_t const a0 = smem_u32(smem + S::kV16); + uint32_t const b0 = smem_u32(smem + S::kP); + if (kWarpMma ? elect_one() : true) + { +#pragma unroll + for (int ks = 0; ks < kPage / 16; ++ks) + { + umma_f16(tm_u + BPC * kRows, umma_desc(a0 + ks * 4096, 2048, 128), + umma_desc(b0 + ks * 256, 128, 2048), umma_idesc(kRows, 1, 1), ks > 0); + } + umma_commit(mma_bar); + } + if (kWarpMma) + { + __syncwarp(); + } + } + M3FA_PROF_T(19, 0); + float const lw = warp_reduce_scatter8(pa, lane); + if ((lane & 3) == 0) + { + red_l[(8 * rg + rs8) * 4 + wq] = lw; + } + rg_bar(); + M3FA_PROF_T(20, 0); + if (wq == 0 && lane < 8) + { + int const r = 8 * rg + lane; + float4 const mq = *reinterpret_cast(red_m + r * 4); + float4 const lq = *reinterpret_cast(red_l + r * 4); + float const m = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + float const l = (lq.x + lq.y) + (lq.z + lq.w); + float const lse = l > 0.0F ? m + lg2(l) : kNegInf; + float const scale = lse > kNegInf ? ex2(m - lse) : 0.0F; + row_scale[r] = scale * vgs; + row_lse[r] = lse; + } + } + __syncthreads(); // row scales and LSEs visible (PV was issued above) + M3FA_PROF(9); + if (warp < 8) + { + mbar_wait_cta(mma_bar, 1); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + } + } + else + { + // ---------------- Softmax: thread = block token; the (block, 16-row chunk) pairs are spread over the + // four groups of four compute warps (each group covers the 128 TMEM lanes = block tokens) + // ---------------- + if (warp < kComputeWarps && warp / 4 < kPairs) + { + int const g = warp / 4; + int const wq = warp % 4; + int const tk = wq * 32 + lane; + auto group_bar = [&]() + { + if (g == 0) + { + asm volatile("bar.sync 10, 128;\n" ::: "memory"); + } + else if (g == 1) + { + asm volatile("bar.sync 11, 128;\n" ::: "memory"); + } + else if (g == 2) + { + asm volatile("bar.sync 12, 128;\n" ::: "memory"); + } + else + { + asm volatile("bar.sync 13, 128;\n" ::: "memory"); + } + }; + // Row r(lane) this lane holds after the reduce-scatter: bits 4..1 of the lane. + int const rs_row + = ((lane >> 4) & 1) * 8 + ((lane >> 3) & 1) * 4 + ((lane >> 2) & 1) * 2 + ((lane >> 1) & 1); + mbar_wait_cta(mma_bar, 0); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + M3FA_PROF(7); + float v[kPairsPerGroup][16]; +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + tmem_ld16(tmem + ((wq * 32) << 16) + h * kRows + c * 16, v[k]); + bool const valid + = cta_tok0 + h * kPage + tk < kv_max && cta_tok0 + h * kPage + tk < kv_len[c]; + float a[16]; +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[k][j] = valid ? v[k][j] * sl2e : kNegInf; + a[j] = v[k][j]; + } + float const mw = warp_reduce_scatter16(a, lane); + if ((lane & 1) == 0) + { + red_m[((h * QG + c) * 16 + rs_row) * 4 + wq] = mw; + } + } + } + group_bar(); + M3FA_PROF(8); + float pa[kPairsPerGroup][16]; +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + uint8_t* pdst = smem + S::kP + h * kRows * kPage * 2 + (tk >> 3) * 128 + (tk & 7) * 16; + // Row maxima: lane j (< 16) reduces row j's four warp partials, then broadcasts it. + float mrow = kNegInf; + if (lane < 16) + { + float4 const mq + = *reinterpret_cast(red_m + ((h * QG + c) * 16 + lane) * 4); + mrow = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + } +#pragma unroll + for (int j = 0; j < 16; ++j) + { + float const m = __shfl_sync(0xffffffffu, mrow, j); + pa[k][j] = m == kNegInf ? 0.0F : ex2(v[k][j] - m); + } + if (tid == 0) + { + M3FA_CLK(41); + } + uint32_t pw[8]; +#pragma unroll + for (int j = 0; j < 8; ++j) + { + __half2 const hp = __floats2half2_rn(pa[k][2 * j], pa[k][2 * j + 1]); + pw[j] = *reinterpret_cast(&hp); + } + // P^T as the MN-major B operand: 8 query rows of one token are one 16-byte core-matrix row. + *reinterpret_cast(pdst + (2 * c) * 2048) = make_uint4(pw[0], pw[1], pw[2], pw[3]); + *reinterpret_cast(pdst + (2 * c + 1) * 2048) + = make_uint4(pw[4], pw[5], pw[6], pw[7]); + M3FA_PROF_T(18, 0); + } + } + if (tid == 0) + { + M3FA_CLK(42); + } + M3FA_FENCE_PROXY_ASYNC(); // P -> tensor core reads + if (tid == 0) + { + M3FA_CLK(43); + } + // PV starts as soon as P and the V tiles are in: thread 0 waits for the softmax warps and the + // producer warps (V); the row sums and statistics below overlap the MMAs. + if (warp == 0) + { + asm volatile("bar.sync 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + else + { + asm volatile("bar.arrive 6, %0;\n" ::"n"(kPvBarThreads) : "memory"); + } + if (kWarpMma ? warp == 0 : tid == 0) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + uint32_t const tm_u = tmem; + if (kWarpMma ? elect_one() : true) + { +#pragma unroll + for (int h = 0; h < BPC; ++h) + { + uint32_t const a0 = smem_u32(smem + S::kV16 + h * S::kTileBytes); + uint32_t const b0 = smem_u32(smem + S::kP + h * kRows * kPage * 2); +#pragma unroll + for (int ks = 0; ks < kPage / 16; ++ks) + { + umma_f16(tm_u + (BPC + h) * kRows, umma_desc(a0 + ks * 4096, 2048, 128), + umma_desc(b0 + ks * 256, 128, 2048), umma_idesc(kRows, 1, 1), ks > 0); + } + } + umma_commit(mma_bar); + } + if (kWarpMma) + { + __syncwarp(); + } + } + M3FA_PROF_T(19, 0); +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + float const lw = warp_reduce_scatter16(pa[k], lane); + if ((lane & 1) == 0) + { + red_l[((h * QG + c) * 16 + rs_row) * 4 + wq] = lw; + } + } + } + group_bar(); + M3FA_PROF_T(20, 0); + if (wq == 0 && lane < 16) + { +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + int const r = (h * QG + c) * 16 + lane; + float4 const mq = *reinterpret_cast(red_m + r * 4); + float4 const lq = *reinterpret_cast(red_l + r * 4); + float const m = fmaxf(fmaxf(mq.x, mq.y), fmaxf(mq.z, mq.w)); + float const l = (lq.x + lq.y) + (lq.z + lq.w); + float const lse = l > 0.0F ? m + lg2(l) : kNegInf; + float const scale = lse > kNegInf ? ex2(m - lse) : 0.0F; + row_scale[h * kRows + c * 16 + lane] = scale * vgs; + row_lse[h * kRows + c * 16 + lane] = lse; + } + } + } + } + __syncthreads(); // row scales and LSEs visible (PV was issued above) + M3FA_PROF(9); + if (warp < kComputeWarps && warp / 4 < kPairs) + { + mbar_wait_cta(mma_bar, 1); + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + } + } + } + M3FA_PROF(3); + } + else + { + asm volatile("cp.async.wait_all;\n" ::: "memory"); +#if M3FA_TRIGGER_AT == 2 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (!FQKV || M3FA_MEGA_TRIGGER == 1) + { + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + } +#endif + for (int r = tid; r < BPC * kRows; r += kThreads) + { + row_lse[r] = kNegInf; + row_scale[r] = 0.0F; + } + __syncthreads(); + } + + // ---------------- Push every partial row to the CTA that merges its head (CTA r merges heads r*BPC..) -------- + if constexpr (S::kRecvAlias) + { + // Peers' partials overlay this CTA's tiles: every CTA's PV MMAs have completed. + asm volatile("barrier.cluster.arrive.aligned;\n" ::: "memory"); + } + if (!(kEarlyClw && !S::kRecvAlias)) + { + asm volatile("barrier.cluster.wait.aligned;\n" ::: "memory"); // peers' mbarriers are initialized + } + M3FA_PROF(4); + uint32_t const recv_base = smem_u32(recv); + if (kRs2 && warp < 8) + { + // Split push: warp group rg pushes rows 8 rg .. 8 rg + 7 (row j goes to CTA j, which merges head j). + int const rg = warp / 4; + int const d = (warp % 4) * 32 + lane; + float v[8]; + if (any_valid) + { + tmem_ld8(tmem + ((warp % 4) * 32 << 16) + BPC * kRows + 8 * rg, v); + } + else + { +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + v[j2] = 0.0F; + } + } +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + v[j2] *= row_scale[8 * rg + j2]; + } +#pragma unroll + for (int j2 = 0; j2 < 8; ++j2) + { + uint32_t const dst_cta = 8 * rg + j2; + uint32_t const row_addr = recv_base + static_cast((cr * kOPitch + d) * 4); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_shared(row_addr, dst_cta)), + "r"(__float_as_uint(v[j2])), "r"(mapa_shared(mbar, dst_cta)) + : "memory"); + } + } + else if (!kRs2 && warp < kComputeWarps && warp / 4 < kPairs) + { + int const g = warp / 4; + int const d = (warp % 4) * 32 + lane; // output dim +#pragma unroll + for (int k = 0; k < kPairsPerGroup; ++k) + { + int const q = g + 4 * k; + if (q < kPairs) + { + int const h = q % BPC; + int const c = q / BPC; + int const chunk = cr * BPC + h; + float v[16]; + if (any_valid && kRegCore) + { + float const* obuf + = reinterpret_cast(smem + S::kK16 + h * S::kTileBytes + S::kObufOff); +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[j] = obuf[(c * 16 + j) * kOPitch + d]; + } + } + else if (any_valid) + { + tmem_ld16(tmem + ((warp % 4) * 32 << 16) + (BPC + h) * kRows + c * 16, v); + } + else + { +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[j] = 0.0F; + } + } +#pragma unroll + for (int j = 0; j < 16; ++j) + { + v[j] *= row_scale[h * kRows + c * 16 + j]; + } + // One scalar store per row: a vector store's lanes would address several destination CTAs at once, + // which measured several times slower. +#pragma unroll + for (int j = 0; j < 16; ++j) + { + uint32_t const dst_cta = j / BPC; + uint32_t const row_addr + = recv_base + static_cast((((chunk * QG + c) * BPC + j % BPC) * kOPitch + d) * 4); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_shared(row_addr, dst_cta)), + "r"(__float_as_uint(v[j])), "r"(mapa_shared(mbar, dst_cta)) + : "memory"); + } + } + } + } + else if (tid >= kComputeThreads) + { + // Every partial row's LSE (BPC * kRows of them; more than the producer threads for the widest geometries). + for (int r = tid - kComputeThreads; r < BPC * kRows; r += kThreads - kComputeThreads) + { + int const h = r / kRows; + int const c = (r % kRows) / 16; + int const j = r % 16; + uint32_t const dst_cta = j / BPC; + uint32_t const row_addr + = recv_base + static_cast((((cr * BPC + h) * QG + c) * BPC + j % BPC) * kOPitch * 4); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_shared(row_addr + kHeadDim * 4, dst_cta)), + "r"(__float_as_uint(row_lse[r])), "r"(mapa_shared(mbar, dst_cta)) + : "memory"); + } + } + M3FA_PROF(10); +#pragma unroll + for (int k = 0; k < kMaxT; ++k) + { + if (wr_kind[k] == 1) + { + *reinterpret_cast(wr_ptr[k]) = static_cast(wr_val[k] & 0xFFFFu); + if (lane % 4 == 0) + { + *wr_sptr[k] = static_cast(wr_val[k] >> 16); + } + } + else if (wr_kind[k] == 2) + { + *reinterpret_cast(wr_ptr[k]) = wr_val[k]; + } + } + { + uint32_t ready = 0; +#ifdef M3FA_DBG_HANG + long long spins = 0; +#endif + while (!ready) + { +#ifdef M3FA_DBG_HANG + if (++spins == (1LL << 24)) + { + printf("M3FA hang: block %d thread %d merge recv\n", blockIdx.x, threadIdx.x); + } +#endif + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(mbar) + : "memory"); + } + } + M3FA_PROF(5); + + // ---------------- Merge heads [cr * BPC, cr * BPC + BPC) (split-K merge kernel's LSE weighting) ---------------- + constexpr int kMergeRows = QG * BPC; // (token, local head) pairs, 16 chunk weights each + // Wide geometries: chunk weights once per merged row (into the softmax scratch, free by now), then the weighted + // sums. Narrow ones recompute the weights per output element: no extra barrier on the critical path. + constexpr bool kRowWeights = kMergeRows >= 4; + float* wmerge = red_m; // [kMergeRows][kBlocks]; red_m holds BPC * kRows * 4 = kMergeRows * 64 floats + [[maybe_unused]] auto chunk_weights = [&](int mr, float(&lc)[kBlocks]) + { + float lmax = kNegInf; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + lc[c] = recv[(c * kMergeRows + mr) * kOPitch + kHeadDim]; + lmax = fmaxf(lmax, lc[c]); + } + lmax = lmax == kNegInf ? 0.0F : lmax; + float denom = 0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + lc[c] = ex2(lc[c] - lmax); // now the chunk weight; 0 for an empty chunk + denom += lc[c]; + } + denom = denom > 0.0F ? denom : 1.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + // -0.0 marks an empty chunk (skipped below, as the merge kernel skips zero weights). + lc[c] = lc[c] != 0.0F ? __fdividef(lc[c], denom) : -0.0F; + } + }; + if constexpr (kRowWeights) + { + for (int mr = tid; mr < kMergeRows; mr += kThreads) + { + float lc[kBlocks]; + chunk_weights(mr, lc); +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + wmerge[mr * kBlocks + c] = lc[c]; + } + } + __syncthreads(); + } +#pragma unroll + for (int idx = tid; idx < kMergeRows * kHeadDim; idx += kThreads) + { + int const mr = idx >> 7; + int const d = idx & (kHeadDim - 1); + float wn[kBlocks]; + if constexpr (kRowWeights) + { +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + wn[c] = wmerge[mr * kBlocks + c]; + } + } + else + { +#ifndef M3FA_MEGA_MERGE_LOCAL +#define M3FA_MEGA_MERGE_LOCAL \ + 0 // 1 (measured +0.6 us slower): every merging thread derives the 16 chunk weights itself (broadcast shared loads, + // independent exps): no shuffle chain; the same arithmetic as chunk_weights +#endif + if constexpr (M3FA_MEGA_MERGE_LOCAL != 0) + { + chunk_weights(mr, wn); + } + else + { + // Narrow geometries: the warp (32 dims of one merged row) derives the 16 chunk weights together, lane c + // holding chunk c's, and broadcasts them. + float lc = (lane < kBlocks) ? recv[(lane * kMergeRows + mr) * kOPitch + kHeadDim] : kNegInf; + float lmax = lc; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + lmax = fmaxf(lmax, __shfl_xor_sync(0xffffffffu, lmax, o)); + } + lmax = lmax == kNegInf ? 0.0F : lmax; + float const wc = lane < kBlocks ? ex2(lc - lmax) : 0.0F; + // The denominator in chunk order from 0.0, as chunk_weights (and the merge kernel) add it: a butterfly + // reduction rounds differently in the last bit. + float denom = 0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + denom += __shfl_sync(0xffffffffu, wc, c); + } + denom = denom > 0.0F ? denom : 1.0F; + float const wl = wc != 0.0F ? __fdividef(wc, denom) : -0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + wn[c] = __shfl_sync(0xffffffffu, wl, c); + } + } + } + if (tid == 0) + { + M3FA_CLK(45); + } + float acc = 0.0F; +#pragma unroll + for (int c = 0; c < kBlocks; ++c) + { + float const prod = recv[(c * kMergeRows + mr) * kOPitch + d] * wn[c]; + acc = __float_as_uint(wn[c]) != 0x80000000u ? acc + prod : acc; + } + if (tid == 0) + { + M3FA_CLK(46); + } + int const tl = mr / BPC; + int const head = cr * BPC + mr % BPC; + __nv_bfloat16 const ob = __float2bfloat16_rn(acc); + int const out_row = tok0 + tfirst + tl; + p.out[static_cast(out_row) * p.out_stride + head * kHeadDim + d] = ob; + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + { + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, mask)); + } + float const sfv = amax * rcp_approx_ftz(448.0F); + int const bits = __float_as_int(sfv); + int ex = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0 ? 1 : 0); + ex = min(ex, 254); + float scale = __int_as_float((254 - ex) << 23); + scale = amax != 0.0F ? scale : 0.0F; + __nv_fp8_e4m3 const q8v(x * scale); + int64_t const q8off = static_cast(out_row) * (p.num_sf_k_tiles * 128) + head * kHeadDim + d; + if constexpr (FQKV) + { + // Megakernel hand-off to the o_proj units (Lamport): this call's parity buffer, the other one reset to + // 0xFF. 0xFF is never written: a NaN code is canonicalized to 0x7F; the UE8M0 code is at most 254. + uint8_t const code = q8v.__x == 0xFFu ? static_cast(0x7Fu) : q8v.__x; + int64_t const cur = (mega_calls & 1u) * kMaxTok * kOpK; + int64_t const nxt = ((mega_calls & 1u) ^ 1u) * kMaxTok * kOpK; + asm volatile( + "st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.q8 + cur + q8off), "h"(static_cast(code)) + : "memory"); + asm volatile( + "st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.q8 + nxt + q8off), "h"(static_cast(0xFFu)) + : "memory"); + } + else + { + p.q8[q8off] = q8v.__x; + } + if (lane == 0) + { + int const k_sf = head * (kHeadDim / 32) + (d >> 5); + int64_t const off = static_cast(out_row / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (out_row % 32) * 16 + ((out_row % 128) / 32) * 4 + k_sf % 4; + if constexpr (FQKV) + { + int64_t const cur = (mega_calls & 1u) * kOpSfBytes; + int64_t const nxt = ((mega_calls & 1u) ^ 1u) * kOpSfBytes; + asm volatile( + "st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.sf + cur + off), "h"(static_cast(ex)) + : "memory"); + asm volatile("st.relaxed.gpu.global.u8 [%0], %1;\n" ::"l"(p.sf + nxt + off), + "h"(static_cast(0xFFu)) + : "memory"); + } + else + { + p.sf[off] = static_cast(ex); + } + } + } + } + M3FA_PROF(60); + if (any_valid) + { + // Every TMEM read completed above (tcgen05.wait::ld); release the columns. + asm volatile("tcgen05.fence::before_thread_sync;\n" ::: "memory"); + } + __syncthreads(); + if (FQKV && tid == 0) + { + p.flags[32 + blockIdx.x] = mega_calls + 1u; + } + M3FA_PROF(61); + if (warp == kWarps - 1) + { + asm volatile("tcgen05.fence::after_thread_sync;\n" ::: "memory"); + asm volatile("tcgen05.dealloc.cta_group::1.sync.aligned.b32 %0, %1;\n" ::"r"(*tmem_slot), "r"(S::kTmemCols) + : "memory"); + } + M3FA_PROF(6); +#else + __trap(); +#endif // MINIMAX_M3_ATTN_DEVICE_ENABLED +} + +template +cudaError_t launch_impl(Params const& prm, int num_reqs, bool pdl, cudaStream_t stream, int* max_clusters) +{ + auto kernel = m3_fused_decode_attn_kernel; + constexpr int smem = SmemX::kBytes; + constexpr int kCs = kBlocks / BPC; + static bool const configured = [&]() + { + bool ok = cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, smem) == cudaSuccess; + ok = ok + && (kCs <= 8 + || cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1) == cudaSuccess); + if (!ok) + { + cudaGetLastError(); + } + return ok; + }(); + if (!configured) + { + return cudaErrorInvalidConfiguration; + } + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(FQKV ? kQkvGrid : kCs * (QLEN / QG) * num_reqs); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeClusterDimension; + attr[0].val.clusterDim.x = kCs; + attr[0].val.clusterDim.y = 1; + attr[0].val.clusterDim.z = 1; + attr[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[1].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + if (max_clusters != nullptr) + { + cfg.numAttrs = 1; + cudaError_t const err = cudaOccupancyMaxActiveClusters(max_clusters, kernel, &cfg); + if (err != cudaSuccess) + { + cudaGetLastError(); + } + return err; + } + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +// The (QLEN, QG, BPC) geometries the target launches: default_geometry() of every step shape it routes here (one +// query token per 16-CTA cluster, or KV blocks paired into 8-CTA clusters), always with the MXFP8 hand-off. +#define M3_FA_GEOMETRIES(X) \ + X(1, 1, 1) \ + X(1, 1, 2) X(2, 1, 1) X(2, 1, 2) X(3, 1, 1) X(4, 1, 1) X(5, 1, 1) X(6, 1, 1) X(6, 1, 2) X(7, 1, 1) X(8, 1, 1) + +cudaError_t dispatch(Params const& prm, int qlen, int qg, int bpc, int num_reqs, bool pdl, cudaStream_t stream) +{ +#define M3_FA_CASE(Q, G, B) \ + if (qlen == Q && qg == G && bpc == B) \ + return launch_impl(prm, num_reqs, pdl, stream, nullptr); + M3_FA_GEOMETRIES(M3_FA_CASE) +#undef M3_FA_CASE + return cudaErrorInvalidValue; +} + +} // namespace + +bool minimaxM3FusedDecodeAttnSupports(int qlen, int qg, int bpc) +{ +#define M3_FA_CASE(Q, G, B) \ + if (qlen == Q && qg == G && bpc == B) \ + return true; + M3_FA_GEOMETRIES(M3_FA_CASE) +#undef M3_FA_CASE + return false; +} + +cudaError_t invokeMinimaxM3FusedDecodeAttn( + MinimaxM3FusedDecodeAttnParams const& params, int qlen, int qg, int bpc, int numReqs, bool pdl, cudaStream_t stream) +{ + return dispatch(params, qlen, qg, bpc, numReqs, pdl, stream); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.h b/cpp/tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.h new file mode 100644 index 000000000000..e1967d82f556 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.h @@ -0,0 +1,105 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Kernel parameters of the fused decode attention (the device code's Params). +struct MinimaxM3FusedDecodeAttnParams +{ + __nv_bfloat16 const* packed; + int64_t packed_stride; + uint8_t* kv_data; + int64_t d_page, d_plane; + uint8_t* kv_scale; + int64_t s_page, s_plane; + uint8_t* idx_k; + int64_t i_page, i_tok; + int64_t num_pages; + int const* out_cache_loc; + int const* position_ids; + int const* block_table; + int64_t bt_stride; + int bt_cols; + int const* seq_lens; + float const* kv_quant_scale; + float const* k_gs; + float const* v_gs; + __nv_bfloat16 const* q_w; + __nv_bfloat16 const* k_w; + __nv_bfloat16 const* ik_w; + float const* rope; + float eps; + float sm_scale; + __nv_bfloat16* out; + int64_t out_stride; + uint8_t* q8; + uint8_t* sf; + int num_sf_k_tiles; + unsigned long long* prof; // optional per-CTA clock64 probes (-DM3FA_PROFILE builds only) + // Fused QKV projection (FQKV kernels only): out[M, 2560] = mxfp8(x)[M, 6144] @ W^T. + __nv_bfloat16 const* x; // [M, 6144] layer input (the all-reduce output) + int ldx; + int num_tokens; // M + uint8_t const* wsf; // UE8M0 weight scales, 128x4-swizzled + float* part; // [2 parities][splits][8 tokens][2560] partial sums, sentinel at rest + int* counters; // [0]: call parity, [1]: CTAs done with this call (zero at rest) + int attn_clusters; // leading clusters that run the attention + uint8_t const* xq; // mega: the layer input's MXFP8 values [M, 6144] (row stride ldxq) ... + uint8_t const* xsf_act; // ... and its 128x4-swizzled UE8M0 scale atoms (atom k at byte 512 k) + int ldxq; + __nv_bfloat16* qkv_out; // mega: the packed QKV rows [2 parities][8][2560] the attention reads + uint8_t const* wsf_o; // mega: the o_proj weight's scale atoms + __nv_bfloat16* y_out; // mega: the o_proj output [M, 6144] + int* ar_ws; // mega, ar_mode: the next m3_mc_allreduce's multicast exchange (int32 words)... + int const* ar_flags; // ... its call flags (parity of a (token, column slice) slot = flags[t * 8 + slice] & 1) + int ar_rank; + int ar_world; + int ar_maxt; + int ar_mode; // 0: none; 2: push the o_proj rows into the exchange (as m3_mx_gemv's ar mode 2) + unsigned int* flags; // mega: [0, 20) tile counters (4 per call), [32 + CTA] the attention CTAs' targets + long long pf_a0, pf_b0, pf_a1, pf_b1; // mega: L2 prefetch regions issued in the attention window (bytes 0: none) + int pf_hint; // 1: evict_last policy on the prefetch + uint8_t const* wpre; // mega: optional pre-tiled QKV weight [N/128][K/128][16 KB swizzled smem image], or null + CUtensorMap wmap; // the [2560, 6144] E4M3 weight, 128 x 128 boxes, 128-byte swizzle + CUtensorMap wmap_o; // mega: the [6144, 2048] E4M3 o_proj weight, same boxes +}; + +//! Whether the (decode_query_len, query tokens per cluster, KV blocks per CTA) geometry is compiled. The MXFP8 +//! hand-off (q8 / sf) is part of every compiled variant. +bool minimaxM3FusedDecodeAttnSupports(int qlen, int qg, int bpc); + +//! Launch the fused norm/RoPE/KV-insert + identity-selection sparse decode + split-K merge kernel on `stream`. +//! `params` holds the packed QKV rows, the paged caches and the outputs (see the field comments). Returns +//! cudaErrorInvalidValue for a geometry minimaxM3FusedDecodeAttnSupports() rejects. +cudaError_t invokeMinimaxM3FusedDecodeAttn(MinimaxM3FusedDecodeAttnParams const& params, int qlen, int qg, int bpc, + int numReqs, bool pdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.cu b/cpp/tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.cu new file mode 100644 index 000000000000..fc60fb3035a9 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.cu @@ -0,0 +1,563 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/tllmException.h" + +#include +#include +#include +#include + +// The kernels use sm_100-family features (thread-block clusters, cp.async.bulk, st.async, mbarrier transaction +// counts, block-scaled conversions). Every other architecture of the build compiles them to empty bodies; the host +// entry points refuse to launch on those devices. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ == 1000 || __CUDA_ARCH__ == 1030) \ + && (defined(__CUDA_ARCH_FEAT_SM100_ALL) || defined(__CUDA_ARCH_FEAT_SM103_ALL) \ + || defined(__CUDA_ARCH_FAMILY_SPECIFIC__)) +#define MINIMAX_M3_KERNELS_ENABLED 1 +#else +#define MINIMAX_M3_KERNELS_ENABLED 0 +#endif + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace cg = cooperative_groups; + +namespace +{ + +constexpr int kEmpty = static_cast(0x80000000u); +constexpr int kFinMaxTopk = 8; // MoE finalize: top-k rows loaded together up to this k + +__device__ __forceinline__ uint32_t canon_bf16x2(uint32_t w) +{ + if ((w & 0xFFFFu) == 0x8000u) + w &= 0xFFFF0000u; + if ((w >> 16) == 0x8000u) + w &= 0x0000FFFFu; + return w; +} + +__device__ __forceinline__ float bf16_lo(uint32_t w) +{ + return __uint_as_float(w << 16); +} + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ float bf16_hi(uint32_t w) +{ + return __uint_as_float(w & 0xFFFF0000u); +} + +__device__ __forceinline__ int4 ld_volatile_v4(int const* p) +{ + int4 v; + asm volatile("ld.volatile.global.v4.s32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(p) + : "memory"); + return v; +} + +struct FinalizeArgs +{ + __nv_bfloat16 const* fc2; // [permuted rows, H] routed-expert FC2 output (null: plain input x) + int const* perm; // [M, topk] expanded index -> permuted row, -1 = dropped + void const* scales; // [M, topk] routing weights (fp32 or bf16) + int scale_fp32; + int topk; +}; + +// grid = min(M, MAXT) * CLUSTER CTAs, cluster (CLUSTER, 1, 1), THREADS threads. +// OPTS (each bit-identical): 1 residual + norm weight loaded at entry (before the poll); 2 cluster RMS exchange by +// st.async + mbarrier complete_tx instead of cluster.sync; 4 call parity rotated right after the poll; 8 max-smem +// carveout so the next kernel's CTAs co-reside early. +template +__global__ void __launch_bounds__(THREADS) mc_allreduce_norm_kernel(__nv_bfloat16 const* __restrict__ x, + __nv_bfloat16 const* __restrict__ x2, FinalizeArgs fin, int M, int H, __nv_bfloat16 const* __restrict__ residual, + __nv_bfloat16 const* __restrict__ norm_w, float eps, int* __restrict__ ws_uc, int* __restrict__ ws_mc, + int* __restrict__ flags, int rank, int world, __nv_bfloat16* __restrict__ out_normed, + __nv_bfloat16* __restrict__ out_residual, uint8_t* __restrict__ out_q, uint8_t* __restrict__ out_sf, int sf_k_tiles, + int prepushed) +{ +#if MINIMAX_M3_KERNELS_ENABLED + __shared__ float warp_sq[THREADS / 32]; + __shared__ alignas(16) float cluster_sq[CLUSTER]; + __shared__ alignas(8) unsigned long long xbar; + cg::cluster_group cluster = cg::this_cluster(); + if constexpr ((OPTS & 2) != 0) + { + if (threadIdx.x == 0) + { + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(b) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + } + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + } + int const crank = static_cast(cluster.block_rank()); + int const t = blockIdx.x / CLUSTER; + int const cols = H / CLUSTER; + int const c0 = crank * cols + threadIdx.x * 8; + bool const active = t < M && threadIdx.x * 8 < cols; + int64_t const half_words = static_cast(MAXT) * world * (H / 2); + + // prepushed 1: the producer (m3_mx_gemv) already pushed this rank's rows into the exchange slots; 2: and it + // launched this grid only after its pushes (its grid wait passed), so the slots are polled without a grid wait; + // 3: as 2, with x in set A and x2 in set B. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (prepushed < 2) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + int const parity = *reinterpret_cast(flags + blockIdx.x) & 1; + uint4 rraw = make_uint4(0u, 0u, 0u, 0u); + uint4 wraw = make_uint4(0u, 0u, 0u, 0u); + if constexpr ((OPTS & 1) != 0) + { + // the residual row was written grids earlier (prepushed >= 2: the producer launched this grid only after its + // own grid wait; otherwise the grid wait above) + if (active) + { + rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + wraw = *reinterpret_cast(norm_w + c0); + } + } + // Push this rank's slice into every rank's buffer. + if (active && !prepushed) + { + uint4 raw; + if (fin.fc2 != nullptr) + { + // MoE finalize: fp32 sum of the routing-weighted top-k expert rows plus the shared expert, rounded once. + float f[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; + if (fin.topk <= kFinMaxTopk) + { + // All top-k indices and scales, then every expert-row load in flight at once, then the same in-order + // FP32 accumulation as the loop below (one memory round trip instead of topk; bit-identical). + int prow[kFinMaxTopk]; + float sc[kFinMaxTopk]; +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + prow[k] = -1; + sc[k] = 0.f; + if (k < fin.topk) + { + int const e = t * fin.topk + k; + prow[k] = fin.perm[e]; + sc[k] = fin.scale_fp32 ? static_cast(fin.scales)[e] + : __bfloat162float(static_cast<__nv_bfloat16 const*>(fin.scales)[e]); + } + } + uint4 rv[kFinMaxTopk]; +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + if (prow[k] >= 0) + rv[k] = *reinterpret_cast(fin.fc2 + static_cast(prow[k]) * H + c0); + } +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + if (prow[k] < 0) + continue; + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv[k]); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] += v.x * sc[k]; + f[2 * q + 1] += v.y * sc[k]; + } + } + } + else + for (int k = 0; k < fin.topk; ++k) + { + int const e = t * fin.topk + k; + int const prow = fin.perm[e]; + if (prow < 0) + continue; + float const sc = fin.scale_fp32 + ? static_cast(fin.scales)[e] + : __bfloat162float(static_cast<__nv_bfloat16 const*>(fin.scales)[e]); + uint4 const rv = *reinterpret_cast(fin.fc2 + static_cast(prow) * H + c0); + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] += v.x * sc; + f[2 * q + 1] += v.y * sc; + } + } + if (x2 != nullptr) + { + // As stock: the routed output is rounded to BF16 first, then the shared expert is added (one more + // BF16 rounding of the FP32 sum). + uint4 const rv = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const routed = __bfloat1622float2(__floats2bfloat162_rn(f[2 * q], f[2 * q + 1])); + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] = routed.x + v.x; + f[2 * q + 1] = routed.y + v.y; + } + } + __nv_bfloat162* o = reinterpret_cast<__nv_bfloat162*>(&raw); +#pragma unroll + for (int q = 0; q < 4; ++q) + o[q] = __floats2bfloat162_rn(f[2 * q], f[2 * q + 1]); + } + else + { + raw = *reinterpret_cast(x + static_cast(t) * H + c0); + } + if (fin.fc2 == nullptr && x2 != nullptr) + { + uint4 const raw2 = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162* a = reinterpret_cast<__nv_bfloat162*>(&raw); + __nv_bfloat162 const* b = reinterpret_cast<__nv_bfloat162 const*>(&raw2); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const fa = __bfloat1622float2(a[q]); + float2 const fb = __bfloat1622float2(b[q]); + a[q] = __floats2bfloat162_rn(fa.x + fb.x, fa.y + fb.y); + } + } + int64_t const slot = parity * half_words + (static_cast(t) * world + rank) * (H / 2) + c0 / 2; + *reinterpret_cast(ws_mc + slot) + = make_int4(static_cast(canon_bf16x2(raw.x)), static_cast(canon_bf16x2(raw.y)), + static_cast(canon_bf16x2(raw.z)), static_cast(canon_bf16x2(raw.w))); + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float s[8] = {}; + float sq = 0.f; + if (active) + { + // The residual row (written grids earlier) and every rank's slot are loaded together; slots still empty are + // re-polled. Ranks are then summed in rank order, as before. + if constexpr ((OPTS & 1) == 0) + rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + constexpr int kMaxWorld = 8; + int* const p0 = ws_uc + parity * half_words + static_cast(t) * world * (H / 2) + c0 / 2; + int4 v[kMaxWorld]; + int4 v2[kMaxWorld]; + bool const two = prepushed == 3; + int* const p1 = p0 + 2 * half_words; // set B + uint32_t pending = (1u << world) - 1u; + uint32_t pending2 = two ? pending : 0u; + long long spins = 0; + while ((pending | pending2) != 0u) + { +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if ((pending >> r) & 1u) + v[r] = ld_volatile_v4(p0 + static_cast(r) * (H / 2)); + if ((pending2 >> r) & 1u) + v2[r] = ld_volatile_v4(p1 + static_cast(r) * (H / 2)); + } +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if (((pending >> r) & 1u) && v[r].x != kEmpty && v[r].y != kEmpty && v[r].z != kEmpty + && v[r].w != kEmpty) + pending &= ~(1u << r); + if (((pending2 >> r) & 1u) && v2[r].x != kEmpty && v2[r].y != kEmpty && v2[r].z != kEmpty + && v2[r].w != kEmpty) + pending2 &= ~(1u << r); + } + if (++spins > (1LL << 31)) + __trap(); + } +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if (r < world) + { + *reinterpret_cast(p0 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t ww[4] = {static_cast(v[r].x), static_cast(v[r].y), + static_cast(v[r].z), static_cast(v[r].w)}; + if (two) + { + *reinterpret_cast(p1 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t const w2[4] = {static_cast(v2[r].x), static_cast(v2[r].y), + static_cast(v2[r].z), static_cast(v2[r].w)}; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + __nv_bfloat162 const u + = __floats2bfloat162_rn(bf16_lo(ww[q]) + bf16_lo(w2[q]), bf16_hi(ww[q]) + bf16_hi(w2[q])); + ww[q] = *reinterpret_cast(&u); + } + } +#pragma unroll + for (int q = 0; q < 4; ++q) + { + s[2 * q] += bf16_lo(ww[q]); + s[2 * q + 1] += bf16_hi(ww[q]); + } + } + } + __nv_bfloat162 const* rr = reinterpret_cast<__nv_bfloat162 const*>(&rraw); + uint32_t upd[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const rf = __bfloat1622float2(rr[q]); + __nv_bfloat162 const u = __floats2bfloat162_rn(s[2 * q] + rf.x, s[2 * q + 1] + rf.y); + upd[q] = *reinterpret_cast(&u); + float2 const uf = __bfloat1622float2(u); + s[2 * q] = uf.x; + s[2 * q + 1] = uf.y; + sq += uf.x * uf.x + uf.y * uf.y; + } + *reinterpret_cast(out_residual + static_cast(t) * H + c0) + = make_uint4(upd[0], upd[1], upd[2], upd[3]); + } + if constexpr ((OPTS & 4) != 0) + { + // every CTA (active or not) rotates its call parity once its own poll is done (read only by later grids) + if (threadIdx.x == 0) + flags[blockIdx.x] = parity ^ 1; + } +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + sq += __shfl_xor_sync(0xffffffffu, sq, off); + if ((threadIdx.x & 31) == 0) + warp_sq[threadIdx.x / 32] = sq; + __syncthreads(); + if constexpr ((OPTS & 2) != 0) + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); // peers' xbar initialized (long since) + if (threadIdx.x == 0) + { + float total = 0.f; +#pragma unroll + for (int w = 0; w < THREADS / 32; ++w) + total += warp_sq[w]; + if constexpr ((OPTS & 2) == 0) + for (int c = 0; c < CLUSTER; ++c) + *cluster.map_shared_rank(&cluster_sq[crank], c) = total; + else + { + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + uint32_t const d = static_cast(__cvta_generic_to_shared(&cluster_sq[crank])); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(b), "r"(CLUSTER * 4) + : "memory"); + for (int c = 0; c < CLUSTER; ++c) + { + uint32_t rd, rb; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rd) : "r"(d), "r"(c)); + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rb) : "r"(b), "r"(c)); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(rd), + "r"(__float_as_uint(total)), "r"(rb) + : "memory"); + } + } + } + if constexpr ((OPTS & 2) == 0) + cluster.sync(); + else + { + // every peer's sum landed (st.async complete_tx on this CTA's barrier); no cluster-wide release-arrive + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + uint32_t done = 0; + while (!done) + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}" + : "=r"(done) + : "r"(b) + : "memory"); + } + if (active) + { + float total = 0.f; +#pragma unroll + for (int c = 0; c < CLUSTER; ++c) + total += cluster_sq[c]; + float const inv = rsqrtf(total / static_cast(H) + eps); + if constexpr ((OPTS & 1) == 0) + wraw = *reinterpret_cast(norm_w + c0); + __nv_bfloat162 const* wv = reinterpret_cast<__nv_bfloat162 const*>(&wraw); + uint32_t o[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const wf = __bfloat1622float2(wv[q]); + __nv_bfloat162 const n = __floats2bfloat162_rn(s[2 * q] * inv * wf.x, s[2 * q + 1] * inv * wf.y); + o[q] = *reinterpret_cast(&n); + float2 const nf = __bfloat1622float2(n); + s[2 * q] = nf.x; + s[2 * q + 1] = nf.y; + } + *reinterpret_cast(out_normed + static_cast(t) * H + c0) = make_uint4(o[0], o[1], o[2], o[3]); + } + if (out_q != nullptr) + { + // MXFP8 copy of the normed row for the next GEMM, as trtllm::mxfp8_quantize(normed, True): four threads + // hold one 32-column scale block. + float amax = 0.f; + if (active) + { +#pragma unroll + for (int e = 0; e < 8; ++e) + amax = fmaxf(amax, fabsf(s[e])); + } + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 1)); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 2)); + if (active) + { + __nv_fp8_e8m0 code; + code.__x = __nv_cvt_float_to_e8m0(amax * rcp_approx_ftz(448.0f), __NV_SATFINITE, cudaRoundPosInf); + float const out_scale = amax != 0.f ? rcp_approx_ftz(static_cast(code)) : 0.0f; + uint32_t packed[2]; +#pragma unroll + for (int h = 0; h < 2; ++h) + { + __nv_fp8x2_e4m3 const lo(make_float2(s[4 * h] * out_scale, s[4 * h + 1] * out_scale)); + __nv_fp8x2_e4m3 const hi(make_float2(s[4 * h + 2] * out_scale, s[4 * h + 3] * out_scale)); + packed[h] = static_cast(lo.__x) | (static_cast(hi.__x) << 16); + } + *reinterpret_cast(out_q + static_cast(t) * H + c0) = make_uint2(packed[0], packed[1]); + if ((threadIdx.x & 3) == 0) + { + int const kb = c0 / 32; + out_sf[static_cast(t / 128) * sf_k_tiles * 512 + (kb / 4) * 512 + (t % 32) * 16 + + ((t % 128) / 32) * 4 + kb % 4] + = code.__x; + } + } + } + if constexpr ((OPTS & 4) == 0) + { + __syncthreads(); + if (threadIdx.x == 0) + flags[blockIdx.x] = parity ^ 1; + } +#endif +} + +template +void launch(MinimaxM3McAllreduceNormParams const& p, bool pdl, cudaStream_t stream) +{ + int const M = p.numTokens, H = p.hidden; + TLLM_CHECK_WITH_INFO(H % (CLUSTER * 8) == 0 && H / CLUSTER / 8 <= THREADS, "unsupported hidden size %d", H); + TLLM_CHECK_WITH_INFO(M >= 1 && M <= MAXT, "%d tokens do not fit the %d-token bucket", M, MAXT); + auto kernel = mc_allreduce_norm_kernel; + static bool const configured = [&]() + { + bool ok = CLUSTER <= 8 + || cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1) == cudaSuccess; + if ((OPTS & 8) != 0) + ok = ok + && cudaFuncSetAttribute( + kernel, cudaFuncAttributePreferredSharedMemoryCarveout, cudaSharedmemCarveoutMaxShared) + == cudaSuccess; + if (!ok) + cudaGetLastError(); + return ok; + }(); + TLLM_CHECK_WITH_INFO(configured, "cluster size not supported"); + FinalizeArgs const fin{p.fc2, p.perm, p.scales, p.scaleFp32 ? 1 : 0, p.topk}; + cudaLaunchConfig_t cfg = {}; + // The grid is sized to the call's rows: min(M, MAXT) clusters. Each (row, slice) slot keeps its own parity flag, + // read by the producers before they push and rotated only by the slot's CTA after it polled: a slot that a call + // does not use keeps its parity, and every rank runs the same calls, so the parities stay consistent when M + // changes between calls. + cfg.gridDim = dim3((M < MAXT ? M : MAXT) * CLUSTER); + cfg.blockDim = dim3(THREADS); + cfg.stream = stream; + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + attr[1].id = cudaLaunchAttributeClusterDimension; + attr[1].val.clusterDim.x = CLUSTER; + attr[1].val.clusterDim.y = 1; + attr[1].val.clusterDim.z = 1; + cfg.attrs = attr; + cfg.numAttrs = 2; + cudaError_t const err + = cudaLaunchKernelEx(&cfg, kernel, p.x, p.x2, fin, M, H, p.residual, p.normWeight, p.eps, p.wsUc, p.wsMc, + p.flags, p.rank, p.world, p.outNormed, p.outResidual, p.outQuant, p.outSf, (H / 32 + 3) / 4, p.prepushed); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +constexpr int kCluster = 8; +constexpr int kThreads = 128; + +} // namespace + +bool isMinimaxM3McAllreduceNormSupported(int32_t maxTokens, int32_t opts) +{ + return (maxTokens == 8 && (opts == 7 || opts == 15)) || (maxTokens == 16 && opts == 15) + || (maxTokens == 48 && opts == 15); +} + +void invokeMinimaxM3McAllreduceNorm( + MinimaxM3McAllreduceNormParams const& params, int32_t maxTokens, int32_t opts, bool pdl, cudaStream_t stream) +{ + if (maxTokens == 8 && opts == 7) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 15) + { + launch(params, pdl, stream); + } + else if (maxTokens == 16 && opts == 15) + { + launch(params, pdl, stream); + } + else if (maxTokens == 48 && opts == 15) + { + launch(params, pdl, stream); + } + else + { + TLLM_THROW("unsupported (bucket %d, opts %d)", maxTokens, opts); + } +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.h b/cpp/tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.h new file mode 100644 index 000000000000..b80a78c9e173 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.h @@ -0,0 +1,76 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Operands of the MiniMax-M3 multicast all-reduce + residual + RMSNorm kernel (decode, at most 48 tokens). +//! +//! Per rank the kernel forms the partial `bf16(x + x2)` (or, in finalize mode, +//! `bf16(bf16(sum_k scale * fc2[perm]) + x2)`, the routed sum rounded before the shared expert is added, as stock), +//! pushes it through the multicast mapping `wsMc` into every rank's exchange, polls its own unicast view `wsUc`, +//! sums the ranks in FP32 in rank order, adds the residual (BF16 round) and applies RMSNorm. With `outQuant` it +//! also writes the MXFP8 copy of the normed rows (E4M3 [numTokens, hidden] and swizzled UE8M0 scales). +struct MinimaxM3McAllreduceNormParams +{ + __nv_bfloat16 const* x{nullptr}; //!< [numTokens, hidden] partial; in finalize mode the FC2 rows [rows, hidden]. + __nv_bfloat16 const* x2{nullptr}; //!< Optional second addend [numTokens, hidden]. + __nv_bfloat16 const* fc2{nullptr}; //!< Finalize mode: the permuted FC2 output (same buffer as x); null otherwise. + int32_t const* perm{nullptr}; //!< Finalize mode: [numTokens, topk] expanded index -> permuted row, -1 dropped. + void const* scales{nullptr}; //!< Finalize mode: [numTokens, topk] routing weights (fp32 or bf16). + bool scaleFp32{false}; + int32_t topk{0}; + int32_t numTokens{0}; + int32_t hidden{0}; + __nv_bfloat16 const* residual{nullptr}; //!< [numTokens, hidden] + __nv_bfloat16 const* normWeight{nullptr}; //!< [hidden] + float eps{0.F}; + int32_t* wsUc{nullptr}; //!< This rank's unicast view of the exchange (int32 words, re-armed after reading). + int32_t* wsMc{nullptr}; //!< The multicast mapping of the exchange. + int32_t* flags{nullptr}; //!< Per-CTA call-parity counters, [maxTokens * 8]. + int32_t rank{0}; + int32_t world{0}; + __nv_bfloat16* outNormed{nullptr}; + __nv_bfloat16* outResidual{nullptr}; + uint8_t* outQuant{nullptr}; //!< Optional E4M3 [numTokens, hidden]. + uint8_t* outSf{nullptr}; //!< Optional UE8M0 scales, 128x4-swizzled. + //! 0: push x (+ x2) here; 2: a producer already pushed x and launched this grid after its pushes; 3: as 2, with x + //! in exchange set A and x2 in set B. + int32_t prepushed{0}; +}; + +//! The (bucket, epilogue options) pairs this launcher is built for: (8, 7), (8, 15), (16, 15) and (48, 15). +bool isMinimaxM3McAllreduceNormSupported(int32_t maxTokens, int32_t opts); + +//! Launch with 8-CTA clusters of 128 threads, one cluster per token. `maxTokens` selects the exchange bucket; +//! `opts` the epilogue variant (every variant is bit-identical). +void invokeMinimaxM3McAllreduceNorm( + MinimaxM3McAllreduceNormParams const& params, int32_t maxTokens, int32_t opts, bool pdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/msaKvUpdate.cu b/cpp/tensorrt_llm/kernels/minimaxM3/msaKvUpdate.cu new file mode 100644 index 000000000000..2fb9f768c1bd --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/msaKvUpdate.cu @@ -0,0 +1,70 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/msaKvUpdate.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ +__global__ void __launch_bounds__(256) + msa_kv_update_kernel(int const* __restrict__ kv_lens, int const* __restrict__ staged, int const* __restrict__ qo, + int const* __restrict__ q_batch_row, int const* __restrict__ q_intra, int const* __restrict__ table, + long long table_ld, int ncols, int page, int batch, int total_q, int* __restrict__ seq_lens_out, + int* __restrict__ cu_kv, int* __restrict__ out_loc, int* __restrict__ n_valid) +{ + for (int b = threadIdx.x; b < batch; b += blockDim.x) + seq_lens_out[b] = min(kv_lens[b], staged[b]); + if (cu_kv != nullptr && threadIdx.x == 0) + { + int acc = 0; + cu_kv[0] = 0; + for (int b = 0; b < batch; ++b) + { + acc += min(kv_lens[b], staged[b]); + cu_kv[b + 1] = acc; + } + } + for (int t = threadIdx.x; t < total_q; t += blockDim.x) + { + int const r = q_batch_row[t]; + int const pos = min(kv_lens[r], staged[r]) - qo[r] + q_intra[t]; + int const ps = max(pos, 0); + int const col = min(ps / page, ncols - 1); + out_loc[t] = table[r * table_ld + col] * page + ps % page; + if (n_valid != nullptr) + n_valid[t] = (max(pos + 1, 0) + page - 1) / page; + } +} +} // namespace + +cudaError_t invokeMinimaxM3MsaKvUpdate(int32_t const* kvLens, int32_t const* staged, int32_t const* qo, + int32_t const* qBatchRow, int32_t const* qIntra, int32_t const* table, int64_t tableLd, int32_t ncols, int32_t page, + int32_t batch, int32_t totalQ, int32_t* seqLensOut, int32_t* cuKv, int32_t* outLoc, int32_t* nValid, + cudaStream_t stream) +{ + msa_kv_update_kernel<<<1, 256, 0, stream>>>(kvLens, staged, qo, qBatchRow, qIntra, table, + static_cast(tableLd), ncols, page, batch, totalQ, seqLensOut, cuKv, outLoc, nValid); + return cudaGetLastError(); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/msaKvUpdate.h b/cpp/tensorrt_llm/kernels/minimaxM3/msaKvUpdate.h new file mode 100644 index 000000000000..d703cef6df73 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/msaKvUpdate.h @@ -0,0 +1,45 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! MiniMax-M3 MSA KV-length correction in one single-CTA launch, for all int32 operands: +//! +//! kvTrue[b] = min(kvLens[b], staged[b]) -> seqLensOut[b]; cuKv = exclusive prefix sum (optional) +//! pos[t] = kvTrue[r] - qo[r] + qIntra[t] (r = qBatchRow[t]) +//! outLoc[t] = table[r, min(max(pos, 0) / page, ncols - 1)] * page + max(pos, 0) % page +//! nValid[t] = (max(pos + 1, 0) + page - 1) / page (optional) +//! +//! cuKv and nValid may be null. table is row-major with row stride tableLd (elements) and ncols columns. +cudaError_t invokeMinimaxM3MsaKvUpdate(int32_t const* kvLens, int32_t const* staged, int32_t const* qo, + int32_t const* qBatchRow, int32_t const* qIntra, int32_t const* table, int64_t tableLd, int32_t ncols, int32_t page, + int32_t batch, int32_t totalQ, int32_t* seqLensOut, int32_t* cuKv, int32_t* outLoc, int32_t* nValid, + cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.cu b/cpp/tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.cu new file mode 100644 index 000000000000..2e942d5b5a57 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.cu @@ -0,0 +1,820 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/tllmException.h" + +#include +#include +#include +#include + +// The kernels use sm_100-family features (thread-block clusters, cp.async.bulk, st.async, mbarrier transaction +// counts, block-scaled conversions). Every other architecture of the build compiles them to empty bodies; the host +// entry points refuse to launch on those devices. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ == 1000 || __CUDA_ARCH__ == 1030) \ + && (defined(__CUDA_ARCH_FEAT_SM100_ALL) || defined(__CUDA_ARCH_FEAT_SM103_ALL) \ + || defined(__CUDA_ARCH_FAMILY_SPECIFIC__)) +#define MINIMAX_M3_KERNELS_ENABLED 1 +#else +#define MINIMAX_M3_KERNELS_ENABLED 0 +#endif + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace cg = cooperative_groups; + +namespace +{ + +constexpr int kMaxM = 48; +constexpr int kRowPad = 32; +constexpr int kEmptyWord = static_cast(0x80000000u); + +// Layer-boundary reduction inputs of the AR prologue (see m3_ar_fused.py); unused otherwise. +struct ArArgs +{ + int const* uc; + int const* cur_buf; + int64_t buf_words; + int world; + __nv_bfloat16 const* residual; + __nv_bfloat16 const* norm_w; + float eps; + __nv_bfloat16* residual_out; + uint8_t* xq_g; // [8][K] quantized activation published by tile 0 + uint8_t* xc_g; // [8][K / 32] its UE8M0 scales + int* flags; // [CLUSTER_K] per K-slice publish generation (call + 1) +}; + +// Optional L2 prefetch of the paged KV blocks the step's sparse decode will read (issued before the PDL wait, so it +// overlaps the producer kernel); all-null when unused. +struct PfArgs +{ + int const* block_table; // [rows, bt_stride] page ids + int const* seq_lens; // [rows] KV length per row + int rows; + int max_blocks; + int bt_stride; + char const* data; // page-major KV data pool + long long data_stride; // bytes between pages + int data_bytes; // contiguous bytes per page + char const* scale; // page-major block-scale pool (or null) + long long scale_stride; + int scale_bytes; +}; + +// Already-quantized activation (E4M3 rows [M, K] + UE8M0 scales in the 128x4-swizzled layout of +// trtllm::mxfp8_quantize(x, True)); all-null to quantize the BF16 activation in-kernel. +struct PreqArgs +{ + uint8_t const* xq; + uint8_t const* xsf; + int sf_k_tiles; // ceil(K / 128) +}; + +__device__ __forceinline__ void prefetch_kv_pages(PfArgs const& pf) +{ + if (pf.block_table == nullptr) + return; + int const units = pf.rows * pf.max_blocks; + for (int u = blockIdx.x; u < units; u += gridDim.x) + { + int const r = u / pf.max_blocks; + int const b = u - r * pf.max_blocks; + if (b * 128 >= pf.seq_lens[r]) + continue; + long long const page = pf.block_table[static_cast(r) * pf.bt_stride + b]; + char const* d = pf.data + page * pf.data_stride; + for (int off = threadIdx.x * 128; off < pf.data_bytes; off += blockDim.x * 128) + asm volatile("prefetch.global.L2::evict_last [%0];" ::"l"(d + off)); + if (pf.scale != nullptr) + { + char const* sc = pf.scale + page * pf.scale_stride; + for (int off = threadIdx.x * 128; off < pf.scale_bytes; off += blockDim.x * 128) + asm volatile("prefetch.global.L2::evict_last [%0];" ::"l"(sc + off)); + } + } +} + +__device__ __forceinline__ uint32_t mapa_shared(uint32_t addr, uint32_t target_rank) +{ + uint32_t r; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;\n" : "=r"(r) : "r"(addr), "r"(target_rank)); + return r; +} + +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ float pow2_from_ue8m0(uint32_t e) +{ + return __uint_as_float(e == 0 ? 0x00400000u : (e << 23)); +} + +__device__ __forceinline__ void mma_e4m3( + float (&d)[4], uint32_t a0, uint32_t a1, uint32_t a2, uint32_t a3, uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k32.row.col.f32.e4m3.e4m3.f32 " + "{%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, {%10,%11,%12,%13};\n" + : "=f"(d[0]), "=f"(d[1]), "=f"(d[2]), "=f"(d[3]) + : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(b0), "r"(b1), "f"(0.f), "f"(0.f), "f"(0.f), "f"(0.f)); +} + +__device__ __forceinline__ void bulk_g2s(void* dst, void const* src, uint32_t bytes, uint64_t* bar) +{ + asm volatile("cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes [%0], [%1], %2, [%3];\n" ::"r"( + smem_u32(dst)), + "l"(src), "r"(bytes), "r"(smem_u32(bar)) + : "memory"); +} + +__device__ __forceinline__ void mbar_wait(uint64_t* bar, uint32_t parity) +{ + asm volatile( + "{\n .reg .pred p;\n WAIT_%=:\n mbarrier.try_wait.parity.shared::cta.b64 p, [%0], %1;\n @!p bra " + "WAIT_%=;\n}\n" ::"r"(smem_u32(bar)), + "r"(parity) + : "memory"); +} + +// grid: (N / (16 * RT) * CLUSTER_K); cluster (CLUSTER_K, 1, 1); block WARPS warps. +// RT: 16-row weight tiles per CTA (amortizes the activation fetch + quantization over more rows). +// NT: number of 8-row activation tiles (MMA n=8 each), 1 for M <= 8 and 2 for M <= 16. +template +__global__ void __launch_bounds__(WARPS * 32) mxfp8_low_m_gemm_tma_kernel(__nv_bfloat16 const* __restrict__ x, int ldx, + int M, int N, int K, uint8_t const* __restrict__ w, uint8_t const* __restrict__ wsf, int num_k_tiles, + __nv_bfloat16* __restrict__ out, int ldo, int red_aliased, ArArgs ar, PfArgs pf, PreqArgs preq) +{ +#if MINIMAX_M3_KERNELS_ENABLED + constexpr int kThreads = WARPS * 32; + constexpr int kRows = 16 * RT; + constexpr int kTile = kRows * 8 * NT; + extern __shared__ __align__(128) uint8_t smem[]; + __shared__ __align__(8) uint64_t bar; + __shared__ __align__(8) uint64_t red_bar; + + int const rank = CLUSTER_K > 1 ? static_cast(cg::this_cluster().block_rank()) : 0; + int const tile = blockIdx.x / CLUSTER_K; + int const row0 = tile * kRows; + int const kc = K / CLUSTER_K; // columns owned by this CTA + int const kc_blocks = kc / 32; + int const k_begin = rank * kc; + int const kb_begin = k_begin / 32; + int const row_stride = kc + kRowPad; + int const warp = threadIdx.x / 32; + int const lane = threadIdx.x % 32; + + uint8_t* w_s = smem; // [kRows][row_stride] + uint8_t* sf_s = w_s + kRows * row_stride; // [RT][kc_blocks / 4][256] + uint8_t* xq_s = sf_s + RT * (kc_blocks / 4) * 256; // [M][kc] + uint8_t* xc_s = xq_s + M * kc; // [M][kc_blocks] + // Peer partial tiles (root only), then the warp partials unless they reuse the dead weight tile. + float* cluster_red = reinterpret_cast(xc_s + ((M * kc_blocks + 15) / 16) * 16); + float* red = red_aliased ? reinterpret_cast(w_s) : cluster_red + (CLUSTER_K - 1) * kTile; + // AR: the reduced + residual-added activation slice, BF16 [M][kc]. + __nv_bfloat16* u_s = reinterpret_cast<__nv_bfloat16*>( + reinterpret_cast(cluster_red + (CLUSTER_K - 1) * kTile) + (red_aliased ? 0 : WARPS * kTile * 4)); + __shared__ float tok_sq[kMaxM]; + __shared__ float peer_sq[CLUSTER_K][kMaxM]; + + uint32_t const w_bytes = kRows * kc; + uint32_t const sf_bytes = RT * (kc_blocks / 4) * 256u; + if (threadIdx.x == 0) + { + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;\n" ::"r"(smem_u32(&bar))); + if (CLUSTER_K > 1 && rank == 0) + { + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;\n" ::"r"(smem_u32(&red_bar))); + } + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + asm volatile( + "mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(smem_u32(&bar)), "r"(w_bytes + sf_bytes) + : "memory"); + if (CLUSTER_K > 1 && rank == 0) + { + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(smem_u32(&red_bar)), + "r"(static_cast((CLUSTER_K - 1) * kTile * sizeof(float))) + : "memory"); + } + } + __syncthreads(); + // The root's reduction barrier must be initialized before any peer pushes into it. + if constexpr (CLUSTER_K > 1) + { + asm volatile("barrier.cluster.arrive.release.aligned;\n" ::: "memory"); + } + // Weight rows and scale chunks do not depend on the producer kernel. + if (warp == 0) + { + for (int r = lane; r < kRows; r += 32) + { + bulk_g2s(w_s + r * row_stride, w + static_cast(row0 + r) * K + k_begin, kc, &bar); + } + int const chunks = kc_blocks / 4; + for (int t = lane; t < RT * chunks; t += 32) + { + int const rt = t / chunks; + int const c = t % chunks; + int const rr = row0 + 16 * rt; + int64_t const sf_base = static_cast(rr / 128) * num_k_tiles * 512 + (rr % 32) * 16; + bulk_g2s(sf_s + t * 256, wsf + sf_base + static_cast(kb_begin / 4 + c) * 512, 256, &bar); + } + } + prefetch_kv_pages(pf); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + if constexpr (AR) + { + // Layer-boundary reduction: sum the ranks' pushed partials for this CTA's K slice, add the residual + // (BF16 round, as the unfused add), then normalize with the RMS statistic of the whole row (the + // cluster spans K) and quantize like the unfused path. + if (threadIdx.x < kMaxM) + tok_sq[threadIdx.x] = 0.f; + __syncthreads(); + int const call = *reinterpret_cast(ar.cur_buf); + if (tile == 0) + { + int* base = const_cast(ar.uc) + static_cast(call & 1) * ar.buf_words; + int const half_k = K / 2; + for (int p = threadIdx.x; p < M * kc_blocks; p += kThreads) + { + int const m = p / kc_blocks; + int const b = p % kc_blocks; + int const k0 = k_begin + b * 32; + float v[32]; +#pragma unroll + for (int e = 0; e < 32; ++e) + v[e] = 0.f; + constexpr int kWorld = 4; + int4 wv[kWorld][4]; + int* src0 = base + static_cast(m) * kWorld * half_k + k0 / 2; +#pragma unroll + for (int r = 0; r < kWorld; ++r) + { +#pragma unroll + for (int q = 0; q < 4; ++q) + { + asm volatile("ld.volatile.global.v4.s32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(wv[r][q].x), "=r"(wv[r][q].y), "=r"(wv[r][q].z), "=r"(wv[r][q].w) + : "l"(src0 + r * half_k + 4 * q) + : "memory"); + } + } + long long spins = 0; + while (true) + { + bool pending = false; +#pragma unroll + for (int r = 0; r < kWorld; ++r) + { +#pragma unroll + for (int q = 0; q < 4; ++q) + { + int4 const c = wv[r][q]; + if (c.x == kEmptyWord || c.y == kEmptyWord || c.z == kEmptyWord || c.w == kEmptyWord) + { + pending = true; + asm volatile("ld.volatile.global.v4.s32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(wv[r][q].x), "=r"(wv[r][q].y), "=r"(wv[r][q].z), "=r"(wv[r][q].w) + : "l"(src0 + r * half_k + 4 * q) + : "memory"); + } + } + } + if (!pending) + break; + if (++spins > (1LL << 30)) + __trap(); + } +#pragma unroll + for (int r = 0; r < kWorld; ++r) + { +#pragma unroll + for (int q = 0; q < 4; ++q) + { + *reinterpret_cast(src0 + r * half_k + 4 * q) + = make_int4(kEmptyWord, kEmptyWord, kEmptyWord, kEmptyWord); + uint32_t const ww[4] = {static_cast(wv[r][q].x), static_cast(wv[r][q].y), + static_cast(wv[r][q].z), static_cast(wv[r][q].w)}; +#pragma unroll + for (int h = 0; h < 4; ++h) + { + v[q * 8 + 2 * h] += __uint_as_float(ww[h] << 16); + v[q * 8 + 2 * h + 1] += __uint_as_float(ww[h] & 0xFFFF0000u); + } + } + } + uint4 const* rsrc = reinterpret_cast(ar.residual + static_cast(m) * K + k0); + uint4* rdst = reinterpret_cast(ar.residual_out + static_cast(m) * K + k0); + uint4* udst = reinterpret_cast(u_s + m * kc + b * 32); + float sq = 0.f; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + uint4 const rraw = rsrc[q]; + __nv_bfloat162 const* rr = reinterpret_cast<__nv_bfloat162 const*>(&rraw); + uint32_t uw[4]; +#pragma unroll + for (int h = 0; h < 4; ++h) + { + float2 const rf = __bfloat1622float2(rr[h]); + __nv_bfloat162 const u + = __floats2bfloat162_rn(v[q * 8 + 2 * h] + rf.x, v[q * 8 + 2 * h + 1] + rf.y); + uw[h] = *reinterpret_cast(&u); + float2 const uf = __bfloat1622float2(u); + sq += uf.x * uf.x + uf.y * uf.y; + } + uint4 const uu = make_uint4(uw[0], uw[1], uw[2], uw[3]); + udst[q] = uu; + if (tile == 0) + rdst[q] = uu; + } + atomicAdd(&tok_sq[m], sq); + } + __syncthreads(); + if constexpr (CLUSTER_K > 1) + { + cg::cluster_group cl = cg::this_cluster(); + asm volatile("barrier.cluster.wait.acquire.aligned;\n" ::: "memory"); + if (threadIdx.x < M) + { + for (int c = 0; c < CLUSTER_K; ++c) + { + *cl.map_shared_rank(&peer_sq[rank][threadIdx.x], c) = tok_sq[threadIdx.x]; + } + } + asm volatile("barrier.cluster.arrive.release.aligned;\n" ::: "memory"); + asm volatile("barrier.cluster.wait.acquire.aligned;\n" ::: "memory"); + // Re-arm for the split-K reduction's wait below. + asm volatile("barrier.cluster.arrive.release.aligned;\n" ::: "memory"); + } + else + { + if (threadIdx.x < M) + peer_sq[0][threadIdx.x] = tok_sq[threadIdx.x]; + __syncthreads(); + } + for (int p = threadIdx.x; p < M * kc_blocks; p += kThreads) + { + int const m = p / kc_blocks; + int const b = p % kc_blocks; + float total = 0.f; +#pragma unroll + for (int c = 0; c < CLUSTER_K; ++c) + total += peer_sq[c][m]; + float const inv = rsqrtf(total / static_cast(K) + ar.eps); + uint4 const* usrc = reinterpret_cast(u_s + m * kc + b * 32); + uint4 const* wsrc = reinterpret_cast(ar.norm_w + k_begin + b * 32); + float v[32]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + uint4 const uraw = usrc[q]; + uint4 const wraw = wsrc[q]; + __nv_bfloat162 const* uu = reinterpret_cast<__nv_bfloat162 const*>(&uraw); + __nv_bfloat162 const* wwv = reinterpret_cast<__nv_bfloat162 const*>(&wraw); +#pragma unroll + for (int h = 0; h < 4; ++h) + { + float2 const uf = __bfloat1622float2(uu[h]); + float2 const wf = __bfloat1622float2(wwv[h]); + float2 const nf + = __bfloat1622float2(__floats2bfloat162_rn(uf.x * inv * wf.x, uf.y * inv * wf.y)); + v[q * 8 + 2 * h] = nf.x; + v[q * 8 + 2 * h + 1] = nf.y; + } + } + float amax = 0.f; +#pragma unroll + for (int t = 0; t < 32; ++t) + amax = fmaxf(amax, fabsf(v[t])); + float const sf = amax * rcp_approx_ftz(448.0f); + __nv_fp8_e8m0 code; + code.__x = __nv_cvt_float_to_e8m0(sf, __NV_SATFINITE, cudaRoundPosInf); + float const out_scale = amax != 0.f ? rcp_approx_ftz(static_cast(code)) : 0.0f; + uint32_t packed[8]; +#pragma unroll + for (int t = 0; t < 8; ++t) + { + __nv_fp8x2_e4m3 const lo(make_float2(v[4 * t] * out_scale, v[4 * t + 1] * out_scale)); + __nv_fp8x2_e4m3 const hi(make_float2(v[4 * t + 2] * out_scale, v[4 * t + 3] * out_scale)); + packed[t] = static_cast(lo.__x) | (static_cast(hi.__x) << 16); + } + uint4* dst = reinterpret_cast(xq_s + m * kc + b * 32); + dst[0] = make_uint4(packed[0], packed[1], packed[2], packed[3]); + dst[1] = make_uint4(packed[4], packed[5], packed[6], packed[7]); + xc_s[m * kc_blocks + b] = code.__x; + } + // Publish the quantized slice for the other tiles. + for (int p = threadIdx.x; p < M * kc_blocks; p += kThreads) + { + int const m = p / kc_blocks; + int const bb = p % kc_blocks; + uint4 const* src = reinterpret_cast(xq_s + m * kc + bb * 32); + uint4* dst = reinterpret_cast(ar.xq_g + static_cast(m) * K + k_begin + bb * 32); + dst[0] = src[0]; + dst[1] = src[1]; + ar.xc_g[static_cast(m) * (K / 32) + kb_begin + bb] = xc_s[m * kc_blocks + bb]; + } + __syncthreads(); + if (threadIdx.x == 0) + { + asm volatile("fence.acq_rel.gpu;" ::: "memory"); + asm volatile("st.release.gpu.global.s32 [%0], %1;" ::"l"(ar.flags + rank), "r"(call + 1) : "memory"); + } + } + else + { + if (threadIdx.x == 0) + { + int v; + long long spins = 0; + do + { + asm volatile("ld.acquire.gpu.global.s32 %0, [%1];" : "=r"(v) : "l"(ar.flags + rank) : "memory"); + if (++spins > (1LL << 31)) + __trap(); + } while (v - (call + 1) < 0); + } + __syncthreads(); + for (int p = threadIdx.x; p < M * kc_blocks; p += kThreads) + { + int const m = p / kc_blocks; + int const bb = p % kc_blocks; + uint4 const* src + = reinterpret_cast(ar.xq_g + static_cast(m) * K + k_begin + bb * 32); + uint4* dst = reinterpret_cast(xq_s + m * kc + bb * 32); + dst[0] = __ldcg(src); + dst[1] = __ldcg(src + 1); + xc_s[m * kc_blocks + bb] = __ldcg(ar.xc_g + static_cast(m) * (K / 32) + kb_begin + bb); + } + if constexpr (CLUSTER_K > 1) + { + // Keep the cluster barrier phases aligned with tile 0's prologue. + asm volatile("barrier.cluster.wait.acquire.aligned;\n" ::: "memory"); + asm volatile("barrier.cluster.arrive.release.aligned;\n" ::: "memory"); + } + } + } + else if (preq.xq != nullptr) + { + // Stage this CTA's K slice of the pre-quantized activation. + int const vec_per_row = kc / 16; + int const vecs = M * vec_per_row; +#pragma unroll 4 + for (int p = threadIdx.x; p < vecs; p += kThreads) + { + int const m = p / vec_per_row; + int const v = p - m * vec_per_row; + reinterpret_cast(xq_s + m * kc)[v] + = __ldcg(reinterpret_cast(preq.xq + static_cast(m) * K + k_begin) + v); + } + for (int p = threadIdx.x; p < M * kc_blocks; p += kThreads) + { + int const m = p / kc_blocks; + int const b = p - m * kc_blocks; + int const kb = kb_begin + b; + xc_s[m * kc_blocks + b] = __ldcg(preq.xsf + static_cast(m / 128) * preq.sf_k_tiles * 512 + + (kb / 4) * 512 + (m % 32) * 16 + ((m % 128) / 32) * 4 + kb % 4); + } + } + else + { + // Quantize the activation columns owned by this CTA (native mxfp8_quantize math). Each thread + // issues the loads of all its blocks before converting any, so the CTA pays one L2 round trip. + int const items = M * kc_blocks; + for (int p0 = threadIdx.x; p0 < items; p0 += 2 * kThreads) + { + uint4 raw[2][4]; +#pragma unroll + for (int u = 0; u < 2; ++u) + { + int const p = p0 + u * kThreads; + if (p < items) + { + int const m = p / kc_blocks; + int const b = p % kc_blocks; + uint4 const* src + = reinterpret_cast(x + static_cast(m) * ldx + k_begin + b * 32); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + raw[u][q] = src[q]; + } + } + } +#pragma unroll + for (int u = 0; u < 2; ++u) + { + int const p = p0 + u * kThreads; + if (p >= items) + { + continue; + } + int const m = p / kc_blocks; + int const b = p % kc_blocks; + float v[32]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&raw[u][q]); +#pragma unroll + for (int t = 0; t < 4; ++t) + { + float2 const f = __bfloat1622float2(h[t]); + v[q * 8 + t * 2] = f.x; + v[q * 8 + t * 2 + 1] = f.y; + } + } + float amax = 0.f; +#pragma unroll + for (int t = 0; t < 32; ++t) + { + amax = fmaxf(amax, fabsf(v[t])); + } + float const sf = amax * rcp_approx_ftz(448.0f); + __nv_fp8_e8m0 code; + code.__x = __nv_cvt_float_to_e8m0(sf, __NV_SATFINITE, cudaRoundPosInf); + float const out_scale = amax != 0.f ? rcp_approx_ftz(static_cast(code)) : 0.0f; + uint32_t packed[8]; +#pragma unroll + for (int t = 0; t < 8; ++t) + { + __nv_fp8x2_e4m3 const lo(make_float2(v[4 * t] * out_scale, v[4 * t + 1] * out_scale)); + __nv_fp8x2_e4m3 const hi(make_float2(v[4 * t + 2] * out_scale, v[4 * t + 3] * out_scale)); + packed[t] = static_cast(lo.__x) | (static_cast(hi.__x) << 16); + } + uint4* dst = reinterpret_cast(xq_s + m * kc + b * 32); + dst[0] = make_uint4(packed[0], packed[1], packed[2], packed[3]); + dst[1] = make_uint4(packed[4], packed[5], packed[6], packed[7]); + xc_s[m * kc_blocks + b] = code.__x; + } + } + } + __syncthreads(); + mbar_wait(&bar, 0); + + int const g = lane >> 2; + int const j = lane & 3; + int const col0 = 2 * j; + int const bpw = kc_blocks / WARPS; + int const chunks = kc_blocks / 4; + float acc[RT][NT][4] = {}; +#pragma unroll 2 + for (int i = 0; i < bpw; ++i) + { + int const b = warp * bpw + i; + int const kb = kb_begin + b; + uint2 xb[NT]; + float xs0[NT], xs1[NT]; +#pragma unroll + for (int t = 0; t < NT; ++t) + { + int const row = 8 * t + g; + int const c0 = 8 * t + col0; + xb[t] = row < M ? *reinterpret_cast(xq_s + row * kc + b * 32 + 8 * j) : make_uint2(0, 0); + xs0[t] = c0 < M ? pow2_from_ue8m0(xc_s[c0 * kc_blocks + b]) : 0.f; + xs1[t] = c0 + 1 < M ? pow2_from_ue8m0(xc_s[(c0 + 1) * kc_blocks + b]) : 0.f; + } +#pragma unroll + for (int rt = 0; rt < RT; ++rt) + { + uint8_t const* wa_s = w_s + (16 * rt + g) * row_stride + 8 * j + b * 32; + uint2 const wa = *reinterpret_cast(wa_s); + uint2 const wb = *reinterpret_cast(wa_s + 8 * row_stride); + int const inner_m = ((row0 + 16 * rt) % 128) / 32; + uint8_t const* sft = sf_s + (rt * chunks + b / 4) * 256 + inner_m * 4 + kb % 4; + float const wsa = pow2_from_ue8m0(sft[g * 16]); + float const wsb = pow2_from_ue8m0(sft[(g + 8) * 16]); +#pragma unroll + for (int t = 0; t < NT; ++t) + { + float d[4]; + mma_e4m3(d, wa.x, wb.x, wa.y, wb.y, xb[t].x, xb[t].y); + acc[rt][t][0] += d[0] * (wsa * xs0[t]); + acc[rt][t][1] += d[1] * (wsa * xs1[t]); + acc[rt][t][2] += d[2] * (wsb * xs0[t]); + acc[rt][t][3] += d[3] * (wsb * xs1[t]); + } + } + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + if (red_aliased) + { + __syncthreads(); + } +#pragma unroll + for (int rt = 0; rt < RT; ++rt) + { +#pragma unroll + for (int t = 0; t < NT; ++t) + { + int const ra = 16 * rt + g; + red[warp * kTile + ra * 8 * NT + 8 * t + col0] = acc[rt][t][0]; + red[warp * kTile + ra * 8 * NT + 8 * t + col0 + 1] = acc[rt][t][1]; + red[warp * kTile + (ra + 8) * 8 * NT + 8 * t + col0] = acc[rt][t][2]; + red[warp * kTile + (ra + 8) * 8 * NT + 8 * t + col0 + 1] = acc[rt][t][3]; + } + } + __syncthreads(); + + // Warp reduction in fixed order. Peers push their partial tile into the root's shared memory with + // st.async (completing bytes on the root's mbarrier) and exit; the root sums ranks in order. + if constexpr (CLUSTER_K > 1) + { + asm volatile("barrier.cluster.wait.acquire.aligned;\n" ::: "memory"); + } + if (CLUSTER_K == 1 || rank == 0) + { + if constexpr (CLUSTER_K > 1) + { + uint32_t ready = 0; + while (!ready) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(ready) + : "r"(smem_u32(&red_bar)) + : "memory"); + } + } + __syncthreads(); + for (int idx = threadIdx.x; idx < kTile; idx += kThreads) + { + int const r = idx / (8 * NT); + int const m = idx % (8 * NT); + if (m < M) + { + float total = 0.f; +#pragma unroll + for (int ww = 0; ww < WARPS; ++ww) + { + total += red[ww * kTile + idx]; + } +#pragma unroll + for (int c = 1; c < CLUSTER_K; ++c) + { + total += cluster_red[(c - 1) * kTile + idx]; + } + out[static_cast(m) * ldo + row0 + r] = __float2bfloat16_rn(total); + } + } + } + else + { + uint32_t const remote_bar = mapa_shared(smem_u32(&red_bar), 0); + uint32_t const remote_dst = mapa_shared(smem_u32(cluster_red + (rank - 1) * kTile), 0); + for (int idx = threadIdx.x; idx < kTile; idx += kThreads) + { + float s = 0.f; +#pragma unroll + for (int ww = 0; ww < WARPS; ++ww) + { + s += red[ww * kTile + idx]; + } + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + remote_dst + idx * 4), + "r"(__float_as_uint(s)), "r"(remote_bar) + : "memory"); + } + } +#endif +} + +template +void launch(__nv_bfloat16 const* x, int ldx, int M, int N, int K, uint8_t const* w, uint8_t const* wsf, + __nv_bfloat16* out, int ldo, bool pdl, cudaStream_t stream) +{ + constexpr bool AR = false; + int const kc = K / CLUSTER_K; + int const kc_blocks = kc / 32; + TLLM_CHECK_WITH_INFO(N % (16 * RT) == 0 && K % (CLUSTER_K * 128) == 0 && kc_blocks % WARPS == 0, + "unsupported shape N=%d K=%d", N, K); + constexpr int kTile = 16 * RT * 8 * NT; + size_t const w_bytes = 16 * RT * (kc + kRowPad); + size_t const red_bytes = static_cast(WARPS) * kTile * sizeof(float); + int const red_aliased = red_bytes <= w_bytes ? 1 : 0; + size_t const smem = w_bytes + RT * (kc_blocks / 4) * 256 + M * kc + ((M * kc_blocks + 15) / 16) * 16 + + (CLUSTER_K - 1) * kTile * sizeof(float) + (red_aliased ? 0 : red_bytes); + auto kernel = mxfp8_low_m_gemm_tma_kernel; + // Configure once, outside any CUDA graph capture, to the largest supported request. + static bool const configured = [&]() + { + int device = 0, max_optin = 0; + cudaGetDevice(&device); + cudaDeviceGetAttribute(&max_optin, cudaDevAttrMaxSharedMemoryPerBlockOptin, device); + cudaFuncAttributes attrs = {}; + if (cudaFuncGetAttributes(&attrs, kernel) != cudaSuccess) + { + cudaGetLastError(); + return false; + } + bool const ok = cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, + max_optin - static_cast(attrs.sharedSizeBytes)) + == cudaSuccess; + if (!ok) + { + cudaGetLastError(); + } + return ok; + }(); + TLLM_CHECK_WITH_INFO(configured, "failed to configure dynamic shared memory"); + cudaFuncAttributes fattrs = {}; + cudaFuncGetAttributes(&fattrs, kernel); + TLLM_CHECK_WITH_INFO(static_cast(smem) <= fattrs.maxDynamicSharedSizeBytes, + "shared memory request exceeds the configured limit"); + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(N / (16 * RT) * CLUSTER_K); + cfg.blockDim = dim3(WARPS * 32); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + attr[1].id = cudaLaunchAttributeClusterDimension; + attr[1].val.clusterDim.x = CLUSTER_K; + attr[1].val.clusterDim.y = 1; + attr[1].val.clusterDim.z = 1; + cfg.attrs = attr; + cfg.numAttrs = 2; + cudaError_t const err = cudaLaunchKernelEx( + &cfg, kernel, x, ldx, M, N, K, w, wsf, (K / 32 + 3) / 4, out, ldo, red_aliased, ArArgs{}, PfArgs{}, PreqArgs{}); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +} // namespace + +bool isMinimaxM3Mxfp8LowMGemmSupported(int32_t warps, int32_t clusterK, int32_t rowTiles, int32_t m) +{ + bool const built = (warps == 4 && clusterK == 4 && rowTiles == 2) || (warps == 8 && clusterK == 2 && rowTiles == 4) + || (warps == 4 && clusterK == 4 && rowTiles == 1) || (warps == 4 && clusterK == 2 && rowTiles == 2); + return built && m >= 1 && m <= 8; +} + +void invokeMinimaxM3Mxfp8LowMGemm(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int32_t n, int32_t k, + uint8_t const* w, uint8_t const* wsf, __nv_bfloat16* out, int32_t ldo, int32_t warps, int32_t clusterK, + int32_t rowTiles, bool pdl, cudaStream_t stream) +{ + TLLM_CHECK_WITH_INFO(isMinimaxM3Mxfp8LowMGemmSupported(warps, clusterK, rowTiles, m), + "unsupported configuration (warps %d, cluster_k %d, row_tiles %d) at M=%d", warps, clusterK, rowTiles, m); + // One 8-row activation tile (NT = 1) covers every built row count. + if (warps == 4 && clusterK == 4 && rowTiles == 2) + { + launch<4, 4, 1, 2>(x, ldx, m, n, k, w, wsf, out, ldo, pdl, stream); + } + else if (warps == 8 && clusterK == 2 && rowTiles == 4) + { + launch<8, 2, 1, 4>(x, ldx, m, n, k, w, wsf, out, ldo, pdl, stream); + } + else if (warps == 4 && clusterK == 4 && rowTiles == 1) + { + launch<4, 4, 1, 1>(x, ldx, m, n, k, w, wsf, out, ldo, pdl, stream); + } + else + { + launch<4, 2, 1, 2>(x, ldx, m, n, k, w, wsf, out, ldo, pdl, stream); + } +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.h b/cpp/tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.h new file mode 100644 index 000000000000..433cb6a1fe07 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.h @@ -0,0 +1,46 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Whether `invokeMinimaxM3Mxfp8LowMGemm` is built for this (warps, clusterK, rowTiles) configuration at M rows: +//! (4, 4, 2), (8, 2, 4), (4, 4, 1) and (4, 2, 2), each for 1 <= M <= 8. +bool isMinimaxM3Mxfp8LowMGemmSupported(int32_t warps, int32_t clusterK, int32_t rowTiles, int32_t m); + +//! Fused MXFP8 activation quantization + low-M MXFP8 GEMM: `out[M, N] = mxfp8(x)[M, K] @ w[N, K]^T` in BF16. +//! +//! `x` is BF16 [M, K] with row stride `ldx`; `w` is E4M3 [N, K] row-major with UE8M0 scales `wsf` in the 128x4 +//! swizzled layout of `mxfp8_quantize(w, True)`; `out` is BF16 [M, N] with row stride `ldo`. Each CTA owns +//! 16 * rowTiles weight rows and K / clusterK columns; `warps` warps per CTA. +void invokeMinimaxM3Mxfp8LowMGemm(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int32_t n, int32_t k, + uint8_t const* w, uint8_t const* wsf, __nv_bfloat16* out, int32_t ldo, int32_t warps, int32_t clusterK, + int32_t rowTiles, bool pdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/producerPw.cu b/cpp/tensorrt_llm/kernels/minimaxM3/producerPw.cu new file mode 100644 index 000000000000..fb3e4bfb3a89 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/producerPw.cu @@ -0,0 +1,324 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/producerPw.h" + +#include +#include +#include + +// MiniMax-M3 NVFP4 QKV + indexer norm/RoPE/KV-insert producer with an early PDL trigger. +// +// Same math and outputs as minimaxM3Nvfp4QKVIndexerNormRopeKVInsertKernel (fusedQKNormRopeKernel.cu, built with +// --use_fast_math like the rest of the library), specialized to 1 KV head, 1 index head, head_dim 128, rotary_dim 64, +// page 128, NVFP4 main KV and FP8 index-K, with 8 (TP8) or 16 (TP4) query heads per rank. One warp per (token, +// packed head). +// +// The decode producer is latency bound: after griddepcontrol.wait the native kernel runs a chain of dependent global +// loads (packed row -> norm weight -> position id -> RoPE row -> cache slot -> quant scale). This kernel reads the +// constant norm weights and KV quant scale before griddepcontrol.wait and issues the packed row, position id and cache +// slot loads together after it, followed by the RoPE row. With `early` it also calls griddepcontrol.launch_dependents +// on entry: consumers read its outputs only after their own griddepcontrol.wait, which waits for this grid to complete, +// so the early trigger only lets the next kernel launch and run its pre-wait prologue while this one runs. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +namespace producerPw +{ +namespace +{ + +constexpr int kHeadDim = 128; +constexpr int kRotaryDim = 64; +constexpr int kPageSize = 128; +constexpr int kElems = kHeadDim / 32; +constexpr int kHeadsKV = 1; +constexpr int kHeadsIndex = 1; +constexpr int kBlock = 256; + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ uint32_t sf_group_mask4() +{ + uint32_t laneId; + asm("mov.u32 %0, %%laneid;" : "=r"(laneId)); + return 0xFu << (laneId & ~3u); +} + +__device__ __forceinline__ uint32_t fp32x8_to_e2m1(float (&a)[8]) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + uint32_t val; + asm volatile( + "{\n" + ".reg .b8 byte0;\n" + ".reg .b8 byte1;\n" + ".reg .b8 byte2;\n" + ".reg .b8 byte3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte0, %2, %1;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte1, %4, %3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte2, %6, %5;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte3, %8, %7;\n" + "mov.b32 %0, {byte0, byte1, byte2, byte3};\n" + "}" + : "=r"(val) + : "f"(a[0]), "f"(a[1]), "f"(a[2]), "f"(a[3]), "f"(a[4]), "f"(a[5]), "f"(a[6]), "f"(a[7])); + return val; +#else + return 0; +#endif +} + +__device__ __forceinline__ void store_fp8x4(__nv_fp8_e4m3* out, int64_t off, float const (&e)[kElems]) +{ + __nv_fp8x2_e4m3 const low(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const high(make_float2(e[2], e[3])); + *reinterpret_cast(out + off) = static_cast(low.__x) | (static_cast(high.__x) << 16); +} + +__device__ __forceinline__ uint16_t quantize_fp4x4(float const (&e)[4], float secondLevelScale, uint8_t* scaleOut) +{ + __nv_bfloat16 rounded[4]; + float localMax = 0.0F; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + rounded[i] = __float2bfloat16_rn(e[i]); + localMax = fmaxf(localMax, fabsf(__bfloat162float(rounded[i]))); + } + uint32_t const groupMask = sf_group_mask4(); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 1)); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 2)); + float const scaleValue = secondLevelScale * (localMax * rcp_approx_ftz(6.0F)); + __nv_fp8_e4m3 const encodedScale(scaleValue); + if (scaleOut != nullptr) + *scaleOut = encodedScale.__x; + float const narrowedScale = static_cast(encodedScale); + float const outputScale + = localMax != 0.0F ? rcp_approx_ftz(narrowedScale * rcp_approx_ftz(secondLevelScale)) : 0.0F; + float q[8] = {}; +#pragma unroll + for (int i = 0; i < 4; ++i) + q[i] = __bfloat162float(rounded[i]) * outputScale; + return static_cast(fp32x8_to_e2m1(q)); +} + +template +__global__ void __launch_bounds__(kBlock) m3_producer_kernel(Params p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + constexpr int kTotalHeads = kHeadsQ + 2 * kHeadsKV + kHeadsIndex + 1; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (kEarly) + { + asm volatile("griddepcontrol.launch_dependents;"); + } +#endif + int const warpId = threadIdx.x / 32; + int const laneId = threadIdx.x % 32; + int const globalWarp = blockIdx.x * (kBlock / 32) + warpId; + int const tokenIdx = globalWarp / kTotalHeads; + int const localHead = globalWarp % kTotalHeads; + if (tokenIdx >= p.numTokens) + return; + + constexpr int kBegin = kHeadsQ; + constexpr int vBegin = kBegin + kHeadsKV; + constexpr int indexQBegin = vBegin + kHeadsKV; + constexpr int indexKHead = indexQBegin + kHeadsIndex; + bool const isQ = localHead < kBegin; + bool const isK = localHead >= kBegin && localHead < vBegin; + bool const isV = localHead >= vBegin && localHead < indexQBegin; + bool const isIndexQ = localHead >= indexQBegin && localHead < indexKHead; + bool const isIndexK = localHead == indexKHead; + + // Only constant data is read before the wait: norm weights and the KV quant scale. + float w[kElems] = {}; + float quantScale = 0.0F; + __nv_bfloat16 const* normWeight + = isQ ? p.qWeight : (isK ? p.kWeight : (isIndexQ ? p.indexQWeight : p.indexKWeight)); + if (!isV) + { + uint2 const wv = *reinterpret_cast(normWeight + laneId * kElems); + float2 const w01 = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&wv.x)); + float2 const w23 = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&wv.y)); + w[0] = w01.x; + w[1] = w01.y; + w[2] = w23.x; + w[3] = w23.y; + } + if (isK || isV) + { + quantScale = p.kvQuantScale[isV ? 2 : 1]; + } + + int64_t const inputOffset = (static_cast(tokenIdx) * kTotalHeads + localHead) * kHeadDim + laneId * kElems; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + // Position ids and cache slots are per-step metadata, read after the wait. The packed row, the position id and + // the slot are independent loads, and the RoPE row load is issued before the norm reduction, so the + // native kernel's serial chain of dependent loads becomes two round trips. + uint2 const packed = *reinterpret_cast(p.packed + inputOffset); + bool const rotates = !isV && laneId * kElems < kRotaryDim; + int const positionId = rotates ? p.positionIds[tokenIdx] : 0; + int const slot = (isK || isV || isIndexK) ? p.outCacheLoc[tokenIdx] : 0; + float cosine[kElems] = {}; + float sine[kElems] = {}; + if (rotates) + { + int64_t const ropeRow = static_cast(positionId) * kRotaryDim; + int const coefficient = (laneId * kElems) % (kRotaryDim / 2); + float4 const c = *reinterpret_cast(p.rotaryCosSin + ropeRow + coefficient); + float4 const s = *reinterpret_cast(p.rotaryCosSin + ropeRow + kRotaryDim / 2 + coefficient); + cosine[0] = c.x; + cosine[1] = c.y; + cosine[2] = c.z; + cosine[3] = c.w; + sine[0] = s.x; + sine[1] = s.y; + sine[2] = s.z; + sine[3] = s.w; + } + + float e[kElems]; + float sumSquares = 0.0F; +#pragma unroll + for (int pair = 0; pair < 2; ++pair) + { + float2 const v = __bfloat1622float2( + *reinterpret_cast<__nv_bfloat162 const*>(reinterpret_cast(&packed) + pair)); + e[2 * pair] = v.x; + e[2 * pair + 1] = v.y; + if (!isV) + sumSquares += v.x * v.x + v.y * v.y; + } + + if (!isV) + { +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + sumSquares += __shfl_xor_sync(0xffffffff, sumSquares, mask, 32); + float const rmsReciprocal = rsqrtf(sumSquares / static_cast(kHeadDim) + p.eps); +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + e[i] *= rmsReciprocal * (1.0F + w[i]); + } + constexpr int kPairOffset = (kRotaryDim / 2) / kElems; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + int const dim = laneId * kElems + i; + float paired = __shfl_xor_sync(0xffffffff, e[i], kPairOffset); + if (dim < kRotaryDim) + { + if (dim < kRotaryDim / 2) + paired = -paired; + // The native kernel's contraction of e * cos + paired * sin; spelled out so this build matches it. + e[i] = __fmaf_rn(paired, sine[i], __fmul_rn(e[i], cosine[i])); + } + } + } + + if (isQ) + { + store_fp8x4(p.qOut, (static_cast(tokenIdx) * kHeadsQ + localHead) * kHeadDim + laneId * kElems, e); + return; + } + if (isIndexQ) + { + store_fp8x4(p.indexQOut, + (static_cast(tokenIdx) * kHeadsIndex + localHead - indexQBegin) * kHeadDim + laneId * kElems, e); + return; + } + + if (slot < 0) + return; + int const page = slot / kPageSize; + if (page >= p.numPages) + return; + int const withinPage = slot & (kPageSize - 1); + if (isIndexK) + { + store_fp8x4(p.indexK, + static_cast(page) * p.indexPageStride + static_cast(withinPage) * p.indexTokenStride + + laneId * kElems, + e); + return; + } + + int const plane = isV ? 1 : 0; + int const scaleColumn = laneId / 4; + int64_t const scaleWithinPage = isV + ? static_cast(withinPage / 4) * 32 + static_cast(scaleColumn) * 4 + withinPage % 4 + : static_cast(withinPage) * 8 + scaleColumn; + int64_t const scaleOffset = static_cast(page) * p.scalePageStride + + static_cast(plane) * p.scalePlaneStride + scaleWithinPage; + uint8_t* scaleOut = laneId % 4 == 0 ? p.kvScale + scaleOffset : nullptr; + uint16_t const packedFp4 = quantize_fp4x4(e, quantScale, scaleOut); + int64_t const dataOffset = static_cast(page) * p.dataPageStride + + static_cast(plane) * p.dataPlaneStride + static_cast(withinPage) * p.dataTokenStride + + laneId * 2; + *reinterpret_cast(p.kvData + dataOffset) = packedFp4; +#endif +} + +template +cudaError_t launch(Params const& p, int numTokens, bool pdl, cudaStream_t stream) +{ + constexpr int kTotalHeads = kHeadsQ + 2 * kHeadsKV + kHeadsIndex + 1; + int const warps = numTokens * kTotalHeads; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3((warps + kBlock / 32 - 1) / (kBlock / 32)); + cfg.blockDim = dim3(kBlock); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, m3_producer_kernel, p); +} + +} // namespace + +cudaError_t invokeProducerPw(Params const& p, int numTokens, int headsQ, bool pdl, bool early, cudaStream_t stream) +{ + if (numTokens <= 0) + return cudaSuccess; + if (headsQ == 16) + return early ? launch<16, true>(p, numTokens, pdl, stream) : launch<16, false>(p, numTokens, pdl, stream); + if (headsQ == 8) + return early ? launch<8, true>(p, numTokens, pdl, stream) : launch<8, false>(p, numTokens, pdl, stream); + return cudaErrorInvalidValue; +} + +} // namespace producerPw +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/producerPw.h b/cpp/tensorrt_llm/kernels/minimaxM3/producerPw.h new file mode 100644 index 000000000000..6d6848e0e199 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/producerPw.h @@ -0,0 +1,69 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +//! The MiniMax-M3 NVFP4 QKV + indexer RMSNorm / RoPE / KV-insert producer with its constant loads before the PDL +//! grid-dependency wait and an optional early dependents trigger (sm_103a). Same outputs, bit for bit, as +//! minimaxM3Nvfp4QKVIndexerNormRopeKVInsert. See producerPw.cu. +namespace producerPw +{ + +struct Params +{ + __nv_bfloat16 const* packed; + __nv_fp8_e4m3* qOut; + __nv_fp8_e4m3* indexQOut; + uint8_t* kvData; + uint8_t* kvScale; + __nv_fp8_e4m3* indexK; + int const* outCacheLoc; + float const* kvQuantScale; + int64_t dataPageStride, dataPlaneStride, dataHeadStride, dataTokenStride; + int64_t scalePageStride, scalePlaneStride, scaleHeadStride; + int64_t indexPageStride, indexTokenStride, numPages; + int numTokens; + float eps; + __nv_bfloat16 const* qWeight; + __nv_bfloat16 const* kWeight; + __nv_bfloat16 const* indexQWeight; + __nv_bfloat16 const* indexKWeight; + float const* rotaryCosSin; + int const* positionIds; +}; + +//! Launch over `numTokens` tokens, one warp per (token, packed head); headsQ is 8 or 16. +cudaError_t invokeProducerPw(Params const& p, int numTokens, int headsQ, bool pdl, bool early, cudaStream_t stream); + +} // namespace producerPw +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.cu b/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.cu new file mode 100644 index 000000000000..d12b765e5ecf --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.cu @@ -0,0 +1,606 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/routerGemv.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" + +#include +#include +#include + +// The kernels use sm_100-family features (thread-block clusters, cp.async.bulk, st.async, mbarrier transaction +// counts, block-scaled conversions). Every other architecture of the build compiles them to empty bodies; the host +// entry points refuse to launch on those devices. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ == 1000 || __CUDA_ARCH__ == 1030) \ + && (defined(__CUDA_ARCH_FEAT_SM100_ALL) || defined(__CUDA_ARCH_FEAT_SM103_ALL) \ + || defined(__CUDA_ARCH_FAMILY_SPECIFIC__)) +#define MINIMAX_M3_KERNELS_ENABLED 1 +#else +#define MINIMAX_M3_KERNELS_ENABLED 0 +#endif + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kMaxM = 16; + +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ void bulk_g2s(void* dst, void const* src, uint32_t bytes, uint64_t* bar) +{ + asm volatile("cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes [%0], [%1], %2, [%3];\n" ::"r"( + smem_u32(dst)), + "l"(src), "r"(bytes), "r"(smem_u32(bar)) + : "memory"); +} + +__device__ __forceinline__ void mbar_wait(uint64_t* bar, uint32_t parity) +{ + asm volatile( + "{\n .reg .pred p;\n WAIT_%=:\n mbarrier.try_wait.parity.shared::cta.b64 p, [%0], %1;\n @!p bra " + "WAIT_%=;\n}\n" ::"r"(smem_u32(bar)), + "r"(parity) + : "memory"); +} + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ uint32_t e2m1x8(float2 const (&v)[4]) +{ + uint32_t r; + asm("{\n .reg .b8 b0, b1, b2, b3;\n" + " cvt.rn.satfinite.e2m1x2.f32 b0, %2, %1;\n cvt.rn.satfinite.e2m1x2.f32 b1, %4, %3;\n" + " cvt.rn.satfinite.e2m1x2.f32 b2, %6, %5;\n cvt.rn.satfinite.e2m1x2.f32 b3, %8, %7;\n" + " mov.b32 %0, {b0, b1, b2, b3};\n}\n" + : "=r"(r) + : "f"(v[0].x), "f"(v[0].y), "f"(v[1].x), "f"(v[1].y), "f"(v[2].x), "f"(v[2].y), "f"(v[3].x), "f"(v[3].y)); + return r; +} + +// NVFP4 (16-element E4M3 block scale, per-tensor global scale) quantization of one 16-value block, bit-identical +// to trtllm fp4_quantize (cvt_warp_fp16_to_fp4, linear scale layout). +__device__ __forceinline__ void quant_block16(__nv_bfloat16 const* src, float sf_scale, uint2* q, uint8_t* sf) +{ + uint4 const v0 = *reinterpret_cast(src); + uint4 const v1 = *reinterpret_cast(src + 8); + __nv_bfloat162 h[8]; + *reinterpret_cast(&h[0]) = v0; + *reinterpret_cast(&h[4]) = v1; + __nv_bfloat162 mx = __habs2(h[0]); +#pragma unroll + for (int i = 1; i < 8; ++i) + mx = __hmax2(mx, __habs2(h[i])); + float const vec_max = fmaxf(__low2float(mx), __high2float(mx)); + float sf_value = sf_scale * (vec_max * rcp_approx_ftz(6.0f)); + __nv_fp8_e4m3 const tmp = __nv_fp8_e4m3(sf_value); + sf_value = static_cast(tmp); + float const out_scale = vec_max != 0.f ? rcp_approx_ftz(sf_value * rcp_approx_ftz(sf_scale)) : 0.f; + float2 lo[4], hi[4]; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + lo[i] = __bfloat1622float2(h[i]); + lo[i].x *= out_scale; + lo[i].y *= out_scale; + hi[i] = __bfloat1622float2(h[i + 4]); + hi[i].x *= out_scale; + hi[i].y *= out_scale; + } + *q = make_uint2(e2m1x8(lo), e2m1x8(hi)); + *sf = tmp.__x; +} + +// grid: E / ROWS; block: 256 threads. smem: ROWS * K fp32 weights + M * K bf16 activations. +// With q_out, the CTAs also emit the NVFP4 activation ([M, K / 2] packed, [M, K / 16] E4M3 scales) for the MoE. +template +__global__ void __launch_bounds__(256) m3_router_gemv_kernel(__nv_bfloat16 const* __restrict__ x, int ldx, + float const* __restrict__ w, float* __restrict__ out, int M, int K, int E, uint8_t* __restrict__ q_out, + uint8_t* __restrict__ sf_out, float const* __restrict__ sf_scale) +{ +#if MINIMAX_M3_KERNELS_ENABLED + extern __shared__ __align__(128) uint8_t smem[]; + __shared__ __align__(8) uint64_t w_bar; + __shared__ __align__(8) uint64_t x_bar; + __shared__ float red[8][ROWS][kMaxM]; + float* w_s = reinterpret_cast(smem); + __nv_bfloat16* x_s = reinterpret_cast<__nv_bfloat16*>(smem + ROWS * K * sizeof(float)); + int const e0 = blockIdx.x * ROWS; + int const warp = threadIdx.x / 32; + int const lane = threadIdx.x % 32; + if (threadIdx.x == 0) + { + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;\n" ::"r"(smem_u32(&w_bar))); + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;\n" ::"r"(smem_u32(&x_bar))); + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(smem_u32(&w_bar)), + "r"(static_cast(ROWS * K * sizeof(float))) + : "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(smem_u32(&x_bar)), + "r"(static_cast(M * K * sizeof(__nv_bfloat16))) + : "memory"); + } + __syncthreads(); + // Router weights do not depend on the producer kernel. + if (warp == 0 && lane < ROWS) + { + bulk_g2s(w_s + lane * K, w + static_cast(e0 + lane) * K, K * sizeof(float), &w_bar); + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + if (warp == 1 && lane < M) + { + bulk_g2s(x_s + lane * K, x + static_cast(lane) * ldx, K * sizeof(__nv_bfloat16), &x_bar); + } + mbar_wait(&x_bar, 0); + if (q_out != nullptr) + { + float const s = *sf_scale; + int const kblocks = K / 16; + for (int b = blockIdx.x * blockDim.x + threadIdx.x; b < M * kblocks; b += gridDim.x * blockDim.x) + { + int const m = b / kblocks; + int const kb = b - m * kblocks; + quant_block16(x_s + m * K + kb * 16, s, + reinterpret_cast(q_out + static_cast(m) * (K / 2) + kb * 8), + sf_out + static_cast(m) * kblocks + kb); + } + } + mbar_wait(&w_bar, 0); + + float acc[ROWS][kMaxM]; +#pragma unroll + for (int r = 0; r < ROWS; ++r) + { +#pragma unroll + for (int m = 0; m < kMaxM; ++m) + { + acc[r][m] = 0.f; + } + } + // Each thread owns 4 consecutive columns per 1024-column stride. + for (int k = threadIdx.x * 4; k < K; k += 256 * 4) + { + float4 wv[ROWS]; +#pragma unroll + for (int r = 0; r < ROWS; ++r) + { + wv[r] = *reinterpret_cast(w_s + r * K + k); + } +#pragma unroll + for (int m = 0; m < kMaxM; ++m) + { + if (m < M) + { + uint2 const raw = *reinterpret_cast(x_s + m * K + k); + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&raw); + float2 const a = __bfloat1622float2(h[0]); + float2 const b = __bfloat1622float2(h[1]); +#pragma unroll + for (int r = 0; r < ROWS; ++r) + { + acc[r][m] = fmaf(a.x, wv[r].x, acc[r][m]); + acc[r][m] = fmaf(a.y, wv[r].y, acc[r][m]); + acc[r][m] = fmaf(b.x, wv[r].z, acc[r][m]); + acc[r][m] = fmaf(b.y, wv[r].w, acc[r][m]); + } + } + } + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif +#pragma unroll + for (int r = 0; r < ROWS; ++r) + { +#pragma unroll + for (int m = 0; m < kMaxM; ++m) + { + if (m < M) + { + float v = acc[r][m]; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + v += __shfl_xor_sync(0xFFFFFFFFu, v, o); + } + if (lane == 0) + { + red[warp][r][m] = v; + } + } + } + } + __syncthreads(); + if (threadIdx.x < ROWS * M) + { + int const r = threadIdx.x / M; + int const m = threadIdx.x % M; + float s = 0.f; +#pragma unroll + for (int ww = 0; ww < 8; ++ww) + { + s += red[ww][r][m]; + } + out[static_cast(m) * E + e0 + r] = s; + } +#endif +} + +// ---------------- 9..48 tokens: TF32 tensor-core router (same precision as the native TF32 GEMM) ---------------- +// grid: 8 expert tiles (16 experts) x CL CTAs, one CL-CTA cluster per tile, 256 threads. CTA r of a cluster computes +// the [16 experts x M] partials of its 16 / CL K slices (16 slices of 384), each over the same 8-warp split. CTA r +// owns 16 / CL experts of the tile: every CTA st.asyncs its partials for those experts into the owner's mailbox, and +// the owner sums the 16 slices in slice order, so the logits do not depend on CL. Optionally each CTA also emits the +// NVFP4 activation for its K range and the token rows m with m % 8 == expert tile. +constexpr int kTcExperts = 16; +constexpr int kTcSlices = 16; +constexpr int kTcMaxM = 48; + +__device__ __forceinline__ uint32_t f2tf32(float f) +{ + uint32_t r; + asm("cvt.rna.tf32.f32 %0, %1;\n" : "=r"(r) : "f"(f)); + return r; +} + +__device__ __forceinline__ uint32_t mapa_u32(void const* p, uint32_t rank) +{ + uint32_t r; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;\n" : "=r"(r) : "r"(smem_u32(p)), "r"(rank)); + return r; +} + +template +__global__ void __launch_bounds__(256) m3_router_tc_kernel(__nv_bfloat16 const* __restrict__ x, int ldx, + float const* __restrict__ w, float* __restrict__ out, int M, int K, uint8_t* __restrict__ q_out, + uint8_t* __restrict__ sf_out, float const* __restrict__ sf_scale) +{ +#if MINIMAX_M3_KERNELS_ENABLED + extern __shared__ __align__(128) uint8_t smem[]; + __shared__ __align__(8) uint64_t w_bar; + __shared__ __align__(8) uint64_t x_bar; + __shared__ __align__(8) uint64_t mb_bar; + constexpr int SPC = kTcSlices / CL; // K slices per CTA + constexpr int EPC = kTcExperts / CL; // experts owned per CTA + __shared__ float red[8][kTcExperts][kTcMaxM]; + __shared__ float part[SPC][kTcExperts][kTcMaxM]; // this CTA's slice partials + __shared__ float mailbox[CL][SPC][EPC][kTcMaxM]; // [source CTA][slice][owned expert][m] + int const tile = blockIdx.x / CL; + int const crank = blockIdx.x % CL; + int const ks = K / kTcSlices; // 384 + int const kc = SPC * ks; + int const k0 = crank * kc; + int const e0 = tile * kTcExperts; + int const wst = kc + 4; // padded row strides (bank-conflict-free fragment loads) + int const xst = kc + 8; + float* w_s = reinterpret_cast(smem); // [16][wst] + __nv_bfloat16* x_s = reinterpret_cast<__nv_bfloat16*>(smem + kTcExperts * wst * 4); // [M][xst] + int const warp = threadIdx.x / 32; + int const lane = threadIdx.x % 32; + if (threadIdx.x == 0) + { + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;\n" ::"r"(smem_u32(&w_bar))); + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;\n" ::"r"(smem_u32(&x_bar))); + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;\n" ::"r"(smem_u32(&mb_bar))); + asm volatile("fence.mbarrier_init.release.cluster;\n" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(smem_u32(&w_bar)), + "r"(static_cast(kTcExperts * kc * 4)) + : "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(smem_u32(&x_bar)), + "r"(static_cast(M * kc * 2)) + : "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;\n" ::"r"(smem_u32(&mb_bar)), + "r"(static_cast((CL - 1) * SPC * EPC * M * 4)) + : "memory"); + } + __syncthreads(); + // The peers' mailbox barriers must be initialized before this CTA's st.async lands there. + asm volatile("barrier.cluster.arrive.release.aligned;\n" ::: "memory"); + if (warp == 0 && lane < kTcExperts) + bulk_g2s(w_s + lane * wst, w + static_cast(e0 + lane) * K + k0, kc * 4, &w_bar); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + if (warp == 1) + { + for (int m = lane; m < M; m += 32) + bulk_g2s(x_s + m * xst, x + static_cast(m) * ldx + k0, kc * 2, &x_bar); + } + mbar_wait(&x_bar, 0); + if (q_out != nullptr) + { + float const sc = *sf_scale; + int const kb_cta = kc / 16; + for (int b = threadIdx.x; b < M * kb_cta; b += blockDim.x) + { + int const m = b / kb_cta; + if (m % 8 != tile) + continue; // 8 expert tiles share the row quantization + int const kb = b - m * kb_cta; + int const kbg = k0 / 16 + kb; + quant_block16(x_s + m * xst + kb * 16, sc, + reinterpret_cast(q_out + static_cast(m) * (K / 2) + kbg * 8), + sf_out + static_cast(m) * (K / 16) + kbg); + } + } + mbar_wait(&w_bar, 0); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + // Per slice h: warp w owns k8 steps [w * steps, (w + 1) * steps) of the slice; all token tiles (n = 8 each). + int const g = lane >> 2; + int const t = lane & 3; + int const ntiles = (M + 7) / 8; + int const steps = ks / 8 / 8; +#pragma unroll 1 + for (int h = 0; h < SPC; ++h) + { + float acc[6][4]; +#pragma unroll + for (int i = 0; i < 6; ++i) + acc[i][0] = acc[i][1] = acc[i][2] = acc[i][3] = 0.f; + for (int st = warp * steps; st < (warp + 1) * steps; ++st) + { + int const kk = h * ks + st * 8; + uint32_t const a0 = f2tf32(w_s[g * wst + kk + t]); + uint32_t const a1 = f2tf32(w_s[(g + 8) * wst + kk + t]); + uint32_t const a2 = f2tf32(w_s[g * wst + kk + t + 4]); + uint32_t const a3 = f2tf32(w_s[(g + 8) * wst + kk + t + 4]); +#pragma unroll + for (int nt = 0; nt < 6; ++nt) + { + if (nt < ntiles) + { + int const m = nt * 8 + g; + float bx0 = 0.f, bx1 = 0.f; + if (m < M) + { + bx0 = __bfloat162float(x_s[m * xst + kk + t]); + bx1 = __bfloat162float(x_s[m * xst + kk + t + 4]); + } + uint32_t const b0 = f2tf32(bx0), b1 = f2tf32(bx1); + asm volatile( + "mma.sync.aligned.m16n8k8.row.col.f32.tf32.tf32.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, " + "{%8,%9}, {%0,%1,%2,%3};\n" + : "+f"(acc[nt][0]), "+f"(acc[nt][1]), "+f"(acc[nt][2]), "+f"(acc[nt][3]) + : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(b0), "r"(b1)); + } + } + } +#pragma unroll + for (int nt = 0; nt < 6; ++nt) + { + if (nt < ntiles) + { + int const c = nt * 8 + 2 * t; + red[warp][g][c] = acc[nt][0]; + red[warp][g][c + 1] = acc[nt][1]; + red[warp][g + 8][c] = acc[nt][2]; + red[warp][g + 8][c + 1] = acc[nt][3]; + } + } + __syncthreads(); + for (int i = threadIdx.x; i < kTcExperts * M; i += blockDim.x) + { + int const e = i / M, m = i % M; + float v = 0.f; +#pragma unroll + for (int ww = 0; ww < 8; ++ww) + v += red[ww][e][m]; + part[h][e][m] = v; + } + __syncthreads(); + } + asm volatile("barrier.cluster.wait.acquire.aligned;\n" ::: "memory"); + // Expert e of the tile belongs to CTA e / EPC: its slice partials go to that CTA's mailbox[crank]. + for (int i = threadIdx.x; i < SPC * kTcExperts * M; i += blockDim.x) + { + int const h = i / (kTcExperts * M); + int const r = i - h * kTcExperts * M; + int const e = r / M, m = r % M; + int const owner = e / EPC; + float const v = part[h][e][m]; + float* dst = &mailbox[crank][h][e % EPC][m]; + if (owner == crank) + { + *dst = v; + } + else + { + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];\n" ::"r"( + mapa_u32(dst, owner)), + "r"(__float_as_uint(v)), "r"(mapa_u32(&mb_bar, owner)) + : "memory"); + } + } + __syncthreads(); + { + uint32_t done = 0; + while (!done) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}\n" + : "=r"(done) + : "r"(smem_u32(&mb_bar)) + : "memory"); + } + } + for (int i = threadIdx.x; i < EPC * M; i += blockDim.x) + { + int const le = i / M, m = i % M; + float v = 0.f; +#pragma unroll + for (int src = 0; src < CL; ++src) + { +#pragma unroll + for (int h = 0; h < SPC; ++h) + v += mailbox[src][h][le][m]; + } + out[static_cast(m) * (kTcExperts * 8) + e0 + EPC * crank + le] = v; + } +#endif +} + +template +void launchGemv(__nv_bfloat16 const* x, int ldx, int M, int K, float const* w, int E, float* out, uint8_t* q_out, + uint8_t* sf_out, float const* sf_scale, bool pdl, cudaStream_t stream) +{ + size_t const smem = static_cast(ROWS) * K * sizeof(float) + static_cast(M) * K * 2; + auto kernel = m3_router_gemv_kernel; + static bool const configured = [&]() + { + int device = 0, max_optin = 0; + cudaGetDevice(&device); + cudaDeviceGetAttribute(&max_optin, cudaDevAttrMaxSharedMemoryPerBlockOptin, device); + cudaFuncAttributes attrs = {}; + if (cudaFuncGetAttributes(&attrs, kernel) != cudaSuccess) + { + return false; + } + return cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, + max_optin - static_cast(attrs.sharedSizeBytes)) + == cudaSuccess; + }(); + TLLM_CHECK_WITH_INFO(configured, "failed to configure dynamic shared memory"); + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(E / ROWS); + cfg.blockDim = dim3(256); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + cudaError_t const err = cudaLaunchKernelEx(&cfg, kernel, x, ldx, w, out, M, K, E, q_out, sf_out, sf_scale); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +} // namespace + +void invokeMinimaxM3RouterGemv(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int32_t k, float const* w, int32_t e, + float* out, uint8_t* qOut, uint8_t* sfOut, float const* sfScale, bool pdl, cudaStream_t stream) +{ + TLLM_CHECK_WITH_INFO(m >= 1 && m <= kMaxM && k % 1024 == 0, "unsupported router GEMV shape M=%d K=%d", m, k); + launchGemv<1>(x, ldx, m, k, w, e, out, qOut, sfOut, sfScale, pdl, stream); +} + +void invokeMinimaxM3RouterTc(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int32_t k, float const* w, float* out, + uint8_t* qOut, uint8_t* sfOut, float const* sfScale, bool pdl, cudaStream_t stream) +{ + int64_t const M = m, K = k; + TLLM_CHECK_WITH_INFO( + M >= 1 && M <= kTcMaxM && K % (kTcSlices * 64) == 0, "unsupported router shape M=%d K=%d", m, k); + int const ks = K / kTcSlices; + auto launch_tc = [&](auto kernel, int cl) -> cudaError_t + { + int const kc = (kTcSlices / cl) * ks; + size_t const smem = static_cast(kTcExperts) * (kc + 4) * 4 + static_cast(M) * (kc + 8) * 2; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(8 * cl); + cfg.blockDim = dim3(256); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + attr[1].id = cudaLaunchAttributeClusterDimension; + attr[1].val.clusterDim.x = cl; + attr[1].val.clusterDim.y = 1; + attr[1].val.clusterDim.z = 1; + cfg.attrs = attr; + cfg.numAttrs = 2; + return cudaLaunchKernelEx( + &cfg, kernel, x, ldx, w, out, static_cast(M), static_cast(K), qOut, sfOut, sfScale); + }; + // 16-CTA clusters (128 CTAs, one K slice each) when the device co-schedules all 8, else 8-CTA clusters. + static int const cluster = [&]() + { + int device = 0, max_optin = 0; + cudaGetDevice(&device); + cudaDeviceGetAttribute(&max_optin, cudaDevAttrMaxSharedMemoryPerBlockOptin, device); + auto configure = [&](auto kernel, bool non_portable) + { + cudaFuncAttributes attrs = {}; + if (cudaFuncGetAttributes(&attrs, kernel) != cudaSuccess) + return false; + if (non_portable + && cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1) != cudaSuccess) + return false; + return cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, + max_optin - static_cast(attrs.sharedSizeBytes)) + == cudaSuccess; + }; + TLLM_CHECK_WITH_INFO(configure(m3_router_tc_kernel<8>, false), "failed to configure the router kernel"); + if (!configure(m3_router_tc_kernel<16>, true)) + { + cudaGetLastError(); + return 8; + } + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(8 * 16); + cfg.blockDim = dim3(256); + cfg.dynamicSmemBytes + = static_cast(kTcExperts) * (ks + 4) * 4 + static_cast(kTcMaxM) * (ks + 8) * 2; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeClusterDimension; + attr[0].val.clusterDim.x = 16; + attr[0].val.clusterDim.y = 1; + attr[0].val.clusterDim.z = 1; + cfg.attrs = attr; + cfg.numAttrs = 1; + int clusters = 0; + if (cudaOccupancyMaxActiveClusters(&clusters, m3_router_tc_kernel<16>, &cfg) != cudaSuccess) + { + cudaGetLastError(); + return 8; + } + return clusters >= 8 ? 16 : 8; + }(); + cudaError_t const err + = cluster == 16 ? launch_tc(m3_router_tc_kernel<16>, 16) : launch_tc(m3_router_tc_kernel<8>, 8); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.h b/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.h new file mode 100644 index 000000000000..6f4467883b24 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.h @@ -0,0 +1,46 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Low-M fp32 router GEMV: `out[M, E] = float(x[M, K]) @ w[E, K]^T` with exact fp32 FMA, one CTA per expert, +//! 1 <= M <= 16, K % 1024 == 0. With `qOut` the CTAs also emit the NVFP4 activation (packed e2m1 [M, K / 2], linear +//! E4M3 scales [M, K / 16]) for global scale `*sfScale`, bit-identical to `fp4_quantize(x, sfScale, 16, False, +//! False)`. +void invokeMinimaxM3RouterGemv(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int32_t k, float const* w, int32_t e, + float* out, uint8_t* qOut, uint8_t* sfOut, float const* sfScale, bool pdl, cudaStream_t stream); + +//! TF32 tensor-core router for 128 experts and 1 <= M <= 48 (K % 1024 == 0): `out[M, 128]` with the weights rounded +//! to TF32 (round to nearest, ties away) and fp32 accumulation, independent of the cluster size picked. Optional +//! NVFP4 activation as in `invokeMinimaxM3RouterGemv`. +void invokeMinimaxM3RouterTc(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int32_t k, float const* w, float* out, + uint8_t* qOut, uint8_t* sfOut, float const* sfScale, bool pdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.cu b/cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.cu new file mode 100644 index 000000000000..91b4c38d0fda --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.cu @@ -0,0 +1,525 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/sampledAccept.h" + +#include "tensorrt_llm/common/assert.h" + +#include +#include + +// Fused sampled verification for the MiniMax-M3 Eagle3 target head (TP-sharded vocabulary, decode rows <= 48). +// +// Replaces, per decode step, the full-vocabulary all-gather of the target logits, the fused sampling kernel and the +// strict-acceptance ops for rows sampled with temperature and top-p (the AgentX contract). Rows with top-k or min-p +// active must stay on the generic path; the caller decides. +// +// Sampling is top-p rejection sampling, distributed over the TP ranks. With w = exp(logit / T - max) over the FULL +// vocabulary and Z = sum(w), a token is in the nucleus iff the mass strictly above its weight is at most top_p * Z +// (the fused sampling kernel's kept set, ties included). A candidate is drawn proportionally to w and accepted iff it +// is in the nucleus; a rejected candidate lies below the nucleus, so the next candidate is drawn over the tokens +// heavier than it. The accepted token is distributed as w restricted to the nucleus. The first candidate lands in the +// nucleus with probability >= top_p, so most rows decide on their second exchange. +// +// Per verify row, 8 CTAs (rows <= 32) or 4 CTAs (more rows) on every rank each own a slice of the rank's logits shard, +// staged in shared memory once. A candidate is drawn in two stages without per-token noise: each CTA draws a local +// candidate by inverse CDF over its slice, and the CTA whose log-mass plus Gumbel noise is largest supplies the +// candidate (CTA j wins with probability T_j / sum T, its token with w / T_j). Uniforms come from a counter-based +// Philox keyed by the request's seed and offset, the row, the attempt and the CTA. Each round every CTA multicasts one +// 4-word record (key, candidate logit, and max and mass in round 0 or the mass above the current candidate later) and +// reads the records of all CTAs of the row on all ranks. Every CTA takes the same decision from the same records in the +// same order, so all ranks agree without any intra-GPU handshake. Words carry the call's tag: a slot needs no clearing +// and two call halves keep a rank one call ahead from overwriting a slot still being read. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +namespace sampledAccept +{ +namespace +{ +constexpr int kThreads = 256; +constexpr int kWarps = kThreads / 32; +constexpr int kMaxRec = kMaxWorld * kMaxSplits; +constexpr int kMaxAttempts = 16; // exchange slots per row, reused cyclically by the rounds (no round cap) +constexpr int kEx = 4; // words per record: key, logit of key, f2, f3 +constexpr int kTagBits = 64 - 32 - kIdBits; +constexpr float kTempEpsilon = 1e-6f; // the fused sampling kernel's 1/T guard + +// Multicast exchange, per call half: [kMaxAttempts][kMaxRows][world * kMaxSplits][kEx] u64. +__host__ __device__ inline long long half_words(int world) +{ + return static_cast(kMaxAttempts) * kMaxRows * world * kMaxSplits * kEx; +} + +__device__ __forceinline__ uint32_t ordered(float v) +{ + uint32_t const b = __float_as_uint(v); + return (b & 0x80000000u) ? ~b : (b | 0x80000000u); +} + +// Philox4x32-10. +__device__ __forceinline__ uint4 philox(uint4 c, uint2 k) +{ +#pragma unroll + for (int i = 0; i < 10; ++i) + { + uint32_t const hi0 = __umulhi(0xD2511F53u, c.x), lo0 = 0xD2511F53u * c.x; + uint32_t const hi1 = __umulhi(0xCD9E8D57u, c.z), lo1 = 0xCD9E8D57u * c.z; + c = make_uint4(hi1 ^ c.y ^ k.x, lo1, hi0 ^ c.w ^ k.y, lo0); + k.x += 0x9E3779B9u; + k.y += 0xBB67AE85u; + } + return c; +} + +// u in (0, 1): (k + 1/2) / 2^23, exact in fp32. +__device__ __forceinline__ float uniform(uint32_t bits) +{ + return (static_cast(bits >> 9) + 0.5f) * (1.0f / 8388608.0f); +} + +// Key on the wire: perturbed log-mass, then the candidate token id; 0 = no candidate. Tag bits are added on the wire. +__device__ __forceinline__ unsigned long long gkey(float lg, int id) +{ + return (static_cast(ordered(lg)) << kIdBits) + | static_cast((1u << kIdBits) - 1u - static_cast(id)); +} + +__device__ __forceinline__ int key_token(unsigned long long k) +{ + return static_cast((1u << kIdBits) - 1u - static_cast(k & ((1u << kIdBits) - 1u))); +} + +__device__ __forceinline__ void st_sys(unsigned long long* p, unsigned long long v) +{ + asm volatile("st.global.relaxed.sys.u64 [%0], %1;" ::"l"(p), "l"(v) : "memory"); +} + +__device__ __forceinline__ unsigned long long ld_sys(unsigned long long const* p) +{ + unsigned long long v; + asm volatile("ld.global.relaxed.sys.u64 %0, [%1];" : "=l"(v) : "l"(p) : "memory"); + return v; +} + +template +struct Shared +{ + __nv_bfloat16 row[kMaxVs / kSplits]; + float redf[kWarps]; + unsigned long long redk[kWarps]; + unsigned long long rk[kMaxRec]; + float rv[kMaxRec], r2[kMaxRec], r3[kMaxRec]; + float wsum[kWarps]; + int owner; + int pick; + float pickx; + bool last; +}; + +template +__device__ __forceinline__ float block_max(float v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, o)); + if ((threadIdx.x & 31) == 0) + s.redf[threadIdx.x >> 5] = v; + __syncthreads(); + float r = s.redf[0]; + for (int i = 1; i < kWarps; ++i) + r = fmaxf(r, s.redf[i]); + __syncthreads(); + return r; +} + +// Fixed-order block sum: identical result for identical inputs. +template +__device__ __forceinline__ float block_sum(float v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + if ((threadIdx.x & 31) == 0) + s.redf[threadIdx.x >> 5] = v; + __syncthreads(); + float r = 0.f; + for (int i = 0; i < kWarps; ++i) + r += s.redf[i]; + __syncthreads(); + return r; +} + +template +__device__ __forceinline__ unsigned long long block_max_u64(unsigned long long v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + unsigned long long const w = __shfl_xor_sync(0xffffffffu, v, o); + v = w > v ? w : v; + } + if ((threadIdx.x & 31) == 0) + s.redk[threadIdx.x >> 5] = v; + __syncthreads(); + unsigned long long r = s.redk[0]; + for (int i = 1; i < kWarps; ++i) + r = s.redk[i] > r ? s.redk[i] : r; + __syncthreads(); + return r; +} + +// Exclusive prefix of v over the threads in thread order, and the total. +template +__device__ __forceinline__ float block_excl_scan(float v, float& total, S& s) +{ + int const lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + float inc = v; +#pragma unroll + for (int o = 1; o < 32; o <<= 1) + { + float const t = __shfl_up_sync(0xffffffffu, inc, o); + if (lane >= o) + inc += t; + } + if (lane == 31) + s.wsum[warp] = inc; + __syncthreads(); + float base = 0.f; + total = 0.f; + for (int i = 0; i < kWarps; ++i) + { + base += i < warp ? s.wsum[i] : 0.f; + total += s.wsum[i]; + } + __syncthreads(); + return base + inc - v; +} + +// Every CTA of the grid must be co-resident: a row's CTAs wait for each other's records. +template +__global__ void __launch_bounds__(kThreads, 2) sampled_accept_kernel(Args const a) +{ + extern __shared__ __align__(16) unsigned char smem[]; + Shared& s = *reinterpret_cast*>(smem); + float constexpr kNegInf = -__builtin_huge_valf(); + int const tid = threadIdx.x; + int const row = blockIdx.x / kSplits; + int const split = blockIdx.x % kSplits; + int const world = a.world; + int const nrec = world * kSplits; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + unsigned int const call = *reinterpret_cast(a.counters + 1); + // Wire tags: never 0 (fresh memory), different between calls of the same half and between consecutive laps of + // the slot ring (round r uses slot r % kMaxAttempts and lap parity (r / kMaxAttempts) & 1). A rank decides round + // r only after reading every record of round r, so no rank is more than one round ahead and a slot is rewritten + // only after every reader has consumed it; the lap bit tells the new record from the previous lap's. + auto ktag_of = [&](int r) -> unsigned long long + { + unsigned int const lap = static_cast(r / kMaxAttempts) & 1u; + return (((call >> 1) << 1) | lap) % ((1u << kTagBits) - 1u) + 1u; + }; + auto ftag_of = [&](int r) -> unsigned long long + { + unsigned int const lap = static_cast(r / kMaxAttempts) & 1u; + return static_cast(((((call >> 1) << 1) | lap) & 0x7FFFFFFFu) | 0x80000000u) << 32; + }; + long long const rec_stride = static_cast(world) * kMaxSplits * kEx; + unsigned long long* const xu + = a.ws_uc + static_cast(call & 1u) * half_words(world) + static_cast(row) * rec_stride; + unsigned long long* const xm = a.ws_mc + static_cast(call & 1u) * half_words(world) + + static_cast(row) * rec_stride + (a.rank * kMaxSplits + split) * kEx; + long long const round_stride = static_cast(kMaxRows) * rec_stride; + + float const tempInv = 1.0f / (a.temperatures[row] + kTempEpsilon); + float const topP = a.top_ps[row]; + unsigned long long const seed = static_cast(a.seeds[row]); + unsigned long long const offset = static_cast(a.offsets[row]); + + // ---- stage this CTA's slice and its max ------------------------------------------------------------------------- + __nv_bfloat16 const* src = a.logits + row * a.ld; + int const sl_len = ((a.Vs + kSplits - 1) / kSplits + 7) / 8 * 8; + int const s0 = min(split * sl_len, a.Vs); + int const n = min(s0 + sl_len, a.Vs) - s0; + int const id0 = a.vocab_offset + s0; + float tmax = kNegInf; + bool const vec = (a.ld % 8) == 0 && (reinterpret_cast(a.logits) & 15) == 0; + int const nvec = vec ? n / 8 : 0; + for (int v = tid; v < nvec; v += kThreads) + { + uint4 const w = *reinterpret_cast(src + s0 + 8 * v); + *reinterpret_cast(&s.row[8 * v]) = w; + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&w); +#pragma unroll + for (int j = 0; j < 8; ++j) + tmax = fmaxf(tmax, __fmul_rn(__bfloat162float(e[j]), tempInv)); + } + for (int i = nvec * 8 + tid; i < n; i += kThreads) + { + s.row[i] = src[s0 + i]; + tmax = fmaxf(tmax, __fmul_rn(__bfloat162float(src[s0 + i]), tempInv)); + } + float const smax = block_max(tmax, s); // the staged slice is visible to all threads after this barrier + float M = 0.f, wc = 0.f, target = 0.f, cv = 0.f; // global max, candidate weight and logit, top_p * Z + unsigned long long ckey = 0ULL; + int const gid = a.rank * kMaxSplits + split; + for (int r = 0;; ++r) + { + // ---- local candidate by inverse CDF over the slice's tokens heavier than the candidate (w > 0 in round 0) + float const ref = r == 0 ? smax : M; + float const wmin = r == 0 ? 0.f : wc; + float ts = 0.f; + for (int i = tid; i < n; i += kThreads) + { + float const w = __expf(__fmul_rn(__bfloat162float(s.row[i]), tempInv) - ref); + ts += w > wmin ? w : 0.f; + } + float total; + float const excl = block_excl_scan(ts, total, s); + uint4 const rnd = philox(make_uint4(static_cast(gid), static_cast(row + 64 * r), + static_cast(offset), static_cast(offset >> 32)), + make_uint2(static_cast(seed), static_cast(seed >> 32))); + float const tgt = uniform(rnd.x) * total; + if (tid == 0) + s.owner = -1; + __syncthreads(); + if (ts > 0.f && excl <= tgt) + atomicMax(&s.owner, tid); + __syncthreads(); + if (tid == s.owner) + { + float acc = excl; + int pick = -1; + float px = 0.f; + for (int i = tid; i < n; i += kThreads) + { + float const x = __fmul_rn(__bfloat162float(s.row[i]), tempInv); + float const w = __expf(x - ref); + if (w > wmin) + { + acc += w; + pick = i; + px = x; + if (acc > tgt) + break; + } + } + s.pick = pick; + s.pickx = px; + } + __syncthreads(); + unsigned long long sb = 0ULL; + if (total > 0.f && s.owner >= 0) + sb = gkey(__logf(total) + ref - logf(-logf(uniform(rnd.y))), id0 + s.pick); + float const smab = total; + unsigned long long const ktag = ktag_of(r); + unsigned long long const ftag = ftag_of(r); + unsigned long long* const xr = xu + (r % kMaxAttempts) * round_stride; + if (tid == 0) + { + unsigned long long* const xw = xm + (r % kMaxAttempts) * round_stride; + st_sys(xw + 0, (sb << kTagBits) | ktag); + st_sys(xw + 1, ftag | __float_as_uint(sb != 0ULL ? s.pickx : 0.f)); + st_sys(xw + 2, ftag | __float_as_uint(r == 0 ? smax : smab)); + st_sys(xw + 3, ftag | __float_as_uint(r == 0 ? smab : 0.f)); + } + // ---- every record of the row, every rank --------------------------------------------------------------- + if (tid < nrec * kEx) + { + int const q = tid / kEx, f = tid % kEx; + unsigned long long v; + if (f == 0) + { + do + v = ld_sys(xr + (q / kSplits * kMaxSplits + q % kSplits) * kEx); + while ((v & ((1ULL << kTagBits) - 1ULL)) != ktag); + s.rk[q] = v >> kTagBits; + } + else + { + do + v = ld_sys(xr + (q / kSplits * kMaxSplits + q % kSplits) * kEx + f); + while ((v & 0xFFFFFFFF00000000ULL) != ftag); + float const fv = __uint_as_float(static_cast(v)); + (f == 1 ? s.rv : f == 2 ? s.r2 : s.r3)[q] = fv; + } + } + __syncthreads(); + // ---- the decision, identical in every CTA of the row on every rank: fixed record order ----------------- + unsigned long long gk = 0ULL; + float gv = 0.f; + for (int q = 0; q < nrec; ++q) + { + if (s.rk[q] > gk) + { + gk = s.rk[q]; + gv = s.rv[q]; + } + } + bool accept; + if (r == 0) + { + M = kNegInf; + for (int q = 0; q < nrec; ++q) + M = fmaxf(M, s.r2[q]); + float Z = 0.f; + for (int q = 0; q < nrec; ++q) + Z += s.r2[q] == kNegInf ? 0.f : s.r3[q] * __expf(s.r2[q] - M); + target = topP * Z; + accept = !(topP < 1.0f); // no nucleus cut: the first draw stands + ckey = gk; + cv = gv; + } + else + { + float S = 0.f; // mass strictly above the current candidate + for (int q = 0; q < nrec; ++q) + S += s.r2[q]; + accept = S <= target || gk == 0ULL; + if (!accept) + { + ckey = gk; + cv = gv; + } + } + wc = __expf(cv - M); + __syncthreads(); // records consumed before the next round's writes to s + if (accept) + { + if (split != 0) + return; + if (tid == 0) + { + a.tokens[row] = key_token(ckey); + if (a.extra_attempts != nullptr && r > 1) + atomicAdd(a.extra_attempts, r - 1); + } + break; + } + } + + // ---- split 0: the last row to finish runs the strict acceptance ---------------------------------------------- +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + __threadfence(); + __syncthreads(); + if (tid == 0) + s.last = atomicAdd(a.counters, 1u) == static_cast(a.rows) - 1; + __syncthreads(); + if (!s.last) + return; + __threadfence(); + if (tid == 0) + { + a.counters[0] = 0u; + a.counters[1] = call + 1u; + } + if (tid < a.nreq) + { + int const* tok = a.tokens; + int const b = tid; + int const k1 = a.draft_len + 1; + int nacc = 1; + bool run = true; + for (int k = 0; k < a.draft_len; ++k) + { + int const tt = *reinterpret_cast(tok + b * k1 + k); + a.acc_tokens[b * k1 + k] = tt; + run = run && a.draft[b * a.draft_len + k] == tt; + nacc += run ? 1 : 0; + } + a.acc_tokens[b * k1 + a.draft_len] = *reinterpret_cast(tok + b * k1 + a.draft_len); + if (a.force_vec != nullptr) + nacc = a.force_vec[b]; + else if (a.force_total > 0) + nacc = a.force_total; + a.num_acc[b] = nacc; + a.gather_ids[b] = static_cast(a.batch_idx[b]) * k1 + nacc - 1; + if (a.next_new != nullptr) + a.next_new[b * a.next_stride] + = *reinterpret_cast(tok + a.batch_idx[b] * k1 + nacc - 1); + } +} + +template +cudaError_t launch(Args const& a, bool use_pdl, cudaStream_t stream) +{ + static int max_ctas = 0; + if (max_ctas == 0) + { + int dev = 0, sms = 0, per_sm = 0; + cudaError_t err = cudaGetDevice(&dev); + if (err == cudaSuccess) + { + err = cudaDeviceGetAttribute(&sms, cudaDevAttrMultiProcessorCount, dev); + } + if (err == cudaSuccess) + { + err = cudaOccupancyMaxActiveBlocksPerMultiprocessor( + &per_sm, sampled_accept_kernel, kThreads, sizeof(Shared)); + } + if (err != cudaSuccess) + { + return err; + } + max_ctas = per_sm * sms; + } + TLLM_CHECK_WITH_INFO(a.rows * kSplits <= max_ctas, "m3_sampled_accept: %d CTAs cannot all be co-resident (max %d)", + a.rows * kSplits, max_ctas); + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(a.rows * kSplits); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = sizeof(Shared); + cfg.stream = stream; + cudaLaunchAttribute attr; + attr.id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr.val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = &attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, sampled_accept_kernel, a); +} + +} // namespace + +long long sampledAcceptWsWords(int world) +{ + return 2 * half_words(world); +} + +cudaError_t invokeSampledAccept(Args const& a, bool usePdl, cudaStream_t stream) +{ + // TP4 caps 2 / 4 / 8 (8 / 16 / 32 rows) and TP8 cap 1 (4 rows): 8 CTAs per row, at most 256 CTAs (2 per SM); + // more rows: 4 CTAs per row. + if (a.rows <= 32) + { + return launch<8>(a, usePdl, stream); + } + return launch<4>(a, usePdl, stream); +} + +} // namespace sampledAccept +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.h b/cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.h new file mode 100644 index 000000000000..1dc211a6f8a8 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.h @@ -0,0 +1,79 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +//! Fused sampled (temperature + top-p) verification of the MiniMax-M3 Eagle3 target head over the TP-sharded +//! vocabulary, decode rows <= kMaxRows; see sampledAccept.cu. +namespace sampledAccept +{ + +constexpr int kMaxSplits = 8; // CTAs per verify row (at most) +constexpr int kMaxRows = 48; // verify rows per call +constexpr int kMaxWorld = 8; // TP ranks +constexpr int kMaxVs = 51200; // vocab shard width +constexpr int kIdBits = 18; // token ids < 2^18 + +struct Args +{ + __nv_bfloat16 const* logits; // [rows, Vs] this rank's shard + long long ld; + int rows, Vs, vocab_offset; + float const* temperatures; + float const* top_ps; + long long const* seeds; + long long const* offsets; + unsigned long long* ws_uc; + unsigned long long* ws_mc; + unsigned int* counters; // [0] finished rows, [1] call + int* tokens; // [rows] sampled token per row + int* extra_attempts; // optional [1]: rounds beyond the second, summed over rows + int rank, world; + int const* draft; + int const* batch_idx; + int* acc_tokens; + int* num_acc; + long long* gather_ids; + int draft_len, nreq, force_total; + int* next_new; + int next_stride; + int const* force_vec; +}; + +//! Words (uint64) of the multicast exchange workspace for `world` ranks: two call halves. +long long sampledAcceptWsWords(int world); + +//! Launch the sampled verification over a.rows * (8 or 4) co-resident CTAs (8 per row up to 32 rows). +cudaError_t invokeSampledAccept(Args const& a, bool usePdl, cudaStream_t stream); + +} // namespace sampledAccept +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/selectLong.cu b/cpp/tensorrt_llm/kernels/minimaxM3/selectLong.cu new file mode 100644 index 000000000000..39351f0ad3ce --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/selectLong.cu @@ -0,0 +1,235 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Top-16 block select for MiniMax-M3 decode rows of 129 to 9216 blocks. +// +// One 1024-thread CTA per (query, KV head) row holds the row's ordered score keys in registers (2, 4 or 9 per +// thread, by the row's valid count). The weakest of the 16 maxima of the interleaved groups `block % 16` is a lower +// bound on the 16th-largest key, so the top 16 is among the keys at or above it; those are compacted to shared +// memory and ranked exactly by (key, block). More than 1024 keys tied at the bound (degenerate rows) rank every +// candidate against the whole row (re-read). + +#include "tensorrt_llm/kernels/minimaxM3/selectLong.h" + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr float kInitScore = 1.0e30f; +constexpr float kLocalScore = 1.0e29f; +constexpr uint32_t kNegInfKey = 0xff800000u; +constexpr int kItems = 9; // keys per thread: rows up to 9216 blocks (1M tokens + draft tokens) + +// Descending order key: a smaller key is a larger score (NaN above +inf), as in the reference op. +__device__ __forceinline__ uint32_t orderedKey(float x) +{ + uint32_t const b = __float_as_uint(x); + return (b & 0x80000000u) ? b : (~b & 0x7fffffffu); +} + +__device__ __forceinline__ float applyForcing(float raw, int block, int initBlocks, long long localStart) +{ + if (block >= localStart) + return kLocalScore; + if (block < initBlocks) + return kInitScore; + return raw; +} + +struct SelectSmem +{ + uint32_t groupMin[16]; + unsigned long long cand[1024]; + int numCand; + unsigned long long picked[16]; +}; + +// One row with up to kItems * 1024 valid blocks; the caller picks kItems from the row's valid count. +template +__device__ __forceinline__ void selectRow( + SelectSmem& sm, float const* rowBase, long long blockStride, int valid, int initBlocks, long long localStart) +{ + int const tid = threadIdx.x, lane = tid & 31; + uint32_t key[kItems]; + uint32_t localMin = 0xffffffffu; +#pragma unroll + for (int s = 0; s < kItems; ++s) + { + int const block = s * 1024 + tid; + key[s] = block < valid ? orderedKey(applyForcing(rowBase[block * blockStride], block, initBlocks, localStart)) + : 0xffffffffu; + localMin = min(localMin, key[s]); + } + if (valid <= 16) + { + if (tid < valid) + sm.picked[tid] = (static_cast(key[0]) << 32) | static_cast(tid); + return; + } + // Group g = block % 16 = tid % 16: lanes l and l ^ 16 of every warp. + localMin = min(localMin, __shfl_xor_sync(0xffffffffu, localMin, 16)); + if (lane < 16) + atomicMin(&sm.groupMin[lane], localMin); + __syncthreads(); + uint32_t bound = 0; +#pragma unroll + for (int g = 0; g < 16; ++g) + bound = max(bound, sm.groupMin[g]); +#pragma unroll + for (int s = 0; s < kItems; ++s) + { + int const block = s * 1024 + tid; + bool const isCand = block < valid && key[s] <= bound; + unsigned const ballot = __ballot_sync(0xffffffffu, isCand); + int base = 0; + if (lane == 0 && ballot) + base = atomicAdd(&sm.numCand, __popc(ballot)); + base = __shfl_sync(0xffffffffu, base, 0); + if (isCand) + { + int const i = base + __popc(ballot & ((1u << lane) - 1)); + if (i < 1024) + sm.cand[i] = (static_cast(key[s]) << 32) | static_cast(block); + } + } + __syncthreads(); + int const n = sm.numCand; + if (n <= 1024) + { + for (int i = tid; i < n; i += 1024) + { + unsigned long long const mine = sm.cand[i]; + int rank = 0; + for (int j = 0; j < n; ++j) + rank += sm.cand[j] < mine; + if (rank < 16) + sm.picked[rank] = mine; + } + return; + } + // More than 1024 keys tied at the bound (degenerate rows): rank every candidate against the whole row. +#pragma unroll + for (int s = 0; s < kItems; ++s) + { + int const block = s * 1024 + tid; + if (block < valid && key[s] <= bound) + { + int rank = 0; + for (int j = 0; j < valid; ++j) + { + uint32_t const other = orderedKey(applyForcing(rowBase[j * blockStride], j, initBlocks, localStart)); + rank += other < key[s] || (other == key[s] && j < block); + } + if (rank < 16) + sm.picked[rank] = (static_cast(key[s]) << 32) | static_cast(block); + } + } +} + +__global__ void __launch_bounds__(1024) + m3_select_long_kernel(float const* __restrict__ scores, long long headStride, long long blockStride, + long long queryStride, int const* __restrict__ nValid, int* __restrict__ out, long long outQueryStride, + long long outHeadStride, int numKvHeads, int numBlocks, int initBlocks, int localBlocks, bool earlyTrigger) +{ + __shared__ SelectSmem sm; + int const row = blockIdx.x, query = row / numKvHeads, head = row % numKvHeads; + int const tid = threadIdx.x, lane = tid & 31; + if (tid < 16) + { + sm.groupMin[tid] = 0xffffffffu; + sm.picked[tid] = ~0ull; + } + if (tid == 0) + sm.numCand = 0; +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + cudaGridDependencySynchronize(); + // The sparse decode that follows stages its table-independent inputs before its own grid wait; past this grid's + // wait those inputs are final, so it may launch now rather than at this grid's exit. + if (earlyTrigger) + cudaTriggerProgrammaticLaunchCompletion(); +#endif + int const raw = nValid[query]; + int const valid = max(0, min(raw, numBlocks)); + long long const localStart = max(static_cast(raw) - localBlocks, 0LL); + float const* rowBase = scores + head * headStride + query * queryStride; + __syncthreads(); + if (valid <= 2048) + selectRow<2>(sm, rowBase, blockStride, valid, initBlocks, localStart); + else if (valid <= 4096) + selectRow<4>(sm, rowBase, blockStride, valid, initBlocks, localStart); + else + selectRow(sm, rowBase, blockStride, valid, initBlocks, localStart); + __syncthreads(); + if (tid >= 32) + return; + uint32_t blockKey = ~0u; + if (lane < 16) + { + unsigned long long const c = sm.picked[lane]; + if (c != ~0ull && static_cast(c >> 32) != kNegInfKey) + blockKey = static_cast(c & 0xffffffffu); + } +#pragma unroll + for (int size = 2; size <= 32; size <<= 1) + { +#pragma unroll + for (int stride = size >> 1; stride > 0; stride >>= 1) + { + uint32_t const other = __shfl_xor_sync(0xffffffffu, blockKey, stride); + bool const ascending = (lane & size) == 0, lower = (lane & stride) == 0; + blockKey = (lower == ascending) ? min(blockKey, other) : max(blockKey, other); + } + } + if (lane < 16) + out[query * outQueryStride + head * outHeadStride + lane] = blockKey == ~0u ? -1 : static_cast(blockKey); +} + +} // namespace + +cudaError_t invokeMinimaxM3SelectLong(float const* scores, int64_t headStride, int64_t blockStride, int64_t queryStride, + int32_t const* nValidBlocks, int32_t* out, int64_t outQueryStride, int64_t outHeadStride, int32_t numKvHeads, + int32_t numBlocks, int32_t totalQ, int32_t initBlocks, int32_t localBlocks, bool usePdl, bool earlyTrigger, + cudaStream_t stream) +{ + if (numBlocks > kMinimaxM3SelectLongMaxBlocks) + { + return cudaErrorInvalidValue; + } + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(totalQ * numKvHeads); + cfg.blockDim = dim3(1024); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = usePdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, m3_select_long_kernel, scores, static_cast(headStride), + static_cast(blockStride), static_cast(queryStride), nValidBlocks, out, + static_cast(outQueryStride), static_cast(outHeadStride), numKvHeads, numBlocks, + initBlocks, localBlocks, earlyTrigger); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/selectLong.h b/cpp/tensorrt_llm/kernels/minimaxM3/selectLong.h new file mode 100644 index 000000000000..258954235317 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/selectLong.h @@ -0,0 +1,45 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Largest row (blocks per query) the long-row top-16 select covers: 9 keys per thread of one 1024-thread CTA. +constexpr int32_t kMinimaxM3SelectLongMaxBlocks = 9216; + +//! MiniMax-M3 top-16 block select over per-KV-head scores [numKvHeads, numBlocks, totalQ] (element strides given), +//! one 1024-thread CTA per (query, KV head). Same contract as minimax_m3_select_blocks: init / local forcing, the +//! per-query valid-block count, ties to the smaller block, forced -inf picks emitted as -1, ascending ids with -1 +//! tail padding. Writes out[query * outQueryStride + head * outHeadStride + 0..15]. With earlyTrigger the dependent +//! grid is released right after this grid's dependency wait instead of at its exit. +cudaError_t invokeMinimaxM3SelectLong(float const* scores, int64_t headStride, int64_t blockStride, int64_t queryStride, + int32_t const* nValidBlocks, int32_t* out, int64_t outQueryStride, int64_t outHeadStride, int32_t numKvHeads, + int32_t numBlocks, int32_t totalQ, int32_t initBlocks, int32_t localBlocks, bool usePdl, bool earlyTrigger, + cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/shardAccept.cu b/cpp/tensorrt_llm/kernels/minimaxM3/shardAccept.cu new file mode 100644 index 000000000000..afc70e0e2c35 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/shardAccept.cu @@ -0,0 +1,204 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/shardAccept.h" + +#include +#include +#include + +// Fused greedy Eagle3 verification for the MiniMax-M3 target head (GB300, TP-sharded vocabulary, <= 48 verify rows). +// +// One kernel replaces the all-gather of the target logits, the argmax and the strict-acceptance ops: +// +// * Each logit of this rank's BF16 shard becomes a 64-bit key ordered(value) << 32 | ~global_index (ties pick the +// lowest index, as torch.argmax), max-reduced per row: warp shuffles, then atomicMax. +// * The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping +// (Lamport: 0 = empty, the key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the +// slots and runs the strict acceptance. Call n uses slot half n % 2. +// +// The keys order the BF16 values exactly, so the tokens are the argmax of the gathered logits. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +namespace shardAccept +{ +namespace +{ +__device__ __forceinline__ unsigned long long make_key(float v, uint32_t index) +{ + uint32_t u = __float_as_uint(v); + // -0.0 and +0.0 compare equal (torch.argmax takes the lower index among them): one key for both. + u = (u == 0x80000000u) ? 0u : u; + u = (u & 0x80000000u) ? ~u : (u | 0x80000000u); + return (static_cast(u) << 32) | static_cast(0xFFFFFFFFu - index); +} + +__device__ __forceinline__ unsigned long long shfl_max(unsigned long long k, int o) +{ + unsigned long long const ok = __shfl_xor_sync(0xffffffffu, k, o); + return ok > k ? ok : k; +} + +// Greedy verification from this rank's logits shard (verify-row count <= kMaxRows): per-row argmax keys over the +// shard, the last CTA exchanges them across ranks (multicast slots [2][kMaxRows][world], Lamport 0 = empty) and runs +// the acceptance epilogue. +__global__ void __launch_bounds__(kThreads, 1) shard_accept_kernel(__nv_bfloat16 const* __restrict__ logits, int M, + int Vs, long long ld, int vocab_offset, unsigned long long* __restrict__ keys, unsigned int* __restrict__ counters, + unsigned long long* __restrict__ ws_uc, unsigned long long* __restrict__ ws_mc, int rank, int world, AcceptArgs acc) +{ + __shared__ bool s_last; + __shared__ int s_tok[kMaxRows]; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + int const warp = threadIdx.x >> 5; + int const lane = threadIdx.x & 31; + int const nwarps = blockDim.x >> 5; + int const groups = (Vs + 7) / 8; + int const g0 = static_cast((static_cast(blockIdx.x) * groups) / gridDim.x); + int const g1 = static_cast((static_cast(blockIdx.x + 1) * groups) / gridDim.x); + bool const vec = (ld % 8) == 0; + for (int r = warp; r < M; r += nwarps) + { + unsigned long long best = 0ull; + __nv_bfloat16 const* row = logits + r * ld; + for (int gi = g0 + lane; gi < g1; gi += 32) + { + int const c = gi * 8; + if (vec && c + 8 <= Vs) + { + uint4 const raw = *reinterpret_cast(row + c); + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&raw); +#pragma unroll + for (int q = 0; q < 8; ++q) + { + unsigned long long const k + = make_key(__bfloat162float(e[q]), static_cast(vocab_offset + c + q)); + best = k > best ? k : best; + } + } + else + { + for (int q = 0; q < 8 && c + q < Vs; ++q) + { + unsigned long long const k + = make_key(__bfloat162float(row[c + q]), static_cast(vocab_offset + c + q)); + best = k > best ? k : best; + } + } + } +#pragma unroll + for (int o = 1; o < 32; o <<= 1) + best = shfl_max(best, o); + if (lane == 0 && best != 0ull) + atomicMax(keys + r, best); + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + unsigned int const prev = atomicAdd(counters, 1u); + s_last = prev == gridDim.x - 1; + } + __syncthreads(); + if (!s_last) + return; + __threadfence(); + unsigned int const call = *reinterpret_cast(counters + 1); + int const half = static_cast(call & 1u); + for (int t = threadIdx.x; t < M; t += blockDim.x) + { + unsigned long long const mine = atomicExch(keys + t, 0ull); + unsigned long long* slot_mc = ws_mc + (half * kMaxRows + t) * world; + unsigned long long* slot_uc = ws_uc + (half * kMaxRows + t) * world; + slot_mc[rank] = mine; + unsigned long long best = 0ull; + for (int r = 0; r < world; ++r) + { + unsigned long long v; + do + { + v = *reinterpret_cast(slot_uc + r); + } while (v == 0ull); + best = v > best ? v : best; + slot_uc[r] = 0ull; + } + s_tok[t] = static_cast(0xFFFFFFFFu - static_cast(best & 0xFFFFFFFFull)); + } + __syncthreads(); + if (threadIdx.x == 0) + { + counters[0] = 0u; + counters[1] = call + 1u; + } + if (threadIdx.x < acc.nreq) + { + int const b = threadIdx.x; + int const k1 = acc.draft_len + 1; + int n = 1; + bool run = true; + for (int k = 0; k < acc.draft_len; ++k) + { + int const tt = s_tok[b * k1 + k]; + acc.acc_tokens[b * k1 + k] = tt; + run = run && acc.draft[b * acc.draft_len + k] == tt; + n += run ? 1 : 0; + } + acc.acc_tokens[b * k1 + acc.draft_len] = s_tok[b * k1 + acc.draft_len]; + if (acc.force_vec != nullptr) + n = acc.force_vec[b]; + else if (acc.force_total > 0) + n = acc.force_total; + acc.num_acc[b] = n; + acc.gather_ids[b] = static_cast(acc.batch_idx[b]) * k1 + n - 1; + if (acc.next_new != nullptr) + acc.next_new[b * acc.next_stride] = s_tok[acc.batch_idx[b] * k1 + n - 1]; + } +} + +} // namespace + +cudaError_t invokeShardAccept(__nv_bfloat16 const* logits, int M, int Vs, long long ld, int vocabOffset, + unsigned long long* keys, unsigned int* counters, unsigned long long* wsUc, unsigned long long* wsMc, int rank, + int world, AcceptArgs const& acc, int numCtas, cudaStream_t stream) +{ + cudaLaunchAttribute attr[1]; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(static_cast(numCtas)); + cfg.blockDim = dim3(kThreads); + cfg.stream = stream; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = 1; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx( + &cfg, shard_accept_kernel, logits, M, Vs, ld, vocabOffset, keys, counters, wsUc, wsMc, rank, world, acc); +} + +} // namespace shardAccept +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/shardAccept.h b/cpp/tensorrt_llm/kernels/minimaxM3/shardAccept.h new file mode 100644 index 000000000000..e41982b9d357 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/shardAccept.h @@ -0,0 +1,67 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +//! Fused greedy Eagle3 verification of the MiniMax-M3 NVFP4 TP4 decode path (sm_103a): the cross-rank argmax of the +//! target logits shard and the strict acceptance in one kernel. See shardAccept.cu. +namespace shardAccept +{ + +constexpr int kMaxRows = 48; // verify rows of one launch +constexpr int kThreads = 512; // threads per CTA + +// Greedy-acceptance epilogue run by the last CTA once the global per-row argmax is known (rows = nreq requests x +// (draft_len + 1) verify positions, request-major). Mirrors the strict path of SpecWorkerBase: accepted = the target +// tokens, count = 1 + length of the matching draft prefix, optionally forced; plus the first draft pass's row index. +struct AcceptArgs +{ + int const* draft; // [nreq, draft_len] previous draft tokens + int const* batch_idx; // [nreq] request row in the verify layout + int* acc_tokens; // [nreq, draft_len + 1] + int* num_acc; // [nreq] + long long* gather_ids; // [nreq] last accepted row + int draft_len; + int nreq; + int force_total; // > 0 overrides the accepted count + int* next_new; // optional [nreq, next_stride]: column 0 = last accepted token (next step's input) + int next_stride; + int const* force_vec; // optional [nreq] per-request accepted count; overrides force_total +}; + +//! Greedy verification from this rank's BF16 logits shard [M, Vs] (row stride ld), M <= kMaxRows, over numCtas +//! kThreads-thread CTAs, followed by the strict acceptance epilogue. +cudaError_t invokeShardAccept(__nv_bfloat16 const* logits, int M, int Vs, long long ld, int vocabOffset, + unsigned long long* keys, unsigned int* counters, unsigned long long* wsUc, unsigned long long* wsMc, int rank, + int world, AcceptArgs const& acc, int numCtas, cudaStream_t stream); + +} // namespace shardAccept +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.cu b/cpp/tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.cu new file mode 100644 index 000000000000..e01fa78927a5 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.cu @@ -0,0 +1,561 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3 long-context sparse decode attention in one launch. +// +// Grid: one cluster of 16 CTAs per query token; CTA `r` attends the token's `r`-th selected block (the Triton +// producer's chunk `r` at 16 chunks), keeps the partial in shared memory, and after a cluster barrier merges head +// `r` from all 16 partials over DSMEM, writing the BF16 row and, optionally, its MXFP8 copy and swizzled UE8M0 +// scales exactly as the Triton merge's `_store_mxfp8_head` lays them out. +// +// Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and +// PV on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +// log-domain recurrence. + +#include "tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kHeads = 16; +constexpr int kDim = 128; +constexpr int kPage = 128; +constexpr int kTopk = 16; +constexpr int kThreads = 128; +constexpr int kPitch = kDim + 8; // halves per smem row: 272 B, conflict-free ldmatrix +constexpr float kNegInf = -__builtin_huge_valf(); + +using Params = MinimaxM3SparseDecodeLcParams; + +struct Smem +{ + __half qs[kHeads * kPitch]; + __half vs[kPage * kPitch]; + __half ps[kHeads * kPitch]; + float recv[kTopk][kDim]; // head `rank` partials pushed by the 16 CTAs, chunk-major + float recv_lse[kTopk]; + float red[4][kHeads]; + float rowm[kHeads]; + float rowl[kHeads]; + alignas(8) unsigned long long rbar; // the split-K partials of head `rank` landed (complete_tx) +}; + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t e2m1x2_to_f16x2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 s;\n cvt.u8.u32 s, %1;\n cvt.rn.f16x2.e2m1x2 %0, s;\n}" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t e4m3x2_to_f16x2(uint16_t pair) +{ + uint32_t out; + asm("cvt.rn.f16x2.e4m3x2 %0, %1;" : "=r"(out) : "h"(pair)); + return out; +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ void ldsm_x4(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x4_t(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma(float (&c)[4], uint32_t const (&a)[4], uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};" + : "+f"(c[0]), "+f"(c[1]), "+f"(c[2]), "+f"(c[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} + +__device__ __forceinline__ uint32_t map_cluster(void const* local, uint32_t rank) +{ + uint32_t remote; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(remote) : "r"(smem_u32(local)), "r"(rank)); + return remote; +} + +// Posted DSMEM stores that complete_tx on the receiving CTA's mbarrier `bar`. +__device__ __forceinline__ void st_async_v2(uint32_t addr, float a, float b, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.v2.f32 [%0], {%1, %2}, [%3];" ::"r"(addr), + "f"(a), "f"(b), "r"(bar) + : "memory"); +} + +__device__ __forceinline__ void st_async_f32(uint32_t addr, float a, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(addr), + "r"(__float_as_uint(a)), "r"(bar) + : "memory"); +} + +// One packed NVFP4 row (64 bytes, 128 values) with its 8 E4M3 scales -> 128 fp16 values in smem. +__device__ __forceinline__ void dequant_row(__half* dst, uint8_t const* packed, uint8_t const (&sc)[8], bool valid) +{ + uint4 const* src = reinterpret_cast(packed); + uint32_t* d = reinterpret_cast(dst); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + uint32_t words[4] = {w.x, w.y, w.z, w.w}; + // 16 bytes = 32 values = scale columns 2c and 2c + 1. +#pragma unroll + for (int h = 0; h < 2; ++h) + { + uint8_t const s = valid ? sc[2 * c + h] : 0; + uint32_t const sh = e4m3x2_to_f16x2(static_cast(s | (s << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); +#pragma unroll + for (int j = 0; j < 2; ++j) + { + uint32_t const word = words[2 * h + j]; +#pragma unroll + for (int b = 0; b < 4; ++b) + { + uint32_t const v2 = e2m1x2_to_f16x2((word >> (8 * b)) & 0xFF); + __half2 const x = __hmul2(*reinterpret_cast<__half2 const*>(&v2), scale); + d[c * 16 + h * 8 + j * 4 + b] = *reinterpret_cast(&x); + } + } + } + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +template +__global__ void __launch_bounds__(kThreads) m3_lc_attn_kernel(__grid_constant__ Params const p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(128) uint8_t smem_raw[]; + Smem& s = *reinterpret_cast(smem_raw); + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const tok = blockIdx.x / kTopk; + uint32_t rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(rank)); + + if (tid == 0) + { + // One arrival (here) plus every cluster CTA's partial row (kDim floats) and LSE for head `rank`. + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(smem_u32(&s.rbar)) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(smem_u32(&s.rbar)), + "r"(kTopk * (kDim + 1) * 4) + : "memory"); + } + // Every peer must be running before its shared memory takes a DSMEM store: arrive now, wait before pushing. + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + + // Everything that does not depend on the top-16 table is read before the grid dependency: the request's length + // (written before the step's graph runs) and the query rows (written by the producer, which completed before the + // select ahead of this grid could release it: the select triggers its dependents only after its own grid wait). + int const req = tok / p.dql; + int const qoff = tok - req * p.dql; + int const seq = p.seq_lens[req]; + int const kv_len = max(seq - p.dql + qoff + 1, 0); + int const nblocks = (kv_len + kPage - 1) / kPage; + int const real_topk = min(kTopk, nblocks); + // q: 16 heads x 128 e4m3 -> fp16; thread = (head, 16-value chunk). + { + int const h = tid >> 3; + int const c = (tid & 7) * 16; + uint4 const w = *reinterpret_cast(p.q + tok * p.q_tok + h * p.q_head + c); + uint32_t const words[4] = {w.x, w.y, w.z, w.w}; + uint32_t* d = reinterpret_cast(s.qs + h * kPitch + c); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + d[2 * i] = e4m3x2_to_f16x2(static_cast(words[i] & 0xFFFF)); + d[2 * i + 1] = e4m3x2_to_f16x2(static_cast(words[i] >> 16)); + } + } + + asm volatile("griddepcontrol.wait;" ::: "memory"); + if constexpr (EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } + int blk = static_cast(rank) < real_topk ? p.topk[tok * p.t_tok + rank * p.t_k] : -1; + bool const active = blk >= 0; + int64_t const page_pw = !active ? 0 : p.block_table[req * p.bt_stride + blk]; + + float m_row[2] = {kNegInf, kNegInf}; + float acc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + acc[j][i] = 0.F; + float l_row[2] = {0.F, 0.F}; + + // K stays packed in registers: the MMA B fragment of token n = lane / 4 at k-step s is the f16x2 pair of + // packed bytes 8s + q and 8s + 4 + q (q = lane % 4), both under scale column s. + uint32_t kw[4][16]; + uint2 ksc[4]; + int64_t page = 0; + if (active) + { + page = page_pw; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const n = warp * 32 + j * 8 + (lane >> 2); + bool const valid = blk * kPage + n < kv_len; + uint4 const* src = reinterpret_cast(p.k + page * p.k_blk + n * 64); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 const w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + kw[j][4 * c] = w.x; + kw[j][4 * c + 1] = w.y; + kw[j][4 * c + 2] = w.z; + kw[j][4 * c + 3] = w.w; + } + ksc[j] = valid ? __ldg(reinterpret_cast(p.ksc + page * p.ksc_blk + n * 8)) : make_uint2(0, 0); + } + bool const valid = blk * kPage + tid < kv_len; + uint8_t vsc[8]; + if (valid) + { + uint8_t const* vb = p.vsc + page * p.vsc_blk + (tid >> 2) * 32 + (tid & 3); +#pragma unroll + for (int c = 0; c < 8; ++c) + vsc[c] = __ldg(vb + 4 * c); + } + dequant_row(s.vs + tid * kPitch, p.v + page * p.v_blk + tid * 64, vsc, valid); + } + __syncthreads(); + + if (active) + { + float const scale_log2 = p.sm_scale * 1.4426950409F * __ldg(p.kgs); + int const q4 = (lane & 3) * 8; + // S = q K^T for this warp's 32 tokens: rows g / g + 8, columns 2 * (lane % 4) + {0, 1} of each n-tile. + float sacc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + sacc[j][i] = 0.F; +#pragma unroll + for (int ks = 0; ks < kDim / 16; ++ks) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.qs + (lane & 15) * kPitch + ks * 16 + (lane >> 4) * 8)); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + uint32_t const scw = ks < 4 ? ksc[j].x : ksc[j].y; + uint8_t const sc = static_cast(scw >> (8 * (ks & 3))); + uint32_t const sh = e4m3x2_to_f16x2(static_cast(sc | (sc << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); + uint32_t const r0 = e2m1x2_to_f16x2((kw[j][2 * ks] >> q4) & 0xFF); + uint32_t const r1 = e2m1x2_to_f16x2((kw[j][2 * ks + 1] >> q4) & 0xFF); + __half2 const h0 = __hmul2(*reinterpret_cast<__half2 const*>(&r0), scale); + __half2 const h1 = __hmul2(*reinterpret_cast<__half2 const*>(&r1), scale); + mma(sacc[j], a, *reinterpret_cast(&h0), *reinterpret_cast(&h1)); + } + } + int const g = lane >> 2; + float mloc[2] = {kNegInf, kNegInf}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const t = warp * 32 + j * 8 + 2 * (lane & 3) + (i & 1); + float const v = blk * kPage + t < kv_len ? 0.F + sacc[j][i] * scale_log2 : kNegInf; + sacc[j][i] = v; + mloc[i >> 1] = fmaxf(mloc[i >> 1], v); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 1)); + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 2)); + } + if ((lane & 3) == 0) + { + s.red[warp][g] = mloc[0]; + s.red[warp][g + 8] = mloc[1]; + } + __syncthreads(); + if (tid < kHeads) + s.rowm[tid] = fmaxf(fmaxf(s.red[0][tid], s.red[1][tid]), fmaxf(s.red[2][tid], s.red[3][tid])); + __syncthreads(); + m_row[0] = s.rowm[g]; + m_row[1] = s.rowm[g + 8]; + float lloc[2] = {0.F, 0.F}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; i += 2) + { + float const p0 = ex2(sacc[j][i] - m_row[i >> 1]); + float const p1 = ex2(sacc[j][i + 1] - m_row[i >> 1]); + lloc[i >> 1] += p0 + p1; + int const row = g + (i >> 1) * 8; + int const t = warp * 32 + j * 8 + 2 * (lane & 3); + *reinterpret_cast<__half2*>(s.ps + row * kPitch + t) = __floats2half2_rn(p0, p1); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 1); + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 2); + } + __syncthreads(); // rowm reads done before red is reused; ps complete + if ((lane & 3) == 0) + { + s.red[warp][g] = lloc[0]; + s.red[warp][g + 8] = lloc[1]; + } + // O[:, warp's 32 dims] = P V over all 128 tokens. +#pragma unroll + for (int kk = 0; kk < kPage; kk += 16) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.ps + (lane & 15) * kPitch + kk + (lane >> 4) * 8)); +#pragma unroll + for (int jp = 0; jp < 2; ++jp) + { + uint32_t b[4]; + int const t = kk + (lane & 7) + ((lane >> 3) & 1) * 8; + int const d = warp * 32 + jp * 16 + (lane >> 4) * 8; + ldsm_x4_t(b, smem_u32(s.vs + t * kPitch + d)); + mma(acc[2 * jp], a, b[0], b[1]); + mma(acc[2 * jp + 1], a, b[2], b[3]); + } + } + __syncthreads(); + if (tid < kHeads) + s.rowl[tid] = (s.red[0][tid] + s.red[1][tid]) + (s.red[2][tid] + s.red[3][tid]); + __syncthreads(); + l_row[0] = s.rowl[g]; + l_row[1] = s.rowl[g + 8]; + } + + // Normalize the partial and push head h's row with its LSE (log2) into CTA h's recv[rank] slot, as the Triton + // producer stores chunk `rank`; st.async stores that complete_tx on CTA h's barrier. + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); + { + int const g = lane >> 2; + float const vgs = __ldg(p.vgs); +#pragma unroll + for (int r = 0; r < 2; ++r) + { + float scale, lse; + if constexpr (LINEAR) + { + scale = l_row[r] > 0.F ? 1.0F / l_row[r] : 0.F; + lse = l_row[r] > 0.F ? m_row[r] + lg2(l_row[r]) : kNegInf; + } + else + { + float const lse_i = active ? m_row[r] + lg2(ex2(kNegInf - m_row[r]) + l_row[r]) : kNegInf; + scale = lse_i > kNegInf ? ex2(m_row[r] - lse_i) : 0.F; + lse = lse_i; + } + scale = scale * vgs; + int const row = g + r * 8; + uint32_t const dst = map_cluster(&s.recv[rank][0], row); + uint32_t const rb = map_cluster(&s.rbar, row); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const d = warp * 32 + j * 8 + 2 * (lane & 3); + st_async_v2(dst + d * 4, acc[j][2 * r] * scale, acc[j][2 * r + 1] * scale, rb); + } + if (warp == 0 && (lane & 3) == 0) + st_async_f32(map_cluster(&s.recv_lse[rank], row), lse, rb); + } + } + { + // Every CTA's partial row and LSE for head `rank` landed (complete_tx on this CTA's barrier). + uint32_t done = 0; + while (!done) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}" + : "=r"(done) + : "r"(smem_u32(&s.rbar)) + : "memory"); + } + } + + // CTA `rank` merges head `rank` across the 16 partials; thread = output dim. + int const h = static_cast(rank); + float lse_c[kTopk]; + float lse_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + lse_c[c] = s.recv_lse[c]; + lse_max = fmaxf(lse_max, lse_c[c]); + } + lse_max = lse_max == kNegInf ? 0.F : lse_max; + float denom = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + lse_c[c] = ex2(lse_c[c] - lse_max); + denom += lse_c[c]; + } + denom = denom > 0.F ? denom : 1.F; + float o = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + o += s.recv[c][tid] * __fdiv_rn(lse_c[c], denom); + __nv_bfloat16 const ob = __float2bfloat16_rn(o); + p.out[tok * p.out_stride + h * kDim + tid] = ob; + + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + amax = fmaxf(amax, __shfl_xor_sync(0xffffffff, amax, off)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + float const xq = x * qs; + uint16_t q2; + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2) : "f"(0.F), "f"(xq)); + p.q8[static_cast(tok) * (p.num_sf_k_tiles * 128) + h * kDim + tid] = static_cast(q2 & 0xFF); + if (lane == 0) + { + int const k_sf = h * 4 + warp; + int64_t const off = static_cast(tok / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (tok % 32) * 16 + ((tok % 128) / 32) * 4 + k_sf % 4; + p.sf[off] = static_cast(e); + } + } + if constexpr (!EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +} + +template +cudaError_t configure() +{ + auto kernel = m3_lc_attn_kernel; + cudaError_t err = cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, sizeof(Smem)); + if (err != cudaSuccess) + { + return err; + } + return cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1); +} + +template +cudaError_t launch(Params const& prm, int num_tokens, bool use_pdl, cudaStream_t stream) +{ + // Kernel attributes are set once per process, on the first launch (outside CUDA-graph capture: warmup runs + // every variant eagerly first). + static cudaError_t const configured = configure(); + if (configured != cudaSuccess) + { + return configured; + } + auto kernel = m3_lc_attn_kernel; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(num_tokens * kTopk); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = sizeof(Smem); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kTopk; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +} // namespace + +cudaError_t invokeMinimaxM3SparseDecodeLc(MinimaxM3SparseDecodeLcParams const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool earlyTrigger, bool usePdl, cudaStream_t stream) +{ + if (!earlyTrigger) + { + return cudaErrorNotSupported; + } + if (linearSoftmax) + { + return mxfp8 ? launch(params, numTokens, usePdl, stream) + : launch(params, numTokens, usePdl, stream); + } + return mxfp8 ? launch(params, numTokens, usePdl, stream) + : launch(params, numTokens, usePdl, stream); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.h b/cpp/tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.h new file mode 100644 index 000000000000..659531f725bf --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.h @@ -0,0 +1,70 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Operands of the MiniMax-M3 one-launch long-context sparse decode. Strides are in elements of the pointed-to type. +//! Geometry is fixed: 16 query heads on one KV head, head_dim 128, page 128, top-k 16. +struct MinimaxM3SparseDecodeLcParams +{ + uint8_t const* q; // e4m3 [T, 16, 128] + int64_t q_tok, q_head; + uint8_t const* k; // packed e2m1 [pages, 1, 128, 64] + uint8_t const* v; + int64_t k_blk, v_blk; + uint8_t const* ksc; // e4m3 [pages, 1, 128, 8], token-major + uint8_t const* vsc; // e4m3, 4x4 token-quad order + int64_t ksc_blk, vsc_blk; + float const* kgs; + float const* vgs; + int const* topk; // [T, 16] view (kv head 0) + int64_t t_tok, t_k; + int const* block_table; + int64_t bt_stride; + int const* seq_lens; + int dql; + float sm_scale; + __nv_bfloat16* out; // [T, 2048] + int64_t out_stride; + uint8_t* q8; // e4m3 [rows, num_sf_k_tiles * 128] + uint8_t* sf; + int num_sf_k_tiles; +}; + +//! One cluster of 16 CTAs per query token: CTA r attends the token's r-th selected block, then the cluster merges +//! the 16 partials over DSMEM into the BF16 row (and, when mxfp8, its MXFP8 copy with swizzled UE8M0 scales). +//! +//! Instantiated variants: linearSoftmax in {false, true} x mxfp8 in {false, true}, with earlyTrigger (release PDL +//! dependents at entry). Returns cudaErrorNotSupported for any other variant. The device code is SM100-family only; +//! on other architectures the kernel compiles to an empty body, so callers check the device first. +cudaError_t invokeMinimaxM3SparseDecodeLc(MinimaxM3SparseDecodeLcParams const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool earlyTrigger, bool usePdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu b/cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu new file mode 100644 index 000000000000..4069da14faf0 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu @@ -0,0 +1,425 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/specTail.h" + +#include +#include +#include + +// Small fused kernels of the MiniMax-M3 Eagle3 step (GB300, decode). +// +// chunk_norm / capture_norm apply the drafter's per-layer fc_norm RMSNorms to the captured target hidden states with +// the in-tree RMSNorm arithmetic (fp32 statistic, (x * rsqrt) * w, one bf16 rounding); step_gather / slot_scatter move +// the step's integer outputs between the spec sampler's store and the engine's input buffers. +// +// draft_input replaces a fused draft pass's embedding lookup, its two RMSNorms (input_layernorm on the embedding, +// hidden_norm on the carried hidden state, the same arithmetic) and their concatenation with one launch. It optionally +// gathers the carried hidden rows (the second pass reads the last accepted row of each request) and writes them out +// as the residual of the decoder layer. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +namespace specTail +{ +namespace +{ +constexpr int kVec = 8; // bf16 per 16-byte access + +__device__ __forceinline__ float block_sum(float v, float* red) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + int const warp = threadIdx.x >> 5; + if ((threadIdx.x & 31) == 0) + red[warp] = v; + __syncthreads(); + float s = 0.f; +#pragma unroll + for (int w = 0; w < kThreads / 32; ++w) + s += red[w]; + return s; +} + +// blockIdx.x = row, blockIdx.y = 0 (embedding half) / 1 (hidden half). H % (kThreads * kVec) == 0 is not required; +// H % kVec == 0 and H <= kMaxPerThread * kThreads * kVec are. +template +__global__ void __launch_bounds__(kThreads) + draft_input_kernel(int const* __restrict__ ids, int vocab, __nv_bfloat16 const* __restrict__ table, + __nv_bfloat16 const* __restrict__ hidden, long long const* __restrict__ hidden_rows, + __nv_bfloat16 const* __restrict__ w_embed, __nv_bfloat16 const* __restrict__ w_hidden, float eps_embed, + float eps_hidden, int H, __nv_bfloat16* __restrict__ out, __nv_bfloat16* __restrict__ residual) +{ + __shared__ float red[kThreads / 32]; + int const row = blockIdx.x; + bool const is_hidden = blockIdx.y == 1; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + __nv_bfloat16 const* src; + bool valid = true; + if (is_hidden) + { + long long const r = hidden_rows ? hidden_rows[row] : row; + src = hidden + r * H; + } + else + { + int const id = ids[row]; + valid = id >= 0 && id < vocab; + src = table + static_cast(valid ? id : 0) * H; + } + float v[kIters][kVec]; + float ss = 0.f; +#pragma unroll + for (int it = 0; it < kIters; ++it) + { + int const c = (it * kThreads + threadIdx.x) * kVec; + if (c < H) + { + uint4 const raw = *reinterpret_cast(src + c); + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&raw); +#pragma unroll + for (int j = 0; j < kVec; ++j) + { + v[it][j] = valid ? __bfloat162float(e[j]) : 0.f; + ss += v[it][j] * v[it][j]; + } + if (is_hidden && residual) + *reinterpret_cast(residual + static_cast(row) * H + c) = raw; + } + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float const rs = rsqrtf(block_sum(ss, red) / H + (is_hidden ? eps_hidden : eps_embed)); + __nv_bfloat16 const* w = is_hidden ? w_hidden : w_embed; + __nv_bfloat16* dst = out + static_cast(row) * 2 * H + (is_hidden ? H : 0); +#pragma unroll + for (int it = 0; it < kIters; ++it) + { + int const c = (it * kThreads + threadIdx.x) * kVec; + if (c < H) + { + uint4 const wr = *reinterpret_cast(w + c); + __nv_bfloat16 const* we = reinterpret_cast<__nv_bfloat16 const*>(&wr); + uint4 o; + __nv_bfloat16* oe = reinterpret_cast<__nv_bfloat16*>(&o); +#pragma unroll + for (int j = 0; j < kVec; ++j) + oe[j] = __float2bfloat16(v[it][j] * rs * __bfloat162float(we[j])); + *reinterpret_cast(dst + c) = o; + } + } +} + +// blockIdx.x = row, blockIdx.y = chunk c: dst[row, c*H:(c+1)*H] = rmsnorm(src[row, c*H:(c+1)*H]) * w[c]. +template +__global__ void __launch_bounds__(kThreads) chunk_norm_kernel(__nv_bfloat16 const* __restrict__ src, + long long src_stride, __nv_bfloat16 const* __restrict__ w, float eps, int H, int C, __nv_bfloat16* __restrict__ dst) +{ + __shared__ float red[kThreads / 32]; + int const row = blockIdx.x; + int const c = blockIdx.y; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + __nv_bfloat16 const* in = src + row * src_stride + static_cast(c) * H; + float v[kIters][kVec]; + float ss = 0.f; +#pragma unroll + for (int it = 0; it < kIters; ++it) + { + int const col = (it * kThreads + threadIdx.x) * kVec; + if (col < H) + { + uint4 const raw = *reinterpret_cast(in + col); + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&raw); +#pragma unroll + for (int j = 0; j < kVec; ++j) + { + v[it][j] = __bfloat162float(e[j]); + ss += v[it][j] * v[it][j]; + } + } + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float const rs = rsqrtf(block_sum(ss, red) / H + eps); + __nv_bfloat16 const* wc = w + static_cast(c) * H; + __nv_bfloat16* out = dst + (static_cast(row) * C + c) * H; +#pragma unroll + for (int it = 0; it < kIters; ++it) + { + int const col = (it * kThreads + threadIdx.x) * kVec; + if (col < H) + { + uint4 const wr = *reinterpret_cast(wc + col); + __nv_bfloat16 const* we = reinterpret_cast<__nv_bfloat16 const*>(&wr); + uint4 o; + __nv_bfloat16* oe = reinterpret_cast<__nv_bfloat16*>(&o); +#pragma unroll + for (int j = 0; j < kVec; ++j) + oe[j] = __float2bfloat16(v[it][j] * rs * __bfloat162float(we[j])); + *reinterpret_cast(out + col) = o; + } + } +} + +// Eagle3 hidden-state capture with the drafter's fc_norm folded in (one row per CTA): +// dst[row, :H] = rmsnorm(bf16(h[row] + r[row])) * w (r optional) +// with chunk_norm's arithmetic, so fc reads the same values the capture + chunk_norm pair produced. +template +__global__ void __launch_bounds__(kThreads) capture_norm_kernel(__nv_bfloat16 const* __restrict__ h, long long h_stride, + __nv_bfloat16 const* __restrict__ r, long long r_stride, __nv_bfloat16 const* __restrict__ w, float eps, int H, + __nv_bfloat16* __restrict__ dst, long long dst_stride) +{ + __shared__ float red[kThreads / 32]; + int const row = blockIdx.x; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + float v[kIters][kVec]; + float ss = 0.f; +#pragma unroll + for (int it = 0; it < kIters; ++it) + { + int const col = (it * kThreads + threadIdx.x) * kVec; + if (col < H) + { + uint4 const raw = *reinterpret_cast(h + row * h_stride + col); + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&raw); + uint4 rraw = make_uint4(0u, 0u, 0u, 0u); + if (r != nullptr) + rraw = *reinterpret_cast(r + row * r_stride + col); + __nv_bfloat16 const* re = reinterpret_cast<__nv_bfloat16 const*>(&rraw); +#pragma unroll + for (int j = 0; j < kVec; ++j) + { + float const x = __bfloat162float(e[j]); + v[it][j] = r != nullptr ? __bfloat162float(__float2bfloat16(x + __bfloat162float(re[j]))) : x; + ss += v[it][j] * v[it][j]; + } + } + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float const rs = rsqrtf(block_sum(ss, red) / H + eps); + __nv_bfloat16* out = dst + row * dst_stride; +#pragma unroll + for (int it = 0; it < kIters; ++it) + { + int const col = (it * kThreads + threadIdx.x) * kVec; + if (col < H) + { + uint4 const wr = *reinterpret_cast(w + col); + __nv_bfloat16 const* we = reinterpret_cast<__nv_bfloat16 const*>(&wr); + uint4 o; + __nv_bfloat16* oe = reinterpret_cast<__nv_bfloat16*>(&o); +#pragma unroll + for (int j = 0; j < kVec; ++j) + oe[j] = __float2bfloat16(v[it][j] * rs * __bfloat162float(we[j])); + *reinterpret_cast(out + col) = o; + } + } +} + +// Overlap-scheduler step inputs from the previous step's device outputs (one launch for the engine's gathers): +// ids[ids0 + p*T + t] = new_tok[t * nt_s0 + slot_p * nt_s1] +// draft[d0 + p*D + d] = next_draft[slot_p * nd_s0 + d * nd_s1] +// pos_off[i] = i in [b0*T, (b0+P)*T) ? lens[ppi[i - b0*T]] : 0 (i < n_pos) +// kv_off[i] = i in [b0, b0+P) ? lens[slot_{i-b0}] - T : 0 (i < n_kv) +__global__ void step_gather_kernel(int const* __restrict__ new_tok, long long nt_s0, long long nt_s1, + int const* __restrict__ next_draft, long long nd_s0, long long nd_s1, int const* __restrict__ lens, + int const* __restrict__ slots, int const* __restrict__ ppi, int P, int T, int D, int* __restrict__ ids, int ids0, + int* __restrict__ draft, int d0, int* __restrict__ pos_off, int n_pos, int* __restrict__ kv_off, int n_kv, int b0) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + int const tid = blockIdx.x * blockDim.x + threadIdx.x; + int const stride = gridDim.x * blockDim.x; + for (int i = tid; i < P * T; i += stride) + { + int const p = i / T, t = i - p * T; + ids[ids0 + i] = new_tok[t * nt_s0 + static_cast(slots[p]) * nt_s1]; + } + for (int i = tid; i < P * D; i += stride) + { + int const p = i / D, d = i - p * D; + draft[d0 + i] = next_draft[static_cast(slots[p]) * nd_s0 + d * nd_s1]; + } + for (int i = tid; i < n_pos; i += stride) + { + int const k = i - b0 * T; + pos_off[i] = (k >= 0 && k < P * T) ? lens[ppi[k]] : 0; + } + for (int i = tid; i < n_kv; i += stride) + { + int const k = i - b0; + kv_off[i] = (k >= 0 && k < P) ? lens[slots[k]] - T : 0; + } +} + +// Sampler stores from one step's outputs (one launch for its index_copy_s), zero-padding to the store widths: +// a[slot_i][c] for c < wa (from src_a[i][c] if c < sa_w), b likewise, lens[slot_i] = src_lens[i], d likewise. +__global__ void slot_scatter_kernel(long long const* __restrict__ slots, int n, int const* __restrict__ sa, + long long sa_s0, long long sa_s1, int sa_w, int* __restrict__ da, long long da_s0, long long da_s1, int wa, + int const* __restrict__ sb, long long sb_s0, long long sb_s1, int sb_w, int* __restrict__ db, long long db_s0, + long long db_s1, int wb, int const* __restrict__ sl, int* __restrict__ dl, int const* __restrict__ sd, + long long sd_s0, long long sd_s1, int sd_w, int* __restrict__ dd, long long dd_s0, long long dd_s1, int wd) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + int const i = blockIdx.x; + if (i >= n) + return; + long long const s = slots[i]; + for (int c = threadIdx.x; c < wa; c += blockDim.x) + da[s * da_s0 + c * da_s1] = c < sa_w ? sa[i * sa_s0 + c * sa_s1] : 0; + for (int c = threadIdx.x; c < wb; c += blockDim.x) + db[s * db_s0 + c * db_s1] = c < sb_w ? sb[i * sb_s0 + c * sb_s1] : 0; + for (int c = threadIdx.x; c < wd; c += blockDim.x) + dd[s * dd_s0 + c * dd_s1] = c < sd_w ? sd[i * sd_s0 + c * sd_s1] : 0; + if (threadIdx.x == 0) + dl[s] = sl[i]; +} + +cudaLaunchConfig_t small_cfg(int grid, int block, bool pdl, cudaLaunchAttribute* attr, cudaStream_t stream) +{ + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(grid); + cfg.blockDim = dim3(block); + cfg.stream = stream; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cfg; +} + +int norm_iters(int H) +{ + return (H / kVec + kThreads - 1) / kThreads; +} + +} // namespace + +cudaError_t invokeDraftInput(int const* ids, int vocab, __nv_bfloat16 const* table, __nv_bfloat16 const* hidden, + long long const* rows, __nv_bfloat16 const* wEmbed, __nv_bfloat16 const* wHidden, float epsEmbed, float epsHidden, + int H, int M, __nv_bfloat16* out, __nv_bfloat16* residual, bool pdl, cudaStream_t stream) +{ + int const iters = norm_iters(H); + cudaLaunchAttribute attr[1]; + cudaLaunchConfig_t cfg = small_cfg(1, kThreads, pdl, attr, stream); + cfg.gridDim = dim3(M, 2); +#define LAUNCH(I) \ + cudaLaunchKernelEx(&cfg, draft_input_kernel, ids, vocab, table, hidden, rows, wEmbed, wHidden, epsEmbed, \ + epsHidden, H, out, residual) + switch (iters) + { + case 0: + case 1: return LAUNCH(1); + case 2: return LAUNCH(2); + case 3: return LAUNCH(3); + case 4: return LAUNCH(4); + default: return cudaErrorInvalidValue; + } +#undef LAUNCH +} + +cudaError_t invokeChunkNorm(__nv_bfloat16 const* src, long long srcStride, __nv_bfloat16 const* w, float eps, int H, + int C, int M, __nv_bfloat16* dst, bool pdl, cudaStream_t stream) +{ + int const iters = norm_iters(H); + cudaLaunchAttribute attr[1]; + cudaLaunchConfig_t cfg = small_cfg(1, kThreads, pdl, attr, stream); + cfg.gridDim = dim3(M, C); +#define LAUNCH(I) cudaLaunchKernelEx(&cfg, chunk_norm_kernel, src, srcStride, w, eps, H, C, dst) + switch (iters) + { + case 0: + case 1: return LAUNCH(1); + case 2: return LAUNCH(2); + case 3: return LAUNCH(3); + case 4: return LAUNCH(4); + default: return cudaErrorInvalidValue; + } +#undef LAUNCH +} + +cudaError_t invokeCaptureNorm(__nv_bfloat16 const* h, long long hStride, __nv_bfloat16 const* res, long long resStride, + __nv_bfloat16 const* w, float eps, int H, int M, __nv_bfloat16* dst, long long dstStride, cudaStream_t stream) +{ + if (M == 0) + { + return cudaSuccess; + } + int const iters = norm_iters(H); + cudaLaunchAttribute attr[1]; + cudaLaunchConfig_t cfg = small_cfg(M, kThreads, true, attr, stream); +#define LAUNCH(I) \ + cudaLaunchKernelEx(&cfg, capture_norm_kernel, h, hStride, res, resStride, w, eps, H, dst, dstStride) + switch (iters) + { + case 0: + case 1: return LAUNCH(1); + case 2: return LAUNCH(2); + case 3: return LAUNCH(3); + case 4: return LAUNCH(4); + default: return cudaErrorInvalidValue; + } +#undef LAUNCH +} + +cudaError_t invokeStepGather(int const* newTok, long long ntS0, long long ntS1, int const* nextDraft, long long ndS0, + long long ndS1, int const* lens, int const* slots, int const* ppi, int P, int T, int D, int* ids, int ids0, + int* draft, int d0, int* posOff, int nPos, int* kvOff, int nKv, int b0, int work, cudaStream_t stream) +{ + cudaLaunchAttribute attr[1]; + cudaLaunchConfig_t cfg = small_cfg(std::max(1, std::min(64, (work + 255) / 256)), 256, true, attr, stream); + return cudaLaunchKernelEx(&cfg, step_gather_kernel, newTok, ntS0, ntS1, nextDraft, ndS0, ndS1, lens, slots, ppi, P, + T, D, ids, ids0, draft, d0, posOff, nPos, kvOff, nKv, b0); +} + +cudaError_t invokeSlotScatter(long long const* slots, int n, IntView sa, IntView da, IntView sb, IntView db, + int const* sl, int* dl, IntView sd, IntView dd, cudaStream_t stream) +{ + if (n == 0) + { + return cudaSuccess; + } + cudaLaunchAttribute attr[1]; + cudaLaunchConfig_t cfg = small_cfg(n, 32, true, attr, stream); + return cudaLaunchKernelEx(&cfg, slot_scatter_kernel, slots, n, sa.data, sa.s0, sa.s1, sa.width, da.data, da.s0, + da.s1, da.width, sb.data, sb.s0, sb.s1, sb.width, db.data, db.s0, db.s1, db.width, sl, dl, sd.data, sd.s0, + sd.s1, sd.width, dd.data, dd.s0, dd.s1, dd.width); +} + +} // namespace specTail +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/specTail.h b/cpp/tensorrt_llm/kernels/minimaxM3/specTail.h new file mode 100644 index 000000000000..c5bdf75ff482 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/specTail.h @@ -0,0 +1,80 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +//! Small fused kernels of the MiniMax-M3 Eagle3 step: the drafter's fc_norm RMSNorms over the captured hidden states +//! and the step's input / output bookkeeping (GB300, decode). All launches use programmatic dependent launch where +//! noted. +namespace specTail +{ + +//! Threads per CTA of the norm kernels, and the hidden-size bound of their register tiles: +//! H % 8 == 0 and H <= kMaxIters * kThreads * 8. +constexpr int kThreads = 256; +constexpr int kMaxIters = 4; + +//! out[r] = [rmsnorm(table[ids[r]]) * wEmbed, rmsnorm(hidden[rows ? rows[r] : r]) * wHidden] ([M, 2H] BF16; ids +//! outside [0, vocab) embed to zero); residual[r] = hidden[rows[r]] when residual is set. +cudaError_t invokeDraftInput(int const* ids, int vocab, __nv_bfloat16 const* table, __nv_bfloat16 const* hidden, + long long const* rows, __nv_bfloat16 const* wEmbed, __nv_bfloat16 const* wHidden, float epsEmbed, float epsHidden, + int H, int M, __nv_bfloat16* out, __nv_bfloat16* residual, bool pdl, cudaStream_t stream); + +//! dst[r, c*H:(c+1)*H] = rmsnorm(src[r, c*H:(c+1)*H]) * w[c] for c < C (src row stride srcStride), dst [M, C*H]. +cudaError_t invokeChunkNorm(__nv_bfloat16 const* src, long long srcStride, __nv_bfloat16 const* w, float eps, int H, + int C, int M, __nv_bfloat16* dst, bool pdl, cudaStream_t stream); + +//! dst[r, :H] = rmsnorm(bf16(h[r] + res[r])) * w (res optional), with chunk-norm rounding. +cudaError_t invokeCaptureNorm(__nv_bfloat16 const* h, long long hStride, __nv_bfloat16 const* res, long long resStride, + __nv_bfloat16 const* w, float eps, int H, int M, __nv_bfloat16* dst, long long dstStride, cudaStream_t stream); + +//! The overlap scheduler's per-step input gathers (see step_gather_kernel); work = the largest of its four loops. +cudaError_t invokeStepGather(int const* newTok, long long ntS0, long long ntS1, int const* nextDraft, long long ndS0, + long long ndS1, int const* lens, int const* slots, int const* ppi, int P, int T, int D, int* ids, int ids0, + int* draft, int d0, int* posOff, int nPos, int* kvOff, int nKv, int b0, int work, cudaStream_t stream); + +//! A strided int32 2-D view: element (i, c) at data[i * s0 + c * s1], `width` columns. +struct IntView +{ + int* data; + long long s0; + long long s1; + int width; +}; + +//! The spec sampler's slot stores (see slot_scatter_kernel): for i < n, da / db / dd rows slots[i] get sa / sb / sd +//! row i, zero-padded to the destination widths, and dl[slots[i]] = sl[i]. +cudaError_t invokeSlotScatter(long long const* slots, int n, IntView sa, IntView da, IntView sb, IntView db, + int const* sl, int* dl, IntView sd, IntView dd, cudaStream_t stream); + +} // namespace specTail +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.cu b/cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.cu new file mode 100644 index 000000000000..2e80de17bfd9 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.cu @@ -0,0 +1,69 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/stepInputsScatter.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kThreads = 256; + +__device__ __forceinline__ void copyBytes(uint8_t* dst, uint8_t const* src, int64_t bytes) +{ + for (int64_t b = threadIdx.x; b < bytes; b += blockDim.x) + { + dst[b] = src[b]; + } +} + +// One CTA: the batches this serves are a step's few hundred bytes of metadata, where a single launch is the point and +// the copy itself is negligible. +__global__ void __launch_bounds__(kThreads) stepInputsScatterKernel(uint8_t const* __restrict__ stage) +{ + auto const* header = reinterpret_cast(stage); + int64_t const numHostCopies = header[0]; + int64_t const numDeviceCopies = header[1]; + int64_t const* desc = header + 2; + // Copies complete in descriptor order: one may overwrite or read another's destination. + for (int64_t i = 0; i < numHostCopies; ++i) + { + copyBytes(reinterpret_cast(desc[3 * i]), stage + desc[3 * i + 1], desc[3 * i + 2]); + __syncthreads(); + } + for (int64_t i = numHostCopies; i < numHostCopies + numDeviceCopies; ++i) + { + copyBytes(reinterpret_cast(desc[3 * i]), reinterpret_cast(desc[3 * i + 1]), + desc[3 * i + 2]); + __syncthreads(); + } +} + +} // namespace + +void invokeStepInputsScatter(uint8_t const* stage, cudaStream_t stream) +{ + stepInputsScatterKernel<<<1, kThreads, 0, stream>>>(stage); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.h b/cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.h new file mode 100644 index 000000000000..353197a479f9 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/stepInputsScatter.h @@ -0,0 +1,40 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Write a packed batch of byte copies described by a staging buffer. +//! +//! The staging buffer starts with an int64 header `[numHostCopies, numDeviceCopies]`, followed by one +//! `(dst, src, bytes)` int64 triple per copy. A host copy's `src` is a byte offset into the staging buffer (its +//! bytes were uploaded with the header); a device copy's `src` is a device address. Every host copy is written +//! before any device copy, so a device copy may read a destination a host copy of the same batch filled. +void invokeStepInputsScatter(uint8_t const* stage, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/targetHead.cu b/cpp/tensorrt_llm/kernels/minimaxM3/targetHead.cu new file mode 100644 index 000000000000..847c59779a6e --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/targetHead.cu @@ -0,0 +1,372 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/targetHead.h" + +#include +#include +#include + +// The MiniMax-M3 target lm_head shard GEMV of a verify step of at most 8 rows (GB300, TP4), at stock precision: BF16 +// weight and activation, mma.m16n8k16 BF16 with FP32 accumulation over the full K, one rounding to BF16 per logit +// (the cuBLAS head differs only in the FP32 summation order). The full logit row is produced. +// +// * Lossless 12-bit weight coding: the BF16 exponents of a trained weight sit almost all in a window of 15 +// consecutive values [base, base + 14]. Each weight is one byte sign << 7 | mantissa plus a 4-bit code +// exponent - base; the decoded BF16 bits are the checkpoint's. Code 15 marks an exception: it decodes to +0 (it +// adds nothing to the MMA sums), and the segment epilogue adds the exact product x * w from a per-tile list (a +// product of two BF16 values is exact in FP32), so every product enters the sum exactly as in the BF16 head. +// * Layout: the weight is permuted into mma A-fragment order (per 16-row tile and k16 step, lane (g, j) holds +// {row g k 2j..2j+1, row g+8 k 2j.., row g k 2j+8.., row g+8 k 2j+8..}); per unit of 4 k16 steps a lane reads +// 48 bytes: the sign/mantissa bytes of steps 0-1 and of steps 2-3, then 4 code words (even values' nibbles in the +// low half, odd values' in the high half, so the codes of value pair p are (word >> 4p) & 0x000f000f). +// * A persistent grid (one 16-warp CTA per SM) splits the (tile, unit) space into equal contiguous ranges, one per +// warp, each streamed through a 3-unit cp.async ring in shared memory. The first units are issued before the PDL +// grid-dependency wait (the weight does not depend on the previous kernel); the activation is staged in shared +// memory in B-fragment order after it. +// * A tile split between warps writes per-warp FP32 partials; the last finisher (per-tile counter) adds them in warp +// order, so the result is deterministic run to run. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +namespace targetHead +{ +namespace +{ + +constexpr int kUnit = 4; // k16 steps per coded unit + +struct Args +{ + Params p; + int tiles; + int ks; // K / 16 + long long units; // tiles * ks / kUnit +}; + +__device__ __forceinline__ void cp_async16(void* smem, void const* gmem) +{ + uint32_t const sa = static_cast(__cvta_generic_to_shared(smem)); + asm volatile("cp.async.cg.shared.global.L2::256B [%0], [%1], 16;\n" ::"r"(sa), "l"(gmem) : "memory"); +} + +__device__ __forceinline__ void cp_async_commit() +{ + asm volatile("cp.async.commit_group;\n" ::: "memory"); +} + +template +__device__ __forceinline__ void cp_async_wait() +{ + asm volatile("cp.async.wait_group %0;\n" ::"n"(N) : "memory"); +} + +__device__ __forceinline__ void mma_bf16(float (&d)[4], uint4 const& a, uint2 const& b) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.bf16.bf16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};\n" + : "+f"(d[0]), "+f"(d[1]), "+f"(d[2]), "+f"(d[3]) + : "r"(a.x), "r"(a.y), "r"(a.z), "r"(a.w), "r"(b.x), "r"(b.y)); +} + +// Accumulator layout of one tile: c[0..1] = (row g, tokens 2j, 2j + 1), c[2..3] = (row g + 8, ...). +__device__ __forceinline__ void store_tile(Args const& a, int tile, int lane, float const (&c)[4]) +{ + int const g = lane >> 2, j = lane & 3; +#pragma unroll + for (int e = 0; e < 4; ++e) + { + int const n = 2 * j + (e & 1); + int const row = tile * 16 + g + (e >> 1) * 8; + if (n < a.p.M && row < a.p.N) + a.p.out[n * a.p.out_stride + row] = __float2bfloat16_rn(c[e]); + } +} + +// Flush the warp's sums of `tile` over k16 steps [k0, k1). +__device__ __forceinline__ void flush(Args const& a, int warp, int tile, int k0, int k1, int lane, float const (&c)[4]) +{ + if (k0 == 0 && k1 == a.ks) + { + store_tile(a, tile, lane, c); + return; + } + // Warps owning parts of this tile: the owner of its first unit up to the owner of its last one, where + // owner(u) = the largest w with w * units / W <= u. The first owner stores into its slot 1 (the tile is its last + // segment), every later owner into slot 0 (its first segment), so a warp never reuses a slot. + long long const W = static_cast(gridDim.x) * (blockDim.x >> 5); + long long const first = static_cast(tile) * (a.ks / kUnit); + auto owner = [&](long long u) + { + long long w = (u * W) / a.units; + while (w + 1 < W && ((w + 1) * a.units) / W <= u) + ++w; + while (w > 0 && (w * a.units) / W > u) + --w; + return static_cast(w); + }; + int const wf = owner(first); + int const wl = owner(first + a.ks / kUnit - 1); + constexpr int kVals = 4 * 32; + float* mine = a.p.part + (static_cast(warp) * 2 + (warp == wf ? 1 : 0)) * kVals; +#pragma unroll + for (int e = 0; e < 4; ++e) + mine[e * 32 + lane] = c[e]; + __threadfence(); + __syncwarp(); + int done = 0; + if (lane == 0) + done = atomicAdd(&a.p.counter[tile], 1) == wl - wf; + done = __shfl_sync(0xffffffffu, done, 0); + if (!done) + return; + __threadfence(); + float s[4] = {0.f, 0.f, 0.f, 0.f}; + for (int w = wf; w <= wl; ++w) + { + float const* q = a.p.part + (static_cast(w) * 2 + (w == wf ? 1 : 0)) * kVals; +#pragma unroll + for (int e = 0; e < 4; ++e) + s[e] += __ldcg(q + e * 32 + lane); + } + store_tile(a, tile, lane, s); + if (lane == 0) + a.p.counter[tile] = 0; +} + +// BF16 pair p (values 2p, 2p + 1) of a lane's 8 coded values: sign/mantissa bytes sm (4 per word), codes word cw. +__device__ __forceinline__ uint32_t decode_pair(uint32_t sm_word, int byte0, uint32_t cw, int p, uint32_t base) +{ + uint32_t const b = __byte_perm(sm_word, 0u, byte0 == 0 ? 0x4140 : 0x4342); // {b0, 0, b1, 0} + uint32_t const q = (cw >> (4 * p)) & 0x000f000fu; + uint32_t const e = q * 128u + base; // exponent field of both halves + // Code 15 (an exception, sign/mantissa byte 0) decodes to +0. + return ((b & 0x007f007fu) | ((b << 8) & 0x80008000u) | e) & __vcmpne2(q, 0x000f000fu); +} + +// WARPS warps per CTA, each with a cp.async ring of S units (1536 bytes each). The range is walked tile segment by +// tile segment, so the inner loop has no boundary checks; k16 steps alternate between two accumulators (added at the +// segment end). x is staged as [ks / 4][32][4] B fragments, two 16-byte shared loads per unit. +template +__global__ void __launch_bounds__(WARPS * 32, 1) coded_kernel(Args const a) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ uint4 smem4[]; + int const lane = threadIdx.x & 31; + int const wid = threadIdx.x >> 5; + int const warp = blockIdx.x * WARPS + wid; + long long const W = static_cast(gridDim.x) * WARPS; + long long const u0 = warp * a.units / W; + long long const u1 = (warp + 1) * a.units / W; + int const n_units = static_cast(u1 - u0); + int const upt = a.ks / kUnit; // units per tile + uint4 const* wp = a.p.w + u0 * 96 + lane; + uint4* const xs4 = smem4; // [upt][32][2] uint4 + uint4* const ring = smem4 + upt * 64 + wid * (S * 96) + lane; + +#pragma unroll + for (int u = 0; u < S; ++u) + { + if (u < n_units) +#pragma unroll + for (int l = 0; l < 3; ++l) + cp_async16(ring + (u * 3 + l) * 32, wp + (u * 3 + l) * 32); + cp_async_commit(); + } + asm volatile("griddepcontrol.wait;" ::: "memory"); + { + // x[n][8q..8q+7] -> word h = q & 1 of entries (s = q / 2, lane = n * 4 + j), entry index (s / 4, lane, s % 4). + uint32_t* xs32 = reinterpret_cast(xs4); + int const qn = a.ks * 2; + int const items = kMaxRows * qn; + constexpr int kSU = 8; + for (int i0 = threadIdx.x; i0 < items; i0 += kSU * blockDim.x) + { + uint4 v[kSU]; +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + int const n = i / qn, q = i - n * qn; + v[u] = (i < items && n < a.p.M) ? *reinterpret_cast(a.p.x + n * a.p.x_stride + q * 8) + : make_uint4(0u, 0u, 0u, 0u); + } +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + if (i >= items) + continue; + int const n = i / qn, q = i - n * qn; + int const sstep = q >> 1; + int const base = (((sstep >> 2) * 32 + n * 4) * 4 + (sstep & 3)) * 2 + (q & 1); + xs32[base + 0] = v[u].x; + xs32[base + 8] = v[u].y; + xs32[base + 16] = v[u].z; + xs32[base + 24] = v[u].w; + } + } + } + __syncthreads(); + asm volatile("griddepcontrol.launch_dependents;"); + + uint4 const* xl = xs4 + lane * 2; + int tile = static_cast(u0 / upt); + int ku = static_cast(u0 % upt); // unit within the tile + int done = 0, slot = 0; + while (done < n_units) + { + int const seg = min(n_units - done, upt - ku); + float ca[4] = {0.f, 0.f, 0.f, 0.f}; + float cb[4] = {0.f, 0.f, 0.f, 0.f}; + int const e0 = __ldg(a.p.exc_off + tile), e1 = __ldg(a.p.exc_off + tile + 1); + for (int i = 0; i < seg; ++i) + { + cp_async_wait(); + uint4 const sm01 = ring[slot * 96]; + uint4 const sm23 = ring[slot * 96 + 32]; + uint4 const cw = ring[slot * 96 + 64]; + uint4 const xa = xl[(ku + i) * 64]; + uint4 const xb = xl[(ku + i) * 64 + 1]; + uint32_t const smw[8] = {sm01.x, sm01.y, sm01.z, sm01.w, sm23.x, sm23.y, sm23.z, sm23.w}; + uint32_t const cws[4] = {cw.x, cw.y, cw.z, cw.w}; + uint2 const xb2[4] + = {make_uint2(xa.x, xa.y), make_uint2(xa.z, xa.w), make_uint2(xb.x, xb.y), make_uint2(xb.z, xb.w)}; +#pragma unroll + for (int ci = 0; ci < 4; ++ci) + { + uint4 wv; + wv.x = decode_pair(smw[2 * ci], 0, cws[ci], 0, a.p.exp_base); + wv.y = decode_pair(smw[2 * ci], 2, cws[ci], 1, a.p.exp_base); + wv.z = decode_pair(smw[2 * ci + 1], 0, cws[ci], 2, a.p.exp_base); + wv.w = decode_pair(smw[2 * ci + 1], 2, cws[ci], 3, a.p.exp_base); + if (ci & 1) + mma_bf16(cb, wv, xb2[ci]); + else + mma_bf16(ca, wv, xb2[ci]); + } + // Refill the slot only after its values were consumed (no ordering between ld.shared and cp.async). + int const nxt = done + i + S; + if (nxt < n_units) +#pragma unroll + for (int l = 0; l < 3; ++l) + cp_async16(ring + (slot * 3 + l) * 32, wp + (nxt * 3 + l) * 32); + cp_async_commit(); + slot = slot + 1 == S ? 0 : slot + 1; + } +#pragma unroll + for (int e = 0; e < 4; ++e) + ca[e] += cb[e]; + // Exceptions of this tile with k in the segment: lanes load 32 at a time, each is broadcast; the owner lanes + // (row g or g + 8) add the exact product x[n][k] * w with x from shared memory. + { + int const g = lane >> 2, j = lane & 3; + int const kb = ku * 64, ke = (ku + seg) * 64; + uint32_t const* xs32 = reinterpret_cast(xs4); + for (int b0 = e0; b0 < e1; b0 += 32) + { + uint32_t pos = 0xffffffffu; + float2 wv = make_float2(0.f, 0.f); + if (b0 + lane < e1) + { + pos = __ldg(a.p.exc_pos + b0 + lane); + wv = __ldg(a.p.exc_w + b0 + lane); + } + int const cnt = min(32, e1 - b0); + for (int q = 0; q < cnt; ++q) + { + uint32_t const pq = __shfl_sync(0xffffffffu, pos, q); + float const w0 = __shfl_sync(0xffffffffu, wv.x, q); + int const r = static_cast(pq >> 16), k = static_cast(pq & 0xffffu); + // No early continue: every lane reaches each iteration's shuffles together. + if (k >= kb && k < ke && (r & 7) == g) + { + int const st = k >> 4, rr = k & 15; + bool const hi = r >= 8; +#pragma unroll + for (int q2 = 0; q2 < 2; ++q2) + { + int const n = 2 * j + q2; + uint32_t const word + = xs32[((((st >> 2) * 32 + n * 4 + ((rr & 7) >> 1)) * 4 + (st & 3)) * 2) + (rr >> 3)]; + float const xv = __uint_as_float((rr & 1) ? (word & 0xffff0000u) : (word << 16)); + float const d = __fmul_rn(xv, w0); + ca[q2] += hi ? 0.f : d; + ca[2 + q2] += hi ? d : 0.f; + } + } + __syncwarp(); + } + } + } + flush(a, warp, tile, ku * kUnit, (ku + seg) * kUnit, lane, ca); + done += seg; + ++tile; + ku = 0; + } +#endif +} + +} // namespace + +size_t targetHeadSmemBytes(int K) +{ + return static_cast(K / 64) * 64 * 16 + static_cast(kWarps) * kStages * 96 * 16; +} + +cudaError_t invokeTargetHead(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + if (p.M < 1 || p.M > kMaxRows || K % 64 != 0 || p.N < 1 || grid < 1) + return cudaErrorInvalidValue; + Args a; + a.p = p; + a.tiles = (p.N + 15) / 16; + a.ks = K / 16; + a.units = static_cast(a.tiles) * a.ks / kUnit; + // Every warp must own at least one unit: a split tile's finisher counts the warps between its first and last + // owner, so a warp with an empty range in between would leave the tile unstored. + if (a.units < kWarps) + return cudaErrorInvalidValue; + grid = static_cast(std::min(grid, a.units / kWarps)); + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(static_cast(grid)); + cfg.blockDim = dim3(kWarps * 32); + cfg.dynamicSmemBytes = targetHeadSmemBytes(K); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + auto kern = coded_kernel; + cudaError_t const err = cudaFuncSetAttribute( + kern, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(cfg.dynamicSmemBytes)); + if (err != cudaSuccess) + return err; + return cudaLaunchKernelEx(&cfg, kern, a); +} + +} // namespace targetHead +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/targetHead.h b/cpp/tensorrt_llm/kernels/minimaxM3/targetHead.h new file mode 100644 index 000000000000..84b68ee2152a --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3/targetHead.h @@ -0,0 +1,68 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3 +{ +//! The MiniMax-M3 target lm_head shard GEMV over a lossless 12-bit coding of its BF16 weight (verify rows <= 8, +//! sm_103a): BF16 x BF16 MMAs with FP32 accumulation over the full K, one BF16 rounding per logit. See targetHead.cu. +namespace targetHead +{ + +constexpr int kMaxRows = 8; // activation rows (verify rows) of one launch +constexpr int kWarps = 16; // warps per CTA +constexpr int kStages = 3; // coded units in flight per warp (cp.async ring) + +struct Params +{ + uint4 const* w; // [tiles][K / 64][3][32] coded weight (see targetHead.cu) + __nv_bfloat16 const* x; // [M, K] activation rows + long long x_stride; // elements between rows of x + __nv_bfloat16* out; // [M, N] logits + long long out_stride; // elements between rows of out + float* part; // [grid * kWarps][2][4 * 32] partial sums of tiles split between warps + int* counter; // [tiles] finishers per split tile, zero between calls (re-armed by the kernel) + int const* exc_off; // [tiles + 1] exception list offsets + uint32_t const* exc_pos; // row in tile << 16 | k, per exception + float2 const* exc_w; // {w, 0} per exception: the exact weight + uint32_t exp_base; // exponent base << 7 in both BF16 halves (base << 7 | base << 23) + int M; // rows, 1..kMaxRows + int N; // logits per row (shard rows) +}; + +//! Shared memory a launch over hidden size K takes (the staged activation and the cp.async rings). +size_t targetHeadSmemBytes(int K); + +//! Launch over `grid` 512-thread CTAs (K a multiple of 64). +cudaError_t invokeTargetHead(Params const& p, int K, int grid, bool pdl, cudaStream_t stream); + +} // namespace targetHead +} // namespace minimaxM3 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.cu b/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.cu new file mode 100644 index 000000000000..fa62f8357e4d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.cu @@ -0,0 +1,418 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// ST-01-REJ: the stock chainSpeculativeSamplingKernel<1024, 4> on ctasPerRow CTAs +// per request, bitwise. v2 (bitwise v1 and stock, 24.8 vs 32.3 us at one request) +// moves loads only, every add and compare in its v1 order: the acceptance loop's probabilities are loaded in parallel +// first, the chain loop is unrolled, and the leader reads the per-tile values from shared memory. See +// chainSpecSamplingMcSm103.h for the contract. Compiled with the repo's default flags, as +// kernels/speculativeDecoding/chainSpeculativeSamplingKernels.cu. + +#include "tensorrt_llm/kernels/fusedSampling/fusedSamplingKernels.h" +#include "tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h" + +#include +#include +#include + +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +namespace +{ + +using speculative_decoding::ChainSpeculativeSamplingParams; + +constexpr int32_t kBlockSize = 1024; +constexpr int32_t kVecSize = 4; +constexpr int32_t kTileSize = kBlockSize * kVecSize; +constexpr int32_t kMaxTiles = kChainSpecMcMaxVocab / kTileSize; + +//! One request's cross-CTA state. +struct RowWorkspace +{ + //! The stock kernel's per-thread (target, residual) mass chains. + float2 chain[kBlockSize]; + //! Per tile: the stock per-tile residual (or target) sum, and the last index with mass (-1 if none). + float tileSum[kMaxTiles]; + int32_t tileLast[kMaxTiles]; + float total; + int32_t targetOnly; + //! The row barrier: CTAs arrived in this generation, and the generation (only increases). + unsigned arrive; + unsigned gen; + int32_t pad[3]; +}; + +// --------------------------------------------------------------------------------------------------------------------- +// Identical to kernels/speculativeDecoding/chainSpeculativeSamplingKernels.cu (Float2SumOp, loadProbs, loadResidual). +// --------------------------------------------------------------------------------------------------------------------- +struct Float2SumOp +{ + __device__ __forceinline__ float2 operator()(float2 a, float2 b) const + { + return make_float2(a.x + b.x, a.y + b.y); + } +}; + +template +__device__ __forceinline__ void loadProbs(float const* row, uint32_t base, uint32_t vocabSize, float (&out)[VEC_SIZE]) +{ + if constexpr (VEC_SIZE == 4) + { + if (base + VEC_SIZE <= vocabSize) + { + float4 const v = *reinterpret_cast(row + base); + out[0] = v.x; + out[1] = v.y; + out[2] = v.z; + out[3] = v.w; + return; + } + } +#pragma unroll + for (int32_t j = 0; j < VEC_SIZE; ++j) + { + out[j] = base + j < vocabSize ? row[base + j] : 0.0f; + } +} + +//! The distribution a request resamples from: max(0, target - draft) at the first rejected +//! position, or the target itself when every draft token was accepted (draft == nullptr) or +//! when the residual has no mass. +template +__device__ __forceinline__ void loadResidual( + float const* target, float const* draft, bool targetOnly, uint32_t base, uint32_t vocabSize, float (&out)[VEC_SIZE]) +{ + loadProbs(target, base, vocabSize, out); + if (targetOnly) + { + return; + } + float d[VEC_SIZE]; + loadProbs(draft, base, vocabSize, d); +#pragma unroll + for (int32_t j = 0; j < VEC_SIZE; ++j) + { + out[j] = fmaxf(out[j] - d[j], 0.0f); + } +} + +// --------------------------------------------------------------------------------------------------------------------- + +__device__ __forceinline__ unsigned loadAcquire(unsigned* p) +{ + return cuda::atomic_ref(*p).load(cuda::memory_order_acquire); +} + +//! A barrier of the request's G CTAs; the last CTA to arrive runs leader() (all its threads) before releasing the +//! others. +template +__device__ __forceinline__ void rowSync(RowWorkspace* w, int32_t G, F leader) +{ + __shared__ unsigned sGen; + __shared__ int32_t sLast; + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + sGen = loadAcquire(&w->gen); + sLast = atomicAdd(&w->arrive, 1u) == static_cast(G - 1); + } + __syncthreads(); + if (sLast) + { + __threadfence(); + leader(); + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + atomicExch(&w->arrive, 0u); + cuda::atomic_ref(w->gen).store(sGen + 1, cuda::memory_order_release); + } + } + else if (threadIdx.x == 0) + { + while (loadAcquire(&w->gen) == sGen) + { + __nanosleep(20); + } + } + __syncthreads(); +} + +__global__ void __launch_bounds__(kBlockSize, 1) + chainSpecSamplingMcKernel(ChainSpeculativeSamplingParams params, RowWorkspace* ws) +{ + using BlockReduce = cub::BlockReduce; + using BlockReducePair = cub::BlockReduce; + using BlockScan = cub::BlockScan; + + __shared__ union + { + typename BlockReduce::TempStorage reduce; + typename BlockReducePair::TempStorage reducePair; + typename BlockScan::TempStorage scan; + } tempStorage; + + __shared__ int32_t sPos; + __shared__ float sU; + __shared__ int32_t sWinner; + __shared__ float sPrefix; + __shared__ int32_t sTile; + __shared__ int32_t sTileLast; + + int64_t const row = blockIdx.y; + int32_t const G = static_cast(gridDim.x); + int32_t const cta = static_cast(blockIdx.x); + int32_t const tid = static_cast(threadIdx.x); + int32_t const numDraft = params.numDraftTokens; + uint32_t const vocabSize = static_cast(params.vocabSize); + RowWorkspace* w = ws + row; + int32_t* outTokens = params.outputTokenIds + row * (numDraft + 1); + + // 1. Acceptance: the stock tid-0 loop, identical in every CTA of the request. The probabilities it may read are + // loaded by numDraft threads in parallel first (the loop's values and order unchanged). + __shared__ float sPTarget[kChainSpecMcMaxDraft]; + __shared__ float sPDraft[kChainSpecMcMaxDraft]; + if (tid < numDraft) + { + int32_t const token = params.draftTokenIds[row * numDraft + tid]; + sPTarget[tid] = params.targetProbs[(row * (numDraft + 1) + tid) * vocabSize + token]; + sPDraft[tid] = params.draftProbs[(row * numDraft + tid) * vocabSize + token]; + } + __syncthreads(); + if (tid == 0) + { + curandStatePhilox4_32_10_t state; + curand_init(static_cast(params.seed[row]), + kPerRowSubsequenceBase | static_cast(params.offset[row]), 0, &state); + int32_t pos = numDraft; + for (int32_t i = 0; i < numDraft; ++i) + { + float const pTarget = sPTarget[i]; + float const pDraft = sPDraft[i]; + if (!(curand_uniform(&state) * pDraft < pTarget)) + { + pos = i; + break; + } + } + sPos = pos; + // curand_uniform is in (0, 1]; the inverse-CDF search below wants [0, 1). + sU = 1.0f - curand_uniform(&state); + } + __syncthreads(); + + int32_t const pos = sPos; + float const* target = params.targetProbs + (row * (numDraft + 1) + pos) * vocabSize; + float const* draft = pos < numDraft ? params.draftProbs + (row * numDraft + pos) * vocabSize : nullptr; + + // 2. The stock per-thread (target, residual) chains, spread over the request's CTAs; one CTA reduces them. + int32_t const chainsPerCta = kBlockSize / G; + if (tid < chainsPerCta) + { + int32_t const t = cta * chainsPerCta + tid; + // x: target mass, y: residual mass. + float2 sums = make_float2(0.0f, 0.0f); +#pragma unroll 8 + for (uint32_t base = t * kVecSize; base < vocabSize; base += kTileSize) + { + float tv[kVecSize]; + loadProbs(target, base, vocabSize, tv); + float dv[kVecSize] = {}; + if (draft != nullptr) + { + loadProbs(draft, base, vocabSize, dv); + } +#pragma unroll + for (int32_t j = 0; j < kVecSize; ++j) + { + sums.x += tv[j]; + sums.y += fmaxf(tv[j] - dv[j], 0.0f); + } + } + w->chain[t] = sums; + } + rowSync(w, G, + [&] + { + float2 const blockSums + = BlockReducePair(tempStorage.reducePair).Reduce(__ldcg(&w->chain[tid]), Float2SumOp{}); + if (tid == 0) + { + int32_t const targetOnly = draft == nullptr || !(blockSums.y > 0.0f); + w->targetOnly = targetOnly; + w->total = targetOnly ? blockSums.x : blockSums.y; + } + }); + bool const targetOnly = __ldcg(&w->targetOnly) != 0; + + // 3. Tile sums (the stock per-tile cub reduce) and each tile's last index with mass (the stock fallback takes the + // max over the tiles it visited, i.e. up to and including the tile where the threshold falls). + int32_t const numTiles = static_cast((vocabSize + kTileSize - 1) / kTileSize); + for (int32_t tile = cta; tile < numTiles; tile += G) + { + if (tid == 0) + { + sTileLast = -1; + } + __syncthreads(); + uint32_t const base = tile * kTileSize + tid * kVecSize; + float value[kVecSize]; + loadResidual(target, draft, targetOnly, base, vocabSize, value); + int32_t localLast = -1; +#pragma unroll + for (int32_t j = 0; j < kVecSize; ++j) + { + if (value[j] > 0.0f) + { + localLast = static_cast(base + j); + } + } + if (localLast >= 0) + { + atomicMax(&sTileLast, localLast); + } + float const tileSum = BlockReduce(tempStorage.reduce).Sum(value); + __syncthreads(); + if (tid == 0) + { + w->tileSum[tile] = tileSum; + w->tileLast[tile] = sTileLast; + } + __syncthreads(); + } + + // 4. One CTA: the tile prefix in tile order, then the stock in-tile scan; it writes the outputs. + rowSync(w, G, + [&] + { + float const threshold = sU * __ldcg(&w->total); + // The per-tile values in shared memory, read by numTiles threads in parallel. + __shared__ float sTileSum[kMaxTiles]; + __shared__ int32_t sTileLastOf[kMaxTiles]; + if (tid < numTiles) + { + sTileSum[tid] = __ldcg(&w->tileSum[tid]); + sTileLastOf[tid] = __ldcg(&w->tileLast[tid]); + } + __syncthreads(); + if (tid == 0) + { + float prefix = 0.0f; + int32_t found = -1; + for (int32_t tile = 0; tile < numTiles; ++tile) + { + float const tileSum = sTileSum[tile]; + if (prefix + tileSum > threshold) + { + found = tile; + break; + } + prefix += tileSum; + } + sTile = found; + sPrefix = prefix; + sWinner = static_cast(vocabSize); + } + __syncthreads(); + if (sTile >= 0) + { + uint32_t const base = sTile * kTileSize + tid * kVecSize; + float value[kVecSize]; + loadResidual(target, draft, targetOnly, base, vocabSize, value); + float inclusive[kVecSize]; + BlockScan(tempStorage.scan).InclusiveSum(value, inclusive); + float const prefix = sPrefix; +#pragma unroll + for (int32_t j = 0; j < kVecSize; ++j) + { + if (value[j] > 0.0f && prefix + inclusive[j] > threshold) + { + atomicMin(&sWinner, static_cast(base + j)); + break; + } + } + } + __syncthreads(); + if (tid == 0) + { + // Float rounding can leave the summed prefix just short of the threshold; the last token with mass + // (over the tiles the stock loop visits) is then the one the threshold falls on. + int32_t const upto = sTile >= 0 ? sTile : numTiles - 1; + int32_t lastValid = -1; + for (int32_t tile = 0; tile <= upto; ++tile) + { + lastValid = max(lastValid, sTileLastOf[tile]); + } + int32_t const token = sWinner < static_cast(vocabSize) ? sWinner : max(lastValid, 0); + for (int32_t i = 0; i < pos; ++i) + { + outTokens[i] = params.draftTokenIds[row * numDraft + i]; + } + outTokens[pos] = token; + for (int32_t i = pos + 1; i <= numDraft; ++i) + { + outTokens[i] = -1; + } + params.outputNumTokens[row] = pos + 1; + } + }); +} + +} // namespace + +cudaError_t invokeChainSpecSamplingMc( + ChainSpeculativeSamplingParams const& params, int ctasPerRow, void* workspace, cudaStream_t stream) +{ + if (params.batchSize <= 0) + { + return cudaSuccess; + } + if (ctasPerRow < 1 || ctasPerRow > kBlockSize || kBlockSize % ctasPerRow != 0 || params.vocabSize <= 0 + || params.vocabSize % kVecSize != 0 || params.vocabSize > kChainSpecMcMaxVocab || params.numDraftTokens < 0 + || params.numDraftTokens > kChainSpecMcMaxDraft || params.batchSize > 65535) + { + return cudaErrorInvalidValue; + } + cudaLaunchConfig_t cfg{}; + cfg.gridDim = dim3(static_cast(ctasPerRow), static_cast(params.batchSize)); + cfg.blockDim = dim3(kBlockSize); + cfg.dynamicSmemBytes = 0; + cfg.stream = stream; + return cudaLaunchKernelEx(&cfg, chainSpecSamplingMcKernel, params, static_cast(workspace)); +} + +size_t chainSpecSamplingMcWorkspaceBytesPerRow() +{ + return sizeof(RowWorkspace); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h b/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h new file mode 100644 index 000000000000..33a37dc071c7 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h @@ -0,0 +1,84 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" +#include "tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! The widest vocab invokeChainSpecSamplingMc takes (64 tiles of 4096). +constexpr int kChainSpecMcMaxVocab = 64 * 4096; + +//! The most draft tokens per request invokeChainSpecSamplingMc takes (its acceptance probabilities sit in shared +//! memory). +constexpr int kChainSpecMcMaxDraft = 32; + +//! ST-01-REJ: chain speculative sampling (kernels/speculativeDecoding/chainSpeculativeSamplingKernels.cu, the +//! VEC_SIZE 4 kernel behind trtllm::chain_speculative_sampling) as ONE launch of batch x ctasPerRow CTAs (1024 threads +//! each) instead of one CTA per request. +//! +//! Semantics: the stock kernel's (see ChainSpeculativeSamplingParams): per request, the Philox acceptance loop, then +//! the corrected (or bonus) token by an inverse-CDF search over max(0, target - draft) (or the target), the outputs +//! [batch, numDraft + 1] tokens and [batch] counts. +//! +//! Bitwise contract vs the stock kernel (tokens and counts, every input of its vectorized path): +//! 1. every CTA of a request runs the stock tid-0 acceptance loop unchanged (same seed, subsequence +//! kPerRowSubsequenceBase | offset, same draws), so all agree on the rejected position and the uniform; +//! 2. the (target, residual) mass: the stock kernel's 1024 per-thread chains (thread t adds the elements at +//! 4t + 4096k in k order) are spread over the request's CTAs unchanged, and one CTA reduces them with the same +//! cub BlockReduce and Float2SumOp; +//! 3. each 4096-element tile's sum is the stock cub BlockReduce::Sum over the same per-thread vec4, and +//! each tile's last index with mass is a max (order-free); +//! 4. one CTA walks the tile sums in tile order with the stock fp32 prefix adds, and in the tile where the threshold +//! falls runs the stock cub BlockScan::InclusiveSum and atomicMin search; the fallback (no winner) is +//! the last index with mass over the tiles the stock loop visits. +//! Validated bitwise (tokens + counts) vs trtllm::chain_speculative_sampling on GB300, K7, vocab 200064, batch +//! 1 / 2 / 4 / 8 x 5 cases (realistic, same, far, ties, onehot) x 64 seeds at ctasPerRow 4 / 8 / 16, eager and graph +//! (microbenchmark): 31.2 vs 41.2 us per call at batch 1 with ctasPerRow 16. +//! +//! Compiled with the repo's default flags, as the stock kernel (only adds, max and compares on the probs; no +//! --use_fast_math). +//! +//! Contract: vocabSize % 4 == 0, vocabSize <= kChainSpecMcMaxVocab, draftProbs / targetProbs 16-byte aligned (the +//! stock kernel's VEC_SIZE 4 path); 1024 % ctasPerRow == 0. The request's CTAs synchronize through global-memory +//! barriers, so all batch x ctasPerRow CTAs must be co-resident (one 1024-thread CTA per SM): pick ctasPerRow with +//! batch x ctasPerRow <= the SM count (ctasPerRow 1 needs no co-residency). +//! +//! Workspace: chainSpecSamplingMcWorkspaceBytesPerRow() x batch bytes, 16-byte aligned, ZERO-initialized once before +//! the first launch and then reused as is: every launch leaves it ready for the next (arrival counters reset, the +//! barrier generation only increases), so it is CUDA-graph safe without an init kernel. Launches that share one +//! workspace must be stream-ordered. +cudaError_t invokeChainSpecSamplingMc(speculative_decoding::ChainSpeculativeSamplingParams const& params, + int ctasPerRow, void* workspace, cudaStream_t stream); + +//! Workspace bytes per request of invokeChainSpecSamplingMc. +size_t chainSpecSamplingMcWorkspaceBytesPerRow(); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.cu b/cpp/tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.cu new file mode 100644 index 000000000000..9627b1ce8cd0 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.cu @@ -0,0 +1,232 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.h" +#include "tensorrt_llm/kernels/quantization.cuh" + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::minimaxM31Sm103 +{ +namespace +{ + +constexpr int kPackedSize = 8; // BF16 elements per 16-byte access, = CVT_ELTS_PER_THREAD +constexpr int kWarpSize = 32; + +union PackedBf16 +{ + int4 packed; + __nv_bfloat162 unpacked[4]; +}; + +// The helpers below are customAllReduceKernels.cu's (reduce_fusion::warp_reduce_sum / block_reduce_sum, +// accumulate, rms_norm, add128b): the same arithmetic in the same order, so the norm is bitwise +// rms_norm_rows_kernel's. +__device__ __forceinline__ float warpReduceSum(float val) +{ + val += __shfl_xor_sync(~0, val, 16); + val += __shfl_xor_sync(~0, val, 8); + val += __shfl_xor_sync(~0, val, 4); + val += __shfl_xor_sync(~0, val, 2); + val += __shfl_xor_sync(~0, val, 1); + return val; +} + +__device__ __forceinline__ float blockReduceSum(float val) +{ + __shared__ float smem[kWarpSize]; + int const laneId = threadIdx.x % kWarpSize; + int const warpId = threadIdx.x / kWarpSize; + int const warpNum = blockDim.x / kWarpSize; + val = warpReduceSum(val); + if (laneId == 0) + { + smem[warpId] = val; + } + __syncthreads(); + val = laneId < warpNum ? smem[laneId] : 0.f; + val = warpReduceSum(val); + return val; +} + +__device__ __forceinline__ int4 add128b(PackedBf16& a, PackedBf16& b) +{ + PackedBf16 c; + c.unpacked[0] = a.unpacked[0] + b.unpacked[0]; + c.unpacked[1] = a.unpacked[1] + b.unpacked[1]; + c.unpacked[2] = a.unpacked[2] + b.unpacked[2]; + c.unpacked[3] = a.unpacked[3] + b.unpacked[3]; + return c.packed; +} + +__device__ __forceinline__ float accumulate(float acc, PackedBf16& vec) +{ +#pragma unroll + for (int i = 0; i < kPackedSize; ++i) + { + float v = static_cast(reinterpret_cast<__nv_bfloat16*>(vec.unpacked)[i]); + acc += v * v; + } + return acc; +} + +__device__ __forceinline__ int4 rmsNorm(float denom, PackedBf16& vec, PackedBf16& weight) +{ + PackedBf16 ret; +#pragma unroll + for (int i = 0; i < kPackedSize; ++i) + { + float v1 = static_cast(reinterpret_cast<__nv_bfloat16*>(vec.unpacked)[i]); + float v2 = static_cast(reinterpret_cast<__nv_bfloat16*>(weight.unpacked)[i]); + reinterpret_cast<__nv_bfloat16*>(ret.unpacked)[i] = static_cast<__nv_bfloat16>(v1 * denom * v2); + } + return ret.packed; +} + +//! rms_norm_rows_kernel (persistent rows, next row prefetched, +//! one 8-element chunk per thread) whose per-row output also goes through quantize_mxfp8_rows' per-element math +//! (cvt_warp_fp16_to_mxfp8, swizzled K32 scales) while it is still in registers. The scale rows that pad +//! the row count to a multiple of 128 are zero-filled, as quantize_mxfp8_rows does. +template +__global__ void ctxResidualRmsNormMxfp8Kernel(__nv_bfloat16* inter, __nv_bfloat16 const* residual, + __nv_bfloat16 const* weight, __nv_bfloat16* normOut, uint64_t* qOut, uint32_t* sfOut, uint32_t* fp4Out, + uint32_t* fp4SfOut, float const* fp4GlobalScale, int rows, int hidden, float eps) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + static constexpr int kThreadsPerSf = 32 / CVT_ELTS_PER_THREAD; + int const colIdx = threadIdx.x; + int const offset = colIdx * kPackedSize; + int const colThreads = hidden / kPackedSize; + std::optional const batchIdx = 0; + std::optional const numRows = rows; + + cudaGridDependencySynchronize(); + + // quantize_with_block_size's per-tensor scale (SFScaleVal) for the NVFP4 copy. + float const fp4SfScale = kFp4 ? fp4GlobalScale[0] : 1.0f; + PackedBf16 weightVec; + weightVec.packed = *reinterpret_cast(weight + offset); + PackedBf16 interVec, residualVec; + int row = blockIdx.x; + if (row < rows) + { + interVec.packed = *reinterpret_cast(inter + static_cast(row) * hidden + offset); + residualVec.packed = *reinterpret_cast(residual + static_cast(row) * hidden + offset); + } + for (; row < rows; row += gridDim.x) + { + int64_t const rowOffset = static_cast(row) * hidden + offset; + int const next = row + gridDim.x; + PackedBf16 nextInterVec, nextResidualVec; + if (next < rows) + { + nextInterVec.packed = *reinterpret_cast(inter + static_cast(next) * hidden + offset); + nextResidualVec.packed + = *reinterpret_cast(residual + static_cast(next) * hidden + offset); + } + interVec.packed = add128b(interVec, residualVec); + *reinterpret_cast(inter + rowOffset) = interVec.packed; + float acc = accumulate(0.f, interVec); + acc = blockReduceSum(acc); + float const denom = rsqrtf(acc / hidden + eps); + PackedBf16 outVec; + outVec.packed = rmsNorm(denom, interVec, weightVec); + if constexpr (kWriteNorm) + { + *reinterpret_cast(normOut + rowOffset) = outVec.packed; + } + PackedVec<__nv_bfloat16> quantIn; + *reinterpret_cast(&quantIn) = outVec.packed; + auto sf = cvt_quant_get_sf_out_offset( + batchIdx, row, colIdx, numRows, hidden / 32, sfOut, QuantizationSFLayout::SWIZZLED); + qOut[static_cast(row) * colThreads + colIdx] = cvt_warp_fp16_to_mxfp8<__nv_bfloat16, 32>(quantIn, sf); + if constexpr (kFp4) + { + // fp4_quantize(norm, globalScale, 16, ue8m0 = false, swizzled = false): E2M1 x 8 per thread, E4M3 scale per + // 16 elements in the linear layout (no row padding). + PackedVec<__nv_bfloat16> fp4In; + *reinterpret_cast(&fp4In) = outVec.packed; + auto sf4 = cvt_quant_get_sf_out_offset( + batchIdx, row, colIdx, numRows, hidden / 16, fp4SfOut, QuantizationSFLayout::LINEAR); + fp4Out[static_cast(row) * colThreads + colIdx] + = cvt_warp_fp16_to_fp4<__nv_bfloat16, 16, false>(fp4In, fp4SfScale, sf4); + } + // blockReduceSum's shared memory is written again by the next row. + __syncthreads(); + interVec.packed = nextInterVec.packed; + residualVec.packed = nextResidualVec.packed; + } + int const paddedRows = (rows + 127) / 128 * 128; + for (int padRow = rows + blockIdx.x; padRow < paddedRows; padRow += gridDim.x) + { + auto sf = cvt_quant_get_sf_out_offset( + batchIdx, padRow, colIdx, numRows, hidden / 32, sfOut, QuantizationSFLayout::SWIZZLED); + cvt_store_replicated_sf<1>(sf, 0x00); + } + // Every thread's stores are visible before any thread reports PDL completion (as in quantize_mxfp8_rows). + __threadfence(); + __syncthreads(); + cudaTriggerProgrammaticLaunchCompletion(); +#endif +} + +} // namespace + +void invokeCtxResidualRmsNormMxfp8(__nv_bfloat16* inter, __nv_bfloat16 const* residual, __nv_bfloat16 const* weight, + __nv_bfloat16* normOut, uint64_t* qOut, uint32_t* sfOut, uint32_t* fp4Out, uint32_t* fp4SfOut, + float const* fp4GlobalScale, int rows, int hidden, float eps, bool enablePdl, cudaStream_t stream) +{ + static int const multiProcessorCount = [] + { + int device = 0; + int count = 0; + cudaGetDevice(&device); + cudaDeviceGetAttribute(&count, cudaDevAttrMultiProcessorCount, device); + return count; + }(); + cudaLaunchConfig_t config = {0}; + config.gridDim = std::max(1, std::min(rows, multiProcessorCount)); + config.blockDim = hidden / kPackedSize; + config.dynamicSmemBytes = 0; + config.stream = stream; + cudaLaunchAttribute attribute[1]; + attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attribute[0].val.programmaticStreamSerializationAllowed = enablePdl; + config.attrs = attribute; + config.numAttrs = 1; + auto launch = [&](auto kernel) + { + cudaLaunchKernelEx(&config, kernel, inter, residual, weight, normOut, qOut, sfOut, fp4Out, fp4SfOut, + fp4GlobalScale, rows, hidden, eps); + }; + bool const fp4 = fp4Out != nullptr; + if (normOut != nullptr) + { + fp4 ? launch(ctxResidualRmsNormMxfp8Kernel) : launch(ctxResidualRmsNormMxfp8Kernel); + } + else + { + fp4 ? launch(ctxResidualRmsNormMxfp8Kernel) : launch(ctxResidualRmsNormMxfp8Kernel); + } +} + +} // namespace kernels::minimaxM31Sm103 + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.h b/cpp/tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.h new file mode 100644 index 000000000000..9cba469d348a --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.h @@ -0,0 +1,43 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::minimaxM31Sm103 +{ + +//! MiniMax-M3 CTX prefill: the residual add + RMSNorm of the all-reduce's RESIDUAL_RMS_NORM epilogue +//! (rms_norm_rows_kernel, bitwise the same) fused with the MXFP8 quantization of its output that the next MXFP8 +//! linear performs (quantize_mxfp8_rows: K32 UE8M0 scales, swizzled 128x4 layout, bitwise the same). +//! In place: `inter` (the all-reduce output) becomes `inter + residual`, as in the stock epilogue. +//! normOut may be null (the BF16 norm output is then not written). With fp4Out (and fp4SfOut / fp4GlobalScale) set, +//! it also writes fp4_quantize(norm, globalScale, 16, ue8m0 = false, swizzled = false) of the same rows. +//! hidden % 256 == 0, hidden / 8 <= 1024. +void invokeCtxResidualRmsNormMxfp8(__nv_bfloat16* inter, __nv_bfloat16 const* residual, __nv_bfloat16 const* weight, + __nv_bfloat16* normOut, uint64_t* qOut, uint32_t* sfOut, uint32_t* fp4Out, uint32_t* fp4SfOut, + float const* fp4GlobalScale, int rows, int hidden, float eps, bool enablePdl, cudaStream_t stream); + +} // namespace kernels::minimaxM31Sm103 + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.cu b/cpp/tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.cu new file mode 100644 index 000000000000..2a40178ca22c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.cu @@ -0,0 +1,545 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3.1 GB300 (sm_103) target-local copy of the #20629 prefill tile select +// (cpp/tensorrt_llm/kernels/minimaxM3/selectPrefillTile.* @ cf8a77e816); namespace and op name are M3.1's. + +// Prefill top-16 block select over head-major scores [num_kv_heads, num_blocks, total_q] with unit query stride. +// +// One CTA per (KV head, 8 consecutive queries). The 8 queries of one block are one 32-byte sector, so the CTA reads +// each sector it needs exactly once, as two 16-byte loads. The tile's rows are staged in shared memory as ordered keys +// of the forced scores. A long tile's key rows fill most of shared memory, so one CTA runs per SM; its loads then come +// from extra staging warps (kWideThreads from kWideMinBlocks blocks), which exit before the selection. Each of the +// first 8 warps then selects one row: +// - the 32 lane-stride group minima are sorted in the warp and the 16th of them bounds the row's 16th smallest key, +// so every key at or below that bound is a candidate; the candidates (compacted in ascending block order) are +// ranked exactly by (key, block); +// - a row with more than kMaxCand candidates (heavy ties) falls back to an exact 4 x 8-bit radix select. +// Ordering is the descending ordered key of the float bits used by minimax_m3_select_blocks, ties to the smaller +// block, so the result is the same top-16 set. + +#include "tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.h" + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/envUtils.h" +#include "tensorrt_llm/common/logger.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::minimaxM31Sm103 +{ + +namespace +{ +constexpr int kTopK = 16; +constexpr int kTileQ = 8; +constexpr int kThreads = kTileQ * 32; +// From this many blocks on, the tile is staged by kWideThreads threads with kWideUnroll blocks in flight each. +constexpr int kWideMinBlocks = 2600; +constexpr int kWideThreads = 1024; +constexpr int kWideUnroll = 4; +constexpr uint32_t kFull = 0xFFFFFFFFu; +constexpr float kInitScore = 1.0e30f; +constexpr float kLocalScore = 1.0e29f; +constexpr uint32_t kNegInfKey = 0xFF800000u; +// Per-warp scratch: the radix histogram (256 bins) or the candidate list (keys then blocks). +constexpr int kMaxCand = 256; +constexpr int kWarpScratch = 2 * kMaxCand; +// Blocks each thread has in flight while staging a narrow tile. +constexpr int kStageUnroll = 8; + +__device__ __forceinline__ uint32_t orderedKey(float x) +{ + uint32_t const bits = __float_as_uint(x); + return (bits & 0x80000000u) ? bits : (~bits & 0x7fffffffu); +} + +// Row pitch in keys: >= num_blocks and == 4 (mod 32), so the staging writes of +// 8 rows x 4 consecutive blocks per warp hit 32 distinct banks. +__host__ __device__ __forceinline__ int rowPitch(int numBlocks) +{ + int p = (numBlocks + 31) / 32 * 32 + 4; + return p; +} + +// kNt staging threads with kUnroll blocks in flight each; warps 0 .. kTileQ - 1 select. kIlp builds each row's 32 +// stride-class minima while staging (a staging thread only ever sees one class) and issues the candidate scan's +// shared-memory loads four at a time; the minima, the bound and every decision are the same as the scalar scans'. +template +__global__ void __launch_bounds__(kNt) selectPrefillTileKernel(float const* __restrict__ scores, int64_t headStride, + int64_t blockStride, int32_t const* __restrict__ nValid, int32_t* __restrict__ out, int64_t outQueryStride, + int64_t outHeadStride, int numBlocks, int totalQ, int initBlocks, int localBlocks) +{ + extern __shared__ uint32_t smem[]; + __shared__ uint32_t sClassMin[kTileQ * 32]; // [row][b % 32] (kIlp) + int const pitch = rowPitch(numBlocks); + uint32_t* keys = smem; // [kTileQ][pitch] + uint32_t* hist = smem + kTileQ * pitch; // [kTileQ][kWarpScratch] + + int const head = blockIdx.y; + int const q0 = blockIdx.x * kTileQ; + int const tid = threadIdx.x; + int const warp = tid / 32; + int const lane = tid % 32; + +#if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)) + cudaGridDependencySynchronize(); +#endif + + // ---- stage: thread loads queries 4 * jh .. 4 * jh + 3 of block b = tid / 2 + (kNt / 2) * i as one float4 ---- + { + int const jh = tid % 2; + int rowEnd[4]; + int64_t localStart[4]; + int tileEnd = 0; +#pragma unroll + for (int c = 0; c < 4; ++c) + { + int const q = q0 + 4 * jh + c; + rowEnd[c] = 0; + localStart[c] = 0; + if (q < totalQ) + { + int const raw = nValid[q]; + rowEnd[c] = max(0, min(raw, numBlocks)); + localStart[c] = max(static_cast(raw) - localBlocks, static_cast(0)); + } + tileEnd = max(tileEnd, rowEnd[c]); + } + tileEnd = max(tileEnd, __shfl_xor_sync(kFull, tileEnd, 1)); + if constexpr (kIlp) + { + if (tid < kTileQ * 32) + { + sClassMin[tid] = 0xFFFFFFFFu; + } + } + __shared__ int sTileEnd[kNt / 32]; + if (lane == 0) + { + sTileEnd[warp] = tileEnd; + } + __syncthreads(); + tileEnd = 0; +#pragma unroll + for (int w = 0; w < kNt / 32; ++w) + { + tileEnd = max(tileEnd, sTileEnd[w]); + } + // Two threads read a tile's 8 queries of one block, one aligned 32-byte sector, when the base and the + // strides keep 16-byte alignment; otherwise, and for a partial last tile, they read one float at a time. + bool const fullVec = (reinterpret_cast(scores) & 15) == 0 && ((headStride | blockStride) & 3) == 0 + && (q0 + 4 * jh + 3) < totalQ; + float const* base = scores + static_cast(head) * headStride + q0 + 4 * jh; + constexpr int kBlocksPerIter = kNt / 2; + static_assert(kBlocksPerIter % 32 == 0, "a staging thread stays in one stride class b % 32"); + uint32_t classMin[4] = {0xFFFFFFFFu, 0xFFFFFFFFu, 0xFFFFFFFFu, 0xFFFFFFFFu}; + for (int b0 = tid / 2; b0 < tileEnd; b0 += kBlocksPerIter * kUnroll) + { + float4 v[kUnroll]; +#pragma unroll + for (int u = 0; u < kUnroll; ++u) + { + int const b = b0 + u * kBlocksPerIter; + float4 x = make_float4(0.f, 0.f, 0.f, 0.f); + if (b < tileEnd) + { + float const* p = base + static_cast(b) * blockStride; + if (fullVec) + { + x = __ldg(reinterpret_cast(p)); + } + else + { + x.x = b < rowEnd[0] ? __ldg(p) : 0.f; + x.y = b < rowEnd[1] ? __ldg(p + 1) : 0.f; + x.z = b < rowEnd[2] ? __ldg(p + 2) : 0.f; + x.w = b < rowEnd[3] ? __ldg(p + 3) : 0.f; + } + } + v[u] = x; + } +#pragma unroll + for (int u = 0; u < kUnroll; ++u) + { + int const b = b0 + u * kBlocksPerIter; + if (b < tileEnd) + { + float const vals[4] = {v[u].x, v[u].y, v[u].z, v[u].w}; +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint32_t key = kNegInfKey; + if (b < rowEnd[c]) + { + float s = vals[c]; + if (b < initBlocks) + { + s = kInitScore; + } + if (b >= localStart[c]) + { + s = kLocalScore; + } + key = orderedKey(s); + if constexpr (kIlp) + { + classMin[c] = min(classMin[c], key); + } + } + keys[(4 * jh + c) * pitch + b] = key; + } + } + } + } + if constexpr (kIlp) + { + int const cls = (tid / 2) % 32; +#pragma unroll + for (int c = 0; c < 4; ++c) + { + if (classMin[c] != 0xFFFFFFFFu) + { + atomicMin(&sClassMin[(4 * jh + c) * 32 + cls], classMin[c]); + } + } + } + } + __syncthreads(); + if (warp >= kTileQ) + { + return; + } + + // ---- select: warp w owns query q0 + w ---- + int const q = q0 + warp; + if (q >= totalQ) + { + return; + } + int const raw = nValid[q]; + int const len = max(0, min(raw, numBlocks)); + uint32_t const* row = keys + warp * pitch; + uint32_t* h = hist + warp * kWarpScratch; + int32_t* dst = out + static_cast(q) * outQueryStride + static_cast(head) * outHeadStride; + + // Threshold key K and how many keys equal to K to take (tieTake). Keys < K are all in. + if (len > kTopK) + { + // Fast path. Group g = lanes' stride classes b % 32: each lane's best (smallest) key. + // T = the 16th smallest of the 32 group bests, so at least 16 keys are <= T and the + // row's 16th smallest key is <= T: every top-16 key (and every key tied with one) is a + // candidate. Candidates are compacted in ascending block order and ranked exactly. + uint32_t best = 0xFFFFFFFFu; + bool const any = lane < len; + if constexpr (kIlp) + { + best = sClassMin[warp * 32 + lane]; + } + else + { + for (int b = lane; b < len; b += 32) + { + best = min(best, row[b]); + } + } + // Ascending bitonic sort of the 32 group bests; empty groups sort last. + uint32_t v = any ? best : 0xFFFFFFFFu; + bool vEmpty = !any; +#pragma unroll + for (int size = 2; size <= 32; size *= 2) + { +#pragma unroll + for (int stride = size / 2; stride > 0; stride /= 2) + { + uint32_t const pv = __shfl_xor_sync(kFull, v, stride); + bool const pe = __shfl_xor_sync(kFull, vEmpty, stride); + bool const up = ((lane & size) == 0) == ((lane & stride) == 0); + // order: (empty, value); smaller first when 'up' + bool const pLess = (!pe && vEmpty) || (pe == vEmpty && pv < v); + bool const pGreater = (pe && !vEmpty) || (pe == vEmpty && pv > v); + if ((up && pLess) || (!up && pGreater)) + { + v = pv; + vEmpty = pe; + } + } + } + uint32_t const bound = __shfl_sync(kFull, v, kTopK - 1); + uint32_t* candKey = h; + uint32_t* candBlk = h + kMaxCand; + int nCand = 0; + int cb0 = 0; + if constexpr (kIlp) + { + for (; cb0 + 128 <= len; cb0 += 128) + { + uint32_t k[4]; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + k[j] = row[cb0 + 32 * j + lane]; + } +#pragma unroll + for (int j = 0; j < 4; ++j) + { + bool const c = k[j] <= bound; + uint32_t const m = __ballot_sync(kFull, c); + int const slot = nCand + __popc(m & ((1u << lane) - 1u)); + if (c && slot < kMaxCand) + { + candKey[slot] = k[j]; + candBlk[slot] = static_cast(cb0 + 32 * j + lane); + } + nCand += __popc(m); + } + } + } + for (int b0 = cb0; b0 < len; b0 += 32) + { + int const b = b0 + lane; + bool const c = b < len && row[b] <= bound; + uint32_t const m = __ballot_sync(kFull, c); + int const slot = nCand + __popc(m & ((1u << lane) - 1u)); + if (c && slot < kMaxCand) + { + candKey[slot] = row[b]; + candBlk[slot] = static_cast(b); + } + nCand += __popc(m); + } + __syncwarp(); + if (nCand <= kMaxCand) + { + // Exact rank by (key, block); candidates are already in ascending block order, so a + // second ballot compaction of the rank < 16 ones writes the output ascending. + int written = 0; + for (int i0 = 0; i0 < nCand; i0 += 32) + { + int const i = i0 + lane; + bool sel = false; + uint32_t k = 0xFFFFFFFFu; + uint32_t blk = 0; + if (i < nCand) + { + k = candKey[i]; + blk = candBlk[i]; + int rank = 0; + for (int jdx = 0; jdx < nCand; ++jdx) + { + uint32_t const kj = candKey[jdx]; + rank += (kj < k) || (kj == k && jdx < i); + } + sel = rank < kTopK; + } + uint32_t const m = __ballot_sync(kFull, sel && k != kNegInfKey); + int const slot = written + __popc(m & ((1u << lane) - 1u)); + if (sel && k != kNegInfKey) + { + dst[slot] = static_cast(blk); + } + written += __popc(m); + } + for (int r = written + lane; r < kTopK; r += 32) + { + dst[r] = -1; + } + return; + } + __syncwarp(); + } + + uint32_t thresh = 0xFFFFFFFFu; + int tieTake = kTopK; + if (len > kTopK) + { + uint32_t prefix = 0; + int remaining = kTopK; // rank (1-based) of the wanted key among keys matching prefix +#pragma unroll 1 + for (int pass = 0; pass < 4; ++pass) + { + int const shift = 24 - 8 * pass; + uint32_t const hiMask = pass == 0 ? 0u : (0xFFFFFFFFu << (shift + 8)); + for (int i = lane; i < 256; i += 32) + { + h[i] = 0; + } + __syncwarp(); + for (int b0 = 0; b0 < len; b0 += 32) + { + int const b = b0 + lane; + bool const active = b < len; + uint32_t const k = active ? row[b] : 0u; + bool const match = active && ((k & hiMask) == prefix); + uint32_t const digit = (k >> shift) & 0xFFu; + uint32_t const m = __ballot_sync(kFull, match); + if (match) + { + uint32_t const peers = __match_any_sync(m, digit); + if ((__ffs(peers) - 1) == lane) + { + atomicAdd(&h[digit], static_cast(__popc(peers))); + } + } + } + __syncwarp(); + // Find the digit holding the remaining-th smallest key: each lane owns 8 bins. + uint32_t c[8]; + uint32_t local = 0; +#pragma unroll + for (int i = 0; i < 8; ++i) + { + c[i] = h[lane * 8 + i]; + local += c[i]; + } + uint32_t incl = local; +#pragma unroll + for (int o = 1; o < 32; o *= 2) + { + uint32_t const v = __shfl_up_sync(kFull, incl, o); + if (lane >= o) + { + incl += v; + } + } + uint32_t const excl = incl - local; + bool const mine = excl < static_cast(remaining) && incl >= static_cast(remaining); + uint32_t const owner = __ballot_sync(kFull, mine); + int const ownerLane = __ffs(owner) - 1; + int digitSel = 0; + uint32_t below = 0; + if (lane == ownerLane) + { + uint32_t acc = excl; +#pragma unroll + for (int i = 0; i < 8; ++i) + { + if (acc + c[i] >= static_cast(remaining)) + { + digitSel = lane * 8 + i; + below = acc; + break; + } + acc += c[i]; + } + } + digitSel = __shfl_sync(kFull, digitSel, ownerLane); + below = __shfl_sync(kFull, below, ownerLane); + prefix |= static_cast(digitSel) << shift; + remaining -= static_cast(below); + __syncwarp(); + } + thresh = prefix; + tieTake = remaining; // keys equal to thresh to take, smallest blocks first + } + + // ---- collect in ascending block order ---- + // len <= 16: every valid block is a pick (thresh = max, tieTake = 16 covers all). + int count = 0; + int ties = 0; + for (int b0 = 0; b0 < len && count < kTopK; b0 += 32) + { + int const b = b0 + lane; + bool const active = b < len; + uint32_t const k = active ? row[b] : 0xFFFFFFFFu; + bool const strict = active && (len <= kTopK || k < thresh); + bool const tie = active && len > kTopK && k == thresh; + uint32_t const tieMask = __ballot_sync(kFull, tie); + int const tieRank = ties + __popc(tieMask & ((1u << lane) - 1u)); + bool const take = strict || (tie && tieRank < tieTake); + uint32_t const takeMask = __ballot_sync(kFull, take); + int const slot = count + __popc(takeMask & ((1u << lane) - 1u)); + if (take && slot < kTopK) + { + dst[slot] = (k == kNegInfKey) ? -1 : b; + } + count += __popc(takeMask); + ties += __popc(tieMask); + } + __syncwarp(); + // Picks forced to -inf were written as -1 in place; compact the valid ids to the front, -1 tail. + if (lane == 0) + { + int32_t ids[kTopK]; + int n = 0; + int const picks = min(count, kTopK); + for (int i = 0; i < picks; ++i) + { + int32_t const v = dst[i]; + if (v >= 0) + { + ids[n++] = v; + } + } + for (int i = 0; i < kTopK; ++i) + { + dst[i] = i < n ? ids[i] : -1; + } + } +} +} // namespace + +cudaError_t invokeMinimaxM3SelectPrefillTile(float const* scores, int64_t headStride, int64_t blockStride, + int32_t const* nValidBlocks, int32_t* out, int64_t outQueryStride, int64_t outHeadStride, int32_t numKvHeads, + int32_t numBlocks, int32_t totalQ, int32_t initBlocks, int32_t localBlocks, cudaStream_t stream) +{ + if (totalQ == 0 || numKvHeads == 0) + { + return cudaSuccess; + } + if (numBlocks > kMinimaxM3SelectPrefillTileMaxBlocks) + { + return cudaErrorInvalidValue; + } + // TRTLLM_M3_SELECT_TILE_V2=1 keeps the previous launch: 256 threads for every tile, one load at a time per scan. + static bool const v2Only = common::getBoolEnv("TRTLLM_M3_SELECT_TILE_V2"); + static std::once_flag logged; + std::call_once(logged, + [] + { + TLLM_LOG_INFO(v2Only + ? "minimaxM3SelectPrefillTile: staging reads each block's sector as two 16-byte loads" + : "minimaxM3SelectPrefillTile: v4, stride-class minima while staging, 4-wide candidate " + "scan and 1024 staging threads from 2600 blocks"); + }); + size_t const smemBytes + = (static_cast(kTileQ) * rowPitch(numBlocks) + kTileQ * kWarpScratch) * sizeof(uint32_t); + bool const wide = !v2Only && numBlocks >= kWideMinBlocks; + auto kernel = v2Only ? selectPrefillTileKernel + : wide ? selectPrefillTileKernel + : selectPrefillTileKernel; + cudaError_t err + = cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smemBytes)); + if (err != cudaSuccess) + { + return err; + } + dim3 const grid((totalQ + kTileQ - 1) / kTileQ, numKvHeads); + common::launchWithPdlWhenEnabled("minimaxM3SelectPrefillTile", kernel, grid, dim3(wide ? kWideThreads : kThreads), + smemBytes, stream, scores, headStride, blockStride, nValidBlocks, out, outQueryStride, outHeadStride, numBlocks, + totalQ, initBlocks, localBlocks); + return cudaGetLastError(); +} + +} // namespace kernels::minimaxM31Sm103 + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.h b/cpp/tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.h new file mode 100644 index 000000000000..0e5d5cc030af --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.h @@ -0,0 +1,51 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3.1 GB300 (sm_103) target-local copy of the #20629 prefill tile select +// (cpp/tensorrt_llm/kernels/minimaxM3/selectPrefillTile.* @ cf8a77e816); namespace and op name are M3.1's. + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::minimaxM31Sm103 +{ + +//! Largest row (blocks per query) the prefill tile select stages in shared memory: 8 query rows of keys plus the +//! per-warp scratch stay under 200 KB. +constexpr int32_t kMinimaxM3SelectPrefillTileMaxBlocks = 5856; + +//! MiniMax-M3 top-16 block select for prefill: per-KV-head scores [numKvHeads, numBlocks, totalQ] with unit query +//! stride, one CTA per (KV head, 8 consecutive queries) so every 32-byte sector of the scores is read once. +//! +//! Same contract as minimax_m3_select_blocks: per (query, head) row the clamp(nValidBlocks[query], 0, numBlocks) +//! leading blocks; the last localBlocks of them (counted from the unclamped count) forced to 1e29 and the first +//! initBlocks to 1e30, local winning; ranked by the descending ordered key of the float bits (NaN above +inf, +0 above +//! -0, negative NaN below -inf), ties to the smaller block; picks whose forced score is -inf emitted as -1; ascending +//! ids with -1 tail padding. Writes out[query * outQueryStride + head * outHeadStride + 0..15]. +cudaError_t invokeMinimaxM3SelectPrefillTile(float const* scores, int64_t headStride, int64_t blockStride, + int32_t const* nValidBlocks, int32_t* out, int64_t outQueryStride, int64_t outHeadStride, int32_t numKvHeads, + int32_t numBlocks, int32_t totalQ, int32_t initBlocks, int32_t localBlocks, cudaStream_t stream); + +} // namespace kernels::minimaxM31Sm103 + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.cu b/cpp/tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.cu new file mode 100644 index 000000000000..6fc16f1d1147 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.cu @@ -0,0 +1,295 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// ST-01-SM: FlashInfer softmax(logits, temperature) Path A with the Reduce kernel's +// write pass spread over many CTAs per row. See softmaxMcSm103.h for the contract. Compiled with --use_fast_math (set +// on this source in kernels/CMakeLists.txt), as FlashInfer's JIT compiles it. + +#include "tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.h" + +#include +#include + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +namespace +{ + +// --------------------------------------------------------------------------------------------------------------------- +// Vendored from FlashInfer (v0.6.15; identical in v0.6.18): include/flashinfer/utils.cuh (ceil_div, round_up), +// include/flashinfer/sampling.cuh (MaxReduceOp, Float2SoftmaxReduceOp, PartialSoftmaxResult) and +// include/flashinfer/vec_dtypes.cuh (vec_t: only the members the kernels below use; cast_load / +// cast_store are the same-type branch of FlashInfer's cast_load_impl / cast_store_impl, i.e. load / store). +// Copyright (c) 2023-2026 by FlashInfer team, licensed under the Apache License, Version 2.0. Copied verbatim +// (formatting kept) so that the element order, the float4 accesses and the reduction operators are FlashInfer's own; +// FlashInfer's headers are not on the TensorRT-LLM C++ include path. +// --------------------------------------------------------------------------------------------------------------------- +namespace fi +{ +// clang-format off +#define FLASHINFER_INLINE inline __attribute__((always_inline)) __device__ + +template +__forceinline__ __device__ __host__ constexpr T1 ceil_div(const T1 x, const T2 y) noexcept { + return (x + y - 1) / y; +} + +template +__forceinline__ __device__ __host__ constexpr T1 round_up(const T1 x, const T2 y) noexcept { + return ceil_div(x, y) * y; +} + +using MaxReduceOp = cuda::maximum<>; + +struct Float2SoftmaxReduceOp { + __device__ __forceinline__ float2 operator()(const float2& a, const float2& b) const { + if (isinf(a.x)) return b; + if (isinf(b.x)) return a; + + float new_max = max(a.x, b.x); + float new_denom = a.y * __expf(a.x - new_max) + b.y * __expf(b.x - new_max); + return make_float2(new_max, new_denom); + } +}; + +struct PartialSoftmaxResult { + float max_val; + float denominator; +}; + +template +struct vec_t; + +// float x 4 or more +template +struct vec_t { + static_assert(vec_size % 4 == 0, "Invalid vector size"); + float4 data[vec_size / 4]; + + FLASHINFER_INLINE float& operator[](size_t i) { return ((float*)(data))[i]; } + FLASHINFER_INLINE const float& operator[](size_t i) const { return ((const float*)(data))[i]; } + FLASHINFER_INLINE void fill(float val) { +#pragma unroll + for (size_t i = 0; i < vec_size / 4; ++i) { + data[i] = make_float4(val, val, val, val); + } + } + FLASHINFER_INLINE void load(const float* ptr) { +#pragma unroll + for (size_t i = 0; i < vec_size / 4; ++i) { + data[i] = ((float4*)ptr)[i]; + } + } + FLASHINFER_INLINE void store(float* ptr) const { +#pragma unroll + for (size_t i = 0; i < vec_size / 4; ++i) { + ((float4*)ptr)[i] = data[i]; + } + } + template + FLASHINFER_INLINE void cast_load(const T* ptr) { + static_assert(std::is_same_v, "only the same-type branch of cast_load_impl is vendored"); + load(ptr); + } + template + FLASHINFER_INLINE void cast_store(T* ptr) const { + static_assert(std::is_same_v, "only the same-type branch of cast_store_impl is vendored"); + store(ptr); + } +}; + +#undef FLASHINFER_INLINE +// clang-format on +} // namespace fi + +// FlashInfer's OnlineSoftmax on sm_80+ (DISPATCH_COMPUTE_CAP_NUM_THREADS) and vec_size = gcd(16 / sizeof(float), d) +// = 4 for d % 4 == 0. +constexpr uint32_t kThreads = 1024; +constexpr uint32_t kVec = 4; + +// FlashInfer OnlineSoftmaxMapKernel<1024, 4, float>: the same body (the shared temp storage is static instead of +// FlashInfer's dynamic OnlineSoftmaxTempStorage union; no PDL griddepcontrol, as the launches here are plain). +__global__ void __launch_bounds__(kThreads) softmaxMcMapKernel(float const* logits, fi::PartialSoftmaxResult* partials, + float const* temperature_arr, float temperature_val, uint32_t d, uint32_t num_slices) +{ + // clang-format off + const uint32_t bx = blockIdx.x; + const uint32_t by = blockIdx.y; // slice index + const uint32_t tx = threadIdx.x; + float temperature = temperature_arr == nullptr ? temperature_val : temperature_arr[bx]; + const float inv_temp = (temperature == 0.f) ? 0.f : 1.f / temperature; + const uint32_t vec_alignment_elems = alignof(fi::vec_t) / sizeof(float); + const uint32_t slice_stride = fi::round_up(fi::ceil_div(d, num_slices), vec_alignment_elems); + const uint32_t slice_start = by * slice_stride; + const uint32_t slice_size = min((by + 1) * slice_stride, d) - slice_start; + if (slice_start >= d) return; + __shared__ typename cub::BlockReduce::TempStorage reduce; + __shared__ float s_max, s_den; + fi::vec_t logits_vec; + float running_max = -cuda::std::numeric_limits::infinity(); + float running_denominator = 0.0f; + float threadlocal_running_denominator = 0.0f; +#pragma unroll 2 + for (uint32_t i = 0; i < fi::ceil_div(slice_size, kThreads * kVec); ++i) { + logits_vec.fill(-cuda::std::numeric_limits::infinity()); + if ((i * kThreads + tx) * kVec < slice_size) { + logits_vec.cast_load(logits + bx * d + slice_start + (i * kThreads + tx) * kVec); + } + float thread_max = -cuda::std::numeric_limits::infinity(); +#pragma unroll + for (uint32_t j = 0; j < kVec; ++j) { + logits_vec[j] *= inv_temp; + thread_max = max(thread_max, logits_vec[j]); + } + float block_max = cub::BlockReduce(reduce).Reduce(thread_max, fi::MaxReduceOp{}); + if (tx == 0) s_max = block_max; + __syncthreads(); + block_max = s_max; + // if block_max is -inf, then this block contains all -inf values, so we can skip updating + if (!isinf(block_max)) { + float threadlocal_sum = 0.0f; +#pragma unroll + for (uint32_t j = 0; j < kVec; ++j) { + threadlocal_sum += __expf(logits_vec[j] - block_max); + } + float new_max = max(running_max, block_max); + threadlocal_running_denominator = threadlocal_running_denominator * __expf(running_max - new_max) + + threadlocal_sum * __expf(block_max - new_max); + running_max = new_max; + } + } + running_denominator = cub::BlockReduce(reduce).Sum(threadlocal_running_denominator); + if (tx == 0) s_den = running_denominator; + __syncthreads(); + running_denominator = s_den; + if (tx == 0) { + partials[bx * num_slices + by] = {running_max, running_denominator}; + } + // clang-format on +} + +// FlashInfer OnlineSoftmaxReduceKernel<1024, 4, float> with its write loop split over gridDim.y CTAs per row: the +// prologue (the partials' reduction) is FlashInfer's, unchanged, in every CTA of the row. +__global__ void __launch_bounds__(kThreads) + softmaxMcReduceWriteKernel(float const* logits, float* output, fi::PartialSoftmaxResult const* partials, + float const* temperature_arr, float temperature_val, uint32_t d, uint32_t num_slices) +{ + // clang-format off + const uint32_t bx = blockIdx.x; + const uint32_t tx = threadIdx.x; + float temperature = temperature_arr == nullptr ? temperature_val : temperature_arr[bx]; + const float inv_temp = (temperature == 0.f) ? 0.f : 1.f / temperature; + __shared__ typename cub::BlockReduce::TempStorage reduce_pair; + __shared__ float s_max, s_den; + const fi::Float2SoftmaxReduceOp reduce_op; + float2 thread_aggregate = make_float2(-cuda::std::numeric_limits::infinity(), 0.0f); + for (uint32_t i = tx; i < num_slices; i += kThreads) { + fi::PartialSoftmaxResult partial = partials[bx * num_slices + i]; + float2 partial_pair = make_float2(partial.max_val, partial.denominator); + thread_aggregate = reduce_op(thread_aggregate, partial_pair); + } + float2 block_result = cub::BlockReduce(reduce_pair).Reduce(thread_aggregate, reduce_op); + if (tx == 0) { + s_max = block_result.x; + s_den = block_result.y; + } + __syncthreads(); + const float final_max = s_max; + const float inv_denominator = 1.0f / s_den; + // FlashInfer's write loop iterates i over ceil_div(d, 1024 * 4) tiles of 4096 elements; this CTA takes tiles + // i = blockIdx.y, blockIdx.y + gridDim.y, ... (elementwise, so the split does not change any value). + fi::vec_t logits_vec; + fi::vec_t prob_vec; + for (uint32_t i = blockIdx.y; i < fi::ceil_div(d, kThreads * kVec); i += gridDim.y) { + logits_vec.fill(-cuda::std::numeric_limits::infinity()); + if ((i * kThreads + tx) * kVec < d) { + logits_vec.cast_load(logits + bx * d + (i * kThreads + tx) * kVec); + } +#pragma unroll + for (uint32_t j = 0; j < kVec; ++j) { + logits_vec[j] *= inv_temp; + float p = __expf(static_cast(logits_vec[j]) - final_max) * inv_denominator; + prob_vec[j] = static_cast(p); + } + if ((i * kThreads + tx) * kVec < d) { + prob_vec.cast_store(output + bx * d + (i * kThreads + tx) * kVec); + } + } + // clang-format on +} + +uint32_t numSlices(int d) +{ + return fi::ceil_div(static_cast(d), static_cast(kSoftmaxMcSliceSize)); +} + +} // namespace + +cudaError_t invokeSoftmaxMc(float const* logits, float* out, float const* temperatureArr, float temperatureVal, + int rows, int d, int writeCtas, void* partials, cudaStream_t stream) +{ + if (rows <= 0 || d <= 0) + { + return cudaSuccess; + } + if (d % static_cast(kVec) != 0 || writeCtas < 1 || writeCtas > 65535 || rows > 65535) + { + return cudaErrorInvalidValue; + } + uint32_t const slices = numSlices(d); + auto* part = static_cast(partials); + cudaLaunchConfig_t cfg{}; + cfg.gridDim = dim3(static_cast(rows), slices); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = 0; + cfg.stream = stream; + cudaError_t const err = cudaLaunchKernelEx( + &cfg, softmaxMcMapKernel, logits, part, temperatureArr, temperatureVal, static_cast(d), slices); + if (err != cudaSuccess) + { + return err; + } + cfg.gridDim = dim3(static_cast(rows), static_cast(writeCtas)); + return cudaLaunchKernelEx(&cfg, softmaxMcReduceWriteKernel, logits, out, + static_cast(part), temperatureArr, temperatureVal, static_cast(d), + slices); +} + +size_t softmaxMcPartialsBytes(int rows, int d) +{ + if (rows <= 0 || d <= 0) + { + return 0; + } + return static_cast(rows) * numSlices(d) * sizeof(fi::PartialSoftmaxResult); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.h b/cpp/tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.h new file mode 100644 index 000000000000..37326130f669 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.h @@ -0,0 +1,71 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! FlashInfer's Path A slice size (OnlineSoftmax DEFAULT_SLICE_SIZE): the Map kernel's grid is rows x +//! ceil(d / kSoftmaxMcSliceSize) CTAs, one partial (max, denominator) float2 each. +constexpr int kSoftmaxMcSliceSize = 8192; + +//! ST-01-SM: FlashInfer softmax(logits, temperature) Path A (OnlineSoftmaxMapKernel + OnlineSoftmaxReduceKernel, +//! batch <= 128 and vocab >= 24576) with the Reduce kernel's write pass spread over writeCtas CTAs per row instead of +//! one (FlashInfer: one 1024-thread CTA per row, ~19 us at 7-8 rows of a 200064 vocab on GB300). +//! +//! Semantics: logits fp32 [rows, d] (contiguous, d % 4 == 0, 16-byte aligned) -> out fp32 [rows, d] (same layout), +//! out[r, i] = softmax(logits[r,:] / T_r)[i] with T_r = temperatureArr[r] when temperatureArr is not null, else +//! temperatureVal (inverse temperature 0 when T_r == 0, as FlashInfer). +//! +//! Bitwise contract vs flashinfer.sampling.softmax(logits, temperature) (FlashInfer 0.6.15 == 0.6.18, Path A): +//! - the Map kernel is FlashInfer's OnlineSoftmaxMapKernel<1024, 4, float> (same grid, same cub BlockReduce calls, +//! same per-element source form), writing the same per-slice partials; +//! - every one of a row's writeCtas CTAs runs FlashInfer's Reduce prologue unchanged (the same cub +//! BlockReduce with Float2SoftmaxReduceOp over the same partials in the same order), so each gets +//! FlashInfer's final max and 1 / denominator bit for bit, and writes only its share of the row's 4096-element +//! tiles (tiles writeCtas apart) with the Reduce kernel's per-element arithmetic. The write is elementwise, so the +//! split changes no value. +//! Validated bitwise at writeCtas 4 / 8 / 16 on GB300 (microbenchmark): 9.2 vs FlashInfer 21.8 us per call +//! at 7-8 rows with writeCtas 16. +//! +//! MUST be compiled with --use_fast_math (FlashInfer's JIT flags: __expf, the approximate 1 / x, flush-to-zero); +//! cpp/tensorrt_llm/kernels/CMakeLists.txt sets the flag on this one source. +//! +//! partials: rows x ceil(d / kSoftmaxMcSliceSize) float2 (softmaxMcPartialsBytes), 8-byte aligned; fully overwritten +//! by every launch (no initialization, CUDA-graph safe). Launches that share it must be stream-ordered. Two plain +//! launches (no PDL), no co-residency requirement. The caller keeps the contract rows <= 128 and d >= 24576 (the +//! shapes where FlashInfer itself takes Path A); 1 <= writeCtas <= 65535. +cudaError_t invokeSoftmaxMc(float const* logits, float* out, float const* temperatureArr, float temperatureVal, + int rows, int d, int writeCtas, void* partials, cudaStream_t stream); + +//! Bytes of invokeSoftmaxMc's partials workspace for rows x d. +size_t softmaxMcPartialsBytes(int rows, int d); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.cu b/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.cu new file mode 100644 index 000000000000..567986ea2f1c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.cu @@ -0,0 +1,621 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3.1 TP8 long-context sparse decode attention in one launch (8 query heads on one KV head). The M3.1 +// target's copy of kernels/minimaxM3Tp8/sparseDecodeLc8.cu with one change (SYNC3): every thread reads its two +// heads' row max and row sum from the four per-warp partials itself, in the same order as the single-thread +// rowm / rowl steps it replaces, so two of the softmax's five __syncthreads go and the rows are bit-identical; and one +// Reserved shared-memory layout for the kernel. +// +// Grid: one cluster of 16 CTAs per query token; CTA `r` attends the token's `r`-th selected block. The QK and PV +// MMAs are transposed (S^T = K q^T, O^T = V^T P^T) so the 8 heads fill the MMA n dimension. Each CTA pushes its +// normalized partial over DSMEM so that CTA `r` holds (head r / 2, dims (r % 2) * 64 + [0, 64)) of all 16 partials, +// merges them, and writes the BF16 row and, optionally, its MXFP8 copy and swizzled UE8M0 scales. +// +// Per-block math: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and PV on FP16 tensor cores with +// FP32 accumulation, log2-domain softmax with either the linear denominator or the log-domain recurrence. +// +// PREWAIT stages everything that does not depend on the top-16 table before the grid dependency wait: the request's +// seq_lens entry, the q rows and the request's block-table row (in dynamic shared memory after Smem8, when it has at +// most kBtCap entries). The arithmetic is the same in both variants. + +#include "tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kDim = 128; +constexpr int kPage = 128; +constexpr int kTopk = 16; +constexpr int kThreads = 128; +constexpr int kPitch = kDim + 8; // halves per smem row: 272 B, conflict-free ldmatrix +constexpr float kNegInf = -__builtin_huge_valf(); +constexpr int kHeads8 = 8; +constexpr int kHalfDim = kDim / 2; +constexpr int kBtCap = 8192; // block-table entries staged in shared memory (8192 x 128 = 1M tokens) + +using Params = MinimaxM31Tp8SparseDecodeLc8Params; + +struct Smem8 +{ + __half qs[kHeads8 * kPitch]; + __half vs[kPage * kPitch]; + __half ps[kHeads8 * kPitch]; // P^T: [head][token] + float recv[kTopk][kHalfDim]; // (head rank / 2, dims (rank % 2) * 64 + [0, 64)) partials of the 16 CTAs + float recv_lse[kTopk]; // Reserved shared-memory layout for the kernel. + float recv_l[kTopk]; // Reserved shared-memory layout for the kernel. + float part[kHalfDim]; + float red[4][kHeads8]; +}; + +template +constexpr size_t smemBytes() +{ + return PREWAIT ? sizeof(Smem8) + kBtCap * sizeof(int) : sizeof(Smem8); +} + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t e2m1x2_to_f16x2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 s;\n cvt.u8.u32 s, %1;\n cvt.rn.f16x2.e2m1x2 %0, s;\n}" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t e4m3x2_to_f16x2(uint16_t pair) +{ + uint32_t out; + asm("cvt.rn.f16x2.e4m3x2 %0, %1;" : "=r"(out) : "h"(pair)); + return out; +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ void ldsm_x4(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x4_t(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma(float (&c)[4], uint32_t const (&a)[4], uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};" + : "+f"(c[0]), "+f"(c[1]), "+f"(c[2]), "+f"(c[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} + +__device__ __forceinline__ void cluster_sync() +{ + asm volatile("barrier.cluster.arrive.release.aligned;\nbarrier.cluster.wait.acquire.aligned;" ::: "memory"); +} + +__device__ __forceinline__ uint32_t map_cluster(void const* local, uint32_t rank) +{ + uint32_t remote; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(remote) : "r"(smem_u32(local)), "r"(rank)); + return remote; +} + +__device__ __forceinline__ void st_cluster_f32(uint32_t addr, float a) +{ + asm volatile("st.shared::cluster.f32 [%0], %1;" ::"r"(addr), "f"(a) : "memory"); +} + +// One packed NVFP4 row (64 bytes, 128 values) with its 8 E4M3 scales -> 128 fp16 values in smem. +__device__ __forceinline__ void dequant_row(__half* dst, uint8_t const* packed, uint8_t const (&sc)[8], bool valid) +{ + uint4 const* src = reinterpret_cast(packed); + uint32_t* d = reinterpret_cast(dst); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + uint32_t words[4] = {w.x, w.y, w.z, w.w}; + // 16 bytes = 32 values = scale columns 2c and 2c + 1. +#pragma unroll + for (int h = 0; h < 2; ++h) + { + uint8_t const s = valid ? sc[2 * c + h] : 0; + uint32_t const sh = e4m3x2_to_f16x2(static_cast(s | (s << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); +#pragma unroll + for (int j = 0; j < 2; ++j) + { + uint32_t const word = words[2 * h + j]; +#pragma unroll + for (int b = 0; b < 4; ++b) + { + uint32_t const v2 = e2m1x2_to_f16x2((word >> (8 * b)) & 0xFF); + __half2 const x = __hmul2(*reinterpret_cast<__half2 const*>(&v2), scale); + d[c * 16 + h * 8 + j * 4 + b] = *reinterpret_cast(&x); + } + } + } + } +} + +// q: 8 heads x 128 e4m3 -> fp16; threads 0..63 = (head, 16-value chunk). +__device__ __forceinline__ void load_q(Smem8& s, Params const& p, int tok, int tid) +{ + if (tid < kHeads8 * 8) + { + int const h = tid >> 3; + int const c = (tid & 7) * 16; + uint4 const w = *reinterpret_cast(p.q + tok * p.q_tok + h * p.q_head + c); + uint32_t const words[4] = {w.x, w.y, w.z, w.w}; + uint32_t* d = reinterpret_cast(s.qs + h * kPitch + c); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + d[2 * i] = e4m3x2_to_f16x2(static_cast(words[i] & 0xFFFF)); + d[2 * i + 1] = e4m3x2_to_f16x2(static_cast(words[i] >> 16)); + } + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +template +__global__ void __launch_bounds__(kThreads) m31_tp8_lc8_attn_kernel(__grid_constant__ Params const p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(128) uint8_t smem_raw[]; + Smem8& s = *reinterpret_cast(smem_raw); + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const g = lane >> 2; + int const qd = lane & 3; + int const tok = blockIdx.x / kTopk; + uint32_t rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(rank)); + + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + if constexpr (!PREWAIT) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + + int const req = tok / p.dql; + int const qoff = tok - req * p.dql; + int const seq = p.seq_lens[req]; + int const kv_len = max(seq - p.dql + qoff + 1, 0); + int const nblocks = (kv_len + kPage - 1) / kPage; + int const real_topk = min(kTopk, nblocks); + + // The request's block-table row, staged before the grid dependency (written before the graph runs). + int* const s_bt = reinterpret_cast(smem_raw + sizeof(Smem8)); + bool bt_smem = false; + if constexpr (PREWAIT) + { + int const nbt = (max(seq, 0) + kPage - 1) / kPage; + bt_smem = nbt <= kBtCap; + if (bt_smem) + { + for (int i = tid; i < nbt; i += kThreads) + { + s_bt[i] = __ldg(p.block_table + req * p.bt_stride + i); + } + } + load_q(s, p, tok, tid); + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + int blk = static_cast(rank) < real_topk ? p.topk[tok * p.t_tok + rank * p.t_k] : -1; + bool const active = blk >= 0; + if constexpr (PREWAIT) + { + __syncthreads(); // s_bt and the q rows are complete + } + else + { + load_q(s, p, tok, tid); + } + + // Lane holds heads 2 * qd + {0, 1}; acc[mt]: dims warp * 32 + mt * 16 + g + {0, 8}. + float m_row[2] = {kNegInf, kNegInf}; + float l_row[2] = {0.F, 0.F}; + float acc[2][4]; +#pragma unroll + for (int j = 0; j < 2; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + acc[j][i] = 0.F; + + // K rows n = warp * 32 + j * 8 + g stay packed in registers. As the MMA A operand, m-tile mt takes rows + // j = 2 mt (row g) and 2 mt + 1 (row g + 8); k-step ks takes packed bytes 8 ks + qd and 8 ks + 4 + qd. + uint32_t kw[4][16]; + uint2 ksc[4]; + if (active) + { + int64_t const page = bt_smem ? s_bt[blk] : p.block_table[req * p.bt_stride + blk]; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const n = warp * 32 + j * 8 + g; + bool const valid = blk * kPage + n < kv_len; + uint4 const* src = reinterpret_cast(p.k + page * p.k_blk + n * 64); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 const w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + kw[j][4 * c] = w.x; + kw[j][4 * c + 1] = w.y; + kw[j][4 * c + 2] = w.z; + kw[j][4 * c + 3] = w.w; + } + ksc[j] = valid ? __ldg(reinterpret_cast(p.ksc + page * p.ksc_blk + n * 8)) : make_uint2(0, 0); + } + bool const valid = blk * kPage + tid < kv_len; + uint8_t vsc[8]; + if (valid) + { + uint8_t const* vb = p.vsc + page * p.vsc_blk + (tid >> 2) * 32 + (tid & 3); +#pragma unroll + for (int c = 0; c < 8; ++c) + vsc[c] = __ldg(vb + 4 * c); + } + dequant_row(s.vs + tid * kPitch, p.v + page * p.v_blk + tid * 64, vsc, valid); + } + __syncthreads(); + + if (active) + { + float const scale_log2 = p.sm_scale * 1.4426950409F * __ldg(p.kgs); + int const q4 = qd * 8; + // S^T = K q^T for this warp's 32 tokens: rows (tokens) g / g + 8 of each m-tile, columns (heads) 2 qd + {0, 1}. + float sacc[2][4]; +#pragma unroll + for (int j = 0; j < 2; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + sacc[j][i] = 0.F; +#pragma unroll + for (int ks2 = 0; ks2 < kDim / 16; ks2 += 2) + { + uint32_t bq[4]; + ldsm_x4(bq, smem_u32(s.qs + (lane & 7) * kPitch + ks2 * 16 + (lane >> 3) * 8)); +#pragma unroll + for (int u = 0; u < 2; ++u) + { + int const ks = ks2 + u; +#pragma unroll + for (int mt = 0; mt < 2; ++mt) + { + uint32_t a[4]; +#pragma unroll + for (int hi = 0; hi < 2; ++hi) + { + int const j = 2 * mt + hi; + uint32_t const scw = ks < 4 ? ksc[j].x : ksc[j].y; + uint8_t const sc = static_cast(scw >> (8 * (ks & 3))); + uint32_t const sh = e4m3x2_to_f16x2(static_cast(sc | (sc << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); + uint32_t const r0 = e2m1x2_to_f16x2((kw[j][2 * ks] >> q4) & 0xFF); + uint32_t const r1 = e2m1x2_to_f16x2((kw[j][2 * ks + 1] >> q4) & 0xFF); + __half2 const h0 = __hmul2(*reinterpret_cast<__half2 const*>(&r0), scale); + __half2 const h1 = __hmul2(*reinterpret_cast<__half2 const*>(&r1), scale); + a[hi] = *reinterpret_cast(&h0); + a[2 + hi] = *reinterpret_cast(&h1); + } + mma(sacc[mt], a, bq[2 * u], bq[2 * u + 1]); + } + } + } + float mloc[2] = {kNegInf, kNegInf}; +#pragma unroll + for (int mt = 0; mt < 2; ++mt) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const t = warp * 32 + mt * 16 + g + 8 * (i >> 1); + float const v = blk * kPage + t < kv_len ? 0.F + sacc[mt][i] * scale_log2 : kNegInf; + sacc[mt][i] = v; + mloc[i & 1] = fmaxf(mloc[i & 1], v); + } +#pragma unroll + for (int e = 0; e < 2; ++e) + { + mloc[e] = fmaxf(mloc[e], __shfl_xor_sync(0xffffffff, mloc[e], 4)); + mloc[e] = fmaxf(mloc[e], __shfl_xor_sync(0xffffffff, mloc[e], 8)); + mloc[e] = fmaxf(mloc[e], __shfl_xor_sync(0xffffffff, mloc[e], 16)); + } + if (g == 0) + { + s.red[warp][2 * qd] = mloc[0]; + s.red[warp][2 * qd + 1] = mloc[1]; + } + __syncthreads(); +#pragma unroll + for (int e = 0; e < 2; ++e) + { + int const h = 2 * qd + e; + m_row[e] = fmaxf(fmaxf(s.red[0][h], s.red[1][h]), fmaxf(s.red[2][h], s.red[3][h])); + } + float lloc[2] = {0.F, 0.F}; +#pragma unroll + for (int mt = 0; mt < 2; ++mt) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + float const pv = ex2(sacc[mt][i] - m_row[i & 1]); + lloc[i & 1] += pv; + int const t = warp * 32 + mt * 16 + g + 8 * (i >> 1); + s.ps[(2 * qd + (i & 1)) * kPitch + t] = __float2half_rn(pv); + } +#pragma unroll + for (int e = 0; e < 2; ++e) + { + lloc[e] += __shfl_xor_sync(0xffffffff, lloc[e], 4); + lloc[e] += __shfl_xor_sync(0xffffffff, lloc[e], 8); + lloc[e] += __shfl_xor_sync(0xffffffff, lloc[e], 16); + } + __syncthreads(); // row-max reads of red done before it is reused; ps complete + if (g == 0) + { + s.red[warp][2 * qd] = lloc[0]; + s.red[warp][2 * qd + 1] = lloc[1]; + } + // O^T[warp's 32 dims, 8 heads] = V^T P^T over all 128 tokens. +#pragma unroll + for (int kk = 0; kk < kPage; kk += 32) + { + uint32_t b[4]; + ldsm_x4(b, smem_u32(s.ps + (lane & 7) * kPitch + kk + (lane >> 3) * 8)); +#pragma unroll + for (int u = 0; u < 2; ++u) + { +#pragma unroll + for (int mt = 0; mt < 2; ++mt) + { + uint32_t a[4]; + int const t = kk + u * 16 + (lane & 7) + ((lane >> 4) & 1) * 8; + int const d = warp * 32 + mt * 16 + ((lane >> 3) & 1) * 8; + ldsm_x4_t(a, smem_u32(s.vs + t * kPitch + d)); + mma(acc[mt], a, b[2 * u], b[2 * u + 1]); + } + } + } + __syncthreads(); +#pragma unroll + for (int e = 0; e < 2; ++e) + { + int const h = 2 * qd + e; + l_row[e] = (s.red[0][h] + s.red[1][h]) + (s.red[2][h] + s.red[3][h]); + } + } + + // Normalize; push (head, dim) to CTA 2 * head + dim / 64, slot recv[rank][dim % 64], plus each head's LSE (log2) + // Reserved shared-memory layout for the kernel. + // stores, made visible by the cluster barrier. + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); + { + float const vgs = __ldg(p.vgs); + float scale[2], lse[2]; +#pragma unroll + for (int e = 0; e < 2; ++e) + { + if constexpr (LINEAR) + { + scale[e] = l_row[e] > 0.F ? 1.0F / l_row[e] : 0.F; + lse[e] = l_row[e] > 0.F ? m_row[e] + lg2(l_row[e]) : kNegInf; + } + else + { + float const lse_i = active ? m_row[e] + lg2(ex2(kNegInf - m_row[e]) + l_row[e]) : kNegInf; + scale[e] = lse_i > kNegInf ? ex2(m_row[e] - lse_i) : 0.F; + lse[e] = lse_i; + } + scale[e] = scale[e] * vgs; + } + uint32_t dst[2]; +#pragma unroll + for (int e = 0; e < 2; ++e) + dst[e] = map_cluster(&s.recv[rank][0], (2 * qd + e) * 2 + (warp >> 1)); +#pragma unroll + for (int mt = 0; mt < 2; ++mt) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const dl = (warp & 1) * 32 + mt * 16 + g + 8 * (i >> 1); + st_cluster_f32(dst[i & 1] + dl * 4, acc[mt][i] * scale[i & 1]); + } + if (warp == 0 && g == 0) + { +#pragma unroll + for (int e = 0; e < 2; ++e) +#pragma unroll + for (int hf = 0; hf < 2; ++hf) + { + st_cluster_f32(map_cluster(&s.recv_lse[rank], (2 * qd + e) * 2 + hf), lse[e]); + } + } + } + cluster_sync(); + + // CTA `rank` merges head rank / 2, dims (rank % 2) * 64 + dl; thread = (chunk half, dl). + int const h = static_cast(rank) >> 1; + int const dh = static_cast(rank) & 1; + int const dl = tid & (kHalfDim - 1); + int const hc = tid >> 6; + float o = 0.F; + { + float lse_c[kTopk]; + float lse_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + lse_c[c] = s.recv_lse[c]; + lse_max = fmaxf(lse_max, lse_c[c]); + } + lse_max = lse_max == kNegInf ? 0.F : lse_max; + float denom = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + lse_c[c] = ex2(lse_c[c] - lse_max); + denom += lse_c[c]; + } + denom = denom > 0.F ? denom : 1.F; +#pragma unroll + for (int c = 0; c < kTopk / 2; ++c) + { + int const cc = hc * (kTopk / 2) + c; + o += s.recv[cc][dl] * __fdiv_rn(lse_c[cc], denom); + } + } + if (hc) + s.part[dl] = o; + __syncthreads(); + if (!hc) + { + o += s.part[dl]; + + int const col = h * kDim + dh * kHalfDim + dl; + __nv_bfloat16 const ob = __float2bfloat16_rn(o); + p.out[tok * p.out_stride + col] = ob; + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + amax = fmaxf(amax, __shfl_xor_sync(0xffffffff, amax, off)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + float const xq = x * qs; + uint16_t q2; + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2) : "f"(0.F), "f"(xq)); + p.q8[static_cast(tok) * (p.num_sf_k_tiles * 128) + col] = static_cast(q2 & 0xFF); + if (lane == 0) + { + int const k_sf = col / 32; + int64_t const off = static_cast(tok / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (tok % 32) * 16 + ((tok % 128) / 32) * 4 + k_sf % 4; + p.sf[off] = static_cast(e); + } + } + } +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +} + +template +cudaError_t configure() +{ + auto kernel = m31_tp8_lc8_attn_kernel; + cudaError_t err = cudaFuncSetAttribute( + kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smemBytes())); + if (err != cudaSuccess) + { + return err; + } + return cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1); +} + +template +cudaError_t launch(Params const& prm, int num_tokens, bool use_pdl, cudaStream_t stream) +{ + // Kernel attributes are set once per process, on the first launch (outside CUDA-graph capture: warmup runs + // every variant eagerly first). + static cudaError_t const configured = configure(); + if (configured != cudaSuccess) + { + return configured; + } + auto kernel = m31_tp8_lc8_attn_kernel; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(num_tokens * kTopk); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smemBytes(); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kTopk; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +template +cudaError_t launchPrewait(Params const& prm, int num_tokens, bool prewait, bool use_pdl, cudaStream_t stream) +{ + return prewait ? launch(prm, num_tokens, use_pdl, stream) + : launch(prm, num_tokens, use_pdl, stream); +} + +} // namespace + +cudaError_t invokeMinimaxM31Tp8SparseDecodeLc8(MinimaxM31Tp8SparseDecodeLc8Params const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, cudaStream_t stream) +{ + if (linearSoftmax) + { + return mxfp8 ? launchPrewait(params, numTokens, prewait, usePdl, stream) + : launchPrewait(params, numTokens, prewait, usePdl, stream); + } + return mxfp8 ? launchPrewait(params, numTokens, prewait, usePdl, stream) + : launchPrewait(params, numTokens, prewait, usePdl, stream); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.h b/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.h new file mode 100644 index 000000000000..84689bcf778a --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.h @@ -0,0 +1,76 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Operands of the MiniMax-M3.1 TP8 one-launch long-context sparse decode (the M3.1 target's copy of +//! kernels/minimaxM3Tp8/sparseDecodeLc8; the M3 kernel is unchanged). Strides are in elements of the pointed-to +//! type. Geometry is fixed: 8 query heads on one KV head, head_dim 128, page 128, top-k 16. +struct MinimaxM31Tp8SparseDecodeLc8Params +{ + uint8_t const* q; // e4m3 [T, 8, 128] + int64_t q_tok, q_head; + uint8_t const* k; // packed e2m1 [pages, 1, 128, 64] + uint8_t const* v; + int64_t k_blk, v_blk; + uint8_t const* ksc; // e4m3 [pages, 1, 128, 8], token-major + uint8_t const* vsc; // e4m3, 4x4 token-quad order + int64_t ksc_blk, vsc_blk; + float const* kgs; + float const* vgs; + int const* topk; // [T, 16] view (kv head 0) + int64_t t_tok, t_k; + int const* block_table; + int64_t bt_stride; + int const* seq_lens; + int dql; + float sm_scale; + __nv_bfloat16* out; // [T, 1024] + int64_t out_stride; + uint8_t* q8; // e4m3 [rows, num_sf_k_tiles * 128] + uint8_t* sf; + int num_sf_k_tiles; +}; + +//! One cluster of 16 CTAs per query token: CTA r attends the token's r-th selected block with transposed MMAs (the 8 +//! heads fill the MMA n dimension), then CTA r merges (head r / 2, dims (r % 2) * 64 + [0, 64)) of the 16 partials +//! over DSMEM into the BF16 [T, 1024] row (and, when mxfp8, its MXFP8 copy with swizzled UE8M0 scales). Dependents +//! are released right after the grid dependency wait. +//! +//! With prewait, the request's seq_lens entry, its q rows (widened to FP16) and its block-table row (up to 8192 +//! entries, in dynamic shared memory; longer rows are read from global memory) are staged before the grid dependency +//! wait; outputs are bit-identical to prewait=false. +//! +//! +//! SM100-family only; on other architectures the kernel compiles to an empty body, so callers check the device first. +cudaError_t invokeMinimaxM31Tp8SparseDecodeLc8(MinimaxM31Tp8SparseDecodeLc8Params const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.cu b/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.cu new file mode 100644 index 000000000000..a9277a3fd2ea --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.cu @@ -0,0 +1,635 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3.1 TP4 long-context sparse decode attention in one launch: the M3.1 target's copy of +// kernels/minimaxM3/sparseDecodeLc.cu with two exact options (BTSMEM: the block-table row staged in shared memory +// before the grid wait; SYNC3: the softmax's row max / sum read by every thread from the per-warp partials) and one +// Reserved shared-memory layout for the kernel. +// +// +// Grid: one cluster of 16 CTAs per query token; CTA `r` attends the token's `r`-th selected block (the Triton +// producer's chunk `r` at 16 chunks), keeps the partial in shared memory, and after a cluster barrier merges head +// `r` from all 16 partials over DSMEM, writing the BF16 row and, optionally, its MXFP8 copy and swizzled UE8M0 +// scales exactly as the Triton merge's `_store_mxfp8_head` lays them out. +// +// Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and +// PV on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +// log-domain recurrence. + +#include "tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kHeads = 16; +constexpr int kDim = 128; +constexpr int kPage = 128; +constexpr int kTopk = 16; +constexpr int kThreads = 128; +constexpr int kPitch = kDim + 8; // halves per smem row: 272 B, conflict-free ldmatrix +constexpr float kNegInf = -__builtin_huge_valf(); + +using Params = MinimaxM31Tp4SparseDecodeLcParams; +constexpr int kBtCap = 8192; // block-table entries staged in shared memory (BTSMEM; 8192 x 128 = 1M tokens) + +struct Smem +{ + __half qs[kHeads * kPitch]; + __half vs[kPage * kPitch]; + __half ps[kHeads * kPitch]; + float recv[kTopk][kDim]; // head `rank` partials pushed by the 16 CTAs, chunk-major + float recv_lse[kTopk]; // Reserved shared-memory layout for the kernel. + float recv_l[kTopk]; // Reserved shared-memory layout for the kernel. + float wgt[kTopk]; // Reserved shared-memory layout for the kernel. + float wl[kTopk]; // Reserved shared-memory layout for the kernel. + float red[4][kHeads]; + float rowm[kHeads]; + float rowl[kHeads]; + alignas(8) unsigned long long rbar; // the split-K partials of head `rank` landed (complete_tx) +}; + +template +constexpr size_t smemBytes() +{ + return BTSMEM ? sizeof(Smem) + kBtCap * sizeof(int) : sizeof(Smem); +} + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t e2m1x2_to_f16x2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 s;\n cvt.u8.u32 s, %1;\n cvt.rn.f16x2.e2m1x2 %0, s;\n}" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t e4m3x2_to_f16x2(uint16_t pair) +{ + uint32_t out; + asm("cvt.rn.f16x2.e4m3x2 %0, %1;" : "=r"(out) : "h"(pair)); + return out; +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ void ldsm_x4(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x4_t(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma(float (&c)[4], uint32_t const (&a)[4], uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};" + : "+f"(c[0]), "+f"(c[1]), "+f"(c[2]), "+f"(c[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} + +__device__ __forceinline__ uint32_t map_cluster(void const* local, uint32_t rank) +{ + uint32_t remote; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(remote) : "r"(smem_u32(local)), "r"(rank)); + return remote; +} + +// Posted DSMEM stores that complete_tx on the receiving CTA's mbarrier `bar`. +__device__ __forceinline__ void st_async_v2(uint32_t addr, float a, float b, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.v2.f32 [%0], {%1, %2}, [%3];" ::"r"(addr), + "f"(a), "f"(b), "r"(bar) + : "memory"); +} + +__device__ __forceinline__ void st_async_f32(uint32_t addr, float a, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(addr), + "r"(__float_as_uint(a)), "r"(bar) + : "memory"); +} + +// One packed NVFP4 row (64 bytes, 128 values) with its 8 E4M3 scales -> 128 fp16 values in smem. +__device__ __forceinline__ void dequant_row(__half* dst, uint8_t const* packed, uint8_t const (&sc)[8], bool valid) +{ + uint4 const* src = reinterpret_cast(packed); + uint32_t* d = reinterpret_cast(dst); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + uint32_t words[4] = {w.x, w.y, w.z, w.w}; + // 16 bytes = 32 values = scale columns 2c and 2c + 1. +#pragma unroll + for (int h = 0; h < 2; ++h) + { + uint8_t const s = valid ? sc[2 * c + h] : 0; + uint32_t const sh = e4m3x2_to_f16x2(static_cast(s | (s << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); +#pragma unroll + for (int j = 0; j < 2; ++j) + { + uint32_t const word = words[2 * h + j]; +#pragma unroll + for (int b = 0; b < 4; ++b) + { + uint32_t const v2 = e2m1x2_to_f16x2((word >> (8 * b)) & 0xFF); + __half2 const x = __hmul2(*reinterpret_cast<__half2 const*>(&v2), scale); + d[c * 16 + h * 8 + j * 4 + b] = *reinterpret_cast(&x); + } + } + } + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +template +__global__ void __launch_bounds__(kThreads) m31_tp4_lc_attn_kernel(__grid_constant__ Params const p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(128) uint8_t smem_raw[]; + Smem& s = *reinterpret_cast(smem_raw); + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const tok = blockIdx.x / kTopk; + uint32_t rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(rank)); + + if (tid == 0) + { + // Reserved shared-memory layout for the kernel. + // head `rank`. + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(smem_u32(&s.rbar)) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(smem_u32(&s.rbar)), + "r"(kTopk * (kDim + 1) * 4) + : "memory"); + } + // Every peer must be running before its shared memory takes a DSMEM store: arrive now, wait before pushing. + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + + // Everything that does not depend on the top-16 table is read before the grid dependency: the request's length + // (written before the step's graph runs) and the query rows (written by the producer, which completed before the + // select ahead of this grid could release it: the select triggers its dependents only after its own grid wait). + int const req = tok / p.dql; + int const qoff = tok - req * p.dql; + int const seq = p.seq_lens[req]; + int const kv_len = max(seq - p.dql + qoff + 1, 0); + int const nblocks = (kv_len + kPage - 1) / kPage; + int const real_topk = min(kTopk, nblocks); + // q: 16 heads x 128 e4m3 -> fp16; thread = (head, 16-value chunk). + { + int const h = tid >> 3; + int const c = (tid & 7) * 16; + uint4 const w = *reinterpret_cast(p.q + tok * p.q_tok + h * p.q_head + c); + uint32_t const words[4] = {w.x, w.y, w.z, w.w}; + uint32_t* d = reinterpret_cast(s.qs + h * kPitch + c); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + d[2 * i] = e4m3x2_to_f16x2(static_cast(words[i] & 0xFFFF)); + d[2 * i + 1] = e4m3x2_to_f16x2(static_cast(words[i] >> 16)); + } + } + + // BTSMEM: the request's block-table row (written before the step's graph runs) is staged before the wait too. + int* const s_bt = reinterpret_cast(smem_raw + sizeof(Smem)); + bool bt_smem = false; + if constexpr (BTSMEM) + { + int const nbt = (max(seq, 0) + kPage - 1) / kPage; + bt_smem = nbt <= kBtCap; + if (bt_smem) + { + for (int i = tid; i < nbt; i += kThreads) + s_bt[i] = __ldg(p.block_table + req * p.bt_stride + i); + } + } + + asm volatile("griddepcontrol.wait;" ::: "memory"); + if constexpr (EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } + int blk = static_cast(rank) < real_topk ? p.topk[tok * p.t_tok + rank * p.t_k] : -1; + bool const active = blk >= 0; + if constexpr (BTSMEM) + { + __syncthreads(); // s_bt complete + } + int64_t const page_pw = !active ? 0 : (bt_smem ? s_bt[blk] : p.block_table[req * p.bt_stride + blk]); + + float m_row[2] = {kNegInf, kNegInf}; + float acc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + acc[j][i] = 0.F; + float l_row[2] = {0.F, 0.F}; + + // K stays packed in registers: the MMA B fragment of token n = lane / 4 at k-step s is the f16x2 pair of + // packed bytes 8s + q and 8s + 4 + q (q = lane % 4), both under scale column s. + uint32_t kw[4][16]; + uint2 ksc[4]; + int64_t page = 0; + if (active) + { + page = page_pw; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const n = warp * 32 + j * 8 + (lane >> 2); + bool const valid = blk * kPage + n < kv_len; + uint4 const* src = reinterpret_cast(p.k + page * p.k_blk + n * 64); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 const w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + kw[j][4 * c] = w.x; + kw[j][4 * c + 1] = w.y; + kw[j][4 * c + 2] = w.z; + kw[j][4 * c + 3] = w.w; + } + ksc[j] = valid ? __ldg(reinterpret_cast(p.ksc + page * p.ksc_blk + n * 8)) : make_uint2(0, 0); + } + bool const valid = blk * kPage + tid < kv_len; + uint8_t vsc[8]; + if (valid) + { + uint8_t const* vb = p.vsc + page * p.vsc_blk + (tid >> 2) * 32 + (tid & 3); +#pragma unroll + for (int c = 0; c < 8; ++c) + vsc[c] = __ldg(vb + 4 * c); + } + dequant_row(s.vs + tid * kPitch, p.v + page * p.v_blk + tid * 64, vsc, valid); + } + __syncthreads(); + + if (active) + { + float const scale_log2 = p.sm_scale * 1.4426950409F * __ldg(p.kgs); + int const q4 = (lane & 3) * 8; + // S = q K^T for this warp's 32 tokens: rows g / g + 8, columns 2 * (lane % 4) + {0, 1} of each n-tile. + float sacc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + sacc[j][i] = 0.F; +#pragma unroll + for (int ks = 0; ks < kDim / 16; ++ks) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.qs + (lane & 15) * kPitch + ks * 16 + (lane >> 4) * 8)); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + uint32_t const scw = ks < 4 ? ksc[j].x : ksc[j].y; + uint8_t const sc = static_cast(scw >> (8 * (ks & 3))); + uint32_t const sh = e4m3x2_to_f16x2(static_cast(sc | (sc << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); + uint32_t const r0 = e2m1x2_to_f16x2((kw[j][2 * ks] >> q4) & 0xFF); + uint32_t const r1 = e2m1x2_to_f16x2((kw[j][2 * ks + 1] >> q4) & 0xFF); + __half2 const h0 = __hmul2(*reinterpret_cast<__half2 const*>(&r0), scale); + __half2 const h1 = __hmul2(*reinterpret_cast<__half2 const*>(&r1), scale); + mma(sacc[j], a, *reinterpret_cast(&h0), *reinterpret_cast(&h1)); + } + } + int const g = lane >> 2; + float mloc[2] = {kNegInf, kNegInf}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const t = warp * 32 + j * 8 + 2 * (lane & 3) + (i & 1); + float const v = blk * kPage + t < kv_len ? 0.F + sacc[j][i] * scale_log2 : kNegInf; + sacc[j][i] = v; + mloc[i >> 1] = fmaxf(mloc[i >> 1], v); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 1)); + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 2)); + } + if ((lane & 3) == 0) + { + s.red[warp][g] = mloc[0]; + s.red[warp][g + 8] = mloc[1]; + } + __syncthreads(); + if constexpr (SYNC3) + { +#pragma unroll + for (int r = 0; r < 2; ++r) + { + int const h = g + r * 8; + m_row[r] = fmaxf(fmaxf(s.red[0][h], s.red[1][h]), fmaxf(s.red[2][h], s.red[3][h])); + } + } + else + { + if (tid < kHeads) + s.rowm[tid] = fmaxf(fmaxf(s.red[0][tid], s.red[1][tid]), fmaxf(s.red[2][tid], s.red[3][tid])); + __syncthreads(); + m_row[0] = s.rowm[g]; + m_row[1] = s.rowm[g + 8]; + } + float lloc[2] = {0.F, 0.F}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; i += 2) + { + float const p0 = ex2(sacc[j][i] - m_row[i >> 1]); + float const p1 = ex2(sacc[j][i + 1] - m_row[i >> 1]); + lloc[i >> 1] += p0 + p1; + int const row = g + (i >> 1) * 8; + int const t = warp * 32 + j * 8 + 2 * (lane & 3); + *reinterpret_cast<__half2*>(s.ps + row * kPitch + t) = __floats2half2_rn(p0, p1); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 1); + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 2); + } + __syncthreads(); // rowm reads done before red is reused; ps complete + if ((lane & 3) == 0) + { + s.red[warp][g] = lloc[0]; + s.red[warp][g + 8] = lloc[1]; + } + // O[:, warp's 32 dims] = P V over all 128 tokens. +#pragma unroll + for (int kk = 0; kk < kPage; kk += 16) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.ps + (lane & 15) * kPitch + kk + (lane >> 4) * 8)); +#pragma unroll + for (int jp = 0; jp < 2; ++jp) + { + uint32_t b[4]; + int const t = kk + (lane & 7) + ((lane >> 3) & 1) * 8; + int const d = warp * 32 + jp * 16 + (lane >> 4) * 8; + ldsm_x4_t(b, smem_u32(s.vs + t * kPitch + d)); + mma(acc[2 * jp], a, b[0], b[1]); + mma(acc[2 * jp + 1], a, b[2], b[3]); + } + } + __syncthreads(); + if constexpr (SYNC3) + { +#pragma unroll + for (int r = 0; r < 2; ++r) + { + int const h = g + r * 8; + l_row[r] = (s.red[0][h] + s.red[1][h]) + (s.red[2][h] + s.red[3][h]); + } + } + else + { + if (tid < kHeads) + s.rowl[tid] = (s.red[0][tid] + s.red[1][tid]) + (s.red[2][tid] + s.red[3][tid]); + __syncthreads(); + l_row[0] = s.rowl[g]; + l_row[1] = s.rowl[g + 8]; + } + } + + // Normalize the partial and push head h's row with its LSE (log2) into CTA h's recv[rank] slot, as the Triton + // producer stores chunk `rank`; st.async stores that complete_tx on CTA h's barrier. + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); + { + int const g = lane >> 2; + float const vgs = __ldg(p.vgs); +#pragma unroll + for (int r = 0; r < 2; ++r) + { + float scale, lse; + if constexpr (LINEAR) + { + scale = l_row[r] > 0.F ? 1.0F / l_row[r] : 0.F; + lse = l_row[r] > 0.F ? m_row[r] + lg2(l_row[r]) : kNegInf; + } + else + { + float const lse_i = active ? m_row[r] + lg2(ex2(kNegInf - m_row[r]) + l_row[r]) : kNegInf; + scale = lse_i > kNegInf ? ex2(m_row[r] - lse_i) : 0.F; + lse = lse_i; + } + scale = scale * vgs; + int const row = g + r * 8; + uint32_t const dst = map_cluster(&s.recv[rank][0], row); + uint32_t const rb = map_cluster(&s.rbar, row); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const d = warp * 32 + j * 8 + 2 * (lane & 3); + st_async_v2(dst + d * 4, acc[j][2 * r] * scale, acc[j][2 * r + 1] * scale, rb); + } + if (warp == 0 && (lane & 3) == 0) + { + st_async_f32(map_cluster(&s.recv_lse[rank], row), lse, rb); + } + } + } + { + // Every CTA's partial row and LSE for head `rank` landed (complete_tx on this CTA's barrier). + uint32_t done = 0; + while (!done) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}" + : "=r"(done) + : "r"(smem_u32(&s.rbar)) + : "memory"); + } + } + + // CTA `rank` merges head `rank` across the 16 partials; thread = output dim. + int const h = static_cast(rank); + float o = 0.F; + { + float lse_c[kTopk]; + float lse_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + lse_c[c] = s.recv_lse[c]; + lse_max = fmaxf(lse_max, lse_c[c]); + } + lse_max = lse_max == kNegInf ? 0.F : lse_max; + float denom = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + lse_c[c] = ex2(lse_c[c] - lse_max); + denom += lse_c[c]; + } + denom = denom > 0.F ? denom : 1.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + o += s.recv[c][tid] * __fdiv_rn(lse_c[c], denom); + } + __nv_bfloat16 const ob = __float2bfloat16_rn(o); + p.out[tok * p.out_stride + h * kDim + tid] = ob; + + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + amax = fmaxf(amax, __shfl_xor_sync(0xffffffff, amax, off)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + float const xq = x * qs; + uint16_t q2; + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2) : "f"(0.F), "f"(xq)); + p.q8[static_cast(tok) * (p.num_sf_k_tiles * 128) + h * kDim + tid] = static_cast(q2 & 0xFF); + if (lane == 0) + { + int const k_sf = h * 4 + warp; + int64_t const off = static_cast(tok / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (tok % 32) * 16 + ((tok % 128) / 32) * 4 + k_sf % 4; + p.sf[off] = static_cast(e); + } + } + if constexpr (!EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +} + +template +cudaError_t configure() +{ + auto kernel = m31_tp4_lc_attn_kernel; + cudaError_t err = cudaFuncSetAttribute( + kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smemBytes())); + if (err != cudaSuccess) + { + return err; + } + return cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1); +} + +template +cudaError_t launch(Params const& prm, int num_tokens, bool use_pdl, cudaStream_t stream) +{ + // Kernel attributes are set once per process, on the first launch (outside CUDA-graph capture: warmup runs + // every variant eagerly first). + static cudaError_t const configured = configure(); + if (configured != cudaSuccess) + { + return configured; + } + auto kernel = m31_tp4_lc_attn_kernel; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(num_tokens * kTopk); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smemBytes(); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kTopk; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +template +cudaError_t launchOptions(Params const& prm, int num_tokens, bool btsmem, bool sync3, bool use_pdl, cudaStream_t stream) +{ + if (btsmem) + { + return sync3 ? launch(prm, num_tokens, use_pdl, stream) + : launch(prm, num_tokens, use_pdl, stream); + } + return sync3 ? launch(prm, num_tokens, use_pdl, stream) + : launch(prm, num_tokens, use_pdl, stream); +} + +} // namespace + +cudaError_t invokeMinimaxM31Tp4SparseDecodeLc(MinimaxM31Tp4SparseDecodeLcParams const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool earlyTrigger, bool btsmem, bool sync3, bool usePdl, cudaStream_t stream) +{ + if (!earlyTrigger) + { + return cudaErrorNotSupported; + } + if (linearSoftmax) + { + return mxfp8 ? launchOptions(params, numTokens, btsmem, sync3, usePdl, stream) + : launchOptions(params, numTokens, btsmem, sync3, usePdl, stream); + } + return mxfp8 ? launchOptions(params, numTokens, btsmem, sync3, usePdl, stream) + : launchOptions(params, numTokens, btsmem, sync3, usePdl, stream); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.h b/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.h new file mode 100644 index 000000000000..b228ff3d7cd0 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.h @@ -0,0 +1,77 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Operands of the MiniMax-M3.1 TP4 one-launch long-context sparse decode (the M3.1 target's copy of +//! kernels/minimaxM3/sparseDecodeLc; the M3 kernel is unchanged). Strides are in elements of the pointed-to type. +//! Geometry is fixed: 16 query heads on one KV head, head_dim 128, page 128, top-k 16. +struct MinimaxM31Tp4SparseDecodeLcParams +{ + uint8_t const* q; // e4m3 [T, 16, 128] + int64_t q_tok, q_head; + uint8_t const* k; // packed e2m1 [pages, 1, 128, 64] + uint8_t const* v; + int64_t k_blk, v_blk; + uint8_t const* ksc; // e4m3 [pages, 1, 128, 8], token-major + uint8_t const* vsc; // e4m3, 4x4 token-quad order + int64_t ksc_blk, vsc_blk; + float const* kgs; + float const* vgs; + int const* topk; // [T, 16] view (kv head 0) + int64_t t_tok, t_k; + int const* block_table; + int64_t bt_stride; + int const* seq_lens; + int dql; + float sm_scale; + __nv_bfloat16* out; // [T, 2048] + int64_t out_stride; + uint8_t* q8; // e4m3 [rows, num_sf_k_tiles * 128] + uint8_t* sf; + int num_sf_k_tiles; +}; + +//! One cluster of 16 CTAs per query token: CTA r attends the token's r-th selected block, then the cluster merges +//! the 16 partials over DSMEM into the BF16 row (and, when mxfp8, its MXFP8 copy with swizzled UE8M0 scales). +//! +//! Two exact options over the M3 kernel, each bit-identical to it: +//! - btsmem: the request's block-table row (up to 8192 entries) is staged in dynamic shared memory before the grid +//! dependency wait, so the selected block's page id is read from shared memory after it (longer rows: global). +//! - sync3: every thread reads its heads' row max and row sum from the four per-warp partials itself, in the same +//! order as the single-thread rowm / rowl steps it replaces (two of the softmax's five __syncthreads go). +//! +//! Early trigger releases PDL dependents at entry. Returns cudaErrorNotSupported for any other variant. The device code +//! is SM100-family only; on other architectures the kernel compiles to an empty body, so callers check the device +//! first. +cudaError_t invokeMinimaxM31Tp4SparseDecodeLc(MinimaxM31Tp4SparseDecodeLcParams const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool earlyTrigger, bool btsmem, bool sync3, bool usePdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.cu b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.cu new file mode 100644 index 000000000000..19b548860e6f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.cu @@ -0,0 +1,535 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// ST-01 v2: FlashInfer AirTopPRenormProb in one +// persistent launch, with the per-row keep-all bypass for top_p >= 1. See toppRenormDetSm103.h for the contract. +// Compiled with --use_fast_math (set on this source in kernels/CMakeLists.txt), as FlashInfer's JIT compiles it. + +#include "tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.h" + +#include +#include +#include +#include + +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +namespace +{ + +// --------------------------------------------------------------------------------------------------------------------- +// Vendored from FlashInfer, include/flashinfer/air_top_p.cuh (FlashInfer v0.6.15; identical in v0.6.18), namespace +// flashinfer::sampling::air_top_p. Copyright (c) 2026 by FlashInfer team, licensed under the Apache License, +// Version 2.0. Copied verbatim (only the pieces this kernel uses; formatting kept) so that the bucket keys, masks +// and the deterministic histogram -> float conversion are FlashInfer's own; FlashInfer's headers are not on the +// TensorRT-LLM C++ include path. +// --------------------------------------------------------------------------------------------------------------------- +namespace fi_air_top_p +{ +// clang-format off +static constexpr int BITS_PER_PASS = 11; +static constexpr int NUM_BUCKETS = 1 << BITS_PER_PASS; + +template +static constexpr int NUM_PASSES = (sizeof(T) * 8 + BITS_PER_PASS - 1) / BITS_PER_PASS; + +template +__device__ int constexpr calcStartBit(int pass) { + int startBit = static_cast(sizeof(T) * 8) - (pass + 1) * BITS_PER_PASS; + return startBit < 0 ? 0 : startBit; +} + +template +__device__ uint32_t constexpr calcMask(int pass) { + int numBits = calcStartBit(pass - 1) - calcStartBit(pass); + return (1 << numBits) - 1; +} + +template +__device__ typename cub::Traits::UnsignedBits twiddleIn(T key, bool selectMin) { + auto bits = reinterpret_cast::UnsignedBits&>(key); + bits = cub::Traits::TwiddleIn(bits); + if (!selectMin) bits = ~bits; + return bits; +} + +template +__device__ T twiddleOut(typename cub::Traits::UnsignedBits bits, bool selectMin) { + if (!selectMin) bits = ~bits; + bits = cub::Traits::TwiddleOut(bits); + return reinterpret_cast(bits); +} + +__device__ inline uint32_t calcMantissa(float value) { + union { + uint32_t bits; + float value; + } input; + input.value = value; + constexpr uint32_t numMantissa = 23; + return input.bits & ((1u << numMantissa) - 1); +} + +__device__ inline uint32_t calcExponent(float value) { + union { + uint32_t bits; + float value; + } input; + input.value = value; + constexpr uint32_t numMantissa = 23; + return input.bits & ~((1u << numMantissa) - 1); +} + +__device__ inline float calcFloatValue(uint32_t count, uint32_t exponent, uint64_t bitSum) { + constexpr uint32_t numTotalBits = 64; + constexpr uint32_t numMantissa = 23; + uint64_t extraInMantissa = (bitSum >> numMantissa); + extraInMantissa = (exponent == 0) ? extraInMantissa : extraInMantissa + count; + uint32_t numExtra = numTotalBits - __clzll(extraInMantissa); + int numNorm = (exponent == 0) ? 0 : -1; + exponent = exponent + ((numExtra + numNorm) << numMantissa); + uint32_t mantissa; + if (extraInMantissa != 0) { + int numMove = numMantissa - (numExtra - 1); + uint32_t mask = (1u << (numExtra - 1)) - 1; + extraInMantissa = extraInMantissa & mask; + if (numMove > 0) { + extraInMantissa = extraInMantissa << numMove; + mask = (1u << numMantissa) - 1; + mantissa = ((bitSum & mask) >> (numExtra - 1)) | extraInMantissa; + } else { + mantissa = extraInMantissa >> (-1 * numMove); + } + } else { + mantissa = bitSum; + } + uint32_t bitFloat = exponent | mantissa; + return reinterpret_cast(bitFloat); +} + +// clang-format on +} // namespace fi_air_top_p + +// --------------------------------------------------------------------------------------------------------------------- +// End of the vendored FlashInfer code. +// --------------------------------------------------------------------------------------------------------------------- + +namespace fa = fi_air_top_p; +namespace cg = cooperative_groups; + +constexpr int kThreads = 1024; +constexpr int kBuckets = fa::NUM_BUCKETS; // 2048 +constexpr int kSmemBytes = 32768; +constexpr int kCopies = 4; // privatized pass histograms +constexpr size_t kDynSmemBytes = kCopies * kBuckets * (sizeof(unsigned long long) + sizeof(int)); // 96 KB + +// One row's persistent state. Zero once; every launch leaves it zeroed except `gen` (monotonic) and the per-launch +// scalars it rewrites before reading. +struct RowWorkspace +{ + unsigned long long hist[kBuckets]; // the pass's det mantissa-sum histogram (FlashInfer's uint64 HisT) + int cnt[kBuckets]; // the pass's count histogram + float chain[kThreads]; // the 1024 per-thread chain sums of the kept values + unsigned arrive; // group barrier: CTAs arrived + unsigned gen; // group barrier: generation + unsigned kbits; // the selected key prefix (FlashInfer's kthValueBits) + float sum; // the remaining target sum (FlashInfer's counter->sum) + float norm; + int kept; + int pad[2]; +}; + +__device__ __forceinline__ unsigned ldAcquire(unsigned* p) +{ + return cuda::atomic_ref(*p).load(cuda::memory_order_acquire); +} + +__device__ __forceinline__ void stRelease(unsigned* p, unsigned v) +{ + cuda::atomic_ref(*p).store(v, cuda::memory_order_release); +} + +// Group barrier of the row's G CTAs; the last CTA to arrive runs `leader` (whole CTA) before releasing the others. +template +__device__ __forceinline__ void groupSync(RowWorkspace* w, int G, F leader) +{ + __shared__ unsigned sGen; + __shared__ int sLast; + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + unsigned gen = ldAcquire(&w->gen); + sGen = gen; + unsigned old = atomicAdd(&w->arrive, 1u); + sLast = (old == static_cast(G - 1)); + } + __syncthreads(); + if (sLast) + { + __threadfence(); + leader(); + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + atomicExch(&w->arrive, 0u); + stRelease(&w->gen, sGen + 1); + } + } + else if (threadIdx.x == 0) + { + while (ldAcquire(&w->gen) == sGen) + { + __nanosleep(20); + } + } + __syncthreads(); +} + +// FlashInfer's last-block selection for `pass` (air_top_p.cuh AirTopPRenormRadixKernel, IsDeterministic branch), on +// the row's global integer histograms. histVal and cntS (the bucket counts, cached for the sequential scan) reuse the +// CTA's scratch. +__device__ __forceinline__ void selectPass(RowWorkspace* w, int pass, float p, float* histVal, int* cntS) +{ + constexpr int WARP_SIZE = 32; + constexpr int WARP_COUNT = kBuckets / WARP_SIZE; + cg::thread_block block = cg::this_thread_block(); + cg::thread_block_tile<32> warp = cg::tiled_partition<32>(block); + __shared__ float warpSum[WARP_COUNT]; + __shared__ cuda::atomic blockSum; + for (int i = threadIdx.x; i < kBuckets; i += blockDim.x) + { + unsigned long long value = __ldcg(&w->hist[i]); + int count = __ldcg(&w->cnt[i]); + cntS[i] = count; + if (count != 0) + { + uint32_t sb = fa::calcStartBit(pass); + uint32_t bv = __ldcg(&w->kbits); + if (pass == 0) + { + bv = i << sb; + } + histVal[i] = fa::calcFloatValue( + static_cast(count), fa::calcExponent(fa::twiddleOut(bv, false)), value); + } + else + { + histVal[i] = 0.0f; + } + } + __syncthreads(); + for (int i = threadIdx.x; i < kBuckets; i += blockDim.x) + { + cg::reduce_store_async(warp, warpSum + i / WARP_SIZE, histVal[i], cg::plus{}); + } + __syncthreads(); + if (threadIdx.x < WARP_SIZE) + { + cg::reduce_store_async(warp, blockSum, warpSum[threadIdx.x], cg::plus{}); + cg::reduce_update_async(warp, blockSum, warpSum[threadIdx.x + WARP_SIZE], cg::plus{}); + } + __syncthreads(); + if (threadIdx.x == 0) + { + float currentSum = pass == 0 ? static_cast(blockSum) * p : __ldcg(&w->sum); + float prev = 0; + int targetStep = 0; + for (int i = 0; i < WARP_COUNT; i++) + { + if (warpSum[i]) + { + targetStep = i; + if ((prev + warpSum[i]) >= currentSum) + { + break; + } + prev += warpSum[i]; + } + } + int targetIdx = 0; + for (int i = targetStep * WARP_SIZE; i < kBuckets; i++) + { + if (cntS[i]) + { + targetIdx = i; + if ((prev + histVal[i]) >= currentSum) + { + break; + } + prev += histVal[i]; + } + } + w->sum = currentSum - prev; + unsigned kb = pass == 0 ? 0u : __ldcg(&w->kbits); + w->kbits = kb | (static_cast(targetIdx) << fa::calcStartBit(pass)); + } + __syncthreads(); + for (int i = threadIdx.x; i < kBuckets; i += blockDim.x) + { + w->hist[i] = 0; + w->cnt[i] = 0; + } +} + +__global__ void __launch_bounds__(kThreads, 1) toppRenormDetKernel( + float const* __restrict__ probs, float* __restrict__ out, float const* topP, int d, RowWorkspace* ws, int* keptOut) +{ + __shared__ __align__(16) unsigned char smem[kSmemBytes]; + // Privatized pass histograms: kCopies copies, one per group of warps (dynamic shared memory). + extern __shared__ __align__(16) unsigned char dyn[]; + unsigned long long* hsCopies = reinterpret_cast(dyn); + int* csCopies = reinterpret_cast(dyn + kCopies * kBuckets * sizeof(unsigned long long)); + int const row = blockIdx.y; + int const G = gridDim.x; + int const cta = blockIdx.x; + int const tid = threadIdx.x; + int const lane = tid & 31; + RowWorkspace* w = ws + row; + float const* in = probs + static_cast(row) * d; + float* o = out + static_cast(row) * d; + float const p = topP[row]; + float4 const* in4 = reinterpret_cast(in); + int const n4 = d / 4; + int const stride4 = G * kThreads; + int const start4 = cta * kThreads + tid; + + unsigned long long* hs = reinterpret_cast(smem); // 16 KB + int* cs = reinterpret_cast(smem + kBuckets * sizeof(unsigned long long)); // 8 KB + unsigned kbits = 0; + + // top_p >= 1 (the keep-all exception): no radix passes, the threshold is -inf. + bool const keepAll = p >= 1.0f; + for (int pass = 0; pass < (keepAll ? 0 : fa::NUM_PASSES); ++pass) + { + for (int i = tid; i < kCopies * kBuckets; i += kThreads) + { + hsCopies[i] = 0; + csCopies[i] = 0; + } + __syncthreads(); + int const copy = (tid >> 5) % kCopies; + unsigned long long* myHs = hsCopies + copy * kBuckets; + int* myCs = csCopies + copy * kBuckets; + int const sb = fa::calcStartBit(pass); + uint32_t const mask = fa::calcMask(pass); + int const psb = pass == 0 ? 0 : fa::calcStartBit(pass - 1); + // Warp-uniform trip count: every lane runs the same iterations (masked), so the warp collectives are + // full-warp. + for (int j0 = start4 - lane; j0 < n4; j0 += stride4) + { + int const j = j0 + lane; + bool const valid = j < n4; + float4 v = valid ? in4[j] : make_float4(0.f, 0.f, 0.f, 0.f); + float const e[4] = {v.x, v.y, v.z, v.w}; +#pragma unroll + for (int k = 0; k < 4; ++k) + { + float const x = e[k]; + auto const bits = fa::twiddleIn(x, false); + bool const ok = valid && (pass == 0 || ((bits >> psb) << psb) == kbits); + int const b = ok ? static_cast((bits >> sb) & mask) : -1; + unsigned const m = ok ? fa::calcMantissa(x) : 0u; + // Adaptive per warp (integer sums, so order-free and exact): a warp whose 32 lanes share one bucket + // (near-uniform rows) adds it with ONE aggregated atomic; otherwise every eligible lane adds into its + // warp group's private copy (few lanes collide on rows spread over many buckets). + int const b0 = __shfl_sync(0xffffffffu, b, 0); + if (__all_sync(0xffffffffu, b == b0)) + { + if (b0 >= 0) + { + unsigned const s = __reduce_add_sync(0xffffffffu, m); + if (lane == 0) + { + atomicAdd(myHs + b0, static_cast(s)); + atomicAdd(myCs + b0, 32); + } + } + } + else if (ok) + { + atomicAdd(myHs + b, static_cast(m)); + atomicAdd(myCs + b, 1); + } + } + } + __syncthreads(); + for (int i = tid; i < kBuckets; i += kThreads) + { + unsigned long long h = 0; + int c = 0; +#pragma unroll + for (int k = 0; k < kCopies; ++k) + { + h += hsCopies[k * kBuckets + i]; + c += csCopies[k * kBuckets + i]; + } + hs[i] = h; + cs[i] = c; + } + __syncthreads(); + for (int i = tid; i < kBuckets; i += kThreads) + { + if (hs[i] != 0) + { + atomicAdd(&w->hist[i], hs[i]); + } + if (cs[i] != 0) + { + atomicAdd(&w->cnt[i], cs[i]); + } + } + groupSync(w, G, + [&] { + selectPass(w, pass, p, reinterpret_cast(smem), + reinterpret_cast(smem + kBuckets * sizeof(float))); + }); + kbits = __ldcg(&w->kbits); + } + + // Apply: the threshold, FlashInfer's per-thread chain sums (spread over the G CTAs), the cub block sum, then the + // renorm write. + float const thr = keepAll ? -INFINITY : fa::twiddleOut(kbits, false); + int const cpc = kThreads / G; // chains per CTA + int const c0 = cta * cpc; + int const kTile = (kSmemBytes / 4) / cpc; + int const kTotal = (d + kThreads - 1) / kThreads; + float* stage = reinterpret_cast(smem); + float acc = 0.0f; + int kept = 0; + for (int k0 = 0; k0 < kTotal; k0 += kTile) + { + int const kn = min(kTile, kTotal - k0); + for (int q = tid; q < kn * cpc; q += kThreads) + { + int const kk = q / cpc, c = q - kk * cpc; + int const i = c0 + c + kThreads * (k0 + kk); + stage[q] = i < d ? in[i] : 0.0f; + } + __syncthreads(); + if (tid < cpc) + { + for (int kk = 0; kk < kn; ++kk) + { + int const i = c0 + tid + kThreads * (k0 + kk); + float const val = stage[kk * cpc + tid]; + if (i < d && val >= thr) + { + acc += val; + ++kept; + } + } + } + __syncthreads(); + } + if (tid < cpc) + { + w->chain[c0 + tid] = acc; + } + typedef cub::BlockReduce BlockReduce; + __shared__ typename BlockReduce::TempStorage temp; + __shared__ int ctaKept; + if (tid == 0) + { + ctaKept = 0; + } + __syncthreads(); + if (tid < cpc && kept) + { + atomicAdd(&ctaKept, kept); + } + __syncthreads(); + if (tid == 0 && ctaKept) + { + atomicAdd(&w->kept, ctaKept); + } + groupSync(w, G, + [&] + { + float totalSum = BlockReduce(temp).Sum(__ldcg(&w->chain[tid])); + if (tid == 0) + { + w->norm = (totalSum > 1e-8f) ? (1.0f / totalSum) : 1.0f; + if (keptOut) + { + keptOut[row] = __ldcg(&w->kept); + } + w->kept = 0; + } + }); + float const norm = __ldcg(&w->norm); + float4* o4 = reinterpret_cast(o); + for (int j = start4; j < n4; j += stride4) + { + float4 v = in4[j]; + float4 r; + r.x = (v.x >= thr) ? v.x * norm : 0.0f; + r.y = (v.y >= thr) ? v.y * norm : 0.0f; + r.z = (v.z >= thr) ? v.z * norm : 0.0f; + r.w = (v.w >= thr) ? v.w * norm : 0.0f; + o4[j] = r; + } +} + +} // namespace + +cudaError_t invokeToppRenormDet(float const* probs, float* out, float const* topP, int rows, int d, int ctasPerRow, + void* workspace, int* keptOut, cudaStream_t stream) +{ + if (rows <= 0 || d <= 0) + { + return cudaSuccess; + } + if (ctasPerRow < 1 || ctasPerRow > kThreads || kThreads % ctasPerRow != 0 || d % 4 != 0) + { + return cudaErrorInvalidValue; + } + static cudaError_t const attrErr = cudaFuncSetAttribute( + toppRenormDetKernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(kDynSmemBytes)); + if (attrErr != cudaSuccess) + { + return attrErr; + } + cudaLaunchConfig_t cfg{}; + cfg.gridDim = dim3(static_cast(ctasPerRow), static_cast(rows)); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = kDynSmemBytes; + cfg.stream = stream; + return cudaLaunchKernelEx( + &cfg, toppRenormDetKernel, probs, out, topP, d, static_cast(workspace), keptOut); +} + +size_t toppRenormDetWorkspaceBytesPerRow() +{ + return sizeof(RowWorkspace); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.h b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.h new file mode 100644 index 000000000000..0e5e1eb7912e --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.h @@ -0,0 +1,80 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! ST-01: the spec-decoding sampler's top-p renorm (FlashInfer top_p_renorm_probs) as ONE persistent launch of +//! rows x ctasPerRow CTAs (1024 threads each), flat in the kept count. +//! +//! Semantics: probs fp32 [rows, d] (contiguous, d % 4 == 0, 16-byte aligned), topP fp32 [rows]. Per row, the +//! threshold thr is the value FlashInfer's AirTopPRenormProb finds with its three +//! radix passes (11 / 11 / 10 bits of the descending twiddled key); out[i] = probs[i] * norm where probs[i] >= thr, +//! else 0, with norm = 1 / sum (1 when sum <= 1e-8) and sum the kept values' total in FlashInfer's order. +//! +//! Bitwise contract vs FlashInfer top_p_renorm_probs(probs, top_p, is_deterministic=True) (air_top_p.cuh, FlashInfer +//! 0.6.15 == 0.6.18), for every row with top_p < 1: +//! - the det histograms are INTEGER (uint64 mantissa sums + int counts), so the warp-aggregated updates here give +//! FlashInfer's exact histograms in any order; +//! - each pass's bucket selection is FlashInfer's last-block code (calcFloatValue, cooperative_groups warp / block +//! sums, the sequential scan) run on integer-identical histograms with the same lane <-> bucket mapping; +//! - the kept sum: thread chain t adds the kept values at i = t (mod 1024) in order (FlashInfer's per-thread input +//! to cub BlockReduce); the chains are spread over the row's CTAs and one CTA runs the same +//! cub BlockReduce::Sum on them. +//! Validated bitwise (renormed probs and the sampled token) on GB300 over R 7 / 8 / 56 / 64 x 8 row classes x +//! 4 top_p modes x 16 seeds, eager and graph (microbenchmark); v1.2's binning (each pass's histogram adds: +//! one aggregated atomic for a warp whose 32 lanes share a bucket, else per-lane atomics into one of 4 privatized +//! shared-memory copies; v1.1's cached bucket counts) is bitwise v1 on every row of the MB grid at 54.3 vs v1's 85.7 us +//! per 7-row call. +//! +//! Exception: a row with top_p >= 1 skips the radix search and keeps EVERY element +//! (threshold -inf), then the same sum and write. FlashInfer itself drops tail buckets at top_p 1 in ~43% of rows, +//! in BOTH its default and its deterministic mode; keep-all is the intended law. The choice is row-local (all CTAs of +//! a row take the same branch), so the other rows of a batch stay bitwise. +//! +//! MUST be compiled with --use_fast_math (FlashInfer's JIT flags: flush-to-zero and the approximate 1 / x of the norm); +//! without it the output is not bitwise FlashInfer's. cpp/tensorrt_llm/kernels/CMakeLists.txt sets the flag on this +//! one source. +//! +//! Workspace: toppRenormDetWorkspaceBytesPerRow() x rows bytes, ZERO-initialized once before the first launch and +//! then reused as is: every launch leaves it ready for the next (histograms cleared, arrival counters reset, the +//! barrier generation only increases), so it is CUDA-graph safe without an init kernel. Launches that share one +//! workspace must be stream-ordered. The row's CTAs synchronize through global-memory barriers, so all +//! rows x ctasPerRow CTAs must be co-resident (one 1024-thread CTA per SM): pick ctasPerRow with +//! rows x ctasPerRow <= the SM count (ctasPerRow 1 needs no co-residency). 1024 % ctasPerRow == 0. +//! keptOut (optional, int [rows]) receives the per-row count of elements >= the threshold. +cudaError_t invokeToppRenormDet(float const* probs, float* out, float const* topP, int rows, int d, int ctasPerRow, + void* workspace, int* keptOut, cudaStream_t stream); + +//! Workspace bytes per row of invokeToppRenormDet. +size_t toppRenormDetWorkspaceBytesPerRow(); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu new file mode 100644 index 000000000000..3ba14d32938d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu @@ -0,0 +1,650 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// TOPP_DET v3: toppRenormDetSm103.cu (v1.2) with the redundant selection and an optional programmatic dependent +// launch. See toppRenormDetV3Sm103.h for the contract (bitwise v1.2's output). Compiled with --use_fast_math (set on +// this source in kernels/CMakeLists.txt), as FlashInfer's JIT compiles it. + +#include "tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h" + +#include +#include +#include +#include + +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +namespace toppv3 +{ + +// --------------------------------------------------------------------------------------------------------------------- +// Vendored from FlashInfer, include/flashinfer/air_top_p.cuh (FlashInfer v0.6.15; identical in v0.6.18), namespace +// flashinfer::sampling::air_top_p. Copyright (c) 2026 by FlashInfer team, licensed under the Apache License, +// Version 2.0. Copied verbatim (only the pieces this kernel uses; formatting kept) so that the bucket keys, masks +// and the deterministic histogram -> float conversion are FlashInfer's own; FlashInfer's headers are not on the +// TensorRT-LLM C++ include path. +// --------------------------------------------------------------------------------------------------------------------- +namespace fi_air_top_p +{ +// clang-format off +static constexpr int BITS_PER_PASS = 11; +static constexpr int NUM_BUCKETS = 1 << BITS_PER_PASS; + +template +static constexpr int NUM_PASSES = (sizeof(T) * 8 + BITS_PER_PASS - 1) / BITS_PER_PASS; + +template +__device__ int constexpr calcStartBit(int pass) { + int startBit = static_cast(sizeof(T) * 8) - (pass + 1) * BITS_PER_PASS; + return startBit < 0 ? 0 : startBit; +} + +template +__device__ uint32_t constexpr calcMask(int pass) { + int numBits = calcStartBit(pass - 1) - calcStartBit(pass); + return (1 << numBits) - 1; +} + +template +__device__ typename cub::Traits::UnsignedBits twiddleIn(T key, bool selectMin) { + auto bits = reinterpret_cast::UnsignedBits&>(key); + bits = cub::Traits::TwiddleIn(bits); + if (!selectMin) bits = ~bits; + return bits; +} + +template +__device__ T twiddleOut(typename cub::Traits::UnsignedBits bits, bool selectMin) { + if (!selectMin) bits = ~bits; + bits = cub::Traits::TwiddleOut(bits); + return reinterpret_cast(bits); +} + +__device__ inline uint32_t calcMantissa(float value) { + union { + uint32_t bits; + float value; + } input; + input.value = value; + constexpr uint32_t numMantissa = 23; + return input.bits & ((1u << numMantissa) - 1); +} + +__device__ inline uint32_t calcExponent(float value) { + union { + uint32_t bits; + float value; + } input; + input.value = value; + constexpr uint32_t numMantissa = 23; + return input.bits & ~((1u << numMantissa) - 1); +} + +__device__ inline float calcFloatValue(uint32_t count, uint32_t exponent, uint64_t bitSum) { + constexpr uint32_t numTotalBits = 64; + constexpr uint32_t numMantissa = 23; + uint64_t extraInMantissa = (bitSum >> numMantissa); + extraInMantissa = (exponent == 0) ? extraInMantissa : extraInMantissa + count; + uint32_t numExtra = numTotalBits - __clzll(extraInMantissa); + int numNorm = (exponent == 0) ? 0 : -1; + exponent = exponent + ((numExtra + numNorm) << numMantissa); + uint32_t mantissa; + if (extraInMantissa != 0) { + int numMove = numMantissa - (numExtra - 1); + uint32_t mask = (1u << (numExtra - 1)) - 1; + extraInMantissa = extraInMantissa & mask; + if (numMove > 0) { + extraInMantissa = extraInMantissa << numMove; + mask = (1u << numMantissa) - 1; + mantissa = ((bitSum & mask) >> (numExtra - 1)) | extraInMantissa; + } else { + mantissa = extraInMantissa >> (-1 * numMove); + } + } else { + mantissa = bitSum; + } + uint32_t bitFloat = exponent | mantissa; + return reinterpret_cast(bitFloat); +} + +// clang-format on +} // namespace fi_air_top_p + +// --------------------------------------------------------------------------------------------------------------------- +// End of the vendored FlashInfer code. +// --------------------------------------------------------------------------------------------------------------------- + +namespace fa = fi_air_top_p; +namespace cg = cooperative_groups; + +constexpr int kThreads = 1024; +constexpr int kBuckets = fa::NUM_BUCKETS; // 2048 +constexpr int kSmemBytes = 32768; +constexpr int kCopies = 4; // privatized pass histograms +constexpr size_t kDynSmemBytes = kCopies * kBuckets * (sizeof(unsigned long long) + sizeof(int)); // 96 KB + +constexpr int kSyncs = fa::NUM_PASSES + 1; // one per radix pass, then the norm +// Candidate list: passes 1 .. NUM_PASSES - 1 run locally on the members of pass 0's threshold bucket when it holds at +// most this many elements (the list lives in the workspace, then in the dynamic shared memory of each CTA). +constexpr int kCandMax = 4096; + +// One row's persistent state. Zero once; every launch leaves it zeroed except the arrival counters, which only grow. +struct RowWorkspace +{ + unsigned long long hist[fa::NUM_PASSES][kBuckets]; // per pass: det mantissa-sum histogram (uint64 HisT) + int cnt[fa::NUM_PASSES][kBuckets]; // per pass: count histogram + float chain[kThreads]; // the 1024 per-thread chain sums of the kept values + unsigned long long arrive[kSyncs]; // per sync: CTAs arrived, monotone across launches + float cand[kCandMax]; // the candidate list (pass 0's bucket members) + unsigned nCand; // its length, zero between launches + int kept; + int pad[2]; +}; + +// Arrive at sync `si` and wait until all G CTAs of the row have arrived. The counter is never reset: each CTA adds +// kThreads / G (G divides kThreads), so every launch adds exactly kThreads per sync it runs whatever its G (a row's +// CTAs all take the same branch) and the counter is a multiple of kThreads between launches; the target is the next +// one. +__device__ __forceinline__ void allArrive(RowWorkspace* w, int G, int si) +{ + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + unsigned long long const step = static_cast(kThreads / G); + unsigned long long const old = atomicAdd(&w->arrive[si], step); + unsigned long long const target = (old / kThreads + 1) * kThreads; + cuda::atomic_ref ref(w->arrive[si]); + while (ref.load(cuda::memory_order_acquire) < target) + { + __nanosleep(20); + } + } + __syncthreads(); +} + +// FlashInfer's last-block selection for `pass` (air_top_p.cuh AirTopPRenormRadixKernel, IsDeterministic branch), on +// the row's global integer histograms of this pass, run by EVERY CTA of the row (identical inputs, identical code, so +// identical bits). The running sum and key prefix are CTA-local (shared memory) instead of the workspace. histVal and +// cntS (the bucket counts, cached for the sequential scan) reuse the CTA's scratch. +// +// kGlobal: the histograms are the row's global ones (read through L2); otherwise this CTA's shared-memory histograms of +// the candidate list, which hold the same integers. +template +__device__ __forceinline__ void selectPassOn(unsigned long long const* hist, int const* cnt, int pass, float p, + float* histVal, int* cntS, float* sSum, unsigned* sKbits) +{ + constexpr int WARP_SIZE = 32; + constexpr int WARP_COUNT = kBuckets / WARP_SIZE; + cg::thread_block block = cg::this_thread_block(); + cg::thread_block_tile<32> warp = cg::tiled_partition<32>(block); + __shared__ float warpSum[WARP_COUNT]; + __shared__ cuda::atomic blockSum; + for (int i = threadIdx.x; i < kBuckets; i += blockDim.x) + { + unsigned long long value = kGlobal ? __ldcg(&hist[i]) : hist[i]; + int count = kGlobal ? __ldcg(&cnt[i]) : cnt[i]; + cntS[i] = count; + if (count != 0) + { + uint32_t sb = fa::calcStartBit(pass); + uint32_t bv = *sKbits; + if (pass == 0) + { + bv = i << sb; + } + histVal[i] = fa::calcFloatValue( + static_cast(count), fa::calcExponent(fa::twiddleOut(bv, false)), value); + } + else + { + histVal[i] = 0.0f; + } + } + __syncthreads(); + for (int i = threadIdx.x; i < kBuckets; i += blockDim.x) + { + cg::reduce_store_async(warp, warpSum + i / WARP_SIZE, histVal[i], cg::plus{}); + } + __syncthreads(); + if (threadIdx.x < WARP_SIZE) + { + cg::reduce_store_async(warp, blockSum, warpSum[threadIdx.x], cg::plus{}); + cg::reduce_update_async(warp, blockSum, warpSum[threadIdx.x + WARP_SIZE], cg::plus{}); + } + __syncthreads(); + if (threadIdx.x == 0) + { + float currentSum = pass == 0 ? static_cast(blockSum) * p : *sSum; + float prev = 0; + int targetStep = 0; + for (int i = 0; i < WARP_COUNT; i++) + { + if (warpSum[i]) + { + targetStep = i; + if ((prev + warpSum[i]) >= currentSum) + { + break; + } + prev += warpSum[i]; + } + } + int targetIdx = 0; + for (int i = targetStep * WARP_SIZE; i < kBuckets; i++) + { + if (cntS[i]) + { + targetIdx = i; + if ((prev + histVal[i]) >= currentSum) + { + break; + } + prev += histVal[i]; + } + } + *sSum = currentSum - prev; + unsigned kb = pass == 0 ? 0u : *sKbits; + *sKbits = kb | (static_cast(targetIdx) << fa::calcStartBit(pass)); + } + __syncthreads(); +} + +__device__ __forceinline__ void selectPass( + RowWorkspace* w, int pass, float p, float* histVal, int* cntS, float* sSum, unsigned* sKbits) +{ + selectPassOn(w->hist[pass], w->cnt[pass], pass, p, histVal, cntS, sSum, sKbits); +} + +// CTA 0 clears pass `pass`'s global histograms once every CTA of the row has finished reading them. +__device__ __forceinline__ void clearPass(RowWorkspace* w, int pass) +{ + if (blockIdx.x == 0) + { + for (int i = threadIdx.x; i < kBuckets; i += blockDim.x) + { + w->hist[pass][i] = 0; + w->cnt[pass][i] = 0; + } + } +} + +template +__global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( + float const* __restrict__ probs, float* __restrict__ out, float const* topP, int d, RowWorkspace* ws, int* keptOut) +{ + __shared__ __align__(16) unsigned char smem[kSmemBytes]; + // Privatized pass histograms: kCopies copies, one per group of warps (dynamic shared memory). + extern __shared__ __align__(16) unsigned char dyn[]; + unsigned long long* hsCopies = reinterpret_cast(dyn); + int* csCopies = reinterpret_cast(dyn + kCopies * kBuckets * sizeof(unsigned long long)); + int const row = blockIdx.y; + int const G = gridDim.x; + int const cta = blockIdx.x; + int const tid = threadIdx.x; + int const lane = tid & 31; + RowWorkspace* w = ws + row; + float const* in = probs + static_cast(row) * d; + float* o = out + static_cast(row) * d; + __shared__ float sSum; + __shared__ unsigned sKbits; + if (tid == 0) + { + sSum = 0.0f; + sKbits = 0u; + } + // Everything above is CTA-private; probs, top_p and the shared workspace are only touched after the wait. + if constexpr (kPdl) + { + cudaGridDependencySynchronize(); + } + __syncthreads(); + float const p = topP[row]; + float4 const* in4 = reinterpret_cast(in); + int const n4 = d / 4; + int const stride4 = G * kThreads; + int const start4 = cta * kThreads + tid; + + unsigned long long* hs = reinterpret_cast(smem); // 16 KB + int* cs = reinterpret_cast(smem + kBuckets * sizeof(unsigned long long)); // 8 KB + unsigned kbits = 0; + + // top_p >= 1 (the keep-all exception): no radix passes, the threshold is -inf. + bool const keepAll = p >= 1.0f; + for (int pass = 0; pass < (keepAll ? 0 : fa::NUM_PASSES); ++pass) + { + for (int i = tid; i < kCopies * kBuckets; i += kThreads) + { + hsCopies[i] = 0; + csCopies[i] = 0; + } + __syncthreads(); + int const copy = (tid >> 5) % kCopies; + unsigned long long* myHs = hsCopies + copy * kBuckets; + int* myCs = csCopies + copy * kBuckets; + int const sb = fa::calcStartBit(pass); + uint32_t const mask = fa::calcMask(pass); + int const psb = pass == 0 ? 0 : fa::calcStartBit(pass - 1); + // Warp-uniform trip count: every lane runs the same iterations (masked), so the warp collectives are + // full-warp. + for (int j0 = start4 - lane; j0 < n4; j0 += stride4) + { + int const j = j0 + lane; + bool const valid = j < n4; + float4 v = valid ? in4[j] : make_float4(0.f, 0.f, 0.f, 0.f); + float const e[4] = {v.x, v.y, v.z, v.w}; +#pragma unroll + for (int k = 0; k < 4; ++k) + { + float const x = e[k]; + auto const bits = fa::twiddleIn(x, false); + bool const ok = valid && (pass == 0 || ((bits >> psb) << psb) == kbits); + int const b = ok ? static_cast((bits >> sb) & mask) : -1; + unsigned const m = ok ? fa::calcMantissa(x) : 0u; + // Adaptive per warp (integer sums, so order-free and exact): a warp whose 32 lanes share one bucket + // (near-uniform rows) adds it with ONE aggregated atomic; otherwise every eligible lane adds into its + // warp group's private copy (few lanes collide on rows spread over many buckets). + int const b0 = __shfl_sync(0xffffffffu, b, 0); + if (__all_sync(0xffffffffu, b == b0)) + { + if (b0 >= 0) + { + unsigned const s = __reduce_add_sync(0xffffffffu, m); + if (lane == 0) + { + atomicAdd(myHs + b0, static_cast(s)); + atomicAdd(myCs + b0, 32); + } + } + } + else if (ok) + { + atomicAdd(myHs + b, static_cast(m)); + atomicAdd(myCs + b, 1); + } + } + } + __syncthreads(); + for (int i = tid; i < kBuckets; i += kThreads) + { + unsigned long long h = 0; + int c = 0; +#pragma unroll + for (int k = 0; k < kCopies; ++k) + { + h += hsCopies[k * kBuckets + i]; + c += csCopies[k * kBuckets + i]; + } + hs[i] = h; + cs[i] = c; + } + __syncthreads(); + for (int i = tid; i < kBuckets; i += kThreads) + { + if (hs[i] != 0) + { + atomicAdd(&w->hist[pass][i], hs[i]); + } + if (cs[i] != 0) + { + atomicAdd(&w->cnt[pass][i], cs[i]); + } + } + allArrive(w, G, pass); + if (pass > 0) + { + clearPass(w, pass - 1); // every CTA finished selectPass(pass - 1) before arriving here + } + selectPass(w, pass, p, reinterpret_cast(smem), reinterpret_cast(smem + kBuckets * sizeof(float)), + &sSum, &sKbits); + kbits = sKbits; + if constexpr (kCandList) + { + // Pass 0's threshold bucket count (cached by selectPass) is the same in every CTA of the row, so all of + // them take the same branch. A small bucket: list its members once, then finish the radix search locally. + int const sb0 = fa::calcStartBit(0); + int const nCand = reinterpret_cast(smem + kBuckets * sizeof(float))[kbits >> sb0]; + if (pass == 0 && nCand <= kCandMax) + { + for (int j0 = start4 - lane; j0 < n4; j0 += stride4) + { + int const j = j0 + lane; + if (j < n4) + { + float4 const v = in4[j]; + float const e[4] = {v.x, v.y, v.z, v.w}; +#pragma unroll + for (int k = 0; k < 4; ++k) + { + if (((fa::twiddleIn(e[k], false) >> sb0) << sb0) == kbits) + { + w->cand[atomicAdd(&w->nCand, 1u)] = e[k]; + } + } + } + } + allArrive(w, G, 1); + clearPass(w, 0); // every CTA finished selectPass(0) before arriving here + // The list in this CTA's dynamic shared memory (privatized copy 1's space), the local histograms in + // copy 0's: the same integer histograms the global passes 1 .. NUM_PASSES - 1 would build. + float* candS = reinterpret_cast(hsCopies + kBuckets); + for (int i = tid; i < nCand; i += kThreads) + { + candS[i] = __ldcg(&w->cand[i]); + } + __syncthreads(); + for (int lp = 1; lp < fa::NUM_PASSES; ++lp) + { + for (int i = tid; i < kBuckets; i += kThreads) + { + hsCopies[i] = 0; + csCopies[i] = 0; + } + __syncthreads(); + int const lsb = fa::calcStartBit(lp); + uint32_t const lmask = fa::calcMask(lp); + int const lpsb = fa::calcStartBit(lp - 1); + for (int i = tid; i < nCand; i += kThreads) + { + float const x = candS[i]; + auto const bits = fa::twiddleIn(x, false); + if (((bits >> lpsb) << lpsb) == kbits) + { + int const b = static_cast((bits >> lsb) & lmask); + atomicAdd(hsCopies + b, static_cast(fa::calcMantissa(x))); + atomicAdd(csCopies + b, 1); + } + } + __syncthreads(); + selectPassOn(hsCopies, csCopies, lp, p, reinterpret_cast(smem), + reinterpret_cast(smem + kBuckets * sizeof(float)), &sSum, &sKbits); + kbits = sKbits; + } + break; + } + } + } + + // Apply: the threshold, FlashInfer's per-thread chain sums (spread over the G CTAs), the cub block sum, then the + // renorm write. + float const thr = keepAll ? -INFINITY : fa::twiddleOut(kbits, false); + int const cpc = kThreads / G; // chains per CTA + int const c0 = cta * cpc; + int const kTile = (kSmemBytes / 4) / cpc; + int const kTotal = (d + kThreads - 1) / kThreads; + float* stage = reinterpret_cast(smem); + float acc = 0.0f; + int kept = 0; + for (int k0 = 0; k0 < kTotal; k0 += kTile) + { + int const kn = min(kTile, kTotal - k0); + for (int q = tid; q < kn * cpc; q += kThreads) + { + int const kk = q / cpc, c = q - kk * cpc; + int const i = c0 + c + kThreads * (k0 + kk); + stage[q] = i < d ? in[i] : 0.0f; + } + __syncthreads(); + if (tid < cpc) + { + for (int kk = 0; kk < kn; ++kk) + { + int const i = c0 + tid + kThreads * (k0 + kk); + float const val = stage[kk * cpc + tid]; + if (i < d && val >= thr) + { + acc += val; + ++kept; + } + } + } + __syncthreads(); + } + if (tid < cpc) + { + w->chain[c0 + tid] = acc; + } + typedef cub::BlockReduce BlockReduce; + __shared__ typename BlockReduce::TempStorage temp; + __shared__ int ctaKept; + if (tid == 0) + { + ctaKept = 0; + } + __syncthreads(); + if (tid < cpc && kept) + { + atomicAdd(&ctaKept, kept); + } + __syncthreads(); + if (tid == 0 && ctaKept) + { + atomicAdd(&w->kept, ctaKept); + } + allArrive(w, G, fa::NUM_PASSES); + if (!keepAll) + { + clearPass(w, fa::NUM_PASSES - 1); // every CTA finished the last selectPass before arriving here + } + __shared__ float sNorm; + { + float totalSum = BlockReduce(temp).Sum(__ldcg(&w->chain[tid])); + if (tid == 0) + { + sNorm = (totalSum > 1e-8f) ? (1.0f / totalSum) : 1.0f; + } + } + __syncthreads(); + if (cta == 0 && tid == 0) + { + if (keptOut) + { + keptOut[row] = __ldcg(&w->kept); + } + w->kept = 0; // only CTA 0 reads it; the next launch adds to it after its own grid-dependency wait + if constexpr (kCandList) + { + w->nCand = 0; // every CTA copied the list before arriving at the norm barrier + } + } + // The workspace is left ready for the next launch here (the last clear above), so dependents may start; their + // grid-dependency wait still covers this launch's renorm write. + if constexpr (kPdl) + { + cudaTriggerProgrammaticLaunchCompletion(); + } + float const norm = sNorm; + float4* o4 = reinterpret_cast(o); + for (int j = start4; j < n4; j += stride4) + { + float4 v = in4[j]; + float4 r; + r.x = (v.x >= thr) ? v.x * norm : 0.0f; + r.y = (v.y >= thr) ? v.y * norm : 0.0f; + r.z = (v.z >= thr) ? v.z * norm : 0.0f; + r.w = (v.w >= thr) ? v.w * norm : 0.0f; + o4[j] = r; + } +} + +} // namespace toppv3 + +cudaError_t invokeToppRenormDetV3(float const* probs, float* out, float const* topP, int rows, int d, int ctasPerRow, + void* workspace, int* keptOut, bool pdl, bool candidateList, cudaStream_t stream) +{ + using namespace toppv3; + if (rows <= 0 || d <= 0) + { + return cudaSuccess; + } + if (ctasPerRow < 1 || ctasPerRow > kThreads || kThreads % ctasPerRow != 0 || d % 4 != 0) + { + return cudaErrorInvalidValue; + } + static cudaError_t const attrErr = [] + { + for (auto* k : {toppRenormDetV3Kernel, toppRenormDetV3Kernel, + toppRenormDetV3Kernel, toppRenormDetV3Kernel}) + { + cudaError_t const e + = cudaFuncSetAttribute(k, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(kDynSmemBytes)); + if (e != cudaSuccess) + { + return e; + } + } + return cudaSuccess; + }(); + if (attrErr != cudaSuccess) + { + return attrErr; + } + cudaLaunchConfig_t cfg{}; + cfg.gridDim = dim3(static_cast(ctasPerRow), static_cast(rows)); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = kDynSmemBytes; + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + auto* ws = static_cast(workspace); + auto* kernel = pdl ? (candidateList ? toppRenormDetV3Kernel : toppRenormDetV3Kernel) + : (candidateList ? toppRenormDetV3Kernel : toppRenormDetV3Kernel); + return cudaLaunchKernelEx(&cfg, kernel, probs, out, topP, d, ws, keptOut); +} + +size_t toppRenormDetV3WorkspaceBytesPerRow() +{ + return sizeof(toppv3::RowWorkspace); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h new file mode 100644 index 000000000000..f7a08d3fb619 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h @@ -0,0 +1,58 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! TOPP_DET v3: the same renorm as invokeToppRenormDet (toppRenormDetSm103.h: semantics, the bitwise contract vs +//! FlashInfer's deterministic top_p_renorm_probs, the top_p >= 1 keep-all rule, --use_fast_math, the co-residency +//! rule for ctasPerRow), with the output bitwise equal to v1.2's on every row. What changes is the synchronization: +//! - no leader CTA: after each radix pass every CTA of the row waits for all of them, then runs FlashInfer's +//! selection itself on the row's global integer histograms of that pass (identical inputs and code, so identical +//! bits in every CTA); the norm is likewise reduced by every CTA from the same 1024 chain sums; +//! - one global histogram buffer per pass, cleared by CTA 0 one barrier later (once every CTA has read it); +//! - per-barrier arrival counters that only grow (each launch adds 1024 per barrier it runs, whatever its +//! ctasPerRow), so a workspace needs no reset between launches and replays, and launches with different +//! ctasPerRow may share it. +//! candidateList: when pass 0's threshold bucket holds at most 4096 elements, its members are listed once (one barrier) +//! and the remaining radix passes run locally in every CTA on the list (the same integer histograms, so the same +//! bits); larger buckets take the global passes. +//! pdl: launched as a programmatic dependent; everything that reads probs, top_p or the workspace waits for the +//! preceding grid, and dependents are released once the workspace is ready for the next launch. +//! Workspace: toppRenormDetV3WorkspaceBytesPerRow() x rows bytes, ZERO-initialized once before the first launch, +//! then reused as is (CUDA-graph safe). Launches sharing one workspace must be stream-ordered. +cudaError_t invokeToppRenormDetV3(float const* probs, float* out, float const* topP, int rows, int d, int ctasPerRow, + void* workspace, int* keptOut, bool pdl, bool candidateList, cudaStream_t stream); + +//! Workspace bytes per row of invokeToppRenormDetV3. +size_t toppRenormDetV3WorkspaceBytesPerRow(); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.cu b/cpp/tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.cu new file mode 100644 index 000000000000..dfc570239a5c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.cu @@ -0,0 +1,770 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3 TP8 fused index score + top-16 block select in one launch. +// +// Scoring: grid (splits, batch), one CTA per SM. CTA (s, b) owns blocks s, s + splits, ... of request b and TMA-loads +// their index-K pages through a stage ring (one producer warp, 8 consumer warps; each consumer warp scores a 32-row +// slice of a page against all q columns with m16n8k16 FP16 MMAs, column = head * 4 + query row). Block scores go to a +// row-major scratch; the last CTA of the request (arrival counter) bulk-copies the request's rows back into its idle +// stage ring and selects the top 16 per (query row, KV head): a threshold bound from per-thread maxima, a candidate +// list, exact ranking by (key desc, block asc), and an exact radix select when ties overflow the candidate list. + +#include "tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.h" + +#include "tensorrt_llm/common/cudaDriverWrapper.h" + +#include +#include +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kTopK = 16; +constexpr int kBlockK = 128; +constexpr int kHeadDim = 128; +constexpr int kStageBytes = kBlockK * kHeadDim; // fp8 +constexpr int kConsumerWarps = 8; +constexpr int kConsumerThreads = kConsumerWarps * 32; +constexpr int kThreads = kConsumerThreads + 32; // + producer warp +constexpr int kMaxDql = 4; // B-operand column = head * kMaxDql + query row +constexpr int kCols = 8; // one n8 tile: heads * kMaxDql <= 8 +constexpr int kMaxRows = 8; // kv_heads * dql +constexpr int kMaxLocal = 128; // blocks per CTA +constexpr int kCandCap = 256; +constexpr int kEpiBytes = kMaxLocal * 4 * kCols * 4; +constexpr float kInitScore = 1.0e30F; +constexpr float kLocalScore = 1.0e29F; +constexpr int kNStage = 12; // TMA stage ring depth (pages in flight per CTA) + +using Params = MinimaxM3Tp8FusedScoreSelectParams; + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +__device__ __forceinline__ uint32_t twiddle(float x) +{ + uint32_t const bits = __float_as_uint(x); + return bits ^ ((bits & 0x80000000u) ? 0xFFFFFFFFu : 0x80000000u); +} + +__device__ __forceinline__ uint32_t cvt_e4m3x2(uint32_t pair16) +{ + uint32_t out; + unsigned short const in = static_cast(pair16 & 0xFFFFu); + asm("cvt.rn.f16x2.e4m3x2 %0, %1;" : "=r"(out) : "h"(in)); + return out; +} + +__device__ __forceinline__ void mma16816( + float (&c)[4], uint32_t a0, uint32_t a1, uint32_t a2, uint32_t a3, uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};" + : "+f"(c[0]), "+f"(c[1]), "+f"(c[2]), "+f"(c[3]) + : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(b0), "r"(b1)); +} + +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ void mbar_init(uint32_t bar, int count) +{ + asm volatile("mbarrier.init.shared::cta.b64 [%0], %1;" ::"r"(bar), "r"(count) : "memory"); +} + +__device__ __forceinline__ void mbar_expect_tx(uint32_t bar, uint32_t bytes) +{ + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(bar), "r"(bytes) : "memory"); +} + +__device__ __forceinline__ void mbar_arrive(uint32_t bar) +{ + asm volatile("mbarrier.arrive.shared::cta.b64 _, [%0];" ::"r"(bar) : "memory"); +} + +__device__ __forceinline__ void mbar_wait(uint32_t bar, uint32_t parity) +{ + uint32_t done; + do + { + asm volatile( + "{\n\t.reg .pred p;\n\tmbarrier.try_wait.parity.shared::cta.b64 p, [%1], %2;\n\tselp.u32 %0, 1, 0, p;\n\t}" + : "=r"(done) + : "r"(bar), "r"(parity) + : "memory"); + } while (!done); +} + +__device__ __forceinline__ void tma_load_page( + uint32_t dst, CUtensorMap const* tmap, int page, uint32_t bar, uint64_t policy) +{ + asm volatile( + "cp.async.bulk.tensor.3d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.L2::cache_hint" + " [%0], [%1, {%2, %3, %4}], [%5], %6;" ::"r"(dst), + "l"(reinterpret_cast(tmap)), "r"(0), "r"(0), "r"(page), "r"(bar), "l"(policy) + : "memory"); +} + +// Exact radix select over n smem keys (fallback when the threshold bound leaves too many candidates). +__device__ void radix_emit(uint32_t const* kr, int n, uint32_t* hist, int32_t* dst, uint32_t neg_inf_key) +{ + int const lane = threadIdx.x & 31; + uint32_t prefix = 0, mask = 0; + int need = kTopK; + bool exact_bin = false; + for (int shift = 24; shift >= 0; shift -= 8) + { + for (int i = lane; i < 256; i += 32) + hist[i] = 0; + __syncwarp(); + for (int i = lane; i < n; i += 32) + { + uint32_t const k = kr[i]; + if ((k & mask) == prefix) + atomicAdd(&hist[(k >> shift) & 255u], 1u); + } + __syncwarp(); + uint32_t c[8]; + uint32_t sum = 0; +#pragma unroll + for (int j = 0; j < 8; ++j) + { + c[j] = hist[255 - 8 * lane - j]; + sum += c[j]; + } + uint32_t incl = sum; +#pragma unroll + for (int o = 1; o < 32; o <<= 1) + { + uint32_t const v = __shfl_up_sync(0xFFFFFFFFu, incl, o); + if (lane >= o) + incl += v; + } + uint32_t above = incl - sum; + int found_bin = -1; + uint32_t found_above = 0, found_cnt = 0; +#pragma unroll + for (int j = 0; j < 8; ++j) + { + if (found_bin < 0 && above < static_cast(need) && above + c[j] >= static_cast(need)) + { + found_bin = 255 - 8 * lane - j; + found_above = above; + found_cnt = c[j]; + } + above += c[j]; + } + unsigned const bal = __ballot_sync(0xFFFFFFFFu, found_bin >= 0); + int const src = __ffs(bal) - 1; + int const bin = __shfl_sync(0xFFFFFFFFu, found_bin, src); + uint32_t const fa = __shfl_sync(0xFFFFFFFFu, found_above, src); + uint32_t const fc = __shfl_sync(0xFFFFFFFFu, found_cnt, src); + __syncwarp(); + need -= static_cast(fa); + prefix |= static_cast(bin) << shift; + mask |= 255u << shift; + if (fc == static_cast(need)) + { + exact_bin = true; + break; + } + } + int emitted = 0, ties_used = 0; + uint32_t const lt = (1u << lane) - 1u; + for (int c0 = 0; c0 < n; c0 += 32) + { + int const i = c0 + lane; + bool const valid = i < n; + uint32_t const k = valid ? kr[i] : 0u; + uint32_t const km = k & mask; + bool const neg = valid && k == neg_inf_key; + bool sel = valid && (km > prefix || (exact_bin && km == prefix)); + bool const tie = valid && !exact_bin && km == prefix; + unsigned const tie_bal = __ballot_sync(0xFFFFFFFFu, tie); + if (tie) + sel = ties_used + __popc(tie_bal & lt) < need; + ties_used += __popc(tie_bal); + bool const out_ok = sel && !neg; + unsigned const ob = __ballot_sync(0xFFFFFFFFu, out_ok); + if (out_ok) + dst[emitted + __popc(ob & lt)] = i; + emitted += __popc(ob); + } + if (lane >= emitted && lane < kTopK) + dst[lane] = -1; + __syncwarp(); +} + +// Descending bitonic sort of one value per lane (lane 0 ends with the largest). +__device__ __forceinline__ uint32_t warp_sort_desc(uint32_t v, int lane) +{ +#pragma unroll + for (int k = 2; k <= 32; k <<= 1) + { +#pragma unroll + for (int j = k >> 1; j > 0; j >>= 1) + { + uint32_t const o = __shfl_xor_sync(0xFFFFFFFFu, v, j); + bool const desc_block = (lane & k) == 0; + bool const low = (lane & j) == 0; + v = (low == desc_block) ? max(v, o) : min(v, o); + } + } + return v; +} + +__device__ __forceinline__ uint32_t forced_key(float sc, int i, int init_blocks, int local_start) +{ + if (i < init_blocks) + sc = kInitScore; + if (i >= local_start) + sc = kLocalScore; + return twiddle(sc); +} + +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +template +__global__ void __launch_bounds__(kThreads, 1) + m3_fused_score_select_kernel(__grid_constant__ CUtensorMap const tmap, Params p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ uint8_t smem_raw[]; + uint8_t* smem = smem_raw + ((1024u - (smem_u32(smem_raw) & 1023u)) & 1023u); + uint8_t* stages = smem; + float* epi = reinterpret_cast(smem + NSTAGE * kStageBytes); // [kMaxLocal][4 slices][kCols] + uint64_t* bars + = reinterpret_cast(smem + NSTAGE * kStageBytes + kEpiBytes); // full[NSTAGE], empty[NSTAGE] + __shared__ uint32_t s_wtop[kMaxRows][kConsumerWarps][16]; + __shared__ int s_cnt[kMaxRows]; + __shared__ int32_t s_sel_blk[kMaxRows][kTopK]; + __shared__ int s_sel_neg[kMaxRows][kTopK]; + __shared__ int s_last; + __shared__ int32_t s_sb[kMaxRows][32]; + __shared__ __align__(8) uint64_t s_selbar; + + int const s = blockIdx.x, b = blockIdx.y, splits = gridDim.x; + int const tid = threadIdx.x, warp = tid >> 5, lane = tid & 31; + int const g = lane >> 2, tq = lane & 3; + uint32_t const full0 = smem_u32(bars), empty0 = smem_u32(bars + NSTAGE); + + if (tid == 0) + { + for (int i = 0; i < NSTAGE; ++i) + { + mbar_init(full0 + 8 * i, 1); + mbar_init(empty0 + 8 * i, 4); + } + mbar_init(smem_u32(&s_selbar), 1); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + } + if (warp == kConsumerWarps && lane == 0) + asm volatile("prefetch.tensormap [%0];" ::"l"(reinterpret_cast(&tmap)) : "memory"); + __syncthreads(); + + // The producer lanes fetch their first page ids together with seq_lens (bounded by the table width), so a CTA + // pays one dependent round trip before its loads go out. That happens, and the loads of the pages that hold only + // tokens older than this step go out, before the grid dependency wait. + int pid_first = 0; + int seqlen = 0; + uint64_t policy = 0; + if (warp == kConsumerWarps) + asm volatile("createpolicy.fractional.L2::evict_first.b64 %0, 1.0;" : "=l"(policy)); + int issued = 0; // producer lane: its first page is already in flight + { + if (warp == kConsumerWarps && lane < NSTAGE) + { + int const blk = s + lane * splits; + if (blk < p.bt_width) + pid_first = __ldg(p.bt + static_cast(b) * p.bt_stride + blk); + } + seqlen = p.seq_lens[b]; + if (warp == kConsumerWarps) + { + int const nb0 = seqlen > 0 ? min((seqlen + kBlockK - 1) / kBlockK, p.max_tiles) : 0; + int const my0 = s < nb0 ? (nb0 - s + splits - 1) / splits : 0; + int const stable = max(seqlen - p.dql, 0) / kBlockK; // blocks holding only older tokens + int const blk = s + lane * splits; + if (lane < min(my0, NSTAGE) && blk < stable) + { + mbar_expect_tx(full0 + 8 * lane, kStageBytes); + tma_load_page(smem_u32(stages + lane * kStageBytes), &tmap, pid_first, full0 + 8 * lane, policy); + issued = 1; + } + } + } + + asm volatile("griddepcontrol.wait;" ::: "memory"); + + // The select's per-row valid-block count, loaded by every CTA alongside seq_lens (only the last one uses it). + int const sel_tpr = p.dql * p.kv_heads <= 4 ? kConsumerThreads / 4 : kConsumerThreads / 8; + int const nv_raw = tid < kConsumerThreads && tid / sel_tpr < p.dql * p.kv_heads + ? p.n_valid[b * p.dql + (tid / sel_tpr) % p.dql] + : 0; + int const nb = seqlen > 0 ? min((seqlen + kBlockK - 1) / kBlockK, p.max_tiles) : 0; + if (nb == 0) + { + // Nothing to score: every row of the request is all -1. + if (s == 0 && tid < p.dql * p.kv_heads * kTopK) + { + int const r = tid / kTopK, j = tid % kTopK; + int const t = b * p.dql + r % p.dql, kvh = r / p.dql; + p.out[static_cast(kvh) * p.out_h + static_cast(t) * p.out_t + j] = -1; + } + return; + } + if (s >= nb) + return; + int const parts = min(splits, nb); + int const my = (nb - s + splits - 1) / splits; + int const q_start = seqlen - p.dql; + int const group = p.heads / p.kv_heads; + int const rows = p.dql * p.kv_heads; + + if (warp == kConsumerWarps) + { + if (lane < min(my, NSTAGE) && !issued) + { + mbar_expect_tx(full0 + 8 * lane, kStageBytes); + tma_load_page(smem_u32(stages + lane * kStageBytes), &tmap, pid_first, full0 + 8 * lane, policy); + } + if (lane == 0) + { + for (int j = NSTAGE; j < my; ++j) + { + int const st = j % NSTAGE; + int const page = __ldg(p.bt + static_cast(b) * p.bt_stride + s + j * splits); + mbar_wait(empty0 + 8 * st, ((j / NSTAGE) - 1) & 1); + mbar_expect_tx(full0 + 8 * st, kStageBytes); + tma_load_page(smem_u32(stages + st * kStageBytes), &tmap, page, full0 + 8 * st, policy); + } + } + } + else + { + // Q fragments: column g is (head = g / kMaxDql, row = g % kMaxDql). + uint32_t qlo[4][2], qhi[4][2]; + { + int const head = g / kMaxDql, qpos = g % kMaxDql; + bool const valid = head < p.heads && qpos < p.dql; + uint8_t const* qrow = p.q + static_cast(b * p.dql + qpos) * p.q_st + head * p.q_sh; + uint32_t w0[4], w1[4]; +#pragma unroll + for (int k = 0; k < 4; ++k) + { + w0[k] = valid ? *reinterpret_cast(qrow + 32 * k + 4 * tq) : 0u; + w1[k] = valid ? *reinterpret_cast(qrow + 32 * k + 16 + 4 * tq) : 0u; + } +#pragma unroll + for (int k = 0; k < 4; ++k) + { + qlo[k][0] = cvt_e4m3x2(w0[k]); + qhi[k][0] = cvt_e4m3x2(w0[k] >> 16); + qlo[k][1] = cvt_e4m3x2(w1[k]); + qhi[k][1] = cvt_e4m3x2(w1[k] >> 16); + } + } + int const slice = warp & 3; +#pragma unroll 1 + for (int lb = warp >> 2; lb < my; lb += kConsumerWarps / 4) + { + int const st = lb % NSTAGE; + mbar_wait(full0 + 8 * st, (lb / NSTAGE) & 1); + uint8_t const* stg = stages + st * kStageBytes; + float acc[2][4]; +#pragma unroll + for (int m = 0; m < 2; ++m) +#pragma unroll + for (int e = 0; e < 4; ++e) + acc[m][e] = 0.f; +#pragma unroll + for (int k = 0; k < 4; ++k) + { +#pragma unroll + for (int m = 0; m < 2; ++m) + { + int const ra = 32 * slice + 16 * m + g; // rb = ra + 8 has the same (row & 7) + int const sw = ra & 7; + uint8_t const* pa = stg + ra * 128; + uint8_t const* pb = pa + 8 * 128; + uint32_t const d0 = *reinterpret_cast(pa + (((2 * k) ^ sw) << 4) + 4 * tq); + uint32_t const d1 = *reinterpret_cast(pb + (((2 * k) ^ sw) << 4) + 4 * tq); + uint32_t const d2 = *reinterpret_cast(pa + (((2 * k + 1) ^ sw) << 4) + 4 * tq); + uint32_t const d3 = *reinterpret_cast(pb + (((2 * k + 1) ^ sw) << 4) + 4 * tq); + uint32_t const l0 = cvt_e4m3x2(d0), l1 = cvt_e4m3x2(d1), l2 = cvt_e4m3x2(d2), l3 = cvt_e4m3x2(d3); + uint32_t const h0 = cvt_e4m3x2(d0 >> 16), h1 = cvt_e4m3x2(d1 >> 16), h2 = cvt_e4m3x2(d2 >> 16), + h3 = cvt_e4m3x2(d3 >> 16); + mma16816(acc[m], l0, l1, l2, l3, qlo[k][0], qlo[k][1]); + mma16816(acc[m], h0, h1, h2, h3, qhi[k][0], qhi[k][1]); + } + } + __syncwarp(); + if (lane == 0) + mbar_arrive(empty0 + 8 * st); + // Causal mask + max over this warp's 32 positions. + int const blk = s + lb * splits; + int const pos0 = blk * kBlockK + 32 * slice; +#pragma unroll + for (int j = 0; j < 2; ++j) + { + int const col = 2 * tq + j; + int const qabs = q_start + col % kMaxDql; + float v = -INFINITY; +#pragma unroll + for (int m = 0; m < 2; ++m) + { + float const a = qabs >= pos0 + 16 * m + g ? acc[m][j] : -INFINITY; + float const c = qabs >= pos0 + 16 * m + g + 8 ? acc[m][2 + j] : -INFINITY; + v = fmaxf(v, fmaxf(a, c)); + } +#pragma unroll + for (int o = 4; o < 32; o <<= 1) + v = fmaxf(v, __shfl_xor_sync(0xFFFFFFFFu, v, o)); + if (g == 0) + epi[(lb * 4 + slice) * kCols + col] = v; + } + } + } + __syncthreads(); + + // Per block: max over the 4 slices, amax over each KV head's index heads, row-major scratch stores. + for (int lb = tid; lb < my; lb += kThreads) + { + int const blk = s + lb * splits; + float v[kCols]; +#pragma unroll + for (int c = 0; c < kCols; ++c) + { + float x = epi[(lb * 4) * kCols + c]; +#pragma unroll + for (int sl = 1; sl < 4; ++sl) + x = fmaxf(x, epi[(lb * 4 + sl) * kCols + c]); + v[c] = x; + } + float* dst = p.scr + b * p.scr_b + blk; +#pragma unroll + for (int r = 0; r < kMaxRows; ++r) + { + if (r < rows) + { + int const kvh = r / p.dql, qpos = r % p.dql; + float x = -INFINITY; +#pragma unroll + for (int c = 0; c < kCols; ++c) + if (c / kMaxDql >= kvh * group && c / kMaxDql < (kvh + 1) * group && c % kMaxDql == qpos) + x = fmaxf(x, v[c]); + dst[r * p.scr_r] = x; + } + } + } + __syncthreads(); + + __syncthreads(); + if (tid == 0) + { + int old; + asm volatile("atom.add.acq_rel.gpu.s32 %0, [%1], 1;" : "=r"(old) : "l"(p.counters + b) : "memory"); + s_last = old == parts - 1; + } + __syncthreads(); + if (!s_last) + return; + asm volatile("griddepcontrol.launch_dependents;"); + + // ---- last CTA of request b: top-16 of every (query row, KV head) ---- + // Row r is owned by tpr threads (2 warps for <= 4 rows, 1 warp for <= 8); thread rt of the row takes blocks + // rt, rt + tpr, ... and stages their forced keys in the (now idle) stage ring. + if (tid == 0) + p.counters[b] = 0; + if (tid < kMaxRows) + s_cnt[tid] = 0; + int const tpr = sel_tpr; + int const r = tid / tpr, rt = tid % tpr; + bool const active = tid < kConsumerThreads && r < rows; + int const ldk = (p.max_tiles + 3) & ~3; + uint32_t* keys = reinterpret_cast(stages) + (active ? r : 0) * ldk; + // One bulk copy per row brings the request's block scores (blocks [0, nb), contiguous per row) into the stage + // ring, so the whole row arrives in a single L2 round trip. + uint32_t const selbar = smem_u32(&s_selbar); + if (tid == 0) + { + asm volatile("fence.proxy.async.global;" ::: "memory"); + uint32_t const bytes = static_cast((nb + 3) & ~3) * 4u; + mbar_expect_tx(selbar, bytes * rows); + for (int rr = 0; rr < rows; ++rr) + { + asm volatile( + "cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes [%0], [%1], %2, [%3];" ::"r"( + smem_u32(stages) + static_cast(rr * ldk) * 4u), + "l"(p.scr + b * p.scr_b + rr * p.scr_r), "r"(bytes), "r"(selbar) + : "memory"); + } + } + int n = 0, lst = 0; + uint32_t mx = 0u; + if (active) + { + n = max(0, min(nv_raw, p.max_tiles)); + lst = max(nv_raw - p.local_blocks, 0); + mbar_wait(selbar, 0); + float const* rowf = reinterpret_cast(keys); +#pragma unroll 1 + for (int i0 = rt; i0 < n; i0 += 8 * tpr) + { + float x[8]; +#pragma unroll + for (int u = 0; u < 8; ++u) + x[u] = i0 + u * tpr < n ? rowf[i0 + u * tpr] : 0.f; +#pragma unroll + for (int u = 0; u < 8; ++u) + { + int const i = i0 + u * tpr; + if (i < n) + { + uint32_t const k = forced_key(x[u], i, p.init_blocks, lst); + keys[i] = k; + mx = max(mx, k); + } + } + } + } + // T0 = 16th largest thread maximum of the row: 16 threads each hold a key >= T0, so every top-16 key (ties + // included) is >= T0 (a row with fewer than 16 keys gets T0 = 0, i.e. all of them). + uint32_t const sorted = warp_sort_desc(mx, lane); + uint32_t t0 = __shfl_sync(0xFFFFFFFFu, sorted, kTopK - 1); + if (tpr == 64 && active && lane < kTopK) + s_wtop[r][rt >> 5][lane] = sorted; + __syncthreads(); + uint32_t* cand_key = reinterpret_cast(epi); // [rows][kCandCap] + int32_t* cand_blk = reinterpret_cast(epi) + kMaxRows * kCandCap; + if (active) + { + if (tpr == 64) + { + // 16th largest of two descending 16-lists A, B: max over i of min(A[i-1], B[15-i]), A[-1] = B[-1] = inf. + uint32_t const* A = s_wtop[r][0]; + uint32_t const* B = s_wtop[r][1]; + uint32_t x = 0u; + if (lane <= kTopK) + x = min(lane > 0 ? A[lane - 1] : 0xFFFFFFFFu, lane < kTopK ? B[kTopK - 1 - lane] : 0xFFFFFFFFu); + t0 = __reduce_max_sync(0xFFFFFFFFu, x); + } +#pragma unroll 1 + for (int i0 = rt; i0 < n; i0 += 8 * tpr) + { + uint32_t kk[8]; +#pragma unroll + for (int u = 0; u < 8; ++u) + kk[u] = i0 + u * tpr < n ? keys[i0 + u * tpr] : 0u; + uint32_t hit = 0; +#pragma unroll + for (int u = 0; u < 8; ++u) + hit |= static_cast(kk[u] >= t0 && kk[u] != 0u) << u; + if (hit) + { +#pragma unroll + for (int u = 0; u < 8; ++u) + { + if ((hit >> u) & 1u) + { + int const at = atomicAdd(&s_cnt[r], 1); + if (at < kCandCap) + { + cand_key[r * kCandCap + at] = kk[u]; + cand_blk[r * kCandCap + at] = i0 + u * tpr; + } + } + } + } + } + } + __syncthreads(); + // One warp per row: exact rank by (key desc, block asc), drop -inf, emit ascending ids with a -1 tail. + if (warp < rows) + { + int const rr = warp; + int const t = b * p.dql + rr % p.dql, kvh = rr / p.dql; + int32_t* dst = p.out + static_cast(kvh) * p.out_h + static_cast(t) * p.out_t; + int const c = s_cnt[rr]; + uint32_t const neg_key = twiddle(-INFINITY); + if (c <= 32) + { + // Lane j ranks candidate j against all of them (smem broadcasts), then places the selected ids by + // counting the smaller selected ids. + uint32_t const* ck = cand_key + rr * kCandCap; + int32_t const* cb = cand_blk + rr * kCandCap; + uint32_t const kj = lane < c ? ck[lane] : 0u; + int const bj = lane < c ? cb[lane] : 0x7FFFFFFF; + int rank = 0; +#pragma unroll 8 + for (int o = 0; o < 32; ++o) + { + uint32_t const ko = ck[o]; + int const bo = cb[o]; + rank += (o < c) & ((ko > kj) | ((ko == kj) & (bo < bj))); + } + bool const sel = lane < c && rank < kTopK && kj != neg_key; + int const sb = sel ? bj : 0x7FFFFFFF; + s_sb[rr][lane] = sb; + __syncwarp(); + int pos = 0; +#pragma unroll 8 + for (int o = 0; o < 32; ++o) + pos += s_sb[rr][o] < sb; + if (sel) + dst[pos] = bj; + int const cnt = __popc(__ballot_sync(0xFFFFFFFFu, sel)); + if (lane >= cnt && lane < kTopK) + dst[lane] = -1; + } + else if (c <= kCandCap) + { + for (int j = lane; j < c; j += 32) + { + uint32_t const kj = cand_key[rr * kCandCap + j]; + int const bj = cand_blk[rr * kCandCap + j]; + int rank = 0; + for (int o = 0; o < c; ++o) + { + uint32_t const ko = cand_key[rr * kCandCap + o]; + rank += (ko > kj) || (ko == kj && cand_blk[rr * kCandCap + o] < bj); + } + if (rank < kTopK) + { + s_sel_blk[rr][rank] = bj; + s_sel_neg[rr][rank] = kj == neg_key; + } + } + __syncwarp(); + bool const has = lane < kTopK && !s_sel_neg[rr][lane & 15]; + int const sk = has ? s_sel_blk[rr][lane & 15] : 0x7FFFFFFF; + int pos = 0; +#pragma unroll + for (int l = 0; l < kTopK; ++l) + { + int const o = __shfl_sync(0xFFFFFFFFu, sk, l); + pos += (o < sk) || (o == sk && l < lane); + } + if (lane < kTopK) + dst[pos] = sk == 0x7FFFFFFF ? -1 : sk; + } + else + { + // Massive exact ties: exact radix select over the staged keys. + int const nrr = max(0, min(__ldcg(p.n_valid + t), p.max_tiles)); + uint32_t const* kr = reinterpret_cast(stages) + rr * ldk; + uint32_t* hist = reinterpret_cast(stages) + rows * ldk + rr * 256; + radix_emit(kr, nrr, hist, dst, neg_key); + } + } +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +} + +// TMA descriptor of the [pages, 128, 128] e4m3 index-K pool (one page per box, 128-byte swizzle), cached per +// (base, pages, strides). +bool make_page_map(CUtensorMap& map, void const* base, int64_t pages, int64_t stride_row, int64_t stride_page) +{ + using Key = std::tuple; + static std::mutex mu; + static std::map cache; + Key const key{reinterpret_cast(base), pages, stride_row, stride_page}; + std::lock_guard lock(mu); + auto it = cache.find(key); + if (it != cache.end()) + { + map = it->second; + return true; + } + cuuint64_t dims[3] + = {static_cast(kHeadDim), static_cast(kBlockK), static_cast(pages)}; + cuuint64_t strides[2] = {static_cast(stride_row), static_cast(stride_page)}; + cuuint32_t box[3] = {kHeadDim, kBlockK, 1}; + cuuint32_t estr[3] = {1, 1, 1}; + CUresult const res = tensorrt_llm::common::CUDADriverWrapper::getInstance()->cuTensorMapEncodeTiled(&map, + CU_TENSOR_MAP_DATA_TYPE_UINT8, 3, const_cast(base), dims, strides, box, estr, + CU_TENSOR_MAP_INTERLEAVE_NONE, CU_TENSOR_MAP_SWIZZLE_128B, CU_TENSOR_MAP_L2_PROMOTION_L2_256B, + CU_TENSOR_MAP_FLOAT_OOB_FILL_NONE); + if (res != CUDA_SUCCESS) + { + return false; + } + if (cache.size() > 64) + { + cache.clear(); + } + cache.emplace(key, map); + return true; +} + +constexpr size_t kSmemBytes = 1024 + kNStage * kStageBytes + kEpiBytes + 2 * kNStage * 8; + +cudaError_t configure() +{ + return cudaFuncSetAttribute(m3_fused_score_select_kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, + static_cast(kSmemBytes)); +} + +} // namespace + +cudaError_t invokeMinimaxM3Tp8FusedScoreSelect(MinimaxM3Tp8FusedScoreSelectParams const& params, void const* kPages, + int64_t numPages, int64_t kRowStride, int64_t kPageStride, int32_t batch, int32_t splits, bool usePdl, + cudaStream_t stream) +{ + if (batch < 1 || batch > kMinimaxM3Tp8FusedScoreSelectMaxBatch || splits < 1 + || (params.max_tiles + splits - 1) / splits > kMaxLocal) + { + return cudaErrorInvalidValue; + } + CUtensorMap map; + if (!make_page_map(map, kPages, numPages, kRowStride, kPageStride)) + { + return cudaErrorInvalidValue; + } + // Kernel attributes are set once per process, on the first launch (outside CUDA-graph capture: warmup runs the + // kernel eagerly first). + static cudaError_t const configured = configure(); + if (configured != cudaSuccess) + { + return configured; + } + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(splits, batch); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = kSmemBytes; + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = usePdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, m3_fused_score_select_kernel, map, params); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.h b/cpp/tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.h new file mode 100644 index 000000000000..ad92b71bac21 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.h @@ -0,0 +1,77 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Largest number of requests (decode spans) one fused score + select launch takes. +constexpr int32_t kMinimaxM3Tp8FusedScoreSelectMaxBatch = 2; +//! Largest decode query length (query rows per request). +constexpr int32_t kMinimaxM3Tp8FusedScoreSelectMaxDql = 4; +//! Largest number of selected rows per request (num_kv_heads * dql). +constexpr int32_t kMinimaxM3Tp8FusedScoreSelectMaxRows = 8; +//! Largest number of blocks one CTA scores (ceil(max_tiles / splits) must not exceed it). +constexpr int32_t kMinimaxM3Tp8FusedScoreSelectMaxLocalBlocks = 128; +//! Bytes of the CTA's TMA stage ring, reused by the select: rows * ceil4(max_tiles) * 4 + rows * 1024 must fit. +constexpr int64_t kMinimaxM3Tp8FusedScoreSelectRingBytes = 12 * 128 * 128; + +//! Operands of the MiniMax-M3 TP8 fused index score + top-16 block select. Strides are in elements. +struct MinimaxM3Tp8FusedScoreSelectParams +{ + uint8_t const* q; // e4m3 [total_q, heads, 128] + int64_t q_st, q_sh; + int32_t const* bt; // [batch, bt_width] + int64_t bt_stride; + int bt_width; + int32_t const* seq_lens; // [batch], including this step's dql query rows + int32_t const* n_valid; // [total_q] causal valid-block counts + float* scr; // [batch][rows][>= ceil4(max_tiles)] block scores + int64_t scr_b, scr_r; + int32_t* out; // logical [total_q, kv_heads, 16] + int64_t out_t, out_h; + int32_t* counters; // [batch], zero at rest + int heads, kv_heads, dql, max_tiles, init_blocks, local_blocks; +}; + +//! One launch of index scoring and top-16 selection for at most 2 decode requests. Grid (splits, batch), one CTA of +//! 8 consumer warps + 1 TMA producer warp per SM: CTA (s, b) TMA-loads the e4m3 [128, 128] index-K pages of blocks +//! s, s + splits, ... of request b (12-stage ring; the pages holding only tokens older than this step are requested +//! before the grid dependency wait), scores them against the request's q rows on FP16 tensor cores, and stores each +//! block's score (max over its causal positions, amax over each KV head's index heads) to the scratch. The last CTA +//! of a request (per-request arrival counter) releases PDL dependents, resets that counter to zero, and selects the +//! top 16 of every (query row, KV head) row exactly like minimax_m3_select_blocks (init / local forcing, ties to the +//! smaller block, -inf picks dropped, ascending ids with a -1 tail). +//! +//! kPages is the e4m3 index-K page pool [numPages, 128, 128] with the given byte strides (row stride and page stride +//! multiples of 16, base 16-byte aligned); its TMA descriptor is encoded on first use and cached per (base, pages, +//! strides). The device code is SM100-family only; on other architectures the kernel compiles to an empty body. +cudaError_t invokeMinimaxM3Tp8FusedScoreSelect(MinimaxM3Tp8FusedScoreSelectParams const& params, void const* kPages, + int64_t numPages, int64_t kRowStride, int64_t kPageStride, int32_t batch, int32_t splits, bool usePdl, + cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.cu b/cpp/tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.cu new file mode 100644 index 000000000000..5d5b0bf605d2 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.cu @@ -0,0 +1,610 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3 TP8 long-context sparse decode attention in one launch (8 query heads on one KV head). +// +// Grid: one cluster of 16 CTAs per query token; CTA `r` attends the token's `r`-th selected block. The QK and PV +// MMAs are transposed (S^T = K q^T, O^T = V^T P^T) so the 8 heads fill the MMA n dimension. Each CTA pushes its +// normalized partial over DSMEM so that CTA `r` holds (head r / 2, dims (r % 2) * 64 + [0, 64)) of all 16 partials, +// merges them, and writes the BF16 row and, optionally, its MXFP8 copy and swizzled UE8M0 scales. +// +// Per-block math: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and PV on FP16 tensor cores with +// FP32 accumulation, log2-domain softmax with either the linear denominator or the log-domain recurrence. +// +// PREWAIT stages everything that does not depend on the top-16 table before the grid dependency wait: the request's +// seq_lens entry, the q rows and the request's block-table row (in dynamic shared memory after Smem8, when it has at +// most kBtCap entries). The arithmetic is the same in both variants. + +#include "tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kDim = 128; +constexpr int kPage = 128; +constexpr int kTopk = 16; +constexpr int kThreads = 128; +constexpr int kPitch = kDim + 8; // halves per smem row: 272 B, conflict-free ldmatrix +constexpr float kNegInf = -__builtin_huge_valf(); +constexpr int kHeads8 = 8; +constexpr int kHalfDim = kDim / 2; +constexpr int kBtCap = 8192; // block-table entries staged in shared memory (8192 x 128 = 1M tokens) + +using Params = MinimaxM3Tp8SparseDecodeLc8Params; + +struct Smem8 +{ + __half qs[kHeads8 * kPitch]; + __half vs[kPage * kPitch]; + __half ps[kHeads8 * kPitch]; // P^T: [head][token] + float recv[kTopk][kHalfDim]; // (head rank / 2, dims (rank % 2) * 64 + [0, 64)) partials of the 16 CTAs + float recv_lse[kTopk]; + float part[kHalfDim]; + float red[4][kHeads8]; + float rowm[kHeads8]; + float rowl[kHeads8]; +}; + +template +constexpr size_t smemBytes() +{ + return PREWAIT ? sizeof(Smem8) + kBtCap * sizeof(int) : sizeof(Smem8); +} + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t e2m1x2_to_f16x2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 s;\n cvt.u8.u32 s, %1;\n cvt.rn.f16x2.e2m1x2 %0, s;\n}" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t e4m3x2_to_f16x2(uint16_t pair) +{ + uint32_t out; + asm("cvt.rn.f16x2.e4m3x2 %0, %1;" : "=r"(out) : "h"(pair)); + return out; +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ void ldsm_x4(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x4_t(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma(float (&c)[4], uint32_t const (&a)[4], uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};" + : "+f"(c[0]), "+f"(c[1]), "+f"(c[2]), "+f"(c[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} + +__device__ __forceinline__ void cluster_sync() +{ + asm volatile("barrier.cluster.arrive.release.aligned;\nbarrier.cluster.wait.acquire.aligned;" ::: "memory"); +} + +__device__ __forceinline__ uint32_t map_cluster(void const* local, uint32_t rank) +{ + uint32_t remote; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(remote) : "r"(smem_u32(local)), "r"(rank)); + return remote; +} + +__device__ __forceinline__ void st_cluster_f32(uint32_t addr, float a) +{ + asm volatile("st.shared::cluster.f32 [%0], %1;" ::"r"(addr), "f"(a) : "memory"); +} + +// One packed NVFP4 row (64 bytes, 128 values) with its 8 E4M3 scales -> 128 fp16 values in smem. +__device__ __forceinline__ void dequant_row(__half* dst, uint8_t const* packed, uint8_t const (&sc)[8], bool valid) +{ + uint4 const* src = reinterpret_cast(packed); + uint32_t* d = reinterpret_cast(dst); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + uint32_t words[4] = {w.x, w.y, w.z, w.w}; + // 16 bytes = 32 values = scale columns 2c and 2c + 1. +#pragma unroll + for (int h = 0; h < 2; ++h) + { + uint8_t const s = valid ? sc[2 * c + h] : 0; + uint32_t const sh = e4m3x2_to_f16x2(static_cast(s | (s << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); +#pragma unroll + for (int j = 0; j < 2; ++j) + { + uint32_t const word = words[2 * h + j]; +#pragma unroll + for (int b = 0; b < 4; ++b) + { + uint32_t const v2 = e2m1x2_to_f16x2((word >> (8 * b)) & 0xFF); + __half2 const x = __hmul2(*reinterpret_cast<__half2 const*>(&v2), scale); + d[c * 16 + h * 8 + j * 4 + b] = *reinterpret_cast(&x); + } + } + } + } +} + +// q: 8 heads x 128 e4m3 -> fp16; threads 0..63 = (head, 16-value chunk). +__device__ __forceinline__ void load_q(Smem8& s, Params const& p, int tok, int tid) +{ + if (tid < kHeads8 * 8) + { + int const h = tid >> 3; + int const c = (tid & 7) * 16; + uint4 const w = *reinterpret_cast(p.q + tok * p.q_tok + h * p.q_head + c); + uint32_t const words[4] = {w.x, w.y, w.z, w.w}; + uint32_t* d = reinterpret_cast(s.qs + h * kPitch + c); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + d[2 * i] = e4m3x2_to_f16x2(static_cast(words[i] & 0xFFFF)); + d[2 * i + 1] = e4m3x2_to_f16x2(static_cast(words[i] >> 16)); + } + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +template +__global__ void __launch_bounds__(kThreads) m3_tp8_lc8_attn_kernel(__grid_constant__ Params const p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(128) uint8_t smem_raw[]; + Smem8& s = *reinterpret_cast(smem_raw); + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const g = lane >> 2; + int const qd = lane & 3; + int const tok = blockIdx.x / kTopk; + uint32_t rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(rank)); + + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + if constexpr (!PREWAIT) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + + int const req = tok / p.dql; + int const qoff = tok - req * p.dql; + int const seq = p.seq_lens[req]; + int const kv_len = max(seq - p.dql + qoff + 1, 0); + int const nblocks = (kv_len + kPage - 1) / kPage; + int const real_topk = min(kTopk, nblocks); + + // The request's block-table row, staged before the grid dependency (written before the graph runs). + int* const s_bt = reinterpret_cast(smem_raw + sizeof(Smem8)); + bool bt_smem = false; + if constexpr (PREWAIT) + { + int const nbt = (max(seq, 0) + kPage - 1) / kPage; + bt_smem = nbt <= kBtCap; + if (bt_smem) + { + for (int i = tid; i < nbt; i += kThreads) + { + s_bt[i] = __ldg(p.block_table + req * p.bt_stride + i); + } + } + load_q(s, p, tok, tid); + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + int blk = static_cast(rank) < real_topk ? p.topk[tok * p.t_tok + rank * p.t_k] : -1; + bool const active = blk >= 0; + if constexpr (PREWAIT) + { + __syncthreads(); // s_bt and the q rows are complete + } + else + { + load_q(s, p, tok, tid); + } + + // Lane holds heads 2 * qd + {0, 1}; acc[mt]: dims warp * 32 + mt * 16 + g + {0, 8}. + float m_row[2] = {kNegInf, kNegInf}; + float l_row[2] = {0.F, 0.F}; + float acc[2][4]; +#pragma unroll + for (int j = 0; j < 2; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + acc[j][i] = 0.F; + + // K rows n = warp * 32 + j * 8 + g stay packed in registers. As the MMA A operand, m-tile mt takes rows + // j = 2 mt (row g) and 2 mt + 1 (row g + 8); k-step ks takes packed bytes 8 ks + qd and 8 ks + 4 + qd. + uint32_t kw[4][16]; + uint2 ksc[4]; + if (active) + { + int64_t const page = bt_smem ? s_bt[blk] : p.block_table[req * p.bt_stride + blk]; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const n = warp * 32 + j * 8 + g; + bool const valid = blk * kPage + n < kv_len; + uint4 const* src = reinterpret_cast(p.k + page * p.k_blk + n * 64); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 const w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + kw[j][4 * c] = w.x; + kw[j][4 * c + 1] = w.y; + kw[j][4 * c + 2] = w.z; + kw[j][4 * c + 3] = w.w; + } + ksc[j] = valid ? __ldg(reinterpret_cast(p.ksc + page * p.ksc_blk + n * 8)) : make_uint2(0, 0); + } + bool const valid = blk * kPage + tid < kv_len; + uint8_t vsc[8]; + if (valid) + { + uint8_t const* vb = p.vsc + page * p.vsc_blk + (tid >> 2) * 32 + (tid & 3); +#pragma unroll + for (int c = 0; c < 8; ++c) + vsc[c] = __ldg(vb + 4 * c); + } + dequant_row(s.vs + tid * kPitch, p.v + page * p.v_blk + tid * 64, vsc, valid); + } + __syncthreads(); + + if (active) + { + float const scale_log2 = p.sm_scale * 1.4426950409F * __ldg(p.kgs); + int const q4 = qd * 8; + // S^T = K q^T for this warp's 32 tokens: rows (tokens) g / g + 8 of each m-tile, columns (heads) 2 qd + {0, 1}. + float sacc[2][4]; +#pragma unroll + for (int j = 0; j < 2; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + sacc[j][i] = 0.F; +#pragma unroll + for (int ks2 = 0; ks2 < kDim / 16; ks2 += 2) + { + uint32_t bq[4]; + ldsm_x4(bq, smem_u32(s.qs + (lane & 7) * kPitch + ks2 * 16 + (lane >> 3) * 8)); +#pragma unroll + for (int u = 0; u < 2; ++u) + { + int const ks = ks2 + u; +#pragma unroll + for (int mt = 0; mt < 2; ++mt) + { + uint32_t a[4]; +#pragma unroll + for (int hi = 0; hi < 2; ++hi) + { + int const j = 2 * mt + hi; + uint32_t const scw = ks < 4 ? ksc[j].x : ksc[j].y; + uint8_t const sc = static_cast(scw >> (8 * (ks & 3))); + uint32_t const sh = e4m3x2_to_f16x2(static_cast(sc | (sc << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); + uint32_t const r0 = e2m1x2_to_f16x2((kw[j][2 * ks] >> q4) & 0xFF); + uint32_t const r1 = e2m1x2_to_f16x2((kw[j][2 * ks + 1] >> q4) & 0xFF); + __half2 const h0 = __hmul2(*reinterpret_cast<__half2 const*>(&r0), scale); + __half2 const h1 = __hmul2(*reinterpret_cast<__half2 const*>(&r1), scale); + a[hi] = *reinterpret_cast(&h0); + a[2 + hi] = *reinterpret_cast(&h1); + } + mma(sacc[mt], a, bq[2 * u], bq[2 * u + 1]); + } + } + } + float mloc[2] = {kNegInf, kNegInf}; +#pragma unroll + for (int mt = 0; mt < 2; ++mt) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const t = warp * 32 + mt * 16 + g + 8 * (i >> 1); + float const v = blk * kPage + t < kv_len ? 0.F + sacc[mt][i] * scale_log2 : kNegInf; + sacc[mt][i] = v; + mloc[i & 1] = fmaxf(mloc[i & 1], v); + } +#pragma unroll + for (int e = 0; e < 2; ++e) + { + mloc[e] = fmaxf(mloc[e], __shfl_xor_sync(0xffffffff, mloc[e], 4)); + mloc[e] = fmaxf(mloc[e], __shfl_xor_sync(0xffffffff, mloc[e], 8)); + mloc[e] = fmaxf(mloc[e], __shfl_xor_sync(0xffffffff, mloc[e], 16)); + } + if (g == 0) + { + s.red[warp][2 * qd] = mloc[0]; + s.red[warp][2 * qd + 1] = mloc[1]; + } + __syncthreads(); + if (tid < kHeads8) + s.rowm[tid] = fmaxf(fmaxf(s.red[0][tid], s.red[1][tid]), fmaxf(s.red[2][tid], s.red[3][tid])); + __syncthreads(); + m_row[0] = s.rowm[2 * qd]; + m_row[1] = s.rowm[2 * qd + 1]; + float lloc[2] = {0.F, 0.F}; +#pragma unroll + for (int mt = 0; mt < 2; ++mt) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + float const pv = ex2(sacc[mt][i] - m_row[i & 1]); + lloc[i & 1] += pv; + int const t = warp * 32 + mt * 16 + g + 8 * (i >> 1); + s.ps[(2 * qd + (i & 1)) * kPitch + t] = __float2half_rn(pv); + } +#pragma unroll + for (int e = 0; e < 2; ++e) + { + lloc[e] += __shfl_xor_sync(0xffffffff, lloc[e], 4); + lloc[e] += __shfl_xor_sync(0xffffffff, lloc[e], 8); + lloc[e] += __shfl_xor_sync(0xffffffff, lloc[e], 16); + } + __syncthreads(); // rowm reads done before red is reused; ps complete + if (g == 0) + { + s.red[warp][2 * qd] = lloc[0]; + s.red[warp][2 * qd + 1] = lloc[1]; + } + // O^T[warp's 32 dims, 8 heads] = V^T P^T over all 128 tokens. +#pragma unroll + for (int kk = 0; kk < kPage; kk += 32) + { + uint32_t b[4]; + ldsm_x4(b, smem_u32(s.ps + (lane & 7) * kPitch + kk + (lane >> 3) * 8)); +#pragma unroll + for (int u = 0; u < 2; ++u) + { +#pragma unroll + for (int mt = 0; mt < 2; ++mt) + { + uint32_t a[4]; + int const t = kk + u * 16 + (lane & 7) + ((lane >> 4) & 1) * 8; + int const d = warp * 32 + mt * 16 + ((lane >> 3) & 1) * 8; + ldsm_x4_t(a, smem_u32(s.vs + t * kPitch + d)); + mma(acc[mt], a, b[2 * u], b[2 * u + 1]); + } + } + } + __syncthreads(); + if (tid < kHeads8) + s.rowl[tid] = (s.red[0][tid] + s.red[1][tid]) + (s.red[2][tid] + s.red[3][tid]); + __syncthreads(); + l_row[0] = s.rowl[2 * qd]; + l_row[1] = s.rowl[2 * qd + 1]; + } + + // Normalize; push (head, dim) to CTA 2 * head + dim / 64, slot recv[rank][dim % 64], plus each head's LSE (log2) + // to both of its CTAs. Posted DSMEM stores, made visible by the cluster barrier. + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); + { + float const vgs = __ldg(p.vgs); + float scale[2], lse[2]; +#pragma unroll + for (int e = 0; e < 2; ++e) + { + if constexpr (LINEAR) + { + scale[e] = l_row[e] > 0.F ? 1.0F / l_row[e] : 0.F; + lse[e] = l_row[e] > 0.F ? m_row[e] + lg2(l_row[e]) : kNegInf; + } + else + { + float const lse_i = active ? m_row[e] + lg2(ex2(kNegInf - m_row[e]) + l_row[e]) : kNegInf; + scale[e] = lse_i > kNegInf ? ex2(m_row[e] - lse_i) : 0.F; + lse[e] = lse_i; + } + scale[e] = scale[e] * vgs; + } + uint32_t dst[2]; +#pragma unroll + for (int e = 0; e < 2; ++e) + dst[e] = map_cluster(&s.recv[rank][0], (2 * qd + e) * 2 + (warp >> 1)); +#pragma unroll + for (int mt = 0; mt < 2; ++mt) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const dl = (warp & 1) * 32 + mt * 16 + g + 8 * (i >> 1); + st_cluster_f32(dst[i & 1] + dl * 4, acc[mt][i] * scale[i & 1]); + } + if (warp == 0 && g == 0) + { +#pragma unroll + for (int e = 0; e < 2; ++e) +#pragma unroll + for (int hf = 0; hf < 2; ++hf) + st_cluster_f32(map_cluster(&s.recv_lse[rank], (2 * qd + e) * 2 + hf), lse[e]); + } + } + cluster_sync(); + + // CTA `rank` merges head rank / 2, dims (rank % 2) * 64 + dl; thread = (chunk half, dl). + int const h = static_cast(rank) >> 1; + int const dh = static_cast(rank) & 1; + int const dl = tid & (kHalfDim - 1); + int const hc = tid >> 6; + float lse_c[kTopk]; + float lse_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + lse_c[c] = s.recv_lse[c]; + lse_max = fmaxf(lse_max, lse_c[c]); + } + lse_max = lse_max == kNegInf ? 0.F : lse_max; + float denom = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + lse_c[c] = ex2(lse_c[c] - lse_max); + denom += lse_c[c]; + } + denom = denom > 0.F ? denom : 1.F; + float o = 0.F; +#pragma unroll + for (int c = 0; c < kTopk / 2; ++c) + { + int const cc = hc * (kTopk / 2) + c; + o += s.recv[cc][dl] * __fdiv_rn(lse_c[cc], denom); + } + if (hc) + s.part[dl] = o; + __syncthreads(); + if (!hc) + { + o += s.part[dl]; + int const col = h * kDim + dh * kHalfDim + dl; + __nv_bfloat16 const ob = __float2bfloat16_rn(o); + p.out[tok * p.out_stride + col] = ob; + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + amax = fmaxf(amax, __shfl_xor_sync(0xffffffff, amax, off)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + float const xq = x * qs; + uint16_t q2; + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2) : "f"(0.F), "f"(xq)); + p.q8[static_cast(tok) * (p.num_sf_k_tiles * 128) + col] = static_cast(q2 & 0xFF); + if (lane == 0) + { + int const k_sf = col / 32; + int64_t const off = static_cast(tok / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (tok % 32) * 16 + ((tok % 128) / 32) * 4 + k_sf % 4; + p.sf[off] = static_cast(e); + } + } + } +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +} + +template +cudaError_t configure() +{ + auto kernel = m3_tp8_lc8_attn_kernel; + cudaError_t err = cudaFuncSetAttribute( + kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smemBytes())); + if (err != cudaSuccess) + { + return err; + } + return cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1); +} + +template +cudaError_t launch(Params const& prm, int num_tokens, bool use_pdl, cudaStream_t stream) +{ + // Kernel attributes are set once per process, on the first launch (outside CUDA-graph capture: warmup runs + // every variant eagerly first). + static cudaError_t const configured = configure(); + if (configured != cudaSuccess) + { + return configured; + } + auto kernel = m3_tp8_lc8_attn_kernel; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(num_tokens * kTopk); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smemBytes(); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kTopk; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +template +cudaError_t launchPrewait(Params const& prm, int num_tokens, bool prewait, bool use_pdl, cudaStream_t stream) +{ + return prewait ? launch(prm, num_tokens, use_pdl, stream) + : launch(prm, num_tokens, use_pdl, stream); +} + +} // namespace + +cudaError_t invokeMinimaxM3Tp8SparseDecodeLc8(MinimaxM3Tp8SparseDecodeLc8Params const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, cudaStream_t stream) +{ + if (linearSoftmax) + { + return mxfp8 ? launchPrewait(params, numTokens, prewait, usePdl, stream) + : launchPrewait(params, numTokens, prewait, usePdl, stream); + } + return mxfp8 ? launchPrewait(params, numTokens, prewait, usePdl, stream) + : launchPrewait(params, numTokens, prewait, usePdl, stream); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.h b/cpp/tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.h new file mode 100644 index 000000000000..ad65a577eab8 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.h @@ -0,0 +1,75 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Operands of the MiniMax-M3 TP8 one-launch long-context sparse decode. Strides are in elements of the pointed-to +//! type. Geometry is fixed: 8 query heads on one KV head, head_dim 128, page 128, top-k 16. +struct MinimaxM3Tp8SparseDecodeLc8Params +{ + uint8_t const* q; // e4m3 [T, 8, 128] + int64_t q_tok, q_head; + uint8_t const* k; // packed e2m1 [pages, 1, 128, 64] + uint8_t const* v; + int64_t k_blk, v_blk; + uint8_t const* ksc; // e4m3 [pages, 1, 128, 8], token-major + uint8_t const* vsc; // e4m3, 4x4 token-quad order + int64_t ksc_blk, vsc_blk; + float const* kgs; + float const* vgs; + int const* topk; // [T, 16] view (kv head 0) + int64_t t_tok, t_k; + int const* block_table; + int64_t bt_stride; + int const* seq_lens; + int dql; + float sm_scale; + __nv_bfloat16* out; // [T, 1024] + int64_t out_stride; + uint8_t* q8; // e4m3 [rows, num_sf_k_tiles * 128] + uint8_t* sf; + int num_sf_k_tiles; +}; + +//! One cluster of 16 CTAs per query token: CTA r attends the token's r-th selected block with transposed MMAs (the 8 +//! heads fill the MMA n dimension), then CTA r merges (head r / 2, dims (r % 2) * 64 + [0, 64)) of the 16 partials +//! over DSMEM into the BF16 [T, 1024] row (and, when mxfp8, its MXFP8 copy with swizzled UE8M0 scales). Dependents +//! are released right after the grid dependency wait. +//! +//! With prewait, the request's seq_lens entry, its q rows (widened to FP16) and its block-table row (up to 8192 +//! entries, in dynamic shared memory; longer rows are read from global memory) are staged before the grid dependency +//! wait; outputs are bit-identical to prewait=false. +//! +//! Instantiated variants: linearSoftmax x mxfp8 x prewait, all in {false, true}. The device code is SM100-family +//! only; on other architectures the kernel compiles to an empty body, so callers check the device first. +cudaError_t invokeMinimaxM3Tp8SparseDecodeLc8(MinimaxM3Tp8SparseDecodeLc8Params const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.cu b/cpp/tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.cu new file mode 100644 index 000000000000..f919dc72bf68 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.cu @@ -0,0 +1,283 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "chainSpeculativeSamplingKernels.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/fusedSampling/fusedSamplingKernels.h" + +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::speculative_decoding +{ + +namespace +{ + +constexpr int32_t kBlockSize = 1024; + +struct Float2SumOp +{ + __device__ __forceinline__ float2 operator()(float2 a, float2 b) const + { + return make_float2(a.x + b.x, a.y + b.y); + } +}; + +struct MaxOp +{ + __device__ __forceinline__ int32_t operator()(int32_t a, int32_t b) const + { + return a > b ? a : b; + } +}; + +template +__device__ __forceinline__ void loadProbs(float const* row, uint32_t base, uint32_t vocabSize, float (&out)[VEC_SIZE]) +{ + if constexpr (VEC_SIZE == 4) + { + if (base + VEC_SIZE <= vocabSize) + { + float4 const v = *reinterpret_cast(row + base); + out[0] = v.x; + out[1] = v.y; + out[2] = v.z; + out[3] = v.w; + return; + } + } +#pragma unroll + for (int32_t j = 0; j < VEC_SIZE; ++j) + { + out[j] = base + j < vocabSize ? row[base + j] : 0.0f; + } +} + +//! The distribution a request resamples from: max(0, target - draft) at the first rejected +//! position, or the target itself when every draft token was accepted (draft == nullptr) or +//! when the residual has no mass. +template +__device__ __forceinline__ void loadResidual( + float const* target, float const* draft, bool targetOnly, uint32_t base, uint32_t vocabSize, float (&out)[VEC_SIZE]) +{ + loadProbs(target, base, vocabSize, out); + if (targetOnly) + { + return; + } + float d[VEC_SIZE]; + loadProbs(draft, base, vocabSize, d); +#pragma unroll + for (int32_t j = 0; j < VEC_SIZE; ++j) + { + out[j] = fmaxf(out[j] - d[j], 0.0f); + } +} + +template +__global__ void __launch_bounds__(BLOCK_SIZE) chainSpeculativeSamplingKernel(ChainSpeculativeSamplingParams params) +{ + using BlockReduce = cub::BlockReduce; + using BlockReducePair = cub::BlockReduce; + using BlockReduceInt = cub::BlockReduce; + using BlockScan = cub::BlockScan; + + __shared__ union + { + typename BlockReduce::TempStorage reduce; + typename BlockReducePair::TempStorage reducePair; + typename BlockReduceInt::TempStorage reduceInt; + typename BlockScan::TempStorage scan; + } tempStorage; + + __shared__ int32_t sPos; + __shared__ float sU; + __shared__ bool sTargetOnly; + __shared__ float sTotal; + __shared__ float sTileSum[2]; + __shared__ int32_t sWinner; + __shared__ int32_t sLastValid; + + int64_t const row = blockIdx.x; + int32_t const tid = static_cast(threadIdx.x); + int32_t const numDraft = params.numDraftTokens; + uint32_t const vocabSize = static_cast(params.vocabSize); + int32_t* outTokens = params.outputTokenIds + row * (numDraft + 1); + + if (tid == 0) + { + curandStatePhilox4_32_10_t state; + curand_init(static_cast(params.seed[row]), + kPerRowSubsequenceBase | static_cast(params.offset[row]), 0, &state); + int32_t pos = numDraft; + for (int32_t i = 0; i < numDraft; ++i) + { + int32_t const token = params.draftTokenIds[row * numDraft + i]; + float const pTarget = params.targetProbs[(row * (numDraft + 1) + i) * vocabSize + token]; + float const pDraft = params.draftProbs[(row * numDraft + i) * vocabSize + token]; + if (curand_uniform(&state) * pDraft < pTarget) + { + outTokens[i] = token; + } + else + { + pos = i; + break; + } + } + sPos = pos; + // curand_uniform is in (0, 1]; the inverse-CDF search below wants [0, 1). + sU = 1.0f - curand_uniform(&state); + sWinner = static_cast(vocabSize); + } + __syncthreads(); + + int32_t const pos = sPos; + float const* target = params.targetProbs + (row * (numDraft + 1) + pos) * vocabSize; + float const* draft = pos < numDraft ? params.draftProbs + (row * numDraft + pos) * vocabSize : nullptr; + uint32_t const tileSize = BLOCK_SIZE * VEC_SIZE; + + // x: target mass, y: residual mass. + float2 sums = make_float2(0.0f, 0.0f); + for (uint32_t base = tid * VEC_SIZE; base < vocabSize; base += tileSize) + { + float t[VEC_SIZE]; + loadProbs(target, base, vocabSize, t); + float d[VEC_SIZE] = {}; + if (draft != nullptr) + { + loadProbs(draft, base, vocabSize, d); + } +#pragma unroll + for (int32_t j = 0; j < VEC_SIZE; ++j) + { + sums.x += t[j]; + sums.y += fmaxf(t[j] - d[j], 0.0f); + } + } + float2 const blockSums = BlockReducePair(tempStorage.reducePair).Reduce(sums, Float2SumOp{}); + if (tid == 0) + { + sTargetOnly = draft == nullptr || !(blockSums.y > 0.0f); + sTotal = sTargetOnly ? blockSums.x : blockSums.y; + } + __syncthreads(); + + // Inverse-CDF search: every tile is only summed, and the one tile the threshold falls in is + // scanned to find the token. + bool const targetOnly = sTargetOnly; + float const threshold = sU * sTotal; + float prefix = 0.0f; + int32_t localLastValid = -1; + int32_t parity = 0; + for (uint32_t tileBase = 0; tileBase < vocabSize; tileBase += tileSize, parity ^= 1) + { + uint32_t const base = tileBase + tid * VEC_SIZE; + float value[VEC_SIZE]; + loadResidual(target, draft, targetOnly, base, vocabSize, value); +#pragma unroll + for (int32_t j = 0; j < VEC_SIZE; ++j) + { + if (value[j] > 0.0f) + { + localLastValid = static_cast(base + j); + } + } + float const tileSum = BlockReduce(tempStorage.reduce).Sum(value); + if (tid == 0) + { + sTileSum[parity] = tileSum; + } + __syncthreads(); + if (prefix + sTileSum[parity] > threshold) + { + float inclusive[VEC_SIZE]; + BlockScan(tempStorage.scan).InclusiveSum(value, inclusive); +#pragma unroll + for (int32_t j = 0; j < VEC_SIZE; ++j) + { + if (value[j] > 0.0f && prefix + inclusive[j] > threshold) + { + atomicMin(&sWinner, static_cast(base + j)); + break; + } + } + __syncthreads(); + break; + } + prefix += sTileSum[parity]; + } + + if (sWinner == static_cast(vocabSize)) + { + // Float rounding can leave the summed prefix just short of the threshold; the last + // token with mass is then the one the threshold falls on. + int32_t const lastValid = BlockReduceInt(tempStorage.reduceInt).Reduce(localLastValid, MaxOp{}); + if (tid == 0) + { + sLastValid = lastValid; + } + } + + if (tid == 0) + { + int32_t const token = sWinner < static_cast(vocabSize) ? sWinner : max(sLastValid, 0); + outTokens[pos] = token; + for (int32_t i = pos + 1; i <= numDraft; ++i) + { + outTokens[i] = -1; + } + params.outputNumTokens[row] = pos + 1; + } +} + +bool isAligned(void const* ptr, uintptr_t alignment) +{ + return reinterpret_cast(ptr) % alignment == 0; +} + +} // namespace + +void invokeChainSpeculativeSampling(ChainSpeculativeSamplingParams const& params, cudaStream_t stream) +{ + if (params.batchSize == 0) + { + return; + } + dim3 const grid(params.batchSize); + dim3 const block(kBlockSize); + bool const vectorized = params.vocabSize % 4 == 0 && isAligned(params.targetProbs, sizeof(float4)) + && isAligned(params.draftProbs, sizeof(float4)); + if (vectorized) + { + chainSpeculativeSamplingKernel<<>>(params); + } + else + { + chainSpeculativeSamplingKernel<<>>(params); + } + sync_check_cuda_error(stream); +} + +} // namespace kernels::speculative_decoding + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.h b/cpp/tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.h new file mode 100644 index 000000000000..e3da5ec5b78f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.h @@ -0,0 +1,65 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::speculative_decoding +{ + +//! Chain speculative sampling: verifies each request's linear draft against the target. +//! +//! Draft token i is accepted with probability min(1, p_target / p_draft); the first rejected +//! position is replaced by a sample from norm(max(0, p_target - p_draft)), and when every +//! draft token is accepted a bonus token is sampled from the last target distribution. The +//! output distribution is the target's. +//! +//! Each request draws from the Philox subsequence its own offset names, so its result does not +//! depend on its row or on the other requests. +struct ChainSpeculativeSamplingParams +{ + //! [batchSize, numDraftTokens, vocabSize] + float const* draftProbs{nullptr}; + //! [batchSize, numDraftTokens] + int32_t const* draftTokenIds{nullptr}; + //! [batchSize, numDraftTokens + 1, vocabSize] + float const* targetProbs{nullptr}; + //! [batchSize] + int64_t const* seed{nullptr}; + //! [batchSize] + int64_t const* offset{nullptr}; + //! Output [batchSize, numDraftTokens + 1]: the accepted draft tokens, then the corrected or + //! bonus token, then -1. + int32_t* outputTokenIds{nullptr}; + //! Output [batchSize]: accepted draft tokens + 1. + int32_t* outputNumTokens{nullptr}; + int32_t batchSize{0}; + int32_t numDraftTokens{0}; + int32_t vocabSize{0}; +}; + +void invokeChainSpeculativeSampling(ChainSpeculativeSamplingParams const& params, cudaStream_t stream); + +} // namespace kernels::speculative_decoding + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/thop/CMakeLists.txt b/cpp/tensorrt_llm/thop/CMakeLists.txt index ab424318c862..5b23408cf16c 100644 --- a/cpp/tensorrt_llm/thop/CMakeLists.txt +++ b/cpp/tensorrt_llm/thop/CMakeLists.txt @@ -76,6 +76,38 @@ add_th_common_family( nvfp4MlaKvCacheGatherOp.cpp trtllmGenQKVProcessOp.cpp) +# MiniMax-M3.1 GB300 native dependencies; explicit to avoid duplicate +# registrations. +target_sources( + th_common_attention + PRIVATE minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp + minimaxM31/minimaxM31Sm103CtxNormMxfp8Op.cpp + minimaxM31/minimaxM31Sm103SelectPrefillTileOp.cpp + minimaxM31/minimaxM31SoftmaxMcOp.cpp + minimaxM31/minimaxM31ToppRenormDetOp.cpp + minimaxM31/minimaxM31ToppRenormDetV3Op.cpp + minimaxM31/minimaxM31Tp4SparseDecodeLcOp.cpp + minimaxM31/minimaxM31Tp8SparseDecodeLc8Op.cpp + chainSpeculativeSamplingOp.cpp + minimaxM3/minimaxM3AttnC1Op.cpp + minimaxM3/minimaxM3AttnMegaOp.cpp + minimaxM3/minimaxM3DraftStepOp.cpp + minimaxM3/minimaxM3FusedDecodeAttnOp.cpp + minimaxM3/minimaxM3McAllreduceNormOp.cpp + minimaxM3/minimaxM3MsaKvUpdateOp.cpp + minimaxM3/minimaxM3Mxfp8LowMGemmOp.cpp + minimaxM3/minimaxM3ProducerPwOp.cpp + minimaxM3/minimaxM3RouterGemvOp.cpp + minimaxM3/minimaxM3SampledAcceptOp.cpp + minimaxM3/minimaxM3SelectLongOp.cpp + minimaxM3/minimaxM3ShardAcceptOp.cpp + minimaxM3/minimaxM3SparseDecodeLcOp.cpp + minimaxM3/minimaxM3SpecTailOp.cpp + minimaxM3/minimaxM3TargetHeadOp.cpp + minimaxM3/minimaxM3Tp8FusedScoreSelectOp.cpp + minimaxM3/minimaxM3Tp8SparseDecodeLc8Op.cpp + minimaxM3/stepInputsScatterOp.cpp) + add_th_common_family( th_common_gemm_quant cutlassScaledMM.cpp diff --git a/cpp/tensorrt_llm/thop/chainSpeculativeSamplingOp.cpp b/cpp/tensorrt_llm/thop/chainSpeculativeSamplingOp.cpp new file mode 100644 index 000000000000..ae9b5cb80a2b --- /dev/null +++ b/cpp/tensorrt_llm/thop/chainSpeculativeSamplingOp.cpp @@ -0,0 +1,91 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/speculativeDecoding/chainSpeculativeSamplingKernels.h" + +#include +#include + +#include + +namespace torch_ext +{ + +namespace tk = tensorrt_llm::kernels::speculative_decoding; + +//! Returns (output_token_ids [batch, numDraft + 1], num_tokens [batch]); see +//! ChainSpeculativeSamplingParams for the semantics. +std::tuple chain_speculative_sampling(torch::Tensor const& draftProbs, + torch::Tensor const& draftTokenIds, torch::Tensor const& targetProbs, torch::Tensor const& seed, + torch::Tensor const& offset) +{ + TORCH_CHECK(draftProbs.dim() == 3, "draft_probs must be [batch, numDraft, vocab]"); + TORCH_CHECK(draftTokenIds.dim() == 2, "draft_token_ids must be [batch, numDraft]"); + TORCH_CHECK(targetProbs.dim() == 3, "target_probs must be [batch, numDraft + 1, vocab]"); + + auto const batchSize = draftProbs.size(0); + auto const numDraft = draftProbs.size(1); + auto const vocabSize = draftProbs.size(2); + TORCH_CHECK(draftTokenIds.size(0) == batchSize && draftTokenIds.size(1) == numDraft, + "draft_token_ids must be [batch, numDraft]"); + TORCH_CHECK( + targetProbs.size(0) == batchSize && targetProbs.size(1) == numDraft + 1 && targetProbs.size(2) == vocabSize, + "target_probs must be [batch, numDraft + 1, vocab]"); + + for (auto const* t : {&draftProbs, &draftTokenIds, &targetProbs, &seed, &offset}) + { + TORCH_CHECK(t->is_cuda() && t->is_contiguous(), "chain_speculative_sampling inputs must be contiguous CUDA"); + } + TORCH_CHECK(draftProbs.scalar_type() == torch::kFloat32 && targetProbs.scalar_type() == torch::kFloat32, + "draft_probs and target_probs must be float32"); + TORCH_CHECK(draftTokenIds.scalar_type() == torch::kInt32, "draft_token_ids must be int32"); + TORCH_CHECK(seed.scalar_type() == torch::kInt64 && offset.scalar_type() == torch::kInt64, + "seed and offset must be int64 tensors"); + TORCH_CHECK( + seed.numel() == batchSize && offset.numel() == batchSize, "seed and offset must hold one entry per request"); + + auto outputTokenIds = torch::empty({batchSize, numDraft + 1}, draftTokenIds.options()); + auto outputNumTokens = torch::empty({batchSize}, draftTokenIds.options()); + + tk::ChainSpeculativeSamplingParams params; + params.draftProbs = draftProbs.const_data_ptr(); + params.draftTokenIds = draftTokenIds.const_data_ptr(); + params.targetProbs = targetProbs.const_data_ptr(); + params.seed = seed.const_data_ptr(); + params.offset = offset.const_data_ptr(); + params.outputTokenIds = outputTokenIds.data_ptr(); + params.outputNumTokens = outputNumTokens.data_ptr(); + params.batchSize = static_cast(batchSize); + params.numDraftTokens = static_cast(numDraft); + params.vocabSize = static_cast(vocabSize); + tk::invokeChainSpeculativeSampling(params, at::cuda::getCurrentCUDAStream()); + return {outputTokenIds, outputNumTokens}; +} + +} // namespace torch_ext + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "chain_speculative_sampling(Tensor draft_probs, Tensor draft_token_ids, Tensor target_probs, " + "Tensor seed, Tensor offset) -> (Tensor, Tensor)"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("chain_speculative_sampling", &torch_ext::chain_speculative_sampling); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3AttnC1Op.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3AttnC1Op.cpp new file mode 100644 index 000000000000..cfc915f939a9 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3AttnC1Op.cpp @@ -0,0 +1,191 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/attnC1.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace dims = tensorrt_llm::kernels::minimax_m3_attn_c1; +using tensorrt_llm::kernels::MinimaxM3AttnC1Params; + +constexpr int64_t kHeadDim = 128; + +void checkByteTensor(at::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, name, " must be a CUDA tensor of one-byte elements"); +} + +} // namespace + +//! One launch for a one-request MiniMax-M3 MSA verify layer of 6 query tokens: the fused norm/RoPE/KV-insert + +//! identity-selection sparse decode + merge of minimax_m3_fused_decode_attn on the packed QKV rows, and the MXFP8 +//! o_proj GEMV, 96 CTAs that are each an attention CTA and an o_proj unit (fed through the MXFP8 hand-off). +//! +//! @param packed bf16 [6, 2560] QKV+index projection rows. +//! @param flags int32 [>= 32 + 96]: the per-CTA call counters (zero-initialized once); written. +//! @param mxfp8Data / mxfp8Scale the two call-parity buffers of the attention's MXFP8 hand-off (0xFF at rest). +//! @param y bf16 [T, 6144] o_proj output. arWs / arFlags / arRank / arWorld / arMaxTokens / arMode: the next +//! multicast all-reduce's exchange (device addresses) to push y into when arMode is 2; 0 for none. +void minimaxM3AttnC1(at::Tensor const& packed, at::Tensor& flags, at::Tensor& kvData, at::Tensor& kvScale, + at::Tensor& idxK, at::Tensor const& outCacheLoc, at::Tensor const& positionIds, at::Tensor const& blockTable, + at::Tensor const& seqLens, at::Tensor const& kvQuantScale, at::Tensor const& kGlobalScale, + at::Tensor const& vGlobalScale, at::Tensor const& qNormWeight, at::Tensor const& kNormWeight, + at::Tensor const& indexKNormWeight, at::Tensor const& rotaryCosSin, double eps, double smScale, + int64_t decodeQueryLen, int64_t tokensPerCluster, int64_t blocksPerCta, at::Tensor& output, at::Tensor& mxfp8Data, + at::Tensor& mxfp8Scale, bool usePdl, at::Tensor const& oWeight, at::Tensor const& oWeightScale, at::Tensor& y, + int64_t arWs, int64_t arFlags, int64_t arRank, int64_t arWorld, int64_t arMaxTokens, int64_t arMode) +{ + TORCH_CHECK(packed.is_cuda() && packed.scalar_type() == at::kBFloat16 && packed.dim() == 2 + && packed.size(1) == 20 * kHeadDim && packed.stride(1) == 1, + "packed must be CUDA bf16 [T, 2560] with contiguous rows"); + checkByteTensor(oWeight, "o_weight"); + checkByteTensor(oWeightScale, "o_weight_scale"); + checkByteTensor(kvData, "kv_data"); + checkByteTensor(kvScale, "kv_scale"); + checkByteTensor(idxK, "idx_k"); + checkByteTensor(mxfp8Data, "mxfp8_data"); + checkByteTensor(mxfp8Scale, "mxfp8_scale"); + TORCH_CHECK( + oWeight.dim() == 2 && oWeight.size(0) == dims::kOpN && oWeight.size(1) == dims::kOpK && oWeight.is_contiguous(), + "o_weight must be contiguous [6144, 2048]"); + TORCH_CHECK( + kvData.dim() == 5 && kvScale.dim() == 5 && idxK.dim() == 4, "kv_data, kv_scale and idx_k must be paged"); + for (auto const* t : {&outCacheLoc, &positionIds, &blockTable, &seqLens}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == at::kInt, + "slot, position, block-table and length tensors " + "must be CUDA int32"); + } + TORCH_CHECK(blockTable.dim() == 2 && blockTable.stride(1) == 1, "block_table must be row-major 2-D"); + for (auto const* t : {&kvQuantScale, &kGlobalScale, &vGlobalScale, &rotaryCosSin}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == at::kFloat, "scales and rotary_cos_sin must be CUDA fp32"); + } + for (auto const* t : {&qNormWeight, &kNormWeight, &indexKNormWeight}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == at::kBFloat16 && t->numel() == kHeadDim, + "norm weights must be CUDA bf16 [128]"); + } + TORCH_CHECK(flags.is_cuda() && flags.scalar_type() == at::kInt && flags.numel() >= 32 + dims::kCtas, + "bad c1 flags: int32 [>= 128]"); + TORCH_CHECK(mxfp8Data.numel() == 2 * dims::kMaxTokens * dims::kOpK && mxfp8Scale.numel() == 2 * dims::kOpSfBytes + && mxfp8Data.is_contiguous() && mxfp8Scale.is_contiguous(), + "c1 MXFP8 hand-off"); + int64_t const numTokens = packed.size(0); + int64_t const numReqs = seqLens.size(0); + TORCH_CHECK(numTokens == numReqs * decodeQueryLen && numTokens <= dims::kMaxTokens, "token count"); + TORCH_CHECK(output.is_cuda() && output.scalar_type() == at::kBFloat16 && output.dim() == 2 + && output.size(0) >= numTokens && output.size(1) == 16 * kHeadDim && output.stride(1) == 1, + "output must be CUDA bf16 [T, 2048] with contiguous rows"); + TORCH_CHECK(y.is_cuda() && y.scalar_type() == at::kBFloat16 && y.dim() == 2 && y.size(0) == numTokens + && y.size(1) == dims::kOpN && y.is_contiguous(), + "y must be [M, 6144]"); + TORCH_CHECK(decodeQueryLen == 6 && tokensPerCluster == 1 && blocksPerCta == 1 && numReqs == 1, + "minimax_m3_attn_c1 is compiled for one request with six query tokens (geometry (6, 1, 1)), got ", numReqs, + " x (", decodeQueryLen, ", ", tokensPerCluster, ", ", blocksPerCta, ")"); + TORCH_CHECK(arMode == 0 || arMode == 2, "ar_mode must be 0 (none) or 2 (push)"); + + c10::cuda::CUDAGuard const deviceGuard{packed.device()}; + auto const* prop = at::cuda::getCurrentDeviceProperties(); + TORCH_CHECK( + prop->major == 10, "minimax_m3_attn_c1 requires an sm_100-family GPU, got sm_", prop->major, prop->minor); + + MinimaxM3AttnC1Params prm = {}; + prm.kv_data = reinterpret_cast(kvData.data_ptr()); + prm.d_page = kvData.stride(0); + prm.d_plane = kvData.stride(1); + prm.kv_scale = reinterpret_cast(kvScale.data_ptr()); + prm.s_page = kvScale.stride(0); + prm.s_plane = kvScale.stride(1); + prm.idx_k = reinterpret_cast(idxK.data_ptr()); + prm.i_page = idxK.stride(0); + prm.i_tok = idxK.stride(2); + prm.num_pages = kvData.size(0); + prm.out_cache_loc = outCacheLoc.data_ptr(); + prm.position_ids = positionIds.data_ptr(); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.bt_cols = static_cast(blockTable.size(1)); + prm.seq_lens = seqLens.data_ptr(); + prm.kv_quant_scale = kvQuantScale.data_ptr(); + prm.k_gs = kGlobalScale.data_ptr(); + prm.v_gs = vGlobalScale.data_ptr(); + prm.q_w = reinterpret_cast<__nv_bfloat16 const*>(qNormWeight.data_ptr()); + prm.k_w = reinterpret_cast<__nv_bfloat16 const*>(kNormWeight.data_ptr()); + prm.ik_w = reinterpret_cast<__nv_bfloat16 const*>(indexKNormWeight.data_ptr()); + prm.rope = rotaryCosSin.data_ptr(); + prm.eps = static_cast(eps); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(output.data_ptr()); + prm.out_stride = output.stride(0); + prm.q8 = reinterpret_cast(mxfp8Data.data_ptr()); + prm.sf = reinterpret_cast(mxfp8Scale.data_ptr()); + prm.num_sf_k_tiles = static_cast(output.size(1) / 128); + prm.packed = reinterpret_cast<__nv_bfloat16 const*>(packed.data_ptr()); + prm.packed_stride = packed.stride(0); + prm.num_tokens = static_cast(numTokens); + prm.flags = reinterpret_cast(flags.data_ptr()); + prm.wsf_o = reinterpret_cast(oWeightScale.data_ptr()); + prm.y_out = reinterpret_cast<__nv_bfloat16*>(y.data_ptr()); + prm.ar_ws = reinterpret_cast(arWs); + prm.ar_flags = reinterpret_cast(arFlags); + prm.ar_rank = static_cast(arRank); + prm.ar_world = static_cast(arWorld); + prm.ar_maxt = static_cast(arMaxTokens); + prm.ar_mode = static_cast(arMode); + + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM3AttnC1(prm, oWeight.data_ptr(), + static_cast(decodeQueryLen), static_cast(tokensPerCluster), static_cast(blocksPerCta), + static_cast(numReqs), usePdl, at::cuda::getCurrentCUDAStream(packed.get_device())); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "minimax_m3_attn_c1 launch failed (qlen=", decodeQueryLen, " qg=", tokensPerCluster, + " bpc=", blocksPerCta, "): ", cudaGetErrorString(err)); + } +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_attn_c1(Tensor packed, Tensor(a!) flags, Tensor(b!) kv_data, Tensor(c!) kv_scale, " + "Tensor(d!) idx_k, Tensor out_cache_loc, Tensor position_ids, Tensor block_table, Tensor seq_lens, " + "Tensor kv_quant_scale, Tensor k_global_scale, Tensor v_global_scale, Tensor q_norm_weight, " + "Tensor k_norm_weight, Tensor index_k_norm_weight, Tensor rotary_cos_sin, float eps, float sm_scale, " + "int decode_query_len, int tokens_per_cluster, int blocks_per_cta, Tensor(e!) output, Tensor(f!) mxfp8_data, " + "Tensor(g!) mxfp8_scale, bool use_pdl, Tensor o_weight, Tensor o_weight_scale, Tensor(h!) y, " + "int ar_workspace, int ar_flags, int ar_rank, int ar_world, int ar_max_tokens, int ar_mode) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_attn_c1", &tensorrt_llm::torch_ext::minimaxM3AttnC1); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3AttnMegaOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3AttnMegaOp.cpp new file mode 100644 index 000000000000..b94942e052f5 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3AttnMegaOp.cpp @@ -0,0 +1,207 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/attnMega.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace dims = tensorrt_llm::kernels::minimax_m3_attn_mega; +using tensorrt_llm::kernels::MinimaxM3AttnMegaParams; + +constexpr int64_t kHeadDim = 128; + +void checkByteTensor(at::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, name, " must be a CUDA tensor of one-byte elements"); +} + +} // namespace + +//! One launch for a one-request, one-token MiniMax-M3 MSA decode layer: the MXFP8 QKV GEMV of the layer input, the +//! fused norm/RoPE/KV-insert + identity-selection sparse decode + merge of minimax_m3_fused_decode_attn, and the MXFP8 +//! o_proj GEMV, joined by per-tile ready counters in `flags`. +//! +//! @param xq / xsf the layer input's E4M3 rows [T, 6144] and 128x4-swizzled UE8M0 scale atoms. +//! @param qkvWeight / qkvWeightScale E4M3 [2560, 6144] and its swizzled UE8M0 scales. +//! @param qkvOut bf16 [2, 8, 2560] call-parity buffers of the packed QKV rows (0xFFFF "not yet" at rest); written. +//! @param flags int32 [>= 32 + 112]: tile counters and the per-CTA call counters (zero-initialized once); written. +//! @param mxfp8Data / mxfp8Scale the two call-parity buffers of the attention's MXFP8 hand-off (0xFF at rest). +//! @param y bf16 [T, 6144] o_proj output. arWs / arFlags / arRank / arWorld / arMaxTokens / arMode: the next +//! multicast all-reduce's exchange (device addresses) to push y into when arMode is 2; 0 for none. +void minimaxM3AttnMega(at::Tensor const& xq, at::Tensor const& xsf, at::Tensor const& qkvWeight, + at::Tensor const& qkvWeightScale, at::Tensor& qkvOut, at::Tensor& flags, at::Tensor& kvData, at::Tensor& kvScale, + at::Tensor& idxK, at::Tensor const& outCacheLoc, at::Tensor const& positionIds, at::Tensor const& blockTable, + at::Tensor const& seqLens, at::Tensor const& kvQuantScale, at::Tensor const& kGlobalScale, + at::Tensor const& vGlobalScale, at::Tensor const& qNormWeight, at::Tensor const& kNormWeight, + at::Tensor const& indexKNormWeight, at::Tensor const& rotaryCosSin, double eps, double smScale, + int64_t decodeQueryLen, int64_t tokensPerCluster, int64_t blocksPerCta, at::Tensor& output, at::Tensor& mxfp8Data, + at::Tensor& mxfp8Scale, bool usePdl, at::Tensor const& oWeight, at::Tensor const& oWeightScale, at::Tensor& y, + int64_t arWs, int64_t arFlags, int64_t arRank, int64_t arWorld, int64_t arMaxTokens, int64_t arMode) +{ + checkByteTensor(xq, "xq"); + checkByteTensor(xsf, "xsf"); + checkByteTensor(qkvWeight, "qkv_weight"); + checkByteTensor(qkvWeightScale, "qkv_weight_scale"); + checkByteTensor(oWeight, "o_weight"); + checkByteTensor(oWeightScale, "o_weight_scale"); + checkByteTensor(kvData, "kv_data"); + checkByteTensor(kvScale, "kv_scale"); + checkByteTensor(idxK, "idx_k"); + checkByteTensor(mxfp8Data, "mxfp8_data"); + checkByteTensor(mxfp8Scale, "mxfp8_scale"); + TORCH_CHECK(xq.dim() == 2 && xq.size(1) == dims::kQkvK && xq.stride(1) == 1, "xq must be [T, 6144] row-major"); + TORCH_CHECK(qkvWeight.dim() == 2 && qkvWeight.size(0) == dims::kQkvN && qkvWeight.size(1) == dims::kQkvK + && qkvWeight.is_contiguous(), + "qkv_weight must be contiguous [2560, 6144]"); + TORCH_CHECK( + oWeight.dim() == 2 && oWeight.size(0) == dims::kOpN && oWeight.size(1) == dims::kOpK && oWeight.is_contiguous(), + "o_weight must be contiguous [6144, 2048]"); + TORCH_CHECK( + kvData.dim() == 5 && kvScale.dim() == 5 && idxK.dim() == 4, "kv_data, kv_scale and idx_k must be paged"); + for (auto const* t : {&outCacheLoc, &positionIds, &blockTable, &seqLens}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == at::kInt, + "slot, position, block-table and length tensors " + "must be CUDA int32"); + } + TORCH_CHECK(blockTable.dim() == 2 && blockTable.stride(1) == 1, "block_table must be row-major 2-D"); + for (auto const* t : {&kvQuantScale, &kGlobalScale, &vGlobalScale, &rotaryCosSin}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == at::kFloat, "scales and rotary_cos_sin must be CUDA fp32"); + } + for (auto const* t : {&qNormWeight, &kNormWeight, &indexKNormWeight}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == at::kBFloat16 && t->numel() == kHeadDim, + "norm weights must be CUDA bf16 [128]"); + } + TORCH_CHECK(flags.is_cuda() && flags.scalar_type() == at::kInt && flags.numel() >= 32 + dims::kGrid, + "flags must be CUDA int32 [>= 144]"); + TORCH_CHECK(qkvOut.is_cuda() && qkvOut.element_size() == 2 && qkvOut.dim() == 3 && qkvOut.size(0) == 2 + && qkvOut.size(1) == dims::kMaxTokens && qkvOut.size(2) == dims::kQkvN && qkvOut.is_contiguous(), + "bad megakernel workspace: qkv_out must be contiguous [2, 8, 2560]"); + TORCH_CHECK(mxfp8Data.numel() == 2 * dims::kMaxTokens * dims::kOpK && mxfp8Scale.numel() == 2 * dims::kOpSfBytes + && mxfp8Data.is_contiguous() && mxfp8Scale.is_contiguous(), + "megakernel MXFP8 hand-off"); + int64_t const numTokens = xq.size(0); + int64_t const numReqs = seqLens.size(0); + TORCH_CHECK(numTokens == numReqs * decodeQueryLen && numTokens <= dims::kMaxTokens, "token count"); + TORCH_CHECK(output.is_cuda() && output.scalar_type() == at::kBFloat16 && output.dim() == 2 + && output.size(0) >= numTokens && output.size(1) == 16 * kHeadDim && output.stride(1) == 1, + "output must be CUDA bf16 [T, 2048] with contiguous rows"); + TORCH_CHECK(y.is_cuda() && y.scalar_type() == at::kBFloat16 && y.dim() == 2 && y.size(0) == numTokens + && y.size(1) == dims::kOpN && y.is_contiguous(), + "y must be [M, 6144]"); + TORCH_CHECK(decodeQueryLen == 1 && tokensPerCluster == 1 && blocksPerCta == 1 && numReqs == 1, + "minimax_m3_attn_mega is compiled for one request with one query token (geometry (1, 1, 1)), got ", numReqs, + " x (", decodeQueryLen, ", ", tokensPerCluster, ", ", blocksPerCta, ")"); + TORCH_CHECK(arMode == 0 || arMode == 2, "ar_mode must be 0 (none) or 2 (push)"); + + c10::cuda::CUDAGuard const deviceGuard{xq.device()}; + auto const* prop = at::cuda::getCurrentDeviceProperties(); + TORCH_CHECK( + prop->major == 10, "minimax_m3_attn_mega requires an sm_100-family GPU, got sm_", prop->major, prop->minor); + + MinimaxM3AttnMegaParams prm = {}; + prm.kv_data = reinterpret_cast(kvData.data_ptr()); + prm.d_page = kvData.stride(0); + prm.d_plane = kvData.stride(1); + prm.kv_scale = reinterpret_cast(kvScale.data_ptr()); + prm.s_page = kvScale.stride(0); + prm.s_plane = kvScale.stride(1); + prm.idx_k = reinterpret_cast(idxK.data_ptr()); + prm.i_page = idxK.stride(0); + prm.i_tok = idxK.stride(2); + prm.num_pages = kvData.size(0); + prm.out_cache_loc = outCacheLoc.data_ptr(); + prm.position_ids = positionIds.data_ptr(); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.bt_cols = static_cast(blockTable.size(1)); + prm.seq_lens = seqLens.data_ptr(); + prm.kv_quant_scale = kvQuantScale.data_ptr(); + prm.k_gs = kGlobalScale.data_ptr(); + prm.v_gs = vGlobalScale.data_ptr(); + prm.q_w = reinterpret_cast<__nv_bfloat16 const*>(qNormWeight.data_ptr()); + prm.k_w = reinterpret_cast<__nv_bfloat16 const*>(kNormWeight.data_ptr()); + prm.ik_w = reinterpret_cast<__nv_bfloat16 const*>(indexKNormWeight.data_ptr()); + prm.rope = rotaryCosSin.data_ptr(); + prm.eps = static_cast(eps); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(output.data_ptr()); + prm.out_stride = output.stride(0); + prm.q8 = reinterpret_cast(mxfp8Data.data_ptr()); + prm.sf = reinterpret_cast(mxfp8Scale.data_ptr()); + prm.num_sf_k_tiles = static_cast(output.size(1) / 128); + prm.xq = reinterpret_cast(xq.data_ptr()); + prm.ldxq = static_cast(xq.stride(0)); + prm.xsf_act = reinterpret_cast(xsf.data_ptr()); + prm.num_tokens = static_cast(numTokens); + prm.wsf = reinterpret_cast(qkvWeightScale.data_ptr()); + prm.qkv_out = reinterpret_cast<__nv_bfloat16*>(qkvOut.data_ptr()); + prm.flags = reinterpret_cast(flags.data_ptr()); + prm.wsf_o = reinterpret_cast(oWeightScale.data_ptr()); + prm.y_out = reinterpret_cast<__nv_bfloat16*>(y.data_ptr()); + prm.ar_ws = reinterpret_cast(arWs); + prm.ar_flags = reinterpret_cast(arFlags); + prm.ar_rank = static_cast(arRank); + prm.ar_world = static_cast(arWorld); + prm.ar_maxt = static_cast(arMaxTokens); + prm.ar_mode = static_cast(arMode); + + cudaError_t const err + = tensorrt_llm::kernels::invokeMinimaxM3AttnMega(prm, qkvWeight.data_ptr(), oWeight.data_ptr(), + static_cast(decodeQueryLen), static_cast(tokensPerCluster), static_cast(blocksPerCta), + static_cast(numReqs), usePdl, at::cuda::getCurrentCUDAStream(xq.get_device())); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "minimax_m3_attn_mega launch failed (qlen=", decodeQueryLen, " qg=", tokensPerCluster, + " bpc=", blocksPerCta, "): ", cudaGetErrorString(err)); + } +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_attn_mega(Tensor xq, Tensor xsf, Tensor qkv_weight, Tensor qkv_weight_scale, Tensor(a!) qkv_out, " + "Tensor(b!) flags, Tensor(c!) kv_data, Tensor(d!) kv_scale, Tensor(e!) idx_k, Tensor out_cache_loc, " + "Tensor position_ids, Tensor block_table, Tensor seq_lens, Tensor kv_quant_scale, Tensor k_global_scale, " + "Tensor v_global_scale, Tensor q_norm_weight, Tensor k_norm_weight, Tensor index_k_norm_weight, " + "Tensor rotary_cos_sin, float eps, float sm_scale, int decode_query_len, int tokens_per_cluster, " + "int blocks_per_cta, Tensor(f!) output, Tensor(g!) mxfp8_data, Tensor(h!) mxfp8_scale, bool use_pdl, " + "Tensor o_weight, Tensor o_weight_scale, Tensor(i!) y, int ar_workspace, int ar_flags, int ar_rank, " + "int ar_world, int ar_max_tokens, int ar_mode) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_attn_mega", &tensorrt_llm::torch_ext::minimaxM3AttnMega); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3DraftStepOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3DraftStepOp.cpp new file mode 100644 index 000000000000..6bfc88d79213 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3DraftStepOp.cpp @@ -0,0 +1,198 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3/draftStep.h" + +#include +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace ds = tensorrt_llm::kernels::minimaxM3::draftStep; + +using OptTensor = std::optional; + +void checkArch(torch::Tensor const& t) +{ + TORCH_CHECK(t.is_cuda(), "minimax_m3_draft_step expects CUDA tensors"); + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_draft_step requires SM100 or newer"); +} + +void check(cudaError_t err) +{ + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, cudaGetErrorString(err)); + } +} + +void checkInt(torch::Tensor const& t, int64_t n, char const* name) +{ + TORCH_CHECK(t.scalar_type() == at::kInt && t.is_contiguous() && t.numel() >= n, "draft_step: ", name, + " must be contiguous int32 with at least ", n, " elements"); +} + +} // namespace + +//! One fused greedy draft pass over `rows` <= 8 requests: the BF16 head (this rank's fragment-ordered shard wf +//! [N / 16, K / 16, 256], or with excOff / excPos / excW / expBase its 12-bit coding, wf uint8 [N * K * 3 / 2]; +//! global ids vocabOffset..) over x[xRows[b]] (or x[b]), the global argmax across the TP ranks through the multicast +//! slots wsMcPtr / wsUc, then draftOut[b, step] = idsNext[b] = token, posNext[b] = posSrc[xRows[b] or b] + 1 and the +//! KV-length bookkeeping (see draftStep.h). +void minimaxM3DraftStep(torch::Tensor const& x, OptTensor const& xRows, int64_t rows, torch::Tensor const& wf, + int64_t vocabOffset, torch::Tensor& keys, torch::Tensor& counters, torch::Tensor& wsUc, int64_t wsMcPtr, + int64_t rank, int64_t world, torch::Tensor& draftOut, int64_t step, torch::Tensor& idsNext, + torch::Tensor const& posSrc, torch::Tensor& posNext, torch::Tensor& kvLens, OptTensor const& numAcc, + OptTensor const& seqLens, int64_t draftLen, int64_t numCtas, bool pdl, OptTensor const& outKeys, + OptTensor const& excOff, OptTensor const& excPos, OptTensor const& excW, int64_t expBase) +{ + checkArch(x); + for (torch::Tensor const* t : std::initializer_list{ + &wf, &keys, &counters, &wsUc, &draftOut, &idsNext, &posSrc, &posNext, &kvLens}) + { + TORCH_CHECK(t->device() == x.device(), "draft_step: every tensor must be on x's device"); + } + for (OptTensor const* t : + std::initializer_list{&xRows, &numAcc, &seqLens, &outKeys, &excOff, &excPos, &excW}) + { + TORCH_CHECK(!t->has_value() || (*t)->device() == x.device(), "draft_step: every tensor must be on x's device"); + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + TORCH_CHECK(x.scalar_type() == at::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "draft_step: x must be contiguous BF16 [rows, K]"); + int const K = static_cast(x.size(1)); + TORCH_CHECK(K % 16 == 0, "draft_step: K must be a multiple of 16"); + TORCH_CHECK(rows >= 1 && rows <= ds::kMaxTokens, "draft_step: 1..", ds::kMaxTokens, " rows"); + bool const coded = excOff.has_value(); + TORCH_CHECK(coded == excPos.has_value() && coded == excW.has_value(), + "draft_step: exc_off, exc_pos and exc_w come together"); + int nTiles = 0; + if (coded) + { + TORCH_CHECK(K % 64 == 0 && K / 16 <= 0xFFFF, "draft_step: a coded head needs K a multiple of 64"); + TORCH_CHECK(excOff->scalar_type() == at::kInt && excOff->is_contiguous() && excOff->numel() >= 2, + "draft_step: exc_off must be int32 [tiles + 1]"); + nTiles = static_cast(excOff->numel() - 1); + TORCH_CHECK(wf.scalar_type() == at::kByte && wf.is_contiguous() + && wf.numel() == static_cast(nTiles) * 16 * K * 3 / 2, + "draft_step: a coded wf must be the contiguous uint8 stream of [tiles * 16, K]"); + TORCH_CHECK(excPos->scalar_type() == at::kInt && excPos->is_contiguous(), "draft_step: exc_pos must be int32"); + TORCH_CHECK(excW->scalar_type() == at::kFloat && excW->is_contiguous() && excW->numel() == 2 * excPos->numel(), + "draft_step: exc_w must be fp32 [E, 2]"); + TORCH_CHECK(expBase >= 0 && expBase <= 240, "draft_step: exponent base out of range"); + } + else + { + TORCH_CHECK(wf.scalar_type() == at::kBFloat16 && wf.dim() == 3 && wf.is_contiguous() && wf.size(1) == K / 16 + && wf.size(2) == 256, + "draft_step: wf must be the fragment-ordered BF16 weight [N / 16, K / 16, 256]"); + nTiles = static_cast(wf.size(0)); + } + TORCH_CHECK(nTiles >= 1 && numCtas >= 1 && (nTiles + numCtas - 1) / numCtas <= ds::kMaxTilesPerCta, + "draft_step: too many weight tiles per CTA"); + TORCH_CHECK(vocabOffset >= 0 && vocabOffset + 16LL * nTiles <= 0xFFFFFFFFLL, "draft_step: vocab offset"); + if (xRows.has_value()) + { + TORCH_CHECK(xRows->scalar_type() == at::kLong && xRows->is_contiguous() && xRows->numel() >= rows, + "draft_step: x_rows must be contiguous int64 [rows]"); + } + else + { + TORCH_CHECK(x.size(0) >= rows, "draft_step: x must hold a row per request"); + } + TORCH_CHECK(world >= 1 && rank >= 0 && rank < world, "draft_step: rank / world"); + TORCH_CHECK(keys.scalar_type() == at::kLong && keys.numel() >= rows, "draft_step: keys must be int64 [>= rows]"); + TORCH_CHECK(counters.scalar_type() == at::kInt && counters.numel() >= 2, "draft_step: counters must be int32 [2]"); + TORCH_CHECK(wsUc.scalar_type() == at::kLong && wsUc.numel() >= 2 * ds::kMaxTokens * world, + "draft_step: ws_uc must hold [2][8][world] slots"); + TORCH_CHECK(draftOut.scalar_type() == at::kInt && draftOut.dim() == 2 && draftOut.stride(1) == 1 + && draftOut.size(0) >= rows && step >= 0 && step < draftOut.size(1), + "draft_step: draft_out must be int32 [rows, > step] with unit column stride"); + checkInt(idsNext, rows, "ids_next"); + checkInt(posNext, rows, "pos_next"); + checkInt(kvLens, rows, "kv_lens"); + TORCH_CHECK(posSrc.scalar_type() == at::kInt && posSrc.is_contiguous(), "draft_step: pos_src must be int32"); + TORCH_CHECK(numAcc.has_value() == seqLens.has_value(), "draft_step: num_acc and seq_lens come together"); + TORCH_CHECK(!numAcc.has_value() || step == 0, "draft_step: the KV rollback (num_acc) is pass 0's"); + if (numAcc.has_value()) + { + checkInt(*numAcc, rows, "num_acc"); + checkInt(*seqLens, rows, "seq_lens"); + } + if (outKeys.has_value()) + { + TORCH_CHECK(outKeys->scalar_type() == at::kLong && outKeys->is_contiguous() && outKeys->numel() >= rows, + "draft_step: out_keys must be contiguous int64 [rows]"); + } + ds::Params p{}; + p.x = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.x_rows = xRows.has_value() ? reinterpret_cast(xRows->data_ptr()) : nullptr; + p.M = static_cast(rows); + p.wf = reinterpret_cast(wf.data_ptr()); + p.n_tiles = nTiles; + p.vocab_offset = static_cast(vocabOffset); + p.keys = reinterpret_cast(keys.data_ptr()); + p.counters = reinterpret_cast(counters.data_ptr()); + p.ws_uc = reinterpret_cast(wsUc.data_ptr()); + p.ws_mc = reinterpret_cast(wsMcPtr); + p.rank = static_cast(rank); + p.world = static_cast(world); + p.out_keys = outKeys.has_value() ? reinterpret_cast(outKeys->data_ptr()) : nullptr; + p.st = ds::StepArgs{draftOut.data_ptr(), static_cast(draftOut.stride(0)), static_cast(step), + idsNext.data_ptr(), posSrc.data_ptr(), posNext.data_ptr(), kvLens.data_ptr(), + numAcc.has_value() ? numAcc->data_ptr() : nullptr, + seqLens.has_value() ? seqLens->data_ptr() : nullptr, static_cast(draftLen)}; + p.cd = ds::CodedArgs{coded ? excOff->data_ptr() : nullptr, + coded ? reinterpret_cast(excPos->data_ptr()) : nullptr, + coded ? reinterpret_cast(excW->data_ptr()) : nullptr, + static_cast((expBase << 7) | (expBase << 23))}; + check(ds::invokeDraftStep(p, K, static_cast(numCtas), pdl, at::cuda::getCurrentCUDAStream(x.get_device()))); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +// Mutations through the raw multicast address ws_mc_ptr are not visible to the schema: that buffer is owned by the +// Python workspace and never aliased by the op's tensor arguments. +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_draft_step(Tensor x, Tensor? x_rows, int rows, Tensor wf, int vocab_offset, Tensor(a!) keys, " + "Tensor(b!) counters, Tensor(c!) ws_uc, int ws_mc_ptr, int rank, int world, Tensor(d!) draft_out, int step, " + "Tensor(e!) ids_next, Tensor pos_src, Tensor(f!) pos_next, Tensor(g!) kv_lens, Tensor? num_acc, " + "Tensor(h!)? seq_lens, int draft_len, int num_ctas, bool pdl, Tensor(i!)? out_keys, Tensor? exc_off=None, " + "Tensor? exc_pos=None, Tensor? exc_w=None, int exp_base=0) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_draft_step", &tensorrt_llm::torch_ext::minimaxM3DraftStep); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3FusedDecodeAttnOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3FusedDecodeAttnOp.cpp new file mode 100644 index 000000000000..e431a62f0028 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3FusedDecodeAttnOp.cpp @@ -0,0 +1,175 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +using tensorrt_llm::kernels::MinimaxM3FusedDecodeAttnParams; + +constexpr int64_t kNumHeads = 16; +constexpr int64_t kHeadDim = 128; +constexpr int64_t kPackedWidth = (kNumHeads + 4) * kHeadDim; + +void checkByteTensor(at::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, name, " must be a CUDA tensor of one-byte elements"); +} + +} // namespace + +//! Fused per-head Gemma RMSNorm + partial NeoX RoPE + NVFP4 K/V and FP8 index-K cache insert + identity-selection +//! sparse decode attention over the paged NVFP4 cache + split-K merge, for the MiniMax-M3 MSA sparse layers in +//! short-sequence decode (every request attends at most 16 KV blocks of 128 tokens). +//! +//! @param packed bf16 [T, 2560] QKV+index projection rows ([Q x16 | K | V | idxQ | idxK] x 128), T = R * qlen. +//! @param kvData uint8 [pages, 2, 1, 128, 64] HND NVFP4 pool (K plane 0, V plane 1); the new tokens are written. +//! @param kvScale uint8 [pages, 2, 1, 128, 8] E4M3 block scales (K linear, V 4x4 token quads); written. +//! @param idxK one-byte [pages, 1, 128, 128] FP8 index-K cache; written. +//! @param output bf16 [T, 2048] attention output; mxfp8Data / mxfp8Scale its MXFP8 copy (E4M3 rows and 128x4-swizzled +//! UE8M0 scales) for the o_proj GEMM. +void minimaxM3FusedDecodeAttn(at::Tensor const& packed, at::Tensor& kvData, at::Tensor& kvScale, at::Tensor& idxK, + at::Tensor const& outCacheLoc, at::Tensor const& positionIds, at::Tensor const& blockTable, + at::Tensor const& seqLens, at::Tensor const& kvQuantScale, at::Tensor const& kGlobalScale, + at::Tensor const& vGlobalScale, at::Tensor const& qNormWeight, at::Tensor const& kNormWeight, + at::Tensor const& indexKNormWeight, at::Tensor const& rotaryCosSin, double eps, double smScale, + int64_t decodeQueryLen, int64_t tokensPerCluster, int64_t blocksPerCta, at::Tensor& output, at::Tensor& mxfp8Data, + at::Tensor& mxfp8Scale, bool usePdl) +{ + TORCH_CHECK(packed.is_cuda() && packed.scalar_type() == at::kBFloat16 && packed.dim() == 2 + && packed.size(1) == kPackedWidth && packed.stride(1) == 1, + "packed must be CUDA bf16 [T, 2560] with contiguous rows"); + checkByteTensor(kvData, "kv_data"); + checkByteTensor(kvScale, "kv_scale"); + checkByteTensor(idxK, "idx_k"); + TORCH_CHECK( + kvData.dim() == 5 && kvScale.dim() == 5 && idxK.dim() == 4, "kv_data, kv_scale and idx_k must be paged"); + for (auto const* t : {&outCacheLoc, &positionIds, &blockTable, &seqLens}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == at::kInt, + "slot, position, block-table and length tensors " + "must be CUDA int32"); + } + TORCH_CHECK(blockTable.dim() == 2 && blockTable.stride(1) == 1, "block_table must be row-major 2-D"); + for (auto const* t : {&kvQuantScale, &kGlobalScale, &vGlobalScale, &rotaryCosSin}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == at::kFloat, "scales and rotary_cos_sin must be CUDA fp32"); + } + for (auto const* t : {&qNormWeight, &kNormWeight, &indexKNormWeight}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == at::kBFloat16 && t->numel() == kHeadDim, + "norm weights must be CUDA bf16 [128]"); + } + TORCH_CHECK(output.is_cuda() && output.scalar_type() == at::kBFloat16 && output.dim() == 2 + && output.size(1) == kNumHeads * kHeadDim && output.stride(1) == 1, + "output must be CUDA bf16 [T, 2048] with contiguous rows"); + checkByteTensor(mxfp8Data, "mxfp8_data"); + checkByteTensor(mxfp8Scale, "mxfp8_scale"); + int64_t const numTokens = packed.size(0); + int64_t const numReqs = seqLens.size(0); + TORCH_CHECK(numTokens == numReqs * decodeQueryLen, "packed rows must equal num_requests * decode_query_len"); + TORCH_CHECK(output.size(0) >= numTokens && mxfp8Data.dim() == 2 && mxfp8Data.size(0) >= numTokens + && mxfp8Data.size(1) == kNumHeads * kHeadDim && mxfp8Data.is_contiguous() && mxfp8Scale.is_contiguous(), + "mxfp8_data must be contiguous [>= T, 2048]"); + TORCH_CHECK(tensorrt_llm::kernels::minimaxM3FusedDecodeAttnSupports(static_cast(decodeQueryLen), + static_cast(tokensPerCluster), static_cast(blocksPerCta)), + "minimax_m3_fused_decode_attn: geometry (", decodeQueryLen, ", ", tokensPerCluster, ", ", blocksPerCta, + ") is not compiled"); + if (numTokens == 0) + { + return; + } + + c10::cuda::CUDAGuard const deviceGuard{packed.device()}; + auto const* prop = at::cuda::getCurrentDeviceProperties(); + TORCH_CHECK(prop->major == 10, "minimax_m3_fused_decode_attn requires an sm_100-family GPU, got sm_", prop->major, + prop->minor); + + MinimaxM3FusedDecodeAttnParams prm = {}; + prm.packed = reinterpret_cast<__nv_bfloat16 const*>(packed.data_ptr()); + prm.packed_stride = packed.stride(0); + prm.kv_data = reinterpret_cast(kvData.data_ptr()); + prm.d_page = kvData.stride(0); + prm.d_plane = kvData.stride(1); + prm.kv_scale = reinterpret_cast(kvScale.data_ptr()); + prm.s_page = kvScale.stride(0); + prm.s_plane = kvScale.stride(1); + prm.idx_k = reinterpret_cast(idxK.data_ptr()); + prm.i_page = idxK.stride(0); + prm.i_tok = idxK.stride(2); + prm.num_pages = kvData.size(0); + prm.out_cache_loc = outCacheLoc.data_ptr(); + prm.position_ids = positionIds.data_ptr(); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.bt_cols = static_cast(blockTable.size(1)); + prm.seq_lens = seqLens.data_ptr(); + prm.kv_quant_scale = kvQuantScale.data_ptr(); + prm.k_gs = kGlobalScale.data_ptr(); + prm.v_gs = vGlobalScale.data_ptr(); + prm.q_w = reinterpret_cast<__nv_bfloat16 const*>(qNormWeight.data_ptr()); + prm.k_w = reinterpret_cast<__nv_bfloat16 const*>(kNormWeight.data_ptr()); + prm.ik_w = reinterpret_cast<__nv_bfloat16 const*>(indexKNormWeight.data_ptr()); + prm.rope = rotaryCosSin.data_ptr(); + prm.eps = static_cast(eps); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(output.data_ptr()); + prm.out_stride = output.stride(0); + prm.q8 = reinterpret_cast(mxfp8Data.data_ptr()); + prm.sf = reinterpret_cast(mxfp8Scale.data_ptr()); + prm.num_sf_k_tiles = static_cast(output.size(1) / 128); + + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM3FusedDecodeAttn(prm, static_cast(decodeQueryLen), + static_cast(tokensPerCluster), static_cast(blocksPerCta), static_cast(numReqs), usePdl, + at::cuda::getCurrentCUDAStream(packed.get_device())); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "minimax_m3_fused_decode_attn launch failed (qlen=", decodeQueryLen, + " qg=", tokensPerCluster, " bpc=", blocksPerCta, "): ", cudaGetErrorString(err)); + } +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_fused_decode_attn(Tensor packed, Tensor(a!) kv_data, Tensor(b!) kv_scale, Tensor(c!) idx_k, " + "Tensor out_cache_loc, Tensor position_ids, Tensor block_table, Tensor seq_lens, Tensor kv_quant_scale, " + "Tensor k_global_scale, Tensor v_global_scale, Tensor q_norm_weight, Tensor k_norm_weight, " + "Tensor index_k_norm_weight, Tensor rotary_cos_sin, float eps, float sm_scale, int decode_query_len, " + "int tokens_per_cluster, int blocks_per_cta, Tensor(d!) output, Tensor(e!) mxfp8_data, " + "Tensor(f!) mxfp8_scale, bool use_pdl) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_fused_decode_attn", &tensorrt_llm::torch_ext::minimaxM3FusedDecodeAttn); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3McAllreduceNormOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3McAllreduceNormOp.cpp new file mode 100644 index 000000000000..3f4558f8a330 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3McAllreduceNormOp.cpp @@ -0,0 +1,159 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3/mcAllreduceNorm.h" + +#include +#include +#include + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kMaxTokens = 48; +constexpr int64_t kCluster = 8; +constexpr int64_t kMaxWorld = 8; + +} // namespace + +//! MiniMax-M3 multicast all-reduce + residual + RMSNorm (+ optional MoE finalize, + optional MXFP8 output). +//! +//! @param x BF16 [M, H] partial, or with `perm` the permuted FC2 output [rows, H]. +//! @param x2 Optional BF16 [M, H] second addend (the shared expert in finalize mode). +//! @param residual BF16 [M, H], 1 <= M <= 48. +//! @param normWeight BF16 [H]. +//! @param wsUc This rank's unicast view of the exchange workspace (int32 words); re-armed in place. +//! @param wsMcPtr Device address of the workspace's multicast mapping. +//! @param flags Int32 per-CTA call-parity counters, at least maxt * 8; updated in place. +//! @param cluster Cluster size; 8 is the only one built. +//! @param perm, scales Finalize mode: int32 [M, topk] permuted-row index and fp32 / bf16 [M, topk] routing weights. +//! @param maxt Token capacity of the workspace bucket (8, 16 or 48). +//! @param quant Also return the MXFP8 copy of the normed rows (E4M3 [M, H], swizzled UE8M0 scales). +//! @param prepushed 0, or 2 / 3 when producers already pushed this rank's rows (see the kernel header). +//! @param opts Epilogue variant; every variant is bit-identical. +//! @return [normed, updated residual] or, with `quant`, [normed, updated residual, E4M3 rows, scale bytes]. +std::vector minimaxM3McAllreduceNorm(torch::Tensor const& x, std::optional const& x2, + torch::Tensor const& residual, torch::Tensor const& normWeight, double eps, torch::Tensor& wsUc, int64_t wsMcPtr, + torch::Tensor& flags, int64_t rank, int64_t world, int64_t cluster, bool pdl, + std::optional const& perm, std::optional const& scales, int64_t maxt, bool quant, + int64_t prepushed, int64_t opts) +{ + int const sm = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sm == 100 || sm == 103, "minimax_m3_mc_allreduce_norm requires SM100 or SM103, got SM", sm); + TORCH_CHECK(cluster == kCluster, "minimax_m3_mc_allreduce_norm: only cluster 8 is built, got ", cluster); + TORCH_CHECK(tensorrt_llm::kernels::isMinimaxM3McAllreduceNormSupported( + static_cast(maxt), static_cast(opts)), + "minimax_m3_mc_allreduce_norm: unsupported (bucket ", maxt, ", opts ", opts, ")"); + TORCH_CHECK( + world >= 1 && world <= kMaxWorld && rank >= 0 && rank < world, "invalid rank ", rank, " / world ", world); + + tensorrt_llm::kernels::MinimaxM3McAllreduceNormParams p; + if (x2.has_value()) + { + TORCH_CHECK(x2->scalar_type() == at::kBFloat16 && x2->is_contiguous() && x2->sizes() == residual.sizes(), + "x2 must be a contiguous bf16 tensor shaped like residual"); + p.x2 = reinterpret_cast<__nv_bfloat16 const*>(x2->data_ptr()); + } + if (perm.has_value()) + { + // Finalize mode: x is the permuted FC2 output [rows, H]. + TORCH_CHECK(scales.has_value() && perm->scalar_type() == at::kInt && perm->is_contiguous() && perm->dim() == 2 + && perm->size(0) == residual.size(0), + "perm must be a contiguous int32 [M, topk] tensor, with scales"); + TORCH_CHECK(scales->is_contiguous() && scales->numel() == perm->numel() + && (scales->scalar_type() == at::kFloat || scales->scalar_type() == at::kBFloat16), + "scales must be a contiguous fp32 or bf16 tensor with one weight per perm entry"); + TORCH_CHECK(x.dim() == 2 && x.size(1) == residual.size(1), "fc2 must be [rows, H]"); + p.fc2 = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.perm = perm->data_ptr(); + p.scales = scales->data_ptr(); + p.scaleFp32 = scales->scalar_type() == at::kFloat; + p.topk = static_cast(perm->size(1)); + } + TORCH_CHECK(x.is_cuda() && x.scalar_type() == at::kBFloat16 && x.is_contiguous() && x.dim() == 2, + "x must be a contiguous 2-D bf16 CUDA tensor"); + TORCH_CHECK(residual.dim() == 2 && residual.size(0) >= 1 && residual.size(0) <= std::min(kMaxTokens, maxt), + "residual must hold 1..min(48, bucket) rows"); + TORCH_CHECK(residual.scalar_type() == at::kBFloat16 && residual.is_contiguous() + && (perm.has_value() || residual.sizes() == x.sizes()), + "residual must be a contiguous bf16 tensor shaped like x"); + TORCH_CHECK( + normWeight.scalar_type() == at::kBFloat16 && normWeight.numel() == residual.size(1), "norm_w must be bf16 [H]"); + TORCH_CHECK(wsUc.scalar_type() == at::kInt && wsUc.is_cuda(), "ws_uc must be an int32 CUDA tensor"); + TORCH_CHECK(flags.scalar_type() == at::kInt && flags.is_contiguous() && flags.numel() >= maxt * kCluster, + "flags must be a contiguous int32 tensor of at least bucket * 8 counters"); + + int64_t const m = residual.size(0), h = residual.size(1); + TORCH_CHECK(!quant || h % 128 == 0, "MXFP8 output needs H % 128 == 0"); + c10::cuda::CUDAGuard const deviceGuard{residual.device()}; + auto outNormed = torch::empty_like(residual); + auto outResidual = torch::empty_like(residual); + auto outQ = quant ? torch::empty({m, h}, residual.options().dtype(at::kFloat8_e4m3fn)) : torch::Tensor(); + auto outSf = quant ? torch::empty({(m + 127) / 128 * 128 * (h / 32)}, residual.options().dtype(at::kByte)) + : torch::Tensor(); + + p.x = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.numTokens = static_cast(m); + p.hidden = static_cast(h); + p.residual = reinterpret_cast<__nv_bfloat16 const*>(residual.data_ptr()); + p.normWeight = reinterpret_cast<__nv_bfloat16 const*>(normWeight.data_ptr()); + p.eps = static_cast(eps); + p.wsUc = wsUc.data_ptr(); + p.wsMc = reinterpret_cast(wsMcPtr); + p.flags = flags.data_ptr(); + p.rank = static_cast(rank); + p.world = static_cast(world); + p.outNormed = reinterpret_cast<__nv_bfloat16*>(outNormed.data_ptr()); + p.outResidual = reinterpret_cast<__nv_bfloat16*>(outResidual.data_ptr()); + p.outQuant = quant ? reinterpret_cast(outQ.data_ptr()) : nullptr; + p.outSf = quant ? outSf.data_ptr() : nullptr; + p.prepushed = static_cast(prepushed); + tensorrt_llm::kernels::invokeMinimaxM3McAllreduceNorm(p, static_cast(maxt), static_cast(opts), + pdl, at::cuda::getCurrentCUDAStream(residual.get_device())); + if (quant) + { + return {outNormed, outResidual, outQ, outSf}; + } + return {outNormed, outResidual}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_mc_allreduce_norm(Tensor x, Tensor? x2, Tensor residual, Tensor norm_w, float eps, " + "Tensor(a!) ws_uc, int ws_mc_ptr, Tensor(b!) flags, int rank, int world, int cluster, bool pdl, " + "Tensor? perm, Tensor? scales, int maxt, bool quant, int prepushed, int opts) -> Tensor[]"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_mc_allreduce_norm", &tensorrt_llm::torch_ext::minimaxM3McAllreduceNorm); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3MsaKvUpdateOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3MsaKvUpdateOp.cpp new file mode 100644 index 000000000000..dfea9d3af430 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3MsaKvUpdateOp.cpp @@ -0,0 +1,110 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/msaKvUpdate.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +void checkInt32Cuda(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda(), "minimax_m3_msa_kv_update: ", name, " must be a CUDA tensor"); + TORCH_CHECK( + t.scalar_type() == torch::kInt32, "minimax_m3_msa_kv_update: ", name, " must be int32, got ", t.scalar_type()); +} + +} // namespace + +//! MiniMax-M3 MSA KV-length correction after the overlap scheduler moved kv_lens (see invokeMinimaxM3MsaKvUpdate). +//! +//! Every tensor is int32 on one CUDA device. block_table is [rows, ncols] with unit column stride; every other +//! operand is contiguous with at least `batch` (per request) or `total_q` (per query token) elements, and cu_kv_lens +//! at least `batch + 1`. Writes msa_seq_lens, out_cache_loc and, when given, cu_kv_lens and n_valid_blocks. +void minimaxM3MsaKvUpdate(torch::Tensor const& kvLens, torch::Tensor const& kvLensStaged, torch::Tensor const& seqLens, + torch::Tensor const& qBatchRow, torch::Tensor const& qIntra, torch::Tensor const& blockTable, + int64_t tokensPerBlock, int64_t batch, int64_t totalQ, torch::Tensor& msaSeqLens, + std::optional cuKvLens, torch::Tensor& outCacheLoc, std::optional nValidBlocks) +{ + checkInt32Cuda(kvLens, "kv_lens"); + checkInt32Cuda(kvLensStaged, "kv_lens_staged"); + checkInt32Cuda(seqLens, "seq_lens"); + checkInt32Cuda(qBatchRow, "q_batch_row"); + checkInt32Cuda(qIntra, "q_intra"); + checkInt32Cuda(blockTable, "block_table"); + checkInt32Cuda(msaSeqLens, "msa_seq_lens"); + checkInt32Cuda(outCacheLoc, "out_cache_loc"); + TORCH_CHECK(blockTable.dim() == 2 && blockTable.stride(1) == 1, + "minimax_m3_msa_kv_update: block_table must be 2-D with unit column stride"); + TORCH_CHECK(tokensPerBlock > 0, "minimax_m3_msa_kv_update: tokens_per_block must be positive"); + TORCH_CHECK(batch >= 0 && totalQ >= 0, "minimax_m3_msa_kv_update: batch and total_q must be non-negative"); + TORCH_CHECK(kvLens.numel() >= batch && kvLensStaged.numel() >= batch && seqLens.numel() >= batch + && msaSeqLens.numel() >= batch, + "minimax_m3_msa_kv_update: a per-request operand has fewer than batch elements"); + TORCH_CHECK(qBatchRow.numel() >= totalQ && qIntra.numel() >= totalQ && outCacheLoc.numel() >= totalQ, + "minimax_m3_msa_kv_update: a per-token operand has fewer than total_q elements"); + int32_t* cuKvPtr = nullptr; + if (cuKvLens.has_value()) + { + checkInt32Cuda(*cuKvLens, "cu_kv_lens"); + TORCH_CHECK(cuKvLens->numel() >= batch + 1, "minimax_m3_msa_kv_update: cu_kv_lens needs batch + 1 elements"); + cuKvPtr = cuKvLens->data_ptr(); + } + int32_t* nValidPtr = nullptr; + if (nValidBlocks.has_value()) + { + checkInt32Cuda(*nValidBlocks, "n_valid_blocks"); + TORCH_CHECK(nValidBlocks->numel() >= totalQ, "minimax_m3_msa_kv_update: n_valid_blocks needs total_q elements"); + nValidPtr = nValidBlocks->data_ptr(); + } + + c10::cuda::CUDAGuard const deviceGuard{kvLens.device()}; + auto const stream = at::cuda::getCurrentCUDAStream(kvLens.get_device()); + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM3MsaKvUpdate(kvLens.data_ptr(), + kvLensStaged.data_ptr(), seqLens.data_ptr(), qBatchRow.data_ptr(), + qIntra.data_ptr(), blockTable.data_ptr(), blockTable.stride(0), + static_cast(blockTable.size(1)), static_cast(tokensPerBlock), static_cast(batch), + static_cast(totalQ), msaSeqLens.data_ptr(), cuKvPtr, outCacheLoc.data_ptr(), + nValidPtr, stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_msa_kv_update launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_msa_kv_update(Tensor kv_lens, Tensor kv_lens_staged, Tensor seq_lens, Tensor q_batch_row, " + "Tensor q_intra, Tensor block_table, int tokens_per_block, int batch, int total_q, Tensor(a!) msa_seq_lens, " + "Tensor(b!)? cu_kv_lens, Tensor(c!) out_cache_loc, Tensor(d!)? n_valid_blocks) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_msa_kv_update", &tensorrt_llm::torch_ext::minimaxM3MsaKvUpdate); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Mxfp8LowMGemmOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Mxfp8LowMGemmOp.cpp new file mode 100644 index 000000000000..c8f6b4a0355d --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Mxfp8LowMGemmOp.cpp @@ -0,0 +1,83 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3/mxfp8LowMGemm.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! MiniMax-M3 fused MXFP8 activation quantization + low-M MXFP8 GEMM (decode). +//! +//! @param x BF16 [M, K] activation, unit column stride, 16-byte aligned rows (row stride a multiple of 8). +//! @param w E4M3 [N, K] weight, contiguous. +//! @param wsf UE8M0 weight scales in the 128x4 swizzled layout of `mxfp8_quantize(w, True)`. +//! @param warps, clusterK, rowTiles The kernel configuration; see `isMinimaxM3Mxfp8LowMGemmSupported`. +//! @return BF16 [M, N]. +torch::Tensor minimaxM3Mxfp8LowMGemm(torch::Tensor const& x, torch::Tensor const& w, torch::Tensor const& wsf, + int64_t warps, int64_t clusterK, int64_t rowTiles, bool pdl) +{ + int const sm = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sm == 100 || sm == 103, "minimax_m3_mxfp8_low_m_gemm requires SM100 or SM103, got SM", sm); + TORCH_CHECK(x.is_cuda() && x.scalar_type() == at::kBFloat16 && x.dim() == 2 && x.stride(1) == 1, + "x must be a 2-D bf16 CUDA tensor with unit column stride"); + TORCH_CHECK(w.scalar_type() == at::kFloat8_e4m3fn && w.is_contiguous() && w.dim() == 2 && w.size(1) == x.size(1), + "w must be a contiguous e4m3 [N, K] tensor"); + TORCH_CHECK(wsf.scalar_type() == at::kByte, "wsf must be uint8 UE8M0 scales"); + TORCH_CHECK( + reinterpret_cast(x.data_ptr()) % 16 == 0 && x.stride(0) % 8 == 0, "x rows must be 16-byte aligned"); + TORCH_CHECK( + reinterpret_cast(w.data_ptr()) % 16 == 0 && reinterpret_cast(wsf.data_ptr()) % 16 == 0, + "w and wsf must be 16-byte aligned"); + TORCH_CHECK(tensorrt_llm::kernels::isMinimaxM3Mxfp8LowMGemmSupported(static_cast(warps), + static_cast(clusterK), static_cast(rowTiles), static_cast(x.size(0))), + "minimax_m3_mxfp8_low_m_gemm: unsupported configuration (warps ", warps, ", cluster_k ", clusterK, + ", row_tiles ", rowTiles, ") at M=", x.size(0)); + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto out = torch::empty({x.size(0), w.size(0)}, x.options()); + tensorrt_llm::kernels::invokeMinimaxM3Mxfp8LowMGemm(reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()), + static_cast(x.stride(0)), static_cast(x.size(0)), static_cast(w.size(0)), + static_cast(x.size(1)), reinterpret_cast(w.data_ptr()), + reinterpret_cast(wsf.data_ptr()), reinterpret_cast<__nv_bfloat16*>(out.data_ptr()), + static_cast(out.stride(0)), static_cast(warps), static_cast(clusterK), + static_cast(rowTiles), pdl, at::cuda::getCurrentCUDAStream(x.get_device())); + return out; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_mxfp8_low_m_gemm(Tensor x, Tensor w, Tensor wsf, int warps, int cluster_k, int row_tiles, " + "bool pdl) -> Tensor"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_mxfp8_low_m_gemm", &tensorrt_llm::torch_ext::minimaxM3Mxfp8LowMGemm); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3ProducerPwOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3ProducerPwOp.cpp new file mode 100644 index 000000000000..ec73b498e40e --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3ProducerPwOp.cpp @@ -0,0 +1,181 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3/producerPw.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace pw = tensorrt_llm::kernels::minimaxM3::producerPw; + +constexpr int64_t kHeadDim = 128; +constexpr int64_t kPageSize = 128; + +void checkCuda(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda(), "producer_pw: ", name, " must be a CUDA tensor"); +} + +} // namespace + +//! The NVFP4 QKV + indexer norm/RoPE/KV-insert producer (same arguments and outputs as +//! minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert for 1 KV head, 1 index head, head_dim 128, rotary_dim 64): +//! returns (q FP8 [T, headsQ, 128], index_q FP8 [T, 1, 128]) and writes the NVFP4 K/V and FP8 index-K of each token's +//! cache slot. `early` triggers the dependents on entry. +std::vector minimaxM3ProducerPw(torch::Tensor const& packed, torch::Tensor& kvData, + torch::Tensor& kvScale, torch::Tensor& indexK, torch::Tensor const& outCacheLoc, torch::Tensor const& kvQuantScale, + double eps, torch::Tensor const& qWeight, torch::Tensor const& kWeight, torch::Tensor const& indexQWeight, + torch::Tensor const& indexKWeight, torch::Tensor const& rotaryCosSin, torch::Tensor const& positionIds, + int64_t headsQ, bool pdl, bool early) +{ + checkCuda(packed, "packed"); + for (torch::Tensor const* t : std::initializer_list{&kvData, &kvScale, &indexK, &outCacheLoc, + &kvQuantScale, &qWeight, &kWeight, &indexQWeight, &indexKWeight, &rotaryCosSin, &positionIds}) + { + TORCH_CHECK(t->is_cuda() && t->get_device() == packed.get_device(), + "producer_pw: every operand must be on packed's CUDA device"); + } + TORCH_CHECK(reinterpret_cast(packed.data_ptr()) % 8 == 0 + && reinterpret_cast(kvData.data_ptr()) % 2 == 0 + && reinterpret_cast(indexK.data_ptr()) % 4 == 0, + "producer_pw: packed and the caches must start at the vector-load / store alignment"); + c10::cuda::CUDAGuard const deviceGuard{packed.device()}; + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_producer_pw requires SM100 or newer"); + TORCH_CHECK(headsQ == 8 || headsQ == 16, "producer_pw: 8 or 16 query heads"); + int64_t const totalHeads = headsQ + 4; + TORCH_CHECK(packed.scalar_type() == at::kBFloat16 && packed.dim() == 2 && packed.size(1) == totalHeads * kHeadDim + && packed.is_contiguous(), + "producer_pw: packed must be contiguous BF16 [T, (headsQ + 4) * 128]"); + // The kernel's layout (the native producer's checkMinimaxM3Nvfp4HndKVPools at one KV head): HND pools + // [pages, 2 planes, 1 head, 128 tokens, 64 data / 8 scale bytes], dense token rows and head blocks, planes and + // pages that do not overlap. + TORCH_CHECK(kvData.scalar_type() == at::kByte && kvData.dim() == 5 && kvData.size(0) > 0 && kvData.size(1) == 2 + && kvData.size(2) == 1 && kvData.size(3) == kPageSize && kvData.size(4) == kHeadDim / 2, + "producer_pw: kv_data must be the NVFP4 pool [pages, 2, 1, 128, 64] uint8"); + TORCH_CHECK(kvScale.scalar_type() == at::kByte && kvScale.dim() == 5 && kvScale.size(0) == kvData.size(0) + && kvScale.size(1) == 2 && kvScale.size(2) == 1 && kvScale.size(3) == kPageSize + && kvScale.size(4) == kHeadDim / 16, + "producer_pw: kv_scale must be the E4M3 scale pool [pages, 2, 1, 128, 8] uint8"); + TORCH_CHECK(kvData.stride(4) == 1 && kvData.stride(3) == kvData.size(4) + && kvData.stride(2) == kvData.size(3) * kvData.stride(3), + "producer_pw: kv_data must have dense token rows and head blocks"); + TORCH_CHECK(kvScale.stride(4) == 1 && kvScale.stride(3) == kvScale.size(4) + && kvScale.stride(2) == kvScale.size(3) * kvScale.stride(3), + "producer_pw: kv_scale must have dense token rows and head blocks"); + TORCH_CHECK(kvData.stride(1) >= kvData.size(2) * kvData.stride(2) && kvData.stride(0) >= 2 * kvData.stride(1) + && kvScale.stride(1) >= kvScale.size(2) * kvScale.stride(2) && kvScale.stride(0) >= 2 * kvScale.stride(1), + "producer_pw: pool planes or pages overlap"); + TORCH_CHECK(kvData.stride(0) % 2 == 0 && kvData.stride(1) % 2 == 0, + "producer_pw: kv_data page and plane strides must keep the uint16 stores aligned"); + TORCH_CHECK(indexK.scalar_type() == at::kFloat8_e4m3fn && indexK.dim() == 4 && indexK.size(0) == kvData.size(0) + && indexK.size(1) == 1 && indexK.size(2) == kPageSize && indexK.size(3) == kHeadDim && indexK.stride(3) == 1 + && indexK.stride(2) == kHeadDim && indexK.stride(0) >= kPageSize * kHeadDim && indexK.stride(0) % 4 == 0, + "producer_pw: index_k must be FP8 [pages, 1, 128, 128] with a page stride keeping the 32-bit stores aligned"); + int64_t const numTokens = packed.size(0); + TORCH_CHECK( + outCacheLoc.scalar_type() == at::kInt && outCacheLoc.is_contiguous() && outCacheLoc.numel() >= numTokens, + "producer_pw: out_cache_loc must be int32 [>= T]"); + TORCH_CHECK( + positionIds.scalar_type() == at::kInt && positionIds.is_contiguous() && positionIds.numel() >= numTokens, + "producer_pw: position_ids must be int32 [>= T]"); + TORCH_CHECK(kvQuantScale.scalar_type() == at::kFloat && kvQuantScale.is_contiguous() && kvQuantScale.numel() >= 3, + "producer_pw: kv_quant_scale must be fp32 [>= 3]"); + TORCH_CHECK(rotaryCosSin.scalar_type() == at::kFloat && rotaryCosSin.is_contiguous() && rotaryCosSin.dim() == 3 + && rotaryCosSin.size(1) == 2 && rotaryCosSin.size(2) == 32 + && reinterpret_cast(rotaryCosSin.data_ptr()) % 16 == 0, + "producer_pw: rotary_cos_sin must be fp32 [positions, 2, 32], 16-byte aligned (read as float4)"); + for (auto const* w : {&qWeight, &kWeight, &indexQWeight, &indexKWeight}) + { + TORCH_CHECK(w->scalar_type() == at::kBFloat16 && w->is_contiguous() && w->numel() == kHeadDim + && reinterpret_cast(w->data_ptr()) % 8 == 0, + "producer_pw: norm weights must be BF16 [128], 8-byte aligned (read as 4-value words)"); + } + auto const fp8 = packed.options().dtype(at::kFloat8_e4m3fn); + auto qOut = torch::empty({numTokens, headsQ, kHeadDim}, fp8); + auto indexQOut = torch::empty({numTokens, 1, kHeadDim}, fp8); + if (numTokens == 0) + { + return {qOut, indexQOut}; + } + pw::Params p{}; + p.packed = reinterpret_cast<__nv_bfloat16 const*>(packed.data_ptr()); + p.qOut = reinterpret_cast<__nv_fp8_e4m3*>(qOut.data_ptr()); + p.indexQOut = reinterpret_cast<__nv_fp8_e4m3*>(indexQOut.data_ptr()); + p.kvData = kvData.data_ptr(); + p.kvScale = kvScale.data_ptr(); + p.indexK = reinterpret_cast<__nv_fp8_e4m3*>(indexK.data_ptr()); + p.outCacheLoc = outCacheLoc.data_ptr(); + p.kvQuantScale = kvQuantScale.data_ptr(); + p.dataPageStride = kvData.stride(0); + p.dataPlaneStride = kvData.stride(1); + p.dataHeadStride = kvData.stride(2); + p.dataTokenStride = kvData.stride(3); + p.scalePageStride = kvScale.stride(0); + p.scalePlaneStride = kvScale.stride(1); + p.scaleHeadStride = kvScale.stride(2); + p.indexPageStride = indexK.stride(0); + p.indexTokenStride = indexK.stride(2); + p.numPages = kvData.size(0); + p.numTokens = static_cast(numTokens); + p.eps = static_cast(eps); + p.qWeight = reinterpret_cast<__nv_bfloat16 const*>(qWeight.data_ptr()); + p.kWeight = reinterpret_cast<__nv_bfloat16 const*>(kWeight.data_ptr()); + p.indexQWeight = reinterpret_cast<__nv_bfloat16 const*>(indexQWeight.data_ptr()); + p.indexKWeight = reinterpret_cast<__nv_bfloat16 const*>(indexKWeight.data_ptr()); + p.rotaryCosSin = rotaryCosSin.data_ptr(); + p.positionIds = positionIds.data_ptr(); + cudaError_t const err = pw::invokeProducerPw(p, static_cast(numTokens), static_cast(headsQ), pdl, early, + at::cuda::getCurrentCUDAStream(packed.get_device())); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "producer_pw launch: ", cudaGetErrorString(err)); + } + return {qOut, indexQOut}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_producer_pw(Tensor packed, Tensor(a!) kv_data, Tensor(b!) kv_scale, Tensor(c!) index_k, " + "Tensor out_cache_loc, Tensor kv_quant_scale, float eps, Tensor q_weight, Tensor k_weight, " + "Tensor index_q_weight, Tensor index_k_weight, Tensor rotary_cos_sin, Tensor position_ids, int heads_q, " + "bool pdl, bool early) -> Tensor[]"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_producer_pw", &tensorrt_llm::torch_ext::minimaxM3ProducerPw); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3RouterGemvOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3RouterGemvOp.cpp new file mode 100644 index 000000000000..059d29e43d85 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3RouterGemvOp.cpp @@ -0,0 +1,149 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3/routerGemv.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kGemvMaxM = 16; +constexpr int64_t kTcMaxM = 48; +constexpr int64_t kTcExperts = 128; + +void checkSmVersion(char const* op) +{ + int const sm = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sm == 100 || sm == 103, op, " requires SM100 or SM103, got SM", sm); +} + +void checkActivation(torch::Tensor const& x, torch::Tensor const& w, int64_t maxM) +{ + TORCH_CHECK(x.is_cuda() && x.scalar_type() == at::kBFloat16 && x.dim() == 2 && x.stride(1) == 1, + "x must be a 2-D bf16 CUDA tensor with unit column stride"); + TORCH_CHECK(x.size(0) >= 1 && x.size(0) <= maxM, "x must hold 1..", maxM, " rows, got ", x.size(0)); + TORCH_CHECK(w.scalar_type() == at::kFloat && w.is_contiguous() && w.dim() == 2 && w.size(1) == x.size(1), + "w must be a contiguous fp32 [E, K] tensor"); + TORCH_CHECK(x.size(1) % 1024 == 0 && (x.stride(0) * 2) % 16 == 0, "K must be a multiple of 1024 with 16-byte rows"); +} + +} // namespace + +//! fp32 router logits `x @ w^T` for 1..16 bf16 rows (exact fp32 FMA, one CTA per expert). +torch::Tensor minimaxM3RouterGemv(torch::Tensor const& x, torch::Tensor const& w, int64_t rows, bool pdl) +{ + checkSmVersion("minimax_m3_router_gemv"); + checkActivation(x, w, kGemvMaxM); + TORCH_CHECK(rows == 1, "minimax_m3_router_gemv: only rows=1 is built, got ", rows); + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto out = torch::empty({x.size(0), w.size(0)}, x.options().dtype(at::kFloat)); + tensorrt_llm::kernels::invokeMinimaxM3RouterGemv(reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()), + static_cast(x.stride(0)), static_cast(x.size(0)), static_cast(x.size(1)), + w.data_ptr(), static_cast(w.size(0)), out.data_ptr(), nullptr, nullptr, nullptr, pdl, + at::cuda::getCurrentCUDAStream(x.get_device())); + return out; +} + +//! Router logits as `minimax_m3_router_gemv` plus the NVFP4 activation: [logits fp32 [M, E], packed e2m1 +//! [M, K / 2] uint8, linear E4M3 scales [M * K / 16] uint8]. +std::vector minimaxM3RouterGemvFp4( + torch::Tensor const& x, torch::Tensor const& w, torch::Tensor const& sfScale, bool pdl) +{ + checkSmVersion("minimax_m3_router_gemv_fp4"); + checkActivation(x, w, kGemvMaxM); + TORCH_CHECK(sfScale.scalar_type() == at::kFloat && sfScale.numel() == 1 && sfScale.is_cuda(), + "sf_scale must be a one-element fp32 CUDA tensor"); + int64_t const M = x.size(0), K = x.size(1); + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto out = torch::empty({M, w.size(0)}, x.options().dtype(at::kFloat)); + auto q = torch::empty({M, K / 2}, x.options().dtype(at::kByte)); + auto sf = torch::empty({M * (K / 16)}, x.options().dtype(at::kByte)); + tensorrt_llm::kernels::invokeMinimaxM3RouterGemv(reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()), + static_cast(x.stride(0)), static_cast(M), static_cast(K), w.data_ptr(), + static_cast(w.size(0)), out.data_ptr(), q.data_ptr(), sf.data_ptr(), + sfScale.data_ptr(), pdl, at::cuda::getCurrentCUDAStream(x.get_device())); + return {out, q, sf}; +} + +//! TF32 tensor-core router logits for 128 experts and 1..48 rows: [logits fp32 [M, 128]] or, with `sf_scale`, +//! [logits, packed e2m1 [M, K / 2], linear E4M3 scales [M * K / 16]]. +std::vector minimaxM3RouterTc( + torch::Tensor const& x, torch::Tensor const& w, std::optional const& sfScale, bool pdl) +{ + checkSmVersion("minimax_m3_router_tc"); + TORCH_CHECK(x.is_cuda() && x.scalar_type() == at::kBFloat16 && x.dim() == 2 && x.stride(1) == 1, + "x must be a 2-D bf16 CUDA tensor with unit column stride"); + int64_t const M = x.size(0), K = x.size(1); + TORCH_CHECK(M >= 1 && M <= kTcMaxM && w.scalar_type() == at::kFloat && w.dim() == 2 && w.size(0) == kTcExperts + && w.size(1) == K && w.is_contiguous(), + "minimax_m3_router_tc takes 1..48 rows and a contiguous fp32 [128, K] weight"); + TORCH_CHECK(K % 1024 == 0 && (x.stride(0) * 2) % 16 == 0, "K must be a multiple of 1024 with 16-byte rows"); + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto out = torch::empty({M, w.size(0)}, x.options().dtype(at::kFloat)); + torch::Tensor q, sf; + uint8_t* qp = nullptr; + uint8_t* sfp = nullptr; + float const* scp = nullptr; + if (sfScale.has_value()) + { + TORCH_CHECK(sfScale->scalar_type() == at::kFloat && sfScale->numel() == 1 && sfScale->is_cuda(), + "sf_scale must be a one-element fp32 CUDA tensor"); + q = torch::empty({M, K / 2}, x.options().dtype(at::kByte)); + sf = torch::empty({M * (K / 16)}, x.options().dtype(at::kByte)); + qp = q.data_ptr(); + sfp = sf.data_ptr(); + scp = sfScale->data_ptr(); + } + tensorrt_llm::kernels::invokeMinimaxM3RouterTc(reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()), + static_cast(x.stride(0)), static_cast(M), static_cast(K), w.data_ptr(), + out.data_ptr(), qp, sfp, scp, pdl, at::cuda::getCurrentCUDAStream(x.get_device())); + if (sfScale.has_value()) + { + return {out, q, sf}; + } + return {out}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_router_gemv(Tensor x, Tensor w, int rows, bool pdl) -> Tensor"); + m.def("minimax_m3_router_gemv_fp4(Tensor x, Tensor w, Tensor sf_scale, bool pdl) -> Tensor[]"); + m.def("minimax_m3_router_tc(Tensor x, Tensor w, Tensor? sf_scale, bool pdl) -> Tensor[]"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_router_gemv", &tensorrt_llm::torch_ext::minimaxM3RouterGemv); + m.impl("minimax_m3_router_gemv_fp4", &tensorrt_llm::torch_ext::minimaxM3RouterGemvFp4); + m.impl("minimax_m3_router_tc", &tensorrt_llm::torch_ext::minimaxM3RouterTc); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SampledAcceptOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SampledAcceptOp.cpp new file mode 100644 index 000000000000..1c2183647374 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SampledAcceptOp.cpp @@ -0,0 +1,105 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/sampledAccept.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace sa = tensorrt_llm::kernels::minimaxM3::sampledAccept; + +//! uint64 words of the sampled-accept exchange workspace for `world` ranks. +int64_t minimaxM3SampledAcceptWsWords(int64_t world) +{ + return sa::sampledAcceptWsWords(static_cast(world)); +} + +//! Sampled (temperature + top-p) verification from this rank's BF16 logits shard [rows, Vs], rows = nreq * +//! (draft_len + 1) <= 48: the sampled token of every row (tokens), then the strict acceptance against draft (accTokens, +//! numAcc, gatherIds, optional nextNew column 0). Per-row Philox (seeds, offsets); the exchange goes through the +//! multicast workspace wsMcPtr / wsUc with the (finished rows, call) counters. +void minimaxM3SampledAccept(torch::Tensor const& logits, int64_t vocabOffset, torch::Tensor const& temperatures, + torch::Tensor const& topPs, torch::Tensor const& seeds, torch::Tensor const& offsets, torch::Tensor& wsUc, + int64_t wsMcPtr, torch::Tensor& counters, torch::Tensor& tokens, int64_t rank, int64_t world, + torch::Tensor const& draft, torch::Tensor const& batchIdx, torch::Tensor& accTokens, torch::Tensor& numAcc, + torch::Tensor& gatherIds, int64_t forceTotal, std::optional const& nextNew, + std::optional const& forceVec, std::optional const& extraAttempts, bool usePdl) +{ + c10::cuda::CUDAGuard const deviceGuard{logits.device()}; + sa::Args a; + a.logits = reinterpret_cast<__nv_bfloat16 const*>(logits.data_ptr()); + a.ld = logits.stride(0); + a.rows = static_cast(logits.size(0)); + a.Vs = static_cast(logits.size(1)); + TORCH_CHECK(a.Vs <= sa::kMaxVs && a.rows <= sa::kMaxRows && world <= sa::kMaxWorld + && static_cast(vocabOffset) + a.Vs <= (1LL << sa::kIdBits)); + a.vocab_offset = static_cast(vocabOffset); + a.temperatures = temperatures.data_ptr(); + a.top_ps = topPs.data_ptr(); + a.seeds = reinterpret_cast(seeds.data_ptr()); + a.offsets = reinterpret_cast(offsets.data_ptr()); + a.ws_uc = reinterpret_cast(wsUc.data_ptr()); + a.ws_mc = reinterpret_cast(wsMcPtr); + a.counters = reinterpret_cast(counters.data_ptr()); + a.tokens = tokens.data_ptr(); + a.extra_attempts = extraAttempts.has_value() ? extraAttempts->data_ptr() : nullptr; + a.rank = static_cast(rank); + a.world = static_cast(world); + a.draft = draft.data_ptr(); + a.batch_idx = batchIdx.data_ptr(); + a.acc_tokens = accTokens.data_ptr(); + a.num_acc = numAcc.data_ptr(); + a.gather_ids = reinterpret_cast(gatherIds.data_ptr()); + a.draft_len = static_cast(draft.size(1)); + a.nreq = static_cast(draft.size(0)); + a.force_total = static_cast(forceTotal); + a.next_new = nextNew.has_value() ? nextNew->data_ptr() : nullptr; + a.next_stride = nextNew.has_value() ? static_cast(nextNew->stride(0)) : 0; + a.force_vec = forceVec.has_value() ? forceVec->data_ptr() : nullptr; + cudaError_t const err = sa::invokeSampledAccept(a, usePdl, at::cuda::getCurrentCUDAStream(logits.get_device())); + TORCH_CHECK(err == cudaSuccess, "m3_sampled_accept launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +// ws_mc_ptr is the multicast address of ws_uc's mapping: the op writes through it, which the schema cannot show. +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + // No tensor arguments, so no dispatch key: registered with the schema as a catch-all kernel. + m.def("minimax_m3_sampled_accept_ws_words(int world) -> int", + &tensorrt_llm::torch_ext::minimaxM3SampledAcceptWsWords); + m.def( + "minimax_m3_sampled_accept(Tensor logits, int vocab_offset, Tensor temperatures, Tensor top_ps, " + "Tensor seeds, Tensor offsets, Tensor(a!) ws_uc, int ws_mc_ptr, Tensor(b!) counters, Tensor(c!) tokens, " + "int rank, int world, Tensor draft, Tensor batch_idx, Tensor(d!) acc_tokens, Tensor(e!) num_acc, " + "Tensor(f!) gather_ids, int force_total, Tensor(g!)? next_new, Tensor? force_vec, " + "Tensor(h!)? extra_attempts, bool use_pdl) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sampled_accept", &tensorrt_llm::torch_ext::minimaxM3SampledAccept); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SelectLongOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SelectLongOp.cpp new file mode 100644 index 000000000000..2def696ce875 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SelectLongOp.cpp @@ -0,0 +1,80 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/selectLong.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! Top-16 block select for MiniMax-M3 decode rows of up to 9216 blocks, into a caller-owned table. +//! +//! @param scores Float32 CUDA tensor [num_kv_heads, num_blocks, total_q], any strides. +//! @param nValidBlocks Contiguous int32 CUDA tensor with at least total_q elements. +//! @param out Int32 CUDA tensor of logical shape [total_q, num_kv_heads, 16] with unit stride in the last dim (the +//! head-major backing the target hands over is a permuted view). Every entry is written. +void minimaxM3SelectLong(torch::Tensor const& scores, torch::Tensor const& nValidBlocks, torch::Tensor& out, + int64_t initBlocks, int64_t localBlocks, bool usePdl, bool earlyTrigger) +{ + TORCH_CHECK(scores.is_cuda() && scores.scalar_type() == torch::kFloat32 && scores.dim() == 3, + "minimax_m3_select_long: scores must be a 3-D float32 CUDA tensor [num_kv_heads, num_blocks, total_q]"); + TORCH_CHECK(nValidBlocks.is_cuda() && nValidBlocks.scalar_type() == torch::kInt32 && nValidBlocks.is_contiguous() + && nValidBlocks.numel() >= scores.size(2), + "minimax_m3_select_long: n_valid_blocks must be contiguous int32 CUDA with at least total_q elements"); + int64_t const numKvHeads = scores.size(0); + int64_t const numBlocks = scores.size(1); + int64_t const totalQ = scores.size(2); + TORCH_CHECK(numBlocks <= tensorrt_llm::kernels::kMinimaxM3SelectLongMaxBlocks, + "minimax_m3_select_long supports at most ", tensorrt_llm::kernels::kMinimaxM3SelectLongMaxBlocks, + " blocks, got ", numBlocks); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kInt32 && out.dim() == 3 && out.size(0) == totalQ + && out.size(1) == numKvHeads && out.size(2) == 16 && out.stride(2) == 1, + "minimax_m3_select_long: out must be int32 [total_q, num_kv_heads, 16] with unit last-dim stride"); + + c10::cuda::CUDAGuard const deviceGuard{scores.device()}; + auto const stream = at::cuda::getCurrentCUDAStream(scores.get_device()); + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM3SelectLong(scores.data_ptr(), scores.stride(0), + scores.stride(1), scores.stride(2), nValidBlocks.data_ptr(), out.data_ptr(), out.stride(0), + out.stride(1), static_cast(numKvHeads), static_cast(numBlocks), static_cast(totalQ), + static_cast(initBlocks), static_cast(localBlocks), usePdl, earlyTrigger, stream); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "minimax_m3_select_long launch failed: ", cudaGetErrorString(err)); + } +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_select_long(Tensor scores, Tensor n_valid_blocks, Tensor(a!) out, int init_blocks, " + "int local_blocks, bool use_pdl=False, bool early_trigger=False) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_select_long", &tensorrt_llm::torch_ext::minimaxM3SelectLong); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3ShardAcceptOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3ShardAcceptOp.cpp new file mode 100644 index 000000000000..5b4ca3e6004f --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3ShardAcceptOp.cpp @@ -0,0 +1,118 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3/shardAccept.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace sa = tensorrt_llm::kernels::minimaxM3::shardAccept; + +using OptTensor = std::optional; + +void checkArch(torch::Tensor const& t) +{ + TORCH_CHECK(t.is_cuda(), "minimax_m3_shard_accept expects CUDA tensors"); + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_shard_accept requires SM100 or newer"); +} + +void check(cudaError_t err) +{ + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, cudaGetErrorString(err)); + } +} + +cudaStream_t currentStream(torch::Tensor const& t) +{ + return at::cuda::getCurrentCUDAStream(t.get_device()); +} + +//! Per-request accepted counts for fractional forced acceptance (int32 [nreq]), or nullptr. +int const* forceVecPtr(OptTensor const& forceVec, int nreq) +{ + if (!forceVec.has_value()) + { + return nullptr; + } + TORCH_CHECK(forceVec->scalar_type() == at::kInt && forceVec->is_contiguous() && forceVec->numel() >= nreq, + "accept: force_vec must be contiguous int32 [nreq]"); + return forceVec->data_ptr(); +} + +} // namespace + +//! Greedy verification of nreq * (draft_len + 1) <= 48 verify rows from this rank's BF16 logits shard [rows, V / tp] +//! (global ids vocabOffset..), exchanged across the TP ranks through the multicast slots wsMcPtr / wsUc, plus the +//! strict acceptance (accepted tokens, counts, first-pass gather rows, optional nextNew[:, 0]). +void minimaxM3ShardAccept(torch::Tensor const& logits, int64_t vocabOffset, torch::Tensor& keys, + torch::Tensor& counters, torch::Tensor& wsUc, int64_t wsMcPtr, int64_t rank, int64_t world, + torch::Tensor const& draft, torch::Tensor const& batchIdx, torch::Tensor& accTokens, torch::Tensor& numAcc, + torch::Tensor& gatherIds, int64_t forceTotal, OptTensor const& nextNew, int64_t numCtas, OptTensor const& forceVec) +{ + checkArch(logits); + c10::cuda::CUDAGuard const deviceGuard{logits.device()}; + TORCH_CHECK(logits.scalar_type() == at::kBFloat16 && logits.dim() == 2 && logits.stride(1) == 1); + int const M = logits.size(0); + int const nreq = accTokens.size(0); + int const k1 = accTokens.size(1); + TORCH_CHECK(M >= 1 && M <= sa::kMaxRows && nreq * k1 == M && nreq <= sa::kThreads); + TORCH_CHECK(draft.scalar_type() == at::kInt && batchIdx.scalar_type() == at::kInt && accTokens.is_contiguous()); + sa::AcceptArgs acc{draft.data_ptr(), batchIdx.data_ptr(), accTokens.data_ptr(), + numAcc.data_ptr(), reinterpret_cast(gatherIds.data_ptr()), k1 - 1, nreq, + static_cast(forceTotal), nextNew.has_value() ? nextNew->data_ptr() : nullptr, + nextNew.has_value() ? static_cast(nextNew->stride(0)) : 0, forceVecPtr(forceVec, nreq)}; + check(sa::invokeShardAccept(reinterpret_cast<__nv_bfloat16 const*>(logits.data_ptr()), M, + static_cast(logits.size(1)), static_cast(logits.stride(0)), static_cast(vocabOffset), + reinterpret_cast(keys.data_ptr()), reinterpret_cast(counters.data_ptr()), + reinterpret_cast(wsUc.data_ptr()), reinterpret_cast(wsMcPtr), + static_cast(rank), static_cast(world), acc, static_cast(numCtas), currentStream(logits))); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +// Mutations through the raw multicast address ws_mc_ptr are not visible to the schema: that buffer is owned by the +// Python workspace and never aliased by the op's tensor arguments. +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_shard_accept(Tensor logits, int vocab_offset, Tensor(a!) keys, Tensor(b!) counters, " + "Tensor(c!) ws_uc, int ws_mc_ptr, int rank, int world, Tensor draft, Tensor batch_idx, " + "Tensor(d!) acc_tokens, Tensor(e!) num_acc, Tensor(f!) gather_ids, int force_total, Tensor(g!)? next_new, " + "int num_ctas, Tensor? force_vec) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_shard_accept", &tensorrt_llm::torch_ext::minimaxM3ShardAccept); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SparseDecodeLcOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SparseDecodeLcOp.cpp new file mode 100644 index 000000000000..62a56779f88b --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SparseDecodeLcOp.cpp @@ -0,0 +1,139 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/sparseDecodeLc.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kHeads = 16; +constexpr int64_t kHeadDim = 128; +constexpr int64_t kTopK = 16; + +void checkBytes(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, "minimax_m3_sparse_decode_lc: ", name, + " must be a one-byte CUDA view (uint8 / e4m3 / packed e2m1)"); +} + +} // namespace + +//! One-launch MiniMax-M3 long-context sparse decode over the selected NVFP4 blocks (TP4 geometry: 16 query heads on +//! one KV head, head_dim 128, page 128, top-k 16). +//! +//! @param q E4M3 bytes [T, 16, 128], unit stride in the last dim. +//! @param k, v Packed E2M1 bytes [pages, 1, 128, 64] (the per-layer HND views). +//! @param kBlockScale, vBlockScale E4M3 bytes [pages, 1, 128, 8]: K token-major, V in 4x4 token-quad order. +//! @param kGlobalScale, vGlobalScale One-element float32 per-tensor dequant scales. +//! @param topk Int32 [T, 16] ascending selected block ids with -1 tail (kv head 0 of the head-major table). +//! @param out BF16 [T, 2048], written. q8 / sf, when given, receive the MXFP8 copy and its swizzled UE8M0 scales. +void minimaxM3SparseDecodeLc(torch::Tensor const& q, torch::Tensor const& k, torch::Tensor const& v, + torch::Tensor const& kBlockScale, torch::Tensor const& vBlockScale, torch::Tensor const& kGlobalScale, + torch::Tensor const& vGlobalScale, torch::Tensor const& topk, torch::Tensor const& blockTable, + torch::Tensor const& seqLens, torch::Tensor& out, std::optional q8, std::optional sf, + int64_t numSfKTiles, int64_t decodeQueryLen, double smScale, bool linearSoftmax, bool usePdl, bool earlyTrigger) +{ + TORCH_CHECK(earlyTrigger, "minimax_m3_sparse_decode_lc is instantiated with early_trigger=True only"); + TORCH_CHECK(q.is_cuda(), "minimax_m3_sparse_decode_lc: q must be a CUDA tensor"); + c10::cuda::CUDAGuard const deviceGuard{q.device()}; + TORCH_CHECK(at::cuda::getCurrentDeviceProperties()->major >= 10, + "minimax_m3_sparse_decode_lc requires an SM100-family GPU"); + checkBytes(q, "q"); + checkBytes(k, "k"); + checkBytes(v, "v"); + checkBytes(kBlockScale, "k_block_scale"); + checkBytes(vBlockScale, "v_block_scale"); + TORCH_CHECK(q.dim() == 3 && q.size(1) == kHeads && q.size(2) == kHeadDim && q.stride(2) == 1, + "minimax_m3_sparse_decode_lc: q must be [T, 16, 128] with unit last-dim stride"); + TORCH_CHECK(kGlobalScale.scalar_type() == torch::kFloat32 && vGlobalScale.scalar_type() == torch::kFloat32, + "minimax_m3_sparse_decode_lc: global scales must be float32"); + TORCH_CHECK(topk.scalar_type() == torch::kInt32 && topk.dim() == 2 && topk.size(1) == kTopK, + "minimax_m3_sparse_decode_lc: topk must be int32 [T, 16]"); + TORCH_CHECK(blockTable.scalar_type() == torch::kInt32 && seqLens.scalar_type() == torch::kInt32, + "minimax_m3_sparse_decode_lc: block_table and seq_lens must be int32"); + TORCH_CHECK(out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.size(0) == q.size(0) + && out.size(1) == kHeads * kHeadDim && out.stride(1) == 1, + "minimax_m3_sparse_decode_lc: out must be bfloat16 [T, 2048] with unit last-dim stride"); + TORCH_CHECK(q8.has_value() == sf.has_value(), "minimax_m3_sparse_decode_lc: pass both q8 and sf, or neither"); + bool const mxfp8 = q8.has_value(); + if (mxfp8) + { + checkBytes(*q8, "q8"); + checkBytes(*sf, "sf"); + } + + tensorrt_llm::kernels::MinimaxM3SparseDecodeLcParams prm{}; + prm.q = reinterpret_cast(q.data_ptr()); + prm.q_tok = q.stride(0); + prm.q_head = q.stride(1); + prm.k = reinterpret_cast(k.data_ptr()); + prm.v = reinterpret_cast(v.data_ptr()); + prm.k_blk = k.stride(0); + prm.v_blk = v.stride(0); + prm.ksc = reinterpret_cast(kBlockScale.data_ptr()); + prm.vsc = reinterpret_cast(vBlockScale.data_ptr()); + prm.ksc_blk = kBlockScale.stride(0); + prm.vsc_blk = vBlockScale.stride(0); + prm.kgs = kGlobalScale.data_ptr(); + prm.vgs = vGlobalScale.data_ptr(); + prm.topk = topk.data_ptr(); + prm.t_tok = topk.stride(0); + prm.t_k = topk.stride(1); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.seq_lens = seqLens.data_ptr(); + prm.dql = static_cast(decodeQueryLen); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(out.data_ptr()); + prm.out_stride = out.stride(0); + prm.q8 = mxfp8 ? reinterpret_cast(q8->data_ptr()) : nullptr; + prm.sf = mxfp8 ? reinterpret_cast(sf->data_ptr()) : nullptr; + prm.num_sf_k_tiles = static_cast(numSfKTiles); + + auto const stream = at::cuda::getCurrentCUDAStream(q.get_device()); + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM3SparseDecodeLc( + prm, static_cast(q.size(0)), linearSoftmax, mxfp8, earlyTrigger, usePdl, stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sparse_decode_lc launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sparse_decode_lc(Tensor q, Tensor k, Tensor v, Tensor k_block_scale, Tensor v_block_scale, " + "Tensor k_global_scale, Tensor v_global_scale, Tensor topk, Tensor block_table, Tensor seq_lens, " + "Tensor(a!) out, Tensor(b!)? q8, Tensor(c!)? sf, int num_sf_k_tiles, int decode_query_len, float sm_scale, " + "bool linear_softmax, bool use_pdl, bool early_trigger) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sparse_decode_lc", &tensorrt_llm::torch_ext::minimaxM3SparseDecodeLc); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SpecTailOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SpecTailOp.cpp new file mode 100644 index 000000000000..cea04a45cd86 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3SpecTailOp.cpp @@ -0,0 +1,202 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/specTail.h" + +#include +#include +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace st = tensorrt_llm::kernels::minimaxM3::specTail; + +void check(cudaError_t err) +{ + TORCH_CHECK(err == cudaSuccess, cudaGetErrorString(err)); +} + +cudaStream_t currentStream(torch::Tensor const& t) +{ + return at::cuda::getCurrentCUDAStream(t.get_device()); +} + +__nv_bfloat16 const* bf16In(torch::Tensor const& t) +{ + return reinterpret_cast<__nv_bfloat16 const*>(t.data_ptr()); +} + +__nv_bfloat16* bf16Out(torch::Tensor const& t) +{ + return reinterpret_cast<__nv_bfloat16*>(t.data_ptr()); +} + +//! Register-tile count of the norm kernels for hidden size H (H / 8 elements over kThreads threads). +int normIters(int H) +{ + return (H / 8 + st::kThreads - 1) / st::kThreads; +} + +st::IntView intView(torch::Tensor const& t) +{ + return st::IntView{t.data_ptr(), t.stride(0), t.stride(1), static_cast(t.size(1))}; +} + +} // namespace + +//! out = [rmsnorm(embed(ids)) * wEmbed, rmsnorm(hidden[rows]) * wHidden] [M, 2H]; residual = hidden[rows]. +void minimaxM3DraftInput(torch::Tensor const& ids, torch::Tensor const& table, torch::Tensor const& hidden, + std::optional const& rows, torch::Tensor const& wEmbed, torch::Tensor const& wHidden, + double epsEmbed, double epsHidden, torch::Tensor& out, std::optional const& residual, bool pdl) +{ + c10::cuda::CUDAGuard const deviceGuard{table.device()}; + int const M = ids.size(0); + int const H = table.size(1); + TORCH_CHECK(ids.scalar_type() == at::kInt && ids.is_contiguous() && table.scalar_type() == at::kBFloat16 + && table.is_contiguous() && H % 8 == 0); + TORCH_CHECK( + hidden.scalar_type() == at::kBFloat16 && hidden.dim() == 2 && hidden.is_contiguous() && hidden.size(1) == H); + TORCH_CHECK(out.scalar_type() == at::kBFloat16 && out.size(0) == M && out.size(1) == 2 * H && out.is_contiguous()); + TORCH_CHECK(wEmbed.scalar_type() == at::kBFloat16 && wEmbed.numel() == H && wHidden.scalar_type() == at::kBFloat16 + && wHidden.numel() == H); + TORCH_CHECK(!rows.has_value() || (rows->scalar_type() == at::kLong && rows->is_contiguous() && rows->size(0) == M)); + TORCH_CHECK(rows.has_value() || hidden.size(0) >= M, "draft_input: hidden must hold a row per id"); + TORCH_CHECK(!residual.has_value() + || (residual->scalar_type() == at::kBFloat16 && residual->is_contiguous() && residual->size(0) == M + && residual->size(1) == H), + "draft_input: residual must be contiguous [M, H]"); + TORCH_CHECK(normIters(H) <= st::kMaxIters, "draft_input: hidden size too large"); + check(st::invokeDraftInput(ids.data_ptr(), static_cast(table.size(0)), bf16In(table), bf16In(hidden), + rows.has_value() ? reinterpret_cast(rows->data_ptr()) : nullptr, bf16In(wEmbed), + bf16In(wHidden), static_cast(epsEmbed), static_cast(epsHidden), H, M, bf16Out(out), + residual.has_value() ? bf16Out(*residual) : nullptr, pdl, currentStream(table))); +} + +//! Per-chunk RMSNorm of src [M, >= C*H] (row stride may exceed C*H) with weights w [C, H] into dst [M, C*H]. +void minimaxM3ChunkNorm(torch::Tensor const& src, torch::Tensor const& w, double eps, torch::Tensor& dst, bool pdl) +{ + c10::cuda::CUDAGuard const deviceGuard{src.device()}; + int const M = src.size(0); + int const C = w.size(0); + int const H = w.size(1); + TORCH_CHECK(src.scalar_type() == at::kBFloat16 && w.scalar_type() == at::kBFloat16 && src.stride(1) == 1); + TORCH_CHECK(src.size(1) >= C * H && dst.is_contiguous() && dst.numel() == static_cast(M) * C * H); + TORCH_CHECK(H % 8 == 0 && src.stride(0) % 8 == 0); + TORCH_CHECK(normIters(H) <= st::kMaxIters, "chunk_norm: hidden size too large"); + check(st::invokeChunkNorm(bf16In(src), src.stride(0), bf16In(w), static_cast(eps), H, C, M, bf16Out(dst), + pdl, currentStream(src))); +} + +//! dst [M, H] (row stride may exceed H) = rmsnorm(bf16(h + r)) * w, r optional. +void minimaxM3CaptureNorm(torch::Tensor const& h, std::optional const& r, torch::Tensor const& w, + double eps, torch::Tensor& dst) +{ + c10::cuda::CUDAGuard const deviceGuard{h.device()}; + int const M = h.size(0); + int const H = w.size(0); + TORCH_CHECK( + h.scalar_type() == at::kBFloat16 && w.scalar_type() == at::kBFloat16 && dst.scalar_type() == at::kBFloat16); + TORCH_CHECK(h.dim() == 2 && h.size(1) == H && h.stride(1) == 1 && dst.dim() == 2 && dst.size(0) == M + && dst.size(1) == H && dst.stride(1) == 1 && H % 8 == 0 && h.stride(0) % 8 == 0 && dst.stride(0) % 8 == 0); + TORCH_CHECK(!r.has_value() + || (r->sizes() == h.sizes() && r->stride(1) == 1 && r->stride(0) % 8 == 0 + && r->scalar_type() == at::kBFloat16)); + TORCH_CHECK(normIters(H) <= st::kMaxIters, "capture_norm: hidden size too large"); + check(st::invokeCaptureNorm(bf16In(h), h.stride(0), r.has_value() ? bf16In(*r) : nullptr, + r.has_value() ? r->stride(0) : 0, bf16In(w), static_cast(eps), H, M, bf16Out(dst), dst.stride(0), + currentStream(h))); +} + +//! The overlap scheduler's step inputs from the previous step's device outputs (see step_gather_kernel). +void minimaxM3StepGather(torch::Tensor const& newTok, torch::Tensor const& nextDraft, torch::Tensor const& lens, + torch::Tensor const& slots, torch::Tensor const& ppi, int64_t P, int64_t T, int64_t D, torch::Tensor& ids, + int64_t ids0, torch::Tensor& draft, int64_t d0, torch::Tensor& posOff, torch::Tensor& kvOff, int64_t b0) +{ + c10::cuda::CUDAGuard const deviceGuard{newTok.device()}; + for (auto const* t : std::initializer_list{ + &newTok, &nextDraft, &lens, &slots, &ppi, &ids, &draft, &posOff, &kvOff}) + { + TORCH_CHECK(t->scalar_type() == at::kInt, "step_gather: int32 tensors expected"); + } + TORCH_CHECK(newTok.dim() >= 2 && nextDraft.dim() == 2 && lens.is_contiguous() && posOff.is_contiguous() + && kvOff.is_contiguous() && ids.is_contiguous() && draft.is_contiguous()); + int const work = std::max(std::max(P * T, P * D), std::max(posOff.numel(), kvOff.numel())); + check(st::invokeStepGather(newTok.data_ptr(), newTok.stride(0), newTok.stride(1), nextDraft.data_ptr(), + nextDraft.stride(0), nextDraft.stride(1), lens.data_ptr(), slots.data_ptr(), ppi.data_ptr(), + static_cast(P), static_cast(T), static_cast(D), ids.data_ptr(), static_cast(ids0), + draft.data_ptr(), static_cast(d0), posOff.data_ptr(), static_cast(posOff.numel()), + kvOff.data_ptr(), static_cast(kvOff.numel()), static_cast(b0), work, currentStream(newTok))); +} + +//! The spec sampler's slot stores from one step's outputs (see slot_scatter_kernel). +void minimaxM3SlotScatter(torch::Tensor const& slots, torch::Tensor const& sa, torch::Tensor& da, + torch::Tensor const& sb, torch::Tensor& db, torch::Tensor const& sl, torch::Tensor& dl, torch::Tensor const& sd, + torch::Tensor& dd) +{ + c10::cuda::CUDAGuard const deviceGuard{slots.device()}; + for (auto const* t : std::initializer_list{&sa, &da, &sb, &db, &sl, &dl, &sd, &dd}) + { + TORCH_CHECK(t->scalar_type() == at::kInt && t->dim() <= 2, "slot_scatter: int32 tensors expected"); + } + TORCH_CHECK(slots.scalar_type() == at::kLong); + int const n = slots.numel(); + if (n == 0) + { + return; + } + check(st::invokeSlotScatter(reinterpret_cast(slots.data_ptr()), n, intView(sa), intView(da), + intView(sb), intView(db), sl.data_ptr(), dl.data_ptr(), intView(sd), intView(dd), + currentStream(slots))); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_draft_input(Tensor ids, Tensor table, Tensor hidden, Tensor? rows, Tensor w_embed, " + "Tensor w_hidden, float eps_embed, float eps_hidden, Tensor(a!) out, Tensor(b!)? residual, bool pdl) -> ()"); + m.def("minimax_m3_chunk_norm(Tensor src, Tensor w, float eps, Tensor(a!) dst, bool pdl) -> ()"); + m.def("minimax_m3_capture_norm(Tensor h, Tensor? r, Tensor w, float eps, Tensor(a!) dst) -> ()"); + m.def( + "minimax_m3_step_gather(Tensor new_tok, Tensor next_draft, Tensor lens, Tensor slots, Tensor ppi, int P, " + "int T, int D, Tensor(a!) ids, int ids0, Tensor(b!) draft, int d0, Tensor(c!) pos_off, Tensor(d!) kv_off, " + "int b0) -> ()"); + m.def( + "minimax_m3_slot_scatter(Tensor slots, Tensor sa, Tensor(a!) da, Tensor sb, Tensor(b!) db, Tensor sl, " + "Tensor(c!) dl, Tensor sd, Tensor(d!) dd) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_draft_input", &tensorrt_llm::torch_ext::minimaxM3DraftInput); + m.impl("minimax_m3_chunk_norm", &tensorrt_llm::torch_ext::minimaxM3ChunkNorm); + m.impl("minimax_m3_capture_norm", &tensorrt_llm::torch_ext::minimaxM3CaptureNorm); + m.impl("minimax_m3_step_gather", &tensorrt_llm::torch_ext::minimaxM3StepGather); + m.impl("minimax_m3_slot_scatter", &tensorrt_llm::torch_ext::minimaxM3SlotScatter); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3TargetHeadOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3TargetHeadOp.cpp new file mode 100644 index 000000000000..448989e08c3a --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3TargetHeadOp.cpp @@ -0,0 +1,123 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3/targetHead.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace th = tensorrt_llm::kernels::minimaxM3::targetHead; + +void checkArch(torch::Tensor const& t) +{ + TORCH_CHECK(t.is_cuda(), "minimax_m3_target_head expects CUDA tensors"); + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_target_head requires SM100 or newer"); +} + +} // namespace + +//! out [M, n] (BF16) = x [M <= 8, K] (BF16) x W^T for the 12-bit coded BF16 shard W [n, K]: w the coded stream +//! (uint8 [tiles * K / 64 * 1536]), exc_off int32 [tiles + 1] / exc_pos int32 [E] / exc_w fp32 [E, 2] its exception +//! lists, exp_base its exponent window's base. part (fp32 [grid * 16 * 256]) and counter (int32 [tiles], zero) are +//! the split-tile scratch. +void minimaxM3TargetHead(torch::Tensor const& w, torch::Tensor const& x, torch::Tensor& out, torch::Tensor& part, + torch::Tensor& counter, torch::Tensor const& excOff, torch::Tensor const& excPos, torch::Tensor const& excW, + int64_t expBase, int64_t n, int64_t grid, bool pdl) +{ + checkArch(x); + for (torch::Tensor const* t : + std::initializer_list{&w, &out, &part, &counter, &excOff, &excPos, &excW}) + { + TORCH_CHECK(t->device() == x.device(), "target_head: every tensor must be on x's device"); + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + TORCH_CHECK(x.scalar_type() == at::kBFloat16 && x.dim() == 2, "target_head: x must be BF16 [M, K]"); + int const M = static_cast(x.size(0)); + int const K = static_cast(x.size(1)); + TORCH_CHECK(M >= 1 && M <= th::kMaxRows, "target_head: 1..", th::kMaxRows, " rows"); + TORCH_CHECK(K % 64 == 0 && K / 16 <= 0xFFFF, "target_head: K must be a multiple of 64"); + TORCH_CHECK(x.stride(1) == 1 && x.stride(0) % 8 == 0 && reinterpret_cast(x.data_ptr()) % 16 == 0, + "target_head: x rows must be unit-stride and 16-byte aligned"); + TORCH_CHECK(n >= 16 && n % 16 == 0, "target_head: n must be a positive multiple of 16 (whole tiles)"); + int64_t const tiles = (n + 15) / 16; + TORCH_CHECK(w.scalar_type() == at::kByte && w.is_contiguous() && w.numel() == tiles * 16 * K * 3 / 2, + "target_head: w must be the contiguous uint8 coded stream of [ceil(n / 16) * 16, K]"); + TORCH_CHECK(out.scalar_type() == at::kBFloat16 && out.dim() == 2 && out.size(0) >= M && out.size(1) >= n + && out.stride(1) == 1, + "target_head: out must be BF16 [M, >= n] with unit column stride"); + TORCH_CHECK(grid >= 1, "target_head: grid must be positive"); + TORCH_CHECK(part.scalar_type() == at::kFloat && part.is_contiguous() && part.numel() >= grid * th::kWarps * 2 * 128, + "target_head: part must be fp32 [grid * 16 * 256]"); + TORCH_CHECK(counter.scalar_type() == at::kInt && counter.is_contiguous() && counter.numel() >= tiles, + "target_head: counter must be int32 [tiles]"); + TORCH_CHECK(excOff.scalar_type() == at::kInt && excOff.is_contiguous() && excOff.numel() == tiles + 1, + "target_head: exc_off must be int32 [tiles + 1]"); + TORCH_CHECK(excPos.scalar_type() == at::kInt && excPos.is_contiguous(), "target_head: exc_pos must be int32"); + TORCH_CHECK(excW.scalar_type() == at::kFloat && excW.is_contiguous() && excW.numel() == 2 * excPos.numel(), + "target_head: exc_w must be fp32 [E, 2]"); + TORCH_CHECK(expBase >= 0 && expBase <= 240, "target_head: exponent base out of range"); + th::Params p{}; + p.w = reinterpret_cast(w.data_ptr()); + p.x = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.x_stride = x.stride(0); + p.out = reinterpret_cast<__nv_bfloat16*>(out.data_ptr()); + p.out_stride = out.stride(0); + p.part = part.data_ptr(); + p.counter = counter.data_ptr(); + p.exc_off = excOff.data_ptr(); + p.exc_pos = reinterpret_cast(excPos.data_ptr()); + p.exc_w = reinterpret_cast(excW.data_ptr()); + p.exp_base = static_cast((expBase << 7) | (expBase << 23)); + p.M = M; + p.N = static_cast(n); + cudaError_t const err + = th::invokeTargetHead(p, K, static_cast(grid), pdl, at::cuda::getCurrentCUDAStream(x.get_device())); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, cudaGetErrorString(err)); + } +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_target_head(Tensor w, Tensor x, Tensor(a!) out, Tensor(b!) part, Tensor(c!) counter, " + "Tensor exc_off, Tensor exc_pos, Tensor exc_w, int exp_base, int n, int grid, bool pdl) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_target_head", &tensorrt_llm::torch_ext::minimaxM3TargetHead); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Tp8FusedScoreSelectOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Tp8FusedScoreSelectOp.cpp new file mode 100644 index 000000000000..1d28f41fdc7a --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Tp8FusedScoreSelectOp.cpp @@ -0,0 +1,173 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Tp8/fusedScoreSelect.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kHeadDim = 128; +constexpr int64_t kBlockK = 128; +constexpr int64_t kTopK = 16; + +} // namespace + +//! MiniMax-M3 TP8 fused index score + top-16 block select for decode spans of at most 2 requests. Equals +//! minimax_m3_select_blocks applied to the block scores of cute_dsl_minimax_m3_index_decode_score. Grid (splits, +//! batch) with splits = max(SM count / batch, ceil(max_tiles / 128), 1). +//! +//! @param idxQ E4M3 [total_q, heads, 128] (total_q = batch * dql, heads * 4 <= 8), unit last-dim stride, row and head +//! strides multiples of 4 bytes, 4-byte aligned. +//! @param idxKPages E4M3 [num_pages, 128, 128], unit last-dim stride, page and row strides multiples of 16 bytes, +//! 16-byte aligned. +//! @param blockTable Int32 [batch, >= blocks], unit last-dim stride. +//! @param seqLens Int32 [batch] (1 or 2 requests), including this step's dql query rows. +//! @param nValid Contiguous int32 [total_q] per-token causal valid-block counts. +//! @param out Int32, logical [total_q, num_kv_heads, 16] with unit last-dim stride (a permuted head-major backing is +//! fine). Every entry is written: ascending block ids with a -1 tail. +//! @param scratch Contiguous float32 [>= batch, >= num_kv_heads * dql, L] with L >= max_tiles and L % 4 == 0, 16-byte +//! aligned; [batch, 8, ceil4(max_tiles)] covers every supported geometry. Contents need no initialization. +//! @param counters Contiguous int32 with >= batch elements, all zero before the first call. The kernel returns each +//! request's counter to zero before it exits, so the buffer stays reusable across calls and graph replays without a +//! reset (as long as launches sharing it are stream-ordered). +//! @param maxTiles Worst-case block count n_valid is clamped to; num_kv_heads * dql * ceil4(max_tiles) * 4 + +//! num_kv_heads * dql * 1024 must not exceed 196608 bytes. +void minimaxM3Tp8FusedScoreSelect(torch::Tensor const& idxQ, torch::Tensor const& idxKPages, + torch::Tensor const& blockTable, torch::Tensor const& seqLens, torch::Tensor const& nValid, torch::Tensor& out, + torch::Tensor& scratch, torch::Tensor& counters, int64_t decodeQueryLen, int64_t numKvHeads, int64_t initBlocks, + int64_t localBlocks, int64_t maxTiles, bool usePdl) +{ + namespace tk = tensorrt_llm::kernels; + TORCH_CHECK(idxQ.is_cuda(), "minimax_m3_tp8_fused_score_select: idx_q must be a CUDA tensor"); + c10::cuda::CUDAGuard const deviceGuard{idxQ.device()}; + auto const* props = at::cuda::getCurrentDeviceProperties(); + TORCH_CHECK(props->major >= 10, "minimax_m3_tp8_fused_score_select requires an SM100-family GPU"); + + TORCH_CHECK(idxQ.dim() == 3 && idxQ.size(2) == kHeadDim && idxQ.stride(2) == 1 && idxQ.element_size() == 1, + "minimax_m3_tp8_fused_score_select: idx_q must be one-byte [total_q, heads, 128] with unit last-dim stride"); + TORCH_CHECK( + idxQ.stride(0) % 4 == 0 && idxQ.stride(1) % 4 == 0 && (reinterpret_cast(idxQ.data_ptr()) & 3) == 0, + "minimax_m3_tp8_fused_score_select: idx_q strides and base must be 4-byte aligned"); + TORCH_CHECK(idxKPages.is_cuda() && idxKPages.element_size() == 1 && idxKPages.dim() == 3 + && idxKPages.size(1) == kBlockK && idxKPages.size(2) == kHeadDim && idxKPages.stride(2) == 1, + "minimax_m3_tp8_fused_score_select: idx_k_pages must be one-byte [num_pages, 128, 128] with unit last-dim " + "stride"); + TORCH_CHECK(idxKPages.stride(1) % 16 == 0 && idxKPages.stride(0) % 16 == 0 + && (reinterpret_cast(idxKPages.data_ptr()) & 15) == 0, + "minimax_m3_tp8_fused_score_select: idx_k_pages strides and base must be 16-byte aligned"); + TORCH_CHECK(blockTable.scalar_type() == torch::kInt32 && blockTable.dim() == 2 && blockTable.stride(1) == 1, + "minimax_m3_tp8_fused_score_select: block_table must be int32 [batch, blocks] with unit last-dim stride"); + TORCH_CHECK( + seqLens.scalar_type() == torch::kInt32 && nValid.scalar_type() == torch::kInt32 && nValid.is_contiguous(), + "minimax_m3_tp8_fused_score_select: seq_lens and n_valid must be int32, n_valid contiguous"); + TORCH_CHECK(out.scalar_type() == torch::kInt32 && out.dim() == 3 && out.size(2) == kTopK && out.stride(2) == 1, + "minimax_m3_tp8_fused_score_select: out must be int32 [total_q, num_kv_heads, 16] with unit last-dim stride"); + + int64_t const batch = seqLens.size(0); + int64_t const totalQ = idxQ.size(0); + int64_t const heads = idxQ.size(1); + int64_t const rows = decodeQueryLen * numKvHeads; + TORCH_CHECK(batch >= 1 && batch <= tk::kMinimaxM3Tp8FusedScoreSelectMaxBatch, + "minimax_m3_tp8_fused_score_select takes 1 to ", tk::kMinimaxM3Tp8FusedScoreSelectMaxBatch, " requests, got ", + batch); + TORCH_CHECK(decodeQueryLen >= 1 && decodeQueryLen <= tk::kMinimaxM3Tp8FusedScoreSelectMaxDql + && totalQ == batch * decodeQueryLen, + "minimax_m3_tp8_fused_score_select: decode_query_len must be in [1, 4] with total_q = batch * " + "decode_query_len"); + TORCH_CHECK(numKvHeads >= 1 && heads % numKvHeads == 0 && heads * tk::kMinimaxM3Tp8FusedScoreSelectMaxDql <= 8 + && rows <= tk::kMinimaxM3Tp8FusedScoreSelectMaxRows, + "minimax_m3_tp8_fused_score_select: unsupported head geometry"); + TORCH_CHECK(out.size(0) == totalQ && out.size(1) == numKvHeads, + "minimax_m3_tp8_fused_score_select: out must be [total_q, num_kv_heads, 16]"); + TORCH_CHECK(nValid.numel() >= totalQ, "minimax_m3_tp8_fused_score_select: n_valid needs total_q elements"); + TORCH_CHECK(maxTiles >= 1, "minimax_m3_tp8_fused_score_select: max_tiles must be positive"); + + TORCH_CHECK(scratch.is_cuda() && scratch.scalar_type() == torch::kFloat32 && scratch.dim() == 3 + && scratch.is_contiguous() && scratch.size(0) >= batch && scratch.size(1) >= rows + && scratch.size(2) >= maxTiles && scratch.stride(1) % 4 == 0 + && (reinterpret_cast(scratch.data_ptr()) & 15) == 0, + "minimax_m3_tp8_fused_score_select: scratch must be contiguous 16-byte aligned float32 [>= batch, >= rows, " + "L] with L >= max_tiles and L % 4 == 0"); + TORCH_CHECK(counters.is_cuda() && counters.scalar_type() == torch::kInt32 && counters.is_contiguous() + && counters.numel() >= batch, + "minimax_m3_tp8_fused_score_select: counters must be contiguous int32 with >= batch elements"); + TORCH_CHECK(rows * ((maxTiles + 3) & ~int64_t{3}) * 4 + rows * 1024 <= tk::kMinimaxM3Tp8FusedScoreSelectRingBytes, + "minimax_m3_tp8_fused_score_select: select scratch does not fit the stage ring"); + + int64_t const splits = std::max({props->multiProcessorCount / batch, + (maxTiles + tk::kMinimaxM3Tp8FusedScoreSelectMaxLocalBlocks - 1) + / tk::kMinimaxM3Tp8FusedScoreSelectMaxLocalBlocks, + int64_t{1}}); + TORCH_CHECK((maxTiles + splits - 1) / splits <= tk::kMinimaxM3Tp8FusedScoreSelectMaxLocalBlocks, + "minimax_m3_tp8_fused_score_select: too few splits"); + + tk::MinimaxM3Tp8FusedScoreSelectParams p{}; + p.q = reinterpret_cast(idxQ.data_ptr()); + p.q_st = idxQ.stride(0); + p.q_sh = idxQ.stride(1); + p.bt = blockTable.data_ptr(); + p.bt_stride = blockTable.stride(0); + p.bt_width = static_cast(blockTable.size(1)); + p.seq_lens = seqLens.data_ptr(); + p.n_valid = nValid.data_ptr(); + p.scr = scratch.data_ptr(); + p.scr_b = scratch.stride(0); + p.scr_r = scratch.stride(1); + p.out = out.data_ptr(); + p.out_t = out.stride(0); + p.out_h = out.stride(1); + p.counters = counters.data_ptr(); + p.heads = static_cast(heads); + p.kv_heads = static_cast(numKvHeads); + p.dql = static_cast(decodeQueryLen); + p.max_tiles = static_cast(maxTiles); + p.init_blocks = static_cast(initBlocks); + p.local_blocks = static_cast(localBlocks); + + auto const stream = at::cuda::getCurrentCUDAStream(idxQ.get_device()); + cudaError_t const err + = tk::invokeMinimaxM3Tp8FusedScoreSelect(p, idxKPages.data_ptr(), idxKPages.size(0), idxKPages.stride(1), + idxKPages.stride(0), static_cast(batch), static_cast(splits), usePdl, stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_tp8_fused_score_select launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_tp8_fused_score_select(Tensor idx_q, Tensor idx_k_pages, Tensor block_table, Tensor seq_lens, " + "Tensor n_valid, Tensor(a!) out, Tensor(b!) scratch, Tensor(c!) counters, int decode_query_len, " + "int num_kv_heads, int init_blocks, int local_blocks, int max_tiles, bool use_pdl) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_tp8_fused_score_select", &tensorrt_llm::torch_ext::minimaxM3Tp8FusedScoreSelect); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Tp8SparseDecodeLc8Op.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Tp8SparseDecodeLc8Op.cpp new file mode 100644 index 000000000000..3f40efdb9bb6 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Tp8SparseDecodeLc8Op.cpp @@ -0,0 +1,139 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Tp8/sparseDecodeLc8.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kHeads = 8; +constexpr int64_t kHeadDim = 128; +constexpr int64_t kTopK = 16; + +void checkBytes(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, "minimax_m3_tp8_sparse_decode_lc8: ", name, + " must be a one-byte CUDA view (uint8 / e4m3 / packed e2m1)"); +} + +} // namespace + +//! One-launch MiniMax-M3 TP8 long-context sparse decode over the selected NVFP4 blocks (8 query heads on one KV head, +//! head_dim 128, page 128, top-k 16). Dependents are released right after the grid dependency wait. +//! +//! @param q E4M3 bytes [T, 8, 128], unit stride in the last dim. +//! @param k, v Packed E2M1 bytes [pages, 1, 128, 64] (the per-layer HND views). +//! @param kBlockScale, vBlockScale E4M3 bytes [pages, 1, 128, 8]: K token-major, V in 4x4 token-quad order. +//! @param kGlobalScale, vGlobalScale One-element float32 per-tensor dequant scales. +//! @param topk Int32 [T, 16] ascending selected block ids with -1 tail (kv head 0 of the head-major table). +//! @param out BF16 [T, 1024], written. q8 / sf, when given, receive the MXFP8 copy and its swizzled UE8M0 scales. +//! @param prewait Stage seq_lens, q and the request's block-table row before the grid dependency wait (same output). +void minimaxM3Tp8SparseDecodeLc8(torch::Tensor const& q, torch::Tensor const& k, torch::Tensor const& v, + torch::Tensor const& kBlockScale, torch::Tensor const& vBlockScale, torch::Tensor const& kGlobalScale, + torch::Tensor const& vGlobalScale, torch::Tensor const& topk, torch::Tensor const& blockTable, + torch::Tensor const& seqLens, torch::Tensor& out, std::optional q8, std::optional sf, + int64_t numSfKTiles, int64_t decodeQueryLen, double smScale, bool linearSoftmax, bool usePdl, bool prewait) +{ + TORCH_CHECK(q.is_cuda(), "minimax_m3_tp8_sparse_decode_lc8: q must be a CUDA tensor"); + c10::cuda::CUDAGuard const deviceGuard{q.device()}; + TORCH_CHECK(at::cuda::getCurrentDeviceProperties()->major >= 10, + "minimax_m3_tp8_sparse_decode_lc8 requires an SM100-family GPU"); + checkBytes(q, "q"); + checkBytes(k, "k"); + checkBytes(v, "v"); + checkBytes(kBlockScale, "k_block_scale"); + checkBytes(vBlockScale, "v_block_scale"); + TORCH_CHECK(q.dim() == 3 && q.size(1) == kHeads && q.size(2) == kHeadDim && q.stride(2) == 1, + "minimax_m3_tp8_sparse_decode_lc8: q must be [T, 8, 128] with unit last-dim stride"); + TORCH_CHECK(kGlobalScale.scalar_type() == torch::kFloat32 && vGlobalScale.scalar_type() == torch::kFloat32, + "minimax_m3_tp8_sparse_decode_lc8: global scales must be float32"); + TORCH_CHECK(topk.scalar_type() == torch::kInt32 && topk.dim() == 2 && topk.size(1) == kTopK, + "minimax_m3_tp8_sparse_decode_lc8: topk must be int32 [T, 16]"); + TORCH_CHECK(blockTable.scalar_type() == torch::kInt32 && seqLens.scalar_type() == torch::kInt32, + "minimax_m3_tp8_sparse_decode_lc8: block_table and seq_lens must be int32"); + TORCH_CHECK(out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.size(0) == q.size(0) + && out.size(1) == kHeads * kHeadDim && out.stride(1) == 1, + "minimax_m3_tp8_sparse_decode_lc8: out must be bfloat16 [T, 1024] with unit last-dim stride"); + TORCH_CHECK(q8.has_value() == sf.has_value(), "minimax_m3_tp8_sparse_decode_lc8: pass both q8 and sf, or neither"); + bool const mxfp8 = q8.has_value(); + if (mxfp8) + { + checkBytes(*q8, "q8"); + checkBytes(*sf, "sf"); + } + + tensorrt_llm::kernels::MinimaxM3Tp8SparseDecodeLc8Params prm{}; + prm.q = reinterpret_cast(q.data_ptr()); + prm.q_tok = q.stride(0); + prm.q_head = q.stride(1); + prm.k = reinterpret_cast(k.data_ptr()); + prm.v = reinterpret_cast(v.data_ptr()); + prm.k_blk = k.stride(0); + prm.v_blk = v.stride(0); + prm.ksc = reinterpret_cast(kBlockScale.data_ptr()); + prm.vsc = reinterpret_cast(vBlockScale.data_ptr()); + prm.ksc_blk = kBlockScale.stride(0); + prm.vsc_blk = vBlockScale.stride(0); + prm.kgs = kGlobalScale.data_ptr(); + prm.vgs = vGlobalScale.data_ptr(); + prm.topk = topk.data_ptr(); + prm.t_tok = topk.stride(0); + prm.t_k = topk.stride(1); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.seq_lens = seqLens.data_ptr(); + prm.dql = static_cast(decodeQueryLen); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(out.data_ptr()); + prm.out_stride = out.stride(0); + prm.q8 = mxfp8 ? reinterpret_cast(q8->data_ptr()) : nullptr; + prm.sf = mxfp8 ? reinterpret_cast(sf->data_ptr()) : nullptr; + prm.num_sf_k_tiles = static_cast(numSfKTiles); + + auto const stream = at::cuda::getCurrentCUDAStream(q.get_device()); + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM3Tp8SparseDecodeLc8( + prm, static_cast(q.size(0)), linearSoftmax, mxfp8, prewait, usePdl, stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_tp8_sparse_decode_lc8 launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_tp8_sparse_decode_lc8(Tensor q, Tensor k, Tensor v, Tensor k_block_scale, Tensor v_block_scale, " + "Tensor k_global_scale, Tensor v_global_scale, Tensor topk, Tensor block_table, Tensor seq_lens, " + "Tensor(a!) out, Tensor(b!)? q8, Tensor(c!)? sf, int num_sf_k_tiles, int decode_query_len, float sm_scale, " + "bool linear_softmax, bool use_pdl, bool prewait) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_tp8_sparse_decode_lc8", &tensorrt_llm::torch_ext::minimaxM3Tp8SparseDecodeLc8); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/stepInputsScatterOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/stepInputsScatterOp.cpp new file mode 100644 index 000000000000..b78a779a29d1 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/stepInputsScatterOp.cpp @@ -0,0 +1,57 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3/stepInputsScatter.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! Issue a batch of packed host-to-device and device-to-device copies in one launch. +//! +//! @param stage Contiguous uint8 CUDA tensor laid out as described in kernels/minimaxM3/stepInputsScatter.h. Its +//! destinations are raw device addresses: the caller guarantees each one is live and that no other work on the stream +//! writes it between staging and this call. +void stepInputsScatter(torch::Tensor const& stage) +{ + TORCH_CHECK(stage.is_cuda(), "step_inputs_scatter expects a CUDA staging buffer"); + TORCH_CHECK(stage.scalar_type() == torch::kUInt8, "step_inputs_scatter expects a uint8 staging buffer"); + TORCH_CHECK(stage.is_contiguous(), "step_inputs_scatter expects a contiguous staging buffer"); + TORCH_CHECK(stage.numel() >= 16, "step_inputs_scatter staging buffer is shorter than its header"); + c10::cuda::CUDAGuard const deviceGuard{stage.device()}; + tensorrt_llm::kernels::invokeStepInputsScatter( + stage.data_ptr(), at::cuda::getCurrentCUDAStream(stage.get_device())); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("step_inputs_scatter(Tensor stage) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("step_inputs_scatter", &tensorrt_llm::torch_ext::stepInputsScatter); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp new file mode 100644 index 000000000000..aa2e7e5fdd68 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp @@ -0,0 +1,140 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! ST-01-REJ's multi-CTA chain speculative sampling (see kernels/minimaxM31/chainSpecSamplingMcSm103.h): the inputs +//! and outputs of trtllm::chain_speculative_sampling (output_token_ids [batch, numDraft + 1], num_tokens [batch]), +//! bitwise its results, on batch x ctas_per_row CTAs. workspace: a zero-initialized byte buffer of at least +//! batch x minimax_m31_chain_speculative_sampling_mc_ws_bytes(), reused across calls; ctas_per_row divides 1024 and +//! batch x ctas_per_row must not exceed the SM count when ctas_per_row > 1. +std::tuple minimaxM31ChainSpecSamplingMc(torch::Tensor const& draftProbs, + torch::Tensor const& draftTokenIds, torch::Tensor const& targetProbs, torch::Tensor const& seed, + torch::Tensor const& offset, torch::Tensor& workspace, int64_t ctasPerRow) +{ + TORCH_CHECK(draftProbs.dim() == 3, + "minimax_m31_chain_speculative_sampling_mc: draft_probs must be [batch, " + "numDraft, vocab]"); + TORCH_CHECK(draftTokenIds.dim() == 2, + "minimax_m31_chain_speculative_sampling_mc: draft_token_ids must be [batch, " + "numDraft]"); + TORCH_CHECK(targetProbs.dim() == 3, + "minimax_m31_chain_speculative_sampling_mc: target_probs must be [batch, numDraft + 1, vocab]"); + auto const batchSize = draftProbs.size(0); + auto const numDraft = draftProbs.size(1); + auto const vocabSize = draftProbs.size(2); + TORCH_CHECK(draftTokenIds.size(0) == batchSize && draftTokenIds.size(1) == numDraft, + "minimax_m31_chain_speculative_sampling_mc: draft_token_ids must be [batch, numDraft]"); + TORCH_CHECK( + targetProbs.size(0) == batchSize && targetProbs.size(1) == numDraft + 1 && targetProbs.size(2) == vocabSize, + "minimax_m31_chain_speculative_sampling_mc: target_probs must be [batch, numDraft + 1, vocab]"); + for (auto const* t : {&draftProbs, &draftTokenIds, &targetProbs, &seed, &offset}) + { + TORCH_CHECK(t->is_cuda() && t->is_contiguous() && t->get_device() == draftProbs.get_device(), + "minimax_m31_chain_speculative_sampling_mc: inputs must be contiguous CUDA tensors on one device"); + } + TORCH_CHECK(draftProbs.scalar_type() == torch::kFloat32 && targetProbs.scalar_type() == torch::kFloat32, + "minimax_m31_chain_speculative_sampling_mc: draft_probs and target_probs must be float32"); + TORCH_CHECK(draftTokenIds.scalar_type() == torch::kInt32, + "minimax_m31_chain_speculative_sampling_mc: draft_token_ids must be int32"); + TORCH_CHECK(seed.scalar_type() == torch::kInt64 && offset.scalar_type() == torch::kInt64, + "minimax_m31_chain_speculative_sampling_mc: seed and offset must be int64 tensors"); + TORCH_CHECK(seed.numel() == batchSize && offset.numel() == batchSize, + "minimax_m31_chain_speculative_sampling_mc: seed and offset must hold one entry per request"); + TORCH_CHECK(vocabSize % 4 == 0 && vocabSize <= tensorrt_llm::kernels::minimaxM31::kChainSpecMcMaxVocab + && reinterpret_cast(draftProbs.data_ptr()) % 16 == 0 + && reinterpret_cast(targetProbs.data_ptr()) % 16 == 0, + "minimax_m31_chain_speculative_sampling_mc: the probs must be 16-byte aligned with a vocab divisible by 4 and " + "at most ", + tensorrt_llm::kernels::minimaxM31::kChainSpecMcMaxVocab); + TORCH_CHECK(batchSize <= 65535, "minimax_m31_chain_speculative_sampling_mc: batch must be <= 65535"); + TORCH_CHECK(ctasPerRow >= 1 && ctasPerRow <= 1024 && 1024 % ctasPerRow == 0, + "minimax_m31_chain_speculative_sampling_mc: ctas_per_row must divide 1024"); + auto const wsBytesPerRow + = static_cast(tensorrt_llm::kernels::minimaxM31::chainSpecSamplingMcWorkspaceBytesPerRow()); + TORCH_CHECK(workspace.is_cuda() && workspace.is_contiguous() && workspace.get_device() == draftProbs.get_device() + && reinterpret_cast(workspace.data_ptr()) % 16 == 0 + && workspace.numel() * workspace.element_size() >= wsBytesPerRow * batchSize, + "minimax_m31_chain_speculative_sampling_mc: workspace must be a contiguous, 16-byte aligned CUDA buffer of at " + "least ", + wsBytesPerRow * batchSize, " bytes on draft_probs' device"); + + c10::cuda::CUDAGuard const deviceGuard{draftProbs.device()}; + TORCH_CHECK( + ctasPerRow == 1 || batchSize * ctasPerRow <= at::cuda::getCurrentDeviceProperties()->multiProcessorCount, + "minimax_m31_chain_speculative_sampling_mc: batch x ctas_per_row must not exceed the SM count"); + auto outputTokenIds = torch::empty({batchSize, numDraft + 1}, draftTokenIds.options()); + auto outputNumTokens = torch::empty({batchSize}, draftTokenIds.options()); + if (batchSize == 0) + { + return {outputTokenIds, outputNumTokens}; + } + + tensorrt_llm::kernels::speculative_decoding::ChainSpeculativeSamplingParams params; + params.draftProbs = draftProbs.const_data_ptr(); + params.draftTokenIds = draftTokenIds.const_data_ptr(); + params.targetProbs = targetProbs.const_data_ptr(); + params.seed = seed.const_data_ptr(); + params.offset = offset.const_data_ptr(); + params.outputTokenIds = outputTokenIds.data_ptr(); + params.outputNumTokens = outputNumTokens.data_ptr(); + params.batchSize = static_cast(batchSize); + params.numDraftTokens = static_cast(numDraft); + params.vocabSize = static_cast(vocabSize); + auto const err = tensorrt_llm::kernels::minimaxM31::invokeChainSpecSamplingMc(params, static_cast(ctasPerRow), + workspace.data_ptr(), at::cuda::getCurrentCUDAStream(draftProbs.get_device())); + TORCH_CHECK( + err == cudaSuccess, "minimax_m31_chain_speculative_sampling_mc launch failed: ", cudaGetErrorString(err)); + return {outputTokenIds, outputNumTokens}; +} + +//! Workspace bytes per request of minimax_m31_chain_speculative_sampling_mc. +int64_t minimaxM31ChainSpecSamplingMcWsBytes() +{ + return static_cast(tensorrt_llm::kernels::minimaxM31::chainSpecSamplingMcWorkspaceBytesPerRow()); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_chain_speculative_sampling_mc(Tensor draft_probs, Tensor draft_token_ids, Tensor target_probs, " + "Tensor seed, Tensor offset, Tensor(a!) workspace, int ctas_per_row) -> (Tensor, Tensor)"); + // No tensor arguments: a catch-all kernel (as moe_a2a_get_timeout), callable without a device. + m.def("minimax_m31_chain_speculative_sampling_mc_ws_bytes() -> int", + &tensorrt_llm::torch_ext::minimaxM31ChainSpecSamplingMcWsBytes); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_chain_speculative_sampling_mc", &tensorrt_llm::torch_ext::minimaxM31ChainSpecSamplingMc); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Sm103CtxNormMxfp8Op.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Sm103CtxNormMxfp8Op.cpp new file mode 100644 index 000000000000..580623a89393 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Sm103CtxNormMxfp8Op.cpp @@ -0,0 +1,100 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/ctxResidualRmsNormMxfp8Sm103.h" + +#include +#include +#include + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! MiniMax-M3.1 sm_103 targets (target-local copy of the MiniMax-M3 SM107 CTX op; PDL is a caller argument). +//! Residual add + RMSNorm of an all-reduce output (in place: inter becomes inter + residual), with the MXFP8 +//! quantization of the normalized rows in the same pass. Returns (norm_out, mxfp8, sf, fp4, fp4_sf) where mxfp8 / sf +//! match mxfp8_quantize(norm_out, swizzled=True) bit for bit (mxfp8 [rows, hidden] Float8_e4m3fn, a 1-D UINT8 scale +//! tensor of the swizzled 128x4 layout). With fp4GlobalScale, fp4 / fp4_sf match +//! fp4_quantize(norm_out, fp4GlobalScale, 16, sf_use_ue8m0=False, is_sf_swizzled=False) bit for bit (fp4 [rows, +//! hidden / 2] UINT8, a 1-D UINT8 linear scale tensor of rows * hidden / 16); otherwise both are empty. norm_out is +//! empty when writeNorm is false. +std::tuple minimaxM31CtxResidualRmsNormMxfp8( + torch::Tensor& inter, torch::Tensor const& residual, torch::Tensor const& weight, double eps, bool writeNorm, + std::optional const& fp4GlobalScale, bool enablePdl) +{ + TORCH_CHECK(inter.is_cuda() && inter.is_contiguous() && inter.scalar_type() == at::kBFloat16 && inter.dim() >= 2); + TORCH_CHECK( + residual.is_contiguous() && residual.scalar_type() == at::kBFloat16 && residual.sizes() == inter.sizes()); + int64_t const hidden = inter.size(-1); + int64_t const rows = inter.numel() / hidden; + TORCH_CHECK(weight.is_contiguous() && weight.scalar_type() == at::kBFloat16 && weight.numel() == hidden); + TORCH_CHECK(hidden % 256 == 0 && hidden / 8 <= 1024, "hidden must be a multiple of 256 and at most 8192"); + at::cuda::CUDAGuard const guard(inter.device()); + auto const* deviceProperties = at::cuda::getCurrentDeviceProperties(); + TORCH_CHECK(deviceProperties->major == 10 && deviceProperties->minor == 3, + "minimax_m31_ctx_residual_rmsnorm_mxfp8 requires SM103"); + auto const opts = inter.options(); + torch::Tensor normOut = writeNorm ? torch::empty_like(inter) : torch::empty({0}, opts); + torch::Tensor qOut = torch::empty(inter.sizes(), opts.dtype(at::ScalarType::Float8_e4m3fn)); + int64_t const sfSize = (rows + 127) / 128 * 128 * ((hidden / 32 + 3) / 4 * 4); + torch::Tensor sfOut = torch::empty({sfSize}, opts.dtype(at::ScalarType::Byte)); + bool const fp4 = fp4GlobalScale.has_value(); + if (fp4) + { + TORCH_CHECK( + fp4GlobalScale->is_cuda() && fp4GlobalScale->scalar_type() == at::kFloat && fp4GlobalScale->numel() == 1); + } + std::vector fp4Shape(inter.sizes().begin(), inter.sizes().end()); + fp4Shape.back() = hidden / 2; + torch::Tensor fp4Out = fp4 ? torch::empty(fp4Shape, opts.dtype(at::ScalarType::Byte)) : torch::empty({0}, opts); + torch::Tensor fp4SfOut + = fp4 ? torch::empty({rows * (hidden / 16)}, opts.dtype(at::ScalarType::Byte)) : torch::empty({0}, opts); + tensorrt_llm::kernels::minimaxM31Sm103::invokeCtxResidualRmsNormMxfp8( + reinterpret_cast<__nv_bfloat16*>(inter.data_ptr()), reinterpret_cast<__nv_bfloat16 const*>(residual.data_ptr()), + reinterpret_cast<__nv_bfloat16 const*>(weight.data_ptr()), + writeNorm ? reinterpret_cast<__nv_bfloat16*>(normOut.data_ptr()) : nullptr, + reinterpret_cast(qOut.data_ptr()), reinterpret_cast(sfOut.data_ptr()), + fp4 ? reinterpret_cast(fp4Out.data_ptr()) : nullptr, + fp4 ? reinterpret_cast(fp4SfOut.data_ptr()) : nullptr, + fp4 ? fp4GlobalScale->data_ptr() : nullptr, static_cast(rows), static_cast(hidden), + static_cast(eps), enablePdl, at::cuda::getCurrentCUDAStream(inter.get_device())); + return {normOut, qOut, sfOut, fp4Out, fp4SfOut}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_ctx_residual_rmsnorm_mxfp8(Tensor(a!) inter, Tensor residual, Tensor weight, float eps, " + "bool write_norm, Tensor? fp4_global_scale=None, bool enable_pdl=False) -> (Tensor, Tensor, Tensor, Tensor, " + "Tensor)"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_ctx_residual_rmsnorm_mxfp8", &tensorrt_llm::torch_ext::minimaxM31CtxResidualRmsNormMxfp8); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Sm103SelectPrefillTileOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Sm103SelectPrefillTileOp.cpp new file mode 100644 index 000000000000..6242fe6ce896 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Sm103SelectPrefillTileOp.cpp @@ -0,0 +1,86 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3.1 GB300 (sm_103) target-local copy of the #20629 prefill tile select +// (cpp/tensorrt_llm/kernels/minimaxM3/selectPrefillTile.* @ cf8a77e816); namespace and op name are M3.1's. + +#include "tensorrt_llm/kernels/minimaxM31/selectPrefillTileSm103.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! Prefill top-16 block select over per-KV-head scores with unit query stride, into a caller-owned table. +//! +//! @param scores Float32 CUDA tensor [num_kv_heads, num_blocks, total_q] with stride(2) == 1. +//! @param nValidBlocks Contiguous int32 CUDA tensor with at least total_q elements. +//! @param out Int32 CUDA tensor of logical shape [total_q, num_kv_heads, 16] with unit stride in the last dim (a +//! head-major backing is a permuted view). Every entry is written. +void minimaxM31SelectPrefillTile(torch::Tensor const& scores, torch::Tensor const& nValidBlocks, torch::Tensor& out, + int64_t initBlocks, int64_t localBlocks) +{ + TORCH_CHECK(scores.is_cuda() && scores.scalar_type() == torch::kFloat32 && scores.dim() == 3, + "minimax_m31_select_prefill_tile: scores must be a 3-D float32 CUDA tensor [num_kv_heads, num_blocks, " + "total_q]"); + TORCH_CHECK(scores.stride(2) == 1, "minimax_m31_select_prefill_tile: scores must have unit query stride"); + TORCH_CHECK(nValidBlocks.is_cuda() && nValidBlocks.scalar_type() == torch::kInt32 && nValidBlocks.is_contiguous() + && nValidBlocks.numel() >= scores.size(2), + "minimax_m31_select_prefill_tile: n_valid_blocks must be contiguous int32 CUDA with at least total_q elements"); + int64_t const numKvHeads = scores.size(0); + int64_t const numBlocks = scores.size(1); + int64_t const totalQ = scores.size(2); + TORCH_CHECK(numBlocks <= tensorrt_llm::kernels::minimaxM31Sm103::kMinimaxM3SelectPrefillTileMaxBlocks, + "minimax_m31_select_prefill_tile supports at most ", + tensorrt_llm::kernels::minimaxM31Sm103::kMinimaxM3SelectPrefillTileMaxBlocks, " blocks, got ", numBlocks); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kInt32 && out.dim() == 3 && out.size(0) == totalQ + && out.size(1) == numKvHeads && out.size(2) == 16 && out.stride(2) == 1, + "minimax_m31_select_prefill_tile: out must be int32 [total_q, num_kv_heads, 16] with unit last-dim stride"); + + c10::cuda::CUDAGuard const deviceGuard{scores.device()}; + auto const stream = at::cuda::getCurrentCUDAStream(scores.get_device()); + cudaError_t const err + = tensorrt_llm::kernels::minimaxM31Sm103::invokeMinimaxM3SelectPrefillTile(scores.data_ptr(), + scores.stride(0), scores.stride(1), nValidBlocks.data_ptr(), out.data_ptr(), + out.stride(0), out.stride(1), static_cast(numKvHeads), static_cast(numBlocks), + static_cast(totalQ), static_cast(initBlocks), static_cast(localBlocks), stream); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "minimax_m31_select_prefill_tile launch failed: ", cudaGetErrorString(err)); + } +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_select_prefill_tile(Tensor scores, Tensor n_valid_blocks, Tensor(a!) out, int init_blocks, " + "int local_blocks) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_select_prefill_tile", &tensorrt_llm::torch_ext::minimaxM31SelectPrefillTile); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SoftmaxMcOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SoftmaxMcOp.cpp new file mode 100644 index 000000000000..6d64fbadc8a4 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SoftmaxMcOp.cpp @@ -0,0 +1,95 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/softmaxMcSm103.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! ST-01-SM's multi-CTA softmax (see kernels/minimaxM31/softmaxMcSm103.h): logits fp32 [rows, d] -> out fp32 +//! [rows, d], bitwise flashinfer.sampling.softmax(logits, temperature) on FlashInfer's Path A shapes (rows <= 128, +//! d >= 24576). temperature: an fp32 [rows] tensor, or None for the scalar temperature_val. partials: a CUDA buffer +//! of at least rows x ceil(d / 8192) float2, overwritten by every call; write_ctas: the CTAs per row of the write pass. +void minimaxM31SoftmaxMc(torch::Tensor const& logits, std::optional const& temperature, + double temperatureVal, torch::Tensor& partials, torch::Tensor& out, int64_t writeCtas) +{ + TORCH_CHECK( + logits.is_cuda() && logits.scalar_type() == torch::kFloat32 && logits.dim() == 2 && logits.is_contiguous(), + "minimax_m31_softmax_mc: logits must be a contiguous 2-D float32 CUDA tensor"); + int64_t const rows = logits.size(0); + int64_t const d = logits.size(1); + TORCH_CHECK(d % 4 == 0 && reinterpret_cast(logits.data_ptr()) % 16 == 0, + "minimax_m31_softmax_mc: logits must be 16-byte aligned with a width divisible by 4"); + TORCH_CHECK(rows <= 128 && d >= 24576, + "minimax_m31_softmax_mc: FlashInfer's Path A shapes only (rows <= 128, width >= 24576)"); + TORCH_CHECK(rows * d <= std::numeric_limits::max(), "minimax_m31_softmax_mc: logits is too large"); + float const* temperatureArr = nullptr; + if (temperature.has_value()) + { + auto const& t = temperature.value(); + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kFloat32 && t.dim() == 1 && t.is_contiguous() + && t.size(0) == rows && t.get_device() == logits.get_device(), + "minimax_m31_softmax_mc: temperature must be a contiguous float32 [rows] tensor on logits' device"); + temperatureArr = t.data_ptr(); + } + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kFloat32 && out.dim() == 2 && out.is_contiguous() + && out.size(0) == rows && out.size(1) == d && reinterpret_cast(out.data_ptr()) % 16 == 0 + && out.get_device() == logits.get_device(), + "minimax_m31_softmax_mc: out must be a contiguous, 16-byte aligned float32 [rows, d] on logits' device"); + TORCH_CHECK(writeCtas >= 1 && writeCtas <= 65535, "minimax_m31_softmax_mc: write_ctas must be in [1, 65535]"); + auto const partialsBytes = static_cast( + tensorrt_llm::kernels::minimaxM31::softmaxMcPartialsBytes(static_cast(rows), static_cast(d))); + TORCH_CHECK(partials.is_cuda() && partials.is_contiguous() && partials.get_device() == logits.get_device() + && reinterpret_cast(partials.data_ptr()) % 8 == 0 + && partials.numel() * partials.element_size() >= partialsBytes, + "minimax_m31_softmax_mc: partials must be a contiguous, 8-byte aligned CUDA buffer of at least ", partialsBytes, + " bytes on logits' device"); + if (rows == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{logits.device()}; + auto const err = tensorrt_llm::kernels::minimaxM31::invokeSoftmaxMc(logits.data_ptr(), out.data_ptr(), + temperatureArr, static_cast(temperatureVal), static_cast(rows), static_cast(d), + static_cast(writeCtas), partials.data_ptr(), at::cuda::getCurrentCUDAStream(logits.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m31_softmax_mc launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_softmax_mc(Tensor logits, Tensor? temperature, float temperature_val, Tensor(a!) partials, " + "Tensor(b!) out, int write_ctas) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_softmax_mc", &tensorrt_llm::torch_ext::minimaxM31SoftmaxMc); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetOp.cpp new file mode 100644 index 000000000000..bea1b0bbc95f --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetOp.cpp @@ -0,0 +1,98 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/toppRenormDetSm103.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! ST-01's det top-p renorm (see kernels/minimaxM31/toppRenormDetSm103.h): probs fp32 [rows, d] and top_p fp32 +//! [rows] -> out fp32 [rows, d], bitwise FlashInfer's top_p_renorm_probs(is_deterministic=True) on rows with +//! top_p < 1, keep-all on rows with top_p >= 1. workspace: a zero-initialized byte buffer of at least +//! rows x minimax_m31_topp_renorm_det_ws_bytes(), reused across calls; ctas_per_row divides 1024 and +//! rows x ctas_per_row must not exceed the SM count when ctas_per_row > 1. +void minimaxM31ToppRenormDet(torch::Tensor const& probs, torch::Tensor const& topP, torch::Tensor& workspace, + torch::Tensor& out, int64_t ctasPerRow) +{ + TORCH_CHECK(probs.is_cuda() && probs.scalar_type() == torch::kFloat32 && probs.dim() == 2 && probs.is_contiguous(), + "minimax_m31_topp_renorm_det: probs must be a contiguous 2-D float32 CUDA tensor"); + int64_t const rows = probs.size(0); + int64_t const d = probs.size(1); + TORCH_CHECK(d % 4 == 0 && reinterpret_cast(probs.data_ptr()) % 16 == 0, + "minimax_m31_topp_renorm_det: probs must be 16-byte aligned with a width divisible by 4"); + TORCH_CHECK(d <= std::numeric_limits::max() && rows <= 65535, + "minimax_m31_topp_renorm_det: probs is too large (rows <= 65535, width < 2^31)"); + TORCH_CHECK(topP.is_cuda() && topP.scalar_type() == torch::kFloat32 && topP.dim() == 1 && topP.is_contiguous() + && topP.size(0) == rows && topP.get_device() == probs.get_device(), + "minimax_m31_topp_renorm_det: top_p must be a contiguous float32 [rows] tensor on probs' device"); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kFloat32 && out.dim() == 2 && out.is_contiguous() + && out.size(0) == rows && out.size(1) == d && reinterpret_cast(out.data_ptr()) % 16 == 0 + && out.get_device() == probs.get_device(), + "minimax_m31_topp_renorm_det: out must be a contiguous, 16-byte aligned float32 [rows, d] on probs' device"); + TORCH_CHECK(ctasPerRow >= 1 && ctasPerRow <= 1024 && 1024 % ctasPerRow == 0, + "minimax_m31_topp_renorm_det: ctas_per_row must divide 1024"); + auto const wsBytesPerRow + = static_cast(tensorrt_llm::kernels::minimaxM31::toppRenormDetWorkspaceBytesPerRow()); + TORCH_CHECK(workspace.is_cuda() && workspace.is_contiguous() && workspace.get_device() == probs.get_device() + && reinterpret_cast(workspace.data_ptr()) % 16 == 0 + && workspace.numel() * workspace.element_size() >= wsBytesPerRow * rows, + "minimax_m31_topp_renorm_det: workspace must be a contiguous, 16-byte aligned CUDA buffer of at least ", + wsBytesPerRow * rows, " bytes on probs' device"); + if (rows == 0 || d == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{probs.device()}; + auto const err + = tensorrt_llm::kernels::minimaxM31::invokeToppRenormDet(probs.data_ptr(), out.data_ptr(), + topP.data_ptr(), static_cast(rows), static_cast(d), static_cast(ctasPerRow), + workspace.data_ptr(), nullptr, at::cuda::getCurrentCUDAStream(probs.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m31_topp_renorm_det launch failed: ", cudaGetErrorString(err)); +} + +//! Workspace bytes per row of minimax_m31_topp_renorm_det. +int64_t minimaxM31ToppRenormDetWsBytes() +{ + return static_cast(tensorrt_llm::kernels::minimaxM31::toppRenormDetWorkspaceBytesPerRow()); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_topp_renorm_det(Tensor probs, Tensor top_p, Tensor(a!) workspace, Tensor(b!) out, " + "int ctas_per_row) -> ()"); + // No tensor arguments: a catch-all kernel (as moe_a2a_get_timeout), callable without a device. + m.def("minimax_m31_topp_renorm_det_ws_bytes() -> int", &tensorrt_llm::torch_ext::minimaxM31ToppRenormDetWsBytes); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_topp_renorm_det", &tensorrt_llm::torch_ext::minimaxM31ToppRenormDet); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetV3Op.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetV3Op.cpp new file mode 100644 index 000000000000..d125c4b79c0e --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetV3Op.cpp @@ -0,0 +1,98 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! TOPP_DET v3 (see kernels/minimaxM31/toppRenormDetV3Sm103.h): minimax_m31_topp_renorm_det's renorm, bitwise its +//! output, with the redundant per-CTA selection and an optional programmatic dependent launch. workspace: a +//! zero-initialized byte buffer of at least rows x minimax_m31_topp_renorm_det_v3_ws_bytes(), reused across calls; +//! ctas_per_row divides 1024 and rows x ctas_per_row must not exceed the SM count when ctas_per_row > 1. +void minimaxM31ToppRenormDetV3(torch::Tensor const& probs, torch::Tensor const& topP, torch::Tensor& workspace, + torch::Tensor& out, int64_t ctasPerRow, bool pdl, bool candidateList) +{ + TORCH_CHECK(probs.is_cuda() && probs.scalar_type() == torch::kFloat32 && probs.dim() == 2 && probs.is_contiguous(), + "minimax_m31_topp_renorm_det_v3: probs must be a contiguous 2-D float32 CUDA tensor"); + int64_t const rows = probs.size(0); + int64_t const d = probs.size(1); + TORCH_CHECK(d % 4 == 0 && reinterpret_cast(probs.data_ptr()) % 16 == 0, + "minimax_m31_topp_renorm_det_v3: probs must be 16-byte aligned with a width divisible by 4"); + TORCH_CHECK(d <= std::numeric_limits::max() && rows <= 65535, + "minimax_m31_topp_renorm_det_v3: probs is too large (rows <= 65535, width < 2^31)"); + TORCH_CHECK(topP.is_cuda() && topP.scalar_type() == torch::kFloat32 && topP.dim() == 1 && topP.is_contiguous() + && topP.size(0) == rows && topP.get_device() == probs.get_device(), + "minimax_m31_topp_renorm_det_v3: top_p must be a contiguous float32 [rows] tensor on probs' device"); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kFloat32 && out.dim() == 2 && out.is_contiguous() + && out.size(0) == rows && out.size(1) == d && reinterpret_cast(out.data_ptr()) % 16 == 0 + && out.get_device() == probs.get_device(), + "minimax_m31_topp_renorm_det_v3: out must be a contiguous, 16-byte aligned float32 [rows, d] on probs' device"); + TORCH_CHECK(ctasPerRow >= 1 && ctasPerRow <= 1024 && 1024 % ctasPerRow == 0, + "minimax_m31_topp_renorm_det_v3: ctas_per_row must divide 1024"); + auto const wsBytesPerRow + = static_cast(tensorrt_llm::kernels::minimaxM31::toppRenormDetV3WorkspaceBytesPerRow()); + TORCH_CHECK(workspace.is_cuda() && workspace.is_contiguous() && workspace.get_device() == probs.get_device() + && reinterpret_cast(workspace.data_ptr()) % 16 == 0 + && workspace.numel() * workspace.element_size() >= wsBytesPerRow * rows, + "minimax_m31_topp_renorm_det_v3: workspace must be a contiguous, 16-byte aligned CUDA buffer of at least ", + wsBytesPerRow * rows, " bytes on probs' device"); + if (rows == 0 || d == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{probs.device()}; + auto const err + = tensorrt_llm::kernels::minimaxM31::invokeToppRenormDetV3(probs.data_ptr(), out.data_ptr(), + topP.data_ptr(), static_cast(rows), static_cast(d), static_cast(ctasPerRow), + workspace.data_ptr(), nullptr, pdl, candidateList, at::cuda::getCurrentCUDAStream(probs.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m31_topp_renorm_det_v3 launch failed: ", cudaGetErrorString(err)); +} + +//! Workspace bytes per row of minimax_m31_topp_renorm_det_v3. +int64_t minimaxM31ToppRenormDetV3WsBytes() +{ + return static_cast(tensorrt_llm::kernels::minimaxM31::toppRenormDetV3WorkspaceBytesPerRow()); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_topp_renorm_det_v3(Tensor probs, Tensor top_p, Tensor(a!) workspace, Tensor(b!) out, " + "int ctas_per_row, bool pdl, bool candidate_list) -> ()"); + // No tensor arguments: a catch-all kernel (as moe_a2a_get_timeout), callable without a device. + m.def( + "minimax_m31_topp_renorm_det_v3_ws_bytes() -> int", &tensorrt_llm::torch_ext::minimaxM31ToppRenormDetV3WsBytes); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_topp_renorm_det_v3", &tensorrt_llm::torch_ext::minimaxM31ToppRenormDetV3); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Tp4SparseDecodeLcOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Tp4SparseDecodeLcOp.cpp new file mode 100644 index 000000000000..5d188c43c702 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Tp4SparseDecodeLcOp.cpp @@ -0,0 +1,144 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/sparseDecodeLcTp4.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kHeads = 16; +constexpr int64_t kHeadDim = 128; +constexpr int64_t kTopK = 16; + +void checkBytes(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, "minimax_m31_tp4_sparse_decode_lc: ", name, + " must be a one-byte CUDA view (uint8 / e4m3 / packed e2m1)"); +} + +} // namespace + +//! One-launch MiniMax-M3.1 TP4 long-context sparse decode over the selected NVFP4 blocks (16 query heads on one KV +//! head, head_dim 128, page 128, top-k 16): the M3.1 target's copy of minimax_m3_sparse_decode_lc with two exact +//! options, bit-identical to it: btsmem (block-table row staged in shared memory before the grid wait) and sync3 +//! (softmax row max / sum read by every thread from the per-warp partials). +//! The compatibility one_div argument must be false. +//! +//! @param q E4M3 bytes [T, 16, 128], unit stride in the last dim. +//! @param k, v Packed E2M1 bytes [pages, 1, 128, 64] (the per-layer HND views). +//! @param kBlockScale, vBlockScale E4M3 bytes [pages, 1, 128, 8]: K token-major, V in 4x4 token-quad order. +//! @param kGlobalScale, vGlobalScale One-element float32 per-tensor dequant scales. +//! @param topk Int32 [T, 16] ascending selected block ids with -1 tail (kv head 0 of the head-major table). +//! @param out BF16 [T, 2048], written. q8 / sf, when given, receive the MXFP8 copy and its swizzled UE8M0 scales. +void minimaxM31Tp4SparseDecodeLc(torch::Tensor const& q, torch::Tensor const& k, torch::Tensor const& v, + torch::Tensor const& kBlockScale, torch::Tensor const& vBlockScale, torch::Tensor const& kGlobalScale, + torch::Tensor const& vGlobalScale, torch::Tensor const& topk, torch::Tensor const& blockTable, + torch::Tensor const& seqLens, torch::Tensor& out, std::optional q8, std::optional sf, + int64_t numSfKTiles, int64_t decodeQueryLen, double smScale, bool linearSoftmax, bool usePdl, bool earlyTrigger, + bool btsmem, bool sync3, bool oneDiv) +{ + TORCH_CHECK(!oneDiv, "one_div is excluded from this exact handoff"); + TORCH_CHECK(earlyTrigger, "minimax_m31_tp4_sparse_decode_lc is instantiated with early_trigger=True only"); + TORCH_CHECK(q.is_cuda(), "minimax_m31_tp4_sparse_decode_lc: q must be a CUDA tensor"); + c10::cuda::CUDAGuard const deviceGuard{q.device()}; + TORCH_CHECK(at::cuda::getCurrentDeviceProperties()->major >= 10, + "minimax_m31_tp4_sparse_decode_lc requires an SM100-family GPU"); + checkBytes(q, "q"); + checkBytes(k, "k"); + checkBytes(v, "v"); + checkBytes(kBlockScale, "k_block_scale"); + checkBytes(vBlockScale, "v_block_scale"); + TORCH_CHECK(q.dim() == 3 && q.size(1) == kHeads && q.size(2) == kHeadDim && q.stride(2) == 1, + "minimax_m31_tp4_sparse_decode_lc: q must be [T, 16, 128] with unit last-dim stride"); + TORCH_CHECK(kGlobalScale.scalar_type() == torch::kFloat32 && vGlobalScale.scalar_type() == torch::kFloat32, + "minimax_m31_tp4_sparse_decode_lc: global scales must be float32"); + TORCH_CHECK(topk.scalar_type() == torch::kInt32 && topk.dim() == 2 && topk.size(1) == kTopK, + "minimax_m31_tp4_sparse_decode_lc: topk must be int32 [T, 16]"); + TORCH_CHECK(blockTable.scalar_type() == torch::kInt32 && seqLens.scalar_type() == torch::kInt32, + "minimax_m31_tp4_sparse_decode_lc: block_table and seq_lens must be int32"); + TORCH_CHECK(out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.size(0) == q.size(0) + && out.size(1) == kHeads * kHeadDim && out.stride(1) == 1, + "minimax_m31_tp4_sparse_decode_lc: out must be bfloat16 [T, 2048] with unit last-dim stride"); + TORCH_CHECK(q8.has_value() == sf.has_value(), "minimax_m31_tp4_sparse_decode_lc: pass both q8 and sf, or neither"); + bool const mxfp8 = q8.has_value(); + if (mxfp8) + { + checkBytes(*q8, "q8"); + checkBytes(*sf, "sf"); + } + + tensorrt_llm::kernels::MinimaxM31Tp4SparseDecodeLcParams prm{}; + prm.q = reinterpret_cast(q.data_ptr()); + prm.q_tok = q.stride(0); + prm.q_head = q.stride(1); + prm.k = reinterpret_cast(k.data_ptr()); + prm.v = reinterpret_cast(v.data_ptr()); + prm.k_blk = k.stride(0); + prm.v_blk = v.stride(0); + prm.ksc = reinterpret_cast(kBlockScale.data_ptr()); + prm.vsc = reinterpret_cast(vBlockScale.data_ptr()); + prm.ksc_blk = kBlockScale.stride(0); + prm.vsc_blk = vBlockScale.stride(0); + prm.kgs = kGlobalScale.data_ptr(); + prm.vgs = vGlobalScale.data_ptr(); + prm.topk = topk.data_ptr(); + prm.t_tok = topk.stride(0); + prm.t_k = topk.stride(1); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.seq_lens = seqLens.data_ptr(); + prm.dql = static_cast(decodeQueryLen); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(out.data_ptr()); + prm.out_stride = out.stride(0); + prm.q8 = mxfp8 ? reinterpret_cast(q8->data_ptr()) : nullptr; + prm.sf = mxfp8 ? reinterpret_cast(sf->data_ptr()) : nullptr; + prm.num_sf_k_tiles = static_cast(numSfKTiles); + + auto const stream = at::cuda::getCurrentCUDAStream(q.get_device()); + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM31Tp4SparseDecodeLc( + prm, static_cast(q.size(0)), linearSoftmax, mxfp8, earlyTrigger, btsmem, sync3, usePdl, stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m31_tp4_sparse_decode_lc launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_tp4_sparse_decode_lc(Tensor q, Tensor k, Tensor v, Tensor k_block_scale, Tensor v_block_scale, " + "Tensor k_global_scale, Tensor v_global_scale, Tensor topk, Tensor block_table, Tensor seq_lens, " + "Tensor(a!) out, Tensor(b!)? q8, Tensor(c!)? sf, int num_sf_k_tiles, int decode_query_len, float sm_scale, " + "bool linear_softmax, bool use_pdl, bool early_trigger, bool btsmem, bool sync3, bool one_div=False) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_tp4_sparse_decode_lc", &tensorrt_llm::torch_ext::minimaxM31Tp4SparseDecodeLc); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Tp8SparseDecodeLc8Op.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Tp8SparseDecodeLc8Op.cpp new file mode 100644 index 000000000000..0290b536fa29 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31Tp8SparseDecodeLc8Op.cpp @@ -0,0 +1,143 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/sparseDecodeLc8Tp8.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kHeads = 8; +constexpr int64_t kHeadDim = 128; +constexpr int64_t kTopK = 16; + +void checkBytes(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, "minimax_m31_tp8_sparse_decode_lc8: ", name, + " must be a one-byte CUDA view (uint8 / e4m3 / packed e2m1)"); +} + +} // namespace + +//! One-launch MiniMax-M3.1 TP8 long-context sparse decode over the selected NVFP4 blocks (8 query heads on one KV +//! head, head_dim 128, page 128, top-k 16): the M3.1 target's copy of the M3 LC8 op, with the SYNC3 softmax (rows +//! bit-identical to minimax_m3_tp8_sparse_decode_lc8). Dependents are released right after the grid dependency wait. +//! +//! @param q E4M3 bytes [T, 8, 128], unit stride in the last dim. +//! @param k, v Packed E2M1 bytes [pages, 1, 128, 64] (the per-layer HND views). +//! @param kBlockScale, vBlockScale E4M3 bytes [pages, 1, 128, 8]: K token-major, V in 4x4 token-quad order. +//! @param kGlobalScale, vGlobalScale One-element float32 per-tensor dequant scales. +//! @param topk Int32 [T, 16] ascending selected block ids with -1 tail (kv head 0 of the head-major table). +//! @param out BF16 [T, 1024], written. q8 / sf, when given, receive the MXFP8 copy and its swizzled UE8M0 scales. +//! @param prewait Stage seq_lens, q and the request's block-table row before the grid dependency wait (same output). +//! @param oneDiv Compatibility argument; must be false. +void minimaxM31Tp8SparseDecodeLc8(torch::Tensor const& q, torch::Tensor const& k, torch::Tensor const& v, + torch::Tensor const& kBlockScale, torch::Tensor const& vBlockScale, torch::Tensor const& kGlobalScale, + torch::Tensor const& vGlobalScale, torch::Tensor const& topk, torch::Tensor const& blockTable, + torch::Tensor const& seqLens, torch::Tensor& out, std::optional q8, std::optional sf, + int64_t numSfKTiles, int64_t decodeQueryLen, double smScale, bool linearSoftmax, bool usePdl, bool prewait, + bool oneDiv) +{ + TORCH_CHECK(!oneDiv, "one_div is excluded from this exact handoff"); + TORCH_CHECK(q.is_cuda(), "minimax_m31_tp8_sparse_decode_lc8: q must be a CUDA tensor"); + c10::cuda::CUDAGuard const deviceGuard{q.device()}; + TORCH_CHECK(at::cuda::getCurrentDeviceProperties()->major >= 10, + "minimax_m31_tp8_sparse_decode_lc8 requires an SM100-family GPU"); + checkBytes(q, "q"); + checkBytes(k, "k"); + checkBytes(v, "v"); + checkBytes(kBlockScale, "k_block_scale"); + checkBytes(vBlockScale, "v_block_scale"); + TORCH_CHECK(q.dim() == 3 && q.size(1) == kHeads && q.size(2) == kHeadDim && q.stride(2) == 1, + "minimax_m31_tp8_sparse_decode_lc8: q must be [T, 8, 128] with unit last-dim stride"); + TORCH_CHECK(kGlobalScale.scalar_type() == torch::kFloat32 && vGlobalScale.scalar_type() == torch::kFloat32, + "minimax_m31_tp8_sparse_decode_lc8: global scales must be float32"); + TORCH_CHECK(topk.scalar_type() == torch::kInt32 && topk.dim() == 2 && topk.size(1) == kTopK, + "minimax_m31_tp8_sparse_decode_lc8: topk must be int32 [T, 16]"); + TORCH_CHECK(blockTable.scalar_type() == torch::kInt32 && seqLens.scalar_type() == torch::kInt32, + "minimax_m31_tp8_sparse_decode_lc8: block_table and seq_lens must be int32"); + TORCH_CHECK(out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.size(0) == q.size(0) + && out.size(1) == kHeads * kHeadDim && out.stride(1) == 1, + "minimax_m31_tp8_sparse_decode_lc8: out must be bfloat16 [T, 1024] with unit last-dim stride"); + TORCH_CHECK(q8.has_value() == sf.has_value(), "minimax_m31_tp8_sparse_decode_lc8: pass both q8 and sf, or neither"); + bool const mxfp8 = q8.has_value(); + if (mxfp8) + { + checkBytes(*q8, "q8"); + checkBytes(*sf, "sf"); + } + + tensorrt_llm::kernels::MinimaxM31Tp8SparseDecodeLc8Params prm{}; + prm.q = reinterpret_cast(q.data_ptr()); + prm.q_tok = q.stride(0); + prm.q_head = q.stride(1); + prm.k = reinterpret_cast(k.data_ptr()); + prm.v = reinterpret_cast(v.data_ptr()); + prm.k_blk = k.stride(0); + prm.v_blk = v.stride(0); + prm.ksc = reinterpret_cast(kBlockScale.data_ptr()); + prm.vsc = reinterpret_cast(vBlockScale.data_ptr()); + prm.ksc_blk = kBlockScale.stride(0); + prm.vsc_blk = vBlockScale.stride(0); + prm.kgs = kGlobalScale.data_ptr(); + prm.vgs = vGlobalScale.data_ptr(); + prm.topk = topk.data_ptr(); + prm.t_tok = topk.stride(0); + prm.t_k = topk.stride(1); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.seq_lens = seqLens.data_ptr(); + prm.dql = static_cast(decodeQueryLen); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(out.data_ptr()); + prm.out_stride = out.stride(0); + prm.q8 = mxfp8 ? reinterpret_cast(q8->data_ptr()) : nullptr; + prm.sf = mxfp8 ? reinterpret_cast(sf->data_ptr()) : nullptr; + prm.num_sf_k_tiles = static_cast(numSfKTiles); + + auto const stream = at::cuda::getCurrentCUDAStream(q.get_device()); + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM31Tp8SparseDecodeLc8( + prm, static_cast(q.size(0)), linearSoftmax, mxfp8, prewait, usePdl, stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m31_tp8_sparse_decode_lc8 launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_tp8_sparse_decode_lc8(Tensor q, Tensor k, Tensor v, Tensor k_block_scale, Tensor v_block_scale, " + "Tensor k_global_scale, Tensor v_global_scale, Tensor topk, Tensor block_table, Tensor seq_lens, " + "Tensor(a!) out, Tensor(b!)? q8, Tensor(c!)? sf, int num_sf_k_tiles, int decode_query_len, float sm_scale, " + "bool linear_softmax, bool use_pdl, bool prewait, bool one_div=False) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_tp8_sparse_decode_lc8", &tensorrt_llm::torch_ext::minimaxM31Tp8SparseDecodeLc8); +} diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/_router_index.py b/tensorrt_llm/_torch/_experimental/modeling_v2/_router_index.py index cdb288f7220f..d54443b4ed74 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/_router_index.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/_router_index.py @@ -37,7 +37,7 @@ import enum import os from importlib import import_module -from typing import TYPE_CHECKING, Optional, Tuple +from typing import TYPE_CHECKING, Dict, Optional, Tuple from tensorrt_llm.logger import logger @@ -49,6 +49,7 @@ # architectures[0] -> routing module, relative to this package. MODELING_V2_ROUTERS = { + "MiniMaxM3SparseForConditionalGeneration": "models.minimax_m3.routing", "GptOssForCausalLM": "models.gpt_oss.routing", "DeepseekV3ForCausalLM": "models.deepseek_v3.routing", } @@ -101,6 +102,62 @@ class NoTarget(Exception): """ +_RESOLVED_MODES: Dict[type, ModelingV2Mode] = {} + + +def resolved_mode(target: type) -> ModelingV2Mode: + """The mode ``modeling_v2_resolve`` returned ``target`` under; ``auto`` for a class built without it. + + A target reads it at construction: under ``require`` its in-target dispatch + raises where it would otherwise fall back to a built-in path inside the + shapes it claims. + """ + return _RESOLVED_MODES.get(target, ModelingV2Mode.AUTO) + + +def tp_reaches_across_nodes(mapping) -> bool: + """Whether an attention-TP allreduce over ``mapping`` can reach every rank + through the workspaces the default targets build. + + Inside one node the IPC workspace (``comm/allreduce``) serves it. Across + nodes the targets take the MNNVL workspace (``comm/mnnvl_allreduce``), which + needs one NVLink domain over those nodes, as on GB200 and GB300 trays. This + is the check the built-in ``AllReduce`` makes before it takes MNNVL. Where it + fails, the built-in model falls back to NCCL, which the targets do not + implement, so a routing module refuses and ``auto`` keeps the built-in model. + Attention DP reduces nothing in its norm slots, and its allgather and + reduce-scatter run over NCCL. + """ + if mapping.enable_attention_dp or mapping.tp_size == 1 or not mapping.is_multi_node(): + return True + import torch + + from tensorrt_llm._torch.distributed.ops import MNNVLAllReduce + + # bf16 is the activation dtype of both default targets. + return MNNVLAllReduce.is_mnnvl(mapping, torch.bfloat16) + + +def _kv_cache_quant_algo(config: "ModelConfig", llm_args: "TorchLlmArgs"): + """The KV-cache algorithm the engine will run with, which routing runs too early to read. + + The loader resolves the target before `validate_and_set_kv_cache_quant` + applies `kv_cache_config.dtype` to the config, so this applies the same + rule here: ``auto`` keeps the checkpoint's algorithm, ``fp8`` and + ``nvfp4`` replace it. Any other dtype comes back as the string given, so a + route refuses it by name: ``fp8_ds_mla`` is FP8 to the loader but also + selects a packed sparse-MLA cache layout, and an unknown dtype is one the + loader raises on later. + """ + from tensorrt_llm.quantization.mode import QuantAlgo + + dtype = llm_args.kv_cache_config.dtype + if dtype == "auto": + algo = config.quant_config.kv_cache_quant_algo + return None if algo is None else QuantAlgo(algo) + return {"fp8": QuantAlgo.FP8, "nvfp4": QuantAlgo.NVFP4}.get(dtype, dtype) + + def routing_module(arch: str): """Import and return the routing module for ``arch``, or None.""" name = MODELING_V2_ROUTERS.get(arch) @@ -143,6 +200,17 @@ def modeling_v2_resolve(config: "ModelConfig", llm_args: "TorchLlmArgs") -> Opti raise NoTarget( f"no routing module for this architecture; routed architectures: {known}" ) + if arch == "MiniMaxM3SparseForConditionalGeneration": + target = routing.route( + pretrained_config, + config.mapping, + _device_compute_capability(), + config.spec_config, + _kv_cache_quant_algo(config, llm_args), + ) + if target is not None: + _RESOLVED_MODES[target] = mode + return target return routing.route( pretrained_config, config.mapping, _device_compute_capability(), config.spec_config ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/attention/fused_qk_norm_rope.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/attention/fused_qk_norm_rope.py new file mode 100644 index 000000000000..97bfeb2cf715 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/attention/fused_qk_norm_rope.py @@ -0,0 +1,360 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-head QK RMS norm followed by RoPE, in place on a packed qkv buffer.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* + +from .._op import Arch, Cell, OpWrapper + + +class FusedQkNormRope(OpWrapper): + """In place on `qkv`: RMS-norm the q and k heads, then rotate them. + Returns `qkv` -- the same buffer, mutated and handed back so a call site + can show what it touched without allocating. + + The v heads are not touched. `qkv` is one `[num_tokens, (Hq + Hk + Hv) * + head_dim]` buffer, which is the layout the fused projection already + produces, so nothing is split or copied. The norm is `rmsnorm(x) * w`, or + the Gemma form `rmsnorm(x) * (1 + w)` with `use_gemma`; it covers the whole + head, the rotation only its first `rotary_dim` dims. With `use_mrope`, + `position_ids` is `[3, T]` and rotary pair `j` takes its position from row 1 + when `j % 3 == 1 and j < 3 * mrope_section1`, from row 2 when + `j % 3 == 2 and j < 3 * mrope_section2`, from row 0 otherwise -- the + interleaved mRoPE layout. The caller owns the QKV projection before the + call and the KV-cache append and attention after it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # gpt-oss-120b's default target asks for the RoPE half only: + # `is_qk_norm=False`, neox pairing, and YaRN-blended frequencies. 64 query + # heads, 8 key and 8 value, head_dim 64, all of which the cells carry + # because the op reads them as scalars rather than deriving them. Under + # attention TP each rank sees its own share of the heads. + _GPT_OSS = dict( + num_heads_q=64, + num_heads_k=8, + num_heads_v=8, + head_dim=64, + rotary_dim=64, + base=150000.0, + is_neox=True, + is_qk_norm=False, + # The checkpoint's YaRN correction range, from beta_fast=32 / + # beta_slow=1 over its original 4096-token context. + factor=32.0, + low=8.0, + high=18.0, + attention_factor=1.0, + ) + + # qwen4_exp (Qwen3.8-Flash-Next) runs Gemma QK norm and an unscaled rope + # on the first 64 dims (32 pairs), base 1e7, at two geometries. Its + # mrope_section is [11, 11, 10], passed as sections 11 / 11. Positions are + # drawn uniformly below `max_position` (a test axis, not an op argument): + # the kernel's fast sincos loses phase as `position * freq` grows, so the + # position range is part of what a cell certifies. + _QWEN_ROPE = dict( + rotary_dim=64, + base=1e7, + is_neox=True, + is_qk_norm=True, + use_gemma=True, + max_position=32768, + ) + _QWEN_MROPE = dict(use_mrope=True, mrope_section1=11, mrope_section2=11, **_QWEN_ROPE) + # Main attention (trunk and MTP): 24 query heads, 2 kv, head_dim 256. + _QWEN_MAIN = dict(num_heads_q=24, num_heads_k=2, num_heads_v=2, head_dim=256, **_QWEN_MROPE) + # The QSA index plane's query rope: 4 index heads plus the one k and one v + # slot of its scratch layout, over index_head_dim 128 (== 2 * rotary_dim, + # so the partial factor is not reapplied). + _QWEN_INDEX_HEADS = dict(num_heads_q=4, num_heads_k=1, num_heads_v=1, head_dim=128) + _QWEN_INDEX_MROPE = dict(**_QWEN_INDEX_HEADS, **_QWEN_MROPE) + _QWEN_INDEX_PLAIN = dict(**_QWEN_INDEX_HEADS, **_QWEN_ROPE) + + CELLS: tuple[Cell, ...] = ( + Cell(why="decode: one token", spec=dict(tokens=1, **_GPT_OSS)), + Cell(why="prefill-shaped batch", spec=dict(tokens=2048, **_GPT_OSS)), + Cell( + why="a token count that is not a multiple of any tile", + spec=dict(tokens=37, **_GPT_OSS), + ), + Cell( + why="default target at attention TP8: 8q/1k/1v heads per rank", + spec=dict( + tokens=64, **{**_GPT_OSS, "num_heads_q": 8, "num_heads_k": 1, "num_heads_v": 1} + ), + ), + Cell( + why="qwen4_exp main rope, decode: one token, Gemma norm and interleaved mRoPE " + "with three distinct position rows", + spec=dict(tokens=1, **_QWEN_MAIN), + ), + Cell( + why="qwen4_exp main rope, 19 tokens: the size its wiring and saturation tests run at", + spec=dict(tokens=19, **_QWEN_MAIN), + ), + Cell( + why="qwen4_exp main rope, 8192-token prefill: the Gemma one-ulp tail, under " + "GEMMA_PREFILL_ATOL", + spec=dict(tokens=8192, gemma_prefill_band=True, **_QWEN_MAIN), + ), + Cell( + why="qwen4_exp index rope (mRoPE), 1 token: one k and one v head, the " + "narrowest layout; 5 warps of an 8-warp block", + spec=dict(tokens=1, **_QWEN_INDEX_MROPE), + ), + Cell( + why="qwen4_exp index rope (mRoPE), 2 tokens: the second token's heads " + "straddle a block boundary", + spec=dict(tokens=2, **_QWEN_INDEX_MROPE), + ), + Cell( + why="qwen4_exp index rope (mRoPE), 5 tokens: the last block runs one warp", + spec=dict(tokens=5, **_QWEN_INDEX_MROPE), + ), + Cell( + why="qwen4_exp index rope (mRoPE), 17 tokens: one past a power of two", + spec=dict(tokens=17, **_QWEN_INDEX_MROPE), + ), + Cell( + why="qwen4_exp index rope (mRoPE), 19 tokens: the size its wiring and " + "saturation tests run at", + spec=dict(tokens=19, **_QWEN_INDEX_MROPE), + ), + Cell( + why="qwen4_exp index rope (mRoPE), 33 tokens: one past a power of two", + spec=dict(tokens=33, **_QWEN_INDEX_MROPE), + ), + Cell( + why="qwen4_exp index rope (mRoPE), 65 tokens: one past a power of two", + spec=dict(tokens=65, **_QWEN_INDEX_MROPE), + ), + Cell( + why="qwen4_exp index rope (mRoPE), 8192-token prefill: the index site sees the " + "whole packed batch; under GEMMA_PREFILL_ATOL", + spec=dict(tokens=8192, gemma_prefill_band=True, **_QWEN_INDEX_MROPE), + ), + Cell( + why="qwen4_exp index rope (plain), decode: the form the index plane runs " + "today, row 0 alone as a flat [T] position_ids", + spec=dict(tokens=1, **_QWEN_INDEX_PLAIN), + ), + Cell( + why="qwen4_exp index rope (plain), 19 tokens: an odd count across blocks", + spec=dict(tokens=19, **_QWEN_INDEX_PLAIN), + ), + Cell( + why="qwen4_exp index rope (plain), 8192-token prefill: under GEMMA_PREFILL_ATOL", + spec=dict(tokens=8192, gemma_prefill_band=True, **_QWEN_INDEX_PLAIN), + ), + ) + + note = """ + Certified by caller: gpt-oss runs `is_qk_norm=False` with YaRN; qwen4_exp + runs `is_qk_norm=True, use_gemma=True`, unscaled, with and without + interleaved mRoPE. Uncertified because no shipped target reaches them: + `is_qk_norm=True` with `use_gemma=False` (the plain-weight norm the op's + name advertises), `is_neox=False`, YaRN together with the norm or with + mRoPE. A target that wants one adds the cell rather than assuming the path + works because the argument exists. + + mRoPE preconditions. `position_ids` is a contiguous int32 `[3, T]`; the op + rejects a strided view loudly, so an engine's `[3, 1, max_num_tokens]` + position buffer has to be copied, not only reshaped. A HuggingFace + `mrope_section` `[s0, s1, s2]` summing to `rotary_dim / 2` is passed as + `mrope_section1 = s1`, `mrope_section2 = s2`; `s0` is implied. Only the + interleaved layout exists here -- the contiguous-section variant is a + different operator, measured ~500x out when wired in its place. Sections + saturate: at 32 pairs `mrope_section2` 10, 11 and 12 are bit-identical (the + largest `j % 3 == 2` below 32 is 29), while `mrope_section1` 10 moves pair + 31 back to row 0. Three equal rows reduce bitwise to the plain + `use_mrope=False` call on that row, so a text batch cannot see a wrong + split; an image span is where one shows. + + Metadata consumed: none. + """ + + def __call__( + self, + qkv: torch.Tensor, + num_heads_q: int, + num_heads_k: int, + num_heads_v: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + is_neox: bool, + position_ids: torch.Tensor, + factor: float = 1.0, + low: float = 0.0, + high: float = 0.0, + attention_factor: float = 1.0, + is_qk_norm: bool = True, + use_gemma: bool = False, + use_mrope: bool = False, + mrope_section1: int = 0, + mrope_section2: int = 0, + ) -> torch.Tensor: + """Writes `qkv` in place and returns it -- the op destroys the q/k + heads' prior contents, so a caller that wants them has to have kept + its own copy beforehand.""" + torch.ops.trtllm.fused_qk_norm_rope( + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + return qkv + + def reference( + self, + qkv: torch.Tensor, + num_heads_q: int, + num_heads_k: int, + num_heads_v: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + is_neox: bool, + position_ids: torch.Tensor, + factor: float = 1.0, + low: float = 0.0, + high: float = 0.0, + attention_factor: float = 1.0, + is_qk_norm: bool = True, + use_gemma: bool = False, + use_mrope: bool = False, + mrope_section1: int = 0, + mrope_section2: int = 0, + ) -> torch.Tensor: + """The buffer the op writes back, fp32 throughout. + + Returns rather than mutates so a cell can drive both sides from the + same input; the in-place write is what `__call__` does. + """ + num_heads = num_heads_q + num_heads_k + num_heads_v + x = qkv.float().view(-1, num_heads, head_dim).clone() + half = rotary_dim // 2 + inv_freq = self._inv_freq(rotary_dim, base, factor, low, high, qkv.device) + + if use_mrope: + angle = position_ids.float()[:, :, None] * inv_freq # [3, T, half] + cos, sin = angle.cos(), angle.sin() + + def pick(c: torch.Tensor) -> torch.Tensor: + out = c[0].clone() + out[:, 1 : mrope_section1 * 3 : 3] = c[1][:, 1 : mrope_section1 * 3 : 3] + out[:, 2 : mrope_section2 * 3 : 3] = c[2][:, 2 : mrope_section2 * 3 : 3] + return out + + cos, sin = pick(cos), pick(sin) + else: + angle = position_ids.float()[:, None] * inv_freq # [T, half] + cos, sin = angle.cos(), angle.sin() + cos = cos[:, None, :] * attention_factor + sin = sin[:, None, :] * attention_factor + + for start, count, weight in ( + (0, num_heads_q, q_weight.float()), + (num_heads_q, num_heads_k, k_weight.float()), + ): + h = x[:, start : start + count, :] + if is_qk_norm: + h = h * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + eps) + h = h * ((1.0 + weight) if use_gemma else weight) + r = h[..., :rotary_dim] + if is_neox: + x1, x2 = r[..., :half], r[..., half:] + rotated = torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1) + else: + x1, x2 = r[..., ::2], r[..., 1::2] + rotated = torch.stack([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1).flatten( + -2 + ) + x[:, start : start + count, :] = torch.cat([rotated, h[..., rotary_dim:]], -1) + return x.view(qkv.shape[0], -1).to(qkv.dtype) + + @staticmethod + def _inv_freq( + rotary_dim: int, base: float, factor: float, low: float, high: float, device + ) -> torch.Tensor: + """YaRN-blended inverse frequencies; plain RoPE when factor == 1.""" + j = torch.arange(rotary_dim // 2, dtype=torch.float32, device=device) + pos_freqs = base ** (2.0 * j / rotary_dim) + inv_extrapolation = 1.0 / pos_freqs + inv_interpolation = 1.0 / (factor * pos_freqs) + if high == low: + high = high + 0.001 # the kernel guards the ramp singularity the same way + ramp = ((j - low) / (high - low)).clamp(0.0, 1.0) + extrapolation = 1.0 - ramp + return inv_interpolation * (1.0 - extrapolation) + inv_extrapolation * extrapolation + + def is_valid( + self, + qkv: torch.Tensor, + num_heads_q: int, + num_heads_k: int, + num_heads_v: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + is_neox: bool, + position_ids: torch.Tensor, + *unused_args: object, + **unused_kwargs: object, + ) -> None: + # `eps`, `q_weight`, `k_weight`, `base` and `is_neox` are named though + # nothing here reads them: they mirror `__call__`, and gpt-oss passes + # them by keyword. + # + # The op reads the head counts as scalars and addresses qkv from + # data_ptr(); a width that disagrees with them is read as whatever the + # counts say and the heads silently come out shifted. + expected = (num_heads_q + num_heads_k + num_heads_v) * head_dim + assert qkv.shape[-1] == expected, ( + f"qkv's last dim must be (Hq + Hk + Hv) * head_dim = {expected}; got " + f"{qkv.shape[-1]}. The op trusts the counts and reads the buffer flat" + ) + assert qkv.is_contiguous(), "qkv must be contiguous; the op writes it through data_ptr()" + assert rotary_dim <= head_dim, ( + f"rotary_dim ({rotary_dim}) exceeds head_dim ({head_dim}); the rotation " + "would read past the end of each head" + ) + assert rotary_dim % 2 == 0, f"rotary_dim must be even to pair; got {rotary_dim}" + assert position_ids.shape[-1] == qkv.shape[0], ( + f"one position per token: {position_ids.shape[-1]} ids for {qkv.shape[0]} tokens" + ) + + +fused_qk_norm_rope = FusedQkNormRope() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast.py new file mode 100644 index 000000000000..24d10fae21bc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast.py @@ -0,0 +1,540 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Vocab-dim all-gather of bf16 logit shards into fp32 full-vocabulary rows, in one launch. + +``out = torch.cat(all_gather(shard), -1).float()``: a column-parallel LM head's verify logits as the sampler takes +them, without the NCCL all-gather, the concat copy and the cast. Every rank pushes its shard through a multicast +NVLink workspace and raises per-CTA ready flags (the epoch of the call), so no bit pattern is reserved as a +sentinel and the result is the stock path's bit for bit, -0.0 and NaN payloads included. + +``allgather_f32`` (MC-ALLGATHER): the same gather for fp32 shards, copied as is, ``out = torch.cat(all_gather(shard), +-1)``: the drafter's sharded draft logits as the rejection sampler takes them (fp32 [B, K, S] -> [B, K, world * S]), +without the NCCL all-gather and the concat copy. It shares the workspace (a slot holds MAX_ROWS / 2 fp32 rows). + +Compiled on the first call per shape, which must run outside CUDA-graph capture (``precompile``). The workspace is +collective on its first use: every rank of the TP group calls ``workspace_args`` at the same point. +""" + +from __future__ import annotations + +import importlib +import os +import threading +from typing import Callable, Dict, Optional, Sequence, Tuple + +import torch + +from .._op import Arch, Cell, OpWrapper + +MAX_ROWS = 64 # capacity: fixed: the workspace's rows per call (K3 verify rows at batch 16) +MAX_GRID = 152 # capacity: fixed: the workspace's flag columns per rank +# capacity: fixed: fp32 rows per call (an fp32 row is twice a bf16 row's bytes) +MAX_ROWS_FP32 = MAX_ROWS // 2 +# Launched without programmatic dependent launch: a plain launch waits for all prior stream work, so the previous +# call has completed whatever kernels run between two calls (the kernel's ordering condition holds by construction). +# It still releases its own dependents early. ``pdl=1`` on a call launches it with PDL (measurement only). +LAUNCH_PDL = False + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} +_workspaces: Dict[object, dict] = {} +_modules: Dict[str, object] = {} + + +def _kernel_module(): + """The CuTe DSL kernel beside this file, imported on first use (it pulls in cutlass).""" + mod = _modules.get("kernel") + if mod is None: + mod = _modules["kernel"] = importlib.import_module(".allgather_cast_kernel", __package__) + return mod + + +# CTAs by world size and rows: the fastest bitwise grid measured at each K3 verify row count on GB300 (graph-us at +# 96 / 128 / 152 CTAs). TP8 over two trays (RESULT_M31_N3A1K_MICROBENCH + its addendum-4 sweep): 4 rows 12.54 / 12.83 / +# 12.88, 8 rows 18.79 / 16.55 / 16.56, 16 rows 25.27 / 23.25 / 24.10, the other row counts fastest at 152. TP4 on one +# tray (RESULT_M31_N3A1K4_MICROBENCH): 4 rows 12.24 / 11.90 / 11.96, 8 rows 15.22 / 14.29 / 14.61, 16 rows 22.02 / +# 22.43 / 20.47, 32 rows 36.66 / 35.25 / 34.48 (96 is kept at 4 rows: within 0.5 us). Row counts in between take the +# next measured one's grid, rows past the table MAX_GRID; other world sizes take the TP8 table. A function of the +# world size, the rows and the device only, so every rank picks the same grid. +GRID_BY_ROWS = ((4, 96), (8, 128), (12, 152), (16, 128), (32, 152)) +GRID_BY_WORLD_ROWS = {4: ((4, 96), (8, 128), (32, 152)), 8: GRID_BY_ROWS} + + +def pick_grid(rows: int, device=None, world: int = 8) -> int: + """CTAs for a call of ``rows`` rows over ``world`` ranks (capped by the SM count and MAX_GRID).""" + sms = torch.cuda.get_device_properties( + device if device is not None else torch.cuda.current_device() + ).multi_processor_count + table = GRID_BY_WORLD_ROWS.get(world, GRID_BY_ROWS) + grid = next((g for bound, g in table if rows <= bound), MAX_GRID) + return min(MAX_GRID, sms, grid) + + +def workspace_args(mapping, shard_cols: int, group=None) -> dict: + """This TP group's workspace as the call's ``ws_*``, ``rank`` and ``world`` arguments. + + Three rotating buffers of [world][MAX_ROWS, shard_cols] bf16 and the [world][MAX_GRID] ready flags behind one + multicast mapping, plus the rank-local epoch words; built once per (mapping, shard_cols, group) and reused. + Collective on first use, outside CUDA-graph capture: every rank of the TP group calls it at the same point. + ``group`` overrides the communicator (a torch.distributed group when MPI is disabled). + """ + key = (mapping, shard_cols, id(group) if group is not None else None) + ws = _workspaces.get(key) + if ws is not None: + return ws["args"] + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "allgather_cast: the workspace must be allocated outside CUDA-graph capture" + ) + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + + kern = _kernel_module() + world = mapping.tp_size + comm = group if group is not None else _get_mnnvl_workspace_comm(mapping) + use_fabric_handle = ( + os.environ.get("TRTLLM_FORCE_MNNVL_AR", "0") == "1" or mapping.is_multi_node() + ) + error: Optional[Exception] = None + try: + words = kern.workspace_words(MAX_ROWS, shard_cols, world, MAX_GRID) + handle = _make_mnnvl_mcast_buffer(comm, words * 4, mapping, use_fabric_handle) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + with torch.inference_mode(): + uc.zero_() + flags = torch.zeros(kern.FLAG_WORDS, dtype=torch.int32, device=uc.device) + torch.cuda.synchronize() + args = dict(ws_uc=uc, ws_mc=mc, ws_flags=flags, rank=mapping.tp_rank, world=world) + ws = dict(handle=handle, comm=comm, args=args) + except Exception as exc: # noqa: BLE001 -- reported to every rank below, then re-raised + error = exc + # Also the barrier that keeps any rank from raising a peer's flag before the peer has zeroed it. + if not _mnnvl_workspace_all_succeeded(comm, error is None): + raise RuntimeError("allgather_cast workspace failed on at least one rank") from error + _workspaces[key] = ws + return ws["args"] + + +def _arg(t: torch.Tensor, align: int = 16): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align).mark_layout_dynamic(leading_dim=0) + + +def supports(shard: torch.Tensor, world: int) -> bool: + """Whether the kernel takes this shard: bf16 [rows <= MAX_ROWS, S] contiguous, S a multiple of 8, on CUDA.""" + return ( + shard.is_cuda + and shard.dim() == 2 + and shard.dtype == torch.bfloat16 + and shard.is_contiguous() + and 0 < shard.shape[0] <= MAX_ROWS + and shard.shape[1] % 8 == 0 + and shard.data_ptr() % 16 == 0 + and 1 < world <= 8 + ) + + +def supports_f32(shard: torch.Tensor, world: int) -> bool: + """Whether ``allgather_f32`` takes this shard: fp32 [rows <= MAX_ROWS_FP32, S] contiguous, S a multiple of 4.""" + return ( + shard.is_cuda + and shard.dim() == 2 + and shard.dtype == torch.float32 + and shard.is_contiguous() + and 0 < shard.shape[0] <= MAX_ROWS_FP32 + and shard.shape[1] % 4 == 0 + and shard.data_ptr() % 16 == 0 + and 1 < world <= 8 + ) + + +def _consts(rows: int, cols: int, world: int, grid: int, fp32: bool = False) -> tuple: + kern = _kernel_module() + return ( + grid, + rows, + cols, + world, + kern.slot_words(MAX_ROWS, cols), + kern.flag_base(MAX_ROWS, cols, world), + MAX_GRID, + bool(fp32), + ) + + +def _default_grid(rows: int, device, world: int, fp32: bool) -> int: + """An fp32 row moves a bf16 row's bytes twice: it takes the grid of twice the rows.""" + return pick_grid(2 * rows if fp32 else rows, device, world) + + +def _kernel_for( + shard, + out, + ws_uc, + ws_mc, + ws_flags, + rank: int, + world: int, + grid: int, + pdl: int, + fp32: bool = False, +): + """The compiled launcher of this call's shape (compiled here on first use, outside capture), its tensor + arguments and the stream.""" + import cuda.bindings.driver as cuda_driver + import cutlass.cute as cute + + kern = _kernel_module() + if not (supports_f32(shard, world) if fp32 else supports(shard, world)): + raise ValueError(f"allgather_cast: unsupported shard {tuple(shard.shape)} {shard.dtype}") + rows, cols = shard.shape + if grid <= 0: + grid = _default_grid(rows, shard.device, world, fp32) + if not 0 < grid <= MAX_GRID or ws_uc.numel() < kern.workspace_words( + MAX_ROWS, cols, world, MAX_GRID + ): + raise ValueError("allgather_cast: the call does not fit the workspace") + args = ( + _arg(shard.view(-1).view(torch.int32)), + _arg(out.view(-1).view(torch.int32)), + _arg(ws_uc.view(-1)), + _arg(ws_mc.view(-1)), + _arg(ws_flags.view(-1), align=4), + ) + use_pdl = LAUNCH_PDL if pdl < 0 else bool(pdl) + stream = cuda_driver.CUstream(torch.cuda.current_stream(shard.device).cuda_stream) + consts = _consts(rows, cols, world, grid, fp32) + key = consts + (use_pdl,) + fn = _compiled.get(key) + if fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "trtllm::allgather_cast must run once per shape outside CUDA-graph capture first" + ) + with _lock: + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute.compile( + kern.allgather_cast, *args, int(rank), *consts, use_pdl, stream + ) + return fn, args, stream + + +@torch.library.custom_op("trtllm::allgather_cast", mutates_args=()) +def _allgather_cast( + shard: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + world: int, + grid: int, + pdl: int, +) -> torch.Tensor: + out = torch.empty( + shard.shape[0], world * shard.shape[1], dtype=torch.float32, device=shard.device + ) + fn, args, stream = _kernel_for(shard, out, ws_uc, ws_mc, ws_flags, rank, world, grid, pdl) + fn(*args, int(rank), stream) + return out + + +@_allgather_cast.register_fake +def _(shard, ws_uc, ws_mc, ws_flags, rank, world, grid, pdl): + return shard.new_empty((shard.shape[0], world * shard.shape[1]), dtype=torch.float32) + + +@torch.library.custom_op("trtllm::allgather_f32", mutates_args=()) +def _allgather_f32( + shard: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + world: int, + grid: int, + pdl: int, +) -> torch.Tensor: + out = torch.empty( + shard.shape[0], world * shard.shape[1], dtype=torch.float32, device=shard.device + ) + fn, args, stream = _kernel_for( + shard, out, ws_uc, ws_mc, ws_flags, rank, world, grid, pdl, fp32=True + ) + fn(*args, int(rank), stream) + return out + + +@_allgather_f32.register_fake +def _(shard, ws_uc, ws_mc, ws_flags, rank, world, grid, pdl): + return shard.new_empty((shard.shape[0], world * shard.shape[1]), dtype=torch.float32) + + +def gather_last_dim_f32(logits: torch.Tensor, ws_uc, ws_mc, ws_flags, rank, world, grid=0, pdl=-1): + """``torch.cat(all_gather(logits), -1)`` for an fp32 [..., S] shard (the leading dims flattened into rows), or + None when ``allgather_f32`` does not take it (the caller then runs the stock all-gather; shapes decide, so every + rank declines alike).""" + lead = tuple(logits.shape[:-1]) + flat = logits.reshape(-1, logits.shape[-1]) + if not supports_f32(flat, world): + return None + out = torch.ops.trtllm.allgather_f32( + flat, ws_uc, ws_mc, ws_flags, int(rank), int(world), int(grid), int(pdl) + ) + return out.view(*lead, world * logits.shape[-1]) + + +def is_compiled( + rows: int, cols: int, world: int, grid: int = 0, device=None, fp32: bool = False +) -> bool: + """Whether the (rows, cols, world, grid) shape is compiled (for the caller's capture-time decision).""" + grid = grid if grid > 0 else _default_grid(rows, device, world, fp32) + return _consts(rows, cols, world, grid, fp32) + (LAUNCH_PDL,) in _compiled + + +def precompile( + mapping, + cols: int, + rows: Tuple[int, ...], + device, + group=None, + agree: Optional[Callable[[bool], bool]] = None, +) -> bool: + """Compile every row count in ``rows`` outside capture, then run each shape once (collective: every rank). + + The compiles launch nothing. ``agree`` (every rank's verdict, e.g. a TP all-reduce of success) is consulted + between the compiles and the first launch, so a compile that fails on one rank turns the item off on every rank + instead of leaving its peers polling for flags it never raises. Returns whether the shapes are ready.""" + ws = workspace_args(mapping, cols, group) + ok = True + try: + for r in rows: + shard = torch.zeros(r, cols, dtype=torch.bfloat16, device=device) + out = torch.empty(r, ws["world"] * cols, dtype=torch.float32, device=device) + _kernel_for( + shard, out, ws["ws_uc"], ws["ws_mc"], ws["ws_flags"], ws["rank"], ws["world"], 0, -1 + ) + except Exception as exc: # noqa: BLE001 -- reported to every rank through ``agree`` + from tensorrt_llm.logger import logger + + logger.warning(f"allgather_cast: compile failed on rank {ws['rank']}: {exc}") + ok = False + if agree is not None: + ok = agree(ok) + if not ok: + return False + for r in rows: + allgather_cast(torch.zeros(r, cols, dtype=torch.bfloat16, device=device), **ws) + torch.cuda.synchronize() + return True + + +def precompile_f32( + mapping, + cols: int, + rows: Tuple[int, ...], + device, + group=None, + agree: Optional[Callable[[bool], bool]] = None, +) -> bool: + """``precompile`` for ``allgather_f32`` (fp32 shards of ``cols`` columns, each row count in ``rows``).""" + ws = workspace_args(mapping, cols, group) + ok = True + try: + for r in rows: + shard = torch.zeros(r, cols, dtype=torch.float32, device=device) + out = torch.empty(r, ws["world"] * cols, dtype=torch.float32, device=device) + _kernel_for( + shard, out, ws["ws_uc"], ws["ws_mc"], ws["ws_flags"], ws["rank"], ws["world"], 0, -1, + fp32=True, + ) # fmt: skip + except Exception as exc: # noqa: BLE001 -- reported to every rank through ``agree`` + from tensorrt_llm.logger import logger + + logger.warning(f"allgather_f32: compile failed on rank {ws['rank']}: {exc}") + ok = False + if agree is not None: + ok = agree(ok) + if not ok: + return False + for r in rows: + allgather_f32(torch.zeros(r, cols, dtype=torch.float32, device=device), **ws) + torch.cuda.synchronize() + return True + + +class AllgatherCast(OpWrapper): + """fp32 [rows, world * S]: every rank's bf16 [rows, S] shard, in rank order along the last dim, widened to fp32. + + ``shard`` bf16 [rows, S] contiguous (rows <= MAX_ROWS, S a multiple of 8): this rank's vocabulary columns. + ``ws_*``, ``rank``, ``world``: ``workspace_args``'s. ``grid``: CTAs (0 picks). + ``pdl``: -1 launches as LAUNCH_PDL (off); 0 / 1 force it (measurement only). + + Every rank returns the same tensor. The call is a collective: every rank of the group calls it with the same + rows, S and grid, in the same order. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # MiniMax-M3.1's target LM head at TP8 over two GB300 trays: vocabulary 200064 (shard 25008), K3 verify rows + # B * (K + 1): K3 at batches 1..16, K7 at batches 1..8. + CELLS: tuple[Cell, ...] = ( + Cell( + why="M3.1 TP8, K3 verify at batch 1: 4 rows of the 25008-column shard", + spec=dict(world=8, rows=4, cols=25008), + ), + Cell( + why="M3.1 TP8, K7 verify at batch 1 (the headline c1): 8 rows", + spec=dict(world=8, rows=8, cols=25008), + ), + Cell( + why="M3.1 TP8, K7 verify at batch 2: 16 rows", + spec=dict(world=8, rows=16, cols=25008), + ), + Cell( + why="M3.1 TP8, K3 verify at batch 8: 32 rows", + spec=dict(world=8, rows=32, cols=25008), + ), + Cell( + why="M3.1 TP8, K3 verify at batch 16: 64 rows, the workspace's capacity", + spec=dict(world=8, rows=64, cols=25008), + ), + Cell( + why="a row count between the K3 batch sizes (odd vectors per CTA)", + spec=dict(world=8, rows=7, cols=25008), + ), + Cell( + why="one tray (TP4): the 50016-column shard", + spec=dict(world=4, rows=4, cols=50016), + ), + Cell( + why="one tray (TP4), K3 verify at batch 12: 48 rows", + spec=dict(world=4, rows=48, cols=50016), + ), + ) + + note = """ + Rows past MAX_ROWS (64) and shards that are not whole 16-byte vectors are declined (``supports``); the caller + takes the stock all-gather. Every rank must make the same decision, and does: it depends only on shapes. + """ + + def __call__( + self, + shard: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + world: int, + grid: int = 0, + pdl: int = -1, + ) -> torch.Tensor: + return torch.ops.trtllm.allgather_cast( + shard, ws_uc, ws_mc, ws_flags, int(rank), int(world), int(grid), int(pdl) + ) + + def reference(self, shards_in_rank_order: Sequence[torch.Tensor]) -> torch.Tensor: + """The stock path: the column-parallel head's all-gather, the vocab-dim concat and ``.float()``. + + The documented collective exception to mirroring ``__call__``: it takes every rank's shard in rank order. + """ + return torch.cat(list(shards_in_rank_order), dim=-1).float() + + def is_valid( + self, + shard: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + world: int, + grid: int = 0, + pdl: int = -1, + ) -> None: + assert supports(shard, world), (tuple(shard.shape), shard.dtype, shard.is_contiguous()) + assert 0 <= rank < world, (rank, world) + + +allgather_cast = AllgatherCast() + + +class AllgatherF32(OpWrapper): + """fp32 [rows, world * S]: every rank's fp32 [rows, S] shard, in rank order along the last dim, copied as is. + + ``shard`` fp32 [rows, S] contiguous (rows <= MAX_ROWS_FP32, S a multiple of 4): this rank's vocabulary columns. + ``ws_*``, ``rank``, ``world``: ``workspace_args``'s (the allgather_cast workspace of the same S). ``grid``: CTAs + (0 picks). ``pdl``: as ``allgather_cast``'s. A collective, as ``allgather_cast``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # MiniMax-M3.1's DSpark drafter at TP8 over two GB300 trays: the rejection sampler's draft logits, B * K rows of + # the 25008-column vocabulary shard. + CELLS: tuple[Cell, ...] = ( + Cell( + why="M3.1 TP8, K7 draft logits at batch 1 (the headline c1): 7 rows", + spec=dict(world=8, rows=7, cols=25008), + ), + Cell( + why="M3.1 TP8, K7 draft logits at batch 2: 14 rows", + spec=dict(world=8, rows=14, cols=25008), + ), + Cell( + why="M3.1 TP8, K3 draft logits at batch 8: 24 rows", + spec=dict(world=8, rows=24, cols=25008), + ), + Cell( + why="the fp32 capacity: 32 rows", + spec=dict(world=8, rows=32, cols=25008), + ), + Cell( + why="one tray (TP4): the 50016-column shard, K7 at batch 1", + spec=dict(world=4, rows=7, cols=50016), + ), + ) + + note = """ + Rows past MAX_ROWS_FP32 (32) and shards that are not whole 16-byte vectors are declined (``supports_f32``); the + caller takes the stock all-gather. Every rank declines alike: it depends only on shapes. + """ + + def __call__( + self, + shard: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + world: int, + grid: int = 0, + pdl: int = -1, + ) -> torch.Tensor: + return torch.ops.trtllm.allgather_f32( + shard, ws_uc, ws_mc, ws_flags, int(rank), int(world), int(grid), int(pdl) + ) + + def reference(self, shards_in_rank_order: Sequence[torch.Tensor]) -> torch.Tensor: + """The stock path: all_gather + the vocab-dim concat (the documented collective exception).""" + return torch.cat(list(shards_in_rank_order), dim=-1) + + def is_valid( + self, + shard: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + world: int, + grid: int = 0, + pdl: int = -1, + ) -> None: + assert supports_f32(shard, world), (tuple(shard.shape), shard.dtype, shard.is_contiguous()) + assert 0 <= rank < world, (rank, world) + + +allgather_f32 = AllgatherF32() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast_kernel.py new file mode 100644 index 000000000000..a77ad777283f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast_kernel.py @@ -0,0 +1,327 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# Vocab-dim all-gather of bf16 logit shards into fp32 full-vocabulary rows -- CTM (prims/cute) kernel +# ============================================================================= +# +# Every rank r holds shard_r, bf16 [rows, S] (its vocabulary columns [r S, r S + S)). Every rank writes +# out[i, r S + j] = float(shard_r[i, j]) for every rank r, row i < rows, column j < S +# i.e. torch.cat(all_gather(shard), -1).float(): the gather, the concat and the cast in one launch. The cast is a bit +# shift (bf16 is the high half of fp32), so out holds every rank's bits exactly: -0.0, NaN payloads and infinities +# included. +# +# Geometry: G CTAs of 256 threads. The shard is N = rows S / 8 16-byte vectors (S a multiple of 8); CTA c owns +# vectors [c N / G, (c + 1) N / G) of every rank's shard. CTA c pushes its vectors of this rank's shard into slot +# [buffer][rank] of every rank's workspace (multimem stores through the multicast mapping), then raises its ready +# flag [rank][c] in every rank's workspace (one multimem max, release, system scope). It converts its vectors of this +# rank's own shard straight from the input meanwhile, then polls the flags [r][c] of the other ranks in its own copy +# (acquire, system scope) and converts their vectors from its own buffer. +# +# fp32 mode (MC-ALLGATHER, the draft-logits gather): the shard is fp32 [rows, S] (S a multiple of 4), the 16-byte +# vectors hold 4 fp32, and every vector is stored as is (no widening): out = torch.cat(all_gather(shard), -1), bit for +# bit. A slot then holds max_rows / 2 rows. +# +# Flags carry the call's epoch, not the data: there is no empty-word sentinel, so no bit pattern of the logits is +# reserved. A flag only grows (multimem max of the epoch), so it needs no re-arming, and calls with different grids +# share it: a reader waits until flag >= its epoch. +# +# ORDERING: every call must start after the previous call on its rank completed. The op launches the kernel without +# programmatic dependent launch, so a launch waits for all prior work on the stream, whatever kernels run between two +# calls (a PDL launch would rely on every kernel in between executing griddepcontrol.wait). The kernel still releases +# its own dependents early. +# +# Buffers: 3 rotating buffers, call n using buffer n % 3. Rank r's call n + 3 reuses the buffer that rank q reads in +# call n only after r's call n + 2 completed, which waited for q's pushes of call n + 2, which q issues after its call +# n + 1 (and so n) completed. +# +# flags (rank-local int32): [0] epoch of this call, [1] CTAs arrived, [4] debug code, [5] debug detail. The last CTA to +# arrive advances the epoch. A flag poll that waits longer than POLL_TIMEOUT_NS (a peer that never raises the column, +# e.g. a grid that differs between ranks) writes the debug words and traps instead of hanging. +# ============================================================================= +"""CTM vocab-dim all-gather + concat + fp32 cast of bf16 logit shards over a multicast NVLink workspace, in one +launch.""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +THREADS = 256 +BUFFERS = 3 +FLAG_WORDS = 8 +POLL_TIMEOUT_NS = 30_000_000_000 # 30 s: far past any straggler, short of a hung job +DEBUG_POLL = 2 +VEC_ELEMS = 8 # bf16 per 16-byte vector +VEC_ELEMS_FP32 = 4 # fp32 per 16-byte vector (fp32 mode) +HI_MASK = -65536 # 0xFFFF0000 + + +def vectors(rows: int, shard: int) -> int: + """16-byte vectors of one rank's [rows, shard] bf16 slice.""" + return rows * shard // VEC_ELEMS + + +def slot_words(max_rows: int, shard: int) -> int: + """Int32 words of one rank's slot in one buffer.""" + return max_rows * shard // 2 + + +def workspace_words(max_rows: int, shard: int, world: int, max_grid: int) -> int: + """Int32 words of the multicast workspace: the buffers, then the [world][max_grid] ready flags.""" + return BUFFERS * world * slot_words(max_rows, shard) + world * max_grid + + +def flag_base(max_rows: int, shard: int, world: int) -> int: + return BUFFERS * world * slot_words(max_rows, shard) + + +@dsl_user_op +def _atomic_add_acq_rel(addr_i64, val, *, loc=None, ip=None): + """atom.acq_rel.gpu.global.add.u32, returning the old value.""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [addr_i64.ir_value(loc=loc, ip=ip), val.ir_value(loc=loc, ip=ip)], + "atom.acq_rel.gpu.global.add.u32 $0, [$1], $2;", "=r,l,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _multimem_st_v4(addr_i64, a, b, c, d, *, loc=None, ip=None): + """multimem.st.global.v4.f32: one 16-byte store into every rank's copy (the bits, as stored).""" + _llvm.inline_asm( + None, + [addr_i64.ir_value(loc=loc, ip=ip), a.ir_value(loc=loc, ip=ip), b.ir_value(loc=loc, ip=ip), + c.ir_value(loc=loc, ip=ip), d.ir_value(loc=loc, ip=ip)], + "multimem.st.global.v4.f32 [$0], {$1, $2, $3, $4};", "l,r,r,r,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _multimem_max_release_sys(addr_i64, val, *, loc=None, ip=None): + """multimem.red.release.sys.global.max.u32: raise every rank's copy of a flag to val, ordered after the CTA's + stores (the caller's barrier makes them the releasing thread's).""" + _llvm.inline_asm( + None, + [addr_i64.ir_value(loc=loc, ip=ip), val.ir_value(loc=loc, ip=ip)], + "multimem.red.release.sys.global.max.u32 [$0], $1;", "l,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _ld_acquire_sys(addr_i64, *, loc=None, ip=None): + """ld.acquire.sys.global.u32.""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [addr_i64.ir_value(loc=loc, ip=ip)], + "ld.acquire.sys.global.u32 $0, [$1];", "=r,l", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _globaltimer(*, loc=None, ip=None): + """%globaltimer (ns).""" + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), [], "mov.u64 $0, %globaltimer;", "=l", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _trap(*, loc=None, ip=None): + """fence.sc.sys (the debug words reach memory), then trap.""" + _llvm.inline_asm( + None, [], "fence.sc.sys;\n\ttrap;", "", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +def _store_fp32x8(out, word, got): + """The 8 bf16 of a 16-byte vector, widened to fp32 bit for bit, as two 16-byte stores at out word `word`.""" + lo = [] + for q in range(4): + w = cutlass.Int32(got[q]) + lo.append(cutlass.Int32(w << cutlass.Int32(16))) + lo.append(cutlass.Int32(w & cutlass.Int32(HI_MASK))) + out.store((lo[0], lo[1], lo[2], lo[3]), idx=word, alignment=16) + out.store((lo[4], lo[5], lo[6], lo[7]), idx=word + cutlass.Int32(4), alignment=16) + + +def _store_out(out, word, got, fp32: bool): + """One 16-byte input vector at out word `word`: 4 fp32 copied as is (fp32 mode), else 8 bf16 widened.""" + if cutlass.const_expr(fp32): + out.store( + ( + cutlass.Int32(got[0]), + cutlass.Int32(got[1]), + cutlass.Int32(got[2]), + cutlass.Int32(got[3]), + ), + idx=word, + alignment=16, + ) + else: + _store_fp32x8(out, word, got) + + +@cute.kernel +def allgather_cast_kernel( + shard: cutlass.Array, # int32 words of bf16 [rows, S]: this rank's shard + out: cutlass.Array, # int32 bits of fp32 [rows, world * S], out + buf_uc: cutlass.Array, # int32 words: this rank's workspace + buf_mc: cutlass.Array, # int32 words: its multicast mapping + flags: cutlass.Array, # int32 [FLAG_WORDS], rank-local + rank: cutlass.Int32, + grid: cutlass.Constexpr[int], + rows: cutlass.Constexpr[int], + shard_cols: cutlass.Constexpr[int], # S + world: cutlass.Constexpr[int], + slot: cutlass.Constexpr[int], # int32 words per (buffer, rank) slot + flag_off: cutlass.Constexpr[int], # first flag word of the workspace + max_grid: cutlass.Constexpr[int], # flag columns per rank + fp32: cutlass.Constexpr[bool], # fp32 shard copied as is (else bf16 widened to fp32) +): + ve = VEC_ELEMS_FP32 if fp32 else VEC_ELEMS + n_vec = rows * shard_cols // ve + row_vecs = shard_cols // ve + out_row = world * shard_cols + + tx, _, _ = cute.arch.thread_idx() + cta, _, _ = cute.arch.block_idx() + s_epoch = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + + # The shard is the previous kernel's output and the epoch the previous call's: both after the grid wait. + prims.griddepcontrol(prims.GridDepAction.WAIT) + if tx == 0: + s_epoch.store(flags.load(idx=0, is_volatile=True), idx=0) + prims.barrier_cta_sync(0) + epoch = s_epoch.load(idx=0) + buf_base = (epoch % cutlass.Int32(BUFFERS)) * cutlass.Int32(world * slot) + + lo = cta * cutlass.Int32(n_vec) // cutlass.Int32(grid) + hi = (cta + cutlass.Int32(1)) * cutlass.Int32(n_vec) // cutlass.Int32(grid) + + # Push this CTA's vectors of this rank's shard into every rank's slot [buffer][rank]. + mine = buf_base + rank * cutlass.Int32(slot) + v = lo + tx + while v < hi: + got = shard.load(idx=v * cutlass.Int32(4), vector_size=4, alignment=16) + _multimem_st_v4( + buf_mc.data_ptr(mine + v * cutlass.Int32(4)).toint(), + cutlass.Int32(got[0]), cutlass.Int32(got[1]), cutlass.Int32(got[2]), cutlass.Int32(got[3]), + ) # fmt: skip + v += cutlass.Int32(THREADS) + prims.barrier_cta_sync(0) + target = epoch + cutlass.Int32(1) + if tx == 0: + _multimem_max_release_sys( + buf_mc.data_ptr(cutlass.Int32(flag_off) + rank * cutlass.Int32(max_grid) + cta).toint(), + target, + ) + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + + # This rank's own columns, straight from the input while the peers' pushes are in flight. + v = lo + tx + while v < hi: + got = shard.load(idx=v * cutlass.Int32(4), vector_size=4, alignment=16) + row = v // cutlass.Int32(row_vecs) + col = v - row * cutlass.Int32(row_vecs) + _store_out( + out, + row * cutlass.Int32(out_row) + + rank * cutlass.Int32(shard_cols) + + col * cutlass.Int32(ve), + got, + fp32, + ) + v += cutlass.Int32(THREADS) + + # Thread r waits for rank r's flag [r][cta]; the barrier hands its acquire to the whole CTA. + if tx < cutlass.Int32(world): + if tx != rank: + addr = buf_uc.data_ptr( + cutlass.Int32(flag_off) + tx * cutlass.Int32(max_grid) + cta + ).toint() + start = _globaltimer() + while _ld_acquire_sys(addr) - target < cutlass.Int32(0): + if _globaltimer() - start > cutlass.Int64(POLL_TIMEOUT_NS): + flags.store(cutlass.Int32(DEBUG_POLL), idx=4) + flags.store(tx * cutlass.Int32(65536) + cta, idx=5) + _trap() + prims.barrier_cta_sync(0) + + for r in cutlass.range_constexpr(world): + if cutlass.Int32(r) != rank: + src = buf_base + cutlass.Int32(r * slot) + v = lo + tx + while v < hi: + got = buf_uc.load( + idx=src + v * cutlass.Int32(4), vector_size=4, alignment=16, is_volatile=True + ) + row = v // cutlass.Int32(row_vecs) + col = v - row * cutlass.Int32(row_vecs) + _store_out( + out, row * cutlass.Int32(out_row) + cutlass.Int32(r * shard_cols) + col * cutlass.Int32(ve), + got, fp32, + ) # fmt: skip + v += cutlass.Int32(THREADS) + + # Every CTA read the epoch before arriving; the last one advances it for the next call. + if tx == 0: + arrived = _atomic_add_acq_rel(flags.data_ptr(1).toint(), cutlass.Int32(1)) + if arrived == cutlass.Int32(grid - 1): + flags.store(cutlass.Int32(0), idx=1) + flags.store(target, idx=0) + + +@cute.jit +def allgather_cast( + shard: cute.Tensor, + out: cute.Tensor, + buf_uc: cute.Tensor, + buf_mc: cute.Tensor, + flags: cute.Tensor, + rank: cutlass.Int32, + grid: cutlass.Constexpr[int], + rows: cutlass.Constexpr[int], + shard_cols: cutlass.Constexpr[int], + world: cutlass.Constexpr[int], + slot: cutlass.Constexpr[int], + flag_off: cutlass.Constexpr[int], + max_grid: cutlass.Constexpr[int], + fp32: cutlass.Constexpr[bool], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + allgather_cast_kernel( + shard, out, buf_uc, buf_mc, flags, rank, grid, rows, shard_cols, world, slot, flag_off, max_grid, fp32, + ).launch( + grid=[grid, 1, 1], + block=[THREADS, 1, 1], + stream=stream, + use_pdl=use_pdl, + ) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/__init__.py new file mode 100644 index 000000000000..4cadfaf48c99 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fake (meta) impls of model-specific C++ torch ops, registered for torch.compile / fake-tensor tracing.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/m31_fake_ops.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/m31_fake_ops.py new file mode 100644 index 000000000000..96c98d1b1afd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/m31_fake_ops.py @@ -0,0 +1,161 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fake (meta) impls of the MiniMax-M3.1 C++ torch ops (cpp/tensorrt_llm/thop/minimaxM31/). + +The ops exist only on builds that carry them, either in the main library or in an overlay library the M3.1 targets load +on first use, so each fake is registered only once its op exists: ``register()`` runs at import and again after a +target loads an overlay library. It is idempotent. +""" + +import torch + +_REGISTERED = set() + + +def _ctx_residual_rmsnorm_mxfp8_fake( + inter, residual, weight, eps, write_norm, fp4_global_scale=None, enable_pdl=False +): + """minimaxM31Sm103CtxNormMxfp8Op.cpp: (norm_out, MXFP8 rows, swizzled scale bytes, NVFP4 rows, linear NVFP4 scale + bytes); inter is updated in place. Outputs that are off are empty 1-D tensors of inter's dtype.""" + hidden = inter.shape[-1] + rows = inter.numel() // hidden + empty = inter.new_empty((0,)) + norm_out = torch.empty_like(inter) if write_norm else empty + mxfp8 = torch.empty_like(inter, dtype=torch.float8_e4m3fn) + sf = inter.new_empty( + ((rows + 127) // 128 * 128 * ((hidden // 32 + 3) // 4 * 4),), dtype=torch.uint8 + ) + if fp4_global_scale is None: + return (norm_out, mxfp8, sf, empty, empty) + fp4 = inter.new_empty((*inter.shape[:-1], hidden // 2), dtype=torch.uint8) + fp4_sf = inter.new_empty((rows * (hidden // 16),), dtype=torch.uint8) + return (norm_out, mxfp8, sf, fp4, fp4_sf) + + +def _select_prefill_tile_fake(scores, n_valid_blocks, out, init_blocks, local_blocks): + """minimaxM31Sm103SelectPrefillTileOp.cpp: writes the block table into out; returns nothing.""" + return None + + +def _sparse_decode_lc_fake( + q, + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + out, + q8, + sf, + num_sf_k_tiles, + decode_query_len, + sm_scale, + linear_softmax, + use_pdl, + early_trigger, + btsmem, + sync3, + one_div=False, +): + """minimaxM31Tp4SparseDecodeLcOp.cpp (LC4; LongAttn a7154ae3fa): writes out (and q8 / sf when given) in place; + returns nothing.""" + return None + + +def _sparse_decode_lc8_fake( + q, + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + out, + q8, + sf, + num_sf_k_tiles, + decode_query_len, + sm_scale, + linear_softmax, + use_pdl, + prewait, + one_div=False, +): + """minimaxM31Tp8SparseDecodeLc8Op.cpp (LC8; LongAttn a7154ae3fa): writes out (and q8 / sf when given) in place; + returns nothing.""" + return None + + +def _topp_renorm_det_v3_fake(probs, top_p, workspace, out, ctas_per_row, pdl, candidate_list): + """m31_topp_det v3: as minimax_m31_topp_renorm_det (in place into out, workspace updated); returns nothing. + minimax_m31_topp_renorm_det_v3_ws_bytes takes no tensor and needs no fake.""" + return None + + +def _topp_renorm_det_fake(probs, top_p, workspace, out, ctas_per_row): + """m31_topp_det (ST-01): writes the renormed [rows, d] fp32 probs into out in place (and updates the persistent + workspace); returns nothing. minimax_m31_topp_renorm_det_ws_bytes takes no tensor and needs no fake.""" + return None + + +def _softmax_mc_fake(logits, temperature, temperature_val, partials, out, write_ctas): + """m31_softmax_mc (ST-01-SM): writes the [rows, d] fp32 probs into out in place (and overwrites the partials + workspace); returns nothing.""" + return None + + +def _chain_spec_sampling_mc_fake( + draft_probs, draft_token_ids, target_probs, seed, offset, workspace, ctas_per_row +): + """m31_chain_spec_mc (ST-01-REJ): (output_token_ids [batch, numDraft + 1], num_tokens [batch]), int32 as + trtllm::chain_speculative_sampling; updates the persistent workspace in place. + minimax_m31_chain_speculative_sampling_mc_ws_bytes takes no tensor and needs no fake.""" + batch_size, num_draft = draft_token_ids.shape + return ( + draft_token_ids.new_empty((batch_size, num_draft + 1)), + draft_token_ids.new_empty((batch_size,)), + ) + + +# op name -> fake impl (schemas as of img9, ONE_DIV included). +FAKES = { + "minimax_m31_ctx_residual_rmsnorm_mxfp8": _ctx_residual_rmsnorm_mxfp8_fake, + "minimax_m31_select_prefill_tile": _select_prefill_tile_fake, + "minimax_m31_tp4_sparse_decode_lc": _sparse_decode_lc_fake, + "minimax_m31_tp8_sparse_decode_lc8": _sparse_decode_lc8_fake, + "minimax_m31_topp_renorm_det": _topp_renorm_det_fake, + "minimax_m31_topp_renorm_det_v3": _topp_renorm_det_v3_fake, + "minimax_m31_softmax_mc": _softmax_mc_fake, + "minimax_m31_chain_speculative_sampling_mc": _chain_spec_sampling_mc_fake, +} + + +def _has_fake(qualname): + """True when the op already has a fake: a C++ Meta kernel or a torch.library fake (register_fake raises on both). + simple_registry is private torch API (checked on the image's torch 2.14); torch.library has no public query.""" + if torch._C._dispatch_has_kernel_for_dispatch_key(qualname, "Meta"): + return True + from torch._library.simple_registry import singleton + + return singleton.find(qualname).fake_impl.kernel is not None + + +def register(): + """Register the fake of every M3.1 op this process has loaded and that has none yet (another module may have + registered one first; that one is kept).""" + for name, fake in FAKES.items(): + if name in _REGISTERED or not hasattr(torch.ops.trtllm, name): + continue + if not _has_fake(f"trtllm::{name}"): + torch.library.register_fake(f"trtllm::{name}")(fake) + _REGISTERED.add(name) + + +register() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/moe/__init__.py new file mode 100644 index 000000000000..5e898b4e1168 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/moe/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mixture-of-experts entries.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/moe/fp4_block_scale_moe_runner.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/moe/fp4_block_scale_moe_runner.py new file mode 100644 index 000000000000..978decf58a8a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/moe/fp4_block_scale_moe_runner.py @@ -0,0 +1,450 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""NVFP4-weight / NVFP4-activation mixture-of-experts layer: caller-supplied +top-k + grouped FC1 GEMM + gated activation + NVFP4 requantization + grouped +FC2 GEMM + routing-weighted combine, in one trtllm-gen call.""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: NVFP4 block size: one e4m3 scale per 16 elements along K. +SF_VEC_SIZE = 16 + +#: Largest e2m1 magnitude, and the largest finite e4m3 magnitude. Their product +#: is the global scale a checkpoint's `input_scale` encodes. +E2M1_MAX = 6.0 +E4M3_MAX = 448.0 + +#: `act_type` for SwiGlu, the only activation this entry certifies. +ACT_TYPE_SWIGLU = 0 + +#: The widest unchunked call a stock engine issues: trtllm's default +#: `max_num_tokens`. A caller whose own token count exceeds it must chunk. +MAX_CERTIFIED_TOKENS = 8192 + +#: The 16 e2m1 code points in code order: sign bit 3, exponent bits 2:1, +#: mantissa bit 0. +_E2M1_VALUES = ( + 0.0, + 0.5, + 1.0, + 1.5, + 2.0, + 3.0, + 4.0, + 6.0, + -0.0, + -0.5, + -1.0, + -1.5, + -2.0, + -3.0, + -4.0, + -6.0, +) + +#: One expert stack's `(codes, block scales)`, unshuffled and unswizzled. +Operand = Tuple[torch.Tensor, torch.Tensor] + + +def _e2m1_rne(a: torch.Tensor) -> torch.Tensor: + """Round to nearest even on the e2m1 grid, saturating at +-6. + + The grid step is 0.5 below 2, 1 below 4 and 2 above -- the binade step of + e2m1's three exponents, the subnormal binade sharing the first normal one's. + `torch.round` is banker's rounding, which is the format's ties-to-even-code + rule: 0.25 -> 0, 0.75 -> 1, 1.25 -> 1, 1.75 -> 2, 2.5 -> 2, 3.5 -> 4, + 5.0 -> 4. + """ + step = torch.where(a.abs() < 2.0, 0.5, torch.where(a.abs() < 4.0, 1.0, 2.0)) + return torch.sign(a) * torch.clamp(torch.round(a.abs() / step) * step, max=E2M1_MAX) + + +def quantize_nvfp4(x: torch.Tensor, global_scale: float) -> torch.Tensor: + """NVFP4 quantize-dequantize of `x` under `global_scale`. + + Per 16 consecutive columns: `sf = e4m3(g * blockmax / 6)` and + `data = e2m1(x * g / sf)`, returning `data * sf` -- the value a downstream + block-scaled MMA multiplies, which is `g` times the reconstruction of `x`. + This models the FC1 epilogue, which the entry's test pins bit-exactly. + """ + rows, cols = x.shape + blocks = x.reshape(rows, cols // SF_VEC_SIZE, SF_VEC_SIZE) + sf = ( + (global_scale * blocks.abs().amax(dim=-1, keepdim=True) / E2M1_MAX) + .clamp(max=E4M3_MAX) + .to(torch.float8_e4m3fn) + .float() + ) + out_scale = torch.where(sf == 0, torch.zeros_like(sf), global_scale / sf) + return (_e2m1_rne(blocks * out_scale) * sf).reshape(rows, cols) + + +def dequantize_nvfp4(codes: torch.Tensor, block_scales: torch.Tensor) -> torch.Tensor: + """One expert's `[N, K]` fp32 weight from its codes and e4m3 block scales.""" + table = torch.tensor(_E2M1_VALUES, dtype=torch.float32, device=codes.device) + return table[codes.long()] * block_scales.float().repeat_interleave(SF_VEC_SIZE, dim=1) + + +class Fp4BlockScaleMoeRunner(OpWrapper): + """One NVFP4-weight MoE layer over NVFP4 (e2m1 + e4m3 block-scale) + activations, entered pre-routed. + + The caller owns top-k: `topk_ids` carries global expert ids and + `topk_weights` the combine weights, neither renormalized here. This rank + answers for `[local_expert_offset, local_expert_offset + local_num_experts)` + and contributes zero for every other id, so an expert-parallel split is N + calls whose outputs sum -- which is what the deepseek default target does + under moe_ep, one call per rank over its window; qwen4_exp makes one call + over its whole expert stack. + + With `do_finalize=True` the result is a one-element list holding the + combined `[T, H]` bf16 rows, or an empty tensor when `output` was given. + With `do_finalize=False` it is three tensors: per-slot expert rows, an + unwritten scale buffer, and the expanded-index -> permuted-row map. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # DeepSeek-R1-0528's routed geometry. The deepseek default target makes one + # call per rank over its moe_ep window at its moe_tp slice of the + # intermediate (dep4 with moe ep4: a 64-wide window at the full 2048; tp8: + # all 256 experts at 256), so the window and intermediate-slice cells are + # the shapes it runs and the full stack is the single-rank one. + _R1 = dict(hidden=7168, intermediate=2048, num_experts=256, top_k=8) + + # qwen4_exp's (Qwen3.8-Flash-Next) routed geometry: many small experts, a + # top_k past 8, an intermediate width of 640, all 512 experts local. Its + # router renormalizes the top-10 softmax weights to sum to 1 before the + # call (`routing="renormalized"`, a test axis rather than an op argument; + # the R1 cells draw uniform [0, 1) weights). Nothing is padded: + # H % 256, 2I % 128, H/16 % 4 and I/16 % 4 are all zero. + _QWEN4_EXP = dict( + hidden=2560, + intermediate=640, + num_experts=512, + top_k=10, + local_expert_offset=0, + local_num_experts=512, + routing="renormalized", + ) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="dep4's own call: the 64-wide expert window at offset 0, R1 geometry", + spec=dict(tokens=64, local_expert_offset=0, local_num_experts=64, **_R1), + ), + Cell( + why="a window at a non-zero offset -- the ids this rank must answer for move with it", + spec=dict(tokens=64, local_expert_offset=192, local_num_experts=64, **_R1), + ), + Cell( + why="the single-rank shape: all 256 experts local, and one token, the decode row count", + spec=dict(tokens=1, local_expert_offset=0, local_num_experts=256, **_R1), + ), + Cell( + why="deepseek: all 256 experts, intermediate split 2 ways (1024)", + spec=dict( + tokens=64, + local_expert_offset=0, + local_num_experts=256, + **dict(_R1, intermediate=1024), + ), + ), + Cell( + why="deepseek default at tp4: all 256 experts, intermediate split 4 ways (512)", + spec=dict( + tokens=64, + local_expert_offset=0, + local_num_experts=256, + **dict(_R1, intermediate=512), + ), + ), + Cell( + why="deepseek default at tp8: all 256 experts, intermediate split 8 ways (256)", + spec=dict( + tokens=64, + local_expert_offset=0, + local_num_experts=256, + **dict(_R1, intermediate=256), + ), + ), + Cell( + why="deepseek default at ep8 (tp8 + ep8 and dep8): the last 32-wide window", + spec=dict(tokens=64, local_expert_offset=224, local_num_experts=32, **_R1), + ), + Cell( + why="deepseek default at dep16: a 16-wide window", + spec=dict(tokens=64, local_expert_offset=112, local_num_experts=16, **_R1), + ), + Cell( + why="qwen4_exp decode: one token over 512 experts, top-10 renormalized, H 2560 / I 640", + spec=dict(tokens=1, **_QWEN4_EXP), + ), + Cell( + why="qwen4_exp at 4 rows: the last count the pre-routed path's static-block " + "permutation kernel takes", + spec=dict(tokens=4, **_QWEN4_EXP), + ), + Cell( + why="qwen4_exp at 5 rows: the first count on the single-cluster permutation " + "kernel (the dynamic-block one stops at 256 experts)", + spec=dict(tokens=5, **_QWEN4_EXP), + ), + Cell( + why="qwen4_exp prefill: 8192 rows, the single-cluster kernel's cap and the top " + "of the certified column", + spec=dict(tokens=8192, **_QWEN4_EXP), + ), + ) + + note = """ + Certified pre-routed only: `routing_logits` is `None` at every cell, so the + kernel's own routing is not exercised and `routing_method_type`, `n_group`, + `topk_group` and `routed_scaling_factor` are inert -- they are passed + because the signature takes them, not because they do anything here. + + That holds at 512 experts / top-10 too, where it would matter if it were + read: which `(num_experts, top_k)` pairs the routing kernels are compiled + for is a per-policy tier list, and a pair with no tier is a logged error, + not a raise. The pre-routed path substitutes its own preprocess/postprocess + pair before dispatch, and `routing_method_type` 0, 2, 4, 5, 6 were measured + bitwise identical to 1 there. It picks its permutation kernel by row count + instead: a static-block kernel up to 4 rows, a dynamic-block one up to 16 + rows only at <= 256 experts, a single-cluster one up to 8192 rows. + + `act_type` 0 (SwiGlu) only. `gemm1_bias`, `gemm1_alpha`, `gemm1_beta`, + `gemm1_clamp_limit` and `gemm2_bias` are `None`. `hidden_states_scale` is + the linear (unswizzled) e4m3 block-scale layout, one scale per 16 elements + of each row. + + At 512 experts only the whole stack is certified: no expert-parallel window + of it, `do_finalize=False` or a caller-supplied `output` was measured there. + + `tune_max_num_tokens` is an autotuner bucket hint and was measured inert; + it is not a token count. `T` is `hidden_states.shape[0]`, and 8192 is the + top of the certified column. A warm autotuner cache was measured bitwise + inert against a cold one at the R1 stack, its windows and the 512-expert + stack; `local_num_experts` separates the tuner's cache entries, so each + layout warms its own. + + The caller owns the weight layout entirely: FC1 and FC2 weights are + pre-shuffled and their scales additionally 128x4 swizzled. Nothing here + checks that, and getting it wrong is 300-3400 bf16 ulp with no error -- the + entry's test builds the layout twice, once in torch and once through the + trtllm preprocessing ops, and compares them before either reaches the op. + """ + + def __call__( + self, + routing_logits: Optional[torch.Tensor], + routing_bias: Optional[torch.Tensor], + hidden_states: torch.Tensor, + hidden_states_scale: torch.Tensor, + gemm1_weights: torch.Tensor, + gemm1_weights_scale: torch.Tensor, + gemm1_bias: Optional[torch.Tensor], + gemm1_alpha: Optional[torch.Tensor], + gemm1_beta: Optional[torch.Tensor], + gemm1_clamp_limit: Optional[torch.Tensor], + gemm2_weights: torch.Tensor, + gemm2_weights_scale: torch.Tensor, + gemm2_bias: Optional[torch.Tensor], + output1_scale_scalar: torch.Tensor, + output1_scale_gate_scalar: torch.Tensor, + output2_scale_scalar: torch.Tensor, + num_experts: int, + top_k: int, + n_group: Optional[int], + topk_group: Optional[int], + intermediate_size: int, + local_expert_offset: int, + local_num_experts: int, + routed_scaling_factor: Optional[float], + routing_method_type: int, + do_finalize: bool, + act_type: int = 0, + topk_weights: Optional[torch.Tensor] = None, + topk_ids: Optional[torch.Tensor] = None, + output: Optional[torch.Tensor] = None, + tune_max_num_tokens: int = 8192, + use_dp: bool = False, + ) -> list[torch.Tensor]: + return torch.ops.trtllm.fp4_block_scale_moe_runner( + routing_logits, + routing_bias, + hidden_states, + hidden_states_scale, + gemm1_weights, + gemm1_weights_scale, + gemm1_bias, + gemm1_alpha, + gemm1_beta, + gemm1_clamp_limit, + gemm2_weights, + gemm2_weights_scale, + gemm2_bias, + output1_scale_scalar, + output1_scale_gate_scalar, + output2_scale_scalar, + num_experts, + top_k, + n_group, + topk_group, + intermediate_size, + local_expert_offset, + local_num_experts, + routed_scaling_factor, + routing_method_type, + do_finalize, + act_type, + topk_weights=topk_weights, + topk_ids=topk_ids, + output=output, + tune_max_num_tokens=tune_max_num_tokens, + use_dp=use_dp, + ) + + def reference( + self, + hidden_states: torch.Tensor, + topk_ids: torch.Tensor, + topk_weights: torch.Tensor, + up: Operand, + gate: Operand, + down: Operand, + fc2_global_scale: float, + local_expert_offset: int = 0, + local_num_experts: Optional[int] = None, + ) -> torch.Tensor: + """The layer, in plain torch over dequantized weights, fp32 throughout. + + The documented exception to mirroring: `__call__` takes the *prepared* + operands -- packed e2m1 codes, shuffled rows, swizzled scales -- and a + reference given those would be re-deriving the layout rather than the + computation. So this takes the per-expert `(codes, scales)` pairs the + layout was built from, which is where a target's checkpoint starts too, + plus the routing this call was entered with. + + `up`, `gate` and `down` are each `(codes[E, M, K/2], scales[E, M, K/16])` + in their unshuffled, unswizzled form. `topk_ids` carries *global* ids; + this rank contributes only for the ids its window owns, and every other + slot contributes zero -- which is what makes an expert-parallel split + summable. + + The FC1 activation is requantized to NVFP4 under `fc2_global_scale` + before FC2, because the kernel does: the FC1 epilogue emits `g2 * act` + as NVFP4 and FC2's alpha divides it back out. Skipping that step is a + reference that disagrees with a correct kernel by ~23 ulp RMS, so it is + not an optional refinement. + """ + num_tokens, hidden = hidden_states.shape + up_c, up_s = up + gt_c, gt_s = gate + dn_c, dn_s = down + local_num_experts = up_c.shape[0] if local_num_experts is None else local_num_experts + out = torch.zeros(num_tokens, hidden, dtype=torch.float32, device=hidden_states.device) + for local_e in range(local_num_experts): + tok, slot = (topk_ids == local_expert_offset + local_e).nonzero(as_tuple=True) + if tok.numel() == 0: + continue + xe = hidden_states[tok] + u = xe @ dequantize_nvfp4(up_c[local_e], up_s[local_e]).t() + g = xe @ dequantize_nvfp4(gt_c[local_e], gt_s[local_e]).t() + act = u * g * torch.sigmoid(g) + act = quantize_nvfp4(act, fc2_global_scale) / fc2_global_scale + y = act @ dequantize_nvfp4(dn_c[local_e], dn_s[local_e]).t() + out.index_add_(0, tok, y * topk_weights[tok, slot].float().unsqueeze(1)) + return out + + def is_valid( + self, + routing_logits: Optional[torch.Tensor], + routing_bias: Optional[torch.Tensor], + hidden_states: torch.Tensor, + hidden_states_scale: torch.Tensor, + gemm1_weights: torch.Tensor, + gemm1_weights_scale: torch.Tensor, + gemm1_bias: Optional[torch.Tensor], + gemm1_alpha: Optional[torch.Tensor], + gemm1_beta: Optional[torch.Tensor], + gemm1_clamp_limit: Optional[torch.Tensor], + gemm2_weights: torch.Tensor, + gemm2_weights_scale: torch.Tensor, + gemm2_bias: Optional[torch.Tensor], + output1_scale_scalar: torch.Tensor, + output1_scale_gate_scalar: torch.Tensor, + output2_scale_scalar: torch.Tensor, + num_experts: int, + top_k: int, + n_group: Optional[int], + topk_group: Optional[int], + intermediate_size: int, + local_expert_offset: int, + local_num_experts: int, + routed_scaling_factor: Optional[float], + routing_method_type: int, + do_finalize: bool, + act_type: int = 0, + topk_weights: Optional[torch.Tensor] = None, + topk_ids: Optional[torch.Tensor] = None, + output: Optional[torch.Tensor] = None, + tune_max_num_tokens: int = 8192, + use_dp: bool = False, + ) -> None: + """The two ways this op takes an input and answers it wrongly. + + Everything else a caller can get wrong here the op rejects loudly: a + mismatched `intermediate_size`, a `top_k` outside `(0, num_experts)`, a + hidden width that disagrees between the two weight operands. Those are + not restated -- swapping their errors for ours would only make them + worse. + """ + # The kernel takes raw data pointers and assumes a dense row-major + # layout for every tensor. A strided view is accepted without complaint + # and silently reads -- or writes -- the wrong elements. Observed on + # this machine for hidden_states, topk_weights and the weight operands. + for name, tensor in ( + ("routing_logits", routing_logits), + ("routing_bias", routing_bias), + ("hidden_states", hidden_states), + ("hidden_states_scale", hidden_states_scale), + ("gemm1_weights", gemm1_weights), + ("gemm1_weights_scale", gemm1_weights_scale), + ("gemm1_bias", gemm1_bias), + ("gemm1_alpha", gemm1_alpha), + ("gemm1_beta", gemm1_beta), + ("gemm1_clamp_limit", gemm1_clamp_limit), + ("gemm2_weights", gemm2_weights), + ("gemm2_weights_scale", gemm2_weights_scale), + ("gemm2_bias", gemm2_bias), + ("output1_scale_scalar", output1_scale_scalar), + ("output1_scale_gate_scalar", output1_scale_gate_scalar), + ("output2_scale_scalar", output2_scale_scalar), + ("topk_weights", topk_weights), + ("topk_ids", topk_ids), + ("output", output), + ): + if tensor is not None: + assert tensor.is_contiguous(), ( + f"{name} must be contiguous; a strided view is read as if dense " + "and silently produces wrong results" + ) + + # With the block-scale layout this entry contracts -- row shuffle plus + # 128x4 scale swizzle -- a hidden size that is only a multiple of 128 + # makes the kernel read the FC1 weight scales for the wrong blocks. + # Measured wrong at hidden 384 / 640 / 896: no error, 300-3400 bf16 ulp. + assert gemm1_weights.shape[-1] * 2 % 256 == 0, ( + "hidden size (gemm1_weights.shape[-1] * 2) must be a multiple of 256; " + "the prepared block-scale layout is silently misread otherwise" + ) + + +fp4_block_scale_moe_runner = Fp4BlockScaleMoeRunner() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/quantization/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/quantization/__init__.py new file mode 100644 index 000000000000..7d0c28eefd77 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/quantization/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Quantization entries.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/quantization/fp4_quantize.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/quantization/fp4_quantize.py new file mode 100644 index 000000000000..58e9ee52f2df --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/quantization/fp4_quantize.py @@ -0,0 +1,229 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""NVFP4 quantization: packed e2m1 data plus one e4m3 scale per 16 elements.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +_E4M3_MAX = 448.0 +_E2M1_MAX = 6.0 + +#: Midpoints between consecutive e2m1 magnitudes. A value landing exactly on one +#: is a rounding tie; `_TIE_UP[i]` is True when rounding *up* yields the even +#: code, which is the direction round-to-nearest-even takes. +_E2M1_MIDPOINTS = (0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0) +_TIE_UP = (False, True, False, True, False, True, False) + +#: The kernel builds its per-block output scale through two `rcp.approx.ftz.f32` +#: reciprocals, each ~2^-23 relative. A value within a few fp32 ulps of a +#: midpoint can therefore round to either neighbour, and no reference can say +#: which. Everything further out than this must match bit for bit. Largest +#: deviation observed on this machine: 7.9e-8 relative, ~12x inside the window. +TIE_WINDOW = 2.0**-20 + + +def _pad_up(value: int, multiple: int) -> int: + return (value + multiple - 1) // multiple * multiple + + +class Fp4Quantize(OpWrapper): + """Quantize to FP4: `(packed e2m1 data [..., K/2] uint8, scale bytes, 1-D)`. + + One scale per `sf_vec_size` contiguous elements of the last dim. NVFP4 -- + the only form any shipped target uses -- is `sf_vec_size=16` with + `sf_use_ue8m0=False`, which makes the scale an e4m3 byte derived from the + block maximum and the caller's `global_scale`. + + `is_sf_swizzled_layout` selects the 128x4 swizzled scale order (True) or the + row-major linear order (False); the buffer is 1-D either way, so the layout + is something a consumer has to be told, not something its shape shows. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # deepseek-r1 reaches this op from both layouts: the swizzled one feeding + # nvfp4_gemm in the dense and shared-expert MLPs, the linear one feeding the + # routed-expert path, at its hidden size and its intermediate. qwen3.8 + # (qwen4_exp) calls it only for its routed experts: linear scales at hidden + # 2560. A linear buffer has no row padding, so at a row count that is not a + # multiple of 128 it is shorter than the swizzled one; at a multiple of 128 + # the two are the same length and only the cells' byte order tells them + # apart. + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode, hidden 7168, swizzled scales -- the dense MLP's input", + spec=dict(shape=(1, 7168), dtype=torch.bfloat16, vec=16, swizzled=True), + ), + Cell( + why="prefill chunk, hidden 7168, swizzled", + spec=dict(shape=(2048, 7168), dtype=torch.bfloat16, vec=16, swizzled=True), + ), + Cell( + why="linear scales -- the routed-expert path asks for this order", + spec=dict(shape=(2048, 7168), dtype=torch.bfloat16, vec=16, swizzled=False), + ), + Cell( + why="the activation width, 2 x moe_intermediate, between the two GEMMs", + spec=dict(shape=(512, 4096), dtype=torch.bfloat16, vec=16, swizzled=True), + ), + Cell( + why="a row count that is not a multiple of the swizzle's 128-row tile", + spec=dict(shape=(37, 7168), dtype=torch.bfloat16, vec=16, swizzled=True), + ), + Cell( + why="deepseek default at attention TP8: the shared expert's down input, 2048/8 wide", + spec=dict(shape=(16, 256), dtype=torch.bfloat16, vec=16, swizzled=True), + ), + Cell( + why="qwen3.8 decode: 1 token, hidden 2560, linear -- its routed-expert input", + spec=dict(shape=(1, 2560), dtype=torch.bfloat16, vec=16, swizzled=False), + ), + Cell( + why="qwen3.8, 7 rows at hidden 2560, linear: several rows, no row padding", + spec=dict(shape=(7, 2560), dtype=torch.bfloat16, vec=16, swizzled=False), + ), + Cell( + why="qwen3.8 prefill at max_num_tokens 8192, hidden 2560, linear", + spec=dict(shape=(8192, 2560), dtype=torch.bfloat16, vec=16, swizzled=False), + ), + ) + + note = """ + `sf_use_ue8m0=True` selects power-of-two scales instead of e4m3. No shipped + target uses it and this entry does not certify it. + """ + + def __call__( + self, + input: torch.Tensor, + global_scale: Optional[torch.Tensor], + sf_vec_size: int, + sf_use_ue8m0: bool = False, + is_sf_swizzled_layout: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + return torch.ops.trtllm.fp4_quantize( + input, global_scale, sf_vec_size, sf_use_ue8m0, is_sf_swizzled_layout + ) + + def reference( + self, + input: torch.Tensor, + global_scale: Optional[torch.Tensor], + sf_vec_size: int, + sf_use_ue8m0: bool = False, + is_sf_swizzled_layout: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Native-torch NVFP4, with the scales in the requested layout. + + Per block: `vecmax = max|x|`, the scale is `e4m3(global_scale * vecmax / + 6)`, and the data is the input times `global_scale / scale` rounded to + e2m1. torch's fp32 -> e4m3 cast emits NaN above 464 where the kernel's + saturates, so the scale is clamped first -- between 448 and 464 both + round to 448 anyway. + """ + packed, sf, _ = self._quantize(input, global_scale, sf_vec_size) + if not is_sf_swizzled_layout: + return packed, sf.reshape(-1) + return packed, self._swizzle(sf) + + def scaled( + self, input: torch.Tensor, global_scale: Optional[torch.Tensor], sf_vec_size: int + ) -> torch.Tensor: + """The pre-rounding product, which is what `near_tie` reads. + + Exposed because a caller cannot reconstruct it from the op's outputs: + rounding is the step that loses it. + """ + return self._quantize(input, global_scale, sf_vec_size)[2] + + @staticmethod + def near_tie(scaled: torch.Tensor) -> torch.Tensor: + """Elements whose rounding this entry refuses to predict. + + Inside `TIE_WINDOW` of an e2m1 midpoint the kernel's approximate + reciprocals decide the direction, so a code that differs there is not a + defect. Outside it, a difference is. + """ + mid = torch.tensor(_E2M1_MIDPOINTS, device=scaled.device, dtype=torch.float32) + a = scaled.abs().unsqueeze(-1) + return ((a - mid).abs() <= TIE_WINDOW * a).any(-1) + + def is_valid( + self, + input: torch.Tensor, + global_scale: Optional[torch.Tensor], + sf_vec_size: int, + sf_use_ue8m0: bool = False, + is_sf_swizzled_layout: bool = True, + ) -> None: + # The kernel loads one scalar from global_scale and ignores every + # element past the first, so a per-token [num_tokens] tensor is silently + # applied as global_scale[0] to every row -- observed to return a + # valid-looking result, never to raise. + assert global_scale is None or global_scale.numel() == 1, ( + "global_scale must hold exactly one element; extra elements are " + "silently ignored and global_scale[0] is applied to every row" + ) + assert input.shape[-1] % sf_vec_size == 0, ( + f"the quantized dim must be a whole number of {sf_vec_size}-element " + f"blocks; got {input.shape[-1]}" + ) + assert input.is_contiguous(), "input must be contiguous; the kernel reads it flat" + + # ── the arithmetic, shared by reference() and scaled() ────────────────── + + def _quantize( + self, input: torch.Tensor, global_scale: Optional[torch.Tensor], vec: int + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + gs = 1.0 if global_scale is None else global_scale.float().reshape(()) + k = input.shape[-1] + rows = input.numel() // k + flat = input.reshape(rows, k) + vecmax = flat.abs().reshape(rows, k // vec, vec).amax(-1).float() + sf = (gs * (vecmax / _E2M1_MAX)).clamp(max=_E4M3_MAX).to(torch.float8_e4m3fn) + out_scale = torch.where(vecmax != 0, gs / sf.float(), torch.zeros_like(vecmax)) + scaled = (flat.float().reshape(rows, k // vec, vec) * out_scale.unsqueeze(-1)).reshape( + rows, k + ) + codes = self._e2m1_codes(scaled) + packed = codes[:, 0::2] | (codes[:, 1::2] << 4) + return packed, sf.view(torch.uint8), scaled + + @staticmethod + def _e2m1_codes(v: torch.Tensor) -> torch.Tensor: + """fp32 -> 4-bit e2m1 codes (bit 3 = sign), round-to-nearest-even.""" + mid = torch.tensor(_E2M1_MIDPOINTS, device=v.device, dtype=torch.float32) + tie_up = torch.tensor(_TIE_UP, device=v.device, dtype=torch.bool) + a = v.abs().unsqueeze(-1) + greater = (a > mid).sum(-1) # strict: ties provisionally round down + up = ((a == mid) & tie_up).any(-1) # ties whose even side is up + code = (greater + up).to(torch.uint8) + negative = (v < 0) | ((v == 0) & torch.signbit(v)) + return code | (negative.to(torch.uint8) << 3) + + @staticmethod + def _swizzle(sf: torch.Tensor) -> torch.Tensor: + """Scatter [rows, cols] scale bytes into the 128x4-swizzled buffer.""" + rows, cols = sf.shape + padded_cols, padded_rows = _pad_up(cols, 4), _pad_up(rows, 128) + r = torch.arange(rows, device=sf.device).view(-1, 1) + c = torch.arange(cols, device=sf.device).view(1, -1) + index = ( + (c % 4) + + (c // 4) * (4 * 128) + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * padded_cols) + ) + out = torch.zeros(padded_rows * padded_cols, dtype=torch.uint8, device=sf.device) + out[index.reshape(-1)] = sf.reshape(-1) + return out + + +fp4_quantize = Fp4Quantize() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/__init__.py new file mode 100644 index 000000000000..4e871f35b7f0 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Speculative-decoding entries: the drafter's own ops.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_block_tables.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_block_tables.py new file mode 100644 index 000000000000..f7c89b37486d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_block_tables.py @@ -0,0 +1,173 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark / DFlash drafter block-table refresh: one launch per step +# ============================================================================= +# +# It replaces, bit for bit, what DFlashWorker._refresh_ctx_block_tables does with the draft pool's encoded block +# offsets of this step (field 0 of the K/V pair, pool ``pool_idx``, the first ``num_seqs`` rows): +# counts[r] = number of entries >= 0 in row r (placeholders are negative) +# tables[r, j] = max(offset[r, j], 0) // divisor (int32) +# Integer arithmetic only, so the result is exact; the stock path spends one launch per torch op on it. + +from typing import Optional + +import torch +import triton +import triton.language as tl + + +@triton.jit +def _refresh_kernel( + src_ptr, # encoded offsets, row r at src_ptr + r * src_stride_row, entry j at + j * src_stride_col + src_stride_row, + src_stride_col, + tables_ptr, # int32 [rows, width], row-major + tables_stride, + counts_ptr, # int64 [rows] + width, + divisor, + BLOCK: tl.constexpr, +): + r = tl.program_id(0).to(tl.int64) + count = tl.zeros((), dtype=tl.int64) + for start in range(0, width, BLOCK): + o = start + tl.arange(0, BLOCK) + m = o < width + v = tl.load(src_ptr + r * src_stride_row + o * src_stride_col, mask=m, other=-1).to( + tl.int64 + ) + count += tl.sum((v >= 0).to(tl.int64), axis=0) + # Non-negative after the clamp, so integer division is the stock floor division. + tl.store( + tables_ptr + r * tables_stride + o, (tl.maximum(v, 0) // divisor).to(tl.int32), mask=m + ) + tl.store(counts_ptr + r, count) + + +@triton.jit +def _refresh_kernel_2d( + src_ptr, + src_stride_row, + src_stride_col, + tables_ptr, + tables_stride, + counts_ptr, + width, + divisor, + BLOCK: tl.constexpr, + COUNT_BLOCK: tl.constexpr, +): + """S10c-B2: grid (rows, column chunks). Each program decodes one chunk in int32 (the offsets fit int32 and are + non-negative after the clamp, so int32 floor division is the stock floor division); chunk 0 of a row also counts + the row's non-placeholder entries over the whole row. No atomics: every output has exactly one writer.""" + r = tl.program_id(0).to(tl.int64) + c = tl.program_id(1) + o = c * BLOCK + tl.arange(0, BLOCK) + m = o < width + v = tl.load(src_ptr + r * src_stride_row + o * src_stride_col, mask=m, other=-1).to(tl.int32) + tl.store(tables_ptr + r * tables_stride + o, tl.maximum(v, 0) // divisor, mask=m) + if c == 0: + count = tl.zeros((), dtype=tl.int64) + for start in range(0, width, COUNT_BLOCK): + oc = start + tl.arange(0, COUNT_BLOCK) + vc = tl.load( + src_ptr + r * src_stride_row + oc * src_stride_col, mask=oc < width, other=-1 + ) + count += tl.sum((vc >= 0).to(tl.int64), axis=0) + tl.store(counts_ptr + r, count) + + +def refresh_block_tables( + offsets: torch.Tensor, + pool_idx: int, + num_seqs: int, + divisor: int, + tables: torch.Tensor, + counts: torch.Tensor, +) -> bool: + """Decode this step's draft block offsets ``offsets[pool_idx, :num_seqs, 0]`` into ``tables[:num_seqs]`` and their + non-placeholder counts into ``counts[:num_seqs]``, in one launch; returns False (nothing written) when the layout + is not one the kernel reproduces, so the caller keeps the stock path.""" + if not supported(offsets, tables, counts, divisor): + return False + if num_seqs <= 0: + return True + src = offsets[pool_idx, :num_seqs, 0] + width = src.shape[-1] + if _parallel[0] and divisor <= 2**31 - 1 and (offsets.dtype == torch.int32 or _fits_int32(src)): + block = min(1024, triton.next_power_of_2(width)) + _refresh_kernel_2d[(num_seqs, triton.cdiv(width, block))]( + src, + src.stride(0), + src.stride(1), + tables, + tables.stride(0), + counts, + width, + divisor, + BLOCK=block, + COUNT_BLOCK=min(4096, triton.next_power_of_2(width)), + num_warps=4, + ) + return True + _refresh_kernel[(num_seqs,)]( + src, + src.stride(0), + src.stride(1), + tables, + tables.stride(0), + counts, + width, + divisor, + BLOCK=min(1024, triton.next_power_of_2(width)), + num_warps=1, + ) + return True + + +_parallel = [False] # S10c-B2: the 2-D int32 decode (see enable_parallel_refresh) + + +def enable_parallel_refresh(enabled: bool = True) -> bool: + """S10c-B2: decode with the (row, chunk) grid in int32 instead of one int64 program per row (the same values: the + offsets fit int32 and are non-negative after the clamp). Off by default.""" + _parallel[0] = bool(enabled) + return _parallel[0] + + +def _fits_int32(src: torch.Tensor) -> bool: + """int64 offsets are decoded in int32 only when the caller's layout guarantees they fit; without a cheap + capture-safe check, int64 offsets keep the int64 kernel.""" + return False + + +def supported( + offsets: torch.Tensor, tables: torch.Tensor, counts: torch.Tensor, divisor: Optional[int] +) -> bool: + """[pools, rows, 2, width] int32 / int64 offsets on the tables' device; int32 row-major tables of the same width; + int64 counts; a positive integer divisor.""" + return ( + offsets.dim() == 4 + and offsets.dtype in (torch.int32, torch.int64) + and tables.dtype == torch.int32 + and tables.dim() == 2 + and tables.stride(1) == 1 + and tables.shape[1] == offsets.shape[-1] + and counts.dtype == torch.int64 + and offsets.device == tables.device == counts.device + and isinstance(divisor, int) + and divisor >= 1 + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_advance.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_advance.py new file mode 100644 index 000000000000..5d20f9f90217 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_advance.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark / DFlash drafter context advance + read view: one launch per step +# ============================================================================= +# +# It replaces, bit for bit, what DFlashWorker's generation branch does after the context store on a bound draft pool +# (_advance_ctx_len, then the advertised context length and the rebased read view of dflash_rebased_ctx_view): +# ctx_len[slot] = min(ctx_len[slot] + num_accepted, max_ctx); ctx_len[dummy_slot] = 0 +# num_ctx = min(ctx_len[slot], max(block_counts[row] * page - block_size, 0)) +# rebase = min(base_pages[slot], num_ctx // page) +# num_ctx_out = max(num_ctx - base_pages[slot] * page, 0) (int64) +# read[row, j] = tables[row, min(j + rebase, width - 1)] (int32) +# One program per generation request, so each request's advance is read back by the same program; rows on the dummy +# slot (padding) skip the add and read 0, which the stock path's zeroing after the add makes them read too. + +import torch +import triton +import triton.language as tl + + +@triton.jit +def _advance_kernel( + slots_ptr, # [n] int64 + rows_ptr, # [n] int64: block-table rows of the requests + nacc_ptr, # [n] int + ctx_len_ptr, # [slots] int64, in / out + counts_ptr, # [rows] int64 + base_ptr, # [slots] int64 + tables_ptr, # int32 [rows, width] + read_ptr, # int32 [rows, width] + out_ptr, # [n] int64 advertised context lengths + width, + tables_stride, + read_stride, + max_ctx, + dummy_slot, + page, + block_size, + BLOCK: tl.constexpr, +): + r = tl.program_id(0).to(tl.int64) + slot = tl.load(slots_ptr + r) + row = tl.load(rows_ptr + r) + is_dummy = slot == dummy_slot + old = tl.load(ctx_len_ptr + slot) + new = tl.minimum(old + tl.load(nacc_ptr + r).to(tl.int64), max_ctx) + if not is_dummy: + tl.store(ctx_len_ptr + slot, new) + if r == 0: + tl.store(ctx_len_ptr + dummy_slot, tl.zeros((), dtype=tl.int64)) + new = tl.where(is_dummy, 0, new) + allocated = tl.maximum(tl.load(counts_ptr + row) * page - block_size, 0) + num_ctx = tl.minimum(new, allocated) + base = tl.load(base_ptr + slot) + rebase = tl.minimum(base, num_ctx // page) + tl.store(out_ptr + r, tl.maximum(num_ctx - base * page, 0)) + for start in range(0, width, BLOCK): + j = start + tl.arange(0, BLOCK) + m = j < width + src = tl.minimum(j + rebase, width - 1) + v = tl.load(tables_ptr + row * tables_stride + src, mask=m) + tl.store(read_ptr + row * read_stride + j, v, mask=m) + + +def advance_ctx_and_read_view( + slots: torch.Tensor, + rows: torch.Tensor, + num_accepted: torch.Tensor, + ctx_len: torch.Tensor, + max_ctx: int, + dummy_slot: int, + block_counts: torch.Tensor, + base_pages: torch.Tensor, + tables: torch.Tensor, + tables_read: torch.Tensor, + page_size: int, + block_size: int, +): + """Advance the generation requests' context lengths and build their read view in one launch: returns the + advertised context lengths [n] (int64) and writes ``tables_read[rows]``; None (nothing written) for a layout the + kernel does not reproduce, so the caller keeps the stock path.""" + n = slots.shape[0] + if not ( + slots.dim() == 1 + and rows.shape == (n,) + and num_accepted.shape == (n,) + and num_accepted.dtype in (torch.int32, torch.int64) + and ctx_len.dtype == torch.int64 + and block_counts.dtype == torch.int64 + and base_pages.dtype == torch.int64 + and tables.dtype == torch.int32 + and tables_read.dtype == torch.int32 + and tables.shape == tables_read.shape + and tables.stride(1) == 1 + and tables_read.stride(1) == 1 + and page_size >= 1 + ): + return None + out = torch.empty(n, dtype=torch.int64, device=ctx_len.device) + if n == 0: + ctx_len[dummy_slot] = 0 + return out + width = tables.shape[1] + _advance_kernel[(n,)]( + slots.to(torch.int64).contiguous(), + rows.to(torch.int64).contiguous(), + num_accepted.contiguous(), + ctx_len, + block_counts, + base_pages, + tables, + tables_read, + out, + width, + tables.stride(0), + tables_read.stride(0), + int(max_ctx), + int(dummy_slot), + int(page_size), + int(block_size), + BLOCK=min(1024, triton.next_power_of_2(width)), + num_warps=4, + ) + return out diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_store.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_store.py new file mode 100644 index 000000000000..26b67405c20c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_store.py @@ -0,0 +1,309 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark / DFlash drafter context K / V store after the fused K / V GEMM: one launch for every drafter layer +# ============================================================================= +# +# Adapted from SGLang's DFlash KV materialization (_fused_norm_rope_kernel_stacked in +# python/sglang/kernels/ops/speculative/fused_kv_materialize.py, https://github.com/sgl-project/sglang, commit +# 6ed9843b57aaecbf0566f24a4eaf78c7c92ef25d, Apache-2.0): one program per (row, layer, kv head) reads the stacked +# [K | V] GEMM output, applies the layer's k_norm and NeoX RoPE to K. Changed here: the 0/1 accept mask on K and V, the +# paged HND store through the request's block table (instead of a dense output), and the stock path's rounding points. +# +# It replaces, bit for bit, what DFlashWorker does after DFlashForCausalLM.precompute_context_kv's GEMM. The k_norm +# stays torch's own F.rms_norm (as precompute_context_kv runs it: an in-kernel row sum did not reproduce torch's +# reduction order on every input); the bf16 weight multiply runs in the kernel (``fold_k_norm_weight``, TAILBK_SPEC F: +# fp32 product rounded to bf16, as torch's bf16 multiply) or as torch's own; this one launch does the rest: +# k = rms_norm(k) (fp32, rounded to bf16) * k_norm weight (bf16) -> flashinfer cos/sin-cache NeoX RoPE over the +# first ROT dims, ROT = the cache's width (the rotary dim; a partial rotary leaves dims ROT..HD as they are) (fp32 +# from the bf16 k, x*cos + (-/+)p*sin as fma(x, cos, (-/+)p*sin), rounded to bf16) -> k, v *= accept mask (bf16) -> +# flashinfer append_paged_kv_cache per layer (HND) at (block_table[row, col // page], col % page). +# Row r belongs to request r // (K+1) at block offset r % (K+1); it is accepted when that offset < num_accepted. +# Masked rows still write (zeros) at their clamped column, as the stock fixed-size write does. + +from typing import Dict, Optional, Sequence, Tuple, Union + +import torch +import torch.nn.functional as F +import triton +import triton.language as tl + + +@triton.jit +def _ctx_store_kernel( + kv_ptr, # [N, L * 2 * NKV * HD] the fused K / V GEMM output, per layer [K heads | V heads] (V is read here) + k_ptr, # [N, L, NKV, HD] K after torch's k_norm and (without FOLD_W) the weight multiply + w_ptr, # [L, HD] bf16 k_norm weight (FOLD_W) + cs_ptr, # [max_pos, ROT] fp32 cos | sin (flashinfer's cos/sin cache; ROT = the rotary dim) + pos_ptr, # [N] RoPE positions + nacc_ptr, # [N // KP1] accepted tokens per request + rows_ptr, # [N // KP1] block-table row per request + col_ptr, # [N] cache column per row (unused with COL_FROM_POS) + counts_ptr, # [rows] int64 pages per block-table row (COL_FROM_POS on a bound pool) + max_col, # the private arena's last column (COL_FROM_POS without a bound pool) + table_ptr, # [rows, pages_per_row] page ids + cache_ptr, # stacked [L, pages, 2, NKV, PS, HD] cache, or an int64 array of L per-layer cache addresses + kv_stride_row, + table_stride, + cache_stride_layer, + KP1: tl.constexpr, + L: tl.constexpr, + NKV: tl.constexpr, + HD: tl.constexpr, + ROT: tl.constexpr, + PS: tl.constexpr, + PTR_LIST: tl.constexpr, + COL_FROM_POS: tl.constexpr, # derive the column from the position as the stock clamp does + BOUND: tl.constexpr, # COL_FROM_POS on a bound pool (clamp to the row's pages) vs the private arena + FOLD_W: tl.constexpr, # k_ptr holds rms_norm(k) (bf16); the bf16 weight multiply runs here +): + r = tl.program_id(0) + layer = tl.program_id(1) + h = tl.program_id(2) + half: tl.constexpr = HD // 2 + rhalf: tl.constexpr = ROT // 2 + o = tl.arange(0, half) + ro = tl.arange(0, rhalf) + req = r // KP1 + m = tl.where((r % KP1) < tl.load(nacc_ptr + req), 1.0, 0.0) + vbase = kv_ptr + r * kv_stride_row + (layer * 2 * NKV + NKV + h) * HD + kbase = k_ptr + ((r * L + layer) * NKV + h) * HD + a = tl.load(kbase + ro) + b = tl.load(kbase + rhalf + ro) + if FOLD_W: # torch's bf16 * bf16: the fp32 product rounded to bf16 + wbase = w_ptr + layer * HD + a = (a.to(tl.float32) * tl.load(wbase + ro).to(tl.float32)).to(tl.bfloat16) + b = (b.to(tl.float32) * tl.load(wbase + rhalf + ro).to(tl.float32)).to(tl.bfloat16) + a = a.to(tl.float32) + b = b.to(tl.float32) + p = tl.load(pos_ptr + r) + c = tl.load(cs_ptr + p * ROT + ro) + s = tl.load(cs_ptr + p * ROT + rhalf + ro) + ra = (tl.fma(a, c, -b * s).to(tl.bfloat16).to(tl.float32) * m).to(tl.bfloat16) + rb = (tl.fma(b, c, a * s).to(tl.bfloat16).to(tl.float32) * m).to(tl.bfloat16) + v0 = (tl.load(vbase + o).to(tl.float32) * m).to(tl.bfloat16) + v1 = (tl.load(vbase + half + o).to(tl.float32) * m).to(tl.bfloat16) + row = tl.load(rows_ptr + req) + last = (r % KP1) == KP1 - 1 + if COL_FROM_POS: + # The stock column: ctx_len + offset (= the position), clamped to the request's pages (bound pool: + # min(col, pages * PS - 1).clamp(min=0)) or to the arena (min(col, max_ctx - 1)). + p_nxt = tl.load(pos_ptr + tl.where(last, r, r + 1)) + if BOUND: + cap = tl.load(counts_ptr + row) * PS - 1 + col = tl.maximum(tl.minimum(p, cap), 0) + nxt = tl.maximum(tl.minimum(p_nxt, cap), 0) + else: + col = tl.minimum(p, max_col) + nxt = tl.minimum(p_nxt, max_col) + else: + col = tl.load(col_ptr + r) + nxt = tl.load( + col_ptr + tl.where(last, r, r + 1) + ) # the next row of the same request (itself for the last) + page = tl.load(table_ptr + row * table_stride + col // PS).to(tl.int64) + if PTR_LIST: + cache = tl.load(cache_ptr + layer).to(tl.pointer_type(tl.bfloat16)) + else: + cache = cache_ptr + layer.to(tl.int64) * cache_stride_layer + kdst = cache + ((page * 2 * NKV + h) * PS + col % PS) * HD + vdst = cache + ((page * 2 * NKV + NKV + h) * PS + col % PS) * HD + # Clamped rows of one request share a column; the last of them writes. Stock's order there is a race between the + # append's thread blocks; that no reader reads such a cell is shown in the S10c-A in-process gate. + write = last | (nxt != col) + tl.store(kdst + ro, ra, mask=write) + tl.store(kdst + rhalf + ro, rb, mask=write) + if ROT < HD: # partial rotary: dims ROT..HD are not rotated, only masked + po = ROT + tl.arange(0, HD - ROT) + kp = tl.load(kbase + po) + if FOLD_W: + kp = (kp.to(tl.float32) * tl.load(w_ptr + layer * HD + po).to(tl.float32)).to( + tl.bfloat16 + ) + kp = (kp.to(tl.float32) * m).to(tl.bfloat16) + tl.store(kdst + po, kp, mask=write) + tl.store(vdst + o, v0, mask=write) + tl.store(vdst + half + o, v1, mask=write) + + +_fold_w = [ + False +] # TAILBK_SPEC F: the k_norm weight multiply in the kernel (see fold_k_norm_weight) + + +def fold_k_norm_weight(enabled: bool = True) -> bool: + """Run the stacked k_norm weight multiply inside the store kernel (one launch fewer per call) when the weight is a + contiguous bf16 [L, HD] and rms_norm's output is bf16: the fp32 product rounded to bf16, bitwise torch's bf16 + multiply. Off by default; process-wide.""" + _fold_w[0] = bool(enabled) + return _fold_w[0] + + +# Per-layer cache lists (the bound draft pool) -> their int64 address array, built once outside capture. +_PTR_ARRAYS: Dict[Tuple[int, ...], torch.Tensor] = {} + + +def _cache_arg( + kv_buf: Union[torch.Tensor, Sequence[torch.Tensor]], +) -> Optional[Tuple[torch.Tensor, bool, int]]: + if isinstance(kv_buf, torch.Tensor): + return kv_buf, False, kv_buf.stride(0) + key = tuple(t.data_ptr() for t in kv_buf) + arr = _PTR_ARRAYS.get(key) + if arr is None: + if torch.cuda.is_current_stream_capturing(): + return None # never seen eagerly: the caller keeps the stock store for this capture + arr = torch.tensor(key, dtype=torch.int64, device=kv_buf[0].device) + _PTR_ARRAYS[key] = arr + return arr, True, 0 + + +def _pow2(n: int) -> bool: + return n >= 1 and n & (n - 1) == 0 + + +def _rotary_dim(draft_model, hd: int) -> int: + """The rotary dim the drafter's RoPE uses (the width of its fp32 cos/sin cache: flashinfer rotates that many leading + dims of each head), or 0 when the kernel cannot reproduce it.""" + cos_sin = draft_model._get_cos_sin_cache() + if cos_sin.dim() != 2 or cos_sin.dtype != torch.float32 or not cos_sin.is_contiguous(): + return 0 + rot = cos_sin.shape[1] + if rot % 2 or rot > hd or not _pow2(rot // 2) or (rot < hd and not _pow2(hd - rot)): + return 0 + return rot + + +def ctx_store_supported(draft_model, kv_buf) -> bool: + """Whether this drafter's context store is one the kernel reproduces: GQA K and V into a bf16 HND paged cache, + stacked per-layer k_norm, NeoX RoPE over the leading rotary dims (whole or partial head) from the fp32 cos/sin + cache, head_dim a power of two.""" + first = kv_buf if isinstance(kv_buf, torch.Tensor) else kv_buf[0] + layer_shape = first.shape[1:] if isinstance(kv_buf, torch.Tensor) else first.shape + hd = getattr(draft_model, "_head_dim", 0) + return ( + getattr(draft_model, "_k_norm_stacked", None) is not None + and getattr(draft_model, "_k_norm_eps", None) is not None + and getattr(draft_model, "_is_neox", False) + and getattr(draft_model, "_num_kv_heads", 0) >= 1 + and hd >= 16 + and hd & (hd - 1) == 0 + and first.dtype == torch.bfloat16 + and len(layer_shape) == 5 + and layer_shape[1] == 2 + and layer_shape[2] == draft_model._num_kv_heads + and layer_shape[4] == hd + and all( + t.is_contiguous() for t in ([kv_buf] if isinstance(kv_buf, torch.Tensor) else kv_buf) + ) + and _rotary_dim(draft_model, hd) > 0 + ) + + +def fused_ctx_store( + draft_model, + projected: torch.Tensor, + positions: torch.Tensor, + num_accepted: torch.Tensor, + request_rows: torch.Tensor, + cols: Optional[torch.Tensor], + block_table: torch.Tensor, + kv_buf: Union[torch.Tensor, Sequence[torch.Tensor]], + tokens_per_request: int, + *, + capacity_counts: Optional[torch.Tensor] = None, + max_col: Optional[int] = None, +) -> bool: + """Store the context K / V of ``projected`` (fc + hidden_norm'd rows, request-major, ``tokens_per_request`` rows + each) into the paged draft cache; returns False (nothing written) when the stock path must run instead. + + ``request_rows`` / ``num_accepted`` are per request, ``positions`` / ``cols`` per row. With ``cols`` None the kernel + derives each row's column from its position as the stock path does: clamped to the request's pages on a bound pool + (``capacity_counts``: pages per block-table row, int64) or to ``max_col`` on the private arena.""" + if not ctx_store_supported(draft_model, kv_buf): + return False + if cols is None and ( + (capacity_counts is None) == (max_col is None) + or (capacity_counts is not None and capacity_counts.dtype != torch.int64) + ): + return False + cache = _cache_arg(kv_buf) + if cache is None: + return False + cache_arg, ptr_list, cache_stride_layer = cache + if draft_model._fused_kv_weight is None: + draft_model._build_fused_kv_buffers() + weight = draft_model._fused_kv_weight + # precompute_context_kv's GEMM, unchanged. + if getattr(draft_model, "_input_ln_eps", None) is not None: + ph = projected.float() + ph = ph * torch.rsqrt(ph.pow(2).mean(-1, keepdim=True) + draft_model._input_ln_eps) + projected = ph.to(weight.dtype) + elif projected.dtype != weight.dtype: + projected = projected.to(weight.dtype) + kv_flat = F.linear(projected, weight, draft_model._fused_kv_bias) + n = kv_flat.shape[0] + if n == 0: + return True + num_layers, nkv, hd = ( + draft_model._num_attn_layers, + draft_model._num_kv_heads, + draft_model._head_dim, + ) + # precompute_context_kv's k_norm, unchanged: contiguous K, F.rms_norm, the stacked weight multiply. + k = kv_flat.view(n, num_layers, 2, nkv, hd)[:, :, 0].contiguous() + k = F.rms_norm(k, (hd,), eps=draft_model._k_norm_eps) + w = draft_model._k_norm_stacked + fold = ( + _fold_w[0] + and k.dtype == torch.bfloat16 + and w.dtype == torch.bfloat16 + and w.is_contiguous() + and w.numel() == num_layers * hd + and w.device == k.device + ) + if not fold: + k = k * w.view(1, num_layers, 1, hd) + first = kv_buf if isinstance(kv_buf, torch.Tensor) else kv_buf[0] + cos_sin = draft_model._get_cos_sin_cache() + _ctx_store_kernel[(n, num_layers, nkv)]( + kv_flat, + k, + w, + cos_sin, + positions.contiguous(), + num_accepted.contiguous(), + request_rows.contiguous(), + positions if cols is None else cols.contiguous(), + capacity_counts if capacity_counts is not None else positions, + int(max_col) if max_col is not None else 0, + block_table, + cache_arg, + kv_flat.stride(0), + block_table.stride(0), + cache_stride_layer, + KP1=tokens_per_request, + L=num_layers, + NKV=nkv, + HD=hd, + ROT=cos_sin.shape[1], + PS=first.shape[-2], + PTR_LIST=ptr_list, + COL_FROM_POS=cols is None, + BOUND=capacity_counts is not None, + FOLD_W=fold, + num_warps=1, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_drafter_inputs.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_drafter_inputs.py new file mode 100644 index 000000000000..56d9cef29894 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_drafter_inputs.py @@ -0,0 +1,143 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark / DFlash drafter step inputs: one launch per step +# ============================================================================= +# +# It replaces, bit for bit, the start of DFlashWorker.prepare_1st_drafter_inputs' generation branch, for a drafter whose +# token embedding is a plain row gather of a replicated BF16 table: +# bonus[r] = accepted[r, max(num_accepted[r] - 1, 0)] (int64) +# query_pos[r, j] = min(ctx_len[slot[r]] + num_accepted[r], max_ctx) + j, j < block (int64) +# ctx_pos[r, j] = ctx_len[slot[r]] + j, j < K + 1 (int64) +# noise[r, 0, :] = table[bonus[r]], or +0 for an id outside [0, vocab) (m3_embedding's masked_fill) +# noise[r, j>0, :] = mask_row (the drafter's trained mask embedding, else table[mask_token_id]) +# Integer arithmetic and row copies only, so the result is exact; the stock path spends a launch per torch op on it. + +from typing import Optional, Tuple + +import torch +import triton +import triton.language as tl + + +@triton.jit +def _inputs_kernel( + acc_ptr, # accepted tokens [n, acc_stride] (int32 / int64) + acc_stride, + nacc_ptr, # accepted counts [n] (int) + slots_ptr, # [n] int64: the request's ctx slot + ctx_len_ptr, # [slots] int64 + max_ctx, + table_ptr, # [vocab, hidden] bf16 embedding table + mask_ptr, # [hidden] bf16 mask row + bonus_ptr, # out [n] int64 + qpos_ptr, # out [n, BLOCK_Q] int64 + cpos_ptr, # out [n, KP1] int64 + noise_ptr, # out [n, BLOCK_Q, hidden] bf16 + hidden, + vocab, + BLOCK_Q: tl.constexpr, + KP1: tl.constexpr, + BLOCK_QP: tl.constexpr, # next powers of two of BLOCK_Q / KP1 (tl.arange sizes) + KP1P: tl.constexpr, + BLOCK_H: tl.constexpr, +): + r = tl.program_id(0).to(tl.int64) + j = tl.program_id(1) # the block row this program copies + hb = tl.program_id(2) + nacc = tl.load(nacc_ptr + r).to(tl.int64) + bonus = tl.load(acc_ptr + r * acc_stride + tl.maximum(nacc - 1, 0)).to(tl.int64) + o = hb * BLOCK_H + tl.arange(0, BLOCK_H) + m = o < hidden + if j == 0: + valid = (bonus >= 0) & (bonus < vocab) + row = tl.load(table_ptr + tl.where(valid, bonus, 0) * hidden + o, mask=m & valid, other=0.0) + else: + row = tl.load(mask_ptr + o, mask=m) + tl.store(noise_ptr + (r * BLOCK_Q + j) * hidden + o, row, mask=m) + if (j == 0) & (hb == 0): + ctx = tl.load(ctx_len_ptr + tl.load(slots_ptr + r)) + tl.store(bonus_ptr + r, bonus) + now = tl.minimum(ctx + nacc, max_ctx) + q = tl.arange(0, BLOCK_QP) + tl.store(qpos_ptr + r * BLOCK_Q + q, now + q, mask=q < BLOCK_Q) + c = tl.arange(0, KP1P) + tl.store(cpos_ptr + r * KP1 + c, ctx + c, mask=c < KP1) + + +def drafter_inputs( + accepted: torch.Tensor, + num_accepted: torch.Tensor, + slots: torch.Tensor, + ctx_len: torch.Tensor, + max_ctx: int, + table: torch.Tensor, + mask_row: torch.Tensor, + block_size: int, + tokens_per_request: int, +) -> Optional[Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]]: + """(bonus [n] int64, query positions [n, block_size] int64, ctx positions [n, tokens_per_request] int64, + noise block [n, block_size, hidden]) in one launch, or None when the inputs are not a layout the kernel reproduces + (the caller keeps the stock path). ``mask_row`` [hidden] is the row every slot after the first takes.""" + n = num_accepted.shape[0] + if not ( + accepted.dim() == 2 + and accepted.shape[0] == n + and accepted.stride(1) == 1 + and accepted.dtype in (torch.int32, torch.int64) + and num_accepted.dtype in (torch.int32, torch.int64) + and slots.shape == (n,) + and ctx_len.dtype == torch.int64 + and table.dim() == 2 + and table.is_contiguous() + and mask_row.shape == (table.shape[1],) + and mask_row.dtype == table.dtype + and mask_row.is_contiguous() + and block_size >= 1 + and tokens_per_request >= 1 + ): + return None + hidden = table.shape[1] + dev = table.device + bonus = torch.empty(n, dtype=torch.int64, device=dev) + qpos = torch.empty(n, block_size, dtype=torch.int64, device=dev) + cpos = torch.empty(n, tokens_per_request, dtype=torch.int64, device=dev) + noise = torch.empty(n, block_size, hidden, dtype=table.dtype, device=dev) + if n == 0: + return bonus, qpos, cpos, noise + block_h = 1024 + _inputs_kernel[(n, block_size, triton.cdiv(hidden, block_h))]( + accepted, + accepted.stride(0), + num_accepted.contiguous(), + slots.to(torch.int64).contiguous(), + ctx_len, + int(max_ctx), + table, + mask_row, + bonus, + qpos, + cpos, + noise, + hidden, + table.shape[0], + BLOCK_Q=block_size, + KP1=tokens_per_request, + BLOCK_QP=triton.next_power_of_2(block_size), + KP1P=triton.next_power_of_2(tokens_per_request), + BLOCK_H=block_h, + num_warps=4, + ) + return bonus, qpos, cpos, noise diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_forward_indices.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_forward_indices.py new file mode 100644 index 000000000000..42448af2b144 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_forward_indices.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark / DFlash drafter forward: the paged block decode's index prelude in one launch +# ============================================================================= +# +# It replaces, bit for bit, the integer lines at the top of DFlashForCausalLM.dflash_forward on a paged backend: +# cache_seqlens = num_ctx[:B].to(int32) +# cache_batch_idx = ctx_cache_batch_idx.to(int32) +# seq_lens_after = cache_seqlens + block_size +# kv_indptr = arange(0, (B + 1) * pages_per_slot, pages_per_slot) (int32) +# kv_last_page_len = ((seq_lens_after - 1) % page_size) + 1 (floored modulo, as torch) +# append_positions = (cache_seqlens.view(-1, 1) + arange(block_size)).reshape(-1) (int32) +# query_pos_i32 = query_positions.reshape(-1).to(int32) +# The int64 -> int32 casts keep the low 32 bits, as torch's do. One program writes every output. + +import torch +import triton +import triton.language as tl + + +@triton.jit +def _indices_kernel( + num_ctx_ptr, # [B] int32 / int64 + batch_idx_ptr, # [B] int32 / int64 + qpos_ptr, # [B, block] int32 / int64, strided + qpos_s0, + qpos_s1, + seqlens_ptr, # out [B] int32 + batch_out_ptr, # out [B] int32 + after_ptr, # out [B] int32 + indptr_ptr, # out [B + 1] int32 + last_ptr, # out [B] int32 + append_ptr, # out [B * block] int32 + qflat_ptr, # out [B * block] int32 + B, + block, + pages_per_slot, + page_size, + BLOCK_B: tl.constexpr, + BLOCK_T: tl.constexpr, +): + b = tl.arange(0, BLOCK_B) + mb = b < B + seqlens = tl.load(num_ctx_ptr + b, mask=mb, other=0).to(tl.int32) + tl.store(seqlens_ptr + b, seqlens, mask=mb) + tl.store(batch_out_ptr + b, tl.load(batch_idx_ptr + b, mask=mb, other=0).to(tl.int32), mask=mb) + after = seqlens + block + tl.store(after_ptr + b, after, mask=mb) + r = (after - 1) % page_size + r = tl.where(r < 0, r + page_size, r) # torch's floored modulo for a positive divisor + tl.store(last_ptr + b, r + 1, mask=mb) + mi = b <= B + tl.store(indptr_ptr + b, b * pages_per_slot, mask=mi) + t = tl.arange(0, BLOCK_T) + mt = t < B * block + rb = t // block + j = t - rb * block + base = tl.load(num_ctx_ptr + rb, mask=mt, other=0).to(tl.int32) + tl.store(append_ptr + t, base + j, mask=mt) + q = tl.load(qpos_ptr + rb * qpos_s0 + j * qpos_s1, mask=mt, other=0).to(tl.int32) + tl.store(qflat_ptr + t, q, mask=mt) + + +def forward_indices( + num_ctx: torch.Tensor, + batch_idx: torch.Tensor, + query_positions: torch.Tensor, + block_size: int, + pages_per_slot: int, + page_size: int, +): + """The paged dflash_forward's index prelude in one launch: (cache_seqlens, cache_batch_idx, seq_lens_after, + kv_indptr, kv_last_page_len, append_positions, query_positions_flat), all int32 and bitwise the stock lines; or + None (nothing computed) for a layout the kernel does not reproduce, so the caller keeps the stock lines.""" + ok_dtypes = (torch.int32, torch.int64) + B = num_ctx.shape[0] if num_ctx.dim() == 1 else -1 + if not ( + B >= 1 + and num_ctx.dtype in ok_dtypes + and batch_idx.dtype in ok_dtypes + and query_positions.dtype in ok_dtypes + and batch_idx.shape == (B,) + and query_positions.dim() == 2 + and tuple(query_positions.shape) == (B, block_size) + and block_size >= 1 + and pages_per_slot >= 1 + and page_size >= 1 + and num_ctx.is_cuda + ): + return None + dev = num_ctx.device + i32 = dict(dtype=torch.int32, device=dev) + seqlens = torch.empty(B, **i32) + batch_out = torch.empty(B, **i32) + after = torch.empty(B, **i32) + indptr = torch.empty(B + 1, **i32) + last = torch.empty(B, **i32) + append = torch.empty(B * block_size, **i32) + qflat = torch.empty(B * block_size, **i32) + _indices_kernel[(1,)]( + num_ctx, + batch_idx, + query_positions, + query_positions.stride(0), + query_positions.stride(1), + seqlens, + batch_out, + after, + indptr, + last, + append, + qflat, + B, + block_size, + pages_per_slot, + page_size, + BLOCK_B=triton.next_power_of_2(B + 1), + BLOCK_T=triton.next_power_of_2(B * block_size), + num_warps=1, + ) + return seqlens, batch_out, after, indptr, last, append, qflat diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_bias.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_bias.py new file mode 100644 index 000000000000..5c2cc449e3cf --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_bias.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark vanilla-Markov head: the biased block logits' glue around the bias GEMM, in two launches +# ============================================================================= +# +# It replaces, bit for bit, the elementwise lines of the batched biased-logits form (the tp targets' S4-logits path) +# prevs = cat([first_prev.view(B, 1), tokens[:, :K - 1].long()], 1).reshape(-1) +# emb = markov_prev_embeddings(prevs, w1) (get_masked_input_and_mask, F.embedding, masked_fill: an anchor +# outside [0, V) gives a zero row) +# bias = F.linear(emb, w2_shard) (unchanged: the caller's GEMM) +# out = shard_logits.float() + bias.view(B, K, N).to(float32) +# with ``prev_embeddings`` (one launch for the 11 index / mask / gather / fill launches) and ``widen_add`` (one launch +# for the two widening copies and the add). The gather copies w1's rows bit for bit (zero rows are +0.0, as +# masked_fill(0)); the add is one fp32 IEEE add of two exactly widened bf16 values, as torch's. + +import torch +import triton +import triton.language as tl + + +@triton.jit +def _prev_embeddings_kernel( + first_prev_ptr, # [B] int32 / int64 + tokens_ptr, # [B, K] int32 / int64, contiguous + w1_ptr, # [V, R] bf16, contiguous + out_ptr, # out [B * K, R] bf16 + K, + V, + R: tl.constexpr, +): + row = tl.program_id(0) + b = row // K + k = row - b * K + if k == 0: + prev = tl.load(first_prev_ptr + b).to(tl.int64) + else: + prev = tl.load(tokens_ptr + b * K + k - 1).to(tl.int64) + valid = (prev >= 0) & (prev < V) + safe = tl.where(valid, prev, 0) + c = tl.arange(0, R) + vals = tl.load(w1_ptr + safe * R + c) + vals = tl.where(valid, vals, tl.zeros_like(vals)) + tl.store(out_ptr + row.to(tl.int64) * R + c, vals) + + +@triton.jit +def _widen_add_kernel( + logits_ptr, # [M, N] bf16, contiguous + bias_ptr, # [M, N] bf16, contiguous + out_ptr, # out [M, N] fp32 + N, + BLOCK: tl.constexpr, +): + row = tl.program_id(0).to(tl.int64) + c = tl.program_id(1) * BLOCK + tl.arange(0, BLOCK) + m = c < N + a = tl.load(logits_ptr + row * N + c, mask=m, other=0.0).to(tl.float32) + bb = tl.load(bias_ptr + row * N + c, mask=m, other=0.0).to(tl.float32) + tl.store(out_ptr + row * N + c, a + bb, mask=m) + + +_INT = (torch.int32, torch.int64) +WIDEN_BLOCK = 4096 + + +def supports( + shard_logits: torch.Tensor, first_prev: torch.Tensor, tokens: torch.Tensor, w1: torch.Tensor +) -> bool: + """Whether both launches take these operands; a caller that gets False runs the in-tree lines.""" + if shard_logits.dim() != 3 or tokens.dim() != 2 or first_prev.dim() != 1 or w1.dim() != 2: + return False + B, K, _ = shard_logits.shape + R = w1.shape[1] + return ( + shard_logits.is_cuda + and shard_logits.dtype == torch.bfloat16 + and shard_logits.is_contiguous() + and w1.dtype == torch.bfloat16 + and w1.is_contiguous() + and w1.device == shard_logits.device + and R >= 16 + and (R & (R - 1)) == 0 + and R <= 1024 + and first_prev.dtype in _INT + and tokens.dtype in _INT + and first_prev.is_contiguous() + and tokens.is_contiguous() + and first_prev.numel() == B + and tuple(tokens.shape) == (B, K) + and B > 0 + and K > 0 + ) + + +def prev_embeddings( + first_prev: torch.Tensor, tokens: torch.Tensor, w1: torch.Tensor +) -> torch.Tensor: + """[B * K, R] bf16: row b * K + k is w1[prev] with prev = first_prev[b] (k = 0) or tokens[b, k - 1], and a zero + row for a prev outside [0, V) -- bitwise markov_prev_embeddings of the in-tree prevs.""" + B, K = tokens.shape + V, R = w1.shape + out = torch.empty(B * K, R, dtype=w1.dtype, device=w1.device) + _prev_embeddings_kernel[(B * K,)](first_prev, tokens, w1, out, K, V, R=R, num_warps=1) + return out + + +def widen_add(shard_logits: torch.Tensor, bias: torch.Tensor) -> torch.Tensor: + """fp32 ``shard_logits.float() + bias.to(float32)`` ([B, K, N] bf16 each, contiguous), bitwise torch's.""" + B, K, N = shard_logits.shape + if bias.dtype != torch.bfloat16 or not bias.is_contiguous() or bias.numel() != B * K * N: + raise ValueError("widen_add: bias must be contiguous bf16 with the logits' element count") + out = torch.empty(B, K, N, dtype=torch.float32, device=shard_logits.device) + grid = (B * K, triton.cdiv(N, WIDEN_BLOCK)) + _widen_add_kernel[grid](shard_logits, bias, out, N, BLOCK=WIDEN_BLOCK, num_warps=8) + return out diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain.py new file mode 100644 index 000000000000..d32a58d50c15 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain.py @@ -0,0 +1,651 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The DSpark draft head's vanilla-Markov chain over vocab-sharded draft logits, in one launch per pass. + +For every request b and block position k (greedy): ``bias = bf16(w2_shard . w1[prev])``, ``value = bf16(base[b, k] ++ bias)``, ``token[b, k]`` = the first maximum of ``value`` over the whole vocabulary (every rank's shard), ``prev = +token[b, k]``; ``prev`` for k = 0 is the anchor token. The in-tree chain gathers the full-vocabulary logits and runs +K launches of GEMV + add + argmax on every rank; this is one launch per rank, exchanging one (value, index) entry per +CTA and position over a multicast NVLink buffer. + +Shapes: the kernel splits a shard over an even number of co-resident CTAs, each owning a multiple of 64 rows (at most +``SMEM_ROW_BUDGET``). A shard that does not split so is padded: ``padded_rows`` gives the padded row count; the +caller allocates the draft logits with that last dimension (``pad_logits_buffer``, tail filled with -inf, never a +maximum) and pads ``w2_shard`` once at load (``pad_w2_shard``, zero rows). + +Compiled on the first call per shape, which must run outside CUDA-graph capture (``precompile``). The exchange +workspace is collective on its first use: every rank of the TP group calls ``workspace_args`` at the same point. + +Variants: batch-1 calls can take ``dspark_markov_chain_kernel_b1`` (``enable_batch1_variant``) and, on top of it, its +LL flag-in-payload exchange ``dspark_markov_chain_kernel_b1_ll`` (``enable_ll_exchange``); all are bitwise the same +tokens. Every rank must run the same variant for a call (the switches are process-wide; set them the same way on +every rank). +""" + +from __future__ import annotations + +import importlib +import os +import threading +from typing import Dict, Optional, Sequence, Tuple + +import torch +import torch.nn.functional as F + +from .._op import Arch, Cell, OpWrapper + +MAX_BLOCK = 8 # capacity: fixed: the workspace's positions per call +MAX_GRID = 152 # capacity: fixed: the workspace's CTAs per rank (all co-resident) +CERTIFIED_BATCH = ( + 8 # the largest batch the cells drive (the kernel takes up to the kernel module's MAX_BATCH) +) + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} +_workspaces: Dict[object, dict] = {} +_modules: Dict[str, object] = {} +_dummy: Dict[torch.device, torch.Tensor] = {} +_const_inputs: Dict[tuple, Tuple[torch.Tensor, torch.Tensor, torch.Tensor]] = {} + + +_batch1_variant = [ + False +] # S4B8: the batch-1 calls take dspark_markov_chain_kernel_b1 (see enable_batch1_variant) + + +_ll_exchange = [ + False +] # MARKOV_LL: the batch-1 calls take dspark_markov_chain_kernel_b1_ll (see enable_ll_exchange) + + +_cached_inputs = [ + False +] # TAILBK_SPEC: the kernel's read-only accepted inputs come from a per-shape cache (see enable_cached_inputs) +_cached_taken = [False] # a call took its accepted inputs from the cache (see cached_inputs_taken) + + +def _kernel_module(batch1: bool = False, ll: bool = False): + """The CuTe DSL kernel beside this file (or, for ``batch1``, its batch-1 variant; with ``ll`` too, the batch-1 + variant with the LL exchange), imported on first use (it pulls in cutlass).""" + name = ("kernel_b1_ll" if ll else "kernel_b1") if batch1 else "kernel" + mod = _modules.get(name) + if mod is None: + mod = _modules[name] = importlib.import_module(f".dspark_markov_chain_{name}", __package__) + return mod + + +_fp32_sum = [ + True +] # the in-tree chain's arithmetic: argmax on the unrounded fp32 sum (see enable_fp32_sum) + + +def enable_fp32_sum(enabled: bool = True) -> bool: + """Compare base + bias as the unrounded fp32 sum, as the in-tree chain does: its draft logits are fp32 + (LogitsProcessor's .float()) and it widens the bf16 bias, so its argmax sees fp32 sums. Off: round the sum to + bf16 first (a bf16-logits reference). On by default; set before the first call of a shape.""" + _fp32_sum[0] = bool(enabled) + return _fp32_sum[0] + + +def enable_batch1_variant(enabled: bool = True) -> bool: + """Run batch-1 calls on dspark_markov_chain_kernel_b1: the same chain with the position loop's base loads issued + before the W1 wait and the exchange polled in one first pass (bitwise the same tokens; faster at one request, + slower at more, so only batch 1 takes it). Off by default; set before the first call of a shape.""" + _batch1_variant[0] = bool(enabled) + return _batch1_variant[0] + + +def enable_ll_exchange(enabled: bool = True) -> bool: + """Run the batch-1 variant's calls on dspark_markov_chain_kernel_b1_ll: the same chain with the per-position + exchange as one 16-byte {value, tag, index, tag} multicast store per CTA (no fence, no flag word) polled and + reduced by one warp (no CTA barrier after the poll); bitwise the same tokens. Only batch-1 calls with the batch-1 + variant on take it. Off by default; set it the same way on every rank, before the first call of a shape.""" + _ll_exchange[0] = bool(enabled) + return _ll_exchange[0] + + +def enable_cached_inputs(enabled: bool = True) -> bool: + """Take the kernel's read-only accepted inputs (``num_accepted`` ones, ``accepted_rows`` arange, ``accepted`` + zeros; they only feed ``next_new``, which the op discards) from a cache filled on the first call per (device, + batch, block) outside CUDA-graph capture, instead of three fill/arange launches per call. The kernels never write + them, so the tokens are bitwise the same. Off by default; set it before the first call of a shape.""" + _cached_inputs[0] = bool(enabled) + return _cached_inputs[0] + + +def cached_inputs_taken() -> bool: + """Whether a call since process start took its accepted inputs from the cache (the engagement of + ``enable_cached_inputs``; a cache fill alone does not count).""" + return _cached_taken[0] + + +def _accepted_inputs( + batch: int, block: int, device +) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """(num_accepted ones [B], accepted_rows arange [B], accepted zeros [B, block + 1]), int32: cached under + ``enable_cached_inputs`` (filled outside capture only; a miss under capture allocates afresh, uncached).""" + key = (device, batch, block) + inputs = _const_inputs.get(key) if _cached_inputs[0] else None + if inputs is not None: + _cached_taken[0] = True + else: + inputs = ( + torch.ones(batch, dtype=torch.int32, device=device), + torch.arange(batch, dtype=torch.int32, device=device), + torch.zeros(batch, block + 1, dtype=torch.int32, device=device), + ) + if _cached_inputs[0] and not torch.cuda.is_current_stream_capturing(): + _const_inputs[key] = inputs + return inputs + + +def ll_exchange_engaged(batch: int) -> bool: + """Whether a call of ``batch`` requests runs the LL exchange kernel under the current switches.""" + return _batch1_variant[0] and _ll_exchange[0] and int(batch) == 1 + + +def _sm_count(device) -> int: + return torch.cuda.get_device_properties(device).multi_processor_count + + +def pick_grid(shard: int, block: int, batch: int, device=None) -> int: + """CTAs for a (padded) shard: 128 when supported, else the largest supported count up to the SM count; 0 if + none.""" + kern = _kernel_module() + limit = min(MAX_GRID, _sm_count(device if device is not None else torch.cuda.current_device())) + if 128 <= limit and kern.supports(shard, 128, block, batch): + return 128 + for grid in range(limit, 0, -1): + if kern.supports(shard, grid, block, batch): + return grid + return 0 + + +def padded_rows(shard: int, max_batch: int, device=None) -> int: + """The smallest row count >= ``shard`` that the kernel splits for every batch <= ``max_batch`` and block <= + MAX_BLOCK; 0 if none fits the shared-memory budget.""" + kern = _kernel_module() + step = kern.WARPS * kern.BLOCK_ROWS + rows = -(-shard // step) * step + limit = rows + MAX_GRID * step + while rows <= limit: + if all(pick_grid(rows, MAX_BLOCK, b, device) > 0 for b in (1, max_batch)): + return rows + rows += step + return 0 + + +def pad_w2_shard(w2_shard: torch.Tensor, rows: int) -> torch.Tensor: + """``w2_shard`` [S, R] bf16 with zero rows appended up to ``rows`` (done once at load).""" + out = w2_shard.new_zeros(rows, w2_shard.shape[1]) + out[: w2_shard.shape[0]].copy_(w2_shard) + return out + + +def pad_logits_buffer( + batch: int, block: int, rows: int, device, dtype=torch.bfloat16 +) -> torch.Tensor: + """A [batch, block, rows] draft-logits buffer whose padding columns hold -inf (the draft head writes [..., :S]).""" + return torch.full((batch, block, rows), float("-inf"), dtype=dtype, device=device) + + +def workspace_args(mapping, group=None) -> dict: + """This TP group's exchange workspace as the call's ``ws_*``, ``rank``, ``slots`` and ``buf_words`` arguments. + + Three rotating Lamport buffers (every word 0x80000000 when armed) and, after them, the LL variant's two regions + of the same size (zero-filled: tag 0 is never a call's tag) behind one multicast mapping, and their flag words + (the Lamport variants' [0..3], the LL variant's [4..7]); built once per (mapping, group) and reused. Collective + on first use, outside CUDA-graph capture: every rank of the TP group calls it at the same point. ``group`` + overrides the communicator (a torch.distributed group when MPI is disabled). + """ + key = (mapping, id(group) if group is not None else None) + ws = _workspaces.get(key) + if ws is not None: + return ws["args"] + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "dspark_markov_chain: the workspace must be allocated outside CUDA-graph capture" + ) + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + + kern = _kernel_module() + ll = _kernel_module(batch1=True, ll=True) + world = mapping.tp_size + buf_words = kern.buffer_words(MAX_BLOCK, kern.MAX_BATCH, world, MAX_GRID) + comm = group if group is not None else _get_mnnvl_workspace_comm(mapping) + use_fabric_handle = ( + os.environ.get("TRTLLM_FORCE_MNNVL_AR", "0") == "1" or mapping.is_multi_node() + ) + error: Optional[Exception] = None + try: + lamport_words = kern.BUFFERS * buf_words + words = ll.ll_workspace_words(buf_words) + handle = _make_mnnvl_mcast_buffer(comm, words * 4, mapping, use_fabric_handle) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + with torch.inference_mode(): + uc[:lamport_words].fill_(kern.EMPTY_WORD) + uc[lamport_words:].zero_() + flags = torch.zeros( + max(kern.FLAG_WORDS, ll.LL_FLAG_WORDS), dtype=torch.int32, device=uc.device + ) + torch.cuda.synchronize() + args = dict( + ws_uc=uc, + ws_mc=mc, + ws_flags=flags, + rank=mapping.tp_rank, + slots=world, + buf_words=buf_words, + ) + ws = dict(handle=handle, comm=comm, args=args) + except Exception as exc: # noqa: BLE001 -- reported to every rank below, then re-raised + error = exc + # Also the barrier that keeps any rank from pushing into a peer's buffer before the peer has armed it. + if not _mnnvl_workspace_all_succeeded(comm, error is None): + raise RuntimeError( + "dspark_markov_chain Lamport buffers failed on at least one rank" + ) from error + _workspaces[key] = ws + return ws["args"] + + +def _arg(t: torch.Tensor, align: int = 16): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align).mark_layout_dynamic(leading_dim=0) + + +def supports(base: torch.Tensor, w1: torch.Tensor, w2_shard: torch.Tensor) -> bool: + """Whether the kernel takes these operands (dtypes, ranks, the padded shard's split); a caller that gets False + runs the in-tree chain.""" + kern = _kernel_module() + if base.dim() != 3 or base.dtype != torch.bfloat16 or not base.is_cuda: + return False + batch, block, rows = base.shape + return ( + w1.dtype == torch.bfloat16 + and w2_shard.dtype == torch.bfloat16 + and w1.dim() == 2 + and w1.shape[1] == kern.MARKOV_RANK + and tuple(w2_shard.shape) == (rows, kern.MARKOV_RANK) + and 0 < block <= MAX_BLOCK + and 0 < batch <= kern.MAX_BATCH + and pick_grid(rows, block, batch, base.device) > 0 + ) + + +def _launch( + base: torch.Tensor, + first_prev: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + shard_offset: int, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + slots: int, + buf_words: int, + kv_lens: Optional[torch.Tensor] = None, + rewind: Optional[torch.Tensor] = None, + rewind_first: int = 0, +) -> torch.Tensor: + """One chain launch; with ``kv_lens`` and ``rewind``, also ``kv_lens[rewind_first + i] = max(kv_lens[...] - + rewind[i], 0)`` for every i (after the kernel's grid-dependency wait).""" + import cuda.bindings.driver as cuda_driver + import cutlass.cute as cute + + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + kern = _kernel_module() + if not supports(base, w1, w2_shard): + raise ValueError(f"dspark_markov_chain: unsupported operands base {tuple(base.shape)} {base.dtype}, " + f"w2_shard {tuple(w2_shard.shape)}") # fmt: skip + if first_prev.dtype != torch.int64 or first_prev.numel() != base.shape[0]: + raise ValueError("dspark_markov_chain: first_prev must be int64 [B]") + batch, block, rows = base.shape + batch1 = _batch1_variant[0] and batch == 1 + ll = batch1 and _ll_exchange[0] + grid = pick_grid(rows, block, batch, base.device) + if kern.buffer_words(block, batch, slots, grid) > buf_words: + raise ValueError("dspark_markov_chain: the exchange does not fit the workspace buffers") + if ll: + ll_kern = _kernel_module(batch1=True, ll=True) + if ( + ll_kern.ll_region_words(block, slots, grid) > buf_words + or ws_uc.numel() < ll_kern.ll_workspace_words(buf_words) + or ws_flags.numel() < ll_kern.LL_FLAG_WORDS + ): + raise ValueError("dspark_markov_chain: the LL exchange does not fit the workspace") + device = base.device + tokens = torch.empty(batch, block, dtype=torch.int32, device=device) + next_new = torch.empty(batch, block + 1, dtype=torch.int32, device=device) + dummy = _dummy.get(device) + if dummy is None: + # The kernel's optional outputs and inputs (corrected logits, accepted rows, KV-length rewind) are unused + # here: one zero word stands for each, and an empty rewind leaves it untouched. + dummy = _dummy[device] = torch.zeros(4, dtype=torch.int32, device=device) + rewind_count = 0 if rewind is None else int(rewind.numel()) + if rewind_count: + if not supports_rewind(kv_lens, rewind, rewind_first): + raise ValueError( + "dspark_markov_chain: unsupported KV-length rewind (see supports_rewind())" + ) + else: + kv_lens, rewind = dummy, dummy + nacc, rows_idx, acc = _accepted_inputs(batch, block, device) + args = ( + _arg(base.contiguous().view(-1).view(torch.int16)), + _arg(first_prev.contiguous().view(-1), align=8), + _arg(w1.contiguous().view(-1).view(torch.int32)), + _arg(w2_shard.contiguous().view(-1).view(torch.int32)), + _arg(dummy.view(torch.float32)), + _arg(tokens.view(-1)), + _arg(next_new.view(-1)), + _arg(acc.view(-1), align=4), + _arg(nacc, align=4), + _arg(rows_idx, align=4), + _arg(kv_lens, align=4), + _arg(rewind, align=4), + _arg(ws_uc.view(-1)), + _arg(ws_mc.view(-1)), + _arg(ws_flags.view(-1)), + ) + scalars = ( + int(w1.shape[0]), + int(shard_offset), + int(rank), + int(block + 1), + int(rewind_first), + rewind_count, + ) + use_pdl = get_env_enable_pdl() + stream = cuda_driver.CUstream(torch.cuda.current_stream(device).cuda_stream) + consts = (grid, rows // grid, rows, block, batch, slots, 1, buf_words, True, _fp32_sum[0]) + key = consts + (use_pdl, batch1, ll) + fn = _compiled.get(key) + if fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "trtllm::dspark_markov_chain must run once per shape outside CUDA-graph capture first" + ) + with _lock: + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute.compile( + _kernel_module(batch1, ll).k3_markov, *args, *scalars, *consts, use_pdl, stream + ) + fn(*args, *scalars, stream) + return tokens + + +@torch.library.custom_op("trtllm::dspark_markov_chain", mutates_args=()) +def _dspark_markov_chain( + base: torch.Tensor, + first_prev: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + shard_offset: int, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + slots: int, + buf_words: int, +) -> torch.Tensor: + return _launch( + base, first_prev, w1, w2_shard, shard_offset, ws_uc, ws_mc, ws_flags, rank, slots, buf_words + ) + + +@_dspark_markov_chain.register_fake +def _(base, first_prev, w1, w2_shard, shard_offset, ws_uc, ws_mc, ws_flags, rank, slots, buf_words): + return base.new_empty((base.shape[0], base.shape[1]), dtype=torch.int32) + + +@torch.library.custom_op("trtllm::dspark_markov_chain_rewind", mutates_args=("kv_lens",)) +def _dspark_markov_chain_rewind( + base: torch.Tensor, + first_prev: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + shard_offset: int, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + slots: int, + buf_words: int, + kv_lens: torch.Tensor, + rewind: torch.Tensor, + rewind_first: int, +) -> torch.Tensor: + return _launch( + base, first_prev, w1, w2_shard, shard_offset, ws_uc, ws_mc, ws_flags, rank, slots, buf_words, + kv_lens, rewind, rewind_first, + ) # fmt: skip + + +@_dspark_markov_chain_rewind.register_fake +def _(base, first_prev, w1, w2_shard, shard_offset, ws_uc, ws_mc, ws_flags, rank, slots, buf_words, kv_lens, + rewind, rewind_first): # fmt: skip + return base.new_empty((base.shape[0], base.shape[1]), dtype=torch.int32) + + +def supports_rewind( + kv_lens: Optional[torch.Tensor], rewind: Optional[torch.Tensor], rewind_first: int +) -> bool: + """Whether a chain call can also apply the KV-length rewind ``kv_lens[first + i] = max(kv_lens[first + i] - + rewind[i], 0)``: int32 contiguous CUDA operands on one device, 1..``kernel MAX_BATCH`` amounts, in range. A caller + that gets False applies the rewind itself.""" + if kv_lens is None or rewind is None: + return False + n = rewind.numel() + return ( + kv_lens.is_cuda + and rewind.is_cuda + and kv_lens.device == rewind.device + and kv_lens.dtype == torch.int32 + and rewind.dtype == torch.int32 + and kv_lens.is_contiguous() + and rewind.is_contiguous() + and 0 < n <= _kernel_module().MAX_BATCH + and 0 <= int(rewind_first) + and int(rewind_first) + n <= kv_lens.numel() + ) + + +def dspark_markov_chain_rewind( + base: torch.Tensor, + first_prev: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + shard_offset: int, + kv_lens: torch.Tensor, + rewind: torch.Tensor, + rewind_first: int, + *, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + slots: int, + buf_words: int, +) -> torch.Tensor: + """``dspark_markov_chain``'s tokens (the same launch, bitwise the same tokens) plus the target's KV-length rewind + after the draft forward in the same launch (TAILBK_SPEC site H): ``kv_lens[rewind_first + i] = + max(kv_lens[rewind_first + i] - rewind[i], 0)``, applied after the kernel's grid-dependency wait (every reader of + ``kv_lens`` in the draft forward finished). Check ``supports_rewind`` first; the call is a collective as + ``dspark_markov_chain``'s, and each rank rewinds its own ``kv_lens``. The kernel module's compile is shared with + ``dspark_markov_chain`` (the rewind's operands are runtime arguments).""" + return torch.ops.trtllm.dspark_markov_chain_rewind( + base, first_prev, w1, w2_shard, int(shard_offset), ws_uc, ws_mc, ws_flags, int(rank), int(slots), + int(buf_words), kv_lens, rewind, int(rewind_first), + ) # fmt: skip + + +def precompile( + mapping, + rows: int, + w1: torch.Tensor, + w2_shard: torch.Tensor, + shard_offset: int, + blocks: Tuple[int, ...], + max_batch: int, + group=None, +) -> None: + """Compile every (batch <= ``max_batch``, block in ``blocks``) shape outside capture (collective: every rank).""" + ws = workspace_args(mapping, group) + for block in blocks: + for batch in range(1, max_batch + 1): + base = pad_logits_buffer(batch, block, rows, w2_shard.device) + prev = torch.zeros(batch, dtype=torch.long, device=w2_shard.device) + dspark_markov_chain(base, prev, w1, w2_shard, shard_offset, **ws) + torch.cuda.synchronize() + + +class DsparkMarkovChain(OpWrapper): + """Greedy draft tokens [B, K] (int32) of the DSpark vanilla-Markov head, from this rank's padded vocab shard + of the block logits. + + ``base`` [B, K, rows] bf16: this rank's shard of the draft head's block logits (columns ``[0, shard)``), the + padding columns -inf. ``first_prev`` [B] int64: the anchor tokens. ``w1`` [V, 256] bf16: ``markov_w1``, whole. + ``w2_shard`` [rows, 256] bf16: this rank's rows of ``markov_w2``, zero-padded. ``shard_offset``: the vocabulary + index of this rank's first column. ``ws_*``, ``rank``, ``slots``, ``buf_words``: ``workspace_args``'s. + + Every rank returns the same tokens. The call is a collective: every rank of the group calls it with the same + (B, K), in the same order. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # MiniMax-M3.1's DSpark drafter: vocabulary 200064, Markov rank 256, K = 3 and 7 draft positions, the decode + # batches the M3.1 targets serve with it (1..8). TP4 is one GB300 tray, TP8 two. + _M31 = dict(vocab=200064, batches=tuple(range(1, CERTIFIED_BATCH + 1))) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="M3.1 TP4, K = 3: the 50016-row shard, padded to 50688", + spec=dict(world=4, block=3, **_M31), + ), + Cell( + why="M3.1 TP4, K = 7: seven chained exchanges per call", + spec=dict(world=4, block=7, **_M31), + ), + Cell( + why="a shard that splits without padding (32768 / 4 = 8192 rows)", + spec=dict(world=4, block=3, vocab=32768, batches=(1, 8)), + ), + Cell( + why="M3.1 TP8 over two trays, K = 3: the 25008-row shard, padded to 25088; fabric-handle multicast", + spec=dict(world=8, block=3, **_M31), + ), + Cell(why="M3.1 TP8 over two trays, K = 7", spec=dict(world=8, block=7, **_M31)), + ) + + note = """ + Greedy only, and only the vanilla Markov head: no d2t map, no candidate selector, no sampling. A caller with + any of those runs the in-tree chain. + + The kernel takes batches up to its module's MAX_BATCH (16) and blocks up to MAX_BLOCK (8); the cells certify + batches 1..8. Measured: at 16 requests the one-launch chain is 0.93x the in-tree chain's time, so the M3.1 + targets serve batches up to 8 only. + + The kernel also computes next_new_tokens, the accepted-row bookkeeping and the target's KV-length rewind of + the K3 fused verify; this entry passes none of them (one dummy word each) and returns only the tokens. + """ + + def __call__( + self, + base: torch.Tensor, + first_prev: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + shard_offset: int, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + slots: int, + buf_words: int, + ) -> torch.Tensor: + return torch.ops.trtllm.dspark_markov_chain( + base, + first_prev, + w1, + w2_shard, + int(shard_offset), + ws_uc, + ws_mc, + ws_flags, + int(rank), + int(slots), + int(buf_words), + ) + + def reference( + self, + base_in_rank_order: Sequence[torch.Tensor], + first_prev: torch.Tensor, + w1: torch.Tensor, + w2_shard_in_rank_order: Sequence[torch.Tensor], + shard: int, + ) -> torch.Tensor: + """The DSpark worker's in-tree chain on the gathered full-vocabulary logits: per position ``bias = + F.linear(w1[prev], w2)`` in bf16, ``logits + bias`` in the logits' dtype, ``torch.argmax`` (first maximum). + With fp32_sum (default) the logits are widened to fp32 first, as LogitsProcessor's ``.float()`` hands them + to the in-tree chain, so the bias is widened and the sum is not rounded; without it, bf16 logits + bf16 bias. + + Not more accurate than the in-tree chain: the tokens must equal its tokens bit for bit, so this is that + chain. The documented collective exception to mirroring ``__call__``: it takes every rank's ``base`` and + ``w2_shard`` in rank order, and ``shard``, the unpadded columns of each. + """ + full = torch.cat([b[..., :shard] for b in base_in_rank_order], dim=-1) + if _fp32_sum[0]: + full = full.float() + w2 = torch.cat([w[:shard] for w in w2_shard_in_rank_order], dim=0) + prev = first_prev + tokens = [] + for k in range(full.shape[1]): + bias = F.linear(w1[prev], w2).to(full.dtype) + prev = torch.argmax(full[:, k] + bias, dim=-1) + tokens.append(prev) + return torch.stack(tokens, 1).to(torch.int32) + + def is_valid( + self, + base: torch.Tensor, + first_prev: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + shard_offset: int, + ws_uc: torch.Tensor, + ws_mc: torch.Tensor, + ws_flags: torch.Tensor, + rank: int, + slots: int, + buf_words: int, + ) -> None: + # The kernel takes every column of `base` as a candidate and offsets its index by `shard_offset`: a padding + # column that is not -inf can win and return an index past this rank's shard, and a wrong offset returns + # another rank's token ids. Neither is checked on the way to the kernel. + vocab = w1.shape[0] + assert vocab % slots == 0, (vocab, slots) + shard = vocab // slots + assert shard_offset == rank * shard, (shard_offset, rank, shard) + assert base.shape[-1] >= shard, (base.shape, shard) + assert bool(torch.isneginf(base[..., shard:]).all()), ( + "the padding columns of base must hold -inf" + ) + # An anchor outside the vocabulary indexes `w1` past its end inside the kernel. + assert bool(((first_prev >= 0) & (first_prev < vocab)).all()), ( + "first_prev outside [0, vocab)" + ) + + +dspark_markov_chain = DsparkMarkovChain() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel.py new file mode 100644 index 000000000000..a6bdf3861a22 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel.py @@ -0,0 +1,583 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark vanilla-Markov draft chain for a vocab-sharded draft head -- CTM (prims/cute) kernel +# ============================================================================= +# +# For every request b and block position k = 0 .. K-1 (greedy, temperature 0): +# bias[v] = bf16(markov_w2[v] . markov_w1[prev]) v in this rank's vocab shard +# value[v] = bf16(base[b, k, v] + bias[v]) (the reference's bf16 add: fp32 sum, one rounding) +# token[b, k] = the first maximum of value over the whole vocabulary (all ranks) +# prev = token[b, k] (prev for k = 0 is the anchor, first_prev[b]) +# so the K positions are a chain of dependent global argmaxes across the tensor-parallel ranks. +# +# Numerics are dsparkMarkovChainKernel's (the C++ kernel this replaces): lane l of a warp sums the 8 columns +# 8l .. 8l+7 of a row by FMAs from 0, the 32 lane sums are added by the xor tree 16, 8, 4, 2, 1, the sum is rounded +# to bf16 (cvt.rn), added to the base in fp32 and the sum rounded to bf16 (cvt.rn), as the reference chain's bf16 +# logits + bf16 bias does. Argmax order: larger value, then lower vocabulary index; NaN never +# wins. The order is total, so every split of the reduction gives the same token. +# +# Geometry: G CTAs of 256 threads, CTA c owning the RC = S / G rows [c RC, c RC + RC) of the shard (RC a multiple +# of 64). Warp w owns the RC / 8 contiguous rows [w RC / 8, ...), in blocks of 8 (row j's sum ends on lane 4j after +# the reduce-scatter form of the xor tree). The CTA's rows of markov_w2 are bulk-copied into shared memory before the +# grid-dependency wait (a weight) and stay there for all K positions. markov_w1[prev] (512 bytes per request) is one +# bulk copy per CTA and position into shared memory, issued by the thread that reduces request b's global argmax, on +# one phase of an mbarrier per position. +# +# Exchange (one NVLink hop per position): every CTA reduces its rows to one (value, index) entry per request and +# multicast-stores it into slot [k][b][rank][c] of every rank's Lamport buffer; every CTA then polls all +# W x G entries of [k][b] in its own rank's copy (the data is the ready signal: 0x80000000 = empty; values carry +# -0.0 as +0.0 and indices are non-negative, so no entry is the sentinel) and reduces them with the same order. +# Every CTA of every rank computes the same global argmax: no leader, no grid barrier, no publish word. +# +# Buffers: 3 rotating Lamport buffers. Call n uses buffer n % 3 and re-arms buffer (n - 1) % 3 (its readers, all +# CTAs of call n - 1, finished before this call's grid wait returned; its next writer is a remote rank's call n + 2, +# which needs this rank's call n + 1 pushes). flags: [0] buffer of this call, [1] CTAs arrived, [2] words the +# previous call used (to re-arm), [3] unused. The last CTA to arrive advances them. +# +# Roles: warp 0's elected lane issues the weight bulk copy (and the early dependent trigger); thread b < B pushes +# request b's CTA entry, reduces its global argmax and requests the next markov_w1 row; all 8 warps compute and poll +# (the chain inside a position is serial, and the poll is spread over all 256 threads: one load wave). Per position: +# rows → barrier → push → poll → barrier → (thread b) argmax, next row; no barrier between the argmax and the next +# position (its readers wait for the row's phase). +# ============================================================================= +"""CTM DSpark Markov chain: the draft head's greedy intra-block Markov bias, the per-position global argmax +across the tensor-parallel vocab shards (the draft tokens), next_new_tokens and the target's KV-length rewind after +the draft forward, in one launch.""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +THREADS = 256 +WARPS = THREADS // 32 +MARKOV_RANK = 256 # 32 lanes x 8 columns +ROW_WORDS = MARKOV_RANK // 2 # int32 words (bf16 pairs) per markov row +ROW_BYTES = MARKOV_RANK * 2 +BLOCK_ROWS = 8 # rows per warp block: one per epilogue lane +MAX_BATCH = 16 +MAX_BLOCK = 16 +BUFFERS = 3 +FLAG_WORDS = 4 +EMPTY_WORD = -(2**31) # 0x80000000: fp32 -0.0, never a pushed word +INT_MAX = 2**31 - 1 +NEG_INF_BITS = -8388608 # 0xFF800000 +COPY_ROWS = 32 # rows per weight bulk copy (16 KB) +SMEM_ROW_BUDGET = 400 # 200 KB of markov_w2 rows per CTA +SMEM_BYTES = 227 * 1024 + + +def rows_per_cta(shard: int, grid: int) -> int: + return shard // grid + + +def supports(shard: int, grid: int, block: int, batch: int) -> bool: + """Shapes the kernel runs: whole 8-row blocks for every warp (RC a multiple of 64), the CTA's markov_w2 rows + within the shared-memory budget, block <= 16, batch <= 16, an even entry count per exchange.""" + if grid <= 0 or shard % grid != 0: + return False + rc = rows_per_cta(shard, grid) + return ( + rc % (WARPS * BLOCK_ROWS) == 0 + and rc <= SMEM_ROW_BUDGET + and smem_bytes(rc, batch) <= SMEM_BYTES + and grid % 2 == 0 + and 0 < block <= MAX_BLOCK + and 0 < batch <= MAX_BATCH + ) + + +def smem_bytes(rows: int, batch: int) -> int: + """Shared memory of a CTA: its markov_w2 rows, the markov_w1[prev] rows, the two [b][warp] reductions, the + barriers, flags and prev (with the allocator's alignment padding).""" + return rows * ROW_BYTES + batch * ROW_BYTES + 2 * batch * WARPS * 8 + 1024 + + +def buffer_words(block: int, batch: int, slots: int, grid: int) -> int: + """Int32 words of one Lamport buffer: one (value, index) entry per position, request, rank slot and CTA.""" + return block * batch * slots * grid * 2 + + +@dsl_user_op +def _atomic_add_acq_rel(addr_i64, val, *, loc=None, ip=None): + """atom.acq_rel.gpu.global.add.u32, returning the old value.""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [addr_i64.ir_value(loc=loc, ip=ip), val.ir_value(loc=loc, ip=ip)], + "atom.acq_rel.gpu.global.add.u32 $0, [$1], $2;", "=r,l,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + """fma.rn.f32 (fmaf).""" + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [a.ir_value(loc=loc, ip=ip), b.ir_value(loc=loc, ip=ip), c.ir_value(loc=loc, ip=ip)], + "fma.rn.f32 $0, $1, $2, $3;", "=f,f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + """add.rn.f32: an fp32 add the compiler cannot contract with a neighbouring multiply.""" + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [a.ir_value(loc=loc, ip=ip), b.ir_value(loc=loc, ip=ip)], + "add.rn.f32 $0, $1, $2;", "=f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _pack_bf16x2(hi, lo, *, loc=None, ip=None): + """(bf16(hi) << 16) | bf16(lo), round to nearest even (cvt.rn, as __float2bfloat16_rn).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [hi.ir_value(loc=loc, ip=ip), lo.ir_value(loc=loc, ip=ip)], + "cvt.rn.bf16x2.f32 $0, $1, $2;", "=r,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _lo_f32(word): + """The bf16 in the low half of a word, as fp32 (exact).""" + return cutlass.Int32(word << cutlass.Int32(16)).bitcast(cutlass.Float32) + + +def _hi_f32(word): + """The bf16 in the high half of a word, as fp32 (exact).""" + return cutlass.Int32(word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + + +def _better(value, index, best_value, best_index): + """dsparkMarkovChainKernel's keepBetter order: larger value, or an equal value at a lower index. NaN never + compares better, so it never wins.""" + return (value > best_value) | ((value == best_value) & (index < best_index)) + + +def _keep_better(best_value, best_index, value, index): + take = _better(value, index, best_value, best_index) + return cutlass.Float32(cutlass.select_(take, value, best_value)), cutlass.Int32( + cutlass.select_(take, index, best_index) + ) + + +def _warp_best(value, index): + """Every lane gets the warp's best (value, index) (xor tree: the order is total, so all lanes agree).""" + for offset in [16, 8, 4, 2, 1]: + other_value = cute.arch.shuffle_sync_bfly(value, offset=offset, mask=-1, mask_and_clamp=31) + other_index = cute.arch.shuffle_sync_bfly(index, offset=offset, mask=-1, mask_and_clamp=31) + value, index = _keep_better(value, index, other_value, other_index) + return value, index + + +def _block_dot(s_w2, first_row, lane, w1f): + """markov_w2[first_row + j] . markov_w1[prev] for the block's 8 rows, in the C++ kernel's order: lane l sums the + 8 columns 8l .. 8l+7 by an FMA chain from 0, then the 32 lane sums are added in pairs at lane distances 16, 8, 4, 2, + 1 (the xor tree). Done as a reduce-scatter: at distances 16, 8 and 4 a lane keeps half of its rows and adds its + partner's partial of each kept row (the pairs of the full tree, so the same sums: 9 shuffles instead of 40), then + the full tree at 2 and 1. Returns row (lane >> 2)'s sum.""" + parts = [] + for r in range(BLOCK_ROWS): + w2v = s_w2.load(idx=(first_row + cutlass.Int32(r)) * cutlass.Int32(ROW_WORDS) + lane * cutlass.Int32(4), + vector_size=4, alignment=16) # fmt: skip + acc = cutlass.Float32(0.0) + for q in range(4): + word = cutlass.Int32(w2v[q]) + acc = _fma_rn(w1f[2 * q], _lo_f32(word), acc) + acc = _fma_rn(w1f[2 * q + 1], _hi_f32(word), acc) + parts.append(acc) + for offset in [16, 8, 4]: + upper = (lane & cutlass.Int32(offset)) != cutlass.Int32(0) + half = len(parts) // 2 + kept = [] + for i in range(half): + send = cutlass.Float32(cutlass.select_(upper, parts[i], parts[half + i])) + keep = cutlass.Float32(cutlass.select_(upper, parts[half + i], parts[i])) + kept.append( + _add_rn( + keep, + cute.arch.shuffle_sync_bfly(send, offset=offset, mask=-1, mask_and_clamp=31), + ) + ) + parts = kept + dot = parts[0] + for offset in [2, 1]: + dot = _add_rn( + dot, cute.arch.shuffle_sync_bfly(dot, offset=offset, mask=-1, mask_and_clamp=31) + ) + return dot + + +def _fetch_w1_row(s_w1, s_valid, w1, mbar, b, prev, vocab): + """Request b's markov_w1[prev] row into shared memory: one 512-byte bulk copy on the position's barrier phase (a + prev outside the vocabulary copies row 0 and clears s_valid[b]: the readers zero its bias). s_valid[b] is stored + before the arrive, whose release the readers' phase wait acquires.""" + valid = (prev >= cutlass.Int32(0)) & (prev < vocab) + row = cutlass.Int32(cutlass.select_(valid, prev, cutlass.Int32(0))) + s_valid.store(cutlass.Int32(cutlass.select_(valid, cutlass.Int32(1), cutlass.Int32(0))), idx=b) + prims.mbarrier_arrive_expect_tx(mbar, ROW_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_w1.subview(b * cutlass.Int32(ROW_WORDS)), + w1.subview(row * cutlass.Int32(ROW_WORDS)), + mbar, + ROW_BYTES, + ) + + +def _reduce_slots(slots_arr, first_word): + """The best of 8 (value bits, index) pairs at first_word .. in warp order.""" + value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + index = cutlass.Int32(INT_MAX) + for w in range(WARPS): + pair = slots_arr.load(idx=first_word + cutlass.Int32(w * 2), vector_size=2, alignment=8) + value, index = _keep_better( + value, index, cutlass.Int32(pair[0]).bitcast(cutlass.Float32), cutlass.Int32(pair[1]) + ) + return value, index + + +@cute.kernel +def k3_markov_kernel( + base: cutlass.Array, # fp32 [B * K * S] (or its int16 bf16 view with base_bf16): the draft head's shard logits + first_prev: cutlass.Array, # int64 [B]: the anchor token of each request + w1: cutlass.Array, # int32 words of bf16 markov_w1 [V, 256] + w2: cutlass.Array, # int32 words of bf16 markov_w2[shard rows] [S, 256] + corrected: cutlass.Array, # unused (kept for the launch signature) + tokens: cutlass.Array, # int32 [B * K], out + next_new: cutlass.Array, # int32 [B * (K + 1)], out + accepted: cutlass.Array, # int32 [*, accepted_stride]: accepted tokens + num_accepted: cutlass.Array, # int32 [B] + accepted_rows: cutlass.Array, # int32 [B]: the row of accepted of each request + kv_lens: cutlass.Array, # int32: the target's KV lengths, rewound here after the draft forward + rewind: cutlass.Array, # int32 [rewind_count]: subtracted from kv_lens[rewind_first ...], clamped at 0 + buf_uc: cutlass.Array, # int32 words: this rank's Lamport buffers + buf_mc: cutlass.Array, # int32 words: their multicast mapping + flags: cutlass.Array, # int32 [4] + vocab: cutlass.Int32, # rows of markov_w1: a prev outside [0, vocab) gets a zero bias + vocab_offset: cutlass.Int32, # first vocabulary index of this rank's shard + rank: cutlass.Int32, + accepted_stride: cutlass.Int32, + rewind_first: cutlass.Int32, + rewind_count: cutlass.Int32, # 0: no rewind + grid: cutlass.Constexpr[int], + rows: cutlass.Constexpr[int], # RC + shard: cutlass.Constexpr[int], # S + block: cutlass.Constexpr[int], # K + batch: cutlass.Constexpr[int], # B + slots: cutlass.Constexpr[int], # rank slots per (k, b): world x push_copies + push_copies: cutlass.Constexpr[ + int + ], # test only: each rank's entry goes into push_copies consecutive slots + buf_words: cutlass.Constexpr[int], + base_bf16: cutlass.Constexpr[bool], + fp32_sum: cutlass.Constexpr[bool], +): + rows_per_warp = rows // WARPS + blocks_per_warp = rows_per_warp // BLOCK_ROWS + entries = slots * grid # (value, index) entries per (k, b) + vectors = entries // 2 # 16-byte loads per (k, b) + vec_per_thread = (vectors + THREADS - 1) // THREADS + used_words = block * batch * entries * 2 + + tx, _, _ = cute.arch.thread_idx() + cta, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tx % cutlass.Int32(32) + + s_w2 = cutlass.Array( + cutlass.Int32, rows * ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128 + ) + # [b]: markov_w1[prev] of the position (one bulk copy per request, issued by the thread that learns prev). + s_w1 = cutlass.Array( + cutlass.Int32, batch * ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128 + ) + s_mbar = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + s_mbar_w1 = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + # [b][warp] (value bits, index): the CTA reduction of the rows, then of the polled entries. + s_row_best = cutlass.Array( + cutlass.Int32, batch * WARPS * 2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_poll_best = cutlass.Array( + cutlass.Int32, batch * WARPS * 2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_flags = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_valid = cutlass.Array(cutlass.Int32, MAX_BATCH, space=cutlass.AddressSpace.smem, alignment=16) + + if warp == 0: + if prims.elect_sync(): + prims.mbarrier_init(s_mbar, 1) + prims.mbarrier_init(s_mbar_w1, batch) + prims.fence_mbarrier_init() + prims.barrier_cta_sync(0) + + # The CTA's markov_w2 rows: a weight, loaded before the grid dependency. + if warp == 0: + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(s_mbar, rows * ROW_BYTES) + for c in cutlass.range_constexpr((rows + COPY_ROWS - 1) // COPY_ROWS): + n = min(COPY_ROWS, rows - c * COPY_ROWS) + prims.cp_async_bulk_shared_cluster_global( + s_w2.subview(c * COPY_ROWS * ROW_WORDS), + w2.subview( + cta * cutlass.Int32(rows * ROW_WORDS) + + cutlass.Int32(c * COPY_ROWS * ROW_WORDS) + ), + s_mbar, + n * ROW_BYTES, + ) + # Dependents wait for this whole grid before reading its outputs. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + + prims.griddepcontrol(prims.GridDepAction.WAIT) + if tx == 0: + s_flags.store(flags.load(idx=0, is_volatile=True), idx=0) + s_flags.store(flags.load(idx=2, is_volatile=True), idx=2) + if tx < cutlass.Int32(batch): + _fetch_w1_row( + s_w1, s_valid, w1, s_mbar_w1, tx, cutlass.Int32(first_prev.load(idx=tx)), vocab + ) + prims.barrier_cta_sync(0) + cur = s_flags.load(idx=0) + cur_base = cur * cutlass.Int32(buf_words) + + # Re-arm the previous call's buffer (this CTA's 16-byte chunks of the words it used). + dirty_base = ((cur + cutlass.Int32(2)) % cutlass.Int32(BUFFERS)) * cutlass.Int32(buf_words) + dirty_words = s_flags.load(idx=2) + empty = cutlass.Int32(EMPTY_WORD) + w_clear = (cta * cutlass.Int32(THREADS) + tx) * cutlass.Int32(4) + while w_clear < dirty_words: + buf_uc.store((empty, empty, empty, empty), idx=dirty_base + w_clear, alignment=16) + w_clear += cutlass.Int32(grid * THREADS * 4) + + # next_new_tokens[b] = [accepted[row_b, num_accepted[b] - 1], tokens[b, :]] (-1 wraps, as torch indexing). + if cta == 0: + if tx < cutlass.Int32(batch): + col = num_accepted.load(idx=tx) - cutlass.Int32(1) + col = cutlass.Int32(cutlass.select_(col < cutlass.Int32(0), col + accepted_stride, col)) + row = accepted_rows.load(idx=tx) + next_new.store( + accepted.load(idx=row * accepted_stride + col), idx=tx * cutlass.Int32(block + 1) + ) + # The KV-length rewind after the draft forward (whose kernels, the readers of kv_lens, all finished before + # this grid's wait): kv_lens[first + b] = max(kv_lens[first + b] - rewind[b], 0). + if tx < rewind_count: + left = kv_lens.load(idx=rewind_first + tx) - rewind.load(idx=tx) + kv_lens.store(cutlass.Int32(cutlass.select_(left < cutlass.Int32(0), cutlass.Int32(0), left)), + idx=rewind_first + tx) # fmt: skip + + # The weights have landed (issued before the wait). + while not cute.arch.mbarrier_test_wait(s_mbar.data_ptr(), 0): + pass + + row0 = warp * cutlass.Int32(rows_per_warp) # first row of the warp within the CTA + shard_row0 = cta * cutlass.Int32(rows) + row0 # ... within the shard + for k in cutlass.range(block, unroll=1): + for b in cutlass.range(batch, unroll=1): + base_row = (b * cutlass.Int32(block) + k) * cutlass.Int32(shard) + # Lane 4j finishes row j of each 8-row block (the reduce-scatter leaves row j's sum on lanes 4j .. 4j+3): + # the warp's base logits, in flight during the wait below. + has_row = (lane & cutlass.Int32(3)) == cutlass.Int32(0) + lane_row = shard_row0 + (lane >> cutlass.Int32(2)) + base_vals = [] + for g in cutlass.range_constexpr(blocks_per_warp): + if cutlass.const_expr(base_bf16): + base_vals.append( + cutlass.Int32( + cutlass.Int32( + base.load(idx=base_row + lane_row + cutlass.Int32(g * BLOCK_ROWS)) + ) + << cutlass.Int32(16) + ).bitcast(cutlass.Float32) + ) + else: + base_vals.append( + base.load(idx=base_row + lane_row + cutlass.Int32(g * BLOCK_ROWS)) + ) + # markov_w1[prev] landed (the position's phase of the W1 barrier). + while not cute.arch.mbarrier_test_wait(s_mbar_w1.data_ptr(), k & cutlass.Int32(1)): + pass + valid = s_valid.load(idx=b) != cutlass.Int32(0) + # markov_w1[prev], columns 8 lane .. 8 lane + 7, as fp32 (zeros for an invalid prev). + w1v = s_w1.load( + idx=b * cutlass.Int32(ROW_WORDS) + lane * 4, vector_size=4, alignment=16 + ) + w1f = [] + for q in cutlass.range_constexpr(4): + word = cutlass.Int32( + cutlass.select_(valid, cutlass.Int32(w1v[q]), cutlass.Int32(0)) + ) + w1f.append(_lo_f32(word)) + w1f.append(_hi_f32(word)) + best_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + best_index = cutlass.Int32(INT_MAX) + for g in cutlass.range_constexpr(blocks_per_warp): + dot = _block_dot(s_w2, row0 + cutlass.Int32(g * BLOCK_ROWS), lane, w1f) + bias = _lo_f32(_pack_bf16x2(dot, dot)) + # fp32_sum (default): the in-tree chain's arithmetic. Its logits are fp32 (LogitsProcessor's + # .float()) and the bf16 bias is widened, so the argmax compares the unrounded fp32 sum. Without + # fp32_sum: a bf16 reference (bf16 logits + bf16 bias in bf16) rounds that sum once. The corrected + # logits are not stored (the chain's consumers read only the tokens). + if cutlass.const_expr(fp32_sum): + value = _add_rn(base_vals[g], bias) + else: + value = _lo_f32( + _pack_bf16x2(_add_rn(base_vals[g], bias), _add_rn(base_vals[g], bias)) + ) + if has_row: + my_row = lane_row + cutlass.Int32(g * BLOCK_ROWS) + best_value, best_index = _keep_better( + best_value, best_index, value, vocab_offset + my_row + ) + best_value, best_index = _warp_best(best_value, best_index) + if lane == 0: + s_row_best.store((best_value.bitcast(cutlass.Int32), best_index), + idx=(b * cutlass.Int32(WARPS) + warp) * cutlass.Int32(2), alignment=8) # fmt: skip + prims.barrier_cta_sync(0) + # Thread b pushes request b's CTA maximum into every rank's buffer. + if tx < cutlass.Int32(batch): + cta_value, cta_index = _reduce_slots(s_row_best, tx * cutlass.Int32(WARPS * 2)) + bits = cta_value.bitcast(cutlass.Int32) + bits = cutlass.Int32( + cutlass.select_(bits == cutlass.Int32(EMPTY_WORD), cutlass.Int32(0), bits) + ) + for c in cutlass.range_constexpr(push_copies): + s_idx = rank * cutlass.Int32(push_copies) + cutlass.Int32(c) + word = cur_base + (((k * cutlass.Int32(batch) + tx) * cutlass.Int32(slots) + s_idx) + * cutlass.Int32(grid) + cta) * cutlass.Int32(2) # fmt: skip + buf_mc.store((bits, cta_index), idx=word, alignment=8) + # Drain the posted multicast store now (the polls below issue no release that would). + prims.fence_acq_rel(prims.MemScope.CLUSTER) + # Poll every rank's entries of [k][b] (this thread's 16-byte vectors, all loads in flight together) until + # none is empty; the pass that finds them all complete also reduces them. + for b in cutlass.range(batch, unroll=1): + region = cur_base + (k * cutlass.Int32(batch) + b) * cutlass.Int32(entries * 2) + best_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + best_index = cutlass.Int32(INT_MAX) + pending = cutlass.Boolean(True) + while pending: + pass_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + pass_index = cutlass.Int32(INT_MAX) + pass_pending = cutlass.Boolean(False) + for i in cutlass.range_constexpr(vec_per_thread): + v_idx = tx + cutlass.Int32(i * THREADS) + live = v_idx < cutlass.Int32(vectors) + got = buf_uc.load( + idx=region + + cutlass.Int32(cutlass.select_(live, v_idx, cutlass.Int32(0))) + * cutlass.Int32(4), + vector_size=4, + alignment=16, + is_volatile=True, + ) + for e in cutlass.range_constexpr(2): + vw = cutlass.Int32(got[2 * e]) + iw = cutlass.Int32(got[2 * e + 1]) + pass_pending = pass_pending | (live & ((vw == empty) | (iw == empty))) + take = live & _better( + vw.bitcast(cutlass.Float32), iw, pass_value, pass_index + ) + pass_value = cutlass.Float32( + cutlass.select_(take, vw.bitcast(cutlass.Float32), pass_value) + ) + pass_index = cutlass.Int32(cutlass.select_(take, iw, pass_index)) + best_value = pass_value + best_index = pass_index + pending = pass_pending + best_value, best_index = _warp_best(best_value, best_index) + if lane == 0: + s_poll_best.store((best_value.bitcast(cutlass.Int32), best_index), + idx=(b * cutlass.Int32(WARPS) + warp) * cutlass.Int32(2), alignment=8) # fmt: skip + prims.barrier_cta_sync(0) + if tx < cutlass.Int32(batch): + g_value, g_index = _reduce_slots(s_poll_best, tx * cutlass.Int32(WARPS * 2)) + # The next position's markov_w1 row, requested as soon as prev is known: every reader of s_w1 and + # s_valid for this position passed the barrier before the push. No barrier follows: the other threads + # go on to the next position's base logits and wait for the row's phase. s_poll_best is next written + # after the next push barrier, which this thread reaches only after these reads. + if k + cutlass.Int32(1) < cutlass.Int32(block): + _fetch_w1_row(s_w1, s_valid, w1, s_mbar_w1, tx, g_index, vocab) + if cta == 0: + tokens.store(g_index, idx=tx * cutlass.Int32(block) + k) + next_new.store(g_index, idx=tx * cutlass.Int32(block + 1) + k + cutlass.Int32(1)) + + # Every CTA read the flags before arriving; the last one advances them for the next call. + if tx == 0: + arrived = _atomic_add_acq_rel(flags.data_ptr(1).toint(), cutlass.Int32(1)) + if arrived == cutlass.Int32(grid - 1): + flags.store(cutlass.Int32(0), idx=1) + flags.store(cutlass.Int32(used_words), idx=2) + flags.store((cur + cutlass.Int32(1)) % cutlass.Int32(BUFFERS), idx=0) + + +@cute.jit +def k3_markov( + base: cute.Tensor, + first_prev: cute.Tensor, + w1: cute.Tensor, + w2: cute.Tensor, + corrected: cute.Tensor, + tokens: cute.Tensor, + next_new: cute.Tensor, + accepted: cute.Tensor, + num_accepted: cute.Tensor, + accepted_rows: cute.Tensor, + kv_lens: cute.Tensor, + rewind: cute.Tensor, + buf_uc: cute.Tensor, + buf_mc: cute.Tensor, + flags: cute.Tensor, + vocab: cutlass.Int32, + vocab_offset: cutlass.Int32, + rank: cutlass.Int32, + accepted_stride: cutlass.Int32, + rewind_first: cutlass.Int32, + rewind_count: cutlass.Int32, + grid: cutlass.Constexpr[int], + rows: cutlass.Constexpr[int], + shard: cutlass.Constexpr[int], + block: cutlass.Constexpr[int], + batch: cutlass.Constexpr[int], + slots: cutlass.Constexpr[int], + push_copies: cutlass.Constexpr[int], + buf_words: cutlass.Constexpr[int], + base_bf16: cutlass.Constexpr[bool], + fp32_sum: cutlass.Constexpr[bool], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + k3_markov_kernel( + base, first_prev, w1, w2, corrected, tokens, next_new, accepted, num_accepted, accepted_rows, kv_lens, + rewind, buf_uc, buf_mc, flags, vocab, vocab_offset, rank, accepted_stride, rewind_first, rewind_count, + grid, rows, shard, block, batch, slots, push_copies, buf_words, base_bf16, fp32_sum, + ).launch( + grid=[grid, 1, 1], + block=[THREADS, 1, 1], + stream=stream, + use_pdl=use_pdl, + ) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1.py new file mode 100644 index 000000000000..97c83e4c01be --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1.py @@ -0,0 +1,616 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark vanilla-Markov draft chain for a vocab-sharded draft head -- CTM (prims/cute) kernel, BATCH-1 VARIANT (S4B8) +# Identical to dspark_markov_chain_kernel.py except the position loop: every request's base loads are issued before the +# position's W1 wait, and the exchange is polled in one first pass for every request before the per-request fallback. +# Bitwise the same tokens (same per-row FMA order and roundings, total argmax order). Measured (TP8, S4B8): +# faster at one request (K3 19.7 -> 18.9 us, K7 38.5 -> 35.2), slower at more (register pressure), so the op runs it +# only for batch 1. +# ============================================================================= +# +# For every request b and block position k = 0 .. K-1 (greedy, temperature 0): +# bias[v] = bf16(markov_w2[v] . markov_w1[prev]) v in this rank's vocab shard +# value[v] = bf16(base[b, k, v] + bias[v]) (the reference's bf16 add: fp32 sum, one rounding) +# token[b, k] = the first maximum of value over the whole vocabulary (all ranks) +# prev = token[b, k] (prev for k = 0 is the anchor, first_prev[b]) +# so the K positions are a chain of dependent global argmaxes across the tensor-parallel ranks. +# +# Numerics are dsparkMarkovChainKernel's (the C++ kernel this replaces): lane l of a warp sums the 8 columns +# 8l .. 8l+7 of a row by FMAs from 0, the 32 lane sums are added by the xor tree 16, 8, 4, 2, 1, the sum is rounded +# to bf16 (cvt.rn), added to the base in fp32 and the sum rounded to bf16 (cvt.rn), as the reference chain's bf16 +# logits + bf16 bias does. Argmax order: larger value, then lower vocabulary index; NaN never +# wins. The order is total, so every split of the reduction gives the same token. +# +# Geometry: G CTAs of 256 threads, CTA c owning the RC = S / G rows [c RC, c RC + RC) of the shard (RC a multiple +# of 64). Warp w owns the RC / 8 contiguous rows [w RC / 8, ...), in blocks of 8 (row j's sum ends on lane 4j after +# the reduce-scatter form of the xor tree). The CTA's rows of markov_w2 are bulk-copied into shared memory before the +# grid-dependency wait (a weight) and stay there for all K positions. markov_w1[prev] (512 bytes per request) is one +# bulk copy per CTA and position into shared memory, issued by the thread that reduces request b's global argmax, on +# one phase of an mbarrier per position. +# +# Exchange (one NVLink hop per position): every CTA reduces its rows to one (value, index) entry per request and +# multicast-stores it into slot [k][b][rank][c] of every rank's Lamport buffer; every CTA then polls all +# W x G entries of [k][b] in its own rank's copy (the data is the ready signal: 0x80000000 = empty; values carry +# -0.0 as +0.0 and indices are non-negative, so no entry is the sentinel) and reduces them with the same order. +# Every CTA of every rank computes the same global argmax: no leader, no grid barrier, no publish word. +# +# Buffers: 3 rotating Lamport buffers. Call n uses buffer n % 3 and re-arms buffer (n - 1) % 3 (its readers, all +# CTAs of call n - 1, finished before this call's grid wait returned; its next writer is a remote rank's call n + 2, +# which needs this rank's call n + 1 pushes). flags: [0] buffer of this call, [1] CTAs arrived, [2] words the +# previous call used (to re-arm), [3] unused. The last CTA to arrive advances them. +# +# Roles: warp 0's elected lane issues the weight bulk copy (and the early dependent trigger); thread b < B pushes +# request b's CTA entry, reduces its global argmax and requests the next markov_w1 row; all 8 warps compute and poll +# (the chain inside a position is serial, and the poll is spread over all 256 threads: one load wave). Per position: +# rows → barrier → push → poll → barrier → (thread b) argmax, next row; no barrier between the argmax and the next +# position (its readers wait for the row's phase). +# ============================================================================= +"""CTM DSpark Markov chain: the draft head's greedy intra-block Markov bias, the per-position global argmax +across the tensor-parallel vocab shards (the draft tokens), next_new_tokens and the target's KV-length rewind after +the draft forward, in one launch.""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +THREADS = 256 +WARPS = THREADS // 32 +MARKOV_RANK = 256 # 32 lanes x 8 columns +ROW_WORDS = MARKOV_RANK // 2 # int32 words (bf16 pairs) per markov row +ROW_BYTES = MARKOV_RANK * 2 +BLOCK_ROWS = 8 # rows per warp block: one per epilogue lane +MAX_BATCH = 16 +MAX_BLOCK = 16 +BUFFERS = 3 +FLAG_WORDS = 4 +EMPTY_WORD = -(2**31) # 0x80000000: fp32 -0.0, never a pushed word +INT_MAX = 2**31 - 1 +NEG_INF_BITS = -8388608 # 0xFF800000 +COPY_ROWS = 32 # rows per weight bulk copy (16 KB) +SMEM_ROW_BUDGET = 400 # 200 KB of markov_w2 rows per CTA +SMEM_BYTES = 227 * 1024 + + +def rows_per_cta(shard: int, grid: int) -> int: + return shard // grid + + +def supports(shard: int, grid: int, block: int, batch: int) -> bool: + """Shapes the kernel runs: whole 8-row blocks for every warp (RC a multiple of 64), the CTA's markov_w2 rows + within the shared-memory budget, block <= 16, batch <= 16, an even entry count per exchange.""" + if grid <= 0 or shard % grid != 0: + return False + rc = rows_per_cta(shard, grid) + return ( + rc % (WARPS * BLOCK_ROWS) == 0 + and rc <= SMEM_ROW_BUDGET + and smem_bytes(rc, batch) <= SMEM_BYTES + and grid % 2 == 0 + and 0 < block <= MAX_BLOCK + and 0 < batch <= MAX_BATCH + ) + + +def smem_bytes(rows: int, batch: int) -> int: + """Shared memory of a CTA: its markov_w2 rows, the markov_w1[prev] rows, the two [b][warp] reductions, the + barriers, flags and prev (with the allocator's alignment padding).""" + return rows * ROW_BYTES + batch * ROW_BYTES + 2 * batch * WARPS * 8 + 1024 + + +def buffer_words(block: int, batch: int, slots: int, grid: int) -> int: + """Int32 words of one Lamport buffer: one (value, index) entry per position, request, rank slot and CTA.""" + return block * batch * slots * grid * 2 + + +@dsl_user_op +def _atomic_add_acq_rel(addr_i64, val, *, loc=None, ip=None): + """atom.acq_rel.gpu.global.add.u32, returning the old value.""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [addr_i64.ir_value(loc=loc, ip=ip), val.ir_value(loc=loc, ip=ip)], + "atom.acq_rel.gpu.global.add.u32 $0, [$1], $2;", "=r,l,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + """fma.rn.f32 (fmaf).""" + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [a.ir_value(loc=loc, ip=ip), b.ir_value(loc=loc, ip=ip), c.ir_value(loc=loc, ip=ip)], + "fma.rn.f32 $0, $1, $2, $3;", "=f,f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + """add.rn.f32: an fp32 add the compiler cannot contract with a neighbouring multiply.""" + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [a.ir_value(loc=loc, ip=ip), b.ir_value(loc=loc, ip=ip)], + "add.rn.f32 $0, $1, $2;", "=f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _pack_bf16x2(hi, lo, *, loc=None, ip=None): + """(bf16(hi) << 16) | bf16(lo), round to nearest even (cvt.rn, as __float2bfloat16_rn).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [hi.ir_value(loc=loc, ip=ip), lo.ir_value(loc=loc, ip=ip)], + "cvt.rn.bf16x2.f32 $0, $1, $2;", "=r,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _lo_f32(word): + """The bf16 in the low half of a word, as fp32 (exact).""" + return cutlass.Int32(word << cutlass.Int32(16)).bitcast(cutlass.Float32) + + +def _hi_f32(word): + """The bf16 in the high half of a word, as fp32 (exact).""" + return cutlass.Int32(word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + + +def _better(value, index, best_value, best_index): + """dsparkMarkovChainKernel's keepBetter order: larger value, or an equal value at a lower index. NaN never + compares better, so it never wins.""" + return (value > best_value) | ((value == best_value) & (index < best_index)) + + +def _keep_better(best_value, best_index, value, index): + take = _better(value, index, best_value, best_index) + return cutlass.Float32(cutlass.select_(take, value, best_value)), cutlass.Int32( + cutlass.select_(take, index, best_index) + ) + + +def _warp_best(value, index): + """Every lane gets the warp's best (value, index) (xor tree: the order is total, so all lanes agree).""" + for offset in [16, 8, 4, 2, 1]: + other_value = cute.arch.shuffle_sync_bfly(value, offset=offset, mask=-1, mask_and_clamp=31) + other_index = cute.arch.shuffle_sync_bfly(index, offset=offset, mask=-1, mask_and_clamp=31) + value, index = _keep_better(value, index, other_value, other_index) + return value, index + + +def _block_dot(s_w2, first_row, lane, w1f): + """markov_w2[first_row + j] . markov_w1[prev] for the block's 8 rows, in the C++ kernel's order: lane l sums the + 8 columns 8l .. 8l+7 by an FMA chain from 0, then the 32 lane sums are added in pairs at lane distances 16, 8, 4, 2, + 1 (the xor tree). Done as a reduce-scatter: at distances 16, 8 and 4 a lane keeps half of its rows and adds its + partner's partial of each kept row (the pairs of the full tree, so the same sums: 9 shuffles instead of 40), then + the full tree at 2 and 1. Returns row (lane >> 2)'s sum.""" + parts = [] + for r in range(BLOCK_ROWS): + w2v = s_w2.load(idx=(first_row + cutlass.Int32(r)) * cutlass.Int32(ROW_WORDS) + lane * cutlass.Int32(4), + vector_size=4, alignment=16) # fmt: skip + acc = cutlass.Float32(0.0) + for q in range(4): + word = cutlass.Int32(w2v[q]) + acc = _fma_rn(w1f[2 * q], _lo_f32(word), acc) + acc = _fma_rn(w1f[2 * q + 1], _hi_f32(word), acc) + parts.append(acc) + for offset in [16, 8, 4]: + upper = (lane & cutlass.Int32(offset)) != cutlass.Int32(0) + half = len(parts) // 2 + kept = [] + for i in range(half): + send = cutlass.Float32(cutlass.select_(upper, parts[i], parts[half + i])) + keep = cutlass.Float32(cutlass.select_(upper, parts[half + i], parts[i])) + kept.append( + _add_rn( + keep, + cute.arch.shuffle_sync_bfly(send, offset=offset, mask=-1, mask_and_clamp=31), + ) + ) + parts = kept + dot = parts[0] + for offset in [2, 1]: + dot = _add_rn( + dot, cute.arch.shuffle_sync_bfly(dot, offset=offset, mask=-1, mask_and_clamp=31) + ) + return dot + + +def _fetch_w1_row(s_w1, s_valid, w1, mbar, b, prev, vocab): + """Request b's markov_w1[prev] row into shared memory: one 512-byte bulk copy on the position's barrier phase (a + prev outside the vocabulary copies row 0 and clears s_valid[b]: the readers zero its bias). s_valid[b] is stored + before the arrive, whose release the readers' phase wait acquires.""" + valid = (prev >= cutlass.Int32(0)) & (prev < vocab) + row = cutlass.Int32(cutlass.select_(valid, prev, cutlass.Int32(0))) + s_valid.store(cutlass.Int32(cutlass.select_(valid, cutlass.Int32(1), cutlass.Int32(0))), idx=b) + prims.mbarrier_arrive_expect_tx(mbar, ROW_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_w1.subview(b * cutlass.Int32(ROW_WORDS)), + w1.subview(row * cutlass.Int32(ROW_WORDS)), + mbar, + ROW_BYTES, + ) + + +def _reduce_slots(slots_arr, first_word): + """The best of 8 (value bits, index) pairs at first_word .. in warp order.""" + value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + index = cutlass.Int32(INT_MAX) + for w in range(WARPS): + pair = slots_arr.load(idx=first_word + cutlass.Int32(w * 2), vector_size=2, alignment=8) + value, index = _keep_better( + value, index, cutlass.Int32(pair[0]).bitcast(cutlass.Float32), cutlass.Int32(pair[1]) + ) + return value, index + + +@cute.kernel +def k3_markov_kernel( + base: cutlass.Array, # fp32 [B * K * S] (or its int16 bf16 view with base_bf16): the draft head's shard logits + first_prev: cutlass.Array, # int64 [B]: the anchor token of each request + w1: cutlass.Array, # int32 words of bf16 markov_w1 [V, 256] + w2: cutlass.Array, # int32 words of bf16 markov_w2[shard rows] [S, 256] + corrected: cutlass.Array, # unused (kept for the launch signature) + tokens: cutlass.Array, # int32 [B * K], out + next_new: cutlass.Array, # int32 [B * (K + 1)], out + accepted: cutlass.Array, # int32 [*, accepted_stride]: accepted tokens + num_accepted: cutlass.Array, # int32 [B] + accepted_rows: cutlass.Array, # int32 [B]: the row of accepted of each request + kv_lens: cutlass.Array, # int32: the target's KV lengths, rewound here after the draft forward + rewind: cutlass.Array, # int32 [rewind_count]: subtracted from kv_lens[rewind_first ...], clamped at 0 + buf_uc: cutlass.Array, # int32 words: this rank's Lamport buffers + buf_mc: cutlass.Array, # int32 words: their multicast mapping + flags: cutlass.Array, # int32 [4] + vocab: cutlass.Int32, # rows of markov_w1: a prev outside [0, vocab) gets a zero bias + vocab_offset: cutlass.Int32, # first vocabulary index of this rank's shard + rank: cutlass.Int32, + accepted_stride: cutlass.Int32, + rewind_first: cutlass.Int32, + rewind_count: cutlass.Int32, # 0: no rewind + grid: cutlass.Constexpr[int], + rows: cutlass.Constexpr[int], # RC + shard: cutlass.Constexpr[int], # S + block: cutlass.Constexpr[int], # K + batch: cutlass.Constexpr[int], # B + slots: cutlass.Constexpr[int], # rank slots per (k, b): world x push_copies + push_copies: cutlass.Constexpr[ + int + ], # test only: each rank's entry goes into push_copies consecutive slots + buf_words: cutlass.Constexpr[int], + base_bf16: cutlass.Constexpr[bool], + fp32_sum: cutlass.Constexpr[bool], +): + rows_per_warp = rows // WARPS + blocks_per_warp = rows_per_warp // BLOCK_ROWS + entries = slots * grid # (value, index) entries per (k, b) + vectors = entries // 2 # 16-byte loads per (k, b) + vec_per_thread = (vectors + THREADS - 1) // THREADS + used_words = block * batch * entries * 2 + + tx, _, _ = cute.arch.thread_idx() + cta, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tx % cutlass.Int32(32) + + s_w2 = cutlass.Array( + cutlass.Int32, rows * ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128 + ) + # [b]: markov_w1[prev] of the position (one bulk copy per request, issued by the thread that learns prev). + s_w1 = cutlass.Array( + cutlass.Int32, batch * ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128 + ) + s_mbar = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + s_mbar_w1 = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + # [b][warp] (value bits, index): the CTA reduction of the rows, then of the polled entries. + s_row_best = cutlass.Array( + cutlass.Int32, batch * WARPS * 2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_poll_best = cutlass.Array( + cutlass.Int32, batch * WARPS * 2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_flags = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_valid = cutlass.Array(cutlass.Int32, MAX_BATCH, space=cutlass.AddressSpace.smem, alignment=16) + + if warp == 0: + if prims.elect_sync(): + prims.mbarrier_init(s_mbar, 1) + prims.mbarrier_init(s_mbar_w1, batch) + prims.fence_mbarrier_init() + prims.barrier_cta_sync(0) + + # The CTA's markov_w2 rows: a weight, loaded before the grid dependency. + if warp == 0: + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(s_mbar, rows * ROW_BYTES) + for c in cutlass.range_constexpr((rows + COPY_ROWS - 1) // COPY_ROWS): + n = min(COPY_ROWS, rows - c * COPY_ROWS) + prims.cp_async_bulk_shared_cluster_global( + s_w2.subview(c * COPY_ROWS * ROW_WORDS), + w2.subview( + cta * cutlass.Int32(rows * ROW_WORDS) + + cutlass.Int32(c * COPY_ROWS * ROW_WORDS) + ), + s_mbar, + n * ROW_BYTES, + ) + # Dependents wait for this whole grid before reading its outputs. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + + prims.griddepcontrol(prims.GridDepAction.WAIT) + if tx == 0: + s_flags.store(flags.load(idx=0, is_volatile=True), idx=0) + s_flags.store(flags.load(idx=2, is_volatile=True), idx=2) + if tx < cutlass.Int32(batch): + _fetch_w1_row( + s_w1, s_valid, w1, s_mbar_w1, tx, cutlass.Int32(first_prev.load(idx=tx)), vocab + ) + prims.barrier_cta_sync(0) + cur = s_flags.load(idx=0) + cur_base = cur * cutlass.Int32(buf_words) + + # Re-arm the previous call's buffer (this CTA's 16-byte chunks of the words it used). + dirty_base = ((cur + cutlass.Int32(2)) % cutlass.Int32(BUFFERS)) * cutlass.Int32(buf_words) + dirty_words = s_flags.load(idx=2) + empty = cutlass.Int32(EMPTY_WORD) + w_clear = (cta * cutlass.Int32(THREADS) + tx) * cutlass.Int32(4) + while w_clear < dirty_words: + buf_uc.store((empty, empty, empty, empty), idx=dirty_base + w_clear, alignment=16) + w_clear += cutlass.Int32(grid * THREADS * 4) + + # next_new_tokens[b] = [accepted[row_b, num_accepted[b] - 1], tokens[b,:]] (-1 wraps, as torch indexing). + if cta == 0: + if tx < cutlass.Int32(batch): + col = num_accepted.load(idx=tx) - cutlass.Int32(1) + col = cutlass.Int32(cutlass.select_(col < cutlass.Int32(0), col + accepted_stride, col)) + row = accepted_rows.load(idx=tx) + next_new.store( + accepted.load(idx=row * accepted_stride + col), idx=tx * cutlass.Int32(block + 1) + ) + # The KV-length rewind after the draft forward (whose kernels, the readers of kv_lens, all finished before + # this grid's wait): kv_lens[first + b] = max(kv_lens[first + b] - rewind[b], 0). + if tx < rewind_count: + left = kv_lens.load(idx=rewind_first + tx) - rewind.load(idx=tx) + kv_lens.store(cutlass.Int32(cutlass.select_(left < cutlass.Int32(0), cutlass.Int32(0), left)), + idx=rewind_first + tx) # fmt: skip + + # The weights have landed (issued before the wait). + while not cute.arch.mbarrier_test_wait(s_mbar.data_ptr(), 0): + pass + + row0 = warp * cutlass.Int32(rows_per_warp) # first row of the warp within the CTA + shard_row0 = cta * cutlass.Int32(rows) + row0 # ... within the shard + has_row = (lane & cutlass.Int32(3)) == cutlass.Int32(0) + lane_row = shard_row0 + (lane >> cutlass.Int32(2)) + for k in cutlass.range(block, unroll=1): + # S4B8: every request's base logits are issued together before the position's W1 wait (one exposed + # global-load latency per position instead of one per request). Values, FMA order and argmax order unchanged. + base_all = [] + for b in cutlass.range_constexpr(batch): + base_row = (cutlass.Int32(b * block) + k) * cutlass.Int32(shard) + # Lane 4j finishes row j of each 8-row block (the reduce-scatter leaves row j's sum on lanes 4j .. 4j+3). + base_vals = [] + for g in cutlass.range_constexpr(blocks_per_warp): + if cutlass.const_expr(base_bf16): + base_vals.append( + cutlass.Int32( + cutlass.Int32( + base.load(idx=base_row + lane_row + cutlass.Int32(g * BLOCK_ROWS)) + ) + << cutlass.Int32(16) + ).bitcast(cutlass.Float32) + ) + else: + base_vals.append( + base.load(idx=base_row + lane_row + cutlass.Int32(g * BLOCK_ROWS)) + ) + base_all.append(base_vals) + # markov_w1[prev] landed for every request (the position's phase of the W1 barrier). + while not cute.arch.mbarrier_test_wait(s_mbar_w1.data_ptr(), k & cutlass.Int32(1)): + pass + for b in cutlass.range_constexpr(batch): + base_vals = base_all[b] + valid = s_valid.load(idx=b) != cutlass.Int32(0) + # markov_w1[prev], columns 8 lane .. 8 lane + 7, as fp32 (zeros for an invalid prev). + w1v = s_w1.load( + idx=cutlass.Int32(b * ROW_WORDS) + lane * 4, vector_size=4, alignment=16 + ) + w1f = [] + for q in cutlass.range_constexpr(4): + word = cutlass.Int32( + cutlass.select_(valid, cutlass.Int32(w1v[q]), cutlass.Int32(0)) + ) + w1f.append(_lo_f32(word)) + w1f.append(_hi_f32(word)) + best_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + best_index = cutlass.Int32(INT_MAX) + for g in cutlass.range_constexpr(blocks_per_warp): + dot = _block_dot(s_w2, row0 + cutlass.Int32(g * BLOCK_ROWS), lane, w1f) + bias = _lo_f32(_pack_bf16x2(dot, dot)) + # fp32_sum (default): the in-tree chain's arithmetic (fp32 logits from LogitsProcessor's .float() + # plus the widened bf16 bias): compare the unrounded fp32 sum. Without fp32_sum: a bf16 reference + # rounds that sum once. + if cutlass.const_expr(fp32_sum): + value = _add_rn(base_vals[g], bias) + else: + value = _lo_f32( + _pack_bf16x2(_add_rn(base_vals[g], bias), _add_rn(base_vals[g], bias)) + ) + if has_row: + my_row = lane_row + cutlass.Int32(g * BLOCK_ROWS) + best_value, best_index = _keep_better( + best_value, best_index, value, vocab_offset + my_row + ) + best_value, best_index = _warp_best(best_value, best_index) + if lane == 0: + s_row_best.store((best_value.bitcast(cutlass.Int32), best_index), + idx=cutlass.Int32(b * WARPS * 2) + warp * cutlass.Int32(2), alignment=8) # fmt: skip + prims.barrier_cta_sync(0) + # Thread b pushes request b's CTA maximum into every rank's buffer. + if tx < cutlass.Int32(batch): + cta_value, cta_index = _reduce_slots(s_row_best, tx * cutlass.Int32(WARPS * 2)) + bits = cta_value.bitcast(cutlass.Int32) + bits = cutlass.Int32( + cutlass.select_(bits == cutlass.Int32(EMPTY_WORD), cutlass.Int32(0), bits) + ) + for c in cutlass.range_constexpr(push_copies): + s_idx = rank * cutlass.Int32(push_copies) + cutlass.Int32(c) + word = cur_base + (((k * cutlass.Int32(batch) + tx) * cutlass.Int32(slots) + s_idx) + * cutlass.Int32(grid) + cta) * cutlass.Int32(2) # fmt: skip + buf_mc.store((bits, cta_index), idx=word, alignment=8) + # Drain the posted multicast store now (the polls below issue no release that would). + prims.fence_acq_rel(prims.MemScope.CLUSTER) + # S4B8: one first pass loads every request's [k][b] entries together (B load waves in flight at once + # instead of one after another); a request still incomplete after it falls back to the v1 poll loop. The + # argmax order is total, so the result is the same whichever pass completes a request. + firsts = [] + for b in cutlass.range_constexpr(batch): + region = cur_base + (k * cutlass.Int32(batch) + cutlass.Int32(b)) * cutlass.Int32( + entries * 2 + ) + gots = [] + for i in cutlass.range_constexpr(vec_per_thread): + v_idx = tx + cutlass.Int32(i * THREADS) + live = v_idx < cutlass.Int32(vectors) + gots.append((live, buf_uc.load( + idx=region + cutlass.Int32(cutlass.select_(live, v_idx, cutlass.Int32(0))) * cutlass.Int32(4), + vector_size=4, alignment=16, is_volatile=True))) # fmt: skip + firsts.append((region, gots)) + for b in cutlass.range_constexpr(batch): + region, gots = firsts[b] + best_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + best_index = cutlass.Int32(INT_MAX) + pending = cutlass.Boolean(False) + for live, got in gots: + for e in cutlass.range_constexpr(2): + vw = cutlass.Int32(got[2 * e]) + iw = cutlass.Int32(got[2 * e + 1]) + pending = pending | (live & ((vw == empty) | (iw == empty))) + take = live & _better(vw.bitcast(cutlass.Float32), iw, best_value, best_index) + best_value = cutlass.Float32( + cutlass.select_(take, vw.bitcast(cutlass.Float32), best_value) + ) + best_index = cutlass.Int32(cutlass.select_(take, iw, best_index)) + while pending: + pass_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + pass_index = cutlass.Int32(INT_MAX) + pass_pending = cutlass.Boolean(False) + for i in cutlass.range_constexpr(vec_per_thread): + v_idx = tx + cutlass.Int32(i * THREADS) + live = v_idx < cutlass.Int32(vectors) + got = buf_uc.load( + idx=region + + cutlass.Int32(cutlass.select_(live, v_idx, cutlass.Int32(0))) + * cutlass.Int32(4), + vector_size=4, + alignment=16, + is_volatile=True, + ) + for e in cutlass.range_constexpr(2): + vw = cutlass.Int32(got[2 * e]) + iw = cutlass.Int32(got[2 * e + 1]) + pass_pending = pass_pending | (live & ((vw == empty) | (iw == empty))) + take = live & _better( + vw.bitcast(cutlass.Float32), iw, pass_value, pass_index + ) + pass_value = cutlass.Float32( + cutlass.select_(take, vw.bitcast(cutlass.Float32), pass_value) + ) + pass_index = cutlass.Int32(cutlass.select_(take, iw, pass_index)) + best_value = pass_value + best_index = pass_index + pending = pass_pending + best_value, best_index = _warp_best(best_value, best_index) + if lane == 0: + s_poll_best.store((best_value.bitcast(cutlass.Int32), best_index), + idx=cutlass.Int32(b * WARPS * 2) + warp * cutlass.Int32(2), alignment=8) # fmt: skip + prims.barrier_cta_sync(0) + if tx < cutlass.Int32(batch): + g_value, g_index = _reduce_slots(s_poll_best, tx * cutlass.Int32(WARPS * 2)) + # The next position's markov_w1 row, requested as soon as prev is known: every reader of s_w1 and + # s_valid for this position passed the barrier before the push. No barrier follows: the other threads + # go on to the next position's base logits and wait for the row's phase. s_poll_best is next written + # after the next push barrier, which this thread reaches only after these reads. + if k + cutlass.Int32(1) < cutlass.Int32(block): + _fetch_w1_row(s_w1, s_valid, w1, s_mbar_w1, tx, g_index, vocab) + if cta == 0: + tokens.store(g_index, idx=tx * cutlass.Int32(block) + k) + next_new.store(g_index, idx=tx * cutlass.Int32(block + 1) + k + cutlass.Int32(1)) + + # Every CTA read the flags before arriving; the last one advances them for the next call. + if tx == 0: + arrived = _atomic_add_acq_rel(flags.data_ptr(1).toint(), cutlass.Int32(1)) + if arrived == cutlass.Int32(grid - 1): + flags.store(cutlass.Int32(0), idx=1) + flags.store(cutlass.Int32(used_words), idx=2) + flags.store((cur + cutlass.Int32(1)) % cutlass.Int32(BUFFERS), idx=0) + + +@cute.jit +def k3_markov( + base: cute.Tensor, + first_prev: cute.Tensor, + w1: cute.Tensor, + w2: cute.Tensor, + corrected: cute.Tensor, + tokens: cute.Tensor, + next_new: cute.Tensor, + accepted: cute.Tensor, + num_accepted: cute.Tensor, + accepted_rows: cute.Tensor, + kv_lens: cute.Tensor, + rewind: cute.Tensor, + buf_uc: cute.Tensor, + buf_mc: cute.Tensor, + flags: cute.Tensor, + vocab: cutlass.Int32, + vocab_offset: cutlass.Int32, + rank: cutlass.Int32, + accepted_stride: cutlass.Int32, + rewind_first: cutlass.Int32, + rewind_count: cutlass.Int32, + grid: cutlass.Constexpr[int], + rows: cutlass.Constexpr[int], + shard: cutlass.Constexpr[int], + block: cutlass.Constexpr[int], + batch: cutlass.Constexpr[int], + slots: cutlass.Constexpr[int], + push_copies: cutlass.Constexpr[int], + buf_words: cutlass.Constexpr[int], + base_bf16: cutlass.Constexpr[bool], + fp32_sum: cutlass.Constexpr[bool], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + k3_markov_kernel( + base, first_prev, w1, w2, corrected, tokens, next_new, accepted, num_accepted, accepted_rows, kv_lens, + rewind, buf_uc, buf_mc, flags, vocab, vocab_offset, rank, accepted_stride, rewind_first, rewind_count, + grid, rows, shard, block, batch, slots, push_copies, buf_words, base_bf16, fp32_sum, + ).launch( + grid=[grid, 1, 1], + block=[THREADS, 1, 1], + stream=stream, + use_pdl=use_pdl, + ) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1_ll.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1_ll.py new file mode 100644 index 000000000000..af191f8cbc4b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1_ll.py @@ -0,0 +1,624 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark vanilla-Markov draft chain for a vocab-sharded draft head -- CTM (prims/cute) kernel, BATCH-1 VARIANT WITH +# THE LL (FLAG-IN-PAYLOAD) EXCHANGE (MARKOV_LL) +# A copy of dspark_markov_chain_kernel_b1.py (S4B8) for batch 1 only, with the per-position exchange replaced: +# - post: one 16-byte multicast store per CTA and position, {value bits, tag, index, tag}; no fence and no separate +# flag word (the tag is in the payload); +# - poll + combine: warp 0 alone polls every entry of the position (16-byte loads) until both tag words of every +# entry hold this launch's tag, and reduces them with warp shuffles; no CTA barrier after the poll (the other warps +# go on to the next position's base loads and wait for the markov_w1 row, which only warp 0 can request); +# - the CTA's row reduction before the post: warp 0 waits on a named barrier the other warps only arrive at. +# Everything else is b1's: smem-resident markov_w2, the GEMV FMA order, the bf16 roundings, the fp32-sum option, the +# PDL wait placement, the argmax order (larger value, then lower vocabulary index; NaN never wins). Bitwise the same +# tokens as the b1 kernel: every CTA entry is computed as b1 computes it, and the global reduction folds the same set +# of entries under the same total order (any fold order gives the same winner). +# ============================================================================= +# +# For request 0 (batch 1) and block position k = 0 .. K-1 (greedy, temperature 0): +# bias[v] = bf16(markov_w2[v] . markov_w1[prev]) v in this rank's vocab shard +# value[v] = bf16(base[0, k, v] + bias[v]) (the reference's bf16 add: fp32 sum, one rounding) +# token[0, k] = the first maximum of value over the whole vocabulary (all ranks) +# prev = token[0, k] (prev for k = 0 is the anchor, first_prev[0]) +# +# Numerics: dspark_markov_chain_kernel_b1.py's, unchanged (lane l sums the 8 columns 8l .. 8l+7 of a row by FMAs from +# 0, xor-tree lane sums, cvt.rn to bf16, fp32 add, cvt.rn; with fp32_sum the unrounded fp32 sum is compared). +# +# Geometry: b1's (G CTAs of 256 threads, RC rows per CTA, warp w owns RC / 8 rows in blocks of 8, markov_w2 rows in +# shared memory before the grid-dependency wait, markov_w1[prev] one bulk copy per position on an mbarrier phase). +# +# LL exchange. The workspace (dspark_markov_chain.workspace_args) holds the 3 Lamport buffers of the other variants +# (untouched here) followed by LL_PARITIES = 2 LL regions of buf_words words each (zero-filled at allocation): +# region p = buf_uc[(BUFFERS + p) * buf_words ...], entry e = (k * slots + rank slot) * G + CTA, 4 words at 4 e: +# {value bits, tag, index, tag} +# Each 8-byte half carries the tag, so a reader that sees the tag in both halves has seen both halves of this write +# (8-byte single-copy granularity of a 16-byte store, the assumption of NCCL's LL protocol). +# Tag: seq, the launch sequence number of this workspace's LL calls (flags[LL_SEQ] holds the last one; this call +# uses seq = that + 1, skipping 0 on wrap; every rank runs the same calls in the same order, so every rank computes +# the same seq). Launch seq uses region seq & 1. The position is in the address, not the tag. +# Why a stale entry is never accepted: an entry is accepted only if both tag words equal this call's seq. Every LL +# call writes every entry it reads (k < K, every rank slot, every CTA) with its own seq, so an entry left by another +# call carries a different seq unless 2^32 LL calls passed since it was written (> 40 h of back-to-back calls of +# only a smaller shape), and the zero fill (tag 0) is never a seq. +# Why no entry is overwritten while a reader still needs it: call n + 1 on rank q uses the other region; call n + 2 +# (this region again) on rank q starts after q's call n + 1 completed, which waited for every rank's call n + 1 +# entries, which a rank posts only after its call n completed (griddepcontrol.wait before the first post). So no +# rank still reads call n's region when it is next written. (One region would do for K >= 2; two make K = 1 safe and +# keep calls of different shapes apart: the region stride is fixed, the layout inside a region follows the shape.) +# Memory ordering: the post is multimem.st.relaxed.sys (no release, no fence: the payload is the data, nothing else +# is published); the poll is ld.volatile (relaxed, system scope) 16-byte loads of this rank's copy (no acquire: no +# other data is read through the tag). flags: read after griddepcontrol.wait, advanced by the last CTA to arrive +# (atom.acq_rel.gpu), as in b1. +# +# Roles per position: all 8 warps compute their rows; lane 0 of each warp writes the warp's best to shared memory; +# warps 1..7 arrive at named barrier LL_BAR and go on to the next position's base loads and the markov_w1 wait; warp +# 0 waits at LL_BAR, thread 0 reduces the 8 warp bests and posts the CTA entry, warp 0 polls the W x G entries +# (ceil(W G / 32) 16-byte loads per lane, all issued before any is used), folds them per lane and across the warp +# (xor tree), and thread 0 requests the next markov_w1 row (and, in CTA 0, stores the token). The next position's +# readers of s_row_best / s_w1 / s_valid write or read them only after the row's mbarrier phase, which thread 0 +# completes after its reads of s_row_best (mbarrier arrive: release; the phase wait: acquire). +# ============================================================================= +"""CTM DSpark Markov chain, batch-1 variant with the LL (flag-in-payload) exchange: the draft head's greedy +intra-block Markov bias, the per-position global argmax across the tensor-parallel vocab shards (the draft tokens), +next_new_tokens and the target's KV-length rewind after the draft forward, in one launch.""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +THREADS = 256 +WARPS = THREADS // 32 +MARKOV_RANK = 256 # 32 lanes x 8 columns +ROW_WORDS = MARKOV_RANK // 2 # int32 words (bf16 pairs) per markov row +ROW_BYTES = MARKOV_RANK * 2 +BLOCK_ROWS = 8 # rows per warp block: one per epilogue lane +MAX_BATCH = 16 +MAX_BLOCK = 16 +BUFFERS = 3 +FLAG_WORDS = 4 +EMPTY_WORD = -(2**31) # 0x80000000: fp32 -0.0, never a pushed word +INT_MAX = 2**31 - 1 +NEG_INF_BITS = -8388608 # 0xFF800000 +COPY_ROWS = 32 # rows per weight bulk copy (16 KB) +SMEM_ROW_BUDGET = 400 # 200 KB of markov_w2 rows per CTA +SMEM_BYTES = 227 * 1024 +LL_PARITIES = 2 # LL regions after the Lamport buffers, buf_words words each +LL_FLAG_WORDS = 8 # flags words this variant needs: [4] last LL seq, [5] LL CTAs arrived ([0..3]: the other variants) +LL_SEQ = 4 +LL_ARRIVED = 5 +LL_ENTRY_WORDS = 4 # {value bits, tag, index, tag} +LL_BAR = 1 # named barrier: warp 0 waits for the 8 warp bests (barrier 0 is __syncthreads) + + +def rows_per_cta(shard: int, grid: int) -> int: + return shard // grid + + +def supports(shard: int, grid: int, block: int, batch: int) -> bool: + """Shapes the kernel runs: whole 8-row blocks for every warp (RC a multiple of 64), the CTA's markov_w2 rows + within the shared-memory budget, block <= 16, batch <= 16, an even entry count per exchange.""" + if grid <= 0 or shard % grid != 0: + return False + rc = rows_per_cta(shard, grid) + return ( + rc % (WARPS * BLOCK_ROWS) == 0 + and rc <= SMEM_ROW_BUDGET + and smem_bytes(rc, batch) <= SMEM_BYTES + and grid % 2 == 0 + and 0 < block <= MAX_BLOCK + and 0 < batch <= MAX_BATCH + ) + + +def smem_bytes(rows: int, batch: int) -> int: + """Shared memory of a CTA: its markov_w2 rows, the markov_w1[prev] rows, the two [b][warp] reductions, the + barriers, flags and prev (with the allocator's alignment padding).""" + return rows * ROW_BYTES + batch * ROW_BYTES + 2 * batch * WARPS * 8 + 1024 + + +def buffer_words(block: int, batch: int, slots: int, grid: int) -> int: + """Int32 words of one Lamport buffer: one (value, index) entry per position, request, rank slot and CTA.""" + return block * batch * slots * grid * 2 + + +@dsl_user_op +def _atomic_add_acq_rel(addr_i64, val, *, loc=None, ip=None): + """atom.acq_rel.gpu.global.add.u32, returning the old value.""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [addr_i64.ir_value(loc=loc, ip=ip), val.ir_value(loc=loc, ip=ip)], + "atom.acq_rel.gpu.global.add.u32 $0, [$1], $2;", "=r,l,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + """fma.rn.f32 (fmaf).""" + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [a.ir_value(loc=loc, ip=ip), b.ir_value(loc=loc, ip=ip), c.ir_value(loc=loc, ip=ip)], + "fma.rn.f32 $0, $1, $2, $3;", "=f,f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + """add.rn.f32: an fp32 add the compiler cannot contract with a neighbouring multiply.""" + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [a.ir_value(loc=loc, ip=ip), b.ir_value(loc=loc, ip=ip)], + "add.rn.f32 $0, $1, $2;", "=f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _pack_bf16x2(hi, lo, *, loc=None, ip=None): + """(bf16(hi) << 16) | bf16(lo), round to nearest even (cvt.rn, as __float2bfloat16_rn).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [hi.ir_value(loc=loc, ip=ip), lo.ir_value(loc=loc, ip=ip)], + "cvt.rn.bf16x2.f32 $0, $1, $2;", "=r,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _lo_f32(word): + """The bf16 in the low half of a word, as fp32 (exact).""" + return cutlass.Int32(word << cutlass.Int32(16)).bitcast(cutlass.Float32) + + +def _hi_f32(word): + """The bf16 in the high half of a word, as fp32 (exact).""" + return cutlass.Int32(word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + + +def _better(value, index, best_value, best_index): + """dsparkMarkovChainKernel's keepBetter order: larger value, or an equal value at a lower index. NaN never + compares better, so it never wins.""" + return (value > best_value) | ((value == best_value) & (index < best_index)) + + +def _keep_better(best_value, best_index, value, index): + take = _better(value, index, best_value, best_index) + return cutlass.Float32(cutlass.select_(take, value, best_value)), cutlass.Int32( + cutlass.select_(take, index, best_index) + ) + + +def _warp_best(value, index): + """Every lane gets the warp's best (value, index) (xor tree: the order is total, so all lanes agree).""" + for offset in [16, 8, 4, 2, 1]: + other_value = cute.arch.shuffle_sync_bfly(value, offset=offset, mask=-1, mask_and_clamp=31) + other_index = cute.arch.shuffle_sync_bfly(index, offset=offset, mask=-1, mask_and_clamp=31) + value, index = _keep_better(value, index, other_value, other_index) + return value, index + + +def _block_dot(s_w2, first_row, lane, w1f): + """markov_w2[first_row + j] . markov_w1[prev] for the block's 8 rows, in the C++ kernel's order: lane l sums the + 8 columns 8l .. 8l+7 by an FMA chain from 0, then the 32 lane sums are added in pairs at lane distances 16, 8, 4, 2, + 1 (the xor tree). Done as a reduce-scatter: at distances 16, 8 and 4 a lane keeps half of its rows and adds its + partner's partial of each kept row (the pairs of the full tree, so the same sums: 9 shuffles instead of 40), then + the full tree at 2 and 1. Returns row (lane >> 2)'s sum.""" + parts = [] + for r in range(BLOCK_ROWS): + w2v = s_w2.load(idx=(first_row + cutlass.Int32(r)) * cutlass.Int32(ROW_WORDS) + lane * cutlass.Int32(4), + vector_size=4, alignment=16) # fmt: skip + acc = cutlass.Float32(0.0) + for q in range(4): + word = cutlass.Int32(w2v[q]) + acc = _fma_rn(w1f[2 * q], _lo_f32(word), acc) + acc = _fma_rn(w1f[2 * q + 1], _hi_f32(word), acc) + parts.append(acc) + for offset in [16, 8, 4]: + upper = (lane & cutlass.Int32(offset)) != cutlass.Int32(0) + half = len(parts) // 2 + kept = [] + for i in range(half): + send = cutlass.Float32(cutlass.select_(upper, parts[i], parts[half + i])) + keep = cutlass.Float32(cutlass.select_(upper, parts[half + i], parts[i])) + kept.append( + _add_rn( + keep, + cute.arch.shuffle_sync_bfly(send, offset=offset, mask=-1, mask_and_clamp=31), + ) + ) + parts = kept + dot = parts[0] + for offset in [2, 1]: + dot = _add_rn( + dot, cute.arch.shuffle_sync_bfly(dot, offset=offset, mask=-1, mask_and_clamp=31) + ) + return dot + + +def _fetch_w1_row(s_w1, s_valid, w1, mbar, b, prev, vocab): + """Request b's markov_w1[prev] row into shared memory: one 512-byte bulk copy on the position's barrier phase (a + prev outside the vocabulary copies row 0 and clears s_valid[b]: the readers zero its bias). s_valid[b] is stored + before the arrive, whose release the readers' phase wait acquires.""" + valid = (prev >= cutlass.Int32(0)) & (prev < vocab) + row = cutlass.Int32(cutlass.select_(valid, prev, cutlass.Int32(0))) + s_valid.store(cutlass.Int32(cutlass.select_(valid, cutlass.Int32(1), cutlass.Int32(0))), idx=b) + prims.mbarrier_arrive_expect_tx(mbar, ROW_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_w1.subview(b * cutlass.Int32(ROW_WORDS)), + w1.subview(row * cutlass.Int32(ROW_WORDS)), + mbar, + ROW_BYTES, + ) + + +def _reduce_slots(slots_arr, first_word): + """The best of 8 (value bits, index) pairs at first_word .. in warp order.""" + value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + index = cutlass.Int32(INT_MAX) + for w in range(WARPS): + pair = slots_arr.load(idx=first_word + cutlass.Int32(w * 2), vector_size=2, alignment=8) + value, index = _keep_better( + value, index, cutlass.Int32(pair[0]).bitcast(cutlass.Float32), cutlass.Int32(pair[1]) + ) + return value, index + + +def ll_region_words(block: int, slots: int, grid: int) -> int: + """Int32 words of one LL region a call uses: one 4-word entry per position, rank slot and CTA (batch 1).""" + return block * slots * grid * LL_ENTRY_WORDS + + +def ll_workspace_words(buf_words: int) -> int: + """Int32 words of the whole workspace with the LL regions: the Lamport buffers, then LL_PARITIES regions.""" + return (BUFFERS + LL_PARITIES) * buf_words + + +@dsl_user_op +def _post_ll(addr_i64, a, b, c, d, *, loc=None, ip=None): + """multimem.st.relaxed.sys.global.v4.f32: one 16-byte store of {a, b, c, d} into every rank's copy (the bits, as + stored); relaxed at system scope so the peers' volatile polls observe it, with no release (nothing else is + published through it).""" + _llvm.inline_asm( + None, + [addr_i64.ir_value(loc=loc, ip=ip), a.ir_value(loc=loc, ip=ip), b.ir_value(loc=loc, ip=ip), + c.ir_value(loc=loc, ip=ip), d.ir_value(loc=loc, ip=ip)], + "multimem.st.relaxed.sys.global.v4.f32 [$0], {$1, $2, $3, $4};", "l,r,r,r,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bar_sync(bar_id: int, threads: int, *, loc=None, ip=None): + """bar.sync on a named barrier: wait until `threads` threads arrived (or synced); memory ordering as bar.sync.""" + _llvm.inline_asm( + None, [], f"bar.sync {bar_id}, {threads};", "", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bar_arrive(bar_id: int, threads: int, *, loc=None, ip=None): + """bar.arrive on a named barrier: count this warp's arrival without waiting (the barrier's completion orders the + arriving threads' prior memory accesses before the waiting threads' later ones, as bar.sync).""" + _llvm.inline_asm( + None, [], f"bar.arrive {bar_id}, {threads};", "", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@cute.kernel +def k3_markov_kernel( + base: cutlass.Array, # fp32 [K * S] (or its int16 bf16 view with base_bf16): the draft head's shard logits + first_prev: cutlass.Array, # int64 [1]: the anchor token + w1: cutlass.Array, # int32 words of bf16 markov_w1 [V, 256] + w2: cutlass.Array, # int32 words of bf16 markov_w2[shard rows] [S, 256] + corrected: cutlass.Array, # unused (kept for the launch signature) + tokens: cutlass.Array, # int32 [K], out + next_new: cutlass.Array, # int32 [K + 1], out + accepted: cutlass.Array, # int32 [*, accepted_stride]: accepted tokens + num_accepted: cutlass.Array, # int32 [1] + accepted_rows: cutlass.Array, # int32 [1]: the row of accepted of the request + kv_lens: cutlass.Array, # int32: the target's KV lengths, rewound here after the draft forward + rewind: cutlass.Array, # int32 [rewind_count]: subtracted from kv_lens[rewind_first ...], clamped at 0 + buf_uc: cutlass.Array, # int32 words: this rank's workspace (Lamport buffers, then the LL regions) + buf_mc: cutlass.Array, # int32 words: its multicast mapping + flags: cutlass.Array, # int32 [LL_FLAG_WORDS] + vocab: cutlass.Int32, # rows of markov_w1: a prev outside [0, vocab) gets a zero bias + vocab_offset: cutlass.Int32, # first vocabulary index of this rank's shard + rank: cutlass.Int32, + accepted_stride: cutlass.Int32, + rewind_first: cutlass.Int32, + rewind_count: cutlass.Int32, # 0: no rewind + grid: cutlass.Constexpr[int], + rows: cutlass.Constexpr[int], # RC + shard: cutlass.Constexpr[int], # S + block: cutlass.Constexpr[int], # K + batch: cutlass.Constexpr[int], # B: 1 (the op takes this variant only for batch 1) + slots: cutlass.Constexpr[int], # rank slots per k: world x push_copies + push_copies: cutlass.Constexpr[ + int + ], # test only: each rank's entry goes into push_copies consecutive slots + buf_words: cutlass.Constexpr[int], + base_bf16: cutlass.Constexpr[bool], + fp32_sum: cutlass.Constexpr[bool], +): + rows_per_warp = rows // WARPS + blocks_per_warp = rows_per_warp // BLOCK_ROWS + entries = slots * grid # LL entries per position + vec_per_lane = (entries + 31) // 32 # 16-byte entry loads per lane of warp 0 + ll_base = BUFFERS * buf_words # the first LL region + + tx, _, _ = cute.arch.thread_idx() + cta, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tx % cutlass.Int32(32) + + s_w2 = cutlass.Array( + cutlass.Int32, rows * ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128 + ) + # markov_w1[prev] of the position (one bulk copy, issued by thread 0 once it learns prev). + s_w1 = cutlass.Array(cutlass.Int32, ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128) + s_mbar = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + s_mbar_w1 = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + # [warp] (value bits, index): the CTA reduction of the rows. + s_row_best = cutlass.Array( + cutlass.Int32, WARPS * 2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_flags = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_valid = cutlass.Array(cutlass.Int32, MAX_BATCH, space=cutlass.AddressSpace.smem, alignment=16) + + if warp == 0: + if prims.elect_sync(): + prims.mbarrier_init(s_mbar, 1) + prims.mbarrier_init(s_mbar_w1, 1) + prims.fence_mbarrier_init() + prims.barrier_cta_sync(0) + + # The CTA's markov_w2 rows: a weight, loaded before the grid dependency. + if warp == 0: + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(s_mbar, rows * ROW_BYTES) + for c in cutlass.range_constexpr((rows + COPY_ROWS - 1) // COPY_ROWS): + n = min(COPY_ROWS, rows - c * COPY_ROWS) + prims.cp_async_bulk_shared_cluster_global( + s_w2.subview(c * COPY_ROWS * ROW_WORDS), + w2.subview( + cta * cutlass.Int32(rows * ROW_WORDS) + + cutlass.Int32(c * COPY_ROWS * ROW_WORDS) + ), + s_mbar, + n * ROW_BYTES, + ) + # Dependents wait for this whole grid before reading its outputs. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + + prims.griddepcontrol(prims.GridDepAction.WAIT) + if tx == 0: + s_flags.store(flags.load(idx=LL_SEQ, is_volatile=True), idx=0) + _fetch_w1_row( + s_w1, s_valid, w1, s_mbar_w1, tx, cutlass.Int32(first_prev.load(idx=0)), vocab + ) + prims.barrier_cta_sync(0) + # This call's tag: the last LL call's seq + 1, skipping 0 (the zero fill) on wrap; its region: seq & 1. + seq = s_flags.load(idx=0) + cutlass.Int32(1) + seq = cutlass.Int32(cutlass.select_(seq == cutlass.Int32(0), cutlass.Int32(2), seq)) + region0 = cutlass.Int32(ll_base) + (seq & cutlass.Int32(1)) * cutlass.Int32(buf_words) + + # next_new_tokens[0] = [accepted[row_0, num_accepted[0] - 1], tokens[0, :]] (-1 wraps, as torch indexing). + if cta == 0: + if tx == 0: + col = num_accepted.load(idx=0) - cutlass.Int32(1) + col = cutlass.Int32(cutlass.select_(col < cutlass.Int32(0), col + accepted_stride, col)) + row = accepted_rows.load(idx=0) + next_new.store(accepted.load(idx=row * accepted_stride + col), idx=0) + # The KV-length rewind after the draft forward (whose kernels, the readers of kv_lens, all finished before + # this grid's wait): kv_lens[first + b] = max(kv_lens[first + b] - rewind[b], 0). + if tx < rewind_count: + left = kv_lens.load(idx=rewind_first + tx) - rewind.load(idx=tx) + kv_lens.store(cutlass.Int32(cutlass.select_(left < cutlass.Int32(0), cutlass.Int32(0), left)), + idx=rewind_first + tx) # fmt: skip + + # The weights have landed (issued before the wait). + while not cute.arch.mbarrier_test_wait(s_mbar.data_ptr(), 0): + pass + + row0 = warp * cutlass.Int32(rows_per_warp) # first row of the warp within the CTA + shard_row0 = cta * cutlass.Int32(rows) + row0 # ... within the shard + has_row = (lane & cutlass.Int32(3)) == cutlass.Int32(0) + lane_row = shard_row0 + (lane >> cutlass.Int32(2)) + empty = cutlass.Int32(EMPTY_WORD) + for k in cutlass.range(block, unroll=1): + # The position's base logits, issued before the W1 wait (b1's order). + base_row = k * cutlass.Int32(shard) + # Lane 4j finishes row j of each 8-row block (the reduce-scatter leaves row j's sum on lanes 4j .. 4j+3). + base_vals = [] + for g in cutlass.range_constexpr(blocks_per_warp): + if cutlass.const_expr(base_bf16): + base_vals.append( + cutlass.Int32( + cutlass.Int32( + base.load(idx=base_row + lane_row + cutlass.Int32(g * BLOCK_ROWS)) + ) + << cutlass.Int32(16) + ).bitcast(cutlass.Float32) + ) + else: + base_vals.append(base.load(idx=base_row + lane_row + cutlass.Int32(g * BLOCK_ROWS))) + # markov_w1[prev] landed (the position's phase of the W1 barrier). + while not cute.arch.mbarrier_test_wait(s_mbar_w1.data_ptr(), k & cutlass.Int32(1)): + pass + valid = s_valid.load(idx=0) != cutlass.Int32(0) + # markov_w1[prev], columns 8 lane .. 8 lane + 7, as fp32 (zeros for an invalid prev). + w1v = s_w1.load(idx=lane * 4, vector_size=4, alignment=16) + w1f = [] + for q in cutlass.range_constexpr(4): + word = cutlass.Int32(cutlass.select_(valid, cutlass.Int32(w1v[q]), cutlass.Int32(0))) + w1f.append(_lo_f32(word)) + w1f.append(_hi_f32(word)) + best_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + best_index = cutlass.Int32(INT_MAX) + for g in cutlass.range_constexpr(blocks_per_warp): + dot = _block_dot(s_w2, row0 + cutlass.Int32(g * BLOCK_ROWS), lane, w1f) + bias = _lo_f32(_pack_bf16x2(dot, dot)) + # fp32_sum (default): the in-tree chain's arithmetic (fp32 logits from LogitsProcessor's .float() plus + # the widened bf16 bias): compare the unrounded fp32 sum. Without fp32_sum: a bf16 reference rounds that + # sum once. + if cutlass.const_expr(fp32_sum): + value = _add_rn(base_vals[g], bias) + else: + value = _lo_f32( + _pack_bf16x2(_add_rn(base_vals[g], bias), _add_rn(base_vals[g], bias)) + ) + if has_row: + my_row = lane_row + cutlass.Int32(g * BLOCK_ROWS) + best_value, best_index = _keep_better( + best_value, best_index, value, vocab_offset + my_row + ) + best_value, best_index = _warp_best(best_value, best_index) + if lane == 0: + s_row_best.store((best_value.bitcast(cutlass.Int32), best_index), idx=warp * cutlass.Int32(2), + alignment=8) # fmt: skip + if warp == 0: + # Every warp's best of this position is in shared memory. + _bar_sync(LL_BAR, THREADS) + region = region0 + k * cutlass.Int32(entries * LL_ENTRY_WORDS) + # Thread 0 posts the CTA entry {value bits, tag, index, tag} into every rank's region: one 16-byte + # multicast store, no fence. + if tx == 0: + cta_value, cta_index = _reduce_slots(s_row_best, cutlass.Int32(0)) + bits = cta_value.bitcast(cutlass.Int32) + # -0.0 is posted as +0.0, as b1 posts it (equal under the order either way). + bits = cutlass.Int32(cutlass.select_(bits == empty, cutlass.Int32(0), bits)) + for c in cutlass.range_constexpr(push_copies): + s_idx = rank * cutlass.Int32(push_copies) + cutlass.Int32(c) + word = region + (s_idx * cutlass.Int32(grid) + cta) * cutlass.Int32( + LL_ENTRY_WORDS + ) + _post_ll(buf_mc.data_ptr(word).toint(), bits, seq, cta_index, seq) + # Warp 0 polls all W x G entries of the position in this rank's copy: lane l reads entries l, l + 32, .. + # (every load of a pass issued before any is used); a pass in which any entry of the lane does not + # carry this call's tag in both halves is repeated in full. The order is total, so the lane best and + # the xor tree give b1's winner. + pending = cutlass.Boolean(True) + g_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + g_index = cutlass.Int32(INT_MAX) + while pending: + gots = [] + for i in cutlass.range_constexpr(vec_per_lane): + e_idx = lane + cutlass.Int32(i * 32) + live = e_idx < cutlass.Int32(entries) + gots.append((live, buf_uc.load( + idx=region + cutlass.Int32(cutlass.select_(live, e_idx, cutlass.Int32(0))) + * cutlass.Int32(LL_ENTRY_WORDS), + vector_size=4, alignment=16, is_volatile=True))) # fmt: skip + pass_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) + pass_index = cutlass.Int32(INT_MAX) + pass_pending = cutlass.Boolean(False) + for live, got in gots: + vw = cutlass.Int32(got[0]) + iw = cutlass.Int32(got[2]) + tagged = (cutlass.Int32(got[1]) == seq) & (cutlass.Int32(got[3]) == seq) + stale = (cutlass.Int32(got[1]) != seq) | (cutlass.Int32(got[3]) != seq) + pass_pending = pass_pending | (live & stale) + take = ( + live + & tagged + & _better(vw.bitcast(cutlass.Float32), iw, pass_value, pass_index) + ) + pass_value = cutlass.Float32( + cutlass.select_(take, vw.bitcast(cutlass.Float32), pass_value) + ) + pass_index = cutlass.Int32(cutlass.select_(take, iw, pass_index)) + g_value = pass_value + g_index = pass_index + pending = pass_pending + g_value, g_index = _warp_best(g_value, g_index) + if tx == 0: + # The next position's markov_w1 row, requested as soon as prev is known: every reader of s_w1 and + # s_valid for this position arrived at LL_BAR after its reads. The other warps wait for the row's + # phase before writing s_row_best again, and this thread read s_row_best before the arrive. + if k + cutlass.Int32(1) < cutlass.Int32(block): + _fetch_w1_row(s_w1, s_valid, w1, s_mbar_w1, tx, g_index, vocab) + if cta == 0: + tokens.store(g_index, idx=k) + next_new.store(g_index, idx=k + cutlass.Int32(1)) + else: + _bar_arrive(LL_BAR, THREADS) + + # Every CTA read the flags before arriving; the last one advances the LL seq for the next call. + if tx == 0: + arrived = _atomic_add_acq_rel(flags.data_ptr(LL_ARRIVED).toint(), cutlass.Int32(1)) + if arrived == cutlass.Int32(grid - 1): + flags.store(cutlass.Int32(0), idx=LL_ARRIVED) + flags.store(seq, idx=LL_SEQ) + + +@cute.jit +def k3_markov( + base: cute.Tensor, + first_prev: cute.Tensor, + w1: cute.Tensor, + w2: cute.Tensor, + corrected: cute.Tensor, + tokens: cute.Tensor, + next_new: cute.Tensor, + accepted: cute.Tensor, + num_accepted: cute.Tensor, + accepted_rows: cute.Tensor, + kv_lens: cute.Tensor, + rewind: cute.Tensor, + buf_uc: cute.Tensor, + buf_mc: cute.Tensor, + flags: cute.Tensor, + vocab: cutlass.Int32, + vocab_offset: cutlass.Int32, + rank: cutlass.Int32, + accepted_stride: cutlass.Int32, + rewind_first: cutlass.Int32, + rewind_count: cutlass.Int32, + grid: cutlass.Constexpr[int], + rows: cutlass.Constexpr[int], + shard: cutlass.Constexpr[int], + block: cutlass.Constexpr[int], + batch: cutlass.Constexpr[int], + slots: cutlass.Constexpr[int], + push_copies: cutlass.Constexpr[int], + buf_words: cutlass.Constexpr[int], + base_bf16: cutlass.Constexpr[bool], + fp32_sum: cutlass.Constexpr[bool], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + k3_markov_kernel( + base, first_prev, w1, w2, corrected, tokens, next_new, accepted, num_accepted, accepted_rows, kv_lens, + rewind, buf_uc, buf_mc, flags, vocab, vocab_offset, rank, accepted_stride, rewind_first, rewind_count, + grid, rows, shard, block, batch, slots, push_copies, buf_words, base_bf16, fp32_sum, + ).launch( + grid=[grid, 1, 1], + block=[THREADS, 1, 1], + stream=stream, + use_pdl=use_pdl, + ) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_qk_norm_rope.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_qk_norm_rope.py new file mode 100644 index 000000000000..08b6630400ab --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_qk_norm_rope.py @@ -0,0 +1,309 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# DSpark / DFlash drafter block: q / k RMSNorm + partial NeoX RoPE in one launch +# ============================================================================= +# +# It replaces, bit for bit, the three launches of DFlashForCausalLM.dflash_forward's flashinfer-RoPE branch at head_dim +# 128: flashinfer_rmsnorm on the q heads, flashinfer_rmsnorm on the k heads (both flashinfer's CuTe RMSNormKernel), +# then flashinfer_apply_rope_with_cos_sin_cache_inplace (NeoX, the cache's leading ROT dims rotated, the rest kept). +# The arithmetic is flashinfer's, made explicit (every op inline PTX, so nothing is contracted or reordered): +# RMSNorm: 16 threads x 8 contiguous elements per row; each thread sums its x^2 sequentially from 0; the 16 sums are +# added by the butterfly at offsets 1, 2, 4, 8 (adjacent pairs, then pairs of pairs); mean = sum * (1 / 128); +# rstd = rsqrt.approx.ftz(mean + eps); y = bf16((x * rstd) * w). +# RoPE (on the bf16 norm output, cos / sin fp32 from the cache): out_i = fma(x_i, cos_i, -x_{i+R/2} * sin_i) and +# out_{i+R/2} = fma(x_{i+R/2}, cos_i, x_i * sin_i) for i < R / 2, rounded to bf16. +# Measured against the stock launches: 0 mismatching elements at 4 and 32 rows. +# +# DRAFT_APPEND_FUSE (``qk_norm_rope_append``): the same launch also writes each row's K (the bits it returns) and V +# (copied from qkv) into the layer's paged HND cache [pages, 2, NKV, PS, HD] where flashinfer.page.append_paged_kv_cache +# would: page kv_indices[kv_indptr[b] + pos // PS], slot pos % PS, with b = batch_indices[row] and pos = positions[row]. +# The caller then skips that append launch; the cache bits are the same. + +import torch +import triton +import triton.language as tl + +HEAD_DIM = 128 + + +@triton.jit +def _add(a, b): + return tl.inline_asm_elementwise( + "add.rn.f32 $0, $1, $2;", "=r,r,r", [a, b], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _mul(a, b): + return tl.inline_asm_elementwise( + "mul.rn.f32 $0, $1, $2;", "=r,r,r", [a, b], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _fma(a, b, c): + return tl.inline_asm_elementwise( + "fma.rn.f32 $0, $1, $2, $3;", "=r,r,r,r", [a, b, c], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _rsqrt(a): + return tl.inline_asm_elementwise( + "rsqrt.approx.ftz.f32 $0, $1;", "=r,r", [a], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _pick(v, idx, j): + """Column j of a [R, C] tile as [R] (exact: one nonzero term).""" + return tl.sum(tl.where(idx == j, v, 0.0), axis=1) + + +@triton.jit +def _pair_sum(v, n: tl.constexpr): + """[n] -> [n / 2]: element k = v[2k] + v[2k + 1].""" + v2 = tl.reshape(v, (n // 2, 2)) + idx = tl.arange(0, 2)[None, :] + tl.arange(0, n // 2)[:, None] * 0 + return _add(_pick(v2, idx, 0), _pick(v2, idx, 1)) + + +@triton.jit +def _qk_norm_rope_kernel( + qkv_ptr, + qkv_stride, + pos_ptr, + cs_ptr, + cs_stride, + wq_ptr, + wk_ptr, + q_out, + k_out, + eps_q, + eps_k, + cache_ptr, + cache_page_stride, + bidx_ptr, + apos_ptr, + kvi_ptr, + kvp_ptr, + v_col, + HQ: tl.constexpr, + HKV: tl.constexpr, + HD: tl.constexpr, + HALF: tl.constexpr, + PS: tl.constexpr, + STORE: tl.constexpr, +): + pid = tl.program_id(0) + t = pid // (HQ + HKV) + h = pid % (HQ + HKV) + is_q = h < HQ + base = qkv_ptr + t * qkv_stride + h * HD + r16 = tl.arange(0, 16) + c8 = tl.arange(0, 8) + x2 = tl.load(base + r16[:, None] * 8 + c8[None, :]).to(tl.float32) + sq = _mul(x2, x2) + col = c8[None, :] + r16[:, None] * 0 + s = tl.zeros([16], dtype=tl.float32) + for j in tl.static_range(8): + s = _add(s, _pick(sq, col, j)) + s8 = _pair_sum(s, 16) + s4 = _pair_sum(s8, 8) + s2 = _pair_sum(s4, 4) + two = tl.arange(0, 2) + total = _add(tl.sum(tl.where(two == 0, s2, 0.0)), tl.sum(tl.where(two == 1, s2, 0.0))) + eps = tl.where(is_q, eps_q, eps_k) + rstd = _rsqrt(_add(_mul(total, 1.0 / HD), eps)) + d = tl.arange(0, HD) + x = tl.load(base + d).to(tl.float32) + w = tl.where(is_q, tl.load(wq_ptr + d), tl.load(wk_ptr + d)).to(tl.float32) + y = _mul(_mul(x, rstd), w).to(tl.bfloat16) + pos = tl.load(pos_ptr + t) + i = tl.arange(0, HALF) + cos = tl.load(cs_ptr + pos * cs_stride + i) + sin = tl.load(cs_ptr + pos * cs_stride + HALF + i) + yf = y.to(tl.float32) + x1 = tl.sum(tl.where(d[None, :] == i[:, None], yf[None, :], 0.0), axis=1) + x2r = tl.sum(tl.where(d[None, :] == (i[:, None] + HALF), yf[None, :], 0.0), axis=1) + o1 = _fma(x1, cos, _mul(-x2r, sin)) + o2 = _fma(x2r, cos, _mul(x1, sin)) + obase = tl.where(is_q, q_out + t * (HQ * HD) + h * HD, k_out + t * (HKV * HD) + (h - HQ) * HD) + tl.store(obase + d, y, mask=d >= 2 * HALF) + tl.store(obase + i, o1.to(tl.bfloat16)) + tl.store(obase + HALF + i, o2.to(tl.bfloat16)) + if STORE: + # K / V heads only: K's bits as stored above, V copied from qkv, at the append's page and slot. + is_k = h >= HQ + hk = tl.where(is_k, h - HQ, 0) + b = tl.load(bidx_ptr + t).to(tl.int64) + p = tl.load(apos_ptr + t).to(tl.int64) + page = tl.load(kvi_ptr + tl.load(kvp_ptr + b).to(tl.int64) + p // PS).to(tl.int64) + slot = p % PS + kdst = cache_ptr + page * cache_page_stride + (hk * PS + slot) * HD + vdst = cache_ptr + page * cache_page_stride + ((HKV + hk) * PS + slot) * HD + tl.store(kdst + d, y, mask=(d >= 2 * HALF) & is_k) + tl.store(kdst + i, o1.to(tl.bfloat16), mask=is_k) + tl.store(kdst + HALF + i, o2.to(tl.bfloat16), mask=is_k) + v = tl.load(qkv_ptr + t * qkv_stride + v_col + hk * HD + d, mask=is_k & (d >= 0), other=0.0) + tl.store(vdst + d, v, mask=is_k & (d >= 0)) + + +def _plain_rmsnorm(norm) -> bool: + """The stock q_norm / k_norm takes flashinfer's plain (non-Gemma) RMSNorm on a bf16 input.""" + return ( + getattr(norm, "use_gemma", True) is False + and not getattr(norm, "use_cuda_tile", False) + and not getattr(norm, "is_nvfp4", False) + and not getattr(norm, "return_hp_output", False) + and getattr(norm, "weight", None) is not None + and norm.weight.dtype == torch.bfloat16 + and norm.weight.shape == (HEAD_DIM,) + ) + + +def qk_norm_rope(qkv, q_size, kv_size, head_dim, q_norm, k_norm, positions, cos_sin_cache, is_neox): + """q / k RMSNorm + RoPE of the drafter block in one launch: returns new contiguous (q [T, q_size], k [T, kv_size]) + bitwise equal to the stock flashinfer path, or None (nothing computed) for anything it does not reproduce.""" + if not qk_norm_rope_supported( + qkv, q_size, kv_size, head_dim, q_norm, k_norm, positions, cos_sin_cache, is_neox + ): + return None + return _launch(qkv, q_size, kv_size, q_norm, k_norm, positions, cos_sin_cache) + + +def qk_norm_rope_supported( + qkv, q_size, kv_size, head_dim, q_norm, k_norm, positions, cos_sin_cache, is_neox +) -> bool: + """Whether the one-launch q / k RMSNorm + RoPE reproduces the stock path for these operands.""" + from tensorrt_llm._torch.flashinfer_utils import IS_FLASHINFER_AVAILABLE + + return bool( + IS_FLASHINFER_AVAILABLE + and is_neox + and head_dim == HEAD_DIM + and qkv.dtype == torch.bfloat16 + and qkv.dim() == 2 + and qkv.stride(1) == 1 + and qkv.shape[1] >= q_size + kv_size + and q_size % head_dim == 0 + and kv_size % head_dim == 0 + and positions.dtype == torch.int32 + and positions.dim() == 1 + and positions.shape[0] == qkv.shape[0] + and cos_sin_cache.dtype == torch.float32 + and cos_sin_cache.dim() == 2 + and cos_sin_cache.stride(1) == 1 + and cos_sin_cache.shape[1] in (16, 32, 64, 128) + and _plain_rmsnorm(q_norm) + and _plain_rmsnorm(k_norm) + ) + + +def _launch(qkv, q_size, kv_size, q_norm, k_norm, positions, cos_sin_cache, append=None): + T = qkv.shape[0] + hq, hkv = q_size // HEAD_DIM, kv_size // HEAD_DIM + q = torch.empty(T, q_size, dtype=torch.bfloat16, device=qkv.device) + k = torch.empty(T, kv_size, dtype=torch.bfloat16, device=qkv.device) + if T == 0: + return q, k + if append is None: + cache, bidx, apos, kvi, kvp, ps, store = ( + q, + positions, + positions, + positions, + positions, + 1, + False, + ) + else: + cache, bidx, apos, kvi, kvp = append + ps, store = cache.shape[-2], True + _qk_norm_rope_kernel[(T * (hq + hkv),)]( + qkv, + qkv.stride(0), + positions, + cos_sin_cache, + cos_sin_cache.stride(0), + q_norm.weight, + k_norm.weight, + q, + k, + float(q_norm.variance_epsilon), + float(k_norm.variance_epsilon), + cache, + cache.stride(0) if store else 0, + bidx, + apos, + kvi, + kvp, + q_size + kv_size, + HQ=hq, + HKV=hkv, + HD=HEAD_DIM, + HALF=cos_sin_cache.shape[1] // 2, + PS=ps, + STORE=store, + num_warps=1, + ) + return q, k + + +_INT = (torch.int32, torch.int64) + + +def append_supported( + qkv, kv_size, paged_kv_cache, batch_indices, positions, kv_indices, kv_indptr +) -> bool: + """Whether ``qk_norm_rope_append`` reproduces flashinfer.page.append_paged_kv_cache for these operands: a + contiguous bf16 HND layer cache [pages, 2, NKV, PS, HEAD_DIM] with NKV = kv_size / HEAD_DIM, one append row per qkv + row, integer index tensors on the same device.""" + c = paged_kv_cache + T = qkv.shape[0] + return ( + isinstance(c, torch.Tensor) + and c.dim() == 5 + and c.dtype == torch.bfloat16 + and c.is_contiguous() + and c.device == qkv.device + and c.shape[1] == 2 + and c.shape[2] == kv_size // HEAD_DIM + and c.shape[4] == HEAD_DIM + and c.shape[3] >= 1 + and all( + x.dtype in _INT and x.device == qkv.device and x.dim() == 1 and x.is_contiguous() + for x in (batch_indices, positions, kv_indices, kv_indptr) + ) + and batch_indices.shape[0] == T + and positions.shape[0] == T + ) + + +def qk_norm_rope_append(qkv, q_size, kv_size, head_dim, q_norm, k_norm, positions, cos_sin_cache, is_neox, + paged_kv_cache, append_batch_indices, append_positions, kv_indices, kv_indptr): # fmt: skip + """``qk_norm_rope`` that also appends the rows' K / V into ``paged_kv_cache`` exactly as + flashinfer.page.append_paged_kv_cache(k, v, append_batch_indices, append_positions, paged_kv_cache, kv_indices, + kv_indptr, ..., kv_layout="HND") would (same bits); None (nothing computed or written) when either part is not + reproduced. The caller skips its append on a non-None return.""" + if not qk_norm_rope_supported(qkv, q_size, kv_size, head_dim, q_norm, k_norm, positions, cos_sin_cache, + is_neox): # fmt: skip + return None + if not append_supported(qkv, kv_size, paged_kv_cache, append_batch_indices, append_positions, kv_indices, + kv_indptr): # fmt: skip + return None + return _launch(qkv, q_size, kv_size, q_norm, k_norm, positions, cos_sin_cache, + (paged_kv_cache, append_batch_indices, append_positions, kv_indices, kv_indptr)) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept.py new file mode 100644 index 000000000000..eda460592842 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept.py @@ -0,0 +1,482 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The strict draft acceptance and the next step's input tokens of a decode-only speculative step, in one launch each. + +The in-tree strict path verifies ``D`` draft tokens per request against the target's sampled tokens with about ten +small torch launches (empty / ones / reshape / == / int / cumprod / sum / += / slice copy), and builds the next +step's input tokens with three more (index, unsqueeze, concat). Both are integer copies and compares, so these +kernels are bitwise to that path by construction; the forced-acceptance override stays the caller's (it advances +the in-tree RNG pool and counters). + +Compiled on the first call per (draft length, dtypes), which must run outside CUDA-graph capture (``precompile``). +""" + +from __future__ import annotations + +import importlib +import threading +from typing import Dict, Sequence + +import torch + +from .._op import Arch, Cell, OpWrapper + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} +_modules: Dict[str, object] = {} + + +def _kernel_module(): + """The CuTe DSL kernels beside this file, imported on first use (they pull in cutlass).""" + mod = _modules.get("kernel") + if mod is None: + mod = _modules["kernel"] = importlib.import_module(".strict_accept_kernel", __package__) + return mod + + +def _arg(t: torch.Tensor, align: int): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align).mark_layout_dynamic(leading_dim=0) + + +def _int_align(t: torch.Tensor) -> int: + return t.element_size() + + +def max_requests() -> int: + return _kernel_module().MAX_REQUESTS + + +def _compiled_fn(name: str, key: tuple, args: tuple, scalars: tuple, consts: tuple, stream): + fn = _compiled.get(key) + if fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + f"trtllm::{name} must run once per shape outside CUDA-graph capture first" + ) + import cutlass.cute as cute + + with _lock: + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute.compile( + getattr(_kernel_module(), name), *args, *scalars, *consts, stream + ) + return fn + + +@torch.library.custom_op("trtllm::spec_strict_accept", mutates_args=()) +def _spec_strict_accept(target: torch.Tensor, draft: torch.Tensor) -> list[torch.Tensor]: + import cuda.bindings.driver as cuda_driver + + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + n_req, draft_len = draft.shape + if target.numel() != n_req * (draft_len + 1): + raise ValueError( + f"spec_strict_accept: target {tuple(target.shape)} vs draft {tuple(draft.shape)}" + ) + if n_req > max_requests() or not target.is_contiguous() or not draft.is_contiguous(): + raise ValueError("spec_strict_accept: unsupported operands (see supports())") + device = target.device + accepted = torch.empty(n_req, draft_len + 1, dtype=torch.int32, device=device) + num_accepted = torch.empty(n_req, dtype=torch.int32, device=device) + args = ( + _arg(target.view(-1), _int_align(target)), + _arg(draft.view(-1), _int_align(draft)), + _arg(accepted.view(-1), 4), + _arg(num_accepted, 4), + ) + use_pdl = get_env_enable_pdl() + stream = cuda_driver.CUstream(torch.cuda.current_stream(device).cuda_stream) + key = ("strict_accept", target.dtype, draft.dtype, draft_len, use_pdl) + fn = _compiled_fn("strict_accept", key, args, (n_req,), (draft_len, use_pdl), stream) + fn(*args, n_req, stream) + return [accepted, num_accepted] + + +@_spec_strict_accept.register_fake +def _(target, draft): + n_req, draft_len = draft.shape + return [ + target.new_empty((n_req, draft_len + 1), dtype=torch.int32), + target.new_empty((n_req,), dtype=torch.int32), + ] + + +@torch.library.custom_op("trtllm::spec_next_new_tokens", mutates_args=()) +def _spec_next_new_tokens( + accepted: torch.Tensor, + num_accepted: torch.Tensor, + batch_idx: torch.Tensor, + next_draft: torch.Tensor, +) -> torch.Tensor: + import cuda.bindings.driver as cuda_driver + + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + n_req, draft_len = next_draft.shape + if n_req > max_requests() or accepted.stride(1) != 1 or next_draft.stride(1) != 1: + raise ValueError("spec_next_new_tokens: unsupported operands (see supports())") + device = accepted.device + out = torch.empty( + n_req, + draft_len + 1, + dtype=torch.promote_types(accepted.dtype, next_draft.dtype), + device=device, + ) + # Flat views over the storage each row index reads (strided rows: the row strides are passed). + acc_flat = accepted.as_strided( + ((accepted.shape[0] - 1) * accepted.stride(0) + accepted.shape[1],), (1,) + ) + nd_flat = next_draft.as_strided(((n_req - 1) * next_draft.stride(0) + draft_len,), (1,)) + args = ( + _arg(acc_flat, _int_align(accepted)), + _arg(num_accepted, _int_align(num_accepted)), + _arg(batch_idx, _int_align(batch_idx)), + _arg(nd_flat, _int_align(next_draft)), + _arg(out.view(-1), _int_align(out)), + ) + scalars = (int(accepted.stride(0)), int(next_draft.stride(0)), n_req) + use_pdl = get_env_enable_pdl() + stream = cuda_driver.CUstream(torch.cuda.current_stream(device).cuda_stream) + key = ( + "next_new_tokens", + accepted.dtype, + num_accepted.dtype, + batch_idx.dtype, + next_draft.dtype, + draft_len, + use_pdl, + ) + fn = _compiled_fn("next_new_tokens", key, args, scalars, (draft_len, use_pdl), stream) + fn(*args, *scalars, stream) + return out + + +@_spec_next_new_tokens.register_fake +def _(accepted, num_accepted, batch_idx, next_draft): + n_req, draft_len = next_draft.shape + return accepted.new_empty( + (n_req, draft_len + 1), dtype=torch.promote_types(accepted.dtype, next_draft.dtype) + ) + + +@torch.library.custom_op("trtllm::spec_kv_rewind", mutates_args=("kv_lens", "rewind")) +def _spec_kv_rewind( + num_accepted: torch.Tensor, kv_lens: torch.Tensor, rewind: torch.Tensor, nc: int, n: int +) -> None: + import cuda.bindings.driver as cuda_driver + + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + if not 0 < n <= max_requests() or rewind.numel() < n or rewind.dtype != num_accepted.dtype: + raise ValueError("spec_kv_rewind: unsupported operands (see supports_kv_rewind())") + device = num_accepted.device + args = ( + _arg(num_accepted, _int_align(num_accepted)), + _arg(kv_lens, _int_align(kv_lens)), + _arg(rewind, _int_align(rewind)), + ) + use_pdl = get_env_enable_pdl() + stream = cuda_driver.CUstream(torch.cuda.current_stream(device).cuda_stream) + key = ("kv_rewind", num_accepted.dtype, kv_lens.dtype, use_pdl) + fn = _compiled_fn("kv_rewind", key, args, (int(nc), int(n)), (use_pdl,), stream) + fn(*args, int(nc), int(n), stream) + + +@_spec_kv_rewind.register_fake +def _(num_accepted, kv_lens, rewind, nc, n): + return None + + +_INT_DTYPES = (torch.int32, torch.int64) + + +def supports_strict_accept(target: torch.Tensor, draft: torch.Tensor) -> bool: + """Whether the kernel takes these operands; a caller that gets False runs the in-tree path.""" + return ( + target.is_cuda + and draft.is_cuda + and target.dtype in _INT_DTYPES + and draft.dtype in _INT_DTYPES + and draft.dim() == 2 + and 0 < draft.shape[0] <= max_requests() + and draft.shape[1] > 0 + and target.numel() == draft.shape[0] * (draft.shape[1] + 1) + and target.is_contiguous() + and draft.is_contiguous() + ) + + +def supports_next_new_tokens( + accepted: torch.Tensor, + num_accepted: torch.Tensor, + batch_idx: torch.Tensor, + next_draft: torch.Tensor, +) -> bool: + """Whether the kernel takes these operands; a caller that gets False runs the in-tree path.""" + return ( + all( + t.is_cuda and t.dtype in _INT_DTYPES + for t in (accepted, num_accepted, batch_idx, next_draft) + ) + and accepted.dim() == 2 + and next_draft.dim() == 2 + and accepted.stride(1) == 1 + and next_draft.stride(1) == 1 + and 0 < next_draft.shape[0] <= max_requests() + and num_accepted.dim() == 1 + and batch_idx.dim() == 1 + and num_accepted.is_contiguous() + and batch_idx.is_contiguous() + and num_accepted.numel() >= next_draft.shape[0] + and batch_idx.numel() >= next_draft.shape[0] + ) + + +def supports_kv_rewind( + num_accepted: torch.Tensor, kv_lens: torch.Tensor, rewind: torch.Tensor, nc: int, n: int +) -> bool: + """Whether the kernel takes these operands; a caller that gets False runs the in-tree lines.""" + return ( + all(t.is_cuda and t.dtype in _INT_DTYPES for t in (num_accepted, kv_lens, rewind)) + and all(t.dim() == 1 and t.is_contiguous() for t in (num_accepted, kv_lens, rewind)) + and rewind.dtype == num_accepted.dtype + and 0 < n <= max_requests() + and nc >= 0 + and num_accepted.numel() >= nc + n + and kv_lens.numel() >= nc + n + and rewind.numel() >= n + ) + + +def is_compiled_kv_rewind(num_accepted: torch.Tensor, kv_lens: torch.Tensor) -> bool: + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + return ("kv_rewind", num_accepted.dtype, kv_lens.dtype, get_env_enable_pdl()) in _compiled + + +def precompile_kv_rewind(device, dtypes: Sequence[torch.dtype] = (torch.int32,)) -> None: + """Compile the kv_rewind variants (num_accepted dtype x kv_lens dtype) outside capture.""" + for a_dtype in dtypes: + for k_dtype in dtypes: + num = torch.ones(1, dtype=a_dtype, device=device) + torch.ops.trtllm.spec_kv_rewind( + num, torch.zeros(1, dtype=k_dtype, device=device), torch.empty_like(num), 0, 1 + ) + torch.cuda.synchronize() + + +def precompile( + draft_lens: Sequence[int], device, dtypes: Sequence[torch.dtype] = _INT_DTYPES +) -> None: + """Compile every (draft length, operand dtype) variant a one-model worker passes, outside capture: target + tokens and drafts of either int dtype; int32 accepted / num_accepted (this op's) and batch indices (the spec + metadata's); next drafts of either int dtype.""" + for draft_len in draft_lens: + batch_idx = torch.zeros(1, dtype=torch.int32, device=device) + for t_dtype in dtypes: + for d_dtype in dtypes: + target = torch.zeros(draft_len + 1, dtype=t_dtype, device=device) + draft = torch.zeros(1, draft_len, dtype=d_dtype, device=device) + accepted, num = torch.ops.trtllm.spec_strict_accept(target, draft) + for d_dtype in dtypes: + draft = torch.zeros(1, draft_len, dtype=d_dtype, device=device) + torch.ops.trtllm.spec_next_new_tokens(accepted, num, batch_idx, draft) + torch.cuda.synchronize() + + +def is_compiled_strict_accept(target: torch.Tensor, draft: torch.Tensor) -> bool: + """Whether this call's variant is compiled (a captured step may only call compiled variants).""" + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + key = ("strict_accept", target.dtype, draft.dtype, draft.shape[1], get_env_enable_pdl()) + return key in _compiled + + +def is_compiled_next_new_tokens(accepted, num_accepted, batch_idx, next_draft) -> bool: + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + key = ( + "next_new_tokens", + accepted.dtype, + num_accepted.dtype, + batch_idx.dtype, + next_draft.dtype, + next_draft.shape[1], + get_env_enable_pdl(), + ) + return key in _compiled + + +class SpecStrictAccept(OpWrapper): + """Strict acceptance of a decode-only batch: ``target`` [n_req * (D + 1)] (the target's sampled token per verify + row, request-major) and ``draft`` [n_req, D] (the drafts being verified), int32 or int64. + + Returns ``(accepted [n_req, D + 1] int32, num_accepted [n_req] int32)``: the target tokens, and 1 + the length of + the prefix of drafts equal to the target's tokens. No forced-acceptance override (the caller's). + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="K3 c1: one request, int32 tokens (the fused sampler's) vs int32 drafts", + spec=dict(n_req=1, draft_len=3), + ), + Cell( + why="K3 c8: the 32-row step, mixed prefix lengths", + spec=dict(n_req=8, draft_len=3), + ), + Cell( + why="K3 at 12 requests: the 48-row bound of the decode targets", + spec=dict(n_req=12, draft_len=3), + ), + Cell( + why="K7 c1 and c8: 8 verify rows per request", + spec=dict(n_req=8, draft_len=7), + ), + Cell( + why="int64 target tokens (torch.argmax, the all-greedy path)", + spec=dict(n_req=8, draft_len=3, target_dtype=torch.int64), + ), + Cell( + why="padded / dropped drafts (-1): never equal to a token", + spec=dict(n_req=8, draft_len=3, pad_drafts=True), + ), + Cell( + why="every draft accepted, and none accepted", + spec=dict(n_req=8, draft_len=3, extremes=True), + ), + Cell( + why="the kernel's request bound (one CTA)", + spec=dict(n_req=64, draft_len=3), + ), + ) + + note = """ + Integer compares and copies only: the reference is exact, and the op must equal it bit for bit. Decode-only + batches (no context rows). The ragged verify layout and the forced-acceptance override are not this op's. + """ + + def __call__(self, target: torch.Tensor, draft: torch.Tensor): + return torch.ops.trtllm.spec_strict_accept(target, draft) + + def reference(self, target: torch.Tensor, draft: torch.Tensor): + """The in-tree strict path (``SpecWorkerBase._sample_and_accept_draft_tokens_base``, no contexts).""" + n_req, draft_len = draft.shape + accepted = torch.empty((n_req, draft_len + 1), dtype=torch.int, device=target.device) + num_accepted = torch.ones(n_req, dtype=torch.int, device=target.device) + gen_target = target.reshape(n_req, draft_len + 1) + num_accepted += torch.cumprod((draft == gen_target[:, :draft_len]).int(), dim=-1).sum(1) + accepted[:, : draft_len + 1] = gen_target + return [accepted, num_accepted] + + +class SpecNextNewTokens(OpWrapper): + """The next step's input tokens: ``out[b] = [accepted[batch_idx[b], num_accepted[b] - 1], *next_draft[b]]``. + + ``accepted`` [rows, >= D + 1] (row stride free), ``num_accepted`` [>= n_req], ``batch_idx`` [>= n_req], + ``next_draft`` [n_req, D] (row stride free); int32 or int64 each. The output dtype is torch's promotion of + ``accepted`` and ``next_draft`` (as ``torch.concat``). + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="K3 c1, int32 throughout", + spec=dict(n_req=1, draft_len=3), + ), + Cell( + why="K3 c8 with shuffled batch indices", + spec=dict(n_req=8, draft_len=3, shuffle=True), + ), + Cell( + why="K7 c8", + spec=dict(n_req=8, draft_len=7), + ), + Cell( + why="int64 next drafts: the promoted int64 output", + spec=dict(n_req=8, draft_len=3, draft_dtype=torch.int64), + ), + Cell( + why="a wider accepted buffer (max draft length > runtime) and strided drafts", + spec=dict(n_req=8, draft_len=3, wide=True), + ), + ) + + note = """ + Index copies only: the reference is exact. The op does not bound batch_idx or num_accepted; is_valid states them. + """ + + def __call__(self, accepted, num_accepted, batch_idx, next_draft): + return torch.ops.trtllm.spec_next_new_tokens(accepted, num_accepted, batch_idx, next_draft) + + def reference(self, accepted, num_accepted, batch_idx, next_draft): + """The in-tree ``SpecWorkerBase._prepare_next_new_tokens``.""" + n_req = next_draft.shape[0] + first = accepted[batch_idx[:n_req], num_accepted[:n_req] - 1].unsqueeze(1) + return torch.concat([first, next_draft], dim=1) + + def is_valid(self, accepted, num_accepted, batch_idx, next_draft) -> None: + n_req = next_draft.shape[0] + assert bool(((batch_idx[:n_req] >= 0) & (batch_idx[:n_req] < accepted.shape[0])).all()) + assert bool( + ((num_accepted[:n_req] >= 1) & (num_accepted[:n_req] <= accepted.shape[1])).all() + ) + + +class SpecKvRewind(OpWrapper): + """DFlash's draft-forward KV bookkeeping for the generation rows ``nc .. nc + n`` after the forced-acceptance + override: ``rewind[:n] = 1 - num_accepted[nc : nc + n]`` and ``kv_lens[nc : nc + n] += 1`` (in place). + + ``num_accepted`` and ``kv_lens`` int32 or int64 (1-D, contiguous); ``rewind`` [>= n] of num_accepted's dtype. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="K3 c1: one generation row, int32 counts and lengths", + spec=dict(nc=0, n=1), + ), + Cell( + why="K3 c8: eight generation rows", + spec=dict(nc=0, n=8), + ), + Cell( + why="context rows first (nc > 0): only the generation rows change", + spec=dict(nc=3, n=5), + ), + Cell( + why="int64 kv_lens", + spec=dict(nc=0, n=8, kv_dtype=torch.int64), + ), + Cell( + why="the kernel's request bound (one CTA)", + spec=dict(nc=0, n=64), + ), + ) + + note = """ + Integer arithmetic only: the reference is exact. kv_lens is updated in place; rows outside nc .. nc + n are + untouched. The caller decides the step (it runs where the in-tree lines run, after the forced override). + """ + + def __call__(self, num_accepted, kv_lens, rewind, nc: int, n: int): + torch.ops.trtllm.spec_kv_rewind(num_accepted, kv_lens, rewind, int(nc), int(n)) + return rewind[:n] + + def reference(self, num_accepted, kv_lens, rewind, nc: int, n: int): + """DFlashWorker._prepare_kv_for_draft_forward's two lines.""" + out = 1 - num_accepted[nc : nc + n] + kv_lens[nc : nc + n] += 1 + return out + + +spec_strict_accept = SpecStrictAccept() +spec_next_new_tokens = SpecNextNewTokens() +spec_kv_rewind = SpecKvRewind() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept_kernel.py new file mode 100644 index 000000000000..bfec9e460a7d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept_kernel.py @@ -0,0 +1,174 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================= +# Strict draft acceptance and the next step's input tokens -- CuTe DSL kernels (one CTA, one thread per request) +# ============================================================================= +# +# strict_accept, for request b (draft length D, K1 = D + 1 verify rows per request): +# accepted[b, j] = target[b K1 + j] j = 0 .. D (int32) +# num_accepted[b] = 1 + the length of the prefix k = 0, 1, .. where draft[b, k] == target[b K1 + k] +# which is the strict path's `1 + cumprod(draft == target[:, :D]).sum(1)`. Tokens are compared as int64, so an +# int32 / int64 mix compares the values torch's promotion compares. +# +# next_new_tokens, for request b (D next draft tokens): +# out[b, 0] = accepted[batch_idx[b], num_accepted[b] - 1] +# out[b, 1 + j] = next_draft[b, j] j = 0 .. D - 1 +# which is `concat([accepted[batch_idx, num_accepted - 1].unsqueeze(1), next_draft], 1)`. +# +# kv_rewind, for generation request b (rows nc .. nc + n of the batch), after the forced-acceptance override: +# rewind[b] = 1 - num_accepted[nc + b] +# kv_lens[nc + b] += 1 +# which is DFlash's `_kv_rewind_amount = 1 - num_accepted[nc:bs]` and `kv_lens_cuda[nc:bs] += 1`. +# +# Integer copies and compares only: bitwise to the torch path by construction. Thread b handles request b; the +# grid is one CTA of MAX_REQUESTS threads, so a launch serves up to MAX_REQUESTS requests. +# ============================================================================= +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass.experimental import primitives as prims + +MAX_REQUESTS = 64 # capacity: fixed: one CTA, one thread per request + + +@cute.kernel +def strict_accept_kernel( + target: cute.Tensor, # int32 or int64 [n_req * (D + 1)] + draft: cute.Tensor, # int32 or int64 [n_req * D] + accepted: cute.Tensor, # int32 [n_req * (D + 1)], out + num_accepted: cute.Tensor, # int32 [n_req], out + n_req: cutlass.Int32, + draft_len: cutlass.Constexpr[int], +): + tidx, _, _ = cute.arch.thread_idx() + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + prims.griddepcontrol(prims.GridDepAction.WAIT) + if tidx < n_req: + k1 = draft_len + 1 + run = cutlass.Int32(1) + count = cutlass.Int32(0) + for k in cutlass.range_constexpr(draft_len): + tok = cutlass.Int64(target[tidx * k1 + k]) + drf = cutlass.Int64(draft[tidx * draft_len + k]) + accepted[tidx * k1 + k] = cutlass.Int32(tok) + run = cutlass.select_(tok == drf, run, cutlass.Int32(0)) + count = count + run + accepted[tidx * k1 + draft_len] = cutlass.Int32( + cutlass.Int64(target[tidx * k1 + draft_len]) + ) + num_accepted[tidx] = count + cutlass.Int32(1) + + +@cute.jit +def strict_accept( + target: cute.Tensor, + draft: cute.Tensor, + accepted: cute.Tensor, + num_accepted: cute.Tensor, + n_req: cutlass.Int32, + draft_len: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + strict_accept_kernel(target, draft, accepted, num_accepted, n_req, draft_len).launch( + grid=[1, 1, 1], block=[MAX_REQUESTS, 1, 1], stream=stream, use_pdl=use_pdl + ) + + +@cute.kernel +def next_new_tokens_kernel( + accepted: cute.Tensor, # [rows, accepted_stride] (flat), any int dtype + num_accepted: cute.Tensor, # [n_req], any int dtype + batch_idx: cute.Tensor, # [n_req], any int dtype + next_draft: cute.Tensor, # [n_req, draft_stride] (flat), any int dtype + out: cute.Tensor, # [n_req * (D + 1)], out + accepted_stride: cutlass.Int64, + draft_stride: cutlass.Int64, + n_req: cutlass.Int32, + draft_len: cutlass.Constexpr[int], +): + tidx, _, _ = cute.arch.thread_idx() + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + prims.griddepcontrol(prims.GridDepAction.WAIT) + if tidx < n_req: + k1 = draft_len + 1 + row = cutlass.Int64(batch_idx[tidx]) + col = cutlass.Int64(num_accepted[tidx]) - cutlass.Int64(1) + out[tidx * k1] = out.element_type(accepted[row * accepted_stride + col]) + for j in cutlass.range_constexpr(draft_len): + out[tidx * k1 + 1 + j] = out.element_type( + next_draft[cutlass.Int64(tidx) * draft_stride + j] + ) + + +@cute.jit +def next_new_tokens( + accepted: cute.Tensor, + num_accepted: cute.Tensor, + batch_idx: cute.Tensor, + next_draft: cute.Tensor, + out: cute.Tensor, + accepted_stride: cutlass.Int64, + draft_stride: cutlass.Int64, + n_req: cutlass.Int32, + draft_len: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + next_new_tokens_kernel( + accepted, + num_accepted, + batch_idx, + next_draft, + out, + accepted_stride, + draft_stride, + n_req, + draft_len, + ).launch(grid=[1, 1, 1], block=[MAX_REQUESTS, 1, 1], stream=stream, use_pdl=use_pdl) + + +@cute.kernel +def kv_rewind_kernel( + num_accepted: cute.Tensor, # [>= nc + n], any int dtype + kv_lens: cute.Tensor, # [>= nc + n], any int dtype, updated in place + rewind: cute.Tensor, # [n], num_accepted's dtype, out + nc: cutlass.Int32, + n: cutlass.Int32, +): + tidx, _, _ = cute.arch.thread_idx() + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + prims.griddepcontrol(prims.GridDepAction.WAIT) + if tidx < n: + row = nc + tidx + rewind[tidx] = rewind.element_type(cutlass.Int64(1) - cutlass.Int64(num_accepted[row])) + kv_lens[row] = kv_lens.element_type(cutlass.Int64(kv_lens[row]) + cutlass.Int64(1)) + + +@cute.jit +def kv_rewind( + num_accepted: cute.Tensor, + kv_lens: cute.Tensor, + rewind: cute.Tensor, + nc: cutlass.Int32, + n: cutlass.Int32, + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + kv_rewind_kernel(num_accepted, kv_lens, rewind, nc, n).launch( + grid=[1, 1, 1], block=[MAX_REQUESTS, 1, 1], stream=stream, use_pdl=use_pdl + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/__init__.py new file mode 100644 index 000000000000..a3dfe4a6ce39 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMaxM3SparseForConditionalGeneration targets.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/fake_ops.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/fake_ops.py new file mode 100644 index 000000000000..106f61daffe6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/fake_ops.py @@ -0,0 +1,124 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fake (meta) implementations of the torch ops that the MiniMax-M3 decode targets add (``trtllm::step_inputs_scatter`` +and the ``trtllm::minimax_m3_*`` decode ops in ``cpp/tensorrt_llm/thop/minimaxM3/``), for torch.compile and fake-tensor +tracing. Only loaded native ops without an existing fake are registered. Importing the family must remain safe +when the native extension is absent, so unsupported routes can fall back to the built-in model. Call register() +again after loading an overlay; registration is idempotent.""" + +import torch + +from ...catalog.fakes.m31_fake_ops import _has_fake + +_REGISTERED = set() + + +def _step_inputs_scatter_fake(stage): + # Writes the staged rows into their destinations; returns nothing. + return None + + +def _minimax_m3_mxfp8_low_m_gemm_fake(x, w, wsf, warps, cluster_k, row_tiles, pdl): + return x.new_empty((x.shape[0], w.shape[0])) + + +def _minimax_m3_router_gemv_fake(x, w, rows, pdl): + return x.new_empty((x.shape[0], w.shape[0]), dtype=torch.float32) + + +def _minimax_m3_router_fp4_outputs(x, w): + m, k = x.shape + return [ + x.new_empty((m, w.shape[0]), dtype=torch.float32), + x.new_empty((m, k // 2), dtype=torch.uint8), + x.new_empty((m * (k // 16),), dtype=torch.uint8), + ] + + +def _minimax_m3_router_gemv_fp4_fake(x, w, sf_scale, pdl): + return _minimax_m3_router_fp4_outputs(x, w) + + +def _minimax_m3_router_tc_fake(x, w, sf_scale, pdl): + outputs = _minimax_m3_router_fp4_outputs(x, w) + return outputs if sf_scale is not None else outputs[:1] + + +def _minimax_m3_producer_pw_fake( + packed, + kv_data, + kv_scale, + index_k, + out_cache_loc, + kv_quant_scale, + eps, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + rotary_cos_sin, + position_ids, + heads_q, + pdl, + early, +): + num_tokens = packed.shape[0] + return [ + packed.new_empty((num_tokens, heads_q, 128), dtype=torch.float8_e4m3fn), + packed.new_empty((num_tokens, 1, 128), dtype=torch.float8_e4m3fn), + ] + + +def _minimax_m3_mc_allreduce_norm_fake( + x, + x2, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + perm, + scales, + maxt, + quant, + prepushed, + opts, +): + outputs = [torch.empty_like(residual), torch.empty_like(residual)] + if quant: + m, h = residual.shape + outputs += [ + residual.new_empty((m, h), dtype=torch.float8_e4m3fn), + residual.new_empty(((m + 127) // 128 * 128 * (h // 32),), dtype=torch.uint8), + ] + return outputs + + +FAKES = { + "step_inputs_scatter": _step_inputs_scatter_fake, + "minimax_m3_mxfp8_low_m_gemm": _minimax_m3_mxfp8_low_m_gemm_fake, + "minimax_m3_router_gemv": _minimax_m3_router_gemv_fake, + "minimax_m3_router_gemv_fp4": _minimax_m3_router_gemv_fp4_fake, + "minimax_m3_router_tc": _minimax_m3_router_tc_fake, + "minimax_m3_producer_pw": _minimax_m3_producer_pw_fake, + "minimax_m3_mc_allreduce_norm": _minimax_m3_mc_allreduce_norm_fake, +} + + +def register(): + """Register available native ops, preserving existing Meta kernels or Python fakes.""" + for name, fake in FAKES.items(): + if name in _REGISTERED or not hasattr(torch.ops.trtllm, name): + continue + qualname = f"trtllm::{name}" + if not _has_fake(qualname): + torch.library.register_fake(qualname)(fake) + _REGISTERED.add(name) + + +register() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/__init__.py new file mode 100644 index 000000000000..a5aeb0abbdbb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/__init__.py @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 NVFP4 / sm_103 / tp4.""" + +# Routing has selected this target before its native fake registrations are needed. +from .. import fake_ops + +fake_ops.register() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/dspark.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/dspark.py new file mode 100644 index 000000000000..ed54c31205a7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/dspark.py @@ -0,0 +1,631 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-model DSpark on this target (phase W0): the in-tree DSpark drafter and worker, unchanged. + +The target keeps its own forward; the shell hands the step to ``M3DSparkWorker``, which is the in-tree +``DSparkWorker`` plus the five calls the shell makes on any worker (``capture_hidden_states``, +``arm_capture_norm``, ``tail_eligible``, ``prebuild``, ``__call__``). In W0 none of them takes a fused path: the +captures are the in-tree DFlash captures, the target logits go through the logits processor and the in-tree +strict accept verifies them, so the speculation is the in-tree one by construction. + +The drafter is the in-tree standalone DSpark drafter for the checkpoint's architecture, built through the draft +model registry. Its context K/V lives in the separate draft KV cache, which must keep the drafter's projection +dtype (BF16): a pool of another dtype is not bound and the worker falls back to a private arena sized by the +served sequence length. So the drafter never takes the target's KV quantization: neither the FP8 draft-KV +override (an Eagle3 need) nor the target's own KV algo (NVFP4 under ``kv_cache_config.dtype="nvfp4"``). +""" + +from __future__ import annotations + +import os +import time +from typing import Tuple + +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.comm import ( + allgather_cast as _allgather_cast, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + strict_accept as _strict_accept, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_block_tables import ( + enable_parallel_refresh, + refresh_block_tables, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_advance import ( + advance_ctx_and_read_view, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_store import ( + fused_ctx_store, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_drafter_inputs import ( + drafter_inputs as _drafter_inputs, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_forward_indices import ( + forward_indices, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_qk_norm_rope import ( + qk_norm_rope, +) +from tensorrt_llm._torch.distributed import allgather +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + external_drafter_config_kwargs, + get_draft_model, +) +from tensorrt_llm._torch.speculative.dspark import DSparkWorker +from tensorrt_llm._torch.speculative.interface import should_use_separate_draft_kv_cache +from tensorrt_llm.logger import logger + +from .dspark_s4 import MAX_FAST_BATCH, S4WorkerMixin +from .m31_switches import m31_disabled +from .mxdraft import route_mx_linears +from .ops.embedding.m3_embedding import m3_embedding + +# MXDRAFT (MXFP8 drafter GEMV): ON by default. With the K3 bundle off (the ship config), its powered K7 gate c +# (n 1,776) passes: mean d +0.009 AL, CI [-0.019, +0.037]. Kill switch TRTLLM_M31_DISABLE_MXDRAFT=1, read at import. +MXDRAFT = not m31_disabled("MXDRAFT") + + +def is_standalone_dspark(spec_config) -> bool: + """Whether the run speculates with a standalone DSpark drafter (its own checkpoint, ``decoding_type: DSpark``).""" + return bool( + spec_config is not None + and spec_config.spec_dec_mode.use_one_engine() + and spec_config.spec_dec_mode.is_dspark() + and not getattr(spec_config, "draft_is_embedded_in_target", False) + ) + + +class M3EmbedTokens(nn.Module): + """The target's token embedding as the module the in-tree drafter borrows (``load_weights_from_target_model``). + + The drafter embeds the bonus and mask tokens through ``embed_tokens(ids)``; the target keeps its replicated + table as a raw tensor and gathers it with ``m3_embedding``, so this module calls the same gather on the same + table (an exact gather, so the drafter sees the rows the target embeds). + """ + + def __init__(self, core: nn.Module) -> None: + super().__init__() + self._core = [core] # not a submodule: the table belongs to the target + + @property + def weight(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def forward(self, ids: torch.Tensor) -> torch.Tensor: + return m3_embedding(ids, self.weight) + + +def s10c_c_drafter_inputs( + embed_tokens, + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + mask_token_id, + trained_mask_embedding, + block_size, + tokens_per_request, +): + """S10c-C: the drafter's step inputs in one launch (catalog/spec_decode/dspark_drafter_inputs), for this target's + embedding only: ``M3EmbedTokens`` is an exact gather of the replicated table, so the noise block is table rows (the + trained mask row if the drafter keeps one). None for any other embedding: the stock path runs.""" + if not isinstance(embed_tokens, M3EmbedTokens): + return None + table = embed_tokens.weight + mask_row = ( + trained_mask_embedding if trained_mask_embedding is not None else table[int(mask_token_id)] + ) + return _drafter_inputs( + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + table, + mask_row, + block_size, + tokens_per_request, + ) + + +# N3 A2 (switch TRTLLM_M31_DISABLE_N3A): the target's strict acceptance and next-step input tokens in one launch each +# (catalog/spec_decode/strict_accept), bitwise to the in-tree torch path. Read once, at import. +N3A_DISABLED = m31_disabled("N3A") +# N3A-KV (switch TRTLLM_M31_DISABLE_N3A_KV, and off with N3A): DFlash's draft-forward KV bookkeeping (the rewind amount +# 1 - num_accepted and kv_lens += 1) in one launch where the in-tree lines run, after the forced-acceptance override, +# bitwise (integer arithmetic). Read once, at import. +N3A_KV_DISABLED = N3A_DISABLED or m31_disabled("N3A_KV") +# Gate-only cross-check (TRTLLM_M31_CHECK_N3A_KV=1; never in a perf arm): every N3A-KV call is compared with the stock +# lines on copies (graph-safe device counters; reported on eager steps). +N3A_KV_CHECK = os.environ.get("TRTLLM_M31_CHECK_N3A_KV", "").strip() not in ("", "0") +# N3 P-gid + P-cat (switch TRTLLM_M31_DISABLE_N3P): the verify-row target logits without the identity gather_ids copy +# and with the vocab concat and the fp32 cast in one strided copy, bitwise to the logits processor. Read at import. +N3P_DISABLED = m31_disabled("N3P") +# N3 A1 (switch TRTLLM_M31_DISABLE_A1): the verify logits' vocab all-gather, concat and fp32 cast in one launch over a +# multicast NVLink workspace (catalog/comm/allgather_cast), bitwise to the NCCL all-gather + strided copy below. K3 +# verify rows (B x (K + 1), up to 64 rows: K3 batches 1..16, K7 batches 1..8). Read once, at import. +A1_DISABLED = m31_disabled("A1") +A1_MAX_ROWS = 64 # capacity: fixed: allgather_cast.MAX_ROWS (K3 at batch 16, K7 at batch 8) +# A1 at draft lengths other than 3 (switch TRTLLM_M31_DISABLE_A1_K7, K3 keeps A1). +A1_K7_DISABLED = m31_disabled("A1_K7") +_n3a_logged = False + + +def _n3a_log_once() -> None: + global _n3a_logged + if not _n3a_logged: + _n3a_logged = True + logger.info( + "[M3 dispatch] n3_strict_accept taken: fused strict acceptance + next-step input tokens" + ) + + +_n3p_logged = False +_n3a_kv_logged = False + + +def _n3a_kv_log_once() -> None: + global _n3a_kv_logged + if not _n3a_kv_logged: + _n3a_kv_logged = True + logger.info( + "[M3 dispatch] n3a_kv_rewind taken: draft-forward KV rewind + kv_lens bump in one launch" + ) + + +_a1_logged = set() + + +def _a1_log_once(key, message: str) -> None: + if key not in _a1_logged: + _a1_logged.add(key) + logger.info(message) + + +def _n3p_log_once() -> None: + global _n3p_logged + if not _n3p_logged: + _n3p_logged = True + logger.info( + "[M3 dispatch] n3p_target_logits taken: verify-row logits without the gather_ids copy and the concat" + ) + + +class M3DSparkWorker(S4WorkerMixin, DSparkWorker): + """The in-tree ``DSparkWorker`` with the calls this target's shell makes on its worker. W1: the one-launch Markov + chain (``S4WorkerMixin``, switch ``TRTLLM_M31_DRAFT_DISABLE_S4``). N3 A2: the strict acceptance and the next-step + input tokens in one launch each (switch ``TRTLLM_M31_DISABLE_N3A``). Everything else is the in-tree path.""" + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None + ) -> None: + """The in-tree DFlash capture: layer output (hidden + residual) into the static capture buffer.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model + ) -> None: + """No capture norm: DSpark's fc reads the raw captured layers.""" + return None + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Never: the target logits always go through the logits processor and the in-tree strict accept.""" + return False + + _a1 = None # the N3 A1 workspace (allgather_cast's arguments) once set up + + def prebuild(self, lm_head: nn.Module, draft_model) -> None: + """The one-launch Markov chain's shards, buffers, exchange workspace and compiles, then the N3 A2 compiles + (rank-local) and the N3 A1 workspace and compiles (collective), outside capture, every rank, in this order; + no fused verify, no coded target head, no fused draft loop.""" + rank = getattr(getattr(self, "mapping", None), "tp_rank", None) + # S10: the drafter's strided-view glue (bitwise; kill switch TRTLLM_M31_DISABLE_S10, read at import). + lean = not m31_disabled("S10") + self.enable_lean_glue(draft_model, lean) + k = getattr(getattr(self, "spec_config", None), "max_draft_len", 0) or 0 + logger.info(f"M3.1 DSpark S10: lean drafter glue {'on' if lean else 'off'} on rank {rank}") + # S10c-A: the drafter's context K / V epilogue + accept mask + paged store in one launch after the K / V GEMM + # (bitwise). Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_A, read at import. + s10c_a = self.enable_fused_ctx_store( + None if m31_disabled("S10C") or m31_disabled("S10C_A") else fused_ctx_store + ) + logger.info( + f"M3.1 DSpark S10c-A: fused drafter ctx store {'on' if s10c_a else 'off'} on rank {rank}" + ) + # S10c-C: the drafter's step inputs (bonus, positions, noise block) in one launch (bitwise: integers and row + # copies). Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_C, read at import. + s10c_c = self.enable_fused_drafter_inputs( + None if m31_disabled("S10C") or m31_disabled("S10C_C") else s10c_c_drafter_inputs + ) + logger.info( + f"M3.1 DSpark S10c-C: fused drafter step inputs {'on' if s10c_c else 'off'} on rank {rank}" + ) + # S10c-D: the drafter's context advance + read view in one launch (bitwise, integer). Kill switches + # TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_D, read at import. + s10c_d = self.enable_fused_ctx_advance( + None if m31_disabled("S10C") or m31_disabled("S10C_D") else advance_ctx_and_read_view + ) + logger.info( + f"M3.1 DSpark S10c-D: fused drafter context advance {'on' if s10c_d else 'off'} on rank {rank}" + ) + # S10c-B: the drafter's block-table refresh (decode + placeholder count) in one launch (bitwise, integer). + # Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_B, read at import. + s10c_b = self.enable_fused_block_refresh( + None if m31_disabled("S10C") or m31_disabled("S10C_B") else refresh_block_tables + ) + # S10c-B2: that refresh on a (row, column-chunk) grid (bitwise; flat at short tables, removes the serial + # kernel's growth past ~11K blocks per sequence). Kill switch TRTLLM_M31_DISABLE_S10C_B2, read at import. + s10c_b2 = enable_parallel_refresh(s10c_b and not m31_disabled("S10C_B2")) + logger.info( + f"M3.1 DSpark S10c-B: fused drafter block-table refresh {'on' if s10c_b else 'off'} on rank {rank}" + f" (S10c-B2 2-D grid {'on' if s10c_b2 else 'off'})" + ) + # S10c-E: the draft model's paged block-decode index prelude in one launch (bitwise, integer). Kill switches + # TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_E, read at import. + s10c_e = self.enable_fused_forward_indices( + draft_model, None if m31_disabled("S10C") or m31_disabled("S10C_E") else forward_indices + ) + logger.info( + f"M3.1 DSpark S10c-E: fused drafter forward indices {'on' if s10c_e else 'off'} on rank {rank}" + ) + # EX-11: the drafter block's q / k RMSNorm + partial RoPE in one launch (bitwise to flashinfer's three). Kill + # switch TRTLLM_M31_DISABLE_EX11, read at import. + ex11 = self.enable_fused_qk_norm_rope( + draft_model, None if m31_disabled("EX11") else qk_norm_rope + ) + logger.info( + f"M3.1 DSpark EX-11: fused drafter q/k norm + RoPE {'on' if ex11 else 'off'} on rank {rank}" + ) + # MXDRAFT: the drafter's qkv / o / gate_up / down GEMMs on this target's MXFP8 GEMV for the decode passes' + # rows (B x K, B <= MAX_FAST_BATCH), from the exact MXFP8 form of the shipped weights (drafter-only precision). + # ON by default (see MXDRAFT); kill switch TRTLLM_M31_DISABLE_MXDRAFT. + mx = ( + 0 + if not MXDRAFT or k <= 0 + else route_mx_linears(draft_model, [b * k for b in range(1, MAX_FAST_BATCH + 1)]) + ) + logger.info(f"M3.1 DSpark MXDRAFT: MXFP8 drafter GEMV on {mx} projections on rank {rank}") + self.prebuild_s4(lm_head, draft_model) + if not N3A_DISABLED and lm_head is not None: + _strict_accept.precompile((self.max_draft_len,), lm_head.weight.device) + self._a1 = self._prebuild_a1(lm_head) if lm_head is not None else None + n3a_kv = False + if not N3A_KV_DISABLED and lm_head is not None: + _strict_accept.precompile_kv_rewind(lm_head.weight.device) + self._n3a_kv_buf = torch.empty( + _strict_accept.max_requests(), dtype=torch.int32, device=lm_head.weight.device + ) + n3a_kv = self.enable_fused_kv_rewind(self._n3a_kv_rewind) + logger.info( + f"M3.1 N3A-KV: one-launch draft-forward KV bookkeeping {'on' if n3a_kv else 'off'} on rank " + f"{getattr(getattr(self, 'mapping', None), 'tp_rank', None)}" + ) + + _n3a_kv_buf = None # the persistent rewind amounts of N3A-KV (int32, MAX_REQUESTS) + _n3a_kv_counts = None # gate-only cross-check counters (calls, mismatches) + + def _n3a_kv_rewind(self, num_accepted, kv_lens, nc: int, bs: int): + """N3A-KV for the DFlash hook: ``1 - num_accepted[nc:bs]`` (a view of the persistent buffer) with + ``kv_lens[nc:bs] += 1`` in one launch, or None (the stock lines) for operands or a variant it does not take.""" + n = bs - nc + buf = self._n3a_kv_buf + if buf is None or not _strict_accept.supports_kv_rewind(num_accepted, kv_lens, buf, nc, n): + return None + if not _strict_accept.is_compiled_kv_rewind(num_accepted, kv_lens): + return None + _n3a_kv_log_once() + if not N3A_KV_CHECK: + return _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + want_rewind = 1 - num_accepted[nc:bs] + want_kv = kv_lens[nc:bs] + 1 + out = _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + if self._n3a_kv_counts is None: + self._n3a_kv_counts = torch.zeros(2, dtype=torch.int64, device=kv_lens.device) + bad = (out != want_rewind).any() | (kv_lens[nc:bs] != want_kv).any() + self._n3a_kv_counts[0] += 1 + self._n3a_kv_counts[1] += bad.to(torch.int64) + if not torch.cuda.is_current_stream_capturing(): + calls, mism = self._n3a_kv_counts.tolist() + logger.info(f"M3.1 N3A-KV check: {calls} calls, {mism} mismatches") + return out + + def _a1_all_ok(self, local: bool) -> bool: + """Whether ``local`` holds on every rank of the TP group: A1 is set up on every rank or on none (a rank that + is not eligible, or whose compile failed, turns it off everywhere).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + return _mnnvl_workspace_all_succeeded(_get_mnnvl_workspace_comm(self.mapping), local) + + def _prebuild_a1(self, lm_head: nn.Module): + """The N3 A1 workspace and its K3 verify-row compiles, or None (every rank alike): off by its switch, at + another draft length, without a column-parallel bf16 head over a TP group, or when a compile fails on any + rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + mapping = self.mapping + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return None + cols = int(getattr(lm_head, "out_features", 0)) + sms = torch.cuda.get_device_properties(lm_head.weight.device).multi_processor_count + # Why this rank would not set A1 up (empty: it would). The grid is a function of rows and the SM count: + # requiring every SM up to the workspace's MAX_GRID makes it a function of rows alone, the same on every rank. + off = [ + why + for why, bad in ( + ("TRTLLM_M31_DISABLE_A1 set", A1_DISABLED), + ( + f"draft length {int(self.max_draft_len)} with TRTLLM_M31_DISABLE_A1_K7 set", + int(self.max_draft_len) != 3 and A1_K7_DISABLED, + ), + ( + f"draft length {int(self.max_draft_len)}: more than {A1_MAX_ROWS} verify rows at batch 1", + int(self.max_draft_len) + 1 > A1_MAX_ROWS, + ), + ( + "not a column-parallel, gathered, unpadded, bias-free head", + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or getattr(lm_head, "bias", None) is not None, + ), + (f"head dtype {lm_head.weight.dtype}", lm_head.weight.dtype != torch.bfloat16), + ( + f"{cols} shard columns (whole 16-byte vectors needed)", + cols <= 0 or cols % 8 != 0, + ), + ( + f"{sms} SMs (fewer than {_allgather_cast.MAX_GRID})", + sms < _allgather_cast.MAX_GRID, + ), + ) + if bad + ] + if not self._a1_all_ok(not off): + logger.info( + f"M3.1 N3 A1: off on rank {mapping.tp_rank} ({'; '.join(off) or 'off on another rank'}); " + "the verify logits use the NCCL all-gather" + ) + return None + k1 = int(self.max_draft_len) + 1 + rows = tuple(b * k1 for b in range(1, A1_MAX_ROWS // k1 + 1)) + start = time.perf_counter() + if not _allgather_cast.precompile( + lm_head.mapping, cols, rows, lm_head.weight.device, agree=self._a1_all_ok + ): + logger.warning( + "M3.1 N3 A1: a compile failed on some rank; every rank keeps the NCCL all-gather" + ) + return None + ws = _allgather_cast.workspace_args(lm_head.mapping, cols) + # The engagement line: one per rank, at server start. + logger.info( + f"M3.1 N3 A1: one-launch verify all-gather set up on rank {mapping.tp_rank}/{mapping.tp_size} " + f"(shard {cols} columns, rows {rows}, workspace {ws['ws_uc'].numel() * 4} bytes/rank, " + f"set-up {time.perf_counter() - start:.2f} s)" + ) + return ws + + def _n3a_eligible( + self, logits, draft_tokens, num_contexts: int, batch_size: int, spec_metadata + ) -> bool: + """Whether this step's strict acceptance takes the fused op: a decode-only, non-ragged step whose logits hold + exactly batch_size x (D + 1) verify rows and whose drafts the op takes. Host state and shapes only, so it is + fixed per captured graph and the same on every rank. Decided before the target tokens are sampled, so the + sampler runs exactly once either way.""" + if N3A_DISABLED or spec_metadata is None or spec_metadata.is_ragged_verify: + return False + if num_contexts != 0 or batch_size <= 0 or draft_tokens.dim() != 2: + return False + n_req, draft_len = draft_tokens.shape + if n_req != batch_size or draft_len <= 0 or n_req > _strict_accept.max_requests(): + return False + rows = logits.shape[0] if logits.dim() > 1 else 1 + return rows == batch_size * (draft_len + 1) and draft_tokens.is_contiguous() + + def _sample_and_accept_draft_tokens_base( + self, logits, draft_tokens, num_contexts, batch_size, spec_metadata + ): + """The in-tree strict acceptance with its epilogue in one launch (N3 A2): the same target tokens (the in-tree + sampler, called once), the same accepted tokens and counts, then the in-tree forced-acceptance override (its + RNG pool and counters advance as on an in-tree step).""" + if not self._n3a_eligible(logits, draft_tokens, num_contexts, batch_size, spec_metadata): + return super()._sample_and_accept_draft_tokens_base( + logits, draft_tokens, num_contexts, batch_size, spec_metadata + ) + if logits.dim() == 1: + logits = logits.unsqueeze(0) + target_tokens = self._sample_tokens_for_batch( + logits, spec_metadata, num_contexts, batch_size + ) + if _strict_accept.supports_strict_accept( + target_tokens, draft_tokens + ) and _strict_accept.is_compiled_strict_accept(target_tokens, draft_tokens): + accepted, num_accepted = _strict_accept.spec_strict_accept(target_tokens, draft_tokens) + _n3a_log_once() + else: + # The sampler already ran: finish with the in-tree epilogue (the op's reference is that code). + accepted, num_accepted = _strict_accept.spec_strict_accept.reference( + target_tokens, draft_tokens + ) + num_accepted = self._apply_force_accepted_tokens( + num_accepted, num_contexts, draft_tokens.shape[-1], spec_metadata=spec_metadata + ) + return accepted, num_accepted + + def target_logits(self, hidden_states, attn_metadata, spec_metadata, lm_head: nn.Module): + """The verify rows' gathered fp32 target logits (N3 P-gid + P-cat), or None for the logits processor. + + On a decode-only, non-ragged step every row is a verify row and ``gather_ids`` is ``arange(num_tokens)`` (each + generation request extends it with the next ``1 + D`` row indices), so the rows are a slice, not a gather. + The processor's head is the column-parallel BF16 GEMM, an all-gather along the vocab (NCCL, then a concat) + and ``.float()``; here the same all-gather runs along dim 0 (no concat) and one strided copy writes the + rank-major shards into the fp32 output: the same bytes, an exact widening. Shapes and host state only, so + the decision is fixed per captured graph and the same on every rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + if ( + N3P_DISABLED + or spec_metadata is None + or getattr(spec_metadata, "is_ragged_verify", False) + ): + return None + if attn_metadata.num_contexts != 0 or spec_metadata.gather_ids is None: + return None + rows = hidden_states.shape[0] + if spec_metadata.gather_ids.shape[0] != rows or rows == 0: + return None + if ( + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or getattr(lm_head, "bias", None) is not None + ): + return None + shard = lm_head.apply_linear(hidden_states, None) + _n3p_log_once() + tp = lm_head.tp_size + if tp == 1: + return shard.float() + # N3 A1: one launch for the all-gather, the concat and the cast. The decision depends on the rows and the + # set-up only (the same on every rank, fixed per captured graph). + if self._a1 is not None: + if rows <= A1_MAX_ROWS and _allgather_cast.is_compiled( + rows, shard.shape[-1], tp, 0, shard.device + ): + _a1_log_once( + "taken", + "[M3 dispatch] n3_a1 allgather_cast taken: verify logits gathered in one launch", + ) + return _allgather_cast.allgather_cast(shard, **self._a1) + _a1_log_once( + rows, + f"[M3 dispatch] n3_a1 allgather_cast not taken: {rows} verify rows (compiled: verify rows up to " + f"{A1_MAX_ROWS}); NCCL all-gather", + ) + gathered = allgather(shard, lm_head.mapping, dim=0) + vocab_shard = shard.shape[-1] + out = torch.empty((rows, tp * vocab_shard), dtype=torch.float32, device=shard.device) + out.view(rows, tp, vocab_shard).copy_(gathered.view(tp, rows, vocab_shard).transpose(0, 1)) + return out + + def _prepare_next_new_tokens( + self, + accepted_tokens, + next_draft_tokens, + batch_indices_cuda, + batch_size, + num_accepted_tokens, + ): + """The next step's input tokens in one launch (N3 A2), else the in-tree index + concat.""" + batch_idx = batch_indices_cuda[:batch_size] + if ( + not N3A_DISABLED + and next_draft_tokens.dim() == 2 + and next_draft_tokens.shape[0] == batch_size + and _strict_accept.supports_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + and _strict_accept.is_compiled_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + ): + return _strict_accept.spec_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + return super()._prepare_next_new_tokens( + accepted_tokens, next_draft_tokens, batch_indices_cuda, batch_size, num_accepted_tokens + ) + + +def build_dspark( + shell, model_config: ModelConfig +) -> Tuple[ModelConfig, nn.Module, M3DSparkWorker, bool]: + """The drafter config, the drafter and the worker of one-model standalone DSpark (mirrors + ``SpecDecOneEngineForCausalLM`` for an external drafter).""" + spec = model_config.spec_config + if not is_standalone_dspark(spec): + raise ValueError("build_dspark serves the standalone DSpark drafter only") + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, **external_drafter_config_kwargs(model_config, spec) + ) + # The drop is safe only because draft_config owns its QuantConfig: from_pretrained reads the drafter checkpoint + # and takes no quant_config from the target. Never pass the target's quant_config in here. + keep_drafter_kv_dtype(draft_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = get_draft_model(model_config, draft_config, shell.lm_head, shell.model) + # A parallel drafter (DSpark) projects its block rows through the target's logits processor, as the one-engine + # construction sets it. + draft.logits_processor = shell.logits_processor + worker = M3DSparkWorker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + logger.info( + f"M3.1 target DSpark: {type(worker).__name__} over {type(draft).__name__}, K={spec.max_draft_len}, " + f"separate draft KV cache={use_separate}" + ) + return draft_config, draft, worker, use_separate + + +def load_dspark_draft_weights(shell, weights, weight_mapper=None) -> None: + """The drafter's own checkpoint, then the target's embedding and head (mirrors + ``SpecDecOneEngineForCausalLM.load_draft_weights`` for DSpark). + + The drafter borrows the target's embedding and head, and ``M3EmbedTokens.weight`` IS the target's table, so a + drafter checkpoint that ships either would load into a module the target owns or be shadowed by it: refused. + """ + borrowed = sorted( + k for k in weights.keys() if "embed_tokens." in k or k.split(".")[-2:-1] == ["lm_head"] + ) + if borrowed: + raise ValueError( + f"the DSpark drafter checkpoint ships tensors this target lends it (embedding / head): {borrowed[:4]}" + ) + shell.draft_model.load_weights(weights=weights, weight_mapper=weight_mapper) + shell.draft_model.load_weights_from_target_model(shell) + if shell.draft_model.lm_head is not shell.lm_head or ( + shell.draft_model.draft_model_full.model.embed_tokens is not shell.model.embed_tokens + ): + raise RuntimeError("the DSpark drafter does not borrow this target's embedding and head") + own = sorted( + n for n, _ in shell.named_parameters() if not n.startswith(("model.w.", "draft_model.")) + ) + logger.info(f"M3.1 target DSpark: shell parameters outside the core and the drafter: {own}") + + +def keep_drafter_kv_dtype(draft_config: ModelConfig) -> None: + """Drop any KV quantization from the drafter's config, so its context pool is allocated in its projection dtype. + + The one-engine construction copies the target's KV algo onto the drafter; on this target that is NVFP4, which + allocates a packed int8 pool the worker refuses to bind (it falls back to its private arena, and a reused + prefix's drafter K/V is lost). The cached quant modes are dropped with it, as the draft KV budget does. + """ + for quant in [draft_config.quant_config, *(draft_config.quant_config_dict or {}).values()]: + quant.kv_cache_quant_algo = None + quant.__dict__.pop("quant_mode", None) + quant.__dict__.pop("layer_quant_mode", None) + + +def draft_kv_override_applies(spec_config) -> bool: + """The target's FP8 draft-KV override is an Eagle3 need; a DSpark drafter keeps its BF16 context pool.""" + return not is_standalone_dspark(spec_config) + + +__all__ = [ + "M3DSparkWorker", + "M3EmbedTokens", + "build_dspark", + "draft_kv_override_applies", + "is_standalone_dspark", + "keep_drafter_kv_dtype", + "load_dspark_draft_weights", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/dspark_s4.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/dspark_s4.py new file mode 100644 index 000000000000..5131aa51b300 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/dspark_s4.py @@ -0,0 +1,491 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""S4: the catalog's one-launch Markov chain (``catalog/spec_decode/dspark_markov_chain``) in this target's DSpark +worker (each M3.1 target keeps its own copy of this glue: targets share only the catalog). + +``S4WorkerMixin`` goes before ``DSparkWorker`` in the target's worker. For a generation step it serves (switch not +set, 1 <= generation requests <= ``MAX_FAST_BATCH``, greedy drafting, no d2t / guided decoder / DFlash-2 path, a +vocab-sharded borrowed head): + +- the draft head computes this rank's vocab shard of the block logits (the head's local GEMM, no gather) and copies + it into a padded buffer whose tail holds -inf (the step's logits processor, swapped in for the step); +- ``_refine_block_logits`` runs the catalog op over the shards in one launch and keeps the tokens; +- ``sample_draft_tokens`` returns them. + +The tokens are bitwise the in-tree chain's on the gathered logits: the gathered logits are the concatenation of the +same per-rank GEMMs, and the op reproduces the chain's bf16 bias, bf16 add and first-maximum argmax. When the op +does not support a step after the logits were sharded, the in-tree sharded path (``vocab_slice`` + the TP-aware +argmax) runs instead, with the same tokens. + +Setup (``prebuild_s4``, from the worker's ``prebuild``, outside capture, on every rank): the padded ``markov_w2`` +shard, one padded logits buffer per (batch, block), the exchange workspace (collective) and the compiles. + +Every routing decision is Python, so it is fixed into each decode graph when the graph is captured: +``TRTLLM_M31_DRAFT_DISABLE_S4`` is read once, at import, not a switch that can be toggled at run time. The set-up +decision must be the same on every rank (the workspace is collective); it follows from the config and the +environment, and ``prebuild_s4`` checks that the ranks agree before the collective. + +Kill switches (``m31_switches``, ``TRTLLM_M31_DISABLE_=1``, read once at import; all bitwise the same tokens): +``S4B8`` -- batch-1 calls on the chain's batch-1 kernel (back to the general kernel). ``MARKOV_LL`` -- the batch-1 +kernel's LL flag-in-payload exchange (``dspark_markov_chain_kernel_b1_ll``; back to the S4B8 kernel's Lamport +exchange; no effect with S4B8 off). ``S4_FP32SUM`` -- the fp32 sum compare (back to bf16-rounded sums). ``S4_LOGITS`` +-- S4 under advanced draft sampling (back to the in-tree chain there). + +""" + +from __future__ import annotations + +import atexit +import os +from typing import Dict, Optional, Tuple + +import torch +import torch.nn.functional as F +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_bias as markov_bias, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_chain as chain, +) +from tensorrt_llm._torch.models.modeling_speculative import dspark_markov_step_bias +from tensorrt_llm.logger import logger + +from .m31_switches import m31_disabled + +MARKOV_RANK = 256 +MAX_FAST_BATCH = ( + 8 # capacity: measured: above 8 requests the chain is within 10% of the reference (16: 0.93x) +) +DISABLE_ENV = "TRTLLM_M31_DRAFT_DISABLE_S4" +S4_COUNT_LOG_EVERY = 1000 # generation steps between the per-rank "served x/y" engagement lines + +# MARKOV_LL ( GB TP8 K7 b1 exchange 2.25 us + post 0.5 us per position against a +# ~1.25 us single-hop floor): the batch-1 kernel's per-position exchange as one 16-byte {value, tag, index, tag} +# multicast store per CTA (no fence, no flag word), polled and reduced by warp 0 alone (no CTA barrier after the +# poll). Bitwise the S4B8 kernel's tokens (same entries, same total argmax order). Default ON; kill switch +# TRTLLM_M31_DISABLE_MARKOV_LL=1, read at import. +MARKOV_LL = not m31_disabled("MARKOV_LL") + +# TAILBK_SPEC (DSpark-worker sites): bookkeeping launches +# of the draft tail removed exactly. Here, site C: the chain op's three read-only accepted inputs (ones, arange, +# zeros; 3 launches per step) come from a per-shape cache filled at precompile. Site H: the target's KV-length rewind +# after the draft forward (kv_lens[nc:bs] -= rewind; clamp_(min=0): 2 launches per step) runs in the chain's launch +# (its rewind path, after the grid-dependency wait; nothing between the chain and the stock rewind reads kv_lens). +# Site S4L (S4-logits, the advanced-sampling contract path): the biased block logits' glue around the bias GEMM -- the +# prevs cat / casts, get_masked_input_and_mask, F.embedding, masked_fill, the two widening copies and the add (14 +# launches per step) -- in two launches (catalog dspark_markov_bias; the GEMM unchanged). +# Bitwise the same tokens, kv_lens and biased logits. Default ON; kill switch TRTLLM_M31_DISABLE_TAILBK_SPEC=1, read +# at import. +TAILBK_SPEC = not m31_disabled("TAILBK_SPEC") + +# Read once, at import: every rank and every step of a process sees the same setting. Any value but "" or "0" turns +# the item off, as the targets' m31_switches reads its switches. +_DISABLED = os.environ.get(DISABLE_ENV, "").strip() not in ("", "0") + + +def disabled() -> bool: + return _DISABLED + + +class _S4State: + def __init__(self, mapping, rows: int, shard: int, offset: int, w1: torch.Tensor, w2s: torch.Tensor, + block: int, ws: dict) -> None: # fmt: skip + self.mapping = mapping + self.ws = ws + self.rows = rows + self.shard = shard + self.offset = offset + self.w1 = w1 + self.w2s = w2s + self.block = block + self.bufs: Dict[Tuple[int, int], torch.Tensor] = {} + + def buffer(self, batch: int) -> torch.Tensor: + return self.bufs[(batch, self.block)] + + +# S4-logits bias form: True = all K positions' bias in one GEMM over B*K rows (S4LC: bitwise the per-step GEMMs); +# False = the in-tree per-step loop over the chain's tokens (exact by construction). +S4_LOGITS_BATCHED = True + + +def _biased_shard_logits( + shard_logits: torch.Tensor, + first_prev: torch.Tensor, + tokens: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + fp32: bool = True, +) -> torch.Tensor: + """The in-tree chain's biased block logits on this rank's vocab shard, given the chain's tokens: step k adds + bf16(w2 @ w1[prev_k]) with prev_0 = the anchor and prev_k = token k-1 (dspark_markov_step_bias). With ``fp32`` + (the in-tree arithmetic: LogitsProcessor hands the chain fp32 logits) the bf16 shard logits and the bias are + widened and added in fp32; without it, one bf16 add.""" + if ( + fp32 + and S4_LOGITS_BATCHED + and TAILBK_SPEC + and markov_bias.supports(shard_logits, first_prev, tokens, w1) + ): + # TAILBK_SPEC S4L: the same prevs, embedding rows, GEMM and fp32 sum, in two launches around the GEMM. + B, K, N = shard_logits.shape + bias = F.linear(markov_bias.prev_embeddings(first_prev, tokens, w1), w2_shard) + return markov_bias.widen_add(shard_logits, bias.view(B, K, N)) + if fp32: + shard_logits = shard_logits.float() + B, K, N = shard_logits.shape + prevs = torch.cat([first_prev.view(B, 1), tokens[:, : K - 1].long()], dim=1) + if S4_LOGITS_BATCHED: + bias = dspark_markov_step_bias(prevs.reshape(-1), w1, w2_shard).view(B, K, N) + return shard_logits + bias.to(shard_logits.dtype) + out = torch.empty_like(shard_logits) + for k in range(K): + out[:, k] = shard_logits[:, k] + dspark_markov_step_bias(prevs[:, k], w1, w2_shard).to( + shard_logits.dtype + ) + return out + + +class _ShardedDraftLogits(nn.Module): + """The step's draft logits processor: this rank's vocab shard of ``lm_head`` over the gathered block rows, + copied into the padded buffer; any other call goes to the drafter's own processor.""" + + def __init__(self, inner: nn.Module, state: _S4State, batch: int) -> None: + super().__init__() + self.inner = inner + self._state = [state] + self.batch = batch + + def forward(self, hidden_states, lm_head, attn_metadata, return_context_logits: bool = False): + state = self._state[0] + if hidden_states.dim() != 2 or hidden_states.shape[0] != self.batch * state.block: + return self.inner(hidden_states, lm_head, attn_metadata, return_context_logits) + logits = lm_head.apply_linear(hidden_states, None) + buf = state.buffer(self.batch) + buf[:, :, : state.shard].copy_(logits.view(self.batch, state.block, state.shard)) + return logits + + +def _head_is_sharded(lm_head: nn.Module, vocab: int, tp: int) -> bool: + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + return ( + tp > 1 + and getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and getattr(lm_head, "padding_size", 0) == 0 + and getattr(lm_head, "bias", None) is None + and lm_head.out_features * tp == vocab + and lm_head.weight.dtype == torch.bfloat16 + ) + + +class S4WorkerMixin: + """Mixed into the target's ``DSparkWorker`` subclass (before it in the bases).""" + + _s4: Optional[_S4State] = None + _s4_step: Optional[_S4State] = None + _s4_tokens: Optional[torch.Tensor] = None + _s4_step_meta: Optional[Tuple[int, int, object]] = None + _s4_draft: Optional[nn.Module] = None + _s4_served: int = 0 + _s4_gen_steps: int = 0 + _s4_ll_logged: bool = False + _s4_step_attn = ( + None # the step's attention metadata (the KV-length rewind's kv_lens), while S4 serves it + ) + _tailbk_rewound: bool = ( + False # TAILBK_SPEC H: this step's chain launch applied the KV-length rewind + ) + _tailbk_h_logged: bool = False + _tailbk_s4l_logged: bool = False + _tailbk_c_logged: bool = False + + def prebuild_s4(self, lm_head: nn.Module, draft_model) -> bool: + """Set up the one-launch chain for this worker; False (and nothing set up) when it does not apply.""" + self._s4 = None + if disabled(): + logger.info(f"M3.1 DSpark S4: off ({DISABLE_ENV} set); the in-tree Markov chain runs") + return False + if not getattr(draft_model, "has_markov_head", False): + return False + w1 = draft_model.markov_w1 + w2 = draft_model.markov_w2 + mapping = self.mapping + tp = mapping.tp_size + vocab = w1.shape[0] + if ( + w1.dtype != torch.bfloat16 + or w2.dtype != torch.bfloat16 + or w1.shape[1] != MARKOV_RANK + or tuple(w2.shape) != tuple(w1.shape) + or mapping.enable_attention_dp + or not _head_is_sharded(lm_head, vocab, tp) + ): + return False + block = int(self.max_draft_len) + shard = vocab // tp + rows = chain.padded_rows(shard, MAX_FAST_BATCH, w1.device) + if not self._s4_ranks_agree(rows != 0 and block <= chain.MAX_BLOCK): + return False + offset = mapping.tp_rank * shard + w2s = chain.pad_w2_shard(w2[offset : offset + shard].contiguous(), rows) + state = _S4State( + mapping, rows, shard, offset, w1, w2s, block, chain.workspace_args(mapping) + ) + for batch in range(1, MAX_FAST_BATCH + 1): + state.bufs[(batch, block)] = chain.pad_logits_buffer(batch, block, rows, w1.device) + # S4B8: batch-1 calls take the chain's batch-1 kernel (bitwise the same tokens; faster at one request only). + # Kill switch TRTLLM_M31_DISABLE_S4B8, read at import. + s4b8 = chain.enable_batch1_variant(not m31_disabled("S4B8")) + # MARKOV_LL: those batch-1 calls exchange through the LL flag-in-payload kernel (bitwise; see MARKOV_LL). + markov_ll = chain.enable_ll_exchange(MARKOV_LL) + # TAILBK_SPEC site C: the chain's read-only accepted inputs from a per-shape cache (bitwise; see TAILBK_SPEC). + tailbk_c = chain.enable_cached_inputs(TAILBK_SPEC) + # S4 fp32 sum: the chain compares base + bias as the in-tree chain does, the unrounded fp32 sum (its logits are + # fp32 from LogitsProcessor's .float()). Kill switch TRTLLM_M31_DISABLE_S4_FP32SUM (bf16-rounded sums), read + # at import. + self._s4_fp32 = chain.enable_fp32_sum(not m31_disabled("S4_FP32SUM")) + # S4-logits: under advanced (rejection) draft sampling the chain still computes the greedy chain's tokens; the + # biased block logits are rebuilt from them (_biased_shard_logits) and the stock sampler gathers and samples + # them as it would the in-tree chain's. Kill switch TRTLLM_M31_DISABLE_S4_LOGITS, read at import. + self._s4_logits_on = not m31_disabled("S4_LOGITS") + chain.precompile(mapping, rows, w1, w2s, offset, blocks=(block,), max_batch=MAX_FAST_BATCH) + self._s4 = state + self._s4_draft = draft_model + atexit.register(self._s4_log_count) + # The engagement line: one per rank, at server start. + logger.info( + f"M3.1 DSpark S4: one-launch Markov chain set up on rank {mapping.tp_rank}/{tp} " + f"(shard {shard} rows padded to {rows}, K={block}, batches 1..{MAX_FAST_BATCH}; S4B8 batch-1 kernel " + f"{'on' if s4b8 else 'off'}; MARKOV_LL {'on' if markov_ll and s4b8 else 'off'}; " + f"TAILBK_SPEC cached inputs {'on' if tailbk_c else 'off'}; " + f"S4-logits {'on' if self._s4_logits_on else 'off'}; fp32 sum " + f"{'on' if self._s4_fp32 else 'off'})" + ) + return True + + def _s4_ranks_agree(self, local: bool) -> bool: + """``local`` if every rank of the TP group decided the same; raises when they differ (a rank going on into + the collective workspace alone would hang its peers).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + comm = _get_mnnvl_workspace_comm(self.mapping) + all_on = _mnnvl_workspace_all_succeeded(comm, local) + all_off = _mnnvl_workspace_all_succeeded(comm, not local) + if not (all_on or all_off): + raise RuntimeError( + "m31 DSpark S4: the TP ranks disagree on setting up the one-launch Markov chain" + ) + return all_on + + def _s4_serves(self, num_gens: int, spec_metadata, draft_model) -> bool: + return self._s4_mode(num_gens, spec_metadata, draft_model) is not None + + def _s4_mode(self, num_gens: int, spec_metadata, draft_model): + """ "tokens" (greedy drafting: the chain's tokens are the draft), "logits" (advanced draft sampling with + S4-logits on: the chain's corrected logits go to the stock sampler), or None (the in-tree chain runs).""" + if not self._s4_common(num_gens, draft_model): + return None + if not getattr(spec_metadata, "wants_advanced_draft_sampling", False): + return "tokens" + return "logits" if getattr(self, "_s4_logits_on", False) else None + + def _s4_common(self, num_gens: int, draft_model) -> bool: + return ( + self._s4 is not None + and 1 <= num_gens <= MAX_FAST_BATCH + and getattr(self, "guided_decoder", None) is None + and getattr(self, "_d2t", None) is None + and getattr(draft_model, "lilicorr", None) is None + and not getattr(draft_model, "has_candidate_selector", False) + ) + + def _host_step_hook(self, spec_metadata) -> None: + """The engagement counter (host only, no CUDA): called by the spec metadata's prepare() on every iteration, + outside any captured graph, it evaluates the same eligibility the captured step was routed by and counts + the generation steps the chain serves. One line per rank every S4_COUNT_LOG_EVERY generation steps and at + exit: "M3.1 DSpark S4: served / generation steps". The counts start at server start, so they include + warmup and padding steps: read engagement as the difference of two lines logged after the server is ready.""" + if self._s4 is None: + return + num_gens = int(getattr(spec_metadata, "num_generations", 0) or 0) + if num_gens <= 0: + return + self._s4_gen_steps += 1 + self._s4_served += int(self._s4_serves(num_gens, spec_metadata, self._s4_draft)) + if self._s4_gen_steps % S4_COUNT_LOG_EVERY == 0: + self._s4_log_count() + + def _s4_log_count(self) -> None: + if self._s4 is not None and self._s4_gen_steps: + logger.info( + f"M3.1 DSpark S4: served {self._s4_served}/{self._s4_gen_steps} generation steps on rank " + f"{self.mapping.tp_rank}" + ) + + def _forward_impl(self, input_ids, position_ids, hidden_states, logits, attn_metadata, spec_metadata, draft_model, + resource_manager=None): # fmt: skip + num_gens = attn_metadata.num_seqs - attn_metadata.num_contexts + mode = self._s4_mode(num_gens, spec_metadata, draft_model) + served = mode is not None + if not served: + return super()._forward_impl(input_ids, position_ids, hidden_states, logits, attn_metadata, + spec_metadata, draft_model, resource_manager) # fmt: skip + inner = draft_model.logits_processor + draft_model.logits_processor = _ShardedDraftLogits(inner, self._s4, num_gens) + self._s4_step = self._s4 + self._s4_step_mode = mode + self._s4_step_meta = (attn_metadata.num_contexts, attn_metadata.num_seqs, spec_metadata) + self._s4_step_attn = attn_metadata + try: + return super()._forward_impl(input_ids, position_ids, hidden_states, logits, attn_metadata, + spec_metadata, draft_model, resource_manager) # fmt: skip + finally: + draft_model.logits_processor = inner + self._s4_step = None + self._s4_step_mode = None + self._s4_tokens = None + self._s4_step_meta = None + self._s4_step_attn = None + self._tailbk_rewound = False + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + state = self._s4_step + if state is not None and gen_logits.dim() == 3 and gen_logits.shape[-1] == state.shard: + base = state.buffer(gen_logits.shape[0]) + if ( + chain.supports(base, state.w1, state.w2s) + and getattr(self, "_s4_step_mode", None) == "logits" + ): + # S4-logits: the chain's tokens are the in-tree greedy chain's (bitwise); the biased shard logits are + # rebuilt from them with the in-tree bias arithmetic and handed to the stock advanced sampler. + first_prev = inputs["first_prev_tokens"].long() + tokens = self._s4_chain(state, base, first_prev, spec_metadata) + self._s4_log_ll(base.shape[0]) + biased = _biased_shard_logits( + gen_logits, + first_prev, + tokens, + state.w1, + state.w2s[: state.shard], + self._s4_fp32, + ) + self._s4l_log(gen_logits, first_prev, tokens, state.w1) + return biased + if chain.supports(base, state.w1, state.w2s): + tokens = self._s4_chain( + state, base, inputs["first_prev_tokens"].long(), spec_metadata + ) + self._s4_log_ll(base.shape[0]) + self._s4_tokens = tokens + return gen_logits + return super()._refine_block_logits(draft_model, gen_logits, inputs, spec_metadata) + + def _s4_chain( + self, state: _S4State, base: torch.Tensor, first_prev: torch.Tensor, spec_metadata + ): + """The chain launch; under TAILBK_SPEC (site H) it also applies the step's deferred KV-length rewind when + the stock _apply_kv_rewind_after_draft would (a real step: not the eager warmup of a graph batch, whose + kv_lens is restored wholesale), and marks it done for that method.""" + rewind = self._tailbk_rewind_args(base.shape[0], spec_metadata) + if rewind is None: + tokens = chain.dspark_markov_chain( + base, first_prev, state.w1, state.w2s, state.offset, **state.ws + ) + self._tailbk_c_log() + return tokens + kv_lens, amount, first = rewind + tokens = chain.dspark_markov_chain_rewind(base, first_prev, state.w1, state.w2s, state.offset, kv_lens, + amount, first, **state.ws) # fmt: skip + self._tailbk_rewound = True + self._tailbk_c_log() + if not self._tailbk_h_logged: + self._tailbk_h_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC H: KV-length rewind in the Markov chain launch " + f"on rank {self.mapping.tp_rank}" + ) + return tokens + + def _tailbk_c_log(self) -> None: + """The TAILBK_SPEC C engagement line: once per rank, after the first chain call that took its accepted inputs + from the precompile cache (host only; a captured step logs it while it is captured).""" + if self._tailbk_c_logged or not getattr(chain, "cached_inputs_taken", lambda: False)(): + return + self._tailbk_c_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC C: chain accepted inputs from precompile cache (taken) " + f"on rank {self.mapping.tp_rank}" + ) + + def _tailbk_rewind_args(self, batch: int, spec_metadata): + """(kv_lens, rewind amounts, first row) for the chain to apply, or None (the stock rewind runs).""" + if not TAILBK_SPEC or not getattr(self, "_kv_rewind_pending", False): + return None + amount = getattr(self, "_kv_rewind_amount", None) + attn = self._s4_step_attn + if amount is None or attn is None or not hasattr(attn, "kv_lens_cuda"): + return None + if spec_metadata.is_cuda_graph and not torch.cuda.is_current_stream_capturing(): + return None # the warmup: the stock method skips the rewind (kv_lens is restored wholesale) + nc, bs = self._kv_rewind_nc, self._kv_rewind_bs + if bs - nc != batch or amount.numel() != batch: + return None + kv_lens = attn.kv_lens_cuda + if not chain.supports_rewind(kv_lens, amount, nc): + return None + return kv_lens, amount, nc + + def _apply_kv_rewind_after_draft(self, attn_metadata, spec_metadata): + if self._tailbk_rewound: + # TAILBK_SPEC H: the chain launch applied it (see _s4_chain). + self._tailbk_rewound = False + self._kv_rewind_pending = False + return + super()._apply_kv_rewind_after_draft(attn_metadata, spec_metadata) + + def _s4l_log(self, shard_logits, first_prev, tokens, w1) -> None: + """The TAILBK_SPEC S4L engagement line: once per rank, on the first step whose biased logits took the + two-launch glue (host only; a captured step logs it while it is captured).""" + if self._tailbk_s4l_logged or not (self._s4_fp32 and S4_LOGITS_BATCHED and TAILBK_SPEC): + return + if markov_bias.supports(shard_logits, first_prev, tokens, w1): + self._tailbk_s4l_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC S4L: biased block logits glue in two launches " + f"on rank {self.mapping.tp_rank}" + ) + + def _s4_log_ll(self, batch: int) -> None: + """The MARKOV_LL engagement line: once per rank, on the first served step whose chain call took the LL + exchange kernel (host only; a captured step logs it while it is captured).""" + if not self._s4_ll_logged and chain.ll_exchange_engaged(batch): + self._s4_ll_logged = True + logger.info( + "[M3 dispatch] MARKOV_LL: k3_markov LL flag-in-payload exchange (batch 1) " + f"on rank {self.mapping.tp_rank}" + ) + + def sample_draft_tokens(self, logits, spec_metadata, batch_size, **kwargs): + tokens = self._s4_tokens + if tokens is not None and logits.dim() == 3 and tokens.shape == logits.shape[:2]: + self._s4_tokens = None + return tokens if tokens.dtype == torch.int32 else tokens.to(torch.int32) + return super().sample_draft_tokens(logits, spec_metadata, batch_size, **kwargs) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/m31_switches.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/m31_switches.py new file mode 100644 index 000000000000..51f966c595fd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/m31_switches.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Kill switches of this MiniMax-M3.1 target's default-on items (each target keeps its own copy: targets share +only the catalog). + +Every item is on by default; ``TRTLLM_M31_DISABLE_=1`` turns item ```` off and restores the path it replaced. +The environment is read once, at import, so every rank and every step of a process sees the same setting (a step must +not change its graph mid-run). Each item lists its id and what the switch restores where it reads it. + +Values (case-insensitive, surrounding blanks ignored): ``1`` / ``true`` / ``on`` / ``yes`` set a switch; empty / ``0`` / +``false`` / ``off`` / ``no`` leave it unset; anything else raises at import, naming the variable, so a typo fails loudly +instead of silently turning an item on or off. +""" + +import os +from typing import FrozenSet + +_PREFIX = "TRTLLM_M31_DISABLE_" +_SET = frozenset(("1", "true", "on", "yes")) +_UNSET = frozenset(("", "0", "false", "off", "no")) + + +def _is_set(name: str, value: str) -> bool: + """Whether the switch variable ``name`` is set to ``value``; raises on a value of neither kind.""" + word = value.strip().lower() + if word in _SET: + return True + if word in _UNSET: + return False + raise ValueError( + f"{name}={value!r}: expected 1 / true / on / yes (set) or 0 / false / off / no / empty (unset)" + ) + + +# The item ids switched off in this process, read once at import. +_DISABLED: FrozenSet[str] = frozenset( + name[len(_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_PREFIX) and _is_set(name, value) +) + + +def m31_disabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_DISABLE_`` turned the item off for this process.""" + return item_id in _DISABLED + + +# Items that are OFF by default (rule 1b without a passing gate c, or a failed gate c): ``TRTLLM_M31_ENABLE_=1`` +# opts in; the item's ``TRTLLM_M31_DISABLE_`` still wins. +_ENABLE_PREFIX = "TRTLLM_M31_ENABLE_" + +_ENABLED: FrozenSet[str] = frozenset( + name[len(_ENABLE_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_ENABLE_PREFIX) and _is_set(name, value) +) + + +def m31_enabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_ENABLE_`` opted a default-off item in for this process.""" + return item_id in _ENABLED diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/modeling.py new file mode 100644 index 000000000000..afe87215a221 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/modeling.py @@ -0,0 +1,5647 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: MiniMax-M3.1 (MXFP8 + packed NVFP4 experts) / sm_103 / tp4 -- self-contained modeling code. + +MiniMax-M3.1's language model has MiniMax-M3's shape (60 layers, hidden 6144, 64 query heads and 4 KV heads of dim +128, 128 routed experts top 4 plus one shared expert) with a 1e7 RoPE base and MSA block-sparse attention on EVERY +layer (an index branch scores 128-token blocks and the top 16 are attended); layers 0-2 keep the dense SwiGLU-OAI MLP, +layers 3-59 the MoE. The checkpoint is an MXFP8 export whose routed experts are packed NVFP4 (compressed-tensors +names, global scales stored as multipliers; ``weights`` takes their FP32 reciprocals); the KV cache is NVFP4 with an +FP8 index-K on every layer. The layer kinds are read from the config (``sparse_attention_freq``, ``moe_layer_freq``), +so this code is the MiniMax-M3 target's with no dense-attention layer in use. + +``tp4`` is four GB300s: head-split attention (16 query heads, one KV head and one index head per rank) and +tensor-parallel experts (every expert on every rank, intermediate 768 per rank), no attention DP. Speculative +decoding, when configured, is one-model Eagle3 (draft length 3): the BF16 drafter (``M3Eagle3Draft``) and the BF16 +target head, both at checkpoint precision; decode steps of up to 8 requests verify and draft through fused kernels +at that precision (``M3Eagle3Worker``). + +**The core** (``MiniMaxM3Tp4ModelingV2``) holds the weights (``weights.MODEL_WEIGHTS``) and the runtime resources the +steps use -- one MSA attention backend per layer, the TP all-reduce communicator, the RoPE table, the MXFP8 GEMM +dispatch state the engine tunes -- and dispatches each step, on the host, from its context count alone +(``_select_target``): + +* ``DecodeTarget`` takes every pure generation step. Up to ``DECODE_MAX_TOKENS`` tokens (up to 8 requests, each one + token or a draft verify of up to 1 + 7 rows) with 1-D input ids it runs the decode-step kernels and picks each layer + boundary's collective by the step's token bucket (``DECODE_PLANS``); it runs its own generic layer for the first + layer when the fused embedding does not take the step, for every layer whose output Eagle3 captures, and for every + layer of a wider step. +* ``PrefillTarget`` takes every step with context rows: prefill chunks and mixed steps, every layer in its generic + form over the MSA backend. + +The two targets share no body: each owns its layer, attention and MoE compositions and its ``step_args``, the one +projection of the step metadata its forward reads. What both call are the module-level op adapters, each picking the +kernel for one call from shapes and layouts (``_linear``, ``_qk_norm_rope``, ``_sparse_qkv_producer``, the router and +MoE-runner calls, the all-reduce + norm calls), which read no step metadata themselves. + +Every fast path the decode step can take, the rows each covers (``FAST_PATHS``, each paired with the entry whose +``CELLS`` certify it) and what a step that cannot take one does are below: **uncovered** (outside every covered +range: log once, run the generic path) or **missed** (inside a covered range but a precondition is missing: raise +``FastPathUnavailable`` under ``modeling_v2='require'``, else log once and fall back). + +**Hand-offs between kernels.** A producer that already holds an activation in registers writes its MXFP8 form for the +next MXFP8 projection: the decode boundary all-reduce for the next layer's QKV, the sparse-decode merge and the fused +decode attention for the o_proj. It publishes it in ``ops/m3_prequant.SLOT`` keyed by the BF16 tensor; the next +MXFP8 projection on a decode graph takes it (``_linear``), and every take clears the slot. The decode boundaries' +push targets (the o_proj / shared-expert GEMVs and the MoE engine writing straight into the boundary all-reduce's +exchange) are passed explicitly from the layer to the producer and back. + +**Kill switches** (``m31_switches``, ``TRTLLM_M31_DISABLE_=1``, read once at import): ``R1`` -- the 64-row +decode step bound (back to 48: steps of 49..64 rows run every layer's generic form). ``PROXYPB`` -- the MSA indexer +proxy's fmha_sm100 variants built before warmup (back to compiling each on its first call, which can be after ready). +``LC4_BTSMEM`` / ``LC4_SYNC3`` -- the long-context sparse decode's block-table staging / softmax sync trim (back to the +M3 op once all LC4 options are off; ``ops/attention/m3_sparse_decode_lc``). ``TOPP_DET`` -- the spec sampler's +top-p renorm as the ST-01 C++ op, exact vs FlashInfer's deterministic renorm (back to the stock FlashInfer call; +``ops/sampling/m31_topp_det``). ``SOFTMAX_MC`` -- the spec sampler's softmax with its write pass on 16 CTAs per row +(ST-01-SM), exact vs FlashInfer's softmax (back to the stock FlashInfer call; ``ops/sampling/m31_softmax_mc``). +``REJ_MC`` -- the spec sampler's chain speculative sampling on 16 CTAs per request (ST-01-REJ), exact vs the stock +op (back to ``trtllm::chain_speculative_sampling``; ``ops/sampling/m31_chain_spec_mc``). +``HOSTTAIL`` -- all six between-replay +host items (each also alone: ``HOST_D2H_SIDE``, ``SMALL_H2D``, ``BATCH_CHANGED_ONLY``, ``SKIP_ACCEPTED_UPLOAD``, +``PHILOX_PACK``, ``KV_LENS_ONCE``; back to the stock copies, see ``HOSTTAIL_*``). ``PACK_IN_GRAPH`` / +``PACK_EARLY_COPY`` / ``H2D_CHANGED_SPAN`` (VR2GB-01/-03) -- the step-input pack issued by the decode graph's prologue, +its window uploaded ahead of the replay, changed-span page-table uploads (back to the eager pack and whole-view +uploads; see ``PACK_IN_GRAPH``). ``BOUNDARY_PACK`` (BP-1, needs ``DSFOLD``) -- under DSpark, a batch-1 decode graph's +tail does the spec sampler's four slot stores and the next step's step gather (back to both between the graph replays; +see ``BOUNDARY_PACK``). + +**What this target asserts rather than adapts**: SM 10.3, the topology and checkpoint (``routing.py``); the +MiniMax-M3 MSA sparse attention runtime with the fused five-way QKV/index projection, an FP8 indexer and an FP8 or +NVFP4 KV cache, no torch.compile and no piecewise prefill graphs (``_validate_runtime``); text-only input (the vision +tower is not built and a step carrying image input raises). +""" + +from __future__ import annotations + +import atexit +import copy +import dataclasses +import os +import threading +import time +from dataclasses import dataclass +from types import MappingProxyType +from typing import ( + Any, + Callable, + Dict, + FrozenSet, + Hashable, + Iterable, + List, + Mapping, + Optional, + Tuple, + Union, +) + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch._experimental.modeling_v2._router_index import ( + ModelingV2Mode, + resolved_mode, + step_contract_enabled, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( # noqa: E501 + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_rmsnorm import ( + flashinfer_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.quantization.fp4_quantize import ( + fp4_quantize, +) +from tensorrt_llm._torch.attention.backends import AttentionMetadata +from tensorrt_llm._torch.attention.backends.fmha.msa_prefill import _aligned_nvfp4_dequant_scales +from tensorrt_llm._torch.attention.backends.interface import PositionalEmbeddingParams, RopeParams +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import MiniMaxM3MsaSparseAttention +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + MSA_REQUIRED_TOPK, + msa_paged_kv, + require_msa_module, +) +from tensorrt_llm._torch.attention.backends.utils import create_attention +from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner +from tensorrt_llm._torch.distributed import AllReduce, AllReduceFusionOp, AllReduceParams +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + _ensure_draft_vocab_size, + _get_requested_draft_moe_backend, + _set_draft_kv_cache_quant_algo, + _slice_spec_position_ids, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.modules.embedding import LMHead +from tensorrt_llm._torch.modules.linear import ( + MXFP8LinearMethod, + TensorParallelMode, + flashinfer_mxfp8_autotune_active, + flashinfer_mxfp8_decode_graph_capture_active, +) +from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor +from tensorrt_llm._torch.modules.multi_stream_utils import maybe_execute_in_parallel +from tensorrt_llm._torch.moe.fused_moe.routing import RoutingMethodType +from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph import is_in_breakable_cuda_graph +from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import MambaHybridCacheManager +from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy, RuntimeHooks +from tensorrt_llm._torch.pyexecutor.sampler.ops.flashinfer import ( + set_softmax_override, + set_top_p_renorm_override, +) +from tensorrt_llm._torch.speculative import SpecMetadata +from tensorrt_llm._torch.speculative.eagle3 import Eagle3OneModelWorker +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + set_chain_speculative_sampling_override, + should_use_separate_draft_kv_cache, +) +from tensorrt_llm._torch.utils import ActType_TrtllmGen +from tensorrt_llm._utils import prefer_pinned +from tensorrt_llm.functional import AllReduceStrategy, PositionEmbeddingType +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.mode import QuantAlgo + +from . import dspark as _dspark +from . import weights as _weights +from .m31_switches import m31_disabled, m31_enabled +from .ops import gqamix_sparse_decode as _gqamix_kernel +from .ops import m3_attn_mega as _attn_mega_kernel +from .ops import m3_attn_mega_c1 as _attn_c1_kernel +from .ops import m3_bp_tail, m3_mc_allreduce, m3_prequant, m3_spec_tail +from .ops import m3_draft_step as m3_draft_step_kernel +from .ops import m3_fused_attn as _fused_attn_kernel +from .ops import m3_index_score as _index_score_kernel +from .ops import m3_sampled_accept as m3_sampled_accept_kernel +from .ops import m3_shard_accept as m3_shard_accept_kernel +from .ops import m3_target_head as m3_target_head_kernel +from .ops import msa_kv_update as _msa_kv_update_kernel +from .ops.activation.m3_silu_and_mul import m3_silu_and_mul +from .ops.activation.m3_swq import m3_swq +from .ops.attention.m3_attn_c1 import m3_attn_c1 +from .ops.attention.m3_attn_mega import m3_attn_mega +from .ops.attention.m3_dense_decode import m3_dense_decode +from .ops.attention.m3_draft_attention import m3_draft_attention +from .ops.attention.m3_fp8_indexer_qk_norm_rope import m3_fp8_indexer_qk_norm_rope +from .ops.attention.m3_fused_decode_attn import m3_fused_decode_attn +from .ops.attention.m3_index_score import SPLIT_K as INDEX_SCORE_SPLIT_K +from .ops.attention.m3_index_score import m3_index_score +from .ops.attention.m3_msa_attention import m3_msa_attention +from .ops.attention.m3_producer import m3_producer +from .ops.attention.m3_qk_norm_rope_fp8 import m3_qk_norm_rope_fp8 +from .ops.attention.m3_qkv_insert import m3_qkv_insert +from .ops.attention.m3_select_blocks import m3_select_blocks +from .ops.attention.m3_select_long import MAX_BLOCKS as SELECT_LONG_MAX_BLOCKS +from .ops.attention.m3_select_long import m3_select_long +from .ops.attention.m3_sparse_decode import m3_sparse_decode +from .ops.comm.m3_allreduce import m3_allreduce +from .ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm +from .ops.comm.m3_mc_finalize_allreduce_norm import m3_mc_finalize_allreduce_norm +from .ops.embedding.m3_embedding import m3_embedding +from .ops.gemm.m3_bf16_linear import m3_bf16_linear +from .ops.gemm.m3_fp32_linear import m3_fp32_linear +from .ops.gemm.m3_mx_gemv import ( + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, + m3_mx_gemv, +) +from .ops.gemm.m3_mx_gemv_bf16 import m3_mx_gemv_bf16 +from .ops.gemm.m3_mxfp8_gemm import m3_mxfp8_gemm, m3_mxfp8_gemm_cute_dsl +from .ops.gemm.m3_mxfp8_linear import m3_mxfp8_linear +from .ops.gemm.m3_mxfp8_pinned_quantize import m3_mxfp8_pinned_quantize +from .ops.gemm.m3_target_head import MAX_ROWS as TARGET_HEAD_KERNEL_MAX_ROWS +from .ops.gemm.m3_target_head import m3_target_head +from .ops.gemm.mxfp8_low_m_gemm import mxfp8_low_m_gemm +from .ops.long_select import long_select_blocks +from .ops.m3_router_gemv import m3_router_gemv as _router_gemv +from .ops.m3_router_gemv import m3_router_gemv_fp4 as _router_gemv_fp4 +from .ops.m3_router_gemv import m3_router_tc as _router_tc +from .ops.moe.m3_moe16_engine import M3Moe16State, m3_moe16_engine +from .ops.moe.m3_moe_engine import M3MoeState, ar_push_desc, m3_moe_engine +from .ops.moe.m3_router_gemv import m3_router_gemv +from .ops.moe.m3_router_gemv_fp4 import m3_router_gemv_fp4 +from .ops.moe.m3_router_tc import m3_router_tc +from .ops.mx_gemv import op as mx_gemv +from .ops.mxfp8_low_m_gemm import mxfp8_low_m_gemm as _low_m_gemm +from .ops.norm.m3_embedding_firstnorm import DecodeEmbedding, m3_embedding_firstnorm +from .ops.sampling import m31_chain_spec_mc as _chain_spec_mc +from .ops.sampling import m31_softmax_mc as _softmax_mc +from .ops.sampling import m31_topp_det as _topp_det +from .ops.spec.m3_draft_input import m3_draft_input +from .ops.spec.m3_draft_step import MAX_TOKENS as DRAFT_STEP_KERNEL_MAX_TOKENS +from .ops.spec.m3_draft_step import m3_draft_step +from .ops.spec.m3_force_vec import m3_force_vec +from .ops.spec.m3_shard_accept import m3_shard_accept +from .ops.triton_sparse_decode import MERGE_MXFP8, MERGE_MXFP8_MIN_TOKENS + +_SM = (10, 3) +# -------------------------------------------------------------------------------------------------------------------- +# Row ranges +# -------------------------------------------------------------------------------------------------------------------- + +# Kill switch ``TRTLLM_M31_DISABLE_PROXYPB``: compile the MSA indexer proxy's fmha_sm100 variants before warmup +# (``_prebuild_msa_proxy``). Compile only; off, each compiles on its first call. +MSA_PROXY_PREBUILD = not m31_disabled("PROXYPB") +# CTXNORMQ (CTX lane): a prefill layer boundary of at least ``CTXNORMQ_MIN_ROWS`` rows runs the plain all-reduce, then +# one kernel for the residual add + RMSNorm that also writes the next MXFP8 projection's activation (and, at a MoE +# boundary, the routed experts' NVFP4 input), instead of the fused all-reduce + norm followed by the consumers' own +# quantize passes. The quantized outputs equal mxfp8_quantize / fp4_quantize of the norm bit for bit; the norm itself +# is rule 1b against the fused all-reduce's (reduction order). Kill switch: TRTLLM_M31_DISABLE_CTXNORMQ=1. +CTXNORMQ = not m31_disabled("CTXNORMQ") +CTXNORMQ_MIN_ROWS = ( + 1024 # capacity: NORMQ microbenchmark: the fused pass wins at 1160 and 16384 rows +) +# AR64AUTO: a generation step past the multicast all-reduce's rows (K7 c8: 64 +# verify rows) runs the built-in fused all-reduce + residual + RMSNorm with the fixed strategy MNNVL instead of the +# configured one; the serving configs pin NCCL, 29.1 vs 11.6 us per boundary at 64 rows on GB300 TP8 (with the next +# projection's MXFP8 quantize). Pinned, not AUTO: AUTO's choice is timing-autotuned per process; MNNVL is bitwise +# identical across processes at 49..64 rows. Rule 1b against the configured strategy (another reduction order +# across ranks); every other boundary keeps the configured instance. Kill switch: TRTLLM_M31_DISABLE_AR64AUTO=1. +AR64AUTO = not m31_disabled("AR64AUTO") +# TSEL (CTX lane): a PrefillTarget sparse layer's top-16 block select runs the #20629 prefill tile select (one CTA per +# KV head and 8 queries, each 32-byte score sector read once) on sm_103, registered on the MSA indexer's selector hook. +# The table equals minimax_m3_select_blocks' bit for bit; steps outside its rows / blocks range keep the built-in +# select. Kill switch: TRTLLM_M31_DISABLE_TSEL=1. +TSEL = not m31_disabled("TSEL") +TSEL_MIN_ROWS = 256 # microbenchmark: x1.4 (128 blocks) to x4.4 (1024), x1.6 at 1160 rows +TSEL_MAX_BLOCKS = 5856 # the op's shared-memory bound (kMinimaxM3SelectPrefillTileMaxBlocks) +# Gate a in-model (correctness runs only): TRTLLM_M31_CHECK_TSEL=1 also runs minimax_m3_select_blocks on every +# tile-selected step and raises on any table difference. +TSEL_CHECK = os.environ.get("TRTLLM_M31_CHECK_TSEL") == "1" +# GQAMIX (MSA): a mixed ctx+gen step's sparse generation rows (MsaDecodeFmha) run the Triton block-sparse decode +# through ops/gqamix_sparse_decode, a verbatim copy of the in-tree kernel module whose two decode kernels do not +# specialize on the step's row count or on the [num_ctx:] slices' alignment, registered on MsaDecodeFmha's +# sparse_decode_hook. Those specializations compiled new variants after the server was ready (microbenchmark: +# bitwise in every cell, 0 new variants vs 1, at most +0.07 us). Kill switch: TRTLLM_M31_DISABLE_GQAMIX=1. +GQAMIX = not m31_disabled("GQAMIX") +# DSFOLD: under the standalone DSpark worker, the engine's previous-batch input gathers (input ids, draft tokens, +# position / KV-length offsets) run as one m3_spec_tail.step_gather launch instead of ~a dozen eager index and copy +# launches at the step boundary. Integer copies of the same values into the same buffers; the sampler keeps its +# in-tree slot stores. Kill switch: TRTLLM_M31_DISABLE_DSFOLD=1. +DSFOLD = not m31_disabled("DSFOLD") +# BOUNDARY_PACK BP-1 (with DSFOLD): a batch-1 decode graph ends with the spec sampler's four +# slot stores (the same values at the same slot of the same store) and the engine's next-step gathers, in one launch +# (ops/m3_bp_tail) gated by a device (slot, live) word the fold writes before the replay, so neither runs eagerly +# between two graph replays: the sampler skips its stores (``sampler_rows_stored``) and the engine its gathers +# (``begin_step``) on the steps the graph did them, and a replay with no identified row (a dummy-only batch) writes +# nothing. Only batch-1 graphs (never padded: the row is the step's one request); every other step keeps today's path. +# Kill switch: TRTLLM_M31_DISABLE_BOUNDARY_PACK=1. +BOUNDARY_PACK = DSFOLD and not m31_disabled("BOUNDARY_PACK") +BOUNDARY_PACK_MAX_ROWS = 1 +# The key of the forward outputs that tells the sampler the step's graph stored its rows (BOUNDARY_PACK). +_BOUNDARY_PACK_MARK = "m31_boundary_pack" +# Prebuild/workspace capacity retains the historical R1 shapes. Fused-layer dispatch +# is limited separately below; each operation also retains its own supported row range. +# Wider pure-generation steps use the target generic layers. +DECODE_MAX_TOKENS = ( + 48 if m31_disabled("R1") else 64 +) # prebuild/workspace capacity; dispatch is bounded separately +# Fused-layer dispatch scope; keep the larger prebuild/workspace capacity above. +FUSED_DECODE_MAX_TOKENS = 32 + + +@dataclass(frozen=True) +class DecodePlan: + """The layer-boundary collectives of one decode token bucket (GB300 tp4 measurements in the comments).""" + + name: str + max_tokens: int + # The attention boundary's multicast all-reduce + residual + RMSNorm (5.05 vs 5.27 us native at 1 token). + attn_mc_ar: bool + # The same kernel at the MoE boundary with the shared + routed add folded in: wins up to 48 tokens. + moe_mc_ar: bool + # The MoE-boundary all-reduce also emits the next QKV activation in MXFP8 for the decode GEMV / GEMM. + qkv_prequant: bool + + +# tiny: every step up to 8 tokens (c1..c8 without speculation, the c1 verify, every draft pass); small: the c2 +# verify; verify: the c4 / c8 verifies. Each is one multicast all-reduce bucket (ops/m3_mc_allreduce.BUCKETS); +# rows of the verify plan past the all-reduce's rows (m3_mc_allreduce.max_rows()) take the built-in all-reduce. +DECODE_PLANS = ( + DecodePlan("tiny", 8, attn_mc_ar=True, moe_mc_ar=True, qkv_prequant=True), + DecodePlan("small", 16, attn_mc_ar=True, moe_mc_ar=True, qkv_prequant=True), + DecodePlan("verify", DECODE_MAX_TOKENS, attn_mc_ar=True, moe_mc_ar=True, qkv_prequant=True), +) + +# The dense layers' decode attention (trtllm-gen, msa_decode.dense_decode_max_seq_len): the max_seq_len it is handed +# per request count. It only sizes the multi-CTA KV split (every CTA still covers its share of the actual length; the +# result changes only by the split-merge order), and the model maximum over-splits AgentX-length caches at one +# request: GB300 TP4, 16 heads, FP8 KV, 64K context: 15.1 us at 16K vs 28.6 us at 1M. Two requests are neutral (the +# grid is unchanged) and keep the step's bound. +DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH: Mapping[int, int] = MappingProxyType({1: 16384}) + +# The sparse layers' NVFP4 QKV + indexer producer (ops/attention/m3_producer, every step its geometry fits) triggers its +# dependent kernel on entry, so the index scorer launches and stages its pre-wait loads while the producer runs (its +# outputs are read only after the scorer's own grid wait). +PRODUCER_EARLY_TRIGGER = True + +# The fused embedding + first RMSNorm kernel: one owner per row count and index dtype. +EMBED_FIRSTNORM_MAX_TOKENS = 8 # capacity: fixed: built rows; wider steps embed generically +# The o_proj pushes its rows into the attention boundary's multicast all-reduce (prepushed mode 2) up to this many +# tokens; wider steps run the GEMV and a plain multicast all-reduce. +ATTN_PUSH_MAX_TOKENS = 16 # capacity: fixed: measured push crossover +AR_PUSH_MODE = 2 +# The 1..8-token MoE engine (router, routed experts and the shared expert in one launch) pushes its sum into the MoE +# boundary's all-reduce (prepushed mode 2). +MOE_ENGINE_MAX_TOKENS = 8 # capacity: fixed: the engine's built rows +MOE_PUSH_MAX_TOKENS = MOE_ENGINE_MAX_TOKENS # capacity: fixed: the engine's rows +# The 9..13-row engine (router, routed and shared experts in one kernel); its bf16 sum goes to the boundary. +MOE16_MIN_TOKENS = 9 # capacity: fixed: the 9..13-row engine's lower bound +MOE16_MAX_TOKENS = 13 # capacity: fixed: the 9..13-row engine's built rows +# From this many tokens the MoE runs unfinalized through the TRTLLM-Gen runner and its finalize (top-k gather, +# routing weights, shared expert add) runs inside the boundary all-reduce. 9..13 take it only when their engine is +# missing (outside require mode). +MC_FINALIZE_MIN_TOKENS = 9 # capacity: fixed: the rows above the engines +# The router: the fp32 GEMV at one row, the tensor-core router (fused with the routed experts' NVFP4 activation +# quantization) at 2..48 rows. GB300 tp4, behind a PDL producer, with the NVFP4 activation: 2.90 / 2.93 / 3.20 / +# 3.82 / 7.36 us at 1 / 4 / 8 / 12 / 48 tokens against the fp32 GEMV's 2.85 / 3.96 / 4.88 / 6.01 / - us. +ROUTER_GEMV_MAX_TOKENS = 1 # capacity: fixed: measured router crossover +ROUTER_TC_MAX_TOKENS = 48 # capacity: fixed: the tensor-core router's built rows +# The fused norm/RoPE/KV-insert/sparse-decode/merge kernel: up to 8 query rows per request and 16 query rows per +# step (above, its 16-CTA clusters run in waves and the three-kernel path is faster: GB300 -13% fAL spec c8). +FUSED_ATTN_MAX_QUERY_LEN = 8 # capacity: fixed: compiled query rows +FUSED_ATTN_MAX_TOKENS = 16 # capacity: fixed: measured wave crossover +# One request verifying a draft of 6 query rows runs attention + o_proj as one launch (96 CTAs: 6 clusters of 16). +ATTN_C1_QUERY_LEN = 6 # capacity: fixed: the verify kernel's query rows +# A greedy Eagle3 verify step of up to this many requests verifies with the fused shard argmax + acceptance kernel +# over the target lm_head's BF16 shard logits (ops/m3_shard_accept), at the configured draft length. +SHARD_ACCEPT_DRAFT_LEN = 3 # capacity: fixed: the built Eagle3 draft length +SHARD_ACCEPT_MAX_REQUESTS = min(DECODE_MAX_TOKENS, m3_shard_accept_kernel.SHARD_ROWS) // ( + SHARD_ACCEPT_DRAFT_LEN + 1 +) # capacity: fixed: the kernel's verify rows / (draft length + 1) +# A fused-verify step of up to this many verify rows (one or two requests at draft length 3) computes the target +# logits shard from the 12-bit coded copy of the BF16 head (ops/m3_target_head; the same BF16 x BF16 -> fp32 +# arithmetic, 75% of the bytes) instead of the in-tree head GEMM; wider verifies keep the in-tree GEMM. +TARGET_HEAD_MAX_ROWS = TARGET_HEAD_KERNEL_MAX_ROWS +# On a fused-verify step of up to this many verify rows the drafter's first fc is forked onto the aux stream right +# after the last captured target layer, overlapping the remaining target layers and the head (spec_worker). +FC_EARLY_MAX_ROWS = 8 # capacity: fixed: measured fork crossover +# A decode-only Eagle3 step of up to this many requests drafts with the fused draft loop (spec_worker): per pass the +# draft input in one launch (ops/m3_spec_tail.draft_input), the drafter layer's in-tree attention, MLP and +# all-reduces, and one kernel for the draft head (its 12-bit coded copy, or the BF16 fragments when the coding does +# not decode bit for bit), the cross-rank argmax and the pass's bookkeeping (ops/m3_draft_step), at the +# configured draft length. The head kernel takes one row per request, at most 8. +# The fused draft loop streams the 12-bit coded copy of the draft head (when it decodes bit for bit); False keeps the +# BF16 fragments. +DRAFT_HEAD_CODED = True +DRAFT_STEP_DRAFT_LEN = SHARD_ACCEPT_DRAFT_LEN # capacity: fixed: the built draft length +DRAFT_STEP_MAX_REQUESTS = DRAFT_STEP_KERNEL_MAX_TOKENS +# The identity-selection sparse decode moves from the per-row producer to the multi-query producer at this many rows +# (ops/triton_sparse_decode.MQ_MIN_TOKENS). +SPARSE_DECODE_MQ_MIN_TOKENS = 17 # capacity: fixed: the multi-query crossover +# The long-row top-16 selector takes engines of 129..LONG_SELECT_MAX_BLOCKS KV blocks of 128 tokens; it holds 2, 4 or +# 9 keys per thread by the row's valid-block count. At most 128 blocks the built-in selector runs. +LONG_SELECT_MIN_BLOCKS = 129 # capacity: fixed: past the built-in selector's 128 +LONG_SELECT_MAX_BLOCKS = SELECT_LONG_MAX_BLOCKS +LONG_SELECT_KEY_WIDTHS = (2, 4, 9) +# The long-row select releases the sparse decode after it right after its own grid wait (the decode stages q and the +# request length before its wait) up to this many query rows; on wider steps the early-launched attention CTAs crowd +# the select's CTAs off the SMs (8 requests x 4 rows: +3.1 us per layer), so they keep the trigger at the select's exit. +SELECT_EARLY_MAX_ROWS = 8 # capacity: fixed: measured early-release crossover +# The MSA index scorer (ops/attention/m3_index_score) prefetches the older index-K pages into L2 before its grid wait +# at every batch; up to this many requests it also issues Q and the first K tile right after the wait (larger batches +# are bandwidth bound, where the extra early tiles only crowd the prefetch). +INDEX_SCORE_EARLY_MAX_REQUESTS = 1 # capacity: fixed: measured early-issue crossover + +# The decode MXFP8 GEMVs, per projection (N, K): the rows each serves. On an activation its producer quantized +# (the MoE boundary for the QKV, the attention merge for the o_proj) the GEMV serves every row up to its widest MMA +# (mx_gemv.MAX_TOKENS) but the unsplit K=768 shared down projection at 17..32 rows, where FlashInfer's GEMM is faster. +_ALL_ROWS = frozenset(range(1, min(DECODE_MAX_TOKENS, mx_gemv.MAX_TOKENS) + 1)) +MX_GEMV_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = { + QKV_SPARSE: _ALL_ROWS, + QKV: _ALL_ROWS, + O_PROJ: _ALL_ROWS, + SHARED_GATE_UP: _ALL_ROWS, + SHARED_DOWN: _ALL_ROWS - frozenset(range(17, 33)), + DENSE_GATE_UP: _ALL_ROWS, + DENSE_DOWN: _ALL_ROWS, +} +# SHDOWN pin: the shared down projection on the rows the GEMV does not serve runs FlashInfer's CuTe DSL MXFP8 GEMM, +# fixed. The AutoTuner tunes this shape live per start and can pick CUTLASS (cold decode weights, 57 per step, GB300: +# 6.5 vs 3.1 us per call; microbenchmark); the pin removes that per-start choice. TRTLLM_M31_DISABLE_SHDOWN_PIN=1 +# restores the tuned backend. +SHDOWN_PIN_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + {} if m31_disabled("SHDOWN_PIN") else {SHARED_DOWN: frozenset(range(17, 33))} +) +_SHDOWN_LOGGED: set = set() +# SWQ: at these rows the shared expert's SwiGLU writes the down projection's MXFP8 activation directly (ops/activation +# /m3_swq.py, bitwise to silu_and_mul + mxfp8_quantize), which then takes the prequant branch of ``_linear`` (the SHDOWN +# pin at 17..32). GB300 TP4, 32 rows: 5.41 vs 6.38 us per call with its GEMM; at <= 16 rows the +# bf16 GEMV (in-kernel quantize) stays faster or is a different quantize. TRTLLM_M31_DISABLE_SWQ=1 restores the bf16 +# SwiGLU + quantize. +SWQ_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + {} if m31_disabled("SWQ") else {SHARED_DOWN: frozenset(range(17, 33))} +) +_SWQ_LOGGED: set = set() +# TOPP_DET: the spec sampler's top-p renorm (FlashInfer top_p_renorm_probs in the no-top-k rejection +# path, the draft's and the verify's call each step) runs as one persistent C++ launch +# (ops/sampling/m31_topp_det, kernels/minimaxM31/toppRenormDetSm103.cu) through the sampler's top-p renorm override, +# instead of FlashInfer's five launches with a one-CTA-per-row apply. Bitwise FlashInfer's deterministic renorm on +# rows with top_p < 1 (microbenchmark: probs and sampled token, 128 cells, eager and graph); rows with top_p >= 1 +# keep every token (FlashInfer drops tail buckets there in ~43% of rows). GB300 per call at +# 7 / 8 peaked rows: 54.3 / 54.8 vs 104.2 / 105.1 us (v1.2 MB; -100 us/step at K7 c1), flat on high-entropy +# rows (near-uniform 59.0 vs 248.5; v1.2 bitwise v1 on every MB row). Taken up to +# ``_topp_det.MAX_ROWS`` rows; larger calls, inputs outside the contract and builds without the op keep the stock +# call. Kill switch: TRTLLM_M31_DISABLE_TOPP_DET=1 (stock FlashInfer top_p_renorm_probs). +TOPP_DET = not m31_disabled("TOPP_DET") +_TOPP_DET_STATE: dict = {} + + +def _install_topp_det() -> None: + """Once per process: install ``_topp_det.top_p_renorm_det`` as the sampler's top-p renorm override when TOPP_DET + is on and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _TOPP_DET_STATE: + return + if not TOPP_DET: + state = "off (TRTLLM_M31_DISABLE_TOPP_DET=1); stock FlashInfer top_p_renorm_probs" + elif _topp_det.topp_det_op() is None: + state = "not taken: the op is not built; stock FlashInfer top_p_renorm_probs" + else: + set_top_p_renorm_override(_topp_det.top_p_renorm_det) + ctas = _topp_det.ctas_per_row(_topp_det.MAX_ROWS, _topp_det.sm_count(torch.device("cuda"))) + state = f"engaged (ctas_per_row={ctas}, rows <= {_topp_det.MAX_ROWS})" + _TOPP_DET_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01 TOPP_DET det top-p renorm: {state}") + + +def _prebuild_topp_det() -> None: + """The override's persistent workspace on this rank's device, allocated before warmup and any graph capture.""" + if _TOPP_DET_STATE.get("installed"): + _topp_det.prebuild() + + +# SOFTMAX_MC: the spec sampler's softmax (FlashInfer softmax(logits, temperature) in softmax_op and +# compute_probs_from_logits_op, the draft's and the verify's call each step) keeps FlashInfer's Map kernel and runs +# its Reduce kernel's write pass on ``_softmax_mc.WRITE_CTAS`` CTAs per row (ops/sampling/m31_softmax_mc, +# kernels/minimaxM31/softmaxMcSm103.cu) through the sampler's softmax override, instead of one 1024-thread CTA per +# row. Bitwise FlashInfer's softmax on its Path A shapes (rows <= 128, vocab >= 24576; microbenchmark at write +# CTAs 4 / 8 / 16). GB300 per call at 7-8 rows: 9.2 vs 21.8 us. Taken up to ``_softmax_mc.MAX_ROWS`` rows; larger +# calls, inputs outside the contract and builds without the op keep the stock call. Kill switch: +# TRTLLM_M31_DISABLE_SOFTMAX_MC=1 (stock FlashInfer softmax). +SOFTMAX_MC = not m31_disabled("SOFTMAX_MC") +_SOFTMAX_MC_STATE: dict = {} + + +def _install_softmax_mc() -> None: + """Once per process: install ``_softmax_mc.softmax_mc`` as the sampler's softmax override when SOFTMAX_MC is on + and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _SOFTMAX_MC_STATE: + return + if not SOFTMAX_MC: + state = "off (TRTLLM_M31_DISABLE_SOFTMAX_MC=1); stock FlashInfer softmax" + elif _softmax_mc.softmax_mc_op() is None: + state = "not taken: the op is not built; stock FlashInfer softmax" + else: + set_softmax_override(_softmax_mc.softmax_mc) + state = f"engaged (write_ctas={_softmax_mc.WRITE_CTAS}, rows <= {_softmax_mc.MAX_ROWS})" + _SOFTMAX_MC_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01-SM softmax multi-CTA write: {state}") + + +def _prebuild_softmax_mc() -> None: + """The override's persistent partials workspace on this rank's device, allocated before warmup and any graph + capture.""" + if _SOFTMAX_MC_STATE.get("installed"): + _softmax_mc.prebuild() + + +# REJ_MC: the spec sampler's rejection step (trtllm::chain_speculative_sampling in +# speculative/interface.rejection_sampling_one_model, once per verify step) runs on ctas_per_row CTAs per request +# (ops/sampling/m31_chain_spec_mc, kernels/minimaxM31/chainSpecSamplingMcSm103.cu) through the rejection sampler's +# override, instead of one 1024-thread CTA per request. Bitwise the stock op's tokens and counts (microbenchmark: +# K7, batch 1 / 2 / 4 / 8 x 5 cases x 64 seeds at ctas_per_row 4 / 8 / 16, eager and graph). GB300 per call at batch 1: +# 31.2 vs 41.2 us. Taken up to ``_chain_spec_mc.MAX_BATCH`` requests; larger calls, inputs outside the contract and +# builds without the op keep the stock op. Kill switch: TRTLLM_M31_DISABLE_REJ_MC=1 (stock +# trtllm::chain_speculative_sampling). +REJ_MC = not m31_disabled("REJ_MC") +_REJ_MC_STATE: dict = {} + + +def _install_rej_mc() -> None: + """Once per process: install ``_chain_spec_mc.chain_spec_mc`` as the rejection sampler's override when REJ_MC is + on and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _REJ_MC_STATE: + return + if not REJ_MC: + state = "off (TRTLLM_M31_DISABLE_REJ_MC=1); stock trtllm::chain_speculative_sampling" + elif _chain_spec_mc.chain_spec_mc_op() is None: + state = "not taken: the op is not built; stock trtllm::chain_speculative_sampling" + else: + set_chain_speculative_sampling_override(_chain_spec_mc.chain_spec_mc) + ctas = _chain_spec_mc.ctas_per_row( + _chain_spec_mc.MAX_BATCH, _chain_spec_mc.sm_count(torch.device("cuda")) + ) + state = f"engaged (ctas_per_row={ctas}, batch <= {_chain_spec_mc.MAX_BATCH})" + _REJ_MC_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01-REJ chain speculative sampling multi-CTA: {state}") + + +def _prebuild_rej_mc() -> None: + """The override's persistent workspace on this rank's device, allocated before warmup and any graph capture.""" + if _REJ_MC_STATE.get("installed"): + _chain_spec_mc.prebuild() + + +# QPIN: the decode-graph MXFP8 quantize ahead of the tuned GEMM runs a fixed tactic instead of the AutoTuner's pick, +# which PERSIST freezes from one learn run's profiling (stack4: (16, 6144) persisted TRT-LLM at a noisy 10.5 us; +# measured 2.85 vs CuTe DSL 1.78 us, microbenchmark). Both tactics are bitwise equal; CuTe DSL is faster at every +# measured shape but (32, 768), where TRT-LLM is (1.88 vs 2.03). The GEMM keeps its tuned tactic (same cache key). +# TRTLLM_M31_DISABLE_QPIN=1 restores the tuned quantize + GEMM. +QPIN = not m31_disabled("QPIN") +QPIN_TACTIC_EXCEPTIONS: Dict[Tuple[int, int], int] = {(32, 768): MXFP8QuantizeRunner.TRTLLM} +_QPIN_LOGGED: set = set() +# HOSTTAIL (LongAttn; VR lineage c972d6f162): six exact host-side items between two decode graph replays, each a +# RuntimeHooks field below (``_runtime_hooks``). Same bytes reach the device; only where / whether a copy is issued +# changes. ``TRTLLM_M31_DISABLE_HOSTTAIL=1`` turns all six off; each also has its own kill switch: +# * ``HOST_D2H_SIDE`` (KIMI-HO-04, sampler_d2h_side_stream): the spec sampler's store D2H copies go on its D2H side +# stream, so they no longer sit between graph N's stores and graph N+1 on the main stream. +# * ``SMALL_H2D`` (KIMI-HO-12a, spec_small_h2d): the sampler's slot ids stay in a device buffer re-uploaded only on +# change, and the accepted-draft counts ride the step-input pack. +# * ``BATCH_CHANGED_ONLY`` (KIMI-HO-03 / ST-18(d), spec_batch_map_changed_only): the DFlash/DSpark drafter's +# batch_indices and batch-to-slot map are uploaded only when they change (constant in steady decode). +# * ``SKIP_ACCEPTED_UPLOAD`` (KIMI-HO-09, skip_accepted_draft_upload): the per-step accepted-draft-count upload is +# skipped; nothing on this target reads spec_metadata.num_accepted_draft_tokens (it is None). +# * ``PHILOX_PACK`` (KIMI-ST-18(e), pack_spec_rng_uploads): a sampled step's four Philox seed / offset uploads ride the +# step-input pack instead of four copies between the graphs. +# * ``KV_LENS_ONCE`` (VR2GB-04, kv_lens_update_once): a speculative overlap step re-derives the MSA KV-length metadata +# once, after the KV-length correction; the first pass at the head of the decode graph is rewritten in full. +_HOSTTAIL_OFF = m31_disabled("HOSTTAIL") +HOSTTAIL_D2H_SIDE = not (_HOSTTAIL_OFF or m31_disabled("HOST_D2H_SIDE")) +HOSTTAIL_SMALL_H2D = not (_HOSTTAIL_OFF or m31_disabled("SMALL_H2D")) +HOSTTAIL_BATCH_CHANGED_ONLY = not (_HOSTTAIL_OFF or m31_disabled("BATCH_CHANGED_ONLY")) +HOSTTAIL_SKIP_ACCEPTED_UPLOAD = not (_HOSTTAIL_OFF or m31_disabled("SKIP_ACCEPTED_UPLOAD")) +HOSTTAIL_PHILOX_PACK = not (_HOSTTAIL_OFF or m31_disabled("PHILOX_PACK")) +HOSTTAIL_KV_LENS_ONCE = not (_HOSTTAIL_OFF or m31_disabled("KV_LENS_ONCE")) +# PACK_PSCATTER (VR2GB-02, pack_parallel_scatter): the step-input pack issues its non-overlapping copies with one +# program per copy instead of the serial one-CTA scatter (same bytes); TRTLLM_M31_DISABLE_PACK_PSCATTER=1 restores it. +PACK_PSCATTER = not m31_disabled("PACK_PSCATTER") +# VR2GB-01 / VR2GB-03 (between-replay step-input hooks; each switch also turns off the items that need it): +# * ``PACK_IN_GRAPH`` (pack_in_graph; needs PACK_PSCATTER): the decode graphs are captured with the step-input pack's +# prologue, so the pack's upload and scatter run inside the replay and no operation is left between the graphs. +# TRTLLM_M31_DISABLE_PACK_IN_GRAPH=1 (or a disabled PACK_PSCATTER) restores the eager upload and scatter. +# * ``PACK_EARLY_COPY`` (pack_early_copy; needs PACK_IN_GRAPH): the prologue's window is uploaded on a side copy stream +# while the previous graph still runs, so the replay does not start with a host copy. +# TRTLLM_M31_DISABLE_PACK_EARLY_COPY=1 restores the prologue's own copy of the pinned window. +# * ``H2D_CHANGED_SPAN`` (h2d_changed_span): a changed packed page-table upload stages only each row's changed column +# span, so a page-crossing step keeps fitting the in-graph window. TRTLLM_M31_DISABLE_H2D_CHANGED_SPAN=1 restores +# the whole-view upload. +PACK_IN_GRAPH = PACK_PSCATTER and not m31_disabled("PACK_IN_GRAPH") +PACK_EARLY_COPY = PACK_IN_GRAPH and not m31_disabled("PACK_EARLY_COPY") +H2D_CHANGED_SPAN = not m31_disabled("H2D_CHANGED_SPAN") +# On a bf16 activation the GEMV quantizes it in its prologue, which pays only up to 8 32-element blocks per thread: +# the most rows each projection takes. +MX_GEMV_BF16_MAX_TOKENS: Dict[Tuple[int, int], int] = { + QKV_SPARSE: 16, + QKV: 16, + O_PROJ: 32, + SHARED_GATE_UP: 16, + SHARED_DOWN: 16, + DENSE_GATE_UP: 8, + DENSE_DOWN: 16, +} +# The fused low-M MXFP8 GEMM (projections marked for it), per projection: its built configurations' row limit +# (ops/mxfp8_low_m_gemm). The bf16 GEMV serves every one of these rows first; the GEMM takes them only if it +# declines. +LOW_M_GEMM_MAX_TOKENS: Dict[Tuple[int, int], int] = { + QKV_SPARSE: 8, + O_PROJ: 8, + DENSE_GATE_UP: 1, + DENSE_DOWN: 8, +} + +Key = Hashable + + +@dataclass(frozen=True) +class RowRange: + """The shapes one fast path takes, and the catalog entry whose cells certify each of them. + + ``keys`` are the covered shapes: row counts for most paths, (query rows per request, requests) for the attention + kernels, a variant name for thresholds. ``certifies`` maps a cell's spec of ``entry`` to the keys that cell + drives (empty for a cell of another variant). + """ + + path: str + keys: FrozenSet[Key] + entry: OpWrapper + certifies: Callable[[Mapping[str, Any]], Iterable[Key]] + + def covers(self, key: Key) -> bool: + return key in self.keys + + def certified(self) -> FrozenSet[Key]: + return frozenset(k for cell in self.entry.CELLS for k in self.certifies(cell.spec)) + + +def _rows(lo: int, hi: int) -> FrozenSet[int]: + return frozenset(range(lo, hi + 1)) + + +def _spec_rows(field: str, **match: Any) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + """A cell's ``spec[field]`` when the cell's other fields match ``match``.""" + + def certifies(spec: Mapping[str, Any]) -> Iterable[Key]: + if any(spec.get(name) != value for name, value in match.items()): + return () + return (spec[field],) + + return certifies + + +def _const(*keys: Key) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + return lambda spec: keys + + +def _fused_attn_keys() -> FrozenSet[Tuple[int, int]]: + return frozenset( + (qlen, reqs) + for qlen in range(1, FUSED_ATTN_MAX_QUERY_LEN + 1) + for reqs in range(1, FUSED_ATTN_MAX_TOKENS // qlen + 1) + ) + + +def _sparse_decode_producer(spec: Mapping[str, Any]) -> Iterable[Key]: + if not spec["identity"]: + return () + rows = spec["decode_query_len"] * len(spec["seq_lens"]) + return ("multi_query" if rows >= SPARSE_DECODE_MQ_MIN_TOKENS else "per_row",) + + +def _long_select_widths(spec: Mapping[str, Any]) -> Iterable[Key]: + """The key widths a cell's rows reach: every width up to its blocks' when rows of mixed lengths are drawn.""" + blocks = spec["num_blocks"] + widest = 2 if blocks <= 2048 else 4 if blocks <= 4096 else 9 + if spec["valid"] == "mixed": + return tuple(w for w in LONG_SELECT_KEY_WIDTHS if w <= widest) + return (widest,) + + +EMBED_FIRSTNORM = RowRange( + "embed_firstnorm", + _rows(1, EMBED_FIRSTNORM_MAX_TOKENS), + m3_embedding_firstnorm, + _spec_rows("rows"), +) +# One request decoding one token on an MXFP8 layer input: QKV GEMV + attention + o_proj in one launch. +ATTN_MEGA = RowRange("attn_mega", frozenset({(1, 1)}), m3_attn_mega, _const((1, 1))) +ATTN_C1 = RowRange( + "attn_c1", frozenset({(ATTN_C1_QUERY_LEN, 1)}), m3_attn_c1, _const((ATTN_C1_QUERY_LEN, 1)) +) +FUSED_DECODE_ATTN = RowRange( + "fused_decode_attn", + _fused_attn_keys(), + m3_fused_decode_attn, + lambda spec: ((spec["qlen"], len(spec["seq_lens"])),), +) +SPARSE_DECODE_PRODUCER = RowRange( + "sparse_decode_producer", + frozenset({"per_row", "multi_query"}), + m3_sparse_decode, + _sparse_decode_producer, +) +# Every token count of a sparse layer on an NVFP4 KV cache at the TP4 heads (prefill and decode): the producer takes the +# geometry, not a row range. +PRODUCER = RowRange("m3_producer", frozenset({"nvfp4_tp4"}), m3_producer, _const("nvfp4_tp4")) +INDEX_SCORE = RowRange( + "m3_index_score", + frozenset({"early", "prefetch"}), + m3_index_score, + lambda spec: ("early" if spec["early"] else "prefetch",), +) +LONG_SELECT = RowRange( + "long_select", frozenset(LONG_SELECT_KEY_WIDTHS), m3_select_long, _long_select_widths +) +MC_ALLREDUCE = RowRange( + "mc_allreduce_norm", + _rows(1, min(DECODE_MAX_TOKENS, m3_mc_allreduce.max_rows())), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=0), +) +ATTN_PUSH = RowRange( + "attn_push", + _rows(1, ATTN_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +MOE_PUSH = RowRange( + "moe_push", + _rows(1, MOE_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +MC_FINALIZE = RowRange( + "mc_finalize_allreduce_norm", + _rows(MC_FINALIZE_MIN_TOKENS, min(DECODE_MAX_TOKENS, m3_mc_allreduce.max_rows())), + m3_mc_finalize_allreduce_norm, + _spec_rows("m"), +) +MOE_ENGINE = RowRange( + "m3_moe_engine", _rows(1, MOE_ENGINE_MAX_TOKENS), m3_moe_engine, _spec_rows("tokens") +) +MOE16_ENGINE = RowRange( + "m3_moe16_engine", + _rows(MOE16_MIN_TOKENS, MOE16_MAX_TOKENS), + m3_moe16_engine, + _spec_rows("tokens"), +) +ROUTER_GEMV = RowRange( + "m3_router_gemv", _rows(1, ROUTER_GEMV_MAX_TOKENS), m3_router_gemv, _spec_rows("m") +) +ROUTER_GEMV_FP4 = RowRange( + "m3_router_gemv_fp4", _rows(1, ROUTER_GEMV_MAX_TOKENS), m3_router_gemv_fp4, _spec_rows("m") +) +ROUTER_TC = RowRange( + "m3_router_tc", + _rows(ROUTER_GEMV_MAX_TOKENS + 1, ROUTER_TC_MAX_TOKENS), + m3_router_tc, + _spec_rows("m"), +) +# The GEMM paths are keyed (projection (N, K), rows): each projection runs its own split-K / ring / MMA +# configuration, so a row count is certified per projection. +MX_GEMV = RowRange( + "m3_mx_gemv", + frozenset((nk, m) for nk, rows in MX_GEMV_ROWS.items() for m in rows), + m3_mx_gemv, + lambda spec: ((spec["nk"], spec["m"]),), +) +MX_GEMV_BF16 = RowRange( + "m3_mx_gemv_bf16", + frozenset((nk, m) for nk, hi in MX_GEMV_BF16_MAX_TOKENS.items() for m in _rows(1, hi)), + m3_mx_gemv_bf16, + lambda spec: ((spec["nk"], spec["m"]),), +) +LOW_M_GEMM = RowRange( + "mxfp8_low_m_gemm", + frozenset((nk, m) for nk, hi in LOW_M_GEMM_MAX_TOKENS.items() for m in _rows(1, hi)), + mxfp8_low_m_gemm, + lambda spec: (((spec["n"], spec["k"]), spec["m"]),), +) +# (draft length, requests) of a greedy verify step. +SHARD_ACCEPT = RowRange( + "m3_shard_accept", + frozenset((SHARD_ACCEPT_DRAFT_LEN, n) for n in range(1, SHARD_ACCEPT_MAX_REQUESTS + 1)), + m3_shard_accept, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) + +# Requests of a decode step whose dense-layer attention runs at a capped KV split. +DENSE_DECODE_SPLIT = RowRange( + "m3_dense_decode", + frozenset(DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH), + m3_dense_decode, + lambda spec: (len(spec["kv_lens"]),) if spec["capped"] else (), +) + +# Verify rows of a fused-verify step whose target logits come from the coded head. +TARGET_HEAD = RowRange( + "m3_target_head", + _rows(1, TARGET_HEAD_MAX_ROWS), + m3_target_head, + _spec_rows("m"), +) + +# (draft length, requests) of a fused draft loop, and the rows of its passes' draft input: the first pass's +# requests x (draft length + 1), one per request on the later passes. +DRAFT_STEP = RowRange( + "m3_draft_step", + frozenset((DRAFT_STEP_DRAFT_LEN, n) for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1)), + m3_draft_step, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) +DRAFT_INPUT = RowRange( + "m3_draft_input", + frozenset( + rows + for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1) + for rows in (n, n * (DRAFT_STEP_DRAFT_LEN + 1)) + ), + m3_draft_input, + _spec_rows("m"), +) + +FAST_PATHS: Tuple[RowRange, ...] = ( + EMBED_FIRSTNORM, + ATTN_MEGA, + ATTN_C1, + FUSED_DECODE_ATTN, + DENSE_DECODE_SPLIT, + SPARSE_DECODE_PRODUCER, + PRODUCER, + INDEX_SCORE, + LONG_SELECT, + MC_ALLREDUCE, + ATTN_PUSH, + MOE_PUSH, + MC_FINALIZE, + MOE_ENGINE, + MOE16_ENGINE, + ROUTER_GEMV, + ROUTER_GEMV_FP4, + ROUTER_TC, + MX_GEMV, + MX_GEMV_BF16, + LOW_M_GEMM, + SHARD_ACCEPT, + TARGET_HEAD, + DRAFT_STEP, + DRAFT_INPUT, +) + +# -------------------------------------------------------------------------------------------------------------------- +# Selection: pure functions of the step shape +# -------------------------------------------------------------------------------------------------------------------- + + +def plan_for(num_tokens: int) -> DecodePlan: + for plan in DECODE_PLANS: + if num_tokens <= plan.max_tokens: + return plan + raise AssertionError(f"no decode plan covers {num_tokens} tokens") + + +def is_decode_step(num_tokens: int) -> bool: + """Whether a pure generation step of ``num_tokens`` rows runs the decode kernels (else generic layers).""" + return num_tokens <= min(FUSED_DECODE_MAX_TOKENS, DECODE_MAX_TOKENS) + + +def moe_path(num_tokens: int) -> Optional[str]: + """The MoE kernel of a decode step: an engine, the unfinalized MoE + finalize all-reduce, or None (generic).""" + if MOE_ENGINE.covers(num_tokens): + return MOE_ENGINE.path + if MOE16_ENGINE.covers(num_tokens): + return MOE16_ENGINE.path + if MC_FINALIZE.covers(num_tokens): + return MC_FINALIZE.path + return None + + +def router_path(num_tokens: int) -> Optional[str]: + """The router kernel when the MoE runs unfused: the fp32 GEMV, the tensor-core router, or None (built-in gate).""" + if ROUTER_GEMV.covers(num_tokens): + return ROUTER_GEMV.path + if ROUTER_TC.covers(num_tokens): + return ROUTER_TC.path + return None + + +MX_GEMV_PATH = MX_GEMV.path +MX_GEMV_BF16_PATH = MX_GEMV_BF16.path +LOW_M_GEMM_PATH = LOW_M_GEMM.path +# FlashInfer's CuTe DSL MXFP8 GEMM on the producer's MXFP8 activation (rows the GEMV does not serve). +PREQUANT_GEMM_PATH = "flashinfer_mxfp8_gemm" +# The MXFP8 quantize (tuned backend) and FlashInfer's CuTe DSL GEMM: every row the decode kernels do not serve. +GENERIC_GEMM_PATH = "mxfp8_gemm" + + +def linear_path(n: int, k: int, num_tokens: int, prequant: bool, low_m_gemm: bool) -> str: + """The decode-graph MXFP8 GEMM of an [N, K] projection on ``num_tokens`` rows. + + ``prequant``: the producer published the activation in MXFP8. ``low_m_gemm``: the projection is marked for the + fused low-M GEMM. + """ + if prequant: + if num_tokens in MX_GEMV_ROWS.get((n, k), ()): + return MX_GEMV_PATH + return PREQUANT_GEMM_PATH + if num_tokens <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0): + return MX_GEMV_BF16_PATH + if low_m_gemm and num_tokens <= LOW_M_GEMM_MAX_TOKENS.get((n, k), 0): + return LOW_M_GEMM_PATH + return GENERIC_GEMM_PATH + + +# -------------------------------------------------------------------------------------------------------------------- +# Fallback policy +# -------------------------------------------------------------------------------------------------------------------- + + +class FastPathUnavailable(RuntimeError): + """A covered decode shape could not take its fast path under ``modeling_v2='require'``.""" + + +_lock = threading.Lock() +_require = [False] +_logged: set = set() + + +def set_require(require: bool) -> None: + """Whether a miss raises (``modeling_v2='require'``); set by the target at construction.""" + _require[0] = bool(require) + + +def require_mode() -> bool: + return _require[0] + + +def _log_once(level: Callable[[str], None], path: str, reason: str, message: str) -> None: + key = (path, reason) + with _lock: + if key in _logged: + return + _logged.add(key) + level(message) + + +def missed(path: str, rows: Optional[int], reason: str) -> None: + """``path`` covers this step but cannot run: raise under require, else log once; the caller falls back.""" + at = "" if rows is None else f" at {rows} rows" + if _require[0]: + raise FastPathUnavailable( + f"modeling_v2='require': the MiniMax-M3 decode path {path!r} covers this step{at} but cannot run: " + f"{reason}" + ) + _log_once( + logger.warning, + path, + reason, + f"[M3 dispatch] {path}{at}: {reason}; falling back (logged once per reason)", + ) + + +def capture_missed(path: str, rows: Optional[int], what: str) -> None: + """``what`` is built on eager steps but a CUDA-graph capture found it missing (see ``missed``).""" + missed(path, rows, f"{what} is not built and a CUDA graph is being captured") + + +def uncovered(path: str, reason: str) -> None: + """The step is outside ``path``'s covered shapes: log once per (path, reason); the generic path runs.""" + _log_once(logger.info, path, reason, f"[M3 dispatch] {path} not taken: {reason} (logged once)") + + +def reset_logged() -> None: + """Forget which (path, reason) pairs were logged.""" + with _lock: + _logged.clear() + + +# -------------------------------------------------------------------------------------------------------------------- +# Target constants +# -------------------------------------------------------------------------------------------------------------------- + +# The generic boundary takes the low-latency multicast all-reduce + residual + RMSNorm up to this many tokens; with +# the shared + routed add folded in, up to its own maximum (m3_mc_allreduce.max_rows()). +MC_AR_MAX_TOKENS = 8 # capacity: fixed: measured crossover vs the one-shot all-reduce +# The shared expert folds into the routed engine from one token (dedicated shared-expert CTAs). +MOE_FOLD_MIN_TOKENS = 1 # capacity: fixed: a row lower bound +# SMs left to the concurrent kernels while a persistent routed engine runs. +MOE_ENGINE_RESERVED_SMS = 24 # capacity: fixed: an SM split, not a size +# Id of the target's final hidden state in Eagle3's ``layers_to_capture``. +FINAL_HIDDEN_LAYER_ID = -1 # capacity: fixed: an Eagle3 layer id, not a size +# The trtllm-gen MoE runner's routing kind for MiniMax-M3 (sigmoid + correction bias, renormalized, scaled) and its +# gated activation (SwiGLU with the per-expert alpha / beta / clamp operands). +ROUTING_MINIMAX2 = int(RoutingMethodType.MiniMax2) +ACT_SWIGLU = int(ActType_TrtllmGen.SwiGlu) + +# -------------------------------------------------------------------------------------------------------------------- +# The runtime-value dump (TRTLLM_MODELING_V2_VALUE_DUMP=1): the metadata of every fast-path and entry decision a step +# takes, recorded on eager steps and captures (replay runs no Python), printed once at exit. +# -------------------------------------------------------------------------------------------------------------------- + +_VALUE_DUMP = os.environ.get("TRTLLM_MODELING_V2_VALUE_DUMP") == "1" + +_VALUES: set = set() + + +def _record(*meta: Hashable) -> None: + if _VALUE_DUMP: + _VALUES.add(meta) + + +def _print_values() -> None: + if _VALUES: + rank = os.environ.get("OMPI_COMM_WORLD_RANK", os.environ.get("RANK", "0")) + for meta in sorted(_VALUES, key=repr): + print(f"[modeling_v2 value dump] rank {rank}: {meta}", flush=True) + + +if _VALUE_DUMP: + atexit.register(_print_values) + +# -------------------------------------------------------------------------------------------------------------------- +# Configuration: the text tower and the runtime the target assumes +# -------------------------------------------------------------------------------------------------------------------- + + +def _is_vl_config(pretrained_config: PretrainedConfig) -> bool: + model_type = getattr(pretrained_config, "model_type", None) + if model_type == "minimax_m3_vl": + return True + if model_type == "minimax_m3": + return False + return any( + "ForConditionalGeneration" in a + for a in (getattr(pretrained_config, "architectures", None) or []) + ) + + +def _text_model_config(model_config: ModelConfig) -> ModelConfig: + """The model config with the language model's ``text_config`` as its pretrained config. + + The checkpoint declares the vision-language wrapper; its ``text_config`` arrives as a plain dict (no typed + sub-config is registered), so it is materialized into a ``PretrainedConfig`` with the wrapper's dtype and + embedding tie carried over. A text config is returned unchanged. + """ + if not _is_vl_config(model_config.pretrained_config): + return model_config + cfg = copy.deepcopy(model_config) + text = getattr(cfg.pretrained_config, "text_config", None) + if text is None: + raise ValueError("MiniMax-M3 VL config has no `text_config`") + if isinstance(text, dict): + out = PretrainedConfig() + for k, v in text.items(): + setattr(out, k, v) + text = out + if ( + getattr(text, "torch_dtype", None) is None + and getattr(cfg.pretrained_config, "torch_dtype", None) is not None + ): + text.torch_dtype = cfg.pretrained_config.torch_dtype + if ( + getattr(text, "tie_word_embeddings", None) is None + and getattr(cfg.pretrained_config, "tie_word_embeddings", None) is not None + ): + text.tie_word_embeddings = cfg.pretrained_config.tie_word_embeddings + return dataclasses.replace(cfg, pretrained_config=text) + + +def _validate_runtime(model_config: ModelConfig) -> None: + """The runtime this target is written for. It is set by engine arguments rather than the checkpoint, so routing + sees only its KV cache algorithm; this check at construction refuses every other runtime. + + The MiniMax-M3 MSA backend (it owns the step metadata and the index-K cache), the fused five-way QKV/index + projection or the separate QKV and index projections, an FP8 indexer and an FP8 or NVFP4 main KV cache. + """ + sparse = model_config.sparse_attention_config + if ( + sparse is None + or getattr(sparse, "algorithm", None) != "minimax_m3" + or sparse.implementation != "msa" + ): + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.algorithm='minimax_m3' with implementation='msa'" + ) + if getattr(sparse, "indexer_kv_dtype", None) != "fp8": + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.indexer_kv_dtype='fp8'" + ) + mode = model_config.quant_config.quant_mode + if mode is None or not (mode.has_fp8_kv_cache() or mode.has_fp4_kv_cache()): + raise ValueError("this MiniMax-M3 target requires an FP8 or NVFP4 KV cache") + + +def _sparse_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = cfg.sparse_attention_config["sparse_attention_freq"] + return [i for i, f in enumerate(freq) if int(f) != 0] + + +def _moe_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = getattr(cfg, "moe_layer_freq", None) + if freq is None: + return list(range(cfg.num_hidden_layers)) + return [i for i, f in enumerate(freq) if int(f) != 0] + + +class _Mxfp8GemmState(nn.Module): + """The MXFP8 GEMM backend state the engine's warmup drives. + + The engine finds MXFP8 projections by a module's ``quant_method`` and, for every one, opts it into native GEMM + tuning and (on decode CUDA graphs) the auto-dispatched FlashInfer GEMM, then marks each tuned. Every projection of + this target dispatches identically, so one method object carries that state for all of them; the forward reads it + at each call (``_mx_mode``) and passes the resolved backend to the GEMM entries. + """ + + def __init__(self) -> None: + super().__init__() + self.quant_method = MXFP8LinearMethod() + + +@dataclass(frozen=True) +class MxMode: + """The MXFP8 GEMM dispatch of one call, resolved from ``_Mxfp8GemmState`` and the engine's contexts.""" + + decode_graph: bool # inside a decode CUDA-graph warmup or capture + autotune: bool # inside the engine's FlashInfer MXFP8 tuning pass + tuned_quantize_gemm: ( + bool # per-bucket tuned quantize + GEMM (decode graphs, tune_decode_graph_backends) + ) + flashinfer: bool # FlashInfer's MXFP8 GEMM on the quantized activation + native_tuning: ( + bool # the native GEMM through the autotuner (warmup, before its tactics are registered) + ) + + +def _mx_mode(state: _Mxfp8GemmState) -> MxMode: + method = state.quant_method + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + autotune = flashinfer_mxfp8_autotune_active() + return MxMode( + decode_graph=decode_graph, + autotune=autotune, + tuned_quantize_gemm=method.tune_decode_graph_backends and decode_graph, + flashinfer=method.backend == "flashinfer" + or ( + method.backend == "auto" + and (autotune or (method._flashinfer_autotuned and decode_graph)) + ), + native_tuning=method.needs_native_autotune, + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# Projections: the MXFP8 GEMM dispatch (``linear_path``) and its hand-offs +# -------------------------------------------------------------------------------------------------------------------- + + +def _mxfp8_linear( + core, x: torch.Tensor, w: torch.Tensor, s: torch.Tensor, mode: MxMode +) -> torch.Tensor: + """One MXFP8 projection on the engine-tuned path: MXFP8 quantize + GEMM, backend per ``mode``.""" + _record( + "m3_mxfp8_linear", + tuple(w.shape), + mode.tuned_quantize_gemm, + mode.flashinfer, + mode.native_tuning, + ) + return m3_mxfp8_linear( + x.contiguous(), + w, + s, + core.mx_alpha, + torch.bfloat16, + mode.tuned_quantize_gemm, + mode.autotune, + mode.flashinfer, + mode.native_tuning, + ) + + +def _gemv_variants(n: int, k: int, push: Optional[str], world: int): + """Every (rows, bf16 activation, push (world, bucket tokens, mode)) call the decode dispatch routes to an [N, K] + projection's GEMV: an MXFP8 activation on the rows ``MX_GEMV_ROWS`` gives it, a bf16 one up to + ``MX_GEMV_BF16_MAX_TOKENS``; each pushing when the projection's boundary push covers the rows.""" + push_rows = {"attn": ATTN_PUSH.keys, "moe": MOE_PUSH.keys}.get(push, frozenset()) + variants = [] + for rows in range(1, DECODE_MAX_TOKENS + 1): + for x_bf16, served in ( + (False, rows in MX_GEMV_ROWS.get((n, k), ())), + (True, rows <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0)), + ): + if not served: + continue + variants.append((rows, x_bf16, None)) + if rows in push_rows: + bucket = next(b for b in m3_mc_allreduce.BUCKETS if rows <= b) + variants.append((rows, x_bf16, (world, bucket, AR_PUSH_MODE))) + return variants + + +def _low_m_gemm_supports(x: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor) -> bool: + """Whether the fused MXFP8 low-M kernel handles these operands.""" + return ( + x.dim() == 2 + and x.dtype in (torch.bfloat16, torch.float16) + and x.stride(-1) == 1 + and weight.dtype == torch.float8_e4m3fn + and weight.is_contiguous() + and weight_scale.dtype == torch.uint8 + and x.shape[1] == weight.shape[1] + and weight.shape[1] % 256 == 0 + ) + + +def _uncompiled_note() -> str: + key = mx_gemv.last_uncompiled + return "" if key is None else f" (its kernel {key} was not compiled before warmup)" + + +def _linear( + core, x: torch.Tensor, wk: str, sk: str, low_m_gemm: bool, push: Optional[tuple] = None +) -> Tuple[torch.Tensor, bool]: + """One MXFP8 projection ``x @ w[wk]^T``; returns (output, whether its GEMV pushed into ``push``). + + Outside decode CUDA graphs: the engine-tuned MXFP8 quantize + GEMM. Inside them the projection's (N, K) and the + step's rows pick one path (``linear_path``): + + 1. an activation its producer already quantized to MXFP8 (``m3_prequant.SLOT``, taken here): the tcgen05 decode + GEMV on the rows it serves, else the decode-graph MXFP8 GEMM on that activation (AutoTuner-selected backend + per token bucket); + 2. a bf16 activation on the rows the GEMV serves quantizing in its prologue; + 3. ``low_m_gemm`` projections, on its rows: the fused low-M MXFP8 GEMM; + 4. otherwise the engine-tuned MXFP8 quantize + GEMM. + + A kernel declining rows its path covers is a miss; the chain then continues down the list. ``push`` (an + ``m3_mx_gemv`` ``ar`` tuple) is the boundary all-reduce exchange the GEMV writes its rows into. + """ + w, s = core.w[wk], core.w[sk] + x = x.contiguous() + mode = _mx_mode(core.mxfp8) + if not mode.decode_graph: + prequant = _CTXNORMQ_MX.take(x) if CTXNORMQ and not mode.tuned_quantize_gemm else None + if prequant is not None: + return _ctxnormq_mxfp8_gemm(core, prequant, w, s, mode), False + return _mxfp8_linear(core, x, w, s, mode), False + use_pdl = get_env_enable_pdl() + rows = x.shape[0] + n, k = w.shape + prequant = m3_prequant.SLOT.take(x) + if prequant is None and x.dtype != torch.bfloat16: + path = GENERIC_GEMM_PATH + else: + path = linear_path(n, k, rows, prequant is not None, low_m_gemm) + _record("linear", (n, k), rows, path, push is not None) + output = None + if prequant is not None: + act, act_scale = prequant + if path == MX_GEMV_PATH: + output = m3_mx_gemv(act, act_scale, w, s, True, None, use_pdl, push) + if output is not None: + return output, push is not None and mx_gemv.last_pushed + missed( + path, + rows, + f"the GEMV declined the [{n}, {k}] projection's MXFP8 activation" + + _uncompiled_note(), + ) + if path == PREQUANT_GEMM_PATH: + uncovered(MX_GEMV_PATH, f"the [{n}, {k}] projection at rows the GEMV does not serve") + if rows in SHDOWN_PIN_ROWS.get((n, k), ()): + return _shdown_pinned(act, act_scale, w, s, "mxfp8"), False + return m3_mxfp8_gemm(act, act_scale, w, s, torch.bfloat16, mode.autotune), False + if path == MX_GEMV_BF16_PATH: + output = m3_mx_gemv_bf16(x, w, s, True, None, use_pdl, push) + if output is not None: + return output, push is not None and mx_gemv.last_pushed + missed( + path, rows, f"the GEMV declined the [{n}, {k}] projection's input" + _uncompiled_note() + ) + if ( + path in (MX_GEMV_BF16_PATH, LOW_M_GEMM_PATH) + and low_m_gemm + and rows <= LOW_M_GEMM_MAX_TOKENS.get((n, k), 0) + and _low_m_gemm_supports(x, w, s) + ): + output = _low_m_gemm(x, w, s, torch.bfloat16, use_pdl=use_pdl) + if output is not None: + return output, False + if path == LOW_M_GEMM_PATH: + missed(path, rows, f"the GEMM declined the [{n}, {k}] projection's input") + if path == GENERIC_GEMM_PATH: + uncovered( + MX_GEMV_BF16_PATH, + f"the [{n}, {k}] projection's bf16 input beyond the decode GEMV's rows", + ) + if x.dtype == torch.bfloat16 and rows in SHDOWN_PIN_ROWS.get((n, k), ()): + act, act_scale = torch.ops.trtllm.mxfp8_quantize(x, True) + return _shdown_pinned(act.view(rows, k), act_scale, w, s, "bf16"), False + if QPIN and mode.tuned_quantize_gemm and x.dtype == torch.bfloat16: + tactic = QPIN_TACTIC_EXCEPTIONS.get((rows, k), MXFP8QuantizeRunner.CUTE_DSL) + act, act_scale = m3_mxfp8_pinned_quantize(x, tactic) + if (rows, n, k) not in _QPIN_LOGGED: + _QPIN_LOGGED.add((rows, n, k)) + logger.info( + f"[M3 dispatch] QPIN: mxfp8 quantize tactic {tactic} rows={rows} [{n}, {k}]" + ) + return m3_mxfp8_gemm( + act.view(rows, k), act_scale, w, s, torch.bfloat16, mode.autotune + ), False + return _mxfp8_linear(core, x, w, s, mode), False + + +def _shdown_pinned(act, act_scale, w, s, source: str) -> torch.Tensor: + """The SHDOWN pin (SHDOWN_PIN_ROWS): FlashInfer's CuTe DSL MXFP8 GEMM, whatever the AutoTuner (or a persisted + tactic cache) holds for the shape. The shared expert's down projection reaches it with a bf16 activation (the + SwiGLU output), quantized here with the native MXFP8 quantize.""" + rows, (n, k) = act.shape[0], w.shape + if (rows, source) not in _SHDOWN_LOGGED: + _SHDOWN_LOGGED.add((rows, source)) + logger.info( + f"[M3 dispatch] SHDOWN pin: cute_dsl rows={rows} [{n}, {k}] ({source} activation)" + ) + return m3_mxfp8_gemm_cute_dsl(act, act_scale, w, s, torch.bfloat16) + + +# -------------------------------------------------------------------------------------------------------------------- +# Layer boundaries: all-reduce + residual + RMSNorm +# -------------------------------------------------------------------------------------------------------------------- + + +def _can_mc(x, folded: bool = False) -> bool: + """Whether the generic boundary takes the multicast all-reduce + residual + RMSNorm.""" + return ( + isinstance(x, torch.Tensor) + and x.dim() == 2 + and x.shape[0] <= (m3_mc_allreduce.max_rows() if folded else MC_AR_MAX_TOKENS) + and x.dtype == torch.bfloat16 + and x.is_contiguous() + ) + + +def _mc_allreduce_norm(core, x, x2, residual, norm_w, *, quant=False, prepushed=0): + """The multicast all-reduce + optional second addend + residual + RMSNorm (``m3_mc_allreduce_norm``), or None + without a workspace or for operands the kernel does not take (the same on every rank). With ``quant`` the normed + rows are also published, MXFP8-quantized, to ``m3_prequant.SLOT`` for the next MXFP8 projection.""" + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + if workspace is None: + return None + _record("m3_mc_allreduce_norm", x.shape[0], x2 is not None, quant, prepushed) + return m3_mc_allreduce.m3_mc_allreduce_norm( + x, + residual.contiguous(), + norm_w, + core.rms_eps, + workspace, + use_pdl=get_env_enable_pdl(), + x2=x2, + quant=quant, + prepushed=prepushed, + ) + + +# The CTXNORMQ hand-offs: the boundary's MXFP8 activation for the first prefill MXFP8 projection reading the normed +# rows, and its NVFP4 form for the routed experts. Separate from ``m3_prequant.SLOT`` so only a CTXNORMQ boundary +# feeds the prefill path. +_CTXNORMQ_MX = m3_prequant.PrequantSlot() +_CTXNORMQ_FP4 = m3_prequant.PrequantSlot() +_CTXNORMQ_STATE = {} + + +def _ctxnormq_op(): + """``torch.ops.trtllm.minimax_m31_ctx_residual_rmsnorm_mxfp8``, or None. A build without the op in its library + loads it from ``tensorrt_llm/libs/libm31_ctxnormq.so`` when that file exists (an overlay); otherwise the boundary + keeps the fused all-reduce. Decided once per process.""" + if "op" not in _CTXNORMQ_STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_ctx_residual_rmsnorm_mxfp8", None) + if op is None: + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_ctxnormq.so") + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_ctx_residual_rmsnorm_mxfp8", None) + _CTXNORMQ_STATE["op"] = op + logger.info( + f"[M3 dispatch] CTXNORMQ prefill boundary norm + quantize: " + f"{'engaged' if op is not None else 'not taken: the op is not built'} (rows >= {CTXNORMQ_MIN_ROWS})" + ) + return _CTXNORMQ_STATE["op"] + + +def _ctxnormq_allreduce_norm(core, x, residual, norm_w, fp4_scale): + """A prefill boundary as the plain all-reduce + the fused residual add, RMSNorm and quantize, or None (switched + off, fewer than ``CTXNORMQ_MIN_ROWS`` rows, op not built). Returns (normed, updated) like the fused all-reduce; + the MXFP8 activation (and with ``fp4_scale`` the NVFP4 one) wait in the CTXNORMQ slots for their consumers.""" + if not CTXNORMQ or x.dim() != 2 or x.shape[0] < CTXNORMQ_MIN_ROWS: + return None + op = _ctxnormq_op() + if op is None: + return None + reduced = m3_allreduce(x, core.allreduce, None) + _record("m31_ctxnormq", x.shape[0], fp4_scale is not None) + normed, xq, xsf, xfp4, xfp4_sf = op( + reduced, residual.contiguous(), norm_w, core.rms_eps, True, fp4_scale, get_env_enable_pdl() + ) + _CTXNORMQ_MX.publish(normed, xq, xsf) + if fp4_scale is not None: + _CTXNORMQ_FP4.publish(normed, xfp4, xfp4_sf) + else: + _CTXNORMQ_FP4.clear() + return normed, reduced + + +def _ctxnormq_mxfp8_gemm(core, prequant, w, s, mode): + """The GEMM ``m3_mxfp8_linear`` runs after its own quantize, on the CTXNORMQ activation (bit-identical operands).""" + act, act_scale = prequant + _record("m31_ctxnormq_gemm", tuple(w.shape), act.shape[0], mode.flashinfer, mode.native_tuning) + if mode.flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8(act, act_scale, w, s, torch.bfloat16) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if mode.native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, w, s, core.mx_alpha, torch.bfloat16) + + +_TSEL_STATE = {} + + +def _tsel_op(): + """``torch.ops.trtllm.minimax_m31_select_prefill_tile`` on sm_103, or None. A build without the op in its library + loads it from ``tensorrt_llm/libs/libm31_tsel.so`` when that file exists (an overlay). Decided once per process.""" + if "op" not in _TSEL_STATE: + op = None + if torch.cuda.get_device_capability() == (10, 3): + op = getattr(torch.ops.trtllm, "minimax_m31_select_prefill_tile", None) + if op is None: + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_tsel.so") + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_select_prefill_tile", None) + _TSEL_STATE["op"] = op + logger.info( + f"[M3 dispatch] TSEL prefill tile select: " + f"{'engaged' if op is not None else 'not taken: not sm_103 or the op is not built'} " + f"(rows >= {TSEL_MIN_ROWS}, blocks <= {TSEL_MAX_BLOCKS})" + ) + return _TSEL_STATE["op"] + + +def _tsel_select(max_score_kv, n_valid_blocks, init_blocks, local_blocks): + """The MSA indexer's selector hook: the head-major [total_q, num_kv_heads, 16] table of + ``select_blocks_from_maxscore``, or None (outside the tile select's rows / blocks range, or no op) to keep it.""" + if ( + torch.cuda.is_current_stream_capturing() # the host-side rows / blocks gate is not graph-stable + or max_score_kv.dtype != torch.float32 + or max_score_kv.dim() != 3 + or max_score_kv.stride(2) != 1 + or max_score_kv.shape[2] < TSEL_MIN_ROWS + or max_score_kv.shape[1] > TSEL_MAX_BLOCKS + ): + return None + op = _tsel_op() + if op is None: + return None + num_kv_heads, num_blocks, total_q = max_score_kv.shape + _record("m31_tsel", num_blocks, total_q >= 1024) + out = torch.empty( + (num_kv_heads, total_q, MSA_REQUIRED_TOPK), dtype=torch.int32, device=max_score_kv.device + ).permute(1, 0, 2) + nvb = n_valid_blocks.to(device=max_score_kv.device, dtype=torch.int32).contiguous() + op(max_score_kv, nvb, out, init_blocks, local_blocks) + if TSEL_CHECK: + ref = torch.ops.trtllm.minimax_m3_select_blocks( + max_score_kv, nvb, MSA_REQUIRED_TOPK, init_blocks, local_blocks, True + ) + if not torch.equal(out, ref): + raise AssertionError( + f"TSEL check: tile table mismatch against minimax_m3_select_blocks " + f"({int((out != ref).any(dim=-1).sum())} rows; {num_blocks} blocks, {total_q} rows)" + ) + _TSEL_STATE["checked"] = _TSEL_STATE.get("checked", 0) + 1 + if _TSEL_STATE["checked"] in (1, 100, 1000): + logger.info( + f"[M3 dispatch] TSEL check: {_TSEL_STATE['checked']} tile-selected steps equal" + ) + return out + + +def _tsel_register(backend) -> None: + """Register ``_tsel_select`` on the layer backend's MSA indexer (an indexer without the hook is left alone).""" + indexer = getattr(backend, "indexer", None) + if indexer is not None and getattr(indexer, "selector_hook", False) is None: + indexer.selector_hook = _tsel_select + + +_GQAMIX_STATE: dict = {} + + +def _gqamix_decode(*args, **kwargs) -> None: + """``minimax_m3_sparse_attn_decode`` for MsaDecodeFmha's sparse generation rows, through the target-local copy.""" + if not _GQAMIX_STATE.get("logged"): + _GQAMIX_STATE["logged"] = True + logger.info( + "[M3 dispatch] GQAMIX: mixed-step sparse decode through ops/gqamix_sparse_decode" + ) + _gqamix_kernel.minimax_m3_sparse_attn_decode(*args, **kwargs) + + +def _gqamix_register(backend) -> None: + """Register ``_gqamix_decode`` on the layer backend's MsaDecodeFmha (a library without the hook is left alone).""" + manager = getattr(backend, "_fmha_manager", None) + for fmha in getattr(manager, "fmha_libs", ()): + if getattr(fmha, "sparse_decode_hook", False) is None: + fmha.sparse_decode_hook = _gqamix_decode + + +def _native_allreduce_norm(core, x, residual, norm_w): + """The built-in fused all-reduce + residual + RMSNorm: with AR64AUTO, the strategy-MNNVL instance for the rows past + the multicast all-reduce's, up to the decode bound; the configured instance otherwise.""" + params = AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual, + norm_weight=norm_w, + eps=core.rms_eps, + trigger_completion_at_end=False, + ) + wide = getattr(core, "allreduce_wide", None) + rows = x.shape[0] + if wide is not None and m3_mc_allreduce.max_rows() < rows <= DECODE_MAX_TOKENS: + # The MNNVL module itself (not AllReduce(MNNVL)), so a declined shape returns None here instead of falling + # through to AllReduce's AUTO / autotuned path: the configured strategy runs then (fixed order either way). + out = m3_allreduce(x.contiguous(), wide, params) + if out is not None: + _record("ar64auto", rows) + _log_once( + logger.info, + "ar64auto", + "taken", + f"[M3.1] AR64AUTO taken: MNNVL all-reduce at {rows} rows", + ) + return out + _log_once( + logger.warning, + "ar64auto", + f"declined {rows}", + f"[M3.1] AR64AUTO declined: MNNVL refused {rows} rows; the configured all-reduce runs", + ) + _record("m3_allreduce", rows) + return m3_allreduce(x, core.allreduce, params) + + +# -------------------------------------------------------------------------------------------------------------------- +# The MoE: router, routed experts (trtllm-gen NVFP4), shared expert +# -------------------------------------------------------------------------------------------------------------------- + + +@dataclass +class _MoeScalars: + """The routed experts' kernel scalars of one layer, derived from the checkpoint's per-tensor scales once the + weights are real (``MiniMaxM3Tp4ModelingV2._derive_moe``).""" + + fc31_input_scale: torch.Tensor + fc2_input_scale: torch.Tensor + fc31_alpha: torch.Tensor + fc2_alpha: torch.Tensor + fc31_scale_c: torch.Tensor + act_alpha: torch.Tensor + act_beta: torch.Tensor + act_clamp: torch.Tensor + + +def _router_logits(core, i: int, h: torch.Tensor) -> torch.Tensor: + """The fp32 router logits of MoE layer ``i``: the fp32 GEMV at one row, the tensor-core router up to 48, else the + fp32 linear.""" + rows = h.shape[0] if h.dim() == 2 else 0 + path = router_path(rows) + w = core.w[f"l{i}_router"] + if path == ROUTER_GEMV.path: + logits = _router_gemv(h, w, use_pdl=get_env_enable_pdl()) + if logits is not None: + return logits + missed(path, rows, "the router GEMV declined the operands") + elif path == ROUTER_TC.path: + fused = _router_tc(h, w, use_pdl=get_env_enable_pdl()) + if fused is not None: + return fused[0] + missed(path, rows, "the tensor-core router declined the operands") + else: + uncovered(ROUTER_TC.path, f"more than {ROUTER_TC_MAX_TOKENS} rows") + return m3_fp32_linear(h, w) + + +def _router_fp4(core, i: int, h: torch.Tensor): + """Router logits and the routed experts' NVFP4 activation from one kernel, or None (then ``_router_logits``).""" + if h.dim() != 2 or h.shape[0] > ROUTER_TC_MAX_TOKENS: + return None + rows = h.shape[0] + path = router_path(rows) + if path == ROUTER_GEMV.path: + path = ROUTER_GEMV_FP4.path + scale = core.moe[i].fc31_input_scale + if path == ROUTER_TC.path: + fused = _router_tc(h, core.w[f"l{i}_router"], scale, use_pdl=get_env_enable_pdl()) + else: + fused = _router_gemv_fp4(h, core.w[f"l{i}_router"], scale, use_pdl=get_env_enable_pdl()) + if fused is None: + missed(path, rows, "the router declined the operands") + return fused + + +def _routed_experts(core, i: int, x, x_sf, router_logits, do_finalize: bool): + """The trtllm-gen NVFP4 MoE runner over this rank's tensor-parallel slice of every expert, routing in-kernel. + + ``x_sf`` None means ``x`` is the bf16 activation, quantized here with the experts' global input scale. Returns + the finalized rows, or the unfinalized ``[gemm2 out, expert scales, expanded->permuted index]``. + """ + moe = core.moe[i] + if x_sf is None: + x, x_sf = fp4_quantize(x, moe.fc31_input_scale, 16, False, False) + rows = x.shape[0] + x_sf = x_sf.view(rows, -1).flatten() + _record("fp4_block_scale_moe_runner", rows, do_finalize) + outs = fp4_block_scale_moe_runner( + router_logits, + core.w[f"l{i}_router_bias"], + x, + x_sf.view(torch.float8_e4m3fn), + core.w[f"l{i}_e_w31"], + core.w[f"l{i}_e_s31"].view(torch.float8_e4m3fn), + None, + moe.act_alpha, + moe.act_beta, + moe.act_clamp, + core.w[f"l{i}_e_w2"], + core.w[f"l{i}_e_s2"].view(torch.float8_e4m3fn), + None, + moe.fc31_scale_c, + moe.fc31_alpha, + moe.fc2_alpha, + core.num_experts, + core.top_k, + None, + None, + core.moe_inter, + 0, + core.num_experts, + core.routed_scaling_factor, + ROUTING_MINIMAX2, + do_finalize, + act_type=ACT_SWIGLU, + topk_weights=None, + topk_ids=None, + output=None, + tune_max_num_tokens=core.max_num_tokens, + use_dp=False, + ) + return outs[0] if do_finalize else outs + + +def _split_chunk(rows: int, num_chunks: int) -> List[int]: + """The MoE scheduler's even split: the first ``rows % num_chunks`` chunks take one more row.""" + base, extra = divmod(rows, num_chunks) + return [base + 1] * extra + [base] * (num_chunks - extra) + + +# -------------------------------------------------------------------------------------------------------------------- +# Attention +# -------------------------------------------------------------------------------------------------------------------- + + +def _nvfp4_layer(manager, i: int) -> bool: + return bool(getattr(manager, "is_nvfp4_layer", lambda _layer_idx: False)(i)) + + +def _producer_covered(core) -> bool: + """Whether ``PRODUCER`` covers this target's sparse layers: an NVFP4 KV cache at the TP4 heads (16 query, one KV + and one index head). A covered layer that cannot take ``m3_producer`` is a miss.""" + return core.main_kv_is_nvfp4 and (core.heads, core.kv_heads, core.index_heads) == (16, 1, 1) + + +def _producer_layout(kv_data, kv_scale) -> bool: + """Whether a sparse layer's NVFP4 pools are the layout ``m3_producer`` is built for: [pages, 2, 1, 128, 64] with + its scale pool [pages, 2, 1, 128, 8].""" + return ( + kv_data is not None + and kv_scale is not None + and kv_data.dim() == 5 + and tuple(kv_data.shape[1:]) == (2, 1, 128, 64) + and kv_data.stride()[2:] == (128 * 64, 64, 1) + and kv_scale.dim() == 5 + and tuple(kv_scale.shape[1:]) == (2, 1, 128, 8) + and kv_scale.stride()[2:] == (128 * 8, 8, 1) + and kv_scale.shape[0] == kv_data.shape[0] + ) + + +def _nvfp4_fused_write_layout(data: torch.Tensor, scales: torch.Tensor, num_heads: int) -> bool: + """Reject invalid HND pools; select the fused producer only for its packed layout.""" + if ( + data.dim() != 5 + or scales.dim() != 5 + or data.shape[0] <= 0 + or data.shape[0] != scales.shape[0] + or tuple(data.shape[1:]) != (2, num_heads, 128, 64) + or tuple(scales.shape[1:]) != (2, num_heads, 128, 8) + or data.dtype not in (torch.int8, torch.uint8) + or scales.dtype != torch.uint8 + ): + raise ValueError("MiniMax-M3 NVFP4 data and scale caches must have matching HND geometry") + for cache in (data, scales): + if ( + cache.stride(4) != 1 + or cache.stride(3) != cache.shape[4] + or cache.stride(2) < 128 * cache.stride(3) + or cache.stride(1) < num_heads * cache.stride(2) + or cache.stride(0) < 2 * cache.stride(1) + ): + raise ValueError( + "MiniMax-M3 NVFP4 cache rows must be packed and heads/planes/pages must not overlap" + ) + if scales.stride(2) != 128 * 8: + raise ValueError("MiniMax-M3 NVFP4 MSA scale cache must have packed head blocks") + return data.stride(2) == 128 * 64 and data.stride(0) % 2 == 0 and data.stride(1) % 2 == 0 + + +def _producer_miss(covered: bool, rows: Optional[int], reason: str) -> None: + if covered: + missed(PRODUCER.path, rows, reason) + return None + + +def _sparse_qkv_producer( + core, i: int, packed, position_ids, kv_cache_manager, msa_idx_k_cache, msa_out_cache_loc +): + """The horizontal producer of a sparse layer: Q / index-Q (FP8) out, K / V / index-K written into the paged + caches (NVFP4 or FP8 main cache), for every step type (token-major). An NVFP4 cache at the TP4 heads takes + ``m3_producer`` (``PRODUCER``); a covered layer that cannot is a miss. Returns (q, index_q), or None for caches + neither kernel takes. + + ``kv_cache_manager``, ``msa_idx_k_cache`` (the per-layer index-K cache getter) and ``msa_out_cache_loc`` are the + calling target's ``step_args`` entries.""" + covered = _producer_covered(core) + rows = int(packed.shape[0]) + if packed.dtype != torch.bfloat16 or position_ids is None: + return _producer_miss( + covered, + rows, + "the packed QKV dtype, position ids, head dims, norm or RoPE configuration", + ) + manager = kv_cache_manager + if manager is None: + return _producer_miss(covered, rows, "the step has no KV cache manager") + buffers = manager.get_buffers(i, kv_layout="HND") + index_k_cache = msa_idx_k_cache(i) + out_cache_loc = msa_out_cache_loc + layer_uses_nvfp4 = _nvfp4_layer(manager, i) + scale_buffers = ( + manager.get_block_scale_buffers(i, kv_layout="HND") if layer_uses_nvfp4 else None + ) + kv_quant_scale = core.inv_kv_scales[i] if layer_uses_nvfp4 else None + if layer_uses_nvfp4: + supported_main = ( + buffers is not None + and scale_buffers is not None + and _nvfp4_fused_write_layout(buffers, scale_buffers, core.kv_heads) + and buffers.is_cuda + and scale_buffers.is_cuda + and scale_buffers.device == buffers.device + and kv_quant_scale.is_cuda + and kv_quant_scale.dtype == torch.float32 + and kv_quant_scale.numel() >= 3 + and kv_quant_scale.is_contiguous() + ) + else: + supported_main = ( + buffers is not None + and buffers.is_cuda + and buffers.dtype == torch.float8_e4m3fn + and buffers.dim() == 5 + and tuple(buffers.shape[1:]) == (2, core.kv_heads, 128, 128) + and buffers.stride(4) == 1 + and buffers.stride(3) == 128 + and buffers.stride(2) == 128 * 128 + and buffers.stride(1) >= core.kv_heads * buffers.stride(2) + and buffers.stride(0) >= 2 * buffers.stride(1) + and buffers.stride(0) % 4 == 0 + and buffers.stride(1) % 4 == 0 + ) + supported_index = ( + index_k_cache.is_cuda + and index_k_cache.dtype == torch.float8_e4m3fn + and index_k_cache.dim() == 4 + and tuple(index_k_cache.shape[1:]) == (1, 128, 128) + and index_k_cache.stride(3) == 1 + and index_k_cache.stride(2) == 128 + and index_k_cache.stride(1) >= 128 * 128 + and index_k_cache.stride(0) >= index_k_cache.stride(1) + and index_k_cache.stride(0) % 4 == 0 + and index_k_cache.stride(1) % 4 == 0 + and buffers is not None + and index_k_cache.shape[0] == buffers.shape[0] + ) + cos_sin = core.rotary_cos_sin + supported_rope = ( + cos_sin.is_cuda + and cos_sin.dtype == torch.float32 + and cos_sin.is_contiguous() + and cos_sin.dim() == 3 + and tuple(cos_sin.shape[1:]) == (2, 32) + ) + if ( + not supported_main + or not supported_index + or not supported_rope + or out_cache_loc is None + or not out_cache_loc.is_cuda + or out_cache_loc.dtype != torch.int32 + or not out_cache_loc.is_contiguous() + or out_cache_loc.numel() < rows + ): + return _producer_miss( + covered and layer_uses_nvfp4, + rows, + "the layer's NVFP4 / index caches, RoPE table, KV quant scale or cache slots are not the layout the " + "producer is built for", + ) + w = core.w + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + if ( + layer_uses_nvfp4 + and (core.heads, core.kv_heads, core.index_heads) == (16, 1, 1) + and _producer_layout(buffers, scale_buffers) + ): + _record("m3_producer", rows) + q, index_q = m3_producer( + packed.contiguous(), + buffers.view(torch.uint8), + scale_buffers.view(torch.uint8), + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + core.heads, + get_env_enable_pdl(), + PRODUCER_EARLY_TRIGGER, + ) + else: + if layer_uses_nvfp4 and covered: + missed( + PRODUCER.path, + rows, + f"the NVFP4 pool {tuple(buffers.shape)} / scale pool {tuple(scale_buffers.shape)} is not the " + "producer's layout", + ) + _record("m3_qkv_insert", rows, layer_uses_nvfp4) + q, index_q = m3_qkv_insert( + packed.contiguous(), + buffers.view(torch.uint8) if layer_uses_nvfp4 else buffers, + scale_buffers, + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.heads, + core.kv_heads, + core.index_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + ) + return q.flatten(1), index_q.flatten(1) + + +def _qk_norm_rope(core, i: int, qkv: torch.Tensor, position_ids, out_fp8: bool) -> torch.Tensor: + """The per-head Gemma RMSNorm of Q and K and the partial NeoX RoPE of a dense layer's packed [Q | K | V], in one + kernel: in place, or an FP8 copy (Q / K normed and roped, V cast) when ``out_fp8``.""" + qkv = qkv.contiguous() + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + args = ( + qkv, + core.heads, + core.kv_heads, + core.kv_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_q_norm"], + core.w[f"l{i}_k_norm"], + core.rope_theta, + True, + pos, + 1.0, # factor: no YARN + 0.0, # low + 0.0, # high + 1.0, # attention_factor + True, # is_qk_norm + True, # use_gemma + False, # use_mrope + 0, + 0, + ) + _record("qk_norm_rope", qkv.shape[0], out_fp8) + if out_fp8: + return m3_qk_norm_rope_fp8(*args) + return fused_qk_norm_rope(*args) + + +def _attn_mega(core, i: int, hidden: torch.Tensor, args: dict, push: Optional[tuple]): + """(o_proj output, pushed) of the one-launch QKV + attention + o_proj kernel for one request decoding one token + whose layer input the preceding all-reduce handed over in MXFP8; None, with nothing consumed, otherwise.""" + if not ATTN_MEGA.covers((args["decode_query_len"], int(args["seq_lens"].shape[0]))): + return None + if not flashinfer_mxfp8_decode_graph_capture_active(): + return None + pending = m3_prequant.SLOT.pending + if ( + pending is None + or hidden.dim() != 2 + or pending[0] != hidden.data_ptr() + or pending[1] != 1 + or pending[2] != hidden.shape[1] + ): + # After a layer whose boundary does not quantize (a dense layer, an Eagle3 capture layer). + uncovered(ATTN_MEGA.path, "the layer input has no MXFP8 hand-off") + return None + w = core.w + if w[f"l{i}_qkv_w"].shape != (2560, 6144) or w[f"l{i}_o_w"].shape != (6144, 2048): + uncovered(ATTN_MEGA.path, "not the tp4 QKV / o_proj shapes") + return None + workspace = core.mega_ws.get(i) + if workspace is None: + # The first call JIT-builds the module; both only outside capture (the eager warmup steps). + if torch.cuda.is_current_stream_capturing(): + capture_missed(ATTN_MEGA.path, 1, "the attn_mega module and workspace") + return None + _attn_mega_kernel._load_module() + workspace = core.mega_ws[i] = _attn_mega_kernel.MegaWorkspace(hidden.device) + xq, xsf = m3_prequant.SLOT.take(hidden) + output = hidden.new_empty((1, core.heads * core.head_dim), dtype=torch.bfloat16) + y = hidden.new_empty((1, w[f"l{i}_o_w"].shape[0]), dtype=torch.bfloat16) + _record("m3_attn_mega", push is not None) + _attn_mega_kernel.m3_attn_mega( + xq, + xsf, + w[f"l{i}_qkv_w"], + w[f"l{i}_qkv_s"], + workspace, + args["kv_data"], + args["kv_scale"], + args["index_k_cache"], + args["out_cache_loc"], + args["position_ids"], + args["block_table"], + args["seq_lens"], + args["kv_quant_scale"], + args["k_global_scale"], + args["v_global_scale"], + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_k_norm"], + core.rotary_cos_sin, + w[f"l{i}_o_w"], + w[f"l{i}_o_s"], + eps=core.rms_eps, + sm_scale=core.sm_scale(i), + decode_query_len=1, + output=output, + y=y, + use_pdl=get_env_enable_pdl(), + ar=push, + ) + return y, push is not None + + +def _attn_c1(core, i: int, packed: torch.Tensor, args: dict, push: Optional[tuple]): + """(o_proj output, True) of the one-launch attention + o_proj kernel for one request verifying a draft, pushed + into the attention boundary's all-reduce; None, with nothing consumed, when the step or the layer does not + qualify.""" + qlen, num_tokens = args["decode_query_len"], args["num_tokens"] + if not ATTN_C1.covers((qlen, int(args["seq_lens"].shape[0]))): + return None + if not flashinfer_mxfp8_decode_graph_capture_active(): + return None + if push is None: + missed(ATTN_C1.path, num_tokens, "no armed attention-boundary push to write into") + return None + w = core.w + if w[f"l{i}_o_w"].shape != (6144, 2048): + uncovered(ATTN_C1.path, "not the tp4 o_proj shape") + return None + workspace = core.c1_ws.get(i) + if workspace is None: + # The first call JIT-builds the module; both only outside capture (the eager warmup steps). + if torch.cuda.is_current_stream_capturing(): + capture_missed(ATTN_C1.path, num_tokens, "the attn_c1 module and workspace") + return None + _attn_c1_kernel._load_module() + workspace = core.c1_ws[i] = _attn_c1_kernel.MegaWorkspace(packed.device) + output = packed.new_empty((num_tokens, core.heads * core.head_dim), dtype=torch.bfloat16) + y = packed.new_empty((num_tokens, w[f"l{i}_o_w"].shape[0]), dtype=torch.bfloat16) + _record("m3_attn_c1", num_tokens) + _attn_c1_kernel.m3_attn_c1( + packed.contiguous(), + workspace, + args["kv_data"], + args["kv_scale"], + args["index_k_cache"], + args["out_cache_loc"], + args["position_ids"], + args["block_table"], + args["seq_lens"], + args["kv_quant_scale"], + args["k_global_scale"], + args["v_global_scale"], + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_k_norm"], + core.rotary_cos_sin, + w[f"l{i}_o_w"], + w[f"l{i}_o_s"], + eps=core.rms_eps, + sm_scale=core.sm_scale(i), + decode_query_len=qlen, + output=output, + y=y, + use_pdl=get_env_enable_pdl(), + ar=push[:5], + ) + return y, True + + +def _fused_attn(core, i: int, packed: torch.Tensor, args: dict) -> torch.Tensor: + """The fused norm/RoPE/KV-insert/sparse-decode/merge kernel after the packed QKV projection; its MXFP8 copy of + the output goes to the o_proj through ``m3_prequant.SLOT``.""" + num_tokens = args["num_tokens"] + width = core.heads * core.head_dim + output = packed.new_empty((num_tokens, width), dtype=torch.bfloat16) + mxfp8 = None + if MERGE_MXFP8_MIN_TOKENS <= num_tokens <= MERGE_MXFP8.MAX_ROWS: + mxfp8 = MERGE_MXFP8.buffers(width, output.device) + w = core.w + _record("m3_fused_decode_attn", num_tokens, args["decode_query_len"]) + _fused_attn_kernel.m3_fused_decode_attn( + packed.contiguous(), + args["kv_data"], + args["kv_scale"], + args["index_k_cache"], + args["out_cache_loc"], + args["position_ids"], + args["block_table"], + args["seq_lens"], + args["kv_quant_scale"], + args["k_global_scale"], + args["v_global_scale"], + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_k_norm"], + core.rotary_cos_sin, + eps=core.rms_eps, + sm_scale=core.sm_scale(i), + decode_query_len=args["decode_query_len"], + output=output, + mxfp8_out=mxfp8, + use_pdl=get_env_enable_pdl(), + ) + if mxfp8 is not None: + m3_prequant.SLOT.publish(output, mxfp8[0][:num_tokens], mxfp8[1]) + return output + + +def _fp8_index_qk_norm_rope( + core, i: int, idx_qk: torch.Tensor, position_ids, msa_idx_k_cache, msa_out_cache_loc +) -> torch.Tensor: + """The index branch of an unfused sparse layer: index-Q in FP8 out, index-K into the FP8 index-K cache, normed and + roped in one kernel (the built-in ``_fused_fp8_index_qk_norm_rope``). ``msa_idx_k_cache`` / ``msa_out_cache_loc`` + are the calling target's ``step_args`` entries.""" + if position_ids is None or idx_qk.dtype != torch.bfloat16: + raise NotImplementedError( + "MiniMax-M3 fused FP8 indexer requires BF16 activations and position_ids." + ) + index_k_cache = msa_idx_k_cache(i) + if index_k_cache.dtype != torch.float8_e4m3fn: + raise ValueError( + "MiniMax-M3 fused FP8 indexer requires an E4M3 index-K cache, " + f"got {index_k_cache.dtype}." + ) + rows = int(idx_qk.shape[0]) + _record("m3_fp8_indexer_qk_norm_rope", rows) + return m3_fp8_indexer_qk_norm_rope( + idx_qk.contiguous(), + index_k_cache, + msa_out_cache_loc[:rows], + core.index_heads, + core.index_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_idx_q_norm"], + core.w[f"l{i}_idx_k_norm"], + core.rope_theta, + position_ids.reshape(-1).contiguous().to(torch.int32), + ).flatten(1) + + +# -------------------------------------------------------------------------------------------------------------------- +# Layer helpers both targets' bodies call: the Eagle3 capture hook and the norm after a layer +# -------------------------------------------------------------------------------------------------------------------- + + +def _plain_capture(spec_metadata, layer_id: int, hidden_states, residual) -> None: + """The in-tree Eagle3 hidden-state capture.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + +def _next_norm(core, i: int) -> torch.Tensor: + """The norm the boundary after layer ``i`` applies: the next layer's input norm, the final norm after the last.""" + return core.w[f"l{i + 1}_in_norm"] if i + 1 < core.num_layers else core.w["final_norm"] + + +# -------------------------------------------------------------------------------------------------------------------- +# The core +# -------------------------------------------------------------------------------------------------------------------- + +# Engine states (compiled kernel + workspace) per (device, token count[, shared fold]), shared by every layer: a step +# runs the layers one after another on one stream. +_ENGINE_STATES: Dict[tuple, object] = {} +_MOE16_STATES: Dict[tuple, object] = {} + + +# AR-DET: AllReduce AUTO's per-process timing autotuner made +# VR runs differ across server starts (0 / 888 same-node and 0 / 296 across node +# pairs with it off, bitwise to the modal autotuned output). The target turns it off for the whole process at init (AUTO +# then takes the C++ size / world heuristic, a pure function of the shape), unless TRTLLM_M31_ENABLE_AR_AUTOTUNE=1. An +# explicit TLLM_DISABLE_ALLREDUCE_AUTOTUNE is left as given. Read once, at import. +AR_AUTOTUNE_ENABLED = m31_enabled("AR_AUTOTUNE") # the m31_switches value contract (1/true/on/yes) + + +def _pin_allreduce_heuristic(rank: int) -> bool: + """Default the AllReduce autotune off for this process (see AR-DET); returns whether it is off. Logs one line.""" + if not AR_AUTOTUNE_ENABLED: + os.environ.setdefault("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "1") + off = os.environ.get("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "0") == "1" + logger.info( + f"M3.1 AR-DET: AllReduce autotune {'off (C++ size / world heuristic)' if off else 'on'} on rank {rank}" + ) + return off + + +class MiniMaxM3Tp4ModelingV2(DecoderModel): + """The MiniMax-M3 text model of this target: its weights, its runtime resources, and the per-step dispatch.""" + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + _validate_runtime(model_config) + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + _pin_allreduce_heuristic(mapping.tp_rank) + self.mapping = mapping + self.vocab_size = cfg.vocab_size + d = _weights.MODEL_WEIGHTS.dims(self) + self.num_layers = d.num_layers + self.hidden = d.hidden + self.heads = d.heads + self.kv_heads = d.kv_heads + self.head_dim = d.head_dim + self.index_heads = d.index_heads + self.index_dim = d.index_dim + self.q_size = d.heads * d.head_dim + self.kv_size = d.kv_heads * d.head_dim + self.moe_inter = d.moe_inter + self.num_experts = d.num_experts + self.top_k = int(cfg.num_experts_per_tok) + self.routed_scaling_factor = float(getattr(cfg, "routed_scaling_factor", 1.0)) + self.rms_eps = float(cfg.rms_norm_eps) + self.swiglu_alpha = float(getattr(cfg, "swiglu_alpha", 1.702)) + self.swiglu_beta = 1.0 # the (up + 1) offset of SwiGLU-OAI + self.swiglu_limit = float(getattr(cfg, "swiglu_limit", 7.0)) + self.max_num_tokens = model_config.max_num_tokens + self.moe_max_num_tokens = model_config.moe_max_num_tokens + self.sparse_layers = frozenset(_sparse_layer_ids(cfg)) + self.moe_layers = frozenset(_moe_layer_ids(cfg)) + mode = model_config.quant_config.quant_mode + self.main_kv_is_fp8 = bool(mode is not None and mode.has_fp8_kv_cache()) + self.main_kv_is_nvfp4 = bool(mode is not None and mode.has_fp4_kv_cache()) + # The sparse layers' QKV and index projections: one five-way packed projection, or the QKV projection and a + # separate MXFP8 index projection (fuse_qkv_index_projection=False) run on two streams. Packed unless the + # deployment sets the flag explicitly (``weights.fused_projection``). + self.fused_projection = _weights.fused_projection(model_config.sparse_attention_config) + + self.w = _weights.MODEL_WEIGHTS.declare(d) + + # The engine reads these two off the module tree: the MXFP8 GEMM state it tunes, and the opt-in to the + # automatic FlashInfer MXFP8 path on decode graphs. + self.mxfp8 = _Mxfp8GemmState() + self._use_flashinfer_mxfp8_decode_graph_default = True + + # The built-in fused all-reduce + residual + RMSNorm of the layer boundaries the multicast kernel does not take. + self.allreduce = AllReduce( + mapping=mapping, strategy=model_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + # AR64AUTO: the strategy-MNNVL instance of the generation boundaries past the multicast all-reduce's rows. + # The MNNVL module of an AllReduce(MNNVL) (kept alive by self._allreduce_wide_owner); None leaves every + # boundary on the configured instance. + self.allreduce_wide = None + self._allreduce_wide_owner = None + if AR64AUTO and DECODE_MAX_TOKENS > m3_mc_allreduce.max_rows(): + self._allreduce_wide_owner = AllReduce( + mapping=mapping, strategy=AllReduceStrategy.MNNVL, dtype=cfg.torch_dtype + ) + self.allreduce_wide = getattr(self._allreduce_wide_owner, "mnnvl_allreduce", None) + if self.allreduce_wide is not None: + logger.info( + f"[M3.1] AR64AUTO: boundaries of {m3_mc_allreduce.max_rows() + 1}..{DECODE_MAX_TOKENS} rows take " + f"the strategy-MNNVL all-reduce (configured: {model_config.allreduce_strategy})" + ) + else: + logger.warning( + "[M3.1] AR64AUTO: MNNVL unavailable on this mapping; every boundary keeps the configured all-reduce" + ) + + # Streams: the shared / routed expert overlap and, when the MoE chunks its input, the chunking overlap. + self.aux_stream_moe = torch.cuda.Stream() + chunked = self.moe_max_num_tokens < model_config.max_num_tokens * mapping.dp_size + self.aux_stream_moe_chunk = torch.cuda.Stream() if chunked else None + self.moe_events = {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers} + self.moe_chunk_events = { + i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers + } + # The unfused sparse layers' index branch overlaps the QKV branch on its own stream. + self.aux_stream_attn = None if self.fused_projection else torch.cuda.Stream() + self.attn_events = ( + {} + if self.fused_projection + else {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.sparse_layers} + ) + + # Partial NeoX RoPE (rotary_dim of head_dim): the cos/sin table every attention kernel reads. + rope = RopeParams.from_config(cfg) + self.rope_theta = rope.theta + self.rotary_dim = int(rope.dim) + self.rotary_cos_sin = rope.create_rope_const_params(interleave=False)[1].reshape( + rope.max_positions, 2, -1 + ) + + # One MSA attention backend per layer (dense and sparse; the step metadata is the backend's). + quant_config = model_config.get_quant_config() + self.attn: List[MiniMaxM3MsaSparseAttention] = [] + for i in range(self.num_layers): + backend = create_attention( + model_config.attn_backend, + i, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=None, + quant_config=quant_config, + skip_create_weights_in_init=model_config.skip_create_weights_in_init, + q_scaling=1.0, + sparse_params=model_config.sparse_attention_config.to_sparse_params( + pretrained_config=cfg, layer_idx=i + ), + ) + backend.update_quant_config(quant_config) + if not isinstance(backend, MiniMaxM3MsaSparseAttention): + raise ValueError( + f"layer {i}: expected the MiniMax-M3 MSA backend, got {type(backend).__name__}" + ) + if i not in self.sparse_layers: + # The KV split trtllm-gen dense decode runs at, per request count. + backend.dense_decode_max_seq_len_by_batch = DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH + self.attn.append(backend) + + # Built once the weights are real (``post_load_weights``): the KV (Q, K, V) scales, the MoE scalars, the + # decode engines and workspaces, the two targets. + self.kv_scales: Dict[int, torch.Tensor] = {} + self.inv_kv_scales: Dict[int, torch.Tensor] = {} + self.moe: Dict[int, _MoeScalars] = {} + self.mx_alpha: Optional[torch.Tensor] = None + self.moe_engines: Dict[int, Dict[Tuple[int, bool, bool], Optional[object]]] = {} + self.moe16_engines: Dict[int, Dict[int, Optional[object]]] = {} + self.embed_owners: Dict[tuple, DecodeEmbedding] = {} + self.mega_ws: Dict[int, object] = {} + self.c1_ws: Dict[int, object] = {} + self._built_for: Optional[tuple] = None + self._prefill: Optional[PrefillTarget] = None + self._decode: Optional[DecodeTarget] = None + + def sm_scale(self, i: int) -> float: + return (self.head_dim**-0.5) / float(self.attn[i].q_scaling) + + # ---------------------------------------------------------------------------------------------- after the load + + def post_load_weights(self) -> None: + """Derive the kernels' state from the loaded weights, then build both targets and every decode kernel. + + Idempotent per set of weight storages: the loader's post-load walk reaches this module directly and through + the shell, and a second call on the same storages does nothing. + """ + identity = ( + self.w["embed"].data_ptr(), + self.w["l0_in_norm"].data_ptr(), + self.w["final_norm"].data_ptr(), + ) + if self._built_for == identity: + return + device = self.w["embed"].device + derived = nn.Module() + derived.register_buffer( + "mx_alpha", torch.ones(1, dtype=torch.float32, device=device), persistent=False + ) + for i in range(self.num_layers): + # No KV calibration in the checkpoint: unit (Q, K, V) scales and their reciprocals. + derived.register_buffer( + f"l{i}_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + derived.register_buffer( + f"l{i}_inv_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + for i in sorted(self.moe_layers): + for name, value in dataclasses.asdict(self._derive_moe(i)).items(): + derived.register_buffer(f"l{i}_{name}", value, persistent=False) + self.derived = derived + self.mx_alpha = derived.mx_alpha + self.kv_scales = {i: getattr(derived, f"l{i}_kv_scales") for i in range(self.num_layers)} + self.inv_kv_scales = { + i: getattr(derived, f"l{i}_inv_kv_scales") for i in range(self.num_layers) + } + self.moe = { + i: _MoeScalars( + **{ + f.name: getattr(derived, f"l{i}_{f.name}") + for f in dataclasses.fields(_MoeScalars) + } + ) + for i in sorted(self.moe_layers) + } + self._build_embed_owners() + self.prebuild_kernels() + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + self._built_for = identity + + def cache_derived_state(self) -> None: + """The staged loaders' entry to the same build (``post_load_weights``).""" + self.post_load_weights() + + def _derive_moe(self, i: int) -> _MoeScalars: + """The routed experts' kernel scalars from the checkpoint's per-tensor scales (trtllm-gen NVFP4 layout). + + Global input scales: the reciprocal of the largest per-expert input scale (gate and up must agree). Per expert: + ``alpha = 1 / (input_scale * (1 / weight_scale_2))`` for the gate (fc31_alpha), the up half (its GEMM2-input + rescale ``fc31_scale_c = fc2_input_scale * up_alpha``) and the down projection; SwiGLU-OAI's activation + constants per expert, with beta and the clamp in the pre-dequant GEMM output domain (divided by fc31_alpha). + """ + w = self.w + isc1, isc3, isc2 = w[f"l{i}_e_isc1"].data, w[f"l{i}_e_isc3"].data, w[f"l{i}_e_isc2"].data + if not torch.allclose(isc1, isc3): + raise ValueError(f"layer {i}: the routed experts' w1 and w3 input scales differ") + fc31_input_scale = isc1.max().reciprocal() + fc2_input_scale = isc2.max().reciprocal() + ws2_1, ws2_3, ws2_2 = ( + w[f"l{i}_e_ws2_1"].data, + w[f"l{i}_e_ws2_3"].data, + w[f"l{i}_e_ws2_2"].data, + ) + fc31_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_1)) + up_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_3)) + fc2_alpha = 1.0 / (fc2_input_scale * (1.0 / ws2_2)) + fc31_scale_c = fc2_input_scale * up_alpha + n = self.num_experts + device = fc31_alpha.device + act_alpha = torch.full((n,), self.swiglu_alpha, dtype=torch.float32, device=device) + act_beta = torch.full((n,), self.swiglu_beta, dtype=torch.float32, device=device) + act_clamp = torch.full((n,), self.swiglu_limit, dtype=torch.float32, device=device) + act_beta.div_(fc31_alpha) + act_clamp.div_(fc31_alpha) + return _MoeScalars( + fc31_input_scale=fc31_input_scale, + fc2_input_scale=fc2_input_scale, + fc31_alpha=fc31_alpha, + fc2_alpha=fc2_alpha, + fc31_scale_c=fc31_scale_c, + act_alpha=act_alpha, + act_beta=act_beta, + act_clamp=act_clamp, + ) + + def _build_embed_owners(self) -> None: + """The fused embedding + first RMSNorm's caller-owned buffers, one per (rows, id dtype) of a decode step.""" + table, weight = self.w["embed"], self.w["l0_in_norm"] + self.embed_owners = { + (rows, dtype): DecodeEmbedding(table, weight, self.rms_eps, rows, dtype) + for rows in range(1, EMBED_FIRSTNORM_MAX_TOKENS + 1) + for dtype in (torch.int32, torch.int64) + } + + def projections(self) -> List[Tuple[str, str, Optional[str]]]: + """Every MXFP8 projection in layer order: (weight key, scale key, the boundary its GEMV pushes into).""" + out = [] + for i in range(self.num_layers): + out.append((f"l{i}_qkv_w", f"l{i}_qkv_s", None)) + out.append((f"l{i}_o_w", f"l{i}_o_s", "attn")) + if i in self.moe_layers: + out.append((f"l{i}_sh_gu_w", f"l{i}_sh_gu_s", None)) + out.append((f"l{i}_sh_dn_w", f"l{i}_sh_dn_s", "moe")) + else: + out.append((f"l{i}_mlp_gu_w", f"l{i}_mlp_gu_s", None)) + out.append((f"l{i}_mlp_dn_w", f"l{i}_mlp_dn_s", None)) + return out + + def decode_query_lens(self) -> List[int]: + """The query rows per request a decode step can have: 1, and 1 + each draft length speculation runs.""" + lens = {1} + spec = getattr(self.model_config, "spec_config", None) + if spec is not None: + # The verify width the config itself declares, whatever it names its draft count. + per_step = getattr(spec, "tokens_per_gen_step", None) + if isinstance(per_step, int) and per_step > 1: + lens.add(per_step) + for drafts in ( + getattr(spec, "max_draft_len", None), + getattr(spec, "max_total_draft_tokens", None), + *(getattr(spec, "draft_len_schedule", None) or {}).values(), + ): + if drafts: + lens.add(1 + int(drafts)) + return sorted(lens) + + def prebuild_kernels(self) -> Dict[str, object]: + """Compile (and load) every kernel a decode forward dispatches to that would otherwise compile on its first + call: each projection's decode GEMVs (every row count, activation and boundary push), the MoE engines of + ``MOE_ENGINE`` / ``MOE16_ENGINE``'s token counts, and the index scorer at the decode query lengths. Runs on the + current (main) stream before the executor's warmup, in the same order on every rank, then synchronizes; after + it, GEMV compiles are frozen and the forward only looks kernels up (a missing one is a miss). Returns the + seconds each part took and the scorer keys.""" + times: Dict[str, object] = {} + if not self.mxfp8.quant_method.use_cutlass: + raise ValueError("the MiniMax-M3 decode dispatch needs the CUTLASS MXFP8 GEMM") + mx_gemv.freeze(False) + use_pdl = get_env_enable_pdl() + world = int(self.mapping.tp_size) + start = time.perf_counter() + for wk, sk, push in self.projections(): + weight, scale = self.w[wk], self.w[sk] + mx_gemv.ensure_compiled(weight, scale, use_pdl) + n, k = weight.shape + for rows, x_bf16, ar in _gemv_variants(n, k, push, world): + mx_gemv.precompile(weight, scale, rows, x_bf16, use_pdl=use_pdl, ar=ar) + torch.cuda.synchronize() + times["gemv_s"] = time.perf_counter() - start + start = time.perf_counter() + self._prebuild_shdown_pin() + torch.cuda.synchronize() + times["shdown_pin_s"] = time.perf_counter() - start + start = time.perf_counter() + for i in sorted(self.moe_layers): + self._prebuild_engines(i) + torch.cuda.synchronize() + times["moe_s"] = time.perf_counter() - start + start = time.perf_counter() + times["scorer_keys"] = self._prebuild_index_scorer( + self.decode_query_lens(), self.w["embed"].device + ) + torch.cuda.synchronize() + times["scorer_s"] = time.perf_counter() - start + start = time.perf_counter() + times["proxy_variants"] = self._prebuild_msa_proxy() if MSA_PROXY_PREBUILD else [] + times["proxy_s"] = time.perf_counter() - start + mx_gemv.freeze(True) + logger.info( + "[M3 prebuild] decode kernels built before warmup: GEMVs %.1f s, MoE engines %.1f s, index scorer " + "%.1f s (%d variants), MSA indexer proxy %.1f s (%d variants)" + % ( + times["gemv_s"], + times["moe_s"], + times["scorer_s"], + len(times["scorer_keys"]), + times["proxy_s"], + len(times["proxy_variants"]), + ) + ) + return times + + def _prebuild_shdown_pin(self) -> None: + """Compile and load the pinned CuTe DSL GEMM (SHDOWN_PIN_ROWS) at every pinned row count before warmup and + capture: FlashInfer compiles it on first use per configuration, which must not happen inside a graph capture.""" + for wk, sk, _ in self.projections(): + weight, scale = self.w[wk], self.w[sk] + pinned = SHDOWN_PIN_ROWS.get(tuple(weight.shape), ()) + if not pinned: + continue + k = weight.shape[1] + for rows in sorted(pinned): + x = torch.zeros(rows, k, dtype=torch.bfloat16, device=weight.device) + act, act_scale = torch.ops.trtllm.mxfp8_quantize(x, True) + m3_mxfp8_gemm_cute_dsl(act.view(rows, k), act_scale, weight, scale, torch.bfloat16) + logger.info( + f"[M3 prebuild] SHDOWN pin: cute_dsl compiled for {list(weight.shape)} " + f"rows {min(pinned)}..{max(pinned)}" + ) + # SWQ (Triton) compiles on first use too; one shape covers every row count. + if SWQ_ROWS.get(tuple(weight.shape)): + m3_swq( + torch.zeros(max(pinned), 2 * k, dtype=torch.bfloat16, device=weight.device), + self.swiglu_limit, + self.swiglu_alpha, + self.swiglu_beta, + ) + logger.info(f"[M3 prebuild] SWQ: compiled for intermediate {k}") + return + + # ------------------------------------------------------------------------------------------ the MoE engines + + def _expert_operands(self, i: int): + """The routed experts' operands the engines read (trtllm-gen NVFP4 SwiGLU-OAI layout), and the input scale.""" + w, m = self.w, self.moe[i] + tensors = [ + w[f"l{i}_e_w31"].data, + w[f"l{i}_e_s31"].data, + w[f"l{i}_e_w2"].data, + w[f"l{i}_e_s2"].data, + m.fc31_alpha, + m.fc31_scale_c, + m.act_alpha, + m.act_beta, + m.act_clamp, + m.fc2_alpha, + ] + return tensors, m.fc31_input_scale + + def _shared_weights(self, i: int): + w = self.w + return ( + w[f"l{i}_sh_gu_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_gu_s"].data, + w[f"l{i}_sh_dn_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_dn_s"].data, + ) + + def engine(self, i: int, num_tokens: int, shared: bool) -> Optional[object]: + """Layer ``i``'s prebuilt routed-expert engine for ``num_tokens`` (1..8), router folded in (and the shared + expert with ``shared``). None after a miss: the build failed or was never made (a forward never builds one).""" + return self._lookup( + i, self.moe_engines[i], (num_tokens, True, shared), MOE_ENGINE.path, num_tokens + ) + + def moe16_engine(self, i: int, num_tokens: int) -> Optional[object]: + """Layer ``i``'s prebuilt 9..13-row engine (router, routed and shared experts in one kernel), or None.""" + if not MOE16_MIN_TOKENS <= num_tokens <= MOE16_MAX_TOKENS: + return None + return self._lookup(i, self.moe16_engines[i], num_tokens, MOE16_ENGINE.path, num_tokens) + + def _lookup(self, i: int, engines: dict, key, path: str, num_tokens: int) -> Optional[object]: + if key not in engines: + what = f"layer {i}'s engine" + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, what) + else: + missed(path, num_tokens, f"{what} was not built before warmup") + return None + engine = engines[key] + if engine is None: + missed(path, num_tokens, "its build failed (logged)") + return engine + + def _prebuild_engines(self, i: int) -> None: + """Build (compile, on first use of each shared state, and warm) every engine a decode step of layer ``i`` can + take: ``MOE_ENGINE``'s token counts with the shared expert folded in, and ``MOE16_ENGINE``'s. Whether an + engine is kept is agreed across the TP group (one host collective per engine): a build that fails on one rank + drops that engine on every rank, so the ranks never mix an engine (and its boundary push) with the generic + MoE.""" + engines = self.moe_engines.setdefault(i, {}) + engines16 = self.moe16_engines.setdefault(i, {}) + for num_tokens in sorted(MOE_ENGINE.keys): + key = (num_tokens, True, True) + if key not in engines: + engines[key] = m3_mc_allreduce.tp_agree_built( + self.mapping, + self._try_build( + i, MOE_ENGINE.path, num_tokens, self._build_engine, shared=True + ), + f"[M3 dispatch] {MOE_ENGINE.path} at {num_tokens} rows: layer {i}'s engine", + ) + for num_tokens in sorted(MOE16_ENGINE.keys): + if num_tokens not in engines16: + engines16[num_tokens] = m3_mc_allreduce.tp_agree_built( + self.mapping, + self._try_build(i, MOE16_ENGINE.path, num_tokens, self._build_moe16_engine), + f"[M3 dispatch] {MOE16_ENGINE.path} at {num_tokens} rows: layer {i}'s engine", + ) + + def _try_build(self, i: int, path: str, num_tokens: int, build, **kwargs) -> Optional[object]: + try: + return build(i, num_tokens, **kwargs) + except (ImportError, RuntimeError, ValueError) as err: + logger.warning( + f"[M3 dispatch] {path} at {num_tokens} rows: layer {i}'s engine failed to build: {err}" + ) + return None + + def _build_engine(self, i: int, num_tokens: int, shared: bool) -> object: + tensors, scale = self._expert_operands(i) + device = tensors[0].device + sms = torch.cuda.get_device_properties(device).multi_processor_count + kwargs = {} + if shared: + kwargs["shared_weights"] = self._shared_weights(i) + state = _ENGINE_STATES.get((str(device), num_tokens, shared)) + if state is None: + state = _ENGINE_STATES[(str(device), num_tokens, shared)] = M3MoeState( + device, num_tokens=num_tokens, num_ctas=sms - MOE_ENGINE_RESERVED_SMS, shared=shared + ) + engine = state.layer( + *tensors, + self.w[f"l{i}_router_bias"].data, + router_weight=self.w[f"l{i}_router"].data, + fc31_input_scale=scale.reshape(1).float(), + **kwargs, + ) + # Compile (first use of the state) and warm. + m3_moe_engine( + torch.zeros(num_tokens, tensors[0].shape[-1] * 2, dtype=torch.bfloat16, device=device), + engine, + ) + return engine + + def _build_moe16_engine(self, i: int, num_tokens: int) -> object: + tensors, scale = self._expert_operands(i) + device = tensors[0].device + state = _MOE16_STATES.get((str(device), num_tokens)) + if state is None: + state = _MOE16_STATES[(str(device), num_tokens)] = M3Moe16State( + device, num_tokens=num_tokens + ) + engine = state.layer( + *tensors, + self.w[f"l{i}_router_bias"].data, + router_weight=self.w[f"l{i}_router"].data, + fc31_input_scale=scale.reshape(1).float(), + shared_weights=self._shared_weights(i), + ) + # Compile (first use of the state) and warm. + m3_moe16_engine( + torch.zeros(num_tokens, tensors[0].shape[-1] * 2, dtype=torch.bfloat16, device=device), + engine, + ) + return engine + + # ------------------------------------------------------------------------------------------ the index scorer + + def _scorer_variants(self, query_lens) -> list: + """The scorer keys a decode step can take: every sparse layer's (index dtype, index heads per rank, index dim) + at each decode query length, with the early-issue schedule and without it, where the kernel serves it.""" + runner = _index_score_kernel.IndexScoreRunner + keys = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + config = backend.m3_config + dtype = torch.float8_e4m3fn if backend.indexer_kv_dtype == "fp8" else torch.bfloat16 + for qlen in sorted(set(query_lens)): + if not runner.is_supported( + q_dtype=dtype, + num_heads=config.num_index_heads, + head_dim=config.sparse_index_dim, + page_size=runner.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ): + continue + for early in (True, False): + key = ( + dtype, + int(config.num_index_heads), + int(qlen), + int(config.sparse_index_dim), + int(INDEX_SCORE_SPLIT_K), + bool(early), + ) + if key not in keys: + keys.append(key) + return keys + + def _prebuild_msa_proxy(self) -> list: + """Compile and load every fmha_sm100 variant the MSA indexer's proxy pass can pick (OnlyScore over the paged + index-K: one KV head, page 128, no split-KV), so no context chunk compiles one after the server is ready (a + 65..128-token chunk's variant did, stalling the steps behind it by 0.8-1.6 ms). The variant follows only the + longest query row: fmha_sm100 packs a row of up to 32 tokens by its pack factor, then takes the 128- or 256-row + query tile and the single-warpgroup kernel from the packed length, so query lengths 1..129 reach every variant. + Compile and load only, nothing runs: no numerics change. fmha_sm100 keys its cache by its installed sources, so + a source overlay builds its own variants. Returns the variant names.""" + require_msa_module() + from fmha_sm100 import api as fmha_api + from fmha_sm100 import jit as fmha_jit + + only_score, page = 2, 128 + names = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + dtype = torch.float8_e4m3fn if backend.indexer_kv_dtype == "fp8" else torch.bfloat16 + heads = int(backend.m3_config.num_index_heads) + for qlen in range(1, 130): + pack = fmha_api._compute_pack_factor(qlen, heads, 1) + packed = qlen * pack + key = ( + fmha_jit._dlpack_dtype_code(dtype), + 128 if packed <= 128 else 256, + packed <= 64, + only_score, + page, + False, + pack, + ) + name, _ = fmha_jit._variant_key_from_runtime(*key) + if name not in names: + fmha_jit.get_fmha_variant(*key) + names.append(name) + return names + + def _prebuild_index_scorer(self, query_lens, device) -> list: + """Compile every scorer key and launch each once on real-shaped dummy operands (loading its module), on the + current stream. The kernel's batch, token, page and block counts are symbolic: its compile needs no step + metadata. Returns the keys.""" + runner = _index_score_kernel.IndexScoreRunner + keys = self._scorer_variants(query_lens) + page = runner.SUPPORTED_PAGE_SIZE + for key in keys: + dtype, heads, qlen, dim, split_k, early = key + runner._compile(dtype, heads, qlen, dim, split_k, early) + rows = 1 if early else INDEX_SCORE_EARLY_MAX_REQUESTS + 1 + blocks = 2 + tokens = rows * qlen + idx_q = torch.zeros(tokens, heads, dim, dtype=torch.bfloat16, device=device).to(dtype) + idx_k = torch.zeros( + rows * blocks, 1, page, dim, dtype=torch.bfloat16, device=device + ).to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device=device).view( + rows, blocks + ) + seq_lens = torch.full((rows,), blocks * page, dtype=torch.int32, device=device) + score = torch.empty(heads, blocks, tokens, dtype=torch.float32, device=device) + runner.kernel_cache[key]( + idx_q, idx_k.squeeze(1), block_table, score.transpose(1, 2), seq_lens + ) + return keys + + # --------------------------------------------------------------------------------------------- the dispatch + + def _select_target(self, attn_metadata: AttentionMetadata): + """Which target runs this step: one host-side integer, fixed per CUDA-graph capture. A step with context rows + (a mixed step included) is prefill's; every pure generation step is decode's, whatever its width.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.IntTensor] = None, + position_ids: Optional[torch.IntTensor] = None, + inputs_embeds: Optional[torch.FloatTensor] = None, + spec_metadata: Optional[SpecMetadata] = None, + capture: Callable[..., None] = _plain_capture, + **kwargs, + ) -> torch.Tensor: + if (input_ids is None) ^ (inputs_embeds is not None): + raise ValueError("You must specify exactly one of input_ids or inputs_embeds") + if kwargs.get("lora_params") is not None: + raise ValueError("this MiniMax-M3 target serves no LoRA adapters") + return self._select_target(attn_metadata).forward( + attn_metadata, input_ids, position_ids, inputs_embeds, spec_metadata, capture + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# The targets +# -------------------------------------------------------------------------------------------------------------------- + + +def _swiglu_act(core, gu: torch.Tensor) -> torch.Tensor: + """The measured default SwiGLU-OAI activation path.""" + return m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + + +class PrefillTarget: + """A step with context rows: prefill chunks, and the mixed steps that carry generation rows beside them. Every + layer runs in its generic form; the attention core is the MSA backend's for every row.""" + + def __init__(self, core: MiniMaxM3Tp4ModelingV2) -> None: + self.core = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it; read once here rather than per forward. + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + Counts and per-step host scalars are read here; device tensors are the metadata's persistent buffers, held + by reference; ``msa_idx_k_cache`` is the metadata's per-layer index-K cache getter. ``metadata`` is the + prepared object itself: the MSA backend's step (``m3_msa_attention``) takes it as its argument. + """ + return dict( + num_contexts=md.num_contexts, + num_ctx_tokens=md.num_ctx_tokens, + num_tokens=md.num_tokens, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + metadata=md, + ) + + def forward( + self, attn_metadata, input_ids, position_ids, inputs_embeds, spec_metadata, capture + ): + core = self.core + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) prefill CUDA graphs" + ) + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + _record("prefill", int(step["num_tokens"]), int(step["num_contexts"])) + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + hidden, residual = self._layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + return hidden + + def _swiglu_mlp(self, h: torch.Tensor, prefix: str, low_m_gemm: bool) -> torch.Tensor: + """A gated MXFP8 MLP (the dense MLP, the shared expert): gate_up, SwiGLU-OAI, down, no reduction.""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s", low_m_gemm) + act = _swiglu_act(core, gu) + out, _ = _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm) + return out + + def _routed_chunks(self, i: int, x, router_logits): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: + the MoE scheduler's even split, even chunks on the MoE chunking stream.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + # CTXNORMQ: the boundary already wrote this activation's NVFP4 form. + fp4 = _CTXNORMQ_FP4.take(x) if CTXNORMQ else None + if num_chunks <= 1: + if fp4 is not None: + return _routed_experts(core, i, fp4[0], fp4[1], router_logits, True) + return _routed_experts(core, i, x, None, router_logits, True) + sizes = _split_chunk(rows, num_chunks) + xs, logits = x.split(sizes), router_logits.split(sizes) + sfs = [None] * num_chunks + if fp4 is not None: + xs = fp4[0].split(sizes) + sfs = [t.flatten() for t in fp4[1].view(rows, -1).split(sizes)] + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + else: + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor): + """Router + routed experts, finalized: the fused router with the NVFP4 activation up to 48 rows, else the + router logits and the experts quantizing their bf16 input (chunked).""" + core = self.core + fused = _router_fp4(core, i, h) + if fused is not None: + router_logits, x_fp4, x_sf = fused + return _routed_experts(core, i, x_fp4, x_sf, router_logits, True) + router_logits = _router_logits(core, i, h) + return self._routed_chunks(i, h, router_logits) + + def _moe(self, i: int, h: torch.Tensor, *, return_parts: bool): + """The MoE block of layer ``i`` without its reduction. Returns ``(out, routed)``: with ``return_parts`` the + shared and routed parts (routed None when the engine folded the shared expert in), else their sum in + ``out``.""" + core = self.core + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + h.dim() == 2 + and h.dtype == torch.bfloat16 + and 1 <= rows <= MOE_ENGINE_MAX_TOKENS + and rows >= MOE_FOLD_MIN_TOKENS + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + _record("m3_moe_engine", rows, False) + folded = m3_moe_engine(h.contiguous(), engine, None, None) + return folded, None + routed_out, shared_out = maybe_execute_in_parallel( + lambda: self._routed(i, h), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False), + core.moe_events[i][0], + core.moe_events[i][1], + # The warmup tuning pass profiles the shared expert's GEMMs: keep it on the main stream, between the + # same collectives on every rank (the capture pass and serving still overlap it on the aux stream). + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if return_parts: + return shared_out, routed_out + # In-place add into the shared output to avoid allocating a temporary. + return shared_out.add_(routed_out), None + + def _generic_boundary(self, x, x2, residual, norm_w): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce.""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + if norm_w is not core.w["final_norm"]: + fused = _ctxnormq_allreduce_norm(core, x, residual, norm_w, None) + if fused is not None: + return fused + return _native_allreduce_norm(core, x, residual, norm_w) + + def _layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + else: + fused = _ctxnormq_allreduce_norm( + core, + hidden, + residual, + core.w[f"l{i}_post_norm"], + core.moe[i].fc31_input_scale if i in core.moe_layers else None, + ) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + if i in core.moe_layers: + out, routed = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out = self._swiglu_mlp(hidden, f"l{i}_mlp", True) + routed = None + normed, updated = self._generic_boundary(out, routed, residual, _next_norm(core, i)) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + capture(spec_metadata, i, updated, None) + return normed, updated + + def _attention(self, i: int, position_ids, hidden, step): + """Attention of layer ``i`` up to its o_proj (no reduction). Returns (o_proj output, pushed): a prefill step + arms no boundary push, so pushed is always False.""" + core = self.core + if i not in core.sparse_layers: + # Dense layers 0-2: QKV, the fused norm + RoPE (FP8 q/k/v for an FP8 cache, else in place in bf16), the + # MSA backend (it writes the layer's cache), o_proj. + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + if not core.fused_projection: + return self._unfused_sparse_attention(i, position_ids, hidden, step) + packed, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + ) + if horizontal is None: + raise RuntimeError( + f"MiniMax-M3 sparse layer {i}: no horizontal producer takes this step's caches (an FP8 or NVFP4 HND " + "pool with an FP8 index-K cache)" + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output.""" + core = self.core + if _producer_covered(core): + # Not a miss: the producer packs the five-way projection's output, which this configuration lacks. + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): the MSA + backend (``m3_msa_attention``: the KV write unless a producer wrote the caches, the sparse layers' top-k block + selection, the FMHA over the context rows and any generation rows beside them).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + live = int(step["num_tokens"]) + q_live, out_live = q[:live], output[:live] + k_live = k[:live] if k is not None else None + v_live = v[:live] if v is not None else None + idx_q_live = idx_q[:live] if idx_q is not None else None + sparse = i in core.sparse_layers + nvfp4 = core.main_kv_is_nvfp4 + if TSEL and sparse: + _tsel_register(core.attn[i]) + if GQAMIX and sparse: + _gqamix_register(core.attn[i]) + _record("m3_msa_attention", sparse, k_live is not None) + m3_msa_attention( + core.attn[i], + q_live, + k_live, + v_live, + idx_q_live, + step["metadata"], + out_live, + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + return output + + +class DecodeTarget: + """A pure generation step (no context rows), of any width. + + Up to 32 tokens with 1-D input ids, per layer: attention (o_proj all-reduce deferred, its rows + pushed into the boundary all-reduce up to 16 tokens) -> attention boundary (all-reduce + residual + post-attention + RMSNorm) -> MoE (the 1..8 / 9..13-row engines, the first pushing its sum into the boundary all-reduce; the + unfinalized MoE with the finalize in the all-reduce from 14 rows) or dense MLP -> MoE boundary (all-reduce + + shared/routed add + residual + the next layer's RMSNorm, emitting the next QKV's MXFP8 activation). A layer whose + output Eagle3 captures runs the generic layer, as does the first layer when the fused embedding does not take the + step. A wider step (or one carrying embeddings) runs every layer in its generic form, with this target's attention + (the decode selection and sparse decode where the producer wrote the caches). + """ + + def __init__(self, core: MiniMaxM3Tp4ModelingV2) -> None: + self.core = core + self._moe_push_descs: Dict[tuple, Any] = {} + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it; read once here rather than per forward. + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + States the two context counts instead of reading them back: this target runs only when ``num_contexts`` is + zero, so reading them would ask the metadata to confirm what the routing already decided. + + Graph classes: counts, the generation span, the identity-selection flag and the worst-case score extent are + host values fixed per capture (prepare() stages them before the forward); the block table, lengths, cache + slots, identity table and valid-block counts are the metadata's persistent buffers, held by reference and + refreshed in place every step; ``msa_idx_k_cache`` / ``msa_proxy_max_score_view`` are the metadata's getters + of the per-layer index-K cache and of the score scratch view. ``metadata`` is the prepared object itself: the + MSA backend's step (``m3_msa_attention``) takes it as its argument. + """ + return dict( + num_contexts=0, + num_ctx_tokens=0, + num_tokens=md.num_tokens, + num_seqs=md.num_seqs, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + msa_decode_span=md.msa_decode_span, + msa_block_table=md.msa_block_table, + msa_seq_lens_cuda=md.msa_seq_lens_cuda, + msa_short_ok=md._msa_short_ok, + msa_short_table=md.msa_short_table, + msa_proxy_max_score_view=md.msa_proxy_max_score_view, + msa_worst_case_max_k_tiles=md.msa_worst_case_max_k_tiles, + msa_n_valid_blocks=md.msa_n_valid_blocks, + metadata=md, + ) + + def forward( + self, attn_metadata, input_ids, position_ids, inputs_embeds, spec_metadata, capture + ): + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + # Fused-layer rows are limited independently of prebuild/workspace capacity. + kernels = ( + inputs_embeds is None and input_ids.ndim == 1 and is_decode_step(input_ids.numel()) + ) + if kernels: + num_tokens = int(input_ids.numel()) + plan = plan_for(num_tokens) + _record("decode", num_tokens, plan.name) + hidden, residual = self._embed(input_ids, num_tokens) + else: + uncovered( + "decode_kernels", + f"outside {FUSED_DECODE_MAX_TOKENS}-row fused scope, embeddings or 2-D ids: generic layers", + ) + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) CUDA graphs" + ) + plan = None + _record("decode", int(step["num_tokens"]), "generic") + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + captured = spec_metadata is not None and spec_metadata.is_layer_capture(i) + if not kernels or residual is None or captured: + hidden, residual = self._generic_layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + else: + hidden, residual = self._layer(i, plan, position_ids, hidden, residual, step) + return hidden + + def _embed(self, input_ids: torch.Tensor, num_tokens: int): + """(hidden, residual) of the fused embedding + first RMSNorm; (embedding, None) when it does not run, so the + first layer runs its own input norm.""" + core = self.core + if not EMBED_FIRSTNORM.covers(num_tokens): + uncovered(EMBED_FIRSTNORM.path, f"more than {EMBED_FIRSTNORM_MAX_TOKENS} tokens") + return m3_embedding(input_ids, core.w["embed"]), None + owner = core.embed_owners.get((num_tokens, input_ids.dtype)) + if owner is None or not input_ids.is_contiguous(): + missed( + EMBED_FIRSTNORM.path, + num_tokens, + f"no embedding owner for {input_ids.dtype} ids (built for int32 / int64 after the load)" + if owner is None + else "the input ids are not contiguous", + ) + return m3_embedding(input_ids, core.w["embed"]), None + _record("m3_embedding_firstnorm", num_tokens, input_ids.dtype) + return m3_embedding_firstnorm( + input_ids, owner.table, owner.weight, owner.eps, owner.raw, owner.output, owner.use_pdl + ) + + def _bucket_flags(self, num_tokens: int): + """(bucket workspace, its 8-CTA call counters) of the boundary all-reduce for ``num_tokens``, or None.""" + workspace = m3_mc_allreduce.get_workspace(self.core.mapping, self.core.hidden) + workspace = None if workspace is None else m3_mc_allreduce._bucket(workspace, num_tokens) + flags = None if workspace is None else workspace.flags_for(8) + return None if flags is None else (workspace, flags) + + @staticmethod + def _no_bucket(path: str, num_tokens: int) -> None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, "the multicast all-reduce workspace") + else: + missed(path, num_tokens, "no multicast all-reduce bucket holds the step") + + def _attn_push_target(self, num_tokens: int): + """The o_proj's push tuple into the attention boundary's all-reduce, or None (after a miss). + + None comes only from the token count or from workspace state created collectively in the same eager call on + every rank, so all ranks agree. (A lone non-pushing rank would still be safe: its all-reduce pushes the same + bf16 rows into the same set A.)""" + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(ATTN_PUSH.path, num_tokens) + return None + workspace, flags = found + return ( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + AR_PUSH_MODE, + ) + + def _moe_push_targets(self, num_tokens: int): + """(engine push descriptor, shared down-projection push tuple) for the MoE boundary's exchange sets, or None + (after a miss). + + This decision must match across ranks: prepushed 3 reads set B of every rank, which a non-pushing rank never + fills. None comes only from the token count or from state created in the same eager call on every rank + (workspace, flags, the descriptor cache), and the engine it is armed for is agreed at build time + (``_prebuild_engines``), so it does.""" + hidden = self.core.hidden + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(MOE_PUSH.path, num_tokens) + return None + workspace, flags = found + key = (workspace.mc.data_ptr(), flags.data_ptr()) + desc = self._moe_push_descs.get(key) + if desc is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(MOE_PUSH.path, num_tokens, "the MoE engine's push descriptor") + return None + desc = self._moe_push_descs[key] = ar_push_desc( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + flags.device, + ) + set_b = workspace.mc.data_ptr() + 2 * 4 * workspace.max_tokens * workspace.world_size * ( + hidden // 2 + ) + return desc, ( + set_b, + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + 2, + ) + + def _boundary(self, use_mc: bool, x, residual, norm_w, x2=None, quant=False): + """All-reduce + (second addend) + residual + RMSNorm: the multicast kernel, else the built-in all-reduce.""" + core = self.core + if use_mc and not MC_ALLREDUCE.covers(x.shape[0]): + uncovered( + MC_ALLREDUCE.path, + f"more than {m3_mc_allreduce.max_rows()} rows: the built-in all-reduce runs", + ) + use_mc = False + if use_mc: + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w, quant=quant) + if fused is not None: + return fused + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + missed( + MC_ALLREDUCE.path, + x.shape[0], + m3_mc_allreduce.decline_reason(x, residual, workspace, x2=x2), + ) + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _prepushed(self, i: int, x, residual, norm_w, quant, prepushed, x2=None): + fused = _mc_allreduce_norm( + self.core, x, x2, residual, norm_w, quant=quant, prepushed=prepushed + ) + if fused is None: + raise RuntimeError(f"layer {i}: rows were pushed into an all-reduce that did not run") + return fused + + def _layer(self, i: int, plan: DecodePlan, position_ids, hidden, residual, step): + core = self.core + num_tokens = hidden.shape[0] + # Padded rows: the boundary all-reduce polls and re-arms rows t < M = residual.shape[0], CUDA-graph padding + # rows included. Every producer pushes all rows of the same tensor: the o_proj / shared down GEMV t < m = + # x.shape[0] (its row count comes from the host shape, not a live count), the MoE engine every row of the + # num_tokens it was built for (= num_tokens here). The all-reduce checks x.sizes() == residual.sizes(), so + # no padded row is left with a stale flag. + # The GEMV dispatch (and with it every push into a boundary all-reduce) is a decode-graph one. + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + if not decode_graph: + uncovered( + "decode_graph", + "an eager decode step: the decode-graph GEMV dispatch and the boundary pushes are off", + ) + push = plan.attn_mc_ar and decode_graph and ATTN_PUSH.covers(num_tokens) + armed = self._attn_push_target(num_tokens) if push else None + hidden, attn_pushed = self._attention(i, position_ids, hidden, step, armed) + post_norm = core.w[f"l{i}_post_norm"] + if armed is not None and attn_pushed: + hidden, residual = self._prepushed(i, hidden, residual, post_norm, False, AR_PUSH_MODE) + else: + if armed is not None: + missed( + ATTN_PUSH.path, + num_tokens, + "the o_proj did not push its rows into the armed boundary all-reduce", + ) + hidden, residual = self._boundary(plan.attn_mc_ar, hidden, residual, post_norm) + + norm_w = _next_norm(core, i) + if i not in core.moe_layers: + out, _ = self._swiglu_mlp(hidden, f"l{i}_mlp", True) + return self._boundary(plan.attn_mc_ar, out, residual, norm_w, quant=False) + + quant = plan.qkv_prequant and i + 1 < core.num_layers + path = moe_path(num_tokens) + if path is None: + uncovered( + MC_FINALIZE.path, + f"more than {m3_mc_allreduce.max_rows()} rows: the finalized MoE and the boundary all-reduce run", + ) + moe_push = None + if path == MOE_ENGINE.path: + # The engine runs inside the MoE; looked up (or found missing) here first so a push is armed only for an + # engine that will take it. + engine = core.engine(i, num_tokens, shared=True) + if engine is not None and plan.moe_mc_ar and decode_graph: + moe_push = self._moe_push_targets(num_tokens) + elif path == MOE16_ENGINE.path and decode_graph: + # Router, routed and shared experts in one kernel; its bf16 sum goes to the boundary all-reduce. + engine = core.moe16_engine(i, num_tokens) + if engine is not None: + _record("m3_moe16_engine", num_tokens) + out = m3_moe16_engine(hidden.contiguous(), engine) + return self._boundary(plan.moe_mc_ar, out, residual, norm_w, quant=quant) + if path in (MOE16_ENGINE.path, MC_FINALIZE.path) and plan.moe_mc_ar: + if self._bucket_flags(num_tokens) is None: + self._no_bucket(MC_FINALIZE.path, num_tokens) + else: + (shared, fc2, scales, perm), _, _, _ = self._moe( + i, hidden, return_parts=False, do_finalize=False + ) + _record("m3_mc_finalize_allreduce_norm", num_tokens, quant) + fused = m3_mc_allreduce.m3_mc_finalize_allreduce_norm( + fc2, + perm.view(num_tokens, -1).to(torch.int32), + scales.reshape(num_tokens, -1), + shared, + residual, + norm_w, + core.rms_eps, + m3_mc_allreduce.get_workspace(core.mapping, core.hidden), + use_pdl=get_env_enable_pdl(), + quant=quant, + ) + if fused is None: + raise RuntimeError( + "the MC finalize all-reduce rejected the unfinalized MoE outputs" + ) + return fused + shared, routed, engine_pushed, shared_pushed = self._moe( + i, hidden, return_parts=True, push=moe_push + ) + if moe_push is not None: + if routed is None: + # The engine folded the shared expert in and pushed routed + shared into set A. + if engine_pushed: + return self._prepushed(i, shared, residual, norm_w, quant, 2) + else: + if engine_pushed and shared_pushed: + return self._prepushed(i, shared, residual, norm_w, quant, 3, x2=routed) + if engine_pushed or shared_pushed: + raise RuntimeError("only one MoE part pushed into the boundary all-reduce") + missed( + MOE_PUSH.path, + num_tokens, + "the MoE did not push its output into the armed boundary all-reduce", + ) + return self._boundary(plan.moe_mc_ar, shared, residual, norm_w, x2=routed, quant=quant) + + def _swiglu_mlp( + self, h: torch.Tensor, prefix: str, low_m_gemm: bool, push: Optional[tuple] = None + ): + """A gated MXFP8 MLP (the dense MLP, the shared expert): gate_up, SwiGLU-OAI, down, no reduction. Returns + (output, whether the down projection's GEMV pushed into ``push``).""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s", low_m_gemm) + n, k = core.w[f"{prefix}_dn_w"].shape + rows = gu.shape[0] + if rows in SWQ_ROWS.get((n, k), ()) and _mx_mode(core.mxfp8).decode_graph: + # The MXFP8 activation is published keyed by ``gu`` and taken by the down projection's prequant branch. + act, act_scale = m3_swq(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + m3_prequant.SLOT.publish(gu, act, act_scale) + if rows not in _SWQ_LOGGED: + _SWQ_LOGGED.add(rows) + logger.info(f"[M3 dispatch] SWQ: SwiGLU + MXFP8 producer rows={rows} [{n}, {k}]") + return _linear(core, gu, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm, push) + act = _swiglu_act(core, gu) + return _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm, push) + + def _routed_chunks(self, i: int, x, router_logits): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: the MoE + scheduler's even split, even chunks on the MoE chunking stream.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + if num_chunks <= 1: + return _routed_experts(core, i, x, None, router_logits, True) + sizes = _split_chunk(rows, num_chunks) + xs, logits = x.split(sizes), router_logits.split(sizes) + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True)) + else: + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True)) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor, do_finalize: bool): + """Router + routed experts: the fused router with the NVFP4 activation up to 48 rows, else the router logits and + the experts quantizing their bf16 input.""" + core = self.core + fused = _router_fp4(core, i, h) + if fused is not None: + router_logits, x_fp4, x_sf = fused + return _routed_experts(core, i, x_fp4, x_sf, router_logits, do_finalize) + router_logits = _router_logits(core, i, h) + if not do_finalize: + return _routed_experts(core, i, h, None, router_logits, False) + return self._routed_chunks(i, h, router_logits) + + def _moe( + self, i: int, h: torch.Tensor, *, return_parts: bool, do_finalize: bool = True, push=None + ): + """The MoE block of layer ``i`` without its reduction. + + ``push`` = (the engine's ``arp`` descriptor, the shared down projection's ``ar`` tuple) the decode layer armed + for the MoE boundary's exchange sets, or None. Returns ``(out, routed, engine_pushed, shared_pushed)``: with + ``return_parts`` the shared and routed parts (routed None when the engine folded the shared expert in), else + their sum in ``out``; with ``do_finalize=False`` ``out`` is ``[shared, fc2 out, expert scales, permuted + index]``. + """ + core = self.core + engine_arp, shared_ar = push if push is not None else (None, None) + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + do_finalize + and h.dim() == 2 + and h.dtype == torch.bfloat16 + and 1 <= rows <= MOE_ENGINE_MAX_TOKENS + and rows >= MOE_FOLD_MIN_TOKENS + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + _record("m3_moe_engine", rows, engine_arp is not None) + folded = m3_moe_engine(h.contiguous(), engine, None, engine_arp) + return folded, None, engine_arp is not None, False + routed_out, (shared_out, shared_pushed) = maybe_execute_in_parallel( + lambda: self._routed(i, h, do_finalize), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False, shared_ar), + core.moe_events[i][0], + core.moe_events[i][1], + # The warmup tuning pass profiles the shared expert's GEMMs: keep it on the main stream, between the same + # collectives on every rank (the capture pass and serving still overlap it on the aux stream). + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if not do_finalize: + return [shared_out, *routed_out], None, False, shared_pushed + if return_parts: + return shared_out, routed_out, False, shared_pushed + # In-place add into the shared output to avoid allocating a temporary. + return shared_out.add_(routed_out), None, False, shared_pushed + + def _generic_boundary(self, x, x2, residual, norm_w): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce.""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _generic_layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + else: + hidden, residual = _native_allreduce_norm( + core, hidden, residual, core.w[f"l{i}_post_norm"] + ) + if i in core.moe_layers: + out, routed, _, _ = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out, _ = self._swiglu_mlp(hidden, f"l{i}_mlp", True) + routed = None + normed, updated = self._generic_boundary(out, routed, residual, _next_norm(core, i)) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + capture(spec_metadata, i, updated, None) + return normed, updated + + def _attention(self, i: int, position_ids, hidden, step, push: Optional[tuple] = None): + """Attention of layer ``i`` up to its o_proj (no reduction). ``push`` is the attention boundary's exchange the + o_proj may write into. Returns (o_proj output, pushed).""" + core = self.core + if i not in core.sparse_layers: + # Dense layers 0-2: QKV, the fused norm + RoPE (FP8 q/k/v: an FP8 cache, or the FP8 pool the dense layers + # sit in within a hybrid NVFP4 cache on a decode step), the MSA backend, o_proj. + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + out_fp8 = core.main_kv_is_fp8 or core.main_kv_is_nvfp4 + fused = _qk_norm_rope(core, i, qkv, position_ids, out_fp8) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + if not core.fused_projection: + return self._unfused_sparse_attention(i, position_ids, hidden, step, push) + # Gathered before the QKV projection so nothing runs between it and the fused attention (PDL hand-off). + fused = self._fused_attn_args(i, position_ids, step) + if fused is not None: + mega = _attn_mega(core, i, hidden, fused, push) + if mega is not None: + return mega + packed, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + if fused is not None and ( + packed.shape[0] != fused["num_tokens"] or packed.shape[1] != 2560 + ): + missed( + FUSED_DECODE_ATTN.path, + fused["num_tokens"], + f"the packed QKV is {tuple(packed.shape)}, not [rows, 2560]", + ) + fused = None + if fused is not None: + c1 = _attn_c1(core, i, packed, fused, push) + if c1 is not None: + return c1 + o = _fused_attn(core, i, packed, fused) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + ) + if horizontal is None: + raise RuntimeError( + f"MiniMax-M3 sparse layer {i}: no horizontal producer takes this step's caches (an FP8 or NVFP4 HND " + "pool with an FP8 index-K cache)" + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step, push): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output. ``push`` is the attention + boundary exchange the o_proj GEMV writes its rows into, as in the five-way path.""" + core = self.core + if _producer_covered(core): + # Not a miss: the producer packs the five-way projection's output, which this configuration lacks. + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + + def _fused_attn_args(self, i: int, position_ids, step) -> Optional[dict]: + """Arguments of the fused decode attention for this step, or None when it does not apply. + + The layer must be an NVFP4 sparse MSA layer and the step a pure generation step whose selection the metadata + staged as the identity (every request within 16 blocks of 128 tokens), with at most 8 query tokens per request. + """ + core = self.core + path = FUSED_DECODE_ATTN.path + if i not in core.sparse_layers or not core.main_kv_is_nvfp4: + uncovered(path, "not an NVFP4 sparse MSA layer with an FP8 indexer") + return None + if position_ids is None: + uncovered(path, "not a pure generation step with positions") + return None + if not step["msa_short_ok"]: + uncovered( + path, "a request beyond 16 KV blocks: the block selection is not the identity" + ) + return None + span = step["msa_decode_span"] + if span is None or span.row_first != 0: + uncovered(path, "no decode span starting at row 0") + return None + num_reqs = int(step["num_seqs"]) + qlen = int(span.query_len) + if not FUSED_DECODE_ATTN.covers((qlen, num_reqs)): + uncovered( + path, + f"more than {FUSED_ATTN_MAX_QUERY_LEN} query rows per request or {FUSED_ATTN_MAX_TOKENS} per step", + ) + return None + manager = step["kv_cache_manager"] + if manager is not None and not manager.is_nvfp4_layer(i): + uncovered(path, "the layer's KV cache is not NVFP4") + return None + if _fused_attn_kernel._module is None: + # Building compiles every geometry (minutes): only outside capture (the eager warmup steps). + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_reqs * qlen, "the fused attention module") + return None + _fused_attn_kernel._load_module() + if manager is None: + missed(path, num_reqs * qlen, "no KV-cache manager") + return None + k_scale, v_scale = _aligned_nvfp4_dequant_scales(core.attn[i], core.kv_scales[i]) + return dict( + num_tokens=num_reqs * qlen, + decode_query_len=qlen, + kv_data=manager.get_buffers(i, kv_layout="HND").view(torch.uint8), + kv_scale=manager.get_block_scale_buffers(i, kv_layout="HND"), + index_k_cache=step["msa_idx_k_cache"](i), + out_cache_loc=step["msa_out_cache_loc"], + position_ids=position_ids.reshape(-1).contiguous().to(torch.int32), + block_table=step["msa_block_table"][:num_reqs], + seq_lens=step["msa_seq_lens_cuda"][:num_reqs], + kv_quant_scale=core.inv_kv_scales[i], + k_global_scale=k_scale, + v_global_scale=v_scale, + ) + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): a sparse + layer whose caches the producer wrote runs this target's selection and sparse decode when the generation span + starts at row 0; everything else the MSA backend (``m3_msa_attention``: the KV write unless a producer wrote the + caches, the sparse layers' top-k block selection, the FMHA).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + live = int(step["num_tokens"]) + q_live, out_live = q[:live], output[:live] + k_live = k[:live] if k is not None else None + v_live = v[:live] if v is not None else None + idx_q_live = idx_q[:live] if idx_q is not None else None + span = step["msa_decode_span"] + sparse = i in core.sparse_layers + if sparse and k_live is None and span is not None and span.row_first == 0: + self._decode_sparse_core(i, q_live, idx_q_live, step, out_live) + return output + nvfp4 = core.main_kv_is_nvfp4 + _record("m3_msa_attention", sparse, k_live is not None) + m3_msa_attention( + core.attn[i], + q_live, + k_live, + v_live, + idx_q_live, + step["metadata"], + out_live, + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + return output + + def _decode_sparse_core(self, i: int, q, idx_q, step, output) -> None: + """Top-16 block selection and the block-sparse decode of one pure generation step of a sparse layer whose caches + the producer already wrote. + + The selection is the identity when the metadata staged it (every request within 16 blocks). Otherwise the CuTe + DSL scorer writes the per-index-head block maxima, reduced to the KV head, and the long-row selector takes the + top 16 (the built-in selector beyond its shapes). + """ + core = self.core + backend = core.attn[i] + config = backend.m3_config + num_tokens = int(q.shape[0]) + rows = int(step["num_seqs"]) + qlen = int(step["msa_decode_span"].query_len) + block_table = step["msa_block_table"][:rows] + seq_lens = step["msa_seq_lens_cuda"][:rows] + num_kv_heads = int(backend.indexer.config.num_kv_heads) + identity = step["msa_short_ok"] and step["msa_short_table"] is not None + if identity: + topk = step["msa_short_table"][: num_kv_heads * num_tokens * MSA_REQUIRED_TOPK].view( + num_kv_heads, num_tokens, MSA_REQUIRED_TOPK + ) + else: + idx_q_view = idx_q.view(num_tokens, config.num_index_heads, config.sparse_index_dim) + idx_k_cache = step["msa_idx_k_cache"](i) + max_score = step["msa_proxy_max_score_view"]( + config.num_index_heads, step["msa_worst_case_max_k_tiles"], num_tokens + ) + early = rows <= INDEX_SCORE_EARLY_MAX_REQUESTS + key = ( + idx_q_view.dtype, + int(config.num_index_heads), + int(qlen), + int(idx_q_view.shape[-1]), + int(INDEX_SCORE_SPLIT_K), + bool(early), + ) + if key not in _index_score_kernel.IndexScoreRunner.kernel_cache: + # Outside require mode the op below compiles it (the only scorer this step has). + missed( + INDEX_SCORE.path, + num_tokens, + f"the index scorer {key} was not compiled before warmup", + ) + _record("m3_index_score", key) + m3_index_score( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens, + max_score.transpose(1, 2), + qlen, + INDEX_SCORE_SPLIT_K, + early, + ) + n_valid_blocks = step["msa_n_valid_blocks"][:num_tokens] + group = config.num_index_heads // config.num_kv_heads + max_score_kv = ( + max_score.view( + config.num_kv_heads, group, max_score.shape[1], max_score.shape[2] + ).amax(dim=1) + if group > 1 + else max_score + ) + table = long_select_blocks( + max_score_kv, + n_valid_blocks=n_valid_blocks, + init_blocks=config.init_blocks, + local_blocks=config.local_blocks, + use_pdl=get_env_enable_pdl(), + early_trigger=num_tokens <= SELECT_EARLY_MAX_ROWS, + ) + if table is None: + uncovered( + LONG_SELECT.path, + f"{max_score_kv.shape[1]} KV blocks, outside " + f"{LONG_SELECT_MIN_BLOCKS}..{LONG_SELECT_MAX_BLOCKS}: the built-in selector", + ) + table = m3_select_blocks( + max_score_kv, + MSA_REQUIRED_TOPK, + n_valid_blocks, + config.init_blocks, + config.local_blocks, + True, + ) + topk = table.permute(1, 0, 2) + manager = step["kv_cache_manager"] + k_paged, v_paged = msa_paged_kv(manager, i) + nvfp4 = {} + if manager.is_nvfp4_layer(i): + k_scale, v_scale = _aligned_nvfp4_dequant_scales(backend, core.kv_scales[i]) + scales = manager.get_block_scale_buffers(i, "HND") + nvfp4 = dict( + k_block_scale=scales[:, 0], + v_block_scale=scales[:, 1], + k_global_scale=k_scale, + v_global_scale=v_scale, + ) + _record("m3_sparse_decode", num_tokens, qlen, identity) + m3_sparse_decode( + q.view(num_tokens, core.heads, core.head_dim), + k_paged, + v_paged, + topk, + block_table, + seq_lens, + sm_scale=core.sm_scale(i), + output=output.view(num_tokens, core.heads, core.head_dim), + decode_query_len=qlen, + identity_blocks=identity, + **nvfp4, + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# Speculative decoding: one-model Eagle3 at checkpoint precision +# -------------------------------------------------------------------------------------------------------------------- + + +class M3SpecStepFold: + """The target's speculative step-input staging (the in-tree ``SpecStepFold`` protocol). + + Between two decode graphs of a one-model Eagle3 step, the spec sampler stores the step's outputs into its per-slot + store and the engine gathers them back into its flat input buffers for the next step. This object does both in + one launch each (``m3_spec_tail.slot_scatter`` / ``step_gather``): integer copies only, the same values the + engine's and the sampler's own copies write. The forward never stores the sampler's outputs itself, so + ``begin_step`` and ``sampler_rows_stored`` always report that nothing was staged. + """ + + def __init__(self) -> None: + self.store: Optional[object] = None + self.step_inputs: Optional[tuple] = None + + def bind_engine_buffers( + self, input_ids, draft_tokens, position_id_offsets, kv_lens_offsets, max_batch_size + ): + self.step_inputs = (input_ids, draft_tokens, position_id_offsets, kv_lens_offsets) + + def bind_sampler_store(self, store: object) -> None: + self.store = store + + def begin_step(self, seq_slots: Optional[list], num_rows: int) -> bool: + return False + + def gather( + self, + new_tokens, + next_draft_tokens, + new_tokens_lens, + previous_slots, + previous_pos_indices, + previous_batch_len: int, + tokens_per_gen_step: int, + draft_token_buffer_width: int, + num_tokens: int, + num_draft_tokens: int, + first_previous_row: int, + ) -> bool: + if self.step_inputs is None: + return False + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets = self.step_inputs + return m3_spec_tail.step_gather( + new_tokens, + next_draft_tokens, + new_tokens_lens, + previous_slots, + previous_pos_indices, + previous_batch_len, + tokens_per_gen_step, + draft_token_buffer_width, + input_ids, + num_tokens, + draft_tokens, + num_draft_tokens, + position_id_offsets, + kv_lens_offsets, + first_previous_row, + ) + + def sampler_rows_stored(self, outputs: dict, num_skip: int, seq_slots: list) -> bool: + return False + + def scatter( + self, seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens + ) -> bool: + store = self.store + if store is None: + return False + return m3_spec_tail.slot_scatter( + seq_slots, + new_tokens, + store.new_tokens.squeeze(-1).T, + next_new_tokens, + store.next_new_tokens.squeeze(-1).T, + new_tokens_lens, + store.new_tokens_lens, + next_draft_tokens, + store.next_draft_tokens, + ) + + +class M3DSparkStepFold(M3SpecStepFold): + """The step fold under the standalone DSpark worker (DSFOLD): the engine's previous-batch gathers in one launch + (``step_gather``). ``taken`` / ``declined`` count the steps the gather ran / fell back to the engine's own copies. + + With ``boundary_pack`` (BOUNDARY_PACK BP-1) a batch-1 decode graph also ends with the sampler's slot stores and the + next step's gather (``graph_tail``: one ``ops/m3_bp_tail`` launch), so the boundary between two replays runs + neither: + + * ``begin_step`` (before every replay) writes the device control word (slot, live) when it changed: live 1 with + the step's slot when the step is one row the fold identifies (a slot inside the store), else live 0, so a replay + the fold did not take (a dummy-only replay, an unidentified step) writes nothing at all; it reports the inputs + staged when the previous graph gathered them for exactly these rows; + * the graph's tail waits for the sampler's previous side-stream store copies (``store_copies_done``), then with live + 1 stores the row as the sampler's ``_store_outputs`` (same output row, same store, zero padding / truncation to + the store widths, at the slot) and, with a static draft length, writes the engine's next-step gathers of that row + (P = 1, ids0 = d0 = b0 = 0, both offset buffers zeroed elsewhere), and marks its outputs; + * ``sampler_rows_stored`` (after the replay) accepts a marked step whose rows are the live ones, and arms the next + ``begin_step``. + + Every other step (context or mixed steps, batches over one row, eager forwards, a draft length that can change, + first steps, slots the fold could not identify) keeps the sampler's stores and the engine's gathers. + ``bp_stores`` / ``bp_gathers`` count the steps whose stores / gathers the graph took.""" + + store_copies_done: Optional[torch.cuda.Event] = None + + def __init__(self, boundary_pack: bool = False, tokens_per_step: Optional[int] = None) -> None: + super().__init__() + self.taken = 0 + self.declined = 0 + self.boundary_pack = boundary_pack + # T of every decode step (the draft length cannot change at runtime), else None: the graph then stores only. + self.tokens_per_step = tokens_per_step + self.bp_stores = 0 + self.bp_gathers = 0 + self._bp_ctl: Optional[torch.Tensor] = None # int64 [2] on the device: (slot, live) + self._bp_ctl_host: Optional[list] = None # what it holds (the last write) + self._bp_block = 0 # the tail's row block (m3_bp_tail.block_for the store) + self._bp_rows: Optional[list] = None # the step's rows while live, else None + self._bp_staged: Optional[list] = None # the rows the last stored step's graph gathered for + self._bp_token = id(self) + + def bind_engine_buffers( + self, input_ids, draft_tokens, position_id_offsets, kv_lens_offsets, max_batch_size + ): + super().bind_engine_buffers( + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets, max_batch_size + ) + self._bp_reset() + + def bind_sampler_store(self, store: object) -> None: + super().bind_sampler_store(store) + self._bp_reset() + + def _bp_reset(self) -> None: + """A new binding (an engine or a sampler built again): BOUNDARY_PACK's buffers are made again for it.""" + self._bp_ctl = self._bp_ctl_host = None + self._bp_rows = self._bp_staged = None + self.store_copies_done = None + + def gather(self, *args, **kwargs) -> bool: + if not super().gather(*args, **kwargs): + self.declined += 1 + uncovered( + "DSFOLD", + "step_gather declined the inputs (dtype / layout); the engine's copies run", + ) + return False + if self.taken == 0: + logger.info( + "[M3 dispatch] DSFOLD DSpark step-input gather: engaged (one step_gather launch)" + ) + self.taken += 1 + return True + + def scatter( + self, seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens + ) -> bool: + return False + + def _bp_ready(self) -> bool: + """BOUNDARY_PACK: the control word (live 0), the tail kernel's two variants and the store-copies event, made + once outside capture when the sampler store and the engine buffers are bound and the tail can take them.""" + if not self.boundary_pack: + return False + if self._bp_ctl is not None: + return True + store = self.store + names = ("new_tokens", "next_new_tokens", "new_tokens_lens", "next_draft_tokens") + if ( + self.step_inputs is None + or not all(isinstance(getattr(store, name, None), torch.Tensor) for name in names) + or torch.cuda.is_current_stream_capturing() + ): + return False + tensors = [getattr(store, name) for name in names] + list(self.step_inputs) + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in tensors) + or not all(t.is_contiguous() for t in self.step_inputs) + or not m3_bp_tail.aligned(*tensors) + ): + uncovered( + "BOUNDARY_PACK", + "the store or the engine buffers are not aligned int32 CUDA tensors", + ) + return False + ctl = torch.zeros(2, dtype=torch.long, device=store.new_tokens_lens.device) + block = m3_bp_tail.block_for(store) + m3_bp_tail.warm_up(ctl, store, self.step_inputs, block) + # Recorded by the sampler after its side-stream store copies; the graph's tail waits on it. A CUDA event is + # created on its first record and a wait on one never created is skipped (a capture would drop the wait). + done = torch.cuda.Event(external=True) + done.record() + self.store_copies_done = done + self._bp_ctl, self._bp_ctl_host, self._bp_block = ctl, [0, 0], block + return True + + def begin_step(self, seq_slots: Optional[list], num_rows: int) -> bool: + staged, self._bp_staged = self._bp_staged, None + self._bp_rows = None + if not self._bp_ready(): + return False + num_slots = self.store.new_tokens_lens.shape[0] + rows = list(seq_slots) if seq_slots is not None else None + live = ( + rows is not None + and len(rows) == num_rows + and 0 < num_rows <= BOUNDARY_PACK_MAX_ROWS + and all(isinstance(slot, int) and 0 <= slot < num_slots for slot in rows) + ) + # Live: this step's slot. Not live: the graph (if one replays) writes nothing; the slot word keeps its value. + ctl = [rows[0], 1] if live else [self._bp_ctl_host[0], 0] + if ctl != self._bp_ctl_host: + # Ordered after the previous graph's tail (which read the last word) and before this step's replay. + self._bp_ctl.copy_( + torch.tensor(ctl, dtype=torch.long, pin_memory=prefer_pinned()), non_blocking=True + ) + self._bp_ctl_host = ctl + if not live: + return False + self._bp_rows = rows + if staged != rows: + return False + if self.bp_gathers == 0: + logger.info( + "[M3 dispatch] BOUNDARY-PACK BP-1: sampler stores + step gather in the step graph (taken)" + ) + self.bp_gathers += 1 + return True + + def sampler_rows_stored(self, outputs: dict, num_skip: int, seq_slots: list) -> bool: + self._bp_staged = None + mark = outputs.get(_BOUNDARY_PACK_MARK) if isinstance(outputs, dict) else None + if mark is None or mark[0] != self._bp_token or self._bp_rows is None: + # Unmarked, or the replay was not live (it wrote nothing): the sampler stores. + return False + rows, gathered = mark[1], mark[2] + if num_skip != 0 or len(seq_slots) != rows or list(seq_slots) != self._bp_rows: + # The graph stored the live rows, which are not the sampler's: it stores its own. + uncovered( + "BOUNDARY_PACK", + "a step graph stored rows that are not the sampler's (skipped context rows or other slot ids); " + "the sampler stores them", + ) + return False + self.bp_stores += 1 + if gathered: + self._bp_staged = list(seq_slots) + return True + + def graph_tail(self, outputs, attn_metadata): + """BOUNDARY_PACK, at the end of a decode step's forward: under capture of a batch-1 generation graph, the + sampler's slot stores and the next step's gather, gated by the live word (see the class docstring); returns + the outputs, marked when the graph has the tail. Anything else returns them unchanged (the eager path).""" + if not self.boundary_pack or not torch.cuda.is_current_stream_capturing(): + return outputs + path = "BOUNDARY_PACK" + if not isinstance(outputs, dict) or self._bp_ctl is None: + uncovered( + path, + "the sampler store or the engine buffers are not bound: eager stores and gathers", + ) + return outputs + if attn_metadata.num_contexts != 0: + return outputs + names = ("new_tokens", "next_new_tokens", "new_tokens_lens", "next_draft_tokens") + srcs = [outputs.get(name) for name in names] + if any( + not isinstance(t, torch.Tensor) or t.dtype != torch.int32 or not t.is_cuda for t in srcs + ) or [t.dim() for t in srcs] != [2, 2, 1, 2]: + uncovered( + path, + "the worker outputs are not int32 [rows, width] / [rows]: eager stores and gathers", + ) + return outputs + rows = srcs[0].shape[0] + if any(t.shape[0] != rows for t in srcs): + uncovered(path, "the worker outputs disagree on their rows: eager stores and gathers") + return outputs + if not 0 < rows <= BOUNDARY_PACK_MAX_ROWS: + uncovered( + path, + f"a {rows}-row decode graph (only batch-1 graphs, which are never padded): eager stores and gathers", + ) + return outputs + src = [t[:1] for t in srcs] + if not m3_bp_tail.aligned(*src): + uncovered(path, "a worker output is not 16-byte aligned: eager stores and gathers") + return outputs + store = self.store + # The next step's gather for the same row: the engine's host ints of a steady step (no rows before it, no + # padding), T fixed by the static draft length. + num_tokens = int(attn_metadata.num_tokens) + t = num_tokens // rows + gather = ( + self.tokens_per_step is not None + and t == self.tokens_per_step + and num_tokens == rows * t + and t <= store.next_new_tokens.shape[0] + and t - 1 <= store.next_draft_tokens.shape[1] + ) + if not gather: + uncovered( + path, + f"the next step's tokens per request may differ from this graph's {t} (a draft length that can " + f"change, or {num_tokens} tokens over {rows} rows): the graph stores, the engine gathers", + ) + # The previous step's side-stream copies of the store have read it (an external wait node). + torch.cuda.current_stream().wait_event(self.store_copies_done) + m3_bp_tail.bp_tail(self._bp_ctl, src, store, self.step_inputs, t, gather, self._bp_block) + marked = dict(outputs) + marked[_BOUNDARY_PACK_MARK] = (self._bp_token, rows, gather) + return marked + + +def _static_tokens_per_step(spec_config) -> Optional[int]: + """BOUNDARY_PACK: the tokens per request of every decode step when the draft length cannot change at runtime (no + draft-length schedule, no acceptance-rate gate that turns speculation off), else None.""" + if getattr(spec_config, "draft_len_schedule", None) is not None: + return None + if ( + getattr(spec_config, "acceptance_rate_window_size", None) + and getattr(spec_config, "acceptance_rate_threshold", None) is not None + ): + return None + from tensorrt_llm._torch.speculative.utils import get_static_draft_len + + return int(spec_config.get_runtime_tokens_per_gen_step(get_static_draft_len(spec_config))) + + +class _DraftForward: + """``draft_model.model`` for the in-tree worker: the drafter layer's forward, called as + ``model(input_ids=, position_ids=, hidden_states=, attn_metadata=, spec_metadata=, all_rank_num_tokens=)`` and + returning (hidden, hidden to carry to the next pass). ``d2t`` None: the draft vocabulary is the target's.""" + + d2t = None + + def __init__(self, draft: "M3Eagle3Draft") -> None: + self._draft = draft + + def __call__(self, **kwargs): + return self._draft.forward_layer(**kwargs) + + +class M3Eagle3Draft(nn.Module): + """The one-layer BF16 Eagle3 drafter (``speculative_model``): its weights (``weights.DRAFT_WEIGHTS``), its + vocab-parallel BF16 head and logits processor, the drafter layer's TRTLLM attention backend (layer index = the + target's layer count, FP8 KV) and its TP all-reduce. + + What the worker reads: ``model`` (the layer forward), ``apply_eagle3_fc`` (the per-layer fc_norms + fc over the + captured target hidden states), ``lm_head`` and ``logits_processor``; the fused draft loop reads the layer's parts + (``draft_input_weights``, ``attention``, ``mlp``, ``ar_norm``). The drafter embeds with the target's table. + """ + + def __init__( + self, + core: MiniMaxM3Tp4ModelingV2, + draft_config: ModelConfig, + start_layer_idx: int, + capture_layers: int, + ) -> None: + super().__init__() + cfg = draft_config.pretrained_config + _ensure_draft_vocab_size(cfg) + eagle = getattr(cfg, "eagle_config", None) or {} + mapping = draft_config.mapping + self.draft_config = draft_config + # The engine's draft weight mapper binds to these two. + self.model_config = draft_config + self.config = cfg + self.capture_layers = int(capture_layers) + self.layer_idx = int(start_layer_idx) + self.dtype = cfg.torch_dtype + self.hidden = int(cfg.hidden_size) + self.hidden_size_in = int(getattr(cfg, "target_hidden_size", cfg.hidden_size)) + self.rms_eps = float(cfg.rms_norm_eps) + self.return_hidden_post_norm = bool( + eagle.get("return_hidden_post_norm", False) or getattr(cfg, "norm_output", False) + ) + if ( + int(cfg.num_hidden_layers) != 1 + or eagle.get("eh_proj_before_attn", False) + or eagle.get("norm_before_fc", False) + or not getattr(cfg, "fc_norm", False) + or getattr(cfg, "kv_lora_rank", None) + or self.capture_layers <= 1 + or self.hidden_size_in != self.hidden + or cfg.draft_vocab_size != cfg.vocab_size + or getattr(cfg, "attention_bias", False) + or getattr(cfg, "mlp_bias", False) + or getattr(cfg, "bias", False) + or getattr(cfg, "tie_word_embeddings", False) + ): + raise ValueError( + "this MiniMax-M3 target drafts with a one-layer Llama-style Eagle3 drafter with per-layer " + "fc_norms over " + "several captured layers, the target's hidden size and vocabulary and no biases" + ) + tp = mapping.tp_size + self.heads = cfg.num_attention_heads // tp + self.kv_heads = (cfg.num_key_value_heads + tp - 1) // tp + self.head_dim = int( + getattr(cfg, "head_dim", None) or cfg.hidden_size // cfg.num_attention_heads + ) + self.q_size = self.heads * self.head_dim + self.kv_size = self.kv_heads * self.head_dim + self.w = _weights.DRAFT_WEIGHTS.declare(_weights.DRAFT_WEIGHTS.dims(self)) + self.lm_head = LMHead( + cfg.draft_vocab_size, + cfg.hidden_size, + dtype=cfg.torch_dtype, + mapping=mapping, + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=draft_config.lm_head_gather_output, + reduce_output=False, + use_custom_cublas_mm=False, + quant_config=None, + ) + self.logits_processor = LogitsProcessor() + self.allreduce = AllReduce( + mapping=mapping, strategy=draft_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + quant_config = draft_config.get_quant_config() + self.attn = create_attention( + draft_config.attn_backend, + self.layer_idx, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=PositionalEmbeddingParams( + type=PositionEmbeddingType.rope_gpt_neox, rope=RopeParams.from_config(cfg) + ), + quant_config=quant_config, + skip_create_weights_in_init=draft_config.skip_create_weights_in_init, + q_scaling=1.0, + attention_chunk_size=None, + sparse_params=None, + ) + if not self.attn.support_fused_rope() or not self.attn.support_fused_qkv(): + raise ValueError( + f"the drafter's attention backend {type(self.attn).__name__} must fuse RoPE and QKV" + ) + self.attn.update_quant_config(quant_config) + # The drafter embeds with the target's table, read off the core on every call (the engine materializes the + # core's weights after construction). + self._core = [core] + self.model = _DraftForward(self) + + @property + def embed_table(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def load_weights_from_target_model(self, target_model: nn.Module) -> None: + """Nothing to alias: the drafter reads the target's embedding table through the core and owns its head.""" + + # ------------------------------------------------------------------------------------------------- the layer + + def fc(self, hs: torch.Tensor) -> torch.Tensor: + return m3_bf16_linear(hs, self.w["fc"]) + + def apply_eagle3_fc(self, hidden_states: torch.Tensor) -> torch.Tensor: + """The captured target hidden states [T, layers x H] to the drafter's input width: each layer's fc_norm, + then the fc (the in-tree ``Eagle3ForCausalLM.apply_eagle3_fc``).""" + hidden_states = hidden_states.to(self.dtype) + if hidden_states.shape[-1] != self.hidden: + chunks = hidden_states.chunk(self.capture_layers, dim=-1) + hidden_states = torch.cat( + [ + flashinfer_rmsnorm(c, self.w[f"fc_norm{i}"], self.rms_eps) + for i, c in enumerate(chunks) + ], + dim=-1, + ) + hidden_states = self.fc(hidden_states) + return hidden_states + + def attention(self, x: torch.Tensor, md, reduce: bool) -> torch.Tensor: + """QKV, the backend's fused RoPE + KV append + causal attention, o_proj (all-reduced unless ``reduce`` is + False, where the caller fuses the reduction).""" + qkv = m3_bf16_linear(x, self.w["qkv"]) + _record("m3_draft_attention", int(md.num_tokens)) + o = m3_draft_attention(self.attn, qkv, md) + out = m3_bf16_linear(o, self.w["o"]) + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def mlp(self, h: torch.Tensor, reduce: bool) -> torch.Tensor: + gu = m3_bf16_linear(h, self.w["gu"]) + act = m3_silu_and_mul(gu, None, None, None) + out = m3_bf16_linear(act, self.w["dn"]) + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def ar_norm(self, x: torch.Tensor, residual: torch.Tensor, norm_key: str): + """One draft-layer boundary in the in-tree structure: the projection's plain all-reduce (BF16 out), then the + fused residual add + RMSNorm (FP32 add, norm of the FP32 sum). Returns (normed, residual sum).""" + return flashinfer_fused_add_rmsnorm( + m3_allreduce(x, self.allreduce, None), residual, self.w[norm_key], self.rms_eps + ) + + def forward_layer( + self, + input_ids: torch.Tensor, + position_ids: torch.Tensor, + hidden_states: torch.Tensor, + attn_metadata, + spec_metadata, + all_rank_num_tokens: Optional[List[int]] = None, + ): + """The drafter layer (the in-tree ``Eagle3DraftModel.forward`` for one layer): the embedded tokens and the + fc'd hidden states normed and concatenated, attention, the post-attention norm with the residual, the MLP and + the final norm. Returns (normed output, the hidden state the next pass carries).""" + previous = attn_metadata.all_rank_num_tokens + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = all_rank_num_tokens + try: + embeds = m3_embedding(input_ids, self.embed_table).to(self.dtype) + residual = hidden_states + h = flashinfer_rmsnorm(hidden_states, self.w["hidden_norm"], self.rms_eps) + e = flashinfer_rmsnorm(embeds, self.w["in_norm"], self.rms_eps) + h = torch.cat([e, h], dim=-1) + h = self.attention(h, attn_metadata, reduce=True) + h, residual = flashinfer_fused_add_rmsnorm( + h, residual, self.w["post_norm"], self.rms_eps + ) + h = self.mlp(h, reduce=True) + spec_metadata.maybe_capture_hidden_states(self.layer_idx, h, residual) + h, to_save = flashinfer_fused_add_rmsnorm( + h, residual, self.w["final_norm"], self.rms_eps + ) + return (h, h) if self.return_hidden_post_norm else (h, to_save) + finally: + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = previous + + +def _eagle3_capture(spec_metadata, layer_id: int, hidden_states, residual, capture_norm) -> None: + """The Eagle3 hidden-state capture of one target layer. + + Mirrors ``Eagle3OneModelSpecMetadata.maybe_capture_hidden_states``. With ``capture_norm`` = (the drafter's stacked + fc_norm weights [layers, H], eps), the capture writes rmsnorm(bf16(h + residual)) * fc_norm[i] in one kernel (the + in-tree capture's bf16 add, then the in-tree fc_norm arithmetic), which the fused first drafter inputs read as is; + otherwise it is the in-tree capture. + """ + if capture_norm is None: + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + return + for i, captured_layer_id in enumerate(spec_metadata.layers_to_capture): + if captured_layer_id == layer_id: + assert spec_metadata.hidden_states is not None + num_tokens = spec_metadata.num_capture_tokens + if num_tokens > spec_metadata.hidden_states.shape[0]: + raise RuntimeError( + "EAGLE3 hidden-state capture token count exceeds buffer capacity: " + f"num_tokens={num_tokens}, capacity={spec_metadata.hidden_states.shape[0]}" + ) + if num_tokens > hidden_states.shape[0]: + raise RuntimeError( + "EAGLE3 hidden-state capture token count exceeds available hidden states: " + f"num_tokens={num_tokens}, hidden_states={hidden_states.shape[0]}" + ) + hidden_size = spec_metadata.hidden_size + m3_spec_tail.capture_norm( + hidden_states[:num_tokens], + residual[:num_tokens] if residual is not None else None, + capture_norm[0][i], + capture_norm[1], + spec_metadata.hidden_states[:num_tokens, i * hidden_size : (i + 1) * hidden_size], + ) + break + + +def _sharded_head(lm_head: nn.Module) -> bool: + """A vocab-sharded TP head without padding (the fused kernels read the local shard).""" + return ( + getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and getattr(lm_head, "padding_size", 0) == 0 + and getattr(lm_head, "tp_size", 1) > 1 + ) + + +class M3Eagle3Worker(Eagle3OneModelWorker): + """``Eagle3OneModelWorker`` with the MiniMax-M3 fused verify and fused draft loop, at the in-tree numerics. + + * Verify: on decode-only batches whose verify rows the fused kernels cover (``tail_eligible``), the target's BF16 + lm_head computes this rank's vocabulary shard of the logits (the in-tree head's own local GEMM; at most + ``TARGET_HEAD_MAX_ROWS`` verify rows stream a lossless 12-bit coding of the same BF16 weight) and one kernel + replaces the logits all-gather, the argmax (or the temperature + top-p draw) and the strict acceptance. + * Draft: the drafter's first pass may take the per-layer fc_norms in one launch (or folded into the hidden-state + capture) and run its fc beside the target head GEMM (forked right after the last captured target layer at most + ``FC_EARLY_MAX_ROWS`` verify rows). On a decode-only step ``DRAFT_STEP`` covers, the draft loop is fused: each + pass's draft input in one launch, the drafter layer's attention and MLP with their all-reduces fused with the + residual add and the next RMSNorm, and one kernel for the BF16 head, the cross-rank argmax and the pass's + bookkeeping. + + Every other batch takes the in-tree paths. Methods whose docstring says "Mirrors" copy the named in-tree method + with the fused branches inserted. + """ + + def __init__(self, spec_config, mapping, use_separate_draft_kv_cache: bool = False) -> None: + super().__init__( + spec_config, mapping=mapping, use_separate_draft_kv_cache=use_separate_draft_kv_cache + ) + # Per-step state, reset at the top of every _forward_impl: the first draft pass's gather rows when this step + # takes the fused first drafter inputs, and the fc output a fork computed beside the target head. + self._m3_first_gather_ids: Optional[torch.Tensor] = None + self._m3_fc_hidden: Optional[torch.Tensor] = None + self._m3_target_lm_head: Optional[nn.Module] = None + # (fc_norm0 weight, stacked fc_norm weights [layers, H]), built eagerly on the first fused first pass. + self._m3_fc_norm_w: Optional[tuple] = None + # Armed per step by arm_capture_norm: (stacked fc_norm weights, eps) or None. + self._m3_capture_norm: Optional[tuple] = None + self._m3_aux_stream: Optional[torch.cuda.Stream] = None + # Armed per step by arm_capture_norm: the drafter whose first fc is forked at the last captured target layer, + # and the fork that did it, joined by _m3_target_accept. + self._m3_fc_early_drafter: Optional[M3Eagle3Draft] = None + self._m3_fc_early_stream: Optional[torch.cuda.Stream] = None + # The 12-bit coded copy of the target head shard the fused verify streams, built before the warmup. + self._m3_target_head = None + self._m3_force_vec_warm = False + # The streamed copy of the drafter's BF16 head shard the fused draft loop reads, built before the warmup. + self._m3_draft_head = None + # Per step: the next step's input tokens [nreq, D + 1], column 0 written by the fused verify (or the seed) and + # columns 1.. by the fused draft passes; set as _m3_next_new_tokens once the fused draft loop filled it. + self._m3_next_new: Optional[torch.Tensor] = None + self._m3_next_new_tokens: Optional[torch.Tensor] = None + # Per step: the first draft inputs are the fused ones (accepted tokens as they lie, fc output of every row). + self._m3_fused_first_inputs = False + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None + ) -> None: + """The Eagle3 capture with this step's capture norm; on a step armed for the early fc fork + (``arm_capture_norm``), the last captured layer then forks the drafter's first fc onto the aux stream, where + it overlaps the remaining target layers and the head (joined by ``_m3_target_accept``).""" + _eagle3_capture(spec_metadata, layer_id, hidden_states, residual, self._m3_capture_norm) + drafter = self._m3_fc_early_drafter + layers = spec_metadata.layers_to_capture + if drafter is not None and layers and layer_id == layers[-1]: + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = self._m3_fc_fork( + spec_metadata, spec_metadata.num_capture_tokens, drafter + ) + + def prebuild(self, lm_head: nn.Module, draft: Optional[M3Eagle3Draft]) -> None: + """The fused verify's coded target head and the fused draft loop's head copy, from the loaded weights, + before the executor's warmup, on every rank.""" + start = time.perf_counter() + if self.prebuild_target_head(lm_head): + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] coded target head built before warmup: %.1f s" + % (time.perf_counter() - start) + ) + start = time.perf_counter() + if self.prebuild_draft_head(draft): + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] fused draft head built before warmup: %.1f s" + % (time.perf_counter() - start) + ) + + def _forward_impl( + self, + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + resource_manager=None, + ): + """Mirrors Eagle3OneModelWorker._forward_impl, with the fused verify branch.""" + runtime_draft_len = spec_metadata.runtime_draft_len + # skip the draft forward if the runtime draft length is 0 + if runtime_draft_len == 0: + return self.skip_drafting( + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + ) + + batch_size = attn_metadata.num_seqs + num_contexts = attn_metadata.num_contexts + num_gens = batch_size - num_contexts + + raw_logits = logits + + self._m3_first_gather_ids = None + self._m3_next_new = None + self._m3_next_new_tokens = None + self._m3_fused_first_inputs = False + if logits is None: + # tail_eligible() skipped the gathered target logits: verify from this rank's logits shard. + accepted_tokens, num_accepted_tokens, self._m3_first_gather_ids = ( + self._m3_target_accept( + hidden_states, spec_metadata, batch_size, runtime_draft_len, draft_model + ) + ) + else: + self._execute_guided_decoder_if_present(logits) + + # Sample and accept tokens. ``input_ids`` is required by the relaxed-acceptance path (scans for + # thinking-phase tokens); ignored otherwise. + accepted_tokens, num_accepted_tokens = self.sample_and_accept_draft_tokens( + input_ids, logits, attn_metadata, spec_metadata + ) + if self.draft_only_eligible(attn_metadata, spec_metadata, draft_model): + self._m3_seed_first_pass(num_accepted_tokens, spec_metadata, batch_size) + + # Mamba hybrid models need state updates after token acceptance (a no-op on this target's cache manager). + if self._is_mamba_hybrid_cache is None: + self._is_mamba_hybrid_cache = isinstance( + attn_metadata.kv_cache_manager, MambaHybridCacheManager + ) + if num_gens > 0 and self._is_mamba_hybrid_cache: + attn_metadata.kv_cache_manager.update_mamba_states( + attn_metadata=attn_metadata, + num_accepted_tokens=num_accepted_tokens, + state_indices=attn_metadata.mamba_metadata.state_indices, + ) + + sa_manager = getattr(spec_metadata.spec_resource_manager, "sa_manager", None) + if self.sa_enhancer is not None and sa_manager is not None: + self.sa_enhancer.extend_and_prepare( + sa_manager=sa_manager, + request_ids=spec_metadata.request_ids, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_gens=num_gens, + num_contexts=num_contexts, + max_draft_len=runtime_draft_len, + ) + + if self._uses_external_shared_target_kv: + next_draft_tokens = self._forward_external_shared_target_kv_draft_loop( + position_ids=position_ids, + hidden_states=hidden_states, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_contexts=num_contexts, + batch_size=batch_size, + ) + else: + # Save the old attn_metadata and spec_metadata + self._prepare_attn_metadata_for_spec_dec(attn_metadata) + + # Prepare inputs for the 1st draft model forward + position_ids = position_ids.squeeze(0) + inputs = self.prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + + # Predict draft tokens. ``original_all_rank_num_tokens`` is saved here so the post-loop restore (below) + # can put attn_metadata back into a state the target model expects. + original_all_rank_num_tokens = attn_metadata.all_rank_num_tokens + + # Get the draft KV cache manager if using separate layouts + draft_kv_cache_manager = self.get_draft_kv_cache_manager(resource_manager) + + next_draft_tokens = self._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + # restore attn_metadata to support cuda graph + self._restore_attn_metadata_from_spec_dec(attn_metadata) + # restore all_rank_num_tokens for attention DP + if original_all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = original_all_rank_num_tokens + + # prepare next new tokens to support overlap scheduler + if self._m3_next_new_tokens is not None: + # Filled in place by the fused verify or the seed (column 0) and the fused draft passes (columns 1..). + next_new_tokens = self._m3_next_new_tokens + else: + next_new_tokens = self._prepare_next_new_tokens( + accepted_tokens, + next_draft_tokens, + spec_metadata.batch_indices_cuda, + batch_size, + num_accepted_tokens, + ) + + attn_metadata.use_spec_decoding = True + + # Commit model-owned target state only after draft preparation has succeeded; SpecWorkerBase aborts the + # still-pending snapshots on any exception above this point. + if self._auxiliary_state_handlers: + self.commit_auxiliary_speculative_states( + num_accepted_tokens, + attn_metadata.mamba_metadata.state_indices[:batch_size], + num_contexts, + ) + + return { + "logits": raw_logits, + "new_tokens": accepted_tokens, + "new_tokens_lens": num_accepted_tokens, + "next_draft_tokens": next_draft_tokens, + "next_new_tokens": next_new_tokens, + } + + def prepare_1st_drafter_inputs( + self, + input_ids, + position_ids, + hidden_states, + accepted_tokens, + attn_metadata, + spec_metadata, + draft_model, + ): + """The fused first draft inputs on a step that verified with the fused kernels (or was seeded for them), + else the in-tree ones.""" + fused = self._m3_first_drafter_inputs( + position_ids, + accepted_tokens, + input_ids.shape[0], + attn_metadata.num_contexts, + spec_metadata, + draft_model, + ) + self._m3_fused_first_inputs = fused is not None + if fused is not None: + fused.update(attn_metadata=attn_metadata, spec_metadata=spec_metadata) + return fused + return super().prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + + def _m3_structural_ok(self, attn_metadata, spec_metadata) -> bool: + """The batch conditions shared by the fused verify and the fused first drafter inputs: a decode-only plain + Eagle3 step without attention DP, guided decoding, relaxed acceptance or a suffix-automaton enhancer, of at + most DECODE_MAX_TOKENS verify rows. Host state only, so the TP ranks agree.""" + if self.is_mtp_eagle or self.use_dynamic_tree: + return False + if self._uses_external_shared_target_kv or self.sa_enhancer is not None: + return False + if ( + self.guided_decoder is not None + or self.mapping is None + or self.mapping.enable_attention_dp + ): + return False + if getattr(self.spec_config, "use_relaxed_acceptance_for_thinking", False): + return False + if attn_metadata.num_contexts != 0 or attn_metadata.num_seqs == 0: + return False + draft_len = spec_metadata.runtime_draft_len + if draft_len == 0: + return False + if attn_metadata.num_seqs * (draft_len + 1) > DECODE_MAX_TOKENS: + uncovered("spec_tail", f"more than {DECODE_MAX_TOKENS} verify rows") + return False + return not spec_metadata.wants_advanced_draft_sampling + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Whether this step verifies with the fused kernels instead of the gathered target logits. + + Decode-only Eagle3 batches (``_m3_structural_ok``) over an unpadded vocab-sharded bias-free target head, all + greedy at a request count ``SHARD_ACCEPT`` covers, or sampled under the temperature + top-p contract (see + _m3_sampled_verify_ok), with nothing else reading the target logits. Every input is host state shared by all + TP ranks, so the ranks agree. + """ + if not self._m3_structural_ok(attn_metadata, spec_metadata): + return False + if getattr(spec_metadata, "enable_penalty", False): + return False + if not _sharded_head(lm_head) or getattr(lm_head, "bias", None) is not None: + return False + draft_len = spec_metadata.runtime_draft_len + if spec_metadata.is_all_greedy_sample: + key = (draft_len, attn_metadata.num_seqs) + if not SHARD_ACCEPT.covers(key): + uncovered( + SHARD_ACCEPT.path, + f"draft length {draft_len} at {attn_metadata.num_seqs} requests", + ) + return False + elif attn_metadata.num_seqs * (draft_len + 1) > m3_sampled_accept_kernel.MAX_ROWS: + uncovered( + "spec_tail", + f"a sampled verify of more than {m3_sampled_accept_kernel.MAX_ROWS} rows", + ) + return False + elif not self._m3_sampled_verify_ok(spec_metadata): + return False + self._m3_target_lm_head = lm_head + return True + + def draft_only_eligible(self, attn_metadata, spec_metadata, draft_model) -> bool: + """Whether a step verified by the in-tree path can still take the fused first drafter inputs: the structural + conditions of tail_eligible, without the target-sampling ones.""" + return self._m3_structural_ok(attn_metadata, spec_metadata) and draft_model is not None + + def _m3_seed_first_pass(self, num_accepted_tokens, spec_metadata, batch_size: int) -> None: + """The first draft pass's gather rows (the last accepted row of each request) from the in-tree acceptance, as + _m3_target_accept leaves them.""" + k1 = spec_metadata.runtime_draft_len + 1 + n = num_accepted_tokens[:batch_size].to(torch.int64) + rows = spec_metadata.batch_indices_cuda[:batch_size].to(torch.int64) + self._m3_first_gather_ids = rows * k1 + n - 1 + + def _m3_first_drafter_inputs( + self, + position_ids, + accepted_tokens, + num_tokens: int, + num_contexts: int, + spec_metadata, + draft_model: Optional[M3Eagle3Draft], + ): + """The in-tree first draft inputs of a decode-only step, with the per-layer fc_norm RMSNorms of + ``apply_eagle3_fc`` in one kernel (unless the capture already applied them) before the fc; the input ids are + the accepted tokens as they lie. None to use the in-tree path.""" + if ( + self._m3_first_gather_ids is None + or self.is_mtp_eagle + or num_contexts != 0 + or draft_model is None + ): + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + ): + return None + cache = self._m3_fc_norm_w + if cache is None or cache[0] is not draft_model.w["fc_norm0"]: + if torch.cuda.is_current_stream_capturing(): + capture_missed("spec_tail", None, "the stacked Eagle3 fc_norm weights") + return None + cache = ( + draft_model.w["fc_norm0"], + torch.stack( + [draft_model.w[f"fc_norm{i}"].detach() for i in range(layers)] + ).contiguous(), + ) + self._m3_fc_norm_w = cache + hidden = self._m3_fc_hidden + self._m3_fc_hidden = None + if hidden is None or hidden.shape[0] != num_tokens: + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + hs = m3_spec_tail.chunk_norm(hs, cache[1], draft_model.rms_eps) + hidden = draft_model.fc(hs) + input_ids = accepted_tokens[:, : spec_metadata.runtime_draft_len + 1].reshape(-1) + return {"input_ids": input_ids, "position_ids": position_ids, "hidden_states": hidden} + + def _m3_fc_fork( + self, + spec_metadata, + num_tokens: int, + draft_model: Optional[M3Eagle3Draft], + after: Optional[torch.cuda.Event] = None, + ) -> Optional[torch.cuda.Stream]: + """Launch the first draft pass's fc_norm + fc on a forked stream beside the target head GEMM (after the event + ``after`` when given, else after everything queued on the current stream); the result is picked up by + _m3_first_drafter_inputs. Returns the stream to join, or None when the fc path does not apply (the first + drafter inputs then compute it).""" + self._m3_fc_hidden = None + if draft_model is None: + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + or self._m3_fc_norm_w is None + ): + return None + aux = self._m3_aux_stream + if aux is None: + aux = self._m3_aux_stream = torch.cuda.Stream() + if after is not None: + aux.wait_event(after) + else: + aux.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(aux): + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + hs = m3_spec_tail.chunk_norm( + hs, self._m3_fc_norm_w[1], draft_model.rms_eps, use_pdl=False + ) + self._m3_fc_hidden = draft_model.fc(hs) + return aux + + def _m3_fc_join(self, fork: torch.cuda.Stream) -> None: + """Join the fc fork onto the current stream and keep its output's memory alive for that stream. + + _m3_fc_hidden is allocated in fork (aux) stream order and read on the current stream. Without the record, + dropping it returns its block to the aux stream's pool, where later aux work can take it while the current + stream's reads are still queued (eager warm-up / mixed steps). Nothing is recorded under graph capture, as in + maybe_execute_in_parallel: a captured graph relies on the next fork's aux.wait_stream(main) to order the + reuse.""" + main = torch.cuda.current_stream() + main.wait_stream(fork) + hidden = self._m3_fc_hidden + if ( + hidden is not None + and hidden.is_cuda + and not (torch.compiler.is_compiling() or torch.cuda.is_current_stream_capturing()) + ): + hidden.record_stream(main) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model: Optional[M3Eagle3Draft] + ) -> None: + """Before every target forward: whether this step's Eagle3 captures apply the drafter's fc_norm themselves. + + On a fused-verify step whose fc_norm weights are already stacked (by an earlier fused first pass), the + capture writes rmsnorm(h + r) * fc_norm[i] (one kernel per captured layer instead of add + copy) and the fused + first drafter inputs read it without their own chunk_norm. A fused-verify step of at most + ``FC_EARLY_MAX_ROWS`` verify rows is also armed to fork the drafter's first fc at the last captured layer + (``capture_hidden_states``). + """ + self._m3_capture_norm = None + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = None + if draft_model is None: + return + if not self.tail_eligible(attn_metadata, spec_metadata, lm_head): + return + # The fork reads every captured column at the last captured layer, so a final-state capture, which is written + # after the model returns, keeps the fc at its stock place. + if ( + attn_metadata.num_seqs * (spec_metadata.runtime_draft_len + 1) <= FC_EARLY_MAX_ROWS + and FINAL_HIDDEN_LAYER_ID not in spec_metadata.layers_to_capture + ): + self._m3_fc_early_drafter = draft_model + fc_norm_w = self._m3_fc_norm_w + if ( + fc_norm_w is not None + and fc_norm_w[0] is draft_model.w["fc_norm0"] + and len(spec_metadata.layers_to_capture) == fc_norm_w[1].shape[0] + ): + self._m3_capture_norm = (fc_norm_w[1], draft_model.rms_eps) + + @staticmethod + def _m3_sampled_verify_ok(spec_metadata) -> bool: + """Whether sampled verify rows can use the fused sampled op: only under advanced_sampling_mode=no_topk, the + deploy-time guarantee that no row uses top-k or min-p (that op applies neither).""" + mode = getattr(spec_metadata, "advanced_sampling_mode", None) + return mode is not None and mode.skips_top_k and not mode.skips_top_p + + def _m3_target_accept( + self, + hidden_states, + spec_metadata, + batch_size: int, + draft_len: int, + draft_model: Optional[M3Eagle3Draft] = None, + ): + """Fused verification (see tail_eligible); same outputs as the strict acceptance path.""" + force_total = 0 + force_vec = None + if self.force_num_accepted_tokens != 0.0: + is_warmup = spec_metadata.is_cuda_graph and not torch.cuda.is_current_stream_capturing() + if not torch.cuda.is_current_stream_capturing(): + # The draw pool is built by an H2D copy and the draw kernel JIT-compiles per block width, so both + # happen here, before the first captured step (a scratch counter keeps the real draw sequence). + self._ensure_force_accept_rng_state(hidden_states.device) + if not self._m3_force_vec_warm: + scratch = torch.zeros(1, dtype=torch.int64, device=hidden_states.device) + dummy = torch.empty(64, dtype=torch.int32, device=hidden_states.device) + for block in (16, 32, 64): + m3_force_vec( + self._force_accept_rng_pool, + scratch, + dummy, + block, + 1, + 2, + 0.5, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + self._m3_force_vec_warm = True + if not is_warmup: + force_total = min(int(self.force_num_accepted_tokens) + 1, draft_len + 1) + force_vec = self._m3_force_vec( + batch_size, force_total, draft_len + 1, hidden_states.device + ) + draft_tokens = spec_metadata.draft_tokens.view(batch_size, draft_len) + if draft_tokens.dtype != torch.int32: + draft_tokens = draft_tokens.to(torch.int32) + # Column 0 receives the last accepted token, the next step's first input (the fused draft loop adds the rest). + self._m3_next_new = torch.empty( + (batch_size, draft_len + 1), dtype=torch.int32, device=hidden_states.device + ) + rows = hidden_states[: batch_size * (draft_len + 1)] + lm_head = self._m3_target_lm_head + batch_idx = spec_metadata.batch_indices_cuda[:batch_size] + head = self._m3_coded_head(lm_head, rows.shape[0]) + # The Eagle3 fc on the aux stream: the fork made at the last captured layer, else one beside the head. A + # sampled verify's coded head holds every SM while it streams: its fc is forked after it instead (overlapping + # the sampled acceptance), where it does not delay the head's start. + fork = self._m3_fc_early_stream + self._m3_fc_early_stream = None + head_first = fork is None and head is not None and not spec_metadata.is_all_greedy_sample + if fork is None and not head_first: + fork = self._m3_fc_fork(spec_metadata, rows.shape[0], draft_model) + # This rank's vocabulary shard of the target logits: the head's own local GEMM (or its coded copy), whose + # all-gather the fused kernels replace (they exchange per-row records across ranks, not logits). + if head is not None: + logits = m3_target_head_kernel.logits(head, rows) + else: + logits = lm_head.apply_linear(rows, None) + head_done = None + if head_first: + head_done = torch.cuda.Event() + head_done.record() + vocab_offset = self.mapping.tp_rank * lm_head.weight.shape[0] + if spec_metadata.is_all_greedy_sample: + result = m3_shard_accept_kernel.shard_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + force_total, + self._m3_next_new, + force_vec=force_vec, + ) + path = SHARD_ACCEPT.path + else: + n_rows = rows.shape[0] + seeds, offsets = self._rng_state_per_token(spec_metadata, n_rows) + result = m3_sampled_accept_kernel.sampled_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + spec_metadata.temperatures[:n_rows], + spec_metadata.top_ks[:n_rows], + spec_metadata.top_ps[:n_rows], + None, + seeds, + offsets, + force_total, + self._m3_next_new, + force_vec=force_vec, + ) + path = "spec_tail" + if head_done is not None: + # Enqueued after the sampled acceptance (which starts first) but waits on the head only. + fork = self._m3_fc_fork(spec_metadata, rows.shape[0], draft_model, after=head_done) + if fork is not None: + self._m3_fc_join(fork) + if result is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, None, "the fused verify's exchange workspace") + raise RuntimeError( + "M3 fused verify unavailable after tail_eligible() accepted the step" + ) + return result + + def prebuild_target_head(self, lm_head: nn.Module) -> bool: + """Build the 12-bit coded copy of the target head shard the fused verify streams at <= TARGET_HEAD_MAX_ROWS + rows (rebuilt from the current weight values on every call), on every rank. True when the coded copy decodes + to the BF16 weight bit for bit; otherwise (or for a head the fused verify does not read) nothing is kept and + the head's own GEMM runs.""" + self._m3_target_head = None + if not _sharded_head(lm_head) or not m3_target_head_kernel.head_supported(lm_head): + return False + head = m3_target_head_kernel.prepare_head(lm_head) + self._m3_target_head = head + if not head.coded: + logger.warning( + f"[M3 target head] the coded shard {tuple(lm_head.weight.shape)} does not decode to the BF16 weight " + "bit for bit; the fused verify keeps the head's BF16 GEMM" + ) + return head.coded + + def _m3_coded_head(self, lm_head: nn.Module, rows: int): + """The coded target head for a fused verify of ``rows`` rows, or None for the head's own GEMM: above + ``TARGET_HEAD_MAX_ROWS`` rows (uncovered), on a shard whose coding failed its round trip (uncovered: the GEMM + is the exact path), and when the copy was not built for this weight (a miss).""" + path = TARGET_HEAD.path + if not TARGET_HEAD.covers(rows): + uncovered(path, f"{rows} verify rows (the head's own GEMM runs)") + return None + head = self._m3_target_head + if head is None or head.weight is not lm_head.weight: + missed( + path, + rows, + "the coded target head is not built for the head's current weight (prebuild)", + ) + return None + if not head.coded: + uncovered( + path, "the coded shard failed its bitwise round trip (the head's own GEMM runs)" + ) + return None + return head + + def _m3_force_vec( + self, batch_size: int, base_total: int, max_total: int, device + ) -> Optional[torch.Tensor]: + """Per-request forced accepted counts for a fractional forced acceptance, else None. + + Draws exactly as _apply_force_accepted_tokens does for a decode-only batch (same fixed-seed pool, one counter + bump per step, the same (counter, slot) hash), so every TP rank and the generic path agree on the counts. + """ + frac = self.force_num_accepted_tokens - int(self.force_num_accepted_tokens) + if frac <= 0.0 or base_total >= max_total: + return None + self._ensure_force_accept_rng_state(device) + out = torch.empty(batch_size, dtype=torch.int32, device=device) + return m3_force_vec( + self._force_accept_rng_pool, + self._force_accept_rng_counter, + out, + batch_size, + base_total, + max_total, + float(frac), + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + max(16, 1 << (batch_size - 1).bit_length()), + ) + + # ---------------------------------------------------------------------------------------------- fused draft loop + + def prebuild_draft_head(self, draft_model: Optional[M3Eagle3Draft]) -> bool: + """Build the streamed copy of the drafter's BF16 head shard that the fused draft loop reads (rebuilt from the + current weight values on every call), on every rank. False when the drafter is not one the loop serves.""" + self._m3_draft_head = None + if draft_model is None or not self._m3_drafter_supported(draft_model): + return False + self._m3_draft_head = m3_draft_step_kernel.prepare_head( + draft_model.lm_head, self.mapping.tp_rank, DRAFT_HEAD_CODED + ) + return True + + def _m3_drafter_supported(self, draft_model: M3Eagle3Draft) -> bool: + """The drafter the fused draft loop runs (``M3Eagle3Draft`` validates its shape at construction) over this + worker's tensor-parallel group with a bias-free vocab-sharded BF16 head without padding.""" + if self.mapping is None or self.mapping.tp_size <= 1 or self.mapping.enable_attention_dp: + return False + if self._d2t is not None: + return False + lm_head = draft_model.lm_head + return _sharded_head(lm_head) and m3_draft_step_kernel.head_supported(lm_head) + + def _m3_fused_loop_ok( + self, attn_metadata, spec_metadata, batch_size: int, draft_model: Optional[M3Eagle3Draft] + ) -> bool: + """Whether this step drafts with the fused loop: a decode-only step the fused verify or the seed prepared + (``_m3_first_gather_ids``) with the fused first draft inputs, a supported drafter, a (draft length, requests) + ``DRAFT_STEP`` covers, the prebuilt head and the exchange workspace. Host state only, so the TP ranks + agree.""" + if ( + self._m3_first_gather_ids is None + or not self._m3_fused_first_inputs + or draft_model is None + ): + return False + path = DRAFT_STEP.path + if not self._m3_drafter_supported(draft_model) or not hasattr( + attn_metadata, "kv_lens_cuda" + ): + uncovered( + path, "the drafter or its attention metadata is not one the fused loop serves" + ) + return False + draft_len = spec_metadata.runtime_draft_len + if not DRAFT_STEP.covers((draft_len, batch_size)): + uncovered(path, f"draft length {draft_len} at {batch_size} requests") + return False + head = self._m3_draft_head + if head is None or head.weight is not draft_model.lm_head.weight: + missed( + path, + batch_size, + "the drafter head's streamed copy is not built for its current weight (prebuild)", + ) + return False + if m3_draft_step_kernel.workspace(self.mapping, head.wf.device) is None: + capture_missed(path, None, "the draft-step exchange workspace") + return False + return True + + def _forward_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ): + """The fused draft loop on a step it covers (``_m3_fused_loop_ok``), else the in-tree loop.""" + if self._m3_fused_loop_ok(attn_metadata, spec_metadata, batch_size, draft_model): + return self._m3_fused_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + batch_size, + num_accepted_tokens, + ) + return super()._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + + def _m3_fused_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model: M3Eagle3Draft, + draft_kv_cache_manager, + batch_size: int, + num_accepted_tokens, + ): + """Mirrors Eagle3OneModelWorker._forward_linear_draft_loop for a decode-only Eagle3 step, with each pass's + glue fused (see ``_m3_fused_loop_ok``): + + * the draft input -- the embedding lookup, the layer's input norm and hidden norm and their concat -- in one + launch (``m3_spec_tail.draft_input``; the second pass gathers the last accepted rows there); + * the drafter layer's attention and MLP, each followed by the all-reduce with the residual add and the next + RMSNorm (the post-attention norm, then the drafter's final norm); + * one kernel for the BF16 head over the head rows (pass 0: the last accepted row of each request), the + cross-rank argmax and the bookkeeping the in-tree loop does between passes: the draft token, the next pass's + input id and position, the draft KV lengths (and pass 0's sequence lengths) (``m3_draft_step``). + + The host metadata transitions are the in-tree loop's. The drafts land in columns 1.. of the step's next-input + tokens (column 0 is the last accepted token), which ``_forward_impl`` returns as they lie. + """ + draft_len = spec_metadata.runtime_draft_len + table = draft_model.embed_table + dev = table.device + head = self._m3_draft_head + gather_ids = self._m3_first_gather_ids + ids = inputs["input_ids"] + hidden = inputs["hidden_states"] + pos = inputs["position_ids"] + if ids.dtype != torch.int32: + ids = ids.to(torch.int32) + if pos.dtype != torch.int32: + pos = pos.to(torch.int32) + num_acc = num_accepted_tokens[:batch_size] + if num_acc.dtype != torch.int32: + num_acc = num_acc.to(torch.int32) + next_new = self._m3_next_new + if next_new is None: + # Seeded from the in-tree acceptance: the last accepted token is the first-pass input at the gather row. + next_new = torch.empty((batch_size, draft_len + 1), dtype=torch.int32, device=dev) + next_new[:, 0] = ids[gather_ids] + draft_tokens = next_new[:, 1:] + w = draft_model.w + eps = draft_model.rms_eps + + with self.draft_kv_cache_context(attn_metadata, draft_kv_cache_manager) as md: + for i in range(draft_len): + x, residual = m3_spec_tail.draft_input( + ids, + table, + hidden, + gather_ids if i == 1 else None, + w["in_norm"], + eps, + w["hidden_norm"], + eps, + ) + h = draft_model.attention(x, md, reduce=False) + h, residual = draft_model.ar_norm(h, residual, "post_norm") + h = draft_model.mlp(h, reduce=False) + h, to_save = draft_model.ar_norm(h, residual, "final_norm") + if draft_model.return_hidden_post_norm: + to_save = h + ids_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + pos_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + if not m3_draft_step_kernel.draft_step( + head, + h, + batch_size, + gather_ids if i == 0 else None, + self.mapping, + draft_tokens, + i, + ids_next, + pos, + pos_next, + md.kv_lens_cuda, + num_acc if i == 0 else None, + md._seq_lens_cuda if i == 0 else None, + draft_len, + ): + raise RuntimeError( + "M3 fused draft step unavailable after _m3_fused_loop_ok() accepted the step" + ) + ids, pos, hidden = ids_next, pos_next, to_save + if i == 0: + # The draft step set the device sequence lengths to 1 and rolled the KV lengths back. + md._seq_lens[:batch_size].fill_(1) + md.on_update() + has_kv_cache = md.kv_cache_manager is not None + if has_kv_cache and hasattr(md, "host_request_types"): + md.host_request_types[: md.num_contexts].fill_(1) + if has_kv_cache: + md.num_contexts = 0 + self._prepare_flash_mla_generation_layout(md, 0, batch_size) + md.update_for_spec_dec() + md.use_spec_decoding = False + else: + md.update_for_spec_dec() + self._m3_next_new_tokens = next_new + return draft_tokens + + +def _build_speculation(shell, model_config: ModelConfig): + """The drafter config, the drafter and the worker of one-model Eagle3 (the in-tree one-engine construction for + the llama3 drafter architecture), or of one-model standalone DSpark (``dspark.build_dspark``: the in-tree drafter + and worker).""" + spec = model_config.spec_config + if _dspark.is_standalone_dspark(spec): + shell.model.embed_tokens = _dspark.M3EmbedTokens(shell.model) + return _dspark.build_dspark(shell, model_config) + if ( + not spec.spec_dec_mode.use_one_engine() + or not spec.spec_dec_mode.is_eagle3_one_model() + or spec.eagle3_model_arch != "llama3" + or getattr(spec, "use_dynamic_tree", False) + ): + raise ValueError( + "this MiniMax-M3 target speculates with linear one-model Eagle3 (llama3 drafter) only" + ) + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, + trust_remote_code=True, + attn_backend=model_config.attn_backend, + moe_backend=_get_requested_draft_moe_backend(model_config, spec), + mapping=model_config.mapping, + spec_config=model_config.spec_config, + max_num_tokens=model_config.max_num_tokens, + moe_max_num_tokens=model_config.moe_max_num_tokens, + ) + _set_draft_kv_cache_quant_algo(draft_config, model_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = M3Eagle3Draft( + shell.model, + draft_config, + model_config.pretrained_config.num_hidden_layers, + spec.num_capture_layers, + ) + worker = M3Eagle3Worker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + return draft_config, draft, worker, use_separate + + +# -------------------------------------------------------------------------------------------------------------------- +# Runtime hooks and the shell +# -------------------------------------------------------------------------------------------------------------------- + + +def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: + """The step-loop hooks this target opts into, all measured on GB300 tp4 AgentX decode. + + * ``pack_step_inputs``: a decode step's small input copies go out as one upload and one scatter at the graph + replay instead of a chain of copies the next graph waits on (~30 us of idle GPU at one request). + * ``skip_unchanged_h2d``: steady decode re-uploads slot indices, query lengths and page tables that rarely change. + * ``loop_on_execution_stream``: the overlap loop's own stream work and the forward on one stream, so step N+1's + inputs wait on graph N in program order rather than through two cross-stream event hops. + * ``merge_tuned_tactics_across_ranks``: tensor-parallel peers capture the same GEMM tactics (rank-local profiling + noise otherwise splits them, and the slowest rank sets every step). + * ``skip_greedy_rng_upload``: all-greedy speculative batches never read the Philox seeds. + * ``spec_step_fold``: the spec sampler's slot stores and the engine's next-step input gathers in one launch each; + under DSpark with ``BOUNDARY_PACK``, both inside a batch-1 decode graph's tail. + * MSA: identity selections for short sequences (and short/long graph families), the one-launch KV-length update, + unchanged block-table copies skipped, pool views built once. + * HOSTTAIL (kill switches above ``HOSTTAIL_*``): ``sampler_d2h_side_stream`` (the sampler's store copies on its + D2H side stream), ``spec_small_h2d`` (static slot ids, packed accepted counts), ``spec_batch_map_changed_only`` + (the drafter's batch_indices / batch-to-slot uploads only on change), ``skip_accepted_draft_upload`` (no reader + of the accepted counts here), ``pack_spec_rng_uploads`` (Philox uploads in the pack), ``kv_lens_update_once`` + (one MSA KV-length re-derivation per speculative overlap step). + * ``pack_parallel_scatter`` (``PACK_PSCATTER``): the pack's non-overlapping copies scatter in parallel. + * ``pack_in_graph`` (``PACK_IN_GRAPH``) / ``pack_early_copy`` (``PACK_EARLY_COPY``): the decode graph's prologue + issues the pack, from a window uploaded on a side copy stream while the previous graph runs (no H2D copy or + scatter between the graphs). ``h2d_changed_span`` (``H2D_CHANGED_SPAN``): page-crossing steps stage only the + changed page-table entries, so their pack still fits the prologue's window. + """ + return RuntimeHooks( + pack_step_inputs=True, + pack_parallel_scatter=PACK_PSCATTER, + pack_in_graph=PACK_IN_GRAPH, + pack_early_copy=PACK_EARLY_COPY, + skip_unchanged_h2d=True, + h2d_changed_span=H2D_CHANGED_SPAN, + loop_on_execution_stream=True, + merge_tuned_tactics_across_ranks=True, + skip_greedy_rng_upload=True, + spec_step_fold=spec_fold, + kv_lens_update_once=HOSTTAIL_KV_LENS_ONCE, + pack_spec_rng_uploads=HOSTTAIL_PHILOX_PACK, + spec_small_h2d=HOSTTAIL_SMALL_H2D, + skip_accepted_draft_upload=HOSTTAIL_SKIP_ACCEPTED_UPLOAD, + spec_batch_map_changed_only=HOSTTAIL_BATCH_CHANGED_ONLY, + sampler_d2h_side_stream=HOSTTAIL_D2H_SIDE, + msa=MsaRuntimePolicy( + short_sequence_identity=True, + kv_update=_msa_kv_update_kernel.msa_kv_update, + skip_unchanged_block_offsets=True, + reuse_pool_views=True, + ), + ) + + +class ModelingV2MiniMaxM31Nvfp4Sm103Tp4( + DecoderModelForCausalLM[MiniMaxM3Tp4ModelingV2, PretrainedConfig] +): + """MiniMax-M3 NVFP4 / sm_103 / tp4, text only, with one-model Eagle3 or no speculation. + + The shell owns the vocab-parallel BF16 ``lm_head`` and the logits processor (``DecoderModelForCausalLM``); with + speculation, the drafter (``M3Eagle3Draft``, its weights from the speculative checkpoint through + ``load_draft_weights``) and the worker (``M3Eagle3Worker``) that verifies and drafts each step. + """ + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + assert torch.cuda.get_device_capability() == _SM, ( + f"this target is built for sm_{_SM[0]}{_SM[1]}, the device is sm_" + f"{''.join(map(str, torch.cuda.get_device_capability()))}" + ) + # Under modeling_v2='require' a covered decode shape that cannot take its fast path raises. + set_require(resolved_mode(type(self)) is ModelingV2Mode.REQUIRE) + model_config = _text_model_config(model_config) + cfg = model_config.pretrained_config + spec_config = getattr(model_config, "spec_config", None) + if ( + model_config.quant_config.kv_cache_quant_algo == QuantAlgo.NVFP4 + and _dspark.draft_kv_override_applies(spec_config) + ): + # The one-model Eagle3 layer has no NVFP4 trtllm-gen decode kernel: keep its modules and the shared + # draft cache on FP8 while the target stays NVFP4. + model_config.extra_attrs["draft_kv_cache_quant_algo_override"] = QuantAlgo.FP8 + super().__init__( + MiniMaxM3Tp4ModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.spec_config = spec_config + self.draft_model: Optional[nn.Module] = None + self.draft_config: Optional[ModelConfig] = None + self.spec_worker: Optional[Union[M3Eagle3Worker, _dspark.M3DSparkWorker]] = None + self.use_separate_draft_kv_cache = False + self.layer_idx = FINAL_HIDDEN_LAYER_ID + spec_fold = None + if spec_config is not None: + ( + self.draft_config, + self.draft_model, + self.spec_worker, + self.use_separate_draft_kv_cache, + ) = _build_speculation(self, model_config) + # The DSpark worker keeps the in-tree sampler slot stores; only the engine's input gathers fold (DSFOLD). + if not _dspark.is_standalone_dspark(spec_config): + spec_fold = M3SpecStepFold() + elif DSFOLD: + spec_fold = M3DSparkStepFold( + boundary_pack=BOUNDARY_PACK, + tokens_per_step=_static_tokens_per_step(spec_config) if BOUNDARY_PACK else None, + ) + # BOUNDARY_PACK: the fold whose graph_tail ends each decode forward (None: the forward returns the worker's). + self.boundary_pack_fold = ( + spec_fold + if isinstance(spec_fold, M3DSparkStepFold) and spec_fold.boundary_pack + else None + ) + self.runtime_hooks = _runtime_hooks(spec_fold) + _install_topp_det() + _install_softmax_mc() + _install_rej_mc() + + def load_weights(self, weights: Dict, *args, **kwargs) -> None: + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_draft_weights(self, weights: Dict, weight_mapper=None) -> None: + """The drafter's checkpoint (``speculative_model``), loaded by the engine after the target's.""" + if _dspark.is_standalone_dspark(self.spec_config): + _dspark.load_dspark_draft_weights(self, weights, weight_mapper) + return + _weights.DRAFT_WEIGHTS.load(self.draft_model, weights) + + def set_guided_decoder(self, guided_decoder) -> bool: + """Mirrors ``SpecDecOneEngineForCausalLM.set_guided_decoder``: guided decoding with speculation is installed + on the worker.""" + if hasattr(self.spec_worker, "set_guided_decoder"): + return self.spec_worker.set_guided_decoder(guided_decoder) + return False + + def cache_derived_state(self) -> None: + """Build the targets and every decode kernel once the weights are real, before the executor's warmup; with + speculation also the fused verify's coded target head and the fused draft loop's head copy.""" + super().cache_derived_state() + self.model.post_load_weights() + _prebuild_topp_det() + _prebuild_softmax_mc() + _prebuild_rej_mc() + if self.spec_worker is not None: + self.spec_worker.prebuild(self.lm_head, self.draft_model) + + def post_load_weights(self) -> None: + super().post_load_weights() + self.model.post_load_weights() + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.Tensor] = None, + position_ids: Optional[torch.Tensor] = None, + inputs_embeds: Optional[torch.Tensor] = None, + return_context_logits: bool = False, + spec_metadata: Optional[SpecMetadata] = None, + resource_manager: Optional[Any] = None, + **kwargs, + ) -> Any: + """One engine step: the target forward, then the logits (or, with speculation, the worker's verify and + draft). Text only.""" + multimodal_params = kwargs.pop("multimodal_params", None) or [] + if any(p is not None and getattr(p, "multimodal_data", None) for p in multimodal_params): + raise ValueError( + "this MiniMax-M3 target is text only; the request carries multimodal input" + ) + worker = self.spec_worker + capture = _plain_capture + if worker is not None: + capture = worker.capture_hidden_states + if spec_metadata is not None: + worker.arm_capture_norm( + attn_metadata, spec_metadata, self.lm_head, self.draft_model + ) + hidden_states = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + capture=capture, + **kwargs, + ) + if getattr(attn_metadata, "csa2_remote_tail_mode", None) == "source": + # A context worker stopped at the encoder/decoder boundary: a disposable token for the context-only + # completion, and no draft worker. + return self.logits_processor.forward(hidden_states, self.lm_head, attn_metadata, False) + if spec_metadata is not None and spec_metadata.is_layer_capture(FINAL_HIDDEN_LAYER_ID): + capture(spec_metadata, FINAL_HIDDEN_LAYER_ID, hidden_states, None) + if attn_metadata.padded_num_tokens is not None: + hidden_states = hidden_states[: attn_metadata.num_tokens] + if worker is None: + return self.logits_processor.forward( + hidden_states, self.lm_head, attn_metadata, return_context_logits + ) + # The fused verify computes this rank's logits shard itself and needs no gathered logits. + if worker.tail_eligible(attn_metadata, spec_metadata, self.lm_head): + logits = None + else: + # A worker may produce the gathered target logits itself (bitwise to the processor); None = the processor. + target_logits = getattr(worker, "target_logits", None) + logits = ( + target_logits(hidden_states, attn_metadata, spec_metadata, self.lm_head) + if target_logits is not None + else None + ) + if logits is None: + logits = self.logits_processor.forward( + hidden_states[spec_metadata.gather_ids], self.lm_head, attn_metadata, True + ) + spec_input_ids = input_ids if input_ids is not None else kwargs.get("orig_input_ids") + spec_position_ids = position_ids + if attn_metadata.padded_num_tokens is not None: + if spec_input_ids is not None: + spec_input_ids = spec_input_ids[: attn_metadata.num_tokens] + if position_ids is not None: + spec_position_ids = _slice_spec_position_ids(position_ids, attn_metadata.num_tokens) + outputs = worker( + input_ids=spec_input_ids, + position_ids=spec_position_ids, + hidden_states=hidden_states, + logits=logits, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=resource_manager, + ) + if self.boundary_pack_fold is not None and spec_metadata is not None: + # BOUNDARY_PACK: a batch-1 decode graph ends with the sampler's stores and the next step's gather. + outputs = self.boundary_pack_fold.graph_tail(outputs, attn_metadata) + return outputs diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/mxdraft.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/mxdraft.py new file mode 100644 index 000000000000..da8a096aaa15 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/mxdraft.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MXDRAFT: the DSpark drafter's decode projections on this target's MXFP8 GEMV. + +The drafter checkpoint ships its linears as MXFP8 (E4M3 codes, one UE8M0 scale per 32 columns) and the stock drafter +dequantizes them to BF16 at load, so every decode pass streams twice the bytes it needs. MXDRAFT recovers the MXFP8 +form of each BF16 weight (exactly: the recovery is checked to dequantize back to the same BF16 bits, else the +projection is left alone) and routes the projection's GEMM to ``m3_mx_gemv_bf16`` for decode inputs it serves; any +other call runs the Linear unchanged. The activation is quantized to MXFP8 inside the GEMV, so the output is not the +stock BF16 GEMM's: drafter-only precision, gated on natural AL. Row-parallel projections keep +their all-reduce (only the Linear's GEMM is replaced). +""" + +from __future__ import annotations + +from typing import Iterable, Optional, Tuple + +import torch + +from .ops.comm.m3_mc_allreduce_norm import swizzle_sf +from .ops.mx_gemv import op as _gemv + +E4M3_MAX = 448.0 +BLOCK = 32 + + +def mxfp8_from_bf16(weight: torch.Tensor) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """The MXFP8 form (E4M3 codes [N, K], UE8M0 scale bytes [N, K / 32]) of a BF16 weight that is exactly an MXFP8 + dequantization, or None. Each block takes the smallest power-of-two scale that fits its amax in E4M3, which is at + most the shipped scale, so the codes are exact whenever the weight came from MXFP8.""" + if weight.dtype != torch.bfloat16 or weight.dim() != 2 or weight.shape[1] % BLOCK != 0: + return None + n, k = weight.shape + blocks = weight.float().view(n, k // BLOCK, BLOCK) + amax = blocks.abs().amax(-1) + exp = torch.ceil(torch.log2(amax.clamp_min(2.0**-126) / E4M3_MAX)).clamp(-127, 127) + scale = torch.exp2(exp).unsqueeze(-1) + codes = (blocks / scale).to(torch.float8_e4m3fn) + back = (codes.float() * scale).view(n, k).to(torch.bfloat16) + if not torch.equal(back, weight): + return None + return codes.view(n, k).contiguous(), (exp + 127).to(torch.uint8).contiguous() + + +def _projections(draft_model) -> Iterable[torch.nn.Module]: + for layer in getattr(getattr(draft_model, "model", None), "layers", []) or []: + attn, mlp = getattr(layer, "self_attn", None), getattr(layer, "mlp", None) + for lin in ( + getattr(attn, "qkv_proj", None), + getattr(attn, "o_proj", None), + getattr(mlp, "gate_up_proj", None), + getattr(mlp, "down_proj", None), + ): + if lin is not None: + yield lin + + +def route_mx_linears(draft_model, rows: Iterable[int]) -> int: + """Route each drafter projection whose BF16 weight is an exact MXFP8 dequantization to the MXFP8 GEMV for 2-D + contiguous bf16 inputs of the given row counts (the decode passes'); compiles those kernels now (outside capture). + Returns how many projections were routed.""" + rows = sorted({int(m) for m in rows if 0 < int(m) <= _gemv.MAX_TOKENS}) + n_routed = 0 + for lin in _projections(draft_model): + weight = getattr(lin, "weight", None) + if ( + weight is None + or getattr(lin, "bias", None) is not None + or getattr(lin, "_m31_mxdraft", None) is not None + ): + continue + rec = mxfp8_from_bf16(weight.data) + if rec is None: + continue + n, k = weight.shape + served = tuple(m for m in rows if _gemv.serves(n, k, m, True)) + if not served: + continue + w8, sf = rec[0], swizzle_sf(rec[1]) + for m in served: + _gemv.precompile(w8, sf, m, True) + original = lin.apply_linear + + def apply_linear( + input, + bias, + lora_params=None, + layer_idx=None, + *, + _o=original, + _w8=w8, + _sf=sf, + _served=served, + ): + if ( + bias is None + and not lora_params + and isinstance(input, torch.Tensor) + and input.dtype == torch.bfloat16 + and input.dim() == 2 + and input.shape[0] in _served + and input.is_contiguous() + ): + out = _gemv.m3_mx_gemv_bf16(input, _w8, _sf) + if out is not None: + return out + return _o(input, bias, lora_params, layer_idx) + + lin._m31_mxdraft = (w8, sf, served) + lin.apply_linear = apply_linear + n_routed += 1 + return n_routed diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/__init__.py new file mode 100644 index 000000000000..1059445198db --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Ops used only by this target. Shared ops stay in the catalog.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/__init__.py new file mode 100644 index 000000000000..b8c7426e3933 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Activation ops of this target: the gated MLPs' fused SwiGLU.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/m3_silu_and_mul.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/m3_silu_and_mul.py new file mode 100644 index 000000000000..64cb00558760 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/m3_silu_and_mul.py @@ -0,0 +1,83 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The gated MLPs' fused SwiGLU: ``silu_and_mul`` with MiniMax-M3's SwiGLU-OAI gain, offset and clamp.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SiluAndMul(OpWrapper): + """``out = g * sigmoid(alpha * g) * (u + beta)`` over ``x = [g | u]`` (bf16 [M, 2I] -> [M, I]), with + ``g = min(gate, limit)`` and ``u = clamp(up, -limit, limit)``; ``limit``, ``alpha`` and ``beta`` None give plain + SwiGLU ``silu(gate) * up``. + + The MXFP8 dense MLPs (layers 0-2, intermediate 3072 per rank) and the MoE's shared expert (intermediate 768 per + rank) run it at MiniMax-M3's (limit 7, alpha 1.702, beta 1); the BF16 Eagle3 drafter's MLP (intermediate 4608 per + rank) runs it plain. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert, one decode token", spec=dict(m=1, inter=768, oai=True)), + Cell(why="shared expert, a 48-row decode verify", spec=dict(m=48, inter=768, oai=True)), + Cell(why="dense MLP, a prefill chunk", spec=dict(m=2048, inter=3072, oai=True)), + Cell(why="dense MLP, one decode token", spec=dict(m=1, inter=3072, oai=True)), + Cell( + why="drafter MLP (plain SwiGLU), eight decode rows", + spec=dict(m=8, inter=4608, oai=False), + ), + Cell( + why="drafter MLP (plain SwiGLU), a prefill chunk", + spec=dict(m=1024, inter=4608, oai=False), + ), + ) + + note = """ + The kernel computes in fp32 and rounds once to bf16; `reference` does the same arithmetic in fp32, so the two + differ only by the fast sigmoid's last bits (the test's band). + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + + def reference( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + gate, up = x.float().chunk(2, dim=-1) + if limit is not None: + gate = gate.clamp(max=limit) + up = up.clamp(min=-limit, max=limit) + a = 1.0 if alpha is None else alpha + b = 0.0 if beta is None else beta + return (gate * torch.sigmoid(a * gate) * (up + b)).to(x.dtype) + + def is_valid( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> None: + assert x.stride(-1) == 1, "the gate/up halves are read from contiguous rows" + + +m3_silu_and_mul = M3SiluAndMul() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/m3_swq.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/m3_swq.py new file mode 100644 index 000000000000..95efc345ea4a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/activation/m3_swq.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SWQ: the shared expert's SwiGLU-OAI written directly as the down projection's MXFP8 activation.""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +@triton.jit +def _rcp_approx_ftz(x): + return tl.inline_asm_elementwise( + "rcp.approx.ftz.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _swq_kernel( + x_ptr, + x_stride, + q_ptr, + sf_ptr, + M, + INTER: tl.constexpr, + NB: tl.constexpr, + NB_PAD: tl.constexpr, + P2: tl.constexpr, + swiglu_limit: tl.constexpr, + swiglu_alpha: tl.constexpr, + swiglu_beta: tl.constexpr, + HAS_LIMIT: tl.constexpr, +): + # One program per row of the 128-row-padded scale layout; rows past M only zero their scale bytes. + r = tl.program_id(0) + blk = tl.arange(0, P2) + blk_ok = blk < NB_PAD + # The 128x4 swizzle of trtllm::mxfp8_quantize(x, True) (quantization.cuh get_sf_out_offset_128x4). + sf_off = ( + (blk % 4) + + (blk // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * NB_PAD) + ) + if r >= M: + tl.store(sf_ptr + sf_off, tl.zeros([P2], dtype=tl.uint8), mask=blk_ok) + return + cols = tl.arange(0, P2 * 32) + valid = cols < INTER + row = x_ptr + r.to(tl.int64) * x_stride + # silu_and_mul_kernel's arithmetic (modules/swiglu.py), then its bf16 store. + a = tl.load(row + cols, mask=valid, other=0.0).to(tl.float32) + b = tl.load(row + cols + INTER, mask=valid, other=0.0).to(tl.float32) + if HAS_LIMIT: + a = tl.minimum(a, swiglu_limit) + b = tl.clamp(b, -swiglu_limit, swiglu_limit) + v = (a * tl.sigmoid(swiglu_alpha * a) * (b + swiglu_beta)).to(tl.bfloat16).to(tl.float32) + # cvt_warp_fp16_to_mxfp8: SF = amax * rcp.approx(448), UE8M0 toward +inf, scale = rcp.approx(SF), E4M3 RN. + v2 = tl.reshape(v, (P2, 32)) + amax = tl.max(tl.abs(v2), axis=1) + sf = amax * _rcp_approx_ftz(tl.full([P2], 448.0, tl.float32)) + bits = sf.to(tl.int32, bitcast=True) + exp = (bits >> 23) & 0xFF + mant = bits & 0x7FFFFF + e = tl.where(exp > 0, exp + (mant != 0).to(tl.int32), tl.where(bits > 0x400000, 1, 0)) + e = tl.minimum(e, 254) + # at code 0 the scale 2^-127 is a float subnormal, so rcp.approx.ftz returns inf (native publishes + # inf/NaN there). Use its exact reciprocal 2^127 instead; every other code keeps + # native's rcp.approx (exact on a normal power of two), so each block native emits finite stays bitwise. + pinv = tl.where( + e == 0, 1.7014118346046923e38, _rcp_approx_ftz((e << 23).to(tl.float32, bitcast=True)) + ) + inv = tl.where(amax != 0.0, pinv, 0.0) + q = tl.reshape((v2 * inv[:, None]).to(tl.float8e4nv), (P2 * 32,)) + tl.store(q_ptr + r.to(tl.int64) * INTER + cols, q, mask=valid) + tl.store(sf_ptr + sf_off, tl.where(blk < NB, e, 0).to(tl.uint8), mask=blk_ok) + + +class M3Swq(OpWrapper): + """``silu_and_mul`` (SwiGLU-OAI) over ``x = [g | u]`` (bf16 [M, 2I]) rounded to bf16, then MXFP8-quantized as + ``trtllm::mxfp8_quantize(h, True)``: (E4M3 [M, I], UE8M0 scales in the 128x4-swizzled layout), bit for bit, in one + launch. The shared expert's down projection reads it at the rows its GEMV does not serve (``SWQ_ROWS``).""" + + ARCHS = frozenset({Arch.SM_103}) + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert, the smallest row count it serves", spec=dict(m=17, inter=768)), + Cell(why="shared expert, a 32-row decode verify (c8)", spec=dict(m=32, inter=768)), + ) + + note = """ + Bitwise to the two-kernel path (silu_and_mul's fp32 arithmetic and bf16 rounding, then mxfp8_quantize's + rcp.approx scale rule): q bytes and every scale byte, padding included. + One deliberate exception: a block whose scale code is 0 (amax / 448 <= 2^-127), where native's + rcp.approx.ftz of the subnormal scale is inf and its e4m3 bytes encode inf/NaN, is quantized with the exact + reciprocal 2^127 instead, so it stays finite; its scale byte is native's. + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> tuple[torch.Tensor, torch.Tensor]: + m, i = x.shape[0], x.shape[1] // 2 + nb = i // 32 + nb_pad = (nb + 3) // 4 * 4 + q = torch.empty(m, i, dtype=torch.float8_e4m3fn, device=x.device) + sf = torch.empty((m + 127) // 128 * 128 * nb_pad, dtype=torch.uint8, device=x.device) + _swq_kernel[((m + 127) // 128 * 128,)]( + x, + x.stride(0), + q, + sf, + m, + INTER=i, + NB=nb, + NB_PAD=nb_pad, + P2=triton.next_power_of_2(nb_pad), + swiglu_limit=limit or 0.0, + swiglu_alpha=alpha if alpha is not None else 1.0, + swiglu_beta=beta if beta is not None else 0.0, + HAS_LIMIT=limit is not None and limit > 0.0, + ) + return q, sf + + def reference( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> tuple[torch.Tensor, torch.Tensor]: + """The two kernels it replaces (bitwise is the contract).""" + h = torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + q, sf = torch.ops.trtllm.mxfp8_quantize(h, True) + return q.view(h.shape), sf + + def is_valid( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> None: + assert x.dtype == torch.bfloat16 and x.stride(-1) == 1, "bf16 [M, 2I] with contiguous rows" + assert (x.shape[1] // 2) % 32 == 0, "I must be a multiple of the 32-element MXFP8 block" + + +m3_swq = M3Swq() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/__init__.py new file mode 100644 index 000000000000..a7f7fee7a234 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Attention ops of this target: MSA block selection, the sparse decode and the KV-length update.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_attn_c1.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_attn_c1.py new file mode 100644 index 000000000000..625a7d5d4edc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_attn_c1.py @@ -0,0 +1,280 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The c1 kernel: fused MSA decode attention + o_proj GEMV in one 96-CTA launch, for a one-request, 6-token +verify layer.""" + +from __future__ import annotations + +from typing import NamedTuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_attn_mega import HIDDEN, check_handoff_at_rest, workspace_parity +from .m3_fused_decode_attn import ( + HEAD_DIM, + MXFP8_BLOCK, + NUM_HEADS, + check_attention_inputs, + mx_gemm, + mxfp8_rows, + reference_attention, + unswizzle_scales, +) + +#: 6 clusters of 16 CTAs, each an attention CTA and an o_proj unit: ``flags[32 + CTA]`` are the call counters. +GRID = 96 + + +class C1State(NamedTuple): + """The c1 kernel's results: the attention's bf16 output, the caches after the insert, and the o_proj output.""" + + output: torch.Tensor + kv_data: torch.Tensor + kv_scale: torch.Tensor + idx_k: torch.Tensor + y: torch.Tensor + + +class M3AttnC1(OpWrapper): + """The attention of a one-request verify step of 6 query tokens and its o_proj, in one 96-CTA launch. + + The fused attention of ``m3_fused_decode_attn`` on the packed QKV rows (bf16 ``output``, MXFP8 hand-off into the + call's parity of ``mxfp8_data`` / ``mxfp8_scale``), and the MXFP8 o_proj GEMV into ``y``: every attention CTA + is also an o_proj unit (48 row tiles x 2 K splits). ``flags[32 + CTA]`` are the per-CTA call counters; the + workspace is allocated once and never reset. With ``ar_mode`` 2, ``y`` is also pushed into the next multicast + all-reduce's exchange at ``ar_workspace`` / ``ar_flags``. + + Returns None; the cells read ``output``, ``y``, the hand-off and the caches. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="Eagle3 verify of 5 drafts (qlen 6) a few blocks in; both call parities (two calls, one workspace)", + spec=dict(seq_len=300, calls=2), + ), + Cell( + why="the full 16-block window: the six tokens fill the last rows of the last page", + spec=dict(seq_len=2048, calls=1), + ), + Cell( + why="the six tokens straddle a page boundary (positions 126..131)", + spec=dict(seq_len=132, calls=1), + ), + Cell(why="the first six tokens of a request", spec=dict(seq_len=6, calls=1)), + ) + + note = """ + Compiled for the one shape the target launches: one request of 6 query tokens (geometry (6, 1, 1), 96 CTAs), + the MXFP8 hand-off. The all-reduce push (`ar_mode` 2) needs a multicast workspace and is not driven here: the + cells run `ar_mode` 0, where `y` is the o_proj output and nothing else is written. + """ + + def __call__( + self, + packed: torch.Tensor, + flags: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + idx_k: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + eps: float, + sm_scale: float, + decode_query_len: int, + tokens_per_cluster: int, + blocks_per_cta: int, + output: torch.Tensor, + mxfp8_data: torch.Tensor, + mxfp8_scale: torch.Tensor, + use_pdl: bool, + o_weight: torch.Tensor, + o_weight_scale: torch.Tensor, + y: torch.Tensor, + ar_workspace: int, + ar_flags: int, + ar_rank: int, + ar_world: int, + ar_max_tokens: int, + ar_mode: int, + ) -> None: + torch.ops.trtllm.minimax_m3_attn_c1( + packed, + flags, + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + eps, + sm_scale, + decode_query_len, + tokens_per_cluster, + blocks_per_cta, + output, + mxfp8_data, + mxfp8_scale, + use_pdl, + o_weight, + o_weight_scale, + y, + ar_workspace, + ar_flags, + ar_rank, + ar_world, + ar_max_tokens, + ar_mode, + ) + + def reference( + self, + packed: torch.Tensor, + flags: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + idx_k: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + eps: float, + sm_scale: float, + decode_query_len: int, + tokens_per_cluster: int, + blocks_per_cta: int, + output: torch.Tensor, + mxfp8_data: torch.Tensor, + mxfp8_scale: torch.Tensor, + use_pdl: bool, + o_weight: torch.Tensor, + o_weight_scale: torch.Tensor, + y: torch.Tensor, + ar_workspace: int, + ar_flags: int, + ar_rank: int, + ar_world: int, + ar_max_tokens: int, + ar_mode: int, + ) -> C1State: + """The composition: the fused attention's reference on ``packed``, then + ``y = bf16(mxfp8(output) @ dequant(W_o)^T)`` (fp64 accumulation, one rounding). + + The workspace, the output buffers, the geometry, ``use_pdl`` and the all-reduce arguments are ignored. + """ + attn = reference_attention( + packed, + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + eps, + sm_scale, + decode_query_len, + ) + q8, ex = mxfp8_rows(attn.output) + width = NUM_HEADS * HEAD_DIM + out_y = mx_gemm( + q8, + ex, + o_weight.view(torch.uint8), + unswizzle_scales(o_weight_scale, HIDDEN, width // MXFP8_BLOCK), + ) + return C1State(attn.output, attn.kv_data, attn.kv_scale, attn.idx_k, out_y) + + def is_valid( + self, + packed: torch.Tensor, + flags: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + idx_k: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + eps: float, + sm_scale: float, + decode_query_len: int, + tokens_per_cluster: int, + blocks_per_cta: int, + output: torch.Tensor, + mxfp8_data: torch.Tensor, + mxfp8_scale: torch.Tensor, + use_pdl: bool, + o_weight: torch.Tensor, + o_weight_scale: torch.Tensor, + y: torch.Tensor, + ar_workspace: int, + ar_flags: int, + ar_rank: int, + ar_world: int, + ar_max_tokens: int, + ar_mode: int, + ) -> None: + check_attention_inputs( + packed.shape[0], + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + decode_query_len, + ) + check_handoff_at_rest(mxfp8_data, mxfp8_scale, workspace_parity(flags, GRID)) + assert ar_mode != 2 or (ar_workspace != 0 and ar_flags != 0), ( + "ar_mode 2 needs the exchange's addresses" + ) + + +m3_attn_c1 = M3AttnC1() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_attn_mega.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_attn_mega.py new file mode 100644 index 000000000000..3445c8c65cbd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_attn_mega.py @@ -0,0 +1,340 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The attention megakernel: QKV GEMV + fused MSA decode attention + o_proj GEMV in one launch, for a one-request, +one-token decode layer.""" + +from __future__ import annotations + +from typing import NamedTuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_fused_decode_attn import ( + HEAD_DIM, + MXFP8_BLOCK, + NUM_HEADS, + PACKED_WIDTH, + check_attention_inputs, + mx_gemm, + mxfp8_rows, + reference_attention, + unswizzle_scales, +) + +#: The layer input (hidden size) and the o_proj output width at tp4. +HIDDEN = 6144 +#: Persistent grid of the megakernel (7 clusters of 16): ``flags[32 + CTA]`` are its per-CTA call counters. +GRID = 112 +#: Rows of each call-parity buffer. +MAX_TOKENS = 8 + + +class MegaState(NamedTuple): + """The megakernel's results: the packed QKV rows of the call, the attention's bf16 output, the caches after the + insert, and the o_proj output.""" + + packed: torch.Tensor + output: torch.Tensor + kv_data: torch.Tensor + kv_scale: torch.Tensor + idx_k: torch.Tensor + y: torch.Tensor + + +def workspace_parity(flags: torch.Tensor, grid: int) -> int: + """The call parity every CTA of the grid will use (their counters move together), checked.""" + counters = flags[32 : 32 + grid] + assert bool((counters == counters[0]).all()), ( + "the per-CTA call counters disagree: the workspace is corrupt" + ) + return int(counters[0]) & 1 + + +def check_handoff_at_rest(mxfp8_data: torch.Tensor, mxfp8_scale: torch.Tensor, parity: int) -> None: + """The MXFP8 hand-off's parity buffer this call writes must hold the 0xFF "not yet" marker (Lamport).""" + assert bool((mxfp8_data.view(torch.uint8).reshape(2, -1)[parity] == 0xFF).all()) and bool( + (mxfp8_scale.view(torch.uint8).reshape(2, -1)[parity] == 0xFF).all() + ), ( + "the MXFP8 hand-off's buffer for this call is not at rest (0xFF): the o_proj units would read stale codes" + ) + + +class M3AttnMega(OpWrapper): + """The whole attention half of a decode layer for one request and one query token, in one launch. + + The MXFP8 QKV GEMV of the layer input (as the preceding all-reduce hands it over: E4M3 rows + 128x4-swizzled + UE8M0 scales) into the call's parity buffer of ``qkv_out``, the fused attention of ``m3_fused_decode_attn`` on + those rows (bf16 ``output``, MXFP8 hand-off into the call's parity of ``mxfp8_data`` / ``mxfp8_scale``), and the + MXFP8 o_proj GEMV into ``y``. ``flags`` holds the per-tile ready counters and the per-CTA call counters; the + workspace is allocated once and never reset. With ``ar_mode`` 2, ``y`` is also pushed into the next multicast + all-reduce's exchange at ``ar_workspace`` / ``ar_flags``. + + Returns None; the cells read ``qkv_out``, ``output``, ``y``, the hand-off and the caches. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode of one request a few blocks in; both call parities (two calls on one workspace)", + spec=dict(seq_len=300, calls=2), + ), + Cell( + why="the full 16-block window, the new token on the last row of the last page", + spec=dict(seq_len=2048, calls=1), + ), + Cell( + why="the new token opens a page (position 1024): its own page's other rows are past the length", + spec=dict(seq_len=1025, calls=1), + ), + Cell(why="the first token of a request: one attended row", spec=dict(seq_len=1, calls=1)), + ) + + note = """ + Compiled for the one shape the target launches: one request, one query token (geometry (1, 1, 1)), the MXFP8 + hand-off. The all-reduce push (`ar_mode` 2) needs a multicast workspace and is not driven here: the cells run + `ar_mode` 0, where `y` is the o_proj output and nothing else is written. + + The QKV rows are handed from the GEMV CTAs to the attention CTAs by polling for the 0xFFFF marker the + previous call left: a GEMV output whose bf16 bits are 0xFFFF (a NaN) would never be seen as written. Finite + inputs cannot produce it. + """ + + def __call__( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + qkv_weight: torch.Tensor, + qkv_weight_scale: torch.Tensor, + qkv_out: torch.Tensor, + flags: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + idx_k: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + eps: float, + sm_scale: float, + decode_query_len: int, + tokens_per_cluster: int, + blocks_per_cta: int, + output: torch.Tensor, + mxfp8_data: torch.Tensor, + mxfp8_scale: torch.Tensor, + use_pdl: bool, + o_weight: torch.Tensor, + o_weight_scale: torch.Tensor, + y: torch.Tensor, + ar_workspace: int, + ar_flags: int, + ar_rank: int, + ar_world: int, + ar_max_tokens: int, + ar_mode: int, + ) -> None: + torch.ops.trtllm.minimax_m3_attn_mega( + xq, + xsf, + qkv_weight, + qkv_weight_scale, + qkv_out, + flags, + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + eps, + sm_scale, + decode_query_len, + tokens_per_cluster, + blocks_per_cta, + output, + mxfp8_data, + mxfp8_scale, + use_pdl, + o_weight, + o_weight_scale, + y, + ar_workspace, + ar_flags, + ar_rank, + ar_world, + ar_max_tokens, + ar_mode, + ) + + def reference( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + qkv_weight: torch.Tensor, + qkv_weight_scale: torch.Tensor, + qkv_out: torch.Tensor, + flags: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + idx_k: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + eps: float, + sm_scale: float, + decode_query_len: int, + tokens_per_cluster: int, + blocks_per_cta: int, + output: torch.Tensor, + mxfp8_data: torch.Tensor, + mxfp8_scale: torch.Tensor, + use_pdl: bool, + o_weight: torch.Tensor, + o_weight_scale: torch.Tensor, + y: torch.Tensor, + ar_workspace: int, + ar_flags: int, + ar_rank: int, + ar_world: int, + ar_max_tokens: int, + ar_mode: int, + ) -> MegaState: + """The composition: ``packed = bf16(dequant(xq) @ dequant(W_qkv)^T)``, the fused attention's reference on + those rows, then ``y = bf16(mxfp8(output) @ dequant(W_o)^T)`` (fp64 accumulation, one rounding per GEMV). + + The workspace, the output buffers, the geometry, ``use_pdl`` and the all-reduce arguments are ignored. + """ + t = xq.shape[0] + xq_b, w_b = xq.view(torch.uint8), qkv_weight.view(torch.uint8) + packed = mx_gemm( + xq_b, + unswizzle_scales(xsf, t, HIDDEN // MXFP8_BLOCK), + w_b, + unswizzle_scales(qkv_weight_scale, PACKED_WIDTH, HIDDEN // MXFP8_BLOCK), + ) + attn = reference_attention( + packed, + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + eps, + sm_scale, + decode_query_len, + ) + q8, ex = mxfp8_rows(attn.output) + width = NUM_HEADS * HEAD_DIM + out_y = mx_gemm( + q8, + ex, + o_weight.view(torch.uint8), + unswizzle_scales(o_weight_scale, HIDDEN, width // MXFP8_BLOCK), + ) + return MegaState(packed, attn.output, attn.kv_data, attn.kv_scale, attn.idx_k, out_y) + + def is_valid( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + qkv_weight: torch.Tensor, + qkv_weight_scale: torch.Tensor, + qkv_out: torch.Tensor, + flags: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + idx_k: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + eps: float, + sm_scale: float, + decode_query_len: int, + tokens_per_cluster: int, + blocks_per_cta: int, + output: torch.Tensor, + mxfp8_data: torch.Tensor, + mxfp8_scale: torch.Tensor, + use_pdl: bool, + o_weight: torch.Tensor, + o_weight_scale: torch.Tensor, + y: torch.Tensor, + ar_workspace: int, + ar_flags: int, + ar_rank: int, + ar_world: int, + ar_max_tokens: int, + ar_mode: int, + ) -> None: + check_attention_inputs( + xq.shape[0], + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + decode_query_len, + ) + assert xsf.numel() >= 128 * (HIDDEN // MXFP8_BLOCK), ( + "xsf must hold one 128-row tile of swizzled scales" + ) + parity = workspace_parity(flags, GRID) + assert bool((qkv_out.view(torch.int16)[parity] == -1).all()), ( + "qkv_out's buffer for this call is not at rest (0xFFFF): the attention would read stale QKV rows" + ) + check_handoff_at_rest(mxfp8_data, mxfp8_scale, parity) + assert ar_mode != 2 or (ar_workspace != 0 and ar_flags != 0), ( + "ar_mode 2 needs the exchange's addresses" + ) + + +m3_attn_mega = M3AttnMega() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_dense_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_dense_decode.py new file mode 100644 index 000000000000..38d5f1342c78 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_dense_decode.py @@ -0,0 +1,187 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' decode attention (trtllm-gen over the full page table) with the KV split this target sizes per +request count (``dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH``).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.trtllm_gen_dense_decode import ( # noqa: E501 + minimax_m3_trtllm_gen_dense_decode, +) + +__all__ = ["m3_dense_decode"] + +#: The TP4 dense layers' geometry: 16 query heads and one KV head of dim 128 per rank, an FP8 KV pool. +HEADS = 16 +KV_HEADS = 1 +HEAD_DIM = 128 +#: The split this target hands trtllm-gen at one request (dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH), and a model +#: maximum the in-tree step passes otherwise. +CAPPED = 16384 +MODEL_MAX = 1 << 20 + + +def dense_attention_reference( + q: torch.Tensor, + keys: torch.Tensor, + values: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, +) -> torch.Tensor: + """fp32 causal decode attention: query row ``b * qlen + i`` of request b attends its first + ``seq_lens[b] - qlen + i + 1`` keys. ``keys`` / ``values``: [batch, kv_heads, max_len, head_dim] (the request's + tokens in order). Returns fp32 [total_q, heads, head_dim].""" + batch, kv_heads, max_len, _ = keys.shape + heads = q.shape[1] + group = heads // kv_heads + out = torch.zeros(q.shape, dtype=torch.float32, device=q.device) + pos = torch.arange(max_len, device=q.device) + with true_fp32_matmul(): + for b in range(batch): + full = int(seq_lens[b]) + rows = q[b * decode_query_len : (b + 1) * decode_query_len].float() # [qlen, heads, d] + limit = full - decode_query_len + 1 + torch.arange(decode_query_len, device=q.device) + valid = pos[None, :] < limit[:, None] # [qlen, max_len] + for h in range(kv_heads): + qh = rows[:, h * group : (h + 1) * group] # [qlen, group, d] + logits = torch.einsum("qgd,kd->qgk", qh, keys[b, h].float()) * sm_scale + logits = logits.masked_fill(~valid[:, None, :], float("-inf")) + probs = torch.softmax(logits, dim=-1) + out[ + b * decode_query_len : (b + 1) * decode_query_len, h * group : (h + 1) * group + ] = torch.einsum("qgk,kd->qgd", probs, values[b, h].float()) + return out + + +def gather_paged_kv( + kv_cache_manager, layer_idx: int, block_table: torch.Tensor, seq_lens: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor]: + """The K and V of each request in token order, [batch, kv_heads, max_len, head_dim], from the flat sub-page pool + (``get_kv_subpage_pool``: K of slot s at sub-page s * subpages_per_slot, V one sub-page later).""" + pool, subpages = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + page = pool.shape[2] + pages = (int(seq_lens.max()) + page - 1) // page + slots = block_table[:, :pages].long() + k = pool[slots * subpages] # [batch, pages, kv_heads, page, d] + v = pool[slots * subpages + 1] + batch, _, kv_heads, _, d = k.shape + + def order(t): + return t.permute(0, 2, 1, 3, 4).reshape(batch, kv_heads, pages * page, d) + + return order(k), order(v) + + +class M3DenseDecode(OpWrapper): + """``minimax_m3_trtllm_gen_dense_decode`` as ``MsaDecodeFmha`` calls it for a dense layer, with the max_seq_len + ``msa_decode.dense_decode_max_seq_len`` resolves: this target's per-request-count cap, else the step's bound. + + Decode attention of every query row over its request's full causal context in the paged FP8 pool, written into + ``output`` (BF16 [T, heads, head_dim]) in place. ``max_seq_len`` only sizes trtllm-gen's multi-CTA KV split: + every CTA still covers its share of each request's actual length, which may exceed it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over a 20000-token context, the split sized for 16384", + spec=dict(kv_lens=(20000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request, one row, over a 65536-token context (an AgentX-length cache) at the capped split", + spec=dict(kv_lens=(65536,), qlen=1, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request verifying 1 + 3 rows over a 3000-token context: the cap above the length", + spec=dict(kv_lens=(3000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="two requests keep the model maximum (no cap at two requests)", + spec=dict(kv_lens=(20000, 5000), qlen=4, max_seq_len=MODEL_MAX, capped=False), + ), + ) + + note = """ + The split only reorders the merge of the CTAs' partial softmax sums, so the capped split's output is the uncapped + one's to a few BF16 ulps (the test holds it there), and both are held to the fp32 reference within the FP8-KV + band of the in-tree dense decode test. + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + minimax_m3_trtllm_gen_dense_decode( + q, + kv_cache_manager, + layer_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + max_seq_len=max_seq_len, + max_num_requests=max_num_requests, + ) + return output + + def reference( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + """fp32 attention over the pool's K / V (q rounded to the pool's dtype first, as the kernel runs it). + + ``output``, ``max_seq_len`` and ``max_num_requests`` do not change the result.""" + keys, values = gather_paged_kv(kv_cache_manager, layer_idx, block_table, seq_lens) + if keys.dtype == torch.float8_e4m3fn: + q = q.to(torch.float8_e4m3fn) + return dense_attention_reference(q, keys, values, seq_lens, decode_query_len, sm_scale) + + def is_valid( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> None: + # The multi-CTA KV counters are sized for max_num_requests: more requests overrun them. + assert seq_lens.shape[0] <= max_num_requests, "more requests than the counters cover" + # Each verify row attends a causal prefix of at least one key. + assert bool((seq_lens >= decode_query_len).all()), "a request shorter than its query rows" + + +m3_dense_decode = M3DenseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_draft_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_draft_attention.py new file mode 100644 index 000000000000..6347348d1681 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_draft_attention.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter layer's attention on the TRTLLM backend: RoPE, KV-cache append and causal attention.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + PredefinedAttentionMask, +) +from tensorrt_llm.functional import PositionEmbeddingType, RotaryScalingType + + +class M3DraftAttention(OpWrapper): + """The drafter layer's attention over its fused QKV rows ``qkv`` [T, (q + 2 kv) heads x 128] (bf16): the + backend applies the drafter's RoPE (fused), appends K / V to the drafter layer's paged cache (FP8, the target's + NVFP4 cache keeps its draft layer in FP8) and runs causal attention over the step's live rows (``metadata`` + ``num_tokens``); returns [num_tokens, q heads x 128] bf16. + + ``backend`` (``TrtllmAttention``) and ``metadata`` are the runtime's objects for the drafter layer and step. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="first draft pass of a 4-request verify (4 x 4 rows)", spec=dict(reqs=4, qlen=4)), + Cell(why="a later draft pass, one row per request", spec=dict(reqs=8, qlen=1)), + Cell(why="a context chunk's draft rows", spec=dict(reqs=1, context=512)), + ) + + note = """ + `reference` is exact (float64) causal softmax attention: Q rotated by the backend's NeoX RoPE at each token's + position, over the drafter layer's K / V rows read back from its FP8 cache (dequantized by the backend's + ``kv_scale_quant_orig``). What remains between the two is the FMHA's own arithmetic (Q's rounding to the + kernel's input precision, its P rounding and accumulation order), which the test bounds. + """ + + def __call__(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + num_tokens = metadata.num_tokens + out = backend.forward( + qkv[:num_tokens, :], + None, + None, + metadata, + forward_args=AttentionForwardArgs(attention_mask=PredefinedAttentionMask.CAUSAL), + ) + return out[0] if isinstance(out, tuple) else out + + def reference(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + """The call's [num_tokens, q heads x 128] output, in ``qkv``'s dtype. + + Query token ``j`` of request ``r`` sits at position ``cached_r + j`` (the metadata's cached lengths) and + attends positions up to it. Its Q is rotated here; every K / V row, the step's own included, is read from + the drafter layer's paged cache at the slots the metadata's KV cache manager holds, so this is evaluated + after the call (whose roped, quantized append it then reads back; the test checks that append itself). + Mirrors one KV head per rank and un-scaled NeoX RoPE, the drafter's configuration. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + assert backend.num_kv_heads == 1, "mirrors the one-KV-head-per-rank layout only" + rope = backend.rope_params + assert backend.position_embedding_type == int(PositionEmbeddingType.rope_gpt_neox) + assert rope.scale_type == RotaryScalingType.none and rope.alpha == 1.0 + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + kv_scale = float(backend.kv_scale_quant_orig.reshape(-1)[0]) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = int(metadata.num_tokens) + assert sum(qlens) == total + device = qkv.device + + rot = int(rope.dim) + inv_freq = float(rope.theta) ** ( + -torch.arange(0, rot, 2, dtype=torch.float64, device=device) / rot + ) + pool = manager.get_buffers(layer, "HND") + q = qkv[:total, : heads * dim].to(torch.float64).reshape(total, heads, dim) + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][ + : (length + page - 1) // page + ], + dtype=torch.int64, + device=device, + ) + keys = pool[pages, 0, 0].to(torch.float64).reshape(-1, dim)[:length] * kv_scale + values = pool[pages, 1, 0].to(torch.float64).reshape(-1, dim)[:length] * kv_scale + qpos = cached[row] + torch.arange(n, device=device) + angles = qpos.to(torch.float64)[:, None] * inv_freq[None, :] + cos, sin = angles.cos()[:, None, :], angles.sin()[:, None, :] + rows = q[first : first + n] + x1, x2 = rows[..., : rot // 2], rows[..., rot // 2 : rot] + rows = torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin, rows[..., rot:]], dim=-1) + allowed = torch.arange(length, device=device)[None, :] <= qpos[:, None] + logits = torch.einsum("thd,kd->thk", rows, keys) * sm_scale + logits = logits.masked_fill(~allowed[:, None, :], float("-inf")) + out[first : first + n] = torch.softmax(logits, dim=-1) @ values + first += n + return out.reshape(total, heads * dim).to(qkv.dtype) + + +m3_draft_attention = M3DraftAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_fp8_indexer_qk_norm_rope.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_fp8_indexer_qk_norm_rope.py new file mode 100644 index 000000000000..bcd928fccf5b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_fp8_indexer_qk_norm_rope.py @@ -0,0 +1,114 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The unfused sparse layers' index branch: per-head Gemma RMSNorm + partial NeoX RoPE of [index-Q | index-K], index-Q +out in FP8 E4M3 and index-K inserted into the FP8 index-K cache, in one kernel.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3Fp8IndexerQkNormRope(OpWrapper): + """``qk`` [T, (heads + 1) * 128] BF16 (the index projection's output): returns index-Q [T, heads, 128] E4M3 and + writes index-K (E4M3) at ``out_cache_loc`` of ``index_k_cache`` [pages, 1, 128, 128]. Gemma RMSNorm of each head, + then RoPE on its leading ``rotary_dim`` channels (NeoX). + + The unfused sparse layers (``fuse_qkv_index_projection=False``) run it on the index branch with an FP8 indexer; + the fused layers' producer writes the index cache itself. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, one index head per rank", spec=dict(m=1, heads=1)), + Cell(why="a 48-row decode verify", spec=dict(m=48, heads=1)), + Cell(why="a prefill chunk", spec=dict(m=2048, heads=1)), + ) + + note = """ + `reference` is the in-place fused norm + RoPE in BF16 followed by the E4M3 cast (the kernel rounds through BF16 + before the cast); its index-K is what the kernel writes to the cache. The two kernels may round a handful of + values differently: the test allows fewer than 0.1% of the bytes to differ, as the in-tree test does. + """ + + def __call__( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> torch.Tensor: + return torch.ops.trtllm.minimax_m3_fp8_indexer_qk_norm_rope( + qk, + index_k_cache, + out_cache_loc, + num_heads_q, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + position_ids, + ) + + def reference( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor]: + """(index-Q [T, heads, head_dim] E4M3, index-K [T, head_dim] E4M3): the catalog's in-place kernel on a copy, + then the E4M3 cast.""" + out = fused_qk_norm_rope( + qk.clone(), + num_heads_q, + 1, + 0, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + True, + position_ids, + 1.0, + 0.0, + 0.0, + 1.0, + True, + True, + False, + 0, + 0, + ) + q, k = out.split([num_heads_q * head_dim, head_dim], dim=-1) + return ( + q.reshape(q.shape[0], num_heads_q, head_dim).to(torch.float8_e4m3fn), + k.to(torch.float8_e4m3fn), + ) + + +m3_fp8_indexer_qk_norm_rope = M3Fp8IndexerQkNormRope() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_fused_decode_attn.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_fused_decode_attn.py new file mode 100644 index 000000000000..1e1084aeccdf --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_fused_decode_attn.py @@ -0,0 +1,643 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The fused MSA short-sequence decode attention of this target's sparse layers, and the plain-torch +composition it fuses (shared with the megakernel and c1 entries, which wrap the same attention).""" + +from __future__ import annotations + +import math +from typing import NamedTuple, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Per-rank attention geometry at tp4: 16 query heads, one KV head, one index head, head_dim 128. +NUM_HEADS = 16 +HEAD_DIM = 128 +#: The packed QKV(+index) projection row: [Q x16 | K | V | idxQ | idxK] x 128. +PACKED_WIDTH = (NUM_HEADS + 4) * HEAD_DIM +#: Partial NeoX RoPE over the first 64 dims of every head. +ROTARY_DIM = 64 +#: One MSA block is one 128-token KV page; identity selection covers at most 16 of them. +PAGE = 128 +MAX_BLOCKS = 16 +MAX_SEQ_LEN = MAX_BLOCKS * PAGE +#: NVFP4 block: 16 elements per E4M3 scale. +NVFP4_BLOCK = 16 +#: MXFP8 block of the o_proj hand-off: 32 elements per UE8M0 scale. +MXFP8_BLOCK = 32 + +_E2M1_VALUES = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0) +_E4M3_MAX = 448.0 + + +def e2m1_values(device) -> torch.Tensor: + """The 16 E2M1 codes' values (sign in bit 3).""" + pos = torch.tensor(_E2M1_VALUES, dtype=torch.float32, device=device) + return torch.cat([pos, -pos]) + + +def e4m3_bytes(x: torch.Tensor) -> torch.Tensor: + """Round fp32 to E4M3 the way ``__nv_fp8_e4m3(float)`` does (RN-even, saturating to +-448), as raw bytes.""" + return x.clamp(-_E4M3_MAX, _E4M3_MAX).to(torch.float8_e4m3fn).view(torch.uint8) + + +def e4m3_value(b: torch.Tensor) -> torch.Tensor: + return b.view(torch.float8_e4m3fn).float() + + +def e2m1_codes(x: torch.Tensor) -> torch.Tensor: + """``cvt.rn.satfinite.e2m1x2.f32`` per element: round-to-nearest-even onto {0, .5, 1, 1.5, 2, 3, 4, 6}, + saturating at 6, sign in bit 3 (a negative value that rounds to zero keeps its sign).""" + a = x.abs() + # Ties go to the even code: 0.25 -> 0, 0.75 -> 1.0, 1.25 -> 1.0, 1.75 -> 2.0, 2.5 -> 2.0, 3.5 -> 4.0, 5 -> 4. + code = ( + (a > 0.25).int() + + (a >= 0.75).int() + + (a > 1.25).int() + + (a >= 1.75).int() + + (a > 2.5).int() + + (a >= 3.5).int() + + (a > 5.0).int() + ) + return (code | (torch.signbit(x).int() << 3)).to(torch.uint8) + + +def nvfp4_quantize( + x: torch.Tensor, second_level_scale: torch.Tensor +) -> Tuple[torch.Tensor, torch.Tensor]: + """The MiniMax-M3 producer's NVFP4 recipe for rows of 128: (packed data [N, 64] uint8, E4M3 scales [N, 8]). + + Elements are rounded to bf16 first; each 16-element block's scale is ``e4m3(sls * amax / 6)`` and the codes are + ``e2m1(x / (scale / sls))``. Element ``2i`` is the low nibble of byte ``i``. + """ + n = x.shape[0] + xb = x.to(torch.bfloat16).float().view(n, HEAD_DIM // NVFP4_BLOCK, NVFP4_BLOCK) + sls = second_level_scale.float().reshape(()) + amax = xb.abs().amax(-1) + scale = e4m3_bytes(sls * (amax * (1.0 / 6.0))) + out_scale = torch.where( + amax != 0, 1.0 / (e4m3_value(scale) * (1.0 / sls)), torch.zeros_like(amax) + ) + codes = e2m1_codes(xb * out_scale[..., None]).view(n, HEAD_DIM) + data = codes[:, 0::2] | (codes[:, 1::2] << 4) + return data, scale + + +def nvfp4_dequantize(data: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + """[N, 64] packed E2M1 + [N, 8] E4M3 block scales -> fp32 [N, 128] (the per-tensor scale not applied).""" + table = e2m1_values(data.device) + lo = table[(data & 0xF).long()] + hi = table[(data >> 4).long()] + values = torch.stack([lo, hi], dim=-1).reshape(data.shape[0], HEAD_DIM) + return values * e4m3_value(scale).repeat_interleave(NVFP4_BLOCK, dim=-1) + + +def v_scale_rows(scale_plane: torch.Tensor) -> torch.Tensor: + """A page's V scale plane in the 4x4 token-quad order [token // 4][column][token % 4] -> [128, 8] row-major.""" + return ( + scale_plane.reshape(PAGE // 4, HEAD_DIM // NVFP4_BLOCK, 4) + .permute(0, 2, 1) + .reshape(PAGE, -1) + ) + + +def v_scale_plane(rows: torch.Tensor) -> torch.Tensor: + """Inverse of :func:`v_scale_rows`: [128, 8] row-major scales -> the plane's [128, 8] storage.""" + return rows.reshape(PAGE // 4, 4, HEAD_DIM // NVFP4_BLOCK).permute(0, 2, 1).reshape(PAGE, -1) + + +def norm_rope( + x: torch.Tensor, weight: torch.Tensor, eps: float, cos: torch.Tensor, sin: torch.Tensor +) -> torch.Tensor: + """Per-head Gemma RMSNorm (``x * rsqrt(mean(x^2) + eps) * (1 + w)``) then NeoX RoPE over the first 64 dims. + + x: fp32 [N, 128]; cos / sin: fp32 [N, 32] for each row's position.""" + y = x * (torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * (1.0 + weight.float())) + x1, x2 = y[:, : ROTARY_DIM // 2], y[:, ROTARY_DIM // 2 : ROTARY_DIM] + return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin, y[:, ROTARY_DIM:]], dim=-1) + + +def swizzled_scale_offsets(rows: int, k_blocks: int, device) -> torch.Tensor: + """Byte offset of scale (row, k-block) in the 128x4-swizzled UE8M0 layout: [rows, k_blocks] int64.""" + r = torch.arange(rows, device=device)[:, None] + k = torch.arange(k_blocks, device=device)[None, :] + k_tiles = (k_blocks + 3) // 4 + return ( + (r // 128) * k_tiles * 512 + (k // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + k % 4 + ) + + +def mxfp8_rows(x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: + """The merge's MXFP8 hand-off recipe on bf16 rows [T, K]: (E4M3 bytes [T, K], UE8M0 exponents [T, K / 32]). + + Per 32-element block: ``e = ceil_exp(amax / 448)`` (biased, at most 254), values ``e4m3(x * 2^(127 - e))``.""" + t, k = x.shape + xf = x.float().view(t, k // MXFP8_BLOCK, MXFP8_BLOCK) + amax = xf.abs().amax(-1) + bits = (amax * (1.0 / _E4M3_MAX)).view(torch.int32) + ex = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).int() + ex = ex.clamp(max=254) + scale = torch.where(amax != 0, ((254 - ex) << 23).view(torch.float32), torch.zeros_like(amax)) + q8 = e4m3_bytes(xf * scale[..., None]).view(t, k) + return q8, ex.to(torch.uint8) + + +def mxfp8_dequantize(q8: torch.Tensor, ex: torch.Tensor) -> torch.Tensor: + """E4M3 bytes [T, K] with UE8M0 exponents [T, K / 32] -> fp64 values.""" + scale = torch.exp2(ex.double() - 127.0).repeat_interleave(MXFP8_BLOCK, dim=-1) + return e4m3_value(q8).double() * scale + + +def unswizzle_scales(swizzled: torch.Tensor, rows: int, k_blocks: int) -> torch.Tensor: + """The [rows, k_blocks] UE8M0 bytes of a 128x4-swizzled scale buffer.""" + flat = swizzled.reshape(-1).view(torch.uint8) + return flat[swizzled_scale_offsets(rows, k_blocks, flat.device)] + + +def mx_gemm( + a: torch.Tensor, a_exp: torch.Tensor, w: torch.Tensor, w_exp: torch.Tensor +) -> torch.Tensor: + """bf16 ``dequant(a) @ dequant(w)^T`` of MXFP8 operands (E4M3 bytes, row-major UE8M0 exponents), accumulated in + fp64 and rounded once.""" + return (mxfp8_dequantize(a, a_exp) @ mxfp8_dequantize(w, w_exp).T).to(torch.bfloat16) + + +class AttnState(NamedTuple): + """What the fused attention leaves behind: its bf16 output and the three caches after the new tokens' insert.""" + + output: torch.Tensor + kv_data: torch.Tensor + kv_scale: torch.Tensor + idx_k: torch.Tensor + + +def query_rows(packed, position_ids, q_norm_weight, eps, rotary_cos_sin) -> torch.Tensor: + """fp32 [T, 16, 128]: the query heads of packed rows after Gemma RMSNorm + NeoX RoPE (before E4M3).""" + rows = packed.float().view(packed.shape[0], NUM_HEADS + 4, HEAD_DIM) + pos_ids = position_ids.long() + cos = rotary_cos_sin[pos_ids, 0].float() + sin = rotary_cos_sin[pos_ids, 1].float() + return torch.stack( + [norm_rope(rows[:, h], q_norm_weight, eps, cos, sin) for h in range(NUM_HEADS)], dim=1 + ) + + +def attend_cached( + q, + kv_data, + kv_scale, + block_table, + seq_lens, + k_global_scale, + v_global_scale, + sm_scale, + decode_query_len, +) -> torch.Tensor: + """bf16 [T, 2048]: the identity-selection decode of fp32 queries ``q`` [T, 16, 128] over caches that already + hold the step's tokens. q is rounded to E4M3 (no scale), the operand the kernel's tensor core reads; request + ``r``'s token ``t`` sits at ``pos = seq_lens[r] - qlen + t`` and attends positions ``[0, pos]`` through + ``block_table``: scores ``q . dequant(K) * sm_scale * k_global_scale``, softmax, values + ``dequant(V) * v_global_scale``. fp64 throughout, one bf16 rounding at the end (the kernel's split-K partials + and their LSE merge are exact algebra).""" + q8 = e4m3_value(e4m3_bytes(q)).double() + t_total = q.shape[0] + qlen = int(decode_query_len) + out = torch.empty(t_total, NUM_HEADS, HEAD_DIM, dtype=torch.float64, device=q.device) + k_gs = float(k_global_scale.reshape(-1)[0]) + v_gs = float(v_global_scale.reshape(-1)[0]) + for r, seq_len in enumerate(seq_lens.tolist()): + pages = block_table[r, : math.ceil(seq_len / PAGE)].tolist() + kd = torch.cat([nvfp4_dequantize(kv_data[p, 0, 0], kv_scale[p, 0, 0]) for p in pages]) + vd = torch.cat( + [nvfp4_dequantize(kv_data[p, 1, 0], v_scale_rows(kv_scale[p, 1, 0])) for p in pages] + ) + kd, vd = kd.double() * k_gs, vd.double() * v_gs + for t in range(qlen): + row = r * qlen + t + n = seq_len - qlen + t + 1 + p = torch.softmax(q8[row] @ kd[:n].T * sm_scale, dim=-1) + out[row] = p @ vd[:n] + return out.to(torch.bfloat16).view(t_total, NUM_HEADS * HEAD_DIM) + + +def reference_attention( + packed, + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + eps, + sm_scale, + decode_query_len, +) -> AttnState: + """The composition the fused kernel replaces, in plain torch, on copies of the caches. + + 1. Per token, q heads / k / index-k: Gemma RMSNorm + NeoX RoPE at ``position_ids``; v as is. + 2. k / v -> NVFP4 (``kv_quant_scale[1]`` / ``[2]``) into the token's ``out_cache_loc`` slot (K scales linear, V + scales in 4x4 token quads); index-k -> E4M3 into the index cache. + 3. :func:`attend_cached` over the caches after the insert. + """ + kv_data, kv_scale, idx_k = kv_data.clone(), kv_scale.clone(), idx_k.clone() + idx_bytes = idx_k.view(torch.uint8) + t_total = packed.shape[0] + rows = packed.float().view(t_total, NUM_HEADS + 4, HEAD_DIM) + pos_ids = position_ids.long() + cos = rotary_cos_sin[pos_ids, 0].float() + sin = rotary_cos_sin[pos_ids, 1].float() + q = query_rows(packed, position_ids, q_norm_weight, eps, rotary_cos_sin) + k = norm_rope(rows[:, NUM_HEADS], k_norm_weight, eps, cos, sin) + v = rows[:, NUM_HEADS + 1] + ik = norm_rope(rows[:, NUM_HEADS + 3], index_k_norm_weight, eps, cos, sin) + k_data, k_sf = nvfp4_quantize(k, kv_quant_scale[1]) + v_data, v_sf = nvfp4_quantize(v, kv_quant_scale[2]) + ik_bytes = e4m3_bytes(ik) + for t, slot in enumerate(out_cache_loc[:t_total].tolist()): + page, row = divmod(slot, PAGE) + kv_data[page, 0, 0, row] = k_data[t] + kv_data[page, 1, 0, row] = v_data[t] + kv_scale[page, 0, 0, row] = k_sf[t] + vrows = v_scale_rows(kv_scale[page, 1, 0]) + vrows[row] = v_sf[t] + kv_scale[page, 1, 0] = v_scale_plane(vrows) + idx_bytes[page, 0, row] = ik_bytes[t] + output = attend_cached( + q, + kv_data, + kv_scale, + block_table, + seq_lens, + k_global_scale, + v_global_scale, + sm_scale, + decode_query_len, + ) + return AttnState(output, kv_data, kv_scale, idx_k) + + +def check_attention_inputs( + packed_rows, + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + decode_query_len, +) -> None: + """The inputs the attention kernels accept and answer wrongly (shared by the three entries' ``is_valid``).""" + assert ( + kv_data.dim() == 5 + and tuple(kv_data.shape[1:]) == (2, 1, PAGE, HEAD_DIM // 2) + and kv_data.stride(4) == 1 + and kv_data.stride(3) == HEAD_DIM // 2 + ), "kv_data must be [pages, 2, 1, 128, 64] with contiguous head blocks" + assert ( + kv_scale.dim() == 5 + and tuple(kv_scale.shape[1:]) == (2, 1, PAGE, HEAD_DIM // NVFP4_BLOCK) + and kv_scale.stride(4) == 1 + and kv_scale.stride(3) == HEAD_DIM // NVFP4_BLOCK + ), "kv_scale must be [pages, 2, 1, 128, 8] with contiguous head blocks" + assert ( + idx_k.dim() == 4 and tuple(idx_k.shape[1:]) == (1, PAGE, HEAD_DIM) and idx_k.stride(3) == 1 + ), "idx_k must be [pages, 1, 128, 128] with contiguous rows" + for name, w in ( + ("q_norm_weight", q_norm_weight), + ("k_norm_weight", k_norm_weight), + ("index_k_norm_weight", index_k_norm_weight), + ): + assert w.is_contiguous(), f"{name} must be contiguous (read as 4-element vectors)" + assert rotary_cos_sin.is_contiguous() and tuple(rotary_cos_sin.shape[1:]) == ( + 2, + ROTARY_DIM // 2, + ), "rotary_cos_sin must be contiguous [max_pos, 2, 32]" + for name, t in ( + ("out_cache_loc", out_cache_loc), + ("position_ids", position_ids), + ("seq_lens", seq_lens), + ): + assert t.is_contiguous(), f"{name} must be contiguous" + qlen = int(decode_query_len) + num_reqs = seq_lens.shape[0] + assert packed_rows == num_reqs * qlen, "one row per (request, query token)" + assert position_ids.numel() == packed_rows, "one position per token" + lens = seq_lens.tolist() + assert all(qlen <= n <= MAX_SEQ_LEN for n in lens), ( + f"every seq_len must be in [decode_query_len, {MAX_SEQ_LEN}]: identity selection covers 16 blocks of 128, " + "a longer request is silently cut to them" + ) + assert block_table.shape[1] >= math.ceil(max(lens) / PAGE), ( + "block_table must hold every attended block (missing columns read page 0)" + ) + assert int(position_ids.max()) < rotary_cos_sin.shape[0], "position past the rotary table" + table = block_table.tolist() + slots = out_cache_loc.tolist() + for r, n in enumerate(lens): + for t in range(qlen): + pos = n - qlen + t + want = table[r][pos // PAGE] * PAGE + pos % PAGE + assert slots[r * qlen + t] == want, ( + "out_cache_loc must be the slot block_table gives the token's position seq_len - qlen + t: the " + "attention reads that page's other rows from the cache it loaded before the insert" + ) + + +class M3FusedDecodeAttn(OpWrapper): + """One launch after the packed QKV GEMM of a sparse layer in short-sequence decode. + + Fuses q / k / index-k Gemma RMSNorm + NeoX partial RoPE, the NVFP4 K/V and FP8 index-K cache insert, the + block-sparse decode over every attended block (selection is the identity: at most 16 blocks of 128 tokens) and + the split-K LSE merge, writing the bf16 output and its MXFP8 copy for the o_proj GEMM. + + ``(tokens_per_cluster, blocks_per_cta)`` is the launch geometry; the target passes + ``ops.m3_fused_attn.default_geometry(qlen, num_requests)``. Returns None: the cells read ``output``, + ``mxfp8_data`` / ``mxfp8_scale`` and the three caches. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # Every compiled geometry, each at a step shape the target routes to it (decode_query_len x requests, at most + # 16 tokens). seq_lens is a per-request list; positions are seq_len - qlen + t. + CELLS: tuple[Cell, ...] = ( + Cell( + why="plain decode, one request, a few blocks: geometry (1, 1, 1)", + spec=dict(qlen=1, seq_lens=(300,), geometry=(1, 1)), + ), + Cell( + why="plain decode, 6 requests: the widest one-block-per-CTA batch; full 16 blocks and a page's 1st row", + spec=dict(qlen=1, seq_lens=(2048, 129, 1, 700, 1500, 256), geometry=(1, 1)), + ), + Cell( + why="plain decode, 8 requests: KV blocks paired per CTA (8-CTA clusters), geometry (1, 1, 2)", + spec=dict(qlen=1, seq_lens=(2048, 33, 128, 129, 1024, 1900, 640, 7), geometry=(1, 2)), + ), + Cell( + why="plain decode at the 16-token cap: 16 paired-block clusters", + spec=dict(qlen=1, seq_lens=tuple(range(97, 2048, 125))[:16], geometry=(1, 2)), + ), + Cell( + why="Eagle3 verify of 1 draft (qlen 2), 2 requests: geometry (2, 1, 1); a draft crossing a page", + spec=dict(qlen=2, seq_lens=(129, 900), geometry=(1, 1)), + ), + Cell( + why="qlen 2 at 8 requests: geometry (2, 1, 2)", + spec=dict(qlen=2, seq_lens=(2, 2048, 130, 257, 640, 1000, 1500, 64), geometry=(1, 2)), + ), + Cell( + why="qlen 3 at 5 requests (15 tokens): geometry (3, 1, 1)", + spec=dict(qlen=3, seq_lens=(3, 2048, 130, 385, 777), geometry=(1, 1)), + ), + Cell( + why="qlen 4 at 4 requests: geometry (4, 1, 1)", + spec=dict(qlen=4, seq_lens=(4, 2048, 131, 1030), geometry=(1, 1)), + ), + Cell( + why="qlen 5 at 3 requests: geometry (5, 1, 1)", + spec=dict(qlen=5, seq_lens=(5, 1100, 2048), geometry=(1, 1)), + ), + Cell( + why="qlen 6, one request (the c1 shape when c1 does not apply): geometry (6, 1, 1)", + spec=dict(qlen=6, seq_lens=(1283,), geometry=(1, 1)), + ), + Cell( + why="qlen 6 at 2 requests: geometry (6, 1, 2)", + spec=dict(qlen=6, seq_lens=(6, 2048), geometry=(1, 2)), + ), + Cell( + why="qlen 7, two requests: geometry (7, 1, 1)", + spec=dict(qlen=7, seq_lens=(132, 2000), geometry=(1, 1)), + ), + Cell( + why="qlen 8 at 2 requests (the 16-token cap): geometry (8, 1, 1)", + spec=dict(qlen=8, seq_lens=(8, 1029), geometry=(1, 1)), + ), + # Every other (query rows per request, requests) the decode dispatch sends here (dispatch.FUSED_DECODE_ATTN): + # lengths spread over the 16-block window, the geometry the target picks for the shape. + *( + Cell( + why=f"qlen {qlen} at {reqs} requests: a decode shape between the ones above", + spec=dict( + qlen=qlen, + seq_lens=tuple( + qlen + (r * 389 + 97 * qlen) % (2049 - qlen) for r in range(reqs) + ), + geometry=geometry, + ), + ) + for qlen, reqs, geometry in ( + (1, 2, (1, 1)), + (1, 3, (1, 1)), + (1, 4, (1, 1)), + (1, 5, (1, 1)), + (1, 7, (1, 2)), + (1, 9, (1, 2)), + (1, 10, (1, 2)), + (1, 11, (1, 2)), + (1, 12, (1, 2)), + (1, 13, (1, 2)), + (1, 14, (1, 2)), + (1, 15, (1, 2)), + (2, 1, (1, 1)), + (2, 3, (1, 1)), + (2, 4, (1, 2)), + (2, 5, (1, 2)), + (2, 6, (1, 2)), + (2, 7, (1, 2)), + (3, 1, (1, 1)), + (3, 2, (1, 1)), + (3, 3, (1, 1)), + (3, 4, (1, 1)), + (4, 1, (1, 1)), + (4, 2, (1, 1)), + (4, 3, (1, 1)), + (5, 1, (1, 1)), + (5, 2, (1, 1)), + (7, 1, (1, 1)), + (8, 1, (1, 1)), + ) + ), + ) + + note = """ + Only the variants the target launches are compiled: the geometries above, each with the MXFP8 hand-off (the + target always passes it). Q is quantized to E4M3 with no scale (`kv_quant_scale[0]` is not read): its operand + on the tensor core is the E4M3 value, which `reference` reproduces. + + Not certified: a KV pool whose pages interleave other layers (any `stride(0)` / `stride(1)` that is a multiple + of 16 is accepted; the cells use a contiguous pool), `position_ids` other than `seq_len - qlen + t` (RoPE reads + `position_ids`, placement reads `seq_lens`; the target passes equal values), and `use_pdl=False` timing. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + idx_k: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + eps: float, + sm_scale: float, + decode_query_len: int, + tokens_per_cluster: int, + blocks_per_cta: int, + output: torch.Tensor, + mxfp8_data: torch.Tensor, + mxfp8_scale: torch.Tensor, + use_pdl: bool, + ) -> None: + torch.ops.trtllm.minimax_m3_fused_decode_attn( + packed, + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + eps, + sm_scale, + decode_query_len, + tokens_per_cluster, + blocks_per_cta, + output, + mxfp8_data, + mxfp8_scale, + use_pdl, + ) + + def reference( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + idx_k: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + eps: float, + sm_scale: float, + decode_query_len: int, + tokens_per_cluster: int, + blocks_per_cta: int, + output: torch.Tensor, + mxfp8_data: torch.Tensor, + mxfp8_scale: torch.Tensor, + use_pdl: bool, + ) -> AttnState: + """The bf16 output and the caches after the insert, as :func:`reference_attention` computes them. + + The geometry, the output buffers and ``use_pdl`` do not change what is computed and are ignored; the + MXFP8 hand-off is :func:`mxfp8_rows` of the bf16 output, which the test checks on the op's own output. + """ + return reference_attention( + packed, + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + eps, + sm_scale, + decode_query_len, + ) + + def is_valid( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + idx_k: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + eps: float, + sm_scale: float, + decode_query_len: int, + tokens_per_cluster: int, + blocks_per_cta: int, + output: torch.Tensor, + mxfp8_data: torch.Tensor, + mxfp8_scale: torch.Tensor, + use_pdl: bool, + ) -> None: + check_attention_inputs( + packed.shape[0], + kv_data, + kv_scale, + idx_k, + out_cache_loc, + position_ids, + block_table, + seq_lens, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + decode_query_len, + ) + k_blocks = output.shape[1] // MXFP8_BLOCK + needed = math.ceil(packed.shape[0] / 128) * ((k_blocks + 3) // 4) * 512 + assert mxfp8_scale.numel() >= needed, ( + "mxfp8_scale must hold the swizzled scales of every output row tile" + ) + + +m3_fused_decode_attn = M3FusedDecodeAttn() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_index_score.py new file mode 100644 index 000000000000..89deacf68826 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_index_score.py @@ -0,0 +1,167 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# PyTorch oracle ported from vLLM (Apache-2.0), _reference_decode_index_score: +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/tests/kernels/attention/test_minimax_m3.py#L188 +"""The MSA index scorer of a sparse decode step: per (index head, query token, KV block) the causal max of Q . K over +the block's 128 index-K positions, with the pre-wait schedule of ``ops/m3_index_score``.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_index_score"] + +PAGE = 128 +HEAD_DIM = 128 +#: CTAs per request (each scores blocks split, split + SPLIT_K, ...; the scores do not depend on it). +SPLIT_K = 512 + + +def compiled_scorer(key: tuple): + """The compiled kernel of ``key`` (dtype, heads, query length, head dim, split_k, early), compiling it on a miss + (the target compiles every key it can take before warmup: ``layers.prebuild_index_scorer``).""" + from ..m3_index_score import IndexScoreRunner + + if key not in IndexScoreRunner.kernel_cache: + IndexScoreRunner._compile(*key) + return IndexScoreRunner.kernel_cache[key] + + +class M3IndexScore(OpWrapper): + """``score[h, t, j]`` (fp32, a transposed view of the [heads, blocks, tokens] selector buffer) = the max over the + positions ``p`` of block ``j`` of request ``b = t // qlen`` with ``p <= seq_lens[b] - qlen + t % qlen`` of + ``idx_q[t, h] . index_k[block_table[b, j], p % 128]``, for ``j < ceil(seq_lens[b] / 128)`` (no other entry is + written). FP8 or BF16 q and index-K, fp32 products and sums. + + ``early`` (the target's one-request kernel) issues the Q and first K tiles right after the grid wait from a + pre-wait length hint and prefetches one older block per CTA into L2 before the wait; otherwise every older block of + the CTA is prefetched. Neither changes a score. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over 20000 tokens: the early-issue kernel", + spec=dict(seq_lens=(20000,), qlen=4, early=True), + ), + Cell( + why="one request, one row, 65536 tokens: early issue, 512 blocks on 512 CTAs", + spec=dict(seq_lens=(65536,), qlen=1, early=True), + ), + Cell( + why="one request of 129 tokens: the early tiles of most CTAs lie past the length and are drained", + spec=dict(seq_lens=(129,), qlen=4, early=True), + ), + Cell( + why="one request longer than 512 blocks (the CTAs loop): early issue", + spec=dict(seq_lens=(70000,), qlen=4, early=True), + ), + Cell( + why="one request verifying 1 + 7 rows over 20000 tokens: the early-issue kernel", + spec=dict(seq_lens=(20000,), qlen=8, early=True), + ), + Cell( + why="two requests verifying 1 + 7 rows: the prefetching kernel", + spec=dict(seq_lens=(20000, 3000), qlen=8, early=False), + ), + Cell( + why="two requests: the prefetching kernel", + spec=dict(seq_lens=(20000, 3000), qlen=4, early=False), + ), + Cell( + why="eight requests of mixed lengths: the prefetching kernel", + spec=dict( + seq_lens=(300, 1500, 2049, 4096, 5000, 7777, 8192, 33001), qlen=4, early=False + ), + ), + ) + + note = """ + The products of FP8 (or BF16) values are exact in fp32 and the kernel sums 128 of them in its k-split order, so it + differs from the reference matmul by a few fp32 ulps; it is held to the in-tree scorer bit for bit. + """ + + def __call__( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + _, num_heads, head_dim = idx_q.shape + key = (idx_q.dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + compiled_scorer(key)(idx_q, index_k, block_table, score, seq_lens) + return score + + def reference( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + """The fp32 [heads, total_q, blocks] scores (``-inf`` where nothing is written). ``score``, ``split_k`` and + ``early`` do not change them.""" + total_q, heads, _ = idx_q.shape + qlen = max_decode_query_len + out = torch.full( + (heads, total_q, score.shape[2]), + -float("inf"), + dtype=torch.float32, + device=idx_q.device, + ) + with true_fp32_matmul(): + for b, seq in enumerate(seq_lens.tolist()): + nblocks = (seq + PAGE - 1) // PAGE + if nblocks == 0: + continue + t0 = b * qlen + q = idx_q[t0 : t0 + qlen].float() + k = index_k[block_table[b, :nblocks].long()].reshape(nblocks * PAGE, -1).float() + s = torch.einsum("qhd,kd->hqk", q, k) + q_pos = seq - qlen + torch.arange(qlen, device=idx_q.device) + k_pos = torch.arange(k.shape[0], device=idx_q.device) + s.masked_fill_(k_pos[None, :] > q_pos[:, None], -float("inf")) + out[:, t0 : t0 + qlen, :nblocks] = s.reshape(heads, qlen, nblocks, PAGE).amax(dim=3) + return out + + def is_valid( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> None: + # The block-table row and the score row are read and written up to ceil(seq_len / 128) with no bound. + nblocks = (seq_lens + PAGE - 1) // PAGE + assert bool((nblocks <= block_table.shape[1]).all()), ( + "a request runs past its block-table row" + ) + assert bool((nblocks <= score.shape[2]).all()), "a request runs past the score buffer" + assert idx_q.shape[0] == seq_lens.shape[0] * max_decode_query_len, ( + "idx_q must hold max_decode_query_len rows per request" + ) + + +m3_index_score = M3IndexScore() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_msa_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_msa_attention.py new file mode 100644 index 000000000000..dac983d8982b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_msa_attention.py @@ -0,0 +1,224 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MiniMax-M3 MSA attention backend's step: KV write, sparse top-k block selection, FMHA.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + SparseBackendForwardArgs, +) + +from .m3_sparse_decode_lc import dequant_nvfp4_pages + + +class M3MsaAttention(OpWrapper): + """One layer's attention core on the MSA backend (``MiniMaxM3MsaSparseAttention``), written into ``output`` + [T, heads * 128] (bf16, the step's live rows). + + * ``k`` / ``v`` given (the dense layers): written into the layer's paged main cache first (NVFP4 with + ``kv_scale_orig_quant``, else FP8 / BF16 as the pool is). None: a producer already wrote the caches. + * ``sparse``: the indexer scores the layer's index-K cache (already written) with ``idx_q`` and publishes the + top-k blocks; the FMHA attends those blocks. Otherwise the FMHA attends the whole page table. + + The FMHA dispatches on the step's metadata (context rows: the MSA prefill FMHA; generation rows: the MSA decode + FMHA, the dense layers' trtllm-gen decode at their capped KV split). ``backend`` and ``metadata`` are the runtime's + objects for this layer and step. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="dense layer, a prefill chunk, FP8 pool", + spec=dict(layer="dense", phase="context", tokens=1024), + ), + Cell( + why="dense layer, a decode step of 4 requests", + spec=dict(layer="dense", phase="generation", reqs=4), + ), + Cell( + why="sparse layer, a prefill chunk beyond 16 blocks, NVFP4 pool", + spec=dict(layer="sparse", phase="context", tokens=4096), + ), + Cell( + why="sparse layer, a mixed step (context + generation rows)", + spec=dict(layer="sparse", phase="mixed", tokens=512, reqs=2), + ), + ) + + note = """ + `reference` is exact (float64) softmax attention over the cache rows the step attends, read back from the paged + cache the call wrote (or a producer wrote before it) and dequantized as the FMHA dequantizes them: FP8 rows as + E4M3, NVFP4 rows as E2M1 x E4M3 block scale x the per-tensor ``kv_scale_quant_orig``. A sparse layer's top-k blocks + are re-selected from ``idx_q`` and the index-K cache with the backend's rule (per-block max over the causal + keys, the query's own block forced in, ties to the lower block). On an FP8 pool the FMHA takes Q as a plain E4M3 + cast, which the reference applies too. What remains between the two is the FMHA's own arithmetic (its P + rounding and accumulation order), which the test bounds. + """ + + def __call__( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + if k is not None: + backend.write_layer_caches( + k, v, None, metadata, kv_scale_orig_quant=kv_scale_orig_quant + ) + if sparse: + blocks = backend.run_indexer(idx_q, None, metadata, idx_k_prewritten=True) + args = AttentionForwardArgs( + output=output, sparse_backend_args=SparseBackendForwardArgs(topk_indices=blocks) + ) + else: + args = AttentionForwardArgs(output=output) + args.kv_scale_orig_quant = kv_scale_orig_quant + args.kv_scale_quant_orig = kv_scale_quant_orig + backend.forward(q, None, None, metadata, forward_args=args) + return output + + def reference( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + """``output`` as the call is supposed to write it, [T, heads * 128] in ``output``'s dtype. + + Every attended row -- the step's own included -- is read from the layer's paged cache, at the slots the + metadata's KV cache manager holds for each request, so this is evaluated after the call (whose write of + ``k`` / ``v`` it then reads back). ``k``, ``v`` and ``kv_scale_orig_quant`` are therefore not read; the + test checks the write itself. Query token ``j`` of request ``r`` sits at position ``cached_r + j`` and + attends positions up to it; on a sparse layer only those in its top-k blocks. Mirrors one KV head per rank, + the target's layout. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + assert backend.num_kv_heads == 1, "mirrors the one-KV-head-per-rank layout only" + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = sum(qlens) + assert q.shape[0] == total and output.shape[0] == total + + pool = manager.get_buffers(layer, "HND") + nvfp4 = manager.is_nvfp4_layer(layer) + if nvfp4: + scale_pool = manager.get_block_scale_buffers(layer, "HND") + k_global = float(kv_scale_quant_orig.reshape(-1)[1]) + v_global = float(kv_scale_quant_orig.reshape(-1)[2]) + q_rows = q.reshape(total, heads, dim) + if q_rows.dtype != torch.float8_e4m3fn and pool.dtype == torch.float8_e4m3fn: + # The FP8 FMHA shares one dtype across Q / K / V: Q is cast to E4M3, unscaled. + q_rows = q_rows.to(torch.float8_e4m3fn) + q64 = q_rows.to(torch.float64) + if sparse: + config = backend.m3_config + idx_cache = metadata.msa_idx_k_cache(layer) + idx64 = idx_q.reshape(total, config.num_index_heads, config.sparse_index_dim).to( + torch.float64 + ) + + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=q.device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + blocks = (length + page - 1) // page + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][:blocks], + dtype=torch.int64, + device=q.device, + ) + if nvfp4: + keys = dequant_nvfp4_pages( + pool[:, 0].view(torch.uint8), scale_pool[:, 0], pages, quad_scales=False + ) + values = dequant_nvfp4_pages( + pool[:, 1].view(torch.uint8), scale_pool[:, 1], pages, quad_scales=True + ) + keys, values = keys * k_global, values * v_global + else: + keys = pool[pages, 0, 0].to(torch.float64) + values = pool[pages, 1, 0].to(torch.float64) + keys = keys.reshape(-1, dim)[:length] + values = values.reshape(-1, dim)[:length] + qpos = cached[row] + torch.arange(n, device=q.device) + allowed = torch.arange(length, device=q.device)[None, :] <= qpos[:, None] + if sparse: + idx_keys = idx_cache[pages, 0].to(torch.float64).reshape(-1, dim)[:length] + chosen = _selected_blocks( + idx64[first : first + n], idx_keys, allowed, qpos, page, config + ) + allowed &= chosen.repeat_interleave(page, dim=1)[:, :length] + for lo in range(0, n, 256): + hi = min(n, lo + 256) + logits = torch.einsum("thd,kd->thk", q64[first + lo : first + hi], keys) * sm_scale + logits = logits.masked_fill(~allowed[lo:hi, None, :], float("-inf")) + out[first + lo : first + hi] = torch.softmax(logits, dim=-1) @ values + first += n + return out.reshape(total, heads * dim).to(output.dtype) + + +def _selected_blocks( + idx_q: torch.Tensor, + idx_keys: torch.Tensor, + allowed: torch.Tensor, + qpos: torch.Tensor, + page: int, + config, +) -> torch.Tensor: + """The blocks each query token of one request attends, [n, blocks] bool. + + A block's score is the maximum, over the KV head's index heads and the block's causally visible keys, of + ``idx_q . idx_k``. Of the token's ``ceil((pos + 1) / page)`` valid blocks the first ``init_blocks`` and the last + ``local_blocks`` are forced in, and the ``topk`` highest-scoring are kept, ties to the lower block (the + backend's ``select_blocks_from_maxscore`` rule). The positive score scale the scorers apply does not change the + ranking, so none is applied. + """ + n, length = allowed.shape + blocks = (length + page - 1) // page + scores = torch.einsum("thd,kd->thk", idx_q, idx_keys).amax(dim=1) + scores = scores.masked_fill(~allowed, float("-inf")) + padded = torch.full((n, blocks * page), float("-inf"), dtype=scores.dtype, device=scores.device) + padded[:, :length] = scores + block_scores = padded.view(n, blocks, page).amax(dim=-1) + ids = torch.arange(blocks, device=scores.device)[None, :] + n_valid = torch.div(qpos + page, page, rounding_mode="floor")[:, None] + block_scores = block_scores.masked_fill(ids >= n_valid, float("-inf")) + forced = (ids < torch.clamp(n_valid, max=config.init_blocks)) | ( + (ids >= n_valid - config.local_blocks) & (ids < n_valid) + ) + block_scores = block_scores.masked_fill(forced, float("inf")) + order = torch.sort(block_scores, dim=1, descending=True, stable=True).indices[:, : config.topk] + keep = torch.arange(order.shape[1], device=scores.device)[None, :] < n_valid + chosen = torch.zeros(n, blocks, dtype=torch.bool, device=scores.device) + rows = torch.arange(n, device=scores.device)[:, None].expand_as(order) + chosen[rows[keep], order[keep]] = True + return chosen + + +m3_msa_attention = M3MsaAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_producer.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_producer.py new file mode 100644 index 000000000000..9b63def4c494 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_producer.py @@ -0,0 +1,202 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' NVFP4 QKV + indexer RMSNorm / RoPE / KV-insert producer, with its constant loads staged +before the PDL grid-dependency wait and an early dependents trigger (``kernels/minimaxM3/producerPw.cu``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +__all__ = ["m3_producer"] + +HEAD_DIM = 128 +ROTARY_DIM = 64 +PAGE = 128 +#: Query heads per rank at TP4 (MiniMax-M3: 64 query heads, 1 KV head and 1 index head per rank). +HEADS_Q = 16 +#: NVFP4 block: 16 values share one E4M3 scale. +FP4_BLOCK = 16 +E2M1_VALUES = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0) + + +def _norm_rope( + x: torch.Tensor, weight: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor, eps: float +): + """Gemma RMSNorm (``x * rsqrt(mean(x^2) + eps) * (1 + w)``) then NeoX RoPE on the first 64 dims, in fp32. + x [T, 128] fp32; cos / sin [T, 32].""" + y = x * torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) * (1.0 + weight.float()) + lo, hi, rest = y[:, :32], y[:, 32:64], y[:, 64:] + return torch.cat([lo * cos - hi * sin, hi * cos + lo * sin, rest], dim=-1) + + +def decode_nvfp4_token( + data: torch.Tensor, scales: torch.Tensor, quant_scale: float +) -> torch.Tensor: + """fp32 [128] of one token's packed E2M1 row (64 bytes, low nibble first) and its 8 E4M3 block scales.""" + table = torch.tensor(E2M1_VALUES, dtype=torch.float32, device=data.device) + nib = torch.stack([data & 0xF, data >> 4], dim=-1).reshape(-1).long() + vals = table[nib & 7] * torch.where(nib >= 8, -1.0, 1.0) + s = scales.view(torch.float8_e4m3fn).float() / quant_scale + return vals * s.repeat_interleave(FP4_BLOCK) + + +def scale_index(within_page: int, plane: int) -> torch.Tensor: + """Positions of a token's 8 block scales in its page's [128 * 8] scale plane: row-major for K, the 4-token + swizzle for V.""" + cols = torch.arange(8) + if plane == 0: + return within_page * 8 + cols + return (within_page // 4) * 32 + cols * 4 + within_page % 4 + + +class M3Producer(OpWrapper): + """The NVFP4 QKV + indexer producer of a sparse layer: ``minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert`` for one + KV head, one index head, head dim 128 and rotary dim 64, with the same arguments and outputs bit for bit. + + ``packed`` [T, (heads_q + 4) * 128] BF16 holds each token's q heads, k, v, index-q and index-k. Every head but v + gets the Gemma RMSNorm with its weight and NeoX RoPE on its first 64 dims (``rotary_cos_sin[position]``). Returns + (q FP8 [T, heads_q, 128], index-q FP8 [T, 1, 128]); writes the token's slot ``out_cache_loc[t]`` (skipped when + negative or past the pool): k and v as NVFP4 (16-value blocks, E4M3 scales against ``kv_quant_scale[1]`` / + ``[2]``, the V scales in the 4-token swizzle) and index-k as FP8. ``early`` triggers the dependent kernel on + entry (its outputs are read only after the dependent's own grid wait). + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token (c1 without speculation)", spec=dict(tokens=1, scale=1.0)), + Cell(why="a c1 verify: 4 tokens", spec=dict(tokens=4, scale=3.0)), + Cell(why="a K7 c1 verify: 8 tokens", spec=dict(tokens=8, scale=3.0)), + Cell(why="a c4 verify: 16 tokens, small activations", spec=dict(tokens=16, scale=1e-3)), + Cell( + why="a c8 verify: 32 tokens, large activations, a skipped and an out-of-pool slot", + spec=dict(tokens=32, scale=40.0, bad_slots=True), + ), + Cell(why="the widest decode step before R1: 48 tokens", spec=dict(tokens=48, scale=1.0)), + Cell( + why="the widest decode step (R1): a K7 c8 verify, 64 tokens, a skipped and an out-of-pool slot", + spec=dict(tokens=64, scale=40.0, bad_slots=True), + ), + Cell( + why="a prefill chunk of 1000 tokens, early trigger off", + spec=dict(tokens=1000, scale=1.0, early=False), + ), + ) + + note = """ + The kernel is the native producer's arithmetic, compiled with the library's --use_fast_math, so it is held to that + kernel bit for bit on every output and cache byte. The torch reference states the math in fp32: the FP8 outputs are + held to it within one E4M3 step, and the NVFP4 K/V, decoded, within one E2M1 step of their block scale. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + q, index_q = torch.ops.trtllm.minimax_m3_producer_pw( + packed, + kv_data, + kv_scale, + index_k, + out_cache_loc, + kv_quant_scale, + eps, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + rotary_cos_sin, + position_ids, + heads_q, + pdl, + early, + ) + return q, index_q + + def reference( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + ) -> dict[str, torch.Tensor]: + """fp32 values before their quantization: "q" [T, heads_q, 128], "index_q" [T, 128], and per token "k", "v", + "index_k" [T, 128] (the values written to the token's slot). The caches, ``pdl`` and ``early`` do not change + them.""" + t = packed.shape[0] + x = packed.float().view(t, heads_q + 4, HEAD_DIM) + rope = rotary_cos_sin[position_ids[:t].long()] + cos, sin = rope[:, 0], rope[:, 1] + q = torch.stack( + [_norm_rope(x[:, h], q_weight, cos, sin, eps) for h in range(heads_q)], dim=1 + ) + return { + "q": q, + "k": _norm_rope(x[:, heads_q], k_weight, cos, sin, eps), + "v": x[:, heads_q + 1], + "index_q": _norm_rope(x[:, heads_q + 2], index_q_weight, cos, sin, eps), + "index_k": _norm_rope(x[:, heads_q + 3], index_k_weight, cos, sin, eps), + } + + def is_valid( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + ) -> None: + t = packed.shape[0] + # The RoPE row is read at the position with no bound. + pos = position_ids[:t] + assert bool(((pos >= 0) & (pos < rotary_cos_sin.shape[0])).all()), ( + "position_ids must index rotary_cos_sin" + ) + # Two tokens writing one slot race. + slots = out_cache_loc[:t] + live = slots[(slots >= 0) & (slots < kv_data.shape[0] * PAGE)] + assert live.unique().numel() == live.numel(), "out_cache_loc must not repeat a slot" + + +m3_producer = M3Producer() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_qk_norm_rope_fp8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..6493c4a45025 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_qk_norm_rope_fp8.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' per-head Gemma RMSNorm + partial NeoX RoPE of the packed [Q | K | V], into a fresh FP8 copy.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3QkNormRopeFp8(OpWrapper): + """``fused_qk_norm_rope`` written out of place in FP8 E4M3: Q and K heads normed and roped in fp32 and rounded + once to E4M3 (saturating at +/-448), V cast, ``qkv`` left untouched. Same arguments as the catalog's in-place + entry. + + The dense layers (0-2) feed the FP8 main KV cache with it: on every step of an FP8 cache, and on the decode steps + of a hybrid NVFP4 cache (whose dense layers sit in its FP8 pool). + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, tp4 heads (16 q, 1 kv)", spec=dict(m=1, q=16, kv=1)), + Cell(why="a 48-row decode verify", spec=dict(m=48, q=16, kv=1)), + Cell(why="a prefill chunk on an FP8 cache", spec=dict(m=2048, q=16, kv=1)), + ) + + note = """ + `reference` is the in-place kernel followed by a cast to E4M3. V matches it bit for bit; Q and K round once from + fp32 where the composition rounds through bf16 first, so they are within one E4M3 step of it. + """ + + def __call__( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + return torch.ops.trtllm.fused_qk_norm_rope_to_fp8( + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + + def reference( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + """The catalog's in-place kernel on a copy, then the E4M3 cast (V exact; Q and K within one E4M3 step).""" + out = fused_qk_norm_rope( + qkv.clone(), + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + return out.to(torch.float8_e4m3fn) + + +m3_qk_norm_rope_fp8 = M3QkNormRopeFp8() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_qkv_insert.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_qkv_insert.py new file mode 100644 index 000000000000..91927ab0bc39 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_qkv_insert.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' generic horizontal producer: packed QKV/index rows -> FP8 Q / index-Q, K / V / index-K into +the paged caches.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3QkvInsert(OpWrapper): + """The packed ``[Q | K | V | index-Q | index-K]`` bf16 rows of a sparse layer: per-head Gemma RMSNorm and the + partial NeoX RoPE of Q, K, index-Q and index-K, K / V written to the main paged cache at ``out_cache_loc`` and + index-K to the FP8 index cache; returns (Q, index-Q) in FP8 E4M3 [M, heads, 128]. + + ``kv_scale`` given: the main cache is NVFP4 (packed E2M1 data [pages, 2, kv, 128, 64] uint8 + E4M3 scale pool + [pages, 2, kv, 128, 8], ``kv_quant_scale`` the fp32 (Q, K, V) quantization scales); None: FP8 E4M3 + [pages, 2, kv, 128, 128]. Token-major and step-type agnostic. ``m3_producer`` takes the NVFP4 cache at the tp4 + heads; this entry takes the FP8 cache and NVFP4 pools outside that layout. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="FP8 main cache, a prefill chunk", spec=dict(m=1024, cache="fp8")), + Cell(why="FP8 main cache, one decode token", spec=dict(m=1, cache="fp8")), + ) + + note = """ + The arithmetic is the CUDA producer's (fp32 norm and RoPE with bf16 roundings, then the E4M3 casts); `reference` + returns the fp32 values, and the test holds the FP8 outputs and cache rows within one E4M3 step of them. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + common = ( + num_heads, + num_kv_heads, + num_index_heads, + head_dim, + rotary_dim, + eps, + q_norm, + k_norm, + index_q_norm, + index_k_norm, + rotary_cos_sin, + position_ids, + ) + if kv_scale is not None: + return torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, kv_scale, index_k_cache, out_cache_loc, kv_quant_scale, *common + ) + return torch.ops.trtllm.minimax_m3_fp8_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, index_k_cache, out_cache_loc, *common + ) + + def reference( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + """The fp32 values the producer rounds (a new dict, nothing written): Gemma RMSNorm + NeoX partial RoPE of + Q [M, heads, D], index-Q [M, index heads, D], K [M, kv heads, D] and index-K [M, D]; V [M, kv heads, D] as + packed. The cache rows at ``out_cache_loc`` and the returned FP8 Q / index-Q are these values cast.""" + m = packed.shape[0] + sizes = [ + num_heads * head_dim, + num_kv_heads * head_dim, + num_kv_heads * head_dim, + num_index_heads * head_dim, + head_dim, + ] + q, k, v, iq, ik = packed.float().split(sizes, dim=-1) + cos, sin = rotary_cos_sin[position_ids.long()].unbind(1) + half = rotary_dim // 2 + + def norm_rope(x, w, heads): + x = x.reshape(m, heads, head_dim) + x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * (1.0 + w.float()) + x1, x2, rest = x[..., :half], x[..., half:rotary_dim], x[..., rotary_dim:] + c, s = cos.unsqueeze(1), sin.unsqueeze(1) + return torch.cat([x1 * c - x2 * s, x2 * c + x1 * s, rest], dim=-1) + + return { + "q": norm_rope(q, q_norm, num_heads), + "index_q": norm_rope(iq, index_q_norm, num_index_heads), + "k": norm_rope(k, k_norm, num_kv_heads), + "v": v.reshape(m, num_kv_heads, head_dim), + "index_k": norm_rope(ik, index_k_norm, 1)[:, 0], + } + + +m3_qkv_insert = M3QkvInsert() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_select_blocks.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_select_blocks.py new file mode 100644 index 000000000000..6bf1764bfc0d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_select_blocks.py @@ -0,0 +1,82 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MSA top-k block selection from per-KV-head block scores (the built-in selector).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SelectBlocks(OpWrapper): + """Per query row the ``topk`` highest-scoring KV blocks of ``max_score_kv`` [kv_heads, n_blocks, total_q] (fp32), + with the first ``init_blocks`` and the last ``local_blocks`` valid blocks forced in and blocks at or beyond the + row's ``n_valid_blocks`` excluded; [total_q, kv_heads, topk] int32 ascending ids, -1 padded. ``head_major_output`` + backs the result head-major so ``permute(1, 0, 2)`` is contiguous. + + The decode dispatch runs it where the long-row selector does not apply: requests of 17..128 KV blocks. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="eight decode rows of 17..128 blocks", + spec=dict(total_q=8, n_blocks=128, mixed=True), + ), + Cell( + why="one decode row at the 128-block edge", + spec=dict(total_q=1, n_blocks=128, mixed=False), + ), + Cell(why="a 4-row verify at 40 blocks", spec=dict(total_q=4, n_blocks=40, mixed=False)), + ) + + note = """ + Exact: ties between equal scores break toward the lower block id; `reference` sorts with the same rule. + """ + + def __call__( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + nvb = n_valid_blocks.to(device=max_score_kv.device, dtype=torch.int32).contiguous() + return torch.ops.trtllm.minimax_m3_select_blocks( + max_score_kv, nvb, topk, init_blocks, local_blocks, head_major_output + ) + + def reference( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + """Plain torch selection; ``head_major_output`` changes the backing only.""" + heads, n_blocks, total_q = max_score_kv.shape + out = torch.full((total_q, heads, topk), -1, dtype=torch.int32, device=max_score_kv.device) + blocks = torch.arange(n_blocks, device=max_score_kv.device) + for t in range(total_q): + nv = int(n_valid_blocks[t]) + for h in range(heads): + s = max_score_kv[h, :, t].clone().double() + s[blocks >= nv] = float("-inf") + forced = (blocks < min(init_blocks, nv)) | ( + (blocks >= nv - local_blocks) & (blocks < nv) + ) + s[forced] = float("inf") + order = sorted(range(nv), key=lambda b: (-float(s[b]), b))[: min(topk, nv)] + ids = torch.tensor(sorted(order), dtype=torch.int32, device=out.device) + out[t, h, : ids.numel()] = ids + return out + + +m3_select_blocks = M3SelectBlocks() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_select_long.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_select_long.py new file mode 100644 index 000000000000..91c1817f1610 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_select_long.py @@ -0,0 +1,172 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 MSA block selection for decode rows of up to 9216 blocks.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Blocks per decode row the op covers (9 keys per thread of one 1024-thread CTA). +MAX_BLOCKS = 9216 +#: Blocks the selection keeps per (query, KV head). +TOPK = 16 +#: Scores the forcing substitutes: initial blocks rank above local ones, and both above any finite score. +INIT_SCORE = 1.0e30 +LOCAL_SCORE = 1.0e29 + +# M3's checkpoint: no forced initial blocks, the one block holding the query forced in. +_M3 = dict(num_kv_heads=1, init_blocks=0, local_blocks=1) + + +class M3SelectLong(OpWrapper): + """Fill ``out`` [total_q, num_kv_heads, 16] with each query's top-16 blocks. + + ``scores`` is the per-KV-head block maxima [num_kv_heads, num_blocks, + total_q] the CuTe DSL index scorer wrote (amax over the KV head's index + heads). Per (query, KV head) row the op keeps the ``n_valid_blocks[query]`` + leading blocks (clamped to ``[0, num_blocks]``), forces the last + ``local_blocks`` of them (counted from the *unclamped* valid count) to + ``LOCAL_SCORE`` and the first ``init_blocks`` to ``INIT_SCORE`` -- local wins + where both apply -- and selects the 16 largest, ties to the smaller block. + A pick whose forced score is -inf is emitted as -1. The ids are written + ascending with a -1 tail. + + Same contract as ``torch.ops.trtllm.minimax_m3_select_blocks`` (the in-tree + ``select_blocks_from_maxscore``) and bit-identical to it, except on a row + where more than 1024 valid blocks tie at the 16th-largest score: this op + breaks that tie to the smallest blocks as the contract says, the in-tree op + does not. The test checks both against ``reference``. Returns ``out``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="32K-context engine (257 worst-case k-tiles), Eagle3 verify at one request: 4 query rows", + spec=dict(num_blocks=257, total_q=4, valid="near_full", scores="random", **_M3), + ), + Cell( + why="decode without speculation at batch 8 on a 2048-block engine: the 2-key-per-thread path at its edge", + spec=dict(num_blocks=2048, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="1M-context engine at batch 8 x qlen 4: rows on all three key widths, short rows and padding rows", + spec=dict(num_blocks=8193, total_q=32, valid="mixed", scores="random", **_M3), + ), + Cell( + why="duplicated scores: ties broken to the smaller block, and > 1024 keys at the bound (re-read path)", + spec=dict(num_blocks=4500, total_q=4, valid="near_full", scores="ties", **_M3), + ), + Cell( + why="non-finite scores: +inf, NaN ranked above +inf, and rows whose picks run into -inf (emitted -1)", + spec=dict(num_blocks=600, total_q=8, valid="mixed", scores="nonfinite", **_M3), + ), + ) + + note = """ + Certified at one KV head per rank (TP4) with M3's forcing (`init_blocks=0`, + `local_blocks=1`). The kernel takes any head count and forcing; neither is + driven here. + + `scores` may be strided (the op reads it by stride); the target hands over + the scorer's contiguous view, which is what the cells drive. Rows of at + most 128 blocks are within the kernel's range, but the target keeps those on + the in-tree op, so they are not certified here. + + Ordering is by the descending-score key `orderedKey`: a positive NaN ranks + above +inf, +0 above -0, and a negative NaN below -inf -- which, unlike + -inf, is emitted as a block id rather than -1. The cells drive positive + NaN only. + """ + + def __call__( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_select_long( + scores, n_valid_blocks, out, init_blocks, local_blocks, use_pdl, early_trigger + ) + return out + + def reference( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + ) -> torch.Tensor: + """The [total_q, num_kv_heads, 16] int32 table the op is supposed to write. + + Exact integer ranking, no floating point beyond the forcing: each + score's bit pattern maps to an unsigned key that is smaller for a larger + score (NaN above +inf), packed with the block id so one sort ranks by + (score descending, block ascending). ``out``, ``use_pdl`` and + ``early_trigger`` only say where and how the op writes; they do not enter the result. + """ + num_kv_heads, num_blocks, total_q = scores.shape + device = scores.device + blocks = torch.arange(num_blocks, device=device, dtype=torch.int64) + raw = n_valid_blocks[:total_q].to(torch.int64) + valid = raw.clamp(0, num_blocks) + local_start = (raw - local_blocks).clamp_min(0) + result = torch.full((total_q, num_kv_heads, TOPK), -1, dtype=torch.int32, device=device) + for head in range(num_kv_heads): + forced = scores[head].float().t().clone() # [total_q, num_blocks] + forced = torch.where(blocks[None, :] < init_blocks, INIT_SCORE, forced) + forced = torch.where(blocks[None, :] >= local_start[:, None], LOCAL_SCORE, forced) + bits = forced.view(torch.int32).to(torch.int64) & 0xFFFFFFFF + key = torch.where(bits >= 0x80000000, bits, (~bits) & 0x7FFFFFFF) + in_row = blocks[None, :] < valid[:, None] + packed = torch.where( + in_row, (key << 16) | blocks[None, :], torch.iinfo(torch.int64).max + ) + k = min(TOPK, num_blocks) + picked = torch.topk(packed, k, dim=1, largest=False).values + picked_key = picked >> 16 + keep = (picked != torch.iinfo(torch.int64).max) & (picked_key != 0xFF800000) + ids = torch.where(keep, picked & 0xFFFF, torch.iinfo(torch.int64).max) + ids = ids.sort(dim=1).values + ids = torch.where(ids == torch.iinfo(torch.int64).max, -1, ids) + result[:, head, :k] = ids.to(torch.int32) + return result + + def is_valid( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + ) -> None: + """The inputs this op takes and answers wrongly. + + Negative forcing counts are accepted and push the forced ranges past the + row instead of failing; the in-tree op this one stands in for refuses + them, so a caller relying on this one would silently diverge from it. + An ``out`` overlapping ``scores`` is overwritten while it is still read. + """ + assert init_blocks >= 0 and local_blocks >= 0, ( + f"init_blocks ({init_blocks}) and local_blocks ({local_blocks}) must be non-negative; the " + "op accepts negative counts and selects as if the forced ranges were elsewhere" + ) + storage = scores.untyped_storage() + assert not (storage.data_ptr() <= out.data_ptr() < storage.data_ptr() + storage.nbytes()), ( + "out must not alias scores: the op writes it while scores are still read" + ) + + +m3_select_long = M3SelectLong() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_sparse_decode.py new file mode 100644 index 000000000000..b82d733c263b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_sparse_decode.py @@ -0,0 +1,215 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""This target's block-sparse decode attention (Triton, with the one-launch long-context route).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..triton_sparse_decode import minimax_m3_sparse_attn_decode +from .m3_sparse_decode_lc import PAGE, sparse_attention_reference + + +class M3SparseDecode(OpWrapper): + """``minimax_m3_sparse_attn_decode`` as ``M3Attention._decode_sparse_core`` calls it. + + Attention of each query token over the blocks ``topk_idx`` selected for it + (``[num_kv_heads, T, topk]``, ascending ids with a -1 tail), each block + truncated at the token's causal extent, written into ``output`` in place. + The target passes FP8 q, the packed NVFP4 sparse-layer cache with its block + and per-tensor scales, and BF16 ``output`` [T, 16, 128]. Returns ``output``. + + Which kernel runs is decided inside, per step: a selected (non-identity) + table on the TP4 geometry goes to the one-launch ``m3_sparse_decode_lc`` + kernel; an identity table (every request within 16 blocks, staged by the + metadata) runs the Triton producer, or from 17 query rows the multi-query + producer, then the LSE merge. Every route also publishes the MXFP8 o_proj + hand-off to ``m3_prequant.SLOT``; this entry certifies the attention output. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="identity selection, decode without speculation at batch 8: the Triton producer + merge", + spec=dict( + seq_lens=(1, 128, 129, 700, 2048, 1500, 64, 0), decode_query_len=1, identity=True + ), + ), + Cell( + why="identity selection, Eagle3 verify at one request (T=4): one block per producer program", + spec=dict(seq_lens=(1900,), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 4 (T=16): the last row count below the MQ producer", + spec=dict(seq_lens=(300, 2048, 4, 1000), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 8 (T=32): the multi-query producer, padding rows", + spec=dict( + seq_lens=(2048, 130, 5, 0, 1777, 900, 2000, 0), decode_query_len=4, identity=True + ), + ), + Cell( + why="selected blocks, Eagle3 verify at batch 2: routed to the one-launch long-context kernel", + spec=dict(seq_lens=(9000, 40000), decode_query_len=4, identity=False), + ), + Cell( + why="selected blocks, decode without speculation at batch 8: the long-context route at T=8", + spec=dict( + seq_lens=(3000, 2100, 9999, 2048, 2049, 4500, 16000, 2200), + decode_query_len=1, + identity=False, + ), + ), + ) + + note = """ + Certified only as the target calls it: one KV head of 16 query heads, + head_dim 128, FP8 q, packed NVFP4 cache, BF16 output, `num_topk_chunks` + left to the dispatcher. The function also takes BF16 / FP8 caches, other + head geometries and an explicit split-K factor (the in-tree tests drive + those); none is certified here. + + The MXFP8 hand-off each route publishes is certified on the long-context + kernel's own entry (`m3_sparse_decode_lc`); the Triton merge writes it with + the same recipe (`_store_mxfp8_head`). + """ + + def __call__( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + minimax_m3_sparse_attn_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + kv_scale=kv_scale, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + num_topk_chunks=num_topk_chunks, + identity_blocks=identity_blocks, + ) + return output + + def reference( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + """``output`` as the call is supposed to write it: float64 block-sparse attention over the + dequantized NVFP4 blocks (``sparse_attention_reference``), rounded once to ``output``'s dtype. + + Only the packed NVFP4 cache with one KV head is mirrored -- the target's + only configuration. ``kv_scale``, ``num_topk_chunks`` and + ``identity_blocks`` do not change what is computed. + """ + assert k_block_scale is not None and k_paged.shape[1] == 1, ( + "mirrors the one-KV-head NVFP4 cache only" + ) + attn = sparse_attention_reference( + q, + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx[0], + block_table, + seq_lens, + decode_query_len, + sm_scale, + ) + return attn.to(output.dtype) + + def is_valid( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> None: + """The inputs this call takes and answers wrongly. + + Every producer walks a token's first ``min(topk, ceil(kv_len / 128))`` + table entries and indexes the block table with them unchecked, so they + must be real blocks of the request. ``identity_blocks`` is a promise the + multi-query producer relies on: every token of a request reads the same + blocks, all of them. Reads ``topk_idx`` and ``seq_lens`` back to the + host: a test-only cost. + """ + total_q = q.shape[0] + seq = seq_lens.tolist() + max_topk = topk_idx.shape[-1] + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk_idx[:, tok, : min(max_topk, nblocks)] + assert bool(((ids >= 0) & (ids < min(nblocks, block_table.shape[1]))).all()), ( + f"token {tok}: the first min(topk, block count) selected ids must be blocks of the request; " + "the block table is indexed with them unchecked" + ) + if identity_blocks: + assert nblocks <= max_topk and bool( + (ids == torch.arange(nblocks, device=ids.device, dtype=ids.dtype)).all() + ), ( + f"token {tok}: identity_blocks promises the table holds every block of the request in order" + ) + + +m3_sparse_decode = M3SparseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_sparse_decode_lc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_sparse_decode_lc.py new file mode 100644 index 000000000000..b5a1f9b750fd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/m3_sparse_decode_lc.py @@ -0,0 +1,487 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-launch long-context sparse decode over the selected NVFP4 blocks (16-CTA clusters). + +Also the plain-torch block-sparse attention and MXFP8 hand-off mirrors the +Triton sparse decode entry (``m3_sparse_decode``) reuses: both compute the same +thing over the same cache. +""" + +from __future__ import annotations + +import os +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) + +from ...m31_switches import m31_disabled + +# LC4 (LongAttn): this target's copy of the LC kernel (``minimax_m31_tp4_sparse_decode_lc``, +# kernels/minimaxM31/sparseDecodeLcTp4) with two exact options, rows and hand-off bit-identical to the M3 op. +# LC4_BTSMEM stages the request's block-table row in shared memory before the grid wait, on steps of at most +# ``LC4_BTSMEM_MAX_ROWS`` rows (TP4 K3 chain microbench: -0.16 us/layer at 4 rows and -0.39 at 8, but +3.3 at 16 and +# +7.9 at 32: the extra 32 KB of shared memory per CTA costs the larger grids their occupancy). LC4_SYNC3 reads the +# softmax row max / sum in every thread (two of five __syncthreads go; -0.10 us/layer at 4 rows, -0.05 at 32). +# A build without the op in its library loads it from ``tensorrt_llm/libs/libm31_lc4.so`` when that file exists (an +# overlay); otherwise, or with no option on for the step, the M3 op runs. Kill switches: +# TRTLLM_M31_DISABLE_LC4_BTSMEM=1, TRTLLM_M31_DISABLE_LC4_SYNC3=1. +LC4_BTSMEM = not m31_disabled("LC4_BTSMEM") +LC4_SYNC3 = not m31_disabled("LC4_SYNC3") +LC4_BTSMEM_MAX_ROWS = 8 # capacity: fixed: measured crossover (wins at 8 rows, loses at 16) +_LC4_STATE = {} + + +def _lc4_op(): + """``minimax_m31_tp4_sparse_decode_lc`` when an LC4 option is on and the op is built (or overlaid), else None + (the M3 op runs). Decided once per process.""" + if "op" not in _LC4_STATE: + op = None + if LC4_BTSMEM or LC4_SYNC3: + op = getattr(torch.ops.trtllm, "minimax_m31_tp4_sparse_decode_lc", None) + if op is None: + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_lc4.so") + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_tp4_sparse_decode_lc", None) + from tensorrt_llm.logger import logger + + if op is not None: + reason = f"engaged (btsmem={LC4_BTSMEM}, sync3={LC4_SYNC3})" + else: + reason = ( + "switched off" + if not (LC4_BTSMEM or LC4_SYNC3) + else "not taken: the op is not built" + ) + logger.info(f"[M3 dispatch] LC4 sparse decode options: {reason}") + _LC4_STATE["op"] = op + return _LC4_STATE["op"] + + +def _lc4_btsmem(rows: int) -> bool: + """Whether a step of ``rows`` decode rows stages its block-table rows (LC4_BTSMEM, at most LC4_BTSMEM_MAX_ROWS).""" + return LC4_BTSMEM and rows <= LC4_BTSMEM_MAX_ROWS + + +#: The TP4 geometry the kernel is compiled for. +NUM_HEADS = 16 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 +#: NVFP4 elements per E4M3 block scale. +SF_VEC = 16 +#: MXFP8 elements per UE8M0 scale, and the o_proj width the hand-off is laid out for. +MX_BLOCK = 32 +WIDTH = NUM_HEADS * HEAD_DIM + +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) + + +def dequant_nvfp4_pages( + packed: torch.Tensor, block_scale: torch.Tensor, pages: torch.Tensor, quad_scales: bool +) -> torch.Tensor: + """Dequantize KV head 0 of the given pages of a packed NVFP4 cache to float64 [n, PAGE, HEAD_DIM]. + + ``packed`` is the [pages, kv_heads, PAGE, HEAD_DIM // 2] byte view (low + nibble = even element), ``block_scale`` the [pages, kv_heads, PAGE, + HEAD_DIM // 16] E4M3 bytes ``fused_write_layer_caches_nvfp4`` writes: token + ``t``'s column ``c`` at byte ``t * 8 + c`` of the page's region for K, and at + ``(t // 4) * 32 + c * 4 + t % 4`` for V (``quad_scales``). The per-tensor + global scale is not applied. Every product is exact in float64. + """ + table = torch.tensor(_E2M1, dtype=torch.float64, device=packed.device) + rows = packed[pages, 0].to(torch.int64) # [n, PAGE, HEAD_DIM // 2] + nibbles = torch.stack([rows & 0x0F, rows >> 4], dim=-1).reshape(rows.shape[0], PAGE, HEAD_DIM) + cols = HEAD_DIM // SF_VEC + region = block_scale[pages, 0].reshape(rows.shape[0], PAGE * cols) + t = torch.arange(PAGE, device=packed.device)[:, None] + c = torch.arange(cols, device=packed.device)[None, :] + index = (t // 4) * 32 + c * 4 + t % 4 if quad_scales else t * cols + c + scales = region[:, index.reshape(-1)].reshape(rows.shape[0], PAGE, cols) + scales = scales.view(torch.float8_e4m3fn).to(torch.float64) + return table[nibbles] * scales.repeat_interleave(SF_VEC, dim=-1) + + +def sparse_attention_reference( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, +) -> torch.Tensor: + """Block-sparse decode attention over one KV head of a packed NVFP4 cache, float64 [T, heads, HEAD_DIM]. + + Query token ``tok`` belongs to request ``tok // decode_query_len`` at offset + ``tok % decode_query_len`` and attends the first ``min(16, ceil(kv_len / + PAGE))`` entries of ``topk[tok]`` that are not -1, each block truncated at + ``kv_len = max(seq_len - decode_query_len + offset + 1, 0)``. A token that + attends nothing (a CUDA-graph padding row) is all zeros. ``q`` is E4M3 (or + its byte view); softmax and both products are float64. + """ + q64 = ( + q.view(torch.float8_e4m3fn).to(torch.float64) + if q.element_size() == 1 + else q.to(torch.float64) + ) + total_q, num_heads, _ = q64.shape + kgs = float(k_global_scale.reshape(-1)[0]) + vgs = float(v_global_scale.reshape(-1)[0]) + out = torch.zeros(total_q, num_heads, HEAD_DIM, dtype=torch.float64, device=q.device) + seq = seq_lens.tolist() + positions = torch.arange(PAGE, device=q.device) + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + real_topk = min(TOPK, (kv_len + PAGE - 1) // PAGE) + blocks = topk[tok, :real_topk] + blocks = blocks[blocks >= 0].to(torch.int64) + if blocks.numel() == 0: + continue + pages = block_table[req, blocks].to(torch.int64) + keys = ( + dequant_nvfp4_pages(k, k_block_scale, pages, quad_scales=False).reshape(-1, HEAD_DIM) + * kgs + ) + values = ( + dequant_nvfp4_pages(v, v_block_scale, pages, quad_scales=True).reshape(-1, HEAD_DIM) + * vgs + ) + valid = ((blocks[:, None] * PAGE + positions[None, :]) < kv_len).reshape(-1) + logits = (q64[tok] @ keys.T) * sm_scale + logits = logits.masked_fill(~valid[None, :], float("-inf")) + out[tok] = torch.softmax(logits, dim=-1) @ values + return out + + +def mxfp8_handoff_reference( + out: torch.Tensor, num_rows_alloc: int +) -> Tuple[torch.Tensor, torch.Tensor]: + """The MXFP8 o_proj hand-off the merge writes for BF16 rows ``out`` [T, WIDTH]: ``(q8 [T, WIDTH] uint8, + sf [num_rows_alloc * WIDTH // 32] uint8)``. + + Per 32-element block: ``e`` is the biased exponent of ``amax * rcp(448)`` + rounded up when its mantissa is non-zero (capped at 254), the elements are + scaled by ``2^(127 - e)`` and rounded to E4M3 (saturating), and ``e`` lands + in the 128x4-swizzled scale layout the MXFP8 GEMM reads. Entries of ``sf`` + for rows past ``T`` are zero. Computed from the op's own BF16 rows, so a + comparison against it is exact. + """ + total_q = out.shape[0] + x = out.float().reshape(total_q, WIDTH // MX_BLOCK, MX_BLOCK) + amax = x.abs().amax(dim=-1) + sfv = amax * torch.tensor(1.0 / 448.0, dtype=torch.float32, device=out.device) + bits = sfv.view(torch.int32) + e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + e = e.clamp(max=254) + qs = torch.where(amax != 0, ((254 - e) << 23).view(torch.float32), torch.zeros_like(amax)) + xq = (x * qs[..., None]).clamp(-448.0, 448.0) + q8 = xq.to(torch.float8_e4m3fn).view(torch.uint8).reshape(total_q, WIDTH) + num_sf_k_tiles = WIDTH // 128 + sf = torch.zeros(num_rows_alloc * (WIDTH // MX_BLOCK), dtype=torch.uint8, device=out.device) + tok = torch.arange(total_q, device=out.device)[:, None] + k_sf = torch.arange(WIDTH // MX_BLOCK, device=out.device)[None, :] + offset = ( + (tok // 128) * num_sf_k_tiles * 512 + + (k_sf // 4) * 512 + + (tok % 32) * 16 + + ((tok % 128) // 32) * 4 + + k_sf % 4 + ) + sf[offset.reshape(-1)] = e.to(torch.uint8).reshape(-1) + return q8, sf + + +class M3SparseDecodeLc(OpWrapper): + """Sparse decode attention, split-K merge and MXFP8 hand-off of one TP4 decode step in one launch. + + One cluster of 16 CTAs per query token: CTA ``r`` attends the token's + ``r``-th selected block (``topk[tok, r]``), and the cluster merges the 16 + partials over distributed shared memory into the BF16 row ``out[tok]``. With + ``q8`` / ``sf`` it also writes the row's MXFP8 copy and swizzled UE8M0 + scales for the o_proj GEMM. All operands are the byte views + ``m3_attn_lc.m3_lc_attn`` hands over (E4M3 ``q``, packed E2M1 pages, E4M3 + block scales). Returns ``out``. + + Per block: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16 + (exact), QK and PV on FP16 tensor cores with FP32 accumulation, the softmax + probabilities rounded to FP16 for PV, log2-domain softmax with ``ex2.approx``. + ``linear_softmax`` picks the denominator recurrence (the two agree to + rounding); the target picks it per batch shape. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode without speculation, one request: T=1, log-domain recurrence, MXFP8 hand-off", + spec=dict(seq_lens=(9000,), decode_query_len=1, linear_softmax=False, mxfp8=True), + ), + Cell( + why="Eagle3 verify at one request: T=4, the log-domain recurrence the target keeps for local batch 1", + spec=dict(seq_lens=(20000,), decode_query_len=4, linear_softmax=False, mxfp8=True), + ), + Cell( + why="Eagle3 verify at two requests: T=8 on the linear denominator", + spec=dict(seq_lens=(5000, 12345), decode_query_len=4, linear_softmax=True, mxfp8=True), + ), + Cell( + why="7-draft verify at one request: T=8, the log-domain recurrence the target keeps for local batch 1", + spec=dict(seq_lens=(20000,), decode_query_len=8, linear_softmax=False, mxfp8=True), + ), + Cell( + why="7-draft verify at two requests: T=16 on the linear denominator", + spec=dict(seq_lens=(5000, 12345), decode_query_len=8, linear_softmax=True, mxfp8=True), + ), + Cell( + why="Eagle3 verify at batch 8 (T=32): short requests with -1 tails and zero-length padding rows", + spec=dict( + seq_lens=(16384, 2049, 130, 0, 7777, 4, 30000, 0), + decode_query_len=4, + linear_softmax=True, + mxfp8=True, + ), + ), + Cell( + why="decode without speculation at batch 8: T=8 rows of distinct requests", + spec=dict( + seq_lens=(3000, 2100, 9999, 2048, 2049, 4500, 16000, 2200), + decode_query_len=1, + linear_softmax=False, + mxfp8=True, + ), + ), + Cell( + why="no hand-off buffers (the variant the call site takes past 128 rows), driven at T=8", + spec=dict(seq_lens=(5000, 12345), decode_query_len=4, linear_softmax=True, mxfp8=False), + ), + ) + + note = """ + Certified: the four instantiated variants (`linear_softmax` x hand-off on / + off), all with `early_trigger=True` -- the op refuses `False`. The + hand-off cells are certified bit-exactly against `mxfp8_handoff_reference` + applied to the op's own BF16 rows; the BF16 rows themselves within the band + the test derives. + + `topk` is read for the first `min(16, ceil(kv_len / 128))` entries of each + token; a -1 among them skips that CTA's block, entries past them are never + read. `use_pdl` only changes the launch attribute. The hand-off assumes + `num_sf_k_tiles = 16` (a 2048-wide o_proj input) and buffers with at least + as many rows as the step; `MERGE_MXFP8` sizes them at 128 rows. + """ + + def __call__( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + early_trigger: bool, + ) -> torch.Tensor: + lc4 = _lc4_op() + btsmem = _lc4_btsmem(int(q.shape[0])) + if lc4 is not None and (btsmem or LC4_SYNC3): + lc4( + q, + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + out, + q8, + sf, + num_sf_k_tiles, + decode_query_len, + sm_scale, + linear_softmax, + use_pdl, + early_trigger, + btsmem, + LC4_SYNC3, + False, # Preserve the measured normal LSE merge; no opt-in one-division path. + ) + return out + torch.ops.trtllm.minimax_m3_sparse_decode_lc( + q, + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + out, + q8, + sf, + num_sf_k_tiles, + decode_query_len, + sm_scale, + linear_softmax, + use_pdl, + early_trigger, + ) + return out + + def reference( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + early_trigger: bool, + ) -> torch.Tensor: + """The BF16 ``out`` [T, 2048] the op is supposed to write: ``sparse_attention_reference`` in float64, + rounded once. ``linear_softmax``, ``use_pdl`` and ``early_trigger`` do not change what is computed; the + hand-off buffers are checked separately with ``mxfp8_handoff_reference``. + """ + attn = sparse_attention_reference( + q.view(-1, NUM_HEADS, HEAD_DIM), + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + decode_query_len, + sm_scale, + ) + return attn.reshape(out.shape).to(out.dtype) + + def is_valid( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + early_trigger: bool, + ) -> None: + """The inputs this op takes and answers wrongly (``m3_attn_lc.lc_applies`` is the target's form of the + layout half). + + The kernel addresses a page's token rows at a fixed 64-byte (data) and + 8-byte (scale) pitch and loads them 16 / 8 bytes at a time, so only the + page stride is read from the views. It reads the selected block ids, + the block table and the hand-off buffers unchecked. Reads ``topk`` and + ``seq_lens`` back to the host: a test-only cost. + """ + for name, t in (("k", k), ("v", v)): + assert t.dim() == 4 and t.shape[1] == 1 and t.shape[2:] == (PAGE, HEAD_DIM // 2), ( + f"{name} must be the [pages, 1, 128, 64] byte view of one KV head" + ) + assert t.stride(3) == 1 and t.stride(2) == HEAD_DIM // 2, ( + f"{name} rows must be dense 64-byte rows: the kernel ignores the row stride" + ) + assert t.data_ptr() % 16 == 0 and t.stride(0) % 16 == 0, ( + f"{name} pages must be 16-byte aligned" + ) + for name, t in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + assert t.dim() == 4 and t.stride(3) == 1 and t.stride(2) == HEAD_DIM // SF_VEC, ( + f"{name} rows must be dense 8-byte rows: the kernel ignores the row stride" + ) + assert t.data_ptr() % 8 == 0 and t.stride(0) % 8 == 0, ( + f"{name} pages must be 8-byte aligned" + ) + assert q.data_ptr() % 16 == 0 and q.stride(0) % 16 == 0 and q.stride(1) % 16 == 0, ( + "q rows are loaded 16 bytes at a time and must be 16-byte aligned" + ) + assert k_global_scale.numel() == 1 and v_global_scale.numel() == 1, ( + "global scales are single values" + ) + total_q = q.shape[0] + assert ( + total_q % decode_query_len == 0 and seq_lens.numel() >= total_q // decode_query_len + ), ( + "every query token must belong to a request: seq_lens is read at tok // decode_query_len unchecked" + ) + seq = seq_lens.tolist() + num_cols = block_table.shape[1] + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk[tok, : min(TOPK, nblocks)] + assert bool(((ids >= -1) & (ids < min(nblocks, num_cols))).all()), ( + f"token {tok}: selected block ids must be -1 or below the request's block count; the block " + "table is indexed with them unchecked" + ) + if q8 is not None: + assert num_sf_k_tiles == WIDTH // 128, ( + f"num_sf_k_tiles must be {WIDTH // 128}: the hand-off rows are {WIDTH} wide and a smaller " + "value overlaps them silently" + ) + assert ( + q8.numel() >= total_q * WIDTH + and sf is not None + and sf.numel() >= ((total_q + 127) // 128 * 128 * (WIDTH // MX_BLOCK)) + ), "the hand-off buffers must hold every row of the step: they are written unchecked" + + +m3_sparse_decode_lc = M3SparseDecodeLc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/msa_kv_update.py new file mode 100644 index 000000000000..ad05b934d4a9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/attention/msa_kv_update.py @@ -0,0 +1,200 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MSA KV-length correction in one launch (this target's ``MsaKvUpdate`` hook).""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The MSA pool's page: one sparse block is one 128-token KV page. +TOKENS_PER_BLOCK = 128 + + +class MsaKvUpdate(OpWrapper): + """Re-derive the MSA decode state after the overlap scheduler moved ``kv_lens``. + + ``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` calls this with its + own buffers at the head of every captured speculative decode step. One + single-CTA launch writes, for ``batch`` requests and ``total_q`` query + tokens (all int32): + + * ``msa_seq_lens[b] = min(kv_lens[b], kv_lens_staged[b])`` and, when given, + ``cu_kv_lens`` as its exclusive prefix sum (``batch + 1`` entries); + * ``out_cache_loc[t]``, the KV write slot of the position token ``t`` + attends up to, through ``block_table``; + * ``n_valid_blocks[t]``, when given, the block count top-k selection is + bounded by. + + It returns None; the cells read the four buffers. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode without speculation at one request: qo = 1, every buffer written", + spec=dict(batch=1, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="decode without speculation at the captured batch 8", + spec=dict(batch=8, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="Eagle3 verify (draft 3): qo = 4 per request, partial acceptance shortens kv_lens below staged", + spec=dict(batch=8, query_len=4, accept="partial", cu_kv=True, n_valid=True), + ), + Cell( + why="the metadata without cu_kv_lens (no extend_ctx promotion): the optional prefix sum is skipped", + spec=dict(batch=4, query_len=4, accept="partial", cu_kv=False, n_valid=True), + ), + Cell( + why="CUDA-graph padding rows (kv_lens 0, pos -1) and no n_valid_blocks buffer", + spec=dict(batch=8, query_len=4, accept="padded", cu_kv=True, n_valid=False), + ), + Cell( + why="positions crossing a page boundary: the slot moves to the previous page of the request", + spec=dict(batch=3, query_len=4, accept="page_edge", cu_kv=True, n_valid=True), + ), + ) + + note = """ + The kernel covers any batch up to the block size's loop (it strides over + 256 threads), and any page size; only `tokens_per_block = 128`, the MSA + pool's page, is certified. + + The op does not bounds-check the request index a token reads + (`q_batch_row`) or the block-table column it derives: both come from the + metadata's own staging, which keeps them in range by construction, and an + out-of-range value reads whatever memory sits there. `is_valid` states both. + """ + + def __call__( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + torch.ops.trtllm.minimax_m3_msa_kv_update( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + + def reference( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> Tuple[torch.Tensor, Optional[torch.Tensor], torch.Tensor, Optional[torch.Tensor]]: + """``(msa_seq_lens[:batch], cu_kv_lens[:batch + 1], out_cache_loc[:total_q], + n_valid_blocks[:total_q])`` as the op is supposed to write them. + + The in-tree metadata's own torch re-derivation + (``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` past its hook + call), statement for statement, returning new tensors rather than + writing the buffers. The buffer arguments only say which outputs exist: + a ``None`` buffer gives a ``None`` result. + """ + page = int(tokens_per_block) + kv_true = torch.minimum(kv_lens[:batch], kv_lens_staged[:batch]) + cu_kv = None + if cu_kv_lens is not None: + cu_kv = torch.zeros(batch + 1, dtype=torch.int32, device=kv_lens.device) + cu_kv[1:] = torch.cumsum(kv_true, 0) + qbr = q_batch_row[:total_q].to(torch.long) + qo_dev = seq_lens[:batch] + pos = kv_true[qbr] - qo_dev[qbr] + q_intra[:total_q] + pos_slot = pos.clamp_min(0) + block_col = torch.div(pos_slot, page, rounding_mode="floor").clamp( + max=int(block_table.shape[1]) - 1 + ) + slots = block_table[qbr, block_col.to(torch.long)] * page + torch.remainder(pos_slot, page) + n_valid = None + if n_valid_blocks is not None: + n_valid = torch.div( + (pos + 1).clamp_min(0) + (page - 1), page, rounding_mode="floor" + ).to(torch.int32) + return kv_true.to(torch.int32), cu_kv, slots.to(torch.int32), n_valid + + def is_valid( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + """The inputs this op takes and answers wrongly. + + Every operand but the block table is read as a dense run from its first + element, so a strided view reads the wrong elements. The request index of + each token and the table row it selects are not bounds-checked. Reads + ``q_batch_row`` back to the host: a test-only cost. + """ + for name, t in ( + ("kv_lens", kv_lens), + ("kv_lens_staged", kv_lens_staged), + ("seq_lens", seq_lens), + ("q_batch_row", q_batch_row), + ("q_intra", q_intra), + ("msa_seq_lens", msa_seq_lens), + ("out_cache_loc", out_cache_loc), + ("cu_kv_lens", cu_kv_lens), + ("n_valid_blocks", n_valid_blocks), + ): + assert t is None or t.is_contiguous(), ( + f"{name} must be contiguous: the kernel indexes it densely from data_ptr and ignores strides" + ) + rows = q_batch_row[:total_q] + assert total_q == 0 or ( + int(rows.min()) >= 0 and int(rows.max()) < min(batch, block_table.shape[0]) + ), ( + "q_batch_row must index a request below batch (and a block-table row): the op reads the " + "request's lengths and table row unchecked" + ) + + +msa_kv_update = MsaKvUpdate() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/__init__.py new file mode 100644 index 000000000000..5da86fe24bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collective-communication entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_allreduce.py new file mode 100644 index 000000000000..1a275c4ad863 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_allreduce.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The built-in TP all-reduce, fused with the residual add and the next RMSNorm.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.distributed import AllReduce, AllReduceParams + + +class M3AllReduce(OpWrapper): + """``comm(x, all_reduce_params=params)``: the runtime's ``AllReduce`` over the TP group. ``params`` None: the + plain sum over the ranks (the drafter's o_proj and down projection). With ``RESIDUAL_RMS_NORM`` params: + ``updated = residual + sum_ranks(x)``, ``normed = rmsnorm(updated) * norm_weight``; + returns (normed, updated). The strategy (``AllReduceStrategy.AUTO``: one-shot / two-shot Lamport, NCCL) is the + communicator's per call. + + The layer boundaries the multicast all-reduce does not take run it: prefill and mixed steps, and generation steps + beyond the multicast kernel's rows. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="tp4, a prefill chunk", spec=dict(world=4, m=2048, hidden=6144)), + Cell(why="tp4, a 64-row generation step", spec=dict(world=4, m=64, hidden=6144)), + Cell( + why="tp4, plain sum of eight drafter rows", + spec=dict(world=4, m=8, hidden=6144, plain=True), + ), + ) + + note = """ + The reduction order across ranks is the strategy's; `reference` sums the ranks' rows in fp32 in rank order, then + adds the residual and normalizes (one bf16 rounding of the sum and one of the norm). A strategy may round the sum + to bf16 before the residual add, so the residual sum agrees within one bf16 ulp of its operands' magnitude. + """ + + def __call__(self, x: torch.Tensor, comm: AllReduce, params: Optional[AllReduceParams]): + return comm(x, all_reduce_params=params) + + def reference(self, xs, comm: AllReduce, params: Optional[AllReduceParams]): + """``xs``: every rank's ``x`` in rank order (a collective's reference takes the group's inputs).""" + total = torch.stack([x.float() for x in xs]).sum(0) + if params is None: + return total.to(xs[0].dtype) + updated = (params.residual.float() + total).to(xs[0].dtype) + h = updated.float() + normed = ( + h + * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + params.eps) + * params.norm_weight.float() + ) + return normed.to(xs[0].dtype), updated + + +m3_allreduce = M3AllReduce() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_mc_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_mc_allreduce_norm.py new file mode 100644 index 000000000000..0da06e59b14e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_mc_allreduce_norm.py @@ -0,0 +1,277 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode layer boundary, 4 ranks.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +_E4M3_MAX = 448.0 + + +def mxfp8_rows(normed: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """MXFP8 of BF16 rows as ``trtllm::mxfp8_quantize(normed, True)``: E4M3 [M, H] and 128x4-swizzled UE8M0 bytes. + + One scale per 32 columns, the smallest power of two at or above ``amax / 448`` (biased exponent, round up), + data = row / scale rounded to nearest E4M3. The scale buffer is padded to 128 rows, padding bytes left as the + op leaves them (the comparison masks them out). + """ + m, h = normed.shape + blocks = normed.float().view(m, h // 32, 32) + amax = blocks.abs().amax(-1) + bits = (amax / _E4M3_MAX).view(torch.int32) + byte = (((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) > 0).to(torch.int32)).clamp(0, 254) + scale = torch.exp2(byte.float() - 127.0).unsqueeze(-1) + data = torch.where(amax.unsqueeze(-1) != 0, blocks / scale, torch.zeros_like(blocks)) + data = data.view(m, h).to(torch.float8_e4m3fn) + return data, swizzle_sf(byte.to(torch.uint8)) + + +def swizzle_sf(byte: torch.Tensor) -> torch.Tensor: + """Scatter [rows, H / 32] scale bytes into the 128x4-swizzled buffer the op writes (zero padding).""" + rows, cols = byte.shape + padded_rows = (rows + 127) // 128 * 128 + r = torch.arange(rows, device=byte.device).view(-1, 1) + c = torch.arange(cols, device=byte.device).view(1, -1) + index = ( + (c % 4) + (c // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + (r // 128) * (128 * cols) + ) + out = torch.zeros(padded_rows * cols, dtype=torch.uint8, device=byte.device) + out[index.reshape(-1)] = byte.reshape(-1) + return out + + +def sf_written(rows: int, cols: int, device) -> torch.Tensor: + """Mask of the swizzled scale bytes the op writes for ``rows`` rows (the padding rows are left unwritten).""" + return swizzle_sf(torch.ones(rows, cols, dtype=torch.uint8, device=device)).bool() + + +def boundary( + partials: Sequence[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, +) -> tuple[torch.Tensor, ...]: + """The boundary arithmetic once each rank's BF16 partial is known (shared with the finalize entry). + + Ranks are summed in FP32 in rank order, the residual added and rounded to BF16 once; the RMS statistic and the + normalization are done in float64 and rounded to BF16 once. + """ + total = torch.zeros(partials[0].shape, dtype=torch.float32, device=residual.device) + for p in partials: + total = total + p.float() + updated = (total + residual.float()).to(torch.bfloat16) + u = updated.double() + inv = torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + eps) + normed = (u * inv * norm_w.double()).to(torch.bfloat16) + if not quant: + return normed, updated + return (normed, updated, *mxfp8_rows(normed)) + + +class M3McAllreduceNorm(OpWrapper): + """One TP group's boundary: ``updated = bf16(residual + sum_r bf16(x_r + x2_r))``, ``normed = RMSNorm(updated)``. + + Every rank pushes its partial through the multicast mapping ``ws_mc_ptr`` into every rank's exchange, then polls + its own view ``ws_uc`` (re-arming the words it read) and rotates the per-CTA call parities in ``flags``; the + workspace is the ``McAllReduceWorkspace`` bucket of ``maxt`` tokens. All ranks must make the same calls in the + same stream order. ``prepushed`` 2 means a producer already pushed ``x`` (and launched this grid after its + pushes); 3 that two producers pushed ``x`` into exchange set A and ``x2`` into set B. Returns + ``[normed, updated]`` or, with ``quant``, also the MXFP8 copy of ``normed`` (E4M3 rows, swizzled UE8M0). + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The tp4 decode boundaries (hidden 6144, 4 ranks, eps 1e-6). Buckets 8 / 16 / 48 are separate workspaces; the + # 8-token bucket runs epilogue option 7 at 1, 2, 7, 8 tokens and 15 at 3..6, the others 15. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one decode token at the attention boundary, x only, bucket 8 option 7", + spec=dict(m=1, maxt=8, opts=7, x2=False, quant=False, prepushed=0), + ), + Cell( + why="MoE boundary at 4 tokens: shared + routed summed in-kernel, MXFP8 copy for the next QKV, option 15", + spec=dict(m=4, maxt=8, opts=15, x2=True, quant=True, prepushed=0), + ), + Cell( + why="a full 8-token bucket with the second addend", + spec=dict(m=8, maxt=8, opts=7, x2=True, quant=False, prepushed=0), + ), + Cell( + why="attention boundary whose rows the o_proj GEMV already pushed (prepushed 2)", + spec=dict(m=8, maxt=8, opts=7, x2=False, quant=False, prepushed=2), + ), + Cell( + why="routed engine pushed set A, shared down projection set B (prepushed 3), with the MXFP8 copy", + spec=dict(m=6, maxt=8, opts=15, x2=True, quant=True, prepushed=3), + ), + Cell( + why="12 tokens pushed by the o_proj GEMV into the 16-token bucket", + spec=dict(m=12, maxt=16, opts=15, x2=False, quant=False, prepushed=2), + ), + Cell( + why="the 9..13-token MoE kernel's sum into the 16-token bucket, MXFP8 copy", + spec=dict(m=16, maxt=16, opts=15, x2=False, quant=True, prepushed=0), + ), + Cell( + why="the 48-token bucket at a partial fill", + spec=dict(m=30, maxt=48, opts=15, x2=False, quant=False, prepushed=0), + ), + Cell( + why="the 48-token bucket full, second addend and MXFP8 copy", + spec=dict(m=48, maxt=48, opts=15, x2=True, quant=True, prepushed=0), + ), + # Every other row count of the decode boundaries (dispatch.MC_ALLREDUCE): the bucket and epilogue options the + # target picks for it (ops/m3_mc_allreduce), the second addend and the MXFP8 copy alternating. + *( + Cell( + why=f"{m} tokens: a boundary row count between the ones above", + spec=dict( + m=m, + maxt=8 if m <= 8 else 16 if m <= 16 else 48, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1] if m <= 8 else 15, + x2=m % 2 == 0, + quant=m % 3 == 0, + prepushed=0, + ), + ) + for m in range(1, 49) + if m not in (1, 4, 8, 16, 30, 48) + ), + # Every other row count pushed by the o_proj (1..16) or the 1..8-token MoE engine (with the MXFP8 copy): + # dispatch.ATTN_PUSH / dispatch.MOE_PUSH. + *( + Cell( + why=f"{m} tokens prepushed: a pushed row count between the ones above", + spec=dict( + m=m, + maxt=8 if m <= 8 else 16, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1] if m <= 8 else 15, + x2=False, + quant=m <= 8, + prepushed=2, + ), + ) + for m in range(1, 17) + if m not in (8, 12) + ), + ) + + note = """ + The epilogue option bits (1 early residual / weight load, 2 st.async cluster exchange, 4 early parity rotation, + 8 max carveout) are bit-identical by construction; only the four (bucket, option) pairs the target selects are + built. `prepushed` 1 (pushed, grid wait kept) is accepted by the op but no caller passes it. `pdl` changes only + launch overlap and is driven off in the cells. + + `normed` is gated to one BF16 rounding step: the kernel's RMS statistic is a warp / cluster tree of FP32 sums + and its `rsqrtf` is approximate, so a row whose scaled value lands next to a BF16 midpoint can round either + way. `updated` and the MXFP8 copy of the op's own `normed` are bit-exact. + """ + + def __call__( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_mc_allreduce_norm( + x, + x2, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + None, + None, + maxt, + quant, + prepushed, + opts, + ) + + def reference( + self, + per_rank_x: Sequence[torch.Tensor], + per_rank_x2: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute. + + The documented collective exception: it takes every rank's ``x`` (and ``x2``) in rank order, not the calling + rank's. Each rank's partial is ``bf16(x + x2)`` (``x`` alone without ``x2``), whatever pushed it. + """ + partials = [] + for r, x in enumerate(per_rank_x): + if per_rank_x2 is None: + partials.append(x) + else: + partials.append((x.float() + per_rank_x2[r].float()).to(torch.bfloat16)) + return boundary(partials, residual, norm_w, eps, quant) + + def is_valid( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + ) -> None: + h = residual.shape[-1] + # The op checks norm_w's element count, not its layout: a strided view is read as packed memory. + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + # The workspace is addressed as [set][parity][maxt][world][H / 2] words and never bounds-checked: a smaller + # buffer (another bucket's, or one built for fewer ranks) is polled and re-armed out of range. + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + # Flags are per (token, cluster CTA) of the bucket; the op checks the count, not that they belong to the + # workspace. Flags of another bucket hold another call history, and the parity they select is not the one the + # peers push into: the poll then waits on the other half forever or reads last call's rows. + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + + +m3_mc_allreduce_norm = M3McAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_mc_finalize_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_mc_finalize_allreduce_norm.py new file mode 100644 index 000000000000..7e6ebeef7d31 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/comm/m3_mc_finalize_allreduce_norm.py @@ -0,0 +1,197 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MoE finalize + multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode MoE boundary, 4 ranks.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_mc_allreduce_norm import boundary + + +def finalize_partial( + fc2: torch.Tensor, perm: torch.Tensor, scales: torch.Tensor, shared: Optional[torch.Tensor] +) -> torch.Tensor: + """One rank's BF16 partial: ``bf16(bf16(sum_k scale_k * fc2[perm_k]) + shared)``, the routed sum in FP32 in k + order and rounded to BF16, then the shared expert added as stock adds it (one more BF16 rounding). + + The kernel's per-k step is a fused multiply-add; it is reproduced exactly by the float64 product (exact for two + FP32 operands) plus the FP32 accumulator, rounded back to FP32. Dropped entries (perm -1) are skipped. + """ + m, topk = perm.shape + acc = torch.zeros(m, fc2.shape[1], dtype=torch.float32, device=fc2.device) + sc = scales.reshape(m, topk).float() + for k in range(topk): + rows = perm[:, k].long() + keep = (rows >= 0).unsqueeze(-1) + gathered = fc2[rows.clamp_min(0)].double() + step = (gathered * sc[:, k : k + 1].double() + acc.double()).float() + acc = torch.where(keep, step, acc) + if shared is not None: + acc = acc.to(torch.bfloat16).float() + shared.float() + return acc.to(torch.bfloat16) + + +class M3McFinalizeAllreduceNorm(OpWrapper): + """The MoE boundary with the routed experts' finalize folded in. + + Per rank the partial is ``bf16(bf16(sum_k scales[t, k] * fc2[perm[t, k]]) + shared[t])`` (``perm`` -1 = dropped, + ``scales`` fp32 or bf16), then the group's all-reduce, residual add and RMSNorm exactly as + ``m3_mc_allreduce_norm``: same workspace, flags and call-order contract. Returns ``[normed, updated]`` or, with + ``quant``, also the MXFP8 copy of ``normed``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The tp4 target takes this path from 9 tokens (top-4 of 128 experts, fp32 routing weights, the shared expert + # always present): the 16- and 48-token buckets, epilogue option 15. + CELLS: tuple[Cell, ...] = ( + Cell( + why="the smallest finalize call (9 tokens, 16-token bucket) with the next QKV's MXFP8 copy", + spec=dict(m=9, maxt=16, opts=15, topk=4, quant=True, dropped=0), + ), + Cell( + why="a full 16-token bucket with dropped expert slots", + spec=dict(m=16, maxt=16, opts=15, topk=4, quant=False, dropped=5), + ), + Cell( + why="the 48-token bucket at a partial fill", + spec=dict(m=24, maxt=48, opts=15, topk=4, quant=False, dropped=0), + ), + Cell( + why="the 48-token bucket full, MXFP8 copy", + spec=dict(m=48, maxt=48, opts=15, topk=4, quant=True, dropped=3), + ), + # Every other row count of the unfinalized MoE (dispatch.MC_FINALIZE): the bucket and options the target picks. + *( + Cell( + why=f"{m} tokens: a finalize row count between the ones above", + spec=dict( + m=m, + maxt=16 if m <= 16 else 48, + opts=15, + topk=4, + quant=m % 2 == 0, + dropped=m % 4, + ), + ) + for m in range(9, 49) + if m not in (9, 16, 24, 48) + ), + ) + + note = """ + BF16 routing weights and the 8-token bucket are accepted by the op (option 7 there) but the target only enters + this path at 9 tokens and up, with the built-in MoE's fp32 weights; neither is a cell. The rest is as in + `m3_mc_allreduce_norm`'s note. + """ + + def __call__( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_mc_allreduce_norm( + fc2, + shared, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + perm, + scales, + maxt, + quant, + 0, + opts, + ) + + def reference( + self, + per_rank_fc2: Sequence[torch.Tensor], + per_rank_perm: Sequence[torch.Tensor], + per_rank_scales: Sequence[torch.Tensor], + per_rank_shared: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute, from every rank's MoE outputs in rank order (the + collective exception, as in ``m3_mc_allreduce_norm.reference``).""" + partials = [ + finalize_partial( + fc2, + per_rank_perm[r], + per_rank_scales[r], + None if per_rank_shared is None else per_rank_shared[r], + ) + for r, fc2 in enumerate(per_rank_fc2) + ] + return boundary(partials, residual, norm_w, eps, quant) + + def is_valid( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + ) -> None: + h = residual.shape[-1] + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + # The kernel gathers fc2[perm] without a bound: an index past the FC2 rows reads whatever follows them. + lo, hi = int(perm.min()), int(perm.max()) + assert lo >= -1 and hi < fc2.shape[0], ( + f"perm spans [{lo}, {hi}]; entries must be -1 (dropped) or a row of fc2 (< {fc2.shape[0]})" + ) + + +m3_mc_finalize_allreduce_norm = M3McFinalizeAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding/__init__.py new file mode 100644 index 000000000000..e9f4d8e20b94 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Embedding ops of this target: the generic token-embedding gather.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding/m3_embedding.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding/m3_embedding.py new file mode 100644 index 000000000000..f979fdecf376 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding/m3_embedding.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The generic token-embedding gather over the replicated table (an id outside the table gives a zero row).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modules.embedding import pre_comm_embedding_ops + + +class M3Embedding(OpWrapper): + """``table[ids]`` with a zero row for an id outside ``[0, V)`` (the speculative sampler pads with -1). + + The table is the full (replicated) bf16 embedding [V, 6144]; ids int32 / int64 of any count. The fused decode + embedding (``m3_embedding_firstnorm``) takes decode steps of up to 8 rows; this entry takes every other step. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="a prefill chunk, int32 ids", spec=dict(rows=2048, id_dtype=torch.int32, pad=False) + ), + Cell( + why="a 48-row decode verify, int64 ids", + spec=dict(rows=48, id_dtype=torch.int64, pad=False), + ), + Cell( + why="draft rows padded with -1 ids", spec=dict(rows=32, id_dtype=torch.int32, pad=True) + ), + ) + + note = """ + An exact gather: `reference` must match bit for bit. + """ + + def __call__(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + return pre_comm_embedding_ops(ids, table, 1, 0, None, 0, table.shape[0], False, 0) + + def reference(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] + return rows * valid.unsqueeze(-1).to(table.dtype) + + +m3_embedding = M3Embedding() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/__init__.py new file mode 100644 index 000000000000..4c6fc92cd49c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 decode embedding gather + first-layer RMSNorm as one CuTe DSL kernel (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/m3_embedding_firstnorm_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/m3_embedding_firstnorm_kernel.py new file mode 100644 index 000000000000..78563b4db40d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/m3_embedding_firstnorm_kernel.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fixed M3 H6144 gather/raw residual/RMS, reusing pinned K3 norm kernel. + +128 threads x 8 BF16 x 6 iterations covers the row. Unlike the original +K3 wrapper this target does not promise FlashInfer's bit-identical reduction. +""" + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass.experimental import primitives as prims + +NORM_THREADS = 128 +NORM_VEC = 8 +NORM_WARPS = 4 # capacity: fixed: the CTA's warps + + +@cute.kernel +def m3_embed_norm_kernel( + ids: cute.Tensor, # int32 or int64 [N] + table: cute.Tensor, # bf16 [V, H] + weight: cute.Tensor, # bf16 [H] + raw: cute.Tensor, # bf16 [N, H], out: the rows + out: cute.Tensor, # bf16 [N, H], out: the normed rows + vocab: cutlass.Int64, + eps: cutlass.Float32, + tv_layout: cute.Layout, + tiler_mn: cute.Shape, + hidden: cutlass.Constexpr[int], +): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + prims.griddepcontrol(prims.GridDepAction.WAIT) + + smem = cutlass.utils.SmemAllocator() + reduction_buffer = smem.allocate_tensor( + cutlass.Float32, cute.make_layout((1, NORM_WARPS)), byte_alignment=4 + ) + + tok = cutlass.Int64(ids[bidx]) + valid = (tok >= cutlass.Int64(0)) & (tok < vocab) + row = cutlass.Int64(cutlass.select_(valid, tok, cutlass.Int64(0))) + + gX = cute.local_tile(table, tiler_mn, (row, 0)) + gR = cute.local_tile(raw, tiler_mn, (bidx, 0)) + gY = cute.local_tile(out, tiler_mn, (bidx, 0)) + w_layout = cute.prepend(weight.layout, cute.make_layout((tiler_mn[0],), stride=(0,))) + gW = cute.local_tile(cute.make_tensor(weight.iterator, w_layout), tiler_mn, (0, 0)) + + copy_atom_load = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), table.element_type, num_bits_per_copy=128 + ) + copy_atom_store = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), out.element_type, num_bits_per_copy=128 + ) + thr_copy_X = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_W = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_O = cute.make_tiled_copy(copy_atom_store, tv_layout, tiler_mn).get_slice(tidx) + + tXgX = thr_copy_X.partition_S(gX) + tXrX = cute.make_fragment_like(tXgX) + tWgW = thr_copy_W.partition_S(gW) + tWrW = cute.make_fragment_like(tWgW) + tXrW = thr_copy_X.retile(tWrW) + tXgR = thr_copy_O.partition_D(gR) + tXgO = thr_copy_O.partition_D(gY) + tXrO = cute.make_fragment_like(tXgO) + + # The row (zero for an id outside [0, V)), straight to the snapshot slot; the weight. + tXrX.store(cute.zeros_like(tXrX, dtype=table.element_type)) + if valid: + cute.copy(copy_atom_load, tXgX, tXrX) + cute.copy(copy_atom_load, tWgW, tWrW) + cute.copy(copy_atom_store, tXrX, tXgR) + + # flashinfer's RMSNormKernel arithmetic, in its order (see the header). + x = tXrX.load().to(cutlass.Float32) + x_sq = x * x + sum_sq = x_sq.reduce(cute.ReductionOp.ADD, init_val=cutlass.Float32(0.0), reduction_profile=0) + for i in cutlass.range_constexpr(5): + sum_sq = sum_sq + cute.arch.shuffle_sync_bfly(sum_sq, offset=1 << i) + lane = cute.arch.lane_idx() + warp = cute.arch.warp_idx() + if lane == 0: + reduction_buffer[0, warp] = sum_sq + cute.arch.barrier() + total = cutlass.Float32(0.0) + if lane < NORM_WARPS: + total = reduction_buffer[0, lane] + for i in cutlass.range_constexpr(5): + total = total + cute.arch.shuffle_sync_bfly(total, offset=1 << i) + mean_sq = total / cutlass.Float32(hidden) + rstd = cute.math.rsqrt(mean_sq + eps, fastmath=True) + w = tXrW.load().to(cutlass.Float32) + y = x * rstd * (w + cutlass.Float32(0.0)) + tXrO.store(y.to(out.element_type)) + cute.copy(copy_atom_store, tXrO, tXgO) + + +@cute.jit +def m3_embed_norm( + ids: cute.Tensor, + table: cute.Tensor, + weight: cute.Tensor, + raw: cute.Tensor, + out: cute.Tensor, + vocab: cutlass.Int64, + eps: cutlass.Float32, + n_tokens: cutlass.Constexpr[int], + hidden: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + # flashinfer's thread-value layout for one row: thread t, value (v, k) -> column 8 t + v + 1024 k. + blocks = hidden // (NORM_VEC * NORM_THREADS) + tv_layout = cute.make_layout(((NORM_THREADS, 1), (NORM_VEC, blocks)), + stride=((NORM_VEC, 1), (1, NORM_VEC * NORM_THREADS))) # fmt: skip + m3_embed_norm_kernel( + ids, table, weight, raw, out, vocab, eps, tv_layout, (1, hidden), hidden + ).launch( + grid=[n_tokens, 1, 1], + block=[NORM_THREADS, 1, 1], + smem=NORM_WARPS * 4, + stream=stream, + use_pdl=use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/op.py new file mode 100644 index 000000000000..276f8c32023b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/embedding_firstnorm/op.py @@ -0,0 +1,142 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Host side of the MiniMax-M3 decode embedding + first-layer RMSNorm (m3_embedding_firstnorm_kernel.py): H 6144, +1..8 rows, into caller-owned buffers. + +Compiled on the first call per (rows, id dtype, table shape, PDL), which must happen outside CUDA-graph capture +(:class:`DecodeEmbedding` compiles at construction). +""" + +import threading +from typing import Dict, Tuple + +import cuda.bindings.driver as cuda_driver +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import from_dlpack + +from .m3_embedding_firstnorm_kernel import m3_embed_norm as _m3_embed_norm_jit + +HIDDEN = 6144 # capacity: fixed: the checkpoint's hidden size +MAX_ROWS = 8 # capacity: fixed: the kernel's built rows (decode steps) + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + + +def arg(tensor): + return from_dlpack( + tensor.detach(), + assumed_align=tensor.element_size() + if tensor.ndim == 1 and tensor.dtype in (torch.int32, torch.int64) + else 16, + ) + + +def _check(ids, table, weight, raw, out) -> None: + rows = ids.shape[0] if ids.dim() == 1 else 0 + if not ( + 1 <= rows <= MAX_ROWS + and ids.dtype in (torch.int32, torch.int64) + and ids.is_contiguous() + and table.dtype == torch.bfloat16 + and table.dim() == 2 + and table.shape[1] == HIDDEN + and table.is_contiguous() + and weight.dtype == torch.bfloat16 + and tuple(weight.shape) == (HIDDEN,) + and all( + t.dtype == torch.bfloat16 and tuple(t.shape) == (rows, HIDDEN) and t.is_contiguous() + for t in (raw, out) + ) + ): + raise ValueError( + "m3_embed_norm: expects ids int32/int64 [1..8], a contiguous bf16 table [V, 6144], bf16 weight [6144] " + "and contiguous bf16 raw / out [rows, 6144]" + ) + + +def _compile(ids, table, weight, eps, raw, out, use_pdl: bool): + key = (ids.shape[0], ids.dtype, tuple(table.shape), table.device, bool(use_pdl)) + fn = _compiled.get(key) + if fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_embed_norm compiles on its first call per shape: call it once before capture" + ) + with _lock: + fn = _compiled.get(key) + if fn is None: + stream = cuda_driver.CUstream(torch.cuda.current_stream(table.device).cuda_stream) + fn = _compiled[key] = cute.compile( + _m3_embed_norm_jit, + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + ids.shape[0], + HIDDEN, + bool(use_pdl), + stream, + ) + return fn + + +def m3_embed_norm( + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """Write ``table[ids]`` into ``raw`` (a zero row for an id outside ``[0, V)``) and its RMSNorm with ``weight`` / + ``eps`` into ``out``; returns ``(out, raw)``.""" + _check(ids, table, weight, raw, out) + fn = _compile(ids, table, weight, eps, raw, out, use_pdl) + stream = cuda_driver.CUstream(torch.cuda.current_stream(ids.device).cuda_stream) + fn( + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + stream, + ) + return out, raw + + +class DecodeEmbedding: + """Caller-owned buffers of one (rows, id dtype) decode embedding: ``raw`` (the gathered rows, the first residual) + and ``output`` (their RMSNorm), over ``table`` and the first layer's plain (non-Gemma) input norm ``weight`` / + ``eps``. Built (and compiled) before CUDA-graph capture.""" + + def __init__(self, table, weight, eps, rows, id_dtype, use_pdl=True): + assert not torch.cuda.is_current_stream_capturing() + assert table.dtype == torch.bfloat16 and table.is_contiguous() and table.shape[1] == HIDDEN + assert weight.dtype == torch.bfloat16 and weight.shape == (HIDDEN,) + assert 1 <= rows <= MAX_ROWS and id_dtype in (torch.int32, torch.int64) + self.table, self.weight, self.rows, self.id_dtype, self.use_pdl = ( + table, + weight, + rows, + id_dtype, + use_pdl, + ) + self.eps = float(eps) + self.raw = torch.empty(rows, HIDDEN, device=table.device, dtype=torch.bfloat16) + self.output = torch.empty_like(self.raw) + ids = torch.empty(rows, device=table.device, dtype=id_dtype) + _compile(ids, table, weight, self.eps, self.raw, self.output, use_pdl) + + def __call__(self, ids): + assert ids.shape == (self.rows,) and ids.dtype == self.id_dtype and ids.is_contiguous() + return m3_embed_norm( + ids, self.table, self.weight, self.eps, self.raw, self.output, self.use_pdl + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/__init__.py new file mode 100644 index 000000000000..9cc374bdf1cc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GEMM entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_bf16_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_bf16_linear.py new file mode 100644 index 000000000000..d74fc709dbdf --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_bf16_linear.py @@ -0,0 +1,41 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The BF16 Eagle3 drafter's projections: ``x @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3Bf16Linear(OpWrapper): + """``F.linear(x, w)`` for bf16 ``x`` [M, K] and a bf16 weight [N, K] (cuBLAS, fp32 accumulation), bf16 out. + + The drafter's fc over the captured target layers (K = 3 x 6144), its fused QKV (K = 2 x 6144, this rank's heads), + o_proj (this rank's input columns), gate_up and down projections run it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="drafter fc, 32 verify rows", spec=dict(m=32, n=6144, k=18432)), + Cell(why="drafter QKV, eight draft rows", spec=dict(m=8, n=2304, k=12288)), + Cell(why="drafter o_proj, a prefill chunk", spec=dict(m=1024, n=6144, k=2048)), + Cell(why="drafter gate_up, one row", spec=dict(m=1, n=9216, k=6144)), + Cell(why="drafter down, eight rows", spec=dict(m=8, n=6144, k=4608)), + ) + + note = """ + cuBLAS's split and accumulation order: `reference` multiplies in fp32 (one bf16 rounding), within the test's band. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x, w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return (x.float() @ w.float().t()).to(x.dtype) + + +m3_bf16_linear = M3Bf16Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_fp32_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_fp32_linear.py new file mode 100644 index 000000000000..8d7374810675 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_fp32_linear.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The router's fp32 projection beyond the decode routers' rows: ``float(x) @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + + +class M3Fp32Linear(OpWrapper): + """fp32 router logits ``F.linear(float(x[M, 6144]), w[128, 6144])`` for prefill and steps above 48 rows (the + fp32 GEMV and the tensor-core router take 1..48). Whatever fp32 matmul precision the process runs with applies + (the engine leaves torch's default).""" + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="a prefill chunk", spec=dict(m=2048, e=128, k=6144)), + Cell( + why="a generation step above the tensor-core router's 48 rows", + spec=dict(m=64, e=128, k=6144), + ), + ) + + note = """ + Under torch's default fp32 matmul precision the product may be TF32: the test's band covers that rounding against + the true fp32 reference. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x.to(torch.float32), w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + with true_fp32_matmul(): + return x.float() @ w.t() + + +m3_fp32_linear = M3Fp32Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mx_gemv.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mx_gemv.py new file mode 100644 index 000000000000..a4cbb12d8085 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mx_gemv.py @@ -0,0 +1,166 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 x MXFP8 GEMV (tcgen05, block-scaled, CuTe DSL) on an activation its producer already quantized.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..moe.m3_moe_engine import mxfp8_dequant +from ..mx_gemv import op as _gemv + +__all__ = ["PROJECTIONS", "gemv_is_valid", "m3_mx_gemv", "served_cells"] + +# (N, K) of this target's MXFP8 projections at tp4, the GEMV's tuned shapes. +QKV_SPARSE = (2560, 6144) +QKV = (2304, 6144) +O_PROJ = (6144, 2048) +SHARED_GATE_UP = (1536, 6144) +SHARED_DOWN = (6144, 768) +DENSE_GATE_UP = (6144, 6144) +DENSE_DOWN = (6144, 3072) +PROJECTIONS = { + "sparse-layer QKV": QKV_SPARSE, + "dense-attention QKV": QKV, + "o_proj": O_PROJ, + "shared gate_up": SHARED_GATE_UP, + "shared down": SHARED_DOWN, + "dense gate_up": DENSE_GATE_UP, + "dense down": DENSE_DOWN, +} + + +def served_cells(named: tuple[Cell, ...], x_bf16: bool, max_m: int) -> tuple[Cell, ...]: + """A cell for every (projection, rows) the GEMV serves up to ``max_m`` rows that ``named`` does not list. + + Certification is per projection: a projection's split-K, ring and MMA width differ, so each (N, K, M) the + decode dispatch can send here is its own cell. + """ + listed = {(cell.spec["nk"], cell.spec["m"]) for cell in named} + return tuple( + Cell(why=f"{name}, {m} rows", spec=dict(nk=nk, m=m)) + for name, nk in PROJECTIONS.items() + for m in range(1, max_m + 1) + if (nk, m) not in listed and _gemv.serves(nk[0], nk[1], m, x_bf16) + ) + + +def gemv_reference( + x: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor +) -> torch.Tensor: + """bf16(x @ dequant(weight)^T) for fp32 activation values x [M, K], summed in fp64 and rounded once.""" + return (x.double() @ mxfp8_dequant(weight, weight_scale).double().t()).to(torch.bfloat16) + + +def gemv_is_valid(weight: torch.Tensor, weight_scale: torch.Tensor) -> None: + n, k = weight.shape + # The kernel reads both from data_ptr() as dense buffers at the layout it was compiled for. + assert weight.dtype == torch.float8_e4m3fn and weight.is_contiguous(), ( + "weight must be contiguous e4m3 [N, K]" + ) + assert weight_scale.numel() == n * k // 32, ( + "weight_scale must be the 128x4-swizzled UE8M0 buffer [N * K / 32]" + ) + + +class M3MxGemv(OpWrapper): + """bf16 [M, N] = xq [M, K] (e4m3, 128x4-swizzled UE8M0 scales ``xsf``) x weight [N, K]^T (e4m3, swizzled + ``weight_scale``), M <= 48; None when the shape is not served (the caller falls back). + + ``trigger_early``: release the PDL dependents before the epilogue. ``cfg``: a (split, ring[, contig, nacc, a_tmem, + st_async]) tuning override; default the table's. ``use_pdl``: default the environment's. ``ar`` = (multicast + workspace address, flags address, rank, world, bucket tokens, mode): also push the rows into the next boundary + all-reduce's exchange (then ``op.last_pushed`` is True). + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The projections' (N, K) at the token counts each build differs in: the MMA width (8 / 16 / 32 / 48 columns) a + # call uses is the smallest that holds M, and the shared gate_up takes another split at 2 tokens and above 8. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token (split 4)", spec=dict(nk=QKV_SPARSE, m=1)), + Cell(why="sparse-layer QKV, 8 tokens", spec=dict(nk=QKV_SPARSE, m=8)), + Cell( + why="sparse-layer QKV, 48 verify rows: the widest MMA", spec=dict(nk=QKV_SPARSE, m=48) + ), + Cell(why="dense-attention QKV, one token", spec=dict(nk=QKV, m=1)), + Cell(why="dense-attention QKV, 13 rows: the 16-column MMA", spec=dict(nk=QKV, m=13)), + Cell(why="o_proj, one token (split 2)", spec=dict(nk=O_PROJ, m=1)), + Cell(why="o_proj, 24 rows: the 32-column MMA", spec=dict(nk=O_PROJ, m=24)), + Cell( + why="shared gate_up, one token (A from shared memory)", + spec=dict(nk=SHARED_GATE_UP, m=1), + ), + Cell(why="shared gate_up, 2 tokens: its split-4 config", spec=dict(nk=SHARED_GATE_UP, m=2)), + Cell( + why="shared gate_up, 12 rows: the wide split-4 config", + spec=dict(nk=SHARED_GATE_UP, m=12), + ), + Cell(why="shared down, one token (unsplit, K 768)", spec=dict(nk=SHARED_DOWN, m=1)), + Cell(why="shared down, 8 tokens", spec=dict(nk=SHARED_DOWN, m=8)), + Cell(why="dense gate_up, one token", spec=dict(nk=DENSE_GATE_UP, m=1)), + Cell(why="dense down, 8 tokens", spec=dict(nk=DENSE_DOWN, m=8)), + ) + # Every other (projection, rows) the decode dispatch can send here (dispatch.MX_GEMV_ROWS). + CELLS = CELLS + served_cells(CELLS, x_bf16=False, max_m=48) + + note = """ + The kernel sums in fp32 in split-K order where `reference` sums in fp64: the bf16 outputs agree within the band the + test derives. Not covered: the all-reduce push (`ar`), which needs a multicast exchange; `cfg` overrides beyond the + table (the target passes none). Shapes outside the table, M > 48, and 17..32 rows at K < 1024 return None. + """ + + def __call__( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv(xq, xsf, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The dequantized product; the launch options are ignored.""" + return gemv_reference(mxfp8_dequant(xq, xsf), weight, weight_scale) + + def is_valid( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + m, k = xq.shape + assert xq.dtype == torch.float8_e4m3fn and xq.is_contiguous(), ( + "xq must be contiguous e4m3 [M, K]" + ) + # A linear-layout [M, K / 32] scale buffer is read as the swizzled one: wrong scales, no error. + assert xsf.numel() >= (m + 127) // 128 * 128 * (k // 32), ( + "xsf must be the 128x4-swizzled scale buffer (rows padded to 128), not the linear one" + ) + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv = M3MxGemv() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mx_gemv_bf16.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mx_gemv_bf16.py new file mode 100644 index 000000000000..1c6bbec4c10a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mx_gemv_bf16.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 GEMV (CuTe DSL) on a bf16 activation, quantized to MXFP8 in the kernel's prologue.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..moe.m3_moe_engine import mxfp8_quant_dequant +from ..mx_gemv import op as _gemv +from .m3_mx_gemv import ( + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, + gemv_is_valid, + gemv_reference, + served_cells, +) + +__all__ = ["m3_mx_gemv_bf16"] + + +class M3MxGemvBf16(OpWrapper): + """bf16 [M, N] = mxfp8_quantize(x) x weight [N, K]^T for a contiguous, 16-byte aligned bf16 ``x`` [M, K]; the + quantization is mxfp8_quantize's (UE8M0 scale per 32 = the smallest power of two >= amax / 448). None when the + shape is not served. The other arguments as ``m3_mx_gemv``'s. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The prologue quantizes one 32-element block per thread and round: the projections at the token counts the + # target sends here (no producer-quantized activation), up to the widest each serves. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token", spec=dict(nk=QKV_SPARSE, m=1)), + Cell( + why="sparse-layer QKV, 16 rows: the most its split-4 prologue serves", + spec=dict(nk=QKV_SPARSE, m=16), + ), + Cell(why="dense-attention QKV, 8 tokens", spec=dict(nk=QKV, m=8)), + Cell(why="o_proj, one token", spec=dict(nk=O_PROJ, m=1)), + Cell(why="o_proj, 32 rows: the 32-column MMA", spec=dict(nk=O_PROJ, m=32)), + Cell(why="shared gate_up, one token", spec=dict(nk=SHARED_GATE_UP, m=1)), + Cell(why="shared gate_up, 2 tokens: its split-4 config", spec=dict(nk=SHARED_GATE_UP, m=2)), + Cell(why="shared down, 8 tokens (K 768)", spec=dict(nk=SHARED_DOWN, m=8)), + Cell( + why="dense gate_up, 8 tokens: the most split 2 at K 6144 serves", + spec=dict(nk=DENSE_GATE_UP, m=8), + ), + Cell(why="dense down, one token", spec=dict(nk=DENSE_DOWN, m=1)), + ) + # Every other (projection, rows) the decode dispatch can send here (dispatch.MX_GEMV_BF16_MAX_TOKENS). + CELLS = CELLS + served_cells(CELLS, x_bf16=True, max_m=32) + + note = """ + As `m3_mx_gemv`. The in-kernel quantization is mxfp8_quantize's bit for bit except where the kernel's + rcp.approx(448) and an exact 1/448 put a block's amax on different sides of a power of two (not observed on the + cells). Not covered: `ar`, `cfg` overrides. Returns None for rows beyond 8 rounds of the prologue per CTA. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv_bf16(x, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The MXFP8 round trip of x times the dequantized weight; the launch options are ignored.""" + return gemv_reference(mxfp8_quant_dequant(x), weight, weight_scale) + + def is_valid( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv_bf16 = M3MxGemvBf16() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_gemm.py new file mode 100644 index 000000000000..8671a81c2227 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_gemm.py @@ -0,0 +1,76 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 GEMM on an activation its producer already quantized (rows the GEMV does not serve).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import ( + flashinfer_mxfp8_gemm_cute_dsl, + mxfp8_gemm_autotuned, +) + +from .m3_mxfp8_linear import mxfp8_reference + + +class M3Mxfp8Gemm(OpWrapper): + """``act @ weight^T`` for an MXFP8 activation (e4m3 [M, K] + 128x4-swizzled UE8M0 scales) and an MXFP8 weight + [N, K] (swizzled scales), bf16 out. The GEMM backend per token bucket is the AutoTuner's (the same runner, op key + and tactic cache as the engine-tuned quantize + GEMM); ``tune`` profiles in the engine's tuning pass.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The decode dispatch hands it the boundary / merge MXFP8 activation on rows the GEMV does not serve: the + # unsplit shared down projection at 17..32 rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert down projection, 17 rows", spec=dict(m=17, n=6144, k=768)), + Cell(why="shared expert down projection, 32 rows", spec=dict(m=32, n=6144, k=768)), + ) + + note = """ + Exactly the MXFP8 product (fp32 accumulation, one bf16 rounding): `reference` dequantizes both operands and + multiplies in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + return mxfp8_gemm_autotuned(act, act_scale, weight, weight_scale, output_dtype, tune=tune) + + def reference( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + """The dequantized product; ``tune`` is ignored.""" + return mxfp8_reference(act, act_scale, weight, weight_scale).to(output_dtype) + + +m3_mxfp8_gemm = M3Mxfp8Gemm() + + +def m3_mxfp8_gemm_cute_dsl( + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, +) -> torch.Tensor: + """The same product on FlashInfer's CuTe DSL backend, fixed (its heuristic config), for the projections the + target pins to it: the AutoTuner, profiling warm weights per start, can pick the CUTLASS backend there, which + streams the cold decode weight about 2x slower.""" + return flashinfer_mxfp8_gemm_cute_dsl( + act, act_scale, weight, weight_scale, output_dtype, sweep_configs=False + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_linear.py new file mode 100644 index 000000000000..799cda007837 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_linear.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One MXFP8 projection on the engine-tuned path: dynamic MXFP8 quantize of a bf16 activation + block-scaled GEMM.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import mxfp8_quantize_gemm_autotuned + +_BLOCK = 32 + + +def _unswizzle(scale: torch.Tensor, rows: int, cols: int) -> torch.Tensor: + """[rows, cols] UE8M0 bytes from the 128x4-swizzled layout (rows padded to 128, columns to 4).""" + pad_r = (rows + 127) // 128 * 128 + pad_c = (cols + 3) // 4 * 4 + s = scale.view(torch.uint8).reshape(pad_r // 128, pad_c // 4, 32, 4, 4) + return s.permute(0, 3, 2, 1, 4).reshape(pad_r, pad_c)[:rows, :cols] + + +def _dequant(x: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + rows, k = x.shape + s = _unswizzle(scale, rows, k // _BLOCK).to(torch.int32) + factor = torch.pow(2.0, (s - 127).to(torch.float32)) + return ( + x.view(torch.float8_e4m3fn).float().view(rows, k // _BLOCK, _BLOCK) * factor.unsqueeze(-1) + ).view(rows, k) + + +def mxfp8_reference(act, act_scale, weight, weight_scale) -> torch.Tensor: + """fp32 product of two dequantized MXFP8 operands (swizzled UE8M0 scales).""" + with true_fp32_matmul(): + return _dequant(act, act_scale) @ _dequant(weight, weight_scale).t() + + +class M3Mxfp8Linear(OpWrapper): + """``x @ weight^T`` for bf16 ``x`` [M, K] and an MXFP8 weight [N, K] (e4m3 + 128x4-swizzled UE8M0 scales), bf16 + out, on the backend the engine selected: + + * ``tuned_quantize_gemm`` (decode CUDA graphs once the engine tuned per-bucket backends): the AutoTuner's quantize + and GEMM per token bucket (``tune`` profiles them in the engine's tuning pass); + * otherwise the native ``mxfp8_quantize`` (swizzled scales) and then FlashInfer's MXFP8 GEMM (``flashinfer``) or + the CUTLASS block-scaled GEMM, through the autotuner while the engine tunes it (``native_tuning``), scaled by + ``alpha`` (fp32 [1], 1.0 for these checkpoints). + + Every MXFP8 projection of the target (the attention QKV and o_proj, the dense MLPs and the shared expert) runs it + outside the decode-step kernels. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="sparse-layer QKV, a prefill chunk, native GEMM", + spec=dict(m=2048, n=2560, k=6144, path="native"), + ), + Cell( + why="o_proj, a prefill chunk, native GEMM", + spec=dict(m=2048, n=6144, k=2048, path="native"), + ), + Cell( + why="dense down projection, eager decode, native GEMM", + spec=dict(m=4, n=6144, k=3072, path="native"), + ), + Cell( + why="shared gate_up, FlashInfer GEMM", + spec=dict(m=64, n=1536, k=6144, path="flashinfer"), + ), + Cell(why="o_proj, tuned quantize + GEMM", spec=dict(m=64, n=6144, k=2048, path="tuned")), + ) + + note = """ + Exactly the MXFP8 product of the MXFP8 round trip of x (fp32 accumulation, one bf16 rounding): `reference` + quantizes x with the same block rule (UE8M0 scale = the block's power-of-two ceiling over e4m3's 448) and + multiplies dequantized operands in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + if tuned_quantize_gemm: + return mxfp8_quantize_gemm_autotuned(x, weight, weight_scale, output_dtype, tune=tune) + act, act_scale = torch.ops.trtllm.mxfp8_quantize(x, True) + if flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8( + act, act_scale, weight, weight_scale, output_dtype + ) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, weight, weight_scale, alpha, output_dtype) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + """The MXFP8 round trip of ``x`` times the dequantized weight, times ``alpha``; the backend flags are + ignored.""" + m, k = x.shape + blocks = x.float().view(m, k // _BLOCK, _BLOCK) + amax = blocks.abs().amax(-1, keepdim=True) + exp = torch.ceil(torch.log2(torch.where(amax > 0, amax / 448.0, torch.ones_like(amax)))) + scale = torch.pow(2.0, exp) + q = (blocks / scale).to(torch.float8_e4m3fn).float() * scale + with true_fp32_matmul(): + out = q.view(m, k) @ _dequant(weight, weight_scale).t() + return (out * alpha.float()).to(output_dtype) + + +m3_mxfp8_linear = M3Mxfp8Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_pinned_quantize.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_pinned_quantize.py new file mode 100644 index 000000000000..a141b7d639c4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_mxfp8_pinned_quantize.py @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 activation quantize at a fixed tactic (QPIN), ahead of the tuned ``m3_mxfp8_gemm``.""" + +from __future__ import annotations + +from typing import Dict, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner + +_RUNNERS: Dict[torch.dtype, object] = {} + + +def _runner(dtype: torch.dtype): + runner = _RUNNERS.get(dtype) + if runner is None: + runner = _RUNNERS[dtype] = MXFP8QuantizeRunner(dtype) + return runner + + +class M3Mxfp8PinnedQuantize(OpWrapper): + """A bf16 activation [M, K] to MXFP8: (e4m3 [M, K], swizzled UE8M0 scales), on the runner's ``tactic``.""" + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="a 16-row decode verify on a K 6144 input (c4)", spec=dict(m=16, k=6144)), + Cell(why="dense-MLP down projection input, 32 rows", spec=dict(m=32, k=3072)), + ) + + note = """ + Bit-exact: `reference` is the stock MXFP8 quantize (``torch.ops.trtllm.mxfp8_quantize``, swizzled scales) of the + same input; every tactic computes the same block maxima and roundings. + """ + + def __call__(self, x: torch.Tensor, tactic: int) -> Tuple[torch.Tensor, torch.Tensor]: + return _runner(x.dtype).forward([x], tactic=tactic) + + def reference(self, x: torch.Tensor, tactic: int) -> Tuple[torch.Tensor, torch.Tensor]: + """The stock quantize; ``tactic`` is ignored.""" + return torch.ops.trtllm.mxfp8_quantize(x, True) + + +m3_mxfp8_pinned_quantize = M3Mxfp8PinnedQuantize() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_target_head.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_target_head.py new file mode 100644 index 000000000000..5f3b4ad1e2b0 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/m3_target_head.py @@ -0,0 +1,346 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The target lm_head shard GEMV of a verify step of at most 8 rows over a lossless 12-bit coding of its BF16 weight, +and that coding (``encode_chunked`` / ``decode_reference`` / ``round_trip_ok``). + +The coding: the BF16 exponents of a trained weight sit almost all in a window of 15 consecutive values +``[base, base + 14]``. Each weight is one byte ``sign << 7 | mantissa`` plus a 4-bit code ``exponent - base``; code 15 +marks an exception, which decodes to +0 (it adds nothing to the MMA sums); its exact product ``x * w`` is added from a +per-tile list. The bytes are laid out in ``mma.m16n8k16`` A-fragment order: +per 16-row tile and unit of 4 k16 steps, each lane reads the sign/mantissa bytes of steps 0-1, of steps 2-3, then 4 +code words (``[tiles][K / 64][3][32 lanes][16 bytes]``). A coded weight is used only after ``round_trip_ok`` proved +it decodes to the BF16 weight bit for bit. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_target_head"] + +#: Verify rows per call (requests x (draft length + 1)). +MAX_ROWS = 8 +#: Warps per CTA (the split-tile scratch holds 2 x 4 x 32 fp32 per warp). +CTA_WARPS = 16 +#: The target lm_head's TP4 shard (200064-id vocabulary) and the hidden size. +TARGET_SHARD = 50016 +HIDDEN = 6144 +#: Codes per window: 0..14 decode, 15 marks an exception. +WINDOW = 15 +#: The largest exponent base whose window top plus one (base + 15) is still a finite BF16 exponent. +MAX_BASE = 254 - WINDOW +#: Rows per block of encode_chunked / round_trip_ok (bounded temporaries). +ROWS_PER_CHUNK = 4096 + + +def _pad_rows(weight: torch.Tensor) -> torch.Tensor: + n = weight.shape[0] + tiles = (n + 15) // 16 + if tiles * 16 == n: + return weight + return torch.cat([weight, weight.new_zeros(tiles * 16 - n, weight.shape[1])]) + + +def _fragment_view(w: torch.Tensor) -> torch.Tensor: + """[tiles * 16, K] -> [tiles, K / 16, 32 lanes, 8 values] in mma A-fragment order (a permutation of the values).""" + n, k = w.shape + # rows: (tile, rh, g); cols: (ks, ch, j, e) -> (tile, ks, g, j, ch, rh, e) + return ( + w.view(n // 16, 2, 8, k // 16, 2, 4, 2) + .permute(0, 3, 2, 5, 4, 1, 6) + .reshape(n // 16, k // 16, 32, 8) + ) + + +def _exponents(weight: torch.Tensor) -> torch.Tensor: + return (weight.contiguous().view(torch.int16).to(torch.int32) >> 7) & 0xFF + + +def _window_base(hist: torch.Tensor) -> int: + """The base of the 15-exponent window holding the most weights (the lowest such base).""" + window = torch.stack([hist[b : b + WINDOW].sum() for b in range(0, MAX_BASE + 1)]) + return int(window.argmax()) + + +def encode(weight: torch.Tensor, base: Optional[int] = None): + """Lossless 12-bit coding of a BF16 [N, K] weight (see the module docstring); ``base`` defaults to this weight's + own window. + + Returns (packed uint8 [tiles * K / 64 * 1536], base, exc_off int32 [tiles + 1], exc_pos int32 [E] (row in tile + << 16 | k), exc_w fp32 [E, 2] ({w, 0})). + """ + w = _pad_rows(weight) + n, k = w.shape + tiles, ks = n // 16, k // 16 + assert ks % 4 == 0, "K must be a multiple of 64" + bits = w.contiguous().view(torch.int16).to(torch.int32) & 0xFFFF + expo = (bits >> 7) & 0xFF + if base is None: + base = _window_base(torch.bincount(expo.flatten(), minlength=256)) + code = expo - base + exc = (code < 0) | (code >= WINDOW) + code = torch.where(exc, torch.full_like(code, WINDOW), code) + sm = ((bits >> 8) & 0x80) | (bits & 0x7F) + sm = torch.where(exc, torch.zeros_like(sm), sm) + # Exceptions: (tile, row in tile, k) in tile-major order; code 15 decodes to +0. + rows, cols = exc.nonzero(as_tuple=True) + order = torch.argsort((rows // 16).long() * (16 * k) + (rows % 16).long() * k + cols.long()) + rows, cols = rows[order], cols[order] + exc_w = torch.stack( + [ + w[rows, cols].float(), + torch.zeros((rows.numel(),), device=w.device), + ], + dim=1, + ) + exc_pos = (((rows % 16) << 16) | cols).to(torch.int32) + exc_off = torch.zeros(tiles + 1, dtype=torch.int32, device=w.device) + exc_off[1:] = torch.cumsum(torch.bincount((rows // 16).long(), minlength=tiles), 0).to( + torch.int32 + ) + # Fragment order, then per (tile, group of 4 k16 steps, lane): 32 sign/mantissa bytes and 4 code words. + smf = _fragment_view(sm.to(torch.uint8).view(n, k)).view(tiles, ks // 4, 4, 32, 8) + cf = ( + _fragment_view(code.to(torch.uint8).view(n, k)) + .view(tiles, ks // 4, 4, 32, 8) + .to(torch.int32) + ) + # Code word of a k16 step: value 2p -> bits 4p..4p+3, value 2p+1 -> bits 16+4p.. + cw = torch.zeros(tiles, ks // 4, 4, 32, dtype=torch.int64, device=w.device) + for p in range(4): + cw |= (cf[..., 2 * p].long() << (4 * p)) | (cf[..., 2 * p + 1].long() << (16 + 4 * p)) + cw = cw.to(torch.int32) # wraps for bit 31; the kernel reads it as uint32 + # Word 0 of a lane = the bytes of steps 0 and 1; word 1 = steps 2 and 3; word 2 = the code words of steps 0..3. + seg01 = ( + smf.permute(0, 1, 3, 2, 4).reshape(tiles, ks // 4, 32, 2, 16).permute(0, 1, 3, 2, 4) + ) # [.., 2, 32, 16] + seg2 = cw.permute(0, 1, 3, 2).contiguous().view(torch.uint8).view(tiles, ks // 4, 1, 32, 16) + packed = torch.cat([seg01.contiguous(), seg2], dim=2) # [tiles, ks / 4, 3, 32, 16] + return packed.contiguous().view(-1), base, exc_off, exc_pos, exc_w.contiguous() + + +def encode_chunked(weight: torch.Tensor, rows_per_chunk: int = ROWS_PER_CHUNK): + """``encode`` over row blocks (bounded temporaries), with the exponent window chosen on the whole weight.""" + n, _ = weight.shape + assert rows_per_chunk % 16 == 0 + hist = torch.zeros(256, dtype=torch.int64, device=weight.device) + for r in range(0, n, rows_per_chunk): + hist += torch.bincount(_exponents(weight[r : r + rows_per_chunk]).flatten(), minlength=256) + base = _window_base(hist) + packed, offs, poss, ws = [], [], [], [] + n_exc = 0 + for r in range(0, n, rows_per_chunk): + p, _, off, pos, ew = encode(weight[r : r + rows_per_chunk], base=base) + packed.append(p) + offs.append(off[:-1] + n_exc) + poss.append(pos) + ws.append(ew) + n_exc += pos.numel() + exc_off = torch.cat(offs + [torch.tensor([n_exc], dtype=torch.int32, device=weight.device)]) + return torch.cat(packed), base, exc_off, torch.cat(poss), torch.cat(ws) + + +def decode_reference(packed: torch.Tensor, base: int, n: int, k: int) -> torch.Tensor: + """BF16 [N, K] from the coded stream with the in-kernel rule (exceptions, code 15, decode to +0).""" + tiles, ks = (n + 15) // 16, k // 16 + p = packed.view(tiles, ks // 4, 3, 32, 16) + sm = ( + p[:, :, :2] + .permute(0, 1, 3, 2, 4) + .reshape(tiles, ks // 4, 32, 4, 8) + .permute(0, 1, 3, 2, 4) + .to(torch.int32) + ) + cw = ( + p[:, :, 2] + .contiguous() + .view(torch.int32) + .view(tiles, ks // 4, 32, 4) + .permute(0, 1, 3, 2) + .long() + & 0xFFFFFFFF + ) + codes = torch.stack([(cw >> (4 * (i // 2) + 16 * (i % 2))) & 0xF for i in range(8)], dim=-1).to( + torch.int32 + ) + bits = ((sm & 0x80) << 8) | ((codes + base) << 7) | (sm & 0x7F) + bits = torch.where(codes == WINDOW, torch.zeros_like(bits), bits) + frag = bits.reshape(tiles, ks, 32, 8) + # Inverse of _fragment_view: (tile, ks, g, j, ch, rh, e) -> (tile, rh, g, ks, ch, j, e). + w = frag.view(tiles, ks, 8, 4, 2, 2, 2).permute(0, 5, 2, 1, 4, 3, 6).reshape(tiles * 16, k) + return w.to(torch.int16).view(torch.bfloat16)[:n] + + +def decode( + packed: torch.Tensor, + base: int, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + n: int, + k: int, + row0: int = 0, + row1: Optional[int] = None, +) -> torch.Tensor: + """BF16 rows [row0, row1) of the coded weight [N, K] with the exceptions applied (what the kernel multiplies by). + ``row0`` must be a multiple of 16.""" + row1 = n if row1 is None else row1 + assert row0 % 16 == 0 + t0, t1 = row0 // 16, (row1 + 15) // 16 + unit_bytes = (k // 64) * 3 * 32 * 16 + dec = decode_reference(packed[t0 * unit_bytes : t1 * unit_bytes], base, row1 - row0, k).clone() + counts = (exc_off[1:] - exc_off[:-1]).long() + tile_of = torch.repeat_interleave(torch.arange(counts.numel(), device=packed.device), counts) + rows = tile_of * 16 + (exc_pos >> 16).long() + cols = (exc_pos & 0xFFFF).long() + sel = (rows >= row0) & (rows < row1) + dec[rows[sel] - row0, cols[sel]] = exc_w.view(-1, 2)[sel, 0].to(torch.bfloat16) + return dec + + +def round_trip_ok( + packed: torch.Tensor, + base: int, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + weight: torch.Tensor, + rows_per_chunk: int = ROWS_PER_CHUNK, +) -> bool: + """Whether the coded weight (code values + exception list) decodes to ``weight`` bit for bit, with its exponent + window below the BF16 infinity exponent.""" + n, k = weight.shape + if not 0 <= base <= MAX_BASE or k % 64: + return False + if ( + packed.numel() != ((n + 15) // 16) * 16 * k * 3 // 2 + or exc_off.numel() != (n + 15) // 16 + 1 + ): + return False + for r0 in range(0, n, rows_per_chunk): + r1 = min(n, r0 + rows_per_chunk) + dec = decode(packed, base, exc_off, exc_pos, exc_w, n, k, r0, r1) + if not torch.equal(dec.view(torch.int16), weight[r0:r1].contiguous().view(torch.int16)): + return False + return True + + +class M3TargetHead(OpWrapper): + """``out[:M, :n]`` (BF16) = x [M, K] (BF16, M <= 8) times the coded BF16 shard W [n, K] transposed: each logit + the fp32 sum of the exact BF16 products, rounded once to BF16 (the in-tree BF16 head GEMM's arithmetic, in + another summation order). + + ``w`` / ``exp_base`` / ``exc_off`` / ``exc_pos`` / ``exc_w`` are ``encode_chunked``'s outputs for W; ``part`` + (fp32 [grid * 16 * 256]) and ``counter`` (int32 [ceil(n / 16)], zero between calls, re-armed by the kernel) are + the split-tile scratch; ``grid`` is the number of 16-warp CTAs (one per SM). + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The verify of a fused-verify step at most 8 rows wide: 1 + 3 drafts for one or two requests (4 / 8 rows), and + # every other row count the kernel's one n8 MMA tile serves; one cell with an exception in many tiles, one with + # a shard whose rows are not a multiple of 16. + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"{m} verify row(s) of the TP4 target head shard", + spec=dict(m=m, n=TARGET_SHARD, k=HIDDEN), + ) + for m in range(1, MAX_ROWS + 1) + ), + Cell( + why="4 rows with outliers and exact zeros in many tiles (the exception lists)", + spec=dict(m=4, n=TARGET_SHARD, k=HIDDEN, exceptions=True), + ), + Cell( + why="2 rows over a shard of 4000 rows (250 tiles), K 1024", + spec=dict(m=2, n=4000, k=1024, exceptions=True), + ), + ) + + note = """ + The weight the kernel multiplies by is the checkpoint's BF16 weight bit for bit (round_trip_ok); an exception + adds nothing to the MMA sums and its exact product x * w is added at its tile's flush. A logit therefore differs + from the in-tree head's only by the fp32 summation order before its single BF16 rounding (ulp-level); + integer-valued data, whose sums are exact in any order, is equal bit for bit. Split tiles are summed in warp + order: the result is deterministic. + """ + + def __call__( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_target_head( + w, x, out, part, counter, exc_off, exc_pos, exc_w, exp_base, n, grid, pdl + ) + return out + + def reference( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> torch.Tensor: + """The BF16 logits [M, n] of x against the decoded weight: the fp32 product, rounded once to BF16. + + ``out``, ``part``, ``counter``, ``grid`` and ``pdl`` do not change the result.""" + weight = decode(w, exp_base, exc_off, exc_pos, exc_w, n, x.shape[1]) + with true_fp32_matmul(): + logits = x.float() @ weight.float().t() + return logits.to(torch.bfloat16) + + def is_valid( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> None: + # A split tile's last finisher is the CTA whose arrival makes the count reach its owners: a stale count + # stores a partial sum. + assert not bool(counter[: (n + 15) // 16].any()), "counter must be zero between calls" + # The exceptions are read per tile in list order with no bound against the coded stream. + assert int(exc_off[-1].item()) == exc_pos.numel(), "exc_off must end at the exception count" + assert bool((exc_off[1:] >= exc_off[:-1]).all()), "exc_off must be non-decreasing" + assert out.data_ptr() != x.data_ptr(), "out must not alias x" + + +m3_target_head = M3TargetHead() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/mxfp8_low_m_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/mxfp8_low_m_gemm.py new file mode 100644 index 000000000000..915c401a3ca1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gemm/mxfp8_low_m_gemm.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused MXFP8 activation quantization + low-M MXFP8 GEMM of the decode projections.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +_E4M3_MAX = 448.0 + + +def _ue8m0_unswizzle(wsf: torch.Tensor, n: int, k: int) -> torch.Tensor: + """[N, K / 32] scale bytes out of the 128x4-swizzled buffer of ``mxfp8_quantize(w, True)``.""" + cols = k // 32 + padded_cols = (cols + 3) // 4 * 4 + r = torch.arange(n, device=wsf.device).view(-1, 1) + c = torch.arange(cols, device=wsf.device).view(1, -1) + index = ( + (c % 4) + + (c // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * padded_cols) + ) + return wsf.reshape(-1)[index] + + +def _mxfp8_dequant_rows(x: torch.Tensor) -> torch.Tensor: + """The activation as the kernel quantizes it (``trtllm::mxfp8_quantize``), dequantized back to fp32. + + Per 32 columns: scale = the smallest power of two at or above ``amax / 448`` (UE8M0, round up), data = x / scale + rounded to nearest E4M3; an all-zero block stays zero. + """ + m, k = x.shape + blocks = x.float().view(m, k // 32, 32) + amax = blocks.abs().amax(-1, keepdim=True) + bits = (amax / _E4M3_MAX).view(torch.int32) + byte = (((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) > 0).to(torch.int32)).clamp(0, 254) + scale = torch.exp2(byte.float() - 127.0) + data = torch.where(amax != 0, blocks / scale, torch.zeros_like(blocks)).to(torch.float8_e4m3fn) + return (data.float() * scale).view(m, k) + + +# The target's configuration per (N, K) projection marked for this kernel, and the most rows it routes here +# (dispatch.LOW_M_GEMM_MAX_TOKENS): (warps, cluster_k, row_tiles, max rows). +CONFIGS = { + (2560, 6144): (4, 4, 2, 8), + (6144, 2048): (8, 2, 4, 8), + (6144, 6144): (4, 4, 1, 1), + (6144, 3072): (4, 2, 2, 8), +} + + +def _config_cells(named: tuple[Cell, ...]) -> tuple[Cell, ...]: + """A cell for every (projection, rows) of ``CONFIGS`` that ``named`` does not list: certification is per + projection, since each runs its own configuration.""" + listed = {(cell.spec["n"], cell.spec["k"], cell.spec["m"]) for cell in named} + return tuple( + Cell( + why=f"{n} x {k}, {m} rows", + spec=dict(m=m, n=n, k=k, warps=warps, cluster_k=cluster_k, row_tiles=row_tiles), + ) + for (n, k), (warps, cluster_k, row_tiles, max_m) in CONFIGS.items() + for m in range(1, max_m + 1) + if (n, k, m) not in listed + ) + + +class Mxfp8LowMGemm(OpWrapper): + """``out[M, N] = mxfp8(x)[M, K] @ dequant(w, wsf)[N, K]^T`` in BF16, quantizing ``x`` in the kernel. + + ``w`` is E4M3 [N, K] with UE8M0 block-32 scales ``wsf`` in the 128x4 swizzled layout; ``x`` BF16 [M, K] with unit + column stride. ``(warps, cluster_k, row_tiles)`` is the kernel configuration the target's table picks per + weight shape; each CTA owns ``16 * row_tiles`` weight rows and ``K / cluster_k`` columns, split-K partials meet + in the cluster root and are summed in rank order, one BF16 rounding at the end. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The tp4 projections marked for this kernel (the decode GEMV runs them first and hands over when it declines), + # each at the configuration the target's table assigns and up to the row count it routes here. + CELLS: tuple[Cell, ...] = ( + Cell( + why="QKV with the index q / k projections (2560 x 6144), one token", + spec=dict(m=1, n=2560, k=6144, warps=4, cluster_k=4, row_tiles=2), + ), + Cell( + why="QKV at the configured row limit, 8 tokens", + spec=dict(m=8, n=2560, k=6144, warps=4, cluster_k=4, row_tiles=2), + ), + Cell( + why="o_proj (6144 x 2048), 8 warps x 4 row tiles, odd row count", + spec=dict(m=5, n=6144, k=2048, warps=8, cluster_k=2, row_tiles=4), + ), + Cell( + why="dense MLP gate_up (6144 x 6144), configured for one token only", + spec=dict(m=1, n=6144, k=6144, warps=4, cluster_k=4, row_tiles=1), + ), + Cell( + why="dense MLP down (6144 x 3072), 8 tokens", + spec=dict(m=8, n=6144, k=3072, warps=4, cluster_k=2, row_tiles=2), + ), + ) + # Every other (projection, rows) of its configurations. + CELLS = CELLS + _config_cells(CELLS) + + note = """ + Only the four configurations the target's table assigns are built, each for 1..8 rows (one 8-row MMA tile). + `pdl` changes launch overlap only and is driven off. Activation blocks whose amax is below `448 * 2^-127` take a + flush-to-zero path the reference does not model; no decode activation is that small. + """ + + def __call__( + self, + x: torch.Tensor, + w: torch.Tensor, + wsf: torch.Tensor, + warps: int, + cluster_k: int, + row_tiles: int, + pdl: bool, + ) -> torch.Tensor: + return torch.ops.trtllm.minimax_m3_mxfp8_low_m_gemm( + x, w, wsf, warps, cluster_k, row_tiles, pdl + ) + + def reference( + self, + x: torch.Tensor, + w: torch.Tensor, + wsf: torch.Tensor, + warps: int, + cluster_k: int, + row_tiles: int, + pdl: bool, + ) -> torch.Tensor: + """Dequantized fp32 product of the MXFP8 activation and weight, rounded to BF16 once. + + ``warps``, ``cluster_k``, ``row_tiles`` and ``pdl`` only pick the kernel's split; they do not enter the + arithmetic. + """ + n, k = w.shape + w_scale = torch.exp2(_ue8m0_unswizzle(wsf, n, k).float() - 127.0) + w_deq = (w.float().view(n, k // 32, 32) * w_scale.unsqueeze(-1)).view(n, k) + with true_fp32_matmul(): + out = _mxfp8_dequant_rows(x) @ w_deq.t() + return out.to(torch.bfloat16) + + def is_valid( + self, + x: torch.Tensor, + w: torch.Tensor, + wsf: torch.Tensor, + warps: int, + cluster_k: int, + row_tiles: int, + pdl: bool, + ) -> None: + n, k = w.shape + # The kernel bulk-copies 256-byte scale chunks by computed offset; the op checks the dtype only, so a short + # or linear-layout scale buffer is read past its end or in the wrong order. + need = (n + 127) // 128 * 128 * ((k // 32 + 3) // 4 * 4) + assert wsf.is_contiguous() and wsf.numel() >= need, ( + f"wsf must be the contiguous 128x4-swizzled UE8M0 buffer of {need} bytes; got {wsf.numel()}" + ) + + +mxfp8_low_m_gemm = Mxfp8LowMGemm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gqamix_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gqamix_sparse_decode.py new file mode 100644 index 000000000000..e6b824de1b96 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/gqamix_sparse_decode.py @@ -0,0 +1,1311 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. +""" + +from __future__ import annotations + +import os +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +def _pdl_enabled() -> bool: + return os.environ.get("TRTLLM_ENABLE_PDL", "1") == "1" + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for _ in tl.range(chunk_start, chunk_end): + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for _ in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on both datacenter Blackwell targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3)) +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + if kv_nvfp4: + capability = torch.cuda.get_device_capability(q.device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else 1 + ) + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = _pdl_enabled() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + if not direct_output: + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + **pdl_launch, + **merge_launch_options, + ) + + +__all__ = [ + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/long_select.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/long_select.py new file mode 100644 index 000000000000..c01a82f0da41 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/long_select.py @@ -0,0 +1,62 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 block select for MiniMax-M3 decode rows of 129 to 9216 blocks. + +Same contract and bit-identical output as ``torch.ops.trtllm.minimax_m3_select_blocks`` on per-KV-head scores +[num_kv_heads, n_blocks, total_q]: init / local forcing, per-query valid-block count, ties to the smaller block index, +forced -inf picks emitted as -1, ascending block ids with -1 tail padding. + +One 1024-thread CTA per (query, KV head) row holds the row's ordered score keys in registers (2, 4 or 9 per +thread, by the row's valid count). The weakest of the 16 maxima of the interleaved groups ``block % 16`` is a lower +bound on the 16th-largest key, so the top 16 is among the keys at or above it; those are compacted to shared memory +and ranked exactly by (key, block). More than 1024 keys tied at the bound (degenerate rows) rank every candidate +against the whole row (re-read). + +The kernel is ``torch.ops.trtllm.minimax_m3_select_long``, called through this target's ``m3_select_long`` entry. +""" + +from typing import Optional + +import torch + +from ..ops.attention.m3_select_long import MAX_BLOCKS, m3_select_long + + +def long_select_blocks( + max_score_kv: torch.Tensor, + *, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, +) -> Optional[torch.Tensor]: + """Top-16 select over per-KV-head scores [num_kv_heads, n_blocks, total_q] with 129..9216 blocks. + + Returns the head-major backed [total_q, num_kv_heads, 16] int32 table, or None outside this kernel's shapes + (callers keep the reference op). ``early_trigger`` releases the dependent sparse decode right after the select's + grid wait (``SELECT_EARLY_MAX_ROWS`` in modeling.py). + """ + if ( + max_score_kv.dtype != torch.float32 + or not max_score_kv.is_cuda + or max_score_kv.dim() != 3 + or not 128 < max_score_kv.shape[1] <= MAX_BLOCKS + or n_valid_blocks.dtype != torch.int32 + or not n_valid_blocks.is_cuda + or n_valid_blocks.shape[0] < max_score_kv.shape[2] + ): + return None + num_kv_heads, _, total_q = max_score_kv.shape + out = torch.empty( + (num_kv_heads, total_q, 16), dtype=torch.int32, device=max_score_kv.device + ).permute(1, 0, 2) + return m3_select_long( + max_score_kv, + n_valid_blocks.contiguous(), + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_lc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_lc.py new file mode 100644 index 000000000000..88e64f35ceb4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_lc.py @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 long-context sparse decode attention in one launch. + +Replaces the generic chain's split-K producer (``_gqa_sparse_decode_kernel``, one block per chunk) and the LSE +merge (``_merge_topk_attn_out_kernel``, with its MXFP8 copy for the o_proj) once the top-k block table exists. + +Grid: one cluster of 16 CTAs per query token; CTA ``r`` attends the token's ``r``-th selected block (the Triton +producer's chunk ``r`` at 16 chunks), keeps the partial in shared memory, and after a cluster barrier merges head +``r`` from all 16 partials over DSMEM, writing the BF16 row and, optionally, its MXFP8 copy and swizzled UE8M0 +scales exactly as ``_store_mxfp8_head`` lays them out. + +Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and +PV on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +log-domain recurrence. Accumulation order inside the MMAs and the reductions differs, so results match to rounding, +not bit for bit. + +Schedule: the request length and the query rows (FP8 widened to FP16 in shared memory) are read before the PDL grid +wait, which overlaps them with the select ahead when it releases this grid early (modeling's +``SELECT_EARLY_MAX_ROWS``); +the split-K partials travel by ``st.async`` into each receiving CTA's own mbarrier instead of a cluster-wide barrier. +Neither changes a value or the merge order. + +Geometry: 16 query heads per KV head (TP4), one KV head, head_dim 128, page 128, top-k 16. The kernel is +``torch.ops.trtllm.minimax_m3_sparse_decode_lc``, called through this target's ``m3_sparse_decode_lc`` entry. +""" + +import torch + +from ..ops.attention.m3_sparse_decode_lc import m3_sparse_decode_lc + + +def lc_applies(q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output) -> bool: + """Whether the step's operands fit the kernel: TP4 geometry, packed NVFP4 pages, FP8 q, BF16 [T, 16, 128] out.""" + if q.dim() != 3 or tuple(q.shape[1:]) != (16, 128) or q.dtype != torch.float8_e4m3fn: + return False + if q.stride(2) != 1 or q.stride(1) % 16 or q.stride(0) % 16 or q.data_ptr() % 16: + return False + if int(k_paged.shape[1]) != 1 or int(topk_idx.shape[0]) != 1 or int(topk_idx.shape[-1]) != 16: + return False + for t in (k_paged, v_paged): + if ( + t.stride(3) != 1 + or t.stride(2) != 64 + or int(t.shape[3]) != 64 + or t.data_ptr() % 16 + or t.stride(0) % 16 + ): + return False + for t in (k_block_scale, v_block_scale): + if t is None or t.stride(3) != 1 or t.stride(2) != 8 or t.data_ptr() % 8 or t.stride(0) % 8: + return False + return ( + output.dtype == torch.bfloat16 + and output.dim() == 3 + and output.is_contiguous() + and tuple(output.shape) == tuple(q.shape) + ) + + +def m3_lc_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + linear_softmax=True, + mxfp8=None, + use_pdl=True, + early_trigger=True, +): + """Drop-in for ``minimax_m3_sparse_attn_decode`` on the TP4 NVFP4 shape; callers check ``lc_applies`` first. + + ``topk_idx`` is the head-major [1, total_q, 16] view the Triton producer takes. ``mxfp8`` is ``(q8, sf)`` from + ``MERGE_MXFP8.buffers(2048, dev)`` to also emit the o_proj hand-off. The op is built with the early PDL + trigger only (``early_trigger=True``). + """ + total_q = int(q.shape[0]) + q8, sf = ( + (mxfp8[0].view(torch.uint8), mxfp8[1].view(torch.uint8)) + if mxfp8 is not None + else (None, None) + ) + m3_sparse_decode_lc( + q.view(torch.uint8), + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx[0], + block_table, + seq_lens, + output.view(total_q, 2048), + q8, + sf, + 2048 // 128, + int(decode_query_len), + float(sm_scale), + bool(linear_softmax), + bool(use_pdl), + bool(early_trigger), + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_mega.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_mega.py new file mode 100644 index 000000000000..964a138afd67 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_mega.py @@ -0,0 +1,206 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 attention megakernel: ``m3_attn_mega``, one launch for a one-request, one-token decode layer. + +The QKV GEMV (bit-identical to ``m3_mx_gemv``: split 4 with contiguous k-tiles, one accumulator, the MXFP8 +activation as produced by the all-reduce, the split-K partials reduced in the owner CTA over DSMEM in rank order +and rounded to bf16), the fused decode attention and the o_proj GEMV run in one persistent grid, joined by per-tile +ready counters instead of a PDL boundary. + +The kernel is ``torch.ops.trtllm.minimax_m3_attn_mega`` (``cpp/tensorrt_llm/kernels/minimaxM3/attnMega.cu``), +compiled for geometry (1, 1, 1) with the MXFP8 hand-off only. + +Grid: 7 clusters of 16 CTAs. The first attention cluster(s) run the attention; the last 80 CTAs are the QKV units +(row tile u / 4, split u % 4; the four splits of a tile are consecutive CTAs of one cluster). Each unit's owner CTA +of a 32-row slice stores it to ``packed_out`` and release-increments the tile's counter; every attention CTA waits +for 4 more increments on all 20 tiles than at its last call (a per-CTA target it alone keeps), then reads the rows. + +One kernel replaces the per-layer chain that follows the packed QKV GEMM in short-sequence decode mode (every +request attends at most 16 KV blocks of 128 tokens, so block selection is the identity): + +* q/k/index-k RMSNorm + NeoX partial RoPE, NVFP4 K/V cache insert and FP8 index-K insert + (``minimaxM3Nvfp4QKVIndexerNormRopeKVInsertKernel``), +* block-sparse GQA decode over the 16 blocks (``_gqa_sparse_decode_kernel``), +* the split-K LSE merge, optionally with the MXFP8 copy of the output (``_merge_topk_attn_out_kernel``). + +Geometry: 16 query heads, one KV head, one index head, head_dim 128, rotary_dim 64, page 128. + +Launch: clusters of ``16 / BPC`` CTAs; CTA ``r`` owns KV blocks ``[r * BPC, r * BPC + BPC)`` for ``QG`` query +tokens of one request (``decode_query_len / QG`` clusters per request). Each block keeps its own softmax partial, +exactly the Triton producer's chunk, so the merge sees the same 16 partials. + +* Before ``griddepcontrol.wait``: request metadata, RoPE rows, norm weights, the packed K chunks (registers) and the + packed V tile (``cp.async``). These are staged before the CUDA graph or written by earlier steps; nothing the QKV + GEMM writes is read, and nothing is written, before the wait. +* After the wait: K is dequantized to fp16 (exact) while the packed-QKV loads are in flight; every CTA + normalizes/ropes the query rows it needs (recomputed rather than exchanged); the CTA whose block holds a new + token's position also normalizes, ropes and quantizes that token's K, V and index-K exactly as the C++ producer + does, and writes the dequantized row into its fp16 tiles. The PDL dependents are released once every K/V byte + has been read: an earlier release lets their weight streams delay the KV reads. +* S^T = K Q^T and O^T = V^T P^T run on ``tcgen05.mma`` (FP16 operands from shared memory, FP32 accumulators in + TMEM). The softmax runs in the TMEM-reading warps with the producer's per-block math (block max, exp2, fp16 P, + log-domain LSE), so each partial equals the Triton producer's chunk partial up to accumulation order. The + producer warps dequantize V meanwhile. +* Partials go from registers to the merging CTA with ``st.async`` + mbarrier transaction counts (DSMEM); CTA ``r`` + merges its heads with the merge kernel's LSE weighting and writes the BF16 output (and the MXFP8 hand-off). + +CUDA C++ rather than Triton or CuTe DSL: the design needs cluster DSMEM pushes, loads issued before the PDL wait, +tcgen05/TMEM, and bit-exact reuse of the C++ producer's norm/RoPE/NVFP4 arithmetic. +""" + +from typing import Optional + +import torch + +from ..ops.attention.m3_attn_mega import m3_attn_mega as _m3_attn_mega_op + +NUM_HEADS = 16 +HEAD_DIM = 128 +PAGE_SIZE = 128 +MAX_BLOCKS = 16 # capacity: fixed: the MSA top-16 blocks +PACKED_WIDTH = (NUM_HEADS + 4) * HEAD_DIM # capacity: fixed: the tp4 packed QKV width + +_module = None + + +def _load_module(): + """The compiled ops' namespace (``torch.ops.trtllm``), resolved once; ``_module`` stays None until then.""" + global _module + if _module is None: + if not hasattr(torch.ops.trtllm, "minimax_m3_attn_mega"): + raise RuntimeError( + "torch.ops.trtllm.minimax_m3_attn_mega is not registered: this TensorRT-LLM build lacks " + "cpp/tensorrt_llm/kernels/minimaxM3/attnMega.cu" + ) + _module = torch.ops.trtllm + return _module + + +def _check(cond: bool, msg: str) -> None: + if not cond: + raise ValueError(f"m3_fused_decode_attn: {msg}") + + +class MegaWorkspace: + """Per-layer persistent state of the megakernel (allocate before CUDA-graph capture; nothing is ever reset): the + two call-parity buffers of the packed QKV rows and of the attention's MXFP8 hand-off (marker-filled), and + ``flags[32 + CTA]``, each CTA's call counter (its low bit selects the parity).""" + + def __init__(self, device, max_tokens: int = 8): + # Two call-parity buffers of the packed QKV rows, all set to the 0xFFFF "not yet" marker. + self.qkv_out = torch.full( + (2, max_tokens, PACKED_WIDTH), -1, dtype=torch.int16, device=device + ).view(torch.bfloat16) + self.flags = torch.zeros(256, dtype=torch.int32, device=device) + # Two call-parity buffers of the attention's MXFP8 hand-off to the o_proj units, all 0xFF ("not yet"). + self.q8 = torch.full( + (2 * max_tokens * 2048,), 255, dtype=torch.uint8, device=device + ) # capacity: fixed: a workspace of the kernel's fixed geometry + self.sf = torch.full( + (2 * 16 * 512,), 255, dtype=torch.uint8, device=device + ) # capacity: fixed: a workspace of the kernel's fixed geometry + self.calls = 0 # host mirror of the call count (eager use only; graph replays advance the device copy) + + +def m3_attn_mega( + xq: torch.Tensor, + xsf: torch.Tensor, + qkv_weight: torch.Tensor, + qkv_weight_scale: torch.Tensor, + ws: MegaWorkspace, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + o_weight: torch.Tensor, + o_weight_scale: torch.Tensor, + *, + eps: float, + sm_scale: float, + decode_query_len: int, + output: torch.Tensor, + y: torch.Tensor, + use_pdl: bool = True, + ar: Optional[tuple] = None, +) -> torch.Tensor: + """QKV projection (bit-identical to m3_mx_gemv) of the MXFP8 layer input ``xq`` / ``xsf``, the fused decode + attention (bit-identical to m3_fused_decode_attn) and the o_proj projection (bit-identical to m3_mx_gemv) into + ``y`` [T, 6144], in one launch. + ``output`` receives the attention rows (bf16). ``ar`` = m3_mx_gemv's (multicast workspace address, flags address, + rank, world, bucket tokens, mode 2): also push y into the next m3_mc_allreduce's exchange. Returns the packed QKV + rows of this call.""" + num_tokens = int(xq.shape[0]) + num_reqs = int(seq_lens.shape[0]) + _check( + num_tokens == num_reqs * decode_query_len and num_tokens <= ws.qkv_out.shape[0], + "token count", + ) + _check( + num_reqs == 1 and decode_query_len == 1, + "the megakernel runs one request with one query token (one " + "attention cluster); use the m3_mx_gemv + m3_fused_decode_attn + m3_mx_gemv path otherwise", + ) + _check( + xq.dtype == torch.float8_e4m3fn and xq.shape[1] == 6144 and xq.stride(1) == 1, + "xq must be e4m3 [T, 6144]", + ) + _check(qkv_weight.shape == (PACKED_WIDTH, 6144), "QKV weight must be [2560, 6144]") + out2d = output.view(num_tokens, NUM_HEADS * HEAD_DIM) + q8, sf = ws.q8, ws.sf + _check( + y.dtype == torch.bfloat16 and y.shape == (num_tokens, 6144) and y.is_contiguous(), + "y must be bf16 [T, 6144]", + ) + qg, bpc = 1, 1 # one 16-CTA attention cluster, one KV block per CTA + _load_module() + _m3_attn_mega_op( + xq.view(torch.uint8), + xsf.view(torch.uint8), + qkv_weight.view(torch.uint8), + qkv_weight_scale.view(torch.uint8), + ws.qkv_out, + ws.flags, + kv_data, + kv_scale, + index_k_cache.view(torch.uint8), + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + float(eps), + float(sm_scale), + int(decode_query_len), + qg, + bpc, + out2d, + q8, + sf, + bool(use_pdl), + o_weight.view(torch.uint8), + o_weight_scale.view(torch.uint8), + y, + *((ar[0], ar[1], ar[2], ar[3], ar[4], 2) if ar is not None else (0, 0, 0, 1, 8, 0)), + ) + out_rows = ws.qkv_out[ws.calls & 1, :num_tokens] + ws.calls += 1 + return out_rows + + +__all__ = ["MegaWorkspace", "m3_attn_mega"] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_mega_c1.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_mega_c1.py new file mode 100644 index 000000000000..82c3b1cc3593 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_attn_mega_c1.py @@ -0,0 +1,365 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 attention + o_proj kernel: ``m3_attn_c1``, one launch for a one-request, 6-token verify layer. + +The fused decode attention on the packed QKV rows and the o_proj GEMV run in one 96-CTA launch (6 clusters of 16): +every attention CTA is also an o_proj unit, fed by the attention's MXFP8 hand-off through the workspace (Lamport, +call parity in its flags), and the o_proj rows are pushed into the next multicast all-reduce's exchange. + +The kernel is ``torch.ops.trtllm.minimax_m3_attn_c1`` (``cpp/tensorrt_llm/kernels/minimaxM3/attnC1.cu``), compiled +for ``C1_GEOMETRIES`` only. + +One kernel replaces the per-layer chain that follows the packed QKV GEMM in short-sequence decode mode (every +request attends at most 16 KV blocks of 128 tokens, so block selection is the identity): + +* q/k/index-k RMSNorm + NeoX partial RoPE, NVFP4 K/V cache insert and FP8 index-K insert + (``minimaxM3Nvfp4QKVIndexerNormRopeKVInsertKernel``), +* block-sparse GQA decode over the 16 blocks (``_gqa_sparse_decode_kernel``), +* the split-K LSE merge, optionally with the MXFP8 copy of the output (``_merge_topk_attn_out_kernel``). + +Geometry: 16 query heads, one KV head, one index head, head_dim 128, rotary_dim 64, page 128. + +Launch: clusters of ``16 / BPC`` CTAs; CTA ``r`` owns KV blocks ``[r * BPC, r * BPC + BPC)`` for ``QG`` query +tokens of one request (``decode_query_len / QG`` clusters per request). Each block keeps its own softmax partial, +exactly the Triton producer's chunk, so the merge sees the same 16 partials. + +* Before ``griddepcontrol.wait``: request metadata, RoPE rows, norm weights, the packed K chunks (registers) and the + packed V tile (``cp.async``). These are staged before the CUDA graph or written by earlier steps; nothing the QKV + GEMM writes is read, and nothing is written, before the wait. +* After the wait: K is dequantized to fp16 (exact) while the packed-QKV loads are in flight; every CTA + normalizes/ropes the query rows it needs (recomputed rather than exchanged); the CTA whose block holds a new + token's position also normalizes, ropes and quantizes that token's K, V and index-K exactly as the C++ producer + does, and writes the dequantized row into its fp16 tiles. The PDL dependents are released once every K/V byte + has been read: an earlier release lets their weight streams delay the KV reads. +* S^T = K Q^T and O^T = V^T P^T run on ``tcgen05.mma`` (FP16 operands from shared memory, FP32 accumulators in + TMEM). The softmax runs in the TMEM-reading warps with the producer's per-block math (block max, exp2, fp16 P, + log-domain LSE), so each partial equals the Triton producer's chunk partial up to accumulation order. The + producer warps dequantize V meanwhile. +* Partials go from registers to the merging CTA with ``st.async`` + mbarrier transaction counts (DSMEM); CTA ``r`` + merges its heads with the merge kernel's LSE weighting and writes the BF16 output (and the MXFP8 hand-off). + +CUDA C++ rather than Triton or CuTe DSL: the design needs cluster DSMEM pushes, loads issued before the PDL wait, +tcgen05/TMEM, and bit-exact reuse of the C++ producer's norm/RoPE/NVFP4 arithmetic. +""" + +from typing import Optional + +import torch + +from ..ops.attention.m3_attn_c1 import m3_attn_c1 as _m3_attn_c1_op + +NUM_HEADS = 16 +HEAD_DIM = 128 # capacity: fixed: the checkpoint's head dim +PAGE_SIZE = 128 +MAX_BLOCKS = 16 # capacity: fixed: the MSA top-16 blocks +PACKED_WIDTH = (NUM_HEADS + 4) * HEAD_DIM # capacity: fixed: the tp4 packed QKV width + +# The geometries of m3_fused_attn, which default_geometry() reads (only C1_GEOMETRIES is compiled into this op). +GEOMETRIES = frozenset( + [ + (1, 1, 1), + (1, 1, 2), + (2, 1, 1), + (2, 1, 2), + (3, 1, 1), + (4, 1, 1), + (5, 1, 1), + (6, 1, 1), + (6, 1, 2), + (7, 1, 1), + (8, 1, 1), + ] +) + +_module = None + + +def _load_module(): + """The compiled ops' namespace (``torch.ops.trtllm``), resolved once; ``_module`` stays None until then.""" + global _module + if _module is None: + if not hasattr(torch.ops.trtllm, "minimax_m3_attn_c1"): + raise RuntimeError( + "torch.ops.trtllm.minimax_m3_attn_c1 is not registered: this TensorRT-LLM build lacks " + "cpp/tensorrt_llm/kernels/minimaxM3/attnC1.cu" + ) + _module = torch.ops.trtllm + return _module + + +def default_geometry(decode_query_len: int, num_requests: int) -> tuple: + """(tokens per cluster, KV blocks per CTA) for a step shape, as m3_fused_attn.default_geometry.""" + clusters = decode_query_len * num_requests + if clusters <= 6: + return (1, 1) + if (decode_query_len, 1, 2) in GEOMETRIES and clusters <= 16: + return (1, 2) + return (1, 1) + + +def _check(cond: bool, msg: str) -> None: + if not cond: + raise ValueError(f"m3_fused_decode_attn: {msg}") + + +def m3_fused_decode_attn_checks( + num_tokens, + num_reqs, + decode_query_len, + kv_data, + kv_scale, + index_k_cache, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + output, +) -> None: + """Argument checks shared by the attention entry points (the packed or fused-projection QKV source).""" + _check( + 1 <= decode_query_len <= 8, f"decode_query_len must be in [1, 8], got {decode_query_len}" + ) + _check( + num_tokens == num_reqs * decode_query_len, + "token rows must equal num_requests * decode_query_len", + ) + _check( + kv_data.dtype == torch.uint8 + and kv_data.dim() == 5 + and tuple(kv_data.shape[1:]) == (2, 1, 128, 64) + and kv_data.stride(4) == 1 + and kv_data.stride(3) == 64 + and kv_data.stride(0) % 16 == 0 + and kv_data.stride(1) % 16 == 0 + and kv_data.data_ptr() % 16 == 0, + "kv_data must be uint8 [pages, 2, 1, 128, 64] with contiguous 16-byte aligned head blocks", + ) + _check( + kv_scale.dtype == torch.uint8 + and kv_scale.dim() == 5 + and tuple(kv_scale.shape[1:]) == (2, 1, 128, 8) + and kv_scale.stride(4) == 1 + and kv_scale.stride(3) == 8 + and kv_scale.stride(0) % 16 == 0 + and kv_scale.stride(1) % 16 == 0 + and kv_scale.data_ptr() % 16 == 0 + and kv_scale.shape[0] == kv_data.shape[0], + "kv_scale must be uint8 [pages, 2, 1, 128, 8] with contiguous 16-byte aligned head blocks", + ) + _check( + index_k_cache.dtype == torch.float8_e4m3fn + and index_k_cache.dim() == 4 + and tuple(index_k_cache.shape[1:]) == (1, 128, 128) + and index_k_cache.stride(3) == 1 + and index_k_cache.stride(2) == 128 + and index_k_cache.stride(0) % 4 == 0 + and index_k_cache.shape[0] == kv_data.shape[0], + "index_k_cache must be fp8 [pages, 1, 128, 128]", + ) + for name, t in ( + ("out_cache_loc", out_cache_loc), + ("position_ids", position_ids), + ("block_table", block_table), + ("seq_lens", seq_lens), + ): + _check(t.dtype == torch.int32 and t.is_cuda, f"{name} must be CUDA int32") + _check( + out_cache_loc.is_contiguous() and out_cache_loc.numel() >= num_tokens, + "out_cache_loc too short", + ) + _check( + position_ids.is_contiguous() and position_ids.numel() == num_tokens, + "position_ids must have T entries", + ) + _check( + seq_lens.is_contiguous() + and block_table.dim() == 2 + and block_table.stride(1) == 1 + and block_table.shape[0] >= num_reqs, + "block_table must be [num_requests, max_blocks] row-major", + ) + for name, t in ( + ("kv_quant_scale", kv_quant_scale), + ("k_global_scale", k_global_scale), + ("v_global_scale", v_global_scale), + ): + _check( + t.dtype == torch.float32 and t.is_cuda and t.is_contiguous(), + f"{name} must be CUDA fp32", + ) + _check(kv_quant_scale.numel() >= 3, "kv_quant_scale must hold [Q, K, V]") + for name, t in ( + ("q_norm_weight", q_norm_weight), + ("k_norm_weight", k_norm_weight), + ("index_k_norm_weight", index_k_norm_weight), + ): + _check( + t.dtype == torch.bfloat16 + and t.numel() == HEAD_DIM + and t.is_contiguous() + and t.data_ptr() % 8 == 0, + f"{name} must be contiguous bf16 [128]", + ) + _check( + rotary_cos_sin.dtype == torch.float32 + and rotary_cos_sin.is_contiguous() + and rotary_cos_sin.dim() == 3 + and tuple(rotary_cos_sin.shape[1:]) == (2, 32), + "rotary_cos_sin must be fp32 [max_pos, 2, 32]", + ) + out2d = output.view(num_tokens, NUM_HEADS * HEAD_DIM) + _check( + output.dtype == torch.bfloat16 and out2d.stride(1) == 1 and out2d.data_ptr() % 8 == 0, + "output must be bf16 [T, 2048] with contiguous rows", + ) + + +class MegaWorkspace: + """Per-layer persistent state of the megakernel (allocate before CUDA-graph capture; nothing is ever reset): the + two call-parity buffers of the packed QKV rows and of the attention's MXFP8 hand-off (marker-filled), and + ``flags[32 + CTA]``, each CTA's call counter (its low bit selects the parity).""" + + def __init__(self, device, max_tokens: int = 8): + # Two call-parity buffers of the packed QKV rows, all set to the 0xFFFF "not yet" marker. + self.qkv_out = torch.full( + (2, max_tokens, PACKED_WIDTH), -1, dtype=torch.int16, device=device + ).view(torch.bfloat16) + self.flags = torch.zeros(256, dtype=torch.int32, device=device) + # Two call-parity buffers of the attention's MXFP8 hand-off to the o_proj units, all 0xFF ("not yet"). + self.q8 = torch.full( + (2 * max_tokens * 2048,), 255, dtype=torch.uint8, device=device + ) # capacity: fixed: a workspace of the kernel's fixed geometry + self.sf = torch.full( + (2 * 16 * 512,), 255, dtype=torch.uint8, device=device + ) # capacity: fixed: a workspace of the kernel's fixed geometry + self.calls = 0 # host mirror of the call count (eager use only; graph replays advance the device copy) + + +C1_GEOMETRIES = frozenset([(6, 1, 1)]) + + +def c1_applies(decode_query_len: int, num_requests: int) -> bool: + """Whether the step shape has a c1 kernel: 96 attention CTAs (6 clusters of 16), one o_proj unit each.""" + qg, bpc = default_geometry(decode_query_len, num_requests) + return ( + num_requests * (decode_query_len // qg) * (16 // bpc) == 96 + and (decode_query_len, qg, bpc) in C1_GEOMETRIES + ) + + +def m3_attn_c1( + packed: torch.Tensor, + ws: MegaWorkspace, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + o_weight: torch.Tensor, + o_weight_scale: torch.Tensor, + *, + eps: float, + sm_scale: float, + decode_query_len: int, + output: torch.Tensor, + y: torch.Tensor, + use_pdl: bool = True, + ar: Optional[tuple] = None, +) -> None: + """The fused decode attention on the packed QKV rows (bit-identical to m3_fused_decode_attn) and the o_proj + projection (bit-identical to m3_mx_gemv) into ``y`` [T, 6144], in one 96-CTA launch: every attention CTA is also + an o_proj unit, fed by the attention's MXFP8 hand-off through ``ws`` (Lamport, call parity in ``ws.flags``). + ``ar`` = m3_mx_gemv's (multicast workspace address, flags address, rank, world, bucket tokens): also push y into + the next m3_mc_allreduce's exchange.""" + num_tokens = int(packed.shape[0]) + num_reqs = int(seq_lens.shape[0]) + _check( + packed.dtype == torch.bfloat16 + and packed.dim() == 2 + and packed.shape[1] == PACKED_WIDTH + and packed.stride(1) == 1 + and packed.stride(0) % 4 == 0 + and packed.data_ptr() % 8 == 0, + "packed must be bf16 [T, 2560] with 8-byte aligned contiguous rows", + ) + m3_fused_decode_attn_checks( + num_tokens, + num_reqs, + decode_query_len, + kv_data, + kv_scale, + index_k_cache, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + output, + ) + qg, bpc = default_geometry(decode_query_len, num_reqs) + _check( + c1_applies(decode_query_len, num_reqs), f"no c1 kernel for {num_reqs} x {decode_query_len}" + ) + _check( + y.dtype == torch.bfloat16 and y.shape == (num_tokens, 6144) and y.is_contiguous(), + "y must be bf16 [T, 6144]", + ) + out2d = output.view(num_tokens, NUM_HEADS * HEAD_DIM) + _load_module() + _m3_attn_c1_op( + packed, + ws.flags, + kv_data, + kv_scale, + index_k_cache.view(torch.uint8), + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + float(eps), + float(sm_scale), + int(decode_query_len), + qg, + bpc, + out2d, + ws.q8, + ws.sf, + bool(use_pdl), + o_weight.view(torch.uint8), + o_weight_scale.view(torch.uint8), + y, + *((ar[0], ar[1], ar[2], ar[3], ar[4], 2) if ar is not None else (0, 0, 0, 1, 8, 0)), + ) + ws.calls += 1 + + +__all__ = ["MegaWorkspace", "c1_applies", "m3_attn_c1"] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_bp_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_bp_tail.py new file mode 100644 index 000000000000..419a5baef898 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_bp_tail.py @@ -0,0 +1,192 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""BOUNDARY_PACK BP-1's graph tail for a batch-1 decode step, in one launch: the spec sampler's four slot stores and +the engine's next-step gathers, both gated by a device-side live word. + +``ctl`` (int64 [2]) holds (slot, live), written by the step fold's ``begin_step`` before the replay. With ``live`` 0 +the launch writes nothing (a step the fold did not take: a dummy-only replay, a step it could not identify). With +``live`` 1, for the one row: + +* the stores, as the sampler's ``_store_outputs`` (zero padding / truncation to the store widths, then index_copy_ + at ``slot``):: + + new_tokens[slot][c] = src_new[c] if c < src width else 0 (c < store width), likewise + next_new_tokens[slot][c] , next_draft_tokens[slot][c]; new_tokens_lens[slot] = src_lens + +* with ``GATHER``, the engine's gathers for the next step of the same row (P = 1, no rows before it), as + ``step_gather`` reads them back from the store just written:: + + ids[t] = next_new_tokens[slot][t] (t < T) draft[d] = next_draft_tokens[slot][d] (d < T - 1) + pos_off[i] = lens[slot] if i < T else 0 kv_off[i] = lens[slot] - T if i < 1 else 0 + (every i of pos_off / kv_off: the engine zeroes both buffers before its gathers) + +Integer copies only. Program 0 does the row; every program zero-fills its ``ZBLOCK`` chunk of the offsets. The int +arguments are not specialized and every pointer is 16-byte aligned (the caller checks), so the two ``GATHER`` +variants compiled by ``warm_up`` outside capture serve every graph. +""" + +import torch +import triton +import triton.language as tl + +ZBLOCK = 1024 +_INTS = ( + "s_new_s1", + "s_new_w", + "d_new_s0", + "d_new_s1", + "d_new_w", + "s_nnew_s1", + "s_nnew_w", + "d_nnew_s0", + "d_nnew_s1", + "d_nnew_w", + "d_lens_s0", + "s_draft_s1", + "s_draft_w", + "d_draft_s0", + "d_draft_s1", + "d_draft_w", + "n_pos", + "n_kv", + "T", +) + + +@triton.jit(do_not_specialize=list(_INTS)) +def _bp_tail_kernel( + ctl, + s_new, + s_new_s1, + s_new_w, + d_new, + d_new_s0, + d_new_s1, + d_new_w, + s_nnew, + s_nnew_s1, + s_nnew_w, + d_nnew, + d_nnew_s0, + d_nnew_s1, + d_nnew_w, + s_lens, + d_lens, + d_lens_s0, + s_draft, + s_draft_s1, + s_draft_w, + d_draft, + d_draft_s0, + d_draft_s1, + d_draft_w, + ids, + draft, + pos_off, + n_pos, + kv_off, + n_kv, + T, + GATHER: tl.constexpr, + BLOCK: tl.constexpr, + ZBLOCK_: tl.constexpr, +): + slot = tl.load(ctl) + live = tl.load(ctl + 1) + if live != 0: + pid = tl.program_id(0) + lens = tl.load(s_lens) + if pid == 0: + c = tl.arange(0, BLOCK) + new = tl.load(s_new + c * s_new_s1, mask=c < s_new_w, other=0) + tl.store(d_new + slot * d_new_s0 + c * d_new_s1, new, mask=c < d_new_w) + nnew = tl.load(s_nnew + c * s_nnew_s1, mask=c < s_nnew_w, other=0) + tl.store(d_nnew + slot * d_nnew_s0 + c * d_nnew_s1, nnew, mask=c < d_nnew_w) + dr = tl.load(s_draft + c * s_draft_s1, mask=c < s_draft_w, other=0) + tl.store(d_draft + slot * d_draft_s0 + c * d_draft_s1, dr, mask=c < d_draft_w) + tl.store(d_lens + slot * d_lens_s0, lens) + if GATHER: + tl.store(ids + c, nnew, mask=c < T) + tl.store(draft + c, dr, mask=c < T - 1) + if GATHER: + z = pid * ZBLOCK_ + tl.arange(0, ZBLOCK_) + tl.store(pos_off + z, tl.where(z < T, lens, 0), mask=z < n_pos) + tl.store(kv_off + z, tl.where(z < 1, lens - T, 0), mask=z < n_kv) + + +def block_for(store) -> int: + """The row block of a sampler store: every store width fits it (fixed per store, so per fold).""" + widths = ( + store.new_tokens.shape[0], + store.next_new_tokens.shape[0], + store.next_draft_tokens.shape[1], + ) + return triton.next_power_of_2(max(*widths, 1)) + + +def aligned(*tensors: torch.Tensor) -> bool: + """Every pointer the launch takes is 16-byte aligned (the compiled variants' assumption).""" + return all(t.data_ptr() % 16 == 0 for t in tensors) + + +def bp_tail(ctl, src, store, engine, t: int, gather: bool, block: int) -> None: + """One launch: ``src`` = (new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens) row 0 of the worker's + int32 outputs ([1, w] / [1]); ``store`` the sampler's store; ``engine`` = (input_ids, draft_tokens, + position_id_offsets, kv_lens_offsets) int32 contiguous; ``t`` tokens per request (gather only).""" + s_new, s_nnew, s_lens, s_draft = src + d_new = store.new_tokens.squeeze(-1).T + d_nnew = store.next_new_tokens.squeeze(-1).T + d_lens, d_draft = store.new_tokens_lens, store.next_draft_tokens + ids, draft, pos_off, kv_off = engine + n_pos, n_kv = pos_off.numel(), kv_off.numel() + grid = (max(1, triton.cdiv(max(n_pos, n_kv), ZBLOCK)) if gather else 1,) + _bp_tail_kernel[grid]( + ctl, + s_new, + s_new.stride(1), + s_new.shape[1], + d_new, + d_new.stride(0), + d_new.stride(1), + d_new.shape[1], + s_nnew, + s_nnew.stride(1), + s_nnew.shape[1], + d_nnew, + d_nnew.stride(0), + d_nnew.stride(1), + d_nnew.shape[1], + s_lens, + d_lens, + d_lens.stride(0), + s_draft, + s_draft.stride(1), + s_draft.shape[1], + d_draft, + d_draft.stride(0), + d_draft.stride(1), + d_draft.shape[1], + ids, + draft, + pos_off, + n_pos, + kv_off, + n_kv, + t, + GATHER=gather, + BLOCK=block, + ZBLOCK_=ZBLOCK, + num_warps=4, + ) + + +def warm_up(ctl, store, engine, block: int) -> None: + """Compile both variants outside capture, with ``ctl`` live 0 (the launch writes nothing).""" + src = ( + store.new_tokens.squeeze(-1).T[:1], + store.next_new_tokens.squeeze(-1).T[:1], + store.new_tokens_lens[:1], + store.next_draft_tokens[:1], + ) + for gather in (False, True): + bp_tail(ctl, src, store, engine, 2, gather, block) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_draft_step.py new file mode 100644 index 000000000000..31768ebcd2ee --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_draft_step.py @@ -0,0 +1,178 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 draft pass for the MiniMax-M3 drafter's vocab-sharded BF16 head (TP decode, <= 8 requests). + +One kernel replaces a draft pass's head and the draft loop's per-pass bookkeeping, at the in-tree precision: + +* The drafter's BF16 lm_head shard is kept a second time, built by ``prepare_head`` before the executor's warmup: in + the lossless 12-bit coding of the target head (``ops.gemm.m3_target_head``; 75% of the bytes, kept only when it + decodes to the BF16 shard bit for bit), else in ``mma.m16n8k16`` A-fragment order (``bf16_fragments``, the same + values). The kernel streams it with BF16 MMAs and FP32 accumulation over the BF16 head input rows; a coded + exception decodes to +0 and its exact product x * w is added at its tile's flush. +* Each logit is rounded to BF16 (the in-tree logits are the BF16 head GEMM's output) and keyed + ``ordered(value) << 32 | ~global_index`` with -0 as +0, so the larger value wins and ties take the lowest index, as + the in-tree ``torch.max`` / ``torch.argmax`` over the gathered logits (``greedy_sample_draft_with_tp_gather``). +* The last CTA exchanges each row's key across the TP ranks through a multicast mapping (Lamport slots, see + ``m3_shard_accept``), then stores the draft token, the next pass's input id and position, and advances the draft KV + lengths as the in-tree draft loop does between passes. + +The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/draftStep.cu) and called through the +catalog entry ``ops.spec.m3_draft_step``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +import torch + +from ..ops.gemm.m3_target_head import encode_chunked, round_trip_ok +from ..ops.spec.m3_draft_step import MAX_TILES_PER_CTA, MAX_TOKENS, bf16_fragments +from ..ops.spec.m3_draft_step import m3_draft_step as _draft_step_op +from .m3_shard_accept import AcceptWorkspace, create_workspace + + +@dataclass(eq=False) +class DraftHead: + """The streamed copy of one rank's draft lm_head shard and its first global id: ``wf`` is the 12-bit coded stream + with its exception lists (``coded``), or the fragment-ordered BF16 copy (``bf16_fragments``).""" + + weight: torch.Tensor + wf: torch.Tensor + vocab_offset: int + coded: bool = False + exc_off: Optional[torch.Tensor] = None + exc_pos: Optional[torch.Tensor] = None + exc_w: Optional[torch.Tensor] = None + exp_base: int = 0 + + +def head_supported(lm_head) -> bool: + """A bias-free BF16 head whose local shard the kernel can own: whole 16-row tiles, K a multiple of 16, at most + MAX_TILES_PER_CTA tiles per CTA of a one-CTA-per-SM grid.""" + weight = getattr(lm_head, "weight", None) + if weight is None or weight.dim() != 2 or weight.dtype != torch.bfloat16: + return False + if getattr(lm_head, "bias", None) is not None: + return False + n, k = weight.shape + if n % 16 or k % 16: + return False + return (n // 16 + _num_sms() - 1) // _num_sms() <= MAX_TILES_PER_CTA + + +def prepare_head(lm_head, tp_rank: int, coded: bool) -> DraftHead: + """The streamed copy of ``lm_head``'s local shard (global ids from tp_rank * rows): its 12-bit coding when + ``coded``, K is a multiple of 64 and the coding decodes to the shard bit for bit, else (fail + closed) the fragment-ordered BF16 copy. + Built once, eagerly, before the executor's warmup (``M3Eagle3Worker.prebuild_draft_head``).""" + weight = lm_head.weight + offset = tp_rank * weight.shape[0] + with torch.no_grad(): + w = weight.detach() + if coded and w.shape[1] % 64 == 0: + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + if round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w): + return DraftHead( + weight=weight, + wf=packed, + vocab_offset=offset, + coded=True, + exc_off=exc_off, + exc_pos=exc_pos, + exc_w=exc_w.view(-1), + exp_base=base, + ) + del packed, exc_off, exc_pos, exc_w + wf = bf16_fragments(w) + return DraftHead(weight=weight, wf=wf, vocab_offset=offset) + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace ([2][MAX_TOKENS][world] slots), created collectively on the first eager call; None + during a CUDA-graph capture that finds it missing.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = create_workspace(mapping, device, MAX_TOKENS) + return _WS + + +def draft_step( + head: DraftHead, + x: torch.Tensor, + rows: int, + x_rows: Optional[torch.Tensor], + mapping, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, +) -> bool: + """One fused greedy draft pass over ``rows`` requests (see the module docstring); False when the exchange + workspace is missing during a capture. Collective over the TP group. + + Args: + head: the prepared head of this rank (``prepare_head``), coded or BF16 fragments. + x: [*, K] BF16 head input rows (the drafter's final-norm output); row ``x_rows[b]`` (int64) or ``b`` is + request b's. + draft_out: int32 [rows, >= step + 1], unit column stride: column ``step`` receives the tokens. + ids_next / pos_next: int32 [rows]: the tokens, and ``pos_src[x_rows[b] or b] + 1`` (pos_src int32). + kv_lens: int32 draft KV lengths: ``-= draft_len - num_acc`` with ``seq_lens = 1`` on pass 0 when ``num_acc`` + is given, ``+= 1`` on later passes. + """ + ws = workspace(mapping, x.device) + if ws is None: + return False + _draft_step_op( + x, + x_rows, + rows, + head.wf, + head.vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + _num_sms(), + True, + None, + head.exc_off, + head.exc_pos, + head.exc_w, + head.exp_base, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_force_vec.py new file mode 100644 index 000000000000..b4de37794ca9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_force_vec.py @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance (the in-tree speculative worker's +``force_num_accepted_tokens``), drawn exactly as its ``_apply_force_accepted_tokens`` does, in one launch. + +Triton: one CTA bumps the shared step counter and draws each request's count from the fixed-seed pool with the +in-tree (counter, slot) hash. Reached through the ``ops/spec`` entry ``m3_force_vec``. +""" + +import triton +import triton.language as tl + + +@triton.jit +def force_vec_kernel( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK: tl.constexpr, +): + c = tl.load(counter) + 1 + slots = tl.arange(0, BLOCK) + tl.store(counter, c) + idx = (c * counter_stride + slots.to(tl.int64) * slot_stride) & pool_mask + live = slots < batch_size + r = tl.load(pool + idx, mask=live, other=1.0) + v = tl.minimum(base_total + (r < frac).to(tl.int32), max_total) + tl.store(out + slots, v, mask=live) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_fused_attn.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_fused_attn.py new file mode 100644 index 000000000000..fefd30acf333 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_fused_attn.py @@ -0,0 +1,352 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 fused short-sequence decode attention: ``m3_fused_decode_attn``. + +The kernel is ``torch.ops.trtllm.minimax_m3_fused_decode_attn`` (``cpp/tensorrt_llm/kernels/minimaxM3/ +fusedDecodeAttn.cu``); it is compiled for the geometries in ``GEOMETRIES``, each with the MXFP8 hand-off. + +One kernel replaces the per-layer chain that follows the packed QKV GEMM in short-sequence decode mode (every +request attends at most 16 KV blocks of 128 tokens, so block selection is the identity): + +* q/k/index-k RMSNorm + NeoX partial RoPE, NVFP4 K/V cache insert and FP8 index-K insert + (``minimaxM3Nvfp4QKVIndexerNormRopeKVInsertKernel``), +* block-sparse GQA decode over the 16 blocks (``_gqa_sparse_decode_kernel``), +* the split-K LSE merge, optionally with the MXFP8 copy of the output (``_merge_topk_attn_out_kernel``). + +Geometry: 16 query heads, one KV head, one index head, head_dim 128, rotary_dim 64, page 128. + +Launch: clusters of ``16 / BPC`` CTAs; CTA ``r`` owns KV blocks ``[r * BPC, r * BPC + BPC)`` for ``QG`` query +tokens of one request (``decode_query_len / QG`` clusters per request). Each block keeps its own softmax partial, +exactly the Triton producer's chunk, so the merge sees the same 16 partials. + +* Before ``griddepcontrol.wait``: request metadata, RoPE rows, norm weights, the packed K chunks (registers) and the + packed V tile (``cp.async``). These are staged before the CUDA graph or written by earlier steps; nothing the QKV + GEMM writes is read, and nothing is written, before the wait. +* After the wait: K is dequantized to fp16 (exact) while the packed-QKV loads are in flight; every CTA + normalizes/ropes the query rows it needs (recomputed rather than exchanged); the CTA whose block holds a new + token's position also normalizes, ropes and quantizes that token's K, V and index-K exactly as the C++ producer + does, and writes the dequantized row into its fp16 tiles. The PDL dependents are released once every K/V byte + has been read: an earlier release lets their weight streams delay the KV reads. +* S^T = K Q^T and O^T = V^T P^T run on ``tcgen05.mma`` (FP16 operands from shared memory, FP32 accumulators in + TMEM). The softmax runs in the TMEM-reading warps with the producer's per-block math (block max, exp2, fp16 P, + log-domain LSE), so each partial equals the Triton producer's chunk partial up to accumulation order. The + producer warps dequantize V meanwhile. +* Partials go from registers to the merging CTA with ``st.async`` + mbarrier transaction counts (DSMEM); CTA ``r`` + merges its heads with the merge kernel's LSE weighting and writes the BF16 output (and the MXFP8 hand-off). + +CUDA C++ rather than Triton or CuTe DSL: the design needs cluster DSMEM pushes, loads issued before the PDL wait, +tcgen05/TMEM, and bit-exact reuse of the C++ producer's norm/RoPE/NVFP4 arithmetic. +""" + +from typing import Optional + +import torch + +from ..ops.attention.m3_fused_decode_attn import m3_fused_decode_attn as _m3_fused_decode_attn_op + +NUM_HEADS = 16 # capacity: fixed: the tp4 query heads +HEAD_DIM = 128 # capacity: fixed: the checkpoint's head dim +PAGE_SIZE = 128 +MAX_BLOCKS = 16 # capacity: fixed: the MSA top-16 blocks +PACKED_WIDTH = (NUM_HEADS + 4) * HEAD_DIM # capacity: fixed: the tp4 packed QKV width + +# (decode_query_len, tokens per cluster, KV blocks per CTA) geometries compiled into the op: what +# default_geometry() returns for every step shape the target routes here (at most 16 tokens). +GEOMETRIES = frozenset( + [ + (1, 1, 1), + (1, 1, 2), + (2, 1, 1), + (2, 1, 2), + (3, 1, 1), + (4, 1, 1), + (5, 1, 1), + (6, 1, 1), + (6, 1, 2), + (7, 1, 1), + (8, 1, 1), + ] +) + +_module = None + + +def _load_module(): + """The compiled ops' namespace (``torch.ops.trtllm``), resolved once; ``_module`` stays None until then.""" + global _module + if _module is None: + if not hasattr(torch.ops.trtllm, "minimax_m3_fused_decode_attn"): + raise RuntimeError( + "torch.ops.trtllm.minimax_m3_fused_decode_attn is not registered: this TensorRT-LLM " + "build lacks cpp/tensorrt_llm/kernels/minimaxM3/fusedDecodeAttn.cu" + ) + _module = torch.ops.trtllm + return _module + + +def default_geometry(decode_query_len: int, num_requests: int) -> tuple: + """(tokens per cluster, KV blocks per CTA) for a step shape: one token per 16-CTA cluster up to 6 clusters, + then KV blocks paired into 8-CTA clusters up to 16 clusters where that geometry is compiled.""" + clusters = decode_query_len * num_requests + if clusters <= 6: + return (1, 1) + if (decode_query_len, 1, 2) in GEOMETRIES and clusters <= 16: + return (1, 2) + return (1, 1) + + +def _check(cond: bool, msg: str) -> None: + if not cond: + raise ValueError(f"m3_fused_decode_attn: {msg}") + + +def m3_fused_decode_attn_checks( + num_tokens, + num_reqs, + decode_query_len, + kv_data, + kv_scale, + index_k_cache, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + output, +) -> None: + """Argument checks shared by the attention entry points (the packed or fused-projection QKV source).""" + _check( + 1 <= decode_query_len <= 8, f"decode_query_len must be in [1, 8], got {decode_query_len}" + ) + _check( + num_tokens == num_reqs * decode_query_len, + "token rows must equal num_requests * decode_query_len", + ) + _check( + kv_data.dtype == torch.uint8 + and kv_data.dim() == 5 + and tuple(kv_data.shape[1:]) == (2, 1, 128, 64) + and kv_data.stride(4) == 1 + and kv_data.stride(3) == 64 + and kv_data.stride(0) % 16 == 0 + and kv_data.stride(1) % 16 == 0 + and kv_data.data_ptr() % 16 == 0, + "kv_data must be uint8 [pages, 2, 1, 128, 64] with contiguous 16-byte aligned head blocks", + ) + _check( + kv_scale.dtype == torch.uint8 + and kv_scale.dim() == 5 + and tuple(kv_scale.shape[1:]) == (2, 1, 128, 8) + and kv_scale.stride(4) == 1 + and kv_scale.stride(3) == 8 + and kv_scale.stride(0) % 16 == 0 + and kv_scale.stride(1) % 16 == 0 + and kv_scale.data_ptr() % 16 == 0 + and kv_scale.shape[0] == kv_data.shape[0], + "kv_scale must be uint8 [pages, 2, 1, 128, 8] with contiguous 16-byte aligned head blocks", + ) + _check( + index_k_cache.dtype == torch.float8_e4m3fn + and index_k_cache.dim() == 4 + and tuple(index_k_cache.shape[1:]) == (1, 128, 128) + and index_k_cache.stride(3) == 1 + and index_k_cache.stride(2) == 128 + and index_k_cache.stride(0) % 4 == 0 + and index_k_cache.shape[0] == kv_data.shape[0], + "index_k_cache must be fp8 [pages, 1, 128, 128]", + ) + for name, t in ( + ("out_cache_loc", out_cache_loc), + ("position_ids", position_ids), + ("block_table", block_table), + ("seq_lens", seq_lens), + ): + _check(t.dtype == torch.int32 and t.is_cuda, f"{name} must be CUDA int32") + _check( + out_cache_loc.is_contiguous() and out_cache_loc.numel() >= num_tokens, + "out_cache_loc too short", + ) + _check( + position_ids.is_contiguous() and position_ids.numel() == num_tokens, + "position_ids must have T entries", + ) + _check( + seq_lens.is_contiguous() + and block_table.dim() == 2 + and block_table.stride(1) == 1 + and block_table.shape[0] >= num_reqs, + "block_table must be [num_requests, max_blocks] row-major", + ) + for name, t in ( + ("kv_quant_scale", kv_quant_scale), + ("k_global_scale", k_global_scale), + ("v_global_scale", v_global_scale), + ): + _check( + t.dtype == torch.float32 and t.is_cuda and t.is_contiguous(), + f"{name} must be CUDA fp32", + ) + _check(kv_quant_scale.numel() >= 3, "kv_quant_scale must hold [Q, K, V]") + for name, t in ( + ("q_norm_weight", q_norm_weight), + ("k_norm_weight", k_norm_weight), + ("index_k_norm_weight", index_k_norm_weight), + ): + _check( + t.dtype == torch.bfloat16 + and t.numel() == HEAD_DIM + and t.is_contiguous() + and t.data_ptr() % 8 == 0, + f"{name} must be contiguous bf16 [128]", + ) + _check( + rotary_cos_sin.dtype == torch.float32 + and rotary_cos_sin.is_contiguous() + and rotary_cos_sin.dim() == 3 + and tuple(rotary_cos_sin.shape[1:]) == (2, 32), + "rotary_cos_sin must be fp32 [max_pos, 2, 32]", + ) + out2d = output.view(num_tokens, NUM_HEADS * HEAD_DIM) + _check( + output.dtype == torch.bfloat16 and out2d.stride(1) == 1 and out2d.data_ptr() % 8 == 0, + "output must be bf16 [T, 2048] with contiguous rows", + ) + + +def m3_fused_decode_attn( + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + position_ids: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + kv_quant_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + index_k_norm_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + *, + eps: float, + sm_scale: float, + decode_query_len: int, + output: torch.Tensor, + mxfp8_out: Optional[tuple] = None, + use_pdl: bool = True, + geometry: Optional[tuple] = None, +) -> None: + """Fused norm/RoPE/KV-insert + short-sequence sparse decode + split-K merge, written into ``output``. + + packed: [T, 2560] bf16 QKV+index GEMM output, T = num_requests * decode_query_len. + kv_data: uint8 [num_pages, 2, 1, 128, 64] HND NVFP4 pool (K plane 0, V plane 1); written. + kv_scale: uint8 [num_pages, 2, 1, 128, 8] E4M3 block scales (K linear, V 4x4 token quads); written. + index_k_cache: float8_e4m3fn [num_pages, 1, 128, 128]; written. + out_cache_loc, position_ids: int32 [>= T] / [T]; block_table int32 [num_requests, >= 1]; seq_lens int32 + [num_requests] (attended length including this step's tokens, at most 16 * 128). + kv_quant_scale: fp32 [3] (Q, K, V) quantization scales; k/v_global_scale: fp32 [1] dequantization scales. + output: bf16 [T, 2048] (or [T, 16, 128], contiguous rows). + mxfp8_out: (e4m3 [>= T, 2048], uint8 swizzled UE8M0 scales) hand-off for the o_proj GEMM (required: only + the MXFP8 variants are compiled), + same layout as the merge kernel's MERGE_MXFP8 output. + geometry: (query tokens per cluster, KV blocks per CTA); default_geometry() when None. + """ + num_tokens = int(packed.shape[0]) + num_reqs = int(seq_lens.shape[0]) + _check( + 1 <= decode_query_len <= 8, f"decode_query_len must be in [1, 8], got {decode_query_len}" + ) + _check( + num_tokens == num_reqs * decode_query_len, + "packed rows must equal num_requests * decode_query_len", + ) + _check( + packed.dtype == torch.bfloat16 + and packed.dim() == 2 + and packed.shape[1] == PACKED_WIDTH + and packed.stride(1) == 1 + and packed.stride(0) % 4 == 0 + and packed.data_ptr() % 8 == 0, + "packed must be bf16 [T, 2560] with 8-byte aligned contiguous rows", + ) + m3_fused_decode_attn_checks( + num_tokens, + num_reqs, + decode_query_len, + kv_data, + kv_scale, + index_k_cache, + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + output, + ) + out2d = output.view(num_tokens, NUM_HEADS * HEAD_DIM) + _check( + mxfp8_out is not None, + "only the variants with the MXFP8 hand-off are compiled; pass mxfp8_out", + ) + q8, sf = mxfp8_out + _check( + q8.dtype == torch.float8_e4m3fn + and q8.dim() == 2 + and q8.shape[1] == NUM_HEADS * HEAD_DIM + and q8.shape[0] >= num_tokens + and q8.is_contiguous(), + "mxfp8 data must be e4m3 [>= T, 2048]", + ) + _check(sf.dtype == torch.uint8 and sf.is_contiguous(), "mxfp8 scales must be contiguous uint8") + q8 = q8.view(torch.uint8) + if num_tokens == 0: + return + qg, bpc = geometry if geometry is not None else default_geometry(decode_query_len, num_reqs) + _check( + (decode_query_len, qg, bpc) in GEOMETRIES, + f"geometry {(decode_query_len, qg, bpc)} is not compiled", + ) + _load_module() + _m3_fused_decode_attn_op( + packed, + kv_data, + kv_scale, + index_k_cache.view(torch.uint8), + out_cache_loc, + position_ids, + block_table, + seq_lens, + kv_quant_scale, + k_global_scale, + v_global_scale, + q_norm_weight, + k_norm_weight, + index_k_norm_weight, + rotary_cos_sin, + float(eps), + float(sm_scale), + int(decode_query_len), + qg, + bpc, + out2d, + q8, + sf, + bool(use_pdl), + ) + + +__all__ = ["GEOMETRIES", "default_geometry", "m3_fused_decode_attn"] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_index_score.py new file mode 100644 index 000000000000..812410b130ed --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_index_score.py @@ -0,0 +1,645 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py +"""MiniMax-M3 index decode block scorer (CuTe DSL, Blackwell) with its pre-wait schedule. + +The kernel of ``tensorrt_llm/_torch/attention/kernels/blackwell/minimax_m3_index_decode_score.py`` (the same scores, bit +for bit) with two schedule options this target compiles in: + +* ``prefetch_old``: before the grid-dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that hold only + tokens older than this step (the producer ahead writes only this step's tokens), up to ``prefetch_depth`` blocks per + CTA (0 = all of them). +* ``early_issue``: the TMA warp reads a ``seq_lens`` hint and its first page id before the wait and issues the Q tile + and the first K tile right after it. The tiles are read after the wait, so their contents are final; the block table + is only written between graph replays, so the early page id is the right one whenever the block is valid. A CTA + whose early tiles lie past the request's true length drains them before it exits. + +Every kernel is compiled before the executor's warmup (``layers.prebuild_index_scorer``) through ``IndexScoreRunner`` +and called through this target's ``m3_index_score`` entry. +""" + +import cutlass +import torch +from cuda.bindings.driver import CUstream +from cutlass import Float8E4M3FN, Float16, Float32, Int32, Int64, Uint32, cute +from cutlass._mlir.dialects import llvm +from cutlass.cute.nvgpu import cpasync, warp +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + EVICT_FIRST, + fp8x4_to_fp16x4, + mma_sync, + simple_tma_copy, +) +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + TRTLLM_ENABLE_PDL, + griddepcontrol_wait, +) + +__all__ = ["IndexDecodeScoreKernel", "IndexScoreRunner"] + + +@dsl_user_op +def _l2_prefetch_bulk(addr: Int64, size: Int32, *, loc=None, ip=None) -> None: + """Prefetch ``size`` contiguous bytes (multiple of 16) at global ``addr`` into L2.""" + llvm.inline_asm( + res=None, + operands_=[Int64(addr).ir_value(loc=loc, ip=ip), Int32(size).ir_value(loc=loc, ip=ip)], + asm_string="cp.async.bulk.prefetch.L2.global [$0], $1;", + constraints="l,r", + has_side_effects=True, + asm_dialect=llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@cute.jit +def _fp8_to_f16_mma_fragments(src: cute.Tensor): + """Split an FP8 ldmatrix fragment into the two FP16 k-fragments MMA wants.""" + src_elems = cute.size(src) + src_u32 = cute.recast_tensor(src, Uint32) + src_f16 = cute.make_rmem_tensor(src_elems, Float16) + src_f16_u32 = cute.recast_tensor(src_f16, Uint32) + # Packed conversion; faster and fewer SASS instructions than + # src.load().to(Float16). + for i in cutlass.range_constexpr(src_elems // 4): + converted = fp8x4_to_fp16x4(src_u32[i]) + src_f16_u32[i * 2] = converted[0] + src_f16_u32[i * 2 + 1] = converted[1] + lower = cute.make_rmem_tensor(src_elems // 2, Float16) + upper = cute.make_rmem_tensor(src_elems // 2, Float16) + + # FP8 ldmatrix yields four consecutive values along K. Split each group + # into the lower two and upper two for the two FP16 MMA k-fragments. + for i in cutlass.range_constexpr(src_elems // 2): + lower[i] = src_f16[(i // 2) * 4 + i % 2] + upper[i] = src_f16[(i // 2) * 4 + 2 + i % 2] + return lower, upper + + +class IndexDecodeScoreKernel: + """Per-block max index score for one decode step. + + Grid is (batch, split_k): CTA (b, s) walks KV blocks + s, s + split_k, s + 2 * split_k, ... of request b and writes each block's + score directly, so no cross-CTA reduction is needed. CTAs whose split_id + exceeds the request's block count exit immediately. + """ + + BLOCK_K = 128 + BAR_MMA = 1 + num_stages = 2 + + def __init__( + self, + dtype: type[cutlass.Numeric], + num_heads: int, + max_decode_query_len: int, + split_k: int, + head_dim: int = 128, + prefetch_old: bool = False, + early_issue: bool = False, + prefetch_depth: int = 0, + ): + # prefetch_old: before the grid dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that + # hold only tokens older than this step (the predecessor writes only this step's tokens). + self.prefetch_old = prefetch_old + # early_issue: the TMA warp reads a seq_lens hint and this CTA's first page id before the grid dependency + # wait, then issues the Q tile and the first K tile right after it, overlapping the post-wait seq_lens load. + # The tiles are read after the wait, so their contents are final; the block table is only written between + # graph replays, so the early page id is the right one whenever the block is valid. A CTA whose early tiles + # turn out to lie past the request's true length drains them before exiting. + self.early_issue = early_issue + # prefetch_depth: blocks per CTA that prefetch_old covers (0 = all of them). Prefetching every older page at + # large batches floods L2; the first few per CTA are the ones whose latency the pipeline cannot hide. + self.prefetch_depth = prefetch_depth + self.dtype = dtype + self.num_heads = num_heads + self.max_decode_query_len = max_decode_query_len + self.split_k = split_k + self.head_dim = head_dim + + def _swizzle_elems(self) -> int: + """Elements spanned by one 128-byte swizzle atom.""" + return 128 * 8 // self.dtype.width + + def _sq_layout(self): + """Composed SMEM layout for the Q tile, shared by the descriptor and the kernel.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_q = self.num_heads * self.max_decode_query_len + layout = cute.make_layout( + (self.max_decode_query_len, self.num_heads, (elems, head_dim // elems)), + stride=(elems, self.max_decode_query_len * elems, (1, block_q * elems)), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + def _sk_layout(self): + """Composed SMEM layout for the pipelined K tiles.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_k = self.BLOCK_K + layout = cute.make_layout( + (1, block_k, (elems, head_dim // elems), self.num_stages), + stride=(0, elems, (1, block_k * elems), block_k * head_dim), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + @cute.jit + def __call__( + self, + gQ: cute.Tensor, # [bs * runtime_decode_query_len, num_heads, head_dim] + gK_cache: cute.Tensor, # [num_pages, page_size, head_dim] + block_table: cute.Tensor, # [bs, max_pages] + score: cute.Tensor, # [num_heads, bs * runtime_decode_query_len, max_pages] + seq_lens: cute.Tensor, # [bs] + stream: CUstream, + ): + num_heads = self.num_heads + head_dim = self.head_dim + MAX_DQL = self.max_decode_query_len + assert num_heads * MAX_DQL <= 32 + + batch = seq_lens.shape[0] + decode_query_len = gQ.shape[0] // batch + grid = (batch, self.split_k, 1) + block = (32 * 5, 1, 1) + + tma_g2s = cpasync.CopyBulkTensorTileG2SOp() + elems = self._swizzle_elems() + + q_tma_atom, q_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gQ, (None, None, elems)), + self._sq_layout(), + (MAX_DQL, num_heads, head_dim), + ) + k_tma_atom, k_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gK_cache, (None, None, elems)), + self._sk_layout(), + (1, self.BLOCK_K, head_dim), + ) + + self.kernel( + q_tma_atom, + q_tma_tensor, + k_tma_atom, + k_tma_tensor, + gK_cache, + block_table, + score, + seq_lens, + decode_query_len, + ).launch(grid=grid, block=block, stream=stream, use_pdl=TRTLLM_ENABLE_PDL) + + @cute.kernel + def kernel( + self, + q_tma_atom: cute.CopyAtom, + q_tma_tensor: cute.Tensor, + k_tma_atom: cute.CopyAtom, + k_tma_tensor: cute.Tensor, + gK_cache: cute.Tensor, + block_table: cute.Tensor, + score: cute.Tensor, + seq_lens: cute.Tensor, + decode_query_len, + ): + batch_id, split_id, _ = cute.arch.block_idx() + _, split_k, _ = cute.arch.grid_dim() + warp_id = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane_id = cute.arch.lane_idx() + + NUM_HEADS = self.num_heads + MAX_DQL = self.max_decode_query_len + BLOCK_Q = NUM_HEADS * MAX_DQL + BLOCK_K = self.BLOCK_K + head_dim = self.head_dim + dtype = self.dtype + MMA_N = 8 + num_stages = self.num_stages + Q_TILES = cute.ceil_div(BLOCK_Q, MMA_N) + EPI_Q = Q_TILES * MMA_N + + sq_layout = self._sq_layout() + sk_layout = self._sk_layout() + + smem = SmemAllocator() + sK = smem.allocate_tensor( + dtype, + sk_layout.outer, + byte_alignment=128, + swizzle=sk_layout.inner, + )[0, None, None, None] + # sQ aliases the first K stage: Q is consumed into registers before the + # first K tile is needed, so the two never overlap in time. + sQ_tma = cute.make_tensor(sK[None, None, 0].iterator, layout=sq_layout.outer) + # TMA sees Q as (query, head, dim) while ldmatrix consumes a flattened Q + # column mode. The target profile keeps the rank-2 view even for + # degenerate shapes such as MAX_DQL == 1. + q_tma_elems = self._swizzle_elems() + sQ = cute.coalesce( + cute.group_modes(sQ_tma, 0, 2), + target_profile=(BLOCK_Q, (q_tma_elems, head_dim // q_tma_elems)), + ) + epi_buffer = smem.allocate_tensor(Float32, cute.make_layout((EPI_Q, 4))) + + tma_full_mbar = smem.allocate_array(Int64, num_stages) + tma_empty_mbar = smem.allocate_array(Int64, num_stages) + + # Barrier init and descriptor prefetch do not depend on the predecessor + # grid, so they are the work PDL can overlap. Every CTA runs them, even + # the ones that own no blocks, since ownership is not known until + # seq_lens is read below. + if warp_id == 0: + with cute.arch.elect_one(): + for i in cutlass.range_constexpr(num_stages): + cute.arch.mbarrier_init(tma_full_mbar + i, 1) + cute.arch.mbarrier_init(tma_empty_mbar + i, 128) + cute.arch.mbarrier_init_fence() + elif warp_id == 1: + cpasync.prefetch_descriptor(q_tma_atom) + cpasync.prefetch_descriptor(k_tma_atom) + cute.arch.sync_threads() + + if cutlass.const_expr(self.prefetch_old): + if warp_id == 4: + with cute.arch.elect_one(): + seqlen0 = seq_lens[batch_id] + num_blocks0 = cute.ceil_div(seqlen0, BLOCK_K) + stable = (seqlen0 - decode_query_len) // BLOCK_K + if stable > num_blocks0: + stable = num_blocks0 + if cutlass.const_expr(self.prefetch_depth > 0): + if stable > split_id + self.prefetch_depth * split_k: + stable = split_id + self.prefetch_depth * split_k + for block_id in range(split_id, stable, split_k): + page = block_table[batch_id, block_id] + _l2_prefetch_bulk( + gK_cache[page, None, None].iterator.toint().to(Int64), + Int32(BLOCK_K * head_dim * (dtype.width // 8)), + ) + + # No explicit launch_dependents: the only consumer is the block selector, + # which reduces over every block this grid writes, so triggering it early + # would leave it resident and spinning in its own wait. The implicit + # trigger as the CTAs exit still gives it the launch overlap. + early_q = False + early_page = Int32(0) + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + hint_blocks = cute.ceil_div(seq_lens[batch_id], BLOCK_K) + if hint_blocks > block_table.shape[1]: + hint_blocks = block_table.shape[1] + if split_id < hint_blocks: + early_q = True + early_page = block_table[batch_id, split_id] + + griddepcontrol_wait() + + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + gQ_early = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar, BLOCK_Q * head_dim * (dtype.width // 8) + ) + simple_tma_copy(q_tma_atom, gQ_early, sQ_tma, tma_full_mbar) + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, BLOCK_K * head_dim * (dtype.width // 8) + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + # seq_lens must be read after the wait. It is a graph-stable buffer that + # on_update_kv_lens patches every step, and num_blocks bounds an unmasked + # block_table load and an unmasked score store, so a stale length is an + # out-of-bounds write rather than just a wrong answer. + seqlen = seq_lens[batch_id] + num_blocks = cute.ceil_div(seqlen, BLOCK_K) + + if split_id < num_blocks: + if warp_id == 4: + # TMA warp + tma_stage = 0 + tma_parity = 1 + first_block = split_id + + if not early_q: + gQ_tile = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, tma_parity) + with cute.arch.elect_one(): + Q_size = BLOCK_Q * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(tma_full_mbar, Q_size) + # TMA bounds-checks rows when the runtime decode_query_len is + # below MAX_DQL; padded Q columns are masked before the stores. + simple_tma_copy(q_tma_atom, gQ_tile, sQ_tma, tma_full_mbar) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + if early_q: + # The first K tile was issued right after the wait. + first_block = split_id + split_k + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + for block_id in range(first_block, num_blocks, split_k): + page_id = block_table[batch_id, block_id] + gK_tile = k_tma_tensor[page_id, None, None] + k_mbar = tma_full_mbar + tma_stage + + cute.arch.mbarrier_wait(tma_empty_mbar + tma_stage, tma_parity) + with cute.arch.elect_one(): + K_size = BLOCK_K * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(k_mbar, K_size) + simple_tma_copy( + k_tma_atom, + gK_tile, + sK[None, None, tma_stage], + k_mbar, + cache_policy=EVICT_FIRST, + ) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + else: + # MMA warps; each handles K[32, head_dim] @ Q[BLOCK_Q, head_dim].T + sK_warp = cute.local_tile(sK, (32, head_dim, num_stages), (warp_id, 0, 0)) + q_start = seqlen - decode_query_len + + elems = 128 // dtype.width # 16B + MMA_K = 32 * 8 // dtype.width # 32B + + # Pre-compute ldmatrix addresses. + # sK loads a [16 x 16B] tile: + # ((16, (16B, 2), 1), (32 / 16, head_dim / 32B, num_stages)) + # sQ loads an [8 x 32B] tile: + # ((8, (16B, 4)), (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = cute.zipped_divide(sK_warp, (16, cute.make_layout((elems, 2)), 1)) + sQ_ldsm = cute.zipped_divide(sQ, (MMA_N, cute.make_layout((elems, 4)))) + + # sK: (16B, (32 / 16, head_dim / 32B, num_stages)) + # sQ: (16B, (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = sK_ldsm[(lane_id % 16, (None, lane_id // 16), 0), None] + sQ_ldsm = sQ_ldsm[(lane_id % MMA_N, (None, lane_id // 8)), None] + + ldsm_op = warp.LdMatrix8x8x16bOp(num_matrices=4) + ldsm_atom = cute.make_copy_atom(ldsm_op, dtype) + + rQ = cute.make_rmem_tensor( + ((elems // 2, 2), head_dim // (MMA_K * 2), Q_TILES), dtype + ) + rK = cute.make_rmem_tensor((elems, 2, head_dim // MMA_K), dtype) + rC = cute.make_rmem_tensor((4, 2, Q_TILES), Float32) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + for q in cutlass.range_constexpr(Q_TILES): + cute.copy(ldsm_atom, sQ_ldsm[None, (q, None)], rQ[None, None, q]) + cute.arch.mbarrier_arrive(tma_empty_mbar) + + tma_stage = 1 % self.num_stages + tma_parity = 0 + if tma_stage == 0: + tma_parity ^= 1 + + if cutlass.const_expr(dtype is Float8E4M3FN): + rQ_f16 = cute.make_rmem_tensor((4, head_dim // MMA_K, Q_TILES, 2), Float16) + q_lower, q_upper = _fp8_to_f16_mma_fragments(rQ) + rQ_f16[None, None, None, 0].store(q_lower.load()) + rQ_f16[None, None, None, 1].store(q_upper.load()) + + for block_id in range(split_id, num_blocks, split_k): + rC.fill(0.0) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar + tma_stage, tma_parity) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + for k in cutlass.range_constexpr(head_dim // MMA_K): + cute.copy( + ldsm_atom, + sK_ldsm[None, (None, k, tma_stage)], + rK[None, None, k], + ) + for m in cutlass.range_constexpr(2): + if cutlass.const_expr(dtype is Float8E4M3FN): + rK_lower, rK_upper = _fp8_to_f16_mma_fragments(rK[None, m, k]) + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK_lower, + rQ_f16[None, k, n, 0], + rC[None, m, n], + ) + rC[None, m, n] = mma_sync( + rK_upper, + rQ_f16[None, k, n, 1], + rC[None, m, n], + ) + else: + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK[None, m, k], + rQ[(None, k % 2), k // 2, n], + rC[None, m, n], + ) + + cute.arch.mbarrier_arrive(tma_empty_mbar + tma_stage) + + k_start = block_id * BLOCK_K + warp_id * 32 + + for q in cutlass.range_constexpr(Q_TILES): + for i in cutlass.range_constexpr(4): + for j in cutlass.range_constexpr(2): + col = q * 8 + (lane_id % 4) * 2 + j + q_local_pos = col % MAX_DQL + q_pos = q_start + q_local_pos + k_pos = k_start + i * 8 + lane_id // 4 + rC[q * 8 + i * 2 + j] = ( + rC[q * 8 + i * 2 + j] if q_pos >= k_pos else float("-inf") + ) + + for q in cutlass.range_constexpr(Q_TILES): + # Thread-local reduction along the BLOCK_K dim. + rScore = cute.make_rmem_tensor(2, Float32) + rScore.fill(float("-inf")) + for i in cutlass.range_constexpr(4): + rScore[0] = cute.arch.fmax(rScore[0], rC[i * 2 + 0 + q * 8]) + rScore[1] = cute.arch.fmax(rScore[1], rC[i * 2 + 1 + q * 8]) + + # Warp reduction among lanes 0, 4, 8, 12, ... + for i in cutlass.range_constexpr(3): + offset = 4 << i + other0 = cute.arch.shuffle_sync_bfly( + rScore[0], offset=offset, mask=-1, mask_and_clamp=31 + ) + other1 = cute.arch.shuffle_sync_bfly( + rScore[1], offset=offset, mask=-1, mask_and_clamp=31 + ) + rScore[0] = cute.arch.fmax(rScore[0], other0) + rScore[1] = cute.arch.fmax(rScore[1], other1) + + if lane_id * 2 < MMA_N: + epi_buffer[q * MMA_N + lane_id * 2 + 0, warp_id] = rScore[0] + epi_buffer[q * MMA_N + lane_id * 2 + 1, warp_id] = rScore[1] + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + head_id = lane_id // MAX_DQL + q_local_pos = lane_id - head_id * MAX_DQL + valid_q = head_id < NUM_HEADS and q_local_pos < decode_query_len + if lane_id < BLOCK_Q and valid_q: + final_score = epi_buffer[lane_id, 0] + for i in cutlass.range_constexpr(1, 4): + final_score = cute.arch.fmax(final_score, epi_buffer[lane_id, i]) + + t = batch_id * decode_query_len + q_local_pos + score[head_id, t, block_id] = final_score + + tma_stage = (tma_stage + 1) % self.num_stages + if tma_stage == 0: + tma_parity ^= 1 + else: + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + # Early tiles past the request's true length: let them land before the CTA exits. + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.mbarrier_wait(tma_full_mbar + 1 % num_stages, 0) + + +class IndexScoreRunner: + """Compiled ``IndexDecodeScoreKernel`` variants, keyed by (index dtype, index heads, decode query length, head dim, + split_k, early issue). Batch, query-token count, page count and block-table width stay symbolic, so one compile + serves every decode step of its key. ``prefetch_old`` is always on; ``early`` also caps the prefetch at one block + per CTA.""" + + kernel_cache: dict = {} + SUPPORTED_HEAD_DIM = 128 # capacity: fixed: the checkpoint's index head dim + SUPPORTED_PAGE_SIZE = IndexDecodeScoreKernel.BLOCK_K + # BLOCK_Q must fit one warp's worth of epilogue lanes. + MAX_BLOCK_Q = 32 # capacity: fixed: the scorer's query tile + EARLY_PREFETCH_DEPTH = 1 + + @classmethod + def is_supported( + cls, + *, + q_dtype: torch.dtype, + num_heads: int, + head_dim: int, + page_size: int, + max_decode_query_len: int, + ) -> bool: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + from tensorrt_llm._utils import is_sm_100f + + return ( + is_sm_100f() + and q_dtype in TORCH_TO_CUTE_DTYPE + and head_dim == cls.SUPPORTED_HEAD_DIM + and page_size == cls.SUPPORTED_PAGE_SIZE + and num_heads * max_decode_query_len <= cls.MAX_BLOCK_Q + and max_decode_query_len >= 1 + ) + + @classmethod + def _compile( + cls, + q_dtype: torch.dtype, + num_heads: int, + max_decode_query_len: int, + head_dim: int, + split_k: int, + early: bool, + ) -> None: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + + key = (q_dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + if key in cls.kernel_cache: + return + cute_dtype = TORCH_TO_CUTE_DTYPE[q_dtype] + page_size = cls.SUPPORTED_PAGE_SIZE + sym_total_tokens = cute.sym_int() + sym_batch = cute.sym_int() + + # 16-element divisibility on every non-innermost stride is what the TMA descriptors for Q and K assume. + def _sym_stride(): + return cute.sym_int64(divisibility=16) + + q_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (sym_total_tokens, num_heads, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + # The index-K pool may be coalesced with the main K/V cache (per-page stride above page_size * head_dim). + k_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (cute.sym_int(), page_size, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + bt_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch, cute.sym_int()), stride_order=(1, 0) + ) + # The target passes a transposed view of the [heads, blocks, tokens] selector buffer. + score_fake = cute.runtime.make_fake_tensor( + cutlass.Float32, + (num_heads, sym_total_tokens, cute.sym_int()), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + sl_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch,), stride_order=(0,) + ) + fake_stream = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True) + kernel = IndexDecodeScoreKernel( + cute_dtype, + num_heads, + max_decode_query_len, + split_k, + head_dim, + prefetch_old=True, + early_issue=early, + prefetch_depth=cls.EARLY_PREFETCH_DEPTH if early else 0, + ) + cls.kernel_cache[key] = cute.compile( + kernel, + q_fake, + k_fake, + bt_fake, + score_fake, + sl_fake, + fake_stream, + options="--enable-tvm-ffi", + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_mc_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_mc_allreduce.py new file mode 100644 index 000000000000..fc992b83d152 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_mc_allreduce.py @@ -0,0 +1,376 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Low-latency TP all-reduce + residual + RMSNorm for decode (M <= 8 tokens) over a multicast mapping. + + partial = bf16(x + x2) (optional second input, e.g. shared + routed experts) + updated = bf16(residual + sum_r partial_r) (ranks summed in FP32, rank order) + normed = bf16(updated * rsqrt(mean(updated^2) + eps) * w) + +One cluster of CLUSTER CTAs per token; CTA c owns columns [c * H / CLUSTER, ...), one 16-byte chunk (8 BF16) +per thread. Each CTA pushes its slice of this rank's partial through the multicast mapping (one store lands in +every rank's buffer), releases dependents, polls its own buffer until all ranks' slices arrived, sums them, adds +the residual and reduces the RMS statistic across the cluster through DSMEM. + +Exchange protocol as in the K3 sandwich: int32 words, 0x80000000 = empty (pushed -0.0 halves become +0.0), each +CTA keeps a call counter whose parity selects the buffer half, readers re-arm the words they read. All ranks of +the TP group must issue the same calls in the same stream order. + +The workspace holds two exchange sets. With prepushed 3 two producers pushed this rank's x (set A) and x2 (set B), +e.g. the routed experts' finalize and the shared expert's down projection, and the kernel forms bf16(x + x2) per +rank from the two slots, exactly as the unpushed path does. + +Producers that push (the o_proj / shared-expert GEMV ``ar`` tuple, the routed-expert engine's ``arp`` descriptor) are +pointed at a bucket's exchange by its ``mc`` address, ``flags_for(8)`` counters, ``rank``, ``world_size`` and +``max_tokens``; ``get_workspace`` and ``_bucket`` select it. + +The kernel is ``torch.ops.trtllm.minimax_m3_mc_allreduce_norm``, reached through the ``ops/comm`` entries +``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm``; this module owns the workspaces, the bucket and +epilogue-option choice and the MXFP8 publication. +""" + +from dataclasses import dataclass +from typing import Any + +import torch + +from ..ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm as _allreduce_norm_op +from ..ops.comm.m3_mc_finalize_allreduce_norm import ( + m3_mc_finalize_allreduce_norm as _finalize_allreduce_norm_op, +) + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +BUCKETS = (8, 16, 48) # token capacity of each workspace (grid = bucket * cluster CTAs) +EMPTY_WORD = -2147483648 + + +def max_rows() -> int: + """The most rows one boundary all-reduce call (plain or MoE-finalize) serves: the decode plans and the + MC_ALLREDUCE / MC_FINALIZE ranges read this, so a wider bucket or a row split here widens them with no change in + the modeling.""" + return MAX_TOKENS + + +# Attention / MoE boundary epilogue options (each bit-identical): 1 residual + norm weight loaded at entry (before +# the poll); 2 cluster RMS exchange by st.async + mbarrier complete_tx instead of cluster.sync; 4 call parity rotated +# right after the poll (no trailing __syncthreads); 8 max-smem carveout so the next kernel's CTAs co-reside early. +# The 8-token bucket, by token count 1..8 (GB300 TP4 tuning). +AR_OPTS_BY_M = (7, 7, 15, 15, 15, 15, 7, 7) +# The 16- and 48-token buckets. +AR_OPTS_BY_BUCKET = {16: 15, 48: 15} +# The MoE-finalize all-reduce, per bucket. +AR_FIN_OPTS_BY_BUCKET = {8: 7, 16: 15, 48: 15} + + +# Option 2 is also the safe ordering of the cluster RMS exchange: its stores into peer CTAs' shared memory follow a +# cluster barrier arrive / wait pair. Without it they run before any cluster barrier, when a peer need not have +# started, so every call keeps it. +_OPT_CLUSTER_BARRIER = 2 + + +def _ar_opts(m: int, bucket: int, finalize: bool = False) -> int: + """Epilogue options for an m-row call on the given bucket; every option is bit-identical. A bucket missing from + the tables gets 0, and every result keeps _OPT_CLUSTER_BARRIER.""" + if finalize: + opts = AR_FIN_OPTS_BY_BUCKET.get(bucket, 0) + elif bucket == 8: + opts = AR_OPTS_BY_M[m - 1] + else: + opts = AR_OPTS_BY_BUCKET.get(bucket, 0) + return opts | _OPT_CLUSTER_BARRIER + + +@dataclass(eq=False) +class McAllReduceWorkspace: + """Two halves of [max_tokens][world][H] BF16 (int32 words) per rank behind one multicast mapping, plus one call + counter per CTA per cluster size. Create collectively and eagerly on every TP rank.""" + + uc: torch.Tensor + mc: torch.Tensor + flags: dict + rank: int + world_size: int + hidden: int + handle: Any + comm: Any + max_tokens: int = 8 + + @classmethod + def create(cls, mapping, hidden: int, max_tokens: int = 8) -> "McAllReduceWorkspace": + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + words = 4 * max_tokens * mapping.tp_size * (hidden // 2) # two exchange sets of two halves + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "McAllReduceWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 4, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("McAllReduceWorkspace.create failed on another rank") + return cls( + uc=uc, + mc=mc, + flags={}, + rank=mapping.tp_rank, + world_size=mapping.tp_size, + hidden=hidden, + handle=handle, + comm=comm, + max_tokens=max_tokens, + ) + + def flags_for(self, cluster: int): + """This cluster size's per-CTA call counters, or None when they do not exist yet during a capture. + get_workspace creates the cluster-8 counters with the workspace, so for cluster 8 (the only size built) + that None never happens and cannot differ between ranks.""" + flags = self.flags.get(cluster) + if flags is None: + if torch.cuda.is_current_stream_capturing(): + return None + flags = torch.zeros(self.max_tokens * cluster, dtype=torch.int32, device=self.uc.device) + self.flags[cluster] = flags + return flags + + +class McBuckets: + """One workspace per token bucket (8 / 16 / 48): calls of different sizes never share exchange slots.""" + + def __init__(self, workspaces): + self.workspaces = workspaces + self.hidden = workspaces[0].hidden + + def for_tokens(self, m: int): + for ws in self.workspaces: + if m <= ws.max_tokens: + return ws + return None + + +def _bucket(workspace, m): + return workspace.for_tokens(m) if isinstance(workspace, McBuckets) else workspace + + +def decline_reason(x, residual, workspace, cluster=8, x2=None) -> str: + """Why ``m3_mc_allreduce_norm`` returned None for these operands (for the dispatch's miss message).""" + if workspace is None: + return "no multicast workspace: it is created collectively on the first eager decode step" + bucket = _bucket(workspace, x.shape[0]) + if bucket is None: + return f"no workspace bucket holds {x.shape[0]} tokens (buckets {BUCKETS})" + if bucket.flags.get(cluster) is None: + return f"no {cluster}-CTA call counters for the {bucket.max_tokens}-token bucket" + return ( + f"operands outside the kernel: x {tuple(x.shape)} {x.dtype} contiguous={x.is_contiguous()}, " + f"residual contiguous={residual.is_contiguous()}, hidden {bucket.hidden}" + + ("" if x2 is None else f", x2 {tuple(x2.shape)} contiguous={x2.is_contiguous()}") + ) + + +def m3_mc_allreduce_norm( + x, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + x2=None, + quant=False, + prepushed=0, +): + """Return (normed, updated_residual) or None when unsupported. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM. + + A None makes the caller take another collective, so it must be the same on every TP rank. It is: it depends only + on the shapes, dtypes and layouts of the operands (the same code builds them on every rank) and on whether the + workspace's flags exist, which ``get_workspace`` creates collectively, before any capture, for cluster 8.""" + workspace = _bucket(workspace, x.shape[0]) + if workspace is None: + return None + if ( + x.dim() != 2 + or x.shape[0] > MAX_TOKENS + or x.dtype != torch.bfloat16 + or x.shape[1] != workspace.hidden + or not x.is_contiguous() + or not residual.is_contiguous() + or (x2 is not None and (not x2.is_contiguous() or x2.shape != x.shape)) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + outs = _allreduce_norm_op( + x, + x2, + residual, + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and x.shape[1] % 128 == 0), + int(prepushed), + _ar_opts(x.shape[0], workspace.max_tokens), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +def m3_mc_finalize_allreduce_norm( + fc2, + perm, + scales, + shared, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + quant=False, +): + """MoE finalize (sum_k scale * fc2[perm] + shared) + all-reduce + residual + RMSNorm in one kernel. + Returns (normed, updated_residual) or None when unsupported. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM.""" + m = residual.shape[0] + workspace = _bucket(workspace, m) + if workspace is None: + return None + if ( + residual.dim() != 2 + or m > MAX_TOKENS + or residual.shape[1] != workspace.hidden + or fc2.dtype != torch.bfloat16 + or not fc2.is_contiguous() + or fc2.dim() != 2 + or perm.dtype != torch.int32 + or perm.dim() != 2 + or perm.shape[0] != m + or not perm.is_contiguous() + or scales.dtype not in (torch.float32, torch.bfloat16) + or not scales.is_contiguous() + or (shared is not None and (shared.shape != residual.shape or not shared.is_contiguous())) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + outs = _finalize_allreduce_norm_op( + fc2, + perm, + scales, + shared, + residual.contiguous(), + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and residual.shape[1] % 128 == 0), + _ar_opts(m, workspace.max_tokens, finalize=True), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +_WORKSPACE: dict = {} +_AGREE_COMMS: dict = {} + + +def _tp_comm(mapping): + """A communicator over mapping's TP group, created once (collective over that group only).""" + key = tuple(mapping.tp_group) + comm = _AGREE_COMMS.get(key) + if comm is None: + from tensorrt_llm._utils import mpi_comm, mpi_disabled + + if mpi_disabled(): + comm = mapping.tp_group_pg + else: + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + _AGREE_COMMS[key] = comm + return comm + + +def tp_agree(mapping, ok: bool) -> bool: + """Whether ``ok`` holds on every rank of mapping's TP group (one blocking host collective). + + For a fast path whose availability is decided from rank-local state (a kernel build that can fail on one rank): + every rank must take the same path, so that they issue the same collectives with the same exchange protocol and + the same kernels. Call it once per decision, at build time, at the same point on every rank, outside CUDA-graph + capture; never per step. With one rank, or attention DP (whose ranks run different batches), it returns ``ok``. + """ + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return ok + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("tp_agree must run outside CUDA-graph capture") + from tensorrt_llm._torch.distributed.ops import _mnnvl_workspace_all_succeeded + + return _mnnvl_workspace_all_succeeded(_tp_comm(mapping), bool(ok)) + + +def tp_agree_built(mapping, built, what: str): + """``built`` when it is not None on every TP rank, else None on every rank (see ``tp_agree``).""" + if tp_agree(mapping, built is not None) or built is None: + return built + from tensorrt_llm.logger import logger + + logger.warning(f"{what}: built here but not on another TP rank; every rank falls back") + return None + + +def get_workspace(mapping, hidden: int): + """Process-wide workspace (created collectively on first eager use); None while capturing before creation. + + The first eager call for a hidden size (the warmup forward, before any CUDA-graph capture) creates one + McAllReduceWorkspace per bucket in BUCKETS on every TP rank (MPI collective over the TP group) plus each + bucket's 8-CTA-cluster call counters; later calls, captured or not, return the same McBuckets. The workspaces + live until process exit. + """ + ws = _WORKSPACE.get(hidden) + if ws is None: + if torch.cuda.is_current_stream_capturing(): + return None + wss = [McAllReduceWorkspace.create(mapping, hidden, b) for b in BUCKETS] + for w in wss: + w.flags_for(8) + ws = McBuckets(wss) + _WORKSPACE[hidden] = ws + return ws diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/__init__.py new file mode 100644 index 000000000000..0aa75c0bf82c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/__init__.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 MoE layers for low-batch decode as CuTe DSL kernels: ``m3_moe`` for 1..8 tokens (op.py) and ``m3_moe16`` +for 9..16 tokens (op16.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/m3_moe_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/m3_moe_kernel.py new file mode 100644 index 000000000000..1ad72e733c64 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/m3_moe_kernel.py @@ -0,0 +1,3719 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 routed experts for up to 8 decode tokens on one TP rank: ``m3_moe`` (group-phased weight stream). + +One persistent kernel replaces TRTLLM-Gen's MiniMax2 routing, FC1 (NVFP4 x NVFP4 + SwiGLU-OAI + NVFP4 requantization), +FC2 and finalize, optionally with the router GEMV and the activation's NVFP4 quantization folded in (FUSED_ROUTER). It +writes the rank's routed partial [M, 6144] bf16, bit for bit what the TRTLLM-Gen op returns. + +- Routing (warps 4..7, token t on warp 4 + t % 4): TRTLLM-Gen's MiniMax2 top-4, operation for operation; warp 4 then + makes the distinct experts slots in (token, top-k) order (match.any), slot s in group s / 4. +- Work: the groups' FC1 stages and FC2 tiles stream through one ring per CTA in phases: phase p = FC1 of group p (a + stream-K range of the group's unit-major stages), then FC2 of group p - LAG (this CTA's 32-row output blocks, 4 + slots' slices stacked in one 128-row MMA, B row j M + t). LAG phases of streaming hide the hand-off of group + p - LAG's intermediate between CTAs. +- FC1 stage: a 32-row unit's rows at 4 consecutive 256-K tiles (rows 32 q + r) against resident B rows QS q + t. + FC2 stage: 4 slots' 32 rows at one 256-K tile. Both carry 8 KB of raw scale atoms, spliced into the 4 instruction + atoms by warp 6 with one formula (word [r][q] <- raw atom 4 q + kb, word [r][m1]). +- FC1 epilogue: k-quarters summed in order, SwiGLU-OAI, NVFP4 requant into tagged 16-byte records (3 words of 3 data + bytes + the call's tag); a unit split between two CTAs is summed by its tail's holder (Lamport slot). +- Warps 5 and 8.. (XW warps) build each group's FC2 B (tiles + scale atoms) from the records once they carry the + call's tag; warp 5 copies the scales to TMEM. +- Finalize: bf16(fc2_alpha * acc) per (slot, token) into its top-k position, then per token the fma chain over its + top-4 in order, bf16 out. With an all-reduce descriptor, every output word is also pushed to the exchange + (AR_PRE: call parities and descriptor loaded during the last group's FC2). + +Configuration (trace-time constants) via M3_CONFIG = {"m_max", "num_ctas", "stages", "nb2_cap", "fused_router", "lag", +"xw", ...}; the plan (op.make_phase_plan) must be built for the same lag. +Launched with PDL: barrier setup, TMEM allocation, tensormap prefetch, bias / plan / router weight loads before +griddepcontrol.wait; every producer-output read and every global write after it. +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +_CFG = globals().get("M3_CONFIG") or {} + + +def _cfg(key: str, default): + return type(default)(_CFG.get(key, default)) + + +# ============================================================================= +# Problem shape (trace-time constants). +# ============================================================================= +H = 6144 # capacity: fixed: the checkpoint's hidden size +I_TP = 768 +TWO_I = 2 * I_TP +E_LOCAL = 128 +TOP_K = 4 # capacity: fixed: the checkpoint's top-k +ROUTE_SCALE = 2.0 +SUM_EPS = 1e-20 +M_MAX = _cfg("m_max", 1) +NCTA = _cfg("num_ctas", 152) +FUSED_ROUTER = bool(_cfg("fused_router", 0)) +NROUTE = E_LOCAL if FUSED_ROUTER else 0 # experts whose logits the router GEMV computes +NGEMV = (NROUTE + NCTA - 1) // NCTA # experts per CTA: bx, bx + NCTA, .. +TIMING = bool(_cfg("timing", 0)) +HANG_DBG = bool(_cfg("hang_dbg", 0)) +XVDUMP = bool( + _cfg("xvdump", 0) +) # debug only (needs a plan without split units): FC1 sums into `partials` +YDUMP = bool( + _cfg("ydump", 0) +) # debug only: FC2 outputs per (token, k) into `partials` (breaks its protocol) +LAG = _cfg("lag", 2) # phase p runs FC1 of group p and FC2 of group p - LAG +G_CAP = TOP_K * M_MAX # distinct experts per call +NG_CAP = M_MAX # groups of 4 slots +XW = _cfg("xw", 5 if M_MAX > 4 else (4 if M_MAX > 2 else 1)) # FC2 B builder warps: 5, then 8 .. +THREADS = 256 + 32 * (XW - 1) +GEMV_THREADS = 256 # capacity: fixed: the CTA size +# SHARED: the MoE's shared expert (MXFP8 gate_up [1536, 6144] -> SwiGLU-OAI -> MXFP8 down [6144, 768]) on NSH more +# CTAs (warps 0-3), its output added in finalize: out = bf16(shared + routed) (shared_output.add_(routed_output)). +# The math is mxfp8_low_m_gemm's (native MXFP8 activation quant, mma.sync m16n8k32 e4m3 per 32-K block, block scales +# as one fma, its partial-sum order, bf16) and the Triton silu_and_mul's, operation for operation. Shared CTA s owns +# intermediate positions 32 s .. 32 s + 31 (gate and up rows: MMA tiles = warps 0-3) and down rows 256 s .. + 255; +# 16-row x 768-byte weight chunks (+ their scale atoms) stream through NBW buffers per warp, the first NBW before +# the grid dependency. h crosses the shared CTAs, y_sh reaches the routed CTAs' finalize (both tagged words). +SHARED = bool(_cfg("shared", 0)) +# LATE_TRIGGER: release the dependents at teardown, not after the grid dependency. An early release lets the next +# kernels' CTAs (the all-reduce, then the attention it releases at entry) become resident during this kernel, packed +# onto the SMs it leaves free, where the attention then runs on a fraction of the GPU. +LATE_TRIGGER = bool(_cfg("late_trigger", 1)) +# AR_PRE: the finalize's all-reduce push uses the call parities and descriptor fields loaded while the last group's +# FC2 runs (not two dependent global loads per push on the kernel's tail). +AR_PRE = bool(_cfg("ar_pre", 1)) +# XSTAGE (fused router): the hidden states land in the (still idle) ring by one bulk copy per token right after the +# grid dependency; the NVFP4 quantization of B1 reads them from shared memory (one HBM / L2 round trip, not ~8). +XSTAGE = bool(_cfg("xstage", 1)) and bool(_cfg("fused_router", 0)) +NSH = 24 if SHARED else 0 +SH_I = 768 +SH_KB = H // 32 # gate_up K blocks # capacity: fixed: a kernel tile constant +SH_DKB = SH_I // 32 # down K blocks # capacity: fixed: a kernel tile constant +SH_SPLIT = 2 # gate_up K slices (= m3_mx_gemv's cluster ranks, contiguous K ranges): MMA rows 64 q + .. = slice q +SH_GT = ( + SH_KB // 4 // SH_SPLIT +) # gate_up k-tiles (128 K) per slice # capacity: fixed: a kernel tile constant +SH_STAGES_GU = SH_GT +SH_STAGES = SH_GT + 2 * ( + SH_I // 128 +) # + 2 down tiles x 6 k-tiles # capacity: fixed: pipeline stages +SHX_BAR = 6 +# SH_PRETILE: the shared weights come pre-tiled (op._pretile_shared): per shared CTA its 36 stages, each the 16 KB +# shared-memory image of its 128-row MMA tile (128-byte swizzle) followed by the stage's 512-byte instruction scale +# atom, so a stage is two contiguous bulk copies (not 128 row segments of 128 bytes) and needs no splicing. +SH_PRETILE = bool( + _cfg("sh_pretile", 1 if M_MAX <= 2 else 0) +) # (in-chain: a win at M <= 2, not at 4..8) +SH_STAGE_BYTES = 16384 + 512 +MMA_M = 128 +KT_BYTES = 128 +INC_K = 2 +NUM_KB = 4 +# FC1 +N1 = 16 if M_MAX <= 4 else 32 # B rows: QS q + t (token t at the stage's k-tile 4 j + q) +QS = N1 // 4 +R1 = 32 +C1 = 4 +U1 = TWO_I // R1 # 48 units per slot +K1_TILES = H // 256 +S1 = K1_TILES // C1 # 6 stages per unit # capacity: fixed: pipeline stages +NUM_BYTES_A = MMA_M * KT_BYTES # capacity: fixed: a stage's operand bytes +NUM_BYTES_BOX1 = R1 * KT_BYTES +SF1_KA = H // 64 +SF1_RB = TWO_I // 128 +NUM_BYTES_SFA_RAW = 16 * 512 +NUM_WORDS_SFA_RAW = NUM_BYTES_SFA_RAW // 4 +NUM_BYTES_SFA = NUM_KB * 512 # capacity: fixed: a stage's scale bytes +NUM_WORDS_SFA = NUM_BYTES_SFA // 4 +NUM_BYTES_B1 = N1 * KT_BYTES +NUM_SFB1_ATOMS = S1 * NUM_KB +# FC2 +R2 = 32 +NB2 = H // R2 # 192 +KT2 = I_TP // 256 # 3 +SF2_KA = I_TP // 64 +SF2_RB = H // 128 +N2 = max(8, (4 * M_MAX + 7) // 8 * 8) # B rows j M + t +NUM_BYTES_B2 = N2 * KT_BYTES +NB2_CAP = _cfg("nb2_cap", 2) +UNITS_ROW = TWO_I // R1 # 48 records per (slot, token) row +H_ROW = UNITS_ROW * 16 +PARTIAL_EMPTY = -1 # capacity: fixed: a sentinel, not a size +EPI_THREADS = 128 +EPI_BAR = 1 +ROUTE_BAR = 2 +GEMV_BAR = 3 +B2_BAR = 5 +TD_STRIDE = 64 # tdbg columns per CTA +TOPK_BAR = 4 +B1_AFTER_TOPK = bool( + _cfg("b1_after_topk", 0) +) # (experiment) the B1 build waits for the routing warps' top-k +TOPK2_BAR = 7 +EARLY0 = bool( + _cfg("early0", 1) +) # warp 4 streams group 0 (token 0's experts) right after its own top-k +EARLY0_N = _cfg("early0_n", 99) # stages issued early (capped at STAGES) +# routing warps: 4..7, then the FC2 B builder warps 8.. (idle until the routing is done) up to one warp per token +RW_EXTRA = max(0, min(M_MAX - 4, (THREADS // 32) - 8)) if bool(_cfg("rw_extra", 1)) else 0 +NRW0 = ( + min(4, M_MAX) if bool(_cfg("rw_trim", 1)) else 4 +) # warps 4.. with a token (no idle pollers in the barrier) +NRW = NRW0 + RW_EXTRA +LG_REPL = _cfg("lg_repl", 32 if M_MAX > 4 else 128) +GEMV_SMEM = bool( + _cfg("gemv_smem", 1) +) # (XSTAGE) the router GEMV reads the staged hidden states from smem +LG_STRIDE = M_MAX * E_LOCAL + 32 +POLL_NS = _cfg("poll_ns", 150) +B2_POLL_NS = _cfg("b2_poll_ns", 200) +# B2IMG (needs SFA_CP: warp 6 is free): the FC1 epilogue writes each finished unit's FP4 bytes and scale byte straight +# into a per-group global image laid out as the FC2 B tile + its scale atoms (exactly what the builder warps assembled +# from the tagged records); per group, warp 6 joins the epilogue (fc1_done), fences the async proxy and does one +# release add of this CTA's finished units to the group counter; warp 5 acquires the counter (every group gains +# CNT_PER_CALL per call, CTA 0 pads absent slots), fences, and bulk-copies the image into the B2 buffer. +B2IMG = bool(_cfg("b2img", 0)) and bool(_cfg("sfa_cp", 0)) +IMG_B2 = KT2 * NUM_BYTES_B2 +IMG_SF = KT2 * NUM_KB * 512 +IMG_BYTES = IMG_B2 + IMG_SF +CNT_PER_CALL = 4 * (TWO_I // R1) +B2IMG_NS = _cfg("b2img_ns", 16) # (B2IMG) counter poll back-off +B2_DLY = _cfg( + "b2_dly", 0 +) # (stress) the release warp spins up to this many ns (CTA- and group-dependent) first +W_WARP = 4 # capacity: fixed: a warp role index +X_WARP = 5 # capacity: fixed: a warp role index +S_WARP = 6 # capacity: fixed: a warp role index +MMA_WARP = 7 # capacity: fixed: a warp role index +# (M <= 3) the early stages are issued by the idle MMA warp, not by warp 4 (routing's critical path) +E0_WARP = MMA_WARP # capacity: fixed: a warp role index +E0_BAR = 8 +E1_BAR = 9 +E0_OFFLOAD = EARLY0 and NRW0 <= 3 and bool(_cfg("e0_off", 1)) +EVICT_FIRST = 0x12F0000000000000 +_LOG2E = 1.4426950408889634 +LD_POW2 = ( + 1 if M_MAX == 1 else 2 if M_MAX == 2 else 4 if M_MAX <= 4 else 8 +) # tcgen05.ld .num covering M columns + +_FIXED_SMEM = ( + S1 * NUM_BYTES_B1 + + 2 * KT2 * NUM_BYTES_B2 + + 2 * KT2 * NUM_KB * 512 + + NB2_CAP * M_MAX * TOP_K * R2 * 4 + + 2 * M_MAX * C1 * R1 * 4 + + (M_MAX * (H // 16) if FUSED_ROUTER else 0) + + 6 * 1024 +) +# SFA_CP: the A scale words go by 4-byte cp.async straight into the instruction atom layout (no raw scale box, no +# splicer warp; the MMA warp tcgen05.cp's them), completing on the stage's full barrier: 18 KB stages instead of 26 KB +SFA_CP = bool(_cfg("sfa_cp", 0)) +_RAW_STAGE = 0 if SFA_CP else NUM_BYTES_SFA_RAW +_STAGE_SMEM = NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA +_STAGES_FIT = (227 * 1024 - _FIXED_SMEM - (128 if SHARED else 0)) // _STAGE_SMEM +ACC1_COL = 0 +ACC2_COL = 2 * N1 +SFB1_COL = ACC2_COL + 2 * N2 +SFB2_COL = SFB1_COL + NUM_SFB1_ATOMS * 4 +SFA1_COL = SFB2_COL + 2 * KT2 * NUM_KB * 4 +STAGES = min(_cfg("stages", 12 if SFA_CP else 8), _STAGES_FIT, (512 - SFA1_COL) // 16) +TMEM_COLS = 512 +assert 1 <= M_MAX <= 8, M_MAX +assert STAGES >= 3, (STAGES, _STAGES_FIT, SFA1_COL) +assert M_MAX * H * 2 <= STAGES * (NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA), ( + "x staging must fit the ring" +) +# one shared-memory region for the ring and the FC1 / FC2 B operands (a shared CTA reuses all of it) +_OFF_SFAR = STAGES * NUM_BYTES_A +_OFF_SFA = _OFF_SFAR + STAGES * _RAW_STAGE +_OFF_B1 = _OFF_SFA + STAGES * NUM_BYTES_SFA +_OFF_B2 = _OFF_B1 + S1 * NUM_BYTES_B1 +_OFF_SFB2 = _OFF_B2 + 2 * KT2 * NUM_BYTES_B2 +BIG = _OFF_SFB2 + 2 * KT2 * NUM_KB * 512 +# shared CTA layout in BIG: weight buffers, x quarters (2), x scales [M, 192], gate / up [4, 16, M], h, h scales +# shared CTA layout in BIG: A ring (16 KB stages), raw scale atoms (4 per stage), spliced atoms, B (x: [stage][16 rows] +# [128 B]; h: [6][8][128 B]), x / h scale codes, gate_up partial exchange +_SB_A = 0 +_SB_RAW = None +_SH_FIX = ( + SH_GT * 16 * 128 + + 6 * 8 * 128 + + M_MAX * SH_KB + + M_MAX * SH_DKB + + 64 * 8 * 4 + + 2 * 32 * 8 * 4 + + 1024 +) +SH_NST = min(8, (BIG - _SH_FIX) // (NUM_BYTES_A + 4 * 512 + 512)) if SHARED else 1 +_SB_RAW = SH_NST * NUM_BYTES_A +_SB_SPL = _SB_RAW + SH_NST * 2048 +_SB_BX = (_SB_SPL + SH_NST * 512 + 1023) // 1024 * 1024 +_SB_BH = _SB_BX + SH_GT * 16 * 128 +_SB_XC = _SB_BH + 6 * 8 * 128 +_SB_HC = _SB_XC + M_MAX * SH_KB +_SB_XCH = (_SB_HC + M_MAX * SH_DKB + 15) // 16 * 16 +_SB_GU = _SB_XCH + 64 * 8 * 4 +_SB_END = _SB_GU + 2 * 32 * 8 * 4 +# shared CTA TMEM columns: gate_up accumulator (N 16), down accumulators (2 x N 8), A scales per stage, B scales +SH_ACC_GU = 0 +SH_ACC_DN = 16 +SH_SFA = 32 +SH_SFB_GU = SH_SFA + 4 * 8 +SH_SFB_DN = SH_SFB_GU + 4 * SH_GT # capacity: fixed: a kernel tile constant +SH_TMEM = 256 # capacity: fixed: TMEM columns +if SHARED: + assert FUSED_ROUTER, "shared expert: fused router only" + assert SH_STAGES - SH_NST <= 64, SH_STAGES + assert SH_NST >= 3 and _SB_END <= BIG and SH_SFB_DN + 24 <= SH_TMEM, (SH_NST, _SB_END, BIG) + + +# ============================================================================= +# Inline-PTX helpers. +# ============================================================================= +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +# ---- all-reduce push (the MoE boundary's multicast exchange, set A): +# arp = int64 [enable, mc, flags, rank, world, maxt] +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32_if(addr, value, pred, *, loc=None, ip=None): + """st.global.b32 when pred != 0.""" + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip), + cutlass.Int32(pred).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, $2, 0;\n\t@p st.global.b32 [$0], $1;\n\t}", "l,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _ar_push(arp, v, low, t, cl): + """Push bf16 channel pair (cl, cl + 1) of token t (v = channel cl's bf16 value on this lane, its neighbour's on + lane ^ 8) into this rank's set-A slot of the exchange; the slot half is the all-reduce CTA (t, cl // 768)'s call + parity. Only lanes with ``low`` store; every lane of the warp must call it (shuffle).""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + world = arp.load(idx=4) + parity = cutlass.Int64( + _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cs // cutlass.Int32(H // 8)) + ) + & cutlass.Int32(1) + ) + off = ( + parity * arp.load(idx=5) * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + arp.load(idx=3)) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(arp.load(idx=1) + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +def _ar_parity(arp, t, cl): + """The call parity _ar_push reads for token t's channel cl (fixed between this kernel's grid dependency and the + next all-reduce, which this kernel releases at teardown): loaded ahead of the finalize (AR_PRE).""" + return _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cl // cutlass.Int32(H // 8)) + ) + + +def _ar_push_pre(ard, parity32, v, low, t, cl): + """_ar_push with the parity (_ar_parity) and the descriptor fields ard = (mc, rank, world, maxt) preloaded.""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + mc, rk, world, maxt = ard + parity = cutlass.Int64(parity32 & cutlass.Int32(1)) + off = ( + parity * maxt * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + rk) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(mc + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +@dsl_user_op +def _bar_arrive(bar_id: int, threads: int, *, loc=None, ip=None): + """bar.arrive: signal a named barrier without waiting (its memory ordering is bar.sync's).""" + _llvm.inline_asm( + None, + [], + f"bar.arrive {bar_id}, {threads};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _st_relaxed_u32(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "st.relaxed.gpu.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _st_relaxed_u64(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int64(val)], + "st.relaxed.gpu.global.u64 [$0], $1;", + "l,l", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm( + _T.i64(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u64 $0, [$1];", + "=l,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _nanosleep(ns: int, *, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + f"nanosleep.u32 {ns};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _gtime(*, loc=None, ip=None): + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), + [], + "mov.u64 $0, %globaltimer;", + "=l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _smid(*, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [], + "mov.u32 $0, %smid;", + "=r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@cute.jit +def _stamp(tdbg: cutlass.Array, bx, i: cutlass.Constexpr): + if cutlass.const_expr(TIMING): + tdbg.store(_gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(i)) + + +@dsl_user_op +def _rcp(a, *, loc=None, ip=None): + """rcp.approx.ftz.f32 (the reciprocal TRT-LLM's FP4 quantization uses).""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _div_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _sub_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "sub.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _mul_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3_bits(x, *, loc=None, ip=None): + """e4m3 (round to nearest even, saturating) of x as an int 0..255.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, 0f00000000, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3_to_f32(b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(b)], + "{\n.reg .b16 t, l, u;\n.reg .b32 h;\ncvt.u16.u32 t, $1;\ncvt.rn.f16x2.e4m3x2 h, t;\nmov.b32 {l, u}, h;\n" + "cvt.f32.f16 $0, l;\n}", + "=f,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e2m1x2(hi, lo, *, loc=None, ip=None): + """One byte of two E2M1 values (lo in the low nibble), round to nearest even, saturating.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b8 t;\ncvt.rn.satfinite.e2m1x2.f32 t, $1, $2;\ncvt.u32.u8 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2(x, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _bf16_round(x, *, loc=None, ip=None): + """x rounded to bf16 (nearest even), as f32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\nmov.b32 $0, {0, t};\n}", + "=f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _redux_max_u32(v, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(v)], + "redux.sync.max.u32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16 (nearest even) bits of x in the low 16 bits.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ue8m0_rp(x, *, loc=None, ip=None): + """UE8M0 code of x rounded toward +inf, saturating (__nv_cvt_float_to_e8m0(x, SATFINITE, RoundPosInf)).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rp.satfinite.ue8m0x2.f32 t, $1, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3x2(hi, lo, *, loc=None, ip=None): + """Two e4m3 (nearest even, saturating) in 16 bits, lo in the low byte.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, $1, $2;\ncvt.u32.u16 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _min_xorsign_abs(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "min.xorsign.abs.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _div_full(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.full.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2_full(x, *, loc=None, ip=None): + """ex2.approx.f32 (no flush to zero), Triton's tl.exp.""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _l2_prefetch(addr, nbytes, *, loc=None, ip=None): + """cp.async.bulk.prefetch.L2 of nbytes (a multiple of 16) at global address addr (16-byte aligned).""" + _llvm.inline_asm( + None, + [ + cutlass.Int64(addr).ir_value(loc=loc, ip=ip), + cutlass.Int32(nbytes).ir_value(loc=loc, ip=ip), + ], + "cp.async.bulk.prefetch.L2.global [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +def _mx_osc(amax): + """mxfp8 quantization of a block with absolute max amax: (UE8M0 code, the element multiplier).""" + code = _ue8m0_rp(_mul_rn(amax, _rcp(cutlass.Float32(448.0)))) + cf = cutlass.Int32( + cutlass.select_( + code == cutlass.Int32(255), + cutlass.Int32(0x7FFFFFFF), + cutlass.select_( + code == cutlass.Int32(0), cutlass.Int32(0x00400000), code << cutlass.Int32(23) + ), + ) + ).bitcast(cutlass.Float32) + return code, cutlass.select_(amax != cutlass.Float32(0.0), _rcp(cf), cutlass.Float32(0.0)) + + +def _mx_words(v, osc): + """e4m3 words of len(v) (a multiple of 4) values times osc, in element order.""" + words = [] + for t in range(len(v) // 4): + lo = _e4m3x2(_mul_rn(v[4 * t + 1], osc), _mul_rn(v[4 * t], osc)) + hi = _e4m3x2(_mul_rn(v[4 * t + 3], osc), _mul_rn(v[4 * t + 2], osc)) + words.append(lo | (hi << cutlass.Int32(16))) + return words + + +def _twiddle(score): + """Float bits mapped so that unsigned order is float order (cub TwiddleIn).""" + bits = ( + score.bitcast(cutlass.Int32) + if hasattr(score, "bitcast") + else cutlass.Float32(score).bitcast(cutlass.Int32) + ) + return cutlass.select_(bits < cutlass.Int32(0), ~bits, bits | cutlass.Int32(-2147483648)) + + +@cute.jit +def _wait(bar: cutlass.Array, parity): + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +def _tptr(base_row, col, dtype=cutlass.Int32): + return cutlass.inttoptr((base_row << 16) | col, 6, dtype) + + +@dsl_user_op +def _match_any(value, *, loc=None, ip=None): + """The mask of the warp's lanes holding the same 32-bit value.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(value)], + "match.any.sync.b32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@cute.jit +def _waitd(bar: cutlass.Array, parity, tdbg: cutlass.Array, site: cutlass.Constexpr): + """_wait, or (HANG_DBG) give up after ~2^22 polls and record the site in tdbg[bx, 24 + warp].""" + if cutlass.const_expr(HANG_DBG): + n = cutlass.Int32(0) + while (not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity)) & ( + n < cutlass.Int32(1 << 22) + ): + n = n + cutlass.Int32(1) + if n >= cutlass.Int32(1 << 22): + bx, _, _ = cute.arch.block_idx() + w = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + tdbg.store( + cutlass.Int64(site), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + w + ) + else: + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +@cute.jit +def _adv(stage, phase): + """The next ring position's (stage, phase).""" + stage = stage + cutlass.Int32(1) + wrap = stage == cutlass.Int32(STAGES) + return cutlass.select_(wrap, cutlass.Int32(0), stage), cutlass.select_( + wrap, phase ^ cutlass.Int32(1), phase + ) + + +@cute.jit +def _seg(i0, i1, k): + """Segment k of the FC1 item range [i0, i1) (items = unit-major stages): (unit, first stage, end stage). Order: + the last unit (its head, the part a split unit's next CTA does not cover), then the first unit, then the rest.""" + uf = i0 // cutlass.Int32(S1) + ul = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) + unit = uf + k - cutlass.Int32(1) + unit = cutlass.select_(k == cutlass.Int32(0), ul, unit) + unit = cutlass.select_(ul == uf, uf, unit) + j0 = i0 - unit * cutlass.Int32(S1) + j0 = cutlass.select_(j0 < cutlass.Int32(0), cutlass.Int32(0), j0) + j1 = i1 - unit * cutlass.Int32(S1) + j1 = cutlass.select_(j1 > cutlass.Int32(S1), cutlass.Int32(S1), j1) + return unit, j0, j1 + + +@cute.jit +def _nsegs(i0, i1): + n = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) - i0 // cutlass.Int32(S1) + cutlass.Int32(1) + return cutlass.select_(i1 > i0, n, cutlass.Int32(0)) + + +@cute.jit +def _sfb1_to_tmem(warp, lane, base_row, base_col, src: cutlass.Array, b1_ready: cutlass.Array): + """B1 scales to TMEM: instruction (j, kb)'s column holds, at lane n = QS q + t, token t's 4 scales of k-tile + 4 j + q, K 64 kb .. (src: int32 words [M, 96] of the token scales); each warp writes its sub-partition.""" + q = lane // cutlass.Int32(QS) + tl = lane - q * cutlass.Int32(QS) + sf_ok = (lane < cutlass.Int32(N1)) & (tl < cutlass.Int32(M_MAX)) + sfv = [] + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + j = a // NUM_KB + kb = a % NUM_KB + wsf = src.load( + idx=cutlass.select_( + sf_ok, + tl * cutlass.Int32(H // 64) + + (cutlass.Int32(4 * j) + q) * cutlass.Int32(4) + + cutlass.Int32(kb), + cutlass.Int32(0), + ) + ) + sfv.append(cutlass.select_(sf_ok, cutlass.Int32(wsf), cutlass.Int32(0))) + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + prims.tcgen05_st( + "32x32b", + _tptr(base_row + warp * cutlass.Int32(32), base_col + SFB1_COL + a * 4), + sfv[a], + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(b1_ready) + + +@cute.jit +def _b1_addr(t, kb16): + """Word index in the resident B tiles of the 8 FP4 bytes of token t's 16-value block kb16.""" + kt = kb16 >> cutlass.Int32(4) + j = kt >> cutlass.Int32(2) + n = (kt & cutlass.Int32(3)) * cutlass.Int32(QS) + t + cc = (kb16 >> cutlass.Int32(1)) & cutlass.Int32(7) + return ( + j * cutlass.Int32(NUM_BYTES_B1) + + n * cutlass.Int32(KT_BYTES) + + (cc ^ (n & cutlass.Int32(7))) * cutlass.Int32(16) + + (kb16 & cutlass.Int32(1)) * cutlass.Int32(8) + ) // cutlass.Int32(4) + + +@cute.jit +def _build_b1( + tidx, + warp, + lane, + base_row, + base_col, + x32: cutlass.Array, + xsf32: cutlass.Array, + sB1: cutlass.Array, + b1_ready: cutlass.Array, +): + """B (the M tokens' whole K) into the resident tiles from the NVFP4 activation, its scales to TMEM.""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + for r in cutlass.range_constexpr((M_MAX * (H // 16) + EPI_THREADS - 1) // EPI_THREADS): + b = tidx + cutlass.Int32(r * EPI_THREADS) + if b < cutlass.Int32(M_MAX * (H // 16)): + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + v2 = x32.load(idx=b * cutlass.Int32(2), vector_size=2, alignment=8) + sB1_32.store( + (cutlass.Int32(v2[0]), cutlass.Int32(v2[1])), idx=_b1_addr(t, kb), alignment=8 + ) + _sfb1_to_tmem(warp, lane, base_row, base_col, xsf32, b1_ready) + + +@cute.jit +def _build_b1_fused( + tidx, + warp, + lane, + base_row, + base_col, + xb32: cutlass.Array, + sfsc: cutlass.Array, + sB1: cutlass.Array, + s_xsf: cutlass.Array, + b1_ready: cutlass.Array, +): + """_build_b1 from the bf16 hidden states, quantized exactly as fp4_quantize(x, sf_scale, 16).""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + s_xsf8 = cutlass.Array( + s_xsf.data_ptr(0), shape=(M_MAX * (H // 16),), dtype=cutlass.Int8, alignment=16 + ) + sc = cutlass.Float32(sfsc.load(idx=0)) + rsc = _rcp(sc) + rsix = _rcp(cutlass.Float32(6.0)) + NBLK = M_MAX * (H // 16) // EPI_THREADS + QB = ( + 3 if NBLK % 3 == 0 else (2 if NBLK % 2 == 0 else 1) + ) # blocks whose loads are in flight together + # a rolled loop over groups of QB blocks (a fully unrolled one is ~50 KB of SASS at M = 8: instruction fetch bound) + for rg in cutlass.range(NBLK // QB, unroll=1): + wall = [] + for rr in cutlass.range_constexpr(QB): + bq = tidx + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(EPI_THREADS) + for h in cutlass.range_constexpr(2): + wall.append( + xb32.load( + idx=bq * cutlass.Int32(8) + cutlass.Int32(4 * h), + vector_size=4, + alignment=16, + ) + ) + for rr in cutlass.range_constexpr(QB): + b = tidx + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(EPI_THREADS) + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + w = [] + for h in cutlass.range_constexpr(2): + v4 = wall[rr * 2 + h] + for z in cutlass.range_constexpr(4): + w.append(cutlass.Int32(v4[z])) + xs = [] + for z in cutlass.range_constexpr(8): + xs.append((w[z] << cutlass.Int32(16)).bitcast(cutlass.Float32)) + xs.append((w[z] & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + vmax = cute.math.abs(xs[0]) + for z in cutlass.range_constexpr(1, 16): + vmax = cute.arch.fmax(vmax, cute.math.abs(xs[z])) + sfb = _e4m3_bits(_mul_rn(sc, _mul_rn(vmax, rsix))) + osc = cutlass.select_( + vmax != cutlass.Float32(0.0), + _rcp(_mul_rn(_e4m3_to_f32(sfb), rsc)), + cutlass.Float32(0.0), + ) + by = [ + _e2m1x2(_mul_rn(xs[2 * z + 1], osc), _mul_rn(xs[2 * z], osc)) & cutlass.Int32(0xFF) + for z in range(8) + ] + lo = ( + by[0] + | (by[1] << cutlass.Int32(8)) + | (by[2] << cutlass.Int32(16)) + | (by[3] << cutlass.Int32(24)) + ) + hi = ( + by[4] + | (by[5] << cutlass.Int32(8)) + | (by[6] << cutlass.Int32(16)) + | (by[7] << cutlass.Int32(24)) + ) + sB1_32.store((lo, hi), idx=_b1_addr(t, kb), alignment=8) + s_xsf8.store(cutlass.Int8(sfb), idx=b) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + _sfb1_to_tmem(warp, lane, base_row, base_col, s_xsf, b1_ready) + + +@cute.jit +def _swiglu_quant(xv, gate_raw, is_up, a, c, alpha, beta, lim): + """TRTLLM-Gen's FC1 epilogue (the swiGlu Bmm E2m1 output), operation for operation: lane value xv is the up row, + gate_raw its gate row (fp32 sums). Returns (q, the scale's e4m3 bits) of the 16-row NVFP4 block (lanes r, r ^ 16 + and r ^ 8 cover it; amax over the warp's up lanes of a half). a = fc31_alpha, c = fc31_scale_c.""" + neg_max = cutlass.Float32(-3.40282346638528859812e38) + gc = cutlass.select_(gate_raw > lim, lim, gate_raw) + gc = cutlass.select_(gate_raw < neg_max, neg_max, gc) + uc = cutlass.select_(xv > lim, lim, xv) + uc = cutlass.select_(xv < -lim, -lim, uc) + c1 = _mul_rn(a, _mul_rn(alpha, cutlass.Float32(1.4426950216293334961))) + sgm = _rcp(_add_rn(_ex2(-_mul_rn(gc, c1)), cutlass.Float32(1.0))) + gs = _mul_rn(gc, sgm) + h = _mul_rn(_fma_rn(uc, a, _mul_rn(a, beta)), gs) + h = cutlass.select_(is_up, h, cutlass.Float32(0.0)) + amax = cute.arch.fmax(_mul_rn(cute.math.abs(h), c), cutlass.Float32(0.0)) + for off in (16, 8, 4, 2, 1): + amax = cute.arch.fmax(amax, cute.arch.shuffle_sync_bfly(amax, off)) + sfb = _e4m3_bits(_mul_rn(amax, cutlass.Float32(0.16666667163372039795))) + sff = _e4m3_to_f32(sfb) + osc = cutlass.select_(sff != cutlass.Float32(0.0), _rcp(sff), cutlass.Float32(0.0)) + return _mul_rn(_mul_rn(h, osc), c), sfb + + +@cute.jit +def _plan_range(s_plan: cutlass.Array, p): + w = s_plan.load(idx=p) + return w & cutlass.Int32(0xFFFF), w >> cutlass.Int32(16) + + +@cute.jit +def _sh_issue( + lane, + sid, + g, + s_big: cutlass.Array, + sh_full: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, +): + """Stage g of the shared CTA's stream into ring slot g % SH_NST (warp 0). g < SH_GT: gate_up k-tile g of both + slices: quadrant Q = 2 q + u (q slice, u 0 gate / 1 up) = rows 32 sid .. (+ 768 for up) at k-tile SH_GT q + g, its + raw scale atom at raw slot Q. Else down tile j = (g - SH_GT) / 6 (rows 256 sid + 128 j ..), k-tile + (g - SH_GT) % 6.""" + st = g % cutlass.Int32(SH_NST) + bar = sh_full.subview(st) + if cutlass.const_expr(SH_PRETILE): + if lane == 0: + src = (sid * cutlass.Int32(SH_STAGES) + g) * cutlass.Int32(SH_STAGE_BYTES) + prims.mbarrier_arrive_expect_tx(bar, SH_STAGE_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), gusf.subview(src), bar, 16384 + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + gusf.subview(src + cutlass.Int32(16384)), + bar, + 512, + ) + cute.arch.sync_warp() + else: + if g < cutlass.Int32(SH_GT): + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 4 * 512) + cute.arch.sync_warp() + if lane < 4: + q = lane >> cutlass.Int32(1) + u = lane & cutlass.Int32(1) + row0 = sid * cutlass.Int32(32) + u * cutlass.Int32(SH_I) + kt = q * cutlass.Int32(SH_GT) + g + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A) + lane * cutlass.Int32(4096)), + tma_gu, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview( + cutlass.Int32(_SB_RAW) + + st * cutlass.Int32(2048) + + lane * cutlass.Int32(512) + ), + gusf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_KB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(6) + kt = gd - jt * cutlass.Int32(6) + row0 = sid * cutlass.Int32(256) + jt * cutlass.Int32(128) + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 512) + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), + tma_dn, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_RAW) + st * cutlass.Int32(2048)), + dnsf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_DKB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + cute.arch.sync_warp() + + +@dsl_user_op +def _red_release_add(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "red.release.gpu.global.add.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_acquire_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.acquire.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fence_proxy_async_global(*, loc=None, ip=None): + _asm(None, [], "fence.proxy.async.global;", "", True, loc, ip) + + +@cute.jit +def _sfa_to_tmem(sSFA: cutlass.Array, stage, base_row, base_col): + """(SFA_CP, elected MMA thread) the stage's 4 scale atoms smem -> TMEM (what the splicer warp does otherwise).""" + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SFA1_COL + stage * cutlass.Int32(16) + kb * 4), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + + +@cute.jit +def _sf1_cp( + lane, stage, sf_line, c, sSFA: cutlass.Array, s1w: cutlass.Array, ab_full: cutlass.Array +): + """(SFA_CP) FC1 stage scale atoms: word [r][q] of atom kb <- raw atom 4 q + kb (raw line sf_line = 4 x atom) + word [r][c % 4] (what the splicer copies out of the raw box), 4-byte cp.async by the warp's 32 lanes; their + arrivals (noinc) complete the stage's full barrier together with the A boxes' bytes.""" + src0 = ( + (sf_line >> cutlass.Int32(2)) * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (c & cutlass.Int32(3)) + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for q in cutlass.range_constexpr(C1): + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + q)), + s1w.subview(src0 + cutlass.Int32((4 * q + kb) * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _sf2_cp( + lane, + stage, + g, + blk, + t2: cutlass.Constexpr, + n_here, + s_el: cutlass.Array, + sSFA: cutlass.Array, + s2w: cutlass.Array, + ab_full: cutlass.Array, +): + """(SFA_CP) FC2 k-tile t2 scale atoms: word [r][jj] of atom kb <- raw atom (rows / 128, K block 4 t2 + kb) of slot + jj's expert, word [r][blk % 4].""" + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + src0 = ( + ( + (e * cutlass.Int32(SF2_RB) + (blk >> cutlass.Int32(2))) * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (blk & cutlass.Int32(3)) + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + jj)), + s2w.subview(src0 + cutlass.Int32(kb * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _shared_cta( + tidx, + warp, + lane, + sid, + s_big: cutlass.Array, + sh_bar: cutlass.Array, + epochs: cutlass.Array, + tdbg: cutlass.Array, + bx, + xb32: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, + tmem_ptr_i32: cutlass.Array, + gu8: cutlass.Array, + dn8: cutlass.Array, +): + """Shared CTA sid (see SHARED): tcgen05 MXF8 (E4M3 x E4M3, UE8M0 per 32 K), m3_mx_gemv's tile order. gate_up: one + 128-row MMA tile = 2 K slices x (32 gate + 32 up rows), B rows 8 q + t = token t's slice-q k-tile; down: 2 tiles + of 128 rows, full K. Warps: 0 weights (TMA), 1 TMEM + scale splice + MMA, 4-7 epilogue; 2.. activation quant.""" + big32 = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Int32, alignment=16) + bigf = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Float32, alignment=16) + big8 = cutlass.Array(s_big.data_ptr(0), shape=(BIG,), dtype=cutlass.Int8, alignment=16) + sh_full = sh_bar # [0, SH_NST): stage full + sh_empty = cutlass.Array(sh_bar.data_ptr(8), shape=(8,), dtype=cutlass.Int64, alignment=8) + b_gu_rdy = sh_bar.subview(16) + b_dn_rdy = sh_bar.subview(17) + acc_gu = sh_bar.subview(18) + acc_dn = sh_bar.subview(19) + if warp == 0: + prims.prefetch_tensormap(tma_gu) + prims.prefetch_tensormap(tma_dn) + if lane < SH_NST: + prims.mbarrier_init(sh_full.subview(lane), 1) + prims.mbarrier_init( + sh_empty.subview(lane), 33 + ) # the MMAs' commit + each splicer lane's reads done + if lane == 0: + prims.mbarrier_init(b_gu_rdy, 1) + prims.mbarrier_init(b_dn_rdy, 1) + prims.mbarrier_init(acc_gu, 1) + prims.mbarrier_init(acc_dn, 1) + if warp == 1: + prims.tcgen05_alloc(tmem_ptr_i32, SH_TMEM) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + if warp == 0: + for g in cutlass.range_constexpr(SH_NST): # layer constants: before the grid dependency + _sh_issue(lane, sid, cutlass.Int32(g), s_big, sh_full, tma_gu, tma_dn, gusf, dnsf) + # the rest of this CTA's weights into L2 (64 gate / up rows of 6 KB, 256 down rows of 768 B) + if cutlass.const_expr(SH_PRETILE): + # this CTA's stages beyond the ring, 4 per lane + for r in cutlass.range_constexpr(2): + pg = cutlass.Int32(SH_NST) + lane * cutlass.Int32(2) + cutlass.Int32(r) + if pg < cutlass.Int32(SH_STAGES): + _l2_prefetch( + gusf.subview( + (sid * cutlass.Int32(SH_STAGES) + pg) * cutlass.Int32(SH_STAGE_BYTES) + ) + .data_ptr() + .toint(), + SH_STAGE_BYTES, + ) + else: + for r in cutlass.range_constexpr(2): + prw = sid * cutlass.Int32(32) + cutlass.Int32(r * SH_I) + lane + _l2_prefetch(gu8.subview(prw * cutlass.Int32(H)).data_ptr().toint(), H) + if lane == 0: + _l2_prefetch( + dn8.subview(sid * cutlass.Int32(256 * SH_I)).data_ptr().toint(), 256 * SH_I + ) + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(not LATE_TRIGGER): + if arp.load(idx=0) == cutlass.Int64(0): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + NQ = THREADS - 64 # activation quant threads (warps 2 ..) + if warp == 0: + for g in cutlass.range(SH_NST, SH_STAGES, unroll=1): + _wait( + sh_empty.subview(g % cutlass.Int32(SH_NST)), + ((g // cutlass.Int32(SH_NST)) - cutlass.Int32(1)) & cutlass.Int32(1), + ) + _sh_issue(lane, sid, g, s_big, sh_full, tma_gu, tma_dn, gusf, dnsf) + elif warp == 1: + # ---- MMA: gate_up (scale atoms spliced to the tile's row order), then down + idesc_gu = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=16, + m_dim=128, + ) + idesc_dn = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=8, + m_dim=128, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + desc_a0 = prims.Tcgen05SmemDesc.build( + s_big.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bx = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BX), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bh = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BH), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + sq = sid & cutlass.Int32(3) + _wait(b_gu_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for g in cutlass.range(SH_STAGES, unroll=1): + st = g % cutlass.Int32(SH_NST) + if g == cutlass.Int32(SH_GT): + _wait(b_dn_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + _wait(sh_full.subview(st), (g // cutlass.Int32(SH_NST)) & cutlass.Int32(1)) + raw = cutlass.Int32(_SB_RAW // 4) + st * cutlass.Int32(512) + if (g < cutlass.Int32(SH_GT)) & cutlass.Boolean(not SH_PRETILE): + # MMA row 32 Q + i = quadrant Q's row i: word 4 i + Q <- raw atom Q's word 4 i + sid % 4 + w4 = [ + big32.load(idx=raw + cutlass.Int32(128 * qq) + lane * cutlass.Int32(4) + sq) + for qq in range(4) + ] + big32.store( + ( + cutlass.Int32(w4[0]), + cutlass.Int32(w4[1]), + cutlass.Int32(w4[2]), + cutlass.Int32(w4[3]), + ), + idx=cutlass.Int32(_SB_SPL // 4) + + st * cutlass.Int32(128) + + lane * cutlass.Int32(4), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + prims.mbarrier_arrive( + sh_empty.subview(st) + ) # this lane's generic reads of the slot are done + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + if g < cutlass.Int32(SH_GT): + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bx + cutlass.Int64(g * cutlass.Int32(2048 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr(tcol + SH_ACC_GU, 6, cutlass.Float32), + da + 2 * kb, + db + 2 * kb, + idesc_gu.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (g > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_GU + g * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_GT - 1): + prims.tcgen05_commit(acc_gu) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(6) + kt = gd - jt * cutlass.Int32(6) + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview( + cutlass.Int32(_SB_SPL if SH_PRETILE else _SB_RAW) + + st * cutlass.Int32(512 if SH_PRETILE else 2048) + ), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bh + cutlass.Int64(kt * cutlass.Int32(1024 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr( + tcol + SH_ACC_DN + jt * cutlass.Int32(8), 6, cutlass.Float32 + ), + da + 2 * kb, + db + 2 * kb, + idesc_dn.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (kt > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_DN + kt * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_STAGES - 1): + prims.tcgen05_commit(acc_dn) + cute.arch.sync_warp() + if warp >= 2: + # ---- the activation, MXFP8 per 32-K block (native mxfp8_quantize), into B (row 8 q + t of stage tile i = + # token t's k-tile SH_GT q + i, 128-byte swizzle) and its codes + qt = tidx - cutlass.Int32(64) + for r in cutlass.range_constexpr((M_MAX * SH_KB + NQ - 1) // NQ): + q_ = qt + cutlass.Int32(r * NQ) + if q_ < cutlass.Int32(M_MAX * SH_KB): + t = q_ // cutlass.Int32(SH_KB) + b = q_ - t * cutlass.Int32(SH_KB) + wv = [ + xb32.load( + idx=t * cutlass.Int32(H // 2) + + b * cutlass.Int32(16) + + cutlass.Int32(4 * z), + vector_size=4, + alignment=16, + ) + for z in range(4) + ] + v = [] + for z in cutlass.range_constexpr(4): + for zz in cutlass.range_constexpr(4): + w = cutlass.Int32(wv[z][zz]) + v.append((w << cutlass.Int32(16)).bitcast(cutlass.Float32)) + v.append((w & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(v[z])) + code, osc = _mx_osc(amax) + words = _mx_words(v, osc) + kt = b >> cutlass.Int32(2) + sl = kt // cutlass.Int32(SH_GT) + it = kt - sl * cutlass.Int32(SH_GT) + n = sl * cutlass.Int32(8) + t + c0 = (b & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BX) + it * cutlass.Int32(2048) + n * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + + (((c0 + cutlass.Int32(1)) ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_XC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR, number_of_threads=NQ) + if warp >= 4: + if warp < 8: + qd = warp - cutlass.Int32(4) + # B scales of gate_up to TMEM: column of stage i, lane n = 8 q + t: token t's codes of k-tile SH_GT q + i + n = lane + t = n & cutlass.Int32(7) + ok = (n < cutlass.Int32(16)) & (t < cutlass.Int32(M_MAX)) + for i in cutlass.range_constexpr(SH_GT): + kt = (n >> cutlass.Int32(3)) * cutlass.Int32(SH_GT) + cutlass.Int32(i) + wsf = big32.load( + idx=( + cutlass.Int32(_SB_XC) + + cutlass.select_(ok, t, cutlass.Int32(0)) * cutlass.Int32(SH_KB) + + kt * cutlass.Int32(4) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_GU + i * 4), + cutlass.select_(ok, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + et = tidx - cutlass.Int32(128) + if et == 0: + prims.mbarrier_arrive(b_gu_rdy) + _stamp(tdbg, bx, 54) + # ---- gate_up out: quadrant qd lane l = MMA row 32 qd + l; slice q = qd / 2 holds token t at column 8 q + t + _wait(acc_gu, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + av = prims.tcgen05_ld( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_ACC_GU, cutlass.Float32), + num=16, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(2): + for tt in cutlass.range_constexpr(8): + bigf.store( + cutlass.Float32(av[8 + tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(2)) * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd < cutlass.Int32(2): + # m3_mx_gemv's owner sum: (0 + rank 0) + rank 1, then bf16 + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + # h = silu_and_mul(gate, up) of positions 32 sid + p, published under the call's tag + for r in cutlass.range_constexpr((32 * M_MAX + 127) // 128): + pq = et + cutlass.Int32(r * 128) + if pq < cutlass.Int32(32 * M_MAX): + p = pq // cutlass.Int32(M_MAX) + tt = pq - p * cutlass.Int32(M_MAX) + ga = bigf.load(idx=cutlass.Int32(_SB_GU // 4) + p * cutlass.Int32(8) + tt) + up = bigf.load( + idx=cutlass.Int32(_SB_GU // 4) + + (cutlass.Int32(32) + p) * cutlass.Int32(8) + + tt + ) + q_a = cute.arch.fmin(cutlass.Float32(ga), cutlass.Float32(7.0)) + q_bu = _min_xorsign_abs(cutlass.Float32(up), cutlass.Float32(7.0)) + q_ub = _add_rn(q_bu, cutlass.Float32(1.0)) + q_e2 = _ex2_full( + _mul_rn( + _mul_rn(q_a, cutlass.Float32(-1.70200002193450927734)), + cutlass.Float32(1.4426950216293334961), + ) + ) + q_sg = _div_full(cutlass.Float32(1.0), _add_rn(q_e2, cutlass.Float32(1.0))) + q_h = _mul_rn(q_ub, _mul_rn(q_sg, q_a)) + _st_relaxed_u32( + hx.subview(tt * cutlass.Int32(SH_I) + sid * cutlass.Int32(32) + p) + .data_ptr() + .toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(q_h), + ) + if et == 0: + _stamp(tdbg, bx, 50) + # every shared CTA's h: (token, K block) per thread, 32 tagged words, MXFP8 into B (h) and its codes + for rr in cutlass.range_constexpr((M_MAX * SH_DKB + 127) // 128): + q_ = et + cutlass.Int32(rr * 128) + if q_ < cutlass.Int32(M_MAX * SH_DKB): + t = q_ // cutlass.Int32(SH_DKB) + kb = q_ - t * cutlass.Int32(SH_DKB) + base = t * cutlass.Int32(SH_I) + kb * cutlass.Int32(32) + hv = [cutlass.Int32(0) for _ in range(32)] + pend = cutlass.Int32(1) + while pend != cutlass.Int32(0): + for q4 in cutlass.range_constexpr(8): + v4 = prims.load_ext( + hx.subview(base + cutlass.Int32(4 * q4)), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for z in cutlass.range_constexpr(4): + hv[4 * q4 + z] = cutlass.Int32(v4[z]) + okw = cutlass.Boolean(True) + for z in cutlass.range_constexpr(32): + okw = okw & ( + ((hv[z] >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) == tag16 + ) + pend = cutlass.select_(okw, cutlass.Int32(0), cutlass.Int32(1)) + vals = [ + (hv[z] << cutlass.Int32(16)).bitcast(cutlass.Float32) for z in range(32) + ] + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(vals[z])) + code, osc = _mx_osc(amax) + words = _mx_words(vals, osc) + kt = kb >> cutlass.Int32(2) + c0 = (kb & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BH) + kt * cutlass.Int32(1024) + t * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + (((c0 + cutlass.Int32(1)) ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_HC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + okh = lane < cutlass.Int32(M_MAX) + for kt in cutlass.range_constexpr(6): + wsf = big32.load( + idx=( + cutlass.Int32(_SB_HC) + + cutlass.select_(okh, lane, cutlass.Int32(0)) * cutlass.Int32(SH_DKB) + + cutlass.Int32(4 * kt) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_DN + kt * 4), + cutlass.select_(okh, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if et == 0: + prims.mbarrier_arrive(b_dn_rdy) + _stamp(tdbg, bx, 51) + # ---- down out: tile j row 32 qd + l -> y_sh row 256 sid + 128 j + 32 qd + l + _wait(acc_dn, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for jt in cutlass.range_constexpr(2): + dv = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_DN + jt * 8, + cutlass.Float32, + ), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + rowo = ( + sid * cutlass.Int32(256) + + cutlass.Int32(128 * jt) + + qd * cutlass.Int32(32) + + lane + ) + for tt in cutlass.range_constexpr(M_MAX): + _st_relaxed_u32( + dnx.subview(cutlass.Int32(tt * H) + rowo).data_ptr().toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(cutlass.Float32(dv[tt])), + ) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if et == 0: + _stamp(tdbg, bx, 52) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if warp == 1: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc(cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), SH_TMEM) + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + + +@cute.kernel +def m3_moe_kernel( + tma_a1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_a2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s1_desc: cutlass.GridConstant[cuda.TensorMap], # w3_w1_weight_scale as 128-byte lines + tma_s2_desc: cutlass.GridConstant[cuda.TensorMap], # w2_weight_scale as 128-byte lines + tma_gu_desc: cutlass.GridConstant[ + cuda.TensorMap + ], # SHARED: gate_up e4m3 [1536, 6144], box 128 B x 32 rows + tma_dn_desc: cutlass.GridConstant[ + cuda.TensorMap + ], # SHARED: down e4m3 [6144, 768], box 128 B x 128 rows + logits: cutlass.Array, # f32 [M, 128] (not FUSED_ROUTER) + bias: cutlass.Array, # f32 [128] + x32: cutlass.Array, # int32 words of x_fp4 [M, 3072 B] (not FUSED_ROUTER) + xsf32: cutlass.Array, # int32 words of x_sf [M, 384 B] (not FUSED_ROUTER) + escal: cutlass.Array, # f32 [6, 128]: fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha + plan: cutlass.Array, # int32 [4 M + 1, NCTA, NG_CAP + 1] by G: per group i0 | i1 << 16, then b0 | nb << 16 + partials: cutlass.Array, # int32 bits of f32 [G_CAP * 48, M, 32]: split units' head sums + hbuf32: cutlass.Array, # int32 [G_CAP * M, 48, 4]: tagged intermediate records + epochs: cutlass.Array, # int32 [NCTA] + out: cutlass.Array, # bf16 [M, 6144] + route_dbg: cutlass.Array, # int32 [M, 8]: ids, weight bits (CTA 0) + tdbg: cutlass.Array, # int64 [NCTA, 32] timing stamps (TIMING builds) + xb32: cutlass.Array, # FUSED_ROUTER: int32 words (bf16 pairs) of the hidden states [M, 6144] + rw: cutlass.Array, # FUSED_ROUTER: f32 router weight [128, 6144] + sfsc: cutlass.Array, # FUSED_ROUTER: f32 [1] fc31_input_scale + lgbuf: cutlass.Array, # FUSED_ROUTER: int64 [LG_REPL, LG_STRIDE] logits: tag << 32 | f32 bits + guw: cutlass.Array, # SHARED: uint8 gate_up e4m3 [1536, 6144] + gusf: cutlass.Array, # SHARED: uint8 its swizzled UE8M0 scales + dnw: cutlass.Array, # SHARED: uint8 down e4m3 [6144, 768] + dnsf: cutlass.Array, # SHARED: uint8 its swizzled UE8M0 scales + hx: cutlass.Array, # SHARED: int32 [M, 768]: h as (call tag << 16 | bf16), tag 0 = never written + dnx: cutlass.Array, # SHARED: int32 [M, 6144]: y_sh as (call tag << 16 | bf16), tag 0 = never written + arp: cutlass.Array, # int64 [8]: all-reduce push (enable, mc, flags, rank, world, maxt); enable 0 = off + s1w: cutlass.Array, # SFA_CP: int32 words of w3_w1_weight_scale + s2w: cutlass.Array, # SFA_CP: int32 words of w2_weight_scale + img32: cutlass.Array, # B2IMG: int32 [NG_CAP, IMG_BYTES / 4] per-group FC2 B image (tile + scale atoms) + b2cnt: cutlass.Array, # B2IMG: int32 [NG_CAP * 8 + 8] per-group unit counters (index 8 g) +): + tidx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tidx % 32 + if tidx == 0: + _stamp(tdbg, bx, 0) + if cutlass.const_expr(TIMING): + if tidx == 0: + tdbg.store( + cutlass.Int64(_smid()), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(63) + ) + + s_big = cutlass.Array(cutlass.Int8, BIG, space=cutlass.AddressSpace.smem, alignment=1024) + sA = cutlass.Array( + s_big.data_ptr(0), shape=(NUM_BYTES_A * STAGES,), dtype=cutlass.Int8, alignment=1024 + ) + sSFAraw = cutlass.Array( + s_big.data_ptr(_OFF_SFAR), + shape=(max(1, _RAW_STAGE // 4) * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sSFA = cutlass.Array( + s_big.data_ptr(_OFF_SFA), + shape=(NUM_WORDS_SFA * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sB1 = cutlass.Array( + s_big.data_ptr(_OFF_B1), shape=(NUM_BYTES_B1 * S1,), dtype=cutlass.Int8, alignment=1024 + ) + sB2 = cutlass.Array( + s_big.data_ptr(_OFF_B2), shape=(2 * KT2 * NUM_BYTES_B2,), dtype=cutlass.Int8, alignment=1024 + ) + sSFB2 = cutlass.Array( + s_big.data_ptr(_OFF_SFB2), + shape=(2 * KT2 * NUM_KB * 128,), + dtype=cutlass.Int32, + alignment=1024, + ) + ys = cutlass.Array( + cutlass.Float32, NB2_CAP * M_MAX * TOP_K * R2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_part = cutlass.Array( + cutlass.Float32, 2 * M_MAX * C1 * R1, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_bias = cutlass.Array(cutlass.Float32, E_LOCAL, space=cutlass.AddressSpace.smem, alignment=16) + s_red = cutlass.Array(cutlass.Float32, 9 * M_MAX, space=cutlass.AddressSpace.smem, alignment=16) + s_xsf = cutlass.Array( + cutlass.Int32, + M_MAX * (H // 64) if FUSED_ROUTER else 4, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + ab_full = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + ab_empty = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + sfa_ready = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + acc_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_ready = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_free = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + # this CTA's FC1 share of group g is out: one barrier per group parity (a single barrier could complete twice + # while warp 5 still works on an earlier group, and its parity wait would then miss them) + fc1_done = cutlass.Array(cutlass.Int64, 4, space=cutlass.AddressSpace.smem, alignment=8) + b1_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + x_bar = cutlass.Array( + cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8 + ) # x landed, x consumed + img_bar = cutlass.Array( + cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8 + ) # (B2IMG) image landed + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + s_el = cutlass.Array( + cutlass.Int32, G_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16 + ) # slots, then G + s_kpos = cutlass.Array( + cutlass.Int32, G_CAP * M_MAX, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_w = cutlass.Array( + cutlass.Float32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_ids = cutlass.Array( + cutlass.Int32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_plan = cutlass.Array(cutlass.Int32, NG_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + # this CTA's plan rows for every G (static data: loaded before the grid dependency) + s_misc = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_plan_all = cutlass.Array( + cutlass.Int32, (G_CAP + 1) * (NG_CAP + 1), space=cutlass.AddressSpace.smem, alignment=16 + ) + s_esc = cutlass.Array( + cutlass.Float32, 8 * G_CAP, space=cutlass.AddressSpace.smem, alignment=32 + ) # slot s at 8 s + + is_routed = cutlass.Boolean(True) + if cutlass.const_expr(SHARED): + sh_full = cutlass.Array(cutlass.Int64, 32, space=cutlass.AddressSpace.smem, alignment=8) + is_routed = bx < cutlass.Int32(NCTA) + if bx >= cutlass.Int32(NCTA): + _shared_cta( + tidx, + warp, + lane, + bx - cutlass.Int32(NCTA), + s_big, + sh_full, + epochs, + tdbg, + bx, + xb32, + tma_gu_desc.get_ptr(), + tma_dn_desc.get_ptr(), + gusf, + dnsf, + hx, + dnx, + arp, + tmem_ptr_i32, + guw, + dnw, + ) + if is_routed: + # ---- before the grid dependency + if warp == 0: + if tidx < STAGES: + prims.mbarrier_init(ab_full.subview(tidx), 33 if SFA_CP else 1) + prims.mbarrier_init(ab_empty.subview(tidx), 1) + prims.mbarrier_init(sfa_ready.subview(tidx), 1) + if tidx < 2: + prims.mbarrier_init(acc_full.subview(tidx), 1) + prims.mbarrier_init(acc_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(acc2_full.subview(tidx), 1) + prims.mbarrier_init(acc2_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(b2_ready.subview(tidx), 1) + prims.mbarrier_init(b2_free.subview(tidx), EPI_THREADS) + if cutlass.const_expr(B2IMG): + prims.mbarrier_init(img_bar.subview(tidx), 1) + if tidx < 4: + prims.mbarrier_init(fc1_done.subview(tidx), EPI_THREADS) + if tidx == 0: + prims.mbarrier_init(b1_ready.subview(0), EPI_THREADS) + if cutlass.const_expr(XSTAGE): + prims.mbarrier_init(x_bar.subview(0), 1) + prims.mbarrier_init(x_bar.subview(1), EPI_THREADS) + if warp == MMA_WARP: + prims.tcgen05_alloc(tmem_ptr_i32, TMEM_COLS) + prims.tcgen05_relinquish_alloc_permit() + if warp == W_WARP: + prims.prefetch_tensormap(tma_a1_desc.get_ptr()) + prims.prefetch_tensormap(tma_a2_desc.get_ptr()) + prims.prefetch_tensormap(tma_s1_desc.get_ptr()) + prims.prefetch_tensormap(tma_s2_desc.get_ptr()) + for i in cutlass.range_constexpr(E_LOCAL // 32): + s_bias.store(cutlass.Float32(bias.load(idx=lane + i * 32)), idx=lane + i * 32) + for i in cutlass.range_constexpr(((G_CAP + 1) * (NG_CAP + 1) + 31) // 32): + q = lane + cutlass.Int32(32 * i) + if q < cutlass.Int32((G_CAP + 1) * (NG_CAP + 1)): + gq = q // cutlass.Int32(NG_CAP + 1) + s_plan_all.store( + plan.load( + idx=(gq * cutlass.Int32(NCTA) + bx) * cutlass.Int32(NG_CAP + 1) + + q + - gq * cutlass.Int32(NG_CAP + 1) + ), + idx=q, + ) + rwv = [] + if cutlass.const_expr(FUSED_ROUTER): + tg = cutlass.select_(tidx < cutlass.Int32(GEMV_THREADS), tidx, cutlass.Int32(0)) + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + rbx = cutlass.select_(eg < cutlass.Int32(NROUTE), eg, cutlass.Int32(0)) + rwv.append( + [ + rw.load( + idx=rbx * cutlass.Int32(H) + + tg * cutlass.Int32(4) + + cutlass.Int32(1024 * j), + vector_size=4, + alignment=16, + ) + for j in range(H // 1024) + ] + ) + if cutlass.const_expr(TIMING): + if tidx == 0: + s_el.store(cutlass.Int32(-7), idx=G_CAP) + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(XSTAGE): + if warp == 0: + if lane == 0: + prims.mbarrier_arrive_expect_tx(x_bar.subview(0), M_MAX * H * 2) + cute.arch.sync_warp() + if lane < M_MAX: + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(lane * cutlass.Int32(H * 2)), + xb32.subview(lane * cutlass.Int32(H // 2)), + x_bar.subview(0), + H * 2, + ) + ar_on = arp.load(idx=0) != cutlass.Int64(0) + ar_off = arp.load(idx=0) == cutlass.Int64(0) + if cutlass.const_expr(not LATE_TRIGGER): + if ar_off: + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + + ep = epochs.load(idx=bx) + tagv = ((ep & cutlass.Int32(0x7F)) | cutlass.Int32(0x80)) << cutlass.Int32(24) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + ltag = cutlass.Int64(ep + cutlass.Int32(1)) << cutlass.Int64(32) + if cutlass.const_expr(FUSED_ROUTER): + # ---- router GEMV (experts bx, bx + NCTA, .., threads 0..255), m3_router_gemv's order exactly + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + if (eg < cutlass.Int32(NROUTE)) & (tidx < cutlass.Int32(GEMV_THREADS)): + accs = [cutlass.Float32(0.0) for _ in range(M_MAX)] + xsrc = xb32 + if cutlass.const_expr(GEMV_SMEM and XSTAGE): + _wait(x_bar.subview(0), 0) + xsrc = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + for j in cutlass.range_constexpr(H // 1024): + for t in cutlass.range_constexpr(M_MAX): + xw = xsrc.load( + idx=cutlass.Int32(t * (H // 2) + 512 * j) + tidx * cutlass.Int32(2), + vector_size=2, + alignment=8, + ) + x0 = cutlass.Int32(xw[0]) + x1 = cutlass.Int32(xw[1]) + a = accs[t] + a = _fma_rn( + (x0 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][0]), + a, + ) + a = _fma_rn( + (x0 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][1]), + a, + ) + a = _fma_rn( + (x1 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][2]), + a, + ) + a = _fma_rn( + (x1 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][3]), + a, + ) + accs[t] = a + for t in cutlass.range_constexpr(M_MAX): + v = accs[t] + for off in (16, 8, 4, 2, 1): + v = _add_rn(v, cute.arch.shuffle_sync_bfly(v, off)) + if lane == 0: + s_red.store(v, idx=warp * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < M_MAX: + sacc = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(8): + sacc = _add_rn(sacc, s_red.load(idx=cutlass.Int32(w * M_MAX) + tidx)) + s_red.store(sacc, idx=cutlass.Int32(8 * M_MAX) + tidx) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < LG_REPL: + for t in cutlass.range_constexpr(M_MAX): + lw = ltag | ( + cutlass.Int64( + cutlass.Float32(s_red.load(idx=8 * M_MAX + t)).bitcast( + cutlass.Int32 + ) + ) + & cutlass.Int64(0xFFFFFFFF) + ) + _st_relaxed_u64( + lgbuf.subview( + tidx * cutlass.Int32(LG_STRIDE) + + cutlass.Int32(t * E_LOCAL) + + eg + ) + .data_ptr() + .toint(), + lw, + ) + if cutlass.const_expr(NGEMV > 1): + cute.arch.sync_warp() + cute.arch.barrier( + barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS + ) # s_red reused next + if tidx == 0: + _stamp(tdbg, bx, 21) + + # ---- routing (warps 4..7, token t on warp 4 + t % 4); the epilogue warps build B meanwhile + if ((warp >= W_WARP) & (warp < cutlass.Int32(W_WARP + NRW0))) | ( + (warp >= cutlass.Int32(8)) & (warp < cutlass.Int32(8 + RW_EXTRA)) + ): + rw_i = cutlass.select_( + warp < cutlass.Int32(8), warp - cutlass.Int32(W_WARP), warp - cutlass.Int32(4) + ) + # ILP: poll every token's logits first, then run the independent top-k chains back to back + lgv_all = [] + t_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = rw_i + cutlass.Int32(NRW * tr) + if cutlass.const_expr(M_MAX % NRW != 0): + t = cutlass.select_(t < cutlass.Int32(M_MAX), t, cutlass.Int32(M_MAX - 1)) + t_all.append(t) + lgv = [] + if cutlass.const_expr(FUSED_ROUTER): + want = ltag >> cutlass.Int64(32) + rbase = ( + (bx % cutlass.Int32(LG_REPL)) * cutlass.Int32(LG_STRIDE) + + t * cutlass.Int32(E_LOCAL) + + lane + ) + done = cutlass.Int32(0) + res = [cutlass.Int64(0) for _ in range(4)] + while done == cutlass.Int32(0): + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(4): + res[z] = _ld_relaxed_u64( + lgbuf.subview(cutlass.Int32(z * 32) + rbase).data_ptr().toint() + ) + ok = ok & ((res[z] >> cutlass.Int64(32)) == want) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32( + -1 + ): + done = cutlass.Int32(1) + else: + _nanosleep(POLL_NS) + for z in cutlass.range_constexpr(4): + lgv.append(cutlass.Int32(res[z]).bitcast(cutlass.Float32)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 22) + else: + for i in cutlass.range_constexpr(4): + lgv.append( + cutlass.Float32( + logits.load( + idx=t * cutlass.Int32(E_LOCAL) + cutlass.Int32(i * 32) + lane + ) + ) + ) + lgv_all.append(lgv) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 61) + res_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = t_all[tr] + lgv = lgv_all[tr] + own = (rw_i + cutlass.Int32(NRW * tr)) < cutlass.Int32(M_MAX) + # top-4 (keys sigmoid + bias, ties to the lower id) + keys = [] + for i in cutlass.range_constexpr(4): + sg = _fma_rn( + cutlass.Float32(0.5), + cute.math.tanh(_mul_rn(cutlass.Float32(0.5), lgv[i])), + cutlass.Float32(0.5), + ) + keys.append(_add_rn(sg, s_bias.load(idx=i * 32 + lane))) + hi = [_twiddle(keys[i]) for i in range(4)] + taken = [cutlass.Boolean(False) for _ in range(4)] + ids_t = [] + kt_t = [] + for k in cutlass.range_constexpr(TOP_K): + bh = cutlass.Int32(0) + bl = cutlass.Int32(0) + for i in cutlass.range_constexpr(4): + lo_i = cutlass.Int32(65535 - i * 32) - lane + hv = cutlass.select_(taken[i], cutlass.Int32(0), hi[i]) + lv = cutlass.select_(taken[i], cutlass.Int32(0), lo_i) + better = (cutlass.Uint32(hv) > cutlass.Uint32(bh)) | ( + (hv == bh) & (cutlass.Uint32(lv) > cutlass.Uint32(bl)) + ) + bh = cutlass.select_(better, hv, bh) + bl = cutlass.select_(better, lv, bl) + mh = _redux_max_u32(bh) + ml = _redux_max_u32(cutlass.select_(bh == mh, bl, cutlass.Int32(0))) + e_k = cutlass.Int32(65535) - ml + for i in cutlass.range_constexpr(4): + taken[i] = taken[i] | (e_k == cutlass.Int32(i * 32) + lane) + ids_t.append(e_k) + kv = cutlass.Float32(0.0) + for i in cutlass.range_constexpr(4): + kv = cutlass.select_(e_k == cutlass.Int32(i * 32) + lane, keys[i], kv) + kt_t.append(cute.arch.shuffle_sync(kv, e_k % cutlass.Int32(32))) + sig = cutlass.Float32(0.0) + e_l = cutlass.Int32(0) + for k in cutlass.range_constexpr(TOP_K): + sk = _sub_rn(kt_t[k], s_bias.load(idx=ids_t[k])) + sig = cutlass.select_(lane == cutlass.Int32(k), sk, sig) + e_l = cutlass.select_(lane == cutlass.Int32(k), ids_t[k], e_l) + total = sig + for off in (16, 8, 4, 2, 1): + total = _add_rn(total, cute.arch.shuffle_sync_bfly(total, off)) + wv = _bf16_round( + _div_rn( + _mul_rn(sig, cutlass.Float32(ROUTE_SCALE)), + _add_rn(total, cutlass.Float32(SUM_EPS)), + ) + ) + res_all.append((t, own, wv, e_l)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 62) + for tr in cutlass.range_constexpr( + (M_MAX + NRW - 1) // NRW + ): # stores after every chain (one branch-free block) + t, own, wv, e_l = res_all[tr] + if (lane < TOP_K) & own: + s_w.store(wv, idx=t * cutlass.Int32(TOP_K) + lane) + s_ids.store(e_l, idx=t * cutlass.Int32(TOP_K) + lane) + if bx == 0: + route_dbg.store( + wv.bitcast(cutlass.Int32), + idx=t * cutlass.Int32(8) + cutlass.Int32(4) + lane, + ) + cute.arch.sync_warp() + if cutlass.const_expr(EARLY0 and not E0_OFFLOAD): + if warp == cutlass.Int32(W_WARP): + # slots 0..3 are always token 0's four (distinct) experts and group 0's stage ranges are the same + # for every G (the plan guarantees it): fill the ring with this CTA's first group-0 stages now + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(W_WARP): + _bar_arrive( + E0_BAR, 64 + ) # token 0's experts are in s_ids: warp 7 issues the early stages + if cutlass.const_expr(B1_AFTER_TOPK): + _bar_arrive(TOPK2_BAR, 128 + 32 * NRW) + cute.arch.barrier(barrier_id=TOPK_BAR, number_of_threads=32 * NRW) + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(E0_WARP): + cute.arch.barrier(barrier_id=E0_BAR, number_of_threads=64) + # slots 0..3 are always token 0's four (distinct) experts and group 0's stage ranges are the same + # for every G (the plan guarantees it): fill the ring with this CTA's first group-0 stages now + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + _bar_arrive(E1_BAR, 64) + if warp == W_WARP: + if lane == 0: + _stamp(tdbg, bx, 20) + # slots: lane l < 4 M is (token l / 4, top-k l % 4); the distinct experts in lane order (match.any) + valid = lane < cutlass.Int32(TOP_K * M_MAX) + my_e = cutlass.select_( + valid, + s_ids.load(idx=cutlass.select_(valid, lane, cutlass.Int32(0))), + cutlass.Int32(-1) - lane, + ) + same = _match_any(my_e) + leader = cute.arch.popc((same & (cutlass.Int32(0) - same)) - cutlass.Int32(1)) + first = valid & (lane == leader) + bal = prims.vote_sync(0xFFFFFFFF, first, prims.VoteSync.BALLOT) + rank = cute.arch.popc(bal & cutlass.Int32(cute.arch.lanemask_lt())) + slot_l = cute.arch.shuffle_sync(rank, leader) + for z in cutlass.range_constexpr((G_CAP * M_MAX + 31) // 32): + if lane + cutlass.Int32(32 * z) < cutlass.Int32(G_CAP * M_MAX): + s_kpos.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + cute.arch.sync_warp() + if first: + s_el.store(my_e, idx=rank) + if valid: + s_kpos.store( + lane % cutlass.Int32(TOP_K), + idx=slot_l * cutlass.Int32(M_MAX) + lane // cutlass.Int32(TOP_K), + ) + if bx == 0: + route_dbg.store( + my_e, + idx=(lane // cutlass.Int32(TOP_K)) * cutlass.Int32(8) + + lane % cutlass.Int32(TOP_K), + ) + gcnt = cute.arch.popc(bal) + if lane == 0: + s_el.store(gcnt, idx=G_CAP) + if lane < cutlass.Int32(NG_CAP + 1): # the plan row of this G (preloaded) + s_plan.store(s_plan_all.load(idx=gcnt * cutlass.Int32(NG_CAP + 1) + lane), idx=lane) + cute.arch.sync_warp() + if cutlass.const_expr(B2IMG): + # CTA 0 pads every group counter to exactly CNT_PER_CALL for this call (absent slots, unused groups) + if (bx == 0) & (lane < cutlass.Int32(NG_CAP)): + nh = gcnt - lane * cutlass.Int32(4) + nh = cutlass.select_(nh > cutlass.Int32(4), cutlass.Int32(4), nh) + nh = cutlass.select_(nh < cutlass.Int32(0), cutlass.Int32(0), nh) + pad = cutlass.Int32(CNT_PER_CALL) - nh * cutlass.Int32(UNITS_ROW) + if pad > cutlass.Int32(0): + _red_release_add( + b2cnt.subview(lane * cutlass.Int32(8)).data_ptr().toint(), pad + ) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 24) + cute.arch.sync_warp() + _bar_arrive(ROUTE_BAR, THREADS) + if warp < 4: + if cutlass.const_expr(B1_AFTER_TOPK): + cute.arch.barrier(barrier_id=TOPK2_BAR, number_of_threads=128 + 32 * NRW) + if cutlass.const_expr(FUSED_ROUTER): + if tidx == 0: + _stamp(tdbg, bx, 57) + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(0), 0) + if tidx == 0: + _stamp(tdbg, bx, 58) + xs32 = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xs32, sfsc, sB1, s_xsf, b1_ready + ) + prims.mbarrier_arrive(x_bar.subview(1)) # the ring is free for the weights + else: + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xb32, sfsc, sB1, s_xsf, b1_ready + ) + if tidx == 0: + _stamp(tdbg, bx, 23) + else: + _build_b1(tidx, warp, lane, base_row, base_col, x32, xsf32, sB1, b1_ready) + if warp != W_WARP: # one sync site for every thread but warp 4's (which arrived) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ROUTE_BAR, number_of_threads=THREADS) + if tidx == 0: + _stamp(tdbg, bx, 2) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(s_el.load(idx=G_CAP)), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(28), + ) + + g_n = s_el.load(idx=G_CAP) + ng = (g_n + cutlass.Int32(3)) >> cutlass.Int32(2) + pr2 = s_plan.load(idx=NG_CAP) + blk0 = pr2 & cutlass.Int32(0xFFFF) + nb = pr2 >> cutlass.Int32(16) + + # ---- weights producer (warp 4): phase p = FC1 stages of group p, then FC2 tiles of group p - LAG + if warp == W_WARP: + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) # the staged hidden states are quantized + stage = cutlass.Int32(0) + phase = cutlass.Int32(1) + skip = cutlass.Int32(0) + if cutlass.const_expr(E0_OFFLOAD): + cute.arch.barrier(barrier_id=E1_BAR, number_of_threads=64) + if cutlass.const_expr(EARLY0): + skip = s_misc.load(idx=0) # group-0 stages already in the ring + stage = cutlass.select_(skip == cutlass.Int32(STAGES), cutlass.Int32(0), skip) + phase = cutlass.select_( + skip == cutlass.Int32(STAGES), cutlass.Int32(0), cutlass.Int32(1) + ) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_el.load(idx=slot) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if skip > cutlass.Int32(0): + skip = skip - cutlass.Int32(1) + else: + _waitd(ab_empty.subview(stage), phase, tdbg, 1) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + ), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, stage, sf_line, c, sSFA, s1w, ab_full) + stage, phase = _adv(stage, phase) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + for bb in range(nb): + blk = blk0 + bb + for t2 in cutlass.range_constexpr(KT2): + _waitd(ab_empty.subview(stage), phase, tdbg, 2) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + n_here + * cutlass.Int32( + NUM_BYTES_BOX1 + if SFA_CP + else NUM_BYTES_BOX1 + NUM_BYTES_SFA_RAW // 4 + ), + ) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(jj * NUM_BYTES_BOX1) + ), + tma_a2_desc.get_ptr(), + ( + cutlass.Int32(t2 * KT_BYTES), + blk * cutlass.Int32(R2), + e, + ), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + cutlass.Int32(jj * NUM_WORDS_SFA_RAW // 4) + ), + tma_s2_desc.get_ptr(), + ( + cutlass.Int32(0), + ( + ( + e * cutlass.Int32(SF2_RB) + + (blk >> cutlass.Int32(2)) + ) + * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(4), + ), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf2_cp(lane, stage, g, blk, t2, n_here, s_el, sSFA, s2w, ab_full) + stage, phase = _adv(stage, phase) + if lane == 0: + _stamp(tdbg, bx, 3) + + # ---- scale splicer (warp 6): every stage, atom kb word [r][q] = raw atom 4 q + kb word [r][m1] + if (warp == S_WARP) & cutlass.Boolean(not SFA_CP): # (SFA_CP: no splicer) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + for p in range(ng + cutlass.Int32(LAG)): + i0 = cutlass.Int32(0) + i1 = cutlass.Int32(0) + for part in cutlass.range_constexpr(2): + cnt = cutlass.Int32(0) + m1 = cutlass.Int32(0) + nk = cutlass.Int32(0) + if cutlass.const_expr(part == 0): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + nk = _nsegs(i0, i1) + else: + if p >= cutlass.Int32(LAG): + nk = nb + for k in range(nk): + if cutlass.const_expr(part == 0): + ug, j0, j1 = _seg(i0, i1, k) + m1 = ( + ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + ) & cutlass.Int32(3) + cnt = j1 - j0 + else: + m1 = (blk0 + k) & cutlass.Int32(3) + cnt = cutlass.Int32(KT2) + for _j in cutlass.range(cnt, unroll=1): + _waitd(ab_full.subview(stage), phase, tdbg, 3) + raw0 = ( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + lane * cutlass.Int32(4) + + m1 + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for kb in cutlass.range_constexpr(NUM_KB): + w0 = sSFAraw.load(idx=raw0 + cutlass.Int32((0 * 4 + kb) * 128)) + w1 = sSFAraw.load(idx=raw0 + cutlass.Int32((1 * 4 + kb) * 128)) + w2 = sSFAraw.load(idx=raw0 + cutlass.Int32((2 * 4 + kb) * 128)) + w3 = sSFAraw.load(idx=raw0 + cutlass.Int32((3 * 4 + kb) * 128)) + sSFA.store( + ( + cutlass.Int32(w0), + cutlass.Int32(w1), + cutlass.Int32(w2), + cutlass.Int32(w3), + ), + idx=dst0 + cutlass.Int32(kb * 128), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + prims.tcgen05_commit(sfa_ready.subview(stage)) + stage, phase = _adv(stage, phase) + + if cutlass.const_expr(B2IMG): + # ---- (B2IMG) group releases (warp 6): this CTA's finished units of group p, one release add per group + if warp == cutlass.Int32(S_WARP): + for p in range(ng): + i0, i1 = _plan_range(s_plan, p) + n_done = cutlass.Int32(0) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + n_done = n_done + cutlass.select_( + (j0 != cutlass.Int32(0)) | (j1 == cutlass.Int32(S1)), + cutlass.Int32(1), + cutlass.Int32(0), + ) + _wait( + fc1_done.subview(p & cutlass.Int32(3)), + (p >> cutlass.Int32(2)) & cutlass.Int32(1), + ) + if cutlass.const_expr(B2_DLY > 0): + t0d = _gtime() + dl = cutlass.Int64( + ( + (bx * cutlass.Int32(37) + p * cutlass.Int32(11) + ep) + & cutlass.Int32(63) + ) + * cutlass.Int32(B2_DLY // 64) + ) + while _gtime() - t0d < dl: + pass + if (lane == 0) & (n_done > cutlass.Int32(0)): + _fence_proxy_async_global() + _red_release_add( + b2cnt.subview(p * cutlass.Int32(8)).data_ptr().toint(), n_done + ) + cute.arch.sync_warp() + # ---- (B2IMG) FC2 B loader (warp 5): group g's image once its counter is complete + if warp == cutlass.Int32(X_WARP): + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + cnt_target = (ep + cutlass.Int32(1)) * cutlass.Int32(CNT_PER_CALL) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + ((g >> cutlass.Int32(1)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + if lane == 0: + _stamp(tdbg, bx, 29) + caddr = b2cnt.subview(g * cutlass.Int32(8)).data_ptr().toint() + if lane == 0: + cv = _ld_acquire_u32(caddr) + while (cv - cnt_target) < cutlass.Int32(0): + _nanosleep(B2IMG_NS) + cv = _ld_acquire_u32(caddr) + _fence_proxy_async_global() + prims.mbarrier_arrive_expect_tx(img_bar.subview(buf), IMG_BYTES) + prims.cp_async_bulk_shared_cluster_global( + sB2.subview(buf * cutlass.Int32(IMG_B2)), + img32.subview(g * cutlass.Int32(IMG_BYTES // 4)), + img_bar.subview(buf), + IMG_B2, + ) + prims.cp_async_bulk_shared_cluster_global( + sSFB2.subview(buf * cutlass.Int32(IMG_SF // 4)), + img32.subview( + g * cutlass.Int32(IMG_BYTES // 4) + cutlass.Int32(IMG_B2 // 4) + ), + img_bar.subview(buf), + IMG_SF, + ) + cute.arch.sync_warp() + _waitd( + img_bar.subview(buf), (g >> cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 108 + ) + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 25) + # ---- FC2 B builder (warps 5, 8 ..): group g's records (all 48 per (slot, token), the call's tag on + # every word) into B2 buffer g % 2 (row j M + t, k-tile t2 = record / 16) and its scale atoms, then + # tcgen05.cp to TMEM. + if ((warp == X_WARP) | (warp >= cutlass.Int32(8))) & cutlass.Boolean(not B2IMG): + xi = cutlass.select_( + warp == cutlass.Int32(X_WARP), cutlass.Int32(0), warp - cutlass.Int32(7) + ) + if cutlass.const_expr(G_CAP > TOP_K): + if (bx == 0) & (xi == cutlass.Int32(0)): + # rows of slots this call does not use: tag 0, never taken for a later call's + for i in range( + g_n * cutlass.Int32(M_MAX * UNITS_ROW) + lane, + cutlass.Int32(G_CAP * M_MAX * UNITS_ROW), + 32, + ): + hbuf32.store( + ( + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + ), + idx=i * cutlass.Int32(4), + alignment=16, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sSFB2_8 = cutlass.Array( + sSFB2.data_ptr(0), shape=(2 * KT2 * NUM_KB * 512,), dtype=cutlass.Int8, alignment=16 + ) + sB2_32 = cutlass.Array( + sB2.data_ptr(0), + shape=(2 * KT2 * NUM_BYTES_B2 // 4,), + dtype=cutlass.Int32, + alignment=16, + ) + nrec_cap = 4 * M_MAX * UNITS_ROW # records of one group + NZ = (nrec_cap + 32 * XW - 1) // (32 * XW) # records per lane: lane + 32 (XW z + xi) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + ((g >> cutlass.Int32(1)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + _waitd( + fc1_done.subview(g & cutlass.Int32(3)), + (g >> cutlass.Int32(2)) & cutlass.Int32(1), + tdbg, + 4, + ) # this CTA's own share of group g is out + if cutlass.const_expr(TIMING): + if (lane == 0) & (xi == cutlass.Int32(0)) & (g < cutlass.Int32(8)): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(32) + g + ) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + nrec = n_here * cutlass.Int32(M_MAX * UNITS_ROW) + rec0 = g * cutlass.Int32( + 4 * M_MAX * UNITS_ROW + ) # rows (4 g + j) M + t are contiguous + done = cutlass.Int32(0) + rspin = cutlass.Int32(0) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 29) + npoll = cutlass.Int32(0) + got = [[cutlass.Int32(0) for _ in range(3)] for _ in range(NZ)] + while done == cutlass.Int32(0): + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + v4 = prims.load_ext( + hbuf32.subview((rec0 + qi) * cutlass.Int32(4)), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for wq in cutlass.range_constexpr(3): + got[z][wq] = cutlass.Int32(v4[wq]) + ok = ok & ( + ( + ((got[z][0] ^ tagv) | (got[z][1] ^ tagv) | (got[z][2] ^ tagv)) + & cutlass.Int32(-16777216) + ) + == cutlass.Int32(0) + ) + npoll = npoll + cutlass.Int32(1) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32(-1): + done = cutlass.Int32(1) + else: + _nanosleep(B2_POLL_NS) + if cutlass.const_expr(HANG_DBG): + rspin = rspin + cutlass.Int32(1) + if rspin > cutlass.Int32(1 << 14): + done = cutlass.Int32(1) + tdbg.store( + cutlass.Int64(200), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + warp, + ) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + w0, w1, w2 = got[z][0], got[z][1], got[z][2] + rr = qi // cutlass.Int32( + UNITS_ROW + ) # row within the group: j M + t = B row n + c = qi - rr * cutlass.Int32(UNITS_ROW) + t2 = c >> cutlass.Int32(4) + cc = (c >> cutlass.Int32(1)) & cutlass.Int32(7) + lo = (w0 & cutlass.Int32(0xFFFFFF)) | (w1 << cutlass.Int32(24)) + hi = ((w1 >> cutlass.Int32(8)) & cutlass.Int32(0xFFFF)) | ( + (w2 & cutlass.Int32(0xFFFF)) << cutlass.Int32(16) + ) + sB2_32.store( + (lo, hi), + idx=( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_BYTES_B2) + + rr * cutlass.Int32(KT_BYTES) + + (cc ^ (rr & cutlass.Int32(7))) * cutlass.Int32(16) + + (c & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + # scale of block c: atom (t2, kb = (c % 16) / 4), row rr, byte c % 4 + sSFB2_8.store( + cutlass.Int8((w2 >> cutlass.Int32(16)) & cutlass.Int32(0xFF)), + idx=( + ( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_KB) + + ((c & cutlass.Int32(15)) >> cutlass.Int32(2)) + ) + * cutlass.Int32(128) + + rr * cutlass.Int32(4) + ) + * cutlass.Int32(4) + + (c & cutlass.Int32(3)), + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + if cutlass.const_expr(XW > 1): + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=B2_BAR, number_of_threads=32 * XW) + cute.arch.sync_warp() + if (xi == cutlass.Int32(0)) & prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + SFB2_COL + buf * cutlass.Int32(KT2 * NUM_KB * 4) + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 25) + if cutlass.const_expr(TIMING): + if g < cutlass.Int32(8): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(11) + g + ) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(npoll), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(30), + ) + + # ---- MMA (warp 7) + if warp == MMA_WARP: + idesc1 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N1, + m_dim=MMA_M, + sparsity_version=0, + ) + idesc2 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N2, + m_dim=MMA_M, + sparsity_version=0, + ) + desc_a0 = prims.Tcgen05SmemDesc.build( + sA.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b10 = prims.Tcgen05SmemDesc.build( + sB1.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b20 = prims.Tcgen05SmemDesc.build( + sB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + _waitd(b1_ready, 0, tdbg, 5) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + acc_b2 = cutlass.Int64(0) + acc_sf = cutlass.Int64(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + _waitd( + acc_empty.subview(ab), + ((k1 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + tdbg, + 102, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc_ptr = cutlass.inttoptr( + tcol + ACC1_COL + ab * cutlass.Int32(N1), 6, cutlass.Float32 + ) + for j in cutlass.range(j0, j1, unroll=1): + tw0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 6) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 6) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b10 + cutlass.Int64(j * cutlass.Int32(NUM_BYTES_B1 >> 4)) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = tcol + SFB1_COL + j * cutlass.Int32(NUM_KB * 4) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc1, + (j > j0) | cutlass.Boolean(kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc_full.subview(ab)) + k1 = k1 + cutlass.Int32(1) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + buf = g & cutlass.Int32(1) + tw2 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + _waitd( + b2_ready.subview(buf), (g >> cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 103 + ) + if cutlass.const_expr(TIMING): + acc_b2 = acc_b2 + (_gtime() - tw2) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 4) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_empty.subview(a2), + ((k2 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + tdbg, + 104, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc2_ptr = cutlass.inttoptr( + tcol + ACC2_COL + a2 * cutlass.Int32(N2), 6, cutlass.Float32 + ) + for t2 in cutlass.range_constexpr(KT2): + tw1 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 7) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 7) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw1) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b20 + cutlass.Int64( + (buf * cutlass.Int32(KT2) + cutlass.Int32(t2)) + * cutlass.Int32(NUM_BYTES_B2 >> 4) + ) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = ( + tcol + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + cutlass.Int32(t2 * NUM_KB * 4) + ) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc2_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc2, + cutlass.Boolean(t2 > 0 or kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc2_full.subview(a2)) + k2 = k2 + cutlass.Int32(1) + if cutlass.const_expr(TIMING): + if lane == 0: + tdbg.store(acc_b2, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(26)) + tdbg.store(acc_sf, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(27)) + + # ---- epilogue warps (0-3) + if warp < 4: + row_w = base_row + warp * 32 + for r in cutlass.range_constexpr((6 * G_CAP + EPI_THREADS - 1) // EPI_THREADS): + qi = tidx + cutlass.Int32(r * EPI_THREADS) + sl_e = qi // cutlass.Int32(6) + if (qi < cutlass.Int32(6 * G_CAP)) & (sl_e < g_n): + fld = qi - sl_e * cutlass.Int32(6) + s_esc.store( + cutlass.Float32( + escal.load(idx=fld * cutlass.Int32(E_LOCAL) + s_el.load(idx=sl_e)) + ), + idx=sl_e * cutlass.Int32(8) + fld, + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + SH_ADD = SHARED + wys = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + pars = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + ard = (cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0)) + for p in range(ng + cutlass.Int32(LAG)): + if cutlass.const_expr(AR_PRE): + if (p == ng + cutlass.Int32(LAG - 1)) & ar_on: + # one 8-byte load each (the vectorizer merges adjacent arp.load()s into 16-byte reads + # at 8-byte offsets: misaligned) + ard = tuple( + cutlass.Int64(_ld_relaxed_u64(arp.subview(i_).data_ptr().toint())) + for i_ in (1, 3, 4, 5) + ) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_cl = cutlass.select_( + f_bb < nb, (blk0 + f_bb) * cutlass.Int32(R2), cutlass.Int32(0) + ) + pars[bt] = _ar_parity(arp, f_t, f_cl) + if cutlass.const_expr(SH_ADD): + if p == ng + cutlass.Int32(LAG - 1): + # this CTA's y_sh words, loaded while the last group's FC2 runs (re-polled in finalize) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_ch = cutlass.select_( + f_bb < nb, + f_t * cutlass.Int32(H) + + (blk0 + f_bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + cutlass.Int32(0), + ) + wys[bt] = _ld_relaxed_u32(dnx.subview(f_ch).data_ptr().toint()) + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + cblk = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + unit = slot * cutlass.Int32(U1) + cblk + _waitd( + acc_full.subview(ab), + (k1 >> cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 105, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, base_col + ACC1_COL + ab * cutlass.Int32(N1), cutlass.Float32 + ), + num=N1, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc_empty.subview(ab)) + pb = ab * cutlass.Int32(M_MAX * C1 * R1) + for t in cutlass.range_constexpr(M_MAX): + part = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(4): + part = cutlass.select_( + warp == cutlass.Int32(w), cutlass.Float32(acc[QS * w + t]), part + ) + s_part.store(part, idx=pb + (t * C1 + warp) * R1 + lane) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + for tr in cutlass.range_constexpr((M_MAX + 3) // 4): + t = warp + cutlass.Int32(4 * tr) + if t < cutlass.Int32(M_MAX): + xv = _add_rn( + _add_rn( + _add_rn( + s_part.load(idx=pb + (t * C1 + 0) * R1 + lane), + s_part.load(idx=pb + (t * C1 + 1) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 2) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 3) * R1 + lane), + ) + finish = cutlass.Int32(1) + if (j0 != cutlass.Int32(0)) | (j1 != cutlass.Int32(S1)): + paddr = ( + partials.subview( + (unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane + ) + .data_ptr() + .toint() + ) + if j0 == cutlass.Int32(0): + _st_relaxed_u32(paddr, xv.bitcast(cutlass.Int32)) + finish = cutlass.Int32(0) + else: + pbits = _ld_relaxed_u32(paddr) + spins = cutlass.Int32(0) + while pbits == cutlass.Int32(PARTIAL_EMPTY): + pbits = _ld_relaxed_u32(paddr) + spins = spins + cutlass.Int32(1) + _st_relaxed_u32(paddr, cutlass.Int32(PARTIAL_EMPTY)) + xv = _add_rn(pbits.bitcast(cutlass.Float32), xv) + cute.arch.sync_warp() + if cutlass.const_expr( + XVDUMP + ): # debug: FC1 sums into the (unused) partial slots + partials.store( + xv.bitcast(cutlass.Int32), + idx=(unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane, + ) + if finish == cutlass.Int32(1): + gate_raw = cute.arch.shuffle_sync_bfly(xv, 8) + is_up = ( + (lane >> cutlass.Int32(3)) & cutlass.Int32(1) + ) == cutlass.Int32(0) + es = slot * cutlass.Int32(8) + qv, sfb = _swiglu_quant( + xv, + gate_raw, + is_up, + s_esc.load(idx=es), + s_esc.load(idx=es + cutlass.Int32(1)), + s_esc.load(idx=es + cutlass.Int32(2)), + s_esc.load(idx=es + cutlass.Int32(3)), + s_esc.load(idx=es + cutlass.Int32(4)), + ) + byte = _e2m1x2(cute.arch.shuffle_sync_down(qv, 16), qv) + bt = [ + cute.arch.shuffle_sync(byte, i) & cutlass.Int32(0xFF) + for i in range(8) + ] + rw0 = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | tagv + ) + rw1 = ( + bt[3] + | (bt[4] << cutlass.Int32(8)) + | (bt[5] << cutlass.Int32(16)) + | tagv + ) + rw2 = ( + bt[6] + | (bt[7] << cutlass.Int32(8)) + | ((sfb & cutlass.Int32(0xFF)) << cutlass.Int32(16)) + | tagv + ) + if cutlass.const_expr(B2IMG): + if lane == 0: + # B2 row n = j M + t, k-tile c / 16, 16-byte chunk (c / 2) % 8 + # (swizzled), half c % 2; scale: atom (t2, kb = (c % 16) / 4), + # row n at byte 16 n, byte c % 4 + n = (ug // cutlass.Int32(U1)) * cutlass.Int32(M_MAX) + t + t2i = cblk >> cutlass.Int32(4) + cci = (cblk >> cutlass.Int32(1)) & cutlass.Int32(7) + gbase = p * cutlass.Int32(IMG_BYTES) + lo = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | (bt[3] << cutlass.Int32(24)) + ) + hi = ( + bt[4] + | (bt[5] << cutlass.Int32(8)) + | (bt[6] << cutlass.Int32(16)) + | (bt[7] << cutlass.Int32(24)) + ) + img32.store( + (lo, hi), + idx=( + gbase + + t2i * cutlass.Int32(NUM_BYTES_B2) + + n * cutlass.Int32(KT_BYTES) + + (cci ^ (n & cutlass.Int32(7))) + * cutlass.Int32(16) + + (cblk & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + img8 = cutlass.Array( + img32.data_ptr(0), + shape=(NG_CAP * IMG_BYTES,), + dtype=cutlass.Int8, + alignment=16, + ) + img8.store( + cutlass.Int8(sfb & cutlass.Int32(0xFF)), + idx=gbase + + cutlass.Int32(IMG_B2) + + ( + ( + t2i * cutlass.Int32(NUM_KB) + + ( + (cblk & cutlass.Int32(15)) + >> cutlass.Int32(2) + ) + ) + * cutlass.Int32(512) + ) + + n * cutlass.Int32(16) + + (cblk & cutlass.Int32(3)), + ) + else: + if lane == 0: + hbuf32.store( + (rw0, rw1, rw2, tagv), + idx=( + (slot * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(UNITS_ROW) + + cblk + ) + * cutlass.Int32(4), + alignment=16, + ) + k1 = k1 + cutlass.Int32(1) + prims.mbarrier_arrive( + fc1_done.subview(p & cutlass.Int32(3)) + ) # warp 5 may poll group p + if tidx == 0: + _stamp(tdbg, bx, 10) + if cutlass.const_expr(TIMING): + if p < cutlass.Int32(8): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(40) + p, + ) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_full.subview(a2), + (k2 >> cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 106, + ) + if cutlass.const_expr(TIMING): + if (tidx == 0) & (g == ng - cutlass.Int32(1)): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(8) + bb, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + yv = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, + base_col + + ACC2_COL + + a2 * cutlass.Int32(N2) + + warp * cutlass.Int32(M_MAX), + cutlass.Float32, + ), + num=LD_POW2, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc2_empty.subview(a2)) + sl = g * cutlass.Int32(4) + warp + if sl < g_n: + a2s = s_esc.load(idx=sl * cutlass.Int32(8) + cutlass.Int32(5)) + for t in cutlass.range_constexpr(M_MAX): + kp = s_kpos.load(idx=sl * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + if kp >= cutlass.Int32(0): + ys.store( + _bf16_round(_mul_rn(cutlass.Float32(yv[t]), a2s)), + idx=( + (bb * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + * cutlass.Int32(TOP_K) + + kp + ) + * cutlass.Int32(R2) + + lane, + ) + k2 = k2 + cutlass.Int32(1) + prims.mbarrier_arrive(b2_free.subview(g & cutlass.Int32(1))) + if tidx == 0: + _stamp(tdbg, bx, 6) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + if tidx == 0: + _stamp(tdbg, bx, 19) + # finalize: (block bb, token t) per warp round: the fma chain over t's top-4 in order + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + pair = warp + cutlass.Int32(bt * 4) + bb = pair // cutlass.Int32(M_MAX) + t = pair - bb * cutlass.Int32(M_MAX) + if bb < nb: + acc_o = cutlass.Float32(0.0) + for k in cutlass.range_constexpr(TOP_K): + acc_o = _fma_rn( + s_w.load(idx=t * cutlass.Int32(TOP_K) + cutlass.Int32(k)), + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ), + acc_o, + ) + if cutlass.const_expr( + YDUMP + ): # debug: y of (token, k) into the partials workspace + partials.store( + cutlass.Float32( + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ) + ).bitcast(cutlass.Int32), + idx=(t * cutlass.Int32(TOP_K) + cutlass.Int32(k)) * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + ) + ch = ( + t * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(SH_ADD): + # + y_sh (this row's shared output, from its shared CTA), as shared_output.add_(routed_output) + wy = wys[bt] + while ((wy >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) != tag16: + wy = _ld_relaxed_u32(dnx.subview(ch).data_ptr().toint()) + acc_o = _add_rn( + (wy << cutlass.Int32(16)).bitcast(cutlass.Float32), _bf16_round(acc_o) + ) + out.store(cutlass.BFloat16(acc_o), idx=ch) + if ar_on: + cl = ( + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(AR_PRE): + _ar_push_pre( + ard, + pars[bt], + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + else: + _ar_push( + arp, + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + + # ---- teardown + if tidx == 0: + _stamp(tdbg, bx, 7) + epochs.store(ep + cutlass.Int32(1), idx=bx) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if ar_on | cutlass.Boolean(LATE_TRIGGER): + # the all-reduce polls the pushed slots without a grid wait: release it after every push of this CTA + cute.arch.griddepcontrol_launch_dependents() + if warp == MMA_WARP: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Float32), TMEM_COLS + ) + + +@cute.jit +def m3_moe( + a1_tensor: cute.Tensor, + sfa1_tensor: cute.Tensor, + a2_tensor: cute.Tensor, + sfa2_tensor: cute.Tensor, + logits: cute.Tensor, + bias: cute.Tensor, + x32: cute.Tensor, + xsf32: cute.Tensor, + escal: cute.Tensor, + plan: cute.Tensor, + partials: cute.Tensor, + hbuf32: cute.Tensor, + epochs: cute.Tensor, + out: cute.Tensor, + route_dbg: cute.Tensor, + tdbg: cute.Tensor, + xb32: cute.Tensor, + rw: cute.Tensor, + sfsc: cute.Tensor, + lgbuf: cute.Tensor, + guw: cute.Tensor, + gusf: cute.Tensor, + dnw: cute.Tensor, + dnsf: cute.Tensor, + hx: cute.Tensor, + dnx: cute.Tensor, + arp: cute.Tensor, + s1w: cute.Tensor, + s2w: cute.Tensor, + img32: cute.Tensor, + b2cnt: cute.Tensor, + stream: cuda_driver.CUstream, +): + tma_a1_desc = cuda.create_tensor_map_tiled( + global_address=a1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H // 2, TWO_I, E_LOCAL], + global_strides=[H // 2 // 16, TWO_I * (H // 2) // 16], + box_dims=(KT_BYTES, R1, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_a2_desc = cuda.create_tensor_map_tiled( + global_address=a2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[I_TP // 2, H, E_LOCAL], + global_strides=[I_TP // 2 // 16, H * (I_TP // 2) // 16], + box_dims=(KT_BYTES, R2, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_s1_desc = cuda.create_tensor_map_tiled( + global_address=sfa1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * TWO_I * (H // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_s2_desc = cuda.create_tensor_map_tiled( + global_address=sfa2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * H * (I_TP // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 4 // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_gu_desc = tma_a1_desc + tma_dn_desc = tma_a2_desc + if cutlass.const_expr(SHARED and not SH_PRETILE): + tma_gu_desc = cuda.create_tensor_map_tiled( + global_address=guw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H, 2 * SH_I], + global_strides=[H // 16], + box_dims=(KT_BYTES, 32), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_dn_desc = cuda.create_tensor_map_tiled( + global_address=dnw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[SH_I, H], + global_strides=[SH_I // 16], + box_dims=(KT_BYTES, 128), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + m3_moe_kernel( + tma_a1_desc, + tma_a2_desc, + tma_s1_desc, + tma_s2_desc, + tma_gu_desc, + tma_dn_desc, + logits, + bias, + x32, + xsf32, + escal, + plan, + partials, + hbuf32, + epochs, + out, + route_dbg, + tdbg, + xb32, + rw, + sfsc, + lgbuf, + guw, + gusf, + dnw, + dnsf, + hx, + dnx, + arp, + s1w, + s2w, + img32, + b2cnt, + ).launch( + grid=(NCTA + NSH, 1, 1), + block=(THREADS, 1, 1), + cluster=(1, 1, 1), + stream=stream, + use_pdl=True, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/op.py new file mode 100644 index 000000000000..87605e245fe3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/op.py @@ -0,0 +1,622 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Host side of the MiniMax-M3 routed-expert kernel on one TP4 rank, over the TRTLLM-Gen NVFP4 buffers read in place. + +``m3_moe`` (m3_moe_kernel.py, :class:`M3MoeState`): 1..8 decode tokens (decode batches, spec-decode verify). The call +takes the bf16 hidden states and folds in the router GEMV (bit-exact with m3_router_gemv_fp4) and the activation's +NVFP4 quantization. With ``shared=True`` it also runs the shared expert on dedicated CTAs and returns +bf16(shared + routed); otherwise it returns the rank's routed partial, bit for bit what the stock fused MoE returns. + +Usage (build before CUDA-graph capture; every call is capture-safe afterwards):: + + state = M3MoeState( + device, num_tokens=6, shared=True + ) # workspace + compiled kernel, shared by all layers + layer = state.layer( + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=router_weight, + fc31_input_scale=fc31_input_scale, + shared_weights=(gate_up_w, gate_up_sf, down_w, down_sf), + ) + out = layer(hidden) # bf16 [M, 6144] +""" + +from __future__ import annotations + +import importlib.util +import os +import sys +from typing import Dict, Optional, Tuple + +import torch + +from ...m31_switches import m31_disabled + +HIDDEN = 6144 # capacity: fixed: the checkpoint's hidden size +INTER = 768 +NUM_EXPERTS = 128 +# The shared expert's intermediate per TP rank. +SHARED_INTER = 768 # capacity: fixed: the shared expert's width per tp4 rank +# m3_moe's shared-expert CTAs (beside num_ctas routed CTAs). +SHARED_CTAS = 24 # capacity: fixed: an SM split, not a size +TOP_K = 4 +U1 = 2 * INTER // 32 # FC1 units per expert +# The router GEMV takes at most 2 experts per CTA (router weights held in registers). +MIN_CTAS = 64 # capacity: fixed: an SM split, not a size + +_UNIT_BYTES = 32 * (HIDDEN // 2 + HIDDEN // 16) # one FC1 unit's weight + scale bytes +_ROW2_BYTES = INTER // 2 + INTER // 16 # one FC2 weight row's bytes (values + scales) +_S1 = 6 # FC1 stages (items) per unit # capacity: fixed: pipeline stages +_ITEM_BYTES = _UNIT_BYTES // _S1 # capacity: fixed: a ring item's bytes +_modules: Dict[tuple, object] = {} +_compiled: Dict[tuple, object] = {} +# m3_moe trace-time options by token count 1..8 (index num_tokens - 1). +# ``sfa_cp``: 18 KB ring stages, the A scales copied by cp.async into the instruction layout (every token count). +SFA_CP_BY_M: Tuple[int, ...] = (1, 1, 1, 1, 1, 1, 1, 1) +# ``b2img``: FC1 -> FC2 hand-off through per-group global B images + release counters (7 and 8 tokens). +B2IMG_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 1, 1) +# ``ar_pre``: the finalize's all-reduce push with its call parities / descriptor preloaded (all but 2 tokens). +AR_PRE_BY_M: Tuple[int, ...] = (1, 0, 1, 1, 1, 1, 1, 1) +# Ring ``stages`` (0 = the kernel's default, the most that fit). 6 at 8 tokens measured 0.4-0.5 us faster than the +# default 7 (same MMAs, bit-identical outputs). +STAGES_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 0, 6) +# Early group-0 stages warp 4 issues right after its top-k (``early0_n``; 0 = the kernel's default, every ring stage). +# Fewer at 4 and 8 tokens measured faster on GB300 (MOE_EARLY0N_READ: 6 at 4 tokens -0.37 us, 3 at 8 tokens -0.15 us +# per call, bit-identical outputs); other token counts were flat or unmeasured. Kill switch +# ``TRTLLM_M31_DISABLE_MOE_EARLY0N`` restores the default everywhere. +EARLY0_N_BY_M: Tuple[int, ...] = (0, 0, 0, 6, 0, 0, 0, 3) +# Per-token-count overrides of the tables above, measured together: at 4 tokens the B-image FC1 -> FC2 hand-off plus 8 +# ring stages run 1.62 us/call faster on GB300 with bit-identical outputs (MOE_CLK64_MR6C_READ; each alone ~0.8). +# Kill switch ``TRTLLM_M31_DISABLE_MOE_M4CFG`` restores the tables' values. +M4_CFG_BY_M: Dict[int, Dict[str, int]] = {4: {"b2img": 1, "stages": 8}} +# m3_moe phase lag: phase p streams FC1 of group p and FC2 of group p - M3_MOE_LAG (the kernel's default lag). +M3_MOE_LAG = 2 + + +_ENGINE_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "m3_moe_kernel.py") + + +def _kernel_module(config: dict, path: str): + key = (os.path.basename(path),) + tuple(sorted(config.items())) + mod = _modules.get(key) + if mod is None: + name = ( + f"{__name__}_" + + os.path.splitext(os.path.basename(path))[0] + + "_" + + "_".join(f"{k}{v}" for k, v in key[1:]) + ) + spec = importlib.util.spec_from_file_location(name, path) + mod = importlib.util.module_from_spec(spec) + mod.M3_CONFIG = dict(config) + sys.modules[name] = mod + spec.loader.exec_module(mod) + _modules[key] = mod + return mod + + +def _view(t: torch.Tensor, align: int): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align) + + +def is_supported(device: Optional[torch.device] = None) -> tuple: + if not torch.cuda.is_available(): + return False, "no CUDA" + major, _ = torch.cuda.get_device_capability(device) + if major != 10: + return False, "needs the sm_100 family (tcgen05 block-scaled NVFP4 MMA)" + try: + import cutlass # noqa: F401 + except ImportError: + return False, "CuTe DSL (nvidia-cutlass-dsl) is not installed" + return True, "" + + +def ar_push_desc( + mc_ptr: int, flags_ptr: int, rank: int, world: int, max_tokens: int, device +) -> torch.Tensor: + """The ``arp`` descriptor that makes a call also push its bf16 output into a multicast all-reduce exchange (set A + of m3_mc_allreduce's workspace at ``mc_ptr``, slot half = the all-reduce CTA's call parity in ``flags_ptr``). + With it the kernel releases its PDL dependents only after every push; the default (enable 0) keeps the old + behaviour.""" + return torch.tensor( + [1, mc_ptr, flags_ptr, rank, world, max_tokens, 0, 0], dtype=torch.int64, device=device + ) + + +def _shared_views(shared_weights, who): + """(gate_up, its scales, down, its scales) of the shared expert as flat uint8 views, checked.""" + if shared_weights is None: + raise ValueError(f"{who} (shared): shared_weights are required") + gu_w, gu_sf, dn_w, dn_sf = shared_weights + want = {"gate_up": (gu_w, (2 * SHARED_INTER, HIDDEN)), "down": (dn_w, (HIDDEN, SHARED_INTER))} + for name, (t, shape) in want.items(): + if t.dtype != torch.float8_e4m3fn or tuple(t.shape) != shape or not t.is_contiguous(): + raise ValueError(f"{who} (shared): {name} weight must be contiguous e4m3 {shape}") + if ( + gu_sf.numel() != 2 * SHARED_INTER * HIDDEN // 32 + or dn_sf.numel() != HIDDEN * SHARED_INTER // 32 + ): + raise ValueError( + f"{who} (shared): the UE8M0 scales must be the swizzled [N, K / 32] buffers" + ) + return ( + gu_w.view(torch.uint8).view(-1), + gu_sf.view(torch.uint8).reshape(-1), + dn_w.view(torch.uint8).view(-1), + dn_sf.view(torch.uint8).reshape(-1), + ) + + +def _pretile_shared(gu8, gusf8, dn8, dnsf8, n_cta: int = 24) -> torch.Tensor: + """The shared expert's weights as m3_moe's shared CTAs stream them (SH_PRETILE): per CTA s, per stage g, the 16 KB + shared-memory image (128-byte swizzle: 16-byte chunk j of row r at j ^ (r % 8)) of the stage's 128-row MMA tile and + its 512-byte instruction scale atom (row R's 4 codes at word 4 (R % 32) + R / 32). Stages 0..23: gate_up k-tiles + g (slice 0) / 24 + g (slice 1) of rows [gate 32 s .., up 768 + 32 s .., gate (slice 1), up (slice 1)]; 24..35: + down tile (g - 24) / 6 (rows 256 s + 128 j ..), k-tile (g - 24) % 6. Built once per layer.""" + dev = gu8.device + H_, I_ = HIDDEN, SHARED_INTER + gu = gu8.view(2 * I_, H_) + dn = dn8.view(H_, I_) + gkb, dkb = H_ // 32, I_ // 32 + + def sf_bytes(sf, n, b, kbn): # code of (row n, K block b), broadcast + return sf[ + ((n >> 7) * (kbn // 4) + (b >> 2)) * 512 + (n & 31) * 16 + ((n >> 5) & 3) * 4 + (b & 3) + ] + + r = torch.arange(128, device=dev) + q = r >> 5 + ri = r & 31 + chunk = torch.arange(8, device=dev) + swz = chunk[None, :] ^ (r[:, None] & 7) # source chunk of image chunk j: j ^ (r % 8) + out = torch.empty( + n_cta, 36, 16384 + 512, dtype=torch.uint8, device=dev + ) # capacity: fixed: a workspace of the kernel's fixed geometry + for s in range(n_cta): + # gate_up: MMA row R -> weight row, slice + nrow = torch.where((q & 1) == 0, 32 * s + ri, I_ + 32 * s + ri) + sl = q >> 1 + for g in range(24): + kt = sl * 24 + g # per row + tile = gu[ + nrow[:, None], (kt[:, None] * 128 + torch.arange(128, device=dev)[None, :]) + ] # [128, 128] + img = tile.view(128, 8, 16)[r[:, None], swz] # image chunk j = tile chunk j ^ (r % 8) + out[s, g, :16384] = img.reshape(-1) + kb = torch.arange(4, device=dev) + codes = sf_bytes(gusf8, nrow[:, None], kt[:, None] * 4 + kb[None, :], gkb) # [128, 4] + atom = torch.empty(512, dtype=torch.uint8, device=dev) + atom[(ri[:, None] * 16 + q[:, None] * 4 + kb[None, :]).reshape(-1)] = codes.reshape(-1) + out[s, g, 16384:] = atom + for gd in range(12): + jt, kt = gd // 6, gd % 6 + row0 = 256 * s + 128 * jt + tile = dn[row0 : row0 + 128, kt * 128 : (kt + 1) * 128] + img = tile.reshape(128, 8, 16)[r[:, None], swz] + out[s, 24 + gd, :16384] = img.reshape(-1) + out[s, 24 + gd, 16384:] = dnsf8[ + ((row0 >> 7) * (dkb // 4) + kt) * 512 : ((row0 >> 7) * (dkb // 4) + kt) * 512 + 512 + ] + return out.view(-1) + + +# --------------------------------------------------------------------------------------------------------------------- +# m3_moe (m3_moe_kernel.py): 1..8 tokens per call, group-phased weight stream. +# --------------------------------------------------------------------------------------------------------------------- +# Pre-tiled shared weights per layer, keyed by the source buffers (which the engines read in place): every token-count +# state of a layer streams the same copy. An entry holds its sources, so their addresses are not reused while cached. +_PRETILED: Dict[tuple, tuple] = {} + + +def _pretiled(gu8, gusf8, dn8, dnsf8) -> torch.Tensor: + key = (gu8.device, gu8.data_ptr(), gusf8.data_ptr(), dn8.data_ptr(), dnsf8.data_ptr()) + hit = _PRETILED.get(key) + if hit is None: + hit = _PRETILED[key] = ((gu8, gusf8, dn8, dnsf8), _pretile_shared(gu8, gusf8, dn8, dnsf8)) + return hit[1] + + +def make_phase_plan( + num_ctas: int, m_max: int, lag: int = 2, tail_w: float = 1.0, early0: bool = False +): + """The work split of ``m3_moe`` for each distinct-expert count G (4 .. 4 m_max): FC2 32-row output blocks in + contiguous per-CTA ranges (the same for every group), and per group p (slots 4 p ..) a range of its unit-major FC1 + stages per CTA, balancing the bytes each CTA streams in phase p (FC1 of group p plus FC2 of group p - lag, and the + trailing FC2s with the last group). Ranges are empty or >= 6 stages (a unit is split over at most two CTAs); a + group too small for every CTA gets whole units on the CTAs with the least FC2 work. Returns int32 + [4 m_max + 1, num_ctas, m_max + 1] (per group i0 | i1 << 16, then b0 | nb << 16) and the largest block count.""" + gcap = TOP_K * m_max + nb2 = HIDDEN // 32 + plan = torch.zeros(gcap + 1, num_ctas, m_max + 1, dtype=torch.int32) + nbs = [nb2 // num_ctas + (1 if c < nb2 % num_ctas else 0) for c in range(num_ctas)] + b0s = [sum(nbs[:c]) for c in range(num_ctas)] + blk_bytes = 32 * _ROW2_BYTES # one block, one slot + for g in range(TOP_K, gcap + 1): + ng = (g + 3) // 4 + for c in range(num_ctas): + plan[g, c, m_max] = b0s[c] | (nbs[c] << 16) + for p in range(ng): + n_p = min(4, g - 4 * p) + total = n_p * U1 * _S1 + fc2 = [0] * num_ctas + for c in range(num_ctas): + if p >= lag: + fc2[c] += nbs[c] * min(4, g - 4 * (p - lag)) * blk_bytes + if p == ng - 1 and not ( + early0 and p == 0 and m_max >= 2 + ): # early0: group 0's ranges the same for every G + for q in range(max(0, ng - lag), ng): + fc2[c] += int(tail_w * nbs[c] * min(4, g - 4 * q) * blk_bytes) + items = [0] * num_ctas + if total >= _S1 * num_ctas: + + def counts_for(limit): + return [max(_S1, (limit - f) // _ITEM_BYTES) for f in fc2] + + lo, hi = 0, max(fc2) + total * _ITEM_BYTES + while lo < hi: + mid = (lo + hi) // 2 + if sum(counts_for(mid)) >= total: + hi = mid + else: + lo = mid + 1 + items = counts_for(lo) + extra = sum(items) - total + while extra > 0: + c = min( + (c for c in range(num_ctas) if items[c] > _S1), + key=lambda c: (-(fc2[c] + items[c] * _ITEM_BYTES), c), + ) + items[c] -= 1 + extra -= 1 + else: + chosen = sorted(sorted(range(num_ctas), key=lambda c: (fc2[c], c))[: total // _S1]) + for c in chosen: + items[c] = _S1 + i0 = 0 + for c in range(num_ctas): + if items[c]: + plan[g, c, p] = i0 | ((i0 + items[c]) << 16) + i0 += items[c] + assert i0 == total + return plan, max(nbs) + + +class M3MoeState: + """Workspace + compiled kernel of ``m3_moe`` (1..8 tokens per call) on one device, shared by the layers that run + on it in one stream order. The call takes the bf16 hidden states and computes the router logits (bit-exact with + m3_router_gemv_fp4) and the NVFP4 activation itself. ``shared``: the shared expert runs on SHARED_CTAS dedicated + CTAs beside the ``num_ctas`` routed ones and the call returns bf16(shared + routed). Build before CUDA-graph + capture; the first call compiles.""" + + fused_router = True + + def __init__( + self, + device: torch.device, + num_tokens: int, + num_ctas: Optional[int] = None, + shared: bool = False, + ): + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "M3MoeState allocates and compiles: build it before CUDA-graph capture" + ) + sms_all = torch.cuda.get_device_properties(device).multi_processor_count + if shared and (num_ctas or sms_all) + SHARED_CTAS > sms_all: + raise ValueError( + f"m3_moe (shared): num_ctas + {SHARED_CTAS} shared-expert CTAs must fit the {sms_all} SMs " + f"(all resident at once)" + ) + if not 1 <= num_tokens <= 8: + raise ValueError(f"m3_moe takes 1..8 tokens per call, not {num_tokens}") + self.device = torch.device(device) + ok, why = is_supported(self.device) + if not ok: + raise RuntimeError(f"m3_moe: {why}") + sms = torch.cuda.get_device_properties(self.device).multi_processor_count + self.num_ctas = num_ctas or sms + if not MIN_CTAS <= self.num_ctas <= sms: + raise ValueError( + f"m3_moe runs {MIN_CTAS} .. {sms} CTAs (all resident at once), not {self.num_ctas}" + ) + self.m = num_tokens + # One token: the trailing FC2s weigh nothing in the last phase's balance. Group 0's FC1 ranges are the same + # for every distinct-expert count (early0). + plan, nb_cap = make_phase_plan( + self.num_ctas, num_tokens, M3_MOE_LAG, 0.0 if num_tokens == 1 else 1.0, True + ) + self.config = { + "m_max": num_tokens, + "num_ctas": self.num_ctas, + "nb2_cap": nb_cap, + "fused_router": 1, + "timing": 0, + } + if shared: + self.config["shared"] = 1 + self.shared = shared + # 18 KB ring stages (A scales by cp.async into the instruction layout) per token count, unless configured + self.config["sfa_cp"] = SFA_CP_BY_M[num_tokens - 1] + # FC1 -> FC2 hand-off through per-group global B images + release counters, per token count + self.config["b2img"] = B2IMG_BY_M[num_tokens - 1] + # the finalize's all-reduce push with its call parities / descriptor preloaded, per token count + self.config["ar_pre"] = AR_PRE_BY_M[num_tokens - 1] + if STAGES_BY_M[num_tokens - 1] > 0: + self.config["stages"] = STAGES_BY_M[num_tokens - 1] + if EARLY0_N_BY_M[num_tokens - 1] > 0 and not m31_disabled("MOE_EARLY0N"): + self.config["early0_n"] = EARLY0_N_BY_M[num_tokens - 1] + if num_tokens in M4_CFG_BY_M and not m31_disabled("MOE_M4CFG"): + self.config.update(M4_CFG_BY_M[num_tokens]) + try: + self.mod = mod = _kernel_module(self.config, _ENGINE_PATH) + except AssertionError as err: + raise ValueError( + f"m3_moe: num_tokens={num_tokens}, num_ctas={self.num_ctas} does not fit the shared " + f"memory / TMEM budget (fewer CTAs means more FC2 blocks per CTA): {err}" + ) from err + kw = dict(device=self.device) + self.plan = plan.to(self.device) + self.partials = torch.full((mod.G_CAP * U1 * num_tokens * 32,), -1, dtype=torch.int32, **kw) + self.hbuf32 = torch.zeros(mod.G_CAP * num_tokens * U1 * 4, dtype=torch.int32, **kw) + self.epochs = torch.zeros(self.num_ctas + mod.NSH, dtype=torch.int32, **kw) + self.route_dbg = torch.zeros(num_tokens, 8, dtype=torch.int32, **kw) + self.tdbg = torch.zeros(self.num_ctas + mod.NSH, 64, dtype=torch.int64, **kw) + self.lgbuf = torch.zeros(mod.LG_REPL * mod.LG_STRIDE, dtype=torch.int64, **kw) + self._dummy = torch.zeros(4, dtype=torch.int32, **kw) + self._ar_off = torch.zeros(8, dtype=torch.int64, **kw) + # B2IMG: per-group FC2 B images and their unit counters (int32; every group gains CNT_PER_CALL per call) + self.img32 = torch.zeros(max(1, mod.NG_CAP * mod.IMG_BYTES // 4), dtype=torch.int32, **kw) + self.cnt = torch.zeros(mod.NG_CAP * 8 + 8, dtype=torch.int32, **kw) + # shared expert: h and y_sh as (call tag << 16 | bf16), tag 0 = never written + self.hx = torch.zeros(num_tokens * SHARED_INTER, dtype=torch.int32, **kw) + self.dnx = torch.zeros(num_tokens * HIDDEN, dtype=torch.int32, **kw) + self._fn = None + + def layer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=None, + fc31_input_scale=None, + shared_weights=None, + ) -> "M3MoeLayer": + """One MoE layer's operands (the TRTLLM-Gen NVFP4 expert buffers, read in place, never copied). + ``shared_weights`` (a ``shared`` state): the shared expert's (gate_up e4m3 [1536, 6144], its swizzled UE8M0 + scales, down e4m3 [6144, 768], its swizzled UE8M0 scales).""" + layer = M3MoeLayer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight, + fc31_input_scale, + ) + if self.shared: + layer.shared_weights = _shared_views(shared_weights, "m3_moe") + layer.shared_mxfp8 = tuple(shared_weights) + if self.mod.SH_PRETILE: + # (gate_up, pre-tiled stages, down, dummy): the kernel streams the pre-tiled copy + gu8, _, dn8, _ = layer.shared_weights + layer.shared_weights = ( + gu8, + _pretiled(*layer.shared_weights), + dn8, + self._dummy.view(torch.uint8), + ) + return layer + + def run(self, layer, hidden, out=None, arp=None) -> torch.Tensor: + m = self.m + if ( + hidden is None + or hidden.dtype != torch.bfloat16 + or tuple(hidden.shape) != (m, HIDDEN) + or not hidden.is_contiguous() + ): + raise ValueError( + f"m3_moe (fused router): expects contiguous bf16 hidden states [{m}, {HIDDEN}]" + ) + fz = ( + hidden.view(-1).view(torch.int32), + layer.router_weight.view(-1), + layer.fc31_input_scale.view(-1), + self.lgbuf, + ) + lgv, xq, xs = self._dummy.view(torch.float32), self._dummy, self._dummy + if out is None: + out = torch.empty(m, HIDDEN, dtype=torch.bfloat16, device=self.device) + sh = layer.shared_weights if self.shared else (self._dummy.view(torch.uint8),) * 4 + args = ( + layer.w3_w1_weight, + layer.w3_w1_weight_scale.view(-1), + layer.w2_weight, + layer.w2_weight_scale.view(-1), + lgv, + layer.bias, + xq, + xs, + layer.escal, + self.plan.view(-1), + self.partials, + self.hbuf32, + self.epochs, + out.view(-1), + self.route_dbg.view(-1), + self.tdbg.view(-1), + *fz, + *sh, + self.hx, + self.dnx, + self._ar_off if arp is None else arp, + layer.w3_w1_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + layer.w2_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + self.img32, + self.cnt, + ) + if self._fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_moe compiles on its first call: call it once before capture") + import cutlass.cute as cute + + aligns = [ + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + ] + sig = [_view(t, a) for t, a in zip(args, aligns)] + key = (_ENGINE_PATH,) + tuple(sorted(self.config.items())) + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute.compile( + self.mod.m3_moe, + *sig, + cute.runtime.make_fake_stream(), + options="--enable-tvm-ffi", + ) + self._fn = fn + self._fn(*args, torch.cuda.current_stream(self.device).cuda_stream) + return out + + +class M3MoeLayer: + """One MoE layer on a :class:`M3MoeState` (or op16's state): ``layer(hidden)``. The expert buffers are read in + place, never copied; the per-expert scalars are packed once (layer constants: rebuild the layer after reloading + weights).""" + + shared_weights = None # the kernel's view of the shared expert (a shared state) + shared_mxfp8 = None # the shared expert's (gate_up, its scales, down, its scales) as given + + def __init__( + self, + state, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + bias, + router_weight=None, + fc31_input_scale=None, + ): + e = NUM_EXPERTS + expected = { + "w3_w1_weight": (w3_w1_weight, (e, 2 * INTER, HIDDEN // 2)), + "w3_w1_weight_scale": (w3_w1_weight_scale, (e, 2 * INTER, HIDDEN // 16)), + "w2_weight": (w2_weight, (e, HIDDEN, INTER // 2)), + "w2_weight_scale": (w2_weight_scale, (e, HIDDEN, INTER // 16)), + } + for name, (t, shape) in expected.items(): + if t.element_size() != 1 or tuple(t.shape) != shape or not t.is_contiguous(): + raise ValueError( + f"m3_moe: {name} is {t.dtype} {tuple(t.shape)}, expected contiguous 1-byte {shape}" + ) + self.state = state + self.w3_w1_weight = w3_w1_weight.view(torch.uint8) + self.w3_w1_weight_scale = w3_w1_weight_scale.view(torch.uint8) + self.w2_weight = w2_weight.view(torch.uint8) + self.w2_weight_scale = w2_weight_scale.view(torch.uint8) + self.escal = torch.stack( + [fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha] + ).float() + self.escal = self.escal.reshape(6, e).contiguous() + self.bias = bias.float().contiguous() + if ( + router_weight is None + or fc31_input_scale is None + or router_weight.dtype != torch.float32 + or tuple(router_weight.shape) != (e, HIDDEN) + or not router_weight.is_contiguous() + or fc31_input_scale.dtype != torch.float32 + or fc31_input_scale.numel() != 1 + ): + raise ValueError( + "m3_moe (fused router): needs router_weight f32 [128, 6144] and fc31_input_scale f32 [1]" + ) + self.router_weight = router_weight + self.fc31_input_scale = fc31_input_scale + + def __call__( + self, + hidden: torch.Tensor, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """``hidden``: contiguous bf16 [M, 6144]. ``arp`` (int64 [8], see :func:`ar_push_desc`): also push the output + into an all-reduce exchange.""" + return self.state.run(self, hidden, out, arp=arp) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/op16.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/op16.py new file mode 100644 index 000000000000..446776419315 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_moe/op16.py @@ -0,0 +1,288 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Host side of ``m3_moe16`` (m3_moe16_kernel.py): the MiniMax-M3 MoE layer (fused router + routed experts + shared +expert) for 9..16 decode tokens on one TP4 rank. The 1..8-token engine (op.M3MoeState) is untouched; select this +state only for num_tokens > 8. + +Output: bf16 [M, 6144] = bf16(shared + routed), the routed part bit for bit TRTLLM-Gen's fused MoE on the fp32 router +GEMV logits (m3_router_gemv's math), the shared part as the in-model m3_mx_gemv shared expert. + + state = M3Moe16State(device, num_tokens=12) # before CUDA-graph capture; one per token count + layer = state.layer(w3_w1_weight, w3_w1_weight_scale, w2_weight, w2_weight_scale, fc31_alpha, fc31_scale_c, + act_alpha, act_beta, act_clamp, fc2_alpha, e_score_correction_bias, + router_weight=router_weight, fc31_input_scale=fc31_input_scale, + shared_weights=(gate_up_w, gate_up_sf, down_w, down_sf)) + out = layer(hidden) # first call compiles: call once before capture + +Protocol state (per state object, shared by the layers that run on it in stream order): per-group FC2-B images and +unit counters, the B1 image + counter, per-CTA epochs. Every call adds exactly 192 to each group counter and NCTA to +the B1 counter, so nothing is reset between calls (graph replay safe). +""" + +from __future__ import annotations + +import os +from typing import Optional + +import torch + +from .op import ( + HIDDEN, + SHARED_CTAS, + SHARED_INTER, + U1, + M3MoeLayer, + _compiled, + _kernel_module, + _shared_views, + _view, + is_supported, +) + + +def _group_slots(g: int, p: int, perm: bool, min_ng: int = 5) -> int: + """Slots of group p when the call has g distinct experts (the kernel's slot order).""" + ng = (g + 3) // 4 + tail = g - 4 * (ng - 1) + if p >= ng: + return 0 + if perm and tail < 4 and ng >= max(3, min_ng): + return tail if p == 1 else 4 + return min(4, g - 4 * p) + + +def make_phase_plan16(num_ctas: int, m_max: int, lag: int = 2, perm: bool = True, min_ng: int = 5): + """op.make_phase_plan (early0, tail_w 1) with the kernel's group sizes (PERM: a partial last group is group 1).""" + S1 = 6 + UNIT_BYTES = 32 * (HIDDEN // 2 + HIDDEN // 16) + ITEM = UNIT_BYTES // S1 + blk_bytes = 32 * (768 // 2 + 768 // 16) + gcap = 4 * m_max + nb2 = HIDDEN // 32 + plan = torch.zeros(gcap + 1, num_ctas, m_max + 1, dtype=torch.int32) + nbs = [nb2 // num_ctas + (1 if c < nb2 % num_ctas else 0) for c in range(num_ctas)] + b0s = [sum(nbs[:c]) for c in range(num_ctas)] + for g in range(4, gcap + 1): + ng = (g + 3) // 4 + for c in range(num_ctas): + plan[g, c, m_max] = b0s[c] | (nbs[c] << 16) + for p in range(ng): + n_p = _group_slots(g, p, perm, min_ng) + total = n_p * U1 * S1 + fc2 = [0] * num_ctas + for c in range(num_ctas): + if p >= lag: + fc2[c] += nbs[c] * _group_slots(g, p - lag, perm, min_ng) * blk_bytes + if p == ng - 1 and not (p == 0 and m_max >= 2): + for q in range(max(0, ng - lag), ng): + fc2[c] += nbs[c] * _group_slots(g, q, perm, min_ng) * blk_bytes + items = [0] * num_ctas + if total >= S1 * num_ctas: + + def counts_for(limit): + return [max(S1, (limit - f) // ITEM) for f in fc2] + + lo, hi = 0, max(fc2) + total * ITEM + while lo < hi: + mid = (lo + hi) // 2 + if sum(counts_for(mid)) >= total: + hi = mid + else: + lo = mid + 1 + items = counts_for(lo) + extra = sum(items) - total + while extra > 0: + c = min( + (c for c in range(num_ctas) if items[c] > S1), + key=lambda c: (-(fc2[c] + items[c] * ITEM), c), + ) + items[c] -= 1 + extra -= 1 + else: + for c in sorted(sorted(range(num_ctas), key=lambda c: (fc2[c], c))[: total // S1]): + items[c] = S1 + i0 = 0 + for c in range(num_ctas): + if items[c]: + plan[g, c, p] = i0 | ((i0 + items[c]) << 16) + i0 += items[c] + assert i0 == total + return plan, max(nbs) + + +_ENGINE16_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "m3_moe16_kernel.py") +M16_MIN_TOKENS, M16_MAX_TOKENS = 9, 16 +# Default routed CTAs (beside the SHARED_CTAS shared-expert CTAs). +M16_NUM_CTAS = 128 +# Phase lag: phase p streams FC1 of group p and FC2 of group p - M16_LAG (the kernel's default lag). +M16_LAG = 3 + + +class M3Moe16State: + """Workspace + compiled kernel of ``m3_moe16`` for ``num_tokens`` (9..16) tokens per call.""" + + fused_router = True + shared = True + + def __init__(self, device: torch.device, num_tokens: int, num_ctas: Optional[int] = None): + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "M3Moe16State allocates and compiles: build it before CUDA-graph capture" + ) + if not M16_MIN_TOKENS <= num_tokens <= M16_MAX_TOKENS: + raise ValueError( + f"m3_moe16 takes {M16_MIN_TOKENS}..{M16_MAX_TOKENS} tokens per call, not {num_tokens}" + ) + self.device = torch.device(device) + ok, why = is_supported(self.device) + if not ok: + raise RuntimeError(f"m3_moe16: {why}") + sms = torch.cuda.get_device_properties(self.device).multi_processor_count + self.num_ctas = num_ctas or M16_NUM_CTAS + if not 128 <= self.num_ctas <= sms - SHARED_CTAS: + raise ValueError( + f"m3_moe16 runs 128 .. {sms - SHARED_CTAS} routed CTAs + {SHARED_CTAS} shared (all resident)" + ) + self.m = num_tokens + plan, nb_cap = make_phase_plan16(self.num_ctas, num_tokens, M16_LAG) + self.config = { + "m_max": num_tokens, + "num_ctas": self.num_ctas, + "nb2_cap": nb_cap, + "fused_router": 1, + "timing": 0, + "shared": 1, + } + try: + self.mod = mod = _kernel_module(self.config, _ENGINE16_PATH) + except AssertionError as err: + raise ValueError( + f"m3_moe16: num_tokens={num_tokens} does not fit the smem / TMEM budget: {err}" + ) from err + kw = dict(device=self.device) + self.plan = plan.to(self.device) + # split units' head sums: every slot empty (0xFFFFFFFF) between calls + self.partials = torch.full((mod.G_CAP * U1 * num_tokens * 32,), -1, dtype=torch.int32, **kw) + # FC2-B images per group + the B1 image; counters: group g at 8 g, B1 at 8 NG_CAP, CTA tickets at 8 NG_CAP + 8 + self.img = torch.zeros(mod.IMG_TOTAL // 4, dtype=torch.int32, **kw) + self.cnt = torch.zeros(mod.NG_CAP * 8 + 16, dtype=torch.int32, **kw) + self.epochs = torch.zeros(self.num_ctas + mod.NSH, dtype=torch.int32, **kw) + self.route_dbg = torch.zeros(num_tokens, 8, dtype=torch.int32, **kw) + self.tdbg = torch.zeros(self.num_ctas + mod.NSH, 64, dtype=torch.int64, **kw) + self.lgbuf = torch.zeros( + mod.LG_REPL * mod.LG_STRIDE, dtype=torch.int64, **kw + ) # logits: tag << 32 | f32 + self._ar_off = torch.zeros(8, dtype=torch.int64, **kw) + # shared expert: h and y_sh as (call tag << 16 | bf16), tag 0 = never written + self.hx = torch.zeros(num_tokens * SHARED_INTER, dtype=torch.int32, **kw) + self.dnx = torch.zeros(num_tokens * HIDDEN, dtype=torch.int32, **kw) + self._fn = None + + def layer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=None, + fc31_input_scale=None, + shared_weights=None, + ) -> M3MoeLayer: + layer = M3MoeLayer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight, + fc31_input_scale, + ) + layer.shared_weights = _shared_views(shared_weights, "m3_moe16") + layer.shared_mxfp8 = tuple(shared_weights) + return layer + + def run(self, layer, hidden, out=None, arp=None) -> torch.Tensor: + m = self.m + if ( + hidden is None + or hidden.dtype != torch.bfloat16 + or tuple(hidden.shape) != (m, HIDDEN) + or not hidden.is_contiguous() + ): + raise ValueError(f"m3_moe16: expects contiguous bf16 hidden states [{m}, {HIDDEN}]") + if out is None: + out = torch.empty(m, HIDDEN, dtype=torch.bfloat16, device=self.device) + args = ( + layer.w3_w1_weight, + layer.w3_w1_weight_scale.view(-1), + layer.w2_weight, + layer.w2_weight_scale.view(-1), + layer.bias, + layer.escal, + self.plan.view(-1), + self.partials, + self.img, + self.cnt, + layer.w3_w1_weight_scale.reshape(-1).view(torch.int32), + layer.w2_weight_scale.reshape(-1).view(torch.int32), + layer.w3_w1_weight.reshape(-1).view(torch.int8), + layer.w2_weight.reshape(-1).view(torch.int8), + self.epochs, + out.view(-1), + self.route_dbg.view(-1), + self.tdbg.view(-1), + hidden.view(-1).view(torch.int32), + layer.router_weight.view(-1), + layer.fc31_input_scale.view(-1), + self.lgbuf, + *layer.shared_weights, + self.hx, + self.dnx, + self._ar_off if arp is None else arp, + ) + if self._fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_moe16 compiles on its first call: call it once before capture" + ) + import cutlass.cute as cute + + sig = [_view(t, 16 if t.element_size() * t.numel() % 16 == 0 else 4) for t in args] + key = (_ENGINE16_PATH,) + tuple(sorted(self.config.items())) + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute.compile( + self.mod.m3_moe, + *sig, + cute.runtime.make_fake_stream(), + options="--enable-tvm-ffi", + ) + self._fn = fn + self._fn(*args, torch.cuda.current_stream(self.device).cuda_stream) + return out diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_prequant.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_prequant.py new file mode 100644 index 000000000000..48a7827f59b3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_prequant.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Hand-off of an MXFP8-quantized activation from the kernel that produced it to the next MXFP8 GEMM. + +A producer that already holds a BF16 row in registers (the multicast all-reduce + RMSNorm, the split-K attention +merge) also writes its MXFP8 form (E4M3 rows + UE8M0 scales in the 128x4-swizzled layout of +``trtllm::mxfp8_quantize(x, True)``) and publishes it here keyed by the BF16 tensor. The MXFP8 linear whose input +is that tensor takes it and skips its own quantize kernel. Every take clears the slot, so a hand-off is only ever +consumed by the first MXFP8 linear that runs after the producer. + +Module state shared across files (one instance each per process): + +* ``SLOT`` (PrequantSlot): published by ``m3_mc_allreduce`` (``quant=True``), the sparse-decode merge and the fused + decode attention (MXFP8 output); cleared by the sparse decode before it runs; taken by the MXFP8 linear dispatch + and by the one-launch QKV + attention + o_proj path (which also peeks ``pending``). +* ``AR_PUSH`` (ArPushSlot): armed by the fast forward around the attention call; the o_proj GEMV takes the target + and marks ``pushed``; the fast forward checks ``take_pushed`` before the attention boundary all-reduce. +* ``MOE_PUSH`` (MoePushSlot): armed by the fast forward before the MoE; the routed-expert engine takes ``engine`` + and the shared expert's down projection takes ``shared``, each marks its pushed output; the fast forward + checks ``take_pushed`` before the MoE boundary all-reduce. +""" + + +class PrequantSlot: + def __init__(self): + self.pending = None # (data_ptr, rows, width, e4m3 [rows, width], scales) + + def publish(self, x, xq, xsf): + self.pending = (x.data_ptr(), x.shape[0], x.shape[-1], xq, xsf) + + def clear(self): + self.pending = None + + def take(self, x): + """(e4m3 [x.shape[0], width], scales) if x is the published tensor, else None.""" + pending, self.pending = self.pending, None + if ( + pending is None + or x.dim() != 2 + or x.data_ptr() != pending[0] + or x.shape[0] != pending[1] + or x.shape[1] != pending[2] + ): + return None + return pending[3], pending[4] + + +SLOT = PrequantSlot() + + +class ArPushSlot: + """The attention boundary's all-reduce exchange for the o_proj GEMV to push into. + + The fast forward arms ``target`` (m3_mx_gemv ``ar`` tuple) around the attention call; the o_proj marks the output + it pushed in ``pushed``; the boundary then runs the all-reduce with ``prepushed`` = the ``ar`` tuple's mode.""" + + def __init__(self): + self.target = None + self.pushed = None # data_ptr of the pushed output + + def arm(self, target): + self.target = target + self.pushed = None + + def take_target(self): + target, self.target = self.target, None + return target + + def take_pushed(self, x): + pushed, self.pushed = self.pushed, None + return pushed is not None and x.data_ptr() == pushed + + +AR_PUSH = ArPushSlot() + + +class MoePushSlot: + """The MoE boundary's all-reduce exchange for its two producers to push into: the routed-expert engine (set A, + ``engine`` = its ``arp`` descriptor) and the shared expert's down projection (set B, ``shared`` = m3_mx_gemv + ``ar`` tuple). Each marks the output it pushed; the boundary then runs the all-reduce with ``prepushed`` = 3.""" + + def __init__(self): + self.engine = None + self.shared = None + self.engine_pushed = None # data_ptr of the pushed routed output + self.shared_pushed = None # data_ptr of the pushed shared output + + def arm(self, engine, shared): + self.engine, self.shared = engine, shared + self.engine_pushed = self.shared_pushed = None + + def disarm(self): + self.engine = self.shared = None + + def take_engine(self): + engine, self.engine = self.engine, None + return engine + + def take_shared(self): + shared, self.shared = self.shared, None + return shared + + def take_pushed(self, shared, routed): + """(routed pushed, shared pushed) for these outputs; clears the marks.""" + e = routed is not None and self.engine_pushed == routed.data_ptr() + s = shared is not None and self.shared_pushed == shared.data_ptr() + self.engine_pushed = self.shared_pushed = None + return e, s + + +MOE_PUSH = MoePushSlot() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_router_gemv.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_router_gemv.py new file mode 100644 index 000000000000..30855eb7f05d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_router_gemv.py @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Low-M fp32 router projection: ``logits[M, E] = float(x[M, K]) @ W[E, K]^T`` in one kernel. + +Replaces the bf16->fp32 cast, the TF32 GEMM and its split-K reduce on decode +steps. One CTA per ``ROWS`` experts: the fp32 weight rows are bulk-copied into +shared memory before ``griddepcontrol.wait`` (overlapping the producer kernel +under PDL), the bf16 activations after it. Products and sums are exact fp32 FMA. + +The kernels are ``torch.ops.trtllm.minimax_m3_router_gemv`` / ``_router_gemv_fp4`` / ``_router_tc``, reached +through the ``ops/moe`` entries. +""" + +from typing import Optional + +import torch + +from ..ops.moe.m3_router_gemv import m3_router_gemv as _router_gemv_op +from ..ops.moe.m3_router_gemv_fp4 import m3_router_gemv_fp4 as _router_gemv_fp4_op +from ..ops.moe.m3_router_tc import m3_router_tc as _router_tc_op + +# Opt-in dynamic shared memory left after the kernel's static reduction buffer (SM100: 227 KiB). +_MAX_DYNAMIC_SMEM = 227 * 1024 - 1024 # capacity: fixed: the device's shared-memory limit + + +def m3_router_gemv( + x: torch.Tensor, weight: torch.Tensor, rows: int = 1, use_pdl: bool = False +) -> Optional[torch.Tensor]: + """fp32 ``x @ weight.T`` for bf16 ``x`` with at most 16 rows; None when unsupported (only ``rows=1`` is built).""" + if ( + rows != 1 + or not x.is_cuda + or x.dtype != torch.bfloat16 + or x.dim() != 2 + or x.stride(1) != 1 + or not 1 <= x.shape[0] <= 16 + or weight.dtype != torch.float32 + or not weight.is_contiguous() + or weight.shape[1] != x.shape[1] + or x.shape[1] % 1024 != 0 + or weight.shape[0] % rows != 0 + or x.data_ptr() % 16 != 0 + or (x.stride(0) * 2) % 16 != 0 + or torch.cuda.get_device_capability(x.device)[0] < 10 + or rows * x.shape[1] * 4 + x.shape[0] * x.shape[1] * 2 > _MAX_DYNAMIC_SMEM + ): + return None + return _router_gemv_op(x, weight, rows, use_pdl) + + +def m3_router_gemv_fp4( + x: torch.Tensor, weight: torch.Tensor, sf_scale: torch.Tensor, use_pdl: bool = False +) -> Optional[tuple]: + """Router logits plus the NVFP4 activation the routed experts consume (bit-identical to + ``fp4_quantize(x, sf_scale, 16, False, False)``): (logits [M, E] fp32, x_fp4 [M, K / 2] uint8, + x_sf [M * K / 16] uint8 linear). None when unsupported.""" + if ( + not x.is_cuda + or x.dtype != torch.bfloat16 + or x.dim() != 2 + or x.stride(1) != 1 + or not 1 <= x.shape[0] <= 16 + or weight.dtype != torch.float32 + or not weight.is_contiguous() + or weight.shape[1] != x.shape[1] + or x.shape[1] % 1024 != 0 + or x.data_ptr() % 16 != 0 + or (x.stride(0) * 2) % 16 != 0 + or sf_scale is None + or sf_scale.dtype != torch.float32 + or sf_scale.numel() != 1 + or torch.cuda.get_device_capability(x.device)[0] < 10 + or x.shape[1] * 4 + x.shape[0] * x.shape[1] * 2 > _MAX_DYNAMIC_SMEM + ): + return None + return tuple(_router_gemv_fp4_op(x, weight, sf_scale.reshape(1), use_pdl)) + + +def m3_router_tc( + x: torch.Tensor, + weight: torch.Tensor, + sf_scale: Optional[torch.Tensor] = None, + use_pdl: bool = False, +) -> Optional[tuple]: + """Router logits (TF32 tensor cores, like the native TF32 GEMM) for 1..48 tokens with 128 experts, plus the + NVFP4 activation when ``sf_scale`` is given. Returns (logits,) or (logits, x_fp4, x_sf); None when unsupported.""" + if ( + not x.is_cuda + or x.dtype != torch.bfloat16 + or x.dim() != 2 + or x.stride(1) != 1 + or not 1 <= x.shape[0] <= 48 + or weight.dtype != torch.float32 + or weight.shape[0] != 128 + or not weight.is_contiguous() + or weight.shape[1] != x.shape[1] + or x.shape[1] % 1024 != 0 + or (x.stride(0) * 2) % 16 != 0 + or x.data_ptr() % 16 != 0 + or torch.cuda.get_device_capability(x.device)[0] < 10 + or 16 * (x.shape[1] // 8 + 4) * 4 + x.shape[0] * (x.shape[1] // 8 + 8) * 2 + > _MAX_DYNAMIC_SMEM + or (sf_scale is not None and (sf_scale.dtype != torch.float32 or sf_scale.numel() != 1)) + ): + return None + res = _router_tc_op(x, weight, None if sf_scale is None else sf_scale.reshape(1), use_pdl) + return tuple(res) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_sampled_accept.py new file mode 100644 index 000000000000..59c58336c5d7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_sampled_accept.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused sampled verification for the MiniMax-M3 Eagle3 target head (TP-sharded vocabulary, decode rows <= 48). + +Replaces, per decode step, the full-vocabulary all-gather of the target logits, the fused sampling kernel and the +strict-acceptance ops for rows sampled with temperature and top-p (the AgentX contract). Rows with top-k or min-p +active must stay on the generic path; the caller decides. + +Sampling is top-p rejection sampling, distributed over the TP ranks. With w = exp(logit / T - max) over the FULL +vocabulary and Z = sum(w), a token is in the nucleus iff the mass strictly above its weight is at most top_p * Z +(the fused sampling kernel's kept set, ties included). A candidate is drawn proportionally to w and accepted iff it +is in the nucleus; a rejected candidate lies below the nucleus, so the next candidate is drawn over the tokens heavier +than it. The accepted token is distributed as w restricted to the nucleus. The first candidate lands in the nucleus +with probability >= top_p, so most rows decide on their second exchange. + +Per verify row, 8 CTAs (rows <= 32) or 4 CTAs (more rows) on every rank each own a slice of the rank's logits shard, +staged in shared memory once. A candidate is drawn in two stages without per-token noise: each CTA draws a local +candidate by inverse CDF over its slice, and the CTA whose log-mass plus Gumbel noise is largest supplies the +candidate (CTA j wins with probability T_j / sum T, its token with w / T_j). Uniforms come from a counter-based Philox +keyed by the request's seed and offset, the row, the attempt and the CTA. Each round every CTA multicasts one 4-word +record (key, candidate logit, and max and mass in round 0 or the mass above the current candidate later) and reads +the records of all CTAs of the row on all ranks. Every CTA takes the same decision from the same records in the same +order, so all ranks agree without any intra-GPU handshake. Words carry the call's tag: a slot needs no clearing and +two call halves keep a rank one call ahead from overwriting a slot still being read. +""" + +from typing import Optional + +import torch + +from ..ops.spec.m3_sampled_accept import m3_sampled_accept as _sampled_accept_op +from ..ops.spec.m3_sampled_accept_ws_words import m3_sampled_accept_ws_words as _ws_words_op + +# The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.cu) and reached through +# the ops/spec entry m3_sampled_accept. +MAX_ROWS = 48 # capacity: fixed: the kernel's built rows (decode steps) + + +class SampledAcceptWorkspace: + """Exchange slots behind one multicast mapping (two call halves), the (finished rows, call) counters, the per-row + token scratch and an extra-attempt counter. Created collectively, outside CUDA-graph capture.""" + + def __init__(self, uc, mc_ptr, rank, world, handle=None, comm=None): + self.uc, self.mc_ptr, self.rank, self.world = uc, mc_ptr, rank, world + self.handle, self.comm = handle, comm + dev = uc.device + self.counters = torch.zeros(2, dtype=torch.int32, device=dev) + self.tokens = torch.zeros(MAX_ROWS, dtype=torch.int32, device=dev) + self.extra_attempts = torch.zeros(1, dtype=torch.int32, device=dev) + + @classmethod + def create(cls, mapping, device) -> "SampledAcceptWorkspace": + words = int(_ws_words_op(max(mapping.tp_size, 1))) + if mapping.tp_size == 1: + uc = torch.zeros(words, dtype=torch.int64, device=device) + return cls(uc, uc.data_ptr(), 0, 1) + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "SampledAcceptWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 8, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("SampledAcceptWorkspace.create failed on another rank") + return cls(uc, mc.data_ptr(), mapping.tp_rank, mapping.tp_size, handle, comm) + + +_WS: Optional[SampledAcceptWorkspace] = None + + +def workspace(mapping, device) -> Optional[SampledAcceptWorkspace]: + global _WS + if _WS is None: + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = SampledAcceptWorkspace.create(mapping, device) + return _WS + + +def sampled_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + temperatures: torch.Tensor, + top_ks: torch.Tensor, + top_ps: torch.Tensor, + min_ps: Optional[torch.Tensor], + seeds: torch.Tensor, + offsets: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, + ws: Optional[SampledAcceptWorkspace] = None, + use_pdl: bool = True, +): + """Sampled verification (temperature + top-p) from this rank's bf16 logits shard [nreq * (draft_len + 1), V / tp]: + outputs as shard_accept (accepted [nreq, D+1], num_accepted [nreq], gather_ids [nreq]); None + when unavailable. Per-row sampling parameters and Philox (seed, offset) are the spec metadata's, one per row. + top_ks / min_ps are not applied: rows with top-k or min-p active must be routed to the generic path by the + caller.""" + del top_ks, min_ps + nreq, draft_len = draft_tokens.shape + rows = nreq * (draft_len + 1) + if logits.dim() != 2 or logits.shape[0] != rows or rows > MAX_ROWS or logits.shape[1] > 51200: + return None + if logits.dtype != torch.bfloat16 or logits.stride(1) != 1: + return None + ws = ws if ws is not None else workspace(mapping, logits.device) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + _sampled_accept_op( + logits, + int(vocab_offset), + temperatures[:rows].float().contiguous(), + top_ps[:rows].float().contiguous(), + seeds[:rows].long().contiguous(), + offsets[:rows].long().contiguous(), + ws.uc, + ws.mc_ptr, + ws.counters, + ws.tokens, + ws.rank, + ws.world, + draft_tokens.int().contiguous(), + batch_idx.int().contiguous(), + accepted, + num_accepted, + gather_ids, + int(force_total), + next_new, + force_vec, + ws.extra_attempts, + bool(use_pdl), + ) + return accepted, num_accepted, gather_ids diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_shard_accept.py new file mode 100644 index 000000000000..2b7d05ad920f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_shard_accept.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 verification for the MiniMax-M3 target head (vocab-sharded TP head, verify rows <= 48). + +Replaces the all-gather of the target logits, the argmax and the strict-acceptance ops of a greedy verify step with +one kernel over this rank's BF16 logits shard (the target lm_head's own local GEMM output): + +* Every logit becomes a 64-bit key ``ordered(value) << 32 | ~global_index`` (ties pick the lowest index, as + ``torch.argmax``), max-reduced per row: warp shuffles, then ``atomicMax``. +* The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping + (Lamport: 0 = empty, the key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the + slots and runs the strict acceptance. Call n uses slot half n % 2. + +The argmax is taken over the BF16 logits without arithmetic, so the tokens are those of the in-tree path's argmax +over the gathered logits. The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/ +shardAccept.cu) and reached through the ``ops/spec`` entry ``m3_shard_accept``. +""" + +from dataclasses import dataclass +from typing import Any, Optional + +import torch + +from ..ops.spec.m3_shard_accept import m3_shard_accept as _shard_accept_op + +#: Verify rows per call, and exchange rows per call half of the workspace. +SHARD_ROWS = 48 # capacity: fixed: DECODE_MAX_TOKENS verify rows + + +@dataclass(eq=False) +class AcceptWorkspace: + """Per-rank exchange slots ([2 halves][SHARD_ROWS rows][world] uint64) behind one multicast mapping, the local + key accumulators and the (finished CTAs, call) counters.""" + + uc: torch.Tensor + mc_ptr: int + keys: torch.Tensor + counters: torch.Tensor + rank: int + world_size: int + handle: Any = None + comm: Any = None + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def create_workspace(mapping, device, slot_rows: int) -> AcceptWorkspace: + """Exchange slots ([2 halves][slot_rows rows][world] uint64) behind one multicast mapping over the TP group, the + local key accumulators [slot_rows] and the (finished CTAs, call) counters. Collective over the TP group; call + outside CUDA-graph capture, in the same order on every rank.""" + words = 2 * slot_rows * max(mapping.tp_size, 1) + if mapping.tp_size > 1: + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + if not _mnnvl_workspace_all_succeeded(comm, True): + comm.Free() + raise RuntimeError("exchange workspace creation failed on another rank") + handle = _make_mnnvl_mcast_buffer( + comm, max(words * 8, 4096), mapping, mapping.is_multi_node() + ) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("exchange workspace creation failed on another rank") + mc_ptr, rank = mc.data_ptr(), mapping.tp_rank + else: + uc = torch.zeros(words, dtype=torch.int64, device=device) + mc_ptr, rank, handle, comm = uc.data_ptr(), 0, None, None + return AcceptWorkspace( + uc=uc, + mc_ptr=mc_ptr, + keys=torch.zeros(slot_rows, dtype=torch.int64, device=uc.device), + counters=torch.zeros(2, dtype=torch.int32, device=uc.device), + rank=rank, + world_size=max(mapping.tp_size, 1), + handle=handle, + comm=comm, + ) + + +def _workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace, created collectively outside CUDA-graph capture on first use.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = create_workspace(mapping, device, SHARD_ROWS) + return _WS + + +def shard_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, +): + """Greedy verification from this rank's BF16 logits shard. + + Args: + logits: [nreq * (draft_len + 1) <= SHARD_ROWS, V / tp] BF16, unit column stride: this rank's columns + (global ids ``vocab_offset ..``) of the verify rows, request-major. + draft_tokens: [nreq, draft_len] int32, the drafts being verified. + batch_idx: [nreq] int32, each request's row in the verify layout. + force_total / force_vec: forced acceptance (see ``m3_shard_accept``). + next_new: optional int32 [nreq, >= 1]; column 0 receives the last accepted token. + + Returns: + (accepted_tokens [nreq, draft_len + 1] int32, num_accepted [nreq] int32, first-pass gather ids [nreq] + int64), or None when the shape is not served or the workspace is missing during a capture. Collective over + the TP group. + """ + nreq, draft_len = draft_tokens.shape + if ( + logits.dim() != 2 + or logits.shape[0] != nreq * (draft_len + 1) + or logits.shape[0] > SHARD_ROWS + or logits.dtype != torch.bfloat16 + or logits.stride(1) != 1 + ): + return None + ws = _workspace(mapping, logits.device) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + _shard_accept_op( + logits, + vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_tokens.contiguous(), + batch_idx, + accepted, + num_accepted, + gather_ids, + force_total, + next_new, + _num_sms(), + force_vec, + ) + return accepted, num_accepted, gather_ids diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_spec_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_spec_tail.py new file mode 100644 index 000000000000..0d4dc9cd1b64 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_spec_tail.py @@ -0,0 +1,116 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Small fused kernels of the MiniMax-M3 Eagle3 step (GB300, decode). + +``chunk_norm`` / ``capture_norm`` apply the drafter's per-layer fc_norm RMSNorms to the captured target hidden states +(the in-tree ``apply_eagle3_fc`` norms, same fp32 arithmetic); ``draft_input`` is a fused draft pass's input (the +embedding lookup, the drafter layer's input_layernorm / hidden_norm and their concat, same arithmetic); +``step_gather`` / ``slot_scatter`` move the step's integer outputs between the spec sampler's store and the engine's +input buffers. +""" + +from typing import Optional, Tuple + +import torch + +# The kernels are compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu) and reached through the +# ops/spec entries. +from ..ops.spec.m3_capture_norm import m3_capture_norm as _capture_norm_op +from ..ops.spec.m3_chunk_norm import m3_chunk_norm as _chunk_norm_op +from ..ops.spec.m3_draft_input import m3_draft_input as _draft_input_op +from ..ops.spec.m3_slot_scatter import m3_slot_scatter as _slot_scatter_op +from ..ops.spec.m3_step_gather import m3_step_gather as _step_gather_op + + +def draft_input( + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + eps_embed: float, + w_hidden: torch.Tensor, + eps_hidden: float, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """(cat([rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden], -1) [M, 2H], hidden[rows] [M, H]) for + int32 ids [M] and int64 rows [M] (rows None: hidden[:M]); ids outside the table embed to zero.""" + m, h = ids.shape[0], table.shape[1] + out = torch.empty((m, 2 * h), dtype=table.dtype, device=table.device) + residual = torch.empty((m, h), dtype=table.dtype, device=table.device) + return _draft_input_op( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, use_pdl + ) + + +def chunk_norm( + src: torch.Tensor, weights: torch.Tensor, eps: float, use_pdl: bool = True +) -> torch.Tensor: + """Per-chunk RMSNorm of src [M, >= C*H] (row stride may exceed C*H) with weights [C, H] -> [M, C*H].""" + c, h = weights.shape + out = torch.empty((src.shape[0], c * h), dtype=src.dtype, device=src.device) + return _chunk_norm_op(src, weights, eps, out, use_pdl) + + +# The overlap scheduler's per-step gathers and the spec sampler's slot stores as one kernel each (step_gather / +# slot_scatter) instead of ~10 and 4 small torch ops; called by the target's SpecStepFold. + + +def step_gather( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 +) -> bool: + """See step_gather_kernel; False when the inputs are outside what the kernel covers (caller falls back).""" + ts = (new_tok, next_draft, lens, slots, ppi, ids, draft, pos_off, kv_off) + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or next_draft.dim() != 2 + or new_tok.dim() < 2 + ): + return False + if (new_tok.dim() == 3 and new_tok.shape[2] != 1) or not ( + lens.is_contiguous() + and ids.is_contiguous() + and draft.is_contiguous() + and pos_off.is_contiguous() + and kv_off.is_contiguous() + ): + return False + _step_gather_op( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 + ) + return True + + +def slot_scatter( + slots, src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft +) -> bool: + """See slot_scatter_kernel (2-D int32 views, zero padding); False to fall back.""" + ts = (src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft) + if any(t.dtype != torch.int32 or not t.is_cuda for t in ts) or slots.dtype != torch.long: + return False + two = (src_new, dst_new, src_next_new, dst_next_new, src_draft, dst_draft) + if any(t.dim() != 2 for t in two) or src_lens.dim() != 1 or dst_lens.dim() != 1: + return False + _slot_scatter_op( + slots, + src_new, + dst_new, + src_next_new, + dst_next_new, + src_lens, + dst_lens, + src_draft, + dst_draft, + ) + return True + + +def capture_norm( + h: torch.Tensor, + residual: Optional[torch.Tensor], + weight: torch.Tensor, + eps: float, + dst: torch.Tensor, +) -> None: + """dst [M, H] (row stride may exceed H) = rmsnorm(h + residual) * weight with chunk_norm's rounding.""" + _capture_norm_op(h, residual, weight, eps, dst) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_target_head.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_target_head.py new file mode 100644 index 000000000000..f5a1b229c777 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/m3_target_head.py @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The target lm_head shard of a verify step of at most 8 rows, streamed from a lossless 12-bit coding of its BF16 +weight (75% of the bytes; ``ops.gemm.m3_target_head`` has the format). + +``prepare_head`` codes this rank's shard once, before the executor's warmup, and keeps it only when it decodes to the +BF16 weight bit for bit (``round_trip_ok``); a shard that does not keeps the in-tree BF16 GEMM. ``logits`` runs the +kernel: BF16 x BF16 MMAs with fp32 accumulation over the full K and one BF16 rounding per logit, the in-tree head's +arithmetic in another summation order. The kernel is compiled into TensorRT-LLM +(cpp/tensorrt_llm/kernels/minimaxM3/targetHead.cu) and called through the catalog entry ``ops.gemm.m3_target_head``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +import torch + +from ..ops.gemm.m3_target_head import CTA_WARPS, MAX_ROWS, encode_chunked, round_trip_ok +from ..ops.gemm.m3_target_head import m3_target_head as _target_head_op + + +@dataclass(eq=False) +class TargetHead: + """One rank's target lm_head shard in the 12-bit coding, and the kernel's split-tile scratch. + + ``coded`` is False when the coding did not reproduce the weight bit for bit (the in-tree GEMM then runs).""" + + weight: torch.Tensor + coded: bool + w: Optional[torch.Tensor] = None + base: int = 0 + exc_off: Optional[torch.Tensor] = None + exc_pos: Optional[torch.Tensor] = None + exc_w: Optional[torch.Tensor] = None + part: Optional[torch.Tensor] = None + counter: Optional[torch.Tensor] = None + grid: int = 0 + + +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def head_supported(lm_head) -> bool: + """A bias-free 2-D BF16 head of whole 16-row tiles whose K the coding covers (a multiple of 64).""" + weight = getattr(lm_head, "weight", None) + if weight is None or weight.dim() != 2 or weight.dtype != torch.bfloat16: + return False + return ( + getattr(lm_head, "bias", None) is None + and weight.shape[0] % 16 == 0 + and weight.shape[1] % 64 == 0 + ) + + +def prepare_head(lm_head, grid: Optional[int] = None) -> TargetHead: + """Code ``lm_head``'s local shard and check it decodes to the BF16 weight bit for bit (fails closed: a shard that + does not is returned with ``coded=False`` and nothing kept). Built once, eagerly, before the executor's warmup + (``M3Eagle3Worker.prebuild_target_head``).""" + weight = lm_head.weight + with torch.no_grad(): + w = weight.detach() + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + if not round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w): + return TargetHead(weight=weight, coded=False) + if grid is None: + grid = _num_sms() + n = weight.shape[0] + return TargetHead( + weight=weight, + coded=True, + w=packed, + base=base, + exc_off=exc_off, + exc_pos=exc_pos, + exc_w=exc_w.view(-1), + part=torch.zeros(grid * CTA_WARPS * 2 * 4 * 32, dtype=torch.float32, device=weight.device), + counter=torch.zeros((n + 15) // 16, dtype=torch.int32, device=weight.device), + grid=grid, + ) + + +def logits(head: TargetHead, x: torch.Tensor, use_pdl: bool = True) -> torch.Tensor: + """This rank's BF16 logits shard [rows, n] of the rows x [rows <= MAX_ROWS, K] (BF16) from the coded shard.""" + assert head.coded and 1 <= x.shape[0] <= MAX_ROWS, (head.coded, x.shape) + if x.stride(1) != 1 or x.stride(0) % 8 or x.data_ptr() % 16: + x = x.contiguous() + n = head.weight.shape[0] + out = torch.empty((x.shape[0], n), dtype=torch.bfloat16, device=x.device) + return _target_head_op( + head.w, + x, + out, + head.part, + head.counter, + head.exc_off, + head.exc_pos, + head.exc_w, + head.base, + n, + head.grid, + use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/__init__.py new file mode 100644 index 000000000000..77d7b84d9bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mixture-of-experts entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_moe16_engine.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_moe16_engine.py new file mode 100644 index 000000000000..950891839022 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_moe16_engine.py @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3's MoE layer for 9..13 decode rows in one CuTe DSL launch (``m3_moe16``): the router, the routed NVFP4 +experts and the shared MXFP8 expert, on caller-owned state (an ``M3Moe16State`` and one ``M3MoeLayer`` per MoE +layer).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..m3_moe.op import M3MoeLayer +from ..m3_moe.op16 import M3Moe16State +from .m3_moe_engine import moe_reference + +__all__ = ["M3Moe16State", "m3_moe16_engine"] + + +class M3Moe16Engine(OpWrapper): + """One MiniMax-M3 MoE layer for M (9..13 here; the state takes 9..16) decode rows: + ``layer.state.run(layer, hidden, out, arp)``, bf16(shared + routed) [M, 6144]. + + The same arguments and the same math as ``m3_moe_engine`` (the routed part operation for operation, the shared + expert always folded in) on an ``M3Moe16State``: a different schedule for more rows. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # One build per row count (m_max is a trace-time constant); the target routes 9..13 rows here (spec-decode verify + # of 2 x 6 rows lands at 12). + CELLS: tuple[Cell, ...] = ( + Cell(why="9 rows: the smallest build, a partial last slot group", spec=dict(tokens=9)), + Cell(why="10 rows: its own build (m_max is a trace-time constant)", spec=dict(tokens=10)), + Cell(why="11 rows: its own build, an odd row count", spec=dict(tokens=11)), + Cell(why="12 rows: two requests verifying 5-token drafts", spec=dict(tokens=12)), + Cell(why="13 rows: the largest the target routes here", spec=dict(tokens=13)), + ) + + note = """ + Bands as `m3_moe_engine` (same approximations, same quantization boundaries). Not covered: 14..16 rows, which the + state takes and this target never routes here; the all-reduce push (`arp`). No `is_valid`, as `m3_moe_engine`. + """ + + def __call__( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return layer.state.run(layer, hidden, out, arp=arp) + + def reference( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """``m3_moe_engine``'s reference with the shared expert; `out` and `arp` are ignored.""" + return moe_reference(hidden, layer, shared=True) + + +m3_moe16_engine = M3Moe16Engine() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_moe_engine.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_moe_engine.py new file mode 100644 index 000000000000..b6b0111b949f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_moe_engine.py @@ -0,0 +1,326 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3's MoE layer for 1..8 decode tokens in one CuTe DSL launch (``m3_moe``): the router, the routed NVFP4 +experts and the shared MXFP8 expert, on caller-owned state (an ``M3MoeState`` and one ``M3MoeLayer`` per MoE layer).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.quantization.utils.fp4_utils import ( + get_reorder_rows_for_gated_act_gemm_row_indices, + get_shuffle_matrix_a_row_indices, +) + +from ..m3_moe.op import ( + HIDDEN, + INTER, + SHARED_INTER, + TOP_K, + M3MoeLayer, + M3MoeState, + ar_push_desc, + is_supported, +) + +__all__ = [ + "M3MoeLayer", + "M3MoeState", + "ar_push_desc", + "is_supported", + "m3_moe_engine", + "moe_reference", + "moe_route", + "mxfp8_dequant", + "mxfp8_quant_dequant", + "unswizzle_sf", +] + +# The model's constants the kernels are built with: routed_scaling_factor, the shared expert's SwiGLU-OAI +# (swiglu_alpha, swiglu_limit, the (up + 1) offset). +ROUTED_SCALING = 2.0 +SHARED_ALPHA = 1.702 +SHARED_LIMIT = 7.0 + +_LOG2E = 1.4426950216293334961 +_RCP6 = 0.16666667163372039795 # fp32(1 / 6), the kernels' reciprocal of the E2M1 maximum +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) +_F32_MAX = 3.40282346638528859812e38 + + +def _f32(x: float) -> torch.Tensor: + return torch.tensor(x, dtype=torch.float32) + + +def _e2m1_decode(packed: torch.Tensor) -> torch.Tensor: + """fp32 values of packed E2M1 bytes [..., K / 2] (low nibble first) -> [..., K].""" + lut = torch.tensor(_E2M1, dtype=torch.float32, device=packed.device) + lo = lut[(packed & 0xF).long()] + hi = lut[(packed >> 4).long()] + return torch.stack((lo, hi), dim=-1).flatten(-2) + + +def _e2m1_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to the nearest E2M1 value, ties to even, saturating at 6 (cvt.rn.satfinite.e2m1x2).""" + a = x.abs() + q = torch.full_like(a, 6.0) + for hi, value, inclusive in ( + (5.0, 4.0, True), + (3.5, 3.0, False), + (2.5, 2.0, True), + (1.75, 1.5, False), + (1.25, 1.0, True), + (0.75, 0.5, False), + (0.25, 0.0, True), + ): + q = torch.where(a <= hi if inclusive else a < hi, torch.full_like(a, value), q) + return torch.where(x < 0, -q, q) + + +def _e4m3_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to E4M3, nearest even, saturating (cvt.rn.satfinite.e4m3x2), as fp32.""" + return x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn).float() + + +def _ue8m0_up(x: torch.Tensor) -> torch.Tensor: + """UE8M0 code of x >= 0 rounded toward +inf, saturating (cvt.rp.satfinite.ue8m0x2), int32.""" + bits = x.contiguous().view(torch.int32) + code = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + return code.clamp(0, 254) + + +def unswizzle_sf(sf: torch.Tensor, rows: int, blocks: int) -> torch.Tensor: + """The [rows, blocks] scale bytes of a 128x4-swizzled scale buffer (int64 indices, any byte dtype).""" + flat = sf.reshape(-1).view(torch.uint8) + r = torch.arange(rows, device=flat.device).view(-1, 1) + b = torch.arange(blocks, device=flat.device).view(1, -1) + padded_blocks = (blocks + 3) // 4 * 4 + idx = ( + (r // 128) * (128 * padded_blocks) + + (b // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (b % 4) + ) + return flat[idx] + + +def mxfp8_dequant(q: torch.Tensor, sf: torch.Tensor) -> torch.Tensor: + """fp32 values of e4m3 ``q`` [R, K] with its 128x4-swizzled UE8M0 scales (one per 32 elements).""" + rows, k = q.shape + codes = unswizzle_sf(sf, rows, k // 32).float() + return q.view(torch.float8_e4m3fn).float() * torch.exp2(codes - 127.0).repeat_interleave( + 32, dim=1 + ) + + +def mxfp8_quant_dequant(x: torch.Tensor) -> torch.Tensor: + """x [R, K] through MXFP8 (one UE8M0 scale per 32, the smallest power of two >= amax / 448; e4m3 data), back to + fp32 -- mxfp8_quantize's recipe, which every MXFP8 activation here reproduces.""" + rows, k = x.shape + blocks = x.float().view(rows, k // 32, 32) + code = _ue8m0_up(blocks.abs().amax(dim=-1) * _f32(1.0 / 448.0).to(x.device)) + scale = torch.exp2(code.float() - 127.0).unsqueeze(-1) + return (_e4m3_round(blocks / scale) * scale).view(rows, k) + + +def _bf16(x: torch.Tensor) -> torch.Tensor: + return x.to(torch.bfloat16).float() + + +def _gemm(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """a [M, K] @ b [N, K]^T, accumulated in fp64 and rounded once to fp32.""" + return (a.double() @ b.double().t()).float() + + +def _row_perm(rows: int, gated: bool) -> torch.Tensor: + """The TRTLLM-Gen row order: stored row j holds logical row perm[j] (the gated interleave, then the 32-row shuffle + of epilogue tile 128).""" + probe = torch.empty(rows, 1) + perm = get_shuffle_matrix_a_row_indices(probe, epilogue_tile_m=128) + if gated: + perm = get_reorder_rows_for_gated_act_gemm_row_indices(probe)[perm] + return perm + + +def _nvfp4_expert(w: torch.Tensor, sf: torch.Tensor, perm_inv: torch.Tensor) -> torch.Tensor: + """One expert's stored NVFP4 matrix (values [R, K / 2], swizzled e4m3 scales) as raw fp32 [R, K] in logical row + order (the global scale is not applied: it is the per-expert alpha).""" + rows, half = w.shape + codes = unswizzle_sf(sf, rows, 2 * half // 16).view(torch.float8_e4m3fn).float() + vals = _e2m1_decode(w) * codes.repeat_interleave(16, dim=1) + return vals[perm_inv] + + +def moe_route(x: torch.Tensor, router_weight: torch.Tensor, bias: torch.Tensor): + """MiniMax2 routing on the fp32 router GEMV: (top-4 expert ids [M, 4] int64, bf16-rounded weights [M, 4] fp32).""" + logits = _gemm(x, router_weight) + sig = (0.5 * torch.tanh(0.5 * logits).double() + 0.5).float() + keys = sig + bias + ids = torch.sort(keys, dim=-1, descending=True, stable=True).indices[:, :TOP_K] + sk = keys.gather(1, ids) - bias[ids] + total = (sk[:, 0] + sk[:, 2]) + (sk[:, 1] + sk[:, 3]) + w = (sk * _f32(ROUTED_SCALING).to(x.device)) / (total + _f32(1e-20).to(x.device)).unsqueeze(1) + return ids, _bf16(w) + + +def _nvfp4_act(x: torch.Tensor, sc: torch.Tensor) -> torch.Tensor: + """The activation's NVFP4 quantization (fp4_quantize(x, sc, 16)), as raw fp32 values (q x block scale).""" + rows, k = x.shape + blocks = x.view(rows, k // 16, 16) + vmax = blocks.abs().amax(dim=-1, keepdim=True) + sff = _e4m3_round(sc * (vmax * _f32(_RCP6).to(x.device))) + # The kernels' rcp.approx(sf x rcp.approx(sc)) lands where the correctly rounded sc / sf does on every element + # measured (the 1 / (sf x (1 / sc)) spelling differs on ~0.06% of them). + osc = torch.where(vmax != 0, sc / sff, torch.zeros_like(vmax)) + return (_e2m1_round(blocks * osc) * sff).view(rows, k) + + +def _fc1_epilogue(up: torch.Tensor, gate: torch.Tensor, esc: torch.Tensor) -> torch.Tensor: + """TRTLLM-Gen's SwiGLU-OAI FC1 epilogue on the raw FC1 sums (fp32 [I]) and its NVFP4 requantization: the raw FC2 + input values. ``esc`` = (fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha) of the expert, with + act_beta and act_clamp already divided by fc31_alpha (the loader's convention).""" + a, c, alpha, beta, lim = (esc[i] for i in range(5)) + gc = torch.where(gate > lim, lim, gate) + gc = torch.where(gate < -_F32_MAX, torch.full_like(gc, -_F32_MAX), gc) + uc = torch.minimum(torch.maximum(up, -lim), lim) + c1 = a * (alpha * _f32(_LOG2E).to(up.device)) + sgm = 1.0 / (torch.exp2(-(gc * c1)) + 1.0) + h = (uc.double() * a.double() + (a * beta).double()).float() * (gc * sgm) + blocks = h.view(-1, 16) + amax = (blocks.abs() * c).amax(dim=-1, keepdim=True) + sff = _e4m3_round(amax * _f32(_RCP6).to(up.device)) + osc = torch.where(sff != 0, 1.0 / sff, torch.zeros_like(sff)) + return (_e2m1_round((blocks * osc) * c) * sff).view(-1) + + +def _shared_expert(x: torch.Tensor, shared_mxfp8) -> torch.Tensor: + """The shared expert as the target runs it outside the engine: MXFP8 gate_up (bf16 out), SwiGLU-OAI with the + model's alpha / limit / (up + 1) in fp32 (bf16 out), MXFP8 down (bf16 out).""" + gu_w, gu_sf, dn_w, dn_sf = shared_mxfp8 + gu = _bf16(_gemm(mxfp8_quant_dequant(x), mxfp8_dequant(gu_w, gu_sf))) + gate, up = gu[:, :SHARED_INTER], gu[:, SHARED_INTER:] + a = torch.minimum(gate, torch.full_like(gate, SHARED_LIMIT)) + ub = up.clamp(-SHARED_LIMIT, SHARED_LIMIT) + 1.0 + sg = 1.0 / ( + torch.exp2((a * _f32(-SHARED_ALPHA).to(x.device)) * _f32(_LOG2E).to(x.device)) + 1.0 + ) + h = _bf16(ub * (sg * a)) + return _bf16(_gemm(mxfp8_quant_dequant(h), mxfp8_dequant(dn_w, dn_sf))) + + +def moe_reference(hidden: torch.Tensor, layer: M3MoeLayer, shared: bool) -> torch.Tensor: + """The MoE layer of the decode engines (both token ranges compute this), in plain torch: bf16 [M, 6144].""" + x = hidden.float() + m = x.shape[0] + ids, weights = moe_route(x, layer.router_weight, layer.bias) + xr = _nvfp4_act(x, layer.fc31_input_scale.reshape(()).float()) + inv31 = torch.argsort(_row_perm(2 * INTER, gated=True)).to(x.device) + inv2 = torch.argsort(_row_perm(HIDDEN, gated=False)).to(x.device) + y = torch.zeros(m, TOP_K, HIDDEN, dtype=torch.float32, device=x.device) + for e in torch.unique(ids).tolist(): + w31 = _nvfp4_expert(layer.w3_w1_weight[e], layer.w3_w1_weight_scale[e], inv31) + w2 = _nvfp4_expert(layer.w2_weight[e], layer.w2_weight_scale[e], inv2) + esc = layer.escal[:, e] + for t, k in (ids == e).nonzero().tolist(): + acc1 = _gemm(xr[t : t + 1], w31)[0] + # The buffer holds [w3 (up); w1 (gate)]. + h = _fc1_epilogue(acc1[:INTER], acc1[INTER:], esc) + y[t, k] = _bf16(_gemm(h.unsqueeze(0), w2)[0] * esc[5]) + acc = torch.zeros(m, HIDDEN, dtype=torch.float64, device=x.device) + for k in range(TOP_K): + acc = (acc + weights[:, k : k + 1].double() * y[:, k].double()).float().double() + routed = acc.float() + if not shared: + return routed.to(torch.bfloat16) + return (_shared_expert(x, layer.shared_mxfp8) + _bf16(routed)).to(torch.bfloat16) + + +class M3MoeEngine(OpWrapper): + """One MiniMax-M3 MoE layer for M (1..8) decode tokens: ``layer.state.run(layer, hidden, out, arp)``. + + ``hidden`` bf16 [M, 6144] contiguous, M = the state's token count. ``layer`` (from ``M3MoeState.layer``) carries the + TRTLLM-Gen NVFP4 expert buffers read in place, the per-expert scalars, the router weight / bias, the activation's + NVFP4 global scale and, on a ``shared`` state, the shared expert's MXFP8 weights; its state carries the compiled + kernel and the workspace every layer on it shares. Returns bf16 [M, 6144]: bf16(shared + routed) on a ``shared`` + state, else the routed partial. ``out``: write there instead of a new tensor. ``arp`` (``ar_push_desc``): also + push the rows into a multicast all-reduce exchange. + + The routed part is TRTLLM-Gen's NVFP4 MoE on the fp32 router logits: sigmoid (tanh form) + correction bias top-4 + of 128 (ties to the lower id), weights sigmoid / sum * 2.0 rounded to bf16; the activation quantized to NVFP4 with + the layer's global scale; per expert the raw FC1 sums, SwiGLU-OAI (alpha / beta / clamp), the NVFP4 requant with + fc31_scale_c, FC2, bf16(fc2_alpha x sum); the fma chain over the top-4 in order. The shared expert is the target's + MXFP8 SwiGLU-OAI MLP. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # One cell per token count: m_max is a trace-time constant, so each is its own build, and the target builds all + # eight (decode batches of 1..8 requests, spec-decode verify rows up to 8). The whys name what the build differs in. + CELLS: tuple[Cell, ...] = ( + Cell( + why="1 token: pre-tiled shared expert, last phase balanced without trailing FC2", + spec=dict(tokens=1), + ), + Cell( + why="2 tokens: pre-tiled shared expert, the one build without AR_PRE", + spec=dict(tokens=2), + ), + Cell( + why="3 tokens: 4 FC2-B builder warps, the shared expert from swizzled scales", + spec=dict(tokens=3), + ), + Cell( + why="4 tokens: one token per routing warp, logits replicated 128x", spec=dict(tokens=4) + ), + Cell( + why="5 tokens: 5 builder warps, a routing warp takes two tokens, logits 32x", + spec=dict(tokens=5), + ), + Cell(why="6 tokens: one request verifying a 5-token draft", spec=dict(tokens=6)), + Cell(why="7 tokens: the B2IMG global FC1 -> FC2 hand-off", spec=dict(tokens=7)), + Cell( + why="8 tokens: B2IMG with a 6-stage ring, 32 distinct experts at most", + spec=dict(tokens=8), + ), + ) + + note = """ + The kernel's elementwise math uses the PTX approximations (rcp.approx, ex2.approx, tanh) where `reference` uses + torch's correctly rounded ones, and its GEMMs sum in tensor-core order where `reference` sums in fp64: a + quantization boundary (NVFP4 activation / intermediate code, routing near-tie) can fall on the other side, so the + two agree within the band the test derives, not bit for bit. The routed part is bit for bit the stock TRTLLM-Gen + NVFP4 MoE (fp4_quantize + fp4_block_scale_moe_runner) on the same routing; the test checks that too. + + Not covered: the all-reduce push (`arp`), which needs a multicast exchange (the decode forward's boundary all-reduce + exercises it); the non-shared state (`shared=False`), which this target never builds. Every layer on one state + must run in one stream order: the workspace is shared and tagged per call, never reset. + + No `is_valid`: the compiled call checks every buffer's dtype, shape and alignment against its signature and + raises on a mismatch (a misaligned `hidden`, an `out` that is not bf16 [M, 6144], a malformed `arp`). + """ + + def __call__( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return layer.state.run(layer, hidden, out, arp=arp) + + def reference( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """Plain torch over the layer's buffers (see the class docstring); `out` and `arp` are ignored.""" + return moe_reference(hidden, layer, shared=layer.state.shared) + + +m3_moe_engine = M3MoeEngine() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_gemv.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_gemv.py new file mode 100644 index 000000000000..37870d0e6cd7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_gemv.py @@ -0,0 +1,101 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""fp32 router GEMV of the decode MoE: ``logits[M, E] = float(x) @ W^T`` at one token.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +_E4M3_MAX = 448.0 +_E2M1_MAX = 6.0 +#: Midpoints between consecutive e2m1 magnitudes; ``_TIE_UP[i]``: round-to-nearest-even takes the upper code. +_E2M1_MIDPOINTS = (0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0) +_TIE_UP = (False, True, False, True, False, True, False) +#: The kernel builds each block's output scale through two ``rcp.approx.ftz.f32`` reciprocals (~2^-23 relative): +#: a value this close to an e2m1 midpoint may round to either neighbour, everything further out must match. +TIE_WINDOW = 2.0**-20 + + +def nvfp4_quantize( + x: torch.Tensor, sf_scale: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """``fp4_quantize(x, sf_scale, 16, False, False)``: (packed e2m1 [M, K / 2], linear E4M3 scales [M * K / 16], + the pre-rounding scaled values [M, K] that ``near_tie`` reads). + + Per 16 columns: scale = e4m3(sf_scale * amax / 6), data = x * sf_scale / scale rounded to nearest-even e2m1. + """ + gs = sf_scale.float().reshape(()) + m, k = x.shape + blocks = x.float().view(m, k // 16, 16) + vecmax = blocks.abs().amax(-1) + sf = (gs * (vecmax / _E2M1_MAX)).clamp(max=_E4M3_MAX).to(torch.float8_e4m3fn) + out_scale = torch.where(vecmax != 0, gs / sf.float(), torch.zeros_like(vecmax)) + scaled = (blocks * out_scale.unsqueeze(-1)).view(m, k) + mid = torch.tensor(_E2M1_MIDPOINTS, device=x.device, dtype=torch.float32) + tie_up = torch.tensor(_TIE_UP, device=x.device, dtype=torch.bool) + a = scaled.abs().unsqueeze(-1) + code = ((a > mid).sum(-1) + ((a == mid) & tie_up).any(-1)).to(torch.uint8) + negative = (scaled < 0) | ((scaled == 0) & torch.signbit(scaled)) + code = code | (negative.to(torch.uint8) << 3) + packed = code[:, 0::2] | (code[:, 1::2] << 4) + return packed, sf.view(torch.uint8).reshape(-1), scaled + + +def near_tie(scaled: torch.Tensor) -> torch.Tensor: + """Elements whose e2m1 rounding the kernel's approximate reciprocals decide (see ``TIE_WINDOW``).""" + mid = torch.tensor(_E2M1_MIDPOINTS, device=scaled.device, dtype=torch.float32) + a = scaled.abs().unsqueeze(-1) + return ((a - mid).abs() <= TIE_WINDOW * a).any(-1) + + +def fp32_logits(x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + """``float(x) @ w^T`` as a true fp32 product (TF32 off).""" + with true_fp32_matmul(): + return x.float() @ w.t() + + +class M3RouterGemv(OpWrapper): + """fp32 router logits ``float(x[M, K]) @ w[E, K]^T`` with exact fp32 FMA, one CTA per expert. + + ``x`` is BF16 with 1..16 rows, unit column stride and 16-byte rows; ``w`` is the fp32 router weight; K a + multiple of 1024. ``rows`` experts per CTA; only 1 is built. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The target takes the GEMV at exactly one token (2..48 go to the tensor-core router): 128 experts, hidden 6144. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one decode token, 128 experts x hidden 6144", + spec=dict(m=1, e=128, k=6144), + ), + ) + + note = """ + The op accepts up to 16 rows; the target sends one, so wider calls are not cells. The logits differ from a true + fp32 product only in accumulation order (per-thread FMA chains over K / 1024 strided column groups, a warp tree + and an 8-warp sum). + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor, rows: int, pdl: bool) -> torch.Tensor: + return torch.ops.trtllm.minimax_m3_router_gemv(x, w, rows, pdl) + + def reference(self, x: torch.Tensor, w: torch.Tensor, rows: int, pdl: bool) -> torch.Tensor: + """True fp32 product; ``rows`` and ``pdl`` do not enter the arithmetic.""" + return fp32_logits(x, w) + + def is_valid(self, x: torch.Tensor, w: torch.Tensor, rows: int, pdl: bool) -> None: + # The activation rows are bulk-copied from x's base address; the op checks the row stride but not the base, + # and a misaligned bulk copy faults the context rather than raising. + assert x.data_ptr() % 16 == 0, "x must start 16-byte aligned (cp.async.bulk source)" + + +m3_router_gemv = M3RouterGemv() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_gemv_fp4.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_gemv_fp4.py new file mode 100644 index 000000000000..58f809e82cf6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_gemv_fp4.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""fp32 router GEMV plus the routed experts' NVFP4 activation, one token.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_router_gemv import fp32_logits, nvfp4_quantize + + +class M3RouterGemvFp4(OpWrapper): + """``[logits fp32 [M, E], x_fp4 [M, K / 2] uint8, x_sf [M * K / 16] uint8]`` from one kernel. + + The logits are ``m3_router_gemv``'s; the NVFP4 activation is ``fp4_quantize(x, sf_scale, 16, False, False)`` + (16-element blocks, E4M3 scales in the linear layout, per-tensor ``sf_scale``) for the routed experts' FC1. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The fused-router decode step at one token: 128 experts, hidden 6144, the experts' input scale. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one decode token: logits for 128 experts and the NVFP4 activation of hidden 6144", + spec=dict(m=1, e=128, k=6144, sf_scale=448.0 * 6.0 / 8.0), + ), + Cell( + why="a global scale that puts block scales near the E4M3 top, where the scale cast saturates", + spec=dict(m=1, e=128, k=6144, sf_scale=448.0 * 6.0 / 2.0), + ), + ) + + note = """ + The e2m1 codes are exact except within `TIE_WINDOW` of a rounding midpoint, where the kernel's approximate + reciprocals decide the direction. Up to 16 rows are accepted; the target sends one. + """ + + def __call__( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: torch.Tensor, pdl: bool + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_router_gemv_fp4(x, w, sf_scale, pdl) + + def reference( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: torch.Tensor, pdl: bool + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """True fp32 logits and the NVFP4 recipe; ``pdl`` does not enter the arithmetic.""" + packed, sf, _ = nvfp4_quantize(x, sf_scale) + return fp32_logits(x, w), packed, sf + + def is_valid(self, x: torch.Tensor, w: torch.Tensor, sf_scale: torch.Tensor, pdl: bool) -> None: + assert x.data_ptr() % 16 == 0, "x must start 16-byte aligned (cp.async.bulk source)" + # A zero or non-finite global scale makes every block scale 0 or NaN and the data codes garbage; the op + # reads the scalar on the device and cannot refuse it. + value = float(sf_scale.reshape(())) + assert value > 0.0 and value != float("inf"), ( + f"sf_scale must be positive and finite, got {value}" + ) + + +m3_router_gemv_fp4 = M3RouterGemvFp4() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_tc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_tc.py new file mode 100644 index 000000000000..e88af51f7fa2 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/moe/m3_router_tc.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TF32 tensor-core router of the decode MoE for 2..48 tokens, optionally with the NVFP4 activation.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +from .m3_router_gemv import nvfp4_quantize + + +def tf32_rna(w: torch.Tensor) -> torch.Tensor: + """fp32 -> TF32 as ``cvt.rna.tf32.f32``: keep 10 mantissa bits, round to nearest with ties away from zero.""" + bits = w.contiguous().view(torch.int32) + rounded = (bits + 0x1000) & ~0x1FFF + finite = (bits & 0x7F800000) != 0x7F800000 + return torch.where(finite, rounded, bits).view(torch.float32) + + +class M3RouterTc(OpWrapper): + """Router logits on TF32 tensor cores for 128 experts and 1..48 tokens: ``x @ tf32(w)^T``, fp32 accumulate. + + ``x`` is BF16 (exact in TF32), ``w`` the fp32 router weight, rounded to TF32 in the kernel. The 16 K slices' + partials are summed in slice order whatever cluster size the device co-schedules (16 or 8 CTAs), so the result + does not depend on it. With ``sf_scale`` also the NVFP4 activation as ``m3_router_gemv_fp4``: returns + ``[logits]`` or ``[logits, x_fp4, x_sf]``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The target routes 2..48 tokens here: plain logits on the unfused path, logits + NVFP4 on the fused-router path. + CELLS: tuple[Cell, ...] = ( + Cell( + why="two tokens, logits only (the smallest call the target makes here)", + spec=dict(m=2, k=6144, fp4=False), + ), + Cell( + why="8 tokens with the NVFP4 activation: one 8-token MMA tile, every expert tile quantizes one row", + spec=dict(m=8, k=6144, fp4=True), + ), + Cell( + why="13 tokens: a partial second token tile", + spec=dict(m=13, k=6144, fp4=True), + ), + Cell( + why="the 48-token maximum, six token tiles, logits only", + spec=dict(m=48, k=6144, fp4=False), + ), + Cell( + why="the 48-token maximum with the NVFP4 activation", + spec=dict(m=48, k=6144, fp4=True), + ), + # Every other row count the router takes (dispatch.ROUTER_TC), with and without the NVFP4 activation. + *( + Cell( + why=f"{m} rows: a router row count between the ones above", + spec=dict(m=m, k=6144, fp4=m % 2 == 1), + ) + for m in range(2, 49) + if m not in (2, 8, 13, 48) + ), + ) + + note = """ + The kernel uses 16-CTA clusters when the device co-schedules eight of them and 8-CTA clusters otherwise; both + sum the same 16 slice partials in the same order. Only the one the test device picks is driven. + """ + + def __call__( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_router_tc(x, w, sf_scale, pdl) + + def reference( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> tuple[torch.Tensor, ...]: + """fp32 product of ``x`` and the TF32-rounded weight, plus the NVFP4 recipe with ``sf_scale``.""" + with true_fp32_matmul(): + logits = x.float() @ tf32_rna(w).t() + if sf_scale is None: + return (logits,) + packed, sf, _ = nvfp4_quantize(x, sf_scale) + return logits, packed, sf + + def is_valid( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> None: + assert x.data_ptr() % 16 == 0, "x must start 16-byte aligned (cp.async.bulk source)" + if sf_scale is not None: + value = float(sf_scale.reshape(())) + assert value > 0.0 and value != float("inf"), ( + f"sf_scale must be positive and finite, got {value}" + ) + + +m3_router_tc = M3RouterTc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/msa_kv_update.py new file mode 100644 index 000000000000..8c2a67356d3f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/msa_kv_update.py @@ -0,0 +1,89 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One kernel for MSA's device-side KV-length correction (MsaAttentionMetadata.on_update_kv_lens). + +The torch formulation is ~23 one-CTA kernels (min, cumsum, gathers, div / clamp / remainder, the block-table lookup, +the valid-block counts) and runs at the head of every captured decode step, twice. This kernel computes the same +int32 results in one launch: + + kv_true[b] = min(kv_lens[b], staged[b]) -> seq_lens_out[b], cu_kv[b + 1] = prefix sum (cu_kv[0] = 0) + pos[t] = kv_true[r] - qo[r] + intra[t] (r = q_batch_row[t]) + out_loc[t] = table[r, min(max(pos, 0) // page, ncols - 1)] * page + max(pos, 0) % page + n_valid[t] = (max(pos + 1, 0) + page - 1) // page + +``msa_kv_update`` is the target's ``MsaKvUpdate`` (``tensorrt_llm._torch.pyexecutor.runtime_hooks``): the MSA +metadata calls it with its own buffers and falls back to the torch formulation when it returns False. The kernel is +``torch.ops.trtllm.minimax_m3_msa_kv_update``, called through this target's ``msa_kv_update`` entry. +""" + +from typing import Optional + +import torch + +from ..ops.attention.msa_kv_update import msa_kv_update as _kv_update_op + + +def msa_kv_update( + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], +) -> bool: + """The fused correction (see module doc); False, with nothing written, when an operand is outside what the + kernel covers. Arguments in ``MsaKvUpdate`` order; ``seq_lens`` are the per-request query lengths (``qo``).""" + ts = [ + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + msa_seq_lens, + out_cache_loc, + ] + ts += [t for t in (cu_kv_lens, n_valid_blocks) if t is not None] + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or block_table.dim() != 2 + or block_table.stride(1) != 1 + ): + return False + if any(not t.is_contiguous() for t in ts if t is not block_table): + return False + if ( + kv_lens.numel() < batch + or kv_lens_staged.numel() < batch + or seq_lens.numel() < batch + or msa_seq_lens.numel() < batch + or q_batch_row.numel() < total_q + or q_intra.numel() < total_q + or out_cache_loc.numel() < total_q + or (cu_kv_lens is not None and cu_kv_lens.numel() < batch + 1) + or (n_valid_blocks is not None and n_valid_blocks.numel() < total_q) + ): + return False + _kv_update_op( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/__init__.py new file mode 100644 index 000000000000..ef3a4c5847ea --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) as a CuTe DSL kernel (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/m3_mx_gemv_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/m3_mx_gemv_kernel.py new file mode 100644 index 000000000000..e3d7751231ee --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/m3_mx_gemv_kernel.py @@ -0,0 +1,1238 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 decode GEMV on tcgen05: ``y[M, N] = x[M, K] @ W[N, K]^T`` for M <= 8, block-scaled E4M3 x E4M3. + +The weight is the MMA's A operand (128 output features per CTA), the activation is B (8 token columns), +fp32 accumulation in TMEM. Both operands come in their MXFP8 storage: E4M3 values (K contiguous) and UE8M0 +scales in the 128x4-swizzled layout, whose 512-byte atom per (128 rows, 128 K) is exactly the source of a +``tcgen05.cp`` 32x128b.warpx4 scale copy, so both scale tensors are bulk-copied unchanged. For an activation of at +most 32 rows the atom of k-tile ``k`` sits at byte ``512 k``. + +A cluster of SPLIT CTAs shares one 128-row weight tile; rank r streams the interleaved k-tiles r, r + SPLIT, ... +through a RING-stage ring. Before the grid dependency each CTA issues its first RING weight tiles (and their scale +atoms) and prefetches the rest into L2; after it, the activation slice of the CTA (values and scale atoms) lands in +one buffer. Rank r owns rows [r * 128 / SPLIT, (r + 1) * 128 / SPLIT) of the tile: the other ranks store their fp32 +partials of those rows into its shared memory and arrive on its barrier (release, cluster scope); the owner adds the +SPLIT partials in rank order and stores bf16. + +Warps: 0 weight TMA, 1 activation TMA, 2 TMEM allocation + scale copies + MMA, 3 idle, 4-7 epilogue (warp 4 + i +reads TMEM lanes 32 i .. 32 i + 31). +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass import dsl_user_op +from cutlass.experimental import primitives as prims + +CTA_M = 128 # weight rows per CTA = the MMA's M # capacity: fixed: a kernel tile constant +MMA_N = 8 # token columns +CTA_K = 128 # one k-tile: 128 E4M3 bytes per row = one 128-byte swizzle row # capacity: fixed: a kernel tile constant +MMA_INST_K = 32 +K_BLOCKS = ( + CTA_K // MMA_INST_K +) # 4 MMAs per k-tile, one per scale byte of the atom # capacity: fixed: a kernel tile constant +THREADS = 256 +SF_ATOM_BYTES = 512 +SF_ATOM_U16 = SF_ATOM_BYTES // 2 +SF_COLS = 4 # TMEM columns of one scale atom +W_TILE_BYTES = CTA_M * CTA_K +X_TILE_BYTES = MMA_N * CTA_K +EVICT_FIRST = 0x12F0000000000000 +A_COLS_PER_MMA = MMA_INST_K // 4 # TMEM columns of one 128 x K32 E4M3 granule of A + +a_dtype = cutlass.Float8E4M3FN +b_dtype = cutlass.Float8E4M3FN + + +def num_k_tiles(k_in: int) -> int: + return k_in // CTA_K + + +def acc_cols(nacc: int, mma_n: int = MMA_N) -> int: + # nacc independent accumulators of MMA_N columns: MMA j of the CTA accumulates into j % nacc, so the M128 x N8 + # MMAs do not form one serial dependency chain; the epilogue adds them. + return max(16, nacc * mma_n) + + +def a_col0(nacc: int, my_tiles: int, mma_n: int = MMA_N) -> int: + # a_tmem: [acc | SFA per k-tile | one SFB slot (reused in issue order) | A: 32 columns (128 x 128 E4M3) per k-tile] + return (acc_cols(nacc, mma_n) + (my_tiles + 1) * SF_COLS + 31) // 32 * 32 + + +def tmem_cols( + ring: int, nacc: int, a_tmem: bool = False, my_tiles: int = 0, mma_n: int = MMA_N +) -> int: + need = acc_cols(nacc, mma_n) + 2 * ring * SF_COLS + if a_tmem: + need = a_col0(nacc, my_tiles, mma_n) + my_tiles * K_BLOCKS * A_COLS_PER_MMA + cols = 32 + while cols < need: + cols *= 2 + return cols + + +@dsl_user_op +def read_globaltimer(*, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), [], "mov.u64 $0, %globaltimer;", "=l", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _mapa_u32(smem_ptr, peer, *, loc=None, ip=None): + """The shared::cluster address of this CTA's shared-memory location in cluster CTA ``peer``.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [smem_ptr.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(peer).ir_value(loc=loc, ip=ip)], + "mapa.shared::cluster.u32 $0, $1, $2;", "=r,r,r", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_async_v4(dst, w0, w1, w2, w3, mbar, *, loc=None, ip=None): + """st.async of four int32 words to a shared::cluster address, completing ``mbar`` there by 16 bytes.""" + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, + [cutlass.Int32(dst).ir_value(loc=loc, ip=ip), cutlass.Int32(w0).ir_value(loc=loc, ip=ip), + cutlass.Int32(w1).ir_value(loc=loc, ip=ip), cutlass.Int32(w2).ir_value(loc=loc, ip=ip), + cutlass.Int32(w3).ir_value(loc=loc, ip=ip), cutlass.Int32(mbar).ir_value(loc=loc, ip=ip)], + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [$0], {$1, $2, $3, $4}, [$5];", "r,r,r,r,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _ld_shared_v4_f32(smem_ptr, *, loc=None, ip=None): + """ld.shared.v4.f32 of four fp32 at a 16-byte-aligned shared-memory pointer.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + out = _llvm.inline_asm( + _llvm.StructType.get_literal([_T.f32()] * 4), [smem_ptr.toint(loc=loc, ip=ip).ir_value()], + "ld.shared.v4.f32 {$0, $1, $2, $3}, [$4];", "=f,=f,=f,=f,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + return [ + cutlass.Float32(_llvm.extractvalue(_T.f32(), out, [i], loc=loc, ip=ip)) for i in range(4) + ] + + +@dsl_user_op +def _test_wait_cluster(mbar, parity, *, loc=None, ip=None): + """mbarrier.test_wait.parity.acquire.cluster on a barrier of this CTA completed by other CTAs' st.async.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + done = cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [mbar.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(parity).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tmbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [$1], $2;\n\t" + "selp.u32 $0, 1, 0, p;\n\t}", "=r,r,r,~{memory}", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + return done != cutlass.Int32(0) + + +@dsl_user_op +def _absmax(acc, v, *, loc=None, ip=None): + """max(acc, |v|).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(acc).ir_value(loc=loc, ip=ip), cutlass.Float32(v).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .f32 a;\n\tabs.f32 a, $2;\n\tmax.f32 $0, $1, a;\n\t}", "=f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _rcp_approx_ftz(x, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _e4m3x4(v0, v1, v2, v3, *, loc=None, ip=None): + """Four f32 to E4M3 (RN, satfinite), packed little-endian into one int32 word (v0 in the low byte).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (v0, v1, v2, v3)], + "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32(addr, value, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip)], + "st.global.b32 [$0], $1;", "l,r", has_side_effects=True, is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16(x) (RN) as the fp32 bit pattern of the rounded value.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b16 h, z;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _bf16_round(x): + return _bf16_bits(x).bitcast(cutlass.Float32) + + +@dsl_user_op +def _rope_bits(c, a, s, b, *, loc=None, ip=None): + """bf16(c * a + s * b) with separately rounded products (no FMA), as fp32 bits: the stock RoPE arithmetic.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (c, a, s, b)], + "{\n\t.reg .f32 p, q, r;\n\t.reg .b16 h, z;\n\tmul.rn.f32 p, $1, $2;\n\tmul.rn.f32 q, $3, $4;\n\t" + "add.rn.f32 r, p, q;\n\tcvt.rn.bf16.f32 h, r;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _din_load4(table, hidden, hidden_half, word_idx): + """Four int32 words at word_idx of the hidden rows (hidden_half == 1) or the embedding table: one CTA-uniform + base-address select instead of a branch.""" + addr = cutlass.select_( + hidden_half == cutlass.Int32(1), hidden.data_ptr(0).toint(), table.data_ptr(0).toint() + ) + ptr = cutlass.inttoptr( + addr + cutlass.Int64(word_idx) * cutlass.Int64(4), mem_space=1, dtype=cutlass.Int32 + ) + return ptr.load(count=4, alignment=16) + + +def _stamp(timers, timed: bool, bx, slot: int): + # Diagnostic: %globaltimer of milestone `slot` of CTA bx (timers [grid, 8] int64). + if timed: + timers.store(read_globaltimer(), idx=bx * cutlass.Int32(8) + cutlass.Int32(slot)) + + +def _k_tile(rank, i: int, split: int, my_tiles: int, contig: bool): + # contig: rank r streams k-tiles [r * my_tiles, (r + 1) * my_tiles); else the interleaved r, r + split, ... + if contig: + return rank * cutlass.Int32(my_tiles) + cutlass.Int32(i) + return rank + cutlass.Int32(i * split) + + +@cute.kernel +def m3_mx_gemv_kernel( + tma_desc_w: cutlass.GridConstant[cuda.TensorMap], # W [N, K] E4M3 as bytes, box 128 x 128 + tma_desc_wsf: cutlass.GridConstant[ + cuda.TensorMap + ], # W scale atoms [N / 128 * K / 128, 256] u16 + tma_desc_x: cutlass.GridConstant[cuda.TensorMap], # x [M, K] E4M3 as bytes, box 128 x 8 + tma_desc_xsf: cutlass.GridConstant[cuda.TensorMap], # x scale atoms [K / 128, 256] u16 + y: cutlass.Array, # [M * N] bf16, token-major + x_words: cutlass.Array, # x_bf16: the bf16 activation [M, K] as int32 words + timers: cutlass.Array, # [grid, 8] int64 milestones when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, # ar_push: the all-reduce workspace's multicast mapping (int32 words) + ar_flags: cutlass.Int64, # ar_push: its call flags (parity of the next call = flags[0] & 1) + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[ + int + ], # 1: push the output rows into the next all-reduce; 2: and trigger after it + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], # token columns of the MMA (8, 16 or 48): M <= mma_n + din_i: cutlass.Array, # din: int32 token ids [M] + din_table: cutlass.Array, # din: embedding [V, K / 2] bf16 as int32 words + din_hidden: cutlass.Array, # din: hidden rows [M, K / 2] bf16 as int32 words + din_w: cutlass.Array, # din: [embedding norm | hidden norm] weights [K] bf16 as int32 words + din_f: cutlass.Array, # din: fp32 [eps_e, eps_h] + din_vocab: cutlass.Int32, + din_g: cutlass.Array, # din, flag 1: int64 hidden row of each token, as int32 words + din_res: cutlass.Array, # din, flag 2: the gathered hidden rows out [M, K / 2] bf16 as int32 words + din_flags: cutlass.Int32, # din: 1 = gather the hidden rows, 2 = write them to din_res + rope_cs: cutlass.Array, # rope: fp32 (cos, sin) [positions][64][2] + rope_i: cutlass.Array, # rope: int32 cache lengths [M / T] + rope_bt: cutlass.Array, # rope: int32 K / V page tables [M / T][2][max_blocks] + rope_q: cutlass.Array, # rope: E4M3 Q out [M][hq][128] as uint8 + rope_pool: cutlass.Array, # rope: paged E4M3 cache [pages][hkv][tokens per page][128] as uint8 + rope_scale: cutlass.Array, # rope: fp32 [1] E4M3 quantization scale + rope_dims: cutlass.Array, # rope: int32 [max_blocks, tokens-per-page log2, hq, hkv, T] + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], +): + k_tiles = num_k_tiles(k_in) + x_tile_bytes = mma_n * CTA_K + my_tiles = k_tiles // split + rows_per_rank = CTA_M // split + tx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp_id = cute.arch.warp_idx() + rank = cutlass.Int32(0) + if cutlass.const_expr(split > 1): + rank = cute.arch.block_idx_in_cluster() + tma_ptr_w = tma_desc_w.get_ptr() + tma_ptr_wsf = tma_desc_wsf.get_ptr() + tma_ptr_x = tma_desc_x.get_ptr() + tma_ptr_xsf = tma_desc_xsf.get_ptr() + n_tile = bx // cutlass.Int32(split) + m_offset = n_tile * cutlass.Int32(CTA_M) + + smem_a = cutlass.Array( + cutlass.Uint8, ring * W_TILE_BYTES, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_b = cutlass.Array( + cutlass.Uint8, my_tiles * x_tile_bytes, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_sfa = cutlass.Array( + cutlass.Uint8, ring * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + smem_sfb = cutlass.Array( + cutlass.Uint8, my_tiles * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + w_full = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + mma_done = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + act_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + acc_done = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + reduce_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + # Partials of the rows this rank owns: [source rank][row][token] fp32 (its own slot stays unused). Rows are padded + # to mma_n + 4 words: the owner's 16-byte row reads (lane = row) then hit all 32 banks once per 8 lanes; at a + # stride of mma_n they were 8- to 16-way conflicted (4.3 us of the 48-token QKV tail). + mb_stride = mma_n + 4 + mailbox = cutlass.Array( + cutlass.Float32, + split * rows_per_rank * mb_stride, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + # din: per-token RMS factor of the half of x this CTA's k-tiles lie in. + din_rs = None + if cutlass.const_expr(din): + din_rs = cutlass.Array(cutlass.Float32, 8, space=cutlass.AddressSpace.smem, alignment=16) + + if warp_id == 0: + prims.prefetch_tensormap(tma_ptr_w) + prims.prefetch_tensormap(tma_ptr_wsf) + prims.prefetch_tensormap(tma_ptr_x) + prims.prefetch_tensormap(tma_ptr_xsf) + if prims.elect_sync(): + for s in cutlass.range_constexpr(ring): + prims.mbarrier_init(w_full.subview(s), 1) + prims.mbarrier_init(mma_done.subview(s), 1) + prims.mbarrier_init(act_full, 1) + prims.mbarrier_init(acc_done, 1) + if cutlass.const_expr(split > 1 and st_async): + # Completed by the peers' st.async bytes; this CTA's own arrival carries the expected count. + prims.mbarrier_init(reduce_ready, 1) + prims.mbarrier_arrive_expect_tx( + reduce_ready, (split - 1) * rows_per_rank * mma_n * 4 + ) + elif cutlass.const_expr(split > 1): + prims.mbarrier_init(reduce_ready, (split - 1) * rows_per_rank) + if warp_id == 2: + prims.tcgen05_alloc(tmem_ptr_i32, tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n)) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + if cutlass.const_expr(split > 1): + prims.barrier_cluster_arrive_relaxed() + prims.barrier_cluster_wait() + prims.barrier_cta_sync(0) + if tx == 0: + _stamp(timers, timed, bx, 0) + + if warp_id == 0: + if prims.elect_sync(): + for i in cutlass.range_constexpr(ring, my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [k_global * cutlass.Int32(CTA_K), m_offset], [] + ) + if cutlass.const_expr((n_out // CTA_M) * split >= 80): + # Wide grids launch in part only after the predecessor (e.g. the fused attention) frees its SMs, which + # loses their pre-wait weight prefetch: the CTAs of the first half also stage their partner's + # (bx + grid / 2) weights into L2. + if bx < cutlass.Int32((n_out // CTA_M) * split // 2): + pb = bx + cutlass.Int32((n_out // CTA_M) * split // 2) + p_rank = pb % cutlass.Int32(split) + p_m_offset = (pb // cutlass.Int32(split)) * cutlass.Int32(CTA_M) + for i in cutlass.range_constexpr(my_tiles): + p_k = _k_tile(p_rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [p_k * cutlass.Int32(CTA_K), p_m_offset], [] + ) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + if cutlass.const_expr(i >= ring): + while not cute.arch.mbarrier_try_wait( + mma_done.subview(stage).data_ptr(), (i // ring - 1) % 2 + ): + pass + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.mbarrier_arrive_expect_tx(w_full.subview(stage), W_TILE_BYTES + SF_ATOM_BYTES) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_a.subview(stage * W_TILE_BYTES), + tma_ptr_w, + (k_global * cutlass.Int32(CTA_K), m_offset), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfa.subview(stage * SF_ATOM_BYTES), + tma_ptr_wsf, + (cutlass.Int32(0), n_tile * cutlass.Int32(k_tiles) + k_global), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + elif warp_id == 1: + prims.griddepcontrol(prims.GridDepAction.WAIT) + if cutlass.const_expr(trigger_early): + # Dependents launch once this grid's own dependency is satisfied (not at entry): consumers that read + # step metadata or caches before their own grid wait rely on every grid ahead of them having passed its + # wait, so the chain of releases never runs ahead of a producer still in flight. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if tx == 32: + _stamp(timers, timed, bx, 1) + if cutlass.const_expr(not x_bf16): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(act_full, my_tiles * (x_tile_bytes + SF_ATOM_BYTES)) + for i in cutlass.range_constexpr(my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_b.subview(i * x_tile_bytes), + tma_ptr_x, + (k_global * cutlass.Int32(CTA_K), cutlass.Int32(0)), + act_full, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfb.subview(i * SF_ATOM_BYTES), + tma_ptr_xsf, + (cutlass.Int32(0), k_global), + act_full, + ) + elif warp_id == 2: + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + ring * SF_COLS + idesc = prims.Tcgen05MxInstrDesc.build( + a_dtype=a_dtype, b_dtype=b_dtype, scale_format=1, n_dim=mma_n, m_dim=CTA_M + ) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + if cutlass.const_expr(a_tmem): + # A staged in TMEM: before the grid wait every weight k-tile and its scale atom go on into TMEM as they + # land (the copies of a stage refilled later are committed to its mma_done); once the activation lands, + # its scale atoms and the MMAs, A read from TMEM. + base_row = tmem_raw >> 16 + base_col = tmem_raw & 0xFFFF + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + my_tiles * SF_COLS + a_base = tmem_raw + cutlass.Int32(a_col0(nacc, my_tiles, mma_n)) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + if prims.elect_sync(): + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_cp( + prims.Tcgen05CpShape.SHAPE_128X256B, + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), + 6, + cutlass.Int32, + ), + desc_a + ((MMA_INST_K >> 4) * kb), + group=prims.CTAGroup.CTA_1, + ) + prims.tcgen05_cp( + s2t_shape, + cutlass.inttoptr((base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32), + desc_sfa, multicast=s2t_multicast, + ) # fmt: skip + if cutlass.const_expr(i + ring < my_tiles): + prims.tcgen05_commit(mma_done.subview(stage)) + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + for i in cutlass.range_constexpr(my_tiles): + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + sfa_tmem = cutlass.inttoptr( + (base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32 + ) + sfb_tmem = cutlass.inttoptr((base_row << 16) | sfb_col0, 6, cutlass.Int32) + prims.tcgen05_cp(s2t_shape, sfb_tmem, desc_sfb, multicast=s2t_multicast) + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), 6, cutlass.Int32 + ), + desc_b + ((MMA_INST_K >> 4) * kb), idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + i > 0 or kb > 0, + sfa_tmem, sfb_tmem, + ) # fmt: skip + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + else: + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + sfa_tmem = (base_row << 16) | (sfa_col0 + stage * SF_COLS) + sfb_tmem = (base_row << 16) | (sfb_col0 + stage * SF_COLS) + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + if prims.elect_sync(): + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), desc_sfa, + multicast=s2t_multicast) # fmt: skip + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), desc_sfb, + multicast=s2t_multicast) # fmt: skip + for kb in cutlass.range_constexpr(K_BLOCKS): + inc = ((MMA_INST_K * 8 // 8) >> 4) * kb + j = i * K_BLOCKS + kb + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw + cutlass.Int32((j % nacc) * mma_n), 6, cutlass.Float32), + desc_a + inc, desc_b + inc, idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), j >= nacc, + cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), + ) # fmt: skip + prims.tcgen05_commit(mma_done.subview(stage)) + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + elif warp_id >= 4: + lane = tx % 32 + row = (warp_id % 4) * 32 + lane + if cutlass.const_expr(x_bf16): + # The bf16 activation of this CTA's k-tiles to MXFP8 in shared memory, as + # torch.ops.trtllm.mxfp8_quantize(x, True) does (UE8M0 = round-up(qz_amax * rcp(448)), + # E4M3 RN-satfinite): one 32-element block (token qz_t, k-tile qz_i, block qz_kb) per thread and round, + # the E4M3 bytes into the 128-byte-swizzled B tile, the qz_scale byte into the tile's qz_scale atom. + qz_tid = tx - cutlass.Int32(128) + prims.griddepcontrol(prims.GridDepAction.WAIT) + din_half_k = k_in // 2 + din_hidden_half = cutlass.Int32(0) + if cutlass.const_expr(din): + # Draft input x = [rmsnorm(table[ids]) * w_e, rmsnorm(hidden) * w_h] (an out-of-range id embeds to + # zero): this CTA's k-tiles lie in one half, so it needs that half's statistic of every token. 16 + # threads per token, 16-byte loads, all of a thread's loads in flight at once. + din_hidden_half = cutlass.select_( + rank * cutlass.Int32(my_tiles * CTA_K) >= cutlass.Int32(din_half_k), + cutlass.Int32(1), + cutlass.Int32(0), + ) + din_tok = qz_tid // cutlass.Int32(16) + din_part = qz_tid % cutlass.Int32(16) + din_tok_ld = cutlass.select_(din_tok < num_tokens, din_tok, cutlass.Int32(0)) + din_id = din_i.load(idx=din_tok_ld) + din_valid = (din_id >= cutlass.Int32(0)) & (din_id < din_vocab) + din_hrow = din_tok_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + din_hrow = din_g.load(idx=din_tok_ld * cutlass.Int32(2)) + din_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + din_hrow, + cutlass.select_(din_valid, din_id, cutlass.Int32(0)), + ) + # The first hidden-half CTA of tile 0 reads whole hidden rows here: it also writes them out. + din_copy = ( + ((din_flags & cutlass.Int32(2)) != cutlass.Int32(0)) + & (bx == cutlass.Int32(split // 2)) + & (din_hidden_half == cutlass.Int32(1)) + & (din_tok < num_tokens) + ) + din_base = din_row * cutlass.Int32(din_half_k // 2) + din_ss = cutlass.Float32(0.0) + din_vecs_per_part = din_half_k // 8 // 16 + for din_round in cutlass.range_constexpr(0, din_vecs_per_part, 16): + din_n = min(16, din_vecs_per_part - din_round) + din_loaded = [] + for din_j in cutlass.range_constexpr(din_n): + din_widx = din_base + ( + din_part + cutlass.Int32((din_round + din_j) * 16) + ) * cutlass.Int32(4) + din_loaded.append( + _din_load4(din_table, din_hidden, din_hidden_half, din_widx) + ) + if din_copy: + for din_j in cutlass.range_constexpr(din_n): + din_v = din_loaded[din_j] + din_res.store( + (din_v[0], din_v[1], din_v[2], din_v[3]), + idx=din_tok * cutlass.Int32(din_half_k // 2) + + (din_part + cutlass.Int32((din_round + din_j) * 16)) + * cutlass.Int32(4), + alignment=16, + ) + for din_j in cutlass.range_constexpr(din_n): + for din_e in cutlass.range_constexpr(4): + din_word = cutlass.Int32(din_loaded[din_j][din_e]) + din_lo = (din_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + din_hi = (din_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + din_ss = din_ss + din_lo * din_lo + din_hi * din_hi + for din_o in cutlass.range_constexpr(4): + din_ss = din_ss + cute.arch.shuffle_sync_bfly(din_ss, offset=1 << din_o) + if din_part == cutlass.Int32(0): + din_eps = din_f.load(idx=din_hidden_half) + din_r = cute.math.rsqrt( + din_ss / cutlass.Float32(din_half_k) + din_eps, fastmath=True + ) + din_keep = din_valid | (din_hidden_half == cutlass.Int32(1)) + din_rs.store( + cutlass.select_(din_keep, din_r, cutlass.Float32(0.0)), idx=din_tok + ) + prims.barrier_cta_sync(3, thread_count=128) + qz_b_words = cutlass.Array(smem_b.data_ptr(0), shape=(my_tiles * x_tile_bytes // 4,), dtype=cutlass.Int32, + alignment=16) # fmt: skip + qz_items = my_tiles * K_BLOCKS * mma_n + qz_rounds = (qz_items + 127) // 128 + # A group's loads first (clamped to a valid row), then its math: one L2 round trip per group of + # QZ_GROUP rounds instead of one per round, without holding every round's data in registers. + QZ_GROUP = qz_rounds if qz_rounds <= 8 else 4 + qz_loaded = [] + for qz_r in cutlass.range_constexpr(qz_rounds): + qz_item = qz_tid + cutlass.Int32(qz_r * 128) + qz_t = qz_item % cutlass.Int32(mma_n) + qz_kb = (qz_item // cutlass.Int32(mma_n)) % cutlass.Int32(K_BLOCKS) + qz_i = qz_item // cutlass.Int32(mma_n * K_BLOCKS) + if cutlass.const_expr(qz_r * 128 + 127 >= qz_items): + qz_i = cutlass.select_( + qz_item < cutlass.Int32(qz_items), qz_i, cutlass.Int32(0) + ) + qz_k_global = rank + qz_i * cutlass.Int32(split) + if cutlass.const_expr(contig): + qz_k_global = rank * cutlass.Int32(my_tiles) + qz_i + qz_t_ld = cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + qz_x_base = ( + qz_t_ld * cutlass.Int32(k_in // 2) + + qz_k_global * cutlass.Int32(CTA_K // 2) + + qz_kb * cutlass.Int32(MMA_INST_K // 2) + ) + qz_wvecs = None + if cutlass.const_expr(din): + # The draft input's raw values (table row of the id / hidden row) and its norm weights. + qz_col = qz_k_global * cutlass.Int32(CTA_K) + qz_kb * cutlass.Int32(MMA_INST_K) + qz_hcol = qz_col - din_hidden_half * cutlass.Int32(din_half_k) + qz_id = din_i.load(idx=qz_t_ld) + qz_hrow = qz_t_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + qz_hrow = din_g.load(idx=qz_t_ld * cutlass.Int32(2)) + qz_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + qz_hrow, + cutlass.select_( + (qz_id >= cutlass.Int32(0)) & (qz_id < din_vocab), + qz_id, + cutlass.Int32(0), + ), + ) + qz_src_w = qz_row * cutlass.Int32(din_half_k // 2) + qz_hcol // cutlass.Int32(2) + qz_vecs = [ + _din_load4( + din_table, + din_hidden, + din_hidden_half, + qz_src_w + cutlass.Int32(qz_v * 4), + ) + for qz_v in range(4) + ] + qz_wvecs = [ + din_w.load( + idx=qz_col // cutlass.Int32(2) + cutlass.Int32(qz_v * 4), + vector_size=4, + alignment=16, + ) + for qz_v in range(4) + ] + else: + qz_vecs = [ + x_words.load( + idx=qz_x_base + cutlass.Int32(qz_v * 4), vector_size=4, alignment=16 + ) + for qz_v in range(4) + ] + qz_loaded.append((qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs)) + if cutlass.const_expr((qz_r + 1) % QZ_GROUP == 0 or qz_r + 1 == qz_rounds): + for qz_g in cutlass.range_constexpr(len(qz_loaded)): + qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs = qz_loaded[qz_g] + qz_vals = [] + qz_amax = cutlass.Float32(0.0) + if cutlass.const_expr(din): + qz_rs = din_rs.load( + idx=cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + ) + for qz_v in cutlass.range_constexpr(4): + for qz_e4 in cutlass.range_constexpr(4): + qz_word = cutlass.Int32(qz_vecs[qz_v][qz_e4]) + qz_lo = (qz_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + qz_hi = (qz_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + if cutlass.const_expr(din): + # bf16(x * rs * w), the draft-input kernel's rounding. + qz_wword = cutlass.Int32(qz_wvecs[qz_v][qz_e4]) + qz_lo = _bf16_round( + qz_lo + * qz_rs + * (qz_wword << cutlass.Int32(16)).bitcast(cutlass.Float32) + ) + qz_hi = _bf16_round( + qz_hi + * qz_rs + * (qz_wword & cutlass.Int32(-65536)).bitcast( + cutlass.Float32 + ) + ) + qz_vals.append(qz_lo) + qz_vals.append(qz_hi) + qz_amax = _absmax(_absmax(qz_amax, qz_lo), qz_hi) + qz_bits = (qz_amax * _rcp_approx_ftz(cutlass.Float32(448.0))).bitcast( + cutlass.Int32 + ) + qz_e = ( + (qz_bits >> cutlass.Int32(23)) & cutlass.Int32(0xFF) + ) + cutlass.select_( + (qz_bits & cutlass.Int32(0x7FFFFF)) != cutlass.Int32(0), + cutlass.Int32(1), + cutlass.Int32(0), + ) + qz_e = cutlass.select_(qz_e > cutlass.Int32(254), cutlass.Int32(254), qz_e) + qz_scale = ((cutlass.Int32(254) - qz_e) << cutlass.Int32(23)).bitcast( + cutlass.Float32 + ) + qz_scale = cutlass.select_( + qz_amax != cutlass.Float32(0.0), qz_scale, cutlass.Float32(0.0) + ) + qz_q = [_e4m3x4(qz_vals[4 * qz_j] * qz_scale, qz_vals[4 * qz_j + 1] * qz_scale, + qz_vals[4 * qz_j + 2] * qz_scale, qz_vals[4 * qz_j + 3] * qz_scale) + for qz_j in range(8)] # fmt: skip + if qz_item < cutlass.Int32(qz_items): + if qz_t < num_tokens: + qz_row_w = qz_i * cutlass.Int32( + x_tile_bytes // 4 + ) + qz_t * cutlass.Int32(CTA_K // 4) + qz_c0 = qz_kb * cutlass.Int32(2) + qz_b_words.store( + (qz_q[0], qz_q[1], qz_q[2], qz_q[3]), + idx=qz_row_w + + (qz_c0 ^ (qz_t & cutlass.Int32(7))) * cutlass.Int32(4), + alignment=16, + ) + qz_b_words.store( + (qz_q[4], qz_q[5], qz_q[6], qz_q[7]), + idx=qz_row_w + + ((qz_c0 + cutlass.Int32(1)) ^ (qz_t & cutlass.Int32(7))) + * cutlass.Int32(4), + alignment=16, + ) + smem_sfb.store( + cutlass.Uint8(qz_e), + idx=qz_i * cutlass.Int32(SF_ATOM_BYTES) + + (qz_t % cutlass.Int32(32)) * cutlass.Int32(16) + + (qz_t // cutlass.Int32(32)) * cutlass.Int32(4) + + qz_kb, + ) + qz_loaded = [] + # Generic-proxy stores made visible to the MMA's async-proxy reads, then one arrival for all. + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.barrier_cta_sync(2, thread_count=128) + if qz_tid == cutlass.Int32(0): + prims.mbarrier_arrive(act_full) + if cutlass.const_expr(st_async): + while not cute.arch.mbarrier_test_wait(acc_done.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(acc_done.data_ptr(), 0): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + # tcgen05.ld widths are powers of two: the accumulator columns in descending power-of-two pieces. + acc_all = [] + ld_col = 0 + for ld_w in (128, 64, 32, 16, 8): + if cutlass.const_expr((mma_n * nacc - ld_col) >= ld_w): + piece = prims.tcgen05_ld( + "32x32b", + cutlass.inttoptr( + tmem_ptr_i32.load() + cutlass.Int32(ld_col), 6, cutlass.Float32 + ), + num=ld_w, + ) + acc_all = acc_all + [piece[c] for c in range(ld_w)] + ld_col += ld_w + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + acc = [] + for t in cutlass.range_constexpr(mma_n): + v = cutlass.Float32(acc_all[t]) + for a in cutlass.range_constexpr(1, nacc): + v = v + cutlass.Float32(acc_all[a * mma_n + t]) + acc.append(v) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if tx == 128: + _stamp(timers, timed, bx, 4) + n = m_offset + row + if cutlass.const_expr(split == 1): + for t in cutlass.range_constexpr(mma_n): + if cutlass.Int32(t) < num_tokens: + y.store( + cutlass.BFloat16(cutlass.Float32(acc[t])), idx=cutlass.Int32(t * n_out) + n + ) + if cutlass.const_expr(ar_push > 0): + prims.griddepcontrol(prims.GridDepAction.WAIT) + # The parity of each (token, column slice) slot: the all-reduce's CTA t * 8 + slice flag. All loads are + # issued before the first store so they overlap (one L2 round trip, not one per token). + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_v = cutlass.Float32(acc[t]) + ar_hi = cute.arch.shuffle_sync_bfly(ar_v, offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), _bf16x2_word(ar_v, ar_hi) + ) + else: + owner = row // cutlass.Int32(rows_per_rank) + local = row % cutlass.Int32(rows_per_rank) + if cutlass.const_expr(rope): + # Rank r owns head dims [r * P, (r + 1) * P) and their rotation partners 64 + ..., P = rows / 2. + pairs = rows_per_rank // 2 + owner = (row % cutlass.Int32(64)) // cutlass.Int32(pairs) + local = (row // cutlass.Int32(64)) * cutlass.Int32(pairs) + ( + row % cutlass.Int32(64) + ) % cutlass.Int32(pairs) + if owner != rank: + base = (rank * cutlass.Int32(rows_per_rank) + local) * cutlass.Int32(mb_stride) + if cutlass.const_expr(st_async): + dst = _mapa_u32(mailbox.subview(base).data_ptr(), owner) + mb = _mapa_u32(reduce_ready.data_ptr(), owner) + words = [cutlass.Float32(acc[t]).bitcast(cutlass.Int32) for t in range(mma_n)] + for g in cutlass.range_constexpr(mma_n // 4): + _st_async_v4( + dst + cutlass.Int32(16 * g), + words[4 * g], + words[4 * g + 1], + words[4 * g + 2], + words[4 * g + 3], + mb, + ) + else: + for t in cutlass.range_constexpr(mma_n): + prims.mapa(mailbox.subview(base + cutlass.Int32(t)), owner).store( + cutlass.Float32(acc[t]) + ) + prims.mbarrier_arrive( + prims.mapa(reduce_ready, owner), scope=prims.MemScope.CLUSTER + ) + else: + if cutlass.const_expr(st_async): + while not _test_wait_cluster(reduce_ready.data_ptr(), 0): + pass + else: + while not prims.mbarrier_try_wait_parity( + reduce_ready, 0, scope=prims.MBarrierScope.CLUSTER + ): + pass + if lane == 0: + if row // cutlass.Int32(rows_per_rank) == rank: + if (row % cutlass.Int32(rows_per_rank)) == 0: + _stamp(timers, timed, bx, 5) + totals = [] + for g in cutlass.range_constexpr(mma_n // 4): + sums = [cutlass.Float32(0.0)] * 4 + for q in cutlass.range_constexpr(split): + parts = _ld_shared_v4_f32( + mailbox.subview( + (cutlass.Int32(q * rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g) + ).data_ptr() + ) + sums = [sums[j] + cutlass.select_(rank == cutlass.Int32(q), cutlass.Float32(acc[4 * g + j]), + parts[j]) for j in range(4)] # fmt: skip + for j in cutlass.range_constexpr(4): + totals.append(sums[j]) + if cutlass.const_expr(rope): + # The own-rank mailbox slot is unused: it carries the totals to the RoPE stage. + mailbox.store( + sums[j], + idx=(rank * cutlass.Int32(rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g + j), + ) + elif cutlass.Int32(4 * g + j) < num_tokens: + y.store( + cutlass.BFloat16(sums[j]), + idx=cutlass.Int32((4 * g + j) * n_out) + n, + ) + if cutlass.const_expr(ar_push > 0): + # Every lane of an owner warp is an owner (rows_per_rank >= 32), so lane pairs hold row pairs. + prims.griddepcontrol(prims.GridDepAction.WAIT) + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_hi = cute.arch.shuffle_sync_bfly(totals[t], offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), + _bf16x2_word(totals[t], ar_hi), + ) + prims.barrier_cta_sync(1, thread_count=128) + if cutlass.const_expr(rope): + # Token t of request t / T sits at cache position kv_len - T + t % T. Q / K heads: GPT-NeoX RoPE over + # the 128-wide head, bf16(cos * x -/+ sin * pair) on the bf16 projection; then E4M3 of x * scale, Q to + # rope_q, K / V into the page of that position. + pairs = rows_per_rank // 2 + rp_e = tx - cutlass.Int32(128) + rp_p = rp_e % cutlass.Int32(pairs) + rp_t = rp_e // cutlass.Int32(pairs) + if rp_t < num_tokens: + rp_max_blocks = rope_dims.load(idx=0) + rp_tpb_log2 = rope_dims.load(idx=1) + rp_hq = rope_dims.load(idx=2) + rp_hkv = rope_dims.load(idx=3) + rp_T = rope_dims.load(idx=4) + rp_d = rank * cutlass.Int32(pairs) + rp_p + rp_a = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + rp_p) * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_b = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + cutlass.Int32(pairs) + rp_p) + * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_req = rp_t // rp_T + rp_tok = rope_i.load(idx=rp_req) - rp_T + (rp_t - rp_req * rp_T) + rp_head = n_tile + if rp_head < rp_hq + rp_hkv: + rp_c = rope_cs.load(idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2)) + rp_s = rope_cs.load( + idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2) + + cutlass.Int32(1) + ) + rp_ra = _rope_bits(rp_c, rp_a, -rp_s, rp_b).bitcast(cutlass.Float32) + rp_rb = _rope_bits(rp_c, rp_b, rp_s, rp_a).bitcast(cutlass.Float32) + rp_a = rp_ra + rp_b = rp_rb + rp_qs = rope_scale.load(idx=0) + rp_w = _e4m3x4( + rp_a * rp_qs, rp_b * rp_qs, cutlass.Float32(0.0), cutlass.Float32(0.0) + ) + rp_lo = cutlass.Uint8(rp_w & cutlass.Int32(0xFF)) + rp_hi = cutlass.Uint8((rp_w >> cutlass.Int32(8)) & cutlass.Int32(0xFF)) + if rp_head < rp_hq: + rp_q = (rp_t * rp_hq + rp_head) * cutlass.Int32(128) + rp_d + rope_q.store(rp_lo, idx=rp_q) + rope_q.store(rp_hi, idx=rp_q + cutlass.Int32(64)) + else: + rp_kv = cutlass.select_( + rp_head < rp_hq + rp_hkv, cutlass.Int32(0), cutlass.Int32(1) + ) + rp_kh = rp_head - rp_hq - rp_kv * rp_hkv + rp_page = rope_bt.load( + idx=(rp_req * cutlass.Int32(2) + rp_kv) * rp_max_blocks + + (rp_tok >> rp_tpb_log2) + ) + # 64-bit byte offset: the pool view can exceed 2^31 bytes. + rp_dst = ( + (cutlass.Int64(rp_page) * cutlass.Int64(rp_hkv) + cutlass.Int64(rp_kh)) + << cutlass.Int64(rp_tpb_log2 + cutlass.Int32(7)) + ) + cutlass.Int64( + (rp_tok & ((cutlass.Int32(1) << rp_tpb_log2) - cutlass.Int32(1))) + * cutlass.Int32(128) + + rp_d + ) + rp_addr = rope_pool.data_ptr(0).toint() + rp_dst + cutlass.inttoptr(rp_addr, mem_space=1, dtype=cutlass.Uint8).store(rp_lo) + cutlass.inttoptr( + rp_addr + cutlass.Int64(64), mem_space=1, dtype=cutlass.Uint8 + ).store(rp_hi) + if tx == 128: + _stamp(timers, timed, bx, 6) + if cutlass.const_expr(ar_push == 2): + # The all-reduce polls the pushed words; it launches after this CTA's rows are out, not at grid end. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if warp_id == 4: + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Int32), + tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n), + ) + + +def _bytes_map(t, cols, rows, box_rows, swizzle): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[cols, rows], + global_strides=[cols // 16], + box_dims=[CTA_K, box_rows], + swizzle=swizzle, + ) + + +def _atom_map(t, atoms): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint16, + global_dims=[SF_ATOM_U16, atoms], + global_strides=[SF_ATOM_BYTES // 16], + box_dims=[SF_ATOM_U16, 1], + swizzle=cuda.TensorMapSwizzle.none, + ) + + +@cute.jit +def m3_mx_gemv( + w: cute.Tensor, # [N, K] E4M3 as uint8, K contiguous + wsf: cute.Tensor, # weight scales, 128x4-swizzled, as uint8 + x: cute.Tensor, # [M, K] E4M3 as uint8, M <= 8 + xsf: cute.Tensor, # activation scales, 128x4-swizzled (atom k at byte 512 k), as uint8 + y: cute.Tensor, # [M * N] bf16 + x_words: cute.Tensor, # x_bf16: bf16 x [M, K] as int32 words (x / xsf are then unused placeholders) + timers: cute.Tensor, # [grid * 8] int64, written when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[int], + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, + din_i: cute.Tensor, # din / rope inputs: see m3_mx_gemv_kernel (placeholders when off) + din_table: cute.Tensor, + din_hidden: cute.Tensor, + din_w: cute.Tensor, + din_f: cute.Tensor, + din_vocab: cutlass.Int32, + din_g: cute.Tensor, + din_res: cute.Tensor, + din_flags: cutlass.Int32, + rope_cs: cute.Tensor, + rope_i: cute.Tensor, + rope_bt: cute.Tensor, + rope_q: cute.Tensor, + rope_pool: cute.Tensor, + rope_scale: cute.Tensor, + rope_dims: cute.Tensor, + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], +) -> None: + k_tiles = num_k_tiles(k_in) + tma_desc_w = _bytes_map(w, k_in, n_out, CTA_M, cuda.TensorMapSwizzle.s128b) + tma_desc_wsf = _atom_map(wsf, (n_out // CTA_M) * k_tiles) + tma_desc_x = _bytes_map(x, k_in, num_tokens, mma_n, cuda.TensorMapSwizzle.s128b) + tma_desc_xsf = _atom_map(xsf, k_tiles) + m3_mx_gemv_kernel( + tma_desc_w, tma_desc_wsf, tma_desc_x, tma_desc_xsf, y, x_words, timers, num_tokens, n_out, k_in, split, ring, + trigger_early, contig, timed, nacc, a_tmem, st_async, x_bf16, ar_ws, ar_flags, ar_rank, ar_push, ar_world, + ar_maxt, mma_n, + din_i, din_table, din_hidden, din_w, din_f, din_vocab, din_g, din_res, din_flags, rope_cs, + rope_i, rope_bt, rope_q, rope_pool, rope_scale, rope_dims, din, rope, + ).launch( + grid=((n_out // CTA_M) * split, 1, 1), + block=(THREADS, 1, 1), + cluster=(split, 1, 1), + stream=stream, + use_pdl=use_pdl, + ) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/op.py new file mode 100644 index 000000000000..d5440088cbc8 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mx_gemv/op.py @@ -0,0 +1,474 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) for already-quantized activations of at most 8 tokens. + +Compiled on the first call per (N, K, split, ring, early trigger, PDL, push, MMA width), which must happen outside +CUDA-graph capture, or ahead of any call by ``precompile``; once ``freeze`` closes compiles, a call whose kernel is not +compiled declines instead. +""" + +from __future__ import annotations + +import threading +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +MMA_WIDTHS = ( + 8, + 16, + 32, + 48, +) # token columns of the compiled MMA; a call uses the smallest that holds M +_SMEM_LIMIT = 220 * 1024 # capacity: fixed: the device's shared-memory limit + + +def _mma_n(m: int) -> int: + return next(w for w in MMA_WIDTHS if m <= w) + + +def _smem_bytes(ring: int, tiles: int, split: int, mma_n: int) -> int: + return ( + ring * (128 * 128 + 512) + + tiles * (mma_n * 128 + 512) + + split * (128 // split) * (mma_n + 4) * 4 + + 1024 + ) + + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + +# (N, K) -> (split, ring): GB300 exposed time behind a 4.25 us PDL predecessor at M 1 / 8, vs the fused low-M GEMM: +# QKV 2.78 / 3.16 vs 4.58 / 6.21 us, o_proj 2.23 / 2.41 vs 4.40 / 5.09, shared gate_up 2.43 / 2.74 vs 3.56 / 4.30, +# shared down 1.74 / 1.66 vs 2.92 / 3.38. A config is (split, ring[, contig, nacc, a_tmem, st_async]); the optional +# fields default to _CONTIG, _NACC, _A_TMEM, _ST_ASYNC. +_CONFIGS: Dict[Tuple[int, int], tuple] = { + (2560, 6144): (4, 12), + (6144, 2048): (2, 8), + # Shared-expert gate_up: split 2, ring 12, single accumulator, A operand from shared memory. + (1536, 6144): (2, 12, 1, 1, 0, 1), + # Shared-expert down: unsplit, ring 3, single accumulator, A operand from shared memory. + (6144, 768): (1, 3, 1, 1, 0, 1), + (2304, 6144): (4, 12), + (6144, 3072): (2, 12), + # Dense-layer gate_up: 6.4 / 6.3 us at M 1 / 8 vs FlashInfer 8.8. + (6144, 6144): (2, 12), +} +# Per-token-count configs (N, K, M): the grid that suits the concurrent routed-expert engine depends on how long +# that engine runs. At 2 tokens the shared-expert gate_up splits K over 4 CTAs. +_CONFIGS_BY_M: Dict[Tuple[int, int, int], tuple] = { + (1536, 6144, 2): (4, 12, 1, 1, 0, 1), +} +# Above 8 tokens (spec-decode verify rows): the shared-expert gate_up splits K over 4 CTAs. +_CONFIGS_WIDE: Dict[Tuple[int, int], tuple] = { + (1536, 6144): (4, 12, 1, 1, 0, 1), +} + +# Defaults of the optional config fields: contiguous k-tile ranges per split rank, one TMEM accumulator, the A +# operand staged in TMEM (when its columns fit), st.async split-K partial exchange. +_CONTIG = True +_NACC = 1 +_A_TMEM = True +_ST_ASYNC = True +_ensured = set() + + +_DUMMY: Dict[torch.device, torch.Tensor] = {} + + +def _dummy_timers(device) -> torch.Tensor: + """Placeholder for the kernel's timer buffer, which it only writes when compiled with timers on.""" + t = _DUMMY.get(device) + if t is None: + t = _DUMMY[device] = torch.zeros(8, dtype=torch.int64, device=device) + return t + + +def _arg(t: torch.Tensor, align: int = 16): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align).mark_layout_dynamic(leading_dim=t.dim() - 1) + + +def get_config(n: int, k: int, m: Optional[int] = None) -> Optional[tuple]: + cfg = _CONFIGS_BY_M.get((n, k, m)) if m is not None else None + if cfg is None and m is not None and m > 8: + cfg = _CONFIGS_WIDE.get((n, k)) + cfg = cfg or _CONFIGS.get((n, k)) + # The split-K epilogue gives each of the split CTAs 128 // split rows of a 128-row output tile. + if cfg is None or n % 128 != 0 or k % 128 != 0 or (k // 128) % cfg[0] != 0 or 128 % cfg[0] != 0: + return None + return cfg + + +def _declines(n: int, k: int, m: int, x_bf16: bool, cfg: tuple) -> bool: + """Whether a (split, ring, ...) config declines an [M, K] x [N, K]^T call on shape grounds.""" + split = cfg[0] + if not 0 < m <= MAX_TOKENS or 128 % split != 0 or (k // 128) % split != 0: + return True + tiles = k // 128 // split + mma_n = _mma_n(m) + # GB300 exposed time vs the FlashInfer prequantized GEMM (behind a PDL predecessor), at 24 / 48 tokens: QKV 3.76 / + # 4.63 vs 7.46 / 11.44 us, o_proj 2.74 / 3.33 vs 4.46 / 5.84, shared gate_up 4.57 / 5.71 vs 7.16 / 11.25. Behind + # only for K = 768 (unsplit shared down) at 17-32 tokens: 3.1 vs 2.9. + # The in-kernel bf16 -> MXFP8 quantization (x_bf16) costs one 32-element block per thread and round on 128 threads; + # beyond 8 rounds per CTA, mxfp8_quantize + this GEMV (or the fallback GEMM) is faster. + if (16 < m <= 32 and k < 1024) or (x_bf16 and tiles * 4 * mma_n > 8 * 128): + return True + return _smem_bytes(min(cfg[1], tiles, 2), tiles, split, mma_n) > _SMEM_LIMIT + + +def serves(n: int, k: int, m: int, x_bf16: bool) -> bool: + """Whether the GEMV with its default config takes an [M, K] activation (bf16 or MXFP8) against an [N, K] weight.""" + cfg = get_config(n, k, m) + return cfg is not None and not _declines(n, k, m, x_bf16, cfg) + + +def _resolve( + m: int, + n: int, + k: int, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple], +) -> Optional[tuple]: + """The compile key of an [M, K] x [N, K]^T call and its runtime push words, or None when the call is declined. + + ``ar`` is the push tuple of ``_run``; only its (world, bucket tokens, mode) enter the key, the addresses and the + rank are runtime arguments.""" + cfg = cfg or get_config(n, k, m) + if cfg is None or _declines(n, k, m, x_bf16, cfg): + return None + split, ring = cfg[0], cfg[1] + contig = bool(cfg[2]) if len(cfg) > 2 else _CONTIG + nacc = int(cfg[3]) if len(cfg) > 3 else _NACC + a_tmem = bool(cfg[4]) if len(cfg) > 4 else _A_TMEM + st_async = bool(cfg[5]) if len(cfg) > 5 else _ST_ASYNC + ring = min(ring, k // 128 // split) + tiles = k // 128 // split + mma_n = _mma_n(m) + while ring > 2 and _smem_bytes(ring, tiles, split, mma_n) > _SMEM_LIMIT: + ring -= 1 + if a_tmem: + from .m3_mx_gemv_kernel import tmem_cols + + if tmem_cols(ring, nacc, True, tiles, mma_n) > 512: + a_tmem = False + # ar = (multicast workspace address, flags address, rank, world, bucket tokens, mode): push the output rows into + # the next m3_mc_allreduce call's exchange slots (mode 2 also defers the dependents' launch until after the push). + ar_ws, ar_flags, ar_rank, ar_mode, ar_world, ar_maxt = 0, 0, 0, 0, 1, 8 + if ar is not None and split <= 4 and n % 2 == 0: + ar_ws, ar_flags, ar_rank, ar_world, ar_maxt, ar_mode = ar + if ar_mode == 2: + trigger_early = False + if use_pdl is None: + use_pdl = get_env_enable_pdl() + # The kernel's ``timed`` flag (per-CTA milestone timers) is always off. + key = ( + n, + k, + split, + ring, + trigger_early, + contig, + False, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + ) + return key, (ar_ws, ar_flags, ar_rank) + + +def _compile(key: tuple, args: tuple, m: int, runtime_ar: tuple, stream, extra: tuple): + """cute.compile the kernel of ``key`` (see ``_resolve``) on these argument views; cached.""" + import cutlass.cute as cute + + from . import m3_mx_gemv_kernel as kernel + + ( + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + ) = key + ar_ws, ar_flags, ar_rank = runtime_ar + with _lock: + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute.compile( + kernel.m3_mx_gemv, + *args, + m, + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_ws, + ar_flags, + ar_rank, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + stream, + *extra, + False, + False, + ) + return fn + + +def _views( + x: torch.Tensor, xsf: Optional[torch.Tensor], weight, weight_scale, y, x_bf16: bool +) -> tuple: + k = x.shape[1] + if x_bf16: + x_bytes = x.view(torch.uint8) + xsf = _placeholder_scales(k, x.device) + x_words = x.view(torch.int32) + else: + x_bytes = x.view(torch.uint8) + x_words = _placeholder_words(x.device) + return ( + _arg(weight.view(torch.uint8)), + _arg(weight_scale.view(torch.uint8).reshape(-1)), + _arg(x_bytes), + _arg(xsf.view(torch.uint8).reshape(-1)), + _arg(y.view(-1) if y.numel() else _placeholder_words(x.device).view(torch.bfloat16)), + _arg(x_words.reshape(-1)), + _arg(_dummy_timers(x.device)), + ) + + +def _run( + x: torch.Tensor, + xsf: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + global last_pushed, last_uncompiled + last_pushed = False + last_uncompiled = None + import cuda.bindings.driver as cuda_driver + + m, k = x.shape + n = weight.shape[0] + if weight.shape[1] != k: + return None + resolved = _resolve(m, n, k, x_bf16, trigger_early, cfg, use_pdl, ar) + if resolved is None: + return None + key, runtime_ar = resolved + fn = _compiled.get(key) + if fn is None: + if _frozen[0]: + # Compiles are closed (the model built every kernel it dispatches to before warmup): decline, the caller + # reports the miss. + last_uncompiled = key + return None + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_mx_gemv must run once per shape outside CUDA-graph capture first" + ) + y = torch.empty(m, n, dtype=torch.bfloat16, device=x.device) + args = _views(x, xsf, weight, weight_scale, y, x_bf16) + extra = _extra_args(x.device) + stream = cuda_driver.CUstream(torch.cuda.current_stream(x.device).cuda_stream) + if fn is None: + fn = _compile(key, args, m, runtime_ar, stream, extra) + ar_ws, ar_flags, ar_rank = runtime_ar + fn(*args, m, ar_ws, ar_flags, ar_rank, stream, *extra) + last_pushed = key[11] > 0 + return y + + +def precompile( + weight: torch.Tensor, + weight_scale: torch.Tensor, + m: int, + x_bf16: bool, + use_pdl: Optional[bool] = None, + ar: Optional[Tuple[int, int, int]] = None, +) -> Optional[tuple]: + """Compile, without launching, the kernel an M-row call against ``weight`` takes; returns its key, or None when + such a call is declined. ``ar`` = (world, bucket tokens, mode) of a pushing call (its addresses and rank are + runtime arguments, so none is needed here). Runs outside capture, even while compiles are frozen.""" + import cuda.bindings.driver as cuda_driver + + n, k = weight.shape + push = None if ar is None else (0, 0, 0, *ar) + resolved = _resolve(m, n, k, x_bf16, True, None, use_pdl, push) + if resolved is None: + return None + key, runtime_ar = resolved + if key in _compiled: + return key + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_mx_gemv.precompile allocates and compiles: call it outside capture") + dev = weight.device + x = torch.zeros(m, k, dtype=torch.bfloat16 if x_bf16 else torch.float8_e4m3fn, device=dev) + xsf = None if x_bf16 else torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=dev) + y = torch.empty(m, n, dtype=torch.bfloat16, device=dev) + args = _views(x, xsf, weight, weight_scale, y, x_bf16) + stream = cuda_driver.CUstream(torch.cuda.current_stream(dev).cuda_stream) + _compile(key, args, m, runtime_ar, stream, _extra_args(dev)) + return key + + +def freeze(frozen: bool = True) -> None: + """Close (or reopen) compiles: while frozen, a call whose kernel is not compiled yet declines (returns None and + sets ``last_uncompiled``) instead of compiling. ``precompile`` and ``ensure_compiled`` still compile.""" + _frozen[0] = bool(frozen) + + +def frozen() -> bool: + return _frozen[0] + + +def is_compiled(key: tuple) -> bool: + return key in _compiled + + +def _extra_args(device) -> tuple: + """Placeholder arguments of the kernel's draft-input prologue and RoPE / KV epilogue (compiled off).""" + i32 = _placeholder_words(device) + f32 = i32.view(torch.float32) + u8 = i32.view(torch.uint8) + d = (_arg(i32), _arg(i32), _arg(i32), _arg(i32), _arg(f32), 0, _arg(i32), _arg(i32), 0) + r = (_arg(f32), _arg(i32), _arg(i32), _arg(u8), _arg(u8), _arg(f32), _arg(i32)) + return d + r + + +def m3_mx_gemv( + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """``xq`` E4M3 [M <= 8, K] with ``xsf`` its 128x4-swizzled UE8M0 scales; ``weight`` E4M3 [N, K] with + ``weight_scale`` swizzled. Returns bf16 [M, N], or None when the shape is not served.""" + return _run(xq, xsf, weight, weight_scale, False, trigger_early, cfg, use_pdl, ar) + + +def m3_mx_gemv_bf16( + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """The same GEMV for a bf16 ``x`` [M <= 8, K] (contiguous), quantized to MXFP8 inside the kernel exactly as + torch.ops.trtllm.mxfp8_quantize(x, True) does.""" + if x.dtype != torch.bfloat16 or x.dim() != 2 or not x.is_contiguous() or x.data_ptr() % 16 != 0: + return None + return _run(x, None, weight, weight_scale, True, trigger_early, cfg, use_pdl, ar) + + +_PLACEHOLDERS: Dict[tuple, torch.Tensor] = {} +# Whether the last call pushed its output into an all-reduce (see _run's ``ar``). +last_pushed = False +# The key of the last call declined because compiles are frozen and its kernel was not compiled (see ``freeze``). +last_uncompiled: Optional[tuple] = None +_frozen = [False] + + +def _placeholder_scales(k: int, device) -> torch.Tensor: + key = ("sf", k, device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(k // 128 * 512, dtype=torch.uint8, device=device) + return t + + +def _placeholder_words(device) -> torch.Tensor: + key = ("w", device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(16, dtype=torch.int32, device=device) + return t + + +def ensure_compiled(weight: torch.Tensor, weight_scale: torch.Tensor, use_pdl: bool) -> None: + """Compiles the kernel of this weight's shape outside capture (a one-token dummy call).""" + if ( + weight.dim() != 2 + or weight.dtype != torch.float8_e4m3fn + or get_config(*weight.shape) is None + ): + return + n, k = weight.shape + if (n, k, use_pdl) in _ensured: + return + _ensured.add((n, k, use_pdl)) + xq = torch.zeros(1, k, dtype=torch.float8_e4m3fn, device=weight.device) + xsf = torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=weight.device) + m3_mx_gemv(xq, xsf, weight, weight_scale, use_pdl=use_pdl) + for width in MMA_WIDTHS[1:]: + m3_mx_gemv( + torch.zeros(width, k, dtype=torch.float8_e4m3fn, device=weight.device), + xsf, + weight, + weight_scale, + use_pdl=use_pdl, + ) + m3_mx_gemv_bf16( + torch.zeros(1, k, dtype=torch.bfloat16, device=weight.device), + weight, + weight_scale, + use_pdl=use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mxfp8_low_m_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mxfp8_low_m_gemm.py new file mode 100644 index 000000000000..cc635f93a836 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/mxfp8_low_m_gemm.py @@ -0,0 +1,109 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused MXFP8 activation quantization + low-M MXFP8 GEMM for decode (SM100+). + +``out[M, N] = mxfp8(x)[M, K] @ mxfp8(W)[N, K]^T`` for ``1 <= M <= 8`` in one kernel +(``torch.ops.trtllm.minimax_m3_mxfp8_low_m_gemm``, reached through the ``ops/gemm`` entry). + +* Each CTA owns ``16 * ROW_TILES`` weight rows and ``K / CLUSTER_K`` columns. + Before ``griddepcontrol.wait`` it issues ``cp.async.bulk`` copies of its weight + tile and UE8M0 scale chunks (existing swizzled 128x4 layout) into shared memory, + so the weight stream overlaps the producer kernel under PDL. Configurations + keep the whole grid resident in one wave so every CTA prefetches. +* Activation quantization reproduces ``trtllm::mxfp8_quantize`` with identical + intrinsics (block 32, UE8M0 rounded toward +inf, saturating RN E4M3); each + thread issues all of its loads before converting, one L2 round trip per CTA. +* One ``mma.sync.m16n8k32`` E4M3 per 32-wide scale block and row tile with exact + power-of-two rescaling and FP32 accumulation. Warps reduce through shared + memory (reusing the consumed weight tile); split-K peers push their partial + tiles into the cluster root with ``st.async`` and exit, and the root sums the + ranks in order. +""" + +from typing import Optional + +import torch + +from tensorrt_llm.logger import logger + +from ..ops.gemm.mxfp8_low_m_gemm import mxfp8_low_m_gemm as _low_m_gemm_op + +# (N, K) -> (warps, cluster_k, max_m, row_tiles); measured on GB300 TP4 M3 shapes. The fused kernel beats native +# quantize + GEMM only up to max_m rows; larger M, and every other shape, keeps the native path. These four +# configurations are the ones built into ``torch.ops.trtllm.minimax_m3_mxfp8_low_m_gemm``. +_CONFIGS = { + (2560, 6144): (4, 4, 8, 2), # QKV (q, k, v, index q / k) + (6144, 2048): (8, 2, 8, 4), # o_proj + (6144, 6144): (4, 4, 1, 1), # dense MLP gate_up + (6144, 3072): (4, 2, 8, 2), # dense MLP down +} +# Opt-in shared memory per CTA on SM100 (227 KiB) less a margin for the static mbarriers. +_MAX_DYNAMIC_SMEM = 227 * 1024 - 1024 # capacity: fixed: the device's shared-memory limit + +_logged_shapes = set() + + +def _smem_bytes(m: int, k: int, warps: int, cluster_k: int, row_tiles: int) -> int: + """Dynamic shared memory of one CTA, mirroring the host-side size in ``launch``.""" + kc = k // cluster_k + kc_blocks = kc // 32 + tile = 16 * row_tiles * 8 * (6 if m > 32 else 4 if m > 16 else 2 if m > 8 else 1) + w_bytes = 16 * row_tiles * (kc + 32) + red_bytes = warps * tile * 4 + return ( + w_bytes + + row_tiles * (kc_blocks // 4) * 256 + + m * kc + + (m * kc_blocks + 15) // 16 * 16 + + (cluster_k - 1) * tile * 4 + + (0 if red_bytes <= w_bytes else red_bytes) + ) + + +def get_config(n: int, k: int) -> Optional[tuple]: + """Return (warps, cluster_k, max_m, row_tiles) for an (N, K) weight, or None when no configuration is built.""" + return _CONFIGS.get((n, k)) + + +def mxfp8_low_m_gemm( + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + use_pdl: bool = False, + config: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """Fused MXFP8 quantize + GEMM; ``weight_scale`` is the swizzled 128x4 buffer. + + Returns None for unsupported operands so callers keep the native path. + """ + if ( + output_dtype != torch.bfloat16 + or x.dtype != torch.bfloat16 + or torch.cuda.get_device_capability(x.device)[0] < 10 + ): + return None + n, k = weight.shape + config = config or get_config(n, k) + if config is None or config not in _CONFIGS.values(): + return None + warps, cluster_k, max_m, row_tiles = config + if ( + x.shape[0] > max_m + or n % (16 * row_tiles) != 0 + or k % (cluster_k * 128) != 0 + or (k // cluster_k // 32) % warps != 0 + or x.data_ptr() % 16 != 0 + or x.stride(0) % 8 != 0 + or weight.data_ptr() % 16 != 0 + or weight_scale.data_ptr() % 16 != 0 + or _smem_bytes(x.shape[0], k, warps, cluster_k, row_tiles) > _MAX_DYNAMIC_SMEM + ): + return None + if (n, k) not in _logged_shapes: + _logged_shapes.add((n, k)) + logger.info( + f"[MXFP8 low-M] fused decode GEMM engaged: N={n} K={k} " + f"warps={warps} cluster_k={cluster_k} row_tiles={row_tiles} pdl={use_pdl}" + ) + return _low_m_gemm_op(x, weight, weight_scale, warps, cluster_k, row_tiles, use_pdl) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/norm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/norm/__init__.py new file mode 100644 index 000000000000..acec8fdbb26f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/norm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Normalization entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/norm/m3_embedding_firstnorm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/norm/m3_embedding_firstnorm.py new file mode 100644 index 000000000000..1842171bde1a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/norm/m3_embedding_firstnorm.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-step embedding gather and the first layer's input RMSNorm in one CuTe DSL launch, into caller buffers.""" + +from __future__ import annotations + +from typing import Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..embedding_firstnorm.op import DecodeEmbedding, m3_embed_norm + +__all__ = ["DecodeEmbedding", "m3_embedding_firstnorm"] + +VOCAB = 200064 # MiniMax-M3's vocabulary; the decode table is the full (replicated) embedding + + +class M3EmbeddingFirstnorm(OpWrapper): + """Write ``table[ids]`` into ``raw`` (bf16 [M, 6144], a zero row for an id outside ``[0, V)``) and + ``raw * rsqrt(mean(raw^2) + eps) * weight`` into ``out``; returns ``(out, raw)`` -- the first layer's normed input + and its residual. ``ids`` int32 or int64 [M], M = 1..8; ``table`` bf16 [V, 6144] contiguous; ``weight`` bf16 + [6144]. The first call per (M, id dtype, table shape, PDL) compiles (``DecodeEmbedding`` does it at build).""" + + ARCHS = frozenset({Arch.SM_103}) + + # The target builds every (rows 1..8, id dtype) owner; the kernel is one CTA per row, so the rows differ only in + # grid size: the ends of the range and both id dtypes. + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, int32 ids", spec=dict(rows=1, id_dtype=torch.int32)), + Cell(why="eight requests, int32 ids", spec=dict(rows=8, id_dtype=torch.int32)), + Cell(why="one request, int64 ids", spec=dict(rows=1, id_dtype=torch.int64)), + Cell( + why="six rows (one request verifying a draft), int64 ids", + spec=dict(rows=6, id_dtype=torch.int64), + ), + # Every other row count the decode dispatch takes (dispatch.EMBED_FIRSTNORM), both id dtypes. + *( + Cell( + why=f"{rows} rows: a decode row count between the ones above", + spec=dict(rows=rows, id_dtype=torch.int32 if rows % 2 else torch.int64), + ) + for rows in (2, 3, 4, 5, 7) + ), + ) + + note = """ + The gathered rows are exact. The norm is flashinfer's RMSNorm arithmetic in its reduction order with a fast + rsqrt, where `reference` sums in fp64 and takes torch's rsqrt: within the band the test derives, not bit for bit. + An id outside [0, V) gives a zero row and a zero norm (torch's embedding would raise). No accepted input is known + to compute wrongly: the host side rejects any other dtype, shape or a strided buffer. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + return m3_embed_norm(ids, table, weight, eps, raw, out, use_pdl) + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """New tensors (``raw`` / ``out`` are not written); `use_pdl` is ignored.""" + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] * valid.unsqueeze(1).to( + table.dtype + ) + x = rows.double() + rstd = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) + return (x * rstd * weight.double()).to(torch.bfloat16), rows + + +m3_embedding_firstnorm = M3EmbeddingFirstnorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/__init__.py new file mode 100644 index 000000000000..b3611a4b7866 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampling entries this target alone calls: the spec sampler's top-p renorm override (ST-01 TOPP_DET), its softmax +override (ST-01-SM SOFTMAX_MC) and its chain speculative sampling override (ST-01-REJ REJ_MC).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_chain_spec_mc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_chain_spec_mc.py new file mode 100644 index 000000000000..2d51fd37b047 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_chain_spec_mc.py @@ -0,0 +1,266 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ST-01-REJ REJ_MC: the spec sampler's chain speculative sampling (rejection) on many CTAs per request, exact vs +``torch.ops.trtllm.chain_speculative_sampling`` (``trtllm::minimax_m31_chain_speculative_sampling_mc``, +kernels/minimaxM31/chainSpecSamplingMcSm103.cu). + +The target installs ``chain_spec_mc`` as the rejection sampler's override +(``speculative/interface.set_chain_speculative_sampling_override``); it returns None, and the sampler keeps the stock +op, whenever the op is not built or the input is outside the kernel's contract. + +Workspace: one zero-initialized byte buffer per device, ``MAX_BATCH`` requests of +``minimax_m31_chain_speculative_sampling_mc_ws_bytes()``, kept for the life of the process (CUDA graphs captured with +it keep its address). The kernel leaves it ready for the next launch, so it is zeroed once. ``prebuild`` allocates it +outside any CUDA-graph capture (the target calls it from ``cache_derived_state``, before the executor's warmup and +captures); a call that finds no workspace allocates one when the stream is not capturing, and inside a capture returns +None (the stock op for that call) rather than allocating from the graph's private pool. Calls that share it must be +stream-ordered (the sampler's are). +""" + +from __future__ import annotations + +import os +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) + +# The largest CTA count per request (the grid is batch x ctas_per_row 1024-thread CTAs, all co-resident). +MAX_CTAS_PER_ROW = 16 + +# The requests the workspace holds and the override takes: the TP8 / TP4 K7 calls up to c8. Microbenchmarks +# gated it bitwise at batch 1 / 2 / 4 / 8 and measured 31.2 vs 41.2 us per call at batch 1; larger calls keep the stock +# kernel. +MAX_BATCH = 8 + +# The widest vocab the kernel takes (64 tiles of 4096; kChainSpecMcMaxVocab). +MAX_VOCAB = 64 * 4096 + +# The op when it is built (or overlaid as tensorrt_llm/libs/libm31_chain_spec_mc.so), decided once per process. +_STATE: dict = {} +# Per device index: the persistent workspace and the SM count. +_WORKSPACES: dict = {} +_SMS: dict = {} + + +def chain_spec_mc_op(): + """``trtllm::minimax_m31_chain_speculative_sampling_mc`` if this build carries it (in-tree, or the overlay + library), else None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_chain_speculative_sampling_mc", None) + if op is None: + import tensorrt_llm + + lib = os.path.join( + os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_chain_spec_mc.so" + ) + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_chain_speculative_sampling_mc", None) + _STATE["op"] = op + _STATE["ws_bytes"] = ( + int(torch.ops.trtllm.minimax_m31_chain_speculative_sampling_mc_ws_bytes()) + if op is not None + else 0 + ) + return _STATE["op"] + + +def ws_bytes_per_row() -> int: + """Workspace bytes per request (0 when the op is not built).""" + chain_spec_mc_op() + return _STATE["ws_bytes"] + + +def sm_count(device: torch.device) -> int: + """The device's SM count, cached per device.""" + idx = torch.device(device).index + idx = torch.cuda.current_device() if idx is None else idx + if idx not in _SMS: + _SMS[idx] = torch.cuda.get_device_properties(idx).multi_processor_count + return _SMS[idx] + + +def ctas_per_row(batch: int, sms: int) -> int: + """The largest power of two <= MAX_CTAS_PER_ROW with ctas x batch <= sms (1 past that: one CTA per request needs + no co-residency).""" + g = MAX_CTAS_PER_ROW + while g > 1 and g * batch > sms: + g //= 2 + return g + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's zeroed workspace for MAX_BATCH requests; must run outside a CUDA-graph capture. + Returns it, or None when the op is not built or the stream is capturing.""" + if chain_spec_mc_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.zeros( + MAX_BATCH * ws_bytes_per_row(), dtype=torch.uint8, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31ChainSpecMc(OpWrapper): + """``trtllm::chain_speculative_sampling(draft_probs, draft_token_ids, target_probs, seed, offset)`` bit for bit + (accepted tokens [batch, D + 1] and token counts [batch]) in ONE launch of batch x ctas_per_row CTAs (stock: one + CTA per request). ``workspace`` is the persistent zeroed buffer of ``prebuild``. ``__call__`` returns None (the + caller keeps the stock op) for a non-fp32, non-contiguous or misaligned probs, a vocab not divisible by 4 or above + MAX_VOCAB, mismatched shapes, non-int32 draft tokens, non-int64 or mis-sized seed / offset, inputs on different + devices, more requests than the workspace holds, a batch x ctas_per_row above the SM count, or when the op is not + built.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The K7 calls were checked bitwise on GB300 over 5 cases x 64 seeds at ctas_per_row 4 / 8 / 16. + CELLS: tuple[Cell, ...] = ( + Cell(why="K7 at c1 (vocab 200064, 16 CTAs)", spec=dict(batch=1, k=7, v=200064)), + Cell(why="K7 at c2 (vocab 200064)", spec=dict(batch=2, k=7, v=200064)), + Cell(why="K7 at c4 (vocab 200064)", spec=dict(batch=4, k=7, v=200064)), + Cell(why="K7 at c8 (vocab 200064, 128 CTAs)", spec=dict(batch=8, k=7, v=200064)), + ) + + note = """ + Exact by construction: every CTA of a request runs the stock Philox acceptance loop, the (target, residual) mass is + the stock 1024 per-thread chains into the same cub BlockReduce, each tile sum is the stock per-tile + cub reduce, and one CTA runs the stock tile-order prefix and in-tile cub BlockScan search. Default flags, as the + stock kernel. + """ + + def __call__( + self, + draft_probs: torch.Tensor, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, + workspace: Optional[torch.Tensor], + ctas_per_row: int, + ) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + if ( + workspace is None + or not all( + isinstance(t, torch.Tensor) + for t in (draft_probs, draft_token_ids, target_probs, seed, offset) + ) + or not draft_probs.is_cuda + or draft_probs.dtype != torch.float32 + or target_probs.dtype != torch.float32 + or draft_token_ids.dtype != torch.int32 + or seed.dtype != torch.int64 + or offset.dtype != torch.int64 + or draft_probs.dim() != 3 + or target_probs.dim() != 3 + or draft_token_ids.dim() != 2 + ): + return None + batch, k, v = draft_probs.shape + if ( + batch < 1 + or tuple(draft_token_ids.shape) != (batch, k) + or tuple(target_probs.shape) != (batch, k + 1, v) + or seed.numel() != batch + or offset.numel() != batch + or v % 4 != 0 + or v > MAX_VOCAB + or any( + not t.is_contiguous() or t.device != draft_probs.device + for t in (draft_token_ids, target_probs, seed, offset, workspace) + ) + or not draft_probs.is_contiguous() + or draft_probs.data_ptr() % 16 != 0 + or target_probs.data_ptr() % 16 != 0 + or ctas_per_row < 1 + or 1024 % ctas_per_row != 0 + or (ctas_per_row > 1 and batch * ctas_per_row > sm_count(draft_probs.device)) + ): + return None + op = chain_spec_mc_op() + if op is None or workspace.numel() * workspace.element_size() < batch * ws_bytes_per_row(): + return None + tokens, num_tokens = op( + draft_probs, draft_token_ids, target_probs, seed, offset, workspace, int(ctas_per_row) + ) + return tokens, num_tokens + + def reference( + self, + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace=None, + ctas_per_row=None, + ) -> tuple[torch.Tensor, torch.Tensor]: + """The stock op (workspace and ctas_per_row are ignored).""" + return torch.ops.trtllm.chain_speculative_sampling( + draft_probs, draft_token_ids, target_probs, seed, offset + ) + + def is_valid( + self, + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace=None, + ctas_per_row=None, + **kwargs, + ) -> None: + assert draft_probs.dtype == torch.float32 and draft_probs.dim() == 3, ( + "draft_probs: fp32 [batch, D, vocab]" + ) + + +m31_chain_spec_mc = M31ChainSpecMc() + + +def chain_spec_mc( + draft_probs: torch.Tensor, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, +) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + """The rejection sampler's override: the stock op's (accepted tokens, token counts), or None (the sampler runs the + stock op) when the op is not built, the batch exceeds MAX_BATCH, no workspace can be had (first use inside a + capture) or the input is outside the contract.""" + if ( + not isinstance(draft_probs, torch.Tensor) + or not draft_probs.is_cuda + or draft_probs.dim() != 3 + ): + return None + batch = draft_probs.shape[0] + if batch < 1 or batch > MAX_BATCH or chain_spec_mc_op() is None: + return None + workspace = prebuild(draft_probs.device) + if workspace is None: + return None + return m31_chain_spec_mc( + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace, + ctas_per_row(batch, sm_count(draft_probs.device)), + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_softmax_mc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_softmax_mc.py new file mode 100644 index 000000000000..ac4d6dc8add2 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_softmax_mc.py @@ -0,0 +1,199 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ST-01-SM SOFTMAX_MC: the spec sampler's softmax with its write pass on many CTAs per row, exact vs +``flashinfer.sampling.softmax(logits, temperature)`` (``trtllm::minimax_m31_softmax_mc``, +kernels/minimaxM31/softmaxMcSm103.cu). + +The target installs ``softmax_mc`` as the sampler's softmax override (``sampler/ops/flashinfer.set_softmax_override``); +it returns None, and the sampler keeps the stock FlashInfer call, whenever the op is not built or the input is outside +the kernel's contract (FlashInfer's Path A shapes: rows <= 128 and vocab >= 24576; here also rows <= ``MAX_ROWS``). + +Workspace: one partials buffer per device (``MAX_ROWS`` x ceil(``MAX_VOCAB`` / 8192) float2, 8 KB), kept for the life +of the process (CUDA graphs captured with it keep its address). Every launch overwrites what it reads, so it needs no +initialization. ``prebuild`` allocates it outside any CUDA-graph capture (the target calls it from +``cache_derived_state``, before the executor's warmup and captures); a call that finds no workspace allocates one when +the stream is not capturing, and inside a capture returns None (stock FlashInfer for that call) rather than allocating +from the graph's private pool. Calls that share it must be stream-ordered (the sampler's are). +""" + +from __future__ import annotations + +import os +from typing import Optional, Union + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) + +# The write pass's CTAs per row (microbenchmark: 9.2 vs FlashInfer 21.8 us per call at 7-8 rows; bitwise at +# 4 / 8 / 16). +WRITE_CTAS = 16 + +# The rows the override takes: the TP8 / TP4 K7 c1 calls (the draft's 7 rows and the verify's 8), where the MB measured +# it. Larger calls keep the stock kernel. +MAX_ROWS = 8 + +# FlashInfer's Path A bounds (OnlineSoftmax: batch <= 128 and d >= 24576); outside them FlashInfer takes its +# single-block Path B, which this kernel does not reproduce. +PATH_A_MAX_ROWS = 128 +PATH_A_MIN_VOCAB = 24576 +# FlashInfer's Path A slice (DEFAULT_SLICE_SIZE) and the widest vocab the workspace holds. +SLICE = 8192 +MAX_VOCAB = 1 << 20 + +# The op when it is built (or overlaid as tensorrt_llm/libs/libm31_softmax_mc.so), decided once per process. +_STATE: dict = {} +# Per device index: the persistent partials workspace. +_WORKSPACES: dict = {} + + +def softmax_mc_op(): + """``trtllm::minimax_m31_softmax_mc`` if this build carries it (in-tree, or the overlay library), else None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_softmax_mc", None) + if op is None: + import tensorrt_llm + + lib = os.path.join( + os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_softmax_mc.so" + ) + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_softmax_mc", None) + _STATE["op"] = op + return _STATE["op"] + + +def partials_numel(rows: int, d: int) -> int: + """fp32 elements of the partials workspace for rows x d (one float2 per row and 8192-wide slice).""" + return 2 * rows * ((d + SLICE - 1) // SLICE) + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's partials workspace for MAX_ROWS rows of up to MAX_VOCAB; must run outside a + CUDA-graph capture. Returns it, or None when the op is not built or the stream is capturing.""" + if softmax_mc_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.empty( + partials_numel(MAX_ROWS, MAX_VOCAB), dtype=torch.float32, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31SoftmaxMc(OpWrapper): + """FlashInfer's ``softmax(logits, temperature)`` of fp32 logits [rows, d], bit for bit, on FlashInfer's Path A + shapes (rows <= 128, d >= 24576): FlashInfer's Map kernel, then its Reduce kernel's prologue in each of + ``write_ctas`` CTAs per row, each writing its share of the row (FlashInfer: one CTA writes the whole row). + ``temperature`` is None (1.0), a Python number (one temperature) or an fp32 [rows] tensor (one per row), as + FlashInfer takes it. ``partials`` is the persistent workspace of ``prebuild``. ``__call__`` returns None (the caller + keeps the stock call) for a non-fp32, non-contiguous or misaligned input, a width not divisible by 4, a shape + outside Path A or the workspace, a temperature tensor that is not a contiguous fp32 [rows] tensor on logits' + device, or when the op is not built.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The c1 K7 calls were checked bitwise on GB300. + CELLS: tuple[Cell, ...] = ( + Cell(why="the K7 draft's 7 rows at c1 (vocab 200064)", spec=dict(rows=7, d=200064)), + Cell(why="the K7 verify's 8 rows at c1 (vocab 200064)", spec=dict(rows=8, d=200064)), + Cell(why="one row (the smallest grid)", spec=dict(rows=1, d=200064)), + ) + + note = """ + Exact by construction: the Map kernel is FlashInfer's, every write CTA reduces the same partials with the same cub + BlockReduce and Float2SoftmaxReduceOp as FlashInfer's single Reduce CTA, and the per-element write + keeps FlashInfer's source form. Built with --use_fast_math, as FlashInfer's JIT builds it. + """ + + def __call__( + self, + logits: torch.Tensor, + temperature: Union[torch.Tensor, float, None], + partials: Optional[torch.Tensor], + write_ctas: int, + ) -> Optional[torch.Tensor]: + if ( + partials is None + or not isinstance(logits, torch.Tensor) + or not logits.is_cuda + or logits.dtype != torch.float32 + or logits.dim() != 2 + or not logits.is_contiguous() + or logits.shape[0] < 1 + or logits.shape[0] > PATH_A_MAX_ROWS + or logits.shape[1] < PATH_A_MIN_VOCAB + or logits.shape[1] % 4 != 0 + or logits.data_ptr() % 16 != 0 + or partials.device != logits.device + or partials.numel() * partials.element_size() + < 4 * partials_numel(logits.shape[0], logits.shape[1]) + or write_ctas < 1 + or write_ctas > 65535 + ): + return None + if temperature is None: + temp_arr, temp_val = None, 1.0 + elif isinstance(temperature, torch.Tensor): + if ( + temperature.dtype != torch.float32 + or temperature.dim() != 1 + or temperature.shape[0] != logits.shape[0] + or not temperature.is_contiguous() + or temperature.device != logits.device + ): + return None + temp_arr, temp_val = temperature, 0.0 + elif isinstance(temperature, (int, float)) and not isinstance(temperature, bool): + temp_arr, temp_val = None, float(temperature) + else: + return None + op = softmax_mc_op() + if op is None: + return None + out = torch.empty_like(logits) + op(logits, temp_arr, temp_val, partials, out, int(write_ctas)) + return out + + def reference(self, logits, temperature, partials=None, write_ctas=None) -> torch.Tensor: + """FlashInfer's softmax (partials and write_ctas are ignored).""" + import flashinfer.sampling + + return flashinfer.sampling.softmax(logits, temperature) + + def is_valid(self, logits, temperature, partials=None, write_ctas=None, **kwargs) -> None: + assert logits.dtype == torch.float32 and logits.dim() == 2, "logits: fp32 [rows, d]" + + +m31_softmax_mc = M31SoftmaxMc() + + +def softmax_mc( + logits: torch.Tensor, temperature: Union[torch.Tensor, float, None] +) -> Optional[torch.Tensor]: + """The sampler's softmax override: FlashInfer's softmax of logits [rows, d] at temperature, or None (the sampler + runs stock FlashInfer) when the op is not built, rows exceed MAX_ROWS, d exceeds MAX_VOCAB, no workspace can be had + (first use inside a capture) or the input is outside the contract.""" + if not isinstance(logits, torch.Tensor) or not logits.is_cuda or logits.dim() != 2: + return None + rows = logits.shape[0] + if rows < 1 or rows > MAX_ROWS or logits.shape[1] > MAX_VOCAB or softmax_mc_op() is None: + return None + partials = prebuild(logits.device) + if partials is None: + return None + return m31_softmax_mc(logits, temperature, partials, WRITE_CTAS) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_topp_det.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_topp_det.py new file mode 100644 index 000000000000..0d28b1e2699c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/sampling/m31_topp_det.py @@ -0,0 +1,248 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ST-01 TOPP_DET: the spec sampler's top-p renorm as one persistent C++ launch, exact vs FlashInfer's deterministic +``top_p_renorm_probs`` (``trtllm::minimax_m31_topp_renorm_det``, kernels/minimaxM31/toppRenormDetSm103.cu). + +TOPP_DET v3 (``trtllm::minimax_m31_topp_renorm_det_v3``, kernels/minimaxM31/toppRenormDetV3Sm103.cu) is the same +renorm, bitwise v1.2's output, with the redundant per-CTA selection and a programmatic dependent launch. It is taken +when it is built (in-tree, or overlaid as tensorrt_llm/libs/libm31_topp_det_v3.so) unless +``TRTLLM_M31_DISABLE_TOPPDET_V3`` is set; v1.2 otherwise. ``TRTLLM_M31_DISABLE_TOPPDET_T3`` keeps v3 on its global +radix passes (no candidate list). + +The target installs ``top_p_renorm_det`` as the sampler's top-p renorm override +(``sampler/ops/flashinfer.set_top_p_renorm_override``); it returns None, and the sampler keeps the stock FlashInfer +call, whenever the op is not built or the input is outside the kernel's contract. + +Workspace: one zero-initialized byte buffer per device, ``MAX_ROWS`` rows of +``minimax_m31_topp_renorm_det_ws_bytes()``, kept for the life of the process (CUDA graphs captured with it keep its +address). The kernel leaves it ready for the next launch, so it is zeroed once. ``prebuild`` allocates it outside any +CUDA-graph capture (the target calls it from ``cache_derived_state``, before the executor's warmup and captures); a +call that finds no workspace allocates one when the stream is not capturing, and inside a capture returns None +(stock FlashInfer for that call) rather than allocating from the graph's private pool. +""" + +from __future__ import annotations + +import hashlib +import os +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled + +# The largest CTA count per row (the persistent grid is rows x ctas_per_row 1024-thread CTAs, all co-resident). +MAX_CTAS_PER_ROW = 16 + +# The rows the workspace holds and the override takes: the TP8 / TP4 K7 c1 calls (the draft's 7 rows and the +# verify's 8), where the microbenchmarks measured it faster than stock FlashInfer (v1.2: 54.3 vs 104.2 us per +# call at 7 peaked rows, 54.8 vs 105.1 at 8). At 56 / 64 rows v1 was slower than FlashInfer's default mode (340 vs +# 187 / 199 us) and larger shapes are not re-measured, so larger calls keep the stock kernel. +MAX_ROWS = 8 + +# The op when it is built (or overlaid as tensorrt_llm/libs/libm31_topp_det.so), decided once per process. +_STATE: dict = {} +# Per device index: the persistent workspace and the SM count. +_WORKSPACES: dict = {} +_SMS: dict = {} + + +def _load_op(name: str, lib_name: str): + """``torch.ops.trtllm.`` if registered (in-tree), else after loading tensorrt_llm/libs/ when it + exists, else None. Returns (op, the library path loaded or None).""" + op = getattr(torch.ops.trtllm, name, None) + if op is not None: + return op, None + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", lib_name) + if not os.path.exists(lib): + return None, None + torch.ops.load_library(lib) + _m31_fake_ops.register() + return getattr(torch.ops.trtllm, name, None), lib + + +def topp_det_op(): + """The renorm op this process uses, decided once: ``trtllm::minimax_m31_topp_renorm_det_v3`` when it is built + and TOPPDET_V3 is not disabled, else ``trtllm::minimax_m31_topp_renorm_det`` when built, else None.""" + if "op" not in _STATE: + op, ws, v3 = None, 0, False + if not m31_disabled("TOPPDET_V3"): + op, lib = _load_op("minimax_m31_topp_renorm_det_v3", "libm31_topp_det_v3.so") + if op is not None: + v3 = True + ws = int(torch.ops.trtllm.minimax_m31_topp_renorm_det_v3_ws_bytes()) + tag = "in-tree" + if lib is not None: + with open(lib, "rb") as f: + digest = hashlib.md5(f.read(), usedforsecurity=False).hexdigest()[:8] + tag = f"libm31_topp_det_v3.so md5 {digest}" + logger.info(f"[M3.1] TOPP_DET v3 private op engaged ({tag})") + if op is None: + op, _ = _load_op("minimax_m31_topp_renorm_det", "libm31_topp_det.so") + ws = ( + int(torch.ops.trtllm.minimax_m31_topp_renorm_det_ws_bytes()) + if op is not None + else 0 + ) + _STATE["op"], _STATE["ws_bytes"], _STATE["v3"] = op, ws, v3 + # v3's candidate-list radix passes (bitwise the same output) unless TOPPDET_T3 is disabled. + _STATE["cand_list"] = v3 and not m31_disabled("TOPPDET_T3") + if v3: + logger.info( + "[M3.1] TOPP_DET v3 T3 candidate-list fused passes: " + f"{'on' if _STATE['cand_list'] else 'off'} (cap 4096)" + ) + return _STATE["op"] + + +def is_v3() -> bool: + """Whether the engaged op is TOPP_DET v3.""" + topp_det_op() + return _STATE["v3"] + + +def ws_bytes_per_row() -> int: + """Workspace bytes per row (0 when the op is not built).""" + topp_det_op() + return _STATE["ws_bytes"] + + +def sm_count(device: torch.device) -> int: + """The device's SM count, cached per device.""" + idx = torch.device(device).index + idx = torch.cuda.current_device() if idx is None else idx + if idx not in _SMS: + _SMS[idx] = torch.cuda.get_device_properties(idx).multi_processor_count + return _SMS[idx] + + +def ctas_per_row(rows: int, sms: int) -> int: + """The largest power of two <= MAX_CTAS_PER_ROW with ctas x rows <= sms (1 past that: one CTA per row needs no + co-residency).""" + g = MAX_CTAS_PER_ROW + while g > 1 and g * rows > sms: + g //= 2 + return g + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's zeroed workspace for MAX_ROWS rows; must run outside a CUDA-graph capture. + Returns it, or None when the op is not built or the stream is capturing.""" + if topp_det_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.zeros( + MAX_ROWS * ws_bytes_per_row(), dtype=torch.uint8, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31ToppDet(OpWrapper): + """FlashInfer's ``top_p_renorm_probs(probs, top_p, is_deterministic=True)`` of fp32 probs [rows, d] with fp32 + per-row top_p [rows], bit for bit on every row with top_p < 1, in ONE launch of rows x ctas_per_row CTAs + (FlashInfer's: five launches, apply on one CTA per row, slow on high-entropy rows). Rows with top_p >= 1 keep every + element (FlashInfer drops tail buckets there in ~43% of rows, in both of its modes); their output is + ``ftz(ftz(p) * norm)``. ``workspace`` is the persistent zeroed buffer of ``prebuild``. ``__call__`` returns None + (the caller keeps the stock call) for a non-fp32, non-contiguous or misaligned input, a width not divisible by 4, + a top_p that is not a contiguous fp32 [rows] tensor on probs' device, more rows than the workspace holds, or when + the op is not built.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The c1 K7 calls were checked bitwise on GB300 over 8 row classes x 4 top_p modes x 16 seeds. + CELLS: tuple[Cell, ...] = ( + Cell(why="the K7 draft's 7 rows at c1 (vocab 200064)", spec=dict(rows=7, d=200064)), + Cell(why="the K7 verify's 8 rows at c1 (vocab 200064)", spec=dict(rows=8, d=200064)), + Cell(why="one row (the smallest grid, 16 CTAs)", spec=dict(rows=1, d=200064)), + ) + + note = """ + Exact by construction: the det histograms are integer (any aggregation order is FlashInfer's), the per-pass + selection is FlashInfer's last-block code on them, and the kept sum is FlashInfer's 1024 per-thread chains into + the same cub BlockReduce. Built with --use_fast_math, as FlashInfer's JIT builds it. + """ + + def __call__( + self, + probs: torch.Tensor, + top_p: torch.Tensor, + workspace: Optional[torch.Tensor], + ctas_per_row: int, + ) -> Optional[torch.Tensor]: + if ( + not isinstance(top_p, torch.Tensor) + or workspace is None + or not probs.is_cuda + or probs.dtype != torch.float32 + or probs.dim() != 2 + or not probs.is_contiguous() + or probs.shape[0] < 1 + or probs.shape[1] % 4 != 0 + or probs.data_ptr() % 16 != 0 + or top_p.dtype != torch.float32 + or top_p.dim() != 1 + or top_p.shape[0] != probs.shape[0] + or not top_p.is_contiguous() + or top_p.device != probs.device + or workspace.device != probs.device + or ctas_per_row < 1 + or 1024 % ctas_per_row != 0 + ): + return None + op = topp_det_op() + if ( + op is None + or workspace.numel() * workspace.element_size() < probs.shape[0] * ws_bytes_per_row() + ): + return None + out = torch.empty_like(probs) + if is_v3(): + op(probs, top_p, workspace, out, int(ctas_per_row), True, _STATE["cand_list"]) + else: + op(probs, top_p, workspace, out, int(ctas_per_row)) + return out + + def reference(self, probs, top_p, workspace=None, ctas_per_row=None) -> torch.Tensor: + """FlashInfer's deterministic renorm (rows with top_p < 1; workspace and ctas_per_row are ignored).""" + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + def is_valid(self, probs, top_p, workspace=None, ctas_per_row=None, **kwargs) -> None: + assert probs.dtype == torch.float32 and probs.dim() == 2, "probs: fp32 [rows, d]" + + +m31_topp_det = M31ToppDet() + + +def top_p_renorm_det(probs: torch.Tensor, top_p: torch.Tensor) -> Optional[torch.Tensor]: + """The sampler's top-p renorm override: the det renorm of probs [rows, d] at per-row top_p, or None (the sampler + runs stock FlashInfer) when the op is not built, rows exceed MAX_ROWS, no workspace can be had (first use inside + a capture) or the input is outside the contract.""" + if not isinstance(top_p, torch.Tensor) or not probs.is_cuda or probs.dim() != 2: + return None + rows = probs.shape[0] + if rows < 1 or rows > MAX_ROWS or topp_det_op() is None: + return None + workspace = prebuild(probs.device) + if workspace is None: + return None + return m31_topp_det(probs, top_p, workspace, ctas_per_row(rows, sm_count(probs.device))) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/__init__.py new file mode 100644 index 000000000000..d25c41dc9557 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Speculative-decoding entries this target alone calls: the Eagle3 verify ops, the capture / fc_norm RMSNorms and +the step-input staging.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_capture_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_capture_norm.py new file mode 100644 index 000000000000..131269ae7d07 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_capture_norm.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Eagle3 hidden-state capture with the drafter's fc_norm folded in (``m3_spec_tail.capture_norm``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_chunk_norm import rmsnorm_rows + + +class M3CaptureNorm(OpWrapper): + """``dst [M, H] = rmsnorm(bf16(h + r)) * w`` (``r`` optional), the rounding chunk_norm applies, into a capture + buffer slice whose row stride may exceed H. Returns ``dst``.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The target's captured layers (hidden 6144) into the 3-layer capture buffer (row stride 3 x 6144), over the + # verify rows of draft_len 3 (4 .. 32 rows); the residual is present for every captured decoder layer. + CELLS: tuple[Cell, ...] = ( + Cell( + why="4 verify rows of one request, layer slot 0, residual added", + spec=dict(m=4, slot=0, residual=True), + ), + Cell( + why="cap 8 (32 rows) into the last layer slot", spec=dict(m=32, slot=2, residual=True) + ), + Cell( + why="no residual (a capture point without one): plain RMSNorm", + spec=dict(m=8, slot=1, residual=False), + ), + ) + + note = """ + fp32 statistic (block sum, rsqrtf), one bf16 rounding after the bf16 sum h + r; the band states the occasional + one-ulp move against torch's statistic. + """ + + def __call__( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_capture_norm(h, r, w, eps, dst) + return dst + + def reference( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + """bf16 sum, then the normalized row; ``dst`` only says where the op writes.""" + x = (h.float() + r.float()).to(torch.bfloat16) if r is not None else h + return rmsnorm_rows(x, w, eps) + + def is_valid( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> None: + # The norm weight is read as H dense bf16 values; a strided view (a column of a stacked weight) is read + # past its elements without an error. + assert w.is_contiguous(), "w must be dense [H]" + + +m3_capture_norm = M3CaptureNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_chunk_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_chunk_norm.py new file mode 100644 index 000000000000..f6adb2207a20 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_chunk_norm.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 fc_norm over the captured layers in one launch: per-chunk RMSNorm of the concatenated hidden states +(``m3_spec_tail.chunk_norm``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3ChunkNorm(OpWrapper): + """``dst[r, c*H:(c+1)*H] = rmsnorm(src[r, c*H:(c+1)*H]) * w[c]`` for the C chunks of ``w [C, H]``; ``src`` may + have a row stride wider than C * H (the capture buffer). Writes ``dst [M, C*H]`` and returns it.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The Eagle3 capture buffer of the target (3 captured layers x hidden 6144, buffer rows wider than the layers + # when the final hidden is also captured), over the verify rows of draft_len 3: 4 / 8 / 32 rows. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request's 4 verify rows, dense [4, 3 x 6144]", + spec=dict(m=4, c=3, h=6144, extra_cols=0), + ), + Cell( + why="cap 8 (32 rows) read from the capture buffer with a fourth (final-hidden) slot in the row stride", + spec=dict(m=32, c=3, h=6144, extra_cols=6144), + ), + ) + + note = """ + fp32 statistic per chunk (block sum, rsqrtf), one bf16 rounding: an output moves by one bf16 ulp now and then + against torch's statistic; the cells' band states it. + """ + + def __call__( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + torch.ops.trtllm.minimax_m3_chunk_norm(src, w, eps, dst, pdl) + return dst + + def reference( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + """Each chunk normalized on its own; ``dst`` only says where the op writes, ``pdl`` shapes the launch.""" + c, h = w.shape + x = src[:, : c * h].reshape(src.shape[0], c, h) + return rmsnorm_rows(x, w, eps).reshape(src.shape[0], c * h) + + def is_valid( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ) -> None: + c, h = w.shape + assert w.is_contiguous(), "w must be dense [C, H] (chunk c read at c * H)" + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + + +m3_chunk_norm = M3ChunkNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_draft_input.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_draft_input.py new file mode 100644 index 000000000000..75f7a44dfcfb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_draft_input.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 draft pass's input in one launch: embedding lookup, both RMSNorms and the concat +(``m3_spec_tail.draft_input``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3DraftInput(OpWrapper): + """``out [M, 2H] = [rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden]`` and ``residual [M, H] = + hidden[rows]`` (rows default to 0..M-1) in one launch; ids outside the table embed to zero (the masked lookup of + a vocab-local embedding). Returns ``(out, residual)``.""" + + ARCHS = frozenset({Arch.SM_103}) + + # Every pass of the fused draft loop (dispatch.DRAFT_INPUT) at draft_len 3, drafter hidden 6144, eps 1e-6: the + # first pass of 1..8 requests (4..32 rows, the carried hidden read in place) and the later passes of 1..8 requests + # (one row each; the second pass gathers the last accepted rows from the first pass's 4 rows per request). + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"first pass of {n} request(s): {4 * n} rows, hidden read in place", + spec=dict(m=4 * n, h=6144, gather=False, invalid_id=False), + ) + for n in range(1, 9) + ), + *( + Cell( + why=f"second pass of {n} request(s): the last accepted rows gathered from {4 * n} rows", + spec=dict(m=n, h=6144, gather=True, invalid_id=False), + ) + for n in range(1, 9) + ), + Cell( + why="third pass of 3 requests: one row each, hidden read in place", + spec=dict(m=3, h=6144, gather=False, invalid_id=False), + ), + Cell( + why="one row with an out-of-table id: the embedding half is zero", + spec=dict(m=1, h=6144, gather=False, invalid_id=True), + ), + ) + + note = """ + The arithmetic is the in-tree RMSNorm's (fp32 statistic, (x * rsqrt) * w, one bf16 rounding); the statistic's + summation order differs from FlashInfer's, which moves an output by one bf16 ulp now and then (the test's band, + also held against the in-tree modules). The out-of-table id is the masked lookup of a vocab-local embedding; the + drafter's embedding is the whole table, so its ids are always in range. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + torch.ops.trtllm.minimax_m3_draft_input( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, pdl + ) + return out, residual + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + """The two normalized halves (fp32 statistic, one bf16 rounding) and the gathered hidden rows. ``out`` / + ``residual`` only say where the op writes; ``pdl`` only shapes the launch.""" + valid = (ids >= 0) & (ids < table.shape[0]) + emb = rmsnorm_rows(table[torch.where(valid, ids, 0).long()], w_embed, eps_embed) + emb = torch.where(valid[:, None], emb, torch.zeros_like(emb)) + hid = hidden[rows.long()] if rows is not None else hidden[: ids.shape[0]] + ref = torch.cat([emb, rmsnorm_rows(hid, w_hidden, eps_hidden)], dim=-1) + return ref, (hid.clone() if residual is not None else None) + + def is_valid( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> None: + h = table.shape[1] + # Rows are read as hidden + rows[r] * H and table + id * H: both must be dense H-wide rows. + assert table.is_contiguous() and hidden.is_contiguous(), ( + "table / hidden must be dense [*, H]" + ) + if rows is not None: + assert bool(((rows >= 0) & (rows < hidden.shape[0])).all()), "rows must index hidden" + else: + assert hidden.shape[0] >= ids.shape[0], "hidden must hold a row per id" + # Vector loads of 16 bytes: H must be a multiple of 8 within the register tile, and the weights aligned. + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + assert residual is None or ( + residual.is_contiguous() and residual.shape == (ids.shape[0], h) + ), "residual must be contiguous [M, H]" + + +m3_draft_input = M3DraftInput() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_draft_step.py new file mode 100644 index 000000000000..a4750f4c1238 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_draft_step.py @@ -0,0 +1,367 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One pass of the fused greedy Eagle3 draft loop: the drafter's vocab-sharded BF16 head (streamed from its lossless +12-bit coding, ``ops.gemm.m3_target_head``, or as BF16 fragments), the cross-rank argmax and the loop's per-pass +bookkeeping in one kernel (``m3_draft_step.draft_step``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +from ..gemm.m3_target_head import decode as decode_coded +from .m3_shard_accept import exchange_tokens, local_argmax, make_keys + +#: Requests (head rows) per pass, and exchange rows per call half of the workspace. +MAX_TOKENS = 8 +#: 16-row weight tiles one CTA can own (the kernel's shared-memory accumulator). +MAX_TILES_PER_CTA = 32 +#: Draft length of the fused loop, and the drafter's TP4 head shard (200064-id vocabulary) and hidden size. +DRAFT_LEN = 3 +DRAFT_SHARD = 50016 +HIDDEN = 6144 + + +def bf16_fragments(weight: torch.Tensor) -> torch.Tensor: + """BF16 [N, K] -> the same values in A-fragment order [N / 16, K / 16, 256]. + + One lane's 16 bytes of a 16-row tile's k16 step are the m16n8k16 A fragment: element (row h * 8 + g, k ks * 16 + + hk * 8 + jj * 2 + e) goes to [tile][ks][g][jj][hk][h][e] (lane = g * 4 + jj). + """ + n, k = weight.shape + assert n % 16 == 0 and k % 16 == 0 and weight.dtype == torch.bfloat16, (n, k, weight.dtype) + q = weight.view(n // 16, 2, 8, k // 16, 2, 4, 2).permute(0, 3, 2, 5, 4, 1, 6).contiguous() + return q.view(n // 16, k // 16, 256) + + +def from_fragments(wf: torch.Tensor) -> torch.Tensor: + """The inverse of ``bf16_fragments``: [N / 16, K / 16, 256] -> [N, K].""" + tiles, ks, _ = wf.shape + return ( + wf.view(tiles, ks, 8, 4, 2, 2, 2).permute(0, 5, 2, 1, 4, 3, 6).reshape(tiles * 16, ks * 16) + ) + + +def head_weight( + wf: torch.Tensor, + k: int, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The BF16 head shard [N, K] the kernel multiplies by: the fragment-ordered copy, or (``exc_off`` given) the + 12-bit coded stream with its exception lists decoded.""" + if exc_off is None: + return from_fragments(wf) + n = (exc_off.numel() - 1) * 16 + return decode_coded(wf, exp_base, exc_off, exc_pos, exc_w, n, k) + + +def shard_logits( + x: torch.Tensor, + wf: torch.Tensor, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The in-tree head's BF16 logits of the rows x [M, K] against the shard (``head_weight``): the fp32 product, + rounded once to BF16 (the BF16 GEMM's output), as fp32.""" + w = head_weight(wf, x.shape[1], exc_off, exc_pos, exc_w, exp_base) + with true_fp32_matmul(): + logits = x.float() @ w.float().t() + return logits.to(torch.bfloat16).float() + + +class M3DraftStep(OpWrapper): + """One fused greedy draft pass over ``rows <= 8`` requests (one head row each). + + Head input ``x[x_rows[b]]`` (pass 0: the last accepted verify row of each request) or ``x[b]``. The head is this + rank's BF16 lm_head shard in fragment order (``m3_draft_step.bf16_fragments``, global ids ``vocab_offset ..``), + or with ``exc_off`` / ``exc_pos`` / ``exc_w`` / ``exp_base`` its lossless 12-bit coding (``wf`` = the uint8 stream + of ``m3_target_head.encode_chunked``, which must round-trip to the BF16 shard); each logit is the fp32 dot product + rounded to BF16 and the row's token is the global argmax over all ranks' + shards (ties, -0 / +0 included, to the lowest global id), exchanged through the multicast slots + ``[2][8][world]``. The last CTA then writes ``draft_out[b, step]`` and ``ids_next[b]`` (the token), ``pos_next[b] + = pos_src[x_rows[b] or b] + 1``, and advances the draft KV lengths as the in-tree draft loop does between passes + (pass 0 with ``num_acc``: ``kv_lens -= draft_len - num_acc`` and ``seq_lens = 1``; later passes: ``kv_lens += + 1``). ``out_keys`` (optional) receives each row's winning key. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # Every pass of the fused draft loop (dispatch.DRAFT_STEP): draft_len 3 at 1..8 requests, the drafter's + # full-vocabulary BF16 head (50016 rows per rank of 4, hidden 6144). Pass 0 reads the verify rows' hidden (4 per + # request) through x_rows and rolls the KV back; the later passes read one row per request. + _PASSES: tuple[Cell, ...] = ( + *( + Cell( + why=f"pass 0 at {n} request(s): head rows gathered from {4 * n} verify rows, KV rollback, seq_lens = 1", + spec=dict( + nreq=n, + draft_len=DRAFT_LEN, + step=0, + rank=n % 4, + world=4, + vs=DRAFT_SHARD, + k=HIDDEN, + ), + ) + for n in range(1, MAX_TOKENS + 1) + ), + Cell( + why="pass 1 at two requests: one row each, kv_lens + 1, rank 3's ids", + spec=dict( + nreq=2, draft_len=DRAFT_LEN, step=1, rank=3, world=4, vs=DRAFT_SHARD, k=HIDDEN + ), + ), + Cell( + why="last pass (step 2) at 8 requests, rank 0", + spec=dict( + nreq=8, draft_len=DRAFT_LEN, step=2, rank=0, world=4, vs=DRAFT_SHARD, k=HIDDEN + ), + ), + Cell( + why="pass 2 at 5 requests with exact cross-rank ties (the lower global id wins)", + spec=dict( + nreq=5, + draft_len=DRAFT_LEN, + step=2, + rank=1, + world=4, + vs=DRAFT_SHARD, + k=HIDDEN, + ties=True, + ), + ), + ) + # The coded head (the fused draft loop's default) at every pass shape, and the BF16 fragments (the coding's + # fail-closed path) at the first and last pass shapes and with ties. + CELLS: tuple[Cell, ...] = ( + *(Cell(why=f"coded: {c.why}", spec=dict(c.spec, coded=True)) for c in _PASSES), + *( + Cell(why=f"bf16: {c.why}", spec=dict(c.spec, coded=False)) + for c in (_PASSES[0], _PASSES[MAX_TOKENS - 1], _PASSES[-1]) + ), + ) + + note = """ + The head is the in-tree BF16 head's arithmetic (BF16 weights and activations, fp32 accumulation, a BF16 logit; + the coded head's weight is the BF16 shard bit for bit; its exceptions decode to +0 in the MMAs and their exact + products are added at the tile's flush) + and the argmax is the in-tree one over BF16 logits, so a token differs from the in-tree path's only where the fp32 + sum, accumulated in another order, rounds to the other side of a BF16 boundary and that decides the row's maximum. + The cells' rows are decidable and the peers' keys sit 0.25 away (or tie exactly); the edge tests (forced BF16 + ties, -0 / +0, top-2 one fp32 step apart that round to one BF16 value, adjacent BF16 values) use integer data + whose fp32 sums are exact in any order and hold the kernel to the in-tree path bit for bit. The cross-rank + exchange is certified on one device with the peers' keys written into the slots (see m3_shard_accept). + """ + + def __call__( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + ) -> None: + torch.ops.trtllm.minimax_m3_draft_step( + x, + x_rows, + rows, + wf, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + num_ctas, + pdl, + out_keys, + exc_off, + exc_pos, + exc_w, + exp_base, + ) + + def reference( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + ) -> dict[str, torch.Tensor]: + """Every buffer the pass writes, computed on copies: "logits" (this rank's BF16 logits as fp32), "local_keys" + (this rank's key per row), "keys" (the winning key), "tokens", "draft_out", "ids_next", "pos_next", "kv_lens" + and "seq_lens". + + Reads ``ws_uc``, ``counters`` and ``kv_lens`` before the call. ``keys``, ``ws_mc_ptr``, ``num_ctas``, ``pdl`` + and ``out_keys`` do not change the result. + """ + h = x[x_rows.long()] if x_rows is not None else x[:rows] + logits = shard_logits(h, wf, exc_off, exc_pos, exc_w, exp_base) + best, idx = local_argmax(logits, vocab_offset) + local_keys = make_keys(best, idx) + tokens = exchange_tokens(local_keys, ws_uc, counters, rank, world, MAX_TOKENS) + half = int(counters[1].item()) & 1 + slots = ws_uc.view(torch.int64)[ + half * MAX_TOKENS * world : (half * MAX_TOKENS + rows) * world + ] + slots = slots.view(rows, world).clone() + slots[:, rank] = local_keys + sign = torch.iinfo(torch.int64).min + win = (slots ^ sign).max(dim=1).values ^ sign + d_out = draft_out.clone() + d_out[:rows, step] = tokens + src = x_rows.long() if x_rows is not None else torch.arange(rows, device=x.device) + lens = kv_lens.clone() + sl = seq_lens.clone() if seq_lens is not None else None + if step == 0 and num_acc is not None: + lens[:rows] -= draft_len - num_acc[:rows] + sl[:rows] = 1 + elif step > 0: + lens[:rows] += 1 + out = { + "logits": logits, + "local_keys": local_keys, + "keys": win, + "tokens": tokens, + "draft_out": d_out, + "ids_next": tokens.clone(), + "pos_next": pos_src[src] + 1, + "kv_lens": lens, + } + if sl is not None: + out["seq_lens"] = sl + return out + + def is_valid( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + ) -> None: + assert not bool(keys[:rows].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + # The kernel reads the head row x + x_rows[b] * K and the position pos_src[x_rows[b]] with no bound. + if x_rows is not None: + assert bool(((x_rows[:rows] >= 0) & (x_rows[:rows] < x.shape[0])).all()), ( + "x_rows must index x" + ) + assert bool((x_rows[:rows] < pos_src.shape[0]).all()), ( + "pos_src must cover the head rows" + ) + else: + assert pos_src.shape[0] >= rows, "pos_src must cover the head rows" + # Each CTA accumulates at most MAX_TILES_PER_CTA tiles in shared memory. + tiles = wf.shape[0] if exc_off is None else exc_off.numel() - 1 + assert -(-tiles // num_ctas) <= MAX_TILES_PER_CTA, "too many weight tiles per CTA" + if exc_off is not None: + # The exceptions are read per tile in list order with no bound against the coded stream. + assert int(exc_off[-1].item()) == exc_pos.numel(), ( + "exc_off must end at the exception count" + ) + assert bool((exc_off[1:] >= exc_off[:-1]).all()), "exc_off must be non-decreasing" + # A NaN logit maps to a key above every finite one and wins the row silently. + h = x[x_rows.long()] if x_rows is not None else x[:rows] + assert not bool(torch.isnan(h).any()), "NaN head inputs win the argmax" + if num_acc is not None: + na = num_acc[:rows] + assert bool(((na >= 1) & (na <= draft_len + 1)).all()), ( + "num_acc counts must lie in [1, draft_len + 1]" + ) + + +m3_draft_step = M3DraftStep() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_force_vec.py new file mode 100644 index 000000000000..af4b06e3999f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_force_vec.py @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance, drawn as the in-tree worker draws them.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..m3_force_vec import force_vec_kernel + + +class M3ForceVec(OpWrapper): + """Bump ``counter`` (int64 [1]) by one, then for each of the ``batch_size`` requests ``b`` write + ``min(base_total + (pool[(c * counter_stride + b * slot_stride) & pool_mask] < frac), max_total)`` into ``out`` + (int32), ``c`` the bumped counter: the in-tree ``_apply_force_accepted_tokens`` draw over its fixed-seed fp32 + ``pool``. ``block`` is the launch width (a power of two >= batch_size; 16, 32 or 64 are compiled at warmup). + + Only the forced-acceptance testing knob (``TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`` with a fractional part) + runs it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, fraction 0.5", spec=dict(batch=1, block=16, frac=0.5)), + Cell(why="eight requests, fraction 0.3", spec=dict(batch=8, block=16, frac=0.3)), + ) + + note = """ + Integer arithmetic and one fp32 comparison: `reference` must match exactly. + """ + + def __call__( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + force_vec_kernel[(1,)]( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK=block, + ) + return out + + def reference( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + """The counts (a new tensor; ``counter`` and ``out`` are not written).""" + c = int(counter.item()) + 1 + slots = torch.arange(batch_size, dtype=torch.int64, device=pool.device) + idx = (c * counter_stride + slots * slot_stride) & pool_mask + r = pool[idx] + return torch.clamp(base_total + (r < frac).to(torch.int32), max=max_total).to(torch.int32) + + +m3_force_vec = M3ForceVec() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_sampled_accept.py new file mode 100644 index 000000000000..e904608fec13 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_sampled_accept.py @@ -0,0 +1,269 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampled (temperature + top-p) Eagle3 verification over the TP-sharded vocabulary in one kernel +(``m3_sampled_accept.sampled_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_shard_accept import strict_accept + +#: Verify rows per call. +MAX_ROWS = 48 +#: The fused sampling kernel's 1 / T guard. +TEMP_EPSILON = 1e-6 + + +def nucleus_probs( + logits: torch.Tensor, temperatures: torch.Tensor, top_ps: torch.Tensor +) -> torch.Tensor: + """Per-row sampling distribution [rows, Vs] (fp32): w = exp(x / (T + 1e-6) - max), kept iff the mass strictly + above its weight is at most top_p * sum(w) (ties included), renormalized over the kept tokens.""" + inv = 1.0 / (temperatures.float() + TEMP_EPSILON) + x = logits.float() * inv[:, None] + w = torch.exp(x - x.max(dim=1, keepdim=True).values).double() + z = w.sum(dim=1, keepdim=True) + order = torch.sort(w, dim=1, descending=True) + sw = order.values + # mass strictly above each sorted weight: the cumulative sum before the first entry of its tie group + csum = torch.cumsum(sw, dim=1) - sw + first = torch.ones_like(sw, dtype=torch.bool) + first[:, 1:] = sw[:, 1:] != sw[:, :-1] + group = torch.cumsum(first.long(), dim=1) - 1 + starts = torch.zeros_like(csum) + starts.scatter_reduce_(1, group, csum, reduce="amin", include_self=False) + above = torch.gather(starts, 1, group) + keep_sorted = above <= top_ps.double()[:, None] * z + keep = torch.zeros_like(keep_sorted) + keep.scatter_(1, order.indices, keep_sorted) + p = torch.where(keep, w, torch.zeros_like(w)) + return (p / p.sum(dim=1, keepdim=True)).float() + + +class M3SampledAccept(OpWrapper): + """Sampled verification of ``rows = nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard. + + Each row draws one token from its top-p nucleus of softmax(logits / T) over the full (all-rank) vocabulary by + distributed rejection sampling (Philox keyed by the row's seed / offset, the row, the attempt and the CTA), then + the strict acceptance runs against ``draft`` as in ``m3_shard_accept``: ``acc_tokens``, ``num_acc``, + ``gather_ids``, optional ``next_new[:, 0]``. ``tokens`` receives the sampled token per row; ``extra_attempts`` + accumulates exchange rounds beyond the second. The exchange goes through the multicast workspace (two call + halves of ``minimax_m3_sampled_accept_ws_words(world)`` words) with the (finished rows, call) ``counters``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # draft_len 3 (T = 4 rows per request) at the TP4 caps: 1 / 2 / 8 requests (4 / 8 / 32 rows, 8 CTAs per row) + # and the 48-row maximum (4 CTAs per row); the 50016-column target shard. Single rank (see note). + CELLS: tuple[Cell, ...] = ( + Cell( + why="cap 1 (4 rows), top_p 0.8 over a 12-token head: the draws follow the nucleus distribution", + spec=dict( + nreq=1, vs=50016, vocab_offset=0, check="distribution", top_p=0.8, temperature=0.7 + ), + ), + Cell( + why="cap 8 (32 rows, 256 co-resident CTAs): top_p ~ 0 keeps only the argmax, token exact", + spec=dict( + nreq=8, + vs=50016, + vocab_offset=2 * 50016, + check="argmax", + top_p=1e-7, + temperature=1.0, + ), + ), + Cell( + why="cap 2 (8 rows) with top_p = 1 (no cut: the first draw stands), next_new and forced counts", + spec=dict( + nreq=2, + vs=50016, + vocab_offset=50016, + check="distribution", + top_p=1.0, + temperature=1.0, + force="vec", + ), + ), + Cell( + why="48 rows (4 CTAs per row): the second launch geometry, nucleus distribution at top_p 0.5", + spec=dict( + nreq=12, + vs=50016, + vocab_offset=3 * 50016, + check="distribution", + top_p=0.5, + temperature=1.3, + force="total", + ), + ), + ) + + note = """ + What is sampled is gated statistically, the rest exactly. Every drawn token must lie in the reference nucleus; + the empirical distribution over many calls (distinct Philox offsets) must sit within a stated total-variation + band of the reference's; the acceptance outputs must equal the strict acceptance of the drawn tokens exactly. + The Philox stream itself is not reproduced in torch: the kernel's draw uses the hardware exp / log + approximations (__expf, __logf) and a CTA-order inverse CDF, which torch does not reproduce bit for bit. + + Certified at world 1 only. The decision is replicated from every rank's per-round records, which another rank + computes from its own shard on the fly, so peers cannot be stood in for on one device; the cross-rank path + needs the multi-device job and is not covered here. Rows with top-k or min-p must not reach this op (it applies + neither); is_valid states it. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + ) -> None: + torch.ops.trtllm.minimax_m3_sampled_accept( + logits, + vocab_offset, + temperatures, + top_ps, + seeds, + offsets, + ws_uc, + ws_mc_ptr, + counters, + tokens, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + force_vec, + extra_attempts, + use_pdl, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + ) -> dict: + """{"probs": the per-row nucleus distribution over this shard [rows, Vs] (global id = vocab_offset + + column), "accept": tokens -> the strict acceptance outputs}. The draw itself is random; ``seeds`` / + ``offsets`` select it and do not change the distribution. World 1: this shard is the whole vocabulary.""" + rows = logits.shape[0] + probs = nucleus_probs(logits, temperatures[:rows], top_ps[:rows]) + + def accept(drawn: torch.Tensor) -> dict[str, torch.Tensor]: + return strict_accept(drawn, draft, batch_idx, force_total, next_new, force_vec) + + return {"probs": probs, "accept": accept} + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + ) -> None: + rows = logits.shape[0] + # top-k and min-p are not applied: a row that has them gets a token outside its real sampling set. + if top_ks is not None: + k = top_ks[:rows] + assert bool(((k <= 0) | (k >= logits.shape[1] * world)).all()), ( + "top-k rows must take the generic path" + ) + if min_ps is not None: + assert not bool((min_ps[:rows] > 0).any()), "min-p rows must take the generic path" + # Rows are request-major verify rows: tokens[b * (D + 1) + k]; a mismatched layout pairs drafts with the + # wrong rows. + assert rows == draft.shape[0] * (draft.shape[1] + 1), ( + "logits rows must be nreq * (draft_len + 1)" + ) + # Per-row parameters are read at the row index, unchecked. + for t in (temperatures, top_ps, seeds, offsets): + assert t.numel() >= rows and t.is_contiguous(), ( + "per-row parameters must cover every row" + ) + assert temperatures.dtype == torch.float32 and top_ps.dtype == torch.float32, ( + "fp32 sampling parameters" + ) + assert seeds.dtype == torch.int64 and offsets.dtype == torch.int64, ( + "int64 Philox seeds / offsets" + ) + # The finished-row counter elects the CTA that runs the acceptance. + assert int(counters[0].item()) == 0, ( + "counters[0] (finished rows) must be zero between calls" + ) + + +m3_sampled_accept = M3SampledAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_sampled_accept_ws_words.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_sampled_accept_ws_words.py new file mode 100644 index 000000000000..3e42fd5c50ad --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_sampled_accept_ws_words.py @@ -0,0 +1,43 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Size of the sampled-verify exchange workspace (``m3_sampled_accept.SampledAcceptWorkspace.create``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Exchange rounds per row, verify rows per call, CTAs per row (at most), uint64 words per record. +MAX_ATTEMPTS = 16 +MAX_ROWS = 48 +MAX_SPLITS = 8 +RECORD_WORDS = 4 + + +class M3SampledAcceptWsWords(OpWrapper): + """uint64 words of the multicast workspace ``minimax_m3_sampled_accept`` exchanges through for ``world`` ranks: + two call halves of ``[attempts][rows][world * splits][record words]``.""" + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="the tp4 target's workspace", spec=dict(world=4)), + Cell(why="a single rank (local workspace, no multicast)", spec=dict(world=1)), + ) + + note = """ + A host computation (no device work); it is an op so the workspace size and the kernel's indexing come from one + definition. + """ + + def __call__(self, world: int) -> int: + return torch.ops.trtllm.minimax_m3_sampled_accept_ws_words(world) + + def reference(self, world: int) -> int: + """Two halves of attempts x rows x (world x splits) records x record words.""" + return 2 * MAX_ATTEMPTS * MAX_ROWS * world * MAX_SPLITS * RECORD_WORDS + + +m3_sampled_accept_ws_words = M3SampledAcceptWsWords() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_shard_accept.py new file mode 100644 index 000000000000..90bb11ef84b3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_shard_accept.py @@ -0,0 +1,284 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Greedy Eagle3 verification from this rank's logits shard: per-row argmax, the cross-rank (value, index) exchange +and the strict acceptance in one kernel (``m3_shard_accept.shard_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Verify rows per call, and exchange rows per call half of its workspace. +SHARD_ROWS = 48 + +_LOW32 = 0xFFFFFFFF + + +def ordered_bits(v: torch.Tensor) -> torch.Tensor: + """The kernel's order-preserving uint32 image of fp32 values, as int64 (-0.0 maps to +0.0's image: they tie).""" + b = v.float().contiguous().view(torch.int32).to(torch.int64) & _LOW32 + b = torch.where(b == 0x80000000, torch.zeros_like(b), b) + return torch.where(b >= 0x80000000, _LOW32 - b, b | 0x80000000) + + +def make_keys(values: torch.Tensor, index: torch.Tensor) -> torch.Tensor: + """Exchange keys of (fp32 value, global token index) pairs: larger value wins, ties take the lower index.""" + return (ordered_bits(values) << 32) | (_LOW32 - index.to(torch.int64)) + + +def unsigned_max(keys: torch.Tensor, dim: int) -> torch.Tensor: + """Max of int64-stored uint64 keys (the kernel compares them unsigned).""" + sign = torch.iinfo(torch.int64).min + return (keys ^ sign).max(dim=dim).values ^ sign + + +def key_token(key: torch.Tensor) -> torch.Tensor: + return (_LOW32 - (key & _LOW32)).to(torch.int32) + + +def local_argmax(logits: torch.Tensor, vocab_offset: int) -> tuple[torch.Tensor, torch.Tensor]: + """(max value, lowest global index reaching it) per row of a [M, Vs] fp32 shard.""" + best = logits.max(dim=1).values + idx = (logits == best[:, None]).float().argmax(dim=1) # first (lowest) index at the max + return best, idx.to(torch.int64) + vocab_offset + + +def exchange_tokens( + own_keys: torch.Tensor, + ws_uc: torch.Tensor, + counters: torch.Tensor, + rank: int, + world: int, + slot_rows: int, +) -> torch.Tensor: + """Global tokens of the cross-rank pick: this rank's keys against the other ranks' keys already in its slots. + + Slot (half * slot_rows + t) * world + r of ``ws_uc`` holds rank r's key of row t; half = call counter & 1. A + single-device run stands in for the other ranks by writing their keys there before the call. + """ + m = own_keys.shape[0] + half = int(counters[1].item()) & 1 + slots = ( + ws_uc.view(torch.int64)[half * slot_rows * world : (half * slot_rows + m) * world] + .view(m, world) + .clone() + ) + slots[:, rank] = own_keys + return key_token(unsigned_max(slots, dim=1)) + + +def strict_accept( + tokens: torch.Tensor, + draft: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], +) -> dict[str, torch.Tensor]: + """The strict acceptance epilogue from the verify-row tokens [nreq * (D + 1)] (request-major). + + accepted[b] = tokens of request b's rows; num_accepted[b] = 1 + the matching prefix of draft[b] (overridden by + force_vec[b], else by force_total > 0); gather_ids[b] = batch_idx[b] * (D + 1) + num_accepted[b] - 1, and + next_new[b, 0] (when given) = the token at that row. Returns the outputs plus ``next_new`` as written. + """ + nreq, d = draft.shape + k1 = d + 1 + tokens = tokens.to(torch.int32) + accepted = tokens.view(nreq, k1).clone() + match = (draft.to(torch.int32) == accepted[:, :d]).to(torch.int32) + num = 1 + match.cumprod(dim=1).sum(dim=1) + if force_vec is not None: + num = force_vec[:nreq].to(num.dtype) + elif force_total > 0: + num = torch.full_like(num, force_total) + gather = batch_idx.to(torch.int64) * k1 + num.to(torch.int64) - 1 + out = {"accepted": accepted, "num_accepted": num.to(torch.int32), "gather_ids": gather} + if next_new is not None: + nn = next_new.clone() + nn[:, 0] = tokens[gather] + out["next_new"] = nn + return out + + +class M3ShardAccept(OpWrapper): + """Greedy verification of ``nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard ``[rows, Vs]`` + (global ids ``vocab_offset ..``). + + Each row's token is its global argmax over all ranks' shards: keys ``ordered(value) << 32 | ~index`` (ties take + the lowest index, as ``torch.argmax``) are max-reduced over the shard, pushed into every rank's exchange slot + through the multicast slots ``[2][48][world]`` (Lamport: 0 = empty) and max-reduced over the ranks by the last + CTA, which then runs the strict acceptance: ``acc_tokens [nreq, D + 1]``, ``num_acc [nreq]`` (``force_vec`` / + ``force_total`` override the counts), ``gather_ids [nreq]`` (the first draft pass's row per request) and, when + given, ``next_new[:, 0]`` (the last accepted token). + """ + + ARCHS = frozenset({Arch.SM_103}) + + # Every greedy verify step of the target (dispatch.SHARD_ACCEPT): draft_len 3 at 1..12 requests (4..48 rows), the + # target lm_head's BF16 shard logits (50016 columns per rank at TP4). The logits come from the head GEMM, so the + # op only reduces. + CELLS: tuple[Cell, ...] = ( + Cell( + why="cap 1: 4 verify rows (1 request x T = 4), rank 0 of 4, next_new staged", + spec=dict(nreq=1, draft_len=3, vs=50016, world=4, rank=0, force="none", strided=False), + ), + Cell( + why="cap 2: 8 verify rows, rank 2, global ids offset by 2 x 50016", + spec=dict(nreq=2, draft_len=3, vs=50016, world=4, rank=2, force="none", strided=False), + ), + Cell( + why="cap 4: 16 verify rows (4 requests x T = 4), rank 1 of 4, next_new staged", + spec=dict(nreq=4, draft_len=3, vs=50016, world=4, rank=1, force="none", strided=False), + ), + Cell( + why="cap 8: 32 verify rows, rank 3, global ids offset by 3 x 50016", + spec=dict(nreq=8, draft_len=3, vs=50016, world=4, rank=3, force="none", strided=False), + ), + Cell( + why="12 rows (3 requests) read from a row-strided view of a wider logits buffer, forced counts", + spec=dict(nreq=3, draft_len=3, vs=50016, world=4, rank=0, force="vec", strided=True), + ), + Cell( + why="the 48-row maximum (12 requests at draft 3), force_total, rows with exact cross-rank ties", + spec=dict( + nreq=12, + draft_len=3, + vs=50016, + world=4, + rank=2, + force="total", + strided=False, + ties=True, + ), + ), + # Every other request count of the verify range, on the 4-rank 50016-column shards. + *( + Cell( + why=f"{n} requests ({4 * n} verify rows): a request count between the ones above", + spec=dict( + nreq=n, draft_len=3, vs=50016, world=4, rank=n % 4, force="none", strided=False + ), + ) + for n in (5, 6, 7, 9, 10, 11) + ), + ) + + note = """ + The op reduces BF16 logits exactly, so the reference is bit-exact, including cross-rank ties (lowest global + index wins). The cross-rank exchange is certified on one device: the other ranks' keys are written into this + rank's slots before the call, which is what their multicast pushes would have put there; the multicast mapping + itself (ws_mc_ptr = this rank's slots, as at world 1) is not exercised across devices here. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + torch.ops.trtllm.minimax_m3_shard_accept( + logits, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + num_ctas, + force_vec, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> dict[str, torch.Tensor]: + """Lowest-index maximum of each row's BF16 logits, the largest key among it and the peers' keys in + ``ws_uc``, then the strict acceptance. Reads ``ws_uc`` / ``counters`` before the call; the scratch and + output buffers and ``num_ctas`` do not change the result.""" + best, idx = local_argmax(logits.float(), vocab_offset) + tokens = exchange_tokens(make_keys(best, idx), ws_uc, counters, rank, world, SHARD_ROWS) + out = strict_accept(tokens, draft, batch_idx, force_total, next_new, force_vec) + out["tokens"] = tokens + return out + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + assert not bool(keys[: logits.shape[0]].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + assert ws_uc.numel() >= 2 * SHARD_ROWS * world, "ws_uc must hold [2][48][world] slots" + # A NaN logit maps to a key above every finite one and wins the row silently. + assert not bool(torch.isnan(logits).any()), "NaN logits win the argmax" + nreq = draft.shape[0] + assert bool(((batch_idx[:nreq] >= 0) & (batch_idx[:nreq] < nreq)).all()), ( + "batch_idx must index the nreq requests" + ) + if force_vec is not None: + fv = force_vec[:nreq] + assert bool(((fv >= 1) & (fv <= draft.shape[1] + 1)).all()), ( + "force_vec counts must lie in [1, D + 1]" + ) + + +m3_shard_accept = M3ShardAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_slot_scatter.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_slot_scatter.py new file mode 100644 index 000000000000..470a6372f2b6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_slot_scatter.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The spec sampler's per-slot stores of one step's outputs, in one launch (``M3SpecStepFold.scatter`` -> +``m3_spec_tail.slot_scatter``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +def padded_rows(src: torch.Tensor, width: int) -> torch.Tensor: + """src rows zero-padded (or cut) to ``width`` columns.""" + out = torch.zeros(src.shape[0], width, dtype=src.dtype, device=src.device) + w = min(width, src.shape[1]) + out[:, :w] = src[:, :w] + return out + + +class M3SlotScatter(OpWrapper): + """For i < len(slots), all int32 2-D strided views: ``da[slots[i]] = sa[i]``, ``db[slots[i]] = sb[i]``, + ``dd[slots[i]] = sd[i]`` (each zero-padded / cut to the destination width) and ``dl[slots[i]] = sl[i]``. The + destinations are the store's transposed [slots, width] views.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The sampler store at draft_len 3 (8 slots): new tokens / next new tokens [slots, 4] (transposed views of + # [4, slots, 1]), next draft tokens [slots, 3]; a worker output narrower than the store (dynamic draft length) + # is zero-padded. + CELLS: tuple[Cell, ...] = ( + Cell( + why="8 rows into shuffled slots, widths equal to the store's", + spec=dict(n=8, slots=8, src_w=4, src_d=3, store_w=4, store_d=3), + ), + Cell( + why="3 rows of a narrower step (runtime draft 2): zero padding to the store width", + spec=dict(n=3, slots=8, src_w=3, src_d=2, store_w=4, store_d=3), + ), + ) + + note = """ + Pure index copies: the reference is exact. Slots are not bounds-checked; is_valid states it. + """ + + def __call__(self, slots, sa, da, sb, db, sl, dl, sd, dd): + torch.ops.trtllm.minimax_m3_slot_scatter(slots, sa, da, sb, db, sl, dl, sd, dd) + + def reference(self, slots, sa, da, sb, db, sl, dl, sd, dd): + """The four destinations after the call, on copies.""" + s = slots.long() + out = {} + for name, src, dst in (("da", sa, da), ("db", sb, db), ("dd", sd, dd)): + r = dst.clone() + r[s] = padded_rows(src[: s.numel()], dst.shape[1]) + out[name] = r + lr = dl.clone() + lr[s] = sl[: s.numel()] + out["dl"] = lr + return out + + def is_valid(self, slots, sa, da, sb, db, sl, dl, sd, dd): + assert bool(((slots >= 0) & (slots < da.shape[0]) & (slots < dl.numel())).all()), ( + "slots must index the store" + ) + # Two rows into one slot race (which one lands is unordered). + assert slots.unique().numel() == slots.numel(), "slots must be distinct" + assert sa.shape[0] >= slots.numel() and sl.numel() >= slots.numel(), "a source row per slot" + + +m3_slot_scatter = M3SlotScatter() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_step_gather.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_step_gather.py new file mode 100644 index 000000000000..abea2cf620e9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/spec/m3_step_gather.py @@ -0,0 +1,152 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The overlap scheduler's per-step input gathers from the previous step's device outputs, in one launch +(``M3SpecStepFold.gather`` -> ``m3_spec_tail.step_gather``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3StepGather(OpWrapper): + """For the ``P`` previous requests (sequence slots ``slots``, ``T`` tokens and ``D`` draft tokens each), all + int32:: + + ids[ids0 + p*T + t] = new_tok[t, slots[p]] (new_tok [T', slots, (1)]) + draft[d0 + p*D + d] = next_draft[slots[p], d] + pos_off[i] = lens[ppi[i - b0*T]] for i in [b0*T, (b0+P)*T), else 0 (every i < pos_off.numel()) + kv_off[i] = lens[slots[i - b0]] - T for i in [b0, b0+P), else 0 (every i < kv_off.numel()) + + Every other element of ``ids`` / ``draft`` is left as it was. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # draft_len 3: T = 4, D = 3; the store's [T, slots, 1] new tokens and [slots, D] next drafts (8 slots, the + # max batch); the engine's flat buffers sized for the max batch. + CELLS: tuple[Cell, ...] = ( + Cell( + why="an unchanged batch of 8 previous requests in shuffled slots, nothing before them (b0 = 0)", + spec=dict(p=8, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0), + ), + Cell( + why="2 previous requests after 3 new ones: offsets ids0 / d0 / b0 nonzero, zero-filled offsets outside", + spec=dict(p=2, t=4, d=3, slots=8, b0=3, ids0=12, d0=9, extra_rows=3), + ), + Cell( + why="one request, the store's 2-D view of new tokens (beam dim squeezed)", + spec=dict(p=1, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0, two_d=True), + ), + ) + + note = """ + Pure index copies: the reference is exact. The op does not bound slots, ppi or the destination offsets; is_valid + states them. + """ + + def __call__( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + torch.ops.trtllm.minimax_m3_step_gather( + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ) + + def reference( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + """The four destinations after the call, on copies (``ids`` / ``draft`` keep their other elements).""" + nt = new_tok.reshape(new_tok.shape[0], new_tok.shape[1]) + s = slots[:P].long() + ids_r, draft_r = ids.clone(), draft.clone() + ids_r[ids0 : ids0 + P * T] = nt[:T, s].t().reshape(-1) + draft_r[d0 : d0 + P * D] = next_draft[s, :D].reshape(-1) + pos_r = torch.zeros_like(pos_off) + pos_r[b0 * T : (b0 + P) * T] = lens[ppi[: P * T].long()] + kv_r = torch.zeros_like(kv_off) + kv_r[b0 : b0 + P] = lens[s] - T + return {"ids": ids_r, "draft": draft_r, "pos_off": pos_r, "kv_off": kv_r} + + def is_valid( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + # Reads through slots / ppi and writes at the offsets are unchecked: any of them out of range reads or + # writes another buffer. + s = slots[:P] + assert bool( + ( + (s >= 0) & (s < next_draft.shape[0]) & (s < new_tok.shape[1]) & (s < lens.numel()) + ).all() + ), "slots must index the store" + assert bool(((ppi[: P * T] >= 0) & (ppi[: P * T] < lens.numel())).all()), ( + "ppi must index lens" + ) + assert new_tok.shape[0] >= T and next_draft.shape[1] >= D, "store narrower than T / D" + assert ids.numel() >= ids0 + P * T and draft.numel() >= d0 + P * D, "destinations too short" + assert pos_off.numel() >= (b0 + P) * T and kv_off.numel() >= b0 + P, ( + "offset buffers too short" + ) + + +m3_step_gather = M3StepGather() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/triton_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/triton_sparse_decode.py new file mode 100644 index 000000000000..064a3c0b2b10 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/triton_sparse_decode.py @@ -0,0 +1,1651 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. + +This is the MiniMax-M3 NVFP4 / sm_103 / tp4 target's copy. On top of the +in-tree decode it routes long-context TP4 steps to the one-launch +``m3_attn_lc`` kernel, short-sequence verify steps of at least +``MQ_MIN_TOKENS`` rows to the multi-query producer, releases PDL dependents +early, and has the merge emit the MXFP8 o_proj hand-off (``m3_prequant.SLOT``). +""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers + +from . import m3_attn_lc, m3_prequant + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 # capacity: fixed: the MSA block size of the checkpoint + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 # capacity: fixed: a launch-grid target, not a size + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +# Fewest query rows for which the merge writes the MXFP8 o_proj hand-off: every decode step, so the MXFP8 o_proj +# never runs its own quantize kernel after attention. +MERGE_MXFP8_MIN_TOKENS = 1 # capacity: fixed: a row lower bound + + +class MergeMxfp8: + """MXFP8 copy of the split-K merge output for the o_proj GEMM. + + From ``MERGE_MXFP8_MIN_TOKENS`` to ``MAX_ROWS`` rows the merge kernel writes the E4M3 rows and swizzled UE8M0 + scales next to the BF16 output and publishes them to ``m3_prequant.SLOT``, so the MXFP8 o_proj skips its own + quantization. One 128-row buffer per width serves every layer: each o_proj consumes it before the next layer's + merge rewrites it, in stream order. + """ + + MAX_ROWS = 128 # capacity: fixed: the merge hand-off's rows (decode steps) + + def __init__(self): + self._buffers = {} + + def buffers(self, width: int, device: torch.device): + key = (width, device) + if key not in self._buffers: + self._buffers[key] = ( + torch.empty((self.MAX_ROWS, width), dtype=torch.float8_e4m3fn, device=device), + torch.zeros((self.MAX_ROWS * (width // 32),), dtype=torch.uint8, device=device), + ) + return self._buffers[key] + + +MERGE_MXFP8 = MergeMxfp8() + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +# Mixed ctx+gen steps launch on [num_ctx:] slices (unaligned seq_lens / block_table) with arbitrary +# total_q; specializing on those compiled a new variant mid-serving (~650 ms TTFT stall at c8). +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + out_ptr, # final [total_q, num_heads, head_dim] for FUSED_MERGE, else unused + counter_ptr, # int32 [total_q, num_kv_heads] arrival counters for FUSED_MERGE, zero at rest + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + stride_out_n, + stride_out_h, + stride_out_d, + num_kv_heads, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge + FUSED_MERGE: tl.constexpr, # the last chunk to finish merges the partials in place + EARLY_TRIGGER: tl.constexpr, # release PDL dependents (weight prefetch) at entry +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + # Dependents still wait for this grid to complete; triggering here only + # lets them start their input-independent prologue (e.g. weight loads). + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for it in tl.range(chunk_start, chunk_end): + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + if FUSED_MERGE: + # Split-K serial reduction: every chunk publishes its partial, and the + # chunk that arrives last for this (token, KV head) merges all of them + # exactly as _merge_topk_attn_out_kernel does, then re-arms the counter. + tl.debug_barrier() + counter = counter_ptr + pid_b * num_kv_heads + pid_kh + arrived = tl.atomic_add(counter, 1, sem="acq_rel", scope="gpu") + if arrived == NUM_TOPK_CHUNKS - 1: + tl.debug_barrier() + o_rows = ( + o_ptr + + pid_b * stride_o_b + + (pid_h + off_h[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d + ) + l_rows = lse_ptr + pid_b * stride_l_b + (pid_h + off_h) * stride_l_h + lse_max = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + lse_max = tl.maximum(lse_max, lse_c) + # All chunks empty (a CUDA-graph padding row): zero, never NaN. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + denom = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + denom += tl.exp2(lse_c - lse_max) + denom = tl.where(denom > 0, denom, 1.0) + merged = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + w_c = tl.exp2(lse_c - lse_max) / denom + o_c = tl.load( + o_rows + c * stride_o_c, mask=hd_mask, other=0.0, cache_modifier=".cg" + ).to(tl.float32) + merged += o_c * w_c[:, None] + tl.store( + out_ptr + + pid_b * stride_out_n + + (pid_h + off_h[:, None]) * stride_out_h + + off_d[None, :] * stride_out_d, + merged.to(out_ptr.dtype.element_ty), + mask=hd_mask, + ) + tl.atomic_xchg(counter, 0, sem="relaxed", scope="gpu") + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for _ in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.jit +def _rcp_approx_ftz(x): + return tl.inline_asm_elementwise( + "rcp.approx.ftz.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _store_mxfp8_head(x, q8_ptr, sf_ptr, row, head, num_sf_k_tiles, HEAD_DIM: tl.constexpr): + """Quantize one row's [head * HEAD_DIM, (head + 1) * HEAD_DIM) slice to MXFP8 as + torch.ops.trtllm.mxfp8_quantize(x, True) does: UE8M0 scale = round-up(amax * rcp(448)), E4M3 RN-satfinite, + scales in the 128x4-swizzled layout.""" + NUM_SF: tl.constexpr = HEAD_DIM // 32 + xr = tl.reshape(x.to(tl.float32), (NUM_SF, 32)) + amax = tl.max(tl.abs(xr), axis=1) + sfv = amax * _rcp_approx_ftz(tl.full((NUM_SF,), 448.0, tl.float32)) + bits = sfv.to(tl.int32, bitcast=True) + e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(tl.int32) + e = tl.minimum(e, 254) + scale = ((254 - e) << 23).to(tl.float32, bitcast=True) + scale = tl.where(amax != 0, scale, 0.0) + q = (xr * scale[:, None]).to(tl.float8e4nv) + off_q = tl.reshape(tl.arange(0, HEAD_DIM), (NUM_SF, 32)) + tl.store(q8_ptr + row * (num_sf_k_tiles * 128) + head * HEAD_DIM + off_q, q) + k_sf = head * NUM_SF + tl.arange(0, NUM_SF) + off_sf = ( + (row // 128) * num_sf_k_tiles * 512 + + (k_sf // 4) * 512 + + (row % 32) * 16 + + ((row % 128) // 32) * 4 + + k_sf % 4 + ) + tl.store(sf_ptr + off_sf, e.to(tl.uint8)) + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr = False, + q8_ptr=None, # MXFP8 copy of out: [total_q, num_heads * head_dim] E4M3 + sf_ptr=None, # its UE8M0 block scales, 128x4-swizzled + num_sf_k_tiles=0, # ceil(num_heads * head_dim / 128) + MXFP8_OUT: tl.constexpr = False, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + if EARLY_TRIGGER: + # Release the dependent (the o_proj GEMM) before waiting on the attention producer, so its weight + # prefetch overlaps the attention kernel as well as this merge; its griddepcontrol.wait still waits + # for this grid (and therefore the producer) to complete. + tl.extra.cuda.gdc_launch_dependents() + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + if MXFP8_OUT: + _store_mxfp8_head( + o_merged.to(out_ptr.dtype.element_ty), + q8_ptr, + sf_ptr, + pid_b, + pid_h, + num_sf_k_tiles, + BLOCK_SIZE_D, + ) + + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on both datacenter Blackwell targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3)) +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +# Multi-query producer (triton_sparse_decode_mq) for identity-selection verify steps with at least this many query +# tokens; below it the per-token producer is as fast. +MQ_MIN_TOKENS = 17 # capacity: fixed: the multi-query producer's crossover +# Warps per multi-query producer program. +MQ_NUM_WARPS = 8 + + +def _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, +) -> None: + """minimax_m3_sparse_attn_decode through m3_attn_lc: same table, output and MXFP8 hand-off contract.""" + use_pdl = get_env_enable_pdl() + total_q, num_heads, head_dim = q.shape + linear = _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=1, + gqa_group_size=num_heads, + max_topk=int(topk_idx.shape[-1]), + decode_query_len=decode_query_len, + ) + mxfp8 = None + m3_prequant.SLOT.clear() + if MERGE_MXFP8_MIN_TOKENS <= total_q <= MergeMxfp8.MAX_ROWS: + mxfp8 = MERGE_MXFP8.buffers(num_heads * head_dim, output.device) + # The early trigger lets the o_proj launch and stream its weights while attention runs. + m3_attn_lc.m3_lc_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + linear_softmax=linear, + mxfp8=mxfp8, + use_pdl=use_pdl, + early_trigger=True, + ) + if mxfp8 is not None: + m3_prequant.SLOT.publish( + output.view(total_q, num_heads * head_dim), mxfp8[0][:total_q], mxfp8[1] + ) + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + if kv_nvfp4: + capability = torch.cuda.get_device_capability(q.device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + # Long-context TP4 steps: attention, split-K merge and MXFP8 hand-off in one launch. + if ( + kv_nvfp4 + and widen_q + and not identity_blocks + and num_topk_chunks is None + and m3_attn_lc.lc_applies( + q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output + ) + ): + _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + ) + return + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else 1 + ) + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + if kv_nvfp4 and total_q * num_kv_heads <= 64: + # GB300 TP4 Eagle verify (1 KV head, group 16, 6 query tokens): one block per program is fastest up to + # batch 8 (lowm5/bench_decode_chunks.py: b8 11.7 vs 14.8 us with the default split of 4). + num_topk_chunks = max_topk + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + # Every query token of a request reads the same blocks under identity selection (short-sequence mode): one + # program per (request, block) covers all of them and dequantizes each block once. + use_mq = ( + identity_blocks + and total_q >= MQ_MIN_TOKENS + and kv_nvfp4 + and num_kv_heads == 1 + and decode_query_len > 1 + and head_dim == 128 + and gqa_group_size * decode_query_len <= 128 + ) + if use_mq: + num_topk_chunks = max_topk + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = get_env_enable_pdl() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + single_kernel_args = ( + { + # No split-K arrival counters: the separate merge kernel combines the chunks. + "merge": (output, None), + "tail": (output.stride(0), output.stride(1), output.stride(2), num_kv_heads), + # An early release pays off when the dependent is the consumer of the + # final output (it can prefetch), or a merge grid that itself releases + # its dependent at entry: the o_proj then launches during this kernel and + # streams its weights while attention runs. Above 8 query rows the + # concurrent weight stream slows the larger attention grid more than the + # prefetch saves (GB300 TP4: -1.1% at 48 rows). + "options": {"FUSED_MERGE": False, "EARLY_TRIGGER": direct_output or total_q <= 8}, + } + if query_group_size == 1 + else {"merge": (), "tail": (), "options": {}} + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + if use_mq: + from .triton_sparse_decode_mq import launch_mq_producer + + launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + sm_scale=sm_scale, + decode_query_len=decode_query_len, + use_pdl=use_pdl, + early_trigger=False, + num_warps=MQ_NUM_WARPS, + ) + else: + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + *single_kernel_args["merge"], + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + *single_kernel_args["tail"], + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + **single_kernel_args["options"], + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + m3_prequant.SLOT.clear() + if not direct_output: + width = num_heads * head_dim + mxfp8 = ( + MERGE_MXFP8.buffers(width, output.device) + if MERGE_MXFP8_MIN_TOKENS <= total_q <= MergeMxfp8.MAX_ROWS + and head_dim == 128 + and output.dim() == 3 + and output.is_contiguous() + and output.dtype == torch.bfloat16 + else None + ) + mxfp8_args = ( + { + "q8_ptr": mxfp8[0], + "sf_ptr": mxfp8[1], + "num_sf_k_tiles": width // 128, + "MXFP8_OUT": True, + } + if mxfp8 is not None + else {} + ) + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + # Release the o_proj at the merge's entry so it streams its weights during the merge. + EARLY_TRIGGER=True, + **mxfp8_args, + **pdl_launch, + **merge_launch_options, + ) + if mxfp8 is not None: + m3_prequant.SLOT.publish(output.view(total_q, width), mxfp8[0][:total_q], mxfp8[1]) + + +__all__ = [ + "MERGE_MXFP8", + "MERGE_MXFP8_MIN_TOKENS", + "MergeMxfp8", + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/triton_sparse_decode_mq.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/triton_sparse_decode_mq.py new file mode 100644 index 000000000000..048d31b82d68 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/ops/triton_sparse_decode_mq.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multi-query block-sparse decode producer for MiniMax-M3 short-sequence verify steps. + +In short-sequence mode every request attends at most ``topk`` KV blocks and block selection is the identity, so +all ``decode_query_len`` query tokens of a request read the same blocks. The per-token producer +(``_gqa_sparse_decode_kernel``) then dequantizes every block once per query token. This producer runs one program +per (request, block) for all of the request's query tokens: rows are (token, head) pairs, each block is +dequantized once, and the causal limit is a per-row mask. + +It writes the same per-block partials (output scaled by 1/l and the V global scale, log2 LSE) as the per-token +producer with one block per chunk, so ``_merge_topk_attn_out_kernel`` merges them unchanged. +""" + +import triton +import triton.language as tl + +from .triton_sparse_decode import _dequant_nvfp4_rows + + +@triton.jit( + do_not_specialize=["num_reqs"], do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"] +) +def _gqa_sparse_decode_mq_kernel( + q_ptr, # [num_reqs * QLEN, num_heads, head_dim], FP8 or FP16/BF16 + k_ptr, # packed NVFP4 K: [num_pages, 1, page_size, head_dim / 2] uint8 + v_ptr, # packed NVFP4 V: same layout + k_sc_ptr, # K block scales [num_pages, 1, page_size, head_dim / 16], token-linear + v_sc_ptr, # V block scales, 4x4 token-quad order + k_gs_ptr, + v_gs_ptr, + t_ptr, # topk table [1, total_q, topk]; the request's last token's row is read + o_ptr, # partials [topk, total_q, num_heads, head_dim] fp32 + lse_ptr, # partials [topk, total_q, num_heads] fp32 (log2) + block_table_ptr, + seq_lens, + num_reqs, + sm_scale, + stride_qn, + stride_qh, + stride_k_blk, + stride_k_pos, + stride_v_blk, + stride_v_pos, + stride_ksc_blk, + stride_vsc_blk, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + QLEN: tl.constexpr, + GROUP: tl.constexpr, # query heads per KV head (16) + BLOCK_R: tl.constexpr, # next power of two of QLEN * GROUP + HEAD_DIM: tl.constexpr, + BLOCK_SIZE_K: tl.constexpr, + SCALE_COLS: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr, +): + pid_r = tl.program_id(0) + pid_c = tl.program_id(1) # top-k position == block index under identity selection + sm_scale_log2e = sm_scale * 1.4426950409 * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + + off_r = tl.arange(0, BLOCK_R) + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, HEAD_DIM) + off_dp = tl.arange(0, HEAD_DIM // 2) + off_s = tl.arange(0, SCALE_COLS) + row_t = off_r // GROUP + row_h = off_r % GROUP + r_mask = off_r < QLEN * GROUP + tok = pid_r * QLEN + row_t + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + pid_r) + # Token t of the request sits at seq_len - QLEN + t and attends positions [0, kv_len_t). + kv_len = tl.maximum(seq_len - QLEN + row_t + 1, 0) + blk = tl.load(t_ptr + (pid_r * QLEN + QLEN - 1) * stride_tn + pid_c * stride_tk) + blk_ok = blk >= 0 + blk = tl.where(blk_ok, blk, 0) + page = tl.load(block_table_ptr + pid_r * stride_bt_b + blk, mask=blk_ok, other=0).to(tl.int64) + # Rows past the last token's position are never read: their cache bytes may be stale (0 * NaN in the PV dot). + n_valid = tl.where( + blk_ok, tl.minimum(tl.maximum(seq_len - blk * BLOCK_SIZE_K, 0), BLOCK_SIZE_K), 0 + ) + n_mask = off_n < n_valid + + q = tl.load( + q_ptr + tok[:, None] * stride_qn + row_h[:, None] * stride_qh + off_d[None, :], + mask=r_mask[:, None], + other=0.0, + ).to(tl.float16) + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + page * stride_k_blk + off_n[:, None] * stride_k_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + page * stride_ksc_blk + off_n[:, None] * SCALE_COLS + off_s[None, :], + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ) + valid = ( + (blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None]) & n_mask[None, :] & r_mask[:, None] + ) + qk = tl.dot(q, tl.trans(k_deq).to(tl.float16)) * sm_scale_log2e + qk = tl.where(valid, qk, float("-inf")) + m = tl.max(qk, axis=1) + m_safe = tl.where(m == float("-inf"), 0.0, m) + p = tl.exp2(qk - m_safe[:, None]) + p_sum = tl.sum(p, axis=1) + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + page * stride_v_blk + off_n[:, None] * stride_v_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ).to(tl.float16) + acc = tl.dot(p.to(tl.float16), v) + # An empty partial stores zero with LSE -inf, so the merge gives it weight 0. + scale = tl.where(p_sum > 0.0, 1.0 / p_sum, 0.0) * v_global_scale + lse = tl.where(p_sum > 0.0, m + tl.log2(p_sum), float("-inf")) + acc = acc * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + tok[:, None] * stride_o_b + + row_h[:, None] * stride_o_h + + off_d[None, :], + acc, + mask=r_mask[:, None], + ) + tl.store(lse_ptr + pid_c * stride_l_c + tok * stride_l_b + row_h * stride_l_h, lse, mask=r_mask) + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + *, + sm_scale, + decode_query_len, + use_pdl, + early_trigger, + num_warps=8, + num_stages=1, +): + """Launch the multi-query producer; arguments as minimax_m3_sparse_attn_decode after its NVFP4 views.""" + total_q, num_heads, head_dim = q.shape + num_reqs = total_q // decode_query_len + max_topk = int(topk_idx.shape[-1]) + rows = decode_query_len * num_heads + pdl_launch = {"launch_pdl": True} if use_pdl else {} + _gqa_sparse_decode_mq_kernel[(num_reqs, max_topk)]( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + num_reqs, + sm_scale, + q.stride(0), + q.stride(1), + k_paged.stride(0), + k_paged.stride(2), + v_paged.stride(0), + v_paged.stride(2), + k_block_scale.stride(0), + v_block_scale.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + o_arg.stride(0), + o_arg.stride(1), + o_arg.stride(2), + lse_arg.stride(0), + lse_arg.stride(1), + lse_arg.stride(2), + block_table.stride(0), + QLEN=decode_query_len, + GROUP=num_heads, + BLOCK_R=triton.next_power_of_2(rows), + HEAD_DIM=head_dim, + BLOCK_SIZE_K=int(k_paged.shape[2]), + SCALE_COLS=head_dim // 16, + USE_PDL=use_pdl, + EARLY_TRIGGER=early_trigger, + num_warps=num_warps, + num_stages=num_stages, + **pdl_launch, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/weights.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/weights.py new file mode 100644 index 000000000000..7eaac1fb8d31 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/weights.py @@ -0,0 +1,910 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight tables and loaders: MiniMax-M3.1 (MXFP8 + packed NVFP4 experts) / sm_103 / tp4. + +Two tables, one per checkpoint the target reads: + +* ``MODEL_WEIGHTS`` (``MiniMaxM3Tp4Weights``): the vision-language checkpoint's language model, + ``language_model.*``. Every rank is handed the whole checkpoint, so every tensor-parallel split is this + table's: the Q heads, the K/V/index-Q heads (one each per rank), the dense MLPs and the shared expert's + intermediate, the o_proj / down-projection input columns, every routed expert's intermediate (all 128 + experts live on every rank), and the vocabulary of the shell's ``lm_head``. The index-K head is replicated. +* ``DRAFT_WEIGHTS`` (``MiniMaxM3Tp4DraftWeights``): the one-layer BF16 Eagle3 drafter's own checkpoint + (``speculative_model``), loaded through the shell's ``load_draft_weights`` when speculation is on. + +Storage is HF ``[out, in]`` row-major. Five families need a transform, each a layout a kernel demands: + +* **MXFP8 block scales** (every attention projection, the dense MLPs and the shared expert): the checkpoint + stores UE8M0 bytes ``[N, K / 32]``; the GEMMs read the 128x4-swizzled order + (``trtllm::block_scale_interleave``). A fused projection concatenates its shards' scales first and swizzles + once. +* **The five-way sparse-layer projection** ``[Q | K | V | index-Q | index-K]`` and the three-way dense one + ``[Q | K | V]``: the per-rank shards concatenated along N. +* **Gated MLPs** (dense and shared expert): ``[gate ; up]`` rows. +* **NVFP4 routed experts** (layers 3..59): ``[up (w3) ; gate (w1)]`` rows, then the trtllm-gen row shuffle of + the weight and of its E4M3 scales (gated-activation reorder + 128-row epilogue tile shuffle, + ``trtllmgen_maybe_get_cached_*_permute_indices``) and the 128x4 swizzle of the scales. +* **Gemma boundary norms**: ``input_layernorm``, ``post_attention_layernorm`` and ``model.norm`` are stored as + Gemma norms, ``(1 + w) * x``; the fused all-reduce + RMSNorm kernels apply ``w * x``, so ``1`` is added + (in BF16) at load. The per-head Q/K/index norms stay Gemma: their kernels apply the offset themselves. + +The routed experts' per-tensor scalars (input scales and second-level weight scales, every expert) load raw; +``modeling`` derives the kernels' global scales, alphas and activation constants from them once the weights +are real. + +Loading contract: ``load(model, weights)`` consumes the engine-provided checkpoint dict, fills every declared +parameter exactly once and asserts coverage in both directions: every declared parameter written, and every +checkpoint key either consumed or in ``expected_unconsumed`` (the vision tower and its projector, which this +text-only target does not build, and the language model's multi-token-prediction head). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn + +from tensorrt_llm._torch.models.checkpoints.hf.minimaxm3_weight_mapper import ( + reciprocal_global_scale, +) +from tensorrt_llm._torch.moe.fused_moe.quantization import ( + trtllmgen_maybe_get_cached_w2_permute_indices, + trtllmgen_maybe_get_cached_w3_w1_permute_indices, +) + +#: ``layers=ALL_LAYERS`` is the common case: a weight every decoder layer carries. +ALL_LAYERS = object() + +_LM = "language_model." +#: Checkpoint key families this target does not build: the vision tower and its two projectors (text only), and the +#: language model's multi-token-prediction head (speculation, when on, is the separate Eagle3 drafter). +_UNBUILT_PREFIXES = ( + "vision_tower.", + "multi_modal_projector.", + "patch_merge_mlp.", + _LM + "model.mtp.", +) + +MX_BLOCK = 32 # MXFP8: one UE8M0 scale per 32 elements along K +NV_BLOCK = 16 # NVFP4: one E4M3 scale per 16 elements along K +EPILOGUE_TILE_M = ( + 128 # the trtllm-gen MoE runner's row tile: fixes its weight and scale row shuffles +) + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + ``shape`` takes the dimension bundle and returns the declared shape. ``src`` is a callable + ``(d, i) -> [(key or key tuple, index into the param | None), ...]`` with keys relative to the checkpoint + root; a key tuple hands the transform those tensors together. ``layers`` is ``ALL_LAYERS``, ``None`` (not + per-layer) or ``d -> layer indices``. ``transform(src, d)`` runs on the destination's device before the copy. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + +# ------------------------------------------------------------------------------------------------ shards + + +def fused_projection(sparse_config) -> bool: + """Whether the sparse layers run the five-way packed QKV + index projection. + + On by default: this target requires the MSA implementation and an FP8 indexer, which the packed projection needs. + An explicit ``fuse_qkv_index_projection`` in the deployment's sparse attention config wins, so setting it to False + is the kill switch (the QKV projection plus a separate MXFP8 index projection). + """ + if sparse_config is None: + return False + # A config that does not record which fields were set (a plain namespace or dataclass) is taken at its word. + fields_set = getattr(sparse_config, "model_fields_set", None) + if fields_set is None or "fuse_qkv_index_projection" in fields_set: + return bool(getattr(sparse_config, "fuse_qkv_index_projection", False)) + return True + + +def _col( + t: torch.Tensor, d: SimpleNamespace, tp: int | None = None, rank: int | None = None +) -> torch.Tensor: + """This rank's rows (column-parallel output split), even split, sliced on the host and moved to the + destination device (only this rank's bytes are read).""" + tp = d.tp if tp is None else tp + rank = d.rank if rank is None else rank + n = t.shape[0] // tp + return _dev(t[rank * n : (rank + 1) * n], d) + + +def _row(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """This rank's input columns (row-parallel input split), even split, on the destination device.""" + n = t.shape[1] // d.tp + return _dev(t[:, d.rank * n : (d.rank + 1) * n], d) + + +def _dev(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """``t`` on the destination device (``d.dev``, set per fill).""" + return t.to(d.dev, non_blocking=True) + + +def _swizzle(scale: torch.Tensor) -> torch.Tensor: + """128x4 swizzle of a row-major ``[N, K / 32]`` UE8M0 scale matrix, flat.""" + return torch.ops.trtllm.block_scale_interleave(scale.contiguous()) + + +# --------------------------------------------------------------------------------------------- transforms + + +def _gemma_fold(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A Gemma boundary norm as the plain norm the fused all-reduce epilogue applies: ``w + 1`` in BF16.""" + return t + 1.0 + + +def _f32(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return t.to(torch.float32) + + +def _divisor(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A stored global scale (multiplier) -> the modelopt divisor: its FP32 reciprocal.""" + return reciprocal_global_scale("expert global scale", t) + + +def _qkv_sparse_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[Q | K | V | index-Q | index-K]``: Q by attention heads, K/V/index-Q by KV heads, index-K replicated.""" + q, k, v, iq, ik = t + return torch.cat( + [_col(q, d), _col(k, d), _col(v, d), _col(iq, d), _dev(ik, d)], dim=0 + ).contiguous() + + +def _qkv_sparse_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v, iq, ik = t + return _swizzle( + torch.cat([_col(q, d), _col(k, d), _col(v, d), _col(iq, d), _dev(ik, d)], dim=0) + ) + + +def _qkv_dense_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + return torch.cat([_col(q, d), _col(k, d), _col(v, d)], dim=0).contiguous() + + +def _qkv_dense_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + return _swizzle(torch.cat([_col(q, d), _col(k, d), _col(v, d)], dim=0)) + + +def _index_qk_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[index-Q | index-K]`` as the checkpoint's MXFP8 (E4M3) values: this rank's index-Q heads and the replicated + index-K, the same rows the five-way projection packs after Q / K / V.""" + iq, ik = t + return torch.cat([_col(iq, d), _dev(ik, d)], dim=0).contiguous() + + +def _index_qk_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """The UE8M0 block scales of ``_index_qk_w``, 128x4-swizzled as every MXFP8 projection's.""" + iq, ik = t + return _swizzle(torch.cat([_col(iq, d), _dev(ik, d)], dim=0)) + + +def _gate_up_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return torch.cat([_col(gate, d), _col(up, d)], dim=0).contiguous() + + +def _gate_up_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return _swizzle(torch.cat([_col(gate, d), _col(up, d)], dim=0)) + + +def _row_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _row(t, d).contiguous() + + +def _row_s(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _swizzle(_row(t, d)) + + +def _col_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _col(t, d).contiguous() + + +#: Permute indices of the trtllm-gen expert relayouts, keyed by (shape, family, scale group): one expert's are +#: every expert's, so each is computed once per load. +_PERMUTE_CACHE: dict = {} + + +def _expert_w31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 operand: ``[up ; gate]`` packed-FP4 rows of this rank, then the gated-activation row + reorder and the epilogue-tile shuffle.""" + up, gate = t + w = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + w, _PERMUTE_CACHE, EPILOGUE_TILE_M, is_gated_act_gemm=True + ) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 E4M3 scales: ``[up ; gate]`` rows of this rank, the same row reorder at scale + granularity, then the 128x4 swizzle.""" + up, gate = t + s = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK, is_gated_act_gemm=True + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled.view(torch.uint8).reshape(shape)) + .view(torch.float8_e4m3fn) + .reshape(shape) + ) + + +def _expert_w2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 operand: this rank's input columns, then the epilogue-tile row shuffle.""" + w = _row(t, d).contiguous() + perm = trtllmgen_maybe_get_cached_w2_permute_indices(w, _PERMUTE_CACHE, EPILOGUE_TILE_M) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 E4M3 scales: this rank's columns, the row shuffle at scale granularity, the swizzle.""" + s = _row(t, d).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w2_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled).view(torch.float8_e4m3fn).reshape(shape) + ) + + +# ------------------------------------------------------------------------------------------------ sources + + +def _one(template: str): + return lambda d, i, t=template: [(_LM + t.format(i=i), None)] + + +def _many(*templates: str): + return lambda d, i, ts=templates: [(tuple(_LM + t.format(i=i) for t in ts), None)] + + +def _per_expert(*suffixes: str): + """One source per routed expert, written at its expert index (all experts are local at tp4).""" + + def build(d, i, sx=suffixes): + out = [] + for e in range(d.num_experts): + keys = tuple(_LM + f"model.layers.{i}.block_sparse_moe.experts.{e}.{s}" for s in sx) + out.append((keys if len(keys) > 1 else keys[0], (e,))) + return out + + return build + + +def _dense_mlp_layers(d): + """Layers with the dense SwiGLU-OAI MLP (``moe_layer_freq`` 0).""" + return d.dense_mlp + + +def _moe_layers(d): + return d.moe + + +def _sparse_attn_layers(d): + """Layers with MSA sparse attention (``sparse_attention_freq`` 1): every layer of MiniMax-M3.1.""" + return d.sparse_attn + + +def _three_way_qkv_layers(d): + """Layers whose QKV is the three-way [Q | K | V]: the dense-attention layers, and every layer without the + five-way packed projection (``fuse_qkv_index_projection=False``).""" + return d.dense_attn if d.fused else range(d.num_layers) + + +def _five_way_qkv_layers(d): + return d.sparse_attn if d.fused else range(0) + + +def _unfused_sparse_layers(d): + return range(0) if d.fused else d.sparse_attn + + +_ATTN = "model.layers.{i}.self_attn" +_MOE = "model.layers.{i}.block_sparse_moe" + + +class MiniMaxM3Tp4Weights: + """MiniMax-M3 NVFP4 / sm_103 / tp4: the language model's table and the passes over it. + + ``declare`` allocates every weight it names, ``manifest`` maps checkpoint keys onto them, ``load`` fills + them; ``_targets`` is the one place that decides which parameters exist. + """ + + #: Free the per-expert relayout scratch after each expert stack. + RELEASE_AFTER = "_e_s2" + + WEIGHTS: tuple[W, ...] = ( + # --- the two boundary norms of every layer (Gemma, folded) --- + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.input_layernorm.weight"), + transform=_gemma_fold, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.post_attention_layernorm.weight"), + transform=_gemma_fold, + ), + # --- attention: per-head Q/K norms (Gemma, unfolded), the packed QKV and the output projection --- + W("q_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".q_norm.weight")), + W("k_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".k_norm.weight")), + W( + "qkv_w", + shape=lambda d: (d.qkv_dense_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in "qkv")), + transform=_qkv_dense_w, + layers=_three_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_dense_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many(*(f"{_ATTN}.{p}_proj.weight_scale_inv" for p in "qkv")), + transform=_qkv_dense_s, + layers=_three_way_qkv_layers, + ), + W( + "qkv_w", + shape=lambda d: (d.qkv_sparse_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in ("q", "k", "v", "index_q", "index_k"))), + transform=_qkv_sparse_w, + layers=_five_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_sparse_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + *( + f"{_ATTN}.{p}_proj.weight_scale_inv" + for p in ("q", "k", "v", "index_q", "index_k") + ) + ), + transform=_qkv_sparse_s, + layers=_five_way_qkv_layers, + ), + # The separate index projection [index-Q | index-K] of the unfused sparse layers: the checkpoint's MXFP8 + # values with their block scales (the rows of the five-way projection's index part), run as an MXFP8 GEMM. + W( + "idx_w", + shape=lambda d: (d.index_heads * d.index_dim + d.index_dim, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(_ATTN + ".index_q_proj.weight", _ATTN + ".index_k_proj.weight"), + transform=_index_qk_w, + layers=_unfused_sparse_layers, + ), + W( + "idx_s", + shape=lambda d: ((d.index_heads * d.index_dim + d.index_dim) * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _ATTN + ".index_q_proj.weight_scale_inv", _ATTN + ".index_k_proj.weight_scale_inv" + ), + transform=_index_qk_s, + layers=_unfused_sparse_layers, + ), + W( + "idx_q_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_q_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "idx_k_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_k_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "o_w", + shape=lambda d: (d.hidden, d.q_width), + dtype=torch.float8_e4m3fn, + src=_one(_ATTN + ".o_proj.weight"), + transform=_row_w, + ), + W( + "o_s", + shape=lambda d: (d.hidden * (d.q_width // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_ATTN + ".o_proj.weight_scale_inv"), + transform=_row_s, + ), + # --- dense SwiGLU-OAI MLP (layers 0..2) --- + W( + "mlp_gu_w", + shape=lambda d: (2 * d.dense_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight", "model.layers.{i}.mlp.up_proj.weight" + ), + transform=_gate_up_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_gu_s", + shape=lambda d: (2 * d.dense_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight_scale_inv", + "model.layers.{i}.mlp.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_w", + shape=lambda d: (d.hidden, d.dense_inter), + dtype=torch.float8_e4m3fn, + src=_one("model.layers.{i}.mlp.down_proj.weight"), + transform=_row_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_s", + shape=lambda d: (d.hidden * (d.dense_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one("model.layers.{i}.mlp.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_dense_mlp_layers, + ), + # --- MoE (layers 3..59): the fp32 router and its bias, the shared expert --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + dtype=torch.float32, + src=_one(_MOE + ".gate.weight"), + transform=_f32, + layers=_moe_layers, + ), + W( + "router_bias", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_one(_MOE + ".e_score_correction_bias"), + transform=_f32, + layers=_moe_layers, + ), + W( + "sh_gu_w", + shape=lambda d: (2 * d.shared_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + _MOE + ".shared_experts.gate_proj.weight", _MOE + ".shared_experts.up_proj.weight" + ), + transform=_gate_up_w, + layers=_moe_layers, + ), + W( + "sh_gu_s", + shape=lambda d: (2 * d.shared_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _MOE + ".shared_experts.gate_proj.weight_scale_inv", + _MOE + ".shared_experts.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_moe_layers, + ), + W( + "sh_dn_w", + shape=lambda d: (d.hidden, d.shared_inter), + dtype=torch.float8_e4m3fn, + src=_one(_MOE + ".shared_experts.down_proj.weight"), + transform=_row_w, + layers=_moe_layers, + ), + W( + "sh_dn_s", + shape=lambda d: (d.hidden * (d.shared_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_MOE + ".shared_experts.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_moe_layers, + ), + # --- the routed experts, NVFP4 (trtllm-gen layouts), all 128 on every rank --- + W( + "e_w31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // 2), + dtype=torch.uint8, + src=_per_expert("w3.weight_packed", "w1.weight_packed"), + transform=_expert_w31, + layers=_moe_layers, + ), + W( + "e_w2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // 2), + dtype=torch.uint8, + src=_per_expert("w2.weight_packed"), + transform=_expert_w2, + layers=_moe_layers, + ), + W( + "e_s31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w3.weight_scale", "w1.weight_scale"), + transform=_expert_s31, + layers=_moe_layers, + ), + W( + "e_s2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w2.weight_scale"), + transform=_expert_s2, + layers=_moe_layers, + ), + # The per-tensor scalars of every expert: the checkpoint stores the multipliers into the quantized range + # (``*_global_scale``); the modelopt divisors modeling derives the kernels' scalars from are their FP32 + # reciprocals, taken by the same helper as the built-in loader. + *( + W( + f"e_{name}", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_per_expert(suffix), + transform=_divisor, + layers=_moe_layers, + ) + for name, suffix in ( + ("isc1", "w1.input_global_scale"), + ("isc3", "w3.input_global_scale"), + ("isc2", "w2.input_global_scale"), + ("ws2_1", "w1.weight_global_scale"), + ("ws2_3", "w3.weight_global_scale"), + ("ws2_2", "w2.weight_global_scale"), + ) + ), + # --- not per-layer --- + W( + "embed", + shape=lambda d: (d.vocab, d.hidden), + src=_one("model.embed_tokens.weight"), + layers=None, + ), + W( + "final_norm", + shape=lambda d: (d.hidden,), + src=_one("model.norm.weight"), + transform=_gemma_fold, + layers=None, + ), + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + mapping = core.model_config.mapping + sparse = cfg.sparse_attention_config + freq = [int(f) for f in sparse["sparse_attention_freq"]] + moe_freq = [ + int(f) for f in getattr(cfg, "moe_layer_freq", None) or [1] * cfg.num_hidden_layers + ] + tp = mapping.tp_size + head_dim = int(cfg.head_dim) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + index_heads = max(int(sparse["sparse_num_index_heads"]) // tp, 1) + index_dim = int(sparse["sparse_index_dim"]) + return SimpleNamespace( + fused=fused_projection(core.model_config.sparse_attention_config), + num_layers=cfg.num_hidden_layers, + sparse_attn=[i for i, f in enumerate(freq) if f], + dense_attn=[i for i, f in enumerate(freq) if not f], + moe=[i for i, f in enumerate(moe_freq) if f], + dense_mlp=[i for i, f in enumerate(moe_freq) if not f], + hidden=cfg.hidden_size, + head_dim=head_dim, + heads=heads, + kv_heads=kv_heads, + index_heads=index_heads, + index_dim=index_dim, + q_width=heads * head_dim, + qkv_dense_rows=(heads + 2 * kv_heads) * head_dim, + qkv_sparse_rows=(heads + 2 * kv_heads) * head_dim + index_heads * index_dim + index_dim, + dense_inter=int(cfg.dense_intermediate_size) // tp, + shared_inter=int(cfg.shared_intermediate_size) * int(cfg.n_shared_experts) // tp, + moe_inter=int(cfg.intermediate_size) // tp, + num_experts=int(cfg.num_local_experts), + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None)] + return [(f"l{i}_{entry.name}", i) for i in layers] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares (meta under the engine's meta init).""" + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, core) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + """param key -> [(checkpoint key or key tuple, index into the param | None, transform | None)].""" + d = self.dims(core) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, core, weights) -> set: + """The vision tower and its projectors, and the multi-token-prediction head: in the checkpoint, never built + by this target.""" + return {k for k in weights.keys() if k.startswith(_UNBUILT_PREFIXES)} + + def load(self, model, weights) -> None: + """Fill every declared weight and the shell's ``lm_head`` shard; assert coverage both ways.""" + core = model.model + d = self.dims(core) + manifest = self.manifest(core) + assert set(manifest) == set(core.w.keys()), ( + "manifest/parameter drift", + set(manifest) ^ set(core.w.keys()), + ) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(core, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(core.w[param_key], ckpt, index, transform, weights, d, consumed) + if param_key.endswith(self.RELEASE_AFTER): + torch.cuda.empty_cache() + # The shell owns the vocab-parallel head: this rank's rows of the full table. + _fill(model.lm_head.weight, _LM + "lm_head.weight", None, _col_w, weights, d, consumed) + shell = { + n for n, _ in model.named_parameters() if not n.startswith(("model.w.", "draft_model.")) + } + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + "checkpoint coverage: leftover keys are not exactly the unbuilt families (vision, MTP head); " + f"unexpected {sorted(leftover - expected)[:8]}, missed {sorted(expected - leftover)[:8]}" + ) + + +def _materialize(entry) -> torch.Tensor: + """``[:]`` on a lazy safetensors slice; a 0-dim tensor (the per-tensor scalars) rejects that index.""" + return entry if getattr(entry, "ndim", 1) == 0 else entry[:] + + +def _fill(param, ckpt, index, transform, weights, d, consumed) -> None: + keys = ckpt if isinstance(ckpt, tuple) else (ckpt,) + for key in keys: + assert key in weights, f"checkpoint key missing: {key}" + dst = param.data if index is None else param.data[index] + d.dev = dst.device + # Host tensors: a transform slices this rank's part on the host and moves only that to the device. + src_all = tuple(_materialize(weights[key]) for key in keys) + if transform is not None: + src = transform(src_all if isinstance(ckpt, tuple) else src_all[0], d) + else: + assert len(src_all) == 1, "a multi-key row needs a transform" + src = src_all[0] + if src.dim() == 0 and dst.dim() == 0: + pass + elif src.numel() == 1 and dst.numel() == 1: + src = src.reshape(dst.shape) + src = src.to(dst.device, non_blocking=True) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(keys) + _release(weights, keys) + + +def _release(weights, keys) -> None: + """Drop consumed checkpoint tensors from the engine's dict as the load goes (bounded host memory).""" + if hasattr(weights, "mark_consumed_keys"): + weights.mark_consumed_keys(keys) + else: + for key in keys: + weights.pop(key, None) + + +MODEL_WEIGHTS = MiniMaxM3Tp4Weights() + + +# -------------------------------------------------------------------------------------------- the drafter + + +def _draft_one(template: str): + return lambda d, i, t=template: [(t, None)] + + +def _draft_many(*templates: str): + return lambda d, i, ts=templates: [(tuple(ts), None)] + + +def _draft_fc_norms(d): + return range(d.capture_layers) + + +def _draft_qkv(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + return torch.cat([_col(q, d), _col(k, d), _col(v, d)], dim=0).contiguous() + + +class MiniMaxM3Tp4DraftWeights: + """The one-layer BF16 Eagle3 drafter (``speculative_model``), tp4. + + The fc over the three captured target layers and its per-layer fc_norms, the layer's three norms, its QKV + (input ``[embeds | hidden]``, so twice the hidden width; Q by heads, K/V by KV heads) and o_proj, its gated + MLP, the final norm and the full-vocabulary head (vocab-parallel). The checkpoint's ``embed_tokens`` is the + target's table (the drafter reads the target's); it is a predicted non-load. + """ + + WEIGHTS: tuple[W, ...] = ( + W( + "fc", + shape=lambda d: (d.hidden, d.capture_layers * d.hidden), + src=_draft_one("fc.weight"), + layers=None, + ), + W( + "fc_norm", + shape=lambda d: (d.hidden,), + src=lambda d, i: [(f"fc_norm.{i}.weight", None)], + layers=_draft_fc_norms, + ), + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.input_layernorm.weight"), + layers=None, + ), + W( + "hidden_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.hidden_norm.weight"), + layers=None, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.post_attention_layernorm.weight"), + layers=None, + ), + W( + "qkv", + shape=lambda d: (d.qkv_rows, 2 * d.hidden), + src=_draft_many(*(f"layers.0.self_attn.{p}_proj.weight" for p in "qkv")), + transform=_draft_qkv, + layers=None, + ), + W( + "o", + shape=lambda d: (d.hidden, d.q_width), + src=_draft_one("layers.0.self_attn.o_proj.weight"), + transform=_row_w, + layers=None, + ), + W( + "gu", + shape=lambda d: (2 * d.inter, d.hidden), + src=_draft_many("layers.0.mlp.gate_proj.weight", "layers.0.mlp.up_proj.weight"), + transform=_gate_up_w, + layers=None, + ), + W( + "dn", + shape=lambda d: (d.hidden, d.inter), + src=_draft_one("layers.0.mlp.down_proj.weight"), + transform=_row_w, + layers=None, + ), + W("final_norm", shape=lambda d: (d.hidden,), src=_draft_one("norm.weight"), layers=None), + ) + + def dims(self, draft) -> SimpleNamespace: + cfg = draft.draft_config.pretrained_config + mapping = draft.draft_config.mapping + tp = mapping.tp_size + head_dim = int(getattr(cfg, "head_dim", cfg.hidden_size // cfg.num_attention_heads)) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + return SimpleNamespace( + hidden=cfg.hidden_size, + capture_layers=draft.capture_layers, + head_dim=head_dim, + q_width=heads * head_dim, + qkv_rows=(heads + 2 * kv_heads) * head_dim, + inter=cfg.intermediate_size // tp, + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + if entry.layers is None: + return [(entry.name, None)] + return [(f"{entry.name}{i}", i) for i in entry.layers(d)] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, draft) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + d = self.dims(draft) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, draft, weights) -> set: + """The drafter checkpoint's own embedding: the drafter reads the target's table.""" + return {"embed_tokens.weight"} & set(weights.keys()) + + def load(self, draft, weights) -> None: + d = self.dims(draft) + manifest = self.manifest(draft) + assert set(manifest) == set(draft.w.keys()), set(manifest) ^ set(draft.w.keys()) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(draft, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(draft.w[param_key], ckpt, index, transform, weights, d, consumed) + _fill(draft.lm_head.weight, "lm_head.weight", None, _col_w, weights, d, consumed) + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + f"drafter checkpoint coverage: unexpected {sorted(leftover - expected)[:8]}, " + f"missed {sorted(expected - leftover)[:8]}" + ) + + +DRAFT_WEIGHTS = MiniMaxM3Tp4DraftWeights() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/__init__.py new file mode 100644 index 000000000000..2e340ff7284d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 NVFP4 / sm_103 / tp8.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/dspark.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/dspark.py new file mode 100644 index 000000000000..71af3ca118cc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/dspark.py @@ -0,0 +1,829 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP8 implementation.""" + +from __future__ import annotations + +import os +import time +from typing import Tuple + +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.comm import ( + allgather_cast as _allgather_cast, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + strict_accept as _strict_accept, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_block_tables import ( + enable_parallel_refresh, + refresh_block_tables, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_advance import ( + advance_ctx_and_read_view, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_store import ( + fold_k_norm_weight, + fused_ctx_store, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_drafter_inputs import ( + drafter_inputs as _drafter_inputs, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_forward_indices import ( + forward_indices, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_qk_norm_rope import ( + qk_norm_rope, + qk_norm_rope_append, +) +from tensorrt_llm._torch.distributed import allgather +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + external_drafter_config_kwargs, + get_draft_model, +) +from tensorrt_llm._torch.speculative.dspark import DSparkWorker +from tensorrt_llm._torch.speculative.interface import ( + _RNG_SLOT_SPAN, + should_use_separate_draft_kv_cache, + tail_bookkeep_draft_len, + tail_bookkeep_log_once, +) +from tensorrt_llm.logger import logger + +from .dspark_s4 import MAX_FAST_BATCH, TAILBK_SPEC, S4WorkerMixin +from .m31_switches import m31_disabled +from .mxdraft import route_mx_linears +from .ops.embedding.m3_embedding import m3_embedding + +MXDRAFT = not m31_disabled("MXDRAFT") +DRAFT_APPEND_FUSE = not m31_disabled("DRAFT_APPEND_FUSE") +DRAFT_ATTN_PDL = not m31_disabled("DRAFT_ATTN_PDL") +_APPEND_FUSE_LOGGED = [False] + + +def _qk_norm_rope_append_logged(*args): + """catalog qk_norm_rope_append, logging the engagement once per process on its first taken call (host only; a + captured step logs while it is captured).""" + out = qk_norm_rope_append(*args) + if out is not None and (not _APPEND_FUSE_LOGGED[0]): + _APPEND_FUSE_LOGGED[0] = True + logger.info( + "[M3 dispatch] DRAFT_APPEND_FUSE taken: drafter K / V append in the EX-11 launch" + ) + return out + + +def is_standalone_dspark(spec_config) -> bool: + """Whether the run speculates with a standalone DSpark drafter (its own checkpoint, ``decoding_type: DSpark``).""" + return bool( + spec_config is not None + and spec_config.spec_dec_mode.use_one_engine() + and spec_config.spec_dec_mode.is_dspark() + and (not getattr(spec_config, "draft_is_embedded_in_target", False)) + ) + + +class M3EmbedTokens(nn.Module): + """The target's token embedding as the module the in-tree drafter borrows (``load_weights_from_target_model``). + + The drafter embeds the bonus and mask tokens through ``embed_tokens(ids)``; the target keeps its replicated + table as a raw tensor and gathers it with ``m3_embedding``, so this module calls the same gather on the same + table (an exact gather, so the drafter sees the rows the target embeds). + """ + + def __init__(self, core: nn.Module) -> None: + super().__init__() + self._core = [core] # not a submodule: the table belongs to the target + + @property + def weight(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def forward(self, ids: torch.Tensor) -> torch.Tensor: + return m3_embedding(ids, self.weight) + + +def s10c_c_drafter_inputs( + embed_tokens, + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + mask_token_id, + trained_mask_embedding, + block_size, + tokens_per_request, +): + """S10c-C: the drafter's step inputs in one launch (catalog/spec_decode/dspark_drafter_inputs), for this target's + embedding only: ``M3EmbedTokens`` is an exact gather of the replicated table, so the noise block is table rows (the + trained mask row if the drafter keeps one). None for any other embedding: the stock path runs.""" + if not isinstance(embed_tokens, M3EmbedTokens): + return None + table = embed_tokens.weight + mask_row = ( + trained_mask_embedding if trained_mask_embedding is not None else table[int(mask_token_id)] + ) + return _drafter_inputs( + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + table, + mask_row, + block_size, + tokens_per_request, + ) + + +N3A_DISABLED = m31_disabled("N3A") +N3A_KV_DISABLED = N3A_DISABLED or m31_disabled("N3A_KV") +N3A_KV_CHECK = os.environ.get("TRTLLM_M31_CHECK_N3A_KV", "").strip() not in ("", "0") +N3P_DISABLED = m31_disabled("N3P") +A1_DISABLED = m31_disabled("A1") +A1_MAX_ROWS = 64 +MCAG = not m31_disabled("MCAG") +A1_K7_DISABLED = m31_disabled("A1_K7") +_n3a_logged = False + + +def _n3a_log_once() -> None: + global _n3a_logged + if not _n3a_logged: + _n3a_logged = True + logger.info( + "[M3 dispatch] n3_strict_accept taken: fused strict acceptance + next-step input tokens" + ) + + +_n3p_logged = False +_n3a_kv_logged = False + + +def _n3a_kv_log_once() -> None: + global _n3a_kv_logged + if not _n3a_kv_logged: + _n3a_kv_logged = True + logger.info( + "[M3 dispatch] n3a_kv_rewind taken: draft-forward KV rewind + kv_lens bump in one launch" + ) + + +_a1_logged = set() + + +def _a1_log_once(key, message: str) -> None: + if key not in _a1_logged: + _a1_logged.add(key) + logger.info(message) + + +def _n3p_log_once() -> None: + global _n3p_logged + if not _n3p_logged: + _n3p_logged = True + logger.info( + "[M3 dispatch] n3p_target_logits taken: verify-row logits without the gather_ids copy and the concat" + ) + + +class M3DSparkWorker(S4WorkerMixin, DSparkWorker): + """The in-tree ``DSparkWorker`` with the calls this target's shell makes on its worker. W1: the one-launch Markov + chain (``S4WorkerMixin``, switch ``TRTLLM_M31_DRAFT_DISABLE_S4``). N3 A2: the strict acceptance and the next-step + input tokens in one launch each (switch ``TRTLLM_M31_DISABLE_N3A``). TAILBK R1 / Z1: the one-request Philox rows + from the per-token window and no padding-row zeroing in captured batch-1 graphs, bitwise (spec_metadata's + tail_bookkeep; switch ``TRTLLM_M31_DISABLE_TAILBK``). Everything else is the in-tree path.""" + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None + ) -> None: + """The in-tree DFlash capture: layer output (hidden + residual) into the static capture buffer.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model + ) -> None: + """No capture norm: DSpark's fc reads the raw captured layers.""" + return None + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Never: the target logits always go through the logits processor and the in-tree strict accept.""" + return False + + _a1 = None + + def prebuild(self, lm_head: nn.Module, draft_model) -> None: + """The one-launch Markov chain's shards, buffers, exchange workspace and compiles, then the N3 A2 compiles + (rank-local) and the N3 A1 workspace and compiles (collective), outside capture, every rank, in this order; + no fused verify, no coded target head, no fused draft loop.""" + rank = getattr(getattr(self, "mapping", None), "tp_rank", None) + lean = not m31_disabled("S10") + self.enable_lean_glue(draft_model, lean) + k = getattr(getattr(self, "spec_config", None), "max_draft_len", 0) or 0 + logger.info( + f"M3.1 DSpark S10: lean drafter glue {('on' if lean else 'off')} on rank {rank}" + ) + s10c_a = self.enable_fused_ctx_store( + None if m31_disabled("S10C") or m31_disabled("S10C_A") else fused_ctx_store + ) + tailbk_f = fold_k_norm_weight(bool(s10c_a) and TAILBK_SPEC) + logger.info( + ( + "M3.1 DSpark S10c-A: fused drafter ctx store " + f"{('on' if s10c_a else 'off')}" + "; TAILBK_SPEC F k_norm weight in the store kernel " + f"{('on' if tailbk_f else 'off')}" + " on rank " + f"{rank}" + ) + ) + s10c_c = self.enable_fused_drafter_inputs( + None if m31_disabled("S10C") or m31_disabled("S10C_C") else s10c_c_drafter_inputs + ) + logger.info( + f"M3.1 DSpark S10c-C: fused drafter step inputs {('on' if s10c_c else 'off')} on rank {rank}" + ) + s10c_d = self.enable_fused_ctx_advance( + None if m31_disabled("S10C") or m31_disabled("S10C_D") else advance_ctx_and_read_view + ) + logger.info( + f"M3.1 DSpark S10c-D: fused drafter context advance {('on' if s10c_d else 'off')} on rank {rank}" + ) + s10c_b = self.enable_fused_block_refresh( + None if m31_disabled("S10C") or m31_disabled("S10C_B") else refresh_block_tables + ) + s10c_b2 = enable_parallel_refresh(s10c_b and (not m31_disabled("S10C_B2"))) + logger.info( + ( + "M3.1 DSpark S10c-B: fused drafter block-table refresh " + f"{('on' if s10c_b else 'off')}" + " on rank " + f"{rank}" + " (S10c-B2 2-D grid " + f"{('on' if s10c_b2 else 'off')}" + ")" + ) + ) + s10c_e = self.enable_fused_forward_indices( + draft_model, None if m31_disabled("S10C") or m31_disabled("S10C_E") else forward_indices + ) + logger.info( + f"M3.1 DSpark S10c-E: fused drafter forward indices {('on' if s10c_e else 'off')} on rank {rank}" + ) + ex11 = self.enable_fused_qk_norm_rope( + draft_model, None if m31_disabled("EX11") else qk_norm_rope + ) + logger.info( + f"M3.1 DSpark EX-11: fused drafter q/k norm + RoPE {('on' if ex11 else 'off')} on rank {rank}" + ) + append_fuse = self.enable_fused_qk_norm_rope_append( + draft_model, _qk_norm_rope_append_logged if ex11 and DRAFT_APPEND_FUSE else None + ) + logger.info( + ( + "M3.1 DSpark DRAFT_APPEND_FUSE: drafter K / V append in the EX-11" + " launch " + f"{('on' if append_fuse else 'off')}" + " on rank " + f"{rank}" + ) + ) + attn_pdl = self.enable_attention_pdl(draft_model, DRAFT_ATTN_PDL) + logger.info( + f"M3.1 DSpark DRAFT_ATTN_PDL: drafter attention PDL {('on' if attn_pdl else 'off')} on rank {rank}" + ) + mx = ( + 0 + if not MXDRAFT or k <= 0 + else route_mx_linears(draft_model, [b * k for b in range(1, MAX_FAST_BATCH + 1)]) + ) + logger.info(f"M3.1 DSpark MXDRAFT: MXFP8 drafter GEMV on {mx} projections on rank {rank}") + self.prebuild_s4(lm_head, draft_model) + if not N3A_DISABLED and lm_head is not None: + _strict_accept.precompile((self.max_draft_len,), lm_head.weight.device) + self._a1 = self._prebuild_a1(lm_head) if lm_head is not None else None + self._mcag_cols = self._prebuild_mcag(lm_head) if self._a1 is not None and MCAG else 0 + mcag = self.enable_draft_logits_gather_op(self._mcag_gather if self._mcag_cols else None) + logger.info( + f"M3.1 MCAG: draft-logits all-gather on the multicast kernel {('on' if mcag else 'off')} on rank {rank}" + ) + n3a_kv = False + if not N3A_KV_DISABLED and lm_head is not None: + _strict_accept.precompile_kv_rewind(lm_head.weight.device) + self._n3a_kv_buf = torch.empty( + _strict_accept.max_requests(), dtype=torch.int32, device=lm_head.weight.device + ) + n3a_kv = self.enable_fused_kv_rewind(self._n3a_kv_rewind) + logger.info( + ( + "M3.1 N3A-KV: one-launch draft-forward KV bookkeeping " + f"{('on' if n3a_kv else 'off')}" + " on rank " + f"{getattr(getattr(self, 'mapping', None), 'tp_rank', None)}" + ) + ) + + _n3a_kv_buf = None + _n3a_kv_counts = None + + def _n3a_kv_rewind(self, num_accepted, kv_lens, nc: int, bs: int): + """N3A-KV for the DFlash hook: ``1 - num_accepted[nc:bs]`` (a view of the persistent buffer) with + ``kv_lens[nc:bs] += 1`` in one launch, or None (the stock lines) for operands or a variant it does not take.""" + n = bs - nc + buf = self._n3a_kv_buf + if buf is None or not _strict_accept.supports_kv_rewind(num_accepted, kv_lens, buf, nc, n): + return None + if not _strict_accept.is_compiled_kv_rewind(num_accepted, kv_lens): + return None + _n3a_kv_log_once() + if not N3A_KV_CHECK: + return _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + want_rewind = 1 - num_accepted[nc:bs] + want_kv = kv_lens[nc:bs] + 1 + out = _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + if self._n3a_kv_counts is None: + self._n3a_kv_counts = torch.zeros(2, dtype=torch.int64, device=kv_lens.device) + bad = (out != want_rewind).any() | (kv_lens[nc:bs] != want_kv).any() + self._n3a_kv_counts[0] += 1 + self._n3a_kv_counts[1] += bad.to(torch.int64) + if not torch.cuda.is_current_stream_capturing(): + calls, mism = self._n3a_kv_counts.tolist() + logger.info(f"M3.1 N3A-KV check: {calls} calls, {mism} mismatches") + return out + + def _a1_all_ok(self, local: bool) -> bool: + """Whether ``local`` holds on every rank of the TP group: A1 is set up on every rank or on none (a rank that + is not eligible, or whose compile failed, turns it off everywhere).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + return _mnnvl_workspace_all_succeeded(_get_mnnvl_workspace_comm(self.mapping), local) + + def _prebuild_a1(self, lm_head: nn.Module): + """The N3 A1 workspace and its K3 verify-row compiles, or None (every rank alike): off by its switch, at + another draft length, without a column-parallel bf16 head over a TP group, or when a compile fails on any + rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + mapping = self.mapping + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return None + cols = int(getattr(lm_head, "out_features", 0)) + sms = torch.cuda.get_device_properties(lm_head.weight.device).multi_processor_count + off = [ + why + for why, bad in ( + ("TRTLLM_M31_DISABLE_A1 set", A1_DISABLED), + ( + f"draft length {int(self.max_draft_len)} with TRTLLM_M31_DISABLE_A1_K7 set", + int(self.max_draft_len) != 3 and A1_K7_DISABLED, + ), + ( + f"draft length {int(self.max_draft_len)}: more than {A1_MAX_ROWS} verify rows at batch 1", + int(self.max_draft_len) + 1 > A1_MAX_ROWS, + ), + ( + "not a column-parallel, gathered, unpadded, bias-free head", + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or (getattr(lm_head, "bias", None) is not None), + ), + (f"head dtype {lm_head.weight.dtype}", lm_head.weight.dtype != torch.bfloat16), + ( + f"{cols} shard columns (whole 16-byte vectors needed)", + cols <= 0 or cols % 8 != 0, + ), + ( + f"{sms} SMs (fewer than {_allgather_cast.MAX_GRID})", + sms < _allgather_cast.MAX_GRID, + ), + ) + if bad + ] + if not self._a1_all_ok(not off): + logger.info( + ( + "M3.1 N3 A1: off on rank " + f"{mapping.tp_rank}" + " (" + f"{'; '.join(off) or 'off on another rank'}" + "); the verify logits use the NCCL all-gather" + ) + ) + return None + k1 = int(self.max_draft_len) + 1 + rows = tuple((b * k1 for b in range(1, A1_MAX_ROWS // k1 + 1))) + start = time.perf_counter() + if not _allgather_cast.precompile( + lm_head.mapping, cols, rows, lm_head.weight.device, agree=self._a1_all_ok + ): + logger.warning( + "M3.1 N3 A1: a compile failed on some rank; every rank keeps the NCCL all-gather" + ) + return None + ws = _allgather_cast.workspace_args(lm_head.mapping, cols) + logger.info( + ( + "M3.1 N3 A1: one-launch verify all-gather set up on rank " + f"{mapping.tp_rank}" + "/" + f"{mapping.tp_size}" + " (shard " + f"{cols}" + " columns, rows " + f"{rows}" + ", workspace " + f"{ws['ws_uc'].numel() * 4}" + " bytes/rank, set-up " + f"{time.perf_counter() - start:.2f}" + " s)" + ) + ) + return ws + + def _prebuild_mcag(self, lm_head: nn.Module) -> int: + """Compile the fp32 draft-logits gather for B * K rows up to its capacity, on every rank (A1's workspace and + agreement); its shard columns, or 0 (every rank alike) when a compile fails anywhere.""" + cols = int(getattr(lm_head, "out_features", 0)) + k = int(self.max_draft_len) + if k <= 0 or cols % 4 != 0: + return 0 + rows = tuple((b * k for b in range(1, _allgather_cast.MAX_ROWS_FP32 // k + 1))) + if not _allgather_cast.precompile_f32( + lm_head.mapping, cols, rows, lm_head.weight.device, agree=self._a1_all_ok + ): + logger.warning( + "M3.1 MCAG: a compile failed on some rank; every rank keeps the NCCL all-gather" + ) + return 0 + return cols + + def _mcag_gather(self, logits: torch.Tensor): + """The draft logits gathered in one launch on the A1 workspace, or None (the stock all-gather) for a shape + that is not compiled; the decision depends on the shape only, the same on every rank.""" + rows = logits.numel() // max(int(logits.shape[-1]), 1) + if ( + logits.dtype != torch.float32 + or int(logits.shape[-1]) != self._mcag_cols + or ( + not _allgather_cast.is_compiled( + rows, self._mcag_cols, self._a1["world"], device=logits.device, fp32=True + ) + ) + ): + _a1_log_once( + ("mcag", rows, logits.dtype), + f"[M3 dispatch] MCAG draft-logits all-gather not taken: {rows} rows {logits.dtype}; NCCL all-gather", + ) + return None + _a1_log_once( + "mcag", + "[M3 dispatch] MCAG draft-logits all-gather taken: one launch on the A1 workspace", + ) + return _allgather_cast.gather_last_dim_f32(logits.contiguous(), **self._a1) + + def _n3a_eligible( + self, logits, draft_tokens, num_contexts: int, batch_size: int, spec_metadata + ) -> bool: + """Whether this step's strict acceptance takes the fused op: a decode-only, non-ragged step whose logits hold + exactly batch_size x (D + 1) verify rows and whose drafts the op takes. Host state and shapes only, so it is + fixed per captured graph and the same on every rank. Decided before the target tokens are sampled, so the + sampler runs exactly once either way.""" + if N3A_DISABLED or spec_metadata is None or spec_metadata.is_ragged_verify: + return False + if num_contexts != 0 or batch_size <= 0 or draft_tokens.dim() != 2: + return False + n_req, draft_len = draft_tokens.shape + if n_req != batch_size or draft_len <= 0 or n_req > _strict_accept.max_requests(): + return False + rows = logits.shape[0] if logits.dim() > 1 else 1 + return rows == batch_size * (draft_len + 1) and draft_tokens.is_contiguous() + + def _sample_and_accept_draft_tokens_base( + self, logits, draft_tokens, num_contexts, batch_size, spec_metadata + ): + """The in-tree strict acceptance with its epilogue in one launch (N3 A2): the same target tokens (the in-tree + sampler, called once), the same accepted tokens and counts, then the in-tree forced-acceptance override (its + RNG pool and counters advance as on an in-tree step).""" + if not self._n3a_eligible(logits, draft_tokens, num_contexts, batch_size, spec_metadata): + return super()._sample_and_accept_draft_tokens_base( + logits, draft_tokens, num_contexts, batch_size, spec_metadata + ) + if logits.dim() == 1: + logits = logits.unsqueeze(0) + target_tokens = self._sample_tokens_for_batch( + logits, spec_metadata, num_contexts, batch_size + ) + if _strict_accept.supports_strict_accept( + target_tokens, draft_tokens + ) and _strict_accept.is_compiled_strict_accept(target_tokens, draft_tokens): + accepted, num_accepted = _strict_accept.spec_strict_accept(target_tokens, draft_tokens) + _n3a_log_once() + else: + accepted, num_accepted = _strict_accept.spec_strict_accept.reference( + target_tokens, draft_tokens + ) + num_accepted = self._apply_force_accepted_tokens( + num_accepted, num_contexts, draft_tokens.shape[-1], spec_metadata=spec_metadata + ) + return (accepted, num_accepted) + + def target_logits(self, hidden_states, attn_metadata, spec_metadata, lm_head: nn.Module): + """The verify rows' gathered fp32 target logits (N3 P-gid + P-cat), or None for the logits processor. + + On a decode-only, non-ragged step every row is a verify row and ``gather_ids`` is ``arange(num_tokens)`` (each + generation request extends it with the next ``1 + D`` row indices), so the rows are a slice, not a gather. + The processor's head is the column-parallel BF16 GEMM, an all-gather along the vocab (NCCL, then a concat) + and ``.float()``; here the same all-gather runs along dim 0 (no concat) and one strided copy writes the + rank-major shards into the fp32 output: the same bytes, an exact widening. Shapes and host state only, so + the decision is fixed per captured graph and the same on every rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + if ( + N3P_DISABLED + or spec_metadata is None + or getattr(spec_metadata, "is_ragged_verify", False) + ): + return None + if attn_metadata.num_contexts != 0 or spec_metadata.gather_ids is None: + return None + rows = hidden_states.shape[0] + if spec_metadata.gather_ids.shape[0] != rows or rows == 0: + return None + if ( + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or (getattr(lm_head, "bias", None) is not None) + ): + return None + shard = lm_head.apply_linear(hidden_states, None) + _n3p_log_once() + tp = lm_head.tp_size + if tp == 1: + return shard.float() + if self._a1 is not None: + if rows <= A1_MAX_ROWS and _allgather_cast.is_compiled( + rows, shard.shape[-1], tp, 0, shard.device + ): + _a1_log_once( + "taken", + "[M3 dispatch] n3_a1 allgather_cast taken: verify logits gathered in one launch", + ) + return _allgather_cast.allgather_cast(shard, **self._a1) + _a1_log_once( + rows, + ( + "[M3 dispatch] n3_a1 allgather_cast not taken: " + f"{rows}" + " verify rows (compiled: verify rows up to " + f"{A1_MAX_ROWS}" + "); NCCL all-gather" + ), + ) + gathered = allgather(shard, lm_head.mapping, dim=0) + vocab_shard = shard.shape[-1] + out = torch.empty((rows, tp * vocab_shard), dtype=torch.float32, device=shard.device) + out.view(rows, tp, vocab_shard).copy_(gathered.view(tp, rows, vocab_shard).transpose(0, 1)) + return out + + def _prepare_next_new_tokens( + self, + accepted_tokens, + next_draft_tokens, + batch_indices_cuda, + batch_size, + num_accepted_tokens, + ): + """The next step's input tokens in one launch (N3 A2), else the in-tree index + concat.""" + batch_idx = batch_indices_cuda[:batch_size] + if ( + not N3A_DISABLED + and next_draft_tokens.dim() == 2 + and (next_draft_tokens.shape[0] == batch_size) + and _strict_accept.supports_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + and _strict_accept.is_compiled_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + ): + return _strict_accept.spec_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + return super()._prepare_next_new_tokens( + accepted_tokens, next_draft_tokens, batch_indices_cuda, batch_size, num_accepted_tokens + ) + + _tailbk_win = None + + def _forward_impl(self, *args, **kwargs): + """The S4 / DFlash step; R1's offsets window lives for one step only (set by its accept call, read by its + draft block), so a step whose accept did not stash one recomputes it.""" + self._tailbk_win = None + try: + return super()._forward_impl(*args, **kwargs) + finally: + self._tailbk_win = None + + def _rng_state_per_request(self, spec_metadata, start=0, end=None, repeat=1, *, slot): + """TAILBK R1: at one decode request, the accept and draft-block Philox (seed, offset) rows read from the + per-token buffers (one int64 add per step instead of eight launches), else the in-tree expansion. + + ``_populate_request_rng_state`` writes row r (0..K) of the request's per-token window as + ``seeds[r] = seed`` and ``offsets[r] = o + r * SPAN`` (o = ``request_offsets[0]``; the host sum fits int64, + or its upload would have raised), uploaded with ``request_seeds`` / ``request_offsets`` on every sampled step. + The in-tree rows are ``(o + slot * SPAN) + k * SPAN``: draft row k at slot K + 1 is ``offsets[k] + (K + 1) * + SPAN`` and the accept row at slot 2K + 1 is ``offsets[K] + (K + 1) * SPAN``. Integer addition is associative + and commutative mod 2^64, so the window ``offsets[0:K + 1] + (K + 1) * SPAN`` holds exactly the in-tree + offsets, even where they wrap, and the seeds are the request's seed by construction. The returned tensors are + contiguous int64 views with one entry per row, as the in-tree ones. Preconditions are host state and shapes + (``tail_bookkeep_draft_len``), so the choice is fixed per captured graph.""" + state = self._tailbk_rng_window(spec_metadata, start, end, repeat, slot) + if state is not None: + return state + return super()._rng_state_per_request(spec_metadata, start, end, repeat, slot=slot) + + def _tailbk_rng_window(self, spec_metadata, start, end, repeat: int, slot: int): + """R1's (seed, offset) rows, or None (the in-tree expansion) outside its preconditions.""" + k = tail_bookkeep_draft_len(spec_metadata, start, end) + if k == 0: + return None + first = spec_metadata.rng_draft_slot(0) + if first != k + 1 or spec_metadata.rng_accept_slot != first + k: + return None + accept = repeat == 1 and slot == spec_metadata.rng_accept_slot + block = repeat == k and slot == first + if not (accept or block): + return None + stash = self._tailbk_win + if block and stash is not None and (stash[0] is spec_metadata): + win = stash[1] + else: + win = spec_metadata.offsets[: k + 1] + first * _RNG_SLOT_SPAN + tail_bookkeep_log_once( + "R1", "Philox seeds / offsets from the per-token window (one add per step)" + ) + if accept: + self._tailbk_win = (spec_metadata, win) + return (spec_metadata.request_seeds[0:1], win[k : k + 1]) + self._tailbk_win = None + return (spec_metadata.seeds[:k], win[:k]) + + def _zero_padding_rows( + self, + logits, + spec_metadata, + num_contexts: int, + batch_size: int, + rows_per_request: int, + *, + packed_verify: bool = False, + ): + """TAILBK Z1: no padding-row zeroing while a batch-1 decode graph is captured, else the in-tree zeroing. + + Why the in-tree result is a bitwise copy of ``logits`` on every replay of such a graph: + (i) a batch-1 graph replays exactly one real request. ``CUDAGraphRunner._get_padded_batch`` only appends + dummies after the batch's own requests, up to the graph's size, and the executor forwards no empty batch + without attention DP (``_can_queue``: batch_size > 0), so a size-1 graph never holds a dummy. Under + attention DP an empty rank does run a dummy, so attention DP is excluded. + (ii) a real request's slot is its ``py_seq_slot`` in ``[0, slot_capacity)`` and the padding marker is + ``dummy_slot_row == slot_capacity`` (``prepare_rejection_sampling_buffers``), so the padding mask is + all-false and the in-tree ``masked_fill`` copies ``logits`` unchanged. + (iii) every consumer is out-of-place (the non-fused probs: the softmax / top-k mask ops and the m31_softmax_mc + override allocate their outputs, top-p reads probs), so returning ``logits`` itself instead of that copy + exposes no write to the caller's verify logits. The fused modes are not audited and keep the copy. + (iv) the consumers see the layout the copy would have: the skip needs contiguous logits at a 256-byte aligned + address (the copy is a fresh contiguous allocation, which the caching allocator aligns at least that much), + since the m31_softmax_mc override takes contiguous, 16-byte aligned inputs only. + Eager steps (the graph warmup's dummy included) keep the in-tree zeroing; so do padded graphs (2..8), which + can hold dummies, and ragged verification (its layout check). Host state only, fixed per captured graph.""" + if self._tailbk_skips_padding_rows(logits, spec_metadata, num_contexts, batch_size): + tail_bookkeep_log_once("Z1", "padding-row zeroing skipped in captured batch-1 graphs") + return logits + return super()._zero_padding_rows( + logits, + spec_metadata, + num_contexts, + batch_size, + rows_per_request, + packed_verify=packed_verify, + ) + + def _tailbk_skips_padding_rows( + self, logits, spec_metadata, num_contexts: int, batch_size: int + ) -> bool: + """Z1's exact condition: tail bookkeeping on, one generation request and no context rows (num_gens == 1), + no ragged verification, a non-fused sampling mode, no attention DP, contiguous 256-byte aligned logits and a + capturing stream.""" + if not getattr(spec_metadata, "tail_bookkeep", False): + return False + if not logits.is_contiguous() or logits.data_ptr() % 256 != 0: + return False + if num_contexts != 0 or batch_size != 1 or getattr(spec_metadata, "is_ragged_verify", True): + return False + mode = getattr(spec_metadata, "advanced_sampling_mode", None) + if mode is None or mode.is_fused: + return False + mapping = getattr(self, "mapping", None) + if mapping is None or getattr(mapping, "enable_attention_dp", True): + return False + return torch.cuda.is_current_stream_capturing() + + +def build_dspark( + shell, model_config: ModelConfig +) -> Tuple[ModelConfig, nn.Module, M3DSparkWorker, bool]: + """The drafter config, the drafter and the worker of one-model standalone DSpark (mirrors + ``SpecDecOneEngineForCausalLM`` for an external drafter).""" + spec = model_config.spec_config + if not is_standalone_dspark(spec): + raise ValueError("build_dspark serves the standalone DSpark drafter only") + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, **external_drafter_config_kwargs(model_config, spec) + ) + keep_drafter_kv_dtype(draft_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = get_draft_model(model_config, draft_config, shell.lm_head, shell.model) + draft.logits_processor = shell.logits_processor + worker = M3DSparkWorker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + logger.info( + ( + "M3.1 target DSpark: " + f"{type(worker).__name__}" + " over " + f"{type(draft).__name__}" + ", K=" + f"{spec.max_draft_len}" + ", separate draft KV cache=" + f"{use_separate}" + ) + ) + return (draft_config, draft, worker, use_separate) + + +def load_dspark_draft_weights(shell, weights, weight_mapper=None) -> None: + """The drafter's own checkpoint, then the target's embedding and head (mirrors + ``SpecDecOneEngineForCausalLM.load_draft_weights`` for DSpark). + + The drafter borrows the target's embedding and head, and ``M3EmbedTokens.weight`` IS the target's table, so a + drafter checkpoint that ships either would load into a module the target owns or be shadowed by it: refused. + """ + borrowed = sorted( + (k for k in weights.keys() if "embed_tokens." in k or k.split(".")[-2:-1] == ["lm_head"]) + ) + if borrowed: + raise ValueError( + f"the DSpark drafter checkpoint ships tensors this target lends it (embedding / head): {borrowed[:4]}" + ) + shell.draft_model.load_weights(weights=weights, weight_mapper=weight_mapper) + shell.draft_model.load_weights_from_target_model(shell) + if ( + shell.draft_model.lm_head is not shell.lm_head + or shell.draft_model.draft_model_full.model.embed_tokens is not shell.model.embed_tokens + ): + raise RuntimeError("the DSpark drafter does not borrow this target's embedding and head") + own = sorted( + (n for n, _ in shell.named_parameters() if not n.startswith(("model.w.", "draft_model."))) + ) + logger.info(f"M3.1 target DSpark: shell parameters outside the core and the drafter: {own}") + + +def keep_drafter_kv_dtype(draft_config: ModelConfig) -> None: + """Drop any KV quantization from the drafter's config, so its context pool is allocated in its projection dtype. + + The one-engine construction copies the target's KV algo onto the drafter; on this target that is NVFP4, which + allocates a packed int8 pool the worker refuses to bind (it falls back to its private arena, and a reused + prefix's drafter K/V is lost). The cached quant modes are dropped with it, as the draft KV budget does. + """ + for quant in [draft_config.quant_config, *(draft_config.quant_config_dict or {}).values()]: + quant.kv_cache_quant_algo = None + quant.__dict__.pop("quant_mode", None) + quant.__dict__.pop("layer_quant_mode", None) + + +def draft_kv_override_applies(spec_config) -> bool: + """The target's FP8 draft-KV override is an Eagle3 need; a DSpark drafter keeps its BF16 context pool.""" + return not is_standalone_dspark(spec_config) + + +__all__ = [ + "M3DSparkWorker", + "M3EmbedTokens", + "build_dspark", + "draft_kv_override_applies", + "is_standalone_dspark", + "keep_drafter_kv_dtype", + "load_dspark_draft_weights", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/dspark_s4.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/dspark_s4.py new file mode 100644 index 000000000000..79bb6aa118e3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/dspark_s4.py @@ -0,0 +1,477 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP8 implementation.""" + +from __future__ import annotations + +import atexit +import os +from typing import Dict, Optional, Tuple + +import torch +import torch.nn.functional as F +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_bias as markov_bias, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_chain as chain, +) +from tensorrt_llm._torch.models.modeling_speculative import dspark_markov_step_bias +from tensorrt_llm.logger import logger + +from .m31_switches import m31_disabled + +MARKOV_RANK = 256 +MAX_FAST_BATCH = 8 +DISABLE_ENV = "TRTLLM_M31_DRAFT_DISABLE_S4" +S4_COUNT_LOG_EVERY = 1000 +MARKOV_LL = not m31_disabled("MARKOV_LL") +TAILBK_SPEC = not m31_disabled("TAILBK_SPEC") +_DISABLED = os.environ.get(DISABLE_ENV, "").strip() not in ("", "0") + + +def disabled() -> bool: + return _DISABLED + + +class _S4State: + def __init__( + self, + mapping, + rows: int, + shard: int, + offset: int, + w1: torch.Tensor, + w2s: torch.Tensor, + block: int, + ws: dict, + ) -> None: + self.mapping = mapping + self.ws = ws + self.rows = rows + self.shard = shard + self.offset = offset + self.w1 = w1 + self.w2s = w2s + self.block = block + self.bufs: Dict[Tuple[int, int], torch.Tensor] = {} + + def buffer(self, batch: int) -> torch.Tensor: + return self.bufs[batch, self.block] + + +S4_LOGITS_BATCHED = True + + +def _biased_shard_logits( + shard_logits: torch.Tensor, + first_prev: torch.Tensor, + tokens: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + fp32: bool = True, +) -> torch.Tensor: + """The in-tree chain's biased block logits on this rank's vocab shard, given the chain's tokens: step k adds + bf16(w2 @ w1[prev_k]) with prev_0 = the anchor and prev_k = token k-1 (dspark_markov_step_bias). With ``fp32`` + (the in-tree arithmetic: LogitsProcessor hands the chain fp32 logits) the bf16 shard logits and the bias are + widened and added in fp32; without it, one bf16 add.""" + if ( + fp32 + and S4_LOGITS_BATCHED + and TAILBK_SPEC + and markov_bias.supports(shard_logits, first_prev, tokens, w1) + ): + B, K, N = shard_logits.shape + bias = F.linear(markov_bias.prev_embeddings(first_prev, tokens, w1), w2_shard) + return markov_bias.widen_add(shard_logits, bias.view(B, K, N)) + if fp32: + shard_logits = shard_logits.float() + B, K, N = shard_logits.shape + prevs = torch.cat([first_prev.view(B, 1), tokens[:, : K - 1].long()], dim=1) + if S4_LOGITS_BATCHED: + bias = dspark_markov_step_bias(prevs.reshape(-1), w1, w2_shard).view(B, K, N) + return shard_logits + bias.to(shard_logits.dtype) + out = torch.empty_like(shard_logits) + for k in range(K): + out[:, k] = shard_logits[:, k] + dspark_markov_step_bias(prevs[:, k], w1, w2_shard).to( + shard_logits.dtype + ) + return out + + +class _ShardedDraftLogits(nn.Module): + """The step's draft logits processor: this rank's vocab shard of ``lm_head`` over the gathered block rows, + copied into the padded buffer; any other call goes to the drafter's own processor.""" + + def __init__(self, inner: nn.Module, state: _S4State, batch: int) -> None: + super().__init__() + self.inner = inner + self._state = [state] + self.batch = batch + + def forward(self, hidden_states, lm_head, attn_metadata, return_context_logits: bool = False): + state = self._state[0] + if hidden_states.dim() != 2 or hidden_states.shape[0] != self.batch * state.block: + return self.inner(hidden_states, lm_head, attn_metadata, return_context_logits) + logits = lm_head.apply_linear(hidden_states, None) + buf = state.buffer(self.batch) + buf[:, :, : state.shard].copy_(logits.view(self.batch, state.block, state.shard)) + return logits + + +def _head_is_sharded(lm_head: nn.Module, vocab: int, tp: int) -> bool: + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + return ( + tp > 1 + and getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and (getattr(lm_head, "padding_size", 0) == 0) + and (getattr(lm_head, "bias", None) is None) + and (lm_head.out_features * tp == vocab) + and (lm_head.weight.dtype == torch.bfloat16) + ) + + +class S4WorkerMixin: + """Mixed into the target's ``DSparkWorker`` subclass (before it in the bases).""" + + _s4: Optional[_S4State] = None + _s4_step: Optional[_S4State] = None + _s4_tokens: Optional[torch.Tensor] = None + _s4_step_meta: Optional[Tuple[int, int, object]] = None + _s4_draft: Optional[nn.Module] = None + _s4_served: int = 0 + _s4_gen_steps: int = 0 + _s4_ll_logged: bool = False + _s4_step_attn = None + _tailbk_rewound: bool = False + _tailbk_h_logged: bool = False + _tailbk_s4l_logged: bool = False + _tailbk_c_logged: bool = False + + def prebuild_s4(self, lm_head: nn.Module, draft_model) -> bool: + """Set up the one-launch chain for this worker; False (and nothing set up) when it does not apply.""" + self._s4 = None + if disabled(): + logger.info(f"M3.1 DSpark S4: off ({DISABLE_ENV} set); the in-tree Markov chain runs") + return False + if not getattr(draft_model, "has_markov_head", False): + return False + w1 = draft_model.markov_w1 + w2 = draft_model.markov_w2 + mapping = self.mapping + tp = mapping.tp_size + vocab = w1.shape[0] + if ( + w1.dtype != torch.bfloat16 + or w2.dtype != torch.bfloat16 + or w1.shape[1] != MARKOV_RANK + or (tuple(w2.shape) != tuple(w1.shape)) + or mapping.enable_attention_dp + or (not _head_is_sharded(lm_head, vocab, tp)) + ): + return False + block = int(self.max_draft_len) + shard = vocab // tp + rows = chain.padded_rows(shard, MAX_FAST_BATCH, w1.device) + if not self._s4_ranks_agree(rows != 0 and block <= chain.MAX_BLOCK): + return False + offset = mapping.tp_rank * shard + w2s = chain.pad_w2_shard(w2[offset : offset + shard].contiguous(), rows) + state = _S4State( + mapping, rows, shard, offset, w1, w2s, block, chain.workspace_args(mapping) + ) + for batch in range(1, MAX_FAST_BATCH + 1): + state.bufs[batch, block] = chain.pad_logits_buffer(batch, block, rows, w1.device) + s4b8 = chain.enable_batch1_variant(not m31_disabled("S4B8")) + markov_ll = chain.enable_ll_exchange(MARKOV_LL) + tailbk_c = chain.enable_cached_inputs(TAILBK_SPEC) + self._s4_fp32 = chain.enable_fp32_sum(not m31_disabled("S4_FP32SUM")) + self._s4_logits_on = not m31_disabled("S4_LOGITS") + chain.precompile(mapping, rows, w1, w2s, offset, blocks=(block,), max_batch=MAX_FAST_BATCH) + self._s4 = state + self._s4_draft = draft_model + atexit.register(self._s4_log_count) + logger.info( + ( + "M3.1 DSpark S4: one-launch Markov chain set up on rank " + f"{mapping.tp_rank}" + "/" + f"{tp}" + " (shard " + f"{shard}" + " rows padded to " + f"{rows}" + ", K=" + f"{block}" + ", batches 1.." + f"{MAX_FAST_BATCH}" + "; S4B8 batch-1 kernel " + f"{('on' if s4b8 else 'off')}" + "; MARKOV_LL " + f"{('on' if markov_ll and s4b8 else 'off')}" + "; TAILBK_SPEC cached inputs " + f"{('on' if tailbk_c else 'off')}" + "; S4-logits " + f"{('on' if self._s4_logits_on else 'off')}" + "; fp32 sum " + f"{('on' if self._s4_fp32 else 'off')}" + ")" + ) + ) + return True + + def _s4_ranks_agree(self, local: bool) -> bool: + """``local`` if every rank of the TP group decided the same; raises when they differ (a rank going on into + the collective workspace alone would hang its peers).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + comm = _get_mnnvl_workspace_comm(self.mapping) + all_on = _mnnvl_workspace_all_succeeded(comm, local) + all_off = _mnnvl_workspace_all_succeeded(comm, not local) + if not (all_on or all_off): + raise RuntimeError( + "m31 DSpark S4: the TP ranks disagree on setting up the one-launch Markov chain" + ) + return all_on + + def _s4_serves(self, num_gens: int, spec_metadata, draft_model) -> bool: + return self._s4_mode(num_gens, spec_metadata, draft_model) is not None + + def _s4_mode(self, num_gens: int, spec_metadata, draft_model): + """ "tokens" (greedy drafting: the chain's tokens are the draft), "logits" (advanced draft sampling with + S4-logits on: the chain's corrected logits go to the stock sampler), or None (the in-tree chain runs).""" + if not self._s4_common(num_gens, draft_model): + return None + if not getattr(spec_metadata, "wants_advanced_draft_sampling", False): + return "tokens" + return "logits" if getattr(self, "_s4_logits_on", False) else None + + def _s4_common(self, num_gens: int, draft_model) -> bool: + return ( + self._s4 is not None + and 1 <= num_gens <= MAX_FAST_BATCH + and (getattr(self, "guided_decoder", None) is None) + and (getattr(self, "_d2t", None) is None) + and (getattr(draft_model, "lilicorr", None) is None) + and (not getattr(draft_model, "has_candidate_selector", False)) + ) + + def _host_step_hook(self, spec_metadata) -> None: + """The engagement counter (host only, no CUDA): called by the spec metadata's prepare() on every iteration, + outside any captured graph, it evaluates the same eligibility the captured step was routed by and counts + the generation steps the chain serves. One line per rank every S4_COUNT_LOG_EVERY generation steps and at + exit: "M3.1 DSpark S4: served / generation steps". The counts start at server start, so they include + warmup and padding steps: read engagement as the difference of two lines logged after the server is ready.""" + if self._s4 is None: + return + num_gens = int(getattr(spec_metadata, "num_generations", 0) or 0) + if num_gens <= 0: + return + self._s4_gen_steps += 1 + self._s4_served += int(self._s4_serves(num_gens, spec_metadata, self._s4_draft)) + if self._s4_gen_steps % S4_COUNT_LOG_EVERY == 0: + self._s4_log_count() + + def _s4_log_count(self) -> None: + if self._s4 is not None and self._s4_gen_steps: + logger.info( + ( + "M3.1 DSpark S4: served " + f"{self._s4_served}" + "/" + f"{self._s4_gen_steps}" + " generation steps on rank " + f"{self.mapping.tp_rank}" + ) + ) + + def _forward_impl( + self, + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + resource_manager=None, + ): + num_gens = attn_metadata.num_seqs - attn_metadata.num_contexts + mode = self._s4_mode(num_gens, spec_metadata, draft_model) + served = mode is not None + if not served: + return super()._forward_impl( + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + resource_manager, + ) + inner = draft_model.logits_processor + draft_model.logits_processor = _ShardedDraftLogits(inner, self._s4, num_gens) + self._s4_step = self._s4 + self._s4_step_mode = mode + self._s4_step_meta = (attn_metadata.num_contexts, attn_metadata.num_seqs, spec_metadata) + self._s4_step_attn = attn_metadata + try: + return super()._forward_impl( + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + resource_manager, + ) + finally: + draft_model.logits_processor = inner + self._s4_step = None + self._s4_step_mode = None + self._s4_tokens = None + self._s4_step_meta = None + self._s4_step_attn = None + self._tailbk_rewound = False + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + state = self._s4_step + if state is not None and gen_logits.dim() == 3 and (gen_logits.shape[-1] == state.shard): + base = state.buffer(gen_logits.shape[0]) + if ( + chain.supports(base, state.w1, state.w2s) + and getattr(self, "_s4_step_mode", None) == "logits" + ): + first_prev = inputs["first_prev_tokens"].long() + tokens = self._s4_chain(state, base, first_prev, spec_metadata) + self._s4_log_ll(base.shape[0]) + biased = _biased_shard_logits( + gen_logits, + first_prev, + tokens, + state.w1, + state.w2s[: state.shard], + self._s4_fp32, + ) + self._s4l_log(gen_logits, first_prev, tokens, state.w1) + return biased + if chain.supports(base, state.w1, state.w2s): + tokens = self._s4_chain( + state, base, inputs["first_prev_tokens"].long(), spec_metadata + ) + self._s4_log_ll(base.shape[0]) + self._s4_tokens = tokens + return gen_logits + return super()._refine_block_logits(draft_model, gen_logits, inputs, spec_metadata) + + def _s4_chain( + self, state: _S4State, base: torch.Tensor, first_prev: torch.Tensor, spec_metadata + ): + """The chain launch; under TAILBK_SPEC (site H) it also applies the step's deferred KV-length rewind when + the stock _apply_kv_rewind_after_draft would (a real step: not the eager warmup of a graph batch, whose + kv_lens is restored wholesale), and marks it done for that method.""" + rewind = self._tailbk_rewind_args(base.shape[0], spec_metadata) + if rewind is None: + tokens = chain.dspark_markov_chain( + base, first_prev, state.w1, state.w2s, state.offset, **state.ws + ) + self._tailbk_c_log() + return tokens + kv_lens, amount, first = rewind + tokens = chain.dspark_markov_chain_rewind( + base, first_prev, state.w1, state.w2s, state.offset, kv_lens, amount, first, **state.ws + ) + self._tailbk_rewound = True + self._tailbk_c_log() + if not self._tailbk_h_logged: + self._tailbk_h_logged = True + logger.info( + ( + "[M3 dispatch] TAILBK_SPEC H: KV-length rewind in the Markov chai" + "n launch on rank " + f"{self.mapping.tp_rank}" + ) + ) + return tokens + + def _tailbk_c_log(self) -> None: + """The TAILBK_SPEC C engagement line: once per rank, after the first chain call that took its accepted inputs + from the precompile cache (host only; a captured step logs it while it is captured).""" + if self._tailbk_c_logged or not getattr(chain, "cached_inputs_taken", lambda: False)(): + return + self._tailbk_c_logged = True + logger.info( + ( + "[M3 dispatch] TAILBK_SPEC C: chain accepted inputs from precompi" + "le cache (taken) on rank " + f"{self.mapping.tp_rank}" + ) + ) + + def _tailbk_rewind_args(self, batch: int, spec_metadata): + """(kv_lens, rewind amounts, first row) for the chain to apply, or None (the stock rewind runs).""" + if not TAILBK_SPEC or not getattr(self, "_kv_rewind_pending", False): + return None + amount = getattr(self, "_kv_rewind_amount", None) + attn = self._s4_step_attn + if amount is None or attn is None or (not hasattr(attn, "kv_lens_cuda")): + return None + if spec_metadata.is_cuda_graph and (not torch.cuda.is_current_stream_capturing()): + return None + nc, bs = (self._kv_rewind_nc, self._kv_rewind_bs) + if bs - nc != batch or amount.numel() != batch: + return None + kv_lens = attn.kv_lens_cuda + if not chain.supports_rewind(kv_lens, amount, nc): + return None + return (kv_lens, amount, nc) + + def _apply_kv_rewind_after_draft(self, attn_metadata, spec_metadata): + if self._tailbk_rewound: + self._tailbk_rewound = False + self._kv_rewind_pending = False + return + super()._apply_kv_rewind_after_draft(attn_metadata, spec_metadata) + + def _s4l_log(self, shard_logits, first_prev, tokens, w1) -> None: + """The TAILBK_SPEC S4L engagement line: once per rank, on the first step whose biased logits took the + two-launch glue (host only; a captured step logs it while it is captured).""" + if self._tailbk_s4l_logged or not (self._s4_fp32 and S4_LOGITS_BATCHED and TAILBK_SPEC): + return + if markov_bias.supports(shard_logits, first_prev, tokens, w1): + self._tailbk_s4l_logged = True + logger.info( + ( + "[M3 dispatch] TAILBK_SPEC S4L: biased block logits glue in two l" + "aunches on rank " + f"{self.mapping.tp_rank}" + ) + ) + + def _s4_log_ll(self, batch: int) -> None: + """The MARKOV_LL engagement line: once per rank, on the first served step whose chain call took the LL + exchange kernel (host only; a captured step logs it while it is captured).""" + if not self._s4_ll_logged and chain.ll_exchange_engaged(batch): + self._s4_ll_logged = True + logger.info( + ( + "[M3 dispatch] MARKOV_LL: k3_markov LL flag-in-payload exchange (" + "batch 1) on rank " + f"{self.mapping.tp_rank}" + ) + ) + + def sample_draft_tokens(self, logits, spec_metadata, batch_size, **kwargs): + tokens = self._s4_tokens + if tokens is not None and logits.dim() == 3 and (tokens.shape == logits.shape[:2]): + self._s4_tokens = None + return tokens if tokens.dtype == torch.int32 else tokens.to(torch.int32) + return super().sample_draft_tokens(logits, spec_metadata, batch_size, **kwargs) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/m31_switches.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/m31_switches.py new file mode 100644 index 000000000000..51f966c595fd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/m31_switches.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Kill switches of this MiniMax-M3.1 target's default-on items (each target keeps its own copy: targets share +only the catalog). + +Every item is on by default; ``TRTLLM_M31_DISABLE_=1`` turns item ```` off and restores the path it replaced. +The environment is read once, at import, so every rank and every step of a process sees the same setting (a step must +not change its graph mid-run). Each item lists its id and what the switch restores where it reads it. + +Values (case-insensitive, surrounding blanks ignored): ``1`` / ``true`` / ``on`` / ``yes`` set a switch; empty / ``0`` / +``false`` / ``off`` / ``no`` leave it unset; anything else raises at import, naming the variable, so a typo fails loudly +instead of silently turning an item on or off. +""" + +import os +from typing import FrozenSet + +_PREFIX = "TRTLLM_M31_DISABLE_" +_SET = frozenset(("1", "true", "on", "yes")) +_UNSET = frozenset(("", "0", "false", "off", "no")) + + +def _is_set(name: str, value: str) -> bool: + """Whether the switch variable ``name`` is set to ``value``; raises on a value of neither kind.""" + word = value.strip().lower() + if word in _SET: + return True + if word in _UNSET: + return False + raise ValueError( + f"{name}={value!r}: expected 1 / true / on / yes (set) or 0 / false / off / no / empty (unset)" + ) + + +# The item ids switched off in this process, read once at import. +_DISABLED: FrozenSet[str] = frozenset( + name[len(_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_PREFIX) and _is_set(name, value) +) + + +def m31_disabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_DISABLE_`` turned the item off for this process.""" + return item_id in _DISABLED + + +# Items that are OFF by default (rule 1b without a passing gate c, or a failed gate c): ``TRTLLM_M31_ENABLE_=1`` +# opts in; the item's ``TRTLLM_M31_DISABLE_`` still wins. +_ENABLE_PREFIX = "TRTLLM_M31_ENABLE_" + +_ENABLED: FrozenSet[str] = frozenset( + name[len(_ENABLE_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_ENABLE_PREFIX) and _is_set(name, value) +) + + +def m31_enabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_ENABLE_`` opted a default-off item in for this process.""" + return item_id in _ENABLED diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/modeling.py new file mode 100644 index 000000000000..2ec5fe79f89b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/modeling.py @@ -0,0 +1,5107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP8 implementation.""" + +from __future__ import annotations + +import copy +import dataclasses +import os +import threading +import time +from dataclasses import dataclass +from types import MappingProxyType +from typing import ( + Any, + Callable, + Dict, + FrozenSet, + Hashable, + Iterable, + List, + Mapping, + Optional, + Set, + Tuple, + Union, +) + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch._experimental.modeling_v2._router_index import ( + ModelingV2Mode, + resolved_mode, + step_contract_enabled, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_rmsnorm import ( + flashinfer_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.quantization.fp4_quantize import ( + fp4_quantize, +) +from tensorrt_llm._torch.attention.backends import AttentionMetadata +from tensorrt_llm._torch.attention.backends.fmha.msa_prefill import _aligned_nvfp4_dequant_scales +from tensorrt_llm._torch.attention.backends.interface import PositionalEmbeddingParams, RopeParams +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import MiniMaxM3MsaSparseAttention +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + MSA_REQUIRED_TOPK, + msa_paged_kv, + require_msa_module, +) +from tensorrt_llm._torch.attention.backends.utils import create_attention +from tensorrt_llm._torch.distributed import AllReduce, AllReduceFusionOp, AllReduceParams +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + _ensure_draft_vocab_size, + _get_requested_draft_moe_backend, + _set_draft_kv_cache_quant_algo, + _slice_spec_position_ids, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.modules.embedding import LMHead +from tensorrt_llm._torch.modules.linear import ( + MXFP8LinearMethod, + TensorParallelMode, + flashinfer_mxfp8_autotune_active, + flashinfer_mxfp8_decode_graph_capture_active, +) +from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor +from tensorrt_llm._torch.modules.multi_stream_utils import maybe_execute_in_parallel +from tensorrt_llm._torch.moe.fused_moe.routing import RoutingMethodType +from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph import is_in_breakable_cuda_graph +from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import MambaHybridCacheManager +from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy, RuntimeHooks +from tensorrt_llm._torch.pyexecutor.sampler.ops.flashinfer import ( + set_softmax_override, + set_top_p_renorm_override, +) +from tensorrt_llm._torch.speculative import SpecMetadata +from tensorrt_llm._torch.speculative.eagle3 import Eagle3OneModelWorker +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + set_chain_speculative_sampling_override, + should_use_separate_draft_kv_cache, +) +from tensorrt_llm._torch.utils import ActType_TrtllmGen +from tensorrt_llm._utils import prefer_pinned +from tensorrt_llm.functional import AllReduceStrategy, PositionEmbeddingType +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.mode import QuantAlgo + +from . import dspark as _dspark +from . import weights as _weights +from .m31_switches import m31_disabled, m31_enabled +from .ops import gqamix_sparse_decode as _gqamix_kernel +from .ops import m3_bp_tail, m3_mc_allreduce, m3_prequant, m3_spec_tail +from .ops import m3_draft_step as m3_draft_step_kernel +from .ops import m3_fused_score_select as _fused_score_select_kernel +from .ops import m3_index_score as _index_score_kernel +from .ops import m3_sampled_accept as m3_sampled_accept_kernel +from .ops import m3_select_long_dsl as _select_long_dsl_kernel +from .ops import m3_shard_accept as m3_shard_accept_kernel +from .ops import msa_kv_update as _msa_kv_update_kernel +from .ops.activation.m3_silu_and_mul import m3_silu_and_mul +from .ops.activation.m3_swq import m3_swq +from .ops.attention.m3_dense_decode import m3_dense_decode +from .ops.attention.m3_draft_attention import m3_draft_attention +from .ops.attention.m3_fp8_indexer_qk_norm_rope import m3_fp8_indexer_qk_norm_rope +from .ops.attention.m3_fused_score_select import ( + MAX_DECODE_QUERY_LEN as FUSED_SCORE_SELECT_MAX_QUERY_LEN, +) +from .ops.attention.m3_fused_score_select import ( + MAX_REQUESTS as FUSED_SCORE_SELECT_KERNEL_MAX_REQUESTS, +) +from .ops.attention.m3_fused_score_select import m3_fused_score_select +from .ops.attention.m3_index_score import SPLIT_K as INDEX_SCORE_SPLIT_K +from .ops.attention.m3_index_score import m3_index_score +from .ops.attention.m3_msa_attention import m3_msa_attention +from .ops.attention.m3_producer import m3_producer +from .ops.attention.m3_qk_norm_rope_fp8 import m3_qk_norm_rope_fp8 +from .ops.attention.m3_qkv_insert import m3_qkv_insert +from .ops.attention.m3_select_blocks import m3_select_blocks +from .ops.attention.m3_select_long import MAX_BLOCKS as SELECT_LONG_MAX_BLOCKS +from .ops.attention.m3_select_long import m3_select_long +from .ops.attention.m3_select_long_dsl import m3_select_long_dsl +from .ops.attention.m3_sparse_decode import m3_sparse_decode +from .ops.attention.m3_sparse_decode_lc8 import m3_sparse_decode_lc8 +from .ops.comm.m3_allreduce import m3_allreduce +from .ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm +from .ops.comm.m3_mc_finalize_allreduce_norm import m3_mc_finalize_allreduce_norm +from .ops.embedding.m3_embedding import m3_embedding +from .ops.gemm.m3_bf16_linear import m3_bf16_linear +from .ops.gemm.m3_draft_lowm import TACTICS as DRAFT_LOWM_TACTICS +from .ops.gemm.m3_draft_lowm import m3_draft_lowm, tactic_for +from .ops.gemm.m3_fp32_linear import m3_fp32_linear +from .ops.gemm.m3_mx_gemv import DENSE_DOWN, DENSE_GATE_UP, O_PROJ, QKV, QKV_SPARSE, m3_mx_gemv +from .ops.gemm.m3_mx_gemv_bf16 import m3_mx_gemv_bf16 +from .ops.gemm.m3_mxfp8_gemm import m3_mxfp8_gemm +from .ops.gemm.m3_mxfp8_linear import m3_mxfp8_linear +from .ops.long_select import long_select_blocks +from .ops.m3_router_gemv import m3_router_gemv as _router_gemv +from .ops.m3_router_gemv import m3_router_gemv_fp4 as _router_gemv_fp4 +from .ops.m3_router_gemv import m3_router_tc as _router_tc +from .ops.moe.m3_moe_engine import M3MoeState, ar_push_desc, m3_moe_engine +from .ops.moe.m3_router_gemv import m3_router_gemv +from .ops.moe.m3_router_gemv_fp4 import m3_router_gemv_fp4 +from .ops.moe.m3_router_tc import m3_router_tc +from .ops.mx_gemv import op as mx_gemv +from .ops.norm.m3_embedding_firstnorm import DecodeEmbedding, m3_embedding_firstnorm +from .ops.sampling import m31_chain_spec_mc as _chain_spec_mc +from .ops.sampling import m31_softmax_mc as _softmax_mc +from .ops.sampling import m31_topp_det as _topp_det +from .ops.spec.m3_draft_input import m3_draft_input +from .ops.spec.m3_draft_step import MAX_TOKENS as DRAFT_STEP_KERNEL_MAX_TOKENS +from .ops.spec.m3_draft_step import m3_draft_step +from .ops.spec.m3_force_vec import m3_force_vec +from .ops.spec.m3_shard_accept import m3_shard_accept + +_SM = (10, 3) +MSA_PROXY_PREBUILD = not m31_disabled("PROXYPB") +CTXNORMQ = not m31_disabled("CTXNORMQ") +CTXNORMQ_MIN_ROWS = 1024 +FUSED_CAPTURE = not m31_disabled("FUSED_CAPTURE") +AR64AUTO = not m31_disabled("AR64AUTO") +TSEL = not m31_disabled("TSEL") +TSEL_MIN_ROWS = 256 +TSEL_MAX_BLOCKS = 5856 +TSEL_CHECK = os.environ.get("TRTLLM_M31_CHECK_TSEL") == "1" +GQAMIX = not m31_disabled("GQAMIX") +DSFOLD = not m31_disabled("DSFOLD") +BOUNDARY_PACK = DSFOLD and (not m31_disabled("BOUNDARY_PACK")) +BOUNDARY_PACK_MAX_ROWS = 1 +_BOUNDARY_PACK_MARK = "m31_boundary_pack" +QKV_PRODUCER_FOLD = not m31_disabled("QKVPROD") +QKV_PRODUCER_FOLD_PATH = "m3_qkv_producer_fold" +_QKV_FOLD_ENGAGED: Set[int] = set() +DECODE_MAX_TOKENS = 48 if m31_disabled("R1") else 64 +# Fused-layer dispatch scope; keep the larger prebuild/workspace capacity above. +FUSED_DECODE_MAX_TOKENS = 8 +SWQ_SHARED_DOWN = (6144, 384) +SWQ_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + {} if m31_disabled("SWQ") else {SWQ_SHARED_DOWN: frozenset(range(9, 49))} +) +_SWQ_LOGGED: set = set() +TOPP_DET = not m31_disabled("TOPP_DET") +_TOPP_DET_STATE: dict = {} + + +def _install_topp_det() -> None: + """Once per process: install ``_topp_det.top_p_renorm_det`` as the sampler's top-p renorm override when TOPP_DET + is on and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _TOPP_DET_STATE: + return + if not TOPP_DET: + state = "off (TRTLLM_M31_DISABLE_TOPP_DET=1); stock FlashInfer top_p_renorm_probs" + elif _topp_det.topp_det_op() is None: + state = "not taken: the op is not built; stock FlashInfer top_p_renorm_probs" + else: + set_top_p_renorm_override(_topp_det.top_p_renorm_det) + ctas = _topp_det.ctas_per_row(_topp_det.MAX_ROWS, _topp_det.sm_count(torch.device("cuda"))) + state = f"engaged (ctas_per_row={ctas}, rows <= {_topp_det.MAX_ROWS})" + _TOPP_DET_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01 TOPP_DET det top-p renorm: {state}") + + +def _prebuild_topp_det() -> None: + """The override's persistent workspace on this rank's device, allocated before warmup and any graph capture.""" + if _TOPP_DET_STATE.get("installed"): + _topp_det.prebuild() + + +SOFTMAX_MC = not m31_disabled("SOFTMAX_MC") +_SOFTMAX_MC_STATE: dict = {} + + +def _install_softmax_mc() -> None: + """Once per process: install ``_softmax_mc.softmax_mc`` as the sampler's softmax override when SOFTMAX_MC is on + and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _SOFTMAX_MC_STATE: + return + if not SOFTMAX_MC: + state = "off (TRTLLM_M31_DISABLE_SOFTMAX_MC=1); stock FlashInfer softmax" + elif _softmax_mc.softmax_mc_op() is None: + state = "not taken: the op is not built; stock FlashInfer softmax" + else: + set_softmax_override(_softmax_mc.softmax_mc) + state = f"engaged (write_ctas={_softmax_mc.WRITE_CTAS}, rows <= {_softmax_mc.MAX_ROWS})" + _SOFTMAX_MC_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01-SM softmax multi-CTA write: {state}") + + +def _prebuild_softmax_mc() -> None: + """The override's persistent partials workspace on this rank's device, allocated before warmup and any graph + capture.""" + if _SOFTMAX_MC_STATE.get("installed"): + _softmax_mc.prebuild() + + +REJ_MC = not m31_disabled("REJ_MC") +_REJ_MC_STATE: dict = {} + + +def _install_rej_mc() -> None: + """Once per process: install ``_chain_spec_mc.chain_spec_mc`` as the rejection sampler's override when REJ_MC is + on and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _REJ_MC_STATE: + return + if not REJ_MC: + state = "off (TRTLLM_M31_DISABLE_REJ_MC=1); stock trtllm::chain_speculative_sampling" + elif _chain_spec_mc.chain_spec_mc_op() is None: + state = "not taken: the op is not built; stock trtllm::chain_speculative_sampling" + else: + set_chain_speculative_sampling_override(_chain_spec_mc.chain_spec_mc) + ctas = _chain_spec_mc.ctas_per_row( + _chain_spec_mc.MAX_BATCH, _chain_spec_mc.sm_count(torch.device("cuda")) + ) + state = f"engaged (ctas_per_row={ctas}, batch <= {_chain_spec_mc.MAX_BATCH})" + _REJ_MC_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01-REJ chain speculative sampling multi-CTA: {state}") + + +def _prebuild_rej_mc() -> None: + """The override's persistent workspace on this rank's device, allocated before warmup and any graph capture.""" + if _REJ_MC_STATE.get("installed"): + _chain_spec_mc.prebuild() + + +_HOSTTAIL_OFF = m31_disabled("HOSTTAIL") +HOSTTAIL_D2H_SIDE = not (_HOSTTAIL_OFF or m31_disabled("HOST_D2H_SIDE")) +HOSTTAIL_SMALL_H2D = not (_HOSTTAIL_OFF or m31_disabled("SMALL_H2D")) +HOSTTAIL_BATCH_CHANGED_ONLY = not (_HOSTTAIL_OFF or m31_disabled("BATCH_CHANGED_ONLY")) +HOSTTAIL_SKIP_ACCEPTED_UPLOAD = not (_HOSTTAIL_OFF or m31_disabled("SKIP_ACCEPTED_UPLOAD")) +HOSTTAIL_PHILOX_PACK = not (_HOSTTAIL_OFF or m31_disabled("PHILOX_PACK")) +HOSTTAIL_KV_LENS_ONCE = not (_HOSTTAIL_OFF or m31_disabled("KV_LENS_ONCE")) +SLOT_IDS_CHANGED = not m31_disabled("SLOT_IDS_CHANGED") +PACK_PSCATTER = not m31_disabled("PACK_PSCATTER") +PACK_IN_GRAPH = PACK_PSCATTER and (not m31_disabled("PACK_IN_GRAPH")) +PACK_EARLY_COPY = PACK_IN_GRAPH and (not m31_disabled("PACK_EARLY_COPY")) +H2D_CHANGED_SPAN = not m31_disabled("H2D_CHANGED_SPAN") +BLOCKOFF_GUARD = not m31_disabled("BLOCKOFF_GUARD") +TAILBK = not m31_disabled("TAILBK") +_TAILBK_LOGGED: list = [] + + +def _log_tailbk(worker) -> None: + """The TAILBK engagement line, once per process when the model arms it: the items its worker can take (the DSpark + worker all five; another worker's accept only A1). Nothing when TAILBK is off or without speculation.""" + if not TAILBK or worker is None or _TAILBK_LOGGED: + return + _TAILBK_LOGGED.append(True) + items = "R1 B1 Z1 S1 A1" if isinstance(worker, _dspark.M3DSparkWorker) else "A1" + logger.info(f"[M3 host] tail bookkeeping: {items}") + + +@dataclass(frozen=True) +class DecodePlan: + """The layer-boundary collectives of one decode token bucket (GB300 measurements in the comments).""" + + name: str + max_tokens: int + attn_mc_ar: bool + moe_mc_ar: bool + qkv_prequant: bool + + +DECODE_PLANS = ( + DecodePlan("tiny", 8, attn_mc_ar=True, moe_mc_ar=True, qkv_prequant=True), + DecodePlan("small", 16, attn_mc_ar=True, moe_mc_ar=True, qkv_prequant=True), + DecodePlan("verify", DECODE_MAX_TOKENS, attn_mc_ar=True, moe_mc_ar=True, qkv_prequant=True), +) +DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH: Mapping[int, int] = MappingProxyType({1: 16384}) +DRAFT_DECODE_MAX_SEQ_LEN_BY_BATCH: Mapping[int, int] = MappingProxyType({1: 16384}) +PRODUCER_EARLY_TRIGGER = True +PRODUCER_SHORTFIX = not m31_disabled("PRODUCER_EARLY_TRIGGER_SHORTFIX") +_PRODUCER_SHORTFIX_LOGGED: list = [] + + +def _producer_early_trigger(step) -> bool: + """Whether this step's sparse-layer producers release their dependents early (see PRODUCER_SHORTFIX).""" + if not PRODUCER_EARLY_TRIGGER: + return False + no_selector = bool(step.get("msa_short_ok")) and step.get("msa_short_table") is not None + no_selector = no_selector or False + if not (PRODUCER_SHORTFIX and no_selector): + return True + if not _PRODUCER_SHORTFIX_LOGGED: + _PRODUCER_SHORTFIX_LOGGED.append(True) + logger.info( + ( + "[M3 dispatch] PRODUCER_SHORTFIX: producer early trigger off on s" + "teps without a selector (sparse decode loads q before its grid w" + "ait)" + ) + ) + return False + + +EMBED_FIRSTNORM_MAX_TOKENS = 8 +ATTN_PUSH_MAX_TOKENS = 16 +AR_PUSH_MODE = 2 +MOE_ENGINE_MAX_TOKENS = 8 +MOE_PUSH_MAX_TOKENS = MOE_ENGINE_MAX_TOKENS +MC_FINALIZE_MIN_TOKENS = MOE_ENGINE_MAX_TOKENS + 1 +ROUTER_GEMV_MAX_TOKENS = 1 +ROUTER_TC_MAX_TOKENS = 48 +SHARD_ACCEPT_DRAFT_LEN = 3 +SHARD_ACCEPT_MAX_REQUESTS = min(DECODE_MAX_TOKENS, m3_shard_accept_kernel.SHARD_ROWS) // ( + SHARD_ACCEPT_DRAFT_LEN + 1 +) +FC_EARLY_MAX_ROWS = 0 +DRAFT_STEP_DRAFT_LEN = SHARD_ACCEPT_DRAFT_LEN +DRAFT_STEP_MAX_REQUESTS = DRAFT_STEP_KERNEL_MAX_TOKENS +SPARSE_DECODE_MQ_MIN_TOKENS = 17 +FUSED_SCORE_SELECT_MAX_REQUESTS = FUSED_SCORE_SELECT_KERNEL_MAX_REQUESTS +LONG_SELECT_MIN_BLOCKS = 129 +LONG_SELECT_MAX_BLOCKS = SELECT_LONG_MAX_BLOCKS +LONG_SELECT_KEY_WIDTHS = (2, 4, 9) +SELECT_EARLY_MAX_ROWS = 8 +SELECT_LONG_DSL = not m31_disabled("SELECT_DSL") +LONG_SELECT_DSL_KEY_WIDTHS = (4, 11, 24) +_SELECT_DSL_LOGGED: set = set() +LC8_PREWAIT_MAX_ROWS = 8 +INDEX_SCORE_EARLY_MAX_REQUESTS = 0 if m31_disabled("INDEX_EARLY") else 1 +_INDEX_EARLY_LOGGED: set = set() + + +def _index_score_early(rows: int, rank: int) -> bool: + """Whether a decode step of ``rows`` requests runs the index scorer's early-issue schedule. The first time it does, + logs the EARLY-ISSUE engagement line once per rank (host side: the graph capture runs this Python too).""" + early = rows <= INDEX_SCORE_EARLY_MAX_REQUESTS + if early and rank not in _INDEX_EARLY_LOGGED: + _INDEX_EARLY_LOGGED.add(rank) + logger.info( + f"[M3 dispatch] INDEX_EARLY index-scorer early-issue: engaged (rows={rows}) on rank {rank}" + ) + return early + + +_ALL_ROWS = frozenset(range(1, min(DECODE_MAX_TOKENS, mx_gemv.MAX_TOKENS) + 1)) +MX_GEMV_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = { + QKV_SPARSE: _ALL_ROWS, + QKV: _ALL_ROWS, + O_PROJ: _ALL_ROWS, + DENSE_GATE_UP: _ALL_ROWS, + DENSE_DOWN: _ALL_ROWS, +} +MX_GEMV_BF16_MAX_TOKENS: Dict[Tuple[int, int], int] = { + QKV_SPARSE: 16, + QKV: 16, + O_PROJ: 48, + DENSE_GATE_UP: 16, + DENSE_DOWN: 32, +} +DRAFT_LOWM_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = { + nk: frozenset(range(1, table[-1][0] + 1)) for nk, table in DRAFT_LOWM_TACTICS.items() +} +Key = Hashable + + +@dataclass(frozen=True) +class RowRange: + """The shapes one fast path takes, and the catalog entry whose cells certify each of them. + + ``keys`` are the covered shapes: row counts for most paths, (query rows per request, requests) for the attention + kernels, a variant name for thresholds. ``certifies`` maps a cell's spec of ``entry`` to the keys that cell + drives (empty for a cell of another variant). + """ + + path: str + keys: FrozenSet[Key] + entry: OpWrapper + certifies: Callable[[Mapping[str, Any]], Iterable[Key]] + + def covers(self, key: Key) -> bool: + return key in self.keys + + def certified(self) -> FrozenSet[Key]: + return frozenset((k for cell in self.entry.CELLS for k in self.certifies(cell.spec))) + + +def _rows(lo: int, hi: int) -> FrozenSet[int]: + return frozenset(range(lo, hi + 1)) + + +def _spec_rows(field: str, **match: Any) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + """A cell's ``spec[field]`` when the cell's other fields match ``match``.""" + + def certifies(spec: Mapping[str, Any]) -> Iterable[Key]: + if any((spec.get(name) != value for name, value in match.items())): + return () + return (spec[field],) + + return certifies + + +def _const(*keys: Key) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + return lambda spec: keys + + +def _sparse_decode_producer(spec: Mapping[str, Any]) -> Iterable[Key]: + if not spec["identity"]: + return () + rows = spec["decode_query_len"] * len(spec["seq_lens"]) + return ("multi_query" if rows >= SPARSE_DECODE_MQ_MIN_TOKENS else "per_row",) + + +def _long_select_widths(spec: Mapping[str, Any]) -> Iterable[Key]: + """The key widths a cell's rows reach: every width up to its blocks' when rows of mixed lengths are drawn.""" + blocks = spec["num_blocks"] + widest = 2 if blocks <= 2048 else 4 if blocks <= 4096 else 9 + if spec["valid"] == "mixed": + return tuple((w for w in LONG_SELECT_KEY_WIDTHS if w <= widest)) + return (widest,) + + +def _long_select_dsl_widths(spec: Mapping[str, Any]) -> Iterable[Key]: + """The 384-thread select's key widths a cell's rows reach (every width up to its blocks' for mixed rows).""" + blocks = spec["num_blocks"] + widest = 4 if blocks <= 4 * 384 else 11 if blocks <= 11 * 384 else 24 + if spec["valid"] == "mixed": + return tuple((w for w in LONG_SELECT_DSL_KEY_WIDTHS if w <= widest)) + return (widest,) + + +EMBED_FIRSTNORM = RowRange( + "embed_firstnorm", + _rows(1, EMBED_FIRSTNORM_MAX_TOKENS), + m3_embedding_firstnorm, + _spec_rows("rows"), +) +SPARSE_DECODE_PRODUCER = RowRange( + "sparse_decode_producer", + frozenset({"per_row", "multi_query"}), + m3_sparse_decode, + _sparse_decode_producer, +) +SPARSE_DECODE_LC8 = RowRange( + "m3_sparse_decode_lc8", + frozenset({"prewait", "plain"}), + m3_sparse_decode_lc8, + lambda spec: ("prewait" if spec["prewait"] else "plain",), +) +PRODUCER = RowRange("m3_producer", frozenset({"nvfp4_tp8"}), m3_producer, _const("nvfp4_tp8")) +FUSED_SCORE_SELECT = RowRange( + "m3_fused_score_select", + _rows(1, FUSED_SCORE_SELECT_MAX_REQUESTS), + m3_fused_score_select, + _spec_rows("batch"), +) +INDEX_SCORE = RowRange( + "m3_index_score", + frozenset({"prefetch", "early"}) if INDEX_SCORE_EARLY_MAX_REQUESTS else frozenset({"prefetch"}), + m3_index_score, + lambda spec: ("early",) if spec["early"] else ("prefetch",), +) +LONG_SELECT = RowRange( + "long_select", frozenset(LONG_SELECT_KEY_WIDTHS), m3_select_long, _long_select_widths +) +LONG_SELECT_DSL = RowRange( + "long_select_dsl", + frozenset(LONG_SELECT_DSL_KEY_WIDTHS) if SELECT_LONG_DSL else frozenset(), + m3_select_long_dsl, + _long_select_dsl_widths, +) +MC_ALLREDUCE = RowRange( + "mc_allreduce_norm", + _rows(1, min(DECODE_MAX_TOKENS, m3_mc_allreduce.max_rows())), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=0), +) +ATTN_PUSH = RowRange( + "attn_push", + _rows(1, ATTN_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +MOE_PUSH = RowRange( + "moe_push", + _rows(1, MOE_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +MC_FINALIZE = RowRange( + "mc_finalize_allreduce_norm", + _rows(MC_FINALIZE_MIN_TOKENS, min(DECODE_MAX_TOKENS, m3_mc_allreduce.max_rows())), + m3_mc_finalize_allreduce_norm, + _spec_rows("m"), +) +MOE_ENGINE = RowRange( + "m3_moe_engine", _rows(1, MOE_ENGINE_MAX_TOKENS), m3_moe_engine, _spec_rows("tokens") +) +ROUTER_GEMV = RowRange( + "m3_router_gemv", _rows(1, ROUTER_GEMV_MAX_TOKENS), m3_router_gemv, _spec_rows("m") +) +ROUTER_GEMV_FP4 = RowRange( + "m3_router_gemv_fp4", _rows(1, ROUTER_GEMV_MAX_TOKENS), m3_router_gemv_fp4, _spec_rows("m") +) +ROUTER_TC = RowRange( + "m3_router_tc", + _rows(ROUTER_GEMV_MAX_TOKENS + 1, ROUTER_TC_MAX_TOKENS), + m3_router_tc, + _spec_rows("m"), +) +MX_GEMV = RowRange( + "m3_mx_gemv", + frozenset(((nk, m) for nk, rows in MX_GEMV_ROWS.items() for m in rows)), + m3_mx_gemv, + lambda spec: ((spec["nk"], spec["m"]),), +) +MX_GEMV_BF16 = RowRange( + "m3_mx_gemv_bf16", + frozenset(((nk, m) for nk, hi in MX_GEMV_BF16_MAX_TOKENS.items() for m in _rows(1, hi))), + m3_mx_gemv_bf16, + lambda spec: ((spec["nk"], spec["m"]),), +) +DRAFT_LOWM = RowRange( + "m3_draft_lowm", + frozenset(((nk, m) for nk, rows in DRAFT_LOWM_ROWS.items() for m in rows)), + m3_draft_lowm, + lambda spec: ((spec["nk"], spec["m"]),), +) +SHARD_ACCEPT = RowRange( + "m3_shard_accept", + frozenset(((SHARD_ACCEPT_DRAFT_LEN, n) for n in range(1, SHARD_ACCEPT_MAX_REQUESTS + 1))), + m3_shard_accept, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) +DENSE_DECODE_SPLIT = RowRange( + "m3_dense_decode", + frozenset(DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH), + m3_dense_decode, + lambda spec: (len(spec["kv_lens"]),) if spec["capped"] else (), +) +DRAFT_STEP = RowRange( + "m3_draft_step", + frozenset(((DRAFT_STEP_DRAFT_LEN, n) for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1))), + m3_draft_step, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) +DRAFT_INPUT = RowRange( + "m3_draft_input", + frozenset( + ( + rows + for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1) + for rows in (n, n * (DRAFT_STEP_DRAFT_LEN + 1)) + ) + ), + m3_draft_input, + _spec_rows("m"), +) +FAST_PATHS: Tuple[RowRange, ...] = ( + EMBED_FIRSTNORM, + DENSE_DECODE_SPLIT, + SPARSE_DECODE_PRODUCER, + SPARSE_DECODE_LC8, + PRODUCER, + FUSED_SCORE_SELECT, + INDEX_SCORE, + LONG_SELECT, + LONG_SELECT_DSL, + MC_ALLREDUCE, + ATTN_PUSH, + MOE_PUSH, + MC_FINALIZE, + MOE_ENGINE, + ROUTER_GEMV, + ROUTER_GEMV_FP4, + ROUTER_TC, + MX_GEMV, + MX_GEMV_BF16, + DRAFT_LOWM, + SHARD_ACCEPT, + DRAFT_STEP, + DRAFT_INPUT, +) + + +def plan_for(num_tokens: int) -> DecodePlan: + for plan in DECODE_PLANS: + if num_tokens <= plan.max_tokens: + return plan + raise AssertionError(f"no decode plan covers {num_tokens} tokens") + + +def is_decode_step(num_tokens: int) -> bool: + """Whether a pure generation step of ``num_tokens`` rows runs the decode kernels (else generic layers).""" + return num_tokens <= min(FUSED_DECODE_MAX_TOKENS, DECODE_MAX_TOKENS) + + +def moe_path(num_tokens: int) -> Optional[str]: + """The MoE kernel of a decode step: the engine, the unfinalized MoE + finalize all-reduce, or None (generic).""" + if MOE_ENGINE.covers(num_tokens): + return MOE_ENGINE.path + if MC_FINALIZE.covers(num_tokens): + return MC_FINALIZE.path + return None + + +def router_path(num_tokens: int) -> Optional[str]: + """The router kernel when the MoE runs unfused: the fp32 GEMV, the tensor-core router, or None (built-in gate).""" + if ROUTER_GEMV.covers(num_tokens): + return ROUTER_GEMV.path + if ROUTER_TC.covers(num_tokens): + return ROUTER_TC.path + return None + + +MX_GEMV_PATH = MX_GEMV.path +MX_GEMV_BF16_PATH = MX_GEMV_BF16.path +PREQUANT_GEMM_PATH = "flashinfer_mxfp8_gemm" +GENERIC_GEMM_PATH = "mxfp8_gemm" + + +def linear_path(n: int, k: int, num_tokens: int, prequant: bool) -> str: + """The decode-graph MXFP8 GEMM of an [N, K] projection on ``num_tokens`` rows. + + ``prequant``: the producer published the activation in MXFP8. + """ + if prequant: + if num_tokens in MX_GEMV_ROWS.get((n, k), ()): + return MX_GEMV_PATH + return PREQUANT_GEMM_PATH + if num_tokens <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0): + return MX_GEMV_BF16_PATH + return GENERIC_GEMM_PATH + + +class FastPathUnavailable(RuntimeError): + """A covered decode shape could not take its fast path under ``modeling_v2='require'``.""" + + +_lock = threading.Lock() +_require = [False] +_logged: set = set() + + +def set_require(require: bool) -> None: + """Whether a miss raises (``modeling_v2='require'``); set by the target at construction.""" + _require[0] = bool(require) + + +def require_mode() -> bool: + return _require[0] + + +def _log_once(level: Callable[[str], None], path: str, reason: str, message: str) -> None: + key = (path, reason) + with _lock: + if key in _logged: + return + _logged.add(key) + level(message) + + +def missed(path: str, rows: Optional[int], reason: str) -> None: + """``path`` covers this step but cannot run: raise under require, else log once; the caller falls back.""" + at = "" if rows is None else f" at {rows} rows" + if _require[0]: + raise FastPathUnavailable( + f"modeling_v2='require': the MiniMax-M3 decode path {path!r} covers this step{at} but cannot run: {reason}" + ) + _log_once( + logger.warning, + path, + reason, + f"[M3 dispatch] {path}{at}: {reason}; falling back (logged once per reason)", + ) + + +def capture_missed(path: str, rows: Optional[int], what: str) -> None: + """``what`` is built on eager steps but a CUDA-graph capture found it missing (see ``missed``).""" + missed(path, rows, f"{what} is not built and a CUDA graph is being captured") + + +def uncovered(path: str, reason: str) -> None: + """The step is outside ``path``'s covered shapes: log once per (path, reason); the generic path runs.""" + _log_once(logger.info, path, reason, f"[M3 dispatch] {path} not taken: {reason} (logged once)") + + +def long_select_dsl_keys() -> Tuple[Tuple[bool, bool, int], ...]: + """The (PDL launch, early trigger, trigger phase) keys of the 384-thread select a decode step can launch, compiled + before warmup (trigger phase 0: released at its exit when not early, as the 1024-thread kernel); none when + ``LONG_SELECT_DSL`` covers nothing.""" + if not LONG_SELECT_DSL.keys: + return () + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + pdl = get_env_enable_pdl() + keys = [(pdl, False, 0)] + if SELECT_EARLY_MAX_ROWS: + keys.append((pdl, True, 0)) + return tuple(keys) + + +def _note_select_dsl(rank: int) -> None: + """The SELECT_DSL engagement line, once per process and rank (host side: also during graph capture).""" + if rank not in _SELECT_DSL_LOGGED: + _SELECT_DSL_LOGGED.add(rank) + logger.info(f"[M3 dispatch] SELECT_DSL 384-thread long select: engaged on rank {rank}") + + +def reset_logged() -> None: + """Forget which (path, reason) pairs were logged.""" + with _lock: + _logged.clear() + + +MC_AR_MAX_TOKENS = 8 +MOE_FOLD_MIN_TOKENS = 1 +MOE_ENGINE_RESERVED_SMS = 24 +FINAL_HIDDEN_LAYER_ID = -1 +ROUTING_MINIMAX2 = int(RoutingMethodType.MiniMax2) +ACT_SWIGLU = int(ActType_TrtllmGen.SwiGlu) + + +def _is_vl_config(pretrained_config: PretrainedConfig) -> bool: + model_type = getattr(pretrained_config, "model_type", None) + if model_type == "minimax_m3_vl": + return True + if model_type == "minimax_m3": + return False + return any( + ( + "ForConditionalGeneration" in a + for a in getattr(pretrained_config, "architectures", None) or [] + ) + ) + + +def _text_model_config(model_config: ModelConfig) -> ModelConfig: + """The model config with the language model's ``text_config`` as its pretrained config. + + The checkpoint declares the vision-language wrapper; its ``text_config`` arrives as a plain dict (no typed + sub-config is registered), so it is materialized into a ``PretrainedConfig`` with the wrapper's dtype and + embedding tie carried over. A text config is returned unchanged. + """ + if not _is_vl_config(model_config.pretrained_config): + return model_config + cfg = copy.deepcopy(model_config) + text = getattr(cfg.pretrained_config, "text_config", None) + if text is None: + raise ValueError("MiniMax-M3 VL config has no `text_config`") + if isinstance(text, dict): + out = PretrainedConfig() + for k, v in text.items(): + setattr(out, k, v) + text = out + if ( + getattr(text, "torch_dtype", None) is None + and getattr(cfg.pretrained_config, "torch_dtype", None) is not None + ): + text.torch_dtype = cfg.pretrained_config.torch_dtype + if ( + getattr(text, "tie_word_embeddings", None) is None + and getattr(cfg.pretrained_config, "tie_word_embeddings", None) is not None + ): + text.tie_word_embeddings = cfg.pretrained_config.tie_word_embeddings + return dataclasses.replace(cfg, pretrained_config=text) + + +def _validate_runtime(model_config: ModelConfig) -> None: + """The runtime this target is written for. It is set by engine arguments rather than the checkpoint, so routing + sees only its KV cache algorithm; this check at construction refuses every other runtime. + + The MiniMax-M3 MSA backend (it owns the step metadata and the index-K cache), the fused five-way QKV/index + projection or the separate QKV and index projections, an FP8 indexer and an FP8 or NVFP4 main KV cache. + """ + sparse = model_config.sparse_attention_config + if ( + sparse is None + or getattr(sparse, "algorithm", None) != "minimax_m3" + or sparse.implementation != "msa" + ): + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.algorithm='minimax_m3' with implementation='msa'" + ) + if getattr(sparse, "indexer_kv_dtype", None) != "fp8": + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.indexer_kv_dtype='fp8'" + ) + mode = model_config.quant_config.quant_mode + if mode is None or not (mode.has_fp8_kv_cache() or mode.has_fp4_kv_cache()): + raise ValueError("this MiniMax-M3 target requires an FP8 or NVFP4 KV cache") + + +def _sparse_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = cfg.sparse_attention_config["sparse_attention_freq"] + return [i for i, f in enumerate(freq) if int(f) != 0] + + +def _moe_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = getattr(cfg, "moe_layer_freq", None) + if freq is None: + return list(range(cfg.num_hidden_layers)) + return [i for i, f in enumerate(freq) if int(f) != 0] + + +class _Mxfp8GemmState(nn.Module): + """The MXFP8 GEMM backend state the engine's warmup drives. + + The engine finds MXFP8 projections by a module's ``quant_method`` and, for every one, opts it into native GEMM + tuning and (on decode CUDA graphs) the auto-dispatched FlashInfer GEMM, then marks each tuned. Every projection of + this target dispatches identically, so one method object carries that state for all of them; the forward reads it + at each call (``_mx_mode``) and passes the resolved backend to the GEMM entries. + """ + + def __init__(self) -> None: + super().__init__() + self.quant_method = MXFP8LinearMethod() + + +@dataclass(frozen=True) +class MxMode: + """The MXFP8 GEMM dispatch of one call, resolved from ``_Mxfp8GemmState`` and the engine's contexts.""" + + decode_graph: bool + autotune: bool + tuned_quantize_gemm: bool + flashinfer: bool + native_tuning: bool + + +def _mx_mode(state: _Mxfp8GemmState) -> MxMode: + method = state.quant_method + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + autotune = flashinfer_mxfp8_autotune_active() + return MxMode( + decode_graph=decode_graph, + autotune=autotune, + tuned_quantize_gemm=method.tune_decode_graph_backends and decode_graph, + flashinfer=method.backend == "flashinfer" + or ( + method.backend == "auto" + and (autotune or (method._flashinfer_autotuned and decode_graph)) + ), + native_tuning=method.needs_native_autotune, + ) + + +def _mxfp8_linear( + core, x: torch.Tensor, w: torch.Tensor, s: torch.Tensor, mode: MxMode +) -> torch.Tensor: + """One MXFP8 projection on the engine-tuned path: MXFP8 quantize + GEMM, backend per ``mode``.""" + return m3_mxfp8_linear( + x.contiguous(), + w, + s, + core.mx_alpha, + torch.bfloat16, + mode.tuned_quantize_gemm, + mode.autotune, + mode.flashinfer, + mode.native_tuning, + ) + + +def _gemv_variants(n: int, k: int, push: Optional[str], world: int): + """Every (rows, bf16 activation, push (world, bucket tokens, mode)) call the decode dispatch routes to an [N, K] + projection's GEMV: an MXFP8 activation on the rows ``MX_GEMV_ROWS`` gives it, a bf16 one up to + ``MX_GEMV_BF16_MAX_TOKENS``; each pushing when the projection's boundary push covers the rows.""" + push_rows = {"attn": ATTN_PUSH.keys, "moe": MOE_PUSH.keys}.get(push, frozenset()) + variants = [] + for rows in range(1, DECODE_MAX_TOKENS + 1): + for x_bf16, served in ( + (False, rows in MX_GEMV_ROWS.get((n, k), ())), + (True, rows <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0)), + ): + if not served: + continue + variants.append((rows, x_bf16, None)) + if rows in push_rows: + bucket = next((b for b in m3_mc_allreduce.BUCKETS if rows <= b)) + variants.append((rows, x_bf16, (world, bucket, AR_PUSH_MODE))) + return variants + + +def _uncompiled_note() -> str: + key = mx_gemv.last_uncompiled + return "" if key is None else f" (its kernel {key} was not compiled before warmup)" + + +class _DeferredGemv: + """QKVPROD (``QKV_PRODUCER_FOLD``): a sparse layer's QKV + index MXFP8 GEMV, recorded instead of launched. + + ``_linear(..., defer=True)`` returns one on the decode GEMV path of the 12-head packed projection at <= 8 rows with + an MXFP8 hand-off. The producer call site either runs the GEMV with the producer in its epilogue + (``_qkv_producer_fold``) or calls ``materialize`` (the plain GEMV) before anything reads the packed row; + ``placeholder`` (unwritten, the packed shape) stands for that row until then and is never read.""" + + def __init__(self, act, act_scale, weight, weight_scale, use_pdl): + self.placeholder = torch.empty( + act.shape[0], weight.shape[0], dtype=torch.bfloat16, device=act.device + ) + self.act, self.act_scale = (act, act_scale) + self.weight, self.weight_scale = (weight, weight_scale) + self.use_pdl = use_pdl + self.done = False + + def materialize(self) -> torch.Tensor: + """Launch the plain GEMV this record stands for; returns the packed row the projection would have returned.""" + self.done = True + rows, (n, k) = (self.act.shape[0], self.weight.shape) + output = m3_mx_gemv( + self.act, self.act_scale, self.weight, self.weight_scale, True, None, self.use_pdl, None + ) + if output is None: + raise RuntimeError( + f"the deferred [{n}, {k}] GEMV declined at {rows} rows" + _uncompiled_note() + ) + return output + + +def _linear( + core, x: torch.Tensor, wk: str, sk: str, push: Optional[tuple] = None, defer: bool = False +) -> Tuple[Any, bool]: + """One MXFP8 projection ``x @ w[wk]^T``; returns (output, whether its GEMV pushed into ``push``). + + Outside decode CUDA graphs: the engine-tuned MXFP8 quantize + GEMM. Inside them the projection's (N, K) and the + step's rows pick one path (``linear_path``): + + 1. an activation its producer already quantized to MXFP8 (``m3_prequant.SLOT``, taken here): the tcgen05 decode + GEMV on the rows it serves, else the decode-graph MXFP8 GEMM on that activation (AutoTuner-selected backend + per token bucket); + 2. a bf16 activation on the rows the GEMV serves quantizing in its prologue; + 3. otherwise the engine-tuned MXFP8 quantize + GEMM. + + A kernel declining rows its path covers is a miss; the chain then continues down the list. ``push`` (an + ``m3_mx_gemv`` ``ar`` tuple) is the boundary all-reduce exchange the GEMV writes its rows into. + """ + w, s = (core.w[wk], core.w[sk]) + x = x.contiguous() + mode = _mx_mode(core.mxfp8) + if not mode.decode_graph: + prequant = _CTXNORMQ_MX.take(x) if CTXNORMQ and (not mode.tuned_quantize_gemm) else None + if prequant is not None: + return (_ctxnormq_mxfp8_gemm(core, prequant, w, s, mode), False) + return (_mxfp8_linear(core, x, w, s, mode), False) + use_pdl = get_env_enable_pdl() + rows = x.shape[0] + n, k = w.shape + prequant = m3_prequant.SLOT.take(x) + if prequant is None and x.dtype != torch.bfloat16: + path = GENERIC_GEMM_PATH + else: + path = linear_path(n, k, rows, prequant is not None) + output = None + if prequant is not None: + act, act_scale = prequant + if ( + defer + and path == MX_GEMV_PATH + and (n == mx_gemv.PROD_N) + and (rows <= 8) + and (push is None) + ): + return (_DeferredGemv(act, act_scale, w, s, use_pdl), False) + if path == MX_GEMV_PATH: + output = m3_mx_gemv(act, act_scale, w, s, True, None, use_pdl, push) + if output is not None: + return (output, push is not None and mx_gemv.last_pushed) + missed( + path, + rows, + f"the GEMV declined the [{n}, {k}] projection's MXFP8 activation" + + _uncompiled_note(), + ) + if path == PREQUANT_GEMM_PATH: + uncovered(MX_GEMV_PATH, f"the [{n}, {k}] projection at rows the GEMV does not serve") + return (m3_mxfp8_gemm(act, act_scale, w, s, torch.bfloat16, mode.autotune), False) + if path == MX_GEMV_BF16_PATH: + output = m3_mx_gemv_bf16(x, w, s, True, None, use_pdl, push) + if output is not None: + return (output, push is not None and mx_gemv.last_pushed) + missed( + path, rows, f"the GEMV declined the [{n}, {k}] projection's input" + _uncompiled_note() + ) + if path == GENERIC_GEMM_PATH: + uncovered( + MX_GEMV_BF16_PATH, + f"the [{n}, {k}] projection's bf16 input beyond the decode GEMV's rows", + ) + return (_mxfp8_linear(core, x, w, s, mode), False) + + +def _can_mc(x, folded: bool = False) -> bool: + """Whether the generic boundary takes the multicast all-reduce + residual + RMSNorm.""" + return ( + isinstance(x, torch.Tensor) + and x.dim() == 2 + and (x.shape[0] <= (m3_mc_allreduce.max_rows() if folded else MC_AR_MAX_TOKENS)) + and (x.dtype == torch.bfloat16) + and x.is_contiguous() + ) + + +def _mc_allreduce_norm(core, x, x2, residual, norm_w, *, quant=False, prepushed=0): + """The multicast all-reduce + optional second addend + residual + RMSNorm (``m3_mc_allreduce_norm``), or None + without a workspace or for operands the kernel does not take (the same on every rank). With ``quant`` the normed + rows are also published, MXFP8-quantized, to ``m3_prequant.SLOT`` for the next MXFP8 projection.""" + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + if workspace is None: + return None + return m3_mc_allreduce.m3_mc_allreduce_norm( + x, + residual.contiguous(), + norm_w, + core.rms_eps, + workspace, + use_pdl=get_env_enable_pdl(), + x2=x2, + quant=quant, + prepushed=prepushed, + ) + + +_CTXNORMQ_MX = m3_prequant.PrequantSlot() +_CTXNORMQ_FP4 = m3_prequant.PrequantSlot() +_CTXNORMQ_STATE = {} + + +def _ctxnormq_op(): + """``torch.ops.trtllm.minimax_m31_ctx_residual_rmsnorm_mxfp8``, or None. A build without the op in its library + loads it from ``tensorrt_llm/libs/libm31_ctxnormq.so`` when that file exists (an overlay); otherwise the boundary + keeps the fused all-reduce. Decided once per process.""" + if "op" not in _CTXNORMQ_STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_ctx_residual_rmsnorm_mxfp8", None) + if op is None: + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_ctxnormq.so") + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_ctx_residual_rmsnorm_mxfp8", None) + _CTXNORMQ_STATE["op"] = op + logger.info( + ( + "[M3 dispatch] CTXNORMQ prefill boundary norm + quantize: " + f"{('engaged' if op is not None else 'not taken: the op is not built')}" + " (rows >= " + f"{CTXNORMQ_MIN_ROWS}" + ")" + ) + ) + return _CTXNORMQ_STATE["op"] + + +def _ctxnormq_allreduce_norm(core, x, residual, norm_w, fp4_scale): + """A prefill boundary as the plain all-reduce + the fused residual add, RMSNorm and quantize, or None (switched + off, fewer than ``CTXNORMQ_MIN_ROWS`` rows, op not built). Returns (normed, updated) like the fused all-reduce; + the MXFP8 activation (and with ``fp4_scale`` the NVFP4 one) wait in the CTXNORMQ slots for their consumers.""" + if not CTXNORMQ or x.dim() != 2 or x.shape[0] < CTXNORMQ_MIN_ROWS: + return None + op = _ctxnormq_op() + if op is None: + return None + reduced = m3_allreduce(x, core.allreduce, None) + normed, xq, xsf, xfp4, xfp4_sf = op( + reduced, residual.contiguous(), norm_w, core.rms_eps, True, fp4_scale, get_env_enable_pdl() + ) + _CTXNORMQ_MX.publish(normed, xq, xsf) + if fp4_scale is not None: + _CTXNORMQ_FP4.publish(normed, xfp4, xfp4_sf) + else: + _CTXNORMQ_FP4.clear() + return (normed, reduced) + + +def _ctxnormq_mxfp8_gemm(core, prequant, w, s, mode): + """The GEMM ``m3_mxfp8_linear`` runs after its own quantize, on the CTXNORMQ activation (bit-identical operands).""" + act, act_scale = prequant + if mode.flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8(act, act_scale, w, s, torch.bfloat16) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if mode.native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, w, s, core.mx_alpha, torch.bfloat16) + + +_TSEL_STATE = {} + + +def _tsel_op(): + """``torch.ops.trtllm.minimax_m31_select_prefill_tile`` on sm_103, or None. A build without the op in its library + loads it from ``tensorrt_llm/libs/libm31_tsel.so`` when that file exists (an overlay). Decided once per process.""" + if "op" not in _TSEL_STATE: + op = None + if torch.cuda.get_device_capability() == (10, 3): + op = getattr(torch.ops.trtllm, "minimax_m31_select_prefill_tile", None) + if op is None: + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_tsel.so") + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_select_prefill_tile", None) + _TSEL_STATE["op"] = op + logger.info( + ( + "[M3 dispatch] TSEL prefill tile select: " + f"{('engaged' if op is not None else 'not taken: not sm_103 or the op is not built')}" + " (rows >= " + f"{TSEL_MIN_ROWS}" + ", blocks <= " + f"{TSEL_MAX_BLOCKS}" + ")" + ) + ) + return _TSEL_STATE["op"] + + +def _tsel_select(max_score_kv, n_valid_blocks, init_blocks, local_blocks): + """The MSA indexer's selector hook: the head-major [total_q, num_kv_heads, 16] table of + ``select_blocks_from_maxscore``, or None (outside the tile select's rows / blocks range, or no op) to keep it.""" + if ( + torch.cuda.is_current_stream_capturing() + or max_score_kv.dtype != torch.float32 + or max_score_kv.dim() != 3 + or (max_score_kv.stride(2) != 1) + or (max_score_kv.shape[2] < TSEL_MIN_ROWS) + or (max_score_kv.shape[1] > TSEL_MAX_BLOCKS) + ): + return None + op = _tsel_op() + if op is None: + return None + num_kv_heads, num_blocks, total_q = max_score_kv.shape + out = torch.empty( + (num_kv_heads, total_q, MSA_REQUIRED_TOPK), dtype=torch.int32, device=max_score_kv.device + ).permute(1, 0, 2) + nvb = n_valid_blocks.to(device=max_score_kv.device, dtype=torch.int32).contiguous() + op(max_score_kv, nvb, out, init_blocks, local_blocks) + if TSEL_CHECK: + ref = torch.ops.trtllm.minimax_m3_select_blocks( + max_score_kv, nvb, MSA_REQUIRED_TOPK, init_blocks, local_blocks, True + ) + if not torch.equal(out, ref): + raise AssertionError( + ( + "TSEL check: tile table mismatch against minimax_m3_select_blo" + "cks (" + f"{int((out != ref).any(dim=-1).sum())}" + " rows; " + f"{num_blocks}" + " blocks, " + f"{total_q}" + " rows)" + ) + ) + _TSEL_STATE["checked"] = _TSEL_STATE.get("checked", 0) + 1 + if _TSEL_STATE["checked"] in (1, 100, 1000): + logger.info( + f"[M3 dispatch] TSEL check: {_TSEL_STATE['checked']} tile-selected steps equal" + ) + return out + + +def _tsel_register(backend) -> None: + """Register ``_tsel_select`` on the layer backend's MSA indexer (an indexer without the hook is left alone).""" + indexer = getattr(backend, "indexer", None) + if indexer is not None and getattr(indexer, "selector_hook", False) is None: + indexer.selector_hook = _tsel_select + + +_GQAMIX_STATE: dict = {} + + +def _gqamix_decode(*args, **kwargs) -> None: + """``minimax_m3_sparse_attn_decode`` for MsaDecodeFmha's sparse generation rows, through the target-local copy.""" + if not _GQAMIX_STATE.get("logged"): + _GQAMIX_STATE["logged"] = True + logger.info( + "[M3 dispatch] GQAMIX: mixed-step sparse decode through ops/gqamix_sparse_decode" + ) + _gqamix_kernel.minimax_m3_sparse_attn_decode(*args, **kwargs) + + +def _gqamix_register(backend) -> None: + """Register ``_gqamix_decode`` on the layer backend's MsaDecodeFmha (a library without the hook is left alone).""" + manager = getattr(backend, "_fmha_manager", None) + for fmha in getattr(manager, "fmha_libs", ()): + if getattr(fmha, "sparse_decode_hook", False) is None: + fmha.sparse_decode_hook = _gqamix_decode + + +def _native_allreduce_norm(core, x, residual, norm_w): + """The built-in fused all-reduce + residual + RMSNorm: with AR64AUTO, the strategy-MNNVL instance for the rows past + the multicast all-reduce's, up to the decode bound; the configured instance otherwise.""" + params = AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual, + norm_weight=norm_w, + eps=core.rms_eps, + trigger_completion_at_end=False, + ) + wide = getattr(core, "allreduce_wide", None) + rows = x.shape[0] + if wide is not None and m3_mc_allreduce.max_rows() < rows <= DECODE_MAX_TOKENS: + out = m3_allreduce(x.contiguous(), wide, params) + if out is not None: + _log_once( + logger.info, + "ar64auto", + "taken", + f"[M3.1] AR64AUTO taken: MNNVL all-reduce at {rows} rows", + ) + return out + _log_once( + logger.warning, + "ar64auto", + f"declined {rows}", + f"[M3.1] AR64AUTO declined: MNNVL refused {rows} rows; the configured all-reduce runs", + ) + return m3_allreduce(x, core.allreduce, params) + + +@dataclass +class _MoeScalars: + """The routed experts' kernel scalars of one layer, derived from the checkpoint's per-tensor scales once the + weights are real (``MiniMaxM3Tp8ModelingV2._derive_moe``).""" + + fc31_input_scale: torch.Tensor + fc2_input_scale: torch.Tensor + fc31_alpha: torch.Tensor + fc2_alpha: torch.Tensor + fc31_scale_c: torch.Tensor + act_alpha: torch.Tensor + act_beta: torch.Tensor + act_clamp: torch.Tensor + + +def _router_logits(core, i: int, h: torch.Tensor) -> torch.Tensor: + """The fp32 router logits of MoE layer ``i``: the fp32 GEMV at one row, the tensor-core router up to 48, else the + fp32 linear.""" + rows = h.shape[0] if h.dim() == 2 else 0 + path = router_path(rows) + w = core.w[f"l{i}_router"] + if path == ROUTER_GEMV.path: + logits = _router_gemv(h, w, use_pdl=get_env_enable_pdl()) + if logits is not None: + return logits + missed(path, rows, "the router GEMV declined the operands") + elif path == ROUTER_TC.path: + fused = _router_tc(h, w, use_pdl=get_env_enable_pdl()) + if fused is not None: + return fused[0] + missed(path, rows, "the tensor-core router declined the operands") + else: + uncovered(ROUTER_TC.path, f"more than {ROUTER_TC_MAX_TOKENS} rows") + return m3_fp32_linear(h, w) + + +def _router_fp4(core, i: int, h: torch.Tensor): + """Router logits and the routed experts' NVFP4 activation from one kernel, or None (then ``_router_logits``).""" + if h.dim() != 2 or h.shape[0] > ROUTER_TC_MAX_TOKENS: + return None + rows = h.shape[0] + path = router_path(rows) + if path == ROUTER_GEMV.path: + path = ROUTER_GEMV_FP4.path + scale = core.moe[i].fc31_input_scale + if path == ROUTER_TC.path: + fused = _router_tc(h, core.w[f"l{i}_router"], scale, use_pdl=get_env_enable_pdl()) + else: + fused = _router_gemv_fp4(h, core.w[f"l{i}_router"], scale, use_pdl=get_env_enable_pdl()) + if fused is None: + missed(path, rows, "the router declined the operands") + return fused + + +def _routed_experts(core, i: int, x, x_sf, router_logits, do_finalize: bool): + """The trtllm-gen NVFP4 MoE runner over this rank's tensor-parallel slice of every expert, routing in-kernel. + + ``x_sf`` None means ``x`` is the bf16 activation, quantized here with the experts' global input scale. Returns + the finalized rows, or the unfinalized ``[gemm2 out, expert scales, expanded->permuted index]``. + """ + moe = core.moe[i] + if x_sf is None: + x, x_sf = fp4_quantize(x, moe.fc31_input_scale, 16, False, False) + rows = x.shape[0] + x_sf = x_sf.view(rows, -1).flatten() + outs = fp4_block_scale_moe_runner( + router_logits, + core.w[f"l{i}_router_bias"], + x, + x_sf.view(torch.float8_e4m3fn), + core.w[f"l{i}_e_w31"], + core.w[f"l{i}_e_s31"].view(torch.float8_e4m3fn), + None, + moe.act_alpha, + moe.act_beta, + moe.act_clamp, + core.w[f"l{i}_e_w2"], + core.w[f"l{i}_e_s2"].view(torch.float8_e4m3fn), + None, + moe.fc31_scale_c, + moe.fc31_alpha, + moe.fc2_alpha, + core.num_experts, + core.top_k, + None, + None, + core.moe_inter, + 0, + core.num_experts, + core.routed_scaling_factor, + ROUTING_MINIMAX2, + do_finalize, + act_type=ACT_SWIGLU, + topk_weights=None, + topk_ids=None, + output=None, + tune_max_num_tokens=core.max_num_tokens, + use_dp=False, + ) + return outs[0] if do_finalize else outs + + +def _split_chunk(rows: int, num_chunks: int) -> List[int]: + """The MoE scheduler's even split: the first ``rows % num_chunks`` chunks take one more row.""" + base, extra = divmod(rows, num_chunks) + return [base + 1] * extra + [base] * (num_chunks - extra) + + +def _nvfp4_layer(manager, i: int) -> bool: + return bool(getattr(manager, "is_nvfp4_layer", lambda _layer_idx: False)(i)) + + +def _producer_covered(core) -> bool: + """Whether ``PRODUCER`` covers this target's sparse layers: an NVFP4 KV cache at the TP8 heads (8 query, one KV + and one index head). A covered layer that cannot take ``m3_producer`` is a miss.""" + return core.main_kv_is_nvfp4 and (core.heads, core.kv_heads, core.index_heads) == (8, 1, 1) + + +def _producer_layout(kv_data, kv_scale) -> bool: + """Whether a sparse layer's NVFP4 pools are the layout ``m3_producer`` is built for: [pages, 2, 1, 128, 64] with + its scale pool [pages, 2, 1, 128, 8].""" + return ( + kv_data is not None + and kv_scale is not None + and (kv_data.dim() == 5) + and (tuple(kv_data.shape[1:]) == (2, 1, 128, 64)) + and (kv_data.stride()[2:] == (128 * 64, 64, 1)) + and (kv_scale.dim() == 5) + and (tuple(kv_scale.shape[1:]) == (2, 1, 128, 8)) + and (kv_scale.stride()[2:] == (128 * 8, 8, 1)) + and (kv_scale.shape[0] == kv_data.shape[0]) + ) + + +def _nvfp4_fused_write_layout(data: torch.Tensor, scales: torch.Tensor, num_heads: int) -> bool: + """Reject invalid HND pools; select the fused producer only for its packed layout.""" + if ( + data.dim() != 5 + or scales.dim() != 5 + or data.shape[0] <= 0 + or (data.shape[0] != scales.shape[0]) + or (tuple(data.shape[1:]) != (2, num_heads, 128, 64)) + or (tuple(scales.shape[1:]) != (2, num_heads, 128, 8)) + or (data.dtype not in (torch.int8, torch.uint8)) + or (scales.dtype != torch.uint8) + ): + raise ValueError("MiniMax-M3 NVFP4 data and scale caches must have matching HND geometry") + for cache in (data, scales): + if ( + cache.stride(4) != 1 + or cache.stride(3) != cache.shape[4] + or cache.stride(2) < 128 * cache.stride(3) + or (cache.stride(1) < num_heads * cache.stride(2)) + or (cache.stride(0) < 2 * cache.stride(1)) + ): + raise ValueError( + "MiniMax-M3 NVFP4 cache rows must be packed and heads/planes/pages must not overlap" + ) + if scales.stride(2) != 128 * 8: + raise ValueError("MiniMax-M3 NVFP4 MSA scale cache must have packed head blocks") + return data.stride(2) == 128 * 64 and data.stride(0) % 2 == 0 and (data.stride(1) % 2 == 0) + + +def _producer_miss(covered: bool, rows: Optional[int], reason: str) -> None: + if covered: + missed(PRODUCER.path, rows, reason) + return None + + +def _sparse_qkv_producer( + core, + i: int, + packed, + position_ids, + kv_cache_manager, + msa_idx_k_cache, + msa_out_cache_loc, + deferred: Optional[_DeferredGemv] = None, + early_trigger: bool = PRODUCER_EARLY_TRIGGER, +): + """The horizontal producer of a sparse layer: Q / index-Q (FP8) out, K / V / index-K written into the paged + caches (NVFP4 or FP8 main cache), for every step type (token-major). An NVFP4 cache at the TP8 heads takes + ``m3_producer`` (``PRODUCER``); a covered layer that cannot is a miss. Returns (q, index_q), or None for caches + neither kernel takes. + + ``kv_cache_manager``, ``msa_idx_k_cache`` (the per-layer index-K cache getter) and ``msa_out_cache_loc`` are the + calling target's ``step_args`` entries. ``deferred``: the QKV GEMV ``_linear`` recorded instead of launching + (``QKV_PRODUCER_FOLD``; ``packed`` is then its unwritten placeholder). The ``m3_producer`` branch runs it with the + producer in its epilogue (``_qkv_producer_fold``); any other branch launches it first (``materialize``). A None + return leaves it unlaunched.""" + covered = _producer_covered(core) + rows = int(packed.shape[0]) + if packed.dtype != torch.bfloat16 or position_ids is None: + return _producer_miss( + covered, + rows, + "the packed QKV dtype, position ids, head dims, norm or RoPE configuration", + ) + manager = kv_cache_manager + if manager is None: + return _producer_miss(covered, rows, "the step has no KV cache manager") + buffers = manager.get_buffers(i, kv_layout="HND") + index_k_cache = msa_idx_k_cache(i) + out_cache_loc = msa_out_cache_loc + layer_uses_nvfp4 = _nvfp4_layer(manager, i) + scale_buffers = ( + manager.get_block_scale_buffers(i, kv_layout="HND") if layer_uses_nvfp4 else None + ) + kv_quant_scale = core.inv_kv_scales[i] if layer_uses_nvfp4 else None + if layer_uses_nvfp4: + supported_main = ( + buffers is not None + and scale_buffers is not None + and _nvfp4_fused_write_layout(buffers, scale_buffers, core.kv_heads) + and buffers.is_cuda + and scale_buffers.is_cuda + and (scale_buffers.device == buffers.device) + and kv_quant_scale.is_cuda + and (kv_quant_scale.dtype == torch.float32) + and (kv_quant_scale.numel() >= 3) + and kv_quant_scale.is_contiguous() + ) + else: + supported_main = ( + buffers is not None + and buffers.is_cuda + and (buffers.dtype == torch.float8_e4m3fn) + and (buffers.dim() == 5) + and (tuple(buffers.shape[1:]) == (2, core.kv_heads, 128, 128)) + and (buffers.stride(4) == 1) + and (buffers.stride(3) == 128) + and (buffers.stride(2) == 128 * 128) + and (buffers.stride(1) >= core.kv_heads * buffers.stride(2)) + and (buffers.stride(0) >= 2 * buffers.stride(1)) + and (buffers.stride(0) % 4 == 0) + and (buffers.stride(1) % 4 == 0) + ) + supported_index = ( + index_k_cache.is_cuda + and index_k_cache.dtype == torch.float8_e4m3fn + and (index_k_cache.dim() == 4) + and (tuple(index_k_cache.shape[1:]) == (1, 128, 128)) + and (index_k_cache.stride(3) == 1) + and (index_k_cache.stride(2) == 128) + and (index_k_cache.stride(1) >= 128 * 128) + and (index_k_cache.stride(0) >= index_k_cache.stride(1)) + and (index_k_cache.stride(0) % 4 == 0) + and (index_k_cache.stride(1) % 4 == 0) + and (buffers is not None) + and (index_k_cache.shape[0] == buffers.shape[0]) + ) + cos_sin = core.rotary_cos_sin + supported_rope = ( + cos_sin.is_cuda + and cos_sin.dtype == torch.float32 + and cos_sin.is_contiguous() + and (cos_sin.dim() == 3) + and (tuple(cos_sin.shape[1:]) == (2, 32)) + ) + if ( + not supported_main + or not supported_index + or (not supported_rope) + or (out_cache_loc is None) + or (not out_cache_loc.is_cuda) + or (out_cache_loc.dtype != torch.int32) + or (not out_cache_loc.is_contiguous()) + or (out_cache_loc.numel() < rows) + ): + return _producer_miss( + covered and layer_uses_nvfp4, + rows, + ( + "the layer's NVFP4 / index caches, RoPE table, KV quant scale or " + "cache slots are not the layout the producer is built for" + ), + ) + w = core.w + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + if ( + layer_uses_nvfp4 + and (core.heads, core.kv_heads, core.index_heads) == (8, 1, 1) + and _producer_layout(buffers, scale_buffers) + ): + if deferred is not None: + folded = _qkv_producer_fold( + core, + i, + deferred, + buffers, + scale_buffers, + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + cos_sin, + pos, + early_trigger, + ) + if folded is not None: + return folded + packed = deferred.materialize() + q, index_q = m3_producer( + packed.contiguous(), + buffers.view(torch.uint8), + scale_buffers.view(torch.uint8), + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + core.heads, + get_env_enable_pdl(), + early_trigger, + ) + else: + if layer_uses_nvfp4 and covered: + missed( + PRODUCER.path, + rows, + ( + "the NVFP4 pool " + f"{tuple(buffers.shape)}" + " / scale pool " + f"{tuple(scale_buffers.shape)}" + " is not the producer's layout" + ), + ) + if deferred is not None: + packed = deferred.materialize() + q, index_q = m3_qkv_insert( + packed.contiguous(), + buffers.view(torch.uint8) if layer_uses_nvfp4 else buffers, + scale_buffers, + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.heads, + core.kv_heads, + core.index_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + ) + return (q.flatten(1), index_q.flatten(1)) + + +def _qkv_producer_fold( + core, + i: int, + deferred: _DeferredGemv, + buffers, + scale_buffers, + index_k_cache, + out_cache_loc, + kv_quant_scale, + cos_sin, + pos, + early_trigger: bool = True, +) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """QKVPROD: the deferred QKV GEMV with layer ``i``'s producer in its epilogue (``mx_gemv.m3_mx_gemv_producer``), + the same outputs and cache bytes as the GEMV + ``m3_producer``; None (nothing launched) when the kernel declines.""" + rows = int(deferred.act.shape[0]) + device = deferred.act.device + q = torch.empty(rows, core.heads, core.head_dim, dtype=torch.float8_e4m3fn, device=device) + index_q = torch.empty(rows, 1, core.head_dim, dtype=torch.float8_e4m3fn, device=device) + w = core.w + ok = mx_gemv.m3_mx_gemv_producer( + deferred.act, + deferred.act_scale, + deferred.weight, + deferred.weight_scale, + q, + index_q, + buffers.view(torch.uint8), + scale_buffers.view(torch.uint8), + index_k_cache, + out_cache_loc, + kv_quant_scale, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + trigger_early=early_trigger, + use_pdl=deferred.use_pdl, + ) + if not ok: + missed( + QKV_PRODUCER_FOLD_PATH, rows, "the producer GEMV declined the step" + _uncompiled_note() + ) + return None + deferred.done = True + if rows not in _QKV_FOLD_ENGAGED: + _QKV_FOLD_ENGAGED.add(rows) + logger.info( + ( + "[M3 dispatch] QKVPROD producer in the QKV GEMV epilogue: engaged" + " (rows=" + f"{rows}" + ") on rank " + f"{int(core.mapping.tp_rank)}" + ) + ) + return (q.flatten(1), index_q.flatten(1)) + + +def _qk_norm_rope(core, i: int, qkv: torch.Tensor, position_ids, out_fp8: bool) -> torch.Tensor: + """The per-head Gemma RMSNorm of Q and K and the partial NeoX RoPE of a dense layer's packed [Q | K | V], in one + kernel: in place, or an FP8 copy (Q / K normed and roped, V cast) when ``out_fp8``.""" + qkv = qkv.contiguous() + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + args = ( + qkv, + core.heads, + core.kv_heads, + core.kv_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_q_norm"], + core.w[f"l{i}_k_norm"], + core.rope_theta, + True, + pos, + 1.0, + 0.0, + 0.0, + 1.0, + True, + True, + False, + 0, + 0, + ) + if out_fp8: + return m3_qk_norm_rope_fp8(*args) + return fused_qk_norm_rope(*args) + + +def _fp8_index_qk_norm_rope( + core, i: int, idx_qk: torch.Tensor, position_ids, msa_idx_k_cache, msa_out_cache_loc +) -> torch.Tensor: + """The index branch of an unfused sparse layer: index-Q in FP8 out, index-K into the FP8 index-K cache, normed and + roped in one kernel (the built-in ``_fused_fp8_index_qk_norm_rope``). ``msa_idx_k_cache`` / ``msa_out_cache_loc`` + are the calling target's ``step_args`` entries.""" + if position_ids is None or idx_qk.dtype != torch.bfloat16: + raise NotImplementedError( + "MiniMax-M3 fused FP8 indexer requires BF16 activations and position_ids." + ) + index_k_cache = msa_idx_k_cache(i) + if index_k_cache.dtype != torch.float8_e4m3fn: + raise ValueError( + f"MiniMax-M3 fused FP8 indexer requires an E4M3 index-K cache, got {index_k_cache.dtype}." + ) + rows = int(idx_qk.shape[0]) + return m3_fp8_indexer_qk_norm_rope( + idx_qk.contiguous(), + index_k_cache, + msa_out_cache_loc[:rows], + core.index_heads, + core.index_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_idx_q_norm"], + core.w[f"l{i}_idx_k_norm"], + core.rope_theta, + position_ids.reshape(-1).contiguous().to(torch.int32), + ).flatten(1) + + +def _plain_capture(spec_metadata, layer_id: int, hidden_states, residual) -> None: + """The in-tree Eagle3 hidden-state capture.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + +def _next_norm(core, i: int) -> torch.Tensor: + """The norm the boundary after layer ``i`` applies: the next layer's input norm, the final norm after the last.""" + return core.w[f"l{i + 1}_in_norm"] if i + 1 < core.num_layers else core.w["final_norm"] + + +_ENGINE_STATES: Dict[tuple, object] = {} +AR_AUTOTUNE_ENABLED = m31_enabled("AR_AUTOTUNE") + + +def _pin_allreduce_heuristic(rank: int) -> bool: + """Default the AllReduce autotune off for this process (see AR-DET); returns whether it is off. Logs one line.""" + if not AR_AUTOTUNE_ENABLED: + os.environ.setdefault("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "1") + off = os.environ.get("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "0") == "1" + logger.info( + f"M3.1 AR-DET: AllReduce autotune {('off (C++ size / world heuristic)' if off else 'on')} on rank {rank}" + ) + return off + + +class MiniMaxM3Tp8ModelingV2(DecoderModel): + """The MiniMax-M3 text model of this target: its weights, its runtime resources, and the per-step dispatch.""" + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + _validate_runtime(model_config) + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + _pin_allreduce_heuristic(mapping.tp_rank) + self.mapping = mapping + self.vocab_size = cfg.vocab_size + d = _weights.MODEL_WEIGHTS.dims(self) + self.num_layers = d.num_layers + self.hidden = d.hidden + self.heads = d.heads + self.kv_heads = d.kv_heads + self.head_dim = d.head_dim + self.index_heads = d.index_heads + self.index_dim = d.index_dim + self.q_size = d.heads * d.head_dim + self.kv_size = d.kv_heads * d.head_dim + self.moe_inter = d.moe_inter + self.num_experts = d.num_experts + self.top_k = int(cfg.num_experts_per_tok) + self.routed_scaling_factor = float(getattr(cfg, "routed_scaling_factor", 1.0)) + self.rms_eps = float(cfg.rms_norm_eps) + self.swiglu_alpha = float(getattr(cfg, "swiglu_alpha", 1.702)) + self.swiglu_beta = 1.0 + self.swiglu_limit = float(getattr(cfg, "swiglu_limit", 7.0)) + self.max_num_tokens = model_config.max_num_tokens + self.moe_max_num_tokens = model_config.moe_max_num_tokens + self.sparse_layers = frozenset(_sparse_layer_ids(cfg)) + self.moe_layers = frozenset(_moe_layer_ids(cfg)) + mode = model_config.quant_config.quant_mode + self.main_kv_is_fp8 = bool(mode is not None and mode.has_fp8_kv_cache()) + self.main_kv_is_nvfp4 = bool(mode is not None and mode.has_fp4_kv_cache()) + self.fused_projection = _weights.fused_projection(model_config.sparse_attention_config) + self.w = _weights.MODEL_WEIGHTS.declare(d) + self.mxfp8 = _Mxfp8GemmState() + self._use_flashinfer_mxfp8_decode_graph_default = True + self.allreduce = AllReduce( + mapping=mapping, strategy=model_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + self.allreduce_wide = None + self._allreduce_wide_owner = None + if AR64AUTO and DECODE_MAX_TOKENS > m3_mc_allreduce.max_rows(): + self._allreduce_wide_owner = AllReduce( + mapping=mapping, strategy=AllReduceStrategy.MNNVL, dtype=cfg.torch_dtype + ) + self.allreduce_wide = getattr(self._allreduce_wide_owner, "mnnvl_allreduce", None) + if self.allreduce_wide is not None: + logger.info( + ( + "[M3.1] AR64AUTO: boundaries of " + f"{m3_mc_allreduce.max_rows() + 1}" + ".." + f"{DECODE_MAX_TOKENS}" + " rows take the strategy-MNNVL all-reduce (configured: " + f"{model_config.allreduce_strategy}" + ")" + ) + ) + else: + logger.warning( + "[M3.1] AR64AUTO: MNNVL unavailable on this mapping; every boundary keeps the configured all-reduce" + ) + self.aux_stream_moe = torch.cuda.Stream() + chunked = self.moe_max_num_tokens < model_config.max_num_tokens * mapping.dp_size + self.aux_stream_moe_chunk = torch.cuda.Stream() if chunked else None + self.moe_events = {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers} + self.moe_chunk_events = { + i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers + } + self.aux_stream_attn = None if self.fused_projection else torch.cuda.Stream() + self.attn_events = ( + {} + if self.fused_projection + else {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.sparse_layers} + ) + rope = RopeParams.from_config(cfg) + self.rope_theta = rope.theta + self.rotary_dim = int(rope.dim) + self.rotary_cos_sin = rope.create_rope_const_params(interleave=False)[1].reshape( + rope.max_positions, 2, -1 + ) + quant_config = model_config.get_quant_config() + self.attn: List[MiniMaxM3MsaSparseAttention] = [] + for i in range(self.num_layers): + backend = create_attention( + model_config.attn_backend, + i, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=None, + quant_config=quant_config, + skip_create_weights_in_init=model_config.skip_create_weights_in_init, + q_scaling=1.0, + sparse_params=model_config.sparse_attention_config.to_sparse_params( + pretrained_config=cfg, layer_idx=i + ), + ) + backend.update_quant_config(quant_config) + if not isinstance(backend, MiniMaxM3MsaSparseAttention): + raise ValueError( + f"layer {i}: expected the MiniMax-M3 MSA backend, got {type(backend).__name__}" + ) + if i not in self.sparse_layers: + backend.dense_decode_max_seq_len_by_batch = DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH + self.attn.append(backend) + self.kv_scales: Dict[int, torch.Tensor] = {} + self.inv_kv_scales: Dict[int, torch.Tensor] = {} + self.moe: Dict[int, _MoeScalars] = {} + self.mx_alpha: Optional[torch.Tensor] = None + self.moe_engines: Dict[int, Dict[Tuple[int, bool, bool], Optional[object]]] = {} + self.embed_owners: Dict[tuple, DecodeEmbedding] = {} + self._built_for: Optional[tuple] = None + self._prefill: Optional[PrefillTarget] = None + self._decode: Optional[DecodeTarget] = None + + def sm_scale(self, i: int) -> float: + return self.head_dim ** (-0.5) / float(self.attn[i].q_scaling) + + def post_load_weights(self) -> None: + """Derive the kernels' state from the loaded weights, then build both targets and every decode kernel. + + Idempotent per set of weight storages: the loader's post-load walk reaches this module directly and through + the shell, and a second call on the same storages does nothing. + """ + identity = ( + self.w["embed"].data_ptr(), + self.w["l0_in_norm"].data_ptr(), + self.w["final_norm"].data_ptr(), + ) + if self._built_for == identity: + return + device = self.w["embed"].device + derived = nn.Module() + derived.register_buffer( + "mx_alpha", torch.ones(1, dtype=torch.float32, device=device), persistent=False + ) + for i in range(self.num_layers): + derived.register_buffer( + f"l{i}_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + derived.register_buffer( + f"l{i}_inv_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + for i in sorted(self.moe_layers): + for name, value in dataclasses.asdict(self._derive_moe(i)).items(): + derived.register_buffer(f"l{i}_{name}", value, persistent=False) + self.derived = derived + self.mx_alpha = derived.mx_alpha + self.kv_scales = {i: getattr(derived, f"l{i}_kv_scales") for i in range(self.num_layers)} + self.inv_kv_scales = { + i: getattr(derived, f"l{i}_inv_kv_scales") for i in range(self.num_layers) + } + self.moe = { + i: _MoeScalars( + **{ + f.name: getattr(derived, f"l{i}_{f.name}") + for f in dataclasses.fields(_MoeScalars) + } + ) + for i in sorted(self.moe_layers) + } + self._build_embed_owners() + self.prebuild_kernels() + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + self._built_for = identity + + def cache_derived_state(self) -> None: + """The staged loaders' entry to the same build (``post_load_weights``).""" + self.post_load_weights() + + def _derive_moe(self, i: int) -> _MoeScalars: + """The routed experts' kernel scalars from the checkpoint's per-tensor scales (trtllm-gen NVFP4 layout). + + Global input scales: the reciprocal of the largest per-expert input scale (gate and up must agree). Per expert: + ``alpha = 1 / (input_scale * (1 / weight_scale_2))`` for the gate (fc31_alpha), the up half (its GEMM2-input + rescale ``fc31_scale_c = fc2_input_scale * up_alpha``) and the down projection; SwiGLU-OAI's activation + constants per expert, with beta and the clamp in the pre-dequant GEMM output domain (divided by fc31_alpha). + """ + w = self.w + isc1, isc3, isc2 = (w[f"l{i}_e_isc1"].data, w[f"l{i}_e_isc3"].data, w[f"l{i}_e_isc2"].data) + if not torch.allclose(isc1, isc3): + raise ValueError(f"layer {i}: the routed experts' w1 and w3 input scales differ") + fc31_input_scale = isc1.max().reciprocal() + fc2_input_scale = isc2.max().reciprocal() + ws2_1, ws2_3, ws2_2 = ( + w[f"l{i}_e_ws2_1"].data, + w[f"l{i}_e_ws2_3"].data, + w[f"l{i}_e_ws2_2"].data, + ) + fc31_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_1)) + up_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_3)) + fc2_alpha = 1.0 / (fc2_input_scale * (1.0 / ws2_2)) + fc31_scale_c = fc2_input_scale * up_alpha + n = self.num_experts + device = fc31_alpha.device + act_alpha = torch.full((n,), self.swiglu_alpha, dtype=torch.float32, device=device) + act_beta = torch.full((n,), self.swiglu_beta, dtype=torch.float32, device=device) + act_clamp = torch.full((n,), self.swiglu_limit, dtype=torch.float32, device=device) + act_beta.div_(fc31_alpha) + act_clamp.div_(fc31_alpha) + return _MoeScalars( + fc31_input_scale=fc31_input_scale, + fc2_input_scale=fc2_input_scale, + fc31_alpha=fc31_alpha, + fc2_alpha=fc2_alpha, + fc31_scale_c=fc31_scale_c, + act_alpha=act_alpha, + act_beta=act_beta, + act_clamp=act_clamp, + ) + + def _build_embed_owners(self) -> None: + """The fused embedding + first RMSNorm's caller-owned buffers, one per (rows, id dtype) of a decode step.""" + table, weight = (self.w["embed"], self.w["l0_in_norm"]) + self.embed_owners = { + (rows, dtype): DecodeEmbedding(table, weight, self.rms_eps, rows, dtype) + for rows in range(1, EMBED_FIRSTNORM_MAX_TOKENS + 1) + for dtype in (torch.int32, torch.int64) + } + + def projections(self) -> List[Tuple[str, str, Optional[str]]]: + """Every MXFP8 projection in layer order: (weight key, scale key, the boundary its GEMV pushes into).""" + out = [] + for i in range(self.num_layers): + out.append((f"l{i}_qkv_w", f"l{i}_qkv_s", None)) + out.append((f"l{i}_o_w", f"l{i}_o_s", "attn")) + if i in self.moe_layers: + out.append((f"l{i}_sh_gu_w", f"l{i}_sh_gu_s", None)) + out.append((f"l{i}_sh_dn_w", f"l{i}_sh_dn_s", "moe")) + else: + out.append((f"l{i}_mlp_gu_w", f"l{i}_mlp_gu_s", None)) + out.append((f"l{i}_mlp_dn_w", f"l{i}_mlp_dn_s", None)) + return out + + def decode_query_lens(self) -> List[int]: + """The query rows per request a decode step can have: 1, and 1 + each draft length speculation runs.""" + lens = {1} + spec = getattr(self.model_config, "spec_config", None) + if spec is not None: + per_step = getattr(spec, "tokens_per_gen_step", None) + if isinstance(per_step, int) and per_step > 1: + lens.add(per_step) + for drafts in ( + getattr(spec, "max_draft_len", None), + getattr(spec, "max_total_draft_tokens", None), + *(getattr(spec, "draft_len_schedule", None) or {}).values(), + ): + if drafts: + lens.add(1 + int(drafts)) + return sorted(lens) + + def prebuild_kernels(self) -> Dict[str, object]: + """Compile (and load) every kernel a decode forward dispatches to that would otherwise compile on its first + call: each projection's decode GEMVs (every row count, activation and boundary push), the MoE engines of + ``MOE_ENGINE``'s token counts, and the index scorer at the decode query lengths. Runs on the + current (main) stream before the executor's warmup, in the same order on every rank, then synchronizes; after + it, GEMV compiles are frozen and the forward only looks kernels up (a missing one is a miss). Returns the + seconds each part took and the scorer keys.""" + times: Dict[str, object] = {} + if not self.mxfp8.quant_method.use_cutlass: + raise ValueError("the MiniMax-M3 decode dispatch needs the CUTLASS MXFP8 GEMM") + mx_gemv.freeze(False) + use_pdl = get_env_enable_pdl() + world = int(self.mapping.tp_size) + start = time.perf_counter() + for wk, sk, push in self.projections(): + weight, scale = (self.w[wk], self.w[sk]) + mx_gemv.ensure_compiled(weight, scale, use_pdl) + n, k = weight.shape + for rows, x_bf16, ar in _gemv_variants(n, k, push, world): + mx_gemv.precompile(weight, scale, rows, x_bf16, use_pdl=use_pdl, ar=ar) + if ( + QKV_PRODUCER_FOLD + and n == mx_gemv.PROD_N + and (not x_bf16) + and (ar is None) + and (rows <= 8) + ): + mx_gemv.precompile(weight, scale, rows, False, use_pdl=use_pdl, prod=True) + if PRODUCER_EARLY_TRIGGER and PRODUCER_SHORTFIX: + mx_gemv.precompile( + weight, + scale, + rows, + False, + use_pdl=use_pdl, + prod=True, + trigger_early=False, + ) + swq_shapes = {tuple(self.w[wk].shape) for wk, _, _ in self.projections()} & set(SWQ_ROWS) + for n, k in sorted(swq_shapes): + gu = torch.zeros( + max(SWQ_ROWS[n, k]), 2 * k, dtype=torch.bfloat16, device=self.w["embed"].device + ) + m3_swq(gu, self.swiglu_limit, self.swiglu_alpha, self.swiglu_beta) + logger.info(f"[M3 prebuild] SWQ: compiled for intermediate {k}") + torch.cuda.synchronize() + times["gemv_s"] = time.perf_counter() - start + start = time.perf_counter() + for i in sorted(self.moe_layers): + self._prebuild_engines(i) + torch.cuda.synchronize() + times["moe_s"] = time.perf_counter() - start + start = time.perf_counter() + times["scorer_keys"] = self._prebuild_index_scorer( + self.decode_query_lens(), self.w["embed"].device + ) + torch.cuda.synchronize() + times["scorer_s"] = time.perf_counter() - start + start = time.perf_counter() + times["select_keys"] = [] + if long_select_dsl_keys(): + times["select_keys"] = _select_long_dsl_kernel.prebuild( + long_select_dsl_keys(), self.w["embed"].device + ) + torch.cuda.synchronize() + times["select_s"] = time.perf_counter() - start + start = time.perf_counter() + times["proxy_variants"] = self._prebuild_msa_proxy() if MSA_PROXY_PREBUILD else [] + times["proxy_s"] = time.perf_counter() - start + mx_gemv.freeze(True) + logger.info( + ( + "[M3 prebuild] decode kernels built before warmup: GEMVs %.1f s, " + "MoE engines %.1f s, index scorer %.1f s (%d variants), MSA index" + "er proxy %.1f s (%d variants)" + ) + % ( + times["gemv_s"], + times["moe_s"], + times["scorer_s"], + len(times["scorer_keys"]), + times["proxy_s"], + len(times["proxy_variants"]), + ) + ) + return times + + def _expert_operands(self, i: int): + """The routed experts' operands the engines read (trtllm-gen NVFP4 SwiGLU-OAI layout), and the input scale.""" + w, m = (self.w, self.moe[i]) + tensors = [ + w[f"l{i}_e_w31"].data, + w[f"l{i}_e_s31"].data, + w[f"l{i}_e_w2"].data, + w[f"l{i}_e_s2"].data, + m.fc31_alpha, + m.fc31_scale_c, + m.act_alpha, + m.act_beta, + m.act_clamp, + m.fc2_alpha, + ] + return (tensors, m.fc31_input_scale) + + def _shared_weights(self, i: int): + w = self.w + return ( + w[f"l{i}_sh_gu_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_gu_s"].data, + w[f"l{i}_sh_dn_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_dn_s"].data, + ) + + def engine(self, i: int, num_tokens: int, shared: bool) -> Optional[object]: + """Layer ``i``'s prebuilt routed-expert engine for ``num_tokens`` (1..8), router folded in (and the shared + expert with ``shared``). None after a miss: the build failed or was never made (a forward never builds one).""" + return self._lookup( + i, self.moe_engines[i], (num_tokens, True, shared), MOE_ENGINE.path, num_tokens + ) + + def _lookup(self, i: int, engines: dict, key, path: str, num_tokens: int) -> Optional[object]: + if key not in engines: + what = f"layer {i}'s engine" + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, what) + else: + missed(path, num_tokens, f"{what} was not built before warmup") + return None + engine = engines[key] + if engine is None: + missed(path, num_tokens, "its build failed (logged)") + return engine + + def _prebuild_engines(self, i: int) -> None: + """Build (compile, on first use of each shared state, and warm) every engine a decode step of layer ``i`` can + take: ``MOE_ENGINE``'s token counts with the shared expert folded in. Whether an + engine is kept is agreed across the TP group (one host collective per engine): a build that fails on one rank + drops that engine on every rank, so the ranks never mix an engine (and its boundary push) with the generic + MoE.""" + engines = self.moe_engines.setdefault(i, {}) + for num_tokens in sorted(MOE_ENGINE.keys): + key = (num_tokens, True, True) + if key not in engines: + engines[key] = m3_mc_allreduce.tp_agree_built( + self.mapping, + self._try_build( + i, MOE_ENGINE.path, num_tokens, self._build_engine, shared=True + ), + f"[M3 dispatch] {MOE_ENGINE.path} at {num_tokens} rows: layer {i}'s engine", + ) + + def _try_build(self, i: int, path: str, num_tokens: int, build, **kwargs) -> Optional[object]: + try: + return build(i, num_tokens, **kwargs) + except (ImportError, RuntimeError, ValueError) as err: + logger.warning( + f"[M3 dispatch] {path} at {num_tokens} rows: layer {i}'s engine failed to build: {err}" + ) + return None + + def _build_engine(self, i: int, num_tokens: int, shared: bool) -> object: + tensors, scale = self._expert_operands(i) + device = tensors[0].device + sms = torch.cuda.get_device_properties(device).multi_processor_count + kwargs = {} + if shared: + kwargs["shared_weights"] = self._shared_weights(i) + state = _ENGINE_STATES.get((str(device), num_tokens, shared)) + if state is None: + state = _ENGINE_STATES[str(device), num_tokens, shared] = M3MoeState( + device, num_tokens=num_tokens, num_ctas=sms - MOE_ENGINE_RESERVED_SMS, shared=shared + ) + engine = state.layer( + *tensors, + self.w[f"l{i}_router_bias"].data, + router_weight=self.w[f"l{i}_router"].data, + fc31_input_scale=scale.reshape(1).float(), + **kwargs, + ) + m3_moe_engine( + torch.zeros(num_tokens, tensors[0].shape[-1] * 2, dtype=torch.bfloat16, device=device), + engine, + ) + return engine + + def _scorer_variants(self, query_lens) -> list: + """The scorer keys a decode step can take: every sparse layer's (index dtype, index heads per rank, index dim) + at each decode query length, with the early-issue schedule (up to ``INDEX_SCORE_EARLY_MAX_REQUESTS`` requests, + none here) and without it, where the kernel serves it.""" + runner = _index_score_kernel.IndexScoreRunner + keys = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + config = backend.m3_config + dtype = torch.float8_e4m3fn if backend.indexer_kv_dtype == "fp8" else torch.bfloat16 + for qlen in sorted(set(query_lens)): + if not runner.is_supported( + q_dtype=dtype, + num_heads=config.num_index_heads, + head_dim=config.sparse_index_dim, + page_size=runner.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ): + continue + for early in (True, False) if INDEX_SCORE_EARLY_MAX_REQUESTS else (False,): + key = ( + dtype, + int(config.num_index_heads), + int(qlen), + int(config.sparse_index_dim), + int(INDEX_SCORE_SPLIT_K), + bool(early), + ) + if key not in keys: + keys.append(key) + return keys + + def _prebuild_msa_proxy(self) -> list: + """Compile and load every fmha_sm100 variant the MSA indexer's proxy pass can pick (OnlyScore over the paged + index-K: one KV head, page 128, no split-KV), so no context chunk compiles one after the server is ready (a + 65..128-token chunk's variant did, stalling the steps behind it by 0.8-1.6 ms). The variant follows only the + longest query row: fmha_sm100 packs a row of up to 32 tokens by its pack factor, then takes the 128- or 256-row + query tile and the single-warpgroup kernel from the packed length, so query lengths 1..129 reach every variant. + Compile and load only, nothing runs: no numerics change. fmha_sm100 keys its cache by its installed sources, so + a source overlay builds its own variants. Returns the variant names.""" + require_msa_module() + from fmha_sm100 import api as fmha_api + from fmha_sm100 import jit as fmha_jit + + only_score, page = (2, 128) + names = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + dtype = torch.float8_e4m3fn if backend.indexer_kv_dtype == "fp8" else torch.bfloat16 + heads = int(backend.m3_config.num_index_heads) + for qlen in range(1, 130): + pack = fmha_api._compute_pack_factor(qlen, heads, 1) + packed = qlen * pack + key = ( + fmha_jit._dlpack_dtype_code(dtype), + 128 if packed <= 128 else 256, + packed <= 64, + only_score, + page, + False, + pack, + ) + name, _ = fmha_jit._variant_key_from_runtime(*key) + if name not in names: + fmha_jit.get_fmha_variant(*key) + names.append(name) + return names + + def _prebuild_index_scorer(self, query_lens, device) -> list: + """Compile every scorer key and launch each once on real-shaped dummy operands (loading its module), on the + current stream. The kernel's batch, token, page and block counts are symbolic: its compile needs no step + metadata. Returns the keys.""" + runner = _index_score_kernel.IndexScoreRunner + keys = self._scorer_variants(query_lens) + page = runner.SUPPORTED_PAGE_SIZE + for key in keys: + dtype, heads, qlen, dim, split_k, early = key + runner._compile(dtype, heads, qlen, dim, split_k, early) + rows = 1 if early else INDEX_SCORE_EARLY_MAX_REQUESTS + 1 + blocks = 2 + tokens = rows * qlen + idx_q = torch.zeros(tokens, heads, dim, dtype=torch.bfloat16, device=device).to(dtype) + idx_k = torch.zeros( + rows * blocks, 1, page, dim, dtype=torch.bfloat16, device=device + ).to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device=device).view( + rows, blocks + ) + seq_lens = torch.full((rows,), blocks * page, dtype=torch.int32, device=device) + score = torch.empty(heads, blocks, tokens, dtype=torch.float32, device=device) + runner.kernel_cache[key]( + idx_q, idx_k.squeeze(1), block_table, score.transpose(1, 2), seq_lens + ) + return keys + + def _select_target(self, attn_metadata: AttentionMetadata): + """Which target runs this step: one host-side integer, fixed per CUDA-graph capture. A step with context rows + (a mixed step included) is prefill's; every pure generation step is decode's, whatever its width.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.IntTensor] = None, + position_ids: Optional[torch.IntTensor] = None, + inputs_embeds: Optional[torch.FloatTensor] = None, + spec_metadata: Optional[SpecMetadata] = None, + capture: Callable[..., None] = _plain_capture, + **kwargs, + ) -> torch.Tensor: + if (input_ids is None) ^ (inputs_embeds is not None): + raise ValueError("You must specify exactly one of input_ids or inputs_embeds") + if kwargs.get("lora_params") is not None: + raise ValueError("this MiniMax-M3 target serves no LoRA adapters") + return self._select_target(attn_metadata).forward( + attn_metadata, input_ids, position_ids, inputs_embeds, spec_metadata, capture + ) + + +def _swiglu_act(core, gu: torch.Tensor) -> torch.Tensor: + return m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + + +class PrefillTarget: + """A step with context rows: prefill chunks, and the mixed steps that carry generation rows beside them. Every + layer runs in its generic form; the attention core is the MSA backend's for every row.""" + + def __init__(self, core: MiniMaxM3Tp8ModelingV2) -> None: + self.core = core + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + Counts and per-step host scalars are read here; device tensors are the metadata's persistent buffers, held + by reference; ``msa_idx_k_cache`` is the metadata's per-layer index-K cache getter. ``metadata`` is the + prepared object itself: the MSA backend's step (``m3_msa_attention``) takes it as its argument. + """ + return dict( + num_contexts=md.num_contexts, + num_ctx_tokens=md.num_ctx_tokens, + num_tokens=md.num_tokens, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + metadata=md, + ) + + def forward( + self, attn_metadata, input_ids, position_ids, inputs_embeds, spec_metadata, capture + ): + core = self.core + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) prefill CUDA graphs" + ) + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + hidden, residual = self._layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + return hidden + + def _swiglu_mlp(self, h: torch.Tensor, prefix: str) -> torch.Tensor: + """A gated MXFP8 MLP (the dense MLP, the shared expert): gate_up, SwiGLU-OAI, down, no reduction.""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s") + act = _swiglu_act(core, gu) + out, _ = _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s") + return out + + def _routed_chunks(self, i: int, x, router_logits): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: + the MoE scheduler's even split, even chunks on the MoE chunking stream.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + fp4 = _CTXNORMQ_FP4.take(x) if CTXNORMQ else None + if num_chunks <= 1: + if fp4 is not None: + return _routed_experts(core, i, fp4[0], fp4[1], router_logits, True) + return _routed_experts(core, i, x, None, router_logits, True) + sizes = _split_chunk(rows, num_chunks) + xs, logits = (x.split(sizes), router_logits.split(sizes)) + sfs = [None] * num_chunks + if fp4 is not None: + xs = fp4[0].split(sizes) + sfs = [t.flatten() for t in fp4[1].view(rows, -1).split(sizes)] + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + else: + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor): + """Router + routed experts, finalized: the fused router with the NVFP4 activation up to 48 rows, else the + router logits and the experts quantizing their bf16 input (chunked).""" + core = self.core + fused = _router_fp4(core, i, h) + if fused is not None: + router_logits, x_fp4, x_sf = fused + return _routed_experts(core, i, x_fp4, x_sf, router_logits, True) + router_logits = _router_logits(core, i, h) + return self._routed_chunks(i, h, router_logits) + + def _moe(self, i: int, h: torch.Tensor, *, return_parts: bool): + """The MoE block of layer ``i`` without its reduction. Returns ``(out, routed)``: with ``return_parts`` the + shared and routed parts (routed None when the engine folded the shared expert in), else their sum in + ``out``.""" + core = self.core + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + h.dim() == 2 + and h.dtype == torch.bfloat16 + and (1 <= rows <= MOE_ENGINE_MAX_TOKENS) + and (rows >= MOE_FOLD_MIN_TOKENS) + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + folded = m3_moe_engine(h.contiguous(), engine, None, None) + return (folded, None) + routed_out, shared_out = maybe_execute_in_parallel( + lambda: self._routed(i, h), + lambda: self._swiglu_mlp(h, f"l{i}_sh"), + core.moe_events[i][0], + core.moe_events[i][1], + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if return_parts: + return (shared_out, routed_out) + return (shared_out.add_(routed_out), None) + + def _generic_boundary(self, x, x2, residual, norm_w): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce.""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + if norm_w is not core.w["final_norm"]: + fused = _ctxnormq_allreduce_norm(core, x, residual, norm_w, None) + if fused is not None: + return fused + return _native_allreduce_norm(core, x, residual, norm_w) + + def _layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + else: + fused = _ctxnormq_allreduce_norm( + core, + hidden, + residual, + core.w[f"l{i}_post_norm"], + core.moe[i].fc31_input_scale if i in core.moe_layers else None, + ) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + if i in core.moe_layers: + out, routed = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out = self._swiglu_mlp(hidden, f"l{i}_mlp") + routed = None + normed, updated = self._generic_boundary(out, routed, residual, _next_norm(core, i)) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + capture(spec_metadata, i, updated, None) + return (normed, updated) + + def _attention(self, i: int, position_ids, hidden, step): + """Attention of layer ``i`` up to its o_proj (no reduction). Returns (o_proj output, pushed): a prefill step + arms no boundary push, so pushed is always False.""" + core = self.core + if i not in core.sparse_layers: + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s") + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s") + if not core.fused_projection: + return self._unfused_sparse_attention(i, position_ids, hidden, step) + packed, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", defer=QKV_PRODUCER_FOLD) + deferred = None + if isinstance(packed, _DeferredGemv): + deferred, packed = (packed, packed.placeholder) + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + deferred, + early_trigger=_producer_early_trigger(step), + ) + if horizontal is None: + raise RuntimeError( + ( + "MiniMax-M3 sparse layer " + f"{i}" + ": no horizontal producer takes this step's caches (an FP8 or NVF" + "P4 HND pool with an FP8 index-K cache)" + ) + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s") + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output.""" + core = self.core + if _producer_covered(core): + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s") + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s") + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): the MSA + backend (``m3_msa_attention``: the KV write unless a producer wrote the caches, the sparse layers' top-k block + selection, the FMHA over the context rows and any generation rows beside them).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + live = int(step["num_tokens"]) + q_live, out_live = (q[:live], output[:live]) + k_live = k[:live] if k is not None else None + v_live = v[:live] if v is not None else None + idx_q_live = idx_q[:live] if idx_q is not None else None + sparse = i in core.sparse_layers + nvfp4 = core.main_kv_is_nvfp4 + if TSEL and sparse: + _tsel_register(core.attn[i]) + if GQAMIX and sparse: + _gqamix_register(core.attn[i]) + m3_msa_attention( + core.attn[i], + q_live, + k_live, + v_live, + idx_q_live, + step["metadata"], + out_live, + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + return output + + +class DecodeTarget: + """A pure generation step (no context rows), of any width. + + Up to 8 tokens with 1-D input ids from one request, per layer: attention (o_proj all-reduce deferred, its rows + pushed into the boundary all-reduce) -> attention boundary (all-reduce + residual + post-attention RMSNorm) -> + MoE (the 1..8-row engine, pushing its sum into the boundary all-reduce) or dense MLP -> MoE boundary + (all-reduce + shared/routed add + residual + + the next layer's RMSNorm, emitting the next QKV's MXFP8 activation). A layer whose output Eagle3 captures runs the + generic layer, as does the first layer when the fused embedding does not take the step. A wider step (or one + carrying embeddings) runs every layer in its generic form, with this target's attention (the decode selection and + sparse decode where the producer wrote the caches). + """ + + def __init__(self, core: MiniMaxM3Tp8ModelingV2) -> None: + self.core = core + self._moe_push_descs: Dict[tuple, Any] = {} + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + States the two context counts instead of reading them back: this target runs only when ``num_contexts`` is + zero, so reading them would ask the metadata to confirm what the routing already decided. + + Graph classes: counts, the generation span, the identity-selection flag and the worst-case score extent are + host values fixed per capture (prepare() stages them before the forward); the block table, lengths, cache + slots, identity table and valid-block counts are the metadata's persistent buffers, held by reference and + refreshed in place every step; ``msa_idx_k_cache`` / ``msa_proxy_max_score_view`` are the metadata's getters + of the per-layer index-K cache and of the score scratch view. ``metadata`` is the prepared object itself: the + MSA backend's step (``m3_msa_attention``) takes it as its argument. + """ + return dict( + num_contexts=0, + num_ctx_tokens=0, + num_tokens=md.num_tokens, + num_seqs=md.num_seqs, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + msa_decode_span=md.msa_decode_span, + msa_block_table=md.msa_block_table, + msa_seq_lens_cuda=md.msa_seq_lens_cuda, + msa_short_ok=md._msa_short_ok, + msa_short_table=md.msa_short_table, + msa_proxy_max_score_view=md.msa_proxy_max_score_view, + msa_worst_case_max_k_tiles=md.msa_worst_case_max_k_tiles, + msa_n_valid_blocks=md.msa_n_valid_blocks, + metadata=md, + ) + + def forward( + self, attn_metadata, input_ids, position_ids, inputs_embeds, spec_metadata, capture + ): + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + kernels = ( + inputs_embeds is None + and input_ids.ndim == 1 + and attn_metadata.num_seqs == 1 + and is_decode_step(input_ids.numel()) + ) + if kernels: + num_tokens = int(input_ids.numel()) + plan = plan_for(num_tokens) + hidden, residual = self._embed(input_ids, num_tokens) + else: + uncovered( + "decode_kernels", + f"outside single-request {FUSED_DECODE_MAX_TOKENS}-row fused scope, " + "embeddings or 2-D ids: generic layers", + ) + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) CUDA graphs" + ) + plan = None + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + captured = spec_metadata is not None and spec_metadata.is_layer_capture(i) + if not kernels or residual is None or (captured and (not FUSED_CAPTURE)): + hidden, residual = self._generic_layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + else: + hidden, residual = self._layer(i, plan, position_ids, hidden, residual, step) + if captured: + capture(spec_metadata, i, residual, None) + _log_once( + logger.info, + "B-4", + "fused capture", + ( + "[M3 dispatch] B-4 fused capture taken: capture layer " + f"{i}" + " runs the decode-kernel layer (logged once)" + ), + ) + return hidden + + def _embed(self, input_ids: torch.Tensor, num_tokens: int): + """(hidden, residual) of the fused embedding + first RMSNorm; (embedding, None) when it does not run, so the + first layer runs its own input norm.""" + core = self.core + if not EMBED_FIRSTNORM.covers(num_tokens): + uncovered(EMBED_FIRSTNORM.path, f"more than {EMBED_FIRSTNORM_MAX_TOKENS} tokens") + return (m3_embedding(input_ids, core.w["embed"]), None) + owner = core.embed_owners.get((num_tokens, input_ids.dtype)) + if owner is None or not input_ids.is_contiguous(): + missed( + EMBED_FIRSTNORM.path, + num_tokens, + f"no embedding owner for {input_ids.dtype} ids (built for int32 / int64 after the load)" + if owner is None + else "the input ids are not contiguous", + ) + return (m3_embedding(input_ids, core.w["embed"]), None) + return m3_embedding_firstnorm( + input_ids, owner.table, owner.weight, owner.eps, owner.raw, owner.output, owner.use_pdl + ) + + def _bucket_flags(self, num_tokens: int): + """(bucket workspace, its 8-CTA call counters) of the boundary all-reduce for ``num_tokens``, or None.""" + workspace = m3_mc_allreduce.get_workspace(self.core.mapping, self.core.hidden) + workspace = None if workspace is None else m3_mc_allreduce._bucket(workspace, num_tokens) + flags = None if workspace is None else workspace.flags_for(8) + return None if flags is None else (workspace, flags) + + @staticmethod + def _no_bucket(path: str, num_tokens: int) -> None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, "the multicast all-reduce workspace") + else: + missed(path, num_tokens, "no multicast all-reduce bucket holds the step") + + def _attn_push_target(self, num_tokens: int): + """The o_proj's push tuple into the attention boundary's all-reduce, or None (after a miss). + + None comes only from the token count or from workspace state created collectively in the same eager call on + every rank, so all ranks agree. (A lone non-pushing rank would still be safe: its all-reduce pushes the same + bf16 rows into the same set A.)""" + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(ATTN_PUSH.path, num_tokens) + return None + workspace, flags = found + return ( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + AR_PUSH_MODE, + ) + + def _moe_push_targets(self, num_tokens: int): + """(engine push descriptor, shared down-projection push tuple) for the MoE boundary's exchange sets, or None + (after a miss). + + This decision must match across ranks: prepushed 3 reads set B of every rank, which a non-pushing rank never + fills. None comes only from the token count or from state created in the same eager call on every rank + (workspace, flags, the descriptor cache), and the engine it is armed for is agreed at build time + (``_prebuild_engines``), so it does.""" + hidden = self.core.hidden + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(MOE_PUSH.path, num_tokens) + return None + workspace, flags = found + key = (workspace.mc.data_ptr(), flags.data_ptr()) + desc = self._moe_push_descs.get(key) + if desc is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(MOE_PUSH.path, num_tokens, "the MoE engine's push descriptor") + return None + desc = self._moe_push_descs[key] = ar_push_desc( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + flags.device, + ) + set_b = workspace.mc.data_ptr() + 2 * 4 * workspace.max_tokens * workspace.world_size * ( + hidden // 2 + ) + return ( + desc, + ( + set_b, + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + 2, + ), + ) + + def _boundary(self, use_mc: bool, x, residual, norm_w, x2=None, quant=False): + """All-reduce + (second addend) + residual + RMSNorm: the multicast kernel, else the built-in all-reduce.""" + core = self.core + if use_mc and (not MC_ALLREDUCE.covers(x.shape[0])): + uncovered( + MC_ALLREDUCE.path, + f"more than {m3_mc_allreduce.max_rows()} rows: the built-in all-reduce runs", + ) + use_mc = False + if use_mc: + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w, quant=quant) + if fused is not None: + return fused + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + missed( + MC_ALLREDUCE.path, + x.shape[0], + m3_mc_allreduce.decline_reason(x, residual, workspace, x2=x2), + ) + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _prepushed(self, i: int, x, residual, norm_w, quant, prepushed, x2=None): + fused = _mc_allreduce_norm( + self.core, x, x2, residual, norm_w, quant=quant, prepushed=prepushed + ) + if fused is None: + raise RuntimeError(f"layer {i}: rows were pushed into an all-reduce that did not run") + return fused + + def _layer(self, i: int, plan: DecodePlan, position_ids, hidden, residual, step): + core = self.core + num_tokens = hidden.shape[0] + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + if not decode_graph: + uncovered( + "decode_graph", + "an eager decode step: the decode-graph GEMV dispatch and the boundary pushes are off", + ) + push = plan.attn_mc_ar and decode_graph and ATTN_PUSH.covers(num_tokens) + armed = self._attn_push_target(num_tokens) if push else None + hidden, attn_pushed = self._attention(i, position_ids, hidden, step, armed) + post_norm = core.w[f"l{i}_post_norm"] + if armed is not None and attn_pushed: + hidden, residual = self._prepushed(i, hidden, residual, post_norm, False, AR_PUSH_MODE) + else: + if armed is not None: + missed( + ATTN_PUSH.path, + num_tokens, + "the o_proj did not push its rows into the armed boundary all-reduce", + ) + hidden, residual = self._boundary(plan.attn_mc_ar, hidden, residual, post_norm) + norm_w = _next_norm(core, i) + if i not in core.moe_layers: + out, _ = self._swiglu_mlp(hidden, f"l{i}_mlp") + quant = False + return self._boundary(plan.attn_mc_ar, out, residual, norm_w, quant=quant) + quant = plan.qkv_prequant and i + 1 < core.num_layers + path = moe_path(num_tokens) + if path is None: + uncovered( + MC_FINALIZE.path, + f"more than {m3_mc_allreduce.max_rows()} rows: the finalized MoE and the boundary all-reduce run", + ) + moe_push = None + if path == MOE_ENGINE.path: + engine = core.engine(i, num_tokens, shared=True) + if engine is not None and plan.moe_mc_ar and decode_graph: + moe_push = self._moe_push_targets(num_tokens) + if path == MC_FINALIZE.path and plan.moe_mc_ar: + if self._bucket_flags(num_tokens) is None: + self._no_bucket(MC_FINALIZE.path, num_tokens) + else: + (shared, fc2, scales, perm), _, _, _ = self._moe( + i, hidden, return_parts=False, do_finalize=False + ) + fused = m3_mc_allreduce.m3_mc_finalize_allreduce_norm( + fc2, + perm.view(num_tokens, -1).to(torch.int32), + scales.reshape(num_tokens, -1), + shared, + residual, + norm_w, + core.rms_eps, + m3_mc_allreduce.get_workspace(core.mapping, core.hidden), + use_pdl=get_env_enable_pdl(), + quant=quant, + ) + if fused is None: + raise RuntimeError( + "the MC finalize all-reduce rejected the unfinalized MoE outputs" + ) + return fused + shared, routed, engine_pushed, shared_pushed = self._moe( + i, hidden, return_parts=True, push=moe_push + ) + if moe_push is not None: + if routed is None: + if engine_pushed: + return self._prepushed(i, shared, residual, norm_w, quant, 2) + else: + if engine_pushed and shared_pushed: + return self._prepushed(i, shared, residual, norm_w, quant, 3, x2=routed) + if engine_pushed or shared_pushed: + raise RuntimeError("only one MoE part pushed into the boundary all-reduce") + missed( + MOE_PUSH.path, + num_tokens, + "the MoE did not push its output into the armed boundary all-reduce", + ) + return self._boundary(plan.moe_mc_ar, shared, residual, norm_w, x2=routed, quant=quant) + + def _swiglu_mlp(self, h: torch.Tensor, prefix: str, push: Optional[tuple] = None): + """A gated MXFP8 MLP (the dense MLP, the shared expert): gate_up, SwiGLU-OAI, down, no reduction. Returns + (output, whether the down projection's GEMV pushed into ``push``).""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s") + n, k = core.w[f"{prefix}_dn_w"].shape + rows = gu.shape[0] + if rows in SWQ_ROWS.get((n, k), ()) and _mx_mode(core.mxfp8).decode_graph: + act, act_scale = m3_swq(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + m3_prequant.SLOT.publish(gu, act, act_scale) + if rows not in _SWQ_LOGGED: + _SWQ_LOGGED.add(rows) + logger.info(f"[M3 dispatch] SWQ: SwiGLU + MXFP8 producer rows={rows} [{n}, {k}]") + return _linear(core, gu, f"{prefix}_dn_w", f"{prefix}_dn_s", push) + act = _swiglu_act(core, gu) + return _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s", push) + + def _routed_chunks(self, i: int, x, router_logits): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: the MoE + scheduler's even split, even chunks on the MoE chunking stream.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + if num_chunks <= 1: + return _routed_experts(core, i, x, None, router_logits, True) + sizes = _split_chunk(rows, num_chunks) + xs, logits = (x.split(sizes), router_logits.split(sizes)) + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True)) + else: + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True)) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor, do_finalize: bool): + """Router + routed experts: the fused router with the NVFP4 activation up to 48 rows, else the router logits and + the experts quantizing their bf16 input.""" + core = self.core + fused = _router_fp4(core, i, h) + if fused is not None: + router_logits, x_fp4, x_sf = fused + return _routed_experts(core, i, x_fp4, x_sf, router_logits, do_finalize) + router_logits = _router_logits(core, i, h) + if not do_finalize: + return _routed_experts(core, i, h, None, router_logits, False) + return self._routed_chunks(i, h, router_logits) + + def _moe( + self, i: int, h: torch.Tensor, *, return_parts: bool, do_finalize: bool = True, push=None + ): + """The MoE block of layer ``i`` without its reduction. + + ``push`` = (the engine's ``arp`` descriptor, the shared down projection's ``ar`` tuple) the decode layer armed + for the MoE boundary's exchange sets, or None. Returns ``(out, routed, engine_pushed, shared_pushed)``: with + ``return_parts`` the shared and routed parts (routed None when the engine folded the shared expert in), else + their sum in ``out``; with ``do_finalize=False`` ``out`` is ``[shared, fc2 out, expert scales, permuted + index]``. + """ + core = self.core + engine_arp, shared_ar = push if push is not None else (None, None) + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + do_finalize + and h.dim() == 2 + and (h.dtype == torch.bfloat16) + and (1 <= rows <= MOE_ENGINE_MAX_TOKENS) + and (rows >= MOE_FOLD_MIN_TOKENS) + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + folded = m3_moe_engine(h.contiguous(), engine, None, engine_arp) + return (folded, None, engine_arp is not None, False) + routed_out, (shared_out, shared_pushed) = maybe_execute_in_parallel( + lambda: self._routed(i, h, do_finalize), + lambda: self._swiglu_mlp(h, f"l{i}_sh", shared_ar), + core.moe_events[i][0], + core.moe_events[i][1], + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if not do_finalize: + return ([shared_out, *routed_out], None, False, shared_pushed) + if return_parts: + return (shared_out, routed_out, False, shared_pushed) + return (shared_out.add_(routed_out), None, False, shared_pushed) + + def _generic_boundary(self, x, x2, residual, norm_w): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce.""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _generic_layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + else: + hidden, residual = _native_allreduce_norm( + core, hidden, residual, core.w[f"l{i}_post_norm"] + ) + if i in core.moe_layers: + out, routed, _, _ = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out, _ = self._swiglu_mlp(hidden, f"l{i}_mlp") + routed = None + normed, updated = self._generic_boundary(out, routed, residual, _next_norm(core, i)) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + capture(spec_metadata, i, updated, None) + return (normed, updated) + + def _attention(self, i: int, position_ids, hidden, step, push: Optional[tuple] = None): + """Attention of layer ``i`` up to its o_proj (no reduction). ``push`` is the attention boundary's exchange the + o_proj may write into. Returns (o_proj output, pushed).""" + core = self.core + if i not in core.sparse_layers: + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s") + out_fp8 = core.main_kv_is_fp8 or core.main_kv_is_nvfp4 + fused = _qk_norm_rope(core, i, qkv, position_ids, out_fp8) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", push) + if not core.fused_projection: + return self._unfused_sparse_attention(i, position_ids, hidden, step, push) + packed, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", defer=QKV_PRODUCER_FOLD) + deferred = None + if isinstance(packed, _DeferredGemv): + deferred, packed = (packed, packed.placeholder) + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + deferred, + early_trigger=_producer_early_trigger(step), + ) + if horizontal is None: + raise RuntimeError( + ( + "MiniMax-M3 sparse layer " + f"{i}" + ": no horizontal producer takes this step's caches (an FP8 or NVF" + "P4 HND pool with an FP8 index-K cache)" + ) + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", push) + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step, push): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output. ``push`` is the attention + boundary exchange the o_proj GEMV writes its rows into, as in the five-way path.""" + core = self.core + if _producer_covered(core): + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s") + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", push) + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): a sparse + layer whose caches the producer wrote runs this target's selection and sparse decode when the generation span + starts at row 0; everything else the MSA backend (``m3_msa_attention``: the KV write unless a producer wrote the + caches, the sparse layers' top-k block selection, the FMHA).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + live = int(step["num_tokens"]) + q_live, out_live = (q[:live], output[:live]) + k_live = k[:live] if k is not None else None + v_live = v[:live] if v is not None else None + idx_q_live = idx_q[:live] if idx_q is not None else None + span = step["msa_decode_span"] + sparse = i in core.sparse_layers + if sparse and k_live is None and (span is not None) and (span.row_first == 0): + self._decode_sparse_core(i, q_live, idx_q_live, step, out_live) + return output + nvfp4 = core.main_kv_is_nvfp4 + m3_msa_attention( + core.attn[i], + q_live, + k_live, + v_live, + idx_q_live, + step["metadata"], + out_live, + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + return output + + def _decode_sparse_core(self, i: int, q, idx_q, step, output) -> None: + """Top-16 block selection and the block-sparse decode of one pure generation step of a sparse layer whose caches + the producer already wrote. + + The selection is the identity when the metadata staged it (every request within 16 blocks). Otherwise a span of + at most ``FUSED_SCORE_SELECT_MAX_REQUESTS`` requests scores and selects in one launch; a wider span runs the + CuTe DSL scorer (the per-index-head block maxima, reduced to the KV head) and the long-row selector (the + built-in selector beyond its shapes). + """ + core = self.core + backend = core.attn[i] + config = backend.m3_config + num_tokens = int(q.shape[0]) + rows = int(step["num_seqs"]) + qlen = int(step["msa_decode_span"].query_len) + block_table = step["msa_block_table"][:rows] + seq_lens = step["msa_seq_lens_cuda"][:rows] + num_kv_heads = int(backend.indexer.config.num_kv_heads) + identity = step["msa_short_ok"] and step["msa_short_table"] is not None + fused = ( + None + if identity or False + else self._fused_score_select(i, idx_q, step, rows, qlen, num_tokens) + ) + if identity: + topk = step["msa_short_table"][: num_kv_heads * num_tokens * MSA_REQUIRED_TOPK].view( + num_kv_heads, num_tokens, MSA_REQUIRED_TOPK + ) + elif fused is not None: + topk = fused.permute(1, 0, 2) + else: + for _rep in range(1): + idx_q_view = idx_q.view(num_tokens, config.num_index_heads, config.sparse_index_dim) + idx_k_cache = step["msa_idx_k_cache"](i) + max_score = step["msa_proxy_max_score_view"]( + config.num_index_heads, step["msa_worst_case_max_k_tiles"], num_tokens + ) + early = _index_score_early(rows, int(core.mapping.tp_rank)) + group = config.num_index_heads // config.num_kv_heads + key = ( + idx_q_view.dtype, + int(config.num_index_heads), + int(qlen), + int(idx_q_view.shape[-1]), + int(INDEX_SCORE_SPLIT_K), + bool(early), + ) + if key not in _index_score_kernel.IndexScoreRunner.kernel_cache: + missed( + INDEX_SCORE.path, + num_tokens, + f"the index scorer {key} was not compiled before warmup", + ) + score_view = max_score.transpose(1, 2) + m3_index_score( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens, + score_view, + qlen, + INDEX_SCORE_SPLIT_K, + early, + ) + n_valid_blocks = step["msa_n_valid_blocks"][:num_tokens] + max_score_kv = ( + max_score.view( + config.num_kv_heads, group, max_score.shape[1], max_score.shape[2] + ).amax(dim=1) + if group > 1 + else max_score + ) + early_trigger = num_tokens <= SELECT_EARLY_MAX_ROWS + dsl = False + if LONG_SELECT_DSL.keys: + dsl_key = _select_long_dsl_kernel.kernel_key( + get_env_enable_pdl(), early_trigger, 0 + ) + dsl = dsl_key in _select_long_dsl_kernel.kernel_cache + if dsl: + _note_select_dsl(int(core.mapping.tp_rank)) + else: + missed( + LONG_SELECT_DSL.path, + num_tokens, + f"the 384-thread select {dsl_key} was not compiled before warmup", + ) + table = long_select_blocks( + max_score_kv, + n_valid_blocks=n_valid_blocks, + init_blocks=config.init_blocks, + local_blocks=config.local_blocks, + use_pdl=get_env_enable_pdl(), + early_trigger=early_trigger, + dsl=dsl, + ) + if table is None: + uncovered( + LONG_SELECT.path, + ( + f"{max_score_kv.shape[1]}" + " KV blocks, outside " + f"{LONG_SELECT_MIN_BLOCKS}" + ".." + f"{LONG_SELECT_MAX_BLOCKS}" + ": the built-in selector" + ), + ) + table = m3_select_blocks( + max_score_kv, + MSA_REQUIRED_TOPK, + n_valid_blocks, + config.init_blocks, + config.local_blocks, + True, + ) + topk = table.permute(1, 0, 2) + manager = step["kv_cache_manager"] + k_paged, v_paged = msa_paged_kv(manager, i) + nvfp4 = {} + if manager.is_nvfp4_layer(i): + k_scale, v_scale = _aligned_nvfp4_dequant_scales(backend, core.kv_scales[i]) + scales = manager.get_block_scale_buffers(i, "HND") + nvfp4 = dict( + k_block_scale=scales[:, 0], + v_block_scale=scales[:, 1], + k_global_scale=k_scale, + v_global_scale=v_scale, + ) + m3_sparse_decode( + q.view(num_tokens, core.heads, core.head_dim), + k_paged, + v_paged, + topk, + block_table, + seq_lens, + sm_scale=core.sm_scale(i), + output=output.view(num_tokens, core.heads, core.head_dim), + decode_query_len=qlen, + identity_blocks=identity, + **nvfp4, + ) + + def _fused_score_select(self, i: int, idx_q, step, rows: int, qlen: int, num_tokens: int): + """The top-16 table [total_q, num_kv_heads, 16] of a covered decode span from the fused score + select, or None + (after a miss when the span is covered) for the scorer + selector.""" + core = self.core + path = FUSED_SCORE_SELECT.path + if not FUSED_SCORE_SELECT.covers(rows): + uncovered( + path, f"more than {FUSED_SCORE_SELECT_MAX_REQUESTS} requests: scorer + selector" + ) + return None + if qlen > FUSED_SCORE_SELECT_MAX_QUERY_LEN: + uncovered( + path, + f"{qlen} query rows per request (more than {FUSED_SCORE_SELECT_MAX_QUERY_LEN}): scorer + selector", + ) + return None + config = core.attn[i].m3_config + idx_q_view = idx_q.view(num_tokens, config.num_index_heads, config.sparse_index_dim) + idx_k_pages = step["msa_idx_k_cache"](i).squeeze(1) + max_tiles = int(step["msa_worst_case_max_k_tiles"]) + n_valid = step["msa_n_valid_blocks"] + reason = _fused_score_select_kernel.supported( + idx_q_view, idx_k_pages, qlen, config.num_kv_heads, max_tiles + ) + if reason is None and ( + num_tokens != rows * qlen + or n_valid is None + or n_valid.dtype != torch.int32 + or (int(n_valid.shape[0]) < num_tokens) + ): + reason = "the step's query rows or valid-block counts are not one decode span" + if reason is not None: + missed(path, rows, reason) + return None + bufs = _fused_score_select_kernel.buffers(idx_q.device, max_tiles) + if bufs is None: + capture_missed(path, rows, "the fused score + select scratch") + return None + return _fused_score_select_kernel.fused_score_select( + idx_q_view, + idx_k_pages, + step["msa_block_table"][:rows], + step["msa_seq_lens_cuda"][:rows], + n_valid[:num_tokens].contiguous(), + decode_query_len=qlen, + num_kv_heads=config.num_kv_heads, + init_blocks=config.init_blocks, + local_blocks=config.local_blocks, + max_tiles=max_tiles, + scratch=bufs[0], + counters=bufs[1], + ) + + +class M3SpecStepFold: + """The target's speculative step-input staging (the in-tree ``SpecStepFold`` protocol). + + Between two decode graphs of a one-model Eagle3 step, the spec sampler stores the step's outputs into its per-slot + store and the engine gathers them back into its flat input buffers for the next step. This object does both in + one launch each (``m3_spec_tail.slot_scatter`` / ``step_gather``): integer copies only, the same values the + engine's and the sampler's own copies write. The forward never stores the sampler's outputs itself, so + ``begin_step`` and ``sampler_rows_stored`` always report that nothing was staged. + """ + + def __init__(self) -> None: + self.store: Optional[object] = None + self.step_inputs: Optional[tuple] = None + + def bind_engine_buffers( + self, input_ids, draft_tokens, position_id_offsets, kv_lens_offsets, max_batch_size + ): + self.step_inputs = (input_ids, draft_tokens, position_id_offsets, kv_lens_offsets) + + def bind_sampler_store(self, store: object) -> None: + self.store = store + + def begin_step(self, seq_slots: Optional[list], num_rows: int) -> bool: + return False + + def gather( + self, + new_tokens, + next_draft_tokens, + new_tokens_lens, + previous_slots, + previous_pos_indices, + previous_batch_len: int, + tokens_per_gen_step: int, + draft_token_buffer_width: int, + num_tokens: int, + num_draft_tokens: int, + first_previous_row: int, + ) -> bool: + if self.step_inputs is None: + return False + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets = self.step_inputs + return m3_spec_tail.step_gather( + new_tokens, + next_draft_tokens, + new_tokens_lens, + previous_slots, + previous_pos_indices, + previous_batch_len, + tokens_per_gen_step, + draft_token_buffer_width, + input_ids, + num_tokens, + draft_tokens, + num_draft_tokens, + position_id_offsets, + kv_lens_offsets, + first_previous_row, + ) + + def sampler_rows_stored(self, outputs: dict, num_skip: int, seq_slots: list) -> bool: + return False + + def scatter( + self, seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens + ) -> bool: + store = self.store + if store is None: + return False + return m3_spec_tail.slot_scatter( + seq_slots, + new_tokens, + store.new_tokens.squeeze(-1).T, + next_new_tokens, + store.next_new_tokens.squeeze(-1).T, + new_tokens_lens, + store.new_tokens_lens, + next_draft_tokens, + store.next_draft_tokens, + ) + + +class M3DSparkStepFold(M3SpecStepFold): + """The step fold under the standalone DSpark worker (DSFOLD): the engine's previous-batch gathers in one launch + (``step_gather``). ``taken`` / ``declined`` count the steps the gather ran / fell back to the engine's own copies. + + With ``boundary_pack`` (BOUNDARY_PACK BP-1) a batch-1 decode graph also ends with the sampler's slot stores and the + next step's gather (``graph_tail``: one ``ops/m3_bp_tail`` launch), so the boundary between two replays runs + neither: + + * ``begin_step`` (before every replay) writes the device control word (slot, live) when it changed: live 1 with + the step's slot when the step is one row the fold identifies (a slot inside the store), else live 0, so a replay + the fold did not take (a dummy-only replay, an unidentified step) writes nothing at all; it reports the inputs + staged when the previous graph gathered them for exactly these rows; + * the graph's tail waits for the sampler's previous side-stream store copies (``store_copies_done``), then with live + 1 stores the row as the sampler's ``_store_outputs`` (same output row, same store, zero padding / truncation to + the store widths, at the slot) and, with a static draft length, writes the engine's next-step gathers of that row + (P = 1, ids0 = d0 = b0 = 0, both offset buffers zeroed elsewhere), and marks its outputs; + * ``sampler_rows_stored`` (after the replay) accepts a marked step whose rows are the live ones, and arms the next + ``begin_step``. + + Every other step (context or mixed steps, batches over one row, eager forwards, a draft length that can change, + first steps, slots the fold could not identify) keeps the sampler's stores and the engine's gathers. + ``bp_stores`` / ``bp_gathers`` count the steps whose stores / gathers the graph took.""" + + store_copies_done: Optional[torch.cuda.Event] = None + + def __init__(self, boundary_pack: bool = False, tokens_per_step: Optional[int] = None) -> None: + super().__init__() + self.taken = 0 + self.declined = 0 + self.boundary_pack = boundary_pack + self.tokens_per_step = tokens_per_step + self.bp_stores = 0 + self.bp_gathers = 0 + self._bp_ctl: Optional[torch.Tensor] = None + self._bp_ctl_host: Optional[list] = None + self._bp_block = 0 + self._bp_rows: Optional[list] = None + self._bp_staged: Optional[list] = None + self._bp_token = id(self) + + def bind_engine_buffers( + self, input_ids, draft_tokens, position_id_offsets, kv_lens_offsets, max_batch_size + ): + super().bind_engine_buffers( + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets, max_batch_size + ) + self._bp_reset() + + def bind_sampler_store(self, store: object) -> None: + super().bind_sampler_store(store) + self._bp_reset() + + def _bp_reset(self) -> None: + """A new binding (an engine or a sampler built again): BOUNDARY_PACK's buffers are made again for it.""" + self._bp_ctl = self._bp_ctl_host = None + self._bp_rows = self._bp_staged = None + self.store_copies_done = None + + def gather(self, *args, **kwargs) -> bool: + if not super().gather(*args, **kwargs): + self.declined += 1 + uncovered( + "DSFOLD", + "step_gather declined the inputs (dtype / layout); the engine's copies run", + ) + return False + if self.taken == 0: + logger.info( + "[M3 dispatch] DSFOLD DSpark step-input gather: engaged (one step_gather launch)" + ) + self.taken += 1 + return True + + def scatter( + self, seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens + ) -> bool: + return False + + def _bp_ready(self) -> bool: + """BOUNDARY_PACK: the control word (live 0), the tail kernel's two variants and the store-copies event, made + once outside capture when the sampler store and the engine buffers are bound and the tail can take them.""" + if not self.boundary_pack: + return False + if self._bp_ctl is not None: + return True + store = self.store + names = ("new_tokens", "next_new_tokens", "new_tokens_lens", "next_draft_tokens") + if ( + self.step_inputs is None + or not all((isinstance(getattr(store, name, None), torch.Tensor) for name in names)) + or torch.cuda.is_current_stream_capturing() + ): + return False + tensors = [getattr(store, name) for name in names] + list(self.step_inputs) + if ( + any((t.dtype != torch.int32 or not t.is_cuda for t in tensors)) + or not all((t.is_contiguous() for t in self.step_inputs)) + or (not m3_bp_tail.aligned(*tensors)) + ): + uncovered( + "BOUNDARY_PACK", + "the store or the engine buffers are not aligned int32 CUDA tensors", + ) + return False + ctl = torch.zeros(2, dtype=torch.long, device=store.new_tokens_lens.device) + block = m3_bp_tail.block_for(store) + m3_bp_tail.warm_up(ctl, store, self.step_inputs, block) + done = torch.cuda.Event(external=True) + done.record() + self.store_copies_done = done + self._bp_ctl, self._bp_ctl_host, self._bp_block = (ctl, [0, 0], block) + return True + + def begin_step(self, seq_slots: Optional[list], num_rows: int) -> bool: + staged, self._bp_staged = (self._bp_staged, None) + self._bp_rows = None + if not self._bp_ready(): + return False + num_slots = self.store.new_tokens_lens.shape[0] + rows = list(seq_slots) if seq_slots is not None else None + live = ( + rows is not None + and len(rows) == num_rows + and (0 < num_rows <= BOUNDARY_PACK_MAX_ROWS) + and all((isinstance(slot, int) and 0 <= slot < num_slots for slot in rows)) + ) + ctl = [rows[0], 1] if live else [self._bp_ctl_host[0], 0] + if ctl != self._bp_ctl_host: + self._bp_ctl.copy_( + torch.tensor(ctl, dtype=torch.long, pin_memory=prefer_pinned()), non_blocking=True + ) + self._bp_ctl_host = ctl + if not live: + return False + self._bp_rows = rows + if staged != rows: + return False + if self.bp_gathers == 0: + logger.info( + "[M3 dispatch] BOUNDARY-PACK BP-1: sampler stores + step gather in the step graph (taken)" + ) + self.bp_gathers += 1 + return True + + def sampler_rows_stored(self, outputs: dict, num_skip: int, seq_slots: list) -> bool: + self._bp_staged = None + mark = outputs.get(_BOUNDARY_PACK_MARK) if isinstance(outputs, dict) else None + if mark is None or mark[0] != self._bp_token or self._bp_rows is None: + return False + rows, gathered = (mark[1], mark[2]) + if num_skip != 0 or len(seq_slots) != rows or list(seq_slots) != self._bp_rows: + uncovered( + "BOUNDARY_PACK", + ( + "a step graph stored rows that are not the sampler's (skipped con" + "text rows or other slot ids); the sampler stores them" + ), + ) + return False + self.bp_stores += 1 + if gathered: + self._bp_staged = list(seq_slots) + return True + + def graph_tail(self, outputs, attn_metadata): + """BOUNDARY_PACK, at the end of a decode step's forward: under capture of a batch-1 generation graph, the + sampler's slot stores and the next step's gather, gated by the live word (see the class docstring); returns + the outputs, marked when the graph has the tail. Anything else returns them unchanged (the eager path).""" + if not self.boundary_pack or not torch.cuda.is_current_stream_capturing(): + return outputs + path = "BOUNDARY_PACK" + if not isinstance(outputs, dict) or self._bp_ctl is None: + uncovered( + path, + "the sampler store or the engine buffers are not bound: eager stores and gathers", + ) + return outputs + if attn_metadata.num_contexts != 0: + return outputs + names = ("new_tokens", "next_new_tokens", "new_tokens_lens", "next_draft_tokens") + srcs = [outputs.get(name) for name in names] + if any( + ( + not isinstance(t, torch.Tensor) or t.dtype != torch.int32 or (not t.is_cuda) + for t in srcs + ) + ) or [t.dim() for t in srcs] != [2, 2, 1, 2]: + uncovered( + path, + "the worker outputs are not int32 [rows, width] / [rows]: eager stores and gathers", + ) + return outputs + rows = srcs[0].shape[0] + if any((t.shape[0] != rows for t in srcs)): + uncovered(path, "the worker outputs disagree on their rows: eager stores and gathers") + return outputs + if not 0 < rows <= BOUNDARY_PACK_MAX_ROWS: + uncovered( + path, + f"a {rows}-row decode graph (only batch-1 graphs, which are never padded): eager stores and gathers", + ) + return outputs + src = [t[:1] for t in srcs] + if not m3_bp_tail.aligned(*src): + uncovered(path, "a worker output is not 16-byte aligned: eager stores and gathers") + return outputs + store = self.store + num_tokens = int(attn_metadata.num_tokens) + t = num_tokens // rows + gather = ( + self.tokens_per_step is not None + and t == self.tokens_per_step + and (num_tokens == rows * t) + and (t <= store.next_new_tokens.shape[0]) + and (t - 1 <= store.next_draft_tokens.shape[1]) + ) + if not gather: + uncovered( + path, + ( + "the next step's tokens per request may differ from this graph's " + f"{t}" + " (a draft length that can change, or " + f"{num_tokens}" + " tokens over " + f"{rows}" + " rows): the graph stores, the engine gathers" + ), + ) + torch.cuda.current_stream().wait_event(self.store_copies_done) + m3_bp_tail.bp_tail(self._bp_ctl, src, store, self.step_inputs, t, gather, self._bp_block) + marked = dict(outputs) + marked[_BOUNDARY_PACK_MARK] = (self._bp_token, rows, gather) + return marked + + +def _static_tokens_per_step(spec_config) -> Optional[int]: + """BOUNDARY_PACK: the tokens per request of every decode step when the draft length cannot change at runtime (no + draft-length schedule, no acceptance-rate gate that turns speculation off), else None.""" + if getattr(spec_config, "draft_len_schedule", None) is not None: + return None + if ( + getattr(spec_config, "acceptance_rate_window_size", None) + and getattr(spec_config, "acceptance_rate_threshold", None) is not None + ): + return None + from tensorrt_llm._torch.speculative.utils import get_static_draft_len + + return int(spec_config.get_runtime_tokens_per_gen_step(get_static_draft_len(spec_config))) + + +_DRAFT_LOWM_BUILT: set = set() + + +def _draft_linear(x: torch.Tensor, weight: torch.Tensor) -> torch.Tensor: + """A drafter BF16 linear ``x @ weight^T``: FlashInfer's fixed-tactic low-M kernel on the rows ``DRAFT_LOWM`` + covers (compiled before warmup; a covered call found uncompiled is a miss), else the in-tree BF16 GEMM.""" + n, k = weight.shape + m = int(x.shape[0]) + path = DRAFT_LOWM.path + if x.dim() != 2 or not DRAFT_LOWM.covers(((n, k), m)): + uncovered(path, f"the drafter's {n}x{k} linear past its tactic table's rows") + elif not x.is_contiguous(): + uncovered(path, "a strided activation (the in-tree GEMM reads it in place)") + elif x.dtype != torch.bfloat16 or weight.dtype != torch.bfloat16: + missed(path, m, "the activation or weight is not BF16") + elif (n, k, m, get_env_enable_pdl()) not in _DRAFT_LOWM_BUILT: + missed(path, m, f"the {n}x{k} kernel at {m} rows was not compiled before warmup") + else: + kind, tactic = tactic_for(n, k, m) + out = torch.empty((m, n), dtype=x.dtype, device=x.device) + return m3_draft_lowm(x, weight, out, kind, tactic, get_env_enable_pdl()) + return m3_bf16_linear(x, weight) + + +def _prebuild_draft_lowm(draft: "M3Eagle3Draft") -> int: + """Compile (and run once) every covered row count of the drafter's linears, outside capture, on the current + stream. Returns the number of calls compiled.""" + use_pdl = get_env_enable_pdl() + built = 0 + shapes = {tuple(draft.w[key].shape): draft.w[key] for key in ("qkv", "o", "gu", "dn", "fc")} + for (n, k), weight in sorted(shapes.items()): + for _, m in sorted((key for key in DRAFT_LOWM.keys if key[0] == (n, k))): + if (n, k, m, use_pdl) in _DRAFT_LOWM_BUILT: + continue + kind, tactic = tactic_for(n, k, m) + x = torch.zeros(m, k, dtype=torch.bfloat16, device=weight.device) + out = torch.empty(m, n, dtype=torch.bfloat16, device=weight.device) + m3_draft_lowm(x, weight, out, kind, tactic, use_pdl) + _DRAFT_LOWM_BUILT.add((n, k, m, use_pdl)) + built += 1 + return built + + +class _DraftForward: + """``draft_model.model`` for the in-tree worker: the drafter layer's forward, called as + ``model(input_ids=, position_ids=, hidden_states=, attn_metadata=, spec_metadata=, all_rank_num_tokens=)`` and + returning (hidden, hidden to carry to the next pass). ``d2t`` None: the draft vocabulary is the target's.""" + + d2t = None + + def __init__(self, draft: "M3Eagle3Draft") -> None: + self._draft = draft + + def __call__(self, **kwargs): + return self._draft.forward_layer(**kwargs) + + +class M3Eagle3Draft(nn.Module): + """The one-layer BF16 Eagle3 drafter (``speculative_model``): its weights (``weights.DRAFT_WEIGHTS``), its + vocab-parallel BF16 head and logits processor, the drafter layer's TRTLLM attention backend (layer index = the + target's layer count, FP8 KV) and its TP all-reduce. + + What the worker reads: ``model`` (the layer forward), ``apply_eagle3_fc`` (the per-layer fc_norms + fc over the + captured target hidden states), ``lm_head`` and ``logits_processor``; the fused draft loop reads the layer's parts + (``draft_input_weights``, ``attention``, ``mlp``, ``ar_norm``). The drafter embeds with the target's table. + """ + + def __init__( + self, + core: MiniMaxM3Tp8ModelingV2, + draft_config: ModelConfig, + start_layer_idx: int, + capture_layers: int, + ) -> None: + super().__init__() + cfg = draft_config.pretrained_config + _ensure_draft_vocab_size(cfg) + eagle = getattr(cfg, "eagle_config", None) or {} + mapping = draft_config.mapping + self.draft_config = draft_config + self.model_config = draft_config + self.config = cfg + self.capture_layers = int(capture_layers) + self.layer_idx = int(start_layer_idx) + self.dtype = cfg.torch_dtype + self.hidden = int(cfg.hidden_size) + self.hidden_size_in = int(getattr(cfg, "target_hidden_size", cfg.hidden_size)) + self.rms_eps = float(cfg.rms_norm_eps) + self.return_hidden_post_norm = bool( + eagle.get("return_hidden_post_norm", False) or getattr(cfg, "norm_output", False) + ) + if ( + int(cfg.num_hidden_layers) != 1 + or eagle.get("eh_proj_before_attn", False) + or eagle.get("norm_before_fc", False) + or (not getattr(cfg, "fc_norm", False)) + or getattr(cfg, "kv_lora_rank", None) + or (self.capture_layers <= 1) + or (self.hidden_size_in != self.hidden) + or (cfg.draft_vocab_size != cfg.vocab_size) + or getattr(cfg, "attention_bias", False) + or getattr(cfg, "mlp_bias", False) + or getattr(cfg, "bias", False) + or getattr(cfg, "tie_word_embeddings", False) + ): + raise ValueError( + ( + "this MiniMax-M3 target drafts with a one-layer Llama-style Eagle" + "3 drafter with per-layer fc_norms over several captured layers, " + "the target's hidden size and vocabulary and no biases" + ) + ) + tp = mapping.tp_size + self.heads = cfg.num_attention_heads // tp + self.kv_heads = (cfg.num_key_value_heads + tp - 1) // tp + self.head_dim = int( + getattr(cfg, "head_dim", None) or cfg.hidden_size // cfg.num_attention_heads + ) + self.q_size = self.heads * self.head_dim + self.kv_size = self.kv_heads * self.head_dim + self.w = _weights.DRAFT_WEIGHTS.declare(_weights.DRAFT_WEIGHTS.dims(self)) + self.lm_head = LMHead( + cfg.draft_vocab_size, + cfg.hidden_size, + dtype=cfg.torch_dtype, + mapping=mapping, + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=draft_config.lm_head_gather_output, + reduce_output=False, + use_custom_cublas_mm=False, + quant_config=None, + ) + self.logits_processor = LogitsProcessor() + self.allreduce = AllReduce( + mapping=mapping, strategy=draft_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + quant_config = draft_config.get_quant_config() + self.attn = create_attention( + draft_config.attn_backend, + self.layer_idx, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=PositionalEmbeddingParams( + type=PositionEmbeddingType.rope_gpt_neox, rope=RopeParams.from_config(cfg) + ), + quant_config=quant_config, + skip_create_weights_in_init=draft_config.skip_create_weights_in_init, + q_scaling=1.0, + attention_chunk_size=None, + sparse_params=None, + ) + if not self.attn.support_fused_rope() or not self.attn.support_fused_qkv(): + raise ValueError( + f"the drafter's attention backend {type(self.attn).__name__} must fuse RoPE and QKV" + ) + self.attn.update_quant_config(quant_config) + self._core = [core] + self.model = _DraftForward(self) + + @property + def embed_table(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def load_weights_from_target_model(self, target_model: nn.Module) -> None: + """Nothing to alias: the drafter reads the target's embedding table through the core and owns its head.""" + + def fc(self, hs: torch.Tensor) -> torch.Tensor: + return _draft_linear(hs, self.w["fc"]) + + def apply_eagle3_fc(self, hidden_states: torch.Tensor) -> torch.Tensor: + """The captured target hidden states [T, layers x H] to the drafter's input width: each layer's fc_norm, + then the fc (the in-tree ``Eagle3ForCausalLM.apply_eagle3_fc``).""" + hidden_states = hidden_states.to(self.dtype) + if hidden_states.shape[-1] != self.hidden: + chunks = hidden_states.chunk(self.capture_layers, dim=-1) + hidden_states = torch.cat( + [ + flashinfer_rmsnorm(c, self.w[f"fc_norm{i}"], self.rms_eps) + for i, c in enumerate(chunks) + ], + dim=-1, + ) + hidden_states = self.fc(hidden_states) + return hidden_states + + def attention(self, x: torch.Tensor, md, reduce: bool) -> torch.Tensor: + """QKV, the backend's fused RoPE + KV append + causal attention, o_proj (all-reduced unless ``reduce`` is + False, where the caller fuses the reduction).""" + qkv = _draft_linear(x, self.w["qkv"]) + o = m3_draft_attention(self.attn, qkv, md) + out = _draft_linear(o, self.w["o"]) + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def mlp(self, h: torch.Tensor, reduce: bool) -> torch.Tensor: + gu = _draft_linear(h, self.w["gu"]) + act = m3_silu_and_mul(gu, None, None, None) + out = _draft_linear(act, self.w["dn"]) + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def ar_norm(self, x: torch.Tensor, residual: torch.Tensor, norm_key: str): + """One draft-layer boundary in the in-tree structure: the projection's plain all-reduce (BF16 out), then the + fused residual add + RMSNorm (FP32 add, norm of the FP32 sum). Returns (normed, residual sum).""" + return flashinfer_fused_add_rmsnorm( + m3_allreduce(x, self.allreduce, None), residual, self.w[norm_key], self.rms_eps + ) + + def forward_layer( + self, + input_ids: torch.Tensor, + position_ids: torch.Tensor, + hidden_states: torch.Tensor, + attn_metadata, + spec_metadata, + all_rank_num_tokens: Optional[List[int]] = None, + ): + """The drafter layer (the in-tree ``Eagle3DraftModel.forward`` for one layer): the embedded tokens and the + fc'd hidden states normed and concatenated, attention, the post-attention norm with the residual, the MLP and + the final norm. Returns (normed output, the hidden state the next pass carries).""" + previous = attn_metadata.all_rank_num_tokens + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = all_rank_num_tokens + try: + embeds = m3_embedding(input_ids, self.embed_table).to(self.dtype) + residual = hidden_states + h = flashinfer_rmsnorm(hidden_states, self.w["hidden_norm"], self.rms_eps) + e = flashinfer_rmsnorm(embeds, self.w["in_norm"], self.rms_eps) + h = torch.cat([e, h], dim=-1) + h = self.attention(h, attn_metadata, reduce=True) + h, residual = flashinfer_fused_add_rmsnorm( + h, residual, self.w["post_norm"], self.rms_eps + ) + h = self.mlp(h, reduce=True) + spec_metadata.maybe_capture_hidden_states(self.layer_idx, h, residual) + h, to_save = flashinfer_fused_add_rmsnorm( + h, residual, self.w["final_norm"], self.rms_eps + ) + return (h, h) if self.return_hidden_post_norm else (h, to_save) + finally: + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = previous + + +def _eagle3_capture(spec_metadata, layer_id: int, hidden_states, residual, capture_norm) -> None: + """The Eagle3 hidden-state capture of one target layer. + + Mirrors ``Eagle3OneModelSpecMetadata.maybe_capture_hidden_states``. With ``capture_norm`` = (the drafter's stacked + fc_norm weights [layers, H], eps), the capture writes rmsnorm(bf16(h + residual)) * fc_norm[i] in one kernel (the + in-tree capture's bf16 add, then the in-tree fc_norm arithmetic), which the fused first drafter inputs read as is; + otherwise it is the in-tree capture. + """ + if capture_norm is None: + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + return + for i, captured_layer_id in enumerate(spec_metadata.layers_to_capture): + if captured_layer_id == layer_id: + assert spec_metadata.hidden_states is not None + num_tokens = spec_metadata.num_capture_tokens + if num_tokens > spec_metadata.hidden_states.shape[0]: + raise RuntimeError( + ( + "EAGLE3 hidden-state capture token count exceeds buffer capacity:" + " num_tokens=" + f"{num_tokens}" + ", capacity=" + f"{spec_metadata.hidden_states.shape[0]}" + ) + ) + if num_tokens > hidden_states.shape[0]: + raise RuntimeError( + ( + "EAGLE3 hidden-state capture token count exceeds available hidden" + " states: num_tokens=" + f"{num_tokens}" + ", hidden_states=" + f"{hidden_states.shape[0]}" + ) + ) + hidden_size = spec_metadata.hidden_size + m3_spec_tail.capture_norm( + hidden_states[:num_tokens], + residual[:num_tokens] if residual is not None else None, + capture_norm[0][i], + capture_norm[1], + spec_metadata.hidden_states[:num_tokens, i * hidden_size : (i + 1) * hidden_size], + ) + break + + +def _sharded_head(lm_head: nn.Module) -> bool: + """A vocab-sharded TP head without padding (the fused kernels read the local shard).""" + return ( + getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and getattr(lm_head, "padding_size", 0) == 0 + and (getattr(lm_head, "tp_size", 1) > 1) + ) + + +class M3Eagle3Worker(Eagle3OneModelWorker): + """``Eagle3OneModelWorker`` with the MiniMax-M3 fused verify and fused draft loop, at the in-tree numerics. + + * Verify: on decode-only batches whose verify rows the fused kernels cover (``tail_eligible``), the target's BF16 + lm_head computes this rank's vocabulary shard of the logits (the in-tree head's own local GEMM) and one kernel + replaces the logits all-gather, the argmax (or the temperature + top-p draw) and the strict acceptance. + * Draft: the drafter's first pass may take the per-layer fc_norms in one launch (or folded into the hidden-state + capture) and run its fc beside the target head GEMM. Its BF16 linears run FlashInfer's fixed-tactic low-M + kernels at the rows ``DRAFT_LOWM`` covers (``_draft_linear``). On a decode-only step ``DRAFT_STEP`` covers, + the draft loop is fused: each pass's draft input in one launch, the drafter layer's attention and MLP with + their all-reduces fused with the residual add and the next RMSNorm, and one kernel for the BF16 head, the + cross-rank argmax and the pass's bookkeeping. + + Every other batch takes the in-tree paths. Methods whose docstring says "Mirrors" copy the named in-tree method + with the fused branches inserted. + """ + + def __init__(self, spec_config, mapping, use_separate_draft_kv_cache: bool = False) -> None: + super().__init__( + spec_config, mapping=mapping, use_separate_draft_kv_cache=use_separate_draft_kv_cache + ) + self._m3_first_gather_ids: Optional[torch.Tensor] = None + self._m3_fc_hidden: Optional[torch.Tensor] = None + self._m3_target_lm_head: Optional[nn.Module] = None + self._m3_fc_norm_w: Optional[tuple] = None + self._m3_capture_norm: Optional[tuple] = None + self._m3_aux_stream: Optional[torch.cuda.Stream] = None + self._m3_fc_early_drafter: Optional[M3Eagle3Draft] = None + self._m3_fc_early_stream: Optional[torch.cuda.Stream] = None + self._m3_force_vec_warm = False + self._m3_draft_head = None + self._m3_next_new: Optional[torch.Tensor] = None + self._m3_next_new_tokens: Optional[torch.Tensor] = None + self._m3_fused_first_inputs = False + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None + ) -> None: + """The Eagle3 capture with this step's capture norm; on a step armed for the early fc fork + (``arm_capture_norm``), the last captured layer then forks the drafter's first fc onto the aux stream, where + it overlaps the remaining target layers and the head (joined by ``_m3_target_accept``).""" + _eagle3_capture(spec_metadata, layer_id, hidden_states, residual, self._m3_capture_norm) + drafter = self._m3_fc_early_drafter + layers = spec_metadata.layers_to_capture + if drafter is not None and layers and (layer_id == layers[-1]): + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = self._m3_fc_fork( + spec_metadata, spec_metadata.num_capture_tokens, drafter + ) + + def prebuild(self, lm_head: nn.Module, draft: Optional[M3Eagle3Draft]) -> None: + """The fused draft loop's head copy, the drafter linears' low-M kernels and the drafter attention's KV-split + cap, from the loaded weights, before the executor's warmup, on every rank.""" + start = time.perf_counter() + if self.prebuild_draft_head(draft): + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] fused draft head built before warmup: %.1f s" + % (time.perf_counter() - start) + ) + start = time.perf_counter() + built = self.prebuild_drafter_linears(draft) + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] drafter low-M GEMMs built before warmup: %d calls, %.1f s" + % (built, time.perf_counter() - start) + ) + + def _forward_impl( + self, + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + resource_manager=None, + ): + """Mirrors Eagle3OneModelWorker._forward_impl, with the fused verify branch.""" + runtime_draft_len = spec_metadata.runtime_draft_len + if runtime_draft_len == 0: + return self.skip_drafting( + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + ) + batch_size = attn_metadata.num_seqs + num_contexts = attn_metadata.num_contexts + num_gens = batch_size - num_contexts + raw_logits = logits + self._m3_first_gather_ids = None + self._m3_next_new = None + self._m3_next_new_tokens = None + self._m3_fused_first_inputs = False + if logits is None: + accepted_tokens, num_accepted_tokens, self._m3_first_gather_ids = ( + self._m3_target_accept( + hidden_states, spec_metadata, batch_size, runtime_draft_len, draft_model + ) + ) + else: + self._execute_guided_decoder_if_present(logits) + accepted_tokens, num_accepted_tokens = self.sample_and_accept_draft_tokens( + input_ids, logits, attn_metadata, spec_metadata + ) + if self.draft_only_eligible(attn_metadata, spec_metadata, draft_model): + self._m3_seed_first_pass(num_accepted_tokens, spec_metadata, batch_size) + if self._is_mamba_hybrid_cache is None: + self._is_mamba_hybrid_cache = isinstance( + attn_metadata.kv_cache_manager, MambaHybridCacheManager + ) + if num_gens > 0 and self._is_mamba_hybrid_cache: + attn_metadata.kv_cache_manager.update_mamba_states( + attn_metadata=attn_metadata, + num_accepted_tokens=num_accepted_tokens, + state_indices=attn_metadata.mamba_metadata.state_indices, + ) + sa_manager = getattr(spec_metadata.spec_resource_manager, "sa_manager", None) + if self.sa_enhancer is not None and sa_manager is not None: + self.sa_enhancer.extend_and_prepare( + sa_manager=sa_manager, + request_ids=spec_metadata.request_ids, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_gens=num_gens, + num_contexts=num_contexts, + max_draft_len=runtime_draft_len, + ) + if self._uses_external_shared_target_kv: + next_draft_tokens = self._forward_external_shared_target_kv_draft_loop( + position_ids=position_ids, + hidden_states=hidden_states, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_contexts=num_contexts, + batch_size=batch_size, + ) + else: + self._prepare_attn_metadata_for_spec_dec(attn_metadata) + position_ids = position_ids.squeeze(0) + inputs = self.prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + original_all_rank_num_tokens = attn_metadata.all_rank_num_tokens + draft_kv_cache_manager = self.get_draft_kv_cache_manager(resource_manager) + next_draft_tokens = self._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + self._restore_attn_metadata_from_spec_dec(attn_metadata) + if original_all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = original_all_rank_num_tokens + if self._m3_next_new_tokens is not None: + next_new_tokens = self._m3_next_new_tokens + else: + next_new_tokens = self._prepare_next_new_tokens( + accepted_tokens, + next_draft_tokens, + spec_metadata.batch_indices_cuda, + batch_size, + num_accepted_tokens, + ) + attn_metadata.use_spec_decoding = True + if self._auxiliary_state_handlers: + self.commit_auxiliary_speculative_states( + num_accepted_tokens, + attn_metadata.mamba_metadata.state_indices[:batch_size], + num_contexts, + ) + return { + "logits": raw_logits, + "new_tokens": accepted_tokens, + "new_tokens_lens": num_accepted_tokens, + "next_draft_tokens": next_draft_tokens, + "next_new_tokens": next_new_tokens, + } + + def prepare_1st_drafter_inputs( + self, + input_ids, + position_ids, + hidden_states, + accepted_tokens, + attn_metadata, + spec_metadata, + draft_model, + ): + """The fused first draft inputs on a step that verified with the fused kernels (or was seeded for them), + else the in-tree ones.""" + fused = self._m3_first_drafter_inputs( + position_ids, + accepted_tokens, + input_ids.shape[0], + attn_metadata.num_contexts, + spec_metadata, + draft_model, + ) + self._m3_fused_first_inputs = fused is not None + if fused is not None: + fused.update(attn_metadata=attn_metadata, spec_metadata=spec_metadata) + return fused + return super().prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + + def _m3_structural_ok(self, attn_metadata, spec_metadata) -> bool: + """The batch conditions shared by the fused verify and the fused first drafter inputs: a decode-only plain + Eagle3 step without attention DP, guided decoding, relaxed acceptance or a suffix-automaton enhancer, of at + most DECODE_MAX_TOKENS verify rows. Host state only, so the TP ranks agree.""" + if self.is_mtp_eagle or self.use_dynamic_tree: + return False + if self._uses_external_shared_target_kv or self.sa_enhancer is not None: + return False + if ( + self.guided_decoder is not None + or self.mapping is None + or self.mapping.enable_attention_dp + ): + return False + if getattr(self.spec_config, "use_relaxed_acceptance_for_thinking", False): + return False + if attn_metadata.num_contexts != 0 or attn_metadata.num_seqs == 0: + return False + draft_len = spec_metadata.runtime_draft_len + if draft_len == 0: + return False + if attn_metadata.num_seqs * (draft_len + 1) > DECODE_MAX_TOKENS: + uncovered("spec_tail", f"more than {DECODE_MAX_TOKENS} verify rows") + return False + return not spec_metadata.wants_advanced_draft_sampling + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Whether this step verifies with the fused kernels instead of the gathered target logits. + + Decode-only Eagle3 batches (``_m3_structural_ok``) over an unpadded vocab-sharded bias-free target head, all + greedy at a request count ``SHARD_ACCEPT`` covers, or sampled under the temperature + top-p contract (see + _m3_sampled_verify_ok), with nothing else reading the target logits. Every input is host state shared by all + TP ranks, so the ranks agree. + """ + if not self._m3_structural_ok(attn_metadata, spec_metadata): + return False + if getattr(spec_metadata, "enable_penalty", False): + return False + if not _sharded_head(lm_head) or getattr(lm_head, "bias", None) is not None: + return False + draft_len = spec_metadata.runtime_draft_len + if spec_metadata.is_all_greedy_sample: + key = (draft_len, attn_metadata.num_seqs) + if not SHARD_ACCEPT.covers(key): + uncovered( + SHARD_ACCEPT.path, + f"draft length {draft_len} at {attn_metadata.num_seqs} requests", + ) + return False + elif attn_metadata.num_seqs * (draft_len + 1) > m3_sampled_accept_kernel.MAX_ROWS: + uncovered( + "spec_tail", + f"a sampled verify of more than {m3_sampled_accept_kernel.MAX_ROWS} rows", + ) + return False + elif not self._m3_sampled_verify_ok(spec_metadata): + return False + self._m3_target_lm_head = lm_head + return True + + def draft_only_eligible(self, attn_metadata, spec_metadata, draft_model) -> bool: + """Whether a step verified by the in-tree path can still take the fused first drafter inputs: the structural + conditions of tail_eligible, without the target-sampling ones.""" + return self._m3_structural_ok(attn_metadata, spec_metadata) and draft_model is not None + + def _m3_seed_first_pass(self, num_accepted_tokens, spec_metadata, batch_size: int) -> None: + """The first draft pass's gather rows (the last accepted row of each request) from the in-tree acceptance, as + _m3_target_accept leaves them.""" + k1 = spec_metadata.runtime_draft_len + 1 + n = num_accepted_tokens[:batch_size].to(torch.int64) + rows = spec_metadata.batch_indices_cuda[:batch_size].to(torch.int64) + self._m3_first_gather_ids = rows * k1 + n - 1 + + def _m3_first_drafter_inputs( + self, + position_ids, + accepted_tokens, + num_tokens: int, + num_contexts: int, + spec_metadata, + draft_model: Optional[M3Eagle3Draft], + ): + """The in-tree first draft inputs of a decode-only step, with the per-layer fc_norm RMSNorms of + ``apply_eagle3_fc`` in one kernel (unless the capture already applied them) before the fc; the input ids are + the accepted tokens as they lie. None to use the in-tree path.""" + if ( + self._m3_first_gather_ids is None + or self.is_mtp_eagle + or num_contexts != 0 + or (draft_model is None) + ): + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + ): + return None + cache = self._m3_fc_norm_w + if cache is None or cache[0] is not draft_model.w["fc_norm0"]: + if torch.cuda.is_current_stream_capturing(): + capture_missed("spec_tail", None, "the stacked Eagle3 fc_norm weights") + return None + cache = ( + draft_model.w["fc_norm0"], + torch.stack( + [draft_model.w[f"fc_norm{i}"].detach() for i in range(layers)] + ).contiguous(), + ) + self._m3_fc_norm_w = cache + hidden = self._m3_fc_hidden + self._m3_fc_hidden = None + if hidden is None or hidden.shape[0] != num_tokens: + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + hs = m3_spec_tail.chunk_norm(hs, cache[1], draft_model.rms_eps) + hidden = draft_model.fc(hs) + input_ids = accepted_tokens[:, : spec_metadata.runtime_draft_len + 1].reshape(-1) + return {"input_ids": input_ids, "position_ids": position_ids, "hidden_states": hidden} + + def _m3_fc_fork( + self, spec_metadata, num_tokens: int, draft_model: Optional[M3Eagle3Draft] + ) -> Optional[torch.cuda.Stream]: + """Launch the first draft pass's fc_norm + fc on a forked stream beside the target head GEMM (after everything + queued on the current stream); the result is picked up by _m3_first_drafter_inputs. Returns the stream to + join, or None when the fc path does not apply (the first drafter inputs then compute it).""" + self._m3_fc_hidden = None + if draft_model is None: + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + or self._m3_fc_norm_w is None + ): + return None + aux = self._m3_aux_stream + if aux is None: + aux = self._m3_aux_stream = torch.cuda.Stream() + aux.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(aux): + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + hs = m3_spec_tail.chunk_norm( + hs, self._m3_fc_norm_w[1], draft_model.rms_eps, use_pdl=False + ) + self._m3_fc_hidden = draft_model.fc(hs) + return aux + + def _m3_fc_join(self, fork: torch.cuda.Stream) -> None: + """Join the fc fork onto the current stream and keep its output's memory alive for that stream. + + _m3_fc_hidden is allocated in fork (aux) stream order and read on the current stream. Without the record, + dropping it returns its block to the aux stream's pool, where later aux work can take it while the current + stream's reads are still queued (eager warm-up / mixed steps). Nothing is recorded under graph capture, as in + maybe_execute_in_parallel: a captured graph relies on the next fork's aux.wait_stream(main) to order the + reuse.""" + main = torch.cuda.current_stream() + main.wait_stream(fork) + hidden = self._m3_fc_hidden + if ( + hidden is not None + and hidden.is_cuda + and (not (torch.compiler.is_compiling() or torch.cuda.is_current_stream_capturing())) + ): + hidden.record_stream(main) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model: Optional[M3Eagle3Draft] + ) -> None: + """Before every target forward: whether this step's Eagle3 captures apply the drafter's fc_norm themselves. + + On a fused-verify step whose fc_norm weights are already stacked (by an earlier fused first pass), the + capture writes rmsnorm(h + r) * fc_norm[i] (one kernel per captured layer instead of add + copy) and the fused + first drafter inputs read it without their own chunk_norm. A fused-verify step of at most + ``FC_EARLY_MAX_ROWS`` verify rows is also armed to fork the drafter's first fc at the last captured layer + (``capture_hidden_states``). + """ + self._m3_capture_norm = None + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = None + if draft_model is None: + return + if not self.tail_eligible(attn_metadata, spec_metadata, lm_head): + return + if ( + attn_metadata.num_seqs * (spec_metadata.runtime_draft_len + 1) <= FC_EARLY_MAX_ROWS + and FINAL_HIDDEN_LAYER_ID not in spec_metadata.layers_to_capture + ): + self._m3_fc_early_drafter = draft_model + fc_norm_w = self._m3_fc_norm_w + if ( + fc_norm_w is not None + and fc_norm_w[0] is draft_model.w["fc_norm0"] + and (len(spec_metadata.layers_to_capture) == fc_norm_w[1].shape[0]) + ): + self._m3_capture_norm = (fc_norm_w[1], draft_model.rms_eps) + + @staticmethod + def _m3_sampled_verify_ok(spec_metadata) -> bool: + """Whether sampled verify rows can use the fused sampled op: only under advanced_sampling_mode=no_topk, the + deploy-time guarantee that no row uses top-k or min-p (that op applies neither).""" + mode = getattr(spec_metadata, "advanced_sampling_mode", None) + return mode is not None and mode.skips_top_k and (not mode.skips_top_p) + + def _m3_target_accept( + self, + hidden_states, + spec_metadata, + batch_size: int, + draft_len: int, + draft_model: Optional[M3Eagle3Draft] = None, + ): + """Fused verification (see tail_eligible); same outputs as the strict acceptance path.""" + force_total = 0 + force_vec = None + if self.force_num_accepted_tokens != 0.0: + is_warmup = spec_metadata.is_cuda_graph and ( + not torch.cuda.is_current_stream_capturing() + ) + if not torch.cuda.is_current_stream_capturing(): + self._ensure_force_accept_rng_state(hidden_states.device) + if not self._m3_force_vec_warm: + scratch = torch.zeros(1, dtype=torch.int64, device=hidden_states.device) + dummy = torch.empty(64, dtype=torch.int32, device=hidden_states.device) + for block in (16, 32, 64): + m3_force_vec( + self._force_accept_rng_pool, + scratch, + dummy, + block, + 1, + 2, + 0.5, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + self._m3_force_vec_warm = True + if not is_warmup: + force_total = min(int(self.force_num_accepted_tokens) + 1, draft_len + 1) + force_vec = self._m3_force_vec( + batch_size, force_total, draft_len + 1, hidden_states.device + ) + draft_tokens = spec_metadata.draft_tokens.view(batch_size, draft_len) + if draft_tokens.dtype != torch.int32: + draft_tokens = draft_tokens.to(torch.int32) + self._m3_next_new = torch.empty( + (batch_size, draft_len + 1), dtype=torch.int32, device=hidden_states.device + ) + rows = hidden_states[: batch_size * (draft_len + 1)] + lm_head = self._m3_target_lm_head + batch_idx = spec_metadata.batch_indices_cuda[:batch_size] + fork = self._m3_fc_early_stream + self._m3_fc_early_stream = None + if fork is None: + fork = self._m3_fc_fork(spec_metadata, rows.shape[0], draft_model) + logits = lm_head.apply_linear(rows, None) + vocab_offset = self.mapping.tp_rank * lm_head.weight.shape[0] + if spec_metadata.is_all_greedy_sample: + result = m3_shard_accept_kernel.shard_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + force_total, + self._m3_next_new, + force_vec=force_vec, + ) + path = SHARD_ACCEPT.path + else: + n_rows = rows.shape[0] + seeds, offsets = self._rng_state_per_token(spec_metadata, n_rows) + result = m3_sampled_accept_kernel.sampled_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + spec_metadata.temperatures[:n_rows], + spec_metadata.top_ks[:n_rows], + spec_metadata.top_ps[:n_rows], + None, + seeds, + offsets, + force_total, + self._m3_next_new, + force_vec=force_vec, + ) + path = "spec_tail" + if fork is not None: + self._m3_fc_join(fork) + if result is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, None, "the fused verify's exchange workspace") + raise RuntimeError( + "M3 fused verify unavailable after tail_eligible() accepted the step" + ) + return result + + def _m3_force_vec( + self, batch_size: int, base_total: int, max_total: int, device + ) -> Optional[torch.Tensor]: + """Per-request forced accepted counts for a fractional forced acceptance, else None. + + Draws exactly as _apply_force_accepted_tokens does for a decode-only batch (same fixed-seed pool, one counter + bump per step, the same (counter, slot) hash), so every TP rank and the generic path agree on the counts. + """ + frac = self.force_num_accepted_tokens - int(self.force_num_accepted_tokens) + if frac <= 0.0 or base_total >= max_total: + return None + self._ensure_force_accept_rng_state(device) + out = torch.empty(batch_size, dtype=torch.int32, device=device) + return m3_force_vec( + self._force_accept_rng_pool, + self._force_accept_rng_counter, + out, + batch_size, + base_total, + max_total, + float(frac), + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + max(16, 1 << (batch_size - 1).bit_length()), + ) + + def prebuild_draft_head(self, draft_model: Optional[M3Eagle3Draft]) -> bool: + """Build the streamed copy of the drafter's BF16 head shard that the fused draft loop reads (rebuilt from the + current weight values on every call), on every rank. False when the drafter is not one the loop serves.""" + self._m3_draft_head = None + if draft_model is None or not self._m3_drafter_supported(draft_model): + return False + self._m3_draft_head = m3_draft_step_kernel.prepare_head( + draft_model.lm_head, self.mapping.tp_rank + ) + return True + + def prebuild_drafter_linears(self, draft_model: Optional[M3Eagle3Draft]) -> int: + """Compile every drafter linear call ``DRAFT_LOWM`` covers, and cap the drafter attention's trtllm-gen KV + split per request count (``DRAFT_DECODE_MAX_SEQ_LEN_BY_BATCH``). Called before the executor's warmup, on every + rank. Returns the number of low-M calls compiled.""" + if draft_model is None: + return 0 + draft_model.attn.dense_decode_max_seq_len_by_batch = DRAFT_DECODE_MAX_SEQ_LEN_BY_BATCH + return _prebuild_draft_lowm(draft_model) + + def _m3_drafter_supported(self, draft_model: M3Eagle3Draft) -> bool: + """The drafter the fused draft loop runs (``M3Eagle3Draft`` validates its shape at construction) over this + worker's tensor-parallel group with a bias-free vocab-sharded BF16 head without padding.""" + if self.mapping is None or self.mapping.tp_size <= 1 or self.mapping.enable_attention_dp: + return False + if self._d2t is not None: + return False + lm_head = draft_model.lm_head + return _sharded_head(lm_head) and m3_draft_step_kernel.head_supported(lm_head) + + def _m3_fused_loop_ok( + self, attn_metadata, spec_metadata, batch_size: int, draft_model: Optional[M3Eagle3Draft] + ) -> bool: + """Whether this step drafts with the fused loop: a decode-only step the fused verify or the seed prepared + (``_m3_first_gather_ids``) with the fused first draft inputs, a supported drafter, a (draft length, requests) + ``DRAFT_STEP`` covers, the prebuilt head and the exchange workspace. Host state only, so the TP ranks + agree.""" + if ( + self._m3_first_gather_ids is None + or not self._m3_fused_first_inputs + or draft_model is None + ): + return False + path = DRAFT_STEP.path + if not self._m3_drafter_supported(draft_model) or not hasattr( + attn_metadata, "kv_lens_cuda" + ): + uncovered( + path, "the drafter or its attention metadata is not one the fused loop serves" + ) + return False + draft_len = spec_metadata.runtime_draft_len + if not DRAFT_STEP.covers((draft_len, batch_size)): + uncovered(path, f"draft length {draft_len} at {batch_size} requests") + return False + head = self._m3_draft_head + if head is None or head.weight is not draft_model.lm_head.weight: + missed( + path, + batch_size, + "the drafter head's streamed copy is not built for its current weight (prebuild)", + ) + return False + if m3_draft_step_kernel.workspace(self.mapping, head.wf.device) is None: + capture_missed(path, None, "the draft-step exchange workspace") + return False + return True + + def _forward_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ): + """The fused draft loop on a step it covers (``_m3_fused_loop_ok``), else the in-tree loop.""" + if self._m3_fused_loop_ok(attn_metadata, spec_metadata, batch_size, draft_model): + return self._m3_fused_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + batch_size, + num_accepted_tokens, + ) + return super()._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + + def _m3_fused_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model: M3Eagle3Draft, + draft_kv_cache_manager, + batch_size: int, + num_accepted_tokens, + ): + """Mirrors Eagle3OneModelWorker._forward_linear_draft_loop for a decode-only Eagle3 step, with each pass's + glue fused (see ``_m3_fused_loop_ok``): + + * the draft input -- the embedding lookup, the layer's input norm and hidden norm and their concat -- in one + launch (``m3_spec_tail.draft_input``; the second pass gathers the last accepted rows there); + * the drafter layer's attention and MLP, each followed by the all-reduce with the residual add and the next + RMSNorm (the post-attention norm, then the drafter's final norm); + * one kernel for the BF16 head over the head rows (pass 0: the last accepted row of each request), the + cross-rank argmax and the bookkeeping the in-tree loop does between passes: the draft token, the next pass's + input id and position, the draft KV lengths (and pass 0's sequence lengths) (``m3_draft_step``). + + The host metadata transitions are the in-tree loop's. The drafts land in columns 1.. of the step's next-input + tokens (column 0 is the last accepted token), which ``_forward_impl`` returns as they lie. + """ + draft_len = spec_metadata.runtime_draft_len + table = draft_model.embed_table + dev = table.device + head = self._m3_draft_head + gather_ids = self._m3_first_gather_ids + ids = inputs["input_ids"] + hidden = inputs["hidden_states"] + pos = inputs["position_ids"] + if ids.dtype != torch.int32: + ids = ids.to(torch.int32) + if pos.dtype != torch.int32: + pos = pos.to(torch.int32) + num_acc = num_accepted_tokens[:batch_size] + if num_acc.dtype != torch.int32: + num_acc = num_acc.to(torch.int32) + next_new = self._m3_next_new + if next_new is None: + next_new = torch.empty((batch_size, draft_len + 1), dtype=torch.int32, device=dev) + next_new[:, 0] = ids[gather_ids] + draft_tokens = next_new[:, 1:] + w = draft_model.w + eps = draft_model.rms_eps + with self.draft_kv_cache_context(attn_metadata, draft_kv_cache_manager) as md: + for i in range(draft_len): + x, residual = m3_spec_tail.draft_input( + ids, + table, + hidden, + gather_ids if i == 1 else None, + w["in_norm"], + eps, + w["hidden_norm"], + eps, + ) + h = draft_model.attention(x, md, reduce=False) + h, residual = draft_model.ar_norm(h, residual, "post_norm") + h = draft_model.mlp(h, reduce=False) + h, to_save = draft_model.ar_norm(h, residual, "final_norm") + if draft_model.return_hidden_post_norm: + to_save = h + ids_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + pos_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + if not m3_draft_step_kernel.draft_step( + head, + h, + batch_size, + gather_ids if i == 0 else None, + self.mapping, + draft_tokens, + i, + ids_next, + pos, + pos_next, + md.kv_lens_cuda, + num_acc if i == 0 else None, + md._seq_lens_cuda if i == 0 else None, + draft_len, + ): + raise RuntimeError( + "M3 fused draft step unavailable after _m3_fused_loop_ok() accepted the step" + ) + ids, pos, hidden = (ids_next, pos_next, to_save) + if i == 0: + md._seq_lens[:batch_size].fill_(1) + md.on_update() + has_kv_cache = md.kv_cache_manager is not None + if has_kv_cache and hasattr(md, "host_request_types"): + md.host_request_types[: md.num_contexts].fill_(1) + if has_kv_cache: + md.num_contexts = 0 + self._prepare_flash_mla_generation_layout(md, 0, batch_size) + md.update_for_spec_dec() + md.use_spec_decoding = False + else: + md.update_for_spec_dec() + self._m3_next_new_tokens = next_new + return draft_tokens + + +def _build_speculation(shell, model_config: ModelConfig): + """The drafter config, the drafter and the worker of one-model Eagle3 (the in-tree one-engine construction for + the llama3 drafter architecture), or of one-model standalone DSpark (``dspark.build_dspark``: the in-tree drafter + and worker).""" + spec = model_config.spec_config + if _dspark.is_standalone_dspark(spec): + shell.model.embed_tokens = _dspark.M3EmbedTokens(shell.model) + return _dspark.build_dspark(shell, model_config) + if ( + not spec.spec_dec_mode.use_one_engine() + or not spec.spec_dec_mode.is_eagle3_one_model() + or spec.eagle3_model_arch != "llama3" + or getattr(spec, "use_dynamic_tree", False) + ): + raise ValueError( + "this MiniMax-M3 target speculates with linear one-model Eagle3 (llama3 drafter) only" + ) + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, + trust_remote_code=True, + attn_backend=model_config.attn_backend, + moe_backend=_get_requested_draft_moe_backend(model_config, spec), + mapping=model_config.mapping, + spec_config=model_config.spec_config, + max_num_tokens=model_config.max_num_tokens, + moe_max_num_tokens=model_config.moe_max_num_tokens, + ) + _set_draft_kv_cache_quant_algo(draft_config, model_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = M3Eagle3Draft( + shell.model, + draft_config, + model_config.pretrained_config.num_hidden_layers, + spec.num_capture_layers, + ) + worker = M3Eagle3Worker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + return (draft_config, draft, worker, use_separate) + + +def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: + """The step-loop hooks this target opts into, measured on GB300 AgentX decode (tp4 and tp8). + + * ``pack_step_inputs``: a decode step's small input copies go out as one upload and one scatter at the graph + replay instead of a chain of copies the next graph waits on (~30 us of idle GPU at one request). + * ``skip_unchanged_h2d``: steady decode re-uploads slot indices, query lengths and page tables that rarely change. + * ``loop_on_execution_stream``: the overlap loop's own stream work and the forward on one stream, so step N+1's + inputs wait on graph N in program order rather than through two cross-stream event hops. + * ``merge_tuned_tactics_across_ranks``: tensor-parallel peers capture the same GEMM tactics (rank-local profiling + noise otherwise splits them, and the slowest rank sets every step). + * ``skip_greedy_rng_upload``: all-greedy speculative batches never read the Philox seeds. + * ``spec_step_fold``: the spec sampler's slot stores and the engine's next-step input gathers in one launch each; + under DSpark with ``BOUNDARY_PACK``, both inside a batch-1 decode graph's tail. + * MSA: identity selections for short sequences (and short/long graph families), the one-launch KV-length update, + unchanged block-table copies skipped, pool views built once. + * HOSTTAIL (kill switches above ``HOSTTAIL_*``): ``sampler_d2h_side_stream`` (the sampler's store copies on its + D2H side stream), ``spec_small_h2d`` (static slot ids, packed accepted counts), ``spec_batch_map_changed_only`` + (the drafter's batch_indices / batch-to-slot uploads only on change), ``skip_accepted_draft_upload`` (no reader + of the accepted counts here), ``pack_spec_rng_uploads`` (Philox uploads in the pack), ``kv_lens_update_once`` + (one MSA KV-length re-derivation per speculative overlap step). + * ``pack_parallel_scatter`` (``PACK_PSCATTER``): the pack's non-overlapping copies scatter in parallel. + * ``pack_in_graph`` (``PACK_IN_GRAPH``) / ``pack_early_copy`` (``PACK_EARLY_COPY``): the decode graph's prologue + issues the pack, from a window uploaded on a side copy stream while the previous graph runs (no H2D copy or + scatter between the graphs). ``h2d_changed_span`` (``H2D_CHANGED_SPAN``): page-crossing steps stage only the + changed page-table entries, so their pack still fits the prologue's window. + * ``spec_tail_bookkeep`` (``TAILBK``): the sampler tail's no-op bookkeeping launches dropped, bitwise. + """ + return RuntimeHooks( + pack_step_inputs=True, + pack_parallel_scatter=PACK_PSCATTER, + pack_in_graph=PACK_IN_GRAPH, + pack_early_copy=PACK_EARLY_COPY, + skip_unchanged_h2d=True, + h2d_changed_span=H2D_CHANGED_SPAN, + loop_on_execution_stream=True, + merge_tuned_tactics_across_ranks=True, + skip_greedy_rng_upload=True, + spec_step_fold=spec_fold, + kv_lens_update_once=HOSTTAIL_KV_LENS_ONCE, + pack_spec_rng_uploads=HOSTTAIL_PHILOX_PACK, + spec_small_h2d=HOSTTAIL_SMALL_H2D, + skip_accepted_draft_upload=HOSTTAIL_SKIP_ACCEPTED_UPLOAD, + spec_batch_map_changed_only=HOSTTAIL_BATCH_CHANGED_ONLY, + spec_slot_ids_changed_only=SLOT_IDS_CHANGED, + sampler_d2h_side_stream=HOSTTAIL_D2H_SIDE, + skip_unchanged_draft_block_offsets=BLOCKOFF_GUARD, + spec_tail_bookkeep=TAILBK, + msa=MsaRuntimePolicy( + short_sequence_identity=True, + kv_update=_msa_kv_update_kernel.msa_kv_update, + skip_unchanged_block_offsets=True, + reuse_pool_views=True, + ), + ) + + +class ModelingV2MiniMaxM31Nvfp4Sm103Tp8( + DecoderModelForCausalLM[MiniMaxM3Tp8ModelingV2, PretrainedConfig] +): + """MiniMax-M3 NVFP4 / sm_103 / tp8, text only, with one-model Eagle3 or no speculation. + + The shell owns the vocab-parallel BF16 ``lm_head`` and the logits processor (``DecoderModelForCausalLM``); with + speculation, the drafter (``M3Eagle3Draft``, its weights from the speculative checkpoint through + ``load_draft_weights``) and the worker (``M3Eagle3Worker``) that verifies and drafts each step. + """ + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + assert torch.cuda.get_device_capability() == _SM, ( + "this target is built for sm_" + f"{_SM[0]}" + f"{_SM[1]}" + ", the device is sm_" + f"{''.join(map(str, torch.cuda.get_device_capability()))}" + ) + set_require(resolved_mode(type(self)) is ModelingV2Mode.REQUIRE) + model_config = _text_model_config(model_config) + cfg = model_config.pretrained_config + spec_config = getattr(model_config, "spec_config", None) + if ( + model_config.quant_config.kv_cache_quant_algo == QuantAlgo.NVFP4 + and _dspark.draft_kv_override_applies(spec_config) + ): + model_config.extra_attrs["draft_kv_cache_quant_algo_override"] = QuantAlgo.FP8 + super().__init__( + MiniMaxM3Tp8ModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.spec_config = spec_config + self.draft_model: Optional[nn.Module] = None + self.draft_config: Optional[ModelConfig] = None + self.spec_worker: Optional[Union[M3Eagle3Worker, _dspark.M3DSparkWorker]] = None + self.use_separate_draft_kv_cache = False + self.layer_idx = FINAL_HIDDEN_LAYER_ID + spec_fold = None + if spec_config is not None: + ( + self.draft_config, + self.draft_model, + self.spec_worker, + self.use_separate_draft_kv_cache, + ) = _build_speculation(self, model_config) + if not _dspark.is_standalone_dspark(spec_config): + spec_fold = M3SpecStepFold() + elif DSFOLD: + spec_fold = M3DSparkStepFold( + boundary_pack=BOUNDARY_PACK, + tokens_per_step=_static_tokens_per_step(spec_config) if BOUNDARY_PACK else None, + ) + self.boundary_pack_fold = ( + spec_fold + if isinstance(spec_fold, M3DSparkStepFold) and spec_fold.boundary_pack + else None + ) + self.runtime_hooks = _runtime_hooks(spec_fold) + _log_tailbk(self.spec_worker) + _install_topp_det() + _install_softmax_mc() + _install_rej_mc() + + def load_weights(self, weights: Dict, *args, **kwargs) -> None: + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_draft_weights(self, weights: Dict, weight_mapper=None) -> None: + """The drafter's checkpoint (``speculative_model``), loaded by the engine after the target's.""" + if _dspark.is_standalone_dspark(self.spec_config): + _dspark.load_dspark_draft_weights(self, weights, weight_mapper) + return + _weights.DRAFT_WEIGHTS.load(self.draft_model, weights) + + def set_guided_decoder(self, guided_decoder) -> bool: + """Mirrors ``SpecDecOneEngineForCausalLM.set_guided_decoder``: guided decoding with speculation is installed + on the worker.""" + if hasattr(self.spec_worker, "set_guided_decoder"): + return self.spec_worker.set_guided_decoder(guided_decoder) + return False + + def cache_derived_state(self) -> None: + """Build the targets and every decode kernel once the weights are real, before the executor's warmup; with + speculation also the fused draft loop's head copy and the drafter linears' low-M kernels.""" + super().cache_derived_state() + self.model.post_load_weights() + _prebuild_topp_det() + _prebuild_softmax_mc() + _prebuild_rej_mc() + if self.spec_worker is not None: + self.spec_worker.prebuild(self.lm_head, self.draft_model) + + def post_load_weights(self) -> None: + super().post_load_weights() + self.model.post_load_weights() + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.Tensor] = None, + position_ids: Optional[torch.Tensor] = None, + inputs_embeds: Optional[torch.Tensor] = None, + return_context_logits: bool = False, + spec_metadata: Optional[SpecMetadata] = None, + resource_manager: Optional[Any] = None, + **kwargs, + ) -> Any: + """One engine step: the target forward, then the logits (or, with speculation, the worker's verify and + draft). Text only.""" + multimodal_params = kwargs.pop("multimodal_params", None) or [] + if any((p is not None and getattr(p, "multimodal_data", None) for p in multimodal_params)): + raise ValueError( + "this MiniMax-M3 target is text only; the request carries multimodal input" + ) + worker = self.spec_worker + capture = _plain_capture + if worker is not None: + capture = worker.capture_hidden_states + if spec_metadata is not None: + worker.arm_capture_norm( + attn_metadata, spec_metadata, self.lm_head, self.draft_model + ) + hidden_states = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + capture=capture, + **kwargs, + ) + if getattr(attn_metadata, "csa2_remote_tail_mode", None) == "source": + return self.logits_processor.forward(hidden_states, self.lm_head, attn_metadata, False) + if spec_metadata is not None and spec_metadata.is_layer_capture(FINAL_HIDDEN_LAYER_ID): + capture(spec_metadata, FINAL_HIDDEN_LAYER_ID, hidden_states, None) + if attn_metadata.padded_num_tokens is not None: + hidden_states = hidden_states[: attn_metadata.num_tokens] + if worker is None: + return self.logits_processor.forward( + hidden_states, self.lm_head, attn_metadata, return_context_logits + ) + if worker.tail_eligible(attn_metadata, spec_metadata, self.lm_head): + logits = None + else: + target_logits = getattr(worker, "target_logits", None) + logits = ( + target_logits(hidden_states, attn_metadata, spec_metadata, self.lm_head) + if target_logits is not None + else None + ) + if logits is None: + logits = self.logits_processor.forward( + hidden_states[spec_metadata.gather_ids], self.lm_head, attn_metadata, True + ) + spec_input_ids = input_ids if input_ids is not None else kwargs.get("orig_input_ids") + spec_position_ids = position_ids + if attn_metadata.padded_num_tokens is not None: + if spec_input_ids is not None: + spec_input_ids = spec_input_ids[: attn_metadata.num_tokens] + if position_ids is not None: + spec_position_ids = _slice_spec_position_ids(position_ids, attn_metadata.num_tokens) + outputs = worker( + input_ids=spec_input_ids, + position_ids=spec_position_ids, + hidden_states=hidden_states, + logits=logits, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=resource_manager, + ) + if self.boundary_pack_fold is not None and spec_metadata is not None: + outputs = self.boundary_pack_fold.graph_tail(outputs, attn_metadata) + return outputs diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/mxdraft.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/mxdraft.py new file mode 100644 index 000000000000..da8a096aaa15 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/mxdraft.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MXDRAFT: the DSpark drafter's decode projections on this target's MXFP8 GEMV. + +The drafter checkpoint ships its linears as MXFP8 (E4M3 codes, one UE8M0 scale per 32 columns) and the stock drafter +dequantizes them to BF16 at load, so every decode pass streams twice the bytes it needs. MXDRAFT recovers the MXFP8 +form of each BF16 weight (exactly: the recovery is checked to dequantize back to the same BF16 bits, else the +projection is left alone) and routes the projection's GEMM to ``m3_mx_gemv_bf16`` for decode inputs it serves; any +other call runs the Linear unchanged. The activation is quantized to MXFP8 inside the GEMV, so the output is not the +stock BF16 GEMM's: drafter-only precision, gated on natural AL. Row-parallel projections keep +their all-reduce (only the Linear's GEMM is replaced). +""" + +from __future__ import annotations + +from typing import Iterable, Optional, Tuple + +import torch + +from .ops.comm.m3_mc_allreduce_norm import swizzle_sf +from .ops.mx_gemv import op as _gemv + +E4M3_MAX = 448.0 +BLOCK = 32 + + +def mxfp8_from_bf16(weight: torch.Tensor) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """The MXFP8 form (E4M3 codes [N, K], UE8M0 scale bytes [N, K / 32]) of a BF16 weight that is exactly an MXFP8 + dequantization, or None. Each block takes the smallest power-of-two scale that fits its amax in E4M3, which is at + most the shipped scale, so the codes are exact whenever the weight came from MXFP8.""" + if weight.dtype != torch.bfloat16 or weight.dim() != 2 or weight.shape[1] % BLOCK != 0: + return None + n, k = weight.shape + blocks = weight.float().view(n, k // BLOCK, BLOCK) + amax = blocks.abs().amax(-1) + exp = torch.ceil(torch.log2(amax.clamp_min(2.0**-126) / E4M3_MAX)).clamp(-127, 127) + scale = torch.exp2(exp).unsqueeze(-1) + codes = (blocks / scale).to(torch.float8_e4m3fn) + back = (codes.float() * scale).view(n, k).to(torch.bfloat16) + if not torch.equal(back, weight): + return None + return codes.view(n, k).contiguous(), (exp + 127).to(torch.uint8).contiguous() + + +def _projections(draft_model) -> Iterable[torch.nn.Module]: + for layer in getattr(getattr(draft_model, "model", None), "layers", []) or []: + attn, mlp = getattr(layer, "self_attn", None), getattr(layer, "mlp", None) + for lin in ( + getattr(attn, "qkv_proj", None), + getattr(attn, "o_proj", None), + getattr(mlp, "gate_up_proj", None), + getattr(mlp, "down_proj", None), + ): + if lin is not None: + yield lin + + +def route_mx_linears(draft_model, rows: Iterable[int]) -> int: + """Route each drafter projection whose BF16 weight is an exact MXFP8 dequantization to the MXFP8 GEMV for 2-D + contiguous bf16 inputs of the given row counts (the decode passes'); compiles those kernels now (outside capture). + Returns how many projections were routed.""" + rows = sorted({int(m) for m in rows if 0 < int(m) <= _gemv.MAX_TOKENS}) + n_routed = 0 + for lin in _projections(draft_model): + weight = getattr(lin, "weight", None) + if ( + weight is None + or getattr(lin, "bias", None) is not None + or getattr(lin, "_m31_mxdraft", None) is not None + ): + continue + rec = mxfp8_from_bf16(weight.data) + if rec is None: + continue + n, k = weight.shape + served = tuple(m for m in rows if _gemv.serves(n, k, m, True)) + if not served: + continue + w8, sf = rec[0], swizzle_sf(rec[1]) + for m in served: + _gemv.precompile(w8, sf, m, True) + original = lin.apply_linear + + def apply_linear( + input, + bias, + lora_params=None, + layer_idx=None, + *, + _o=original, + _w8=w8, + _sf=sf, + _served=served, + ): + if ( + bias is None + and not lora_params + and isinstance(input, torch.Tensor) + and input.dtype == torch.bfloat16 + and input.dim() == 2 + and input.shape[0] in _served + and input.is_contiguous() + ): + out = _gemv.m3_mx_gemv_bf16(input, _w8, _sf) + if out is not None: + return out + return _o(input, bias, lora_params, layer_idx) + + lin._m31_mxdraft = (w8, sf, served) + lin.apply_linear = apply_linear + n_routed += 1 + return n_routed diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/__init__.py new file mode 100644 index 000000000000..1059445198db --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Ops used only by this target. Shared ops stay in the catalog.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/__init__.py new file mode 100644 index 000000000000..b8c7426e3933 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Activation ops of this target: the gated MLPs' fused SwiGLU.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/m3_silu_and_mul.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/m3_silu_and_mul.py new file mode 100644 index 000000000000..7a494d97d788 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/m3_silu_and_mul.py @@ -0,0 +1,83 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The gated MLPs' fused SwiGLU: ``silu_and_mul`` with MiniMax-M3's SwiGLU-OAI gain, offset and clamp.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SiluAndMul(OpWrapper): + """``out = g * sigmoid(alpha * g) * (u + beta)`` over ``x = [g | u]`` (bf16 [M, 2I] -> [M, I]), with + ``g = min(gate, limit)`` and ``u = clamp(up, -limit, limit)``; ``limit``, ``alpha`` and ``beta`` None give plain + SwiGLU ``silu(gate) * up``. + + The MXFP8 dense MLPs (layers 0-2, intermediate 3072 per rank) and the MoE's shared expert (intermediate 768 per + rank) run it at MiniMax-M3's (limit 7, alpha 1.702, beta 1); the BF16 Eagle3 drafter's MLP (intermediate 4608 per + rank) runs it plain. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert, one decode token", spec=dict(m=1, inter=384, oai=True)), + Cell(why="shared expert, a 48-row decode verify", spec=dict(m=48, inter=384, oai=True)), + Cell(why="dense MLP, a prefill chunk", spec=dict(m=2048, inter=1536, oai=True)), + Cell(why="dense MLP, one decode token", spec=dict(m=1, inter=1536, oai=True)), + Cell( + why="drafter MLP (plain SwiGLU), eight decode rows", + spec=dict(m=8, inter=2304, oai=False), + ), + Cell( + why="drafter MLP (plain SwiGLU), a prefill chunk", + spec=dict(m=1024, inter=2304, oai=False), + ), + ) + + note = """ + The kernel computes in fp32 and rounds once to bf16; `reference` does the same arithmetic in fp32, so the two + differ only by the fast sigmoid's last bits (the test's band). + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + + def reference( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + gate, up = x.float().chunk(2, dim=-1) + if limit is not None: + gate = gate.clamp(max=limit) + up = up.clamp(min=-limit, max=limit) + a = 1.0 if alpha is None else alpha + b = 0.0 if beta is None else beta + return (gate * torch.sigmoid(a * gate) * (up + b)).to(x.dtype) + + def is_valid( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> None: + assert x.stride(-1) == 1, "the gate/up halves are read from contiguous rows" + + +m3_silu_and_mul = M3SiluAndMul() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/m3_swq.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/m3_swq.py new file mode 100644 index 000000000000..5c8b970549f1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/activation/m3_swq.py @@ -0,0 +1,162 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SWQ: the shared expert's SwiGLU-OAI written directly as the down projection's MXFP8 activation.""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +@triton.jit +def _rcp_approx_ftz(x): + return tl.inline_asm_elementwise( + "rcp.approx.ftz.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _swq_kernel( + x_ptr, + x_stride, + q_ptr, + sf_ptr, + M, + INTER: tl.constexpr, + NB: tl.constexpr, + NB_PAD: tl.constexpr, + P2: tl.constexpr, + swiglu_limit: tl.constexpr, + swiglu_alpha: tl.constexpr, + swiglu_beta: tl.constexpr, + HAS_LIMIT: tl.constexpr, +): + # One program per row of the 128-row-padded scale layout; rows past M only zero their scale bytes. + r = tl.program_id(0) + blk = tl.arange(0, P2) + blk_ok = blk < NB_PAD + # The 128x4 swizzle of trtllm::mxfp8_quantize(x, True) (quantization.cuh get_sf_out_offset_128x4). + sf_off = ( + (blk % 4) + + (blk // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * NB_PAD) + ) + if r >= M: + tl.store(sf_ptr + sf_off, tl.zeros([P2], dtype=tl.uint8), mask=blk_ok) + return + cols = tl.arange(0, P2 * 32) + valid = cols < INTER + row = x_ptr + r.to(tl.int64) * x_stride + # silu_and_mul_kernel's arithmetic (modules/swiglu.py), then its bf16 store. + a = tl.load(row + cols, mask=valid, other=0.0).to(tl.float32) + b = tl.load(row + cols + INTER, mask=valid, other=0.0).to(tl.float32) + if HAS_LIMIT: + a = tl.minimum(a, swiglu_limit) + b = tl.clamp(b, -swiglu_limit, swiglu_limit) + v = (a * tl.sigmoid(swiglu_alpha * a) * (b + swiglu_beta)).to(tl.bfloat16).to(tl.float32) + # cvt_warp_fp16_to_mxfp8: SF = amax * rcp.approx(448), UE8M0 toward +inf, scale = rcp.approx(SF), E4M3 RN. + v2 = tl.reshape(v, (P2, 32)) + amax = tl.max(tl.abs(v2), axis=1) + sf = amax * _rcp_approx_ftz(tl.full([P2], 448.0, tl.float32)) + bits = sf.to(tl.int32, bitcast=True) + exp = (bits >> 23) & 0xFF + mant = bits & 0x7FFFFF + e = tl.where(exp > 0, exp + (mant != 0).to(tl.int32), tl.where(bits > 0x400000, 1, 0)) + e = tl.minimum(e, 254) + # at code 0 the scale 2^-127 is a float subnormal, so rcp.approx.ftz returns inf (native publishes + # inf/NaN there). Use its exact reciprocal 2^127 instead; every other code keeps + # native's rcp.approx (exact on a normal power of two), so each block native emits finite stays bitwise. + pinv = tl.where( + e == 0, 1.7014118346046923e38, _rcp_approx_ftz((e << 23).to(tl.float32, bitcast=True)) + ) + inv = tl.where(amax != 0.0, pinv, 0.0) + q = tl.reshape((v2 * inv[:, None]).to(tl.float8e4nv), (P2 * 32,)) + tl.store(q_ptr + r.to(tl.int64) * INTER + cols, q, mask=valid) + tl.store(sf_ptr + sf_off, tl.where(blk < NB, e, 0).to(tl.uint8), mask=blk_ok) + + +class M3Swq(OpWrapper): + """``silu_and_mul`` (SwiGLU-OAI) over ``x = [g | u]`` (bf16 [M, 2I]) rounded to bf16, then MXFP8-quantized as + ``trtllm::mxfp8_quantize(h, True)``: (E4M3 [M, I], UE8M0 scales in the 128x4-swizzled layout), bit for bit, in one + launch. The TP8 shared expert's down projection (intermediate 384, no GEMV row table) reads it at ``SWQ_ROWS``.""" + + ARCHS = frozenset({Arch.SM_103}) + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert, the smallest row count it serves", spec=dict(m=9, inter=384)), + Cell(why="shared expert, a 32-row decode verify (c8)", spec=dict(m=32, inter=384)), + Cell( + why="shared expert, the widest decode row count it serves", spec=dict(m=48, inter=384) + ), + ) + + note = """ + Bitwise to the two-kernel path (silu_and_mul's fp32 arithmetic and bf16 rounding, then mxfp8_quantize's + rcp.approx scale rule): q bytes and every scale byte, padding included. + One deliberate exception: a block whose scale code is 0 (amax / 448 <= 2^-127), where native's + rcp.approx.ftz of the subnormal scale is inf and its e4m3 bytes encode inf/NaN, is quantized with the exact + reciprocal 2^127 instead, so it stays finite; its scale byte is native's. + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> tuple[torch.Tensor, torch.Tensor]: + m, i = x.shape[0], x.shape[1] // 2 + nb = i // 32 + nb_pad = (nb + 3) // 4 * 4 + q = torch.empty(m, i, dtype=torch.float8_e4m3fn, device=x.device) + sf = torch.empty((m + 127) // 128 * 128 * nb_pad, dtype=torch.uint8, device=x.device) + _swq_kernel[((m + 127) // 128 * 128,)]( + x, + x.stride(0), + q, + sf, + m, + INTER=i, + NB=nb, + NB_PAD=nb_pad, + P2=triton.next_power_of_2(nb_pad), + swiglu_limit=limit or 0.0, + swiglu_alpha=alpha if alpha is not None else 1.0, + swiglu_beta=beta if beta is not None else 0.0, + HAS_LIMIT=limit is not None and limit > 0.0, + ) + return q, sf + + def reference( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> tuple[torch.Tensor, torch.Tensor]: + """The two kernels it replaces (bitwise is the contract).""" + h = torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + q, sf = torch.ops.trtllm.mxfp8_quantize(h, True) + return q.view(h.shape), sf + + def is_valid( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> None: + assert x.dtype == torch.bfloat16 and x.stride(-1) == 1, "bf16 [M, 2I] with contiguous rows" + assert (x.shape[1] // 2) % 32 == 0, "I must be a multiple of the 32-element MXFP8 block" + + +m3_swq = M3Swq() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/__init__.py new file mode 100644 index 000000000000..a7f7fee7a234 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Attention ops of this target: MSA block selection, the sparse decode and the KV-length update.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_dense_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_dense_decode.py new file mode 100644 index 000000000000..525de997aaf9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_dense_decode.py @@ -0,0 +1,187 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' decode attention (trtllm-gen over the full page table) with the KV split this target sizes per +request count (``dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH``).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.trtllm_gen_dense_decode import ( # noqa: E501 + minimax_m3_trtllm_gen_dense_decode, +) + +__all__ = ["m3_dense_decode"] + +#: The TP8 dense layers' geometry: 8 query heads and one KV head of dim 128 per rank, an FP8 KV pool. +HEADS = 8 +KV_HEADS = 1 +HEAD_DIM = 128 +#: The split this target hands trtllm-gen at one request (dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH), and a model +#: maximum the in-tree step passes otherwise. +CAPPED = 16384 +MODEL_MAX = 1 << 20 + + +def dense_attention_reference( + q: torch.Tensor, + keys: torch.Tensor, + values: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, +) -> torch.Tensor: + """fp32 causal decode attention: query row ``b * qlen + i`` of request b attends its first + ``seq_lens[b] - qlen + i + 1`` keys. ``keys`` / ``values``: [batch, kv_heads, max_len, head_dim] (the request's + tokens in order). Returns fp32 [total_q, heads, head_dim].""" + batch, kv_heads, max_len, _ = keys.shape + heads = q.shape[1] + group = heads // kv_heads + out = torch.zeros(q.shape, dtype=torch.float32, device=q.device) + pos = torch.arange(max_len, device=q.device) + with true_fp32_matmul(): + for b in range(batch): + full = int(seq_lens[b]) + rows = q[b * decode_query_len : (b + 1) * decode_query_len].float() # [qlen, heads, d] + limit = full - decode_query_len + 1 + torch.arange(decode_query_len, device=q.device) + valid = pos[None, :] < limit[:, None] # [qlen, max_len] + for h in range(kv_heads): + qh = rows[:, h * group : (h + 1) * group] # [qlen, group, d] + logits = torch.einsum("qgd,kd->qgk", qh, keys[b, h].float()) * sm_scale + logits = logits.masked_fill(~valid[:, None, :], float("-inf")) + probs = torch.softmax(logits, dim=-1) + out[ + b * decode_query_len : (b + 1) * decode_query_len, h * group : (h + 1) * group + ] = torch.einsum("qgk,kd->qgd", probs, values[b, h].float()) + return out + + +def gather_paged_kv( + kv_cache_manager, layer_idx: int, block_table: torch.Tensor, seq_lens: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor]: + """The K and V of each request in token order, [batch, kv_heads, max_len, head_dim], from the flat sub-page pool + (``get_kv_subpage_pool``: K of slot s at sub-page s * subpages_per_slot, V one sub-page later).""" + pool, subpages = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + page = pool.shape[2] + pages = (int(seq_lens.max()) + page - 1) // page + slots = block_table[:, :pages].long() + k = pool[slots * subpages] # [batch, pages, kv_heads, page, d] + v = pool[slots * subpages + 1] + batch, _, kv_heads, _, d = k.shape + + def order(t): + return t.permute(0, 2, 1, 3, 4).reshape(batch, kv_heads, pages * page, d) + + return order(k), order(v) + + +class M3DenseDecode(OpWrapper): + """``minimax_m3_trtllm_gen_dense_decode`` as ``MsaDecodeFmha`` calls it for a dense layer, with the max_seq_len + ``msa_decode.dense_decode_max_seq_len`` resolves: this target's per-request-count cap, else the step's bound. + + Decode attention of every query row over its request's full causal context in the paged FP8 pool, written into + ``output`` (BF16 [T, heads, head_dim]) in place. ``max_seq_len`` only sizes trtllm-gen's multi-CTA KV split: + every CTA still covers its share of each request's actual length, which may exceed it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over a 20000-token context, the split sized for 16384", + spec=dict(kv_lens=(20000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request, one row, over a 65536-token context (an AgentX-length cache) at the capped split", + spec=dict(kv_lens=(65536,), qlen=1, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request verifying 1 + 3 rows over a 3000-token context: the cap above the length", + spec=dict(kv_lens=(3000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="two requests keep the model maximum (no cap at two requests)", + spec=dict(kv_lens=(20000, 5000), qlen=4, max_seq_len=MODEL_MAX, capped=False), + ), + ) + + note = """ + The split only reorders the merge of the CTAs' partial softmax sums, so the capped split's output is the uncapped + one's to a few BF16 ulps (the test holds it there), and both are held to the fp32 reference within the FP8-KV + band of the in-tree dense decode test. + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + minimax_m3_trtllm_gen_dense_decode( + q, + kv_cache_manager, + layer_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + max_seq_len=max_seq_len, + max_num_requests=max_num_requests, + ) + return output + + def reference( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + """fp32 attention over the pool's K / V (q rounded to the pool's dtype first, as the kernel runs it). + + ``output``, ``max_seq_len`` and ``max_num_requests`` do not change the result.""" + keys, values = gather_paged_kv(kv_cache_manager, layer_idx, block_table, seq_lens) + if keys.dtype == torch.float8_e4m3fn: + q = q.to(torch.float8_e4m3fn) + return dense_attention_reference(q, keys, values, seq_lens, decode_query_len, sm_scale) + + def is_valid( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> None: + # The multi-CTA KV counters are sized for max_num_requests: more requests overrun them. + assert seq_lens.shape[0] <= max_num_requests, "more requests than the counters cover" + # Each verify row attends a causal prefix of at least one key. + assert bool((seq_lens >= decode_query_len).all()), "a request shorter than its query rows" + + +m3_dense_decode = M3DenseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_draft_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_draft_attention.py new file mode 100644 index 000000000000..6347348d1681 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_draft_attention.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter layer's attention on the TRTLLM backend: RoPE, KV-cache append and causal attention.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + PredefinedAttentionMask, +) +from tensorrt_llm.functional import PositionEmbeddingType, RotaryScalingType + + +class M3DraftAttention(OpWrapper): + """The drafter layer's attention over its fused QKV rows ``qkv`` [T, (q + 2 kv) heads x 128] (bf16): the + backend applies the drafter's RoPE (fused), appends K / V to the drafter layer's paged cache (FP8, the target's + NVFP4 cache keeps its draft layer in FP8) and runs causal attention over the step's live rows (``metadata`` + ``num_tokens``); returns [num_tokens, q heads x 128] bf16. + + ``backend`` (``TrtllmAttention``) and ``metadata`` are the runtime's objects for the drafter layer and step. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="first draft pass of a 4-request verify (4 x 4 rows)", spec=dict(reqs=4, qlen=4)), + Cell(why="a later draft pass, one row per request", spec=dict(reqs=8, qlen=1)), + Cell(why="a context chunk's draft rows", spec=dict(reqs=1, context=512)), + ) + + note = """ + `reference` is exact (float64) causal softmax attention: Q rotated by the backend's NeoX RoPE at each token's + position, over the drafter layer's K / V rows read back from its FP8 cache (dequantized by the backend's + ``kv_scale_quant_orig``). What remains between the two is the FMHA's own arithmetic (Q's rounding to the + kernel's input precision, its P rounding and accumulation order), which the test bounds. + """ + + def __call__(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + num_tokens = metadata.num_tokens + out = backend.forward( + qkv[:num_tokens, :], + None, + None, + metadata, + forward_args=AttentionForwardArgs(attention_mask=PredefinedAttentionMask.CAUSAL), + ) + return out[0] if isinstance(out, tuple) else out + + def reference(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + """The call's [num_tokens, q heads x 128] output, in ``qkv``'s dtype. + + Query token ``j`` of request ``r`` sits at position ``cached_r + j`` (the metadata's cached lengths) and + attends positions up to it. Its Q is rotated here; every K / V row, the step's own included, is read from + the drafter layer's paged cache at the slots the metadata's KV cache manager holds, so this is evaluated + after the call (whose roped, quantized append it then reads back; the test checks that append itself). + Mirrors one KV head per rank and un-scaled NeoX RoPE, the drafter's configuration. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + assert backend.num_kv_heads == 1, "mirrors the one-KV-head-per-rank layout only" + rope = backend.rope_params + assert backend.position_embedding_type == int(PositionEmbeddingType.rope_gpt_neox) + assert rope.scale_type == RotaryScalingType.none and rope.alpha == 1.0 + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + kv_scale = float(backend.kv_scale_quant_orig.reshape(-1)[0]) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = int(metadata.num_tokens) + assert sum(qlens) == total + device = qkv.device + + rot = int(rope.dim) + inv_freq = float(rope.theta) ** ( + -torch.arange(0, rot, 2, dtype=torch.float64, device=device) / rot + ) + pool = manager.get_buffers(layer, "HND") + q = qkv[:total, : heads * dim].to(torch.float64).reshape(total, heads, dim) + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][ + : (length + page - 1) // page + ], + dtype=torch.int64, + device=device, + ) + keys = pool[pages, 0, 0].to(torch.float64).reshape(-1, dim)[:length] * kv_scale + values = pool[pages, 1, 0].to(torch.float64).reshape(-1, dim)[:length] * kv_scale + qpos = cached[row] + torch.arange(n, device=device) + angles = qpos.to(torch.float64)[:, None] * inv_freq[None, :] + cos, sin = angles.cos()[:, None, :], angles.sin()[:, None, :] + rows = q[first : first + n] + x1, x2 = rows[..., : rot // 2], rows[..., rot // 2 : rot] + rows = torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin, rows[..., rot:]], dim=-1) + allowed = torch.arange(length, device=device)[None, :] <= qpos[:, None] + logits = torch.einsum("thd,kd->thk", rows, keys) * sm_scale + logits = logits.masked_fill(~allowed[:, None, :], float("-inf")) + out[first : first + n] = torch.softmax(logits, dim=-1) @ values + first += n + return out.reshape(total, heads * dim).to(qkv.dtype) + + +m3_draft_attention = M3DraftAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_fp8_indexer_qk_norm_rope.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_fp8_indexer_qk_norm_rope.py new file mode 100644 index 000000000000..bcd928fccf5b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_fp8_indexer_qk_norm_rope.py @@ -0,0 +1,114 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The unfused sparse layers' index branch: per-head Gemma RMSNorm + partial NeoX RoPE of [index-Q | index-K], index-Q +out in FP8 E4M3 and index-K inserted into the FP8 index-K cache, in one kernel.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3Fp8IndexerQkNormRope(OpWrapper): + """``qk`` [T, (heads + 1) * 128] BF16 (the index projection's output): returns index-Q [T, heads, 128] E4M3 and + writes index-K (E4M3) at ``out_cache_loc`` of ``index_k_cache`` [pages, 1, 128, 128]. Gemma RMSNorm of each head, + then RoPE on its leading ``rotary_dim`` channels (NeoX). + + The unfused sparse layers (``fuse_qkv_index_projection=False``) run it on the index branch with an FP8 indexer; + the fused layers' producer writes the index cache itself. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, one index head per rank", spec=dict(m=1, heads=1)), + Cell(why="a 48-row decode verify", spec=dict(m=48, heads=1)), + Cell(why="a prefill chunk", spec=dict(m=2048, heads=1)), + ) + + note = """ + `reference` is the in-place fused norm + RoPE in BF16 followed by the E4M3 cast (the kernel rounds through BF16 + before the cast); its index-K is what the kernel writes to the cache. The two kernels may round a handful of + values differently: the test allows fewer than 0.1% of the bytes to differ, as the in-tree test does. + """ + + def __call__( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> torch.Tensor: + return torch.ops.trtllm.minimax_m3_fp8_indexer_qk_norm_rope( + qk, + index_k_cache, + out_cache_loc, + num_heads_q, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + position_ids, + ) + + def reference( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor]: + """(index-Q [T, heads, head_dim] E4M3, index-K [T, head_dim] E4M3): the catalog's in-place kernel on a copy, + then the E4M3 cast.""" + out = fused_qk_norm_rope( + qk.clone(), + num_heads_q, + 1, + 0, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + True, + position_ids, + 1.0, + 0.0, + 0.0, + 1.0, + True, + True, + False, + 0, + 0, + ) + q, k = out.split([num_heads_q * head_dim, head_dim], dim=-1) + return ( + q.reshape(q.shape[0], num_heads_q, head_dim).to(torch.float8_e4m3fn), + k.to(torch.float8_e4m3fn), + ) + + +m3_fp8_indexer_qk_norm_rope = M3Fp8IndexerQkNormRope() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_fused_score_select.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_fused_score_select.py new file mode 100644 index 000000000000..0457f456024e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_fused_score_select.py @@ -0,0 +1,189 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MSA index scoring and top-16 block selection of a pure decode span of at most 2 requests, in one launch.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_index_score import m3_index_score +from .m3_select_long import m3_select_long + +#: Requests per launch the kernel takes. +MAX_REQUESTS = 2 +#: Query rows per request it takes (decode, or a verify of up to 3 drafts). +MAX_DECODE_QUERY_LEN = 4 +#: (KV head x query row) score rows of one request its stage ring holds. +MAX_ROWS = 8 +#: Bytes of the stage ring the select scratch shares. +RING_BYTES = 12 * 16384 + +# M3's checkpoint: no forced initial blocks, the one block holding the query forced in. +_M3 = dict(num_kv_heads=1, init_blocks=0, local_blocks=1) + + +def ring_fits(rows: int, max_tiles: int) -> bool: + """Whether a request's select scratch of ``rows`` score rows over ``max_tiles`` blocks fits the stage ring.""" + return rows * ((max_tiles + 3) // 4 * 4) * 4 + rows * 1024 <= RING_BYTES + + +def scratch_shape(max_tiles: int) -> tuple[int, int, int]: + """The scratch [requests, rows, L] that serves every request count and geometry the op takes.""" + return (MAX_REQUESTS, MAX_ROWS, (max_tiles + 3) // 4 * 4) + + +class M3FusedScoreSelect(OpWrapper): + """Fill ``out`` [total_q, num_kv_heads, 16] with each query row's top-16 KV blocks, straight from the FP8 index Q + and the paged FP8 index-K cache. + + The table is ``m3_select_long`` (ascending ids, -1 tail, ties to the smaller block, the last + ``local_blocks`` valid blocks and the first ``init_blocks`` forced in) applied to the per-KV-head maxima over its + index heads of the ``m3_index_score`` scores, with each row's valid-block count ``n_valid`` clamped to + ``max_tiles``. Grid (splits, requests): each CTA scores a share of its request's blocks through a 12-stage TMA + ring with ``mma.sync`` (the products and the 128-term sums of the CuTe DSL scorer, so the same fp32 scores); the + last CTA of a request (an arrival counter it returns to zero) selects. The request's block-table row and length + are read and its older index-K pages are issued before the grid wait; only the last CTA releases the PDL + dependents. ``scratch`` holds the per-request scores between the two phases and needs no initialization; + ``counters`` must be zero before the first call and stays zero between calls. Returns ``out``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="Eagle3 verify at one request (the TP8 operating point): 4 rows over 20000 tokens", + spec=dict(batch=1, qlen=4, seq_lens=(20000,), max_tiles=8192, **_M3), + ), + Cell( + why="one request decoding one row over 65536 tokens: 512 blocks across every split", + spec=dict(batch=1, qlen=1, seq_lens=(65536,), max_tiles=8192, **_M3), + ), + Cell( + why="one request of 129 tokens: two blocks, most splits empty", + spec=dict(batch=1, qlen=4, seq_lens=(129,), max_tiles=8192, **_M3), + ), + Cell( + why="one request of 200000 tokens: more blocks than one split holds in its 128 local slots", + spec=dict(batch=1, qlen=4, seq_lens=(200000,), max_tiles=8192, **_M3), + ), + Cell( + why="two requests verifying 4 rows each, of different lengths", + spec=dict(batch=2, qlen=4, seq_lens=(20000, 3000), max_tiles=8192, **_M3), + ), + Cell( + why="two requests, one of them shorter than 16 blocks: its whole row is selected and -1 padded", + spec=dict(batch=2, qlen=4, seq_lens=(1500, 70000), max_tiles=8192, **_M3), + ), + ) + + note = """ + Certified at one index head and one KV head per rank (TP8) with M3's forcing (`init_blocks=0`, + `local_blocks=1`), FP8 index Q and index-K, a 1M-token engine (`max_tiles=8192`). The selection is checked bit + for bit against `reference`; scores are compared through the table only. Rows with more than 1024 blocks tied at + the 16th score are not driven. + """ + + def __call__( + self, + idx_q: torch.Tensor, + idx_k_pages: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + n_valid: torch.Tensor, + out: torch.Tensor, + scratch: torch.Tensor, + counters: torch.Tensor, + decode_query_len: int, + num_kv_heads: int, + init_blocks: int, + local_blocks: int, + max_tiles: int, + use_pdl: bool, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_tp8_fused_score_select( + idx_q, + idx_k_pages, + block_table, + seq_lens, + n_valid, + out, + scratch, + counters, + decode_query_len, + num_kv_heads, + init_blocks, + local_blocks, + max_tiles, + use_pdl, + ) + return out + + def reference( + self, + idx_q: torch.Tensor, + idx_k_pages: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + n_valid: torch.Tensor, + out: torch.Tensor, + scratch: torch.Tensor, + counters: torch.Tensor, + decode_query_len: int, + num_kv_heads: int, + init_blocks: int, + local_blocks: int, + max_tiles: int, + use_pdl: bool, + ) -> torch.Tensor: + """The int32 [total_q, num_kv_heads, 16] table: ``m3_select_long.reference`` over the per-KV-head maxima of + ``m3_index_score.reference``. ``out``, ``scratch``, ``counters`` and ``use_pdl`` only say where and how the op + works.""" + total_q, heads, _ = idx_q.shape + placeholder = torch.empty(heads, total_q, max_tiles, device=idx_q.device) + scores = m3_index_score.reference( + idx_q, idx_k_pages, block_table, seq_lens, placeholder, decode_query_len, 1, False + ) # [heads, total_q, max_tiles] + per_kv = scores.view(num_kv_heads, heads // num_kv_heads, total_q, max_tiles).amax(dim=1) + table = torch.empty(total_q, num_kv_heads, 16, dtype=torch.int32, device=idx_q.device) + return m3_select_long.reference( + per_kv.transpose(1, 2), n_valid, table, init_blocks, local_blocks + ) + + def is_valid( + self, + idx_q: torch.Tensor, + idx_k_pages: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + n_valid: torch.Tensor, + out: torch.Tensor, + scratch: torch.Tensor, + counters: torch.Tensor, + decode_query_len: int, + num_kv_heads: int, + init_blocks: int, + local_blocks: int, + max_tiles: int, + use_pdl: bool, + ) -> None: + """The inputs this op takes and answers wrongly. + + A non-zero counter makes a request's selection run early, on partial scores. The block-table row is read up + to ``ceil(seq_len / 128)`` entries unchecked. Reads ``counters`` and ``seq_lens`` back to the host: a test-only + cost. + """ + assert bool((counters[: seq_lens.shape[0]] == 0).all()), ( + "counters must be zero on entry: the kernel selects once a request's arrivals reach its split count" + ) + nblocks = (seq_lens + 127) // 128 + assert bool((nblocks <= block_table.shape[1]).all()), ( + "a request runs past its block-table row" + ) + assert bool((nblocks <= max_tiles).all()), "a request has more blocks than max_tiles" + assert init_blocks >= 0 and local_blocks >= 0, "forcing counts must be non-negative" + + +m3_fused_score_select = M3FusedScoreSelect() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_index_score.py new file mode 100644 index 000000000000..139b216086eb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_index_score.py @@ -0,0 +1,165 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# PyTorch oracle ported from vLLM (Apache-2.0), _reference_decode_index_score: +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/tests/kernels/attention/test_minimax_m3.py#L188 +"""The MSA index scorer of a sparse decode step: per (index head, query token, KV block) the causal max of Q . K over +the block's 128 index-K positions, with the pre-wait schedule of ``ops/m3_index_score``.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_index_score"] + +PAGE = 128 +HEAD_DIM = 128 +#: CTAs per request (each scores blocks split, split + SPLIT_K, ...; the scores do not depend on it). +SPLIT_K = 512 + + +def compiled_scorer(key: tuple): + """The compiled kernel of ``key`` (dtype, heads, query length, head dim, split_k, early), compiling it on a miss + (the target compiles every key it can take before warmup: ``layers.prebuild_index_scorer``).""" + from ..m3_index_score import IndexScoreRunner + + if key not in IndexScoreRunner.kernel_cache: + IndexScoreRunner._compile(*key) + return IndexScoreRunner.kernel_cache[key] + + +class M3IndexScore(OpWrapper): + """``score[h, t, j]`` (fp32, a transposed view of the [heads, blocks, tokens] selector buffer) = the max over the + positions ``p`` of block ``j`` of request ``b = t // qlen`` with ``p <= seq_lens[b] - qlen + t % qlen`` of + ``idx_q[t, h] . index_k[block_table[b, j], p % 128]``, for ``j < ceil(seq_lens[b] / 128)`` (no other entry is + written). FP8 or BF16 q and index-K, fp32 products and sums. + + ``early`` (the target's one-request kernel) issues the Q and first K tiles right after the grid wait from a + pre-wait length hint and prefetches one older block per CTA into L2 before the wait; otherwise every older block of + the CTA is prefetched. Neither changes a score. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="three requests verifying 1 + 3 rows (the narrowest span the fused score + select leaves to it)", + spec=dict(seq_lens=(20000, 3000, 129), qlen=4, early=False), + ), + Cell( + why="three requests, one longer than 512 blocks (the CTAs loop)", + spec=dict(seq_lens=(70000, 2049, 4096), qlen=4, early=False), + ), + Cell( + why="one request verifying 1 + 7 rows (a 7-draft verify the fused score + select does not take)", + spec=dict(seq_lens=(20000,), qlen=8, early=False), + ), + Cell( + why="two requests verifying 1 + 7 rows, one longer than 512 blocks (the CTAs loop)", + spec=dict(seq_lens=(70000, 3000), qlen=8, early=False), + ), + Cell( + why="one request verifying 1 + 7 rows on the early-issue schedule (the K7 batch-1 decode step)", + spec=dict(seq_lens=(131072,), qlen=8, early=True), + ), + Cell( + why="eight requests of mixed lengths: the prefetching kernel", + spec=dict( + seq_lens=(300, 1500, 2049, 4096, 5000, 7777, 8192, 33001), qlen=4, early=False + ), + ), + Cell( + why="eight requests decoding one row each", + spec=dict( + seq_lens=(300, 1500, 2049, 4096, 5000, 7777, 8192, 33001), qlen=1, early=False + ), + ), + ) + + note = """ + The products of FP8 (or BF16) values are exact in fp32 and the kernel sums 128 of them in its k-split order, so it + differs from the reference matmul by a few fp32 ulps; it is held to the in-tree scorer bit for bit. + """ + + def __call__( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + _, num_heads, head_dim = idx_q.shape + key = (idx_q.dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + compiled_scorer(key)(idx_q, index_k, block_table, score, seq_lens) + return score + + def reference( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + """The fp32 [heads, total_q, blocks] scores (``-inf`` where nothing is written). ``score``, ``split_k`` and + ``early`` do not change them.""" + total_q, heads, _ = idx_q.shape + qlen = max_decode_query_len + out = torch.full( + (heads, total_q, score.shape[2]), + -float("inf"), + dtype=torch.float32, + device=idx_q.device, + ) + with true_fp32_matmul(): + for b, seq in enumerate(seq_lens.tolist()): + nblocks = (seq + PAGE - 1) // PAGE + if nblocks == 0: + continue + t0 = b * qlen + q = idx_q[t0 : t0 + qlen].float() + k = index_k[block_table[b, :nblocks].long()].reshape(nblocks * PAGE, -1).float() + s = torch.einsum("qhd,kd->hqk", q, k) + q_pos = seq - qlen + torch.arange(qlen, device=idx_q.device) + k_pos = torch.arange(k.shape[0], device=idx_q.device) + s.masked_fill_(k_pos[None, :] > q_pos[:, None], -float("inf")) + out[:, t0 : t0 + qlen, :nblocks] = s.reshape(heads, qlen, nblocks, PAGE).amax(dim=3) + return out + + def is_valid( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> None: + # The block-table row and the score row are read and written up to ceil(seq_len / 128) with no bound. + nblocks = (seq_lens + PAGE - 1) // PAGE + assert bool((nblocks <= block_table.shape[1]).all()), ( + "a request runs past its block-table row" + ) + assert bool((nblocks <= score.shape[2]).all()), "a request runs past the score buffer" + assert idx_q.shape[0] == seq_lens.shape[0] * max_decode_query_len, ( + "idx_q must hold max_decode_query_len rows per request" + ) + + +m3_index_score = M3IndexScore() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_msa_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_msa_attention.py new file mode 100644 index 000000000000..60990ce4a415 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_msa_attention.py @@ -0,0 +1,224 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MiniMax-M3 MSA attention backend's step: KV write, sparse top-k block selection, FMHA.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + SparseBackendForwardArgs, +) + +from .m3_sparse_decode_lc8 import dequant_nvfp4_pages + + +class M3MsaAttention(OpWrapper): + """One layer's attention core on the MSA backend (``MiniMaxM3MsaSparseAttention``), written into ``output`` + [T, heads * 128] (bf16, the step's live rows). + + * ``k`` / ``v`` given (the dense layers): written into the layer's paged main cache first (NVFP4 with + ``kv_scale_orig_quant``, else FP8 / BF16 as the pool is). None: a producer already wrote the caches. + * ``sparse``: the indexer scores the layer's index-K cache (already written) with ``idx_q`` and publishes the + top-k blocks; the FMHA attends those blocks. Otherwise the FMHA attends the whole page table. + + The FMHA dispatches on the step's metadata (context rows: the MSA prefill FMHA; generation rows: the MSA decode + FMHA, the dense layers' trtllm-gen decode at their capped KV split). ``backend`` and ``metadata`` are the runtime's + objects for this layer and step. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="dense layer, a prefill chunk, FP8 pool", + spec=dict(layer="dense", phase="context", tokens=1024), + ), + Cell( + why="dense layer, a decode step of 4 requests", + spec=dict(layer="dense", phase="generation", reqs=4), + ), + Cell( + why="sparse layer, a prefill chunk beyond 16 blocks, NVFP4 pool", + spec=dict(layer="sparse", phase="context", tokens=4096), + ), + Cell( + why="sparse layer, a mixed step (context + generation rows)", + spec=dict(layer="sparse", phase="mixed", tokens=512, reqs=2), + ), + ) + + note = """ + `reference` is exact (float64) softmax attention over the cache rows the step attends, read back from the paged + cache the call wrote (or a producer wrote before it) and dequantized as the FMHA dequantizes them: FP8 rows as + E4M3, NVFP4 rows as E2M1 x E4M3 block scale x the per-tensor ``kv_scale_quant_orig``. A sparse layer's top-k blocks + are re-selected from ``idx_q`` and the index-K cache with the backend's rule (per-block max over the causal + keys, the query's own block forced in, ties to the lower block). On an FP8 pool the FMHA takes Q as a plain E4M3 + cast, which the reference applies too. What remains between the two is the FMHA's own arithmetic (its P + rounding and accumulation order), which the test bounds. + """ + + def __call__( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + if k is not None: + backend.write_layer_caches( + k, v, None, metadata, kv_scale_orig_quant=kv_scale_orig_quant + ) + if sparse: + blocks = backend.run_indexer(idx_q, None, metadata, idx_k_prewritten=True) + args = AttentionForwardArgs( + output=output, sparse_backend_args=SparseBackendForwardArgs(topk_indices=blocks) + ) + else: + args = AttentionForwardArgs(output=output) + args.kv_scale_orig_quant = kv_scale_orig_quant + args.kv_scale_quant_orig = kv_scale_quant_orig + backend.forward(q, None, None, metadata, forward_args=args) + return output + + def reference( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + """``output`` as the call is supposed to write it, [T, heads * 128] in ``output``'s dtype. + + Every attended row -- the step's own included -- is read from the layer's paged cache, at the slots the + metadata's KV cache manager holds for each request, so this is evaluated after the call (whose write of + ``k`` / ``v`` it then reads back). ``k``, ``v`` and ``kv_scale_orig_quant`` are therefore not read; the + test checks the write itself. Query token ``j`` of request ``r`` sits at position ``cached_r + j`` and + attends positions up to it; on a sparse layer only those in its top-k blocks. Mirrors one KV head per rank, + the target's layout. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + assert backend.num_kv_heads == 1, "mirrors the one-KV-head-per-rank layout only" + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = sum(qlens) + assert q.shape[0] == total and output.shape[0] == total + + pool = manager.get_buffers(layer, "HND") + nvfp4 = manager.is_nvfp4_layer(layer) + if nvfp4: + scale_pool = manager.get_block_scale_buffers(layer, "HND") + k_global = float(kv_scale_quant_orig.reshape(-1)[1]) + v_global = float(kv_scale_quant_orig.reshape(-1)[2]) + q_rows = q.reshape(total, heads, dim) + if q_rows.dtype != torch.float8_e4m3fn and pool.dtype == torch.float8_e4m3fn: + # The FP8 FMHA shares one dtype across Q / K / V: Q is cast to E4M3, unscaled. + q_rows = q_rows.to(torch.float8_e4m3fn) + q64 = q_rows.to(torch.float64) + if sparse: + config = backend.m3_config + idx_cache = metadata.msa_idx_k_cache(layer) + idx64 = idx_q.reshape(total, config.num_index_heads, config.sparse_index_dim).to( + torch.float64 + ) + + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=q.device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + blocks = (length + page - 1) // page + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][:blocks], + dtype=torch.int64, + device=q.device, + ) + if nvfp4: + keys = dequant_nvfp4_pages( + pool[:, 0].view(torch.uint8), scale_pool[:, 0], pages, quad_scales=False + ) + values = dequant_nvfp4_pages( + pool[:, 1].view(torch.uint8), scale_pool[:, 1], pages, quad_scales=True + ) + keys, values = keys * k_global, values * v_global + else: + keys = pool[pages, 0, 0].to(torch.float64) + values = pool[pages, 1, 0].to(torch.float64) + keys = keys.reshape(-1, dim)[:length] + values = values.reshape(-1, dim)[:length] + qpos = cached[row] + torch.arange(n, device=q.device) + allowed = torch.arange(length, device=q.device)[None, :] <= qpos[:, None] + if sparse: + idx_keys = idx_cache[pages, 0].to(torch.float64).reshape(-1, dim)[:length] + chosen = _selected_blocks( + idx64[first : first + n], idx_keys, allowed, qpos, page, config + ) + allowed &= chosen.repeat_interleave(page, dim=1)[:, :length] + for lo in range(0, n, 256): + hi = min(n, lo + 256) + logits = torch.einsum("thd,kd->thk", q64[first + lo : first + hi], keys) * sm_scale + logits = logits.masked_fill(~allowed[lo:hi, None, :], float("-inf")) + out[first + lo : first + hi] = torch.softmax(logits, dim=-1) @ values + first += n + return out.reshape(total, heads * dim).to(output.dtype) + + +def _selected_blocks( + idx_q: torch.Tensor, + idx_keys: torch.Tensor, + allowed: torch.Tensor, + qpos: torch.Tensor, + page: int, + config, +) -> torch.Tensor: + """The blocks each query token of one request attends, [n, blocks] bool. + + A block's score is the maximum, over the KV head's index heads and the block's causally visible keys, of + ``idx_q . idx_k``. Of the token's ``ceil((pos + 1) / page)`` valid blocks the first ``init_blocks`` and the last + ``local_blocks`` are forced in, and the ``topk`` highest-scoring are kept, ties to the lower block (the + backend's ``select_blocks_from_maxscore`` rule). The positive score scale the scorers apply does not change the + ranking, so none is applied. + """ + n, length = allowed.shape + blocks = (length + page - 1) // page + scores = torch.einsum("thd,kd->thk", idx_q, idx_keys).amax(dim=1) + scores = scores.masked_fill(~allowed, float("-inf")) + padded = torch.full((n, blocks * page), float("-inf"), dtype=scores.dtype, device=scores.device) + padded[:, :length] = scores + block_scores = padded.view(n, blocks, page).amax(dim=-1) + ids = torch.arange(blocks, device=scores.device)[None, :] + n_valid = torch.div(qpos + page, page, rounding_mode="floor")[:, None] + block_scores = block_scores.masked_fill(ids >= n_valid, float("-inf")) + forced = (ids < torch.clamp(n_valid, max=config.init_blocks)) | ( + (ids >= n_valid - config.local_blocks) & (ids < n_valid) + ) + block_scores = block_scores.masked_fill(forced, float("inf")) + order = torch.sort(block_scores, dim=1, descending=True, stable=True).indices[:, : config.topk] + keep = torch.arange(order.shape[1], device=scores.device)[None, :] < n_valid + chosen = torch.zeros(n, blocks, dtype=torch.bool, device=scores.device) + rows = torch.arange(n, device=scores.device)[:, None].expand_as(order) + chosen[rows[keep], order[keep]] = True + return chosen + + +m3_msa_attention = M3MsaAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_producer.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_producer.py new file mode 100644 index 000000000000..783304219ae5 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_producer.py @@ -0,0 +1,202 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' NVFP4 QKV + indexer RMSNorm / RoPE / KV-insert producer, with its constant loads staged +before the PDL grid-dependency wait and an early dependents trigger (``kernels/minimaxM3/producerPw.cu``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +__all__ = ["m3_producer"] + +HEAD_DIM = 128 +ROTARY_DIM = 64 +PAGE = 128 +#: Query heads per rank at TP8 (MiniMax-M3: 64 query heads; 1 KV head and 1 index head per rank). +HEADS_Q = 8 +#: NVFP4 block: 16 values share one E4M3 scale. +FP4_BLOCK = 16 +E2M1_VALUES = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0) + + +def _norm_rope( + x: torch.Tensor, weight: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor, eps: float +): + """Gemma RMSNorm (``x * rsqrt(mean(x^2) + eps) * (1 + w)``) then NeoX RoPE on the first 64 dims, in fp32. + x [T, 128] fp32; cos / sin [T, 32].""" + y = x * torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) * (1.0 + weight.float()) + lo, hi, rest = y[:, :32], y[:, 32:64], y[:, 64:] + return torch.cat([lo * cos - hi * sin, hi * cos + lo * sin, rest], dim=-1) + + +def decode_nvfp4_token( + data: torch.Tensor, scales: torch.Tensor, quant_scale: float +) -> torch.Tensor: + """fp32 [128] of one token's packed E2M1 row (64 bytes, low nibble first) and its 8 E4M3 block scales.""" + table = torch.tensor(E2M1_VALUES, dtype=torch.float32, device=data.device) + nib = torch.stack([data & 0xF, data >> 4], dim=-1).reshape(-1).long() + vals = table[nib & 7] * torch.where(nib >= 8, -1.0, 1.0) + s = scales.view(torch.float8_e4m3fn).float() / quant_scale + return vals * s.repeat_interleave(FP4_BLOCK) + + +def scale_index(within_page: int, plane: int) -> torch.Tensor: + """Positions of a token's 8 block scales in its page's [128 * 8] scale plane: row-major for K, the 4-token + swizzle for V.""" + cols = torch.arange(8) + if plane == 0: + return within_page * 8 + cols + return (within_page // 4) * 32 + cols * 4 + within_page % 4 + + +class M3Producer(OpWrapper): + """The NVFP4 QKV + indexer producer of a sparse layer: ``minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert`` for one + KV head, one index head, head dim 128 and rotary dim 64, with the same arguments and outputs bit for bit. + + ``packed`` [T, (heads_q + 4) * 128] BF16 holds each token's q heads, k, v, index-q and index-k. Every head but v + gets the Gemma RMSNorm with its weight and NeoX RoPE on its first 64 dims (``rotary_cos_sin[position]``). Returns + (q FP8 [T, heads_q, 128], index-q FP8 [T, 1, 128]); writes the token's slot ``out_cache_loc[t]`` (skipped when + negative or past the pool): k and v as NVFP4 (16-value blocks, E4M3 scales against ``kv_quant_scale[1]`` / + ``[2]``, the V scales in the 4-token swizzle) and index-k as FP8. ``early`` triggers the dependent kernel on + entry (its outputs are read only after the dependent's own grid wait). + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token (c1 without speculation)", spec=dict(tokens=1, scale=1.0)), + Cell(why="a c1 verify: 4 tokens", spec=dict(tokens=4, scale=3.0)), + Cell(why="a K7 c1 verify: 8 tokens", spec=dict(tokens=8, scale=3.0)), + Cell(why="a c4 verify: 16 tokens, small activations", spec=dict(tokens=16, scale=1e-3)), + Cell( + why="a c8 verify: 32 tokens, large activations, a skipped and an out-of-pool slot", + spec=dict(tokens=32, scale=40.0, bad_slots=True), + ), + Cell(why="the widest decode step before R1: 48 tokens", spec=dict(tokens=48, scale=1.0)), + Cell( + why="the widest decode step (R1): a K7 c8 verify, 64 tokens, a skipped and an out-of-pool slot", + spec=dict(tokens=64, scale=40.0, bad_slots=True), + ), + Cell( + why="a prefill chunk of 1000 tokens, early trigger off", + spec=dict(tokens=1000, scale=1.0, early=False), + ), + ) + + note = """ + The kernel is the native producer's arithmetic, compiled with the library's --use_fast_math, so it is held to that + kernel bit for bit on every output and cache byte. The torch reference states the math in fp32: the FP8 outputs are + held to it within one E4M3 step, and the NVFP4 K/V, decoded, within one E2M1 step of their block scale. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + q, index_q = torch.ops.trtllm.minimax_m3_producer_pw( + packed, + kv_data, + kv_scale, + index_k, + out_cache_loc, + kv_quant_scale, + eps, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + rotary_cos_sin, + position_ids, + heads_q, + pdl, + early, + ) + return q, index_q + + def reference( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + ) -> dict[str, torch.Tensor]: + """fp32 values before their quantization: "q" [T, heads_q, 128], "index_q" [T, 128], and per token "k", "v", + "index_k" [T, 128] (the values written to the token's slot). The caches, ``pdl`` and ``early`` do not change + them.""" + t = packed.shape[0] + x = packed.float().view(t, heads_q + 4, HEAD_DIM) + rope = rotary_cos_sin[position_ids[:t].long()] + cos, sin = rope[:, 0], rope[:, 1] + q = torch.stack( + [_norm_rope(x[:, h], q_weight, cos, sin, eps) for h in range(heads_q)], dim=1 + ) + return { + "q": q, + "k": _norm_rope(x[:, heads_q], k_weight, cos, sin, eps), + "v": x[:, heads_q + 1], + "index_q": _norm_rope(x[:, heads_q + 2], index_q_weight, cos, sin, eps), + "index_k": _norm_rope(x[:, heads_q + 3], index_k_weight, cos, sin, eps), + } + + def is_valid( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + ) -> None: + t = packed.shape[0] + # The RoPE row is read at the position with no bound. + pos = position_ids[:t] + assert bool(((pos >= 0) & (pos < rotary_cos_sin.shape[0])).all()), ( + "position_ids must index rotary_cos_sin" + ) + # Two tokens writing one slot race. + slots = out_cache_loc[:t] + live = slots[(slots >= 0) & (slots < kv_data.shape[0] * PAGE)] + assert live.unique().numel() == live.numel(), "out_cache_loc must not repeat a slot" + + +m3_producer = M3Producer() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_qk_norm_rope_fp8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..681aec29472e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_qk_norm_rope_fp8.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' per-head Gemma RMSNorm + partial NeoX RoPE of the packed [Q | K | V], into a fresh FP8 copy.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3QkNormRopeFp8(OpWrapper): + """``fused_qk_norm_rope`` written out of place in FP8 E4M3: Q and K heads normed and roped in fp32 and rounded + once to E4M3 (saturating at +/-448), V cast, ``qkv`` left untouched. Same arguments as the catalog's in-place + entry. + + The dense layers (0-2) feed the FP8 main KV cache with it: on every step of an FP8 cache, and on the decode steps + of a hybrid NVFP4 cache (whose dense layers sit in its FP8 pool). + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, tp8 heads (8 q, 1 kv)", spec=dict(m=1, q=8, kv=1)), + Cell(why="a 48-row decode verify", spec=dict(m=48, q=8, kv=1)), + Cell(why="a prefill chunk on an FP8 cache", spec=dict(m=2048, q=8, kv=1)), + ) + + note = """ + `reference` is the in-place kernel followed by a cast to E4M3. V matches it bit for bit; Q and K round once from + fp32 where the composition rounds through bf16 first, so they are within one E4M3 step of it. + """ + + def __call__( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + return torch.ops.trtllm.fused_qk_norm_rope_to_fp8( + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + + def reference( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + """The catalog's in-place kernel on a copy, then the E4M3 cast (V exact; Q and K within one E4M3 step).""" + out = fused_qk_norm_rope( + qkv.clone(), + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + return out.to(torch.float8_e4m3fn) + + +m3_qk_norm_rope_fp8 = M3QkNormRopeFp8() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_qkv_insert.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_qkv_insert.py new file mode 100644 index 000000000000..fc46936cc839 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_qkv_insert.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' generic horizontal producer: packed QKV/index rows -> FP8 Q / index-Q, K / V / index-K into +the paged caches.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3QkvInsert(OpWrapper): + """The packed ``[Q | K | V | index-Q | index-K]`` bf16 rows of a sparse layer: per-head Gemma RMSNorm and the + partial NeoX RoPE of Q, K, index-Q and index-K, K / V written to the main paged cache at ``out_cache_loc`` and + index-K to the FP8 index cache; returns (Q, index-Q) in FP8 E4M3 [M, heads, 128]. + + ``kv_scale`` given: the main cache is NVFP4 (packed E2M1 data [pages, 2, kv, 128, 64] uint8 + E4M3 scale pool + [pages, 2, kv, 128, 8], ``kv_quant_scale`` the fp32 (Q, K, V) quantization scales); None: FP8 E4M3 + [pages, 2, kv, 128, 128]. Token-major and step-type agnostic. ``m3_producer`` takes the NVFP4 cache at the tp8 + heads; this entry takes the FP8 cache and NVFP4 pools outside that layout. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="FP8 main cache, a prefill chunk", spec=dict(m=1024, cache="fp8")), + Cell(why="FP8 main cache, one decode token", spec=dict(m=1, cache="fp8")), + ) + + note = """ + The arithmetic is the CUDA producer's (fp32 norm and RoPE with bf16 roundings, then the E4M3 casts); `reference` + returns the fp32 values, and the test holds the FP8 outputs and cache rows within one E4M3 step of them. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + common = ( + num_heads, + num_kv_heads, + num_index_heads, + head_dim, + rotary_dim, + eps, + q_norm, + k_norm, + index_q_norm, + index_k_norm, + rotary_cos_sin, + position_ids, + ) + if kv_scale is not None: + return torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, kv_scale, index_k_cache, out_cache_loc, kv_quant_scale, *common + ) + return torch.ops.trtllm.minimax_m3_fp8_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, index_k_cache, out_cache_loc, *common + ) + + def reference( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + """The fp32 values the producer rounds (a new dict, nothing written): Gemma RMSNorm + NeoX partial RoPE of + Q [M, heads, D], index-Q [M, index heads, D], K [M, kv heads, D] and index-K [M, D]; V [M, kv heads, D] as + packed. The cache rows at ``out_cache_loc`` and the returned FP8 Q / index-Q are these values cast.""" + m = packed.shape[0] + sizes = [ + num_heads * head_dim, + num_kv_heads * head_dim, + num_kv_heads * head_dim, + num_index_heads * head_dim, + head_dim, + ] + q, k, v, iq, ik = packed.float().split(sizes, dim=-1) + cos, sin = rotary_cos_sin[position_ids.long()].unbind(1) + half = rotary_dim // 2 + + def norm_rope(x, w, heads): + x = x.reshape(m, heads, head_dim) + x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * (1.0 + w.float()) + x1, x2, rest = x[..., :half], x[..., half:rotary_dim], x[..., rotary_dim:] + c, s = cos.unsqueeze(1), sin.unsqueeze(1) + return torch.cat([x1 * c - x2 * s, x2 * c + x1 * s, rest], dim=-1) + + return { + "q": norm_rope(q, q_norm, num_heads), + "index_q": norm_rope(iq, index_q_norm, num_index_heads), + "k": norm_rope(k, k_norm, num_kv_heads), + "v": v.reshape(m, num_kv_heads, head_dim), + "index_k": norm_rope(ik, index_k_norm, 1)[:, 0], + } + + +m3_qkv_insert = M3QkvInsert() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_blocks.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_blocks.py new file mode 100644 index 000000000000..6bf1764bfc0d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_blocks.py @@ -0,0 +1,82 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MSA top-k block selection from per-KV-head block scores (the built-in selector).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SelectBlocks(OpWrapper): + """Per query row the ``topk`` highest-scoring KV blocks of ``max_score_kv`` [kv_heads, n_blocks, total_q] (fp32), + with the first ``init_blocks`` and the last ``local_blocks`` valid blocks forced in and blocks at or beyond the + row's ``n_valid_blocks`` excluded; [total_q, kv_heads, topk] int32 ascending ids, -1 padded. ``head_major_output`` + backs the result head-major so ``permute(1, 0, 2)`` is contiguous. + + The decode dispatch runs it where the long-row selector does not apply: requests of 17..128 KV blocks. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="eight decode rows of 17..128 blocks", + spec=dict(total_q=8, n_blocks=128, mixed=True), + ), + Cell( + why="one decode row at the 128-block edge", + spec=dict(total_q=1, n_blocks=128, mixed=False), + ), + Cell(why="a 4-row verify at 40 blocks", spec=dict(total_q=4, n_blocks=40, mixed=False)), + ) + + note = """ + Exact: ties between equal scores break toward the lower block id; `reference` sorts with the same rule. + """ + + def __call__( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + nvb = n_valid_blocks.to(device=max_score_kv.device, dtype=torch.int32).contiguous() + return torch.ops.trtllm.minimax_m3_select_blocks( + max_score_kv, nvb, topk, init_blocks, local_blocks, head_major_output + ) + + def reference( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + """Plain torch selection; ``head_major_output`` changes the backing only.""" + heads, n_blocks, total_q = max_score_kv.shape + out = torch.full((total_q, heads, topk), -1, dtype=torch.int32, device=max_score_kv.device) + blocks = torch.arange(n_blocks, device=max_score_kv.device) + for t in range(total_q): + nv = int(n_valid_blocks[t]) + for h in range(heads): + s = max_score_kv[h, :, t].clone().double() + s[blocks >= nv] = float("-inf") + forced = (blocks < min(init_blocks, nv)) | ( + (blocks >= nv - local_blocks) & (blocks < nv) + ) + s[forced] = float("inf") + order = sorted(range(nv), key=lambda b: (-float(s[b]), b))[: min(topk, nv)] + ids = torch.tensor(sorted(order), dtype=torch.int32, device=out.device) + out[t, h, : ids.numel()] = ids + return out + + +m3_select_blocks = M3SelectBlocks() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_long.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_long.py new file mode 100644 index 000000000000..867822e7fe41 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_long.py @@ -0,0 +1,172 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 MSA block selection for decode rows of up to 9216 blocks.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Blocks per decode row the op covers (9 keys per thread of one 1024-thread CTA). +MAX_BLOCKS = 9216 +#: Blocks the selection keeps per (query, KV head). +TOPK = 16 +#: Scores the forcing substitutes: initial blocks rank above local ones, and both above any finite score. +INIT_SCORE = 1.0e30 +LOCAL_SCORE = 1.0e29 + +# M3's checkpoint: no forced initial blocks, the one block holding the query forced in. +_M3 = dict(num_kv_heads=1, init_blocks=0, local_blocks=1) + + +class M3SelectLong(OpWrapper): + """Fill ``out`` [total_q, num_kv_heads, 16] with each query's top-16 blocks. + + ``scores`` is the per-KV-head block maxima [num_kv_heads, num_blocks, + total_q] the CuTe DSL index scorer wrote (amax over the KV head's index + heads). Per (query, KV head) row the op keeps the ``n_valid_blocks[query]`` + leading blocks (clamped to ``[0, num_blocks]``), forces the last + ``local_blocks`` of them (counted from the *unclamped* valid count) to + ``LOCAL_SCORE`` and the first ``init_blocks`` to ``INIT_SCORE`` -- local wins + where both apply -- and selects the 16 largest, ties to the smaller block. + A pick whose forced score is -inf is emitted as -1. The ids are written + ascending with a -1 tail. + + Same contract as ``torch.ops.trtllm.minimax_m3_select_blocks`` (the in-tree + ``select_blocks_from_maxscore``) and bit-identical to it, except on a row + where more than 1024 valid blocks tie at the 16th-largest score: this op + breaks that tie to the smallest blocks as the contract says, the in-tree op + does not. The test checks both against ``reference``. Returns ``out``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="32K-context engine (257 worst-case k-tiles), Eagle3 verify at one request: 4 query rows", + spec=dict(num_blocks=257, total_q=4, valid="near_full", scores="random", **_M3), + ), + Cell( + why="decode without speculation at batch 8 on a 2048-block engine: the 2-key-per-thread path at its edge", + spec=dict(num_blocks=2048, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="1M-context engine at batch 8 x qlen 4: rows on all three key widths, short rows and padding rows", + spec=dict(num_blocks=8193, total_q=32, valid="mixed", scores="random", **_M3), + ), + Cell( + why="duplicated scores: ties broken to the smaller block, and > 1024 keys at the bound (re-read path)", + spec=dict(num_blocks=4500, total_q=4, valid="near_full", scores="ties", **_M3), + ), + Cell( + why="non-finite scores: +inf, NaN ranked above +inf, and rows whose picks run into -inf (emitted -1)", + spec=dict(num_blocks=600, total_q=8, valid="mixed", scores="nonfinite", **_M3), + ), + ) + + note = """ + Certified at one KV head per rank (TP8) with M3's forcing (`init_blocks=0`, + `local_blocks=1`). The kernel takes any head count and forcing; neither is + driven here. + + `scores` may be strided (the op reads it by stride); the target hands over + the scorer's contiguous view, which is what the cells drive. Rows of at + most 128 blocks are within the kernel's range, but the target keeps those on + the in-tree op, so they are not certified here. + + Ordering is by the descending-score key `orderedKey`: a positive NaN ranks + above +inf, +0 above -0, and a negative NaN below -inf -- which, unlike + -inf, is emitted as a block id rather than -1. The cells drive positive + NaN only. + """ + + def __call__( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_select_long( + scores, n_valid_blocks, out, init_blocks, local_blocks, use_pdl, early_trigger + ) + return out + + def reference( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + ) -> torch.Tensor: + """The [total_q, num_kv_heads, 16] int32 table the op is supposed to write. + + Exact integer ranking, no floating point beyond the forcing: each + score's bit pattern maps to an unsigned key that is smaller for a larger + score (NaN above +inf), packed with the block id so one sort ranks by + (score descending, block ascending). ``out``, ``use_pdl`` and + ``early_trigger`` only say where and how the op writes; they do not enter the result. + """ + num_kv_heads, num_blocks, total_q = scores.shape + device = scores.device + blocks = torch.arange(num_blocks, device=device, dtype=torch.int64) + raw = n_valid_blocks[:total_q].to(torch.int64) + valid = raw.clamp(0, num_blocks) + local_start = (raw - local_blocks).clamp_min(0) + result = torch.full((total_q, num_kv_heads, TOPK), -1, dtype=torch.int32, device=device) + for head in range(num_kv_heads): + forced = scores[head].float().t().clone() # [total_q, num_blocks] + forced = torch.where(blocks[None, :] < init_blocks, INIT_SCORE, forced) + forced = torch.where(blocks[None, :] >= local_start[:, None], LOCAL_SCORE, forced) + bits = forced.view(torch.int32).to(torch.int64) & 0xFFFFFFFF + key = torch.where(bits >= 0x80000000, bits, (~bits) & 0x7FFFFFFF) + in_row = blocks[None, :] < valid[:, None] + packed = torch.where( + in_row, (key << 16) | blocks[None, :], torch.iinfo(torch.int64).max + ) + k = min(TOPK, num_blocks) + picked = torch.topk(packed, k, dim=1, largest=False).values + picked_key = picked >> 16 + keep = (picked != torch.iinfo(torch.int64).max) & (picked_key != 0xFF800000) + ids = torch.where(keep, picked & 0xFFFF, torch.iinfo(torch.int64).max) + ids = ids.sort(dim=1).values + ids = torch.where(ids == torch.iinfo(torch.int64).max, -1, ids) + result[:, head, :k] = ids.to(torch.int32) + return result + + def is_valid( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + ) -> None: + """The inputs this op takes and answers wrongly. + + Negative forcing counts are accepted and push the forced ranges past the + row instead of failing; the in-tree op this one stands in for refuses + them, so a caller relying on this one would silently diverge from it. + An ``out`` overlapping ``scores`` is overwritten while it is still read. + """ + assert init_blocks >= 0 and local_blocks >= 0, ( + f"init_blocks ({init_blocks}) and local_blocks ({local_blocks}) must be non-negative; the " + "op accepts negative counts and selects as if the forced ranges were elsewhere" + ) + storage = scores.untyped_storage() + assert not (storage.data_ptr() <= out.data_ptr() < storage.data_ptr() + storage.nbytes()), ( + "out must not alias scores: the op writes it while scores are still read" + ) + + +m3_select_long = M3SelectLong() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_long_dsl.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_long_dsl.py new file mode 100644 index 000000000000..425fe62da388 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_select_long_dsl.py @@ -0,0 +1,126 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 MSA block selection for decode rows of up to 9216 blocks, one row per 384-thread CTA (CuTe DSL).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_select_long import _M3, MAX_BLOCKS, M3SelectLong + + +class M3SelectLongDsl(OpWrapper): + """``m3_select_long``'s contract, cells and reference, run by ``ops/m3_select_long_dsl``: the same selection + bit for bit as the 1024-thread ``minimax_m3_select_long`` (also on rows where more than 2048 blocks tie + at the 16th-largest score), from one 384-thread CTA per row. Returns ``out``.""" + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 4 rows over 165521 tokens (1294 blocks, the AgentX ISL): the 2-key path", + spec=dict(num_blocks=1294, total_q=4, valid="near_full", scores="random", **_M3), + ), + Cell( + why="5888-block rows at two requests x 4 rows: the 9-key path", + spec=dict(num_blocks=5888, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="the longest covered rows (9216 blocks, 1.18M tokens), 2 requests x 4 rows: the 9-key path at its end", + spec=dict(num_blocks=9216, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="~1500 blocks tied at the 16th score: the in-tree op also takes the smaller blocks (bit for bit)", + spec=dict(num_blocks=2000, total_q=8, valid="near_full", scores="ties1500", **_M3), + ), + Cell( + why="two requests x 4 rows of mixed lengths, 4500-block engine: every key width, short and padding rows", + spec=dict(num_blocks=4500, total_q=8, valid="mixed", scores="random", **_M3), + ), + Cell( + why="duplicated scores: ties broken to the smaller block, and > 2048 keys at the bound (re-read path)", + spec=dict(num_blocks=4500, total_q=4, valid="near_full", scores="ties", **_M3), + ), + Cell( + why="non-finite scores: +inf, NaN ranked above +inf, and rows whose picks run into -inf (emitted -1)", + spec=dict(num_blocks=600, total_q=8, valid="mixed", scores="nonfinite", **_M3), + ), + ) + + def reference( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + ) -> torch.Tensor: + """``m3_select_long``'s reference (which takes no trigger phase): the selection is the same bit for bit.""" + return M3SelectLong.reference( + self, scores, n_valid_blocks, out, init_blocks, local_blocks, use_pdl, early_trigger + ) + + def is_valid( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + ) -> None: + """``m3_select_long``'s domain.""" + M3SelectLong.is_valid( + self, scores, n_valid_blocks, out, init_blocks, local_blocks, use_pdl, early_trigger + ) + + note = """ + Certified at one KV head per rank (TP8, GB300) with M3's forcing + (`init_blocks=0`, `local_blocks=1`), and bit for bit against the + 1024-thread `m3_select_long` on every cell. The kernel takes any head count + and forcing; other forcing is not driven here. + + The kernel is compiled per (PDL launch, early trigger, trigger phase) with + symbolic shapes and strides; the target compiles its keys before warmup + (`ops/m3_select_long_dsl.prebuild`), and a call outside them compiles + on first use. + """ + + def __call__( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + ) -> torch.Tensor: + from ..m3_select_long_dsl import select_long_dsl + + if scores.shape[1] > MAX_BLOCKS: + raise RuntimeError( + f"m3_select_long_dsl supports at most {MAX_BLOCKS} blocks, got {scores.shape[1]}" + ) + select_long_dsl( + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + ) + return out + + +m3_select_long_dsl = M3SelectLongDsl() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_sparse_decode.py new file mode 100644 index 000000000000..0438d20109b5 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_sparse_decode.py @@ -0,0 +1,215 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""This target's block-sparse decode attention (Triton, with the one-launch long-context route).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..triton_sparse_decode import minimax_m3_sparse_attn_decode +from .m3_sparse_decode_lc8 import PAGE, sparse_attention_reference + + +class M3SparseDecode(OpWrapper): + """``minimax_m3_sparse_attn_decode`` as ``M3Attention._decode_sparse_core`` calls it. + + Attention of each query token over the blocks ``topk_idx`` selected for it + (``[num_kv_heads, T, topk]``, ascending ids with a -1 tail), each block + truncated at the token's causal extent, written into ``output`` in place. + The target passes FP8 q, the packed NVFP4 sparse-layer cache with its block + and per-tensor scales, and BF16 ``output`` [T, 8, 128]. Returns ``output``. + + Which kernel runs is decided inside, per step: a selected (non-identity) + table on the TP8 geometry goes to the one-launch ``m3_sparse_decode_lc8`` + kernel; an identity table (every request within 16 blocks, staged by the + metadata) runs the Triton producer, or from 17 query rows the multi-query + producer, then the LSE merge. Every route also publishes the MXFP8 o_proj + hand-off to ``m3_prequant.SLOT``; this entry certifies the attention output. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="identity selection, decode without speculation at batch 8: the Triton producer + merge", + spec=dict( + seq_lens=(1, 128, 129, 700, 2048, 1500, 64, 0), decode_query_len=1, identity=True + ), + ), + Cell( + why="identity selection, Eagle3 verify at one request (T=4): one block per producer program", + spec=dict(seq_lens=(1900,), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 4 (T=16): the last row count below the MQ producer", + spec=dict(seq_lens=(300, 2048, 4, 1000), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 8 (T=32): the multi-query producer, padding rows", + spec=dict( + seq_lens=(2048, 130, 5, 0, 1777, 900, 2000, 0), decode_query_len=4, identity=True + ), + ), + Cell( + why="selected blocks, Eagle3 verify at batch 2: routed to the one-launch long-context kernel", + spec=dict(seq_lens=(9000, 40000), decode_query_len=4, identity=False), + ), + Cell( + why="selected blocks, decode without speculation at batch 8: the long-context route at T=8", + spec=dict( + seq_lens=(3000, 2100, 9999, 2048, 2049, 4500, 16000, 2200), + decode_query_len=1, + identity=False, + ), + ), + ) + + note = """ + Certified only as the target calls it: one KV head of 8 query heads, + head_dim 128, FP8 q, packed NVFP4 cache, BF16 output, `num_topk_chunks` + left to the dispatcher. The function also takes BF16 / FP8 caches, other + head geometries and an explicit split-K factor (the in-tree tests drive + those); none is certified here. + + The MXFP8 hand-off each route publishes is certified on the long-context + kernel's own entry (`m3_sparse_decode_lc8`); the Triton merge writes it with + the same recipe (`_store_mxfp8_head`). + """ + + def __call__( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + minimax_m3_sparse_attn_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + kv_scale=kv_scale, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + num_topk_chunks=num_topk_chunks, + identity_blocks=identity_blocks, + ) + return output + + def reference( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + """``output`` as the call is supposed to write it: float64 block-sparse attention over the + dequantized NVFP4 blocks (``sparse_attention_reference``), rounded once to ``output``'s dtype. + + Only the packed NVFP4 cache with one KV head is mirrored -- the target's + only configuration. ``kv_scale``, ``num_topk_chunks`` and + ``identity_blocks`` do not change what is computed. + """ + assert k_block_scale is not None and k_paged.shape[1] == 1, ( + "mirrors the one-KV-head NVFP4 cache only" + ) + attn = sparse_attention_reference( + q, + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx[0], + block_table, + seq_lens, + decode_query_len, + sm_scale, + ) + return attn.to(output.dtype) + + def is_valid( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> None: + """The inputs this call takes and answers wrongly. + + Every producer walks a token's first ``min(topk, ceil(kv_len / 128))`` + table entries and indexes the block table with them unchecked, so they + must be real blocks of the request. ``identity_blocks`` is a promise the + multi-query producer relies on: every token of a request reads the same + blocks, all of them. Reads ``topk_idx`` and ``seq_lens`` back to the + host: a test-only cost. + """ + total_q = q.shape[0] + seq = seq_lens.tolist() + max_topk = topk_idx.shape[-1] + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk_idx[:, tok, : min(max_topk, nblocks)] + assert bool(((ids >= 0) & (ids < min(nblocks, block_table.shape[1]))).all()), ( + f"token {tok}: the first min(topk, block count) selected ids must be blocks of the request; " + "the block table is indexed with them unchecked" + ) + if identity_blocks: + assert nblocks <= max_topk and bool( + (ids == torch.arange(nblocks, device=ids.device, dtype=ids.dtype)).all() + ), ( + f"token {tok}: identity_blocks promises the table holds every block of the request in order" + ) + + +m3_sparse_decode = M3SparseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_sparse_decode_lc8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_sparse_decode_lc8.py new file mode 100644 index 000000000000..9c5912667cde --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/m3_sparse_decode_lc8.py @@ -0,0 +1,499 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP8 implementation.""" + +from __future__ import annotations + +import os +from typing import Optional, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) + +from ...m31_switches import m31_disabled + +LC8_SYNC3 = not m31_disabled("LC8_SYNC3") +_LC8_STATE = {} + + +def _lc8_launcher(): + """The sparse decode op the entry launches, decided once per process: the SYNC3 copy when switched on and built + (or overlaid), else ``minimax_m3_tp8_sparse_decode_lc8``.""" + if "op" not in _LC8_STATE: + op = None + if LC8_SYNC3: + op = getattr(torch.ops.trtllm, "minimax_m31_tp8_sparse_decode_lc8", None) + if op is None: + import tensorrt_llm + + lib = os.path.join( + os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_lc8sync3.so" + ) + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_tp8_sparse_decode_lc8", None) + from tensorrt_llm.logger import logger + + reason = ( + "engaged" + if op is not None + else "switched off" + if not LC8_SYNC3 + else "not taken: the op is not built" + ) + logger.info(f"[M3 dispatch] LC8_SYNC3 sparse decode softmax: {reason}") + if op is not None: + logger.info(f"[M3 dispatch] LC8 one-division merge: one_div={False}") + _LC8_STATE["m31"] = op is not None + _LC8_STATE["op"] = ( + op if op is not None else torch.ops.trtllm.minimax_m3_tp8_sparse_decode_lc8 + ) + return _LC8_STATE["op"] + + +def _lc8_onediv(rows: int) -> bool: + _lc8_launcher() + return False + + +NUM_HEADS = 8 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 +SF_VEC = 16 +MX_BLOCK = 32 +WIDTH = NUM_HEADS * HEAD_DIM +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) + + +def dequant_nvfp4_pages( + packed: torch.Tensor, block_scale: torch.Tensor, pages: torch.Tensor, quad_scales: bool +) -> torch.Tensor: + """Dequantize KV head 0 of the given pages of a packed NVFP4 cache to float64 [n, PAGE, HEAD_DIM]. + + ``packed`` is the [pages, kv_heads, PAGE, HEAD_DIM // 2] byte view (low + nibble = even element), ``block_scale`` the [pages, kv_heads, PAGE, + HEAD_DIM // 16] E4M3 bytes ``fused_write_layer_caches_nvfp4`` writes: token + ``t``'s column ``c`` at byte ``t * 8 + c`` of the page's region for K, and at + ``(t // 4) * 32 + c * 4 + t % 4`` for V (``quad_scales``). The per-tensor + global scale is not applied. Every product is exact in float64. + """ + table = torch.tensor(_E2M1, dtype=torch.float64, device=packed.device) + rows = packed[pages, 0].to(torch.int64) + nibbles = torch.stack([rows & 15, rows >> 4], dim=-1).reshape(rows.shape[0], PAGE, HEAD_DIM) + cols = HEAD_DIM // SF_VEC + region = block_scale[pages, 0].reshape(rows.shape[0], PAGE * cols) + t = torch.arange(PAGE, device=packed.device)[:, None] + c = torch.arange(cols, device=packed.device)[None, :] + index = t // 4 * 32 + c * 4 + t % 4 if quad_scales else t * cols + c + scales = region[:, index.reshape(-1)].reshape(rows.shape[0], PAGE, cols) + scales = scales.view(torch.float8_e4m3fn).to(torch.float64) + return table[nibbles] * scales.repeat_interleave(SF_VEC, dim=-1) + + +def sparse_attention_reference( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, +) -> torch.Tensor: + """Block-sparse decode attention over one KV head of a packed NVFP4 cache, float64 [T, heads, HEAD_DIM]. + + Query token ``tok`` belongs to request ``tok // decode_query_len`` at offset + ``tok % decode_query_len`` and attends the first ``min(16, ceil(kv_len / + PAGE))`` entries of ``topk[tok]`` that are not -1, each block truncated at + ``kv_len = max(seq_len - decode_query_len + offset + 1, 0)``. A token that + attends nothing (a CUDA-graph padding row) is all zeros. ``q`` is E4M3 (or + its byte view); softmax and both products are float64. + """ + q64 = ( + q.view(torch.float8_e4m3fn).to(torch.float64) + if q.element_size() == 1 + else q.to(torch.float64) + ) + total_q, num_heads, _ = q64.shape + kgs = float(k_global_scale.reshape(-1)[0]) + vgs = float(v_global_scale.reshape(-1)[0]) + out = torch.zeros(total_q, num_heads, HEAD_DIM, dtype=torch.float64, device=q.device) + seq = seq_lens.tolist() + positions = torch.arange(PAGE, device=q.device) + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + real_topk = min(TOPK, (kv_len + PAGE - 1) // PAGE) + blocks = topk[tok, :real_topk] + blocks = blocks[blocks >= 0].to(torch.int64) + if blocks.numel() == 0: + continue + pages = block_table[req, blocks].to(torch.int64) + keys = ( + dequant_nvfp4_pages(k, k_block_scale, pages, quad_scales=False).reshape(-1, HEAD_DIM) + * kgs + ) + values = ( + dequant_nvfp4_pages(v, v_block_scale, pages, quad_scales=True).reshape(-1, HEAD_DIM) + * vgs + ) + valid = (blocks[:, None] * PAGE + positions[None, :] < kv_len).reshape(-1) + logits = q64[tok] @ keys.T * sm_scale + logits = logits.masked_fill(~valid[None, :], float("-inf")) + out[tok] = torch.softmax(logits, dim=-1) @ values + return out + + +def mxfp8_handoff_reference( + out: torch.Tensor, num_rows_alloc: int +) -> Tuple[torch.Tensor, torch.Tensor]: + """The MXFP8 o_proj hand-off the merge writes for BF16 rows ``out`` [T, WIDTH]: ``(q8 [T, WIDTH] uint8, + sf [num_rows_alloc * WIDTH // 32] uint8)``. + + Per 32-element block: ``e`` is the biased exponent of ``amax * rcp(448)`` + rounded up when its mantissa is non-zero (capped at 254), the elements are + scaled by ``2^(127 - e)`` and rounded to E4M3 (saturating), and ``e`` lands + in the 128x4-swizzled scale layout the MXFP8 GEMM reads. Entries of ``sf`` + for rows past ``T`` are zero. Computed from the op's own BF16 rows, so a + comparison against it is exact. + """ + total_q = out.shape[0] + x = out.float().reshape(total_q, WIDTH // MX_BLOCK, MX_BLOCK) + amax = x.abs().amax(dim=-1) + sfv = amax * torch.tensor(1.0 / 448.0, dtype=torch.float32, device=out.device) + bits = sfv.view(torch.int32) + e = (bits >> 23 & 255) + (bits & 8388607 != 0).to(torch.int32) + e = e.clamp(max=254) + qs = torch.where(amax != 0, (254 - e << 23).view(torch.float32), torch.zeros_like(amax)) + xq = (x * qs[..., None]).clamp(-448.0, 448.0) + q8 = xq.to(torch.float8_e4m3fn).view(torch.uint8).reshape(total_q, WIDTH) + num_sf_k_tiles = WIDTH // 128 + sf = torch.zeros(num_rows_alloc * (WIDTH // MX_BLOCK), dtype=torch.uint8, device=out.device) + tok = torch.arange(total_q, device=out.device)[:, None] + k_sf = torch.arange(WIDTH // MX_BLOCK, device=out.device)[None, :] + offset = ( + tok // 128 * num_sf_k_tiles * 512 + + k_sf // 4 * 512 + + tok % 32 * 16 + + tok % 128 // 32 * 4 + + k_sf % 4 + ) + sf[offset.reshape(-1)] = e.to(torch.uint8).reshape(-1) + return (q8, sf) + + +class M3SparseDecodeLc8(OpWrapper): + """Sparse decode attention, split-K merge and MXFP8 hand-off of one TP8 decode step in one launch. + + One cluster of 16 CTAs per query token: CTA ``r`` attends the token's + ``r``-th selected block (``topk[tok, r]``) for all 8 heads (transposed MMAs, + the heads on the n dimension), and the cluster merges the 16 partials over + distributed shared memory into the BF16 row ``out[tok]`` [1024]. With ``q8`` / + ``sf`` it also writes the row's MXFP8 copy and swizzled UE8M0 scales for the + o_proj GEMM. All operands are byte views (E4M3 ``q``, packed E2M1 pages, E4M3 + block scales). The kernel releases its PDL dependents right after its grid + wait. Returns ``out``. + + ``prewait`` stages the request length, the query rows (widened to FP16) and + the request's block-table row (up to 8192 entries, longer rows are read from + global memory) before the grid wait; the arithmetic is the same, so both + variants write the same bits. + + Per block: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16 + (exact), QK and PV on FP16 tensor cores with FP32 accumulation, the softmax + probabilities rounded to FP16 for PV, log2-domain softmax with ``ex2.approx``. + ``linear_softmax`` picks the denominator recurrence (the two agree to + rounding); the target picks it per batch shape. + """ + + ARCHS = frozenset({Arch.SM_103}) + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode without speculation, one request: T=1, log-domain recurrence, MXFP8 hand-off, prewait", + spec=dict( + seq_lens=(9000,), decode_query_len=1, linear_softmax=False, mxfp8=True, prewait=True + ), + ), + Cell( + why="Eagle3 verify at one request (the TP8 operating point): T=4, log-domain recurrence, prewait", + spec=dict( + seq_lens=(20000,), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=True, + ), + ), + Cell( + why="Eagle3 verify at two requests: T=8 on the linear denominator, prewait at its row limit", + spec=dict( + seq_lens=(5000, 12345), + decode_query_len=4, + linear_softmax=True, + mxfp8=True, + prewait=True, + ), + ), + Cell( + why="7-draft verify at one request: T=8, log-domain recurrence, prewait at its row limit", + spec=dict( + seq_lens=(20000,), + decode_query_len=8, + linear_softmax=False, + mxfp8=True, + prewait=True, + ), + ), + Cell( + why="7-draft verify at two requests: T=16 on the linear denominator, past the prewait row limit", + spec=dict( + seq_lens=(5000, 12345), + decode_query_len=8, + linear_softmax=True, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="a 1M-token request: a block-table row longer than the staged 8192 entries, read from global", + spec=dict( + seq_lens=(1048576,), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=True, + ), + ), + Cell( + why="Eagle3 verify at batch 8 (T=32): short requests with -1 tails and zero-length padding rows", + spec=dict( + seq_lens=(16384, 2049, 130, 0, 7777, 4, 30000, 0), + decode_query_len=4, + linear_softmax=True, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="decode without speculation at batch 8: T=8 rows of distinct requests, staged after the wait", + spec=dict( + seq_lens=(3000, 2100, 9999, 2048, 2049, 4500, 16000, 2200), + decode_query_len=1, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="no hand-off buffers (the variant the call site takes past 128 rows), driven at T=8", + spec=dict( + seq_lens=(5000, 12345), + decode_query_len=4, + linear_softmax=True, + mxfp8=False, + prewait=False, + ), + ), + ) + note = ( + "\n Certified: `linear_softmax` x hand-off on / off x `prewait`, the " + "variants\n the cells drive. The hand-off cells are certified " + "bit-exactly against\n `mxfp8_handoff_reference` applied to the op's " + "own BF16 rows; the BF16 rows\n themselves within the band the test " + "derives, and the two `prewait` variants\n against each other bit for" + " bit.\n\n `topk` is read for the first `min(16, ceil(kv_len / 128))` " + "entries of each\n token; a -1 among them skips that CTA's block, " + "entries past them are never\n read. `use_pdl` only changes the " + "launch attribute. The hand-off assumes\n `num_sf_k_tiles = 8` (a " + "1024-wide o_proj input) and buffers with at least\n as many rows as " + "the step; `MERGE_MXFP8` sizes them at 128 rows.\n " + ) + + def __call__( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + prewait: bool, + ) -> torch.Tensor: + one_div = _lc8_onediv(int(q.shape[0])) + _lc8_launcher()( + q, + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + out, + q8, + sf, + num_sf_k_tiles, + decode_query_len, + sm_scale, + linear_softmax, + use_pdl, + prewait, + **{"one_div": True} if one_div else {}, + ) + return out + + def reference( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + prewait: bool, + ) -> torch.Tensor: + """The BF16 ``out`` [T, 1024] the op is supposed to write: ``sparse_attention_reference`` in float64, + rounded once. ``linear_softmax``, ``use_pdl`` and ``prewait`` do not change what is computed; the + hand-off buffers are checked separately with ``mxfp8_handoff_reference``. + """ + attn = sparse_attention_reference( + q.view(-1, NUM_HEADS, HEAD_DIM), + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + decode_query_len, + sm_scale, + ) + return attn.reshape(out.shape).to(out.dtype) + + def is_valid( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + prewait: bool, + ) -> None: + """The inputs this op takes and answers wrongly (``m3_attn_lc8.lc8_applies`` is the target's form of the + layout half). + + The kernel addresses a page's token rows at a fixed 64-byte (data) and + 8-byte (scale) pitch and loads them 16 / 8 bytes at a time, so only the + page stride is read from the views. It reads the selected block ids, + the block table and the hand-off buffers unchecked. Reads ``topk`` and + ``seq_lens`` back to the host: a test-only cost. + """ + for name, t in (("k", k), ("v", v)): + assert t.dim() == 4 and t.shape[1] == 1 and (t.shape[2:] == (PAGE, HEAD_DIM // 2)), ( + f"{name} must be the [pages, 1, 128, 64] byte view of one KV head" + ) + assert t.stride(3) == 1 and t.stride(2) == HEAD_DIM // 2, ( + f"{name} rows must be dense 64-byte rows: the kernel ignores the row stride" + ) + assert t.data_ptr() % 16 == 0 and t.stride(0) % 16 == 0, ( + f"{name} pages must be 16-byte aligned" + ) + for name, t in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + assert t.dim() == 4 and t.stride(3) == 1 and (t.stride(2) == HEAD_DIM // SF_VEC), ( + f"{name} rows must be dense 8-byte rows: the kernel ignores the row stride" + ) + assert t.data_ptr() % 8 == 0 and t.stride(0) % 8 == 0, ( + f"{name} pages must be 8-byte aligned" + ) + assert q.data_ptr() % 16 == 0 and q.stride(0) % 16 == 0 and (q.stride(1) % 16 == 0), ( + "q rows are loaded 16 bytes at a time and must be 16-byte aligned" + ) + assert k_global_scale.numel() == 1 and v_global_scale.numel() == 1, ( + "global scales are single values" + ) + total_q = q.shape[0] + assert ( + total_q % decode_query_len == 0 and seq_lens.numel() >= total_q // decode_query_len + ), ( + "every query token must belong to a request: seq_lens is read at tok // decode_query_len unchecked" + ) + seq = seq_lens.tolist() + num_cols = block_table.shape[1] + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk[tok, : min(TOPK, nblocks)] + assert bool(((ids >= -1) & (ids < min(nblocks, num_cols))).all()), ( + "token " + f"{tok}" + ": selected block ids must be -1 or below the request's block cou" + "nt; the block table is indexed with them unchecked" + ) + if q8 is not None: + assert num_sf_k_tiles == WIDTH // 128, ( + "num_sf_k_tiles must be " + f"{WIDTH // 128}" + ": the hand-off rows are " + f"{WIDTH}" + " wide and a smaller value overlaps them silently" + ) + assert ( + q8.numel() >= total_q * WIDTH + and sf is not None + and (sf.numel() >= (total_q + 127) // 128 * 128 * (WIDTH // MX_BLOCK)) + ), "the hand-off buffers must hold every row of the step: they are written unchecked" + + +m3_sparse_decode_lc8 = M3SparseDecodeLc8() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/msa_kv_update.py new file mode 100644 index 000000000000..ad05b934d4a9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/attention/msa_kv_update.py @@ -0,0 +1,200 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MSA KV-length correction in one launch (this target's ``MsaKvUpdate`` hook).""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The MSA pool's page: one sparse block is one 128-token KV page. +TOKENS_PER_BLOCK = 128 + + +class MsaKvUpdate(OpWrapper): + """Re-derive the MSA decode state after the overlap scheduler moved ``kv_lens``. + + ``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` calls this with its + own buffers at the head of every captured speculative decode step. One + single-CTA launch writes, for ``batch`` requests and ``total_q`` query + tokens (all int32): + + * ``msa_seq_lens[b] = min(kv_lens[b], kv_lens_staged[b])`` and, when given, + ``cu_kv_lens`` as its exclusive prefix sum (``batch + 1`` entries); + * ``out_cache_loc[t]``, the KV write slot of the position token ``t`` + attends up to, through ``block_table``; + * ``n_valid_blocks[t]``, when given, the block count top-k selection is + bounded by. + + It returns None; the cells read the four buffers. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode without speculation at one request: qo = 1, every buffer written", + spec=dict(batch=1, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="decode without speculation at the captured batch 8", + spec=dict(batch=8, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="Eagle3 verify (draft 3): qo = 4 per request, partial acceptance shortens kv_lens below staged", + spec=dict(batch=8, query_len=4, accept="partial", cu_kv=True, n_valid=True), + ), + Cell( + why="the metadata without cu_kv_lens (no extend_ctx promotion): the optional prefix sum is skipped", + spec=dict(batch=4, query_len=4, accept="partial", cu_kv=False, n_valid=True), + ), + Cell( + why="CUDA-graph padding rows (kv_lens 0, pos -1) and no n_valid_blocks buffer", + spec=dict(batch=8, query_len=4, accept="padded", cu_kv=True, n_valid=False), + ), + Cell( + why="positions crossing a page boundary: the slot moves to the previous page of the request", + spec=dict(batch=3, query_len=4, accept="page_edge", cu_kv=True, n_valid=True), + ), + ) + + note = """ + The kernel covers any batch up to the block size's loop (it strides over + 256 threads), and any page size; only `tokens_per_block = 128`, the MSA + pool's page, is certified. + + The op does not bounds-check the request index a token reads + (`q_batch_row`) or the block-table column it derives: both come from the + metadata's own staging, which keeps them in range by construction, and an + out-of-range value reads whatever memory sits there. `is_valid` states both. + """ + + def __call__( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + torch.ops.trtllm.minimax_m3_msa_kv_update( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + + def reference( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> Tuple[torch.Tensor, Optional[torch.Tensor], torch.Tensor, Optional[torch.Tensor]]: + """``(msa_seq_lens[:batch], cu_kv_lens[:batch + 1], out_cache_loc[:total_q], + n_valid_blocks[:total_q])`` as the op is supposed to write them. + + The in-tree metadata's own torch re-derivation + (``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` past its hook + call), statement for statement, returning new tensors rather than + writing the buffers. The buffer arguments only say which outputs exist: + a ``None`` buffer gives a ``None`` result. + """ + page = int(tokens_per_block) + kv_true = torch.minimum(kv_lens[:batch], kv_lens_staged[:batch]) + cu_kv = None + if cu_kv_lens is not None: + cu_kv = torch.zeros(batch + 1, dtype=torch.int32, device=kv_lens.device) + cu_kv[1:] = torch.cumsum(kv_true, 0) + qbr = q_batch_row[:total_q].to(torch.long) + qo_dev = seq_lens[:batch] + pos = kv_true[qbr] - qo_dev[qbr] + q_intra[:total_q] + pos_slot = pos.clamp_min(0) + block_col = torch.div(pos_slot, page, rounding_mode="floor").clamp( + max=int(block_table.shape[1]) - 1 + ) + slots = block_table[qbr, block_col.to(torch.long)] * page + torch.remainder(pos_slot, page) + n_valid = None + if n_valid_blocks is not None: + n_valid = torch.div( + (pos + 1).clamp_min(0) + (page - 1), page, rounding_mode="floor" + ).to(torch.int32) + return kv_true.to(torch.int32), cu_kv, slots.to(torch.int32), n_valid + + def is_valid( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + """The inputs this op takes and answers wrongly. + + Every operand but the block table is read as a dense run from its first + element, so a strided view reads the wrong elements. The request index of + each token and the table row it selects are not bounds-checked. Reads + ``q_batch_row`` back to the host: a test-only cost. + """ + for name, t in ( + ("kv_lens", kv_lens), + ("kv_lens_staged", kv_lens_staged), + ("seq_lens", seq_lens), + ("q_batch_row", q_batch_row), + ("q_intra", q_intra), + ("msa_seq_lens", msa_seq_lens), + ("out_cache_loc", out_cache_loc), + ("cu_kv_lens", cu_kv_lens), + ("n_valid_blocks", n_valid_blocks), + ): + assert t is None or t.is_contiguous(), ( + f"{name} must be contiguous: the kernel indexes it densely from data_ptr and ignores strides" + ) + rows = q_batch_row[:total_q] + assert total_q == 0 or ( + int(rows.min()) >= 0 and int(rows.max()) < min(batch, block_table.shape[0]) + ), ( + "q_batch_row must index a request below batch (and a block-table row): the op reads the " + "request's lengths and table row unchecked" + ) + + +msa_kv_update = MsaKvUpdate() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/__init__.py new file mode 100644 index 000000000000..5da86fe24bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collective-communication entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_allreduce.py new file mode 100644 index 000000000000..7e03223a5dba --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_allreduce.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The built-in TP all-reduce, fused with the residual add and the next RMSNorm.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.distributed import AllReduce, AllReduceParams + + +class M3AllReduce(OpWrapper): + """``comm(x, all_reduce_params=params)``: the runtime's ``AllReduce`` over the TP group. ``params`` None: the + plain sum over the ranks (the drafter's o_proj and down projection). With ``RESIDUAL_RMS_NORM`` params: + ``updated = residual + sum_ranks(x)``, ``normed = rmsnorm(updated) * norm_weight``; + returns (normed, updated). The strategy (``AllReduceStrategy.AUTO``: one-shot / two-shot Lamport, NCCL) is the + communicator's per call. + + The layer boundaries the multicast all-reduce does not take run it: prefill and mixed steps, and generation steps + beyond the multicast kernel's rows. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # No cell names a world size: the result does not depend on it, and the certifying matrix runs at every world + # size one node can form (the target's two-tray group of 8 cannot be formed on one node). + CELLS: tuple[Cell, ...] = ( + Cell(why="a prefill chunk", spec=dict(m=2048, hidden=6144)), + Cell(why="a 64-row generation step", spec=dict(m=64, hidden=6144)), + Cell(why="plain sum of eight drafter rows", spec=dict(m=8, hidden=6144, plain=True)), + ) + + note = """ + The reduction order across ranks is the strategy's; `reference` sums the ranks' rows in fp32 in rank order, then + adds the residual and normalizes (one bf16 rounding of the sum and one of the norm). A strategy may round the sum + to bf16 before the residual add, so the residual sum agrees within one bf16 ulp of its operands' magnitude. + """ + + def __call__(self, x: torch.Tensor, comm: AllReduce, params: Optional[AllReduceParams]): + return comm(x, all_reduce_params=params) + + def reference(self, xs, comm: AllReduce, params: Optional[AllReduceParams]): + """``xs``: every rank's ``x`` in rank order (a collective's reference takes the group's inputs).""" + total = torch.stack([x.float() for x in xs]).sum(0) + if params is None: + return total.to(xs[0].dtype) + updated = (params.residual.float() + total).to(xs[0].dtype) + h = updated.float() + normed = ( + h + * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + params.eps) + * params.norm_weight.float() + ) + return normed.to(xs[0].dtype), updated + + +m3_allreduce = M3AllReduce() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_mc_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_mc_allreduce_norm.py new file mode 100644 index 000000000000..f7054de397dc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_mc_allreduce_norm.py @@ -0,0 +1,277 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode layer boundary, 8 ranks.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +_E4M3_MAX = 448.0 + + +def mxfp8_rows(normed: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """MXFP8 of BF16 rows as ``trtllm::mxfp8_quantize(normed, True)``: E4M3 [M, H] and 128x4-swizzled UE8M0 bytes. + + One scale per 32 columns, the smallest power of two at or above ``amax / 448`` (biased exponent, round up), + data = row / scale rounded to nearest E4M3. The scale buffer is padded to 128 rows, padding bytes left as the + op leaves them (the comparison masks them out). + """ + m, h = normed.shape + blocks = normed.float().view(m, h // 32, 32) + amax = blocks.abs().amax(-1) + bits = (amax / _E4M3_MAX).view(torch.int32) + byte = (((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) > 0).to(torch.int32)).clamp(0, 254) + scale = torch.exp2(byte.float() - 127.0).unsqueeze(-1) + data = torch.where(amax.unsqueeze(-1) != 0, blocks / scale, torch.zeros_like(blocks)) + data = data.view(m, h).to(torch.float8_e4m3fn) + return data, swizzle_sf(byte.to(torch.uint8)) + + +def swizzle_sf(byte: torch.Tensor) -> torch.Tensor: + """Scatter [rows, H / 32] scale bytes into the 128x4-swizzled buffer the op writes (zero padding).""" + rows, cols = byte.shape + padded_rows = (rows + 127) // 128 * 128 + r = torch.arange(rows, device=byte.device).view(-1, 1) + c = torch.arange(cols, device=byte.device).view(1, -1) + index = ( + (c % 4) + (c // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + (r // 128) * (128 * cols) + ) + out = torch.zeros(padded_rows * cols, dtype=torch.uint8, device=byte.device) + out[index.reshape(-1)] = byte.reshape(-1) + return out + + +def sf_written(rows: int, cols: int, device) -> torch.Tensor: + """Mask of the swizzled scale bytes the op writes for ``rows`` rows (the padding rows are left unwritten).""" + return swizzle_sf(torch.ones(rows, cols, dtype=torch.uint8, device=device)).bool() + + +def boundary( + partials: Sequence[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, +) -> tuple[torch.Tensor, ...]: + """The boundary arithmetic once each rank's BF16 partial is known (shared with the finalize entry). + + Ranks are summed in FP32 in rank order, the residual added and rounded to BF16 once; the RMS statistic and the + normalization are done in float64 and rounded to BF16 once. + """ + total = torch.zeros(partials[0].shape, dtype=torch.float32, device=residual.device) + for p in partials: + total = total + p.float() + updated = (total + residual.float()).to(torch.bfloat16) + u = updated.double() + inv = torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + eps) + normed = (u * inv * norm_w.double()).to(torch.bfloat16) + if not quant: + return normed, updated + return (normed, updated, *mxfp8_rows(normed)) + + +class M3McAllreduceNorm(OpWrapper): + """One TP group's boundary: ``updated = bf16(residual + sum_r bf16(x_r + x2_r))``, ``normed = RMSNorm(updated)``. + + Every rank pushes its partial through the multicast mapping ``ws_mc_ptr`` into every rank's exchange, then polls + its own view ``ws_uc`` (re-arming the words it read) and rotates the per-CTA call parities in ``flags``; the + workspace is the ``McAllReduceWorkspace`` bucket of ``maxt`` tokens. All ranks must make the same calls in the + same stream order. ``prepushed`` 2 means a producer already pushed ``x`` (and launched this grid after its + pushes); 3 that two producers pushed ``x`` into exchange set A and ``x2`` into set B. Returns + ``[normed, updated]`` or, with ``quant``, also the MXFP8 copy of ``normed`` (E4M3 rows, swizzled UE8M0). + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The tp8 decode boundaries (hidden 6144, 8 ranks over two trays, eps 1e-6). Buckets 8 / 16 / 48 are separate + # workspaces; the 8-token bucket runs epilogue option 7 at 1, 2, 7, 8 tokens and 15 at 3..6, the others 15. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one decode token at the attention boundary, x only, bucket 8 option 7", + spec=dict(m=1, maxt=8, opts=7, x2=False, quant=False, prepushed=0), + ), + Cell( + why="MoE boundary at 4 tokens: shared + routed summed in-kernel, MXFP8 copy for the next QKV, option 15", + spec=dict(m=4, maxt=8, opts=15, x2=True, quant=True, prepushed=0), + ), + Cell( + why="a full 8-token bucket with the second addend", + spec=dict(m=8, maxt=8, opts=7, x2=True, quant=False, prepushed=0), + ), + Cell( + why="attention boundary whose rows the o_proj GEMV already pushed (prepushed 2)", + spec=dict(m=8, maxt=8, opts=7, x2=False, quant=False, prepushed=2), + ), + Cell( + why="routed engine pushed set A, shared down projection set B (prepushed 3), with the MXFP8 copy", + spec=dict(m=6, maxt=8, opts=15, x2=True, quant=True, prepushed=3), + ), + Cell( + why="12 tokens pushed by the o_proj GEMV into the 16-token bucket", + spec=dict(m=12, maxt=16, opts=15, x2=False, quant=False, prepushed=2), + ), + Cell( + why="the 9..13-token MoE kernel's sum into the 16-token bucket, MXFP8 copy", + spec=dict(m=16, maxt=16, opts=15, x2=False, quant=True, prepushed=0), + ), + Cell( + why="the 48-token bucket at a partial fill", + spec=dict(m=30, maxt=48, opts=15, x2=False, quant=False, prepushed=0), + ), + Cell( + why="the 48-token bucket full, second addend and MXFP8 copy", + spec=dict(m=48, maxt=48, opts=15, x2=True, quant=True, prepushed=0), + ), + # Every other row count of the decode boundaries (dispatch.MC_ALLREDUCE): the bucket and epilogue options the + # target picks for it (ops/m3_mc_allreduce), the second addend and the MXFP8 copy alternating. + *( + Cell( + why=f"{m} tokens: a boundary row count between the ones above", + spec=dict( + m=m, + maxt=8 if m <= 8 else 16 if m <= 16 else 48, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1] if m <= 8 else 15, + x2=m % 2 == 0, + quant=m % 3 == 0, + prepushed=0, + ), + ) + for m in range(1, 49) + if m not in (1, 4, 8, 16, 30, 48) + ), + # Every other row count pushed by the o_proj (1..16) or the 1..8-token MoE engine (with the MXFP8 copy): + # dispatch.ATTN_PUSH / dispatch.MOE_PUSH. + *( + Cell( + why=f"{m} tokens prepushed: a pushed row count between the ones above", + spec=dict( + m=m, + maxt=8 if m <= 8 else 16, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1] if m <= 8 else 15, + x2=False, + quant=m <= 8, + prepushed=2, + ), + ) + for m in range(1, 17) + if m not in (8, 12) + ), + ) + + note = """ + The epilogue option bits (1 early residual / weight load, 2 st.async cluster exchange, 4 early parity rotation, + 8 max carveout) are bit-identical by construction; only the four (bucket, option) pairs the target selects are + built. `prepushed` 1 (pushed, grid wait kept) is accepted by the op but no caller passes it. `pdl` changes only + launch overlap and is driven off in the cells. + + `normed` is gated to one BF16 rounding step: the kernel's RMS statistic is a warp / cluster tree of FP32 sums + and its `rsqrtf` is approximate, so a row whose scaled value lands next to a BF16 midpoint can round either + way. `updated` and the MXFP8 copy of the op's own `normed` are bit-exact. + """ + + def __call__( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_mc_allreduce_norm( + x, + x2, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + None, + None, + maxt, + quant, + prepushed, + opts, + ) + + def reference( + self, + per_rank_x: Sequence[torch.Tensor], + per_rank_x2: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute. + + The documented collective exception: it takes every rank's ``x`` (and ``x2``) in rank order, not the calling + rank's. Each rank's partial is ``bf16(x + x2)`` (``x`` alone without ``x2``), whatever pushed it. + """ + partials = [] + for r, x in enumerate(per_rank_x): + if per_rank_x2 is None: + partials.append(x) + else: + partials.append((x.float() + per_rank_x2[r].float()).to(torch.bfloat16)) + return boundary(partials, residual, norm_w, eps, quant) + + def is_valid( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + ) -> None: + h = residual.shape[-1] + # The op checks norm_w's element count, not its layout: a strided view is read as packed memory. + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + # The workspace is addressed as [set][parity][maxt][world][H / 2] words and never bounds-checked: a smaller + # buffer (another bucket's, or one built for fewer ranks) is polled and re-armed out of range. + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + # Flags are per (token, cluster CTA) of the bucket; the op checks the count, not that they belong to the + # workspace. Flags of another bucket hold another call history, and the parity they select is not the one the + # peers push into: the poll then waits on the other half forever or reads last call's rows. + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + + +m3_mc_allreduce_norm = M3McAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_mc_finalize_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_mc_finalize_allreduce_norm.py new file mode 100644 index 000000000000..64feae8122d5 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/comm/m3_mc_finalize_allreduce_norm.py @@ -0,0 +1,197 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MoE finalize + multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode MoE boundary, 8 ranks.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_mc_allreduce_norm import boundary + + +def finalize_partial( + fc2: torch.Tensor, perm: torch.Tensor, scales: torch.Tensor, shared: Optional[torch.Tensor] +) -> torch.Tensor: + """One rank's BF16 partial: ``bf16(bf16(sum_k scale_k * fc2[perm_k]) + shared)``, the routed sum in FP32 in k + order and rounded to BF16, then the shared expert added as stock adds it (one more BF16 rounding). + + The kernel's per-k step is a fused multiply-add; it is reproduced exactly by the float64 product (exact for two + FP32 operands) plus the FP32 accumulator, rounded back to FP32. Dropped entries (perm -1) are skipped. + """ + m, topk = perm.shape + acc = torch.zeros(m, fc2.shape[1], dtype=torch.float32, device=fc2.device) + sc = scales.reshape(m, topk).float() + for k in range(topk): + rows = perm[:, k].long() + keep = (rows >= 0).unsqueeze(-1) + gathered = fc2[rows.clamp_min(0)].double() + step = (gathered * sc[:, k : k + 1].double() + acc.double()).float() + acc = torch.where(keep, step, acc) + if shared is not None: + acc = acc.to(torch.bfloat16).float() + shared.float() + return acc.to(torch.bfloat16) + + +class M3McFinalizeAllreduceNorm(OpWrapper): + """The MoE boundary with the routed experts' finalize folded in. + + Per rank the partial is ``bf16(bf16(sum_k scales[t, k] * fc2[perm[t, k]]) + shared[t])`` (``perm`` -1 = dropped, + ``scales`` fp32 or bf16), then the group's all-reduce, residual add and RMSNorm exactly as + ``m3_mc_allreduce_norm``: same workspace, flags and call-order contract. Returns ``[normed, updated]`` or, with + ``quant``, also the MXFP8 copy of ``normed``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The tp8 target takes this path from 9 tokens (top-4 of 128 experts, fp32 routing weights, the shared expert + # always present): the 16- and 48-token buckets, epilogue option 15. + CELLS: tuple[Cell, ...] = ( + Cell( + why="the smallest finalize call (9 tokens, 16-token bucket) with the next QKV's MXFP8 copy", + spec=dict(m=9, maxt=16, opts=15, topk=4, quant=True, dropped=0), + ), + Cell( + why="a full 16-token bucket with dropped expert slots", + spec=dict(m=16, maxt=16, opts=15, topk=4, quant=False, dropped=5), + ), + Cell( + why="the 48-token bucket at a partial fill", + spec=dict(m=24, maxt=48, opts=15, topk=4, quant=False, dropped=0), + ), + Cell( + why="the 48-token bucket full, MXFP8 copy", + spec=dict(m=48, maxt=48, opts=15, topk=4, quant=True, dropped=3), + ), + # Every other row count of the unfinalized MoE (dispatch.MC_FINALIZE): the bucket and options the target picks. + *( + Cell( + why=f"{m} tokens: a finalize row count between the ones above", + spec=dict( + m=m, + maxt=16 if m <= 16 else 48, + opts=15, + topk=4, + quant=m % 2 == 0, + dropped=m % 4, + ), + ) + for m in range(9, 49) + if m not in (9, 16, 24, 48) + ), + ) + + note = """ + BF16 routing weights and the 8-token bucket are accepted by the op (option 7 there) but the target only enters + this path at 9 tokens and up, with the built-in MoE's fp32 weights; neither is a cell. The rest is as in + `m3_mc_allreduce_norm`'s note. + """ + + def __call__( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_mc_allreduce_norm( + fc2, + shared, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + perm, + scales, + maxt, + quant, + 0, + opts, + ) + + def reference( + self, + per_rank_fc2: Sequence[torch.Tensor], + per_rank_perm: Sequence[torch.Tensor], + per_rank_scales: Sequence[torch.Tensor], + per_rank_shared: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute, from every rank's MoE outputs in rank order (the + collective exception, as in ``m3_mc_allreduce_norm.reference``).""" + partials = [ + finalize_partial( + fc2, + per_rank_perm[r], + per_rank_scales[r], + None if per_rank_shared is None else per_rank_shared[r], + ) + for r, fc2 in enumerate(per_rank_fc2) + ] + return boundary(partials, residual, norm_w, eps, quant) + + def is_valid( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + ) -> None: + h = residual.shape[-1] + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + # The kernel gathers fc2[perm] without a bound: an index past the FC2 rows reads whatever follows them. + lo, hi = int(perm.min()), int(perm.max()) + assert lo >= -1 and hi < fc2.shape[0], ( + f"perm spans [{lo}, {hi}]; entries must be -1 (dropped) or a row of fc2 (< {fc2.shape[0]})" + ) + + +m3_mc_finalize_allreduce_norm = M3McFinalizeAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding/__init__.py new file mode 100644 index 000000000000..e9f4d8e20b94 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Embedding ops of this target: the generic token-embedding gather.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding/m3_embedding.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding/m3_embedding.py new file mode 100644 index 000000000000..f979fdecf376 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding/m3_embedding.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The generic token-embedding gather over the replicated table (an id outside the table gives a zero row).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modules.embedding import pre_comm_embedding_ops + + +class M3Embedding(OpWrapper): + """``table[ids]`` with a zero row for an id outside ``[0, V)`` (the speculative sampler pads with -1). + + The table is the full (replicated) bf16 embedding [V, 6144]; ids int32 / int64 of any count. The fused decode + embedding (``m3_embedding_firstnorm``) takes decode steps of up to 8 rows; this entry takes every other step. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="a prefill chunk, int32 ids", spec=dict(rows=2048, id_dtype=torch.int32, pad=False) + ), + Cell( + why="a 48-row decode verify, int64 ids", + spec=dict(rows=48, id_dtype=torch.int64, pad=False), + ), + Cell( + why="draft rows padded with -1 ids", spec=dict(rows=32, id_dtype=torch.int32, pad=True) + ), + ) + + note = """ + An exact gather: `reference` must match bit for bit. + """ + + def __call__(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + return pre_comm_embedding_ops(ids, table, 1, 0, None, 0, table.shape[0], False, 0) + + def reference(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] + return rows * valid.unsqueeze(-1).to(table.dtype) + + +m3_embedding = M3Embedding() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/__init__.py new file mode 100644 index 000000000000..4c6fc92cd49c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 decode embedding gather + first-layer RMSNorm as one CuTe DSL kernel (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/m3_embedding_firstnorm_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/m3_embedding_firstnorm_kernel.py new file mode 100644 index 000000000000..78563b4db40d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/m3_embedding_firstnorm_kernel.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fixed M3 H6144 gather/raw residual/RMS, reusing pinned K3 norm kernel. + +128 threads x 8 BF16 x 6 iterations covers the row. Unlike the original +K3 wrapper this target does not promise FlashInfer's bit-identical reduction. +""" + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass.experimental import primitives as prims + +NORM_THREADS = 128 +NORM_VEC = 8 +NORM_WARPS = 4 # capacity: fixed: the CTA's warps + + +@cute.kernel +def m3_embed_norm_kernel( + ids: cute.Tensor, # int32 or int64 [N] + table: cute.Tensor, # bf16 [V, H] + weight: cute.Tensor, # bf16 [H] + raw: cute.Tensor, # bf16 [N, H], out: the rows + out: cute.Tensor, # bf16 [N, H], out: the normed rows + vocab: cutlass.Int64, + eps: cutlass.Float32, + tv_layout: cute.Layout, + tiler_mn: cute.Shape, + hidden: cutlass.Constexpr[int], +): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + prims.griddepcontrol(prims.GridDepAction.WAIT) + + smem = cutlass.utils.SmemAllocator() + reduction_buffer = smem.allocate_tensor( + cutlass.Float32, cute.make_layout((1, NORM_WARPS)), byte_alignment=4 + ) + + tok = cutlass.Int64(ids[bidx]) + valid = (tok >= cutlass.Int64(0)) & (tok < vocab) + row = cutlass.Int64(cutlass.select_(valid, tok, cutlass.Int64(0))) + + gX = cute.local_tile(table, tiler_mn, (row, 0)) + gR = cute.local_tile(raw, tiler_mn, (bidx, 0)) + gY = cute.local_tile(out, tiler_mn, (bidx, 0)) + w_layout = cute.prepend(weight.layout, cute.make_layout((tiler_mn[0],), stride=(0,))) + gW = cute.local_tile(cute.make_tensor(weight.iterator, w_layout), tiler_mn, (0, 0)) + + copy_atom_load = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), table.element_type, num_bits_per_copy=128 + ) + copy_atom_store = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), out.element_type, num_bits_per_copy=128 + ) + thr_copy_X = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_W = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_O = cute.make_tiled_copy(copy_atom_store, tv_layout, tiler_mn).get_slice(tidx) + + tXgX = thr_copy_X.partition_S(gX) + tXrX = cute.make_fragment_like(tXgX) + tWgW = thr_copy_W.partition_S(gW) + tWrW = cute.make_fragment_like(tWgW) + tXrW = thr_copy_X.retile(tWrW) + tXgR = thr_copy_O.partition_D(gR) + tXgO = thr_copy_O.partition_D(gY) + tXrO = cute.make_fragment_like(tXgO) + + # The row (zero for an id outside [0, V)), straight to the snapshot slot; the weight. + tXrX.store(cute.zeros_like(tXrX, dtype=table.element_type)) + if valid: + cute.copy(copy_atom_load, tXgX, tXrX) + cute.copy(copy_atom_load, tWgW, tWrW) + cute.copy(copy_atom_store, tXrX, tXgR) + + # flashinfer's RMSNormKernel arithmetic, in its order (see the header). + x = tXrX.load().to(cutlass.Float32) + x_sq = x * x + sum_sq = x_sq.reduce(cute.ReductionOp.ADD, init_val=cutlass.Float32(0.0), reduction_profile=0) + for i in cutlass.range_constexpr(5): + sum_sq = sum_sq + cute.arch.shuffle_sync_bfly(sum_sq, offset=1 << i) + lane = cute.arch.lane_idx() + warp = cute.arch.warp_idx() + if lane == 0: + reduction_buffer[0, warp] = sum_sq + cute.arch.barrier() + total = cutlass.Float32(0.0) + if lane < NORM_WARPS: + total = reduction_buffer[0, lane] + for i in cutlass.range_constexpr(5): + total = total + cute.arch.shuffle_sync_bfly(total, offset=1 << i) + mean_sq = total / cutlass.Float32(hidden) + rstd = cute.math.rsqrt(mean_sq + eps, fastmath=True) + w = tXrW.load().to(cutlass.Float32) + y = x * rstd * (w + cutlass.Float32(0.0)) + tXrO.store(y.to(out.element_type)) + cute.copy(copy_atom_store, tXrO, tXgO) + + +@cute.jit +def m3_embed_norm( + ids: cute.Tensor, + table: cute.Tensor, + weight: cute.Tensor, + raw: cute.Tensor, + out: cute.Tensor, + vocab: cutlass.Int64, + eps: cutlass.Float32, + n_tokens: cutlass.Constexpr[int], + hidden: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + # flashinfer's thread-value layout for one row: thread t, value (v, k) -> column 8 t + v + 1024 k. + blocks = hidden // (NORM_VEC * NORM_THREADS) + tv_layout = cute.make_layout(((NORM_THREADS, 1), (NORM_VEC, blocks)), + stride=((NORM_VEC, 1), (1, NORM_VEC * NORM_THREADS))) # fmt: skip + m3_embed_norm_kernel( + ids, table, weight, raw, out, vocab, eps, tv_layout, (1, hidden), hidden + ).launch( + grid=[n_tokens, 1, 1], + block=[NORM_THREADS, 1, 1], + smem=NORM_WARPS * 4, + stream=stream, + use_pdl=use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/op.py new file mode 100644 index 000000000000..276f8c32023b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/embedding_firstnorm/op.py @@ -0,0 +1,142 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Host side of the MiniMax-M3 decode embedding + first-layer RMSNorm (m3_embedding_firstnorm_kernel.py): H 6144, +1..8 rows, into caller-owned buffers. + +Compiled on the first call per (rows, id dtype, table shape, PDL), which must happen outside CUDA-graph capture +(:class:`DecodeEmbedding` compiles at construction). +""" + +import threading +from typing import Dict, Tuple + +import cuda.bindings.driver as cuda_driver +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import from_dlpack + +from .m3_embedding_firstnorm_kernel import m3_embed_norm as _m3_embed_norm_jit + +HIDDEN = 6144 # capacity: fixed: the checkpoint's hidden size +MAX_ROWS = 8 # capacity: fixed: the kernel's built rows (decode steps) + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + + +def arg(tensor): + return from_dlpack( + tensor.detach(), + assumed_align=tensor.element_size() + if tensor.ndim == 1 and tensor.dtype in (torch.int32, torch.int64) + else 16, + ) + + +def _check(ids, table, weight, raw, out) -> None: + rows = ids.shape[0] if ids.dim() == 1 else 0 + if not ( + 1 <= rows <= MAX_ROWS + and ids.dtype in (torch.int32, torch.int64) + and ids.is_contiguous() + and table.dtype == torch.bfloat16 + and table.dim() == 2 + and table.shape[1] == HIDDEN + and table.is_contiguous() + and weight.dtype == torch.bfloat16 + and tuple(weight.shape) == (HIDDEN,) + and all( + t.dtype == torch.bfloat16 and tuple(t.shape) == (rows, HIDDEN) and t.is_contiguous() + for t in (raw, out) + ) + ): + raise ValueError( + "m3_embed_norm: expects ids int32/int64 [1..8], a contiguous bf16 table [V, 6144], bf16 weight [6144] " + "and contiguous bf16 raw / out [rows, 6144]" + ) + + +def _compile(ids, table, weight, eps, raw, out, use_pdl: bool): + key = (ids.shape[0], ids.dtype, tuple(table.shape), table.device, bool(use_pdl)) + fn = _compiled.get(key) + if fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_embed_norm compiles on its first call per shape: call it once before capture" + ) + with _lock: + fn = _compiled.get(key) + if fn is None: + stream = cuda_driver.CUstream(torch.cuda.current_stream(table.device).cuda_stream) + fn = _compiled[key] = cute.compile( + _m3_embed_norm_jit, + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + ids.shape[0], + HIDDEN, + bool(use_pdl), + stream, + ) + return fn + + +def m3_embed_norm( + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """Write ``table[ids]`` into ``raw`` (a zero row for an id outside ``[0, V)``) and its RMSNorm with ``weight`` / + ``eps`` into ``out``; returns ``(out, raw)``.""" + _check(ids, table, weight, raw, out) + fn = _compile(ids, table, weight, eps, raw, out, use_pdl) + stream = cuda_driver.CUstream(torch.cuda.current_stream(ids.device).cuda_stream) + fn( + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + stream, + ) + return out, raw + + +class DecodeEmbedding: + """Caller-owned buffers of one (rows, id dtype) decode embedding: ``raw`` (the gathered rows, the first residual) + and ``output`` (their RMSNorm), over ``table`` and the first layer's plain (non-Gemma) input norm ``weight`` / + ``eps``. Built (and compiled) before CUDA-graph capture.""" + + def __init__(self, table, weight, eps, rows, id_dtype, use_pdl=True): + assert not torch.cuda.is_current_stream_capturing() + assert table.dtype == torch.bfloat16 and table.is_contiguous() and table.shape[1] == HIDDEN + assert weight.dtype == torch.bfloat16 and weight.shape == (HIDDEN,) + assert 1 <= rows <= MAX_ROWS and id_dtype in (torch.int32, torch.int64) + self.table, self.weight, self.rows, self.id_dtype, self.use_pdl = ( + table, + weight, + rows, + id_dtype, + use_pdl, + ) + self.eps = float(eps) + self.raw = torch.empty(rows, HIDDEN, device=table.device, dtype=torch.bfloat16) + self.output = torch.empty_like(self.raw) + ids = torch.empty(rows, device=table.device, dtype=id_dtype) + _compile(ids, table, weight, self.eps, self.raw, self.output, use_pdl) + + def __call__(self, ids): + assert ids.shape == (self.rows,) and ids.dtype == self.id_dtype and ids.is_contiguous() + return m3_embed_norm( + ids, self.table, self.weight, self.eps, self.raw, self.output, self.use_pdl + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/__init__.py new file mode 100644 index 000000000000..9cc374bdf1cc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GEMM entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_bf16_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_bf16_linear.py new file mode 100644 index 000000000000..5007e1da7386 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_bf16_linear.py @@ -0,0 +1,41 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The BF16 Eagle3 drafter's projections: ``x @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3Bf16Linear(OpWrapper): + """``F.linear(x, w)`` for bf16 ``x`` [M, K] and a bf16 weight [N, K] (cuBLAS, fp32 accumulation), bf16 out. + + The drafter's fc over the captured target layers (K = 3 x 6144), its fused QKV (K = 2 x 6144, this rank's heads), + o_proj (this rank's input columns), gate_up and down projections run it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="drafter fc, 32 verify rows", spec=dict(m=32, n=6144, k=18432)), + Cell(why="drafter QKV, nine draft rows", spec=dict(m=9, n=1280, k=12288)), + Cell(why="drafter o_proj, a prefill chunk", spec=dict(m=1024, n=6144, k=1024)), + Cell(why="drafter gate_up, 16 rows", spec=dict(m=16, n=4608, k=6144)), + Cell(why="drafter down, eight rows", spec=dict(m=8, n=6144, k=2304)), + ) + + note = """ + cuBLAS's split and accumulation order: `reference` multiplies in fp32 (one bf16 rounding), within the test's band. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x, w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return (x.float() @ w.float().t()).to(x.dtype) + + +m3_bf16_linear = M3Bf16Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_draft_lowm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_draft_lowm.py new file mode 100644 index 000000000000..1caeda1aa98d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_draft_lowm.py @@ -0,0 +1,150 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""BF16 x BF16 -> BF16 low-M GEMM of the Eagle3 drafter's linears at fixed FlashInfer tactics (SM10x).""" + +from __future__ import annotations + +from typing import Mapping, Optional, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: (N, K) of the TP8 drafter shards -> ((max rows, kind, tactic), ...), rows ascending. Kind "d" is FlashInfer's direct +#: SIMT kernel with tactic (block, outputs per block, rows per block); kind "s" is its split-K tensor-core kernel with +#: tactic (mma_m, mma_n, split_k, ab_stages). A row count is served by the first entry whose max rows holds it; rows +#: past the last entry are not served. GB300 cold L2, CUDA-graph timing, us cuBLAS -> low-M at 1 / 4 rows: qkv 11.9 -> +#: 6.4 / 12.0 -> 7.8, o 3.6 -> 2.3 / -, gate_up 11.9 -> 9.4 / 12.1 -> 11.0, down 6.8 -> 5.6 / -, fc 37.9 -> 31.9 / +#: 39.2 -> 35.5. +TACTICS: Mapping[Tuple[int, int], tuple] = { + # qkv + (1280, 12288): ( + (1, "d", (384, 2, 1)), + (2, "d", (128, 4, 2)), + (3, "d", (128, 4, 3)), + (8, "s", (64, 8, 4, 12)), + ), + # o_proj + (6144, 1024): ((1, "d", (32, 4, 1)), (2, "d", (32, 4, 2))), + # gate_up + (4608, 6144): ( + (1, "d", (128, 1, 1)), + (2, "d", (256, 2, 2)), + (3, "s", (128, 8, 3, 6)), + (4, "s", (128, 8, 3, 5)), + (8, "s", (64, 8, 2, 10)), + ), + # down + (6144, 2304): ((1, "d", (96, 1, 1)), (2, "d", (96, 2, 2))), + # the Eagle3 fc over the three captured hidden states + (6144, 18432): ( + (1, "d", (384, 2, 1)), + (2, "d", (64, 4, 2)), + (3, "d", (32, 2, 3)), + (8, "s", (64, 8, 1, 12)), + ), +} + + +def tactic_for(n: int, k: int, m: int) -> Optional[Tuple[str, tuple]]: + """(kind, tactic) of an [m, k] x [n, k]^T call, or None when the table does not serve it.""" + for rows, kind, tactic in TACTICS.get((n, k), ()): + if m <= rows: + return kind, tactic + return None + + +#: The drafter linear each (N, K) of the table is. +_NAMES: Mapping[Tuple[int, int], str] = { + (1280, 12288): "qkv", + (6144, 1024): "o_proj", + (4608, 6144): "gate_up", + (6144, 2304): "down", + (6144, 18432): "fc", +} + + +class M3DraftLowM(OpWrapper): + """``out`` [M, N] BF16 = ``x`` [M, K] BF16 @ ``weight`` [N, K]^T BF16, FP32 accumulation. + + ``kind`` / ``tactic`` pick FlashInfer's kernel: "d" runs ``run_direct_dense`` (SIMT, one dot product per output in + K order), "s" runs ``run_splitk_dense`` (tcgen05 MMA, K split over ``split_k`` CTAs and summed in FP32). Both + round once to BF16. The kernel is compiled on its first call per (M, N, K, tactic, PDL); the target compiles every + covered call before warmup (modeling's ``_prebuild_draft_lowm``). + """ + + ARCHS = frozenset({Arch.SM_103}) + # Every row count of every table entry. + CELLS: tuple[Cell, ...] = tuple( + Cell( + why=f"drafter {_NAMES[nk]}, {m} rows: {tactic_for(nk[0], nk[1], m)[0]} tactic", + spec=dict(nk=nk, m=m), + ) + for nk, table in TACTICS.items() + for m in range(1, table[-1][0] + 1) + ) + + note = """ + The table is the TP8 drafter's: its shapes are the per-rank shards of the Eagle3 layer at 8-way tensor + parallelism (64 query and 4 KV heads, intermediate 18432) and the replicated fc. Certified at those shapes and + rows only; FlashInfer's kernels take other shapes, which are not measured here. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + ) -> torch.Tensor: + if kind == "d": + from flashinfer.gemm.kernels.dense_bf16_gemm_direct import ( + DirectTactic, + run_direct_dense, + ) + + run_direct_dense(x, weight.t(), out, use_pdl, DirectTactic(*tactic)) + else: + from flashinfer.gemm.kernels.dense_bf16_gemm_sm100_splitk import ( + SplitKTactic, + run_splitk_dense, + ) + + run_splitk_dense(x, weight.t(), None, out, use_pdl, SplitKTactic(*tactic)) + return out + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + ) -> torch.Tensor: + """bf16(x @ weight^T) summed in fp64 and rounded once; ``out``, ``kind``, ``tactic`` and ``use_pdl`` only + choose where and how the op writes.""" + return (x.double() @ weight.double().t()).to(torch.bfloat16) + + def is_valid( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + ) -> None: + """The kernels index ``x`` and ``out`` densely: a strided view is read as if it were packed.""" + assert x.dim() == 2 and x.is_contiguous(), "x must be a contiguous [M, K] matrix" + assert weight.dim() == 2 and weight.is_contiguous(), ( + "weight must be a contiguous [N, K] matrix" + ) + assert out.is_contiguous() and tuple(out.shape) == (x.shape[0], weight.shape[0]), ( + "out must be a contiguous [M, N] matrix" + ) + + +m3_draft_lowm = M3DraftLowM() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_fp32_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_fp32_linear.py new file mode 100644 index 000000000000..8d7374810675 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_fp32_linear.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The router's fp32 projection beyond the decode routers' rows: ``float(x) @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + + +class M3Fp32Linear(OpWrapper): + """fp32 router logits ``F.linear(float(x[M, 6144]), w[128, 6144])`` for prefill and steps above 48 rows (the + fp32 GEMV and the tensor-core router take 1..48). Whatever fp32 matmul precision the process runs with applies + (the engine leaves torch's default).""" + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="a prefill chunk", spec=dict(m=2048, e=128, k=6144)), + Cell( + why="a generation step above the tensor-core router's 48 rows", + spec=dict(m=64, e=128, k=6144), + ), + ) + + note = """ + Under torch's default fp32 matmul precision the product may be TF32: the test's band covers that rounding against + the true fp32 reference. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x.to(torch.float32), w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + with true_fp32_matmul(): + return x.float() @ w.t() + + +m3_fp32_linear = M3Fp32Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mx_gemv.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mx_gemv.py new file mode 100644 index 000000000000..17c48fe4f7c6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mx_gemv.py @@ -0,0 +1,160 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 x MXFP8 GEMV (tcgen05, block-scaled, CuTe DSL) on an activation its producer already quantized.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..moe.m3_moe_engine import mxfp8_dequant +from ..mx_gemv import op as _gemv + +__all__ = ["PROJECTIONS", "gemv_is_valid", "m3_mx_gemv", "served_cells"] + +# (N, K) of this target's MXFP8 projections at tp8, the GEMV's tuned shapes. The shared expert's projections +# (768 x 6144, 6144 x 384) run inside the MoE engine at decode rows and on the generic GEMM otherwise. +QKV_SPARSE = (1536, 6144) +QKV = (1280, 6144) +O_PROJ = (6144, 1024) +DENSE_GATE_UP = (3072, 6144) +DENSE_DOWN = (6144, 1536) +PROJECTIONS = { + "sparse-layer QKV": QKV_SPARSE, + "dense-attention QKV": QKV, + "o_proj": O_PROJ, + "dense gate_up": DENSE_GATE_UP, + "dense down": DENSE_DOWN, +} + + +def served_cells(named: tuple[Cell, ...], x_bf16: bool, max_m: int) -> tuple[Cell, ...]: + """A cell for every (projection, rows) the GEMV serves up to ``max_m`` rows that ``named`` does not list. + + Certification is per projection: a projection's split-K, ring and MMA width differ, so each (N, K, M) the + decode dispatch can send here is its own cell. + """ + listed = {(cell.spec["nk"], cell.spec["m"]) for cell in named} + return tuple( + Cell(why=f"{name}, {m} rows", spec=dict(nk=nk, m=m)) + for name, nk in PROJECTIONS.items() + for m in range(1, max_m + 1) + if (nk, m) not in listed and _gemv.serves(nk[0], nk[1], m, x_bf16) + ) + + +def gemv_reference( + x: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor +) -> torch.Tensor: + """bf16(x @ dequant(weight)^T) for fp32 activation values x [M, K], summed in fp64 and rounded once.""" + return (x.double() @ mxfp8_dequant(weight, weight_scale).double().t()).to(torch.bfloat16) + + +def gemv_is_valid(weight: torch.Tensor, weight_scale: torch.Tensor) -> None: + n, k = weight.shape + # The kernel reads both from data_ptr() as dense buffers at the layout it was compiled for. + assert weight.dtype == torch.float8_e4m3fn and weight.is_contiguous(), ( + "weight must be contiguous e4m3 [N, K]" + ) + assert weight_scale.numel() == n * k // 32, ( + "weight_scale must be the 128x4-swizzled UE8M0 buffer [N * K / 32]" + ) + + +class M3MxGemv(OpWrapper): + """bf16 [M, N] = xq [M, K] (e4m3, 128x4-swizzled UE8M0 scales ``xsf``) x weight [N, K]^T (e4m3, swizzled + ``weight_scale``), M <= 48; None when the shape is not served (the caller falls back). + + ``trigger_early``: release the PDL dependents before the epilogue. ``cfg``: a (split, ring[, contig, nacc, a_tmem, + st_async]) tuning override; default the table's. ``use_pdl``: default the environment's. ``ar`` = (multicast + workspace address, flags address, rank, world, bucket tokens, mode): also push the rows into the next boundary + all-reduce's exchange (then ``op.last_pushed`` is True). + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The projections' (N, K) at the token counts each build differs in: the MMA width (8 / 16 / 32 / 48 columns) a + # call uses is the smallest that holds M, and the sparse-layer QKV takes another split at 2 tokens and above 8. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token (split 8)", spec=dict(nk=QKV_SPARSE, m=1)), + Cell( + why="sparse-layer QKV, 2 tokens: its split-4 config, A from shared memory", + spec=dict(nk=QKV_SPARSE, m=2), + ), + Cell( + why="sparse-layer QKV, 4 rows: one request verifying 3 drafts", + spec=dict(nk=QKV_SPARSE, m=4), + ), + Cell( + why="sparse-layer QKV, 12 rows: the wide split-4 config", + spec=dict(nk=QKV_SPARSE, m=12), + ), + Cell(why="dense-attention QKV, one token", spec=dict(nk=QKV, m=1)), + Cell(why="dense-attention QKV, 13 rows: the 16-column MMA", spec=dict(nk=QKV, m=13)), + Cell(why="o_proj, one token (split 4)", spec=dict(nk=O_PROJ, m=1)), + Cell(why="o_proj, 24 rows: the 32-column MMA", spec=dict(nk=O_PROJ, m=24)), + Cell(why="dense gate_up, one token", spec=dict(nk=DENSE_GATE_UP, m=1)), + Cell(why="dense down, 8 tokens", spec=dict(nk=DENSE_DOWN, m=8)), + ) + # Every other (projection, rows) the decode dispatch can send here (dispatch.MX_GEMV_ROWS). + CELLS = CELLS + served_cells(CELLS, x_bf16=False, max_m=48) + + note = """ + The kernel sums in fp32 in split-K order where `reference` sums in fp64: the bf16 outputs agree within the band the + test derives. Not covered: the all-reduce push (`ar`), which needs a multicast exchange; `cfg` overrides beyond the + table (the target passes none). Shapes outside the table and M > 48 return None. + """ + + def __call__( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv(xq, xsf, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The dequantized product; the launch options are ignored.""" + return gemv_reference(mxfp8_dequant(xq, xsf), weight, weight_scale) + + def is_valid( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + m, k = xq.shape + assert xq.dtype == torch.float8_e4m3fn and xq.is_contiguous(), ( + "xq must be contiguous e4m3 [M, K]" + ) + # A linear-layout [M, K / 32] scale buffer is read as the swizzled one: wrong scales, no error. + assert xsf.numel() >= (m + 127) // 128 * 128 * (k // 32), ( + "xsf must be the 128x4-swizzled scale buffer (rows padded to 128), not the linear one" + ) + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv = M3MxGemv() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mx_gemv_bf16.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mx_gemv_bf16.py new file mode 100644 index 000000000000..08abc2249da3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mx_gemv_bf16.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 GEMV (CuTe DSL) on a bf16 activation, quantized to MXFP8 in the kernel's prologue.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..moe.m3_moe_engine import mxfp8_quant_dequant +from ..mx_gemv import op as _gemv +from .m3_mx_gemv import ( + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + gemv_is_valid, + gemv_reference, + served_cells, +) + +__all__ = ["m3_mx_gemv_bf16"] + + +class M3MxGemvBf16(OpWrapper): + """bf16 [M, N] = mxfp8_quantize(x) x weight [N, K]^T for a contiguous, 16-byte aligned bf16 ``x`` [M, K]; the + quantization is mxfp8_quantize's (UE8M0 scale per 32 = the smallest power of two >= amax / 448). None when the + shape is not served. The other arguments as ``m3_mx_gemv``'s. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The prologue quantizes one 32-element block per thread and round: the projections at the token counts the + # target sends here (no producer-quantized activation), up to the widest each serves. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token", spec=dict(nk=QKV_SPARSE, m=1)), + Cell(why="sparse-layer QKV, 2 tokens: its split-4 config", spec=dict(nk=QKV_SPARSE, m=2)), + Cell( + why="sparse-layer QKV, 16 rows: the most its split-4 prologue serves", + spec=dict(nk=QKV_SPARSE, m=16), + ), + Cell(why="dense-attention QKV, 8 tokens", spec=dict(nk=QKV, m=8)), + Cell(why="o_proj, one token", spec=dict(nk=O_PROJ, m=1)), + Cell(why="o_proj, 48 rows: the 48-column MMA", spec=dict(nk=O_PROJ, m=48)), + Cell( + why="dense gate_up, 16 rows: the most split 4 at K 6144 serves", + spec=dict(nk=DENSE_GATE_UP, m=16), + ), + Cell(why="dense down, one token", spec=dict(nk=DENSE_DOWN, m=1)), + Cell( + why="dense down, 32 rows: the most split 2 at K 1536 serves", + spec=dict(nk=DENSE_DOWN, m=32), + ), + ) + # Every other (projection, rows) the decode dispatch can send here (dispatch.MX_GEMV_BF16_MAX_TOKENS). + CELLS = CELLS + served_cells(CELLS, x_bf16=True, max_m=48) + + note = """ + As `m3_mx_gemv`. The in-kernel quantization is mxfp8_quantize's bit for bit except where the kernel's + rcp.approx(448) and an exact 1/448 put a block's amax on different sides of a power of two (not observed on the + cells). Not covered: `ar`, `cfg` overrides. Returns None for rows beyond 8 rounds of the prologue per CTA. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv_bf16(x, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The MXFP8 round trip of x times the dequantized weight; the launch options are ignored.""" + return gemv_reference(mxfp8_quant_dequant(x), weight, weight_scale) + + def is_valid( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv_bf16 = M3MxGemvBf16() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mxfp8_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mxfp8_gemm.py new file mode 100644 index 000000000000..8fc131a7bb61 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mxfp8_gemm.py @@ -0,0 +1,58 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 GEMM on an activation its producer already quantized (rows the GEMV does not serve).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import mxfp8_gemm_autotuned + +from .m3_mxfp8_linear import mxfp8_reference + + +class M3Mxfp8Gemm(OpWrapper): + """``act @ weight^T`` for an MXFP8 activation (e4m3 [M, K] + 128x4-swizzled UE8M0 scales) and an MXFP8 weight + [N, K] (swizzled scales), bf16 out. The GEMM backend per token bucket is the AutoTuner's (the same runner, op key + and tactic cache as the engine-tuned quantize + GEMM); ``tune`` profiles in the engine's tuning pass.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The decode dispatch hands it the boundary / merge MXFP8 activation when the GEMV declines rows it serves: the + # sparse-layer QKV and the o_proj at the tp8 shards. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, 4 rows", spec=dict(m=4, n=1536, k=6144)), + Cell(why="o_proj, 32 rows", spec=dict(m=32, n=6144, k=1024)), + ) + + note = """ + Exactly the MXFP8 product (fp32 accumulation, one bf16 rounding): `reference` dequantizes both operands and + multiplies in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + return mxfp8_gemm_autotuned(act, act_scale, weight, weight_scale, output_dtype, tune=tune) + + def reference( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + """The dequantized product; ``tune`` is ignored.""" + return mxfp8_reference(act, act_scale, weight, weight_scale).to(output_dtype) + + +m3_mxfp8_gemm = M3Mxfp8Gemm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mxfp8_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mxfp8_linear.py new file mode 100644 index 000000000000..1bdc1829508d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gemm/m3_mxfp8_linear.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One MXFP8 projection on the engine-tuned path: dynamic MXFP8 quantize of a bf16 activation + block-scaled GEMM.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import mxfp8_quantize_gemm_autotuned + +_BLOCK = 32 + + +def _unswizzle(scale: torch.Tensor, rows: int, cols: int) -> torch.Tensor: + """[rows, cols] UE8M0 bytes from the 128x4-swizzled layout (rows padded to 128, columns to 4).""" + pad_r = (rows + 127) // 128 * 128 + pad_c = (cols + 3) // 4 * 4 + s = scale.view(torch.uint8).reshape(pad_r // 128, pad_c // 4, 32, 4, 4) + return s.permute(0, 3, 2, 1, 4).reshape(pad_r, pad_c)[:rows, :cols] + + +def _dequant(x: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + rows, k = x.shape + s = _unswizzle(scale, rows, k // _BLOCK).to(torch.int32) + factor = torch.pow(2.0, (s - 127).to(torch.float32)) + return ( + x.view(torch.float8_e4m3fn).float().view(rows, k // _BLOCK, _BLOCK) * factor.unsqueeze(-1) + ).view(rows, k) + + +def mxfp8_reference(act, act_scale, weight, weight_scale) -> torch.Tensor: + """fp32 product of two dequantized MXFP8 operands (swizzled UE8M0 scales).""" + with true_fp32_matmul(): + return _dequant(act, act_scale) @ _dequant(weight, weight_scale).t() + + +class M3Mxfp8Linear(OpWrapper): + """``x @ weight^T`` for bf16 ``x`` [M, K] and an MXFP8 weight [N, K] (e4m3 + 128x4-swizzled UE8M0 scales), bf16 + out, on the backend the engine selected: + + * ``tuned_quantize_gemm`` (decode CUDA graphs once the engine tuned per-bucket backends): the AutoTuner's quantize + and GEMM per token bucket (``tune`` profiles them in the engine's tuning pass); + * otherwise the native ``mxfp8_quantize`` (swizzled scales) and then FlashInfer's MXFP8 GEMM (``flashinfer``) or + the CUTLASS block-scaled GEMM, through the autotuner while the engine tunes it (``native_tuning``), scaled by + ``alpha`` (fp32 [1], 1.0 for these checkpoints). + + Every MXFP8 projection of the target (the attention QKV and o_proj, the dense MLPs and the shared expert) runs it + outside the decode-step kernels. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="sparse-layer QKV, a prefill chunk, native GEMM", + spec=dict(m=2048, n=1536, k=6144, path="native"), + ), + Cell( + why="o_proj, a prefill chunk, native GEMM", + spec=dict(m=2048, n=6144, k=1024, path="native"), + ), + Cell( + why="dense down projection, eager decode, native GEMM", + spec=dict(m=4, n=6144, k=1536, path="native"), + ), + Cell( + why="shared gate_up, FlashInfer GEMM", + spec=dict(m=64, n=768, k=6144, path="flashinfer"), + ), + Cell(why="o_proj, tuned quantize + GEMM", spec=dict(m=64, n=6144, k=1024, path="tuned")), + ) + + note = """ + Exactly the MXFP8 product of the MXFP8 round trip of x (fp32 accumulation, one bf16 rounding): `reference` + quantizes x with the same block rule (UE8M0 scale = the block's power-of-two ceiling over e4m3's 448) and + multiplies dequantized operands in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + if tuned_quantize_gemm: + return mxfp8_quantize_gemm_autotuned(x, weight, weight_scale, output_dtype, tune=tune) + act, act_scale = torch.ops.trtllm.mxfp8_quantize(x, True) + if flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8( + act, act_scale, weight, weight_scale, output_dtype + ) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, weight, weight_scale, alpha, output_dtype) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + """The MXFP8 round trip of ``x`` times the dequantized weight, times ``alpha``; the backend flags are + ignored.""" + m, k = x.shape + blocks = x.float().view(m, k // _BLOCK, _BLOCK) + amax = blocks.abs().amax(-1, keepdim=True) + exp = torch.ceil(torch.log2(torch.where(amax > 0, amax / 448.0, torch.ones_like(amax)))) + scale = torch.pow(2.0, exp) + q = (blocks / scale).to(torch.float8_e4m3fn).float() * scale + with true_fp32_matmul(): + out = q.view(m, k) @ _dequant(weight, weight_scale).t() + return (out * alpha.float()).to(output_dtype) + + +m3_mxfp8_linear = M3Mxfp8Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gqamix_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gqamix_sparse_decode.py new file mode 100644 index 000000000000..e6b824de1b96 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/gqamix_sparse_decode.py @@ -0,0 +1,1311 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. +""" + +from __future__ import annotations + +import os +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +def _pdl_enabled() -> bool: + return os.environ.get("TRTLLM_ENABLE_PDL", "1") == "1" + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for _ in tl.range(chunk_start, chunk_end): + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for _ in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on both datacenter Blackwell targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3)) +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: tuple[int, int], +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + if kv_nvfp4: + capability = torch.cuda.get_device_capability(q.device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else 1 + ) + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = _pdl_enabled() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + if not direct_output: + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + **pdl_launch, + **merge_launch_options, + ) + + +__all__ = [ + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/long_select.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/long_select.py new file mode 100644 index 000000000000..8a028e1cc288 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/long_select.py @@ -0,0 +1,77 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 block select for MiniMax-M3 decode rows of 129 to 9216 blocks. + +Same contract and bit-identical output as ``torch.ops.trtllm.minimax_m3_select_blocks`` on per-KV-head scores +[num_kv_heads, n_blocks, total_q]: init / local forcing, per-query valid-block count, ties to the smaller block index, +forced -inf picks emitted as -1, ascending block ids with -1 tail padding. + +One 1024-thread CTA per (query, KV head) row holds the row's ordered score keys in registers (2, 4 or 9 per +thread, by the row's valid count). The weakest of the 16 maxima of the interleaved groups ``block % 16`` is a lower +bound on the 16th-largest key, so the top 16 is among the keys at or above it; those are compacted to shared memory +and ranked exactly by (key, block). More than 1024 keys tied at the bound (degenerate rows) rank every candidate +against the whole row (re-read). + +The kernel is ``torch.ops.trtllm.minimax_m3_select_long``, called through this target's ``m3_select_long`` entry, or +with ``dsl`` the 384-thread CuTe DSL kernel (``ops/m3_select_long_dsl``, entry ``m3_select_long_dsl``): the same table. +""" + +from typing import Optional + +import torch + +from ..ops.attention.m3_select_long import MAX_BLOCKS, m3_select_long +from ..ops.attention.m3_select_long_dsl import m3_select_long_dsl + + +def long_select_blocks( + max_score_kv: torch.Tensor, + *, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + dsl: bool = False, +) -> Optional[torch.Tensor]: + """Top-16 select over per-KV-head scores [num_kv_heads, n_blocks, total_q] with 129..9216 blocks. + + Returns the head-major backed [total_q, num_kv_heads, 16] int32 table, or None outside this kernel's shapes + (callers keep the reference op). ``early_trigger`` releases the dependent sparse decode right after the select's + grid wait (``SELECT_EARLY_MAX_ROWS`` in modeling.py). ``dsl`` runs the 384-thread kernel (``SELECT_LONG_DSL``; + the same table), released at its exit unless ``early_trigger``. + """ + if ( + max_score_kv.dtype != torch.float32 + or not max_score_kv.is_cuda + or max_score_kv.dim() != 3 + or not 128 < max_score_kv.shape[1] <= MAX_BLOCKS + or n_valid_blocks.dtype != torch.int32 + or not n_valid_blocks.is_cuda + or n_valid_blocks.shape[0] < max_score_kv.shape[2] + ): + return None + num_kv_heads, _, total_q = max_score_kv.shape + out = torch.empty( + (num_kv_heads, total_q, 16), dtype=torch.int32, device=max_score_kv.device + ).permute(1, 0, 2) + if dsl: + return m3_select_long_dsl( + max_score_kv, + n_valid_blocks.contiguous(), + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + 0, + ) + return m3_select_long( + max_score_kv, + n_valid_blocks.contiguous(), + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_attn_lc8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_attn_lc8.py new file mode 100644 index 000000000000..2c4c55cf089f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_attn_lc8.py @@ -0,0 +1,116 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 long-context sparse decode attention at 8 query heads in one launch. + +Replaces the generic chain's split-K producer (``_gqa_sparse_decode_kernel``, one block per chunk) and the LSE +merge (``_merge_topk_attn_out_kernel``, with its MXFP8 copy for the o_proj) once the top-k block table exists. + +Grid: one cluster of 16 CTAs per query token; CTA ``r`` attends the token's ``r``-th selected block for all 8 heads, +keeps the partial in shared memory, and the cluster merges the 16 partials over DSMEM into the BF16 row and, +optionally, its MXFP8 copy and swizzled UE8M0 scales exactly as ``_store_mxfp8_head`` lays them out. + +Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and PV +on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +log-domain recurrence. Accumulation order inside the MMAs and the reductions differs, so results match to rounding, +not bit for bit. + +Schedule: the kernel releases its PDL dependents right after its grid wait, so the o_proj launches and streams its +weights while attention runs. Up to ``LC8_PREWAIT_MAX_ROWS`` (modeling.py) query rows it also stages the request +length, the query rows and the request's block-table row before that wait, overlapping them with the select ahead +when it releases this grid early (``SELECT_EARLY_MAX_ROWS`` in modeling.py). Neither changes a value. + +Geometry: 8 query heads per KV head (TP8), one KV head, head_dim 128, page 128, top-k 16. The kernel is +``torch.ops.trtllm.minimax_m3_tp8_sparse_decode_lc8``, called through this target's ``m3_sparse_decode_lc8`` entry. +""" + +import torch + +from ..ops.attention.m3_sparse_decode_lc8 import NUM_HEADS, WIDTH, m3_sparse_decode_lc8 + + +def _prewait_max_rows() -> int: + """``LC8_PREWAIT_MAX_ROWS`` of the target's dispatch (imported at call time: modeling imports this module).""" + from .. import modeling + + return modeling.LC8_PREWAIT_MAX_ROWS + + +def lc8_applies(q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output) -> bool: + """Whether the step's operands fit the kernel: TP8 geometry, packed NVFP4 pages, FP8 q, BF16 [T, 8, 128] out.""" + if q.dim() != 3 or tuple(q.shape[1:]) != (NUM_HEADS, 128) or q.dtype != torch.float8_e4m3fn: + return False + if q.stride(2) != 1 or q.stride(1) % 16 or q.stride(0) % 16 or q.data_ptr() % 16: + return False + if int(k_paged.shape[1]) != 1 or int(topk_idx.shape[0]) != 1 or int(topk_idx.shape[-1]) != 16: + return False + for t in (k_paged, v_paged): + if ( + t.stride(3) != 1 + or t.stride(2) != 64 + or int(t.shape[3]) != 64 + or t.data_ptr() % 16 + or t.stride(0) % 16 + ): + return False + for t in (k_block_scale, v_block_scale): + if t is None or t.stride(3) != 1 or t.stride(2) != 8 or t.data_ptr() % 8 or t.stride(0) % 8: + return False + return ( + output.dtype == torch.bfloat16 + and output.dim() == 3 + and output.is_contiguous() + and tuple(output.shape) == tuple(q.shape) + ) + + +def m3_lc8_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + linear_softmax=True, + mxfp8=None, + use_pdl=True, +): + """Drop-in for ``minimax_m3_sparse_attn_decode`` on the TP8 NVFP4 shape; callers check ``lc8_applies`` first. + + ``topk_idx`` is the head-major [1, total_q, 16] view the Triton producer takes. ``mxfp8`` is ``(q8, sf)`` from + ``MERGE_MXFP8.buffers(1024, dev)`` to also emit the o_proj hand-off. + """ + total_q = int(q.shape[0]) + q8, sf = ( + (mxfp8[0].view(torch.uint8), mxfp8[1].view(torch.uint8)) + if mxfp8 is not None + else (None, None) + ) + m3_sparse_decode_lc8( + q.view(torch.uint8), + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx[0], + block_table, + seq_lens, + output.view(total_q, WIDTH), + q8, + sf, + WIDTH // 128, + int(decode_query_len), + float(sm_scale), + bool(linear_softmax), + bool(use_pdl), + total_q <= _prewait_max_rows(), + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_bp_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_bp_tail.py new file mode 100644 index 000000000000..419a5baef898 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_bp_tail.py @@ -0,0 +1,192 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""BOUNDARY_PACK BP-1's graph tail for a batch-1 decode step, in one launch: the spec sampler's four slot stores and +the engine's next-step gathers, both gated by a device-side live word. + +``ctl`` (int64 [2]) holds (slot, live), written by the step fold's ``begin_step`` before the replay. With ``live`` 0 +the launch writes nothing (a step the fold did not take: a dummy-only replay, a step it could not identify). With +``live`` 1, for the one row: + +* the stores, as the sampler's ``_store_outputs`` (zero padding / truncation to the store widths, then index_copy_ + at ``slot``):: + + new_tokens[slot][c] = src_new[c] if c < src width else 0 (c < store width), likewise + next_new_tokens[slot][c] , next_draft_tokens[slot][c]; new_tokens_lens[slot] = src_lens + +* with ``GATHER``, the engine's gathers for the next step of the same row (P = 1, no rows before it), as + ``step_gather`` reads them back from the store just written:: + + ids[t] = next_new_tokens[slot][t] (t < T) draft[d] = next_draft_tokens[slot][d] (d < T - 1) + pos_off[i] = lens[slot] if i < T else 0 kv_off[i] = lens[slot] - T if i < 1 else 0 + (every i of pos_off / kv_off: the engine zeroes both buffers before its gathers) + +Integer copies only. Program 0 does the row; every program zero-fills its ``ZBLOCK`` chunk of the offsets. The int +arguments are not specialized and every pointer is 16-byte aligned (the caller checks), so the two ``GATHER`` +variants compiled by ``warm_up`` outside capture serve every graph. +""" + +import torch +import triton +import triton.language as tl + +ZBLOCK = 1024 +_INTS = ( + "s_new_s1", + "s_new_w", + "d_new_s0", + "d_new_s1", + "d_new_w", + "s_nnew_s1", + "s_nnew_w", + "d_nnew_s0", + "d_nnew_s1", + "d_nnew_w", + "d_lens_s0", + "s_draft_s1", + "s_draft_w", + "d_draft_s0", + "d_draft_s1", + "d_draft_w", + "n_pos", + "n_kv", + "T", +) + + +@triton.jit(do_not_specialize=list(_INTS)) +def _bp_tail_kernel( + ctl, + s_new, + s_new_s1, + s_new_w, + d_new, + d_new_s0, + d_new_s1, + d_new_w, + s_nnew, + s_nnew_s1, + s_nnew_w, + d_nnew, + d_nnew_s0, + d_nnew_s1, + d_nnew_w, + s_lens, + d_lens, + d_lens_s0, + s_draft, + s_draft_s1, + s_draft_w, + d_draft, + d_draft_s0, + d_draft_s1, + d_draft_w, + ids, + draft, + pos_off, + n_pos, + kv_off, + n_kv, + T, + GATHER: tl.constexpr, + BLOCK: tl.constexpr, + ZBLOCK_: tl.constexpr, +): + slot = tl.load(ctl) + live = tl.load(ctl + 1) + if live != 0: + pid = tl.program_id(0) + lens = tl.load(s_lens) + if pid == 0: + c = tl.arange(0, BLOCK) + new = tl.load(s_new + c * s_new_s1, mask=c < s_new_w, other=0) + tl.store(d_new + slot * d_new_s0 + c * d_new_s1, new, mask=c < d_new_w) + nnew = tl.load(s_nnew + c * s_nnew_s1, mask=c < s_nnew_w, other=0) + tl.store(d_nnew + slot * d_nnew_s0 + c * d_nnew_s1, nnew, mask=c < d_nnew_w) + dr = tl.load(s_draft + c * s_draft_s1, mask=c < s_draft_w, other=0) + tl.store(d_draft + slot * d_draft_s0 + c * d_draft_s1, dr, mask=c < d_draft_w) + tl.store(d_lens + slot * d_lens_s0, lens) + if GATHER: + tl.store(ids + c, nnew, mask=c < T) + tl.store(draft + c, dr, mask=c < T - 1) + if GATHER: + z = pid * ZBLOCK_ + tl.arange(0, ZBLOCK_) + tl.store(pos_off + z, tl.where(z < T, lens, 0), mask=z < n_pos) + tl.store(kv_off + z, tl.where(z < 1, lens - T, 0), mask=z < n_kv) + + +def block_for(store) -> int: + """The row block of a sampler store: every store width fits it (fixed per store, so per fold).""" + widths = ( + store.new_tokens.shape[0], + store.next_new_tokens.shape[0], + store.next_draft_tokens.shape[1], + ) + return triton.next_power_of_2(max(*widths, 1)) + + +def aligned(*tensors: torch.Tensor) -> bool: + """Every pointer the launch takes is 16-byte aligned (the compiled variants' assumption).""" + return all(t.data_ptr() % 16 == 0 for t in tensors) + + +def bp_tail(ctl, src, store, engine, t: int, gather: bool, block: int) -> None: + """One launch: ``src`` = (new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens) row 0 of the worker's + int32 outputs ([1, w] / [1]); ``store`` the sampler's store; ``engine`` = (input_ids, draft_tokens, + position_id_offsets, kv_lens_offsets) int32 contiguous; ``t`` tokens per request (gather only).""" + s_new, s_nnew, s_lens, s_draft = src + d_new = store.new_tokens.squeeze(-1).T + d_nnew = store.next_new_tokens.squeeze(-1).T + d_lens, d_draft = store.new_tokens_lens, store.next_draft_tokens + ids, draft, pos_off, kv_off = engine + n_pos, n_kv = pos_off.numel(), kv_off.numel() + grid = (max(1, triton.cdiv(max(n_pos, n_kv), ZBLOCK)) if gather else 1,) + _bp_tail_kernel[grid]( + ctl, + s_new, + s_new.stride(1), + s_new.shape[1], + d_new, + d_new.stride(0), + d_new.stride(1), + d_new.shape[1], + s_nnew, + s_nnew.stride(1), + s_nnew.shape[1], + d_nnew, + d_nnew.stride(0), + d_nnew.stride(1), + d_nnew.shape[1], + s_lens, + d_lens, + d_lens.stride(0), + s_draft, + s_draft.stride(1), + s_draft.shape[1], + d_draft, + d_draft.stride(0), + d_draft.stride(1), + d_draft.shape[1], + ids, + draft, + pos_off, + n_pos, + kv_off, + n_kv, + t, + GATHER=gather, + BLOCK=block, + ZBLOCK_=ZBLOCK, + num_warps=4, + ) + + +def warm_up(ctl, store, engine, block: int) -> None: + """Compile both variants outside capture, with ``ctl`` live 0 (the launch writes nothing).""" + src = ( + store.new_tokens.squeeze(-1).T[:1], + store.next_new_tokens.squeeze(-1).T[:1], + store.new_tokens_lens[:1], + store.next_draft_tokens[:1], + ) + for gather in (False, True): + bp_tail(ctl, src, store, engine, 2, gather, block) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_draft_step.py new file mode 100644 index 000000000000..917f04005cc9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_draft_step.py @@ -0,0 +1,149 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 draft pass for the MiniMax-M3 drafter's vocab-sharded BF16 head (TP decode, <= 8 requests). + +One kernel replaces a draft pass's head and the draft loop's per-pass bookkeeping, at the in-tree precision: + +* The drafter's BF16 lm_head shard is kept a second time, built by ``prepare_head`` before the executor's warmup, in + ``mma.m16n8k16`` A-fragment order (``bf16_fragments``, the same values). The kernel streams it with BF16 MMAs and + FP32 accumulation over the BF16 head input rows. +* Each logit is rounded to BF16 (the in-tree logits are the BF16 head GEMM's output) and keyed + ``ordered(value) << 32 | ~global_index`` with -0 as +0, so the larger value wins and ties take the lowest index, as + the in-tree ``torch.max`` / ``torch.argmax`` over the gathered logits (``greedy_sample_draft_with_tp_gather``). +* The last CTA exchanges each row's key across the TP ranks through a multicast mapping (Lamport slots, see + ``m3_shard_accept``), then stores the draft token, the next pass's input id and position, and advances the draft KV + lengths as the in-tree draft loop does between passes. + +The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/draftStep.cu) and called through the +catalog entry ``ops.spec.m3_draft_step``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +import torch + +from ..ops.spec.m3_draft_step import MAX_TILES_PER_CTA, MAX_TOKENS, bf16_fragments +from ..ops.spec.m3_draft_step import m3_draft_step as _draft_step_op +from .m3_shard_accept import AcceptWorkspace, create_workspace + + +@dataclass(eq=False) +class DraftHead: + """The streamed copy of one rank's draft lm_head shard and its first global id: ``wf`` is the fragment-ordered BF16 + copy (``bf16_fragments``).""" + + weight: torch.Tensor + wf: torch.Tensor + vocab_offset: int + + +def head_supported(lm_head) -> bool: + """A bias-free BF16 head whose local shard the kernel can own: whole 16-row tiles, K a multiple of 16, at most + MAX_TILES_PER_CTA tiles per CTA of a one-CTA-per-SM grid.""" + weight = getattr(lm_head, "weight", None) + if weight is None or weight.dim() != 2 or weight.dtype != torch.bfloat16: + return False + if getattr(lm_head, "bias", None) is not None: + return False + n, k = weight.shape + if n % 16 or k % 16: + return False + return (n // 16 + _num_sms() - 1) // _num_sms() <= MAX_TILES_PER_CTA + + +def prepare_head(lm_head, tp_rank: int) -> DraftHead: + """The fragment-ordered BF16 copy of ``lm_head``'s local shard (global ids from tp_rank * rows). + Built once, eagerly, before the executor's warmup (``M3Eagle3Worker.prebuild_draft_head``).""" + weight = lm_head.weight + offset = tp_rank * weight.shape[0] + with torch.no_grad(): + wf = bf16_fragments(weight.detach()) + return DraftHead(weight=weight, wf=wf, vocab_offset=offset) + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace ([2][MAX_TOKENS][world] slots), created collectively on the first eager call; None + during a CUDA-graph capture that finds it missing.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = create_workspace(mapping, device, MAX_TOKENS) + return _WS + + +def draft_step( + head: DraftHead, + x: torch.Tensor, + rows: int, + x_rows: Optional[torch.Tensor], + mapping, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, +) -> bool: + """One fused greedy draft pass over ``rows`` requests (see the module docstring); False when the exchange + workspace is missing during a capture. Collective over the TP group. + + Args: + head: the prepared head of this rank (``prepare_head``), BF16 fragments. + x: [*, K] BF16 head input rows (the drafter's final-norm output); row ``x_rows[b]`` (int64) or ``b`` is + request b's. + draft_out: int32 [rows, >= step + 1], unit column stride: column ``step`` receives the tokens. + ids_next / pos_next: int32 [rows]: the tokens, and ``pos_src[x_rows[b] or b] + 1`` (pos_src int32). + kv_lens: int32 draft KV lengths: ``-= draft_len - num_acc`` with ``seq_lens = 1`` on pass 0 when ``num_acc`` + is given, ``+= 1`` on later passes. + """ + ws = workspace(mapping, x.device) + if ws is None: + return False + _draft_step_op( + x, + x_rows, + rows, + head.wf, + head.vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + _num_sms(), + True, + None, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_force_vec.py new file mode 100644 index 000000000000..b4de37794ca9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_force_vec.py @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance (the in-tree speculative worker's +``force_num_accepted_tokens``), drawn exactly as its ``_apply_force_accepted_tokens`` does, in one launch. + +Triton: one CTA bumps the shared step counter and draws each request's count from the fixed-seed pool with the +in-tree (counter, slot) hash. Reached through the ``ops/spec`` entry ``m3_force_vec``. +""" + +import triton +import triton.language as tl + + +@triton.jit +def force_vec_kernel( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK: tl.constexpr, +): + c = tl.load(counter) + 1 + slots = tl.arange(0, BLOCK) + tl.store(counter, c) + idx = (c * counter_stride + slots.to(tl.int64) * slot_stride) & pool_mask + live = slots < batch_size + r = tl.load(pool + idx, mask=live, other=1.0) + v = tl.minimum(base_total + (r < frac).to(tl.int32), max_total) + tl.store(out + slots, v, mask=live) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_fused_score_select.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_fused_score_select.py new file mode 100644 index 000000000000..3ef07ce6dbff --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_fused_score_select.py @@ -0,0 +1,110 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The fused index score + top-16 select of a decode span of at most ``dispatch.FUSED_SCORE_SELECT_MAX_REQUESTS`` +requests (``ops/attention/m3_fused_score_select``), with its scratch and arrival counters. + +One scratch [2, 8, ceil4(max_tiles)] and one zeroed counter pair per (device, engine block count) serve every sparse +layer: the layers run one after another on one stream, and the kernel returns each counter to zero. They are +created on an eager decode step; a CUDA-graph capture only reads them. +""" + +from __future__ import annotations + +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + +from ..ops.attention.m3_fused_score_select import ( + MAX_DECODE_QUERY_LEN, + MAX_ROWS, + m3_fused_score_select, + ring_fits, + scratch_shape, +) + +_BUFFERS: Dict[Tuple[str, int], Tuple[torch.Tensor, torch.Tensor]] = {} + + +def supported( + idx_q: torch.Tensor, + idx_k_pages: torch.Tensor, + decode_query_len: int, + num_kv_heads: int, + max_tiles: int, +): + """None when the kernel takes this span's geometry, else the reason it does not.""" + if ( + idx_q.dim() != 3 + or idx_q.dtype != torch.float8_e4m3fn + or idx_k_pages.dtype != torch.float8_e4m3fn + ): + return "the index Q / index-K cache is not FP8" + heads = int(idx_q.shape[1]) + rows = decode_query_len * num_kv_heads + if ( + idx_q.shape[2] != 128 + or idx_k_pages.dim() != 3 + or tuple(idx_k_pages.shape[1:]) != (128, 128) + or not 1 <= decode_query_len <= MAX_DECODE_QUERY_LEN + or heads % num_kv_heads + or heads * MAX_DECODE_QUERY_LEN > MAX_ROWS + or rows > MAX_ROWS + ): + return f"{heads} index heads x {decode_query_len} query rows is not the kernel's geometry" + if not ring_fits(rows, max_tiles): + return f"{max_tiles} blocks of select scratch do not fit the stage ring" + return None + + +def buffers(device: torch.device, max_tiles: int) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """(scratch, counters) for this engine's block count; None during a capture that finds them missing.""" + key = (str(device), int(max_tiles)) + found = _BUFFERS.get(key) + if found is None: + if torch.cuda.is_current_stream_capturing(): + return None + found = _BUFFERS[key] = ( + torch.empty(scratch_shape(max_tiles), dtype=torch.float32, device=device), + torch.zeros(scratch_shape(max_tiles)[0], dtype=torch.int32, device=device), + ) + return found + + +def fused_score_select( + idx_q, + idx_k_pages, + block_table, + seq_lens, + n_valid, + *, + decode_query_len, + num_kv_heads, + init_blocks, + local_blocks, + max_tiles, + scratch, + counters, +): + """The [total_q, num_kv_heads, 16] table (a view of a head-major [num_kv_heads, total_q, 16] backing).""" + total_q = int(idx_q.shape[0]) + out = torch.empty((num_kv_heads, total_q, 16), dtype=torch.int32, device=idx_q.device).permute( + 1, 0, 2 + ) + return m3_fused_score_select( + idx_q, + idx_k_pages, + block_table, + seq_lens, + n_valid, + out, + scratch, + counters, + int(decode_query_len), + int(num_kv_heads), + int(init_blocks), + int(local_blocks), + int(max_tiles), + get_env_enable_pdl(), + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_index_score.py new file mode 100644 index 000000000000..c6a7f6d80080 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_index_score.py @@ -0,0 +1,700 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py +"""MiniMax-M3 index decode block scorer (CuTe DSL, Blackwell) with its pre-wait schedule. + +The kernel of ``tensorrt_llm/_torch/attention/kernels/blackwell/minimax_m3_index_decode_score.py`` (the same scores, bit +for bit) with four schedule options this target compiles in: + +* ``prefetch_old``: before the grid-dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that hold only + tokens older than this step (the producer ahead writes only this step's tokens), up to ``prefetch_depth`` blocks per + CTA (0 = all of them). +* ``early_issue``: the TMA warp reads a ``seq_lens`` hint and its first page id before the wait and issues the Q tile + and the first K tile right after it. The tiles are read after the wait, so their contents are final; the block table + is only written between graph replays, so the early page id is the right one whenever the block is valid. A CTA + whose early tiles lie past the request's true length drains them before it exits. +* ``prestage_old`` (with ``early_issue``, ``INDEX_SCORE_PRESTAGE_OLD``): when the CTA's first block holds only + tokens older than this step (block < (seq_lens hint - decode query length) / 128), its K tile is TMA-loaded into its + stage before the wait; Q (and a first block that is not all old) is still loaded after it. +* ``early_trigger`` (``INDEX_SCORE_SET``): every CTA releases the dependent block selector right after its + setup, so the selector launches and makes its pre-wait reads while this grid scores; the selector still waits for + this whole grid before it reads a score, so the scores and the selection are unchanged. + +Every kernel is compiled before the executor's warmup (``layers.prebuild_index_scorer``) through ``IndexScoreRunner`` +and called through this target's ``m3_index_score`` entry. +""" + +import cutlass +import torch +from cuda.bindings.driver import CUstream +from cutlass import Float8E4M3FN, Float16, Float32, Int32, Int64, Uint32, cute +from cutlass._mlir.dialects import llvm +from cutlass.cute.nvgpu import cpasync, warp +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + EVICT_FIRST, + fp8x4_to_fp16x4, + mma_sync, + simple_tma_copy, +) +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + TRTLLM_ENABLE_PDL, + griddepcontrol_launch_dependents, + griddepcontrol_wait, +) +from tensorrt_llm.logger import logger + +from ..m31_switches import m31_disabled + +# PRESTAGE_OLD (ported from the sm_107 TP8 target, the M3 Rubin Indexer's G1): with the early issue, a CTA whose first +# block holds only tokens older than this step also TMA-loads that block's index-K tile before the grid wait +# (bit-identical scores). Kill switch: TRTLLM_M31_DISABLE_PRESTAGE_OLD=1. +INDEX_SCORE_PRESTAGE_OLD = not m31_disabled("PRESTAGE_OLD") +# SET (ported from the sm_107 TP8 target): every CTA releases the dependent block selector right after its setup +# instead of at its exit (the selector's own grid wait still returns only once the whole scoring grid is complete, so +# the picks are unchanged), at every key. Kill switch: TRTLLM_M31_DISABLE_SCORE_SET=1. +INDEX_SCORE_SET = not m31_disabled("SCORE_SET") + +__all__ = ["IndexDecodeScoreKernel", "IndexScoreRunner"] + + +@dsl_user_op +def _l2_prefetch_bulk(addr: Int64, size: Int32, *, loc=None, ip=None) -> None: + """Prefetch ``size`` contiguous bytes (multiple of 16) at global ``addr`` into L2.""" + llvm.inline_asm( + res=None, + operands_=[Int64(addr).ir_value(loc=loc, ip=ip), Int32(size).ir_value(loc=loc, ip=ip)], + asm_string="cp.async.bulk.prefetch.L2.global [$0], $1;", + constraints="l,r", + has_side_effects=True, + asm_dialect=llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@cute.jit +def _fp8_to_f16_mma_fragments(src: cute.Tensor): + """Split an FP8 ldmatrix fragment into the two FP16 k-fragments MMA wants.""" + src_elems = cute.size(src) + src_u32 = cute.recast_tensor(src, Uint32) + src_f16 = cute.make_rmem_tensor(src_elems, Float16) + src_f16_u32 = cute.recast_tensor(src_f16, Uint32) + # Packed conversion; faster and fewer SASS instructions than + # src.load().to(Float16). + for i in cutlass.range_constexpr(src_elems // 4): + converted = fp8x4_to_fp16x4(src_u32[i]) + src_f16_u32[i * 2] = converted[0] + src_f16_u32[i * 2 + 1] = converted[1] + lower = cute.make_rmem_tensor(src_elems // 2, Float16) + upper = cute.make_rmem_tensor(src_elems // 2, Float16) + + # FP8 ldmatrix yields four consecutive values along K. Split each group + # into the lower two and upper two for the two FP16 MMA k-fragments. + for i in cutlass.range_constexpr(src_elems // 2): + lower[i] = src_f16[(i // 2) * 4 + i % 2] + upper[i] = src_f16[(i // 2) * 4 + 2 + i % 2] + return lower, upper + + +class IndexDecodeScoreKernel: + """Per-block max index score for one decode step. + + Grid is (batch, split_k): CTA (b, s) walks KV blocks + s, s + split_k, s + 2 * split_k, ... of request b and writes each block's + score directly, so no cross-CTA reduction is needed. CTAs whose split_id + exceeds the request's block count exit immediately. + """ + + BLOCK_K = 128 # capacity: fixed: the MSA block size of the checkpoint + BAR_MMA = 1 + num_stages = 2 + + def __init__( + self, + dtype: type[cutlass.Numeric], + num_heads: int, + max_decode_query_len: int, + split_k: int, + head_dim: int = 128, + prefetch_old: bool = False, + early_issue: bool = False, + prefetch_depth: int = 0, + prestage_old: bool = False, + early_trigger: bool = False, + ): + # prefetch_old: before the grid dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that + # hold only tokens older than this step (the predecessor writes only this step's tokens). + self.prefetch_old = prefetch_old + # early_issue: the TMA warp reads a seq_lens hint and this CTA's first page id before the grid dependency + # wait, then issues the Q tile and the first K tile right after it, overlapping the post-wait seq_lens load. + # The tiles are read after the wait, so their contents are final; the block table is only written between + # graph replays, so the early page id is the right one whenever the block is valid. A CTA whose early tiles + # turn out to lie past the request's true length drains them before exiting. + self.early_issue = early_issue + # prefetch_depth: blocks per CTA that prefetch_old covers (0 = all of them). Prefetching every older page at + # large batches floods L2; the first few per CTA are the ones whose latency the pipeline cannot hide. + self.prefetch_depth = prefetch_depth + # prestage_old (with early_issue): when this CTA's first block holds only tokens older than this step, its K + # tile is TMA-loaded into its stage BEFORE the grid dependency wait (the predecessor writes only this step's + # tokens; the block table and the seq_lens hint are final before the grid starts). Q stays after the wait. + self.prestage_old = prestage_old and early_issue + # early_trigger: release the dependent block selector right after this CTA's setup (see the module docstring). + self.early_trigger = early_trigger + self.dtype = dtype + self.num_heads = num_heads + self.max_decode_query_len = max_decode_query_len + self.split_k = split_k + self.head_dim = head_dim + + def _swizzle_elems(self) -> int: + """Elements spanned by one 128-byte swizzle atom.""" + return 128 * 8 // self.dtype.width + + def _sq_layout(self): + """Composed SMEM layout for the Q tile, shared by the descriptor and the kernel.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_q = self.num_heads * self.max_decode_query_len + layout = cute.make_layout( + (self.max_decode_query_len, self.num_heads, (elems, head_dim // elems)), + stride=(elems, self.max_decode_query_len * elems, (1, block_q * elems)), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + def _sk_layout(self): + """Composed SMEM layout for the pipelined K tiles.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_k = self.BLOCK_K + layout = cute.make_layout( + (1, block_k, (elems, head_dim // elems), self.num_stages), + stride=(0, elems, (1, block_k * elems), block_k * head_dim), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + @cute.jit + def __call__( + self, + gQ: cute.Tensor, # [bs * runtime_decode_query_len, num_heads, head_dim] + gK_cache: cute.Tensor, # [num_pages, page_size, head_dim] + block_table: cute.Tensor, # [bs, max_pages] + score: cute.Tensor, # [num_heads, bs * runtime_decode_query_len, max_pages] + seq_lens: cute.Tensor, # [bs] + stream: CUstream, + ): + num_heads = self.num_heads + head_dim = self.head_dim + MAX_DQL = self.max_decode_query_len + assert num_heads * MAX_DQL <= 32 + + batch = seq_lens.shape[0] + decode_query_len = gQ.shape[0] // batch + grid = (batch, self.split_k, 1) + block = (32 * 5, 1, 1) + + tma_g2s = cpasync.CopyBulkTensorTileG2SOp() + elems = self._swizzle_elems() + + q_tma_atom, q_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gQ, (None, None, elems)), + self._sq_layout(), + (MAX_DQL, num_heads, head_dim), + ) + k_tma_atom, k_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gK_cache, (None, None, elems)), + self._sk_layout(), + (1, self.BLOCK_K, head_dim), + ) + + self.kernel( + q_tma_atom, + q_tma_tensor, + k_tma_atom, + k_tma_tensor, + gK_cache, + block_table, + score, + seq_lens, + decode_query_len, + ).launch(grid=grid, block=block, stream=stream, use_pdl=TRTLLM_ENABLE_PDL) + + @cute.kernel + def kernel( + self, + q_tma_atom: cute.CopyAtom, + q_tma_tensor: cute.Tensor, + k_tma_atom: cute.CopyAtom, + k_tma_tensor: cute.Tensor, + gK_cache: cute.Tensor, + block_table: cute.Tensor, + score: cute.Tensor, + seq_lens: cute.Tensor, + decode_query_len, + ): + batch_id, split_id, _ = cute.arch.block_idx() + _, split_k, _ = cute.arch.grid_dim() + warp_id = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane_id = cute.arch.lane_idx() + + NUM_HEADS = self.num_heads + MAX_DQL = self.max_decode_query_len + BLOCK_Q = NUM_HEADS * MAX_DQL + BLOCK_K = self.BLOCK_K + head_dim = self.head_dim + dtype = self.dtype + MMA_N = 8 + num_stages = self.num_stages + Q_TILES = cute.ceil_div(BLOCK_Q, MMA_N) + EPI_Q = Q_TILES * MMA_N + + sq_layout = self._sq_layout() + sk_layout = self._sk_layout() + + smem = SmemAllocator() + sK = smem.allocate_tensor( + dtype, + sk_layout.outer, + byte_alignment=128, + swizzle=sk_layout.inner, + )[0, None, None, None] + # sQ aliases the first K stage: Q is consumed into registers before the + # first K tile is needed, so the two never overlap in time. + sQ_tma = cute.make_tensor(sK[None, None, 0].iterator, layout=sq_layout.outer) + # TMA sees Q as (query, head, dim) while ldmatrix consumes a flattened Q + # column mode. The target profile keeps the rank-2 view even for + # degenerate shapes such as MAX_DQL == 1. + q_tma_elems = self._swizzle_elems() + sQ = cute.coalesce( + cute.group_modes(sQ_tma, 0, 2), + target_profile=(BLOCK_Q, (q_tma_elems, head_dim // q_tma_elems)), + ) + epi_buffer = smem.allocate_tensor(Float32, cute.make_layout((EPI_Q, 4))) + + tma_full_mbar = smem.allocate_array(Int64, num_stages) + tma_empty_mbar = smem.allocate_array(Int64, num_stages) + + # Barrier init and descriptor prefetch do not depend on the predecessor + # grid, so they are the work PDL can overlap. Every CTA runs them, even + # the ones that own no blocks, since ownership is not known until + # seq_lens is read below. + if warp_id == 0: + with cute.arch.elect_one(): + for i in cutlass.range_constexpr(num_stages): + cute.arch.mbarrier_init(tma_full_mbar + i, 1) + cute.arch.mbarrier_init(tma_empty_mbar + i, 128) + cute.arch.mbarrier_init_fence() + elif warp_id == 1: + cpasync.prefetch_descriptor(q_tma_atom) + cpasync.prefetch_descriptor(k_tma_atom) + cute.arch.sync_threads() + if cutlass.const_expr(self.early_trigger): + griddepcontrol_launch_dependents() + + if cutlass.const_expr(self.prefetch_old): + if warp_id == 4: + with cute.arch.elect_one(): + seqlen0 = seq_lens[batch_id] + num_blocks0 = cute.ceil_div(seqlen0, BLOCK_K) + stable = (seqlen0 - decode_query_len) // BLOCK_K + if stable > num_blocks0: + stable = num_blocks0 + if cutlass.const_expr(self.prefetch_depth > 0): + if stable > split_id + self.prefetch_depth * split_k: + stable = split_id + self.prefetch_depth * split_k + for block_id in range(split_id, stable, split_k): + page = block_table[batch_id, block_id] + _l2_prefetch_bulk( + gK_cache[page, None, None].iterator.toint().to(Int64), + Int32(BLOCK_K * head_dim * (dtype.width // 8)), + ) + + # Without early_trigger the dependent block selector is released as the CTAs exit. + early_q = False + early_k = False + early_page = Int32(0) + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + seqlen_hint = seq_lens[batch_id] + hint_blocks = cute.ceil_div(seqlen_hint, BLOCK_K) + if hint_blocks > block_table.shape[1]: + hint_blocks = block_table.shape[1] + if split_id < hint_blocks: + early_q = True + early_page = block_table[batch_id, split_id] + if cutlass.const_expr(self.prestage_old): + # Stage the first K tile now when every token of the block predates this step. + if split_id < (seqlen_hint - decode_query_len) // BLOCK_K: + early_k = True + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, + BLOCK_K * head_dim * (dtype.width // 8), + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + griddepcontrol_wait() + + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + gQ_early = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar, BLOCK_Q * head_dim * (dtype.width // 8) + ) + simple_tma_copy(q_tma_atom, gQ_early, sQ_tma, tma_full_mbar) + if not early_k: + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, + BLOCK_K * head_dim * (dtype.width // 8), + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + # The length that bounds this grid's work is read after the wait (the + # pre-wait reads above are prefetch / early-issue hints only, clamped to + # the table width). seq_lens is a graph-stable buffer that on_update_kv_lens + # patches every step, and num_blocks bounds an unmasked block_table load + # and an unmasked score store, so a stale length is an out-of-bounds write + # rather than just a wrong answer. + seqlen = seq_lens[batch_id] + num_blocks = cute.ceil_div(seqlen, BLOCK_K) + + if split_id < num_blocks: + if warp_id == 4: + # TMA warp + tma_stage = 0 + tma_parity = 1 + first_block = split_id + + if not early_q: + gQ_tile = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, tma_parity) + with cute.arch.elect_one(): + Q_size = BLOCK_Q * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(tma_full_mbar, Q_size) + # TMA bounds-checks rows when the runtime decode_query_len is + # below MAX_DQL; padded Q columns are masked before the stores. + simple_tma_copy(q_tma_atom, gQ_tile, sQ_tma, tma_full_mbar) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + if early_q: + # The first K tile was issued right after the wait. + first_block = split_id + split_k + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + for block_id in range(first_block, num_blocks, split_k): + page_id = block_table[batch_id, block_id] + gK_tile = k_tma_tensor[page_id, None, None] + k_mbar = tma_full_mbar + tma_stage + + cute.arch.mbarrier_wait(tma_empty_mbar + tma_stage, tma_parity) + with cute.arch.elect_one(): + K_size = BLOCK_K * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(k_mbar, K_size) + simple_tma_copy( + k_tma_atom, + gK_tile, + sK[None, None, tma_stage], + k_mbar, + cache_policy=EVICT_FIRST, + ) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + else: + # MMA warps; each handles K[32, head_dim] @ Q[BLOCK_Q, head_dim].T + sK_warp = cute.local_tile(sK, (32, head_dim, num_stages), (warp_id, 0, 0)) + q_start = seqlen - decode_query_len + + elems = 128 // dtype.width # 16B + MMA_K = 32 * 8 // dtype.width # 32B + + # Pre-compute ldmatrix addresses. + # sK loads a [16 x 16B] tile: + # ((16, (16B, 2), 1), (32 / 16, head_dim / 32B, num_stages)) + # sQ loads an [8 x 32B] tile: + # ((8, (16B, 4)), (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = cute.zipped_divide(sK_warp, (16, cute.make_layout((elems, 2)), 1)) + sQ_ldsm = cute.zipped_divide(sQ, (MMA_N, cute.make_layout((elems, 4)))) + + # sK: (16B, (32 / 16, head_dim / 32B, num_stages)) + # sQ: (16B, (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = sK_ldsm[(lane_id % 16, (None, lane_id // 16), 0), None] + sQ_ldsm = sQ_ldsm[(lane_id % MMA_N, (None, lane_id // 8)), None] + + ldsm_op = warp.LdMatrix8x8x16bOp(num_matrices=4) + ldsm_atom = cute.make_copy_atom(ldsm_op, dtype) + + rQ = cute.make_rmem_tensor( + ((elems // 2, 2), head_dim // (MMA_K * 2), Q_TILES), dtype + ) + rK = cute.make_rmem_tensor((elems, 2, head_dim // MMA_K), dtype) + rC = cute.make_rmem_tensor((4, 2, Q_TILES), Float32) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + for q in cutlass.range_constexpr(Q_TILES): + cute.copy(ldsm_atom, sQ_ldsm[None, (q, None)], rQ[None, None, q]) + cute.arch.mbarrier_arrive(tma_empty_mbar) + + tma_stage = 1 % self.num_stages + tma_parity = 0 + if tma_stage == 0: + tma_parity ^= 1 + + if cutlass.const_expr(dtype is Float8E4M3FN): + rQ_f16 = cute.make_rmem_tensor((4, head_dim // MMA_K, Q_TILES, 2), Float16) + q_lower, q_upper = _fp8_to_f16_mma_fragments(rQ) + rQ_f16[None, None, None, 0].store(q_lower.load()) + rQ_f16[None, None, None, 1].store(q_upper.load()) + + for block_id in range(split_id, num_blocks, split_k): + rC.fill(0.0) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar + tma_stage, tma_parity) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + for k in cutlass.range_constexpr(head_dim // MMA_K): + cute.copy( + ldsm_atom, + sK_ldsm[None, (None, k, tma_stage)], + rK[None, None, k], + ) + for m in cutlass.range_constexpr(2): + if cutlass.const_expr(dtype is Float8E4M3FN): + rK_lower, rK_upper = _fp8_to_f16_mma_fragments(rK[None, m, k]) + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK_lower, + rQ_f16[None, k, n, 0], + rC[None, m, n], + ) + rC[None, m, n] = mma_sync( + rK_upper, + rQ_f16[None, k, n, 1], + rC[None, m, n], + ) + else: + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK[None, m, k], + rQ[(None, k % 2), k // 2, n], + rC[None, m, n], + ) + + cute.arch.mbarrier_arrive(tma_empty_mbar + tma_stage) + + k_start = block_id * BLOCK_K + warp_id * 32 + + for q in cutlass.range_constexpr(Q_TILES): + for i in cutlass.range_constexpr(4): + for j in cutlass.range_constexpr(2): + col = q * 8 + (lane_id % 4) * 2 + j + q_local_pos = col % MAX_DQL + q_pos = q_start + q_local_pos + k_pos = k_start + i * 8 + lane_id // 4 + rC[q * 8 + i * 2 + j] = ( + rC[q * 8 + i * 2 + j] if q_pos >= k_pos else float("-inf") + ) + + for q in cutlass.range_constexpr(Q_TILES): + # Thread-local reduction along the BLOCK_K dim. + rScore = cute.make_rmem_tensor(2, Float32) + rScore.fill(float("-inf")) + for i in cutlass.range_constexpr(4): + rScore[0] = cute.arch.fmax(rScore[0], rC[i * 2 + 0 + q * 8]) + rScore[1] = cute.arch.fmax(rScore[1], rC[i * 2 + 1 + q * 8]) + + # Warp reduction among lanes 0, 4, 8, 12, ... + for i in cutlass.range_constexpr(3): + offset = 4 << i + other0 = cute.arch.shuffle_sync_bfly( + rScore[0], offset=offset, mask=-1, mask_and_clamp=31 + ) + other1 = cute.arch.shuffle_sync_bfly( + rScore[1], offset=offset, mask=-1, mask_and_clamp=31 + ) + rScore[0] = cute.arch.fmax(rScore[0], other0) + rScore[1] = cute.arch.fmax(rScore[1], other1) + + if lane_id * 2 < MMA_N: + epi_buffer[q * MMA_N + lane_id * 2 + 0, warp_id] = rScore[0] + epi_buffer[q * MMA_N + lane_id * 2 + 1, warp_id] = rScore[1] + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + head_id = lane_id // MAX_DQL + q_local_pos = lane_id - head_id * MAX_DQL + valid_q = head_id < NUM_HEADS and q_local_pos < decode_query_len + if lane_id < BLOCK_Q and valid_q: + final_score = epi_buffer[lane_id, 0] + for i in cutlass.range_constexpr(1, 4): + final_score = cute.arch.fmax(final_score, epi_buffer[lane_id, i]) + + t = batch_id * decode_query_len + q_local_pos + score[head_id, t, block_id] = final_score + + tma_stage = (tma_stage + 1) % self.num_stages + if tma_stage == 0: + tma_parity ^= 1 + else: + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + # Early tiles past the request's true length: let them land before the CTA exits. + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.mbarrier_wait(tma_full_mbar + 1 % num_stages, 0) + + +class IndexScoreRunner: + """Compiled ``IndexDecodeScoreKernel`` variants, keyed by (index dtype, index heads, decode query length, head dim, + split_k, early issue). Batch, query-token count, page count and block-table width stay symbolic, so one compile + serves every decode step of its key. ``prefetch_old`` is always on; ``early`` also caps the prefetch at one block + per CTA; ``early_trigger`` is ``INDEX_SCORE_SET`` for every key of the process.""" + + kernel_cache: dict = {} + SUPPORTED_HEAD_DIM = 128 # capacity: fixed: the checkpoint's index head dim + SUPPORTED_PAGE_SIZE = IndexDecodeScoreKernel.BLOCK_K + # BLOCK_Q must fit one warp's worth of epilogue lanes. + MAX_BLOCK_Q = 32 # capacity: fixed: the scorer's query tile + EARLY_PREFETCH_DEPTH = 1 + + @classmethod + def is_supported( + cls, + *, + q_dtype: torch.dtype, + num_heads: int, + head_dim: int, + page_size: int, + max_decode_query_len: int, + ) -> bool: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + from tensorrt_llm._utils import is_sm_100f + + return ( + is_sm_100f() + and q_dtype in TORCH_TO_CUTE_DTYPE + and head_dim == cls.SUPPORTED_HEAD_DIM + and page_size == cls.SUPPORTED_PAGE_SIZE + and num_heads * max_decode_query_len <= cls.MAX_BLOCK_Q + and max_decode_query_len >= 1 + ) + + @classmethod + def _compile( + cls, + q_dtype: torch.dtype, + num_heads: int, + max_decode_query_len: int, + head_dim: int, + split_k: int, + early: bool, + ) -> None: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + + key = (q_dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + if key in cls.kernel_cache: + return + cute_dtype = TORCH_TO_CUTE_DTYPE[q_dtype] + page_size = cls.SUPPORTED_PAGE_SIZE + sym_total_tokens = cute.sym_int() + sym_batch = cute.sym_int() + + # 16-element divisibility on every non-innermost stride is what the TMA descriptors for Q and K assume. + def _sym_stride(): + return cute.sym_int64(divisibility=16) + + q_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (sym_total_tokens, num_heads, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + # The index-K pool may be coalesced with the main K/V cache (per-page stride above page_size * head_dim). + k_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (cute.sym_int(), page_size, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + bt_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch, cute.sym_int()), stride_order=(1, 0) + ) + # The target passes a transposed view of the [heads, blocks, tokens] selector buffer. + score_fake = cute.runtime.make_fake_tensor( + cutlass.Float32, + (num_heads, sym_total_tokens, cute.sym_int()), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + sl_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch,), stride_order=(0,) + ) + fake_stream = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True) + kernel = IndexDecodeScoreKernel( + cute_dtype, + num_heads, + max_decode_query_len, + split_k, + head_dim, + prefetch_old=True, + early_issue=early, + prefetch_depth=cls.EARLY_PREFETCH_DEPTH if early else 0, + prestage_old=early and INDEX_SCORE_PRESTAGE_OLD, + early_trigger=INDEX_SCORE_SET, + ) + if INDEX_SCORE_SET and not cls.kernel_cache: + logger.info( + "[M3 dispatch] SCORE_SET tp8 index scorer: the block selector is released after the scorer's setup" + ) + cls.kernel_cache[key] = cute.compile( + kernel, + q_fake, + k_fake, + bt_fake, + score_fake, + sl_fake, + fake_stream, + options="--enable-tvm-ffi", + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_layer/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_layer/__init__.py new file mode 100644 index 000000000000..467079831e16 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_layer/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_layer/phase_producer.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_layer/phase_producer.py new file mode 100644 index 000000000000..02f3f91b7ed8 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_layer/phase_producer.py @@ -0,0 +1,489 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode producer as a phase of the TP8 per-layer kernel (m3_layer): q / k / index-q / index-k RMSNorm + NeoX +partial RoPE, NVFP4 K / V and FP8 index-K cache insert, FP8 q / index-q out. + +Bit for bit ``torch.ops.trtllm.minimax_m3_producer_pw`` (cpp/tensorrt_llm/kernels/minimaxM3/producerPw.cu, +m3_producer_kernel<8, *>) at the TP8 geometry: 8 query heads, 1 KV head, 1 index head, head_dim 128, rotary_dim 64, +page 128. One work item = one decode token; the CTA's warp w (< 12) owns packed head w of it, lane l holds elements +[4l, 4l + 4) exactly as the C++ kernel, so every reduction (the RMS sum of squares, the RoPE pair exchange, the +NVFP4 block max) runs the same shuffle tree in the same order. Every floating-point operation is written as its PTX +instruction (mul.rn / add.rn / fma.rn / rsqrt.approx / rcp.approx.ftz / cvt) so the DSL compiler cannot contract +or reassociate differently from the library build (--use_fast_math). The places where that build contracted are +marked CONTRACT and follow the shipped SASS (msa/results/sassdump-*). + +Packed row layout per token: [8 q | k | v | index_q | index_k] x 128 BF16 (the QKV GEMV output rounded to BF16). +""" + +from __future__ import annotations + +import cutlass +import cutlass.cute as cute +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op + +HEADS_Q = 8 # capacity: fixed: the tp8 query heads per rank +PACKED_HEADS = HEADS_Q + 4 # q x 8, k, v, index_q, index_k +HEAD_DIM = 128 +ROTARY_DIM = 64 # capacity: fixed: the checkpoint's rotary dim +PAGE = 128 +ELEMS = HEAD_DIM // 32 # capacity: fixed: elements per lane (per lane) +K_HEAD, V_HEAD, IQ_HEAD, IK_HEAD = HEADS_Q, HEADS_Q + 1, HEADS_Q + 2, HEADS_Q + 3 + + +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +# ---- exact float arithmetic (no contraction) --------------------------------------------------------------------- +@dsl_user_op +def f_mul(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_add(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_fma(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_max(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "max.f32 $0, $1, $2;", + "=f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_abs(a, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "abs.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def f_neg(a, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "neg.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def f_rsqrt(a, *, loc=None, ip=None): + # NOT ftz on this target: the sm_103 library build of producerPw.cu runs rsqrt.approx.f32 with denormal scaling + # (MSA's SASS read; edge row denormal_rms with eps 0, differs under .ftz). The sm_107 copy keeps .ftz. + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rsqrt.approx.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def f_rcp(a, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def bf16_bits_to_f32(bits16, *, loc=None, ip=None): + """The float value of a BF16 given in the low 16 bits of an Int32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(bits16)], + "{\n\t.reg .b32 t;\n\tshl.b32 t, $1, 16;\n\tmov.b32 $0, t;\n\t}", + "=f,r", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def round_bf16(a, *, loc=None, ip=None): + """__bfloat162float(__float2bfloat16_rn(a)).""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a)], + "{\n\t.reg .b16 h;\n\t.reg .b32 t;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b32 t, {0, h};\n\tmov.b32 $0, t;\n\t}", + "=f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def e4m3x4_word(e0, e1, e2, e3, *, loc=None, ip=None): + """store_fp8x4: bytes [e0, e1, e2, e3] (cvt.rn.satfinite.e4m3x2.f32 puts its first source in the high byte).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(e0), cutlass.Float32(e1), cutlass.Float32(e2), cutlass.Float32(e3)], + "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}", + "=r,f,f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def e4m3_byte_and_value(x, *, loc=None, ip=None): + """(byte, float value) of __nv_fp8_e4m3(x): the encoded scale and static_cast of it.""" + byte = cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n\t.reg .b16 p;\n\t.reg .b32 w;\n\tcvt.rn.satfinite.e4m3x2.f32 p, 0f00000000, $1;\n\t" + "cvt.u32.u16 w, p;\n\tand.b32 $0, w, 0xFF;\n\t}", + "=r,f", + loc=loc, + ip=ip, + ) + ) + value = cutlass.Float32( + _asm( + _T.f32(), + [byte], + "{\n\t.reg .b16 p;\n\t.reg .b32 h2;\n\t.reg .b16 h;\n\tcvt.u16.u32 p, $1;\n\t" + "cvt.rn.f16x2.e4m3x2 h2, p;\n\tmov.b32 {h, _}, h2;\n\tcvt.f32.f16 $0, h;\n\t}", + "=f,r", + loc=loc, + ip=ip, + ) + ) + return byte, value + + +@dsl_user_op +def e2m1x4_halfword(q0, q1, q2, q3, *, loc=None, ip=None): + """The low 16 bits of fp32x8_to_e2m1 for q[0..3] (q[4..7] = 0): byte0 = (q1, q0), byte1 = (q3, q2).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(q0), cutlass.Float32(q1), cutlass.Float32(q2), cutlass.Float32(q3)], + "{\n\t.reg .b8 b0, b1;\n\t.reg .b32 w;\n\tcvt.rn.satfinite.e2m1x2.f32 b0, $2, $1;\n\t" + "cvt.rn.satfinite.e2m1x2.f32 b1, $4, $3;\n\tmov.b32 w, {b0, b1, 0, 0};\n\tmov.b32 $0, w;\n\t}", + "=r,f,f,f,f", + loc=loc, + ip=ip, + ) + ) + + +# ---- global memory ----------------------------------------------------------------------------------------------- +@dsl_user_op +def ld_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm(_T.i64(), [cutlass.Int64(addr)], "ld.global.u64 $0, [$1];", "=l,l", True, loc, ip) + ) + + +@dsl_user_op +def ld_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm(_T.i32(), [cutlass.Int64(addr)], "ld.global.s32 $0, [$1];", "=r,l", True, loc, ip) + ) + + +@dsl_user_op +def ld_f32(addr, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Int64(addr)], "ld.global.f32 $0, [$1];", "=f,l", True, loc, ip) + ) + + +@dsl_user_op +def st_u32(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(v)], + "st.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def st_u16(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(v)], + "{\n\t.reg .b16 h;\n\tcvt.u16.u32 h, $1;\n\tst.global.u16 [$0], h;\n\t}", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def st_u8(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(v)], + "{\n\t.reg .b16 h;\n\tcvt.u16.u32 h, $1;\n\tst.global.u8 [$0], h;\n\t}", + "l,r", + True, + loc, + ip, + ) + + +# ---- CONTRACT points (follow the shipped SASS; see the module docstring) ------------------------------------------- +def sumsq_pair(acc, x, y): + """sumSquares += x * x + y * y (C++: one pair of the packed row). CONTRACT: fma(y, y, x * x) then add.""" + return f_add(acc, f_fma(y, y, f_mul(x, x))) + + +def rms_reciprocal(sum_squares, eps): + """rsqrtf(sumSquares / 128 + eps). CONTRACT: the / 128 is an exact * 2^-7; fused with + eps into one fma.""" + return f_rsqrt(f_fma(sum_squares, cutlass.Float32(1.0 / HEAD_DIM), eps)) + + +class ProducerArgs: + """Raw device addresses (int64) + strides, as the C++ Params (producerPw.h) carries them.""" + + def __init__( + self, + packed, + q_out, + index_q_out, + index_k, + kv_data, + kv_scale, + out_cache_loc, + kv_quant_scale, + position_ids, + rotary_cos_sin, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + eps, + num_pages, + index_page_stride, + index_token_stride, + data_page_stride, + data_plane_stride, + data_token_stride, + scale_page_stride, + scale_plane_stride, + ): + self.__dict__.update(locals()) + del self.__dict__["self"] + + +@cute.jit +def phase_producer(tok, a: ProducerArgs): + """Producer work item ``tok``: warp w (< 12) of the calling CTA does packed head w of decode token tok. + The caller guarantees tok < rows and that warps >= 12 skip. Stores only (no CTA barrier inside).""" + tid = cute.arch.thread_idx()[0] + lane = tid % 32 + head = tid // 32 + in_off = (cutlass.Int64(tok) * PACKED_HEADS + cutlass.Int64(head)) * HEAD_DIM + cutlass.Int64( + lane * ELEMS + ) + pk = ld_u64(a.packed + in_off * 2) + x0 = bf16_bits_to_f32(cutlass.Int32(pk & cutlass.Int64(0xFFFF))) + x1 = bf16_bits_to_f32(cutlass.Int32((pk >> cutlass.Int64(16)) & cutlass.Int64(0xFFFF))) + x2 = bf16_bits_to_f32(cutlass.Int32((pk >> cutlass.Int64(32)) & cutlass.Int64(0xFFFF))) + x3 = bf16_bits_to_f32(cutlass.Int32((pk >> cutlass.Int64(48)) & cutlass.Int64(0xFFFF))) + producer_head(tok, head, lane, x0, x1, x2, x3, a) + + +@cute.jit +def producer_head(tok, head, lane, x0, x1, x2, x3, a: ProducerArgs): + """Everything after the packed-row load for packed head ``head`` (warp-uniform; dynamic Int32 or Constexpr) of + decode token ``tok``, on the calling warp: lane ``lane`` holds the head's elements [4 lane, 4 lane + 4) as the + float values x0..x3 of the BF16 packed row (the QKV GEMV output rounded to BF16). Norm weights, KV quant scale, + position and RoPE loads, the RMS tree, RoPE, and the q / index-q / index-K / K-V NVFP4 stores. Shared by + phase_producer and the QKV GEMV owner epilogue (MK-A), so the producer math has one implementation. + + Data flow is branch-free (every load is in bounds for every head: V reads the q norm weight, a non-rotating lane + reads its token's RoPE row, and the results are then discarded with select); only the stores branch, on + warp-uniform conditions, so every shuffle runs on the full warp as in the C++ kernel.""" + l4 = cutlass.Int64(lane * ELEMS) + is_v = head == V_HEAD + zero = cutlass.Float32(0.0) + + # Norm weights (BF16): q / k / index-q / index-k weight by head (V: q's, unused). + w_base = cutlass.select_( + head == K_HEAD, + a.k_weight, + cutlass.select_( + head == IQ_HEAD, + a.index_q_weight, + cutlass.select_(head == IK_HEAD, a.index_k_weight, a.q_weight), + ), + ) + ww = ld_u64(w_base + l4 * 2) + w0 = bf16_bits_to_f32(cutlass.Int32(ww & cutlass.Int64(0xFFFF))) + w1 = bf16_bits_to_f32(cutlass.Int32((ww >> cutlass.Int64(16)) & cutlass.Int64(0xFFFF))) + w2 = bf16_bits_to_f32(cutlass.Int32((ww >> cutlass.Int64(32)) & cutlass.Int64(0xFFFF))) + w3 = bf16_bits_to_f32(cutlass.Int32((ww >> cutlass.Int64(48)) & cutlass.Int64(0xFFFF))) + quant = ld_f32(a.kv_quant_scale + cutlass.select_(is_v, cutlass.Int64(8), cutlass.Int64(4))) + + # Position id, RoPE row. + pos = ld_s32(a.position_ids + cutlass.Int64(tok) * 4) + row = cutlass.Int64(pos) * ROTARY_DIM + coef = cutlass.Int64((lane * ELEMS) % (ROTARY_DIM // 2)) + c0 = ld_f32(a.rotary_cos_sin + (row + coef) * 4) + c1 = ld_f32(a.rotary_cos_sin + (row + coef + 1) * 4) + c2 = ld_f32(a.rotary_cos_sin + (row + coef + 2) * 4) + c3 = ld_f32(a.rotary_cos_sin + (row + coef + 3) * 4) + s0 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef) * 4) + s1 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef + 1) * 4) + s2 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef + 2) * 4) + s3 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef + 3) * 4) + + # RMSNorm (all heads compute it; V keeps x). + ss = sumsq_pair(zero, x0, x1) + ss = sumsq_pair(ss, x2, x3) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=16)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=8)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=4)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=2)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=1)) + r = rms_reciprocal(ss, a.eps) + one = cutlass.Float32(1.0) + n0 = f_mul(x0, f_mul(r, f_add(one, w0))) + n1 = f_mul(x1, f_mul(r, f_add(one, w1))) + n2 = f_mul(x2, f_mul(r, f_add(one, w2))) + n3 = f_mul(x3, f_mul(r, f_add(one, w3))) + # NeoX partial RoPE: lane l's dim 4l + i pairs with lane l ^ 8 (dim +-32) within the first 64 dims. + pair_off = (ROTARY_DIM // 2) // ELEMS + p0 = cute.arch.shuffle_sync_bfly(n0, offset=pair_off) + p1 = cute.arch.shuffle_sync_bfly(n1, offset=pair_off) + p2 = cute.arch.shuffle_sync_bfly(n2, offset=pair_off) + p3 = cute.arch.shuffle_sync_bfly(n3, offset=pair_off) + rot = lane * ELEMS < ROTARY_DIM + low = lane * ELEMS < ROTARY_DIM // 2 + p0 = cutlass.select_(low, f_neg(p0), p0) + p1 = cutlass.select_(low, f_neg(p1), p1) + p2 = cutlass.select_(low, f_neg(p2), p2) + p3 = cutlass.select_(low, f_neg(p3), p3) + r0 = cutlass.select_(rot, f_fma(p0, s0, f_mul(n0, c0)), n0) + r1 = cutlass.select_(rot, f_fma(p1, s1, f_mul(n1, c1)), n1) + r2 = cutlass.select_(rot, f_fma(p2, s2, f_mul(n2, c2)), n2) + r3 = cutlass.select_(rot, f_fma(p3, s3, f_mul(n3, c3)), n3) + e0 = cutlass.select_(is_v, x0, r0) + e1 = cutlass.select_(is_v, x1, r1) + e2 = cutlass.select_(is_v, x2, r2) + e3 = cutlass.select_(is_v, x3, r3) + + if head < HEADS_Q: + st_u32( + a.q_out + (cutlass.Int64(tok) * HEADS_Q + cutlass.Int64(head)) * HEAD_DIM + l4, + e4m3x4_word(e0, e1, e2, e3), + ) + if head == IQ_HEAD: + st_u32(a.index_q_out + cutlass.Int64(tok) * HEAD_DIM + l4, e4m3x4_word(e0, e1, e2, e3)) + if head >= K_HEAD: + if head != IQ_HEAD: + slot = ld_s32(a.out_cache_loc + cutlass.Int64(tok) * 4) + page = slot // PAGE + if slot >= 0: + if page < a.num_pages: + within = slot & (PAGE - 1) + if head == IK_HEAD: + st_u32( + a.index_k + + cutlass.Int64(page) * a.index_page_stride + + cutlass.Int64(within) * a.index_token_stride + + l4, + e4m3x4_word(e0, e1, e2, e3), + ) + else: + plane = cutlass.select_(is_v, cutlass.Int64(1), cutlass.Int64(0)) + col = lane // 4 + sw = cutlass.select_( + is_v, + cutlass.Int64((within // 4) * 32 + col * 4 + within % 4), + cutlass.Int64(within * 8 + col), + ) + s_off = ( + cutlass.Int64(page) * a.scale_page_stride + + plane * a.scale_plane_stride + + sw + ) + # quantize_fp4x4: BF16-rounded values, 16-element block max over the 4-lane group. + q0 = round_bf16(e0) + q1 = round_bf16(e1) + q2 = round_bf16(e2) + q3 = round_bf16(e3) + lmax = f_max( + f_max(f_max(f_max(zero, f_abs(q0)), f_abs(q1)), f_abs(q2)), f_abs(q3) + ) + lmax = f_max(lmax, cute.arch.shuffle_sync_bfly(lmax, offset=1)) + lmax = f_max(lmax, cute.arch.shuffle_sync_bfly(lmax, offset=2)) + sbyte, sval = e4m3_byte_and_value( + f_mul(quant, f_mul(lmax, f_rcp(cutlass.Float32(6.0)))) + ) + if lane % 4 == 0: + st_u8(a.kv_scale + s_off, sbyte) + outs = cutlass.select_(lmax != zero, f_rcp(f_mul(sval, f_rcp(quant))), zero) + hw = e2m1x4_halfword( + f_mul(q0, outs), f_mul(q1, outs), f_mul(q2, outs), f_mul(q3, outs) + ) + d_off = ( + cutlass.Int64(page) * a.data_page_stride + + plane * a.data_plane_stride + + cutlass.Int64(within) * a.data_token_stride + + cutlass.Int64(lane * 2) + ) + st_u16(a.kv_data + d_off, hw) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_mc_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_mc_allreduce.py new file mode 100644 index 000000000000..d5266a50dd3c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_mc_allreduce.py @@ -0,0 +1,376 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Low-latency TP all-reduce + residual + RMSNorm for decode (M <= 8 tokens) over a multicast mapping. + + partial = bf16(x + x2) (optional second input, e.g. shared + routed experts) + updated = bf16(residual + sum_r partial_r) (ranks summed in FP32, rank order) + normed = bf16(updated * rsqrt(mean(updated^2) + eps) * w) + +One cluster of CLUSTER CTAs per token; CTA c owns columns [c * H / CLUSTER, ...), one 16-byte chunk (8 BF16) +per thread. Each CTA pushes its slice of this rank's partial through the multicast mapping (one store lands in +every rank's buffer), releases dependents, polls its own buffer until all ranks' slices arrived, sums them, adds +the residual and reduces the RMS statistic across the cluster through DSMEM. + +Exchange protocol as in the K3 sandwich: int32 words, 0x80000000 = empty (pushed -0.0 halves become +0.0), each +CTA keeps a call counter whose parity selects the buffer half, readers re-arm the words they read. All ranks of +the TP group must issue the same calls in the same stream order. + +The workspace holds two exchange sets. With prepushed 3 two producers pushed this rank's x (set A) and x2 (set B), +e.g. the routed experts' finalize and the shared expert's down projection, and the kernel forms bf16(x + x2) per +rank from the two slots, exactly as the unpushed path does. + +Producers that push (the o_proj / shared-expert GEMV ``ar`` tuple, the routed-expert engine's ``arp`` descriptor) are +pointed at a bucket's exchange by its ``mc`` address, ``flags_for(8)`` counters, ``rank``, ``world_size`` and +``max_tokens``; ``get_workspace`` and ``_bucket`` select it. + +The kernel is ``torch.ops.trtllm.minimax_m3_mc_allreduce_norm``, reached through the ``ops/comm`` entries +``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm``; this module owns the workspaces, the bucket and +epilogue-option choice and the MXFP8 publication. +""" + +from dataclasses import dataclass +from typing import Any + +import torch + +from ..ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm as _allreduce_norm_op +from ..ops.comm.m3_mc_finalize_allreduce_norm import ( + m3_mc_finalize_allreduce_norm as _finalize_allreduce_norm_op, +) + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +BUCKETS = (8, 16, 48) # token capacity of each workspace (grid = bucket * cluster CTAs) +EMPTY_WORD = -2147483648 + + +def max_rows() -> int: + """The most rows one boundary all-reduce call (plain or MoE-finalize) serves: the decode plans and the + MC_ALLREDUCE / MC_FINALIZE ranges read this, so a wider bucket or a row split here widens them with no change in + the modeling.""" + return MAX_TOKENS + + +# Attention / MoE boundary epilogue options (each bit-identical): 1 residual + norm weight loaded at entry (before +# the poll); 2 cluster RMS exchange by st.async + mbarrier complete_tx instead of cluster.sync; 4 call parity rotated +# right after the poll (no trailing __syncthreads); 8 max-smem carveout so the next kernel's CTAs co-reside early. +# The 8-token bucket, by token count 1..8 (GB300 tuning, the same at TP4 and TP8). +AR_OPTS_BY_M = (7, 7, 15, 15, 15, 15, 7, 7) +# The 16- and 48-token buckets. +AR_OPTS_BY_BUCKET = {16: 15, 48: 15} +# The MoE-finalize all-reduce, per bucket. +AR_FIN_OPTS_BY_BUCKET = {8: 7, 16: 15, 48: 15} + + +# Option 2 is also the safe ordering of the cluster RMS exchange: its stores into peer CTAs' shared memory follow a +# cluster barrier arrive / wait pair. Without it they run before any cluster barrier, when a peer need not have +# started, so every call keeps it. +_OPT_CLUSTER_BARRIER = 2 + + +def _ar_opts(m: int, bucket: int, finalize: bool = False) -> int: + """Epilogue options for an m-row call on the given bucket; every option is bit-identical. A bucket missing from + the tables gets 0, and every result keeps _OPT_CLUSTER_BARRIER.""" + if finalize: + opts = AR_FIN_OPTS_BY_BUCKET.get(bucket, 0) + elif bucket == 8: + opts = AR_OPTS_BY_M[m - 1] + else: + opts = AR_OPTS_BY_BUCKET.get(bucket, 0) + return opts | _OPT_CLUSTER_BARRIER + + +@dataclass(eq=False) +class McAllReduceWorkspace: + """Two halves of [max_tokens][world][H] BF16 (int32 words) per rank behind one multicast mapping, plus one call + counter per CTA per cluster size. Create collectively and eagerly on every TP rank.""" + + uc: torch.Tensor + mc: torch.Tensor + flags: dict + rank: int + world_size: int + hidden: int + handle: Any + comm: Any + max_tokens: int = 8 + + @classmethod + def create(cls, mapping, hidden: int, max_tokens: int = 8) -> "McAllReduceWorkspace": + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + words = 4 * max_tokens * mapping.tp_size * (hidden // 2) # two exchange sets of two halves + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "McAllReduceWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 4, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("McAllReduceWorkspace.create failed on another rank") + return cls( + uc=uc, + mc=mc, + flags={}, + rank=mapping.tp_rank, + world_size=mapping.tp_size, + hidden=hidden, + handle=handle, + comm=comm, + max_tokens=max_tokens, + ) + + def flags_for(self, cluster: int): + """This cluster size's per-CTA call counters, or None when they do not exist yet during a capture. + get_workspace creates the cluster-8 counters with the workspace, so for cluster 8 (the only size built) + that None never happens and cannot differ between ranks.""" + flags = self.flags.get(cluster) + if flags is None: + if torch.cuda.is_current_stream_capturing(): + return None + flags = torch.zeros(self.max_tokens * cluster, dtype=torch.int32, device=self.uc.device) + self.flags[cluster] = flags + return flags + + +class McBuckets: + """One workspace per token bucket (8 / 16 / 48): calls of different sizes never share exchange slots.""" + + def __init__(self, workspaces): + self.workspaces = workspaces + self.hidden = workspaces[0].hidden + + def for_tokens(self, m: int): + for ws in self.workspaces: + if m <= ws.max_tokens: + return ws + return None + + +def _bucket(workspace, m): + return workspace.for_tokens(m) if isinstance(workspace, McBuckets) else workspace + + +def decline_reason(x, residual, workspace, cluster=8, x2=None) -> str: + """Why ``m3_mc_allreduce_norm`` returned None for these operands (for the dispatch's miss message).""" + if workspace is None: + return "no multicast workspace: it is created collectively on the first eager decode step" + bucket = _bucket(workspace, x.shape[0]) + if bucket is None: + return f"no workspace bucket holds {x.shape[0]} tokens (buckets {BUCKETS})" + if bucket.flags.get(cluster) is None: + return f"no {cluster}-CTA call counters for the {bucket.max_tokens}-token bucket" + return ( + f"operands outside the kernel: x {tuple(x.shape)} {x.dtype} contiguous={x.is_contiguous()}, " + f"residual contiguous={residual.is_contiguous()}, hidden {bucket.hidden}" + + ("" if x2 is None else f", x2 {tuple(x2.shape)} contiguous={x2.is_contiguous()}") + ) + + +def m3_mc_allreduce_norm( + x, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + x2=None, + quant=False, + prepushed=0, +): + """Return (normed, updated_residual) or None when unsupported. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM. + + A None makes the caller take another collective, so it must be the same on every TP rank. It is: it depends only + on the shapes, dtypes and layouts of the operands (the same code builds them on every rank) and on whether the + workspace's flags exist, which ``get_workspace`` creates collectively, before any capture, for cluster 8.""" + workspace = _bucket(workspace, x.shape[0]) + if workspace is None: + return None + if ( + x.dim() != 2 + or x.shape[0] > MAX_TOKENS + or x.dtype != torch.bfloat16 + or x.shape[1] != workspace.hidden + or not x.is_contiguous() + or not residual.is_contiguous() + or (x2 is not None and (not x2.is_contiguous() or x2.shape != x.shape)) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + outs = _allreduce_norm_op( + x, + x2, + residual, + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and x.shape[1] % 128 == 0), + int(prepushed), + _ar_opts(x.shape[0], workspace.max_tokens), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +def m3_mc_finalize_allreduce_norm( + fc2, + perm, + scales, + shared, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + quant=False, +): + """MoE finalize (sum_k scale * fc2[perm] + shared) + all-reduce + residual + RMSNorm in one kernel. + Returns (normed, updated_residual) or None when unsupported. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM.""" + m = residual.shape[0] + workspace = _bucket(workspace, m) + if workspace is None: + return None + if ( + residual.dim() != 2 + or m > MAX_TOKENS + or residual.shape[1] != workspace.hidden + or fc2.dtype != torch.bfloat16 + or not fc2.is_contiguous() + or fc2.dim() != 2 + or perm.dtype != torch.int32 + or perm.dim() != 2 + or perm.shape[0] != m + or not perm.is_contiguous() + or scales.dtype not in (torch.float32, torch.bfloat16) + or not scales.is_contiguous() + or (shared is not None and (shared.shape != residual.shape or not shared.is_contiguous())) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + outs = _finalize_allreduce_norm_op( + fc2, + perm, + scales, + shared, + residual.contiguous(), + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and residual.shape[1] % 128 == 0), + _ar_opts(m, workspace.max_tokens, finalize=True), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +_WORKSPACE: dict = {} +_AGREE_COMMS: dict = {} + + +def _tp_comm(mapping): + """A communicator over mapping's TP group, created once (collective over that group only).""" + key = tuple(mapping.tp_group) + comm = _AGREE_COMMS.get(key) + if comm is None: + from tensorrt_llm._utils import mpi_comm, mpi_disabled + + if mpi_disabled(): + comm = mapping.tp_group_pg + else: + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + _AGREE_COMMS[key] = comm + return comm + + +def tp_agree(mapping, ok: bool) -> bool: + """Whether ``ok`` holds on every rank of mapping's TP group (one blocking host collective). + + For a fast path whose availability is decided from rank-local state (a kernel build that can fail on one rank): + every rank must take the same path, so that they issue the same collectives with the same exchange protocol and + the same kernels. Call it once per decision, at build time, at the same point on every rank, outside CUDA-graph + capture; never per step. With one rank, or attention DP (whose ranks run different batches), it returns ``ok``. + """ + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return ok + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("tp_agree must run outside CUDA-graph capture") + from tensorrt_llm._torch.distributed.ops import _mnnvl_workspace_all_succeeded + + return _mnnvl_workspace_all_succeeded(_tp_comm(mapping), bool(ok)) + + +def tp_agree_built(mapping, built, what: str): + """``built`` when it is not None on every TP rank, else None on every rank (see ``tp_agree``).""" + if tp_agree(mapping, built is not None) or built is None: + return built + from tensorrt_llm.logger import logger + + logger.warning(f"{what}: built here but not on another TP rank; every rank falls back") + return None + + +def get_workspace(mapping, hidden: int): + """Process-wide workspace (created collectively on first eager use); None while capturing before creation. + + The first eager call for a hidden size (the warmup forward, before any CUDA-graph capture) creates one + McAllReduceWorkspace per bucket in BUCKETS on every TP rank (MPI collective over the TP group) plus each + bucket's 8-CTA-cluster call counters; later calls, captured or not, return the same McBuckets. The workspaces + live until process exit. + """ + ws = _WORKSPACE.get(hidden) + if ws is None: + if torch.cuda.is_current_stream_capturing(): + return None + wss = [McAllReduceWorkspace.create(mapping, hidden, b) for b in BUCKETS] + for w in wss: + w.flags_for(8) + ws = McBuckets(wss) + _WORKSPACE[hidden] = ws + return ws diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/__init__.py new file mode 100644 index 000000000000..e361dc614231 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/__init__.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 MoE layer of the TP8 target for low-batch decode as a CuTe DSL kernel: ``m3_moe`` for 1..8 tokens +(op.py) at the TP8 routed and shared intermediate of 384 per rank.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/m3_moe_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/m3_moe_kernel.py new file mode 100644 index 000000000000..97cef6b7ea10 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/m3_moe_kernel.py @@ -0,0 +1,3673 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP8 implementation.""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +_CFG = globals().get("M3_CONFIG") or {} + + +def _cfg(key: str, default): + return type(default)(_CFG.get(key, default)) + + +H = 6144 +I_TP = _cfg("i_tp", 768) +TWO_I = 2 * I_TP +assert I_TP % 128 == 0 and 128 <= I_TP <= 768, I_TP +E_LOCAL = 128 +TOP_K = 4 +ROUTE_SCALE = 2.0 +SUM_EPS = 1e-20 +M_MAX = _cfg("m_max", 1) +NCTA = _cfg("num_ctas", 152) +FUSED_ROUTER = bool(_cfg("fused_router", 0)) +NROUTE = E_LOCAL if FUSED_ROUTER else 0 +NGEMV = (NROUTE + NCTA - 1) // NCTA +TIMING = bool(_cfg("timing", 0)) +HANG_DBG = bool(_cfg("hang_dbg", 0)) +XVDUMP = bool(_cfg("xvdump", 0)) +YDUMP = bool(_cfg("ydump", 0)) +LAG = _cfg("lag", 2) +G_CAP = TOP_K * M_MAX +NG_CAP = M_MAX +XW = _cfg("xw", 5 if M_MAX > 4 else 4 if M_MAX > 2 else 1) +THREADS = 256 + 32 * (XW - 1) +GEMV_THREADS = 256 +SHARED = bool(_cfg("shared", 0)) +LATE_TRIGGER = bool(_cfg("late_trigger", 1)) +AR_PRE = bool(_cfg("ar_pre", 1)) +TAIL_TRIGGER = bool(_cfg("tail_trigger", 0)) +assert True, "early_ar and tail_trigger are alternative release points" +EARLY_EPOCH = TAIL_TRIGGER +XSTAGE = bool(_cfg("xstage", 1)) and bool(_cfg("fused_router", 0)) +B1WIDE = bool(_cfg("b1wide", 0)) +B1W_BAR = 10 +NSH = 24 if SHARED else 0 +SH_I = _cfg("sh_i", 768) +SH_ROWS = SH_I // 24 +SH_SPLIT = 128 // (2 * SH_ROWS) +SH_NB = 8 * SH_SPLIT +SH_KB = H // 32 +SH_DKB = SH_I // 32 +SH_DKT = SH_I // 128 +SH_GT = SH_KB // 4 // SH_SPLIT +SH_STAGES_GU = SH_GT +SH_STAGES = SH_GT + 2 * SH_DKT +SHX_BAR = 6 +SH_PRETILE = bool(_cfg("sh_pretile", 1 if M_MAX <= 2 or SH_SPLIT != 2 else 0)) +if SHARED: + assert SH_I in (384, 768), SH_I + assert SH_PRETILE or SH_SPLIT == 2, ( + "the un-pretiled shared stream is the 32-row (TP4) layout only" + ) +SH_STAGE_BYTES = 16384 + 512 +MMA_M = 128 +KT_BYTES = 128 +INC_K = 2 +NUM_KB = 4 +N1 = 16 if M_MAX <= 4 else 32 +QS = N1 // 4 +R1 = 32 +C1 = 4 +U1 = TWO_I // R1 +K1_TILES = H // 256 +S1 = K1_TILES // C1 +NUM_BYTES_A = MMA_M * KT_BYTES +NUM_BYTES_BOX1 = R1 * KT_BYTES +SF1_KA = H // 64 +SF1_RB = TWO_I // 128 +NUM_BYTES_SFA_RAW = 16 * 512 +NUM_WORDS_SFA_RAW = NUM_BYTES_SFA_RAW // 4 +NUM_BYTES_SFA = NUM_KB * 512 +NUM_WORDS_SFA = NUM_BYTES_SFA // 4 +NUM_BYTES_B1 = N1 * KT_BYTES +NUM_SFB1_ATOMS = S1 * NUM_KB +R2 = 32 +NB2 = H // R2 +KT2 = (I_TP + 255) // 256 +KB2_LAST = (I_TP - 256 * (KT2 - 1)) // 64 + + +def _nkb2(t2: int) -> int: + """K-64 MMA instructions (= scale atoms) of FC2 k-tile t2.""" + return NUM_KB if t2 < KT2 - 1 else KB2_LAST + + +SF2_KA = I_TP // 64 +SF2_RB = H // 128 +N2 = max(8, (4 * M_MAX + 7) // 8 * 8) +NUM_BYTES_B2 = N2 * KT_BYTES +NB2_CAP = _cfg("nb2_cap", 2) +UNITS_ROW = TWO_I // R1 +H_ROW = UNITS_ROW * 16 +HB_REPL = _cfg("hb_repl", 1) +assert 1 <= HB_REPL <= 32, HB_REPL +HB_STRIDE = (G_CAP * M_MAX * UNITS_ROW * 4 + 255) // 256 * 256 + 32 +PARTIAL_EMPTY = -1 +EPI_THREADS = 128 +EPI_BAR = 1 +ROUTE_BAR = 2 +GEMV_BAR = 3 +B2_BAR = 5 +TD_STRIDE = 64 +TOPK_BAR = 4 +B1_AFTER_TOPK = bool(_cfg("b1_after_topk", 0)) +TOPK2_BAR = 7 +EARLY0 = bool(_cfg("early0", 1)) +EARLY0_N = _cfg("early0_n", 99) +RW_EXTRA = max(0, min(M_MAX - 4, THREADS // 32 - 8)) if bool(_cfg("rw_extra", 1)) else 0 +NRW0 = min(4, M_MAX) if bool(_cfg("rw_trim", 1)) else 4 +NRW = NRW0 + RW_EXTRA +LG_REPL = _cfg("lg_repl", 32 if M_MAX > 4 else 128) +GEMV_SMEM = bool(_cfg("gemv_smem", 1)) +LG_STRIDE = M_MAX * E_LOCAL + 32 +POLL_NS = _cfg("poll_ns", 150) +B2_POLL_NS = _cfg("b2_poll_ns", 200) +B2IMG = bool(_cfg("b2img", 0)) and bool(_cfg("sfa_cp", 0)) +IMG_B2 = KT2 * NUM_BYTES_B2 +IMG_SF = KT2 * NUM_KB * 512 +IMG_BYTES = IMG_B2 + IMG_SF +CNT_PER_CALL = 4 * (TWO_I // R1) +B2IMG_NS = _cfg("b2img_ns", 16) +B2_DLY = _cfg("b2_dly", 0) +W_WARP = 4 +X_WARP = 5 +S_WARP = 6 +MMA_WARP = 7 +E0_WARP = MMA_WARP +E0_BAR = 8 +E1_BAR = 9 +E0_OFFLOAD = EARLY0 and NRW0 <= 3 and bool(_cfg("e0_off", 1)) +EVICT_FIRST = 1364590687093260288 +_LOG2E = 1.4426950408889634 +LD_POW2 = 1 if M_MAX == 1 else 2 if M_MAX == 2 else 4 if M_MAX <= 4 else 8 +_FIXED_SMEM = ( + S1 * NUM_BYTES_B1 + + 2 * KT2 * NUM_BYTES_B2 + + 2 * KT2 * NUM_KB * 512 + + NB2_CAP * M_MAX * TOP_K * R2 * 4 + + 2 * M_MAX * C1 * R1 * 4 + + (M_MAX * (H // 16) if FUSED_ROUTER else 0) + + 6 * 1024 +) +SFA_CP = bool(_cfg("sfa_cp", 0)) +_RAW_STAGE = 0 if SFA_CP else NUM_BYTES_SFA_RAW +_STAGE_SMEM = NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA +_STAGES_FIT = (227 * 1024 - _FIXED_SMEM - (128 if SHARED else 0)) // _STAGE_SMEM +ACC1_COL = 0 +ACC2_COL = 2 * N1 +SFB1_COL = ACC2_COL + 2 * N2 +SFB2_COL = SFB1_COL + NUM_SFB1_ATOMS * 4 +SFA1_COL = SFB2_COL + 2 * KT2 * NUM_KB * 4 +STAGES = min(_cfg("stages", 12 if SFA_CP else 8), _STAGES_FIT, (512 - SFA1_COL) // 16) +TMEM_COLS = 512 +assert 1 <= M_MAX <= 8, M_MAX +assert STAGES >= 3, (STAGES, _STAGES_FIT, SFA1_COL) +if B1WIDE: + assert ( + XSTAGE + and FUSED_ROUTER + and (W_WARP == 4) + and (NRW0 == 4) + and (RW_EXTRA == THREADS // 32 - 8) + ), ( + "b1wide: every warp of the CTA must be warps 0..3 or a routing warp", + NRW0, + RW_EXTRA, + THREADS, + ) + assert M_MAX * (H // 16) % THREADS == 0, (M_MAX, THREADS) +assert M_MAX * H * 2 <= STAGES * (NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA), ( + "x staging must fit the ring" +) +_OFF_SFAR = STAGES * NUM_BYTES_A +_OFF_SFA = _OFF_SFAR + STAGES * _RAW_STAGE +_OFF_B1 = _OFF_SFA + STAGES * NUM_BYTES_SFA +_OFF_B2 = _OFF_B1 + S1 * NUM_BYTES_B1 +_OFF_SFB2 = _OFF_B2 + 2 * KT2 * NUM_BYTES_B2 +BIG = _OFF_SFB2 + 2 * KT2 * NUM_KB * 512 +_SB_A = 0 +_SB_RAW = None +_SH_XCH_BYTES = (SH_SPLIT - 1) * 2 * SH_ROWS * 8 * 4 +_SH_FIX = ( + SH_GT * SH_NB * 128 + + SH_DKT * 8 * 128 + + M_MAX * SH_KB + + M_MAX * SH_DKB + + _SH_XCH_BYTES + + 2 * SH_ROWS * 8 * 4 + + 1024 +) +SH_NST = min(8, (BIG - _SH_FIX) // (NUM_BYTES_A + 4 * 512 + 512)) if SHARED else 1 +_SB_RAW = SH_NST * NUM_BYTES_A +_SB_SPL = _SB_RAW + SH_NST * 2048 +_SB_BX = (_SB_SPL + SH_NST * 512 + 1023) // 1024 * 1024 +_SB_BH = _SB_BX + SH_GT * SH_NB * 128 +_SB_XC = _SB_BH + SH_DKT * 8 * 128 +_SB_HC = _SB_XC + M_MAX * SH_KB +_SB_XCH = (_SB_HC + M_MAX * SH_DKB + 15) // 16 * 16 +_SB_GU = _SB_XCH + _SH_XCH_BYTES +_SB_END = _SB_GU + 2 * SH_ROWS * 8 * 4 +SH_ACC_GU = 0 +SH_ACC_DN = SH_NB +SH_SFA = SH_ACC_DN + 16 +SH_SFB_GU = SH_SFA + 4 * 8 +SH_SFB_DN = SH_SFB_GU + 4 * SH_GT +SH_TMEM = 256 +if SHARED: + assert FUSED_ROUTER, "shared expert: fused router only" + assert SH_STAGES - SH_NST <= 64, SH_STAGES + assert SH_NST >= 3 and _SB_END <= BIG and (SH_SFB_DN + 24 <= SH_TMEM), (SH_NST, _SB_END, BIG) + + +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], + "ld.volatile.global.s32 $0, [$1];", + "=r,l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _st_global_b32_if(addr, value, pred, *, loc=None, ip=None): + """st.global.b32 when pred != 0.""" + _llvm.inline_asm( + None, + [ + cutlass.Int64(addr).ir_value(loc=loc, ip=ip), + cutlass.Int32(value).ir_value(loc=loc, ip=ip), + cutlass.Int32(pred).ir_value(loc=loc, ip=ip), + ], + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, $2, 0;\n\t@p st.global.b32 [$0], $1;\n\t}", + "l,r,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [ + cutlass.Float32(hi).ir_value(loc=loc, ip=ip), + cutlass.Float32(lo).ir_value(loc=loc, ip=ip), + ], + ( + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, " + "$1, $2;\n\tand.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl " + "and.b32 w, w, 0xFFFF0000;\n\tshr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h," + " 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\tmov.b32 $0, w;\n\t}" + ), + "=r,f,f", + has_side_effects=False, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +def _ar_push(arp, v, low, t, cl): + """Push bf16 channel pair (cl, cl + 1) of token t (v = channel cl's bf16 value on this lane, its neighbour's on + lane ^ 8) into this rank's set-A slot of the exchange; the slot half is the all-reduce CTA (t, cl // 768)'s call + parity. Only lanes with ``low`` store; every lane of the warp must call it (shuffle).""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + world = arp.load(idx=4) + parity = cutlass.Int64( + _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cs // cutlass.Int32(H // 8)) + ) + & cutlass.Int32(1) + ) + off = ( + parity * arp.load(idx=5) * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + arp.load(idx=3)) * cutlass.Int64(H // 2) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(arp.load(idx=1) + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +def _ar_parity(arp, t, cl): + """The call parity _ar_push reads for token t's channel cl (fixed between this kernel's grid dependency and the + next all-reduce, which this kernel releases at teardown): loaded ahead of the finalize (AR_PRE).""" + return _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cl // cutlass.Int32(H // 8)) + ) + + +def _ar_push_pre(ard, parity32, v, low, t, cl): + """_ar_push with the parity (_ar_parity) and the descriptor fields ard = (mc, rank, world, maxt) preloaded.""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + mc, rk, world, maxt = ard + parity = cutlass.Int64(parity32 & cutlass.Int32(1)) + off = ( + parity * maxt * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + rk) * cutlass.Int64(H // 2) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(mc + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +@dsl_user_op +def _bar_arrive(bar_id: int, threads: int, *, loc=None, ip=None): + """bar.arrive: signal a named barrier without waiting (its memory ordering is bar.sync's).""" + _llvm.inline_asm( + None, + [], + f"bar.arrive {bar_id}, {threads};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _st_relaxed_u32(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "st.relaxed.gpu.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _st_relaxed_u64(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int64(val)], + "st.relaxed.gpu.global.u64 [$0], $1;", + "l,l", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm( + _T.i64(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u64 $0, [$1];", + "=l,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _nanosleep(ns: int, *, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + f"nanosleep.u32 {ns};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _gtime(*, loc=None, ip=None): + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), + [], + "mov.u64 $0, %globaltimer;", + "=l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _smid(*, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [], + "mov.u32 $0, %smid;", + "=r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@cute.jit +def _stamp(tdbg: cutlass.Array, bx, i: cutlass.Constexpr): + if cutlass.const_expr(TIMING): + tdbg.store(_gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(i)) + + +@dsl_user_op +def _rcp(a, *, loc=None, ip=None): + """rcp.approx.ftz.f32 (the reciprocal TRT-LLM's FP4 quantization uses).""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _div_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _sub_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "sub.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _mul_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3_bits(x, *, loc=None, ip=None): + """e4m3 (round to nearest even, saturating) of x as an int 0..255.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, 0f00000000, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(255) + + +@dsl_user_op +def _e4m3_to_f32(b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(b)], + ( + "{\n.reg .b16 t, l, u;\n.reg .b32 h;\ncvt.u16.u32 t, $1;\ncvt.rn.f16x" + "2.e4m3x2 h, t;\nmov.b32 {l, u}, h;\ncvt.f32.f16 $0, l;\n}" + ), + "=f,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e2m1x2(hi, lo, *, loc=None, ip=None): + """One byte of two E2M1 values (lo in the low nibble), round to nearest even, saturating.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b8 t;\ncvt.rn.satfinite.e2m1x2.f32 t, $1, $2;\ncvt.u32.u8 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2(x, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _bf16_round(x, *, loc=None, ip=None): + """x rounded to bf16 (nearest even), as f32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\nmov.b32 $0, {0, t};\n}", + "=f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _redux_max_u32(v, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(v)], + "redux.sync.max.u32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16 (nearest even) bits of x in the low 16 bits.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ue8m0_rp(x, *, loc=None, ip=None): + """UE8M0 code of x rounded toward +inf, saturating (__nv_cvt_float_to_e8m0(x, SATFINITE, RoundPosInf)).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rp.satfinite.ue8m0x2.f32 t, $1, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(255) + + +@dsl_user_op +def _e4m3x2(hi, lo, *, loc=None, ip=None): + """Two e4m3 (nearest even, saturating) in 16 bits, lo in the low byte.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, $1, $2;\ncvt.u32.u16 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _min_xorsign_abs(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "min.xorsign.abs.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _div_full(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.full.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2_full(x, *, loc=None, ip=None): + """ex2.approx.f32 (no flush to zero), Triton's tl.exp.""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _l2_prefetch(addr, nbytes, *, loc=None, ip=None): + """cp.async.bulk.prefetch.L2 of nbytes (a multiple of 16) at global address addr (16-byte aligned).""" + _llvm.inline_asm( + None, + [ + cutlass.Int64(addr).ir_value(loc=loc, ip=ip), + cutlass.Int32(nbytes).ir_value(loc=loc, ip=ip), + ], + "cp.async.bulk.prefetch.L2.global [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +def _mx_osc(amax): + """mxfp8 quantization of a block with absolute max amax: (UE8M0 code, the element multiplier).""" + code = _ue8m0_rp(_mul_rn(amax, _rcp(cutlass.Float32(448.0)))) + cf = cutlass.Int32( + cutlass.select_( + code == cutlass.Int32(255), + cutlass.Int32(2147483647), + cutlass.select_( + code == cutlass.Int32(0), cutlass.Int32(4194304), code << cutlass.Int32(23) + ), + ) + ).bitcast(cutlass.Float32) + return (code, cutlass.select_(amax != cutlass.Float32(0.0), _rcp(cf), cutlass.Float32(0.0))) + + +def _mx_words(v, osc): + """e4m3 words of len(v) (a multiple of 4) values times osc, in element order.""" + words = [] + for t in range(len(v) // 4): + lo = _e4m3x2(_mul_rn(v[4 * t + 1], osc), _mul_rn(v[4 * t], osc)) + hi = _e4m3x2(_mul_rn(v[4 * t + 3], osc), _mul_rn(v[4 * t + 2], osc)) + words.append(lo | hi << cutlass.Int32(16)) + return words + + +def _twiddle(score): + """Float bits mapped so that unsigned order is float order (cub TwiddleIn).""" + bits = ( + score.bitcast(cutlass.Int32) + if hasattr(score, "bitcast") + else cutlass.Float32(score).bitcast(cutlass.Int32) + ) + return cutlass.select_(bits < cutlass.Int32(0), ~bits, bits | cutlass.Int32(-2147483648)) + + +@cute.jit +def _wait(bar: cutlass.Array, parity): + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +def _tptr(base_row, col, dtype=cutlass.Int32): + return cutlass.inttoptr(base_row << 16 | col, 6, dtype) + + +@dsl_user_op +def _match_any(value, *, loc=None, ip=None): + """The mask of the warp's lanes holding the same 32-bit value.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(value)], + "match.any.sync.b32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@cute.jit +def _waitd(bar: cutlass.Array, parity, tdbg: cutlass.Array, site: cutlass.Constexpr): + """_wait, or (HANG_DBG) give up after ~2^22 polls and record the site in tdbg[bx, 24 + warp].""" + if cutlass.const_expr(HANG_DBG): + n = cutlass.Int32(0) + while (not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity)) & ( + n < cutlass.Int32(1 << 22) + ): + n = n + cutlass.Int32(1) + if n >= cutlass.Int32(1 << 22): + bx, _, _ = cute.arch.block_idx() + w = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + tdbg.store( + cutlass.Int64(site), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + w + ) + else: + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +@cute.jit +def _adv(stage, phase): + """The next ring position's (stage, phase).""" + stage = stage + cutlass.Int32(1) + wrap = stage == cutlass.Int32(STAGES) + return ( + cutlass.select_(wrap, cutlass.Int32(0), stage), + cutlass.select_(wrap, phase ^ cutlass.Int32(1), phase), + ) + + +@cute.jit +def _seg(i0, i1, k): + """Segment k of the FC1 item range [i0, i1) (items = unit-major stages): (unit, first stage, end stage). Order: + the last unit (its head, the part a split unit's next CTA does not cover), then the first unit, then the rest.""" + uf = i0 // cutlass.Int32(S1) + ul = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) + unit = uf + k - cutlass.Int32(1) + unit = cutlass.select_(k == cutlass.Int32(0), ul, unit) + unit = cutlass.select_(ul == uf, uf, unit) + j0 = i0 - unit * cutlass.Int32(S1) + j0 = cutlass.select_(j0 < cutlass.Int32(0), cutlass.Int32(0), j0) + j1 = i1 - unit * cutlass.Int32(S1) + j1 = cutlass.select_(j1 > cutlass.Int32(S1), cutlass.Int32(S1), j1) + return (unit, j0, j1) + + +@cute.jit +def _nsegs(i0, i1): + n = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) - i0 // cutlass.Int32(S1) + cutlass.Int32(1) + return cutlass.select_(i1 > i0, n, cutlass.Int32(0)) + + +@cute.jit +def _sfb1_to_tmem(warp, lane, base_row, base_col, src: cutlass.Array, b1_ready: cutlass.Array): + """B1 scales to TMEM: instruction (j, kb)'s column holds, at lane n = QS q + t, token t's 4 scales of k-tile + 4 j + q, K 64 kb .. (src: int32 words [M, 96] of the token scales); each warp writes its sub-partition.""" + q = lane // cutlass.Int32(QS) + tl = lane - q * cutlass.Int32(QS) + sf_ok = (lane < cutlass.Int32(N1)) & (tl < cutlass.Int32(M_MAX)) + sfv = [] + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + j = a // NUM_KB + kb = a % NUM_KB + wsf = src.load( + idx=cutlass.select_( + sf_ok, + tl * cutlass.Int32(H // 64) + + (cutlass.Int32(4 * j) + q) * cutlass.Int32(4) + + cutlass.Int32(kb), + cutlass.Int32(0), + ) + ) + sfv.append(cutlass.select_(sf_ok, cutlass.Int32(wsf), cutlass.Int32(0))) + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + prims.tcgen05_st( + "32x32b", + _tptr(base_row + warp * cutlass.Int32(32), base_col + SFB1_COL + a * 4), + sfv[a], + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(b1_ready) + + +@cute.jit +def _b1_addr(t, kb16): + """Word index in the resident B tiles of the 8 FP4 bytes of token t's 16-value block kb16.""" + kt = kb16 >> cutlass.Int32(4) + j = kt >> cutlass.Int32(2) + n = (kt & cutlass.Int32(3)) * cutlass.Int32(QS) + t + cc = kb16 >> cutlass.Int32(1) & cutlass.Int32(7) + return ( + j * cutlass.Int32(NUM_BYTES_B1) + + n * cutlass.Int32(KT_BYTES) + + (cc ^ n & cutlass.Int32(7)) * cutlass.Int32(16) + + (kb16 & cutlass.Int32(1)) * cutlass.Int32(8) + ) // cutlass.Int32(4) + + +@cute.jit +def _build_b1( + tidx, + warp, + lane, + base_row, + base_col, + x32: cutlass.Array, + xsf32: cutlass.Array, + sB1: cutlass.Array, + b1_ready: cutlass.Array, +): + """B (the M tokens' whole K) into the resident tiles from the NVFP4 activation, its scales to TMEM.""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + for r in cutlass.range_constexpr((M_MAX * (H // 16) + EPI_THREADS - 1) // EPI_THREADS): + b = tidx + cutlass.Int32(r * EPI_THREADS) + if b < cutlass.Int32(M_MAX * (H // 16)): + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + v2 = x32.load(idx=b * cutlass.Int32(2), vector_size=2, alignment=8) + sB1_32.store( + (cutlass.Int32(v2[0]), cutlass.Int32(v2[1])), idx=_b1_addr(t, kb), alignment=8 + ) + _sfb1_to_tmem(warp, lane, base_row, base_col, xsf32, b1_ready) + + +@cute.jit +def _b1_quant( + qtid, + NTHR: int, + xb32: cutlass.Array, + sfsc: cutlass.Array, + sB1: cutlass.Array, + s_xsf: cutlass.Array, +): + """(B1WIDE) the quantize loop of _build_b1_fused over NTHR threads (thread qtid): blocks qtid, qtid + NTHR, ..""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + s_xsf8 = cutlass.Array( + s_xsf.data_ptr(0), shape=(M_MAX * (H // 16),), dtype=cutlass.Int8, alignment=16 + ) + sc = cutlass.Float32(sfsc.load(idx=0)) + rsc = _rcp(sc) + rsix = _rcp(cutlass.Float32(6.0)) + NBLK = M_MAX * (H // 16) // NTHR + QB = 3 if NBLK % 3 == 0 else 2 if NBLK % 2 == 0 else 1 + for rg in cutlass.range(NBLK // QB, unroll=1): + wall = [] + for rr in cutlass.range_constexpr(QB): + bq = qtid + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(NTHR) + for h in cutlass.range_constexpr(2): + wall.append( + xb32.load( + idx=bq * cutlass.Int32(8) + cutlass.Int32(4 * h), + vector_size=4, + alignment=16, + ) + ) + for rr in cutlass.range_constexpr(QB): + b = qtid + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(NTHR) + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + w = [] + for h in cutlass.range_constexpr(2): + v4 = wall[rr * 2 + h] + for z in cutlass.range_constexpr(4): + w.append(cutlass.Int32(v4[z])) + xs = [] + for z in cutlass.range_constexpr(8): + xs.append((w[z] << cutlass.Int32(16)).bitcast(cutlass.Float32)) + xs.append((w[z] & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + vmax = cute.math.abs(xs[0]) + for z in cutlass.range_constexpr(1, 16): + vmax = cute.arch.fmax(vmax, cute.math.abs(xs[z])) + sfb = _e4m3_bits(_mul_rn(sc, _mul_rn(vmax, rsix))) + osc = cutlass.select_( + vmax != cutlass.Float32(0.0), + _rcp(_mul_rn(_e4m3_to_f32(sfb), rsc)), + cutlass.Float32(0.0), + ) + by = [ + _e2m1x2(_mul_rn(xs[2 * z + 1], osc), _mul_rn(xs[2 * z], osc)) & cutlass.Int32(255) + for z in range(8) + ] + lo = ( + by[0] + | by[1] << cutlass.Int32(8) + | by[2] << cutlass.Int32(16) + | by[3] << cutlass.Int32(24) + ) + hi = ( + by[4] + | by[5] << cutlass.Int32(8) + | by[6] << cutlass.Int32(16) + | by[7] << cutlass.Int32(24) + ) + sB1_32.store((lo, hi), idx=_b1_addr(t, kb), alignment=8) + s_xsf8.store(cutlass.Int8(sfb), idx=b) + + +def _build_b1_fused( + tidx, + warp, + lane, + base_row, + base_col, + xb32: cutlass.Array, + sfsc: cutlass.Array, + sB1: cutlass.Array, + s_xsf: cutlass.Array, + b1_ready: cutlass.Array, +): + """_build_b1 from the bf16 hidden states, quantized exactly as fp4_quantize(x, sf_scale, 16).""" + _b1_quant(tidx, EPI_THREADS, xb32, sfsc, sB1, s_xsf) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + _sfb1_to_tmem(warp, lane, base_row, base_col, s_xsf, b1_ready) + + +@cute.jit +def _swiglu_quant(xv, gate_raw, is_up, a, c, alpha, beta, lim): + """TRTLLM-Gen's FC1 epilogue (the swiGlu Bmm E2m1 output), operation for operation: lane value xv is the up row, + gate_raw its gate row (fp32 sums). Returns (q, the scale's e4m3 bits) of the 16-row NVFP4 block (lanes r, r ^ 16 + and r ^ 8 cover it; amax over the warp's up lanes of a half). a = fc31_alpha, c = fc31_scale_c.""" + neg_max = cutlass.Float32(-3.4028234663852886e38) + gc = cutlass.select_(gate_raw > lim, lim, gate_raw) + gc = cutlass.select_(gate_raw < neg_max, neg_max, gc) + uc = cutlass.select_(xv > lim, lim, xv) + uc = cutlass.select_(xv < -lim, -lim, uc) + c1 = _mul_rn(a, _mul_rn(alpha, cutlass.Float32(1.4426950216293335))) + sgm = _rcp(_add_rn(_ex2(-_mul_rn(gc, c1)), cutlass.Float32(1.0))) + gs = _mul_rn(gc, sgm) + h = _mul_rn(_fma_rn(uc, a, _mul_rn(a, beta)), gs) + h = cutlass.select_(is_up, h, cutlass.Float32(0.0)) + amax = cute.arch.fmax(_mul_rn(cute.math.abs(h), c), cutlass.Float32(0.0)) + for off in (16, 8, 4, 2, 1): + amax = cute.arch.fmax(amax, cute.arch.shuffle_sync_bfly(amax, off)) + sfb = _e4m3_bits(_mul_rn(amax, cutlass.Float32(0.1666666716337204))) + sff = _e4m3_to_f32(sfb) + osc = cutlass.select_(sff != cutlass.Float32(0.0), _rcp(sff), cutlass.Float32(0.0)) + return (_mul_rn(_mul_rn(h, osc), c), sfb) + + +@cute.jit +def _plan_range(s_plan: cutlass.Array, p): + w = s_plan.load(idx=p) + return (w & cutlass.Int32(65535), w >> cutlass.Int32(16)) + + +@cute.jit +def _sh_issue( + lane, + sid, + g, + s_big: cutlass.Array, + sh_full: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, +): + """Stage g of the shared CTA's stream into ring slot g % SH_NST (warp 0). g < SH_GT: gate_up k-tile g of both + slices: quadrant Q = 2 q + u (q slice, u 0 gate / 1 up) = rows 32 sid .. (+ 768 for up) at k-tile SH_GT q + g, its + raw scale atom at raw slot Q. Else down tile j = (g - SH_GT) / 6 (rows 256 sid + 128 j ..), k-tile + (g - SH_GT) % 6.""" + st = g % cutlass.Int32(SH_NST) + bar = sh_full.subview(st) + if cutlass.const_expr(SH_PRETILE): + if lane == 0: + src = (sid * cutlass.Int32(SH_STAGES) + g) * cutlass.Int32(SH_STAGE_BYTES) + prims.mbarrier_arrive_expect_tx(bar, SH_STAGE_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), gusf.subview(src), bar, 16384 + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + gusf.subview(src + cutlass.Int32(16384)), + bar, + 512, + ) + cute.arch.sync_warp() + elif g < cutlass.Int32(SH_GT): + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 4 * 512) + cute.arch.sync_warp() + if lane < 4: + q = lane >> cutlass.Int32(1) + u = lane & cutlass.Int32(1) + row0 = sid * cutlass.Int32(32) + u * cutlass.Int32(SH_I) + kt = q * cutlass.Int32(SH_GT) + g + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A) + lane * cutlass.Int32(4096)), + tma_gu, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview( + cutlass.Int32(_SB_RAW) + st * cutlass.Int32(2048) + lane * cutlass.Int32(512) + ), + gusf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_KB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(SH_DKT) + kt = gd - jt * cutlass.Int32(SH_DKT) + row0 = sid * cutlass.Int32(256) + jt * cutlass.Int32(128) + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 512) + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), + tma_dn, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_RAW) + st * cutlass.Int32(2048)), + dnsf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_DKB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + cute.arch.sync_warp() + + +@dsl_user_op +def _red_release_add(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "red.release.gpu.global.add.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_acquire_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.acquire.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fence_proxy_async_global(*, loc=None, ip=None): + _asm(None, [], "fence.proxy.async.global;", "", True, loc, ip) + + +@cute.jit +def _sfa_to_tmem(sSFA: cutlass.Array, stage, base_row, base_col): + """(SFA_CP, elected MMA thread) the stage's 4 scale atoms smem -> TMEM (what the splicer warp does otherwise).""" + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SFA1_COL + stage * cutlass.Int32(16) + kb * 4), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + + +@cute.jit +def _sf1_cp( + lane, stage, sf_line, c, sSFA: cutlass.Array, s1w: cutlass.Array, ab_full: cutlass.Array +): + """(SFA_CP) FC1 stage scale atoms: word [r][q] of atom kb <- raw atom 4 q + kb (raw line sf_line = 4 x atom) + word [r][c % 4] (what the splicer copies out of the raw box), 4-byte cp.async by the warp's 32 lanes; their + arrivals (noinc) complete the stage's full barrier together with the A boxes' bytes.""" + src0 = ( + (sf_line >> cutlass.Int32(2)) * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (c & cutlass.Int32(3)) + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for q in cutlass.range_constexpr(C1): + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + q)), + s1w.subview(src0 + cutlass.Int32((4 * q + kb) * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _sf2_cp( + lane, + stage, + g, + blk, + t2: cutlass.Constexpr, + n_here, + s_el: cutlass.Array, + sSFA: cutlass.Array, + s2w: cutlass.Array, + ab_full: cutlass.Array, +): + """(SFA_CP) FC2 k-tile t2 scale atoms: word [r][jj] of atom kb <- raw atom (rows / 128, K block 4 t2 + kb) of slot + jj's expert, word [r][blk % 4].""" + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + src0 = ( + ( + (e * cutlass.Int32(SF2_RB) + (blk >> cutlass.Int32(2))) * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (blk & cutlass.Int32(3)) + ) + for kb in cutlass.range_constexpr(_nkb2(t2)): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + jj)), + s2w.subview(src0 + cutlass.Int32(kb * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _shared_cta( + tidx, + warp, + lane, + sid, + s_big: cutlass.Array, + sh_bar: cutlass.Array, + epochs: cutlass.Array, + tdbg: cutlass.Array, + bx, + xb32: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, + tmem_ptr_i32: cutlass.Array, + gu8: cutlass.Array, + dn8: cutlass.Array, +): + """Shared CTA sid (see SHARED): tcgen05 MXF8 (E4M3 x E4M3, UE8M0 per 32 K), m3_mx_gemv's tile order. gate_up: one + 128-row MMA tile = 2 K slices x (32 gate + 32 up rows), B rows 8 q + t = token t's slice-q k-tile; down: 2 tiles + of 128 rows, full K. Warps: 0 weights (TMA), 1 TMEM + scale splice + MMA, 4-7 epilogue; 2.. activation quant.""" + big32 = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Int32, alignment=16) + bigf = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Float32, alignment=16) + big8 = cutlass.Array(s_big.data_ptr(0), shape=(BIG,), dtype=cutlass.Int8, alignment=16) + sh_full = sh_bar + sh_empty = cutlass.Array(sh_bar.data_ptr(8), shape=(8,), dtype=cutlass.Int64, alignment=8) + b_gu_rdy = sh_bar.subview(16) + b_dn_rdy = sh_bar.subview(17) + acc_gu = sh_bar.subview(18) + acc_dn = sh_bar.subview(19) + if warp == 0: + prims.prefetch_tensormap(tma_gu) + prims.prefetch_tensormap(tma_dn) + if lane < SH_NST: + prims.mbarrier_init(sh_full.subview(lane), 1) + prims.mbarrier_init(sh_empty.subview(lane), 33) + if lane == 0: + prims.mbarrier_init(b_gu_rdy, 1) + prims.mbarrier_init(b_dn_rdy, 1) + prims.mbarrier_init(acc_gu, 1) + prims.mbarrier_init(acc_dn, 1) + if warp == 1: + prims.tcgen05_alloc(tmem_ptr_i32, SH_TMEM) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 65535 + base_row = tmem_raw >> 16 + if warp == 0: + for g in cutlass.range_constexpr(SH_NST): + _sh_issue(lane, sid, cutlass.Int32(g), s_big, sh_full, tma_gu, tma_dn, gusf, dnsf) + if cutlass.const_expr(SH_PRETILE): + for r in cutlass.range_constexpr(2): + pg = cutlass.Int32(SH_NST) + lane * cutlass.Int32(2) + cutlass.Int32(r) + if pg < cutlass.Int32(SH_STAGES): + _l2_prefetch( + gusf.subview( + (sid * cutlass.Int32(SH_STAGES) + pg) * cutlass.Int32(SH_STAGE_BYTES) + ) + .data_ptr() + .toint(), + SH_STAGE_BYTES, + ) + else: + for r in cutlass.range_constexpr(2): + prw = sid * cutlass.Int32(32) + cutlass.Int32(r * SH_I) + lane + _l2_prefetch(gu8.subview(prw * cutlass.Int32(H)).data_ptr().toint(), H) + if lane == 0: + _l2_prefetch( + dn8.subview(sid * cutlass.Int32(256 * SH_I)).data_ptr().toint(), 256 * SH_I + ) + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(not LATE_TRIGGER): + if arp.load(idx=0) == cutlass.Int64(0): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + if cutlass.const_expr(EARLY_EPOCH): + prims.barrier_cta_sync(0) + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + tag16 = ep & cutlass.Int32(32767) | cutlass.Int32(32768) + NQ = THREADS - 64 + if warp == 0: + for g in cutlass.range(SH_NST, SH_STAGES, unroll=1): + _wait( + sh_empty.subview(g % cutlass.Int32(SH_NST)), + g // cutlass.Int32(SH_NST) - cutlass.Int32(1) & cutlass.Int32(1), + ) + _sh_issue(lane, sid, g, s_big, sh_full, tma_gu, tma_dn, gusf, dnsf) + elif warp == 1: + idesc_gu = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=SH_NB, + m_dim=128, + ) + idesc_dn = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=8, + m_dim=128, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + desc_a0 = prims.Tcgen05SmemDesc.build( + s_big.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bx = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BX), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bh = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BH), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = base_row << 16 | base_col + sq = sid & cutlass.Int32(3) + _wait(b_gu_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for g in cutlass.range(SH_STAGES, unroll=1): + st = g % cutlass.Int32(SH_NST) + if g == cutlass.Int32(SH_GT): + _wait(b_dn_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + _wait(sh_full.subview(st), g // cutlass.Int32(SH_NST) & cutlass.Int32(1)) + raw = cutlass.Int32(_SB_RAW // 4) + st * cutlass.Int32(512) + if (g < cutlass.Int32(SH_GT)) & cutlass.Boolean(not SH_PRETILE): + w4 = [ + big32.load(idx=raw + cutlass.Int32(128 * qq) + lane * cutlass.Int32(4) + sq) + for qq in range(4) + ] + big32.store( + ( + cutlass.Int32(w4[0]), + cutlass.Int32(w4[1]), + cutlass.Int32(w4[2]), + cutlass.Int32(w4[3]), + ), + idx=cutlass.Int32(_SB_SPL // 4) + + st * cutlass.Int32(128) + + lane * cutlass.Int32(4), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + prims.mbarrier_arrive(sh_empty.subview(st)) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + if g < cutlass.Int32(SH_GT): + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bx + cutlass.Int64(g * cutlass.Int32(SH_NB * 128 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr(tcol + SH_ACC_GU, 6, cutlass.Float32), + da + 2 * kb, + db + 2 * kb, + idesc_gu.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (g > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_GU + g * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_GT - 1): + prims.tcgen05_commit(acc_gu) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(SH_DKT) + kt = gd - jt * cutlass.Int32(SH_DKT) + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview( + cutlass.Int32(_SB_SPL if SH_PRETILE else _SB_RAW) + + st * cutlass.Int32(512 if SH_PRETILE else 2048) + ), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bh + cutlass.Int64(kt * cutlass.Int32(1024 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr( + tcol + SH_ACC_DN + jt * cutlass.Int32(8), 6, cutlass.Float32 + ), + da + 2 * kb, + db + 2 * kb, + idesc_dn.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (kt > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_DN + kt * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_STAGES - 1): + prims.tcgen05_commit(acc_dn) + cute.arch.sync_warp() + if warp >= 2: + qt = tidx - cutlass.Int32(64) + for r in cutlass.range_constexpr((M_MAX * SH_KB + NQ - 1) // NQ): + q_ = qt + cutlass.Int32(r * NQ) + if q_ < cutlass.Int32(M_MAX * SH_KB): + t = q_ // cutlass.Int32(SH_KB) + b = q_ - t * cutlass.Int32(SH_KB) + wv = [ + xb32.load( + idx=t * cutlass.Int32(H // 2) + + b * cutlass.Int32(16) + + cutlass.Int32(4 * z), + vector_size=4, + alignment=16, + ) + for z in range(4) + ] + v = [] + for z in cutlass.range_constexpr(4): + for zz in cutlass.range_constexpr(4): + w = cutlass.Int32(wv[z][zz]) + v.append((w << cutlass.Int32(16)).bitcast(cutlass.Float32)) + v.append((w & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(v[z])) + code, osc = _mx_osc(amax) + words = _mx_words(v, osc) + kt = b >> cutlass.Int32(2) + sl = kt // cutlass.Int32(SH_GT) + it = kt - sl * cutlass.Int32(SH_GT) + n = sl * cutlass.Int32(8) + t + c0 = (b & cutlass.Int32(3)) * cutlass.Int32(2) + rw = cutlass.Int32(_SB_BX) + it * cutlass.Int32(SH_NB * 128) + n * cutlass.Int32( + 128 + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ n & cutlass.Int32(7)) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + ((c0 + cutlass.Int32(1) ^ n & cutlass.Int32(7)) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_XC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR, number_of_threads=NQ) + if warp >= 4: + if warp < 8: + qd = warp - cutlass.Int32(4) + n = lane + t = n & cutlass.Int32(7) + ok = (n < cutlass.Int32(SH_NB)) & (t < cutlass.Int32(M_MAX)) + for i in cutlass.range_constexpr(SH_GT): + kt = (n >> cutlass.Int32(3)) * cutlass.Int32(SH_GT) + cutlass.Int32(i) + wsf = big32.load( + idx=cutlass.Int32(_SB_XC) + + cutlass.select_(ok, t, cutlass.Int32(0)) * cutlass.Int32(SH_KB) + + kt * cutlass.Int32(4) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_GU + i * 4), + cutlass.select_(ok, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + et = tidx - cutlass.Int32(128) + if et == 0: + prims.mbarrier_arrive(b_gu_rdy) + _stamp(tdbg, bx, 54) + _wait(acc_gu, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if cutlass.const_expr(SH_SPLIT == 2): + av = prims.tcgen05_ld( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_ACC_GU, cutlass.Float32), + num=16, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(2): + for tt in cutlass.range_constexpr(8): + bigf.store( + cutlass.Float32(av[8 + tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(2)) * cutlass.Int32(32) + lane) + * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd < cutlass.Int32(2): + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + else: + av = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_GU + qd * cutlass.Int32(8), + cutlass.Float32, + ), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(1): + for tt in cutlass.range_constexpr(8): + bigf.store( + cutlass.Float32(av[tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(1)) * cutlass.Int32(32) + lane) + * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd == cutlass.Int32(0): + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + for s_ in cutlass.range_constexpr(1, SH_SPLIT): + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (cutlass.Int32((s_ - 1) * 32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + lane * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + for r in cutlass.range_constexpr((SH_ROWS * M_MAX + 127) // 128): + pq = et + cutlass.Int32(r * 128) + if pq < cutlass.Int32(SH_ROWS * M_MAX): + p = pq // cutlass.Int32(M_MAX) + tt = pq - p * cutlass.Int32(M_MAX) + ga = bigf.load(idx=cutlass.Int32(_SB_GU // 4) + p * cutlass.Int32(8) + tt) + up = bigf.load( + idx=cutlass.Int32(_SB_GU // 4) + + (cutlass.Int32(SH_ROWS) + p) * cutlass.Int32(8) + + tt + ) + q_a = cute.arch.fmin(cutlass.Float32(ga), cutlass.Float32(7.0)) + q_bu = _min_xorsign_abs(cutlass.Float32(up), cutlass.Float32(7.0)) + q_ub = _add_rn(q_bu, cutlass.Float32(1.0)) + q_e2 = _ex2_full( + _mul_rn( + _mul_rn(q_a, cutlass.Float32(-1.7020000219345093)), + cutlass.Float32(1.4426950216293335), + ) + ) + q_sg = _div_full(cutlass.Float32(1.0), _add_rn(q_e2, cutlass.Float32(1.0))) + q_h = _mul_rn(q_ub, _mul_rn(q_sg, q_a)) + _st_relaxed_u32( + hx.subview(tt * cutlass.Int32(SH_I) + sid * cutlass.Int32(SH_ROWS) + p) + .data_ptr() + .toint(), + tag16 << cutlass.Int32(16) | _bf16_bits(q_h), + ) + if et == 0: + _stamp(tdbg, bx, 50) + for rr in cutlass.range_constexpr((M_MAX * SH_DKB + 127) // 128): + q_ = et + cutlass.Int32(rr * 128) + if q_ < cutlass.Int32(M_MAX * SH_DKB): + t = q_ // cutlass.Int32(SH_DKB) + kb = q_ - t * cutlass.Int32(SH_DKB) + base = t * cutlass.Int32(SH_I) + kb * cutlass.Int32(32) + hv = [cutlass.Int32(0) for _ in range(32)] + pend = cutlass.Int32(1) + while pend != cutlass.Int32(0): + for q4 in cutlass.range_constexpr(8): + v4 = prims.load_ext( + hx.subview(base + cutlass.Int32(4 * q4)), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for z in cutlass.range_constexpr(4): + hv[4 * q4 + z] = cutlass.Int32(v4[z]) + okw = cutlass.Boolean(True) + for z in cutlass.range_constexpr(32): + okw = okw & (hv[z] >> cutlass.Int32(16) & cutlass.Int32(65535) == tag16) + pend = cutlass.select_(okw, cutlass.Int32(0), cutlass.Int32(1)) + vals = [ + (hv[z] << cutlass.Int32(16)).bitcast(cutlass.Float32) for z in range(32) + ] + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(vals[z])) + code, osc = _mx_osc(amax) + words = _mx_words(vals, osc) + kt = kb >> cutlass.Int32(2) + c0 = (kb & cutlass.Int32(3)) * cutlass.Int32(2) + rw = cutlass.Int32(_SB_BH) + kt * cutlass.Int32(1024) + t * cutlass.Int32( + 128 + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + ((c0 + cutlass.Int32(1) ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_HC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + okh = lane < cutlass.Int32(M_MAX) + for kt in cutlass.range_constexpr(SH_DKT): + wsf = big32.load( + idx=cutlass.Int32(_SB_HC) + + cutlass.select_(okh, lane, cutlass.Int32(0)) * cutlass.Int32(SH_DKB) + + cutlass.Int32(4 * kt) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_DN + kt * 4), + cutlass.select_(okh, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if et == 0: + prims.mbarrier_arrive(b_dn_rdy) + _stamp(tdbg, bx, 51) + _wait(acc_dn, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for jt in cutlass.range_constexpr(2): + dv = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_DN + jt * 8, + cutlass.Float32, + ), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + rowo = ( + sid * cutlass.Int32(256) + + cutlass.Int32(128 * jt) + + qd * cutlass.Int32(32) + + lane + ) + for tt in cutlass.range_constexpr(M_MAX): + _st_relaxed_u32( + dnx.subview(cutlass.Int32(tt * H) + rowo).data_ptr().toint(), + tag16 << cutlass.Int32(16) | _bf16_bits(cutlass.Float32(dv[tt])), + ) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if et == 0: + _stamp(tdbg, bx, 52) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if warp == 1: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc(cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), SH_TMEM) + if cutlass.const_expr(not EARLY_EPOCH): + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + + +@cute.kernel +def m3_moe_kernel( + tma_a1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_a2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_gu_desc: cutlass.GridConstant[cuda.TensorMap], + tma_dn_desc: cutlass.GridConstant[cuda.TensorMap], + logits: cutlass.Array, + bias: cutlass.Array, + x32: cutlass.Array, + xsf32: cutlass.Array, + escal: cutlass.Array, + plan: cutlass.Array, + partials: cutlass.Array, + hbuf32: cutlass.Array, + epochs: cutlass.Array, + out: cutlass.Array, + route_dbg: cutlass.Array, + tdbg: cutlass.Array, + xb32: cutlass.Array, + rw: cutlass.Array, + sfsc: cutlass.Array, + lgbuf: cutlass.Array, + guw: cutlass.Array, + gusf: cutlass.Array, + dnw: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, + s1w: cutlass.Array, + s2w: cutlass.Array, + img32: cutlass.Array, + b2cnt: cutlass.Array, +): + tidx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tidx % 32 + if tidx == 0: + _stamp(tdbg, bx, 0) + if cutlass.const_expr(TIMING): + if tidx == 0: + tdbg.store( + cutlass.Int64(_smid()), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(63) + ) + s_big = cutlass.Array(cutlass.Int8, BIG, space=cutlass.AddressSpace.smem, alignment=1024) + sA = cutlass.Array( + s_big.data_ptr(0), shape=(NUM_BYTES_A * STAGES,), dtype=cutlass.Int8, alignment=1024 + ) + sSFAraw = cutlass.Array( + s_big.data_ptr(_OFF_SFAR), + shape=(max(1, _RAW_STAGE // 4) * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sSFA = cutlass.Array( + s_big.data_ptr(_OFF_SFA), + shape=(NUM_WORDS_SFA * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sB1 = cutlass.Array( + s_big.data_ptr(_OFF_B1), shape=(NUM_BYTES_B1 * S1,), dtype=cutlass.Int8, alignment=1024 + ) + sB2 = cutlass.Array( + s_big.data_ptr(_OFF_B2), shape=(2 * KT2 * NUM_BYTES_B2,), dtype=cutlass.Int8, alignment=1024 + ) + sSFB2 = cutlass.Array( + s_big.data_ptr(_OFF_SFB2), + shape=(2 * KT2 * NUM_KB * 128,), + dtype=cutlass.Int32, + alignment=1024, + ) + ys = cutlass.Array( + cutlass.Float32, NB2_CAP * M_MAX * TOP_K * R2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_part = cutlass.Array( + cutlass.Float32, 2 * M_MAX * C1 * R1, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_bias = cutlass.Array(cutlass.Float32, E_LOCAL, space=cutlass.AddressSpace.smem, alignment=16) + s_red = cutlass.Array(cutlass.Float32, 9 * M_MAX, space=cutlass.AddressSpace.smem, alignment=16) + s_xsf = cutlass.Array( + cutlass.Int32, + M_MAX * (H // 64) if FUSED_ROUTER else 4, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + ab_full = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + ab_empty = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + sfa_ready = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + acc_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_ready = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_free = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + fc1_done = cutlass.Array(cutlass.Int64, 4, space=cutlass.AddressSpace.smem, alignment=8) + b1_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + x_bar = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + img_bar = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + s_el = cutlass.Array(cutlass.Int32, G_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + s_kpos = cutlass.Array( + cutlass.Int32, G_CAP * M_MAX, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_w = cutlass.Array( + cutlass.Float32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_ids = cutlass.Array( + cutlass.Int32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_plan = cutlass.Array(cutlass.Int32, NG_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + s_misc = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_plan_all = cutlass.Array( + cutlass.Int32, (G_CAP + 1) * (NG_CAP + 1), space=cutlass.AddressSpace.smem, alignment=16 + ) + s_esc = cutlass.Array(cutlass.Float32, 8 * G_CAP, space=cutlass.AddressSpace.smem, alignment=32) + is_routed = cutlass.Boolean(True) + if cutlass.const_expr(SHARED): + sh_full = cutlass.Array(cutlass.Int64, 32, space=cutlass.AddressSpace.smem, alignment=8) + is_routed = bx < cutlass.Int32(NCTA) + if bx >= cutlass.Int32(NCTA): + _shared_cta( + tidx, + warp, + lane, + bx - cutlass.Int32(NCTA), + s_big, + sh_full, + epochs, + tdbg, + bx, + xb32, + tma_gu_desc.get_ptr(), + tma_dn_desc.get_ptr(), + gusf, + dnsf, + hx, + dnx, + arp, + tmem_ptr_i32, + guw, + dnw, + ) + if is_routed: + if warp == 0: + if tidx < STAGES: + prims.mbarrier_init(ab_full.subview(tidx), 33 if SFA_CP else 1) + prims.mbarrier_init(ab_empty.subview(tidx), 1) + prims.mbarrier_init(sfa_ready.subview(tidx), 1) + if tidx < 2: + prims.mbarrier_init(acc_full.subview(tidx), 1) + prims.mbarrier_init(acc_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(acc2_full.subview(tidx), 1) + prims.mbarrier_init(acc2_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(b2_ready.subview(tidx), 1) + prims.mbarrier_init(b2_free.subview(tidx), EPI_THREADS) + if cutlass.const_expr(B2IMG): + prims.mbarrier_init(img_bar.subview(tidx), 1) + if tidx < 4: + prims.mbarrier_init(fc1_done.subview(tidx), EPI_THREADS) + if tidx == 0: + prims.mbarrier_init(b1_ready.subview(0), EPI_THREADS) + if cutlass.const_expr(XSTAGE): + prims.mbarrier_init(x_bar.subview(0), 1) + prims.mbarrier_init(x_bar.subview(1), EPI_THREADS) + if warp == MMA_WARP: + prims.tcgen05_alloc(tmem_ptr_i32, TMEM_COLS) + prims.tcgen05_relinquish_alloc_permit() + if warp == W_WARP: + prims.prefetch_tensormap(tma_a1_desc.get_ptr()) + prims.prefetch_tensormap(tma_a2_desc.get_ptr()) + prims.prefetch_tensormap(tma_s1_desc.get_ptr()) + prims.prefetch_tensormap(tma_s2_desc.get_ptr()) + for i in cutlass.range_constexpr(E_LOCAL // 32): + s_bias.store(cutlass.Float32(bias.load(idx=lane + i * 32)), idx=lane + i * 32) + for i in cutlass.range_constexpr(((G_CAP + 1) * (NG_CAP + 1) + 31) // 32): + q = lane + cutlass.Int32(32 * i) + if q < cutlass.Int32((G_CAP + 1) * (NG_CAP + 1)): + gq = q // cutlass.Int32(NG_CAP + 1) + s_plan_all.store( + plan.load( + idx=(gq * cutlass.Int32(NCTA) + bx) * cutlass.Int32(NG_CAP + 1) + + q + - gq * cutlass.Int32(NG_CAP + 1) + ), + idx=q, + ) + rwv = [] + if cutlass.const_expr(FUSED_ROUTER): + tg = cutlass.select_(tidx < cutlass.Int32(GEMV_THREADS), tidx, cutlass.Int32(0)) + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + rbx = cutlass.select_(eg < cutlass.Int32(NROUTE), eg, cutlass.Int32(0)) + rwv.append( + [ + rw.load( + idx=rbx * cutlass.Int32(H) + + tg * cutlass.Int32(4) + + cutlass.Int32(1024 * j), + vector_size=4, + alignment=16, + ) + for j in range(H // 1024) + ] + ) + if cutlass.const_expr(TIMING): + if tidx == 0: + s_el.store(cutlass.Int32(-7), idx=G_CAP) + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 65535 + base_row = tmem_raw >> 16 + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(XSTAGE): + if warp == 0: + if lane == 0: + prims.mbarrier_arrive_expect_tx(x_bar.subview(0), M_MAX * H * 2) + cute.arch.sync_warp() + if lane < M_MAX: + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(lane * cutlass.Int32(H * 2)), + xb32.subview(lane * cutlass.Int32(H // 2)), + x_bar.subview(0), + H * 2, + ) + ar_on = arp.load(idx=0) != cutlass.Int64(0) + ar_off = arp.load(idx=0) == cutlass.Int64(0) + if cutlass.const_expr(not LATE_TRIGGER): + if ar_off: + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + tagv = (ep & cutlass.Int32(127) | cutlass.Int32(128)) << cutlass.Int32(24) + tag16 = ep & cutlass.Int32(32767) | cutlass.Int32(32768) + ltag = cutlass.Int64(ep + cutlass.Int32(1)) << cutlass.Int64(32) + if cutlass.const_expr(FUSED_ROUTER): + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + if (eg < cutlass.Int32(NROUTE)) & (tidx < cutlass.Int32(GEMV_THREADS)): + accs = [cutlass.Float32(0.0) for _ in range(M_MAX)] + xsrc = xb32 + if cutlass.const_expr(GEMV_SMEM and XSTAGE): + _wait(x_bar.subview(0), 0) + xsrc = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + for j in cutlass.range_constexpr(H // 1024): + for t in cutlass.range_constexpr(M_MAX): + xw = xsrc.load( + idx=cutlass.Int32(t * (H // 2) + 512 * j) + tidx * cutlass.Int32(2), + vector_size=2, + alignment=8, + ) + x0 = cutlass.Int32(xw[0]) + x1 = cutlass.Int32(xw[1]) + a = accs[t] + a = _fma_rn( + (x0 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][0]), + a, + ) + a = _fma_rn( + (x0 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][1]), + a, + ) + a = _fma_rn( + (x1 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][2]), + a, + ) + a = _fma_rn( + (x1 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][3]), + a, + ) + accs[t] = a + for t in cutlass.range_constexpr(M_MAX): + v = accs[t] + for off in (16, 8, 4, 2, 1): + v = _add_rn(v, cute.arch.shuffle_sync_bfly(v, off)) + if lane == 0: + s_red.store(v, idx=warp * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < M_MAX: + sacc = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(8): + sacc = _add_rn(sacc, s_red.load(idx=cutlass.Int32(w * M_MAX) + tidx)) + s_red.store(sacc, idx=cutlass.Int32(8 * M_MAX) + tidx) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < LG_REPL: + for t in cutlass.range_constexpr(M_MAX): + lw = ltag | cutlass.Int64( + cutlass.Float32(s_red.load(idx=8 * M_MAX + t)).bitcast( + cutlass.Int32 + ) + ) & cutlass.Int64(4294967295) + _st_relaxed_u64( + lgbuf.subview( + tidx * cutlass.Int32(LG_STRIDE) + + cutlass.Int32(t * E_LOCAL) + + eg + ) + .data_ptr() + .toint(), + lw, + ) + if cutlass.const_expr(NGEMV > 1): + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx == 0: + _stamp(tdbg, bx, 21) + if (warp >= W_WARP) & (warp < cutlass.Int32(W_WARP + NRW0)) | (warp >= cutlass.Int32(8)) & ( + warp < cutlass.Int32(8 + RW_EXTRA) + ): + rw_i = cutlass.select_( + warp < cutlass.Int32(8), warp - cutlass.Int32(W_WARP), warp - cutlass.Int32(4) + ) + if cutlass.const_expr(B1WIDE): + _wait(x_bar.subview(0), 0) + xw32 = cutlass.Array( + s_big.data_ptr(0), shape=(M_MAX * H // 2,), dtype=cutlass.Int32, alignment=16 + ) + _b1_quant(tidx, THREADS, xw32, sfsc, sB1, s_xsf) + _bar_arrive(B1W_BAR, THREADS) + lgv_all = [] + t_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = rw_i + cutlass.Int32(NRW * tr) + if cutlass.const_expr(M_MAX % NRW != 0): + t = cutlass.select_(t < cutlass.Int32(M_MAX), t, cutlass.Int32(M_MAX - 1)) + t_all.append(t) + lgv = [] + if cutlass.const_expr(FUSED_ROUTER): + want = ltag >> cutlass.Int64(32) + rbase = ( + bx % cutlass.Int32(LG_REPL) * cutlass.Int32(LG_STRIDE) + + t * cutlass.Int32(E_LOCAL) + + lane + ) + done = cutlass.Int32(0) + res = [cutlass.Int64(0) for _ in range(4)] + while done == cutlass.Int32(0): + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(4): + res[z] = _ld_relaxed_u64( + lgbuf.subview(cutlass.Int32(z * 32) + rbase).data_ptr().toint() + ) + ok = ok & (res[z] >> cutlass.Int64(32) == want) + if prims.vote_sync(4294967295, ok, prims.VoteSync.BALLOT) == cutlass.Int32( + -1 + ): + done = cutlass.Int32(1) + else: + _nanosleep(POLL_NS) + for z in cutlass.range_constexpr(4): + lgv.append(cutlass.Int32(res[z]).bitcast(cutlass.Float32)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 22) + else: + for i in cutlass.range_constexpr(4): + lgv.append( + cutlass.Float32( + logits.load( + idx=t * cutlass.Int32(E_LOCAL) + cutlass.Int32(i * 32) + lane + ) + ) + ) + lgv_all.append(lgv) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 61) + res_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = t_all[tr] + lgv = lgv_all[tr] + own = rw_i + cutlass.Int32(NRW * tr) < cutlass.Int32(M_MAX) + keys = [] + for i in cutlass.range_constexpr(4): + sg = _fma_rn( + cutlass.Float32(0.5), + cute.math.tanh(_mul_rn(cutlass.Float32(0.5), lgv[i])), + cutlass.Float32(0.5), + ) + keys.append(_add_rn(sg, s_bias.load(idx=i * 32 + lane))) + hi = [_twiddle(keys[i]) for i in range(4)] + taken = [cutlass.Boolean(False) for _ in range(4)] + ids_t = [] + kt_t = [] + for k in cutlass.range_constexpr(TOP_K): + bh = cutlass.Int32(0) + bl = cutlass.Int32(0) + for i in cutlass.range_constexpr(4): + lo_i = cutlass.Int32(65535 - i * 32) - lane + hv = cutlass.select_(taken[i], cutlass.Int32(0), hi[i]) + lv = cutlass.select_(taken[i], cutlass.Int32(0), lo_i) + better = (cutlass.Uint32(hv) > cutlass.Uint32(bh)) | (hv == bh) & ( + cutlass.Uint32(lv) > cutlass.Uint32(bl) + ) + bh = cutlass.select_(better, hv, bh) + bl = cutlass.select_(better, lv, bl) + mh = _redux_max_u32(bh) + ml = _redux_max_u32(cutlass.select_(bh == mh, bl, cutlass.Int32(0))) + e_k = cutlass.Int32(65535) - ml + for i in cutlass.range_constexpr(4): + taken[i] = taken[i] | (e_k == cutlass.Int32(i * 32) + lane) + ids_t.append(e_k) + kv = cutlass.Float32(0.0) + for i in cutlass.range_constexpr(4): + kv = cutlass.select_(e_k == cutlass.Int32(i * 32) + lane, keys[i], kv) + kt_t.append(cute.arch.shuffle_sync(kv, e_k % cutlass.Int32(32))) + sig = cutlass.Float32(0.0) + e_l = cutlass.Int32(0) + for k in cutlass.range_constexpr(TOP_K): + sk = _sub_rn(kt_t[k], s_bias.load(idx=ids_t[k])) + sig = cutlass.select_(lane == cutlass.Int32(k), sk, sig) + e_l = cutlass.select_(lane == cutlass.Int32(k), ids_t[k], e_l) + total = sig + for off in (16, 8, 4, 2, 1): + total = _add_rn(total, cute.arch.shuffle_sync_bfly(total, off)) + wv = _bf16_round( + _div_rn( + _mul_rn(sig, cutlass.Float32(ROUTE_SCALE)), + _add_rn(total, cutlass.Float32(SUM_EPS)), + ) + ) + res_all.append((t, own, wv, e_l)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 62) + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t, own, wv, e_l = res_all[tr] + if (lane < TOP_K) & own: + s_w.store(wv, idx=t * cutlass.Int32(TOP_K) + lane) + s_ids.store(e_l, idx=t * cutlass.Int32(TOP_K) + lane) + if bx == 0: + route_dbg.store( + wv.bitcast(cutlass.Int32), + idx=t * cutlass.Int32(8) + cutlass.Int32(4) + lane, + ) + cute.arch.sync_warp() + if cutlass.const_expr(EARLY0 and (not E0_OFFLOAD)): + if warp == cutlass.Int32(W_WARP): + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(65535) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - ug // cutlass.Int32(U1) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(W_WARP): + _bar_arrive(E0_BAR, 64) + if cutlass.const_expr(B1_AFTER_TOPK): + _bar_arrive(TOPK2_BAR, 128 + 32 * NRW) + cute.arch.barrier(barrier_id=TOPK_BAR, number_of_threads=32 * NRW) + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(E0_WARP): + cute.arch.barrier(barrier_id=E0_BAR, number_of_threads=64) + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(65535) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - ug // cutlass.Int32(U1) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + _bar_arrive(E1_BAR, 64) + if warp == W_WARP: + if lane == 0: + _stamp(tdbg, bx, 20) + valid = lane < cutlass.Int32(TOP_K * M_MAX) + my_e = cutlass.select_( + valid, + s_ids.load(idx=cutlass.select_(valid, lane, cutlass.Int32(0))), + cutlass.Int32(-1) - lane, + ) + same = _match_any(my_e) + leader = cute.arch.popc((same & cutlass.Int32(0) - same) - cutlass.Int32(1)) + first = valid & (lane == leader) + bal = prims.vote_sync(4294967295, first, prims.VoteSync.BALLOT) + rank = cute.arch.popc(bal & cutlass.Int32(cute.arch.lanemask_lt())) + slot_l = cute.arch.shuffle_sync(rank, leader) + for z in cutlass.range_constexpr((G_CAP * M_MAX + 31) // 32): + if lane + cutlass.Int32(32 * z) < cutlass.Int32(G_CAP * M_MAX): + s_kpos.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + cute.arch.sync_warp() + if first: + s_el.store(my_e, idx=rank) + if valid: + s_kpos.store( + lane % cutlass.Int32(TOP_K), + idx=slot_l * cutlass.Int32(M_MAX) + lane // cutlass.Int32(TOP_K), + ) + if bx == 0: + route_dbg.store( + my_e, + idx=lane // cutlass.Int32(TOP_K) * cutlass.Int32(8) + + lane % cutlass.Int32(TOP_K), + ) + gcnt = cute.arch.popc(bal) + if lane == 0: + s_el.store(gcnt, idx=G_CAP) + if lane < cutlass.Int32(NG_CAP + 1): + s_plan.store(s_plan_all.load(idx=gcnt * cutlass.Int32(NG_CAP + 1) + lane), idx=lane) + cute.arch.sync_warp() + if cutlass.const_expr(B2IMG): + if (bx == 0) & (lane < cutlass.Int32(NG_CAP)): + nh = gcnt - lane * cutlass.Int32(4) + nh = cutlass.select_(nh > cutlass.Int32(4), cutlass.Int32(4), nh) + nh = cutlass.select_(nh < cutlass.Int32(0), cutlass.Int32(0), nh) + pad = cutlass.Int32(CNT_PER_CALL) - nh * cutlass.Int32(UNITS_ROW) + if pad > cutlass.Int32(0): + _red_release_add( + b2cnt.subview(lane * cutlass.Int32(8)).data_ptr().toint(), pad + ) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 24) + cute.arch.sync_warp() + _bar_arrive(ROUTE_BAR, THREADS) + if warp < 4: + if cutlass.const_expr(B1_AFTER_TOPK): + cute.arch.barrier(barrier_id=TOPK2_BAR, number_of_threads=128 + 32 * NRW) + if cutlass.const_expr(FUSED_ROUTER): + if tidx == 0: + _stamp(tdbg, bx, 57) + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(0), 0) + if tidx == 0: + _stamp(tdbg, bx, 58) + xs32 = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + if cutlass.const_expr(B1WIDE): + _b1_quant(tidx, THREADS, xs32, sfsc, sB1, s_xsf) + cute.arch.barrier(barrier_id=B1W_BAR, number_of_threads=THREADS) + _sfb1_to_tmem(warp, lane, base_row, base_col, s_xsf, b1_ready) + if cutlass.const_expr(not B1WIDE): + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xs32, sfsc, sB1, s_xsf, b1_ready + ) + prims.mbarrier_arrive(x_bar.subview(1)) + else: + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xb32, sfsc, sB1, s_xsf, b1_ready + ) + if tidx == 0: + _stamp(tdbg, bx, 23) + else: + _build_b1(tidx, warp, lane, base_row, base_col, x32, xsf32, sB1, b1_ready) + if warp != W_WARP: + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ROUTE_BAR, number_of_threads=THREADS) + if cutlass.const_expr(EARLY_EPOCH): + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + if tidx == 0: + _stamp(tdbg, bx, 2) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(s_el.load(idx=G_CAP)), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(28), + ) + g_n = s_el.load(idx=G_CAP) + ng = g_n + cutlass.Int32(3) >> cutlass.Int32(2) + pr2 = s_plan.load(idx=NG_CAP) + blk0 = pr2 & cutlass.Int32(65535) + nb = pr2 >> cutlass.Int32(16) + if warp == W_WARP: + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + stage = cutlass.Int32(0) + phase = cutlass.Int32(1) + skip = cutlass.Int32(0) + if cutlass.const_expr(E0_OFFLOAD): + cute.arch.barrier(barrier_id=E1_BAR, number_of_threads=64) + if cutlass.const_expr(EARLY0): + skip = s_misc.load(idx=0) + stage = cutlass.select_(skip == cutlass.Int32(STAGES), cutlass.Int32(0), skip) + phase = cutlass.select_( + skip == cutlass.Int32(STAGES), cutlass.Int32(0), cutlass.Int32(1) + ) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + c = ug - ug // cutlass.Int32(U1) * cutlass.Int32(U1) + e = s_el.load(idx=slot) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if skip > cutlass.Int32(0): + skip = skip - cutlass.Int32(1) + else: + _waitd(ab_empty.subview(stage), phase, tdbg, 1) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + ), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, stage, sf_line, c, sSFA, s1w, ab_full) + stage, phase = _adv(stage, phase) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + for bb in range(nb): + blk = blk0 + bb + for t2 in cutlass.range_constexpr(KT2): + _waitd(ab_empty.subview(stage), phase, tdbg, 2) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + n_here + * cutlass.Int32( + NUM_BYTES_BOX1 + if SFA_CP + else NUM_BYTES_BOX1 + NUM_BYTES_SFA_RAW // 4 + ), + ) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(jj * NUM_BYTES_BOX1) + ), + tma_a2_desc.get_ptr(), + ( + cutlass.Int32(t2 * KT_BYTES), + blk * cutlass.Int32(R2), + e, + ), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + cutlass.Int32(jj * NUM_WORDS_SFA_RAW // 4) + ), + tma_s2_desc.get_ptr(), + ( + cutlass.Int32(0), + ( + ( + e * cutlass.Int32(SF2_RB) + + (blk >> cutlass.Int32(2)) + ) + * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(4), + ), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf2_cp(lane, stage, g, blk, t2, n_here, s_el, sSFA, s2w, ab_full) + stage, phase = _adv(stage, phase) + if lane == 0: + _stamp(tdbg, bx, 3) + if cutlass.const_expr(TAIL_TRIGGER): + if ar_on: + cute.arch.griddepcontrol_launch_dependents() + if (warp == S_WARP) & cutlass.Boolean(not SFA_CP): + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + for p in range(ng + cutlass.Int32(LAG)): + i0 = cutlass.Int32(0) + i1 = cutlass.Int32(0) + for part in cutlass.range_constexpr(2): + cnt = cutlass.Int32(0) + m1 = cutlass.Int32(0) + nk = cutlass.Int32(0) + if cutlass.const_expr(part == 0): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + nk = _nsegs(i0, i1) + elif p >= cutlass.Int32(LAG): + nk = nb + for k in range(nk): + if cutlass.const_expr(part == 0): + ug, j0, j1 = _seg(i0, i1, k) + m1 = ug - ug // cutlass.Int32(U1) * cutlass.Int32(U1) & cutlass.Int32(3) + cnt = j1 - j0 + else: + m1 = blk0 + k & cutlass.Int32(3) + cnt = cutlass.Int32(KT2) + for _j in cutlass.range(cnt, unroll=1): + _waitd(ab_full.subview(stage), phase, tdbg, 3) + raw0 = ( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + lane * cutlass.Int32(4) + + m1 + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for kb in cutlass.range_constexpr(NUM_KB): + w0 = sSFAraw.load(idx=raw0 + cutlass.Int32((0 * 4 + kb) * 128)) + w1 = sSFAraw.load(idx=raw0 + cutlass.Int32((1 * 4 + kb) * 128)) + w2 = sSFAraw.load(idx=raw0 + cutlass.Int32((2 * 4 + kb) * 128)) + w3 = sSFAraw.load(idx=raw0 + cutlass.Int32((3 * 4 + kb) * 128)) + sSFA.store( + ( + cutlass.Int32(w0), + cutlass.Int32(w1), + cutlass.Int32(w2), + cutlass.Int32(w3), + ), + idx=dst0 + cutlass.Int32(kb * 128), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + prims.tcgen05_commit(sfa_ready.subview(stage)) + stage, phase = _adv(stage, phase) + if cutlass.const_expr(B2IMG): + if warp == cutlass.Int32(S_WARP): + for p in range(ng): + i0, i1 = _plan_range(s_plan, p) + n_done = cutlass.Int32(0) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + n_done = n_done + cutlass.select_( + (j0 != cutlass.Int32(0)) | (j1 == cutlass.Int32(S1)), + cutlass.Int32(1), + cutlass.Int32(0), + ) + _wait( + fc1_done.subview(p & cutlass.Int32(3)), + p >> cutlass.Int32(2) & cutlass.Int32(1), + ) + if cutlass.const_expr(B2_DLY > 0): + t0d = _gtime() + dl = cutlass.Int64( + ( + bx * cutlass.Int32(37) + p * cutlass.Int32(11) + ep + & cutlass.Int32(63) + ) + * cutlass.Int32(B2_DLY // 64) + ) + while _gtime() - t0d < dl: + pass + if (lane == 0) & (n_done > cutlass.Int32(0)): + _fence_proxy_async_global() + _red_release_add( + b2cnt.subview(p * cutlass.Int32(8)).data_ptr().toint(), n_done + ) + cute.arch.sync_warp() + if warp == cutlass.Int32(X_WARP): + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + cnt_target = (ep + cutlass.Int32(1)) * cutlass.Int32(CNT_PER_CALL) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + (g >> cutlass.Int32(1)) - cutlass.Int32(1) & cutlass.Int32(1), + tdbg, + 101, + ) + if lane == 0: + _stamp(tdbg, bx, 29) + caddr = b2cnt.subview(g * cutlass.Int32(8)).data_ptr().toint() + if lane == 0: + cv = _ld_acquire_u32(caddr) + while cv - cnt_target < cutlass.Int32(0): + _nanosleep(B2IMG_NS) + cv = _ld_acquire_u32(caddr) + _fence_proxy_async_global() + prims.mbarrier_arrive_expect_tx(img_bar.subview(buf), IMG_BYTES) + prims.cp_async_bulk_shared_cluster_global( + sB2.subview(buf * cutlass.Int32(IMG_B2)), + img32.subview(g * cutlass.Int32(IMG_BYTES // 4)), + img_bar.subview(buf), + IMG_B2, + ) + prims.cp_async_bulk_shared_cluster_global( + sSFB2.subview(buf * cutlass.Int32(IMG_SF // 4)), + img32.subview( + g * cutlass.Int32(IMG_BYTES // 4) + cutlass.Int32(IMG_B2 // 4) + ), + img_bar.subview(buf), + IMG_SF, + ) + cute.arch.sync_warp() + _waitd( + img_bar.subview(buf), g >> cutlass.Int32(1) & cutlass.Int32(1), tdbg, 108 + ) + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 25) + if ((warp == X_WARP) | (warp >= cutlass.Int32(8))) & cutlass.Boolean(not B2IMG): + xi = cutlass.select_( + warp == cutlass.Int32(X_WARP), cutlass.Int32(0), warp - cutlass.Int32(7) + ) + if cutlass.const_expr(G_CAP > TOP_K): + if (bx == 0) & (xi == cutlass.Int32(0)): + for i in range( + g_n * cutlass.Int32(M_MAX * UNITS_ROW) + lane, + cutlass.Int32(G_CAP * M_MAX * UNITS_ROW), + 32, + ): + for rr in cutlass.range_constexpr(HB_REPL): + hbuf32.store( + ( + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + ), + idx=i * cutlass.Int32(4) + + (cutlass.Int32(rr * HB_STRIDE) if rr else 0), + alignment=16, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sSFB2_8 = cutlass.Array( + sSFB2.data_ptr(0), shape=(2 * KT2 * NUM_KB * 512,), dtype=cutlass.Int8, alignment=16 + ) + sB2_32 = cutlass.Array( + sB2.data_ptr(0), + shape=(2 * KT2 * NUM_BYTES_B2 // 4,), + dtype=cutlass.Int32, + alignment=16, + ) + nrec_cap = 4 * M_MAX * UNITS_ROW + hb_off = cutlass.Int32(0) + if cutlass.const_expr(HB_REPL > 1): + hb_off = bx % cutlass.Int32(HB_REPL) * cutlass.Int32(HB_STRIDE) + NZ = (nrec_cap + 32 * XW - 1) // (32 * XW) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + (g >> cutlass.Int32(1)) - cutlass.Int32(1) & cutlass.Int32(1), + tdbg, + 101, + ) + _waitd( + fc1_done.subview(g & cutlass.Int32(3)), + g >> cutlass.Int32(2) & cutlass.Int32(1), + tdbg, + 4, + ) + if cutlass.const_expr(TIMING): + if (lane == 0) & (xi == cutlass.Int32(0)) & (g < cutlass.Int32(8)): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(32) + g + ) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + nrec = n_here * cutlass.Int32(M_MAX * UNITS_ROW) + rec0 = g * cutlass.Int32(4 * M_MAX * UNITS_ROW) + done = cutlass.Int32(0) + rspin = cutlass.Int32(0) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 29) + npoll = cutlass.Int32(0) + got = [[cutlass.Int32(0) for _ in range(3)] for _ in range(NZ)] + while done == cutlass.Int32(0): + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + v4 = prims.load_ext( + hbuf32.subview( + (rec0 + qi) * cutlass.Int32(4) + (hb_off if HB_REPL > 1 else 0) + ), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for wq in cutlass.range_constexpr(3): + got[z][wq] = cutlass.Int32(v4[wq]) + ok = ok & ( + (got[z][0] ^ tagv | got[z][1] ^ tagv | got[z][2] ^ tagv) + & cutlass.Int32(-16777216) + == cutlass.Int32(0) + ) + npoll = npoll + cutlass.Int32(1) + if prims.vote_sync(4294967295, ok, prims.VoteSync.BALLOT) == cutlass.Int32(-1): + done = cutlass.Int32(1) + else: + _nanosleep(B2_POLL_NS) + if cutlass.const_expr(HANG_DBG): + rspin = rspin + cutlass.Int32(1) + if rspin > cutlass.Int32(1 << 14): + done = cutlass.Int32(1) + tdbg.store( + cutlass.Int64(200), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + warp, + ) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + w0, w1, w2 = (got[z][0], got[z][1], got[z][2]) + rr = qi // cutlass.Int32(UNITS_ROW) + c = qi - rr * cutlass.Int32(UNITS_ROW) + t2 = c >> cutlass.Int32(4) + cc = c >> cutlass.Int32(1) & cutlass.Int32(7) + lo = w0 & cutlass.Int32(16777215) | w1 << cutlass.Int32(24) + hi = w1 >> cutlass.Int32(8) & cutlass.Int32(65535) | ( + w2 & cutlass.Int32(65535) + ) << cutlass.Int32(16) + sB2_32.store( + (lo, hi), + idx=( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_BYTES_B2) + + rr * cutlass.Int32(KT_BYTES) + + (cc ^ rr & cutlass.Int32(7)) * cutlass.Int32(16) + + (c & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + sSFB2_8.store( + cutlass.Int8(w2 >> cutlass.Int32(16) & cutlass.Int32(255)), + idx=( + ( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_KB) + + ((c & cutlass.Int32(15)) >> cutlass.Int32(2)) + ) + * cutlass.Int32(128) + + rr * cutlass.Int32(4) + ) + * cutlass.Int32(4) + + (c & cutlass.Int32(3)), + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + if cutlass.const_expr(XW > 1): + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=B2_BAR, number_of_threads=32 * XW) + cute.arch.sync_warp() + if (xi == cutlass.Int32(0)) & prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + SFB2_COL + buf * cutlass.Int32(KT2 * NUM_KB * 4) + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 25) + if cutlass.const_expr(TIMING): + if g < cutlass.Int32(8): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(11) + g + ) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(npoll), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(30), + ) + if warp == MMA_WARP: + idesc1 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N1, + m_dim=MMA_M, + sparsity_version=0, + ) + idesc2 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N2, + m_dim=MMA_M, + sparsity_version=0, + ) + desc_a0 = prims.Tcgen05SmemDesc.build( + sA.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b10 = prims.Tcgen05SmemDesc.build( + sB1.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b20 = prims.Tcgen05SmemDesc.build( + sB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = base_row << 16 | base_col + _waitd(b1_ready, 0, tdbg, 5) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + acc_b2 = cutlass.Int64(0) + acc_sf = cutlass.Int64(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + _waitd( + acc_empty.subview(ab), + k1 >> cutlass.Int32(1) & cutlass.Int32(1) ^ cutlass.Int32(1), + tdbg, + 102, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc_ptr = cutlass.inttoptr( + tcol + ACC1_COL + ab * cutlass.Int32(N1), 6, cutlass.Float32 + ) + for j in cutlass.range(j0, j1, unroll=1): + tw0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 6) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 6) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b10 + cutlass.Int64(j * cutlass.Int32(NUM_BYTES_B1 >> 4)) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = tcol + SFB1_COL + j * cutlass.Int32(NUM_KB * 4) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc1, + (j > j0) | cutlass.Boolean(kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc_full.subview(ab)) + k1 = k1 + cutlass.Int32(1) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + buf = g & cutlass.Int32(1) + tw2 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + _waitd( + b2_ready.subview(buf), g >> cutlass.Int32(1) & cutlass.Int32(1), tdbg, 103 + ) + if cutlass.const_expr(TIMING): + acc_b2 = acc_b2 + (_gtime() - tw2) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 4) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_empty.subview(a2), + k2 >> cutlass.Int32(1) & cutlass.Int32(1) ^ cutlass.Int32(1), + tdbg, + 104, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc2_ptr = cutlass.inttoptr( + tcol + ACC2_COL + a2 * cutlass.Int32(N2), 6, cutlass.Float32 + ) + for t2 in cutlass.range_constexpr(KT2): + tw1 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 7) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 7) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw1) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b20 + cutlass.Int64( + (buf * cutlass.Int32(KT2) + cutlass.Int32(t2)) + * cutlass.Int32(NUM_BYTES_B2 >> 4) + ) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = ( + tcol + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + cutlass.Int32(t2 * NUM_KB * 4) + ) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(_nkb2(t2)): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc2_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc2, + cutlass.Boolean(t2 > 0 or kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc2_full.subview(a2)) + k2 = k2 + cutlass.Int32(1) + if cutlass.const_expr(TIMING): + if lane == 0: + tdbg.store(acc_b2, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(26)) + tdbg.store(acc_sf, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(27)) + if warp < 4: + row_w = base_row + warp * 32 + for r in cutlass.range_constexpr((6 * G_CAP + EPI_THREADS - 1) // EPI_THREADS): + qi = tidx + cutlass.Int32(r * EPI_THREADS) + sl_e = qi // cutlass.Int32(6) + if (qi < cutlass.Int32(6 * G_CAP)) & (sl_e < g_n): + fld = qi - sl_e * cutlass.Int32(6) + s_esc.store( + cutlass.Float32( + escal.load(idx=fld * cutlass.Int32(E_LOCAL) + s_el.load(idx=sl_e)) + ), + idx=sl_e * cutlass.Int32(8) + fld, + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + SH_ADD = SHARED + wys = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + pars = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + ard = (cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0)) + for p in range(ng + cutlass.Int32(LAG)): + if cutlass.const_expr(AR_PRE): + if (p == ng + cutlass.Int32(LAG - 1)) & ar_on: + ard = tuple( + ( + cutlass.Int64(_ld_relaxed_u64(arp.subview(i_).data_ptr().toint())) + for i_ in (1, 3, 4, 5) + ) + ) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_cl = cutlass.select_( + f_bb < nb, (blk0 + f_bb) * cutlass.Int32(R2), cutlass.Int32(0) + ) + pars[bt] = _ar_parity(arp, f_t, f_cl) + if cutlass.const_expr(SH_ADD): + if p == ng + cutlass.Int32(LAG - 1): + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_ch = cutlass.select_( + f_bb < nb, + f_t * cutlass.Int32(H) + + (blk0 + f_bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + cutlass.Int32(0), + ) + wys[bt] = _ld_relaxed_u32(dnx.subview(f_ch).data_ptr().toint()) + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + cblk = ug - ug // cutlass.Int32(U1) * cutlass.Int32(U1) + unit = slot * cutlass.Int32(U1) + cblk + _waitd( + acc_full.subview(ab), + k1 >> cutlass.Int32(1) & cutlass.Int32(1), + tdbg, + 105, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, base_col + ACC1_COL + ab * cutlass.Int32(N1), cutlass.Float32 + ), + num=N1, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc_empty.subview(ab)) + pb = ab * cutlass.Int32(M_MAX * C1 * R1) + for t in cutlass.range_constexpr(M_MAX): + part = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(4): + part = cutlass.select_( + warp == cutlass.Int32(w), cutlass.Float32(acc[QS * w + t]), part + ) + s_part.store(part, idx=pb + (t * C1 + warp) * R1 + lane) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + for tr in cutlass.range_constexpr((M_MAX + 3) // 4): + t = warp + cutlass.Int32(4 * tr) + if t < cutlass.Int32(M_MAX): + xv = _add_rn( + _add_rn( + _add_rn( + s_part.load(idx=pb + (t * C1 + 0) * R1 + lane), + s_part.load(idx=pb + (t * C1 + 1) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 2) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 3) * R1 + lane), + ) + finish = cutlass.Int32(1) + if (j0 != cutlass.Int32(0)) | (j1 != cutlass.Int32(S1)): + paddr = ( + partials.subview( + (unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane + ) + .data_ptr() + .toint() + ) + if j0 == cutlass.Int32(0): + _st_relaxed_u32(paddr, xv.bitcast(cutlass.Int32)) + finish = cutlass.Int32(0) + else: + pbits = _ld_relaxed_u32(paddr) + spins = cutlass.Int32(0) + while pbits == cutlass.Int32(PARTIAL_EMPTY): + pbits = _ld_relaxed_u32(paddr) + spins = spins + cutlass.Int32(1) + _st_relaxed_u32(paddr, cutlass.Int32(PARTIAL_EMPTY)) + xv = _add_rn(pbits.bitcast(cutlass.Float32), xv) + cute.arch.sync_warp() + if cutlass.const_expr(XVDUMP): + partials.store( + xv.bitcast(cutlass.Int32), + idx=(unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane, + ) + if finish == cutlass.Int32(1): + gate_raw = cute.arch.shuffle_sync_bfly(xv, 8) + is_up = lane >> cutlass.Int32(3) & cutlass.Int32( + 1 + ) == cutlass.Int32(0) + es = slot * cutlass.Int32(8) + qv, sfb = _swiglu_quant( + xv, + gate_raw, + is_up, + s_esc.load(idx=es), + s_esc.load(idx=es + cutlass.Int32(1)), + s_esc.load(idx=es + cutlass.Int32(2)), + s_esc.load(idx=es + cutlass.Int32(3)), + s_esc.load(idx=es + cutlass.Int32(4)), + ) + byte = _e2m1x2(cute.arch.shuffle_sync_down(qv, 16), qv) + bt = [ + cute.arch.shuffle_sync(byte, i) & cutlass.Int32(255) + for i in range(8) + ] + rw0 = ( + bt[0] + | bt[1] << cutlass.Int32(8) + | bt[2] << cutlass.Int32(16) + | tagv + ) + rw1 = ( + bt[3] + | bt[4] << cutlass.Int32(8) + | bt[5] << cutlass.Int32(16) + | tagv + ) + rw2 = ( + bt[6] + | bt[7] << cutlass.Int32(8) + | (sfb & cutlass.Int32(255)) << cutlass.Int32(16) + | tagv + ) + if cutlass.const_expr(B2IMG): + if lane == 0: + n = ug // cutlass.Int32(U1) * cutlass.Int32(M_MAX) + t + t2i = cblk >> cutlass.Int32(4) + cci = cblk >> cutlass.Int32(1) & cutlass.Int32(7) + gbase = p * cutlass.Int32(IMG_BYTES) + lo = ( + bt[0] + | bt[1] << cutlass.Int32(8) + | bt[2] << cutlass.Int32(16) + | bt[3] << cutlass.Int32(24) + ) + hi = ( + bt[4] + | bt[5] << cutlass.Int32(8) + | bt[6] << cutlass.Int32(16) + | bt[7] << cutlass.Int32(24) + ) + img32.store( + (lo, hi), + idx=( + gbase + + t2i * cutlass.Int32(NUM_BYTES_B2) + + n * cutlass.Int32(KT_BYTES) + + (cci ^ n & cutlass.Int32(7)) + * cutlass.Int32(16) + + (cblk & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + img8 = cutlass.Array( + img32.data_ptr(0), + shape=(NG_CAP * IMG_BYTES,), + dtype=cutlass.Int8, + alignment=16, + ) + img8.store( + cutlass.Int8(sfb & cutlass.Int32(255)), + idx=gbase + + cutlass.Int32(IMG_B2) + + ( + t2i * cutlass.Int32(NUM_KB) + + ( + (cblk & cutlass.Int32(15)) + >> cutlass.Int32(2) + ) + ) + * cutlass.Int32(512) + + n * cutlass.Int32(16) + + (cblk & cutlass.Int32(3)), + ) + elif cutlass.const_expr(HB_REPL == 1): + if lane == 0: + hbuf32.store( + (rw0, rw1, rw2, tagv), + idx=( + (slot * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(UNITS_ROW) + + cblk + ) + * cutlass.Int32(4), + alignment=16, + ) + elif lane < HB_REPL: + hbuf32.store( + (rw0, rw1, rw2, tagv), + idx=( + (slot * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(UNITS_ROW) + + cblk + ) + * cutlass.Int32(4) + + lane * cutlass.Int32(HB_STRIDE), + alignment=16, + ) + k1 = k1 + cutlass.Int32(1) + prims.mbarrier_arrive(fc1_done.subview(p & cutlass.Int32(3))) + if tidx == 0: + _stamp(tdbg, bx, 10) + if cutlass.const_expr(TIMING): + if p < cutlass.Int32(8): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(40) + p, + ) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_full.subview(a2), + k2 >> cutlass.Int32(1) & cutlass.Int32(1), + tdbg, + 106, + ) + if cutlass.const_expr(TIMING): + if (tidx == 0) & (g == ng - cutlass.Int32(1)): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(8) + bb, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + yv = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, + base_col + + ACC2_COL + + a2 * cutlass.Int32(N2) + + warp * cutlass.Int32(M_MAX), + cutlass.Float32, + ), + num=LD_POW2, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc2_empty.subview(a2)) + sl = g * cutlass.Int32(4) + warp + if sl < g_n: + a2s = s_esc.load(idx=sl * cutlass.Int32(8) + cutlass.Int32(5)) + for t in cutlass.range_constexpr(M_MAX): + kp = s_kpos.load(idx=sl * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + if kp >= cutlass.Int32(0): + ys.store( + _bf16_round(_mul_rn(cutlass.Float32(yv[t]), a2s)), + idx=( + (bb * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + * cutlass.Int32(TOP_K) + + kp + ) + * cutlass.Int32(R2) + + lane, + ) + k2 = k2 + cutlass.Int32(1) + prims.mbarrier_arrive(b2_free.subview(g & cutlass.Int32(1))) + if tidx == 0: + _stamp(tdbg, bx, 6) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + if tidx == 0: + _stamp(tdbg, bx, 19) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + pair = warp + cutlass.Int32(bt * 4) + bb = pair // cutlass.Int32(M_MAX) + t = pair - bb * cutlass.Int32(M_MAX) + if bb < nb: + acc_o = cutlass.Float32(0.0) + for k in cutlass.range_constexpr(TOP_K): + acc_o = _fma_rn( + s_w.load(idx=t * cutlass.Int32(TOP_K) + cutlass.Int32(k)), + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ), + acc_o, + ) + if cutlass.const_expr(YDUMP): + partials.store( + cutlass.Float32( + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ) + ).bitcast(cutlass.Int32), + idx=(t * cutlass.Int32(TOP_K) + cutlass.Int32(k)) * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + ) + ch = ( + t * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(SH_ADD): + wy = wys[bt] + while wy >> cutlass.Int32(16) & cutlass.Int32(65535) != tag16: + wy = _ld_relaxed_u32(dnx.subview(ch).data_ptr().toint()) + acc_o = _add_rn( + (wy << cutlass.Int32(16)).bitcast(cutlass.Float32), _bf16_round(acc_o) + ) + out.store(cutlass.BFloat16(acc_o), idx=ch) + if ar_on: + cl = ( + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(AR_PRE): + _ar_push_pre( + ard, + pars[bt], + _bf16_round(acc_o), + lane >> cutlass.Int32(3) & cutlass.Int32(1) == cutlass.Int32(0), + t, + cl, + ) + else: + _ar_push( + arp, + _bf16_round(acc_o), + lane >> cutlass.Int32(3) & cutlass.Int32(1) == cutlass.Int32(0), + t, + cl, + ) + if tidx == 0: + _stamp(tdbg, bx, 7) + if cutlass.const_expr(not EARLY_EPOCH): + epochs.store(ep + cutlass.Int32(1), idx=bx) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if ar_on | cutlass.Boolean(LATE_TRIGGER): + cute.arch.griddepcontrol_launch_dependents() + if warp == MMA_WARP: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Float32), TMEM_COLS + ) + + +@cute.jit +def m3_moe( + a1_tensor: cute.Tensor, + sfa1_tensor: cute.Tensor, + a2_tensor: cute.Tensor, + sfa2_tensor: cute.Tensor, + logits: cute.Tensor, + bias: cute.Tensor, + x32: cute.Tensor, + xsf32: cute.Tensor, + escal: cute.Tensor, + plan: cute.Tensor, + partials: cute.Tensor, + hbuf32: cute.Tensor, + epochs: cute.Tensor, + out: cute.Tensor, + route_dbg: cute.Tensor, + tdbg: cute.Tensor, + xb32: cute.Tensor, + rw: cute.Tensor, + sfsc: cute.Tensor, + lgbuf: cute.Tensor, + guw: cute.Tensor, + gusf: cute.Tensor, + dnw: cute.Tensor, + dnsf: cute.Tensor, + hx: cute.Tensor, + dnx: cute.Tensor, + arp: cute.Tensor, + s1w: cute.Tensor, + s2w: cute.Tensor, + img32: cute.Tensor, + b2cnt: cute.Tensor, + stream: cuda_driver.CUstream, +): + tma_a1_desc = cuda.create_tensor_map_tiled( + global_address=a1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H // 2, TWO_I, E_LOCAL], + global_strides=[H // 2 // 16, TWO_I * (H // 2) // 16], + box_dims=(KT_BYTES, R1, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_a2_desc = cuda.create_tensor_map_tiled( + global_address=a2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[I_TP // 2, H, E_LOCAL], + global_strides=[I_TP // 2 // 16, H * (I_TP // 2) // 16], + box_dims=(KT_BYTES, R2, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_s1_desc = cuda.create_tensor_map_tiled( + global_address=sfa1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * TWO_I * (H // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_s2_desc = cuda.create_tensor_map_tiled( + global_address=sfa2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * H * (I_TP // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 4 // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_gu_desc = tma_a1_desc + tma_dn_desc = tma_a2_desc + if cutlass.const_expr(SHARED and (not SH_PRETILE)): + tma_gu_desc = cuda.create_tensor_map_tiled( + global_address=guw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H, 2 * SH_I], + global_strides=[H // 16], + box_dims=(KT_BYTES, 32), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_dn_desc = cuda.create_tensor_map_tiled( + global_address=dnw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[SH_I, H], + global_strides=[SH_I // 16], + box_dims=(KT_BYTES, 128), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + m3_moe_kernel( + tma_a1_desc, + tma_a2_desc, + tma_s1_desc, + tma_s2_desc, + tma_gu_desc, + tma_dn_desc, + logits, + bias, + x32, + xsf32, + escal, + plan, + partials, + hbuf32, + epochs, + out, + route_dbg, + tdbg, + xb32, + rw, + sfsc, + lgbuf, + guw, + gusf, + dnw, + dnsf, + hx, + dnx, + arp, + s1w, + s2w, + img32, + b2cnt, + ).launch( + grid=(NCTA + NSH, 1, 1), + block=(THREADS, 1, 1), + cluster=(1, 1, 1), + stream=stream, + use_pdl=True, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/op.py new file mode 100644 index 000000000000..3f69c9392eb1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_moe/op.py @@ -0,0 +1,670 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP8 implementation.""" + +from __future__ import annotations + +import importlib.util +import os +import sys +from typing import Dict, Optional, Tuple + +import torch + +from ...m31_switches import m31_disabled + +HIDDEN = 6144 +INTER = 384 +NUM_EXPERTS = 128 +SHARED_INTER = 384 +SHARED_CTAS = 24 +TOP_K = 4 +U1 = 2 * INTER // 32 +MIN_CTAS = 64 +_UNIT_BYTES = 32 * (HIDDEN // 2 + HIDDEN // 16) +_ROW2_BYTES = INTER // 2 + INTER // 16 +_S1 = 6 +_ITEM_BYTES = _UNIT_BYTES // _S1 +_modules: Dict[tuple, object] = {} +_compiled: Dict[tuple, object] = {} +SFA_CP_BY_M: Tuple[int, ...] = (1, 1, 1, 1, 1, 1, 1, 1) +B2IMG_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 1, 1) +AR_PRE_BY_M: Tuple[int, ...] = (1, 0, 1, 1, 1, 1, 1, 1) +STAGES_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 0, 6) +HB_REPL_BY_M: Tuple[int, ...] = (32, 32, 8, 8, 1, 1, 1, 1) +M8_CFG_BY_M: Dict[int, Dict[str, int]] = {8: {"b2img": 0}} +B1WIDE_TOKENS = frozenset({8}) +LG_REPL = 32 +LG_REPL_MAX_TOKENS = 4 +LG_REPL_BY_M: Dict[int, int] = {8: 16} +FINE_PLAN = True +PLAN_FC2_W = 2.0 +M3_MOE_LAG = 2 +_ENGINE_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "m3_moe_kernel.py") + + +def _kernel_module(config: dict, path: str): + key = (os.path.basename(path),) + tuple(sorted(config.items())) + mod = _modules.get(key) + if mod is None: + name = ( + f"{__name__}_" + + os.path.splitext(os.path.basename(path))[0] + + "_" + + "_".join((f"{k}{v}" for k, v in key[1:])) + ) + spec = importlib.util.spec_from_file_location(name, path) + mod = importlib.util.module_from_spec(spec) + mod.M3_CONFIG = dict(config) + sys.modules[name] = mod + spec.loader.exec_module(mod) + _modules[key] = mod + return mod + + +def _view(t: torch.Tensor, align: int): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align) + + +def is_supported(device: Optional[torch.device] = None) -> tuple: + if not torch.cuda.is_available(): + return (False, "no CUDA") + major, _ = torch.cuda.get_device_capability(device) + if major != 10: + return (False, "needs the sm_100 family (tcgen05 block-scaled NVFP4 MMA)") + try: + import cutlass # noqa: F401 - dependency availability check + except ImportError: + return (False, "CuTe DSL (nvidia-cutlass-dsl) is not installed") + return (True, "") + + +def ar_push_desc( + mc_ptr: int, flags_ptr: int, rank: int, world: int, max_tokens: int, device +) -> torch.Tensor: + """The ``arp`` descriptor that makes a call also push its bf16 output into a multicast all-reduce exchange (set A + of m3_mc_allreduce's workspace at ``mc_ptr``, slot half = the all-reduce CTA's call parity in ``flags_ptr``). + With it the kernel releases its PDL dependents only after every push; the default (enable 0) keeps the old + behaviour.""" + return torch.tensor( + [1, mc_ptr, flags_ptr, rank, world, max_tokens, 0, 0], dtype=torch.int64, device=device + ) + + +def _shared_views(shared_weights, who): + """(gate_up, its scales, down, its scales) of the shared expert as flat uint8 views, checked.""" + if shared_weights is None: + raise ValueError(f"{who} (shared): shared_weights are required") + gu_w, gu_sf, dn_w, dn_sf = shared_weights + want = {"gate_up": (gu_w, (2 * SHARED_INTER, HIDDEN)), "down": (dn_w, (HIDDEN, SHARED_INTER))} + for name, (t, shape) in want.items(): + if t.dtype != torch.float8_e4m3fn or tuple(t.shape) != shape or (not t.is_contiguous()): + raise ValueError(f"{who} (shared): {name} weight must be contiguous e4m3 {shape}") + if ( + gu_sf.numel() != 2 * SHARED_INTER * HIDDEN // 32 + or dn_sf.numel() != HIDDEN * SHARED_INTER // 32 + ): + raise ValueError( + f"{who} (shared): the UE8M0 scales must be the swizzled [N, K / 32] buffers" + ) + return ( + gu_w.view(torch.uint8).view(-1), + gu_sf.view(torch.uint8).reshape(-1), + dn_w.view(torch.uint8).view(-1), + dn_sf.view(torch.uint8).reshape(-1), + ) + + +def _pretile_shared(gu8, gusf8, dn8, dnsf8, n_cta: int = 24) -> torch.Tensor: + """The shared expert's weights as m3_moe's shared CTAs stream them (SH_PRETILE): per CTA s, per stage g, the 16 KB + shared-memory image (128-byte swizzle: 16-byte chunk j of row r at j ^ (r % 8)) of the stage's 128-row MMA tile and + its 512-byte instruction scale atom (row R's 4 codes at word 4 (R % 32) + R / 32). CTA s owns P = SHARED_INTER / 24 + = 16 positions in SPLIT = 128 / (2 P) = 4 K slices of GT = 48 / SPLIT = 12 k-tiles. Stages 0..GT-1: MMA row + R = P O + i (O = 2 q + u: slice q, u 0 gate / 1 up) = weight row u SHARED_INTER + P s + i at k-tile GT q + g; then + down tile j (rows 256 s + 128 j ..) at k-tile kt, j-major (SHARED_INTER / 128 k-tiles each). Built once per + layer.""" + dev = gu8.device + H_, I_ = (HIDDEN, SHARED_INTER) + P = I_ // n_cta + split = 128 // (2 * P) + gt = H_ // 128 // split + dkt = I_ // 128 + gu = gu8.view(2 * I_, H_) + dn = dn8.view(H_, I_) + gkb, dkb = (H_ // 32, I_ // 32) + + def sf_bytes(sf, n, b, kbn): + return sf[ + ((n >> 7) * (kbn // 4) + (b >> 2)) * 512 + (n & 31) * 16 + (n >> 5 & 3) * 4 + (b & 3) + ] + + r = torch.arange(128, device=dev) + q = r >> 5 + ri = r & 31 + octant = r // P + chunk = torch.arange(8, device=dev) + swz = chunk[None, :] ^ r[:, None] & 7 + out = torch.empty(n_cta, gt + 2 * dkt, 16384 + 512, dtype=torch.uint8, device=dev) + for s in range(n_cta): + nrow = (octant & 1) * I_ + P * s + r % P + sl = octant >> 1 + for g in range(gt): + kt = sl * gt + g + tile = gu[nrow[:, None], kt[:, None] * 128 + torch.arange(128, device=dev)[None, :]] + img = tile.view(128, 8, 16)[r[:, None], swz] + out[s, g, :16384] = img.reshape(-1) + kb = torch.arange(4, device=dev) + codes = sf_bytes(gusf8, nrow[:, None], kt[:, None] * 4 + kb[None, :], gkb) + atom = torch.empty(512, dtype=torch.uint8, device=dev) + atom[(ri[:, None] * 16 + q[:, None] * 4 + kb[None, :]).reshape(-1)] = codes.reshape(-1) + out[s, g, 16384:] = atom + for gd in range(2 * dkt): + jt, kt = (gd // dkt, gd % dkt) + row0 = 256 * s + 128 * jt + tile = dn[row0 : row0 + 128, kt * 128 : (kt + 1) * 128] + img = tile.reshape(128, 8, 16)[r[:, None], swz] + out[s, gt + gd, :16384] = img.reshape(-1) + out[s, gt + gd, 16384:] = dnsf8[ + ((row0 >> 7) * (dkb // 4) + kt) * 512 : ((row0 >> 7) * (dkb // 4) + kt) * 512 + 512 + ] + return out.view(-1) + + +_PRETILED: Dict[tuple, tuple] = {} + + +def _pretiled(gu8, gusf8, dn8, dnsf8) -> torch.Tensor: + key = (gu8.device, gu8.data_ptr(), gusf8.data_ptr(), dn8.data_ptr(), dnsf8.data_ptr()) + hit = _PRETILED.get(key) + if hit is None: + hit = _PRETILED[key] = ((gu8, gusf8, dn8, dnsf8), _pretile_shared(gu8, gusf8, dn8, dnsf8)) + return hit[1] + + +def _fine_ranges(weights, total: int): + """Contiguous FC1 item ranges [s_c, e_c) over the CTAs, sized as close to ``weights`` (ideal real-valued item + counts summing to ``total``) as the split-unit protocol allows: every unit (S1 items) has at most one CTA boundary + strictly inside it (its head and tail parts, summed by the tail's holder), so a range that starts inside a unit + runs at least to that unit's end. Returns the item counts.""" + n = len(weights) + items = [0] * n + s = 0 + acc = 0.0 + for c in range(n): + acc += weights[c] + e = total if c == n - 1 else min(total, max(s, int(acc + 0.5))) + if s % _S1 and e < total: + e = max(e, min(total, (s // _S1 + 1) * _S1)) + items[c] = e - s + s = e + assert s == total + return items + + +def make_phase_plan( + num_ctas: int, + m_max: int, + lag: int = 2, + tail_w: float = 1.0, + early0: bool = False, + fine: bool = FINE_PLAN, + fc2_w: float = 1.0, +): + """The work split of ``m3_moe`` for each distinct-expert count G (4 .. 4 m_max): FC2 32-row output blocks in + contiguous per-CTA ranges (the same for every group), and per group p (slots 4 p ..) a range of its unit-major FC1 + stages per CTA, balancing the bytes each CTA streams in phase p (FC1 of group p plus FC2 of group p - lag, and the + trailing FC2s with the last group). Ranges are empty or >= 6 stages (a unit is split over at most two CTAs); a + group too small for every CTA gets whole units on the CTAs with the least FC2 work, or (``fine``) byte-balanced + ranges of any length with at most one CTA boundary inside a unit (:func:`_fine_ranges`). Returns int32 + [4 m_max + 1, num_ctas, m_max + 1] (per group i0 | i1 << 16, then b0 | nb << 16) and the largest block count.""" + gcap = TOP_K * m_max + nb2 = HIDDEN // 32 + plan = torch.zeros(gcap + 1, num_ctas, m_max + 1, dtype=torch.int32) + nbs = [nb2 // num_ctas + (1 if c < nb2 % num_ctas else 0) for c in range(num_ctas)] + b0s = [sum(nbs[:c]) for c in range(num_ctas)] + blk_bytes = int(round(32 * _ROW2_BYTES * fc2_w)) + for g in range(TOP_K, gcap + 1): + ng = (g + 3) // 4 + for c in range(num_ctas): + plan[g, c, m_max] = b0s[c] | nbs[c] << 16 + for p in range(ng): + n_p = min(4, g - 4 * p) + total = n_p * U1 * _S1 + fc2 = [0] * num_ctas + for c in range(num_ctas): + if p >= lag: + fc2[c] += nbs[c] * min(4, g - 4 * (p - lag)) * blk_bytes + if p == ng - 1 and (not (early0 and p == 0 and (m_max >= 2))): + for q in range(max(0, ng - lag), ng): + fc2[c] += int(tail_w * nbs[c] * min(4, g - 4 * q) * blk_bytes) + items = [0] * num_ctas + if total >= _S1 * num_ctas: + + def counts_for(limit): + return [max(_S1, (limit - f) // _ITEM_BYTES) for f in fc2] + + lo, hi = (0, max(fc2) + total * _ITEM_BYTES) + while lo < hi: + mid = (lo + hi) // 2 + if sum(counts_for(mid)) >= total: + hi = mid + else: + lo = mid + 1 + items = counts_for(lo) + extra = sum(items) - total + while extra > 0: + c = min( + (c for c in range(num_ctas) if items[c] > _S1), + key=lambda c: (-(fc2[c] + items[c] * _ITEM_BYTES), c), + ) + items[c] -= 1 + extra -= 1 + elif fine: + lo_f, hi_f = (0.0, float(max(fc2) + total * _ITEM_BYTES)) + for _ in range(100): + mid = 0.5 * (lo_f + hi_f) + if sum((max(0.0, (mid - f) / _ITEM_BYTES) for f in fc2)) >= total: + hi_f = mid + else: + lo_f = mid + wts = [max(0.0, (hi_f - f) / _ITEM_BYTES) for f in fc2] + sw = sum(wts) + items = _fine_ranges([w * total / sw for w in wts], total) + else: + chosen = sorted(sorted(range(num_ctas), key=lambda c: (fc2[c], c))[: total // _S1]) + for c in chosen: + items[c] = _S1 + i0 = 0 + for c in range(num_ctas): + if items[c]: + plan[g, c, p] = i0 | i0 + items[c] << 16 + i0 += items[c] + assert i0 == total + return (plan, max(nbs)) + + +class M3MoeState: + """Workspace + compiled kernel of ``m3_moe`` (1..8 tokens per call) on one device, shared by the layers that run + on it in one stream order. The call takes the bf16 hidden states and computes the router logits (bit-exact with + m3_router_gemv_fp4) and the NVFP4 activation itself. ``shared``: the shared expert runs on SHARED_CTAS dedicated + CTAs beside the ``num_ctas`` routed ones and the call returns bf16(shared + routed). Build before CUDA-graph + capture; the first call compiles.""" + + fused_router = True + + def __init__( + self, + device: torch.device, + num_tokens: int, + num_ctas: Optional[int] = None, + shared: bool = False, + inter: int = INTER, + ): + """``inter``: the routed intermediate per rank; this engine is built for INTER (384) only.""" + if inter != INTER: + raise ValueError( + f"m3_moe (TP8): routed intermediate {inter} per rank is not supported (only {INTER})" + ) + self.inter = inter + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "M3MoeState allocates and compiles: build it before CUDA-graph capture" + ) + sms_all = torch.cuda.get_device_properties(device).multi_processor_count + if shared and (num_ctas or sms_all) + SHARED_CTAS > sms_all: + raise ValueError( + ( + "m3_moe (shared): num_ctas + " + f"{SHARED_CTAS}" + " shared-expert CTAs must fit the " + f"{sms_all}" + " SMs (all resident at once)" + ) + ) + if not 1 <= num_tokens <= 8: + raise ValueError(f"m3_moe takes 1..8 tokens per call, not {num_tokens}") + self.device = torch.device(device) + ok, why = is_supported(self.device) + if not ok: + raise RuntimeError(f"m3_moe: {why}") + sms = torch.cuda.get_device_properties(self.device).multi_processor_count + self.num_ctas = num_ctas or sms + if not MIN_CTAS <= self.num_ctas <= sms: + raise ValueError( + f"m3_moe runs {MIN_CTAS} .. {sms} CTAs (all resident at once), not {self.num_ctas}" + ) + self.m = num_tokens + plan, nb_cap = make_phase_plan( + self.num_ctas, + num_tokens, + M3_MOE_LAG, + 0.0 if num_tokens == 1 else 1.0, + True, + fc2_w=1.0 if m31_disabled("PLANW") else PLAN_FC2_W, + ) + if not m31_disabled("PLANW") and PLAN_FC2_W != 1.0: + from tensorrt_llm.logger import logger as _planw_log + + _planw_log.info( + ( + "MiniMax-M3.1 MoE engine PLAN-W: FC2 bytes weighted " + f"{PLAN_FC2_W}" + " in the phase plan (tokens=" + f"{num_tokens}" + ")" + ) + ) + self.config = { + "m_max": num_tokens, + "num_ctas": self.num_ctas, + "nb2_cap": nb_cap, + "fused_router": 1, + "timing": 0, + "i_tp": INTER, + } + if HB_REPL_BY_M[num_tokens - 1] > 1: + self.config["hb_repl"] = HB_REPL_BY_M[num_tokens - 1] + if num_tokens <= LG_REPL_MAX_TOKENS: + self.config["lg_repl"] = LG_REPL + elif num_tokens in LG_REPL_BY_M and (not m31_disabled("MOE_LGREPL16")): + self.config["lg_repl"] = LG_REPL_BY_M[num_tokens] + from tensorrt_llm.logger import logger as _lg_log + + _lg_log.info( + ( + "MiniMax-M3.1 MoE engine MOE_LGREPL16: " + f"{LG_REPL_BY_M[num_tokens]}" + " router-logit exchange replicas (tokens=" + f"{num_tokens}" + ")" + ) + ) + if shared: + self.config["sh_i"] = SHARED_INTER + self.config["shared"] = 1 + self.shared = shared + self.config["sfa_cp"] = SFA_CP_BY_M[num_tokens - 1] + self.config["b2img"] = B2IMG_BY_M[num_tokens - 1] + self.config["ar_pre"] = AR_PRE_BY_M[num_tokens - 1] + if STAGES_BY_M[num_tokens - 1] > 0: + self.config["stages"] = STAGES_BY_M[num_tokens - 1] + if num_tokens in M8_CFG_BY_M and (not m31_disabled("MOE_M8CFG")): + self.config.update(M8_CFG_BY_M[num_tokens]) + if ( + num_tokens in B1WIDE_TOKENS + and self.config.get("fused_router") + and (not m31_disabled("MOE_B1WIDE")) + ): + self.config["b1wide"] = 1 + from tensorrt_llm.logger import logger as _b1w_log + + _b1w_log.info( + f"MiniMax-M3.1 MoE engine B1WIDE: B1 quantize on every warp (tokens={num_tokens})" + ) + if not m31_disabled("MOE_TAIL_TRIGGER"): + self.config["tail_trigger"] = 1 + for key in ("tail_trigger",): + if self.config.get(key): + from tensorrt_llm.logger import logger as _rel_log + + _rel_log.info( + f"MiniMax-M3.1 MoE engine dependent release: {key} (tokens={num_tokens}, epoch stored early)" + ) + try: + self.mod = mod = _kernel_module(self.config, _ENGINE_PATH) + except AssertionError as err: + raise ValueError( + ( + "m3_moe: num_tokens=" + f"{num_tokens}" + ", num_ctas=" + f"{self.num_ctas}" + " does not fit the shared memory / TMEM budget (fewer CTAs means " + "more FC2 blocks per CTA): " + f"{err}" + ) + ) from err + kw = dict(device=self.device) + self.plan = plan.to(self.device) + self.partials = torch.full( + (mod.G_CAP * mod.U1 * num_tokens * 32,), -1, dtype=torch.int32, **kw + ) + hb_words = mod.G_CAP * num_tokens * mod.U1 * 4 + if mod.HB_REPL > 1: + hb_words = mod.HB_REPL * mod.HB_STRIDE + self.hbuf32 = torch.zeros(hb_words, dtype=torch.int32, **kw) + self.epochs = torch.zeros(self.num_ctas + mod.NSH, dtype=torch.int32, **kw) + self.route_dbg = torch.zeros(num_tokens, 8, dtype=torch.int32, **kw) + self.tdbg = torch.zeros(self.num_ctas + mod.NSH, 64, dtype=torch.int64, **kw) + self.lgbuf = torch.zeros(mod.LG_REPL * mod.LG_STRIDE, dtype=torch.int64, **kw) + self._dummy = torch.zeros(4, dtype=torch.int32, **kw) + self._ar_off = torch.zeros(8, dtype=torch.int64, **kw) + self.img32 = torch.zeros(max(1, mod.NG_CAP * mod.IMG_BYTES // 4), dtype=torch.int32, **kw) + self.cnt = torch.zeros(mod.NG_CAP * 8 + 8, dtype=torch.int32, **kw) + self.hx = torch.zeros(num_tokens * SHARED_INTER, dtype=torch.int32, **kw) + self.dnx = torch.zeros(num_tokens * HIDDEN, dtype=torch.int32, **kw) + self._fn = None + + def layer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=None, + fc31_input_scale=None, + shared_weights=None, + ) -> "M3MoeLayer": + """One MoE layer's operands (the TRTLLM-Gen NVFP4 expert buffers, read in place, never copied). + ``shared_weights`` (a ``shared`` state): the shared expert's (gate_up e4m3 [768, 6144], its swizzled UE8M0 + scales, down e4m3 [6144, 384], its swizzled UE8M0 scales).""" + layer = M3MoeLayer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight, + fc31_input_scale, + ) + if self.shared: + layer.shared_weights = _shared_views(shared_weights, "m3_moe") + layer.shared_mxfp8 = tuple(shared_weights) + if self.mod.SH_PRETILE: + gu8, _, dn8, _ = layer.shared_weights + layer.shared_weights = ( + gu8, + _pretiled(*layer.shared_weights), + dn8, + self._dummy.view(torch.uint8), + ) + return layer + + def run(self, layer, hidden, out=None, arp=None) -> torch.Tensor: + m = self.m + if ( + hidden is None + or hidden.dtype != torch.bfloat16 + or tuple(hidden.shape) != (m, HIDDEN) + or (not hidden.is_contiguous()) + ): + raise ValueError( + f"m3_moe (fused router): expects contiguous bf16 hidden states [{m}, {HIDDEN}]" + ) + fz = ( + hidden.view(-1).view(torch.int32), + layer.router_weight.view(-1), + layer.fc31_input_scale.view(-1), + self.lgbuf, + ) + lgv, xq, xs = (self._dummy.view(torch.float32), self._dummy, self._dummy) + if out is None: + out = torch.empty(m, HIDDEN, dtype=torch.bfloat16, device=self.device) + sh = layer.shared_weights if self.shared else (self._dummy.view(torch.uint8),) * 4 + args = ( + layer.w3_w1_weight, + layer.w3_w1_weight_scale.view(-1), + layer.w2_weight, + layer.w2_weight_scale.view(-1), + lgv, + layer.bias, + xq, + xs, + layer.escal, + self.plan.view(-1), + self.partials, + self.hbuf32, + self.epochs, + out.view(-1), + self.route_dbg.view(-1), + self.tdbg.view(-1), + *fz, + *sh, + self.hx, + self.dnx, + self._ar_off if arp is None else arp, + layer.w3_w1_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + layer.w2_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + self.img32, + self.cnt, + ) + if self._fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_moe compiles on its first call: call it once before capture") + import cutlass.cute as cute + + aligns = [ + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + ] + sig = [_view(t, a) for t, a in zip(args, aligns)] + key = (_ENGINE_PATH,) + tuple(sorted(self.config.items())) + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute.compile( + self.mod.m3_moe, + *sig, + cute.runtime.make_fake_stream(), + options="--enable-tvm-ffi", + ) + self._fn = fn + self._fn(*args, torch.cuda.current_stream(self.device).cuda_stream) + return out + + +class M3MoeLayer: + """One MoE layer on a:class:`M3MoeState`: ``layer(hidden)``. The expert buffers are read in + place, never copied; the per-expert scalars are packed once (layer constants: rebuild the layer after reloading + weights).""" + + shared_weights = None + shared_mxfp8 = None + + def __init__( + self, + state, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + bias, + router_weight=None, + fc31_input_scale=None, + ): + e = NUM_EXPERTS + expected = { + "w3_w1_weight": (w3_w1_weight, (e, 2 * INTER, HIDDEN // 2)), + "w3_w1_weight_scale": (w3_w1_weight_scale, (e, 2 * INTER, HIDDEN // 16)), + "w2_weight": (w2_weight, (e, HIDDEN, INTER // 2)), + "w2_weight_scale": (w2_weight_scale, (e, HIDDEN, INTER // 16)), + } + for name, (t, shape) in expected.items(): + if t.element_size() != 1 or tuple(t.shape) != shape or (not t.is_contiguous()): + raise ValueError( + f"m3_moe: {name} is {t.dtype} {tuple(t.shape)}, expected contiguous 1-byte {shape}" + ) + self.state = state + self.w3_w1_weight = w3_w1_weight.view(torch.uint8) + self.w3_w1_weight_scale = w3_w1_weight_scale.view(torch.uint8) + self.w2_weight = w2_weight.view(torch.uint8) + self.w2_weight_scale = w2_weight_scale.view(torch.uint8) + self.escal = torch.stack( + [fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha] + ).float() + self.escal = self.escal.reshape(6, e).contiguous() + self.bias = bias.float().contiguous() + if ( + router_weight is None + or fc31_input_scale is None + or router_weight.dtype != torch.float32 + or (tuple(router_weight.shape) != (e, HIDDEN)) + or (not router_weight.is_contiguous()) + or (fc31_input_scale.dtype != torch.float32) + or (fc31_input_scale.numel() != 1) + ): + raise ValueError( + "m3_moe (fused router): needs router_weight f32 [128, 6144] and fc31_input_scale f32 [1]" + ) + self.router_weight = router_weight + self.fc31_input_scale = fc31_input_scale + + def __call__( + self, + hidden: torch.Tensor, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """``hidden``: contiguous bf16 [M, 6144]. ``arp`` (int64 [8], see:func:`ar_push_desc`): also push the output + into an all-reduce exchange.""" + return self.state.run(self, hidden, out, arp=arp) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_prequant.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_prequant.py new file mode 100644 index 000000000000..48a7827f59b3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_prequant.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Hand-off of an MXFP8-quantized activation from the kernel that produced it to the next MXFP8 GEMM. + +A producer that already holds a BF16 row in registers (the multicast all-reduce + RMSNorm, the split-K attention +merge) also writes its MXFP8 form (E4M3 rows + UE8M0 scales in the 128x4-swizzled layout of +``trtllm::mxfp8_quantize(x, True)``) and publishes it here keyed by the BF16 tensor. The MXFP8 linear whose input +is that tensor takes it and skips its own quantize kernel. Every take clears the slot, so a hand-off is only ever +consumed by the first MXFP8 linear that runs after the producer. + +Module state shared across files (one instance each per process): + +* ``SLOT`` (PrequantSlot): published by ``m3_mc_allreduce`` (``quant=True``), the sparse-decode merge and the fused + decode attention (MXFP8 output); cleared by the sparse decode before it runs; taken by the MXFP8 linear dispatch + and by the one-launch QKV + attention + o_proj path (which also peeks ``pending``). +* ``AR_PUSH`` (ArPushSlot): armed by the fast forward around the attention call; the o_proj GEMV takes the target + and marks ``pushed``; the fast forward checks ``take_pushed`` before the attention boundary all-reduce. +* ``MOE_PUSH`` (MoePushSlot): armed by the fast forward before the MoE; the routed-expert engine takes ``engine`` + and the shared expert's down projection takes ``shared``, each marks its pushed output; the fast forward + checks ``take_pushed`` before the MoE boundary all-reduce. +""" + + +class PrequantSlot: + def __init__(self): + self.pending = None # (data_ptr, rows, width, e4m3 [rows, width], scales) + + def publish(self, x, xq, xsf): + self.pending = (x.data_ptr(), x.shape[0], x.shape[-1], xq, xsf) + + def clear(self): + self.pending = None + + def take(self, x): + """(e4m3 [x.shape[0], width], scales) if x is the published tensor, else None.""" + pending, self.pending = self.pending, None + if ( + pending is None + or x.dim() != 2 + or x.data_ptr() != pending[0] + or x.shape[0] != pending[1] + or x.shape[1] != pending[2] + ): + return None + return pending[3], pending[4] + + +SLOT = PrequantSlot() + + +class ArPushSlot: + """The attention boundary's all-reduce exchange for the o_proj GEMV to push into. + + The fast forward arms ``target`` (m3_mx_gemv ``ar`` tuple) around the attention call; the o_proj marks the output + it pushed in ``pushed``; the boundary then runs the all-reduce with ``prepushed`` = the ``ar`` tuple's mode.""" + + def __init__(self): + self.target = None + self.pushed = None # data_ptr of the pushed output + + def arm(self, target): + self.target = target + self.pushed = None + + def take_target(self): + target, self.target = self.target, None + return target + + def take_pushed(self, x): + pushed, self.pushed = self.pushed, None + return pushed is not None and x.data_ptr() == pushed + + +AR_PUSH = ArPushSlot() + + +class MoePushSlot: + """The MoE boundary's all-reduce exchange for its two producers to push into: the routed-expert engine (set A, + ``engine`` = its ``arp`` descriptor) and the shared expert's down projection (set B, ``shared`` = m3_mx_gemv + ``ar`` tuple). Each marks the output it pushed; the boundary then runs the all-reduce with ``prepushed`` = 3.""" + + def __init__(self): + self.engine = None + self.shared = None + self.engine_pushed = None # data_ptr of the pushed routed output + self.shared_pushed = None # data_ptr of the pushed shared output + + def arm(self, engine, shared): + self.engine, self.shared = engine, shared + self.engine_pushed = self.shared_pushed = None + + def disarm(self): + self.engine = self.shared = None + + def take_engine(self): + engine, self.engine = self.engine, None + return engine + + def take_shared(self): + shared, self.shared = self.shared, None + return shared + + def take_pushed(self, shared, routed): + """(routed pushed, shared pushed) for these outputs; clears the marks.""" + e = routed is not None and self.engine_pushed == routed.data_ptr() + s = shared is not None and self.shared_pushed == shared.data_ptr() + self.engine_pushed = self.shared_pushed = None + return e, s + + +MOE_PUSH = MoePushSlot() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_router_gemv.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_router_gemv.py new file mode 100644 index 000000000000..30855eb7f05d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_router_gemv.py @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Low-M fp32 router projection: ``logits[M, E] = float(x[M, K]) @ W[E, K]^T`` in one kernel. + +Replaces the bf16->fp32 cast, the TF32 GEMM and its split-K reduce on decode +steps. One CTA per ``ROWS`` experts: the fp32 weight rows are bulk-copied into +shared memory before ``griddepcontrol.wait`` (overlapping the producer kernel +under PDL), the bf16 activations after it. Products and sums are exact fp32 FMA. + +The kernels are ``torch.ops.trtllm.minimax_m3_router_gemv`` / ``_router_gemv_fp4`` / ``_router_tc``, reached +through the ``ops/moe`` entries. +""" + +from typing import Optional + +import torch + +from ..ops.moe.m3_router_gemv import m3_router_gemv as _router_gemv_op +from ..ops.moe.m3_router_gemv_fp4 import m3_router_gemv_fp4 as _router_gemv_fp4_op +from ..ops.moe.m3_router_tc import m3_router_tc as _router_tc_op + +# Opt-in dynamic shared memory left after the kernel's static reduction buffer (SM100: 227 KiB). +_MAX_DYNAMIC_SMEM = 227 * 1024 - 1024 # capacity: fixed: the device's shared-memory limit + + +def m3_router_gemv( + x: torch.Tensor, weight: torch.Tensor, rows: int = 1, use_pdl: bool = False +) -> Optional[torch.Tensor]: + """fp32 ``x @ weight.T`` for bf16 ``x`` with at most 16 rows; None when unsupported (only ``rows=1`` is built).""" + if ( + rows != 1 + or not x.is_cuda + or x.dtype != torch.bfloat16 + or x.dim() != 2 + or x.stride(1) != 1 + or not 1 <= x.shape[0] <= 16 + or weight.dtype != torch.float32 + or not weight.is_contiguous() + or weight.shape[1] != x.shape[1] + or x.shape[1] % 1024 != 0 + or weight.shape[0] % rows != 0 + or x.data_ptr() % 16 != 0 + or (x.stride(0) * 2) % 16 != 0 + or torch.cuda.get_device_capability(x.device)[0] < 10 + or rows * x.shape[1] * 4 + x.shape[0] * x.shape[1] * 2 > _MAX_DYNAMIC_SMEM + ): + return None + return _router_gemv_op(x, weight, rows, use_pdl) + + +def m3_router_gemv_fp4( + x: torch.Tensor, weight: torch.Tensor, sf_scale: torch.Tensor, use_pdl: bool = False +) -> Optional[tuple]: + """Router logits plus the NVFP4 activation the routed experts consume (bit-identical to + ``fp4_quantize(x, sf_scale, 16, False, False)``): (logits [M, E] fp32, x_fp4 [M, K / 2] uint8, + x_sf [M * K / 16] uint8 linear). None when unsupported.""" + if ( + not x.is_cuda + or x.dtype != torch.bfloat16 + or x.dim() != 2 + or x.stride(1) != 1 + or not 1 <= x.shape[0] <= 16 + or weight.dtype != torch.float32 + or not weight.is_contiguous() + or weight.shape[1] != x.shape[1] + or x.shape[1] % 1024 != 0 + or x.data_ptr() % 16 != 0 + or (x.stride(0) * 2) % 16 != 0 + or sf_scale is None + or sf_scale.dtype != torch.float32 + or sf_scale.numel() != 1 + or torch.cuda.get_device_capability(x.device)[0] < 10 + or x.shape[1] * 4 + x.shape[0] * x.shape[1] * 2 > _MAX_DYNAMIC_SMEM + ): + return None + return tuple(_router_gemv_fp4_op(x, weight, sf_scale.reshape(1), use_pdl)) + + +def m3_router_tc( + x: torch.Tensor, + weight: torch.Tensor, + sf_scale: Optional[torch.Tensor] = None, + use_pdl: bool = False, +) -> Optional[tuple]: + """Router logits (TF32 tensor cores, like the native TF32 GEMM) for 1..48 tokens with 128 experts, plus the + NVFP4 activation when ``sf_scale`` is given. Returns (logits,) or (logits, x_fp4, x_sf); None when unsupported.""" + if ( + not x.is_cuda + or x.dtype != torch.bfloat16 + or x.dim() != 2 + or x.stride(1) != 1 + or not 1 <= x.shape[0] <= 48 + or weight.dtype != torch.float32 + or weight.shape[0] != 128 + or not weight.is_contiguous() + or weight.shape[1] != x.shape[1] + or x.shape[1] % 1024 != 0 + or (x.stride(0) * 2) % 16 != 0 + or x.data_ptr() % 16 != 0 + or torch.cuda.get_device_capability(x.device)[0] < 10 + or 16 * (x.shape[1] // 8 + 4) * 4 + x.shape[0] * (x.shape[1] // 8 + 8) * 2 + > _MAX_DYNAMIC_SMEM + or (sf_scale is not None and (sf_scale.dtype != torch.float32 or sf_scale.numel() != 1)) + ): + return None + res = _router_tc_op(x, weight, None if sf_scale is None else sf_scale.reshape(1), use_pdl) + return tuple(res) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_sampled_accept.py new file mode 100644 index 000000000000..59c58336c5d7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_sampled_accept.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused sampled verification for the MiniMax-M3 Eagle3 target head (TP-sharded vocabulary, decode rows <= 48). + +Replaces, per decode step, the full-vocabulary all-gather of the target logits, the fused sampling kernel and the +strict-acceptance ops for rows sampled with temperature and top-p (the AgentX contract). Rows with top-k or min-p +active must stay on the generic path; the caller decides. + +Sampling is top-p rejection sampling, distributed over the TP ranks. With w = exp(logit / T - max) over the FULL +vocabulary and Z = sum(w), a token is in the nucleus iff the mass strictly above its weight is at most top_p * Z +(the fused sampling kernel's kept set, ties included). A candidate is drawn proportionally to w and accepted iff it +is in the nucleus; a rejected candidate lies below the nucleus, so the next candidate is drawn over the tokens heavier +than it. The accepted token is distributed as w restricted to the nucleus. The first candidate lands in the nucleus +with probability >= top_p, so most rows decide on their second exchange. + +Per verify row, 8 CTAs (rows <= 32) or 4 CTAs (more rows) on every rank each own a slice of the rank's logits shard, +staged in shared memory once. A candidate is drawn in two stages without per-token noise: each CTA draws a local +candidate by inverse CDF over its slice, and the CTA whose log-mass plus Gumbel noise is largest supplies the +candidate (CTA j wins with probability T_j / sum T, its token with w / T_j). Uniforms come from a counter-based Philox +keyed by the request's seed and offset, the row, the attempt and the CTA. Each round every CTA multicasts one 4-word +record (key, candidate logit, and max and mass in round 0 or the mass above the current candidate later) and reads +the records of all CTAs of the row on all ranks. Every CTA takes the same decision from the same records in the same +order, so all ranks agree without any intra-GPU handshake. Words carry the call's tag: a slot needs no clearing and +two call halves keep a rank one call ahead from overwriting a slot still being read. +""" + +from typing import Optional + +import torch + +from ..ops.spec.m3_sampled_accept import m3_sampled_accept as _sampled_accept_op +from ..ops.spec.m3_sampled_accept_ws_words import m3_sampled_accept_ws_words as _ws_words_op + +# The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.cu) and reached through +# the ops/spec entry m3_sampled_accept. +MAX_ROWS = 48 # capacity: fixed: the kernel's built rows (decode steps) + + +class SampledAcceptWorkspace: + """Exchange slots behind one multicast mapping (two call halves), the (finished rows, call) counters, the per-row + token scratch and an extra-attempt counter. Created collectively, outside CUDA-graph capture.""" + + def __init__(self, uc, mc_ptr, rank, world, handle=None, comm=None): + self.uc, self.mc_ptr, self.rank, self.world = uc, mc_ptr, rank, world + self.handle, self.comm = handle, comm + dev = uc.device + self.counters = torch.zeros(2, dtype=torch.int32, device=dev) + self.tokens = torch.zeros(MAX_ROWS, dtype=torch.int32, device=dev) + self.extra_attempts = torch.zeros(1, dtype=torch.int32, device=dev) + + @classmethod + def create(cls, mapping, device) -> "SampledAcceptWorkspace": + words = int(_ws_words_op(max(mapping.tp_size, 1))) + if mapping.tp_size == 1: + uc = torch.zeros(words, dtype=torch.int64, device=device) + return cls(uc, uc.data_ptr(), 0, 1) + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "SampledAcceptWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 8, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("SampledAcceptWorkspace.create failed on another rank") + return cls(uc, mc.data_ptr(), mapping.tp_rank, mapping.tp_size, handle, comm) + + +_WS: Optional[SampledAcceptWorkspace] = None + + +def workspace(mapping, device) -> Optional[SampledAcceptWorkspace]: + global _WS + if _WS is None: + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = SampledAcceptWorkspace.create(mapping, device) + return _WS + + +def sampled_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + temperatures: torch.Tensor, + top_ks: torch.Tensor, + top_ps: torch.Tensor, + min_ps: Optional[torch.Tensor], + seeds: torch.Tensor, + offsets: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, + ws: Optional[SampledAcceptWorkspace] = None, + use_pdl: bool = True, +): + """Sampled verification (temperature + top-p) from this rank's bf16 logits shard [nreq * (draft_len + 1), V / tp]: + outputs as shard_accept (accepted [nreq, D+1], num_accepted [nreq], gather_ids [nreq]); None + when unavailable. Per-row sampling parameters and Philox (seed, offset) are the spec metadata's, one per row. + top_ks / min_ps are not applied: rows with top-k or min-p active must be routed to the generic path by the + caller.""" + del top_ks, min_ps + nreq, draft_len = draft_tokens.shape + rows = nreq * (draft_len + 1) + if logits.dim() != 2 or logits.shape[0] != rows or rows > MAX_ROWS or logits.shape[1] > 51200: + return None + if logits.dtype != torch.bfloat16 or logits.stride(1) != 1: + return None + ws = ws if ws is not None else workspace(mapping, logits.device) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + _sampled_accept_op( + logits, + int(vocab_offset), + temperatures[:rows].float().contiguous(), + top_ps[:rows].float().contiguous(), + seeds[:rows].long().contiguous(), + offsets[:rows].long().contiguous(), + ws.uc, + ws.mc_ptr, + ws.counters, + ws.tokens, + ws.rank, + ws.world, + draft_tokens.int().contiguous(), + batch_idx.int().contiguous(), + accepted, + num_accepted, + gather_ids, + int(force_total), + next_new, + force_vec, + ws.extra_attempts, + bool(use_pdl), + ) + return accepted, num_accepted, gather_ids diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_select_long_dsl.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_select_long_dsl.py new file mode 100644 index 000000000000..b56c93b93a2b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_select_long_dsl.py @@ -0,0 +1,367 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 block select for MiniMax-M3 decode rows of up to 9216 blocks, one row per 384-thread CTA (CuTe DSL). + +Bit for bit the selection of this target's ``m3_select_long`` (``torch.ops.trtllm.minimax_m3_select_long``, +one 1024-thread CTA per row), from a CTA of 384 threads: + +* Every block's score is forced (the first ``init_blocks`` and the last ``local_blocks`` of the row's valid blocks) and + mapped to an order key held in registers: thread t owns blocks t, t + 384, ... (4, 11 or 24 of them by the row's + valid count). Larger scores get smaller keys; NaN sorts above +inf. The key is the 1024-thread kernel's unsigned + key with its top bit flipped, so signed Int32 order here is that kernel's order. Every thread loads at a clamped + block and masks the result, so a thread's loads issue back to back. +* 384 is a multiple of 16, so the 16 interleaved groups ``block % 16`` are the lanes' ``lane % 16``. The weakest of + the 16 group maxima bounds the 16th-largest score, so the top 16 is among the keys at or above it. +* The candidates are compacted by a CTA prefix count, each as one int64 (key << 32 | block), and ranked exactly by + that value (one signed compare per pair); more than 1024 of them (keys tied at the bound) rank every candidate + against the whole row instead. +* Warp 0 emits the 16 picked block ids sorted ascending, -1 for an empty or -inf pick. + +The kernel waits on its grid dependency first; it releases the dependent sparse decode right after that wait +(``early_trigger``), once every score of its row is loaded (trigger phase 1) or at its exit (phase 0). Kernels are +compiled per (PDL launch, early trigger, trigger phase) with symbolic shapes and strides; ``prebuild`` compiles and +loads them before the executor's warmup, so a decode step never compiles one. +""" + +from typing import Dict, Tuple + +import cutlass +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream, make_fake_tensor +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + griddepcontrol_launch_dependents, + griddepcontrol_wait, +) + +# The 384-thread select reads its row's valid-block count before its grid wait instead of after it, which takes one +# dependent global load off the post-wait path; the selection is the same either way. Exact: the count +# (MsaMetadata.msa_n_valid_blocks) is written only before the step's first layer, behind a plain launch. +SELECT_N_VALID_PREWAIT = True + +SELECT_THREADS = 384 +SELECT_WARPS = SELECT_THREADS // 32 +# 1M tokens + draft tokens at 128 tokens per block. +MAX_BLOCKS = 9216 # capacity: fixed: the kernel's built blocks per row +MAX_CAND = 1024 # capacity: fixed: the kernel's candidate staging +TOPK = 16 # capacity: fixed: MSA's top-k +NO_KEY = 0x7FFFFFFF # capacity: fixed: a sentinel, not a size (an unused key slot (the reference's 0xffffffff)) +# The key of -inf (the reference's 0xff800000): never emitted. +NEG_INF_KEY = 0x7F800000 # capacity: fixed: a key encoding, not a size +INIT_SCORE = 1.0e30 +LOCAL_SCORE = 1.0e29 +# (keys per thread, rows up to this many valid blocks) +ITEM_CLASSES = ((4, 4 * SELECT_THREADS), (11, 11 * SELECT_THREADS), (24, MAX_BLOCKS)) + +# Shared memory of the phase, int32 words: candidates [MAX_CAND] int64 (key << 32 | block, so one signed compare +# orders them by (key, block)), picked keys / blocks [TOPK] each, per-warp group minima [SELECT_WARPS][16], group +# minima [16], per-warp candidate counts [SELECT_WARPS]. +_CAND = 0 +_PICK_KEY = _CAND + 2 * MAX_CAND +_PICK_BLOCK = _PICK_KEY + TOPK +_WARP_MIN = _PICK_BLOCK + TOPK +_GROUP_MIN = _WARP_MIN + SELECT_WARPS * 16 +_WARP_CNT = _GROUP_MIN + 16 +SMEM_WORDS = _WARP_CNT + SELECT_WARPS +SMEM_BYTES = 4 * SMEM_WORDS + + +def _key(score): + """Order key of a float score: smaller key = larger score, NaN first; signed order = the reference's unsigned.""" + bits = cutlass.Float32(score).bitcast(cutlass.Int32) + return cutlass.select_(bits < cutlass.Int32(0), bits & cutlass.Int32(0x7FFFFFFF), ~bits) + + +def _imin(a, b): + return cutlass.select_(b < a, b, a) + + +def _imax(a, b): + return cutlass.select_(b > a, b, a) + + +def _forced_key(scores, head, block, query, init_blocks, local_start): + raw = scores[head, block, query] + s = cutlass.select_( + block >= local_start, + cutlass.Float32(LOCAL_SCORE), + cutlass.select_(block < init_blocks, cutlass.Float32(INIT_SCORE), cutlass.Float32(raw)), + ) + return _key(s) + + +@cute.jit +def _select_row( + row, + scores: cute.Tensor, # float32 [num_kv_heads, num_blocks, total_q], any strides + n_valid: cute.Tensor, # int32 [>= total_q] + out: cute.Tensor, # int32 [total_q, num_kv_heads, 16] + init_blocks, + local_blocks, + num_kv_heads, + smem: cute.Tensor, # int32 [SMEM_WORDS], shared memory + bar_id: cutlass.Constexpr, + loaded_trigger: cutlass.Constexpr, + raw_prewait, + n_valid_prewait: cutlass.Constexpr, +): + """Select row ``row`` (= query * num_kv_heads + head) with this CTA's SELECT_THREADS threads. Uses named barrier + ``bar_id`` over SELECT_THREADS threads. ``loaded_trigger`` releases the dependent grid once every score of the + row is loaded (trigger phase 1). With ``n_valid_prewait`` the row's valid-block count is ``raw_prewait``, read by + the caller before its grid wait; otherwise it is read here.""" + tid, _, _ = cute.arch.thread_idx() + lane = tid % cutlass.Int32(32) + warp = tid // cutlass.Int32(32) + query = row // num_kv_heads + head = row % num_kv_heads + if cutlass.const_expr(n_valid_prewait): + raw = raw_prewait + else: + raw = n_valid[query] + valid = _imax(_imin(raw, cutlass.Int32(scores.shape[1])), cutlass.Int32(0)) + local_start = _imax(raw - local_blocks, cutlass.Int32(0)) + cand = cute.make_tensor( + cute.recast_ptr(smem.iterator + _CAND, dtype=cutlass.Int64), cute.make_layout(MAX_CAND) + ) + + if tid < TOPK: + smem[_PICK_KEY + tid] = cutlass.Int32(NO_KEY) + smem[_PICK_BLOCK + tid] = cutlass.Int32(0) + + if valid <= TOPK: + # Every valid block is picked (thread t holds block t). + if tid < valid: + smem[_PICK_KEY + tid] = _forced_key(scores, head, tid, query, init_blocks, local_start) + smem[_PICK_BLOCK + tid] = tid + if cutlass.const_expr(loaded_trigger): + griddepcontrol_launch_dependents() + for ci in cutlass.range_constexpr(len(ITEM_CLASSES)): + items = ITEM_CLASSES[ci][0] + lo = TOPK if ci == 0 else ITEM_CLASSES[ci - 1][1] + hi = ITEM_CLASSES[ci][1] + if (valid > lo) & (valid <= hi): + key = cute.make_rmem_tensor(items, cutlass.Int32) + local_min = cutlass.Int32(NO_KEY) + for s in cutlass.range_constexpr(items): + # Every thread loads (a clamped block past the row) so the loads issue back to back, unbranched. + block = tid + cutlass.Int32(s * SELECT_THREADS) + k = _forced_key( + scores, head, _imin(block, valid - 1), query, init_blocks, local_start + ) + key[s] = cutlass.select_(block < valid, k, cutlass.Int32(NO_KEY)) + local_min = _imin(local_min, key[s]) + local_min = _imin(local_min, cute.arch.shuffle_sync_bfly(local_min, 16)) + if lane < 16: + smem[_WARP_MIN + warp * cutlass.Int32(16) + lane] = local_min + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + if cutlass.const_expr(loaded_trigger): + griddepcontrol_launch_dependents() + if tid < 16: + g = cutlass.Int32(NO_KEY) + for w in cutlass.range_constexpr(SELECT_WARPS): + g = _imin(g, smem[_WARP_MIN + w * 16 + tid]) + smem[_GROUP_MIN + tid] = g + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + bound = cutlass.Int32( + -2147483648 + ) # the smallest key: signed keys, unlike the reference's unsigned ones + for g in cutlass.range_constexpr(16): + bound = _imax(bound, smem[_GROUP_MIN + g]) + + # Candidates (key <= bound; unused slots hold NO_KEY > any bound): a CTA prefix count gives the slots. + cnt = cutlass.Int32(0) + for s in cutlass.range_constexpr(items): + cnt += cutlass.select_(key[s] <= bound, cutlass.Int32(1), cutlass.Int32(0)) + incl = cnt + for d in cutlass.range_constexpr(5): + up = cute.arch.shuffle_sync( + incl, _imax(lane - cutlass.Int32(1 << d), cutlass.Int32(0)) + ) + incl += cutlass.select_(lane >= cutlass.Int32(1 << d), up, cutlass.Int32(0)) + if lane == 31: + smem[_WARP_CNT + warp] = incl + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + before = cutlass.Int32(0) + total = cutlass.Int32(0) + for w in cutlass.range_constexpr(SELECT_WARPS): + c = smem[_WARP_CNT + w] + before += cutlass.select_(cutlass.Int32(w) < warp, c, cutlass.Int32(0)) + total += c + slot = before + incl - cnt + for s in cutlass.range_constexpr(items): + if key[s] <= bound: + if slot < MAX_CAND: + cand[slot] = (cutlass.Int64(key[s]) << cutlass.Int64(32)) | cutlass.Int64( + tid + cutlass.Int32(s * SELECT_THREADS) + ) + slot += 1 + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + + if total <= MAX_CAND: + for i in cutlass.range(tid, total, SELECT_THREADS, unroll=1): + mine = cand[i] + rank = cutlass.Int32(0) + for j in cutlass.range(total, unroll=8): + rank += cutlass.select_(cand[j] < mine, cutlass.Int32(1), cutlass.Int32(0)) + if rank < TOPK: + smem[_PICK_KEY + rank] = cutlass.Int32(mine >> cutlass.Int64(32)) + smem[_PICK_BLOCK + rank] = cutlass.Int32(mine & cutlass.Int64(0xFFFFFFFF)) + else: + # More than MAX_CAND keys tied at the bound: rank each candidate against the whole row. + for s in cutlass.range_constexpr(items): + if key[s] <= bound: + block = tid + cutlass.Int32(s * SELECT_THREADS) + rank = cutlass.Int32(0) + for j in cutlass.range(valid, unroll=1): + kj = _forced_key(scores, head, j, query, init_blocks, local_start) + less = (kj < key[s]) | ((kj == key[s]) & (j < block)) + rank += cutlass.select_(less, cutlass.Int32(1), cutlass.Int32(0)) + if rank < TOPK: + smem[_PICK_KEY + rank] = key[s] + smem[_PICK_BLOCK + rank] = block + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + + if warp == 0: + # The 16 picked block ids ascending (NO_KEY = empty or -inf, emitted as -1), bitonic over the warp. + v = cutlass.Int32(NO_KEY) + if lane < TOPK: + pk = smem[_PICK_KEY + lane] + if (pk != cutlass.Int32(NO_KEY)) & (pk != cutlass.Int32(NEG_INF_KEY)): + v = smem[_PICK_BLOCK + lane] + for size_log in cutlass.range_constexpr(1, 6): + for stride_log in cutlass.range_constexpr(size_log - 1, -1, -1): + other = cute.arch.shuffle_sync_bfly(v, 1 << stride_log) + ascending = (lane & cutlass.Int32(1 << size_log)) == cutlass.Int32(0) + lower = (lane & cutlass.Int32(1 << stride_log)) == cutlass.Int32(0) + v = cutlass.select_(lower == ascending, _imin(v, other), _imax(v, other)) + if lane < TOPK: + out[query, head, lane] = cutlass.select_( + v == cutlass.Int32(NO_KEY), cutlass.Int32(-1), v + ) + + +class _SelectLongDslKernel: + """One row per CTA: the grid wait, then ``_select_row``. ``n_valid_prewait`` reads the row's valid-block count + before the grid wait, so that dependent load leaves the post-wait critical path. That is exact because + ``n_valid`` is written only by ``on_update_kv_lens`` and the host copy in ``prepare()``, both before the step's + first layer and behind a plain (non-PDL) launch, so no grid this select can overlap writes it + (``attention/test_modeling_v2_m3_n_valid_contract.py``).""" + + def __init__( + self, use_pdl: bool, early_trigger: bool, trigger_phase: int, n_valid_prewait: bool + ): + self.use_pdl = use_pdl + self.early_trigger = early_trigger + self.loaded_trigger = (not early_trigger) and trigger_phase == 1 + self.n_valid_prewait = n_valid_prewait + + @cute.kernel + def kernel(self, scores, n_valid, out, init_blocks, local_blocks, num_kv_heads): + row, _, _ = cute.arch.block_idx() + smem = SmemAllocator() + words = smem.allocate_tensor(cutlass.Int32, cute.make_layout(SMEM_WORDS), byte_alignment=16) + raw_prewait = cutlass.Int32(0) + if cutlass.const_expr(self.n_valid_prewait): + raw_prewait = n_valid[row // num_kv_heads] + griddepcontrol_wait() + if cutlass.const_expr(self.early_trigger): + griddepcontrol_launch_dependents() + _select_row( + row, + scores, + n_valid, + out, + init_blocks, + local_blocks, + num_kv_heads, + words, + bar_id=1, + loaded_trigger=self.loaded_trigger, + raw_prewait=raw_prewait, + n_valid_prewait=self.n_valid_prewait, + ) + + @cute.jit + def __call__(self, scores, n_valid, out, init_blocks, local_blocks, stream): + num_kv_heads = scores.shape[0] + self.kernel(scores, n_valid, out, init_blocks, local_blocks, num_kv_heads).launch( + grid=(scores.shape[2] * num_kv_heads, 1, 1), + block=(SELECT_THREADS, 1, 1), + stream=stream, + use_pdl=self.use_pdl, + ) + + +#: Compiled kernels by (PDL launch, early trigger, trigger phase, valid count read before the wait). +kernel_cache: Dict[Tuple[bool, bool, int, bool], object] = {} + + +def kernel_key( + use_pdl: bool, early_trigger: bool, trigger_phase: int +) -> Tuple[bool, bool, int, bool]: + """The compile key of a launch (the trigger phase does not matter with the early trigger), with this target's + ``SELECT_N_VALID_PREWAIT``.""" + return ( + bool(use_pdl), + bool(early_trigger), + 0 if early_trigger else int(trigger_phase), + bool(SELECT_N_VALID_PREWAIT), + ) + + +def _compile(key: Tuple[bool, bool, int, bool]): + if key not in kernel_cache: + heads, blocks, rows = cute.sym_int(), cute.sym_int(), cute.sym_int() + scores = make_fake_tensor( + cutlass.Float32, + (heads, blocks, rows), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + n_valid = make_fake_compact_tensor(cutlass.Int32, (cute.sym_int(),), stride_order=(0,)) + out = make_fake_tensor( + cutlass.Int32, (rows, heads, TOPK), stride=(cute.sym_int64(), cute.sym_int64(), 1) + ) + kernel_cache[key] = cute.compile( + _SelectLongDslKernel(*key), + scores, + n_valid, + out, + cutlass.Int32(0), + cutlass.Int32(0), + make_fake_stream(use_tvm_ffi_env_stream=True), + options="--enable-tvm-ffi", + ) + return kernel_cache[key] + + +def select_long_dsl( + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool, + early_trigger: bool, + trigger_phase: int, +) -> None: + """Launch the select on the current stream (compiling its key first if ``prebuild`` did not).""" + _compile(kernel_key(use_pdl, early_trigger, trigger_phase))( + scores, n_valid_blocks, out, init_blocks, local_blocks + ) + + +def prebuild(keys, device) -> list: + """Compile every key and launch each once on a small dummy row (loading its module). Returns the keys.""" + scores = torch.zeros( + 1, 2048, 4, dtype=torch.float32, device=device + ) # capacity: fixed: a dummy prebuild row, not a size + n_valid = torch.full((4,), 2048, dtype=torch.int32, device=device) + out = torch.empty(1, 4, TOPK, dtype=torch.int32, device=device).permute(1, 0, 2) + built = [] + for key in keys: + key = kernel_key(*key) + _compile(key)(scores, n_valid, out, 0, 1) + built.append(key) + return built diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_shard_accept.py new file mode 100644 index 000000000000..2b7d05ad920f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_shard_accept.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 verification for the MiniMax-M3 target head (vocab-sharded TP head, verify rows <= 48). + +Replaces the all-gather of the target logits, the argmax and the strict-acceptance ops of a greedy verify step with +one kernel over this rank's BF16 logits shard (the target lm_head's own local GEMM output): + +* Every logit becomes a 64-bit key ``ordered(value) << 32 | ~global_index`` (ties pick the lowest index, as + ``torch.argmax``), max-reduced per row: warp shuffles, then ``atomicMax``. +* The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping + (Lamport: 0 = empty, the key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the + slots and runs the strict acceptance. Call n uses slot half n % 2. + +The argmax is taken over the BF16 logits without arithmetic, so the tokens are those of the in-tree path's argmax +over the gathered logits. The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/ +shardAccept.cu) and reached through the ``ops/spec`` entry ``m3_shard_accept``. +""" + +from dataclasses import dataclass +from typing import Any, Optional + +import torch + +from ..ops.spec.m3_shard_accept import m3_shard_accept as _shard_accept_op + +#: Verify rows per call, and exchange rows per call half of the workspace. +SHARD_ROWS = 48 # capacity: fixed: DECODE_MAX_TOKENS verify rows + + +@dataclass(eq=False) +class AcceptWorkspace: + """Per-rank exchange slots ([2 halves][SHARD_ROWS rows][world] uint64) behind one multicast mapping, the local + key accumulators and the (finished CTAs, call) counters.""" + + uc: torch.Tensor + mc_ptr: int + keys: torch.Tensor + counters: torch.Tensor + rank: int + world_size: int + handle: Any = None + comm: Any = None + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def create_workspace(mapping, device, slot_rows: int) -> AcceptWorkspace: + """Exchange slots ([2 halves][slot_rows rows][world] uint64) behind one multicast mapping over the TP group, the + local key accumulators [slot_rows] and the (finished CTAs, call) counters. Collective over the TP group; call + outside CUDA-graph capture, in the same order on every rank.""" + words = 2 * slot_rows * max(mapping.tp_size, 1) + if mapping.tp_size > 1: + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + if not _mnnvl_workspace_all_succeeded(comm, True): + comm.Free() + raise RuntimeError("exchange workspace creation failed on another rank") + handle = _make_mnnvl_mcast_buffer( + comm, max(words * 8, 4096), mapping, mapping.is_multi_node() + ) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("exchange workspace creation failed on another rank") + mc_ptr, rank = mc.data_ptr(), mapping.tp_rank + else: + uc = torch.zeros(words, dtype=torch.int64, device=device) + mc_ptr, rank, handle, comm = uc.data_ptr(), 0, None, None + return AcceptWorkspace( + uc=uc, + mc_ptr=mc_ptr, + keys=torch.zeros(slot_rows, dtype=torch.int64, device=uc.device), + counters=torch.zeros(2, dtype=torch.int32, device=uc.device), + rank=rank, + world_size=max(mapping.tp_size, 1), + handle=handle, + comm=comm, + ) + + +def _workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace, created collectively outside CUDA-graph capture on first use.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = create_workspace(mapping, device, SHARD_ROWS) + return _WS + + +def shard_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, +): + """Greedy verification from this rank's BF16 logits shard. + + Args: + logits: [nreq * (draft_len + 1) <= SHARD_ROWS, V / tp] BF16, unit column stride: this rank's columns + (global ids ``vocab_offset ..``) of the verify rows, request-major. + draft_tokens: [nreq, draft_len] int32, the drafts being verified. + batch_idx: [nreq] int32, each request's row in the verify layout. + force_total / force_vec: forced acceptance (see ``m3_shard_accept``). + next_new: optional int32 [nreq, >= 1]; column 0 receives the last accepted token. + + Returns: + (accepted_tokens [nreq, draft_len + 1] int32, num_accepted [nreq] int32, first-pass gather ids [nreq] + int64), or None when the shape is not served or the workspace is missing during a capture. Collective over + the TP group. + """ + nreq, draft_len = draft_tokens.shape + if ( + logits.dim() != 2 + or logits.shape[0] != nreq * (draft_len + 1) + or logits.shape[0] > SHARD_ROWS + or logits.dtype != torch.bfloat16 + or logits.stride(1) != 1 + ): + return None + ws = _workspace(mapping, logits.device) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + _shard_accept_op( + logits, + vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_tokens.contiguous(), + batch_idx, + accepted, + num_accepted, + gather_ids, + force_total, + next_new, + _num_sms(), + force_vec, + ) + return accepted, num_accepted, gather_ids diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_spec_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_spec_tail.py new file mode 100644 index 000000000000..0d4dc9cd1b64 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/m3_spec_tail.py @@ -0,0 +1,116 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Small fused kernels of the MiniMax-M3 Eagle3 step (GB300, decode). + +``chunk_norm`` / ``capture_norm`` apply the drafter's per-layer fc_norm RMSNorms to the captured target hidden states +(the in-tree ``apply_eagle3_fc`` norms, same fp32 arithmetic); ``draft_input`` is a fused draft pass's input (the +embedding lookup, the drafter layer's input_layernorm / hidden_norm and their concat, same arithmetic); +``step_gather`` / ``slot_scatter`` move the step's integer outputs between the spec sampler's store and the engine's +input buffers. +""" + +from typing import Optional, Tuple + +import torch + +# The kernels are compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu) and reached through the +# ops/spec entries. +from ..ops.spec.m3_capture_norm import m3_capture_norm as _capture_norm_op +from ..ops.spec.m3_chunk_norm import m3_chunk_norm as _chunk_norm_op +from ..ops.spec.m3_draft_input import m3_draft_input as _draft_input_op +from ..ops.spec.m3_slot_scatter import m3_slot_scatter as _slot_scatter_op +from ..ops.spec.m3_step_gather import m3_step_gather as _step_gather_op + + +def draft_input( + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + eps_embed: float, + w_hidden: torch.Tensor, + eps_hidden: float, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """(cat([rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden], -1) [M, 2H], hidden[rows] [M, H]) for + int32 ids [M] and int64 rows [M] (rows None: hidden[:M]); ids outside the table embed to zero.""" + m, h = ids.shape[0], table.shape[1] + out = torch.empty((m, 2 * h), dtype=table.dtype, device=table.device) + residual = torch.empty((m, h), dtype=table.dtype, device=table.device) + return _draft_input_op( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, use_pdl + ) + + +def chunk_norm( + src: torch.Tensor, weights: torch.Tensor, eps: float, use_pdl: bool = True +) -> torch.Tensor: + """Per-chunk RMSNorm of src [M, >= C*H] (row stride may exceed C*H) with weights [C, H] -> [M, C*H].""" + c, h = weights.shape + out = torch.empty((src.shape[0], c * h), dtype=src.dtype, device=src.device) + return _chunk_norm_op(src, weights, eps, out, use_pdl) + + +# The overlap scheduler's per-step gathers and the spec sampler's slot stores as one kernel each (step_gather / +# slot_scatter) instead of ~10 and 4 small torch ops; called by the target's SpecStepFold. + + +def step_gather( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 +) -> bool: + """See step_gather_kernel; False when the inputs are outside what the kernel covers (caller falls back).""" + ts = (new_tok, next_draft, lens, slots, ppi, ids, draft, pos_off, kv_off) + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or next_draft.dim() != 2 + or new_tok.dim() < 2 + ): + return False + if (new_tok.dim() == 3 and new_tok.shape[2] != 1) or not ( + lens.is_contiguous() + and ids.is_contiguous() + and draft.is_contiguous() + and pos_off.is_contiguous() + and kv_off.is_contiguous() + ): + return False + _step_gather_op( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 + ) + return True + + +def slot_scatter( + slots, src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft +) -> bool: + """See slot_scatter_kernel (2-D int32 views, zero padding); False to fall back.""" + ts = (src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft) + if any(t.dtype != torch.int32 or not t.is_cuda for t in ts) or slots.dtype != torch.long: + return False + two = (src_new, dst_new, src_next_new, dst_next_new, src_draft, dst_draft) + if any(t.dim() != 2 for t in two) or src_lens.dim() != 1 or dst_lens.dim() != 1: + return False + _slot_scatter_op( + slots, + src_new, + dst_new, + src_next_new, + dst_next_new, + src_lens, + dst_lens, + src_draft, + dst_draft, + ) + return True + + +def capture_norm( + h: torch.Tensor, + residual: Optional[torch.Tensor], + weight: torch.Tensor, + eps: float, + dst: torch.Tensor, +) -> None: + """dst [M, H] (row stride may exceed H) = rmsnorm(h + residual) * weight with chunk_norm's rounding.""" + _capture_norm_op(h, residual, weight, eps, dst) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/__init__.py new file mode 100644 index 000000000000..77d7b84d9bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mixture-of-experts entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_moe_engine.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_moe_engine.py new file mode 100644 index 000000000000..37d5ab5c6c46 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_moe_engine.py @@ -0,0 +1,325 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3's MoE layer for 1..8 decode tokens in one CuTe DSL launch (``m3_moe``, the TP8 build: routed and shared +expert width 384 per rank): the router, the routed NVFP4 experts and the shared MXFP8 expert, on caller-owned state +(an ``M3MoeState`` and one ``M3MoeLayer`` per MoE layer).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.quantization.utils.fp4_utils import ( + get_reorder_rows_for_gated_act_gemm_row_indices, + get_shuffle_matrix_a_row_indices, +) + +from ..m3_moe.op import ( + HIDDEN, + INTER, + SHARED_INTER, + TOP_K, + M3MoeLayer, + M3MoeState, + ar_push_desc, + is_supported, +) + +__all__ = [ + "M3MoeLayer", + "M3MoeState", + "ar_push_desc", + "is_supported", + "m3_moe_engine", + "moe_reference", + "moe_route", + "mxfp8_dequant", + "mxfp8_quant_dequant", + "unswizzle_sf", +] + +# The model's constants the kernels are built with: routed_scaling_factor, the shared expert's SwiGLU-OAI +# (swiglu_alpha, swiglu_limit, the (up + 1) offset). +ROUTED_SCALING = 2.0 +SHARED_ALPHA = 1.702 +SHARED_LIMIT = 7.0 + +_LOG2E = 1.4426950216293334961 +_RCP6 = 0.16666667163372039795 # fp32(1 / 6), the kernels' reciprocal of the E2M1 maximum +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) +_F32_MAX = 3.40282346638528859812e38 + + +def _f32(x: float) -> torch.Tensor: + return torch.tensor(x, dtype=torch.float32) + + +def _e2m1_decode(packed: torch.Tensor) -> torch.Tensor: + """fp32 values of packed E2M1 bytes [..., K / 2] (low nibble first) -> [..., K].""" + lut = torch.tensor(_E2M1, dtype=torch.float32, device=packed.device) + lo = lut[(packed & 0xF).long()] + hi = lut[(packed >> 4).long()] + return torch.stack((lo, hi), dim=-1).flatten(-2) + + +def _e2m1_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to the nearest E2M1 value, ties to even, saturating at 6 (cvt.rn.satfinite.e2m1x2).""" + a = x.abs() + q = torch.full_like(a, 6.0) + for hi, value, inclusive in ( + (5.0, 4.0, True), + (3.5, 3.0, False), + (2.5, 2.0, True), + (1.75, 1.5, False), + (1.25, 1.0, True), + (0.75, 0.5, False), + (0.25, 0.0, True), + ): + q = torch.where(a <= hi if inclusive else a < hi, torch.full_like(a, value), q) + return torch.where(x < 0, -q, q) + + +def _e4m3_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to E4M3, nearest even, saturating (cvt.rn.satfinite.e4m3x2), as fp32.""" + return x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn).float() + + +def _ue8m0_up(x: torch.Tensor) -> torch.Tensor: + """UE8M0 code of x >= 0 rounded toward +inf, saturating (cvt.rp.satfinite.ue8m0x2), int32.""" + bits = x.contiguous().view(torch.int32) + code = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + return code.clamp(0, 254) + + +def unswizzle_sf(sf: torch.Tensor, rows: int, blocks: int) -> torch.Tensor: + """The [rows, blocks] scale bytes of a 128x4-swizzled scale buffer (int64 indices, any byte dtype).""" + flat = sf.reshape(-1).view(torch.uint8) + r = torch.arange(rows, device=flat.device).view(-1, 1) + b = torch.arange(blocks, device=flat.device).view(1, -1) + padded_blocks = (blocks + 3) // 4 * 4 + idx = ( + (r // 128) * (128 * padded_blocks) + + (b // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (b % 4) + ) + return flat[idx] + + +def mxfp8_dequant(q: torch.Tensor, sf: torch.Tensor) -> torch.Tensor: + """fp32 values of e4m3 ``q`` [R, K] with its 128x4-swizzled UE8M0 scales (one per 32 elements).""" + rows, k = q.shape + codes = unswizzle_sf(sf, rows, k // 32).float() + return q.view(torch.float8_e4m3fn).float() * torch.exp2(codes - 127.0).repeat_interleave( + 32, dim=1 + ) + + +def mxfp8_quant_dequant(x: torch.Tensor) -> torch.Tensor: + """x [R, K] through MXFP8 (one UE8M0 scale per 32, the smallest power of two >= amax / 448; e4m3 data), back to + fp32 -- mxfp8_quantize's recipe, which every MXFP8 activation here reproduces.""" + rows, k = x.shape + blocks = x.float().view(rows, k // 32, 32) + code = _ue8m0_up(blocks.abs().amax(dim=-1) * _f32(1.0 / 448.0).to(x.device)) + scale = torch.exp2(code.float() - 127.0).unsqueeze(-1) + return (_e4m3_round(blocks / scale) * scale).view(rows, k) + + +def _bf16(x: torch.Tensor) -> torch.Tensor: + return x.to(torch.bfloat16).float() + + +def _gemm(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """a [M, K] @ b [N, K]^T, accumulated in fp64 and rounded once to fp32.""" + return (a.double() @ b.double().t()).float() + + +def _row_perm(rows: int, gated: bool) -> torch.Tensor: + """The TRTLLM-Gen row order: stored row j holds logical row perm[j] (the gated interleave, then the 32-row shuffle + of epilogue tile 128).""" + probe = torch.empty(rows, 1) + perm = get_shuffle_matrix_a_row_indices(probe, epilogue_tile_m=128) + if gated: + perm = get_reorder_rows_for_gated_act_gemm_row_indices(probe)[perm] + return perm + + +def _nvfp4_expert(w: torch.Tensor, sf: torch.Tensor, perm_inv: torch.Tensor) -> torch.Tensor: + """One expert's stored NVFP4 matrix (values [R, K / 2], swizzled e4m3 scales) as raw fp32 [R, K] in logical row + order (the global scale is not applied: it is the per-expert alpha).""" + rows, half = w.shape + codes = unswizzle_sf(sf, rows, 2 * half // 16).view(torch.float8_e4m3fn).float() + vals = _e2m1_decode(w) * codes.repeat_interleave(16, dim=1) + return vals[perm_inv] + + +def moe_route(x: torch.Tensor, router_weight: torch.Tensor, bias: torch.Tensor): + """MiniMax2 routing on the fp32 router GEMV: (top-4 expert ids [M, 4] int64, bf16-rounded weights [M, 4] fp32).""" + logits = _gemm(x, router_weight) + sig = (0.5 * torch.tanh(0.5 * logits).double() + 0.5).float() + keys = sig + bias + ids = torch.sort(keys, dim=-1, descending=True, stable=True).indices[:, :TOP_K] + sk = keys.gather(1, ids) - bias[ids] + total = (sk[:, 0] + sk[:, 2]) + (sk[:, 1] + sk[:, 3]) + w = (sk * _f32(ROUTED_SCALING).to(x.device)) / (total + _f32(1e-20).to(x.device)).unsqueeze(1) + return ids, _bf16(w) + + +def _nvfp4_act(x: torch.Tensor, sc: torch.Tensor) -> torch.Tensor: + """The activation's NVFP4 quantization (fp4_quantize(x, sc, 16)), as raw fp32 values (q x block scale).""" + rows, k = x.shape + blocks = x.view(rows, k // 16, 16) + vmax = blocks.abs().amax(dim=-1, keepdim=True) + sff = _e4m3_round(sc * (vmax * _f32(_RCP6).to(x.device))) + # The kernels' rcp.approx(sf x rcp.approx(sc)) lands where the correctly rounded sc / sf does on every element + # measured (the 1 / (sf x (1 / sc)) spelling differs on ~0.06% of them). + osc = torch.where(vmax != 0, sc / sff, torch.zeros_like(vmax)) + return (_e2m1_round(blocks * osc) * sff).view(rows, k) + + +def _fc1_epilogue(up: torch.Tensor, gate: torch.Tensor, esc: torch.Tensor) -> torch.Tensor: + """TRTLLM-Gen's SwiGLU-OAI FC1 epilogue on the raw FC1 sums (fp32 [I]) and its NVFP4 requantization: the raw FC2 + input values. ``esc`` = (fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha) of the expert, with + act_beta and act_clamp already divided by fc31_alpha (the loader's convention).""" + a, c, alpha, beta, lim = (esc[i] for i in range(5)) + gc = torch.where(gate > lim, lim, gate) + gc = torch.where(gate < -_F32_MAX, torch.full_like(gc, -_F32_MAX), gc) + uc = torch.minimum(torch.maximum(up, -lim), lim) + c1 = a * (alpha * _f32(_LOG2E).to(up.device)) + sgm = 1.0 / (torch.exp2(-(gc * c1)) + 1.0) + h = (uc.double() * a.double() + (a * beta).double()).float() * (gc * sgm) + blocks = h.view(-1, 16) + amax = (blocks.abs() * c).amax(dim=-1, keepdim=True) + sff = _e4m3_round(amax * _f32(_RCP6).to(up.device)) + osc = torch.where(sff != 0, 1.0 / sff, torch.zeros_like(sff)) + return (_e2m1_round((blocks * osc) * c) * sff).view(-1) + + +def _shared_expert(x: torch.Tensor, shared_mxfp8) -> torch.Tensor: + """The shared expert as the target runs it outside the engine: MXFP8 gate_up (bf16 out), SwiGLU-OAI with the + model's alpha / limit / (up + 1) in fp32 (bf16 out), MXFP8 down (bf16 out).""" + gu_w, gu_sf, dn_w, dn_sf = shared_mxfp8 + gu = _bf16(_gemm(mxfp8_quant_dequant(x), mxfp8_dequant(gu_w, gu_sf))) + gate, up = gu[:, :SHARED_INTER], gu[:, SHARED_INTER:] + a = torch.minimum(gate, torch.full_like(gate, SHARED_LIMIT)) + ub = up.clamp(-SHARED_LIMIT, SHARED_LIMIT) + 1.0 + sg = 1.0 / ( + torch.exp2((a * _f32(-SHARED_ALPHA).to(x.device)) * _f32(_LOG2E).to(x.device)) + 1.0 + ) + h = _bf16(ub * (sg * a)) + return _bf16(_gemm(mxfp8_quant_dequant(h), mxfp8_dequant(dn_w, dn_sf))) + + +def moe_reference(hidden: torch.Tensor, layer: M3MoeLayer, shared: bool) -> torch.Tensor: + """The MoE layer of the decode engines (both token ranges compute this), in plain torch: bf16 [M, 6144].""" + x = hidden.float() + m = x.shape[0] + ids, weights = moe_route(x, layer.router_weight, layer.bias) + xr = _nvfp4_act(x, layer.fc31_input_scale.reshape(()).float()) + inv31 = torch.argsort(_row_perm(2 * INTER, gated=True)).to(x.device) + inv2 = torch.argsort(_row_perm(HIDDEN, gated=False)).to(x.device) + y = torch.zeros(m, TOP_K, HIDDEN, dtype=torch.float32, device=x.device) + for e in torch.unique(ids).tolist(): + w31 = _nvfp4_expert(layer.w3_w1_weight[e], layer.w3_w1_weight_scale[e], inv31) + w2 = _nvfp4_expert(layer.w2_weight[e], layer.w2_weight_scale[e], inv2) + esc = layer.escal[:, e] + for t, k in (ids == e).nonzero().tolist(): + acc1 = _gemm(xr[t : t + 1], w31)[0] + # The buffer holds [w3 (up); w1 (gate)]. + h = _fc1_epilogue(acc1[:INTER], acc1[INTER:], esc) + y[t, k] = _bf16(_gemm(h.unsqueeze(0), w2)[0] * esc[5]) + acc = torch.zeros(m, HIDDEN, dtype=torch.float64, device=x.device) + for k in range(TOP_K): + acc = (acc + weights[:, k : k + 1].double() * y[:, k].double()).float().double() + routed = acc.float() + if not shared: + return routed.to(torch.bfloat16) + return (_shared_expert(x, layer.shared_mxfp8) + _bf16(routed)).to(torch.bfloat16) + + +class M3MoeEngine(OpWrapper): + """One MiniMax-M3 MoE layer for M (1..8) decode tokens: ``layer.state.run(layer, hidden, out, arp)``. + + ``hidden`` bf16 [M, 6144] contiguous, M = the state's token count. ``layer`` (from ``M3MoeState.layer``) carries the + TRTLLM-Gen NVFP4 expert buffers read in place, the per-expert scalars, the router weight / bias, the activation's + NVFP4 global scale and, on a ``shared`` state, the shared expert's MXFP8 weights; its state carries the compiled + kernel and the workspace every layer on it shares. Returns bf16 [M, 6144]: bf16(shared + routed) on a ``shared`` + state, else the routed partial. ``out``: write there instead of a new tensor. ``arp`` (``ar_push_desc``): also + push the rows into a multicast all-reduce exchange. + + The routed part is TRTLLM-Gen's NVFP4 MoE on the fp32 router logits: sigmoid (tanh form) + correction bias top-4 + of 128 (ties to the lower id), weights sigmoid / sum * 2.0 rounded to bf16; the activation quantized to NVFP4 with + the layer's global scale; per expert the raw FC1 sums, SwiGLU-OAI (alpha / beta / clamp), the NVFP4 requant with + fc31_scale_c, FC2, bf16(fc2_alpha x sum); the fma chain over the top-4 in order. The shared expert is the target's + MXFP8 SwiGLU-OAI MLP. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # One cell per token count: m_max is a trace-time constant, so each is its own build, and the target builds all + # eight (decode batches of 1..8 requests, spec-decode verify rows up to 8). The whys name what the build differs in. + CELLS: tuple[Cell, ...] = ( + Cell( + why="1 token: FC1 -> FC2 records replicated 32x, logits 32x, last phase without trailing FC2", + spec=dict(tokens=1), + ), + Cell( + why="2 tokens: records 32x, logits 32x, the one build without AR_PRE", + spec=dict(tokens=2), + ), + Cell(why="3 tokens: records 8x, logits 32x, 4 FC2-B builder warps", spec=dict(tokens=3)), + Cell( + why="4 tokens: one request verifying 3 drafts (the TP8 operating point), records 8x, logits 32x", + spec=dict(tokens=4), + ), + Cell( + why="5 tokens: single records, a routing warp takes two tokens, logits 32x", + spec=dict(tokens=5), + ), + Cell(why="6 tokens: one request verifying a 5-token draft", spec=dict(tokens=6)), + Cell(why="7 tokens: the B2IMG global FC1 -> FC2 hand-off", spec=dict(tokens=7)), + Cell( + why="8 tokens: B2IMG with a 6-stage ring, 32 distinct experts at most", + spec=dict(tokens=8), + ), + ) + + note = """ + The kernel's elementwise math uses the PTX approximations (rcp.approx, ex2.approx, tanh) where `reference` uses + torch's correctly rounded ones, and its GEMMs sum in tensor-core order where `reference` sums in fp64: a + quantization boundary (NVFP4 activation / intermediate code, routing near-tie) can fall on the other side, so the + two agree within the band the test derives, not bit for bit. The routed part is bit for bit the stock TRTLLM-Gen + NVFP4 MoE (fp4_quantize + fp4_block_scale_moe_runner) on the same routing; the test checks that too. + + Not covered: the all-reduce push (`arp`), which needs a multicast exchange (the decode forward's boundary all-reduce + exercises it); the non-shared state (`shared=False`), which this target never builds. Every layer on one state + must run in one stream order: the workspace is shared and tagged per call, never reset. + + No `is_valid`: the compiled call checks every buffer's dtype, shape and alignment against its signature and + raises on a mismatch (a misaligned `hidden`, an `out` that is not bf16 [M, 6144], a malformed `arp`). + """ + + def __call__( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return layer.state.run(layer, hidden, out, arp=arp) + + def reference( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """Plain torch over the layer's buffers (see the class docstring); `out` and `arp` are ignored.""" + return moe_reference(hidden, layer, shared=layer.state.shared) + + +m3_moe_engine = M3MoeEngine() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_gemv.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_gemv.py new file mode 100644 index 000000000000..37870d0e6cd7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_gemv.py @@ -0,0 +1,101 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""fp32 router GEMV of the decode MoE: ``logits[M, E] = float(x) @ W^T`` at one token.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +_E4M3_MAX = 448.0 +_E2M1_MAX = 6.0 +#: Midpoints between consecutive e2m1 magnitudes; ``_TIE_UP[i]``: round-to-nearest-even takes the upper code. +_E2M1_MIDPOINTS = (0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0) +_TIE_UP = (False, True, False, True, False, True, False) +#: The kernel builds each block's output scale through two ``rcp.approx.ftz.f32`` reciprocals (~2^-23 relative): +#: a value this close to an e2m1 midpoint may round to either neighbour, everything further out must match. +TIE_WINDOW = 2.0**-20 + + +def nvfp4_quantize( + x: torch.Tensor, sf_scale: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """``fp4_quantize(x, sf_scale, 16, False, False)``: (packed e2m1 [M, K / 2], linear E4M3 scales [M * K / 16], + the pre-rounding scaled values [M, K] that ``near_tie`` reads). + + Per 16 columns: scale = e4m3(sf_scale * amax / 6), data = x * sf_scale / scale rounded to nearest-even e2m1. + """ + gs = sf_scale.float().reshape(()) + m, k = x.shape + blocks = x.float().view(m, k // 16, 16) + vecmax = blocks.abs().amax(-1) + sf = (gs * (vecmax / _E2M1_MAX)).clamp(max=_E4M3_MAX).to(torch.float8_e4m3fn) + out_scale = torch.where(vecmax != 0, gs / sf.float(), torch.zeros_like(vecmax)) + scaled = (blocks * out_scale.unsqueeze(-1)).view(m, k) + mid = torch.tensor(_E2M1_MIDPOINTS, device=x.device, dtype=torch.float32) + tie_up = torch.tensor(_TIE_UP, device=x.device, dtype=torch.bool) + a = scaled.abs().unsqueeze(-1) + code = ((a > mid).sum(-1) + ((a == mid) & tie_up).any(-1)).to(torch.uint8) + negative = (scaled < 0) | ((scaled == 0) & torch.signbit(scaled)) + code = code | (negative.to(torch.uint8) << 3) + packed = code[:, 0::2] | (code[:, 1::2] << 4) + return packed, sf.view(torch.uint8).reshape(-1), scaled + + +def near_tie(scaled: torch.Tensor) -> torch.Tensor: + """Elements whose e2m1 rounding the kernel's approximate reciprocals decide (see ``TIE_WINDOW``).""" + mid = torch.tensor(_E2M1_MIDPOINTS, device=scaled.device, dtype=torch.float32) + a = scaled.abs().unsqueeze(-1) + return ((a - mid).abs() <= TIE_WINDOW * a).any(-1) + + +def fp32_logits(x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + """``float(x) @ w^T`` as a true fp32 product (TF32 off).""" + with true_fp32_matmul(): + return x.float() @ w.t() + + +class M3RouterGemv(OpWrapper): + """fp32 router logits ``float(x[M, K]) @ w[E, K]^T`` with exact fp32 FMA, one CTA per expert. + + ``x`` is BF16 with 1..16 rows, unit column stride and 16-byte rows; ``w`` is the fp32 router weight; K a + multiple of 1024. ``rows`` experts per CTA; only 1 is built. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The target takes the GEMV at exactly one token (2..48 go to the tensor-core router): 128 experts, hidden 6144. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one decode token, 128 experts x hidden 6144", + spec=dict(m=1, e=128, k=6144), + ), + ) + + note = """ + The op accepts up to 16 rows; the target sends one, so wider calls are not cells. The logits differ from a true + fp32 product only in accumulation order (per-thread FMA chains over K / 1024 strided column groups, a warp tree + and an 8-warp sum). + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor, rows: int, pdl: bool) -> torch.Tensor: + return torch.ops.trtllm.minimax_m3_router_gemv(x, w, rows, pdl) + + def reference(self, x: torch.Tensor, w: torch.Tensor, rows: int, pdl: bool) -> torch.Tensor: + """True fp32 product; ``rows`` and ``pdl`` do not enter the arithmetic.""" + return fp32_logits(x, w) + + def is_valid(self, x: torch.Tensor, w: torch.Tensor, rows: int, pdl: bool) -> None: + # The activation rows are bulk-copied from x's base address; the op checks the row stride but not the base, + # and a misaligned bulk copy faults the context rather than raising. + assert x.data_ptr() % 16 == 0, "x must start 16-byte aligned (cp.async.bulk source)" + + +m3_router_gemv = M3RouterGemv() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_gemv_fp4.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_gemv_fp4.py new file mode 100644 index 000000000000..58f809e82cf6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_gemv_fp4.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""fp32 router GEMV plus the routed experts' NVFP4 activation, one token.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_router_gemv import fp32_logits, nvfp4_quantize + + +class M3RouterGemvFp4(OpWrapper): + """``[logits fp32 [M, E], x_fp4 [M, K / 2] uint8, x_sf [M * K / 16] uint8]`` from one kernel. + + The logits are ``m3_router_gemv``'s; the NVFP4 activation is ``fp4_quantize(x, sf_scale, 16, False, False)`` + (16-element blocks, E4M3 scales in the linear layout, per-tensor ``sf_scale``) for the routed experts' FC1. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The fused-router decode step at one token: 128 experts, hidden 6144, the experts' input scale. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one decode token: logits for 128 experts and the NVFP4 activation of hidden 6144", + spec=dict(m=1, e=128, k=6144, sf_scale=448.0 * 6.0 / 8.0), + ), + Cell( + why="a global scale that puts block scales near the E4M3 top, where the scale cast saturates", + spec=dict(m=1, e=128, k=6144, sf_scale=448.0 * 6.0 / 2.0), + ), + ) + + note = """ + The e2m1 codes are exact except within `TIE_WINDOW` of a rounding midpoint, where the kernel's approximate + reciprocals decide the direction. Up to 16 rows are accepted; the target sends one. + """ + + def __call__( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: torch.Tensor, pdl: bool + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_router_gemv_fp4(x, w, sf_scale, pdl) + + def reference( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: torch.Tensor, pdl: bool + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """True fp32 logits and the NVFP4 recipe; ``pdl`` does not enter the arithmetic.""" + packed, sf, _ = nvfp4_quantize(x, sf_scale) + return fp32_logits(x, w), packed, sf + + def is_valid(self, x: torch.Tensor, w: torch.Tensor, sf_scale: torch.Tensor, pdl: bool) -> None: + assert x.data_ptr() % 16 == 0, "x must start 16-byte aligned (cp.async.bulk source)" + # A zero or non-finite global scale makes every block scale 0 or NaN and the data codes garbage; the op + # reads the scalar on the device and cannot refuse it. + value = float(sf_scale.reshape(())) + assert value > 0.0 and value != float("inf"), ( + f"sf_scale must be positive and finite, got {value}" + ) + + +m3_router_gemv_fp4 = M3RouterGemvFp4() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_tc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_tc.py new file mode 100644 index 000000000000..e88af51f7fa2 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/moe/m3_router_tc.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TF32 tensor-core router of the decode MoE for 2..48 tokens, optionally with the NVFP4 activation.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +from .m3_router_gemv import nvfp4_quantize + + +def tf32_rna(w: torch.Tensor) -> torch.Tensor: + """fp32 -> TF32 as ``cvt.rna.tf32.f32``: keep 10 mantissa bits, round to nearest with ties away from zero.""" + bits = w.contiguous().view(torch.int32) + rounded = (bits + 0x1000) & ~0x1FFF + finite = (bits & 0x7F800000) != 0x7F800000 + return torch.where(finite, rounded, bits).view(torch.float32) + + +class M3RouterTc(OpWrapper): + """Router logits on TF32 tensor cores for 128 experts and 1..48 tokens: ``x @ tf32(w)^T``, fp32 accumulate. + + ``x`` is BF16 (exact in TF32), ``w`` the fp32 router weight, rounded to TF32 in the kernel. The 16 K slices' + partials are summed in slice order whatever cluster size the device co-schedules (16 or 8 CTAs), so the result + does not depend on it. With ``sf_scale`` also the NVFP4 activation as ``m3_router_gemv_fp4``: returns + ``[logits]`` or ``[logits, x_fp4, x_sf]``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # The target routes 2..48 tokens here: plain logits on the unfused path, logits + NVFP4 on the fused-router path. + CELLS: tuple[Cell, ...] = ( + Cell( + why="two tokens, logits only (the smallest call the target makes here)", + spec=dict(m=2, k=6144, fp4=False), + ), + Cell( + why="8 tokens with the NVFP4 activation: one 8-token MMA tile, every expert tile quantizes one row", + spec=dict(m=8, k=6144, fp4=True), + ), + Cell( + why="13 tokens: a partial second token tile", + spec=dict(m=13, k=6144, fp4=True), + ), + Cell( + why="the 48-token maximum, six token tiles, logits only", + spec=dict(m=48, k=6144, fp4=False), + ), + Cell( + why="the 48-token maximum with the NVFP4 activation", + spec=dict(m=48, k=6144, fp4=True), + ), + # Every other row count the router takes (dispatch.ROUTER_TC), with and without the NVFP4 activation. + *( + Cell( + why=f"{m} rows: a router row count between the ones above", + spec=dict(m=m, k=6144, fp4=m % 2 == 1), + ) + for m in range(2, 49) + if m not in (2, 8, 13, 48) + ), + ) + + note = """ + The kernel uses 16-CTA clusters when the device co-schedules eight of them and 8-CTA clusters otherwise; both + sum the same 16 slice partials in the same order. Only the one the test device picks is driven. + """ + + def __call__( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_router_tc(x, w, sf_scale, pdl) + + def reference( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> tuple[torch.Tensor, ...]: + """fp32 product of ``x`` and the TF32-rounded weight, plus the NVFP4 recipe with ``sf_scale``.""" + with true_fp32_matmul(): + logits = x.float() @ tf32_rna(w).t() + if sf_scale is None: + return (logits,) + packed, sf, _ = nvfp4_quantize(x, sf_scale) + return logits, packed, sf + + def is_valid( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> None: + assert x.data_ptr() % 16 == 0, "x must start 16-byte aligned (cp.async.bulk source)" + if sf_scale is not None: + value = float(sf_scale.reshape(())) + assert value > 0.0 and value != float("inf"), ( + f"sf_scale must be positive and finite, got {value}" + ) + + +m3_router_tc = M3RouterTc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/msa_kv_update.py new file mode 100644 index 000000000000..8c2a67356d3f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/msa_kv_update.py @@ -0,0 +1,89 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One kernel for MSA's device-side KV-length correction (MsaAttentionMetadata.on_update_kv_lens). + +The torch formulation is ~23 one-CTA kernels (min, cumsum, gathers, div / clamp / remainder, the block-table lookup, +the valid-block counts) and runs at the head of every captured decode step, twice. This kernel computes the same +int32 results in one launch: + + kv_true[b] = min(kv_lens[b], staged[b]) -> seq_lens_out[b], cu_kv[b + 1] = prefix sum (cu_kv[0] = 0) + pos[t] = kv_true[r] - qo[r] + intra[t] (r = q_batch_row[t]) + out_loc[t] = table[r, min(max(pos, 0) // page, ncols - 1)] * page + max(pos, 0) % page + n_valid[t] = (max(pos + 1, 0) + page - 1) // page + +``msa_kv_update`` is the target's ``MsaKvUpdate`` (``tensorrt_llm._torch.pyexecutor.runtime_hooks``): the MSA +metadata calls it with its own buffers and falls back to the torch formulation when it returns False. The kernel is +``torch.ops.trtllm.minimax_m3_msa_kv_update``, called through this target's ``msa_kv_update`` entry. +""" + +from typing import Optional + +import torch + +from ..ops.attention.msa_kv_update import msa_kv_update as _kv_update_op + + +def msa_kv_update( + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], +) -> bool: + """The fused correction (see module doc); False, with nothing written, when an operand is outside what the + kernel covers. Arguments in ``MsaKvUpdate`` order; ``seq_lens`` are the per-request query lengths (``qo``).""" + ts = [ + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + msa_seq_lens, + out_cache_loc, + ] + ts += [t for t in (cu_kv_lens, n_valid_blocks) if t is not None] + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or block_table.dim() != 2 + or block_table.stride(1) != 1 + ): + return False + if any(not t.is_contiguous() for t in ts if t is not block_table): + return False + if ( + kv_lens.numel() < batch + or kv_lens_staged.numel() < batch + or seq_lens.numel() < batch + or msa_seq_lens.numel() < batch + or q_batch_row.numel() < total_q + or q_intra.numel() < total_q + or out_cache_loc.numel() < total_q + or (cu_kv_lens is not None and cu_kv_lens.numel() < batch + 1) + or (n_valid_blocks is not None and n_valid_blocks.numel() < total_q) + ): + return False + _kv_update_op( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/__init__.py new file mode 100644 index 000000000000..7d2ec0224cf2 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) as a CuTe DSL kernel, configured for the TP8 +projection shapes (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/m3_mx_gemv_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/m3_mx_gemv_kernel.py new file mode 100644 index 000000000000..3ba6fb0d7c8c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/m3_mx_gemv_kernel.py @@ -0,0 +1,1472 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 decode GEMV on tcgen05: ``y[M, N] = x[M, K] @ W[N, K]^T`` for M <= 8, block-scaled E4M3 x E4M3. + +The weight is the MMA's A operand (128 output features per CTA), the activation is B (8 token columns), +fp32 accumulation in TMEM. Both operands come in their MXFP8 storage: E4M3 values (K contiguous) and UE8M0 +scales in the 128x4-swizzled layout, whose 512-byte atom per (128 rows, 128 K) is exactly the source of a +``tcgen05.cp`` 32x128b.warpx4 scale copy, so both scale tensors are bulk-copied unchanged. For an activation of at +most 32 rows the atom of k-tile ``k`` sits at byte ``512 k``. + +A cluster of SPLIT CTAs shares one 128-row weight tile; rank r streams the interleaved k-tiles r, r + SPLIT, ... +through a RING-stage ring. Before the grid dependency each CTA issues its first RING weight tiles (and their scale +atoms) and prefetches the rest into L2; after it, the activation slice of the CTA (values and scale atoms) lands in +one buffer. Rank r owns rows [r * 128 / SPLIT, (r + 1) * 128 / SPLIT) of the tile: the other ranks store their fp32 +partials of those rows into its shared memory and arrive on its barrier (release, cluster scope); the owner adds the +SPLIT partials in rank order and stores bf16. + +``prod`` (#117, the sparse QKV + index projection at split 8, at most 8 tokens): instead of storing the packed row, the +owners stage their bf16 rows, the CTAs exchange them by st.async so that rank t holds token t's 128-dim row of this +tile's packed head, and rank t's warp 4 runs that head's producer (``m3_layer.phase_producer.producer_head``: RMSNorm, +RoPE, FP8 q / index-q / index-K and NVFP4 K / V stores) on it, bit for bit what the separate producer kernel does with +the stored row. + +Warps: 0 weight TMA, 1 activation TMA, 2 TMEM allocation + scale copies + MMA, 3 idle, 4-7 epilogue (warp 4 + i +reads TMEM lanes 32 i .. 32 i + 31). +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass import dsl_user_op +from cutlass.experimental import primitives as prims + +from ..m3_layer.phase_producer import ProducerArgs, bf16_bits_to_f32, producer_head + +CTA_M = 128 # capacity: fixed: a kernel tile constant (weight rows per CTA = the MMA's M) +MMA_N = 8 # token columns +CTA_K = 128 # capacity: fixed: a kernel tile constant (one k-tile: 128 E4M3 bytes per row = one 128-byte swizzle row) +MMA_INST_K = 32 +# 4 MMAs per k-tile, one per scale byte of the atom. +K_BLOCKS = CTA_K // MMA_INST_K # capacity: fixed: a kernel tile constant +THREADS = 256 +SF_ATOM_BYTES = 512 +SF_ATOM_U16 = SF_ATOM_BYTES // 2 +SF_COLS = 4 # TMEM columns of one scale atom +W_TILE_BYTES = CTA_M * CTA_K +X_TILE_BYTES = MMA_N * CTA_K +EVICT_FIRST = 0x12F0000000000000 +A_COLS_PER_MMA = MMA_INST_K // 4 # TMEM columns of one 128 x K32 E4M3 granule of A + +a_dtype = cutlass.Float8E4M3FN +b_dtype = cutlass.Float8E4M3FN + + +def num_k_tiles(k_in: int) -> int: + return k_in // CTA_K + + +def acc_cols(nacc: int, mma_n: int = MMA_N) -> int: + # nacc independent accumulators of MMA_N columns: MMA j of the CTA accumulates into j % nacc, so the M128 x N8 + # MMAs do not form one serial dependency chain; the epilogue adds them. + return max(16, nacc * mma_n) + + +def a_col0(nacc: int, my_tiles: int, mma_n: int = MMA_N) -> int: + # a_tmem: [acc | SFA per k-tile | one SFB slot (reused in issue order) | A: 32 columns (128 x 128 E4M3) per k-tile] + return (acc_cols(nacc, mma_n) + (my_tiles + 1) * SF_COLS + 31) // 32 * 32 + + +def tmem_cols( + ring: int, nacc: int, a_tmem: bool = False, my_tiles: int = 0, mma_n: int = MMA_N +) -> int: + need = acc_cols(nacc, mma_n) + 2 * ring * SF_COLS + if a_tmem: + need = a_col0(nacc, my_tiles, mma_n) + my_tiles * K_BLOCKS * A_COLS_PER_MMA + cols = 32 + while cols < need: + cols *= 2 + return cols + + +@dsl_user_op +def read_globaltimer(*, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), [], "mov.u64 $0, %globaltimer;", "=l", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _mapa_u32(smem_ptr, peer, *, loc=None, ip=None): + """The shared::cluster address of this CTA's shared-memory location in cluster CTA ``peer``.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [smem_ptr.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(peer).ir_value(loc=loc, ip=ip)], + "mapa.shared::cluster.u32 $0, $1, $2;", "=r,r,r", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_async_v4(dst, w0, w1, w2, w3, mbar, *, loc=None, ip=None): + """st.async of four int32 words to a shared::cluster address, completing ``mbar`` there by 16 bytes.""" + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, + [cutlass.Int32(dst).ir_value(loc=loc, ip=ip), cutlass.Int32(w0).ir_value(loc=loc, ip=ip), + cutlass.Int32(w1).ir_value(loc=loc, ip=ip), cutlass.Int32(w2).ir_value(loc=loc, ip=ip), + cutlass.Int32(w3).ir_value(loc=loc, ip=ip), cutlass.Int32(mbar).ir_value(loc=loc, ip=ip)], + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [$0], {$1, $2, $3, $4}, [$5];", "r,r,r,r,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _ld_shared_v4_f32(smem_ptr, *, loc=None, ip=None): + """ld.shared.v4.f32 of four fp32 at a 16-byte-aligned shared-memory pointer.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + out = _llvm.inline_asm( + _llvm.StructType.get_literal([_T.f32()] * 4), [smem_ptr.toint(loc=loc, ip=ip).ir_value()], + "ld.shared.v4.f32 {$0, $1, $2, $3}, [$4];", "=f,=f,=f,=f,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + return [ + cutlass.Float32(_llvm.extractvalue(_T.f32(), out, [i], loc=loc, ip=ip)) for i in range(4) + ] + + +@dsl_user_op +def _ld_shared_v4_b32(smem_ptr, *, loc=None, ip=None): + """ld.shared.v4.b32 of four int32 words at a 16-byte-aligned shared-memory pointer.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + out = _llvm.inline_asm( + _llvm.StructType.get_literal([_T.i32()] * 4), [smem_ptr.toint(loc=loc, ip=ip).ir_value()], + "ld.shared.v4.b32 {$0, $1, $2, $3}, [$4];", "=r,=r,=r,=r,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + return [cutlass.Int32(_llvm.extractvalue(_T.i32(), out, [i], loc=loc, ip=ip)) for i in range(4)] + + +@dsl_user_op +def _test_wait_cluster(mbar, parity, *, loc=None, ip=None): + """mbarrier.test_wait.parity.acquire.cluster on a barrier of this CTA completed by other CTAs' st.async.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + done = cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [mbar.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(parity).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tmbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [$1], $2;\n\t" + "selp.u32 $0, 1, 0, p;\n\t}", "=r,r,r,~{memory}", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + return done != cutlass.Int32(0) + + +@dsl_user_op +def _absmax(acc, v, *, loc=None, ip=None): + """max(acc, |v|).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(acc).ir_value(loc=loc, ip=ip), cutlass.Float32(v).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .f32 a;\n\tabs.f32 a, $2;\n\tmax.f32 $0, $1, a;\n\t}", "=f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _rcp_approx_ftz(x, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _e4m3x4(v0, v1, v2, v3, *, loc=None, ip=None): + """Four f32 to E4M3 (RN, satfinite), packed little-endian into one int32 word (v0 in the low byte).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (v0, v1, v2, v3)], + "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32(addr, value, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip)], + "st.global.b32 [$0], $1;", "l,r", has_side_effects=True, is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16(x) (RN) as the fp32 bit pattern of the rounded value.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b16 h, z;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _bf16_round(x): + return _bf16_bits(x).bitcast(cutlass.Float32) + + +# The swigluoai activation of MiniMax-M3's dense MLP, gate * sigmoid(1.702 * gate) * (up + 1) with gate clamped to <= 7 +# and up to [-7, 7], in exactly the instruction sequence the Triton silu_and_mul kernel compiles to (modules/swiglu.py +# on SM107): the gate clamp on the packed bf16 pair, the up clamp as min.xorsign.abs, two rounded multiplies into +# ex2.approx, div.full for the reciprocal, two rounded multiplies, round to bf16. No multiply feeds an add, so the +# explicit .rn forms compile to what the Triton kernel's contractable forms do. +SWIGLU_OAI_ALPHA = 1.702 +SWIGLU_OAI_LIMIT = 7.0 + + +@dsl_user_op +def _swiglu_oai_bf16x2(gate_word, up_word, *, loc=None, ip=None): + """The activation of one packed bf16 pair of gate and up values: its two bf16 results as one int32 word (the low + half from the low halves).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [cutlass.Int32(w).ir_value(loc=loc, ip=ip) for w in (gate_word, up_word)], + "{\n\t.reg .b32 g, glim, gl, gh, ul, uh, one, lim, t0, t1, e0, e1, s0, s1, r0, r1;\n\t.reg .b16 hl, hh;\n\t" + "mov.b32 glim, 0x40E040E0;\n\tmin.bf16x2 g, $1, glim;\n\t" + "mov.b32 {hl, hh}, g;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, $2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mov.b32 lim, 0f40E00000;\n\tmin.xorsign.abs.f32 ul, ul, lim;\n\tmin.xorsign.abs.f32 uh, uh, lim;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 ul, ul, one;\n\tadd.rn.f32 uh, uh, one;\n\t" + "mul.rn.f32 t0, gl, 0fBFD9DB23;\n\tmul.rn.f32 t1, gh, 0fBFD9DB23;\n\t" + "mul.rn.f32 t0, t0, 0f3FB8AA3B;\n\tmul.rn.f32 t1, t1, 0f3FB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "add.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 r0, s0, gl;\n\tmul.rn.f32 r1, s1, gh;\n\t" + "mul.rn.f32 r0, ul, r0;\n\tmul.rn.f32 r1, uh, r1;\n\t" + "cvt.rn.bf16x2.f32 $0, r1, r0;\n\t}", "=r,r,r", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _rope_bits(c, a, s, b, *, loc=None, ip=None): + """bf16(c * a + s * b) with separately rounded products (no FMA), as fp32 bits: the stock RoPE arithmetic.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (c, a, s, b)], + "{\n\t.reg .f32 p, q, r;\n\t.reg .b16 h, z;\n\tmul.rn.f32 p, $1, $2;\n\tmul.rn.f32 q, $3, $4;\n\t" + "add.rn.f32 r, p, q;\n\tcvt.rn.bf16.f32 h, r;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _din_load4(table, hidden, hidden_half, word_idx): + """Four int32 words at word_idx of the hidden rows (hidden_half == 1) or the embedding table: one CTA-uniform + base-address select instead of a branch.""" + addr = cutlass.select_( + hidden_half == cutlass.Int32(1), hidden.data_ptr(0).toint(), table.data_ptr(0).toint() + ) + ptr = cutlass.inttoptr( + addr + cutlass.Int64(word_idx) * cutlass.Int64(4), mem_space=1, dtype=cutlass.Int32 + ) + return ptr.load(count=4, alignment=16) + + +def _stamp(timers, timed: bool, bx, slot: int): + # Diagnostic: %globaltimer of milestone `slot` of CTA bx (timers [grid, 8] int64). + if timed: + timers.store(read_globaltimer(), idx=bx * cutlass.Int32(8) + cutlass.Int32(slot)) + + +def _k_tile(rank, i: int, split: int, my_tiles: int, contig: bool): + # contig: rank r streams k-tiles [r * my_tiles, (r + 1) * my_tiles); else the interleaved r, r + split, ... + if contig: + return rank * cutlass.Int32(my_tiles) + cutlass.Int32(i) + return rank + cutlass.Int32(i * split) + + +@cute.kernel +def m3_mx_gemv_kernel( + tma_desc_w: cutlass.GridConstant[cuda.TensorMap], # W [N, K] E4M3 as bytes, box 128 x 128 + tma_desc_wsf: cutlass.GridConstant[ + cuda.TensorMap + ], # W scale atoms [N / 128 * K / 128, 256] u16 + tma_desc_x: cutlass.GridConstant[cuda.TensorMap], # x [M, K] E4M3 as bytes, box 128 x 8 + tma_desc_xsf: cutlass.GridConstant[cuda.TensorMap], # x scale atoms [K / 128, 256] u16 + y: cutlass.Array, # [M * N] bf16, token-major + x_words: cutlass.Array, # x_bf16: the bf16 activation [M, K] as int32 words + timers: cutlass.Array, # [grid, 8] int64 milestones when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, # ar_push: the all-reduce workspace's multicast mapping (int32 words) + ar_flags: cutlass.Int64, # ar_push: its call flags (parity of the next call = flags[0] & 1) + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[ + int + ], # 1: push the output rows into the next all-reduce; 2: and trigger after it + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], # token columns of the MMA (8, 16 or 48): M <= mma_n + din_i: cutlass.Array, # din: int32 token ids [M] + din_table: cutlass.Array, # din: embedding [V, K / 2] bf16 as int32 words + din_hidden: cutlass.Array, # din: hidden rows [M, K / 2] bf16 as int32 words + din_w: cutlass.Array, # din: [embedding norm | hidden norm] weights [K] bf16 as int32 words + din_f: cutlass.Array, # din: fp32 [eps_e, eps_h] + din_vocab: cutlass.Int32, + din_g: cutlass.Array, # din, flag 1: int64 hidden row of each token, as int32 words + din_res: cutlass.Array, # din, flag 2: the gathered hidden rows out [M, K / 2] bf16 as int32 words + din_flags: cutlass.Int32, # din: 1 = gather the hidden rows, 2 = write them to din_res + rope_cs: cutlass.Array, # rope: fp32 (cos, sin) [positions][64][2] + rope_i: cutlass.Array, # rope: int32 cache lengths [M / T] + rope_bt: cutlass.Array, # rope: int32 K / V page tables [M / T][2][max_blocks] + rope_q: cutlass.Array, # rope: E4M3 Q out [M][hq][128] as uint8 + rope_pool: cutlass.Array, # rope: paged E4M3 cache [pages][hkv][tokens per page][128] as uint8 + rope_scale: cutlass.Array, # rope: fp32 [1] E4M3 quantization scale + rope_dims: cutlass.Array, # rope: int32 [max_blocks, tokens-per-page log2, hq, hkv, T] + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], + x_swiglu: cutlass.Constexpr[ + bool + ], # x_bf16: x_words is [M, 2K] gate | up; the activation is swigluoai(gate, up) + # prod: the producer's addresses and strides (ProducerArgs fields; `packed` is unused), then num_pages and eps. + pr_q: cutlass.Int64, + pr_iq: cutlass.Int64, + pr_ik: cutlass.Int64, + pr_kvd: cutlass.Int64, + pr_kvs: cutlass.Int64, + pr_loc: cutlass.Int64, + pr_quant: cutlass.Int64, + pr_pos: cutlass.Int64, + pr_rope: cutlass.Int64, + pr_qw: cutlass.Int64, + pr_kw: cutlass.Int64, + pr_iqw: cutlass.Int64, + pr_ikw: cutlass.Int64, + pr_ips: cutlass.Int64, + pr_its: cutlass.Int64, + pr_dps: cutlass.Int64, + pr_dls: cutlass.Int64, + pr_dts: cutlass.Int64, + pr_sps: cutlass.Int64, + pr_sls: cutlass.Int64, + pr_pages: cutlass.Int32, + pr_eps: cutlass.Float32, + prod: cutlass.Constexpr[bool], +): + k_tiles = num_k_tiles(k_in) + x_tile_bytes = mma_n * CTA_K + my_tiles = k_tiles // split + rows_per_rank = CTA_M // split + tx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp_id = cute.arch.warp_idx() + rank = cutlass.Int32(0) + if cutlass.const_expr(split > 1): + rank = cute.arch.block_idx_in_cluster() + tma_ptr_w = tma_desc_w.get_ptr() + tma_ptr_wsf = tma_desc_wsf.get_ptr() + tma_ptr_x = tma_desc_x.get_ptr() + tma_ptr_xsf = tma_desc_xsf.get_ptr() + n_tile = bx // cutlass.Int32(split) + m_offset = n_tile * cutlass.Int32(CTA_M) + + smem_a = cutlass.Array( + cutlass.Uint8, ring * W_TILE_BYTES, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_b = cutlass.Array( + cutlass.Uint8, my_tiles * x_tile_bytes, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_sfa = cutlass.Array( + cutlass.Uint8, ring * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + smem_sfb = cutlass.Array( + cutlass.Uint8, my_tiles * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + w_full = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + mma_done = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + act_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + acc_done = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + reduce_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + # Partials of the rows this rank owns: [source rank][row][token] fp32 (its own slot stays unused). Rows are padded + # to mma_n + 4 words: the owner's 16-byte row reads (lane = row) then hit all 32 banks once per 8 lanes; at a + # stride of mma_n they were 8- to 16-way conflicted (4.3 us of the 48-token QKV tail). + mb_stride = mma_n + 4 + mailbox = cutlass.Array( + cutlass.Float32, + split * rows_per_rank * mb_stride, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + # prod: this CTA's bf16 owner rows by token ([8][rows_per_rank]), and the packed-head row of token `rank` gathered + # from the cluster (128 bf16 as 64 words, completed on pr_full by 8 x rows_per_rank * 2 bytes of st.async). + pr_stage = None + pr_row = None + pr_full = None + if cutlass.const_expr(prod): + pr_stage = cutlass.Array( + cutlass.Uint16, 8 * rows_per_rank, space=cutlass.AddressSpace.smem, alignment=16 + ) + pr_row = cutlass.Array(cutlass.Int32, 64, space=cutlass.AddressSpace.smem, alignment=16) + pr_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + # din: per-token RMS factor of the half of x this CTA's k-tiles lie in. + din_rs = None + if cutlass.const_expr(din): + din_rs = cutlass.Array(cutlass.Float32, 8, space=cutlass.AddressSpace.smem, alignment=16) + + if warp_id == 0: + prims.prefetch_tensormap(tma_ptr_w) + prims.prefetch_tensormap(tma_ptr_wsf) + prims.prefetch_tensormap(tma_ptr_x) + prims.prefetch_tensormap(tma_ptr_xsf) + if prims.elect_sync(): + for s in cutlass.range_constexpr(ring): + prims.mbarrier_init(w_full.subview(s), 1) + prims.mbarrier_init(mma_done.subview(s), 1) + prims.mbarrier_init(act_full, 1) + prims.mbarrier_init(acc_done, 1) + if cutlass.const_expr(split > 1 and st_async): + # Completed by the peers' st.async bytes; this CTA's own arrival carries the expected count. + prims.mbarrier_init(reduce_ready, 1) + prims.mbarrier_arrive_expect_tx( + reduce_ready, (split - 1) * rows_per_rank * mma_n * 4 + ) + elif cutlass.const_expr(split > 1): + prims.mbarrier_init(reduce_ready, (split - 1) * rows_per_rank) + if cutlass.const_expr(prod): + # Completed by the 8 ranks' st.async (this one's included) only on a rank that receives a token row. + prims.mbarrier_init(pr_full, 1) + if rank < num_tokens: + prims.mbarrier_arrive_expect_tx(pr_full, split * rows_per_rank * 2) + if warp_id == 2: + prims.tcgen05_alloc(tmem_ptr_i32, tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n)) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + if cutlass.const_expr(split > 1): + prims.barrier_cluster_arrive_relaxed() + prims.barrier_cluster_wait() + prims.barrier_cta_sync(0) + if tx == 0: + _stamp(timers, timed, bx, 0) + + if warp_id == 0: + if prims.elect_sync(): + for i in cutlass.range_constexpr(ring, my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [k_global * cutlass.Int32(CTA_K), m_offset], [] + ) + if cutlass.const_expr((n_out // CTA_M) * split >= 80): + # Wide grids launch in part only after the predecessor (e.g. the fused attention) frees its SMs, which + # loses their pre-wait weight prefetch: the CTAs of the first half also stage their partner's + # (bx + grid / 2) weights into L2. + if bx < cutlass.Int32((n_out // CTA_M) * split // 2): + pb = bx + cutlass.Int32((n_out // CTA_M) * split // 2) + p_rank = pb % cutlass.Int32(split) + p_m_offset = (pb // cutlass.Int32(split)) * cutlass.Int32(CTA_M) + for i in cutlass.range_constexpr(my_tiles): + p_k = _k_tile(p_rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [p_k * cutlass.Int32(CTA_K), p_m_offset], [] + ) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + if cutlass.const_expr(i >= ring): + while not cute.arch.mbarrier_try_wait( + mma_done.subview(stage).data_ptr(), (i // ring - 1) % 2 + ): + pass + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.mbarrier_arrive_expect_tx(w_full.subview(stage), W_TILE_BYTES + SF_ATOM_BYTES) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_a.subview(stage * W_TILE_BYTES), + tma_ptr_w, + (k_global * cutlass.Int32(CTA_K), m_offset), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfa.subview(stage * SF_ATOM_BYTES), + tma_ptr_wsf, + (cutlass.Int32(0), n_tile * cutlass.Int32(k_tiles) + k_global), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + elif warp_id == 1: + prims.griddepcontrol(prims.GridDepAction.WAIT) + if cutlass.const_expr(trigger_early): + # Dependents launch once this grid's own dependency is satisfied (not at entry): consumers that read + # step metadata or caches before their own grid wait rely on every grid ahead of them having passed its + # wait, so the chain of releases never runs ahead of a producer still in flight. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if tx == 32: + _stamp(timers, timed, bx, 1) + if cutlass.const_expr(not x_bf16): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(act_full, my_tiles * (x_tile_bytes + SF_ATOM_BYTES)) + for i in cutlass.range_constexpr(my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_b.subview(i * x_tile_bytes), + tma_ptr_x, + (k_global * cutlass.Int32(CTA_K), cutlass.Int32(0)), + act_full, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfb.subview(i * SF_ATOM_BYTES), + tma_ptr_xsf, + (cutlass.Int32(0), k_global), + act_full, + ) + elif warp_id == 2: + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + ring * SF_COLS + idesc = prims.Tcgen05MxInstrDesc.build( + a_dtype=a_dtype, b_dtype=b_dtype, scale_format=1, n_dim=mma_n, m_dim=CTA_M + ) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + if cutlass.const_expr(a_tmem): + # A staged in TMEM: before the grid wait every weight k-tile and its scale atom go on into TMEM as they + # land (the copies of a stage refilled later are committed to its mma_done); once the activation lands, + # its scale atoms and the MMAs, A read from TMEM. + base_row = tmem_raw >> 16 + base_col = tmem_raw & 0xFFFF + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + my_tiles * SF_COLS + a_base = tmem_raw + cutlass.Int32(a_col0(nacc, my_tiles, mma_n)) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + if prims.elect_sync(): + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_cp( + prims.Tcgen05CpShape.SHAPE_128X256B, + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), + 6, + cutlass.Int32, + ), + desc_a + ((MMA_INST_K >> 4) * kb), + group=prims.CTAGroup.CTA_1, + ) + prims.tcgen05_cp( + s2t_shape, + cutlass.inttoptr((base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32), + desc_sfa, multicast=s2t_multicast, + ) # fmt: skip + if cutlass.const_expr(i + ring < my_tiles): + prims.tcgen05_commit(mma_done.subview(stage)) + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + for i in cutlass.range_constexpr(my_tiles): + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + sfa_tmem = cutlass.inttoptr( + (base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32 + ) + sfb_tmem = cutlass.inttoptr((base_row << 16) | sfb_col0, 6, cutlass.Int32) + prims.tcgen05_cp(s2t_shape, sfb_tmem, desc_sfb, multicast=s2t_multicast) + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), 6, cutlass.Int32 + ), + desc_b + ((MMA_INST_K >> 4) * kb), idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + i > 0 or kb > 0, + sfa_tmem, sfb_tmem, + ) # fmt: skip + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + else: + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + sfa_tmem = (base_row << 16) | (sfa_col0 + stage * SF_COLS) + sfb_tmem = (base_row << 16) | (sfb_col0 + stage * SF_COLS) + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + if prims.elect_sync(): + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), desc_sfa, + multicast=s2t_multicast) # fmt: skip + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), desc_sfb, + multicast=s2t_multicast) # fmt: skip + for kb in cutlass.range_constexpr(K_BLOCKS): + inc = ((MMA_INST_K * 8 // 8) >> 4) * kb + j = i * K_BLOCKS + kb + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw + cutlass.Int32((j % nacc) * mma_n), 6, cutlass.Float32), + desc_a + inc, desc_b + inc, idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), j >= nacc, + cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), + ) # fmt: skip + prims.tcgen05_commit(mma_done.subview(stage)) + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + elif warp_id >= 4: + lane = tx % 32 + row = (warp_id % 4) * 32 + lane + if cutlass.const_expr(x_bf16): + # The bf16 activation of this CTA's k-tiles to MXFP8 in shared memory, as + # torch.ops.trtllm.mxfp8_quantize(x, True) does (UE8M0 = round-up(qz_amax * rcp(448)), + # E4M3 RN-satfinite): one 32-element block (token qz_t, k-tile qz_i, block qz_kb) per thread and round, + # the E4M3 bytes into the 128-byte-swizzled B tile, the qz_scale byte into the tile's qz_scale atom. + qz_tid = tx - cutlass.Int32(128) + prims.griddepcontrol(prims.GridDepAction.WAIT) + din_half_k = k_in // 2 + din_hidden_half = cutlass.Int32(0) + if cutlass.const_expr(din): + # Draft input x = [rmsnorm(table[ids]) * w_e, rmsnorm(hidden) * w_h] (an out-of-range id embeds to + # zero): this CTA's k-tiles lie in one half, so it needs that half's statistic of every token. 16 + # threads per token, 16-byte loads, all of a thread's loads in flight at once. + din_hidden_half = cutlass.select_( + rank * cutlass.Int32(my_tiles * CTA_K) >= cutlass.Int32(din_half_k), + cutlass.Int32(1), + cutlass.Int32(0), + ) + din_tok = qz_tid // cutlass.Int32(16) + din_part = qz_tid % cutlass.Int32(16) + din_tok_ld = cutlass.select_(din_tok < num_tokens, din_tok, cutlass.Int32(0)) + din_id = din_i.load(idx=din_tok_ld) + din_valid = (din_id >= cutlass.Int32(0)) & (din_id < din_vocab) + din_hrow = din_tok_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + din_hrow = din_g.load(idx=din_tok_ld * cutlass.Int32(2)) + din_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + din_hrow, + cutlass.select_(din_valid, din_id, cutlass.Int32(0)), + ) + # The first hidden-half CTA of tile 0 reads whole hidden rows here: it also writes them out. + din_copy = ( + ((din_flags & cutlass.Int32(2)) != cutlass.Int32(0)) + & (bx == cutlass.Int32(split // 2)) + & (din_hidden_half == cutlass.Int32(1)) + & (din_tok < num_tokens) + ) + din_base = din_row * cutlass.Int32(din_half_k // 2) + din_ss = cutlass.Float32(0.0) + din_vecs_per_part = din_half_k // 8 // 16 + for din_round in cutlass.range_constexpr(0, din_vecs_per_part, 16): + din_n = min(16, din_vecs_per_part - din_round) + din_loaded = [] + for din_j in cutlass.range_constexpr(din_n): + din_widx = din_base + ( + din_part + cutlass.Int32((din_round + din_j) * 16) + ) * cutlass.Int32(4) + din_loaded.append( + _din_load4(din_table, din_hidden, din_hidden_half, din_widx) + ) + if din_copy: + for din_j in cutlass.range_constexpr(din_n): + din_v = din_loaded[din_j] + din_res.store( + (din_v[0], din_v[1], din_v[2], din_v[3]), + idx=din_tok * cutlass.Int32(din_half_k // 2) + + (din_part + cutlass.Int32((din_round + din_j) * 16)) + * cutlass.Int32(4), + alignment=16, + ) + for din_j in cutlass.range_constexpr(din_n): + for din_e in cutlass.range_constexpr(4): + din_word = cutlass.Int32(din_loaded[din_j][din_e]) + din_lo = (din_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + din_hi = (din_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + din_ss = din_ss + din_lo * din_lo + din_hi * din_hi + for din_o in cutlass.range_constexpr(4): + din_ss = din_ss + cute.arch.shuffle_sync_bfly(din_ss, offset=1 << din_o) + if din_part == cutlass.Int32(0): + din_eps = din_f.load(idx=din_hidden_half) + din_r = cute.math.rsqrt( + din_ss / cutlass.Float32(din_half_k) + din_eps, fastmath=True + ) + din_keep = din_valid | (din_hidden_half == cutlass.Int32(1)) + din_rs.store( + cutlass.select_(din_keep, din_r, cutlass.Float32(0.0)), idx=din_tok + ) + prims.barrier_cta_sync(3, thread_count=128) + qz_b_words = cutlass.Array(smem_b.data_ptr(0), shape=(my_tiles * x_tile_bytes // 4,), dtype=cutlass.Int32, + alignment=16) # fmt: skip + qz_items = my_tiles * K_BLOCKS * mma_n + qz_rounds = (qz_items + 127) // 128 + # A group's loads first (clamped to a valid row), then its math: one L2 round trip per group of + # QZ_GROUP rounds instead of one per round, without holding every round's data in registers. + QZ_GROUP = qz_rounds if qz_rounds <= 8 else 4 + qz_loaded = [] + for qz_r in cutlass.range_constexpr(qz_rounds): + qz_item = qz_tid + cutlass.Int32(qz_r * 128) + qz_t = qz_item % cutlass.Int32(mma_n) + qz_kb = (qz_item // cutlass.Int32(mma_n)) % cutlass.Int32(K_BLOCKS) + qz_i = qz_item // cutlass.Int32(mma_n * K_BLOCKS) + if cutlass.const_expr(qz_r * 128 + 127 >= qz_items): + qz_i = cutlass.select_( + qz_item < cutlass.Int32(qz_items), qz_i, cutlass.Int32(0) + ) + qz_k_global = rank + qz_i * cutlass.Int32(split) + if cutlass.const_expr(contig): + qz_k_global = rank * cutlass.Int32(my_tiles) + qz_i + qz_t_ld = cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + qz_x_base = ( + qz_t_ld * cutlass.Int32(k_in if x_swiglu else k_in // 2) + + qz_k_global * cutlass.Int32(CTA_K // 2) + + qz_kb * cutlass.Int32(MMA_INST_K // 2) + ) + qz_wvecs = None + if cutlass.const_expr(din): + # The draft input's raw values (table row of the id / hidden row) and its norm weights. + qz_col = qz_k_global * cutlass.Int32(CTA_K) + qz_kb * cutlass.Int32(MMA_INST_K) + qz_hcol = qz_col - din_hidden_half * cutlass.Int32(din_half_k) + qz_id = din_i.load(idx=qz_t_ld) + qz_hrow = qz_t_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + qz_hrow = din_g.load(idx=qz_t_ld * cutlass.Int32(2)) + qz_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + qz_hrow, + cutlass.select_( + (qz_id >= cutlass.Int32(0)) & (qz_id < din_vocab), + qz_id, + cutlass.Int32(0), + ), + ) + qz_src_w = qz_row * cutlass.Int32(din_half_k // 2) + qz_hcol // cutlass.Int32(2) + qz_vecs = [ + _din_load4( + din_table, + din_hidden, + din_hidden_half, + qz_src_w + cutlass.Int32(qz_v * 4), + ) + for qz_v in range(4) + ] + qz_wvecs = [ + din_w.load( + idx=qz_col // cutlass.Int32(2) + cutlass.Int32(qz_v * 4), + vector_size=4, + alignment=16, + ) + for qz_v in range(4) + ] + else: + qz_vecs = [ + x_words.load( + idx=qz_x_base + cutlass.Int32(qz_v * 4), vector_size=4, alignment=16 + ) + for qz_v in range(4) + ] + if cutlass.const_expr(x_swiglu): + # The up half of the same columns, K values after the gate in the row. + qz_wvecs = [ + x_words.load( + idx=qz_x_base + cutlass.Int32(k_in // 2 + qz_v * 4), + vector_size=4, + alignment=16, + ) + for qz_v in range(4) + ] + qz_loaded.append((qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs)) + if cutlass.const_expr((qz_r + 1) % QZ_GROUP == 0 or qz_r + 1 == qz_rounds): + for qz_g in cutlass.range_constexpr(len(qz_loaded)): + qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs = qz_loaded[qz_g] + qz_vals = [] + qz_amax = cutlass.Float32(0.0) + if cutlass.const_expr(din): + qz_rs = din_rs.load( + idx=cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + ) + for qz_v in cutlass.range_constexpr(4): + for qz_e4 in cutlass.range_constexpr(4): + qz_word = cutlass.Int32(qz_vecs[qz_v][qz_e4]) + if cutlass.const_expr(x_swiglu): + qz_word = _swiglu_oai_bf16x2( + qz_word, cutlass.Int32(qz_wvecs[qz_v][qz_e4]) + ) + qz_lo = (qz_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + qz_hi = (qz_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + if cutlass.const_expr(din): + # bf16(x * rs * w), the draft-input kernel's rounding. + qz_wword = cutlass.Int32(qz_wvecs[qz_v][qz_e4]) + qz_lo = _bf16_round( + qz_lo + * qz_rs + * (qz_wword << cutlass.Int32(16)).bitcast(cutlass.Float32) + ) + qz_hi = _bf16_round( + qz_hi + * qz_rs + * (qz_wword & cutlass.Int32(-65536)).bitcast( + cutlass.Float32 + ) + ) + qz_vals.append(qz_lo) + qz_vals.append(qz_hi) + qz_amax = _absmax(_absmax(qz_amax, qz_lo), qz_hi) + qz_bits = (qz_amax * _rcp_approx_ftz(cutlass.Float32(448.0))).bitcast( + cutlass.Int32 + ) + qz_e = ( + (qz_bits >> cutlass.Int32(23)) & cutlass.Int32(0xFF) + ) + cutlass.select_( + (qz_bits & cutlass.Int32(0x7FFFFF)) != cutlass.Int32(0), + cutlass.Int32(1), + cutlass.Int32(0), + ) + qz_e = cutlass.select_(qz_e > cutlass.Int32(254), cutlass.Int32(254), qz_e) + qz_scale = ((cutlass.Int32(254) - qz_e) << cutlass.Int32(23)).bitcast( + cutlass.Float32 + ) + qz_scale = cutlass.select_( + qz_amax != cutlass.Float32(0.0), qz_scale, cutlass.Float32(0.0) + ) + qz_q = [_e4m3x4(qz_vals[4 * qz_j] * qz_scale, qz_vals[4 * qz_j + 1] * qz_scale, + qz_vals[4 * qz_j + 2] * qz_scale, qz_vals[4 * qz_j + 3] * qz_scale) + for qz_j in range(8)] # fmt: skip + if qz_item < cutlass.Int32(qz_items): + if qz_t < num_tokens: + qz_row_w = qz_i * cutlass.Int32( + x_tile_bytes // 4 + ) + qz_t * cutlass.Int32(CTA_K // 4) + qz_c0 = qz_kb * cutlass.Int32(2) + qz_b_words.store( + (qz_q[0], qz_q[1], qz_q[2], qz_q[3]), + idx=qz_row_w + + (qz_c0 ^ (qz_t & cutlass.Int32(7))) * cutlass.Int32(4), + alignment=16, + ) + qz_b_words.store( + (qz_q[4], qz_q[5], qz_q[6], qz_q[7]), + idx=qz_row_w + + ((qz_c0 + cutlass.Int32(1)) ^ (qz_t & cutlass.Int32(7))) + * cutlass.Int32(4), + alignment=16, + ) + smem_sfb.store( + cutlass.Uint8(qz_e), + idx=qz_i * cutlass.Int32(SF_ATOM_BYTES) + + (qz_t % cutlass.Int32(32)) * cutlass.Int32(16) + + (qz_t // cutlass.Int32(32)) * cutlass.Int32(4) + + qz_kb, + ) + qz_loaded = [] + # Generic-proxy stores made visible to the MMA's async-proxy reads, then one arrival for all. + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.barrier_cta_sync(2, thread_count=128) + if qz_tid == cutlass.Int32(0): + prims.mbarrier_arrive(act_full) + if cutlass.const_expr(st_async): + while not cute.arch.mbarrier_test_wait(acc_done.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(acc_done.data_ptr(), 0): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + # tcgen05.ld widths are powers of two: the accumulator columns in descending power-of-two pieces. + acc_all = [] + ld_col = 0 + for ld_w in (128, 64, 32, 16, 8): + if cutlass.const_expr((mma_n * nacc - ld_col) >= ld_w): + piece = prims.tcgen05_ld( + "32x32b", + cutlass.inttoptr( + tmem_ptr_i32.load() + cutlass.Int32(ld_col), 6, cutlass.Float32 + ), + num=ld_w, + ) + acc_all = acc_all + [piece[c] for c in range(ld_w)] + ld_col += ld_w + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + acc = [] + for t in cutlass.range_constexpr(mma_n): + v = cutlass.Float32(acc_all[t]) + for a in cutlass.range_constexpr(1, nacc): + v = v + cutlass.Float32(acc_all[a * mma_n + t]) + acc.append(v) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if tx == 128: + _stamp(timers, timed, bx, 4) + n = m_offset + row + if cutlass.const_expr(split == 1): + for t in cutlass.range_constexpr(mma_n): + if cutlass.Int32(t) < num_tokens: + y.store( + cutlass.BFloat16(cutlass.Float32(acc[t])), idx=cutlass.Int32(t * n_out) + n + ) + if cutlass.const_expr(ar_push > 0): + prims.griddepcontrol(prims.GridDepAction.WAIT) + # The flags are rotated by the previous all-reduce after its dependent release; this load is ordered by + # the wait above, which waits for ALL in-flight upstream grids, not only the immediate predecessor (PTX + # griddepcontrol; MoE test_pdl_transitive_wait.py). Keep the wait before this load. + # The parity of each (token, column slice) slot: the all-reduce's CTA t * 8 + slice flag. All loads are + # issued before the first store so they overlap (one L2 round trip, not one per token). + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_v = cutlass.Float32(acc[t]) + ar_hi = cute.arch.shuffle_sync_bfly(ar_v, offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), _bf16x2_word(ar_v, ar_hi) + ) + else: + owner = row // cutlass.Int32(rows_per_rank) + local = row % cutlass.Int32(rows_per_rank) + if cutlass.const_expr(rope): + # Rank r owns head dims [r * P, (r + 1) * P) and their rotation partners 64 + ..., P = rows / 2. + pairs = rows_per_rank // 2 + owner = (row % cutlass.Int32(64)) // cutlass.Int32(pairs) + local = (row // cutlass.Int32(64)) * cutlass.Int32(pairs) + ( + row % cutlass.Int32(64) + ) % cutlass.Int32(pairs) + if owner != rank: + base = (rank * cutlass.Int32(rows_per_rank) + local) * cutlass.Int32(mb_stride) + if cutlass.const_expr(st_async): + dst = _mapa_u32(mailbox.subview(base).data_ptr(), owner) + mb = _mapa_u32(reduce_ready.data_ptr(), owner) + words = [cutlass.Float32(acc[t]).bitcast(cutlass.Int32) for t in range(mma_n)] + for g in cutlass.range_constexpr(mma_n // 4): + _st_async_v4( + dst + cutlass.Int32(16 * g), + words[4 * g], + words[4 * g + 1], + words[4 * g + 2], + words[4 * g + 3], + mb, + ) + else: + for t in cutlass.range_constexpr(mma_n): + prims.mapa(mailbox.subview(base + cutlass.Int32(t)), owner).store( + cutlass.Float32(acc[t]) + ) + prims.mbarrier_arrive( + prims.mapa(reduce_ready, owner), scope=prims.MemScope.CLUSTER + ) + else: + if cutlass.const_expr(st_async): + while not _test_wait_cluster(reduce_ready.data_ptr(), 0): + pass + else: + while not prims.mbarrier_try_wait_parity( + reduce_ready, 0, scope=prims.MBarrierScope.CLUSTER + ): + pass + if lane == 0: + if row // cutlass.Int32(rows_per_rank) == rank: + if (row % cutlass.Int32(rows_per_rank)) == 0: + _stamp(timers, timed, bx, 5) + totals = [] + for g in cutlass.range_constexpr(mma_n // 4): + sums = [cutlass.Float32(0.0)] * 4 + for q in cutlass.range_constexpr(split): + parts = _ld_shared_v4_f32( + mailbox.subview( + (cutlass.Int32(q * rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g) + ).data_ptr() + ) + sums = [sums[j] + cutlass.select_(rank == cutlass.Int32(q), cutlass.Float32(acc[4 * g + j]), + parts[j]) for j in range(4)] # fmt: skip + for j in cutlass.range_constexpr(4): + totals.append(sums[j]) + if cutlass.const_expr(rope): + # The own-rank mailbox slot is unused: it carries the totals to the RoPE stage. + mailbox.store( + sums[j], + idx=(rank * cutlass.Int32(rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g + j), + ) + elif cutlass.const_expr(prod): + # The bf16 the packed row would hold (RN, as the y store below), staged by token. + if cutlass.Int32(4 * g + j) < num_tokens: + pr_stage.store( + cutlass.Uint16( + (_bf16_bits(sums[j]) >> cutlass.Int32(16)) + & cutlass.Int32(0xFFFF) + ), + idx=cutlass.Int32((4 * g + j) * rows_per_rank) + local, + ) + else: + if cutlass.Int32(4 * g + j) < num_tokens: + y.store( + cutlass.BFloat16(sums[j]), + idx=cutlass.Int32((4 * g + j) * n_out) + n, + ) + if cutlass.const_expr(ar_push > 0): + # Every lane of an owner warp is an owner (rows_per_rank >= 32), so lane pairs hold row pairs. + prims.griddepcontrol(prims.GridDepAction.WAIT) + # The flags are rotated by the previous all-reduce after its dependent release; this load is ordered + # by the wait above, which waits for ALL in-flight upstream grids, not only the immediate + # predecessor (PTX griddepcontrol; MoE test_pdl_transitive_wait.py). Keep the wait before this load. + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_hi = cute.arch.shuffle_sync_bfly(totals[t], offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), + _bf16x2_word(totals[t], ar_hi), + ) + prims.barrier_cta_sync(1, thread_count=128) + if cutlass.const_expr(prod): + # Transpose: thread e < 2 T sends half e % 2 (16 bytes = 8 dims) of token e / 2's staged rows to cluster + # rank e / 2, at dims [rank * rows_per_rank + 8 (e % 2), ...) of that rank's packed-head row. + pr_e = tx - cutlass.Int32(128) + if pr_e < num_tokens * cutlass.Int32(2): + pr_t = pr_e // cutlass.Int32(2) + pr_h = pr_e % cutlass.Int32(2) + pr_w = _ld_shared_v4_b32( + pr_stage.subview( + pr_t * cutlass.Int32(rows_per_rank) + pr_h * cutlass.Int32(8) + ).data_ptr() + ) + _st_async_v4( + _mapa_u32( + pr_row.subview( + rank * cutlass.Int32(rows_per_rank // 2) + pr_h * cutlass.Int32(4) + ).data_ptr(), + pr_t, + ), + pr_w[0], + pr_w[1], + pr_w[2], + pr_w[3], + _mapa_u32(pr_full.data_ptr(), pr_t), + ) + if warp_id == 4: + if rank < num_tokens: + # The producer reads step inputs (positions, slots) and writes the caches: ordered after the + # upstream grids by this thread's own grid wait (already satisfied: the activation landed). + prims.griddepcontrol(prims.GridDepAction.WAIT) + while not _test_wait_cluster(pr_full.data_ptr(), 0): + pass + pr_lo = pr_row.load(idx=lane * cutlass.Int32(2)) + pr_hi = pr_row.load(idx=lane * cutlass.Int32(2) + cutlass.Int32(1)) + mask = cutlass.Int32(0xFFFF) + producer_head( + rank, + n_tile, + lane, + bf16_bits_to_f32(pr_lo & mask), + bf16_bits_to_f32((pr_lo >> cutlass.Int32(16)) & mask), + bf16_bits_to_f32(pr_hi & mask), + bf16_bits_to_f32((pr_hi >> cutlass.Int32(16)) & mask), + ProducerArgs( + packed=cutlass.Int64(0), + q_out=pr_q, + index_q_out=pr_iq, + index_k=pr_ik, + kv_data=pr_kvd, + kv_scale=pr_kvs, + out_cache_loc=pr_loc, + kv_quant_scale=pr_quant, + position_ids=pr_pos, + rotary_cos_sin=pr_rope, + q_weight=pr_qw, + k_weight=pr_kw, + index_q_weight=pr_iqw, + index_k_weight=pr_ikw, + eps=pr_eps, + num_pages=pr_pages, + index_page_stride=pr_ips, + index_token_stride=pr_its, + data_page_stride=pr_dps, + data_plane_stride=pr_dls, + data_token_stride=pr_dts, + scale_page_stride=pr_sps, + scale_plane_stride=pr_sls, + ), + ) + if cutlass.const_expr(rope): + # Token t of request t / T sits at cache position kv_len - T + t % T. Q / K heads: GPT-NeoX RoPE over + # the 128-wide head, bf16(cos * x -/+ sin * pair) on the bf16 projection; then E4M3 of x * scale, Q to + # rope_q, K / V into the page of that position. + pairs = rows_per_rank // 2 + rp_e = tx - cutlass.Int32(128) + rp_p = rp_e % cutlass.Int32(pairs) + rp_t = rp_e // cutlass.Int32(pairs) + if rp_t < num_tokens: + rp_max_blocks = rope_dims.load(idx=0) + rp_tpb_log2 = rope_dims.load(idx=1) + rp_hq = rope_dims.load(idx=2) + rp_hkv = rope_dims.load(idx=3) + rp_T = rope_dims.load(idx=4) + rp_d = rank * cutlass.Int32(pairs) + rp_p + rp_a = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + rp_p) * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_b = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + cutlass.Int32(pairs) + rp_p) + * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_req = rp_t // rp_T + rp_tok = rope_i.load(idx=rp_req) - rp_T + (rp_t - rp_req * rp_T) + rp_head = n_tile + if rp_head < rp_hq + rp_hkv: + rp_c = rope_cs.load(idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2)) + rp_s = rope_cs.load( + idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2) + + cutlass.Int32(1) + ) + rp_ra = _rope_bits(rp_c, rp_a, -rp_s, rp_b).bitcast(cutlass.Float32) + rp_rb = _rope_bits(rp_c, rp_b, rp_s, rp_a).bitcast(cutlass.Float32) + rp_a = rp_ra + rp_b = rp_rb + rp_qs = rope_scale.load(idx=0) + rp_w = _e4m3x4( + rp_a * rp_qs, rp_b * rp_qs, cutlass.Float32(0.0), cutlass.Float32(0.0) + ) + rp_lo = cutlass.Uint8(rp_w & cutlass.Int32(0xFF)) + rp_hi = cutlass.Uint8((rp_w >> cutlass.Int32(8)) & cutlass.Int32(0xFF)) + if rp_head < rp_hq: + rp_q = (rp_t * rp_hq + rp_head) * cutlass.Int32(128) + rp_d + rope_q.store(rp_lo, idx=rp_q) + rope_q.store(rp_hi, idx=rp_q + cutlass.Int32(64)) + else: + rp_kv = cutlass.select_( + rp_head < rp_hq + rp_hkv, cutlass.Int32(0), cutlass.Int32(1) + ) + rp_kh = rp_head - rp_hq - rp_kv * rp_hkv + rp_page = rope_bt.load( + idx=(rp_req * cutlass.Int32(2) + rp_kv) * rp_max_blocks + + (rp_tok >> rp_tpb_log2) + ) + # 64-bit byte offset: the pool view can exceed 2^31 bytes. + rp_dst = ( + (cutlass.Int64(rp_page) * cutlass.Int64(rp_hkv) + cutlass.Int64(rp_kh)) + << cutlass.Int64(rp_tpb_log2 + cutlass.Int32(7)) + ) + cutlass.Int64( + (rp_tok & ((cutlass.Int32(1) << rp_tpb_log2) - cutlass.Int32(1))) + * cutlass.Int32(128) + + rp_d + ) + rp_addr = rope_pool.data_ptr(0).toint() + rp_dst + cutlass.inttoptr(rp_addr, mem_space=1, dtype=cutlass.Uint8).store(rp_lo) + cutlass.inttoptr( + rp_addr + cutlass.Int64(64), mem_space=1, dtype=cutlass.Uint8 + ).store(rp_hi) + if tx == 128: + _stamp(timers, timed, bx, 6) + if cutlass.const_expr(ar_push == 2): + # The all-reduce polls the pushed words; it launches after this CTA's rows are out, not at grid end. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if warp_id == 4: + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Int32), + tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n), + ) + + +def _bytes_map(t, cols, rows, box_rows, swizzle): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[cols, rows], + global_strides=[cols // 16], + box_dims=[CTA_K, box_rows], + swizzle=swizzle, + ) + + +def _atom_map(t, atoms): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint16, + global_dims=[SF_ATOM_U16, atoms], + global_strides=[SF_ATOM_BYTES // 16], + box_dims=[SF_ATOM_U16, 1], + swizzle=cuda.TensorMapSwizzle.none, + ) + + +@cute.jit +def m3_mx_gemv( + w: cute.Tensor, # [N, K] E4M3 as uint8, K contiguous + wsf: cute.Tensor, # weight scales, 128x4-swizzled, as uint8 + x: cute.Tensor, # [M, K] E4M3 as uint8, M <= 8 + xsf: cute.Tensor, # activation scales, 128x4-swizzled (atom k at byte 512 k), as uint8 + y: cute.Tensor, # [M * N] bf16 + x_words: cute.Tensor, # x_bf16: bf16 x [M, K] as int32 words (x / xsf are then unused placeholders) + timers: cute.Tensor, # [grid * 8] int64, written when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[int], + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, + din_i: cute.Tensor, # din / rope inputs: see m3_mx_gemv_kernel (placeholders when off) + din_table: cute.Tensor, + din_hidden: cute.Tensor, + din_w: cute.Tensor, + din_f: cute.Tensor, + din_vocab: cutlass.Int32, + din_g: cute.Tensor, + din_res: cute.Tensor, + din_flags: cutlass.Int32, + rope_cs: cute.Tensor, + rope_i: cute.Tensor, + rope_bt: cute.Tensor, + rope_q: cute.Tensor, + rope_pool: cute.Tensor, + rope_scale: cute.Tensor, + rope_dims: cute.Tensor, + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], + x_swiglu: cutlass.Constexpr[bool], + # prod: see m3_mx_gemv_kernel (zeros when off). + pr_q: cutlass.Int64, + pr_iq: cutlass.Int64, + pr_ik: cutlass.Int64, + pr_kvd: cutlass.Int64, + pr_kvs: cutlass.Int64, + pr_loc: cutlass.Int64, + pr_quant: cutlass.Int64, + pr_pos: cutlass.Int64, + pr_rope: cutlass.Int64, + pr_qw: cutlass.Int64, + pr_kw: cutlass.Int64, + pr_iqw: cutlass.Int64, + pr_ikw: cutlass.Int64, + pr_ips: cutlass.Int64, + pr_its: cutlass.Int64, + pr_dps: cutlass.Int64, + pr_dls: cutlass.Int64, + pr_dts: cutlass.Int64, + pr_sps: cutlass.Int64, + pr_sls: cutlass.Int64, + pr_pages: cutlass.Int32, + pr_eps: cutlass.Float32, + prod: cutlass.Constexpr[bool], +) -> None: + # x_swiglu's up half rides the slot din uses for its norm weights, and it activates the bf16 x. + assert not (din and x_swiglu) and (x_bf16 or not x_swiglu), ( + "x_swiglu needs x_bf16 and excludes din" + ) + k_tiles = num_k_tiles(k_in) + tma_desc_w = _bytes_map(w, k_in, n_out, CTA_M, cuda.TensorMapSwizzle.s128b) + tma_desc_wsf = _atom_map(wsf, (n_out // CTA_M) * k_tiles) + tma_desc_x = _bytes_map(x, k_in, num_tokens, mma_n, cuda.TensorMapSwizzle.s128b) + tma_desc_xsf = _atom_map(xsf, k_tiles) + m3_mx_gemv_kernel( + tma_desc_w, tma_desc_wsf, tma_desc_x, tma_desc_xsf, y, x_words, timers, num_tokens, n_out, k_in, split, ring, + trigger_early, contig, timed, nacc, a_tmem, st_async, x_bf16, ar_ws, ar_flags, ar_rank, ar_push, ar_world, + ar_maxt, mma_n, + din_i, din_table, din_hidden, din_w, din_f, din_vocab, din_g, din_res, din_flags, rope_cs, + rope_i, rope_bt, rope_q, rope_pool, rope_scale, rope_dims, din, rope, x_swiglu, + pr_q, pr_iq, pr_ik, pr_kvd, pr_kvs, pr_loc, pr_quant, pr_pos, pr_rope, pr_qw, pr_kw, pr_iqw, pr_ikw, pr_ips, + pr_its, pr_dps, pr_dls, pr_dts, pr_sps, pr_sls, pr_pages, pr_eps, prod, + ).launch( + grid=((n_out // CTA_M) * split, 1, 1), + block=(THREADS, 1, 1), + cluster=(split, 1, 1), + stream=stream, + use_pdl=use_pdl, + ) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/op.py new file mode 100644 index 000000000000..882fc18b3431 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/mx_gemv/op.py @@ -0,0 +1,649 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) at the TP8 per-rank projection shapes. + +Compiled on the first call per (N, K, split, ring, early trigger, PDL, push, MMA width), which must happen outside +CUDA-graph capture, or ahead of any call by ``precompile``; once ``freeze`` closes compiles, a call whose kernel is not +compiled declines instead. +""" + +from __future__ import annotations + +import threading +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +MMA_WIDTHS = ( + 8, + 16, + 32, + 48, +) # token columns of the compiled MMA; a call uses the smallest that holds M +_SMEM_LIMIT = 220 * 1024 # capacity: fixed: the device's shared-memory limit + + +def _mma_n(m: int) -> int: + return next(w for w in MMA_WIDTHS if m <= w) + + +def _smem_bytes(ring: int, tiles: int, split: int, mma_n: int) -> int: + return ( + ring * (128 * 128 + 512) + + tiles * (mma_n * 128 + 512) + + split * (128 // split) * (mma_n + 4) * 4 + + 1024 + ) + + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + +# (N, K) -> (split, ring[, contig, nacc, a_tmem, st_async]) of the TP8 per-rank projections: GB300 exposed time +# behind a PDL predecessor. The optional fields default to _CONTIG, _NACC, _A_TMEM, _ST_ASYNC. The sparse-layer QKV +# splits K over 8 CTAs (96 CTAs at 1 token). The split count sets the fp32 association of the K partials, so splits +# 8, 4, 2 and 1 are NOT bitwise equal in general (wide-range block scales show it): an output +# row's bits depend on the split its row count selects (batch invariance holds within a split band only, see +# test_modeling_v2_m3_mx_gemv_split_bands.py). +_CONFIGS: Dict[Tuple[int, int], tuple] = { + # sparse-layer QKV + index projection + (1536, 6144): (8, 8), + # dense-layer QKV + (1280, 6144): (4, 12), + # o_proj + (6144, 1024): (4, 8), + # dense-layer gate_up + (3072, 6144): (4, 12), + # dense-layer down + (6144, 1536): (2, 12), +} +# Per-token-count configs (N, K, M): at 2 tokens the sparse-layer QKV splits K over 4 CTAs, A operand from shared +# memory. +_CONFIGS_BY_M: Dict[Tuple[int, int, int], tuple] = { + (1536, 6144, 2): (4, 12, 1, 1, 0, 1), +} +# Above 8 tokens (spec-decode verify rows): the same 4-CTA split for the sparse-layer QKV. +_CONFIGS_WIDE: Dict[Tuple[int, int], tuple] = { + (1536, 6144): (4, 12, 1, 1, 0, 1), +} + +# Defaults of the optional config fields: contiguous k-tile ranges per split rank, one TMEM accumulator, the A +# operand staged in TMEM (when its columns fit), st.async split-K partial exchange. +_CONTIG = True +_NACC = 1 +_A_TMEM = True +_ST_ASYNC = True +_ensured = set() + + +_DUMMY: Dict[torch.device, torch.Tensor] = {} + + +def _dummy_timers(device) -> torch.Tensor: + """Placeholder for the kernel's timer buffer, which it only writes when compiled with timers on.""" + t = _DUMMY.get(device) + if t is None: + t = _DUMMY[device] = torch.zeros(8, dtype=torch.int64, device=device) + return t + + +def _arg(t: torch.Tensor, align: int = 16): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align).mark_layout_dynamic(leading_dim=t.dim() - 1) + + +# #117: the producer epilogue serves the sparse-layer QKV + index projection, 12 packed heads of 128 (8 q, k, v, +# index-q, index-k), one head per split-8 cluster. +PROD_N = 12 * 128 # capacity: fixed: the producer's packed heads +# The kernel's producer arguments on a call without the producer epilogue (20 addresses / strides, pages, eps). +PROD_OFF = (0,) * 21 + (0.0,) + + +def get_config(n: int, k: int, m: Optional[int] = None) -> Optional[tuple]: + cfg = _CONFIGS_BY_M.get((n, k, m)) if m is not None else None + if cfg is None and m is not None and m > 8: + cfg = _CONFIGS_WIDE.get((n, k)) + cfg = cfg or _CONFIGS.get((n, k)) + # The split-K epilogue gives each of the split CTAs 128 // split rows of a 128-row output tile. + if cfg is None or n % 128 != 0 or k % 128 != 0 or (k // 128) % cfg[0] != 0 or 128 % cfg[0] != 0: + return None + return cfg + + +def _declines(n: int, k: int, m: int, x_bf16: bool, cfg: tuple) -> bool: + """Whether a (split, ring, ...) config declines an [M, K] x [N, K]^T call on shape grounds.""" + split = cfg[0] + if not 0 < m <= MAX_TOKENS or 128 % split != 0 or (k // 128) % split != 0: + return True + tiles = k // 128 // split + mma_n = _mma_n(m) + # 17..32 rows at K < 1024 decline (an unsplit K below 1024 is slower than the FlashInfer GEMM there); no TP8 + # projection has K < 1024. + # The in-kernel bf16 -> MXFP8 quantization (x_bf16) costs one 32-element block per thread and round on 128 threads; + # beyond 8 rounds per CTA, mxfp8_quantize + this GEMV (or the fallback GEMM) is faster. + if (16 < m <= 32 and k < 1024) or (x_bf16 and tiles * 4 * mma_n > 8 * 128): + return True + return _smem_bytes(min(cfg[1], tiles, 2), tiles, split, mma_n) > _SMEM_LIMIT + + +def serves(n: int, k: int, m: int, x_bf16: bool) -> bool: + """Whether the GEMV with its default config takes an [M, K] activation (bf16 or MXFP8) against an [N, K] weight.""" + cfg = get_config(n, k, m) + return cfg is not None and not _declines(n, k, m, x_bf16, cfg) + + +def _resolve( + m: int, + n: int, + k: int, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple], + x_swiglu: bool = False, + prod: bool = False, +) -> Optional[tuple]: + """The compile key of an [M, K] x [N, K]^T call and its runtime push words, or None when the call is declined. + + ``ar`` is the push tuple of ``_run``; only its (world, bucket tokens, mode) enter the key, the addresses and the + rank are runtime arguments.""" + cfg = cfg or get_config(n, k, m) + if cfg is None or _declines(n, k, m, x_bf16, cfg): + return None + split, ring = cfg[0], cfg[1] + contig = bool(cfg[2]) if len(cfg) > 2 else _CONTIG + nacc = int(cfg[3]) if len(cfg) > 3 else _NACC + a_tmem = bool(cfg[4]) if len(cfg) > 4 else _A_TMEM + st_async = bool(cfg[5]) if len(cfg) > 5 else _ST_ASYNC + if prod and not ( + n == PROD_N + and split == 8 + and m <= split + and st_async + and not x_bf16 + and not x_swiglu + and ar is None + ): + # The producer epilogue: the sparse QKV + index projection (one 128-row packed head per cluster of 8), at most + # one token per cluster rank, the st.async exchange, MXFP8 input, no all-reduce push. + return None + ring = min(ring, k // 128 // split) + tiles = k // 128 // split + mma_n = _mma_n(m) + while ring > 2 and _smem_bytes(ring, tiles, split, mma_n) > _SMEM_LIMIT: + ring -= 1 + if a_tmem: + from .m3_mx_gemv_kernel import tmem_cols + + if tmem_cols(ring, nacc, True, tiles, mma_n) > 512: + a_tmem = False + # ar = (multicast workspace address, flags address, rank, world, bucket tokens, mode): push the output rows into + # the next m3_mc_allreduce call's exchange slots (mode 2 also defers the dependents' launch until after the push). + ar_ws, ar_flags, ar_rank, ar_mode, ar_world, ar_maxt = 0, 0, 0, 0, 1, 8 + if ar is not None and split <= 4 and n % 2 == 0: + ar_ws, ar_flags, ar_rank, ar_world, ar_maxt, ar_mode = ar + if ar_mode == 2: + trigger_early = False + if use_pdl is None: + use_pdl = get_env_enable_pdl() + # The kernel's ``timed`` flag (per-CTA milestone timers) is always off. + key = ( + n, + k, + split, + ring, + trigger_early, + contig, + False, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + x_swiglu, + prod, + ) + return key, (ar_ws, ar_flags, ar_rank) + + +def _compile( + key: tuple, args: tuple, m: int, runtime_ar: tuple, stream, extra: tuple, prod_args: tuple +): + """cute.compile the kernel of ``key`` (see ``_resolve``) on these argument views; cached.""" + import cutlass.cute as cute + + from . import m3_mx_gemv_kernel as kernel + + ( + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + x_swiglu, + prod, + ) = key + ar_ws, ar_flags, ar_rank = runtime_ar + with _lock: + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute.compile( + kernel.m3_mx_gemv, + *args, + m, + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_ws, + ar_flags, + ar_rank, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + stream, + *extra, + False, + False, + x_swiglu, + *prod_args, + prod, + ) + return fn + + +def _views( + x: torch.Tensor, + xsf: Optional[torch.Tensor], + weight, + weight_scale, + y, + x_bf16: bool, + x_swiglu: bool = False, +) -> tuple: + # x_swiglu: x is [M, 2K] (gate | up); the GEMV's K is half its width. + k = x.shape[1] // 2 if x_swiglu else x.shape[1] + if x_bf16: + x_bytes = x.view(torch.uint8) + xsf = _placeholder_scales(k, x.device) + x_words = x.view(torch.int32) + else: + x_bytes = x.view(torch.uint8) + x_words = _placeholder_words(x.device) + return ( + _arg(weight.view(torch.uint8)), + _arg(weight_scale.view(torch.uint8).reshape(-1)), + _arg(x_bytes), + _arg(xsf.view(torch.uint8).reshape(-1)), + _arg(y.view(-1) if y.numel() else _placeholder_words(x.device).view(torch.bfloat16)), + _arg(x_words.reshape(-1)), + _arg(_dummy_timers(x.device)), + ) + + +def _run( + x: torch.Tensor, + xsf: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple] = None, + x_swiglu: bool = False, + prod_args: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + global last_pushed, last_uncompiled + last_pushed = False + last_uncompiled = None + import cuda.bindings.driver as cuda_driver + + m, k = x.shape + if x_swiglu: + k //= 2 + n = weight.shape[0] + if weight.shape[1] != k: + return None + prod = prod_args is not None + resolved = _resolve(m, n, k, x_bf16, trigger_early, cfg, use_pdl, ar, x_swiglu, prod) + if resolved is None: + return None + key, runtime_ar = resolved + if not prod: + prod_args = PROD_OFF + fn = _compiled.get(key) + if fn is None: + if _frozen[0]: + # Compiles are closed (the model built every kernel it dispatches to before warmup): decline, the caller + # reports the miss. + last_uncompiled = key + return None + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_mx_gemv must run once per shape outside CUDA-graph capture first" + ) + # prod: no packed row is stored (the producer epilogue consumes it); y is the placeholder. + y = torch.empty(0 if prod else m, n, dtype=torch.bfloat16, device=x.device) + args = _views(x, xsf, weight, weight_scale, y, x_bf16, x_swiglu) + extra = _extra_args(x.device) + stream = cuda_driver.CUstream(torch.cuda.current_stream(x.device).cuda_stream) + if fn is None: + fn = _compile(key, args, m, runtime_ar, stream, extra, prod_args) + ar_ws, ar_flags, ar_rank = runtime_ar + fn(*args, m, ar_ws, ar_flags, ar_rank, stream, *extra, *prod_args) + last_pushed = key[11] > 0 + return y + + +def precompile( + weight: torch.Tensor, + weight_scale: torch.Tensor, + m: int, + x_bf16: bool, + use_pdl: Optional[bool] = None, + ar: Optional[Tuple[int, int, int]] = None, + x_swiglu: bool = False, + prod: bool = False, + trigger_early: bool = True, +) -> Optional[tuple]: + """Compile, without launching, the kernel an M-row call against ``weight`` takes; returns its key, or None when + such a call is declined. ``trigger_early`` is part of the key (the early PDL release is a compile-time choice). + ``ar`` = (world, bucket tokens, mode) of a pushing call (its addresses and rank are + runtime arguments, so none is needed here). Runs outside capture, even while compiles are frozen.""" + import cuda.bindings.driver as cuda_driver + + n, k = weight.shape + push = None if ar is None else (0, 0, 0, *ar) + # prod: the producer epilogue needs the split-8 cluster; the TP8 table's default (N, K) config is split 8 at every + # row count. The plain GEMV takes split 4 at 2 rows, which is not bitwise split 8 (see _CONFIGS). + resolved = _resolve( + m, + n, + k, + x_bf16, + trigger_early, + get_config(n, k) if prod else None, + use_pdl, + push, + x_swiglu, + prod, + ) + if resolved is None: + return None + key, runtime_ar = resolved + if key in _compiled: + return key + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_mx_gemv.precompile allocates and compiles: call it outside capture") + dev = weight.device + x = torch.zeros( + m, + 2 * k if x_swiglu else k, + dtype=torch.bfloat16 if x_bf16 else torch.float8_e4m3fn, + device=dev, + ) + xsf = None if x_bf16 else torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=dev) + y = torch.empty(m, n, dtype=torch.bfloat16, device=dev) + args = _views(x, xsf, weight, weight_scale, y, x_bf16, x_swiglu) + stream = cuda_driver.CUstream(torch.cuda.current_stream(dev).cuda_stream) + _compile(key, args, m, runtime_ar, stream, _extra_args(dev), PROD_OFF) + return key + + +def freeze(frozen: bool = True) -> None: + """Close (or reopen) compiles: while frozen, a call whose kernel is not compiled yet declines (returns None and + sets ``last_uncompiled``) instead of compiling. ``precompile`` and ``ensure_compiled`` still compile.""" + _frozen[0] = bool(frozen) + + +def frozen() -> bool: + return _frozen[0] + + +def is_compiled(key: tuple) -> bool: + return key in _compiled + + +def _extra_args(device) -> tuple: + """Placeholder arguments of the kernel's draft-input prologue and RoPE / KV epilogue (compiled off).""" + i32 = _placeholder_words(device) + f32 = i32.view(torch.float32) + u8 = i32.view(torch.uint8) + d = (_arg(i32), _arg(i32), _arg(i32), _arg(i32), _arg(f32), 0, _arg(i32), _arg(i32), 0) + r = (_arg(f32), _arg(i32), _arg(i32), _arg(u8), _arg(u8), _arg(f32), _arg(i32)) + return d + r + + +def m3_mx_gemv( + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """``xq`` E4M3 [M <= 8, K] with ``xsf`` its 128x4-swizzled UE8M0 scales; ``weight`` E4M3 [N, K] with + ``weight_scale`` swizzled. Returns bf16 [M, N], or None when the shape is not served.""" + return _run(xq, xsf, weight, weight_scale, False, trigger_early, cfg, use_pdl, ar) + + +def m3_mx_gemv_bf16( + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """The same GEMV for a bf16 ``x`` [M <= 8, K] (contiguous), quantized to MXFP8 inside the kernel exactly as + torch.ops.trtllm.mxfp8_quantize(x, True) does.""" + if x.dtype != torch.bfloat16 or x.dim() != 2 or not x.is_contiguous() or x.data_ptr() % 16 != 0: + return None + return _run(x, None, weight, weight_scale, True, trigger_early, cfg, use_pdl, ar) + + +def m3_mx_gemv_swiglu( + gate_up: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """``m3_mx_gemv_bf16(swiglu(gate_up), ...)`` in one launch: ``gate_up`` bf16 [M <= 8, 2K] (contiguous, gate then + up) is activated in the kernel's quantize prologue with the swigluoai arithmetic of the Triton silu_and_mul kernel + (alpha 1.702, limit 7, the (up + 1) offset; bit for bit), then quantized as the bf16 path does.""" + if ( + gate_up.dtype != torch.bfloat16 + or gate_up.dim() != 2 + or not gate_up.is_contiguous() + or gate_up.data_ptr() % 16 != 0 + or gate_up.shape[1] % 2 != 0 + ): + return None + return _run(gate_up, None, weight, weight_scale, True, trigger_early, cfg, use_pdl, ar, True) + + +def m3_mx_gemv_producer( + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + q: torch.Tensor, + index_q: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + trigger_early: bool = True, + use_pdl: Optional[bool] = None, +) -> bool: + """#117: ``m3_producer(m3_mx_gemv(xq, xsf, weight, weight_scale), ...)`` in one launch, bit for bit: the QKV + + index GEMV of the sparse layer (``weight`` [12 x 128, K]) whose owner epilogue hands each token's packed head to + the producer phase (``m3_layer.phase_producer.producer_head``). Writes ``q`` FP8 [T, 8, 128] and ``index_q`` FP8 + [T, 1, 128] and the token's cache slot (NVFP4 K / V + scales, FP8 index-K) exactly as the producer does; the + caches and tensors are those ``minimax_m3_producer_pw`` takes (kv_data [pages, 2, 1, 128, 64] u8, kv_scale [pages, + 2, 1, 128, 8] u8, index_k [pages, 1, 128, 128] e4m3). Returns False (nothing launched) when the call is declined. + """ + m = xq.shape[0] + if ( + m > 8 + or weight.shape[0] != PROD_N + or q.shape != (m, 8, 128) + or index_q.shape != (m, 1, 128) + or not (q.is_contiguous() and index_q.is_contiguous()) + or kv_data.dim() != 5 + or kv_scale.dim() != 5 + or index_k.dim() != 4 + ): + return False + tensors = ( + q, + index_q, + index_k, + kv_data, + kv_scale, + out_cache_loc, + kv_quant_scale, + position_ids, + rotary_cos_sin, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + ) + prod_args = ( + *(t.data_ptr() for t in tensors), + index_k.stride(0), + index_k.stride(2), + kv_data.stride(0), + kv_data.stride(1), + kv_data.stride(3), + kv_scale.stride(0), + kv_scale.stride(1), + kv_data.shape[0], + float(eps), + ) + out = _run( + xq, + xsf, + weight, + weight_scale, + False, + trigger_early, + get_config(weight.shape[0], weight.shape[1]), # split 8 at every row count (see precompile) + use_pdl, + None, + False, + prod_args, + ) + return out is not None + + +_PLACEHOLDERS: Dict[tuple, torch.Tensor] = {} +# Whether the last call pushed its output into an all-reduce (see _run's ``ar``). +last_pushed = False +# The key of the last call declined because compiles are frozen and its kernel was not compiled (see ``freeze``). +last_uncompiled: Optional[tuple] = None +_frozen = [False] + + +def _placeholder_scales(k: int, device) -> torch.Tensor: + key = ("sf", k, device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(k // 128 * 512, dtype=torch.uint8, device=device) + return t + + +def _placeholder_words(device) -> torch.Tensor: + key = ("w", device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(16, dtype=torch.int32, device=device) + return t + + +def ensure_compiled(weight: torch.Tensor, weight_scale: torch.Tensor, use_pdl: bool) -> None: + """Compiles the kernel of this weight's shape outside capture (a one-token dummy call).""" + if ( + weight.dim() != 2 + or weight.dtype != torch.float8_e4m3fn + or get_config(*weight.shape) is None + ): + return + n, k = weight.shape + if (n, k, use_pdl) in _ensured: + return + _ensured.add((n, k, use_pdl)) + xq = torch.zeros(1, k, dtype=torch.float8_e4m3fn, device=weight.device) + xsf = torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=weight.device) + m3_mx_gemv(xq, xsf, weight, weight_scale, use_pdl=use_pdl) + for width in MMA_WIDTHS[1:]: + m3_mx_gemv( + torch.zeros(width, k, dtype=torch.float8_e4m3fn, device=weight.device), + xsf, + weight, + weight_scale, + use_pdl=use_pdl, + ) + m3_mx_gemv_bf16( + torch.zeros(1, k, dtype=torch.bfloat16, device=weight.device), + weight, + weight_scale, + use_pdl=use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/norm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/norm/__init__.py new file mode 100644 index 000000000000..acec8fdbb26f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/norm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Normalization entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/norm/m3_embedding_firstnorm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/norm/m3_embedding_firstnorm.py new file mode 100644 index 000000000000..1842171bde1a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/norm/m3_embedding_firstnorm.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-step embedding gather and the first layer's input RMSNorm in one CuTe DSL launch, into caller buffers.""" + +from __future__ import annotations + +from typing import Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..embedding_firstnorm.op import DecodeEmbedding, m3_embed_norm + +__all__ = ["DecodeEmbedding", "m3_embedding_firstnorm"] + +VOCAB = 200064 # MiniMax-M3's vocabulary; the decode table is the full (replicated) embedding + + +class M3EmbeddingFirstnorm(OpWrapper): + """Write ``table[ids]`` into ``raw`` (bf16 [M, 6144], a zero row for an id outside ``[0, V)``) and + ``raw * rsqrt(mean(raw^2) + eps) * weight`` into ``out``; returns ``(out, raw)`` -- the first layer's normed input + and its residual. ``ids`` int32 or int64 [M], M = 1..8; ``table`` bf16 [V, 6144] contiguous; ``weight`` bf16 + [6144]. The first call per (M, id dtype, table shape, PDL) compiles (``DecodeEmbedding`` does it at build).""" + + ARCHS = frozenset({Arch.SM_103}) + + # The target builds every (rows 1..8, id dtype) owner; the kernel is one CTA per row, so the rows differ only in + # grid size: the ends of the range and both id dtypes. + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, int32 ids", spec=dict(rows=1, id_dtype=torch.int32)), + Cell(why="eight requests, int32 ids", spec=dict(rows=8, id_dtype=torch.int32)), + Cell(why="one request, int64 ids", spec=dict(rows=1, id_dtype=torch.int64)), + Cell( + why="six rows (one request verifying a draft), int64 ids", + spec=dict(rows=6, id_dtype=torch.int64), + ), + # Every other row count the decode dispatch takes (dispatch.EMBED_FIRSTNORM), both id dtypes. + *( + Cell( + why=f"{rows} rows: a decode row count between the ones above", + spec=dict(rows=rows, id_dtype=torch.int32 if rows % 2 else torch.int64), + ) + for rows in (2, 3, 4, 5, 7) + ), + ) + + note = """ + The gathered rows are exact. The norm is flashinfer's RMSNorm arithmetic in its reduction order with a fast + rsqrt, where `reference` sums in fp64 and takes torch's rsqrt: within the band the test derives, not bit for bit. + An id outside [0, V) gives a zero row and a zero norm (torch's embedding would raise). No accepted input is known + to compute wrongly: the host side rejects any other dtype, shape or a strided buffer. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + return m3_embed_norm(ids, table, weight, eps, raw, out, use_pdl) + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """New tensors (``raw`` / ``out`` are not written); `use_pdl` is ignored.""" + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] * valid.unsqueeze(1).to( + table.dtype + ) + x = rows.double() + rstd = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) + return (x * rstd * weight.double()).to(torch.bfloat16), rows + + +m3_embedding_firstnorm = M3EmbeddingFirstnorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/__init__.py new file mode 100644 index 000000000000..b3611a4b7866 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampling entries this target alone calls: the spec sampler's top-p renorm override (ST-01 TOPP_DET), its softmax +override (ST-01-SM SOFTMAX_MC) and its chain speculative sampling override (ST-01-REJ REJ_MC).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_chain_spec_mc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_chain_spec_mc.py new file mode 100644 index 000000000000..a0b26091aced --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_chain_spec_mc.py @@ -0,0 +1,271 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ST-01-REJ REJ_MC: the spec sampler's chain speculative sampling (rejection) on many CTAs per request, exact vs +``torch.ops.trtllm.chain_speculative_sampling`` (``trtllm::minimax_m31_chain_speculative_sampling_mc``, +kernels/minimaxM31/chainSpecSamplingMcSm103.cu). + +The target installs ``chain_spec_mc`` as the rejection sampler's override +(``speculative/interface.set_chain_speculative_sampling_override``); it returns None, and the sampler keeps the stock +op, whenever the op is not built or the input is outside the kernel's contract. + +Workspace: one zero-initialized byte buffer per device, ``MAX_BATCH`` requests of +``minimax_m31_chain_speculative_sampling_mc_ws_bytes()``, kept for the life of the process (CUDA graphs captured with +it keep its address). The kernel leaves it ready for the next launch, so it is zeroed once. ``prebuild`` allocates it +outside any CUDA-graph capture (the target calls it from ``cache_derived_state``, before the executor's warmup and +captures); a call that finds no workspace allocates one when the stream is not capturing, and inside a capture returns +None (the stock op for that call) rather than allocating from the graph's private pool. Calls that share it must be +stream-ordered (the sampler's are). +""" + +from __future__ import annotations + +import os +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) + +# The largest CTA count per request (the grid is batch x ctas_per_row 1024-thread CTAs, all co-resident). REJ v2 MB +#: one request 32.3 us at 16 (v1) -> 24.8 us at 64 (v2), 25.8 at 32, 25.5 at 128; bitwise at every count. +MAX_CTAS_PER_ROW = 64 + +# The requests the workspace holds and the override takes: the TP8 / TP4 K7 calls up to c8. Microbenchmarks +# gated it bitwise at batch 1 / 2 / 4 / 8 and measured 31.2 vs 41.2 us per call at batch 1; larger calls keep the stock +# kernel. +MAX_BATCH = 8 + +# The widest vocab the kernel takes (64 tiles of 4096; kChainSpecMcMaxVocab). +MAX_VOCAB = 64 * 4096 +MAX_DRAFT = ( + 32 # capacity: fixed: the kernel's shared acceptance probabilities (kChainSpecMcMaxDraft) +) + +# The op when it is built (or overlaid as tensorrt_llm/libs/libm31_chain_spec_mc.so), decided once per process. +_STATE: dict = {} +# Per device index: the persistent workspace and the SM count. +_WORKSPACES: dict = {} +_SMS: dict = {} + + +def chain_spec_mc_op(): + """``trtllm::minimax_m31_chain_speculative_sampling_mc`` if this build carries it (in-tree, or the overlay + library), else None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_chain_speculative_sampling_mc", None) + if op is None: + import tensorrt_llm + + lib = os.path.join( + os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_chain_spec_mc.so" + ) + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_chain_speculative_sampling_mc", None) + _STATE["op"] = op + _STATE["ws_bytes"] = ( + int(torch.ops.trtllm.minimax_m31_chain_speculative_sampling_mc_ws_bytes()) + if op is not None + else 0 + ) + return _STATE["op"] + + +def ws_bytes_per_row() -> int: + """Workspace bytes per request (0 when the op is not built).""" + chain_spec_mc_op() + return _STATE["ws_bytes"] + + +def sm_count(device: torch.device) -> int: + """The device's SM count, cached per device.""" + idx = torch.device(device).index + idx = torch.cuda.current_device() if idx is None else idx + if idx not in _SMS: + _SMS[idx] = torch.cuda.get_device_properties(idx).multi_processor_count + return _SMS[idx] + + +def ctas_per_row(batch: int, sms: int) -> int: + """The largest power of two <= MAX_CTAS_PER_ROW with ctas x batch <= sms (1 past that: one CTA per request needs + no co-residency).""" + g = MAX_CTAS_PER_ROW + while g > 1 and g * batch > sms: + g //= 2 + return g + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's zeroed workspace for MAX_BATCH requests; must run outside a CUDA-graph capture. + Returns it, or None when the op is not built or the stream is capturing.""" + if chain_spec_mc_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.zeros( + MAX_BATCH * ws_bytes_per_row(), dtype=torch.uint8, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31ChainSpecMc(OpWrapper): + """``trtllm::chain_speculative_sampling(draft_probs, draft_token_ids, target_probs, seed, offset)`` bit for bit + (accepted tokens [batch, D + 1] and token counts [batch]) in ONE launch of batch x ctas_per_row CTAs (stock: one + CTA per request). ``workspace`` is the persistent zeroed buffer of ``prebuild``. ``__call__`` returns None (the + caller keeps the stock op) for a non-fp32, non-contiguous or misaligned probs, a vocab not divisible by 4 or above + MAX_VOCAB, mismatched shapes, non-int32 draft tokens, non-int64 or mis-sized seed / offset, inputs on different + devices, more requests than the workspace holds, a batch x ctas_per_row above the SM count, or when the op is not + built.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The K7 calls were checked bitwise on GB300 over 5 cases x 64 seeds at ctas_per_row 4 / 8 / 16. + CELLS: tuple[Cell, ...] = ( + Cell(why="K7 at c1 (vocab 200064, 16 CTAs)", spec=dict(batch=1, k=7, v=200064)), + Cell(why="K7 at c2 (vocab 200064)", spec=dict(batch=2, k=7, v=200064)), + Cell(why="K7 at c4 (vocab 200064)", spec=dict(batch=4, k=7, v=200064)), + Cell(why="K7 at c8 (vocab 200064, 128 CTAs)", spec=dict(batch=8, k=7, v=200064)), + ) + + note = """ + Exact by construction: every CTA of a request runs the stock Philox acceptance loop, the (target, residual) mass is + the stock 1024 per-thread chains into the same cub BlockReduce, each tile sum is the stock per-tile + cub reduce, and one CTA runs the stock tile-order prefix and in-tile cub BlockScan search. Default flags, as the + stock kernel. + """ + + def __call__( + self, + draft_probs: torch.Tensor, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, + workspace: Optional[torch.Tensor], + ctas_per_row: int, + ) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + if ( + workspace is None + or not all( + isinstance(t, torch.Tensor) + for t in (draft_probs, draft_token_ids, target_probs, seed, offset) + ) + or not draft_probs.is_cuda + or draft_probs.dtype != torch.float32 + or target_probs.dtype != torch.float32 + or draft_token_ids.dtype != torch.int32 + or seed.dtype != torch.int64 + or offset.dtype != torch.int64 + or draft_probs.dim() != 3 + or target_probs.dim() != 3 + or draft_token_ids.dim() != 2 + ): + return None + batch, k, v = draft_probs.shape + if ( + batch < 1 + or tuple(draft_token_ids.shape) != (batch, k) + or tuple(target_probs.shape) != (batch, k + 1, v) + or seed.numel() != batch + or offset.numel() != batch + or v % 4 != 0 + or v > MAX_VOCAB + or k > MAX_DRAFT + or any( + not t.is_contiguous() or t.device != draft_probs.device + for t in (draft_token_ids, target_probs, seed, offset, workspace) + ) + or not draft_probs.is_contiguous() + or draft_probs.data_ptr() % 16 != 0 + or target_probs.data_ptr() % 16 != 0 + or ctas_per_row < 1 + or 1024 % ctas_per_row != 0 + or (ctas_per_row > 1 and batch * ctas_per_row > sm_count(draft_probs.device)) + ): + return None + op = chain_spec_mc_op() + if op is None or workspace.numel() * workspace.element_size() < batch * ws_bytes_per_row(): + return None + tokens, num_tokens = op( + draft_probs, draft_token_ids, target_probs, seed, offset, workspace, int(ctas_per_row) + ) + return tokens, num_tokens + + def reference( + self, + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace=None, + ctas_per_row=None, + ) -> tuple[torch.Tensor, torch.Tensor]: + """The stock op (workspace and ctas_per_row are ignored).""" + return torch.ops.trtllm.chain_speculative_sampling( + draft_probs, draft_token_ids, target_probs, seed, offset + ) + + def is_valid( + self, + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace=None, + ctas_per_row=None, + **kwargs, + ) -> None: + assert draft_probs.dtype == torch.float32 and draft_probs.dim() == 3, ( + "draft_probs: fp32 [batch, D, vocab]" + ) + + +m31_chain_spec_mc = M31ChainSpecMc() + + +def chain_spec_mc( + draft_probs: torch.Tensor, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, +) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + """The rejection sampler's override: the stock op's (accepted tokens, token counts), or None (the sampler runs the + stock op) when the op is not built, the batch exceeds MAX_BATCH, no workspace can be had (first use inside a + capture) or the input is outside the contract.""" + if ( + not isinstance(draft_probs, torch.Tensor) + or not draft_probs.is_cuda + or draft_probs.dim() != 3 + ): + return None + batch = draft_probs.shape[0] + if batch < 1 or batch > MAX_BATCH or chain_spec_mc_op() is None: + return None + workspace = prebuild(draft_probs.device) + if workspace is None: + return None + return m31_chain_spec_mc( + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace, + ctas_per_row(batch, sm_count(draft_probs.device)), + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_softmax_mc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_softmax_mc.py new file mode 100644 index 000000000000..ac4d6dc8add2 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_softmax_mc.py @@ -0,0 +1,199 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ST-01-SM SOFTMAX_MC: the spec sampler's softmax with its write pass on many CTAs per row, exact vs +``flashinfer.sampling.softmax(logits, temperature)`` (``trtllm::minimax_m31_softmax_mc``, +kernels/minimaxM31/softmaxMcSm103.cu). + +The target installs ``softmax_mc`` as the sampler's softmax override (``sampler/ops/flashinfer.set_softmax_override``); +it returns None, and the sampler keeps the stock FlashInfer call, whenever the op is not built or the input is outside +the kernel's contract (FlashInfer's Path A shapes: rows <= 128 and vocab >= 24576; here also rows <= ``MAX_ROWS``). + +Workspace: one partials buffer per device (``MAX_ROWS`` x ceil(``MAX_VOCAB`` / 8192) float2, 8 KB), kept for the life +of the process (CUDA graphs captured with it keep its address). Every launch overwrites what it reads, so it needs no +initialization. ``prebuild`` allocates it outside any CUDA-graph capture (the target calls it from +``cache_derived_state``, before the executor's warmup and captures); a call that finds no workspace allocates one when +the stream is not capturing, and inside a capture returns None (stock FlashInfer for that call) rather than allocating +from the graph's private pool. Calls that share it must be stream-ordered (the sampler's are). +""" + +from __future__ import annotations + +import os +from typing import Optional, Union + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) + +# The write pass's CTAs per row (microbenchmark: 9.2 vs FlashInfer 21.8 us per call at 7-8 rows; bitwise at +# 4 / 8 / 16). +WRITE_CTAS = 16 + +# The rows the override takes: the TP8 / TP4 K7 c1 calls (the draft's 7 rows and the verify's 8), where the MB measured +# it. Larger calls keep the stock kernel. +MAX_ROWS = 8 + +# FlashInfer's Path A bounds (OnlineSoftmax: batch <= 128 and d >= 24576); outside them FlashInfer takes its +# single-block Path B, which this kernel does not reproduce. +PATH_A_MAX_ROWS = 128 +PATH_A_MIN_VOCAB = 24576 +# FlashInfer's Path A slice (DEFAULT_SLICE_SIZE) and the widest vocab the workspace holds. +SLICE = 8192 +MAX_VOCAB = 1 << 20 + +# The op when it is built (or overlaid as tensorrt_llm/libs/libm31_softmax_mc.so), decided once per process. +_STATE: dict = {} +# Per device index: the persistent partials workspace. +_WORKSPACES: dict = {} + + +def softmax_mc_op(): + """``trtllm::minimax_m31_softmax_mc`` if this build carries it (in-tree, or the overlay library), else None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_softmax_mc", None) + if op is None: + import tensorrt_llm + + lib = os.path.join( + os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_softmax_mc.so" + ) + if os.path.exists(lib): + torch.ops.load_library(lib) + _m31_fake_ops.register() + op = getattr(torch.ops.trtllm, "minimax_m31_softmax_mc", None) + _STATE["op"] = op + return _STATE["op"] + + +def partials_numel(rows: int, d: int) -> int: + """fp32 elements of the partials workspace for rows x d (one float2 per row and 8192-wide slice).""" + return 2 * rows * ((d + SLICE - 1) // SLICE) + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's partials workspace for MAX_ROWS rows of up to MAX_VOCAB; must run outside a + CUDA-graph capture. Returns it, or None when the op is not built or the stream is capturing.""" + if softmax_mc_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.empty( + partials_numel(MAX_ROWS, MAX_VOCAB), dtype=torch.float32, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31SoftmaxMc(OpWrapper): + """FlashInfer's ``softmax(logits, temperature)`` of fp32 logits [rows, d], bit for bit, on FlashInfer's Path A + shapes (rows <= 128, d >= 24576): FlashInfer's Map kernel, then its Reduce kernel's prologue in each of + ``write_ctas`` CTAs per row, each writing its share of the row (FlashInfer: one CTA writes the whole row). + ``temperature`` is None (1.0), a Python number (one temperature) or an fp32 [rows] tensor (one per row), as + FlashInfer takes it. ``partials`` is the persistent workspace of ``prebuild``. ``__call__`` returns None (the caller + keeps the stock call) for a non-fp32, non-contiguous or misaligned input, a width not divisible by 4, a shape + outside Path A or the workspace, a temperature tensor that is not a contiguous fp32 [rows] tensor on logits' + device, or when the op is not built.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The c1 K7 calls were checked bitwise on GB300. + CELLS: tuple[Cell, ...] = ( + Cell(why="the K7 draft's 7 rows at c1 (vocab 200064)", spec=dict(rows=7, d=200064)), + Cell(why="the K7 verify's 8 rows at c1 (vocab 200064)", spec=dict(rows=8, d=200064)), + Cell(why="one row (the smallest grid)", spec=dict(rows=1, d=200064)), + ) + + note = """ + Exact by construction: the Map kernel is FlashInfer's, every write CTA reduces the same partials with the same cub + BlockReduce and Float2SoftmaxReduceOp as FlashInfer's single Reduce CTA, and the per-element write + keeps FlashInfer's source form. Built with --use_fast_math, as FlashInfer's JIT builds it. + """ + + def __call__( + self, + logits: torch.Tensor, + temperature: Union[torch.Tensor, float, None], + partials: Optional[torch.Tensor], + write_ctas: int, + ) -> Optional[torch.Tensor]: + if ( + partials is None + or not isinstance(logits, torch.Tensor) + or not logits.is_cuda + or logits.dtype != torch.float32 + or logits.dim() != 2 + or not logits.is_contiguous() + or logits.shape[0] < 1 + or logits.shape[0] > PATH_A_MAX_ROWS + or logits.shape[1] < PATH_A_MIN_VOCAB + or logits.shape[1] % 4 != 0 + or logits.data_ptr() % 16 != 0 + or partials.device != logits.device + or partials.numel() * partials.element_size() + < 4 * partials_numel(logits.shape[0], logits.shape[1]) + or write_ctas < 1 + or write_ctas > 65535 + ): + return None + if temperature is None: + temp_arr, temp_val = None, 1.0 + elif isinstance(temperature, torch.Tensor): + if ( + temperature.dtype != torch.float32 + or temperature.dim() != 1 + or temperature.shape[0] != logits.shape[0] + or not temperature.is_contiguous() + or temperature.device != logits.device + ): + return None + temp_arr, temp_val = temperature, 0.0 + elif isinstance(temperature, (int, float)) and not isinstance(temperature, bool): + temp_arr, temp_val = None, float(temperature) + else: + return None + op = softmax_mc_op() + if op is None: + return None + out = torch.empty_like(logits) + op(logits, temp_arr, temp_val, partials, out, int(write_ctas)) + return out + + def reference(self, logits, temperature, partials=None, write_ctas=None) -> torch.Tensor: + """FlashInfer's softmax (partials and write_ctas are ignored).""" + import flashinfer.sampling + + return flashinfer.sampling.softmax(logits, temperature) + + def is_valid(self, logits, temperature, partials=None, write_ctas=None, **kwargs) -> None: + assert logits.dtype == torch.float32 and logits.dim() == 2, "logits: fp32 [rows, d]" + + +m31_softmax_mc = M31SoftmaxMc() + + +def softmax_mc( + logits: torch.Tensor, temperature: Union[torch.Tensor, float, None] +) -> Optional[torch.Tensor]: + """The sampler's softmax override: FlashInfer's softmax of logits [rows, d] at temperature, or None (the sampler + runs stock FlashInfer) when the op is not built, rows exceed MAX_ROWS, d exceeds MAX_VOCAB, no workspace can be had + (first use inside a capture) or the input is outside the contract.""" + if not isinstance(logits, torch.Tensor) or not logits.is_cuda or logits.dim() != 2: + return None + rows = logits.shape[0] + if rows < 1 or rows > MAX_ROWS or logits.shape[1] > MAX_VOCAB or softmax_mc_op() is None: + return None + partials = prebuild(logits.device) + if partials is None: + return None + return m31_softmax_mc(logits, temperature, partials, WRITE_CTAS) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_topp_det.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_topp_det.py new file mode 100644 index 000000000000..0d28b1e2699c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/sampling/m31_topp_det.py @@ -0,0 +1,248 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ST-01 TOPP_DET: the spec sampler's top-p renorm as one persistent C++ launch, exact vs FlashInfer's deterministic +``top_p_renorm_probs`` (``trtllm::minimax_m31_topp_renorm_det``, kernels/minimaxM31/toppRenormDetSm103.cu). + +TOPP_DET v3 (``trtllm::minimax_m31_topp_renorm_det_v3``, kernels/minimaxM31/toppRenormDetV3Sm103.cu) is the same +renorm, bitwise v1.2's output, with the redundant per-CTA selection and a programmatic dependent launch. It is taken +when it is built (in-tree, or overlaid as tensorrt_llm/libs/libm31_topp_det_v3.so) unless +``TRTLLM_M31_DISABLE_TOPPDET_V3`` is set; v1.2 otherwise. ``TRTLLM_M31_DISABLE_TOPPDET_T3`` keeps v3 on its global +radix passes (no candidate list). + +The target installs ``top_p_renorm_det`` as the sampler's top-p renorm override +(``sampler/ops/flashinfer.set_top_p_renorm_override``); it returns None, and the sampler keeps the stock FlashInfer +call, whenever the op is not built or the input is outside the kernel's contract. + +Workspace: one zero-initialized byte buffer per device, ``MAX_ROWS`` rows of +``minimax_m31_topp_renorm_det_ws_bytes()``, kept for the life of the process (CUDA graphs captured with it keep its +address). The kernel leaves it ready for the next launch, so it is zeroed once. ``prebuild`` allocates it outside any +CUDA-graph capture (the target calls it from ``cache_derived_state``, before the executor's warmup and captures); a +call that finds no workspace allocates one when the stream is not capturing, and inside a capture returns None +(stock FlashInfer for that call) rather than allocating from the graph's private pool. +""" + +from __future__ import annotations + +import hashlib +import os +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops as _m31_fake_ops, +) +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled + +# The largest CTA count per row (the persistent grid is rows x ctas_per_row 1024-thread CTAs, all co-resident). +MAX_CTAS_PER_ROW = 16 + +# The rows the workspace holds and the override takes: the TP8 / TP4 K7 c1 calls (the draft's 7 rows and the +# verify's 8), where the microbenchmarks measured it faster than stock FlashInfer (v1.2: 54.3 vs 104.2 us per +# call at 7 peaked rows, 54.8 vs 105.1 at 8). At 56 / 64 rows v1 was slower than FlashInfer's default mode (340 vs +# 187 / 199 us) and larger shapes are not re-measured, so larger calls keep the stock kernel. +MAX_ROWS = 8 + +# The op when it is built (or overlaid as tensorrt_llm/libs/libm31_topp_det.so), decided once per process. +_STATE: dict = {} +# Per device index: the persistent workspace and the SM count. +_WORKSPACES: dict = {} +_SMS: dict = {} + + +def _load_op(name: str, lib_name: str): + """``torch.ops.trtllm.`` if registered (in-tree), else after loading tensorrt_llm/libs/ when it + exists, else None. Returns (op, the library path loaded or None).""" + op = getattr(torch.ops.trtllm, name, None) + if op is not None: + return op, None + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", lib_name) + if not os.path.exists(lib): + return None, None + torch.ops.load_library(lib) + _m31_fake_ops.register() + return getattr(torch.ops.trtllm, name, None), lib + + +def topp_det_op(): + """The renorm op this process uses, decided once: ``trtllm::minimax_m31_topp_renorm_det_v3`` when it is built + and TOPPDET_V3 is not disabled, else ``trtllm::minimax_m31_topp_renorm_det`` when built, else None.""" + if "op" not in _STATE: + op, ws, v3 = None, 0, False + if not m31_disabled("TOPPDET_V3"): + op, lib = _load_op("minimax_m31_topp_renorm_det_v3", "libm31_topp_det_v3.so") + if op is not None: + v3 = True + ws = int(torch.ops.trtllm.minimax_m31_topp_renorm_det_v3_ws_bytes()) + tag = "in-tree" + if lib is not None: + with open(lib, "rb") as f: + digest = hashlib.md5(f.read(), usedforsecurity=False).hexdigest()[:8] + tag = f"libm31_topp_det_v3.so md5 {digest}" + logger.info(f"[M3.1] TOPP_DET v3 private op engaged ({tag})") + if op is None: + op, _ = _load_op("minimax_m31_topp_renorm_det", "libm31_topp_det.so") + ws = ( + int(torch.ops.trtllm.minimax_m31_topp_renorm_det_ws_bytes()) + if op is not None + else 0 + ) + _STATE["op"], _STATE["ws_bytes"], _STATE["v3"] = op, ws, v3 + # v3's candidate-list radix passes (bitwise the same output) unless TOPPDET_T3 is disabled. + _STATE["cand_list"] = v3 and not m31_disabled("TOPPDET_T3") + if v3: + logger.info( + "[M3.1] TOPP_DET v3 T3 candidate-list fused passes: " + f"{'on' if _STATE['cand_list'] else 'off'} (cap 4096)" + ) + return _STATE["op"] + + +def is_v3() -> bool: + """Whether the engaged op is TOPP_DET v3.""" + topp_det_op() + return _STATE["v3"] + + +def ws_bytes_per_row() -> int: + """Workspace bytes per row (0 when the op is not built).""" + topp_det_op() + return _STATE["ws_bytes"] + + +def sm_count(device: torch.device) -> int: + """The device's SM count, cached per device.""" + idx = torch.device(device).index + idx = torch.cuda.current_device() if idx is None else idx + if idx not in _SMS: + _SMS[idx] = torch.cuda.get_device_properties(idx).multi_processor_count + return _SMS[idx] + + +def ctas_per_row(rows: int, sms: int) -> int: + """The largest power of two <= MAX_CTAS_PER_ROW with ctas x rows <= sms (1 past that: one CTA per row needs no + co-residency).""" + g = MAX_CTAS_PER_ROW + while g > 1 and g * rows > sms: + g //= 2 + return g + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's zeroed workspace for MAX_ROWS rows; must run outside a CUDA-graph capture. + Returns it, or None when the op is not built or the stream is capturing.""" + if topp_det_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.zeros( + MAX_ROWS * ws_bytes_per_row(), dtype=torch.uint8, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31ToppDet(OpWrapper): + """FlashInfer's ``top_p_renorm_probs(probs, top_p, is_deterministic=True)`` of fp32 probs [rows, d] with fp32 + per-row top_p [rows], bit for bit on every row with top_p < 1, in ONE launch of rows x ctas_per_row CTAs + (FlashInfer's: five launches, apply on one CTA per row, slow on high-entropy rows). Rows with top_p >= 1 keep every + element (FlashInfer drops tail buckets there in ~43% of rows, in both of its modes); their output is + ``ftz(ftz(p) * norm)``. ``workspace`` is the persistent zeroed buffer of ``prebuild``. ``__call__`` returns None + (the caller keeps the stock call) for a non-fp32, non-contiguous or misaligned input, a width not divisible by 4, + a top_p that is not a contiguous fp32 [rows] tensor on probs' device, more rows than the workspace holds, or when + the op is not built.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The c1 K7 calls were checked bitwise on GB300 over 8 row classes x 4 top_p modes x 16 seeds. + CELLS: tuple[Cell, ...] = ( + Cell(why="the K7 draft's 7 rows at c1 (vocab 200064)", spec=dict(rows=7, d=200064)), + Cell(why="the K7 verify's 8 rows at c1 (vocab 200064)", spec=dict(rows=8, d=200064)), + Cell(why="one row (the smallest grid, 16 CTAs)", spec=dict(rows=1, d=200064)), + ) + + note = """ + Exact by construction: the det histograms are integer (any aggregation order is FlashInfer's), the per-pass + selection is FlashInfer's last-block code on them, and the kept sum is FlashInfer's 1024 per-thread chains into + the same cub BlockReduce. Built with --use_fast_math, as FlashInfer's JIT builds it. + """ + + def __call__( + self, + probs: torch.Tensor, + top_p: torch.Tensor, + workspace: Optional[torch.Tensor], + ctas_per_row: int, + ) -> Optional[torch.Tensor]: + if ( + not isinstance(top_p, torch.Tensor) + or workspace is None + or not probs.is_cuda + or probs.dtype != torch.float32 + or probs.dim() != 2 + or not probs.is_contiguous() + or probs.shape[0] < 1 + or probs.shape[1] % 4 != 0 + or probs.data_ptr() % 16 != 0 + or top_p.dtype != torch.float32 + or top_p.dim() != 1 + or top_p.shape[0] != probs.shape[0] + or not top_p.is_contiguous() + or top_p.device != probs.device + or workspace.device != probs.device + or ctas_per_row < 1 + or 1024 % ctas_per_row != 0 + ): + return None + op = topp_det_op() + if ( + op is None + or workspace.numel() * workspace.element_size() < probs.shape[0] * ws_bytes_per_row() + ): + return None + out = torch.empty_like(probs) + if is_v3(): + op(probs, top_p, workspace, out, int(ctas_per_row), True, _STATE["cand_list"]) + else: + op(probs, top_p, workspace, out, int(ctas_per_row)) + return out + + def reference(self, probs, top_p, workspace=None, ctas_per_row=None) -> torch.Tensor: + """FlashInfer's deterministic renorm (rows with top_p < 1; workspace and ctas_per_row are ignored).""" + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + def is_valid(self, probs, top_p, workspace=None, ctas_per_row=None, **kwargs) -> None: + assert probs.dtype == torch.float32 and probs.dim() == 2, "probs: fp32 [rows, d]" + + +m31_topp_det = M31ToppDet() + + +def top_p_renorm_det(probs: torch.Tensor, top_p: torch.Tensor) -> Optional[torch.Tensor]: + """The sampler's top-p renorm override: the det renorm of probs [rows, d] at per-row top_p, or None (the sampler + runs stock FlashInfer) when the op is not built, rows exceed MAX_ROWS, no workspace can be had (first use inside + a capture) or the input is outside the contract.""" + if not isinstance(top_p, torch.Tensor) or not probs.is_cuda or probs.dim() != 2: + return None + rows = probs.shape[0] + if rows < 1 or rows > MAX_ROWS or topp_det_op() is None: + return None + workspace = prebuild(probs.device) + if workspace is None: + return None + return m31_topp_det(probs, top_p, workspace, ctas_per_row(rows, sm_count(probs.device))) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/__init__.py new file mode 100644 index 000000000000..d25c41dc9557 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Speculative-decoding entries this target alone calls: the Eagle3 verify ops, the capture / fc_norm RMSNorms and +the step-input staging.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_capture_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_capture_norm.py new file mode 100644 index 000000000000..131269ae7d07 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_capture_norm.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Eagle3 hidden-state capture with the drafter's fc_norm folded in (``m3_spec_tail.capture_norm``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_chunk_norm import rmsnorm_rows + + +class M3CaptureNorm(OpWrapper): + """``dst [M, H] = rmsnorm(bf16(h + r)) * w`` (``r`` optional), the rounding chunk_norm applies, into a capture + buffer slice whose row stride may exceed H. Returns ``dst``.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The target's captured layers (hidden 6144) into the 3-layer capture buffer (row stride 3 x 6144), over the + # verify rows of draft_len 3 (4 .. 32 rows); the residual is present for every captured decoder layer. + CELLS: tuple[Cell, ...] = ( + Cell( + why="4 verify rows of one request, layer slot 0, residual added", + spec=dict(m=4, slot=0, residual=True), + ), + Cell( + why="cap 8 (32 rows) into the last layer slot", spec=dict(m=32, slot=2, residual=True) + ), + Cell( + why="no residual (a capture point without one): plain RMSNorm", + spec=dict(m=8, slot=1, residual=False), + ), + ) + + note = """ + fp32 statistic (block sum, rsqrtf), one bf16 rounding after the bf16 sum h + r; the band states the occasional + one-ulp move against torch's statistic. + """ + + def __call__( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_capture_norm(h, r, w, eps, dst) + return dst + + def reference( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + """bf16 sum, then the normalized row; ``dst`` only says where the op writes.""" + x = (h.float() + r.float()).to(torch.bfloat16) if r is not None else h + return rmsnorm_rows(x, w, eps) + + def is_valid( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> None: + # The norm weight is read as H dense bf16 values; a strided view (a column of a stacked weight) is read + # past its elements without an error. + assert w.is_contiguous(), "w must be dense [H]" + + +m3_capture_norm = M3CaptureNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_chunk_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_chunk_norm.py new file mode 100644 index 000000000000..f6adb2207a20 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_chunk_norm.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 fc_norm over the captured layers in one launch: per-chunk RMSNorm of the concatenated hidden states +(``m3_spec_tail.chunk_norm``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3ChunkNorm(OpWrapper): + """``dst[r, c*H:(c+1)*H] = rmsnorm(src[r, c*H:(c+1)*H]) * w[c]`` for the C chunks of ``w [C, H]``; ``src`` may + have a row stride wider than C * H (the capture buffer). Writes ``dst [M, C*H]`` and returns it.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The Eagle3 capture buffer of the target (3 captured layers x hidden 6144, buffer rows wider than the layers + # when the final hidden is also captured), over the verify rows of draft_len 3: 4 / 8 / 32 rows. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request's 4 verify rows, dense [4, 3 x 6144]", + spec=dict(m=4, c=3, h=6144, extra_cols=0), + ), + Cell( + why="cap 8 (32 rows) read from the capture buffer with a fourth (final-hidden) slot in the row stride", + spec=dict(m=32, c=3, h=6144, extra_cols=6144), + ), + ) + + note = """ + fp32 statistic per chunk (block sum, rsqrtf), one bf16 rounding: an output moves by one bf16 ulp now and then + against torch's statistic; the cells' band states it. + """ + + def __call__( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + torch.ops.trtllm.minimax_m3_chunk_norm(src, w, eps, dst, pdl) + return dst + + def reference( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + """Each chunk normalized on its own; ``dst`` only says where the op writes, ``pdl`` shapes the launch.""" + c, h = w.shape + x = src[:, : c * h].reshape(src.shape[0], c, h) + return rmsnorm_rows(x, w, eps).reshape(src.shape[0], c * h) + + def is_valid( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ) -> None: + c, h = w.shape + assert w.is_contiguous(), "w must be dense [C, H] (chunk c read at c * H)" + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + + +m3_chunk_norm = M3ChunkNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_draft_input.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_draft_input.py new file mode 100644 index 000000000000..75f7a44dfcfb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_draft_input.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 draft pass's input in one launch: embedding lookup, both RMSNorms and the concat +(``m3_spec_tail.draft_input``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3DraftInput(OpWrapper): + """``out [M, 2H] = [rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden]`` and ``residual [M, H] = + hidden[rows]`` (rows default to 0..M-1) in one launch; ids outside the table embed to zero (the masked lookup of + a vocab-local embedding). Returns ``(out, residual)``.""" + + ARCHS = frozenset({Arch.SM_103}) + + # Every pass of the fused draft loop (dispatch.DRAFT_INPUT) at draft_len 3, drafter hidden 6144, eps 1e-6: the + # first pass of 1..8 requests (4..32 rows, the carried hidden read in place) and the later passes of 1..8 requests + # (one row each; the second pass gathers the last accepted rows from the first pass's 4 rows per request). + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"first pass of {n} request(s): {4 * n} rows, hidden read in place", + spec=dict(m=4 * n, h=6144, gather=False, invalid_id=False), + ) + for n in range(1, 9) + ), + *( + Cell( + why=f"second pass of {n} request(s): the last accepted rows gathered from {4 * n} rows", + spec=dict(m=n, h=6144, gather=True, invalid_id=False), + ) + for n in range(1, 9) + ), + Cell( + why="third pass of 3 requests: one row each, hidden read in place", + spec=dict(m=3, h=6144, gather=False, invalid_id=False), + ), + Cell( + why="one row with an out-of-table id: the embedding half is zero", + spec=dict(m=1, h=6144, gather=False, invalid_id=True), + ), + ) + + note = """ + The arithmetic is the in-tree RMSNorm's (fp32 statistic, (x * rsqrt) * w, one bf16 rounding); the statistic's + summation order differs from FlashInfer's, which moves an output by one bf16 ulp now and then (the test's band, + also held against the in-tree modules). The out-of-table id is the masked lookup of a vocab-local embedding; the + drafter's embedding is the whole table, so its ids are always in range. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + torch.ops.trtllm.minimax_m3_draft_input( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, pdl + ) + return out, residual + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + """The two normalized halves (fp32 statistic, one bf16 rounding) and the gathered hidden rows. ``out`` / + ``residual`` only say where the op writes; ``pdl`` only shapes the launch.""" + valid = (ids >= 0) & (ids < table.shape[0]) + emb = rmsnorm_rows(table[torch.where(valid, ids, 0).long()], w_embed, eps_embed) + emb = torch.where(valid[:, None], emb, torch.zeros_like(emb)) + hid = hidden[rows.long()] if rows is not None else hidden[: ids.shape[0]] + ref = torch.cat([emb, rmsnorm_rows(hid, w_hidden, eps_hidden)], dim=-1) + return ref, (hid.clone() if residual is not None else None) + + def is_valid( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> None: + h = table.shape[1] + # Rows are read as hidden + rows[r] * H and table + id * H: both must be dense H-wide rows. + assert table.is_contiguous() and hidden.is_contiguous(), ( + "table / hidden must be dense [*, H]" + ) + if rows is not None: + assert bool(((rows >= 0) & (rows < hidden.shape[0])).all()), "rows must index hidden" + else: + assert hidden.shape[0] >= ids.shape[0], "hidden must hold a row per id" + # Vector loads of 16 bytes: H must be a multiple of 8 within the register tile, and the weights aligned. + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + assert residual is None or ( + residual.is_contiguous() and residual.shape == (ids.shape[0], h) + ), "residual must be contiguous [M, H]" + + +m3_draft_input = M3DraftInput() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_draft_step.py new file mode 100644 index 000000000000..3f00def536b6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_draft_step.py @@ -0,0 +1,359 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One pass of the fused greedy Eagle3 draft loop: the drafter's vocab-sharded BF16 head (streamed as BF16 fragments), +the cross-rank argmax and the loop's per-pass bookkeeping in one kernel (``m3_draft_step.draft_step``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +from .m3_shard_accept import exchange_tokens, local_argmax, make_keys + +#: Requests (head rows) per pass, and exchange rows per call half of the workspace. +MAX_TOKENS = 8 +#: 16-row weight tiles one CTA can own (the kernel's shared-memory accumulator). +MAX_TILES_PER_CTA = 32 +#: Draft length of the fused loop, and the drafter's TP8 head shard (200064-id vocabulary) and hidden size. +DRAFT_LEN = 3 +DRAFT_SHARD = 25008 +#: Ranks of the TP group the argmax is exchanged over. +WORLD = 8 +HIDDEN = 6144 + + +def bf16_fragments(weight: torch.Tensor) -> torch.Tensor: + """BF16 [N, K] -> the same values in A-fragment order [N / 16, K / 16, 256]. + + One lane's 16 bytes of a 16-row tile's k16 step are the m16n8k16 A fragment: element (row h * 8 + g, k ks * 16 + + hk * 8 + jj * 2 + e) goes to [tile][ks][g][jj][hk][h][e] (lane = g * 4 + jj). + """ + n, k = weight.shape + assert n % 16 == 0 and k % 16 == 0 and weight.dtype == torch.bfloat16, (n, k, weight.dtype) + q = weight.view(n // 16, 2, 8, k // 16, 2, 4, 2).permute(0, 3, 2, 5, 4, 1, 6).contiguous() + return q.view(n // 16, k // 16, 256) + + +def from_fragments(wf: torch.Tensor) -> torch.Tensor: + """The inverse of ``bf16_fragments``: [N / 16, K / 16, 256] -> [N, K].""" + tiles, ks, _ = wf.shape + return ( + wf.view(tiles, ks, 8, 4, 2, 2, 2).permute(0, 5, 2, 1, 4, 3, 6).reshape(tiles * 16, ks * 16) + ) + + +def head_weight( + wf: torch.Tensor, + k: int, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The BF16 head shard [N, K] the kernel multiplies by: the fragment-ordered copy. The op also streams a 12-bit + coded head (``exc_off`` given); this target never builds one, and nothing here certifies it.""" + if exc_off is not None: + raise NotImplementedError("the tp8 target streams the BF16 fragments only") + return from_fragments(wf) + + +def shard_logits( + x: torch.Tensor, + wf: torch.Tensor, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The in-tree head's BF16 logits of the rows x [M, K] against the shard (``head_weight``): the fp32 product, + rounded once to BF16 (the BF16 GEMM's output), as fp32.""" + w = head_weight(wf, x.shape[1], exc_off, exc_pos, exc_w, exp_base) + with true_fp32_matmul(): + logits = x.float() @ w.float().t() + return logits.to(torch.bfloat16).float() + + +class M3DraftStep(OpWrapper): + """One fused greedy draft pass over ``rows <= 8`` requests (one head row each). + + Head input ``x[x_rows[b]]`` (pass 0: the last accepted verify row of each request) or ``x[b]``. The head is this + rank's BF16 lm_head shard in fragment order (``m3_draft_step.bf16_fragments``, global ids ``vocab_offset ..``), + (the op also takes a 12-bit coded stream with ``exc_off`` / ``exc_pos`` / ``exc_w`` / ``exp_base``, which this + target never builds); each logit is the fp32 dot product + rounded to BF16 and the row's token is the global argmax over all ranks' + shards (ties, -0 / +0 included, to the lowest global id), exchanged through the multicast slots + ``[2][8][world]``. The last CTA then writes ``draft_out[b, step]`` and ``ids_next[b]`` (the token), ``pos_next[b] + = pos_src[x_rows[b] or b] + 1``, and advances the draft KV lengths as the in-tree draft loop does between passes + (pass 0 with ``num_acc``: ``kv_lens -= draft_len - num_acc`` and ``seq_lens = 1``; later passes: ``kv_lens += + 1``). ``out_keys`` (optional) receives each row's winning key. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # Every pass of the fused draft loop (dispatch.DRAFT_STEP): draft_len 3 at 1..8 requests, the drafter's + # full-vocabulary BF16 head (25008 rows per rank of 8, hidden 6144). Pass 0 reads the verify rows' hidden (4 per + # request) through x_rows and rolls the KV back; the later passes read one row per request. + _PASSES: tuple[Cell, ...] = ( + *( + Cell( + why=f"pass 0 at {n} request(s): head rows gathered from {4 * n} verify rows, KV rollback, seq_lens = 1", + spec=dict( + nreq=n, + draft_len=DRAFT_LEN, + step=0, + rank=n % WORLD, + world=WORLD, + vs=DRAFT_SHARD, + k=HIDDEN, + ), + ) + for n in range(1, MAX_TOKENS + 1) + ), + Cell( + why="pass 1 at two requests: one row each, kv_lens + 1, rank 7's ids", + spec=dict( + nreq=2, draft_len=DRAFT_LEN, step=1, rank=7, world=WORLD, vs=DRAFT_SHARD, k=HIDDEN + ), + ), + Cell( + why="last pass (step 2) at 8 requests, rank 0", + spec=dict( + nreq=8, draft_len=DRAFT_LEN, step=2, rank=0, world=WORLD, vs=DRAFT_SHARD, k=HIDDEN + ), + ), + Cell( + why="pass 2 at 5 requests with exact cross-rank ties (the lower global id wins)", + spec=dict( + nreq=5, + draft_len=DRAFT_LEN, + step=2, + rank=5, + world=WORLD, + vs=DRAFT_SHARD, + k=HIDDEN, + ties=True, + ), + ), + ) + # The BF16 fragments at every pass shape. + CELLS: tuple[Cell, ...] = tuple( + Cell(why=f"bf16: {c.why}", spec=dict(c.spec, coded=False)) for c in _PASSES + ) + + note = """ + The head is the in-tree BF16 head's arithmetic (BF16 weights and activations, fp32 accumulation, a BF16 logit) + and the argmax is the in-tree one over BF16 logits, so a token differs from the in-tree path's only where the fp32 + sum, accumulated in another order, rounds to the other side of a BF16 boundary and that decides the row's maximum. + The cells' rows are decidable and the peers' keys sit 0.25 away (or tie exactly); the edge tests (forced BF16 + ties, -0 / +0, top-2 one fp32 step apart that round to one BF16 value, adjacent BF16 values) use integer data + whose fp32 sums are exact in any order and hold the kernel to the in-tree path bit for bit. The cross-rank + exchange is certified on one device with the peers' keys written into the slots (see m3_shard_accept). + """ + + def __call__( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + ) -> None: + torch.ops.trtllm.minimax_m3_draft_step( + x, + x_rows, + rows, + wf, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + num_ctas, + pdl, + out_keys, + exc_off, + exc_pos, + exc_w, + exp_base, + ) + + def reference( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + ) -> dict[str, torch.Tensor]: + """Every buffer the pass writes, computed on copies: "logits" (this rank's BF16 logits as fp32), "local_keys" + (this rank's key per row), "keys" (the winning key), "tokens", "draft_out", "ids_next", "pos_next", "kv_lens" + and "seq_lens". + + Reads ``ws_uc``, ``counters`` and ``kv_lens`` before the call. ``keys``, ``ws_mc_ptr``, ``num_ctas``, ``pdl`` + and ``out_keys`` do not change the result. + """ + h = x[x_rows.long()] if x_rows is not None else x[:rows] + logits = shard_logits(h, wf, exc_off, exc_pos, exc_w, exp_base) + best, idx = local_argmax(logits, vocab_offset) + local_keys = make_keys(best, idx) + tokens = exchange_tokens(local_keys, ws_uc, counters, rank, world, MAX_TOKENS) + half = int(counters[1].item()) & 1 + slots = ws_uc.view(torch.int64)[ + half * MAX_TOKENS * world : (half * MAX_TOKENS + rows) * world + ] + slots = slots.view(rows, world).clone() + slots[:, rank] = local_keys + sign = torch.iinfo(torch.int64).min + win = (slots ^ sign).max(dim=1).values ^ sign + d_out = draft_out.clone() + d_out[:rows, step] = tokens + src = x_rows.long() if x_rows is not None else torch.arange(rows, device=x.device) + lens = kv_lens.clone() + sl = seq_lens.clone() if seq_lens is not None else None + if step == 0 and num_acc is not None: + lens[:rows] -= draft_len - num_acc[:rows] + sl[:rows] = 1 + elif step > 0: + lens[:rows] += 1 + out = { + "logits": logits, + "local_keys": local_keys, + "keys": win, + "tokens": tokens, + "draft_out": d_out, + "ids_next": tokens.clone(), + "pos_next": pos_src[src] + 1, + "kv_lens": lens, + } + if sl is not None: + out["seq_lens"] = sl + return out + + def is_valid( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + ) -> None: + assert not bool(keys[:rows].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + # The kernel reads the head row x + x_rows[b] * K and the position pos_src[x_rows[b]] with no bound. + if x_rows is not None: + assert bool(((x_rows[:rows] >= 0) & (x_rows[:rows] < x.shape[0])).all()), ( + "x_rows must index x" + ) + assert bool((x_rows[:rows] < pos_src.shape[0]).all()), ( + "pos_src must cover the head rows" + ) + else: + assert pos_src.shape[0] >= rows, "pos_src must cover the head rows" + # Each CTA accumulates at most MAX_TILES_PER_CTA tiles in shared memory. + tiles = wf.shape[0] if exc_off is None else exc_off.numel() - 1 + assert -(-tiles // num_ctas) <= MAX_TILES_PER_CTA, "too many weight tiles per CTA" + if exc_off is not None: + # The exceptions are read per tile in list order with no bound against the coded stream. + assert int(exc_off[-1].item()) == exc_pos.numel(), ( + "exc_off must end at the exception count" + ) + assert bool((exc_off[1:] >= exc_off[:-1]).all()), "exc_off must be non-decreasing" + # A NaN logit maps to a key above every finite one and wins the row silently. + h = x[x_rows.long()] if x_rows is not None else x[:rows] + assert not bool(torch.isnan(h).any()), "NaN head inputs win the argmax" + if num_acc is not None: + na = num_acc[:rows] + assert bool(((na >= 1) & (na <= draft_len + 1)).all()), ( + "num_acc counts must lie in [1, draft_len + 1]" + ) + + +m3_draft_step = M3DraftStep() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_force_vec.py new file mode 100644 index 000000000000..af4b06e3999f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_force_vec.py @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance, drawn as the in-tree worker draws them.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ..m3_force_vec import force_vec_kernel + + +class M3ForceVec(OpWrapper): + """Bump ``counter`` (int64 [1]) by one, then for each of the ``batch_size`` requests ``b`` write + ``min(base_total + (pool[(c * counter_stride + b * slot_stride) & pool_mask] < frac), max_total)`` into ``out`` + (int32), ``c`` the bumped counter: the in-tree ``_apply_force_accepted_tokens`` draw over its fixed-seed fp32 + ``pool``. ``block`` is the launch width (a power of two >= batch_size; 16, 32 or 64 are compiled at warmup). + + Only the forced-acceptance testing knob (``TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`` with a fractional part) + runs it. + """ + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, fraction 0.5", spec=dict(batch=1, block=16, frac=0.5)), + Cell(why="eight requests, fraction 0.3", spec=dict(batch=8, block=16, frac=0.3)), + ) + + note = """ + Integer arithmetic and one fp32 comparison: `reference` must match exactly. + """ + + def __call__( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + force_vec_kernel[(1,)]( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK=block, + ) + return out + + def reference( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + """The counts (a new tensor; ``counter`` and ``out`` are not written).""" + c = int(counter.item()) + 1 + slots = torch.arange(batch_size, dtype=torch.int64, device=pool.device) + idx = (c * counter_stride + slots * slot_stride) & pool_mask + r = pool[idx] + return torch.clamp(base_total + (r < frac).to(torch.int32), max=max_total).to(torch.int32) + + +m3_force_vec = M3ForceVec() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_sampled_accept.py new file mode 100644 index 000000000000..d0ad11de2ce9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_sampled_accept.py @@ -0,0 +1,269 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampled (temperature + top-p) Eagle3 verification over the TP-sharded vocabulary in one kernel +(``m3_sampled_accept.sampled_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_shard_accept import strict_accept + +#: Verify rows per call. +MAX_ROWS = 48 +#: The fused sampling kernel's 1 / T guard. +TEMP_EPSILON = 1e-6 + + +def nucleus_probs( + logits: torch.Tensor, temperatures: torch.Tensor, top_ps: torch.Tensor +) -> torch.Tensor: + """Per-row sampling distribution [rows, Vs] (fp32): w = exp(x / (T + 1e-6) - max), kept iff the mass strictly + above its weight is at most top_p * sum(w) (ties included), renormalized over the kept tokens.""" + inv = 1.0 / (temperatures.float() + TEMP_EPSILON) + x = logits.float() * inv[:, None] + w = torch.exp(x - x.max(dim=1, keepdim=True).values).double() + z = w.sum(dim=1, keepdim=True) + order = torch.sort(w, dim=1, descending=True) + sw = order.values + # mass strictly above each sorted weight: the cumulative sum before the first entry of its tie group + csum = torch.cumsum(sw, dim=1) - sw + first = torch.ones_like(sw, dtype=torch.bool) + first[:, 1:] = sw[:, 1:] != sw[:, :-1] + group = torch.cumsum(first.long(), dim=1) - 1 + starts = torch.zeros_like(csum) + starts.scatter_reduce_(1, group, csum, reduce="amin", include_self=False) + above = torch.gather(starts, 1, group) + keep_sorted = above <= top_ps.double()[:, None] * z + keep = torch.zeros_like(keep_sorted) + keep.scatter_(1, order.indices, keep_sorted) + p = torch.where(keep, w, torch.zeros_like(w)) + return (p / p.sum(dim=1, keepdim=True)).float() + + +class M3SampledAccept(OpWrapper): + """Sampled verification of ``rows = nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard. + + Each row draws one token from its top-p nucleus of softmax(logits / T) over the full (all-rank) vocabulary by + distributed rejection sampling (Philox keyed by the row's seed / offset, the row, the attempt and the CTA), then + the strict acceptance runs against ``draft`` as in ``m3_shard_accept``: ``acc_tokens``, ``num_acc``, + ``gather_ids``, optional ``next_new[:, 0]``. ``tokens`` receives the sampled token per row; ``extra_attempts`` + accumulates exchange rounds beyond the second. The exchange goes through the multicast workspace (two call + halves of ``minimax_m3_sampled_accept_ws_words(world)`` words) with the (finished rows, call) ``counters``. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # draft_len 3 (T = 4 rows per request) at 1 / 2 / 8 requests (4 / 8 / 32 rows, 8 CTAs per row) and the 48-row + # maximum (4 CTAs per row); the 25008-column TP8 target shard. Single rank (see note). + CELLS: tuple[Cell, ...] = ( + Cell( + why="cap 1 (4 rows), top_p 0.8 over a 12-token head: the draws follow the nucleus distribution", + spec=dict( + nreq=1, vs=25008, vocab_offset=0, check="distribution", top_p=0.8, temperature=0.7 + ), + ), + Cell( + why="cap 8 (32 rows, 256 co-resident CTAs): top_p ~ 0 keeps only the argmax, token exact", + spec=dict( + nreq=8, + vs=25008, + vocab_offset=2 * 25008, + check="argmax", + top_p=1e-7, + temperature=1.0, + ), + ), + Cell( + why="cap 2 (8 rows) with top_p = 1 (no cut: the first draw stands), next_new and forced counts", + spec=dict( + nreq=2, + vs=25008, + vocab_offset=25008, + check="distribution", + top_p=1.0, + temperature=1.0, + force="vec", + ), + ), + Cell( + why="48 rows (4 CTAs per row): the second launch geometry, nucleus distribution at top_p 0.5", + spec=dict( + nreq=12, + vs=25008, + vocab_offset=3 * 25008, + check="distribution", + top_p=0.5, + temperature=1.3, + force="total", + ), + ), + ) + + note = """ + What is sampled is gated statistically, the rest exactly. Every drawn token must lie in the reference nucleus; + the empirical distribution over many calls (distinct Philox offsets) must sit within a stated total-variation + band of the reference's; the acceptance outputs must equal the strict acceptance of the drawn tokens exactly. + The Philox stream itself is not reproduced in torch: the kernel's draw uses the hardware exp / log + approximations (__expf, __logf) and a CTA-order inverse CDF, which torch does not reproduce bit for bit. + + Certified at world 1 only. The decision is replicated from every rank's per-round records, which another rank + computes from its own shard on the fly, so peers cannot be stood in for on one device; the cross-rank path + needs the multi-device job and is not covered here. Rows with top-k or min-p must not reach this op (it applies + neither); is_valid states it. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + ) -> None: + torch.ops.trtllm.minimax_m3_sampled_accept( + logits, + vocab_offset, + temperatures, + top_ps, + seeds, + offsets, + ws_uc, + ws_mc_ptr, + counters, + tokens, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + force_vec, + extra_attempts, + use_pdl, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + ) -> dict: + """{"probs": the per-row nucleus distribution over this shard [rows, Vs] (global id = vocab_offset + + column), "accept": tokens -> the strict acceptance outputs}. The draw itself is random; ``seeds`` / + ``offsets`` select it and do not change the distribution. World 1: this shard is the whole vocabulary.""" + rows = logits.shape[0] + probs = nucleus_probs(logits, temperatures[:rows], top_ps[:rows]) + + def accept(drawn: torch.Tensor) -> dict[str, torch.Tensor]: + return strict_accept(drawn, draft, batch_idx, force_total, next_new, force_vec) + + return {"probs": probs, "accept": accept} + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + ) -> None: + rows = logits.shape[0] + # top-k and min-p are not applied: a row that has them gets a token outside its real sampling set. + if top_ks is not None: + k = top_ks[:rows] + assert bool(((k <= 0) | (k >= logits.shape[1] * world)).all()), ( + "top-k rows must take the generic path" + ) + if min_ps is not None: + assert not bool((min_ps[:rows] > 0).any()), "min-p rows must take the generic path" + # Rows are request-major verify rows: tokens[b * (D + 1) + k]; a mismatched layout pairs drafts with the + # wrong rows. + assert rows == draft.shape[0] * (draft.shape[1] + 1), ( + "logits rows must be nreq * (draft_len + 1)" + ) + # Per-row parameters are read at the row index, unchecked. + for t in (temperatures, top_ps, seeds, offsets): + assert t.numel() >= rows and t.is_contiguous(), ( + "per-row parameters must cover every row" + ) + assert temperatures.dtype == torch.float32 and top_ps.dtype == torch.float32, ( + "fp32 sampling parameters" + ) + assert seeds.dtype == torch.int64 and offsets.dtype == torch.int64, ( + "int64 Philox seeds / offsets" + ) + # The finished-row counter elects the CTA that runs the acceptance. + assert int(counters[0].item()) == 0, ( + "counters[0] (finished rows) must be zero between calls" + ) + + +m3_sampled_accept = M3SampledAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_sampled_accept_ws_words.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_sampled_accept_ws_words.py new file mode 100644 index 000000000000..b0a17e38a983 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_sampled_accept_ws_words.py @@ -0,0 +1,43 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Size of the sampled-verify exchange workspace (``m3_sampled_accept.SampledAcceptWorkspace.create``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Exchange rounds per row, verify rows per call, CTAs per row (at most), uint64 words per record. +MAX_ATTEMPTS = 16 +MAX_ROWS = 48 +MAX_SPLITS = 8 +RECORD_WORDS = 4 + + +class M3SampledAcceptWsWords(OpWrapper): + """uint64 words of the multicast workspace ``minimax_m3_sampled_accept`` exchanges through for ``world`` ranks: + two call halves of ``[attempts][rows][world * splits][record words]``.""" + + ARCHS = frozenset({Arch.SM_103}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="the tp8 target's workspace", spec=dict(world=8)), + Cell(why="a single rank (local workspace, no multicast)", spec=dict(world=1)), + ) + + note = """ + A host computation (no device work); it is an op so the workspace size and the kernel's indexing come from one + definition. + """ + + def __call__(self, world: int) -> int: + return torch.ops.trtllm.minimax_m3_sampled_accept_ws_words(world) + + def reference(self, world: int) -> int: + """Two halves of attempts x rows x (world x splits) records x record words.""" + return 2 * MAX_ATTEMPTS * MAX_ROWS * world * MAX_SPLITS * RECORD_WORDS + + +m3_sampled_accept_ws_words = M3SampledAcceptWsWords() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_shard_accept.py new file mode 100644 index 000000000000..500553c2bc88 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_shard_accept.py @@ -0,0 +1,284 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Greedy Eagle3 verification from this rank's logits shard: per-row argmax, the cross-rank (value, index) exchange +and the strict acceptance in one kernel (``m3_shard_accept.shard_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Verify rows per call, and exchange rows per call half of its workspace. +SHARD_ROWS = 48 + +_LOW32 = 0xFFFFFFFF + + +def ordered_bits(v: torch.Tensor) -> torch.Tensor: + """The kernel's order-preserving uint32 image of fp32 values, as int64 (-0.0 maps to +0.0's image: they tie).""" + b = v.float().contiguous().view(torch.int32).to(torch.int64) & _LOW32 + b = torch.where(b == 0x80000000, torch.zeros_like(b), b) + return torch.where(b >= 0x80000000, _LOW32 - b, b | 0x80000000) + + +def make_keys(values: torch.Tensor, index: torch.Tensor) -> torch.Tensor: + """Exchange keys of (fp32 value, global token index) pairs: larger value wins, ties take the lower index.""" + return (ordered_bits(values) << 32) | (_LOW32 - index.to(torch.int64)) + + +def unsigned_max(keys: torch.Tensor, dim: int) -> torch.Tensor: + """Max of int64-stored uint64 keys (the kernel compares them unsigned).""" + sign = torch.iinfo(torch.int64).min + return (keys ^ sign).max(dim=dim).values ^ sign + + +def key_token(key: torch.Tensor) -> torch.Tensor: + return (_LOW32 - (key & _LOW32)).to(torch.int32) + + +def local_argmax(logits: torch.Tensor, vocab_offset: int) -> tuple[torch.Tensor, torch.Tensor]: + """(max value, lowest global index reaching it) per row of a [M, Vs] fp32 shard.""" + best = logits.max(dim=1).values + idx = (logits == best[:, None]).float().argmax(dim=1) # first (lowest) index at the max + return best, idx.to(torch.int64) + vocab_offset + + +def exchange_tokens( + own_keys: torch.Tensor, + ws_uc: torch.Tensor, + counters: torch.Tensor, + rank: int, + world: int, + slot_rows: int, +) -> torch.Tensor: + """Global tokens of the cross-rank pick: this rank's keys against the other ranks' keys already in its slots. + + Slot (half * slot_rows + t) * world + r of ``ws_uc`` holds rank r's key of row t; half = call counter & 1. A + single-device run stands in for the other ranks by writing their keys there before the call. + """ + m = own_keys.shape[0] + half = int(counters[1].item()) & 1 + slots = ( + ws_uc.view(torch.int64)[half * slot_rows * world : (half * slot_rows + m) * world] + .view(m, world) + .clone() + ) + slots[:, rank] = own_keys + return key_token(unsigned_max(slots, dim=1)) + + +def strict_accept( + tokens: torch.Tensor, + draft: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], +) -> dict[str, torch.Tensor]: + """The strict acceptance epilogue from the verify-row tokens [nreq * (D + 1)] (request-major). + + accepted[b] = tokens of request b's rows; num_accepted[b] = 1 + the matching prefix of draft[b] (overridden by + force_vec[b], else by force_total > 0); gather_ids[b] = batch_idx[b] * (D + 1) + num_accepted[b] - 1, and + next_new[b, 0] (when given) = the token at that row. Returns the outputs plus ``next_new`` as written. + """ + nreq, d = draft.shape + k1 = d + 1 + tokens = tokens.to(torch.int32) + accepted = tokens.view(nreq, k1).clone() + match = (draft.to(torch.int32) == accepted[:, :d]).to(torch.int32) + num = 1 + match.cumprod(dim=1).sum(dim=1) + if force_vec is not None: + num = force_vec[:nreq].to(num.dtype) + elif force_total > 0: + num = torch.full_like(num, force_total) + gather = batch_idx.to(torch.int64) * k1 + num.to(torch.int64) - 1 + out = {"accepted": accepted, "num_accepted": num.to(torch.int32), "gather_ids": gather} + if next_new is not None: + nn = next_new.clone() + nn[:, 0] = tokens[gather] + out["next_new"] = nn + return out + + +class M3ShardAccept(OpWrapper): + """Greedy verification of ``nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard ``[rows, Vs]`` + (global ids ``vocab_offset ..``). + + Each row's token is its global argmax over all ranks' shards: keys ``ordered(value) << 32 | ~index`` (ties take + the lowest index, as ``torch.argmax``) are max-reduced over the shard, pushed into every rank's exchange slot + through the multicast slots ``[2][48][world]`` (Lamport: 0 = empty) and max-reduced over the ranks by the last + CTA, which then runs the strict acceptance: ``acc_tokens [nreq, D + 1]``, ``num_acc [nreq]`` (``force_vec`` / + ``force_total`` override the counts), ``gather_ids [nreq]`` (the first draft pass's row per request) and, when + given, ``next_new[:, 0]`` (the last accepted token). + """ + + ARCHS = frozenset({Arch.SM_103}) + + # Every greedy verify step of the target (dispatch.SHARD_ACCEPT): draft_len 3 at 1..12 requests (4..48 rows), the + # target lm_head's BF16 shard logits (25008 columns per rank at TP8). The logits come from the head GEMM, so the + # op only reduces. + CELLS: tuple[Cell, ...] = ( + Cell( + why="cap 1: 4 verify rows (1 request x T = 4), rank 0 of 8, next_new staged", + spec=dict(nreq=1, draft_len=3, vs=25008, world=8, rank=0, force="none", strided=False), + ), + Cell( + why="cap 2: 8 verify rows, rank 2, global ids offset by 2 x 25008", + spec=dict(nreq=2, draft_len=3, vs=25008, world=8, rank=2, force="none", strided=False), + ), + Cell( + why="cap 4: 16 verify rows (4 requests x T = 4), rank 1 of 8, next_new staged", + spec=dict(nreq=4, draft_len=3, vs=25008, world=8, rank=1, force="none", strided=False), + ), + Cell( + why="cap 8: 32 verify rows, rank 7, global ids offset by 7 x 25008", + spec=dict(nreq=8, draft_len=3, vs=25008, world=8, rank=7, force="none", strided=False), + ), + Cell( + why="12 rows (3 requests) read from a row-strided view of a wider logits buffer, forced counts", + spec=dict(nreq=3, draft_len=3, vs=25008, world=8, rank=0, force="vec", strided=True), + ), + Cell( + why="the 48-row maximum (12 requests at draft 3), force_total, rows with exact cross-rank ties", + spec=dict( + nreq=12, + draft_len=3, + vs=25008, + world=8, + rank=2, + force="total", + strided=False, + ties=True, + ), + ), + # Every other request count of the verify range, on the 8-rank 25008-column shards. + *( + Cell( + why=f"{n} requests ({4 * n} verify rows): a request count between the ones above", + spec=dict( + nreq=n, draft_len=3, vs=25008, world=8, rank=n % 8, force="none", strided=False + ), + ) + for n in (5, 6, 7, 9, 10, 11) + ), + ) + + note = """ + The op reduces BF16 logits exactly, so the reference is bit-exact, including cross-rank ties (lowest global + index wins). The cross-rank exchange is certified on one device: the other ranks' keys are written into this + rank's slots before the call, which is what their multicast pushes would have put there; the multicast mapping + itself (ws_mc_ptr = this rank's slots, as at world 1) is not exercised across devices here. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + torch.ops.trtllm.minimax_m3_shard_accept( + logits, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + num_ctas, + force_vec, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> dict[str, torch.Tensor]: + """Lowest-index maximum of each row's BF16 logits, the largest key among it and the peers' keys in + ``ws_uc``, then the strict acceptance. Reads ``ws_uc`` / ``counters`` before the call; the scratch and + output buffers and ``num_ctas`` do not change the result.""" + best, idx = local_argmax(logits.float(), vocab_offset) + tokens = exchange_tokens(make_keys(best, idx), ws_uc, counters, rank, world, SHARD_ROWS) + out = strict_accept(tokens, draft, batch_idx, force_total, next_new, force_vec) + out["tokens"] = tokens + return out + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + assert not bool(keys[: logits.shape[0]].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + assert ws_uc.numel() >= 2 * SHARD_ROWS * world, "ws_uc must hold [2][48][world] slots" + # A NaN logit maps to a key above every finite one and wins the row silently. + assert not bool(torch.isnan(logits).any()), "NaN logits win the argmax" + nreq = draft.shape[0] + assert bool(((batch_idx[:nreq] >= 0) & (batch_idx[:nreq] < nreq)).all()), ( + "batch_idx must index the nreq requests" + ) + if force_vec is not None: + fv = force_vec[:nreq] + assert bool(((fv >= 1) & (fv <= draft.shape[1] + 1)).all()), ( + "force_vec counts must lie in [1, D + 1]" + ) + + +m3_shard_accept = M3ShardAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_slot_scatter.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_slot_scatter.py new file mode 100644 index 000000000000..470a6372f2b6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_slot_scatter.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The spec sampler's per-slot stores of one step's outputs, in one launch (``M3SpecStepFold.scatter`` -> +``m3_spec_tail.slot_scatter``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +def padded_rows(src: torch.Tensor, width: int) -> torch.Tensor: + """src rows zero-padded (or cut) to ``width`` columns.""" + out = torch.zeros(src.shape[0], width, dtype=src.dtype, device=src.device) + w = min(width, src.shape[1]) + out[:, :w] = src[:, :w] + return out + + +class M3SlotScatter(OpWrapper): + """For i < len(slots), all int32 2-D strided views: ``da[slots[i]] = sa[i]``, ``db[slots[i]] = sb[i]``, + ``dd[slots[i]] = sd[i]`` (each zero-padded / cut to the destination width) and ``dl[slots[i]] = sl[i]``. The + destinations are the store's transposed [slots, width] views.""" + + ARCHS = frozenset({Arch.SM_103}) + + # The sampler store at draft_len 3 (8 slots): new tokens / next new tokens [slots, 4] (transposed views of + # [4, slots, 1]), next draft tokens [slots, 3]; a worker output narrower than the store (dynamic draft length) + # is zero-padded. + CELLS: tuple[Cell, ...] = ( + Cell( + why="8 rows into shuffled slots, widths equal to the store's", + spec=dict(n=8, slots=8, src_w=4, src_d=3, store_w=4, store_d=3), + ), + Cell( + why="3 rows of a narrower step (runtime draft 2): zero padding to the store width", + spec=dict(n=3, slots=8, src_w=3, src_d=2, store_w=4, store_d=3), + ), + ) + + note = """ + Pure index copies: the reference is exact. Slots are not bounds-checked; is_valid states it. + """ + + def __call__(self, slots, sa, da, sb, db, sl, dl, sd, dd): + torch.ops.trtllm.minimax_m3_slot_scatter(slots, sa, da, sb, db, sl, dl, sd, dd) + + def reference(self, slots, sa, da, sb, db, sl, dl, sd, dd): + """The four destinations after the call, on copies.""" + s = slots.long() + out = {} + for name, src, dst in (("da", sa, da), ("db", sb, db), ("dd", sd, dd)): + r = dst.clone() + r[s] = padded_rows(src[: s.numel()], dst.shape[1]) + out[name] = r + lr = dl.clone() + lr[s] = sl[: s.numel()] + out["dl"] = lr + return out + + def is_valid(self, slots, sa, da, sb, db, sl, dl, sd, dd): + assert bool(((slots >= 0) & (slots < da.shape[0]) & (slots < dl.numel())).all()), ( + "slots must index the store" + ) + # Two rows into one slot race (which one lands is unordered). + assert slots.unique().numel() == slots.numel(), "slots must be distinct" + assert sa.shape[0] >= slots.numel() and sl.numel() >= slots.numel(), "a source row per slot" + + +m3_slot_scatter = M3SlotScatter() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_step_gather.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_step_gather.py new file mode 100644 index 000000000000..abea2cf620e9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/spec/m3_step_gather.py @@ -0,0 +1,152 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The overlap scheduler's per-step input gathers from the previous step's device outputs, in one launch +(``M3SpecStepFold.gather`` -> ``m3_spec_tail.step_gather``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3StepGather(OpWrapper): + """For the ``P`` previous requests (sequence slots ``slots``, ``T`` tokens and ``D`` draft tokens each), all + int32:: + + ids[ids0 + p*T + t] = new_tok[t, slots[p]] (new_tok [T', slots, (1)]) + draft[d0 + p*D + d] = next_draft[slots[p], d] + pos_off[i] = lens[ppi[i - b0*T]] for i in [b0*T, (b0+P)*T), else 0 (every i < pos_off.numel()) + kv_off[i] = lens[slots[i - b0]] - T for i in [b0, b0+P), else 0 (every i < kv_off.numel()) + + Every other element of ``ids`` / ``draft`` is left as it was. + """ + + ARCHS = frozenset({Arch.SM_103}) + + # draft_len 3: T = 4, D = 3; the store's [T, slots, 1] new tokens and [slots, D] next drafts (8 slots, the + # max batch); the engine's flat buffers sized for the max batch. + CELLS: tuple[Cell, ...] = ( + Cell( + why="an unchanged batch of 8 previous requests in shuffled slots, nothing before them (b0 = 0)", + spec=dict(p=8, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0), + ), + Cell( + why="2 previous requests after 3 new ones: offsets ids0 / d0 / b0 nonzero, zero-filled offsets outside", + spec=dict(p=2, t=4, d=3, slots=8, b0=3, ids0=12, d0=9, extra_rows=3), + ), + Cell( + why="one request, the store's 2-D view of new tokens (beam dim squeezed)", + spec=dict(p=1, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0, two_d=True), + ), + ) + + note = """ + Pure index copies: the reference is exact. The op does not bound slots, ppi or the destination offsets; is_valid + states them. + """ + + def __call__( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + torch.ops.trtllm.minimax_m3_step_gather( + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ) + + def reference( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + """The four destinations after the call, on copies (``ids`` / ``draft`` keep their other elements).""" + nt = new_tok.reshape(new_tok.shape[0], new_tok.shape[1]) + s = slots[:P].long() + ids_r, draft_r = ids.clone(), draft.clone() + ids_r[ids0 : ids0 + P * T] = nt[:T, s].t().reshape(-1) + draft_r[d0 : d0 + P * D] = next_draft[s, :D].reshape(-1) + pos_r = torch.zeros_like(pos_off) + pos_r[b0 * T : (b0 + P) * T] = lens[ppi[: P * T].long()] + kv_r = torch.zeros_like(kv_off) + kv_r[b0 : b0 + P] = lens[s] - T + return {"ids": ids_r, "draft": draft_r, "pos_off": pos_r, "kv_off": kv_r} + + def is_valid( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + # Reads through slots / ppi and writes at the offsets are unchecked: any of them out of range reads or + # writes another buffer. + s = slots[:P] + assert bool( + ( + (s >= 0) & (s < next_draft.shape[0]) & (s < new_tok.shape[1]) & (s < lens.numel()) + ).all() + ), "slots must index the store" + assert bool(((ppi[: P * T] >= 0) & (ppi[: P * T] < lens.numel())).all()), ( + "ppi must index lens" + ) + assert new_tok.shape[0] >= T and next_draft.shape[1] >= D, "store narrower than T / D" + assert ids.numel() >= ids0 + P * T and draft.numel() >= d0 + P * D, "destinations too short" + assert pos_off.numel() >= (b0 + P) * T and kv_off.numel() >= b0 + P, ( + "offset buffers too short" + ) + + +m3_step_gather = M3StepGather() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/triton_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/triton_sparse_decode.py new file mode 100644 index 000000000000..0e8dc710bc98 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/triton_sparse_decode.py @@ -0,0 +1,1662 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. + +This is the MiniMax-M3 NVFP4 / sm_103 / tp8 target's copy. On top of the +in-tree decode it routes long-context TP8 steps to the one-launch +``m3_attn_lc8`` kernel, short-sequence verify steps of at least +``MQ_MIN_TOKENS`` rows to the multi-query producer, releases PDL dependents +early, and has the merge emit the MXFP8 o_proj hand-off (``m3_prequant.SLOT``). +""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers + +from . import m3_attn_lc8, m3_prequant + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 # capacity: fixed: the MSA block size of the checkpoint + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 # capacity: fixed: a launch-grid target, not a size + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +# Fewest query rows for which the merge writes the MXFP8 o_proj hand-off: every decode step, so the MXFP8 o_proj +# never runs its own quantize kernel after attention. +MERGE_MXFP8_MIN_TOKENS = 1 # capacity: fixed: a row lower bound + + +class MergeMxfp8: + """MXFP8 copy of the split-K merge output for the o_proj GEMM. + + From ``MERGE_MXFP8_MIN_TOKENS`` to ``MAX_ROWS`` rows the merge kernel writes the E4M3 rows and swizzled UE8M0 + scales next to the BF16 output and publishes them to ``m3_prequant.SLOT``, so the MXFP8 o_proj skips its own + quantization. One 128-row buffer per width serves every layer: each o_proj consumes it before the next layer's + merge rewrites it, in stream order. + """ + + MAX_ROWS = 128 # capacity: fixed: the merge hand-off's rows (decode steps) + + def __init__(self): + self._buffers = {} + + def buffers(self, width: int, device: torch.device): + key = (width, device) + if key not in self._buffers: + self._buffers[key] = ( + torch.empty((self.MAX_ROWS, width), dtype=torch.float8_e4m3fn, device=device), + torch.zeros((self.MAX_ROWS * (width // 32),), dtype=torch.uint8, device=device), + ) + return self._buffers[key] + + +MERGE_MXFP8 = MergeMxfp8() + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +# Mixed ctx+gen steps launch on [num_ctx:] slices (unaligned seq_lens / block_table) with arbitrary +# total_q; specializing on those compiled a new variant mid-serving (~650 ms TTFT stall at c8). +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + out_ptr, # final [total_q, num_heads, head_dim] for FUSED_MERGE, else unused + counter_ptr, # int32 [total_q, num_kv_heads] arrival counters for FUSED_MERGE, zero at rest + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + stride_out_n, + stride_out_h, + stride_out_d, + num_kv_heads, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge + FUSED_MERGE: tl.constexpr, # the last chunk to finish merges the partials in place + EARLY_TRIGGER: tl.constexpr, # release PDL dependents (weight prefetch) at entry +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + # Dependents still wait for this grid to complete; triggering here only + # lets them start their input-independent prologue (e.g. weight loads). + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for it in tl.range(chunk_start, chunk_end): + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + if FUSED_MERGE: + # Split-K serial reduction: every chunk publishes its partial, and the + # chunk that arrives last for this (token, KV head) merges all of them + # exactly as _merge_topk_attn_out_kernel does, then re-arms the counter. + tl.debug_barrier() + counter = counter_ptr + pid_b * num_kv_heads + pid_kh + arrived = tl.atomic_add(counter, 1, sem="acq_rel", scope="gpu") + if arrived == NUM_TOPK_CHUNKS - 1: + tl.debug_barrier() + o_rows = ( + o_ptr + + pid_b * stride_o_b + + (pid_h + off_h[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d + ) + l_rows = lse_ptr + pid_b * stride_l_b + (pid_h + off_h) * stride_l_h + lse_max = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + lse_max = tl.maximum(lse_max, lse_c) + # All chunks empty (a CUDA-graph padding row): zero, never NaN. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + denom = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + denom += tl.exp2(lse_c - lse_max) + denom = tl.where(denom > 0, denom, 1.0) + merged = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + w_c = tl.exp2(lse_c - lse_max) / denom + o_c = tl.load( + o_rows + c * stride_o_c, mask=hd_mask, other=0.0, cache_modifier=".cg" + ).to(tl.float32) + merged += o_c * w_c[:, None] + tl.store( + out_ptr + + pid_b * stride_out_n + + (pid_h + off_h[:, None]) * stride_out_h + + off_d[None, :] * stride_out_d, + merged.to(out_ptr.dtype.element_ty), + mask=hd_mask, + ) + tl.atomic_xchg(counter, 0, sem="relaxed", scope="gpu") + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for _ in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.jit +def _rcp_approx_ftz(x): + return tl.inline_asm_elementwise( + "rcp.approx.ftz.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _store_mxfp8_head(x, q8_ptr, sf_ptr, row, head, num_sf_k_tiles, HEAD_DIM: tl.constexpr): + """Quantize one row's [head * HEAD_DIM, (head + 1) * HEAD_DIM) slice to MXFP8 as + torch.ops.trtllm.mxfp8_quantize(x, True) does: UE8M0 scale = round-up(amax * rcp(448)), E4M3 RN-satfinite, + scales in the 128x4-swizzled layout.""" + NUM_SF: tl.constexpr = HEAD_DIM // 32 + xr = tl.reshape(x.to(tl.float32), (NUM_SF, 32)) + amax = tl.max(tl.abs(xr), axis=1) + sfv = amax * _rcp_approx_ftz(tl.full((NUM_SF,), 448.0, tl.float32)) + bits = sfv.to(tl.int32, bitcast=True) + e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(tl.int32) + e = tl.minimum(e, 254) + scale = ((254 - e) << 23).to(tl.float32, bitcast=True) + scale = tl.where(amax != 0, scale, 0.0) + q = (xr * scale[:, None]).to(tl.float8e4nv) + off_q = tl.reshape(tl.arange(0, HEAD_DIM), (NUM_SF, 32)) + tl.store(q8_ptr + row * (num_sf_k_tiles * 128) + head * HEAD_DIM + off_q, q) + k_sf = head * NUM_SF + tl.arange(0, NUM_SF) + off_sf = ( + (row // 128) * num_sf_k_tiles * 512 + + (k_sf // 4) * 512 + + (row % 32) * 16 + + ((row % 128) // 32) * 4 + + k_sf % 4 + ) + tl.store(sf_ptr + off_sf, e.to(tl.uint8)) + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr = False, + q8_ptr=None, # MXFP8 copy of out: [total_q, num_heads * head_dim] E4M3 + sf_ptr=None, # its UE8M0 block scales, 128x4-swizzled + num_sf_k_tiles=0, # ceil(num_heads * head_dim / 128) + MXFP8_OUT: tl.constexpr = False, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + if EARLY_TRIGGER: + # Release the dependent (the o_proj GEMM) before waiting on the attention producer, so its weight + # prefetch overlaps the attention kernel as well as this merge; its griddepcontrol.wait still waits + # for this grid (and therefore the producer) to complete. + tl.extra.cuda.gdc_launch_dependents() + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + if MXFP8_OUT: + _store_mxfp8_head( + o_merged.to(out_ptr.dtype.element_ty), + q8_ptr, + sf_ptr, + pid_b, + pid_h, + num_sf_k_tiles, + BLOCK_SIZE_D, + ) + + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on both datacenter Blackwell targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3)) +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +# Multi-query producer (triton_sparse_decode_mq) for identity-selection verify steps with at least this many query +# tokens; below it the per-token producer is as fast. +MQ_MIN_TOKENS = 17 # capacity: fixed: the multi-query producer's crossover +# Warps per multi-query producer program. +MQ_NUM_WARPS = 8 + + +def _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, +) -> None: + """minimax_m3_sparse_attn_decode through m3_attn_lc8: same table, output and MXFP8 hand-off contract.""" + use_pdl = get_env_enable_pdl() + total_q, num_heads, head_dim = q.shape + linear = _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=1, + gqa_group_size=num_heads, + max_topk=int(topk_idx.shape[-1]), + decode_query_len=decode_query_len, + ) + mxfp8 = None + m3_prequant.SLOT.clear() + if MERGE_MXFP8_MIN_TOKENS <= total_q <= MergeMxfp8.MAX_ROWS: + mxfp8 = MERGE_MXFP8.buffers(num_heads * head_dim, output.device) + # The early trigger lets the o_proj launch and stream its weights while attention runs. + m3_attn_lc8.m3_lc8_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + linear_softmax=linear, + mxfp8=mxfp8, + use_pdl=use_pdl, + ) + if mxfp8 is not None: + m3_prequant.SLOT.publish( + output.view(total_q, num_heads * head_dim), mxfp8[0][:total_q], mxfp8[1] + ) + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + if kv_nvfp4: + capability = torch.cuda.get_device_capability(q.device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + # Long-context TP8 steps: attention, split-K merge and MXFP8 hand-off in one launch. A selected table on an NVFP4 + # cache with FP8 q at the TP8 heads is the kernel's (modeling's SPARSE_DECODE_LC8); operands it declines are a miss. + lc8_covered = ( + kv_nvfp4 + and widen_q + and not identity_blocks + and num_topk_chunks is None + and q.dim() == 3 + and int(q.shape[1]) == m3_attn_lc8.NUM_HEADS + ) + if lc8_covered and not m3_attn_lc8.lc8_applies( + q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output + ): + from .. import modeling + + modeling.missed( + modeling.SPARSE_DECODE_LC8.path, + total_q, + "the step's q / KV pages / scales / output are not the layout the kernel is built for", + ) + lc8_covered = False + if lc8_covered: + _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + ) + return + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else 1 + ) + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + if kv_nvfp4 and total_q * num_kv_heads <= 64: + # GB300 TP4 Eagle verify (1 KV head, group 16, 6 query tokens): one block per program is fastest up to + # batch 8 (lowm5/bench_decode_chunks.py: b8 11.7 vs 14.8 us with the default split of 4). + num_topk_chunks = max_topk + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + # Every query token of a request reads the same blocks under identity selection (short-sequence mode): one + # program per (request, block) covers all of them and dequantizes each block once. + use_mq = ( + identity_blocks + and total_q >= MQ_MIN_TOKENS + and kv_nvfp4 + and num_kv_heads == 1 + and decode_query_len > 1 + and head_dim == 128 + and gqa_group_size * decode_query_len <= 128 + ) + if use_mq: + num_topk_chunks = max_topk + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = get_env_enable_pdl() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + single_kernel_args = ( + { + # No split-K arrival counters: the separate merge kernel combines the chunks. + "merge": (output, None), + "tail": (output.stride(0), output.stride(1), output.stride(2), num_kv_heads), + # An early release pays off when the dependent is the consumer of the + # final output (it can prefetch), or a merge grid that itself releases + # its dependent at entry: the o_proj then launches during this kernel and + # streams its weights while attention runs. Above 8 query rows the + # concurrent weight stream slows the larger attention grid more than the + # prefetch saves (GB300 TP4: -1.1% at 48 rows). + "options": {"FUSED_MERGE": False, "EARLY_TRIGGER": direct_output or total_q <= 8}, + } + if query_group_size == 1 + else {"merge": (), "tail": (), "options": {}} + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + if use_mq: + from .triton_sparse_decode_mq import launch_mq_producer + + launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + sm_scale=sm_scale, + decode_query_len=decode_query_len, + use_pdl=use_pdl, + early_trigger=False, + num_warps=MQ_NUM_WARPS, + ) + else: + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + *single_kernel_args["merge"], + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + *single_kernel_args["tail"], + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + **single_kernel_args["options"], + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + m3_prequant.SLOT.clear() + if not direct_output: + width = num_heads * head_dim + mxfp8 = ( + MERGE_MXFP8.buffers(width, output.device) + if MERGE_MXFP8_MIN_TOKENS <= total_q <= MergeMxfp8.MAX_ROWS + and head_dim == 128 + and output.dim() == 3 + and output.is_contiguous() + and output.dtype == torch.bfloat16 + else None + ) + mxfp8_args = ( + { + "q8_ptr": mxfp8[0], + "sf_ptr": mxfp8[1], + "num_sf_k_tiles": width // 128, + "MXFP8_OUT": True, + } + if mxfp8 is not None + else {} + ) + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + # Release the o_proj at the merge's entry so it streams its weights during the merge. + EARLY_TRIGGER=True, + **mxfp8_args, + **pdl_launch, + **merge_launch_options, + ) + if mxfp8 is not None: + m3_prequant.SLOT.publish(output.view(total_q, width), mxfp8[0][:total_q], mxfp8[1]) + + +__all__ = [ + "MERGE_MXFP8", + "MERGE_MXFP8_MIN_TOKENS", + "MergeMxfp8", + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/triton_sparse_decode_mq.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/triton_sparse_decode_mq.py new file mode 100644 index 000000000000..048d31b82d68 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/ops/triton_sparse_decode_mq.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multi-query block-sparse decode producer for MiniMax-M3 short-sequence verify steps. + +In short-sequence mode every request attends at most ``topk`` KV blocks and block selection is the identity, so +all ``decode_query_len`` query tokens of a request read the same blocks. The per-token producer +(``_gqa_sparse_decode_kernel``) then dequantizes every block once per query token. This producer runs one program +per (request, block) for all of the request's query tokens: rows are (token, head) pairs, each block is +dequantized once, and the causal limit is a per-row mask. + +It writes the same per-block partials (output scaled by 1/l and the V global scale, log2 LSE) as the per-token +producer with one block per chunk, so ``_merge_topk_attn_out_kernel`` merges them unchanged. +""" + +import triton +import triton.language as tl + +from .triton_sparse_decode import _dequant_nvfp4_rows + + +@triton.jit( + do_not_specialize=["num_reqs"], do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"] +) +def _gqa_sparse_decode_mq_kernel( + q_ptr, # [num_reqs * QLEN, num_heads, head_dim], FP8 or FP16/BF16 + k_ptr, # packed NVFP4 K: [num_pages, 1, page_size, head_dim / 2] uint8 + v_ptr, # packed NVFP4 V: same layout + k_sc_ptr, # K block scales [num_pages, 1, page_size, head_dim / 16], token-linear + v_sc_ptr, # V block scales, 4x4 token-quad order + k_gs_ptr, + v_gs_ptr, + t_ptr, # topk table [1, total_q, topk]; the request's last token's row is read + o_ptr, # partials [topk, total_q, num_heads, head_dim] fp32 + lse_ptr, # partials [topk, total_q, num_heads] fp32 (log2) + block_table_ptr, + seq_lens, + num_reqs, + sm_scale, + stride_qn, + stride_qh, + stride_k_blk, + stride_k_pos, + stride_v_blk, + stride_v_pos, + stride_ksc_blk, + stride_vsc_blk, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + QLEN: tl.constexpr, + GROUP: tl.constexpr, # query heads per KV head (16) + BLOCK_R: tl.constexpr, # next power of two of QLEN * GROUP + HEAD_DIM: tl.constexpr, + BLOCK_SIZE_K: tl.constexpr, + SCALE_COLS: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr, +): + pid_r = tl.program_id(0) + pid_c = tl.program_id(1) # top-k position == block index under identity selection + sm_scale_log2e = sm_scale * 1.4426950409 * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + + off_r = tl.arange(0, BLOCK_R) + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, HEAD_DIM) + off_dp = tl.arange(0, HEAD_DIM // 2) + off_s = tl.arange(0, SCALE_COLS) + row_t = off_r // GROUP + row_h = off_r % GROUP + r_mask = off_r < QLEN * GROUP + tok = pid_r * QLEN + row_t + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + pid_r) + # Token t of the request sits at seq_len - QLEN + t and attends positions [0, kv_len_t). + kv_len = tl.maximum(seq_len - QLEN + row_t + 1, 0) + blk = tl.load(t_ptr + (pid_r * QLEN + QLEN - 1) * stride_tn + pid_c * stride_tk) + blk_ok = blk >= 0 + blk = tl.where(blk_ok, blk, 0) + page = tl.load(block_table_ptr + pid_r * stride_bt_b + blk, mask=blk_ok, other=0).to(tl.int64) + # Rows past the last token's position are never read: their cache bytes may be stale (0 * NaN in the PV dot). + n_valid = tl.where( + blk_ok, tl.minimum(tl.maximum(seq_len - blk * BLOCK_SIZE_K, 0), BLOCK_SIZE_K), 0 + ) + n_mask = off_n < n_valid + + q = tl.load( + q_ptr + tok[:, None] * stride_qn + row_h[:, None] * stride_qh + off_d[None, :], + mask=r_mask[:, None], + other=0.0, + ).to(tl.float16) + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + page * stride_k_blk + off_n[:, None] * stride_k_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + page * stride_ksc_blk + off_n[:, None] * SCALE_COLS + off_s[None, :], + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ) + valid = ( + (blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None]) & n_mask[None, :] & r_mask[:, None] + ) + qk = tl.dot(q, tl.trans(k_deq).to(tl.float16)) * sm_scale_log2e + qk = tl.where(valid, qk, float("-inf")) + m = tl.max(qk, axis=1) + m_safe = tl.where(m == float("-inf"), 0.0, m) + p = tl.exp2(qk - m_safe[:, None]) + p_sum = tl.sum(p, axis=1) + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + page * stride_v_blk + off_n[:, None] * stride_v_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ).to(tl.float16) + acc = tl.dot(p.to(tl.float16), v) + # An empty partial stores zero with LSE -inf, so the merge gives it weight 0. + scale = tl.where(p_sum > 0.0, 1.0 / p_sum, 0.0) * v_global_scale + lse = tl.where(p_sum > 0.0, m + tl.log2(p_sum), float("-inf")) + acc = acc * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + tok[:, None] * stride_o_b + + row_h[:, None] * stride_o_h + + off_d[None, :], + acc, + mask=r_mask[:, None], + ) + tl.store(lse_ptr + pid_c * stride_l_c + tok * stride_l_b + row_h * stride_l_h, lse, mask=r_mask) + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + *, + sm_scale, + decode_query_len, + use_pdl, + early_trigger, + num_warps=8, + num_stages=1, +): + """Launch the multi-query producer; arguments as minimax_m3_sparse_attn_decode after its NVFP4 views.""" + total_q, num_heads, head_dim = q.shape + num_reqs = total_q // decode_query_len + max_topk = int(topk_idx.shape[-1]) + rows = decode_query_len * num_heads + pdl_launch = {"launch_pdl": True} if use_pdl else {} + _gqa_sparse_decode_mq_kernel[(num_reqs, max_topk)]( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + num_reqs, + sm_scale, + q.stride(0), + q.stride(1), + k_paged.stride(0), + k_paged.stride(2), + v_paged.stride(0), + v_paged.stride(2), + k_block_scale.stride(0), + v_block_scale.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + o_arg.stride(0), + o_arg.stride(1), + o_arg.stride(2), + lse_arg.stride(0), + lse_arg.stride(1), + lse_arg.stride(2), + block_table.stride(0), + QLEN=decode_query_len, + GROUP=num_heads, + BLOCK_R=triton.next_power_of_2(rows), + HEAD_DIM=head_dim, + BLOCK_SIZE_K=int(k_paged.shape[2]), + SCALE_COLS=head_dim // 16, + USE_PDL=use_pdl, + EARLY_TRIGGER=early_trigger, + num_warps=num_warps, + num_stages=num_stages, + **pdl_launch, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/weights.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/weights.py new file mode 100644 index 000000000000..870f60035f74 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp8/weights.py @@ -0,0 +1,928 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight tables and loaders: MiniMax-M3.1 (MXFP8 + packed NVFP4 experts) / sm_103 / tp8. + +Two tables, one per checkpoint the target reads: + +* ``MODEL_WEIGHTS`` (``MiniMaxM3Tp8Weights``): the vision-language checkpoint's language model, + ``language_model.*``. Every rank is handed the whole checkpoint, so every tensor-parallel split is this + table's: the Q heads, the K/V/index-Q heads (4 each over 8 ranks: rank r holds head r // 2, as the built-in + loader duplicates them), the dense MLPs and the shared expert's + intermediate, the o_proj / down-projection input columns, every routed expert's intermediate (all 128 + experts live on every rank), and the vocabulary of the shell's ``lm_head``. The index-K head is replicated. +* ``DRAFT_WEIGHTS`` (``MiniMaxM3Tp8DraftWeights``): the one-layer BF16 Eagle3 drafter's own checkpoint + (``speculative_model``), loaded through the shell's ``load_draft_weights`` when speculation is on. + +Storage is HF ``[out, in]`` row-major. Five families need a transform, each a layout a kernel demands: + +* **MXFP8 block scales** (every attention projection, the dense MLPs and the shared expert): the checkpoint + stores UE8M0 bytes ``[N, K / 32]``; the GEMMs read the 128x4-swizzled order + (``trtllm::block_scale_interleave``). A fused projection concatenates its shards' scales first and swizzles + once. +* **The five-way sparse-layer projection** ``[Q | K | V | index-Q | index-K]`` and the three-way dense one + ``[Q | K | V]``: the per-rank shards concatenated along N. +* **Gated MLPs** (dense and shared expert): ``[gate ; up]`` rows. +* **NVFP4 routed experts** (layers 3..59): ``[up (w3) ; gate (w1)]`` rows, then the trtllm-gen row shuffle of + the weight and of its E4M3 scales (gated-activation reorder + 128-row epilogue tile shuffle, + ``trtllmgen_maybe_get_cached_*_permute_indices``) and the 128x4 swizzle of the scales. +* **Gemma boundary norms**: ``input_layernorm``, ``post_attention_layernorm`` and ``model.norm`` are stored as + Gemma norms, ``(1 + w) * x``; the fused all-reduce + RMSNorm kernels apply ``w * x``, so ``1`` is added + (in BF16) at load. The per-head Q/K/index norms stay Gemma: their kernels apply the offset themselves. + +The routed experts' per-tensor scalars (input scales and second-level weight scales, every expert) load raw; +``modeling`` derives the kernels' global scales, alphas and activation constants from them once the weights +are real. + +Loading contract: ``load(model, weights)`` consumes the engine-provided checkpoint dict, fills every declared +parameter exactly once and asserts coverage in both directions: every declared parameter written, and every +checkpoint key either consumed or in ``expected_unconsumed`` (the vision tower and its projector, which this +text-only target does not build, and the language model's multi-token-prediction head). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn + +from tensorrt_llm._torch.models.checkpoints.hf.minimaxm3_weight_mapper import ( + reciprocal_global_scale, +) +from tensorrt_llm._torch.moe.fused_moe.quantization import ( + trtllmgen_maybe_get_cached_w2_permute_indices, + trtllmgen_maybe_get_cached_w3_w1_permute_indices, +) + +#: ``layers=ALL_LAYERS`` is the common case: a weight every decoder layer carries. +ALL_LAYERS = object() + +_LM = "language_model." +#: Checkpoint key families this target does not build: the vision tower and its two projectors (text only), and the +#: language model's multi-token-prediction head (speculation, when on, is the separate Eagle3 drafter). +_UNBUILT_PREFIXES = ( + "vision_tower.", + "multi_modal_projector.", + "patch_merge_mlp.", + _LM + "model.mtp.", +) + +MX_BLOCK = 32 # MXFP8: one UE8M0 scale per 32 elements along K +NV_BLOCK = 16 # NVFP4: one E4M3 scale per 16 elements along K +EPILOGUE_TILE_M = ( + 128 # the trtllm-gen MoE runner's row tile: fixes its weight and scale row shuffles +) + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + ``shape`` takes the dimension bundle and returns the declared shape. ``src`` is a callable + ``(d, i) -> [(key or key tuple, index into the param | None), ...]`` with keys relative to the checkpoint + root; a key tuple hands the transform those tensors together. ``layers`` is ``ALL_LAYERS``, ``None`` (not + per-layer) or ``d -> layer indices``. ``transform(src, d)`` runs on the destination's device before the copy. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + +# ------------------------------------------------------------------------------------------------ shards + + +def fused_projection(sparse_config) -> bool: + """Whether the sparse layers run the five-way packed QKV + index projection. + + On by default: this target requires the MSA implementation and an FP8 indexer, which the packed projection needs. + An explicit ``fuse_qkv_index_projection`` in the deployment's sparse attention config wins, so setting it to False + is the kill switch (the QKV projection plus a separate MXFP8 index projection). + """ + if sparse_config is None: + return False + # A config that does not record which fields were set (a plain namespace or dataclass) is taken at its word. + fields_set = getattr(sparse_config, "model_fields_set", None) + if fields_set is None or "fuse_qkv_index_projection" in fields_set: + return bool(getattr(sparse_config, "fuse_qkv_index_projection", False)) + return True + + +def _col( + t: torch.Tensor, d: SimpleNamespace, tp: int | None = None, rank: int | None = None +) -> torch.Tensor: + """This rank's rows (column-parallel output split), even split, sliced on the host and moved to the + destination device (only this rank's bytes are read).""" + tp = d.tp if tp is None else tp + rank = d.rank if rank is None else rank + n = t.shape[0] // tp + return _dev(t[rank * n : (rank + 1) * n], d) + + +def _heads(t: torch.Tensor, d: SimpleNamespace, head_dim: int) -> torch.Tensor: + """This rank's rows of a projection split by heads of ``head_dim`` rows: an even split when there are at least as + many heads as ranks, else the head of rank // (ranks per head), duplicated on those ranks.""" + n = t.shape[0] // head_dim + if n >= d.tp: + return _col(t, d) + h = d.rank // (d.tp // n) + return _dev(t[h * head_dim : (h + 1) * head_dim], d) + + +def _row(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """This rank's input columns (row-parallel input split), even split, on the destination device.""" + n = t.shape[1] // d.tp + return _dev(t[:, d.rank * n : (d.rank + 1) * n], d) + + +def _dev(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """``t`` on the destination device (``d.dev``, set per fill).""" + return t.to(d.dev, non_blocking=True) + + +def _swizzle(scale: torch.Tensor) -> torch.Tensor: + """128x4 swizzle of a row-major ``[N, K / 32]`` UE8M0 scale matrix, flat.""" + return torch.ops.trtllm.block_scale_interleave(scale.contiguous()) + + +# --------------------------------------------------------------------------------------------- transforms + + +def _gemma_fold(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A Gemma boundary norm as the plain norm the fused all-reduce epilogue applies: ``w + 1`` in BF16.""" + return t + 1.0 + + +def _f32(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return t.to(torch.float32) + + +def _divisor(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A stored global scale (multiplier) -> the modelopt divisor: its FP32 reciprocal.""" + return reciprocal_global_scale("expert global scale", t) + + +def _qkv_sparse_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[Q | K | V | index-Q | index-K]``: Q by attention heads, K/V/index-Q by their heads, index-K replicated.""" + q, k, v, iq, ik = t + hd, xd = d.head_dim, d.index_dim + return torch.cat( + [_col(q, d), _heads(k, d, hd), _heads(v, d, hd), _heads(iq, d, xd), _dev(ik, d)], dim=0 + ).contiguous() + + +def _qkv_sparse_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v, iq, ik = t + hd, xd = d.head_dim, d.index_dim + return _swizzle( + torch.cat( + [_col(q, d), _heads(k, d, hd), _heads(v, d, hd), _heads(iq, d, xd), _dev(ik, d)], dim=0 + ) + ) + + +def _qkv_dense_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + hd = d.head_dim + return torch.cat([_col(q, d), _heads(k, d, hd), _heads(v, d, hd)], dim=0).contiguous() + + +def _qkv_dense_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + hd = d.head_dim + return _swizzle(torch.cat([_col(q, d), _heads(k, d, hd), _heads(v, d, hd)], dim=0)) + + +def _index_qk_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[index-Q | index-K]`` as the checkpoint's MXFP8 (E4M3) values: this rank's index-Q head (rank r holds head + r // 2, as the five-way projection's ``_heads`` split) and the replicated index-K, the same rows it packs.""" + iq, ik = t + return torch.cat([_heads(iq, d, d.index_dim), _dev(ik, d)], dim=0).contiguous() + + +def _index_qk_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """The UE8M0 block scales of ``_index_qk_w``, 128x4-swizzled as every MXFP8 projection's.""" + iq, ik = t + return _swizzle(torch.cat([_heads(iq, d, d.index_dim), _dev(ik, d)], dim=0)) + + +def _gate_up_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return torch.cat([_col(gate, d), _col(up, d)], dim=0).contiguous() + + +def _gate_up_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return _swizzle(torch.cat([_col(gate, d), _col(up, d)], dim=0)) + + +def _row_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _row(t, d).contiguous() + + +def _row_s(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _swizzle(_row(t, d)) + + +def _col_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _col(t, d).contiguous() + + +#: Permute indices of the trtllm-gen expert relayouts, keyed by (shape, family, scale group): one expert's are +#: every expert's, so each is computed once per load. +_PERMUTE_CACHE: dict = {} + + +def _expert_w31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 operand: ``[up ; gate]`` packed-FP4 rows of this rank, then the gated-activation row + reorder and the epilogue-tile shuffle.""" + up, gate = t + w = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + w, _PERMUTE_CACHE, EPILOGUE_TILE_M, is_gated_act_gemm=True + ) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 E4M3 scales: ``[up ; gate]`` rows of this rank, the same row reorder at scale + granularity, then the 128x4 swizzle.""" + up, gate = t + s = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK, is_gated_act_gemm=True + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled.view(torch.uint8).reshape(shape)) + .view(torch.float8_e4m3fn) + .reshape(shape) + ) + + +def _expert_w2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 operand: this rank's input columns, then the epilogue-tile row shuffle.""" + w = _row(t, d).contiguous() + perm = trtllmgen_maybe_get_cached_w2_permute_indices(w, _PERMUTE_CACHE, EPILOGUE_TILE_M) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 E4M3 scales: this rank's columns, the row shuffle at scale granularity, the swizzle.""" + s = _row(t, d).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w2_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled).view(torch.float8_e4m3fn).reshape(shape) + ) + + +# ------------------------------------------------------------------------------------------------ sources + + +def _one(template: str): + return lambda d, i, t=template: [(_LM + t.format(i=i), None)] + + +def _many(*templates: str): + return lambda d, i, ts=templates: [(tuple(_LM + t.format(i=i) for t in ts), None)] + + +def _per_expert(*suffixes: str): + """One source per routed expert, written at its expert index (all experts are local at tp8).""" + + def build(d, i, sx=suffixes): + out = [] + for e in range(d.num_experts): + keys = tuple(_LM + f"model.layers.{i}.block_sparse_moe.experts.{e}.{s}" for s in sx) + out.append((keys if len(keys) > 1 else keys[0], (e,))) + return out + + return build + + +def _dense_mlp_layers(d): + """Layers with the dense SwiGLU-OAI MLP (``moe_layer_freq`` 0).""" + return d.dense_mlp + + +def _moe_layers(d): + return d.moe + + +def _sparse_attn_layers(d): + """Layers with MSA sparse attention (``sparse_attention_freq`` 1): every layer of MiniMax-M3.1.""" + return d.sparse_attn + + +def _three_way_qkv_layers(d): + """Layers whose QKV is the three-way [Q | K | V]: the dense-attention layers, and every layer without the + five-way packed projection (``fuse_qkv_index_projection=False``).""" + return d.dense_attn if d.fused else range(d.num_layers) + + +def _five_way_qkv_layers(d): + return d.sparse_attn if d.fused else range(0) + + +def _unfused_sparse_layers(d): + return range(0) if d.fused else d.sparse_attn + + +_ATTN = "model.layers.{i}.self_attn" +_MOE = "model.layers.{i}.block_sparse_moe" + + +class MiniMaxM3Tp8Weights: + """MiniMax-M3 NVFP4 / sm_103 / tp8: the language model's table and the passes over it. + + ``declare`` allocates every weight it names, ``manifest`` maps checkpoint keys onto them, ``load`` fills + them; ``_targets`` is the one place that decides which parameters exist. + """ + + #: Free the per-expert relayout scratch after each expert stack. + RELEASE_AFTER = "_e_s2" + + WEIGHTS: tuple[W, ...] = ( + # --- the two boundary norms of every layer (Gemma, folded) --- + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.input_layernorm.weight"), + transform=_gemma_fold, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.post_attention_layernorm.weight"), + transform=_gemma_fold, + ), + # --- attention: per-head Q/K norms (Gemma, unfolded), the packed QKV and the output projection --- + W("q_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".q_norm.weight")), + W("k_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".k_norm.weight")), + W( + "qkv_w", + shape=lambda d: (d.qkv_dense_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in "qkv")), + transform=_qkv_dense_w, + layers=_three_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_dense_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many(*(f"{_ATTN}.{p}_proj.weight_scale_inv" for p in "qkv")), + transform=_qkv_dense_s, + layers=_three_way_qkv_layers, + ), + W( + "qkv_w", + shape=lambda d: (d.qkv_sparse_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in ("q", "k", "v", "index_q", "index_k"))), + transform=_qkv_sparse_w, + layers=_five_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_sparse_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + *( + f"{_ATTN}.{p}_proj.weight_scale_inv" + for p in ("q", "k", "v", "index_q", "index_k") + ) + ), + transform=_qkv_sparse_s, + layers=_five_way_qkv_layers, + ), + # The separate index projection [index-Q | index-K] of the unfused sparse layers: the checkpoint's MXFP8 + # values with their block scales (the rows of the five-way projection's index part), run as an MXFP8 GEMM. + W( + "idx_w", + shape=lambda d: (d.index_heads * d.index_dim + d.index_dim, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(_ATTN + ".index_q_proj.weight", _ATTN + ".index_k_proj.weight"), + transform=_index_qk_w, + layers=_unfused_sparse_layers, + ), + W( + "idx_s", + shape=lambda d: ((d.index_heads * d.index_dim + d.index_dim) * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _ATTN + ".index_q_proj.weight_scale_inv", _ATTN + ".index_k_proj.weight_scale_inv" + ), + transform=_index_qk_s, + layers=_unfused_sparse_layers, + ), + W( + "idx_q_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_q_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "idx_k_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_k_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "o_w", + shape=lambda d: (d.hidden, d.q_width), + dtype=torch.float8_e4m3fn, + src=_one(_ATTN + ".o_proj.weight"), + transform=_row_w, + ), + W( + "o_s", + shape=lambda d: (d.hidden * (d.q_width // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_ATTN + ".o_proj.weight_scale_inv"), + transform=_row_s, + ), + # --- dense SwiGLU-OAI MLP (layers 0..2) --- + W( + "mlp_gu_w", + shape=lambda d: (2 * d.dense_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight", "model.layers.{i}.mlp.up_proj.weight" + ), + transform=_gate_up_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_gu_s", + shape=lambda d: (2 * d.dense_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight_scale_inv", + "model.layers.{i}.mlp.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_w", + shape=lambda d: (d.hidden, d.dense_inter), + dtype=torch.float8_e4m3fn, + src=_one("model.layers.{i}.mlp.down_proj.weight"), + transform=_row_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_s", + shape=lambda d: (d.hidden * (d.dense_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one("model.layers.{i}.mlp.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_dense_mlp_layers, + ), + # --- MoE (layers 3..59): the fp32 router and its bias, the shared expert --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + dtype=torch.float32, + src=_one(_MOE + ".gate.weight"), + transform=_f32, + layers=_moe_layers, + ), + W( + "router_bias", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_one(_MOE + ".e_score_correction_bias"), + transform=_f32, + layers=_moe_layers, + ), + W( + "sh_gu_w", + shape=lambda d: (2 * d.shared_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + _MOE + ".shared_experts.gate_proj.weight", _MOE + ".shared_experts.up_proj.weight" + ), + transform=_gate_up_w, + layers=_moe_layers, + ), + W( + "sh_gu_s", + shape=lambda d: (2 * d.shared_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _MOE + ".shared_experts.gate_proj.weight_scale_inv", + _MOE + ".shared_experts.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_moe_layers, + ), + W( + "sh_dn_w", + shape=lambda d: (d.hidden, d.shared_inter), + dtype=torch.float8_e4m3fn, + src=_one(_MOE + ".shared_experts.down_proj.weight"), + transform=_row_w, + layers=_moe_layers, + ), + W( + "sh_dn_s", + shape=lambda d: (d.hidden * (d.shared_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_MOE + ".shared_experts.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_moe_layers, + ), + # --- the routed experts, NVFP4 (trtllm-gen layouts), all 128 on every rank --- + W( + "e_w31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // 2), + dtype=torch.uint8, + src=_per_expert("w3.weight_packed", "w1.weight_packed"), + transform=_expert_w31, + layers=_moe_layers, + ), + W( + "e_w2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // 2), + dtype=torch.uint8, + src=_per_expert("w2.weight_packed"), + transform=_expert_w2, + layers=_moe_layers, + ), + W( + "e_s31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w3.weight_scale", "w1.weight_scale"), + transform=_expert_s31, + layers=_moe_layers, + ), + W( + "e_s2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w2.weight_scale"), + transform=_expert_s2, + layers=_moe_layers, + ), + # The per-tensor scalars of every expert: the checkpoint stores the multipliers into the quantized range + # (``*_global_scale``); the modelopt divisors modeling derives the kernels' scalars from are their FP32 + # reciprocals, taken by the same helper as the built-in loader. + *( + W( + f"e_{name}", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_per_expert(suffix), + transform=_divisor, + layers=_moe_layers, + ) + for name, suffix in ( + ("isc1", "w1.input_global_scale"), + ("isc3", "w3.input_global_scale"), + ("isc2", "w2.input_global_scale"), + ("ws2_1", "w1.weight_global_scale"), + ("ws2_3", "w3.weight_global_scale"), + ("ws2_2", "w2.weight_global_scale"), + ) + ), + # --- not per-layer --- + W( + "embed", + shape=lambda d: (d.vocab, d.hidden), + src=_one("model.embed_tokens.weight"), + layers=None, + ), + W( + "final_norm", + shape=lambda d: (d.hidden,), + src=_one("model.norm.weight"), + transform=_gemma_fold, + layers=None, + ), + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + mapping = core.model_config.mapping + sparse = cfg.sparse_attention_config + freq = [int(f) for f in sparse["sparse_attention_freq"]] + moe_freq = [ + int(f) for f in getattr(cfg, "moe_layer_freq", None) or [1] * cfg.num_hidden_layers + ] + tp = mapping.tp_size + head_dim = int(cfg.head_dim) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + index_heads = max(int(sparse["sparse_num_index_heads"]) // tp, 1) + index_dim = int(sparse["sparse_index_dim"]) + return SimpleNamespace( + fused=fused_projection(core.model_config.sparse_attention_config), + num_layers=cfg.num_hidden_layers, + sparse_attn=[i for i, f in enumerate(freq) if f], + dense_attn=[i for i, f in enumerate(freq) if not f], + moe=[i for i, f in enumerate(moe_freq) if f], + dense_mlp=[i for i, f in enumerate(moe_freq) if not f], + hidden=cfg.hidden_size, + head_dim=head_dim, + heads=heads, + kv_heads=kv_heads, + index_heads=index_heads, + index_dim=index_dim, + q_width=heads * head_dim, + qkv_dense_rows=(heads + 2 * kv_heads) * head_dim, + qkv_sparse_rows=(heads + 2 * kv_heads) * head_dim + index_heads * index_dim + index_dim, + dense_inter=int(cfg.dense_intermediate_size) // tp, + shared_inter=int(cfg.shared_intermediate_size) * int(cfg.n_shared_experts) // tp, + moe_inter=int(cfg.intermediate_size) // tp, + num_experts=int(cfg.num_local_experts), + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None)] + return [(f"l{i}_{entry.name}", i) for i in layers] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares (meta under the engine's meta init).""" + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, core) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + """param key -> [(checkpoint key or key tuple, index into the param | None, transform | None)].""" + d = self.dims(core) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, core, weights) -> set: + """The vision tower and its projectors, and the multi-token-prediction head: in the checkpoint, never built + by this target.""" + return {k for k in weights.keys() if k.startswith(_UNBUILT_PREFIXES)} + + def load(self, model, weights) -> None: + """Fill every declared weight and the shell's ``lm_head`` shard; assert coverage both ways.""" + core = model.model + d = self.dims(core) + manifest = self.manifest(core) + assert set(manifest) == set(core.w.keys()), ( + "manifest/parameter drift", + set(manifest) ^ set(core.w.keys()), + ) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(core, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(core.w[param_key], ckpt, index, transform, weights, d, consumed) + if param_key.endswith(self.RELEASE_AFTER): + torch.cuda.empty_cache() + # The shell owns the vocab-parallel head: this rank's rows of the full table. + _fill(model.lm_head.weight, _LM + "lm_head.weight", None, _col_w, weights, d, consumed) + shell = { + n for n, _ in model.named_parameters() if not n.startswith(("model.w.", "draft_model.")) + } + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + "checkpoint coverage: leftover keys are not exactly the unbuilt families (vision, MTP head); " + f"unexpected {sorted(leftover - expected)[:8]}, missed {sorted(expected - leftover)[:8]}" + ) + + +def _materialize(entry) -> torch.Tensor: + """``[:]`` on a lazy safetensors slice; a 0-dim tensor (the per-tensor scalars) rejects that index.""" + return entry if getattr(entry, "ndim", 1) == 0 else entry[:] + + +def _fill(param, ckpt, index, transform, weights, d, consumed) -> None: + keys = ckpt if isinstance(ckpt, tuple) else (ckpt,) + for key in keys: + assert key in weights, f"checkpoint key missing: {key}" + dst = param.data if index is None else param.data[index] + d.dev = dst.device + # Host tensors: a transform slices this rank's part on the host and moves only that to the device. + src_all = tuple(_materialize(weights[key]) for key in keys) + if transform is not None: + src = transform(src_all if isinstance(ckpt, tuple) else src_all[0], d) + else: + assert len(src_all) == 1, "a multi-key row needs a transform" + src = src_all[0] + if src.dim() == 0 and dst.dim() == 0: + pass + elif src.numel() == 1 and dst.numel() == 1: + src = src.reshape(dst.shape) + src = src.to(dst.device, non_blocking=True) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(keys) + _release(weights, keys) + + +def _release(weights, keys) -> None: + """Drop consumed checkpoint tensors from the engine's dict as the load goes (bounded host memory).""" + if hasattr(weights, "mark_consumed_keys"): + weights.mark_consumed_keys(keys) + else: + for key in keys: + weights.pop(key, None) + + +MODEL_WEIGHTS = MiniMaxM3Tp8Weights() + + +# -------------------------------------------------------------------------------------------- the drafter + + +def _draft_one(template: str): + return lambda d, i, t=template: [(t, None)] + + +def _draft_many(*templates: str): + return lambda d, i, ts=templates: [(tuple(ts), None)] + + +def _draft_fc_norms(d): + return range(d.capture_layers) + + +def _draft_qkv(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + hd = d.head_dim + return torch.cat([_col(q, d), _heads(k, d, hd), _heads(v, d, hd)], dim=0).contiguous() + + +class MiniMaxM3Tp8DraftWeights: + """The one-layer BF16 Eagle3 drafter (``speculative_model``), tp8. + + The fc over the three captured target layers and its per-layer fc_norms, the layer's three norms, its QKV + (input ``[embeds | hidden]``, so twice the hidden width; Q by heads, K/V by KV heads, each KV head on two + ranks) and o_proj, its gated MLP, the final norm and the full-vocabulary head (vocab-parallel). The + checkpoint's ``embed_tokens`` is the target's table (the drafter reads the target's); it is a predicted non-load. + """ + + WEIGHTS: tuple[W, ...] = ( + W( + "fc", + shape=lambda d: (d.hidden, d.capture_layers * d.hidden), + src=_draft_one("fc.weight"), + layers=None, + ), + W( + "fc_norm", + shape=lambda d: (d.hidden,), + src=lambda d, i: [(f"fc_norm.{i}.weight", None)], + layers=_draft_fc_norms, + ), + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.input_layernorm.weight"), + layers=None, + ), + W( + "hidden_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.hidden_norm.weight"), + layers=None, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.post_attention_layernorm.weight"), + layers=None, + ), + W( + "qkv", + shape=lambda d: (d.qkv_rows, 2 * d.hidden), + src=_draft_many(*(f"layers.0.self_attn.{p}_proj.weight" for p in "qkv")), + transform=_draft_qkv, + layers=None, + ), + W( + "o", + shape=lambda d: (d.hidden, d.q_width), + src=_draft_one("layers.0.self_attn.o_proj.weight"), + transform=_row_w, + layers=None, + ), + W( + "gu", + shape=lambda d: (2 * d.inter, d.hidden), + src=_draft_many("layers.0.mlp.gate_proj.weight", "layers.0.mlp.up_proj.weight"), + transform=_gate_up_w, + layers=None, + ), + W( + "dn", + shape=lambda d: (d.hidden, d.inter), + src=_draft_one("layers.0.mlp.down_proj.weight"), + transform=_row_w, + layers=None, + ), + W("final_norm", shape=lambda d: (d.hidden,), src=_draft_one("norm.weight"), layers=None), + ) + + def dims(self, draft) -> SimpleNamespace: + cfg = draft.draft_config.pretrained_config + mapping = draft.draft_config.mapping + tp = mapping.tp_size + head_dim = int(getattr(cfg, "head_dim", cfg.hidden_size // cfg.num_attention_heads)) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + return SimpleNamespace( + hidden=cfg.hidden_size, + capture_layers=draft.capture_layers, + head_dim=head_dim, + q_width=heads * head_dim, + qkv_rows=(heads + 2 * kv_heads) * head_dim, + inter=cfg.intermediate_size // tp, + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + if entry.layers is None: + return [(entry.name, None)] + return [(f"{entry.name}{i}", i) for i in entry.layers(d)] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, draft) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + d = self.dims(draft) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, draft, weights) -> set: + """The drafter checkpoint's own embedding: the drafter reads the target's table.""" + return {"embed_tokens.weight"} & set(weights.keys()) + + def load(self, draft, weights) -> None: + d = self.dims(draft) + manifest = self.manifest(draft) + assert set(manifest) == set(draft.w.keys()), set(manifest) ^ set(draft.w.keys()) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(draft, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(draft.w[param_key], ckpt, index, transform, weights, d, consumed) + _fill(draft.lm_head.weight, "lm_head.weight", None, _col_w, weights, d, consumed) + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + f"drafter checkpoint coverage: unexpected {sorted(leftover - expected)[:8]}, " + f"missed {sorted(expected - leftover)[:8]}" + ) + + +DRAFT_WEIGHTS = MiniMaxM3Tp8DraftWeights() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py new file mode 100644 index 000000000000..03977da8e8af --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py @@ -0,0 +1,206 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Route the MiniMax-M3.1 sm_103 TP4/TP8 targets; other shapes decline.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Optional, Tuple + +from tensorrt_llm.quantization.mode import QuantAlgo + +from ..._router_index import NoTarget + +TEXT_ONLY = True +_SM = (10, 3) +_SHAPE = (60, 6144, 128, 3072, 64, 4) +_M31_ROPE_THETA = 10_000_000 +_M31_SPARSE_ATTENTION_FREQ = (1,) * 60 +_M31_EXPERTS = ("NVFP4", "nvfp4-pack-quantized") +_M31_DSPARK_ARCH = "DSparkMiniMaxDraftModel" +_M31_DSPARK_GEOMETRY = dict( + hidden_size=6144, vocab_size=200064, num_key_value_heads=4, head_dim=128 +) +_M31_DSPARK_TARGET_LAYERS = [3, 17, 31, 45, 59] + + +def _field(config: Any, name: str) -> Any: + """Read ``name`` off a sub-config that may be an object or a plain dict.""" + if isinstance(config, dict): + return config.get(name) + return getattr(config, name, None) + + +def _rope_theta(text_config: Any) -> Any: + theta = _field(text_config, "rope_theta") + if theta is None: + rope = _field(text_config, "rope_parameters") or {} + theta = _field( + rope.get("full_attention", rope) if isinstance(rope, dict) else rope, "rope_theta" + ) + return theta + + +def _m31_mismatch( + pretrained_config: Any, text_config: Any, quant_algo: Optional[QuantAlgo] +) -> Optional[str]: + """Why the checkpoint is not the MiniMax-M3.1 export the m31 targets build, or None.""" + if text_config is None: + return "no text_config" + reasons = [] + theta = _rope_theta(text_config) + if theta != _M31_ROPE_THETA: + reasons.append(f"rope_theta {theta}, not {_M31_ROPE_THETA}") + sparse = _field(text_config, "sparse_attention_config") + freq = None if sparse is None else _field(sparse, "sparse_attention_freq") + enabled = sparse is not None and _field(sparse, "use_sparse_attention") is not False + if not enabled or freq is None or tuple(int(f) for f in freq) != _M31_SPARSE_ATTENTION_FREQ: + reasons.append("sparse attention not on every layer") + declared = getattr(pretrained_config, "quantization_config", None) or {} + experts = (_field(declared, "moe_quant_algo"), _field(declared, "moe_quant_format")) + if quant_algo != QuantAlgo.MXFP8 or experts != _M31_EXPERTS: + reasons.append( + f"quantization {quant_algo} with experts {experts}, not MXFP8 with {_M31_EXPERTS}" + ) + return "; ".join(reasons) if reasons else None + + +def _quant_algo(pretrained_config: Any) -> Optional[QuantAlgo]: + """The quantization the checkpoint's config declares, as the engine reads it. + + The NVFP4 checkpoint is a modelopt mixed-precision export (NVFP4 routed + experts, MXFP8 dense projections, bf16 router, embedding and head), declared + in ``config.json`` as ``MIXED_PRECISION``; the per-layer split lives in + ``hf_quant_config.json``, which ``route`` does not see. + """ + from tensorrt_llm._torch.model_config import ModelConfig + + declared = getattr(pretrained_config, "quantization_config", None) + if declared is None: + return None + try: + quant_config, _ = ModelConfig.load_hf_quant_config(declared, "TRTLLM") + except ValueError: + return None + return quant_config.quant_algo + + +def _parallel(m) -> Optional[str]: + """Name the parallel topology, or None if no target implements it. + + Attention is head-split and the routed experts are tensor-parallel with + every expert on every rank: the expert split decides which expert weights + each rank loads and into which shapes (per-rank expert widths 768 at TP4 and 384 at TP8), + so it selects a target rather than a runtime branch. + """ + for tp in (4, 8): + if ( + m.world_size == tp + and m.tp_size == tp + and m.pp_size == 1 + and m.cp_size == 1 + and m.moe_tp_size == tp + and m.moe_ep_size == 1 + and not m.enable_attention_dp + ): + return f"tp{tp}" + return None + + +def _m31_speculation_mismatch(spec_config: Optional[Any]) -> Optional[str]: + """Why the speculation is not one the M3.1 targets serve, or None when it is (none, or a standalone MiniMax DSpark + drafter). Read from the drafter checkpoint's ``config.json``.""" + if spec_config is None: + return None + mode = spec_config.spec_dec_mode + if not (mode.use_one_engine() and mode.is_dspark()): + return f"speculative decoding {mode.name}, not standalone DSpark" + if getattr(spec_config, "draft_is_embedded_in_target", False): + return "a DSpark drafter embedded in the target checkpoint" + path = getattr(spec_config, "speculative_model", None) + if path is None: + return "no DSpark drafter checkpoint (speculative_model)" + try: + with open(Path(path) / "config.json") as f: + cfg = json.load(f) + except (OSError, ValueError) as err: + return f"the DSpark drafter's config.json is not readable ({err})" + text = cfg.get("text_config") or {} + + def field(name): + return text.get(name, cfg.get(name)) + + arches = cfg.get("architectures") or text.get("architectures") or [] + if not arches or arches[0] != _M31_DSPARK_ARCH: + return f"drafter architecture {arches[:1]}, not {_M31_DSPARK_ARCH}" + if list(field("dspark_target_layer_ids") or []) != _M31_DSPARK_TARGET_LAYERS: + return f"drafter target layers {field('dspark_target_layer_ids')}, not {_M31_DSPARK_TARGET_LAYERS}" + for name, want in _M31_DSPARK_GEOMETRY.items(): + if field(name) != want: + return f"drafter {name} {field(name)}, not {want}" + return None + + +def route( + pretrained_config: Any, + mapping: Any, + compute_capability: Tuple[int, int], + spec_config: Optional[Any], + kv_cache_quant_algo: Optional[Any], +) -> type: + """Return the target class for this deployment, or raise ``NoTarget`` saying why none fits. + + ``kv_cache_quant_algo`` is the KV-cache algorithm the engine will run with + (see ``_router_index._kv_cache_quant_algo``), not the checkpoint's alone. + + Each target is imported where it is returned, so declining costs no import + of modeling code. + """ + text_config = getattr(pretrained_config, "text_config", None) + shape = ( + None + if text_config is None + else tuple( + _field(text_config, name) + for name in ( + "num_hidden_layers", + "hidden_size", + "num_local_experts", + "intermediate_size", + "num_attention_heads", + "num_key_value_heads", + ) + ) + ) + quant_algo = _quant_algo(pretrained_config) + parallel = _parallel(mapping) + kv_cache = kv_cache_quant_algo in (QuantAlgo.FP8, QuantAlgo.NVFP4) + m31_miss = _m31_mismatch(pretrained_config, text_config, quant_algo) + m31_spec_miss = _m31_speculation_mismatch(spec_config) if m31_miss is None else None + + if ( + compute_capability == _SM + and shape == _SHAPE + and m31_miss is None + and parallel in ("tp4", "tp8") + and m31_spec_miss is None + and kv_cache + ): + if parallel == "tp4": + from .minimax_m31_nvfp4__sm_103__tp4.modeling import ModelingV2MiniMaxM31Nvfp4Sm103Tp4 + + return ModelingV2MiniMaxM31Nvfp4Sm103Tp4 + from .fake_ops import register + + register() + from .minimax_m31_nvfp4__sm_103__tp8.modeling import ModelingV2MiniMaxM31Nvfp4Sm103Tp8 + + return ModelingV2MiniMaxM31Nvfp4Sm103Tp8 + + raise NoTarget( + f"MiniMax-M3.1 requires sm_103 TP4 or TP8/EP1, shape {_SHAPE}, MXFP8/NVFP4 experts, " + f"FP8 or NVFP4 KV, and standalone MiniMax DSpark or no speculation; " + f"got SM={compute_capability}, shape={shape}, topology={parallel}, " + f"KV={kv_cache_quant_algo}, checkpoint={m31_miss}, speculation={m31_spec_miss}" + ) diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py index 4563331e9754..93af5a432f8c 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py @@ -60,6 +60,37 @@ from tensorrt_llm.llmapi.llm_args import DecodingBaseConfig +class _UnchangedCopy: + """Remembers the inputs of the last decode-only block-table copy into one destination. + + In steady decode the block tables change only when a request grows into a + new page, so a copy whose destination, batch order, decode-only shape and + host block-offset table equal the previous copy's can be skipped. + """ + + def __init__(self) -> None: + self._key: Optional[tuple] = None + self._host: Optional[torch.Tensor] = None + + def skip(self, dst, request_ids, beam_width, num_contexts, num_seqs, max_blocks, host) -> bool: + if num_contexts != 0: + self._key = None + return False + key = (dst.data_ptr(), tuple(request_ids), beam_width, num_seqs, max_blocks) + return key == self._key and self._host is not None and torch.equal(host, self._host) + + def record( + self, dst, request_ids, beam_width, num_contexts, num_seqs, max_blocks, host + ) -> None: + if num_contexts != 0: + return + self._key = (dst.data_ptr(), tuple(request_ids), beam_width, num_seqs, max_blocks) + if self._host is None or self._host.shape != host.shape: + self._host = host.clone() + else: + self._host.copy_(host) + + class MiniMaxM3SparseIndexCache: """Plain-tensor side cache for the M3 sparse index branch. @@ -304,6 +335,10 @@ class MiniMaxM3KVCacheManagerV2(KVCacheManagerV2): own pool. """ + # Decode-step shortcuts a model opts into (RuntimeHooks.msa); see configure_decode_shortcuts. + skip_unchanged_block_offsets = False + reuse_pool_views = False + _main_kv_mapper_kind = MapperKind.NHD # Virtual attention-op pools for shared draft layers: # (attention_op_pool_id, source_storage_pool_id) plus their copy parameters. @@ -370,6 +405,7 @@ def __init__( f"{self.indexer_kv_dtype!r}." ) super().__init__(*args, **kwargs) + self._unchanged_copies: dict[int, _UnchangedCopy] = {} if self.dtype == DataType.NVFP4: dense_target_layers = [ @@ -618,6 +654,20 @@ def copy_batch_block_offsets( num_seqs: int, max_blocks: Optional[int] = None, ) -> None: + if self.skip_unchanged_block_offsets: + guard = self._unchanged_copies.setdefault(dst_tensor.data_ptr(), _UnchangedCopy()) + args = ( + dst_tensor, + request_ids, + beam_width, + num_contexts, + num_seqs, + max_blocks, + self.host_kv_cache_block_offsets, + ) + if guard.skip(*args): + return + guard.record(*args) super().copy_batch_block_offsets( dst_tensor, request_ids, beam_width, num_contexts, num_seqs, max_blocks=max_blocks ) @@ -827,6 +877,20 @@ def _build_kv_buffers( full_view = convert_to_torch_tensor(TensorWrapper(addr_key, torch_dtype, full_slot_shape)) return full_view[:, :2] + def configure_decode_shortcuts( + self, skip_unchanged_block_offsets: bool, reuse_pool_views: bool + ) -> None: + """Turn on the decode-step shortcuts a model opted into. + + ``skip_unchanged_block_offsets``: skip a decode-only block-table copy + whose inputs equal the previous copy's into the same destination. + ``reuse_pool_views``: hand out one pool view per geometry instead of + re-wrapping the pool memory on every attention call. + """ + self.skip_unchanged_block_offsets = skip_unchanged_block_offsets + # The feature base already caches every pool view in _cached_pool_view. + self.reuse_pool_views = reuse_pool_views + def _kv_scale_slot_geometry( self, layer_idx: int, kv_layout: Optional[str] ) -> Tuple[int, torch.dtype, int, int, List[int]]: @@ -1063,6 +1127,7 @@ def __init__(self, manager: MiniMaxM3KVCacheManagerV2, draft_layer_ids: Sequence f"got {len(draft_layer_ids)}. Multiple layers require separate pool roots." ) self._manager = manager + self._unchanged_copy = _UnchangedCopy() layer_id = draft_layer_ids[0] local = int(manager.layer_offsets[layer_id]) self._local_layer_idx = local @@ -1127,6 +1192,19 @@ def copy_batch_block_offsets( """ assert beam_width == 1, "beam_width must be 1 for KVCacheManagerV2" manager = self._manager + if manager.skip_unchanged_block_offsets: + args = ( + dst_tensor, + request_ids, + beam_width, + num_contexts, + num_seqs, + max_blocks, + self.host_kv_cache_block_offsets, + ) + if self._unchanged_copy.skip(*args): + return + self._unchanged_copy.record(*args) copy_idx = manager.index_mapper.get_copy_index(request_ids, num_contexts, beam_width) assert copy_idx.shape[0] == num_seqs copy_batch_block_offsets_to_device( diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py index b5022eb77256..fc9ec623cc0e 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py @@ -39,6 +39,9 @@ from tensorrt_llm._torch.attention.backends.interface import AttentionForwardArgs from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention, TrtllmAttentionMetadata +from tensorrt_llm._torch.pyexecutor import h2d_cache, step_inputs +from tensorrt_llm._torch.pyexecutor.h2d_cache import copy_h2d_if_changed +from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy from tensorrt_llm._utils import maybe_pin_memory from tensorrt_llm.bindings import DataType from tensorrt_llm.models.modeling_utils import QuantConfig @@ -103,6 +106,17 @@ def _worst_case_proxy_max_k_tiles( return int(proxy_plan[3]["max_k_tiles"]) +def _host_prefix_sum(lens_cpu: torch.Tensor, dtype: torch.dtype) -> torch.Tensor: + """[0, cumsum(lens)...] as a pinned host tensor of ``dtype``.""" + sums = torch.cumsum(lens_cpu.to(torch.int64), 0) + return maybe_pin_memory(torch.cat([sums.new_zeros(1), sums]).to(dtype)) + + +def _fill_unused_slots(buf: torch.Tensor, start: int) -> None: + """buf[start:] = -1 after a write of the live slots buf[:start] (see h2d_cache.fill_tail_if_changed).""" + h2d_cache.fill_tail_if_changed(buf, start, -1) + + class MsaDecodeSpan(NamedTuple): """Generation rows served by MiniMax-M3's dedicated decode kernels. @@ -159,6 +173,13 @@ class MiniMaxM3MsaSparseAttentionMetadata(TrtllmAttentionMetadata): msa_kv_indices: Optional[torch.Tensor] = None msa_max_score: Optional[torch.Tensor] = None msa_n_valid_blocks: Optional[torch.Tensor] = None + # Decode-step preparation the model opted into (RuntimeHooks.msa). + msa_policy: MsaRuntimePolicy = MsaRuntimePolicy() + # Short-sequence steps (MsaRuntimePolicy.short_sequence_identity): every query token has at most topk valid + # blocks, so the selection is the identity table [kv_heads, tokens, topk] = (i < n_valid ? i : -1), staged here + # by prepare(). _msa_short_ok says whether this step staged it. + msa_short_table: Optional[torch.Tensor] = None + _msa_short_ok: bool = False # The same page table and lengths as msa_kv_indices / msa_kv_lens, in the # per-request 2-D form the decode kernels index directly # (block_table[request, block] and seq_lens[request]). fmha_sm100 instead @@ -202,6 +223,12 @@ class MiniMaxM3MsaSparseAttentionMetadata(TrtllmAttentionMetadata): # _msa_prefill_n_valid_buf is the persistent backing store for the view. _msa_prefill_n_valid_buf: Optional[torch.Tensor] = None _msa_prefill_n_valid_blocks: Optional[torch.Tensor] = None + # Pure-prefill step where every query token has at most topk valid blocks: selection is the identity, so the + # indexer returns this [total_q, topk] table (per KV head) instead of scoring (built once per step). + _msa_prefill_identity: Optional[torch.Tensor] = None + # (identity table, (batch, rows), k2q CSR worklist) the NVFP4 prefill built for it: the worklist is + # layer-invariant for an identity selection, so the later layers of the step reuse it. + _msa_identity_csr: Optional[tuple] = None # This step's per-request host lengths, staged by _stage_host_lengths. _msa_qo_lens_cpu: Optional[torch.Tensor] = None _msa_kv_lens_cpu: Optional[torch.Tensor] = None @@ -644,6 +671,40 @@ def _ensure_prefill_n_valid_buffer(self, total_q: int, device: torch.device) -> self._msa_prefill_n_valid_buf = buf return buf + def _stage_msa_short_table(self, n_valid: torch.Tensor, params) -> None: + """Stage the identity block table for short-sequence steps (see msa_short_table).""" + self._msa_short_ok = False + if ( + not self.msa_policy.short_sequence_identity + or n_valid.numel() == 0 + or self.num_contexts > 0 + ): + return + topk = int(MSA_REQUIRED_TOPK) + if int(n_valid.max()) > topk: + return + _, num_kv_heads = params.sharded_head_counts(self.mapping) + total_q = int(n_valid.shape[0]) + max_tokens = max(int(self.max_num_sequences), self._msa_max_decode_tokens(), total_q) + if ( + self.msa_short_table is None + or self.msa_short_table.numel() < num_kv_heads * max_tokens * topk + ): + self.msa_short_table = self.get_empty( + self.cuda_graph_buffers, + (num_kv_heads * max_tokens * topk,), + cache_name="msa_short_table", + dtype=torch.int32, + capture_graph=self.is_cuda_graph, + ) + ar = torch.arange(topk, dtype=torch.int32) + table = torch.where(ar[None, :] < n_valid.to(torch.int32)[:, None], ar[None, :], -1) + table = table.unsqueeze(0).expand(num_kv_heads, total_q, topk).contiguous() + copy_h2d_if_changed( + self.msa_short_table[: table.numel()], maybe_pin_memory(table.flatten()), pack=True + ) + self._msa_short_ok = True + def prepare(self) -> None: super().prepare() self._check_beam_width() @@ -841,6 +902,29 @@ def on_update_kv_lens(self) -> None: total_q = int(self.num_tokens) if batch <= 0 or total_q <= 0: return + page = int(self.kv_cache_manager.tokens_per_block) + n_valid_buf = ( + self._msa_prefill_n_valid_blocks + if self._msa_prefill_n_valid_blocks is not None + else self.msa_n_valid_blocks + ) + kv_update = self.msa_policy.kv_update + if kv_update is not None and kv_update( + self.kv_lens_cuda, + self.msa_kv_lens_staged, + self.seq_lens_cuda, + self.msa_q_batch_row, + self.msa_q_intra, + self.msa_block_table, + page, + batch, + total_q, + self.msa_seq_lens_cuda, + None if self.msa_cu_kv_lens is None else self.msa_cu_kv_lens[: batch + 1], + self.msa_out_cache_loc, + None if n_valid_buf is None else n_valid_buf[:total_q], + ): + return # kv_lens_cuda is the attended length (no num_extra_kv_tokens), the same # domain as the staged bound and msa_seq_lens_cuda. kv_true = torch.minimum(self.kv_lens_cuda[:batch], self.msa_kv_lens_staged[:batch]) @@ -901,6 +985,8 @@ def _build_step_plans(self) -> None: self._msa_prefill_gqa_plan = None self._msa_prefill_dense_plan = None self._msa_prefill_n_valid_blocks = None + self._msa_prefill_identity = None + self._msa_short_ok = False if not self._msa_fields_ready: return # Geometry is captured in __post_init__; skip when it is unavailable. @@ -931,7 +1017,9 @@ def _build_step_plans(self) -> None: qo_lens_cpu, kv_lens_cpu, qo_offset_cpu, causal=True, block_size=page_size ) total_q = int(n_valid.shape[0]) - self.msa_n_valid_blocks[:total_q].copy_(n_valid.to(torch.int32), non_blocking=True) + # Packed but never skipped: on_update_kv_lens rewrites this buffer on the device. + step_inputs.copy_(self.msa_n_valid_blocks[:total_q], n_valid.to(torch.int32)) + self._stage_msa_short_table(n_valid, params) return fmha_sm100 = require_msa_module() @@ -962,13 +1050,28 @@ def plan_for(rows: Optional[Tuple[int, int]], **plan_kwargs) -> Optional[tuple]: **plan_kwargs, ) + n_valid_host = per_token_valid_blocks( + qo_lens_cpu, kv_lens_cpu, qo_offset_cpu, causal=True, block_size=page_size + ) + total_q = int(n_valid_host.shape[0]) + short_prefill = ( + self.msa_policy.short_sequence_identity + and self._msa_decode_span is None + and total_q > 0 + and int(n_valid_host.max()) <= int(params.topk) + ) # Proxy plan: MQA (num_kv_heads=1) max-score pass over the index # branch; output_maxscore feeds the indexer's top-k block selection. - self._msa_prefill_proxy_plan = plan_for( - self._msa_proxy_plan_rows(), - num_qo_heads=num_index_heads, - num_kv_heads=1, - output_maxscore=True, + # A short pure-prefill step selects every valid block, so it needs none. + self._msa_prefill_proxy_plan = ( + None + if short_prefill + else plan_for( + self._msa_proxy_plan_rows(), + num_qo_heads=num_index_heads, + num_kv_heads=1, + output_maxscore=True, + ) ) attn_rows = self._msa_attn_plan_rows() # Sparse layers: kv_block_num=topk limits attention to top-k blocks. @@ -988,14 +1091,15 @@ def plan_for(rows: Optional[Tuple[int, int]], **plan_kwargs) -> Optional[tuple]: ) # Stage the valid-block count to the device once for the whole step # (see msa_prefill_n_valid_blocks). - n_valid_host = per_token_valid_blocks( - qo_lens_cpu, kv_lens_cpu, qo_offset_cpu, causal=True, block_size=page_size - ) - total_q = int(n_valid_host.shape[0]) if total_q > 0: dev_buf = self._ensure_prefill_n_valid_buffer(total_q, _cache_device(self)) dev_buf[:total_q].copy_(n_valid_host.to(torch.int32), non_blocking=True) self._msa_prefill_n_valid_blocks = dev_buf[:total_q] + if short_prefill: + topk = int(params.topk) + ar = torch.arange(topk, dtype=torch.int32, device=dev_buf.device) + table = torch.where(ar[None, :] < dev_buf[:total_q, None], ar[None, :], -1) + self._msa_prefill_identity = table def _build_msa_fields(self) -> None: """Populate the MSA cache-write buffers for this step. @@ -1013,11 +1117,11 @@ def _build_msa_fields(self) -> None: kv_lens_cpu = self.msa_kv_lens_cpu qo_offset_cpu = self.msa_qo_offset_cpu if request_ids is None or qo_lens_cpu is None: - self.msa_out_cache_loc.fill_(-1) + _fill_unused_slots(self.msa_out_cache_loc, 0) return batch_size = int(qo_lens_cpu.shape[0]) if batch_size == 0: - self.msa_out_cache_loc.fill_(-1) + _fill_unused_slots(self.msa_out_cache_loc, 0) return kv_cache_manager = self.kv_cache_manager @@ -1034,6 +1138,8 @@ def _build_msa_fields(self) -> None: qo_lens_cpu=qo_lens_cpu, qo_offset_cpu=qo_offset_cpu, device=cache_device, + # Only out_cache_loc and the host block ids are read here; stage the slots straight from the host. + device_tables=not h2d_cache.enabled(), ) out_cache_loc = mapping.out_cache_loc # Only fmha_sm100 reads the flattened page table (the decode kernels @@ -1075,41 +1181,62 @@ def _build_msa_fields(self) -> None: f"smaller than the step's per-request page count ({block_table_cols})." ) - self.msa_out_cache_loc[:total_new_tokens].copy_(out_cache_loc, non_blocking=True) + step_inputs.copy_(self.msa_out_cache_loc[:total_new_tokens], out_cache_loc) # Captured producers also execute padded rows: the fused index producer # sits inside the captured region, so trimming it to a host-side count # would make its shape dynamic, and a negative slot is what makes those # rows cache-write no-ops instead. Invalidate the unwritten tail so they # cannot reuse the previous step's live slots, which address real pages. - if total_new_tokens < self.msa_out_cache_loc.shape[0]: - self.msa_out_cache_loc[total_new_tokens:].fill_(-1) + _fill_unused_slots(self.msa_out_cache_loc, total_new_tokens) if kv_indices is not None: - self.msa_kv_indices[: int(kv_indices.shape[0])].copy_(kv_indices, non_blocking=True) + step_inputs.copy_(self.msa_kv_indices[: int(kv_indices.shape[0])], kv_indices) # 2-D page table and per-request length for the decode kernels, # from the same host block ids the flat page table was built from. # Columns past a request's page count are left stale rather than # cleared: every consumer bounds its walk by seq_lens. - self.msa_block_table[:batch_size, :block_table_cols].copy_( - maybe_pin_memory(block_ids_cpu.to(torch.int32)), non_blocking=True + block_table_changed = copy_h2d_if_changed( + self.msa_block_table[:batch_size, :block_table_cols], + maybe_pin_memory(block_ids_cpu.to(torch.int32)), + # write_subpage_block_table below reads it on the device before the replay. + pack=self.msa_subpage_block_table is None, ) - self.msa_seq_lens_cuda[:batch_size].copy_(kv_lens_cpu, non_blocking=True) + # Packed only when the device scan below (h2d_cache off) does not read it before the replay. + if h2d_cache.enabled(): + step_inputs.copy_(self.msa_seq_lens_cuda[:batch_size], kv_lens_cpu) + else: + self.msa_seq_lens_cuda[:batch_size].copy_(kv_lens_cpu, non_blocking=True) if self.msa_cu_q_lens is not None: self._msa_live_batch = batch_size - self.msa_cu_q_lens[0].zero_() - self.msa_cu_kv_lens[0].zero_() qo_lens_cuda = self.msa_qo_lens_cuda[:batch_size] - qo_lens_cuda.copy_(qo_lens_cpu, non_blocking=True) - torch.cumsum( - qo_lens_cuda, - 0, - out=self.msa_cu_q_lens[1 : batch_size + 1], - ) - torch.cumsum( - self.msa_seq_lens_cuda[:batch_size], - 0, - out=self.msa_cu_kv_lens[1 : batch_size + 1], - ) + if h2d_cache.enabled(): + # Both prefix sums from the host lengths (the values the device scans would give), staged with + # their leading zero: copies instead of zero / scan kernels between the decode graphs. + copy_h2d_if_changed(qo_lens_cuda, qo_lens_cpu, pack=True) + copy_h2d_if_changed( + self.msa_cu_q_lens[: batch_size + 1], + _host_prefix_sum(qo_lens_cpu, self.msa_cu_q_lens.dtype), + pack=True, + ) + # Not through the skip cache: on_update_kv_lens rewrites this buffer on the device. + step_inputs.copy_( + self.msa_cu_kv_lens[: batch_size + 1], + _host_prefix_sum(kv_lens_cpu, self.msa_cu_kv_lens.dtype), + ) + else: + self.msa_cu_q_lens[0].zero_() + self.msa_cu_kv_lens[0].zero_() + qo_lens_cuda.copy_(qo_lens_cpu, non_blocking=True) + torch.cumsum( + qo_lens_cuda, + 0, + out=self.msa_cu_q_lens[1 : batch_size + 1], + ) + torch.cumsum( + self.msa_seq_lens_cuda[:batch_size], + 0, + out=self.msa_cu_kv_lens[1 : batch_size + 1], + ) self._msa_max_q_len = int(qo_lens_cpu.max().item()) self._msa_max_kv_len_all = int(kv_lens_cpu.max().item()) self._msa_total_k = int(kv_lens_cpu.to(torch.int64).sum().item()) @@ -1144,7 +1271,7 @@ def _build_msa_fields(self) -> None: # Sub-page expansion for the trtllm-gen dense layers, staged once here # instead of once per layer, outside capture into a graph-stable # buffer as with the slot table above. - if self.msa_subpage_block_table is not None: + if self.msa_subpage_block_table is not None and block_table_changed: write_subpage_block_table( self.msa_block_table[:batch_size], self._msa_subpages_per_slot, @@ -1168,16 +1295,20 @@ def _build_msa_fields(self) -> None: intra_np = ( np.arange(total_new_tokens, dtype=np.int64) - np.repeat(starts_np, qo_np) ).astype(np.int32) - self.msa_q_batch_row[:total_new_tokens].copy_( - maybe_pin_memory(torch.from_numpy(batch_row_np)), non_blocking=True + copy_h2d_if_changed( + self.msa_q_batch_row[:total_new_tokens], + maybe_pin_memory(torch.from_numpy(batch_row_np)), + pack=True, ) - self.msa_q_intra[:total_new_tokens].copy_( - maybe_pin_memory(torch.from_numpy(intra_np)), non_blocking=True + copy_h2d_if_changed( + self.msa_q_intra[:total_new_tokens], + maybe_pin_memory(torch.from_numpy(intra_np)), + pack=True, ) # The staged lens are the upper bound on_update_kv_lens clamps to: the # same attended lengths msa_seq_lens_cuda was just filled from, kept # apart so the clamp stays fixed however often the hook runs. - self.msa_kv_lens_staged[:batch_size].copy_(kv_lens_cpu, non_blocking=True) + step_inputs.copy_(self.msa_kv_lens_staged[:batch_size], kv_lens_cpu) self._msa_fields_ready = True def msa_idx_k_cache(self, layer_idx: int) -> torch.Tensor: @@ -1470,6 +1601,11 @@ def run_indexer( # proxy plan scores the whole batch instead. gen_first is the span's # first query token: the scorer takes [gen_first, num_tokens) over rows # [ctx_rows, row_last), the proxy the context prefix ahead of both. + identity = metadata._msa_prefill_identity + if span is None and identity is not None and int(identity.shape[0]) == num_tokens: + # Short pure-prefill step: every token selects all its valid blocks (staged once per step). + num_kv_heads = int(self.indexer.config.num_kv_heads) + return identity.unsqueeze(1).expand(num_tokens, num_kv_heads, int(identity.shape[1])) block_table = None seq_lens_cuda = None decode_query_len = None diff --git a/tensorrt_llm/_torch/attention/backends/trtllm.py b/tensorrt_llm/_torch/attention/backends/trtllm.py index d29def811bc5..8a262843975e 100644 --- a/tensorrt_llm/_torch/attention/backends/trtllm.py +++ b/tensorrt_llm/_torch/attention/backends/trtllm.py @@ -39,7 +39,9 @@ from tensorrt_llm.math_utils import ceil_div from tensorrt_llm.models.modeling_utils import QuantConfig +from ...pyexecutor import step_inputs from ...pyexecutor.config_utils import is_mla +from ...pyexecutor.h2d_cache import copy_h2d_if_changed from ...utils import (compute_swizzled_sf_shape, get_global_attrs, get_model_extra_attrs, helix_local_len_tensor) from .fmha.manager import FmhaManager @@ -997,8 +999,8 @@ def prepare(self) -> None: device='cpu', ) self.prompt_lens_cpu[:self.num_seqs].copy_(prompt_lens) - self.prompt_lens_cuda[:self.num_seqs].copy_( - self.prompt_lens_cpu[:self.num_seqs], non_blocking=True) + copy_h2d_if_changed(self.prompt_lens_cuda[:self.num_seqs], + self.prompt_lens_cpu[:self.num_seqs]) # number of tokens in the kv cache for each sequence in the batch cached_token_lens = torch.tensor( @@ -1051,9 +1053,8 @@ def prepare(self) -> None: # the sequence length including the cached tokens and the input tokens. self.kv_lens[:self.num_seqs].copy_( kv_lens + self.kv_cache_params.num_extra_kv_tokens) - self.kv_lens_cuda[:self.num_seqs].copy_(maybe_pin_memory( - kv_lens[:self.num_seqs]), - non_blocking=True) + step_inputs.copy_(self.kv_lens_cuda[:self.num_seqs], + maybe_pin_memory(kv_lens[:self.num_seqs])) # total kv lens for context requests and generation requests, without extra tokens self.host_total_kv_lens[0] = kv_lens[:self.num_contexts].sum().item() self.host_total_kv_lens[1] = kv_lens[self.num_contexts:self. @@ -1160,7 +1161,7 @@ def prepare_encoder_decoder_from_precomputed_lengths( f"({self.kv_cache_manager.max_seq_len}).") num_seqs = self.num_seqs - self.prompt_lens_cuda[:num_seqs].copy_(prompt_lens, non_blocking=True) + copy_h2d_if_changed(self.prompt_lens_cuda[:num_seqs], prompt_lens) self.kv_lens_cuda[:num_seqs].copy_(kv_lens, non_blocking=True) self.host_total_kv_lens[0] = context_kv_tokens self.host_total_kv_lens[1] = generation_kv_tokens diff --git a/tensorrt_llm/_torch/custom_ops/torch_custom_ops.py b/tensorrt_llm/_torch/custom_ops/torch_custom_ops.py index 01b9f9cace7c..7ac21c784c47 100644 --- a/tensorrt_llm/_torch/custom_ops/torch_custom_ops.py +++ b/tensorrt_llm/_torch/custom_ops/torch_custom_ops.py @@ -849,6 +849,65 @@ def mxfp8_quantize_gemm_autotuned( gemm_inputs, tune)) +def mxfp8_gemm_autotuned( + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool = False, +) -> torch.Tensor: + """The GEMM half of mxfp8_quantize_gemm_autotuned for an activation already in MXFP8. + + Same runner, op key and tuning-cache entries as the decode-graph MXFP8 GEMM: + the tactic per token bucket comes from the AutoTuner (merged across ranks + when the runtime asks for it), never from a per-rank backend sweep. + """ + gemm_runner = FlashInferMXFP8GemmRunner(output_dtype) + gemm_inputs = [act, act_scale, weight, weight_scale] + return gemm_runner(gemm_inputs, + tactic=_choose_mxfp8_tactic( + _FLASHINFER_MXFP8_GEMM_AUTOTUNED_OP, gemm_runner, + gemm_inputs, tune)) + + +def flashinfer_mxfp8_gemm_cute_dsl( + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + sweep_configs: bool, +) -> torch.Tensor: + """FlashInfer's CuTeDSL MXFP8 GEMM for an activation its producer already quantized. + + Unlike ``mxfp8_quantize_gemm_autotuned``, the backend is fixed rather than + chosen per token bucket: the AutoTuner profiles with the weight L2-resident, + where CUTLASS can edge out CuTeDSL, while a decode step streams it cold. + + ``sweep_configs`` runs FlashInfer's own CuTeDSL config sweep: it profiles + the configs for this shape while ``flashinfer_mxfp8_autotune`` is active + (the warmup tuning pass) and replays the cached best one afterwards; + without it the heuristic config runs. + """ + if sweep_configs: + from ..modules.linear import flashinfer_mxfp8_autotune_active + context = _flashinfer_autotune( + tune_mode=flashinfer_mxfp8_autotune_active()) + else: + context = _flashinfer_autotune(tune_mode=False, skip_ops="mxfp8_gemm") + with context: + return _flashinfer_mm_mxfp8( + act, + weight.t(), + act_scale, + weight_scale, + out_dtype=output_dtype, + use_8x4_sf_layout=False, + backend="cute-dsl", + ) + + class FP4GemmRunner(TunableRunner): runner_dict = dict() tuning_config = TuningConfig(dynamic_tensor_specs=(DynamicTensorSpec( diff --git a/tensorrt_llm/_torch/models/checkpoints/hf/minimaxm3_weight_mapper.py b/tensorrt_llm/_torch/models/checkpoints/hf/minimaxm3_weight_mapper.py index 261559f186d4..7ce7a546f0d7 100644 --- a/tensorrt_llm/_torch/models/checkpoints/hf/minimaxm3_weight_mapper.py +++ b/tensorrt_llm/_torch/models/checkpoints/hf/minimaxm3_weight_mapper.py @@ -60,6 +60,11 @@ def _read_global_scale(name: str, value) -> Tensor: return scale +def reciprocal_global_scale(name: str, value: Tensor) -> Tensor: + """Validate a packed expert scale and return its FP32 modelopt divisor.""" + return nvfp4_weight_scale_2(_read_global_scale(name, value)) + + def translate_packed_nvfp4_experts(weights: Mapping[str, Tensor]) -> Mapping[str, Tensor]: """Rename packed NVFP4 routed-expert tensors to the NVFP4 MoE contract. diff --git a/tensorrt_llm/_torch/models/modeling_dflash.py b/tensorrt_llm/_torch/models/modeling_dflash.py index f670cdb7937f..d1d80a29efe2 100644 --- a/tensorrt_llm/_torch/models/modeling_dflash.py +++ b/tensorrt_llm/_torch/models/modeling_dflash.py @@ -509,6 +509,13 @@ def __init__(self, draft_config, *, dflash_attention_backend: str = "AUTO"): f"{self.dflash_attention_backend!r} but loads no op set for it." ) self._dflash_trtllm_gen_workspace = None + # Optional target-installed hooks. Unarmed draft models retain the stock path. + self._dflash_lean_glue = False + self._dflash_cache_rows = None + self._dflash_fused_indices = None + self._dflash_fused_qk_norm_rope = None + self._dflash_fused_qk_norm_rope_append = None + self._dflash_attn_pdl = False self._dflash_trtllm_gen_counters = None self.register_buffer("_dflash_batch_indices", None, persistent=False) self.register_buffer("_dflash_block_offsets", None, persistent=False) @@ -1047,7 +1054,8 @@ def precompute_context_kv( # after the select() splits — no extra copy required. kv = kv_flat.view(N, L, 2, nkv, hd) k = kv[:, :, 0].contiguous() - v = kv[:, :, 1].contiguous() + # V only feeds the paged append and an in-place mask multiply, both stride-aware. + v = kv[:, :, 1] if self._dflash_lean_glue else kv[:, :, 1].contiguous() if self._k_norm_stacked is not None: # Fuse L per-layer RMSNorms into one. k is [N, L, nkv, hd]; @@ -1535,10 +1543,26 @@ def dflash_forward( q_rope_cos, q_rope_sin = self._get_rope_cos_sin(query_positions, dtype=rope_dtype) _rope = RotaryEmbedding.apply_rotary_pos_emb - # cache_seqlens BEFORE the block's K/V are stored. - cache_seqlens_i32 = num_ctx_per_req[:B].to(torch.int32) - cache_batch_idx_i32 = ctx_cache_batch_idx.to(torch.int32) - seq_lens_after = cache_seqlens_i32 + block_size + # A target's one-launch index prelude for the paged decode: the same int32 tensors as the stock lines, or None. + fused_indices = ( + self._dflash_fused_indices( + num_ctx_per_req[:B], + ctx_cache_batch_idx, + query_positions, + block_size, + ctx_page_table.size(1), + ctx_kv_cache[0].size(-2), + ) + if is_paged and self._dflash_fused_indices is not None + else None + ) + if fused_indices is not None: + cache_seqlens_i32, cache_batch_idx_i32, seq_lens_after = fused_indices[:3] + else: + # cache_seqlens BEFORE the block's K/V are stored. + cache_seqlens_i32 = num_ctx_per_req[:B].to(torch.int32) + cache_batch_idx_i32 = ctx_cache_batch_idx.to(torch.int32) + seq_lens_after = cache_seqlens_i32 + block_size if is_paged: max_batch_size = ctx_page_table.size(0) @@ -1554,28 +1578,38 @@ def dflash_forward( ) else: # FA4 needs no workspace or counter buffers. self._prepare_dflash_index_buffers(hidden_states.device, max_batch_size, block_size) - block_tables = ctx_page_table.index_select(0, cache_batch_idx_i32.long()) + rows = self._dflash_cache_rows if self._dflash_lean_glue else None + if rows is not None: + block_tables = ctx_page_table[rows[0] : rows[1]] + else: + block_tables = ctx_page_table.index_select(0, cache_batch_idx_i32.long()) pages_per_slot = block_tables.size(1) # Index the layer first: ctx_kv_cache is an [L, ...] tensor for the # private arena but a per-layer list when bound to the draft KV # cache manager's pool. page_size sits at -2 either way. page_size = ctx_kv_cache[0].size(-2) kv_indices = block_tables.flatten() - kv_indptr = torch.arange( - 0, - (B + 1) * pages_per_slot, - pages_per_slot, - dtype=torch.int32, - device=hidden_states.device, - ) - kv_last_page_len = ((seq_lens_after - 1) % page_size) + 1 + if fused_indices is not None: + kv_indptr, kv_last_page_len = fused_indices[3:5] + else: + kv_indptr = torch.arange( + 0, + (B + 1) * pages_per_slot, + pages_per_slot, + dtype=torch.int32, + device=hidden_states.device, + ) + kv_last_page_len = ((seq_lens_after - 1) % page_size) + 1 batch_indices = self._dflash_batch_indices append_batch_indices = batch_indices[:B].reshape(-1) - append_positions = ( - (cache_seqlens_i32.view(-1, 1) + self._dflash_block_offsets) - .reshape(-1) - .contiguous() - ) + if fused_indices is not None: + append_positions = fused_indices[5] + else: + append_positions = ( + (cache_seqlens_i32.view(-1, 1) + self._dflash_block_offsets) + .reshape(-1) + .contiguous() + ) else: # VANILLA # Slot and column that each block K/V row occupies self._prepare_dflash_index_buffers( @@ -1591,8 +1625,11 @@ def dflash_forward( (cache_seqlens_i32.view(-1, 1) + self._dflash_block_offsets).reshape(-1).long() ) - # Flatten query positions once for the fused QK-norm-RoPE kernel. - query_positions_flat_i32 = query_positions.reshape(-1).to(torch.int32) + if fused_indices is not None: + query_positions_flat_i32 = fused_indices[6] + else: + # Flatten query positions once for the fused QK-norm-RoPE kernel. + query_positions_flat_i32 = query_positions.reshape(-1).to(torch.int32) residual = None has_block_conv = self.has_block_conv @@ -1622,6 +1659,9 @@ def dflash_forward( # QKV projection on normed query tokens (2D) qkv_query = attn_mod.qkv_proj(hs_normed_flat) # [B*blk, qkv_size] + kv_appended = ( + False # the fused q / k norm + RoPE also wrote this layer's K / V pages (see below) + ) if use_fused_qk_norm_rope: # One kernel does q_norm + k_norm + RoPE in-place on qkv. # Only safe when the drafter's rope params don't use YaRN / @@ -1635,21 +1675,66 @@ def dflash_forward( k_noise_bshd = k_noise_2d.reshape(B, block_size, num_kv_heads_per_rank, head_dim) v_noise_bshd = v_noise_2d.reshape(B, block_size, num_kv_heads_per_rank, head_dim) elif use_fused_rope: - # Per-head RMSNorm on q/k (returns new contiguous tensors), - # then flashinfer in-place RoPE sharing the same cos/sin cache - # as precompute_context_kv. - q = attn_mod.q_norm(qkv_query[:, :q_size].reshape(-1, head_dim)).view(-1, q_size) - k = attn_mod.k_norm( - qkv_query[:, q_size : q_size + kv_size].reshape(-1, head_dim) - ).view(-1, kv_size) - _flashinfer_rope( - query_positions_flat_i32, - q, - k, - head_dim, - self._get_cos_sin_cache(), - self._is_neox, + # A target's one-launch q / k norm + RoPE: the same new q / k as the lines below, or None. With the + # append form, the same launch also writes the block's K / V into this layer's paged cache. + fused_qk = ( + self._dflash_fused_qk_norm_rope_append( + qkv_query, + q_size, + kv_size, + head_dim, + attn_mod.q_norm, + attn_mod.k_norm, + query_positions_flat_i32, + self._get_cos_sin_cache(), + self._is_neox, + ctx_kv_cache[layer_idx], + append_batch_indices, + append_positions, + kv_indices, + kv_indptr, + ) + if self._dflash_fused_qk_norm_rope_append is not None + and self.dflash_attention_backend == "TRTLLM" + else None ) + kv_appended = fused_qk is not None + if fused_qk is None: + fused_qk = ( + self._dflash_fused_qk_norm_rope( + qkv_query, + q_size, + kv_size, + head_dim, + attn_mod.q_norm, + attn_mod.k_norm, + query_positions_flat_i32, + self._get_cos_sin_cache(), + self._is_neox, + ) + if self._dflash_fused_qk_norm_rope is not None + else None + ) + if fused_qk is not None: + q, k = fused_qk + else: + # Per-head RMSNorm on q/k (returns new contiguous tensors), + # then flashinfer in-place RoPE sharing the same cos/sin cache + # as precompute_context_kv. + q = attn_mod.q_norm(qkv_query[:, :q_size].reshape(-1, head_dim)).view( + -1, q_size + ) + k = attn_mod.k_norm( + qkv_query[:, q_size : q_size + kv_size].reshape(-1, head_dim) + ).view(-1, kv_size) + _flashinfer_rope( + query_positions_flat_i32, + q, + k, + head_dim, + self._get_cos_sin_cache(), + self._is_neox, + ) Q_bshd = q.view(B, block_size, num_heads_per_rank, head_dim) k_noise_bshd = k.view(B, block_size, num_kv_heads_per_rank, head_dim) v_noise_bshd = qkv_query[:, q_size + kv_size :].reshape( @@ -1694,21 +1779,26 @@ def dflash_forward( causal, window_size = self._resolve_block_attention(layer_idx) if self.dflash_attention_backend == "TRTLLM": layer_cache = ctx_kv_cache[layer_idx] - trtllm_gen_ops.append_paged_kv_cache( - append_key=k_noise_bshd.reshape( - -1, num_kv_heads_per_rank, head_dim - ).contiguous(), - append_value=v_noise_bshd.reshape( - -1, num_kv_heads_per_rank, head_dim - ).contiguous(), - batch_indices=append_batch_indices, - positions=append_positions, - paged_kv_cache=layer_cache, - kv_indices=kv_indices, - kv_indptr=kv_indptr, - kv_last_page_len=kv_last_page_len, - kv_layout="HND", - ) + if not kv_appended: + trtllm_gen_ops.append_paged_kv_cache( + append_key=k_noise_bshd.reshape( + -1, num_kv_heads_per_rank, head_dim + ).contiguous(), + append_value=( + v_noise_bshd.reshape(-1, num_kv_heads_per_rank, head_dim) + if self._dflash_lean_glue + else v_noise_bshd.reshape( + -1, num_kv_heads_per_rank, head_dim + ).contiguous() + ), + batch_indices=append_batch_indices, + positions=append_positions, + paged_kv_cache=layer_cache, + kv_indices=kv_indices, + kv_indptr=kv_indptr, + kv_last_page_len=kv_last_page_len, + kv_layout="HND", + ) out = torch.empty_like(Q_bshd) q_flat = Q_bshd.reshape(-1, num_heads_per_rank, head_dim) out_flat = out.reshape(-1, num_heads_per_rank, head_dim) @@ -1726,7 +1816,7 @@ def dflash_forward( window_left=window_left, out=out_flat, sinks=None, - enable_pdl=False, + enable_pdl=self._dflash_attn_pdl, kv_layout="HND", backend="trtllm-gen", q_len_per_req=block_size, @@ -1767,7 +1857,7 @@ def dflash_forward( window_left=window_left, out=out_flat, sinks=None, - enable_pdl=False, + enable_pdl=self._dflash_attn_pdl, kv_layout="HND", kv_cache_sf=None, uses_shared_paged_kv_idx=True, @@ -1780,9 +1870,11 @@ def dflash_forward( append_key=k_noise_bshd.reshape( -1, num_kv_heads_per_rank, head_dim ).contiguous(), - append_value=v_noise_bshd.reshape( - -1, num_kv_heads_per_rank, head_dim - ).contiguous(), + append_value=( + v_noise_bshd.reshape(-1, num_kv_heads_per_rank, head_dim) + if self._dflash_lean_glue + else v_noise_bshd.reshape(-1, num_kv_heads_per_rank, head_dim).contiguous() + ), batch_indices=append_batch_indices, positions=append_positions, paged_kv_cache=layer_cache, diff --git a/tensorrt_llm/_torch/modules/linear.py b/tensorrt_llm/_torch/modules/linear.py index 0b229654a59c..ad794d6b494d 100644 --- a/tensorrt_llm/_torch/modules/linear.py +++ b/tensorrt_llm/_torch/modules/linear.py @@ -3366,6 +3366,16 @@ def _mxfp8_cutlass_op_available() -> bool: "flashinfer_mxfp8_decode_graph_capture_active", default=False) +def flashinfer_mxfp8_autotune_active() -> bool: + """Whether the current forward runs inside ``flashinfer_mxfp8_autotune``.""" + return _FLASHINFER_MXFP8_AUTOTUNE_ACTIVE.get() + + +def flashinfer_mxfp8_decode_graph_capture_active() -> bool: + """Whether the current forward is a decode CUDA-graph warmup or capture.""" + return _FLASHINFER_MXFP8_DECODE_GRAPH_CAPTURE_ACTIVE.get() + + @contextmanager def flashinfer_mxfp8_autotune() -> Iterator[None]: """Tune FlashInfer MXFP8 tactics while enabling auto-dispatched calls.""" diff --git a/tensorrt_llm/_torch/pyexecutor/_util.py b/tensorrt_llm/_torch/pyexecutor/_util.py index b177b8c50894..cf91917ec042 100644 --- a/tensorrt_llm/_torch/pyexecutor/_util.py +++ b/tensorrt_llm/_torch/pyexecutor/_util.py @@ -53,6 +53,7 @@ from ..speculative import (draft_prompt_lookahead, get_num_extra_kv_tokens, get_num_spec_layers, get_spec_decoder, should_use_separate_draft_kv_cache) +from ..speculative.spec_sampler_base import SpecSampler from ..utils import is_gdn_replay_enabled from . import config_utils from .config_utils import (MambaKVCacheParams, _is_sliding_attention_layer, @@ -77,6 +78,7 @@ from .resource_manager import (KVCacheCompressionManager, KVCacheManager, PeftCacheManager, ResourceManager, ResourceManagerType) +from .runtime_hooks import runtime_hooks_of from .sampler import EarlyStopSampler, EarlyStopWithMMResult, TorchSampler from .scheduler import (BindCapacityScheduler, BindMicroBatchScheduler, KVCacheV2Scheduler, MultimodalEagerEncoderScheduler, @@ -4626,7 +4628,18 @@ def instantiate_sampler( ) if engine.spec_config is not None and engine.spec_config.spec_dec_mode.has_spec_decoder( ): - return get_spec_decoder(sampler_args, engine.spec_config) + sampler = get_spec_decoder(sampler_args, engine.spec_config) + runtime_hooks = runtime_hooks_of(engine.model) + if isinstance(sampler, SpecSampler): + if runtime_hooks.spec_step_fold is not None: + sampler.bind_spec_step_fold(runtime_hooks.spec_step_fold) + sampler.d2h_side_stream_copies = runtime_hooks.sampler_d2h_side_stream + sampler.static_slot_ids = runtime_hooks.spec_small_h2d + if runtime_hooks.sampler_d2h_side_stream: + logger.info( + "[M3 host] spec sampler store copies on the D2H side stream" + ) + return sampler if mm_encoder_only: # NOTE: handle model outputs specially for mm encoder executor/engine. diff --git a/tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py b/tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py index 19ed990f33a1..3aea5f022a99 100644 --- a/tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py +++ b/tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py @@ -27,6 +27,7 @@ from ..speculative.spec_sampler_base import SampleStateTensorsSpec from ..speculative.utils import get_draft_kv_cache_manager from ..utils import make_weak_ref, piecewise_cuda_graph +from . import step_inputs from .llm_request import LlmRequest, get_draft_token_length from .resource_manager import (BaseResourceManager, ResourceManager, ResourceManagerType) @@ -149,6 +150,15 @@ class CUDAGraphRunnerConfig: kv_cache_manager_key: Any dynamic_draft_len_mapping: Optional[Dict[int, int]] = None sparse_attention_config: Optional[BaseSparseAttentionConfig] = None + # Total sequence length at or below which a generation batch replays the + # short-sequence graph family; None captures one family. Derived from the + # sparse attention config when it asks for the split, else from the model's + # runtime hooks. + short_long_seq_len_threshold: Optional[int] = None + # Give the graph metadata a buffer pool of its own instead of the + # process-wide one, so no other engine writes its metadata buffers (an + # engine that skips unchanged uploads into them relies on that). + private_metadata_buffers: bool = False enable_encoder_decoder_mixed_cuda_graph: bool = False enable_in_graph_sampling: bool = False @@ -162,6 +172,12 @@ class CUDAGraphRunner: multiple graphs, keyed by batch shape and execution-path specializations. """ WARMUP_STEPS = 1 + # CUDAGraphRunnerConfig.short_long_seq_len_threshold, when the config set one. + _configured_seq_len_threshold: Optional[int] = None + # The graphs captured with a step-input pack's in-graph prologue, and that + # pack (step_inputs.capture_prologue). None until the first such capture: a + # class-level default, so a runner without one replays exactly as before. + _step_input_prologue_keys: Optional[Dict[KeyType, Any]] = None def __init__(self, config: CUDAGraphRunnerConfig): self.config = config @@ -174,6 +190,7 @@ def __init__(self, config: CUDAGraphRunnerConfig): self.max_beam_width = config.max_beam_width self.spec_config = config.spec_config self.sparse_config = config.sparse_attention_config + self._configured_seq_len_threshold = config.short_long_seq_len_threshold self.is_encoder_decoder = config.is_encoder_decoder self.enable_encoder_decoder_mixed_cuda_graph = ( config.enable_encoder_decoder_mixed_cuda_graph) @@ -203,7 +220,9 @@ def __init__(self, config: CUDAGraphRunnerConfig): self.shared_static_tensors: Dict[str, torch.Tensor] = {} if self.enabled: self._create_shared_static_tensors() - self.cuda_graph_meta_buffers = get_memory_buffers() + self.cuda_graph_meta_buffers = (Buffers() + if config.private_metadata_buffers else + get_memory_buffers()) # On-the-fly capture is disabled by default to prevent workspace # tensor reallocation from invalidating addresses baked into existing @@ -301,6 +320,20 @@ def _gather_adp_graph_batch_info(self, batch: ScheduledRequests, tiers = gathered[:, 2].tolist() return list(zip(flags, sizes, tiers)) + @property + def short_long_seq_len_threshold(self) -> Optional[int]: + """The total sequence length splitting the short and long graph families, or None for one family. + + The runner config's value when it set one, else the sparse attention + config's when that asks for the split. + """ + if self._configured_seq_len_threshold is not None: + return self._configured_seq_len_threshold + if (isinstance(self.sparse_config, SeqLenAwareSparseAttentionConfig) + and self.sparse_config.needs_separate_short_long_cuda_graphs()): + return self.sparse_config.seq_len_threshold + return None + def _get_seq_len_mode( self, batch: ScheduledRequests, @@ -313,8 +346,7 @@ def _get_seq_len_mode( that the model engine temporarily placed in the generation list. It is empty for the existing generation-only path. """ - if (isinstance(self.sparse_config, SeqLenAwareSparseAttentionConfig) - and self.sparse_config.needs_separate_short_long_cuda_graphs()): + if self.short_long_seq_len_threshold is not None: # Some sparse attention algorithms need to use different forward paths for short and long sequences. # For example, the DSA can skip the MQA and Top-K in the indexer for short sequences to reduce the # computational overhead. To support this feature, we need to capture separate CUDA graphs for short @@ -356,7 +388,7 @@ def _get_seq_len_mode( from ..speculative import get_num_extra_kv_tokens num_extra_kv_tokens = get_num_extra_kv_tokens(self.spec_config) max_seq_len = max(total_seq_lens) - if max_seq_len <= self.sparse_config.seq_len_threshold - num_extra_kv_tokens: + if max_seq_len <= self.short_long_seq_len_threshold - num_extra_kv_tokens: short_seq_len_mode = True else: short_seq_len_mode = False @@ -771,6 +803,10 @@ def capture(self, # setup/capture; release its reference before entering. output = None with torch.cuda.graph(graph, pool=self.memory_pool): + # A pre-replay hook reads the static inputs before the replay, + # so their copies must land before it: no prologue then. + if not self._pre_replay_hooks: + self._capture_step_input_prologue(key) output = forward_fn(capture_inputs) # Snapshot before postprocessing so the check compares against # the addresses the graph actually baked in. @@ -786,6 +822,35 @@ def capture(self, self.memory_pool = graph.pool() return graph_output + def _capture_step_input_prologue(self, key: KeyType) -> None: + """Inside the capture of ``key``: the active step-input pack's in-graph + prologue, if it has one (step_inputs.capture_prologue).""" + prologue_pack = step_inputs.capture_prologue() + if prologue_pack is not None: + if self._step_input_prologue_keys is None: + self._step_input_prologue_keys = {} + self._step_input_prologue_keys[key] = prologue_pack + elif self._step_input_prologue_keys: + # A recapture of the key without the prologue. + self._step_input_prologue_keys.pop(key, None) + + def _issue_step_inputs(self, key: KeyType) -> None: + """Issue the step's packed input copies before the replay of ``key`` + (a no-op without an open pack). A graph captured with a pack's + prologue issues that pack itself at the start of the replay.""" + prologue_keys = self._step_input_prologue_keys + prologue_pack = prologue_keys.get(key) if prologue_keys else None + if prologue_pack is None: + step_inputs.flush() + elif step_inputs.is_active(prologue_pack): + step_inputs.flush(in_graph=True) + else: + # Replayed outside the pack's context: another (or no) pack is + # active, which the prologue does not read, so it is issued + # eagerly, and nothing filled the prologue pack's window. + step_inputs.flush() + prologue_pack.quiesce_graph_window() + def replay( self, key: KeyType, @@ -816,7 +881,7 @@ def replay( f"replay() got {seqlen} tokens for key {key}, but the graph " f"was captured for {expected_num_tokens} tokens. A shorter " "input_ids leaves the tail of the static input buffer stale.") - static_tensors["input_ids"][:seqlen].copy_(input_ids) + step_inputs.copy_(static_tensors["input_ids"][:seqlen], input_ids) position_ids = current_inputs["position_ids"] if self.config.use_mrope: @@ -863,7 +928,8 @@ def replay( f"for key {key}, but expected {expected_position_ids_shape}. " "torch.Tensor.copy_() silently broadcasts mismatched shapes, " "which would corrupt the static input buffer.") - static_tensors["position_ids"][:, :seqlen].copy_(position_ids) + step_inputs.copy_(static_tensors["position_ids"][:, :seqlen], + position_ids) num_encoder_tokens = key.num_encoder_tokens if num_encoder_tokens: @@ -887,6 +953,10 @@ def replay( encoder_hidden_states) static_encoder_hidden_states[actual_num_encoder_tokens:].zero_() + # Issue the step's packed input copies (no-op without an open pack) before + # anything below reads the static inputs on the device; a graph captured + # with the pack's prologue issues them itself at the start of the replay. + self._issue_step_inputs(key) if self._pre_replay_hooks: hook_tensors = static_tensors if position_id_offsets is not None: @@ -1338,6 +1408,7 @@ def clear(self): self.graph_outputs.clear() self.graph_metadata.clear() self._pre_replay_position_ids = None + self._step_input_prologue_keys = None self.padding_dummy_requests = {} del self.memory_pool self.memory_pool = None diff --git a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py index 3b8d7fa2da91..ca6a0068730e 100644 --- a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py +++ b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py @@ -33,7 +33,7 @@ MoeLoadBalancerIterContext, ) from tensorrt_llm._torch.peft.lora.config import LoraConfig -from tensorrt_llm._torch.peft.lora.manager import LoraModelConfig +from tensorrt_llm._torch.pyexecutor import h2d_cache, step_inputs from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph_runner import BreakableCUDAGraphRunner from tensorrt_llm._torch.pyexecutor.cuda_graph_runner import ( CUDAGraphRunner, @@ -41,7 +41,7 @@ KeyType, get_mrope_dummy_seq_slot, ) -from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder +from tensorrt_llm._torch.pyexecutor.h2d_cache import H2DSkipCache, copy_h2d_if_changed from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import ( BaseMambaCacheManager, @@ -54,11 +54,23 @@ ResourceManager, ResourceManagerType, ) +from tensorrt_llm._torch.pyexecutor.runtime_hooks import ( + DEFAULT_RUNTIME_HOOKS, + MsaRuntimePolicy, + RuntimeHooks, + runtime_hooks_of, +) from tensorrt_llm._torch.pyexecutor.sampler import SampleStateTensors from tensorrt_llm._torch.pyexecutor.sampler.ops.flashinfer import warmup_sampling_module +from tensorrt_llm._torch.pyexecutor.sampler.sampler import ( + GREEDY_HIDDEN_STATES_KEY, + GREEDY_TOKENS_KEY, + greedy_tokens_usable, +) from tensorrt_llm._torch.pyexecutor.sampler.sampler_common import SampleType from tensorrt_llm._torch.pyexecutor.scheduler import ScheduledRequests from tensorrt_llm._torch.pyexecutor.scheduler.scheduler import RemoteTailPhase +from tensorrt_llm._torch.pyexecutor.step_inputs import StepInputPack from tensorrt_llm._torch.pyexecutor.trace_log_utils import log_mem_snapshot from tensorrt_llm._torch.pyexecutor.warmup_timer import _WarmupTimer from tensorrt_llm._torch.pyexecutor.workspace import EagerWorkspaceReclaimer @@ -91,6 +103,7 @@ from tensorrt_llm.llmapi.llm_args import ( BaseSparseAttentionConfig, DecodingBaseConfig, + MiniMaxM3SparseAttentionConfig, PrefillCudaGraphBackend, SeqLenAwareSparseAttentionConfig, ) @@ -296,6 +309,15 @@ def __getattr__(self, name: str) -> Any: class DecoderRunner(ScheduledModelRunner): """Run scheduled decoder generation with its own buffers, graphs and metadata.""" + # Step-loop behaviour the model opted into (RuntimeHooks); all off unless it declares otherwise. + runtime_hooks: RuntimeHooks = DEFAULT_RUNTIME_HOOKS + # This engine's skip cache and step-input pack (None: the hook is off), active only during its own forward. + _h2d_cache: H2DSkipCache | None = None + _step_pack: StepInputPack | None = None + # Whether a batch may keep the greedy tokens a forward sampled in place of the logits. False when the executor + # runs a guided decoder, which reads the logits of every batch. + keep_greedy_only_outputs = True + # Optional execution paths that a subclass can turn off. _context_warmups_supported = True _context_graph_promotion_supported = True @@ -347,9 +369,33 @@ def __init__( self._mm_item_scheduler = mm_item_scheduler self._config = config + self.runtime_hooks = runtime_hooks_of(model) + self._h2d_cache = ( + H2DSkipCache(changed_span=self.runtime_hooks.h2d_changed_span) + if self.runtime_hooks.skip_unchanged_h2d + else None + ) + self._step_pack = ( + StepInputPack( + torch.device("cuda", torch.cuda.current_device()), + parallel_scatter=self.runtime_hooks.pack_parallel_scatter, + in_graph=self.runtime_hooks.pack_in_graph, + early_copy=self.runtime_hooks.pack_early_copy, + ) + if self.runtime_hooks.pack_step_inputs + else None + ) + if self._step_pack is not None and self._step_pack.parallel_scatter: + logger.info("[M3 host] step-input pack: parallel scatter for non-overlapping copies") + if self._step_pack is not None and self._step_pack.in_graph: + logger.info("[M3 host] step-input pack issued by the decode graph's prologue") + if self._step_pack is not None and self._step_pack.early_copy: + logger.info( + "[M3 host] step-input pack: window uploaded ahead of the replay on a side copy stream" + ) + if self._h2d_cache is not None and self._h2d_cache.changed_span: + logger.info("[M3 host] changed metadata uploads stage only each row's changed span") - self.guided_decoder: CapturableGuidedDecoder | None = None - self.lora_model_config: LoraModelConfig | None = None self.forward_pass_callable = None # Optional in-graph sampling hook, registered by PyExecutor. Called at # the tail of _forward_step -- i.e. right after the LM head, and inside @@ -391,7 +437,15 @@ def forward( resource_manager: ResourceManager, is_dummy: bool = False, ) -> Any: - return self._forward_decoder(inputs, resource_manager, is_dummy=is_dummy) + with h2d_cache.active(self._h2d_cache), step_inputs.active(self._step_pack): + try: + return self._forward_decoder(inputs, resource_manager, is_dummy=is_dummy) + except BaseException: + # A failed step issues none of its packed copies, so forget the shadows recorded for them too. + step_inputs.discard() + if self._h2d_cache is not None: + self._h2d_cache.clear() + raise def warmup(self, resource_manager: ResourceManager) -> None: kv_cache_manager = resource_manager.get_resource_manager(self.kv_cache_manager_key) @@ -472,6 +526,14 @@ def _allocate_decoder_buffers(self) -> None: self.position_ids_cuda = torch.empty( (self._config.max_num_tokens,), dtype=torch.int, device="cuda" ) + if self._config.is_spec_decode and self.runtime_hooks.spec_step_fold is not None: + self.runtime_hooks.spec_step_fold.bind_engine_buffers( + self.input_ids_cuda, + self.draft_tokens_cuda, + self.previous_pos_id_offsets_cuda, + self.previous_kv_lens_offsets_cuda, + self._config.max_batch_size, + ) # Host counter of the steady-state generation prepare cache. self._steady_gen_positions_pinned = torch.empty( (self._config.max_num_tokens,), dtype=torch.int, pin_memory=prefer_pinned() @@ -550,9 +612,33 @@ def _cuda_graph_runner_config(self) -> CUDAGraphRunnerConfig: dist=self.dist, kv_cache_manager_key=self.kv_cache_manager_key, sparse_attention_config=self.sparse_attention_config, + short_long_seq_len_threshold=self._short_long_seq_len_threshold(), + private_metadata_buffers=self.runtime_hooks.skips_unchanged_metadata_uploads, enable_in_graph_sampling=self._config.enable_in_graph_sampling, ) + def _short_long_seq_len_threshold(self) -> int | None: + """The total sequence length that splits the short and long decode graph families, or None for one family. + + Set by a sparse attention config that asks for the split, or by a model whose MSA + policy stages identity selections for short sequences: a sequence within the + top-k blocks attends every valid block. + """ + sparse_config = self.sparse_attention_config + if ( + isinstance(sparse_config, SeqLenAwareSparseAttentionConfig) + and sparse_config.needs_separate_short_long_cuda_graphs() + ): + return sparse_config.seq_len_threshold + if self.runtime_hooks.msa.short_sequence_identity: + if not isinstance(sparse_config, MiniMaxM3SparseAttentionConfig): + raise ValueError( + "MsaRuntimePolicy.short_sequence_identity requires a MiniMax-M3 sparse " + f"attention config, got {type(sparse_config).__name__}" + ) + return sparse_config.sparse_topk_blocks * sparse_config.sparse_block_size + return None + def _initialize_breakable_cuda_graph_runner(self) -> BreakableCUDAGraphRunner | None: if ( self.cuda_graph_runner is None @@ -1643,6 +1729,11 @@ def run_autotuner_pass(autotune_context: Any, synchronize_trtllm_cache: bool) -> ran_flashinfer_forward = run_autotuner_pass( flashinfer_mxfp8_autotune(), synchronize_trtllm_cache=False ) + if self.runtime_hooks.merge_tuned_tactics_across_ranks: + # Rank-local profiling noise must not leave tensor-parallel peers + # serving different tactics for one shape: keep the fastest everywhere. + # Collective; before the native MXFP8 cache below registers the tactics. + AutoTuner.get().post_tune_merge_tactics() if enable_flashinfer_mxfp8_autotuner: if ran_flashinfer_forward: @@ -2037,6 +2128,10 @@ def _run_cuda_graph_warmup(self, resource_manager: ResourceManager): ) with flashinfer_autotune_context, flashinfer_mxfp8_decode_graph_capture(): self._capture_generation_cuda_graphs(resource_manager) + if is_warmup_only and self.runtime_hooks.merge_tuned_tactics_across_ranks: + # Tactics tuned while warming the decode graphs agree across + # ranks before the capture pass reads them. + AutoTuner.get().post_tune_merge_tactics() self._capture_additional_cuda_graphs(resource_manager) # Piecewise graphs have separate capture machinery and do not use the # whole-model attention workspace. Capture them only on the second pass. @@ -2081,11 +2176,8 @@ def _capture_generation_cuda_graphs(self, resource_manager: ResourceManager): if self.mapping is not None and self.mapping.has_cp_helix(): effective_max_seq_len = self._config.max_seq_len // self.mapping.cp_size - sparse_config = self.sparse_attention_config - if ( - isinstance(sparse_config, SeqLenAwareSparseAttentionConfig) - and sparse_config.needs_separate_short_long_cuda_graphs() - ): + short_long_seq_len_threshold = self._short_long_seq_len_threshold() + if short_long_seq_len_threshold is not None: # For short sequences, subtract the maximum runtime tokens consumed # by a generation step so all current-step tokens stay within the # sequence length threshold. PARD uses 2K tokens here, not K+1. @@ -2093,7 +2185,7 @@ def _capture_generation_cuda_graphs(self, resource_manager: ResourceManager): self._config.max_draft_len ) # For long sequences, use the default maximum sequence length. - max_seq_len = sparse_config.seq_len_threshold - max_runtime_tokens_per_gen_step + max_seq_len = short_long_seq_len_threshold - max_runtime_tokens_per_gen_step if max_seq_len < effective_max_seq_len: max_seq_len_list = [effective_max_seq_len, max_seq_len] else: @@ -2776,15 +2868,84 @@ def _set_up_attn_metadata( config, "chunk_size", self.attn_metadata.mamba_chunk_size ) self.attn_metadata.mamba_metadata_cls = resolve_mamba_metadata_cls(self.model) + self._arm_msa_policy(kv_cache_manager) + self._arm_draft_block_offset_guard(draft_kv_cache_manager) return self.attn_metadata - def _set_up_spec_metadata(self, spec_resource_manager: BaseResourceManager | None): + def _arm_draft_block_offset_guard( + self, draft_kv_cache_manager: KVCacheManager | KVCacheManagerV2 | None + ) -> None: + """Turn on the separate draft KV cache's unchanged block-table skip the model opted into.""" + if not self.runtime_hooks.skip_unchanged_draft_block_offsets: + return + if not isinstance(draft_kv_cache_manager, KVCacheManagerV2): + return + if not draft_kv_cache_manager.skip_unchanged_block_offsets: + draft_kv_cache_manager.skip_unchanged_block_offsets = True + logger.info("[M3 host] draft block-offset copy skipped when unchanged") + + def _arm_msa_policy(self, kv_cache_manager: Any) -> None: + """Hand the model's MiniMax-M3 MSA preparation choices to the metadata and the KV cache.""" + msa = self.runtime_hooks.msa + if msa == MsaRuntimePolicy(): + return + # Imported here: only a model that armed an MSA policy needs the sparse backend. + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.cache_manager import ( + MiniMaxM3KVCacheManagerV2, + ) + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.msa_backend import ( + MiniMaxM3MsaSparseAttentionMetadata, + ) + + if not isinstance( + self.attn_metadata, MiniMaxM3MsaSparseAttentionMetadata + ) or not isinstance(kv_cache_manager, MiniMaxM3KVCacheManagerV2): + raise ValueError( + "RuntimeHooks.msa requires the MiniMax-M3 MSA sparse attention backend and KV cache, " + f"got {type(self.attn_metadata).__name__} and {type(kv_cache_manager).__name__}" + ) + self.attn_metadata.msa_policy = msa + kv_cache_manager.configure_decode_shortcuts( + skip_unchanged_block_offsets=msa.skip_unchanged_block_offsets, + reuse_pool_views=msa.reuse_pool_views, + ) + + def _set_up_spec_metadata( + self, spec_resource_manager: BaseResourceManager | None + ) -> SpecMetadata | None: if self.spec_metadata is not None: return self.spec_metadata self.spec_metadata = create_spec_metadata( self._config, self.model.config, spec_resource_manager ) + if self.spec_metadata is None: + return None + self.spec_metadata.skip_greedy_rng_upload = self.runtime_hooks.skip_greedy_rng_upload + self.spec_metadata.pack_rng_uploads = self.runtime_hooks.pack_spec_rng_uploads + self.spec_metadata.batch_map_changed_only = self.runtime_hooks.spec_batch_map_changed_only + self.spec_metadata.tail_bookkeep = self.runtime_hooks.spec_tail_bookkeep + self.spec_metadata.slot_ids_changed_only = self.runtime_hooks.spec_slot_ids_changed_only + if self.runtime_hooks.pack_spec_rng_uploads: + logger.info("[M3 host] sampled-step Philox uploads staged into the step-input pack") + if self.runtime_hooks.spec_small_h2d: + logger.info( + "[M3 host] spec step small uploads off the boundary (accepted counts packed, static slot ids)" + ) + if self.runtime_hooks.skip_accepted_draft_upload: + logger.info("[M3 host] spec step accepted-draft-count upload skipped (no reader)") + if self.runtime_hooks.spec_batch_map_changed_only: + logger.info( + "[M3 host] drafter batch_indices / batch_to_slot uploaded only when changed" + ) + if self.runtime_hooks.spec_slot_ids_changed_only: + logger.info( + "[M3 host] spec metadata slot ids (batch_slot_ids) uploaded only when changed" + ) + if self.runtime_hooks.kv_lens_update_once: + logger.info( + "[M3 host] spec step KV-length metadata re-derived once, after the KV-length correction" + ) return self.spec_metadata def _release_decoder_graphs(self) -> None: @@ -2804,6 +2965,20 @@ def _should_use_full_generation_page_table( self._config.disable_overlap_scheduler, spec_config, attn_metadata ) + def _kv_lens_corrected_in_preprocess( + self, inputs: dict[str, Any], enable_spec_decode: bool + ) -> bool: + """Whether preprocess_inputs applies the overlap scheduler's KV-length offsets to kv_lens_cuda and then calls + on_update_kv_lens (the branch below).""" + attn_meta = inputs.get("attn_metadata") + return ( + enable_spec_decode + and not self._config.disable_overlap_scheduler + and attn_meta is not None + and attn_meta.kv_cache_manager is not None + and hasattr(attn_meta, "kv_lens_cuda") + ) + def _preprocess_inputs( self, inputs: dict[str, Any], *, enable_spec_decode: bool, runtime_draft_len: int ): @@ -2811,8 +2986,14 @@ def _preprocess_inputs( Make some changes to the device inputs and avoid blocking the async data transfer """ attn_meta = inputs.get("attn_metadata") - # Invalidate per-forward-pass caches so they are recomputed (and captured) on every _forward_step. - if attn_meta is not None: + # Invalidate per-forward-pass caches so they are recomputed (and captured) on every _forward_step. With + # RuntimeHooks.kv_lens_update_once, a step that corrects the KV lengths below (and then calls it again) calls + # it only there: the second call rewrites everything the first one writes, and nothing reads the metadata + # in between. + if attn_meta is not None and not ( + self.runtime_hooks.kv_lens_update_once + and self._kv_lens_corrected_in_preprocess(inputs, enable_spec_decode) + ): attn_meta.on_update_kv_lens() if enable_spec_decode and not self._config.disable_overlap_scheduler: @@ -4002,7 +4183,7 @@ def previous_seq_slots_device(): previous_batch_indices, dtype=torch.int, pin_memory=prefer_pinned() ) previous_slots = self.previous_batch_indices_cuda[:previous_batch_len] - previous_slots.copy_(previous_batch_indices_host, non_blocking=True) + copy_h2d_if_changed(previous_slots, previous_batch_indices_host) return previous_slots if scheduled_requests.is_attention_dp_phase_idle: @@ -4023,21 +4204,77 @@ def previous_seq_slots_device(): if num_draft_tokens > 0: draft_tokens = torch.tensor(draft_tokens, dtype=torch.int, pin_memory=prefer_pinned()) self.draft_tokens_cuda[: len(draft_tokens)].copy_(draft_tokens, non_blocking=True) - if self._config.is_spec_decode and len(num_accepted_draft_tokens) > 0: + if ( + self._config.is_spec_decode + and len(num_accepted_draft_tokens) > 0 + and not self.runtime_hooks.skip_accepted_draft_upload + ): num_accepted_draft_tokens = torch.tensor( num_accepted_draft_tokens, dtype=torch.int, pin_memory=prefer_pinned() ) - self.num_accepted_draft_tokens_cuda[: len(num_accepted_draft_tokens)].copy_( - num_accepted_draft_tokens, non_blocking=True + copy_h2d_if_changed( + self.num_accepted_draft_tokens_cuda[: len(num_accepted_draft_tokens)], + num_accepted_draft_tokens, + pack=self.runtime_hooks.spec_small_h2d, + ) + step_fold = self.runtime_hooks.spec_step_fold if self._config.is_spec_decode else None + fold_rows = None + fold_staged = False + if step_fold is not None: + # A decode-only step's generation rows; the model's forward may stage the next + # step's inputs for exactly these rows (see SpecStepFold). + fold_rows = ( + [request.py_seq_slot for request in extend_requests] + if num_ctx_requests == 0 + else None ) + if fold_rows is not None and any(slot is None for slot in fold_rows): + fold_rows = None + fold_staged = step_fold.begin_step(fold_rows, len(extend_requests)) if next_draft_tokens_device is not None: - # Initialize these two values to zeros - self.previous_pos_id_offsets_cuda *= 0 - self.previous_kv_lens_offsets_cuda *= 0 runtime_tokens_per_gen_step = self.get_runtime_tokens_per_gen_step(runtime_draft_len) runtime_draft_token_buffer_width = runtime_tokens_per_gen_step - 1 + # The previous forward already staged this step's inputs when it ran for exactly + # these rows and this step adds nothing of its own. + inputs_staged = ( + fold_staged + and len(extend_dummy_requests) == 0 + and num_tokens == 0 + and num_draft_tokens == 0 + and previous_batch_len == len(extend_requests) + and previous_batch_indices == fold_rows + ) + if not inputs_staged and step_fold is not None and previous_batch_len > 0: + previous_slots = previous_seq_slots_device() + previous_pos_indices_host = torch.tensor( + previous_pos_indices, dtype=torch.int, pin_memory=prefer_pinned() + ) + copy_h2d_if_changed( + self.previous_pos_indices_cuda[ + : previous_batch_len * runtime_tokens_per_gen_step + ], + previous_pos_indices_host, + ) + num_extend_reqeust_wo_dummy = len(extend_requests) - len(extend_dummy_requests) + inputs_staged = step_fold.gather( + new_tokens_device, + next_draft_tokens_device, + new_tokens_lens_device, + previous_slots, + self.previous_pos_indices_cuda, + previous_batch_len, + runtime_tokens_per_gen_step, + runtime_draft_token_buffer_width, + num_tokens, + num_draft_tokens, + num_extend_reqeust_wo_dummy - previous_batch_len, + ) + if not inputs_staged: + # Initialize these two values to zeros + self.previous_pos_id_offsets_cuda *= 0 + self.previous_kv_lens_offsets_cuda *= 0 - if previous_batch_len > 0: + if previous_batch_len > 0 and not inputs_staged: previous_slots = previous_seq_slots_device() # previous input ids previous_batch_tokens = previous_batch_len * runtime_tokens_per_gen_step @@ -4064,8 +4301,9 @@ def previous_seq_slots_device(): previous_pos_indices_host = torch.tensor( previous_pos_indices, dtype=torch.int, pin_memory=prefer_pinned() ) - self.previous_pos_indices_cuda[0:previous_batch_tokens].copy_( - previous_pos_indices_host, non_blocking=True + copy_h2d_if_changed( + self.previous_pos_indices_cuda[0:previous_batch_tokens], + previous_pos_indices_host, ) # The order of requests in a batch: [context requests, generation requests] @@ -4164,13 +4402,13 @@ def previous_seq_slots_device(): ) final_position_ids = self.mrope_position_ids_cuda[:, :, :total_num_tokens] else: - self.position_ids_cuda[:total_num_tokens].copy_(host_position_ids, non_blocking=True) + step_inputs.copy_(self.position_ids_cuda[:total_num_tokens], host_position_ids) final_position_ids = self.position_ids_cuda[:total_num_tokens].unsqueeze(0) if enable_spec_decode: - self.gather_ids_cuda[: len(gather_ids)].copy_( + copy_h2d_if_changed( + self.gather_ids_cuda[: len(gather_ids)], torch.tensor(gather_ids, dtype=torch.int, pin_memory=prefer_pinned()), - non_blocking=True, ) if self.mapping.has_cp_helix(): @@ -4413,9 +4651,11 @@ def previous_seq_slots_device(): # num_generations / num_tokens / seq_lens are set above, before the # attention-DP allgather that must agree with prepare(). spec_metadata.host_position_ids = host_position_ids - spec_metadata.num_accepted_draft_tokens = self.num_accepted_draft_tokens_cuda[ - : len(num_accepted_draft_tokens) - ] + spec_metadata.num_accepted_draft_tokens = ( + None + if self.runtime_hooks.skip_accepted_draft_upload + else self.num_accepted_draft_tokens_cuda[: len(num_accepted_draft_tokens)] + ) if context_prompt_lookahead is not None: spec_metadata.populate_context_prompt_lookahead(context_prompt_lookahead) # No-op for non 1-model @@ -4790,6 +5030,16 @@ def _forward_decoder( execution_requests.generation_requests, ) + # Pack the step's small input copies into one upload and one scatter at the + # graph replay. Only a replay of an already captured graph, which always + # reaches the flush, may open a pack. + if ( + self.runtime_hooks.pack_step_inputs + and can_run_graph + and not self.cuda_graph_runner.is_warmup_only + and not self.cuda_graph_runner.needs_capture(key) + ): + step_inputs.begin() with self._cuda_graph_metadata_prepare_scope( attn_metadata, key, @@ -4921,9 +5171,14 @@ def capture_postprocess_fn(inputs: dict[str, Any]): finally: restore_attn_metadata_after_draft_replay(attn_metadata, saved_draft) + # No-op unless a pack was opened above and the replay did not issue it. + step_inputs.flush() + if self.forward_pass_callable is not None: self.forward_pass_callable() + if isinstance(outputs, dict) and outputs.get(GREEDY_TOKENS_KEY) is not None: + outputs = self._logits_for_greedy_tokens(semantic_requests, outputs) self._execute_logit_post_processors(semantic_requests, outputs) if not isinstance(outputs, dict): @@ -5044,6 +5299,39 @@ def _apply_logits_processors( # back would be a self-assignment, which torch rejects for the # non-contiguous slices a TP-padded vocab produces. + def _logits_for_greedy_tokens( + self, scheduled_requests: ScheduledRequests, outputs: dict + ) -> dict: + """Replace greedy tokens sampled by the forward with logits when a request needs logits. + + A forward that samples greedily in place of the LM head returns the head's input + alongside the tokens. When any generation request needs more than its greedy token, + the logits come from that input eagerly, followed by the in-graph sampling and the + logits post-processing the forward skipped. Returns the outputs to use: ``outputs`` + itself, or a new dict (a replayed graph's outputs dict is reused by later replays). + """ + if ( + self.keep_greedy_only_outputs + and self.max_beam_width == 1 + and scheduled_requests.num_context_requests == 0 + # An idle attention-DP rank runs a dummy row for an empty batch, whose outputs are restored from logits. + and len(scheduled_requests.generation_requests) > 0 + and all( + greedy_tokens_usable(request) for request in scheduled_requests.generation_requests + ) + ): + return outputs + outputs = dict(outputs) + outputs.pop(GREEDY_TOKENS_KEY) + hidden_states = outputs.pop(GREEDY_HIDDEN_STATES_KEY) + # One row per generation request: every row is a last token. + outputs["logits"] = self.model.logits_processor.forward( + hidden_states, self.model.lm_head, None, True + ) + if self.sample_in_graph_callable is not None and self.mapping.is_last_pp_rank(): + self.sample_in_graph_callable(outputs) + return outputs + def _execute_logit_post_processors(self, scheduled_requests: ScheduledRequests, outputs: dict): """Apply logit post processors (in-place modify outputs Tensors) if any.""" @@ -5055,6 +5343,10 @@ def _execute_logit_post_processors(self, scheduled_requests: ScheduledRequests, return logits_tensor = outputs["logits"] + if logits_tensor is None: + # Greedy tokens sampled by the forward; _logits_for_greedy_tokens supplied + # logits instead whenever a request needed them. + return logits_row_offset = 0 request_groups = ( diff --git a/tensorrt_llm/_torch/pyexecutor/h2d_cache.py b/tensorrt_llm/_torch/pyexecutor/h2d_cache.py new file mode 100644 index 000000000000..cd7edb316bef --- /dev/null +++ b/tensorrt_llm/_torch/pyexecutor/h2d_cache.py @@ -0,0 +1,204 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Skip host-to-device metadata copies whose values the destination already holds. + +Steady decode re-uploads many small per-step metadata tensors whose values do not change (slot indices, query +lengths, prompt lengths, page tables between page boundaries). Each upload is a separate copy on the compute stream, +serialized between the decode graphs. An ``H2DSkipCache`` keeps one host shadow per destination buffer (the last +values written through it, and the view they were written to) and skips the copy when the same view receives the same +values again. + +Each engine that armed ``RuntimeHooks.skip_unchanged_h2d`` owns one cache and makes it the active one for its forward +(``active``); ``copy_h2d_if_changed`` acts on the calling context's active cache and, with none, is +``dst.copy_(src, non_blocking=True)``. The attention metadata that calls it has no engine to ask, which is why the +cache travels in the context rather than as an argument. + +Route a copy through it only for a destination that no other code writes (no device kernel, no other copy path, no other +engine): the shadow describes the device contents only under that invariant. Pooled metadata buffers satisfy it because +an engine that arms the skip gives its CUDA graph metadata a buffer pool of its own +(``RuntimeHooks.skips_unchanged_metadata_uploads``); eager metadata allocates its own buffers. A write through a +different view of the same buffer replaces the shadow, so overlapping views never skip on stale state. +""" + +from __future__ import annotations + +import contextlib +from contextvars import ContextVar +from typing import Iterator, Optional + +import torch + +from tensorrt_llm.logger import logger + +from . import step_inputs + + +class H2DSkipCache: + """One engine's host shadows of its skip-routed destinations (see the module docstring). + + ``changed_span`` (``RuntimeHooks.h2d_changed_span``): a packed copy into a 2-D view with contiguous rows whose + shadow covers the same rows (same first element and row stride) uploads, per row, only the span from its first to + its last changed column, plus any columns the shadow does not cover. A page table that gained one page then stages + a few bytes instead of every padded row, so the pack still fits the decode graph's prologue window. The span is + sound under the same invariant as the skip itself: the shadow describes the device bytes outside the span. + """ + + # A changed-span upload is used only while it moves at most this fraction of the view's bytes. + CHANGED_SPAN_MAX_FRACTION = 0.5 + # Class-level defaults: a cache built without __init__ never stages a span. + changed_span = False + # Engagement counters: changed-span uploads, the bytes they staged and the bytes the whole views would have staged. + span_uploads = 0 + span_bytes = 0 + span_full_bytes = 0 + + def __init__(self, changed_span: bool = False) -> None: + self.changed_span = changed_span + # Device storage address -> (the storage, kept alive so the address is not reused; the view key; the host + # values). Keyed by storage rather than by view base: tensors made under inference mode do not record their + # base. + self._shadow: dict[int, tuple[object, tuple, torch.Tensor]] = {} + # Device storage address -> (the storage, kept alive as above; the view key; (first row, value) of the view's + # last tail fill). + self._tails: dict[int, tuple[object, tuple, tuple[int, int]]] = {} + + def clear(self) -> None: + """Forget every shadow, so the next copy to each destination is issued (e.g. after a discarded step).""" + self._shadow.clear() + self._tails.clear() + + def copy(self, dst: torch.Tensor, src: torch.Tensor, pack: bool) -> bool: + storage = dst.untyped_storage() + addr = storage.data_ptr() + if src.is_cuda or not dst.is_cuda or torch.cuda.is_current_stream_capturing(): + self._shadow.pop(addr, None) + dst.copy_(src, non_blocking=True) + return True + key = (dst.storage_offset(), tuple(dst.shape), tuple(dst.stride()), dst.dtype) + entry = self._shadow.get(addr) + if ( + entry is not None + and entry[1] == key + and entry[2].dtype == src.dtype + and entry[2].shape == src.shape + and torch.equal(entry[2], src) + ): + return False + if not ( + pack + and self.changed_span + and entry is not None + and self._copy_changed_span(dst, src, entry[1:]) + ): + if pack: + step_inputs.copy_(dst, src) + else: + dst.copy_(src, non_blocking=True) + self._shadow[addr] = (storage, key, src.clone()) + return True + + def _copy_changed_span(self, dst: torch.Tensor, src: torch.Tensor, shadow: tuple) -> bool: + """Stage only the changed span of each row of ``dst`` (see the class docstring); False if not applicable.""" + (offset, shape, stride, dtype), old = shadow + if ( + dst.dim() != 2 + or dst.stride(1) != 1 + or offset != dst.storage_offset() + or len(shape) != 2 + or shape[0] != dst.shape[0] + or stride != tuple(dst.stride()) + or dtype != dst.dtype + or old.dtype != src.dtype + or tuple(src.shape) != tuple(dst.shape) + or not step_inputs.is_open() + ): + return False + cols = dst.shape[1] + known = min(shape[1], cols) + changed = src[:, :known] != old[:, :known] + if cols > known: + # Columns the shadow does not cover hold unknown device values. + changed = torch.cat([changed, changed.new_ones(dst.shape[0], cols - known)], dim=1) + rows = changed.any(dim=1).nonzero().flatten().tolist() + spans = [] + for row in rows: + columns = changed[row].nonzero().flatten() + spans.append((row, int(columns[0]), int(columns[-1]) + 1)) + staged = sum(hi - lo for _, lo, hi in spans) + if staged > self.CHANGED_SPAN_MAX_FRACTION * dst.numel(): + return False + for row, lo, hi in spans: + step_inputs.copy_(dst[row, lo:hi], src[row, lo:hi]) + self.span_uploads += 1 + self.span_bytes += staged * dst.element_size() + self.span_full_bytes += dst.numel() * dst.element_size() + if self.span_uploads & (self.span_uploads - 1) == 0: + logger.info( + f"[M3 host] changed-span uploads: {self.span_uploads}, {self.span_bytes} bytes staged " + f"of {self.span_full_bytes} for the whole views" + ) + return True + + def fill_tail(self, buf: torch.Tensor, start: int, value: int) -> None: + storage = buf.untyped_storage() + addr = storage.data_ptr() + if start >= buf.shape[0]: + # The live rows cover the whole view: the previous tail fill is overwritten. + self._tails.pop(addr, None) + return + if torch.cuda.is_current_stream_capturing(): + self._tails.pop(addr, None) + buf[start:].fill_(value) + return + key = (buf.storage_offset(), tuple(buf.shape), tuple(buf.stride()), buf.dtype) + entry = self._tails.get(addr) + if entry is not None and entry[1] == key and entry[2] == (start, value): + return + buf[start:].fill_(value) + self._tails[addr] = (storage, key, (start, value)) + + +_active: ContextVar[Optional[H2DSkipCache]] = ContextVar("trtllm_h2d_skip_cache", default=None) + + +@contextlib.contextmanager +def active(cache: Optional[H2DSkipCache]) -> Iterator[None]: + """Make ``cache`` the calling context's cache (None: no skipping) for the duration.""" + token = _active.set(cache) + try: + yield + finally: + _active.reset(token) + + +def enabled() -> bool: + """Whether the calling context skips unchanged copies.""" + return _active.get() is not None + + +def copy_h2d_if_changed(dst: torch.Tensor, src: torch.Tensor, pack: bool = False) -> bool: + """``dst.copy_(src, non_blocking=True)`` unless the active cache knows ``dst`` holds ``src``; True if copied. + + ``pack``: the copy may join an open ``step_inputs`` pack, i.e. land only at the pack's flush (the graph replay). + Pass it only for destinations no device work reads before that replay. + """ + cache = _active.get() + if cache is None: + dst.copy_(src, non_blocking=True) + return True + return cache.copy(dst, src, pack) + + +def fill_tail_if_changed(buf: torch.Tensor, start: int, value: int) -> None: + """``buf[start:].fill_(value)`` unless the active cache's last tail fill of ``buf`` was the same. + + Call it after every write of the live rows ``buf[:start]``, including one that covers the whole of ``buf`` + (``start == len(buf)``, which fills nothing): a live-row write past the recorded start overwrites part of the + tail. The skip is sound only while nothing else writes ``buf`` past ``start``. + """ + cache = _active.get() + if cache is None: + if start < buf.shape[0]: + buf[start:].fill_(value) + return + cache.fill_tail(buf, start, value) diff --git a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py index 8ee3b7dff9e8..080c05303dec 100644 --- a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py +++ b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py @@ -1284,6 +1284,14 @@ class KVCacheManagerV2(BaseResourceManager): # The lender sharing.attach_lender installs, one per manager for its life; None without one. # On the class so it exists without running __init__. _sharing = None + # Opt-in (RuntimeHooks.skip_unchanged_draft_block_offsets): copy_batch_block_offsets skips a decode-only copy + # whose destination, copied rows and the rows' host page indices equal the previous copy's into that + # destination (see _block_offsets_unchanged). On the class so it exists without running __init__. + skip_unchanged_block_offsets: bool = False + _block_offset_guard: Optional[dict] = None + _block_offset_host: Optional[np.ndarray] = None + block_offset_copies: int = 0 + block_offset_skips: int = 0 def __init__( self, @@ -6589,6 +6597,13 @@ def copy_batch_block_offsets( ) assert copy_idx.shape[0] == num_seqs + if ( + self.skip_unchanged_block_offsets + and not self._use_per_layer_page_tables + and self._block_offsets_unchanged(dst_tensor, copy_idx, num_contexts) + ): + return + if self._use_per_layer_page_tables: self._copy_batch_block_offsets_per_layer( dst_tensor, request_ids, copy_idx, num_contexts, num_seqs @@ -6604,6 +6619,59 @@ def copy_batch_block_offsets( self._stream.cuda_stream, ) + def _block_offsets_unchanged( + self, dst_tensor: torch.Tensor, copy_idx: torch.Tensor, num_contexts: int + ) -> bool: + """Whether the copy into ``dst_tensor`` would write exactly what the previous copy into it wrote. + + copy_batch_block_offsets_to_device writes, for each copied row i, dst[:, i, 0/1, :] from field 0 of host + row copy_idx[i] alone (the V entry is derived from the same page index) and the per-pool scales and offsets, + which are fixed at construction. So the copy can be skipped when the destination view (storage, offset, + shape, strides), the copied rows and those rows' field-0 page indices equal the last copy's into that + storage: the device bytes are then already the ones the copy would write. Every write into these + destinations goes through this method, and a copy into another view of the same storage replaces the + record. A step with context requests always copies (its rows are being filled). The compare is a + single-threaded numpy compare of the copied rows only (a torch compare of the whole table costs tens of us + on the host per step). + """ + guard = self._block_offset_guard + if guard is None: + guard = self._block_offset_guard = {} + storage = dst_tensor.untyped_storage().data_ptr() + if num_contexts != 0: + guard.pop(storage, None) + return self._count_block_offset_copy(skipped=False) + rows = copy_idx.tolist() + key = ( + dst_tensor.data_ptr(), + tuple(dst_tensor.shape), + tuple(dst_tensor.stride()), + tuple(rows), + ) + host = self._block_offset_host + if host is None: + host = self._block_offset_host = self.host_kv_cache_block_offsets.numpy() + current = host[:, rows, 0, :] + last = guard.get(storage) + if last is not None and last[0] == key and np.array_equal(last[1], current): + return self._count_block_offset_copy(skipped=True) + guard[storage] = (key, current) + return self._count_block_offset_copy(skipped=False) + + def _count_block_offset_copy(self, skipped: bool) -> bool: + """Count one guarded copy (``skipped``: not issued) and log the counts whenever the calls reach a power of + two; returns ``skipped``.""" + if skipped: + self.block_offset_skips += 1 + else: + self.block_offset_copies += 1 + calls = self.block_offset_skips + self.block_offset_copies + if calls & (calls - 1) == 0: + logger.info( + f"[M3 host] block-offset copies skipped when unchanged: {self.block_offset_skips} of {calls}" + ) + return skipped + @staticmethod def _derive_reuse_salt(cache_salt: str | None) -> int | None: """Derive ``ReuseScope.salt`` (int|None) from the ``cache_salt`` string. diff --git a/tensorrt_llm/_torch/pyexecutor/model_engine.py b/tensorrt_llm/_torch/pyexecutor/model_engine.py index 221f9145b17c..d87da7f1e6be 100644 --- a/tensorrt_llm/_torch/pyexecutor/model_engine.py +++ b/tensorrt_llm/_torch/pyexecutor/model_engine.py @@ -854,6 +854,11 @@ def _init_cuda_graph_lora_manager(self, lora_config: LoraConfig): if isinstance(self._runner, DecoderRunner): self._runner.init_cuda_graph_lora_manager(lora_config) + def drop_greedy_only_outputs(self) -> None: + """Have every batch return logits, for an executor-side guided decoder that reads them.""" + if isinstance(self._runner, DecoderRunner): + self._runner.keep_greedy_only_outputs = False + def set_guided_decoder(self, guided_decoder: CapturableGuidedDecoder) -> bool: if hasattr(self.model, "set_guided_decoder"): diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor.py b/tensorrt_llm/_torch/pyexecutor/py_executor.py index 9d179a02f620..87d960f4d2e5 100644 --- a/tensorrt_llm/_torch/pyexecutor/py_executor.py +++ b/tensorrt_llm/_torch/pyexecutor/py_executor.py @@ -114,6 +114,7 @@ get_from_waiting_queue, merge_requests) from .resource_manager import (NoFreeSlotsError, ResourceManager, ResourceManagerType, request_context) +from .runtime_hooks import runtime_hooks_of from .sampler import (AsyncWorkerMixin, Sampler, SamplerEvent, SampleState, SampleStateTensors) from .scheduler import (RequestScheduler, ScheduledRequests, @@ -379,6 +380,13 @@ class PendingEncoderStep: class PyExecutor: + # True while the overlap loop runs with the execution stream current + # (_overlap_loop_stream_scope). + _loop_on_execution_stream = False + # Whether the model opted into running the overlap loop on the execution + # stream (RuntimeHooks.loop_on_execution_stream); read once at construction. + _execution_stream_loop_armed = False + # Minimum number of async micro batches for async PP execution. # This is a trade-off between memory usage and performance. # If the number of micro batches is too small, the executor will spend too much time in synchronization. @@ -490,6 +498,8 @@ def __init__( # related modules self.resource_manager = resource_manager self.model_engine = model_engine + self._execution_stream_loop_armed = runtime_hooks_of( + model_engine.model).loop_on_execution_stream self._enable_adp_dummy_fixes = getattr(model_engine, "_enable_adp_dummy_fixes", False) self._enable_scheduler_aware_adp_dummy = getattr( @@ -5602,11 +5612,42 @@ def _wait_for_model_engine_input_copy(self): if wait_for_input_copy is not None: wait_for_input_copy() + @contextmanager + def _overlap_loop_stream_scope(self): + """Run the overlap loop with the execution stream current, when the model opted in. + + _forward_step makes the execution stream wait for the current stream + before every forward and the current stream wait for the execution + stream after it. On the thread's default stream, which in steady decode + carries only these waits and timing events, step N+1's inputs wait for + decode graph N through two cross-stream event hops. With the execution + stream current the loop's stream work is queued in the same order on one + stream and _forward_step skips both waits. + + Not taken with a two-model drafter, a guided decoder, a KV connector, + an encoder-decoder model or multimodal encoder item scheduling, whose + side streams synchronize with the current stream. + """ + if not (self._execution_stream_loop_armed and self.drafter is None and + self.draft_model_engine is None and self.guided_decoder is None + and self.kv_connector_manager is None + and not self.is_encoder_decoder + and not self._mm_encoder_item_scheduling_enabled): + yield + return + with torch.cuda.stream(self.execution_stream): + self._loop_on_execution_stream = True + try: + yield + finally: + self._loop_on_execution_stream = False + def _executor_loop_overlap(self): torch.cuda.set_device(self.device_id) # ensure the context is created, otherwise, some MPI calls will fail. CUASSERT(cudart.cudaSetDevice(self.device_id)) - with self._profiler() as profile_step, self.hang_detector: + with self._profiler() as profile_step, self.hang_detector, \ + self._overlap_loop_stream_scope(): iter_start_time = time.time() iter_stats = None target_inputs = None @@ -8311,8 +8352,10 @@ def forward(scheduled_requests, resource_manager, new_tensors_device, cache_indirection_buffer = self.sampler.get_cache_indirection() # Run model forward on the execution stream for proper synchronization - # with KVCacheTransferManager's onboard/offload operations. - self.execution_stream.wait_stream(torch.cuda.current_stream()) + # with KVCacheTransferManager's onboard/offload operations. With the + # loop on the execution stream both waits are program order. + if not self._loop_on_execution_stream: + self.execution_stream.wait_stream(torch.cuda.current_stream()) self._attach_encoder_output_to_execution_stream(scheduled_requests) with torch.cuda.stream(self.execution_stream): outputs = forward(scheduled_requests, self.resource_manager, @@ -8326,7 +8369,8 @@ def forward(scheduled_requests, resource_manager, new_tensors_device, # Ensure the default stream waits for execution_stream to complete # before downstream operations use the outputs. - torch.cuda.current_stream().wait_stream(self.execution_stream) + if not self._loop_on_execution_stream: + torch.cuda.current_stream().wait_stream(self.execution_stream) self._kv_connector_wait_for_save() if route_capture is not None: @@ -8455,9 +8499,11 @@ def _sample_async(self, scheduled_batch, beam_width = self.sampler.beam_width( scheduled_batch.all_requests()) + # A forward that sampled greedy tokens in place of the logits + # returns no logits; no request of such a batch asks for them. HandleLogits()(scheduled_batch.context_requests, scheduled_batch.generation_requests, - batch_outputs["logits"], beam_width, + batch_outputs.get("logits"), beam_width, num_context_logits_prefix_sum, self.sampler.is_generation_model()) diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py b/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py index 348328d75d96..84ef85352cf6 100644 --- a/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py +++ b/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py @@ -971,6 +971,7 @@ def allocation_scope(current_stage: ExecutorMemoryType): ): # GuidedDecoder is applicable to non-speculative decoding and two-model speculative decoding. guided_decoder = GuidedDecoder(**kwargs) + model_engine.drop_greedy_only_outputs() elif spec_config.spec_dec_mode.support_capturable_guided_decoder( ): # CapturableGuidedDecoder is applicable to one-model speculative decoding. diff --git a/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py b/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py new file mode 100644 index 000000000000..651777934a81 --- /dev/null +++ b/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py @@ -0,0 +1,255 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Step-loop behaviour a model may opt into, read once by the engine. + +A model declares its hooks as ``runtime_hooks`` (an instance attribute or a class +attribute). Every field defaults to off, and a model that declares nothing gets +``RuntimeHooks()``: the step loop, the samplers and the attention metadata then +run exactly as they do without this module. + +The hooks change *where* and *how* the step loop moves data, never what a +forward computes. Each one is armed only by a model that has measured it on its +own deployment; none is a general default. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING, Optional, Protocol + +if TYPE_CHECKING: + import torch + + +class SpecStepFold(Protocol): + """Speculative step-input staging folded into the previous step's forward. + + A one-model speculative step ends by storing its accepted tokens, draft + tokens and lengths into the sampler's per-slot store, and the next step + begins by gathering those values back into the engine's flat input buffers. + A model that implements this protocol does both inside its last draft pass: + it writes the sampler store and stages the next step's inputs directly, and + the engine and the sampler skip their copies on the steps where it did. + + The object is owned by the model; the engine and the sampler only call it. + Every method is host-side bookkeeping or a single kernel launch on the + current stream. + """ + + def bind_engine_buffers( + self, + input_ids: "torch.Tensor", + draft_tokens: "torch.Tensor", + position_id_offsets: "torch.Tensor", + kv_lens_offsets: "torch.Tensor", + max_batch_size: int, + ) -> None: + """Receive the engine's persistent speculative input buffers, once, at construction.""" + + def bind_sampler_store(self, store: object) -> None: + """Receive the speculative sampler's persistent per-slot store, once, at construction.""" + + def begin_step(self, seq_slots: Optional[list[int]], num_rows: int) -> bool: + """Start a step of ``num_rows`` generation rows. + + ``seq_slots`` are those rows' sequence slots on a decode-only step whose + rows all have one, else None: the forward then stores nothing for them. + Returns whether the previous step's forward staged its inputs for + ``seq_slots``; the engine additionally checks that this step adds no + inputs of its own before it skips its gathers. + """ + + def gather( + self, + new_tokens: "torch.Tensor", + next_draft_tokens: "torch.Tensor", + new_tokens_lens: "torch.Tensor", + previous_slots: "torch.Tensor", + previous_pos_indices: "torch.Tensor", + previous_batch_len: int, + tokens_per_gen_step: int, + draft_token_buffer_width: int, + num_tokens: int, + num_draft_tokens: int, + first_previous_row: int, + ) -> bool: + """Do the engine's previous-batch gathers in one launch. + + Writes the engine buffers exactly as the engine's own gathers would, + including zeroing both offset buffers outside the gathered range. + Returns False, having written nothing, when it cannot take the step. + """ + + def sampler_rows_stored(self, outputs: dict, num_skip: int, seq_slots: list[int]) -> bool: + """Whether the forward already stored this step's outputs for ``seq_slots`` into the sampler store.""" + + def scatter( + self, + seq_slots: "torch.Tensor", + new_tokens: "torch.Tensor", + next_new_tokens: "torch.Tensor", + new_tokens_lens: "torch.Tensor", + next_draft_tokens: "torch.Tensor", + ) -> bool: + """Store the step's outputs into the sampler store in one launch. + + Returns False, having written nothing, when it cannot take the step. + """ + + +class MsaKvUpdate(Protocol): + """Advance the MSA decode kernels' per-request state after the KV lengths moved. + + Called by ``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` with the + metadata's own buffers; recomputes the attended lengths, the new tokens' + cache slots and their valid-block counts in one launch, exactly as the + metadata's own update does. Returns False, having written nothing, when it + cannot take the step. + """ + + def __call__( + self, + kv_lens: "torch.Tensor", + kv_lens_staged: "torch.Tensor", + seq_lens: "torch.Tensor", + q_batch_row: "torch.Tensor", + q_intra: "torch.Tensor", + block_table: "torch.Tensor", + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: "torch.Tensor", + cu_kv_lens: Optional["torch.Tensor"], + out_cache_loc: "torch.Tensor", + n_valid_blocks: Optional["torch.Tensor"], + ) -> bool: ... + + +@dataclass(frozen=True) +class MsaRuntimePolicy: + """MiniMax-M3 MSA metadata preparation a model may opt into. + + Every option changes how a step's selection and cache-write tables reach + the device, not which blocks are attended. + """ + + # A step whose every query token has at most top-k valid blocks attends all + # of them, so its selection is the identity table: prepare stages that table + # instead of a scoring plan, and decode CUDA graphs are keyed by short and + # long contexts. + short_sequence_identity: bool = False + # The model's one-launch replacement for the update after the KV lengths move. + kv_update: Optional[MsaKvUpdate] = None + # Skip a block-table copy whose inputs equal the previous step's. + skip_unchanged_block_offsets: bool = False + # Build each KV pool view once per pool geometry instead of on every attention call. + reuse_pool_views: bool = False + + +@dataclass(frozen=True) +class RuntimeHooks: + """The step-loop hooks a model opts into. All off by default.""" + + # Pack a decode step's small host-to-device input copies into one upload and + # one scatter launch, issued just before the CUDA graph replay. + pack_step_inputs: bool = False + # Issue the step-input pack's host copies with one parallel scatter (a program per copy) instead of the + # serial one-CTA scatter, when their destinations do not overlap (pack_step_inputs). + pack_parallel_scatter: bool = False + # Issue the step-input pack inside the replayed graph (a prologue copies a pinned window and scatters it), so no + # operation is left between the decode graphs (needs pack_parallel_scatter). + pack_in_graph: bool = False + # With pack_in_graph: a side copy stream uploads the pinned window into the prologue's own device buffer while the + # previous graph still runs, so the prologue only waits on that copy and scatters device to device (no host copy + # at the head of the replay). + pack_early_copy: bool = False + # Skip host-to-device metadata copies whose values the destination already holds. + skip_unchanged_h2d: bool = False + # With skip_unchanged_h2d: a changed packed 2-D upload stages only each row's changed column span (a page table that + # gained a page re-sends a few bytes, not every padded row), so the pack keeps fitting the in-graph window. + h2d_changed_span: bool = False + # A speculative step under the overlap scheduler re-derives its KV-length metadata (on_update_kv_lens) once, + # after the forward's KV-length correction, instead of also before it: the first pass's device writes are all + # rewritten by the second. + kv_lens_update_once: bool = False + # Run the overlap executor loop with the execution stream current, so the + # loop's own stream work and the forward are ordered on one stream. + loop_on_execution_stream: bool = False + # After autotuning, keep the fastest profiled tactic per shape on every + # tensor-parallel rank, so the ranks capture the same kernels. + merge_tuned_tactics_across_ranks: bool = False + # Skip the per-step Philox seed and offset uploads on all-greedy + # one-model speculative batches (the argmax path never reads them). + skip_greedy_rng_upload: bool = False + # Stage the one-model speculative sampler's per-step Philox seed and offset + # uploads into the step-input pack (pack_step_inputs), so they ride its one + # upload instead of each being a copy the next graph waits for. + pack_spec_rng_uploads: bool = False + # The one-model speculative step's small per-step uploads: the accepted-draft + # counts join the step-input pack, and the spec sampler keeps its slot ids in + # a device buffer it re-uploads only when they change. + spec_small_h2d: bool = False + # Skip the per-step accepted-draft-count upload (spec_metadata.num_accepted_draft_tokens is then None). Only for a + # model whose forward and drafter never read that tensor. + skip_accepted_draft_upload: bool = False + # The one-model speculative metadata re-uploads its per-request slot-id table (batch_slot_ids, read by rejection + # sampling and the penalties) only when the batch's slot ids change (steady decode keeps them), instead of + # building a pinned tensor and copying it every step. + spec_slot_ids_changed_only: bool = False + # The DFlash / standalone DSpark drafter re-uploads its batch_indices and batch-to-slot map only when they change + # (steady decode keeps both), instead of copying both every step. + spec_batch_map_changed_only: bool = False + # Speculative step-input staging folded into the forward (see SpecStepFold). + spec_step_fold: Optional[SpecStepFold] = None + # Copy the one-model speculative sampler's per-slot store to the host on the sampler's D2H side + # stream, so those copies overlap the next step's input staging instead of preceding it. + sampler_d2h_side_stream: bool = False + # MiniMax-M3 MSA metadata preparation. + msa: MsaRuntimePolicy = MsaRuntimePolicy() + # A separate draft KV cache manager (KVCacheManagerV2) skips a decode-only block-table copy whose destination, + # copied rows and those rows' page indices equal the previous copy's (the standalone DSpark drafter's table is + # unchanged on most decode steps; see KVCacheManagerV2._block_offsets_unchanged). + skip_unchanged_draft_block_offsets: bool = False + # The one-model speculative sampler's tail bookkeeping without its no-op launches, bitwise the same outputs + # (copied to spec_metadata.tail_bookkeep): at one decode request the draft block reads its sampling parameters as + # views of the per-token buffers, a draft block with no d2t map skips the int32 -> int64 -> int32 round trip, and + # a decode-only rejection accept returns the acceptance kernel's outputs instead of copying them into fresh ones. A + # model's own worker may drop more of them under the same flag. + spec_tail_bookkeep: bool = False + # Under disaggregated serving, ship a standalone drafter's context window (its separate draft + # KV pool) alongside the target's KV, so the generation worker drafts from the prompt instead + # of an empty window. Takes effect only with a separate draft KV cache manager on both sides + # (see disaggregation/draft_kv.py). + disagg_draft_kv: bool = False + + @property + def skips_unchanged_metadata_uploads(self) -> bool: + """Whether a hook skips uploads into graph metadata buffers that already hold the values. + + Such a skip is sound only while no other engine writes those buffers, so the engine + then gives its CUDA graph metadata a buffer pool of its own. + """ + return ( + self.skip_unchanged_h2d + or self.msa.skip_unchanged_block_offsets + or self.skip_unchanged_draft_block_offsets + ) + + +DEFAULT_RUNTIME_HOOKS = RuntimeHooks() + + +def runtime_hooks_of(model: object) -> RuntimeHooks: + """The hooks ``model`` declares, or the all-off default when it declares none. + + Declaring hooks is optional for every model class, so absence is the common + case and means the default. + """ + hooks = getattr(model, "runtime_hooks", None) + if hooks is None: + return DEFAULT_RUNTIME_HOOKS + if not isinstance(hooks, RuntimeHooks): + raise TypeError( + f"{type(model).__name__}.runtime_hooks must be a RuntimeHooks, got {type(hooks).__name__}" + ) + return hooks diff --git a/tensorrt_llm/_torch/pyexecutor/sampler/ops/flashinfer.py b/tensorrt_llm/_torch/pyexecutor/sampler/ops/flashinfer.py index ae43820dd1df..7b46e2a09c6f 100644 --- a/tensorrt_llm/_torch/pyexecutor/sampler/ops/flashinfer.py +++ b/tensorrt_llm/_torch/pyexecutor/sampler/ops/flashinfer.py @@ -63,6 +63,40 @@ def __getattr__(self, name: str) -> Any: SeedOrTensor = Union[int, torch.Tensor] +# An optional replacement of ``flashinfer.sampling.top_p_renorm_probs`` with +# per-row ``top_p`` tensors, installed by a model at load time: it returns the +# renormed probs, or None to keep the stock call for that input. None (the +# default) leaves every op below unchanged. +_TOP_P_RENORM_OVERRIDE: Optional[Callable[[torch.Tensor, torch.Tensor], Optional[torch.Tensor]]] = ( + None +) + + +def set_top_p_renorm_override( + fn: Optional[Callable[[torch.Tensor, torch.Tensor], Optional[torch.Tensor]]], +) -> None: + """Install (or, with None, remove) the top-p renorm override used by + ``top_p_renorm_probs_op`` and ``compute_probs_from_logits_op``.""" + global _TOP_P_RENORM_OVERRIDE + _TOP_P_RENORM_OVERRIDE = fn + + +# An optional replacement of ``flashinfer.sampling.softmax(logits, temperature)``, +# installed by a model at load time: it returns the probs, or None to keep the +# stock call for that input. None (the default) leaves every op below unchanged. +_SOFTMAX_OVERRIDE: Optional[ + Callable[[torch.Tensor, Optional[torch.Tensor]], Optional[torch.Tensor]] +] = None + + +def set_softmax_override( + fn: Optional[Callable[[torch.Tensor, Optional[torch.Tensor]], Optional[torch.Tensor]]], +) -> None: + """Install (or, with None, remove) the softmax override used by + ``softmax_op`` and ``compute_probs_from_logits_op``.""" + global _SOFTMAX_OVERRIDE + _SOFTMAX_OVERRIDE = fn + @_compiler_disable def radix_topk_op( @@ -241,6 +275,10 @@ def softmax_op( logits: torch.Tensor, temperature: Optional[torch.Tensor], ) -> torch.Tensor: + if _SOFTMAX_OVERRIDE is not None: + overridden = _SOFTMAX_OVERRIDE(logits, temperature) + if overridden is not None: + return overridden probs: torch.Tensor = flashinfer.sampling.softmax( logits, temperature, enable_pdl=get_env_enable_pdl() ) @@ -270,6 +308,10 @@ def top_p_renorm_probs_op( probs: torch.Tensor, top_p: torch.Tensor, ) -> torch.Tensor: + if _TOP_P_RENORM_OVERRIDE is not None: + overridden = _TOP_P_RENORM_OVERRIDE(probs, top_p) + if overridden is not None: + return overridden renormed: torch.Tensor = flashinfer.sampling.top_p_renorm_probs(probs, top_p) return renormed @@ -289,11 +331,21 @@ def compute_probs_from_logits_op( """ if top_k is not None: logits = flashinfer.sampling.top_k_mask_logits(logits, top_k) - probs: torch.Tensor = flashinfer.sampling.softmax( - logits, temperatures, enable_pdl=get_env_enable_pdl() + softmaxed = _SOFTMAX_OVERRIDE(logits, temperatures) if _SOFTMAX_OVERRIDE is not None else None + probs: torch.Tensor = ( + softmaxed + if softmaxed is not None + else flashinfer.sampling.softmax(logits, temperatures, enable_pdl=get_env_enable_pdl()) ) if top_p is not None: - probs = flashinfer.sampling.top_p_renorm_probs(probs, top_p) + overridden = ( + _TOP_P_RENORM_OVERRIDE(probs, top_p) if _TOP_P_RENORM_OVERRIDE is not None else None + ) + probs = ( + overridden + if overridden is not None + else flashinfer.sampling.top_p_renorm_probs(probs, top_p) + ) return probs diff --git a/tensorrt_llm/_torch/pyexecutor/sampler/sampler.py b/tensorrt_llm/_torch/pyexecutor/sampler/sampler.py index 2464827a1a9a..5bc63d9b0e5a 100644 --- a/tensorrt_llm/_torch/pyexecutor/sampler/sampler.py +++ b/tensorrt_llm/_torch/pyexecutor/sampler/sampler.py @@ -145,6 +145,34 @@ class SampleState(Generic[GenericSampleStateTensorsHost, GenericSampleStateTenso # Generic bounds not supported, https://github.com/python/typing/issues/548 GenericSampleState = TypeVar("GenericSampleState", bound=SampleState) # type: ignore +# Model output keys of a forward that samples greedily in place of its LM head: per-row greedy token ids, and the +# head's input rows so the logits can still be computed for a batch that needs them. Such a forward returns no +# logits. +GREEDY_TOKENS_KEY = "greedy_tokens" +GREEDY_HIDDEN_STATES_KEY = "greedy_hidden_states" + + +def greedy_tokens_usable(request: LlmRequest) -> bool: + """Whether a generation request is served by the forward's greedy token alone, without logits. + + The request conditions of TorchSampler's stable greedy path, plus every other reader of the logits. A dummy + request is not usable: the stable greedy path takes no batch that carries one. + """ + return not request.is_dummy and ( + get_draft_token_length(request) == 0 + and request.py_embedding_bias is None + and not getattr(request, "py_bad_words", None) + and not getattr(request, "py_no_repeat_ngram_size", None) + and not has_occurrence_penalty(request) + and not request.py_min_length + and not request.py_return_log_probs + and not request.py_return_generation_logits + and not request.py_logits_post_processors + and request.py_guided_decoding_params is None + and not check_stop_words_length(request) + and _request_strategy(request, vocab_size=2**31) == GREEDY + ) + def _fast_strategy_params(strategy: Strategy) -> tuple[float, int, float, float]: """Flatten a fast-tier strategy into per-row ``(temperature, top_k, top_p, min_p)``.""" @@ -2513,7 +2541,11 @@ def _process_requests( torch.Tensor, bool, ]: - raw_logits_cuda = model_outputs["logits"] + # A forward that sampled greedy tokens in place of the logits (GREEDY_TOKENS_KEY) may return no logits. + raw_logits_cuda = model_outputs.get("logits") + greedy_tokens = model_outputs.get(GREEDY_TOKENS_KEY) + rows_cuda = raw_logits_cuda if greedy_tokens is None else greedy_tokens + assert rows_cuda is not None, "Expected logits or in-graph greedy tokens" generation_requests = scheduled_requests.generation_requests request_ids = [request.py_request_id for request in generation_requests] @@ -2526,7 +2558,7 @@ def _process_requests( bool(generation_requests) and self.max_beam_width == 1 and scheduled_requests.num_context_requests == 0 - and len(generation_requests) <= raw_logits_cuda.shape[0] + and len(generation_requests) <= rows_cuda.shape[0] and model_outputs.get("d2t") is None and all( not request.is_dummy and get_draft_token_length(request) == 0 @@ -2563,6 +2595,9 @@ def _process_requests( live_requests = [request for request in generation_requests if not request.is_dummy] staged_matches_batch = ( self._in_graph_staged + # A forward that sampled greedily in place of its LM head skipped the + # in-graph sampling, so the staged buffers were never written. + and greedy_tokens is None and self._in_graph_dest_indices is not None # A draft request reuses its target's py_request_id, so the id # list alone cannot tell a drafter batch from the target batch @@ -2660,13 +2695,20 @@ def _process_requests( self._stable_greedy_seq_slots_host = seq_slots_host self._stable_greedy_seq_slots_cuda = seq_slots_cuda - next_tokens = fast_greedy_sample_kernel( - raw_logits_cuda[: len(generation_requests)], - new_tokens_cuda, - seq_slots_cuda, - self.max_beam_width, - None, - ) + if greedy_tokens is not None: + next_tokens = greedy_tokens[: len(generation_requests)].to(new_tokens_cuda.dtype) + scatter_new_tokens( + next_tokens, new_tokens_cuda, seq_slots_cuda, self.max_beam_width + ) + else: + assert raw_logits_cuda is not None + next_tokens = fast_greedy_sample_kernel( + raw_logits_cuda[: len(generation_requests)], + new_tokens_cuda, + seq_slots_cuda, + self.max_beam_width, + None, + ) new_tokens_host = self._copy_to_host(next_tokens) return ( generation_requests, @@ -2681,6 +2723,10 @@ def _process_requests( self._stable_greedy_request_ids = [] self._stable_greedy_seq_slots = [] + if raw_logits_cuda is None: + raise RuntimeError( + "the forward returned greedy tokens without logits for a batch that needs logits" + ) sampling_requests, sampling_requests_metadata, logits_cuda = self._select_generated_logits( scheduled_requests, raw_logits_cuda, diff --git a/tensorrt_llm/_torch/pyexecutor/step_inputs.py b/tensorrt_llm/_torch/pyexecutor/step_inputs.py new file mode 100644 index 000000000000..4207c341cce7 --- /dev/null +++ b/tensorrt_llm/_torch/pyexecutor/step_inputs.py @@ -0,0 +1,471 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Pack a decode step's small per-step input copies into one upload and one scatter launch. + +Between two decode-graph replays, the next step's metadata reaches the device as a chain of tiny copies (positions, +KV lengths, cache slots, and the graph runner's static-input copies). Each is a separate operation on the stream the +next graph waits for, so the GPU idles for the whole chain. + +While a pack is open (``begin`` ... ``flush``), ``copy_(dst, src)`` records a host-to-device copy by snapshotting the +host bytes into a pinned staging buffer, and a device-to-device copy as a (dst, src) descriptor. ``flush`` issues one +host-to-device copy of [descriptors | data] into a device staging buffer and one ``trtllm::step_inputs_scatter`` +launch, which writes every staged host copy and then every device copy (in that order, so a device copy may read a +destination a host copy just filled). With no pack open, ``copy_`` is ``dst.copy_(src, non_blocking=True)``. + +Each engine that armed ``RuntimeHooks.pack_step_inputs`` owns one ``StepInputPack`` (on its device) and makes it the +active pack for its forward (``active``); the module functions act on the active pack of the calling context, so +executors on other threads or devices never share one. The engine opens it only when the step will reach the flush (a +replay of an already captured graph). Route a copy through ``copy_`` only when no device work reads or writes its +destination between the copy site and the flush: the pack moves the copy to the flush point. A copy that cannot be +packed (non-contiguous, device-side dtype change) flushes the pack first and then runs as a plain copy, which keeps +the original order. A pack holds every tensor it staged until it is issued, so a temporary source outlives its copy. + +With ``parallel_scatter`` (``RuntimeHooks.pack_parallel_scatter``) a pack of host copies only, whose destinations do +not overlap, is issued with one program per copy (``_parallel_scatter_kernel``) instead of the serial scatter: the same +bytes land in the same places, and with no overlap their order cannot matter. Any other pack takes the serial scatter. + +With ``in_graph`` (``RuntimeHooks.pack_in_graph``, on top of ``parallel_scatter``) the issue itself moves into the +replayed graph: a graph captured with ``capture_prologue`` starts by copying a fixed pinned window (``ghost``) to the +device staging buffer, records ``ghost_free``, and runs the parallel scatter with the copy counts read from the staged +header (one program per descriptor slot; the slots past the counts write nothing). ``flush(in_graph=True)`` before +such a replay then only fills the window on the host, after ``ghost_free`` shows the previous replay has copied it: no +operation is left between the graphs. A pack that does not fit the window or is not order-free is issued eagerly as +before, with a zero-count window so the graph's prologue writes nothing. + +With ``early_copy`` (``RuntimeHooks.pack_early_copy``, on top of ``in_graph``) the window reaches the device before the +replay: ``flush(in_graph=True)`` fills the pinned window and enqueues its upload on a side copy stream, into a device +buffer of the prologue's own (``gdev``), while the previous graph may still run. The prologue then waits for that upload +(``staged_ready``) and scatters device to device; it zeroes the window's header after the scatter and records +``prologue_done``, which the next upload waits for before it overwrites ``gdev``. The next fill of the pinned window +waits for ``ghost_free``, which the upload records once it has read the window. + +Every in-graph attribute has a class-level default (off), so a pack built without its ``__init__`` and every pack built +with both flags off takes exactly the path above (``flush()`` and ``flush(in_graph=True)`` are then the same call). +""" + +from __future__ import annotations + +import contextlib +from contextvars import ContextVar +from typing import Iterator, Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._utils import prefer_pinned +from tensorrt_llm.logger import logger + + +# first (the copy index offset) varies with the pack's host-copy count: not specialized on its value, so the two +# DEVICE_SRC variants compiled by the warm-up serve every pack (no compile on a serving step). +@triton.jit(do_not_specialize=["first"]) +def _parallel_scatter_kernel( + stage_u8, stage_i64, first, DEVICE_SRC: tl.constexpr, BLOCK: tl.constexpr +): + """Program i writes copy ``first + i`` ([dst, src, bytes] descriptors after the 2-word header). ``src`` is an offset + into the staged buffer for a host copy, a device address for a device copy (``DEVICE_SRC``).""" + d = 2 + 3 * (first + tl.program_id(0)) + dst = tl.load(stage_i64 + d).to(tl.pointer_type(tl.uint8)) + src_word = tl.load(stage_i64 + d + 1) + if DEVICE_SRC: + src = src_word.to(tl.pointer_type(tl.uint8)) + else: + src = stage_u8 + src_word + nbytes = tl.load(stage_i64 + d + 2) + for start in range(0, nbytes, BLOCK): + o = start + tl.arange(0, BLOCK) + tl.store(dst + o, tl.load(src + o, mask=o < nbytes), mask=o < nbytes) + + +# The in-graph prologue's scatter: no integer argument (the counts are read from the staged header and the grid is the +# fixed MAX_DESC), so the two DEVICE_SRC variants its warm-up compiles serve every replay and every capture. +@triton.jit +def _graph_scatter_kernel(stage_u8, stage_i64, DEVICE_SRC: tl.constexpr, BLOCK: tl.constexpr): + """Descriptor slot i of the host copies (``DEVICE_SRC`` False) or of the device copies, with the counts read from + the staged header; a slot past its count writes nothing.""" + i = tl.program_id(0) + num_h2d = tl.load(stage_i64) + if DEVICE_SRC: + count = tl.load(stage_i64 + 1) + d = 2 + 3 * (num_h2d + i) + else: + count = num_h2d + d = 2 + 3 * i + if i < count: + dst = tl.load(stage_i64 + d).to(tl.pointer_type(tl.uint8)) + src_word = tl.load(stage_i64 + d + 1) + if DEVICE_SRC: + src = src_word.to(tl.pointer_type(tl.uint8)) + else: + src = stage_u8 + src_word + nbytes = tl.load(stage_i64 + d + 2) + for start in range(0, nbytes, BLOCK): + o = start + tl.arange(0, BLOCK) + tl.store(dst + o, tl.load(src + o, mask=o < nbytes), mask=o < nbytes) + + +def _disjoint(ranges: list[tuple[int, int]]) -> bool: + """Whether the [start, start + bytes) ranges are pairwise disjoint.""" + end = -1 + for start, nbytes in sorted(ranges): + if start < end: + return False + end = start + nbytes + return True + + +def _parallel_ok(h2d: list[tuple[int, int, int]], d2d: list[tuple[int, int, int]]) -> bool: + """Whether issuing the host copies in parallel and then the device copies in parallel writes what the serial + scatter (every copy in order) writes: the host copies' destinations are disjoint, and the device copies' + destinations are disjoint and none of them is a device copy's source (a host copy may still feed a device copy, + as the launches run in order).""" + if not _disjoint([(dst, n) for dst, _, n in h2d]): + return False + if not d2d: + return True + dsts = [(dst, n) for dst, _, n in d2d] + if not _disjoint(dsts): + return False + return all(_disjoint(dsts + [(src, n)]) for _, src, n in d2d) + + +class StepInputPack: + """One engine's pack of deferred step-input copies (see the module docstring).""" + + CAPACITY = 1 << 16 + MAX_DESC = 256 + # Pinned staging buffers in rotation: a forced flush reopens the pack without waiting on the upload just issued. + RING = 4 + HEADER_BYTES = 16 + DESC_BYTES = 24 + ALIGN = 16 + # The pinned window a graph's prologue copies (in_graph): the header, the descriptors and up to ~10 KB of data. + GRAPH_WINDOW = 1 << 14 + + # Class-level defaults: a pack built without __init__ (or with the flags off) never takes an in-graph branch. + parallel_scatter = False + in_graph = False + early_copy = False + # Engagement counters of this pack (one per engine and rank): packs the replay's prologue issued, packs issued + # eagerly before a prologue graph (over the window or not order-free), and side-stream window uploads + # (early_copy). Logged whenever the in-graph flushes (windows + fallbacks) reach a power of two. + graph_windows = 0 + eager_fallbacks = 0 + early_uploads = 0 + + def __init__( + self, + device: torch.device, + parallel_scatter: bool = False, + in_graph: bool = False, + early_copy: bool = False, + ) -> None: + # Only an opted-in pack needs the native scatter fake registration. + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 import ( # noqa: F401 + fake_ops as _m3_fake_ops, # registers trtllm::step_inputs_scatter's fake impl + ) + + self.parallel_scatter = parallel_scatter + self.in_graph = in_graph and parallel_scatter + self.early_copy = early_copy and self.in_graph + if self.early_copy: + # The prologue's own device window: an eager flush (self.dev) can never leave a header in it. + self.gdev = torch.zeros(self.GRAPH_WINDOW, dtype=torch.uint8, device=device) + self.copy_stream = torch.cuda.Stream(device=device) + # Recorded by the side upload once it landed (the prologue waits on it) ... + self.staged_ready = torch.cuda.Event(external=True) + # ... and by every replayed prologue after its scatter (the next upload waits on it). + self.prologue_done = torch.cuda.Event(external=True) + # A CUDA event is created on its first record, and a wait on one never created is skipped: a capture + # would then leave the prologue's wait out of the graph. Create both now. + self.staged_ready.record(self.copy_stream) + self.prologue_done.record(self.copy_stream) + if self.in_graph: + self.ghost = torch.zeros( + self.GRAPH_WINDOW, dtype=torch.uint8, pin_memory=prefer_pinned() + ) + # Recorded by every replayed prologue right after it copied the window (an event node), or by the side + # upload once it read the window (early_copy). + self.ghost_free = torch.cuda.Event(external=True) + self.host = [ + torch.empty(self.CAPACITY, dtype=torch.uint8, pin_memory=prefer_pinned()) + for _ in range(self.RING) + ] + self.events: list[Optional[torch.cuda.Event]] = [None] * self.RING + self.dev = torch.empty(self.CAPACITY, dtype=torch.uint8, device=device) + self.data_start = self.HEADER_BYTES + self.DESC_BYTES * self.MAX_DESC + self.parity = 0 + self.open = False + self.h2d: list[tuple[int, int, int]] = [] + self.d2d: list[tuple[int, int, int]] = [] + # Every tensor a staged copy reads or writes, kept alive until the pack is issued. + self.staged: list[torch.Tensor] = [] + self.used = self.data_start + if parallel_scatter: + self._warm_up(device) + if self.in_graph: + self._warm_up_graph_kernel() + + def _warm_up(self, device: torch.device) -> None: + """Compile the parallel scatter now, not on a serving step's flush.""" + dst = torch.zeros(8, dtype=torch.int32, device=device) + self.begin() + self.stage(dst[:4], torch.arange(4, dtype=torch.int32)) + self.stage(dst[4:], dst[:4]) + self.flush() + torch.cuda.synchronize(device) + + def _warm_up_graph_kernel(self) -> None: + """Compile both variants of the prologue's scatter now, not inside a capture (a zero-count header: it writes + nothing).""" + self.dev[: self.HEADER_BYTES].zero_() + self._launch_graph_scatter(self.dev) + torch.cuda.synchronize(self.dev.device) + + def _launch_graph_scatter(self, window: torch.Tensor) -> None: + stage_i64 = window.view(torch.int64) + for device_src in (False, True): + _graph_scatter_kernel[(self.MAX_DESC,)]( + window, stage_i64, DEVICE_SRC=device_src, BLOCK=256, num_warps=1 + ) + + def capture_prologue(self) -> bool: + """Inside a graph capture: the prologue that issues the window ``flush(in_graph=True)`` fills (see the module + docstring). Returns whether the graph has it.""" + if not self.in_graph: + return False + if self.early_copy: + torch.cuda.current_stream().wait_event(self.staged_ready) + self._launch_graph_scatter(self.gdev) + # Consumed: a replay with no new upload scatters nothing. + self.gdev[: self.HEADER_BYTES].zero_() + self.prologue_done.record() + return True + self.dev[: self.GRAPH_WINDOW].copy_(self.ghost, non_blocking=True) + self.ghost_free.record() + self._launch_graph_scatter(self.dev) + return True + + def _upload_window(self, nbytes: int) -> None: + """early_copy: upload the filled pinned window into gdev on the side stream, once the last replayed prologue + has consumed gdev; the window is fully written on the host before this is enqueued.""" + stream = self.copy_stream + stream.wait_event(self.prologue_done) + with torch.cuda.stream(stream): + self.gdev[:nbytes].copy_(self.ghost[:nbytes], non_blocking=True) + self.ghost_free.record(stream) + self.staged_ready.record(stream) + + def _ghost_writable(self) -> None: + # The last replayed prologue (or side upload) has read the window (a no-op before the first one). + self.ghost_free.synchronize() + + def quiesce_graph_window(self) -> None: + """Before replaying a graph with the prologue without a flush: a zero-count window, so it writes nothing.""" + if self.in_graph and self.ghost[: self.HEADER_BYTES].any(): + self._ghost_writable() + self.ghost[: self.HEADER_BYTES].zero_() + if self.early_copy: + # A window uploaded but never replayed would still sit in gdev: overwrite its header. + self._upload_window(self.HEADER_BYTES) + + def _count(self, in_window: bool) -> None: + """Count one in-graph flush (and its early upload); log the counters whenever the number of in-graph flushes + reaches a power of two (per rank: one pack per engine).""" + if in_window: + self.graph_windows += 1 + self.early_uploads += int(self.early_copy) + else: + self.eager_fallbacks += 1 + flushes = self.graph_windows + self.eager_fallbacks + if flushes & (flushes - 1) == 0: + logger.info( + f"[M3 host] step-input pack engagement: {self.graph_windows} in-graph windows, " + f"{self.eager_fallbacks} eager fallbacks, {self.early_uploads} early uploads" + ) + + def begin(self) -> None: + if self.open: + self.flush() + event = self.events[self.parity] + if event is not None: + # The upload that last read this pinned buffer has run. + event.synchronize() + self._clear() + self.open = True + + def _clear(self) -> None: + self.h2d.clear() + self.d2d.clear() + self.staged.clear() + self.used = self.data_start + + def stage(self, dst: torch.Tensor, src: torch.Tensor) -> bool: + if not dst.is_cuda or dst.numel() != src.numel(): + return False + nbytes = dst.numel() * dst.element_size() + if src.device.type == "cpu": + # A contiguous destination is one descriptor; a 2-D slice with contiguous rows (a page table's leading + # rows and columns) is one per row. + if dst.is_contiguous(): + rows, row_bytes, row_stride = 1, nbytes, 0 + elif dst.dim() == 2 and dst.stride(1) == 1: + rows = dst.shape[0] + row_bytes = dst.shape[1] * dst.element_size() + row_stride = dst.stride(0) * dst.element_size() + else: + return False + offset = (self.used + self.ALIGN - 1) // self.ALIGN * self.ALIGN + if ( + offset + nbytes > self.CAPACITY + or len(self.h2d) + len(self.d2d) + rows > self.MAX_DESC + ): + return False + # A dtype change converts on the host, as copy_ would on the device. + src = src.to(dst.dtype).contiguous() + self.host[self.parity][offset : offset + nbytes].copy_(src.view(-1).view(torch.uint8)) + base = dst.data_ptr() + for row in range(rows): + self.h2d.append((base + row * row_stride, offset + row * row_bytes, row_bytes)) + self.staged.append(dst) + self.used = offset + nbytes + return True + if len(self.h2d) + len(self.d2d) >= self.MAX_DESC: + return False + if src.is_cuda and src.dtype == dst.dtype and src.is_contiguous() and dst.is_contiguous(): + self.d2d.append((dst.data_ptr(), src.data_ptr(), nbytes)) + self.staged += (dst, src) + return True + return False + + def flush(self, in_graph: bool = False) -> None: + """Issue the pack. ``in_graph``: the next replay is of a graph captured with ``capture_prologue``.""" + if not self.open: + if in_graph: + self.quiesce_graph_window() + return + self.open = False + if in_graph and self.in_graph: + if self._fill_graph_window(): + if self.early_copy: + self._upload_window(self.used) + self._count(in_window=True) + self._clear() + return + self.quiesce_graph_window() + self._count(in_window=False) + if not self.h2d and not self.d2d: + return + host = self.host[self.parity] + desc = [len(self.h2d), len(self.d2d)] + [ + value for copy in self.h2d + self.d2d for value in copy + ] + host[: 8 * len(desc)].view(torch.int64).copy_(torch.tensor(desc, dtype=torch.int64)) + # Descriptors and data in one upload; the gap between them is unused padding. + self.dev[: self.used].copy_(host[: self.used], non_blocking=True) + if self.parallel_scatter and _parallel_ok(self.h2d, self.d2d): + stage_i64 = self.dev.view(torch.int64) + if self.h2d: + _parallel_scatter_kernel[(len(self.h2d),)]( + self.dev, stage_i64, 0, DEVICE_SRC=False, BLOCK=256, num_warps=2 + ) + if self.d2d: + _parallel_scatter_kernel[(len(self.d2d),)]( + self.dev, stage_i64, len(self.h2d), DEVICE_SRC=True, BLOCK=256, num_warps=2 + ) + else: + torch.ops.trtllm.step_inputs_scatter(self.dev) + event = self.events[self.parity] + if event is None: + event = self.events[self.parity] = torch.cuda.Event() + event.record() + self.parity = (self.parity + 1) % self.RING + # The scatter is queued on the current stream: the caching allocator keeps the staged tensors' memory for it. + self._clear() + + def _fill_graph_window(self) -> bool: + """Write the pack into the pinned window the replay's prologue issues; False when it does not fit or is not + order-free (the caller issues it eagerly).""" + if self.used > self.GRAPH_WINDOW or not _parallel_ok(self.h2d, self.d2d): + return False + desc = [len(self.h2d), len(self.d2d)] + [ + value for copy in self.h2d + self.d2d for value in copy + ] + self._ghost_writable() + host = self.host[self.parity] + self.ghost[: self.used].copy_(host[: self.used]) + self.ghost[: 8 * len(desc)].view(torch.int64).copy_(torch.tensor(desc, dtype=torch.int64)) + return True + + def discard(self) -> None: + """Close the pack without issuing it (its step failed).""" + self.open = False + self._clear() + + +_active: ContextVar[Optional[StepInputPack]] = ContextVar("trtllm_step_input_pack", default=None) + + +@contextlib.contextmanager +def active(pack: Optional[StepInputPack]) -> Iterator[None]: + """Make ``pack`` the calling context's pack (None: no packing) for the duration.""" + token = _active.set(pack) + try: + yield + finally: + _active.reset(token) + + +def begin() -> None: + """Open the active pack for the coming copies. Call only when the step will reach ``flush``.""" + pack = _active.get() + if pack is not None: + pack.begin() + + +def is_active(pack: StepInputPack) -> bool: + """Whether ``pack`` is the calling context's pack.""" + return _active.get() is pack + + +def is_open() -> bool: + pack = _active.get() + return pack is not None and pack.open + + +def copy_(dst: torch.Tensor, src: torch.Tensor) -> torch.Tensor: + """``dst.copy_(src, non_blocking=True)``, deferred into the active open pack when there is one.""" + pack = _active.get() + if pack is not None and pack.open and not torch.cuda.is_current_stream_capturing(): + if pack.stage(dst, src): + return dst + # Keep the order: everything staged so far lands before this copy, and the copies after it pack again. + pack.flush() + out = dst.copy_(src, non_blocking=True) + pack.begin() + return out + return dst.copy_(src, non_blocking=True) + + +def discard() -> None: + """Close the active pack without issuing it (its step failed).""" + pack = _active.get() + if pack is not None: + pack.discard() + + +def flush(in_graph: bool = False) -> None: + """Issue the active open pack (one upload and one scatter launch on the current stream) and close it. + + ``in_graph``: the next replay is of a graph whose prologue issues the active pack (``capture_prologue``).""" + pack = _active.get() + if pack is not None: + pack.flush(in_graph=in_graph) + + +def capture_prologue() -> Optional[StepInputPack]: + """Inside a graph capture: add the active pack's in-graph prologue; the pack whose window the graph issues, or + None when the graph has no prologue (no active pack, or one without ``in_graph``: nothing is captured).""" + pack = _active.get() + return pack if pack is not None and pack.capture_prologue() else None diff --git a/tensorrt_llm/_torch/speculative/dflash.py b/tensorrt_llm/_torch/speculative/dflash.py index 0db5892872bd..6c9dc7ab938e 100644 --- a/tensorrt_llm/_torch/speculative/dflash.py +++ b/tensorrt_llm/_torch/speculative/dflash.py @@ -576,12 +576,16 @@ class DFlashSpecMetadata(SpecMetadata): dtype: torch.dtype = torch.bfloat16 captured_hidden_states: Optional[torch.Tensor] = None + # Highest arange prefix uploaded to this metadata instance. + _batch_indices_uploaded = 0 + def __post_init__(self): self.batch_indices_cuda = torch.empty( [self.max_num_requests], dtype=torch.int, device="cuda", ) + self._batch_indices_uploaded = 0 self.is_spec_dec_tree = False self.is_spec_dec_dynamic_tree = False @@ -622,10 +626,17 @@ def prepare(self): assert self.request_ids is not None num_seqs = len(self.request_ids) - batch_indices = torch.arange( - num_seqs, dtype=torch.int, device="cpu", pin_memory=prefer_pinned() - ) - self.batch_indices_cuda[:num_seqs].copy_(batch_indices, non_blocking=True) + # getattr: unit tests drive prepare() with stand-in metadata and workers. + changed_only = getattr(self, "batch_map_changed_only", False) + # The batch indices are an arange, so with batch_map_changed_only a buffer already holding a long enough + # prefix is not re-uploaded (nothing else writes batch_indices_cuda). + if not (changed_only and num_seqs <= self._batch_indices_uploaded): + batch_indices = torch.arange( + num_seqs, dtype=torch.int, device="cpu", pin_memory=prefer_pinned() + ) + self.batch_indices_cuda[:num_seqs].copy_(batch_indices, non_blocking=True) + if changed_only: + self._batch_indices_uploaded = num_seqs # Update slot mapping for DFlash context buffers worker = getattr(self, "_dflash_worker", None) @@ -660,13 +671,25 @@ def prepare(self): # Route unknown request IDs (cuda graph padding or warmup dummies) # to dummy slot to avoid corrupting real request's context - mapping = torch.tensor( - [worker._req_to_slot.get(rid, worker._dummy_slot) for rid in self.request_ids], - dtype=torch.long, - device="cpu", - pin_memory=prefer_pinned(), - ) - worker._batch_to_slot[:num_seqs].copy_(mapping, non_blocking=True) + slots = [worker._req_to_slot.get(rid, worker._dummy_slot) for rid in self.request_ids] + # With batch_map_changed_only the map is re-uploaded only when it differs from the last upload (see + # DFlashWorker._batch_to_slot_uploaded). + if not (changed_only and getattr(worker, "_batch_to_slot_uploaded", None) == slots): + mapping = torch.tensor( + slots, + dtype=torch.long, + device="cpu", + pin_memory=prefer_pinned(), + ) + worker._batch_to_slot[:num_seqs].copy_(mapping, non_blocking=True) + if changed_only: + worker._batch_to_slot_uploaded = slots + + # A target's per-iteration host hook (None in stock), e.g. an engagement counter. prepare() runs on the host + # every iteration, outside any captured graph, after the batch's sampling mode is set. + hook = getattr(worker, "_host_step_hook", None) + if hook is not None: + hook(self) def is_layer_capture(self, layer_id: int) -> bool: return layer_id in self._capture_layer_set @@ -735,6 +758,10 @@ class DFlashWorker(SpecWorkerBase): Reference: https://arxiv.org/pdf/2602.06036 """ + # The map DFlashSpecMetadata.prepare last uploaded into _batch_to_slot[:len(map)] under batch_map_changed_only, or + # None when unknown. Every other writer of _batch_to_slot (its allocation, the context-step rebuild) resets it. + _batch_to_slot_uploaded: Optional[List[int]] = None + def __init__( self, spec_config: "DFlashDecodingConfig", @@ -764,6 +791,14 @@ def __init__( self._ctx_len_restore_pending = False # Deferred kv_lens_cuda rewind state (see _prepare_kv_for_draft_forward, # _apply_kv_rewind_after_draft, _ensure_spec_dec_state_restored). + self._lean_glue = False + self._lean_aranges = {} + self._fused_ctx_store = None + self._fused_block_refresh = None + self._fused_drafter_inputs = None + self._fused_ctx_advance = None + self._fused_kv_rewind = None + self._host_step_hook = None self._kv_rewind_pending = False self._kv_rewind_amount = None self._kv_rewind_nc = None @@ -1070,6 +1105,15 @@ def _refresh_ctx_block_tables(self, attn_metadata, num_seqs: int) -> bool: # pool, and the drafter addresses blocks, not planes. # clone(): .to() is a no-op for an int64 source, and the in-place ops # below would then edit attn_metadata's own tensor. + if self._fused_block_refresh is not None and self._fused_block_refresh( + src, + self._ctx_pool_idx, + num_seqs, + self._ctx_block_divisor, + self._ctx_block_tables, + self._ctx_block_counts, + ): + return True encoded = src[self._ctx_pool_idx, :num_seqs, 0].to(torch.int64).clone() # Before the clamp erases the placeholders. Informative only on the # python V2 backend; the cpp one already mapped BAD_PAGE_INDEX to 0. @@ -1165,6 +1209,7 @@ def _lazy_init_ctx_buffers( self._ctx_base_pages = torch.zeros(num_slots, dtype=torch.long, device="cuda") self._ctx_base_host = [0] * num_slots self._batch_to_slot = torch.zeros(max_batch, dtype=torch.long, device="cuda") + self._batch_to_slot_uploaded = None self._free_slots = deque(range(max_batch)) self._req_to_slot = {} @@ -1399,6 +1444,124 @@ def _get_ctx_paged_append(self) -> Callable[..., None]: self._ctx_paged_append = get_dflash_paged_append() return self._ctx_paged_append + def _static_arange(self, start: int, end: int) -> torch.Tensor: + """torch.arange(start, end) (long, cuda). With lean glue on, cached per (start, end) outside CUDA-graph capture + (read-only uses only); never cached during capture, so no graph-pool tensor outlives its graph.""" + if not self._lean_glue: + return torch.arange(start, end, dtype=torch.long, device="cuda") + key = (start, end) + t = self._lean_aranges.get(key) + if t is None: + t = torch.arange(start, end, dtype=torch.long, device="cuda") + if not torch.cuda.is_current_stream_capturing(): + self._lean_aranges[key] = t + return t + + def enable_lean_glue(self, draft_model, enabled: bool = True) -> None: + """Skip the drafter's copies whose consumer reads strided views (worker and draft model); off by default. + The values are bitwise the same; only the copies go.""" + self._lean_glue = bool(enabled) + if hasattr(draft_model, "_dflash_lean_glue"): + draft_model._dflash_lean_glue = bool(enabled) + + def enable_fused_ctx_store(self, store) -> bool: + """Install a target's one-launch context K / V store (None, the default, keeps the stock path). + + ``store(draft_model, projected, positions, num_accepted, request_rows, cols, block_table, kv_buf, + tokens_per_request)`` must write exactly what precompute_context_kv + the accept mask + _store_context_kv_paged + write, and return False (writing nothing) for a drafter or cache it does not reproduce. It is first called with + ``cols`` None and the keywords ``capacity_counts`` (bound pool: pages per block-table row) / ``max_col`` (the + private arena's last column), to derive each row's column from its position as the stock clamp does.""" + self._fused_ctx_store = store + return store is not None + + def enable_fused_ctx_advance(self, advance) -> bool: + """Install a target's one-launch context advance + read view on a bound draft pool (None, the default, keeps + the stock path). + + ``advance(slots, rows, num_accepted, ctx_len, max_ctx, dummy_slot, block_counts, base_pages, tables, + tables_read, page_size, block_size)`` must leave ``ctx_len`` and ``tables_read[rows]`` exactly as + _advance_ctx_len and the read view below do and return the advertised context lengths, or None (writing + nothing) for a layout it does not reproduce.""" + self._fused_ctx_advance = advance + return advance is not None + + def enable_fused_forward_indices(self, draft_model, indices) -> bool: + """Install a target's one-launch index prelude for the draft model's paged block decode (None, the default, + keeps the stock lines). + + ``indices(num_ctx, batch_idx, query_positions, block_size, pages_per_slot, page_size)`` must return the int32 + (cache_seqlens, cache_batch_idx, seq_lens_after, kv_indptr, kv_last_page_len, append_positions, + query_positions_flat) exactly as the stock lines of dflash_forward compute them, or None for a layout it does + not reproduce.""" + if not hasattr(draft_model, "_dflash_fused_indices"): + return False + draft_model._dflash_fused_indices = indices + return indices is not None + + def enable_fused_qk_norm_rope(self, draft_model, fused) -> bool: + """Install a target's one-launch q / k RMSNorm + RoPE for the draft model's flashinfer-RoPE branch (None, the + default, keeps the stock lines). + + ``fused(qkv, q_size, kv_size, head_dim, q_norm, k_norm, positions, cos_sin_cache, is_neox)`` must return new + (q, k) exactly as the stock q_norm / k_norm + flashinfer RoPE lines leave them, or None for a layout it does + not reproduce.""" + if not hasattr(draft_model, "_dflash_fused_qk_norm_rope"): + return False + draft_model._dflash_fused_qk_norm_rope = fused + return fused is not None + + def enable_fused_qk_norm_rope_append(self, draft_model, fused) -> bool: + """Install a target's one-launch q / k RMSNorm + RoPE that also appends the block's K / V into the layer's + paged cache, for the draft model's flashinfer-RoPE branch on the TRTLLM backend (None, the default, keeps the + stock lines). + + ``fused(qkv, q_size, kv_size, head_dim, q_norm, k_norm, positions, cos_sin_cache, is_neox, paged_kv_cache, + append_batch_indices, append_positions, kv_indices, kv_indptr)`` must return new (q, k) exactly as + ``enable_fused_qk_norm_rope``'s op does AND leave the cache exactly as flashinfer's append_paged_kv_cache of + that K / V would; or return None (nothing computed or written), and the stock lines run.""" + if not hasattr(draft_model, "_dflash_fused_qk_norm_rope_append"): + return False + draft_model._dflash_fused_qk_norm_rope_append = fused + return fused is not None + + def enable_attention_pdl(self, draft_model, enabled: bool) -> bool: + """Launch the draft model's TRTLLM-backend block attention (trtllm-gen decode / context) with programmatic + dependent launch: the same kernel and inputs, launched to overlap the previous kernel's tail (it waits for that + grid before reading). Off by default.""" + if not hasattr(draft_model, "_dflash_attn_pdl"): + return False + draft_model._dflash_attn_pdl = bool(enabled) + return draft_model._dflash_attn_pdl + + def enable_fused_drafter_inputs(self, inputs) -> bool: + """Install a target's one-launch drafter step inputs (None, the default, keeps the stock path). + + ``inputs(embed_tokens, accepted_tokens, num_accepted, slots, ctx_len, max_ctx, mask_token_id, + trained_mask_embedding, block_size, tokens_per_request)`` must return exactly the stock (bonus, + query_position_ids, ctx_position_ids, noise block) of prepare_1st_drafter_inputs' generation branch, or None + (computing nothing) for an embedding or layout it does not reproduce.""" + self._fused_drafter_inputs = inputs + return inputs is not None + + def enable_fused_kv_rewind(self, rewind) -> bool: + """Install a target's one-launch draft-forward KV bookkeeping (None, the default, keeps the stock lines). + + ``rewind(num_accepted, kv_lens, nc, bs)`` must leave ``kv_lens[nc:bs]`` incremented by one and return the + tensor ``1 - num_accepted[nc:bs]`` (the same values and dtype), exactly as _prepare_kv_for_draft_forward's + two lines, or None (writing nothing) for operands it does not take.""" + self._fused_kv_rewind = rewind + return rewind is not None + + def enable_fused_block_refresh(self, refresh) -> bool: + """Install a target's one-launch block-table refresh (None, the default, keeps the stock path). + + ``refresh(offsets, pool_idx, num_seqs, divisor, tables, counts)`` must write exactly what + _refresh_ctx_block_tables writes into ``tables[:num_seqs]`` and ``counts[:num_seqs]``, and return False + (writing nothing) for a layout it does not reproduce.""" + self._fused_block_refresh = refresh + return refresh is not None + def _store_context_kv_paged( self, k: torch.Tensor, @@ -1430,9 +1593,11 @@ def _store_context_kv_paged( positions_i32 = positions.to(torch.int32) kv_indices, kv_indptr = self._ctx_paged_index_args() for layer_idx in range(k.size(1)): + # Lean glue: flashinfer's append is stride-aware (last dim contiguous), so the per-layer slices go + # in as views; the values written are bitwise the same. append_paged_kv_cache( - append_key=k[:, layer_idx].contiguous(), - append_value=v[:, layer_idx].contiguous(), + append_key=k[:, layer_idx] if self._lean_glue else k[:, layer_idx].contiguous(), + append_value=v[:, layer_idx] if self._lean_glue else v[:, layer_idx].contiguous(), batch_indices=rows_i32, positions=positions_i32, paged_kv_cache=self._ctx_kv_buf[layer_idx], @@ -1460,11 +1625,22 @@ def _prepare_kv_for_draft_forward( ): """Adjust kv_lens_cuda so the draft model sees correct RoPE positions.""" if hasattr(attn_metadata, "kv_lens_cuda"): - self._kv_rewind_amount = 1 - num_accepted_tokens[num_contexts:batch_size] + # A target's one-launch bookkeeping: the same rewind amount and kv_lens increment as the lines below. + fused = ( + self._fused_kv_rewind( + num_accepted_tokens, attn_metadata.kv_lens_cuda, num_contexts, batch_size + ) + if self._fused_kv_rewind is not None and batch_size > num_contexts + else None + ) + if fused is not None: + self._kv_rewind_amount = fused + else: + self._kv_rewind_amount = 1 - num_accepted_tokens[num_contexts:batch_size] self._kv_rewind_nc = num_contexts self._kv_rewind_bs = batch_size - if batch_size > num_contexts: + if fused is None and batch_size > num_contexts: attn_metadata.kv_lens_cuda[num_contexts:batch_size] += 1 self._kv_rewind_pending = True @@ -1828,6 +2004,7 @@ def _forward_impl( ), non_blocking=True, ) + self._batch_to_slot_uploaded = None inputs = self.prepare_1st_drafter_inputs( input_ids=input_ids, @@ -1843,6 +2020,12 @@ def _forward_impl( if num_gens > 0: with self.draft_kv_cache_context(attn_metadata, draft_kv_cache_manager): + # Lean glue: in bound mode the cache rows are the contiguous gen rows, so the block table is a slice. + draft_model._dflash_cache_rows = ( + (num_contexts, num_contexts + num_gens) + if self._lean_glue and self._ctx_block_tables is not None + else None + ) hidden_states_out = draft_model.dflash_forward( noise_embedding=inputs["noise_embedding"], query_positions=inputs["query_positions"], @@ -1859,13 +2042,22 @@ def _forward_impl( # Which block slots carry them is a drafter-family convention, # resolved through _draft_slot_ids. block_size = self._compute_block_size - gen_gather_ids = self._draft_slot_ids(draft_model, num_gens, block_size, K) - # Shields only the last request: at block_size == K with - # shift_label off, slots run 1..K, so every request reads the - # next one's slot 0 and the last overruns. Degrades, never raises. - gen_gather_ids = gen_gather_ids.clamp(max=hidden_states_out.shape[0] - 1) - - gen_hidden_states = hidden_states_out[gen_gather_ids] + if ( + self._lean_glue + and getattr(draft_model, "_dspark_shift_label", False) + and block_size == K + and hidden_states_out.shape[0] == num_gens * block_size + ): + # Lean glue: under shift_label with block_size == K the slot ids are arange(num_gens * K), so the + # gather is the identity. + gen_hidden_states = hidden_states_out + else: + gen_gather_ids = self._draft_slot_ids(draft_model, num_gens, block_size, K) + # Shields only the last request: at block_size == K with + # shift_label off, slots run 1..K, so every request reads the + # next one's slot 0 and the last overruns. Degrades, never raises. + gen_gather_ids = gen_gather_ids.clamp(max=hidden_states_out.shape[0] - 1) + gen_hidden_states = hidden_states_out[gen_gather_ids] gen_logits = draft_model.logits_processor( gen_hidden_states, draft_model.lm_head, attn_metadata, True ) @@ -2196,39 +2388,61 @@ def prepare_1st_drafter_inputs( # Get slots for gen requests from pre-computed mapping slots = self._batch_to_slot[num_contexts : num_contexts + num_gens] - gen_rows_out = torch.arange( - num_contexts, num_contexts + num_gens, dtype=torch.long, device="cuda" - ) + gen_rows_out = self._static_arange(num_contexts, num_contexts + num_gens) K_plus_1 = K + 1 - bonus_idx = (gen_num_accepted - 1).clamp_min(0).long().unsqueeze(1) - bonus = gen_accepted_tokens.gather(1, bonus_idx).squeeze(1).long() - - ctx_len_gen = self._ctx_len[slots] - j_block = torch.arange(query_tokens_per_req, dtype=torch.long, device="cuda") - offsets_kp1 = torch.arange(K_plus_1, dtype=torch.long, device="cuda") - - # _ctx_len is clamped to _max_ctx only AFTER this step's accepted - # tokens are folded in (see the update below), so the running length - # used here has to be clamped on its own -- otherwise a request that - # already sits at the ceiling indexes num_accepted positions past - # any position the sequence can legitimately reach. - ctx_len_now = (ctx_len_gen + gen_num_accepted.long()).clamp_(max=self._max_ctx) - query_position_ids = ctx_len_now.unsqueeze(1) + j_block.unsqueeze(0) - ctx_position_ids = ctx_len_gen.unsqueeze(1) + offsets_kp1.unsqueeze(0) - - # The drafter's own trained mask row, if it kept one for the mask id in use. - trained_mask_embedding = ( - getattr(draft_model, "mask_token_embedding", None) - if mask_token_id == getattr(draft_model, "mask_token_id", None) + # A target's one-launch step inputs: the same bonus, positions and noise block as the stock lines below. + fused_inputs = ( + self._fused_drafter_inputs( + embed_tokens, + gen_accepted_tokens, + gen_num_accepted, + slots, + self._ctx_len, + self._max_ctx, + mask_token_id, + getattr(draft_model, "mask_token_embedding", None) + if mask_token_id == getattr(draft_model, "mask_token_id", None) + else None, + query_tokens_per_req, + K_plus_1, + ) + if self._fused_drafter_inputs is not None else None ) - noise_embed_2d = dflash_noise_block_embedding( - embed_tokens, bonus, mask_token_id, query_tokens_per_req, trained_mask_embedding - ) + if fused_inputs is not None: + bonus, query_position_ids, ctx_position_ids, noise_embed_2d = fused_inputs + offsets_kp1 = self._static_arange(0, K_plus_1) + else: + bonus_idx = (gen_num_accepted - 1).clamp_min(0).long().unsqueeze(1) + bonus = gen_accepted_tokens.gather(1, bonus_idx).squeeze(1).long() + + ctx_len_gen = self._ctx_len[slots] + j_block = self._static_arange(0, query_tokens_per_req) + offsets_kp1 = self._static_arange(0, K_plus_1) + + # _ctx_len is clamped to _max_ctx only AFTER this step's accepted + # tokens are folded in (see the update below), so the running length + # used here has to be clamped on its own -- otherwise a request that + # already sits at the ceiling indexes num_accepted positions past + # any position the sequence can legitimately reach. + ctx_len_now = (ctx_len_gen + gen_num_accepted.long()).clamp_(max=self._max_ctx) + query_position_ids = ctx_len_now.unsqueeze(1) + j_block.unsqueeze(0) + ctx_position_ids = ctx_len_gen.unsqueeze(1) + offsets_kp1.unsqueeze(0) + + # The drafter's own trained mask row, if it kept one for the mask id in use. + trained_mask_embedding = ( + getattr(draft_model, "mask_token_embedding", None) + if mask_token_id == getattr(draft_model, "mask_token_id", None) + else None + ) + noise_embed_2d = dflash_noise_block_embedding( + embed_tokens, bonus, mask_token_id, query_tokens_per_req, trained_mask_embedding + ) # Accumulate new accepted features into context buffers + fused_num_ctx = None if has_target_features: gen_start = attn_metadata.num_ctx_tokens # Target now processes exactly K+1 tokens per gen req, so the @@ -2242,81 +2456,142 @@ def prepare_1st_drafter_inputs( projected_rows = projected_to_store.reshape(num_gens, K_plus_1, -1) lilicorr_anchor = last_accepted_hidden(projected_rows, gen_num_accepted) gen_num_accepted_long = gen_num_accepted.long() - col_idx = self._ctx_len[slots].unsqueeze(1) + offsets_kp1.unsqueeze(0) - write_mask = offsets_kp1.unsqueeze(0) < gen_num_accepted_long.unsqueeze(1) - if self._ctx_block_tables is not None: - # Bound by what the manager handed THIS request: the masked - # entries below are still written (fixed-size, for graph - # safety) and a placeholder page resolves to another's. - gen_capacity = self._ctx_block_counts[gen_rows_out] * self._ctx_page_size - col_idx = torch.minimum(col_idx, (gen_capacity - 1).unsqueeze(1)).clamp_(min=0) - else: - col_idx = col_idx.clamp(max=self._max_ctx - 1) - - # Fixed-size writes for CUDA graph compatibility: - # Write ALL entries but zero out invalid ones. Invalid - # writes land at clamped column indices beyond valid - # _ctx_len range, so they're harmless. - slot_flat = slots.unsqueeze(1).expand(-1, K + 1).reshape(-1) - col_flat = col_idx.reshape(-1) - proj_flat = projected_to_store # already [num_gens*(K+1), proj_dim] - pos_flat = ctx_position_ids.long().reshape(-1) - mask_1d = write_mask.reshape(-1) - - # Fast path: store the pre-projected/pre-RoPE'd K/V. - # dflash_forward reads these directly via cache_batch_idx. - cache_dtype = ( - self._ctx_kv_buf[0].dtype if self._ctx_paged else self._ctx_k_buf.dtype - ) - k_new, v_new = draft_model.precompute_context_kv( - proj_flat.to(cache_dtype), pos_flat - ) - mask_bc = mask_1d.view(-1, 1, 1, 1).to(k_new.dtype) - k_new.mul_(mask_bc) - if v_new is not None: - v_new.mul_(mask_bc) - slot_long = slot_flat.long() - col_long = col_flat.long() - if self._ctx_paged: + # S10c-A2: a target's fused store derives each row's column from its position itself (the stock clamp + # below), so the column glue runs only on the stock path. + fused_stored = False + if self._fused_ctx_store is not None and self._ctx_paged: + bound = self._ctx_block_tables is not None + fused_stored = self._fused_ctx_store( + draft_model, + projected_to_store.to(self._ctx_kv_buf[0].dtype), + ctx_position_ids.long().reshape(-1), + gen_num_accepted_long, + gen_rows_out if bound else slots, + None, + self._ctx_block_tables if bound else self._ctx_page_table, + self._ctx_kv_buf, + K + 1, + capacity_counts=self._ctx_block_counts if bound else None, + max_col=None if bound else self._max_ctx - 1, + ) + if not fused_stored: + col_idx = self._ctx_len[slots].unsqueeze(1) + offsets_kp1.unsqueeze(0) + write_mask = offsets_kp1.unsqueeze(0) < gen_num_accepted_long.unsqueeze(1) if self._ctx_block_tables is not None: - # Batch positions of the gen requests, matching the - # per-request block table's row order. - rows_long = gen_rows_out.unsqueeze(1).expand(-1, K + 1).reshape(-1) + # Bound by what the manager handed THIS request: the masked + # entries below are still written (fixed-size, for graph + # safety) and a placeholder page resolves to another's. + gen_capacity = self._ctx_block_counts[gen_rows_out] * self._ctx_page_size + col_idx = torch.minimum(col_idx, (gen_capacity - 1).unsqueeze(1)).clamp_( + min=0 + ) else: - rows_long = slot_long - self._store_context_kv_paged(k_new, v_new, rows_long, col_long) - else: # VANILLA DFlash backend (FlashAttention) - self._ctx_k_buf[slot_long, :, col_long] = k_new - if v_new is not None: - self._ctx_v_buf[slot_long, :, col_long] = v_new - - self._advance_ctx_len(slots, gen_num_accepted_long) - - num_ctx_per_req_t = self._ctx_len[slots] - ctx_page_table = self._ctx_page_table - if self._ctx_block_tables is not None: - # The write above clamps columns to the request's allocation, so - # a context that outruns it has its tail written over the last - # valid slot. Truncate what is advertised to match, or the read - # attends that clobbered value -- or another request's page. - allocated = dflash_allocated_ctx_limit( - self._ctx_block_counts, self._ctx_page_size, block_size - ) - num_ctx_per_req_t = torch.minimum(num_ctx_per_req_t, allocated[gen_rows_out]) - # Pages below a slot's base hold no drafter K/V (a prefix the - # target reused): the read numbers from the base instead, see - # _ctx_base_for_store and dflash_rebased_ctx_view. Writes above - # keep the absolute table. - base_pages = self._ctx_base_pages[slots] - self._ctx_block_tables_read[gen_rows_out], num_ctx_per_req_t = ( - dflash_rebased_ctx_view( - self._ctx_block_tables[gen_rows_out], - base_pages, - num_ctx_per_req_t, + col_idx = col_idx.clamp(max=self._max_ctx - 1) + + # Fixed-size writes for CUDA graph compatibility: + # Write ALL entries but zero out invalid ones. Invalid + # writes land at clamped column indices beyond valid + # _ctx_len range, so they're harmless. + slot_flat = slots.unsqueeze(1).expand(-1, K + 1).reshape(-1) + col_flat = col_idx.reshape(-1) + proj_flat = projected_to_store # already [num_gens*(K+1), proj_dim] + pos_flat = ctx_position_ids.long().reshape(-1) + mask_1d = write_mask.reshape(-1) + + # Fast path: store the pre-projected/pre-RoPE'd K/V. + # dflash_forward reads these directly via cache_batch_idx. + cache_dtype = ( + self._ctx_kv_buf[0].dtype if self._ctx_paged else self._ctx_k_buf.dtype + ) + # A target's fused store does the K / V epilogue, the accept mask and the paged write in one launch. + fused_stored = ( + self._fused_ctx_store is not None + and self._ctx_paged + and self._fused_ctx_store( + draft_model, + proj_flat.to(cache_dtype), + pos_flat, + gen_num_accepted_long, + gen_rows_out if self._ctx_block_tables is not None else slots, + col_flat, + self._ctx_block_tables + if self._ctx_block_tables is not None + else self._ctx_page_table, + self._ctx_kv_buf, + K + 1, + ) + ) + if not fused_stored: + k_new, v_new = draft_model.precompute_context_kv( + proj_flat.to(cache_dtype), pos_flat + ) + mask_bc = mask_1d.view(-1, 1, 1, 1).to(k_new.dtype) + k_new.mul_(mask_bc) + if v_new is not None: + v_new.mul_(mask_bc) + slot_long = slot_flat.long() + col_long = col_flat.long() + if self._ctx_paged: + if self._ctx_block_tables is not None: + # Batch positions of the gen requests, matching the + # per-request block table's row order. + rows_long = gen_rows_out.unsqueeze(1).expand(-1, K + 1).reshape(-1) + else: + rows_long = slot_long + self._store_context_kv_paged(k_new, v_new, rows_long, col_long) + else: # VANILLA DFlash backend (FlashAttention) + self._ctx_k_buf[slot_long, :, col_long] = k_new + if v_new is not None: + self._ctx_v_buf[slot_long, :, col_long] = v_new + + # A target's one-launch advance + read view: the stock lines' context lengths and read rows, or None. + if self._fused_ctx_advance is not None and self._ctx_block_tables is not None: + fused_num_ctx = self._fused_ctx_advance( + slots, + gen_rows_out, + gen_num_accepted_long, + self._ctx_len, + self._max_ctx, + self._dummy_slot, + self._ctx_block_counts, + self._ctx_base_pages, + self._ctx_block_tables, + self._ctx_block_tables_read, self._ctx_page_size, + block_size, ) - ) + if fused_num_ctx is None: + self._advance_ctx_len(slots, gen_num_accepted_long) + + if fused_num_ctx is not None: + num_ctx_per_req_t = fused_num_ctx ctx_page_table = self._ctx_block_tables_read + else: + num_ctx_per_req_t = self._ctx_len[slots] + ctx_page_table = self._ctx_page_table + if self._ctx_block_tables is not None: + # The write above clamps columns to the request's allocation, so + # a context that outruns it has its tail written over the last + # valid slot. Truncate what is advertised to match, or the read + # attends that clobbered value -- or another request's page. + allocated = dflash_allocated_ctx_limit( + self._ctx_block_counts, self._ctx_page_size, block_size + ) + num_ctx_per_req_t = torch.minimum(num_ctx_per_req_t, allocated[gen_rows_out]) + # Pages below a slot's base hold no drafter K/V (a prefix the + # target reused): the read numbers from the base instead, see + # _ctx_base_for_store and dflash_rebased_ctx_view. Writes above + # keep the absolute table. + base_pages = self._ctx_base_pages[slots] + self._ctx_block_tables_read[gen_rows_out], num_ctx_per_req_t = ( + dflash_rebased_ctx_view( + self._ctx_block_tables[gen_rows_out], + base_pages, + num_ctx_per_req_t, + self._ctx_page_size, + ) + ) + ctx_page_table = self._ctx_block_tables_read noise_embedding = noise_embed_2d query_positions = query_position_ids.long() diff --git a/tensorrt_llm/_torch/speculative/eagle3.py b/tensorrt_llm/_torch/speculative/eagle3.py index fc29a0633f41..c766c5027624 100644 --- a/tensorrt_llm/_torch/speculative/eagle3.py +++ b/tensorrt_llm/_torch/speculative/eagle3.py @@ -17,6 +17,7 @@ from ..attention.backends.flashinfer import FlashInferAttentionMetadata from ..attention.backends.sparse.params import MTPIndexShareMetadata from ..model_config import ModelConfig +from ..pyexecutor.h2d_cache import copy_h2d_if_changed from ..pyexecutor.kv_cache.mamba_cache_manager import MambaHybridCacheManager from ..pyexecutor.llm_request import LlmRequest from ..pyexecutor.resource_manager import BaseResourceManager, SlotManager @@ -352,8 +353,7 @@ def prepare(self): dtype=torch.int, device='cpu', pin_memory=prefer_pinned()) - self.batch_indices_cuda[:num_seqs].copy_(batch_indices, - non_blocking=True) + copy_h2d_if_changed(self.batch_indices_cuda[:num_seqs], batch_indices) # Snapshot the real (unpadded) scheduled token count before the decrement # below. maybe_capture_hidden_states runs during the target forward and # must bound the capture by this count; self.num_tokens is about to be @@ -374,8 +374,7 @@ def prepare(self): eagle_slot_ids_tensor = torch.tensor(eagle_slot_ids, dtype=torch.int, pin_memory=prefer_pinned()) - self.slot_ids[:num_seqs].copy_(eagle_slot_ids_tensor, - non_blocking=True) + copy_h2d_if_changed(self.slot_ids[:num_seqs], eagle_slot_ids_tensor) sa_manager = getattr(self.spec_resource_manager, 'sa_manager', None) if sa_manager is not None: diff --git a/tensorrt_llm/_torch/speculative/interface.py b/tensorrt_llm/_torch/speculative/interface.py index 724b2aa15e36..d0bfd76c28cc 100644 --- a/tensorrt_llm/_torch/speculative/interface.py +++ b/tensorrt_llm/_torch/speculative/interface.py @@ -19,7 +19,7 @@ from contextlib import contextmanager from dataclasses import dataclass, field from enum import IntEnum -from typing import TYPE_CHECKING, List, Optional, Protocol, Type +from typing import TYPE_CHECKING, Callable, List, Optional, Protocol, Type import torch from packaging.version import Version @@ -32,6 +32,7 @@ from ..attention.backends.trtllm import (AttentionBackend, TrtllmAttention, TrtllmAttentionMetadata) from ..flashinfer_utils import IS_FLASHINFER_AVAILABLE +from ..pyexecutor import step_inputs from ..pyexecutor.resource_manager import ResourceManagerType if TYPE_CHECKING: @@ -48,14 +49,31 @@ from ..pyexecutor.sampler.ops import flashinfer as flashinfer_sampling from ..pyexecutor.sampler.ops.vanilla import greedy_search_sampling_batch +# An optional replacement of the tensor-seeded deterministic chain sampler, +# installed by a model at load time: it takes the op's arguments and returns +# its (accepted_tokens, num_tokens), or None to keep the stock op for that +# input. None (the default) leaves rejection_sampling_one_model unchanged. +_ChainSpecOverride = Callable[ + [torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor], + Optional[tuple[torch.Tensor, torch.Tensor]]] +_CHAIN_SPEC_OVERRIDE: Optional[_ChainSpecOverride] = None + + +def set_chain_speculative_sampling_override( + fn: Optional[_ChainSpecOverride]) -> None: + """Install (or, with None, remove) the chain speculative sampling override + used by ``rejection_sampling_one_model``.""" + global _CHAIN_SPEC_OVERRIDE + _CHAIN_SPEC_OVERRIDE = fn + def rejection_sampling_one_model( draft_probs: torch.Tensor, draft_token_ids: torch.Tensor, target_probs: torch.Tensor, deterministic: bool = True, - seed: Optional[int] = None, - offset: Optional[int] = None, + seed: Optional[int | torch.Tensor] = None, + offset: Optional[int | torch.Tensor] = None, ) -> tuple[torch.Tensor, torch.Tensor]: # chain_speculative_sampling requires flashinfer>=0.6.4. This entry point can # be reached independently of SpecWorkerBase.__init__ (e.g. via @@ -66,6 +84,15 @@ def rejection_sampling_one_model( raise RuntimeError( "Rejection sampling for one-model speculative decoding requires flashinfer>=0.6.4" ) + # The inherited override accepts per-request tensor Philox state and has no + # nondeterministic mode. Keep feature-base scalar/default RNG calls stock. + if (_CHAIN_SPEC_OVERRIDE is not None and deterministic + and isinstance(seed, torch.Tensor) + and isinstance(offset, torch.Tensor)): + overridden = _CHAIN_SPEC_OVERRIDE(draft_probs, draft_token_ids, + target_probs, seed, offset) + if overridden is not None: + return overridden batch_size = draft_token_ids.shape[0] device = draft_token_ids.device output_accepted_token_num = torch.zeros(batch_size, @@ -466,6 +493,79 @@ def from_string(name: Optional[str]) -> "SpeculativeDecodingMode": # slot leaves them room before the next one starts. _RNG_SLOT_SPAN = 32 +# RuntimeHooks.spec_tail_bookkeep (``SpecMetadata.tail_bookkeep``): the +# per-token buffers a single decode request's rows are read from, with the +# dtype ``populate_sampling_params_for_one_model`` / +# ``_populate_request_rng_state`` allocate them with. +_TAIL_BOOKKEEP_ROW_BUFFERS = ( + ("temperatures", torch.float32), + ("top_ks", torch.int32), + ("top_ps", torch.float32), + ("min_ps", torch.float32), + ("seeds", torch.int64), + ("offsets", torch.int64), +) +# The tail bookkeeping items that already logged their first use. +_TAIL_BOOKKEEP_LOGGED: set = set() + + +def tail_bookkeep_log_once(item: str, message: str) -> None: + """Log a tail bookkeeping item's engagement line once per process. + + Host only, so a captured step logs it while it is captured. + """ + if item not in _TAIL_BOOKKEEP_LOGGED: + _TAIL_BOOKKEEP_LOGGED.add(item) + logger.info(f"[M3 host] tail bookkeeping {item}: {message}") + + +def tail_bookkeep_draft_len(spec_metadata, num_contexts: int, + batch_size: Optional[int]) -> int: + """The draft length when one request owns the per-token rows, else 0. + + ``K`` when this step's one request owns rows ``[0, K]`` of the per-token + sampling and Philox buffers. + + ``SpecMetadata._scan_one_model_sampling`` lays the per-token buffers out as + ``[ctx: 1 row, gen: 1 + runtime_draft_len rows]`` per request, and + ``populate_sampling_params_for_one_model`` / + ``_populate_request_rng_state`` fill a generation request's rows with its + per-request values: the same normalized temperature / top-k / top-p / min-p + as ``request_*``, its seed, and offsets ``request_offsets + r * + _RNG_SLOT_SPAN`` for row ``r``. Both layouts are kept current on every + sampled step (the Philox state is uploaded together; the parameters' + signature check only skips a refill that would write the same values). So + with tail bookkeeping on, a decode-only step of one request (no tree, no + ragged verification, the static draft length) can read its draft block's + parameters and Philox state from those rows instead of expanding the + per-request buffers. + + Host state and shapes only (no device read), so it is fixed per captured + graph. + """ + if not getattr(spec_metadata, "tail_bookkeep", False): + return 0 + if num_contexts != 0 or batch_size != 1: + return 0 + if (spec_metadata.is_spec_dec_tree or spec_metadata.is_spec_dec_dynamic_tree + or spec_metadata.is_ragged_verify): + return 0 + draft_len = spec_metadata.max_draft_len + if draft_len < 1 or spec_metadata.runtime_draft_len != draft_len: + return 0 + for name, dtype in _TAIL_BOOKKEEP_ROW_BUFFERS: + buf = getattr(spec_metadata, name, None) + if (not isinstance(buf, torch.Tensor) or buf.dim() != 1 + or buf.dtype != dtype or buf.numel() < draft_len + 1 + or not buf.is_contiguous()): + return 0 + for name in ("request_seeds", "request_offsets"): + buf = getattr(spec_metadata, name, None) + if (not isinstance(buf, torch.Tensor) or buf.dim() != 1 + or buf.dtype != torch.int64 or buf.numel() < 1): + return 0 + return draft_len + @dataclass class SpecMetadata: @@ -584,6 +684,19 @@ def is_ragged_verify(self) -> bool: group_all_greedy_sample: Optional[bool] = None # Whether to use rejection sampling for one-model speculative decoding. use_rejection_sampling: bool = False + # Skip the Philox seed and offset uploads on all-greedy batches (deploy-time; + # from the model's RuntimeHooks). The argmax path never reads them, and the + # next sampled batch rewrites every row it reads. + skip_greedy_rng_upload: bool = False + # Stage the per-step Philox seed/offset uploads into the engine's step-input + # pack (RuntimeHooks.pack_spec_rng_uploads) instead of copying each one. + pack_rng_uploads: bool = False + # Re-upload a drafter's batch_indices / batch-to-slot map only when they + # change (RuntimeHooks.spec_batch_map_changed_only; read by DFlash). + batch_map_changed_only: bool = False + # Drop the sampler tail's no-op bookkeeping launches, bitwise the same + # outputs (RuntimeHooks.spec_tail_bookkeep; see tail_bookkeep_draft_len). + tail_bookkeep: bool = False # Advanced-sampling specialization (deploy-time; from DecodingBaseConfig.advanced_sampling_mode). advanced_sampling_mode: AdvancedSamplingMode = AdvancedSamplingMode.FULL # Whether the occurrence penalties are enabled (deploy-time; from @@ -730,6 +843,13 @@ def batch_uses_penalty(self, value: bool) -> None: # Used to scatter draft probs by slot at write time and gather them by slot # at the next iter's verify. Shape: [max_num_requests], dtype=long. batch_slot_ids: Optional[torch.Tensor] = None + # Re-upload batch_slot_ids only when the batch's slot ids change (RuntimeHooks.spec_slot_ids_changed_only). + slot_ids_changed_only: bool = False + # [buffer, ids]: the slot ids last copied and the buffer they went into. Nothing else writes batch_slot_ids, so + # equal ids into the same buffer mean equal device values. A list shared with the shallow copies of + # create_cuda_graph_metadata, which share batch_slot_ids too (same reasoning as _sampling_params_signature). + _batch_slot_ids_last: list = field(default_factory=lambda: [None, None], + repr=False) # Draft probs expanded to the target vocab size. Zero-filled once at # prepare(); each rejection iter overwrites only the d2t-selected positions # (or [:draft_vocab] when there is no d2t). @@ -820,6 +940,23 @@ def _take_rng_window_offsets(self, requests: list["LlmRequest"], offsets.append(base + step * window) return offsets + def _upload_batch_slot_ids(self, per_request_slot_ids) -> None: + """Copy the batch's slot ids into batch_slot_ids; with slot_ids_changed_only, skip the copy when the same ids + were last copied into this same buffer (nothing else writes it, so it already holds them). With the hook off + this is the plain copy, and no record is kept.""" + if self.slot_ids_changed_only: + key = tuple(per_request_slot_ids) + last = self._batch_slot_ids_last + if last[0] is self.batch_slot_ids and last[1] == key: + return + last[0], last[1] = self.batch_slot_ids, key + self.batch_slot_ids[:len(per_request_slot_ids)].copy_( + torch.tensor(per_request_slot_ids, + dtype=torch.long, + pin_memory=prefer_pinned()), + non_blocking=True, + ) + def _populate_request_rng_state( self, requests: list["LlmRequest"], per_request_normalized: list[tuple[float, int, float, float, @@ -894,11 +1031,19 @@ def _populate_request_rng_state( device='cuda') def _upload(dst: torch.Tensor, values: list[int]) -> None: - dst[:len(values)].copy_(torch.tensor(values, - dtype=torch.int64, - pin_memory=prefer_pinned()), - non_blocking=True) + src = torch.tensor(values, + dtype=torch.int64, + pin_memory=prefer_pinned()) + if self.pack_rng_uploads: + # The forward is the first reader of these buffers, so the copy + # may move to the pack's flush (a plain copy when no pack is open). + step_inputs.copy_(dst[:len(values)], src) + else: + dst[:len(values)].copy_(src, non_blocking=True) + if self.skip_greedy_rng_upload and self.is_all_greedy_sample: + # The offset window counters above still advanced. + return _upload(self.request_seeds, request_seeds) _upload(self.request_offsets, request_offsets) _upload(self.seeds, flat_seeds) @@ -1435,12 +1580,7 @@ def populate_sampling_params_for_one_model( # same table to map logits rows back to their slot, so they enable it too. if (self.use_rejection_sampling or self.enable_penalty) and self.batch_slot_ids is not None: - self.batch_slot_ids[:len(per_request_slot_ids)].copy_( - torch.tensor(per_request_slot_ids, - dtype=torch.long, - pin_memory=prefer_pinned()), - non_blocking=True, - ) + self._upload_batch_slot_ids(per_request_slot_ids) # Penalties are independent of the greedy/advanced split -- they rewrite the # logits before sampling, so an all-greedy batch is penalized too (its argmax @@ -1644,6 +1784,10 @@ class SpecWorkerBase(nn.Module, ABC): Provides common functionality for sampling and token handling. """ + # Per-worker target-installed collective; None retains the stock gather. + _draft_logits_gather_op: Optional[Callable[[torch.Tensor], + Optional[torch.Tensor]]] = None + def __init__(self, use_separate_draft_kv_cache: bool = False): super().__init__() self.guided_decoder: Optional["CapturableGuidedDecoder"] = None @@ -2262,9 +2406,25 @@ def maybe_gather_sharded_draft_logits(self, # Only plain TP (no attention DP) reaches here -- ADP variants return # early via _draft_logits_are_sharded, since their ranks are # data-parallel and must not gather draft logits across ranks. + op = self._draft_logits_gather_op + if op is not None: + gathered = op(logits) + if gathered is not None: + return gathered from ..distributed.ops import allgather return allgather(logits, self.mapping, dim=-1) + def enable_draft_logits_gather_op(self, op) -> bool: + """Install a target's all-gather of the vocab-sharded draft logits + (None, the default, keeps ``allgather(logits, mapping, dim=-1)``). + + ``op(logits)`` must return ``torch.cat(all_gather(logits), -1)`` bit + for bit, identical on every TP rank, or None for operands it does not + take (decided by shapes / dtypes only, so every rank agrees), and then + the stock all-gather runs. Collective: every rank installs the same.""" + self._draft_logits_gather_op = op + return op is not None + @staticmethod def _sample_from_logits( advanced_sampling_mode: AdvancedSamplingMode, @@ -2588,12 +2748,18 @@ def _sample_and_accept_draft_tokens_rejection( if logits.dim() == 1: logits = logits.unsqueeze(0) - accepted_tokens = torch.empty((batch_size, runtime_draft_len + 1), - dtype=torch.int, - device=device) - num_accepted_tokens = torch.ones(batch_size, - dtype=torch.int, - device=device) + # Tail bookkeeping A1: on a decode-only step the acceptance kernel's + # outputs are the whole result, so they are returned as they are (see + # below) and the result buffers are allocated only if they cannot be. + defer_outputs = (getattr(spec_metadata, "tail_bookkeep", False) + and num_contexts == 0 and num_gens > 0) + if not defer_outputs: + accepted_tokens = torch.empty((batch_size, runtime_draft_len + 1), + dtype=torch.int, + device=device) + num_accepted_tokens = torch.ones(batch_size, + dtype=torch.int, + device=device) # === Context subset: sample target's first token directly === if num_contexts > 0: @@ -2729,8 +2895,30 @@ def _sample_and_accept_draft_tokens_rejection( gen_accepted[:, 1:runtime_draft_len + 1].copy_(full_draft_tokens) - accepted_tokens[num_contexts:] = gen_accepted - num_accepted_tokens[num_contexts:] = gen_num_accepted + if defer_outputs and self._tail_bookkeep_takes_outputs( + gen_accepted, gen_num_accepted, batch_size, + runtime_draft_len, device): + # Exact: with no context rows the two copies below overwrite + # every element of same-shape, same-dtype buffers (the ones + # fill included), and the kernel's outputs are fresh tensors + # nothing else holds, so the in-place updates after this point + # (forced acceptance, the ragged clamp) see the same values. + tail_bookkeep_log_once( + "A1", "rejection outputs returned as the result (no " + "ones fill + 2 copies)") + accepted_tokens = gen_accepted + num_accepted_tokens = gen_num_accepted + else: + if defer_outputs: + accepted_tokens = torch.empty( + (batch_size, runtime_draft_len + 1), + dtype=torch.int, + device=device) + num_accepted_tokens = torch.ones(batch_size, + dtype=torch.int, + device=device) + accepted_tokens[num_contexts:] = gen_accepted + num_accepted_tokens[num_contexts:] = gen_num_accepted num_accepted_tokens = self._apply_force_accepted_tokens( num_accepted_tokens, @@ -2748,6 +2936,26 @@ def _sample_and_accept_draft_tokens_rejection( spec_metadata.verify_lens.to(num_accepted_tokens.dtype)) return accepted_tokens, num_accepted_tokens + @staticmethod + def _tail_bookkeep_takes_outputs(gen_accepted, gen_num_accepted, + batch_size: int, runtime_draft_len: int, + device: torch.device) -> bool: + """Whether the acceptance kernel's outputs can be the accept's result. + + True when they are exactly the buffers the decode-only rejection accept + would copy them into: contiguous int32 ``[batch_size, + runtime_draft_len + 1]`` tokens and ``[batch_size]`` counts on + ``device``. Shapes only. + """ + outputs = ((gen_accepted, (batch_size, runtime_draft_len + 1)), + (gen_num_accepted, (batch_size, ))) + for t, shape in outputs: + if (not isinstance(t, torch.Tensor) or t.dtype != torch.int + or tuple(t.shape) != shape or not t.is_contiguous() + or t.device != device): + return False + return True + def _rng_state_per_request( self, spec_metadata: SpecMetadata, @@ -2910,16 +3118,31 @@ def advanced_sample_draft_block(self, gen_logits: torch.Tensor, dtype=torch.int32, device=gen_logits.device) - # Take the gen slice and repeat each request's value K times to line up - # with the flattened [num_gens*K, vocab] logits (K rows per request). - temps = spec_metadata.request_temperatures[ - num_contexts:batch_size].repeat_interleave(K) - top_ks = spec_metadata.request_top_ks[ - num_contexts:batch_size].repeat_interleave(K) - top_ps = spec_metadata.request_top_ps[ - num_contexts:batch_size].repeat_interleave(K) - min_ps = spec_metadata.request_min_ps[ - num_contexts:batch_size].repeat_interleave(K) + if (num_gens == 1 and tail_bookkeep_draft_len( + spec_metadata, num_contexts, batch_size) == K): + # Tail bookkeeping B1: the one request's per-token rows already + # hold its values K times (tail_bookkeep_draft_len), so read-only + # views replace the four repeats. The rejection accept reads the + # same buffers' views. + tail_bookkeep_log_once( + "B1", "draft-block sampling params as per-token views (no " + "repeat copies)") + temps = spec_metadata.temperatures[:K] + top_ks = spec_metadata.top_ks[:K] + top_ps = spec_metadata.top_ps[:K] + min_ps = spec_metadata.min_ps[:K] + else: + # Take the gen slice and repeat each request's value K times to + # line up with the flattened [num_gens*K, vocab] logits (K rows per + # request). + temps = spec_metadata.request_temperatures[ + num_contexts:batch_size].repeat_interleave(K) + top_ks = spec_metadata.request_top_ks[ + num_contexts:batch_size].repeat_interleave(K) + top_ps = spec_metadata.request_top_ps[ + num_contexts:batch_size].repeat_interleave(K) + min_ps = spec_metadata.request_min_ps[ + num_contexts:batch_size].repeat_interleave(K) flat_logits = gen_logits.reshape(num_gens * K, vocab) # A block sampler emits all K draft positions in one launch; they take @@ -3062,7 +3285,18 @@ def sample_draft_tokens(self, logits, spec_metadata, mapping_lm_head_tp) if is_block: tokens = self.advanced_sample_draft_block( - logits, spec_metadata, num_contexts, batch_size).long() + logits, spec_metadata, num_contexts, batch_size) + if (getattr(spec_metadata, "tail_bookkeep", False) + and self._d2t is None and tokens.dtype == torch.int32 + and tokens.is_contiguous()): + # Tail bookkeeping S1: with no d2t map the int64 widening + # only feeds the int32 cast below (an identity on int32), + # which then returns the sampler's fresh tokens as is. + tail_bookkeep_log_once( + "S1", "int32 draft tokens without the int64 round " + "trip") + else: + tokens = tokens.long() else: tokens = self.advanced_sample_draft(logits, spec_metadata, diff --git a/tensorrt_llm/_torch/speculative/spec_sampler_base.py b/tensorrt_llm/_torch/speculative/spec_sampler_base.py index 5fb5b185dc0b..1c81e24be184 100644 --- a/tensorrt_llm/_torch/speculative/spec_sampler_base.py +++ b/tensorrt_llm/_torch/speculative/spec_sampler_base.py @@ -27,9 +27,11 @@ import torch +from ..._utils import prefer_pinned from ...sampling_params import SamplingParams from ..pyexecutor.llm_request import LlmRequest, LlmRequestState, get_draft_token_length from ..pyexecutor.resource_manager import BaseResourceManager +from ..pyexecutor.runtime_hooks import SpecStepFold from ..pyexecutor.sampler import ( DEFAULT_BEAM_IDX, AsyncWorkerMixin, @@ -165,6 +167,16 @@ class SpecSampler(Sampler[SampleStateSpec], AsyncWorkerMixin): draft length every mode used to compute for itself. """ + # The model's speculative step-input staging, when it opted into one (bind_spec_step_fold). + spec_step_fold: Optional[SpecStepFold] = None + # Copy the per-slot store to the host on the D2H side stream (RuntimeHooks.sampler_d2h_side_stream). + d2h_side_stream_copies: bool = False + # Keep the stored rows' slot ids in a device buffer re-uploaded only when they change + # (RuntimeHooks.spec_small_h2d). + static_slot_ids: bool = False + _slot_ids_host: Optional[list[int]] = None + _slot_ids_cuda: Optional[torch.Tensor] = None + SampleState = SampleStateSpec def is_generation_model(self) -> bool: @@ -313,6 +325,76 @@ def __init__( new_tokens_lens=int_tensor((seq_slots,)), ) + def _device_slot_ids(self, seq_slots: list[int]) -> torch.Tensor: + """The slot ids on the device, uploaded only when they differ from the last upload. + + The buffer is this sampler's own: only this method writes it, so the last uploaded + list describes its contents, and the stream orders each upload after the scatter that + read the previous ids. + """ + n = len(seq_slots) + if self._slot_ids_cuda is None or self._slot_ids_cuda.numel() < n: + self._slot_ids_cuda = torch.empty(max(n, 8), dtype=torch.long, device="cuda") + self._slot_ids_host = None + if self._slot_ids_host != list(seq_slots): + self._slot_ids_cuda[:n].copy_( + torch.tensor(seq_slots, dtype=torch.long, pin_memory=prefer_pinned()), + non_blocking=True, + ) + self._slot_ids_host = list(seq_slots) + return self._slot_ids_cuda[:n] + + def _copy_outputs_to_host( + self, + o_verify_lens: Optional[torch.Tensor], + store_copies_on_side_stream: bool, + store_copies_done: Optional[torch.cuda.Event] = None, + ) -> tuple[SampleStateTensorsSpec, Optional[torch.cuda.Event]]: + """Host copies of the step's stored outputs, and the side-stream event to await them by (or None). + + With ``d2h_side_stream_copies`` the per-slot store is copied on the D2H side stream, so the copies + overlap the next step's input staging instead of preceding it on the main stream. That is sound + only while nothing rewrites the store before the sampler event is synchronized host-side, which + holds when this sampler stored the rows itself: the store is next written by the following + ``sample_async``, after the loop has synchronized this step. ``verify_lens`` is a forward output + buffer the next replay rewrites, so it is always copied on the main stream. + + ``store_copies_done`` (a step fold's ``store_copies_done``) is recorded on the side stream after the copies: + a forward that stores into the store waits on it first, which keeps the side-stream copies sound when the + next replay rather than the next ``sample_async`` writes the store. + """ + if not ( + self.d2h_side_stream_copies + and store_copies_on_side_stream + and not self._async_worker_active() + ): + # The stock copies, in the stock order. + host = SampleStateTensorsSpec( + new_tokens=self._copy_to_host(self.store.new_tokens), + new_tokens_lens=self._copy_to_host(self.store.new_tokens_lens), + next_draft_tokens=self._copy_to_host(self.store.next_draft_tokens), + verify_lens=( + self._copy_to_host(o_verify_lens) if o_verify_lens is not None else None + ), + ) + return host, None + verify_lens = self._copy_to_host(o_verify_lens) if o_verify_lens is not None else None + with self._make_side_stream_copier() as copier: + host = SampleStateTensorsSpec( + new_tokens=copier.stage_copy_to_host(self.store.new_tokens), + new_tokens_lens=copier.stage_copy_to_host(self.store.new_tokens_lens), + next_draft_tokens=copier.stage_copy_to_host(self.store.next_draft_tokens), + verify_lens=verify_lens, + ) + if store_copies_done is not None and copier.event is not None: + store_copies_done.record(self._d2h_side_stream) + return host, copier.event + + def bind_spec_step_fold(self, spec_step_fold: SpecStepFold) -> None: + """Let the model's forward store into this sampler's per-slot store (see SpecStepFold).""" + self.spec_step_fold = spec_step_fold + spec_step_fold.bind_sampler_store(self.store) + def _request_common_handling( self, request: LlmRequest, @@ -396,6 +478,57 @@ def update_requests( req.py_rewind_len = verified_len - req.py_num_accepted_draft_tokens self._request_common_handling(req, next_draft_tokens_list, runtime_draft_len) + def _store_outputs( + self, + seq_slots: list[int], + o_new_tokens: torch.Tensor, + o_next_new_tokens: torch.Tensor, + o_new_tokens_lens: torch.Tensor, + o_next_draft_tokens: torch.Tensor, + ) -> None: + """Store one step's worker outputs into the per-slot store.""" + if self.static_slot_ids: + slots = self._device_slot_ids(seq_slots) + else: + slots = torch.as_tensor(seq_slots, dtype=torch.long) + slots = slots.to(device="cuda", non_blocking=True) + step_fold = self.spec_step_fold + if step_fold is not None and step_fold.scatter( + slots, o_new_tokens, o_next_new_tokens, o_new_tokens_lens, o_next_draft_tokens + ): + return + + # Pad or truncate to match fixed-size store buffers for index_copy_. + # The worker output width tracks runtime_draft_len, which dynamic draft + # length shrinks below the statically allocated store width. + new_tokens_width = self.store.new_tokens.shape[0] + next_new_tokens_width = self.store.next_new_tokens.shape[0] + draft_tokens_width = self.store.next_draft_tokens.shape[1] + if o_new_tokens.shape[1] < new_tokens_width: + o_new_tokens = torch.nn.functional.pad( + o_new_tokens, (0, new_tokens_width - o_new_tokens.shape[1]) + ) + elif o_new_tokens.shape[1] > new_tokens_width: + o_new_tokens = o_new_tokens[:, :new_tokens_width] + if o_next_draft_tokens.shape[1] < draft_tokens_width: + o_next_draft_tokens = torch.nn.functional.pad( + o_next_draft_tokens, (0, draft_tokens_width - o_next_draft_tokens.shape[1]) + ) + elif o_next_draft_tokens.shape[1] > draft_tokens_width: + o_next_draft_tokens = o_next_draft_tokens[:, :draft_tokens_width] + if o_next_new_tokens.shape[1] < next_new_tokens_width: + o_next_new_tokens = torch.nn.functional.pad( + o_next_new_tokens, (0, next_new_tokens_width - o_next_new_tokens.shape[1]) + ) + elif o_next_new_tokens.shape[1] > next_new_tokens_width: + o_next_new_tokens = o_next_new_tokens[:, :next_new_tokens_width] + + # Use index_copy_ for efficient copying (slots are unique) + self.store.new_tokens.squeeze(-1).T.index_copy_(0, slots, o_new_tokens) + self.store.next_new_tokens.squeeze(-1).T.index_copy_(0, slots, o_next_new_tokens) + self.store.new_tokens_lens.index_copy_(0, slots, o_new_tokens_lens) + self.store.next_draft_tokens.index_copy_(0, slots, o_next_draft_tokens) + def sample_async( self, scheduled_requests: ScheduledRequests, @@ -442,8 +575,7 @@ def sample_async( for r in sampling_requests ] - slots = torch.as_tensor([r.py_seq_slot for r in sampling_requests], dtype=torch.long) - slots = slots.to(device="cuda", non_blocking=True) + seq_slots = [r.py_seq_slot for r in sampling_requests] o_new_tokens = outputs["new_tokens"][num_skip : num_skip + num_sampling_requests] o_new_tokens_lens = outputs["new_tokens_lens"][num_skip : num_skip + num_sampling_requests] @@ -489,36 +621,14 @@ def sample_async( raise ValueError("verify_lens must contain one int32 token window per output row") o_verify_lens = o_verify_lens[num_skip : num_skip + num_sampling_requests] - # Pad or truncate to match fixed-size store buffers for index_copy_. - # The worker output width tracks runtime_draft_len, which dynamic draft - # length shrinks below the statically allocated store width. - new_tokens_width = self.store.new_tokens.shape[0] - next_new_tokens_width = self.store.next_new_tokens.shape[0] - draft_tokens_width = self.store.next_draft_tokens.shape[1] - if o_new_tokens.shape[1] < new_tokens_width: - o_new_tokens = torch.nn.functional.pad( - o_new_tokens, (0, new_tokens_width - o_new_tokens.shape[1]) - ) - elif o_new_tokens.shape[1] > new_tokens_width: - o_new_tokens = o_new_tokens[:, :new_tokens_width] - if o_next_draft_tokens.shape[1] < draft_tokens_width: - o_next_draft_tokens = torch.nn.functional.pad( - o_next_draft_tokens, (0, draft_tokens_width - o_next_draft_tokens.shape[1]) - ) - elif o_next_draft_tokens.shape[1] > draft_tokens_width: - o_next_draft_tokens = o_next_draft_tokens[:, :draft_tokens_width] - if o_next_new_tokens.shape[1] < next_new_tokens_width: - o_next_new_tokens = torch.nn.functional.pad( - o_next_new_tokens, (0, next_new_tokens_width - o_next_new_tokens.shape[1]) + step_fold = self.spec_step_fold + stored_here = step_fold is None or not step_fold.sampler_rows_stored( + outputs, num_skip, seq_slots + ) + if stored_here: + self._store_outputs( + seq_slots, o_new_tokens, o_next_new_tokens, o_new_tokens_lens, o_next_draft_tokens ) - elif o_next_new_tokens.shape[1] > next_new_tokens_width: - o_next_new_tokens = o_next_new_tokens[:, :next_new_tokens_width] - - # Use index_copy_ for efficient copying (slots are unique) - self.store.new_tokens.squeeze(-1).T.index_copy_(0, slots, o_new_tokens) - self.store.next_new_tokens.squeeze(-1).T.index_copy_(0, slots, o_next_new_tokens) - self.store.new_tokens_lens.index_copy_(0, slots, o_new_tokens_lens) - self.store.next_draft_tokens.index_copy_(0, slots, o_next_draft_tokens) # Create sample state with async D2H copy device_tensors = SampleStateTensorsSpec( @@ -527,13 +637,20 @@ def sample_async( next_draft_tokens=self.store.next_draft_tokens, ) - host_tensors = SampleStateTensorsSpec( - new_tokens=self._copy_to_host(self.store.new_tokens), - new_tokens_lens=self._copy_to_host(self.store.new_tokens_lens), - next_draft_tokens=self._copy_to_host(self.store.next_draft_tokens), - verify_lens=(self._copy_to_host(o_verify_lens) if o_verify_lens is not None else None), + # A fold whose forward waits for the store copies (store_copies_done) lets them go on the side stream + # on the steps its forward stored the rows too. + store_copies_done = getattr(step_fold, "store_copies_done", None) if step_fold else None + host_tensors, side_stream_event = self._copy_outputs_to_host( + o_verify_lens, + store_copies_on_side_stream=stored_here or store_copies_done is not None, + store_copies_done=store_copies_done, + ) + # Without side-stream copies the call is the stock one (no argument). + sampler_event = ( + self._record_sampler_event(side_stream_event) + if side_stream_event is not None + else self._record_sampler_event() ) - sampler_event = self._record_sampler_event() # Add dummy draft tokens to context requests for KV cache preparation for request in finished_context_requests: diff --git a/tests/unittest/_torch/executor/test_h2d_cache.py b/tests/unittest/_torch/executor/test_h2d_cache.py new file mode 100644 index 000000000000..fb2360a86083 --- /dev/null +++ b/tests/unittest/_torch/executor/test_h2d_cache.py @@ -0,0 +1,580 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Skipping host-to-device metadata copies whose values the destination already holds.""" + +import threading + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor import h2d_cache +from tensorrt_llm._torch.pyexecutor.h2d_cache import ( + H2DSkipCache, + copy_h2d_if_changed, + fill_tail_if_changed, +) + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required") + + +@pytest.fixture +def cache(): + """A fresh cache, active for the test body.""" + skip_cache = H2DSkipCache() + with h2d_cache.active(skip_cache): + yield skip_cache + torch.cuda.synchronize() + + +def _host(values, dtype=torch.int32): + return torch.tensor(values, dtype=dtype).pin_memory() + + +def _device_zeros(n, dtype=torch.int32): + return torch.zeros(n, dtype=dtype, device="cuda") + + +def test_without_an_active_cache_every_copy_is_issued_like_copy_(): + assert not h2d_cache.enabled() + dst = _device_zeros(4) + reference = _device_zeros(4) + src = _host([3, 1, 4, 1]) + + for _ in range(3): + assert copy_h2d_if_changed(dst, src) is True + reference.copy_(src, non_blocking=True) + torch.cuda.synchronize() + assert torch.equal(dst, reference) + + +def test_active_scopes_the_cache_to_the_with_block(): + skip_cache = H2DSkipCache() + dst = _device_zeros(2) + values = _host([1, 2]) + + with h2d_cache.active(skip_cache): + assert h2d_cache.enabled() + assert copy_h2d_if_changed(dst, values) is True + assert copy_h2d_if_changed(dst, values) is False + assert not h2d_cache.enabled() + assert copy_h2d_if_changed(dst, values) is True + # Re-activating the same cache finds its shadow again. + with h2d_cache.active(skip_cache): + assert copy_h2d_if_changed(dst, values) is False + + +def test_active_none_disables_skipping_inside_an_outer_cache(cache): + dst = _device_zeros(2) + values = _host([1, 2]) + assert copy_h2d_if_changed(dst, values) is True + + with h2d_cache.active(None): + assert not h2d_cache.enabled() + assert copy_h2d_if_changed(dst, values) is True + assert copy_h2d_if_changed(dst, values) is False + + +def test_two_caches_do_not_share_shadows(cache): + dst = _device_zeros(3) + values = _host([7, 8, 9]) + other = H2DSkipCache() + + assert copy_h2d_if_changed(dst, values) is True + with h2d_cache.active(other): + # The other cache has never seen dst. + assert copy_h2d_if_changed(dst, values) is True + assert copy_h2d_if_changed(dst, values) is False + assert copy_h2d_if_changed(dst, values) is False + + +def test_a_cache_active_on_one_thread_is_invisible_to_another(cache): + dst = _device_zeros(3) + values = _host([4, 5, 6]) + assert copy_h2d_if_changed(dst, values) is True + seen = {} + + def other_thread(): + seen["enabled"] = h2d_cache.enabled() + seen["copied"] = [copy_h2d_if_changed(dst, values) for _ in range(2)] + torch.cuda.synchronize() + + thread = threading.Thread(target=other_thread) + thread.start() + thread.join() + + assert seen == {"enabled": False, "copied": [True, True]} + assert copy_h2d_if_changed(dst, values) is False + + +def test_threads_with_their_own_caches_keep_them_apart(): + dst = [_device_zeros(2) for _ in range(2)] + values = _host([1, 1]) + caches = [H2DSkipCache(), H2DSkipCache()] + barrier = threading.Barrier(2) + results = {} + + def worker(index): + with h2d_cache.active(caches[index]): + first = copy_h2d_if_changed(dst[index], values) + barrier.wait() + second = copy_h2d_if_changed(dst[index], values) + # The other thread's destination was never shadowed by this cache. + other = copy_h2d_if_changed(dst[1 - index], values) + torch.cuda.synchronize() + results[index] = (first, second, other) + + threads = [threading.Thread(target=worker, args=(i,)) for i in range(2)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + + assert results == {0: (True, False, True), 1: (True, False, True)} + + +def test_a_repeated_identical_copy_is_skipped(cache): + dst = _device_zeros(4) + + assert copy_h2d_if_changed(dst, _host([5, 6, 7, 8])) is True + assert copy_h2d_if_changed(dst, _host([5, 6, 7, 8])) is False + torch.cuda.synchronize() + assert dst.tolist() == [5, 6, 7, 8] + + +def test_clear_makes_the_next_identical_copy_issue(cache): + """The runner clears the cache after a discarded step, whose copies never landed.""" + dst = _device_zeros(3) + values = _host([1, 2, 3]) + assert copy_h2d_if_changed(dst, values) is True + + cache.clear() + + assert copy_h2d_if_changed(dst, values) is True + assert copy_h2d_if_changed(dst, values) is False + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3] + + +def test_changed_values_are_copied(cache): + dst = _device_zeros(4) + + assert copy_h2d_if_changed(dst, _host([1, 2, 3, 4])) is True + assert copy_h2d_if_changed(dst, _host([1, 2, 3, 5])) is True + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 5] + assert copy_h2d_if_changed(dst, _host([1, 2, 3, 5])) is False + + +def test_the_shadow_is_a_snapshot_of_the_source(cache): + """Mutating the host source after the copy must not make a later copy look unchanged.""" + dst = _device_zeros(3) + src = _host([1, 2, 3]) + + assert copy_h2d_if_changed(dst, src) is True + torch.cuda.synchronize() + src[0] = 9 + assert copy_h2d_if_changed(dst, src) is True + torch.cuda.synchronize() + assert dst.tolist() == [9, 2, 3] + + +def test_a_source_dtype_change_copies(cache): + dst = _device_zeros(3) + + assert copy_h2d_if_changed(dst, _host([1, 2, 3], torch.int32)) is True + assert copy_h2d_if_changed(dst, _host([1, 2, 3], torch.int64)) is True + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3] + + +def test_a_copy_through_another_view_replaces_the_shadow(cache): + buf = _device_zeros(8) + head, tail = buf[:4], buf[4:] + values = _host([1, 2, 3, 4]) + + assert copy_h2d_if_changed(head, values) is True + assert copy_h2d_if_changed(tail, _host([5, 6, 7, 8])) is True + # The shadow now describes the tail, so the head copy must not be skipped. + assert copy_h2d_if_changed(head, values) is True + assert copy_h2d_if_changed(head, values) is False + torch.cuda.synchronize() + assert buf.tolist() == [1, 2, 3, 4, 5, 6, 7, 8] + + +def test_an_overlapping_view_never_skips_on_stale_state(cache): + buf = _device_zeros(6) + values = _host([1, 1, 1, 1]) + + assert copy_h2d_if_changed(buf[:4], values) is True + assert copy_h2d_if_changed(buf[2:6], _host([2, 2, 2, 2])) is True + # buf[2:4] was overwritten through the other view: same values, same view, must copy. + assert copy_h2d_if_changed(buf[:4], values) is True + torch.cuda.synchronize() + assert buf.tolist() == [1, 1, 1, 1, 2, 2] + + +def test_a_view_with_another_shape_of_the_same_bytes_copies(cache): + buf = _device_zeros(4) + + assert copy_h2d_if_changed(buf, _host([1, 2, 3, 4])) is True + assert copy_h2d_if_changed(buf.view(2, 2), _host([[1, 2], [3, 4]])) is True + assert copy_h2d_if_changed(buf.view(2, 2), _host([[1, 2], [3, 4]])) is False + + +def test_a_cuda_source_always_copies_and_drops_the_shadow(cache): + dst = _device_zeros(4) + values = _host([1, 2, 3, 4]) + device_src = torch.arange(4, dtype=torch.int32, device="cuda") + + assert copy_h2d_if_changed(dst, values) is True + assert copy_h2d_if_changed(dst, device_src) is True + assert copy_h2d_if_changed(dst, device_src) is True + # The device-side write replaced what the shadow described. + assert copy_h2d_if_changed(dst, values) is True + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 4] + + +def test_a_host_destination_always_copies(cache): + dst = torch.zeros(3, dtype=torch.int32) + + assert copy_h2d_if_changed(dst, _host([1, 2, 3])) is True + assert copy_h2d_if_changed(dst, _host([1, 2, 3])) is True + assert dst.tolist() == [1, 2, 3] + + +def test_capture_always_copies_and_drops_the_shadow(cache): + dst = _device_zeros(4) + values = _host([4, 3, 2, 1]) + assert copy_h2d_if_changed(dst, values) is True + torch.cuda.synchronize() + + graph = torch.cuda.CUDAGraph() + results = [] + with torch.cuda.graph(graph): + results.append(copy_h2d_if_changed(dst, values)) + assert results == [True] + + dst.zero_() + graph.replay() + torch.cuda.synchronize() + assert dst.tolist() == [4, 3, 2, 1] + # The captured copy reruns on every replay, so the shadow cannot vouch for the buffer. + assert copy_h2d_if_changed(dst, values) is True + + +def test_without_an_active_cache_every_tail_fill_runs(): + buf = _device_zeros(6) + fill_tail_if_changed(buf, 2, -1) + buf[3] = 5 + fill_tail_if_changed(buf, 2, -1) + fill_tail_if_changed(buf, 6, -1) + assert buf.tolist() == [0, 0, -1, -1, -1, -1] + + +def test_a_repeated_tail_fill_is_skipped_and_a_moved_start_refills(cache): + buf = _device_zeros(6) + fill_tail_if_changed(buf, 2, -1) + # Writes nothing else is allowed to make; they show which fills ran. + buf[3] = 5 + fill_tail_if_changed(buf, 2, -1) + assert buf[3].item() == 5 + fill_tail_if_changed(buf, 4, -1) + buf[3] = 7 + fill_tail_if_changed(buf, 2, -1) + assert buf.tolist() == [0, 0, -1, -1, -1, -1] + + +def test_a_whole_buffer_write_drops_the_tail_record(cache): + buf = _device_zeros(6) + fill_tail_if_changed(buf, 2, -1) + buf.copy_(torch.arange(6, dtype=torch.int32, device="cuda")) + fill_tail_if_changed(buf, 6, -1) + fill_tail_if_changed(buf, 2, -1) + assert buf.tolist() == [0, 1, -1, -1, -1, -1] + + +def test_clear_drops_the_tail_records(cache): + buf = _device_zeros(6) + fill_tail_if_changed(buf, 2, -1) + buf[4] = 9 + cache.clear() + fill_tail_if_changed(buf, 2, -1) + assert buf[2:].eq(-1).all() + + +def test_a_tail_record_keeps_its_storage_so_a_new_buffer_cannot_inherit_it(cache): + buf = _device_zeros(1 << 20) + fill_tail_if_changed(buf, 3, -1) + address = buf.data_ptr() + del buf + # Without the record holding the storage, the allocator hands this same block back, and the fill below would be + # skipped on the freed buffer's record. + fresh = _device_zeros(1 << 20) + assert fresh.data_ptr() != address + fill_tail_if_changed(fresh, 3, -1) + assert fresh[3:].eq(-1).all() + + +# --------------------------------------------------------------------------- +# changed_span (RuntimeHooks.h2d_changed_span): a changed packed 2-D upload stages only each row's changed span +# --------------------------------------------------------------------------- + +ROWS, WIDTH = 4, 1024 + + +def _span_pack(): + from tensorrt_llm._torch.pyexecutor.step_inputs import StepInputPack + + return StepInputPack(torch.device("cuda"), parallel_scatter=True) + + +def _table(seed, rows=ROWS, width=WIDTH): + generator = torch.Generator().manual_seed(seed) + return torch.randint( + 0, 1 << 20, (rows, width), generator=generator, dtype=torch.int32 + ).pin_memory() + + +def _upload(pack, span_cache, dst, src): + """One step: open the pack, route the copy through the cache, issue the pack; returns the staged copies.""" + from tensorrt_llm._torch.pyexecutor import step_inputs + + with h2d_cache.active(span_cache), step_inputs.active(pack): + step_inputs.begin() + copied = copy_h2d_if_changed(dst, src, pack=True) + staged = list(pack.h2d) + step_inputs.flush() + torch.cuda.synchronize() + return copied, staged + + +def test_changed_span_is_off_by_default_and_without_init(): + assert not H2DSkipCache().changed_span + assert not object.__new__(H2DSkipCache).changed_span + + +@pytest.mark.parametrize("changed_span", [False, True]) +def test_changed_span_device_rows_equal_the_source_after_every_step(changed_span): + """Bitwise: whatever is staged, the device view holds the source after each step.""" + pack = _span_pack() + span_cache = H2DSkipCache(changed_span=changed_span) + buf = torch.full((8, 2 * WIDTH), -7, dtype=torch.int32, device="cuda") + dst = buf[:ROWS, :WIDTH] + src = _table(0) + generator = torch.Generator().manual_seed(1) + for step in range(40): + src = src.clone().pin_memory() + # A request crosses into a new page: one entry of one row changes (sometimes several, in several rows). + for _ in range(1 + step % 3): + row = int(torch.randint(0, ROWS, (1,), generator=generator)) + col = int(torch.randint(0, WIDTH, (1,), generator=generator)) + src[row, col] = step + _upload(pack, span_cache, dst, src) + assert torch.equal(dst.cpu(), src) + # Rows and columns outside the view were never written. + assert bool((buf[ROWS:] == -7).all()) and bool((buf[:, WIDTH:] == -7).all()) + assert (span_cache.span_uploads > 0) is changed_span + + +def test_changed_span_one_changed_entry_stages_one_element(): + pack = _span_pack() + span_cache = H2DSkipCache(changed_span=True) + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + src = _table(2) + _upload(pack, span_cache, dst, src) + + src = src.clone().pin_memory() + src[2, 517] += 1 + copied, staged = _upload(pack, span_cache, dst, src) + + assert copied is True + assert staged == [(dst[2, 517].data_ptr(), staged[0][1], 4)] + assert torch.equal(dst.cpu(), src) + assert (span_cache.span_uploads, span_cache.span_bytes) == (1, 4) + assert span_cache.span_full_bytes == ROWS * WIDTH * 4 + + +def test_changed_span_each_changed_row_stages_its_first_to_last_changed_column(): + pack = _span_pack() + span_cache = H2DSkipCache(changed_span=True) + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + src = _table(3) + _upload(pack, span_cache, dst, src) + + src = src.clone().pin_memory() + src[0, 10] += 1 + src[0, 20] += 1 + src[3, 1000] += 1 + _, staged = _upload(pack, span_cache, dst, src) + + assert [(ptr, nbytes) for ptr, _, nbytes in staged] == [ + (dst[0, 10].data_ptr(), 11 * 4), + (dst[3, 1000].data_ptr(), 4), + ] + assert torch.equal(dst.cpu(), src) + + +def test_changed_span_empty_span_stages_nothing_and_keeps_the_device_values(): + """PR-CTX (d): an empty span. A narrower view of the same rows whose values all equal the shadow's: no column + changed, so nothing is staged (the copy still counts as issued, and the shadow follows the new view).""" + pack = _span_pack() + span_cache = H2DSkipCache(changed_span=True) + buf = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + src = _table(10) + _upload(pack, span_cache, buf, src) + + narrow = src[:, : WIDTH // 2].contiguous().pin_memory() + copied, staged = _upload(pack, span_cache, buf[:, : WIDTH // 2], narrow) + + assert copied is True + assert staged == [] + assert torch.equal(buf.cpu(), src) + assert (span_cache.span_uploads, span_cache.span_bytes) == (1, 0) + # The shadow is the narrow view now: the same values again are skipped. + copied, staged = _upload(pack, span_cache, buf[:, : WIDTH // 2], narrow) + assert copied is False and staged == [] + + +def test_changed_span_whole_buffer_span_is_staged_whole(): + """PR-CTX (d): a whole-buffer span. The first and last column of every row changed, so every row's span is the + whole row: over the fraction, the view is staged whole (one descriptor), not as per-row spans.""" + pack = _span_pack() + span_cache = H2DSkipCache(changed_span=True) + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + src = _table(11) + _upload(pack, span_cache, dst, src) + + src = src.clone().pin_memory() + src[:, 0] += 1 + src[:, -1] += 1 + _, staged = _upload(pack, span_cache, dst, src) + + assert [nbytes for _, _, nbytes in staged] == [ROWS * WIDTH * 4] + assert torch.equal(dst.cpu(), src) + assert span_cache.span_uploads == 0 + + +def test_changed_span_at_the_fraction_is_still_a_span(): + pack = _span_pack() + span_cache = H2DSkipCache(changed_span=True) + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + src = _table(12) + _upload(pack, span_cache, dst, src) + + src = src.clone().pin_memory() + half = int(H2DSkipCache.CHANGED_SPAN_MAX_FRACTION * WIDTH) + src[:, :half] += 1 # exactly the fraction of the view's elements + _, staged = _upload(pack, span_cache, dst, src) + + assert [nbytes for _, _, nbytes in staged] == [half * 4] * ROWS + assert torch.equal(dst.cpu(), src) + + +def test_changed_span_wider_view_uploads_the_columns_the_shadow_does_not_cover(): + pack = _span_pack() + span_cache = H2DSkipCache(changed_span=True) + buf = torch.full((ROWS, WIDTH), -1, dtype=torch.int32, device="cuda") + src = _table(4) + _upload(pack, span_cache, buf[:, : WIDTH // 2], src[:, : WIDTH // 2].contiguous().pin_memory()) + + _, staged = _upload( + pack, + span_cache, + buf[:, : WIDTH // 2 + 3], + src[:, : WIDTH // 2 + 3].contiguous().pin_memory(), + ) + + assert [nbytes for _, _, nbytes in staged] == [3 * 4] * ROWS + assert torch.equal(buf[:, : WIDTH // 2 + 3].cpu(), src[:, : WIDTH // 2 + 3]) + + +def test_changed_span_a_mostly_changed_table_is_staged_whole(): + pack = _span_pack() + span_cache = H2DSkipCache(changed_span=True) + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + _upload(pack, span_cache, dst, _table(5)) + + src = _table(6) + _, staged = _upload(pack, span_cache, dst, src) + + assert [nbytes for _, _, nbytes in staged] == [ROWS * WIDTH * 4] + assert torch.equal(dst.cpu(), src) + + +def test_changed_span_off_by_default_the_whole_view_is_staged(): + pack = _span_pack() + span_cache = H2DSkipCache() + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + src = _table(7) + _upload(pack, span_cache, dst, src) + + src = src.clone().pin_memory() + src[1, 1] += 1 + _, staged = _upload(pack, span_cache, dst, src) + + assert [nbytes for _, _, nbytes in staged] == [ROWS * WIDTH * 4] + + +def test_changed_span_an_unpacked_copy_copies_the_whole_view(): + span_cache = H2DSkipCache(changed_span=True) + dst = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + src = _table(8) + with h2d_cache.active(span_cache): + copy_h2d_if_changed(dst, src) + src = src.clone().pin_memory() + src[1, 1] += 1 + assert copy_h2d_if_changed(dst, src) is True + torch.cuda.synchronize() + assert torch.equal(dst.cpu(), src) + assert span_cache.span_uploads == 0 + + +def test_changed_span_a_view_with_another_row_stride_is_staged_whole(): + """The shadow describes other device bytes: nothing of it may be trusted.""" + pack = _span_pack() + span_cache = H2DSkipCache(changed_span=True) + buf = torch.zeros((2 * ROWS, WIDTH), dtype=torch.int32, device="cuda") + src = _table(9) + _upload(pack, span_cache, buf[:ROWS], src) + + _, staged = _upload(pack, span_cache, buf[::2], src) + + assert [nbytes for _, _, nbytes in staged] == [WIDTH * 4] * ROWS + assert torch.equal(buf[::2].cpu(), src) + + +@pytest.mark.parametrize("changed_span", [False, True]) +def test_changed_span_keeps_a_page_crossing_step_in_the_graph_window(changed_span): + """A 16 KB page table (over the in-graph window) gains one page: with changed_span the step's pack still fits the + window and the replay's prologue issues it; without, the step falls back to an eager upload and scatter.""" + from tensorrt_llm._torch.pyexecutor import step_inputs + from tensorrt_llm._torch.pyexecutor.step_inputs import StepInputPack + + pack = StepInputPack(torch.device("cuda"), parallel_scatter=True, in_graph=True) + span_cache = H2DSkipCache(changed_span=changed_span) + table = torch.zeros((ROWS, WIDTH), dtype=torch.int32, device="cuda") + out = torch.zeros(ROWS, dtype=torch.int32, device="cuda") + graph = torch.cuda.CUDAGraph() + with step_inputs.active(pack): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + assert step_inputs.capture_prologue() is pack + out.copy_(table[:, -1]) + src = _table(13) + table.copy_(src) + with h2d_cache.active(span_cache): + copy_h2d_if_changed(table, src) # the shadow of the table the device holds + torch.cuda.synchronize() + src = src.clone().pin_memory() + src[1, -1] = 4242 + with h2d_cache.active(span_cache), step_inputs.active(pack): + step_inputs.begin() + copy_h2d_if_changed(table, src, pack=True) + step_inputs.flush(in_graph=True) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(table.cpu(), src) + assert out.tolist() == src[:, -1].tolist() + expect = (1, 0) if changed_span else (0, 1) + assert (pack.graph_windows, pack.eager_fallbacks) == expect diff --git a/tests/unittest/_torch/executor/test_instantiate_sampler_hooks.py b/tests/unittest/_torch/executor/test_instantiate_sampler_hooks.py new file mode 100644 index 000000000000..8c87369c9eca --- /dev/null +++ b/tests/unittest/_torch/executor/test_instantiate_sampler_hooks.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""instantiate_sampler hands a one-model speculative sampler the model's host-tail hooks +(RuntimeHooks.sampler_d2h_side_stream / spec_small_h2d) and logs the D2H side-stream engagement line only when it +is on.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor import _util +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks +from tensorrt_llm._torch.speculative.spec_sampler_base import SpecSampler +from tensorrt_llm.mapping import Mapping + +pytestmark = pytest.mark.cpu_only + +_D2H_LINE = "[M3 host] spec sampler store copies on the D2H side stream" + + +class _Fold: + """A SpecStepFold stand-in; binding hands it the sampler's store.""" + + store = None + + def bind_sampler_store(self, store): + self.store = store + + +def _instantiate(monkeypatch, hooks, sampler): + lines = [] + monkeypatch.setattr(_util, "confidential_compute_enabled", lambda: False) + monkeypatch.setattr(_util, "resolve_max_num_sequences", lambda *args, **kwargs: 4) + monkeypatch.setattr(_util, "get_spec_decoder", lambda sampler_args, spec_config: sampler) + monkeypatch.setattr(_util.logger, "info", lines.append) + + class _Model(torch.nn.Module): + runtime_hooks = hooks + + engine = SimpleNamespace( + max_seq_len=64, + model=_Model(), + spec_config=SimpleNamespace( + spec_dec_mode=SimpleNamespace(has_spec_decoder=lambda: True), max_draft_len=3 + ), + ) + llm_args = SimpleNamespace( + sampler_force_async_worker=False, + disable_overlap_scheduler=False, + enable_speculative_beam_history_d2h=False, + ) + result = _util.instantiate_sampler( + engine, + llm_args, + Mapping(), + max_batch_size=4, + max_beam_width=1, + mm_encoder_only=False, + speculative_config=None, + max_num_sequences=4, + ) + assert result is sampler + return lines + + +def _spec_sampler(): + sampler = object.__new__(SpecSampler) + sampler.store = object() + return sampler + + +def test_default_hooks_leave_the_sampler_stock_and_log_nothing(monkeypatch): + sampler = _spec_sampler() + + lines = _instantiate(monkeypatch, RuntimeHooks(), sampler) + + assert sampler.d2h_side_stream_copies is False + assert sampler.static_slot_ids is False + assert sampler.spec_step_fold is None + assert lines == [] + + +@pytest.mark.parametrize( + "side, static", + [(True, False), (False, True), (True, True)], + ids=["d2h_side_stream", "spec_small_h2d", "both"], +) +def test_the_hooks_reach_the_sampler(monkeypatch, side, static): + sampler = _spec_sampler() + fold = _Fold() + hooks = RuntimeHooks(sampler_d2h_side_stream=side, spec_small_h2d=static, spec_step_fold=fold) + + lines = _instantiate(monkeypatch, hooks, sampler) + + assert sampler.d2h_side_stream_copies is side + assert sampler.static_slot_ids is static + assert sampler.spec_step_fold is fold and fold.store is sampler.store + # spec_small_h2d's line is DecoderMetadata.setup_speculative's, not this one. + assert lines == ([_D2H_LINE] if side else []) + + +def test_a_sampler_that_is_not_a_spec_sampler_is_left_alone(monkeypatch): + other = SimpleNamespace() + + lines = _instantiate( + monkeypatch, RuntimeHooks(sampler_d2h_side_stream=True, spec_small_h2d=True), other + ) + + assert not hasattr(other, "d2h_side_stream_copies") + assert not hasattr(other, "static_slot_ids") + assert lines == [] diff --git a/tests/unittest/_torch/executor/test_kv_cache_v2_block_offset_guard.py b/tests/unittest/_torch/executor/test_kv_cache_v2_block_offset_guard.py new file mode 100644 index 000000000000..581cc255684a --- /dev/null +++ b/tests/unittest/_torch/executor/test_kv_cache_v2_block_offset_guard.py @@ -0,0 +1,201 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""KVCacheManagerV2's opt-in unchanged block-table skip (``skip_unchanged_block_offsets``) and the RuntimeHooks field +that arms it on a separate draft KV cache manager (``skip_unchanged_draft_block_offsets``). + +The guard is driven on a manager built without ``__init__`` (every attribute it reads is declared on the class or set +here). The CUDA test checks the bytes: a guarded sequence of copies (the in-tree copy op issued only when the guard +does not skip) leaves the destination equal to always copying, after every step of a random host-table history. +""" + +import random +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.runner import DecoderRunner +from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + +POOLS, CAPACITY, WIDTH = 2, 6, 16 + + +def _manager(pin: bool = False) -> KVCacheManagerV2: + manager = object.__new__(KVCacheManagerV2) + manager.host_kv_cache_block_offsets = torch.zeros( + POOLS, CAPACITY, 2, WIDTH, dtype=torch.int32, pin_memory=pin + ) + manager.host_kv_cache_block_offsets[:, :, 0, :] = torch.arange( + CAPACITY * WIDTH, dtype=torch.int32 + ).view(CAPACITY, WIDTH) + manager.skip_unchanged_block_offsets = True + return manager + + +def _rows(*rows: int) -> torch.Tensor: + return torch.tensor(rows, dtype=torch.int32) + + +@pytest.mark.cpu_only +def test_the_guard_is_off_and_unarmed_by_default(): + assert KVCacheManagerV2.skip_unchanged_block_offsets is False + assert RuntimeHooks().skip_unchanged_draft_block_offsets is False + assert not RuntimeHooks().skips_unchanged_metadata_uploads + assert RuntimeHooks(skip_unchanged_draft_block_offsets=True).skips_unchanged_metadata_uploads + + +@pytest.mark.cpu_only +def test_an_unchanged_decode_copy_is_skipped(): + manager, dst = _manager(), torch.zeros(POOLS, CAPACITY, 2, WIDTH, dtype=torch.int32) + + assert not manager._block_offsets_unchanged(dst, _rows(3), 0) + assert manager._block_offsets_unchanged(dst, _rows(3), 0) + assert manager._block_offsets_unchanged(dst, _rows(3), 0) + assert (manager.block_offset_copies, manager.block_offset_skips) == (1, 2) + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("pool", range(POOLS)) +def test_a_changed_page_index_in_a_copied_row_copies(pool): + manager, dst = _manager(), torch.zeros(POOLS, CAPACITY, 2, WIDTH, dtype=torch.int32) + assert not manager._block_offsets_unchanged(dst, _rows(1, 3), 0) + + manager.host_kv_cache_block_offsets[pool, 3, 0, WIDTH - 1] += 1 + assert not manager._block_offsets_unchanged(dst, _rows(1, 3), 0) + assert manager._block_offsets_unchanged(dst, _rows(1, 3), 0) + + +@pytest.mark.cpu_only +def test_field_one_and_rows_not_copied_do_not_force_a_copy(): + """The copy reads field 0 of the copied rows only.""" + manager, dst = _manager(), torch.zeros(POOLS, CAPACITY, 2, WIDTH, dtype=torch.int32) + assert not manager._block_offsets_unchanged(dst, _rows(2), 0) + + manager.host_kv_cache_block_offsets[:, 2, 1, :] = 7 + manager.host_kv_cache_block_offsets[:, 4, 0, :] = 9 + assert manager._block_offsets_unchanged(dst, _rows(2), 0) + + +@pytest.mark.cpu_only +def test_another_row_set_or_order_copies(): + manager, dst = _manager(), torch.zeros(POOLS, CAPACITY, 2, WIDTH, dtype=torch.int32) + assert not manager._block_offsets_unchanged(dst, _rows(1, 3), 0) + + assert not manager._block_offsets_unchanged(dst, _rows(3, 1), 0) + assert not manager._block_offsets_unchanged(dst, _rows(3), 0) + assert manager._block_offsets_unchanged(dst, _rows(3), 0) + + +@pytest.mark.cpu_only +def test_a_step_with_context_requests_copies_and_drops_the_record(): + manager, dst = _manager(), torch.zeros(POOLS, CAPACITY, 2, WIDTH, dtype=torch.int32) + assert not manager._block_offsets_unchanged(dst, _rows(3), 0) + + assert not manager._block_offsets_unchanged(dst, _rows(3), 1) + assert not manager._block_offsets_unchanged(dst, _rows(3), 0) + assert manager._block_offsets_unchanged(dst, _rows(3), 0) + + +@pytest.mark.cpu_only +def test_another_view_of_the_same_storage_copies_and_replaces_the_record(): + manager = _manager() + storage = torch.zeros(POOLS, CAPACITY + 2, 2, WIDTH, dtype=torch.int32) + whole, prefix = storage, storage[:, :CAPACITY] + assert not manager._block_offsets_unchanged(whole, _rows(3), 0) + + assert not manager._block_offsets_unchanged(prefix, _rows(3), 0) + assert not manager._block_offsets_unchanged(whole, _rows(3), 0) + assert manager._block_offsets_unchanged(whole, _rows(3), 0) + + +@pytest.mark.cpu_only +def test_destinations_in_different_storages_keep_their_own_records(): + manager = _manager() + first, second = (torch.zeros(POOLS, CAPACITY, 2, WIDTH, dtype=torch.int32) for _ in range(2)) + assert not manager._block_offsets_unchanged(first, _rows(3), 0) + assert not manager._block_offsets_unchanged(second, _rows(3), 0) + + assert manager._block_offsets_unchanged(first, _rows(3), 0) + assert manager._block_offsets_unchanged(second, _rows(3), 0) + + +@pytest.mark.cpu_only +def test_the_counts_are_logged_at_powers_of_two(monkeypatch): + lines = [] + monkeypatch.setattr( + "tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2.logger.info", lines.append + ) + manager, dst = _manager(), torch.zeros(POOLS, CAPACITY, 2, WIDTH, dtype=torch.int32) + for _ in range(9): + manager._block_offsets_unchanged(dst, _rows(3), 0) + + assert lines == [ + f"[M3 host] block-offset copies skipped when unchanged: {skips} of {calls}" + for skips, calls in ((0, 1), (1, 2), (3, 4), (7, 8)) + ] + + +@pytest.mark.cpu_only +def test_the_hook_arms_only_a_separate_v2_draft_manager(monkeypatch): + lines = [] + monkeypatch.setattr( + "tensorrt_llm._torch.pyexecutor.engine.runners.decoder.runner.logger.info", lines.append + ) + arm = DecoderRunner._arm_draft_block_offset_guard + draft = object.__new__(KVCacheManagerV2) + + arm(SimpleNamespace(runtime_hooks=RuntimeHooks()), draft) + assert draft.skip_unchanged_block_offsets is False + + armed = SimpleNamespace(runtime_hooks=RuntimeHooks(skip_unchanged_draft_block_offsets=True)) + arm(armed, None) + arm(armed, SimpleNamespace(skip_unchanged_block_offsets=False)) + assert lines == [] + + arm(armed, draft) + arm(armed, draft) + assert draft.skip_unchanged_block_offsets is True + assert KVCacheManagerV2.skip_unchanged_block_offsets is False + assert lines == ["[M3 host] draft block-offset copy skipped when unchanged"] + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="runs the in-tree block-offset copy op") +@pytest.mark.parametrize("seed", range(4)) +def test_guarded_copies_leave_the_bytes_of_always_copying(seed): + from tensorrt_llm.bindings.internal.batch_manager.kv_cache_manager_v2_utils import ( + copy_batch_block_offsets_to_device, + ) + + rng = random.Random(seed) + manager = _manager(pin=True) + host = manager.host_kv_cache_block_offsets + scales = torch.tensor([2, 3], dtype=torch.int32, pin_memory=True) + offsets = torch.tensor([1, 5], dtype=torch.int32, pin_memory=True) + guarded = torch.full((POOLS, CAPACITY, 2, WIDTH), -1, dtype=torch.int32, device="cuda") + reference = guarded.clone() + stream = torch.cuda.current_stream().cuda_stream + rows = [1, 3] + skips = 0 + for step in range(200): + event = rng.random() + if event < 0.15: + host[rng.randrange(POOLS), rng.choice(rows), 0, rng.randrange(WIDTH)] = rng.randrange( + 1000 + ) + elif event < 0.25: + host[rng.randrange(POOLS), rng.randrange(CAPACITY), 1, rng.randrange(WIDTH)] = ( + rng.randrange(1000) + ) + elif event < 0.30: + rows = rng.sample(range(CAPACITY), rng.randint(1, 3)) + num_contexts = 1 if rng.random() < 0.05 else 0 + copy_idx = torch.tensor(rows, dtype=torch.int32, pin_memory=True) + copy_batch_block_offsets_to_device(host, reference, copy_idx, scales, offsets, stream) + if manager._block_offsets_unchanged(guarded, copy_idx, num_contexts): + skips += 1 + else: + copy_batch_block_offsets_to_device(host, guarded, copy_idx, scales, offsets, stream) + torch.cuda.synchronize() + assert torch.equal(guarded, reference), f"seed {seed} step {step}" + assert skips > 50 diff --git a/tests/unittest/_torch/executor/test_runtime_hooks.py b/tests/unittest/_torch/executor/test_runtime_hooks.py new file mode 100644 index 000000000000..2e392661d69f --- /dev/null +++ b/tests/unittest/_torch/executor/test_runtime_hooks.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""How the engine reads the step-loop hooks a model declares (``runtime_hooks_of``).""" + +import dataclasses + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor.runtime_hooks import ( + DEFAULT_RUNTIME_HOOKS, + MsaRuntimePolicy, + RuntimeHooks, + runtime_hooks_of, +) + +pytestmark = pytest.mark.cpu_only + + +class _PlainModel(torch.nn.Module): + """A model that declares no hooks, like every model outside the opt-in.""" + + +class _ClassHooksModel(torch.nn.Module): + runtime_hooks = RuntimeHooks(pack_step_inputs=True, skip_unchanged_h2d=True) + + +def test_model_without_hooks_gets_the_all_off_default(): + hooks = runtime_hooks_of(_PlainModel()) + + assert hooks is DEFAULT_RUNTIME_HOOKS + assert hooks == RuntimeHooks() + assert not hooks.pack_step_inputs + assert not hooks.skip_unchanged_h2d + assert not hooks.loop_on_execution_stream + assert not hooks.merge_tuned_tactics_across_ranks + assert not hooks.skip_greedy_rng_upload + assert not hooks.pack_spec_rng_uploads + assert hooks.spec_step_fold is None + assert not hooks.sampler_d2h_side_stream + assert not hooks.spec_small_h2d + assert not hooks.skip_accepted_draft_upload + assert not hooks.spec_batch_map_changed_only + assert not hooks.kv_lens_update_once + assert not hooks.pack_parallel_scatter + assert not hooks.pack_in_graph + assert not hooks.pack_early_copy + assert not hooks.h2d_changed_span + assert hooks.msa == MsaRuntimePolicy() + assert not hooks.msa.short_sequence_identity + assert hooks.msa.kv_update is None + assert not hooks.msa.skip_unchanged_block_offsets + assert not hooks.msa.reuse_pool_views + + +def test_hooks_declared_as_none_mean_the_default(): + model = _PlainModel() + model.runtime_hooks = None + + assert runtime_hooks_of(model) is DEFAULT_RUNTIME_HOOKS + + +def test_class_attribute_hooks_are_returned_as_declared(): + model = _ClassHooksModel() + + assert runtime_hooks_of(model) is _ClassHooksModel.runtime_hooks + + +def test_instance_attribute_hooks_are_returned_as_declared(): + hooks = RuntimeHooks( + loop_on_execution_stream=True, + msa=MsaRuntimePolicy(short_sequence_identity=True, reuse_pool_views=True), + ) + model = _PlainModel() + model.runtime_hooks = hooks + + assert runtime_hooks_of(model) is hooks + + +def test_instance_attribute_overrides_class_attribute(): + hooks = RuntimeHooks(skip_greedy_rng_upload=True) + model = _ClassHooksModel() + model.runtime_hooks = hooks + + assert runtime_hooks_of(model) is hooks + + +@pytest.mark.parametrize( + "declared", + [True, {"pack_step_inputs": True}, MsaRuntimePolicy(), "RuntimeHooks()"], + ids=["bool", "dict", "msa_policy", "str"], +) +def test_hooks_of_the_wrong_type_raise(declared): + model = _PlainModel() + model.runtime_hooks = declared + + with pytest.raises(TypeError, match=r"_PlainModel\.runtime_hooks must be a RuntimeHooks"): + runtime_hooks_of(model) + + +def test_hooks_are_immutable_once_declared(): + hooks = RuntimeHooks() + + with pytest.raises(dataclasses.FrozenInstanceError): + hooks.pack_step_inputs = True + with pytest.raises(dataclasses.FrozenInstanceError): + hooks.msa.reuse_pool_views = True + + +@pytest.mark.parametrize( + "hooks, skips", + [ + (RuntimeHooks(), False), + (RuntimeHooks(pack_step_inputs=True, skip_greedy_rng_upload=True), False), + (RuntimeHooks(skip_unchanged_h2d=True), True), + (RuntimeHooks(msa=MsaRuntimePolicy(skip_unchanged_block_offsets=True)), True), + ], + ids=["default", "no_skip_of_metadata", "h2d_skip", "msa_block_offset_skip"], +) +def test_every_metadata_upload_skip_asks_for_private_metadata_buffers(hooks, skips): + assert hooks.skips_unchanged_metadata_uploads is skips + + +HOSTTAIL_FIELDS = ( + "sampler_d2h_side_stream", + "spec_small_h2d", + "pack_spec_rng_uploads", + "skip_accepted_draft_upload", + "spec_batch_map_changed_only", + "kv_lens_update_once", +) + + +@pytest.mark.parametrize("field", HOSTTAIL_FIELDS) +def test_each_host_tail_hook_defaults_off_and_is_a_plain_bool(field): + default = next(f for f in dataclasses.fields(RuntimeHooks) if f.name == field) + + assert default.default is False + assert getattr(RuntimeHooks(), field) is False + assert getattr(RuntimeHooks(**{field: True}), field) is True + + +def test_the_default_hooks_still_equal_a_fresh_instance(): + assert RuntimeHooks() == DEFAULT_RUNTIME_HOOKS + assert RuntimeHooks(**{field: False for field in HOSTTAIL_FIELDS}) == DEFAULT_RUNTIME_HOOKS + + +@pytest.mark.parametrize("field", HOSTTAIL_FIELDS) +def test_no_host_tail_hook_asks_for_private_metadata_buffers(field): + """skips_unchanged_metadata_uploads keys only on the metadata-upload skips; the host-tail hooks leave it alone.""" + assert RuntimeHooks(**{field: True}).skips_unchanged_metadata_uploads is False + assert ( + RuntimeHooks(skip_unchanged_h2d=True, **{field: True}).skips_unchanged_metadata_uploads + is True + ) + assert ( + RuntimeHooks(**{name: True for name in HOSTTAIL_FIELDS}).skips_unchanged_metadata_uploads + is False + ) diff --git a/tests/unittest/_torch/executor/test_step_inputs.py b/tests/unittest/_torch/executor/test_step_inputs.py new file mode 100644 index 000000000000..ceb8e36847e2 --- /dev/null +++ b/tests/unittest/_torch/executor/test_step_inputs.py @@ -0,0 +1,1202 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Packing a decode step's small input copies into one upload and one scatter launch.""" + +import threading + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor import h2d_cache, step_inputs +from tensorrt_llm._torch.pyexecutor.h2d_cache import H2DSkipCache, copy_h2d_if_changed +from tensorrt_llm._torch.pyexecutor.step_inputs import StepInputPack + + +def _scatter_op_registered() -> bool: + return torch.cuda.is_available() and hasattr(torch.ops.trtllm, "step_inputs_scatter") + + +pytestmark = [ + pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required"), + pytest.mark.skipif( + not _scatter_op_registered(), + reason="torch.ops.trtllm.step_inputs_scatter is not registered (built from " + "cpp/tensorrt_llm/thop/minimaxM3/stepInputsScatterOp.cpp)", + ), +] + +# About half a second of GPU time: long enough that work queued behind it is still pending when +# the host checks, short enough for a unit test. +_SLEEP_CYCLES = 1_000_000_000 + + +def _device(): + return torch.device("cuda", torch.cuda.current_device()) + + +@pytest.fixture +def pack(): + """A fresh pack, active for the test body.""" + step_pack = StepInputPack(_device()) + with step_inputs.active(step_pack): + yield step_pack + step_inputs.flush() + torch.cuda.synchronize() + + +def _host(values, dtype=torch.int32): + return torch.tensor(values, dtype=dtype).pin_memory() + + +def _zeros(*shape, dtype=torch.int32): + return torch.zeros(*shape, dtype=dtype, device="cuda") + + +# --------------------------------------------------------------------------- +# No active pack +# --------------------------------------------------------------------------- + + +def test_without_an_active_pack_every_call_is_a_plain_copy(): + dst = _zeros(4) + d2d_dst = _zeros(4) + src_dev = torch.arange(4, dtype=torch.int32, device="cuda") + + step_inputs.begin() + assert not step_inputs.is_open() + assert step_inputs.copy_(dst, _host([1, 2, 3, 4])) is dst + step_inputs.copy_(d2d_dst, src_dev) + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 4] + assert d2d_dst.tolist() == [0, 1, 2, 3] + step_inputs.flush() + step_inputs.discard() + + +def test_an_inactive_pack_is_not_used(): + idle = StepInputPack(_device()) + dst = _zeros(2) + + step_inputs.copy_(dst, _host([3, 4])) + torch.cuda.synchronize() + + assert dst.tolist() == [3, 4] + assert not idle.open and idle.staged == [] + + +# --------------------------------------------------------------------------- +# Deferral and ordering +# --------------------------------------------------------------------------- + + +def test_packed_host_copy_lands_only_at_flush(pack): + dst = _zeros(4) + src = _host([1, 2, 3, 4]) + + step_inputs.begin() + assert step_inputs.is_open() and pack.open + assert step_inputs.copy_(dst, src) is dst + # The pack snapshots the host bytes at the copy site. + src.fill_(7) + torch.cuda.synchronize() + assert dst.tolist() == [0, 0, 0, 0] + + step_inputs.flush() + assert not step_inputs.is_open() + assert pack.staged == [] + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3, 4] + + +def test_packed_host_copy_into_a_row_strided_2d_slice(pack): + table = torch.full((4, 6), -1, dtype=torch.int32, device="cuda") + dst = table[1:3, 2:5] + assert not dst.is_contiguous() and dst.stride(1) == 1 + expected = table.clone() + expected[1:3, 2:5] = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.int32) + + step_inputs.begin() + step_inputs.copy_(dst, _host([[1, 2, 3], [4, 5, 6]])) + torch.cuda.synchronize() + assert torch.equal(table, torch.full_like(table, -1)) + + step_inputs.flush() + torch.cuda.synchronize() + assert torch.equal(table, expected) + + +def test_packed_host_copy_converts_dtype_on_the_host(pack): + dst = _zeros(3) + + step_inputs.begin() + step_inputs.copy_(dst, _host([7, -8, 9], dtype=torch.int64)) + step_inputs.flush() + torch.cuda.synchronize() + + assert dst.tolist() == [7, -8, 9] + + +def test_packed_device_copy_lands_only_at_flush(pack): + dst = _zeros(5) + src = torch.arange(5, dtype=torch.int32, device="cuda") + + step_inputs.begin() + step_inputs.copy_(dst, src) + torch.cuda.synchronize() + assert dst.tolist() == [0] * 5 + + step_inputs.flush() + torch.cuda.synchronize() + assert dst.tolist() == [0, 1, 2, 3, 4] + + +def test_device_copy_reads_what_a_host_copy_of_the_same_pack_wrote(pack): + """The graph runner copies the engine's freshly uploaded buffers into its static inputs.""" + staged = _zeros(4) + static = _zeros(1, 8) + + step_inputs.begin() + step_inputs.copy_(staged, _host([10, 11, 12, 13])) + step_inputs.copy_(static[:, :4], staged.unsqueeze(0)) + step_inputs.flush() + torch.cuda.synchronize() + + assert static.tolist() == [[10, 11, 12, 13, 0, 0, 0, 0]] + + +# The scatter's CTA copies descriptor i with thread t on bytes t, t + 256, ... of it. When the +# second copy starts TAIL bytes before the end of the first (TAIL not a multiple of the warp +# size), its first bytes are the ones the first copy writes last, from other warps: without a +# barrier between descriptors, a warp that finished the first copy early writes (or reads) them +# before a slower warp has written its part of the first copy. +_TAILS = [239, 1000, 4073] + + +@pytest.mark.parametrize("tail", _TAILS) +@pytest.mark.parametrize("kind", ["host", "device"]) +def test_overlapping_destinations_land_in_descriptor_order(pack, kind, tail): + """The later copy wins on the overlap, as it would eagerly.""" + # Host copies share the 64 KiB staging buffer; device copies are bounded only by memory, and a + # long first copy lets the warps drift apart. + n = 24 * 1024 if kind == "host" else 1 << 20 + m = 8 * 1024 + first = torch.full((n,), 1, dtype=torch.uint8) + second = torch.full((m,), 2, dtype=torch.uint8) + if kind == "host": + first, second = first.pin_memory(), second.pin_memory() + else: + first, second = first.cuda(), second.cuda() + start = n - tail + expected = torch.ones(start + m, dtype=torch.uint8) + expected[start:] = 2 + + mismatches = [] + for trial in range(50): + buf = torch.zeros(start + m, dtype=torch.uint8, device="cuda") + step_inputs.begin() + step_inputs.copy_(buf[:n], first) + step_inputs.copy_(buf[start:], second) + step_inputs.flush() + torch.cuda.synchronize() + wrong = int((buf.cpu() != expected).sum()) + if wrong: + mismatches.append((trial, wrong)) + + assert not mismatches, f"(trial, overlap bytes where the earlier copy won): {mismatches}" + + +@pytest.mark.parametrize("tail", _TAILS) +def test_a_device_copy_reads_the_destination_of_an_earlier_device_copy(pack, tail): + """A chain of device copies: the second reads the bytes the first writes last.""" + n = 1 << 20 + source = torch.randint(1, 256, (n,), dtype=torch.uint8, device="cuda") + expected = source[n - tail :].clone() + + mismatches = [] + for trial in range(50): + middle = torch.zeros(n, dtype=torch.uint8, device="cuda") + out = torch.zeros(tail, dtype=torch.uint8, device="cuda") + step_inputs.begin() + step_inputs.copy_(middle, source) + step_inputs.copy_(out, middle[n - tail :]) + step_inputs.flush() + torch.cuda.synchronize() + wrong = int((out != expected).sum()) + if wrong: + mismatches.append((trial, wrong)) + + assert not mismatches, ( + f"(trial, stale bytes read before the first copy wrote them): {mismatches}" + ) + + +# --------------------------------------------------------------------------- +# Staged tensors stay alive +# --------------------------------------------------------------------------- + + +def test_a_temporary_device_source_outlives_its_copy(pack): + """The pack holds the source until the scatter is queued, so its memory is not reused.""" + numel = 1 << 16 + expected = torch.arange(numel, dtype=torch.int32) * 3 + 1 + dst = _zeros(numel) + + step_inputs.begin() + # The only reference to this source is the one the pack keeps. + step_inputs.copy_(dst, torch.arange(numel, dtype=torch.int32, device="cuda") * 3 + 1) + assert len(pack.staged) == 2 + # Churn the allocator with same-sized blocks: a freed source block would be handed out here. + garbage = [torch.full((numel,), -7, dtype=torch.int32, device="cuda") for _ in range(16)] + torch.cuda.synchronize() + step_inputs.flush() + del garbage + # Blocks allocated after the flush are ordered behind the scatter on this stream. + after = [torch.full((numel,), -9, dtype=torch.int32, device="cuda") for _ in range(16)] + torch.cuda.synchronize() + + assert torch.equal(dst.cpu(), expected) + del after + + +def test_a_temporary_destination_view_outlives_its_copy(pack): + owner = _zeros(8) + + step_inputs.begin() + step_inputs.copy_(owner.view(2, 4)[1], _host([5, 6, 7, 8])) + assert len(pack.staged) == 1 + step_inputs.flush() + torch.cuda.synchronize() + + assert owner.tolist() == [0, 0, 0, 0, 5, 6, 7, 8] + + +# --------------------------------------------------------------------------- +# Fallbacks +# --------------------------------------------------------------------------- + + +def test_an_unpackable_copy_flushes_first_and_keeps_order(pack): + buf = _zeros(8) + other = _zeros(4) + later = _zeros(2) + + step_inputs.begin() + step_inputs.copy_(buf, _host([1] * 8)) + step_inputs.copy_(other, _host([3, 3, 3, 3])) + # A strided 1-D destination cannot be packed: the pack lands first, then this copy. + step_inputs.copy_(buf[::2], _host([2, 2, 2, 2])) + assert step_inputs.is_open() + torch.cuda.synchronize() + assert buf.tolist() == [2, 1, 2, 1, 2, 1, 2, 1] + assert other.tolist() == [3, 3, 3, 3] + + # The pack reopened: later copies are deferred again. + step_inputs.copy_(later, _host([5, 6])) + torch.cuda.synchronize() + assert later.tolist() == [0, 0] + step_inputs.flush() + torch.cuda.synchronize() + assert later.tolist() == [5, 6] + + +def test_a_device_copy_with_a_dtype_change_is_not_packed(pack): + dst = _zeros(3, dtype=torch.int64) + src = torch.tensor([4, 5, 6], dtype=torch.int32, device="cuda") + + step_inputs.begin() + step_inputs.copy_(dst, src) + torch.cuda.synchronize() + + assert dst.tolist() == [4, 5, 6] + assert step_inputs.is_open() + + +def test_a_copy_larger_than_the_staging_buffer_falls_back_in_order(pack): + numel = StepInputPack.CAPACITY // 4 + big = _zeros(numel) + big_values = torch.arange(numel, dtype=torch.int32).pin_memory() + + step_inputs.begin() + step_inputs.copy_(big[:4], _host([-1, -1, -1, -1])) + step_inputs.copy_(big, big_values) + torch.cuda.synchronize() + + # The staged prefix landed before the full copy, which therefore wins. + assert torch.equal(big.cpu(), big_values) + assert step_inputs.is_open() + + +def test_copies_that_together_overflow_the_staging_buffer_all_land(pack): + # Each copy is a quarter of the staging buffer, so the fourth one does not fit. + chunk = StepInputPack.CAPACITY // 16 + dsts = [_zeros(chunk) for _ in range(6)] + values = [torch.full((chunk,), i + 1, dtype=torch.int32).pin_memory() for i in range(6)] + + step_inputs.begin() + for dst, value in zip(dsts, values): + step_inputs.copy_(dst, value) + step_inputs.flush() + torch.cuda.synchronize() + + for i, dst in enumerate(dsts): + assert torch.equal(dst.cpu(), values[i]), f"copy {i} did not land" + + +def test_more_copies_than_descriptors_all_land(pack): + count = StepInputPack.MAX_DESC + 40 + dst = _zeros(count) + + step_inputs.begin() + for i in range(count): + step_inputs.copy_(dst[i : i + 1], _host([i + 1])) + step_inputs.flush() + torch.cuda.synchronize() + + assert dst.tolist() == list(range(1, count + 1)) + + +def test_a_row_strided_copy_that_needs_more_descriptors_than_remain_falls_back(pack): + rows = 8 + table = torch.full((rows, 4), -1, dtype=torch.int32, device="cuda") + filler = _zeros(StepInputPack.MAX_DESC - 2) + rows_values = torch.arange(rows * 3, dtype=torch.int32).reshape(rows, 3) + + step_inputs.begin() + for i in range(filler.numel()): + step_inputs.copy_(filler[i : i + 1], _host([i + 1])) + # Two descriptors remain; the slice needs one per row, so it flushes and copies directly. + step_inputs.copy_(table[:, :3], rows_values.pin_memory()) + step_inputs.flush() + torch.cuda.synchronize() + + assert filler.tolist() == list(range(1, filler.numel() + 1)) + assert torch.equal(table[:, :3].cpu(), rows_values) + assert table[:, 3].tolist() == [-1] * rows + + +def test_mixed_device_copies_past_the_descriptor_limit_all_land(pack): + count = StepInputPack.MAX_DESC + 10 + src = torch.arange(1, count + 1, dtype=torch.int32, device="cuda") + dst = _zeros(count) + + step_inputs.begin() + for i in range(count): + step_inputs.copy_(dst[i : i + 1], src[i : i + 1]) + step_inputs.flush() + torch.cuda.synchronize() + + assert torch.equal(dst, src) + + +def test_begin_on_an_open_pack_issues_it_first(pack): + dst = _zeros(2) + + step_inputs.begin() + step_inputs.copy_(dst, _host([8, 9])) + step_inputs.begin() + torch.cuda.synchronize() + + assert dst.tolist() == [8, 9] + + +def test_reusing_a_pinned_staging_buffer_waits_for_its_upload(pack): + """With every staging buffer in the ring used once, the next pack reuses the first one. + + Its upload is queued behind a long kernel, so rewriting the buffer without waiting would + upload the newer bytes in place of the older ones. + """ + ring = StepInputPack.RING + dsts = [_zeros(4) for _ in range(ring + 1)] + values = [[i * 10 + j for j in range(4)] for i in range(ring + 1)] + + step_inputs.begin() + torch.cuda._sleep(_SLEEP_CYCLES) + step_inputs.copy_(dsts[0], _host(values[0])) + step_inputs.flush() + first_upload = pack.events[0] + for i in range(1, ring): + step_inputs.begin() + step_inputs.copy_(dsts[i], _host(values[i])) + step_inputs.flush() + assert not first_upload.query(), "the queued kernel finished too early for this check" + + step_inputs.begin() + assert first_upload.query() + step_inputs.copy_(dsts[ring], _host(values[ring])) + step_inputs.flush() + torch.cuda.synchronize() + + for i, dst in enumerate(dsts): + assert dst.tolist() == values[i], f"pack {i} uploaded the wrong bytes" + + +def test_a_copy_while_capturing_is_not_deferred(pack): + dst = _zeros(3) + src = torch.tensor([1, 2, 3], dtype=torch.int32, device="cuda") + + step_inputs.begin() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + step_inputs.copy_(dst, src) + step_inputs.flush() + torch.cuda.synchronize() + assert dst.tolist() == [0, 0, 0] + + graph.replay() + torch.cuda.synchronize() + assert dst.tolist() == [1, 2, 3] + + +# --------------------------------------------------------------------------- +# Discard +# --------------------------------------------------------------------------- + + +def test_discard_drops_the_staged_copies(pack): + dst = _zeros(3) + + step_inputs.begin() + step_inputs.copy_(dst, _host([1, 2, 3])) + step_inputs.copy_(dst, torch.ones(3, dtype=torch.int32, device="cuda")) + step_inputs.discard() + assert not pack.open and pack.staged == [] and pack.h2d == [] and pack.d2d == [] + step_inputs.flush() + torch.cuda.synchronize() + + assert dst.tolist() == [0, 0, 0] + + +def test_after_a_discarded_step_identical_values_are_sent_again(pack): + """The runner's recovery: discard the pack and clear the skip cache.""" + skip_cache = H2DSkipCache() + dst = _zeros(3) + values = _host([4, 5, 6]) + + with h2d_cache.active(skip_cache): + step_inputs.begin() + assert copy_h2d_if_changed(dst, values, pack=True) is True + step_inputs.discard() + torch.cuda.synchronize() + assert dst.tolist() == [0, 0, 0] + # Without clearing, the cache would vouch for values that never landed. + assert copy_h2d_if_changed(dst, values, pack=True) is False + skip_cache.clear() + + step_inputs.begin() + assert copy_h2d_if_changed(dst, values, pack=True) is True + step_inputs.flush() + torch.cuda.synchronize() + + assert dst.tolist() == [4, 5, 6] + + +def test_skip_cache_copies_join_the_active_pack(pack): + skip_cache = H2DSkipCache() + dst = _zeros(3) + values = _host([4, 5, 6]) + + with h2d_cache.active(skip_cache): + step_inputs.begin() + assert copy_h2d_if_changed(dst, values, pack=True) is True + torch.cuda.synchronize() + assert dst.tolist() == [0, 0, 0] + # Already staged with these values: the second copy is skipped. + assert copy_h2d_if_changed(dst, values, pack=True) is False + step_inputs.flush() + torch.cuda.synchronize() + + assert dst.tolist() == [4, 5, 6] + + +# --------------------------------------------------------------------------- +# Isolation between packs +# --------------------------------------------------------------------------- + + +def test_a_nested_pack_does_not_see_the_outer_pack(pack): + outer_dst = _zeros(2) + inner_dst = _zeros(2) + other = StepInputPack(_device()) + + step_inputs.begin() + step_inputs.copy_(outer_dst, _host([1, 2])) + with step_inputs.active(other): + assert not step_inputs.is_open() + step_inputs.copy_(inner_dst, _host([3, 4])) # plain copy: the inner pack is closed + step_inputs.begin() + step_inputs.copy_(inner_dst, _host([5, 6])) + torch.cuda.synchronize() + assert inner_dst.tolist() == [3, 4] + assert outer_dst.tolist() == [0, 0] + step_inputs.flush() + assert step_inputs.is_open() and len(pack.h2d) == 1 + step_inputs.flush() + torch.cuda.synchronize() + + assert inner_dst.tolist() == [5, 6] + assert outer_dst.tolist() == [1, 2] + + +def test_active_none_disables_packing_inside_an_outer_pack(pack): + dst = _zeros(2) + + step_inputs.begin() + with step_inputs.active(None): + assert not step_inputs.is_open() + step_inputs.copy_(dst, _host([7, 8])) + torch.cuda.synchronize() + assert dst.tolist() == [7, 8] + assert pack.h2d == [] + + +def test_an_open_pack_on_one_thread_is_invisible_to_another(pack): + mine = _zeros(2) + theirs = _zeros(2) + seen = {} + + step_inputs.begin() + step_inputs.copy_(mine, _host([1, 1])) + + def other_thread(): + seen["open"] = step_inputs.is_open() + step_inputs.copy_(theirs, _host([2, 2])) + step_inputs.flush() + torch.cuda.synchronize() + seen["theirs"] = theirs.tolist() + seen["mine"] = mine.tolist() + + thread = threading.Thread(target=other_thread) + thread.start() + thread.join() + + assert seen == {"open": False, "theirs": [2, 2], "mine": [0, 0]} + assert step_inputs.is_open() and len(pack.h2d) == 1 + step_inputs.flush() + torch.cuda.synchronize() + assert mine.tolist() == [1, 1] + + +def test_threads_with_their_own_packs_keep_them_apart(): + packs = [StepInputPack(_device()) for _ in range(2)] + dsts = [_zeros(4) for _ in range(2)] + barrier = threading.Barrier(2) + results = {} + + def worker(index): + with step_inputs.active(packs[index]): + step_inputs.begin() + step_inputs.copy_(dsts[index], _host([index + 1] * 4)) + barrier.wait() + # Both packs are open with one staged copy each. + results[index] = (packs[index].open, len(packs[index].h2d)) + barrier.wait() + step_inputs.flush() + torch.cuda.synchronize() + + threads = [threading.Thread(target=worker, args=(i,)) for i in range(2)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + + assert results == {0: (True, 1), 1: (True, 1)} + assert dsts[0].tolist() == [1] * 4 + assert dsts[1].tolist() == [2] * 4 + + +def _random_pack_run(parallel: bool, seed: int, overlap: bool): + """Stage a fixed random mix of host copies (2-D page-table rows included) and device copies that read a host + copy's destination; return every destination after the flush.""" + g = torch.Generator().manual_seed(seed) + pack = StepInputPack(_device(), parallel_scatter=parallel) + big = _zeros(4096) + table = torch.zeros((8, 64), dtype=torch.int32, device=_device()) + out = _zeros(256) + with step_inputs.active(pack): + step_inputs.begin() + for k in range(24): + n = int(torch.randint(1, 64, (1,), generator=g)) + step_inputs.copy_( + big[k * 128 : k * 128 + n], + torch.randint(0, 1 << 30, (n,), generator=g, dtype=torch.int32), + ) + step_inputs.copy_( + table[:3, :40], torch.randint(0, 1 << 30, (3, 40), generator=g, dtype=torch.int32) + ) + if overlap: + # A later host copy overwrites part of an earlier one: order matters, so the serial scatter must run. + step_inputs.copy_(big[0:4], torch.full((4,), -7, dtype=torch.int32)) + step_inputs.copy_(out[:64], big[:64]) + step_inputs.copy_(out[64:96], big[128:160]) + step_inputs.flush() + torch.cuda.synchronize() + return big.cpu(), table.cpu(), out.cpu() + + +@pytest.mark.parametrize("overlap", [False, True], ids=["disjoint", "overlapping"]) +@pytest.mark.parametrize("seed", [0, 1, 2]) +def test_parallel_scatter_writes_what_the_serial_scatter_writes(seed, overlap): + serial = _random_pack_run(False, seed, overlap) + parallel = _random_pack_run(True, seed, overlap) + for a, b in zip(serial, parallel): + assert torch.equal(a, b) + + +def test_parallel_scatter_only_for_order_free_packs(): + assert step_inputs._parallel_ok([(0, 0, 8), (8, 0, 8)], []) + assert not step_inputs._parallel_ok([(0, 0, 8), (4, 0, 8)], []) + # A device copy may read a host copy's destination (the launches run in order) ... + assert step_inputs._parallel_ok([(0, 0, 8)], [(100, 0, 8)]) + # ... but not another device copy's destination, and device destinations must be disjoint. + assert not step_inputs._parallel_ok([], [(100, 0, 8), (200, 100, 8)]) + assert not step_inputs._parallel_ok([], [(100, 0, 8), (104, 300, 8)]) + + +def test_parallel_scatter_is_off_by_default(): + assert not StepInputPack(_device()).parallel_scatter + + +def _parallel_scatter_compiles() -> int: + """How many variants of the parallel scatter kernel Triton holds compiled (any device).""" + kernel = step_inputs._parallel_scatter_kernel + caches = getattr(kernel, "device_caches", None) + if caches is not None: + return sum(len(entry[0]) for entry in caches.values()) + return sum(len(entry) for entry in kernel.cache.values()) + + +def test_parallel_scatter_compiles_nothing_after_the_warm_up(): + # The warm-up compiles both DEVICE_SRC variants; packs with any number of host copies (which moves the device + # copies' offset `first`) must reuse them, so no serving-step flush compiles. + pack = StepInputPack(_device(), parallel_scatter=True) + compiled = _parallel_scatter_compiles() + dst = _zeros(4096) + out = _zeros(64) + with step_inputs.active(pack): + for n_host in (1, 2, 3, 5, 15, 16, 17, 24): + step_inputs.begin() + for k in range(n_host): + step_inputs.copy_(dst[k * 64 : k * 64 + 4], torch.full((4,), k, dtype=torch.int32)) + step_inputs.copy_(out[:4], dst[:4]) + step_inputs.flush() + torch.cuda.synchronize() + assert _parallel_scatter_compiles() == compiled + assert out[:4].tolist() == [0] * 4 + assert dst[23 * 64 : 23 * 64 + 4].tolist() == [23] * 4 + + +# --------------------------------------------------------------------------- +# Flags off: the GB path, byte for byte +# --------------------------------------------------------------------------- + + +def _gb_flush(pack): + """StepInputPack.flush as of the GB base (1af1e3272c), verbatim: the reference the flags-off path must match.""" + if not pack.open: + return + pack.open = False + if not pack.h2d and not pack.d2d: + return + host = pack.host[pack.parity] + desc = [len(pack.h2d), len(pack.d2d)] + [ + value for copy in pack.h2d + pack.d2d for value in copy + ] + host[: 8 * len(desc)].view(torch.int64).copy_(torch.tensor(desc, dtype=torch.int64)) + pack.dev[: pack.used].copy_(host[: pack.used], non_blocking=True) + if pack.parallel_scatter and step_inputs._parallel_ok(pack.h2d, pack.d2d): + stage_i64 = pack.dev.view(torch.int64) + if pack.h2d: + step_inputs._parallel_scatter_kernel[(len(pack.h2d),)]( + pack.dev, stage_i64, 0, DEVICE_SRC=False, BLOCK=256, num_warps=2 + ) + if pack.d2d: + step_inputs._parallel_scatter_kernel[(len(pack.d2d),)]( + pack.dev, stage_i64, len(pack.h2d), DEVICE_SRC=True, BLOCK=256, num_warps=2 + ) + else: + torch.ops.trtllm.step_inputs_scatter(pack.dev) + event = pack.events[pack.parity] + if event is None: + event = pack.events[pack.parity] = torch.cuda.Event() + event.record() + pack.parity = (pack.parity + 1) % pack.RING + pack._clear() + + +def _flags_off_run(parallel: bool, seed: int, overlap: bool, how: str): + """``_random_pack_run`` issued by ``flush()``, ``flush(in_graph=True)`` or the GB reference; returns every + destination and the uploaded staging bytes.""" + g = torch.Generator().manual_seed(seed) + pack = StepInputPack(_device(), parallel_scatter=parallel) + big = _zeros(4096) + table = torch.zeros((8, 64), dtype=torch.int32, device=_device()) + out = _zeros(256) + with step_inputs.active(pack): + step_inputs.begin() + for k in range(24): + n = int(torch.randint(1, 64, (1,), generator=g)) + step_inputs.copy_( + big[k * 128 : k * 128 + n], + torch.randint(0, 1 << 30, (n,), generator=g, dtype=torch.int32), + ) + step_inputs.copy_( + table[:3, :40], torch.randint(0, 1 << 30, (3, 40), generator=g, dtype=torch.int32) + ) + if overlap: + step_inputs.copy_(big[0:4], torch.full((4,), -7, dtype=torch.int32)) + step_inputs.copy_(out[:64], big[:64]) + step_inputs.copy_(out[64:96], big[128:160]) + used = pack.used + if how == "flush": + step_inputs.flush() + elif how == "flush_in_graph": + step_inputs.flush(in_graph=True) + else: + _gb_flush(pack) + torch.cuda.synchronize() + assert not pack.open and pack.staged == [] + assert (pack.graph_windows, pack.eager_fallbacks, pack.early_uploads) == (0, 0, 0) + return big.cpu(), table.cpu(), out.cpu(), _uploaded(pack, used, (big, table, out)), pack.parity + + +def _uploaded(pack, used, tensors): + """The staging bytes the flush uploaded, with each device address written as (tensor index, byte offset) so two + runs compare: the header, every descriptor, and every host copy's staged data bytes.""" + staged = pack.dev[:used].cpu() + words = staged[: pack.data_start].view(torch.int64).tolist() + num_h2d, num_d2d = words[0], words[1] + spans = [(t.data_ptr(), t.numel() * t.element_size()) for t in tensors] + + def where(addr): + for i, (base, nbytes) in enumerate(spans): + if base <= addr < base + nbytes: + return (i, addr - base) + raise AssertionError(f"address {addr:#x} outside the test tensors") + + copies = [] + for k in range(num_h2d + num_d2d): + dst, src, nbytes = words[2 + 3 * k : 5 + 3 * k] + if k < num_h2d: + copies.append((where(dst), src, nbytes, staged[src : src + nbytes].tolist())) + else: + copies.append((where(dst), where(src), nbytes)) + return num_h2d, num_d2d, copies + + +@pytest.mark.parametrize("parallel", [False, True], ids=["serial", "parallel"]) +@pytest.mark.parametrize("overlap", [False, True], ids=["disjoint", "overlapping"]) +@pytest.mark.parametrize("seed", [0, 1]) +def test_flags_off_flush_is_the_gb_flush_byte_for_byte(parallel, overlap, seed): + """With pack_in_graph off, flush() and flush(in_graph=True) upload the same staging bytes, write the same + destinations and advance the ring like the GB base's flush.""" + reference = _flags_off_run(parallel, seed, overlap, "gb") + for how in ("flush", "flush_in_graph"): + got = _flags_off_run(parallel, seed, overlap, how) + for a, b in zip(reference[:3], got[:3]): + assert torch.equal(a, b), how + assert got[3] == reference[3], how + assert got[4] == reference[4], how + + +def test_flags_off_pack_allocates_no_window_and_has_no_prologue(): + for kwargs in ({}, {"parallel_scatter": True}): + pack = StepInputPack(_device(), **kwargs) + assert not pack.in_graph and not pack.early_copy + for name in ("ghost", "ghost_free", "gdev", "copy_stream", "staged_ready", "prologue_done"): + assert not hasattr(pack, name), name + with step_inputs.active(pack): + assert step_inputs.capture_prologue() is None + # A closed flags-off pack: flush(in_graph=True) is a no-op, like flush(). + pack.flush(in_graph=True) + pack.quiesce_graph_window() + assert step_inputs.capture_prologue() is None + + +def test_a_pack_built_without_init_keeps_every_in_graph_item_off(): + """The class-level defaults (object.__new__ executors in tests never run __init__).""" + pack = object.__new__(StepInputPack) + assert not pack.parallel_scatter and not pack.in_graph and not pack.early_copy + assert (pack.graph_windows, pack.eager_fallbacks, pack.early_uploads) == (0, 0, 0) + pack.quiesce_graph_window() + assert not pack.capture_prologue() + + +# --------------------------------------------------------------------------- +# In-graph prologue (pack_in_graph) and early copy (pack_early_copy) +# --------------------------------------------------------------------------- + + +def _graph_case(early=False): + """A pack with the in-graph prologue and a graph that starts with it and then reads the destinations.""" + pack = StepInputPack(_device(), parallel_scatter=True, in_graph=True, early_copy=early) + a, b, c = _zeros(16), _zeros(16), _zeros(16) + out = _zeros(48) + graph = torch.cuda.CUDAGraph() + with step_inputs.active(pack): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + assert step_inputs.capture_prologue() is pack + out.copy_(torch.cat([a, b, c])) + return pack, graph, (a, b, c), out + + +def _step(pack, graph, dsts, values, overlap=False, flush=True): + a, b, c = dsts + with step_inputs.active(pack): + step_inputs.begin() + step_inputs.copy_(a, _host(values[0])) + step_inputs.copy_(b, _host(values[1])) + if overlap: + step_inputs.copy_(a[:4], _host([-1, -2, -3, -4])) + step_inputs.copy_(c, a) # a device copy reading a host copy's destination + if flush: + step_inputs.flush(in_graph=True) + else: + step_inputs.discard() + step_inputs.flush(in_graph=True) + graph.replay() + + +def test_in_graph_pack_is_issued_by_the_replay(): + pack, graph, dsts, out = _graph_case() + for step in range(5): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + _step(pack, graph, dsts, (va, vb)) + torch.cuda.synchronize() + assert out.tolist() == va + vb + va, step + # Nothing was issued outside the graph: the window carried the copies. + assert int(pack.ghost[:8].view(torch.int64)[0]) == 2 + assert (pack.graph_windows, pack.eager_fallbacks, pack.early_uploads) == (5, 0, 0) + + +def test_in_graph_pack_not_order_free_is_issued_eagerly(): + pack, graph, dsts, out = _graph_case() + va = list(range(16)) + vb = list(range(100, 116)) + _step(pack, graph, dsts, (va, vb), overlap=True) + torch.cuda.synchronize() + expect_a = [-1, -2, -3, -4] + va[4:] + assert out.tolist() == expect_a + vb + expect_a + # The window was zeroed, so the prologue wrote nothing over the eager result. + assert not pack.ghost[:16].any() + assert (pack.graph_windows, pack.eager_fallbacks) == (0, 1) + + +@pytest.mark.parametrize("early", [False, True], ids=["prologue_copy", "early_copy"]) +def test_in_graph_pack_over_the_window_is_issued_eagerly(early): + pack, graph, dsts, out = _graph_case(early) + big = _zeros(StepInputPack.GRAPH_WINDOW // 4) + va = list(range(16)) + vb = list(range(200, 216)) + with step_inputs.active(pack): + step_inputs.begin() + step_inputs.copy_(dsts[0], _host(va)) + step_inputs.copy_(dsts[1], _host(vb)) + step_inputs.copy_(big, torch.arange(big.numel(), dtype=torch.int32)) + step_inputs.copy_(dsts[2], dsts[0]) + assert pack.used > StepInputPack.GRAPH_WINDOW + step_inputs.flush(in_graph=True) + graph.replay() + torch.cuda.synchronize() + assert out.tolist() == va + vb + va + assert big.cpu().tolist() == list(range(big.numel())) + assert (pack.graph_windows, pack.eager_fallbacks, pack.early_uploads) == (0, 1, 0) + + +def test_in_graph_replay_without_a_pack_reapplies_nothing(): + pack, graph, dsts, out = _graph_case() + _step(pack, graph, dsts, (list(range(16)), list(range(16)))) + torch.cuda.synchronize() + dsts[0].fill_(7) + _step(pack, graph, dsts, ([0] * 16, [0] * 16), flush=False) + torch.cuda.synchronize() + assert dsts[0].tolist() == [7] * 16 + + +@pytest.mark.parametrize("early", [False, True], ids=["prologue_copy", "early_copy"]) +def test_in_graph_pack_lands_every_step(early): + pack, graph, dsts, out = _graph_case(early) + assert pack.early_copy is early + for step in range(6): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + _step(pack, graph, dsts, (va, vb), overlap=step % 2 == 1) + torch.cuda.synchronize() + head = [-1, -2, -3, -4] if step % 2 == 1 else [] + expect_a = head + va[len(head) :] + assert out.tolist() == expect_a + vb + expect_a, step + assert (pack.graph_windows, pack.eager_fallbacks) == (3, 3) + assert pack.early_uploads == (3 if early else 0) + + +def _history_run(pack, graph, dsts, out, steps, before_replay=None, before_upload=None): + """Steps back to back without host syncs; each replay's output is copied to a history row behind it (same + stream), so a later upload that overwrote the window too early shows up as the wrong step's values.""" + hist = _zeros(steps, out.numel()) + expect = [] + for step in range(steps): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + expect.append(va + vb + va) + a, b, c = dsts + with step_inputs.active(pack): + step_inputs.begin() + step_inputs.copy_(a, _host(va)) + step_inputs.copy_(b, _host(vb)) + step_inputs.copy_(c, a) + if before_upload is not None: + before_upload(step) + step_inputs.flush(in_graph=True) + if before_replay is not None: + before_replay(step) + graph.replay() + hist[step].copy_(out) + torch.cuda.synchronize() + return hist.tolist(), expect + + +def test_early_copy_waits_for_the_previous_prologue_before_overwriting_the_window(): + """A slow GPU: the replay of step N (and its prologue) is held back behind a long kernel while the host already + uploads step N+1; that upload must wait for step N's prologue to consume the window.""" + pack, graph, dsts, out = _graph_case(early=True) + hist, expect = _history_run( + pack, graph, dsts, out, 4, before_replay=lambda step: torch.cuda._sleep(_SLEEP_CYCLES // 4) + ) + assert hist == expect + + +def test_early_copy_prologue_waits_for_a_slow_upload(): + """A slow copy stream: each step's upload is held back behind a long kernel on the copy stream; the prologue must + wait for it (staged_ready), and the next host fill must wait for the upload to have read the window.""" + pack, graph, dsts, out = _graph_case(early=True) + + def slow_copy(step): + with torch.cuda.stream(pack.copy_stream): + torch.cuda._sleep(_SLEEP_CYCLES // 4) + + hist, expect = _history_run(pack, graph, dsts, out, 4, before_upload=slow_copy) + assert hist == expect + + +def test_early_copy_with_a_slow_host(): + """A slow host: the GPU idles between steps (the window lands long before the replay).""" + import time + + pack, graph, dsts, out = _graph_case(early=True) + hist, expect = _history_run( + pack, graph, dsts, out, 3, before_replay=lambda step: time.sleep(0.05) + ) + assert hist == expect + + +def test_early_copy_snapshots_the_host_bytes_before_a_stalled_upload(): + """PR-CTX (d)(1): stall the copy stream, issue the early copy, overwrite the host source, then sync: the device + holds the pre-overwrite values. The next step's host fill also runs while the first upload is still stalled (it + must wait for that upload to have read the pinned window before it overwrites it).""" + pack, graph, dsts, out = _graph_case(early=True) + a, b, c = dsts + hist = _zeros(2, out.numel()) + va, vb = list(range(16)), list(range(500, 516)) + src_a, src_b = _host(va), _host(vb) + with torch.cuda.stream(pack.copy_stream): + torch.cuda._sleep(1 << 30) + with step_inputs.active(pack): + step_inputs.begin() + step_inputs.copy_(a, src_a) + step_inputs.copy_(b, src_b) + step_inputs.copy_(c, a) + step_inputs.flush(in_graph=True) + # The upload is queued behind the stall: it has not read the window yet. + assert not pack.staged_ready.query() + src_a.fill_(-9) + src_b.fill_(-9) + graph.replay() + hist[0].copy_(out) + # Step 2's fill blocks on ghost_free (the stalled upload) before it writes the pinned window. + va2, vb2 = [7] * 16, [8] * 16 + with step_inputs.active(pack): + step_inputs.begin() + step_inputs.copy_(a, _host(va2)) + step_inputs.copy_(b, _host(vb2)) + step_inputs.copy_(c, a) + step_inputs.flush(in_graph=True) + graph.replay() + hist[1].copy_(out) + torch.cuda.synchronize() + assert hist[0].tolist() == va + vb + va + assert hist[1].tolist() == va2 + vb2 + va2 + assert (pack.graph_windows, pack.early_uploads) == (2, 2) + + +@pytest.mark.parametrize("early", [False, True], ids=["prologue_copy", "early_copy"]) +def test_in_graph_replay_after_quiesce_reapplies_nothing(early): + pack, graph, dsts, out = _graph_case(early) + _step(pack, graph, dsts, (list(range(16)), list(range(16)))) + torch.cuda.synchronize() + dsts[0].fill_(7) + pack.quiesce_graph_window() + graph.replay() + torch.cuda.synchronize() + assert dsts[0].tolist() == [7] * 16 + + +def test_early_copy_needs_in_graph(): + assert not StepInputPack(_device(), parallel_scatter=True, early_copy=True).early_copy + + +def test_in_graph_needs_parallel_scatter_and_is_off_by_default(): + assert not StepInputPack(_device()).in_graph + assert not StepInputPack(_device(), in_graph=True).in_graph + + +def test_in_graph_replay_outside_the_pack_context_reapplies_nothing(): + pack, graph, dsts, out = _graph_case() + _step(pack, graph, dsts, (list(range(16)), list(range(16)))) + torch.cuda.synchronize() + dsts[0].fill_(7) + # What the graph runner does for a replay outside the pack's context. + assert not step_inputs.is_active(pack) + pack.quiesce_graph_window() + graph.replay() + torch.cuda.synchronize() + assert dsts[0].tolist() == [7] * 16 + + +def _kernel_compiles(kernel) -> int: + """How many variants of a Triton kernel are compiled (any device).""" + caches = getattr(kernel, "device_caches", None) + if caches is not None: + return sum(len(entry[0]) for entry in caches.values()) + return sum(len(entry) for entry in kernel.cache.values()) + + +@pytest.mark.parametrize("early", [False, True], ids=["prologue_copy", "early_copy"]) +def test_in_graph_kernels_compile_nothing_after_the_warm_up(early): + """The pack's warm-up compiles both scatter kernels' variants; captures and steps with any number of copies (in + the window, over it, not order-free, quiesced) compile nothing more.""" + pack = StepInputPack(_device(), parallel_scatter=True, in_graph=True, early_copy=early) + kernels = (step_inputs._parallel_scatter_kernel, step_inputs._graph_scatter_kernel) + compiled = [_kernel_compiles(kernel) for kernel in kernels] + dst = _zeros(8192) + out = _zeros(64) + graph = torch.cuda.CUDAGraph() + with step_inputs.active(pack): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + assert step_inputs.capture_prologue() is pack + out.add_(1) + for n_host in (1, 2, 3, 5, 15, 16, 17, 24, 120): + with step_inputs.active(pack): + step_inputs.begin() + for k in range(n_host): + step_inputs.copy_(dst[k * 64 : k * 64 + 4], torch.full((4,), k, dtype=torch.int32)) + for k in range(n_host % 7): + step_inputs.copy_(out[k * 4 : k * 4 + 4], dst[k * 64 : k * 64 + 4]) + if n_host == 17: + step_inputs.copy_( + dst[4096:], torch.ones(4096, dtype=torch.int32) + ) # over the window + step_inputs.flush(in_graph=True) + graph.replay() + if n_host == 5: + pack.quiesce_graph_window() + graph.replay() + torch.cuda.synchronize() + assert [_kernel_compiles(kernel) for kernel in kernels] == compiled + assert dst[119 * 64 : 119 * 64 + 4].tolist() == [119] * 4 + assert pack.eager_fallbacks == 1 and pack.graph_windows == 8 + + +# --------------------------------------------------------------------------- +# pack_in_graph through the real CUDAGraphRunner capture -> replay +# --------------------------------------------------------------------------- + + +def _runner_case(early, in_graph=True): + from _torch.helpers import create_mock_cuda_graph_runner + + from tensorrt_llm._torch.pyexecutor.cuda_graph_runner import KeyType + + batch_size = 4 + runner = create_mock_cuda_graph_runner(batch_size, max_num_tokens=128) + key = KeyType(batch_size=batch_size, draft_len=0) + num_tokens = runner._get_num_tokens_for_key(key) + pack = StepInputPack(_device(), parallel_scatter=True, in_graph=in_graph, early_copy=early) + meta = _zeros(8) + out = torch.zeros(2 * num_tokens + meta.numel(), dtype=torch.int64, device="cuda") + attn_metadata = object() + + def forward_fn(inputs): + out.copy_( + torch.cat( + [ + inputs["input_ids"].to(torch.int64), + inputs["position_ids"].flatten().to(torch.int64), + meta.to(torch.int64), + ] + ) + ) + return out + + def inputs(value): + return { + "attn_metadata": attn_metadata, + "input_ids": torch.full((num_tokens,), value, dtype=torch.int32, device="cuda"), + "position_ids": torch.arange(num_tokens, dtype=torch.int32, device="cuda").view(1, -1) + + value, + } + + with step_inputs.active(pack): + runner.capture(key, forward_fn, inputs(0)) + return runner, key, pack, meta, out, inputs, num_tokens + + +def _runner_step(runner, key, pack, meta, inputs, value): + with step_inputs.active(pack): + step_inputs.begin() + # A metadata upload the engine stages before the replay (as the MSA metadata does). + step_inputs.copy_(meta, _host([value * 10 + i for i in range(meta.numel())])) + runner.replay(key, inputs(value)) + + +@pytest.mark.parametrize("early", [False, True], ids=["prologue_copy", "early_copy"]) +def test_in_graph_pack_through_the_graph_runner_capture_and_replay(early): + """capture() adds the active pack's prologue to the graph; replay() with fresh inputs only fills the window (the + static input copies and the metadata upload are issued by the replay's prologue), every step lands.""" + runner, key, pack, meta, out, inputs, n = _runner_case(early) + assert runner._step_input_prologue_keys == {key: pack} + for value in range(1, 6): + _runner_step(runner, key, pack, meta, inputs, value) + torch.cuda.synchronize() + expect = [value] * n + [value + i for i in range(n)] + [value * 10 + i for i in range(8)] + assert out.tolist() == expect, value + assert pack.graph_windows == value and pack.eager_fallbacks == 0 + assert pack.early_uploads == (5 if early else 0) + # Replayed outside the pack's context: the inputs are plain copies and the prologue re-issues nothing. + meta.fill_(-3) + runner.replay(key, inputs(9)) + torch.cuda.synchronize() + assert out.tolist() == [9] * n + [9 + i for i in range(n)] + [-3] * 8 + runner.clear() + assert runner._step_input_prologue_keys is None + + +def test_graph_runner_flags_off_replays_through_the_stock_flush(monkeypatch): + """A pack without in_graph: the capture adds no prologue and replay() issues the pack with flush() exactly as on + the GB base (no in_graph argument), before the graph replay.""" + runner, key, pack, meta, out, inputs, n = _runner_case(False, in_graph=False) + assert runner._step_input_prologue_keys is None + calls = [] + real_flush = step_inputs.flush + + def recording_flush(*args, **kwargs): + calls.append((args, kwargs)) + return real_flush(*args, **kwargs) + + monkeypatch.setattr(step_inputs, "flush", recording_flush) + _runner_step(runner, key, pack, meta, inputs, 3) + torch.cuda.synchronize() + assert calls == [((), {})] + assert out.tolist() == [3] * n + [3 + i for i in range(n)] + [30 + i for i in range(8)] + assert (pack.graph_windows, pack.eager_fallbacks) == (0, 0) diff --git a/tests/unittest/_torch/modeling_v2/comm/_allgather_cast_op_matrix.py b/tests/unittest/_torch/modeling_v2/comm/_allgather_cast_op_matrix.py new file mode 100644 index 000000000000..e39734090c28 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/comm/_allgather_cast_op_matrix.py @@ -0,0 +1,266 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Certify `comm/allgather_cast`: every rank's fp32 rows equal `reference` (the all-gather, the vocab concat and +`.float()`) bit for bit, on every cell certified at this run's world size. + +Every rank builds every rank's shard from one seed, so each rank checks its own output against the reference over +all shards. Per cell: random logits with adversarial bits planted (-0.0, +-inf, NaN payloads, denormals, the +0x80000000 pair word), several seeds, inside `validating`. Also: a captured call replayed with fresh contents copied +into its static input before every replay (the epoch and the buffer rotation advance on the device); call-unique +contents while the grid alternates between 16 CTAs and the picked grid; `supports` refuses more rows than the +workspace holds, fp32 shards and partial 16-byte vectors. + +World 4 (one GB300 tray) is spawned from one process: + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _allgather_cast_op_matrix.py + +World 8 spans two trays and runs as one srun task per rank, like `_mnnvl_allreduce_op_matrix.py`: + + MASTER_ADDR= MASTER_PORT= python _allgather_cast_op_matrix.py --srun + +under ``srun --ntasks=8 --ntasks-per-node=4 --kill-on-bad-exit=1``. +""" + +import importlib.util +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Optional + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "allgather_cast requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 +SEEDS = 4 +GPUS_PER_NODE = 4 +"""A GB300 tray. The mapping needs it to see a world-8 group span two nodes.""" +# bf16 bit patterns planted into every shard: -0.0, +inf, -inf, NaN payloads, denormals. +SPECIAL_BITS = (-32768, 0x7F80, -128, 0x7FC1, -63, 0x0001, -32767, 0x7FFF) + +RANK = 0 +WORLD = 1 +# Bound inside the rank body: `entry` is the module, `op` its `allgather_cast` instance (the call surface, +# `reference`, `is_valid` and `CELLS`), `WS` the workspace arguments of every call, per shard width. +entry: Any = None +op: Any = None +validating: Any = None +WS: dict = {} + + +def _shards(seed: int, rows: int, cols: int): + """Every rank's bf16 [rows, cols] shard (the same on every rank), adversarial bits planted.""" + g = torch.Generator(device="cuda").manual_seed(seed) + out = [] + for _ in range(WORLD): + x = (torch.randn(rows, cols, device="cuda", generator=g) * 3).bfloat16() + w = x.view(torch.int16) + # Distinct columns: scatter_ with a repeated index keeps an arbitrary writer, which can differ between + # devices, and every rank must build the same shards. + cols_hit = torch.randperm(cols, generator=g, device="cuda")[:64] + special = torch.tensor(SPECIAL_BITS, dtype=torch.int16, device="cuda").repeat(8) + w[:, cols_hit] = special + w[:, 0] = 0 + w[:, 1] = -32768 # the int32 word 0x80000000 + out.append(x) + return out + + +def _call(shard: torch.Tensor, grid: int = 0) -> torch.Tensor: + return op(shard, **WS[shard.shape[1]], grid=grid) + + +def _bits_equal(a: torch.Tensor, b: torch.Tensor) -> bool: + return a.shape == b.shape and torch.equal(a.view(torch.int32), b.view(torch.int32)) + + +def check_certified_cells() -> None: + cells = [c for c in op.CELLS if c.spec["world"] == WORLD] + assert cells, f"no cell is certified at world size {WORLD}" + for cell in cells: + rows, cols = cell.spec["rows"], cell.spec["cols"] + for seed in range(SEEDS): + shards = _shards(seed, rows, cols) + with validating(op): + got = _call(shards[RANK]) + assert _bits_equal(got, op.reference(shards)), ( + f"{cell.why}: seed {seed} differs on rank {RANK}" + ) + dist.barrier() + if RANK == 0: + print(f"[cell] world={WORLD} {cell.why}: bitwise", flush=True) + + +def check_graph_replays_fresh_contents() -> None: + cell = [c for c in op.CELLS if c.spec["world"] == WORLD][0] + rows, cols = cell.spec["rows"], cell.spec["cols"] + static = _shards(100, rows, cols)[RANK].clone() + _call(static) + torch.cuda.synchronize() + graph_out = {} + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g): + graph_out["t"] = _call(static) + torch.cuda.current_stream().wait_stream(s) + for i in range(6): + shards = _shards(200 + i, rows, cols) + static.copy_(shards[RANK]) + g.replay() + torch.cuda.synchronize() + assert _bits_equal(graph_out["t"], op.reference(shards)), ( + f"replay {i} returned stale or wrong rows" + ) + dist.barrier() + + +def check_alternating_grids() -> None: + cell = [c for c in op.CELLS if c.spec["world"] == WORLD][0] + rows, cols = cell.spec["rows"], cell.spec["cols"] + for i in range(24): + shards = _shards(300 + i, rows, cols) + got = _call(shards[RANK], grid=16 if i % 2 else 0) + assert _bits_equal(got, op.reference(shards)), ( + f"call {i} (grid {16 if i % 2 else 'picked'}) differs" + ) + dist.barrier() + + +def check_supports_refuses() -> None: + cols = [c for c in op.CELLS if c.spec["world"] == WORLD][0].spec["cols"] + shard = torch.zeros(entry.MAX_ROWS + 1, cols, dtype=torch.bfloat16, device="cuda") + assert not entry.supports(shard, WORLD), "more rows than the workspace holds" + assert not entry.supports(shard[:4].float(), WORLD), "an fp32 shard" + assert not entry.supports( + torch.zeros(4, cols - 4, dtype=torch.bfloat16, device="cuda"), WORLD + ), "a partial 16-byte vector" + assert not entry.supports(shard[:4, :-8], WORLD), "a strided view" + dist.barrier() + + +CHECKS = ( + check_certified_cells, + check_graph_replays_fresh_contents, + check_alternating_grids, + check_supports_refuses, +) + + +def _run_one_rank(device: int) -> int: + global RANK, WORLD, WS, entry, op, validating + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(device) + dist.init_process_group( + "cuda:nccl,cpu:gloo", init_method="env://", device_id=torch.device("cuda", device) + ) + RANK, WORLD = rank, world + + from tensorrt_llm._torch._experimental.modeling_v2.catalog.comm import ( + allgather_cast as entry_module, + ) + from tensorrt_llm.mapping import Mapping + + entry = entry_module + op = entry.allgather_cast + mapping = Mapping(world_size=world, rank=rank, gpus_per_node=GPUS_PER_NODE, tp_size=world) + for cols in sorted({c.spec["cols"] for c in op.CELLS if c.spec["world"] == world}): + WS[cols] = entry.workspace_args(mapping, cols, dist.group.WORLD) + + sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + from _validating import validating as _validating_cm + + validating = _validating_cm + + for check in CHECKS: + try: + check() + except BaseException: + import traceback + + print(f"[rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + print(f"[rank {RANK}] {check.__name__} passed", flush=True) + dist.barrier() + torch.cuda.synchronize() + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict(os.environ, MASTER_ADDR="127.0.0.1", MASTER_PORT=str(port), WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1") # fmt: skip + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) # fmt: skip + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +def _rank_job(): + """`_rank_job.py` beside this file, by path (a sibling directory ships its own module of that name).""" + path = Path(__file__).resolve().with_name("_rank_job.py") + spec = importlib.util.spec_from_file_location("_comm_rank_job", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +if __name__ == "__main__": + if "--srun" in sys.argv: + sys.exit(_run_one_rank(device=_rank_job().adopt_srun_env())) + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank(device=int(os.environ["RANK"]))) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/comm/_rank_job.py b/tests/unittest/_torch/modeling_v2/comm/_rank_job.py index 3784a1a98361..f5aabba7f1e0 100644 --- a/tests/unittest/_torch/modeling_v2/comm/_rank_job.py +++ b/tests/unittest/_torch/modeling_v2/comm/_rank_job.py @@ -65,9 +65,10 @@ def _load_launcher_constants(launcher: Path): return module -def run(entry: str) -> None: +def run(entry: str, *, launcher_dir: Path | None = None) -> None: """Run ``__op_matrix``'s launcher over ``WORLD_SIZE`` devices.""" - launcher = Path(__file__).resolve().with_name(f"_{entry}_op_matrix.py") + directory = Path(__file__).resolve().parent if launcher_dir is None else launcher_dir.resolve() + launcher = directory / f"_{entry}_op_matrix.py" env = dict(os.environ, CUDA_VISIBLE_DEVICES=_devices()) # The ranks import tensorrt_llm absolutely, and a source checkout is not diff --git a/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_allgather_cast_op_matrix.py b/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_allgather_cast_op_matrix.py new file mode 100644 index 000000000000..0d5b6c41ab6e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/comm/test_modeling_v2_allgather_cast_op_matrix.py @@ -0,0 +1,32 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for `comm/allgather_cast`'s certification matrix (``_allgather_cast_op_matrix.py`` beside +this file, which drives the entry's ``.CELLS``). + +World 4 is collected here. The world-8 cells span two GB300 trays and run as one srun task per rank (see the +matrix's docstring), like ``_mnnvl_allreduce_op_matrix.py``. +""" + +import importlib.util +from pathlib import Path + +import pytest +import torch + +assert torch.cuda.is_available(), "allgather_cast requires CUDA devices" + + +def _rank_job(): + # By path: other test directories ship a module of the same name, and `import _rank_job` would get whichever + # one a collected sibling imported first. + path = Path(__file__).resolve().with_name("_rank_job.py") + spec = importlib.util.spec_from_file_location("_comm_rank_job", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +# Spawns 4 rank processes over the visible devices; under xdist workers would fight for them. +@pytest.mark.no_xdist +def test_allgather_cast_op_matrix() -> None: + _rank_job().run("allgather_cast") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/activation/test_modeling_v2_m31_tp4_swq.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/activation/test_modeling_v2_m31_tp4_swq.py new file mode 100644 index 000000000000..bc688eeb3307 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/activation/test_modeling_v2_m31_tp4_swq.py @@ -0,0 +1,99 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_swq entry of the MiniMax-M3.1 TP4 target: SwiGLU-OAI written as MXFP8, bitwise to +silu_and_mul + mxfp8_quantize.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.activation.m3_swq import ( # noqa: E501 + m3_swq as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_swq requires a CUDA device" + +OAI = (7.0, 1.702, 1.0) + + +def _assert_bitwise(x): + q, sf = op(x, *OAI) + rq, rsf = op.reference(x, *OAI) + assert q.shape == rq.shape and q.dtype == torch.float8_e4m3fn + assert torch.equal(q.view(torch.uint8), rq.view(torch.uint8)) + assert sf.numel() <= rsf.numel() and torch.equal(sf, rsf.view(-1)[: sf.numel()]) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = (torch.randn(cell.spec["m"], 2 * cell.spec["inter"], generator=g, device="cuda") * 4.0).to( + torch.bfloat16 + ) + with validating(op): + _assert_bitwise(x) + + +@pytest.mark.parametrize("m", [17, 24, 32]) +@pytest.mark.parametrize("scale", [1e-3, 1.0, 1e3]) +def test_bitwise_across_magnitudes_and_edges(m, scale) -> None: + """Scaled inputs, gates past the clamp, an all-zero row and a -0.0 row: q bytes and every scale byte.""" + g = torch.Generator(device="cuda").manual_seed(m * 1000 + int(scale * 10) % 997) + x = torch.randn(m, 2 * 768, generator=g, device="cuda") * scale + x[1, :768] = x[1, :768].abs() + OAI[0] + x[2] = 0.0 + x[3] = -0.0 + _assert_bitwise(x.to(torch.bfloat16)) + + +def _scale_bytes(sf, m, nb): + """Un-swizzle the 128x4 scale layout to [m, nb] bytes.""" + nbp = (nb + 3) // 4 * 4 + r = torch.arange(m, device=sf.device).view(-1, 1) + c = torch.arange(nb, device=sf.device).view(1, -1) + idx = ( + (c % 4) + (c // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + (r // 128) * (128 * nbp) + ) + return sf.view(-1)[idx] + + +@pytest.mark.parametrize("m", [17, 32]) +def test_tiny_amax_blocks_stay_finite(m) -> None: + """a block whose scale code is 0 (amax / 448 <= 2^-127) is native's inf/NaN; SWQ keeps native's scale + byte, writes data quantized with the exact reciprocal 2^127, and stays bitwise to native on every other block.""" + g = torch.Generator(device="cuda").manual_seed(151 + m) + gate = torch.randn(m, 768, generator=g, device="cuda") + up = torch.randn(m, 768, generator=g, device="cuda") + pattern = torch.tensor([1.0, -0.75, 0.5, -0.25, 0.375, 0.0, -1.0, 0.625] * 4, device="cuda") + for r in range(m): + for j, c in enumerate([2.0**-130, 2.0**-127, 448 * 2.0**-128, 448 * 2.0**-127, 2.0**-118]): + blk = (r * 5 + j * 3) % 24 + gate[r, blk * 32 : (blk + 1) * 32] = pattern * (2 * c) + up[r, blk * 32 : (blk + 1) * 32] = pattern * (2 * c) + x = torch.cat([gate, up], 1).to(torch.bfloat16) + q, sf = op(x, *OAI) + rq, rsf = op.reference(x, *OAI) + h = torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=OAI[0], swiglu_alpha=OAI[1], swiglu_beta=OAI[2] + ).float() + byte, rbyte = _scale_bytes(sf, m, 24), _scale_bytes(rsf, m, 24) + assert torch.equal(byte, rbyte), "scale bytes are native's on every block" + deq = q.float().view(m, 24, 32) * torch.exp2(byte.float() - 127.0).unsqueeze(-1) + rdeq = rq.float().view(m, 24, 32) * torch.exp2(rbyte.float() - 127.0).unsqueeze(-1) + native_finite = torch.isfinite(rdeq).all(-1) + code0 = (byte == 0) & (h.view(m, 24, 32).abs().amax(-1) != 0) + assert bool(code0.any()), "the planted blocks reach scale code 0" + assert not bool(native_finite[code0].any()), ( + "native is inf/NaN on code-0 blocks (the bug this guards)" + ) + assert bool(torch.isfinite(deq).all()), "SWQ is finite everywhere" + qb, rqb = q.view(torch.uint8).view(m, 24, 32), rq.view(torch.uint8).view(m, 24, 32) + assert bool((qb == rqb).all(-1)[native_finite].all()), ( + "bitwise to native wherever native is finite" + ) + exact = (h.view(m, 24, 32) * 2.0**127).to(torch.float8_e4m3fn).view(torch.uint8) + assert bool((qb == exact).all(-1)[code0].all()), "code-0 data = RN-e4m3(h * 2^127)" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_gqamix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_gqamix.py new file mode 100644 index 000000000000..c3aa7c93c210 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_gqamix.py @@ -0,0 +1,189 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GQAMIX: a mixed step's sparse generation rows (MsaDecodeFmha) through ops/gqamix_sparse_decode. + +The copy must be the in-tree kernel module except for the two decode kernels' specialization (no row count, no slice +alignment) and the one import it cannot make relatively, and must write the in-tree kernel's output bit for bit on a +mixed step's [num_ctx:] slices. The shared MsaDecodeFmha hook must change nothing while it is unset. +""" + +import inspect +import os +import subprocess +import sys +import types +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.triton_sparse_decode as in_tree +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( + modeling as m, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops import ( + gqamix_sparse_decode as copy, +) +from tensorrt_llm._torch.attention.backends.fmha.msa_decode import MsaDecodeFmha + +HEADS = 16 # query heads per rank at tp4, one KV head +D, PAGE, TOPK, QLEN, SF_VEC = 128, 128, 16, 4, 16 + +_JIT_OLD = '@triton.jit(do_not_specialize=["decode_query_len"])' +_JIT_NEW = ( + "@triton.jit(\n" + ' do_not_specialize=["decode_query_len", "total_q"],\n' + ' do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"],\n' + ")" +) +_IMPORT_OLD = "from .msa_utils import check_decode_span_shape" +_IMPORT_NEW = ( + "from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import (\n" + " check_decode_span_shape,\n" + ")" +) + + +def _code_lines(text: str) -> list: + return [line for line in text.splitlines() if line.strip()] + + +def test_the_copy_is_the_in_tree_module_but_for_its_specialization() -> None: + """Line for line (blank lines aside: the import sorts into a different block) the in-tree module.""" + ours = Path(copy.__file__).read_text() + theirs = Path(in_tree.__file__).read_text() + assert ( + ours.count(_JIT_NEW) == 2 + and ours.count(_IMPORT_NEW) == 1 + and theirs.count(_IMPORT_OLD) == 1 + ) + ours = ours.replace(_JIT_NEW, _JIT_OLD).replace(_IMPORT_NEW, "") + assert _code_lines(ours) == _code_lines(theirs.replace(_IMPORT_OLD, "")), ( + "ops/gqamix_sparse_decode.py must stay a copy of the in-tree triton_sparse_decode.py: re-copy it" + ) + + +def test_register_sets_the_hook_once_and_only_where_it_exists() -> None: + lib = types.SimpleNamespace(sparse_decode_hook=None) + other_lib = types.SimpleNamespace() # an FMHA library without the hook + manager = types.SimpleNamespace(fmha_libs=[other_lib, lib]) + m._gqamix_register(types.SimpleNamespace(_fmha_manager=manager)) + assert lib.sparse_decode_hook is m._gqamix_decode + assert not hasattr(other_lib, "sparse_decode_hook") + mine = object() + lib.sparse_decode_hook = mine + m._gqamix_register(types.SimpleNamespace(_fmha_manager=manager)) + assert lib.sparse_decode_hook is mine + m._gqamix_register(types.SimpleNamespace()) # a backend without an FMHA manager + + +def test_the_kill_switch_leaves_the_pre_hook_decode() -> None: + """TRTLLM_M31_DISABLE_GQAMIX=1: GQAMIX is off, so no layer registers the hook and MsaDecodeFmha runs the in-tree + decode (whose output the copy matches bit for bit below).""" + target = "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4" + probe = f"import {target}.modeling as m; print(m.GQAMIX)" + env = dict(os.environ, TRTLLM_M31_DISABLE_GQAMIX="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False" + source = inspect.getsource(m.PrefillTarget) + assert "if GQAMIX and sparse:\n _gqamix_register(core.attn[i])" in source + assert source.count("_gqamix_register(") == 1 + + +class _Attn: + is_mla_enable = False + kv_lora_rank = None + v_head_dim = None + head_dim = D + num_heads = HEADS + layer_idx = 3 + q_scaling = 1.0 + + +def _params(attn, num_tokens): + return types.SimpleNamespace( + attn=attn, + num_tokens=num_tokens, + meta=types.SimpleNamespace(kv_cache_manager=object()), + fwd=types.SimpleNamespace(kv_scale_quant_orig=None), + query_input=torch.zeros(num_tokens, HEADS * D), + output=torch.zeros(num_tokens, HEADS * D), + token_offset=2, + input_seq_length=QLEN, + ) + + +@pytest.mark.parametrize("hooked", [False, True], ids=["unset", "registered"]) +def test_the_shared_decode_hook_is_inert_unset(monkeypatch, hooked) -> None: + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import msa_utils + + calls = [] + monkeypatch.setattr(msa_utils, "msa_paged_kv", lambda manager, layer: ("k", "v")) + monkeypatch.setattr( + in_tree, "minimax_m3_sparse_attn_decode", lambda *a, **k: calls.append(("in_tree", a, k)) + ) + attn = _Attn() + fmha = MsaDecodeFmha(attn) + assert fmha.sparse_decode_hook is None + if hooked: + fmha.sparse_decode_hook = lambda *a, **k: calls.append(("hook", a, k)) + indexes = torch.zeros(10, 1, TOPK, dtype=torch.int32) + fmha._run_sparse(_params(attn, QLEN), indexes, "bt", "sl") + assert [c[0] for c in calls] == ["hook" if hooked else "in_tree"] + _, args, kwargs = calls[0] + assert args[1:3] == ("k", "v") and args[4:] == ("bt", "sl") + assert args[3].shape == (1, QLEN, TOPK) + assert kwargs["decode_query_len"] == QLEN and kwargs["sm_scale"] == D**-0.5 + + +def _mixed_step(nreq, seq_len, row_first, seed): + """A mixed step's generation rows: NVFP4 caches, FP8 q, and block-table / length slices past row_first.""" + g = torch.Generator(device="cuda").manual_seed(seed) + nb = seq_len // PAGE + pages = (nreq + row_first) * nb + block_table = torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(-1, nb) + seq_lens = torch.full((nreq + row_first,), seq_len, device="cuda", dtype=torch.int32) + total_q = nreq * QLEN + q = torch.randn(total_q, HEADS, D, device="cuda", generator=g).to(torch.float8_e4m3fn) + kv = [ + torch.randint( + 0, 256, (pages, 1, PAGE, D // 2), device="cuda", generator=g, dtype=torch.uint8 + ) + for _ in "kv" + ] + sc = [ + (torch.rand(pages, 1, PAGE, D // SF_VEC, device="cuda", generator=g) + 0.5).to( + torch.float8_e4m3fn + ) + for _ in "kv" + ] + topk = torch.stack( + [torch.randperm(nb, device="cuda", generator=g)[:TOPK] for _ in range(total_q)] + ) + args = (q, kv[0], kv[1], topk.to(torch.int32).view(total_q, 1, TOPK).permute(1, 0, 2)) + args += (block_table[row_first:], seq_lens[row_first:]) + kwargs = dict( + sm_scale=D**-0.5, + decode_query_len=QLEN, + k_block_scale=sc[0], + v_block_scale=sc[1], + k_global_scale=torch.tensor([0.02], device="cuda"), + v_global_scale=torch.tensor([0.03], device="cuda"), + ) + return args, kwargs, total_q + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA device") +@pytest.mark.parametrize("nreq,row_first", [(1, 0), (1, 1), (3, 2), (5, 3), (8, 1)]) +@pytest.mark.parametrize("seq_len", [16384, 131072]) +def test_the_copy_writes_the_in_tree_output_on_a_mixed_step(nreq, row_first, seq_len) -> None: + args, kwargs, total_q = _mixed_step(nreq, seq_len, row_first, 31 * nreq + row_first) + ref = torch.empty(total_q, HEADS, D, device="cuda", dtype=torch.bfloat16) + out = torch.empty_like(ref) + in_tree.minimax_m3_sparse_attn_decode(*args, output=ref, **kwargs) + copy.minimax_m3_sparse_attn_decode(*args, output=out, **kwargs) + torch.cuda.synchronize() + assert torch.isfinite(ref.float()).all() + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_lc4.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_lc4.py new file mode 100644 index 000000000000..06eaf7caa11f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_lc4.py @@ -0,0 +1,170 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for LC4: this target's copy of the long-context sparse decode +(``torch.ops.trtllm.minimax_m31_tp4_sparse_decode_lc``) with its two exact options, block-table staging (btsmem) and +the softmax sync trim (sync3), must write the same bits as the M3 op (``minimax_m3_sparse_decode_lc``): the BF16 rows, +the MXFP8 hand-off and its UE8M0 scales, for every option combination over the certified cells' shapes, a row longer +than the staged block table, repeated runs and a CUDA-graph replay. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.attention import ( # noqa: E501 + m3_sparse_decode_lc as entry, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.attention.m3_sparse_decode_lc import ( # noqa: E501 + WIDTH, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_nvfp4_cache import build_step # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "LC4 requires a CUDA device" + +_HANDOFF_ROWS = 128 +_OPTIONS = ((True, False), (False, True), (True, True)) + + +def _lc4(): + op = getattr(torch.ops.trtllm, "minimax_m31_tp4_sparse_decode_lc", None) + if op is None and entry._lc4_op() is not None: + op = torch.ops.trtllm.minimax_m31_tp4_sparse_decode_lc + if op is None: + pytest.skip("minimax_m31_tp4_sparse_decode_lc is not built") + return op + + +def _run(fn, case, linear_softmax, mxfp8, *opts): + total_q = case.q.shape[0] + out = torch.full((total_q, WIDTH), float("nan"), dtype=torch.bfloat16, device="cuda") + q8 = torch.zeros(_HANDOFF_ROWS, WIDTH, dtype=torch.uint8, device="cuda") if mxfp8 else None + sf = ( + torch.zeros(_HANDOFF_ROWS * (WIDTH // 32), dtype=torch.uint8, device="cuda") + if mxfp8 + else None + ) + fn( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + WIDTH // 128, + case.decode_query_len, + case.sm_scale, + linear_softmax, + False, + True, + *opts, + ) + return out, q8, sf + + +def _assert_same(a, b, what): + assert torch.equal(a[0].view(torch.int16), b[0].view(torch.int16)), f"{what}: BF16 rows differ" + if a[1] is not None: + assert torch.equal(a[1], b[1]), f"{what}: MXFP8 rows differ" + assert torch.equal(a[2], b[2]), f"{what}: UE8M0 scales differ" + + +@pytest.mark.parametrize( + "cell", + entry.m3_sparse_decode_lc.CELLS, + ids=[c.why[:44] for c in entry.m3_sparse_decode_lc.CELLS], +) +@pytest.mark.parametrize("linear_softmax", (False, True)) +@pytest.mark.parametrize("mxfp8", (False, True)) +@pytest.mark.parametrize("btsmem,sync3", _OPTIONS) +def test_lc4_matches_the_m3_op_bit_for_bit(cell, linear_softmax, mxfp8, btsmem, sync3) -> None: + lc4 = _lc4() + spec = cell.spec + case = build_step( + spec["seq_lens"], + spec["decode_query_len"], + identity=False, + seed=zlib.crc32(cell.why.encode()) % 2**31, + ) + ref = _run(torch.ops.trtllm.minimax_m3_sparse_decode_lc, case, linear_softmax, mxfp8) + got = _run(lc4, case, linear_softmax, mxfp8, btsmem, sync3) + _assert_same(got, ref, f"btsmem={btsmem} sync3={sync3}") + + +def test_lc4_row_longer_than_the_staged_block_table() -> None: + """A request past 8192 blocks reads its block table from global memory under btsmem: same bits.""" + lc4 = _lc4() + case = build_step((8193 * 128 + 5,), 1, identity=False, seed=31) + ref = _run(torch.ops.trtllm.minimax_m3_sparse_decode_lc, case, False, True) + for btsmem, sync3 in _OPTIONS: + _assert_same( + _run(lc4, case, False, True, btsmem, sync3), + ref, + f"long row btsmem={btsmem} sync3={sync3}", + ) + + +def test_lc4_graph_replay_and_repeats_are_bit_identical() -> None: + """Three eager runs and a CUDA-graph replay of the shipped placement (both options, hand-off on) agree.""" + lc4 = _lc4() + case = build_step((33000,), 4, identity=False, seed=21) + first = _run(lc4, case, False, True, True, True) + for _ in range(2): + _assert_same(_run(lc4, case, False, True, True, True), first, "repeat") + total_q = case.q.shape[0] + out = torch.empty(total_q, WIDTH, dtype=torch.bfloat16, device="cuda") + q8 = torch.zeros(_HANDOFF_ROWS, WIDTH, dtype=torch.uint8, device="cuda") + sf = torch.zeros(_HANDOFF_ROWS * (WIDTH // 32), dtype=torch.uint8, device="cuda") + args = ( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + WIDTH // 128, + case.decode_query_len, + case.sm_scale, + False, + True, + True, + True, + True, + ) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + lc4(*args) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + lc4(*args) + out.fill_(float("nan")) + q8.zero_() + sf.zero_() + graph.replay() + torch.cuda.synchronize() + _assert_same((out, q8, sf), first, "graph replay") + + +def test_btsmem_is_scoped_to_small_steps() -> None: + """btsmem costs larger grids their occupancy: the entry enables it only up to LC4_BTSMEM_MAX_ROWS rows.""" + assert entry.LC4_BTSMEM_MAX_ROWS == 8 + if entry.LC4_BTSMEM: + assert entry._lc4_btsmem(4) and entry._lc4_btsmem(8) + assert not entry._lc4_btsmem(9) and not entry._lc4_btsmem(32) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_tsel.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_tsel.py new file mode 100644 index 000000000000..27ceeb61fb8d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/test_modeling_v2_m31_tsel.py @@ -0,0 +1,141 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for TSEL: the PrefillTarget's top-16 block select through the #20629 prefill tile select +(torch.ops.trtllm.minimax_m31_select_prefill_tile), registered on the MSA indexer's selector hook. + +The table must equal minimax_m3_select_blocks' (head-major backing) bit for bit, ties included; outside its rows / +blocks range the hook returns None so the built-in select runs. Skipped when the op is not built (no overlay library). +""" + +import types + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( + modeling as m, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + select_blocks_from_maxscore, +) + +assert torch.cuda.is_available(), "TSEL requires a CUDA device" + +BLOCK = 128 +op = m._tsel_op() +needs_op = pytest.mark.skipif( + op is None, reason="minimax_m31_select_prefill_tile is not built (or not sm_103)" +) + + +def _inputs(ctx_end, rows, ties, seed, heads=1): + g = torch.Generator(device="cuda").manual_seed(seed) + blocks = (ctx_end + BLOCK - 1) // BLOCK + pos = torch.arange(ctx_end - rows, ctx_end, device="cuda") + nvb = ((pos + BLOCK) // BLOCK).to(torch.int32) + if ties: + scores = torch.randint(0, 8, (heads, blocks, rows), generator=g, device="cuda").float() + else: + scores = torch.randn(heads, blocks, rows, generator=g, device="cuda") + return scores.contiguous(), nvb + + +def _ref(scores, nvb, init_blocks, local_blocks): + return select_blocks_from_maxscore( + scores, + topk=16, + n_valid_blocks=nvb, + init_blocks=init_blocks, + local_blocks=local_blocks, + head_major_output=True, + ) + + +@needs_op +@pytest.mark.parametrize( + "ctx_end,rows", [(16384, 16384), (65536, 4096), (163840, 2048), (165000, 1160), (1024, 256)] +) +@pytest.mark.parametrize("ties", [False, True], ids=["random", "ties"]) +@pytest.mark.parametrize("init_blocks,local_blocks", [(0, 1), (2, 3)]) +def test_table_equals_select_blocks(ctx_end, rows, ties, init_blocks, local_blocks) -> None: + scores, nvb = _inputs(ctx_end, rows, ties, ctx_end + rows) + got = m._tsel_select(scores, nvb, init_blocks, local_blocks) + assert got is not None + ref = _ref(scores, nvb, init_blocks, local_blocks) + assert ( + got.shape == ref.shape + and got.stride() == ref.stride() + and got.dtype == ref.dtype == torch.int32 + ) + assert torch.equal(got, ref) + + +@needs_op +def test_two_kv_heads_and_minus_one_tail() -> None: + """Rows with fewer than 16 valid blocks pad with -1; two KV heads keep the head-major backing.""" + scores, nvb = _inputs(2048, 2048, False, 7, heads=2) + ref = _ref(scores, nvb, 0, 1) + got = m._tsel_select(scores, nvb, 0, 1) + assert got.shape == ref.shape and got.stride() == ref.stride() and got.dtype == ref.dtype + assert torch.equal(got, ref) + assert (got[:100] == -1).any() + + +@needs_op +def test_host_n_valid_blocks_are_converted() -> None: + """Callers hand either the device int32 step view or the host per_token_valid_blocks() (int64) tensor.""" + scores, nvb = _inputs(8192, 512, False, 5) + got = m._tsel_select(scores, nvb.cpu().long(), 0, 1) + assert torch.equal(got, _ref(scores, nvb, 0, 1)) + + +def test_outside_the_gate_keeps_the_built_in_select() -> None: + scores, nvb = _inputs(4096, m.TSEL_MIN_ROWS - 1, False, 3) + assert m._tsel_select(scores, nvb, 0, 1) is None + wide = torch.zeros(1, m.TSEL_MAX_BLOCKS + 1, 512, device="cuda") + assert m._tsel_select(wide, torch.ones(512, dtype=torch.int32, device="cuda"), 0, 1) is None + assert m._tsel_select(scores.transpose(1, 2).contiguous().transpose(1, 2), nvb, 0, 1) is None + + +def test_register_sets_the_hook_once_and_only_where_it_exists() -> None: + indexer = types.SimpleNamespace(selector_hook=None) + m._tsel_register(types.SimpleNamespace(indexer=indexer)) + assert indexer.selector_hook is m._tsel_select + other = object() + indexer.selector_hook = other + m._tsel_register(types.SimpleNamespace(indexer=indexer)) + assert indexer.selector_hook is other + m._tsel_register( + types.SimpleNamespace(indexer=types.SimpleNamespace()) + ) # an indexer without the hook + m._tsel_register(types.SimpleNamespace()) # a backend without an indexer + + +def test_the_shared_indexer_hook_is_inert_unset_or_declining() -> None: + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.msa_indexer import MsaIndexer + + cfg = types.SimpleNamespace(init_blocks=0, local_blocks=1, num_index_heads=1, num_kv_heads=1) + indexer = MsaIndexer(cfg) + assert indexer.selector_hook is None + scores, nvb = _inputs(4096, 300, False, 11) + ref = _ref(scores, nvb, 0, 1) + assert torch.equal(indexer._select(scores, nvb), ref) + indexer.selector_hook = lambda *a: None + assert torch.equal(indexer._select(scores, nvb), ref) + + +@needs_op +def test_fake_impl_matches_the_op() -> None: + """The registered fake (catalog m31_fake_ops) returns what the op returns (nothing) and leaves out as is.""" + from torch._subclasses.fake_tensor import FakeTensorMode + + from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import m31_fake_ops + + assert "minimax_m31_select_prefill_tile" in m31_fake_ops._REGISTERED + scores, nvb = _inputs(16384, 1160, False, 7) + out = torch.empty(1160, 1, 16, dtype=torch.int32, device="cuda") # (rows, heads, 16) + assert op(scores, nvb, out, 0, 1) is None + with FakeTensorMode(allow_non_fake_inputs=True) as mode: + fout = mode.from_tensor(out) + assert op(mode.from_tensor(scores), mode.from_tensor(nvb), fout, 0, 1) is None + assert (tuple(fout.shape), fout.dtype) == (tuple(out.shape), out.dtype) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_tp4_mxfp8_pinned_quantize.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_tp4_mxfp8_pinned_quantize.py new file mode 100644 index 000000000000..4b9ee50fc211 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_tp4_mxfp8_pinned_quantize.py @@ -0,0 +1,33 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_pinned_quantize entry of the MiniMax-M3.1 TP4 target (QPIN): every tactic is bitwise the +stock MXFP8 quantize.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.gemm.m3_mxfp8_pinned_quantize import ( # noqa: E501 + m3_mxfp8_pinned_quantize as op, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner as QR + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_pinned_quantize requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +@pytest.mark.parametrize("tactic", [QR.CUTE_DSL, QR.TRTLLM], ids=["cute_dsl", "trtllm"]) +def test_certified_cells_bitwise(cell, tactic) -> None: + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = (torch.randn(cell.spec["m"], cell.spec["k"], generator=g, device="cuda") * 3.0).to( + torch.bfloat16 + ) + with validating(op): + q, sf = op(x, tactic) + rq, rsf = op.reference(x, tactic) + assert torch.equal(q.view(torch.uint8).view(-1), rq.view(torch.uint8).view(-1)) + assert torch.equal(sf.view(-1), rsf.view(-1)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_tp4_shdown_pin.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_tp4_shdown_pin.py new file mode 100644 index 000000000000..fa7fd4ef66d5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_tp4_shdown_pin.py @@ -0,0 +1,238 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The SHDOWN pin of the MiniMax-M3.1 TP4 target: the shared down projection at the rows the decode GEMV does not +serve runs FlashInfer's CuTe DSL MXFP8 GEMM, fixed, instead of the per-start tuned backend.""" + +import os +import subprocess +import sys +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.gemm.m3_mxfp8_gemm import ( # noqa: E501 + m3_mxfp8_gemm, + m3_mxfp8_gemm_cute_dsl, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import ULP, assert_within_ulp # noqa: E402 — needs the path declared above + +_N, _K = modeling.SHARED_DOWN + + +def test_the_pin_covers_exactly_the_rows_the_gemv_leaves_to_the_gemm(): + pinned = modeling.SHDOWN_PIN_ROWS[modeling.SHARED_DOWN] + gemv = modeling.MX_GEMV_ROWS[modeling.SHARED_DOWN] + assert pinned == frozenset(range(17, 33)) + assert not pinned & gemv + assert set(modeling.SHDOWN_PIN_ROWS) == {modeling.SHARED_DOWN} + + +def test_the_kill_switch_restores_the_tuned_backend(): + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_103__tp4 import modeling; print(len(modeling.SHDOWN_PIN_ROWS))" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_SHDOWN_PIN="1") + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "0" + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +@pytest.mark.parametrize("m", [17, 24, 32]) +def test_the_pinned_backend_is_the_mxfp8_product(m): + """Same E4M3 operands and power-of-two scales as the reference; accumulation order then one bf16 rounding: the + band of the target's MXFP8 GEMM test (2 ulp of the row scale, RMS 0.25 ulp). Also reports the largest difference + from the CUTLASS backend the AutoTuner can pick here, in ulp of the row scale (report only).""" + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(f"shdown{m}".encode()) % 2**31) + act, act_sf = mxfp8(m, _K, g) + w, w_sf = mxfp8(_N, _K, g) + out = m3_mxfp8_gemm_cute_dsl(act, act_sf, w, w_sf, torch.bfloat16) + ref = m3_mxfp8_gemm.reference(act, act_sf, w, w_sf, torch.bfloat16, False) + assert out.shape == (m, _N) and out.dtype == torch.bfloat16 + assert_within_ulp(out, ref, element_ulp=2.0, rms_ulp=0.25) + from tensorrt_llm._torch.custom_ops.torch_custom_ops import FlashInferMXFP8GemmRunner as R + + cutlass = R(torch.bfloat16)([act, act_sf, w, w_sf], tactic=R.CUTLASS) + scale = ref.float().abs().amax(dim=1, keepdim=True).clamp_min(1e-9) + ulps = ((out.float() - cutlass.float()).abs() / scale / ULP[torch.bfloat16]).max().item() + print(f"[shdown-pin] m={m}: max |cute_dsl - cutlass| = {ulps:.2f} ulp of the row scale") + + +class _GenericTaken(Exception): + pass + + +def _decode_graph_dispatch(monkeypatch): + """``_linear`` as inside a decode CUDA graph, with no producer-published MXFP8 activation (the shared expert's + down projection reads the SwiGLU's bf16 output) and the tuned quantize + GEMM replaced by a sentinel.""" + mode = modeling.MxMode( + decode_graph=True, + autotune=False, + tuned_quantize_gemm=True, + flashinfer=True, + native_tuning=False, + ) + monkeypatch.setattr(modeling, "_mx_mode", lambda state: mode) + monkeypatch.setattr( + modeling.m3_prequant, "SLOT", type("Slot", (), {"take": staticmethod(lambda x: None)})() + ) + + def generic(*args, **kwargs): + raise _GenericTaken + + monkeypatch.setattr(modeling, "_mxfp8_linear", generic) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +@pytest.mark.parametrize("m", [17, 24, 32]) +def test_the_bf16_shared_down_activation_takes_the_pin(monkeypatch, m): + """The production shared down projection: a bf16 activation at a pinned row count runs the native MXFP8 quantize + and the CuTe DSL GEMM (bitwise), never the tuned quantize + GEMM a persisted tactic cache steers.""" + _decode_graph_dispatch(monkeypatch) + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(f"shdown-bf16-{m}".encode()) % 2**31) + w, w_sf = mxfp8(_N, _K, g) + core = type("Core", (), {"w": {"dn_w": w, "dn_s": w_sf}, "mxfp8": None})() + x = torch.randn(m, _K, generator=g, device="cuda", dtype=torch.bfloat16) + out, pushed = modeling._linear(core, x, "dn_w", "dn_s", False) + act, act_sf = torch.ops.trtllm.mxfp8_quantize(x, True) + expect = m3_mxfp8_gemm_cute_dsl(act.view(m, _K), act_sf, w, w_sf, torch.bfloat16) + assert not pushed + assert torch.equal(out, expect) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +def test_unpinned_bf16_rows_keep_the_tuned_backend(monkeypatch): + _decode_graph_dispatch(monkeypatch) + g = torch.Generator(device="cuda").manual_seed(33) + w, w_sf = mxfp8(_N, _K, g) + core = type("Core", (), {"w": {"dn_w": w, "dn_s": w_sf}, "mxfp8": None})() + x = torch.randn(33, _K, generator=g, device="cuda", dtype=torch.bfloat16) + monkeypatch.setattr(modeling, "QPIN", False) # QPIN's own tests below + with pytest.raises(_GenericTaken): + modeling._linear(core, x, "dn_w", "dn_s", False) + + +class _SwqCore: + """The decode target's core as ``_swiglu_mlp`` reads it: weights, the MXFP8 state (unused: ``_mx_mode`` is + patched) and the SwiGLU-OAI scalars.""" + + def __init__(self, gu_out, w, w_sf): + self.w = {"sh_gu_w": gu_out, "sh_gu_s": None, "sh_dn_w": w, "sh_dn_s": w_sf} + self.mxfp8 = None + self.swiglu_limit, self.swiglu_alpha, self.swiglu_beta = 7.0, 1.702, 1.0 + + +def _swq_dispatch(monkeypatch, m, seed): + """Run DecodeTarget._swiglu_mlp at m rows with the gate_up projection replaced by a fixed bf16 output; returns + (output, the pinned GEMM's activation sources, the expected product).""" + _decode_graph_dispatch(monkeypatch) + monkeypatch.setattr( + modeling, "QPIN", False + ) # the gate_up stub below stands in for the generic path + # A real slot: SWQ publishes the MXFP8 activation that the down projection takes. + monkeypatch.setattr(modeling.m3_prequant, "SLOT", modeling.m3_prequant.PrequantSlot()) + g = torch.Generator(device="cuda").manual_seed(seed) + gu = (torch.randn(m, 2 * _K, generator=g, device="cuda") * 4.0).to(torch.bfloat16) + w, w_sf = mxfp8(_N, _K, g) + core = _SwqCore(gu, w, w_sf) + + def gate_up_or_raise(core_, x, w_, s_, mode): + if w_ is core_.w["sh_gu_w"]: + return gu + raise _GenericTaken + + monkeypatch.setattr(modeling, "_mxfp8_linear", gate_up_or_raise) + sources = [] + pinned = modeling._shdown_pinned + + def recording(act, act_scale, w_, s_, source): + sources.append(source) + return pinned(act, act_scale, w_, s_, source) + + monkeypatch.setattr(modeling, "_shdown_pinned", recording) + h = torch.zeros(m, 6144, device="cuda", dtype=torch.bfloat16) + target = type("Target", (), {"core": core})() + out, pushed = modeling.DecodeTarget._swiglu_mlp(target, h, "sh", False) + act = torch.ops.trtllm.silu_and_mul(gu, swiglu_limit=7.0, swiglu_alpha=1.702, swiglu_beta=1.0) + aq, asf = torch.ops.trtllm.mxfp8_quantize(act, True) + expect = m3_mxfp8_gemm_cute_dsl(aq.view(m, _K), asf, w, w_sf, torch.bfloat16) + return out, pushed, sources, expect + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +@pytest.mark.parametrize("m", [17, 24, 32]) +def test_swq_reaches_the_pinned_gemm_on_its_mxfp8_activation(monkeypatch, m): + """SWQ at 17..32 rows: the down projection takes the prequant branch (the pin's '(mxfp8 activation)' source) and + its product is bitwise the bf16 SwiGLU + quantize + pinned GEMM path it replaces.""" + out, pushed, sources, expect = _swq_dispatch( + monkeypatch, m, zlib.crc32(f"swq{m}".encode()) % 2**31 + ) + assert sources == ["mxfp8"] and not pushed + assert torch.equal(out, expect) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +def test_swq_kill_switch_keeps_the_bf16_pin_path(monkeypatch): + """SWQ off: the same rows take the bf16 SwiGLU and reach the pin with its '(bf16 activation)' source, bitwise + the same product.""" + monkeypatch.setattr(modeling, "SWQ_ROWS", {}) + out, pushed, sources, expect = _swq_dispatch(monkeypatch, 24, 24) + assert sources == ["bf16"] and not pushed + assert torch.equal(out, expect) + + +def test_swq_rows_are_the_pinned_rows(): + assert modeling.SWQ_ROWS[modeling.SHARED_DOWN] == frozenset(range(17, 33)) + assert modeling.SWQ_ROWS[modeling.SHARED_DOWN] <= modeling.SHDOWN_PIN_ROWS[modeling.SHARED_DOWN] + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +@pytest.mark.parametrize( + "m, k, tactic", [(16, 6144, "cute"), (32, 3072, "cute"), (33, 768, "cute"), (32, 768, "trt")] +) +def test_qpin_quantize_is_fixed_and_the_product_bitwise(monkeypatch, m, k, tactic): + """QPIN: a bf16 projection off the decode kernels' rows quantizes on the fixed tactic (CuTe DSL; TRT-LLM at + (32, 768)) and runs the tuned GEMM; the product is bitwise the tuned quantize + GEMM's (both quantize tactics + give the same bytes).""" + from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner as QR + + _decode_graph_dispatch(monkeypatch) + monkeypatch.setattr(modeling, "SHDOWN_PIN_ROWS", {}) + seen = [] + pinned_q = modeling.m3_mxfp8_pinned_quantize + + def recording(x, t): + seen.append(t) + return pinned_q(x, t) + + monkeypatch.setattr(modeling, "m3_mxfp8_pinned_quantize", recording) + g = torch.Generator(device="cuda").manual_seed(m * 7 + k) + w, w_sf = mxfp8(6144, k, g) + core = type("Core", (), {"w": {"dn_w": w, "dn_s": w_sf}, "mxfp8": None})() + x = torch.randn(m, k, generator=g, device="cuda").to(torch.bfloat16) + out, pushed = modeling._linear(core, x, "dn_w", "dn_s", False) + assert seen == [QR.TRTLLM if tactic == "trt" else QR.CUTE_DSL] and not pushed + aq, asf = torch.ops.trtllm.mxfp8_quantize(x, True) + expect = m3_mxfp8_gemm(aq.view(m, k), asf, w, w_sf, torch.bfloat16, False) + assert torch.equal(out, expect) + + +def test_qpin_kill_switch(): + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_103__tp4 import modeling; print(modeling.QPIN)" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_QPIN="1") + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_unfused_index.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_unfused_index.py new file mode 100644 index 000000000000..a3a5098cb9c3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/test_modeling_v2_m31_unfused_index.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test: the unfused sparse layers' separate MXFP8 index projection (``fuse_qkv_index_projection=False``) of the +tp4 target equals the index columns of the fused five-way projection on the same checkpoint tensors. + +Its weight rows are the five-way weight's index rows byte for byte, its block scales are applied (not dropped), and +on every engine MXFP8 backend its output is bitwise the five-way GEMM's index columns. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( + weights as tw, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.gemm.m3_mxfp8_linear import ( # noqa: E501 + m3_mxfp8_linear as op, +) + +assert torch.cuda.is_available(), "the MXFP8 index projection requires a CUDA device" + +TP = 4 +HIDDEN = 6144 +BLOCK = 32 +# Text-model rows of the attention projections (64 query / 4 KV heads, 4 index heads, head and index dim 128). +ROWS = {"q": 64 * 128, "k": 4 * 128, "v": 4 * 128, "index_q": 4 * 128, "index_k": 128} +_FLAGS = { + "native": (False, False, False, False), + "flashinfer": (False, False, True, False), + "tuned": (True, False, False, False), +} + + +def _checkpoint(seed: int): + """E4M3 weights and UE8M0 block scales per projection; the scales spread over 2^-6 .. 2^5 (bytes 121..132).""" + g = torch.Generator(device="cuda").manual_seed(seed) + out = {} + for name, rows in ROWS.items(): + w = (torch.randn(rows, HIDDEN, generator=g, device="cuda") * 2).to(torch.float8_e4m3fn) + s = torch.randint( + 121, 133, (rows, HIDDEN // BLOCK), generator=g, device="cuda", dtype=torch.uint8 + ) + out[name] = (w.cpu(), s.cpu()) + return out + + +def _ns(rank: int) -> SimpleNamespace: + return SimpleNamespace(tp=TP, rank=rank, dev=torch.device("cuda"), head_dim=128, index_dim=128) + + +def _packs(ck, rank): + d = _ns(rank) + names = ("q", "k", "v", "index_q", "index_k") + w5 = tw._qkv_sparse_w(tuple(ck[n][0] for n in names), d) + s5 = tw._qkv_sparse_s( + tuple(ck[n][1] for n in names), d + ) # UE8M0 bytes, as the checkpoint stores them + wi = tw._index_qk_w((ck["index_q"][0], ck["index_k"][0]), d) + si = tw._index_qk_s((ck["index_q"][1], ck["index_k"][1]), d) + return w5, s5.view(torch.uint8), wi, si.view(torch.uint8) + + +@pytest.mark.parametrize("rank", [0, TP - 1]) +def test_weight_rows_are_the_five_way_index_rows(rank) -> None: + ck = _checkpoint(1234 + rank) + w5, _, wi, _ = _packs(ck, rank) + n = wi.shape[0] + assert wi.dtype == torch.float8_e4m3fn and wi.shape == (n, HIDDEN) + assert torch.equal(wi.view(torch.uint8), w5[-n:].view(torch.uint8)) + + +@pytest.mark.parametrize("path", list(_FLAGS)) +@pytest.mark.parametrize("rows", [1, 4, 8, 32, 64, 256]) +def test_unfused_output_is_bitwise_the_five_way_index_columns(path, rows) -> None: + if path == "flashinfer" and getattr(torch.ops.trtllm, "flashinfer_mm_mxfp8", None) is None: + pytest.skip("this build has no trtllm::flashinfer_mm_mxfp8") + ck = _checkpoint(77) + w5, s5, wi, si = _packs(ck, 0) + n = wi.shape[0] + g = torch.Generator(device="cuda").manual_seed(rows) + x = torch.randn(rows, HIDDEN, generator=g, device="cuda").to(torch.bfloat16) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + out5 = op(x, w5, s5, alpha, torch.bfloat16, *_FLAGS[path]) + outi = op(x, wi, si, alpha, torch.bfloat16, *_FLAGS[path]) + assert outi.shape == (rows, n) + assert torch.equal(outi, out5[:, -n:]), ( + f"{path}: max |diff| {(outi.float() - out5[:, -n:].float()).abs().max().item()}" + ) + + +def test_block_scales_are_applied() -> None: + """The output follows the op's reference with the checkpoint scales and differs from a unit-scale (dropped) + projection by far more than rounding.""" + ck = _checkpoint(5) + _, _, wi, si = _packs(ck, 0) + unit = tw._index_qk_s( + (torch.full_like(ck["index_q"][1], 127), torch.full_like(ck["index_k"][1], 127)), _ns(0) + ).view(torch.uint8) + x = torch.randn(8, HIDDEN, device="cuda").to(torch.bfloat16) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + out = op(x, wi, si, alpha, torch.bfloat16, *_FLAGS["native"]) + ref = op.reference(x, wi, si, alpha, torch.bfloat16, *_FLAGS["native"]) + dropped = op(x, wi, unit, alpha, torch.bfloat16, *_FLAGS["native"]) + scale = ref.float().abs().mean() + # The engine GEMM rounds once to BF16 after an fp32 accumulation in its own order: within 2 BF16 ulps (2^-7 + # relative) of the reference per element. + assert bool( + ((out.float() - ref.float()).abs() <= 2.0**-7 * ref.float().abs() + 1e-3 * scale).all() + ) + assert (out.float() - dropped.float()).abs().mean() > 0.5 * scale diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/moe/test_modeling_v2_m31_tp4_moe_early0n.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/moe/test_modeling_v2_m31_tp4_moe_early0n.py new file mode 100644 index 000000000000..56adf9d0869d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/moe/test_modeling_v2_m31_tp4_moe_early0n.py @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The TP4 MoE engine's early group-0 issue depth (MOE_EARLY0N): 6 stages at 4 tokens and 3 at 8, measured faster with +bit-identical outputs; every other token count keeps the kernel default; TRTLLM_M31_DISABLE_MOE_EARLY0N restores the +default everywhere.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.m3_moe import ( + op as moe_op, +) + + +def test_early0n_table_scope() -> None: + assert moe_op.EARLY0_N_BY_M == (0, 0, 0, 6, 0, 0, 0, 3) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize(("num_tokens", "expected"), [(1, None), (4, 6), (8, 3)]) +@pytest.mark.parametrize("killed", [False, True]) +def test_early0n_config_and_kill_switch( + monkeypatch: pytest.MonkeyPatch, num_tokens: int, expected, killed: bool +) -> None: + monkeypatch.setattr(moe_op, "m31_disabled", lambda item_id: killed and item_id == "MOE_EARLY0N") + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + state = moe_op.M3MoeState(dev, num_tokens=num_tokens, num_ctas=sms - 24, shared=True) + assert state.config.get("early0_n") == (None if killed else expected) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/moe/test_modeling_v2_m31_tp4_moe_m4cfg.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/moe/test_modeling_v2_m31_tp4_moe_m4cfg.py new file mode 100644 index 000000000000..f28f315a1365 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/moe/test_modeling_v2_m31_tp4_moe_m4cfg.py @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The TP4 MoE engine's 4-token overrides (MOE_M4CFG): the B-image hand-off plus 8 ring stages, measured faster with +bit-identical outputs; other token counts keep the tables; TRTLLM_M31_DISABLE_MOE_M4CFG restores the tables.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.m3_moe import ( + op as moe_op, +) + + +def test_m4cfg_scope() -> None: + assert moe_op.M4_CFG_BY_M == {4: {"b2img": 1, "stages": 8}} + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("killed", [False, True]) +def test_m4cfg_config_and_kill_switch(monkeypatch: pytest.MonkeyPatch, killed: bool) -> None: + monkeypatch.setattr(moe_op, "m31_disabled", lambda item_id: killed and item_id == "MOE_M4CFG") + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + state = moe_op.M3MoeState(dev, num_tokens=4, num_ctas=sms - 24, shared=True) + if killed: + assert state.config["b2img"] == moe_op.B2IMG_BY_M[3] and "stages" not in state.config + else: + assert state.config["b2img"] == 1 and state.config["stages"] == 8 + assert state.mod.B2IMG and state.mod.STAGES == 8 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/norm/test_modeling_v2_m31_ctxnormq.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/norm/test_modeling_v2_m31_ctxnormq.py new file mode 100644 index 000000000000..48a1b49b0e96 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/norm/test_modeling_v2_m31_ctxnormq.py @@ -0,0 +1,104 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for CTXNORMQ: the prefill boundary's residual add + RMSNorm that also writes the next projection's MXFP8 +activation and the routed experts' NVFP4 input (torch.ops.trtllm.minimax_m31_ctx_residual_rmsnorm_mxfp8). + +The quantized outputs must equal mxfp8_quantize / fp4_quantize of the op's own norm bit for bit; the norm and the +updated residual are checked against an fp32 reference. Skipped when the op is not built (no overlay library). +""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( + modeling as m, +) + +assert torch.cuda.is_available(), "CTXNORMQ requires a CUDA device" + +H = 6144 +EPS = 1e-6 +FP4_SCALE = 448.0 * 6.0 / 8.0 + +op = m._ctxnormq_op() +pytestmark = pytest.mark.skipif( + op is None, reason="minimax_m31_ctx_residual_rmsnorm_mxfp8 is not built" +) + + +def _operands(rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + inter = torch.randn(rows, H, generator=g, device="cuda").to(torch.bfloat16) + residual = (2 * torch.randn(rows, H, generator=g, device="cuda")).to(torch.bfloat16) + weight = (1.0 + 0.1 * torch.randn(H, generator=g, device="cuda")).to(torch.bfloat16) + return inter, residual, weight + + +@pytest.mark.parametrize("rows", [1024, 1160, 4096]) +@pytest.mark.parametrize("fp4", [False, True], ids=["attn", "moe"]) +def test_quantized_outputs_equal_quantizing_the_norm(rows, fp4) -> None: + inter, residual, weight = _operands(rows, 20261021 + rows) + scale = torch.tensor([FP4_SCALE], device="cuda", dtype=torch.float32) if fp4 else None + ref_updated = (inter.float() + residual.float()).to(torch.bfloat16) + normed, q, sf, f4, f4_sf = op(inter, residual, weight, EPS, True, scale, False) + # The updated residual is written in place into inter: one bf16 rounding of the fp32 sum, exact. + assert torch.equal(inter.view(torch.int16), ref_updated.view(torch.int16)) + u = ref_updated.float() + ref_normed = u * torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + EPS) * weight.float() + # fp32 sum of squares + fast rsqrt vs torch's: at most one bf16 rounding apart. + torch.testing.assert_close(normed.float(), ref_normed, rtol=2**-7, atol=2**-7) + q_ref, sf_ref = torch.ops.trtllm.mxfp8_quantize(normed, True) + assert torch.equal(q.view(torch.uint8), q_ref.view(torch.uint8)) + assert torch.equal(sf.view(torch.uint8)[: sf_ref.numel()], sf_ref.view(torch.uint8).flatten()) + if fp4: + f4_ref, f4_sf_ref = torch.ops.trtllm.fp4_quantize(normed, scale, 16, False, False) + assert torch.equal(f4.view(torch.uint8).flatten(), f4_ref.view(torch.uint8).flatten()) + assert torch.equal( + f4_sf.view(torch.uint8).flatten(), + f4_sf_ref.view(torch.uint8).flatten()[: f4_sf.numel()], + ) + # _routed_chunks splits the NVFP4 scales per row chunk. + assert f4_sf.numel() == rows * H // 16 + + +def test_deterministic() -> None: + outs = [] + for _ in range(3): + inter, residual, weight = _operands(2048, 7) + scale = torch.tensor([FP4_SCALE], device="cuda", dtype=torch.float32) + outs.append((inter,) + tuple(op(inter, residual, weight, EPS, True, scale, False))) + for o in outs[1:]: + assert all( + torch.equal(a.view(torch.uint8), b.view(torch.uint8)) for a, b in zip(outs[0], o) + ) + + +def test_small_boundaries_keep_the_fused_allreduce() -> None: + """Below CTXNORMQ_MIN_ROWS the boundary is not taken (the microbench crossover); decided before any collective.""" + x = torch.zeros(m.CTXNORMQ_MIN_ROWS - 1, H, dtype=torch.bfloat16, device="cuda") + assert m._ctxnormq_allreduce_norm(None, x, x, x[0], None) is None + + +@pytest.mark.parametrize("fp4", [False, True], ids=["attn", "moe"]) +@pytest.mark.parametrize("write_norm", [False, True]) +def test_fake_impl_matches_the_op(fp4, write_norm) -> None: + """The registered fake (catalog m31_fake_ops) gives every output the op's shape and dtype.""" + from torch._subclasses.fake_tensor import FakeTensorMode + + from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import m31_fake_ops + + assert "minimax_m31_ctx_residual_rmsnorm_mxfp8" in m31_fake_ops._REGISTERED + inter, residual, weight = _operands(1160, 7) + scale = torch.tensor([FP4_SCALE], device="cuda") if fp4 else None + real = op(inter.clone(), residual, weight, EPS, write_norm, scale, False) + with FakeTensorMode(allow_non_fake_inputs=True) as mode: + fake = op( + mode.from_tensor(inter), + mode.from_tensor(residual), + mode.from_tensor(weight), + EPS, + write_norm, + None if scale is None else mode.from_tensor(scale), + False, + ) + assert [(tuple(t.shape), t.dtype) for t in fake] == [(tuple(t.shape), t.dtype) for t in real] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_chain_spec_mc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_chain_spec_mc.py new file mode 100644 index 000000000000..ed53e4304c7b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_chain_spec_mc.py @@ -0,0 +1,286 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for REJ_MC: the spec sampler's chain speculative sampling as +``torch.ops.trtllm.minimax_m31_chain_speculative_sampling_mc`` through ``ops/sampling/m31_chain_spec_mc``, installed as +the rejection sampler's override. + +It must equal the stock ``torch.ops.trtllm.chain_speculative_sampling`` bit for bit, tokens and counts +(the microbenchmark generators: realistic / same / far / ties / onehot drafts and targets, K7). The shared hook in +``speculative/interface.rejection_sampling_one_model`` must be stock while no override is set, and fall back to the +stock op whenever the override declines. TRTLLM_M31_DISABLE_REJ_MC=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.sampling import ( # noqa: E501 + m31_chain_spec_mc as entry, +) +from tensorrt_llm._torch.speculative import interface as spec_interface + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4" +) +CASES = ("realistic", "same", "far", "ties", "onehot") +op = entry.m31_chain_spec_mc + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 3): + pytest.skip("REJ_MC is an sm_103 kernel") + if entry.chain_spec_mc_op() is None: + pytest.skip("minimax_m31_chain_speculative_sampling_mc is not built") + if not hasattr(torch.ops.trtllm, "chain_speculative_sampling"): + pytest.skip("the stock chain_speculative_sampling op is not built") + pytest.importorskip("flashinfer.sampling") + + +def _probs(logits, top_p=0.95): + """softmax then FlashInfer's top-p renorm, as the sampler feeds the rejection step.""" + import flashinfer.sampling + + v = logits.shape[-1] + p = torch.softmax(logits, -1) + renormed = flashinfer.sampling.top_p_renorm_probs(p.view(-1, v).contiguous(), top_p) + return renormed.view(logits.shape).contiguous() + + +def _make(case, batch, k, v, seed): + """The microbenchmark input generators.""" + g = torch.Generator(device="cuda").manual_seed(seed) + + def rl(shape, scale=3.0): + return torch.randn(*shape, v, generator=g, device="cuda") * scale + + dl = rl((batch, k)) + if case == "realistic": # independent draft / target + tl = torch.cat([dl, rl((batch, 1))], 1) + rl((batch, k + 1), 1.0) + elif case == "same": # target == draft: all-accept and zero-residual paths + tl = torch.cat([dl, rl((batch, 1))], 1) + elif case == "far": # target peaked elsewhere: early reject + tl = rl((batch, k + 1)) + tl[:, :, :8] += 20 + elif case == "ties": + dl = torch.round(dl * 2) / 2 + tl = torch.round(torch.cat([dl, rl((batch, 1))], 1) * 2) / 2 + elif case == "onehot": + tl = torch.full((batch, k + 1, v), -1e4, device="cuda") + tl[:, :, 1234] = 0 + else: + raise ValueError(case) + dp, tp = _probs(dl), _probs(tl) + ids = torch.multinomial(dp.view(-1, v), 1, generator=g).view(batch, k).to(torch.int32) + seeds = torch.randint(0, 2**31, (batch,), generator=g, device="cuda", dtype=torch.int64) + offsets = torch.randint(0, 2**20, (batch,), generator=g, device="cuda", dtype=torch.int64) + return dp, ids.contiguous(), tp, seeds, offsets + + +def _stock(*args): + return torch.ops.trtllm.chain_speculative_sampling(*args) + + +def _same(got, ref, what): + assert torch.equal(got[0], ref[0]), f"{what}: accepted tokens differ" + assert torch.equal(got[1], ref[1]), f"{what}: token counts differ" + + +@pytest.mark.parametrize("case", CASES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_stock_op(cell, case) -> None: + """At each certified (batch, K, vocab): the override equals the stock op (tokens and counts), and the explicit + entry call at ctas_per_row 1 / 4 / 8 gives the same result again.""" + _gpu() + batch, k, v = cell.spec["batch"], cell.spec["k"], cell.spec["v"] + for seed in range(4): + args = _make(case, batch, k, v, 1000 * seed + batch) + got = entry.chain_spec_mc(*args) + assert got is not None, "an in-contract input must be served" + _same(got, _stock(*args), f"{case} batch={batch} seed={seed}") + for ctas in (1, 4, 8): + _same(op(*args, entry.prebuild(args[0].device), ctas), got, f"ctas_per_row={ctas}") + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager result.""" + _gpu() + args = _make("realistic", 8, 7, 200064, 7) + entry.prebuild(args[0].device) + ref = entry.chain_spec_mc(*args) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.chain_spec_mc(*args) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.chain_spec_mc(*args) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out[0].fill_(-7) + out[1].fill_(-7) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """Non-fp32 probs, non-int32 draft tokens, non-int64 or mis-sized seed / offset, a vocab not divisible by 4, a + misaligned or non-contiguous probs, mismatched shapes, CPU inputs or more requests than MAX_BATCH return None (the + sampler runs the stock op).""" + _gpu() + dp, ids, tp, seeds, offsets = _make("realistic", 2, 3, 32768, 1) + assert entry.chain_spec_mc(dp, ids, tp, seeds, offsets) is not None + assert entry.chain_spec_mc(dp.double(), ids, tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids, tp.half(), seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids.long(), tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids, tp, seeds.int(), offsets) is None + assert entry.chain_spec_mc(dp, ids, tp, seeds[:1], offsets) is None + assert entry.chain_spec_mc(dp, ids, tp, seeds, offsets[:1]) is None + assert entry.chain_spec_mc(dp, ids[:, :2].contiguous(), tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids, tp[:, :3].contiguous(), seeds, offsets) is None + assert entry.chain_spec_mc(dp[..., :-4], ids, tp[..., :-4], seeds, offsets) is None + assert ( + entry.chain_spec_mc( + dp[..., :-2].contiguous(), ids, tp[..., :-2].contiguous(), seeds, offsets + ) + is None + ) + flat = torch.empty(dp.numel() + 1, device="cuda") + assert entry.chain_spec_mc(flat[1:].view(dp.shape), ids, tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp.cpu(), ids.cpu(), tp.cpu(), seeds.cpu(), offsets.cpu()) is None + b = entry.MAX_BATCH + 1 + wide = _make("realistic", b, 1, 32768, 2) + assert entry.chain_spec_mc(*wide) is None + assert entry.ctas_per_row(1, 148) == 16 and entry.ctas_per_row(8, 148) == 16 + assert entry.ctas_per_row(16, 148) == 8 and entry.ctas_per_row(64, 148) == 2 + assert entry.ctas_per_row(200, 148) == 1 + + +@pytest.fixture +def _fake_stock_op(monkeypatch): + """Record the feature FlashInfer fallback; retain real torch tensor semantics.""" + stock = (torch.zeros(2, 4, dtype=torch.int32), torch.ones(2, dtype=torch.int32)) + fake_op = mock.Mock(return_value=(stock[0], None, stock[1] - 1)) + fake_fi = types.SimpleNamespace( + __version__="0.6.4", sampling=types.SimpleNamespace(chain_speculative_sampling=fake_op) + ) + monkeypatch.setattr(spec_interface, "flashinfer", fake_fi, raising=False) + monkeypatch.setattr(spec_interface, "IS_FLASHINFER_AVAILABLE", True) + spec_interface.set_chain_speculative_sampling_override(None) + yield fake_op, stock + spec_interface.set_chain_speculative_sampling_override(None) + + +def _cpu_args(): + return ( + torch.rand(2, 3, 8), + torch.zeros(2, 3, dtype=torch.int32), + torch.rand(2, 4, 8), + torch.zeros(2, dtype=torch.int64), + torch.zeros(2, dtype=torch.int64), + ) + + +def test_hook_unset_is_stock(_fake_stock_op) -> None: + """No override: rejection_sampling_one_model makes exactly the stock op call and returns its result.""" + fake_op, stock = _fake_stock_op + args = _cpu_args() + assert spec_interface._CHAIN_SPEC_OVERRIDE is None + result = spec_interface.rejection_sampling_one_model(*args[:3], seed=args[3], offset=args[4]) + assert result[0] is stock[0] + assert torch.equal(result[1], stock[1]) + fake_op.assert_called_once() + assert all(a is b for a, b in zip(fake_op.call_args.args, args[:3])) + assert fake_op.call_args.kwargs["seed"] is args[3] + assert fake_op.call_args.kwargs["offset"] is args[4] + assert fake_op.call_args.kwargs["deterministic"] is True + assert fake_op.call_args.kwargs["generator"] is None + + +def test_hook_override_result_is_used(_fake_stock_op) -> None: + """An override's result replaces the stock op; the stock op does not run.""" + fake_op, _ = _fake_stock_op + mine = (torch.ones(2, 4, dtype=torch.int32), torch.full((2,), 4, dtype=torch.int32)) + override = mock.Mock(return_value=mine) + spec_interface.set_chain_speculative_sampling_override(override) + args = _cpu_args() + assert ( + spec_interface.rejection_sampling_one_model(*args[:3], seed=args[3], offset=args[4]) is mine + ) + override.assert_called_once_with(*args) + fake_op.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_stock_op) -> None: + """An override that returns None leaves the stock op call in place.""" + fake_op, stock = _fake_stock_op + override = mock.Mock(return_value=None) + spec_interface.set_chain_speculative_sampling_override(override) + args = _cpu_args() + result = spec_interface.rejection_sampling_one_model(*args[:3], seed=args[3], offset=args[4]) + assert result[0] is stock[0] + assert torch.equal(result[1], stock[1]) + assert override.call_count == 1 and fake_op.call_count == 1 + + +@pytest.mark.parametrize("state", ["none", "scalar", "mixed", "nondeterministic"]) +def test_ineligible_rng_state_stays_stock(_fake_stock_op, state) -> None: + fake_op, stock = _fake_stock_op + args = _cpu_args() + override = mock.Mock() + spec_interface.set_chain_speculative_sampling_override(override) + seed, offset = args[3:] + if state == "none": + seed = offset = None + elif state == "scalar": + seed, offset = 17, 0 + elif state == "mixed": + offset = 0 + result = spec_interface.rejection_sampling_one_model( + *args[:3], deterministic=state != "nondeterministic", seed=seed, offset=offset + ) + override.assert_not_called() + fake_op.assert_called_once() + assert result[0] is stock[0] + assert torch.equal(result[1], stock[1]) + + +def test_the_target_installs_the_override() -> None: + """With REJ_MC on (the default) and the op built, the target installs ``chain_spec_mc`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( # noqa: E501 + modeling, + ) + + if not modeling.REJ_MC: + pytest.skip("REJ_MC is switched off in this environment") + saved = dict(modeling._REJ_MC_STATE) + modeling._REJ_MC_STATE.clear() + try: + modeling._install_rej_mc() + assert spec_interface._CHAIN_SPEC_OVERRIDE is entry.chain_spec_mc + assert modeling._REJ_MC_STATE["installed"] is True + finally: + spec_interface.set_chain_speculative_sampling_override(None) + modeling._REJ_MC_STATE.clear() + modeling._REJ_MC_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_REJ_MC=1 (read at import): REJ_MC is off and the target leaves the rejection hook unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.speculative import interface as i\n" + "m._install_rej_mc()\n" + "print(m.REJ_MC, i._CHAIN_SPEC_OVERRIDE is None, m._REJ_MC_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_REJ_MC="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_softmax_mc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_softmax_mc.py new file mode 100644 index 000000000000..8b2de70ecddc --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_softmax_mc.py @@ -0,0 +1,256 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for SOFTMAX_MC: the spec sampler's softmax as +``torch.ops.trtllm.minimax_m31_softmax_mc`` through ``ops/sampling/m31_softmax_mc``, installed as the sampler's softmax +override. + +It must equal ``flashinfer.sampling.softmax(logits, temperature)`` bit for bit (the microbenchmark generators: randn / +peaked / ties / near_uniform / all_equal / masked / wide rows, temperature 1 / per-row mixed / scalar). The shared +sampler hook must be byte-for-byte stock while no override is set, and fall back to the stock call whenever the +override declines. TRTLLM_M31_DISABLE_SOFTMAX_MC=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.sampling import ( # noqa: E501 + m31_softmax_mc as entry, +) +from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as fi_ops + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4" +) +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal", "masked", "wide") +TEMPS = ("t1", "mixed", "scalar") +op = entry.m31_softmax_mc + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 3): + pytest.skip("SOFTMAX_MC is an sm_103 kernel") + if entry.softmax_mc_op() is None: + pytest.skip("minimax_m31_softmax_mc is not built") + pytest.importorskip("flashinfer.sampling") + + +def _make(cls, rows, d, seed): + """The microbenchmark logit generators.""" + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + return (torch.randn(rows, d, generator=g, device="cuda") * 3).contiguous() + if cls == "peaked": # few dominant tokens, long tail (real-like rows) + x = torch.randn(rows, d, generator=g, device="cuda") * 2 + idx = torch.randint(0, d, (rows, 8), generator=g, device="cuda") + x.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + return x.contiguous() + if cls == "ties": + return (torch.round(torch.randn(rows, d, generator=g, device="cuda") * 2) / 2).contiguous() + if cls == "near_uniform": + return (torch.randn(rows, d, generator=g, device="cuda") * 1e-3).contiguous() + if cls == "all_equal": + return torch.zeros(rows, d, device="cuda") + if cls == "masked": + x = torch.randn(rows, d, generator=g, device="cuda") * 3 + x[:, d // 2 :] = -1e4 + return x.contiguous() + if cls == "wide": + return (torch.randn(rows, d, generator=g, device="cuda") * 30).contiguous() + raise ValueError(cls) + + +def _temperature(mode, rows, seed): + """FlashInfer's temperature argument: an fp32 [rows] tensor, or a Python float.""" + g = torch.Generator(device="cuda").manual_seed(seed + 3) + if mode == "t1": + return torch.ones(rows, device="cuda") + if mode == "mixed": + return (0.3 + 1.2 * torch.rand(rows, generator=g, device="cuda")).contiguous() + if mode == "scalar": + return 0.7 + raise ValueError(mode) + + +def _fi(logits, temperature): + import flashinfer.sampling + + return flashinfer.sampling.softmax(logits, temperature) + + +def _same(a, b, what): + """Bit for bit.""" + assert torch.equal(a.view(torch.int32), b.view(torch.int32)), f"{what}: fp32 bits differ" + + +@pytest.mark.parametrize("mode", TEMPS) +@pytest.mark.parametrize("cls", CLASSES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_flashinfer_softmax(cell, cls, mode) -> None: + """At each certified (rows, d): the override equals FlashInfer's softmax bit for bit, and the explicit entry call + at write_ctas 4 / 8 gives the same bits again.""" + _gpu() + rows, d = cell.spec["rows"], cell.spec["d"] + for seed in range(2): + logits = _make(cls, rows, d, 100 * seed + rows) + temperature = _temperature(mode, rows, seed) + got = entry.softmax_mc(logits, temperature) + assert got is not None, "an in-contract input must be served" + _same(got, _fi(logits, temperature), f"{cls} {mode} rows={rows} seed={seed}") + for write_ctas in (4, 8): + again = op(logits, temperature, entry.prebuild(logits.device), write_ctas) + _same(again, got, f"write_ctas={write_ctas}") + + +def test_no_temperature_is_one() -> None: + """A None temperature (softmax_op's Optional) is FlashInfer's default, 1.0.""" + _gpu() + logits = _make("peaked", 8, 200064, 11) + got = entry.softmax_mc(logits, None) + assert got is not None + _same(got, _fi(logits, None), "temperature None") + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager bits.""" + _gpu() + logits = _make("peaked", 8, 200064, 5) + temperature = _temperature("mixed", 8, 5) + entry.prebuild(logits.device) + ref = entry.softmax_mc(logits, temperature) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.softmax_mc(logits, temperature) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.softmax_mc(logits, temperature) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """A non-fp32 / misaligned / non-contiguous logits, a d % 4 width, a vocab below FlashInfer's Path A, a mis-sized, + non-fp32 or 2-D temperature, a non-number temperature or more rows than MAX_ROWS return None (the sampler runs + stock FlashInfer).""" + _gpu() + d = 200064 + logits = _make("randn", 4, d, 1) + temperature = torch.full((4,), 0.9, device="cuda") + assert entry.softmax_mc(logits.half(), temperature) is None + assert entry.softmax_mc(logits, temperature.double()) is None + assert entry.softmax_mc(logits, temperature[:3]) is None + assert entry.softmax_mc(logits, temperature.view(4, 1)) is None + assert entry.softmax_mc(logits, "0.9") is None + assert entry.softmax_mc(logits[:, : d - 4], temperature) is None + assert entry.softmax_mc(logits[:, : d - 2].contiguous(), temperature) is None + assert ( + entry.softmax_mc(logits[:, : entry.PATH_A_MIN_VOCAB - 4].contiguous(), temperature) is None + ) + flat = torch.empty(4 * d + 1, device="cuda") + assert entry.softmax_mc(flat[1:].view(4, d), temperature) is None + wide = _make("randn", entry.MAX_ROWS + 1, d, 2) + assert entry.softmax_mc(wide, torch.full((entry.MAX_ROWS + 1,), 0.9, device="cuda")) is None + assert entry.softmax_mc(logits.cpu(), temperature.cpu()) is None + + +@pytest.fixture +def _fake_flashinfer(monkeypatch): + """The sampler ops module with a stand-in flashinfer whose calls are recorded, and no override afterwards.""" + stock = torch.full((2, 8), 0.125) + sampling = types.SimpleNamespace( + softmax=mock.Mock(return_value=stock), + top_p_renorm_probs=mock.Mock(side_effect=lambda probs, top_p: probs), + top_k_mask_logits=mock.Mock(side_effect=lambda logits, top_k: logits), + ) + monkeypatch.setattr(fi_ops, "flashinfer", types.SimpleNamespace(sampling=sampling)) + fi_ops.set_softmax_override(None) + fi_ops.set_top_p_renorm_override(None) + yield sampling, stock + fi_ops.set_softmax_override(None) + + +def test_hook_unset_is_stock(_fake_flashinfer) -> None: + """No override: both ops make exactly the stock flashinfer call and return its result.""" + sampling, stock = _fake_flashinfer + logits, temperature = torch.rand(2, 8), torch.ones(2) + assert fi_ops._SOFTMAX_OVERRIDE is None + assert fi_ops.softmax_op(logits, temperature) is stock + sampling.softmax.assert_called_once_with(logits, temperature, enable_pdl=mock.ANY) + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, None) is stock + assert sampling.softmax.call_count == 2 + + +def test_hook_override_result_is_used(_fake_flashinfer) -> None: + """An override's tensor replaces the stock call in both ops; the stock kernel does not run, and a top_p renorm + after it gets the override's probs.""" + sampling, _ = _fake_flashinfer + mine = torch.zeros(2, 8) + override = mock.Mock(return_value=mine) + fi_ops.set_softmax_override(override) + logits, temperature = torch.rand(2, 8), torch.ones(2) + assert fi_ops.softmax_op(logits, temperature) is mine + override.assert_called_once_with(logits, temperature) + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, None) is mine + top_p = torch.full((2,), 0.9) + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, top_p) is mine + sampling.top_p_renorm_probs.assert_called_once_with(mine, top_p) + assert override.call_count == 3 + sampling.softmax.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_flashinfer) -> None: + """An override that returns None leaves the stock call in place in both ops.""" + sampling, stock = _fake_flashinfer + override = mock.Mock(return_value=None) + fi_ops.set_softmax_override(override) + logits, temperature = torch.rand(2, 8), torch.ones(2) + assert fi_ops.softmax_op(logits, temperature) is stock + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, None) is stock + assert override.call_count == 2 and sampling.softmax.call_count == 2 + + +def test_the_target_installs_the_override() -> None: + """With SOFTMAX_MC on (the default) and the op built, the target installs ``softmax_mc`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( # noqa: E501 + modeling, + ) + + if not modeling.SOFTMAX_MC: + pytest.skip("SOFTMAX_MC is switched off in this environment") + saved = dict(modeling._SOFTMAX_MC_STATE) + modeling._SOFTMAX_MC_STATE.clear() + try: + modeling._install_softmax_mc() + assert fi_ops._SOFTMAX_OVERRIDE is entry.softmax_mc + assert modeling._SOFTMAX_MC_STATE["installed"] is True + finally: + fi_ops.set_softmax_override(None) + modeling._SOFTMAX_MC_STATE.clear() + modeling._SOFTMAX_MC_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_SOFTMAX_MC=1 (read at import): SOFTMAX_MC is off and the target leaves the sampler hook + unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as f\n" + "m._install_softmax_mc()\n" + "print(m.SOFTMAX_MC, f._SOFTMAX_OVERRIDE is None, m._SOFTMAX_MC_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_SOFTMAX_MC="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_topp_det.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_topp_det.py new file mode 100644 index 000000000000..600defd2bff5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/sampling/test_modeling_v2_m31_topp_det.py @@ -0,0 +1,271 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for TOPP_DET: the spec sampler's top-p renorm as +``torch.ops.trtllm.minimax_m31_topp_renorm_det`` through ``ops/sampling/m31_topp_det``, installed as the sampler's +top-p renorm override. + +On rows with top_p < 1 it must equal FlashInfer's ``top_p_renorm_probs(probs, top_p, is_deterministic=True)`` bit for +bit (the microbenchmark generators: randn / peaked / ties / near_uniform / all_equal rows); +rows with top_p >= 1 keep every token. The shared sampler hook must be byte-for-byte stock while no override is set, and +fall back to the stock call whenever the override declines. TRTLLM_M31_DISABLE_TOPP_DET=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.sampling import ( # noqa: E501 + m31_topp_det as entry, +) +from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as fi_ops + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4" +) +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal") +FLT_MIN = torch.finfo(torch.float32).tiny +op = entry.m31_topp_det + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 3): + pytest.skip("TOPP_DET is an sm_103 kernel") + if entry.topp_det_op() is None: + pytest.skip("minimax_m31_topp_renorm_det is not built") + pytest.importorskip("flashinfer.sampling") + + +def _make(cls, rows, d, seed): + """The microbenchmark row generators: softmax of the class's logits.""" + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + logits = torch.randn(rows, d, generator=g, device="cuda") * 3 + elif cls == "peaked": # few dominant tokens, long tail (real-like draft rows) + logits = torch.randn(rows, d, generator=g, device="cuda") * 2 + idx = torch.randint(0, d, (rows, 8), generator=g, device="cuda") + logits.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + elif cls == "ties": + logits = torch.round(torch.randn(rows, d, generator=g, device="cuda") * 2) / 2 + elif cls == "near_uniform": + logits = torch.randn(rows, d, generator=g, device="cuda") * 1e-3 + elif cls == "all_equal": + logits = torch.zeros(rows, d, device="cuda") + else: + raise ValueError(cls) + return torch.softmax(logits, -1).contiguous() + + +def _top_p(mode, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed + 7) + if mode == "p95": + return torch.full((rows,), 0.95, device="cuda") + if mode == "mixed": + return (0.05 + 0.95 * torch.rand(rows, generator=g, device="cuda")).contiguous() + raise ValueError(mode) + + +def _fi_det(probs, top_p): + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + +def _same(a, b, what): + """Bit for bit.""" + assert torch.equal(a.view(torch.int32), b.view(torch.int32)), f"{what}: fp32 bits differ" + + +@pytest.mark.parametrize("mode", ["p95", "mixed"]) +@pytest.mark.parametrize("cls", CLASSES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_flashinfer_det_below_one(cell, cls, mode) -> None: + """Rows with top_p < 1 at each certified (rows, d): the override equals FlashInfer's det renorm bit for bit, and + the explicit entry call gives the same bits again (deterministic).""" + _gpu() + rows, d = cell.spec["rows"], cell.spec["d"] + for seed in range(2): + probs = _make(cls, rows, d, 1000 * rows + seed) + top_p = _top_p(mode, rows, seed) + got = entry.top_p_renorm_det(probs, top_p) + assert got is not None, "an in-contract input must be served" + ref = _fi_det(probs, top_p) + _same(got, ref, f"{cls} {mode} rows={rows} seed={seed}") + again = op( + probs, + top_p, + entry.prebuild(probs.device), + entry.ctas_per_row(rows, entry.sm_count(probs.device)), + ) + _same(again, got, "repeat") + + +@pytest.mark.parametrize("cls", CLASSES) +def test_keep_all_rows_in_a_mixed_batch(cls) -> None: + """A batch of 8 rows, a third at top_p 1: the top_p 1 rows keep every element whose renormed value is a normal float + (output nonzero <=> ftz(p * norm) != 0, the --use_fast_math semantics FlashInfer's apply shares) and sum to 1 within + 1e-5; the top_p < 1 rows are FlashInfer's det renorm bit for bit.""" + _gpu() + rows, d = 8, 200064 + probs = _make(cls, rows, d, 77) + top_p = _top_p("mixed", rows, 3) + keep = torch.zeros(rows, dtype=torch.bool, device="cuda") + keep[::3] = True + top_p[keep] = 1.0 + got = entry.top_p_renorm_det(probs, top_p) + assert got is not None + ref = _fi_det(probs, top_p) + _same(got[~keep], ref[~keep], f"{cls}: top_p < 1 rows") + for r in keep.nonzero().flatten().tolist(): + p64 = probs[r].double() + norm = 1.0 / float(p64[probs[r] >= FLT_MIN].sum()) + # kept <=> ftz(p * norm) != 0; elements within 1e-6 of the FLT_MIN boundary are left to the rounding + live = p64 * norm >= FLT_MIN * (1 + 1e-6) + dead = probs[r] < FLT_MIN + assert bool((got[r][live] != 0).all()), f"{cls} row {r}: a kept token was dropped" + assert bool((got[r][dead] == 0).all()), f"{cls} row {r}: a sub-FLT_MIN token is nonzero" + total = float(got[r].double().sum()) + assert abs(total - 1.0) <= 1e-5, f"{cls} row {r}: sum {total}" + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager bits.""" + _gpu() + probs = _make("peaked", 8, 200064, 5) + top_p = _top_p("mixed", 8, 5) + entry.prebuild(probs.device) + ref = entry.top_p_renorm_det(probs, top_p) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.top_p_renorm_det(probs, top_p) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.top_p_renorm_det(probs, top_p) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """A float top_p, a non-fp32 / misaligned / non-contiguous probs, a d % 4 width, a mis-sized top_p or more rows + than the workspace holds return None (the sampler runs stock FlashInfer).""" + _gpu() + d = 200064 + probs = _make("randn", 4, d, 1) + top_p = torch.full((4,), 0.9, device="cuda") + assert entry.top_p_renorm_det(probs, 0.9) is None + assert entry.top_p_renorm_det(probs.half(), top_p) is None + assert entry.top_p_renorm_det(probs, top_p.double()) is None + assert entry.top_p_renorm_det(probs, top_p[:3]) is None + assert entry.top_p_renorm_det(probs[:, : d - 2], top_p) is None + assert entry.top_p_renorm_det(probs[:, : d - 2].contiguous(), top_p) is None + flat = torch.empty(4 * d + 1, device="cuda") + assert entry.top_p_renorm_det(flat[1:].view(4, d), top_p) is None + wide = _make("randn", entry.MAX_ROWS + 1, d, 2) + assert ( + entry.top_p_renorm_det(wide, torch.full((entry.MAX_ROWS + 1,), 0.9, device="cuda")) is None + ) + assert entry.ctas_per_row(1, 148) == 16 and entry.ctas_per_row(8, 148) == 16 + assert entry.ctas_per_row(16, 148) == 8 and entry.ctas_per_row(64, 148) == 2 + assert entry.ctas_per_row(200, 148) == 1 + + +@pytest.fixture +def _fake_flashinfer(monkeypatch): + """The sampler ops module with a stand-in flashinfer whose calls are recorded, and no override afterwards.""" + stock = torch.full((2, 8), 0.125) + sampling = types.SimpleNamespace( + top_p_renorm_probs=mock.Mock(return_value=stock), + softmax=mock.Mock(side_effect=lambda logits, temps, enable_pdl=False: logits.softmax(-1)), + top_k_mask_logits=mock.Mock(side_effect=lambda logits, top_k: logits), + ) + monkeypatch.setattr(fi_ops, "flashinfer", types.SimpleNamespace(sampling=sampling)) + fi_ops.set_top_p_renorm_override(None) + yield sampling, stock + fi_ops.set_top_p_renorm_override(None) + + +def test_hook_unset_is_stock(_fake_flashinfer) -> None: + """No override: both ops make exactly the stock flashinfer call and return its result.""" + sampling, stock = _fake_flashinfer + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops._TOP_P_RENORM_OVERRIDE is None + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is stock + sampling.top_p_renorm_probs.assert_called_once_with(probs, top_p) + assert ( + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is stock + ) + assert sampling.top_p_renorm_probs.call_count == 2 + + +def test_hook_override_result_is_used(_fake_flashinfer) -> None: + """An override's tensor replaces the stock call in both ops; the stock kernel does not run.""" + sampling, _ = _fake_flashinfer + mine = torch.zeros(2, 8) + override = mock.Mock(return_value=mine) + fi_ops.set_top_p_renorm_override(override) + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is mine + override.assert_called_once_with(probs, top_p) + assert fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is mine + assert override.call_count == 2 + sampling.top_p_renorm_probs.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_flashinfer) -> None: + """An override that returns None leaves the stock call in place; without top_p the override is never asked.""" + sampling, stock = _fake_flashinfer + override = mock.Mock(return_value=None) + fi_ops.set_top_p_renorm_override(override) + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is stock + assert ( + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is stock + ) + assert override.call_count == 2 and sampling.top_p_renorm_probs.call_count == 2 + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, None) + assert override.call_count == 2 + + +def test_the_target_installs_the_override() -> None: + """With TOPP_DET on (the default) and the op built, the target installs ``top_p_renorm_det`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( # noqa: E501 + modeling, + ) + + if not modeling.TOPP_DET: + pytest.skip("TOPP_DET is switched off in this environment") + saved = dict(modeling._TOPP_DET_STATE) + modeling._TOPP_DET_STATE.clear() + try: + modeling._install_topp_det() + assert fi_ops._TOP_P_RENORM_OVERRIDE is entry.top_p_renorm_det + assert modeling._TOPP_DET_STATE["installed"] is True + finally: + fi_ops.set_top_p_renorm_override(None) + modeling._TOPP_DET_STATE.clear() + modeling._TOPP_DET_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_TOPP_DET=1 (read at import): TOPP_DET is off and the target leaves the sampler hook unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as f\n" + "m._install_topp_det()\n" + "print(m.TOPP_DET, f._TOP_P_RENORM_OVERRIDE is None, m._TOPP_DET_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_TOPP_DET="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_n3a.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_n3a.py new file mode 100644 index 000000000000..ad765523c7f6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_n3a.py @@ -0,0 +1,466 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""N3 A2 in this target's DSpark worker: the fused strict acceptance and next-step tokens equal the in-tree path +(N3 P and A1, the verify logits, at the end). + +The worker's two overrides are driven on a bare worker (no model) against the in-tree methods of its base class on +the same inputs: +- eligible steps (K3 / K7, padded drafts, graph-padding dummy rows) are bitwise; +- forced acceptance (fractional, so the RNG pool path runs) over a run that alternates eligible and ineligible + steps keeps the same accepted counts and the same RNG counter as an all-in-tree run; +- the sampler runs exactly once per step; +- the switch and an ineligible step take the in-tree path. +""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import strict_accept +from tensorrt_llm._torch.speculative.dspark import DSparkWorker + +TARGET = "minimax_m31_nvfp4__sm_103__tp4" +dspark = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.dspark" +) + +VOCAB = 200064 + +assert torch.cuda.is_available(), "the N3 A2 worker test requires a CUDA device" + + +def _worker(force: float = 0.0): + w = dspark.M3DSparkWorker.__new__(dspark.M3DSparkWorker) + nn.Module.__init__(w) + w.force_num_accepted_tokens = force + w._force_accept_rng_counter = None + w._force_accept_rng_pool = None + w.calls = 0 + return w + + +def _metadata(ragged: bool = False, graph: bool = False): + return SimpleNamespace(is_ragged_verify=ragged, is_cuda_graph=graph) + + +def _fixed_sampler(tokens: torch.Tensor): + """The in-tree sampler stands in as a fixed function of the logits: the overrides must call it once.""" + + def sample(self, logits, spec_metadata, num_contexts, batch_size): + self.calls += 1 + return tokens + + return sample + + +def _step(n_req: int, draft_len: int, gen: torch.Generator, pad: bool = False, int64: bool = False): + k1 = draft_len + 1 + tokens = torch.randint(0, VOCAB, (n_req * k1,), generator=gen, device="cuda") + tokens = tokens if int64 else tokens.to(torch.int32) + draft = tokens.view(n_req, k1)[:, :draft_len].clone().to(torch.int32) + cut = torch.randint(0, draft_len + 1, (n_req,), generator=gen, device="cuda") + pos = torch.arange(draft_len, device="cuda").unsqueeze(0) + draft = torch.where(pos == cut.unsqueeze(1), (draft + 1) % VOCAB, draft) + if pad: + draft[::2, draft_len // 2 :] = -1 + logits = torch.zeros( + n_req * k1, 8, device="cuda" + ) # shape only: the sampler stand-in ignores the values + return logits, draft, tokens + + +def _in_tree(w, logits, draft, num_contexts, batch_size, md): + return DSparkWorker._sample_and_accept_draft_tokens_base( + w, logits, draft, num_contexts, batch_size, md + ) + + +@pytest.fixture(autouse=True) +def _compiled(): + strict_accept.precompile((3, 7), torch.device("cuda")) + + +@pytest.mark.parametrize( + "n_req,draft_len,pad,int64", + [ + (1, 3, False, False), + (8, 3, False, False), + (12, 3, True, False), + (8, 7, False, False), + (8, 3, False, True), + ], +) +def test_eligible_steps_are_bitwise(monkeypatch, n_req, draft_len, pad, int64): + gen = torch.Generator(device="cuda").manual_seed(3) + logits, draft, tokens = _step(n_req, draft_len, gen, pad=pad, int64=int64) + monkeypatch.setattr(dspark.M3DSparkWorker, "_sample_tokens_for_batch", _fixed_sampler(tokens)) + w, ref = _worker(), _worker() + md = _metadata() + assert w._n3a_eligible(logits, draft, 0, n_req, md) + acc, num = w._sample_and_accept_draft_tokens_base(logits, draft, 0, n_req, md) + racc, rnum = _in_tree(ref, logits, draft, 0, n_req, md) + assert w.calls == 1 and ref.calls == 1 + assert acc.dtype == racc.dtype and num.dtype == rnum.dtype + assert torch.equal(acc, racc) and torch.equal(num, rnum) + + +def test_forced_acceptance_alternating_steps_matches_in_tree(monkeypatch): + """Fractional forced AL (the RNG pool path) over eligible, ragged (ineligible) and graph-warmup steps: the + counts and the RNG counter follow the all-in-tree run step for step.""" + gen = torch.Generator(device="cuda").manual_seed(5) + w, ref = _worker(force=1.6), _worker(force=1.6) + served = 0 + for step in range(12): + logits, draft, tokens = _step(8, 3, gen) + num_contexts, batch_size = 0, 8 + if step % 3 == 2: + # An ineligible step: one context row ahead of the 8 generation requests (the in-tree path runs). + ctx_token = torch.randint(0, VOCAB, (1,), generator=gen, device="cuda").to(tokens.dtype) + tokens = torch.cat([ctx_token, tokens]) + logits = torch.zeros(tokens.numel(), 8, device="cuda") + num_contexts, batch_size = 1, 9 + monkeypatch.setattr( + dspark.M3DSparkWorker, "_sample_tokens_for_batch", _fixed_sampler(tokens) + ) + md = _metadata(graph=(step == 4)) + eligible = w._n3a_eligible(logits, draft, num_contexts, batch_size, md) + assert eligible == (num_contexts == 0), step + served += int(eligible) + acc, num = w._sample_and_accept_draft_tokens_base( + logits, draft, num_contexts, batch_size, md + ) + racc, rnum = _in_tree(ref, logits, draft, num_contexts, batch_size, md) + # A context row defines only column 0 (the in-tree path leaves the rest uninitialized). + assert torch.equal(acc[:num_contexts, 0], racc[:num_contexts, 0]), step + assert torch.equal(acc[num_contexts:], racc[num_contexts:]) and torch.equal(num, rnum), step + if ref._force_accept_rng_counter is not None: + assert torch.equal(w._force_accept_rng_counter, ref._force_accept_rng_counter), step + assert served == 8 + # A ragged verify step is declined (the in-tree ragged path keeps it). + logits, draft, _ = _step(8, 3, gen) + assert not w._n3a_eligible(logits, draft, 0, 8, _metadata(ragged=True)) + + +def test_ineligible_and_switch_take_the_in_tree_path(monkeypatch): + gen = torch.Generator(device="cuda").manual_seed(9) + logits, draft, tokens = _step(4, 3, gen) + w = _worker() + # A context row, a row-count mismatch, too many requests: declined. + assert not w._n3a_eligible(logits, draft, 1, 4, _metadata()) + assert not w._n3a_eligible(logits[:-1], draft, 0, 4, _metadata()) + big_logits, big_draft, _ = _step(strict_accept.max_requests() + 1, 3, gen) + assert not w._n3a_eligible(big_logits, big_draft, 0, big_draft.shape[0], _metadata()) + monkeypatch.setattr(dspark, "N3A_DISABLED", True) + assert not w._n3a_eligible(logits, draft, 0, 4, _metadata()) + + +@pytest.mark.parametrize( + "draft_len,wide,shuffle", [(3, False, False), (3, True, True), (7, False, True)] +) +def test_next_new_tokens_matches_in_tree(draft_len, wide, shuffle): + gen = torch.Generator(device="cuda").manual_seed(13) + n_req = 8 + width = draft_len + 1 + (4 if wide else 0) + accepted = torch.randint( + 0, VOCAB, (n_req, width), generator=gen, device="cuda", dtype=torch.int32 + ) + num = torch.randint(1, draft_len + 2, (n_req,), generator=gen, device="cuda", dtype=torch.int32) + batch_idx = torch.arange(32, device="cuda", dtype=torch.int32) + if shuffle: + batch_idx[:n_req] = torch.randperm(n_req, generator=gen, device="cuda").to(torch.int32) + next_draft = torch.randint( + 0, VOCAB, (n_req, draft_len), generator=gen, device="cuda", dtype=torch.int32 + ) + w = _worker() + out = w._prepare_next_new_tokens(accepted, next_draft, batch_idx, n_req, num) + ref = DSparkWorker._prepare_next_new_tokens(w, accepted, next_draft, batch_idx, n_req, num) + assert out.dtype == ref.dtype and torch.equal(out, ref) + + +# N3 P-gid + P-cat: the worker's target logits equal the logits processor's. + + +@pytest.mark.parametrize("tp", [2, 4, 8]) +def test_rank_major_strided_copy_equals_vocab_concat_and_cast(tp): + """The all-gather along dim 0 + one strided copy writes what the vocab concat + .float() writes.""" + gen = torch.Generator(device="cuda").manual_seed(tp) + rows, shard = 32, 25008 + shards = [ + torch.randn(rows, shard, generator=gen, device="cuda").to(torch.bfloat16) for _ in range(tp) + ] + gathered = torch.cat(shards, dim=0) # what the dim-0 all-gather returns, rank-major + out = torch.empty((rows, tp * shard), dtype=torch.float32, device="cuda") + out.view(rows, tp, shard).copy_(gathered.view(tp, rows, shard).transpose(0, 1)) + assert torch.equal(out, torch.cat(shards, dim=-1).float()) + + +def _lm_head(vocab=50016, hidden=6144): + from tensorrt_llm._torch.modules.embedding import LMHead + from tensorrt_llm._torch.modules.linear import TensorParallelMode + from tensorrt_llm.mapping import Mapping + + head = LMHead( + vocab, + hidden, + dtype=torch.bfloat16, + mapping=Mapping(world_size=1, tp_size=1, rank=0), + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=True, + ).cuda() + with torch.no_grad(): + head.weight.copy_( + torch.randn_like(head.weight, dtype=torch.float32).to(torch.bfloat16) * 0.02 + ) + return head + + +def test_target_logits_equal_the_processor_at_tp1(): + from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor + + head = _lm_head() + rows = 32 + hidden = torch.randn(rows, 6144, device="cuda").to(torch.bfloat16) + md = SimpleNamespace( + is_ragged_verify=False, gather_ids=torch.arange(rows, device="cuda", dtype=torch.int32) + ) + attn = SimpleNamespace(num_contexts=0) + out = _worker().target_logits(hidden, attn, md, head) + ref = LogitsProcessor().forward(hidden[md.gather_ids], head, attn, True) + assert out is not None and out.dtype == ref.dtype and torch.equal(out, ref) + + +def test_target_logits_decline_to_the_processor(monkeypatch): + head = _lm_head(vocab=1024, hidden=256) + hidden = torch.randn(8, 256, device="cuda").to(torch.bfloat16) + ids = torch.arange(8, device="cuda", dtype=torch.int32) + w = _worker() + # A context row, a gather that is not every row, a ragged verify, a padded head: the processor runs. + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=1), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids[:4]), + head, + ) + is None + ) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=True, gather_ids=ids), + head, + ) + is None + ) + monkeypatch.setattr(head, "padding_size", 3) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + monkeypatch.setattr(head, "padding_size", 0) + monkeypatch.setattr(dspark, "N3P_DISABLED", True) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + + +# N3 A1: the verify logits' all-gather + concat + cast in one launch (catalog/comm/allgather_cast) when set up. + + +def _a1_step(monkeypatch, rows: int, compiled: bool = True): + """A TP4-shaped call: the head reports tp 4, A1 is set up, the op and the NCCL all-gather are recorded.""" + head = _lm_head(vocab=1024, hidden=256) + monkeypatch.setattr(head, "tp_size", 4) + calls = {"a1": 0, "nccl": 0} + sentinel = torch.full((rows, 4 * 1024), 7.0, device="cuda") + + def fake_a1(shard, **ws): + assert ws == {"ws": "a1"} and shard.shape == (rows, 1024) + calls["a1"] += 1 + return sentinel + + def fake_allgather(shard, mapping, dim=0): + calls["nccl"] += 1 + return torch.cat([shard] * 4, dim=0) + + monkeypatch.setattr(dspark._allgather_cast, "allgather_cast", fake_a1) + monkeypatch.setattr(dspark._allgather_cast, "is_compiled", lambda *a, **k: compiled) + monkeypatch.setattr(dspark, "allgather", fake_allgather) + w = _worker() + w._a1 = {"ws": "a1"} + hidden = torch.randn(rows, 256, device="cuda").to(torch.bfloat16) + md = SimpleNamespace( + is_ragged_verify=False, gather_ids=torch.arange(rows, device="cuda", dtype=torch.int32) + ) + out = w.target_logits(hidden, SimpleNamespace(num_contexts=0), md, head) + return out, sentinel, calls + + +@pytest.mark.parametrize("rows", [4, 32, 64]) +def test_target_logits_take_a1_when_set_up(monkeypatch, rows): + out, sentinel, calls = _a1_step(monkeypatch, rows) + assert out is sentinel and calls == {"a1": 1, "nccl": 0} + + +@pytest.mark.parametrize("rows,compiled", [(68, True), (4, False)]) +def test_target_logits_fall_back_to_nccl_past_a1(monkeypatch, rows, compiled): + """More rows than the workspace holds, or a shape never compiled: the NCCL all-gather + strided copy.""" + out, _, calls = _a1_step(monkeypatch, rows, compiled=compiled) + assert ( + calls == {"a1": 0, "nccl": 1} + and out.shape == (rows, 4 * 1024) + and out.dtype == torch.float32 + ) + + +def _a1_prebuild_worker(monkeypatch, tp: int = 4, draft_len: int = 3): + w = _worker() + w.mapping = SimpleNamespace(tp_size=tp, tp_rank=0, enable_attention_dp=False) + w.spec_config = SimpleNamespace(max_draft_len=draft_len) # max_draft_len reads it + # Eligibility needs every SM up to the workspace's grid; any CUDA device passes with a 1-CTA workspace grid. + monkeypatch.setattr(dspark._allgather_cast, "MAX_GRID", 1) + monkeypatch.setattr(dspark.M3DSparkWorker, "_a1_all_ok", lambda self, local: local) + return w + + +def test_prebuild_a1_sets_up_k3_rows(monkeypatch): + head = _lm_head(vocab=1024, hidden=256) + seen = {} + + def fake_precompile(mapping, cols, rows, device, group=None, agree=None): + seen.update(cols=cols, rows=rows, agreed=agree(True)) + return True + + ws = {"ws_uc": torch.zeros(4, dtype=torch.int32), "rank": 0} + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr( + dspark._allgather_cast, "workspace_args", lambda mapping, cols, group=None: ws + ) + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is ws + # K3 verify rows of batches 1..16, all within the workspace's 64 rows. + assert seen == {"cols": 1024, "rows": tuple(range(4, 65, 4)), "agreed": True} + + +@pytest.mark.parametrize("draft_len", [3, 7]) +def test_prebuild_a1_sets_up_the_verify_rows_of_k3_and_k7(monkeypatch, draft_len): + """K3: rows 4..64 (batches 1..16); K7, the headline: rows 8..64 (batches 1..8).""" + head = _lm_head(vocab=1024, hidden=256) + seen = {} + + def fake_precompile(mapping, cols, rows, device, group=None, agree=None): + seen.update(rows=rows, agreed=agree(True)) + return True + + ws = {"ws_uc": torch.zeros(4, dtype=torch.int32), "rank": 0} + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr( + dspark._allgather_cast, "workspace_args", lambda mapping, cols, group=None: ws + ) + assert _a1_prebuild_worker(monkeypatch, draft_len=draft_len)._prebuild_a1(head) is ws + k1 = draft_len + 1 + assert seen == {"rows": tuple(range(k1, 65, k1)), "agreed": True} + + +def test_prebuild_a1_off(monkeypatch): + """The switch, another draft length, one rank, a rank that is not eligible, or a compile that failed on any rank: + no A1, no workspace.""" + head = _lm_head(vocab=1024, hidden=256) + built = {"n": 0} + + def fake_precompile(*a, **k): + built["n"] += 1 + return False # a failed compile somewhere + + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr(dspark, "A1_K7_DISABLED", True) + assert _a1_prebuild_worker(monkeypatch, draft_len=7)._prebuild_a1(head) is None + monkeypatch.setattr(dspark, "A1_K7_DISABLED", False) + assert ( + _a1_prebuild_worker(monkeypatch, draft_len=64)._prebuild_a1(head) is None + ) # 65 rows at batch 1 + assert _a1_prebuild_worker(monkeypatch, tp=1)._prebuild_a1(head) is None + assert built["n"] == 0 + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is None + assert built["n"] == 1 + # Another rank not eligible: off here too, nothing compiled. + w = _a1_prebuild_worker(monkeypatch) + monkeypatch.setattr(dspark.M3DSparkWorker, "_a1_all_ok", lambda self, local: False) + assert w._prebuild_a1(head) is None and built["n"] == 1 + monkeypatch.setattr(dspark, "A1_DISABLED", True) + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is None + assert built["n"] == 1 + + +# N3A-KV: the draft-forward KV bookkeeping in one launch, through the DFlash hook (bitwise to the stock lines). + + +def _kv_step(nc: int, bs: int, gen: torch.Generator): + num = torch.randint(1, 5, (bs + 2,), generator=gen, device="cuda", dtype=torch.int32) + kv = torch.randint(0, 1 << 20, (bs + 2,), generator=gen, device="cuda", dtype=torch.int32) + return num, kv + + +def _kv_worker(fused: bool): + w = _worker() + w._fused_kv_rewind = None + if fused: + strict_accept.precompile_kv_rewind(torch.device("cuda")) + w._n3a_kv_buf = torch.empty(strict_accept.max_requests(), dtype=torch.int32, device="cuda") + w._fused_kv_rewind = w._n3a_kv_rewind + return w + + +@pytest.mark.parametrize("nc,bs", [(0, 1), (0, 8), (2, 6), (3, 3)]) +def test_kv_rewind_hook_equals_the_stock_lines(nc, bs): + gen = torch.Generator(device="cuda").manual_seed(5) + num, kv = _kv_step(nc, bs, gen) + outs = [] + for fused in (False, True): + w = _kv_worker(fused) + attn = SimpleNamespace(kv_lens_cuda=kv.clone(), update_for_spec_dec=lambda: None) + w._prepare_kv_for_draft_forward(attn, num, nc, bs) + outs.append( + ( + w._kv_rewind_amount, + attn.kv_lens_cuda, + w._kv_rewind_nc, + w._kv_rewind_bs, + w._kv_rewind_pending, + ) + ) + (r0, k0, *rest0), (r1, k1, *rest1) = outs + assert torch.equal(k0, k1) and rest0 == rest1 + assert r0.dtype == r1.dtype and r0.shape == r1.shape and torch.equal(r0, r1) + + +def test_kv_rewind_hook_declines_unsupported_operands(): + w = _kv_worker(True) + num64 = torch.ones(4, dtype=torch.int64, device="cuda") # not the buffer's dtype + kv = torch.zeros(4, dtype=torch.int32, device="cuda") + assert w._n3a_kv_rewind(num64, kv, 0, 4) is None and torch.equal(kv, torch.zeros_like(kv)) + w._n3a_kv_buf = None + assert w._n3a_kv_rewind(torch.ones(4, dtype=torch.int32, device="cuda"), kv, 0, 4) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_s4.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_s4.py new file mode 100644 index 000000000000..fe633ff9a2c1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_s4.py @@ -0,0 +1,347 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp4 target's worker-side routing of the one-launch Markov chain (``dspark_s4.S4WorkerMixin``), without GPUs +or a model: which steps it serves, that the kill switch sets nothing up, that a declined step is the in-tree path, that +the step's logits processor only takes the block rows, and that the chain's tokens replace the in-tree sampling for +that one call. The chain itself is the catalog's ``spec_decode/dspark_markov_chain``, certified by +``spec_decode/_dspark_markov_chain_op_matrix.py``; the in-model equality by the call-site gate.""" + +import importlib +import os +import subprocess +import sys +from types import SimpleNamespace + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( # noqa: E501 + dspark_s4 as w1, +) + + +class _Base: + def __init__(self): + self.calls = [] + + def _forward_impl(self, *args): + self.calls.append("forward") + return "in-tree" + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + self.calls.append("refine") + return gen_logits + + def sample_draft_tokens(self, logits, spec_metadata, batch_size, **kwargs): + self.calls.append("sample") + return torch.full(logits.shape[:2], -1, dtype=torch.int32) + + def _apply_kv_rewind_after_draft(self, attn_metadata, spec_metadata): + self.calls.append("stock_rewind") + self._kv_rewind_pending = False + + +class _Worker(w1.S4WorkerMixin, _Base): + pass + + +def _state(block=3, shard=8, rows=8): + st = w1._S4State(None, rows, shard, 0, None, None, block, {}) + for b in range(1, w1.MAX_FAST_BATCH + 1): + st.bufs[(b, block)] = torch.full((b, block, rows), float("-inf")) + return st + + +def _meta(advanced=False): + return SimpleNamespace(wants_advanced_draft_sampling=advanced) + + +def test_serves_only_eligible_steps(monkeypatch): + wk = _Worker() + draft = SimpleNamespace(lilicorr=None, has_candidate_selector=False) + assert not wk._s4_serves(1, _meta(), draft) # not set up + wk._s4 = _state() + assert wk._s4_serves(1, _meta(), draft) + assert wk._s4_serves(w1.MAX_FAST_BATCH, _meta(), draft) + assert not wk._s4_serves(0, _meta(), draft) + assert not wk._s4_serves(w1.MAX_FAST_BATCH + 1, _meta(), draft) + assert not wk._s4_serves(1, _meta(advanced=True), draft) + wk._d2t = torch.zeros(1) + assert not wk._s4_serves(1, _meta(), draft) + + +def test_kill_switch_sets_nothing_up(monkeypatch): + wk = _Worker() + wk._s4 = _state() + monkeypatch.setattr(w1, "_DISABLED", True) + assert not wk.prebuild_s4(None, SimpleNamespace(has_markov_head=True)) + assert wk._s4 is None + assert not wk._s4_serves( + 1, _meta(), SimpleNamespace(lilicorr=None, has_candidate_selector=False) + ) + + +def test_declined_step_is_the_in_tree_path(): + wk = _Worker() + attn = SimpleNamespace(num_seqs=12, num_contexts=0) # above MAX_FAST_BATCH + draft = SimpleNamespace(lilicorr=None, has_candidate_selector=False, logits_processor="proc") + wk._s4 = _state() + assert wk._forward_impl(None, None, None, None, attn, _meta(), draft) == "in-tree" + assert draft.logits_processor == "proc" + logits = torch.zeros(2, 3, 16) # full-vocab width: not this step's shard + assert wk._refine_block_logits(draft, logits, {}, None) is logits + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() + assert wk.calls == ["forward", "refine", "sample"] + + +def test_sharded_logits_processor_takes_only_block_rows(): + st = _state(block=3, shard=8, rows=12) + inner_calls = [] + + def inner(h, head, meta, ctx): + inner_calls.append(h.shape) + return "inner" + + head = SimpleNamespace(apply_linear=lambda h, b: h[:, :8] * 2) + proc = w1._ShardedDraftLogits(inner, st, batch=2) + h = torch.arange(6 * 10, dtype=torch.float32).view(6, 10) + out = proc(h, head, None, True) + assert torch.equal(out, h[:, :8] * 2) + buf = st.buffer(2) + assert torch.equal(buf[:, :, :8], (h[:, :8] * 2).view(2, 3, 8)) + assert torch.isinf(buf[:, :, 8:]).all() + assert proc(torch.zeros(5, 10), head, None, True) == "inner" and inner_calls == [ + torch.Size([5, 10]) + ] + + +def test_chain_tokens_replace_sampling_once(monkeypatch): + wk = _Worker() + st = _state(block=3, shard=8, rows=8) + wk._s4_step = st + toks = torch.tensor([[1, 2, 3], [4, 5, 6]]) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, + dspark_markov_chain=lambda *a, **k: toks.int(), + ll_exchange_engaged=lambda batch: False)) # fmt: skip + logits = torch.zeros(2, 3, 8) + assert wk._refine_block_logits(None, logits, {"first_prev_tokens": torch.zeros(2, dtype=torch.long)}, None) \ + is logits # fmt: skip + out = wk.sample_draft_tokens(logits, None, 2) + assert out.dtype == torch.int32 and torch.equal(out, toks.int()) + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() # used once + assert "refine" not in wk.calls + + +def test_unsupported_shapes_fall_back_on_the_shards(monkeypatch): + wk = _Worker() + wk._s4_step = _state(block=3, shard=8, rows=8) + + def _never(*a, **k): + raise AssertionError("an unsupported step must not reach the op") + + monkeypatch.setattr( + w1, "chain", SimpleNamespace(supports=lambda *a: False, dspark_markov_chain=_never) + ) + logits = torch.zeros(2, 3, 8) + wk._refine_block_logits( + None, logits, {"first_prev_tokens": torch.zeros(2, dtype=torch.long)}, None + ) + assert wk.calls == ["refine"] + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() + + +def test_switch_reads_any_value_but_empty_or_zero(monkeypatch): + """Read at import, like the targets' m31_switches: "true" turns S4 off, not only "1".""" + try: + for value, off in ( + ("1", True), + ("true", True), + ("yes", True), + ("0", False), + ("", False), + (" 0 ", False), + ): + monkeypatch.setenv(w1.DISABLE_ENV, value) + importlib.reload(w1) + assert w1.disabled() is off, value + finally: + monkeypatch.delenv(w1.DISABLE_ENV, raising=False) + importlib.reload(w1) + assert not w1.disabled() + + +def test_markov_ll_engagement_logged_once_per_rank(monkeypatch): + """MARKOV_LL's "[M3 dispatch]" line: once per rank, on the first served step that took the LL kernel.""" + wk = _Worker() + wk.mapping = SimpleNamespace(tp_rank=3) + wk._s4_step = _state(block=3, shard=8, rows=8) + lines = [] + monkeypatch.setattr(w1.logger, "info", lambda msg, *a, **k: lines.append(msg)) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, + dspark_markov_chain=lambda *a, **k: torch.zeros( + a[0].shape[0], 3, dtype=torch.int32), + ll_exchange_engaged=lambda batch: batch == 1)) # fmt: skip + inputs2 = {"first_prev_tokens": torch.zeros(2, dtype=torch.long)} + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs2, None) # batch 2: not the LL kernel + assert not [m for m in lines if "MARKOV_LL" in m] + inputs1 = {"first_prev_tokens": torch.zeros(1, dtype=torch.long)} + for _ in range(3): + wk._refine_block_logits(None, torch.zeros(1, 3, 8), inputs1, None) + ll = [m for m in lines if "MARKOV_LL" in m] + assert ll == [ + "[M3 dispatch] MARKOV_LL: k3_markov LL flag-in-payload exchange (batch 1) on rank 3" + ], ll + + +def _markov_ll_in_subprocess(value): + env = {k: v for k, v in os.environ.items() if k != "TRTLLM_M31_DISABLE_MARKOV_LL"} + if value is not None: + env["TRTLLM_M31_DISABLE_MARKOV_LL"] = value + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_103__tp4 import dspark_s4 as m; print('MARKOV_LL=' + str(m.MARKOV_LL))" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=600 + ) + assert out.returncode == 0, out.stderr + return [ln for ln in out.stdout.splitlines() if ln.startswith("MARKOV_LL=")][-1] + + +def test_markov_ll_kill_switch_subprocess(): + """MARKOV_LL is on by default; TRTLLM_M31_DISABLE_MARKOV_LL=1 (read at import) turns it off.""" + assert _markov_ll_in_subprocess(None) == "MARKOV_LL=True" + assert _markov_ll_in_subprocess("0") == "MARKOV_LL=True" + assert _markov_ll_in_subprocess("1") == "MARKOV_LL=False" + + +def _rewind_worker(monkeypatch, batch=2, nc=1, on=True): + """A worker in a served step with a pending KV-length rewind of ``batch`` rows from ``nc``, and a chain fake that + records which launch ran.""" + wk = _Worker() + wk.mapping = SimpleNamespace(tp_rank=5) + wk._s4_step = _state(block=3, shard=8, rows=8) + wk._s4_step_attn = SimpleNamespace(kv_lens_cuda=torch.zeros(nc + batch + 1, dtype=torch.int32)) + wk._kv_rewind_pending = True + wk._kv_rewind_amount = torch.zeros(batch, dtype=torch.int32) + wk._kv_rewind_nc, wk._kv_rewind_bs = nc, nc + batch + launches, lines = [], [] + toks = torch.zeros(batch, 3, dtype=torch.int32) + + def plain(*a, **k): + launches.append("chain") + return toks + + def with_rewind(base, first_prev, w1_, w2s, offset, kv_lens, amount, first, **k): + launches.append( + ( + "rewind", + kv_lens is wk._s4_step_attn.kv_lens_cuda, + amount is wk._kv_rewind_amount, + first, + ) + ) + return toks + + monkeypatch.setattr(w1, "TAILBK_SPEC", on) + monkeypatch.setattr(w1.logger, "info", lambda msg, *a, **k: lines.append(msg)) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, supports_rewind=lambda *a: True, + dspark_markov_chain=plain, + dspark_markov_chain_rewind=with_rewind, + ll_exchange_engaged=lambda b: False)) # fmt: skip + return wk, launches, lines, {"first_prev_tokens": torch.zeros(batch, dtype=torch.long)} + + +def test_tailbk_h_rewind_in_chain_launch(monkeypatch): + """TAILBK_SPEC H: a real step's chain launch applies the pending rewind (this step's kv_lens, amounts, first + row) and the stock rewind lines are skipped for that step; the engagement line is logged once per rank.""" + wk, launches, lines, inputs = _rewind_worker(monkeypatch) + meta = SimpleNamespace(is_cuda_graph=False) + for _ in range(2): + wk._kv_rewind_pending = True + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert not wk._kv_rewind_pending and not wk._tailbk_rewound + assert launches == [("rewind", True, True, 1)] * 2, launches + assert "stock_rewind" not in wk.calls + assert [m for m in lines if "TAILBK_SPEC H" in m] == [ + "[M3 dispatch] TAILBK_SPEC H: KV-length rewind in the Markov chain launch on rank 5" + ] + + +def test_tailbk_h_warmup_keeps_stock_path(monkeypatch): + """The eager warmup of a graph batch (is_cuda_graph, not capturing): the stock method skips the rewind there, so + the chain must not apply it either.""" + wk, launches, _, inputs = _rewind_worker(monkeypatch) + meta = SimpleNamespace(is_cuda_graph=True) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def test_tailbk_h_off_or_mismatch_keeps_stock_path(monkeypatch): + """Kill switch off, no pending rewind, or amounts that are not this chain call's rows: the stock rewind runs.""" + meta = SimpleNamespace(is_cuda_graph=False) + wk, launches, _, inputs = _rewind_worker(monkeypatch, on=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + wk, launches, _, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_pending = False + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert launches == ["chain"] + wk, launches, _, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_amount = torch.zeros(3, dtype=torch.int32) # not the 2 rows of this call + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def test_tailbk_h_unsupported_operands_keep_stock_path(monkeypatch): + """supports_rewind False (e.g. int64 amounts): the plain chain launch and the stock rewind.""" + wk, launches, _, inputs = _rewind_worker(monkeypatch) + monkeypatch.setattr(w1.chain, "supports_rewind", lambda *a: False) + meta = SimpleNamespace(is_cuda_graph=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def _tailbk_in_subprocess(value): + env = {k: v for k, v in os.environ.items() if k != "TRTLLM_M31_DISABLE_TAILBK_SPEC"} + if value is not None: + env["TRTLLM_M31_DISABLE_TAILBK_SPEC"] = value + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_103__tp4 import dspark_s4 as m; print('TAILBK_SPEC=' + str(m.TAILBK_SPEC))" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=600 + ) + assert out.returncode == 0, out.stderr + return [ln for ln in out.stdout.splitlines() if ln.startswith("TAILBK_SPEC=")][-1] + + +def test_tailbk_spec_kill_switch_subprocess(): + """TAILBK_SPEC is on by default; TRTLLM_M31_DISABLE_TAILBK_SPEC=1 (read at import) turns it off.""" + assert _tailbk_in_subprocess(None) == "TAILBK_SPEC=True" + assert _tailbk_in_subprocess("0") == "TAILBK_SPEC=True" + assert _tailbk_in_subprocess("1") == "TAILBK_SPEC=False" + + +def test_tailbk_c_engagement_logged_once_per_rank(monkeypatch): + """TAILBK_SPEC C's "[M3 dispatch]" line: once per rank, only after a chain call took the cached inputs.""" + wk, launches, lines, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_pending = False # the plain chain launch + taken = [False] + monkeypatch.setattr(w1.chain, "cached_inputs_taken", lambda: taken[0], raising=False) + meta = SimpleNamespace(is_cuda_graph=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert not [m for m in lines if "TAILBK_SPEC C" in m] + taken[0] = True + for _ in range(3): + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._kv_rewind_pending = True # and through the rewind launch + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert [m for m in lines if "TAILBK_SPEC C" in m] == [ + "[M3 dispatch] TAILBK_SPEC C: chain accepted inputs from precompile cache (taken) on rank 5" + ] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_w0.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_w0.py new file mode 100644 index 000000000000..e1c9f72e7fcb --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m31_dspark_w0.py @@ -0,0 +1,171 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-model DSpark on the tp4 target, phase W0 (``dspark.py``): the shell's calls on the DSpark worker take no +fused path, the capture is the in-tree DFlash capture, the drafter's borrowed embedding is the target's exact gather, +and the FP8 draft-KV override (an Eagle3 need) is never applied to a DSpark run, so the drafter's context pool keeps +the projection dtype (BF16) and binds to the separate draft KV cache. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( # noqa: E501 + dspark, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops import ( # noqa: E501 + m3_target_head, +) +from tensorrt_llm.quantization.mode import QuantAlgo + + +def _spec( + *, dspark_mode: bool, eagle3: bool = False, one_engine: bool = True, embedded: bool = False +): + mode = SimpleNamespace( + use_one_engine=lambda: one_engine, + is_dspark=lambda: dspark_mode, + is_eagle3_one_model=lambda: eagle3, + ) + return SimpleNamespace(spec_dec_mode=mode, draft_is_embedded_in_target=embedded) + + +def _worker() -> dspark.M3DSparkWorker: + # The shell-facing calls read no worker state, so the worker is not constructed (that needs a mapping and a + # DSpark spec config). + return dspark.M3DSparkWorker.__new__(dspark.M3DSparkWorker) + + +@pytest.mark.parametrize( + "spec, expected", + [ + (_spec(dspark_mode=True), True), + (_spec(dspark_mode=True, embedded=True), False), + (_spec(dspark_mode=True, one_engine=False), False), + (_spec(dspark_mode=False, eagle3=True), False), + (None, False), + ], +) +def test_standalone_dspark_detection(spec, expected): + assert dspark.is_standalone_dspark(spec) is expected + + +def test_fp8_draft_kv_override_is_eagle3_only(): + assert dspark.draft_kv_override_applies(_spec(dspark_mode=False, eagle3=True)) + assert dspark.draft_kv_override_applies(None) + assert not dspark.draft_kv_override_applies(_spec(dspark_mode=True)) + + +@pytest.mark.parametrize("target_algo", [QuantAlgo.NVFP4, QuantAlgo.FP8, None]) +def test_drafter_kv_pool_keeps_the_drafter_dtype(target_algo): + from tensorrt_llm.models.modeling_utils import QuantConfig + + whole, layer = ( + QuantConfig(kv_cache_quant_algo=target_algo), + QuantConfig(kv_cache_quant_algo=target_algo), + ) + _ = whole.quant_mode # cached before the drop, as a loaded config may have it + cfg = SimpleNamespace(quant_config=whole, quant_config_dict={"model.layers.0": layer}) + dspark.keep_drafter_kv_dtype(cfg) + assert whole.kv_cache_quant_algo is None and layer.kv_cache_quant_algo is None + assert not whole.quant_mode.has_fp8_kv_cache() and not whole.quant_mode.has_fp4_kv_cache() + + +def test_shell_calls_take_no_fused_path(monkeypatch): + def _forbidden(*args, **kwargs): + raise AssertionError("W0 must never reach the coded target head") + + monkeypatch.setattr(m3_target_head, "prepare_head", _forbidden) + monkeypatch.setattr(m3_target_head, "logits", _forbidden) + worker = _worker() + assert worker.tail_eligible(None, None, None) is False + assert worker.arm_capture_norm(None, None, None, None) is None + assert worker.prebuild(None, None) is None + + +def test_capture_is_the_in_tree_capture(): + calls = [] + meta = SimpleNamespace(maybe_capture_hidden_states=lambda *a: calls.append(a)) + hidden, residual = object(), object() + _worker().capture_hidden_states(meta, 17, hidden, residual) + _worker().capture_hidden_states(meta, 59, hidden) + assert calls == [(17, hidden, residual), (59, hidden, None)] + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +def test_embed_tokens_is_the_target_gather(): + gen = torch.Generator(device="cuda").manual_seed(31) + table = torch.randn(1000, 64, generator=gen, device="cuda").to(torch.bfloat16) + core = SimpleNamespace(w={"embed": table}) + embed = dspark.M3EmbedTokens(core) + ids = torch.tensor([0, 5, 999, 200, -1], dtype=torch.long, device="cuda") + out = embed(ids) + ref = table[ids.clamp(min=0)] * (ids >= 0).unsqueeze(-1).to(table.dtype) + assert embed.weight is table + assert torch.equal(out, ref) + assert not list(embed.parameters()), ( + "the table belongs to the target, not to the borrowed module" + ) + + +def test_borrowed_embedding_owns_no_parameters(): + # The weights walker asserts every shell parameter is fed: the borrowed module must add none. + model = torch.nn.Module() + model.w = torch.nn.ParameterDict( + {"embed": torch.nn.Parameter(torch.zeros(8, 4), requires_grad=False)} + ) + before = {n for n, _ in model.named_parameters()} + model.embed_tokens = dspark.M3EmbedTokens(model) + assert {n for n, _ in model.named_parameters()} == before == {"w.embed"} + assert model.embed_tokens.weight is model.w["embed"] + + +class _FakeDrafter: + def __init__(self, borrows: bool = True) -> None: + self.loaded = None + self.borrows = borrows + self.lm_head = None + self.draft_model_full = SimpleNamespace(model=SimpleNamespace(embed_tokens=None)) + + def load_weights(self, weights, weight_mapper=None): + self.loaded = dict(weights) + + def load_weights_from_target_model(self, target): + if self.borrows: + self.lm_head = target.lm_head + self.draft_model_full.model.embed_tokens = target.model.embed_tokens + + +def _shell(drafter): + return SimpleNamespace( + draft_model=drafter, + lm_head=object(), + model=SimpleNamespace(embed_tokens=object()), + named_parameters=lambda: iter([("lm_head.weight", None), ("model.w.embed", None)]), + ) + + +@pytest.mark.parametrize( + "name", + ["model.embed_tokens.weight", "language_model.model.embed_tokens.weight", "lm_head.weight"], +) +def test_draft_load_refuses_borrowed_tensors(name): + shell = _shell(_FakeDrafter()) + with pytest.raises(ValueError, match="lends it"): + dspark.load_dspark_draft_weights(shell, {name: torch.zeros(1), "fc.weight": torch.zeros(1)}) + assert shell.draft_model.loaded is None + + +def test_draft_load_borrows_target_modules(): + shell = _shell(_FakeDrafter()) + weights = { + "fc.weight": torch.zeros(1), + "markov_head.markov_w1.weight": torch.zeros(1), + "confidence_head.proj.weight": torch.zeros(1), + } + dspark.load_dspark_draft_weights(shell, weights) + assert shell.draft_model.loaded == weights + assert shell.draft_model.lm_head is shell.lm_head + with pytest.raises(RuntimeError, match="does not borrow"): + dspark.load_dspark_draft_weights(_shell(_FakeDrafter(borrows=False)), weights) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m3_exact_numerics.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m3_exact_numerics.py new file mode 100644 index 000000000000..7279bbc33f9e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/test_modeling_v2_m3_exact_numerics.py @@ -0,0 +1,289 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp4 target's speculative step against the in-tree (stock) Eagle3 numerics, at the real TP4 shapes. + +* Verify: the target logits are the in-tree BF16 LM head's own shard GEMM, and the fused shard argmax + acceptance + gives the in-tree greedy acceptance of the gathered logits, bit for bit, on every rank (peers' keys are the real + per-shard maxima, computed by the same kernel). +* Drafter: the worker runs the in-tree drafter's modules and weights; its own drafter arithmetic is the first pass's + fc_norm (chunk_norm, or capture_norm folded into the capture), held here to the in-tree ``apply_eagle3_fc`` RMSNorm + modules within one bf16 ulp (its effect on the fc output and on the full-vocabulary BF16 head's draft tokens is + measured too), and the fused draft loop's draft input and BF16 head, held to the in-tree modules and greedy draft + sampling by their own tests (test_modeling_v2_m3_draft_input.py, test_modeling_v2_m3_draft_step.py). + +Synthetic weights: hidden 6144, vocabulary 200064 (4 x 50016 shards), 3 captured layers, draft length 3. +""" + +import importlib.util +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops import ( # noqa: E501 + m3_spec_tail, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.spec.m3_shard_accept import ( # noqa: E501 + SHARD_ROWS, + make_keys, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.spec.m3_shard_accept import ( # noqa: E501 + m3_shard_accept as shard_accept_op, +) +from tensorrt_llm._torch.models.modeling_speculative import Eagle3ForCausalLM +from tensorrt_llm._torch.modules.embedding import LMHead +from tensorrt_llm._torch.modules.linear import Linear +from tensorrt_llm._torch.modules.rms_norm import RMSNorm +from tensorrt_llm._torch.speculative.eagle3 import Eagle3OneModelWorker + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 + EPS, + HIDDEN, + TARGET_SHARD, + VOCAB, + WORLD, + assert_bf16_close, + craft_drafts, + gen, + head_workspace, + num_sms, + randn, + seed_of, +) + +assert torch.cuda.is_available(), "the exact-numerics checks require a CUDA device" + +DRAFT_LEN = 3 +CAPTURED_LAYERS = 3 +#: Largest share of fc_norm outputs one bf16 ulp off the in-tree RMSNorm (fp32 statistic summed in another order). +NORM_ULP_FRAC = 1e-3 + + +def _stock_accept(logits: torch.Tensor, draft: torch.Tensor): + """The in-tree greedy strict acceptance (``SpecWorkerBase._sample_and_accept_draft_tokens_base``) of the gathered + float logits of a decode-only batch, no forcing.""" + md = SimpleNamespace(is_all_greedy_sample=True, is_ragged_verify=False, is_cuda_graph=False) + worker = SimpleNamespace(force_num_accepted_tokens=0.0) + worker._sample_tokens_for_batch = lambda *a: Eagle3OneModelWorker._sample_tokens_for_batch( + worker, *a + ) + worker._apply_force_accepted_tokens = lambda *a, **k: ( + Eagle3OneModelWorker._apply_force_accepted_tokens(worker, *a, **k) + ) + nreq = draft.shape[0] + return Eagle3OneModelWorker._sample_and_accept_draft_tokens_base( + worker, logits, draft, 0, nreq, md + ) + + +def _lm_head_shard(weight: torch.Tensor, rank: int) -> LMHead: + """Rank ``rank``'s local part of the target's vocab-sharded BF16 LM head at TP4: an LM head over that rank's + 50016 rows (the TP4 module's local GEMM is this module's; its all-gather is the caller's).""" + head = LMHead(num_embeddings=TARGET_SHARD, embedding_dim=HIDDEN, dtype=torch.bfloat16) + head.weight.data = weight[rank * TARGET_SHARD : (rank + 1) * TARGET_SHARD].contiguous() + return head + + +def _run_shard_accept(logits, rank, world, draft, keys_of_peers=None, call=0): + """The fused verify of one rank: its shard logits, the peers' keys planted in its slots.""" + nreq = draft.shape[0] + rows = logits.shape[0] + uc, keys, counters = head_workspace(world, call=call, slot_rows=SHARD_ROWS) + half = call & 1 + for peer, peer_keys in (keys_of_peers or {}).items(): + for t in range(rows): + uc[(half * SHARD_ROWS + t) * world + peer] = peer_keys[t] + acc = torch.full((nreq, DRAFT_LEN + 1), -7, dtype=torch.int32, device="cuda") + num = torch.full((nreq,), -7, dtype=torch.int32, device="cuda") + gather = torch.full((nreq,), -7, dtype=torch.int64, device="cuda") + next_new = torch.full((nreq, DRAFT_LEN + 1), -7, dtype=torch.int32, device="cuda") + batch_idx = torch.arange(nreq, dtype=torch.int32, device="cuda") + shard_accept_op( + logits, rank * TARGET_SHARD if world > 1 else 0, keys, counters, uc, uc.data_ptr(), rank, world, + draft, batch_idx, acc, num, gather, 0, next_new, num_sms(), None, + ) # fmt: skip + torch.cuda.synchronize() + return acc, num, gather, next_new + + +def _local_keys(logits: torch.Tensor, rank: int) -> torch.Tensor: + """What rank ``rank`` pushes: the key of its own shard maximum per row, found by the same kernel at world 1.""" + rows = logits.shape[0] + nreq = rows // (DRAFT_LEN + 1) + dummy = torch.full((nreq, DRAFT_LEN), -1, dtype=torch.int32, device="cuda") + acc, _, _, _ = _run_shard_accept(logits, 0, 1, dummy) + local = acc.view(-1).to(torch.int64) + value = logits.float().gather(1, local[:, None]).squeeze(1) + return make_keys(value, local + rank * TARGET_SHARD) + + +@pytest.mark.parametrize( + "nreq,ties", [(1, False), (2, False), (4, False), (8, True), (12, False)], ids=str +) +def test_fused_verify_is_the_stock_greedy_acceptance(nreq, ties) -> None: + """Target logits (in-tree BF16 head) and verify outputs bit-identical to the in-tree path, on all four ranks.""" + g = gen(seed_of(f"verify {nreq} {ties}")) + rows = nreq * (DRAFT_LEN + 1) + x = randn((rows, HIDDEN), g, 1.0) + weight = randn((VOCAB, HIDDEN), g, 0.02) + if ties: + # Two equal head rows on ranks 1 and 3 that win every row: the lower global id must be the token. + win = (x.float().sum(0) / x.float().sum(0).norm() * 4.0).to(torch.bfloat16) + weight[TARGET_SHARD + 77] = win + weight[3 * TARGET_SHARD + 5] = win + heads = [_lm_head_shard(weight, r) for r in range(WORLD)] + # Stock: each rank's LMHead forward, all-gathered, then LogitsProcessor's float cast. + shard_logits = [head(x) for head in heads] + for head, stock in zip(heads, shard_logits): + assert torch.equal(head.apply_linear(x, None), stock), ( + "the verify's shard GEMM is not the head's" + ) + gathered = torch.cat(shard_logits, dim=1).float() + target_tokens = gathered.argmax(dim=-1).to(torch.int32) + if ties: + assert bool((target_tokens == TARGET_SHARD + 77).all()) + draft = craft_drafts(target_tokens, nreq, DRAFT_LEN) + stock_acc, stock_num = _stock_accept(gathered, draft) + peer_keys = {r: _local_keys(shard_logits[r], r) for r in range(WORLD)} + for rank in range(WORLD): + peers = {r: k for r, k in peer_keys.items() if r != rank} + acc, num, gather, next_new = _run_shard_accept( + shard_logits[rank], rank, WORLD, draft, peers + ) + assert torch.equal(acc, stock_acc.to(torch.int32)), f"rank {rank}: accepted tokens" + assert torch.equal(num, stock_num.to(torch.int32)), f"rank {rank}: accepted counts" + expect_gather = torch.arange(nreq, device="cuda") * (DRAFT_LEN + 1) + num.long() - 1 + assert torch.equal(gather, expect_gather), f"rank {rank}: first-pass gather rows" + assert torch.equal(next_new[:, 0], target_tokens[expect_gather]), f"rank {rank}: next_new" + + +class _Drafter: + """The in-tree Eagle3 first-pass pieces at the target's shapes: per-layer fc_norm RMSNorms and the fc.""" + + def __init__(self, g: torch.Generator): + self.norms = [] + for _ in range(CAPTURED_LAYERS): + norm = RMSNorm(hidden_size=HIDDEN, eps=EPS, dtype=torch.bfloat16).cuda() + norm.weight.data = (1.0 + 0.1 * torch.randn(HIDDEN, generator=g, device="cuda")).to( + torch.bfloat16 + ) + self.norms.append(norm) + self.fc = Linear(HIDDEN * CAPTURED_LAYERS, HIDDEN, bias=False, dtype=torch.bfloat16).cuda() + self.fc.weight.data = randn((HIDDEN, HIDDEN * CAPTURED_LAYERS), g, 0.02) + self.model = SimpleNamespace( + dtype=torch.bfloat16, + hidden_size=HIDDEN, + fc_norm=torch.nn.ModuleList(self.norms), + _norm_before_fc=False, + fc=self.fc, + ) + self.stacked = torch.stack([n.weight.detach() for n in self.norms]).contiguous() + + def stock_normed(self, hs: torch.Tensor) -> torch.Tensor: + """``apply_eagle3_fc``'s fc input: each captured layer through its own in-tree RMSNorm.""" + chunks = hs.chunk(CAPTURED_LAYERS, dim=-1) + return torch.cat([norm(c) for norm, c in zip(self.norms, chunks)], dim=-1) + + def stock_fc(self, hs: torch.Tensor) -> torch.Tensor: + return Eagle3ForCausalLM.apply_eagle3_fc(self, hs) + + +def _captures(rows: int, g: torch.Generator): + """Per captured layer (hidden, residual) and the in-tree capture buffer of bf16(hidden + residual).""" + layers = [(randn((rows, HIDDEN), g, 1.5), randn((rows, HIDDEN), g, 3.0)) for _ in range(3)] + buf = torch.cat([h + r for h, r in layers], dim=-1) + return layers, buf + + +def _row_identical(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + return (a == b).all(dim=-1) + + +@pytest.mark.parametrize("rows", (4, 8, 32, 48), ids=lambda r: f"rows{r}") +def test_first_drafter_inputs_are_the_stock_fc_norm(rows) -> None: + """chunk_norm and capture_norm against ``apply_eagle3_fc``'s in-tree RMSNorms: within one bf16 ulp on at most + NORM_ULP_FRAC of the elements; the fc (the in-tree module in both paths) then differs only on rows whose input + did.""" + g = gen(seed_of(f"fc_norm {rows}")) + drafter = _Drafter(g) + layers, buf = _captures(rows, g) + stock_in = drafter.stock_normed(buf) + stock_out = drafter.stock_fc(buf) + assert torch.equal(drafter.fc(stock_in), stock_out) + + chunk_in = m3_spec_tail.chunk_norm(buf, drafter.stacked, EPS) + capture_in = torch.empty_like(buf) + for i, (h, r) in enumerate(layers): + m3_spec_tail.capture_norm( + h, r, drafter.stacked[i], EPS, capture_in[:, i * HIDDEN : (i + 1) * HIDDEN] + ) + torch.cuda.synchronize() + for name, ours in (("chunk_norm", chunk_in), ("capture_norm", capture_in)): + assert_bf16_close(ours, stock_in, max_frac=NORM_ULP_FRAC) + out = drafter.fc(ours) + same_rows = _row_identical(ours, stock_in) + assert torch.equal(out[same_rows], stock_out[same_rows]), ( + f"{name}: fc moved an unchanged row" + ) + moved = (ours != stock_in).float().mean().item() + print( + f"[exact-numerics] rows={rows} {name}: {moved:.2e} of fc inputs one ulp off, " + f"{int((~same_rows).sum())}/{rows} rows differ, fc max |diff| " + f"{(out.float() - stock_out.float()).abs().max().item():.3e}" + ) + assert torch.equal(chunk_in, capture_in), "capture_norm and chunk_norm disagree" + + +@pytest.mark.parametrize("rows", (4, 32), ids=lambda r: f"rows{r}") +def test_draft_tokens_through_the_full_vocab_head(rows) -> None: + """The fc_norm's ulp-level differences carried through the full-vocabulary BF16 head (a proxy of the draft head + input): the greedy draft token equals the stock one on every row whose stock top-2 margin exceeds the logit + perturbation.""" + g = gen(seed_of(f"draft head {rows}")) + drafter = _Drafter(g) + _, buf = _captures(rows, g) + weight = randn((VOCAB, HIDDEN), g, 0.02) + head = LMHead(num_embeddings=VOCAB, embedding_dim=HIDDEN, dtype=torch.bfloat16) + head.weight.data = weight + stock_logits = head(drafter.stock_fc(buf)).float() + ours_logits = head(drafter.fc(m3_spec_tail.chunk_norm(buf, drafter.stacked, EPS))).float() + perturb = (ours_logits - stock_logits).abs().max(dim=-1).values + top2 = stock_logits.topk(2, dim=-1).values + decidable = (top2[:, 0] - top2[:, 1]) > 2 * perturb + stock_tok, ours_tok = stock_logits.argmax(-1), ours_logits.argmax(-1) + assert torch.equal(ours_tok[decidable], stock_tok[decidable]) + print( + f"[exact-numerics] rows={rows} draft head: {int((ours_tok == stock_tok).sum())}/{rows} tokens equal, " + f"{int(decidable.sum())} decidable, max logit |diff| {perturb.max().item():.3e}" + ) + + +def test_the_drafter_is_the_in_tree_drafter() -> None: + """Outside the fused draft loop's steps the worker drafts with the in-tree loop, forward and sampling, and nothing + re-quantizes the drafter's linears or the LM heads: no hook of a quantized / pruned head or drafter projection is + left in the target, and the fused head streams the BF16 head weight itself.""" + assert ( + modeling.M3Eagle3Worker._forward_draft_loop is not Eagle3OneModelWorker._forward_draft_loop + ) + for name in ( + "_forward_linear_draft_loop", + "_run_draft_forward", + "sample_draft_tokens", + "sample_and_accept_draft_tokens", + ): + assert getattr(modeling.M3Eagle3Worker, name) is getattr(Eagle3OneModelWorker, name), name + for module, name in ( + (modeling, "M3DraftGemvLinearMethod"), + (modeling, "adopt_draft_linears"), + (modeling, "greedy_head_outputs"), + (modeling, "M3FP8LMHeadMethod"), + (modeling, "adopt_fp8_lm_heads"), + ): + assert not hasattr(module, name), f"{module.__name__}.{name}" + pkg = modeling.__name__.rsplit(".", 1)[0] + for gone in ("ops.m3_draft_head", "ops.spec.m3_draft_head", "ops.gemm.m3_draft_gemv"): + assert importlib.util.find_spec(f"{pkg}.{gone}") is None, gone diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_ar_det.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_ar_det.py new file mode 100644 index 000000000000..bac700e045d8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_ar_det.py @@ -0,0 +1,43 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""AR-DET: the target defaults the AllReduce autotune off for its process (the C++ heuristic), keeps an explicit +TLLM_DISABLE_ALLREDUCE_AUTOTUNE as given, and leaves the autotune on with TRTLLM_M31_ENABLE_AR_AUTOTUNE=1.""" + +import os + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( # noqa: E501 + modeling, +) + +KEY = "TLLM_DISABLE_ALLREDUCE_AUTOTUNE" + + +def test_autotune_defaults_off(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", False) + assert modeling._pin_allreduce_heuristic(0) is True + assert os.environ[KEY] == "1" + + +def test_escape_hatch_keeps_autotune_on(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", True) + assert modeling._pin_allreduce_heuristic(0) is False + assert KEY not in os.environ + + +def test_explicit_setting_is_kept(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {KEY: "0"}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", False) + assert modeling._pin_allreduce_heuristic(0) is False + assert os.environ[KEY] == "0" + + +def test_escape_hatch_follows_the_switch_contract() -> None: + """TRTLLM_M31_ENABLE_AR_AUTOTUNE is parsed by m31_switches (1/true/on/yes set; ""/0/false/off/no unset), not by + a separate loose read.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( # noqa: E501 + m31_switches, + ) + + assert modeling.AR_AUTOTUNE_ENABLED == m31_switches.m31_enabled("AR_AUTOTUNE") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_fmha_sm100_fixes.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_fmha_sm100_fixes.py new file mode 100644 index 000000000000..b551103f0079 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_fmha_sm100_fixes.py @@ -0,0 +1,44 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The two arch-independent fmha_sm100 correctness fixes the M3.1 targets carry in +3rdparty/patches/msa_strided_paged_kv.patch (from #20629 cf8a77e816), checked on the installed package: + +1. Sm100FmhaFwdKernelTmaWarpspecialized is PDL-launched, so it must run griddepcontrol.wait before reading any + input (the upstream wait is commented out: a race with its predecessor; the M3.1 prefill index proxy pass runs it). +2. The NVFP4-KV forward zeroes the V block scales / values of rows at or past kv_valid_cols, so a never-written pool + slot cannot inject NaN through a zero probability. +""" + +import importlib.util +import re +from pathlib import Path + +import pytest + +_SPEC = importlib.util.find_spec("fmha_sm100") +pytestmark = pytest.mark.skipif(_SPEC is None, reason="fmha_sm100 is not installed") + + +def _pkg() -> Path: + return Path(_SPEC.origin).parent + + +def test_the_fwd_kernel_waits_on_its_grid_dependency_before_reading_inputs() -> None: + src = (_pkg() / "csrc/include/sm100_fmha_fwd_kernel_tma_warpspecialized.hpp").read_text() + body = src[src.index("CUTLASS_DEVICE void operator()(const Params& params, char* smem) {") :] + first_lines = body.split("\n")[1:6] + active = [ + line + for line in first_lines + if "griddepcontrol.wait" in line and not line.lstrip().startswith("//") + ] + assert active, "griddepcontrol.wait is missing or commented out at the top of the fwd kernel" + + +def test_the_nvfp4_kv_forward_zeroes_v_rows_past_the_valid_columns() -> None: + src = (_pkg() / "cute/src/sm100/fwd/atten_fwd_nvfp4_kv.py").read_text() + assert re.search(r"scale_e4m3_lo = scale_e4m3_lo if row < kv_valid_cols else Int32\(0\)", src) + assert re.search(r"scale_e4m3_hi = scale_e4m3_hi if row < kv_valid_cols else Int32\(0\)", src) + assert re.search( + r"combined_bf16x2 = combined_bf16x2 if row < kv_valid_cols else Int32\(0\)", src + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_hosttail.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_hosttail.py new file mode 100644 index 000000000000..221e0d9a7e43 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_hosttail.py @@ -0,0 +1,173 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""HOSTTAIL's kill switches on this target: ``_runtime_hooks`` turns on all six host-tail RuntimeHooks fields by +default, ``TRTLLM_M31_DISABLE_`` turns off only its own field and ``TRTLLM_M31_DISABLE_HOSTTAIL`` all six. + +The ``HOSTTAIL_*`` expressions are evaluated from modeling.py's source against a private copy of m31_switches loaded +under each environment (as test_modeling_v2_m31_switches does), so neither the in-tree switches nor the heavy +modeling module is reloaded. CPU only, except the last test, which imports the in-tree module. +""" + +import ast +import importlib +import importlib.util +import os +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.cache_manager import ( + MiniMaxM3KVCacheManagerV2, +) +from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import ( + BaseMambaCacheManager, + MambaHybridCacheManager, +) +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + +TARGET = "minimax_m31_nvfp4__sm_103__tp4" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + +# Kill-switch id -> (the module constant, the RuntimeHooks field it sets). +ITEMS = { + "HOST_D2H_SIDE": ("HOSTTAIL_D2H_SIDE", "sampler_d2h_side_stream"), + "SMALL_H2D": ("HOSTTAIL_SMALL_H2D", "spec_small_h2d"), + "BATCH_CHANGED_ONLY": ("HOSTTAIL_BATCH_CHANGED_ONLY", "spec_batch_map_changed_only"), + "SKIP_ACCEPTED_UPLOAD": ("HOSTTAIL_SKIP_ACCEPTED_UPLOAD", "skip_accepted_draft_upload"), + "PHILOX_PACK": ("HOSTTAIL_PHILOX_PACK", "pack_spec_rng_uploads"), + "KV_LENS_ONCE": ("HOSTTAIL_KV_LENS_ONCE", "kv_lens_update_once"), +} +_SWITCH_IDS = ("HOSTTAIL", *ITEMS) + + +def _module_tree() -> ast.Module: + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _hosttail_statements(tree: ast.Module) -> list[ast.stmt]: + """modeling.py's module-level ``_HOSTTAIL_OFF = ...`` and ``HOSTTAIL_* = ...`` assignments, in source order.""" + statements = [] + for node in tree.body: + if isinstance(node, ast.Assign) and len(node.targets) == 1: + target = node.targets[0] + if isinstance(target, ast.Name) and ( + target.id == "_HOSTTAIL_OFF" or target.id.startswith("HOSTTAIL_") + ): + statements.append(node) + return statements + + +def _switches(monkeypatch, env): + """A private copy of m31_switches imported under ``env`` (the HOSTTAIL variables cleared first).""" + for item in _SWITCH_IDS: + for prefix in ("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_"): + monkeypatch.delenv(prefix + item, raising=False) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_hosttail_{id(env)}", _SWITCHES) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _hosttail(monkeypatch, env) -> dict[str, bool]: + """The six HOSTTAIL_* values modeling.py computes under ``env``, by item id.""" + switches = _switches(monkeypatch, env) + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec( + compile(ast.Module(_hosttail_statements(_module_tree()), []), str(_MODELING), "exec"), + namespace, + ) + return {item: namespace[constant] for item, (constant, _) in ITEMS.items()} + + +def test_the_switches_are_the_six_items(): + names = [node.targets[0].id for node in _hosttail_statements(_module_tree())] + + assert names == ["_HOSTTAIL_OFF"] + [constant for constant, _ in ITEMS.values()] + for _, field in ITEMS.values(): + assert getattr(RuntimeHooks(), field) is False, field + + +def test_every_item_is_on_by_default(monkeypatch): + assert _hosttail(monkeypatch, {}) == {item: True for item in ITEMS} + + +@pytest.mark.parametrize("item", sorted(ITEMS)) +def test_each_kill_switch_turns_off_only_its_own_item(monkeypatch, item): + values = _hosttail(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: "1"}) + + assert values == {other: other != item for other in ITEMS} + + +@pytest.mark.parametrize("value", ["1", "true", "on", "yes"]) +def test_the_bundle_switch_turns_off_all_six(monkeypatch, value): + assert _hosttail(monkeypatch, {"TRTLLM_M31_DISABLE_HOSTTAIL": value}) == { + item: False for item in ITEMS + } + + +@pytest.mark.parametrize("value", ["", "0", "false", "off", "no"]) +def test_an_unset_bundle_switch_keeps_all_six(monkeypatch, value): + assert _hosttail(monkeypatch, {"TRTLLM_M31_DISABLE_HOSTTAIL": value}) == { + item: True for item in ITEMS + } + + +def test_an_enable_variable_does_not_turn_an_item_off_or_on(monkeypatch): + env = {"TRTLLM_M31_ENABLE_" + item: "1" for item in _SWITCH_IDS} + assert _hosttail(monkeypatch, env) == {item: True for item in ITEMS} + + +def test_runtime_hooks_wires_each_field_to_its_switch(): + """``_runtime_hooks`` passes each HOSTTAIL_* constant to its RuntimeHooks field, and nothing else sets those.""" + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + calls = [ + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1 + keywords = {kw.arg: kw.value for kw in calls[0].keywords} + for constant, field in ITEMS.values(): + assert isinstance(keywords.get(field), ast.Name), field + assert keywords[field].id == constant, field + + +def test_the_target_kv_cache_is_not_a_mamba_cache(): + """skip_accepted_draft_upload is on only together with a non-mamba cache: the mamba state update is the one + consumer of accepted counts on the drafter path (it takes them as an argument, not from the spec metadata).""" + assert not issubclass(MiniMaxM3KVCacheManagerV2, BaseMambaCacheManager) + assert not issubclass(MiniMaxM3KVCacheManagerV2, MambaHybridCacheManager) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="imports the target's modeling module") +def test_the_in_tree_hooks_follow_the_switches(): + modeling = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.modeling" + ) + hooks = modeling._runtime_hooks(None) + + for constant, field in ITEMS.values(): + assert getattr(hooks, field) is getattr(modeling, constant), field + + def disabled(item): + return os.environ.get("TRTLLM_M31_DISABLE_" + item, "").strip().lower() in ( + "1", + "true", + "on", + "yes", + ) + + for item, (_, field) in ITEMS.items(): + assert getattr(hooks, field) is not (disabled("HOSTTAIL") or disabled(item)), item diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_switches.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_switches.py new file mode 100644 index 000000000000..81a862f2a624 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_switches.py @@ -0,0 +1,156 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The value contract of this target's kill switches (m31_switches): 1 / true / on / yes set TRTLLM_M31_DISABLE_ / +TRTLLM_M31_ENABLE_, empty / 0 / false / off / no leave it unset (case-insensitive), anything else raises at import +naming the variable; the environment is read once, at import. + +Each case loads a private copy of the module from its file, so the in-tree module (read by every other module of the +target at import) is never reloaded. CPU only. +""" + +import ast +import importlib.util +import os +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_103__tp4" +_SWITCHES = Path(_family.__file__).resolve().parent / TARGET / "m31_switches.py" + +pytestmark = pytest.mark.cpu_only + +KINDS = [("TRTLLM_M31_DISABLE_", "m31_disabled"), ("TRTLLM_M31_ENABLE_", "m31_enabled")] +ITEM = "UTCONTRACT" # an id no item uses, so the real switches of the test process cannot interfere + + +def _load(monkeypatch, env): + """A private copy of m31_switches, imported under ``env`` (every other TRTLLM_M31_* variable cleared).""" + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_contract_{id(env)}", _SWITCHES) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@pytest.mark.parametrize("prefix,query", KINDS) +@pytest.mark.parametrize( + "value", ["", " ", "0", " 0 ", "false", "FALSE", "off", " Off ", "no", "No"] +) +def test_unset_spellings_leave_the_switch_unset(monkeypatch, prefix, query, value): + assert not getattr(_load(monkeypatch, {}), query)(ITEM) + assert not getattr(_load(monkeypatch, {prefix + ITEM: value}), query)(ITEM) + + +@pytest.mark.parametrize("prefix,query", KINDS) +@pytest.mark.parametrize("value", ["1", " 1 ", "true", "TRUE", "on", "On", "yes", "YES"]) +def test_set_spellings_set_the_switch(monkeypatch, prefix, query, value): + assert getattr(_load(monkeypatch, {prefix + ITEM: value}), query)(ITEM) + + +@pytest.mark.parametrize("prefix", [k for k, _ in KINDS]) +@pytest.mark.parametrize("value", ["2", "-1", "1.0", "y", "n", "enable", "disabled", "truee", "of"]) +def test_any_other_value_fails_at_import_naming_the_variable(monkeypatch, prefix, value): + """Typos must fail loudly: invalid switch values must not silently disable a feature.""" + with pytest.raises(ValueError, match=prefix + ITEM): + _load(monkeypatch, {prefix + ITEM: value}) + + +def test_the_id_is_matched_exactly(monkeypatch): + switches = _load(monkeypatch, {"TRTLLM_M31_DISABLE_" + ITEM: "1"}) + assert switches.m31_disabled(ITEM) + for other in (ITEM.lower(), ITEM + "_X", ITEM[:-1], ""): + assert not switches.m31_disabled(other), other + + +def test_disable_and_enable_are_independent(monkeypatch): + """Both read: an item's own code applies "DISABLE wins" (e.g. B1, L-DS1); the module reports each separately.""" + switches = _load( + monkeypatch, {"TRTLLM_M31_DISABLE_" + ITEM: "1", "TRTLLM_M31_ENABLE_" + ITEM: "1"} + ) + assert switches.m31_disabled(ITEM) and switches.m31_enabled(ITEM) + + +def test_the_environment_is_read_once_at_import(monkeypatch): + switches = _load(monkeypatch, {}) + monkeypatch.setenv("TRTLLM_M31_DISABLE_" + ITEM, "1") + monkeypatch.setenv("TRTLLM_M31_ENABLE_" + ITEM, "1") + assert not switches.m31_disabled(ITEM) and not switches.m31_enabled(ITEM) + + +# The step-input pack items (VR2GB-01 / VR2GB-03) and the order each one needs: a killed prerequisite turns off +# every item that needs it. +_PACK_FLAGS = ("PACK_PSCATTER", "PACK_IN_GRAPH", "PACK_EARLY_COPY", "H2D_CHANGED_SPAN") +_PACK_HOOKS = { + "pack_parallel_scatter": "PACK_PSCATTER", + "pack_in_graph": "PACK_IN_GRAPH", + "pack_early_copy": "PACK_EARLY_COPY", + "h2d_changed_span": "H2D_CHANGED_SPAN", +} + + +def _modeling_tree(): + return ast.parse((_SWITCHES.parent / "modeling.py").read_text()) + + +def _pack_flags(disabled): + """The target's pack flags as its module computes them at import, with ``disabled`` the switched-off ids (the + flags' own statements, evaluated with a stand-in m31_disabled: no import of the target).""" + statements = [ + node + for node in _modeling_tree().body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id in _PACK_FLAGS + ] + assert sorted(node.targets[0].id for node in statements) == sorted(_PACK_FLAGS) + namespace = {"m31_disabled": lambda item_id: item_id in disabled} + exec(compile(ast.Module(body=statements, type_ignores=[]), "", "exec"), namespace) + return {name: namespace[name] for name in _PACK_FLAGS} + + +@pytest.mark.parametrize( + "disabled,expect_on", + [ + ((), _PACK_FLAGS), + (("H2D_CHANGED_SPAN",), ("PACK_PSCATTER", "PACK_IN_GRAPH", "PACK_EARLY_COPY")), + (("PACK_EARLY_COPY",), ("PACK_PSCATTER", "PACK_IN_GRAPH", "H2D_CHANGED_SPAN")), + (("PACK_IN_GRAPH",), ("PACK_PSCATTER", "H2D_CHANGED_SPAN")), + (("PACK_PSCATTER",), ("H2D_CHANGED_SPAN",)), + (("PACK_PSCATTER", "H2D_CHANGED_SPAN"), ()), + ], +) +def test_pack_items_default_on_and_a_killed_prerequisite_turns_its_dependents_off( + disabled, expect_on +): + flags = _pack_flags(set(disabled)) + assert {name for name, on in flags.items() if on} == set(expect_on) + + +def test_runtime_hooks_take_the_pack_flags(): + """_runtime_hooks passes each pack flag to its RuntimeHooks field (not a literal True).""" + (function,) = [ + node + for node in _modeling_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ] + (call,) = [ + node + for node in ast.walk(function) + if isinstance(node, ast.Call) and getattr(node.func, "id", None) == "RuntimeHooks" + ] + passed = {kw.arg: kw.value for kw in call.keywords} + for field, flag in _PACK_HOOKS.items(): + assert isinstance(passed.get(field), ast.Name) and passed[field].id == flag, field + assert ( + isinstance(passed["skip_unchanged_h2d"], ast.Constant) + and passed["skip_unchanged_h2d"].value + ) + assert isinstance(passed["pack_step_inputs"], ast.Constant) and passed["pack_step_inputs"].value diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_ar64auto.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_ar64auto.py new file mode 100644 index 000000000000..1ad6d30cf627 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_ar64auto.py @@ -0,0 +1,79 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""AR64AUTO dispatch (CPU): the generation boundaries past the multicast all-reduce's rows, up to the decode bound, +take the strategy-MNNVL instance; every other row count, and a core without it (kill switch), the configured one.""" + +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops import ( + m3_mc_allreduce, +) + + +@pytest.fixture +def taken(monkeypatch): + seen = [] + monkeypatch.setattr( + dispatch, "m3_allreduce", lambda x, comm, params: seen.append(comm) or (x, x) + ) + return seen + + +# AllReduceParams(fusion_op=RESIDUAL_RMS_NORM) asserts a residual; the patched m3_allreduce never reads these. +_RESIDUAL = object() +_NORM_W = object() + + +class _Rows: + def __init__(self, m: int): + self.shape = (m, 6144) + + def contiguous(self): + return self + + +def _core(wide=True): + return SimpleNamespace( + allreduce="configured", allreduce_wide="auto" if wide else None, rms_eps=1e-6 + ) + + +def test_the_wide_rows_take_the_mnnvl_instance(taken) -> None: + lo, hi = m3_mc_allreduce.max_rows(), dispatch.DECODE_MAX_TOKENS + assert hi > lo, ( + "the decode bound must exceed the multicast all-reduce's rows for AR64AUTO to apply" + ) + for m, want in ((1, "configured"), (lo, "configured"), (lo + 1, "auto"), (hi, "auto"), (hi + 1, "configured"), + (2048, "configured")): # fmt: skip + dispatch._native_allreduce_norm(_core(), _Rows(m), _RESIDUAL, _NORM_W) + assert taken[-1] == want, (m, taken[-1]) + + +def test_without_the_auto_instance_every_row_count_takes_the_configured_one(taken) -> None: + for m in (1, m3_mc_allreduce.max_rows() + 1, dispatch.DECODE_MAX_TOKENS): + dispatch._native_allreduce_norm(_core(wide=False), _Rows(m), _RESIDUAL, _NORM_W) + assert taken[-1] == "configured", m + + +def test_a_declined_wide_call_runs_the_configured_instance(monkeypatch) -> None: + """MNNVL returning None (a shape it does not take) must not fall to AUTO: the configured instance runs.""" + seen = [] + + def fake(x, comm, params): + seen.append(comm) + return None if comm == "auto" else (x, x) + + monkeypatch.setattr(dispatch, "m3_allreduce", fake) + dispatch._native_allreduce_norm( + _core(), _Rows(m3_mc_allreduce.max_rows() + 1), _RESIDUAL, _NORM_W + ) + assert seen == ["auto", "configured"] + + +def test_the_kill_switch_is_read_at_import() -> None: + assert isinstance(dispatch.AR64AUTO, bool) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_fused_projection_default.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_fused_projection_default.py new file mode 100644 index 000000000000..90360a858a56 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_fused_projection_default.py @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The M3.1 TP4 target packs the sparse layers' QKV + index projection by default (USER rule: measured wins ship on +by default); an explicit ``fuse_qkv_index_projection`` in the deployment's config wins, so False is the kill switch.""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( + weights as tp_weights, +) +from tensorrt_llm.llmapi.llm_args import MiniMaxM3SparseAttentionConfig + +_MSA = dict(implementation="msa", indexer_kv_dtype="fp8") + + +@pytest.mark.parametrize( + ("explicit", "expected"), + [ + ({}, True), + ({"fuse_qkv_index_projection": True}, True), + ({"fuse_qkv_index_projection": False}, False), + ], +) +def test_fused_projection_default_and_kill(explicit: dict, expected: bool) -> None: + config = MiniMaxM3SparseAttentionConfig(**_MSA, **explicit) + assert tp_weights.fused_projection(config) is expected + + +def test_no_sparse_config_is_unfused() -> None: + assert tp_weights.fused_projection(None) is False + + +@pytest.mark.parametrize("value", [True, False]) +def test_a_config_without_set_fields_is_taken_at_its_word(value: bool) -> None: + from types import SimpleNamespace + + assert tp_weights.fused_projection(SimpleNamespace(fuse_qkv_index_projection=value)) is value diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_weights_layout.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_weights_layout.py new file mode 100644 index 000000000000..8eb0dd6a700d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/test_modeling_v2_m31_tp4_weights_layout.py @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP4 weight table: layer kinds from the config, packed-NVFP4 expert scalars as FP32 divisors.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4 import ( + weights as W, +) + +pytestmark = pytest.mark.cpu_only + + +def _core(sparse_freq, moe_freq, fused=True): + cfg = SimpleNamespace( + num_hidden_layers=len(sparse_freq), + moe_layer_freq=moe_freq, + sparse_attention_config=dict( + sparse_attention_freq=sparse_freq, sparse_num_index_heads=4, sparse_index_dim=128 + ), + head_dim=128, + num_attention_heads=64, + num_key_value_heads=4, + hidden_size=6144, + dense_intermediate_size=12288, + shared_intermediate_size=3072, + n_shared_experts=1, + intermediate_size=3072, + num_local_experts=128, + vocab_size=200064, + torch_dtype=torch.bfloat16, + ) + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=cfg, + mapping=SimpleNamespace(tp_size=4, tp_rank=0), + sparse_attention_config=SimpleNamespace(fuse_qkv_index_projection=fused), + ) + ) + + +def test_layer_kinds_come_from_the_config_independently(): + """Sparse attention on every layer while layers 0-1 keep the dense MLP.""" + d = W.MODEL_WEIGHTS.dims(_core([1] * 6, [0, 0, 1, 1, 1, 1])) + assert list(W._sparse_attn_layers(d)) == [0, 1, 2, 3, 4, 5] + assert list(W._dense_mlp_layers(d)) == [0, 1] + assert list(W._moe_layers(d)) == [2, 3, 4, 5] + assert list(W._five_way_qkv_layers(d)) == [0, 1, 2, 3, 4, 5] + assert list(W._three_way_qkv_layers(d)) == [] + + +def test_unfused_projection_uses_the_three_way_qkv_everywhere(): + d = W.MODEL_WEIGHTS.dims(_core([0, 0, 1, 1], [0, 0, 1, 1], fused=False)) + assert list(W._three_way_qkv_layers(d)) == [0, 1, 2, 3] + assert list(W._unfused_sparse_layers(d)) == [2, 3] + assert list(W._five_way_qkv_layers(d)) == [] + + +def test_expert_scalars_are_the_fp32_reciprocal_of_the_stored_multiplier(): + stored = torch.tensor(12345.678, dtype=torch.float32) + out = W._divisor(stored, None) + assert out.dtype == torch.float32 + assert out.view(torch.int32).item() == torch.reciprocal(stored).view(torch.int32).item() + with pytest.raises(ValueError): + W._divisor(torch.tensor(0.0), None) diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/_dspark_markov_chain_ll_op_matrix.py b/tests/unittest/_torch/modeling_v2/spec_decode/_dspark_markov_chain_ll_op_matrix.py new file mode 100644 index 000000000000..1f314443f93a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/_dspark_markov_chain_ll_op_matrix.py @@ -0,0 +1,671 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Certify `spec_decode/dspark_markov_chain`'s LL exchange variant (``enable_ll_exchange``, kernel +``dspark_markov_chain_kernel_b1_ll``): its batch-1 draft tokens equal the S4B8 batch-1 kernel's +(``dspark_markov_chain_kernel_b1``) bit for bit, on real ranks over the multicast workspace. + +Every rank builds the same full-vocabulary tensors from one seed and passes its padded shard to the op; every rank +checks LL == b1 on its own tokens, rank 0 also checks both against ``reference``. Checks (batch 1, K = 3 and 7, the +M3.1 vocabulary): + +- inputs: random logits, logits on a coarse grid (many exact ties), all-equal values (zero ``markov_w2``: every + position's token is 0), NaN-free extremes (+-bf16 max, +-the smallest bf16 denormal, +-0.0, -inf); +- forced ties at fixed vocabulary columns (their ``markov_w2`` rows zeroed, so the value is the logit exactly): across + ranks at one local column, the last real column of rank r against the first of rank r + 1, the last row of CTA 0 + against the first of CTA 1, two warps of one CTA, the last real column of every rank; the tied value a large + finite value, the bf16 maximum, the smallest bf16 denormal (every other column -inf) or +-0.0 (every other column + -inf). The token must be the lowest tied column at every position; +- stale-tag safety: 64 back-to-back launches with changing inputs (no host sync), K = 7 and K = 3 interleaved with + batch-2 calls of the Lamport kernel, then 60 back-to-back K = 7 launches; every launch's tokens equal the b1 + kernel's on its inputs; +- a CUDA graph of four LL launches replayed 16 times with new inputs copied in before each replay, against eager b1; +- TAILBK_SPEC cached inputs (``enable_cached_inputs``): tokens with the cache equal tokens without it (LL batch 1 and + the Lamport kernel at batch 2, eager and in a CUDA graph); the cache holds ones / arange / zeros, is reused, never + written, and a shape first met under capture allocates without caching; +- TAILBK_SPEC KV-length rewind (``dspark_markov_chain_rewind``): its tokens equal ``dspark_markov_chain``'s and its + ``kv_lens`` equal the stock ``kv_lens[nc:bs] -= rewind; kv_lens[nc:bs].clamp_(min=0)`` (rows outside untouched), + for rewinds of both signs and clamped rows, eager and in a CUDA graph; ``supports_rewind`` rejects bad operands. + +World 4 (one GB300 tray) is spawned from one process: + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _dspark_markov_chain_ll_op_matrix.py + +World 8 spans two trays and runs as one srun task per rank: + + MASTER_ADDR= MASTER_PORT= python _dspark_markov_chain_ll_op_matrix.py --srun + +under ``srun --ntasks=8 --ntasks-per-node=4 --kill-on-bad-exit=1``. +""" + +import importlib.util +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, List, Optional + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "dspark_markov_chain requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 1500 +RANK_DIM = 256 +VOCAB = 200064 +SEEDS = 4 +BACK_TO_BACK = 64 +GPUS_PER_NODE = 4 +"""A GB300 tray. The mapping needs it to see a world-8 group span two nodes.""" + +BF16_MAX = torch.finfo(torch.bfloat16).max +BF16_DENORM_MIN = 2.0**-133 # the smallest positive bf16 subnormal + +RANK = 0 +WORLD = 1 +entry: Any = None +chain: Any = None +WS: dict = {} + + +def _set_ll(on: bool) -> None: + entry.enable_batch1_variant(True) + entry.enable_ll_exchange(on) + + +def _weights(vocab: int, zero_rows: Optional[List[int]] = None, zero_w2: bool = False): + g = torch.Generator(device="cuda").manual_seed(1234) + w1 = (torch.randn(vocab, RANK_DIM, device="cuda", generator=g) * 0.05).bfloat16() + w2 = (torch.randn(vocab, RANK_DIM, device="cuda", generator=g) * 0.05).bfloat16() + if zero_w2: + w2.zero_() + if zero_rows: + w2[torch.tensor(zero_rows, device="cuda")] = 0 + return w1, w2 + + +class _Shard: + """One vocabulary's shards: this rank's padded `w2` rows and logits, and every rank's for the reference.""" + + def __init__(self, vocab: int, **weights): + self.vocab = vocab + self.shard = vocab // WORLD + self.rows = entry.padded_rows(self.shard, entry.CERTIFIED_BATCH) + assert self.rows >= self.shard, f"no padded shard for {self.shard} rows" + self.w1, w2 = _weights(vocab, **weights) + self.w2s_all = [ + entry.pad_w2_shard(w2[r * self.shard : (r + 1) * self.shard].contiguous(), self.rows) + for r in range(WORLD) + ] + self.w2s = self.w2s_all[RANK] + + def base(self, full: torch.Tensor, rank: int) -> torch.Tensor: + b = entry.pad_logits_buffer(full.shape[0], full.shape[1], self.rows, full.device) + b[:, :, : self.shard] = full[:, :, rank * self.shard : (rank + 1) * self.shard] + return b + + def call(self, base: torch.Tensor, prev0: torch.Tensor) -> torch.Tensor: + return chain(base, prev0, self.w1, self.w2s, RANK * self.shard, **WS) + + def both(self, base: torch.Tensor, prev0: torch.Tensor): + _set_ll(False) + b1 = self.call(base, prev0) + _set_ll(True) + ll = self.call(base, prev0) + return b1, ll + + def reference(self, full: torch.Tensor, prev0: torch.Tensor) -> torch.Tensor: + bases = [self.base(full, r) for r in range(WORLD)] + return chain.reference(bases, prev0, self.w1, self.w2s_all, self.shard) + + def rows_per_cta(self, block: int) -> int: + grid = entry.pick_grid(self.rows, block, 1) + assert grid > 0 + return self.rows // grid + + +def _inputs(seed: int, batch: int, block: int, vocab: int, kind: str): + g = torch.Generator(device="cuda").manual_seed(seed) + base = torch.randn(batch, block, vocab, device="cuda", generator=g) * 3 + if kind == "ties": + base = torch.round(base * 2) / 2 + elif kind == "equal": + base = torch.ones_like(base) + elif kind == "extremes": + pool = torch.tensor( + [ + BF16_MAX, + -BF16_MAX, + BF16_DENORM_MIN, + -BF16_DENORM_MIN, + 0.0, + -0.0, + float("-inf"), + 1.0, + -1.0, + ], + device="cuda", + ) + base = pool[torch.randint(0, pool.numel(), base.shape, device="cuda", generator=g)] + prev0 = torch.randint(0, vocab, (batch,), device="cuda", generator=g) + return base.bfloat16(), prev0 + + +def _assert_equal(what: str, ll: torch.Tensor, b1: torch.Tensor) -> None: + assert torch.equal(ll, b1), f"{what}: LL {ll.tolist()} != b1 {b1.tolist()}" + + +def check_ll_matches_b1() -> None: + sh = _Shard(VOCAB) + sh_eq = _Shard(VOCAB, zero_w2=True) + for block in (3, 7): + for kind in ("random", "ties", "equal", "extremes"): + s = sh_eq if kind == "equal" else sh + for seed in range(SEEDS): + full, prev0 = _inputs(seed, 1, block, s.vocab, kind) + b1, ll = s.both(s.base(full, RANK), prev0) + _assert_equal(f"K {block} {kind} seed {seed}", ll, b1) + if kind == "equal": + assert ll.eq(0).all(), f"all-equal values: tokens {ll.tolist()} are not all 0" + if RANK == 0: + ref = s.reference(full, prev0) + _assert_equal(f"K {block} {kind} seed {seed} vs reference", ll, ref) + dist.barrier() + if RANK == 0: + print( + f"[ll] world={WORLD} LL == b1 (random, ties, all-equal, extremes; K 3 / 7)", flush=True + ) + + +def _tie_patterns(shard: int, rc: int) -> List[List[int]]: + """Global vocabulary columns that tie, one list per pattern (each pattern's token: its lowest column).""" + last = ( + shard - 1 + ) # the last real local column (in CTA last // rc, the last CTA holding real rows) + rows_per_warp = rc // 8 + pats = [ + [r * shard + 5 for r in range(1, WORLD)], # across ranks at one local column (not rank 0) + [0 * shard + last, 1 * shard + 0], # last real column of rank 0 vs first column of rank 1 + [ + (WORLD - 1) * shard + 0, + (WORLD - 2) * shard + last, + ], # ... of rank W - 2 vs rank W - 1, listed high first + [rc, rc - 1], # rank 0: CTA 0's last row vs CTA 1's first row + [ + 2 * shard + rc, + 2 * shard + rc - 1, + 3 % WORLD * shard + rc, + ], # the same across CTAs, on other ranks + [1 * shard + rows_per_warp + 3, 1 * shard + 3], # rank 1, CTA 0: warp 1 vs warp 0 + [r * shard + last for r in range(WORLD)][::-1], # every rank's last real column + [(WORLD - 1) * shard + 7, 7], # rank W - 1 vs rank 0 + ] + return [sorted(set(p)) for p in pats] + + +def check_forced_ties() -> None: + probe = _Shard.__new__(_Shard) + probe.shard = VOCAB // WORLD + probe.rows = entry.padded_rows(probe.shard, entry.CERTIFIED_BATCH) + rcs = {block: probe.rows_per_cta(block) for block in (3, 7)} + pats_by_block = {block: _tie_patterns(probe.shard, rcs[block]) for block in (3, 7)} + zero_rows = sorted({c for pats in pats_by_block.values() for p in pats for c in p}) + sh = _Shard(VOCAB, zero_rows=zero_rows) + values = ( + ("large", 100.0, False), + ("bf16max", BF16_MAX, False), + ("denormal", BF16_DENORM_MIN, True), + ("signed-zero", 0.0, True), + ) + checked = 0 + for block in (3, 7): + pats = pats_by_block[block] + for name, tie, others_neg_inf in values: + for offset in range(len(pats)): + full, prev0 = _inputs(1000 + offset, 1, block, sh.vocab, "random") + full = full.float() + expect = [] + for k in range(block): + cols = pats[(offset + k) % len(pats)] + if others_neg_inf: + full[0, k, :] = float("-inf") + idx = torch.tensor(cols, device="cuda") + if name == "signed-zero": + # -0.0 and +0.0 alternate over the tied columns: equal under the order. + signs = torch.tensor( + [(-0.0 if i % 2 == 0 else 0.0) for i in range(len(cols))], device="cuda" + ) + full[0, k, idx] = signs + else: + full[0, k, idx] = tie + expect.append(min(cols)) + full = full.bfloat16() + b1, ll = sh.both(sh.base(full, RANK), prev0) + what = f"forced ties {name} K {block} offset {offset}" + _assert_equal(what, ll, b1) + want = torch.tensor([expect], dtype=torch.int32, device=ll.device) + assert torch.equal(ll, want), ( + f"{what}: tokens {ll.tolist()} != lowest tied columns {expect}" + ) + if RANK == 0: + _assert_equal(f"{what} vs reference", ll, sh.reference(full, prev0)) + checked += 1 + dist.barrier() + if RANK == 0: + print( + f"[ll] world={WORLD} forced ties: {checked} cases, LL == b1 == lowest tied column", + flush=True, + ) + + +def check_back_to_back_stale_tags() -> None: + sh = _Shard(VOCAB) + # Warm every shape once (compiles run outside any sequence below). + for block, batch, ll in ( + (7, 1, True), + (3, 1, True), + (3, 2, True), + (7, 1, False), + (3, 1, False), + ): + full, prev0 = _inputs(1, batch, block, sh.vocab, "random") + _set_ll(ll) + sh.call(sh.base(full, RANK), prev0) + torch.cuda.synchronize() + dist.barrier() + + def run_sequence(shapes, seed0: int, tag: str) -> None: + launches = [] + for i, (block, batch) in enumerate(shapes): + full, prev0 = _inputs( + seed0 + i, batch, block, sh.vocab, "ties" if i % 3 == 0 else "random" + ) + launches.append((sh.base(full, RANK), prev0)) + torch.cuda.synchronize() + _set_ll(True) + outs = [sh.call(base, prev0) for base, prev0 in launches] # back to back, no host sync + torch.cuda.synchronize() + _set_ll(False) + refs = [sh.call(base, prev0) for base, prev0 in launches] + torch.cuda.synchronize() + for i, (got, ref) in enumerate(zip(outs, refs)): + _assert_equal(f"{tag} launch {i} (K {shapes[i][0]}, batch {shapes[i][1]})", got, ref) + + mixed = [((7, 1), (3, 1), (7, 1), (3, 2))[i % 4] for i in range(BACK_TO_BACK)] + run_sequence(mixed, 5000, "mixed back-to-back") + run_sequence([(7, 1)] * 60, 7000, "K7 back-to-back") + dist.barrier() + if RANK == 0: + print( + f"[ll] world={WORLD} back-to-back: {BACK_TO_BACK} mixed + 60 K7 launches equal b1", + flush=True, + ) + + +def check_cuda_graph() -> None: + sh = _Shard(VOCAB) + blocks = (7, 3, 7, 7) + static = [] + for j, block in enumerate(blocks): + full, prev0 = _inputs(300 + j, 1, block, sh.vocab, "random") + static.append((sh.base(full, RANK), prev0.clone())) + _set_ll(True) + for base, prev0 in static: # compiled outside capture + sh.call(base, prev0) + torch.cuda.synchronize() + dist.barrier() + graph_out = [] + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g): + for base, prev0 in static: + graph_out.append(sh.call(base, prev0)) + torch.cuda.current_stream().wait_stream(s) + torch.cuda.synchronize() + dist.barrier() + for rep in range(16): + for j, (block, (base, prev0)) in enumerate(zip(blocks, static)): + full, p = _inputs( + 400 + rep * 8 + j, 1, block, sh.vocab, "ties" if rep % 2 else "random" + ) + base.copy_(sh.base(full, RANK)) + prev0.copy_(p) + g.replay() + torch.cuda.synchronize() + got = [t.clone() for t in graph_out] + _set_ll(False) + refs = [sh.call(base, prev0) for base, prev0 in static] + torch.cuda.synchronize() + _set_ll(True) + for j, (a, b) in enumerate(zip(got, refs)): + _assert_equal(f"graph replay {rep} launch {j} (K {blocks[j]})", a, b) + dist.barrier() + if RANK == 0: + print( + f"[ll] world={WORLD} CUDA graph: 16 replays x 4 LL launches equal eager b1", flush=True + ) + + +def check_cached_inputs() -> None: + sh = _Shard(VOCAB) + shapes = ((7, 1), (3, 1), (7, 2)) + _set_ll(True) + entry._cached_taken[0] = False + entry.enable_cached_inputs(True) + for block, batch in shapes: # outside capture: compiles and fills the cache + full, prev0 = _inputs(11, batch, block, sh.vocab, "random") + sh.call(sh.base(full, RANK), prev0) + torch.cuda.synchronize() + dev = torch.device("cuda", torch.cuda.current_device()) + cached = {} + for block, batch in shapes: + key = (dev, batch, block) + assert key in entry._const_inputs, f"no cached inputs for batch {batch} K {block}" + nacc, rows, acc = entry._const_inputs[key] + assert nacc.dtype == rows.dtype == acc.dtype == torch.int32 + assert torch.equal(nacc, torch.ones(batch, dtype=torch.int32, device=dev)) + assert torch.equal(rows, torch.arange(batch, dtype=torch.int32, device=dev)) + assert torch.equal(acc, torch.zeros(batch, block + 1, dtype=torch.int32, device=dev)) + cached[key] = (nacc, rows, acc) + + assert not entry.cached_inputs_taken(), "a cache fill alone is not the engagement" + + def both(base, prev0): + entry.enable_cached_inputs(False) + off = sh.call(base, prev0) + entry.enable_cached_inputs(True) + on = sh.call(base, prev0) + return off, on + + for block, batch in shapes: + for kind in ("random", "ties", "extremes"): + for seed in range(SEEDS): + full, prev0 = _inputs(600 + seed, batch, block, sh.vocab, kind) + off, on = both(sh.base(full, RANK), prev0) + _assert_equal(f"cached inputs K {block} batch {batch} {kind} seed {seed}", on, off) + assert entry.cached_inputs_taken(), "a call with the cache filled must report the inputs taken" + # A shape compiled with the cache off, then first met with it on under capture: allocated, not cached. + miss = (dev, 3, 7) + entry._const_inputs.pop(miss, None) + entry.enable_cached_inputs(False) + full, prev0 = _inputs(13, 3, 7, sh.vocab, "random") + sh.call(sh.base(full, RANK), prev0) + torch.cuda.synchronize() + dist.barrier() + + graph_shapes = ((7, 1), (3, 1), (7, 2), (7, 3)) + static = [] + for j, (block, batch) in enumerate(graph_shapes): + full, prev0 = _inputs(700 + j, batch, block, sh.vocab, "random") + static.append((sh.base(full, RANK), prev0.clone())) + entry.enable_cached_inputs(True) + graph_out = [] + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g): + for base, prev0 in static: + graph_out.append(sh.call(base, prev0)) + torch.cuda.current_stream().wait_stream(s) + torch.cuda.synchronize() + dist.barrier() + assert miss not in entry._const_inputs, "a shape first met under capture was cached" + for key, tensors in cached.items(): + assert all(a is b for a, b in zip(entry._const_inputs[key], tensors)), ( + f"cache {key} replaced" + ) + for rep in range(8): + for j, ((block, batch), (base, prev0)) in enumerate(zip(graph_shapes, static)): + full, p = _inputs( + 800 + rep * 8 + j, batch, block, sh.vocab, "ties" if rep % 2 else "random" + ) + base.copy_(sh.base(full, RANK)) + prev0.copy_(p) + g.replay() + torch.cuda.synchronize() + got = [t.clone() for t in graph_out] + entry.enable_cached_inputs(False) + refs = [sh.call(base, prev0) for base, prev0 in static] + torch.cuda.synchronize() + entry.enable_cached_inputs(True) + for j, (a, b) in enumerate(zip(got, refs)): + _assert_equal(f"cached-inputs graph replay {rep} launch {j} {graph_shapes[j]}", a, b) + for key, (nacc, rows, acc) in cached.items(): # never written by the kernels + batch, block = key[1], key[2] + assert torch.equal(nacc, torch.ones(batch, dtype=torch.int32, device=dev)) + assert torch.equal(rows, torch.arange(batch, dtype=torch.int32, device=dev)) + assert torch.equal(acc, torch.zeros(batch, block + 1, dtype=torch.int32, device=dev)) + entry.enable_cached_inputs(False) + dist.barrier() + if RANK == 0: + print( + f"[ll] world={WORLD} cached inputs: eager + CUDA graph tokens equal uncached; cache intact", + flush=True, + ) + + +def _stock_rewind(kv_lens: torch.Tensor, amount: torch.Tensor, nc: int) -> torch.Tensor: + want = kv_lens.clone() + bs = nc + amount.numel() + want[nc:bs] -= amount + want[nc:bs].clamp_(min=0) + return want + + +def _rewind_case(seed: int, batch: int, nc: int): + g = torch.Generator(device="cuda").manual_seed(seed) + kv_lens = torch.randint(0, 40, (nc + batch + 3,), dtype=torch.int32, device="cuda", generator=g) + amount = torch.randint(-8, 9, (batch,), dtype=torch.int32, device="cuda", generator=g) + amount[0] = int(kv_lens[nc]) + 5 # a clamped row + return kv_lens, amount + + +def check_rewind() -> None: + sh = _Shard(VOCAB) + entry.enable_cached_inputs(False) + for ll in (True, False): + _set_ll(ll) + for block, batch, nc in ((7, 1, 0), (3, 1, 2), (7, 2, 1), (7, 3, 0)): + if batch > 1 and not ll: + continue + for seed in range(SEEDS): + full, prev0 = _inputs( + 900 + seed, batch, block, sh.vocab, "ties" if seed % 2 else "random" + ) + base = sh.base(full, RANK) + kv_lens, amount = _rewind_case(950 + seed * 7 + batch, batch, nc) + want = _stock_rewind(kv_lens, amount, nc) + ref = sh.call(base, prev0) + assert entry.supports_rewind(kv_lens, amount, nc) + got = entry.dspark_markov_chain_rewind(base, prev0, sh.w1, sh.w2s, RANK * sh.shard, kv_lens, + amount, nc, **WS) # fmt: skip + torch.cuda.synchronize() + tag = f"rewind ll={ll} K {block} batch {batch} nc {nc} seed {seed}" + _assert_equal(f"{tag} tokens", got, ref) + assert torch.equal(kv_lens, want), ( + f"{tag}: kv_lens {kv_lens.tolist()} != {want.tolist()}" + ) + i32 = torch.zeros(8, dtype=torch.int32, device="cuda") + assert not entry.supports_rewind(None, i32[:2], 0) + assert not entry.supports_rewind(i32, i32[:2].long(), 0) + assert not entry.supports_rewind(i32.long(), i32[:2], 0) + assert not entry.supports_rewind(i32, i32[:2], 7) # past the end + assert not entry.supports_rewind(i32, i32[:0], 0) + assert not entry.supports_rewind(i32[::2], i32[:2], 0) + + # A CUDA graph of the rewind launch (batch 1, K 7, nc 1), replayed with new inputs and kv_lens copied in. + _set_ll(True) + full, prev0 = _inputs(990, 1, 7, sh.vocab, "random") + base = sh.base(full, RANK) + kv_lens, amount = _rewind_case(991, 1, 1) + entry.dspark_markov_chain_rewind( + base, prev0, sh.w1, sh.w2s, RANK * sh.shard, kv_lens, amount, 1, **WS + ) + torch.cuda.synchronize() + dist.barrier() + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g): + out = entry.dspark_markov_chain_rewind(base, prev0, sh.w1, sh.w2s, RANK * sh.shard, kv_lens, + amount, 1, **WS) # fmt: skip + torch.cuda.current_stream().wait_stream(s) + torch.cuda.synchronize() + dist.barrier() + for rep in range(8): + f, p = _inputs(1000 + rep, 1, 7, sh.vocab, "ties" if rep % 2 else "random") + base.copy_(sh.base(f, RANK)) + prev0.copy_(p) + k, a = _rewind_case(1100 + rep, 1, 1) + kv_lens.copy_(k) + amount.copy_(a) + want = _stock_rewind(k, a, 1) + g.replay() + torch.cuda.synchronize() + got = out.clone() + ref = sh.call(base, prev0) + torch.cuda.synchronize() + _assert_equal(f"rewind graph replay {rep} tokens", got, ref) + assert torch.equal(kv_lens, want), ( + f"rewind graph replay {rep}: kv_lens {kv_lens.tolist()} != {want.tolist()}" + ) + dist.barrier() + if RANK == 0: + print( + f"[ll] world={WORLD} rewind: tokens equal the chain's, kv_lens equal the stock rewind", + flush=True, + ) + + +def check_switches() -> None: + _set_ll(True) + assert entry.ll_exchange_engaged(1) and not entry.ll_exchange_engaged(2) + entry.enable_batch1_variant(False) + assert not entry.ll_exchange_engaged(1), "LL needs the batch-1 variant" + _set_ll(False) + assert not entry.ll_exchange_engaged(1) + dist.barrier() + + +CHECKS = ( + check_switches, + check_ll_matches_b1, + check_forced_ties, + check_back_to_back_stale_tags, + check_cuda_graph, + check_cached_inputs, + check_rewind, +) + + +def _run_one_rank(device: int) -> int: + global RANK, WORLD, WS, entry, chain + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(device) + dist.init_process_group( + "cuda:nccl,cpu:gloo", init_method="env://", device_id=torch.device("cuda", device) + ) + RANK, WORLD = rank, world + + from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_chain as entry_module, + ) + from tensorrt_llm.mapping import Mapping + + entry = entry_module + chain = entry.dspark_markov_chain + mapping = Mapping(world_size=world, rank=rank, gpus_per_node=GPUS_PER_NODE, tp_size=world) + WS = entry.workspace_args(mapping, dist.group.WORLD) + + for check in CHECKS: + try: + check() + except BaseException: + import traceback + + print(f"[rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + print(f"[rank {RANK}] {check.__name__} passed", flush=True) + dist.barrier() + torch.cuda.synchronize() + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict(os.environ, MASTER_ADDR="127.0.0.1", MASTER_PORT=str(port), WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1") # fmt: skip + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) # fmt: skip + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +def _rank_job(): + """`_rank_job.py` beside this file, by path (a sibling directory ships its own module of that name).""" + path = Path(__file__).resolve().with_name("_rank_job.py") + spec = importlib.util.spec_from_file_location("_spec_decode_rank_job", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +if __name__ == "__main__": + if "--srun" in sys.argv: + sys.exit(_run_one_rank(device=_rank_job().adopt_srun_env())) + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank(device=int(os.environ["RANK"]))) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/_dspark_markov_chain_op_matrix.py b/tests/unittest/_torch/modeling_v2/spec_decode/_dspark_markov_chain_op_matrix.py new file mode 100644 index 000000000000..5f5692a621fa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/_dspark_markov_chain_op_matrix.py @@ -0,0 +1,287 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Certify `spec_decode/dspark_markov_chain`: the one-launch chain's draft tokens against the reference chain's, bit +for bit, on every cell certified at this run's world size. + +Every rank builds the same full-vocabulary tensors from one seed and passes its padded shard to the op; rank 0 checks +the tokens against `reference` over every rank's shard. Per cell: random logits and logits on a coarse grid (many +exact bf16 ties, so the tie order is exercised), each of the cell's batches, several seeds, inside `validating`. Also: +three repeated calls give identical tokens and a CUDA-graph replay gives the eager tokens; `supports` refuses a batch +above the kernel's capacity and fp32 logits; the guard refuses a padding column that is not -inf. + +World 4 (one GB300 tray) is spawned from one process: + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _dspark_markov_chain_op_matrix.py + +World 8 spans two trays and runs as one srun task per rank, like `comm/_mnnvl_allreduce_op_matrix.py`: + + MASTER_ADDR= MASTER_PORT= python _dspark_markov_chain_op_matrix.py --srun + +under ``srun --ntasks=8 --ntasks-per-node=4 --kill-on-bad-exit=1``. +""" + +import importlib.util +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Optional + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "dspark_markov_chain requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 +RANK_DIM = 256 +SEEDS = 8 +GPUS_PER_NODE = 4 +"""A GB300 tray. The mapping needs it to see a world-8 group span two nodes.""" + +RANK = 0 +WORLD = 1 +# Bound inside the rank body: `entry` is the module, `chain` its `dspark_markov_chain` instance (the call surface, +# `reference`, `is_valid` and `CELLS`), `WS` the workspace arguments of every call. +entry: Any = None +chain: Any = None +validating: Any = None +WS: dict = {} + + +def _weights(vocab: int): + g = torch.Generator(device="cuda").manual_seed(1234) + w1 = (torch.randn(vocab, RANK_DIM, device="cuda", generator=g) * 0.05).bfloat16() + w2 = (torch.randn(vocab, RANK_DIM, device="cuda", generator=g) * 0.05).bfloat16() + return w1, w2 + + +def _inputs(seed: int, batch: int, block: int, vocab: int, ties: bool): + g = torch.Generator(device="cuda").manual_seed(seed) + base = torch.randn(batch, block, vocab, device="cuda", generator=g) * 3 + if ties: + base = torch.round(base * 2) / 2 + prev0 = torch.randint(0, vocab, (batch,), device="cuda", generator=g) + return base.bfloat16(), prev0 + + +class _Shard: + """One vocabulary's shards: this rank's padded `w2` rows and logits, and every rank's for the reference.""" + + def __init__(self, vocab: int): + self.vocab = vocab + self.shard = vocab // WORLD + self.rows = entry.padded_rows(self.shard, entry.CERTIFIED_BATCH) + assert self.rows >= self.shard, f"no padded shard for {self.shard} rows" + self.w1, w2 = _weights(vocab) + self.w2s_all = [ + entry.pad_w2_shard(w2[r * self.shard : (r + 1) * self.shard].contiguous(), self.rows) + for r in range(WORLD) + ] + self.w2s = self.w2s_all[RANK] + + def base(self, full: torch.Tensor, rank: int) -> torch.Tensor: + b = entry.pad_logits_buffer(full.shape[0], full.shape[1], self.rows, full.device) + b[:, :, : self.shard] = full[:, :, rank * self.shard : (rank + 1) * self.shard] + return b + + def call(self, base: torch.Tensor, prev0: torch.Tensor) -> torch.Tensor: + return chain(base, prev0, self.w1, self.w2s, RANK * self.shard, **WS) + + def reference(self, full: torch.Tensor, prev0: torch.Tensor) -> torch.Tensor: + bases = [self.base(full, r) for r in range(WORLD)] + return chain.reference(bases, prev0, self.w1, self.w2s_all, self.shard) + + +def check_certified_cells() -> None: + cells = [c for c in chain.CELLS if c.spec["world"] == WORLD] + assert cells, f"no cell is certified at world size {WORLD}" + for cell in cells: + spec = cell.spec + sh = _Shard(spec["vocab"]) + for ties in (False, True): + for batch in spec["batches"]: + for seed in range(SEEDS): + full, prev0 = _inputs(seed, batch, spec["block"], sh.vocab, ties) + with validating(chain): + got = sh.call(sh.base(full, RANK), prev0) + if RANK == 0: + ref = sh.reference(full, prev0) + assert torch.equal(got, ref), ( + f"{cell.why}: ties {ties} batch {batch} seed {seed}: " + f"{(got != ref).any(1).sum().item()} request(s) differ" + ) + dist.barrier() + if RANK == 0: + print(f"[cell] world={WORLD} {cell.why}: bitwise", flush=True) + + +def check_deterministic_and_graph() -> None: + sh = _Shard(200064) + full, prev0 = _inputs(77, entry.CERTIFIED_BATCH, 7, sh.vocab, False) + base = sh.base(full, RANK) + outs = [sh.call(base, prev0) for _ in range(3)] + assert all(torch.equal(outs[0], o) for o in outs[1:]), "repeated calls differ" + graph_out = {} + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g): + graph_out["t"] = sh.call(base, prev0) + torch.cuda.current_stream().wait_stream(s) + for _ in range(3): + g.replay() + torch.cuda.synchronize() + assert torch.equal(graph_out["t"], outs[0]), "graph replay differs from eager" + dist.barrier() + + +def check_supports_refuses() -> None: + sh = _Shard(200064) + kern = entry._kernel_module() + full, prev0 = _inputs(5, kern.MAX_BATCH + 1, 3, sh.vocab, False) + assert not entry.supports(sh.base(full, RANK), sh.w1, sh.w2s), "a batch above MAX_BATCH" + full, prev0 = _inputs(5, 2, 3, sh.vocab, False) + assert not entry.supports(sh.base(full, RANK).float(), sh.w1, sh.w2s), "fp32 logits" + assert not entry.supports(sh.base(full, RANK)[:, :, :-64], sh.w1, sh.w2s), ( + "logits narrower than w2_shard" + ) + dist.barrier() + + +def check_the_guard_refuses_a_live_padding_column() -> None: + sh = _Shard(200064) + assert sh.rows > sh.shard, "the M3.1 shard is padded" + full, prev0 = _inputs(9, 2, 3, sh.vocab, False) + base = sh.base(full, RANK) + base[:, :, -1] = 0.0 + try: + chain.is_valid(base, prev0, sh.w1, sh.w2s, RANK * sh.shard, **WS) + except AssertionError: + pass + else: + raise AssertionError("the guard admitted a padding column that is not -inf") + dist.barrier() + + +CHECKS = ( + check_certified_cells, + check_deterministic_and_graph, + check_supports_refuses, + check_the_guard_refuses_a_live_padding_column, +) + + +def _run_one_rank(device: int) -> int: + global RANK, WORLD, WS, entry, chain, validating + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(device) + dist.init_process_group( + "cuda:nccl,cpu:gloo", init_method="env://", device_id=torch.device("cuda", device) + ) + RANK, WORLD = rank, world + + from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_chain as entry_module, + ) + from tensorrt_llm.mapping import Mapping + + entry = entry_module + chain = entry.dspark_markov_chain + mapping = Mapping(world_size=world, rank=rank, gpus_per_node=GPUS_PER_NODE, tp_size=world) + WS = entry.workspace_args(mapping, dist.group.WORLD) + + sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + from _validating import validating as _validating_cm + + validating = _validating_cm + + for check in CHECKS: + try: + check() + except BaseException: + import traceback + + print(f"[rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + print(f"[rank {RANK}] {check.__name__} passed", flush=True) + dist.barrier() + torch.cuda.synchronize() + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict(os.environ, MASTER_ADDR="127.0.0.1", MASTER_PORT=str(port), WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1") # fmt: skip + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) # fmt: skip + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +def _rank_job(): + """`_rank_job.py` beside this file, by path (a sibling directory ships its own module of that name).""" + path = Path(__file__).resolve().with_name("_rank_job.py") + spec = importlib.util.spec_from_file_location("_spec_decode_rank_job", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +if __name__ == "__main__": + if "--srun" in sys.argv: + sys.exit(_run_one_rank(device=_rank_job().adopt_srun_env())) + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank(device=int(os.environ["RANK"]))) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/_rank_job.py b/tests/unittest/_torch/modeling_v2/spec_decode/_rank_job.py new file mode 100644 index 000000000000..47dc85fb582e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/_rank_job.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Use the existing four-rank launcher wrapper for spec-decode matrices.""" + +import importlib.util +from pathlib import Path + + +def run(entry: str, world_size: int) -> None: + if world_size != 4: + raise ValueError("This collected launcher supports WORLD4 only") + directory = Path(__file__).resolve().parent + path = directory.parent / "comm" / "_rank_job.py" + spec = importlib.util.spec_from_file_location("_spec_decode_comm_rank_job", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + module.run(entry, launcher_dir=directory) diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_block_tables.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_block_tables.py new file mode 100644 index 000000000000..c521d8f78497 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_block_tables.py @@ -0,0 +1,104 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""catalog/spec_decode/dspark_block_tables: the one-launch draft block-table refresh writes, bit for bit, what +DFlashWorker._refresh_ctx_block_tables writes (decoded tables and placeholder counts), eager and in a CUDA graph, with +the per-row kernel and with S10c-B2's (row, chunk) grid.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_block_tables import ( + enable_parallel_refresh, + refresh_block_tables, +) +from tensorrt_llm._torch.speculative.dflash import DFlashWorker + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU (Triton)") + + +@pytest.fixture(params=[False, True], ids=["per_row", "grid2d"]) +def parallel(request): + enable_parallel_refresh(request.param) + yield request.param + enable_parallel_refresh(False) + + +def _worker(rows, width, pool_idx, divisor, fused): + worker = object.__new__(DFlashWorker) + worker._ctx_block_tables = torch.full((rows, width), 77, dtype=torch.int32, device="cuda") + worker._ctx_block_counts = torch.full((rows,), 77, dtype=torch.int64, device="cuda") + worker._ctx_pool_idx = pool_idx + worker._ctx_block_divisor = divisor + worker._fused_block_refresh = refresh_block_tables if fused else None + return worker + + +def _offsets(pools, rows, width, dtype, seed, high=1 << 20): + g = torch.Generator(device="cuda").manual_seed(seed) + off = torch.randint(0, high, (pools, rows, 2, width), generator=g, device="cuda").to(dtype) + # placeholders past each row's allocation (negative), at a random length per row + lens = torch.randint(0, width + 1, (pools, rows, 1, 1), generator=g, device="cuda") + cols = torch.arange(width, device="cuda").view(1, 1, 1, -1) + return torch.where(cols < lens, off, torch.full_like(off, -1)) + + +@pytest.mark.parametrize("dtype", [torch.int32, torch.int64]) +@pytest.mark.parametrize("width", [7, 129, 1100, 4500]) +@pytest.mark.parametrize("divisor", [1, 2, 10]) +@pytest.mark.parametrize("pools,pool_idx", [(1, 0), (3, 2)]) +@pytest.mark.parametrize("high", [1 << 20, (1 << 31) - 1]) +def test_refresh_is_the_stock_refresh_bit_for_bit( + parallel, dtype, width, divisor, pools, pool_idx, high +): + rows = 9 + for seed in range(3): + src = _offsets(pools, rows, width, dtype, seed, high) + md = SimpleNamespace(draft_kv_cache_block_offsets=src) + for num_seqs in (1, 5, rows): + ref = _worker(rows, width, pool_idx, divisor, fused=False) + got = _worker(rows, width, pool_idx, divisor, fused=True) + assert DFlashWorker._refresh_ctx_block_tables(ref, md, num_seqs) + assert DFlashWorker._refresh_ctx_block_tables(got, md, num_seqs) + torch.cuda.synchronize() + assert torch.equal(ref._ctx_block_tables, got._ctx_block_tables), (seed, num_seqs) + assert torch.equal(ref._ctx_block_counts, got._ctx_block_counts), (seed, num_seqs) + assert torch.equal( + src, _offsets(pools, rows, width, dtype, seed, high) + ) # the metadata is not edited + + +@pytest.mark.parametrize("width", [64, 3000]) +def test_refresh_in_a_cuda_graph(parallel, width): + rows, num_seqs = 8, 6 + src = _offsets(2, rows, width, torch.int32, 0) + md = SimpleNamespace(draft_kv_cache_block_offsets=src) + got = _worker(rows, width, 1, 2, fused=True) + DFlashWorker._refresh_ctx_block_tables(got, md, num_seqs) # compile outside capture + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g, stream=s): + DFlashWorker._refresh_ctx_block_tables(got, md, num_seqs) + torch.cuda.current_stream().wait_stream(s) + src.copy_(_offsets(2, rows, width, torch.int32, 1)) # a later step's offsets, same buffer + got._ctx_block_tables.fill_(77) + g.replay() + ref = _worker(rows, width, 1, 2, fused=False) + DFlashWorker._refresh_ctx_block_tables(ref, md, num_seqs) + torch.cuda.synchronize() + assert torch.equal(ref._ctx_block_tables, got._ctx_block_tables) + assert torch.equal(ref._ctx_block_counts, got._ctx_block_counts) + + +def test_unsupported_layouts_keep_the_stock_path(): + src = _offsets(1, 4, 16, torch.int32, 0) + tables = torch.zeros(4, 16, dtype=torch.int32, device="cuda") + counts = torch.zeros(4, dtype=torch.int64, device="cuda") + assert refresh_block_tables(src, 0, 4, 2, tables, counts) + assert not refresh_block_tables(src, 0, 4, 2, tables[:, :8], counts) # width mismatch + assert not refresh_block_tables(src, 0, 4, 2, tables, counts.int()) # counts not int64 + assert not refresh_block_tables(src.float(), 0, 4, 2, tables, counts) # not integer offsets + assert not refresh_block_tables(src, 0, 4, 0, tables, counts) # no divisor diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_ctx_advance.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_ctx_advance.py new file mode 100644 index 000000000000..15763bd310dc --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_ctx_advance.py @@ -0,0 +1,141 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""catalog/spec_decode/dspark_ctx_advance: the one-launch context advance + read view leaves the context lengths and +the read block table exactly as DFlashWorker._advance_ctx_len and the generation branch's read-view lines +(dflash_allocated_ctx_limit, dflash_rebased_ctx_view) do, and returns the same advertised lengths; eager and in a CUDA +graph.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_advance import ( + advance_ctx_and_read_view, +) +from tensorrt_llm._torch.speculative.dflash import ( + DFlashWorker, + dflash_allocated_ctx_limit, + dflash_rebased_ctx_view, +) + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU (Triton)") + +PAGE, BLOCK = 64, 3 + + +def _state(n, width, max_ctx, seed, dummy_rows): + g = torch.Generator(device="cuda").manual_seed(seed) + num_slots = n + 4 + dummy = num_slots - 1 + slots = torch.randperm(num_slots - 1, generator=g, device="cuda")[:n] + for r in range(dummy_rows): + slots[-1 - r] = dummy + ctx_len = torch.randint(0, max_ctx + 1, (num_slots,), generator=g, device="cuda").long() + ctx_len[slots[0]] = max_ctx - 1 # runs into the max_ctx clamp + ctx_len[dummy] = 0 + nacc = torch.randint(1, 5, (n,), generator=g, device="cuda").long() + rows_total = n + 2 + counts = torch.randint(0, width + 1, (rows_total,), generator=g, device="cuda").long() + base = torch.randint(0, 6, (num_slots,), generator=g, device="cuda").long() + base[slots[1 % n]] = (1 << 20) // PAGE # a context shorter than its base + tables = torch.randint(0, 1 << 16, (rows_total, width), generator=g, device="cuda").int() + read = torch.full_like(tables, -9) + rows = torch.arange(1, n + 1, device="cuda") + return dict( + slots=slots, + rows=rows, + nacc=nacc, + ctx_len=ctx_len, + max_ctx=max_ctx, + dummy=dummy, + counts=counts, + base=base, + tables=tables, + read=read, + ) + + +def _stock(st): + """_advance_ctx_len + the generation branch's read-view lines, verbatim.""" + w = object.__new__(DFlashWorker) + w._ctx_len, w._max_ctx, w._dummy_slot = st["ctx_len"], st["max_ctx"], st["dummy"] + DFlashWorker._advance_ctx_len(w, st["slots"], st["nacc"]) + num_ctx = w._ctx_len[st["slots"]] + allocated = dflash_allocated_ctx_limit(st["counts"], PAGE, BLOCK) + num_ctx = torch.minimum(num_ctx, allocated[st["rows"]]) + base_pages = st["base"][st["slots"]] + st["read"][st["rows"]], num_ctx = dflash_rebased_ctx_view( + st["tables"][st["rows"]], base_pages, num_ctx, PAGE + ) + return num_ctx + + +def _fused(st): + return advance_ctx_and_read_view( + st["slots"], + st["rows"], + st["nacc"], + st["ctx_len"], + st["max_ctx"], + st["dummy"], + st["counts"], + st["base"], + st["tables"], + st["read"], + PAGE, + BLOCK, + ) + + +def _copy(st): + return {k: v.clone() if isinstance(v, torch.Tensor) else v for k, v in st.items()} + + +@pytest.mark.parametrize("width", [5, 33, 2112]) +@pytest.mark.parametrize("n,dummy_rows", [(1, 0), (4, 1), (8, 3), (16, 0)]) +def test_advance_is_the_stock_advance_bit_for_bit(width, n, dummy_rows): + for seed in range(4): + st = _state(n, width, 135168, seed, dummy_rows) + ref, got = _copy(st), _copy(st) + ref_ctx = _stock(ref) + got_ctx = _fused(got) + torch.cuda.synchronize() + assert got_ctx is not None and got_ctx.dtype == ref_ctx.dtype + assert torch.equal(ref_ctx, got_ctx), seed + assert torch.equal(ref["ctx_len"], got["ctx_len"]), seed + assert torch.equal(ref["read"], got["read"]), seed + + +def test_advance_in_a_cuda_graph(): + st = _state(8, 64, 4096, 0, 2) + got = _copy(st) + _fused(got) # compile outside capture + got = _copy(st) + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g, stream=s): + out = _fused(got) + torch.cuda.current_stream().wait_stream(s) + for k in ( + "ctx_len", + "read", + ): # the capture ran nothing: reset to the step's inputs, then replay + got[k].copy_(st[k]) + g.replay() + ref = _copy(st) + ref_ctx = _stock(ref) + torch.cuda.synchronize() + assert torch.equal(ref_ctx, out) + assert torch.equal(ref["ctx_len"], got["ctx_len"]) and torch.equal(ref["read"], got["read"]) + + +def test_unsupported_layouts_keep_the_stock_path(): + st = _state(2, 8, 4096, 0, 0) + assert _fused(_copy(st)) is not None + bad = _copy(st) + bad["read"] = bad["read"].long() + assert _fused(bad) is None + bad = _copy(st) + bad["ctx_len"] = bad["ctx_len"].int() + assert _fused(bad) is None diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_ctx_store.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_ctx_store.py new file mode 100644 index 000000000000..d7a8ebddadc8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_ctx_store.py @@ -0,0 +1,266 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""catalog/spec_decode/dspark_ctx_store: the one-launch context store writes, bit for bit, what the stock path writes. + +Stock = DFlashForCausalLM.precompute_context_kv, the accept mask and DFlashWorker._store_context_kv_paged (flashinfer's +paged append per layer), run on a tiny real drafter (2 layers, 2 kv heads of 128, k_norm), with full RoPE and with +the MiniMax-M3.1 drafter's partial RoPE (rotary_dim 64 of head_dim 128: dims 64..127 are not rotated). Both write into +caches filled with a sentinel, compared in an int16 view, for the private arena (one stacked tensor) and the bound pool +(a list of per-layer tensors), with masked rows, clamped columns and a placeholder page. Cells written more than once +(see _single_writer) are excluded: stock's own value there is a race. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_store import ( + ctx_store_supported, + fold_k_norm_weight, + fused_ctx_store, +) +from tensorrt_llm._torch.models.modeling_dflash import DFlashForCausalLM +from tensorrt_llm._torch.speculative.dflash import DFlashWorker + + +def _has_trtllm_gen(): + if not torch.cuda.is_available(): + return False + from tensorrt_llm._utils import is_sm_100f + + return is_sm_100f() + + +pytestmark = pytest.mark.skipif( + not _has_trtllm_gen(), reason="the paged draft cache path needs an SM100-family GPU" +) + +HIDDEN, LAYERS, HEADS, KV_HEADS, HEAD_DIM, INTER, VOCAB, PAGE = 64, 2, 4, 2, 128, 128, 512, 32 + + +def _drafter(partial_rotary_factor=1.0): + from transformers import Qwen3Config + + from tensorrt_llm._torch.model_config import ModelConfig + + cfg = Qwen3Config.from_dict( + dict( + architectures=["DFlashDraftModel"], + model_type="qwen3", + block_size=4, + hidden_size=HIDDEN, + num_hidden_layers=LAYERS, + num_attention_heads=HEADS, + num_key_value_heads=KV_HEADS, + head_dim=HEAD_DIM, + intermediate_size=INTER, + hidden_act="silu", + rms_norm_eps=1e-6, + vocab_size=VOCAB, + max_position_embeddings=8192, + rope_theta=1e7, + rope_scaling=None, + partial_rotary_factor=partial_rotary_factor, + attention_bias=False, + torch_dtype="bfloat16", + num_target_layers=4, + tie_word_embeddings=False, + dflash_config={"target_layer_ids": [0, 1], "mask_token_id": VOCAB - 2}, + ) + ) + model = DFlashForCausalLM( + ModelConfig(pretrained_config=cfg, attn_backend="TRTLLM"), dflash_attention_backend="TRTLLM" + ).to("cuda") + g = torch.Generator().manual_seed(3) + + def rnd(*shape): + return (torch.randn(*shape, generator=g) * 0.05).to(torch.bfloat16) + + w = { + "fc.weight": rnd(HIDDEN, HIDDEN * 2), + "hidden_norm.weight": rnd(HIDDEN) + 1.0, + "norm.weight": rnd(HIDDEN) + 1.0, + } + for i in range(LAYERS): + p = f"layers.{i}." + w[p + "self_attn.q_proj.weight"] = rnd(HEADS * HEAD_DIM, HIDDEN) + w[p + "self_attn.k_proj.weight"] = rnd(KV_HEADS * HEAD_DIM, HIDDEN) + w[p + "self_attn.v_proj.weight"] = rnd(KV_HEADS * HEAD_DIM, HIDDEN) + w[p + "self_attn.o_proj.weight"] = rnd(HIDDEN, HEADS * HEAD_DIM) + w[p + "self_attn.q_norm.weight"] = rnd(HEAD_DIM) + 1.0 + w[p + "self_attn.k_norm.weight"] = rnd(HEAD_DIM) * 4 + 1.0 + w[p + "input_layernorm.weight"] = rnd(HIDDEN) + 1.0 + w[p + "post_attention_layernorm.weight"] = rnd(HIDDEN) + 1.0 + w[p + "mlp.gate_proj.weight"] = rnd(INTER, HIDDEN) + w[p + "mlp.up_proj.weight"] = rnd(INTER, HIDDEN) + w[p + "mlp.down_proj.weight"] = rnd(HIDDEN, INTER) + model.load_weights(w) + model._build_fused_kv_buffers() + assert model._get_cos_sin_cache().shape[-1] == int(HEAD_DIM * partial_rotary_factor) + return model + + +def _case(n_req, kp1, seed, placeholder): + g = torch.Generator(device="cuda").manual_seed(seed) + rows_total, pages_per_row = n_req + 2, 6 + cap = pages_per_row * PAGE + table = ( + torch.randperm(rows_total * pages_per_row, generator=g, device="cuda") + .int() + .view(rows_total, -1) + ) + if placeholder: + table[:, pages_per_row // 2 :] = ( + 0 # past each request's allocation: page 0, which another request owns + ) + rows = torch.arange(1, n_req + 1, device="cuda") + ctx_len = torch.randint(0, cap, (n_req,), generator=g, device="cuda") + ctx_len[0] = cap - 2 # the last request runs into the clamp + off = torch.arange(kp1, device="cuda") + cols = torch.minimum(ctx_len[:, None] + off, torch.tensor(cap - 1, device="cuda")) + nacc = torch.randint(1, kp1 + 1, (n_req,), generator=g, device="cuda") + proj = (torch.randn(n_req * kp1, HIDDEN, generator=g, device="cuda") * 2).bfloat16() + return dict( + table=table, + rows=rows, + pos=(ctx_len[:, None] + off).reshape(-1), + cols=cols.reshape(-1), + nacc=nacc, + proj=proj, + kp1=kp1, + pages=rows_total * pages_per_row, + rows_total=rows_total, + ppr=pages_per_row, + ) + + +def _stock(model, c, kv_buf): + """The worker's own code after the projection: precompute_context_kv, the mask, _store_context_kv_paged.""" + k, v = model.precompute_context_kv(c["proj"], c["pos"]) + mask = (torch.arange(c["kp1"], device="cuda")[None] < c["nacc"][:, None]).reshape(-1) + mb = mask.view(-1, 1, 1, 1).to(k.dtype) + k.mul_(mb) + v.mul_(mb) + worker = object.__new__(DFlashWorker) + worker._lean_glue = False + worker._ctx_paged_append = None + worker._ctx_block_tables = c["table"] + worker._ctx_block_indptr = torch.arange( + 0, (c["rows_total"] + 1) * c["ppr"], c["ppr"], device="cuda", dtype=torch.int32 + ) + worker._ctx_page_size = PAGE + worker._ctx_kv_buf = kv_buf + worker._ctx_kv_last_page_len = torch.full( + (c["rows_total"],), PAGE, device="cuda", dtype=torch.int32 + ) + rows = c["rows"][:, None].expand(-1, c["kp1"]).reshape(-1) + DFlashWorker._store_context_kv_paged(worker, k, v, rows, c["cols"]) + + +def _caches(c, layout): + full = torch.full( + (LAYERS, c["pages"], 2, KV_HEADS, PAGE, HEAD_DIM), 7.0, device="cuda" + ).bfloat16() + return full if layout == "stacked" else [full[i].clone() for i in range(LAYERS)] + + +def _single_writer(c): + """Cells exactly one row writes. A clamped column (several rows of one request) or a placeholder page (several + requests) is written more than once; stock's winner there is a race between the append's thread blocks and no + reader reads it (dflash_allocated_ctx_limit), so those cells are left out of the comparison.""" + rows = c["rows"].repeat_interleave(c["kp1"]) + key = c["table"][rows, c["cols"] // PAGE].long() * PAGE + c["cols"] % PAGE + uniq, counts = torch.unique(key, return_counts=True) + keep = torch.ones( + LAYERS, c["pages"], 2, KV_HEADS, PAGE, HEAD_DIM, dtype=torch.bool, device="cuda" + ) + for k in uniq[counts > 1].tolist(): + keep[:, k // PAGE, :, :, k % PAGE] = False + return keep + + +def _as_int16(kv_buf): + t = kv_buf if isinstance(kv_buf, torch.Tensor) else torch.stack(kv_buf) + return t.view(torch.int16) + + +@pytest.mark.parametrize("partial_rotary_factor", [1.0, 0.5]) +@pytest.mark.parametrize("layout", ["stacked", "list"]) +@pytest.mark.parametrize("n_req,kp1", [(1, 4), (3, 4), (2, 8)]) +@pytest.mark.parametrize("placeholder", [False, True]) +@pytest.mark.parametrize("columns", ["given", "from_counts", "from_max_col"]) +@pytest.mark.parametrize("fold", [False, True]) +def test_fused_store_is_the_stock_store_bit_for_bit( + layout, n_req, kp1, placeholder, partial_rotary_factor, columns, fold +): + """fold: TAILBK_SPEC F, the k_norm weight multiply inside the kernel (the drafter's weight is bf16, so it is + taken); the stores must equal stock's either way.""" + model = _drafter(partial_rotary_factor) + if fold: + assert ( + model._k_norm_stacked.dtype == torch.bfloat16 and model._k_norm_stacked.is_contiguous() + ) + fold_k_norm_weight(fold) + try: + _check_fused_store(model, layout, n_req, kp1, placeholder, columns) + finally: + fold_k_norm_weight(False) + + +def _check_fused_store(model, layout, n_req, kp1, placeholder, columns): + for seed in range(4): + c = _case(n_req, kp1, seed, placeholder) + ref, got = _caches(c, layout), _caches(c, layout) + assert ctx_store_supported(model, got) + _stock(model, c, ref) + # The case's columns are the positions clamped to the rows' pages (every row holds ppr pages), so the + # kernel's own derivation (S10c-A2: from the page counts, or from the arena's last column) must match them. + kwargs = {} + if columns == "from_counts": + kwargs["capacity_counts"] = torch.full( + (c["rows_total"],), c["ppr"], dtype=torch.int64, device="cuda" + ) + elif columns == "from_max_col": + kwargs["max_col"] = c["ppr"] * PAGE - 1 + cols = c["cols"] if columns == "given" else None + assert fused_ctx_store( + model, + c["proj"], + c["pos"], + c["nacc"], + c["rows"], + cols, + c["table"], + got, + c["kp1"], + **kwargs, + ) + torch.cuda.synchronize() + keep = _single_writer(c) + assert torch.equal(_as_int16(ref)[keep], _as_int16(got)[keep]), f"seed {seed}" + + +def test_unsupported_drafters_keep_the_stock_path(): + model = _drafter() + kv = torch.zeros(LAYERS, 4, 2, KV_HEADS, PAGE, HEAD_DIM, device="cuda", dtype=torch.bfloat16) + assert ctx_store_supported(model, kv) + assert not ctx_store_supported(model, kv.float()) # not a bf16 cache + assert not ctx_store_supported(model, kv[:, :, :1]) # not a [K | V] HND page + no_norm = SimpleNamespace( + **{k: getattr(model, k) for k in ("_k_norm_eps", "_is_neox", "_num_kv_heads", "_head_dim")} + ) + no_norm._k_norm_stacked = None + assert not ctx_store_supported(no_norm, kv) + assert not fused_ctx_store(no_norm, None, None, None, None, None, None, kv, 4) + # Derived columns need exactly one bound: the page counts (int64) or the arena's last column. + assert not fused_ctx_store(model, None, None, None, None, None, None, kv, 4) + # A rotary dim the kernel cannot split into power-of-two halves (48 of 128) keeps the stock path. + odd_rotary = SimpleNamespace( + **{ + k: getattr(model, k) + for k in ("_k_norm_eps", "_is_neox", "_num_kv_heads", "_head_dim", "_k_norm_stacked") + } + ) + odd_rotary._get_cos_sin_cache = lambda: torch.zeros(16, 48, device="cuda") + assert not ctx_store_supported(odd_rotary, kv) diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_drafter_inputs.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_drafter_inputs.py new file mode 100644 index 000000000000..14e64c41d22c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_drafter_inputs.py @@ -0,0 +1,105 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""catalog/spec_decode/dspark_drafter_inputs through the M3.1 targets' S10c-C entry: the drafter's step inputs (bonus, +query / ctx positions, noise block) are bit for bit what the stock lines of DFlashWorker.prepare_1st_drafter_inputs +compute, with the target's own embedding (M3EmbedTokens), eager and in a CUDA graph.""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.speculative.dflash import dflash_noise_block_embedding + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU (Triton)") + +_ROOT = "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3" +VOCAB, HIDDEN, MASK_ID, SLOTS = 1000, 6144, 997, 20 + + +def _target(tp): + return importlib.import_module(f"{_ROOT}.minimax_m31_nvfp4__sm_103__{tp}.dspark") + + +def _stock(embed, accepted, nacc, slots, ctx_len, max_ctx, trained, block, kp1): + """The stock lines of prepare_1st_drafter_inputs' generation branch, verbatim.""" + bonus_idx = (nacc - 1).clamp_min(0).long().unsqueeze(1) + bonus = accepted.gather(1, bonus_idx).squeeze(1).long() + ctx_len_gen = ctx_len[slots] + j_block = torch.arange(0, block, dtype=torch.long, device="cuda") + offsets_kp1 = torch.arange(0, kp1, dtype=torch.long, device="cuda") + ctx_len_now = (ctx_len_gen + nacc.long()).clamp_(max=max_ctx) + query_position_ids = ctx_len_now.unsqueeze(1) + j_block.unsqueeze(0) + ctx_position_ids = ctx_len_gen.unsqueeze(1) + offsets_kp1.unsqueeze(0) + noise = dflash_noise_block_embedding(embed, bonus, MASK_ID, block, trained) + return bonus, query_position_ids, ctx_position_ids, noise + + +def _case(n, k, acc_dtype, seed, max_ctx): + g = torch.Generator(device="cuda").manual_seed(seed) + kp1 = k + 1 + accepted = torch.randint(0, VOCAB, (n, kp1), generator=g, device="cuda").to(acc_dtype) + accepted[0, :] = -1 # a padding request: its bonus id is outside the table + nacc = torch.randint(1, kp1 + 1, (n,), generator=g, device="cuda").int() + slots = torch.randperm(SLOTS, generator=g, device="cuda")[:n] + ctx_len = torch.randint(0, max_ctx + 1, (SLOTS,), generator=g, device="cuda").long() + ctx_len[slots[-1]] = max_ctx - 1 # runs into the clamp + return accepted, nacc, slots, ctx_len + + +def _embed(tp): + g = torch.Generator(device="cuda").manual_seed(7) + table = torch.randn(VOCAB, HIDDEN, generator=g, device="cuda").bfloat16() + return _target(tp).M3EmbedTokens(SimpleNamespace(w={"embed": table})) + + +def _equal(a, b): + return all( + x.dtype == y.dtype and x.shape == y.shape and torch.equal(x, y) for x, y in zip(a, b) + ) + + +@pytest.mark.parametrize("tp", ["tp4", "tp8"]) +@pytest.mark.parametrize("k", [3, 7]) +@pytest.mark.parametrize("acc_dtype", [torch.int32, torch.int64]) +@pytest.mark.parametrize("trained", [False, True]) +def test_inputs_are_the_stock_inputs_bit_for_bit(tp, k, acc_dtype, trained): + embed = _embed(tp) + mask_row = (torch.randn(HIDDEN, device="cuda") * 3).bfloat16() if trained else None + for n in (1, 3, 8, 16): + for seed in range(3): + accepted, nacc, slots, ctx_len = _case(n, k, acc_dtype, seed, max_ctx=4096) + args = (embed, accepted, nacc, slots, ctx_len, 4096) + ref = _stock(*args, mask_row, k, k + 1) + got = _target(tp).s10c_c_drafter_inputs(*args, MASK_ID, mask_row, k, k + 1) + assert got is not None + torch.cuda.synchronize() + assert _equal(ref, got), (n, seed) + + +def test_inputs_in_a_cuda_graph(): + embed = _embed("tp8") + accepted, nacc, slots, ctx_len = _case(8, 3, torch.int32, 0, max_ctx=4096) + fn = _target("tp8").s10c_c_drafter_inputs + fn(embed, accepted, nacc, slots, ctx_len, 4096, MASK_ID, None, 3, 4) # compile outside capture + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g, stream=s): + got = fn(embed, accepted, nacc, slots, ctx_len, 4096, MASK_ID, None, 3, 4) + torch.cuda.current_stream().wait_stream(s) + a2, n2, s2, c2 = _case(8, 3, torch.int32, 1, max_ctx=4096) + accepted.copy_(a2), nacc.copy_(n2), slots.copy_(s2), ctx_len.copy_(c2) + g.replay() + ref = _stock(embed, accepted, nacc, slots, ctx_len, 4096, None, 3, 4) + torch.cuda.synchronize() + assert _equal(ref, got) + + +def test_other_embeddings_keep_the_stock_path(): + accepted, nacc, slots, ctx_len = _case(2, 3, torch.int32, 0, max_ctx=4096) + other = torch.nn.Embedding(VOCAB, HIDDEN).cuda().bfloat16() + fn = _target("tp8").s10c_c_drafter_inputs + assert fn(other, accepted, nacc, slots, ctx_len, 4096, MASK_ID, None, 3, 4) is None diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_forward_indices.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_forward_indices.py new file mode 100644 index 000000000000..5b95d126b36e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_forward_indices.py @@ -0,0 +1,97 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""catalog/spec_decode/dspark_forward_indices: the paged dflash_forward's index prelude in one launch is bit for bit +the stock lines of DFlashForCausalLM.dflash_forward (cache_seqlens, cache_batch_idx, seq_lens_after, kv_indptr, +kv_last_page_len, append_positions, query_positions_flat), eager and in a CUDA graph.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_forward_indices import ( + forward_indices, +) + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU (Triton)") + + +def _stock(num_ctx, batch_idx, query_positions, block_size, pages_per_slot, page_size): + """The stock lines of dflash_forward's paged branch, verbatim.""" + B = query_positions.shape[0] + cache_seqlens_i32 = num_ctx[:B].to(torch.int32) + cache_batch_idx_i32 = batch_idx.to(torch.int32) + seq_lens_after = cache_seqlens_i32 + block_size + kv_indptr = torch.arange( + 0, (B + 1) * pages_per_slot, pages_per_slot, dtype=torch.int32, device="cuda" + ) + kv_last_page_len = ((seq_lens_after - 1) % page_size) + 1 + block_offsets = torch.arange(block_size, dtype=torch.int32, device="cuda") + append_positions = (cache_seqlens_i32.view(-1, 1) + block_offsets).reshape(-1).contiguous() + query_positions_flat_i32 = query_positions.reshape(-1).to(torch.int32) + return ( + cache_seqlens_i32, + cache_batch_idx_i32, + seq_lens_after, + kv_indptr, + kv_last_page_len, + append_positions, + query_positions_flat_i32, + ) + + +def _case(B, block, dtype, seed, strided): + g = torch.Generator(device="cuda").manual_seed(seed) + num_ctx = torch.randint(0, 1 << 20, (B,), generator=g, device="cuda").to(dtype) + num_ctx[0] = 0 # an empty context: seq_lens_after = block + batch_idx = torch.randperm(B + 5, generator=g, device="cuda")[:B].to(dtype) + if strided: + q = torch.randint(0, 1 << 21, (block, B), generator=g, device="cuda").to(dtype).t() + else: + q = torch.randint(0, 1 << 21, (B, block), generator=g, device="cuda").to(dtype) + return num_ctx, batch_idx, q + + +def _equal(a, b): + return all( + x.dtype == y.dtype and x.shape == y.shape and torch.equal(x, y) for x, y in zip(a, b) + ) + + +@pytest.mark.parametrize("dtype", [torch.int32, torch.int64]) +@pytest.mark.parametrize("block", [3, 4, 7, 8]) +@pytest.mark.parametrize("page_size,pages_per_slot", [(64, 66), (32, 2049), (16, 5)]) +@pytest.mark.parametrize("strided", [False, True]) +def test_indices_are_the_stock_lines_bit_for_bit(dtype, block, page_size, pages_per_slot, strided): + for B in (1, 2, 3, 8, 16, 33): + for seed in range(3): + num_ctx, batch_idx, q = _case(B, block, dtype, seed, strided) + ref = _stock(num_ctx, batch_idx, q, block, pages_per_slot, page_size) + got = forward_indices(num_ctx, batch_idx, q, block, pages_per_slot, page_size) + assert got is not None + torch.cuda.synchronize() + assert _equal(ref, got), (B, seed) + + +def test_indices_in_a_cuda_graph(): + num_ctx, batch_idx, q = _case(8, 4, torch.int64, 0, False) + forward_indices(num_ctx, batch_idx, q, 4, 66, 64) # compile outside capture + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g, stream=s): + got = forward_indices(num_ctx, batch_idx, q, 4, 66, 64) + torch.cuda.current_stream().wait_stream(s) + n2, b2, q2 = _case(8, 4, torch.int64, 1, False) + num_ctx.copy_(n2), batch_idx.copy_(b2), q.copy_(q2) + g.replay() + ref = _stock(num_ctx, batch_idx, q, 4, 66, 64) + torch.cuda.synchronize() + assert _equal(ref, got) + + +def test_unsupported_layouts_keep_the_stock_path(): + num_ctx, batch_idx, q = _case(2, 4, torch.int64, 0, False) + assert forward_indices(num_ctx, batch_idx, q, 4, 66, 64) is not None + assert forward_indices(num_ctx.float(), batch_idx, q, 4, 66, 64) is None + assert forward_indices(num_ctx, batch_idx, q[:, :3], 4, 66, 64) is None + assert forward_indices(num_ctx, batch_idx[:1], q, 4, 66, 64) is None diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_bias.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_bias.py new file mode 100644 index 000000000000..0263a0fe6590 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_bias.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""`spec_decode/dspark_markov_bias` (TAILBK_SPEC S4L): its two launches around the bias GEMM equal, bit for bit, the +in-tree batched biased-logits lines (prevs cat / casts, markov_prev_embeddings, F.linear, fp32 widen + add), on the +M3.1 TP8 shard (25008 columns, rank 256), K = 3 / 7, batches 1..8, with out-of-vocab anchors (-1, V, a large id), +extreme logits, eagerly and replayed from a CUDA graph; ``supports`` declines what the kernels do not take.""" + +import pytest +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_bias as mb, +) +from tensorrt_llm._torch.models.modeling_speculative import dspark_markov_step_bias + +assert torch.cuda.is_available(), "dspark_markov_bias requires a CUDA device" + +V = 200064 +R = 256 +N = 25008 + + +def _weights(seed=7): + g = torch.Generator(device="cuda").manual_seed(seed) + w1 = (torch.randn(V, R, device="cuda", generator=g) * 0.05).bfloat16() + w2 = (torch.randn(N, R, device="cuda", generator=g) * 0.05).bfloat16() + return w1, w2 + + +def _case(seed, B, K, kind="random"): + g = torch.Generator(device="cuda").manual_seed(seed) + logits = (torch.randn(B, K, N, device="cuda", generator=g) * 4).bfloat16() + if kind == "extremes": + pool = torch.tensor([3.0e38, -3.0e38, 2.0**-133, -(2.0**-133), 0.0, -0.0, float("-inf"), 1.0], + device="cuda") # fmt: skip + logits = pool[ + torch.randint(0, pool.numel(), logits.shape, device="cuda", generator=g) + ].bfloat16() + first_prev = torch.randint(0, V, (B,), device="cuda", generator=g) + tokens = torch.randint(0, V, (B, K), device="cuda", generator=g, dtype=torch.int32) + if kind == "oov": + first_prev[0] = -1 + if B > 1: + first_prev[1] = V + tokens[0, 0] = -1 + tokens[-1, K - 2] = V + 12345 + return logits, first_prev, tokens + + +def _reference(logits, first_prev, tokens, w1, w2): + """The in-tree batched form (tp targets' _biased_shard_logits, fp32 + batched).""" + shard = logits.float() + B, K, n = shard.shape + prevs = torch.cat([first_prev.view(B, 1), tokens[:, : K - 1].long()], dim=1) + bias = dspark_markov_step_bias(prevs.reshape(-1), w1, w2).view(B, K, n) + return shard + bias.to(shard.dtype) + + +def _fused(logits, first_prev, tokens, w1, w2): + B, K, n = logits.shape + bias = F.linear(mb.prev_embeddings(first_prev, tokens, w1), w2) + return mb.widen_add(logits, bias.view(B, K, n)) + + +def _bits_equal(a, b): + return ( + a.dtype == b.dtype + and a.shape == b.shape + and torch.equal(a.view(torch.int32), b.view(torch.int32)) + ) + + +@pytest.mark.parametrize("K", [3, 7]) +@pytest.mark.parametrize("B", [1, 2, 8]) +@pytest.mark.parametrize("kind", ["random", "oov", "extremes"]) +def test_bitwise_in_tree(B, K, kind): + w1, w2 = _weights() + for seed in range(3): + logits, first_prev, tokens = _case(100 * seed + B * 10 + K, B, K, kind) + assert mb.supports(logits, first_prev, tokens, w1) + got = _fused(logits, first_prev, tokens, w1, w2) + want = _reference(logits, first_prev, tokens, w1, w2) + assert _bits_equal(got, want), f"B {B} K {K} {kind} seed {seed}" + + +def test_prev_embeddings_rows(): + """Each row is w1[prev] bit for bit, or +0.0 for an anchor outside [0, V); int32 and int64 ids.""" + w1, _ = _weights() + B, K = 3, 7 + first_prev = torch.tensor([5, -1, V], device="cuda") + tokens = torch.tensor([[1, 2, 3, -7, V - 1, 0, 9]] * B, device="cuda", dtype=torch.int32) + for fp, tk in ((first_prev, tokens), (first_prev.int(), tokens.long())): + e = mb.prev_embeddings(fp, tk, w1) + for b in range(B): + for k in range(K): + prev = int(fp[b]) if k == 0 else int(tk[b, k - 1]) + row = e[b * K + k] + if 0 <= prev < V: + assert torch.equal(row.view(torch.int16), w1[prev].view(torch.int16)), (b, k) + else: + assert torch.equal( + row.view(torch.int16), torch.zeros_like(row).view(torch.int16) + ), (b, k) + + +def test_cuda_graph_replay(): + w1, w2 = _weights() + B, K = 1, 7 + logits, first_prev, tokens = _case(1, B, K) + _fused(logits, first_prev, tokens, w1, w2) # compile outside capture + torch.cuda.synchronize() + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g): + out = _fused(logits, first_prev, tokens, w1, w2) + torch.cuda.current_stream().wait_stream(s) + for rep in range(6): + l2, f2, t2 = _case(500 + rep, B, K, "oov" if rep % 2 else "random") + logits.copy_(l2) + first_prev.copy_(f2) + tokens.copy_(t2) + g.replay() + torch.cuda.synchronize() + assert _bits_equal(out, _reference(logits, first_prev, tokens, w1, w2)), f"replay {rep}" + + +def test_supports_declines(): + w1, _ = _weights() + logits, first_prev, tokens = _case(3, 2, 7) + assert mb.supports(logits, first_prev, tokens, w1) + assert not mb.supports(logits.float(), first_prev, tokens, w1) + assert not mb.supports(logits.transpose(0, 1), first_prev, tokens, w1) + assert not mb.supports(logits, first_prev.float(), tokens, w1) + assert not mb.supports(logits, first_prev, tokens[:, :3], w1) + assert not mb.supports(logits, first_prev[:1], tokens, w1) + assert not mb.supports(logits, first_prev, tokens, w1[:, :200]) + assert not mb.supports(logits, first_prev, tokens, w1.float()) diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_chain_ll_op_matrix.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_chain_ll_op_matrix.py new file mode 100644 index 000000000000..d08d25d97357 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_chain_ll_op_matrix.py @@ -0,0 +1,32 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for `spec_decode/dspark_markov_chain`'s LL exchange variant +(``_dspark_markov_chain_ll_op_matrix.py`` beside this file: LL tokens == the S4B8 batch-1 kernel's, bitwise, on real +ranks, including forced cross-rank ties, back-to-back launches and CUDA-graph replays). + +World 4 is collected here. World 8 spans two GB300 trays and runs as one srun task per rank (see the matrix's +docstring). +""" + +import importlib.util +from pathlib import Path + +import pytest +import torch + +assert torch.cuda.is_available(), "dspark_markov_chain requires CUDA devices" + + +def _rank_job(): + # By path: other test directories ship a module of the same name. + path = Path(__file__).resolve().with_name("_rank_job.py") + spec = importlib.util.spec_from_file_location("_spec_decode_rank_job", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +# Spawns 4 rank processes over the visible devices; under xdist workers would fight for them. +@pytest.mark.no_xdist +def test_dspark_markov_chain_ll_op_matrix() -> None: + _rank_job().run("dspark_markov_chain_ll", 4) diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_chain_op_matrix.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_chain_op_matrix.py new file mode 100644 index 000000000000..505570c17ef5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_markov_chain_op_matrix.py @@ -0,0 +1,32 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for `spec_decode/dspark_markov_chain`'s certification matrix +(``_dspark_markov_chain_op_matrix.py`` beside this file, which drives the entry's ``.CELLS``). + +World 4 is collected here. The world-8 cells span two GB300 trays and run as one srun task per rank (see the +matrix's docstring), like ``comm/_mnnvl_allreduce_op_matrix.py``. +""" + +import importlib.util +from pathlib import Path + +import pytest +import torch + +assert torch.cuda.is_available(), "dspark_markov_chain requires CUDA devices" + + +def _rank_job(): + # By path: other test directories ship a module of the same name, and `import _rank_job` would get whichever + # one a collected sibling imported first. + path = Path(__file__).resolve().with_name("_rank_job.py") + spec = importlib.util.spec_from_file_location("_spec_decode_rank_job", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +# Spawns 4 rank processes over the visible devices; under xdist workers would fight for them. +@pytest.mark.no_xdist +def test_dspark_markov_chain_op_matrix() -> None: + _rank_job().run("dspark_markov_chain", 4) diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_qk_norm_rope.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_qk_norm_rope.py new file mode 100644 index 000000000000..28d9486fce85 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_qk_norm_rope.py @@ -0,0 +1,244 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""catalog/spec_decode/dspark_qk_norm_rope: the drafter block's q / k RMSNorm + partial NeoX RoPE in one launch is bit +for bit the stock lines of DFlashForCausalLM.dflash_forward's flashinfer-RoPE branch (q_norm, k_norm, then +flashinfer_apply_rope_with_cos_sin_cache_inplace), at the M3.1 TP8 and TP4 rank head counts, 16 seeds, rows 3..56, +real-scale activations with tiny / huge / all-zero heads, eager and in a CUDA graph. DRAFT_APPEND_FUSE +(``qk_norm_rope_append``): the same launch's K / V pages are bit for bit flashinfer.page.append_paged_kv_cache's +(HND), with blocks crossing page boundaries, slots re-written, int32 / int64 indices, eager and in a CUDA graph.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_qk_norm_rope import ( + append_supported, + qk_norm_rope, + qk_norm_rope_append, +) +from tensorrt_llm._torch.custom_ops import ( + flashinfer_apply_rope_with_cos_sin_cache_inplace as fi_rope, +) +from tensorrt_llm._torch.modules.rms_norm import RMSNorm + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU (Triton)") + +HD, ROT, EPS, MAX_POS = 128, 64, 1e-6, 1 << 20 + + +def _cache(): + inv = 1.0 / (1e7 ** (torch.arange(0, ROT, 2, device="cuda", dtype=torch.float32) / ROT)) + ang = torch.arange(MAX_POS, device="cuda", dtype=torch.float32)[:, None] * inv[None, :] + return torch.cat([ang.cos(), ang.sin()], dim=1).contiguous() + + +_CS = None + + +def _cs(): + global _CS + if _CS is None: + _CS = _cache() + return _CS + + +def _norms(seed): + g = torch.Generator(device="cuda").manual_seed(1000 + seed) + qn = RMSNorm(hidden_size=HD, eps=EPS, dtype=torch.bfloat16, device="cuda") + kn = RMSNorm(hidden_size=HD, eps=EPS, dtype=torch.bfloat16, device="cuda") + with torch.no_grad(): + qn.weight.copy_((torch.randn(HD, generator=g, device="cuda") * 0.3 + 1).bfloat16()) + kn.weight.copy_((torch.randn(HD, generator=g, device="cuda") * 0.3 + 1).bfloat16()) + return qn, kn + + +def _qkv(T, hq, hkv, seed): + """Real-scale qkv rows: per-head scales spanning 1e-3..3e2, some all-zero heads, a few outlier channels.""" + g = torch.Generator(device="cuda").manual_seed(seed) + width = (hq + 2 * hkv) * HD + x = torch.randn(T, hq + 2 * hkv, HD, generator=g, device="cuda") + scale = torch.exp( + torch.empty(T, hq + 2 * hkv, 1, device="cuda").uniform_(-7.0, 5.7, generator=g) + ) + x = x * scale + x[:, :, :4] *= 20.0 # outlier channels + if T > 1: + x[1, 0] = 0.0 # an all-zero q head + x[T - 1, hq] = 0.0 # an all-zero k head + pos = torch.randint(0, MAX_POS, (T,), generator=g, device="cuda", dtype=torch.int32) + return x.reshape(T, width).bfloat16(), pos + + +def _stock(qkv, hq, hkv, qn, kn, pos): + qs, ks = hq * HD, hkv * HD + q = qn(qkv[:, :qs].reshape(-1, HD)).view(-1, qs) + k = kn(qkv[:, qs : qs + ks].reshape(-1, HD)).view(-1, ks) + fi_rope(pos, q, k, HD, _cs(), True) + return q, k + + +@pytest.mark.parametrize("hq,hkv", [(8, 1), (16, 1)], ids=["tp8", "tp4"]) +def test_bitwise_to_the_stock_lines(hq, hkv): + worst = 0 + for seed in range(16): + qn, kn = _norms(seed) + for T in (3, 4, 7, 8, 12, 24, 28, 32, 56): + qkv, pos = _qkv(T, hq, hkv, 100 * seed + T) + rq, rk = _stock(qkv, hq, hkv, qn, kn, pos) + got = qk_norm_rope(qkv, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), True) + assert got is not None + torch.cuda.synchronize() + bad = int((got[0] != rq).sum() + (got[1] != rk).sum()) + worst = max(worst, bad) + assert bad == 0, (seed, T, bad) + assert worst == 0 + + +def test_in_a_cuda_graph(): + hq, hkv = 8, 1 + qn, kn = _norms(0) + qkv, pos = _qkv(32, hq, hkv, 1) + qk_norm_rope(qkv, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), True) # compile outside capture + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g, stream=s): + got = qk_norm_rope(qkv, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), True) + torch.cuda.current_stream().wait_stream(s) + q2, p2 = _qkv(32, hq, hkv, 2) + qkv.copy_(q2), pos.copy_(p2) + g.replay() + rq, rk = _stock(qkv, hq, hkv, qn, kn, pos) + torch.cuda.synchronize() + assert torch.equal(got[0], rq) and torch.equal(got[1], rk) + + +def test_unsupported_inputs_keep_the_stock_path(): + hq, hkv = 8, 1 + qn, kn = _norms(0) + qkv, pos = _qkv(4, hq, hkv, 3) + assert ( + qk_norm_rope(qkv, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), False) is None + ) # interleaved RoPE + assert qk_norm_rope(qkv.float(), hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), True) is None + assert qk_norm_rope(qkv, hq * HD, hkv * HD, HD, qn, kn, pos.long(), _cs(), True) is None + gem = RMSNorm(hidden_size=HD, eps=EPS, dtype=torch.bfloat16, device="cuda", use_gemma=True) + assert qk_norm_rope(qkv, hq * HD, hkv * HD, HD, gem, kn, pos, _cs(), True) is None + + +PS = 32 # the draft pool's page size + + +def _paged(B, K, hkv, pps, ctx, seed, idx_dtype=torch.int32): + """A random-filled HND cache [pages, 2, hkv, PS, HD], per-request block tables, and the append rows of a K-token + block per request at ctx[b] .. ctx[b] + K - 1 (as dflash_forward builds them).""" + g = torch.Generator(device="cuda").manual_seed(seed) + pages = B * pps + 3 + cache = torch.randn(pages, 2, hkv, PS, HD, generator=g, device="cuda").bfloat16() + perm = torch.randperm(pages, generator=g, device="cuda")[: B * pps] + kv_indices = perm.to(idx_dtype).contiguous() + kv_indptr = torch.arange(0, (B + 1) * pps, pps, device="cuda", dtype=torch.int32) + bidx = torch.arange(B, device="cuda", dtype=idx_dtype).repeat_interleave(K).contiguous() + apos = (torch.tensor(ctx, device="cuda", dtype=torch.int64).view(-1, 1) + + torch.arange(K, device="cuda")).reshape(-1).to(idx_dtype).contiguous() # fmt: skip + return cache, bidx, apos, kv_indices, kv_indptr + + +def _stock_append(qkv, hq, hkv, qn, kn, pos, cache, bidx, apos, kv_indices, kv_indptr): + import flashinfer + + q, k = _stock(qkv, hq, hkv, qn, kn, pos) + qs, ks = hq * HD, hkv * HD + v = qkv[:, qs + ks : qs + 2 * ks].reshape(-1, hkv, HD) + seq = torch.bincount(bidx.long(), minlength=kv_indptr.numel() - 1) + last = ((apos.view(seq.numel(), -1)[:, -1].long() % PS) + 1).to(torch.int32) + i32 = lambda t: t.to(torch.int32).contiguous() # noqa: E731 -- flashinfer takes int32 index tensors + flashinfer.page.append_paged_kv_cache(k.reshape(-1, hkv, HD).contiguous(), v, i32(bidx), i32(apos), cache, + i32(kv_indices), kv_indptr, last, kv_layout="HND") # fmt: skip + return q, k + + +@pytest.mark.parametrize("idx_dtype", [torch.int32, torch.int64], ids=["i32", "i64"]) +@pytest.mark.parametrize("hq,hkv", [(8, 1), (16, 1)], ids=["tp8", "tp4"]) +def test_append_pages_bitwise_to_flashinfer(hq, hkv, idx_dtype): + K, pps = 7, 4 + for seed in range(6): + qn, kn = _norms(seed) + for B, ctx in ( + (1, [29]), + (1, [0]), + (2, [30, 61]), + (3, [5, 95, 31]), + ): # blocks crossing pages, page 0, ... + T = B * K + qkv, pos = _qkv(T, hq, hkv, 500 * seed + 10 * B + ctx[0]) + cache, bidx, apos, kvi, kvp = _paged(B, K, hkv, pps, ctx, seed, idx_dtype) + ref_cache = cache.clone() + rq, rk = _stock_append(qkv, hq, hkv, qn, kn, pos, ref_cache, bidx, apos, kvi, kvp) + assert append_supported(qkv, hkv * HD, cache, bidx, apos, kvi, kvp) + got = qk_norm_rope_append(qkv, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), True, cache, bidx, apos, kvi, + kvp) # fmt: skip + assert got is not None + torch.cuda.synchronize() + assert torch.equal(got[0], rq) and torch.equal(got[1], rk), (seed, B, ctx) + assert torch.equal(cache.view(torch.int16), ref_cache.view(torch.int16)), (seed, B, ctx) + # Slot reuse: the same rows written again with new values (a rejected draft's slots are re-appended). + qkv2, _ = _qkv(T, hq, hkv, 777 + seed) + _stock_append(qkv2, hq, hkv, qn, kn, pos, ref_cache, bidx, apos, kvi, kvp) + qk_norm_rope_append( + qkv2, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), True, cache, bidx, apos, kvi, kvp + ) + torch.cuda.synchronize() + assert torch.equal(cache.view(torch.int16), ref_cache.view(torch.int16)), ( + "reuse", + seed, + B, + ctx, + ) + + +def test_append_in_a_cuda_graph(): + hq, hkv, K, B, pps = 8, 1, 7, 1, 4 + qn, kn = _norms(0) + qkv, pos = _qkv(B * K, hq, hkv, 5) + cache, bidx, apos, kvi, kvp = _paged(B, K, hkv, pps, [29], 0) + qk_norm_rope_append( + qkv, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), True, cache, bidx, apos, kvi, kvp + ) + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s): + with torch.cuda.graph(g, stream=s): + got = qk_norm_rope_append(qkv, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), True, cache, bidx, apos, kvi, + kvp) # fmt: skip + torch.cuda.current_stream().wait_stream(s) + for rep, c in enumerate((29, 60, 3, 95)): + q2, p2 = _qkv(B * K, hq, hkv, 40 + rep) + qkv.copy_(q2), pos.copy_(p2) + _, _, a2, _, _ = _paged(B, K, hkv, pps, [c], 0) + apos.copy_(a2) + ref_cache = cache.clone() + rq, rk = _stock_append(qkv, hq, hkv, qn, kn, pos, ref_cache, bidx, apos, kvi, kvp) + g.replay() + torch.cuda.synchronize() + assert torch.equal(got[0], rq) and torch.equal(got[1], rk), rep + assert torch.equal(cache.view(torch.int16), ref_cache.view(torch.int16)), rep + + +def test_append_unsupported_keeps_the_stock_path(): + hq, hkv, K = 8, 1, 7 + qn, kn = _norms(0) + qkv, pos = _qkv(K, hq, hkv, 9) + cache, bidx, apos, kvi, kvp = _paged(1, K, hkv, 4, [10], 0) + args = (qkv, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), True) + assert qk_norm_rope_append(*args, cache.float(), bidx, apos, kvi, kvp) is None # fp32 cache + assert ( + qk_norm_rope_append(*args, cache.transpose(2, 3), bidx, apos, kvi, kvp) is None + ) # NHD / strided + assert qk_norm_rope_append(*args, cache, bidx[:-1], apos, kvi, kvp) is None # rows mismatch + assert qk_norm_rope_append(*args, cache, bidx.float(), apos, kvi, kvp) is None + before = cache.clone() + assert qk_norm_rope_append(qkv, hq * HD, hkv * HD, HD, qn, kn, pos, _cs(), False, cache, bidx, apos, kvi, + kvp) is None # interleaved RoPE: nothing written # fmt: skip + assert torch.equal(cache.view(torch.int16), before.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_spec_strict_accept.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_spec_strict_accept.py new file mode 100644 index 000000000000..b725150e36c4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_spec_strict_accept.py @@ -0,0 +1,158 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""`spec_decode/strict_accept`: both entries bitwise to their in-tree torch path over every cell of their `.CELLS`. + +One GPU; each cell is also run on a second random draw and inside a CUDA graph (captured after a warm call, replayed +on new inputs written into the captured buffers). +""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.strict_accept import ( + SpecKvRewind, + SpecNextNewTokens, + SpecStrictAccept, + spec_kv_rewind, + spec_next_new_tokens, + spec_strict_accept, +) + +assert torch.cuda.is_available(), "strict_accept requires a CUDA device" + +VOCAB = 200064 + + +def _strict_inputs(spec, gen): + n_req, draft_len = spec["n_req"], spec["draft_len"] + t_dtype = spec.get("target_dtype", torch.int32) + k1 = draft_len + 1 + target = torch.randint(0, VOCAB, (n_req * k1,), generator=gen, device="cuda").to(t_dtype) + draft = target.view(n_req, k1)[:, :draft_len].clone().to(spec.get("draft_dtype", torch.int32)) + # A random prefix of each request's drafts matches: break the draft at a random position (or never). + cut = torch.randint(0, draft_len + 1, (n_req,), generator=gen, device="cuda") + pos = torch.arange(draft_len, device="cuda").unsqueeze(0) + breaks = pos >= cut.unsqueeze(1) + draft = torch.where(breaks & (pos == cut.unsqueeze(1)), (draft + 1) % VOCAB, draft) + # Past the first mismatch the drafts are random again (a later match must not count). + noise = torch.randint(0, VOCAB, draft.shape, generator=gen, device="cuda").to(draft.dtype) + draft = torch.where(breaks & (pos > cut.unsqueeze(1)), noise, draft) + if spec.get("pad_drafts"): + draft[::2, draft_len // 2 :] = -1 + if spec.get("extremes"): + draft[: n_req // 2] = target.view(n_req, k1)[: n_req // 2, :draft_len].to(draft.dtype) + draft[n_req // 2 :] = ( + target.view(n_req, k1)[n_req // 2 :, :draft_len].to(draft.dtype) + 1 + ) % VOCAB + return target, draft + + +def _next_new_inputs(spec, gen): + n_req, draft_len = spec["n_req"], spec["draft_len"] + width = draft_len + 1 + (4 if spec.get("wide") else 0) + rows = n_req + 3 + accepted = torch.randint( + 0, VOCAB, (rows, width), generator=gen, device="cuda", dtype=torch.int32 + ) + num = torch.randint(1, draft_len + 2, (n_req,), generator=gen, device="cuda", dtype=torch.int32) + batch_idx = torch.arange(n_req, device="cuda", dtype=torch.int32) + if spec.get("shuffle"): + batch_idx = torch.randperm(rows, generator=gen, device="cuda")[:n_req].to(torch.int32) + draft_dtype = spec.get("draft_dtype", torch.int32) + if spec.get("wide"): + # A strided view: rows of draft_len + 5 elements, the first draft_len used. + wide_rows = torch.randint(0, VOCAB, (n_req, draft_len + 5), generator=gen, device="cuda") + next_draft = wide_rows.to(draft_dtype)[:, :draft_len] + else: + next_draft = torch.randint(0, VOCAB, (n_req, draft_len), generator=gen, device="cuda").to( + draft_dtype + ) + return accepted, num, batch_idx, next_draft + + +def _assert_same(out, ref): + for o, r in zip( + out if isinstance(out, (list, tuple)) else [out], + ref if isinstance(ref, (list, tuple)) else [ref], + ): + assert o.dtype == r.dtype, (o.dtype, r.dtype) + assert o.shape == r.shape, (o.shape, r.shape) + assert torch.equal(o, r), (o, r) + + +@pytest.mark.parametrize("cell", SpecStrictAccept.CELLS, ids=lambda c: c.why[:60]) +def test_strict_accept_cells(cell): + gen = torch.Generator(device="cuda").manual_seed(7) + for _ in range(2): + target, draft = _strict_inputs(dict(cell.spec), gen) + _assert_same(spec_strict_accept(target, draft), spec_strict_accept.reference(target, draft)) + # Graph replay on new values written into the captured inputs. + target, draft = _strict_inputs(dict(cell.spec), gen) + spec_strict_accept(target, draft) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = spec_strict_accept(target, draft) + new_target, new_draft = _strict_inputs(dict(cell.spec), gen) + target.copy_(new_target) + draft.copy_(new_draft) + graph.replay() + torch.cuda.synchronize() + _assert_same(out, spec_strict_accept.reference(target, draft)) + + +@pytest.mark.parametrize("cell", SpecNextNewTokens.CELLS, ids=lambda c: c.why[:60]) +def test_next_new_tokens_cells(cell): + gen = torch.Generator(device="cuda").manual_seed(11) + for _ in range(2): + args = _next_new_inputs(dict(cell.spec), gen) + spec_next_new_tokens.is_valid(*args) + _assert_same(spec_next_new_tokens(*args), spec_next_new_tokens.reference(*args)) + args = _next_new_inputs(dict(cell.spec), gen) + spec_next_new_tokens(*args) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = spec_next_new_tokens(*args) + new_args = _next_new_inputs(dict(cell.spec), gen) + for dst, src in zip(args, new_args): + dst.copy_(src) + graph.replay() + torch.cuda.synchronize() + _assert_same(out, spec_next_new_tokens.reference(*args)) + + +def _kv_rewind_inputs(spec, gen): + nc, n = spec["nc"], spec["n"] + rows = nc + n + 2 # rows past the generation block must stay untouched + num = torch.randint(1, 5, (rows,), generator=gen, device="cuda", dtype=torch.int32) + kv = torch.randint(0, 1 << 20, (rows,), generator=gen, device="cuda").to( + spec.get("kv_dtype", torch.int32) + ) + rewind = torch.empty(64, dtype=torch.int32, device="cuda") + return num, kv, rewind + + +@pytest.mark.parametrize("cell", SpecKvRewind.CELLS, ids=lambda c: c.why[:60]) +def test_kv_rewind_cells(cell): + spec = dict(cell.spec) + nc, n = spec["nc"], spec["n"] + gen = torch.Generator(device="cuda").manual_seed(13) + for _ in range(2): + num, kv, rewind = _kv_rewind_inputs(spec, gen) + kv_ref = kv.clone() + out = spec_kv_rewind(num, kv, rewind, nc, n) + _assert_same(out, spec_kv_rewind.reference(num, kv_ref, rewind, nc, n)) + _assert_same(kv, kv_ref) + # Graph replay on new values written into the captured inputs. + num, kv, rewind = _kv_rewind_inputs(spec, gen) + spec_kv_rewind(num, kv.clone(), rewind, nc, n) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = spec_kv_rewind(num, kv, rewind, nc, n) + new_num, new_kv, _ = _kv_rewind_inputs(spec, gen) + num.copy_(new_num) + kv.copy_(new_kv) + kv_ref = kv.clone() + graph.replay() + torch.cuda.synchronize() + _assert_same(out, spec_kv_rewind.reference(num, kv_ref, rewind, nc, n)) + _assert_same(kv, kv_ref) diff --git a/tests/unittest/_torch/modeling_v2/test_m31_decode_scope_source.py b/tests/unittest/_torch/modeling_v2/test_m31_decode_scope_source.py new file mode 100644 index 000000000000..fb5698d43384 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/test_m31_decode_scope_source.py @@ -0,0 +1,96 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CPU checks of the production fused-dispatch predicates without CUDA imports.""" + +import ast +import types +import unittest +from pathlib import Path + +ROOT = ( + Path(__file__).resolve().parents[4] + / "tensorrt_llm" + / "_torch" + / "_experimental" + / "modeling_v2" +) + + +def dispatch(tp, rows, requests, capacity=64, ndim=1, embeddings=False): + path = ROOT / "models" / "minimax_m3" / f"minimax_m31_nvfp4__sm_103__tp{tp}" / "modeling.py" + tree = ast.parse(path.read_text()) + helper = next( + node + for node in tree.body + if isinstance(node, ast.FunctionDef) and node.name == "is_decode_step" + ) + limit = next( + node + for node in tree.body + if isinstance(node, ast.Assign) + and any( + isinstance(target, ast.Name) and target.id == "FUSED_DECODE_MAX_TOKENS" + for target in node.targets + ) + ) + cls = next( + node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == "DecodeTarget" + ) + forward = next( + node for node in cls.body if isinstance(node, ast.FunctionDef) and node.name == "forward" + ) + predicate = next( + node.value + for node in ast.walk(forward) + if isinstance(node, ast.Assign) + and any(isinstance(target, ast.Name) and target.id == "kernels" for target in node.targets) + ) + env = { + "DECODE_MAX_TOKENS": capacity, + "input_ids": types.SimpleNamespace(ndim=ndim, numel=lambda: rows), + "inputs_embeds": object() if embeddings else None, + "attn_metadata": types.SimpleNamespace(num_seqs=requests), + } + exec(compile(ast.Module(body=[limit, helper], type_ignores=[]), str(path), "exec"), env) + return eval(compile(ast.Expression(predicate), str(path), "eval"), env) + + +class TestM31DecodeScope(unittest.TestCase): + def test_tp8_single_request_boundary(self): + for capacity in (48, 64): + for rows, expected in ( + (1, True), + (7, True), + (8, True), + (9, False), + (32, False), + (64, False), + ): + with self.subTest(capacity=capacity, rows=rows): + self.assertEqual(dispatch(8, rows, 1, capacity), expected) + + def test_tp8_multiple_requests_decline_even_when_rows_fit(self): + self.assertFalse(dispatch(8, 8, 2)) # K3/c2 must not enter the c1 fused path. + self.assertFalse(dispatch(8, 8, 8)) # Non-speculative c8 also has eight rows. + + def test_tp4_boundary(self): + for capacity in (48, 64): + for rows, requests, expected in ( + (8, 1, True), + (16, 2, True), + (32, 4, True), + (33, 5, False), + (64, 8, False), + ): + with self.subTest(capacity=capacity, rows=rows): + self.assertEqual(dispatch(4, rows, requests, capacity), expected) + + def test_embeddings_and_nonflat_ids_use_generic_path(self): + for tp in (4, 8): + with self.subTest(tp=tp): + self.assertFalse(dispatch(tp, 8, 1, ndim=2)) + self.assertFalse(dispatch(tp, 8, 1, embeddings=True)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/unittest/_torch/modeling_v2/test_m31_tp4_routing_source.py b/tests/unittest/_torch/modeling_v2/test_m31_tp4_routing_source.py new file mode 100644 index 000000000000..6ac180b441f6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/test_m31_tp4_routing_source.py @@ -0,0 +1,325 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CPU source-contract checks; no CUDA/model import or end-to-end gate implied.""" + +import ast +import enum +import importlib.util +import json +import os +import sys +import types +import unittest +from pathlib import Path +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parents[4] +PACKAGE = "tensorrt_llm._torch._experimental.modeling_v2" +SOURCE = ROOT / "tensorrt_llm/_torch/_experimental/modeling_v2" + + +class Quant(enum.Enum): + FP8 = "FP8" + NVFP4 = "NVFP4" + MXFP8 = "MXFP8" + + +class NoTarget(Exception): + pass + + +def load_source(path, **values): + """Evaluate exact definitions with only top-level dependency imports supplied by fixtures.""" + tree = ast.parse(path.read_text()) + tree.body = [ + n + for n in tree.body + if not isinstance(n, (ast.Import, ast.ImportFrom)) + or (isinstance(n, ast.ImportFrom) and n.module == "__future__") + ] + namespace = dict(TYPE_CHECKING=False, enum=enum, os=os, json=json, Path=Path, **values) + exec(compile(tree, str(path), "exec"), namespace) + return namespace + + +def route(): + ns = load_source( + SOURCE / "models/minimax_m3/routing.py", + QuantAlgo=Quant, + NoTarget=NoTarget, + __package__=PACKAGE + ".models.minimax_m3", + ) + # The HF quant parser is a framework prerequisite; these tests supply its result. + ns["_quant_algo"] = lambda config: Quant.MXFP8 + return ns["route"] + + +def inputs(): + config = types.SimpleNamespace( + text_config=dict( + num_hidden_layers=60, + hidden_size=6144, + num_local_experts=128, + intermediate_size=3072, + num_attention_heads=64, + num_key_value_heads=4, + rope_theta=10_000_000, + sparse_attention_config=dict(use_sparse_attention=True, sparse_attention_freq=[1] * 60), + ), + quantization_config=dict(moe_quant_algo="NVFP4", moe_quant_format="nvfp4-pack-quantized"), + ) + mapping = types.SimpleNamespace( + world_size=4, + tp_size=4, + pp_size=1, + cp_size=1, + moe_tp_size=4, + moe_ep_size=1, + enable_attention_dp=False, + ) + return config, mapping + + +class TestM31Tp4RoutingSource(unittest.TestCase): + def test_declines_outside_scope(self): + cases = [("chip", (10, 0)), ("chip", (10, 7))] + cases += [ + (name, value) + for name, value in ( + ("world_size", 8), + ("tp_size", 8), + ("tp_size", 2), + ("pp_size", 2), + ("cp_size", 2), + ("moe_tp_size", 1), + ("moe_ep_size", 2), + ("enable_attention_dp", True), + ) + ] + cases += [ + (name, None) + for name in ("m3_rope", "m3_layers", "wrong_experts", "wrong_shape", "bf16_kv") + ] + for name, value in cases: + with self.subTest(name=name, value=value): + config, mapping = inputs() + chip, kv = (10, 3), Quant.NVFP4 + if name == "chip": + chip = value + elif hasattr(mapping, name): + setattr(mapping, name, value) + elif name == "m3_rope": + config.text_config["rope_theta"] = 5_000_000 + elif name == "m3_layers": + config.text_config["sparse_attention_config"]["sparse_attention_freq"] = [ + 0 + ] * 3 + [1] * 57 + elif name == "wrong_experts": + config.quantization_config["moe_quant_algo"] = "FP8" + elif name == "wrong_shape": + config.text_config["num_key_value_heads"] = 8 + else: + kv = None + with self.assertRaises(NoTarget): + route()(config, mapping, chip, None, kv) + + def test_tp4_routes_only_to_tp4(self): + config, mapping = inputs() + name = PACKAGE + ".models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.modeling" + module = types.ModuleType(name) + sentinel = type("Target", (), {}) + module.ModelingV2MiniMaxM31Nvfp4Sm103Tp4 = sentinel + with patch.dict(sys.modules, {name: module}): + self.assertIs(route()(config, mapping, (10, 3), None, Quant.NVFP4), sentinel) + + def test_router_arity_and_mode(self): + for arch, arity in [ + ("GptOssForCausalLM", 4), + ("DeepseekV3ForCausalLM", 4), + ("MiniMaxM3SparseForConditionalGeneration", 5), + ]: + with self.subTest(arch=arch): + ns = load_source( + SOURCE / "_router_index.py", logger=types.SimpleNamespace(info=lambda *a: None) + ) + calls = [] + target = type("Target", (), {}) + + def call(*args): + calls.append(args) + return target + + ns["routing_module"] = lambda name: types.SimpleNamespace(route=call) + ns["_device_compute_capability"] = lambda: (10, 3) + ns["_kv_cache_quant_algo"] = lambda *args: Quant.NVFP4 + config = types.SimpleNamespace( + pretrained_config=types.SimpleNamespace(architectures=[arch]), + mapping=object(), + spec_config=None, + ) + self.assertIs( + ns["modeling_v2_resolve"](config, types.SimpleNamespace(modeling_v2="require")), + target, + ) + self.assertEqual(len(calls[0]), arity) + if arity == 5: + self.assertIs(ns["resolved_mode"](target), ns["ModelingV2Mode"].REQUIRE) + self.assertEqual( + set(ns["MODELING_V2_ROUTERS"]), + { + "GptOssForCausalLM", + "DeepseekV3ForCausalLM", + "MiniMaxM3SparseForConditionalGeneration", + }, + ) + + def test_parent_import_and_late_native_registration(self): + """Import actual parent with no native ops, then load ops and repeat registration.""" + calls = [] + native = types.SimpleNamespace() + installed = {} + torch = types.ModuleType("torch") + torch.ops = types.SimpleNamespace(trtllm=native) + torch._C = types.SimpleNamespace( + _dispatch_has_kernel_for_dispatch_key=lambda name, key: installed.get(name) == "Meta" + ) + + def register_fake(name): + def decorate(fn): + self.assertNotIn(name, installed) + installed[name] = fn + calls.append(name) + return fn + + return decorate + + torch.library = types.SimpleNamespace(register_fake=register_fake) + registry = types.ModuleType("torch._library.simple_registry") + registry.singleton = types.SimpleNamespace( + find=lambda name: types.SimpleNamespace( + fake_impl=types.SimpleNamespace(kernel=installed.get(name)) + ) + ) + family_name = PACKAGE + ".models.minimax_m3" + catalog_name = PACKAGE + ".catalog.fakes.m31_fake_ops" + + def load_file(name, path, package=False): + spec = importlib.util.spec_from_file_location( + name, path, submodule_search_locations=[str(path.parent)] if package else None + ) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + with patch.dict(sys.modules, {"torch": torch, "torch._library.simple_registry": registry}): + for fullname in (family_name, catalog_name): + parts = fullname.split(".") + for length in range(1, len(parts)): + name = ".".join(parts[:length]) + package = types.ModuleType(name) + package.__path__ = [str(ROOT.joinpath(*parts[:length]))] + sys.modules[name] = package + parent = load_file(family_name, SOURCE / "models/minimax_m3/__init__.py", package=True) + self.assertEqual(calls, []) + self.assertFalse(hasattr(parent, "fake_ops")) + # A route rejected after the parent import can still choose the built-in model. + config, mapping = inputs() + with self.assertRaises(NoTarget): + route()(config, mapping, (10, 7), None, Quant.NVFP4) + load_file(catalog_name, SOURCE / "catalog/fakes/m31_fake_ops.py") + fake_module = load_file( + family_name + ".fake_ops", SOURCE / "models/minimax_m3/fake_ops.py" + ) + self.assertEqual(len(fake_module.FAKES), 7) + names = list(fake_module.FAKES) + for name in names: + setattr(native, name, object()) + installed["trtllm::" + names[0]] = "Meta" + previous_fake = object() + installed["trtllm::" + names[1]] = previous_fake + fake_module.register() + self.assertEqual(len(calls), 5) + fake_module.register() + self.assertEqual(len(calls), 5) + self.assertIs(installed["trtllm::" + names[1]], previous_fake) + self.assertEqual(installed["trtllm::" + names[0]], "Meta") + + def test_dormant_optins_removed_and_baseline_flags_retained(self): + root = SOURCE / "models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4" + rejected = { + "LC4_ONEDIV", + "DENSE_SWIGLU_CPP", + "R6", + "LDS1", + "S5", + "S2", + "S9B", + "FUSED_CAPTURE", + } + for path in root.rglob("*.py"): + for node in ast.walk(ast.parse(path.read_text())): + if isinstance(node, ast.Call) and isinstance(node.func, ast.Name): + if node.func.id in {"m31_enabled", "m31_disabled"} and node.args: + self.assertNotIn(getattr(node.args[0], "value", None), rejected, str(path)) + for diagnostic in ("UB_NOSELECT", "UB_DUPSELECT", "_ub_noselect_table"): + self.assertNotIn(diagnostic, (root / "modeling.py").read_text()) + s4 = (root / "dspark_s4.py").read_text() + for diagnostic in ("DSPARK_GATE_DUMP", "_s4_dump", "_s4_gate_dump", "_s4_logits_gate_dump"): + self.assertNotIn(diagnostic, s4) + model = ast.parse((root / "modeling.py").read_text()) + assigns = { + n.targets[0].id: ast.unparse(n.value) + for n in model.body + if isinstance(n, ast.Assign) and isinstance(n.targets[0], ast.Name) + } + self.assertTrue( + any( + isinstance(n, ast.AnnAssign) + and isinstance(n.target, ast.Name) + and n.target.id == "SWQ_ROWS" + for n in model.body + ) + ) + self.assertIn("AR_AUTOTUNE_ENABLED", assigns) + lc = ast.parse((root / "ops/attention/m3_sparse_decode_lc.py").read_text()) + flags = { + n.targets[0].id: ast.unparse(n.value) + for n in lc.body + if isinstance(n, ast.Assign) and isinstance(n.targets[0], ast.Name) + } + self.assertEqual(flags["LC4_BTSMEM"], "not m31_disabled('LC4_BTSMEM')") + self.assertEqual(flags["LC4_SYNC3"], "not m31_disabled('LC4_SYNC3')") + native_calls = [ + n + for n in ast.walk(lc) + if isinstance(n, ast.Call) and isinstance(n.func, ast.Name) and n.func.id == "lc4" + ] + self.assertEqual(len(native_calls), 1) + self.assertIs(native_calls[0].args[-1].value, False) + draft = (root / "dspark.py").read_text() + for marker in ("MXDRAFT = not m31_disabled", 'm31_disabled("S10")', 'm31_disabled("S10C")'): + self.assertIn(marker, draft) + self.assertFalse((root / "ops/gemm/m3_dense_swiglu.py").exists()) + + def test_pruned_activation_keeps_default_parameter_order(self): + path = SOURCE / "models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/modeling.py" + tree = ast.parse(path.read_text()) + fn = next( + n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == "_swiglu_act" + ) + calls = [] + + def fallback(*args): + calls.append(args) + return "output" + + namespace = {"m3_silu_and_mul": fallback, "torch": types.SimpleNamespace(Tensor=object)} + exec(compile(ast.Module(body=[fn], type_ignores=[]), str(path), "exec"), namespace) + core = types.SimpleNamespace(swiglu_limit=7, swiglu_alpha=1.7, swiglu_beta=1.0) + self.assertEqual(namespace["_swiglu_act"](core, "gate_up"), "output") + self.assertEqual(calls, [("gate_up", 7, 1.7, 1.0)]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/unittest/_torch/modeling_v2/test_m31_tp8_routing_source.py b/tests/unittest/_torch/modeling_v2/test_m31_tp8_routing_source.py new file mode 100644 index 000000000000..57ca48949126 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/test_m31_tp8_routing_source.py @@ -0,0 +1,253 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CPU source-contract checks; no CUDA/model import or end-to-end gate implied.""" + +import ast +import enum +import importlib.util +import json +import os +import sys +import types +import unittest +from pathlib import Path +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parents[4] +PACKAGE = "tensorrt_llm._torch._experimental.modeling_v2" +SOURCE = ROOT / "tensorrt_llm/_torch/_experimental/modeling_v2" + + +class Quant(enum.Enum): + FP8 = "FP8" + NVFP4 = "NVFP4" + MXFP8 = "MXFP8" + + +class NoTarget(Exception): + pass + + +def load_source(path, **values): + """Evaluate exact definitions with only top-level dependency imports supplied by fixtures.""" + tree = ast.parse(path.read_text()) + tree.body = [ + n + for n in tree.body + if not isinstance(n, (ast.Import, ast.ImportFrom)) + or (isinstance(n, ast.ImportFrom) and n.module == "__future__") + ] + namespace = dict(TYPE_CHECKING=False, enum=enum, os=os, json=json, Path=Path, **values) + exec(compile(tree, str(path), "exec"), namespace) + return namespace + + +def route(): + ns = load_source( + SOURCE / "models/minimax_m3/routing.py", + QuantAlgo=Quant, + NoTarget=NoTarget, + __package__=PACKAGE + ".models.minimax_m3", + ) + # The HF quant parser is a framework prerequisite; these tests supply its result. + ns["_quant_algo"] = lambda config: Quant.MXFP8 + return ns["route"] + + +def inputs(): + config = types.SimpleNamespace( + text_config=dict( + num_hidden_layers=60, + hidden_size=6144, + num_local_experts=128, + intermediate_size=3072, + num_attention_heads=64, + num_key_value_heads=4, + rope_theta=10_000_000, + sparse_attention_config=dict(use_sparse_attention=True, sparse_attention_freq=[1] * 60), + ), + quantization_config=dict(moe_quant_algo="NVFP4", moe_quant_format="nvfp4-pack-quantized"), + ) + mapping = types.SimpleNamespace( + world_size=8, + tp_size=8, + pp_size=1, + cp_size=1, + moe_tp_size=8, + moe_ep_size=1, + enable_attention_dp=False, + ) + return config, mapping + + +class TestM31Tp8RoutingSource(unittest.TestCase): + def test_declines_outside_scope(self): + cases = [("chip", (10, 0)), ("chip", (10, 7))] + cases += [ + (name, value) + for name, value in ( + ("world_size", 4), + ("tp_size", 4), + ("tp_size", 2), + ("pp_size", 2), + ("cp_size", 2), + ("moe_tp_size", 1), + ("moe_ep_size", 2), + ("enable_attention_dp", True), + ) + ] + cases += [ + (name, None) + for name in ("m3_rope", "m3_layers", "wrong_experts", "wrong_shape", "bf16_kv") + ] + for name, value in cases: + with self.subTest(name=name, value=value): + config, mapping = inputs() + chip, kv = (10, 3), Quant.NVFP4 + if name == "chip": + chip = value + elif hasattr(mapping, name): + setattr(mapping, name, value) + elif name == "m3_rope": + config.text_config["rope_theta"] = 5_000_000 + elif name == "m3_layers": + config.text_config["sparse_attention_config"]["sparse_attention_freq"] = [ + 0 + ] * 3 + [1] * 57 + elif name == "wrong_experts": + config.quantization_config["moe_quant_algo"] = "FP8" + elif name == "wrong_shape": + config.text_config["num_key_value_heads"] = 8 + else: + kv = None + with self.assertRaises(NoTarget): + route()(config, mapping, chip, None, kv) + + def test_tp8_routes_only_to_tp8(self): + config, mapping = inputs() + name = PACKAGE + ".models.minimax_m3.minimax_m31_nvfp4__sm_103__tp8.modeling" + module = types.ModuleType(name) + sentinel = type("Target", (), {}) + module.ModelingV2MiniMaxM31Nvfp4Sm103Tp8 = sentinel + fake_name = PACKAGE + ".models.minimax_m3.fake_ops" + fake_module = types.ModuleType(fake_name) + fake_module.register = lambda: None + with patch.dict(sys.modules, {name: module, fake_name: fake_module}): + self.assertIs(route()(config, mapping, (10, 3), None, Quant.NVFP4), sentinel) + + def test_router_arity_and_mode(self): + for arch, arity in [ + ("GptOssForCausalLM", 4), + ("DeepseekV3ForCausalLM", 4), + ("MiniMaxM3SparseForConditionalGeneration", 5), + ]: + with self.subTest(arch=arch): + ns = load_source( + SOURCE / "_router_index.py", logger=types.SimpleNamespace(info=lambda *a: None) + ) + calls = [] + target = type("Target", (), {}) + + def call(*args): + calls.append(args) + return target + + ns["routing_module"] = lambda name: types.SimpleNamespace(route=call) + ns["_device_compute_capability"] = lambda: (10, 3) + ns["_kv_cache_quant_algo"] = lambda *args: Quant.NVFP4 + config = types.SimpleNamespace( + pretrained_config=types.SimpleNamespace(architectures=[arch]), + mapping=object(), + spec_config=None, + ) + self.assertIs( + ns["modeling_v2_resolve"](config, types.SimpleNamespace(modeling_v2="require")), + target, + ) + self.assertEqual(len(calls[0]), arity) + if arity == 5: + self.assertIs(ns["resolved_mode"](target), ns["ModelingV2Mode"].REQUIRE) + self.assertEqual( + set(ns["MODELING_V2_ROUTERS"]), + { + "GptOssForCausalLM", + "DeepseekV3ForCausalLM", + "MiniMaxM3SparseForConditionalGeneration", + }, + ) + + def test_parent_import_and_late_native_registration(self): + """Import actual parent with no native ops, then load ops and repeat registration.""" + calls = [] + native = types.SimpleNamespace() + installed = {} + torch = types.ModuleType("torch") + torch.ops = types.SimpleNamespace(trtllm=native) + torch._C = types.SimpleNamespace( + _dispatch_has_kernel_for_dispatch_key=lambda name, key: installed.get(name) == "Meta" + ) + + def register_fake(name): + def decorate(fn): + self.assertNotIn(name, installed) + installed[name] = fn + calls.append(name) + return fn + + return decorate + + torch.library = types.SimpleNamespace(register_fake=register_fake) + registry = types.ModuleType("torch._library.simple_registry") + registry.singleton = types.SimpleNamespace( + find=lambda name: types.SimpleNamespace( + fake_impl=types.SimpleNamespace(kernel=installed.get(name)) + ) + ) + family_name = PACKAGE + ".models.minimax_m3" + catalog_name = PACKAGE + ".catalog.fakes.m31_fake_ops" + + def load_file(name, path, package=False): + spec = importlib.util.spec_from_file_location( + name, path, submodule_search_locations=[str(path.parent)] if package else None + ) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + with patch.dict(sys.modules, {"torch": torch, "torch._library.simple_registry": registry}): + for fullname in (family_name, catalog_name): + parts = fullname.split(".") + for length in range(1, len(parts)): + name = ".".join(parts[:length]) + package = types.ModuleType(name) + package.__path__ = [str(ROOT.joinpath(*parts[:length]))] + sys.modules[name] = package + load_file(catalog_name, SOURCE / "catalog/fakes/m31_fake_ops.py") + parent = load_file(family_name, SOURCE / "models/minimax_m3/__init__.py", package=True) + self.assertEqual(calls, []) + self.assertFalse(hasattr(parent, "fake_ops")) + fake_module = load_file( + family_name + ".fake_ops", SOURCE / "models/minimax_m3/fake_ops.py" + ) + self.assertEqual(len(fake_module.FAKES), 7) + # A route rejected after the parent import can still choose the built-in model. + config, mapping = inputs() + with self.assertRaises(NoTarget): + route()(config, mapping, (10, 7), None, Quant.NVFP4) + names = list(fake_module.FAKES) + for name in names: + setattr(native, name, object()) + installed["trtllm::" + names[0]] = "Meta" + previous_fake = object() + installed["trtllm::" + names[1]] = previous_fake + fake_module.register() + self.assertEqual(len(calls), 5) + fake_module.register() + self.assertEqual(len(calls), 5) + self.assertIs(installed["trtllm::" + names[1]], previous_fake) + self.assertEqual(installed["trtllm::" + names[0]], "Meta") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/unittest/_torch/speculative/test_rng_upload_pack.py b/tests/unittest/_torch/speculative/test_rng_upload_pack.py new file mode 100644 index 000000000000..4964734f7434 --- /dev/null +++ b/tests/unittest/_torch/speculative/test_rng_upload_pack.py @@ -0,0 +1,150 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Philox seed/offset uploads of a sampled one-model step, staged into the step-input pack.""" + +import types +from typing import Optional + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor import step_inputs +from tensorrt_llm._torch.pyexecutor.step_inputs import StepInputPack +from tensorrt_llm._torch.speculative.interface import SpecMetadata + +pytestmark = [ + pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required"), + pytest.mark.skipif( + not (torch.cuda.is_available() and hasattr(torch.ops.trtllm, "step_inputs_scatter")), + reason="torch.ops.trtllm.step_inputs_scatter is not registered", + ), +] + +MAX_DRAFT_LEN = 3 +ROWS_PER_REQUEST = MAX_DRAFT_LEN + 1 + + +def _meta(pack_rng_uploads: bool) -> SpecMetadata: + meta = SpecMetadata( + max_num_requests=4, max_draft_len=MAX_DRAFT_LEN, max_total_draft_tokens=MAX_DRAFT_LEN + ) + meta.pack_rng_uploads = pack_rng_uploads + return meta + + +def _request(slot: int, seed: Optional[int]) -> types.SimpleNamespace: + return types.SimpleNamespace( + py_seq_slot=slot, + py_decoding_iter=0, + py_request_id=slot, + sampling_config=types.SimpleNamespace(seed=seed), + ) + + +def _populate(meta: SpecMetadata) -> None: + requests = [_request(2, seed=1234), _request(0, seed=None)] + normalized = [(0.7, 0, 0.9, 0.0, ROWS_PER_REQUEST) for _ in requests] + meta._populate_request_rng_state(requests, normalized) + + +def _rng_state(meta: SpecMetadata) -> dict[str, torch.Tensor]: + torch.cuda.synchronize() + return { + name: getattr(meta, name).clone() + for name in ("request_seeds", "request_offsets", "seeds", "offsets") + } + + +def test_packed_uploads_land_at_the_flush_with_the_plain_values(): + plain = _meta(pack_rng_uploads=False) + _populate(plain) + expected = _rng_state(plain) + + packed = _meta(pack_rng_uploads=True) + pack = StepInputPack(torch.device("cuda", torch.cuda.current_device())) + with step_inputs.active(pack): + step_inputs.begin() + _populate(packed) + before_flush = _rng_state(packed) + step_inputs.flush() + after_flush = _rng_state(packed) + + for name, tensor in expected.items(): + assert torch.equal(after_flush[name], tensor), name + # Staged, not copied: the buffers still hold their zero init until the flush. + assert not torch.equal(before_flush["seeds"], expected["seeds"]) + + +def test_without_an_open_pack_the_uploads_are_plain_copies(): + plain = _meta(pack_rng_uploads=False) + _populate(plain) + expected = _rng_state(plain) + + packed = _meta(pack_rng_uploads=True) + _populate(packed) + + for name, tensor in _rng_state(packed).items(): + assert torch.equal(tensor, expected[name]), name + + +def test_packing_is_off_by_default(): + assert not SpecMetadata( + max_num_requests=1, max_draft_len=MAX_DRAFT_LEN, max_total_draft_tokens=MAX_DRAFT_LEN + ).pack_rng_uploads + + +def test_under_capture_the_uploads_are_plain_copies(monkeypatch): + """step_inputs.copy_ never defers a copy while the stream captures (the captured copy reruns on every replay), + so a packed Philox upload issued during capture is a plain copy that lands without the flush.""" + plain = _meta(pack_rng_uploads=False) + _populate(plain) + expected = _rng_state(plain) + + packed = _meta(pack_rng_uploads=True) + pack = StepInputPack(torch.device("cuda", torch.cuda.current_device())) + with step_inputs.active(pack): + step_inputs.begin() + monkeypatch.setattr(step_inputs.torch.cuda, "is_current_stream_capturing", lambda: True) + _populate(packed) + monkeypatch.undo() + assert pack.h2d == [] and pack.staged == [] + landed = _rng_state(packed) + step_inputs.flush() + + for name, tensor in expected.items(): + assert torch.equal(landed[name], tensor), name + + +@pytest.mark.parametrize("pack_rng_uploads", [False, True]) +def test_an_all_greedy_batch_with_the_greedy_skip_uploads_nothing(pack_rng_uploads): + meta = _meta(pack_rng_uploads=pack_rng_uploads) + meta.skip_greedy_rng_upload = True + meta.is_all_greedy_sample = True + pack = StepInputPack(torch.device("cuda", torch.cuda.current_device())) + with step_inputs.active(pack): + step_inputs.begin() + _populate(meta) + assert pack.h2d == [] + step_inputs.flush() + + for name, tensor in _rng_state(meta).items(): + assert not bool(tensor.any()), name # still the zero init + + +def test_a_sampled_batch_with_the_greedy_skip_still_packs_its_uploads(): + plain = _meta(pack_rng_uploads=False) + _populate(plain) + expected = _rng_state(plain) + + packed = _meta(pack_rng_uploads=True) + packed.skip_greedy_rng_upload = True + packed.is_all_greedy_sample = False + pack = StepInputPack(torch.device("cuda", torch.cuda.current_device())) + with step_inputs.active(pack): + step_inputs.begin() + _populate(packed) + assert len(pack.h2d) == 4 # request_seeds, request_offsets, seeds, offsets + step_inputs.flush() + + for name, tensor in _rng_state(packed).items(): + assert torch.equal(tensor, expected[name]), name diff --git a/tests/unittest/_torch/speculative/test_spec_sampler_step_fold.py b/tests/unittest/_torch/speculative/test_spec_sampler_step_fold.py new file mode 100644 index 000000000000..a50e884a29d2 --- /dev/null +++ b/tests/unittest/_torch/speculative/test_spec_sampler_step_fold.py @@ -0,0 +1,648 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SpecSampler's per-slot store with and without a model's SpecStepFold.""" + +from typing import Optional + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor.llm_request import LlmRequest +from tensorrt_llm._torch.pyexecutor.runtime_hooks import SpecStepFold +from tensorrt_llm._torch.pyexecutor.sampler import TorchSampler +from tensorrt_llm._torch.pyexecutor.scheduler import ScheduledRequests +from tensorrt_llm._torch.speculative.spec_sampler_base import SpecSampler +from tensorrt_llm.bindings import SamplingConfig + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required") + +MAX_DRAFT_LEN = 3 +MAX_NUM_SEQUENCES = 6 +SEQ_SLOTS = [4, 1] + + +class _RecordingStepFold: + """A SpecStepFold whose store decisions are fixed per test; it records what the sampler asks.""" + + def __init__(self, rows_stored: bool = False, scatter_taken: bool = False) -> None: + self._rows_stored = rows_stored + self._scatter_taken = scatter_taken + self.store: Optional[object] = None + self.rows_stored_calls: list[tuple[int, list[int]]] = [] + self.scatter_calls: list[dict[str, torch.Tensor]] = [] + + def bind_engine_buffers( + self, + input_ids: torch.Tensor, + draft_tokens: torch.Tensor, + position_id_offsets: torch.Tensor, + kv_lens_offsets: torch.Tensor, + max_batch_size: int, + ) -> None: + raise AssertionError("the sampler never binds engine buffers") + + def bind_sampler_store(self, store: object) -> None: + self.store = store + + def begin_step(self, seq_slots: Optional[list[int]], num_rows: int) -> bool: + raise AssertionError("the sampler never begins a step") + + def gather(self, *args: object) -> bool: + raise AssertionError("the sampler never gathers") + + def sampler_rows_stored(self, outputs: dict, num_skip: int, seq_slots: list[int]) -> bool: + self.rows_stored_calls.append((num_skip, list(seq_slots))) + return self._rows_stored + + def scatter( + self, + seq_slots: torch.Tensor, + new_tokens: torch.Tensor, + next_new_tokens: torch.Tensor, + new_tokens_lens: torch.Tensor, + next_draft_tokens: torch.Tensor, + ) -> bool: + self.scatter_calls.append( + dict( + seq_slots=seq_slots.clone(), + new_tokens=new_tokens.clone(), + next_new_tokens=next_new_tokens.clone(), + new_tokens_lens=new_tokens_lens.clone(), + next_draft_tokens=next_draft_tokens.clone(), + ) + ) + return self._scatter_taken + + +def _fold(**kwargs: bool) -> SpecStepFold: + return _RecordingStepFold(**kwargs) + + +def _sampler() -> SpecSampler: + sampler = SpecSampler( + TorchSampler.Args( + max_seq_len=64, + max_draft_len=MAX_DRAFT_LEN, + max_num_sequences=MAX_NUM_SEQUENCES, + max_beam_width=1, + max_total_draft_tokens=MAX_DRAFT_LEN, + disable_overlap_scheduler=True, + ) + ) + # The store is allocated uninitialized; give it a recognizable baseline. + for index, tensor in enumerate( + ( + sampler.store.new_tokens, + sampler.store.next_new_tokens, + sampler.store.next_draft_tokens, + sampler.store.new_tokens_lens, + ) + ): + tensor.fill_(-100 - index) + return sampler + + +def _store_snapshot(sampler: SpecSampler) -> dict[str, torch.Tensor]: + store = sampler.store + return dict( + new_tokens=store.new_tokens.clone(), + next_new_tokens=store.next_new_tokens.clone(), + next_draft_tokens=store.next_draft_tokens.clone(), + new_tokens_lens=store.new_tokens_lens.clone(), + ) + + +def _assert_store_equals(sampler: SpecSampler, expected: dict[str, torch.Tensor]) -> None: + torch.cuda.synchronize() + for name, tensor in _store_snapshot(sampler).items(): + assert torch.equal(tensor, expected[name]), name + + +def _step(draft_width: int): + requests = [] + for request_id, seq_slot in enumerate(SEQ_SLOTS): + request = LlmRequest( + request_id=request_id, + max_new_tokens=8, + input_tokens=[1, 2, 3], + sampling_config=SamplingConfig(), + seq_slot=seq_slot, + is_streaming=False, + ) + request.py_draft_tokens = [7] * MAX_DRAFT_LEN + requests.append(request) + scheduled = ScheduledRequests() + scheduled.generation_requests = requests + rows = len(requests) + 1 # plus a CUDA-graph padding row the sampler must not store + outputs = dict( + new_tokens=torch.arange(rows * (draft_width + 1), dtype=torch.int32, device="cuda").view( + rows, draft_width + 1 + ), + new_tokens_lens=torch.arange(1, rows + 1, dtype=torch.int32, device="cuda"), + next_draft_tokens=( + 100 + torch.arange(rows * draft_width, dtype=torch.int32, device="cuda") + ).view(rows, draft_width), + next_new_tokens=( + 200 + torch.arange(rows * (draft_width + 1), dtype=torch.int32, device="cuda") + ).view(rows, draft_width + 1), + ) + return scheduled, outputs + + +def _reference_store(before: dict[str, torch.Tensor], outputs: dict) -> dict[str, torch.Tensor]: + """What the sampler has always stored: each live row's outputs at its slot, zero-padded.""" + expected = {name: tensor.clone() for name, tensor in before.items()} + count = len(SEQ_SLOTS) + slots = torch.tensor(SEQ_SLOTS, dtype=torch.long, device="cuda") + + def padded(tensor: torch.Tensor, width: int) -> torch.Tensor: + tensor = tensor[:count, :width] + return torch.nn.functional.pad(tensor, (0, width - tensor.shape[1])) + + accepted_width = expected["new_tokens"].shape[0] + wire_width = expected["next_new_tokens"].shape[0] + draft_width = expected["next_draft_tokens"].shape[1] + expected["new_tokens"].squeeze(-1).T.index_copy_( + 0, slots, padded(outputs["new_tokens"], accepted_width) + ) + expected["next_new_tokens"].squeeze(-1).T.index_copy_( + 0, slots, padded(outputs["next_new_tokens"], wire_width) + ) + expected["new_tokens_lens"].index_copy_(0, slots, outputs["new_tokens_lens"][:count]) + expected["next_draft_tokens"].index_copy_( + 0, slots, padded(outputs["next_draft_tokens"], draft_width) + ) + return expected + + +@pytest.mark.parametrize("draft_width", [MAX_DRAFT_LEN, MAX_DRAFT_LEN - 1]) +def test_without_a_fold_the_store_is_written_as_before(draft_width): + sampler = _sampler() + scheduled, outputs = _step(draft_width) + before = _store_snapshot(sampler) + + state = sampler.sample_async(scheduled, outputs, []) + + assert sampler.spec_step_fold is None + expected = _reference_store(before, outputs) + _assert_store_equals(sampler, expected) + state.sampler_event.synchronize() + assert torch.equal(state.host.new_tokens, expected["new_tokens"].cpu()) + assert torch.equal(state.host.next_draft_tokens, expected["next_draft_tokens"].cpu()) + + +def test_binding_a_fold_hands_it_the_sampler_store(): + sampler = _sampler() + fold = _fold() + + sampler.bind_spec_step_fold(fold) + + assert sampler.spec_step_fold is fold + assert fold.store is sampler.store + + +def test_rows_the_forward_already_stored_are_left_alone(): + sampler = _sampler() + fold = _fold(rows_stored=True) + sampler.bind_spec_step_fold(fold) + scheduled, outputs = _step(MAX_DRAFT_LEN) + before = _store_snapshot(sampler) + + state = sampler.sample_async(scheduled, outputs, []) + + assert fold.rows_stored_calls == [(0, SEQ_SLOTS)] + assert fold.scatter_calls == [] + _assert_store_equals(sampler, before) + # The host copies still read the (fold-owned) store. + state.sampler_event.synchronize() + assert torch.equal(state.host.new_tokens, before["new_tokens"].cpu()) + + +def test_a_fold_scatter_replaces_the_index_copies(): + sampler = _sampler() + fold = _fold(scatter_taken=True) + sampler.bind_spec_step_fold(fold) + scheduled, outputs = _step(MAX_DRAFT_LEN) + before = _store_snapshot(sampler) + + sampler.sample_async(scheduled, outputs, []) + + _assert_store_equals(sampler, before) + assert len(fold.scatter_calls) == 1 + call = fold.scatter_calls[0] + count = len(SEQ_SLOTS) + assert call["seq_slots"].tolist() == SEQ_SLOTS + assert call["seq_slots"].dtype == torch.long and call["seq_slots"].is_cuda + assert torch.equal(call["new_tokens"], outputs["new_tokens"][:count]) + assert torch.equal(call["next_new_tokens"], outputs["next_new_tokens"][:count]) + assert torch.equal(call["new_tokens_lens"], outputs["new_tokens_lens"][:count]) + assert torch.equal(call["next_draft_tokens"], outputs["next_draft_tokens"][:count]) + + +@pytest.mark.parametrize("draft_width", [MAX_DRAFT_LEN, MAX_DRAFT_LEN - 1]) +def test_a_fold_that_declines_leaves_the_sampler_store_path(draft_width): + sampler = _sampler() + fold = _fold(rows_stored=False, scatter_taken=False) + sampler.bind_spec_step_fold(fold) + scheduled, outputs = _step(draft_width) + before = _store_snapshot(sampler) + + sampler.sample_async(scheduled, outputs, []) + + assert fold.rows_stored_calls == [(0, SEQ_SLOTS)] + assert len(fold.scatter_calls) == 1 + _assert_store_equals(sampler, _reference_store(before, outputs)) + + +def _host_outputs(state) -> dict[str, torch.Tensor]: + state.sampler_event.synchronize() + return dict( + new_tokens=state.host.new_tokens.clone(), + new_tokens_lens=state.host.new_tokens_lens.clone(), + next_draft_tokens=state.host.next_draft_tokens.clone(), + ) + + +def test_store_copies_stay_on_the_main_stream_by_default(): + sampler = _sampler() + scheduled, outputs = _step(MAX_DRAFT_LEN) + + state = sampler.sample_async(scheduled, outputs, []) + + assert not sampler.d2h_side_stream_copies + assert state.sampler_event.side_stream_event is None + + +@pytest.mark.parametrize("draft_width", [MAX_DRAFT_LEN, MAX_DRAFT_LEN - 1]) +def test_side_stream_store_copies_match_the_main_stream_copies(draft_width): + scheduled, outputs = _step(draft_width) + on_main = _host_outputs(_sampler().sample_async(scheduled, outputs, [])) + sampler = _sampler() + sampler.d2h_side_stream_copies = True + + state = sampler.sample_async(scheduled, outputs, []) + + assert state.sampler_event.side_stream_event is not None + on_side = _host_outputs(state) + for name, tensor in on_main.items(): + assert torch.equal(on_side[name], tensor), name + + +def test_a_fold_that_stored_the_rows_keeps_the_copies_on_the_main_stream(): + sampler = _sampler() + sampler.bind_spec_step_fold(_fold(rows_stored=True)) + sampler.d2h_side_stream_copies = True + scheduled, outputs = _step(MAX_DRAFT_LEN) + + state = sampler.sample_async(scheduled, outputs, []) + + assert state.sampler_event.side_stream_event is None + + +def test_a_fold_that_awaits_the_store_copies_moves_them_to_the_side_stream(): + """A fold with store_copies_done: the copies go on the side stream although its forward stored the rows, and + the event is recorded after them.""" + scheduled, outputs = _step(MAX_DRAFT_LEN) + fold = _fold(rows_stored=True) + fold.store_copies_done = torch.cuda.Event(external=True) + fold.store_copies_done.record() + sampler = _sampler() + sampler.bind_spec_step_fold(fold) + sampler.d2h_side_stream_copies = True + expected = _store_snapshot(sampler) + + state = sampler.sample_async(scheduled, outputs, []) + + assert state.sampler_event.side_stream_event is not None + state.sampler_event.synchronize() + fold.store_copies_done.synchronize() + host = _host_outputs(state) + for name in ("new_tokens", "new_tokens_lens", "next_draft_tokens"): + assert torch.equal(host[name], expected[name].cpu()), name + + +def test_a_fold_that_awaits_the_store_copies_keeps_them_on_the_main_stream_when_side_copies_are_off(): + scheduled, outputs = _step(MAX_DRAFT_LEN) + fold = _fold(rows_stored=True) + fold.store_copies_done = torch.cuda.Event(external=True) + sampler = _sampler() + sampler.bind_spec_step_fold(fold) + + state = sampler.sample_async(scheduled, outputs, []) + + assert state.sampler_event.side_stream_event is None + + +# HO-04 on GB: the folds the M3.1 sm_103 targets bind never store the rows in the forward (sampler_rows_stored is +# always False) and carry no store_copies_done. + + +class _ScatteringStepFold(_RecordingStepFold): + """The shape of the targets' M3SpecStepFold (Eagle3): the forward stores nothing, and ``scatter`` writes the + sampler store itself (as m3_spec_tail.slot_scatter does) and takes the step.""" + + def __init__(self) -> None: + super().__init__(rows_stored=False, scatter_taken=True) + + def scatter(self, seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens): + taken = super().scatter( + seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens + ) + store = self.store + count = seq_slots.shape[0] + + def padded(tensor: torch.Tensor, width: int) -> torch.Tensor: + tensor = tensor[:count, :width] + return torch.nn.functional.pad(tensor, (0, width - tensor.shape[1])) + + store.new_tokens.squeeze(-1).T.index_copy_( + 0, seq_slots, padded(new_tokens, store.new_tokens.shape[0]) + ) + store.next_new_tokens.squeeze(-1).T.index_copy_( + 0, seq_slots, padded(next_new_tokens, store.next_new_tokens.shape[0]) + ) + store.new_tokens_lens.index_copy_(0, seq_slots, new_tokens_lens[:count]) + store.next_draft_tokens.index_copy_( + 0, seq_slots, padded(next_draft_tokens, store.next_draft_tokens.shape[1]) + ) + return taken + + +class _AwaitedEvent: + """Stands in for the side-stream event and records that SamplerEvent.synchronize waited on it.""" + + def __init__(self, event: torch.cuda.Event) -> None: + self._event = event + self.waits = 0 + + def synchronize(self) -> None: + self.waits += 1 + self._event.synchronize() + + +@pytest.mark.parametrize("draft_width", [MAX_DRAFT_LEN, MAX_DRAFT_LEN - 1]) +def test_side_stream_copies_engage_under_the_gb_eagle3_fold(draft_width): + scheduled, outputs = _step(draft_width) + stock = _sampler() + stock.bind_spec_step_fold(_ScatteringStepFold()) + on_main = _host_outputs(stock.sample_async(scheduled, outputs, [])) + sampler = _sampler() + fold = _ScatteringStepFold() + sampler.bind_spec_step_fold(fold) + sampler.d2h_side_stream_copies = True + expected = _reference_store(_store_snapshot(sampler), outputs) + + state = sampler.sample_async(scheduled, outputs, []) + + assert not hasattr(fold, "store_copies_done") + assert fold.rows_stored_calls == [(0, SEQ_SLOTS)] and len(fold.scatter_calls) == 1 + side_event = state.sampler_event.side_stream_event + assert side_event is not None + awaited = _AwaitedEvent(side_event) + state.sampler_event.side_stream_event = awaited + host = _host_outputs(state) + assert awaited.waits == 1 + _assert_store_equals(sampler, expected) + for name, tensor in on_main.items(): + assert torch.equal(host[name], tensor), name + assert torch.equal(host[name], expected[name].cpu()), name + + +@pytest.mark.parametrize("draft_width", [MAX_DRAFT_LEN, MAX_DRAFT_LEN - 1]) +def test_side_stream_copies_engage_under_the_gb_dspark_fold(draft_width): + """The targets' M3DSparkStepFold (standalone DSpark): scatter declines every step, so the store is written only by + the sampler's own _store_outputs, and the copies go on the side stream.""" + scheduled, outputs = _step(draft_width) + sampler = _sampler() + fold = _fold(rows_stored=False, scatter_taken=False) + sampler.bind_spec_step_fold(fold) + sampler.d2h_side_stream_copies = True + before = _store_snapshot(sampler) + expected = _reference_store(before, outputs) + store_writes = [] + store_outputs = sampler._store_outputs + + def recording_store_outputs(*args): + store_writes.append(args[0]) + return store_outputs(*args) + + sampler._store_outputs = recording_store_outputs + + state = sampler.sample_async(scheduled, outputs, []) + + assert not hasattr(fold, "store_copies_done") + assert store_writes == [SEQ_SLOTS] + assert len(fold.scatter_calls) == 1 # offered, declined + assert state.sampler_event.side_stream_event is not None + host = _host_outputs(state) + _assert_store_equals(sampler, expected) + for name, tensor in host.items(): + assert torch.equal(tensor, expected[name].cpu()), name + + +def test_flags_off_issues_the_stock_copies_in_the_stock_order(): + """Flags off, sample_async issues exactly the stock calls: four _copy_to_host (new_tokens, new_tokens_lens, + next_draft_tokens, verify_lens) and one argument-less _record_sampler_event.""" + sampler = _sampler() + scheduled, outputs = _step(MAX_DRAFT_LEN) + rows = outputs["new_tokens_lens"].shape[0] + outputs["verify_lens"] = torch.full( + (rows,), MAX_DRAFT_LEN + 1, dtype=torch.int32, device="cuda" + ) + outputs["verify_lens_in_output_order"] = True + copies = [] + events = [] + copy_to_host = sampler._copy_to_host + record_sampler_event = sampler._record_sampler_event + + def recording_copy(src): + copies.append(src) + return copy_to_host(src) + + def recording_event(*args, **kwargs): + events.append((args, kwargs)) + return record_sampler_event(*args, **kwargs) + + sampler._copy_to_host = recording_copy + sampler._record_sampler_event = recording_event + + state = sampler.sample_async(scheduled, outputs, []) + + assert len(copies) == 4 + assert copies[0] is sampler.store.new_tokens + assert copies[1] is sampler.store.new_tokens_lens + assert copies[2] is sampler.store.next_draft_tokens + assert copies[3].data_ptr() == outputs["verify_lens"].data_ptr() + assert copies[3].shape[0] == len(SEQ_SLOTS) + assert events == [((), {})] + assert state.sampler_event.side_stream_event is None + + +def test_side_stream_copies_fall_back_to_the_main_stream_with_the_async_worker(): + sampler = _sampler() + sampler.d2h_side_stream_copies = True + scheduled, outputs = _step(MAX_DRAFT_LEN) + copies = [] + + def plain_copy(src): + # What _copy_to_host does without a worker thread; the worker itself is not started here. + copies.append(src) + dest = torch.empty_like(src, device="cpu", pin_memory=True) + dest.copy_(src, non_blocking=True) + return dest + + sampler._async_worker_active = lambda: True + sampler._copy_to_host = plain_copy + + state = sampler.sample_async(scheduled, outputs, []) + + assert len(copies) == 3 + assert state.sampler_event.side_stream_event is None + + +def test_a_sampler_built_without_init_reads_both_hooks_off(): + sampler = object.__new__(SpecSampler) + + assert sampler.d2h_side_stream_copies is False + assert sampler.static_slot_ids is False + assert sampler._slot_ids_cuda is None and sampler._slot_ids_host is None + + +# HO-12a: static slot ids. + + +def test_static_slot_ids_are_off_by_default(): + assert not _sampler().static_slot_ids + + +@pytest.mark.parametrize("draft_width", [MAX_DRAFT_LEN, MAX_DRAFT_LEN - 1]) +def test_static_slot_ids_store_the_same_rows(draft_width): + scheduled, outputs = _step(draft_width) + plain = _sampler() + expected = _reference_store(_store_snapshot(plain), outputs) + sampler = _sampler() + sampler.static_slot_ids = True + + sampler.sample_async(scheduled, outputs, []) + sampler.sample_async(scheduled, outputs, []) # second step reuses the uploaded ids + + _assert_store_equals(sampler, expected) + + +def test_static_slot_ids_follow_a_changed_batch(): + sampler = _sampler() + sampler.static_slot_ids = True + + first_view = sampler._device_slot_ids([4, 1]) + first = first_view.clone() + # Unchanged ids reuse the sampler's buffer without a new upload. The returned tensor is a view of that buffer, + # which the next call rewrites, so snapshot it before changing the batch. + reused_view = sampler._device_slot_ids([4, 1]) + assert reused_view.data_ptr() == first_view.data_ptr() + reused = reused_view.clone() + changed = sampler._device_slot_ids([2, 5, 0]).clone() + + torch.cuda.synchronize() + assert first.tolist() == [4, 1] + assert reused.tolist() == [4, 1] + assert changed.tolist() == [2, 5, 0] + + +@pytest.fixture +def slot_id_uploads(monkeypatch): + """Counts the copies into the sampler's slot-id buffer (bound with ``.watch(sampler)``).""" + + class _Uploads: + sampler: Optional[SpecSampler] = None + count = 0 + + def watch(self, sampler: SpecSampler) -> "_Uploads": + self.sampler = sampler + return self + + uploads = _Uploads() + copy_ = torch.Tensor.copy_ + + def counting_copy_(dst, src, *args, **kwargs): + buffer = uploads.sampler._slot_ids_cuda if uploads.sampler is not None else None + if ( + buffer is not None + and dst.is_cuda + and dst.untyped_storage().data_ptr() == buffer.untyped_storage().data_ptr() + ): + uploads.count += 1 + return copy_(dst, src, *args, **kwargs) + + monkeypatch.setattr(torch.Tensor, "copy_", counting_copy_) + return uploads + + +def test_steady_slot_ids_upload_once_over_many_steps(slot_id_uploads): + sampler = _sampler() + sampler.static_slot_ids = True + uploads = slot_id_uploads.watch(sampler) + scheduled, outputs = _step(MAX_DRAFT_LEN) + + for _ in range(5): + sampler.sample_async(scheduled, outputs, []) + + assert uploads.count == 1 + torch.cuda.synchronize() + assert sampler._slot_ids_cuda[: len(SEQ_SLOTS)].tolist() == SEQ_SLOTS + + +def test_a_changed_slot_list_re_uploads(slot_id_uploads): + sampler = _sampler() + uploads = slot_id_uploads.watch(sampler) + + sampler._device_slot_ids([4, 1]) + sampler._device_slot_ids([4, 1]) + assert uploads.count == 1 + sampler._device_slot_ids([1, 4]) + assert uploads.count == 2 + sampler._device_slot_ids([1]) # a prefix of the buffer still differs from the last list + assert uploads.count == 3 + view = sampler._device_slot_ids([1]) + assert uploads.count == 3 + torch.cuda.synchronize() + assert view.tolist() == [1] + + +def test_a_longer_slot_list_reallocates_and_uploads(slot_id_uploads): + sampler = _sampler() + uploads = slot_id_uploads.watch(sampler) + + sampler._device_slot_ids([4, 1]) + small = sampler._slot_ids_cuda + assert small.numel() == 8 # max(n, 8) + long_ids = list(range(10)) + view = sampler._device_slot_ids(long_ids) + + assert sampler._slot_ids_cuda is not small and sampler._slot_ids_cuda.numel() == 10 + assert uploads.count == 2 + torch.cuda.synchronize() + assert view.tolist() == long_ids + # The same ids again: no upload into the new buffer. + sampler._device_slot_ids(long_ids) + assert uploads.count == 2 + + +def test_both_hooks_on_store_and_copy_what_the_stock_sampler_does_over_a_changing_batch(): + """The targets run HO-04 and HO-12a together: over steady, reordered and repeated steps the store and the host + copies are bitwise the stock ones.""" + stock = _sampler() + sampler = _sampler() + sampler.d2h_side_stream_copies = True + sampler.static_slot_ids = True + scheduled, outputs = _step(MAX_DRAFT_LEN) + for step in range(4): + if step == 2: # the batch's slot order changes + scheduled.generation_requests = scheduled.generation_requests[::-1] + step_outputs = {name: tensor + step for name, tensor in outputs.items()} + on_main = _host_outputs(stock.sample_async(scheduled, step_outputs, [])) + state = sampler.sample_async(scheduled, step_outputs, []) + assert state.sampler_event.side_stream_event is not None + on_side = _host_outputs(state) + for name, tensor in on_main.items(): + assert torch.equal(on_side[name], tensor), (step, name) + _assert_store_equals(sampler, _store_snapshot(stock)) diff --git a/tests/unittest/_torch/speculative/test_tail_bookkeep.py b/tests/unittest/_torch/speculative/test_tail_bookkeep.py new file mode 100644 index 000000000000..c8a98fecfed7 --- /dev/null +++ b/tests/unittest/_torch/speculative/test_tail_bookkeep.py @@ -0,0 +1,590 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tail bookkeeping (RuntimeHooks.spec_tail_bookkeep -> SpecMetadata.tail_bookkeep): the shared items against the +in-tree path, bitwise. + +* B1 (``advanced_sample_draft_block``): at one decode request the draft block's sampling parameters are views of the + per-token buffers instead of four repeats of the per-request ones. +* S1 (``sample_draft_tokens``): a draft block with no d2t map returns its int32 tokens without the int64 round trip. +* A1 (``_sample_and_accept_draft_tokens_rejection``): a decode-only accept returns the acceptance kernel's outputs + instead of copying them into a ones-filled and an empty buffer. + +Each test runs the same step twice, on a metadata with the flag off (the in-tree path) and one with it on, and compares +every output (and every tensor the samplers were handed) bit for bit. The samplers are deterministic stand-ins that +record their inputs, so the comparison covers the parameter and Philox tensors, not only the tokens. + +The CPU tests fill the per-token buffers with a mirror of ``populate_sampling_params_for_one_model``'s layout and also +check that the flag off takes the in-tree lines. The CUDA tests fill them with the real populate, eagerly and inside +``torch.cuda.graph`` (captured on one request's values, replayed on another's). R1 and Z1 are the M3.1 target's +overrides and are tested with it (test_modeling_v2_m31_tailbk.py). +""" + +import types +from typing import Optional + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder import runner as runner_module +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder.runner import DecoderRunner +from tensorrt_llm._torch.pyexecutor.llm_request import LlmRequestState +from tensorrt_llm._torch.pyexecutor.runtime_hooks import DEFAULT_RUNTIME_HOOKS, RuntimeHooks +from tensorrt_llm._torch.speculative import interface +from tensorrt_llm._torch.speculative.interface import ( + _RNG_SLOT_SPAN, + SpecMetadata, + SpeculativeDecodingMode, + SpecWorkerBase, + tail_bookkeep_draft_len, +) +from tensorrt_llm.llmapi.llm_args import AdvancedSamplingMode + +CUDA = torch.cuda.is_available() +needs_cuda = pytest.mark.skipif(not CUDA, reason="the real populate allocates CUDA buffers") + +V = 64 # vocab +MAX_REQUESTS = 8 +DRAFT_LENS = list(range(1, 8)) # K = 1..7 +# (temperature, top_k, top_p, min_p, seed), all sampled: unseeded, a seed at the int64 edge, a uint64 seed +# (reinterpreted as negative), seed 0, and top-k 2 with a large min-p (top-k 1 would be greedy). +REQUESTS = [ + (0.7, 0, 0.9, 0.0, None), + (1.3, 17, 0.5, 0.05, (1 << 63) - 1), + (0.25, 3, 1.0, 0.2, (1 << 64) - 1), + (1.0, 0, 0.95, 0.0, 0), + (2.0, 2, 0.1, 0.5, 1234), +] + + +class _Worker(SpecWorkerBase): + """SpecWorkerBase without its constructor (which needs flashinfer): the state the sampler tail reads.""" + + def _forward_impl(self, *args, **kwargs): # pragma: no cover - never run + raise NotImplementedError + + @property + def max_draft_len(self) -> int: + return max(DRAFT_LENS) + + +def _worker(d2t: Optional[torch.Tensor] = None, force_num_accepted_tokens: float = 0.0) -> _Worker: + worker = object.__new__(_Worker) + worker._d2t = d2t + worker.force_num_accepted_tokens = force_num_accepted_tokens + worker.mapping = None + return worker + + +def _request(params, slot: int = 3, request_id: int = 7): + temperature, top_k, top_p, min_p, seed = params + return types.SimpleNamespace( + sampling_config=types.SimpleNamespace( + temperature=temperature, top_k=top_k, top_p=top_p, min_p=min_p, seed=seed + ), + state=LlmRequestState.GENERATION_IN_PROGRESS, + py_seq_slot=slot, + py_request_id=request_id, + py_decoding_iter=0, + ) + + +def _meta(k: int, on: bool, mode=AdvancedSamplingMode.NO_TOPK) -> SpecMetadata: + """A one-model DSpark metadata with rejection sampling (the advanced draft path) at draft length ``k``.""" + meta = SpecMetadata( + max_num_requests=MAX_REQUESTS, + max_draft_len=k, + max_total_draft_tokens=k, + spec_dec_mode=SpeculativeDecodingMode.DSPARK, + ) + assert meta.spec_dec_mode.use_one_engine() + meta.runtime_draft_len = k + meta.advanced_sampling_mode = mode + meta.use_rejection_sampling = True + meta.vocab_size = V + meta.draft_vocab_size = V + meta.tail_bookkeep = on + return meta + + +def _fill_like_populate(meta: SpecMetadata, params, offset: int, device: str = "cpu") -> None: + """CPU mirror of populate_sampling_params_for_one_model + _populate_request_rng_state for one generation request + (the in-tree code allocates on CUDA): request_* hold its values, its K + 1 per-token rows hold the same values and + the offset window ``offset + r * SPAN``; every other row holds junk, so a wrong slice shows.""" + temperature, top_k, top_p, min_p, seed = params + k = meta.max_draft_len + rows = (k + 1) * MAX_REQUESTS + junk = torch.arange(rows, device=device) + + def per_token(value, dtype): + out = (junk + 1000).to(dtype) + out[: k + 1] = value + return out + + def per_request(value, dtype): + out = (junk[:MAX_REQUESTS] + 2000).to(dtype) + out[0] = value + return out + + seed = interface.DEFAULT_SAMPLING_SEED if seed is None else seed + seed = seed - (1 << 64) if seed >= (1 << 63) else seed + meta.temperatures = per_token(temperature, torch.float32) + meta.top_ks = per_token(top_k, torch.int32) + meta.top_ps = per_token(top_p, torch.float32) + meta.min_ps = per_token(min_p, torch.float32) + meta.request_temperatures = per_request(temperature, torch.float32) + meta.request_top_ks = per_request(top_k, torch.int32) + meta.request_top_ps = per_request(top_p, torch.float32) + meta.request_min_ps = per_request(min_p, torch.float32) + meta.request_seeds = per_request(seed, torch.int64) + meta.request_offsets = per_request(offset, torch.int64) + meta.seeds = per_token(seed, torch.int64) + meta.offsets = (junk + 3000).to(torch.int64) + meta.offsets[: k + 1] = torch.tensor([offset + r * _RNG_SLOT_SPAN for r in range(k + 1)]) + meta.is_all_greedy_sample = False + + +def _populate(meta: SpecMetadata, params, slot: int = 3, request_id: int = 7) -> None: + """The real populate for a one-request decode batch (CUDA).""" + meta.populate_sampling_params_for_one_model([_request(params, slot, request_id)]) + assert not meta.is_all_greedy_sample + + +class _Sampler: + """A deterministic stand-in for the advanced samplers: records (a copy of) every tensor it is handed and returns + the argmax as int32 tokens with softmax probs.""" + + def __init__(self): + self.seen = [] + + def __call__( + self, mode, logits, temperatures, top_ks, top_ps, min_ps, *, seed=None, offset=None + ): + self.seen.append( + tuple(t.clone() for t in (temperatures, top_ks, top_ps, min_ps, seed, offset)) + ) + return logits.argmax(-1).to(torch.int32), torch.softmax(logits, dim=-1) + + +def _fake_rejection(draft_probs, draft_token_ids, target_probs, seed, offset): + """A stand-in acceptance kernel: fresh int32 outputs that depend on every input, as the in-tree op's.""" + bonus = target_probs[:, -1].argmax(-1, keepdim=True).to(torch.int32) + accepted = torch.cat([draft_token_ids, bonus], dim=1) + num = ((seed ^ offset) % (draft_token_ids.shape[1] + 1) + 1).to(torch.int32) + num = num + (draft_probs.sum(dim=(1, 2)) < 0).to(torch.int32) # reads the draft probs (adds 0) + return accepted, num + + +def _patch_samplers(monkeypatch) -> _Sampler: + sampler = _Sampler() + monkeypatch.setattr(SpecWorkerBase, "_sample_from_logits_with_probs", staticmethod(sampler)) + monkeypatch.setattr( + SpecWorkerBase, + "_compute_probs_from_logits", + staticmethod(lambda mode, logits, t, k, p, m: torch.softmax(logits / t.unsqueeze(-1), -1)), + ) + monkeypatch.setattr(interface, "rejection_sampling_one_model", _fake_rejection) + return sampler + + +def _assert_same(a, b, what: str) -> None: + if isinstance(a, (tuple, list)): + assert isinstance(b, (tuple, list)) and len(a) == len(b), what + for i, (x, y) in enumerate(zip(a, b)): + _assert_same(x, y, f"{what}[{i}]") + return + if a is None or b is None: + assert a is None and b is None, what + return + assert a.dtype == b.dtype and a.shape == b.shape, ( + f"{what}: {a.dtype}{tuple(a.shape)} vs {b.dtype}{tuple(b.shape)}" + ) + if a.dtype == torch.float32: + a, b = a.view(torch.int32), b.view(torch.int32) # bits, so -0.0 / NaN payloads count too + assert torch.equal(a.cpu(), b.cpu()), what + + +# --------------------------------------------------------------------------------------------------------------------- +# The switch: off by default everywhere, copied to the spec metadata. +# --------------------------------------------------------------------------------------------------------------------- + + +@pytest.mark.cpu_only +def test_the_hook_and_the_metadata_flag_are_off_by_default(): + assert RuntimeHooks().spec_tail_bookkeep is False + assert DEFAULT_RUNTIME_HOOKS.spec_tail_bookkeep is False + assert SpecMetadata.tail_bookkeep is False + assert ( + SpecMetadata(max_num_requests=1, max_draft_len=1, max_total_draft_tokens=1).tail_bookkeep + is False + ) + assert not RuntimeHooks(spec_tail_bookkeep=True).skips_unchanged_metadata_uploads + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("on", [False, True]) +def test_setup_speculative_copies_the_hook_and_logs_nothing(monkeypatch, on): + created = types.SimpleNamespace() + lines = [] + monkeypatch.setattr(runner_module, "create_spec_metadata", lambda *args: created) + monkeypatch.setattr(runner_module.logger, "info", lines.append) + metadata = object.__new__(DecoderRunner) + metadata.runtime_hooks = RuntimeHooks(spec_tail_bookkeep=on) + metadata.spec_metadata = None + metadata._config = None + metadata.model = types.SimpleNamespace(config=None) + + assert metadata._set_up_spec_metadata(None) is created + assert created.tail_bookkeep is on + assert lines == [] # the arm line is the target's (it knows which items its worker takes) + + +# --------------------------------------------------------------------------------------------------------------------- +# The shared precondition (host state and shapes only). +# --------------------------------------------------------------------------------------------------------------------- + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("k", DRAFT_LENS) +def test_the_precondition_holds_for_one_decode_request(k): + meta = _meta(k, on=True) + _fill_like_populate(meta, REQUESTS[0], offset=0) + assert tail_bookkeep_draft_len(meta, 0, 1) == k + + +def _breaks(): + def flag_off(meta): + meta.tail_bookkeep = False + + def tree(meta): + meta.is_spec_dec_tree = True + + def dynamic_tree(meta): + meta.is_spec_dec_dynamic_tree = True + + def ragged(meta): + meta.verify_lens = torch.ones(1, dtype=torch.int32) + + def dynamic_draft_len(meta): + meta.runtime_draft_len = meta.max_draft_len - 1 + + def short_buffer(meta): + meta.seeds = meta.seeds[: meta.max_draft_len] + + def missing_buffer(meta): + meta.min_ps = None + + def wrong_dtype(meta): + meta.offsets = meta.offsets.to(torch.int32) + + def strided_buffer(meta): + meta.top_ps = torch.stack([meta.top_ps, meta.top_ps], dim=1)[:, 0] + + return [flag_off, tree, dynamic_tree, ragged, dynamic_draft_len, short_buffer, missing_buffer, wrong_dtype, + strided_buffer] # fmt: skip + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("brk", _breaks(), ids=lambda f: f.__name__) +def test_the_precondition_fails_closed(brk): + meta = _meta(3, on=True) + _fill_like_populate(meta, REQUESTS[0], offset=0) + brk(meta) + assert tail_bookkeep_draft_len(meta, 0, 1) == 0 + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("num_contexts,batch_size", [(0, 2), (1, 2), (0, 0), (1, 1), (0, None)]) +def test_the_precondition_needs_exactly_one_generation_request(num_contexts, batch_size): + meta = _meta(3, on=True) + _fill_like_populate(meta, REQUESTS[0], offset=0) + assert tail_bookkeep_draft_len(meta, num_contexts, batch_size) == 0 + + +# --------------------------------------------------------------------------------------------------------------------- +# B1 + S1: the draft block (sample_draft_tokens, block form). +# --------------------------------------------------------------------------------------------------------------------- + + +def _draft_block(meta, logits, d2t=None): + worker = _worker(d2t=d2t) + tokens = worker.sample_draft_tokens(logits, meta, 1, num_contexts=0) + return tokens + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("k", DRAFT_LENS) +@pytest.mark.parametrize("params", REQUESTS) +def test_draft_block_cpu_on_matches_off(monkeypatch, k, params): + sampler = _patch_samplers(monkeypatch) + logits = torch.randn(1, k, V, generator=torch.Generator().manual_seed(k)) + outs = {} + for on in (False, True): + meta = _meta(k, on) + _fill_like_populate(meta, params, offset=5 * k) + tokens = _draft_block(meta, logits) + outs[on] = (tokens, sampler.seen[-1]) + _assert_same(outs[False], outs[True], "tokens + sampler inputs") + assert outs[True][0].dtype == torch.int32 + + +@pytest.mark.cpu_only +def test_draft_block_cpu_flag_off_takes_the_in_tree_lines(monkeypatch): + """Off: the parameters are fresh repeats (not views of the per-token buffers) and the tokens a fresh int32 copy + (not the sampler's tensor).""" + k = 4 + seen_tokens = [] + + def sampler(mode, logits, temperatures, top_ks, top_ps, min_ps, *, seed=None, offset=None): + tokens = logits.argmax(-1).to(torch.int32) + seen_tokens.append(tokens) + sampler.params = (temperatures, top_ks, top_ps, min_ps) + return tokens, torch.softmax(logits, -1) + + monkeypatch.setattr(SpecWorkerBase, "_sample_from_logits_with_probs", staticmethod(sampler)) + logits = torch.randn(1, k, V) + for on in (False, True): + meta = _meta(k, on) + _fill_like_populate(meta, REQUESTS[1], offset=0) + tokens = _draft_block(meta, logits) + views = [p.untyped_storage().data_ptr() for p in sampler.params] + bufs = [meta.temperatures, meta.top_ks, meta.top_ps, meta.min_ps] + shared = [v == b.untyped_storage().data_ptr() for v, b in zip(views, bufs)] + aliases_sampler = ( + tokens.untyped_storage().data_ptr() == seen_tokens[-1].untyped_storage().data_ptr() + ) + assert shared == [on] * 4 + assert aliases_sampler is on + + +@pytest.mark.cpu_only +def test_s1_keeps_the_round_trip_with_a_d2t_map(monkeypatch): + _patch_samplers(monkeypatch) + k = 3 + d2t = torch.arange(V, dtype=torch.int64) % 5 + logits = torch.randn(1, k, V) + outs = [] + for on in (False, True): + meta = _meta(k, on) + _fill_like_populate(meta, REQUESTS[0], offset=0) + outs.append(_draft_block(meta, logits, d2t=d2t)) + _assert_same(outs[0], outs[1], "d2t-mapped tokens") + + +@pytest.mark.cpu_only +def test_b1_is_not_taken_for_a_padded_batch(monkeypatch): + """Two generation requests (a padded graph): the per-request repeats run with the flag on too.""" + sampler = _patch_samplers(monkeypatch) + k = 3 + meta = _meta(k, on=True) + _fill_like_populate(meta, REQUESTS[0], offset=0) + worker = _worker() + worker.advanced_sample_draft_block(torch.randn(2, k, V), meta, 0, 2) + temps = sampler.seen[-1][0] + assert torch.equal(temps, meta.request_temperatures[:2].repeat_interleave(k)) + + +# --------------------------------------------------------------------------------------------------------------------- +# A1: the rejection accept. +# --------------------------------------------------------------------------------------------------------------------- + + +def _accept(meta, logits, draft_tokens, draft_probs, num_contexts, batch_size, force=0.0): + worker = _worker(force_num_accepted_tokens=force) + return worker._sample_and_accept_draft_tokens_rejection( + logits, draft_tokens, draft_probs, num_contexts, batch_size, meta + ) + + +def _accept_inputs(k, num_gens, device="cpu", seed=0): + g = torch.Generator(device=device).manual_seed(seed) + logits = torch.randn(num_gens * (k + 1), V, generator=g, device=device) + draft_tokens = torch.randint(0, V, (num_gens, k), generator=g, device=device, dtype=torch.int32) + draft_probs = torch.softmax(torch.randn(num_gens, k, V, generator=g, device=device), -1) + return logits, draft_tokens, draft_probs + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("k", DRAFT_LENS) +@pytest.mark.parametrize("force", [0.0, 2.0]) +def test_accept_cpu_on_matches_off(monkeypatch, k, force): + _patch_samplers(monkeypatch) + logits, draft_tokens, draft_probs = _accept_inputs(k, 1, seed=k) + outs = [] + for on in (False, True): + meta = _meta(k, on) + _fill_like_populate(meta, REQUESTS[2], offset=96 * k) + outs.append(_accept(meta, logits, draft_tokens, draft_probs, 0, 1, force=force)) + _assert_same(outs[0], outs[1], "accepted tokens + counts") + + +@pytest.mark.cpu_only +def test_accept_flag_on_returns_the_kernel_outputs(monkeypatch): + returned = [] + + def rejection(*args, **kwargs): + returned.append(_fake_rejection(*args, **kwargs)) + return returned[-1] + + _patch_samplers(monkeypatch) + monkeypatch.setattr(interface, "rejection_sampling_one_model", rejection) + k = 3 + logits, draft_tokens, draft_probs = _accept_inputs(k, 1) + for on in (False, True): + meta = _meta(k, on) + _fill_like_populate(meta, REQUESTS[0], offset=0) + accepted, num = _accept(meta, logits, draft_tokens, draft_probs, 0, 1) + assert (accepted is returned[-1][0]) is on + assert (num is returned[-1][1]) is on + + +@pytest.mark.cpu_only +@pytest.mark.parametrize( + "bad", + ["int64_tokens", "int64_counts", "strided_tokens"], +) +def test_accept_falls_back_when_the_kernel_outputs_differ_in_layout(monkeypatch, bad): + def rejection(*args, **kwargs): + accepted, num = _fake_rejection(*args, **kwargs) + if bad == "int64_tokens": + accepted = accepted.long() + elif bad == "int64_counts": + num = num.long() + else: + strided = torch.zeros(accepted.shape[0], 2 * accepted.shape[1], dtype=accepted.dtype) + strided[:, ::2] = accepted + accepted = strided[:, ::2] + return accepted, num + + _patch_samplers(monkeypatch) + monkeypatch.setattr(interface, "rejection_sampling_one_model", rejection) + k = 3 + logits, draft_tokens, draft_probs = _accept_inputs(k, 1) + outs = [] + for on in (False, True): + meta = _meta(k, on) + _fill_like_populate(meta, REQUESTS[0], offset=0) + outs.append(_accept(meta, logits, draft_tokens, draft_probs, 0, 1)) + _assert_same(outs[0], outs[1], "fallback") + assert outs[1][0].dtype == torch.int32 and outs[1][0].is_contiguous() + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("num_gens", [2, 4]) +def test_accept_with_several_requests_matches_off(monkeypatch, num_gens): + """A1 is not limited to one request: any decode-only accept returns the kernel's outputs.""" + _patch_samplers(monkeypatch) + k = 3 + logits, draft_tokens, draft_probs = _accept_inputs(k, num_gens) + outs = [] + for on in (False, True): + meta = _meta(k, on) + _fill_like_populate(meta, REQUESTS[0], offset=0) + outs.append(_accept(meta, logits, draft_tokens, draft_probs, 0, num_gens)) + _assert_same(outs[0], outs[1], "accept") + + +@pytest.mark.cpu_only +def test_accept_with_a_context_row_takes_the_in_tree_lines(monkeypatch): + """num_contexts > 0: the outputs are assembled into fresh buffers whatever the flag.""" + _patch_samplers(monkeypatch) + monkeypatch.setattr( + _Worker, + "_sample_tokens_for_batch", + lambda self, logits, meta, n, b: logits.argmax(-1).to(torch.int32), + ) + returned = [] + + def rejection(*args, **kwargs): + returned.append(_fake_rejection(*args, **kwargs)) + return returned[-1] + + monkeypatch.setattr(interface, "rejection_sampling_one_model", rejection) + k = 3 + gen_logits, draft_tokens, draft_probs = _accept_inputs(k, 1) + logits = torch.cat([torch.randn(1, V), gen_logits]) + outs = [] + for on in (False, True): + meta = _meta(k, on) + _fill_like_populate(meta, REQUESTS[0], offset=0) + accepted, num = _accept(meta, logits, draft_tokens, draft_probs, 1, 2) + assert accepted is not returned[-1][0] and num is not returned[-1][1] + # The context row defines only its first token (the rest of its row is the in-tree torch.empty), so compare + # the defined entries: the context row's first token, the generation rows and every count. + outs.append((accepted[:1, :1], accepted[1:], num)) + _assert_same(outs[0], outs[1], "mixed batch") + + +# --------------------------------------------------------------------------------------------------------------------- +# CUDA: the real populate, eager and inside torch.cuda.graph. +# --------------------------------------------------------------------------------------------------------------------- + + +def _graph(fn): + """Capture ``fn`` (after a side-stream warmup, as torch.cuda.graphs asks); returns (graph, captured outputs).""" + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + fn() + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = fn() + return graph, out + + +def _step(meta, worker, logits, accept_inputs, sampler): + """One sampled step's tail: the rejection accept, then the draft block; the outputs and the draft-prob row.""" + v_logits, draft_tokens, draft_probs = accept_inputs + accepted = worker._sample_and_accept_draft_tokens_rejection( + v_logits, draft_tokens, draft_probs, 0, 1, meta + ) + tokens = worker.sample_draft_tokens(logits, meta, 1, num_contexts=0) + return accepted, tokens, sampler.seen[-1] + + +@needs_cuda +@pytest.mark.parametrize("k", DRAFT_LENS) +@pytest.mark.parametrize("mode", [AdvancedSamplingMode.NO_TOPK, AdvancedSamplingMode.FULL]) +def test_cuda_eager_on_matches_off(monkeypatch, k, mode): + sampler = _patch_samplers(monkeypatch) + worker = _worker() + metas = {on: _meta(k, on, mode=mode) for on in (False, True)} + logits = torch.randn(1, k, V, device="cuda") + inputs = _accept_inputs(k, 1, device="cuda", seed=k) + for step, params in enumerate(REQUESTS): + outs = {} + for on, meta in metas.items(): + _populate(meta, params) + outs[on] = _step(meta, worker, logits + step, inputs, sampler) + outs[on] += (meta.draft_probs[3, :k].clone(),) + torch.cuda.synchronize() + _assert_same(outs[False], outs[True], f"step {step}") + + +@needs_cuda +@pytest.mark.parametrize("k", [1, 3, 7]) +def test_cuda_graph_on_matches_off(monkeypatch, k): + """The ON step captured on the first request's values, then replayed after each later populate (same buffers, + new values), against the in-tree step run eagerly on the same values.""" + sampler = _patch_samplers(monkeypatch) + worker = _worker() + on_meta, off_meta = _meta(k, True), _meta(k, False) + logits = torch.randn(1, k, V, device="cuda") + inputs = _accept_inputs(k, 1, device="cuda", seed=k) + _populate(on_meta, REQUESTS[0]) + _populate(off_meta, REQUESTS[0]) + graph, captured = _graph(lambda: _step(on_meta, worker, logits, inputs, sampler)) + for step, params in enumerate(REQUESTS[1:], start=1): + _populate(on_meta, params) + graph.replay() + _populate(off_meta, params) + expected = _step(off_meta, worker, logits, inputs, sampler) + torch.cuda.synchronize() + _assert_same(captured, expected, f"replay {step}") + _assert_same(on_meta.draft_probs[3, :k], off_meta.draft_probs[3, :k], f"draft probs {step}") + + +if __name__ == "__main__": + import sys + + sys.exit(pytest.main([__file__, "-v"])) From 647ee5c13d10fc4e7fb9f1a0a275503fbead4a50 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Fri, 23 Oct 2026 18:09:59 -0700 Subject: [PATCH 19/38] [None][fix] Share per-row RNG subsequence constant with chain sampling Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- cpp/tensorrt_llm/kernels/fusedSampling/fusedSamplingKernels.h | 4 ++++ .../kernels/fusedSampling/fusedSamplingKernelsCommon.cuh | 4 ---- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/cpp/tensorrt_llm/kernels/fusedSampling/fusedSamplingKernels.h b/cpp/tensorrt_llm/kernels/fusedSampling/fusedSamplingKernels.h index 8863035e5740..5a267eb9ca4b 100644 --- a/cpp/tensorrt_llm/kernels/fusedSampling/fusedSamplingKernels.h +++ b/cpp/tensorrt_llm/kernels/fusedSampling/fusedSamplingKernels.h @@ -21,6 +21,10 @@ namespace tensorrt_llm { +//! Philox subsequences from here up hold per-row streams. A shared seed/offset makes the row +//! index the subsequence, which never reaches them. +constexpr uint64_t kPerRowSubsequenceBase = 1ull << 63; + namespace kernels { diff --git a/cpp/tensorrt_llm/kernels/fusedSampling/fusedSamplingKernelsCommon.cuh b/cpp/tensorrt_llm/kernels/fusedSampling/fusedSamplingKernelsCommon.cuh index f8b6c31de06b..62b669ed4a84 100644 --- a/cpp/tensorrt_llm/kernels/fusedSampling/fusedSamplingKernelsCommon.cuh +++ b/cpp/tensorrt_llm/kernels/fusedSampling/fusedSamplingKernelsCommon.cuh @@ -129,10 +129,6 @@ struct RowParams bool needMinP; }; -//! Philox subsequences from here up hold per-row streams. A shared seed/offset makes the row -//! index the subsequence, which never reaches them. -constexpr uint64_t kPerRowSubsequenceBase = 1ull << 63; - //! Seeds one row's Philox stream (see FusedSamplingParams::perRowRng). __device__ inline void initRowRng(FusedSamplingParams const& p, int row, curandStatePhilox4_32_10_t* state) { From 11130604217f9e235ac22e64299fba14ec1e6874 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Sat, 24 Oct 2026 00:01:04 -0700 Subject: [PATCH 20/38] [None][test] Restore GB300 TP4 test operand helpers Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../attention/_m3_nvfp4_cache.py | 104 +++++++++++++++++ .../gemm/_m3_mxfp8_operands.py | 30 +++++ .../spec/_m3_spec_inputs.py | 106 ++++++++++++++++++ 3 files changed, 240 insertions(+) create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/_m3_nvfp4_cache.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/_m3_mxfp8_operands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/_m3_spec_inputs.py diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/_m3_nvfp4_cache.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/_m3_nvfp4_cache.py new file mode 100644 index 000000000000..2254f6c0b68c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/attention/_m3_nvfp4_cache.py @@ -0,0 +1,104 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""A MiniMax-M3 TP4 sparse-layer decode step over a packed NVFP4 cache, for the attention entries' tests. + +The cache is written by the in-tree scatter (``fused_write_layer_caches_nvfp4``, +which quantizes with ``torch.ops.trtllm.fp4_quantize``), into one +``[pages, 2 (K, V), 1, 128, .]`` pool per role exactly as ``msa_paged_kv`` and +``get_block_scale_buffers`` hand the K and V views over. Pages are shuffled so +a kernel that ignored the block table would read the wrong ones. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from typing import Sequence + +import torch + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_scatter import ( + fused_write_layer_caches_nvfp4, +) + +NUM_HEADS = 16 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 + + +def token_kv_len(seq_len: int, decode_query_len: int, offset: int) -> int: + """The KV extent query token ``offset`` of a request attends, as every decode kernel derives it.""" + return max(seq_len - decode_query_len + offset + 1, 0) + + +def build_step( + seq_lens: Sequence[int], decode_query_len: int, *, identity: bool, seed: int +) -> SimpleNamespace: + """One decode step: E4M3 q [T, 16, 128], the packed NVFP4 K/V views and their scales, a [1, T, 16] + head-major top-k table (the identity table, or a random ascending selection with a -1 tail), the block + table and per-request lengths.""" + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + total_q = batch * decode_query_len + max_blocks = max(1, max((s + PAGE - 1) // PAGE for s in seq_lens)) + num_pages = batch * max_blocks + num_slots = num_pages * PAGE + block_table = ( + torch.randperm(num_pages, device="cuda", generator=g) + .to(torch.int32) + .reshape(batch, max_blocks) + ) + data_pool = torch.zeros(num_pages, 2, 1, PAGE, HEAD_DIM // 2, dtype=torch.uint8, device="cuda") + scale_pool = torch.zeros( + num_pages, 2, 1, PAGE, HEAD_DIM // 16, dtype=torch.uint8, device="cuda" + ) + quant = torch.ones(3, dtype=torch.float32, device="cuda") + values = [] + for role in (0, 1): + x = torch.randn(num_slots, HEAD_DIM, device="cuda", generator=g, dtype=torch.float32).to( + torch.bfloat16 + ) + quant[role + 1] = (448.0 * 6.0) / x.abs().max().float().item() + values.append(x) + assert fused_write_layer_caches_nvfp4( + data_pool[:, 0], + data_pool[:, 1], + scale_pool[:, 0], + scale_pool[:, 1], + None, + torch.arange(num_slots, device="cuda", dtype=torch.int32), + values[0], + values[1], + None, + quant, + ) + q = torch.randn(total_q, NUM_HEADS, HEAD_DIM, device="cuda", generator=g).to( + torch.float8_e4m3fn + ) + topk = torch.full((1, total_q, TOPK), -1, dtype=torch.int32, device="cuda") + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + nblocks = (token_kv_len(seq_lens[req], decode_query_len, offset) + PAGE - 1) // PAGE + if identity: + assert nblocks <= TOPK, "an identity step has every request within 16 blocks" + chosen = torch.arange(nblocks, device="cuda") + else: + chosen = torch.randperm(nblocks, device="cuda", generator=g)[:TOPK].sort().values + topk[0, tok, : chosen.numel()] = chosen.to(torch.int32) + return SimpleNamespace( + q=q, + k_paged=data_pool[:, 0], + v_paged=data_pool[:, 1], + k_block_scale=scale_pool[:, 0], + v_block_scale=scale_pool[:, 1], + k_global_scale=(1.0 / quant[1:2]).contiguous(), + v_global_scale=(1.0 / quant[2:3]).contiguous(), + topk=topk, + block_table=block_table, + seq_lens=torch.tensor(list(seq_lens), dtype=torch.int32, device="cuda"), + decode_query_len=decode_query_len, + sm_scale=HEAD_DIM**-0.5, + source=values, + quant=quant, + num_slots=num_slots, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/_m3_mxfp8_operands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/_m3_mxfp8_operands.py new file mode 100644 index 000000000000..e44ad56d743d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/gemm/_m3_mxfp8_operands.py @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MXFP8 operands for the MXFP8 GEMM entries' tests: E4M3 values and 128x4-swizzled UE8M0 block scales.""" + +import torch + + +def swizzle(byte: torch.Tensor) -> torch.Tensor: + """[N, K / 32] scale bytes into the 128x4-swizzled buffer of ``mxfp8_quantize(w, True)``.""" + rows, cols = byte.shape + padded_cols, padded_rows = (cols + 3) // 4 * 4, (rows + 127) // 128 * 128 + r = torch.arange(rows, device=byte.device).view(-1, 1) + c = torch.arange(cols, device=byte.device).view(1, -1) + index = ( + (c % 4) + + (c // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * padded_cols) + ) + out = torch.zeros(padded_rows * padded_cols, dtype=torch.uint8, device=byte.device) + out[index.reshape(-1)] = byte.reshape(-1) + return out + + +def mxfp8(rows: int, k: int, g: torch.Generator, lo: int = 118, hi: int = 125): + """An MXFP8 operand: E4M3 values of a unit normal, UE8M0 block scales 2^(lo-127) .. 2^(hi-128).""" + v = torch.randn(rows, k, generator=g, device="cuda").to(torch.float8_e4m3fn) + sf = torch.randint(lo, hi, (rows, k // 32), generator=g, device="cuda", dtype=torch.int32) + return v, swizzle(sf.to(torch.uint8)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/_m3_spec_inputs.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/_m3_spec_inputs.py new file mode 100644 index 000000000000..1ab56b80f960 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_103__tp4/spec/_m3_spec_inputs.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Inputs shared by the tp4 target's spec entry tests: TP4 dimensions, single-device stand-ins for the cross-rank +exchanges, and the comparison bands.""" + +from __future__ import annotations + +import zlib + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_103__tp4.ops.spec.m3_shard_accept import ( # noqa: E501 + SHARD_ROWS, + make_keys, +) + +#: MiniMax-M3 at TP4: hidden size, vocabulary, the target head's shard and RMSNorm epsilon. +HIDDEN = 6144 +VOCAB = 200064 +WORLD = 4 +TARGET_SHARD = VOCAB // WORLD +EPS = 1e-6 + + +def seed_of(why: str) -> int: + return zlib.crc32(why.encode()) % 2**31 + + +def gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def num_sms() -> int: + return torch.cuda.get_device_properties(torch.cuda.current_device()).multi_processor_count + + +def randn(shape, g: torch.Generator, std: float = 1.0, dtype=torch.bfloat16) -> torch.Tensor: + return (torch.randn(shape, generator=g, device="cuda") * std).to(dtype) + + +def head_workspace(world: int, call: int = 0, slot_rows: int = SHARD_ROWS): + """(ws_uc int64 [2 * slot_rows * world], keys int64 [slot_rows], counters int32 [2]) of a fresh call ``call``.""" + uc = torch.zeros(2 * slot_rows * world, dtype=torch.int64, device="cuda") + keys = torch.zeros(slot_rows, dtype=torch.int64, device="cuda") + counters = torch.tensor([0, call], dtype=torch.int32, device="cuda") + return uc, keys, counters + + +def plant_peers( + uc: torch.Tensor, + counters: torch.Tensor, + own_best: torch.Tensor, + rank: int, + world: int, + shard: int, + g: torch.Generator, + slot_rows: int = SHARD_ROWS, + ties: bool = False, +) -> None: + """Write the other ranks' keys for each row into this rank's slots, as their multicast pushes would. + + Each peer's maximum is the own row maximum shifted by +-0.25 (well clear of the accumulation-order band), so + some rows are won by a peer and some by this rank; with ``ties`` a peer on an even row reports exactly the own + maximum, and the lower global index must win. + """ + if world == 1: + return + m = own_best.shape[0] + half = int(counters[1].item()) & 1 + for r in range(world): + if r == rank: + continue + sign = torch.where(torch.rand(m, generator=g, device="cuda") < 0.5, -1.0, 1.0) + value = own_best.float() + 0.25 * sign + if ties: + value = torch.where(torch.arange(m, device="cuda") % 2 == 0, own_best.float(), value) + index = r * shard + torch.randint(0, shard, (m,), generator=g, device="cuda") + keys = make_keys(value, index) + for t in range(m): + uc[(half * slot_rows + t) * world + r] = keys[t] + + +def assert_bf16_close(out: torch.Tensor, ref: torch.Tensor, max_frac: float = 1e-3) -> None: + """At most one bf16 ulp per element, and only on a ``max_frac`` share of the elements. + + Kernel and reference compute the same fp32 expression and round once; they differ only where a different + summation order (or rsqrtf) puts the fp32 value on the other side of a bf16 rounding boundary. + """ + assert out.shape == ref.shape and out.dtype == ref.dtype == torch.bfloat16 + o, r = out.float(), ref.float() + ulp = torch.ldexp(torch.ones_like(r), torch.frexp(r).exponent - 8) + ulp = torch.where(r == 0, torch.full_like(r, 2.0**-133), ulp) + diff = (o - r).abs() + assert bool((diff <= ulp * 1.0001).all()), f"max |diff| / ulp = {(diff / ulp).max().item():.2f}" + frac = (diff > 0).float().mean().item() + assert frac <= max_frac, f"{frac:.2e} of the elements moved by one ulp (allowed {max_frac})" + + +def craft_drafts(tokens: torch.Tensor, nreq: int, d: int) -> torch.Tensor: + """Drafts matching request b's verify tokens for its first (b + 2) % (d + 1) positions and differing from + there on, so the acceptance counts vary across requests.""" + t = tokens.view(nreq, d + 1)[:, :d].clone() + for b in range(nreq): + n = (b + 2) % (d + 1) + t[b, n:] += 1 + return t.contiguous() From 9d975657e60b8d6232b0f3fac907b5f938305186 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Sat, 24 Oct 2026 18:33:02 -0700 Subject: [PATCH 21/38] [None][fix] Restore DecoderRunner optional state initialization Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../_torch/pyexecutor/engine/runners/decoder/runner.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py index ca6a0068730e..b281c15d0f27 100644 --- a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py +++ b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py @@ -33,6 +33,7 @@ MoeLoadBalancerIterContext, ) from tensorrt_llm._torch.peft.lora.config import LoraConfig +from tensorrt_llm._torch.peft.lora.manager import LoraModelConfig from tensorrt_llm._torch.pyexecutor import h2d_cache, step_inputs from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph_runner import BreakableCUDAGraphRunner from tensorrt_llm._torch.pyexecutor.cuda_graph_runner import ( @@ -41,6 +42,7 @@ KeyType, get_mrope_dummy_seq_slot, ) +from tensorrt_llm._torch.pyexecutor.guided_decoder import CapturableGuidedDecoder from tensorrt_llm._torch.pyexecutor.h2d_cache import H2DSkipCache, copy_h2d_if_changed from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import ( @@ -396,6 +398,8 @@ def __init__( if self._h2d_cache is not None and self._h2d_cache.changed_span: logger.info("[M3 host] changed metadata uploads stage only each row's changed span") + self.guided_decoder: CapturableGuidedDecoder | None = None + self.lora_model_config: LoraModelConfig | None = None self.forward_pass_callable = None # Optional in-graph sampling hook, registered by PyExecutor. Called at # the tail of _forward_step -- i.e. right after the LM head, and inside From 162e93efb6b6d9e9ee46282eae774151d9761661 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 11:32:29 -0700 Subject: [PATCH 22/38] [None][feat] Add Rubin (sm_107) MiniMax-M3.1 low-latency decode targets Add the MiniMax-M3.1 NVFP4 decode targets for Rubin (sm_107) at TP2, TP4 and TP8, stacked on the GB300 low-latency decode targets. - Decode megakernel paths for sm_107: fused attention / MoE engines, multicast all-reduce + norm, the F3 prepushed all-reduce, the S2 MoE tail, DSpark drafting with C++ glue ops, and deterministic top-p. - Native SwiGLU-OAI and MoE finalize ops for the TP2 target, bit for bit the Triton kernels they replace. - CuTe DSL kernels loaded from ahead-of-time objects shipped in the wheel, keyed on the toolchain, kernel sources and model switches. - Bundled tactic pins (name-keyed, plus index-keyed entries under a native-build guard), also served under TRTLLM_CACHE_DIR. - Disaggregated serving: the DSpark draft context window transfers with the KV cache. - LlmRequest takes context-only prompt tokens as an int32 array. - DFlash rejection sampling writes full-width context one-hot draft probabilities. - nvidia-cutlass-dsl is pinned to the 4.8.0 release. Each optimization is on by default with a TRTLLM_M31_DISABLE_ kill switch. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../vendor_patches/flashinfer-prims-ts.patch | 459 +- 3rdparty/vendor_sources.lock.yaml | 4 +- cpp/tensorrt_llm/kernels/CMakeLists.txt | 38 +- .../kernels/minimaxM3/routerGemv.cu | 67 +- .../kernels/minimaxM3/routerGemv.h | 4 + .../kernels/minimaxM3/specTail.cu | 30 +- .../minimaxM31/chainSpecSamplingMcSm103.cu | 489 +- .../minimaxM31/chainSpecSamplingMcSm103.h | 19 +- .../kernels/minimaxM31/dsparkGlueSm107.cu | 209 + .../kernels/minimaxM31/dsparkGlueSm107.h | 59 + .../kernels/minimaxM31/f3ArPrepushedSm107.cu | 467 + .../kernels/minimaxM31/f3ArPrepushedSm107.h | 61 + .../kernels/minimaxM31/markovEpilogueSm107.cu | 178 + .../kernels/minimaxM31/markovEpilogueSm107.h | 68 + .../kernels/minimaxM31/moeFinalizeSm107.cu | 190 + .../kernels/minimaxM31/moeFinalizeSm107.h | 45 + .../kernels/minimaxM31/s1bFoldSm107.cu | 146 + .../kernels/minimaxM31/s1bFoldSm107.h | 46 + .../kernels/minimaxM31/s2MoeTailSm107.cu | 445 + .../kernels/minimaxM31/s2MoeTailSm107.h | 52 + .../kernels/minimaxM31/scatterRowsSm107.cu | 80 + .../kernels/minimaxM31/scatterRowsSm107.h | 43 + .../kernels/minimaxM31/softmaxPathBSm107.cu | 259 + .../kernels/minimaxM31/softmaxPathBSm107.h | 39 + .../kernels/minimaxM31/swigluOaiVecSm107.cu | 176 + .../kernels/minimaxM31/swigluOaiVecSm107.h | 42 + .../kernels/minimaxM31/tailBk3Sm107.cu | 111 + .../kernels/minimaxM31/tailBk3Sm107.h | 51 + .../minimaxM31/toppRenormDetV3Sm103.cu | 393 +- .../kernels/minimaxM31/toppRenormDetV3Sm103.h | 13 +- .../minimaxM31/zeroPaddingRowsSm107.cu | 64 + .../kernels/minimaxM31/zeroPaddingRowsSm107.h | 43 + .../minimaxM3Sm107Tp2/denseQkNormRopeFp8.cu | 267 + .../minimaxM3Sm107Tp2/denseQkNormRopeFp8.h | 64 + .../kernels/minimaxM3Sm107Tp2/draftStep.cu | 620 ++ .../kernels/minimaxM3Sm107Tp2/draftStep.h | 111 + .../minimaxM3Sm107Tp2/drafterRopeFp8Kv.cu | 189 + .../minimaxM3Sm107Tp2/drafterRopeFp8Kv.h | 79 + .../kernels/minimaxM3Sm107Tp2/foldTail.cu | 86 + .../kernels/minimaxM3Sm107Tp2/foldTail.h | 72 + .../minimaxM3Sm107Tp2/fullCtxReduce.cu | 223 + .../kernels/minimaxM3Sm107Tp2/fullCtxReduce.h | 71 + .../minimaxM3Sm107Tp2/mcAllreduceNormTp2.cu | 1092 ++ .../minimaxM3Sm107Tp2/mcAllreduceNormTp2.h | 83 + .../kernels/minimaxM3Sm107Tp2/producerPw2.cu | 339 + .../kernels/minimaxM3Sm107Tp2/producerPw2.h | 76 + .../minimaxM3Sm107Tp2/sampledAccept.cu | 596 ++ .../kernels/minimaxM3Sm107Tp2/sampledAccept.h | 82 + .../kernels/minimaxM3Sm107Tp2/selectLong2.cu | 246 + .../kernels/minimaxM3Sm107Tp2/selectLong2.h | 50 + .../kernels/minimaxM3Sm107Tp2/siluAndMul.cu | 89 + .../kernels/minimaxM3Sm107Tp2/siluAndMul.h | 40 + .../minimaxM3Sm107Tp2/sparseDecodeLc2.cu | 640 ++ .../minimaxM3Sm107Tp2/sparseDecodeLc2.h | 77 + .../minimaxM3Sm107Tp2/sparseDecodeLc2Split.cu | 671 ++ .../minimaxM3Sm107Tp2/sparseDecodeLc2Split.h | 46 + .../kernels/minimaxM3Sm107Tp2/specOffsets.cu | 58 + .../kernels/minimaxM3Sm107Tp2/specOffsets.h | 41 + .../kernels/minimaxM3Sm107Tp2/stepPrologue.cu | 111 + .../kernels/minimaxM3Sm107Tp2/stepPrologue.h | 47 + .../kernels/minimaxM3Sm107Tp2/swigluOai.cu | 100 + .../kernels/minimaxM3Sm107Tp2/swigluOai.h | 43 + .../minimaxM3Sm107Tp4/denseQkNormRopeFp8.cu | 296 + .../minimaxM3Sm107Tp4/denseQkNormRopeFp8.h | 68 + .../kernels/minimaxM3Sm107Tp4/draftStep.cu | 667 ++ .../kernels/minimaxM3Sm107Tp4/draftStep.h | 107 + .../minimaxM3Sm107Tp4/drafterRopeFp8Kv.cu | 189 + .../minimaxM3Sm107Tp4/drafterRopeFp8Kv.h | 79 + .../kernels/minimaxM3Sm107Tp4/foldTail.cu | 64 + .../kernels/minimaxM3Sm107Tp4/foldTail.h | 129 + .../minimaxM3Sm107Tp4/fullCtxReduce.cu | 268 + .../kernels/minimaxM3Sm107Tp4/fullCtxReduce.h | 77 + .../minimaxM3Sm107Tp4/mcAllreduceNormTp4.cu | 1264 +++ .../minimaxM3Sm107Tp4/mcAllreduceNormTp4.h | 83 + .../kernels/minimaxM3Sm107Tp4/producerPw4.cu | 340 + .../kernels/minimaxM3Sm107Tp4/producerPw4.h | 76 + .../minimaxM3Sm107Tp4/sampledAccept.cu | 944 ++ .../kernels/minimaxM3Sm107Tp4/sampledAccept.h | 87 + .../kernels/minimaxM3Sm107Tp4/selectLong2.cu | 312 + .../kernels/minimaxM3Sm107Tp4/selectLong2.h | 52 + .../kernels/minimaxM3Sm107Tp4/siluAndMul.cu | 89 + .../kernels/minimaxM3Sm107Tp4/siluAndMul.h | 40 + .../minimaxM3Sm107Tp4/sparseDecodeLc4.cu | 743 ++ .../minimaxM3Sm107Tp4/sparseDecodeLc4.h | 104 + .../kernels/minimaxM3Sm107Tp4/specOffsets.cu | 58 + .../kernels/minimaxM3Sm107Tp4/specOffsets.h | 41 + .../kernels/minimaxM3Sm107Tp4/stepPrologue.cu | 111 + .../kernels/minimaxM3Sm107Tp4/stepPrologue.h | 47 + .../kernels/minimaxM3Sm107Tp4/swigluOai.cu | 100 + .../kernels/minimaxM3Sm107Tp4/swigluOai.h | 43 + .../kernels/minimaxM3Sm107Tp8/draftStep.cu | 711 ++ .../kernels/minimaxM3Sm107Tp8/draftStep.h | 119 + .../minimaxM3Sm107Tp8/drafterRopeFp8Kv.cu | 189 + .../minimaxM3Sm107Tp8/drafterRopeFp8Kv.h | 79 + .../kernels/minimaxM3Sm107Tp8/foldTail.cu | 64 + .../kernels/minimaxM3Sm107Tp8/foldTail.h | 129 + .../minimaxM3Sm107Tp8/fullCtxReduce.cu | 268 + .../kernels/minimaxM3Sm107Tp8/fullCtxReduce.h | 77 + .../minimaxM3Sm107Tp8/mcAllreduceNormTp8.cu | 1096 ++ .../minimaxM3Sm107Tp8/mcAllreduceNormTp8.h | 83 + .../kernels/minimaxM3Sm107Tp8/producerPw8.cu | 340 + .../kernels/minimaxM3Sm107Tp8/producerPw8.h | 76 + .../minimaxM3Sm107Tp8/sampledAccept.cu | 946 ++ .../kernels/minimaxM3Sm107Tp8/sampledAccept.h | 87 + .../kernels/minimaxM3Sm107Tp8/siluAndMul.cu | 89 + .../kernels/minimaxM3Sm107Tp8/siluAndMul.h | 40 + .../minimaxM3Sm107Tp8/sparseDecodeLc8m.cu | 746 ++ .../minimaxM3Sm107Tp8/sparseDecodeLc8m.h | 105 + .../kernels/minimaxM3Sm107Tp8/specOffsets.cu | 58 + .../kernels/minimaxM3Sm107Tp8/specOffsets.h | 41 + .../kernels/minimaxM3Sm107Tp8/stepPrologue.cu | 111 + .../kernels/minimaxM3Sm107Tp8/stepPrologue.h | 47 + .../kernels/minimaxM3Sm107Tp8/swigluOai.cu | 100 + .../kernels/minimaxM3Sm107Tp8/swigluOai.h | 43 + .../kernels/moe/trtllmGen/runner.cu | 9 + .../kernels/moe/trtllmGen/runner.h | 3 + .../nanobind/batch_manager/bindings.cpp | 21 +- .../nanobind/executor/request.cpp | 14 + cpp/tensorrt_llm/thop/CMakeLists.txt | 56 +- .../thop/minimaxM3/minimaxM3RouterGemvOp.cpp | 23 +- .../minimaxM3Sm107Tp2DenseQkNormRopeFp8Op.cpp | 125 + .../minimaxM3Sm107Tp2DraftStepOp.cpp | 220 + .../minimaxM3Sm107Tp2DrafterRopeFp8KvOp.cpp | 103 + .../minimaxM3/minimaxM3Sm107Tp2FoldTailOp.cpp | 130 + .../minimaxM3Sm107Tp2FullCtxReduceOp.cpp | 74 + .../minimaxM3Sm107Tp2McAllreduceNormOp.cpp | 181 + .../minimaxM3Sm107Tp2ProducerPwOp.cpp | 185 + .../minimaxM3Sm107Tp2SampledAcceptOp.cpp | 110 + .../minimaxM3Sm107Tp2SelectLongOp.cpp | 85 + .../minimaxM3Sm107Tp2SiluAndMulOp.cpp | 66 + .../minimaxM3Sm107Tp2SparseDecodeLc2Op.cpp | 298 + .../minimaxM3Sm107Tp2SpecOffsetsOp.cpp | 74 + .../minimaxM3Sm107Tp2StepPrologueOp.cpp | 68 + .../minimaxM3Sm107Tp2SwigluOaiOp.cpp | 65 + .../minimaxM3Sm107Tp4DenseQkNormRopeFp8Op.cpp | 128 + .../minimaxM3Sm107Tp4DraftStepOp.cpp | 215 + .../minimaxM3Sm107Tp4DrafterRopeFp8KvOp.cpp | 103 + .../minimaxM3/minimaxM3Sm107Tp4FoldTailOp.cpp | 89 + .../minimaxM3Sm107Tp4FoldTailParams.h | 111 + .../minimaxM3Sm107Tp4FullCtxReduceOp.cpp | 92 + .../minimaxM3Sm107Tp4McAllreduceNormOp.cpp | 181 + .../minimaxM3Sm107Tp4ProducerPwOp.cpp | 185 + .../minimaxM3Sm107Tp4SampledAcceptOp.cpp | 113 + .../minimaxM3Sm107Tp4SelectLongOp.cpp | 87 + .../minimaxM3Sm107Tp4SiluAndMulOp.cpp | 66 + .../minimaxM3Sm107Tp4SparseDecodeLcOp.cpp | 191 + .../minimaxM3Sm107Tp4SpecOffsetsOp.cpp | 74 + .../minimaxM3Sm107Tp4StepPrologueOp.cpp | 68 + .../minimaxM3Sm107Tp4SwigluOaiOp.cpp | 65 + .../minimaxM3Sm107Tp8DraftStepOp.cpp | 236 + .../minimaxM3Sm107Tp8DrafterRopeFp8KvOp.cpp | 103 + .../minimaxM3/minimaxM3Sm107Tp8FoldTailOp.cpp | 89 + .../minimaxM3Sm107Tp8FoldTailParams.h | 111 + .../minimaxM3Sm107Tp8FullCtxReduceOp.cpp | 92 + .../minimaxM3Sm107Tp8McAllreduceNormOp.cpp | 181 + .../minimaxM3Sm107Tp8ProducerPwOp.cpp | 185 + .../minimaxM3Sm107Tp8SampledAcceptOp.cpp | 113 + .../minimaxM3Sm107Tp8SiluAndMulOp.cpp | 66 + .../minimaxM3Sm107Tp8SparseDecodeLc8Op.cpp | 189 + .../minimaxM3Sm107Tp8SpecOffsetsOp.cpp | 74 + .../minimaxM3Sm107Tp8StepPrologueOp.cpp | 68 + .../minimaxM3Sm107Tp8SwigluOaiOp.cpp | 65 + .../minimaxM31ChainSpecSamplingMcOp.cpp | 100 + .../minimaxM31/minimaxM31DsparkGlueOp.cpp | 133 + .../thop/minimaxM31/minimaxM31F3ArOp.cpp | 107 + .../minimaxM31/minimaxM31FcShardGemmOp.cpp | 172 + .../minimaxM31/minimaxM31MarkovEpilogueOp.cpp | 261 + .../minimaxM31/minimaxM31MoeFinalizeOp.cpp | 82 + .../thop/minimaxM31/minimaxM31S1bFoldOp.cpp | 81 + .../thop/minimaxM31/minimaxM31S2MoeTailOp.cpp | 87 + .../minimaxM31/minimaxM31SoftmaxPathBOp.cpp | 81 + .../minimaxM31/minimaxM31SwigluOaiVecOp.cpp | 62 + .../minimaxM31ToppRenormDetV3Op.cpp | 14 +- .../thop/moe/trtllmGen/fp4BlockScaleMoe.cpp | 8 + cpp/tensorrt_llm/thop/mxfp8Gemm.cpp | 8 + requirements.txt | 2 +- setup.py | 2 + .../_experimental/modeling_v2/catalog/_op.py | 1 + .../catalog/comm/allgather_cast.py | 49 +- .../modeling_v2/catalog/comm/tp2ctx_ipc.py | 347 + .../modeling_v2/catalog/fakes/m31_fake_ops.py | 191 +- .../catalog/spec_decode/dspark_ctx_advance.py | 178 +- .../spec_decode/dspark_forward_indices.py | 54 + .../spec_decode/dspark_markov_chain.py | 52 +- .../spec_decode/dspark_markov_chain_kernel.py | 88 +- .../dspark_markov_chain_kernel_b1.py | 88 +- .../spec_decode/dspark_qk_norm_rope.py | 11 +- .../catalog/spec_decode/strict_accept.py | 4 +- .../modeling_v2/models/minimax_m3/fake_ops.py | 84 + .../modeling_v2/models/minimax_m3/m31_pins.py | 322 + .../m31_tuning/sm107/tp2/rank_0.json | 1644 +++ .../m31_tuning/sm107/tp2/rank_1.json | 1644 +++ .../__init__.py | 3 + .../minimax_m31_nvfp4__sm_107__tp2/dspark.py | 1065 ++ .../dspark_s4.py | 765 ++ .../kernels/__init__.py | 3 + .../kernels/embedding_firstnorm/__init__.py | 3 + .../m3_embedding_firstnorm_kernel.py | 128 + .../kernels/embedding_firstnorm/op.py | 157 + .../kernels/lc2_mixed.py | 77 + .../kernels/long_select.py | 72 + .../kernels/m31_f3_gemm.py | 1910 ++++ .../kernels/m3_attn_lc.py | 182 + .../kernels/m3_draft_lowm.py | 143 + .../kernels/m3_draft_step.py | 222 + .../kernels/m3_drafter_attn.py | 472 + .../kernels/m3_fold_tail.py | 172 + .../kernels/m3_force_vec.py | 64 + .../kernels/m3_fullctx_decode.py | 313 + .../kernels/m3_index_score.py | 661 ++ .../kernels/m3_index_score_t2sc.py | 702 ++ .../kernels/m3_lowm_splitk.py | 1259 +++ .../kernels/m3_mc_allreduce.py | 422 + .../kernels/m3_moe/__init__.py | 16 + .../kernels/m3_moe/m3_moe16_kernel.py | 3871 +++++++ .../kernels/m3_moe/m3_moe_kernel.py | 3914 +++++++ .../kernels/m3_moe/op.py | 880 ++ .../kernels/m3_moe/op16.py | 304 + .../kernels/m3_moe_finalize.py | 95 + .../kernels/m3_nvfp4_index_decode.py | 121 + .../kernels/m3_prequant.py | 111 + .../kernels/m3_report.py | 108 + .../kernels/m3_sampled_accept.py | 174 + .../kernels/m3_select_long_dsl.py | 382 + .../kernels/m3_shard_accept.py | 176 + .../kernels/m3_spec_tail.py | 116 + .../kernels/m3_target_head.py | 113 + .../kernels/m3_tokmap_guard.py | 185 + .../kernels/msa_kv_update.py | 89 + .../kernels/mx_gemv/__init__.py | 3 + .../kernels/mx_gemv/m3_mx_gemv_kernel.py | 1474 +++ .../kernels/mx_gemv/op.py | 683 ++ .../kernels/mx_gemv/phase_producer.py | 508 + .../kernels/qkv_fusion/__init__.py | 2 + .../kernels/qkv_fusion/clc_tile_scheduler.py | 559 + .../kernels/qkv_fusion/kernel.py | 1996 ++++ .../kernels/triton_sparse_decode.py | 1829 ++++ .../kernels/triton_sparse_decode_mq.py | 237 + .../m31_switches.py | 61 + .../modeling.py | 9237 +++++++++++++++++ .../minimax_m31_nvfp4__sm_107__tp2/mxdraft.py | 116 + .../ops/__init__.py | 3 + .../ops/activation/__init__.py | 3 + .../ops/activation/m3_silu_and_mul.py | 83 + .../ops/activation/m3_swq.py | 170 + .../ops/attention/__init__.py | 3 + .../ops/attention/m3_dense_decode.py | 190 + .../attention/m3_dense_qk_norm_rope_fp8.py | 134 + .../ops/attention/m3_draft_attention.py | 111 + .../attention/m3_fp8_indexer_qk_norm_rope.py | 114 + .../ops/attention/m3_fullctx_decode.py | 142 + .../ops/attention/m3_index_score.py | 159 + .../ops/attention/m3_index_score_t2sc.py | 181 + .../ops/attention/m3_msa_attention.py | 249 + .../ops/attention/m3_producer.py | 216 + .../ops/attention/m3_qk_norm_rope_fp8.py | 137 + .../ops/attention/m3_qkv_insert.py | 134 + .../ops/attention/m3_select_blocks.py | 82 + .../ops/attention/m3_select_long.py | 186 + .../ops/attention/m3_select_long_dsl.py | 140 + .../ops/attention/m3_sparse_decode.py | 215 + .../ops/attention/m3_sparse_decode_lc.py | 607 ++ .../ops/attention/msa_kv_update.py | 200 + .../ops/comm/__init__.py | 3 + .../ops/comm/m3_allreduce.py | 61 + .../ops/comm/m3_mc_allreduce_norm.py | 329 + .../ops/comm/m3_mc_finalize_allreduce_norm.py | 197 + .../ops/embedding/__init__.py | 3 + .../ops/embedding/m3_embedding.py | 48 + .../ops/gemm/__init__.py | 3 + .../ops/gemm/m31_agg_prefill_cublaslt.py | 133 + .../ops/gemm/m3_bf16_linear.py | 41 + .../ops/gemm/m3_dense_swiglu.py | 72 + .../ops/gemm/m3_draft_lowm.py | 158 + .../ops/gemm/m3_fp32_linear.py | 46 + .../ops/gemm/m3_mx_gemv.py | 171 + .../ops/gemm/m3_mx_gemv_bf16.py | 111 + .../ops/gemm/m3_mxfp8_cublaslt.py | 127 + .../ops/gemm/m3_mxfp8_gemm.py | 58 + .../ops/gemm/m3_mxfp8_linear.py | 137 + .../ops/gemm/m3_mxfp8_pinned_gemm.py | 69 + .../ops/gemm/m3_mxfp8_pinned_quantize.py | 47 + .../ops/gemm/m3_target_head.py | 346 + .../ops/gqamix_sparse_decode.py | 1337 +++ .../ops/moe/__init__.py | 3 + .../ops/moe/m31_s1b_fold.py | 173 + .../ops/moe/m3_moe16_engine.py | 67 + .../ops/moe/m3_moe_engine.py | 328 + .../ops/moe/m3_moe_tactic_pin.py | 86 + .../ops/moe/m3_router_tc.py | 137 + .../ops/norm/__init__.py | 3 + .../ops/norm/m3_embedding_firstnorm.py | 87 + .../ops/sampling/__init__.py | 4 + .../ops/sampling/m31_chain_spec_mc.py | 381 + .../ops/sampling/m31_softmax_mc.py | 240 + .../ops/sampling/m31_softmax_pathb.py | 102 + .../ops/sampling/m31_topp_det.py | 303 + .../ops/spec/__init__.py | 4 + .../ops/spec/m31_markov_epilogue.py | 159 + .../ops/spec/m3_capture_norm.py | 80 + .../ops/spec/m3_chunk_norm.py | 70 + .../ops/spec/m3_draft_input.py | 140 + .../ops/spec/m3_draft_silu.py | 58 + .../ops/spec/m3_draft_step.py | 417 + .../ops/spec/m3_force_vec.py | 86 + .../ops/spec/m3_force_vec_req.py | 62 + .../ops/spec/m3_sampled_accept.py | 289 + .../ops/spec/m3_sampled_accept_ws_words.py | 59 + .../ops/spec/m3_shard_accept.py | 284 + .../ops/spec/m3_slot_scatter.py | 72 + .../ops/spec/m3_spec_offsets.py | 60 + .../ops/spec/m3_step_gather.py | 152 + .../ops/spec/m3_step_prologue.py | 76 + .../minimax_m31_nvfp4__sm_107__tp2/weights.py | 910 ++ .../__init__.py | 3 + .../minimax_m31_nvfp4__sm_107__tp4/dspark.py | 823 ++ .../dspark_s4.py | 547 + .../kernels/__init__.py | 3 + .../kernels/embedding_firstnorm/__init__.py | 3 + .../m3_embedding_firstnorm_kernel.py | 128 + .../kernels/embedding_firstnorm/op.py | 157 + .../kernels/long_select.py | 72 + .../kernels/m3_attn_lc.py | 163 + .../kernels/m3_draft_lowm.py | 143 + .../kernels/m3_draft_step.py | 222 + .../kernels/m3_drafter_attn.py | 405 + .../kernels/m3_fold_tail.py | 172 + .../kernels/m3_force_vec.py | 64 + .../kernels/m3_fullctx_decode.py | 313 + .../kernels/m3_index_score.py | 661 ++ .../kernels/m3_index_score_t2sc.py | 702 ++ .../kernels/m3_lowm_splitk.py | 1259 +++ .../kernels/m3_mc_allreduce.py | 422 + .../kernels/m3_moe/__init__.py | 16 + .../kernels/m3_moe/m3_moe16_kernel.py | 3900 +++++++ .../kernels/m3_moe/m3_moe_kernel.py | 3943 +++++++ .../kernels/m3_moe/op.py | 850 ++ .../kernels/m3_moe/op16.py | 327 + .../kernels/m3_prequant.py | 111 + .../kernels/m3_report.py | 34 + .../kernels/m3_sampled_accept.py | 174 + .../kernels/m3_select_long_dsl.py | 382 + .../kernels/m3_shard_accept.py | 176 + .../kernels/m3_spec_tail.py | 116 + .../kernels/m3_target_head.py | 113 + .../kernels/m3_tokmap_guard.py | 185 + .../kernels/msa_kv_update.py | 89 + .../kernels/mx_gemv/__init__.py | 3 + .../kernels/mx_gemv/m3_mx_gemv_kernel.py | 1299 +++ .../kernels/mx_gemv/op.py | 535 + .../kernels/triton_sparse_decode.py | 1732 ++++ .../kernels/triton_sparse_decode_mq.py | 237 + .../m31_switches.py | 61 + .../modeling.py | 6396 ++++++++++++ .../minimax_m31_nvfp4__sm_107__tp4/mxdraft.py | 112 + .../ops/__init__.py | 3 + .../ops/activation/__init__.py | 3 + .../ops/activation/m3_silu_and_mul.py | 83 + .../ops/attention/__init__.py | 3 + .../ops/attention/m3_dense_decode.py | 190 + .../attention/m3_dense_qk_norm_rope_fp8.py | 134 + .../ops/attention/m3_draft_attention.py | 111 + .../attention/m3_fp8_indexer_qk_norm_rope.py | 114 + .../ops/attention/m3_fullctx_decode.py | 142 + .../ops/attention/m3_index_score.py | 159 + .../ops/attention/m3_index_score_t2sc.py | 168 + .../ops/attention/m3_msa_attention.py | 224 + .../ops/attention/m3_producer.py | 223 + .../ops/attention/m3_qk_norm_rope_fp8.py | 137 + .../ops/attention/m3_qkv_insert.py | 134 + .../ops/attention/m3_select_blocks.py | 82 + .../ops/attention/m3_select_long.py | 186 + .../ops/attention/m3_select_long_dsl.py | 140 + .../ops/attention/m3_sparse_decode.py | 215 + .../ops/attention/m3_sparse_decode_lc.py | 436 + .../ops/attention/msa_kv_update.py | 200 + .../ops/comm/__init__.py | 3 + .../ops/comm/m3_allreduce.py | 61 + .../ops/comm/m3_mc_allreduce_norm.py | 329 + .../ops/comm/m3_mc_finalize_allreduce_norm.py | 197 + .../ops/embedding/__init__.py | 3 + .../ops/embedding/m3_embedding.py | 48 + .../ops/gemm/__init__.py | 3 + .../ops/gemm/m31_fc_shard.py | 141 + .../ops/gemm/m3_bf16_linear.py | 41 + .../ops/gemm/m3_dense_swiglu.py | 72 + .../ops/gemm/m3_draft_lowm.py | 158 + .../ops/gemm/m3_fp32_linear.py | 46 + .../ops/gemm/m3_mx_gemv.py | 166 + .../ops/gemm/m3_mx_gemv_bf16.py | 106 + .../ops/gemm/m3_mxfp8_gemm.py | 58 + .../ops/gemm/m3_mxfp8_linear.py | 137 + .../ops/gemm/m3_mxfp8_pinned_gemm.py | 69 + .../ops/gemm/m3_mxfp8_pinned_quantize.py | 47 + .../ops/gemm/m3_target_head.py | 346 + .../ops/gqamix_sparse_decode.py | 1337 +++ .../ops/moe/__init__.py | 3 + .../ops/moe/m3_moe16_engine.py | 67 + .../ops/moe/m3_moe_engine.py | 326 + .../ops/moe/m3_router_tc.py | 119 + .../ops/norm/__init__.py | 3 + .../ops/norm/m3_embedding_firstnorm.py | 87 + .../ops/sampling/__init__.py | 3 + .../ops/sampling/m31_topp_det.py | 271 + .../ops/spec/__init__.py | 4 + .../ops/spec/m3_capture_norm.py | 80 + .../ops/spec/m3_chunk_norm.py | 70 + .../ops/spec/m3_draft_input.py | 140 + .../ops/spec/m3_draft_silu.py | 58 + .../ops/spec/m3_draft_step.py | 417 + .../ops/spec/m3_force_vec.py | 86 + .../ops/spec/m3_force_vec_req.py | 62 + .../ops/spec/m3_sampled_accept.py | 289 + .../ops/spec/m3_sampled_accept_ws_words.py | 59 + .../ops/spec/m3_shard_accept.py | 284 + .../ops/spec/m3_slot_scatter.py | 72 + .../ops/spec/m3_spec_offsets.py | 60 + .../ops/spec/m3_step_gather.py | 152 + .../ops/spec/m3_step_prologue.py | 76 + .../minimax_m31_nvfp4__sm_107__tp4/weights.py | 910 ++ .../__init__.py | 3 + .../minimax_m31_nvfp4__sm_107__tp8/dspark.py | 984 ++ .../dspark_s4.py | 547 + .../kernels/__init__.py | 3 + .../kernels/embedding_firstnorm/__init__.py | 3 + .../m3_embedding_firstnorm_kernel.py | 128 + .../kernels/embedding_firstnorm/op.py | 157 + .../kernels/long_select.py | 70 + .../kernels/m3_attn_lc8.py | 164 + .../kernels/m3_draft_step.py | 355 + .../kernels/m3_drafter_attn.py | 442 + .../kernels/m3_fc_wait.py | 40 + .../kernels/m3_fold_tail.py | 180 + .../kernels/m3_force_vec.py | 63 + .../kernels/m3_fullctx_decode.py | 332 + .../kernels/m3_index_score.py | 712 ++ .../kernels/m3_layer/__init__.py | 14 + .../kernels/m3_layer/phase_producer.py | 489 + .../kernels/m3_locality.py | 448 + .../kernels/m3_lowm_splitk.py | 1602 +++ .../kernels/m3_mc_allreduce.py | 439 + .../kernels/m3_moe/__init__.py | 16 + .../kernels/m3_moe/m3_moe_kernel.py | 3971 +++++++ .../kernels/m3_moe/op.py | 828 ++ .../kernels/m3_prequant.py | 111 + .../kernels/m3_report.py | 34 + .../kernels/m3_sampled_accept.py | 230 + .../kernels/m3_select_long_dsl.py | 382 + .../kernels/m3_shard_accept.py | 176 + .../kernels/m3_spec_tail.py | 113 + .../kernels/m3_target_head.py | 113 + .../kernels/m3_tokmap_guard.py | 185 + .../kernels/msa_kv_update.py | 89 + .../kernels/mx_gemv/__init__.py | 4 + .../kernels/mx_gemv/m3_mx_gemv_kernel.py | 1511 +++ .../kernels/mx_gemv/op.py | 713 ++ .../kernels/triton_sparse_decode.py | 1754 ++++ .../kernels/triton_sparse_decode_mq.py | 237 + .../m31_switches.py | 61 + .../modeling.py | 6919 ++++++++++++ .../minimax_m31_nvfp4__sm_107__tp8/mxdraft.py | 203 + .../ops/__init__.py | 3 + .../ops/activation/__init__.py | 3 + .../ops/activation/m3_silu_and_mul.py | 83 + .../ops/activation/m3_swq.py | 162 + .../ops/attention/__init__.py | 3 + .../ops/attention/m3_dense_decode.py | 190 + .../ops/attention/m3_draft_attention.py | 131 + .../attention/m3_fp8_indexer_qk_norm_rope.py | 114 + .../ops/attention/m3_fullctx_decode.py | 142 + .../ops/attention/m3_index_score.py | 167 + .../ops/attention/m3_msa_attention.py | 249 + .../ops/attention/m3_producer.py | 223 + .../ops/attention/m3_qk_norm_rope_fp8.py | 137 + .../ops/attention/m3_qkv_insert.py | 134 + .../ops/attention/m3_select_blocks.py | 82 + .../ops/attention/m3_select_long.py | 193 + .../ops/attention/m3_select_long_dsl.py | 142 + .../ops/attention/m3_sparse_decode.py | 215 + .../ops/attention/m3_sparse_decode_lc8.py | 487 + .../ops/attention/msa_kv_update.py | 200 + .../ops/comm/__init__.py | 3 + .../ops/comm/m3_allreduce.py | 60 + .../ops/comm/m3_mc_allreduce_norm.py | 317 + .../ops/comm/m3_mc_finalize_allreduce_norm.py | 215 + .../ops/embedding/__init__.py | 3 + .../ops/embedding/m3_embedding.py | 48 + .../ops/gemm/__init__.py | 3 + .../ops/gemm/m31_fc_shard.py | 141 + .../ops/gemm/m3_bf16_linear.py | 41 + .../ops/gemm/m3_dense_swiglu.py | 72 + .../ops/gemm/m3_draft_lowm.py | 160 + .../ops/gemm/m3_fp32_linear.py | 46 + .../ops/gemm/m3_mx_gemv.py | 176 + .../ops/gemm/m3_mx_gemv_bf16.py | 100 + .../ops/gemm/m3_mxfp8_cublaslt.py | 127 + .../ops/gemm/m3_mxfp8_gemm.py | 60 + .../ops/gemm/m3_mxfp8_linear.py | 137 + .../ops/gemm/m3_target_head.py | 346 + .../ops/gqamix_sparse_decode.py | 1337 +++ .../ops/moe/__init__.py | 3 + .../ops/moe/m3_moe_engine.py | 343 + .../ops/moe/m3_router_tc.py | 151 + .../ops/norm/__init__.py | 3 + .../ops/norm/m3_embedding_firstnorm.py | 90 + .../ops/sampling/__init__.py | 4 + .../ops/sampling/m31_chain_spec_mc.py | 382 + .../ops/sampling/m31_softmax_mc.py | 186 + .../ops/sampling/m31_topp_det.py | 257 + .../ops/spec/__init__.py | 4 + .../ops/spec/m3_capture_norm.py | 80 + .../ops/spec/m3_chunk_norm.py | 74 + .../ops/spec/m3_draft_input.py | 140 + .../ops/spec/m3_draft_silu.py | 58 + .../ops/spec/m3_draft_step.py | 440 + .../ops/spec/m3_force_vec.py | 136 + .../ops/spec/m3_sampled_accept.py | 282 + .../ops/spec/m3_sampled_accept_ws_words.py | 59 + .../ops/spec/m3_shard_accept.py | 266 + .../ops/spec/m3_slot_scatter.py | 72 + .../ops/spec/m3_spec_offsets.py | 60 + .../ops/spec/m3_step_gather.py | 160 + .../ops/spec/m3_step_prologue.py | 76 + .../minimax_m31_nvfp4__sm_107__tp8/weights.py | 929 ++ .../modeling_v2/models/minimax_m3/routing.py | 43 +- .../backends/fmha/flashinfer_trtllm_gen.py | 3 + .../attention/backends/fmha/msa_decode.py | 30 +- .../attention/backends/fmha/msa_prefill.py | 561 +- .../attention/backends/prims_ts/__init__.py | 2 + .../prims_ts/_block_sparse/compiler.py | 2 + .../backends/prims_ts/_worktile_scope.py | 105 + .../attention/backends/prims_ts/context.py | 3 + .../attention/backends/prims_ts/decode.py | 110 +- .../kernels/fmha_context/fmha_resources.py | 23 +- .../fmha_decode/block_sparse_inspect.py | 3 + .../attention/backends/prims_ts/mla_decode.py | 2 + .../sparse/minimax_m3/cache_manager.py | 171 +- .../backends/sparse/minimax_m3/common.py | 31 +- .../minimax_m3/kernels/nvfp4_index_k.py | 215 + .../sparse/minimax_m3/kernels/paged_cache.py | 54 +- .../kernels/trtllm_gen_dense_decode.py | 3 +- .../backends/sparse/minimax_m3/msa_backend.py | 223 +- .../backends/sparse/minimax_m3/msa_indexer.py | 269 +- tensorrt_llm/_torch/autotuner.py | 75 +- .../_torch/custom_ops/cpp_custom_ops.py | 7 + .../_torch/custom_ops/cute_dsl_custom_ops.py | 20 +- .../custom_ops/mxfp8_cute_pinned_tactics.py | 160 + .../_torch/custom_ops/torch_custom_ops.py | 16 +- .../custom_ops/trtllm_gen_custom_ops.py | 7 + tensorrt_llm/_torch/cute_aot.py | 224 + .../k00eed757e39938fb731637e794868820.o | Bin 0 -> 103384 bytes .../k0218894bfe0a33f003526be955b1a5a2.o | Bin 0 -> 28240 bytes .../k027cc4c61c652bd6b1876fd15a2cbe78.o | Bin 0 -> 34024 bytes .../k02f9c681999d3610c65efb69f58ef67d.o | Bin 0 -> 48528 bytes .../k06187bd3c325e618ea22baa0688e80f9.o | Bin 0 -> 34072 bytes .../k075b4d53fecff042c8aacfdc99fd270e.o | Bin 0 -> 89896 bytes .../k084b52de5bc59b14164d9d5f14534621.o | Bin 0 -> 34072 bytes .../k0881662f562befcc1c32b65bd7a99070.o | Bin 0 -> 34072 bytes .../k09cc9c2f6949a2cd3d99c8317996d977.o | Bin 0 -> 63464 bytes .../k0a41d63ce5fcc4e48ef95d3f5f0df2d0.o | Bin 0 -> 34072 bytes .../k0c5fc19711de959b6135918a6874bc83.o | Bin 0 -> 317664 bytes .../k0f8ddce1d7ec250fbf885785c6685dcf.o | Bin 0 -> 74024 bytes .../k107e6a9efe22fee8544bed815eb03368.o | Bin 0 -> 34048 bytes .../k1306931fc2e1dae2a5a29cbf4450af9f.o | Bin 0 -> 61128 bytes .../k16be2b692ddbe79e186bc00a13703cbe.o | Bin 0 -> 34072 bytes .../k1a497f11e71eecd11561cfae8dbf9368.o | Bin 0 -> 73272 bytes .../k1b19f4137dccb0b8c55a535ad5d89d27.o | Bin 0 -> 49904 bytes .../k1c3f1a6c44770333636569ea847afd22.o | Bin 0 -> 34072 bytes .../k1c732009be9238cb2ce6a18efbd177f8.o | Bin 0 -> 61128 bytes .../k1dffa8015f154621562b41428d264bb1.o | Bin 0 -> 34072 bytes .../k1ea2c0d63cbc201cbc33ca38478ecb34.o | Bin 0 -> 28192 bytes .../k1fc6e741d499c4f215a1ac023c31ddf4.o | Bin 0 -> 34072 bytes .../k221e3d1cbd479c8444e94c698e544677.o | Bin 0 -> 354304 bytes .../k2531744bfa75437fb5135dfdd624fb5b.o | Bin 0 -> 34072 bytes .../k257864cebcfb331fdedf891003cb1c7a.o | Bin 0 -> 28192 bytes .../k29635f4ef9bf64e611db5121637fa3c3.o | Bin 0 -> 34072 bytes .../k29ad84cf333079e0a1d3b932c4ded45e.o | Bin 0 -> 34072 bytes .../k29d76b4da51071cbe677d0d83ec77d99.o | Bin 0 -> 34072 bytes .../k2a14e4b7e315b91abc4b37e7c8e5912a.o | Bin 0 -> 34072 bytes .../k2a3daf4105dbe50ecee4c53a13c1bb8d.o | Bin 0 -> 73272 bytes .../k2af3de0526072a130bf76c531284a306.o | Bin 0 -> 92968 bytes .../k2c3a02ad45c3d395ae376a6baf737fe8.o | Bin 0 -> 34072 bytes .../k2c5121f2f0610b0471c2b91d16d809e0.o | Bin 0 -> 51264 bytes .../k2d44f4edabc0f33b82391a9d8dc1a6ce.o | Bin 0 -> 88552 bytes .../k2e8bbd58d865a589130563dc4d1a2af0.o | Bin 0 -> 34072 bytes .../k2f391ea29f162b08a27e62beca7c3210.o | Bin 0 -> 34072 bytes .../k317bebfdef9ea155a409089e0bbd5030.o | Bin 0 -> 34072 bytes .../k3399a017f924826ae7af74c8721d0afb.o | Bin 0 -> 88552 bytes .../k342b3e9accc758690d57acd6c1ad4a3c.o | Bin 0 -> 74440 bytes .../k37caad1b21b00553597efe4623e78827.o | Bin 0 -> 28240 bytes .../k39138ce03fbb46891281e57dda50e68c.o | Bin 0 -> 34072 bytes .../k3a806c0edb8d0586bcf26ba141c1bee9.o | Bin 0 -> 34072 bytes .../k3bbc8b2a3a7b5c07d7939606525983dd.o | Bin 0 -> 67256 bytes .../k3bc9b617ff7ca80a27e4613a56b743e3.o | Bin 0 -> 28192 bytes .../k3c6ac10abb8ba633cc91b9123f27f780.o | Bin 0 -> 34072 bytes .../k3deb65d63c241dded8f9f6feee524cfd.o | Bin 0 -> 51616 bytes .../k3e010db787a2ead6b11796e755278465.o | Bin 0 -> 81032 bytes .../k3e29dac67c8e14db8d4c27d6f06ff15c.o | Bin 0 -> 294496 bytes .../k3f1dc08f97f6bc42e42599fb66daa0d8.o | Bin 0 -> 34072 bytes .../k3f64fb35e6c6d45c9c387172cd70e1b5.o | Bin 0 -> 92232 bytes .../k40f506331a47b27d6c476c5df332cfd3.o | Bin 0 -> 57064 bytes .../k410baccb2ba161cb240e3f170f960314.o | Bin 0 -> 75688 bytes .../k41f577a6984ff8bee476d34a62caed8c.o | Bin 0 -> 75688 bytes .../k42514d5f2dc215403c9c11ec79f6625e.o | Bin 0 -> 34072 bytes .../k42677b72c271123e0fde15397661c8d6.o | Bin 0 -> 49904 bytes .../k457ffd832f18f0d27324061d1eb1ce5a.o | Bin 0 -> 28192 bytes .../k459e034c1eff4cabc5badd7bd4dc16ae.o | Bin 0 -> 34072 bytes .../k46dfb5c68e57fa3a4a80950d45a1aa26.o | Bin 0 -> 34072 bytes .../k49b951f8bc5ea8cfe38dd52cea8745d7.o | Bin 0 -> 97944 bytes .../k4ee158f9d78e2f4af22ca8cdf25035bf.o | Bin 0 -> 128520 bytes .../k50781d01e99590a8c8543b21a6721d65.o | Bin 0 -> 34072 bytes .../k52af4c03f6b8e5990e9d227e1421165e.o | Bin 0 -> 34072 bytes .../k536e504c92bdbc994c692ca2c2cca754.o | Bin 0 -> 49288 bytes .../k53a1053964f40eebd57ef8419af84352.o | Bin 0 -> 34072 bytes .../k53dfccd97af20f0564192076fba638e6.o | Bin 0 -> 34072 bytes .../k544d4b9f3ea0ea08886c5dc10e08c42b.o | Bin 0 -> 34072 bytes .../k55ed6be359ce74e2cb00ec59dbd6ad89.o | Bin 0 -> 50000 bytes .../k5768d8426ce95a796c57d497ca754f7d.o | Bin 0 -> 28240 bytes .../k5c30d6e150a9ec66ae0a094c7fd60913.o | Bin 0 -> 34072 bytes .../k5d54e1f60cd25f1f4f8d7e0a394cebf2.o | Bin 0 -> 34072 bytes .../k5f07b3765de26e236ae17b8ff0eac7f0.o | Bin 0 -> 65848 bytes .../k602da62aafbfde53540fa18526658aa0.o | Bin 0 -> 34072 bytes .../k609f04ac135d191eeca3dc8cf40657cf.o | Bin 0 -> 75688 bytes .../k60b43c6114d06189e061601bda5423fd.o | Bin 0 -> 69944 bytes .../k60b5079290ad2088526acd151c5b9a3f.o | Bin 0 -> 34072 bytes .../k60c66d6ef2e5e14f52634e40438b9d85.o | Bin 0 -> 28240 bytes .../k624b4bd74939b58bb85e6b0a212ac48d.o | Bin 0 -> 96584 bytes .../k62d97aff61c66ea719a2978c9072de7e.o | Bin 0 -> 34072 bytes .../k64480700cb22a21c82f517aa00c1365a.o | Bin 0 -> 86568 bytes .../k64849d4081bef6f3ea56a9f09a4c2417.o | Bin 0 -> 34072 bytes .../k65494c2c19a9814c6b2344a80dc7d835.o | Bin 0 -> 34072 bytes .../k67c2ec2335be3981cab8732df48e2960.o | Bin 0 -> 336336 bytes .../k6a230f0482598b0b12a92ec687a861b8.o | Bin 0 -> 34072 bytes .../k6b155ec20fbadbfdafba8d376dabe43b.o | Bin 0 -> 188152 bytes .../k6c8122079f737fb229dab3f7c9f1e88b.o | Bin 0 -> 64032 bytes .../k6d69f27693eca302313fb758e28da794.o | Bin 0 -> 96584 bytes .../k6fe8c0040c9ee3e70a0678e04af81999.o | Bin 0 -> 34072 bytes .../k71abcbc4b33e413259fa590cad8fae89.o | Bin 0 -> 34072 bytes .../k71fddb6eb415bdae83414da29a477c1d.o | Bin 0 -> 128504 bytes .../k729d5c5f515c2989c6f4645046a8fc78.o | Bin 0 -> 34072 bytes .../k730d971c25c04dea469c917abe6da00f.o | Bin 0 -> 28240 bytes .../k73ea43a43a3c248aeb51c39dcc32757c.o | Bin 0 -> 65848 bytes .../k76837390ba7090d03755f38f335984c6.o | Bin 0 -> 76584 bytes .../k77540beeb21572cf40c1465139a8e3e8.o | Bin 0 -> 187944 bytes .../k7895e519c8fd50399a10ee9c9b4722bb.o | Bin 0 -> 34072 bytes .../k7a842663c2f6cfc297b0bd19e12cefee.o | Bin 0 -> 168648 bytes .../k7ac2accb0dfe6f9715bd7db142efee09.o | Bin 0 -> 63448 bytes .../k7ac78b16d6aa49ee5ba408abe9f02706.o | Bin 0 -> 34072 bytes .../k7e11bd01538e0e8fd354e6564ee71ac0.o | Bin 0 -> 92232 bytes .../k81128243050e8617afaf6db477a16040.o | Bin 0 -> 34072 bytes .../k8686d739fe1cf565bc7e78a1ab51c945.o | Bin 0 -> 71288 bytes .../k86d244e9f495308f819e50e86f604433.o | Bin 0 -> 34072 bytes .../k877d7aa659e5e1d816e0442761e33bde.o | Bin 0 -> 34072 bytes .../k87a26cbd91074d4c090e61c7442a511b.o | Bin 0 -> 34072 bytes .../k88642f4ca698fd1087f6caa67efb57e8.o | Bin 0 -> 64384 bytes .../k88fa6b951eb20c313aa59b4adbc69fe1.o | Bin 0 -> 315664 bytes .../k89fbca15f9847079d4dfb2ed49d66e4f.o | Bin 0 -> 34072 bytes .../k8b914d547c30519c09e45ca3937f65b2.o | Bin 0 -> 34072 bytes .../k8be43f1e0571e1f4d5fa83daf0d1b144.o | Bin 0 -> 34072 bytes .../k8bfaf12a78e278972fe4cfa229c5ff5e.o | Bin 0 -> 64488 bytes .../k8d2a1751722203553aa1a28872dfbb04.o | Bin 0 -> 34072 bytes .../k8d38d65c9e1a4e267b294dbf112f052f.o | Bin 0 -> 34072 bytes .../k8e653ebb28adfde7815602f8b4d83f31.o | Bin 0 -> 51616 bytes .../k8f7c307820544eeca13d80f89acf695b.o | Bin 0 -> 34072 bytes .../k91103515cccf539429183d92cfb39807.o | Bin 0 -> 34072 bytes .../k91346849ef2de43f93d0e96a65664eaf.o | Bin 0 -> 34072 bytes .../k91d99bae224a9801fe14037f80954e7c.o | Bin 0 -> 34072 bytes .../k9431f967afdb58beb949df76f58cee5b.o | Bin 0 -> 167992 bytes .../k95bff0816ba2d5a0a7594e89066c750a.o | Bin 0 -> 52104 bytes .../k967c39595b6f81d02c06645d911c9961.o | Bin 0 -> 34072 bytes .../k98add47b626026c2503673796d8185e5.o | Bin 0 -> 34072 bytes .../k99c15121f2eb0d1777f2be633e7e5072.o | Bin 0 -> 64016 bytes .../k9a6d1706c5fcdb285a9042f823e51edb.o | Bin 0 -> 34072 bytes .../ka267b2dcc02441456cd57472dbc874da.o | Bin 0 -> 34072 bytes .../ka4a690575a315b7d9738c20e7f242a36.o | Bin 0 -> 49512 bytes .../ka7d5145bbf7eda9f6df50f622e00dff3.o | Bin 0 -> 65848 bytes .../ka910e7175f9a30bf6e713cb86932fc46.o | Bin 0 -> 34072 bytes .../ka9d2996a6c1881bcab65e7825da4f469.o | Bin 0 -> 34072 bytes .../kab00779b3ea6268c3fa8c5384347c979.o | Bin 0 -> 34072 bytes .../kac95216b414b7faede973efab3c4be64.o | Bin 0 -> 34072 bytes .../kb21b47d8c1c250b3a20cf80112ab527c.o | Bin 0 -> 34072 bytes .../kb2dd3ea5f861e452f4ecbb580b3fca92.o | Bin 0 -> 28192 bytes .../kb4be9a246f5fb4af8d3eed89aa446cf4.o | Bin 0 -> 63464 bytes .../kb624af19573b968e6cd35f81411b3a44.o | Bin 0 -> 363120 bytes .../kb666059f738b7cfb01e45a484e3a0dd2.o | Bin 0 -> 34072 bytes .../kb689e14232999f4c2151d328fb2e0334.o | Bin 0 -> 34072 bytes .../kb74d7bb748b941e9fe6bc1f8fdd93198.o | Bin 0 -> 70024 bytes .../kb7a6af396879822c1534ec7d3163f8df.o | Bin 0 -> 34072 bytes .../kb95baa3070802f80dcd903c53a3964f8.o | Bin 0 -> 28192 bytes .../kbb78dda30e4032356ec7be602bc3b424.o | Bin 0 -> 67256 bytes .../kbe6ee4564892ecd24074538ed0d17f2d.o | Bin 0 -> 34072 bytes .../kbfb313640115702d8139291c633cb61d.o | Bin 0 -> 34072 bytes .../kc1775ee7a18143c76ca13d2b494c4727.o | Bin 0 -> 34072 bytes .../kc1c162425e681df83e1b62b93d0679bf.o | Bin 0 -> 34072 bytes .../kc1c7f67dc4e7c17390bf4c8ace76f391.o | Bin 0 -> 34072 bytes .../kc3053a2621af680d56e409cc084a8db7.o | Bin 0 -> 55624 bytes .../kc3cc186b225f1f23b6d1efb44ec7dbd2.o | Bin 0 -> 52768 bytes .../kc415ff8fa08210a6a77393bf85ddf0be.o | Bin 0 -> 34072 bytes .../kc4b328c6e8f71c2b2fd152f8077055f8.o | Bin 0 -> 34072 bytes .../kc5932f15ac9d608adbc379cb72c7640c.o | Bin 0 -> 34072 bytes .../kc911260e8a4241f05592f229dc40512f.o | Bin 0 -> 34072 bytes .../kc9754f51e923c13afe1180ef003e4fa4.o | Bin 0 -> 34072 bytes .../kce1906dfc7ccb380301b314df59ba00b.o | Bin 0 -> 34072 bytes .../kceb214f8b302ec91a2b88cd52602bb69.o | Bin 0 -> 97944 bytes .../kcf0367abcf989ba1c42a0731b61881c8.o | Bin 0 -> 349872 bytes .../kcf09707be55af5195f4589640abe27f2.o | Bin 0 -> 28240 bytes .../kd008badb8f4e2cb34ba95f2b40b06014.o | Bin 0 -> 56472 bytes .../kd3364a16830f1a3bead73b2be4199b92.o | Bin 0 -> 34072 bytes .../kd3bd7786587938e328844c978cd6b2e3.o | Bin 0 -> 34072 bytes .../kd4187341a82c234fffdcd7d661616fb9.o | Bin 0 -> 34072 bytes .../kd579851969c62870ca4274ebc92d651a.o | Bin 0 -> 34072 bytes .../kd80050a13b9de78420fb0bfa3e80921f.o | Bin 0 -> 34072 bytes .../kd8021408bba1af62bfc43668e1cea7d1.o | Bin 0 -> 34072 bytes .../kda850ab5991c75d665f25508ff6d34b9.o | Bin 0 -> 28240 bytes .../kdbd4019aa02bf65c25564ca198a77085.o | Bin 0 -> 34072 bytes .../kde44e6074303c68d4b290ad3929fb761.o | Bin 0 -> 34072 bytes .../ke0c7c73e4b05f100d7e599c08027ab7a.o | Bin 0 -> 47032 bytes .../ke2a030185a3f56dc9281490537f5c6ab.o | Bin 0 -> 28192 bytes .../ke30c27a09f840d696840df876a6e6ea9.o | Bin 0 -> 34072 bytes .../ke48338ff5e721e5e5119deb5226df72d.o | Bin 0 -> 34072 bytes .../ke4fc62a58d6e4f7e4ae0a2087df8155a.o | Bin 0 -> 34072 bytes .../ke557e36f3e1fad7aebc9847d4b6b62bb.o | Bin 0 -> 59256 bytes .../ke6b56a141d6efee652214e6d50494ac2.o | Bin 0 -> 34072 bytes .../ke8996f833d1f91e81cf2644d8fc15fb7.o | Bin 0 -> 51032 bytes .../ke9a6508eb80d23a7f3ed6591f72e643b.o | Bin 0 -> 34072 bytes .../kea2fa28c84c229f5b6224bcf2285a787.o | Bin 0 -> 34072 bytes .../kebcef6942084641815e5579414860d93.o | Bin 0 -> 78888 bytes .../kec7fea04a276529bce9626a6fdcab07a.o | Bin 0 -> 89896 bytes .../kec9f140f67d7409282564bd81783237c.o | Bin 0 -> 64560 bytes .../kee904649a81dda6daccd4f9b1b52a773.o | Bin 0 -> 34072 bytes .../keebaa803d23596dc67f8cc9d626ca23e.o | Bin 0 -> 34072 bytes .../keef8a85fe494ecd5dc420eb65fb1c61a.o | Bin 0 -> 28192 bytes .../kf0ecb78ca25035a27ab95b3b3d67f2b4.o | Bin 0 -> 34072 bytes .../kf16023d317c918040f4b5c0e25e1405d.o | Bin 0 -> 317952 bytes .../kf3232ddee2a0b12f6c32292a49d23f44.o | Bin 0 -> 34072 bytes .../kf3c2d7335a14ad90793841232ed9c10f.o | Bin 0 -> 34072 bytes .../kf3c4db3cd960b261d0847ebaf22df84c.o | Bin 0 -> 34072 bytes .../kf7b5a92e434ffcac257a03692b343b68.o | Bin 0 -> 34072 bytes .../kf868b615acd9df2fff40fce7e73a7e6e.o | Bin 0 -> 52472 bytes .../kf935e0bf1ab3deb88c6e9fd2a48be7e8.o | Bin 0 -> 28240 bytes .../kfafff4c85c826306b570c6022b95eb40.o | Bin 0 -> 34072 bytes .../kfeb0ee198c6718291355fc1db62e121e.o | Bin 0 -> 34072 bytes .../kfef3ffb9125b7358714c2ccf6aedd14e.o | Bin 0 -> 34072 bytes .../_torch/disaggregation/base/backend.py | 7 + .../_torch/disaggregation/draft_kv.py | 347 + .../disaggregation/kv_cache_transceiver.py | 29 +- .../_torch/disaggregation/native/auxiliary.py | 76 +- .../native/mixers/attention/peer.py | 19 +- .../_torch/disaggregation/native/rank_info.py | 94 +- .../_torch/disaggregation/native/transfer.py | 720 +- .../orchestration/coordinator.py | 46 + .../_torch/disaggregation/resource/utils.py | 40 + .../_torch/disaggregation/transceiver.py | 464 +- .../_torch/flashinfer_autotune_cache.py | 78 + tensorrt_llm/_torch/late_compile.py | 90 + tensorrt_llm/_torch/models/modeling_dflash.py | 119 +- tensorrt_llm/_torch/modules/linear.py | 2 +- tensorrt_llm/_torch/pyexecutor/_util.py | 69 +- .../engine/runners/decoder/runner.py | 106 +- tensorrt_llm/_torch/pyexecutor/h2d_cache.py | 2 + .../kv_cache/kv_cache_manager_v2.py | 343 +- tensorrt_llm/_torch/pyexecutor/llm_request.py | 42 +- tensorrt_llm/_torch/pyexecutor/profiling.py | 31 +- tensorrt_llm/_torch/pyexecutor/py_executor.py | 160 +- .../_torch/pyexecutor/runtime_hooks.py | 114 +- .../pyexecutor/scheduler/scheduler_v2.py | 5 + tensorrt_llm/_torch/pyexecutor/step_inputs.py | 75 +- tensorrt_llm/_torch/speculative/dflash.py | 336 +- tensorrt_llm/_torch/speculative/dspark.py | 4 + tensorrt_llm/_torch/speculative/interface.py | 287 +- .../_torch/speculative/spec_sampler_base.py | 22 +- tensorrt_llm/disaggregated_params.py | 4 + tensorrt_llm/executor/result.py | 2 + tensorrt_llm/llmapi/llm_args.py | 47 +- tensorrt_llm/serve/openai_protocol.py | 7 + .../usage/llm_args_golden_manifest.json | 3 +- .../integration/test_lists/test-db/l0_cpu.yml | 7 +- .../test_lists/test-db/l0_dgx_b200.yml | 3 + .../test_disagg_gen_init_draft_window.py | 361 + .../executor/kv_cache/test_kv_gpu_prefetch.py | 167 + .../executor/test_request_token_array.py | 147 + .../_torch/executor/test_runtime_hooks.py | 1 + .../test_modeling_v2_m31_tp2_swq.py | 151 + .../test_modeling_v2_m3_silu_and_mul.py | 49 + .../attention/_m3_attn_runtime_tp2.py | 275 + .../attention/_m3_nvfp4_cache.py | 108 + .../attention/test_modeling_v2_m31_gqamix.py | 204 + .../test_modeling_v2_m31_lc_tp2_default.py | 44 + ...t_modeling_v2_m31_tp2wide_intree_scorer.py | 254 + ...modeling_v2_m3_fp8_indexer_qk_norm_rope.py | 63 + .../test_modeling_v2_m3_fp8_saturate.py | 102 + .../test_modeling_v2_m3_index_score.py | 163 + .../test_modeling_v2_m3_index_score_t2sc.py | 136 + .../test_modeling_v2_m3_msa_attention.py | 232 + .../test_modeling_v2_m3_n_valid_contract.py | 138 + .../attention/test_modeling_v2_m3_producer.py | 229 + .../test_modeling_v2_m3_qk_norm_rope_fp8.py | 95 + .../test_modeling_v2_m3_qkv_insert.py | 80 + .../test_modeling_v2_m3_select_blocks.py | 45 + .../test_modeling_v2_m3_select_long.py | 230 + .../test_modeling_v2_m3_select_long_dsl.py | 228 + .../test_modeling_v2_m3_sparse_decode.py | 359 + .../test_modeling_v2_m3_sparse_decode_lc.py | 875 ++ ...t_modeling_v2_m3_sparse_decode_lc_split.py | 139 + .../test_modeling_v2_m3_tokmap_guard.py | 226 + .../test_modeling_v2_msa_kv_update.py | 133 + .../comm/_m3_allreduce_op_matrix.py | 440 + .../comm/_m3_mc_allreduce_norm_op_matrix.py | 755 ++ .../comm/_rank_job.py | 87 + .../comm/test_modeling_v2_m3_allreduce.py | 39 + ...eling_v2_m3_mc_allreduce_norm_op_matrix.py | 30 + ...deling_v2_m3_mc_finalize_allreduce_norm.py | 48 + .../test_modeling_v2_m3_embedding.py | 37 + .../gemm/_m3_mxfp8_operands.py | 30 + .../gemm/test_modeling_v2_m31_tp2_gemvcfg.py | 108 + ...t_modeling_v2_m31_tp2_qkv_producer_fold.py | 285 + .../test_modeling_v2_m31_unfused_index.py | 117 + .../gemm/test_modeling_v2_m3_bf16_linear.py | 31 + .../gemm/test_modeling_v2_m3_dense_swiglu.py | 143 + .../gemm/test_modeling_v2_m3_fp32_linear.py | 32 + .../gemm/test_modeling_v2_m3_mx_gemv.py | 266 + .../gemm/test_modeling_v2_m3_mx_gemv_bf16.py | 134 + .../gemm/test_modeling_v2_m3_mx_gemv_order.py | 92 + .../gemm/test_modeling_v2_m3_mxfp8_gemm.py | 38 + .../gemm/test_modeling_v2_m3_mxfp8_linear.py | 46 + .../test_modeling_v2_m3_mxfp8_pinned_gemm.py | 90 + ...st_modeling_v2_m3_mxfp8_pinned_quantize.py | 48 + .../moe/_m3_moe_operands.py | 171 + .../moe/_m3_router.py | 81 + .../moe/test_modeling_v2_m31_moe16_tp2_off.py | 78 + .../test_modeling_v2_m31_tp2w_moe_finalize.py | 73 + ..._modeling_v2_m31_tp2w_moe_finalize_rows.py | 199 + ...est_modeling_v2_m31_tp2w_moe_tactic_pin.py | 111 + ...modeling_v2_m31_tp2w_moe_tactic_pin_gpu.py | 135 + .../moe/test_modeling_v2_m31_x1.py | 213 + .../moe/test_modeling_v2_m3_moe_engine.py | 180 + .../test_modeling_v2_m3_moe_flat_gate_tp2.py | 74 + .../moe/test_modeling_v2_m3_router_tc.py | 87 + ...t_modeling_v2_m31_native_dispatch_lines.py | 121 + .../test_modeling_v2_m31_native_finalize.py | 94 + .../test_modeling_v2_m31_native_swiglu.py | 99 + .../norm/test_modeling_v2_m31_router_f32.py | 94 + ...test_modeling_v2_m3_embedding_firstnorm.py | 88 + .../test_modeling_v2_m31_chain_spec_mc.py | 292 + .../test_modeling_v2_m31_rej_slots_default.py | 25 + .../test_modeling_v2_m31_rej_slots_wide.py | 42 + .../test_modeling_v2_m31_softmax_mc.py | 344 + .../test_modeling_v2_m31_softmax_pathb.py | 144 + .../sampling/test_modeling_v2_m31_topp_det.py | 273 + .../test_modeling_v2_m31_topp_det_wide.py | 236 + .../spec/_m3_spec_inputs.py | 106 + .../spec/test_modeling_v2_m31_dsfold.py | 168 + .../spec/test_modeling_v2_m31_dspark_n3a.py | 571 + .../spec/test_modeling_v2_m31_dspark_s4.py | 460 + .../spec/test_modeling_v2_m31_dspark_w0.py | 171 + .../spec/test_modeling_v2_m31_s4fe.py | 184 + .../spec/test_modeling_v2_m31_tail16_g2.py | 97 + .../spec/test_modeling_v2_m31_tail16_g3.py | 167 + .../spec/test_modeling_v2_m31_tail_bk3.py | 123 + .../spec/test_modeling_v2_m3_spec_offsets.py | 138 + .../spec/test_modeling_v2_m3_step_gather.py | 85 + .../spec/test_modeling_v2_m3_step_prologue.py | 192 + .../test_modeling_v2_m31_agg_qkv_a1.py | 77 + .../test_modeling_v2_m31_ar_det.py | 49 + .../test_modeling_v2_m31_blockoff_guard.py | 165 + .../test_modeling_v2_m31_draft_a1.py | 60 + .../test_modeling_v2_m31_dspark_draft_ar.py | 148 + ...test_modeling_v2_m31_hook_work_counters.py | 383 + ...est_modeling_v2_m31_host_tail_sm107_tp2.py | 274 + .../test_modeling_v2_m31_mxdraft_tp2_scope.py | 50 + .../test_modeling_v2_m31_onediv_cap.py | 55 + .../test_modeling_v2_m31_prefill_ar_mnnvl.py | 64 + .../test_modeling_v2_m31_proxy_prebuild.py | 59 + .../test_modeling_v2_m31_r6_dense_prequant.py | 35 + .../test_modeling_v2_m31_s10_scope.py | 16 + .../test_modeling_v2_m31_s2_tail.py | 137 + .../test_modeling_v2_m31_s4_logits_default.py | 68 + .../test_modeling_v2_m31_s5_scope.py | 16 + ...odeling_v2_m31_slot_ids_changed_default.py | 86 + ...test_modeling_v2_m31_switches_sm107_tp2.py | 83 + .../test_modeling_v2_m31_tp2_dspark_scope.py | 170 + ...ing_v2_m31_tp2_fused_projection_default.py | 37 + .../test_modeling_v2_m31_tp2_prefill_brk.py | 172 + .../test_modeling_v2_m31_tp2_r1vr8.py | 80 + .../test_modeling_v2_m31_tp2_rulings.py | 160 + ...test_modeling_v2_m31_tp2_weights_layout.py | 90 + .../test_modeling_v2_m31_tp2_work_counters.py | 182 + .../test_modeling_v2_m31_tp2agg_sp.py | 197 + .../test_modeling_v2_m31_tp2wide_decode.py | 90 + .../test_modeling_v2_m31_tp2wide_gemm.py | 90 + ...st_modeling_v2_m31_tp2wide_prefill_attn.py | 266 + .../test_modeling_v2_m31_vr_bands.py | 111 + ...st_modeling_v2_m31_vr_draft_fmha_sm100f.py | 99 + .../test_modeling_v2_m31_vr_tp2_mcag.py | 155 + .../test_modeling_v2_m3_capture_boundary.py | 283 + .../test_modeling_v2_m3_dispatch.py | 412 + .../test_modeling_v2_m3_prebuild.py | 372 + .../test_modeling_v2_m3_rank_agreement.py | 195 + .../test_modeling_v2_m3_silu_and_mul.py | 49 + .../attention/_m3_attn_runtime_tp4.py | 274 + .../attention/_m3_nvfp4_cache.py | 104 + .../attention/test_modeling_v2_m31_gqamix.py | 189 + .../test_modeling_v2_m3_dense_decode.py | 153 + ...t_modeling_v2_m3_dense_qk_norm_rope_fp8.py | 180 + .../test_modeling_v2_m3_draft_attention.py | 138 + .../test_modeling_v2_m3_drafter_fullctx.py | 289 + ...test_modeling_v2_m3_drafter_rope_fp8_kv.py | 371 + ...modeling_v2_m3_fp8_indexer_qk_norm_rope.py | 49 + .../test_modeling_v2_m3_fp8_saturate.py | 107 + .../test_modeling_v2_m3_fullctx_decode.py | 312 + .../test_modeling_v2_m3_index_score.py | 133 + .../test_modeling_v2_m3_index_score_t2sc.py | 112 + .../test_modeling_v2_m3_msa_attention.py | 214 + .../test_modeling_v2_m3_n_valid_contract.py | 138 + .../attention/test_modeling_v2_m3_producer.py | 209 + .../test_modeling_v2_m3_qk_norm_rope_fp8.py | 81 + .../test_modeling_v2_m3_qkv_insert.py | 80 + .../test_modeling_v2_m3_select_blocks.py | 45 + .../test_modeling_v2_m3_select_long.py | 183 + .../test_modeling_v2_m3_select_long_dsl.py | 191 + .../test_modeling_v2_m3_sparse_decode.py | 446 + .../test_modeling_v2_m3_sparse_decode_lc.py | 526 + .../test_modeling_v2_m3_tokmap_guard.py | 226 + .../test_modeling_v2_msa_kv_update.py | 133 + .../comm/_m3_allreduce_op_matrix.py | 440 + .../comm/_m3_capture_in_boundary_op_matrix.py | 307 + .../comm/_m3_draft_ar_norm_op_matrix.py | 533 + .../comm/_m3_mc_allreduce_norm_op_matrix.py | 665 ++ .../comm/_rank_job.py | 87 + .../comm/test_modeling_v2_m3_allreduce.py | 39 + ...test_modeling_v2_m3_capture_in_boundary.py | 22 + .../comm/test_modeling_v2_m3_draft_ar_norm.py | 32 + ...eling_v2_m3_mc_allreduce_norm_op_matrix.py | 20 + ...deling_v2_m3_mc_finalize_allreduce_norm.py | 48 + .../test_modeling_v2_m3_embedding.py | 37 + .../gemm/_m3_mxfp8_operands.py | 30 + .../test_modeling_v2_m31_unfused_index.py | 112 + .../gemm/test_modeling_v2_m3_bf16_linear.py | 31 + .../gemm/test_modeling_v2_m3_dense_swiglu.py | 142 + .../gemm/test_modeling_v2_m3_draft_lowm.py | 182 + .../gemm/test_modeling_v2_m3_fp32_linear.py | 32 + .../gemm/test_modeling_v2_m3_mx_gemv.py | 176 + .../gemm/test_modeling_v2_m3_mx_gemv_bf16.py | 68 + .../gemm/test_modeling_v2_m3_mx_gemv_order.py | 78 + .../gemm/test_modeling_v2_m3_mxfp8_gemm.py | 38 + .../gemm/test_modeling_v2_m3_mxfp8_linear.py | 46 + .../test_modeling_v2_m3_mxfp8_pinned_gemm.py | 82 + ...st_modeling_v2_m3_mxfp8_pinned_quantize.py | 48 + .../gemm/test_modeling_v2_m3_target_head.py | 253 + .../moe/_m3_moe_operands.py | 170 + .../moe/_m3_router.py | 39 + ...est_modeling_v2_m31_vr_tp4_moe_out_skip.py | 112 + ...modeling_v2_m31_vr_tp4_moe_tail_trigger.py | 101 + .../moe/test_modeling_v2_m3_moe16_engine.py | 63 + .../moe/test_modeling_v2_m3_moe_engine.py | 139 + .../test_modeling_v2_m3_moe_flat_gate_tp4.py | 74 + .../moe/test_modeling_v2_m3_router_tc.py | 60 + .../norm/test_modeling_v2_m31_ctxnormq.py | 101 + ...test_modeling_v2_m3_embedding_firstnorm.py | 76 + .../sampling/test_modeling_v2_m31_topp_det.py | 271 + .../spec/_m3_spec_inputs.py | 106 + .../spec/test_modeling_v2_m31_dspark_n3a.py | 473 + .../spec/test_modeling_v2_m31_dspark_s4.py | 347 + .../spec/test_modeling_v2_m31_dspark_w0.py | 171 + ...st_modeling_v2_m31_vr_draft_append_fuse.py | 46 + .../spec/test_modeling_v2_m3_capture_norm.py | 62 + .../test_modeling_v2_m3_capture_normed.py | 70 + .../spec/test_modeling_v2_m3_chunk_norm.py | 55 + .../spec/test_modeling_v2_m3_draft_input.py | 103 + .../test_modeling_v2_m3_draft_push_take.py | 42 + .../spec/test_modeling_v2_m3_draft_silu.py | 98 + .../spec/test_modeling_v2_m3_draft_step.py | 694 ++ ...t_modeling_v2_m3_draft_step_fixed_order.py | 148 + .../test_modeling_v2_m3_exact_numerics.py | 289 + .../spec/test_modeling_v2_m3_fold_tail.py | 302 + .../spec/test_modeling_v2_m3_force_vec.py | 47 + .../spec/test_modeling_v2_m3_force_vec_req.py | 93 + .../spec/test_modeling_v2_m3_lowm_splitk.py | 63 + .../test_modeling_v2_m3_sampled_accept.py | 297 + .../test_modeling_v2_m3_sampled_accept_1r.py | 186 + .../spec/test_modeling_v2_m3_shard_accept.py | 125 + .../spec/test_modeling_v2_m3_slot_scatter.py | 62 + .../spec/test_modeling_v2_m3_spec_offsets.py | 138 + .../spec/test_modeling_v2_m3_step_gather.py | 85 + .../spec/test_modeling_v2_m3_step_prologue.py | 192 + .../test_modeling_v2_m31_ar_det.py | 54 + .../test_modeling_v2_m31_dspark_draft_ar.py | 119 + .../test_modeling_v2_m31_fc_shard.py | 168 + ...est_modeling_v2_m31_host_tail_sm107_tp4.py | 187 + .../test_modeling_v2_m31_mxdraft_tp4_scope.py | 34 + .../test_modeling_v2_m31_onediv_cap.py | 54 + .../test_modeling_v2_m31_prefill_ar_mnnvl.py | 64 + .../test_modeling_v2_m31_proxy_prebuild.py | 57 + .../test_modeling_v2_m31_r6_dense_prequant.py | 35 + .../test_modeling_v2_m31_s10_scope.py | 32 + .../test_modeling_v2_m31_s5_scope.py | 15 + ...test_modeling_v2_m31_switches_sm107_tp4.py | 83 + ...ing_v2_m31_tp4_fused_projection_default.py | 37 + ...test_modeling_v2_m31_tp4_weights_layout.py | 68 + .../test_modeling_v2_m31_vr_bands.py | 109 + .../test_modeling_v2_m3_capture_boundary.py | 257 + .../test_modeling_v2_m3_dispatch.py | 318 + .../test_modeling_v2_m3_prebuild.py | 334 + .../test_modeling_v2_m3_rank_agreement.py | 181 + .../test_modeling_v2_m31_tp8_swq.py | 99 + .../test_modeling_v2_m3_silu_and_mul.py | 49 + .../attention/_m3_attn_runtime_tp8.py | 274 + .../attention/_m3_nvfp4_cache.py | 105 + .../attention/test_modeling_v2_m31_gqamix.py | 189 + ...t_modeling_v2_m3_dense_qk_norm_rope_fp8.py | 144 + .../test_modeling_v2_m3_draft_attention.py | 148 + .../test_modeling_v2_m3_drafter_fullctx.py | 271 + ...test_modeling_v2_m3_drafter_rope_fp8_kv.py | 371 + ...modeling_v2_m3_fp8_indexer_qk_norm_rope.py | 49 + .../test_modeling_v2_m3_fp8_saturate.py | 107 + .../test_modeling_v2_m3_fullctx_decode.py | 310 + .../test_modeling_v2_m3_identity_selection.py | 103 + .../test_modeling_v2_m3_index_score.py | 111 + .../test_modeling_v2_m3_msa_attention.py | 224 + .../test_modeling_v2_m3_n_valid_contract.py | 138 + .../attention/test_modeling_v2_m3_producer.py | 209 + .../test_modeling_v2_m3_qk_norm_rope_fp8.py | 81 + .../test_modeling_v2_m3_qkv_insert.py | 80 + .../test_modeling_v2_m3_select_blocks.py | 45 + .../test_modeling_v2_m3_select_long.py | 179 + .../test_modeling_v2_m3_select_long_dsl.py | 191 + .../test_modeling_v2_m3_sparse_decode.py | 626 ++ .../test_modeling_v2_m3_sparse_decode_lc8.py | 360 + .../test_modeling_v2_m3_tokmap_guard.py | 226 + .../test_modeling_v2_msa_kv_update.py | 128 + .../comm/_m3_allreduce_op_matrix.py | 454 + .../comm/_m3_mc_allreduce_norm_op_matrix.py | 788 ++ .../comm/_m3_moe_push_op_matrix.py | 403 + .../comm/_rank_job.py | 111 + .../comm/test_modeling_v2_m3_allreduce.py | 42 + ...eling_v2_m3_mc_allreduce_norm_op_matrix.py | 62 + .../test_modeling_v2_m3_embedding.py | 37 + .../gemm/_m3_mxfp8_operands.py | 30 + .../test_modeling_v2_m31_mxfp8_cublaslt.py | 141 + .../test_modeling_v2_m31_unfused_index.py | 112 + .../test_modeling_v2_m31_vr_mxgemv_vrcfg.py | 76 + .../gemm/test_modeling_v2_m3_bf16_linear.py | 31 + .../gemm/test_modeling_v2_m3_dense_swiglu.py | 142 + .../gemm/test_modeling_v2_m3_draft_lowm.py | 177 + .../gemm/test_modeling_v2_m3_fp32_linear.py | 32 + .../gemm/test_modeling_v2_m3_mx_gemv.py | 87 + .../gemm/test_modeling_v2_m3_mx_gemv_bf16.py | 63 + .../test_modeling_v2_m3_mx_gemv_producer.py | 236 + .../gemm/test_modeling_v2_m3_mxfp8_gemm.py | 38 + .../gemm/test_modeling_v2_m3_mxfp8_linear.py | 46 + .../gemm/test_modeling_v2_m3_pinned_gemm.py | 52 + .../test_modeling_v2_m3_swiglu_oai_bf16x2.py | 149 + .../gemm/test_mxdraft_gu_swiglu.py | 123 + .../gemm/test_r1b1_gemv_order_gate_a.py | 85 + .../moe/_m3_moe_operands.py | 172 + .../moe/_m3_router.py | 79 + ..._modeling_v2_m31_vr_tp8_moe_early_epoch.py | 59 + ...est_modeling_v2_m31_vr_tp8_moe_halfplan.py | 69 + .../test_modeling_v2_m31_vr_tp8_moe_m8cfg.py | 27 + ...est_modeling_v2_m31_vr_tp8_moe_out_skip.py | 96 + .../test_modeling_v2_m31_vr_tp8_moe_sh20.py | 59 + ...modeling_v2_m31_vr_tp8_moe_tail_trigger.py | 93 + .../test_modeling_v2_m31_vr_tp8_moe_vrpoll.py | 56 + .../moe/test_modeling_v2_m3_moe_engine.py | 139 + .../moe/test_modeling_v2_m3_router_tc.py | 57 + .../norm/test_modeling_v2_m31_ctxnormq.py | 79 + ...test_modeling_v2_m3_embedding_firstnorm.py | 102 + .../test_modeling_v2_m31_chain_spec_mc.py | 527 + .../test_modeling_v2_m31_softmax_mc.py | 256 + .../sampling/test_modeling_v2_m31_topp_det.py | 271 + .../test_modeling_v2_m31_topp_det_v3.py | 249 + .../test_modeling_v2_m31_topp_det_v4k8.py | 211 + .../spec/_m3_spec_inputs.py | 106 + .../spec/test_modeling_v2_m31_dsfold.py | 168 + .../spec/test_modeling_v2_m31_dspark_n3a.py | 473 + .../spec/test_modeling_v2_m31_dspark_s4.py | 347 + .../spec/test_modeling_v2_m31_dspark_w0.py | 164 + ...st_modeling_v2_m31_vr_draft_append_fuse.py | 46 + .../spec/test_modeling_v2_m3_capture_norm.py | 57 + .../test_modeling_v2_m3_capture_normed.py | 71 + .../test_modeling_v2_m3_capture_target.py | 82 + .../test_modeling_v2_m3_capture_workspaces.py | 56 + .../spec/test_modeling_v2_m3_chunk_norm.py | 50 + .../spec/test_modeling_v2_m3_di_fc_flag.py | 91 + .../spec/test_modeling_v2_m3_draft_input.py | 98 + .../test_modeling_v2_m3_draft_push_take.py | 43 + .../spec/test_modeling_v2_m3_draft_silu.py | 88 + .../spec/test_modeling_v2_m3_draft_step.py | 747 ++ ...t_modeling_v2_m3_draft_step_fixed_order.py | 150 + .../spec/test_modeling_v2_m3_fold_tail.py | 376 + .../spec/test_modeling_v2_m3_force_al.py | 63 + .../spec/test_modeling_v2_m3_force_vec.py | 47 + .../spec/test_modeling_v2_m3_force_vec_req.py | 48 + .../spec/test_modeling_v2_m3_lowm_splitk.py | 98 + .../test_modeling_v2_m3_sampled_accept.py | 267 + .../test_modeling_v2_m3_sampled_accept_1r.py | 186 + .../spec/test_modeling_v2_m3_shard_accept.py | 120 + .../spec/test_modeling_v2_m3_slot_scatter.py | 57 + .../spec/test_modeling_v2_m3_spec_offsets.py | 138 + .../spec/test_modeling_v2_m3_step_gather.py | 80 + .../spec/test_modeling_v2_m3_step_prologue.py | 228 + .../test_modeling_v2_m31_ar_det.py | 54 + .../test_modeling_v2_m31_blockoff_guard.py | 165 + .../test_modeling_v2_m31_fc_shard.py | 168 + ...est_modeling_v2_m31_host_tail_sm107_tp8.py | 237 + .../test_modeling_v2_m31_onediv_cap.py | 58 + .../test_modeling_v2_m31_proxy_prebuild.py | 57 + .../test_modeling_v2_m31_r6_dense_prequant.py | 35 + .../test_modeling_v2_m31_s10_scope.py | 15 + .../test_modeling_v2_m31_s5_scope.py | 15 + .../test_modeling_v2_m31_s9b_mc.py | 124 + ...odeling_v2_m31_slot_ids_changed_default.py | 86 + ...test_modeling_v2_m31_switches_sm107_tp8.py | 83 + .../test_modeling_v2_m31_swq_rows.py | 29 + .../test_modeling_v2_m31_tailbk.py | 558 + ...ing_v2_m31_tp8_fused_projection_default.py | 37 + .../test_modeling_v2_m31_tp8_r1vr8.py | 79 + ...test_modeling_v2_m31_tp8_weights_layout.py | 68 + .../test_modeling_v2_m31_vr_bands.py | 109 + ...st_modeling_v2_m31_vr_draft_fmha_sm100f.py | 70 + .../test_modeling_v2_m31_vr_tp8_mcag.py | 144 + .../test_modeling_v2_m3_capture_boundary.py | 192 + .../test_modeling_v2_m3_dispatch.py | 612 ++ .../test_modeling_v2_m3_rank_agreement.py | 190 + .../_torch/modeling_v2/test_m31_pins.py | 260 + .../test_m31_tp4_routing_source.py | 13 +- .../test_m31_tp8_routing_source.py | 13 +- .../test_minimax_m31_f3_ar_prepushed.py | 181 + .../multi_gpu/test_minimax_m31_s2_moe_tail.py | 166 + .../hw_agnostic/test_dflash_context_onehot.py | 72 + .../test_dflash_disagg_draft_ctx.py | 322 + .../test_dflash_forward_context_onehot.py | 183 + tests/unittest/_torch/test_cute_aot.py | 176 + .../_torch/test_flashinfer_autotune_cache.py | 71 + .../test_minimax_m31_dspark_glue_ops.py | 135 + .../test_mxfp8_cute_pinned_tactics.py | 100 + .../unittest/disaggregated/f0a_g1d_compare.py | 183 + .../test_disagg_draft_ctx_state_carrier.py | 99 + .../test_disagg_draft_kv_hook_off.py | 635 ++ .../test_disagg_draft_kv_switch.py | 140 + .../test_disagg_draft_kv_targets.py | 53 + .../test_disagg_draft_kv_transfer.py | 1782 ++++ tests/unittest/disaggregated/test_peer.py | 20 + .../disaggregated/test_rank_info_draft_kv.py | 215 + .../test_kv_gpu_prefetch_runtime.py | 145 + 1145 files changed, 218937 insertions(+), 641 deletions(-) create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.h create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.h create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DenseQkNormRopeFp8Op.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DraftStepOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DrafterRopeFp8KvOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2FoldTailOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2FullCtxReduceOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2McAllreduceNormOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2ProducerPwOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SampledAcceptOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SelectLongOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SiluAndMulOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SparseDecodeLc2Op.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SpecOffsetsOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2StepPrologueOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SwigluOaiOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DenseQkNormRopeFp8Op.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DraftStepOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DrafterRopeFp8KvOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailParams.h create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FullCtxReduceOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4McAllreduceNormOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4ProducerPwOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SampledAcceptOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SelectLongOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SiluAndMulOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SparseDecodeLcOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SpecOffsetsOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4StepPrologueOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SwigluOaiOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8DraftStepOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8DrafterRopeFp8KvOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailParams.h create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FullCtxReduceOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8McAllreduceNormOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8ProducerPwOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SampledAcceptOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SiluAndMulOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SparseDecodeLc8Op.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SpecOffsetsOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8StepPrologueOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SwigluOaiOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31DsparkGlueOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31F3ArOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31FcShardGemmOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31MarkovEpilogueOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31MoeFinalizeOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31S1bFoldOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31S2MoeTailOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SoftmaxPathBOp.cpp create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SwigluOaiVecOp.cpp create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/tp2ctx_ipc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_pins.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_tuning/sm107/tp2/rank_0.json create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_tuning/sm107/tp2/rank_1.json create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/dspark.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/dspark_s4.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/lc2_mixed.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/long_select.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m31_f3_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_attn_lc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_draft_lowm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_drafter_attn.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_fold_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_fullctx_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_index_score_t2sc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_lowm_splitk.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_mc_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/m3_moe16_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/m3_moe_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/op16.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe_finalize.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_nvfp4_index_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_prequant.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_report.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_select_long_dsl.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_spec_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_target_head.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_tokmap_guard.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/m3_mx_gemv_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/phase_producer.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/clc_tile_scheduler.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/triton_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/triton_sparse_decode_mq.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/m31_switches.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/mxdraft.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/m3_silu_and_mul.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/m3_swq.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_dense_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_dense_qk_norm_rope_fp8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_draft_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_fp8_indexer_qk_norm_rope.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_fullctx_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_index_score_t2sc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_msa_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_producer.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_qk_norm_rope_fp8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_qkv_insert.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_blocks.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_long.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_long_dsl.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_sparse_decode_lc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_mc_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_mc_finalize_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/embedding/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/embedding/m3_embedding.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m31_agg_prefill_cublaslt.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_bf16_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_dense_swiglu.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_draft_lowm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_fp32_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mx_gemv.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mx_gemv_bf16.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_cublaslt.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_pinned_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_pinned_quantize.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_target_head.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gqamix_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m31_s1b_fold.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe16_engine.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe_engine.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe_tactic_pin.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_router_tc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/norm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/norm/m3_embedding_firstnorm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_chain_spec_mc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_softmax_mc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_softmax_pathb.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_topp_det.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m31_markov_epilogue.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_capture_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_chunk_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_input.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_silu.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_force_vec_req.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_sampled_accept_ws_words.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_slot_scatter.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_spec_offsets.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_step_gather.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_step_prologue.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/weights.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/dspark.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/dspark_s4.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/long_select.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_attn_lc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_draft_lowm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_drafter_attn.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_fold_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_fullctx_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_index_score_t2sc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_lowm_splitk.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_mc_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/m3_moe16_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/m3_moe_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/op16.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_prequant.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_report.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_select_long_dsl.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_spec_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_target_head.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_tokmap_guard.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/m3_mx_gemv_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/triton_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/triton_sparse_decode_mq.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/m31_switches.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/mxdraft.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/activation/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/activation/m3_silu_and_mul.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_dense_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_dense_qk_norm_rope_fp8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_draft_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_fp8_indexer_qk_norm_rope.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_fullctx_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_index_score_t2sc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_msa_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_producer.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_qk_norm_rope_fp8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_qkv_insert.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_blocks.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_long.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_long_dsl.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_sparse_decode_lc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_mc_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_mc_finalize_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/embedding/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/embedding/m3_embedding.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m31_fc_shard.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_bf16_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_dense_swiglu.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_draft_lowm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_fp32_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mx_gemv.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mx_gemv_bf16.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_pinned_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_pinned_quantize.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_target_head.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gqamix_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_moe16_engine.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_moe_engine.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_router_tc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/norm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/norm/m3_embedding_firstnorm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/sampling/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/sampling/m31_topp_det.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_capture_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_chunk_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_input.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_silu.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_force_vec_req.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_sampled_accept_ws_words.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_slot_scatter.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_spec_offsets.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_step_gather.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_step_prologue.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/weights.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/dspark.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/dspark_s4.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/long_select.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_attn_lc8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_drafter_attn.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fc_wait.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fold_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fullctx_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_layer/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_layer/phase_producer.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_locality.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_lowm_splitk.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_mc_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/m3_moe_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_prequant.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_report.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_select_long_dsl.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_spec_tail.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_target_head.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_tokmap_guard.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/m3_mx_gemv_kernel.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/op.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/triton_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/triton_sparse_decode_mq.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/m31_switches.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/mxdraft.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/m3_silu_and_mul.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/m3_swq.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_dense_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_draft_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_fp8_indexer_qk_norm_rope.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_fullctx_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_index_score.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_msa_attention.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_producer.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_qk_norm_rope_fp8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_qkv_insert.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_blocks.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_long.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_long_dsl.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_sparse_decode_lc8.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/msa_kv_update.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_allreduce.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_mc_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_mc_finalize_allreduce_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/embedding/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/embedding/m3_embedding.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m31_fc_shard.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_bf16_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_dense_swiglu.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_draft_lowm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_fp32_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mx_gemv.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mx_gemv_bf16.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_cublaslt.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_gemm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_linear.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_target_head.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gqamix_sparse_decode.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/m3_moe_engine.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/m3_router_tc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/norm/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/norm/m3_embedding_firstnorm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_chain_spec_mc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_softmax_mc.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_topp_det.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/__init__.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_capture_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_chunk_norm.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_input.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_silu.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_step.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_force_vec.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_sampled_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_sampled_accept_ws_words.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_shard_accept.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_slot_scatter.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_spec_offsets.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_step_gather.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_step_prologue.py create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/weights.py create mode 100644 tensorrt_llm/_torch/attention/backends/prims_ts/_worktile_scope.py create mode 100644 tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/nvfp4_index_k.py create mode 100644 tensorrt_llm/_torch/custom_ops/mxfp8_cute_pinned_tactics.py create mode 100644 tensorrt_llm/_torch/cute_aot.py create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k00eed757e39938fb731637e794868820.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0218894bfe0a33f003526be955b1a5a2.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k027cc4c61c652bd6b1876fd15a2cbe78.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k02f9c681999d3610c65efb69f58ef67d.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k06187bd3c325e618ea22baa0688e80f9.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k075b4d53fecff042c8aacfdc99fd270e.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k084b52de5bc59b14164d9d5f14534621.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0881662f562befcc1c32b65bd7a99070.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k09cc9c2f6949a2cd3d99c8317996d977.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0a41d63ce5fcc4e48ef95d3f5f0df2d0.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0c5fc19711de959b6135918a6874bc83.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0f8ddce1d7ec250fbf885785c6685dcf.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k107e6a9efe22fee8544bed815eb03368.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1306931fc2e1dae2a5a29cbf4450af9f.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k16be2b692ddbe79e186bc00a13703cbe.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1a497f11e71eecd11561cfae8dbf9368.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1b19f4137dccb0b8c55a535ad5d89d27.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1c3f1a6c44770333636569ea847afd22.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1c732009be9238cb2ce6a18efbd177f8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1dffa8015f154621562b41428d264bb1.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1ea2c0d63cbc201cbc33ca38478ecb34.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1fc6e741d499c4f215a1ac023c31ddf4.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k221e3d1cbd479c8444e94c698e544677.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2531744bfa75437fb5135dfdd624fb5b.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k257864cebcfb331fdedf891003cb1c7a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k29635f4ef9bf64e611db5121637fa3c3.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k29ad84cf333079e0a1d3b932c4ded45e.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k29d76b4da51071cbe677d0d83ec77d99.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2a14e4b7e315b91abc4b37e7c8e5912a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2a3daf4105dbe50ecee4c53a13c1bb8d.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2af3de0526072a130bf76c531284a306.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2c3a02ad45c3d395ae376a6baf737fe8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2c5121f2f0610b0471c2b91d16d809e0.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2d44f4edabc0f33b82391a9d8dc1a6ce.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2e8bbd58d865a589130563dc4d1a2af0.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2f391ea29f162b08a27e62beca7c3210.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k317bebfdef9ea155a409089e0bbd5030.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3399a017f924826ae7af74c8721d0afb.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k342b3e9accc758690d57acd6c1ad4a3c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k37caad1b21b00553597efe4623e78827.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k39138ce03fbb46891281e57dda50e68c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3a806c0edb8d0586bcf26ba141c1bee9.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3bbc8b2a3a7b5c07d7939606525983dd.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3bc9b617ff7ca80a27e4613a56b743e3.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3c6ac10abb8ba633cc91b9123f27f780.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3deb65d63c241dded8f9f6feee524cfd.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3e010db787a2ead6b11796e755278465.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3e29dac67c8e14db8d4c27d6f06ff15c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3f1dc08f97f6bc42e42599fb66daa0d8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3f64fb35e6c6d45c9c387172cd70e1b5.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k40f506331a47b27d6c476c5df332cfd3.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k410baccb2ba161cb240e3f170f960314.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k41f577a6984ff8bee476d34a62caed8c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k42514d5f2dc215403c9c11ec79f6625e.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k42677b72c271123e0fde15397661c8d6.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k457ffd832f18f0d27324061d1eb1ce5a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k459e034c1eff4cabc5badd7bd4dc16ae.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k46dfb5c68e57fa3a4a80950d45a1aa26.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k49b951f8bc5ea8cfe38dd52cea8745d7.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k4ee158f9d78e2f4af22ca8cdf25035bf.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k50781d01e99590a8c8543b21a6721d65.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k52af4c03f6b8e5990e9d227e1421165e.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k536e504c92bdbc994c692ca2c2cca754.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k53a1053964f40eebd57ef8419af84352.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k53dfccd97af20f0564192076fba638e6.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k544d4b9f3ea0ea08886c5dc10e08c42b.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k55ed6be359ce74e2cb00ec59dbd6ad89.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5768d8426ce95a796c57d497ca754f7d.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5c30d6e150a9ec66ae0a094c7fd60913.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5d54e1f60cd25f1f4f8d7e0a394cebf2.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5f07b3765de26e236ae17b8ff0eac7f0.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k602da62aafbfde53540fa18526658aa0.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k609f04ac135d191eeca3dc8cf40657cf.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k60b43c6114d06189e061601bda5423fd.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k60b5079290ad2088526acd151c5b9a3f.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k60c66d6ef2e5e14f52634e40438b9d85.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k624b4bd74939b58bb85e6b0a212ac48d.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k62d97aff61c66ea719a2978c9072de7e.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k64480700cb22a21c82f517aa00c1365a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k64849d4081bef6f3ea56a9f09a4c2417.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k65494c2c19a9814c6b2344a80dc7d835.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k67c2ec2335be3981cab8732df48e2960.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k6a230f0482598b0b12a92ec687a861b8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k6b155ec20fbadbfdafba8d376dabe43b.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k6c8122079f737fb229dab3f7c9f1e88b.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k6d69f27693eca302313fb758e28da794.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k6fe8c0040c9ee3e70a0678e04af81999.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k71abcbc4b33e413259fa590cad8fae89.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k71fddb6eb415bdae83414da29a477c1d.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k729d5c5f515c2989c6f4645046a8fc78.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k730d971c25c04dea469c917abe6da00f.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k73ea43a43a3c248aeb51c39dcc32757c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k76837390ba7090d03755f38f335984c6.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k77540beeb21572cf40c1465139a8e3e8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k7895e519c8fd50399a10ee9c9b4722bb.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k7a842663c2f6cfc297b0bd19e12cefee.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k7ac2accb0dfe6f9715bd7db142efee09.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k7ac78b16d6aa49ee5ba408abe9f02706.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k7e11bd01538e0e8fd354e6564ee71ac0.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k81128243050e8617afaf6db477a16040.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8686d739fe1cf565bc7e78a1ab51c945.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k86d244e9f495308f819e50e86f604433.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k877d7aa659e5e1d816e0442761e33bde.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k87a26cbd91074d4c090e61c7442a511b.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k88642f4ca698fd1087f6caa67efb57e8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k88fa6b951eb20c313aa59b4adbc69fe1.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k89fbca15f9847079d4dfb2ed49d66e4f.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8b914d547c30519c09e45ca3937f65b2.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8be43f1e0571e1f4d5fa83daf0d1b144.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8bfaf12a78e278972fe4cfa229c5ff5e.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8d2a1751722203553aa1a28872dfbb04.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8d38d65c9e1a4e267b294dbf112f052f.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8e653ebb28adfde7815602f8b4d83f31.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8f7c307820544eeca13d80f89acf695b.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k91103515cccf539429183d92cfb39807.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k91346849ef2de43f93d0e96a65664eaf.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k91d99bae224a9801fe14037f80954e7c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k9431f967afdb58beb949df76f58cee5b.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k95bff0816ba2d5a0a7594e89066c750a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k967c39595b6f81d02c06645d911c9961.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k98add47b626026c2503673796d8185e5.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k99c15121f2eb0d1777f2be633e7e5072.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k9a6d1706c5fcdb285a9042f823e51edb.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka267b2dcc02441456cd57472dbc874da.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka4a690575a315b7d9738c20e7f242a36.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka7d5145bbf7eda9f6df50f622e00dff3.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka910e7175f9a30bf6e713cb86932fc46.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka9d2996a6c1881bcab65e7825da4f469.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kab00779b3ea6268c3fa8c5384347c979.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kac95216b414b7faede973efab3c4be64.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb21b47d8c1c250b3a20cf80112ab527c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb2dd3ea5f861e452f4ecbb580b3fca92.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb4be9a246f5fb4af8d3eed89aa446cf4.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb624af19573b968e6cd35f81411b3a44.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb666059f738b7cfb01e45a484e3a0dd2.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb689e14232999f4c2151d328fb2e0334.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb74d7bb748b941e9fe6bc1f8fdd93198.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb7a6af396879822c1534ec7d3163f8df.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb95baa3070802f80dcd903c53a3964f8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kbb78dda30e4032356ec7be602bc3b424.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kbe6ee4564892ecd24074538ed0d17f2d.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kbfb313640115702d8139291c633cb61d.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc1775ee7a18143c76ca13d2b494c4727.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc1c162425e681df83e1b62b93d0679bf.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc1c7f67dc4e7c17390bf4c8ace76f391.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc3053a2621af680d56e409cc084a8db7.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc3cc186b225f1f23b6d1efb44ec7dbd2.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc415ff8fa08210a6a77393bf85ddf0be.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc4b328c6e8f71c2b2fd152f8077055f8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc5932f15ac9d608adbc379cb72c7640c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc911260e8a4241f05592f229dc40512f.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc9754f51e923c13afe1180ef003e4fa4.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kce1906dfc7ccb380301b314df59ba00b.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kceb214f8b302ec91a2b88cd52602bb69.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kcf0367abcf989ba1c42a0731b61881c8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kcf09707be55af5195f4589640abe27f2.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd008badb8f4e2cb34ba95f2b40b06014.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd3364a16830f1a3bead73b2be4199b92.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd3bd7786587938e328844c978cd6b2e3.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd4187341a82c234fffdcd7d661616fb9.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd579851969c62870ca4274ebc92d651a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd80050a13b9de78420fb0bfa3e80921f.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd8021408bba1af62bfc43668e1cea7d1.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kda850ab5991c75d665f25508ff6d34b9.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kdbd4019aa02bf65c25564ca198a77085.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kde44e6074303c68d4b290ad3929fb761.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke0c7c73e4b05f100d7e599c08027ab7a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke2a030185a3f56dc9281490537f5c6ab.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke30c27a09f840d696840df876a6e6ea9.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke48338ff5e721e5e5119deb5226df72d.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke4fc62a58d6e4f7e4ae0a2087df8155a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke557e36f3e1fad7aebc9847d4b6b62bb.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke6b56a141d6efee652214e6d50494ac2.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke8996f833d1f91e81cf2644d8fc15fb7.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke9a6508eb80d23a7f3ed6591f72e643b.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kea2fa28c84c229f5b6224bcf2285a787.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kebcef6942084641815e5579414860d93.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kec7fea04a276529bce9626a6fdcab07a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kec9f140f67d7409282564bd81783237c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kee904649a81dda6daccd4f9b1b52a773.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/keebaa803d23596dc67f8cc9d626ca23e.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/keef8a85fe494ecd5dc420eb65fb1c61a.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf0ecb78ca25035a27ab95b3b3d67f2b4.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf16023d317c918040f4b5c0e25e1405d.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf3232ddee2a0b12f6c32292a49d23f44.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf3c2d7335a14ad90793841232ed9c10f.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf3c4db3cd960b261d0847ebaf22df84c.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf7b5a92e434ffcac257a03692b343b68.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf868b615acd9df2fff40fce7e73a7e6e.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf935e0bf1ab3deb88c6e9fd2a48be7e8.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kfafff4c85c826306b570c6022b95eb40.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kfeb0ee198c6718291355fc1db62e121e.o create mode 100644 tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kfef3ffb9125b7358714c2ccf6aedd14e.o create mode 100644 tensorrt_llm/_torch/disaggregation/draft_kv.py create mode 100644 tensorrt_llm/_torch/flashinfer_autotune_cache.py create mode 100644 tensorrt_llm/_torch/late_compile.py create mode 100644 tests/unittest/_torch/executor/kv_cache/test_disagg_gen_init_draft_window.py create mode 100644 tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py create mode 100644 tests/unittest/_torch/executor/test_request_token_array.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/activation/test_modeling_v2_m31_tp2_swq.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/activation/test_modeling_v2_m3_silu_and_mul.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/_m3_attn_runtime_tp2.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/_m3_nvfp4_cache.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_gqamix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_lc_tp2_default.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_tp2wide_intree_scorer.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_fp8_saturate.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_index_score.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_index_score_t2sc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_msa_attention.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_n_valid_contract.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_producer.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_qkv_insert.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_blocks.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_long.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_long_dsl.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode_lc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode_lc_split.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_tokmap_guard.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_msa_kv_update.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_m3_allreduce_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_m3_mc_allreduce_norm_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_rank_job.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_allreduce.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_mc_finalize_allreduce_norm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/embedding/test_modeling_v2_m3_embedding.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/_m3_mxfp8_operands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_tp2_gemvcfg.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_tp2_qkv_producer_fold.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_unfused_index.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_bf16_linear.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_dense_swiglu.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_fp32_linear.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv_bf16.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv_order.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_gemm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_linear.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_pinned_gemm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_pinned_quantize.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/_m3_moe_operands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/_m3_router.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_moe16_tp2_off.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_finalize.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_finalize_rows.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_tactic_pin.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_tactic_pin_gpu.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_x1.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_moe_engine.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_moe_flat_gate_tp2.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_router_tc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_dispatch_lines.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_finalize.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_swiglu.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/norm/test_modeling_v2_m31_router_f32.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/norm/test_modeling_v2_m3_embedding_firstnorm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_chain_spec_mc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_rej_slots_default.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_rej_slots_wide.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_softmax_mc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_softmax_pathb.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_topp_det.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_topp_det_wide.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/_m3_spec_inputs.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dsfold.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_n3a.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_s4.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_w0.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_s4fe.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail16_g2.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail16_g3.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail_bk3.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_spec_offsets.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_step_gather.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_step_prologue.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_agg_qkv_a1.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_ar_det.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_blockoff_guard.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_draft_a1.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_dspark_draft_ar.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_hook_work_counters.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_host_tail_sm107_tp2.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_mxdraft_tp2_scope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_onediv_cap.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_prefill_ar_mnnvl.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_proxy_prebuild.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_r6_dense_prequant.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s10_scope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s2_tail.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s4_logits_default.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s5_scope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_slot_ids_changed_default.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_switches_sm107_tp2.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_dspark_scope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_fused_projection_default.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_prefill_brk.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_r1vr8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_rulings.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_weights_layout.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_work_counters.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2agg_sp.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_decode.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_gemm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_prefill_attn.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_bands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_draft_fmha_sm100f.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_tp2_mcag.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_capture_boundary.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_dispatch.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_prebuild.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_rank_agreement.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/activation/test_modeling_v2_m3_silu_and_mul.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/_m3_attn_runtime_tp4.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/_m3_nvfp4_cache.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m31_gqamix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_dense_decode.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_dense_qk_norm_rope_fp8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_draft_attention.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_drafter_fullctx.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_drafter_rope_fp8_kv.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fp8_saturate.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fullctx_decode.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_index_score.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_index_score_t2sc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_msa_attention.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_n_valid_contract.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_producer.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_qkv_insert.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_blocks.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_long.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_long_dsl.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_sparse_decode.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_sparse_decode_lc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_tokmap_guard.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_msa_kv_update.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_allreduce_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_capture_in_boundary_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_draft_ar_norm_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_mc_allreduce_norm_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_rank_job.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_allreduce.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_capture_in_boundary.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_draft_ar_norm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_mc_finalize_allreduce_norm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/embedding/test_modeling_v2_m3_embedding.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/_m3_mxfp8_operands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m31_unfused_index.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_bf16_linear.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_dense_swiglu.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_draft_lowm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_fp32_linear.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv_bf16.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv_order.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_gemm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_linear.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_pinned_gemm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_pinned_quantize.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_target_head.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/_m3_moe_operands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/_m3_router.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m31_vr_tp4_moe_out_skip.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m31_vr_tp4_moe_tail_trigger.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe16_engine.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe_engine.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe_flat_gate_tp4.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_router_tc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/norm/test_modeling_v2_m31_ctxnormq.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/norm/test_modeling_v2_m3_embedding_firstnorm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/sampling/test_modeling_v2_m31_topp_det.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/_m3_spec_inputs.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_n3a.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_s4.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_w0.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_vr_draft_append_fuse.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_capture_norm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_capture_normed.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_chunk_norm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_input.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_push_take.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_silu.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_step.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_step_fixed_order.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_exact_numerics.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_fold_tail.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_force_vec.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_force_vec_req.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_lowm_splitk.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_sampled_accept.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_sampled_accept_1r.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_shard_accept.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_slot_scatter.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_spec_offsets.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_step_gather.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_step_prologue.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_ar_det.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_dspark_draft_ar.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_fc_shard.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_host_tail_sm107_tp4.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_mxdraft_tp4_scope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_onediv_cap.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_prefill_ar_mnnvl.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_proxy_prebuild.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_r6_dense_prequant.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_s10_scope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_s5_scope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_switches_sm107_tp4.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_tp4_fused_projection_default.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_tp4_weights_layout.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_vr_bands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_capture_boundary.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_dispatch.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_prebuild.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_rank_agreement.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/activation/test_modeling_v2_m31_tp8_swq.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/activation/test_modeling_v2_m3_silu_and_mul.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/_m3_attn_runtime_tp8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/_m3_nvfp4_cache.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m31_gqamix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_dense_qk_norm_rope_fp8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_draft_attention.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_drafter_fullctx.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_drafter_rope_fp8_kv.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fp8_saturate.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fullctx_decode.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_identity_selection.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_index_score.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_msa_attention.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_n_valid_contract.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_producer.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_qkv_insert.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_blocks.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_long.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_long_dsl.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_sparse_decode.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_sparse_decode_lc8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_tokmap_guard.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_msa_kv_update.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_allreduce_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_mc_allreduce_norm_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_moe_push_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_rank_job.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/test_modeling_v2_m3_allreduce.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/embedding/test_modeling_v2_m3_embedding.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/_m3_mxfp8_operands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_mxfp8_cublaslt.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_unfused_index.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_vr_mxgemv_vrcfg.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_bf16_linear.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_dense_swiglu.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_draft_lowm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_fp32_linear.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv_bf16.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv_producer.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mxfp8_gemm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mxfp8_linear.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_pinned_gemm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_swiglu_oai_bf16x2.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_mxdraft_gu_swiglu.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_r1b1_gemv_order_gate_a.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/_m3_moe_operands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/_m3_router.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_early_epoch.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_halfplan.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_m8cfg.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_out_skip.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_sh20.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_tail_trigger.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_vrpoll.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m3_moe_engine.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m3_router_tc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/norm/test_modeling_v2_m31_ctxnormq.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/norm/test_modeling_v2_m3_embedding_firstnorm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_chain_spec_mc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_softmax_mc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det_v3.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det_v4k8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/_m3_spec_inputs.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dsfold.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_n3a.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_s4.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_w0.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_vr_draft_append_fuse.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_norm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_normed.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_target.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_workspaces.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_chunk_norm.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_di_fc_flag.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_input.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_push_take.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_silu.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_step.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_step_fixed_order.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_fold_tail.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_al.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_vec.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_vec_req.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_lowm_splitk.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_sampled_accept.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_sampled_accept_1r.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_shard_accept.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_slot_scatter.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_spec_offsets.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_step_gather.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_step_prologue.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_ar_det.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_blockoff_guard.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_fc_shard.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_host_tail_sm107_tp8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_onediv_cap.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_proxy_prebuild.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_r6_dense_prequant.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s10_scope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s5_scope.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s9b_mc.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_slot_ids_changed_default.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_switches_sm107_tp8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_swq_rows.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tailbk.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_fused_projection_default.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_r1vr8.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_weights_layout.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_bands.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_draft_fmha_sm100f.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_tp8_mcag.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_capture_boundary.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_dispatch.py create mode 100644 tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_rank_agreement.py create mode 100644 tests/unittest/_torch/modeling_v2/test_m31_pins.py create mode 100644 tests/unittest/_torch/multi_gpu/test_minimax_m31_f3_ar_prepushed.py create mode 100644 tests/unittest/_torch/multi_gpu/test_minimax_m31_s2_moe_tail.py create mode 100644 tests/unittest/_torch/speculative/hw_agnostic/test_dflash_context_onehot.py create mode 100644 tests/unittest/_torch/speculative/hw_agnostic/test_dflash_disagg_draft_ctx.py create mode 100644 tests/unittest/_torch/speculative/test_dflash_forward_context_onehot.py create mode 100644 tests/unittest/_torch/test_cute_aot.py create mode 100644 tests/unittest/_torch/test_flashinfer_autotune_cache.py create mode 100644 tests/unittest/_torch/thop/parallel/test_minimax_m31_dspark_glue_ops.py create mode 100644 tests/unittest/_torch/thop/parallel/test_mxfp8_cute_pinned_tactics.py create mode 100644 tests/unittest/disaggregated/f0a_g1d_compare.py create mode 100644 tests/unittest/disaggregated/test_disagg_draft_ctx_state_carrier.py create mode 100644 tests/unittest/disaggregated/test_disagg_draft_kv_hook_off.py create mode 100644 tests/unittest/disaggregated/test_disagg_draft_kv_switch.py create mode 100644 tests/unittest/disaggregated/test_disagg_draft_kv_targets.py create mode 100644 tests/unittest/disaggregated/test_disagg_draft_kv_transfer.py create mode 100644 tests/unittest/disaggregated/test_rank_info_draft_kv.py create mode 100644 tests/unittest/kv_cache_manager_v2_tests/test_kv_gpu_prefetch_runtime.py diff --git a/3rdparty/vendor_patches/flashinfer-prims-ts.patch b/3rdparty/vendor_patches/flashinfer-prims-ts.patch index d6f80daccc16..bbd4b392acef 100644 --- a/3rdparty/vendor_patches/flashinfer-prims-ts.patch +++ b/3rdparty/vendor_patches/flashinfer-prims-ts.patch @@ -1,3 +1,147 @@ +diff --git a/__init__.py b/__init__.py +index e40d8ce0d41c48ad17099e097c64a70e1f6d7a4b..c49ad2fc908d8794105d7340d7614347db123c53 100644 +--- a/__init__.py ++++ b/__init__.py +@@ -14,6 +14,8 @@ + + """Experimental task-scheduled attention entry points.""" + ++from . import _worktile_scope # noqa: F401 (first: scopes task scheduling's WorkTileInfo rebind) ++ + from .block_sparse import ( + BlockSparsePagedTSWrapper, + BlockSparseTSWrapper, +diff --git a/_block_sparse/compiler.py b/_block_sparse/compiler.py +index 1a9d42857f94459f8d2bb8bb523080c0b6bf1ea7..d557504e26512148da732a891692e9c5ea295b90 100644 +--- a/_block_sparse/compiler.py ++++ b/_block_sparse/compiler.py +@@ -14,6 +14,7 @@ + + """Compilation and caching for PrimTS block-sparse attention adapters.""" + ++from .._worktile_scope import with_scalar_work_tile_info + from collections.abc import Callable + import functools + +@@ -29,6 +30,7 @@ from .config import _BlockSparseCompileKey, _make_block_sparse_config + _COMPILE_OPTIONS = "--enable-tvm-ffi --opt-level 3" + + ++@with_scalar_work_tile_info + def _compile_block_sparse(key: _BlockSparseCompileKey) -> Callable[..., object]: + """Compile one contiguous or paged attention adapter for ``key``. + +diff --git a/_worktile_scope.py b/_worktile_scope.py +new file mode 100644 +index 0000000000000000000000000000000000000000..45fd51f97d979d56aea3707176dc242f73116058 +--- /dev/null ++++ b/_worktile_scope.py +@@ -0,0 +1,105 @@ ++# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. ++# SPDX-License-Identifier: Apache-2.0 ++"""Keep CuTe DSL task scheduling's WorkTileInfo rebind inside PrimTS. ++ ++Importing ``cutlass.experimental.task_scheduling`` rebinds six attributes of CUTLASS' ++``WorkTileInfo`` (``__init__``, the two MLIR value hooks, ``tile_idx``, ``is_valid_tile`` and ++``update_from``) to a scalar three-axis form, for every kernel in the process. Kernels that ++subclass ``WorkTileInfo`` with a different tile shape (FA4's four-axis tile) then fail to trace. ++ ++This module is imported first by the package. It snapshots the original attributes, imports ++the task scheduling resources (which apply the rebind) and snapshots the scalar set. Until a model ++calls ``arm()`` the rebind stays global, as before. Once armed, the originals are put back and ++``scalar_work_tile_info`` installs the scalar set only while a PrimTS kernel traces and compiles; ++compiled kernels do not call these Python methods. Nothing about the generated code changes. ++Kill switch: TRTLLM_PRIMS_TS_SCOPED_WORKTILE=0 makes ``arm()`` keep the global rebind. ++""" ++ ++import contextlib ++import functools ++import os ++import sys ++import threading ++ ++from cutlass.utils.static_persistent_tile_scheduler import WorkTileInfo as _WorkTileInfo ++ ++_ATTRS = ( ++ "__init__", ++ "__extract_mlir_values__", ++ "__new_from_mlir_values__", ++ "tile_idx", ++ "is_valid_tile", ++ "update_from", ++) ++_MISSING = object() ++_RESOURCES = "cutlass.experimental.task_scheduling.resources" ++ ++ ++def _snapshot() -> dict: ++ return {name: _WorkTileInfo.__dict__.get(name, _MISSING) for name in _ATTRS} ++ ++ ++def _install(attrs: dict) -> None: ++ for name, value in attrs.items(): ++ if value is _MISSING: ++ if name in _WorkTileInfo.__dict__: ++ delattr(_WorkTileInfo, name) ++ else: ++ setattr(_WorkTileInfo, name, value) ++ ++ ++# Only scope the rebind when this module saw the originals, i.e. task scheduling was not imported ++# before PrimTS. Otherwise keep whatever is installed (the previous global behavior). ++_ORIGINAL = None if _RESOURCES in sys.modules else _snapshot() ++import cutlass.experimental.task_scheduling.resources # noqa: E402,F401 (applies the scalar rebind) ++ ++_SCALAR = _snapshot() ++SCOPED = False ++_LOCK = threading.RLock() ++_DEPTH = 0 ++ ++ ++def arm() -> bool: ++ """Scope the rebind to PrimTS traces from now on (a model opt-in); returns whether it is scoped. Keeps the ++ global rebind when task scheduling was imported before PrimTS or TRTLLM_PRIMS_TS_SCOPED_WORKTILE=0.""" ++ global SCOPED ++ with _LOCK: ++ if ( ++ not SCOPED ++ and _ORIGINAL is not None ++ and _DEPTH == 0 ++ and os.environ.get("TRTLLM_PRIMS_TS_SCOPED_WORKTILE", "1") != "0" ++ ): ++ SCOPED = True ++ _install(_ORIGINAL) ++ return SCOPED ++ ++ ++@contextlib.contextmanager ++def scalar_work_tile_info(): ++ """Install the task scheduling WorkTileInfo form while a PrimTS kernel traces (re-entrant).""" ++ global _DEPTH ++ if not SCOPED: ++ yield ++ return ++ with _LOCK: ++ if _DEPTH == 0: ++ _install(_SCALAR) ++ _DEPTH += 1 ++ try: ++ yield ++ finally: ++ _DEPTH -= 1 ++ if _DEPTH == 0: ++ _install(_ORIGINAL) ++ ++ ++def with_scalar_work_tile_info(fn): ++ """Decorator form of ``scalar_work_tile_info`` for the PrimTS compile functions.""" ++ ++ @functools.wraps(fn) ++ def wrapper(*args, **kwargs): ++ with scalar_work_tile_info(): ++ return fn(*args, **kwargs) ++ ++ return wrapper diff --git a/block_sparse.py b/block_sparse.py index d4652b8b68958ee3dab29d062a7cd082ea604c47..bb56724e986d302854722bf33962bea063f9e6e4 100644 --- a/block_sparse.py @@ -61,10 +205,18 @@ index d4652b8b68958ee3dab29d062a7cd082ea604c47..bb56724e986d302854722bf33962bea0 q: torch.Tensor, paged_kv_cache: PagedKVCache, diff --git a/context.py b/context.py -index dd9a29c3e36111a0de9a57c487068b6b5caa4ea2..1163324ec632f2a0938f78a3636fb415450369df 100644 +index dd9a29c3e36111a0de9a57c487068b6b5caa4ea2..171c02224d12fd7d0481a7ba1d905fc8281b1ec4 100644 --- a/context.py +++ b/context.py -@@ -46,7 +46,7 @@ from typing import TYPE_CHECKING, Callable, Literal, Optional +@@ -35,6 +35,7 @@ now; unlike the decode APIs, their ``@flashinfer_experimental_api`` decorators d + register trace templates. + """ + ++from ._worktile_scope import with_scalar_work_tile_info + from dataclasses import dataclass + import functools + import importlib.metadata +@@ -46,7 +47,7 @@ from typing import TYPE_CHECKING, Callable, Literal, Optional import torch @@ -73,7 +225,7 @@ index dd9a29c3e36111a0de9a57c487068b6b5caa4ea2..1163324ec632f2a0938f78a3636fb415 if TYPE_CHECKING: -@@ -488,7 +488,7 @@ def _validate_device(device: torch.device) -> int: +@@ -488,7 +489,7 @@ def _validate_device(device: torch.device) -> int: # Rubin runs through the sm_100f family target; a CuTe DSL older than 4.8 # cannot emit for it unless CUTE_DSL_ARCH=sm_100f is set before import. if capability == (10, 7): @@ -82,8 +234,24 @@ index dd9a29c3e36111a0de9a57c487068b6b5caa4ea2..1163324ec632f2a0938f78a3636fb415 require_cute_dsl_arch(device_index) return device_index +@@ -1753,6 +1754,7 @@ def _paged_context_compile_spec( + + + @functools.cache ++@with_scalar_work_tile_info + def _get_compiled_context( + compile_spec: _ContextCompileSpec, + ): +@@ -1973,6 +1975,7 @@ def _get_compiled_context( + + + @functools.cache ++@with_scalar_work_tile_info + def _get_compiled_paged_context( + compile_spec: _PagedContextCompileSpec, + ): diff --git a/decode.py b/decode.py -index 41479a386d72c27eb0df76107318dc09256aff06..3fcec5efdc2a3825be0dc1d9a238aaaa6d76a609 100644 +index 41479a386d72c27eb0df76107318dc09256aff06..d720bb765993e875789e443029996a012d12a4f0 100644 --- a/decode.py +++ b/decode.py @@ -1,3 +1,4 @@ @@ -91,7 +259,20 @@ index 41479a386d72c27eb0df76107318dc09256aff06..3fcec5efdc2a3825be0dc1d9a238aaaa # Copyright (c) 2026 by FlashInfer team. # # Licensed under the Apache License, Version 2.0 (the "License"); -@@ -24,11 +25,7 @@ from typing import TYPE_CHECKING, Literal, NamedTuple, Optional, Union, cast +@@ -14,21 +15,22 @@ + + """Task-scheduled paged decode with a FlashInfer-style plan/run lifecycle.""" + ++from ._worktile_scope import with_scalar_work_tile_info + from collections.abc import Callable, Sequence ++import dataclasses + from dataclasses import dataclass + import functools + import math + import numbers + import struct ++from pathlib import Path + from typing import TYPE_CHECKING, Literal, NamedTuple, Optional, Union, cast import torch @@ -101,10 +282,12 @@ index 41479a386d72c27eb0df76107318dc09256aff06..3fcec5efdc2a3825be0dc1d9a238aaaa - prims_ts_decode_wrapper_trace_dispatch, -) +from flashinfer.api_logging import flashinfer_api as flashinfer_experimental_api ++ ++from tensorrt_llm._torch import cute_aot from . import _q_token_kv_block_sparse_policy as _sparse_policy from ._block_sparse.common import _num_sparse_pattern_heads -@@ -789,7 +786,7 @@ def _validate_runtime_device(device: torch.device) -> None: +@@ -789,7 +791,7 @@ def _validate_runtime_device(device: torch.device) -> None: # Rubin runs through the sm_100f family target; a CuTe DSL older than 4.8 # cannot emit for it unless CUTE_DSL_ARCH=sm_100f is set before import. if capability == (10, 7): @@ -113,7 +296,138 @@ index 41479a386d72c27eb0df76107318dc09256aff06..3fcec5efdc2a3825be0dc1d9a238aaaa require_cute_dsl_arch(device_index) -@@ -4046,7 +4043,7 @@ class BatchDecodePagedTSWrapper: +@@ -1513,6 +1515,7 @@ def _make_decode_compile_spec( + + + @functools.cache ++@with_scalar_work_tile_info + def _get_compiled_decode( + compile_spec: _DecodeCompileSpec, + ): +@@ -1906,67 +1909,79 @@ def _get_compiled_decode( + ), + ) + ++ # The device index does not change the compiled code: one shipped object serves every device. ++ aot_spec = dataclasses.replace(compile_spec, device_index=0) + with torch.cuda.device(device_index): +- compiled_main = cute.compile( +- main_tensor_adapter, +- q_fake, +- k_fake, +- v_fake, +- k_sf_fake, +- v_sf_fake, +- out_fake, +- seq_lens_fake, +- cu_seqlens_q_fake, +- block_table_fake, +- q_token_kv_block_sparse_page_memberships_fake, +- partial_o_fake, +- partial_stats_fake, +- counter_fake, +- attention_sinks_fake, +- Int64(1), +- Int64(1), +- Int64(1), +- Int64(1), +- Int64(1), +- Int64(1), +- Int64(1), +- Float32(1.0), +- Float32(1.0), +- direct_q1_fake, +- stream_fake, +- cfg, +- seq_len_q, +- num_qo_heads, +- num_kv_heads, +- head_dim, +- max_kv_len, +- max_active_clusters, +- static_full_split_prefix, +- static_native_uniform_kv, +- options=_COMPILE_OPTIONS, +- ) +- compiled_reducer = None +- if cfg.use_separate_reduction_kernel: +- assert reduction_tensor_adapter is not None +- compiled_reducer = cute.compile( +- reduction_tensor_adapter, ++ compiled_main = cute_aot.compile_or_load( ++ f"prims_ts_decode.main {aot_spec!r}", ++ lambda: cute.compile( ++ main_tensor_adapter, ++ q_fake, ++ k_fake, ++ v_fake, ++ k_sf_fake, ++ v_sf_fake, + out_fake, + seq_lens_fake, + cu_seqlens_q_fake, ++ block_table_fake, ++ q_token_kv_block_sparse_page_memberships_fake, + partial_o_fake, + partial_stats_fake, ++ counter_fake, + attention_sinks_fake, ++ Int64(1), ++ Int64(1), ++ Int64(1), ++ Int64(1), ++ Int64(1), ++ Int64(1), ++ Int64(1), + Float32(1.0), + Float32(1.0), + direct_q1_fake, + stream_fake, + cfg, ++ seq_len_q, + num_qo_heads, + num_kv_heads, + head_dim, + max_kv_len, ++ max_active_clusters, + static_full_split_prefix, ++ static_native_uniform_kv, + options=_COMPILE_OPTIONS, ++ ), ++ sources=_aot_sources(), ++ tvm_ffi=True, ++ ) ++ compiled_reducer = None ++ if cfg.use_separate_reduction_kernel: ++ assert reduction_tensor_adapter is not None ++ compiled_reducer = cute_aot.compile_or_load( ++ f"prims_ts_decode.reducer {aot_spec!r}", ++ lambda: cute.compile( ++ reduction_tensor_adapter, ++ out_fake, ++ seq_lens_fake, ++ cu_seqlens_q_fake, ++ partial_o_fake, ++ partial_stats_fake, ++ attention_sinks_fake, ++ Float32(1.0), ++ Float32(1.0), ++ direct_q1_fake, ++ stream_fake, ++ cfg, ++ num_qo_heads, ++ num_kv_heads, ++ head_dim, ++ max_kv_len, ++ static_full_split_prefix, ++ options=_COMPILE_OPTIONS, ++ ), ++ sources=_aot_sources(), ++ tvm_ffi=True, + ) + + return compiled_main, compiled_reducer +@@ -4046,7 +4061,7 @@ class BatchDecodePagedTSWrapper: # previous complete plan revision usable. self._plan_state = candidate @@ -122,7 +436,7 @@ index 41479a386d72c27eb0df76107318dc09256aff06..3fcec5efdc2a3825be0dc1d9a238aaaa def run( self, q: torch.Tensor, -@@ -4228,7 +4225,7 @@ class BatchDecodePagedTSWrapper: +@@ -4228,7 +4243,7 @@ class BatchDecodePagedTSWrapper: ) @@ -131,8 +445,115 @@ index 41479a386d72c27eb0df76107318dc09256aff06..3fcec5efdc2a3825be0dc1d9a238aaaa def batch_decode_with_paged_kv_cache( q: torch.Tensor, paged_kv_cache: PagedKVCache, +@@ -4522,3 +4537,9 @@ __all__ = [ + "validate_q_token_kv_block_sparse_group_size", + "prepare_prims_ts_batch_decode_with_kv_cache", + ] ++ ++ ++def _aot_sources(): ++ """The Python files the PrimTS decode kernels are traced from: the ``sources`` of their shipped AOT objects.""" ++ return cute_aot.py_files(str(Path(__file__).resolve().parent)) ++ +diff --git a/kernels/fmha_context/fmha_resources.py b/kernels/fmha_context/fmha_resources.py +index c4d1b5512f9209793e42ec12bc269ac612c4703f..c21b2c9f5ab8d4f52d02024dcba15fdb2d16f080 100644 +--- a/kernels/fmha_context/fmha_resources.py ++++ b/kernels/fmha_context/fmha_resources.py +@@ -1200,8 +1200,6 @@ class GmemQKVResource(MemoryResource): + if cutlass.const_expr(self.cfg.has_uniform_varlen): + seqlen_k = Int32(self.cfg.uniform_seq_len_k) + else: +- from .helpers_paged import _load_runtime_seq_len_kv +- + seqlen_k = _load_runtime_seq_len_kv( + self.seqlens_kv, self.max_seq_len_kv, batch_coord + ) +@@ -1224,8 +1222,6 @@ class GmemQKVResource(MemoryResource): + kv_request_begin = batch_coord * Int32(self.block_table_row_stride) + kv_page_idx_ub = Int32(self.cfg.max_num_pages_per_seq_kv - 1) + else: +- from .helpers_paged import _load_block_table_row_bounds +- + kv_request_begin, kv_page_idx_ub = _load_block_table_row_bounds( + Int32(self.block_table_row_stride), + self.cfg, +@@ -1301,13 +1297,9 @@ class GmemQKVResource(MemoryResource): + kv_request_begin = batch_coord * Int32(self.block_table_row_stride) + kv_page_idx_ub = Int32(self.cfg.max_num_pages_per_seq_kv - 1) + else: +- from .helpers_paged import _load_runtime_seq_len_kv +- + cached_seqlen_kv = _load_runtime_seq_len_kv( + self.seqlens_kv, self.max_seq_len_kv, batch_coord + ) +- from .helpers_paged import _load_block_table_row_bounds +- + kv_request_begin, kv_page_idx_ub = _load_block_table_row_bounds( + Int32(self.block_table_row_stride), + self.cfg, +@@ -1744,8 +1736,6 @@ class SmemPageOffsetsKvResource(MemoryResource): + kv_request_begin: Int32, + kv_page_idx_ub: Int32, + ) -> None: +- from .helpers_paged import _resolve_kv_tile_idx_context +- + cfg = self.cfg + # Context's K/V tile index is kv_tile_start + loop_offset; for V the + # producer runs one tile ahead in TAIL, so reuse the same expression +@@ -1836,8 +1826,6 @@ class SmemPageOffsetsKvResource(MemoryResource): + tile_offset: cutlass.Constexpr[int] = 0, + ) -> cutlass.Array: + """Retain one tile's page IDs after its SMEM window is released.""" +- from .helpers_paged import _resolve_kv_tile_idx_context +- + tile_idx = _resolve_kv_tile_idx_context( + stage_info, kv_tile_start, tile_offset=tile_offset + ) +@@ -5798,3 +5786,14 @@ class GmemOResource(MemoryResource): + prims.cp_async_bulk_commit_group() + if cutlass.const_expr(self.cfg.gmem_o_store_wait_after_write): + prims.cp_async_bulk_wait_group(0, read=True) ++ ++ ++# helpers_paged imports FmhaConfig from this module, so it is imported here, ++# after FmhaConfig is defined. The methods above call these helpers while being ++# traced inside the kernel's runtime loops, where the CuTe DSL rejects a first ++# import of a module; importing at module scope keeps those calls import-free. ++from .helpers_paged import ( ++ _load_block_table_row_bounds, ++ _load_runtime_seq_len_kv, ++ _resolve_kv_tile_idx_context, ++) +diff --git a/kernels/fmha_decode/block_sparse_inspect.py b/kernels/fmha_decode/block_sparse_inspect.py +index 5728a7c6d373ddd694fa0c740dd01f672a65f648..e5f8de86571f54af313adcc0493bc65e4c9ce055 100644 +--- a/kernels/fmha_decode/block_sparse_inspect.py ++++ b/kernels/fmha_decode/block_sparse_inspect.py +@@ -26,6 +26,7 @@ Both publish one validation status plus the maximum row width in one Int64 + summary; no route payload is constructed. + """ + ++from ..._worktile_scope import with_scalar_work_tile_info + import functools + from collections.abc import Callable + +@@ -450,6 +451,7 @@ def _fake_compact( + + + @functools.cache ++@with_scalar_work_tile_info + def compile_block_sparse_inspection( + *, + device_index: int, +@@ -495,6 +497,7 @@ def compile_block_sparse_inspection( + + + @functools.cache ++@with_scalar_work_tile_info + def compile_paged_block_sparse_metadata_inspection( + *, + device_index: int, diff --git a/mla_decode.py b/mla_decode.py -index 41ba3b9055d229d227ac10cfc6f7d0f403eae69f..3d5fb159a3298d1edf72935b43d9a2eef74a1a4b 100644 +index 41ba3b9055d229d227ac10cfc6f7d0f403eae69f..0e82e5330728f78aa93709145382be5f2a015feb 100644 --- a/mla_decode.py +++ b/mla_decode.py @@ -1,3 +1,4 @@ @@ -140,7 +561,15 @@ index 41ba3b9055d229d227ac10cfc6f7d0f403eae69f..3d5fb159a3298d1edf72935b43d9a2ee # Copyright (c) 2026 by FlashInfer team. # # Licensed under the Apache License, Version 2.0 (the "License"); -@@ -21,10 +22,9 @@ from typing import Any, Literal, Optional, cast +@@ -14,6 +15,7 @@ + + """Task-scheduled paged MLA decode with a plan/run lifecycle.""" + ++from ._worktile_scope import with_scalar_work_tile_info + from collections.abc import Callable + from dataclasses import dataclass, field + import functools +@@ -21,10 +23,9 @@ from typing import Any, Literal, Optional, cast import torch @@ -152,7 +581,15 @@ index 41ba3b9055d229d227ac10cfc6f7d0f403eae69f..3d5fb159a3298d1edf72935b43d9a2ee ) from .decode import ( -@@ -1576,7 +1576,7 @@ class BatchMLADecodePagedTSWrapper: +@@ -1044,6 +1045,7 @@ def _make_mla_decode_compile_spec( + + + @functools.cache ++@with_scalar_work_tile_info + def _get_compiled_mla_decode( + compile_spec: _MLADecodeCompileSpec, + ): +@@ -1576,7 +1578,7 @@ class BatchMLADecodePagedTSWrapper: split_kv=int(dict(policy)["split_kv"]), ) diff --git a/3rdparty/vendor_sources.lock.yaml b/3rdparty/vendor_sources.lock.yaml index 2e914da8f052..e54ddc8f408d 100644 --- a/3rdparty/vendor_sources.lock.yaml +++ b/3rdparty/vendor_sources.lock.yaml @@ -9,8 +9,8 @@ vendors: include: - '**/*.py' patch: 3rdparty/vendor_patches/flashinfer-prims-ts.patch - patch_digest: sha256:65cc08e20a9cc7bdc190eb6f3e79124986dced74394c1f3250ae2d100a8ee340 - digest: sha256-tree-v1:e411b6c6679474d362b166e586df4276f52322d75a35c59c782d6266143e560e + patch_digest: sha256:e1f3f573c2723e432a8f4495c0f5329099b54f00856779996ac9b7f4ff0c21f7 + digest: sha256-tree-v1:b7be1c7ddbbcbd51dac753bdd9b0dd5cbeb5b122330a5c8d1366b79e7e635cba openengine-proto: url: https://github.com/ai-dynamo/openengine.git tag: v0.1.0 diff --git a/cpp/tensorrt_llm/kernels/CMakeLists.txt b/cpp/tensorrt_llm/kernels/CMakeLists.txt index 54a477030e66..86bdc01f11f6 100644 --- a/cpp/tensorrt_llm/kernels/CMakeLists.txt +++ b/cpp/tensorrt_llm/kernels/CMakeLists.txt @@ -81,6 +81,9 @@ list(FILTER SRC_CU EXCLUDE REGEX "marlin/.*") # Scope tcgen05 attention sources to their supported architectures. list(FILTER SRC_CU EXCLUDE REGEX "minimaxM3/(fusedDecodeAttn|attnMega|attnC1)\\.cu$") +# The MiniMax-M3 sm_107 decode kernels are built as their own sm_107-only OBJECT +# library below. +list(FILTER SRC_CU EXCLUDE REGEX "minimaxM3Sm107Tp[248]/.*") if(NOT ENABLE_MULTI_DEVICE) list(FILTER SRC_CU EXCLUDE REGEX "customAllReduceKernels*.*cu$") @@ -121,6 +124,7 @@ set_source_files_properties( ${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/toppRenormDetSm103.cu ${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/toppRenormDetV3Sm103.cu ${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/softmaxMcSm103.cu + ${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/softmaxPathBSm107.cu PROPERTIES COMPILE_OPTIONS "$<$:--use_fast_math>") # Preserve the historical architecture-scoped attention object library. @@ -131,11 +135,43 @@ set_property(TARGET minimax_m3_attn_src PROPERTY POSITION_INDEPENDENT_CODE ON) set_property(TARGET minimax_m3_attn_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON) set_cuda_architectures(minimax_m3_attn_src 100 103) +# MiniMax-M3 sm_107 decode kernels (minimaxM3Sm107Tp{2,4,8}/): written for +# sm_107 only (PDL griddepcontrol, sm_10x-only PTX), so they are compiled for +# the sm_10x entries of this build's architectures only. A build without an +# sm_10x architecture compiles them for sm_107a: their torch ops link against +# the host launchers, and only the sm_107 targets launch them. +file(GLOB MINIMAX_M3_SM107_SRC minimaxM3Sm107Tp2/*.cu minimaxM3Sm107Tp4/*.cu + minimaxM3Sm107Tp8/*.cu) +add_library(minimax_m3_sm107_src OBJECT ${MINIMAX_M3_SM107_SRC}) +set_property(TARGET minimax_m3_sm107_src PROPERTY POSITION_INDEPENDENT_CODE ON) +set_property(TARGET minimax_m3_sm107_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS + ON) +target_include_directories( + minimax_m3_sm107_src + PRIVATE + $ +) +target_link_libraries(minimax_m3_sm107_src PRIVATE trtllm_gen_fmha_interface) +set(MINIMAX_M3_SM107_ARCHS ${CMAKE_CUDA_ARCHITECTURES}) +list(FILTER MINIMAX_M3_SM107_ARCHS INCLUDE REGEX "^10[0-9]") +if(NOT MINIMAX_M3_SM107_ARCHS) + set(MINIMAX_M3_SM107_ARCHS 107a-real) +endif() +# SM107 SASS next to the family build: a family (100f) cubin is scheduled for +# sm_100 and SM107 runs it as is; the LC2 / LC2-split kernels built for sm_107 +# are 3.6-3.9% faster, bit for bit. The driver prefers the exact-arch cubin. +if(NOT "107a-real" IN_LIST MINIMAX_M3_SM107_ARCHS AND NOT "107f-real" IN_LIST + MINIMAX_M3_SM107_ARCHS) + list(APPEND MINIMAX_M3_SM107_ARCHS 107f-real) +endif() +set_property(TARGET minimax_m3_sm107_src PROPERTY CUDA_ARCHITECTURES + ${MINIMAX_M3_SM107_ARCHS}) + add_library( kernels_src STATIC ${SRC_CPP} ${SRC_CU} ${MOE_KERNELS_SRC} $<$:$> - $) + $ $) set_property(TARGET kernels_src PROPERTY POSITION_INDEPENDENT_CODE ON) set_property(TARGET kernels_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON) target_include_directories( diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.cu b/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.cu index d12b765e5ecf..9ef178897f9f 100644 --- a/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.cu +++ b/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.cu @@ -520,36 +520,13 @@ void invokeMinimaxM3RouterGemv(__nv_bfloat16 const* x, int32_t ldx, int32_t m, i launchGemv<1>(x, ldx, m, k, w, e, out, qOut, sfOut, sfScale, pdl, stream); } -void invokeMinimaxM3RouterTc(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int32_t k, float const* w, float* out, - uint8_t* qOut, uint8_t* sfOut, float const* sfScale, bool pdl, cudaStream_t stream) +int minimaxM3RouterTcCluster(int32_t k) { - int64_t const M = m, K = k; - TLLM_CHECK_WITH_INFO( - M >= 1 && M <= kTcMaxM && K % (kTcSlices * 64) == 0, "unsupported router shape M=%d K=%d", m, k); - int const ks = K / kTcSlices; - auto launch_tc = [&](auto kernel, int cl) -> cudaError_t - { - int const kc = (kTcSlices / cl) * ks; - size_t const smem = static_cast(kTcExperts) * (kc + 4) * 4 + static_cast(M) * (kc + 8) * 2; - cudaLaunchConfig_t cfg = {}; - cfg.gridDim = dim3(8 * cl); - cfg.blockDim = dim3(256); - cfg.dynamicSmemBytes = smem; - cfg.stream = stream; - cudaLaunchAttribute attr[2]; - attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; - attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; - attr[1].id = cudaLaunchAttributeClusterDimension; - attr[1].val.clusterDim.x = cl; - attr[1].val.clusterDim.y = 1; - attr[1].val.clusterDim.z = 1; - cfg.attrs = attr; - cfg.numAttrs = 2; - return cudaLaunchKernelEx( - &cfg, kernel, x, ldx, w, out, static_cast(M), static_cast(K), qOut, sfOut, sfScale); - }; - // 16-CTA clusters (128 CTAs, one K slice each) when the device co-schedules all 8, else 8-CTA clusters. - static int const cluster = [&]() + // 16-CTA clusters (128 CTAs, one K slice each) when the device co-schedules all 8, else 8-CTA clusters; decided + // once per process (device-static, for the first K seen: MiniMax-M3 has one hidden size) and configures both + // kernels. + int const ks = k / kTcSlices; + static int const cluster = [ks]() { int device = 0, max_optin = 0; cudaGetDevice(&device); @@ -592,6 +569,38 @@ void invokeMinimaxM3RouterTc(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int } return clusters >= 8 ? 16 : 8; }(); + return cluster; +} + +void invokeMinimaxM3RouterTc(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int32_t k, float const* w, float* out, + uint8_t* qOut, uint8_t* sfOut, float const* sfScale, bool pdl, cudaStream_t stream) +{ + int64_t const M = m, K = k; + TLLM_CHECK_WITH_INFO( + M >= 1 && M <= kTcMaxM && K % (kTcSlices * 64) == 0, "unsupported router shape M=%d K=%d", m, k); + int const ks = K / kTcSlices; + auto launch_tc = [&](auto kernel, int cl) -> cudaError_t + { + int const kc = (kTcSlices / cl) * ks; + size_t const smem = static_cast(kTcExperts) * (kc + 4) * 4 + static_cast(M) * (kc + 8) * 2; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(8 * cl); + cfg.blockDim = dim3(256); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + attr[1].id = cudaLaunchAttributeClusterDimension; + attr[1].val.clusterDim.x = cl; + attr[1].val.clusterDim.y = 1; + attr[1].val.clusterDim.z = 1; + cfg.attrs = attr; + cfg.numAttrs = 2; + return cudaLaunchKernelEx( + &cfg, kernel, x, ldx, w, out, static_cast(M), static_cast(K), qOut, sfOut, sfScale); + }; + int const cluster = minimaxM3RouterTcCluster(k); cudaError_t const err = cluster == 16 ? launch_tc(m3_router_tc_kernel<16>, 16) : launch_tc(m3_router_tc_kernel<8>, 8); if (err != cudaSuccess) diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.h b/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.h index 6f4467883b24..6548cb8ca83d 100644 --- a/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.h +++ b/cpp/tensorrt_llm/kernels/minimaxM3/routerGemv.h @@ -41,6 +41,10 @@ void invokeMinimaxM3RouterGemv(__nv_bfloat16 const* x, int32_t ldx, int32_t m, i void invokeMinimaxM3RouterTc(__nv_bfloat16 const* x, int32_t ldx, int32_t m, int32_t k, float const* w, float* out, uint8_t* qOut, uint8_t* sfOut, float const* sfScale, bool pdl, cudaStream_t stream); +//! The cluster size `invokeMinimaxM3RouterTc` launches with (16 or 8): decided once per process from the device's +//! co-scheduling (device-static) and configures both kernels. SM107 callers use the router only when this is 16. +int minimaxM3RouterTcCluster(int32_t k); + } // namespace kernels TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu b/cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu index 4069da14faf0..035bc514e894 100644 --- a/cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu +++ b/cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu @@ -43,6 +43,14 @@ namespace specTail namespace { constexpr int kVec = 8; // bf16 per 16-byte access +// draft_input loads its constant norm weight row before the grid dependency wait (the same values, bit for bit). Taken +// in the sm_107 device code only, whose callers are the sm_107 MiniMax-M3.1 targets; every other architecture loads the +// row after the wait, with the normalization. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ == 1070) +constexpr bool kDraftInputPrewaitWeight = true; +#else +constexpr bool kDraftInputPrewaitWeight = false; +#endif __device__ __forceinline__ float block_sum(float v, float* red) { @@ -72,6 +80,20 @@ __global__ void __launch_bounds__(kThreads) __shared__ float red[kThreads / 32]; int const row = blockIdx.x; bool const is_hidden = blockIdx.y == 1; + // The norm weight row is constant (no grid of the step writes it): load it before the grid dependency wait, so + // after the wait only the dependent loads (the id / hidden row) remain ahead of the reduction. The same values and + // arithmetic. + __nv_bfloat16 const* const w = is_hidden ? w_hidden : w_embed; + uint4 wv[kIters]; + if constexpr (kDraftInputPrewaitWeight) + { +#pragma unroll + for (int it = 0; it < kIters; ++it) + { + int const c = (it * kThreads + threadIdx.x) * kVec; + wv[it] = c < H ? *reinterpret_cast(w + c) : make_uint4(0u, 0u, 0u, 0u); + } + } #if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) asm volatile("griddepcontrol.wait;" ::: "memory"); #endif @@ -112,7 +134,6 @@ __global__ void __launch_bounds__(kThreads) asm volatile("griddepcontrol.launch_dependents;"); #endif float const rs = rsqrtf(block_sum(ss, red) / H + (is_hidden ? eps_hidden : eps_embed)); - __nv_bfloat16 const* w = is_hidden ? w_hidden : w_embed; __nv_bfloat16* dst = out + static_cast(row) * 2 * H + (is_hidden ? H : 0); #pragma unroll for (int it = 0; it < kIters; ++it) @@ -120,8 +141,11 @@ __global__ void __launch_bounds__(kThreads) int const c = (it * kThreads + threadIdx.x) * kVec; if (c < H) { - uint4 const wr = *reinterpret_cast(w + c); - __nv_bfloat16 const* we = reinterpret_cast<__nv_bfloat16 const*>(&wr); + if constexpr (!kDraftInputPrewaitWeight) + { + wv[it] = *reinterpret_cast(w + c); + } + __nv_bfloat16 const* we = reinterpret_cast<__nv_bfloat16 const*>(&wv[it]); uint4 o; __nv_bfloat16* oe = reinterpret_cast<__nv_bfloat16*>(&o); #pragma unroll diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.cu b/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.cu index fa62f8357e4d..849f825da7ed 100644 --- a/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.cu +++ b/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.cu @@ -15,7 +15,7 @@ * limitations under the License. */ -// ST-01-REJ: the stock chainSpeculativeSamplingKernel<1024, 4> on ctasPerRow CTAs +// REJ_MC: the stock chainSpeculativeSamplingKernel<1024, 4> on ctasPerRow CTAs // per request, bitwise. v2 (bitwise v1 and stock, 24.8 vs 32.3 us at one request) // moves loads only, every add and compare in its v1 order: the acceptance loop's probabilities are loaded in parallel // first, the chain loop is unrolled, and the leader reads the per-tile values from shared memory. See @@ -29,7 +29,11 @@ #include #include +#include #include +#include +#include +#include #include @@ -64,6 +68,10 @@ struct RowWorkspace unsigned arrive; unsigned gen; int32_t pad[3]; + //! v3 only: the target-only variant of tileSum / tileLast (v3 computes both before it knows targetOnly; tileSum / + //! tileLast then hold the residual variant). + float tileSumTarget[kMaxTiles]; + int32_t tileLastTarget[kMaxTiles]; }; // --------------------------------------------------------------------------------------------------------------------- @@ -164,8 +172,21 @@ __device__ __forceinline__ void rowSync(RowWorkspace* w, int32_t G, F leader) __syncthreads(); } +// Request row's draft distribution at position i: params.draftProbs[row, i, :] of a contiguous [batch, numDraft, vocab] +// buffer (rows.rowIds null), or, slot-indexed, rows.rowIds[row] * rows.rowStride + i * rows.posStride into a larger +// [slots, maxDraft, >= vocab] buffer (the gather the caller would otherwise copy out first; the same floats). +__device__ __forceinline__ float const* draftRow(ChainSpeculativeSamplingParams const& params, + ChainSpecMcDraftRows const& rows, int64_t row, int32_t i, int32_t numDraft, uint32_t vocabSize) +{ + if (rows.rowIds == nullptr) + { + return params.draftProbs + (row * numDraft + i) * static_cast(vocabSize); + } + return params.draftProbs + rows.rowIds[row] * rows.rowStride + i * rows.posStride; +} + __global__ void __launch_bounds__(kBlockSize, 1) - chainSpecSamplingMcKernel(ChainSpeculativeSamplingParams params, RowWorkspace* ws) + chainSpecSamplingMcKernel(ChainSpeculativeSamplingParams params, RowWorkspace* ws, ChainSpecMcDraftRows rows) { using BlockReduce = cub::BlockReduce; using BlockReducePair = cub::BlockReduce; @@ -202,7 +223,7 @@ __global__ void __launch_bounds__(kBlockSize, 1) { int32_t const token = params.draftTokenIds[row * numDraft + tid]; sPTarget[tid] = params.targetProbs[(row * (numDraft + 1) + tid) * vocabSize + token]; - sPDraft[tid] = params.draftProbs[(row * numDraft + tid) * vocabSize + token]; + sPDraft[tid] = draftRow(params, rows, row, tid, numDraft, vocabSize)[token]; } __syncthreads(); if (tid == 0) @@ -229,7 +250,7 @@ __global__ void __launch_bounds__(kBlockSize, 1) int32_t const pos = sPos; float const* target = params.targetProbs + (row * (numDraft + 1) + pos) * vocabSize; - float const* draft = pos < numDraft ? params.draftProbs + (row * numDraft + pos) * vocabSize : nullptr; + float const* draft = pos < numDraft ? draftRow(params, rows, row, pos, numDraft, vocabSize) : nullptr; // 2. The stock per-thread (target, residual) chains, spread over the request's CTAs; one CTA reduces them. int32_t const chainsPerCta = kBlockSize / G; @@ -384,10 +405,426 @@ __global__ void __launch_bounds__(kBlockSize, 1) }); } +// --------------------------------------------------------------------------------------------------------------------- +// v3 (VR). The v2 stamps put the chains at 10.4 of 21.3 us per call at one +// request: only 1024 / G threads of each CTA walk them, each through ~49 dependent loads. v3 keeps every add, compare +// and their order (v2's and the stock kernel's) and moves only loads and the cross-CTA schedule: +// A. all 1024 threads stage the CTA's chain elements in shared memory (chunkK 4096-strides at a time), then the chain +// threads run the stock chain loop over them; +// B. every CTA computes both stock tile variants (target only, residual) before the row barrier, so the barrier's +// leader runs the final search directly: one barrier per call, and the other CTAs exit on arrival; +// C. the leader's tile prefix runs unrolled from shared memory (same adds, same first-crossing exit), the fallback's +// last index with mass is an integer max (order-free), and numDraft + 1 threads store the outputs from the draft +// ids loaded in phase 1. +// --------------------------------------------------------------------------------------------------------------------- + +//! Staged chains: float4 slots per probability row (target, draft) in dynamic shared memory, 64 KiB each. +constexpr int32_t kStageVecsPerRow = 4096; +//! Tiles a CTA holds in registers at once while it sums both tile variants. +constexpr int32_t kTileRegs = 2; + +//! The row barrier without a release: the last CTA to arrive runs leader() (all its threads) and resets the barrier +//! (arrival count, generation; the same state rowSync leaves, so v2 and v3 launches may share a workspace); the other +//! CTAs return at once. +template +__device__ __forceinline__ void rowArrive(RowWorkspace* w, int32_t G, F leader) +{ + __shared__ unsigned sGen; + __shared__ int32_t sLast; + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + sGen = loadAcquire(&w->gen); + sLast = atomicAdd(&w->arrive, 1u) == static_cast(G - 1); + } + __syncthreads(); + if (!sLast) + { + return; + } + __threadfence(); + leader(); + __syncthreads(); + if (threadIdx.x == 0) + { + atomicExch(&w->arrive, 0u); + cuda::atomic_ref(w->gen).store(sGen + 1, cuda::memory_order_release); + } +} + +__global__ void __launch_bounds__(kBlockSize, 1) chainSpecSamplingMcKernelV3( + ChainSpeculativeSamplingParams params, RowWorkspace* ws, ChainSpecMcDraftRows rows, int32_t chunkK) +{ + using BlockReduce = cub::BlockReduce; + using BlockReducePair = cub::BlockReduce; + using BlockScan = cub::BlockScan; + + __shared__ union + { + typename BlockReduce::TempStorage reduce; + typename BlockReducePair::TempStorage reducePair; + typename BlockScan::TempStorage scan; + } tempStorage; + + //! [chunkK x chainsPerCta] float4 of the target, then the same of the draft. + extern __shared__ float4 sStage[]; + + __shared__ int32_t sPos; + __shared__ float sU; + __shared__ int32_t sTileLastOfTile; + __shared__ int32_t sDraftToken[kChainSpecMcMaxDraft]; + + int64_t const row = blockIdx.y; + int32_t const G = static_cast(gridDim.x); + int32_t const cta = static_cast(blockIdx.x); + int32_t const tid = static_cast(threadIdx.x); + int32_t const numDraft = params.numDraftTokens; + uint32_t const vocabSize = static_cast(params.vocabSize); + RowWorkspace* w = ws + row; + int32_t* outTokens = params.outputTokenIds + row * (numDraft + 1); + + // 1. Acceptance: v2's (the stock tid-0 loop over probabilities loaded in parallel); the draft ids stay in shared + // memory for the outputs. + __shared__ float sPTarget[kChainSpecMcMaxDraft]; + __shared__ float sPDraft[kChainSpecMcMaxDraft]; + if (tid < numDraft) + { + int32_t const token = params.draftTokenIds[row * numDraft + tid]; + sDraftToken[tid] = token; + sPTarget[tid] = params.targetProbs[(row * (numDraft + 1) + tid) * vocabSize + token]; + sPDraft[tid] = draftRow(params, rows, row, tid, numDraft, vocabSize)[token]; + } + __syncthreads(); + if (tid == 0) + { + curandStatePhilox4_32_10_t state; + curand_init(static_cast(params.seed[row]), + kPerRowSubsequenceBase | static_cast(params.offset[row]), 0, &state); + int32_t pos = numDraft; + for (int32_t i = 0; i < numDraft; ++i) + { + float const pTarget = sPTarget[i]; + float const pDraft = sPDraft[i]; + if (!(curand_uniform(&state) * pDraft < pTarget)) + { + pos = i; + break; + } + } + sPos = pos; + // curand_uniform is in (0, 1]; the inverse-CDF search below wants [0, 1). + sU = 1.0f - curand_uniform(&state); + } + __syncthreads(); + + int32_t const pos = sPos; + float const* target = params.targetProbs + (row * (numDraft + 1) + pos) * vocabSize; + float const* draft = pos < numDraft ? draftRow(params, rows, row, pos, numDraft, vocabSize) : nullptr; + int32_t const chainsPerCta = kBlockSize / G; + int32_t const numTiles = static_cast((vocabSize + kTileSize - 1) / kTileSize); + + // B (loads): the CTA's first kTileRegs tiles (tiles cta, cta + G, ...), issued with A's staged loads. + float tileTarget[kTileRegs][kVecSize]; + float tileDraft[kTileRegs][kVecSize]; + auto loadTiles = [&](int32_t tile0) + { +#pragma unroll + for (int32_t r = 0; r < kTileRegs; ++r) + { + int32_t const tile = tile0 + r * G; +#pragma unroll + for (int32_t j = 0; j < kVecSize; ++j) + { + tileTarget[r][j] = 0.0f; + tileDraft[r][j] = 0.0f; + } + if (tile < numTiles) + { + uint32_t const base = tile * kTileSize + tid * kVecSize; + loadProbs(target, base, vocabSize, tileTarget[r]); + if (draft != nullptr) + { + loadProbs(draft, base, vocabSize, tileDraft[r]); + } + } + } + }; + loadTiles(cta); + + // 2 (A). The stock chains: chain t adds the elements at 4t + 4096k in k order (x: target mass, y: residual mass). + // The CTA's chains are t = cta * chainsPerCta + c; all threads stage their vec4s, chunkK strides at a time, and + // thread c < chainsPerCta adds them in the stock order (a stride past the vocab ends the chain, as the stock + // loop's bound; the vocab is a multiple of 4, so a staged vec4 is whole or absent). + float4* const sTarget = sStage; + float4* const sDraft = sStage + chunkK * chainsPerCta; + int32_t const numK = numTiles; + float2 sums = make_float2(0.0f, 0.0f); + for (int32_t k0 = 0; k0 < numK; k0 += chunkK) + { + int32_t const kc = min(chunkK, numK - k0); + for (int32_t i = tid; i < kc * chainsPerCta; i += kBlockSize) + { + int32_t const k = k0 + i / chainsPerCta; + int32_t const t = cta * chainsPerCta + i % chainsPerCta; + uint32_t const base = static_cast(t) * kVecSize + static_cast(k) * kTileSize; + float4 tv = make_float4(0.0f, 0.0f, 0.0f, 0.0f); + float4 dv = make_float4(0.0f, 0.0f, 0.0f, 0.0f); + if (base < vocabSize) + { + tv = *reinterpret_cast(target + base); + if (draft != nullptr) + { + dv = *reinterpret_cast(draft + base); + } + } + sTarget[i] = tv; + sDraft[i] = dv; + } + __syncthreads(); + if (tid < chainsPerCta) + { + uint32_t const t = static_cast(cta * chainsPerCta + tid); + for (int32_t kk = 0; kk < kc; ++kk) + { + if (t * kVecSize + static_cast(k0 + kk) * kTileSize >= vocabSize) + { + break; + } + float4 const a = sTarget[kk * chainsPerCta + tid]; + float tv[kVecSize] = {a.x, a.y, a.z, a.w}; + float dv[kVecSize] = {}; + if (draft != nullptr) + { + float4 const b = sDraft[kk * chainsPerCta + tid]; + dv[0] = b.x; + dv[1] = b.y; + dv[2] = b.z; + dv[3] = b.w; + } +#pragma unroll + for (int32_t j = 0; j < kVecSize; ++j) + { + sums.x += tv[j]; + sums.y += fmaxf(tv[j] - dv[j], 0.0f); + } + } + } + __syncthreads(); + } + if (tid < chainsPerCta) + { + w->chain[cta * chainsPerCta + tid] = sums; + } + + // 3 (B). Both stock tile variants of the CTA's tiles: the stock per-tile cub reduce and last index with mass over + // the + // target (targetOnly) and over max(0, target - draft) (the residual; only when there is a draft row). + auto tileSums = [&](int32_t tile, float(&value)[kVecSize], float* sumOut, int32_t* lastOut) + { + if (tid == 0) + { + sTileLastOfTile = -1; + } + __syncthreads(); + uint32_t const base = tile * kTileSize + tid * kVecSize; + int32_t localLast = -1; +#pragma unroll + for (int32_t j = 0; j < kVecSize; ++j) + { + if (value[j] > 0.0f) + { + localLast = static_cast(base + j); + } + } + if (localLast >= 0) + { + atomicMax(&sTileLastOfTile, localLast); + } + float const tileSum = BlockReduce(tempStorage.reduce).Sum(value); + __syncthreads(); + if (tid == 0) + { + sumOut[tile] = tileSum; + lastOut[tile] = sTileLastOfTile; + } + __syncthreads(); + }; + for (int32_t tile0 = cta; tile0 < numTiles; tile0 += kTileRegs * G) + { + if (tile0 != cta) + { + loadTiles(tile0); + } +#pragma unroll + for (int32_t r = 0; r < kTileRegs; ++r) + { + int32_t const tile = tile0 + r * G; + if (tile >= numTiles) + { + break; + } + tileSums(tile, tileTarget[r], w->tileSumTarget, w->tileLastTarget); + if (draft != nullptr) + { + float residual[kVecSize]; +#pragma unroll + for (int32_t j = 0; j < kVecSize; ++j) + { + residual[j] = fmaxf(tileTarget[r][j] - tileDraft[r][j], 0.0f); + } + tileSums(tile, residual, w->tileSum, w->tileLast); + } + } + } + + // 4. The last CTA to arrive: the chains' reduce, the tile prefix in tile order, the stock in-tile scan, the + // outputs. + rowArrive(w, G, + [&] + { + __shared__ int32_t sTargetOnly; + __shared__ float sTotal; + __shared__ float sTileSum[kMaxTiles]; + __shared__ int32_t sTileLastOf[kMaxTiles]; + __shared__ float sPrefix; + __shared__ int32_t sTile; + __shared__ int32_t sWinner; + __shared__ int32_t sLastValid; + __shared__ int32_t sToken; + // Both variants' per-tile values, loaded with the chains for the reduce. + float sumTarget = 0.0f; + float sumResidual = 0.0f; + int32_t lastTarget = -1; + int32_t lastResidual = -1; + if (tid < numTiles) + { + sumTarget = __ldcg(&w->tileSumTarget[tid]); + lastTarget = __ldcg(&w->tileLastTarget[tid]); + if (draft != nullptr) + { + sumResidual = __ldcg(&w->tileSum[tid]); + lastResidual = __ldcg(&w->tileLast[tid]); + } + } + float2 const blockSums + = BlockReducePair(tempStorage.reducePair).Reduce(__ldcg(&w->chain[tid]), Float2SumOp{}); + if (tid == 0) + { + int32_t const targetOnly = draft == nullptr || !(blockSums.y > 0.0f); + sTargetOnly = targetOnly; + sTotal = targetOnly ? blockSums.x : blockSums.y; + sWinner = static_cast(vocabSize); + sLastValid = -1; + } + __syncthreads(); + bool const targetOnly = sTargetOnly != 0; + if (tid < numTiles) + { + sTileSum[tid] = targetOnly ? sumTarget : sumResidual; + sTileLastOf[tid] = targetOnly ? lastTarget : lastResidual; + } + __syncthreads(); + float const threshold = sU * sTotal; + if (tid == 0) + { + // The stock walk: the first tile whose prefix + sum crosses the threshold, prefix summed in tile order. + float prefix = 0.0f; + int32_t found = -1; +#pragma unroll + for (int32_t tile = 0; tile < kMaxTiles; ++tile) + { + if (tile < numTiles && found < 0) + { + float const tileSum = sTileSum[tile]; + if (prefix + tileSum > threshold) + { + found = tile; + } + else + { + prefix += tileSum; + } + } + } + sTile = found; + sPrefix = prefix; + } + __syncthreads(); + // The fallback's last index with mass over the tiles the stock loop visits (a max, in any order). + int32_t const upto = sTile >= 0 ? sTile : numTiles - 1; + if (tid <= upto && sTileLastOf[tid] >= 0) + { + atomicMax(&sLastValid, sTileLastOf[tid]); + } + if (sTile >= 0) + { + uint32_t const base = sTile * kTileSize + tid * kVecSize; + float value[kVecSize]; + loadResidual(target, draft, targetOnly, base, vocabSize, value); + float inclusive[kVecSize]; + BlockScan(tempStorage.scan).InclusiveSum(value, inclusive); + float const prefix = sPrefix; +#pragma unroll + for (int32_t j = 0; j < kVecSize; ++j) + { + if (value[j] > 0.0f && prefix + inclusive[j] > threshold) + { + atomicMin(&sWinner, static_cast(base + j)); + break; + } + } + } + __syncthreads(); + if (tid == 0) + { + sToken = sWinner < static_cast(vocabSize) ? sWinner : max(sLastValid, 0); + params.outputNumTokens[row] = pos + 1; + } + __syncthreads(); + if (tid <= numDraft) + { + outTokens[tid] = tid < pos ? sDraftToken[tid] : (tid == pos ? sToken : -1); + } + }); +} + +//! REJ v3 on sm_107 devices unless TRTLLM_M31_DISABLE_REJ_V3=1 (the v2 kernel); other devices launch v2. The switch +//! is read once per process and the compute capability once per device. +bool rejV3Enabled() +{ + static bool const allowed = [] + { + char const* env = std::getenv("TRTLLM_M31_DISABLE_REJ_V3"); + return env == nullptr || std::string(env) != "1"; + }(); + if (!allowed) + { + return false; + } + static std::once_flag smOnce[64]; + static bool sm107[64] = {}; + int device = 0; + if (cudaGetDevice(&device) != cudaSuccess) + { + return false; + } + std::call_once(smOnce[device % 64], + [device] + { + int major = 0; + int minor = 0; + cudaDeviceGetAttribute(&major, cudaDevAttrComputeCapabilityMajor, device); + cudaDeviceGetAttribute(&minor, cudaDevAttrComputeCapabilityMinor, device); + sm107[device % 64] = major == 10 && minor == 7; + }); + return sm107[device % 64]; +} + } // namespace -cudaError_t invokeChainSpecSamplingMc( - ChainSpeculativeSamplingParams const& params, int ctasPerRow, void* workspace, cudaStream_t stream) +cudaError_t invokeChainSpecSamplingMc(ChainSpeculativeSamplingParams const& params, int ctasPerRow, void* workspace, + cudaStream_t stream, ChainSpecMcDraftRows const& rows) { if (params.batchSize <= 0) { @@ -395,7 +832,9 @@ cudaError_t invokeChainSpecSamplingMc( } if (ctasPerRow < 1 || ctasPerRow > kBlockSize || kBlockSize % ctasPerRow != 0 || params.vocabSize <= 0 || params.vocabSize % kVecSize != 0 || params.vocabSize > kChainSpecMcMaxVocab || params.numDraftTokens < 0 - || params.numDraftTokens > kChainSpecMcMaxDraft || params.batchSize > 65535) + || params.numDraftTokens > kChainSpecMcMaxDraft || params.batchSize > 65535 + || (rows.rowIds != nullptr + && (rows.posStride < params.vocabSize || rows.rowStride < rows.posStride * params.numDraftTokens))) { return cudaErrorInvalidValue; } @@ -404,7 +843,36 @@ cudaError_t invokeChainSpecSamplingMc( cfg.blockDim = dim3(kBlockSize); cfg.dynamicSmemBytes = 0; cfg.stream = stream; - return cudaLaunchKernelEx(&cfg, chainSpecSamplingMcKernel, params, static_cast(workspace)); + if (!rejV3Enabled()) + { + return cudaLaunchKernelEx(&cfg, chainSpecSamplingMcKernel, params, static_cast(workspace), rows); + } + // v3 stages chunkK 4096-strides of the CTA's chains per pass (all of them at 16+ CTAs per request). + int32_t const chainsPerCta = kBlockSize / ctasPerRow; + int32_t const numK = (params.vocabSize + kTileSize - 1) / kTileSize; + int32_t const chunkK = std::min(numK, std::max(1, kStageVecsPerRow / chainsPerCta)); + constexpr int kMaxStageBytes = 2 * kStageVecsPerRow * static_cast(sizeof(float4)); + // The attribute is per device; set it on each device's first launch. + static std::once_flag smemOnce[64]; + int device = 0; + if (cudaError_t const err = cudaGetDevice(&device); err != cudaSuccess) + { + return err; + } + cudaError_t attrErr = cudaSuccess; + std::call_once(smemOnce[device % 64], + [&] + { + attrErr = cudaFuncSetAttribute( + chainSpecSamplingMcKernelV3, cudaFuncAttributeMaxDynamicSharedMemorySize, kMaxStageBytes); + }); + if (attrErr != cudaSuccess) + { + return attrErr; + } + cfg.dynamicSmemBytes = 2 * static_cast(chunkK) * chainsPerCta * sizeof(float4); + return cudaLaunchKernelEx( + &cfg, chainSpecSamplingMcKernelV3, params, static_cast(workspace), rows, chunkK); } size_t chainSpecSamplingMcWorkspaceBytesPerRow() @@ -412,6 +880,11 @@ size_t chainSpecSamplingMcWorkspaceBytesPerRow() return sizeof(RowWorkspace); } +int chainSpecSamplingMcKernelVersion() +{ + return rejV3Enabled() ? 3 : 2; +} + } // namespace minimaxM31 } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h b/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h index 33a37dc071c7..1db29049a91b 100644 --- a/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h +++ b/cpp/tensorrt_llm/kernels/minimaxM31/chainSpecSamplingMcSm103.h @@ -58,7 +58,7 @@ constexpr int kChainSpecMcMaxDraft = 32; //! the last index with mass over the tiles the stock loop visits. //! Validated bitwise (tokens + counts) vs trtllm::chain_speculative_sampling on GB300, K7, vocab 200064, batch //! 1 / 2 / 4 / 8 x 5 cases (realistic, same, far, ties, onehot) x 64 seeds at ctasPerRow 4 / 8 / 16, eager and graph -//! (microbenchmark): 31.2 vs 41.2 us per call at batch 1 with ctasPerRow 16. +//! 31.2 vs 41.2 us per call at batch 1 with ctasPerRow 16. //! //! Compiled with the repo's default flags, as the stock kernel (only adds, max and compares on the probs; no //! --use_fast_math). @@ -72,12 +72,27 @@ constexpr int kChainSpecMcMaxDraft = 32; //! the first launch and then reused as is: every launch leaves it ready for the next (arrival counters reset, the //! barrier generation only increases), so it is CUDA-graph safe without an init kernel. Launches that share one //! workspace must be stream-ordered. +//! Slot-indexed draft rows (VR REJ_SLOTS): with rowIds set, request row r's draft distribution at position i is read at +//! draftProbs + rowIds[r] * rowStride + i * posStride (a [slots, maxDraft, >= vocab] buffer) instead of the contiguous +//! [batch, numDraft, vocab] layout; the same floats as the caller's gather, without the copy. rowIds null (the default) +//! keeps the contiguous layout. +struct ChainSpecMcDraftRows +{ + int64_t const* rowIds = nullptr; + int64_t rowStride = 0; + int64_t posStride = 0; +}; + cudaError_t invokeChainSpecSamplingMc(speculative_decoding::ChainSpeculativeSamplingParams const& params, - int ctasPerRow, void* workspace, cudaStream_t stream); + int ctasPerRow, void* workspace, cudaStream_t stream, ChainSpecMcDraftRows const& rows = ChainSpecMcDraftRows{}); //! Workspace bytes per request of invokeChainSpecSamplingMc. size_t chainSpecSamplingMcWorkspaceBytesPerRow(); +//! The kernel invokeChainSpecSamplingMc launches: 3 (REJ v3: staged chains, both tile variants before one row barrier, +//! the same adds and order; the default) or 2 (the v2 kernel, with TRTLLM_M31_DISABLE_REJ_V3=1, read once per process). +int chainSpecSamplingMcKernelVersion(); + } // namespace minimaxM31 } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.cu new file mode 100644 index 000000000000..b5c701c5735e --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.cu @@ -0,0 +1,209 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// DSPARK_GLUE_CPP: C++ / CUDA replacements of two DSpark drafter Triton kernels (see dsparkGlueSm107.h for the exact +// contracts). The kernel bodies are the ones gated in SpecTail's DGLUE oracle / MB (857722 / 857723: bitwise vs the +// Triton kernels on every output; _advance 15.9 -> 7.1 us at the in-model block-table width). + +#include "tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +namespace +{ + +constexpr int kAdvThreads = 1024; +constexpr int kAdvUnroll = 8; +constexpr int kIdxThreads = 256; + +template +__global__ void __launch_bounds__(kAdvThreads) dsparkAdvanceKernel(int64_t const* __restrict__ slots, + int64_t const* __restrict__ rows, NaccT const* __restrict__ nacc, int64_t* __restrict__ ctxLen, + int64_t const* __restrict__ counts, int64_t const* __restrict__ basePages, int32_t const* __restrict__ tables, + int32_t* __restrict__ readView, int64_t* __restrict__ out, int64_t width, int64_t tablesStride, int64_t readStride, + int64_t maxCtx, int64_t dummySlot, int64_t page, int64_t blockSize) +{ + __shared__ int64_t sRebase; + __shared__ int64_t sRow; + int64_t const r = blockIdx.x; + if (threadIdx.x == 0) + { + int64_t const slot = slots[r]; + int64_t const row = rows[r]; + bool const isDummy = slot == dummySlot; + int64_t newLen = min(ctxLen[slot] + static_cast(nacc[r]), maxCtx); + if (!isDummy) + { + ctxLen[slot] = newLen; + } + if (r == 0) + { + ctxLen[dummySlot] = 0; + } + newLen = isDummy ? 0 : newLen; + int64_t const allocated = max(counts[row] * page - blockSize, int64_t{0}); + int64_t const numCtx = min(newLen, allocated); + int64_t const base = basePages[slot]; + sRebase = min(base, numCtx / page); + out[r] = max(numCtx - base * page, int64_t{0}); + sRow = row; + } + __syncthreads(); + int64_t const rebase = sRebase; + int32_t const* src = tables + sRow * tablesStride; + int32_t* dst = readView + sRow * readStride; + int64_t const last = width - 1; + int64_t j = threadIdx.x; + for (; j + (kAdvUnroll - 1) * kAdvThreads < width; j += kAdvUnroll * kAdvThreads) + { + int32_t v[kAdvUnroll]; +#pragma unroll + for (int u = 0; u < kAdvUnroll; ++u) + { + v[u] = __ldg(src + min(j + u * kAdvThreads + rebase, last)); + } +#pragma unroll + for (int u = 0; u < kAdvUnroll; ++u) + { + dst[j + u * kAdvThreads] = v[u]; + } + } + for (; j < width; j += kAdvThreads) + { + dst[j] = __ldg(src + min(j + rebase, last)); + } +} + +template +__global__ void __launch_bounds__(kIdxThreads) dsparkIndicesKernel(CtxT const* __restrict__ numCtx, + BidT const* __restrict__ batchIdx, QT const* __restrict__ qpos, int64_t q0, int64_t q1, + int32_t* __restrict__ seqlens, int32_t* __restrict__ batchOut, int32_t* __restrict__ after, + int32_t* __restrict__ indptr, int32_t* __restrict__ last, int32_t* __restrict__ append, int32_t* __restrict__ qflat, + int32_t B, int32_t block, int32_t pagesPerSlot, int32_t pageSize) +{ + for (int32_t b = threadIdx.x; b <= B; b += blockDim.x) + { + indptr[b] = b * pagesPerSlot; + if (b < B) + { + int32_t const s = static_cast(numCtx[b]); + seqlens[b] = s; + batchOut[b] = static_cast(batchIdx[b]); + int32_t const a = s + block; + after[b] = a; + int32_t m = (a - 1) % pageSize; + m = m < 0 ? m + pageSize : m; // torch's floored modulo for a positive divisor + last[b] = m + 1; + } + } + for (int32_t t = threadIdx.x; t < B * block; t += blockDim.x) + { + int32_t const rb = t / block; + int32_t const j = t - rb * block; + append[t] = static_cast(numCtx[rb]) + j; + qflat[t] = static_cast(qpos[rb * q0 + j * q1]); + } +} + +template +void launchIndices(CtxT const* numCtx, BidT const* batchIdx, void const* qpos, bool qposIs64, int64_t q0, int64_t q1, + int32_t* seqlens, int32_t* batchOut, int32_t* after, int32_t* indptr, int32_t* last, int32_t* append, + int32_t* qflat, int32_t B, int32_t block, int32_t pagesPerSlot, int32_t pageSize, cudaStream_t stream) +{ + if (qposIs64) + { + dsparkIndicesKernel<<<1, kIdxThreads, 0, stream>>>(numCtx, batchIdx, + static_cast(qpos), q0, q1, seqlens, batchOut, after, indptr, last, append, qflat, B, block, + pagesPerSlot, pageSize); + } + else + { + dsparkIndicesKernel<<<1, kIdxThreads, 0, stream>>>(numCtx, batchIdx, + static_cast(qpos), q0, q1, seqlens, batchOut, after, indptr, last, append, qflat, B, block, + pagesPerSlot, pageSize); + } +} + +template +void launchIndicesBid(CtxT const* numCtx, void const* batchIdx, bool batchIdxIs64, void const* qpos, bool qposIs64, + int64_t q0, int64_t q1, int32_t* seqlens, int32_t* batchOut, int32_t* after, int32_t* indptr, int32_t* last, + int32_t* append, int32_t* qflat, int32_t B, int32_t block, int32_t pagesPerSlot, int32_t pageSize, + cudaStream_t stream) +{ + if (batchIdxIs64) + { + launchIndices(numCtx, static_cast(batchIdx), qpos, qposIs64, q0, q1, seqlens, batchOut, after, + indptr, last, append, qflat, B, block, pagesPerSlot, pageSize, stream); + } + else + { + launchIndices(numCtx, static_cast(batchIdx), qpos, qposIs64, q0, q1, seqlens, batchOut, after, + indptr, last, append, qflat, B, block, pagesPerSlot, pageSize, stream); + } +} + +} // namespace + +void invokeDsparkAdvance(int64_t const* slots, int64_t const* rows, void const* nacc, bool naccIs64, int64_t* ctxLen, + int64_t const* counts, int64_t const* basePages, int32_t const* tables, int32_t* readView, int64_t* out, int64_t n, + int64_t width, int64_t tablesStride, int64_t readStride, int64_t maxCtx, int64_t dummySlot, int64_t page, + int64_t blockSize, cudaStream_t stream) +{ + if (n <= 0) + { + return; + } + if (naccIs64) + { + dsparkAdvanceKernel<<(n), kAdvThreads, 0, stream>>>(slots, rows, + static_cast(nacc), ctxLen, counts, basePages, tables, readView, out, width, tablesStride, + readStride, maxCtx, dummySlot, page, blockSize); + } + else + { + dsparkAdvanceKernel<<(n), kAdvThreads, 0, stream>>>(slots, rows, + static_cast(nacc), ctxLen, counts, basePages, tables, readView, out, width, tablesStride, + readStride, maxCtx, dummySlot, page, blockSize); + } +} + +void invokeDsparkIndices(void const* numCtx, bool numCtxIs64, void const* batchIdx, bool batchIdxIs64, void const* qpos, + bool qposIs64, int64_t q0, int64_t q1, int32_t* seqlens, int32_t* batchOut, int32_t* after, int32_t* indptr, + int32_t* last, int32_t* append, int32_t* qflat, int32_t B, int32_t block, int32_t pagesPerSlot, int32_t pageSize, + cudaStream_t stream) +{ + if (numCtxIs64) + { + launchIndicesBid(static_cast(numCtx), batchIdx, batchIdxIs64, qpos, qposIs64, q0, q1, seqlens, + batchOut, after, indptr, last, append, qflat, B, block, pagesPerSlot, pageSize, stream); + } + else + { + launchIndicesBid(static_cast(numCtx), batchIdx, batchIdxIs64, qpos, qposIs64, q0, q1, seqlens, + batchOut, after, indptr, last, append, qflat, B, block, pagesPerSlot, pageSize, stream); + } +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.h new file mode 100644 index 000000000000..37440206d4c5 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.h @@ -0,0 +1,59 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! DSPARK_GLUE_CPP: the DSpark / DFlash drafter's context advance + rebased read view, bit for bit the Triton +//! _advance_kernel (catalog/spec_decode/dspark_ctx_advance.py). For each generation request r (one block each, so the +//! in-place ctx_len update keeps the one-program-per-request ordering): +//! ctxLen[slot] = min(ctxLen[slot] + nacc[r], maxCtx) (not for the dummy slot); ctxLen[dummySlot] = 0 (request 0) +//! numCtx = min(new (0 on the dummy slot), max(counts[row] * page - blockSize, 0)) +//! rebase = min(basePages[slot], numCtx / page); out[r] = max(numCtx - basePages[slot] * page, 0) +//! readView[row, j] = tables[row, min(j + rebase, width - 1)] (int32) +//! nacc is int32 (naccIs64 false) or int64. Integer arithmetic and copies only. Graph-safe. +void invokeDsparkAdvance(int64_t const* slots, int64_t const* rows, void const* nacc, bool naccIs64, int64_t* ctxLen, + int64_t const* counts, int64_t const* basePages, int32_t const* tables, int32_t* readView, int64_t* out, int64_t n, + int64_t width, int64_t tablesStride, int64_t readStride, int64_t maxCtx, int64_t dummySlot, int64_t page, + int64_t blockSize, cudaStream_t stream); + +//! DSPARK_GLUE_CPP: the paged dflash_forward's index prelude in one launch, bit for bit the Triton _indices_kernel +//! (catalog/spec_decode/dspark_forward_indices.py), all outputs int32 (int64 -> int32 keeps the low 32 bits): +//! seqlens = numCtx; batchOut = batchIdx; after = seqlens + block; last = floor-mod(after - 1, pageSize) + 1; +//! indptr[b] = b * pagesPerSlot (b <= B); append[t] = seqlens[t / block] + t % block; +//! qflat[t] = qpos[t / block, t % block] (qpos strided by q0 / q1 elements). +//! numCtx / batchIdx / qpos are int32 or int64 (the *Is64 flags). Graph-safe. +void invokeDsparkIndices(void const* numCtx, bool numCtxIs64, void const* batchIdx, bool batchIdxIs64, void const* qpos, + bool qposIs64, int64_t q0, int64_t q1, int32_t* seqlens, int32_t* batchOut, int32_t* after, int32_t* indptr, + int32_t* last, int32_t* append, int32_t* qflat, int32_t B, int32_t block, int32_t pagesPerSlot, int32_t pageSize, + cudaStream_t stream); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.cu new file mode 100644 index 000000000000..7423110bfbd8 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.cu @@ -0,0 +1,467 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Device code copied from TRT-LLM c972d6f162 cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu +// (md5 546e8c75) and common/reduceKernelUtils.cuh: LamportComm, FusedOp, allreduce_sum +// (Fp32Acc), IndexHelper, blockReduceSumV2, with the push loop removed (the producer pre-pushed) and the MXFP8 output +// of the norm (quantization.cuh cvt_quant_get_sf_out_offset SWIZZLED + cvt_warp_fp16_to_mxfp8, +// as quantize_mxfp8_rows). Same device code as the shipped libm31_f3_ar.so. +#include "tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.h" +#include "tensorrt_llm/kernels/quantization.cuh" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +namespace +{ +namespace cg = cooperative_groups; +#define FINAL_MASK 0xffffffff + +namespace f3 +{ +using DType = __nv_bfloat16; +constexpr int NRanks = 2; +constexpr int kElemsPerAccess = 8; + +using Params = F3ArParams; + +template +__inline__ __device__ T warpReduceSumV2(T* val) +{ +#pragma unroll + for (int i = 0; i < NUM; i++) + { +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + val[i] += __shfl_xor_sync(FINAL_MASK, val[i], mask, 32); + } + return (T) (0.0f); +} + +template +__inline__ __device__ T blockReduceSumV2(T* val) +{ + static __shared__ T shared[NUM][33]; + int lane = threadIdx.x & 0x1f; + int wid = threadIdx.x >> 5; + + warpReduceSumV2(val); + + if (lane == 0) + { +#pragma unroll + for (int i = 0; i < NUM; i++) + { + shared[i][wid] = val[i]; + } + } + + __syncthreads(); + + bool is_mask = threadIdx.x < (blockDim.x / 32.f); +#pragma unroll + for (int i = 0; i < NUM; i++) + { + val[i] = is_mask ? shared[i][lane] : (T) (0.0f); + } + warpReduceSumV2(val); + return (T) 0.0f; +} + +struct LamportComm +{ + __device__ __forceinline__ LamportComm(void** workspace, int rank) + { + counter_ptr = &reinterpret_cast(workspace[NRanks * 3])[0]; + flag_ptr = &reinterpret_cast(workspace[NRanks * 3])[2]; + clear_ptr = &reinterpret_cast(workspace[NRanks * 3 + 1])[0]; + flag_value = *flag_ptr; + auto comm_size = reinterpret_cast(workspace[NRanks * 3 + 1])[1]; + clear_size = *clear_ptr; + int data_offset = flag_value % 3; + int clear_offset = (flag_value + 2) % 3; + for (int r = 0; r < NRanks; ++r) + { + data_bufs[r] = reinterpret_cast(workspace[2 * NRanks + r]) + data_offset * comm_size; + } + clear_buf = reinterpret_cast(workspace[2 * NRanks + rank]) + clear_offset * comm_size; + __syncthreads(); + if (threadIdx.x == 0) + { + atomicAdd(counter_ptr, 1); + } + } + + __device__ __forceinline__ void update(int64_t new_clear_size) + { + if (blockIdx.x == 0 && threadIdx.x == 0) + { + while (*reinterpret_cast(counter_ptr) != gridDim.x) + { + } + *flag_ptr = (flag_value + 1) % 3; + *clear_ptr = new_clear_size; + *counter_ptr = 0; + } + } + + int* counter_ptr; + int* flag_ptr; + int64_t* clear_ptr; + uint8_t* data_bufs[NRanks]; + uint8_t* clear_buf; + int64_t clear_size; + int flag_value; +}; + +template +__device__ __forceinline__ PackedType add128(PackedType const& a, PackedType const& b) +{ + static constexpr int kMathCount = sizeof(PackedType) / sizeof(DType); + PackedType c; +#pragma unroll + for (int i = 0; i < kMathCount; ++i) + { + reinterpret_cast(&c)[i] = reinterpret_cast(&a)[i] + reinterpret_cast(&b)[i]; + } + return c; +} + +// FusedOp: residual add -> residual_out -> rms_norm -> norm_out. +class FusedOp +{ + static constexpr int kMathCount = sizeof(float4) / sizeof(DType); + +public: + __device__ __forceinline__ FusedOp(Params const& params, int access_id, int access_id_in_token) + : m_params(params) + , m_access_id(access_id) + , m_access_id_in_token(access_id_in_token) + { + m_gamma_val = reinterpret_cast(params.rms_gamma)[m_access_id_in_token]; + m_residual_val = reinterpret_cast(params.residual_in)[m_access_id]; + } + + __device__ __forceinline__ void update(int access_id) + { + if (m_access_id != access_id) + { + m_access_id = access_id; + m_residual_val = reinterpret_cast(m_params.residual_in)[m_access_id]; + } + } + + __device__ __forceinline__ void operator()(float4 val, int token_id) + { + val = add128(val, m_residual_val); + reinterpret_cast(m_params.residual_out)[m_access_id] = val; + val = rms_norm(val, m_gamma_val); + reinterpret_cast(m_params.norm_out)[m_access_id] = val; + if (m_params.mx_out != nullptr) + { + using PV = tensorrt_llm::kernels::PackedVec; + PV pv = *reinterpret_cast(&val); + auto sf_out = tensorrt_llm::kernels::cvt_quant_get_sf_out_offset(std::optional(0), + token_id, m_access_id_in_token, std::optional(m_params.size / m_params.hidden_dim), + m_params.hidden_dim / 32, reinterpret_cast(m_params.mx_sf), + tensorrt_llm::QuantizationSFLayout::SWIZZLED); + reinterpret_cast(m_params.mx_out)[m_access_id] + = tensorrt_llm::kernels::cvt_warp_fp16_to_mxfp8(pv, sf_out); + } + } + + // The swizzled scale rows past the last token, up to the 128-row tile, are 0 (as quantize_mxfp8_rows writes). + __device__ __forceinline__ void pad_sf(int token_id) + { + auto sf_out = tensorrt_llm::kernels::cvt_quant_get_sf_out_offset(std::optional(0), token_id, + m_access_id_in_token, std::optional(m_params.size / m_params.hidden_dim), m_params.hidden_dim / 32, + reinterpret_cast(m_params.mx_sf), tensorrt_llm::QuantizationSFLayout::SWIZZLED); + tensorrt_llm::kernels::cvt_store_replicated_sf<1>(sf_out, 0x00); + } + +protected: + __device__ __forceinline__ float4 rms_norm(float4 const& residual, float4 const& gamma) + { + __shared__ float s_val; + float4 norm_out; + float acc = 0.f; +#pragma unroll + for (int i = 0; i < kMathCount; ++i) + { + float v = static_cast(reinterpret_cast(&residual)[i]); + acc += v * v; + } + blockReduceSumV2(&acc); + cg::cluster_group cluster = cg::this_cluster(); + if (cluster.num_blocks() > 1) + { + if (threadIdx.x == 0) + { + s_val = acc; + acc = 0.f; + } + cluster.sync(); + if (threadIdx.x == 0) + { + for (int i = 0; i < cluster.num_blocks(); ++i) + { + acc += *cluster.map_shared_rank(&s_val, i); + } + } + cluster.sync(); + } + if (threadIdx.x == 0) + { + s_val = rsqrtf(acc / m_params.hidden_dim + m_params.rms_eps); + } + __syncthreads(); +#pragma unroll + for (int i = 0; i < kMathCount; ++i) + { + reinterpret_cast(&norm_out)[i] + = static_cast(static_cast(reinterpret_cast(&residual)[i]) * s_val + * static_cast(reinterpret_cast(&gamma)[i])); + } + return norm_out; + } + +private: + Params const& m_params; + int m_access_id; + int m_access_id_in_token; + float4 m_residual_val; + float4 m_gamma_val; +}; + +__device__ __forceinline__ bool is_neg_zero(float v) +{ + return *reinterpret_cast(&v) == 0x80000000; +} + +__device__ __forceinline__ bool is_neg_zero(float4 v) +{ + return is_neg_zero(v.x) || is_neg_zero(v.y) || is_neg_zero(v.z) || is_neg_zero(v.w); +} + +__device__ __forceinline__ float4 get_neg_zero() +{ + float4 vec; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + reinterpret_cast(&vec)[i] = 0x80000000; + } + return vec; +} + +__device__ __forceinline__ float4 ld_global_volatile(float4* addr) +{ + float4 val; + asm volatile("ld.volatile.global.v4.f32 {%0, %1, %2, %3}, [%4];" + : "=f"(val.x), "=f"(val.y), "=f"(val.z), "=f"(val.w) + : "l"(addr)); + return val; +} + +__device__ __forceinline__ float4 allreduce_sum(float4* vals) +{ + float acc_f32[kElemsPerAccess]; +#pragma unroll + for (int i = 0; i < kElemsPerAccess; ++i) + { + acc_f32[i] = static_cast(reinterpret_cast(&vals[0])[i]); + } +#pragma unroll + for (int r = 1; r < NRanks; ++r) + { +#pragma unroll + for (int i = 0; i < kElemsPerAccess; ++i) + { + acc_f32[i] += static_cast(reinterpret_cast(&vals[r])[i]); + } + } + float4 acc; +#pragma unroll + for (int i = 0; i < kElemsPerAccess; ++i) + { + reinterpret_cast(&acc)[i] = static_cast(acc_f32[i]); + } + return acc; +} + +struct IndexHelper +{ + __device__ __forceinline__ IndexHelper(Params const& params) + { + cg::cluster_group cluster = cg::this_cluster(); + cg::grid_group grid = cg::this_grid(); + token_id = grid.cluster_rank(); + access_id_in_token = cluster.thread_rank(); + token_stride = grid.num_clusters(); + access_id = token_id * params.hidden_dim / kElemsPerAccess + access_id_in_token; + access_stride = token_stride * params.hidden_dim / kElemsPerAccess; + tot_access = params.size / kElemsPerAccess; + } + + int token_id; + int access_id_in_token; + int token_stride; + int access_id; + int access_stride; + int tot_access; +}; + +// allreduce_fusion_kernel_oneshot_lamport without +// the push loop: the data slots were filled by the producer. +template +__global__ void __launch_bounds__(1024) f3_ar_prepushed(Params params) +{ + IndexHelper index_helper(params); + int token_id = index_helper.token_id; + int access_id_in_token = index_helper.access_id_in_token; + int token_stride = index_helper.token_stride; + int access_id = index_helper.access_id; + int access_stride = index_helper.access_stride; + int tot_access = index_helper.tot_access; + float4 clear_vec = get_neg_zero(); + FusedOp fused_op(params, access_id, access_id_in_token); + + cudaGridDependencySynchronize(); + if constexpr (!TriggerCompletionAtEnd) + { + cudaTriggerProgrammaticLaunchCompletion(); + } + LamportComm comm(params.workspace, params.rank); + int clear_access = comm.clear_size / kElemsPerAccess; + + for (int idx = access_id; idx < clear_access; idx += access_stride) + { + // Clear comm buffer that previous kernel used + reinterpret_cast(comm.clear_buf)[idx] = clear_vec; + } + + for (int idx = access_id, tidx = token_id; idx < tot_access; idx += access_stride, tidx += token_stride) + { + fused_op.update(idx); + float4 vals[NRanks]; + bool done = false; + while (!done) + { + done = true; +#pragma unroll + for (int r = 0; r < NRanks; ++r) + { + vals[r] + = ld_global_volatile(&reinterpret_cast(comm.data_bufs[params.rank])[r * tot_access + idx]); + done &= !is_neg_zero(vals[r]); + } + } + float4 sum_val = allreduce_sum(vals); + fused_op(sum_val, tidx); + } + if (params.mx_out != nullptr) + { + int const token_num = params.size / params.hidden_dim; + int const padded = (token_num + 127) / 128 * 128; + for (int t = token_num + token_id; t < padded; t += token_stride) + { + fused_op.pad_sf(t); + } + } + + comm.update(params.size * NRanks); + + if constexpr (TriggerCompletionAtEnd) + { + cudaTriggerProgrammaticLaunchCompletion(); + } +} + +int smCount() +{ + static int n = 0; + if (n == 0) + { + int dev; + cudaGetDevice(&dev); + cudaDeviceGetAttribute(&n, cudaDevAttrMultiProcessorCount, dev); + } + return n; +} + +} // namespace f3 +} // namespace + +cudaError_t invokeF3ArPrepushed( + F3ArParams const& params, bool triggerCompletionAtEnd, bool enablePdl, cudaStream_t stream) +{ + using namespace f3; + Params const p = params; + int token_num = p.size / p.hidden_dim; + int cluster_num = token_num; + int threads_per_token = p.hidden_dim / kElemsPerAccess; + int cluster_size = 8; + while (threads_per_token % cluster_size != 0 && cluster_size > 1) + { + cluster_size /= 2; + } + int threads_per_block = threads_per_token / cluster_size; + while (threads_per_block < 128 && cluster_size >= 2) + { + threads_per_block *= 2; + cluster_size /= 2; + } + int sms = smCount(); + while (cluster_num * cluster_size > sms && cluster_size > 1 && threads_per_block <= 512) + { + threads_per_block *= 2; + cluster_size /= 2; + } + if (threads_per_block > 1024 || cluster_size <= 0) + { + return cudaErrorInvalidConfiguration; + } + int grid_size = (std::min(sms, cluster_num * cluster_size) / cluster_size) * cluster_size; + cudaLaunchConfig_t cfg = {}; + cudaLaunchAttribute attr[2]; + cfg.gridDim = grid_size; + cfg.blockDim = threads_per_block; + cfg.dynamicSmemBytes = 0; + cfg.stream = stream; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = enablePdl ? 1 : 0; + attr[1].id = cudaLaunchAttributeClusterDimension; + attr[1].val.clusterDim.x = cluster_size; + attr[1].val.clusterDim.y = 1; + attr[1].val.clusterDim.z = 1; + cfg.attrs = attr; + cfg.numAttrs = 2; + return triggerCompletionAtEnd ? cudaLaunchKernelEx(&cfg, f3_ar_prepushed, p) + : cudaLaunchKernelEx(&cfg, f3_ar_prepushed, p); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.h new file mode 100644 index 000000000000..20be6d80ea49 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.h @@ -0,0 +1,61 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! F3 (TP2 decode, the o_proj -> attention boundary): the one-shot Lamport all-reduce + residual + RMSNorm of +//! allreduce_fusion_kernel_oneshot_lamport with its push loop removed, +//! because the producer (the o_proj GEMM's epilogue fork, kernels/m31_f3_gemm.py) already stored its rows into this +//! rank's section of every rank's Lamport data slot. Optionally also the MXFP8 of norm_out (mx_out e4m3 [rows, hidden], +//! mx_sf the 128x4-swizzled UE8M0 scales of pad128(rows) rows; pad rows get 0 scale bytes) with the stock MXFP8 +//! quantize's device functions. Bitwise to the stock push + all-reduce (+ mxfp8_quantize of the norm). +struct F3ArParams +{ + int rank; + int size; //!< rows * hidden_dim + int hidden_dim; + void** workspace; //!< the AllReduce's Lamport workspace pointer table (2 ranks) + void* allreduce_in; + void* residual_in; + void* residual_out; + void* norm_out; + void* rms_gamma; + float rms_eps; + void* mx_out; //!< nullptr: no MXFP8 output + void* mx_sf; +}; + +//! The stock allreduce_fusion_kernel_launcher's one-shot shape (cluster launch). size % hidden_dim == 0 and +//! hidden_dim % 8 == 0 are the caller's to check. +cudaError_t invokeF3ArPrepushed( + F3ArParams const& params, bool triggerCompletionAtEnd, bool enablePdl, cudaStream_t stream); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.cu new file mode 100644 index 000000000000..fc541df7eeba --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.cu @@ -0,0 +1,178 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +namespace +{ + +constexpr int kThreads = 256; +constexpr int kWarps = kThreads / 32; + +// torch's argmax order (aten SharedReduceOps GreaterOrNan): a NaN beats any number, two NaNs or two equal values keep +// the lower index. A total order, so the reduction's shape cannot change the result. +__device__ __forceinline__ bool better(float va, int32_t ia, float vb, int32_t ib) +{ + bool const na = isnan(va); + bool const nb = isnan(vb); + if (na || nb) + { + return na && (!nb || ia < ib); + } + return va == vb ? ia < ib : va > vb; +} + +__device__ __forceinline__ void take(float& bv, int32_t& bi, float v, int32_t i) +{ + if (better(v, i, bv, bi)) + { + bv = v; + bi = i; + } +} + +__device__ __forceinline__ float bf16Bits(uint32_t lo16) +{ + return __uint_as_float(lo16 << 16); // exact bf16 -> fp32 +} + +__global__ void __launch_bounds__(kThreads) markovEpilogueKernel(float const* __restrict__ base, int64_t baseRowStride, + __nv_bfloat16 const* __restrict__ bias, float* __restrict__ out, int64_t outRowStride, int64_t* __restrict__ prev, + __nv_bfloat16 const* __restrict__ w1, int32_t rank, __nv_bfloat16* __restrict__ embOut, + MarkovEpilogueBest* __restrict__ partials, int32_t* __restrict__ counters, int32_t d, int32_t splits) +{ + __shared__ float sValue[kWarps]; + __shared__ int32_t sIndex[kWarps]; + __shared__ bool sLast; + + int32_t const row = blockIdx.y; + int32_t const split = blockIdx.x; + int32_t const d4 = d / 4; + int32_t const per = (d4 + splits - 1) / splits; + int32_t const begin = split * per; + int32_t const end = min(d4, begin + per); + + float4 const* src = reinterpret_cast(base + row * baseRowStride); + uint2 const* bsrc = reinterpret_cast(bias + static_cast(row) * d); + float4* dst = reinterpret_cast(out + row * outRowStride); + + float bv = -INFINITY; + int32_t bi = 0x7fffffff; + for (int32_t i = begin + threadIdx.x; i < end; i += kThreads) + { + float4 const x = src[i]; + uint2 const w = bsrc[i]; + float4 y; + y.x = __fadd_rn(x.x, bf16Bits(w.x & 0xffffu)); + y.y = __fadd_rn(x.y, bf16Bits(w.x >> 16)); + y.z = __fadd_rn(x.z, bf16Bits(w.y & 0xffffu)); + y.w = __fadd_rn(x.w, bf16Bits(w.y >> 16)); + dst[i] = y; + int32_t const j = 4 * i; + take(bv, bi, y.x, j); + take(bv, bi, y.y, j + 1); + take(bv, bi, y.z, j + 2); + take(bv, bi, y.w, j + 3); + } +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + { + float const ov = __shfl_xor_sync(0xffffffffu, bv, off); + int32_t const oi = __shfl_xor_sync(0xffffffffu, bi, off); + take(bv, bi, ov, oi); + } + int const warp = threadIdx.x / 32; + int const lane = threadIdx.x % 32; + if (lane == 0) + { + sValue[warp] = bv; + sIndex[warp] = bi; + } + __syncthreads(); + if (threadIdx.x == 0) + { + for (int w = 1; w < kWarps; ++w) + { + take(bv, bi, sValue[w], sIndex[w]); + } + partials[row * splits + split] = MarkovEpilogueBest{bv, bi}; + __threadfence(); + int32_t const done = atomicAdd(counters + row, 1); + sLast = done == splits - 1; + } + __syncthreads(); + if (!sLast) + { + return; + } + // The row's last CTA: combine the splits (any order: total order), publish prev and the next embedding. + __threadfence(); + if (threadIdx.x == 0) + { + MarkovEpilogueBest const* p = partials + row * splits; + float v = __ldcg(&p[0].value); + int32_t idx = __ldcg(&p[0].index); + for (int32_t s = 1; s < splits; ++s) + { + take(v, idx, __ldcg(&p[s].value), __ldcg(&p[s].index)); + } + prev[row] = idx; + sIndex[0] = idx; + counters[row] = 0; + } + __syncthreads(); + if (embOut != nullptr) + { + int64_t const token = sIndex[0]; + for (int32_t c = threadIdx.x; c < rank; c += kThreads) + { + embOut[static_cast(row) * rank + c] = w1[token * rank + c]; + } + } +} + +} // namespace + +int32_t markovEpilogueSplits(int32_t rows, int32_t d, int32_t smCount) +{ + int32_t const want = std::max(1, (2 * smCount + rows - 1) / std::max(1, rows)); + int32_t const cap = std::max(1, d / 4 / 1024); + return std::min(want, cap); +} + +void invokeMarkovEpilogue(float const* base, int64_t baseRowStride, __nv_bfloat16 const* bias, float* out, + int64_t outRowStride, int64_t* prev, __nv_bfloat16 const* w1, int32_t rank, __nv_bfloat16* embOut, + MarkovEpilogueBest* partials, int32_t* counters, int32_t rows, int32_t d, int32_t splits, cudaStream_t stream) +{ + dim3 const grid(splits, rows); + markovEpilogueKernel<<>>( + base, baseRowStride, bias, out, outRowStride, prev, w1, rank, embOut, partials, counters, d, splits); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.h new file mode 100644 index 000000000000..405e06039b06 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.h @@ -0,0 +1,68 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! The (value, index) partial of one CTA's slice of a row (the argmax reduction's carry). +struct MarkovEpilogueBest +{ + float value; + int32_t index; +}; + +//! S4FE: one position of the in-tree DSpark Markov chain's elementwise tail, fused +//! (models/modeling_speculative.py dspark_markov_chain with cast_bias_to_logits, full-vocab argmax): +//! +//! out[b, :] = base[b, :] + float(bias[b, :]) (fp32 add.rn; the bf16 -> fp32 cast is exact) +//! prev[b] = torch.argmax(out[b, :]) (NaN wins; ties and NaNs take the lowest index) +//! embOut[b] = w1[prev[b], :] (the next position's markov_prev_embeddings; prev is +//! always in vocab, so the mask is a no-op) +//! +//! Bitwise contract vs the in-tree path: each output element is the same single fp32 rounding as torch's add of the +//! cast bias, and the argmax uses torch's total order (GreaterOrNan: NaN beats any number, equal values or two NaNs +//! keep the lower index), so any split of a row gives torch's index. bias is the in-tree F.linear's own bf16 output. +//! +//! Layout: base and out are rows of d fp32 with row strides baseRowStride / outRowStride elements (a [B, K, d] tensor's +//! position k: stride K * d); bias is a contiguous [rows, d] bf16 tensor; d % 4 == 0 and every row 16-byte aligned +//! (base / out) or 8-byte aligned (bias). w1 is [vocab, rank] bf16 contiguous (rank <= 1024); embOut is [rows, rank] +//! or null. partials holds rows * splits entries; counters holds rows int32 that are zero before the first launch and +//! are left zero by every launch (the last CTA of a row resets its counter): CUDA-graph safe, stream-ordered reuse. +//! Grid (splits, rows), 256 threads; no co-residency requirement. +void invokeMarkovEpilogue(float const* base, int64_t baseRowStride, __nv_bfloat16 const* bias, float* out, + int64_t outRowStride, int64_t* prev, __nv_bfloat16 const* w1, int32_t rank, __nv_bfloat16* embOut, + MarkovEpilogueBest* partials, int32_t* counters, int32_t rows, int32_t d, int32_t splits, cudaStream_t stream); + +//! The default split count: about two CTAs per SM over the rows (at least 1, at most one CTA per 1024 float4s). +int32_t markovEpilogueSplits(int32_t rows, int32_t d, int32_t smCount); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.cu new file mode 100644 index 000000000000..5eeeb5ed5c7e --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.cu @@ -0,0 +1,190 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Exact MoE finalize: the native replacement of kernels/m3_moe_finalize +// (Triton). out[t, h] = bf16(sum_k fma(w[t, k], fc2[idx[t, k], h], acc)) over k in order (idx -1 skipped), float32; +// with shared: bf16(float(shared[t, h]) + float(that)). Bit for bit the Triton kernel and the TRTLLM-Gen runner's +// finalizeKernelVecLoad (+ shared.add_): the same per-element fp32 fma chain and RNE roundings. One CTA per +// (token, THREADS x NV x 8 columns), NV 16-byte vectors per thread (the tile / threads table never changes the order); +// every k gather is issued before the fma chain. +#include "tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +namespace +{ +constexpr int kVec = 8; + +// Explicit, exact widening of the weights (no reliance on the implicit bf16 -> float conversion, which builds that +// define __CUDA_NO_BFLOAT16_CONVERSIONS__ disable). +__device__ __forceinline__ float toFloat(float v) +{ + return v; +} + +__device__ __forceinline__ float toFloat(__nv_bfloat16 v) +{ + return __bfloat162float(v); +} + +template +__global__ void __launch_bounds__(THREADS) moeFinalizeKernel(__nv_bfloat16 const* __restrict__ fc2, + int32_t const* __restrict__ idx, W const* __restrict__ weights, __nv_bfloat16 const* __restrict__ shared, + __nv_bfloat16* __restrict__ out, int64_t ld_fc2, int hidden) +{ + int64_t const t = blockIdx.x; + int const col0 = blockIdx.y * (THREADS * NV * kVec) + threadIdx.x * kVec; + int32_t p[TOPK]; + float w[TOPK]; +#pragma unroll + for (int k = 0; k < TOPK; ++k) + { + p[k] = idx[t * TOPK + k]; + w[k] = toFloat(weights[t * TOPK + k]); + } + uint4 x[NV][TOPK]; + uint4 shv[NV]; +#pragma unroll + for (int v = 0; v < NV; ++v) + { + int const col = col0 + v * THREADS * kVec; + // The shared row is loaded with the gathers, not after the fma chain (load order only). + if constexpr (SHARED) + shv[v] = col < hidden ? __ldg(reinterpret_cast(shared + t * hidden + col)) + : make_uint4(0, 0, 0, 0); +#pragma unroll + for (int k = 0; k < TOPK; ++k) + x[v][k] = (p[k] >= 0 && col < hidden) + ? __ldg(reinterpret_cast(fc2 + static_cast(p[k]) * ld_fc2 + col)) + : make_uint4(0, 0, 0, 0); + } +#pragma unroll + for (int v = 0; v < NV; ++v) + { + int const col = col0 + v * THREADS * kVec; + if (col >= hidden) + continue; + float acc[kVec]; +#pragma unroll + for (int j = 0; j < kVec; ++j) + acc[j] = 0.f; +#pragma unroll + for (int k = 0; k < TOPK; ++k) + { + if (p[k] < 0) + continue; + __nv_bfloat16 const* xv = reinterpret_cast<__nv_bfloat16 const*>(&x[v][k]); +#pragma unroll + for (int j = 0; j < kVec; ++j) + acc[j] = __fmaf_rn(w[k], __bfloat162float(xv[j]), acc[j]); + } + uint4 r; + __nv_bfloat16* rv = reinterpret_cast<__nv_bfloat16*>(&r); +#pragma unroll + for (int j = 0; j < kVec; ++j) + rv[j] = __float2bfloat16_rn(acc[j]); + if constexpr (SHARED) + { + __nv_bfloat16 const* sv = reinterpret_cast<__nv_bfloat16 const*>(&shv[v]); +#pragma unroll + for (int j = 0; j < kVec; ++j) + rv[j] = __float2bfloat16_rn(__bfloat162float(sv[j]) + __bfloat162float(rv[j])); + } + *reinterpret_cast(out + t * hidden + col) = r; + } +} + +template +cudaError_t launchTopk(int topk, int64_t tokens, int hidden, cudaStream_t st, __nv_bfloat16 const* fc2, + int32_t const* idx, W const* w, __nv_bfloat16 const* sh, __nv_bfloat16* out, int64_t ld) +{ + dim3 grid(tokens, (hidden + THREADS * NV * kVec - 1) / (THREADS * NV * kVec)); + // clang-format off + switch (topk) + { +#define M31_FIN_CASE(K) \ + case K: \ + moeFinalizeKernel<<>>(fc2, idx, w, sh, out, ld, hidden); \ + break; + M31_FIN_CASE(1) + M31_FIN_CASE(2) + M31_FIN_CASE(3) + M31_FIN_CASE(4) + M31_FIN_CASE(5) + M31_FIN_CASE(6) + M31_FIN_CASE(7) + M31_FIN_CASE(8) + M31_FIN_CASE(9) + M31_FIN_CASE(10) +#undef M31_FIN_CASE + default: return cudaErrorInvalidValue; + } + // clang-format on + return cudaGetLastError(); +} + +// (THREADS, NV) table; cfg -1 = by rows (the serving default). +template +cudaError_t launchCfg(int cfg, int topk, int64_t tokens, int hidden, cudaStream_t st, __nv_bfloat16 const* f, + int32_t const* ix, W const* w, __nv_bfloat16 const* sh, __nv_bfloat16* o, int64_t ld) +{ + if (cfg < 0) + cfg = 0; + switch (cfg) + { + case 0: return launchTopk(topk, tokens, hidden, st, f, ix, w, sh, o, ld); + case 1: return launchTopk(topk, tokens, hidden, st, f, ix, w, sh, o, ld); + case 2: return launchTopk(topk, tokens, hidden, st, f, ix, w, sh, o, ld); + case 3: return launchTopk(topk, tokens, hidden, st, f, ix, w, sh, o, ld); + case 4: return launchTopk(topk, tokens, hidden, st, f, ix, w, sh, o, ld); + case 5: return launchTopk(topk, tokens, hidden, st, f, ix, w, sh, o, ld); + case 6: return launchTopk(topk, tokens, hidden, st, f, ix, w, sh, o, ld); + case 7: return launchTopk(topk, tokens, hidden, st, f, ix, w, sh, o, ld); + default: return cudaErrorInvalidValue; + } +} +} // namespace + +cudaError_t invokeMoeFinalize(__nv_bfloat16 const* fc2, int64_t ldFc2, void const* weights, bool weightsFp32, + int32_t const* idx, __nv_bfloat16 const* shared, __nv_bfloat16* out, int64_t tokens, int topk, int hidden, int cfg, + cudaStream_t stream) +{ + if (tokens == 0) + { + return cudaSuccess; + } + if (weightsFp32) + { + auto const* w = static_cast(weights); + return shared != nullptr + ? launchCfg(cfg, topk, tokens, hidden, stream, fc2, idx, w, shared, out, ldFc2) + : launchCfg(cfg, topk, tokens, hidden, stream, fc2, idx, w, shared, out, ldFc2); + } + auto const* w = static_cast<__nv_bfloat16 const*>(weights); + return shared != nullptr + ? launchCfg<__nv_bfloat16, true>(cfg, topk, tokens, hidden, stream, fc2, idx, w, shared, out, ldFc2) + : launchCfg<__nv_bfloat16, false>(cfg, topk, tokens, hidden, stream, fc2, idx, w, shared, out, ldFc2); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.h new file mode 100644 index 000000000000..a1d5e2b3a149 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.h @@ -0,0 +1,45 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! NATIVE_FINALIZE: out[t, h] = bf16(sum_k fma(w[t, k], fc2[idx[t, k], h], acc)) over k in order +//! (idx -1 skipped), fp32; with shared: bf16(float(shared[t, h]) + float(that)). Bit for bit kernels/m3_moe_finalize +//! (Triton) and the TRTLLM-Gen runner's finalize (+ shared.add_). weights: bf16 or fp32 [tokens, topk]; topk 1..10; +//! fc2 rows 16-byte aligned (hidden % 8 == 0, ldFc2 % 8 == 0). cfg: the (threads, vectors per thread) table entry 0..7 +//! (only the work split; every element's arithmetic is the same in all of them). +cudaError_t invokeMoeFinalize(__nv_bfloat16 const* fc2, int64_t ldFc2, void const* weights, bool weightsFp32, + int32_t const* idx, __nv_bfloat16 const* shared, __nv_bfloat16* out, int64_t tokens, int topk, int hidden, int cfg, + cudaStream_t stream); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.cu new file mode 100644 index 000000000000..d9718eaf5260 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.cu @@ -0,0 +1,146 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.h" +#include "tensorrt_llm/kernels/moe/trtllmGen/routing/RoutingCustomPolicy.cuh" + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +namespace +{ +namespace cg = cooperative_groups; +namespace rt = moe::dev::routing; +namespace btg = batchedGemm::trtllm::gen; + +using Policy = rt::TopKExpertSelect; + +struct FoldParams +{ + typename Policy::template Params<__nv_bfloat16> mExpertSelectParams; +}; + +constexpr int kE = 128; +constexpr int kK = 4; +constexpr int kWarps = 8; + +template +__global__ void __launch_bounds__(kWarps * 32) s1bFoldKernel(float const* __restrict__ partials, int32_t M, + float* __restrict__ logitsOut, int32_t* __restrict__ ids, __nv_bfloat16* __restrict__ weights, FoldParams p) +{ + __shared__ float row[kWarps][kE]; + auto block = cg::this_thread_block(); + auto warp = cg::tiled_partition<32>(block); + int32_t const lane = threadIdx.x & 31; + int32_t const wid = threadIdx.x >> 5; + int32_t const m = blockIdx.x * kWarps + wid; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); +#endif + if (m >= M) + { + return; + } + // Every slice's value first (all loads in flight), then the slice-order sum (the bits of cuBLASLt's reduce). + float v[kE / 32][S]; +#pragma unroll + for (int i = 0; i < kE / 32; ++i) + { +#pragma unroll + for (int s = 0; s < S; ++s) + { + v[i][s] = __ldg(partials + (static_cast(s) * M + m) * kE + i * 32 + lane); + } + } +#pragma unroll + for (int i = 0; i < kE / 32; ++i) + { + float acc = v[i][0]; +#pragma unroll + for (int s = 1; s < S; ++s) + { + acc = acc + v[i][s]; + } + row[wid][i * 32 + lane] = acc; + if (logitsOut != nullptr) + { + logitsOut[static_cast(m) * kE + i * 32 + lane] = acc; + } + } + __syncwarp(); + float topScore[8]; + int32_t topIdx[8]; + Policy::template apply(warp, topScore, topIdx, lane, kE, kK, &row[wid][0], p); + if (lane < kK) + { + ids[m * kK + lane] = topIdx[lane]; + weights[m * kK + lane] = __nv_bfloat16{topScore[lane]}; + } +} + +template +cudaError_t launch(float const* partials, int32_t M, float* lo, int32_t* ids, __nv_bfloat16* w, FoldParams const& p, + bool pdl, cudaStream_t stream) +{ + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3((M + kWarps - 1) / kWarps); + cfg.blockDim = dim3(kWarps * 32); + cfg.dynamicSmemBytes = 0; + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, s1bFoldKernel, partials, M, lo, ids, w, p); +} +} // namespace + +cudaError_t invokeS1bFold(float const* partials, int32_t S, int32_t M, float const* bias, float routeScale, + float* logitsOut, int32_t* ids, __nv_bfloat16* weights, bool pdl, cudaStream_t stream) +{ + if (M <= 0) + { + return cudaSuccess; + } + FoldParams p; + p.mExpertSelectParams.mPreprocessParams.ptrRoutingBias = bias; + p.mExpertSelectParams.mPreprocessParams.dtypeBias = btg::Dtype::Fp32; + p.mExpertSelectParams.mPostprocessParams.ptrRoutingBias = bias; + p.mExpertSelectParams.mPostprocessParams.dtypeBias = btg::Dtype::Fp32; + p.mExpertSelectParams.mPostprocessParams.routeScale = routeScale; + p.mExpertSelectParams.mPostprocessParams.sumEpsilon = 1e-20f; + if (S == 8) + { + return launch<8>(partials, M, logitsOut, ids, weights, p, pdl, stream); + } + if (S == 16) + { + return launch<16>(partials, M, logitsOut, ids, weights, p, pdl, stream); + } + return cudaErrorInvalidValue; +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.h new file mode 100644 index 000000000000..08552adb06ee --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.h @@ -0,0 +1,46 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! S1b fold: the MiniMax-M3.1 router's split-K reduce + the trtllm-gen MiniMax2 top-4 selection in one launch. +//! partials: contiguous fp32 [S, M, 128], one strided-batched TF32 GEMM over S K-slices of the router projection; at +//! the split factor cuBLASLt's heuristic picks for the shape, summed in slice order they are bitwise its split-K +//! logits. Per row (one warp): the slice-order sum, then moe::dev::routing::TopKExpertSelect (the production E128K8 instance), so ids (top-4, in slot order) and bf16 weights are bitwise +//! what routingIndicesClusterKernel selects from the same logits. bias: fp32 [128]; ids: int32 [M, 4]; weights: bf16 +//! [M, 4]; logitsOut: fp32 [M, 128] or null. S must be 8 or 16. PDL: waits on its predecessor, releases its dependents +//! early. +cudaError_t invokeS1bFold(float const* partials, int32_t S, int32_t M, float const* bias, float routeScale, + float* logitsOut, int32_t* ids, __nv_bfloat16* weights, bool pdl, cudaStream_t stream); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.cu new file mode 100644 index 000000000000..507d8cbd7065 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.cu @@ -0,0 +1,445 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * MiniMax-M3.1 TP2 decode MoE tail in one launch, bit for bit the chain + * m3_moe_finalize (fp32 fma over the top-k slots in order, round to bf16, + shared in fp32, round) + * -> allreduce fusion, one-shot Lamport, pattern kARResidualRMSNorm (residual_out + norm_out), Fp32Acc + * -> mxfp8_quantize(norm_out, swizzled) (cvt_warp_fp16_to_mxfp8, K32 UE8M0 scales, pad rows' scales 0). + * The kernel is allreduce_fusion_kernel_oneshot_lamport (communicationKernels/allReduceFusionKernels.cu) with (1) the + * allreduce_in load replaced by the finalize of the same 8 columns and (2) the MXFP8 quantize of the normed 8 columns + * in the epilogue; the launch configuration (cluster, block, grid) is that file's launcher, so the RMSNorm block / + * cluster reduction order is unchanged. + * Op: trtllm::minimax_m31_s2_moe_tail(fc2, scales, perm, shared, residual, norm_weight, workspace, rank, nranks, eps, + * trigger_completion_at_end) -> (norm_out bf16 [T, H], residual_out bf16 [T, H], mxfp8 [T, H], sf u8 swizzled) + * workspace = the AllReduce module's lamport workspace (the stock op's). + */ + +#include "tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.h" + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/envUtils.h" +#include "tensorrt_llm/common/reduceKernelUtils.cuh" +#include "tensorrt_llm/kernels/quantization.cuh" + +#include +#include + +#include +#include + +namespace cg = cooperative_groups; + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::minimaxM31Sm107::s2tail +{ + +using tensorrt_llm::kernels::PackedVec; +using tensorrt_llm::QuantizationSFLayout; + +template +struct LamportComm +{ + __device__ __forceinline__ LamportComm(void** workspace, int rank) + { + counter_ptr = &reinterpret_cast(workspace[NRanks * 3])[0]; + flag_ptr = &reinterpret_cast(workspace[NRanks * 3])[2]; + clear_ptr = &reinterpret_cast(workspace[NRanks * 3 + 1])[0]; + flag_value = *flag_ptr; + auto comm_size = reinterpret_cast(workspace[NRanks * 3 + 1])[1]; + clear_size = *clear_ptr; + int data_offset = flag_value % 3; + int clear_offset = (flag_value + 2) % 3; + for (int r = 0; r < NRanks; ++r) + { + data_bufs[r] = reinterpret_cast(workspace[2 * NRanks + r]) + data_offset * comm_size; + } + clear_buf = reinterpret_cast(workspace[2 * NRanks + rank]) + clear_offset * comm_size; + __syncthreads(); + if (threadIdx.x == 0) + { + atomicAdd(counter_ptr, 1); + } + } + + __device__ __forceinline__ void update(int64_t new_clear_size) + { + if (blockIdx.x == 0 && threadIdx.x == 0) + { + while (*reinterpret_cast(counter_ptr) != gridDim.x) + { + } + *flag_ptr = (flag_value + 1) % 3; + *clear_ptr = new_clear_size; + *counter_ptr = 0; + } + } + + int* counter_ptr; + int* flag_ptr; + int64_t* clear_ptr; + uint8_t* data_bufs[NRanks]; + uint8_t* clear_buf; + int64_t clear_size; + int flag_value; +}; + +__device__ __forceinline__ bool is_neg_zero(float v) +{ + return *reinterpret_cast(&v) == 0x80000000; +} + +__device__ __forceinline__ bool is_neg_zero(float4 v) +{ + return is_neg_zero(v.x) || is_neg_zero(v.y) || is_neg_zero(v.z) || is_neg_zero(v.w); +} + +__device__ __forceinline__ float4 get_neg_zero() +{ + float4 vec; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + reinterpret_cast(&vec)[i] = 0x80000000; + } + return vec; +} + +__device__ __forceinline__ float4 ld_global_volatile(float4* addr) +{ + float4 val; + asm volatile("ld.volatile.global.v4.f32 {%0, %1, %2, %3}, [%4];" + : "=f"(val.x), "=f"(val.y), "=f"(val.z), "=f"(val.w) + : "l"(addr)); + return val; +} + +__device__ __forceinline__ float4 add128(float4 const& a, float4 const& b) +{ + float4 c; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + reinterpret_cast<__nv_bfloat16*>(&c)[i] + = reinterpret_cast<__nv_bfloat16 const*>(&a)[i] + reinterpret_cast<__nv_bfloat16 const*>(&b)[i]; + } + return c; +} + +template +__device__ __forceinline__ float4 allreduce_sum(float4* vals) +{ + if constexpr (Fp32Acc) + { + float acc_f32[kElems]; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + acc_f32[i] = static_cast(reinterpret_cast<__nv_bfloat16*>(&vals[0])[i]); + } +#pragma unroll + for (int r = 1; r < NRanks; ++r) + { +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + acc_f32[i] += static_cast(reinterpret_cast<__nv_bfloat16*>(&vals[r])[i]); + } + } + float4 acc; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + reinterpret_cast<__nv_bfloat16*>(&acc)[i] = static_cast<__nv_bfloat16>(acc_f32[i]); + } + return acc; + } + else + { + float4 acc = vals[0]; +#pragma unroll + for (int r = 1; r < NRanks; ++r) + { + acc = add128(acc, vals[r]); + } + return acc; + } +} + +// The finalize of 8 columns of one token: m3_moe_finalize's per-element math (fp32 fma over the slots in order, a +// dropped slot leaves the accumulator, bf16 round, + shared in fp32, bf16 round). +template +__device__ __forceinline__ float4 finalize8(Params const& p, int token, int col8) +{ + float acc[kElems]; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + acc[i] = 0.f; + } + for (int k = 0; k < p.top_k; ++k) + { + int const slot = p.perm[token * p.top_k + k]; + float const w = static_cast(reinterpret_cast(p.scales)[token * p.top_k + k]); + if (slot >= 0) + { + float4 x4 = reinterpret_cast(p.fc2 + static_cast(slot) * p.ld_fc2)[col8]; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + acc[i] = __fmaf_rn(w, static_cast(reinterpret_cast<__nv_bfloat16*>(&x4)[i]), acc[i]); + } + } + } + float4 s4 = reinterpret_cast(p.shared + static_cast(token) * p.hidden_dim)[col8]; + float4 out; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + float const r = static_cast(__float2bfloat16_rn(acc[i])); + float const s = static_cast(reinterpret_cast<__nv_bfloat16*>(&s4)[i]); + reinterpret_cast<__nv_bfloat16*>(&out)[i] = __float2bfloat16_rn(s + r); + } + return out; +} + +// allreduce_fusion's FusedOp::rms_norm verbatim (block reduce, then the cluster's partial sums in rank order). +__device__ __forceinline__ float4 rms_norm(float4 const& residual, float4 const& gamma, int hidden_dim, float eps) +{ + __shared__ float s_val; + float4 norm_out; + float acc = 0.f; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + float v = static_cast(reinterpret_cast<__nv_bfloat16 const*>(&residual)[i]); + acc += v * v; + } + tensorrt_llm::common::blockReduceSumV2(&acc); +#if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)) + cg::cluster_group cluster = cg::this_cluster(); + if (cluster.num_blocks() > 1) + { + if (threadIdx.x == 0) + { + s_val = acc; + acc = 0.f; + } + cluster.sync(); + if (threadIdx.x == 0) + { + for (int i = 0; i < cluster.num_blocks(); ++i) + { + acc += *cluster.map_shared_rank(&s_val, i); + } + } + cluster.sync(); + } +#endif + if (threadIdx.x == 0) + { + s_val = rsqrtf(acc / hidden_dim + eps); + } + __syncthreads(); +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + reinterpret_cast<__nv_bfloat16*>(&norm_out)[i] + = static_cast<__nv_bfloat16>(static_cast(reinterpret_cast<__nv_bfloat16 const*>(&residual)[i]) + * s_val * static_cast(reinterpret_cast<__nv_bfloat16 const*>(&gamma)[i])); + } + return norm_out; +} + +template +__global__ void __launch_bounds__(1024) s2_moe_tail_kernel(Params params) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + cg::cluster_group cluster = cg::this_cluster(); + cg::grid_group grid = cg::this_grid(); + int const token_id = grid.cluster_rank(); + int const access_id_in_token = cluster.thread_rank(); + int const token_stride = grid.num_clusters(); + int const access_per_token = params.hidden_dim / kElems; + int const access_id = token_id * access_per_token + access_id_in_token; + int const access_stride = token_stride * access_per_token; + int const tot_access = params.size / kElems; + int const tokens = params.size / params.hidden_dim; + float4 const clear_vec = get_neg_zero(); + float4 const gamma = reinterpret_cast(params.rms_gamma)[access_id_in_token]; + + cudaGridDependencySynchronize(); + if constexpr (!TriggerCompletionAtEnd) + { + cudaTriggerProgrammaticLaunchCompletion(); + } + LamportComm comm(params.workspace, params.rank); + int const clear_access = comm.clear_size / kElems; + + for (int idx = access_id, tidx = token_id; idx < tot_access; idx += access_stride, tidx += token_stride) + { + alignas(16) float val[4]; + *reinterpret_cast(val) = finalize8(params, tidx, access_id_in_token); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + if (is_neg_zero(val[i])) + { + val[i] = 0.f; + } + } +#pragma unroll + for (int r = 0; r < NRanks; ++r) + { + reinterpret_cast(comm.data_bufs[r])[params.rank * tot_access + idx] + = *reinterpret_cast(val); + } + } + for (int idx = access_id; idx < clear_access; idx += access_stride) + { + reinterpret_cast(comm.clear_buf)[idx] = clear_vec; + } + + for (int idx = access_id, tidx = token_id; idx < tot_access; idx += access_stride, tidx += token_stride) + { + float4 const residual = reinterpret_cast(params.residual_in)[idx]; + float4 vals[NRanks]; + bool done = false; + while (!done) + { + done = true; +#pragma unroll + for (int r = 0; r < NRanks; ++r) + { + vals[r] + = ld_global_volatile(&reinterpret_cast(comm.data_bufs[params.rank])[r * tot_access + idx]); + done &= !is_neg_zero(vals[r]); + } + } + float4 v = allreduce_sum(vals); + v = add128(v, residual); + reinterpret_cast(params.residual_out)[idx] = v; + v = rms_norm(v, gamma, params.hidden_dim, params.rms_eps); + reinterpret_cast(params.norm_out)[idx] = v; + // MXFP8 of the normed 8 columns: mxfp8_quantize's per-element math (4 lanes share a K32 scale). + PackedVec<__nv_bfloat16> pack = *reinterpret_cast*>(&v); + auto sf_out + = tensorrt_llm::kernels::cvt_quant_get_sf_out_offset(std::nullopt, tidx, access_id_in_token, + std::optional(tokens), params.hidden_dim / 32, params.scale_out, QuantizationSFLayout::SWIZZLED); + params.quant_out[idx] = tensorrt_llm::kernels::cvt_warp_fp16_to_mxfp8<__nv_bfloat16, 32>(pack, sf_out); + } + // The swizzled scales of the pad rows up to the 128-row tile are 0 (mxfp8_quantize's). + int const padded = (tokens + 127) / 128 * 128; + int const sf_cols = params.hidden_dim / 32; + for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < (padded - tokens) * sf_cols; i += gridDim.x * blockDim.x) + { + int const row = tokens + i / sf_cols; + int const col = i % sf_cols; + int64_t const off = tensorrt_llm::kernels::get_sf_out_offset_128x4( + std::nullopt, row, col, std::optional(tokens), sf_cols); + reinterpret_cast(params.scale_out)[off] = 0; + } + + comm.update(static_cast(params.size) * NRanks); + if constexpr (TriggerCompletionAtEnd) + { + __threadfence(); + cudaTriggerProgrammaticLaunchCompletion(); + } +#endif +} + +int get_sm_count() +{ + static int sm_count = 0; + if (sm_count == 0) + { + int device_id; + TLLM_CUDA_CHECK(cudaGetDevice(&device_id)); + cudaDeviceProp device_prop; + cudaGetDeviceProperties(&device_prop, device_id); + sm_count = device_prop.multiProcessorCount; + } + return sm_count; +} + +bool use_fp32_acc() +{ + static char* fp32_acc = std::getenv("ALL_REDUCE_FUSION_KERNEL_ACC_FP32"); + return fp32_acc == nullptr || fp32_acc[0] != '0'; +} + +template +void launch(Params const& params, cudaStream_t stream) +{ + // allreduce_fusion_kernel_launcher (oneshot) verbatim. + int token_num = params.size / params.hidden_dim; + int cluster_num = token_num; + int threads_per_token = params.hidden_dim / kElems; + int cluster_size = 8; + while (threads_per_token % cluster_size != 0 && cluster_size > 1) + { + cluster_size /= 2; + } + int threads_per_block = threads_per_token / cluster_size; + while (threads_per_block < 128 && cluster_size >= 2) + { + threads_per_block *= 2; + cluster_size /= 2; + } + int sm_count = get_sm_count(); + while (cluster_num * cluster_size > sm_count && cluster_size > 1 && threads_per_block <= 512) + { + threads_per_block *= 2; + cluster_size /= 2; + } + TLLM_CHECK(threads_per_block <= 1024 && cluster_size > 0); + int grid_size = (std::min(sm_count, cluster_num * cluster_size) / cluster_size) * cluster_size; + cudaLaunchConfig_t cfg; + cudaLaunchAttribute attribute[2]; + cfg.gridDim = grid_size; + cfg.blockDim = threads_per_block; + cfg.dynamicSmemBytes = 0; + cfg.stream = stream; + attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attribute[0].val.programmaticStreamSerializationAllowed = tensorrt_llm::common::getEnvEnablePDL() ? 1 : 0; + attribute[1].id = cudaLaunchAttributeClusterDimension; + attribute[1].val.clusterDim.x = cluster_size; + attribute[1].val.clusterDim.y = 1; + attribute[1].val.clusterDim.z = 1; + cfg.attrs = attribute; + cfg.numAttrs = 2; + TLLM_CUDA_CHECK(cudaLaunchKernelEx(&cfg, s2_moe_tail_kernel, params)); +} + +template +void dispatch_acc(Params const& p, bool trigger, cudaStream_t s) +{ + bool const fp32 = use_fp32_acc(); + if (fp32) + { + trigger ? launch(p, s) : launch(p, s); + } + else + { + trigger ? launch(p, s) : launch(p, s); + } +} + +void invokeS2MoeTail(Params const& params, bool scalesBf16, bool triggerCompletionAtEnd, cudaStream_t stream) +{ + TLLM_CHECK_WITH_INFO(params.nranks == 2, "TP2 only"); + if (scalesBf16) + { + dispatch_acc<__nv_bfloat16, 2>(params, triggerCompletionAtEnd, stream); + } + else + { + dispatch_acc(params, triggerCompletionAtEnd, stream); + } +} + +} // namespace kernels::minimaxM31Sm107::s2tail + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.h new file mode 100644 index 000000000000..03a3b2410d20 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.h @@ -0,0 +1,52 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::minimaxM31Sm107::s2tail +{ + +constexpr int kElems = 8; // bf16 per float4 access +// The AUTO all-reduce strategy takes the one-shot Lamport kernel for up to 256 rows; this op covers the same range. +constexpr int kMaxTokens = 256; + +struct Params +{ + int nranks; + int rank; + int size; // tokens * hidden + int hidden_dim; + int top_k; + int ld_fc2; + void** workspace; + __nv_bfloat16 const* fc2; + void const* scales; // [T, top_k] bf16 or fp32 + int const* perm; // [T, top_k] expanded -> permuted row, < 0 = dropped + __nv_bfloat16 const* shared; + __nv_bfloat16 const* residual_in; + __nv_bfloat16 const* rms_gamma; + float rms_eps; + __nv_bfloat16* residual_out; + __nv_bfloat16* norm_out; + uint64_t* quant_out; + uint32_t* scale_out; +}; + +//! TP2W_S2_TAIL: the MoE tail of one TP2 decode boundary in one launch (routed finalize + shared add, the one-shot +//! Lamport all-reduce on the AllReduce module's workspace, residual add + RMSNorm, MXFP8 out). scalesBf16 picks the +//! expert-scale dtype (bf16 or fp32); trigger releases dependents at the end. +void invokeS2MoeTail(Params const& params, bool scalesBf16, bool triggerCompletionAtEnd, cudaStream_t stream); + +} // namespace kernels::minimaxM31Sm107::s2tail + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.cu new file mode 100644 index 000000000000..aa1554d6a8ef --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.cu @@ -0,0 +1,80 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +namespace +{ + +constexpr int kThreads = 256; +constexpr int kChunk = 16384; // elements per CTA + +template +__global__ void __launch_bounds__(kThreads) scatterRowsKernel(float* __restrict__ dst, int64_t dstSlotStride, + int64_t dstRowStride, float const* __restrict__ src, int64_t const* __restrict__ slotIds, int32_t K, int32_t v) +{ + int32_t const row = blockIdx.y; // i * K + k + int32_t const i = row / K; + int32_t const k = row % K; + float* d = dst + slotIds[i] * dstSlotStride + static_cast(k) * dstRowStride; + float const* s = src + static_cast(row) * v; + int32_t const begin = blockIdx.x * kChunk; + int32_t const end = begin + min(v - begin, kChunk); + if constexpr (kVec) + { + for (int32_t j = begin / 4 + threadIdx.x; j < end / 4; j += kThreads) + { + reinterpret_cast(d)[j] = reinterpret_cast(s)[j]; + } + } + else + { + for (int64_t j = static_cast(begin) + threadIdx.x; j < end; j += kThreads) + { + d[j] = s[j]; + } + } +} + +} // namespace + +void invokeScatterRows(float* dst, int64_t dstSlotStride, int64_t dstRowStride, float const* src, + int64_t const* slotIds, int32_t n, int32_t K, int32_t v, cudaStream_t stream) +{ + dim3 const grid((v - 1) / kChunk + 1, n * K); + bool const vec = v % 4 == 0 && dstSlotStride % 4 == 0 && dstRowStride % 4 == 0 + && reinterpret_cast(dst) % 16 == 0 && reinterpret_cast(src) % 16 == 0; + if (vec) + { + scatterRowsKernel<<>>(dst, dstSlotStride, dstRowStride, src, slotIds, K, v); + } + else + { + scatterRowsKernel<<>>(dst, dstSlotStride, dstRowStride, src, slotIds, K, v); + } +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.h new file mode 100644 index 000000000000..3b056ee348fa --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.h @@ -0,0 +1,43 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! TAIL16-G3: the block draft-prob store, dst[slotIds[i], k, 0:v] = src[i, k, 0:v] for i < n, k < K (a row copy; +//! bitwise the index_put it replaces). src: contiguous fp32 [n, K, v]; dst: fp32 with element strides dstSlotStride +//! (between slots) and dstRowStride (between k); unit stride on the last dim. Grid (chunks, n * K); graph-safe. Caller +//! provides valid nonnegative slot ids, nonoverlapping rows and disjoint source/destination storage. Duplicate slots +//! are only defined when their source rows are bitwise identical (e.g. padding). +void invokeScatterRows(float* dst, int64_t dstSlotStride, int64_t dstRowStride, float const* src, + int64_t const* slotIds, int32_t n, int32_t K, int32_t v, cudaStream_t stream); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.cu new file mode 100644 index 000000000000..1d99786ef66d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.cu @@ -0,0 +1,259 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.h" +#include +#include +#include +#include +#include +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +namespace +{ +namespace fi +{ +// Helpers and ordered softmax arithmetic adapted from FlashInfer 0.6.18, Copyright (c) 2023-2026 FlashInfer team, +// Apache-2.0. Float4 helpers are the same subset used by softmaxMcSm103.cu. Keep the source arithmetic form so the +// denominator recurrence and final CUB sum preserve the reference's rounding. Only the chunk maximum is replaced +// with unsigned hardware warp reductions and separate ping-pong shared storage. +// clang-format off +#define FLASHINFER_INLINE inline __attribute__((always_inline)) __device__ + +template +__forceinline__ __device__ __host__ constexpr T1 ceil_div(const T1 x, const T2 y) noexcept { + return (x + y - 1) / y; +} + +template +__forceinline__ __device__ __host__ constexpr T1 round_up(const T1 x, const T2 y) noexcept { + return ceil_div(x, y) * y; +} + +template +struct vec_t; + +// float x 4 or more +template +struct vec_t { + static_assert(vec_size % 4 == 0, "Invalid vector size"); + float4 data[vec_size / 4]; + + FLASHINFER_INLINE float& operator[](size_t i) { return ((float*)(data))[i]; } + FLASHINFER_INLINE const float& operator[](size_t i) const { return ((const float*)(data))[i]; } + FLASHINFER_INLINE void fill(float val) { +#pragma unroll + for (size_t i = 0; i < vec_size / 4; ++i) { + data[i] = make_float4(val, val, val, val); + } + } + FLASHINFER_INLINE void load(const float* ptr) { +#pragma unroll + for (size_t i = 0; i < vec_size / 4; ++i) { + data[i] = ((float4*)ptr)[i]; + } + } + FLASHINFER_INLINE void store(float* ptr) const { +#pragma unroll + for (size_t i = 0; i < vec_size / 4; ++i) { + ((float4*)ptr)[i] = data[i]; + } + } + template + FLASHINFER_INLINE void cast_load(const T* ptr) { + static_assert(std::is_same_v, "only the same-type branch of cast_load_impl is vendored"); + load(ptr); + } + template + FLASHINFER_INLINE void cast_store(T* ptr) const { + static_assert(std::is_same_v, "only the same-type branch of cast_store_impl is vendored"); + store(ptr); + } +}; + +#undef FLASHINFER_INLINE +template +struct OnlineSoftmaxTempStorage { + union { + typename cub::BlockReduce::TempStorage reduce; + typename cub::BlockReduce::TempStorage reduce_pair; + } block_prim; + + struct { + float max_val; + float denominator; + } shared_state; +}; + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1070) +#define M31_PATHB_LAUNCH_BOUNDS(block_threads) __launch_bounds__(block_threads) +#else +#define M31_PATHB_LAUNCH_BOUNDS(block_threads) +#endif +template +__global__ M31_PATHB_LAUNCH_BOUNDS(BLOCK_THREADS) void PathBWarpMaxKernel( + DType* logits, DType* output, DType* temperature_arr, DType temperature_val) { + constexpr uint32_t d = 200064; + static_assert(BLOCK_THREADS == 1024 && VEC_SIZE == 4 && !CACHE_INPUT); + // Separate from final CUB storage: it may be used while slower warps finish the last max load. + // Buffer i is not overwritten until all warps have crossed the next iteration's barrier. + __shared__ uint32_t warp_max_words[2][32]; + const uint32_t bx = blockIdx.x, tx = threadIdx.x; + float temperature = temperature_arr == nullptr ? temperature_val : temperature_arr[bx]; + const float inv_temp = (temperature == 0.f) ? 0.f : 1.f / temperature; + + using TempStorage = OnlineSoftmaxTempStorage; + extern __shared__ __align__(alignof(TempStorage)) uint8_t smem[]; + auto& temp_storage = reinterpret_cast(smem); + + DType* smem_vec_base = nullptr; + if constexpr (CACHE_INPUT) { + constexpr size_t vec_alignment = alignof(vec_t); + size_t aligned_offset = round_up(sizeof(TempStorage), vec_alignment); + smem_vec_base = reinterpret_cast(smem + aligned_offset); + } + + vec_t logits_vec; + + float running_max = -cuda::std::numeric_limits::infinity(); + float running_denominator = 0.0f; + float threadlocal_running_denominator = 0.0f; + +#if (__CUDACC_VER_MAJOR__ >= 12 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)) + asm volatile("griddepcontrol.wait;"); +#endif + + // Pass 1: Compute running max and denominator +#pragma unroll 2 + for (uint32_t i = 0; i < ceil_div(d, BLOCK_THREADS * VEC_SIZE); ++i) { + logits_vec.fill(-cuda::std::numeric_limits::infinity()); + if ((i * BLOCK_THREADS + tx) * VEC_SIZE < d) { + logits_vec.cast_load(logits + bx * d + (i * BLOCK_THREADS + tx) * VEC_SIZE); + +#pragma unroll + for (uint32_t j = 0; j < VEC_SIZE; ++j) { + logits_vec[j] *= inv_temp; + } + + if constexpr (CACHE_INPUT) { + logits_vec.store(smem_vec_base + (i * BLOCK_THREADS + tx) * VEC_SIZE); + } + } + + float thread_max = -cuda::std::numeric_limits::infinity(); +#pragma unroll + for (uint32_t j = 0; j < VEC_SIZE; ++j) { + thread_max = max(thread_max, logits_vec[j]); + } + // Monotonic float keys preserve finite maxima and the -inf padding lanes. + const uint32_t lane = tx & 31u; + const uint32_t warp = tx >> 5; + const uint32_t bits = __float_as_uint(thread_max); + uint32_t key = bits ^ ((bits & 0x80000000u) ? 0xffffffffu : 0x80000000u); + key = __reduce_max_sync(0xffffffffu, key); + if (lane == 0) warp_max_words[i & 1u][warp] = key; + __syncthreads(); + key = __reduce_max_sync(0xffffffffu, warp_max_words[i & 1u][lane]); + float block_max = __uint_as_float(key ^ ((key & 0x80000000u) ? 0x80000000u : 0xffffffffu)); + // if block_max is -inf, then this block contains all -inf values, so we can skip updating + if (!isinf(block_max)) { + float threadlocal_sum = 0.0f; +#pragma unroll + for (uint32_t j = 0; j < VEC_SIZE; ++j) { + threadlocal_sum += __expf(logits_vec[j] - block_max); + } + float new_max = max(running_max, block_max); + threadlocal_running_denominator = + threadlocal_running_denominator * __expf(running_max - new_max) + + threadlocal_sum * __expf(block_max - new_max); + running_max = new_max; + } + } + + running_denominator = cub::BlockReduce(temp_storage.block_prim.reduce) + .Sum(threadlocal_running_denominator); + if (tx == 0) { + temp_storage.shared_state.denominator = running_denominator; + } + __syncthreads(); + running_denominator = temp_storage.shared_state.denominator; + + const float final_max = running_max; + const float inv_denominator = 1.0f / running_denominator; + + // Pass 2: Normalize in place + vec_t prob_vec; + for (uint32_t i = 0; i < ceil_div(d, BLOCK_THREADS * VEC_SIZE); ++i) { + if constexpr (CACHE_INPUT) { + if ((i * BLOCK_THREADS + tx) * VEC_SIZE < d) { + logits_vec.load(smem_vec_base + (i * BLOCK_THREADS + tx) * VEC_SIZE); + } + } else { + if ((i * BLOCK_THREADS + tx) * VEC_SIZE < d) { + logits_vec.cast_load(logits + bx * d + (i * BLOCK_THREADS + tx) * VEC_SIZE); + +#pragma unroll + for (uint32_t j = 0; j < VEC_SIZE; ++j) { + logits_vec[j] *= inv_temp; + } + } + } + +#pragma unroll + for (uint32_t j = 0; j < VEC_SIZE; ++j) { + float p = __expf(static_cast(logits_vec[j]) - final_max) * inv_denominator; + prob_vec[j] = static_cast(p); + } + + if ((i * BLOCK_THREADS + tx) * VEC_SIZE < d) { + prob_vec.cast_store(output + bx * d + (i * BLOCK_THREADS + tx) * VEC_SIZE); + } + } +#if (__CUDACC_VER_MAJOR__ >= 12 && defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)) + asm volatile("griddepcontrol.launch_dependents;"); +#endif +} + +#undef M31_PATHB_LAUNCH_BOUNDS +// clang-format on +} // namespace fi +} // namespace + +cudaError_t invokeSoftmaxPathB( + float* logits, float* output, float* temperature, float temperatureValue, int rows, cudaStream_t stream) +{ + if (rows < kSoftmaxPathBMinRows || rows > kSoftmaxPathBMaxRows) + { + return cudaErrorInvalidValue; + } + constexpr int kThreads = 1024; + constexpr int kVectorSize = 4; + auto kernel = fi::PathBWarpMaxKernel; + constexpr size_t kSharedBytes = sizeof(fi::OnlineSoftmaxTempStorage); + static_assert(kSharedBytes + 256 < 48 * 1024); + void* arguments[] = {&logits, &output, &temperature, &temperatureValue}; + return cudaLaunchKernel(reinterpret_cast(kernel), dim3(static_cast(rows)), dim3(kThreads), + arguments, kSharedBytes, stream); +} +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.h new file mode 100644 index 000000000000..eeec44363e8f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.h @@ -0,0 +1,39 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once +#include "tensorrt_llm/common/config.h" +#include +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +constexpr int kSoftmaxPathBMinRows = 129; +constexpr int kSoftmaxPathBMaxRows = 448; +constexpr int kSoftmaxPathBVocab = 200064; +//! SM107 fp32 softmax for contiguous [129..448, 200064] tensors. Inputs and output must not overlap. +//! A scalar temperature or one fp32 temperature per row; inverse temperature is zero for temperature zero. +//! Preserves FlashInfer Path B's ordered denominator recurrence and final CUB sum. Compile with --use_fast_math. +//! Uses one stream-ordered CTA per row, no workspace and no PDL. All pointers belong to the stream's device. +cudaError_t invokeSoftmaxPathB( + float* logits, float* output, float* temperature, float temperatureValue, int rows, cudaStream_t stream); +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.cu new file mode 100644 index 000000000000..287d8c6fbcab --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.cu @@ -0,0 +1,176 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// SwiGLU-OAI with 16-byte vectors: the in-tree TP2 swigluOai.cu pair +// function (swigluOaiBf16x2, copied verbatim: the Triton kernel's instruction sequence, bitwise) applied to 4 packed +// pairs per thread from one 16-byte load of the gate and one of the up half, with a 32-bit row index. Rows whose +// width is not a multiple of 8 bf16 take the in-tree one-pair-per-thread shape. +#include "tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.h" + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +namespace +{ +constexpr int kThreads = 256; + +__device__ __forceinline__ uint32_t swigluOaiBf16x2(uint32_t gate, uint32_t up) +{ + uint32_t r; + asm("{\n\t.reg .b32 g, glim, gl, gh, ul, uh, one, lim, t0, t1, e0, e1, s0, s1, r0, r1;\n\t.reg .b16 hl, hh;\n\t" + "mov.b32 glim, 0x40E040E0;\n\tmin.bf16x2 g, %1, glim;\n\t" + "mov.b32 {hl, hh}, g;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, %2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mov.b32 lim, 0f40E00000;\n\tmin.xorsign.abs.f32 ul, ul, lim;\n\tmin.xorsign.abs.f32 uh, uh, lim;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 ul, ul, one;\n\tadd.rn.f32 uh, uh, one;\n\t" + "mul.rn.f32 t0, gl, 0fBFD9DB23;\n\tmul.rn.f32 t1, gh, 0fBFD9DB23;\n\t" + "mul.rn.f32 t0, t0, 0f3FB8AA3B;\n\tmul.rn.f32 t1, t1, 0f3FB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "add.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 r0, s0, gl;\n\tmul.rn.f32 r1, s1, gh;\n\t" + "mul.rn.f32 r0, ul, r0;\n\tmul.rn.f32 r1, uh, r1;\n\t" + "cvt.rn.bf16x2.f32 %0, r1, r0;\n\t}" + : "=r"(r) + : "r"(gate), "r"(up)); + return r; +} + +template +__global__ void __launch_bounds__(kThreads) swigluOaiVecKernel( + uint4 const* __restrict__ x, uint4* __restrict__ out, uint32_t rows, uint32_t vecs, uint32_t span) +{ + if constexpr (kPdl) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + // NV output vectors per thread, span apart (span = ceil(rows * vecs / NV)); all loads issued first. + uint32_t const i0 = blockIdx.x * kThreads + threadIdx.x; + uint32_t const total = rows * vecs; + uint4 g[NV], u[NV]; +#pragma unroll + for (int v = 0; v < NV; ++v) + { + uint32_t const i = i0 + v * span; + if (i0 < span && i < total) + { + uint32_t const row = i / vecs; + uint32_t const col = i - row * vecs; + uint4 const* xr = x + static_cast(row) * 2 * vecs; + g[v] = xr[col]; + u[v] = xr[vecs + col]; + } + } +#pragma unroll + for (int v = 0; v < NV; ++v) + { + uint32_t const i = i0 + v * span; + if (i0 < span && i < total) + { + uint4 r; + r.x = swigluOaiBf16x2(g[v].x, u[v].x); + r.y = swigluOaiBf16x2(g[v].y, u[v].y); + r.z = swigluOaiBf16x2(g[v].z, u[v].z); + r.w = swigluOaiBf16x2(g[v].w, u[v].w); + out[i] = r; + } + } +} + +template +__global__ void __launch_bounds__(kThreads) + swigluOaiPairKernel(uint32_t const* __restrict__ x, uint32_t* __restrict__ out, int64_t rows, int64_t halfD) +{ + if constexpr (kPdl) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + int64_t const i = blockIdx.x * static_cast(kThreads) + threadIdx.x; + if (i >= rows * halfD) + { + return; + } + int64_t const row = i / halfD; + int64_t const col = i - row * halfD; + uint32_t const* xr = x + row * 2 * halfD; + out[i] = swigluOaiBf16x2(xr[col], xr[halfD + col]); +} + +template +cudaError_t launch(K kernel, int64_t n, bool pdl, cudaStream_t st, A... args) +{ + cudaLaunchConfig_t cfg{}; + cfg.gridDim = dim3(static_cast((n + kThreads - 1) / kThreads)); + cfg.blockDim = dim3(kThreads); + cfg.stream = st; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, kernel, args...); +} +} // namespace + +cudaError_t invokeSwigluOaiVec(void const* x, void* out, int64_t rows, int64_t d, int nv, bool pdl, cudaStream_t stream) +{ + if (rows == 0) + { + return cudaSuccess; + } + if (d % 2 != 0) + { + return cudaErrorInvalidValue; + } + bool const vec = d % 8 == 0 && reinterpret_cast(x) % 16 == 0 && rows * (d / 8) < (int64_t{1} << 32); + if (vec) + { + if (nv != 1 && nv != 2 && nv != 4) + { + return cudaErrorInvalidValue; + } + auto const* xp = reinterpret_cast(x); + auto* op = reinterpret_cast(out); + uint32_t const r = static_cast(rows), v = static_cast(d / 8); + int64_t const total = rows * (d / 8); + uint32_t const span = static_cast((total + nv - 1) / nv); + if (nv == 1) + return pdl ? launch(swigluOaiVecKernel, span, true, stream, xp, op, r, v, span) + : launch(swigluOaiVecKernel, span, false, stream, xp, op, r, v, span); + if (nv == 2) + return pdl ? launch(swigluOaiVecKernel, span, true, stream, xp, op, r, v, span) + : launch(swigluOaiVecKernel, span, false, stream, xp, op, r, v, span); + return pdl ? launch(swigluOaiVecKernel, span, true, stream, xp, op, r, v, span) + : launch(swigluOaiVecKernel, span, false, stream, xp, op, r, v, span); + } + auto const* xp = reinterpret_cast(x); + auto* op = reinterpret_cast(out); + return pdl ? launch(swigluOaiPairKernel, rows * (d / 2), true, stream, xp, op, rows, d / 2) + : launch(swigluOaiPairKernel, rows * (d / 2), false, stream, xp, op, rows, d / 2); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.h new file mode 100644 index 000000000000..ef096f95abdd --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.h @@ -0,0 +1,42 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! SwiGLU-OAI (limit 7, alpha 1.702, beta 1) of x = [gate | up] bf16 [rows, 2d] -> out bf16 [rows, d], bit for bit the +//! Triton silu_and_mul kernel (the in-tree swigluOai pair sequence: ex2.approx, div.full, one rounding). d % 8 == 0 and +//! a 16-byte aligned x take 16-byte vectors with nv (1 / 2 / 4) output vectors per thread; otherwise one packed pair +//! per thread. d must be even. PDL: waits on its predecessor, releases its dependents early. +cudaError_t invokeSwigluOaiVec( + void const* x, void* out, int64_t rows, int64_t d, int nv, bool pdl, cudaStream_t stream); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.cu new file mode 100644 index 000000000000..abda5ccf5a97 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.cu @@ -0,0 +1,111 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +namespace +{ + +// One block per row; 16-byte copies when the row allows them (rank % 8 == 0 and aligned rows), else element copies. +__global__ void markovPrevEmbedKernel( + int64_t const* prev, __nv_bfloat16 const* w1, int64_t vocab, int32_t rank, __nv_bfloat16* out) +{ + int32_t const i = blockIdx.x; + int64_t const p = prev[i]; + bool const valid = p >= 0 && p < vocab; + __nv_bfloat16* dst = out + static_cast(i) * rank; + __nv_bfloat16 const* src = w1 + (valid ? p : 0) * static_cast(rank); + bool const vec = (rank % 8) == 0 && (reinterpret_cast(src) % 16) == 0 + && (reinterpret_cast(dst) % 16) == 0; + if (vec) + { + int32_t const nv = rank / 8; + for (int32_t c = threadIdx.x; c < nv; c += blockDim.x) + { + uint4 v = valid ? reinterpret_cast(src)[c] : make_uint4(0u, 0u, 0u, 0u); + reinterpret_cast(dst)[c] = v; + } + } + else + { + for (int32_t c = threadIdx.x; c < rank; c += blockDim.x) + { + dst[c] = valid ? src[c] : __float2bfloat16_rn(0.0f); + } + } +} + +__global__ void expandBlockSamplingKernel(float const* temps, int32_t const* topKs, float const* topPs, + float const* minPs, int64_t const* seeds, int64_t const* offsets, int32_t n, int32_t K, int64_t slotOffset, + int64_t span, float* tempsOut, int32_t* topKsOut, float* topPsOut, float* minPsOut, int64_t* seedsOut, + int64_t* offsetsOut) +{ + int32_t const r = blockIdx.x * blockDim.x + threadIdx.x; + if (r >= n * K) + { + return; + } + int32_t const i = r / K; + int32_t const j = r - i * K; + tempsOut[r] = temps[i]; + topKsOut[r] = topKs[i]; + topPsOut[r] = topPs[i]; + minPsOut[r] = minPs[i]; + seedsOut[r] = seeds[i]; + // torch: (offsets + slot * SPAN).repeat_interleave(K) + arange(K).repeat(n) * SPAN, int64 two's-complement wrap. + uint64_t const o = static_cast(offsets[i]) + static_cast(slotOffset) + + static_cast(static_cast(j) * span); + offsetsOut[r] = static_cast(o); +} + +} // namespace + +void invokeMarkovPrevEmbed(int64_t const* prev, int32_t n, __nv_bfloat16 const* w1, int64_t vocab, int32_t rank, + __nv_bfloat16* out, cudaStream_t stream) +{ + if (n <= 0 || rank <= 0) + { + return; + } + int32_t const threads = rank / 8 >= 128 ? 128 : 64; + markovPrevEmbedKernel<<>>(prev, w1, vocab, rank, out); +} + +void invokeExpandBlockSampling(float const* temps, int32_t const* topKs, float const* topPs, float const* minPs, + int64_t const* seeds, int64_t const* offsets, int32_t n, int32_t K, int64_t slotOffset, int64_t span, + float* tempsOut, int32_t* topKsOut, float* topPsOut, float* minPsOut, int64_t* seedsOut, int64_t* offsetsOut, + cudaStream_t stream) +{ + int32_t const total = n * K; + if (total <= 0) + { + return; + } + int32_t const threads = 128; + expandBlockSamplingKernel<<<(total + threads - 1) / threads, threads, 0, stream>>>(temps, topKs, topPs, minPs, + seeds, offsets, n, K, slotOffset, span, tempsOut, topKsOut, topPsOut, minPsOut, seedsOut, offsetsOut); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.h new file mode 100644 index 000000000000..e7ec8b18ce66 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.h @@ -0,0 +1,51 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! TAIL_BK3 (B): the S4FE position-0 Markov embedding in one launch, bitwise the in-tree +//! ``F.embedding(masked prev, w1).masked_fill(invalid, 0)``: out[i, :] = w1[prev[i], :] when 0 <= prev[i] < vocab, else +//! +0. prev: int64 [n]; w1: contiguous bf16 [vocab, rank]; out: contiguous bf16 [n, rank]. Graph-safe. +void invokeMarkovPrevEmbed(int64_t const* prev, int32_t n, __nv_bfloat16 const* w1, int64_t vocab, int32_t rank, + __nv_bfloat16* out, cudaStream_t stream); + +//! TAIL_BK3 (C): the block draft sampler's per-row parameters in one launch, bitwise the in-tree +//! ``request_*[start:end].repeat_interleave(K)`` and ``_rng_state_per_request(..., repeat=K, slot=s)``. For row +//! r = i * K + j (i < n requests, j < K): temps / topKs / topPs / minPs [r] = the request's value; seeds[r] = seeds[i]; +//! offsets[r] = offsets[i] + slotOffset + j * span (int64, wrapping as torch does). Inputs are the request slices +//! (contiguous, n entries); outputs are contiguous [n * K]. Graph-safe. +void invokeExpandBlockSampling(float const* temps, int32_t const* topKs, float const* topPs, float const* minPs, + int64_t const* seeds, int64_t const* offsets, int32_t n, int32_t K, int64_t slotOffset, int64_t span, + float* tempsOut, int32_t* topKsOut, float* topPsOut, float* minPsOut, int64_t* seedsOut, int64_t* offsetsOut, + cudaStream_t stream); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu index 3ba14d32938d..2f16fe10712f 100644 --- a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu +++ b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu @@ -16,8 +16,9 @@ */ // TOPP_DET v3: toppRenormDetSm103.cu (v1.2) with the redundant selection and an optional programmatic dependent -// launch. See toppRenormDetV3Sm103.h for the contract (bitwise v1.2's output). Compiled with --use_fast_math (set on -// this source in kernels/CMakeLists.txt), as FlashInfer's JIT compiles it. +// launch; V4K8: the one-sync list path and 8 privatized pass histograms (see the header). See toppRenormDetV3Sm103.h +// for the contract (bitwise v1.2's output). Compiled with --use_fast_math (set on this source in +// kernels/CMakeLists.txt), as FlashInfer's JIT compiles it. #include "tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h" @@ -26,8 +27,12 @@ #include #include +#include #include #include +#include +#include +#include #include @@ -144,13 +149,23 @@ namespace cg = cooperative_groups; constexpr int kThreads = 1024; constexpr int kBuckets = fa::NUM_BUCKETS; // 2048 constexpr int kSmemBytes = 32768; -constexpr int kCopies = 4; // privatized pass histograms -constexpr size_t kDynSmemBytes = kCopies * kBuckets * (sizeof(unsigned long long) + sizeof(int)); // 96 KB + +// Privatized pass histograms per CTA: kCopies (a kernel template parameter) copies, one per group of warps; 8 (V4K8, +// default) or 4 (TRTLLM_M31_DISABLE_TOPPDET_K8). +template +constexpr size_t dynSmemBytes() +{ + return kCopies * kBuckets * (sizeof(unsigned long long) + sizeof(int)); +} constexpr int kSyncs = fa::NUM_PASSES + 1; // one per radix pass, then the norm // Candidate list: passes 1 .. NUM_PASSES - 1 run locally on the members of pass 0's threshold bucket when it holds at // most this many elements (the list lives in the workspace, then in the dynamic shared memory of each CTA). constexpr int kCandMax = 4096; +// V4 list path: pass 0 also lists the elements >= 2^-12 (a pass-0 bucket edge; probabilities summing to 1 hold at most +// 4096 of them) with their indices; pass-0 histograms and lists are double-buffered by launch parity. +constexpr int kListMax = 4096; +constexpr float kListMin = 0.000244140625f; // 2^-12 // One row's persistent state. Zero once; every launch leaves it zeroed except the arrival counters, which only grow. struct RowWorkspace @@ -163,6 +178,12 @@ struct RowWorkspace unsigned nCand; // its length, zero between launches int kept; int pad[2]; + unsigned long long hist0x[2][kBuckets]; + int cnt0x[2][kBuckets]; + float lval[2][kListMax]; + int lidx[2][kListMax]; + unsigned nList[2]; + unsigned pad2[2]; }; // Arrive at sync `si` and wait until all G CTAs of the row have arrived. The counter is never reset: each CTA adds @@ -293,7 +314,133 @@ __device__ __forceinline__ void clearPass(RowWorkspace* w, int pass) } } -template +// V4 list path, run by every CTA of the row after pass 0 (one barrier so far) when the pass-0 threshold bucket lies at +// or above 2^-12: then every element of that bucket and above, so every kept element, is in the row's list. The +// remaining radix passes run on the list (the same integer histograms as v3's candidate passes), FlashInfer's 1024 +// chain sums are rebuilt from the kept members (chain c = the kept values at indices c, c + 1024, ... in ascending +// order, as v3's chains), reduced by the same cub block sum, and each CTA writes its own slice: zeros, then its kept +// values. +template +__device__ __noinline__ void listPath(RowWorkspace* w, int par, int nL, unsigned kbits, float p, float const* in, + float* o, int d, int row, int cta, int G, int* keptOut, unsigned long long* hsCopies, int* csCopies, + unsigned char* smem, float* sSum, unsigned* sKbits) +{ + int const tid = threadIdx.x; + float* lv = reinterpret_cast(hsCopies + kBuckets); // 16 KB + int* li = reinterpret_cast(hsCopies + 2 * kBuckets); // 16 KB + for (int i = tid; i < nL; i += kThreads) + { + lv[i] = __ldcg(&w->lval[par][i]); + li[i] = __ldcg(&w->lidx[par][i]); + } + __syncthreads(); + for (int lp = 1; lp < fa::NUM_PASSES; ++lp) + { + for (int i = tid; i < kBuckets; i += kThreads) + { + hsCopies[i] = 0; + csCopies[i] = 0; + } + __syncthreads(); + int const lsb = fa::calcStartBit(lp); + uint32_t const lmask = fa::calcMask(lp); + int const lpsb = fa::calcStartBit(lp - 1); + for (int i = tid; i < nL; i += kThreads) + { + float const x = lv[i]; + auto const bits = fa::twiddleIn(x, false); + if (((bits >> lpsb) << lpsb) == kbits) + { + int const b = static_cast((bits >> lsb) & lmask); + atomicAdd(hsCopies + b, static_cast(fa::calcMantissa(x))); + atomicAdd(csCopies + b, 1); + } + } + __syncthreads(); + selectPassOn(hsCopies, csCopies, lp, p, reinterpret_cast(smem), + reinterpret_cast(smem + kBuckets * sizeof(float)), sSum, sKbits); + kbits = *sKbits; + } + float const thr = fa::twiddleOut(kbits, false); + // Compact the kept members (csCopies is free now: 32 KB = 4096 values + 4096 indices). + float* kv = reinterpret_cast(csCopies); + int* ki = csCopies + kListMax; + __shared__ int nK; + if (tid == 0) + { + nK = 0; + } + __syncthreads(); + for (int i = tid; i < nL; i += kThreads) + { + if (lv[i] >= thr) + { + int const s = atomicAdd(&nK, 1); + kv[s] = lv[i]; + ki[s] = li[i]; + } + } + __syncthreads(); + int const nKept = nK; + // Chain tid: its kept values in ascending index order, accumulated from 0 as v3's chains. + float acc = 0.0f; + int last = -1; + while (true) + { + int best = INT_MAX; + float bv = 0.0f; + for (int i = 0; i < nKept; ++i) + { + int const idx = ki[i]; + if ((idx & (kThreads - 1)) == tid && idx > last && idx < best) + { + best = idx; + bv = kv[i]; + } + } + if (best == INT_MAX) + { + break; + } + acc += bv; + last = best; + } + typedef cub::BlockReduce BlockReduce; + __shared__ typename BlockReduce::TempStorage temp; + __shared__ float sNormL; + float const totalSum = BlockReduce(temp).Sum(acc); + if (tid == 0) + { + sNormL = (totalSum > 1e-8f) ? (1.0f / totalSum) : 1.0f; + if (cta == 0 && keptOut) + { + keptOut[row] = nKept; + } + } + __syncthreads(); + if constexpr (kPdl) + { + cudaTriggerProgrammaticLaunchCompletion(); + } + float const norm = sNormL; + float4* o4 = reinterpret_cast(o); + int const n4 = d / 4; + for (int j = cta * kThreads + tid; j < n4; j += G * kThreads) + { + o4[j] = make_float4(0.f, 0.f, 0.f, 0.f); + } + __syncthreads(); + for (int i = tid; i < nKept; i += kThreads) + { + int const idx = ki[i]; + if (((idx / 4) / kThreads) % G == cta) + { + o[idx] = kv[i] * norm; + } + } +} + +template __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( float const* __restrict__ probs, float* __restrict__ out, float const* topP, int d, RowWorkspace* ws, int* keptOut) { @@ -323,6 +470,19 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( cudaGridDependencySynchronize(); } __syncthreads(); + // V4: launch index from the grow-only pass-0 counter (every earlier launch has completed; this launch adds less + // than kThreads until all of its CTAs arrive), so every CTA of the row gets the same parity. + int const par = static_cast((__ldcg(&w->arrive[0]) / kThreads) & 1ull); + // Spread over the row's CTAs (each clears 2048 / G buckets), so no CTA's binning is delayed. + for (int i = cta * kThreads + tid; i < kBuckets; i += G * kThreads) + { + w->hist0x[par ^ 1][i] = 0; + w->cnt0x[par ^ 1][i] = 0; + } + if (cta == 0 && tid == 0) + { + w->nList[par ^ 1] = 0; + } float const p = topP[row]; float4 const* in4 = reinterpret_cast(in); int const n4 = d / 4; @@ -346,6 +506,13 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( int const copy = (tid >> 5) % kCopies; unsigned long long* myHs = hsCopies + copy * kBuckets; int* myCs = csCopies + copy * kBuckets; + // S32 (with the 8 copies): the 64-bit mantissa-sum space holds two 32-bit sums per bucket instead (hi = m >> + // 11, then lo = m & 0x7ff), two cheap shared atomics in place of a contended 64-bit one; the merge rebuilds the + // exact sum (hi << 11) + lo. Per copy and bucket at most d / G elements, so both stay below 2^32 while d / G < + // 2^20. + bool const s32 = kCopies == 8 && d / G < (1 << 20); + unsigned* myHi = reinterpret_cast(hsCopies) + copy * kBuckets; + unsigned* myLo = reinterpret_cast(hsCopies) + kCopies * kBuckets + copy * kBuckets; int const sb = fa::calcStartBit(pass); uint32_t const mask = fa::calcMask(pass); int const psb = pass == 0 ? 0 : fa::calcStartBit(pass - 1); @@ -361,6 +528,32 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( for (int k = 0; k < 4; ++k) { float const x = e[k]; + if (kList && pass == 0) + { + // Warp-aggregated append: one global atomic per warp and element slot (the list's order is + // irrelevant: integer histograms, index-ordered chains). + bool const want = valid && x >= kListMin; + unsigned const m = __ballot_sync(0xffffffffu, want); + if (m) + { + int const leader = __ffs(m) - 1; + unsigned base = 0; + if (lane == leader) + { + base = atomicAdd(&w->nList[par], static_cast(__popc(m))); + } + base = __shfl_sync(0xffffffffu, base, leader); + if (want) + { + unsigned const slot = base + __popc(m & ((1u << lane) - 1u)); + if (slot < kListMax) + { + w->lval[par][slot] = x; + w->lidx[par][slot] = 4 * j + k; + } + } + } + } auto const bits = fa::twiddleIn(x, false); bool const ok = valid && (pass == 0 || ((bits >> psb) << psb) == kbits); int const b = ok ? static_cast((bits >> sb) & mask) : -1; @@ -373,17 +566,39 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( { if (b0 >= 0) { - unsigned const s = __reduce_add_sync(0xffffffffu, m); - if (lane == 0) + if (s32) { - atomicAdd(myHs + b0, static_cast(s)); - atomicAdd(myCs + b0, 32); + unsigned const sh = __reduce_add_sync(0xffffffffu, m >> 11); + unsigned const sl = __reduce_add_sync(0xffffffffu, m & 0x7ffu); + if (lane == 0) + { + atomicAdd(myHi + b0, sh); + atomicAdd(myLo + b0, sl); + atomicAdd(myCs + b0, 32); + } + } + else + { + unsigned const s = __reduce_add_sync(0xffffffffu, m); + if (lane == 0) + { + atomicAdd(myHs + b0, static_cast(s)); + atomicAdd(myCs + b0, 32); + } } } } else if (ok) { - atomicAdd(myHs + b, static_cast(m)); + if (s32) + { + atomicAdd(myHi + b, m >> 11); + atomicAdd(myLo + b, m & 0x7ffu); + } + else + { + atomicAdd(myHs + b, static_cast(m)); + } atomicAdd(myCs + b, 1); } } @@ -393,10 +608,13 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( { unsigned long long h = 0; int c = 0; + unsigned const* hiS = reinterpret_cast(hsCopies); #pragma unroll for (int k = 0; k < kCopies; ++k) { - h += hsCopies[k * kBuckets + i]; + h += s32 ? (static_cast(hiS[k * kBuckets + i]) << 11) + + hiS[kCopies * kBuckets + k * kBuckets + i] + : hsCopies[k * kBuckets + i]; c += csCopies[k * kBuckets + i]; } hs[i] = h; @@ -407,21 +625,40 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( { if (hs[i] != 0) { - atomicAdd(&w->hist[pass][i], hs[i]); + atomicAdd(pass == 0 ? &w->hist0x[par][i] : &w->hist[pass][i], hs[i]); } if (cs[i] != 0) { - atomicAdd(&w->cnt[pass][i], cs[i]); + atomicAdd(pass == 0 ? &w->cnt0x[par][i] : &w->cnt[pass][i], cs[i]); } } allArrive(w, G, pass); - if (pass > 0) + if (pass > 1) { clearPass(w, pass - 1); // every CTA finished selectPass(pass - 1) before arriving here } - selectPass(w, pass, p, reinterpret_cast(smem), reinterpret_cast(smem + kBuckets * sizeof(float)), - &sSum, &sKbits); + if (pass == 0) + { + selectPassOn(w->hist0x[par], w->cnt0x[par], 0, p, reinterpret_cast(smem), + reinterpret_cast(smem + kBuckets * sizeof(float)), &sSum, &sKbits); + } + else + { + selectPass(w, pass, p, reinterpret_cast(smem), + reinterpret_cast(smem + kBuckets * sizeof(float)), &sSum, &sKbits); + } kbits = sKbits; + if (kList && pass == 0) + { + unsigned const nL = __ldcg(&w->nList[par]); + float const lowEdge = fa::twiddleOut(kbits | ((1u << fa::calcStartBit(0)) - 1u), false); + if (nL <= static_cast(kListMax) && lowEdge >= kListMin) + { + listPath(w, par, static_cast(nL), kbits, p, in, o, d, row, cta, G, keptOut, hsCopies, + csCopies, smem, &sSum, &sKbits); + return; + } + } if constexpr (kCandList) { // Pass 0's threshold bucket count (cached by selectPass) is the same in every CTA of the row, so all of @@ -448,7 +685,6 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( } } allArrive(w, G, 1); - clearPass(w, 0); // every CTA finished selectPass(0) before arriving here // The list in this CTA's dynamic shared memory (privatized copy 1's space), the local histograms in // copy 0's: the same integer histograms the global passes 1 .. NUM_PASSES - 1 would build. float* candS = reinterpret_cast(hsCopies + kBuckets); @@ -499,7 +735,49 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( float* stage = reinterpret_cast(smem); float acc = 0.0f; int kept = 0; - for (int k0 = 0; k0 < kTotal; k0 += kTile) + // TOPP_V6 (kChainPf): each chain loads its own values straight from global memory, kPf loads in flight, and adds + // them in the same ascending-k order as the staged loop (the same values, the same fp32 adds and kept counts, so + // the same bits); no shared staging and no per-tile barriers. Kill TRTLLM_M31_DISABLE_TOPP_V6=1 (the staged loop). + if constexpr (kChainPf) + { + constexpr int kPf = 16; + if (tid < cpc) + { + int const cbase = c0 + tid; + int k = 0; + for (; k + kPf <= kTotal; k += kPf) + { + float v[kPf]; +#pragma unroll + for (int u = 0; u < kPf; ++u) + { + int const i = cbase + kThreads * (k + u); + v[u] = i < d ? __ldg(in + i) : 0.0f; + } +#pragma unroll + for (int u = 0; u < kPf; ++u) + { + int const i = cbase + kThreads * (k + u); + if (i < d && v[u] >= thr) + { + acc += v[u]; + ++kept; + } + } + } + for (; k < kTotal; ++k) + { + int const i = cbase + kThreads * k; + float const val = i < d ? __ldg(in + i) : 0.0f; + if (i < d && val >= thr) + { + acc += val; + ++kept; + } + } + } + } + for (int k0 = 0; !kChainPf && k0 < kTotal; k0 += kTile) { int const kn = min(kTile, kTotal - k0); for (int q = tid; q < kn * cpc; q += kThreads) @@ -593,8 +871,56 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( } // namespace toppv3 +namespace +{ +// TOPP_V6: on unless TRTLLM_M31_DISABLE_TOPP_V6 is set to 1 / true / on / yes (read once per process). +bool chainPrefetchOn() +{ + static bool const on = [] + { + char const* v = std::getenv("TRTLLM_M31_DISABLE_TOPP_V6"); + if (v == nullptr) + { + return true; + } + return !(std::strcmp(v, "1") == 0 || strcasecmp(v, "true") == 0 || strcasecmp(v, "on") == 0 + || strcasecmp(v, "yes") == 0); + }(); + return on; +} + +template +cudaError_t launchV3(cudaLaunchConfig_t& cfg, float const* probs, float* out, float const* topP, int d, + toppv3::RowWorkspace* ws, int* keptOut) +{ + auto* kernel = chainPrefetchOn() ? toppv3::toppRenormDetV3Kernel + : toppv3::toppRenormDetV3Kernel; + static cudaError_t const attrErr = cudaFuncSetAttribute( + kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(toppv3::dynSmemBytes())); + if (attrErr != cudaSuccess) + { + return attrErr; + } + cfg.dynamicSmemBytes = toppv3::dynSmemBytes(); + return cudaLaunchKernelEx(&cfg, kernel, probs, out, topP, d, ws, keptOut); +} + +template +cudaError_t launchV3Copies(cudaLaunchConfig_t& cfg, bool listPath, bool wideHist, float const* probs, float* out, + float const* topP, int d, toppv3::RowWorkspace* ws, int* keptOut) +{ + if (listPath) + { + return wideHist ? launchV3(cfg, probs, out, topP, d, ws, keptOut) + : launchV3(cfg, probs, out, topP, d, ws, keptOut); + } + return wideHist ? launchV3(cfg, probs, out, topP, d, ws, keptOut) + : launchV3(cfg, probs, out, topP, d, ws, keptOut); +} +} // namespace + cudaError_t invokeToppRenormDetV3(float const* probs, float* out, float const* topP, int rows, int d, int ctasPerRow, - void* workspace, int* keptOut, bool pdl, bool candidateList, cudaStream_t stream) + void* workspace, int* keptOut, bool pdl, bool candidateList, bool listPath, bool wideHist, cudaStream_t stream) { using namespace toppv3; if (rows <= 0 || d <= 0) @@ -605,28 +931,9 @@ cudaError_t invokeToppRenormDetV3(float const* probs, float* out, float const* t { return cudaErrorInvalidValue; } - static cudaError_t const attrErr = [] - { - for (auto* k : {toppRenormDetV3Kernel, toppRenormDetV3Kernel, - toppRenormDetV3Kernel, toppRenormDetV3Kernel}) - { - cudaError_t const e - = cudaFuncSetAttribute(k, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(kDynSmemBytes)); - if (e != cudaSuccess) - { - return e; - } - } - return cudaSuccess; - }(); - if (attrErr != cudaSuccess) - { - return attrErr; - } cudaLaunchConfig_t cfg{}; cfg.gridDim = dim3(static_cast(ctasPerRow), static_cast(rows)); cfg.blockDim = dim3(kThreads); - cfg.dynamicSmemBytes = kDynSmemBytes; cfg.stream = stream; cudaLaunchAttribute attr[1]; attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; @@ -634,9 +941,13 @@ cudaError_t invokeToppRenormDetV3(float const* probs, float* out, float const* t cfg.attrs = attr; cfg.numAttrs = 1; auto* ws = static_cast(workspace); - auto* kernel = pdl ? (candidateList ? toppRenormDetV3Kernel : toppRenormDetV3Kernel) - : (candidateList ? toppRenormDetV3Kernel : toppRenormDetV3Kernel); - return cudaLaunchKernelEx(&cfg, kernel, probs, out, topP, d, ws, keptOut); + if (pdl) + { + return candidateList ? launchV3Copies(cfg, listPath, wideHist, probs, out, topP, d, ws, keptOut) + : launchV3Copies(cfg, listPath, wideHist, probs, out, topP, d, ws, keptOut); + } + return candidateList ? launchV3Copies(cfg, listPath, wideHist, probs, out, topP, d, ws, keptOut) + : launchV3Copies(cfg, listPath, wideHist, probs, out, topP, d, ws, keptOut); } size_t toppRenormDetV3WorkspaceBytesPerRow() diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h index f7a08d3fb619..155467d559de 100644 --- a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h +++ b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.h @@ -44,10 +44,19 @@ namespace minimaxM31 //! bits); larger buckets take the global passes. //! pdl: launched as a programmatic dependent; everything that reads probs, top_p or the workspace waits for the //! preceding grid, and dependents are released once the workspace is ready for the next launch. +//! listPath (V4): pass 0 also lists the elements >= 2^-12 (at most 4096 in a row summing to 1; 2^-12 is a pass-0 bucket +//! edge). When the list is complete and the selected pass-0 bucket lies at or above 2^-12, every kept element is in +//! it: after that single barrier every CTA runs the remaining radix passes on the list (the same integer histograms), +//! rebuilds FlashInfer's 1024 chain sums from the kept members (chain c = the kept values at indices c, c + 1024, ... +//! in ascending order), reduces them with the same cub block sum and writes its own slice. Otherwise the radix path +//! above. Pass-0 histograms and lists are double-buffered by launch parity (from the grow-only pass-0 counter), so no +//! later barrier has to clear them. +//! wideHist (K8): 8 privatized pass histograms per CTA instead of 4, each bucket's mantissa sum kept as two 32-bit +//! shared counters (hi / lo bits) while d / ctasPerRow < 2^20 (integer merges, so the same bits). //! Workspace: toppRenormDetV3WorkspaceBytesPerRow() x rows bytes, ZERO-initialized once before the first launch, -//! then reused as is (CUDA-graph safe). Launches sharing one workspace must be stream-ordered. +//! then reused as is (CUDA-graph safe) by any mix of the flags. Launches sharing one workspace must be stream-ordered. cudaError_t invokeToppRenormDetV3(float const* probs, float* out, float const* topP, int rows, int d, int ctasPerRow, - void* workspace, int* keptOut, bool pdl, bool candidateList, cudaStream_t stream); + void* workspace, int* keptOut, bool pdl, bool candidateList, bool listPath, bool wideHist, cudaStream_t stream); //! Workspace bytes per row of invokeToppRenormDetV3. size_t toppRenormDetV3WorkspaceBytesPerRow(); diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.cu b/cpp/tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.cu new file mode 100644 index 000000000000..fd3b12327960 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.cu @@ -0,0 +1,64 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ +namespace +{ + +constexpr int kThreads = 256; + +// One CTA per (row, chunk): a row belongs to request row / rowsPerRequest; it is zeroed only when that request's slot +// id is the CUDA-graph padding marker. A real row is never touched, so at a full batch the kernel writes nothing. +__global__ void __launch_bounds__(kThreads) zeroPaddingRowsKernel(float* __restrict__ logits, int64_t rowStride, + int64_t const* __restrict__ slotIds, int64_t dummySlotRow, int32_t rowsPerRequest, int32_t d, int32_t chunk) +{ + int64_t const row = blockIdx.y; + if (slotIds[row / rowsPerRequest] != dummySlotRow) + { + return; + } + int32_t const begin = blockIdx.x * chunk; + int32_t const end = min(d, begin + chunk); + float* dst = logits + row * rowStride; + for (int32_t i = begin + threadIdx.x; i < end; i += kThreads) + { + dst[i] = 0.0f; + } +} + +} // namespace + +void invokeZeroPaddingRows(float* logits, int64_t rowStride, int64_t const* slotIds, int64_t dummySlotRow, int32_t rows, + int32_t rowsPerRequest, int32_t d, cudaStream_t stream) +{ + int32_t const chunk = 16384; + dim3 const grid((d + chunk - 1) / chunk, rows); + zeroPaddingRowsKernel<<>>( + logits, rowStride, slotIds, dummySlotRow, rowsPerRequest, d, chunk); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.h b/cpp/tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.h new file mode 100644 index 000000000000..3176aeaa2448 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.h @@ -0,0 +1,43 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! TAIL16-G2: the speculative worker's _zero_padding_rows (uniform layout) in place and in one launch: +//! logits[r, :] = 0.0f iff slotIds[r / rowsPerRequest] == dummySlotRow +//! for r in 0..rows-1 (slotIds already offset to the first generation request). Stock: an eq, a repeat_interleave and +//! an out-of-place masked_fill over every row. Bitwise the stock result: real rows are untouched, padding rows are +//! +0.0. logits: fp32 rows of d with row stride rowStride elements. Graph-safe (no host state). +void invokeZeroPaddingRows(float* logits, int64_t rowStride, int64_t const* slotIds, int64_t dummySlotRow, int32_t rows, + int32_t rowsPerRequest, int32_t d, cudaStream_t stream); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.cu new file mode 100644 index 000000000000..a66c20a0c867 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.cu @@ -0,0 +1,267 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// The TP2 dense layers' QK norm + RoPE with FP8 output. The norm and RoPE are the generic fusedQKNormRopeKernel's at +// head_dim 128 without YaRN or mRoPE, expression for expression; the store rounds Q/K to BF16 and then to E4M3 +// (saturating), the two roundings of the stock BF16 kernel followed by its FP8 cast. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +namespace denseQkNormRopeFp8 +{ +namespace +{ + +constexpr int kElemsPerThread = kHeadDim / 32; +constexpr int kVecSize = kElemsPerThread * sizeof(__nv_bfloat16) / 4; +constexpr int kBlockSize = 256; +static_assert(kVecSize == 2, "a lane loads 4 BF16 values as one uint2"); + +// tensorrt_llm::common::warpReduceSum: the same butterfly, so the same sum. +__device__ __forceinline__ float warpReduceSum(float val) +{ +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + { + val = val + __shfl_xor_sync(0xffffffff, val, mask, 32); + } + return val; +} + +__device__ __forceinline__ void storeFp8ViaBf16(__nv_fp8_e4m3* out, long long offsetThread, float const (&elements)[4]) +{ +#pragma unroll + for (int i = 0; i < kElemsPerThread; i += 2) + { + float2 const rounded = __bfloat1622float2(__float22bfloat162_rn(make_float2(elements[i], elements[i + 1]))); + __nv_fp8x2_e4m3 const packed(rounded); + reinterpret_cast<__nv_fp8x2_storage_t*>(&out[offsetThread])[i / 2] = packed.__x; + } +} + +// The K (kv 0) or V (kv 1) row of one head of token tokenIdx in the HND paged pool, at the token's cache slot. +__device__ __forceinline__ __nv_fp8_e4m3* cacheRow(KvCacheView const& cache, int kv, int headIdx, int tokenIdx) +{ + long long const slot = static_cast(cache.outCacheLoc[tokenIdx]); + long long const page = slot / cache.tokensPerBlock; + long long const within = slot % cache.tokensPerBlock; + return cache.base + page * cache.pageStride + kv * cache.kvStride + headIdx * cache.headStride + + within * cache.tokenStride; +} + +template +__global__ void denseQkNormRopeFp8Kernel(__nv_bfloat16 const* qkvIn, __nv_fp8_e4m3* qkvOut, int numHeadsQ, + int numHeadsK, int numHeadsV, int rotaryDim, float eps, __nv_bfloat16 const* qWeight, __nv_bfloat16 const* kWeight, + float base, int const* positionIds, int numTokens, bool useGemma, KvCacheView cache) +{ + int const warpsPerBlock = blockDim.x / 32; + int const warpId = threadIdx.x / 32; + int const laneId = threadIdx.x % 32; + int const globalWarpIdx = blockIdx.x * warpsPerBlock + warpId; + int const totalQkHeads = numHeadsQ + numHeadsK; + int const totalHeads = totalQkHeads + numHeadsV; + int const tokenIdx = globalWarpIdx / totalHeads; + int const localHeadIdx = globalWarpIdx % totalHeads; + if (tokenIdx >= numTokens) + { + return; + } + bool const isQ = localHeadIdx < numHeadsQ; + bool const isV = localHeadIdx >= totalQkHeads; + int headIdx; + int segStart; + if (isQ) + { + headIdx = localHeadIdx; + segStart = 0; + } + else if (!isV) + { + headIdx = localHeadIdx - numHeadsQ; + segStart = numHeadsQ * kHeadDim; + } + else + { + headIdx = localHeadIdx - totalQkHeads; + segStart = totalQkHeads * kHeadDim; + } + + using vec_T = uint2; + int const offsetThread + = tokenIdx * totalHeads * kHeadDim + segStart + headIdx * kHeadDim + laneId * kElemsPerThread; + float elements[kElemsPerThread]; + float sumOfSquares = 0.0f; + { + vec_T vec = *reinterpret_cast(&qkvIn[offsetThread]); +#pragma unroll + for (int i = 0; i < kVecSize; i++) + { + float2 vals = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162*>(reinterpret_cast(&vec) + i)); + sumOfSquares += vals.x * vals.x; + sumOfSquares += vals.y * vals.y; + elements[2 * i] = vals.x; + elements[2 * i + 1] = vals.y; + } + } + if (isV) + { + // V is BF16 already: its single E4M3 rounding is the stock cast's. + storeFp8ViaBf16(qkvOut, offsetThread, elements); + if (kWriteCache && tokenIdx < cache.numLiveTokens && cache.outCacheLoc[tokenIdx] >= 0) + { + storeFp8ViaBf16(cacheRow(cache, 1, headIdx, tokenIdx), laneId * kElemsPerThread, elements); + } + return; + } + + sumOfSquares = warpReduceSum(sumOfSquares); + float rms_rcp = rsqrtf(sumOfSquares / static_cast(kHeadDim) + eps); + for (int i = 0; i < kElemsPerThread; i++) + { + int dim = laneId * kElemsPerThread + i; + float weight = isQ ? __bfloat162float(qWeight[dim]) : __bfloat162float(kWeight[dim]); + elements[i] *= rms_rcp * (useGemma ? (1.0f + weight) : weight); + } + + float elements2[kElemsPerThread]; + float cos_vals[kElemsPerThread]; + float sin_vals[kElemsPerThread]; + float const neg2_log2base_over_rd = -2.0f * __log2f(base) / static_cast(rotaryDim); + int const rd_mask = rotaryDim - 1; + bool const rd_is_pow2 = ((rotaryDim & rd_mask) == 0); + float const pos_id = static_cast(positionIds[tokenIdx]); + if constexpr (kInterleave) + { + for (int i = 0; i < kElemsPerThread; i++) + { + if (i % 2 == 0) + { + elements2[i] = -elements[i + 1]; + } + else + { + elements2[i] = elements[i - 1]; + } + int dim_idx = laneId * kElemsPerThread + i; + int half_dim = dim_idx / 2; + float freq = exp2f(static_cast(half_dim) * neg2_log2base_over_rd); + float theta = pos_id * freq; + __sincosf(theta, &sin_vals[i], &cos_vals[i]); + } + } + else + { + __syncwarp(); + int pairOffset = (rotaryDim / 2) / kElemsPerThread; + for (int i = 0; i < kElemsPerThread; i++) + { + elements2[i] = __shfl_xor_sync(0xffffffff, elements[i], pairOffset); + if (laneId < pairOffset) + { + elements2[i] = -elements2[i]; + } + int dim_idx = laneId * kElemsPerThread + i; + dim_idx = rd_is_pow2 ? ((dim_idx * 2) & rd_mask) : ((dim_idx * 2) % rotaryDim); + int half_dim = dim_idx / 2; + float freq = exp2f(static_cast(half_dim) * neg2_log2base_over_rd); + float theta = pos_id * freq; + __sincosf(theta, &sin_vals[i], &cos_vals[i]); + } + __syncwarp(); + } + + // attention_factor is 1 without YaRN: (x * cos + x2 * sin) * 1 rounds as the generic kernel's product does. + float const attention_factor = 1.0f; + if (rotaryDim == kHeadDim) + { + for (int i = 0; i < kElemsPerThread; i++) + { + elements[i] = (elements[i] * cos_vals[i] + elements2[i] * sin_vals[i]) * attention_factor; + } + } + else + { + for (int i = 0; i < kElemsPerThread; i++) + { + int dim_idx = laneId * kElemsPerThread + i; + if (dim_idx < rotaryDim) + { + elements[i] = (elements[i] * cos_vals[i] + elements2[i] * sin_vals[i]) * attention_factor; + } + } + } + storeFp8ViaBf16(qkvOut, offsetThread, elements); + if constexpr (kWriteCache) + { + // Rows past the live count (padded graph rows) own no slot, nor does a negative (sentinel) slot. + if (!isQ && tokenIdx < cache.numLiveTokens && cache.outCacheLoc[tokenIdx] >= 0) + { + storeFp8ViaBf16(cacheRow(cache, 0, headIdx, tokenIdx), laneId * kElemsPerThread, elements); + } + // The compact query [tokens, q heads, 128] the decode reads, beside the packed row. + if (isQ && cache.qCompact != nullptr) + { + storeFp8ViaBf16(cache.qCompact, + (static_cast(tokenIdx) * numHeadsQ + headIdx) * kHeadDim + laneId * kElemsPerThread, + elements); + } + } +} + +} // namespace + +void launch(void const* qkvIn, void* qkvOut, int numTokens, int numHeadsQ, int numHeadsK, int numHeadsV, int rotaryDim, + float eps, void const* qWeight, void const* kWeight, float base, bool interleave, int const* positionIds, + bool useGemma, KvCacheView const* cache, cudaStream_t stream) +{ + int const totalWarps = numTokens * (numHeadsQ + numHeadsK + numHeadsV); + int const warpsPerBlock = kBlockSize / 32; + dim3 const grid((totalWarps + warpsPerBlock - 1) / warpsPerBlock); + auto const* in = static_cast<__nv_bfloat16 const*>(qkvIn); + auto* out = static_cast<__nv_fp8_e4m3*>(qkvOut); + auto const* qw = static_cast<__nv_bfloat16 const*>(qWeight); + auto const* kw = static_cast<__nv_bfloat16 const*>(kWeight); + KvCacheView const view = cache != nullptr ? *cache : KvCacheView{}; + auto const run = [&](auto kernel) + { + kernel<<>>(in, out, numHeadsQ, numHeadsK, numHeadsV, rotaryDim, eps, qw, kw, base, + positionIds, numTokens, useGemma, view); + }; + if (cache != nullptr) + { + interleave ? run(denseQkNormRopeFp8Kernel) : run(denseQkNormRopeFp8Kernel); + } + else + { + interleave ? run(denseQkNormRopeFp8Kernel) : run(denseQkNormRopeFp8Kernel); + } +} + +} // namespace denseQkNormRopeFp8 +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.h new file mode 100644 index 000000000000..ec94c1dbd550 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.h @@ -0,0 +1,64 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +namespace denseQkNormRopeFp8 +{ + +constexpr int kHeadDim = 128; + +//! A layer's FP8 K/V pool in the HND paged layout [pages, 2 (K, V), kv heads, tokens per block, 128] (element +//! strides), with the step's per-token cache slots (page * tokensPerBlock + within) for its first numLiveTokens rows. +struct KvCacheView +{ + __nv_fp8_e4m3* base = nullptr; + int const* outCacheLoc = nullptr; + long long pageStride = 0; + long long kvStride = 0; + long long headStride = 0; + long long tokenStride = 0; + int tokensPerBlock = 1; + int numLiveTokens = 0; + //! Optional: every row's Q also stored compact, [tokens, q heads, 128]. + __nv_fp8_e4m3* qCompact = nullptr; +}; + +//! The MiniMax-M3 TP2 dense layers' per-head QK RMSNorm + partial RoPE of a packed BF16 [tokens, (q + k + v) * 128] +//! row, written out of place as FP8 E4M3 with V copy-cast. Q and K are rounded to BF16 before E4M3, as the stock path +//! (the BF16 fused_qk_norm_rope output cast to FP8) rounds them. With cache, the K and V rows of the live tokens are +//! also stored into the paged pool at their slots (the bytes of the qkvOut rows). +void launch(void const* qkvIn, void* qkvOut, int numTokens, int numHeadsQ, int numHeadsK, int numHeadsV, int rotaryDim, + float eps, void const* qWeight, void const* kWeight, float base, bool interleave, int const* positionIds, + bool useGemma, KvCacheView const* cache, cudaStream_t stream); + +} // namespace denseQkNormRopeFp8 +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.cu new file mode 100644 index 000000000000..67e054e74712 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.cu @@ -0,0 +1,620 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.h" + +#include +#include +#include + +// One pass of the fused greedy Eagle3 draft loop of MiniMax-M3 (GB300, TP4, <= 8 requests per pass). +// +// One kernel replaces the draft lm_head GEMM, the logits all-gather / float cast / argmax of the in-tree greedy draft +// sampling and the draft loop's per-pass bookkeeping, at the in-tree precision: +// +// * The vocab-sharded BF16 lm_head weight is stored in mma.m16n8k16 A-fragment order (the same values): per 16-row +// tile and k16 step, 512 bytes; lane (g, j) holds 16 bytes = {row g k 2j..2j+1, row g+8 k 2j.., row g k 2j+8.., +// row g+8 k 2j+8..}. +// * Or, by default, in a lossless 12-bit coding of those fragments (75% of the bytes; the target head's coding, see +// targetHead.cu): per 16-row tile and unit of 4 k16 steps a lane reads three 16-byte words, the sign/mantissa bytes +// of steps 0-1, of steps 2-3, then 4 code words (exponent - base, 4 bits; 15 = an exception). The decoded values +// are the checkpoint's BF16 bits; an exception decodes to +0 and the segment's flush adds the exact product x * w +// from the tile's list (products of BF16 values are exact in FP32). +// * A persistent grid (one 16-warp CTA per SM) streams whole tiles over the full K with BF16 MMAs and FP32 +// accumulation; the BF16 head input rows sit in shared memory in B-fragment order, unchanged. +// * Each logit is rounded to BF16 (the in-tree logits are the BF16 GEMM output) and becomes a 64-bit key +// ordered(value) << 32 | ~global_index, with -0 taken as +0: the larger value wins and ties pick the lowest index, as +// torch.max / torch.argmax over the gathered logits do. Keys are max-reduced per row: warp shuffles, then shared +// memory, then atomicMax. +// * The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping +// (Lamport: 0 = empty, a key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the +// slots, and runs the pass's bookkeeping. Call n uses slot half n % 2. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +namespace draftStep +{ +namespace +{ +constexpr int kUnroll = 8; // k16 chunks in flight per warp +constexpr int kChunkBytes = 512; // one tile's k16 step: 32 lanes x 16 bytes +constexpr long long kMaxSpins = 1LL << 31; // cross-rank key poll bound + +// Coded units (4 k16 steps, 3 x 16 bytes per lane) in flight per warp: 3 up to 4 rows, 2 at 8 rows. +template +constexpr int codedUnroll() +{ + return MP >= 8 ? 2 : 3; +} + +__device__ __forceinline__ void mma_bf16( + float (&d)[4], uint32_t a0, uint32_t a1, uint32_t a2, uint32_t a3, uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.bf16.bf16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};\n" + : "+f"(d[0]), "+f"(d[1]), "+f"(d[2]), "+f"(d[3]) + : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(b0), "r"(b1)); +} + +// BF16 pair p (values 2p, 2p + 1) of a lane's 8 coded values: sign/mantissa bytes sm (4 per word), codes word cw. +__device__ __forceinline__ uint32_t coded_decode_pair(uint32_t sm_word, int byte0, uint32_t cw, int p, uint32_t base) +{ + uint32_t const b = __byte_perm(sm_word, 0u, byte0 == 0 ? 0x4140 : 0x4342); // {b0, 0, b1, 0} + uint32_t const q = (cw >> (4 * p)) & 0x000f000fu; + uint32_t const e = q * 128u + base; // exponent field of both halves + // Code 15 (an exception, sign/mantissa byte 0) decodes to +0. + return ((b & 0x007f007fu) | ((b << 8) & 0x80008000u) | e) & __vcmpne2(q, 0x000f000fu); +} + +// One coded unit (4 k16 steps) against the BF16 B fragments: steps alternate between the two accumulators, as the +// k16 steps of the plain BF16 head do. +template +__device__ __forceinline__ void chunk_mma_coded(uint4 const& sm01, uint4 const& sm23, uint4 const& cw, uint32_t base, + uint2 const* xs, int unit, int j, int g, bool b_active, float (&da)[4], float (&db)[4]) +{ + uint32_t const smw[8] = {sm01.x, sm01.y, sm01.z, sm01.w, sm23.x, sm23.y, sm23.z, sm23.w}; + uint32_t const cws[4] = {cw.x, cw.y, cw.z, cw.w}; +#pragma unroll + for (int ci = 0; ci < 4; ++ci) + { + uint2 b = make_uint2(0u, 0u); + if (b_active) + b = xs[((unit * 4 + ci) * 4 + j) * MP + g]; + uint32_t const w0 = coded_decode_pair(smw[2 * ci], 0, cws[ci], 0, base); + uint32_t const w1 = coded_decode_pair(smw[2 * ci], 2, cws[ci], 1, base); + uint32_t const w2 = coded_decode_pair(smw[2 * ci + 1], 0, cws[ci], 2, base); + uint32_t const w3 = coded_decode_pair(smw[2 * ci + 1], 2, cws[ci], 3, base); + if (ci & 1) + mma_bf16(db, w0, w1, w2, w3, b.x, b.y); + else + mma_bf16(da, w0, w1, w2, w3, b.x, b.y); + } +} + +// Exceptions of `tile` with k in [kb, ke): v += x * w (exact) for the owner lanes (row g or g + 8; tokens 2j, +// 2j + 1 < M), x from the staged B fragments. Warp-uniform (shuffles). +template +__device__ __forceinline__ void coded_exceptions( + CodedArgs const& cd, int tile, int kb, int ke, uint2 const* xs, int M, int lane, float (&v)[4]) +{ + int const g = lane >> 2, j = lane & 3; + int const e0 = __ldg(cd.exc_off + tile), e1 = __ldg(cd.exc_off + tile + 1); + for (int b0 = e0; b0 < e1; b0 += 32) + { + uint32_t pos = 0xffffffffu; + float2 wv = make_float2(0.f, 0.f); + if (b0 + lane < e1) + { + pos = __ldg(cd.exc_pos + b0 + lane); + wv = __ldg(cd.exc_w + b0 + lane); + } + int const cnt = min(32, e1 - b0); + for (int q = 0; q < cnt; ++q) + { + uint32_t const pq = __shfl_sync(0xffffffffu, pos, q); + float const w0 = __shfl_sync(0xffffffffu, wv.x, q); + int const r = static_cast(pq >> 16), k = static_cast(pq & 0xffffu); + if (k >= kb && k < ke && (r & 7) == g) + { + int const st = k >> 4, rr = k & 15; + bool const hi = r >= 8; +#pragma unroll + for (int q2 = 0; q2 < 2; ++q2) + { + int const n = 2 * j + q2; + if (n < M) + { + uint2 const bw = xs[(st * 4 + ((rr & 7) >> 1)) * MP + n]; + uint32_t const word = rr >= 8 ? bw.y : bw.x; + float const xv = __uint_as_float((rr & 1) ? (word & 0xffff0000u) : (word << 16)); + float const d = __fmul_rn(xv, w0); + if (hi) + v[2 + q2] += d; + else + v[q2] += d; + } + } + } + __syncwarp(); + } + } +} + +__device__ __forceinline__ uint4 ldg_stream(uint4 const* p) +{ + uint4 r; + asm volatile("ld.global.nc.L1::no_allocate.L2::256B.v4.u32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(r.x), "=r"(r.y), "=r"(r.z), "=r"(r.w) + : "l"(p)); + return r; +} + +// The argmax value of a logit: its BF16 rounding (the in-tree logits' dtype), -0 as +0. +__device__ __forceinline__ float key_value(float v) +{ + float const r = __bfloat162float(__float2bfloat16(v)); + return r == 0.f ? 0.f : r; +} + +__device__ __forceinline__ unsigned long long make_key(float v, uint32_t index) +{ + uint32_t u = __float_as_uint(v); + u = (u & 0x80000000u) ? ~u : (u | 0x80000000u); + return (static_cast(u) << 32) | static_cast(0xFFFFFFFFu - index); +} + +__device__ __forceinline__ unsigned long long shfl_max(unsigned long long k, int o) +{ + unsigned long long const ok = __shfl_xor_sync(0xffffffffu, k, o); + return ok > k ? ok : k; +} + +// One k16 step of a tile: the lane's A fragment w against the activation B fragment of step `chunk` (rows g < MP). +// Even and odd steps accumulate separately (two independent MMA chains); the tile's sum is da + db. +template +__device__ __forceinline__ void chunk_mma( + uint4 const& w, uint2 const* xs, int chunk, int j, int g, bool b_active, float (&da)[4], float (&db)[4]) +{ + uint2 b = make_uint2(0u, 0u); + if (b_active) + b = xs[(chunk * 4 + j) * MP + g]; + if (chunk & 1) + mma_bf16(db, w.x, w.y, w.z, w.w, b.x, b.y); + else + mma_bf16(da, w.x, w.y, w.z, w.w, b.x, b.y); +} + +// Add a warp's segment of a tile into the CTA's accumulator (with a coded head, the segment's exceptions first). +template +__device__ __forceinline__ void flush_tile( + float4* acc, float (&da)[4], float (&db)[4], Params const& p, int tile, int kb, int ke, uint2 const* xs, int lane) +{ + float v[4] = {da[0] + db[0], da[1] + db[1], da[2] + db[2], da[3] + db[3]}; + if constexpr (CODED) + coded_exceptions(p.cd, tile, kb, ke, xs, p.M, lane, v); + float* a = reinterpret_cast(acc); + atomicAdd(a + 0, v[0]); + atomicAdd(a + 1, v[1]); + atomicAdd(a + 2, v[2]); + atomicAdd(a + 3, v[3]); +#pragma unroll + for (int e = 0; e < 4; ++e) + { + da[e] = 0.f; + db[e] = 0.f; + } +} + +// Balanced streaming GEMV over the fragment-ordered weight, persistent grid of 16-warp CTAs. CTA c owns tiles +// [t_begin, t_end); its (tile, chunk) space is split into 16 equal contiguous ranges, one per warp, so every warp +// streams the same number of bytes. A tile straddles at most two warps; their partial sums are added into shared +// memory (two addends: the order does not change the sum). MP = rows padded to 1 / 2 / 4 / 8. A chunk is one k16 +// step of the BF16 head, or one coded unit (4 k16 steps) of the coded head (CODED: no shared-memory prologue). +template +__global__ void __launch_bounds__(kWarps * 32, 1) + draft_step_kernel(Params const p, int kchunks, int pf_chunks, int pf_off) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(16) uint2 xs[]; // [K/16 steps][4 (j)][MP (g)], then the weight prologue + __shared__ float4 acc_s[kMaxTilesPerCta + 1][32]; // + 1: an equal-chunk span can touch one more tile + __shared__ unsigned long long red[kWarps][kMaxTokens]; + __shared__ __align__(8) unsigned long long pf_bar[kWarps]; + __shared__ bool s_last; + __shared__ int s_tok[kMaxTokens]; + + int const warp = threadIdx.x >> 5; + int const lane = threadIdx.x & 31; + int const g = lane >> 2; + int const j = lane & 3; + constexpr int kChunkK = CODED ? 64 : 16; // k per chunk + constexpr int kWords = CODED ? 3 : 1; // 16-byte words per lane and chunk + constexpr int KU = CODED ? codedUnroll() : kUnroll; + int const K = kchunks * kChunkK; + int const M = p.M; + int const r_lo = p.tile_lo; + int const r_n = (p.tile_hi > 0 ? p.tile_hi : p.n_tiles) - r_lo; + bool const eq = p.bnd != nullptr; + // Whole tiles: CTA c owns tiles [t_begin, t_end). Equal chunks: CTA c streams chunks [c0, c1) of the range, which + // start off0 chunks into tile t_begin (its first and last tiles may be shared with the neighbouring CTAs). + long long const r_chunks = static_cast(r_n) * kchunks; + long long const c0 = eq ? (static_cast(blockIdx.x) * r_chunks) / gridDim.x : 0; + long long const c1 = eq ? (static_cast(blockIdx.x + 1) * r_chunks) / gridDim.x : 0; + int const t_begin = r_lo + + (eq ? static_cast(c0 / kchunks) + : static_cast((static_cast(blockIdx.x) * r_n) / gridDim.x)); + int const t_end = r_lo + + (eq ? static_cast((c1 + kchunks - 1) / kchunks) + : static_cast((static_cast(blockIdx.x + 1) * r_n) / gridDim.x)); + int const n_local = t_end - t_begin; + int const off0 = eq ? static_cast(c0 - static_cast(t_begin - r_lo) * kchunks) : 0; + int const span = eq ? static_cast(c1 - c0) : n_local * kchunks; + int const q_begin = off0 + warp * span / kWarps; + int const q_end = off0 + (warp + 1) * span / kWarps; + // Equal chunks: whether the first / last local tile is shared with the previous / next CTA of the launch. + bool const shared_first = eq && off0 != 0; + bool const shared_last = eq && ((c1 % kchunks) != 0); + long long const base = static_cast(t_begin) * kchunks; // first chunk of this CTA + uint4 const* const wf = p.wf; + + // Weights are static: start this warp's stream before waiting on the producer of x. The warp's first pf_chunks + // chunks come in by one bulk copy into shared memory (dynamic offset pf_off), which fills the grid-dependency + // wait and the staging; the register ring starts after them. + int const n_pf = CODED ? 0 : min(pf_chunks, q_end - q_begin); + uint4* const wpf = reinterpret_cast(reinterpret_cast(xs) + pf_off) + warp * pf_chunks * 32; + if (n_pf > 0 && lane == 0) + { + uint32_t const bar = static_cast(__cvta_generic_to_shared(&pf_bar[warp])); + uint32_t const bytes = static_cast(n_pf) * kChunkBytes; + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(bar) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(bar), "r"(bytes) : "memory"); + asm volatile("cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes [%0], [%1], %2, [%3];" ::"r"( + static_cast(__cvta_generic_to_shared(wpf))), + "l"(wf + (base + q_begin) * 32), "r"(bytes), "r"(bar) + : "memory"); + } + __syncwarp(); + int const q_reg = q_begin + n_pf; + uint4 buf[KU][kWords]; +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q_reg + u < q_end) + { +#pragma unroll + for (int l = 0; l < kWords; ++l) + buf[u][l] = ldg_stream(wf + ((base + q_reg + u) * kWords + l) * 32 + lane); + } + } + for (int i = threadIdx.x; i < n_local * 32; i += blockDim.x) + acc_s[i / 32][i % 32] = make_float4(0.f, 0.f, 0.f, 0.f); + + asm volatile("griddepcontrol.wait;" ::: "memory"); + + // Stage the head input rows in B-fragment order: item (k16 step S, row gg) = two 16-byte loads of x, four + // fragment words {x[2jj], x[2jj + 1]}, {x[2jj + 8], x[2jj + 9]} (BF16 pairs, moved unchanged). Rows >= M are 0. + constexpr int kSU = MP >= 4 ? 4 : 2; + int const n_items = (K / 16) * MP; + for (int i0 = threadIdx.x; i0 < n_items; i0 += kSU * blockDim.x) + { + uint4 xv[kSU][2]; +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + int const gg = i % MP; + int const S = i / MP; + xv[u][0] = make_uint4(0u, 0u, 0u, 0u); + xv[u][1] = make_uint4(0u, 0u, 0u, 0u); + if (i < n_items && gg < M) + { + long long const xr = p.x_rows != nullptr ? p.x_rows[gg] : gg; + uint4 const* xp = reinterpret_cast(p.x + xr * K + 16 * S); + xv[u][0] = xp[0]; + xv[u][1] = xp[1]; + } + } +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + if (i >= n_items) + continue; + int const gg = i % MP; + int const S = i / MP; + uint32_t const lo[4] = {xv[u][0].x, xv[u][0].y, xv[u][0].z, xv[u][0].w}; + uint32_t const hi[4] = {xv[u][1].x, xv[u][1].y, xv[u][1].z, xv[u][1].w}; +#pragma unroll + for (int jj = 0; jj < 4; ++jj) + xs[(S * 4 + jj) * MP + gg] = make_uint2(lo[jj], hi[jj]); + } + } + __syncthreads(); + + bool const b_active = g < MP; + float da[4] = {0.f, 0.f, 0.f, 0.f}; + float db[4] = {0.f, 0.f, 0.f, 0.f}; + int tl = q_begin / kchunks; // local tile of the current accumulator + int c = q_begin - tl * kchunks; // chunk within that tile + int cs = c; // first chunk of the warp's segment of that tile + if (n_pf > 0) + { + uint32_t const bar = static_cast(__cvta_generic_to_shared(&pf_bar[warp])); + asm volatile( + "{\n .reg .pred p;\n PF_WAIT%=:\n mbarrier.try_wait.parity.shared::cta.b64 p, [%0], 0;\n" + " @!p bra PF_WAIT%=;\n}\n" ::"r"(bar) + : "memory"); + for (int q = q_begin; q < q_reg; ++q) + { + chunk_mma(wpf[(q - q_begin) * 32 + lane], xs, c, j, g, b_active, da, db); + if (++c == kchunks) + { + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, K, xs, lane); + c = 0; + cs = 0; + ++tl; + } + } + } + for (int q = q_reg; q < q_end; q += KU) + { + uint4 nxt[KU][kWords]; +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q + KU + u < q_end) + { +#pragma unroll + for (int l = 0; l < kWords; ++l) + nxt[u][l] = ldg_stream(wf + ((base + q + KU + u) * kWords + l) * 32 + lane); + } + } +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q + u < q_end) + { + if constexpr (CODED) + chunk_mma_coded(buf[u][0], buf[u][kWords > 1 ? 1 : 0], buf[u][kWords > 2 ? 2 : 0], p.cd.base, + xs, c, j, g, b_active, da, db); + else + chunk_mma(buf[u][0], xs, c, j, g, b_active, da, db); + if (++c == kchunks) + { + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, K, xs, lane); + c = 0; + cs = 0; + ++tl; + } + } + } +#pragma unroll + for (int u = 0; u < KU; ++u) +#pragma unroll + for (int l = 0; l < kWords; ++l) + buf[u][l] = nxt[u][l]; + } + if (c != 0) + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, c * kChunkK, xs, lane); + __syncthreads(); + + // Accumulator lane (g, j) of a tile: rows g / g + 8 (x, y / z, w) of tokens 2j / 2j + 1 (x, z / y, w). + unsigned long long best0 = 0ull, best1 = 0ull; // tokens 2j, 2j + 1 + for (int t = warp; t < n_local; t += kWarps) + { + float4 v = acc_s[t][lane]; + if ((t == 0 && shared_first) || (t == n_local - 1 && shared_last)) + { + // A tile shared with a neighbouring CTA: add this CTA's partial into bnd; the second CTA to arrive reads + // the sum (two addends), keys the tile and restores bnd and its counter to zero for the next call. + int const tile = t_begin + t; + float* const bp = reinterpret_cast(p.bnd + static_cast(tile) * 32 + lane); + atomicAdd(bp + 0, v.x); + atomicAdd(bp + 1, v.y); + atomicAdd(bp + 2, v.z); + atomicAdd(bp + 3, v.w); + __threadfence(); + __syncwarp(); + unsigned int prev = 0; + if (lane == 0) + prev = atomicAdd(p.bnd_cnt + tile, 1u); + prev = __shfl_sync(0xffffffffu, prev, 0); + if (prev == 0) + continue; // the neighbour keys it + __threadfence(); + v = __ldcg(p.bnd + static_cast(tile) * 32 + lane); + p.bnd[static_cast(tile) * 32 + lane] = make_float4(0.f, 0.f, 0.f, 0.f); + if (lane == 0) + p.bnd_cnt[tile] = 0u; + } + int const r0 = (t_begin + t) * 16 + g; + uint32_t const i0 = static_cast(p.vocab_offset + r0); + unsigned long long k0 = make_key(key_value(v.x), i0); + unsigned long long const k0b = make_key(key_value(v.z), i0 + 8); + unsigned long long k1 = make_key(key_value(v.y), i0); + unsigned long long const k1b = make_key(key_value(v.w), i0 + 8); + k0 = k0b > k0 ? k0b : k0; + k1 = k1b > k1 ? k1b : k1; + best0 = k0 > best0 ? k0 : best0; + best1 = k1 > best1 ? k1 : best1; + } +#pragma unroll + for (int o = 4; o < 32; o <<= 1) + { + best0 = shfl_max(best0, o); + best1 = shfl_max(best1, o); + } + asm volatile("griddepcontrol.launch_dependents;"); + if (lane < 4) + { + red[warp][2 * lane] = best0; + red[warp][2 * lane + 1] = best1; + } + __syncthreads(); + if (threadIdx.x < M) + { + unsigned long long b = 0ull; +#pragma unroll + for (int w = 0; w < kWarps; ++w) + b = red[w][threadIdx.x] > b ? red[w][threadIdx.x] : b; + if (b != 0ull) + atomicMax(p.keys + threadIdx.x, b); + } + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + unsigned int const prev = atomicAdd(p.counters, 1u); + s_last = prev == (p.total_ctas != 0u ? p.total_ctas : gridDim.x) - 1u; + } + __syncthreads(); + if (!s_last) + return; + + __threadfence(); + unsigned int const call = *reinterpret_cast(p.counters + 1); + int const half = static_cast(call & 1u); + if (threadIdx.x < M) + { + int const t = threadIdx.x; + unsigned long long const mine = atomicExch(p.keys + t, 0ull); + unsigned long long* slot_mc = p.ws_mc + (half * kMaxTokens + t) * p.world; + unsigned long long* slot_uc = p.ws_uc + (half * kMaxTokens + t) * p.world; + slot_mc[p.rank] = mine; + unsigned long long best = 0ull; + for (int r = 0; r < p.world; ++r) + { + unsigned long long v; + long long spins = 0; + do + { + v = *reinterpret_cast(slot_uc + r); + if (++spins > kMaxSpins) + __trap(); // a missing peer key becomes a CUDA error, never a hang (the mc all-reduce's bound) + } while (v == 0ull); + best = v > best ? v : best; + slot_uc[r] = 0ull; + } + s_tok[t] = static_cast(0xFFFFFFFFu - static_cast(best & 0xFFFFFFFFull)); + if (p.out_keys != nullptr) + p.out_keys[t] = best; + } + __syncthreads(); + if (threadIdx.x == 0) + { + p.counters[0] = 0u; + p.counters[1] = call + 1u; + } + if (threadIdx.x < M) + { + StepArgs const& st = p.st; + int const b = threadIdx.x; + int const tok = s_tok[b]; + st.draft_out[b * st.draft_stride + st.step] = tok; + st.ids_next[b] = tok; + long long const r = p.x_rows != nullptr ? p.x_rows[b] : b; + st.pos_next[b] = st.pos_src[r] + 1; + if (st.step == 0 && st.num_acc != nullptr) + { + st.kv_lens[b] -= st.draft_len - st.num_acc[b]; + st.seq_lens[b] = 1; + } + else if (st.step > 0) + { + st.kv_lens[b] += 1; + } + } +#endif +} + +template +cudaError_t launch(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + auto kern = draft_step_kernel; + size_t const xs_bytes = static_cast(K / 16) * 4 * MP * sizeof(uint2); + // Shared-memory weight prologue: chunks per warp that fit next to the kernel's static + dynamic shared memory. + static int const sStaticSmem = [] + { + cudaFuncAttributes fa{}; + return cudaFuncGetAttributes(&fa, draft_step_kernel) == cudaSuccess + ? static_cast(fa.sharedSizeBytes) + : -1; + }(); + static int const sMaxSmemPerBlock = [] + { + int dev = 0, maxSmem = 0; + cudaGetDevice(&dev); + cudaDeviceGetAttribute(&maxSmem, cudaDevAttrMaxSharedMemoryPerBlockOptin, dev); + return maxSmem; + }(); + int pf = 0; + if (!CODED && sStaticSmem >= 0) + { + long long const room + = static_cast(sMaxSmemPerBlock) - sStaticSmem - static_cast(xs_bytes) - 1024; + long long const want = static_cast(kPrefetchKb) * 1024; + pf = static_cast(std::max(0ll, std::min(room, want)) / (kWarps * kChunkBytes)); + } + size_t const smem = xs_bytes + static_cast(pf) * kWarps * kChunkBytes; + cudaError_t err = cudaFuncSetAttribute(kern, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smem)); + if (err != cudaSuccess) + return err; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(static_cast(grid)); + cfg.blockDim = dim3(kWarps * 32); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, kern, p, K / (CODED ? 64 : 16), pf, static_cast(xs_bytes)); +} + +template +cudaError_t dispatchRows(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + if (p.M <= 1) + return launch<1, CODED>(p, K, grid, pdl, stream); + if (p.M <= 2) + return launch<2, CODED>(p, K, grid, pdl, stream); + if (p.M <= 4) + return launch<4, CODED>(p, K, grid, pdl, stream); + return launch<8, CODED>(p, K, grid, pdl, stream); +} + +} // namespace + +cudaError_t invokeDraftStep(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + bool const coded = p.cd.exc_off != nullptr; + if (p.M < 1 || p.M > kMaxTokens || K % (coded ? 64 : 16) != 0 || grid < 1) + return cudaErrorInvalidValue; + return coded ? dispatchRows(p, K, grid, pdl, stream) : dispatchRows(p, K, grid, pdl, stream); +} + +} // namespace draftStep +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.h new file mode 100644 index 000000000000..eb38b73488a4 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.h @@ -0,0 +1,111 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +//! One pass of the fused greedy Eagle3 draft loop of the MiniMax-M3 NVFP4 TP4 decode path (sm_103a): the BF16 +//! vocab-sharded draft head, the cross-rank argmax and the draft loop's per-pass bookkeeping in one kernel. See +//! draftStep.cu. +namespace draftStep +{ + +constexpr int kMaxTokens = 8; // head rows (requests) of one launch +constexpr int kWarps = 16; // warps per CTA +constexpr int kMaxTilesPerCta = 32; // 16-row weight tiles one CTA can own +//! Shared-memory weight prologue per CTA, in KB (capped by the free shared memory). +constexpr int kPrefetchKb = 112; + +// Draft-pass epilogue (last CTA, one thread per request row b): stores the token as draft token `step`, stages the +// next pass's input id and position, and advances the draft KV lengths the way the in-tree draft loop does between +// passes (pass 0 with num_acc: drop the rejected verify rows and switch to one token per request; later passes: +1). +struct StepArgs +{ + int* draft_out; // [nreq, draft_stride]: column `step` receives the token + int draft_stride; + int step; + int* ids_next; // [nreq] + int const* pos_src; // positions of the rows the head read (indexed like x) + int* pos_next; // [nreq] = pos_src[x_rows[b] or b] + 1 + int* kv_lens; // [nreq] draft KV lengths, updated in place + int const* num_acc; // pass 0: accepted counts (nullptr: no rollback) + int* seq_lens; // pass 0: set to 1 + int draft_len; +}; + +// The 12-bit coded head (see draftStep.cu): the per-tile exception lists and the exponent base. exc_off == nullptr: +// the head is the plain BF16 fragment-ordered copy. +struct CodedArgs +{ + int const* exc_off; // [n_tiles + 1] exception list offsets + uint32_t const* exc_pos; // row in tile << 16 | k, per exception + float2 const* exc_w; // {w, 0} per exception: the exact weight + uint32_t base; // exponent base << 7 in both BF16 halves (base << 7 | base << 23) +}; + +// Everything a draft-step launch takes (see invokeDraftStep). +struct Params +{ + __nv_bfloat16 const* x; // [*, K] head input rows + long long const* x_rows; // optional [M]: head input row of request b (pass 0: the last accepted verify row) + int M; // requests, <= kMaxTokens + uint4 const* wf; // [n_tiles][K / 16][32 lanes] 16-byte BF16 A fragments, or the 12-bit coded stream + // [n_tiles][K / 64][3][32 lanes] when cd.exc_off is set (see draftStep.cu) + int n_tiles; + int vocab_offset; // global id of this rank's first head row + unsigned long long* keys; + unsigned int* counters; + unsigned long long* ws_uc; // this rank's exchange slots [2][kMaxTokens][world] + unsigned long long* ws_mc; // their multicast address + int rank; + int world; + unsigned long long* out_keys; // optional [M]: the winning global key of each row + StepArgs st; + CodedArgs cd; + // A pass split over several launches (one per locality domain, each on that domain's SMs): this launch streams + // tiles [tile_lo, tile_hi) (tile_hi 0 = n_tiles), and the last of total_ctas CTAs over all of the pass's launches + // runs the exchange and the bookkeeping (0 = this launch's grid: one launch per pass). + int tile_lo = 0; + int tile_hi = 0; + unsigned int total_ctas = 0; + // Equal-chunk split (set both, zero-filled [n_tiles][32] float4 / [n_tiles], restored to zero by the kernel): each + // CTA of a launch streams an equal share of its range's (tile, chunk) space instead of whole tiles, so a tile at a + // CTA boundary gets one partial sum from each of its two CTAs, added in bnd (two addends: the order does not change + // the sum) and keyed by the second to arrive (bnd_cnt). nullptr: whole tiles per CTA. + float4* bnd = nullptr; + unsigned int* bnd_cnt = nullptr; +}; + +//! Launch the draft step over `grid` 512-thread CTAs (K = the hidden size, a multiple of 16; of 64 for a coded head). +cudaError_t invokeDraftStep(Params const& p, int K, int grid, bool pdl, cudaStream_t stream); + +} // namespace draftStep +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.cu new file mode 100644 index 000000000000..179770d14a0e --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.cu @@ -0,0 +1,189 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// The TP2 drafter layer's generation preprocess in one launch. The rotation is applyBiasRopeUpdateKVCacheV2's GPT-NeoX +// path at BF16 (apply_rotary_embedding_gptneox: cos * x + sin * pair in fp32, the first rotary half's sin negated, +// dims past the rotary width with (cos, sin) = (1, 0), rounded to BF16); the K / V store is its unit-scale E4M3 +// store (__nv_fp8x2_e4m3 from BF16 pairs), which is also the query's E4M3: torch's CUDA cast of the BF16 query +// saturates the same way (448, not NaN). + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +namespace drafterRopeFp8Kv +{ +namespace +{ + +constexpr int kElemsPerThread = kHeadDim / 32; +constexpr int kBlockSize = 256; + +template +__device__ __forceinline__ float rotate(float c, float x, float s, float pair) +{ + if constexpr (kRopeMode == 0) + { + return __fadd_rn(__fmul_rn(c, x), __fmul_rn(s, pair)); + } + else if constexpr (kRopeMode == 1) + { + return __fmaf_rn(c, x, __fmul_rn(s, pair)); + } + else + { + return __fmaf_rn(s, pair, __fmul_rn(c, x)); + } +} + +__device__ __forceinline__ void storeE4m3Pairs(__nv_fp8_e4m3* dst, __nv_bfloat16 const (&v)[kElemsPerThread]) +{ +#pragma unroll + for (int i = 0; i < kElemsPerThread; i += 2) + { + __nv_bfloat162 pair; + pair.x = v[i]; + pair.y = v[i + 1]; + __nv_fp8x2_e4m3 const packed(pair); + reinterpret_cast<__nv_fp8x2_storage_t*>(dst)[i / 2] = packed.__x; + } +} + +template +__global__ void __launch_bounds__(kBlockSize) drafterRopeFp8KvKernel(Params const p) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); +#endif + int const lane = threadIdx.x % 32; + int const warp = (blockIdx.x * blockDim.x + threadIdx.x) / 32; + int const slots = p.numHeadsQ + p.numHeadsKv; + int const token = warp / slots; + if (token >= p.numTokens) + { + return; + } + int const slot = warp % slots; + bool const isQ = slot < p.numHeadsQ; + int const head = isQ ? slot : slot - p.numHeadsQ; + int const request = token / p.qLen; + int const row = token % p.qLen; + int const cacheLen = p.seqLens[request]; + int const kvPos = cacheLen - p.qLen + row; + int const ropePos + = p.specPositionOffsets != nullptr ? p.specPositionOffsets[request * p.qLen + row] + cacheLen - p.qLen : kvPos; + int const rowWidth = (p.numHeadsQ + 2 * p.numHeadsKv) * kHeadDim; + __nv_bfloat16 const* src + = p.qkv + static_cast(token) * rowWidth + (isQ ? head * kHeadDim : (p.numHeadsQ + head) * kHeadDim); + int const half = p.rotaryDim / 2; + float2 const* coef = p.cosSin + static_cast(ropePos) * half; + + __nv_bfloat16 rotated[kElemsPerThread]; +#pragma unroll + for (int i = 0; i < kElemsPerThread; ++i) + { + int const d = lane * kElemsPerThread + i; + bool const firstHalf = d < half; + float const x = __bfloat162float(src[d]); + float const pair = __bfloat162float(src[firstHalf ? d + half : d - half]); + float2 const cs = d < p.rotaryDim ? coef[firstHalf ? d : d - half] : make_float2(1.0f, 0.0f); + float const s = firstHalf ? -cs.y : cs.y; + rotated[i] = __float2bfloat16(rotate(cs.x, x, s, pair)); + } + + if (isQ) + { + storeE4m3Pairs( + p.qOut + (static_cast(token) * p.numHeadsQ + head) * kHeadDim + lane * kElemsPerThread, rotated); + return; + } + + int const block = kvPos / p.tokensPerBlock; + long long const inPage = (static_cast(head) * p.tokensPerBlock + kvPos % p.tokensPerBlock) * kHeadDim + + lane * kElemsPerThread; + long long const pageElems = static_cast(p.numHeadsKv) * p.tokensPerBlock * kHeadDim; + int const* table = p.blockTable + request * p.blockTableRequestStride; + long long const kPage = table[block]; + long long const vPage = table[p.blockTableKvStride + block]; + storeE4m3Pairs(p.pool + kPage * pageElems + inPage, rotated); + + __nv_bfloat16 const* vSrc = p.qkv + static_cast(token) * rowWidth + + (p.numHeadsQ + p.numHeadsKv + head) * kHeadDim + lane * kElemsPerThread; + __nv_bfloat16 v[kElemsPerThread]; +#pragma unroll + for (int i = 0; i < kElemsPerThread; ++i) + { + v[i] = vSrc[i]; + } + storeE4m3Pairs(p.pool + vPage * pageElems + inPage, v); +} + +//! With PDL and maxSharedCarveout, the maximum shared-memory carveout lets the full-context decode's large-smem CTAs +//! (the dependent this kernel releases right after its wait) co-reside on the SMs this small kernel occupies. Set once +//! per instantiation. +template +cudaError_t launchWithMaxSharedCarveout( + cudaLaunchConfig_t const& config, Params const& params, bool pdl, bool maxSharedCarveout) +{ + auto const kernel = drafterRopeFp8KvKernel; + if (pdl && maxSharedCarveout) + { + static cudaError_t const carveout = cudaFuncSetAttribute( + kernel, cudaFuncAttributePreferredSharedMemoryCarveout, cudaSharedmemCarveoutMaxShared); + if (carveout != cudaSuccess) + { + return carveout; + } + } + return cudaLaunchKernelEx(&config, kernel, params); +} + +} // namespace + +cudaError_t launch(Params const& params, bool pdl, bool maxSharedCarveout, cudaStream_t stream) +{ + int const warps = params.numTokens * (params.numHeadsQ + params.numHeadsKv); + int const warpsPerBlock = kBlockSize / 32; + cudaLaunchConfig_t config = {}; + config.gridDim = dim3((warps + warpsPerBlock - 1) / warpsPerBlock); + config.blockDim = dim3(kBlockSize); + config.dynamicSmemBytes = 0; + config.stream = stream; + cudaLaunchAttribute attribute[1]; + attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attribute[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + config.attrs = attribute; + config.numAttrs = 1; + switch (params.ropeMode) + { + case 0: return launchWithMaxSharedCarveout<0>(config, params, pdl, maxSharedCarveout); + case 1: return launchWithMaxSharedCarveout<1>(config, params, pdl, maxSharedCarveout); + case 2: return launchWithMaxSharedCarveout<2>(config, params, pdl, maxSharedCarveout); + default: return cudaErrorInvalidValue; + } +} + +} // namespace drafterRopeFp8Kv +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.h new file mode 100644 index 000000000000..3302c94390f3 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.h @@ -0,0 +1,79 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +namespace drafterRopeFp8Kv +{ + +constexpr int kHeadDim = 128; + +struct Params +{ + //! [tokens, (numHeadsQ + 2 * numHeadsKv) * 128] BF16: the drafter layer's packed Q, K, V rows. + __nv_bfloat16 const* qkv; + //! [positions, rotaryDim / 2] (cos, sin) pairs: the layer's rotary_cos_sin table. + float2 const* cosSin; + //! [requests] cache lengths including this step's rows. + int const* seqLens; + //! [requests, qLen] spec-decoding position offsets of a multi-token pass, or null. + int const* specPositionOffsets; + //! [tokens, numHeadsQ, 128] E4M3 query out. + __nv_fp8_e4m3* qOut; + //! Base of the layer's K / V sub-pages, [*, numHeadsKv, tokensPerBlock, 128] E4M3. + __nv_fp8_e4m3* pool; + //! [requests, 2, maxBlocks] sub-page indices: K in row 0, V in row 1. + int const* blockTable; + long long blockTableRequestStride; + long long blockTableKvStride; + int numTokens; + int qLen; + int numHeadsQ; + int numHeadsKv; + int rotaryDim; + int tokensPerBlock; + //! How the rotation's two products are summed: 0 two products then the add, 1 fma(cos, x, sin * pair), + //! 2 fma(sin, pair, cos * x). + int ropeMode; +}; + +//! The MiniMax-M3 TP2 Eagle3 drafter layer's generation preprocess: GPT-NeoX RoPE from the cos/sin table on Q and K, +//! rounded to BF16 (the in-tree preprocess's q output and cache input), then Q, K and V to saturating E4M3: K / V +//! as the in-tree preprocess stores them at unit KV scale, Q as torch's CUDA cast of the BF16 query. Rows map to +//! (request token / qLen, row token % qLen) at cache position seqLen - qLen + row. +//! +//! PDL (pdl): the launch is programmatic; griddepcontrol.wait is the kernel's first instruction and the kernel +//! releases its dependent right after it. The dependent may read only step-invariant data before its own wait. +cudaError_t launch(Params const& params, bool pdl, bool maxSharedCarveout, cudaStream_t stream); + +} // namespace drafterRopeFp8Kv +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.cu new file mode 100644 index 000000000000..dd0ad05437a6 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.cu @@ -0,0 +1,86 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +namespace +{ + +// Row i of the step (lane c = column), as the Triton fold tail's program i: the store's padded rows, then the next +// step's inputs read from them (zero past each source width). +__global__ void __launch_bounds__(32) foldTailKernel(FoldTailParams const p) +{ + int const i = blockIdx.x; + int const c = threadIdx.x; + if (i >= *p.ctrl) + { + return; + } + int64_t const s = p.slots[i]; + int32_t const nw = c < p.snW ? p.srcNew[i * p.sn0 + c * p.sn1] : 0; + if (c < p.dnW) + { + p.dstNew[s * p.dn0 + c * p.dn1] = nw; + } + int32_t const nxt = c < p.sxW ? p.srcNext[i * p.sx0 + c * p.sx1] : 0; + if (c < p.dxW) + { + p.dstNext[s * p.dx0 + c * p.dx1] = nxt; + } + int32_t const drf = c < p.sdW ? p.srcDraft[i * p.sd0 + c * p.sd1] : 0; + if (c < p.ddW) + { + p.dstDraft[s * p.dd0 + c * p.dd1] = drf; + } + int32_t const ln = p.srcLens[i]; + int32_t const t = p.tokensPerStep; + int32_t const d = t - 1; + if (c == 0) + { + p.dstLens[s] = ln; + p.kvOff[i] = ln - t; + } + if (c < t) + { + p.ids[i * t + c] = c < p.dxW ? nxt : 0; + p.posOff[i * t + c] = ln; + } + if (c < d) + { + p.draftIds[i * d + c] = c < p.ddW ? drf : 0; + } +} + +} // namespace + +cudaError_t invokeFoldTail(FoldTailParams const& p, int32_t rows, cudaStream_t stream) +{ + foldTailKernel<<>>(p); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.h new file mode 100644 index 000000000000..929ac2fed9ed --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.h @@ -0,0 +1,72 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +//! Operands of the speculative step's forward-tail store and next-step staging (kernels/m3_fold_tail.py). Every tensor +//! is int32 except `slots` (int64); strides are in elements. The store views are [slots, width], the sources +//! [rows, width]. +struct FoldTailParams +{ + int32_t const* ctrl; // ctrl[0]: the rows the step armed (0: write nothing) + int64_t const* slots; + int32_t const* srcNew; + int64_t sn0, sn1; + int32_t snW; + int32_t* dstNew; + int64_t dn0, dn1; + int32_t dnW; + int32_t const* srcNext; + int64_t sx0, sx1; + int32_t sxW; + int32_t* dstNext; + int64_t dx0, dx1; + int32_t dxW; + int32_t const* srcDraft; + int64_t sd0, sd1; + int32_t sdW; + int32_t* dstDraft; + int64_t dd0, dd1; + int32_t ddW; + int32_t const* srcLens; + int32_t* dstLens; + int32_t* ids; // [rows * T] + int32_t* draftIds; // [rows * (T - 1)] + int32_t* posOff; // [rows * T] + int32_t* kvOff; // [rows] + int32_t tokensPerStep; +}; + +//! One CTA of 32 lanes per row, `rows` CTAs; the first ctrl[0] rows write. Widths and tokensPerStep must be <= 32. +cudaError_t invokeFoldTail(FoldTailParams const& p, int32_t rows, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.cu new file mode 100644 index 000000000000..236a13e66d1f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.cu @@ -0,0 +1,223 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// The split-KV merge for the MiniMax-M3 NVFP4 TP2 (Rubin) full-context decode. The producer is the PrimTS split-KV +// decode with its separate-reduction scratch: per (request, KV head, split, row) one normalized 16-bit partial O and +// one fp32 log2-LSE. Active splits per row follow the producer's split ranges (the causal union of the row's grouped +// Q CTA, instruction-group aligned): splits past them hold nothing and are not read. +// +// One CTA per (request, KV head, kRowsPerCta rows); per row kGroups split groups x kThreadsPerRow threads x 8 dims. +// A thread issues the loads of every split it owns (strided over the groups) before using any, folds them under its +// group's max, and the groups merge through SMEM with one max rescale; the row is normalized once and rounded to +// BF16 once. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +namespace fullCtxReduce +{ +namespace +{ + +constexpr int kGroups = 8; +constexpr int kThreadsPerRow = kHeadDim / 8; +constexpr int kRowsPerCta = 2; +constexpr int kSplitsPerGroup = kMaxSplits / kGroups; +constexpr int kThreads = kGroups * kThreadsPerRow * kRowsPerCta; + +__device__ int activeSplits(Params const& p, int b, int row) +{ + int const seqKv = p.seqLensKv[b]; + int visible = seqKv; + if (p.causal) + { + int const token = row / p.headsPerKv; + int const base = (token / p.qTokensPerCta) * p.qTokensPerCta; + int const end = min(base + p.qTokensPerCta, p.seqLenQ); + visible = min(max(seqKv - p.seqLenQ + end, 0), seqKv); + } + int const tiles = (visible + p.tileKv - 1) / p.tileKv; + int const tilesPerGroup = p.splits * p.instsKv; + int const groups = (tiles + tilesPerGroup - 1) / tilesPerGroup; + int const local = max(groups * p.instsKv, p.instsKv); + return (tiles + local - 1) / local; +} + +template +__device__ float2 unpack(uint32_t word) +{ + if constexpr (kBf16Partial) + { + return __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&word)); + } + else + { + return __half22float2(*reinterpret_cast<__half2 const*>(&word)); + } +} + +template +__global__ void __launch_bounds__(kThreads) fullCtxReduceKernel(Params const p) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + int const b = blockIdx.z; + int const h = blockIdx.y; + int const localRow = threadIdx.x / (kGroups * kThreadsPerRow); + int const row = blockIdx.x * kRowsPerCta + localRow; + int const g = (threadIdx.x / kThreadsPerRow) % kGroups; + int const t = threadIdx.x % kThreadsPerRow; + bool const validRow = row < p.rows; + int const active = validRow ? activeSplits(p, b, row) : 0; + + __shared__ float sMax[kRowsPerCta][kGroups]; + __shared__ float sSum[kRowsPerCta][kGroups]; + __shared__ float sAcc[kRowsPerCta][kGroups][kHeadDim]; + + uint4 const* partialO = static_cast(p.partialO); + size_t const kvIdx = static_cast(b) * p.numKvHeads + h; + float lse[kSplitsPerGroup]; + uint4 vec[kSplitsPerGroup]; +#pragma unroll + for (int i = 0; i < kSplitsPerGroup; ++i) + { + int const s = g + i * kGroups; + if (s < active) + { + size_t const r = (kvIdx * p.splits + s) * p.rows + row; + lse[i] = __ldcg(p.partialStats + r); + vec[i] = __ldcg(partialO + r * kThreadsPerRow + t); + } + } + float groupMax = -CUDART_INF_F; +#pragma unroll + for (int i = 0; i < kSplitsPerGroup; ++i) + { + if (g + i * kGroups < active) + { + groupMax = fmaxf(groupMax, lse[i]); + } + } + // A split of the CTA's causal union can hold no key of this row (log2-LSE -inf). Such a split, and a group of + // only such splits, is skipped rather than weighted 0: its partial O carries no defined value. + float acc[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; + float sum = 0.f; +#pragma unroll + for (int i = 0; i < kSplitsPerGroup; ++i) + { + if (g + i * kGroups < active && lse[i] != -CUDART_INF_F) + { + float const w = exp2f(lse[i] - groupMax); + sum += w; + uint32_t const* words = reinterpret_cast(&vec[i]); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float2 const f = unpack(words[j]); + acc[2 * j] += w * f.x; + acc[2 * j + 1] += w * f.y; + } + } + } + if (t == 0) + { + sMax[localRow][g] = groupMax; + sSum[localRow][g] = sum; + } +#pragma unroll + for (int j = 0; j < 8; ++j) + { + sAcc[localRow][g][t * 8 + j] = acc[j]; + } + __syncthreads(); + if (g != 0 || !validRow) + { + return; + } + float rowMax = -CUDART_INF_F; +#pragma unroll + for (int k = 0; k < kGroups; ++k) + { + rowMax = fmaxf(rowMax, sMax[localRow][k]); + } + float total = 0.f; + float o[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; +#pragma unroll + for (int k = 0; k < kGroups; ++k) + { + if (sMax[localRow][k] == -CUDART_INF_F) + { + continue; + } + float const scale = exp2f(sMax[localRow][k] - rowMax); + total += scale * sSum[localRow][k]; +#pragma unroll + for (int j = 0; j < 8; ++j) + { + o[j] += scale * sAcc[localRow][k][t * 8 + j]; + } + } + float const inv = total > 0.f ? 1.f / total : 0.f; + int const token = row / p.headsPerKv; + int const head = row % p.headsPerKv; + __nv_bfloat162 packed[4]; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + packed[j] = __floats2bfloat162_rn(o[2 * j] * inv, o[2 * j + 1] * inv); + } + size_t const outRow + = (static_cast(b) * p.seqLenQ + token) * (p.numKvHeads * p.headsPerKv) + h * p.headsPerKv + head; + reinterpret_cast(p.out + outRow * kHeadDim)[t] = *reinterpret_cast(packed); +} + +} // namespace + +cudaError_t invokeFullCtxReduce(Params const& params, bool pdl, cudaStream_t stream) +{ + if (params.splits < 1 || params.splits > kMaxSplits || params.rows < 1) + { + return cudaErrorInvalidValue; + } + cudaLaunchConfig_t config = {}; + config.gridDim = dim3((params.rows + kRowsPerCta - 1) / kRowsPerCta, params.numKvHeads, params.batch); + config.blockDim = dim3(kThreads); + config.stream = stream; + cudaLaunchAttribute attribute[1]; + attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attribute[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + config.attrs = attribute; + config.numAttrs = 1; + return params.bf16Partial ? cudaLaunchKernelEx(&config, fullCtxReduceKernel, params) + : cudaLaunchKernelEx(&config, fullCtxReduceKernel, params); +} + +} // namespace fullCtxReduce +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.h new file mode 100644 index 000000000000..eca13e5d7dab --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.h @@ -0,0 +1,71 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +//! The split-KV merge of the MiniMax-M3 TP2 full-context decode (PrimTS split-KV decode, separate-reduction scratch) +//! straight to BF16. See fullCtxReduce.cu. +namespace fullCtxReduce +{ + +constexpr int kHeadDim = 128; +constexpr int kMaxSplits = 128; + +struct Params +{ + void const* partialO; // [B, Hkv, S, R, 128] normalized 16-bit partial O per split + float const* partialStats; // [B, Hkv, S, R] log2-LSE per split + int const* seqLensKv; // [B] + __nv_bfloat16* out; // [B, SQ, Hkv * hR, 128] + int batch; + int numKvHeads; + int splits; // S, the planned split capacity (<= kMaxSplits) + int rows; // R = hR * SQ, rows [token, local head] + int seqLenQ; // SQ + int headsPerKv; // hR + int qTokensPerCta; // query tokens one producer CTA owns (grouped Q) + int tileKv; // producer K/V tile + int instsKv; // producer K/V instructions per loop step + bool causal; + bool bf16Partial; +}; + +//! Merges the active splits of every row in fp32 with one max rescale and writes BF16. With pdl the launch is +//! programmatic: the grid waits on the producer (griddepcontrol.wait) before reading its scratch. +//! +//! PDL contract: griddepcontrol.wait is the kernel's first instruction, so nothing (scratch, lengths, out) is read or +//! written before every prerequisite grid has completed and flushed; a producer's early trigger +//! (griddepcontrol.launch_dependents) only lets this grid launch and wait sooner. The kernel never triggers its own +//! dependents early: the next kernel's post-wait code sees all of out, and its pre-wait prologue must not read out. +cudaError_t invokeFullCtxReduce(Params const& params, bool pdl, cudaStream_t stream); + +} // namespace fullCtxReduce +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.cu new file mode 100644 index 000000000000..cad2b93558ab --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.cu @@ -0,0 +1,1092 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/tllmException.h" + +#include +#include +#include +#include + +// SM107 (Rubin) copy of minimaxM3/mcAllreduceNorm.cu for the sm_107 TP2 target. Two changes in the arithmetic: the +// rank sum rounds to BF16 before the residual add, the rounding points of the stock all-reduce + fused residual +// RMSNorm on this target (bf16 all-reduce output, then bf16(residual + sum)); and every call but the MoE finalize runs +// the row kernel below, whose RMS statistic is the stock fused norm's reduction tree (bitwise the stock outputs). +// +// The kernels use sm_100-family features (thread-block clusters, cp.async.bulk, st.async, mbarrier transaction +// counts, block-scaled conversions): they are compiled for every sm_10x target of the build, the sm_100f family target +// included (which runs on SM107). A device pass for any other architecture compiles them to a trap, never to a silent +// empty body, and the host entry point checks that the kernel binary it launches holds sm_10x code (see +// checkBinary). +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000 +#define MINIMAX_M3_KERNELS_ENABLED 1 +#else +#define MINIMAX_M3_KERNELS_ENABLED 0 +#endif + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace cg = cooperative_groups; + +namespace +{ + +constexpr int kEmpty = static_cast(0x80000000u); +constexpr int kFinMaxTopk = 8; // MoE finalize: top-k rows loaded together up to this k + +// The Eagle3 capture written by a capture layer's boundary (cap_out, optional): rmsnorm(updated residual) * fc_norm[i], +// bit for bit the m3_spec_tail capture_norm kernel (kernels/minimaxM3/specTail.cu at H 6144: 256 threads, thread c +// sums the squares of the 8-column chunks c, c + 256 and c + 512 in that order, the warp butterfly 16..1, the 8 warp +// sums added in warp order, rsqrtf(sum / H + eps), bf16(v * rs * w)). Only its layout is reproduced here; its +// arithmetic is the same source. +constexpr int kCapThreads = 256; +constexpr int kCapIters = 3; +constexpr int kCapWarps = kCapThreads / 32; +constexpr int kCapHidden = kCapThreads * kCapIters * 8; + +// capture_norm's per-thread chain over one 8-column chunk (continues `ss`). +__device__ __forceinline__ float cap_chunk_sq(uint4 const raw, float ss) +{ + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&raw); +#pragma unroll + for (int q = 0; q < 8; ++q) + { + float const v = __bfloat162float(e[q]); + ss += v * v; + } + return ss; +} + +__device__ __forceinline__ float cap_warp_sum(float v) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + return v; +} + +// bf16(v * rs * w) for one 8-column chunk of the updated residual (v as floats) into cap. +__device__ __forceinline__ void cap_store( + float const (&v)[8], float rs, __nv_bfloat16 const* __restrict__ w, __nv_bfloat16* __restrict__ cap) +{ + uint4 const wr = *reinterpret_cast(w); + __nv_bfloat16 const* we = reinterpret_cast<__nv_bfloat16 const*>(&wr); + uint4 o; + __nv_bfloat16* oe = reinterpret_cast<__nv_bfloat16*>(&o); +#pragma unroll + for (int q = 0; q < 8; ++q) + oe[q] = __float2bfloat16(v[q] * rs * __bfloat162float(we[q])); + *reinterpret_cast(cap) = o; +} + +__device__ __forceinline__ uint32_t canon_bf16x2(uint32_t w) +{ + if ((w & 0xFFFFu) == 0x8000u) + w &= 0xFFFF0000u; + if ((w >> 16) == 0x8000u) + w &= 0x0000FFFFu; + return w; +} + +__device__ __forceinline__ float bf16_lo(uint32_t w) +{ + return __uint_as_float(w << 16); +} + +// The MXFP8 publish quantizes exactly as FlashInfer's CuTe DSL mxfp8 quantize (MXFP8QuantizeRunner.CUTE_DSL, MX-TAC's +// pinned decode quantize; quantization_cute_dsl_utils.py), the three helpers below being its PTX verbatim. It equals +// trtllm::mxfp8_quantize bit for bit except on blocks with 0 < amax < 448 * 2^-127 (UE8M0 byte 0), where the +// TRT-LLM kernel's rcp.approx.ftz of the denormal scale gives inf (NaN / garbage E4M3) and this flushes the block to 0. +__device__ __forceinline__ uint32_t mxfp8_ue8m0(float amax) +{ + float value; + asm("mul.rn.f32 %0, %1, %2;" : "=f"(value) : "f"(amax), "f"(1.0f / 448.0f)); + uint32_t result; + asm("{\n" + " .reg .pred p_zero, p_has_mant, p_exp_zero, p_tiny_sub, p_ovf;\n" + " .reg .u32 bits, exp_biased, mantissa, bump, r;\n" + " setp.le.f32 p_zero, %1, 0f00000000;\n" + " mov.b32 bits, %1;\n" + " shr.b32 exp_biased, bits, 23;\n" + " and.b32 exp_biased, exp_biased, 255;\n" + " and.b32 mantissa, bits, 0x7FFFFF;\n" + " setp.ne.u32 p_has_mant, mantissa, 0;\n" + " selp.u32 bump, 1, 0, p_has_mant;\n" + " setp.eq.u32 p_exp_zero, exp_biased, 0;\n" + " setp.le.u32 p_tiny_sub, mantissa, 0x400000;\n" + " and.pred p_tiny_sub, p_exp_zero, p_tiny_sub;\n" + " @p_tiny_sub mov.u32 bump, 0;\n" + " add.u32 r, exp_biased, bump;\n" + " setp.gt.u32 p_ovf, r, 254;\n" + " selp.u32 r, 254, r, p_ovf;\n" + " selp.u32 %0, 0, r, p_zero;\n" + "}\n" + : "=r"(result) + : "f"(value)); + return result; +} + +__device__ __forceinline__ float mxfp8_inv_scale(uint32_t ue8m0) +{ + float inv; + asm("{\n" + " .reg .s32 new_exp;\n" + " .reg .b32 float_bits;\n" + " .reg .pred p_zero;\n" + " setp.eq.u32 p_zero, %1, 0;\n" + " sub.s32 new_exp, 254, %1;\n" + " max.s32 new_exp, new_exp, 0;\n" + " shl.b32 float_bits, new_exp, 23;\n" + " mov.b32 %0, float_bits;\n" + " @p_zero mov.b32 %0, 0;\n" + "}\n" + : "=f"(inv) + : "r"(ue8m0)); + return inv; +} + +// Two BF16-valued floats (lo, hi) times inv_scale -> two E4M3 bytes (lo in the low byte), saturating to +/-448. +__device__ __forceinline__ uint32_t mxfp8_e4m3x2(float lo, float hi, float inv) +{ + uint16_t pair; + asm("{\n" + " .reg .f32 f0, f1;\n" + " mul.f32 f0, %1, %3;\n" + " mul.f32 f1, %2, %3;\n" + " min.f32 f0, f0, 0f43E00000;\n" + " max.f32 f0, f0, 0fC3E00000;\n" + " min.f32 f1, f1, 0f43E00000;\n" + " max.f32 f1, f1, 0fC3E00000;\n" + " cvt.rn.satfinite.e4m3x2.f32 %0, f1, f0;\n" + "}\n" + : "=h"(pair) + : "f"(lo), "f"(hi), "f"(inv)); + return pair; +} + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ float bf16_hi(uint32_t w) +{ + return __uint_as_float(w & 0xFFFF0000u); +} + +__device__ __forceinline__ int4 ld_volatile_v4(int const* p) +{ + int4 v; + asm volatile("ld.volatile.global.v4.s32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(p) + : "memory"); + return v; +} + +struct FinalizeArgs +{ + __nv_bfloat16 const* fc2; // [permuted rows, H] routed-expert FC2 output (null: plain input x) + int const* perm; // [M, topk] expanded index -> permuted row, -1 = dropped + void const* scales; // [M, topk] routing weights (fp32 or bf16) + int scale_fp32; + int topk; +}; + +// grid = min(M, MAXT) * CLUSTER CTAs, cluster (CLUSTER, 1, 1), THREADS threads. +// OPTS (each bit-identical): 1 residual + norm weight loaded at entry (before the poll); 2 cluster RMS exchange by +// st.async + mbarrier complete_tx instead of cluster.sync; 4 call parity rotated right after the poll; 8 max-smem +// carveout so the next kernel's CTAs co-reside early. +// OPTS 16 (NOT bit-identical with the others; a different boundary): the Eagle3 drafter's boundary, whose stock is a +// plain BF16 all-reduce, then FlashInfer fused_add_rmsnorm: the RMS statistic and the normalization take the +// unrounded FP32 sum bf16(rank sum) + residual (only the stored residual is rounded to BF16). +// CAP: the Eagle3 capture epilogue (cap_out) is compiled in only where a call passes cap_out, so every other call +// runs the kernel without the capture branches, shared arrays or parameters in its body. +template +__global__ void __launch_bounds__(THREADS) mc_allreduce_norm_kernel(__nv_bfloat16 const* __restrict__ x, + __nv_bfloat16 const* __restrict__ x2, FinalizeArgs fin, int M, int H, __nv_bfloat16 const* __restrict__ residual, + __nv_bfloat16 const* __restrict__ norm_w, float eps, int* ws_uc, int* ws_mc, int* __restrict__ flags, int rank, + int world, __nv_bfloat16* __restrict__ out_normed, __nv_bfloat16* __restrict__ out_residual, + uint8_t* __restrict__ out_q, uint8_t* __restrict__ out_sf, int sf_k_tiles, int prepushed, + __nv_bfloat16* __restrict__ cap_out, int64_t cap_stride, __nv_bfloat16 const* __restrict__ cap_w, float cap_eps) +{ +#if MINIMAX_M3_KERNELS_ENABLED + __shared__ float warp_sq[THREADS / 32]; + __shared__ alignas(16) float cluster_sq[CLUSTER]; + extern __shared__ __align__(16) uint4 cap_slice[]; // [cols / 8] this CTA's updated chunks (cap_out only) + __shared__ alignas(8) unsigned long long xbar; + cg::cluster_group cluster = cg::this_cluster(); + if constexpr ((OPTS & 2) != 0) + { + if (threadIdx.x == 0) + { + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(b) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + } + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + } + int const crank = static_cast(cluster.block_rank()); + int const t = blockIdx.x / CLUSTER; + int const cols = H / CLUSTER; + int const c0 = crank * cols + threadIdx.x * 8; + bool const active = t < M && threadIdx.x * 8 < cols; + int64_t const half_words = static_cast(MAXT) * world * (H / 2); + + // prepushed 1: the producer (m3_mx_gemv) already pushed this rank's rows into the exchange slots; 2: and it + // launched this grid only after its pushes (its grid wait passed), so the slots are polled without a grid wait; + // 3: as 2, with x in set A and x2 in set B. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (prepushed < 2) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + int const parity = *reinterpret_cast(flags + blockIdx.x) & 1; + uint4 rraw = make_uint4(0u, 0u, 0u, 0u); + uint4 wraw = make_uint4(0u, 0u, 0u, 0u); + if constexpr ((OPTS & 1) != 0) + { + // the residual row was written grids earlier (prepushed >= 2: the producer launched this grid only after its + // own grid wait; otherwise the grid wait above) + if (active) + { + rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + wraw = *reinterpret_cast(norm_w + c0); + } + } + // Push this rank's slice into every rank's buffer. + if (active && !prepushed) + { + uint4 raw; + if (fin.fc2 != nullptr) + { + // MoE finalize: fp32 sum of the routing-weighted top-k expert rows plus the shared expert, rounded once. + float f[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; + if (fin.topk <= kFinMaxTopk) + { + // All top-k indices and scales, then every expert-row load in flight at once, then the same in-order + // FP32 accumulation as the loop below (one memory round trip instead of topk; bit-identical). + int prow[kFinMaxTopk]; + float sc[kFinMaxTopk]; +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + prow[k] = -1; + sc[k] = 0.f; + if (k < fin.topk) + { + int const e = t * fin.topk + k; + prow[k] = fin.perm[e]; + sc[k] = fin.scale_fp32 ? static_cast(fin.scales)[e] + : __bfloat162float(static_cast<__nv_bfloat16 const*>(fin.scales)[e]); + } + } + uint4 rv[kFinMaxTopk]; +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + if (prow[k] >= 0) + rv[k] = *reinterpret_cast(fin.fc2 + static_cast(prow[k]) * H + c0); + } +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + if (prow[k] < 0) + continue; + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv[k]); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] += v.x * sc[k]; + f[2 * q + 1] += v.y * sc[k]; + } + } + } + else + for (int k = 0; k < fin.topk; ++k) + { + int const e = t * fin.topk + k; + int const prow = fin.perm[e]; + if (prow < 0) + continue; + float const sc = fin.scale_fp32 + ? static_cast(fin.scales)[e] + : __bfloat162float(static_cast<__nv_bfloat16 const*>(fin.scales)[e]); + uint4 const rv = *reinterpret_cast(fin.fc2 + static_cast(prow) * H + c0); + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] += v.x * sc; + f[2 * q + 1] += v.y * sc; + } + } + if (x2 != nullptr) + { + // As stock: the routed output is rounded to BF16 first, then the shared expert is added (one more + // BF16 rounding of the FP32 sum). + uint4 const rv = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const routed = __bfloat1622float2(__floats2bfloat162_rn(f[2 * q], f[2 * q + 1])); + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] = routed.x + v.x; + f[2 * q + 1] = routed.y + v.y; + } + } + __nv_bfloat162* o = reinterpret_cast<__nv_bfloat162*>(&raw); +#pragma unroll + for (int q = 0; q < 4; ++q) + o[q] = __floats2bfloat162_rn(f[2 * q], f[2 * q + 1]); + } + else + { + raw = *reinterpret_cast(x + static_cast(t) * H + c0); + } + if (fin.fc2 == nullptr && x2 != nullptr) + { + uint4 const raw2 = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162* a = reinterpret_cast<__nv_bfloat162*>(&raw); + __nv_bfloat162 const* b = reinterpret_cast<__nv_bfloat162 const*>(&raw2); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const fa = __bfloat1622float2(a[q]); + float2 const fb = __bfloat1622float2(b[q]); + a[q] = __floats2bfloat162_rn(fa.x + fb.x, fa.y + fb.y); + } + } + int64_t const slot = parity * half_words + (static_cast(t) * world + rank) * (H / 2) + c0 / 2; + *reinterpret_cast(ws_mc + slot) + = make_int4(static_cast(canon_bf16x2(raw.x)), static_cast(canon_bf16x2(raw.y)), + static_cast(canon_bf16x2(raw.z)), static_cast(canon_bf16x2(raw.w))); + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float s[8] = {}; + float sq = 0.f; + if (active) + { + // The residual row (written grids earlier) and every rank's slot are loaded together; slots still empty are + // re-polled. Ranks are then summed in rank order, as before. + if constexpr ((OPTS & 1) == 0) + rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + constexpr int kMaxWorld = 8; + int* const p0 = ws_uc + parity * half_words + static_cast(t) * world * (H / 2) + c0 / 2; + int4 v[kMaxWorld]; + int4 v2[kMaxWorld]; + bool const two = prepushed == 3; + int* const p1 = p0 + 2 * half_words; // set B + uint32_t pending = (1u << world) - 1u; + uint32_t pending2 = two ? pending : 0u; + long long spins = 0; + while ((pending | pending2) != 0u) + { +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if ((pending >> r) & 1u) + v[r] = ld_volatile_v4(p0 + static_cast(r) * (H / 2)); + if ((pending2 >> r) & 1u) + v2[r] = ld_volatile_v4(p1 + static_cast(r) * (H / 2)); + } +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if (((pending >> r) & 1u) && v[r].x != kEmpty && v[r].y != kEmpty && v[r].z != kEmpty + && v[r].w != kEmpty) + pending &= ~(1u << r); + if (((pending2 >> r) & 1u) && v2[r].x != kEmpty && v2[r].y != kEmpty && v2[r].z != kEmpty + && v2[r].w != kEmpty) + pending2 &= ~(1u << r); + } + if (++spins > (1LL << 31)) + __trap(); + } +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if (r < world) + { + *reinterpret_cast(p0 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t ww[4] = {static_cast(v[r].x), static_cast(v[r].y), + static_cast(v[r].z), static_cast(v[r].w)}; + if (two) + { + *reinterpret_cast(p1 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t const w2[4] = {static_cast(v2[r].x), static_cast(v2[r].y), + static_cast(v2[r].z), static_cast(v2[r].w)}; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + __nv_bfloat162 const u + = __floats2bfloat162_rn(bf16_lo(ww[q]) + bf16_lo(w2[q]), bf16_hi(ww[q]) + bf16_hi(w2[q])); + ww[q] = *reinterpret_cast(&u); + } + } +#pragma unroll + for (int q = 0; q < 4; ++q) + { + s[2 * q] += bf16_lo(ww[q]); + s[2 * q + 1] += bf16_hi(ww[q]); + } + } + } + // Stock rounding points: the rank sum is a BF16 all-reduce output before the residual add. +#pragma unroll + for (int i = 0; i < 8; ++i) + s[i] = __bfloat162float(__float2bfloat16_rn(s[i])); + __nv_bfloat162 const* rr = reinterpret_cast<__nv_bfloat162 const*>(&rraw); + uint32_t upd[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const rf = __bfloat1622float2(rr[q]); + __nv_bfloat162 const u = __floats2bfloat162_rn(s[2 * q] + rf.x, s[2 * q + 1] + rf.y); + upd[q] = *reinterpret_cast(&u); + if constexpr ((OPTS & 16) != 0) + { + float const x0 = s[2 * q] + rf.x, x1 = s[2 * q + 1] + rf.y; + s[2 * q] = x0; + s[2 * q + 1] = x1; + sq += x0 * x0 + x1 * x1; + } + else + { + float2 const uf = __bfloat1622float2(u); + s[2 * q] = uf.x; + s[2 * q + 1] = uf.y; + sq += uf.x * uf.x + uf.y * uf.y; + } + } + *reinterpret_cast(out_residual + static_cast(t) * H + c0) + = make_uint4(upd[0], upd[1], upd[2], upd[3]); + if constexpr ((OPTS & 16) == 0) + { + if constexpr (CAP) + cap_slice[threadIdx.x] = make_uint4(upd[0], upd[1], upd[2], upd[3]); + } + } + if constexpr ((OPTS & 4) != 0) + { + // every CTA (active or not) rotates its call parity once its own poll is done (read only by later grids) + if (threadIdx.x == 0) + flags[blockIdx.x] = parity ^ 1; + } +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + sq += __shfl_xor_sync(0xffffffffu, sq, off); + if ((threadIdx.x & 31) == 0) + warp_sq[threadIdx.x / 32] = sq; + __syncthreads(); + if constexpr ((OPTS & 2) != 0) + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); // peers' xbar initialized (long since) + if (threadIdx.x == 0) + { + float total = 0.f; +#pragma unroll + for (int w = 0; w < THREADS / 32; ++w) + total += warp_sq[w]; + if constexpr ((OPTS & 2) == 0) + for (int c = 0; c < CLUSTER; ++c) + *cluster.map_shared_rank(&cluster_sq[crank], c) = total; + else + { + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + uint32_t const d = static_cast(__cvta_generic_to_shared(&cluster_sq[crank])); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(b), "r"(CLUSTER * 4) + : "memory"); + for (int c = 0; c < CLUSTER; ++c) + { + uint32_t rd, rb; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rd) : "r"(d), "r"(c)); + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rb) : "r"(b), "r"(c)); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(rd), + "r"(__float_as_uint(total)), "r"(rb) + : "memory"); + } + } + } + if constexpr ((OPTS & 2) == 0) + cluster.sync(); + else + { + // every peer's sum landed (st.async complete_tx on this CTA's barrier); no cluster-wide release-arrive + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + uint32_t done = 0; + while (!done) + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}" + : "=r"(done) + : "r"(b) + : "memory"); + } + if constexpr ((OPTS & 16) == 0) + { + if constexpr (CAP) + { + // The row's chunks are spread over the cluster (CTA c holds chunks [c * per, (c + 1) * per)); every CTA + // forms capture_norm's statistic itself, its 128 threads standing in for capture threads ct and ct + 128. + __shared__ float cap_red[kCapWarps]; + int const per = cols / 8; + asm volatile("barrier.cluster.arrive.release.aligned;" ::: "memory"); + asm volatile("barrier.cluster.wait.acquire.aligned;" ::: "memory"); + float ca = 0.f, cb = 0.f; +#pragma unroll + for (int it = 0; it < kCapIters; ++it) + { + int const ka = it * kCapThreads + static_cast(threadIdx.x); + int const kb = ka + kCapThreads / 2; + ca = cap_chunk_sq(*cluster.map_shared_rank(&cap_slice[ka % per], ka / per), ca); + cb = cap_chunk_sq(*cluster.map_shared_rank(&cap_slice[kb % per], kb / per), cb); + } + // Every peer read is done before any CTA moves on (and later exits). + asm volatile("barrier.cluster.arrive.release.aligned;" ::: "memory"); + asm volatile("barrier.cluster.wait.acquire.aligned;" ::: "memory"); + ca = cap_warp_sum(ca); + cb = cap_warp_sum(cb); + if ((threadIdx.x & 31) == 0) + { + cap_red[threadIdx.x / 32] = ca; + cap_red[threadIdx.x / 32 + kCapWarps / 2] = cb; + } + __syncthreads(); + float tot = 0.f; +#pragma unroll + for (int w = 0; w < kCapWarps; ++w) + tot += cap_red[w]; + float const rs = rsqrtf(tot / H + cap_eps); + if (active) + cap_store(s, rs, cap_w + c0, cap_out + static_cast(t) * cap_stride + c0); + } + } + if (active) + { + float total = 0.f; +#pragma unroll + for (int c = 0; c < CLUSTER; ++c) + total += cluster_sq[c]; + float const inv = rsqrtf(total / static_cast(H) + eps); + if constexpr ((OPTS & 1) == 0) + wraw = *reinterpret_cast(norm_w + c0); + __nv_bfloat162 const* wv = reinterpret_cast<__nv_bfloat162 const*>(&wraw); + uint32_t o[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const wf = __bfloat1622float2(wv[q]); + __nv_bfloat162 const n = __floats2bfloat162_rn(s[2 * q] * inv * wf.x, s[2 * q + 1] * inv * wf.y); + o[q] = *reinterpret_cast(&n); + float2 const nf = __bfloat1622float2(n); + s[2 * q] = nf.x; + s[2 * q + 1] = nf.y; + } + *reinterpret_cast(out_normed + static_cast(t) * H + c0) = make_uint4(o[0], o[1], o[2], o[3]); + } + if (out_q != nullptr) + { + // MXFP8 copy of the normed row for the next GEMM, as the CuTe DSL mxfp8 quantize of `normed` (the tree's + // decode quantize, MX-TAC; see mxfp8_ue8m0): four threads hold one 32-column scale block. + float amax = 0.f; + if (active) + { +#pragma unroll + for (int e = 0; e < 8; ++e) + amax = fmaxf(amax, fabsf(s[e])); + } + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 1)); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 2)); + if (active) + { + uint32_t const code = mxfp8_ue8m0(amax); + float const inv_scale = mxfp8_inv_scale(code); + uint32_t packed[2]; +#pragma unroll + for (int h = 0; h < 2; ++h) + { + packed[h] = mxfp8_e4m3x2(s[4 * h], s[4 * h + 1], inv_scale) + | (mxfp8_e4m3x2(s[4 * h + 2], s[4 * h + 3], inv_scale) << 16); + } + *reinterpret_cast(out_q + static_cast(t) * H + c0) = make_uint2(packed[0], packed[1]); + if ((threadIdx.x & 3) == 0) + { + int const kb = c0 / 32; + out_sf[static_cast(t / 128) * sf_k_tiles * 512 + (kb / 4) * 512 + (t % 32) * 16 + + ((t % 128) / 32) * 4 + kb % 4] + = static_cast(code); + } + } + } + if constexpr ((OPTS & 4) == 0) + { + __syncthreads(); + if (threadIdx.x == 0) + flags[blockIdx.x] = parity ^ 1; + } +#elif defined(__CUDA_ARCH__) + __trap(); // compiled for an architecture without the sm_10x features: never a silent no-op +#endif +} + +// Refuses (raises) to launch a kernel whose loaded binary holds no sm_10x code (PTX and SASS version >= 100): the +// device passes for other architectures compile the kernels to a trap, and a build that left out every sm_10x target +// must not reach the GPU at all. +template +bool checkBinary(Kernel kernel, char const* name) +{ + cudaFuncAttributes attr{}; + TLLM_CUDA_CHECK(cudaFuncGetAttributes(&attr, reinterpret_cast(kernel))); + TLLM_CHECK_WITH_INFO(attr.ptxVersion >= 100 && attr.binaryVersion >= 100, + "%s: the loaded binary holds no sm_10x code (PTX %d, SASS %d): rebuild with an sm_100f / sm_107a target", name, + attr.ptxVersion, attr.binaryVersion); + return true; +} + +template +void launch(MinimaxM3Sm107Tp2McAllreduceNormParams const& p, bool pdl, cudaStream_t stream) +{ + int const M = p.numTokens, H = p.hidden; + TLLM_CHECK_WITH_INFO(H % (CLUSTER * 8) == 0 && H / CLUSTER / 8 <= THREADS, "unsupported hidden size %d", H); + TLLM_CHECK_WITH_INFO(M >= 1 && M <= MAXT, "%d tokens do not fit the %d-token bucket", M, MAXT); + auto kernel = p.capOut != nullptr ? mc_allreduce_norm_kernel + : mc_allreduce_norm_kernel; + static bool const binaryOk + = checkBinary(mc_allreduce_norm_kernel, "mc_allreduce_norm_kernel") + && checkBinary( + mc_allreduce_norm_kernel, "mc_allreduce_norm_kernel (capture)"); + (void) binaryOk; + // Both instantiations (with and without the capture epilogue) get the same attributes, whichever runs first. + static bool const configured = []() + { + auto setup = [](auto k) + { + bool ok = CLUSTER <= 8 + || cudaFuncSetAttribute(k, cudaFuncAttributeNonPortableClusterSizeAllowed, 1) == cudaSuccess; + if ((OPTS & 8) != 0) + ok = ok + && cudaFuncSetAttribute( + k, cudaFuncAttributePreferredSharedMemoryCarveout, cudaSharedmemCarveoutMaxShared) + == cudaSuccess; + return ok; + }; + bool const ok = setup(mc_allreduce_norm_kernel) + && setup(mc_allreduce_norm_kernel); + if (!ok) + cudaGetLastError(); + return ok; + }(); + TLLM_CHECK_WITH_INFO(configured, "cluster size not supported"); + FinalizeArgs const fin{p.fc2, p.perm, p.scales, p.scaleFp32 ? 1 : 0, p.topk}; + cudaLaunchConfig_t cfg = {}; + // The grid is sized to the call's rows: min(M, MAXT) clusters. Each (row, slice) slot keeps its own parity flag, + // read by the producers before they push and rotated only by the slot's CTA after it polled: a slot that a call + // does not use keeps its parity, and every rank runs the same calls, so the parities stay consistent when M + // changes between calls. + cfg.gridDim = dim3((M < MAXT ? M : MAXT) * CLUSTER); + cfg.blockDim = dim3(THREADS); + cfg.stream = stream; + if (p.capOut != nullptr) + { + TLLM_CHECK_WITH_INFO((OPTS & 16) == 0 && H == kCapHidden && 2 * THREADS == kCapThreads, + "the capture epilogue takes H %d on the 128-thread cluster kernel, not the drafter boundary", kCapHidden); + cfg.dynamicSmemBytes = static_cast(H / CLUSTER / 8) * sizeof(uint4); + } + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + attr[1].id = cudaLaunchAttributeClusterDimension; + attr[1].val.clusterDim.x = CLUSTER; + attr[1].val.clusterDim.y = 1; + attr[1].val.clusterDim.z = 1; + cfg.attrs = attr; + cfg.numAttrs = 2; + cudaError_t const err = cudaLaunchKernelEx(&cfg, kernel, p.x, p.x2, fin, M, H, p.residual, p.normWeight, p.eps, + p.wsUc, p.wsMc, p.flags, p.rank, p.world, p.outNormed, p.outResidual, p.outQuant, p.outSf, (H / 32 + 3) / 4, + p.prepushed, p.capOut, p.capStride, p.capWeight, p.capEps); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +// ws_uc / ws_mc are this rank's unicast and multicast views of ONE exchange buffer (they alias): no __restrict__. +// prepushed >= 2 contract: the producer (m3_mx_gemv's o_proj push, the m3_moe engine's finalize push) passes its own +// grid dependency, pushes its rows into the exchange, and only then releases this grid (griddepcontrol +// .launch_dependents); this kernel therefore skips griddepcontrol.wait and reads only the pushed (Lamport-polled) +// words, its parity flags and the residual / norm weight, which were written two or more grids earlier. +// +// The TP2 row kernel: one CTA of H / 8 = 768 threads per token row, used for every call but the MoE finalize. Its +// residual add + RMSNorm is TRT-LLM's rms_norm_kernel (customAllReduceKernels.cu), the stock fused norm after the +// all-reduce on this target, operation for operation at H 6144: thread j owns columns [8j, 8j + 8), its sum of squares +// is acc += v * v over them in order, then the warp butterfly 16..1, the 24 warp sums through shared memory into a +// second 32-lane butterfly (lanes >= 24 zero), rsqrtf(acc / H + eps) and (v * denom) * w. With the BF16 rank sum +// before the residual add, the normed rows, the updated residual and the MXFP8 copy are bitwise the stock ones. The +// exchange slots and the per-(row, slice) parity flags keep the cluster kernel's layout (slice = 96 threads), so +// producers that push (the o_proj GEMV) are unchanged. +constexpr int kSlices = 8; +constexpr int kSliceThreads = 96; +constexpr int kRowThreads = 768; + +// grid = M CTAs (one per row), THREADS = H / 8 threads. WORLD ranks (compile time). DRAFTER (opts bit 16): the Eagle3 +// drafter boundary, whose stock is a plain BF16 all-reduce then FlashInfer fused_add_rmsnorm: the RMS statistic and +// the normalization take the unrounded FP32 sum bf16(rank sum) + residual (only the stored residual is rounded). +// ws_uc / ws_mc alias (one exchange buffer): no __restrict__. +template +__global__ void __launch_bounds__(THREADS, 1) mc_allreduce_norm_row_kernel(__nv_bfloat16 const* __restrict__ x, + __nv_bfloat16 const* __restrict__ x2, int H, __nv_bfloat16 const* __restrict__ residual, + __nv_bfloat16 const* __restrict__ norm_w, float eps, int* ws_uc, int* ws_mc, int* __restrict__ flags, int rank, + int maxt, __nv_bfloat16* __restrict__ out_normed, __nv_bfloat16* __restrict__ out_residual, + uint8_t* __restrict__ out_q, uint8_t* __restrict__ out_sf, int sf_k_tiles, int prepushed, + __nv_bfloat16* __restrict__ cap_out, int64_t cap_stride, __nv_bfloat16 const* __restrict__ cap_w, float cap_eps) +{ +#if MINIMAX_M3_KERNELS_ENABLED + __shared__ float warp_sq[THREADS / 32]; + __shared__ float total_sh; + extern __shared__ __align__(16) uint4 cap_row[]; // [THREADS] the updated row's chunks (cap_out only) + int const t = blockIdx.x; + int const j = threadIdx.x; + int const c0 = j * 8; + int const slice = j / kSliceThreads; + int64_t const half_words = static_cast(maxt) * WORLD * (H / 2); + if (prepushed < 2) + asm volatile("griddepcontrol.wait;" ::: "memory"); + int const parity = *reinterpret_cast(flags + t * kSlices + slice) & 1; + uint4 const rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + uint4 const wraw = *reinterpret_cast(norm_w + c0); + if (!prepushed) + { + uint4 raw = *reinterpret_cast(x + static_cast(t) * H + c0); + if (x2 != nullptr) + { + uint4 const raw2 = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162* a = reinterpret_cast<__nv_bfloat162*>(&raw); + __nv_bfloat162 const* b = reinterpret_cast<__nv_bfloat162 const*>(&raw2); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const fa = __bfloat1622float2(a[q]); + float2 const fb = __bfloat1622float2(b[q]); + a[q] = __floats2bfloat162_rn(fa.x + fb.x, fa.y + fb.y); + } + } + int64_t const slot = parity * half_words + (static_cast(t) * WORLD + rank) * (H / 2) + c0 / 2; + *reinterpret_cast(ws_mc + slot) + = make_int4(static_cast(canon_bf16x2(raw.x)), static_cast(canon_bf16x2(raw.y)), + static_cast(canon_bf16x2(raw.z)), static_cast(canon_bf16x2(raw.w))); + } + asm volatile("griddepcontrol.launch_dependents;"); + float s[8] = {}; + float sq = 0.f; + { + int* const p0 = ws_uc + parity * half_words + static_cast(t) * WORLD * (H / 2) + c0 / 2; + int* const p1 = p0 + 2 * half_words; // set B + bool const two = prepushed == 3; + int4 v[WORLD]; + int4 v2[WORLD]; + uint32_t pending = (1u << WORLD) - 1u; + uint32_t pending2 = two ? pending : 0u; + long long spins = 0; + while ((pending | pending2) != 0u) + { +#pragma unroll + for (int r = 0; r < WORLD; ++r) + { + if ((pending >> r) & 1u) + v[r] = ld_volatile_v4(p0 + static_cast(r) * (H / 2)); + if ((pending2 >> r) & 1u) + v2[r] = ld_volatile_v4(p1 + static_cast(r) * (H / 2)); + } +#pragma unroll + for (int r = 0; r < WORLD; ++r) + { + if (((pending >> r) & 1u) && v[r].x != kEmpty && v[r].y != kEmpty && v[r].z != kEmpty + && v[r].w != kEmpty) + pending &= ~(1u << r); + if (((pending2 >> r) & 1u) && v2[r].x != kEmpty && v2[r].y != kEmpty && v2[r].z != kEmpty + && v2[r].w != kEmpty) + pending2 &= ~(1u << r); + } + if (++spins > (1LL << 31)) + __trap(); + } +#pragma unroll + for (int r = 0; r < WORLD; ++r) + { + *reinterpret_cast(p0 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t ww[4] = {static_cast(v[r].x), static_cast(v[r].y), + static_cast(v[r].z), static_cast(v[r].w)}; + if (two) + { + *reinterpret_cast(p1 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t const w2[4] = {static_cast(v2[r].x), static_cast(v2[r].y), + static_cast(v2[r].z), static_cast(v2[r].w)}; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + __nv_bfloat162 const u + = __floats2bfloat162_rn(bf16_lo(ww[q]) + bf16_lo(w2[q]), bf16_hi(ww[q]) + bf16_hi(w2[q])); + ww[q] = *reinterpret_cast(&u); + } + } +#pragma unroll + for (int q = 0; q < 4; ++q) + { + s[2 * q] += bf16_lo(ww[q]); + s[2 * q + 1] += bf16_hi(ww[q]); + } + } + // Stock rounding points: the rank sum is a BF16 all-reduce output before the residual add. +#pragma unroll + for (int i = 0; i < 8; ++i) + s[i] = __bfloat162float(__float2bfloat16_rn(s[i])); + __nv_bfloat162 const* rr = reinterpret_cast<__nv_bfloat162 const*>(&rraw); + uint32_t upd[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const rf = __bfloat1622float2(rr[q]); + __nv_bfloat162 const u = __floats2bfloat162_rn(s[2 * q] + rf.x, s[2 * q + 1] + rf.y); + upd[q] = *reinterpret_cast(&u); + if constexpr (DRAFTER) + { + s[2 * q] += rf.x; + s[2 * q + 1] += rf.y; + } + else + { + float2 const uf = __bfloat1622float2(u); + s[2 * q] = uf.x; + s[2 * q + 1] = uf.y; + } + } +#pragma unroll + for (int e = 0; e < 8; ++e) + sq += s[e] * s[e]; + *reinterpret_cast(out_residual + static_cast(t) * H + c0) + = make_uint4(upd[0], upd[1], upd[2], upd[3]); + if constexpr (!DRAFTER) + { + if constexpr (CAP) + cap_row[j] = make_uint4(upd[0], upd[1], upd[2], upd[3]); + } + } + // TRT-LLM reduce_fusion::block_reduce_sum. + sq += __shfl_xor_sync(~0, sq, 16); + sq += __shfl_xor_sync(~0, sq, 8); + sq += __shfl_xor_sync(~0, sq, 4); + sq += __shfl_xor_sync(~0, sq, 2); + sq += __shfl_xor_sync(~0, sq, 1); + if ((j & 31) == 0) + warp_sq[j / 32] = sq; + __syncthreads(); + { + float val = (j & 31) < THREADS / 32 ? warp_sq[j & 31] : 0.f; + val += __shfl_xor_sync(~0, val, 16); + val += __shfl_xor_sync(~0, val, 8); + val += __shfl_xor_sync(~0, val, 4); + val += __shfl_xor_sync(~0, val, 2); + val += __shfl_xor_sync(~0, val, 1); + if (j == 0) + total_sh = val; + } + // The parity of this row's slices rotates once every thread has read the exchange (later grids only). + if (j % kSliceThreads == 0) + flags[t * kSlices + slice] = parity ^ 1; + __syncthreads(); + if constexpr (!DRAFTER) + { + if constexpr (CAP) + { + // The whole row is in this CTA: threads 0..255 run capture_norm's chains over the staged chunks. + __shared__ float cap_red[kCapWarps]; + if (j < kCapThreads) + { + float css = 0.f; +#pragma unroll + for (int it = 0; it < kCapIters; ++it) + css = cap_chunk_sq(cap_row[it * kCapThreads + j], css); + css = cap_warp_sum(css); + if ((j & 31) == 0) + cap_red[j / 32] = css; + } + __syncthreads(); + float tot = 0.f; +#pragma unroll + for (int w = 0; w < kCapWarps; ++w) + tot += cap_red[w]; + float const rs = rsqrtf(tot / H + cap_eps); + cap_store(s, rs, cap_w + c0, cap_out + static_cast(t) * cap_stride + c0); + } + } + float const inv = rsqrtf(total_sh / H + eps); + __nv_bfloat162 const* wv = reinterpret_cast<__nv_bfloat162 const*>(&wraw); + uint32_t o[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const wf = __bfloat1622float2(wv[q]); + __nv_bfloat162 const n = __floats2bfloat162_rn(s[2 * q] * inv * wf.x, s[2 * q + 1] * inv * wf.y); + o[q] = *reinterpret_cast(&n); + float2 const nf = __bfloat1622float2(n); + s[2 * q] = nf.x; + s[2 * q + 1] = nf.y; + } + *reinterpret_cast(out_normed + static_cast(t) * H + c0) = make_uint4(o[0], o[1], o[2], o[3]); + if (out_q != nullptr) + { + float amax = 0.f; +#pragma unroll + for (int e = 0; e < 8; ++e) + amax = fmaxf(amax, fabsf(s[e])); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 1)); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 2)); + uint32_t const code = mxfp8_ue8m0(amax); + float const inv_scale = mxfp8_inv_scale(code); + uint32_t packed[2]; +#pragma unroll + for (int h = 0; h < 2; ++h) + { + packed[h] = mxfp8_e4m3x2(s[4 * h], s[4 * h + 1], inv_scale) + | (mxfp8_e4m3x2(s[4 * h + 2], s[4 * h + 3], inv_scale) << 16); + } + *reinterpret_cast(out_q + static_cast(t) * H + c0) = make_uint2(packed[0], packed[1]); + if ((j & 3) == 0) + { + int const kb = c0 / 32; + out_sf[static_cast(t / 128) * sf_k_tiles * 512 + (kb / 4) * 512 + (t % 32) * 16 + + ((t % 128) / 32) * 4 + kb % 4] + = static_cast(code); + } + } +#elif defined(__CUDA_ARCH__) + __trap(); // compiled for an architecture without the sm_10x features: never a silent no-op +#endif +} + +void launchRow( + MinimaxM3Sm107Tp2McAllreduceNormParams const& p, int32_t maxTokens, bool drafter, bool pdl, cudaStream_t stream) +{ + TLLM_CHECK_WITH_INFO(p.hidden == kRowThreads * 8 && p.world == 2, "the row kernel takes H 6144 at 2 ranks"); + static bool const binaryOk + = checkBinary(mc_allreduce_norm_row_kernel<2, kRowThreads, false, false>, "mc_allreduce_norm_row_kernel") + && checkBinary( + mc_allreduce_norm_row_kernel<2, kRowThreads, true, false>, "mc_allreduce_norm_row_kernel (drafter)") + && checkBinary( + mc_allreduce_norm_row_kernel<2, kRowThreads, false, true>, "mc_allreduce_norm_row_kernel (capture)"); + (void) binaryOk; + TLLM_CHECK_WITH_INFO(p.numTokens >= 1 && p.numTokens <= maxTokens, "%d tokens do not fit the %d-token bucket", + p.numTokens, maxTokens); + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(p.numTokens); + cfg.blockDim = dim3(kRowThreads); + cfg.stream = stream; + if (p.capOut != nullptr) + { + TLLM_CHECK_WITH_INFO( + !drafter && p.hidden == kCapHidden, "the capture epilogue takes H %d, not the drafter", kCapHidden); + cfg.dynamicSmemBytes = static_cast(kRowThreads) * sizeof(uint4); + } + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + // The capture epilogue never runs on the drafter boundary (checked above), so there are three instantiations. + auto kernel = drafter ? mc_allreduce_norm_row_kernel<2, kRowThreads, true, false> + : p.capOut != nullptr ? mc_allreduce_norm_row_kernel<2, kRowThreads, false, true> + : mc_allreduce_norm_row_kernel<2, kRowThreads, false, false>; + cudaError_t const err = cudaLaunchKernelEx(&cfg, kernel, p.x, p.x2, p.hidden, p.residual, p.normWeight, p.eps, + p.wsUc, p.wsMc, p.flags, p.rank, maxTokens, p.outNormed, p.outResidual, p.outQuant, p.outSf, + (p.hidden / 32 + 3) / 4, p.prepushed, p.capOut, p.capStride, p.capWeight, p.capEps); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +constexpr int kCluster = 8; +constexpr int kThreads = 128; + +} // namespace + +bool isMinimaxM3Sm107Tp2McAllreduceNormSupported(int32_t maxTokens, int32_t opts) +{ + return (maxTokens == 8 && (opts == 7 || opts == 15 || opts == 23 || opts == 31)) || (maxTokens == 16 && opts == 15) + || (maxTokens == 48 && opts == 15); +} + +void invokeMinimaxM3Sm107Tp2McAllreduceNorm(MinimaxM3Sm107Tp2McAllreduceNormParams const& params, int32_t maxTokens, + int32_t opts, bool pdl, cudaStream_t stream) +{ + if (params.fc2 == nullptr && params.world == 2 && params.hidden == kRowThreads * 8) + { + launchRow(params, maxTokens, (opts & 16) != 0, pdl, stream); + return; + } + if (maxTokens == 8 && opts == 7) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 15) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 23) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 31) + { + launch(params, pdl, stream); + } + else if (maxTokens == 16 && opts == 15) + { + launch(params, pdl, stream); + } + else if (maxTokens == 48 && opts == 15) + { + launch(params, pdl, stream); + } + else + { + TLLM_THROW("unsupported (bucket %d, opts %d)", maxTokens, opts); + } +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.h new file mode 100644 index 000000000000..e8ade9c62c3e --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.h @@ -0,0 +1,83 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Operands of the MiniMax-M3 multicast all-reduce + residual + RMSNorm kernel (decode, at most 48 tokens). +//! +//! Per rank the kernel forms the partial `bf16(x + x2)` (or, in finalize mode, +//! `bf16(bf16(sum_k scale * fc2[perm]) + x2)`, the routed sum rounded before the shared expert is added, as stock), +//! pushes it through the multicast mapping `wsMc` into every rank's exchange, polls its own unicast view `wsUc`, +//! sums the ranks in FP32 in rank order, adds the residual (BF16 round) and applies RMSNorm. With `outQuant` it +//! also writes the MXFP8 copy of the normed rows (E4M3 [numTokens, hidden] and swizzled UE8M0 scales). +struct MinimaxM3Sm107Tp2McAllreduceNormParams +{ + __nv_bfloat16 const* x{nullptr}; //!< [numTokens, hidden] partial; in finalize mode the FC2 rows [rows, hidden]. + __nv_bfloat16 const* x2{nullptr}; //!< Optional second addend [numTokens, hidden]. + __nv_bfloat16 const* fc2{nullptr}; //!< Finalize mode: the permuted FC2 output (same buffer as x); null otherwise. + int32_t const* perm{nullptr}; //!< Finalize mode: [numTokens, topk] expanded index -> permuted row, -1 dropped. + void const* scales{nullptr}; //!< Finalize mode: [numTokens, topk] routing weights (fp32 or bf16). + bool scaleFp32{false}; + int32_t topk{0}; + int32_t numTokens{0}; + int32_t hidden{0}; + __nv_bfloat16 const* residual{nullptr}; //!< [numTokens, hidden] + __nv_bfloat16 const* normWeight{nullptr}; //!< [hidden] + float eps{0.F}; + int32_t* wsUc{nullptr}; //!< This rank's unicast view of the exchange (int32 words, re-armed after reading). + int32_t* wsMc{nullptr}; //!< The multicast mapping of the exchange. + int32_t* flags{nullptr}; //!< Per-CTA call-parity counters, [maxTokens * 8]. + int32_t rank{0}; + int32_t world{0}; + __nv_bfloat16* outNormed{nullptr}; + __nv_bfloat16* outResidual{nullptr}; + uint8_t* outQuant{nullptr}; //!< Optional E4M3 [numTokens, hidden]. + uint8_t* outSf{nullptr}; //!< Optional UE8M0 scales, 128x4-swizzled. + //! 0: push x (+ x2) here; 2: a producer already pushed x and launched this grid after its pushes; 3: as 2, with x + //! in exchange set A and x2 in set B. + int32_t prepushed{0}; + //! Optional Eagle3 capture of a capture layer's boundary: capOut[t * capStride + c] = the m3_spec_tail + //! capture_norm of the updated residual row with weight capWeight [hidden] and capEps, bit for bit (hidden 6144, + //! not the drafter boundary). + __nv_bfloat16* capOut{nullptr}; + int64_t capStride{0}; + __nv_bfloat16 const* capWeight{nullptr}; + float capEps{0.F}; +}; + +//! The (bucket, epilogue options) pairs this launcher is built for: (8, 7), (8, 15), (16, 15) and (48, 15). +bool isMinimaxM3Sm107Tp2McAllreduceNormSupported(int32_t maxTokens, int32_t opts); + +//! Launch with 8-CTA clusters of 128 threads, one cluster per token. `maxTokens` selects the exchange bucket; +//! `opts` the epilogue variant (every variant is bit-identical). +void invokeMinimaxM3Sm107Tp2McAllreduceNorm(MinimaxM3Sm107Tp2McAllreduceNormParams const& params, int32_t maxTokens, + int32_t opts, bool pdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.cu new file mode 100644 index 000000000000..da30d840cc2e --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.cu @@ -0,0 +1,339 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.h" + +#include +#include +#include + +// MiniMax-M3 NVFP4 QKV + indexer norm/RoPE/KV-insert producer with an early PDL trigger, at the SM107 TP2 heads. +// +// The SM107 TP2 target's copy of minimaxM3/producerPw.cu (the sm_103 TP4 / TP8 targets keep theirs). Same math and +// outputs as minimaxM3Nvfp4QKVIndexerNormRopeKVInsertKernel (fusedQKNormRopeKernel.cu, built with --use_fast_math like +// the rest of the library), specialized to 32 query, 2 KV and 2 index heads per rank (one index-K head), head_dim 128, +// rotary_dim 64, page 128, NVFP4 main KV and FP8 index-K. One warp per (token, packed head); K and V rows go to their +// head's block of the HND pools (the native kernel's head strides). +// +// The decode producer is latency bound: after griddepcontrol.wait the native kernel runs a chain of dependent global +// loads (packed row -> norm weight -> position id -> RoPE row -> cache slot -> quant scale). This kernel reads the +// constant norm weights and KV quant scale before griddepcontrol.wait and issues the packed row, position id and cache +// slot loads together after it, followed by the RoPE row; with `prewaitStepInputs` the position id, the slot and the +// RoPE row are read before the wait too, leaving only the packed row after it. With `early` it also calls +// griddepcontrol.launch_dependents on entry: consumers read its outputs only after their own griddepcontrol.wait, which +// waits for this grid to complete, so the early trigger only lets the next kernel launch and run its pre-wait prologue +// while this one runs. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +namespace producerPw2 +{ +namespace +{ + +constexpr int kHeadDim = 128; +constexpr int kRotaryDim = 64; +constexpr int kPageSize = 128; +constexpr int kElems = kHeadDim / 32; +constexpr int kBlock = 256; + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ uint32_t sf_group_mask4() +{ + uint32_t laneId; + asm("mov.u32 %0, %%laneid;" : "=r"(laneId)); + return 0xFu << (laneId & ~3u); +} + +__device__ __forceinline__ uint32_t fp32x8_to_e2m1(float (&a)[8]) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + uint32_t val; + asm volatile( + "{\n" + ".reg .b8 byte0;\n" + ".reg .b8 byte1;\n" + ".reg .b8 byte2;\n" + ".reg .b8 byte3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte0, %2, %1;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte1, %4, %3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte2, %6, %5;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte3, %8, %7;\n" + "mov.b32 %0, {byte0, byte1, byte2, byte3};\n" + "}" + : "=r"(val) + : "f"(a[0]), "f"(a[1]), "f"(a[2]), "f"(a[3]), "f"(a[4]), "f"(a[5]), "f"(a[6]), "f"(a[7])); + return val; +#else + return 0; +#endif +} + +__device__ __forceinline__ void store_fp8x4(__nv_fp8_e4m3* out, int64_t off, float const (&e)[kElems]) +{ + __nv_fp8x2_e4m3 const low(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const high(make_float2(e[2], e[3])); + *reinterpret_cast(out + off) = static_cast(low.__x) | (static_cast(high.__x) << 16); +} + +__device__ __forceinline__ uint16_t quantize_fp4x4(float const (&e)[4], float secondLevelScale, uint8_t* scaleOut) +{ + __nv_bfloat16 rounded[4]; + float localMax = 0.0F; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + rounded[i] = __float2bfloat16_rn(e[i]); + localMax = fmaxf(localMax, fabsf(__bfloat162float(rounded[i]))); + } + uint32_t const groupMask = sf_group_mask4(); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 1)); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 2)); + float const scaleValue = secondLevelScale * (localMax * rcp_approx_ftz(6.0F)); + __nv_fp8_e4m3 const encodedScale(scaleValue); + if (scaleOut != nullptr) + *scaleOut = encodedScale.__x; + float const narrowedScale = static_cast(encodedScale); + float const outputScale + = localMax != 0.0F ? rcp_approx_ftz(narrowedScale * rcp_approx_ftz(secondLevelScale)) : 0.0F; + float q[8] = {}; +#pragma unroll + for (int i = 0; i < 4; ++i) + q[i] = __bfloat162float(rounded[i]) * outputScale; + return static_cast(fp32x8_to_e2m1(q)); +} + +template +__global__ void __launch_bounds__(kBlock) m3_producer_kernel(Params p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + constexpr int kTotalHeads = kHeadsQ + 2 * kHeadsKV + kHeadsIndex + 1; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (kEarly) + { + asm volatile("griddepcontrol.launch_dependents;"); + } +#endif + int const warpId = threadIdx.x / 32; + int const laneId = threadIdx.x % 32; + int const globalWarp = blockIdx.x * (kBlock / 32) + warpId; + int const tokenIdx = globalWarp / kTotalHeads; + int const localHead = globalWarp % kTotalHeads; + if (tokenIdx >= p.numTokens) + return; + + constexpr int kBegin = kHeadsQ; + constexpr int vBegin = kBegin + kHeadsKV; + constexpr int indexQBegin = vBegin + kHeadsKV; + constexpr int indexKHead = indexQBegin + kHeadsIndex; + bool const isQ = localHead < kBegin; + bool const isK = localHead >= kBegin && localHead < vBegin; + bool const isV = localHead >= vBegin && localHead < indexQBegin; + bool const isIndexQ = localHead >= indexQBegin && localHead < indexKHead; + bool const isIndexK = localHead == indexKHead; + + // Constant data read before the wait: norm weights and the KV quant scale. + float w[kElems] = {}; + float quantScale = 0.0F; + __nv_bfloat16 const* normWeight + = isQ ? p.qWeight : (isK ? p.kWeight : (isIndexQ ? p.indexQWeight : p.indexKWeight)); + if (!isV) + { + uint2 const wv = *reinterpret_cast(normWeight + laneId * kElems); + float2 const w01 = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&wv.x)); + float2 const w23 = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&wv.y)); + w[0] = w01.x; + w[1] = w01.y; + w[2] = w23.x; + w[3] = w23.y; + } + if (isK || isV) + { + quantScale = p.kvQuantScale[isV ? 2 : 1]; + } + + int64_t const inputOffset = (static_cast(tokenIdx) * kTotalHeads + localHead) * kHeadDim + laneId * kElems; + bool const rotates = !isV && laneId * kElems < kRotaryDim; + bool const slotted = isK || isV || isIndexK; + int const coefficient = (laneId * kElems) % (kRotaryDim / 2); + // kPrewait: the position id, the cache slot and the RoPE row are read before the wait. They are per-step metadata + // written before the replay (host-to-device) or by plain-launched kernels earlier in the step (position_ids by the + // step's input preprocessing, the slots by msa_kv_update or its torch fallback), so they are final before this + // grid can start whatever its predecessors trigger; the RoPE table is constant. + int positionId = 0; + int slot = 0; + float4 c = make_float4(0.0F, 0.0F, 0.0F, 0.0F); + float4 s = c; + auto const loadStepInputs = [&]() + { + positionId = rotates ? p.positionIds[tokenIdx] : 0; + slot = slotted ? p.outCacheLoc[tokenIdx] : 0; + if (rotates) + { + int64_t const ropeRow = static_cast(positionId) * kRotaryDim; + c = *reinterpret_cast(p.rotaryCosSin + ropeRow + coefficient); + s = *reinterpret_cast(p.rotaryCosSin + ropeRow + kRotaryDim / 2 + coefficient); + } + }; + if constexpr (kPrewait) + { + loadStepInputs(); + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + // The packed row comes from the predecessor grid; without kPrewait the step inputs are issued with it, so the + // native kernel's serial chain of dependent loads becomes two round trips. + uint2 const packed = *reinterpret_cast(p.packed + inputOffset); + if constexpr (!kPrewait) + { + loadStepInputs(); + } + float cosine[kElems] = {c.x, c.y, c.z, c.w}; + float sine[kElems] = {s.x, s.y, s.z, s.w}; + + float e[kElems]; + float sumSquares = 0.0F; +#pragma unroll + for (int pair = 0; pair < 2; ++pair) + { + float2 const v = __bfloat1622float2( + *reinterpret_cast<__nv_bfloat162 const*>(reinterpret_cast(&packed) + pair)); + e[2 * pair] = v.x; + e[2 * pair + 1] = v.y; + if (!isV) + sumSquares += v.x * v.x + v.y * v.y; + } + + if (!isV) + { +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + sumSquares += __shfl_xor_sync(0xffffffff, sumSquares, mask, 32); + float const rmsReciprocal = rsqrtf(sumSquares / static_cast(kHeadDim) + p.eps); +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + e[i] *= rmsReciprocal * (1.0F + w[i]); + } + constexpr int kPairOffset = (kRotaryDim / 2) / kElems; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + int const dim = laneId * kElems + i; + float paired = __shfl_xor_sync(0xffffffff, e[i], kPairOffset); + if (dim < kRotaryDim) + { + if (dim < kRotaryDim / 2) + paired = -paired; + // The native kernel's contraction of e * cos + paired * sin; spelled out so this build matches it. + e[i] = __fmaf_rn(paired, sine[i], __fmul_rn(e[i], cosine[i])); + } + } + } + + if (isQ) + { + store_fp8x4(p.qOut, (static_cast(tokenIdx) * kHeadsQ + localHead) * kHeadDim + laneId * kElems, e); + return; + } + if (isIndexQ) + { + store_fp8x4(p.indexQOut, + (static_cast(tokenIdx) * kHeadsIndex + localHead - indexQBegin) * kHeadDim + laneId * kElems, e); + return; + } + + if (slot < 0) + return; + int const page = slot / kPageSize; + if (page >= p.numPages) + return; + int const withinPage = slot & (kPageSize - 1); + if (isIndexK) + { + store_fp8x4(p.indexK, + static_cast(page) * p.indexPageStride + static_cast(withinPage) * p.indexTokenStride + + laneId * kElems, + e); + return; + } + + int const head = isK ? localHead - kBegin : localHead - vBegin; + int const plane = isV ? 1 : 0; + int const scaleColumn = laneId / 4; + int64_t const scaleWithinPage = isV + ? static_cast(withinPage / 4) * 32 + static_cast(scaleColumn) * 4 + withinPage % 4 + : static_cast(withinPage) * 8 + scaleColumn; + int64_t const scaleOffset = static_cast(page) * p.scalePageStride + + static_cast(plane) * p.scalePlaneStride + static_cast(head) * p.scaleHeadStride + + scaleWithinPage; + uint8_t* scaleOut = laneId % 4 == 0 ? p.kvScale + scaleOffset : nullptr; + uint16_t const packedFp4 = quantize_fp4x4(e, quantScale, scaleOut); + int64_t const dataOffset = static_cast(page) * p.dataPageStride + + static_cast(plane) * p.dataPlaneStride + static_cast(head) * p.dataHeadStride + + static_cast(withinPage) * p.dataTokenStride + laneId * 2; + *reinterpret_cast(p.kvData + dataOffset) = packedFp4; +#endif +} + +template +cudaError_t launch(Params const& p, int numTokens, bool pdl, cudaStream_t stream) +{ + constexpr int kTotalHeads = kHeadsQ + 2 * kHeadsKV + kHeadsIndex + 1; + int const warps = numTokens * kTotalHeads; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3((warps + kBlock / 32 - 1) / (kBlock / 32)); + cfg.blockDim = dim3(kBlock); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, m3_producer_kernel, p); +} + +} // namespace + +cudaError_t invokeProducerPw2( + Params const& p, int numTokens, bool pdl, bool early, bool prewaitStepInputs, cudaStream_t stream) +{ + if (numTokens <= 0) + return cudaSuccess; + if (prewaitStepInputs) + { + return early ? launch(p, numTokens, pdl, stream) : launch(p, numTokens, pdl, stream); + } + return early ? launch(p, numTokens, pdl, stream) : launch(p, numTokens, pdl, stream); +} + +} // namespace producerPw2 +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.h new file mode 100644 index 000000000000..9e94868af843 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.h @@ -0,0 +1,76 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +//! The MiniMax-M3 NVFP4 QKV + indexer RMSNorm / RoPE / KV-insert producer at the SM107 TP2 heads (32 query, 2 KV and +//! 2 index heads per rank) with its constant loads before the PDL grid-dependency wait and an optional early dependents +//! trigger. Same outputs, bit for bit, as minimaxM3Nvfp4QKVIndexerNormRopeKVInsert. See producerPw2.cu. +namespace producerPw2 +{ + +struct Params +{ + __nv_bfloat16 const* packed; + __nv_fp8_e4m3* qOut; + __nv_fp8_e4m3* indexQOut; + uint8_t* kvData; + uint8_t* kvScale; + __nv_fp8_e4m3* indexK; + int const* outCacheLoc; + float const* kvQuantScale; + int64_t dataPageStride, dataPlaneStride, dataHeadStride, dataTokenStride; + int64_t scalePageStride, scalePlaneStride, scaleHeadStride; + int64_t indexPageStride, indexTokenStride, numPages; + int numTokens; + float eps; + __nv_bfloat16 const* qWeight; + __nv_bfloat16 const* kWeight; + __nv_bfloat16 const* indexQWeight; + __nv_bfloat16 const* indexKWeight; + float const* rotaryCosSin; + int const* positionIds; +}; + +//! The heads per rank this producer is built for. +constexpr int kHeadsQ = 32; +constexpr int kHeadsKV = 2; +constexpr int kHeadsIndex = 2; + +//! Launch over `numTokens` tokens, one warp per (token, packed head). `prewaitStepInputs` reads the position ids, the +//! cache slots and the RoPE rows before the grid-dependency wait (they must not be written by a PDL predecessor). +cudaError_t invokeProducerPw2( + Params const& p, int numTokens, bool pdl, bool early, bool prewaitStepInputs, cudaStream_t stream); + +} // namespace producerPw2 +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.cu new file mode 100644 index 000000000000..878282b18839 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.cu @@ -0,0 +1,596 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.h" + +#include "tensorrt_llm/common/assert.h" + +#include +#include + +// Fused sampled verification for the MiniMax-M3 Eagle3 target head (TP-sharded vocabulary, decode rows <= 48). +// +// sm_107 (Rubin) TP2 copy of kernels/minimaxM3/sampledAccept.cu: identical algorithm and arithmetic; the vocab shard +// limit is raised to the TP2 shard (200064 / 2 = 100032 -> kMaxVs 100352), so a CTA stages up to kMaxVs / kSplits +// BF16 logits (25 KB at 8 splits, 50 KB at 4) and the launch opts in to that dynamic shared memory. +// +// Replaces, per decode step, the full-vocabulary all-gather of the target logits, the fused sampling kernel and the +// strict-acceptance ops for rows sampled with temperature and top-p (the AgentX contract). Rows with top-k or min-p +// active must stay on the generic path; the caller decides. +// +// Sampling is top-p rejection sampling, distributed over the TP ranks. With w = exp(logit / T - max) over the FULL +// vocabulary and Z = sum(w), a token is in the nucleus iff the mass strictly above its weight is at most top_p * Z +// (the fused sampling kernel's kept set, ties included). A candidate is drawn proportionally to w and accepted iff it +// is in the nucleus; a rejected candidate lies below the nucleus, so the next candidate is drawn over the tokens +// heavier than it. The accepted token is distributed as w restricted to the nucleus. The first candidate lands in the +// nucleus with probability >= top_p, so most rows decide on their second exchange. +// +// Per verify row, 8 CTAs (rows <= 32) or 4 CTAs (more rows) on every rank each own a slice of the rank's logits shard, +// staged in shared memory once. A candidate is drawn in two stages without per-token noise: each CTA draws a local +// candidate by inverse CDF over its slice, and the CTA whose log-mass plus Gumbel noise is largest supplies the +// candidate (CTA j wins with probability T_j / sum T, its token with w / T_j). Uniforms come from a counter-based +// Philox keyed by the request's seed and offset, the row, the attempt and the CTA. Each round every CTA multicasts one +// 4-word record (key, candidate logit, and max and mass in round 0 or the mass above the current candidate later) and +// reads the records of all CTAs of the row on all ranks. Every CTA takes the same decision from the same records in the +// same order, so all ranks agree without any intra-GPU handshake. Words carry the call's tag: a slot needs no clearing +// and two call halves keep a rank one call ahead from overwriting a slot still being read. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +namespace sampledAccept +{ +namespace +{ +constexpr int kThreads = 256; +constexpr int kWarps = kThreads / 32; +constexpr int kMaxRec = kMaxWorld * kMaxSplits; +constexpr int kMaxAttempts = 16; // exchange slots per row, reused cyclically by the rounds (no round cap) +constexpr int kEx = 4; // words per record: key, logit of key, f2, f3 +constexpr int kTagBits = 64 - 32 - kIdBits; +constexpr float kTempEpsilon = 1e-6f; // the fused sampling kernel's 1/T guard +// Record polls trap after this many spins (the multicast all-reduce's bound): a missing writer becomes a CUDA error, +// never a hang. +constexpr long long kMaxSpins = 1LL << 31; + +// Multicast exchange, per call half: [kMaxAttempts][kMaxRows][world * kMaxSplits][kEx] u64. +__host__ __device__ inline long long half_words(int world) +{ + return static_cast(kMaxAttempts) * kMaxRows * world * kMaxSplits * kEx; +} + +__device__ __forceinline__ uint32_t ordered(float v) +{ + uint32_t const b = __float_as_uint(v); + return (b & 0x80000000u) ? ~b : (b | 0x80000000u); +} + +// Philox4x32-10. +__device__ __forceinline__ uint4 philox(uint4 c, uint2 k) +{ +#pragma unroll + for (int i = 0; i < 10; ++i) + { + uint32_t const hi0 = __umulhi(0xD2511F53u, c.x), lo0 = 0xD2511F53u * c.x; + uint32_t const hi1 = __umulhi(0xCD9E8D57u, c.z), lo1 = 0xCD9E8D57u * c.z; + c = make_uint4(hi1 ^ c.y ^ k.x, lo1, hi0 ^ c.w ^ k.y, lo0); + k.x += 0x9E3779B9u; + k.y += 0xBB67AE85u; + } + return c; +} + +// u in (0, 1): (k + 1/2) / 2^23, exact in fp32. +__device__ __forceinline__ float uniform(uint32_t bits) +{ + return (static_cast(bits >> 9) + 0.5f) * (1.0f / 8388608.0f); +} + +// Key on the wire: perturbed log-mass, then the candidate token id; 0 = no candidate. Tag bits are added on the wire. +__device__ __forceinline__ unsigned long long gkey(float lg, int id) +{ + return (static_cast(ordered(lg)) << kIdBits) + | static_cast((1u << kIdBits) - 1u - static_cast(id)); +} + +__device__ __forceinline__ int key_token(unsigned long long k) +{ + return static_cast((1u << kIdBits) - 1u - static_cast(k & ((1u << kIdBits) - 1u))); +} + +__device__ __forceinline__ void st_sys(unsigned long long* p, unsigned long long v) +{ + asm volatile("st.global.relaxed.sys.u64 [%0], %1;" ::"l"(p), "l"(v) : "memory"); +} + +__device__ __forceinline__ unsigned long long ld_sys(unsigned long long const* p) +{ + unsigned long long v; + asm volatile("ld.global.relaxed.sys.u64 %0, [%1];" : "=l"(v) : "l"(p) : "memory"); + return v; +} + +template +struct Shared +{ + __nv_bfloat16 row[kMaxVs / kSplits]; + float redf[kWarps]; + unsigned long long redk[kWarps]; + unsigned long long rk[kMaxRec]; + float rv[kMaxRec], r2[kMaxRec], r3[kMaxRec]; + float wsum[kWarps]; + int owner; + int pick; + float pickx; + bool last; +}; + +template +__device__ __forceinline__ float block_max(float v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, o)); + if ((threadIdx.x & 31) == 0) + s.redf[threadIdx.x >> 5] = v; + __syncthreads(); + float r = s.redf[0]; + for (int i = 1; i < kWarps; ++i) + r = fmaxf(r, s.redf[i]); + __syncthreads(); + return r; +} + +// Fixed-order block sum: identical result for identical inputs. +template +__device__ __forceinline__ float block_sum(float v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + if ((threadIdx.x & 31) == 0) + s.redf[threadIdx.x >> 5] = v; + __syncthreads(); + float r = 0.f; + for (int i = 0; i < kWarps; ++i) + r += s.redf[i]; + __syncthreads(); + return r; +} + +template +__device__ __forceinline__ unsigned long long block_max_u64(unsigned long long v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + unsigned long long const w = __shfl_xor_sync(0xffffffffu, v, o); + v = w > v ? w : v; + } + if ((threadIdx.x & 31) == 0) + s.redk[threadIdx.x >> 5] = v; + __syncthreads(); + unsigned long long r = s.redk[0]; + for (int i = 1; i < kWarps; ++i) + r = s.redk[i] > r ? s.redk[i] : r; + __syncthreads(); + return r; +} + +// Exclusive prefix of v over the threads in thread order, and the total. +template +__device__ __forceinline__ float block_excl_scan(float v, float& total, S& s) +{ + int const lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + float inc = v; +#pragma unroll + for (int o = 1; o < 32; o <<= 1) + { + float const t = __shfl_up_sync(0xffffffffu, inc, o); + if (lane >= o) + inc += t; + } + if (lane == 31) + s.wsum[warp] = inc; + __syncthreads(); + float base = 0.f; + total = 0.f; + for (int i = 0; i < kWarps; ++i) + { + base += i < warp ? s.wsum[i] : 0.f; + total += s.wsum[i]; + } + __syncthreads(); + return base + inc - v; +} + +// A row's kSplits CTAs wait for each other's records, so they launch as one thread-block cluster: the hardware +// schedules a cluster's CTAs together, whatever other streams hold. Rows never wait on other rows. +template +__global__ void __launch_bounds__(kThreads, 2) sampled_accept_kernel(Args const a) +{ + extern __shared__ __align__(16) unsigned char smem[]; + Shared& s = *reinterpret_cast*>(smem); + float constexpr kNegInf = -__builtin_huge_valf(); + int const tid = threadIdx.x; + int const row = blockIdx.x / kSplits; + int const split = blockIdx.x % kSplits; + int const world = a.world; + int const nrec = world * kSplits; + // The row's sampling parameters are step inputs (the spec metadata, written before the target forward), not the + // predecessor head GEMM's output, so with prewait_params they load before the dependency wait and overlap the + // head's tail; the same values either way. + float temperature = 0.f, topP = 0.f; + long long seed_in = 0, offset_in = 0; + auto load_params = [&]() + { + temperature = a.temperatures[row]; + topP = a.top_ps[row]; + seed_in = a.seeds[row]; + offset_in = a.offsets[row]; + }; + if (a.prewait_params) + load_params(); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + if (!a.prewait_params) + load_params(); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + // early_trigger: the dependent grid (the first draft pass's input) may launch now and park at its own + // griddepcontrol.wait, which still returns only once this grid has completed and flushed; it skips the launch + // latency after this kernel ends. Otherwise dependents are released at the acceptance epilogue (the CTAs of + // splits 1.. exit without releasing). + if (a.early_trigger) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float const tempInv = 1.0f / (temperature + kTempEpsilon); + unsigned long long const seed = static_cast(seed_in); + unsigned long long const offset = static_cast(offset_in); + unsigned int const call = *reinterpret_cast(a.counters + 1); + // Wire tags: never 0 (fresh memory), different between calls of the same half and between consecutive laps of + // the slot ring (round r uses slot r % kMaxAttempts and lap parity (r / kMaxAttempts) & 1). A rank decides round + // r only after reading every record of round r, so no rank is more than one round ahead and a slot is rewritten + // only after every reader has consumed it; the lap bit tells the new record from the previous lap's. + auto ktag_of = [&](int r) -> unsigned long long + { + unsigned int const lap = static_cast(r / kMaxAttempts) & 1u; + return (((call >> 1) << 1) | lap) % ((1u << kTagBits) - 1u) + 1u; + }; + auto ftag_of = [&](int r) -> unsigned long long + { + unsigned int const lap = static_cast(r / kMaxAttempts) & 1u; + return static_cast(((((call >> 1) << 1) | lap) & 0x7FFFFFFFu) | 0x80000000u) << 32; + }; + long long const rec_stride = static_cast(world) * kMaxSplits * kEx; + unsigned long long* const xu + = a.ws_uc + static_cast(call & 1u) * half_words(world) + static_cast(row) * rec_stride; + unsigned long long* const xm = a.ws_mc + static_cast(call & 1u) * half_words(world) + + static_cast(row) * rec_stride + (a.rank * kMaxSplits + split) * kEx; + long long const round_stride = static_cast(kMaxRows) * rec_stride; + + // ---- stage this CTA's slice and its max ------------------------------------------------------------------------- + __nv_bfloat16 const* src = a.logits + row * a.ld; + int const sl_len = ((a.Vs + kSplits - 1) / kSplits + 7) / 8 * 8; + int const s0 = min(split * sl_len, a.Vs); + int const n = min(s0 + sl_len, a.Vs) - s0; + int const id0 = a.vocab_offset + s0; + float tmax = kNegInf; + bool const vec = (a.ld % 8) == 0 && (reinterpret_cast(a.logits) & 15) == 0; + int const nvec = vec ? n / 8 : 0; + for (int v = tid; v < nvec; v += kThreads) + { + uint4 const w = *reinterpret_cast(src + s0 + 8 * v); + *reinterpret_cast(&s.row[8 * v]) = w; + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&w); +#pragma unroll + for (int j = 0; j < 8; ++j) + tmax = fmaxf(tmax, __fmul_rn(__bfloat162float(e[j]), tempInv)); + } + for (int i = nvec * 8 + tid; i < n; i += kThreads) + { + s.row[i] = src[s0 + i]; + tmax = fmaxf(tmax, __fmul_rn(__bfloat162float(src[s0 + i]), tempInv)); + } + float const smax = block_max(tmax, s); // the staged slice is visible to all threads after this barrier + float M = 0.f, wc = 0.f, target = 0.f, cv = 0.f; // global max, candidate weight and logit, top_p * Z + unsigned long long ckey = 0ULL; + int const gid = a.rank * kMaxSplits + split; + for (int r = 0;; ++r) + { + // ---- local candidate by inverse CDF over the slice's tokens heavier than the candidate (w > 0 in round 0) + float const ref = r == 0 ? smax : M; + float const wmin = r == 0 ? 0.f : wc; + float ts = 0.f; + for (int i = tid; i < n; i += kThreads) + { + float const w = __expf(__fmul_rn(__bfloat162float(s.row[i]), tempInv) - ref); + ts += w > wmin ? w : 0.f; + } + float total; + float const excl = block_excl_scan(ts, total, s); + uint4 const rnd = philox(make_uint4(static_cast(gid), static_cast(row + 64 * r), + static_cast(offset), static_cast(offset >> 32)), + make_uint2(static_cast(seed), static_cast(seed >> 32))); + float const tgt = uniform(rnd.x) * total; + if (tid == 0) + s.owner = -1; + __syncthreads(); + if (ts > 0.f && excl <= tgt) + atomicMax(&s.owner, tid); + __syncthreads(); + if (tid == s.owner) + { + float acc = excl; + int pick = -1; + float px = 0.f; + if constexpr (kIlpWalk) + { + // SA-PW: the owner's strided tokens 8 at a time. The 8 x and w (the same expressions) are + // independent, so their MUFU work overlaps; the serial sum then adds them in the same order with + // the same early exit: the same acc sequence, pick and px as the one-token walk, bit for bit. + constexpr int kW = 8; + bool done = false; + for (int i0 = tid; i0 < n && !done; i0 += kW * kThreads) + { + float xs[kW], wv[kW]; +#pragma unroll + for (int k = 0; k < kW; ++k) + { + int const i = i0 + k * kThreads; + xs[k] = i < n ? __fmul_rn(__bfloat162float(s.row[i]), tempInv) : 0.f; + wv[k] = __expf(xs[k] - ref); + } +#pragma unroll + for (int k = 0; k < kW; ++k) + { + int const i = i0 + k * kThreads; + if (!done && i < n && wv[k] > wmin) + { + acc += wv[k]; + pick = i; + px = xs[k]; + if (acc > tgt) + done = true; + } + } + } + } + else + { + for (int i = tid; i < n; i += kThreads) + { + float const x = __fmul_rn(__bfloat162float(s.row[i]), tempInv); + float const w = __expf(x - ref); + if (w > wmin) + { + acc += w; + pick = i; + px = x; + if (acc > tgt) + break; + } + } + } + s.pick = pick; + s.pickx = px; + } + __syncthreads(); + unsigned long long sb = 0ULL; + if (total > 0.f && s.owner >= 0) + sb = gkey(__logf(total) + ref - logf(-logf(uniform(rnd.y))), id0 + s.pick); + float const smab = total; + unsigned long long const ktag = ktag_of(r); + unsigned long long const ftag = ftag_of(r); + unsigned long long* const xr = xu + (r % kMaxAttempts) * round_stride; + if (tid == 0) + { + unsigned long long* const xw = xm + (r % kMaxAttempts) * round_stride; + st_sys(xw + 0, (sb << kTagBits) | ktag); + st_sys(xw + 1, ftag | __float_as_uint(sb != 0ULL ? s.pickx : 0.f)); + st_sys(xw + 2, ftag | __float_as_uint(r == 0 ? smax : smab)); + st_sys(xw + 3, ftag | __float_as_uint(r == 0 ? smab : 0.f)); + } + // ---- every record of the row, every rank --------------------------------------------------------------- + if (tid < nrec * kEx) + { + int const q = tid / kEx, f = tid % kEx; + unsigned long long v; + if (f == 0) + { + long long spins = 0; + do + { + v = ld_sys(xr + (q / kSplits * kMaxSplits + q % kSplits) * kEx); + if (++spins > kMaxSpins) + __trap(); + } while ((v & ((1ULL << kTagBits) - 1ULL)) != ktag); + s.rk[q] = v >> kTagBits; + } + else + { + long long spins = 0; + do + { + v = ld_sys(xr + (q / kSplits * kMaxSplits + q % kSplits) * kEx + f); + if (++spins > kMaxSpins) + __trap(); + } while ((v & 0xFFFFFFFF00000000ULL) != ftag); + float const fv = __uint_as_float(static_cast(v)); + (f == 1 ? s.rv : f == 2 ? s.r2 : s.r3)[q] = fv; + } + } + __syncthreads(); + // ---- the decision, identical in every CTA of the row on every rank: fixed record order ----------------- + unsigned long long gk = 0ULL; + float gv = 0.f; + for (int q = 0; q < nrec; ++q) + { + if (s.rk[q] > gk) + { + gk = s.rk[q]; + gv = s.rv[q]; + } + } + bool accept; + if (r == 0) + { + M = kNegInf; + for (int q = 0; q < nrec; ++q) + M = fmaxf(M, s.r2[q]); + float Z = 0.f; + for (int q = 0; q < nrec; ++q) + Z += s.r2[q] == kNegInf ? 0.f : s.r3[q] * __expf(s.r2[q] - M); + target = topP * Z; + accept = !(topP < 1.0f); // no nucleus cut: the first draw stands + ckey = gk; + cv = gv; + } + else + { + float S = 0.f; // mass strictly above the current candidate + for (int q = 0; q < nrec; ++q) + S += s.r2[q]; + accept = S <= target || gk == 0ULL; + if (!accept) + { + ckey = gk; + cv = gv; + } + } + wc = __expf(cv - M); + __syncthreads(); // records consumed before the next round's writes to s + if (accept) + { + if (split != 0) + return; + if (tid == 0) + { + a.tokens[row] = key_token(ckey); + if (a.extra_attempts != nullptr && r > 1) + atomicAdd(a.extra_attempts, r - 1); + } + break; + } + } + + // ---- split 0: the last row to finish runs the strict acceptance ---------------------------------------------- +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (!a.early_trigger) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + __threadfence(); + __syncthreads(); + if (tid == 0) + s.last = atomicAdd(a.counters, 1u) == static_cast(a.rows) - 1; + __syncthreads(); + if (!s.last) + return; + __threadfence(); + if (tid == 0) + { + a.counters[0] = 0u; + a.counters[1] = call + 1u; + } + if (tid < a.nreq) + { + int const* tok = a.tokens; + int const b = tid; + int const k1 = a.draft_len + 1; + int nacc = 1; + bool run = true; + for (int k = 0; k < a.draft_len; ++k) + { + int const tt = *reinterpret_cast(tok + b * k1 + k); + a.acc_tokens[b * k1 + k] = tt; + run = run && a.draft[b * a.draft_len + k] == tt; + nacc += run ? 1 : 0; + } + a.acc_tokens[b * k1 + a.draft_len] = *reinterpret_cast(tok + b * k1 + a.draft_len); + if (a.force_vec != nullptr) + nacc = a.force_vec[b]; + else if (a.force_total > 0) + nacc = a.force_total; + a.num_acc[b] = nacc; + a.gather_ids[b] = static_cast(a.batch_idx[b]) * k1 + nacc - 1; + if (a.next_new != nullptr) + a.next_new[b * a.next_stride] + = *reinterpret_cast(tok + a.batch_idx[b] * k1 + nacc - 1); + } +} + +template +cudaError_t launch(Args const& a, bool use_pdl, cudaStream_t stream) +{ + static int max_clusters = 0; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(a.rows * kSplits); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = sizeof(Shared); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kSplits; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + if (max_clusters == 0) + { + cudaError_t err = cudaFuncSetAttribute(sampled_accept_kernel, + cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(sizeof(Shared))); + if (err == cudaSuccess) + { + err = cudaOccupancyMaxActiveClusters(&max_clusters, sampled_accept_kernel, &cfg); + } + if (err != cudaSuccess) + { + return err; + } + } + TLLM_CHECK_WITH_INFO(max_clusters >= 1, "m3_sampled_accept: a %d-CTA cluster does not fit on one GPC", kSplits); + return cudaLaunchKernelEx(&cfg, sampled_accept_kernel, a); +} + +} // namespace + +long long sampledAcceptWsWords(int world) +{ + return 2 * half_words(world); +} + +cudaError_t invokeSampledAccept(Args const& a, bool usePdl, cudaStream_t stream) +{ + // TP4 caps 2 / 4 / 8 (8 / 16 / 32 rows) and TP8 cap 1 (4 rows): 8 CTAs per row, at most 256 CTAs (2 per SM); + // more rows: 4 CTAs per row. + if (a.rows <= 32) + { + return a.ilp_walk ? launch<8, true>(a, usePdl, stream) : launch<8, false>(a, usePdl, stream); + } + return a.ilp_walk ? launch<4, true>(a, usePdl, stream) : launch<4, false>(a, usePdl, stream); +} + +} // namespace sampledAccept +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.h new file mode 100644 index 000000000000..cd1e8ec7fefa --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.h @@ -0,0 +1,82 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ +//! Fused sampled (temperature + top-p) verification of the MiniMax-M3 Eagle3 target head over the TP-sharded +//! vocabulary, decode rows <= kMaxRows; see sampledAccept.cu. +namespace sampledAccept +{ + +constexpr int kMaxSplits = 8; // CTAs per verify row (at most) +constexpr int kMaxRows = 48; // verify rows per call +constexpr int kMaxWorld = 8; // TP ranks +constexpr int kMaxVs = 100352; // vocab shard width (TP2: 200064 / 2 = 100032, padded to a multiple of 512) +constexpr int kIdBits = 18; // token ids < 2^18 + +struct Args +{ + __nv_bfloat16 const* logits; // [rows, Vs] this rank's shard + long long ld; + int rows, Vs, vocab_offset; + float const* temperatures; + float const* top_ps; + long long const* seeds; + long long const* offsets; + unsigned long long* ws_uc; + unsigned long long* ws_mc; + unsigned int* counters; // [0] finished rows, [1] call + int* tokens; // [rows] sampled token per row + int* extra_attempts; // optional [1]: rounds beyond the second, summed over rows + int rank, world; + int const* draft; + int const* batch_idx; + int* acc_tokens; + int* num_acc; + long long* gather_ids; + int draft_len, nreq, force_total; + int* next_new; + int next_stride; + int const* force_vec; + int prewait_params; // nonzero: the row's sampling parameters load before the PDL dependency wait + int early_trigger; // nonzero: dependents are released right after the dependency wait, not at the epilogue + int ilp_walk; // nonzero: the owner walk takes its tokens 8 at a time (same weights, same summation order) +}; + +//! Words (uint64) of the multicast exchange workspace for `world` ranks: two call halves. +long long sampledAcceptWsWords(int world); + +//! Launch the sampled verification over a.rows * (8 or 4) co-resident CTAs (8 per row up to 32 rows). +cudaError_t invokeSampledAccept(Args const& a, bool usePdl, cudaStream_t stream); + +} // namespace sampledAccept +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.cu new file mode 100644 index 000000000000..e708083ef9ff --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.cu @@ -0,0 +1,246 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Top-16 block select for MiniMax-M3 decode rows of 129 to 9216 blocks. +// +// One 1024-thread CTA per (query, KV head) row holds the row's ordered score keys in registers (2, 4 or 9 per +// thread, by the row's valid count). The weakest of the 16 maxima of the interleaved groups `block % 16` is a lower +// bound on the 16th-largest key, so the top 16 is among the keys at or above it; those are compacted to shared +// memory and ranked exactly by (key, block). More than 1024 keys tied at the bound (degenerate rows) rank every +// candidate against the whole row (re-read). + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.h" + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +namespace +{ + +constexpr float kInitScore = 1.0e30f; +constexpr float kLocalScore = 1.0e29f; +constexpr uint32_t kNegInfKey = 0xff800000u; +constexpr int kItems = 9; // keys per thread: rows up to 9216 blocks (1M tokens + draft tokens) + +// Descending order key: a smaller key is a larger score (NaN above +inf), as in the reference op. +__device__ __forceinline__ uint32_t orderedKey(float x) +{ + uint32_t const b = __float_as_uint(x); + return (b & 0x80000000u) ? b : (~b & 0x7fffffffu); +} + +__device__ __forceinline__ float applyForcing(float raw, int block, int initBlocks, long long localStart) +{ + if (block >= localStart) + return kLocalScore; + if (block < initBlocks) + return kInitScore; + return raw; +} + +struct SelectSmem +{ + uint32_t groupMin[16]; + unsigned long long cand[1024]; + int numCand; + unsigned long long picked[16]; +}; + +// One row with up to kItems * 1024 valid blocks; the caller picks kItems from the row's valid count. +template +__device__ __forceinline__ void selectRow(SelectSmem& sm, float const* rowBase, long long blockStride, int valid, + int initBlocks, long long localStart, bool loadedTrigger) +{ + int const tid = threadIdx.x, lane = tid & 31; + uint32_t key[kItems]; + uint32_t localMin = 0xffffffffu; +#pragma unroll + for (int s = 0; s < kItems; ++s) + { + int const block = s * 1024 + tid; + key[s] = block < valid ? orderedKey(applyForcing(rowBase[block * blockStride], block, initBlocks, localStart)) + : 0xffffffffu; + localMin = min(localMin, key[s]); + } + if (valid <= 16) + { + if (tid < valid) + sm.picked[tid] = (static_cast(key[0]) << 32) | static_cast(tid); + if (loadedTrigger) + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + return; + } + // Group g = block % 16 = tid % 16: lanes l and l ^ 16 of every warp. + localMin = min(localMin, __shfl_xor_sync(0xffffffffu, localMin, 16)); + if (lane < 16) + atomicMin(&sm.groupMin[lane], localMin); + __syncthreads(); + // Every score of the row is loaded: release the sparse decode, so its launch and its pre-wait prologue overlap + // the rank below. This comes after this grid's own dependency wait, which the sparse decode relies on: it reads + // q and the seq_lens / block-table row before its own wait (sparseDecodeLc2.cu), safe only while every kernel + // between the producer and it releases its dependents after its own wait. + if (loadedTrigger) + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + uint32_t bound = 0; +#pragma unroll + for (int g = 0; g < 16; ++g) + bound = max(bound, sm.groupMin[g]); +#pragma unroll + for (int s = 0; s < kItems; ++s) + { + int const block = s * 1024 + tid; + bool const isCand = block < valid && key[s] <= bound; + unsigned const ballot = __ballot_sync(0xffffffffu, isCand); + int base = 0; + if (lane == 0 && ballot) + base = atomicAdd(&sm.numCand, __popc(ballot)); + base = __shfl_sync(0xffffffffu, base, 0); + if (isCand) + { + int const i = base + __popc(ballot & ((1u << lane) - 1)); + if (i < 1024) + sm.cand[i] = (static_cast(key[s]) << 32) | static_cast(block); + } + } + __syncthreads(); + int const n = sm.numCand; + if (n <= 1024) + { + for (int i = tid; i < n; i += 1024) + { + unsigned long long const mine = sm.cand[i]; + int rank = 0; + for (int j = 0; j < n; ++j) + rank += sm.cand[j] < mine; + if (rank < 16) + sm.picked[rank] = mine; + } + return; + } + // More than 1024 keys tied at the bound (degenerate rows): rank every candidate against the whole row. +#pragma unroll + for (int s = 0; s < kItems; ++s) + { + int const block = s * 1024 + tid; + if (block < valid && key[s] <= bound) + { + int rank = 0; + for (int j = 0; j < valid; ++j) + { + uint32_t const other = orderedKey(applyForcing(rowBase[j * blockStride], j, initBlocks, localStart)); + rank += other < key[s] || (other == key[s] && j < block); + } + if (rank < 16) + sm.picked[rank] = (static_cast(key[s]) << 32) | static_cast(block); + } + } +} + +__global__ void __launch_bounds__(1024) m3_select_long2_kernel(float const* __restrict__ scores, long long headStride, + long long blockStride, long long queryStride, int const* __restrict__ nValid, int* __restrict__ out, + long long outQueryStride, long long outHeadStride, int numKvHeads, int numBlocks, int initBlocks, int localBlocks, + bool earlyTrigger, bool loadedTrigger) +{ + __shared__ SelectSmem sm; + int const row = blockIdx.x, query = row / numKvHeads, head = row % numKvHeads; + int const tid = threadIdx.x, lane = tid & 31; + if (tid < 16) + { + sm.groupMin[tid] = 0xffffffffu; + sm.picked[tid] = ~0ull; + } + if (tid == 0) + sm.numCand = 0; +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + cudaGridDependencySynchronize(); + // The sparse decode that follows stages its table-independent inputs before its own grid wait; past this grid's + // wait those inputs are final, so it may launch now rather than at this grid's exit. + if (earlyTrigger) + cudaTriggerProgrammaticLaunchCompletion(); +#endif + int const raw = nValid[query]; + int const valid = max(0, min(raw, numBlocks)); + long long const localStart = max(static_cast(raw) - localBlocks, 0LL); + float const* rowBase = scores + head * headStride + query * queryStride; + __syncthreads(); + if (valid <= 2048) + selectRow<2>(sm, rowBase, blockStride, valid, initBlocks, localStart, loadedTrigger); + else if (valid <= 4096) + selectRow<4>(sm, rowBase, blockStride, valid, initBlocks, localStart, loadedTrigger); + else + selectRow(sm, rowBase, blockStride, valid, initBlocks, localStart, loadedTrigger); + __syncthreads(); + if (tid >= 32) + return; + uint32_t blockKey = ~0u; + if (lane < 16) + { + unsigned long long const c = sm.picked[lane]; + if (c != ~0ull && static_cast(c >> 32) != kNegInfKey) + blockKey = static_cast(c & 0xffffffffu); + } +#pragma unroll + for (int size = 2; size <= 32; size <<= 1) + { +#pragma unroll + for (int stride = size >> 1; stride > 0; stride >>= 1) + { + uint32_t const other = __shfl_xor_sync(0xffffffffu, blockKey, stride); + bool const ascending = (lane & size) == 0, lower = (lane & stride) == 0; + blockKey = (lower == ascending) ? min(blockKey, other) : max(blockKey, other); + } + } + if (lane < 16) + out[query * outQueryStride + head * outHeadStride + lane] = blockKey == ~0u ? -1 : static_cast(blockKey); +} + +} // namespace + +cudaError_t invokeSelectLong2(float const* scores, int64_t headStride, int64_t blockStride, int64_t queryStride, + int32_t const* nValidBlocks, int32_t* out, int64_t outQueryStride, int64_t outHeadStride, int32_t numKvHeads, + int32_t numBlocks, int32_t totalQ, int32_t initBlocks, int32_t localBlocks, bool usePdl, bool earlyTrigger, + int32_t triggerPhase, cudaStream_t stream) +{ + if (numBlocks > kSelectLong2MaxBlocks || triggerPhase < 0 || triggerPhase > 1) + { + return cudaErrorInvalidValue; + } + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(totalQ * numKvHeads); + cfg.blockDim = dim3(1024); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = usePdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, m3_select_long2_kernel, scores, static_cast(headStride), + static_cast(blockStride), static_cast(queryStride), nValidBlocks, out, + static_cast(outQueryStride), static_cast(outHeadStride), numKvHeads, numBlocks, + initBlocks, localBlocks, earlyTrigger, !earlyTrigger && triggerPhase == 1); +} + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.h new file mode 100644 index 000000000000..9fba25f03c77 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.h @@ -0,0 +1,50 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +//! Largest row (blocks per query) the long-row top-16 select covers: 9 keys per thread of one 1024-thread CTA. +constexpr int32_t kSelectLong2MaxBlocks = 9216; + +//! MiniMax-M3 top-16 block select over per-KV-head scores [numKvHeads, numBlocks, totalQ] (element strides given), +//! one 1024-thread CTA per (query, KV head). Same contract as minimax_m3_select_blocks: init / local forcing, the +//! per-query valid-block count, ties to the smaller block, forced -inf picks emitted as -1, ascending ids with -1 +//! tail padding. Writes out[query * outQueryStride + head * outHeadStride + 0..15]. The SM107 TP2 copy of +//! minimaxM3/selectLong with a choice of where the dependent grid (the sparse decode) is released: at this grid's exit +//! (triggerPhase 0), right after every score of the row is loaded (1), or, with earlyTrigger, right after this grid's +//! dependency wait. Every release point follows this grid's own dependency wait. +cudaError_t invokeSelectLong2(float const* scores, int64_t headStride, int64_t blockStride, int64_t queryStride, + int32_t const* nValidBlocks, int32_t* out, int64_t outQueryStride, int64_t outHeadStride, int32_t numKvHeads, + int32_t numBlocks, int32_t totalQ, int32_t initBlocks, int32_t localBlocks, bool usePdl, bool earlyTrigger, + int32_t triggerPhase, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.cu new file mode 100644 index 000000000000..cb7d83b89bb4 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.cu @@ -0,0 +1,89 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +namespace +{ + +constexpr int kThreads = 128; + +// One packed bf16 pair of gate and up values -> its two bf16 results (the low half from the low halves). The +// instruction sequence of the Triton kernel's PTX at its defaults (host/silu/silu_and_mul_plain.ptx): g * -log2(e) as +// one product, ex2.approx, + 1, div.full 1 / that, g * s, (u + 0.0) as an IEEE add, the product, one rounding to bf16. +__device__ __forceinline__ uint32_t siluMulBf16x2(uint32_t gate, uint32_t up) +{ + uint32_t r; + asm("{\n\t.reg .b32 gl, gh, ul, uh, t0, t1, e0, e1, s0, s1, p0, p1, q0, q1, r0, r1, one, zero;\n\t" + ".reg .b16 hl, hh;\n\t" + "mov.b32 {hl, hh}, %1;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, %2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mul.rn.f32 t0, gl, 0fBFB8AA3B;\n\tmul.rn.f32 t1, gh, 0fBFB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 p0, gl, s0;\n\tmul.rn.f32 p1, gh, s1;\n\t" + "mov.b32 zero, 0f00000000;\n\tadd.rn.f32 q0, ul, zero;\n\tadd.rn.f32 q1, uh, zero;\n\t" + "mul.rn.f32 r0, q0, p0;\n\tmul.rn.f32 r1, q1, p1;\n\t" + "cvt.rn.bf16x2.f32 %0, r1, r0;\n\t}" + : "=r"(r) + : "r"(gate), "r"(up)); + return r; +} + +// Eight outputs per thread: 16-byte loads of the gate and the up half, one 16-byte store. +__global__ void __launch_bounds__(kThreads) + siluAndMulKernel(uint4 const* __restrict__ x, uint4* __restrict__ out, int64_t rows, int64_t vecs) +{ + int64_t const row = blockIdx.y; + int64_t const v = static_cast(blockIdx.x) * kThreads + threadIdx.x; + if (row >= rows || v >= vecs) + { + return; + } + uint4 const g = x[row * 2 * vecs + v]; + uint4 const u = x[row * 2 * vecs + vecs + v]; + uint4 o; + o.x = siluMulBf16x2(g.x, u.x); + o.y = siluMulBf16x2(g.y, u.y); + o.z = siluMulBf16x2(g.z, u.z); + o.w = siluMulBf16x2(g.w, u.w); + out[row * vecs + v] = o; +} + +} // namespace + +cudaError_t invokeSiluAndMul(uint16_t const* x, uint16_t* out, int64_t rows, int64_t d, cudaStream_t stream) +{ + int64_t const vecs = d / 8; + dim3 const grid(static_cast((vecs + kThreads - 1) / kThreads), static_cast(rows)); + siluAndMulKernel<<>>( + reinterpret_cast(x), reinterpret_cast(out), rows, vecs); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.h new file mode 100644 index 000000000000..01b32137ac4a --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.h @@ -0,0 +1,40 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +//! out[m, j] = bf16(silu(g) * (u + 0)) with g = x[m, j], u = x[m, d + j] (x bf16 [rows, 2d], out bf16 [rows, d], both +//! contiguous, d a multiple of 8, 16-byte aligned rows): bit for bit the Triton silu_and_mul kernel at its defaults +//! (no limit, alpha 1, beta 0), the plain SwiGLU of the Eagle3 drafter MLP. +cudaError_t invokeSiluAndMul(uint16_t const* x, uint16_t* out, int64_t rows, int64_t d, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.cu new file mode 100644 index 000000000000..d96f51d5d150 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.cu @@ -0,0 +1,640 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3 long-context sparse decode attention in one launch, SM107 TP2 geometry (32 query heads on 2 KV heads +// per rank; staircase copy of minimaxM3/sparseDecodeLc.cu with a KV-head grid dimension). +// +// Grid: (tokens * 16, kv heads), one cluster of 16 CTAs per (query token, KV head); CTA `r` attends the token's `r`-th +// selected block (the Triton producer's chunk `r` at 16 chunks), keeps the partial in shared memory, and after a +// cluster barrier merges head `r` from all 16 partials over DSMEM, writing the BF16 row and, optionally, its MXFP8 copy +// and swizzled UE8M0 scales exactly as the Triton merge's `_store_mxfp8_head` lays them out. +// +// Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and +// PV on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +// log-domain recurrence. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kHeads = 16; +constexpr int kDim = 128; +constexpr int kPage = 128; +constexpr int kTopk = 16; +constexpr int kThreads = 128; +constexpr int kPitch = kDim + 8; // halves per smem row: 272 B, conflict-free ldmatrix +constexpr float kNegInf = -__builtin_huge_valf(); +constexpr int kBtCap = 4096; // block-table entries staged in smem by PREWAIT (512K tokens) + +using Params = MinimaxM3Sm107Tp2SparseDecodeLcParams; + +struct Smem +{ + __half qs[kHeads * kPitch]; + __half vs[kPage * kPitch]; + __half ps[kHeads * kPitch]; + float recv[kTopk][kDim]; // head `rank` partials pushed by the 16 CTAs, chunk-major + float recv_lse[kTopk]; + float wgt[kTopk]; // the 16 merge weights, computed once per CTA + float red[4][kHeads]; + float rowm[kHeads]; + float rowl[kHeads]; + alignas(8) unsigned long long rbar; // the split-K partials of head `rank` landed (complete_tx) +}; + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t e2m1x2_to_f16x2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 s;\n cvt.u8.u32 s, %1;\n cvt.rn.f16x2.e2m1x2 %0, s;\n}" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t e4m3x2_to_f16x2(uint16_t pair) +{ + uint32_t out; + asm("cvt.rn.f16x2.e4m3x2 %0, %1;" : "=r"(out) : "h"(pair)); + return out; +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ void ldsm_x4(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x4_t(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma(float (&c)[4], uint32_t const (&a)[4], uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};" + : "+f"(c[0]), "+f"(c[1]), "+f"(c[2]), "+f"(c[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} + +__device__ __forceinline__ uint32_t map_cluster(void const* local, uint32_t rank) +{ + uint32_t remote; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(remote) : "r"(smem_u32(local)), "r"(rank)); + return remote; +} + +// Posted DSMEM stores that complete_tx on the receiving CTA's mbarrier `bar`. +__device__ __forceinline__ void st_async_v2(uint32_t addr, float a, float b, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.v2.f32 [%0], {%1, %2}, [%3];" ::"r"(addr), + "f"(a), "f"(b), "r"(bar) + : "memory"); +} + +__device__ __forceinline__ void st_async_f32(uint32_t addr, float a, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(addr), + "r"(__float_as_uint(a)), "r"(bar) + : "memory"); +} + +// One packed NVFP4 row (64 bytes, 128 values) with its 8 E4M3 scales -> 128 fp16 values in smem. +__device__ __forceinline__ void dequant_row(__half* dst, uint8_t const* packed, uint8_t const (&sc)[8], bool valid) +{ + uint4 const* src = reinterpret_cast(packed); + uint32_t* d = reinterpret_cast(dst); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + uint32_t words[4] = {w.x, w.y, w.z, w.w}; + // 16 bytes = 32 values = scale columns 2c and 2c + 1. +#pragma unroll + for (int h = 0; h < 2; ++h) + { + uint8_t const s = valid ? sc[2 * c + h] : 0; + uint32_t const sh = e4m3x2_to_f16x2(static_cast(s | (s << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); +#pragma unroll + for (int j = 0; j < 2; ++j) + { + uint32_t const word = words[2 * h + j]; +#pragma unroll + for (int b = 0; b < 4; ++b) + { + uint32_t const v2 = e2m1x2_to_f16x2((word >> (8 * b)) & 0xFF); + __half2 const x = __hmul2(*reinterpret_cast<__half2 const*>(&v2), scale); + d[c * 16 + h * 8 + j * 4 + b] = *reinterpret_cast(&x); + } + } + } + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +template +constexpr size_t smemBytes() +{ + return PREWAIT ? sizeof(Smem) + kBtCap * sizeof(int) : sizeof(Smem); +} + +template +__global__ void __launch_bounds__(kThreads) m3_lc_attn_kernel(__grid_constant__ Params const p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(128) uint8_t smem_raw[]; + Smem& s = *reinterpret_cast(smem_raw); + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const tok = blockIdx.x / kTopk; + int const kvh = blockIdx.y; + uint32_t rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(rank)); + + if (tid == 0) + { + // One arrival (here) plus every cluster CTA's partial row (kDim floats) and LSE for head `rank`. + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(smem_u32(&s.rbar)) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(smem_u32(&s.rbar)), + "r"(kTopk * (kDim + 1) * 4) + : "memory"); + } + // Every peer must be running before its shared memory takes a DSMEM store: arrive now, wait before pushing. + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + + // Pre-wait read contract: everything that does not depend on the top-16 table is read before the grid dependency. + // - seq_lens: written inside the step's graph by msa_kv_update (the KV-length correction). That kernel is a plain + // launch, not a programmatic dependent, so it has completed before any later kernel of the step starts. + // - the query rows: written by the producer, two grids back. This holds only while every kernel between the + // producer and this grid (the long-row select) triggers its dependents after its own griddepcontrol.wait, so + // this grid cannot start before the producer completed. A kernel inserted into that chain, or a select that + // triggers before its wait, breaks it: these reads then move after the griddepcontrol.wait below. + int const req = tok / p.dql; + int const qoff = tok - req * p.dql; + int const seq = p.seq_lens[req]; + int const kv_len = max(seq - p.dql + qoff + 1, 0); + int const nblocks = (kv_len + kPage - 1) / kPage; + int const real_topk = min(kTopk, nblocks); + // q: 16 heads x 128 e4m3 -> fp16; thread = (head, 16-value chunk). + { + int const h = tid >> 3; + int const c = (tid & 7) * 16; + uint4 const w = *reinterpret_cast(p.q + tok * p.q_tok + (kvh * kHeads + h) * p.q_head + c); + uint32_t const words[4] = {w.x, w.y, w.z, w.w}; + uint32_t* d = reinterpret_cast(s.qs + h * kPitch + c); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + d[2 * i] = e4m3x2_to_f16x2(static_cast(words[i] & 0xFFFF)); + d[2 * i + 1] = e4m3x2_to_f16x2(static_cast(words[i] >> 16)); + } + } + + // PREWAIT: the request's block-table row (written before the graph runs) is staged before the grid dependency. + int* const s_bt = reinterpret_cast(smem_raw + sizeof(Smem)); + bool bt_smem = false; + if constexpr (PREWAIT) + { + int const nbt = (max(seq, 0) + kPage - 1) / kPage; + bt_smem = nbt <= kBtCap; + if (bt_smem) + { + for (int i = tid; i < nbt; i += kThreads) + s_bt[i] = __ldg(p.block_table + req * p.bt_stride + i); + } + } + // N1: the block this (KV head, row, slot) selected at the layer's previous decode step recurs ~2/3 of the time; + // pull its K/V page and scales into L2 while the select runs. The ids were written by this kernel one step back + // (complete before the graph's next replay) and the block-table row before the graph runs. Read and written by + // the same thread, so the record below cannot overtake this read. Out-of-range ids (stale rows) are skipped. + bool const n1 = p.n1_ids != nullptr && tok < p.n1_rows; + int* const n1_slot = n1 ? p.n1_ids + kvh * p.n1_h + tok * kTopk + rank : nullptr; + if (n1 && tid == 64) + { + int const nb = *n1_slot; + if (nb >= 0 && nb < nblocks) + { + int64_t const pg = __ldg(p.block_table + req * p.bt_stride + nb); + if (pg >= 0 && pg < p.n1_pages) + { + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.k + pg * p.k_blk + kvh * p.k_h), + "r"(kPage * 64) + : "memory"); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.v + pg * p.v_blk + kvh * p.v_h), + "r"(kPage * 64) + : "memory"); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.ksc + pg * p.ksc_blk + kvh * p.ksc_h), + "r"(kPage * 8) + : "memory"); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.vsc + pg * p.vsc_blk + kvh * p.vsc_h), + "r"(kPage * 8) + : "memory"); + } + } + } + asm volatile("griddepcontrol.wait;" ::: "memory"); + if constexpr (EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } + int blk = static_cast(rank) < real_topk ? p.topk[kvh * p.t_h + tok * p.t_tok + rank * p.t_k] : -1; + if (n1 && tid == 64) + { + *n1_slot = blk; + } + bool const active = blk >= 0; + if constexpr (PREWAIT) + { + __syncthreads(); // s_bt complete + } + int64_t const page_pw = !active ? 0 : (bt_smem ? s_bt[blk] : p.block_table[req * p.bt_stride + blk]); + + float m_row[2] = {kNegInf, kNegInf}; + float acc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + acc[j][i] = 0.F; + float l_row[2] = {0.F, 0.F}; + + // K stays packed in registers: the MMA B fragment of token n = lane / 4 at k-step s is the f16x2 pair of + // packed bytes 8s + q and 8s + 4 + q (q = lane % 4), both under scale column s. + uint32_t kw[4][16]; + uint2 ksc[4]; + int64_t page = 0; + if (active) + { + page = page_pw; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const n = warp * 32 + j * 8 + (lane >> 2); + bool const valid = blk * kPage + n < kv_len; + uint4 const* src = reinterpret_cast(p.k + page * p.k_blk + kvh * p.k_h + n * 64); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 const w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + kw[j][4 * c] = w.x; + kw[j][4 * c + 1] = w.y; + kw[j][4 * c + 2] = w.z; + kw[j][4 * c + 3] = w.w; + } + ksc[j] = valid ? __ldg(reinterpret_cast(p.ksc + page * p.ksc_blk + kvh * p.ksc_h + n * 8)) + : make_uint2(0, 0); + } + bool const valid = blk * kPage + tid < kv_len; + uint8_t vsc[8]; + if (valid) + { + uint8_t const* vb = p.vsc + page * p.vsc_blk + kvh * p.vsc_h + (tid >> 2) * 32 + (tid & 3); +#pragma unroll + for (int c = 0; c < 8; ++c) + vsc[c] = __ldg(vb + 4 * c); + } + dequant_row(s.vs + tid * kPitch, p.v + page * p.v_blk + kvh * p.v_h + tid * 64, vsc, valid); + } + __syncthreads(); + + if (active) + { + float const scale_log2 = p.sm_scale * 1.4426950409F * __ldg(p.kgs); + int const q4 = (lane & 3) * 8; + // S = q K^T for this warp's 32 tokens: rows g / g + 8, columns 2 * (lane % 4) + {0, 1} of each n-tile. + float sacc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + sacc[j][i] = 0.F; +#pragma unroll + for (int ks = 0; ks < kDim / 16; ++ks) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.qs + (lane & 15) * kPitch + ks * 16 + (lane >> 4) * 8)); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + uint32_t const scw = ks < 4 ? ksc[j].x : ksc[j].y; + uint8_t const sc = static_cast(scw >> (8 * (ks & 3))); + uint32_t const sh = e4m3x2_to_f16x2(static_cast(sc | (sc << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); + uint32_t const r0 = e2m1x2_to_f16x2((kw[j][2 * ks] >> q4) & 0xFF); + uint32_t const r1 = e2m1x2_to_f16x2((kw[j][2 * ks + 1] >> q4) & 0xFF); + __half2 const h0 = __hmul2(*reinterpret_cast<__half2 const*>(&r0), scale); + __half2 const h1 = __hmul2(*reinterpret_cast<__half2 const*>(&r1), scale); + mma(sacc[j], a, *reinterpret_cast(&h0), *reinterpret_cast(&h1)); + } + } + int const g = lane >> 2; + float mloc[2] = {kNegInf, kNegInf}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const t = warp * 32 + j * 8 + 2 * (lane & 3) + (i & 1); + float const v = blk * kPage + t < kv_len ? 0.F + sacc[j][i] * scale_log2 : kNegInf; + sacc[j][i] = v; + mloc[i >> 1] = fmaxf(mloc[i >> 1], v); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 1)); + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 2)); + } + if ((lane & 3) == 0) + { + s.red[warp][g] = mloc[0]; + s.red[warp][g + 8] = mloc[1]; + } + __syncthreads(); + if (tid < kHeads) + s.rowm[tid] = fmaxf(fmaxf(s.red[0][tid], s.red[1][tid]), fmaxf(s.red[2][tid], s.red[3][tid])); + __syncthreads(); + m_row[0] = s.rowm[g]; + m_row[1] = s.rowm[g + 8]; + float lloc[2] = {0.F, 0.F}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; i += 2) + { + float const p0 = ex2(sacc[j][i] - m_row[i >> 1]); + float const p1 = ex2(sacc[j][i + 1] - m_row[i >> 1]); + lloc[i >> 1] += p0 + p1; + int const row = g + (i >> 1) * 8; + int const t = warp * 32 + j * 8 + 2 * (lane & 3); + *reinterpret_cast<__half2*>(s.ps + row * kPitch + t) = __floats2half2_rn(p0, p1); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 1); + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 2); + } + __syncthreads(); // rowm reads done before red is reused; ps complete + if ((lane & 3) == 0) + { + s.red[warp][g] = lloc[0]; + s.red[warp][g + 8] = lloc[1]; + } + // O[:, warp's 32 dims] = P V over all 128 tokens. +#pragma unroll + for (int kk = 0; kk < kPage; kk += 16) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.ps + (lane & 15) * kPitch + kk + (lane >> 4) * 8)); +#pragma unroll + for (int jp = 0; jp < 2; ++jp) + { + uint32_t b[4]; + int const t = kk + (lane & 7) + ((lane >> 3) & 1) * 8; + int const d = warp * 32 + jp * 16 + (lane >> 4) * 8; + ldsm_x4_t(b, smem_u32(s.vs + t * kPitch + d)); + mma(acc[2 * jp], a, b[0], b[1]); + mma(acc[2 * jp + 1], a, b[2], b[3]); + } + } + __syncthreads(); + if (tid < kHeads) + s.rowl[tid] = (s.red[0][tid] + s.red[1][tid]) + (s.red[2][tid] + s.red[3][tid]); + __syncthreads(); + l_row[0] = s.rowl[g]; + l_row[1] = s.rowl[g + 8]; + } + + // Normalize the partial and push head h's row with its LSE (log2) into CTA h's recv[rank] slot, as the Triton + // producer stores chunk `rank`; st.async stores that complete_tx on CTA h's barrier. + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); + { + int const g = lane >> 2; + float const vgs = __ldg(p.vgs); +#pragma unroll + for (int r = 0; r < 2; ++r) + { + float scale, lse; + if constexpr (LINEAR) + { + scale = l_row[r] > 0.F ? 1.0F / l_row[r] : 0.F; + lse = l_row[r] > 0.F ? m_row[r] + lg2(l_row[r]) : kNegInf; + } + else + { + float const lse_i = active ? m_row[r] + lg2(ex2(kNegInf - m_row[r]) + l_row[r]) : kNegInf; + scale = lse_i > kNegInf ? ex2(m_row[r] - lse_i) : 0.F; + lse = lse_i; + } + scale = scale * vgs; + int const row = g + r * 8; + uint32_t const dst = map_cluster(&s.recv[rank][0], row); + uint32_t const rb = map_cluster(&s.rbar, row); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const d = warp * 32 + j * 8 + 2 * (lane & 3); + st_async_v2(dst + d * 4, acc[j][2 * r] * scale, acc[j][2 * r + 1] * scale, rb); + } + if (warp == 0 && (lane & 3) == 0) + st_async_f32(map_cluster(&s.recv_lse[rank], row), lse, rb); + } + } + { + // Every CTA's partial row and LSE for head `rank` landed (complete_tx on this CTA's barrier). + uint32_t done = 0; + while (!done) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}" + : "=r"(done) + : "r"(smem_u32(&s.rbar)) + : "memory"); + } + } + + // CTA `rank` merges head `rank` across the 16 partials; thread = output dim. The weights are the Triton merge's + // (exp2 of each chunk's LSE minus the largest, divided by their sum in chunk order), formed once per CTA: thread + // c < 16 takes chunk c's exponential, then divides it by the sum every one of those threads forms in the same + // order. + int const h = static_cast(rank); + if (tid < kTopk) + { + float lse_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + lse_max = fmaxf(lse_max, s.recv_lse[c]); + lse_max = lse_max == kNegInf ? 0.F : lse_max; + s.red[0][tid] = ex2(s.recv_lse[tid] - lse_max); + } + __syncthreads(); + if (tid < kTopk) + { + float denom = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + denom += s.red[0][c]; + denom = denom > 0.F ? denom : 1.F; + s.wgt[tid] = __fdiv_rn(s.red[0][tid], denom); + } + __syncthreads(); + float o = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + o += s.recv[c][tid] * s.wgt[c]; + __nv_bfloat16 const ob = __float2bfloat16_rn(o); + p.out[tok * p.out_stride + (kvh * kHeads + h) * kDim + tid] = ob; + + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + amax = fmaxf(amax, __shfl_xor_sync(0xffffffff, amax, off)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + float const xq = x * qs; + uint16_t q2; + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2) : "f"(0.F), "f"(xq)); + p.q8[static_cast(tok) * (p.num_sf_k_tiles * 128) + (kvh * kHeads + h) * kDim + tid] + = static_cast(q2 & 0xFF); + if (lane == 0) + { + int const k_sf = (kvh * kHeads + h) * 4 + warp; + int64_t const off = static_cast(tok / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (tok % 32) * 16 + ((tok % 128) / 32) * 4 + k_sf % 4; + p.sf[off] = static_cast(e); + } + } + if constexpr (!EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +} + +template +cudaError_t configure() +{ + auto kernel = m3_lc_attn_kernel; + cudaError_t err = cudaFuncSetAttribute( + kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smemBytes())); + if (err != cudaSuccess) + { + return err; + } + return cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1); +} + +template +cudaError_t launch(Params const& prm, int num_tokens, bool use_pdl, cudaStream_t stream) +{ + // Kernel attributes are set once per process, on the first launch (outside CUDA-graph capture: warmup runs + // every variant eagerly first). + static cudaError_t const configured = configure(); + if (configured != cudaSuccess) + { + return configured; + } + auto kernel = m3_lc_attn_kernel; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(num_tokens * kTopk, prm.num_kv_heads); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smemBytes(); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kTopk; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +} // namespace + +cudaError_t invokeMinimaxM3Sm107Tp2SparseDecodeLc(MinimaxM3Sm107Tp2SparseDecodeLcParams const& params, + int32_t numTokens, bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, cudaStream_t stream) +{ + if (prewait) + { + if (linearSoftmax) + { + return mxfp8 ? launch(params, numTokens, usePdl, stream) + : launch(params, numTokens, usePdl, stream); + } + return mxfp8 ? launch(params, numTokens, usePdl, stream) + : launch(params, numTokens, usePdl, stream); + } + if (linearSoftmax) + { + return mxfp8 ? launch(params, numTokens, usePdl, stream) + : launch(params, numTokens, usePdl, stream); + } + return mxfp8 ? launch(params, numTokens, usePdl, stream) + : launch(params, numTokens, usePdl, stream); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.h new file mode 100644 index 000000000000..b20a1894f255 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.h @@ -0,0 +1,77 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Operands of the MiniMax-M3 one-launch long-context sparse decode, SM107 TP2 geometry. Strides are in elements of +//! the pointed-to type. Geometry is fixed: 16 query heads per KV head, head_dim 128, page 128, top-k 16. +struct MinimaxM3Sm107Tp2SparseDecodeLcParams +{ + uint8_t const* q; // e4m3 [T, kv_heads * 16, 128] + int64_t q_tok, q_head; + uint8_t const* k; // packed e2m1 [pages, kv_heads, 128, 64] + uint8_t const* v; + int64_t k_blk, v_blk, k_h, v_h; + uint8_t const* ksc; // e4m3 [pages, kv_heads, 128, 8], token-major + uint8_t const* vsc; // e4m3, 4x4 token-quad order + int64_t ksc_blk, vsc_blk, ksc_h, vsc_h; + float const* kgs; + float const* vgs; + int const* topk; // [kv_heads, T, 16] + int64_t t_h, t_tok, t_k; + int const* block_table; + int64_t bt_stride; + int const* seq_lens; + int dql; + float sm_scale; + __nv_bfloat16* out; // [T, kv_heads * 2048] + int64_t out_stride; + uint8_t* q8; // e4m3 [rows, num_sf_k_tiles * 128] + uint8_t* sf; + int num_sf_k_tiles; + int num_kv_heads; + // N1 (optional): this layer's selected block id per (KV head, query row, slot) at its previous decode step, + // int32 [kv_heads, n1_rows, 16]. Before the grid dependency each CTA pulls into L2 the K/V page and scales of the + // block its slot selected last step, then records this step's block. A hint: no value changes. + int* n1_ids; + int64_t n1_h; + int n1_rows; + int64_t n1_pages; +}; + +//! One cluster of 16 CTAs per (query token, KV head): CTA r attends the token's r-th selected block of that head, +//! then the cluster merges the 16 partials over DSMEM into the BF16 rows (and, when mxfp8, the MXFP8 copy with +//! swizzled UE8M0 scales). prewait stages the request's block-table row in shared memory before the grid dependency +//! wait (rows longer than 4096 blocks read it from global memory); the arithmetic is the same in both variants. +//! The device code is SM100-family only; on other architectures the kernel compiles to an empty body. +cudaError_t invokeMinimaxM3Sm107Tp2SparseDecodeLc(MinimaxM3Sm107Tp2SparseDecodeLcParams const& params, + int32_t numTokens, bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.cu new file mode 100644 index 000000000000..cedb0d8c0c01 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.cu @@ -0,0 +1,671 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3 long-context sparse decode, SM107 TP2 geometry, LC2-split: LC2 (sparseDecodeLc2.cu) without its thread- +// block cluster. Grid 1 (tokens * 16, kv heads): CTA `r` attends the token's `r`-th selected block exactly as LC2 does +// and stores its normalized partial rows and LSE (LC2's pushed values) to a workspace, head-major. Grid 2 (tokens, kv +// heads): one warp per head combines the 16 partials with LC2's merge weights in slot order and writes the BF16 row +// and, optionally, LC2's MXFP8 copy and swizzled UE8M0 scales. Bit-identical to LC2. +// +// Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and +// PV on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +// log-domain recurrence. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace +{ + +constexpr int kHeads = 16; +constexpr int kDim = 128; +constexpr int kPage = 128; +constexpr int kTopk = 16; +constexpr int kThreads = 128; +constexpr int kPitch = kDim + 8; // halves per smem row: 272 B, conflict-free ldmatrix +constexpr float kNegInf = -__builtin_huge_valf(); +constexpr int kBtCap = 4096; // block-table entries staged in smem by PREWAIT (512K tokens) + +using Params = MinimaxM3Sm107Tp2SparseDecodeLcParams; + +struct Smem +{ + __half qs[kHeads * kPitch]; + __half vs[kPage * kPitch]; + __half ps[kHeads * kPitch]; + float red[4][kHeads]; + float rowm[kHeads]; + float rowl[kHeads]; +}; + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t e2m1x2_to_f16x2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 s;\n cvt.u8.u32 s, %1;\n cvt.rn.f16x2.e2m1x2 %0, s;\n}" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t e4m3x2_to_f16x2(uint16_t pair) +{ + uint32_t out; + asm("cvt.rn.f16x2.e4m3x2 %0, %1;" : "=r"(out) : "h"(pair)); + return out; +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ void ldsm_x4(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x4_t(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma(float (&c)[4], uint32_t const (&a)[4], uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};" + : "+f"(c[0]), "+f"(c[1]), "+f"(c[2]), "+f"(c[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} + +__device__ __forceinline__ uint32_t map_cluster(void const* local, uint32_t rank) +{ + uint32_t remote; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(remote) : "r"(smem_u32(local)), "r"(rank)); + return remote; +} + +// Posted DSMEM stores that complete_tx on the receiving CTA's mbarrier `bar`. +__device__ __forceinline__ void st_async_v2(uint32_t addr, float a, float b, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.v2.f32 [%0], {%1, %2}, [%3];" ::"r"(addr), + "f"(a), "f"(b), "r"(bar) + : "memory"); +} + +__device__ __forceinline__ void st_async_f32(uint32_t addr, float a, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(addr), + "r"(__float_as_uint(a)), "r"(bar) + : "memory"); +} + +// One packed NVFP4 row (64 bytes, 128 values) with its 8 E4M3 scales -> 128 fp16 values in smem. +__device__ __forceinline__ void dequant_row(__half* dst, uint8_t const* packed, uint8_t const (&sc)[8], bool valid) +{ + uint4 const* src = reinterpret_cast(packed); + uint32_t* d = reinterpret_cast(dst); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + uint32_t words[4] = {w.x, w.y, w.z, w.w}; + // 16 bytes = 32 values = scale columns 2c and 2c + 1. +#pragma unroll + for (int h = 0; h < 2; ++h) + { + uint8_t const s = valid ? sc[2 * c + h] : 0; + uint32_t const sh = e4m3x2_to_f16x2(static_cast(s | (s << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); +#pragma unroll + for (int j = 0; j < 2; ++j) + { + uint32_t const word = words[2 * h + j]; +#pragma unroll + for (int b = 0; b < 4; ++b) + { + uint32_t const v2 = e2m1x2_to_f16x2((word >> (8 * b)) & 0xFF); + __half2 const x = __hmul2(*reinterpret_cast<__half2 const*>(&v2), scale); + d[c * 16 + h * 8 + j * 4 + b] = *reinterpret_cast(&x); + } + } + } + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +template +constexpr size_t smemBytes() +{ + return PREWAIT ? sizeof(Smem) + kBtCap * sizeof(int) : sizeof(Smem); +} + +template +__global__ void __launch_bounds__(kThreads) + m3_lcs_attn_kernel(__grid_constant__ Params const p, float* const s_part, float* const s_lse) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(128) uint8_t smem_raw[]; + Smem& s = *reinterpret_cast(smem_raw); + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const tok = blockIdx.x / kTopk; + int const kvh = blockIdx.y; + uint32_t rank; + rank = blockIdx.x % kTopk; // LC2-split: no cluster; slot = the CTA's position in its 16 + + // Pre-wait read contract: everything that does not depend on the top-16 table is read before the grid dependency. + // - seq_lens: written inside the step's graph by msa_kv_update (the KV-length correction). That kernel is a plain + // launch, not a programmatic dependent, so it has completed before any later kernel of the step starts. + // - the query rows: written by the producer, two grids back. This holds only while every kernel between the + // producer and this grid (the long-row select) triggers its dependents after its own griddepcontrol.wait, so + // this grid cannot start before the producer completed. A kernel inserted into that chain, or a select that + // triggers before its wait, breaks it: these reads then move after the griddepcontrol.wait below. + int const req = tok / p.dql; + int const qoff = tok - req * p.dql; + int const seq = p.seq_lens[req]; + int const kv_len = max(seq - p.dql + qoff + 1, 0); + int const nblocks = (kv_len + kPage - 1) / kPage; + int const real_topk = min(kTopk, nblocks); + // q: 16 heads x 128 e4m3 -> fp16; thread = (head, 16-value chunk). + { + int const h = tid >> 3; + int const c = (tid & 7) * 16; + uint4 const w = *reinterpret_cast(p.q + tok * p.q_tok + (kvh * kHeads + h) * p.q_head + c); + uint32_t const words[4] = {w.x, w.y, w.z, w.w}; + uint32_t* d = reinterpret_cast(s.qs + h * kPitch + c); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + d[2 * i] = e4m3x2_to_f16x2(static_cast(words[i] & 0xFFFF)); + d[2 * i + 1] = e4m3x2_to_f16x2(static_cast(words[i] >> 16)); + } + } + + // PREWAIT: the request's block-table row (written before the graph runs) is staged before the grid dependency. + int* const s_bt = reinterpret_cast(smem_raw + sizeof(Smem)); + bool bt_smem = false; + if constexpr (PREWAIT) + { + int const nbt = (max(seq, 0) + kPage - 1) / kPage; + bt_smem = nbt <= kBtCap; + if (bt_smem) + { + for (int i = tid; i < nbt; i += kThreads) + s_bt[i] = __ldg(p.block_table + req * p.bt_stride + i); + } + } + // N1: the block this (KV head, row, slot) selected at the layer's previous decode step recurs ~2/3 of the time; + // pull its K/V page and scales into L2 while the select runs. The ids were written by this kernel one step back + // (complete before the graph's next replay) and the block-table row before the graph runs. Read and written by + // the same thread, so the record below cannot overtake this read. Out-of-range ids (stale rows) are skipped. + bool const n1 = p.n1_ids != nullptr && tok < p.n1_rows; + int* const n1_slot = n1 ? p.n1_ids + kvh * p.n1_h + tok * kTopk + rank : nullptr; + if (n1 && tid == 64) + { + int const nb = *n1_slot; + if (nb >= 0 && nb < nblocks) + { + int64_t const pg = __ldg(p.block_table + req * p.bt_stride + nb); + if (pg >= 0 && pg < p.n1_pages) + { + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.k + pg * p.k_blk + kvh * p.k_h), + "r"(kPage * 64) + : "memory"); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.v + pg * p.v_blk + kvh * p.v_h), + "r"(kPage * 64) + : "memory"); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.ksc + pg * p.ksc_blk + kvh * p.ksc_h), + "r"(kPage * 8) + : "memory"); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.vsc + pg * p.vsc_blk + kvh * p.vsc_h), + "r"(kPage * 8) + : "memory"); + } + } + } + asm volatile("griddepcontrol.wait;" ::: "memory"); + if constexpr (EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } + int blk = static_cast(rank) < real_topk ? p.topk[kvh * p.t_h + tok * p.t_tok + rank * p.t_k] : -1; + if (n1 && tid == 64) + { + *n1_slot = blk; + } + bool const active = blk >= 0; + if constexpr (PREWAIT) + { + __syncthreads(); // s_bt complete + } + int64_t const page_pw = !active ? 0 : (bt_smem ? s_bt[blk] : p.block_table[req * p.bt_stride + blk]); + + float m_row[2] = {kNegInf, kNegInf}; + float acc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + acc[j][i] = 0.F; + float l_row[2] = {0.F, 0.F}; + + // K stays packed in registers: the MMA B fragment of token n = lane / 4 at k-step s is the f16x2 pair of + // packed bytes 8s + q and 8s + 4 + q (q = lane % 4), both under scale column s. + uint32_t kw[4][16]; + uint2 ksc[4]; + int64_t page = 0; + if (active) + { + page = page_pw; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const n = warp * 32 + j * 8 + (lane >> 2); + bool const valid = blk * kPage + n < kv_len; + uint4 const* src = reinterpret_cast(p.k + page * p.k_blk + kvh * p.k_h + n * 64); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 const w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + kw[j][4 * c] = w.x; + kw[j][4 * c + 1] = w.y; + kw[j][4 * c + 2] = w.z; + kw[j][4 * c + 3] = w.w; + } + ksc[j] = valid ? __ldg(reinterpret_cast(p.ksc + page * p.ksc_blk + kvh * p.ksc_h + n * 8)) + : make_uint2(0, 0); + } + bool const valid = blk * kPage + tid < kv_len; + uint8_t vsc[8]; + if (valid) + { + uint8_t const* vb = p.vsc + page * p.vsc_blk + kvh * p.vsc_h + (tid >> 2) * 32 + (tid & 3); +#pragma unroll + for (int c = 0; c < 8; ++c) + vsc[c] = __ldg(vb + 4 * c); + } + dequant_row(s.vs + tid * kPitch, p.v + page * p.v_blk + kvh * p.v_h + tid * 64, vsc, valid); + } + __syncthreads(); + + if (active) + { + float const scale_log2 = p.sm_scale * 1.4426950409F * __ldg(p.kgs); + int const q4 = (lane & 3) * 8; + // S = q K^T for this warp's 32 tokens: rows g / g + 8, columns 2 * (lane % 4) + {0, 1} of each n-tile. + float sacc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + sacc[j][i] = 0.F; +#pragma unroll + for (int ks = 0; ks < kDim / 16; ++ks) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.qs + (lane & 15) * kPitch + ks * 16 + (lane >> 4) * 8)); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + uint32_t const scw = ks < 4 ? ksc[j].x : ksc[j].y; + uint8_t const sc = static_cast(scw >> (8 * (ks & 3))); + uint32_t const sh = e4m3x2_to_f16x2(static_cast(sc | (sc << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); + uint32_t const r0 = e2m1x2_to_f16x2((kw[j][2 * ks] >> q4) & 0xFF); + uint32_t const r1 = e2m1x2_to_f16x2((kw[j][2 * ks + 1] >> q4) & 0xFF); + __half2 const h0 = __hmul2(*reinterpret_cast<__half2 const*>(&r0), scale); + __half2 const h1 = __hmul2(*reinterpret_cast<__half2 const*>(&r1), scale); + mma(sacc[j], a, *reinterpret_cast(&h0), *reinterpret_cast(&h1)); + } + } + int const g = lane >> 2; + float mloc[2] = {kNegInf, kNegInf}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const t = warp * 32 + j * 8 + 2 * (lane & 3) + (i & 1); + float const v = blk * kPage + t < kv_len ? 0.F + sacc[j][i] * scale_log2 : kNegInf; + sacc[j][i] = v; + mloc[i >> 1] = fmaxf(mloc[i >> 1], v); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 1)); + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 2)); + } + if ((lane & 3) == 0) + { + s.red[warp][g] = mloc[0]; + s.red[warp][g + 8] = mloc[1]; + } + __syncthreads(); + if (tid < kHeads) + s.rowm[tid] = fmaxf(fmaxf(s.red[0][tid], s.red[1][tid]), fmaxf(s.red[2][tid], s.red[3][tid])); + __syncthreads(); + m_row[0] = s.rowm[g]; + m_row[1] = s.rowm[g + 8]; + float lloc[2] = {0.F, 0.F}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; i += 2) + { + float const p0 = ex2(sacc[j][i] - m_row[i >> 1]); + float const p1 = ex2(sacc[j][i + 1] - m_row[i >> 1]); + lloc[i >> 1] += p0 + p1; + int const row = g + (i >> 1) * 8; + int const t = warp * 32 + j * 8 + 2 * (lane & 3); + *reinterpret_cast<__half2*>(s.ps + row * kPitch + t) = __floats2half2_rn(p0, p1); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 1); + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 2); + } + __syncthreads(); // rowm reads done before red is reused; ps complete + if ((lane & 3) == 0) + { + s.red[warp][g] = lloc[0]; + s.red[warp][g + 8] = lloc[1]; + } + // O[:, warp's 32 dims] = P V over all 128 tokens. +#pragma unroll + for (int kk = 0; kk < kPage; kk += 16) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.ps + (lane & 15) * kPitch + kk + (lane >> 4) * 8)); +#pragma unroll + for (int jp = 0; jp < 2; ++jp) + { + uint32_t b[4]; + int const t = kk + (lane & 7) + ((lane >> 3) & 1) * 8; + int const d = warp * 32 + jp * 16 + (lane >> 4) * 8; + ldsm_x4_t(b, smem_u32(s.vs + t * kPitch + d)); + mma(acc[2 * jp], a, b[0], b[1]); + mma(acc[2 * jp + 1], a, b[2], b[3]); + } + } + __syncthreads(); + if (tid < kHeads) + s.rowl[tid] = (s.red[0][tid] + s.red[1][tid]) + (s.red[2][tid] + s.red[3][tid]); + __syncthreads(); + l_row[0] = s.rowl[g]; + l_row[1] = s.rowl[g + 8]; + } + + // LC2-split: store this CTA's normalized partial rows and LSE (LC2's pushed values, the same expressions) to the + // workspace, head-major: part[row][kvh][head][slot][128], lse[row][kvh][head][slot]. The merge grid combines them. + { + int const g = lane >> 2; + float const vgs = __ldg(p.vgs); +#pragma unroll + for (int r = 0; r < 2; ++r) + { + float scale, lse; + if constexpr (LINEAR) + { + scale = l_row[r] > 0.F ? 1.0F / l_row[r] : 0.F; + lse = l_row[r] > 0.F ? m_row[r] + lg2(l_row[r]) : kNegInf; + } + else + { + float const lse_i = active ? m_row[r] + lg2(ex2(kNegInf - m_row[r]) + l_row[r]) : kNegInf; + scale = lse_i > kNegInf ? ex2(m_row[r] - lse_i) : 0.F; + lse = lse_i; + } + scale = scale * vgs; + int const row = g + r * 8; + int64_t const hs = ((static_cast(tok) * p.num_kv_heads + kvh) * kHeads + row) * kTopk + rank; + float* const dst = s_part + hs * kDim; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const d = warp * 32 + j * 8 + 2 * (lane & 3); + *reinterpret_cast(dst + d) = make_float2(acc[j][2 * r] * scale, acc[j][2 * r + 1] * scale); + } + if (warp == 0 && (lane & 3) == 0) + s_lse[hs] = lse; + } + } + if constexpr (!EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +} + +template +cudaError_t configure() +{ + auto kernel = m3_lcs_attn_kernel; + cudaError_t err = cudaFuncSetAttribute( + kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smemBytes())); + if (err != cudaSuccess) + { + return err; + } + return cudaSuccess; +} + +template +cudaError_t launch(Params const& prm, int num_tokens, bool use_pdl, float* part, float* lse, cudaStream_t stream) +{ + // Kernel attributes are set once per process, on the first launch (outside CUDA-graph capture: warmup runs + // every variant eagerly first). + static cudaError_t const configured = configure(); + if (configured != cudaSuccess) + { + return configured; + } + auto kernel = m3_lcs_attn_kernel; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(num_tokens * kTopk, prm.num_kv_heads); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smemBytes(); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kTopk; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs + 1; // no cluster + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, kernel, prm, part, lse); +} + +template +__global__ void __launch_bounds__(kHeads * 32) + m3_lcs_merge_kernel(__grid_constant__ Params const p, float const* const s_part, float const* const s_lse) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + // One CTA per (row, KV head), one warp per head; lane l owns dims 4l .. 4l + 3. LC2's merge per head: the 16 slot + // weights (exp2 of each LSE minus the largest, divided by their sum formed in slot order), then the weighted sum of + // the 16 partials in slot order, the BF16 row and the MXFP8 hand-off, value for value. + int const tok = blockIdx.x; + int const kvh = blockIdx.y; + int const h = threadIdx.x >> 5; + int const lane = threadIdx.x & 31; + asm volatile("griddepcontrol.wait;" ::: "memory"); + int64_t const hs = ((static_cast(tok) * p.num_kv_heads + kvh) * kHeads + h) * kTopk; + float const lse_c = lane < kTopk ? s_lse[hs + lane] : kNegInf; + float lse_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + lse_max = fmaxf(lse_max, __shfl_sync(0xffffffff, lse_c, c)); + lse_max = lse_max == kNegInf ? 0.F : lse_max; + float const e_c = ex2(lse_c - lse_max); + float denom = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + denom += __shfl_sync(0xffffffff, e_c, c); + denom = denom > 0.F ? denom : 1.F; + float const w_c = __fdiv_rn(e_c, denom); + float o[4] = {0.F, 0.F, 0.F, 0.F}; + float4 const* const src = reinterpret_cast(s_part + hs * kDim) + lane; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + float const w = __shfl_sync(0xffffffff, w_c, c); + float4 const v = src[c * (kDim / 4)]; + o[0] += v.x * w; + o[1] += v.y * w; + o[2] += v.z * w; + o[3] += v.w * w; + } + __nv_bfloat16 ob[4]; +#pragma unroll + for (int i = 0; i < 4; ++i) + ob[i] = __float2bfloat16_rn(o[i]); + __nv_bfloat16* const orow = p.out + tok * p.out_stride + (kvh * kHeads + h) * kDim + 4 * lane; + *reinterpret_cast(orow) = *reinterpret_cast(ob); + if constexpr (MXFP8) + { + // LC2's per-32-dim group: amax over the group (8 lanes here), the UE8M0 exponent and the E4M3 values. + float amax = fabsf(__bfloat162float(ob[0])); +#pragma unroll + for (int i = 1; i < 4; ++i) + amax = fmaxf(amax, fabsf(__bfloat162float(ob[i]))); +#pragma unroll + for (int off = 4; off > 0; off >>= 1) + amax = fmaxf(amax, __shfl_xor_sync(0xffffffff, amax, off)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + uint8_t q8v[4]; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + float const xq = __bfloat162float(ob[i]) * qs; + uint16_t q2; + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2) : "f"(0.F), "f"(xq)); + q8v[i] = static_cast(q2 & 0xFF); + } + *reinterpret_cast( + p.q8 + static_cast(tok) * (p.num_sf_k_tiles * 128) + (kvh * kHeads + h) * kDim + 4 * lane) + = *reinterpret_cast(q8v); + if ((lane & 7) == 0) + { + int const warp = lane >> 3; // LC2's warp index = the 32-dim group + int const k_sf = (kvh * kHeads + h) * 4 + warp; + int64_t const off = static_cast(tok / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (tok % 32) * 16 + ((tok % 128) / 32) * 4 + k_sf % 4; + p.sf[off] = static_cast(e); + } + } +#endif +} + +template +cudaError_t launchMerge( + Params const& prm, int num_tokens, bool use_pdl, float const* part, float const* lse, cudaStream_t stream) +{ + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(num_tokens, prm.num_kv_heads); + cfg.blockDim = dim3(kHeads * 32); + cfg.stream = stream; + cudaLaunchAttribute attrs[1]; + attrs[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[0].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, m3_lcs_merge_kernel, prm, part, lse); +} + +} // namespace + +size_t minimaxM3Sm107Tp2SparseDecodeLcSplitWorkspaceBytes(int32_t maxTokens, int32_t numKvHeads) +{ + size_t const hs = static_cast(maxTokens) * numKvHeads * kHeads * kTopk; + return hs * kDim * sizeof(float) + hs * sizeof(float); +} + +cudaError_t invokeMinimaxM3Sm107Tp2SparseDecodeLcSplit(MinimaxM3Sm107Tp2SparseDecodeLcParams const& params, + int32_t numTokens, bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, void* workspace, + size_t workspaceBytes, cudaStream_t stream) +{ + size_t const hs = static_cast(numTokens) * params.num_kv_heads * kHeads * kTopk; + if (workspace == nullptr || workspaceBytes < hs * (kDim + 1) * sizeof(float)) + { + return cudaErrorInvalidValue; + } + float* const part = static_cast(workspace); + float* const lse = part + hs * kDim; + // The compute grid triggers its dependents at its end (EARLY off), so no merge CTA waits on a slot it needs. + cudaError_t err; + if (prewait) + { + err = linearSoftmax ? (mxfp8 ? launch(params, numTokens, usePdl, part, lse, stream) + : launch(params, numTokens, usePdl, part, lse, stream)) + : (mxfp8 ? launch(params, numTokens, usePdl, part, lse, stream) + : launch(params, numTokens, usePdl, part, lse, stream)); + } + else + { + err = linearSoftmax + ? (mxfp8 ? launch(params, numTokens, usePdl, part, lse, stream) + : launch(params, numTokens, usePdl, part, lse, stream)) + : (mxfp8 ? launch(params, numTokens, usePdl, part, lse, stream) + : launch(params, numTokens, usePdl, part, lse, stream)); + } + if (err != cudaSuccess) + { + return err; + } + return mxfp8 ? launchMerge(params, numTokens, usePdl, part, lse, stream) + : launchMerge(params, numTokens, usePdl, part, lse, stream); +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.h new file mode 100644 index 000000000000..a79cf29d8b0b --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.h @@ -0,0 +1,46 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Workspace bytes of LC2-split for up to `maxTokens` query rows: the fp32 partial rows and LSEs of every (row, KV +//! head, head, slot). +size_t minimaxM3Sm107Tp2SparseDecodeLcSplitWorkspaceBytes(int32_t maxTokens, int32_t numKvHeads); + +//! LC2-split: the LC2 sparse decode (same operands, bit-identical outputs) without the thread-block cluster. A compute +//! grid writes each (row, slot) CTA's normalized partial and LSE to `workspace`; a merge grid combines the 16 partials +//! of each head with LC2's weights in slot order and writes the BF16 row and, when mxfp8, the MXFP8 copy. prewait and +//! N1 behave as in LC2 (hints; they do not change a value). +cudaError_t invokeMinimaxM3Sm107Tp2SparseDecodeLcSplit(MinimaxM3Sm107Tp2SparseDecodeLcParams const& params, + int32_t numTokens, bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, void* workspace, + size_t workspaceBytes, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.cu new file mode 100644 index 000000000000..22c073f74b3c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.cu @@ -0,0 +1,58 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +namespace +{ + +constexpr int kThreads = 256; + +__global__ void __launch_bounds__(kThreads) + specOffsetsKernel(int32_t* __restrict__ pos, int32_t const* __restrict__ posOff, int64_t nPos, + int32_t* __restrict__ kv, int32_t const* __restrict__ kvOff, int64_t nKv) +{ + for (int64_t i = threadIdx.x; i < nPos; i += kThreads) + { + pos[i] = static_cast(static_cast(pos[i]) + static_cast(posOff[i])); + } + for (int64_t j = threadIdx.x; j < nKv; j += kThreads) + { + kv[j] = static_cast(static_cast(kv[j]) + static_cast(kvOff[j])); + } +} + +} // namespace + +cudaError_t invokeSpecOffsets(int32_t* pos, int32_t const* posOff, int64_t nPos, int32_t* kv, int32_t const* kvOff, + int64_t nKv, cudaStream_t stream) +{ + specOffsetsKernel<<<1, kThreads, 0, stream>>>(pos, posOff, nPos, kv, kvOff, nKv); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.h new file mode 100644 index 000000000000..1de212a37179 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.h @@ -0,0 +1,41 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +//! The overlap scheduler's speculative step correction in one launch: pos[i] += posOff[i] for i < nPos and +//! kv[j] += kvOff[j] for j < nKv (int32, wrapping as the ATen adds do). A plain launch: it is a step-input writer that +//! pre-wait readers rely on (rubin-20261015/PREWAIT_AUDIT.md item b). +cudaError_t invokeSpecOffsets(int32_t* pos, int32_t const* posOff, int64_t nPos, int32_t* kv, int32_t const* kvOff, + int64_t nKv, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.cu new file mode 100644 index 000000000000..9b6dbb42faed --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.cu @@ -0,0 +1,111 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +namespace +{ + +constexpr int kThreads = 512; +constexpr int kWarps = kThreads / 32; + +// One warp per copy. Plain (coherent) loads: a device copy may read bytes another warp of this CTA just stored, which +// the CTA barrier between the two kinds makes visible; a non-coherent load could return the stale line. +__device__ __forceinline__ void copyWarp(uint8_t* dst, uint8_t const* src, int64_t bytes, int lane) +{ + auto const d = reinterpret_cast(dst); + auto const s = reinterpret_cast(src); + if (((d | s | static_cast(bytes)) & 15) == 0) + { + auto* d16 = reinterpret_cast(dst); + auto const* s16 = reinterpret_cast(src); + for (int64_t i = lane; i < bytes / 16; i += 32) + { + d16[i] = s16[i]; + } + } + else if (((d | s | static_cast(bytes)) & 3) == 0) + { + auto* d4 = reinterpret_cast(dst); + auto const* s4 = reinterpret_cast(src); + for (int64_t i = lane; i < bytes / 4; i += 32) + { + d4[i] = s4[i]; + } + } + else + { + for (int64_t i = lane; i < bytes; i += 32) + { + dst[i] = src[i]; + } + } +} + +// One CTA: a step's window is a few dozen copies of a few hundred bytes, so the launch is the cost; a CTA barrier +// orders the host copies before the device copies where the parallel scatter needed a second launch. +__global__ void __launch_bounds__(kThreads) stepPrologueKernel(uint8_t* window, int64_t* epoch, int64_t maxDesc) +{ + auto* header = reinterpret_cast(window); + int64_t const numHost = min(header[0], maxDesc); + int64_t const numDevice = min(header[1], maxDesc); + int64_t const* desc = header + 2; + int const warp = threadIdx.x / 32; + int const lane = threadIdx.x % 32; + if (epoch != nullptr && threadIdx.x == 0) + { + *epoch += 1; + } + for (int64_t i = warp; i < numHost; i += kWarps) + { + int64_t const* c = desc + 3 * i; + copyWarp(reinterpret_cast(c[0]), window + c[1], c[2], lane); + } + __syncthreads(); + for (int64_t i = warp; i < numDevice; i += kWarps) + { + // The device copies' slots follow all header[0] host copies, as the parallel scatter indexes them. + int64_t const* c = desc + 3 * (header[0] + i); + copyWarp(reinterpret_cast(c[0]), reinterpret_cast(c[1]), c[2], lane); + } + // Every thread has read the header and the descriptors: consume the window. + __syncthreads(); + if (threadIdx.x < 2) + { + header[threadIdx.x] = 0; + } +} + +} // namespace + +cudaError_t invokeStepPrologue(uint8_t* window, int64_t* epoch, int64_t maxDesc, cudaStream_t stream) +{ + stepPrologueKernel<<<1, kThreads, 0, stream>>>(window, epoch, maxDesc); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.h new file mode 100644 index 000000000000..44a7b7bcb44d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.h @@ -0,0 +1,47 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +//! The decode graph's step-input prologue in one launch (step_inputs.StepInputPack with early_copy): the work of the +//! two parallel scatter launches and the header clear that follow them, in that order. +//! +//! The window is the pack's staging layout (kernels/stepInputsScatter.h): an int64 header `[numHostCopies, +//! numDeviceCopies]`, then one `(dst, src, bytes)` int64 triple per copy, host copies first; a host copy's `src` is a +//! byte offset into the window, a device copy's a device address. Up to `maxDesc` copies of each kind are written (the +//! slots the parallel scatter's grid covers), every host copy before any device copy, so a device copy may read a +//! destination a host copy filled; the copies of one kind must not overlap (the pack issues the window only then). +//! Afterwards the header is zeroed, so a replay with no new upload writes nothing. With `epoch` non-null, it is +//! incremented once. +cudaError_t invokeStepPrologue(uint8_t* window, int64_t* epoch, int64_t maxDesc, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.cu new file mode 100644 index 000000000000..5da59fcec01a --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.cu @@ -0,0 +1,100 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +namespace +{ + +constexpr int kThreads = 256; + +// One packed bf16 pair of gate and up values -> its two bf16 results (the low half from the low halves): the +// instruction sequence of the Triton kernel at limit 7, alpha 1.702, beta 1 (Indexer kdev17, bitwise on SM107): the +// gate clamp as min.bf16x2 before the widening, the up clamp as min.xorsign.abs, (u + 1), (g * -1.702) * log2(e), +// ex2.approx, + 1, div.full 1 / that, s * g, the product, one rounding to bf16. +__device__ __forceinline__ uint32_t swigluOaiBf16x2(uint32_t gate, uint32_t up) +{ + uint32_t r; + asm("{\n\t.reg .b32 g, glim, gl, gh, ul, uh, one, lim, t0, t1, e0, e1, s0, s1, r0, r1;\n\t.reg .b16 hl, hh;\n\t" + "mov.b32 glim, 0x40E040E0;\n\tmin.bf16x2 g, %1, glim;\n\t" + "mov.b32 {hl, hh}, g;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, %2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mov.b32 lim, 0f40E00000;\n\tmin.xorsign.abs.f32 ul, ul, lim;\n\tmin.xorsign.abs.f32 uh, uh, lim;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 ul, ul, one;\n\tadd.rn.f32 uh, uh, one;\n\t" + "mul.rn.f32 t0, gl, 0fBFD9DB23;\n\tmul.rn.f32 t1, gh, 0fBFD9DB23;\n\t" + "mul.rn.f32 t0, t0, 0f3FB8AA3B;\n\tmul.rn.f32 t1, t1, 0f3FB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "add.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 r0, s0, gl;\n\tmul.rn.f32 r1, s1, gh;\n\t" + "mul.rn.f32 r0, ul, r0;\n\tmul.rn.f32 r1, uh, r1;\n\t" + "cvt.rn.bf16x2.f32 %0, r1, r0;\n\t}" + : "=r"(r) + : "r"(gate), "r"(up)); + return r; +} + +template +__global__ void __launch_bounds__(kThreads) + swigluOaiKernel(uint32_t const* __restrict__ x, uint32_t* __restrict__ out, int64_t rows, int64_t halfD) +{ + if constexpr (kPdl) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + int64_t const i = blockIdx.x * static_cast(kThreads) + threadIdx.x; + if (i >= rows * halfD) + { + return; + } + int64_t const row = i / halfD; + int64_t const col = i - row * halfD; + uint32_t const* xr = x + row * 2 * halfD; + out[i] = swigluOaiBf16x2(xr[col], xr[halfD + col]); +} + +} // namespace + +cudaError_t invokeSwigluOai( + uint32_t const* x, uint32_t* out, int64_t rows, int64_t halfD, bool pdl, cudaStream_t stream) +{ + int64_t const n = rows * halfD; + cudaLaunchConfig_t cfg{}; + cfg.gridDim = dim3(static_cast((n + kThreads - 1) / kThreads)); + cfg.blockDim = dim3(kThreads); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return pdl ? cudaLaunchKernelEx(&cfg, swigluOaiKernel, x, out, rows, halfD) + : cudaLaunchKernelEx(&cfg, swigluOaiKernel, x, out, rows, halfD); +} + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.h new file mode 100644 index 000000000000..c8db72a5efd3 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.h @@ -0,0 +1,43 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp2 +{ + +//! The dense MLP's swigluoai activation: out[m, j] = bf16(min(g, 7) * sigmoid(1.702 * min(g, 7)) * (clamp(u, -7, 7) + +//! 1)) with g = x[m, j], u = x[m, d + j] (x bf16 [rows, 2d], out bf16 [rows, d], both contiguous, d even): bit for bit +//! the Triton silu_and_mul kernel at swiglu_limit 7, swiglu_alpha 1.702, swiglu_beta 1. With `pdl` the launch allows +//! programmatic serialization and the grid waits on its dependency, then releases its dependents before it computes +//! (the down GEMV stages its weights during the activation and reads it only after its own grid wait). +cudaError_t invokeSwigluOai( + uint32_t const* x, uint32_t* out, int64_t rows, int64_t halfD, bool pdl, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp2 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.cu new file mode 100644 index 000000000000..e173c91016c1 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.cu @@ -0,0 +1,296 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// The TP4 dense layers' QK norm + RoPE with FP8 output. The norm and RoPE are the generic fusedQKNormRopeKernel's at +// head_dim 128 without YaRN or mRoPE, expression for expression; the store rounds Q/K to BF16 and then to E4M3 +// (saturating), the two roundings of the stock BF16 kernel followed by its FP8 cast. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace common +{ +// tensorrt_llm/common/envUtils.h (not included: its CUDA utility headers do not build under the extension's +// no-half-conversion flags). +bool getEnvEnablePDL(); +} // namespace common + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +namespace denseQkNormRopeFp8 +{ +namespace +{ + +constexpr int kElemsPerThread = kHeadDim / 32; +constexpr int kVecSize = kElemsPerThread * sizeof(__nv_bfloat16) / 4; +constexpr int kBlockSize = 256; +static_assert(kVecSize == 2, "a lane loads 4 BF16 values as one uint2"); + +// tensorrt_llm::common::warpReduceSum: the same butterfly, so the same sum. +__device__ __forceinline__ float warpReduceSum(float val) +{ +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + { + val = val + __shfl_xor_sync(0xffffffff, val, mask, 32); + } + return val; +} + +__device__ __forceinline__ void storeFp8ViaBf16(__nv_fp8_e4m3* out, long long offsetThread, float const (&elements)[4]) +{ +#pragma unroll + for (int i = 0; i < kElemsPerThread; i += 2) + { + float2 const rounded = __bfloat1622float2(__float22bfloat162_rn(make_float2(elements[i], elements[i + 1]))); + __nv_fp8x2_e4m3 const packed(rounded); + reinterpret_cast<__nv_fp8x2_storage_t*>(&out[offsetThread])[i / 2] = packed.__x; + } +} + +// The K (kv 0) or V (kv 1) row of one head of token tokenIdx in the HND paged pool, at the token's cache slot. +__device__ __forceinline__ __nv_fp8_e4m3* cacheRow(KvCacheView const& cache, int kv, int headIdx, int tokenIdx) +{ + long long const slot = static_cast(cache.outCacheLoc[tokenIdx]); + long long const page = slot / cache.tokensPerBlock; + long long const within = slot % cache.tokensPerBlock; + return cache.base + page * cache.pageStride + kv * cache.kvStride + headIdx * cache.headStride + + within * cache.tokenStride; +} + +template +__global__ void denseQkNormRopeFp8Kernel(__nv_bfloat16 const* qkvIn, __nv_fp8_e4m3* qkvOut, int numHeadsQ, + int numHeadsK, int numHeadsV, int rotaryDim, float eps, __nv_bfloat16 const* qWeight, __nv_bfloat16 const* kWeight, + float base, int const* positionIds, int numTokens, bool useGemma, KvCacheView cache) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + // PDL: every read of this step's q/k/v rows, positions and cache tables follows the wait. The dependent (the + // full-context decode) reads this kernel's outputs only after its own griddepcontrol.wait, so it is released at + // entry and stages its CTAs while this kernel runs. + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); +#endif + int const warpsPerBlock = blockDim.x / 32; + int const warpId = threadIdx.x / 32; + int const laneId = threadIdx.x % 32; + int const globalWarpIdx = blockIdx.x * warpsPerBlock + warpId; + int const totalQkHeads = numHeadsQ + numHeadsK; + int const totalHeads = totalQkHeads + numHeadsV; + int const tokenIdx = globalWarpIdx / totalHeads; + int const localHeadIdx = globalWarpIdx % totalHeads; + if (tokenIdx >= numTokens) + { + return; + } + bool const isQ = localHeadIdx < numHeadsQ; + bool const isV = localHeadIdx >= totalQkHeads; + int headIdx; + int segStart; + if (isQ) + { + headIdx = localHeadIdx; + segStart = 0; + } + else if (!isV) + { + headIdx = localHeadIdx - numHeadsQ; + segStart = numHeadsQ * kHeadDim; + } + else + { + headIdx = localHeadIdx - totalQkHeads; + segStart = totalQkHeads * kHeadDim; + } + + using vec_T = uint2; + int const offsetThread + = tokenIdx * totalHeads * kHeadDim + segStart + headIdx * kHeadDim + laneId * kElemsPerThread; + float elements[kElemsPerThread]; + float sumOfSquares = 0.0f; + { + vec_T vec = *reinterpret_cast(&qkvIn[offsetThread]); +#pragma unroll + for (int i = 0; i < kVecSize; i++) + { + float2 vals = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162*>(reinterpret_cast(&vec) + i)); + sumOfSquares += vals.x * vals.x; + sumOfSquares += vals.y * vals.y; + elements[2 * i] = vals.x; + elements[2 * i + 1] = vals.y; + } + } + if (isV) + { + // V is BF16 already: its single E4M3 rounding is the stock cast's. + storeFp8ViaBf16(qkvOut, offsetThread, elements); + if (kWriteCache && tokenIdx < cache.numLiveTokens && cache.outCacheLoc[tokenIdx] >= 0) + { + storeFp8ViaBf16(cacheRow(cache, 1, headIdx, tokenIdx), laneId * kElemsPerThread, elements); + } + return; + } + + sumOfSquares = warpReduceSum(sumOfSquares); + float rms_rcp = rsqrtf(sumOfSquares / static_cast(kHeadDim) + eps); + for (int i = 0; i < kElemsPerThread; i++) + { + int dim = laneId * kElemsPerThread + i; + float weight = isQ ? __bfloat162float(qWeight[dim]) : __bfloat162float(kWeight[dim]); + elements[i] *= rms_rcp * (useGemma ? (1.0f + weight) : weight); + } + + float elements2[kElemsPerThread]; + float cos_vals[kElemsPerThread]; + float sin_vals[kElemsPerThread]; + float const neg2_log2base_over_rd = -2.0f * __log2f(base) / static_cast(rotaryDim); + int const rd_mask = rotaryDim - 1; + bool const rd_is_pow2 = ((rotaryDim & rd_mask) == 0); + float const pos_id = static_cast(positionIds[tokenIdx]); + if constexpr (kInterleave) + { + for (int i = 0; i < kElemsPerThread; i++) + { + if (i % 2 == 0) + { + elements2[i] = -elements[i + 1]; + } + else + { + elements2[i] = elements[i - 1]; + } + int dim_idx = laneId * kElemsPerThread + i; + int half_dim = dim_idx / 2; + float freq = exp2f(static_cast(half_dim) * neg2_log2base_over_rd); + float theta = pos_id * freq; + __sincosf(theta, &sin_vals[i], &cos_vals[i]); + } + } + else + { + __syncwarp(); + int pairOffset = (rotaryDim / 2) / kElemsPerThread; + for (int i = 0; i < kElemsPerThread; i++) + { + elements2[i] = __shfl_xor_sync(0xffffffff, elements[i], pairOffset); + if (laneId < pairOffset) + { + elements2[i] = -elements2[i]; + } + int dim_idx = laneId * kElemsPerThread + i; + dim_idx = rd_is_pow2 ? ((dim_idx * 2) & rd_mask) : ((dim_idx * 2) % rotaryDim); + int half_dim = dim_idx / 2; + float freq = exp2f(static_cast(half_dim) * neg2_log2base_over_rd); + float theta = pos_id * freq; + __sincosf(theta, &sin_vals[i], &cos_vals[i]); + } + __syncwarp(); + } + + // attention_factor is 1 without YaRN: (x * cos + x2 * sin) * 1 rounds as the generic kernel's product does. + float const attention_factor = 1.0f; + if (rotaryDim == kHeadDim) + { + for (int i = 0; i < kElemsPerThread; i++) + { + elements[i] = (elements[i] * cos_vals[i] + elements2[i] * sin_vals[i]) * attention_factor; + } + } + else + { + for (int i = 0; i < kElemsPerThread; i++) + { + int dim_idx = laneId * kElemsPerThread + i; + if (dim_idx < rotaryDim) + { + elements[i] = (elements[i] * cos_vals[i] + elements2[i] * sin_vals[i]) * attention_factor; + } + } + } + storeFp8ViaBf16(qkvOut, offsetThread, elements); + if constexpr (kWriteCache) + { + // Rows past the live count (padded graph rows) own no slot, nor does a negative (sentinel) slot. + if (!isQ && tokenIdx < cache.numLiveTokens && cache.outCacheLoc[tokenIdx] >= 0) + { + storeFp8ViaBf16(cacheRow(cache, 0, headIdx, tokenIdx), laneId * kElemsPerThread, elements); + } + // The compact query [tokens, q heads, 128] the decode reads, beside the packed row. + if (isQ && cache.qCompact != nullptr) + { + storeFp8ViaBf16(cache.qCompact, + (static_cast(tokenIdx) * numHeadsQ + headIdx) * kHeadDim + laneId * kElemsPerThread, + elements); + } + } +} + +} // namespace + +cudaError_t launch(void const* qkvIn, void* qkvOut, int numTokens, int numHeadsQ, int numHeadsK, int numHeadsV, + int rotaryDim, float eps, void const* qWeight, void const* kWeight, float base, bool interleave, + int const* positionIds, bool useGemma, KvCacheView const* cache, cudaStream_t stream) +{ + int const totalWarps = numTokens * (numHeadsQ + numHeadsK + numHeadsV); + int const warpsPerBlock = kBlockSize / 32; + dim3 const grid((totalWarps + warpsPerBlock - 1) / warpsPerBlock); + auto const* in = static_cast<__nv_bfloat16 const*>(qkvIn); + auto* out = static_cast<__nv_fp8_e4m3*>(qkvOut); + auto const* qw = static_cast<__nv_bfloat16 const*>(qWeight); + auto const* kw = static_cast<__nv_bfloat16 const*>(kWeight); + KvCacheView const view = cache != nullptr ? *cache : KvCacheView{}; + auto const run = [&](auto kernel) -> cudaError_t + { + // The maximum shared-memory carveout lets the dependent's large-smem CTAs co-reside on the SMs this small + // kernel occupies once it releases them. + cudaError_t const err = cudaFuncSetAttribute( + kernel, cudaFuncAttributePreferredSharedMemoryCarveout, cudaSharedmemCarveoutMaxShared); + if (err != cudaSuccess) + { + return err; + } + cudaLaunchConfig_t config = {}; + config.gridDim = grid; + config.blockDim = dim3(kBlockSize); + config.dynamicSmemBytes = 0; + config.stream = stream; + cudaLaunchAttribute attribute[1]; + attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attribute[0].val.programmaticStreamSerializationAllowed = tensorrt_llm::common::getEnvEnablePDL() ? 1 : 0; + config.attrs = attribute; + config.numAttrs = 1; + return cudaLaunchKernelEx(&config, kernel, in, out, numHeadsQ, numHeadsK, numHeadsV, rotaryDim, eps, qw, kw, + base, positionIds, numTokens, useGemma, view); + }; + if (cache != nullptr) + { + return interleave ? run(denseQkNormRopeFp8Kernel) : run(denseQkNormRopeFp8Kernel); + } + return interleave ? run(denseQkNormRopeFp8Kernel) : run(denseQkNormRopeFp8Kernel); +} + +} // namespace denseQkNormRopeFp8 +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.h new file mode 100644 index 000000000000..1049033b07c9 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.h @@ -0,0 +1,68 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +namespace denseQkNormRopeFp8 +{ + +constexpr int kHeadDim = 128; + +//! A layer's FP8 K/V pool in the HND paged layout [pages, 2 (K, V), kv heads, tokens per block, 128] (element +//! strides), with the step's per-token cache slots (page * tokensPerBlock + within) for its first numLiveTokens rows. +struct KvCacheView +{ + __nv_fp8_e4m3* base = nullptr; + int const* outCacheLoc = nullptr; + long long pageStride = 0; + long long kvStride = 0; + long long headStride = 0; + long long tokenStride = 0; + int tokensPerBlock = 1; + int numLiveTokens = 0; + //! Optional: every row's Q also stored compact, [tokens, q heads, 128]. + __nv_fp8_e4m3* qCompact = nullptr; +}; + +//! The MiniMax-M3 TP4 dense layers' per-head QK RMSNorm + partial RoPE of a packed BF16 [tokens, (q + k + v) * 128] +//! row, written out of place as FP8 E4M3 with V copy-cast. Q and K are rounded to BF16 before E4M3, as the stock path +//! (the BF16 fused_qk_norm_rope output cast to FP8) rounds them. With cache, the K and V rows of the live tokens are +//! also stored into the paged pool at their slots (the bytes of the qkvOut rows). +//! +//! PDL (with TRTLLM_ENABLE_PDL): the launch is programmatic; griddepcontrol.wait is the kernel's first instruction, so +//! nothing is read or written before the producer has completed, and the kernel releases its dependent right after +//! it. The dependent may read only step-invariant data before its own griddepcontrol.wait. +cudaError_t launch(void const* qkvIn, void* qkvOut, int numTokens, int numHeadsQ, int numHeadsK, int numHeadsV, + int rotaryDim, float eps, void const* qWeight, void const* kWeight, float base, bool interleave, + int const* positionIds, bool useGemma, KvCacheView const* cache, cudaStream_t stream); + +} // namespace denseQkNormRopeFp8 +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.cu new file mode 100644 index 000000000000..2e67b3ae7ac1 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.cu @@ -0,0 +1,667 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.h" + +#include +#include +#include + +// One pass of the fused greedy Eagle3 draft loop of MiniMax-M3 (GB300, TP4, <= 8 requests per pass). +// +// One kernel replaces the draft lm_head GEMM, the logits all-gather / float cast / argmax of the in-tree greedy draft +// sampling and the draft loop's per-pass bookkeeping, at the in-tree precision: +// +// * The vocab-sharded BF16 lm_head weight is stored in mma.m16n8k16 A-fragment order (the same values): per 16-row +// tile and k16 step, 512 bytes; lane (g, j) holds 16 bytes = {row g k 2j..2j+1, row g+8 k 2j.., row g k 2j+8.., +// row g+8 k 2j+8..}. +// * Or, by default, in a lossless 12-bit coding of those fragments (75% of the bytes; the target head's coding, see +// targetHead.cu): per 16-row tile and unit of 4 k16 steps a lane reads three 16-byte words, the sign/mantissa bytes +// of steps 0-1, of steps 2-3, then 4 code words (exponent - base, 4 bits; 15 = an exception). The decoded values +// are the checkpoint's BF16 bits; an exception decodes to +0 and the segment's flush adds the exact product x * w +// from the tile's list (products of BF16 values are exact in FP32). +// * A persistent grid (one 16-warp CTA per SM) streams whole tiles over the full K with BF16 MMAs and FP32 +// accumulation; the BF16 head input rows sit in shared memory in B-fragment order, unchanged. +// * Each logit is rounded to BF16 (the in-tree logits are the BF16 GEMM output) and becomes a 64-bit key +// ordered(value) << 32 | ~global_index, with -0 taken as +0: the larger value wins and ties pick the lowest index, as +// torch.max / torch.argmax over the gathered logits do. Keys are max-reduced per row: warp shuffles, then shared +// memory, then atomicMax. +// * The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping +// (Lamport: 0 = empty, a key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the +// slots, and runs the pass's bookkeeping. Call n uses slot half n % 2. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +namespace draftStep +{ +namespace +{ +constexpr int kUnroll = 8; // k16 chunks in flight per warp +constexpr int kChunkBytes = 512; // one tile's k16 step: 32 lanes x 16 bytes + +// Coded units (4 k16 steps, 3 x 16 bytes per lane) in flight per warp: 3 up to 4 rows, 2 at 8 rows. +template +constexpr int codedUnroll() +{ + return MP >= 8 ? 2 : 3; +} + +__device__ __forceinline__ void mma_bf16( + float (&d)[4], uint32_t a0, uint32_t a1, uint32_t a2, uint32_t a3, uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.bf16.bf16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};\n" + : "+f"(d[0]), "+f"(d[1]), "+f"(d[2]), "+f"(d[3]) + : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(b0), "r"(b1)); +} + +// BF16 pair p (values 2p, 2p + 1) of a lane's 8 coded values: sign/mantissa bytes sm (4 per word), codes word cw. +__device__ __forceinline__ uint32_t coded_decode_pair(uint32_t sm_word, int byte0, uint32_t cw, int p, uint32_t base) +{ + uint32_t const b = __byte_perm(sm_word, 0u, byte0 == 0 ? 0x4140 : 0x4342); // {b0, 0, b1, 0} + uint32_t const q = (cw >> (4 * p)) & 0x000f000fu; + uint32_t const e = q * 128u + base; // exponent field of both halves + // Code 15 (an exception, sign/mantissa byte 0) decodes to +0. + return ((b & 0x007f007fu) | ((b << 8) & 0x80008000u) | e) & __vcmpne2(q, 0x000f000fu); +} + +// One coded unit (4 k16 steps) against the BF16 B fragments: steps alternate between the two accumulators, as the +// k16 steps of the plain BF16 head do. +template +__device__ __forceinline__ void chunk_mma_coded(uint4 const& sm01, uint4 const& sm23, uint4 const& cw, uint32_t base, + uint2 const* xs, int unit, int j, int g, bool b_active, float (&da)[4], float (&db)[4]) +{ + uint32_t const smw[8] = {sm01.x, sm01.y, sm01.z, sm01.w, sm23.x, sm23.y, sm23.z, sm23.w}; + uint32_t const cws[4] = {cw.x, cw.y, cw.z, cw.w}; +#pragma unroll + for (int ci = 0; ci < 4; ++ci) + { + uint2 b = make_uint2(0u, 0u); + if (b_active) + b = xs[((unit * 4 + ci) * 4 + j) * MP + g]; + uint32_t const w0 = coded_decode_pair(smw[2 * ci], 0, cws[ci], 0, base); + uint32_t const w1 = coded_decode_pair(smw[2 * ci], 2, cws[ci], 1, base); + uint32_t const w2 = coded_decode_pair(smw[2 * ci + 1], 0, cws[ci], 2, base); + uint32_t const w3 = coded_decode_pair(smw[2 * ci + 1], 2, cws[ci], 3, base); + if (ci & 1) + mma_bf16(db, w0, w1, w2, w3, b.x, b.y); + else + mma_bf16(da, w0, w1, w2, w3, b.x, b.y); + } +} + +// Exceptions of `tile` with k in [kb, ke): v += x * w (exact) for the owner lanes (row g or g + 8; tokens 2j, +// 2j + 1 < M), x from the staged B fragments. Warp-uniform (shuffles). +template +__device__ __forceinline__ void coded_exceptions( + CodedArgs const& cd, int tile, int kb, int ke, uint2 const* xs, int M, int lane, float (&v)[4]) +{ + int const g = lane >> 2, j = lane & 3; + int const e0 = __ldg(cd.exc_off + tile), e1 = __ldg(cd.exc_off + tile + 1); + for (int b0 = e0; b0 < e1; b0 += 32) + { + uint32_t pos = 0xffffffffu; + float2 wv = make_float2(0.f, 0.f); + if (b0 + lane < e1) + { + pos = __ldg(cd.exc_pos + b0 + lane); + wv = __ldg(cd.exc_w + b0 + lane); + } + int const cnt = min(32, e1 - b0); + for (int q = 0; q < cnt; ++q) + { + uint32_t const pq = __shfl_sync(0xffffffffu, pos, q); + float const w0 = __shfl_sync(0xffffffffu, wv.x, q); + int const r = static_cast(pq >> 16), k = static_cast(pq & 0xffffu); + if (k >= kb && k < ke && (r & 7) == g) + { + int const st = k >> 4, rr = k & 15; + bool const hi = r >= 8; +#pragma unroll + for (int q2 = 0; q2 < 2; ++q2) + { + int const n = 2 * j + q2; + if (n < M) + { + uint2 const bw = xs[(st * 4 + ((rr & 7) >> 1)) * MP + n]; + uint32_t const word = rr >= 8 ? bw.y : bw.x; + float const xv = __uint_as_float((rr & 1) ? (word & 0xffff0000u) : (word << 16)); + float const d = __fmul_rn(xv, w0); + if (hi) + v[2 + q2] += d; + else + v[q2] += d; + } + } + } + __syncwarp(); + } + } +} + +__device__ __forceinline__ uint4 ldg_stream(uint4 const* p) +{ + uint4 r; + asm volatile("ld.global.nc.L1::no_allocate.L2::256B.v4.u32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(r.x), "=r"(r.y), "=r"(r.z), "=r"(r.w) + : "l"(p)); + return r; +} + +// The argmax value of a logit: its BF16 rounding (the in-tree logits' dtype), -0 as +0. +__device__ __forceinline__ float key_value(float v) +{ + float const r = __bfloat162float(__float2bfloat16(v)); + return r == 0.f ? 0.f : r; +} + +__device__ __forceinline__ unsigned long long make_key(float v, uint32_t index) +{ + uint32_t u = __float_as_uint(v); + u = (u & 0x80000000u) ? ~u : (u | 0x80000000u); + return (static_cast(u) << 32) | static_cast(0xFFFFFFFFu - index); +} + +__device__ __forceinline__ unsigned long long shfl_max(unsigned long long k, int o) +{ + unsigned long long const ok = __shfl_xor_sync(0xffffffffu, k, o); + return ok > k ? ok : k; +} + +// One k16 step of a tile: the lane's A fragment w against the activation B fragment of step `chunk` (rows g < MP). +// Even and odd steps accumulate separately (two independent MMA chains); the tile's sum is da + db. +template +__device__ __forceinline__ void chunk_mma( + uint4 const& w, uint2 const* xs, int chunk, int j, int g, bool b_active, float (&da)[4], float (&db)[4]) +{ + uint2 b = make_uint2(0u, 0u); + if (b_active) + b = xs[(chunk * 4 + j) * MP + g]; + if (chunk & 1) + mma_bf16(db, w.x, w.y, w.z, w.w, b.x, b.y); + else + mma_bf16(da, w.x, w.y, w.z, w.w, b.x, b.y); +} + +// A warp's pieces of the tiles it shares with other warps (its first and last local tile), kept for the warp-order sum. +struct Pieces +{ + float4 head = make_float4(0.f, 0.f, 0.f, 0.f); + float4 tail = make_float4(0.f, 0.f, 0.f, 0.f); +}; + +// Whether the warp's chunk range [q_begin, q_end) covers the CTA's whole range of local tile tl (the CTA streams +// chunks [lo, lo + span) of its local tiles). +__device__ __forceinline__ bool covers_tile(int tl, int kchunks, int lo, int span, int q_begin, int q_end) +{ + return q_begin <= max(tl * kchunks, lo) && q_end >= min((tl + 1) * kchunks, lo + span); +} + +// Local tile t's sum (D-FIX): the accumulator when one warp streamed the CTA's whole range of it, else its pieces +// from the warps that streamed it, added in warp order from 0 (each warp's first tile is its head piece, any later +// one its tail piece). Deterministic: the order is fixed by the launch shape, and with two pieces it is the same +// sum as the atomics' (two addends from 0 commute). +__device__ __forceinline__ float4 ordered_tile( + float4 const* acc_t, float4 const* stage, int t, int kchunks, int lo, int span, int lane) +{ + int const r_lo = max(t * kchunks, lo); + int const r_last = min((t + 1) * kchunks, lo + span) - 1; + int w_lo = 0, w_hi = 0; + for (int w = 1; w < kWarps; ++w) + { + int const qb = lo + w * span / kWarps; + if (qb <= r_lo) + w_lo = w; + if (qb <= r_last) + w_hi = w; + } + if (w_lo == w_hi) + return acc_t[lane]; + float4 s = make_float4(0.f, 0.f, 0.f, 0.f); + for (int w = w_lo; w <= w_hi; ++w) + { + int const qb = lo + w * span / kWarps; + float4 const x = stage[(w * 2 + (qb / kchunks == t ? 0 : 1)) * 32 + lane]; + s.x += x.x; + s.y += x.y; + s.z += x.z; + s.w += x.w; + } + return s; +} + +// Add a warp's segment of a tile into the CTA's accumulator (with a coded head, the segment's exceptions first); +// with `fixed` (D-FIX) a shared tile's piece goes to `pc` instead (ordered_tile adds it). +template +__device__ __forceinline__ void flush_tile(float4* acc, float (&da)[4], float (&db)[4], Params const& p, int tile, + int kb, int ke, uint2 const* xs, int lane, Pieces& pc, bool fixed, bool whole, bool first) +{ + float v[4] = {da[0] + db[0], da[1] + db[1], da[2] + db[2], da[3] + db[3]}; + if constexpr (CODED) + coded_exceptions(p.cd, tile, kb, ke, xs, p.M, lane, v); + if (fixed) + { + // A piece covering the CTA's whole range of the tile is its only addend (0 + v, as the atomic gave); a + // warp's other pieces (at most its first and last tile) wait in registers for the warp-order sum. + if (whole) + *acc = make_float4(0.f + v[0], 0.f + v[1], 0.f + v[2], 0.f + v[3]); + else if (first) + pc.head = make_float4(v[0], v[1], v[2], v[3]); + else + pc.tail = make_float4(v[0], v[1], v[2], v[3]); + } + else + { + float* a = reinterpret_cast(acc); + atomicAdd(a + 0, v[0]); + atomicAdd(a + 1, v[1]); + atomicAdd(a + 2, v[2]); + atomicAdd(a + 3, v[3]); + } +#pragma unroll + for (int e = 0; e < 4; ++e) + { + da[e] = 0.f; + db[e] = 0.f; + } +} + +// Balanced streaming GEMV over the fragment-ordered weight, persistent grid of 16-warp CTAs. CTA c owns tiles +// [t_begin, t_end); its (tile, chunk) space is split into 16 equal contiguous ranges, one per warp, so every warp +// streams the same number of bytes. A tile straddles at most two warps; their partial sums are added into shared +// memory (two addends: the order does not change the sum). MP = rows padded to 1 / 2 / 4 / 8. A chunk is one k16 +// step of the BF16 head, or one coded unit (4 k16 steps) of the coded head (CODED: no shared-memory prologue). +template +__global__ void __launch_bounds__(kWarps * 32, 1) + draft_step_kernel(Params const p, int kchunks, int pf_chunks, int pf_off) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(16) uint2 xs[]; // [K/16 steps][4 (j)][MP (g)], then the weight prologue + __shared__ float4 acc_s[kMaxTilesPerCta][32]; + __shared__ unsigned long long red[kWarps][kMaxTokens]; + __shared__ __align__(8) unsigned long long pf_bar[kWarps]; + __shared__ bool s_last; + __shared__ int s_tok[kMaxTokens]; + + int const warp = threadIdx.x >> 5; + int const lane = threadIdx.x & 31; + int const g = lane >> 2; + int const j = lane & 3; + constexpr int kChunkK = CODED ? 64 : 16; // k per chunk + constexpr int kWords = CODED ? 3 : 1; // 16-byte words per lane and chunk + constexpr int KU = CODED ? codedUnroll() : kUnroll; + int const K = kchunks * kChunkK; + int const M = p.M; + int const t_begin = static_cast((static_cast(blockIdx.x) * p.n_tiles) / gridDim.x); + int const t_end = static_cast((static_cast(blockIdx.x + 1) * p.n_tiles) / gridDim.x); + int const n_local = t_end - t_begin; + int const span = n_local * kchunks; + int const q_begin = warp * span / kWarps; + int const q_end = (warp + 1) * span / kWarps; + long long const base = static_cast(t_begin) * kchunks; // first chunk of this CTA + uint4 const* const wf = p.wf; + + // Weights are static: start this warp's stream before waiting on the producer of x. The warp's first pf_chunks + // chunks come in by one bulk copy into shared memory (dynamic offset pf_off), which fills the grid-dependency + // wait and the staging; the register ring starts after them. + int const n_pf = CODED ? 0 : min(pf_chunks, q_end - q_begin); + uint4* const wpf = reinterpret_cast(reinterpret_cast(xs) + pf_off) + warp * pf_chunks * 32; + if (n_pf > 0 && lane == 0) + { + uint32_t const bar = static_cast(__cvta_generic_to_shared(&pf_bar[warp])); + uint32_t const bytes = static_cast(n_pf) * kChunkBytes; + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(bar) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(bar), "r"(bytes) : "memory"); + asm volatile("cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes [%0], [%1], %2, [%3];" ::"r"( + static_cast(__cvta_generic_to_shared(wpf))), + "l"(wf + (base + q_begin) * 32), "r"(bytes), "r"(bar) + : "memory"); + } + __syncwarp(); + int const q_reg = q_begin + n_pf; + uint4 buf[KU][kWords]; +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q_reg + u < q_end) + { +#pragma unroll + for (int l = 0; l < kWords; ++l) + buf[u][l] = ldg_stream(wf + ((base + q_reg + u) * kWords + l) * 32 + lane); + } + } + for (int i = threadIdx.x; i < n_local * 32; i += blockDim.x) + acc_s[i / 32][i % 32] = make_float4(0.f, 0.f, 0.f, 0.f); + + asm volatile("griddepcontrol.wait;" ::: "memory"); + + // Stage the head input rows in B-fragment order: item (k16 step S, row gg) = two 16-byte loads of x, four + // fragment words {x[2jj], x[2jj + 1]}, {x[2jj + 8], x[2jj + 9]} (BF16 pairs, moved unchanged). Rows >= M are 0. + constexpr int kSU = MP >= 4 ? 4 : 2; + int const n_items = (K / 16) * MP; + for (int i0 = threadIdx.x; i0 < n_items; i0 += kSU * blockDim.x) + { + uint4 xv[kSU][2]; +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + int const gg = i % MP; + int const S = i / MP; + xv[u][0] = make_uint4(0u, 0u, 0u, 0u); + xv[u][1] = make_uint4(0u, 0u, 0u, 0u); + if (i < n_items && gg < M) + { + long long const xr = p.x_rows != nullptr ? p.x_rows[gg] : gg; + uint4 const* xp = reinterpret_cast(p.x + xr * K + 16 * S); + xv[u][0] = xp[0]; + xv[u][1] = xp[1]; + } + } +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + if (i >= n_items) + continue; + int const gg = i % MP; + int const S = i / MP; + uint32_t const lo[4] = {xv[u][0].x, xv[u][0].y, xv[u][0].z, xv[u][0].w}; + uint32_t const hi[4] = {xv[u][1].x, xv[u][1].y, xv[u][1].z, xv[u][1].w}; +#pragma unroll + for (int jj = 0; jj < 4; ++jj) + xs[(S * 4 + jj) * MP + gg] = make_uint2(lo[jj], hi[jj]); + } + } + __syncthreads(); + + bool const b_active = g < MP; + float da[4] = {0.f, 0.f, 0.f, 0.f}; + float db[4] = {0.f, 0.f, 0.f, 0.f}; + int tl = q_begin / kchunks; // local tile of the current accumulator + int c = q_begin - tl * kchunks; // chunk within that tile + int cs = c; // first chunk of the warp's segment of that tile + int const tl0 = tl; // the warp's first local tile (its head piece) + Pieces pc; + if (n_pf > 0) + { + uint32_t const bar = static_cast(__cvta_generic_to_shared(&pf_bar[warp])); + asm volatile( + "{\n .reg .pred p;\n PF_WAIT%=:\n mbarrier.try_wait.parity.shared::cta.b64 p, [%0], 0;\n" + " @!p bra PF_WAIT%=;\n}\n" ::"r"(bar) + : "memory"); + for (int q = q_begin; q < q_reg; ++q) + { + chunk_mma(wpf[(q - q_begin) * 32 + lane], xs, c, j, g, b_active, da, db); + if (++c == kchunks) + { + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, K, xs, lane, pc, + p.fixedOrder, covers_tile(tl, kchunks, 0, span, q_begin, q_end), tl == tl0); + c = 0; + cs = 0; + ++tl; + } + } + } + for (int q = q_reg; q < q_end; q += KU) + { + uint4 nxt[KU][kWords]; +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q + KU + u < q_end) + { +#pragma unroll + for (int l = 0; l < kWords; ++l) + nxt[u][l] = ldg_stream(wf + ((base + q + KU + u) * kWords + l) * 32 + lane); + } + } +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q + u < q_end) + { + if constexpr (CODED) + chunk_mma_coded(buf[u][0], buf[u][kWords > 1 ? 1 : 0], buf[u][kWords > 2 ? 2 : 0], p.cd.base, + xs, c, j, g, b_active, da, db); + else + chunk_mma(buf[u][0], xs, c, j, g, b_active, da, db); + if (++c == kchunks) + { + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, K, xs, lane, pc, + p.fixedOrder, covers_tile(tl, kchunks, 0, span, q_begin, q_end), tl == tl0); + c = 0; + cs = 0; + ++tl; + } + } + } +#pragma unroll + for (int u = 0; u < KU; ++u) +#pragma unroll + for (int l = 0; l < kWords; ++l) + buf[u][l] = nxt[u][l]; + } + if (c != 0) + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, c * kChunkK, xs, lane, pc, + p.fixedOrder, covers_tile(tl, kchunks, 0, span, q_begin, q_end), tl == tl0); + __syncthreads(); + // D-FIX: stage every warp's head / tail piece. xs and the weight prologue are dead after the barrier above (the + // launcher sizes them to hold the 16 x 2 x 32 pieces). + float4* const stage = reinterpret_cast(xs); + if (p.fixedOrder) + { + stage[(warp * 2) * 32 + lane] = pc.head; + stage[(warp * 2 + 1) * 32 + lane] = pc.tail; + __syncthreads(); + } + + // Accumulator lane (g, j) of a tile: rows g / g + 8 (x, y / z, w) of tokens 2j / 2j + 1 (x, z / y, w). + unsigned long long best0 = 0ull, best1 = 0ull; // tokens 2j, 2j + 1 + for (int t = warp; t < n_local; t += kWarps) + { + float4 const v = p.fixedOrder ? ordered_tile(acc_s[t], stage, t, kchunks, 0, span, lane) : acc_s[t][lane]; + int const r0 = (t_begin + t) * 16 + g; + uint32_t const i0 = static_cast(p.vocab_offset + r0); + unsigned long long k0 = make_key(key_value(v.x), i0); + unsigned long long const k0b = make_key(key_value(v.z), i0 + 8); + unsigned long long k1 = make_key(key_value(v.y), i0); + unsigned long long const k1b = make_key(key_value(v.w), i0 + 8); + k0 = k0b > k0 ? k0b : k0; + k1 = k1b > k1 ? k1b : k1; + best0 = k0 > best0 ? k0 : best0; + best1 = k1 > best1 ? k1 : best1; + } +#pragma unroll + for (int o = 4; o < 32; o <<= 1) + { + best0 = shfl_max(best0, o); + best1 = shfl_max(best1, o); + } + asm volatile("griddepcontrol.launch_dependents;"); + if (lane < 4) + { + red[warp][2 * lane] = best0; + red[warp][2 * lane + 1] = best1; + } + __syncthreads(); + if (threadIdx.x < M) + { + unsigned long long b = 0ull; +#pragma unroll + for (int w = 0; w < kWarps; ++w) + b = red[w][threadIdx.x] > b ? red[w][threadIdx.x] : b; + if (b != 0ull) + atomicMax(p.keys + threadIdx.x, b); + } + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + unsigned int const prev = atomicAdd(p.counters, 1u); + s_last = prev == gridDim.x - 1; + } + __syncthreads(); + if (!s_last) + return; + + __threadfence(); + unsigned int const call = *reinterpret_cast(p.counters + 1); + int const half = static_cast(call & 1u); + if (threadIdx.x < M) + { + int const t = threadIdx.x; + unsigned long long const mine = atomicExch(p.keys + t, 0ull); + unsigned long long* slot_mc = p.ws_mc + (half * kMaxTokens + t) * p.world; + unsigned long long* slot_uc = p.ws_uc + (half * kMaxTokens + t) * p.world; + slot_mc[p.rank] = mine; + unsigned long long best = 0ull; + for (int r = 0; r < p.world; ++r) + { + unsigned long long v; + do + { + v = *reinterpret_cast(slot_uc + r); + } while (v == 0ull); + best = v > best ? v : best; + slot_uc[r] = 0ull; + } + s_tok[t] = static_cast(0xFFFFFFFFu - static_cast(best & 0xFFFFFFFFull)); + if (p.out_keys != nullptr) + p.out_keys[t] = best; + } + __syncthreads(); + if (threadIdx.x == 0) + { + p.counters[0] = 0u; + p.counters[1] = call + 1u; + } + if (threadIdx.x < M) + { + StepArgs const& st = p.st; + int const b = threadIdx.x; + int const tok = s_tok[b]; + st.draft_out[b * st.draft_stride + st.step] = tok; + st.ids_next[b] = tok; + long long const r = p.x_rows != nullptr ? p.x_rows[b] : b; + st.pos_next[b] = st.pos_src[r] + 1; + if (st.step == 0 && st.num_acc != nullptr) + { + st.kv_lens[b] -= st.draft_len - st.num_acc[b]; + st.seq_lens[b] = 1; + } + else if (st.step > 0) + { + st.kv_lens[b] += 1; + } + } + if (p.hasFold) + { + // The fold tail (H-R2-3) after this pass's tokens and bookkeeping: the barrier makes this CTA's writes above + // visible to warp 0; every earlier operand was written by grids this one waited on. All CTAs triggered their + // dependents before the exchange, which is safe: a dependent's griddepcontrol.wait waits for this whole grid + // (this tail included) to complete, so only a dependent that read the tail's outputs before its own wait + // could race, and none does (the next step's readers run in the next graph replay). + __syncthreads(); + if (threadIdx.x < 32) + { + foldTailWarp(p.fold, static_cast(threadIdx.x)); + } + } +#endif +} + +template +cudaError_t launch(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + auto kern = draft_step_kernel; + size_t const xs_bytes = static_cast(K / 16) * 4 * MP * sizeof(uint2); + // Shared-memory weight prologue: chunks per warp that fit next to the kernel's static + dynamic shared memory. + static int const sStaticSmem = [] + { + cudaFuncAttributes fa{}; + return cudaFuncGetAttributes(&fa, draft_step_kernel) == cudaSuccess + ? static_cast(fa.sharedSizeBytes) + : -1; + }(); + static int const sMaxSmemPerBlock = [] + { + int dev = 0, maxSmem = 0; + cudaGetDevice(&dev); + cudaDeviceGetAttribute(&maxSmem, cudaDevAttrMaxSharedMemoryPerBlockOptin, dev); + return maxSmem; + }(); + int pf = 0; + if (!CODED && sStaticSmem >= 0) + { + long long const room + = static_cast(sMaxSmemPerBlock) - sStaticSmem - static_cast(xs_bytes) - 1024; + long long const want = static_cast(kPrefetchKb) * 1024; + pf = static_cast(std::max(0ll, std::min(room, want)) / (kWarps * kChunkBytes)); + } + // At least the D-FIX piece staging (16 warps x head / tail x 32 lanes of float4), which reuses this space. + size_t const smem = std::max(xs_bytes + static_cast(pf) * kWarps * kChunkBytes, + static_cast(kWarps) * 2 * 32 * sizeof(float4)); + cudaError_t err = cudaFuncSetAttribute(kern, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smem)); + if (err != cudaSuccess) + return err; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(static_cast(grid)); + cfg.blockDim = dim3(kWarps * 32); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, kern, p, K / (CODED ? 64 : 16), pf, static_cast(xs_bytes)); +} + +template +cudaError_t dispatchRows(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + if (p.M <= 1) + return launch<1, CODED>(p, K, grid, pdl, stream); + if (p.M <= 2) + return launch<2, CODED>(p, K, grid, pdl, stream); + if (p.M <= 4) + return launch<4, CODED>(p, K, grid, pdl, stream); + return launch<8, CODED>(p, K, grid, pdl, stream); +} + +} // namespace + +cudaError_t invokeDraftStep(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + bool const coded = p.cd.exc_off != nullptr; + if (p.M < 1 || p.M > kMaxTokens || K % (coded ? 64 : 16) != 0 || grid < 1) + return cudaErrorInvalidValue; + return coded ? dispatchRows(p, K, grid, pdl, stream) : dispatchRows(p, K, grid, pdl, stream); +} + +} // namespace draftStep +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.h new file mode 100644 index 000000000000..85296a3fd6d7 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.h @@ -0,0 +1,107 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +//! One pass of the fused greedy Eagle3 draft loop of the MiniMax-M3 NVFP4 TP4 decode path (sm_103a): the BF16 +//! vocab-sharded draft head, the cross-rank argmax and the draft loop's per-pass bookkeeping in one kernel. See +//! draftStep.cu. +namespace draftStep +{ + +constexpr int kMaxTokens = 8; // head rows (requests) of one launch +constexpr int kWarps = 16; // warps per CTA +constexpr int kMaxTilesPerCta = 32; // 16-row weight tiles one CTA can own +//! Shared-memory weight prologue per CTA, in KB (capped by the free shared memory). +constexpr int kPrefetchKb = 112; + +// Draft-pass epilogue (last CTA, one thread per request row b): stores the token as draft token `step`, stages the +// next pass's input id and position, and advances the draft KV lengths the way the in-tree draft loop does between +// passes (pass 0 with num_acc: drop the rejected verify rows and switch to one token per request; later passes: +1). +struct StepArgs +{ + int* draft_out; // [nreq, draft_stride]: column `step` receives the token + int draft_stride; + int step; + int* ids_next; // [nreq] + int const* pos_src; // positions of the rows the head read (indexed like x) + int* pos_next; // [nreq] = pos_src[x_rows[b] or b] + 1 + int* kv_lens; // [nreq] draft KV lengths, updated in place + int const* num_acc; // pass 0: accepted counts (nullptr: no rollback) + int* seq_lens; // pass 0: set to 1 + int draft_len; +}; + +// The 12-bit coded head (see draftStep.cu): the per-tile exception lists and the exponent base. exc_off == nullptr: +// the head is the plain BF16 fragment-ordered copy. +struct CodedArgs +{ + int const* exc_off; // [n_tiles + 1] exception list offsets + uint32_t const* exc_pos; // row in tile << 16 | k, per exception + float2 const* exc_w; // {w, 0} per exception: the exact weight + uint32_t base; // exponent base << 7 in both BF16 halves (base << 7 | base << 23) +}; + +// Everything a draft-step launch takes (see invokeDraftStep). +struct Params +{ + __nv_bfloat16 const* x; // [*, K] head input rows + long long const* x_rows; // optional [M]: head input row of request b (pass 0: the last accepted verify row) + int M; // requests, <= kMaxTokens + uint4 const* wf; // [n_tiles][K / 16][32 lanes] 16-byte BF16 A fragments, or the 12-bit coded stream + // [n_tiles][K / 64][3][32 lanes] when cd.exc_off is set (see draftStep.cu) + int n_tiles; + int vocab_offset; // global id of this rank's first head row + unsigned long long* keys; + unsigned int* counters; + unsigned long long* ws_uc; // this rank's exchange slots [2][kMaxTokens][world] + unsigned long long* ws_mc; // their multicast address + int rank; + int world; + unsigned long long* out_keys; // optional [M]: the winning global key of each row + StepArgs st; + CodedArgs cd; + // The step's forward-tail store + next-step staging (foldTail.h), run by the last CTA after this pass's + // tokens and bookkeeping: set on the drafter's last pass only (dispatch.DRAFT_STEP_FOLD_TAIL). + FoldTailParams fold{}; + bool hasFold = false; + // D-FIX: sum the pieces of a tile that several warps stream in warp order (deterministic) instead of by + // shared-memory atomics in arrival order (dispatch.DRAFT_STEP_FIXED_ORDER). + bool fixedOrder = false; +}; + +//! Launch the draft step over `grid` 512-thread CTAs (K = the hidden size, a multiple of 16; of 64 for a coded head). +cudaError_t invokeDraftStep(Params const& p, int K, int grid, bool pdl, cudaStream_t stream); + +} // namespace draftStep +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.cu new file mode 100644 index 000000000000..a506548c1aec --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.cu @@ -0,0 +1,189 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// The TP4 drafter layer's generation preprocess in one launch. The rotation is applyBiasRopeUpdateKVCacheV2's GPT-NeoX +// path at BF16 (apply_rotary_embedding_gptneox: cos * x + sin * pair in fp32, the first rotary half's sin negated, +// dims past the rotary width with (cos, sin) = (1, 0), rounded to BF16); the K / V store is its unit-scale E4M3 +// store (__nv_fp8x2_e4m3 from BF16 pairs), which is also the query's E4M3: torch's CUDA cast of the BF16 query +// saturates the same way (448, not NaN). + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +namespace drafterRopeFp8Kv +{ +namespace +{ + +constexpr int kElemsPerThread = kHeadDim / 32; +constexpr int kBlockSize = 256; + +template +__device__ __forceinline__ float rotate(float c, float x, float s, float pair) +{ + if constexpr (kRopeMode == 0) + { + return __fadd_rn(__fmul_rn(c, x), __fmul_rn(s, pair)); + } + else if constexpr (kRopeMode == 1) + { + return __fmaf_rn(c, x, __fmul_rn(s, pair)); + } + else + { + return __fmaf_rn(s, pair, __fmul_rn(c, x)); + } +} + +__device__ __forceinline__ void storeE4m3Pairs(__nv_fp8_e4m3* dst, __nv_bfloat16 const (&v)[kElemsPerThread]) +{ +#pragma unroll + for (int i = 0; i < kElemsPerThread; i += 2) + { + __nv_bfloat162 pair; + pair.x = v[i]; + pair.y = v[i + 1]; + __nv_fp8x2_e4m3 const packed(pair); + reinterpret_cast<__nv_fp8x2_storage_t*>(dst)[i / 2] = packed.__x; + } +} + +template +__global__ void __launch_bounds__(kBlockSize) drafterRopeFp8KvKernel(Params const p) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); +#endif + int const lane = threadIdx.x % 32; + int const warp = (blockIdx.x * blockDim.x + threadIdx.x) / 32; + int const slots = p.numHeadsQ + p.numHeadsKv; + int const token = warp / slots; + if (token >= p.numTokens) + { + return; + } + int const slot = warp % slots; + bool const isQ = slot < p.numHeadsQ; + int const head = isQ ? slot : slot - p.numHeadsQ; + int const request = token / p.qLen; + int const row = token % p.qLen; + int const cacheLen = p.seqLens[request]; + int const kvPos = cacheLen - p.qLen + row; + int const ropePos + = p.specPositionOffsets != nullptr ? p.specPositionOffsets[request * p.qLen + row] + cacheLen - p.qLen : kvPos; + int const rowWidth = (p.numHeadsQ + 2 * p.numHeadsKv) * kHeadDim; + __nv_bfloat16 const* src + = p.qkv + static_cast(token) * rowWidth + (isQ ? head * kHeadDim : (p.numHeadsQ + head) * kHeadDim); + int const half = p.rotaryDim / 2; + float2 const* coef = p.cosSin + static_cast(ropePos) * half; + + __nv_bfloat16 rotated[kElemsPerThread]; +#pragma unroll + for (int i = 0; i < kElemsPerThread; ++i) + { + int const d = lane * kElemsPerThread + i; + bool const firstHalf = d < half; + float const x = __bfloat162float(src[d]); + float const pair = __bfloat162float(src[firstHalf ? d + half : d - half]); + float2 const cs = d < p.rotaryDim ? coef[firstHalf ? d : d - half] : make_float2(1.0f, 0.0f); + float const s = firstHalf ? -cs.y : cs.y; + rotated[i] = __float2bfloat16(rotate(cs.x, x, s, pair)); + } + + if (isQ) + { + storeE4m3Pairs( + p.qOut + (static_cast(token) * p.numHeadsQ + head) * kHeadDim + lane * kElemsPerThread, rotated); + return; + } + + int const block = kvPos / p.tokensPerBlock; + long long const inPage = (static_cast(head) * p.tokensPerBlock + kvPos % p.tokensPerBlock) * kHeadDim + + lane * kElemsPerThread; + long long const pageElems = static_cast(p.numHeadsKv) * p.tokensPerBlock * kHeadDim; + int const* table = p.blockTable + request * p.blockTableRequestStride; + long long const kPage = table[block]; + long long const vPage = table[p.blockTableKvStride + block]; + storeE4m3Pairs(p.pool + kPage * pageElems + inPage, rotated); + + __nv_bfloat16 const* vSrc = p.qkv + static_cast(token) * rowWidth + + (p.numHeadsQ + p.numHeadsKv + head) * kHeadDim + lane * kElemsPerThread; + __nv_bfloat16 v[kElemsPerThread]; +#pragma unroll + for (int i = 0; i < kElemsPerThread; ++i) + { + v[i] = vSrc[i]; + } + storeE4m3Pairs(p.pool + vPage * pageElems + inPage, v); +} + +//! With PDL and maxSharedCarveout, the maximum shared-memory carveout lets the full-context decode's large-smem CTAs +//! (the dependent this kernel releases right after its wait) co-reside on the SMs this small kernel occupies. Set once +//! per instantiation. +template +cudaError_t launchWithMaxSharedCarveout( + cudaLaunchConfig_t const& config, Params const& params, bool pdl, bool maxSharedCarveout) +{ + auto const kernel = drafterRopeFp8KvKernel; + if (pdl && maxSharedCarveout) + { + static cudaError_t const carveout = cudaFuncSetAttribute( + kernel, cudaFuncAttributePreferredSharedMemoryCarveout, cudaSharedmemCarveoutMaxShared); + if (carveout != cudaSuccess) + { + return carveout; + } + } + return cudaLaunchKernelEx(&config, kernel, params); +} + +} // namespace + +cudaError_t launch(Params const& params, bool pdl, bool maxSharedCarveout, cudaStream_t stream) +{ + int const warps = params.numTokens * (params.numHeadsQ + params.numHeadsKv); + int const warpsPerBlock = kBlockSize / 32; + cudaLaunchConfig_t config = {}; + config.gridDim = dim3((warps + warpsPerBlock - 1) / warpsPerBlock); + config.blockDim = dim3(kBlockSize); + config.dynamicSmemBytes = 0; + config.stream = stream; + cudaLaunchAttribute attribute[1]; + attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attribute[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + config.attrs = attribute; + config.numAttrs = 1; + switch (params.ropeMode) + { + case 0: return launchWithMaxSharedCarveout<0>(config, params, pdl, maxSharedCarveout); + case 1: return launchWithMaxSharedCarveout<1>(config, params, pdl, maxSharedCarveout); + case 2: return launchWithMaxSharedCarveout<2>(config, params, pdl, maxSharedCarveout); + default: return cudaErrorInvalidValue; + } +} + +} // namespace drafterRopeFp8Kv +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.h new file mode 100644 index 000000000000..2a39d03d9d76 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.h @@ -0,0 +1,79 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +namespace drafterRopeFp8Kv +{ + +constexpr int kHeadDim = 128; + +struct Params +{ + //! [tokens, (numHeadsQ + 2 * numHeadsKv) * 128] BF16: the drafter layer's packed Q, K, V rows. + __nv_bfloat16 const* qkv; + //! [positions, rotaryDim / 2] (cos, sin) pairs: the layer's rotary_cos_sin table. + float2 const* cosSin; + //! [requests] cache lengths including this step's rows. + int const* seqLens; + //! [requests, qLen] spec-decoding position offsets of a multi-token pass, or null. + int const* specPositionOffsets; + //! [tokens, numHeadsQ, 128] E4M3 query out. + __nv_fp8_e4m3* qOut; + //! Base of the layer's K / V sub-pages, [*, numHeadsKv, tokensPerBlock, 128] E4M3. + __nv_fp8_e4m3* pool; + //! [requests, 2, maxBlocks] sub-page indices: K in row 0, V in row 1. + int const* blockTable; + long long blockTableRequestStride; + long long blockTableKvStride; + int numTokens; + int qLen; + int numHeadsQ; + int numHeadsKv; + int rotaryDim; + int tokensPerBlock; + //! How the rotation's two products are summed: 0 two products then the add, 1 fma(cos, x, sin * pair), + //! 2 fma(sin, pair, cos * x). + int ropeMode; +}; + +//! The MiniMax-M3 TP4 Eagle3 drafter layer's generation preprocess: GPT-NeoX RoPE from the cos/sin table on Q and K, +//! rounded to BF16 (the in-tree preprocess's q output and cache input), then Q, K and V to saturating E4M3: K / V +//! as the in-tree preprocess stores them at unit KV scale, Q as torch's CUDA cast of the BF16 query. Rows map to +//! (request token / qLen, row token % qLen) at cache position seqLen - qLen + row. +//! +//! PDL (pdl): the launch is programmatic; griddepcontrol.wait is the kernel's first instruction and the kernel +//! releases its dependent right after it. The dependent may read only step-invariant data before its own wait. +cudaError_t launch(Params const& params, bool pdl, bool maxSharedCarveout, cudaStream_t stream); + +} // namespace drafterRopeFp8Kv +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.cu new file mode 100644 index 000000000000..e6a4f0da2039 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.cu @@ -0,0 +1,64 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +namespace +{ + +// Row i of the step (lane c = column), as the Triton fold tail's program i: the store's padded rows, then the next +// step's inputs read from them (zero past each source width). +__global__ void __launch_bounds__(32) foldTailKernel(FoldTailParams const p) +{ + int const i = blockIdx.x; + if (i >= *p.ctrl) + { + return; + } + foldTailRow(p, i, static_cast(threadIdx.x)); +} + +__global__ void __launch_bounds__(32) foldTailOneWarpKernel(FoldTailParams const p) +{ + foldTailWarp(p, static_cast(threadIdx.x)); +} + +} // namespace + +cudaError_t invokeFoldTail(FoldTailParams const& p, int32_t rows, cudaStream_t stream) +{ + foldTailKernel<<>>(p); + return cudaGetLastError(); +} + +cudaError_t invokeFoldTailOneWarp(FoldTailParams const& p, cudaStream_t stream) +{ + foldTailOneWarpKernel<<<1, 32, 0, stream>>>(p); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.h new file mode 100644 index 000000000000..54a58b0569e0 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.h @@ -0,0 +1,129 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +//! Operands of the speculative step's forward-tail store and next-step staging (kernels/m3_fold_tail.py). Every tensor +//! is int32 except `slots` (int64); strides are in elements. The store views are [slots, width], the sources +//! [rows, width]. +struct FoldTailParams +{ + int32_t const* ctrl; // ctrl[0]: the rows the step armed (0: write nothing) + int64_t const* slots; + int32_t const* srcNew; + int64_t sn0, sn1; + int32_t snW; + int32_t* dstNew; + int64_t dn0, dn1; + int32_t dnW; + int32_t const* srcNext; + int64_t sx0, sx1; + int32_t sxW; + int32_t* dstNext; + int64_t dx0, dx1; + int32_t dxW; + int32_t const* srcDraft; + int64_t sd0, sd1; + int32_t sdW; + int32_t* dstDraft; + int64_t dd0, dd1; + int32_t ddW; + int32_t const* srcLens; + int32_t* dstLens; + int32_t* ids; // [rows * T] + int32_t* draftIds; // [rows * (T - 1)] + int32_t* posOff; // [rows * T] + int32_t* kvOff; // [rows] + int32_t tokensPerStep; +}; + +//! One CTA of 32 lanes per row, `rows` CTAs; the first ctrl[0] rows write. Widths and tokensPerStep must be <= 32. +cudaError_t invokeFoldTail(FoldTailParams const& p, int32_t rows, cudaStream_t stream); + +//! The same writes from ONE warp (one CTA of 32 lanes looping over the rows, foldTailWarp): the in-kernel form a caller +//! such as the drafter's last-pass draft_step runs in its last CTA; launched alone it is that form's oracle. +cudaError_t invokeFoldTailOneWarp(FoldTailParams const& p, cudaStream_t stream); + +#ifdef __CUDACC__ +//! Row i of the tail at lane c (0..31), the work of foldTailKernel's CTA i, for a caller that runs the tail inside its +//! own kernel (e.g. the drafter's last-pass draft_step in its last CTA, after its final tokens are written and fenced). +//! The caller runs rows i < *p.ctrl (one warp per row, or one warp looping over the rows); same code as the kernel, so +//! the writes are bit for bit the standalone tail's. +__device__ __forceinline__ void foldTailRow(FoldTailParams const& p, int i, int c) +{ + int64_t const s = p.slots[i]; + int32_t const nw = c < p.snW ? p.srcNew[i * p.sn0 + c * p.sn1] : 0; + if (c < p.dnW) + { + p.dstNew[s * p.dn0 + c * p.dn1] = nw; + } + int32_t const nxt = c < p.sxW ? p.srcNext[i * p.sx0 + c * p.sx1] : 0; + if (c < p.dxW) + { + p.dstNext[s * p.dx0 + c * p.dx1] = nxt; + } + int32_t const drf = c < p.sdW ? p.srcDraft[i * p.sd0 + c * p.sd1] : 0; + if (c < p.ddW) + { + p.dstDraft[s * p.dd0 + c * p.dd1] = drf; + } + int32_t const ln = p.srcLens[i]; + int32_t const t = p.tokensPerStep; + int32_t const d = t - 1; + if (c == 0) + { + p.dstLens[s] = ln; + p.kvOff[i] = ln - t; + } + if (c < t) + { + p.ids[i * t + c] = c < p.dxW ? nxt : 0; + p.posOff[i * t + c] = ln; + } + if (c < d) + { + p.draftIds[i * d + c] = c < p.ddW ? drf : 0; + } +} + +//! The whole tail from one warp of a running kernel (lane = threadIdx.x % 32): rows i < *p.ctrl in order. +__device__ __forceinline__ void foldTailWarp(FoldTailParams const& p, int lane) +{ + int const rows = *p.ctrl; + for (int i = 0; i < rows; ++i) + { + foldTailRow(p, i, lane); + } +} +#endif + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.cu new file mode 100644 index 000000000000..2937d461854c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.cu @@ -0,0 +1,268 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// The split-KV merge for the MiniMax-M3 NVFP4 TP2 (Rubin) full-context decode. The producer is the PrimTS split-KV +// decode with its separate-reduction scratch: per (request, KV head, split, row) one normalized 16-bit partial O and +// one fp32 log2-LSE. Active splits per row follow the producer's split ranges (the causal union of the row's grouped +// Q CTA, instruction-group aligned): splits past them hold nothing and are not read. +// +// One CTA per (request, KV head, kRowsPerCta rows); per row kGroups split groups x kThreadsPerRow threads x 8 dims. +// A thread issues the loads of every split it owns (strided over the groups) before using any, folds them under its +// group's max, and the groups merge through SMEM with one max rescale; the row is normalized once and rounded to +// BF16 once. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +namespace fullCtxReduce +{ +namespace +{ + +constexpr int kGroups = 8; +constexpr int kThreadsPerRow = kHeadDim / 8; +constexpr int kRowsPerCta = 1; +constexpr int kSplitsPerGroup = kMaxSplits / kGroups; +constexpr int kThreads = kGroups * kThreadsPerRow * kRowsPerCta; + +__device__ int activeSplits(Params const& p, int b, int row) +{ + int const seqKv = p.seqLensKv[b]; + int visible = seqKv; + if (p.causal) + { + int const token = row / p.headsPerKv; + int const base = (token / p.qTokensPerCta) * p.qTokensPerCta; + int const end = min(base + p.qTokensPerCta, p.seqLenQ); + visible = min(max(seqKv - p.seqLenQ + end, 0), seqKv); + } + int const tiles = (visible + p.tileKv - 1) / p.tileKv; + int const tilesPerGroup = p.splits * p.instsKv; + int const groups = (tiles + tilesPerGroup - 1) / tilesPerGroup; + int const local = max(groups * p.instsKv, p.instsKv); + return (tiles + local - 1) / local; +} + +template +__device__ float2 unpack(uint32_t word) +{ + if constexpr (kBf16Partial) + { + return __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&word)); + } + else + { + return __half22float2(*reinterpret_cast<__half2 const*>(&word)); + } +} + +template +__global__ void __launch_bounds__(kThreads) fullCtxReduceKernel(Params const p) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + asm volatile("griddepcontrol.wait;" ::: "memory"); + // The dependent (the o_proj activation quantization / GEMV) reads `out` only after its own griddepcontrol.wait, + // which waits for this whole grid, so it is released here and runs its weight-only prologue under the merge. + asm volatile("griddepcontrol.launch_dependents;"); +#endif + int const b = blockIdx.z; + int const h = blockIdx.y; + int const localRow = threadIdx.x / (kGroups * kThreadsPerRow); + int const row = blockIdx.x * kRowsPerCta + localRow; + int const g = (threadIdx.x / kThreadsPerRow) % kGroups; + int const t = threadIdx.x % kThreadsPerRow; + bool const validRow = row < p.rows; + int const active = validRow ? activeSplits(p, b, row) : 0; + + __shared__ float sMax[kRowsPerCta][kGroups]; + __shared__ float sSum[kRowsPerCta][kGroups]; + __shared__ float sAcc[kRowsPerCta][kGroups][kHeadDim]; + + uint4 const* partialO = static_cast(p.partialO); + size_t const kvIdx = static_cast(b) * p.numKvHeads + h; + float lse[kSplitsPerGroup]; + uint4 vec[kSplitsPerGroup]; +#pragma unroll + for (int i = 0; i < kSplitsPerGroup; ++i) + { + int const s = g + i * kGroups; + if (s < active) + { + size_t const r = (kvIdx * p.splits + s) * p.rows + row; + lse[i] = __ldcg(p.partialStats + r); + vec[i] = __ldcg(partialO + r * kThreadsPerRow + t); + } + } + float groupMax = -CUDART_INF_F; +#pragma unroll + for (int i = 0; i < kSplitsPerGroup; ++i) + { + if (g + i * kGroups < active) + { + groupMax = fmaxf(groupMax, lse[i]); + } + } + // A split of the CTA's causal union can hold no key of this row (log2-LSE -inf). Such a split, and a group of + // only such splits, is skipped rather than weighted 0: its partial O carries no defined value. + float acc[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; + float sum = 0.f; +#pragma unroll + for (int i = 0; i < kSplitsPerGroup; ++i) + { + if (g + i * kGroups < active && lse[i] != -CUDART_INF_F) + { + float const w = exp2f(lse[i] - groupMax); + sum += w; + uint32_t const* words = reinterpret_cast(&vec[i]); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float2 const f = unpack(words[j]); + acc[2 * j] += w * f.x; + acc[2 * j + 1] += w * f.y; + } + } + } + if (t == 0) + { + sMax[localRow][g] = groupMax; + sSum[localRow][g] = sum; + } +#pragma unroll + for (int j = 0; j < 8; ++j) + { + sAcc[localRow][g][t * 8 + j] = acc[j]; + } + __syncthreads(); + if (g != 0 || !validRow) + { + return; + } + float rowMax = -CUDART_INF_F; +#pragma unroll + for (int k = 0; k < kGroups; ++k) + { + rowMax = fmaxf(rowMax, sMax[localRow][k]); + } + float total = 0.f; + float o[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; +#pragma unroll + for (int k = 0; k < kGroups; ++k) + { + if (sMax[localRow][k] == -CUDART_INF_F) + { + continue; + } + float const scale = exp2f(sMax[localRow][k] - rowMax); + total += scale * sSum[localRow][k]; +#pragma unroll + for (int j = 0; j < 8; ++j) + { + o[j] += scale * sAcc[localRow][k][t * 8 + j]; + } + } + float const inv = total > 0.f ? 1.f / total : 0.f; + int const token = row / p.headsPerKv; + int const head = row % p.headsPerKv; + __nv_bfloat162 packed[4]; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + packed[j] = __floats2bfloat162_rn(o[2 * j] * inv, o[2 * j + 1] * inv); + } + size_t const outRow + = (static_cast(b) * p.seqLenQ + token) * (p.numKvHeads * p.headsPerKv) + h * p.headsPerKv + head; + reinterpret_cast(p.out + outRow * kHeadDim)[t] = *reinterpret_cast(packed); + if (p.q8 == nullptr) + { + return; + } + // MXFP8 copy of the BF16 row: a 32-column block is 4 neighbouring threads x 8 columns (lanes 0..15 of warp 0 + // hold the row). Block scale = the UE8M0 exponent of ceil(amax / 448), values = satfinite E4M3 of x * 2^-scale. + float x[8]; + float amax = 0.f; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float2 const f = __bfloat1622float2(packed[j]); + x[2 * j] = f.x; + x[2 * j + 1] = f.y; + amax = fmaxf(amax, fmaxf(fabsf(f.x), fabsf(f.y))); + } + amax = fmaxf(amax, __shfl_xor_sync(0x0000ffffu, amax, 1)); + amax = fmaxf(amax, __shfl_xor_sync(0x0000ffffu, amax, 2)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + uint16_t q2[4]; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2[j]) : "f"(x[2 * j + 1] * qs), "f"(x[2 * j] * qs)); + } + int const tok = b * p.seqLenQ + token; + int const col = (h * p.headsPerKv + head) * kHeadDim + t * 8; + *reinterpret_cast(p.q8 + static_cast(tok) * (p.numSfKTiles * 128) + col) + = *reinterpret_cast(q2); + if (t % 4 == 0) + { + int const kSf = col / 32; + int64_t const off = static_cast(tok / 128) * p.numSfKTiles * 512 + (kSf / 4) * 512 + (tok % 32) * 16 + + ((tok % 128) / 32) * 4 + kSf % 4; + p.sf[off] = static_cast(e); + } +} + +} // namespace + +cudaError_t invokeFullCtxReduce(Params const& params, bool pdl, cudaStream_t stream) +{ + if (params.splits < 1 || params.splits > kMaxSplits || params.rows < 1) + { + return cudaErrorInvalidValue; + } + cudaLaunchConfig_t config = {}; + config.gridDim = dim3((params.rows + kRowsPerCta - 1) / kRowsPerCta, params.numKvHeads, params.batch); + config.blockDim = dim3(kThreads); + config.stream = stream; + cudaLaunchAttribute attribute[1]; + attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attribute[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + config.attrs = attribute; + config.numAttrs = 1; + return params.bf16Partial ? cudaLaunchKernelEx(&config, fullCtxReduceKernel, params) + : cudaLaunchKernelEx(&config, fullCtxReduceKernel, params); +} + +} // namespace fullCtxReduce +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.h new file mode 100644 index 000000000000..95af79e2b2ce --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.h @@ -0,0 +1,77 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +//! The split-KV merge of the MiniMax-M3 TP2 full-context decode (PrimTS split-KV decode, separate-reduction scratch) +//! straight to BF16. See fullCtxReduce.cu. +namespace fullCtxReduce +{ + +constexpr int kHeadDim = 128; +constexpr int kMaxSplits = 128; + +struct Params +{ + void const* partialO; // [B, Hkv, S, R, 128] normalized 16-bit partial O per split + float const* partialStats; // [B, Hkv, S, R] log2-LSE per split + int const* seqLensKv; // [B] + __nv_bfloat16* out; // [B, SQ, Hkv * hR, 128] + int batch; + int numKvHeads; + int splits; // S, the planned split capacity (<= kMaxSplits) + int rows; // R = hR * SQ, rows [token, local head] + int seqLenQ; // SQ + int headsPerKv; // hR + int qTokensPerCta; // query tokens one producer CTA owns (grouped Q) + int tileKv; // producer K/V tile + int instsKv; // producer K/V instructions per loop step + bool causal; + bool bf16Partial; + uint8_t* q8 = nullptr; // optional E4M3 copy of out [B * SQ, Hkv * hR * 128] (the o_proj MXFP8 hand-off) + uint8_t* sf = nullptr; // its UE8M0 block scales, 128x4-swizzled as trtllm::mxfp8_quantize(x, True) lays them out + int numSfKTiles = 0; // Hkv * hR * 128 / 128 +}; + +//! Merges the active splits of every row in fp32 with one max rescale and writes BF16; with q8 set, also the MXFP8 +//! copy of the BF16 rows (bit for bit trtllm::mxfp8_quantize of out). With pdl the launch is +//! programmatic: the grid waits on the producer (griddepcontrol.wait) before reading its scratch. +//! +//! PDL contract: griddepcontrol.wait is the kernel's first instruction, so nothing (scratch, lengths, out) is read or +//! written before every prerequisite grid has completed and flushed; a producer's early trigger +//! (griddepcontrol.launch_dependents) only lets this grid launch and wait sooner. The kernel triggers its own +//! dependents right after its wait: the next kernel's griddepcontrol.wait still waits for this whole grid, so its +//! post-wait code sees all of out, and its pre-wait prologue must not read out. +cudaError_t invokeFullCtxReduce(Params const& params, bool pdl, cudaStream_t stream); + +} // namespace fullCtxReduce +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.cu new file mode 100644 index 000000000000..739fb0f72248 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.cu @@ -0,0 +1,1264 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/tllmException.h" + +#include +#include +#include +#include + +// SM107 (Rubin) TP4 copy of minimaxM3Sm107Tp2/mcAllreduceNormTp2.cu for an sm_107 TP4 target: the row kernel at 4 +// ranks. Two changes in the arithmetic: the rank sum rounds to BF16 before the residual add, the rounding points of the +// stock all-reduce + fused residual RMSNorm on this target (bf16 all-reduce output, then bf16(residual + sum)); and +// every call but the MoE finalize runs the row kernel below, whose RMS statistic is the stock fused norm's reduction +// tree (bitwise the stock outputs). +// +// The kernels use sm_100-family features (thread-block clusters, cp.async.bulk, st.async, mbarrier transaction +// counts, block-scaled conversions): they are compiled for every sm_10x target of the build, the sm_100f family target +// included (which runs on SM107). A device pass for any other architecture compiles them to a trap, never to a silent +// empty body, and the host entry point checks that the kernel binary it launches holds sm_10x code (see +// checkBinary). +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000 +#define MINIMAX_M3_KERNELS_ENABLED 1 +#else +#define MINIMAX_M3_KERNELS_ENABLED 0 +#endif + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace cg = cooperative_groups; + +namespace +{ + +constexpr int kEmpty = static_cast(0x80000000u); +constexpr int kFinMaxTopk = 8; // MoE finalize: top-k rows loaded together up to this k + +// The Eagle3 capture written by a capture layer's boundary (cap_out, optional): rmsnorm(updated residual) * fc_norm[i], +// bit for bit the m3_spec_tail capture_norm kernel (kernels/minimaxM3/specTail.cu at H 6144: 256 threads, thread c +// sums the squares of the 8-column chunks c, c + 256 and c + 512 in that order, the warp butterfly 16..1, the 8 warp +// sums added in warp order, rsqrtf(sum / H + eps), bf16(v * rs * w)). Only its layout is reproduced here; its +// arithmetic is the same source. (The TP2 target's mcAllreduceNormTp2.cu has the same epilogue.) +constexpr int kCapThreads = 256; +constexpr int kCapIters = 3; +constexpr int kCapWarps = kCapThreads / 32; +constexpr int kCapHidden = kCapThreads * kCapIters * 8; + +// capture_norm's per-thread chain over one 8-column chunk (continues `ss`). +__device__ __forceinline__ float cap_chunk_sq(uint4 const raw, float ss) +{ + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&raw); +#pragma unroll + for (int q = 0; q < 8; ++q) + { + float const v = __bfloat162float(e[q]); + ss += v * v; + } + return ss; +} + +__device__ __forceinline__ float cap_warp_sum(float v) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + return v; +} + +// bf16(v * rs * w) for one 8-column chunk of the updated residual (v as floats) into cap. +__device__ __forceinline__ void cap_store( + float const (&v)[8], float rs, __nv_bfloat16 const* __restrict__ w, __nv_bfloat16* __restrict__ cap) +{ + uint4 const wr = *reinterpret_cast(w); + __nv_bfloat16 const* we = reinterpret_cast<__nv_bfloat16 const*>(&wr); + uint4 o; + __nv_bfloat16* oe = reinterpret_cast<__nv_bfloat16*>(&o); +#pragma unroll + for (int q = 0; q < 8; ++q) + oe[q] = __float2bfloat16(v[q] * rs * __bfloat162float(we[q])); + *reinterpret_cast(cap) = o; +} + +__device__ __forceinline__ uint32_t canon_bf16x2(uint32_t w) +{ + if ((w & 0xFFFFu) == 0x8000u) + w &= 0xFFFF0000u; + if ((w >> 16) == 0x8000u) + w &= 0x0000FFFFu; + return w; +} + +__device__ __forceinline__ float bf16_lo(uint32_t w) +{ + return __uint_as_float(w << 16); +} + +// The MXFP8 publish quantizes exactly as FlashInfer's CuTe DSL mxfp8 quantize (MXFP8QuantizeRunner.CUTE_DSL, MX-TAC's +// pinned decode quantize; quantization_cute_dsl_utils.py), the three helpers below being its PTX verbatim. It equals +// trtllm::mxfp8_quantize bit for bit except on blocks with 0 < amax < 448 * 2^-127 (UE8M0 byte 0), where the +// TRT-LLM kernel's rcp.approx.ftz of the denormal scale gives inf (NaN / garbage E4M3) and this flushes the block to 0. +__device__ __forceinline__ uint32_t mxfp8_ue8m0(float amax) +{ + float value; + asm("mul.rn.f32 %0, %1, %2;" : "=f"(value) : "f"(amax), "f"(1.0f / 448.0f)); + uint32_t result; + asm("{\n" + " .reg .pred p_zero, p_has_mant, p_exp_zero, p_tiny_sub, p_ovf;\n" + " .reg .u32 bits, exp_biased, mantissa, bump, r;\n" + " setp.le.f32 p_zero, %1, 0f00000000;\n" + " mov.b32 bits, %1;\n" + " shr.b32 exp_biased, bits, 23;\n" + " and.b32 exp_biased, exp_biased, 255;\n" + " and.b32 mantissa, bits, 0x7FFFFF;\n" + " setp.ne.u32 p_has_mant, mantissa, 0;\n" + " selp.u32 bump, 1, 0, p_has_mant;\n" + " setp.eq.u32 p_exp_zero, exp_biased, 0;\n" + " setp.le.u32 p_tiny_sub, mantissa, 0x400000;\n" + " and.pred p_tiny_sub, p_exp_zero, p_tiny_sub;\n" + " @p_tiny_sub mov.u32 bump, 0;\n" + " add.u32 r, exp_biased, bump;\n" + " setp.gt.u32 p_ovf, r, 254;\n" + " selp.u32 r, 254, r, p_ovf;\n" + " selp.u32 %0, 0, r, p_zero;\n" + "}\n" + : "=r"(result) + : "f"(value)); + return result; +} + +__device__ __forceinline__ float mxfp8_inv_scale(uint32_t ue8m0) +{ + float inv; + asm("{\n" + " .reg .s32 new_exp;\n" + " .reg .b32 float_bits;\n" + " .reg .pred p_zero;\n" + " setp.eq.u32 p_zero, %1, 0;\n" + " sub.s32 new_exp, 254, %1;\n" + " max.s32 new_exp, new_exp, 0;\n" + " shl.b32 float_bits, new_exp, 23;\n" + " mov.b32 %0, float_bits;\n" + " @p_zero mov.b32 %0, 0;\n" + "}\n" + : "=f"(inv) + : "r"(ue8m0)); + return inv; +} + +// Two BF16-valued floats (lo, hi) times inv_scale -> two E4M3 bytes (lo in the low byte), saturating to +/-448. +__device__ __forceinline__ uint32_t mxfp8_e4m3x2(float lo, float hi, float inv) +{ + uint16_t pair; + asm("{\n" + " .reg .f32 f0, f1;\n" + " mul.f32 f0, %1, %3;\n" + " mul.f32 f1, %2, %3;\n" + " min.f32 f0, f0, 0f43E00000;\n" + " max.f32 f0, f0, 0fC3E00000;\n" + " min.f32 f1, f1, 0f43E00000;\n" + " max.f32 f1, f1, 0fC3E00000;\n" + " cvt.rn.satfinite.e4m3x2.f32 %0, f1, f0;\n" + "}\n" + : "=h"(pair) + : "f"(lo), "f"(hi), "f"(inv)); + return pair; +} + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ float bf16_hi(uint32_t w) +{ + return __uint_as_float(w & 0xFFFF0000u); +} + +__device__ __forceinline__ int4 ld_volatile_v4(int const* p) +{ + int4 v; + asm volatile("ld.volatile.global.v4.s32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(p) + : "memory"); + return v; +} + +struct FinalizeArgs +{ + __nv_bfloat16 const* fc2; // [permuted rows, H] routed-expert FC2 output (null: plain input x) + int const* perm; // [M, topk] expanded index -> permuted row, -1 = dropped + void const* scales; // [M, topk] routing weights (fp32 or bf16) + int scale_fp32; + int topk; +}; + +// grid = min(M, MAXT) * CLUSTER CTAs, cluster (CLUSTER, 1, 1), THREADS threads. +// OPTS (each bit-identical): 1 residual + norm weight loaded at entry (before the poll); 2 cluster RMS exchange by +// st.async + mbarrier complete_tx instead of cluster.sync; 4 call parity rotated right after the poll; 8 max-smem +// carveout so the next kernel's CTAs co-reside early. +// OPTS 16 (NOT bit-identical with the others; a different boundary): the Eagle3 drafter's boundary, whose stock is a +// plain BF16 all-reduce, then FlashInfer fused_add_rmsnorm: the RMS statistic and the normalization take the +// unrounded FP32 sum bf16(rank sum) + residual (only the stored residual is rounded to BF16). +template +__global__ void __launch_bounds__(THREADS) mc_allreduce_norm_kernel(__nv_bfloat16 const* __restrict__ x, + __nv_bfloat16 const* __restrict__ x2, FinalizeArgs fin, int M, int H, __nv_bfloat16 const* __restrict__ residual, + __nv_bfloat16 const* __restrict__ norm_w, float eps, int* ws_uc, int* ws_mc, int* __restrict__ flags, int rank, + int world, __nv_bfloat16* __restrict__ out_normed, __nv_bfloat16* __restrict__ out_residual, + uint8_t* __restrict__ out_q, uint8_t* __restrict__ out_sf, int sf_k_tiles, int prepushed, + __nv_bfloat16* __restrict__ cap_out, int64_t cap_stride, __nv_bfloat16 const* __restrict__ cap_w, float cap_eps) +{ +#if MINIMAX_M3_KERNELS_ENABLED + __shared__ float warp_sq[THREADS / 32]; + __shared__ alignas(16) float cluster_sq[CLUSTER]; + extern __shared__ __align__(16) uint4 cap_slice[]; // [cols / 8] this CTA's updated chunks (cap_out only) + // Option 32, ONESHOT order (H 6144, CLUSTER 8, not the drafter option 16): every CTA's 3 active warp sums gathered + // in every CTA (global warp 3c + w), then reduced as the stock TP4 oneshot fusion does (see the consumer below). + constexpr bool kOneshotOrder = (OPTS & 32) != 0; + static_assert(!kOneshotOrder || (CLUSTER == 8 && THREADS == 128 && (OPTS & 16) == 0), "option 32: CLUSTER 8 x 128"); + constexpr int kAW = 3; + __shared__ alignas(16) float cluster_wsq[CLUSTER * kAW]; + __shared__ float oneshot_total; + __shared__ alignas(8) unsigned long long xbar; + cg::cluster_group cluster = cg::this_cluster(); + if constexpr ((OPTS & 2) != 0) + { + if (threadIdx.x == 0) + { + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(b) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + } + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + } + int const crank = static_cast(cluster.block_rank()); + int const t = blockIdx.x / CLUSTER; + int const cols = H / CLUSTER; + int const c0 = crank * cols + threadIdx.x * 8; + bool const active = t < M && threadIdx.x * 8 < cols; + int64_t const half_words = static_cast(MAXT) * world * (H / 2); + + // prepushed 1: the producer (m3_mx_gemv) already pushed this rank's rows into the exchange slots; 2: and it + // launched this grid only after its pushes (its grid wait passed), so the slots are polled without a grid wait; + // 3: as 2, with x in set A and x2 in set B. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (prepushed < 2) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + int const parity = *reinterpret_cast(flags + blockIdx.x) & 1; + uint4 rraw = make_uint4(0u, 0u, 0u, 0u); + uint4 wraw = make_uint4(0u, 0u, 0u, 0u); + if constexpr ((OPTS & 1) != 0) + { + // the residual row was written grids earlier (prepushed >= 2: the producer launched this grid only after its + // own grid wait; otherwise the grid wait above) + if (active) + { + rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + wraw = *reinterpret_cast(norm_w + c0); + } + } + // Push this rank's slice into every rank's buffer. + if (active && !prepushed) + { + uint4 raw; + if (fin.fc2 != nullptr) + { + // MoE finalize: fp32 sum of the routing-weighted top-k expert rows plus the shared expert, rounded once. + float f[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; + if (fin.topk <= kFinMaxTopk) + { + // All top-k indices and scales, then every expert-row load in flight at once, then the same in-order + // FP32 accumulation as the loop below (one memory round trip instead of topk; bit-identical). + int prow[kFinMaxTopk]; + float sc[kFinMaxTopk]; +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + prow[k] = -1; + sc[k] = 0.f; + if (k < fin.topk) + { + int const e = t * fin.topk + k; + prow[k] = fin.perm[e]; + sc[k] = fin.scale_fp32 ? static_cast(fin.scales)[e] + : __bfloat162float(static_cast<__nv_bfloat16 const*>(fin.scales)[e]); + } + } + uint4 rv[kFinMaxTopk]; +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + if (prow[k] >= 0) + rv[k] = *reinterpret_cast(fin.fc2 + static_cast(prow[k]) * H + c0); + } +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + if (prow[k] < 0) + continue; + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv[k]); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] += v.x * sc[k]; + f[2 * q + 1] += v.y * sc[k]; + } + } + } + else + for (int k = 0; k < fin.topk; ++k) + { + int const e = t * fin.topk + k; + int const prow = fin.perm[e]; + if (prow < 0) + continue; + float const sc = fin.scale_fp32 + ? static_cast(fin.scales)[e] + : __bfloat162float(static_cast<__nv_bfloat16 const*>(fin.scales)[e]); + uint4 const rv = *reinterpret_cast(fin.fc2 + static_cast(prow) * H + c0); + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] += v.x * sc; + f[2 * q + 1] += v.y * sc; + } + } + if (x2 != nullptr) + { + // As stock: the routed output is rounded to BF16 first, then the shared expert is added (one more + // BF16 rounding of the FP32 sum). + uint4 const rv = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const routed = __bfloat1622float2(__floats2bfloat162_rn(f[2 * q], f[2 * q + 1])); + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] = routed.x + v.x; + f[2 * q + 1] = routed.y + v.y; + } + } + __nv_bfloat162* o = reinterpret_cast<__nv_bfloat162*>(&raw); +#pragma unroll + for (int q = 0; q < 4; ++q) + o[q] = __floats2bfloat162_rn(f[2 * q], f[2 * q + 1]); + } + else + { + raw = *reinterpret_cast(x + static_cast(t) * H + c0); + } + if (fin.fc2 == nullptr && x2 != nullptr) + { + uint4 const raw2 = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162* a = reinterpret_cast<__nv_bfloat162*>(&raw); + __nv_bfloat162 const* b = reinterpret_cast<__nv_bfloat162 const*>(&raw2); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const fa = __bfloat1622float2(a[q]); + float2 const fb = __bfloat1622float2(b[q]); + a[q] = __floats2bfloat162_rn(fa.x + fb.x, fa.y + fb.y); + } + } + int64_t const slot = parity * half_words + (static_cast(t) * world + rank) * (H / 2) + c0 / 2; + *reinterpret_cast(ws_mc + slot) + = make_int4(static_cast(canon_bf16x2(raw.x)), static_cast(canon_bf16x2(raw.y)), + static_cast(canon_bf16x2(raw.z)), static_cast(canon_bf16x2(raw.w))); + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float s[8] = {}; + float sq = 0.f; + if (active) + { + // The residual row (written grids earlier) and every rank's slot are loaded together; slots still empty are + // re-polled. Ranks are then summed in rank order, as before. + if constexpr ((OPTS & 1) == 0) + rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + constexpr int kMaxWorld = 8; + int* const p0 = ws_uc + parity * half_words + static_cast(t) * world * (H / 2) + c0 / 2; + int4 v[kMaxWorld]; + int4 v2[kMaxWorld]; + bool const two = prepushed == 3; + int* const p1 = p0 + 2 * half_words; // set B + uint32_t pending = (1u << world) - 1u; + uint32_t pending2 = two ? pending : 0u; + long long spins = 0; + while ((pending | pending2) != 0u) + { +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if ((pending >> r) & 1u) + v[r] = ld_volatile_v4(p0 + static_cast(r) * (H / 2)); + if ((pending2 >> r) & 1u) + v2[r] = ld_volatile_v4(p1 + static_cast(r) * (H / 2)); + } +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if (((pending >> r) & 1u) && v[r].x != kEmpty && v[r].y != kEmpty && v[r].z != kEmpty + && v[r].w != kEmpty) + pending &= ~(1u << r); + if (((pending2 >> r) & 1u) && v2[r].x != kEmpty && v2[r].y != kEmpty && v2[r].z != kEmpty + && v2[r].w != kEmpty) + pending2 &= ~(1u << r); + } + if (++spins > (1LL << 31)) + __trap(); + } +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if (r < world) + { + *reinterpret_cast(p0 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t ww[4] = {static_cast(v[r].x), static_cast(v[r].y), + static_cast(v[r].z), static_cast(v[r].w)}; + if (two) + { + *reinterpret_cast(p1 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t const w2[4] = {static_cast(v2[r].x), static_cast(v2[r].y), + static_cast(v2[r].z), static_cast(v2[r].w)}; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + __nv_bfloat162 const u + = __floats2bfloat162_rn(bf16_lo(ww[q]) + bf16_lo(w2[q]), bf16_hi(ww[q]) + bf16_hi(w2[q])); + ww[q] = *reinterpret_cast(&u); + } + } +#pragma unroll + for (int q = 0; q < 4; ++q) + { + s[2 * q] += bf16_lo(ww[q]); + s[2 * q + 1] += bf16_hi(ww[q]); + } + } + } + // Stock rounding points: the rank sum is a BF16 all-reduce output before the residual add. +#pragma unroll + for (int i = 0; i < 8; ++i) + s[i] = __bfloat162float(__float2bfloat16_rn(s[i])); + __nv_bfloat162 const* rr = reinterpret_cast<__nv_bfloat162 const*>(&rraw); + uint32_t upd[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const rf = __bfloat1622float2(rr[q]); + __nv_bfloat162 const u = __floats2bfloat162_rn(s[2 * q] + rf.x, s[2 * q + 1] + rf.y); + upd[q] = *reinterpret_cast(&u); + if constexpr ((OPTS & 16) != 0) + { + float const x0 = s[2 * q] + rf.x, x1 = s[2 * q + 1] + rf.y; + s[2 * q] = x0; + s[2 * q + 1] = x1; + sq += x0 * x0 + x1 * x1; + } + else + { + float2 const uf = __bfloat1622float2(u); + s[2 * q] = uf.x; + s[2 * q + 1] = uf.y; + if constexpr (kOneshotOrder) + { + // stock oneshot: acc += v * v, one column at a time + sq += uf.x * uf.x; + sq += uf.y * uf.y; + } + else + sq += uf.x * uf.x + uf.y * uf.y; + } + } + *reinterpret_cast(out_residual + static_cast(t) * H + c0) + = make_uint4(upd[0], upd[1], upd[2], upd[3]); + if constexpr ((OPTS & 16) == 0) + { + if (cap_out != nullptr) + cap_slice[threadIdx.x] = make_uint4(upd[0], upd[1], upd[2], upd[3]); + } + } + if constexpr ((OPTS & 4) != 0) + { + // every CTA (active or not) rotates its call parity once its own poll is done (read only by later grids) + if (threadIdx.x == 0) + flags[blockIdx.x] = parity ^ 1; + } +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + sq += __shfl_xor_sync(0xffffffffu, sq, off); + if ((threadIdx.x & 31) == 0) + warp_sq[threadIdx.x / 32] = sq; + __syncthreads(); + if constexpr ((OPTS & 2) != 0) + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); // peers' xbar initialized (long since) + if (threadIdx.x == 0) + { + if constexpr (kOneshotOrder) + { + // Every CTA gets this CTA's kAW warp sums at [kAW * crank + w]. + if constexpr ((OPTS & 2) == 0) + { + for (int c = 0; c < CLUSTER; ++c) + for (int w = 0; w < kAW; ++w) + *cluster.map_shared_rank(&cluster_wsq[kAW * crank + w], c) = warp_sq[w]; + } + else + { + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(b), "r"(CLUSTER * kAW * 4) + : "memory"); + for (int c = 0; c < CLUSTER; ++c) + { + uint32_t rb; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rb) : "r"(b), "r"(c)); + for (int w = 0; w < kAW; ++w) + { + uint32_t rd; + uint32_t const d + = static_cast(__cvta_generic_to_shared(&cluster_wsq[kAW * crank + w])); + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rd) : "r"(d), "r"(c)); + asm volatile( + "st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(rd), + "r"(__float_as_uint(warp_sq[w])), "r"(rb) + : "memory"); + } + } + } + } + else + { + float total = 0.f; +#pragma unroll + for (int w = 0; w < THREADS / 32; ++w) + total += warp_sq[w]; + if constexpr ((OPTS & 2) == 0) + for (int c = 0; c < CLUSTER; ++c) + *cluster.map_shared_rank(&cluster_sq[crank], c) = total; + else + { + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + uint32_t const d = static_cast(__cvta_generic_to_shared(&cluster_sq[crank])); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(b), "r"(CLUSTER * 4) + : "memory"); + for (int c = 0; c < CLUSTER; ++c) + { + uint32_t rd, rb; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rd) : "r"(d), "r"(c)); + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rb) : "r"(b), "r"(c)); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(rd), + "r"(__float_as_uint(total)), "r"(rb) + : "memory"); + } + } + } + } + if constexpr ((OPTS & 2) == 0) + cluster.sync(); + else + { + // every peer's sum landed (st.async complete_tx on this CTA's barrier); no cluster-wide release-arrive + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + uint32_t done = 0; + while (!done) + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}" + : "=r"(done) + : "r"(b) + : "memory"); + } + if constexpr ((OPTS & 16) == 0) + { + if (cap_out != nullptr) + { + // The row's chunks are spread over the cluster (CTA c holds chunks [c * per, (c + 1) * per)); every CTA + // forms capture_norm's statistic itself, its 128 threads standing in for capture threads ct and ct + 128. + __shared__ float cap_red[kCapWarps]; + int const per = cols / 8; + asm volatile("barrier.cluster.arrive.release.aligned;" ::: "memory"); + asm volatile("barrier.cluster.wait.acquire.aligned;" ::: "memory"); + float ca = 0.f, cb = 0.f; +#pragma unroll + for (int it = 0; it < kCapIters; ++it) + { + int const ka = it * kCapThreads + static_cast(threadIdx.x); + int const kb = ka + kCapThreads / 2; + ca = cap_chunk_sq(*cluster.map_shared_rank(&cap_slice[ka % per], ka / per), ca); + cb = cap_chunk_sq(*cluster.map_shared_rank(&cap_slice[kb % per], kb / per), cb); + } + // Every peer read is done before any CTA moves on (and later exits). + asm volatile("barrier.cluster.arrive.release.aligned;" ::: "memory"); + asm volatile("barrier.cluster.wait.acquire.aligned;" ::: "memory"); + ca = cap_warp_sum(ca); + cb = cap_warp_sum(cb); + if ((threadIdx.x & 31) == 0) + { + cap_red[threadIdx.x / 32] = ca; + cap_red[threadIdx.x / 32 + kCapWarps / 2] = cb; + } + __syncthreads(); + float tot = 0.f; +#pragma unroll + for (int w = 0; w < kCapWarps; ++w) + tot += cap_red[w]; + float const rs = rsqrtf(tot / H + cap_eps); + if (active) + cap_store(s, rs, cap_w + c0, cap_out + static_cast(t) * cap_stride + c0); + } + } + if constexpr (kOneshotOrder) + { + // The stock TP4 oneshot fusion at H 6144 (4 CTAs x 192 threads): blockReduceSumV2 per CTA = a 32-lane + // butterfly over its 6 warp sums (lanes >= 6 zero), lane 0's value; then thread 0 adds the 4 CTA sums in + // order from 0.f. Global warps 6g..6g+5 = this cluster's CTAs 2g, 2g+1. + if (threadIdx.x < 32) + { + int const lane = threadIdx.x; + float acc = 0.f; + for (int g = 0; g < 4; ++g) + { + float val = lane < 6 ? cluster_wsq[6 * g + lane] : 0.f; +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + val += __shfl_xor_sync(0xffffffffu, val, off); + acc += __shfl_sync(0xffffffffu, val, 0); + } + if (lane == 0) + oneshot_total = acc; + } + __syncthreads(); + } + if (active) + { + float total = 0.f; + if constexpr (kOneshotOrder) + total = oneshot_total; + else + { +#pragma unroll + for (int c = 0; c < CLUSTER; ++c) + total += cluster_sq[c]; + } + float const inv = rsqrtf(total / static_cast(H) + eps); + if constexpr ((OPTS & 1) == 0) + wraw = *reinterpret_cast(norm_w + c0); + __nv_bfloat162 const* wv = reinterpret_cast<__nv_bfloat162 const*>(&wraw); + uint32_t o[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const wf = __bfloat1622float2(wv[q]); + __nv_bfloat162 const n = __floats2bfloat162_rn(s[2 * q] * inv * wf.x, s[2 * q + 1] * inv * wf.y); + o[q] = *reinterpret_cast(&n); + float2 const nf = __bfloat1622float2(n); + s[2 * q] = nf.x; + s[2 * q + 1] = nf.y; + } + *reinterpret_cast(out_normed + static_cast(t) * H + c0) = make_uint4(o[0], o[1], o[2], o[3]); + } + if (out_q != nullptr) + { + // MXFP8 copy of the normed row for the next GEMM, as the CuTe DSL mxfp8 quantize of `normed` (the tree's + // decode quantize, MX-TAC; see mxfp8_ue8m0): four threads hold one 32-column scale block. + float amax = 0.f; + if (active) + { +#pragma unroll + for (int e = 0; e < 8; ++e) + amax = fmaxf(amax, fabsf(s[e])); + } + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 1)); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 2)); + if (active) + { + uint32_t const code = mxfp8_ue8m0(amax); + float const inv_scale = mxfp8_inv_scale(code); + uint32_t packed[2]; +#pragma unroll + for (int h = 0; h < 2; ++h) + { + packed[h] = mxfp8_e4m3x2(s[4 * h], s[4 * h + 1], inv_scale) + | (mxfp8_e4m3x2(s[4 * h + 2], s[4 * h + 3], inv_scale) << 16); + } + *reinterpret_cast(out_q + static_cast(t) * H + c0) = make_uint2(packed[0], packed[1]); + if ((threadIdx.x & 3) == 0) + { + int const kb = c0 / 32; + out_sf[static_cast(t / 128) * sf_k_tiles * 512 + (kb / 4) * 512 + (t % 32) * 16 + + ((t % 128) / 32) * 4 + kb % 4] + = static_cast(code); + } + } + } + if constexpr ((OPTS & 4) == 0) + { + __syncthreads(); + if (threadIdx.x == 0) + flags[blockIdx.x] = parity ^ 1; + } +#elif defined(__CUDA_ARCH__) + __trap(); // compiled for an architecture without the sm_10x features: never a silent no-op +#endif +} + +// Refuses (raises) to launch a kernel whose loaded binary holds no sm_10x code (PTX and SASS version >= 100): the +// device passes for other architectures compile the kernels to a trap, and a build that left out every sm_10x target +// must not reach the GPU at all. +template +bool checkBinary(Kernel kernel, char const* name) +{ + cudaFuncAttributes attr{}; + TLLM_CUDA_CHECK(cudaFuncGetAttributes(&attr, reinterpret_cast(kernel))); + TLLM_CHECK_WITH_INFO(attr.ptxVersion >= 100 && attr.binaryVersion >= 100, + "%s: the loaded binary holds no sm_10x code (PTX %d, SASS %d): rebuild with an sm_100f / sm_107a target", name, + attr.ptxVersion, attr.binaryVersion); + return true; +} + +template +void launch(MinimaxM3Sm107Tp4McAllreduceNormParams const& p, bool pdl, cudaStream_t stream) +{ + int const M = p.numTokens, H = p.hidden; + TLLM_CHECK_WITH_INFO(H % (CLUSTER * 8) == 0 && H / CLUSTER / 8 <= THREADS, "unsupported hidden size %d", H); + TLLM_CHECK_WITH_INFO(M >= 1 && M <= MAXT, "%d tokens do not fit the %d-token bucket", M, MAXT); + auto kernel = mc_allreduce_norm_kernel; + static bool const binaryOk = checkBinary(kernel, "mc_allreduce_norm_kernel"); + (void) binaryOk; + static bool const configured = [&]() + { + bool ok = CLUSTER <= 8 + || cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1) == cudaSuccess; + if ((OPTS & 8) != 0) + ok = ok + && cudaFuncSetAttribute( + kernel, cudaFuncAttributePreferredSharedMemoryCarveout, cudaSharedmemCarveoutMaxShared) + == cudaSuccess; + if (!ok) + cudaGetLastError(); + return ok; + }(); + TLLM_CHECK_WITH_INFO(configured, "cluster size not supported"); + FinalizeArgs const fin{p.fc2, p.perm, p.scales, p.scaleFp32 ? 1 : 0, p.topk}; + cudaLaunchConfig_t cfg = {}; + // The grid is sized to the call's rows: min(M, MAXT) clusters. Each (row, slice) slot keeps its own parity flag, + // read by the producers before they push and rotated only by the slot's CTA after it polled: a slot that a call + // does not use keeps its parity, and every rank runs the same calls, so the parities stay consistent when M + // changes between calls. + cfg.gridDim = dim3((M < MAXT ? M : MAXT) * CLUSTER); + cfg.blockDim = dim3(THREADS); + cfg.stream = stream; + if (p.capOut != nullptr) + { + TLLM_CHECK_WITH_INFO((OPTS & 16) == 0 && H == kCapHidden && 2 * THREADS == kCapThreads, + "the capture epilogue takes H %d on the 128-thread cluster kernel, not the drafter boundary", kCapHidden); + cfg.dynamicSmemBytes = static_cast(H / CLUSTER / 8) * sizeof(uint4); + } + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + attr[1].id = cudaLaunchAttributeClusterDimension; + attr[1].val.clusterDim.x = CLUSTER; + attr[1].val.clusterDim.y = 1; + attr[1].val.clusterDim.z = 1; + cfg.attrs = attr; + cfg.numAttrs = 2; + cudaError_t const err = cudaLaunchKernelEx(&cfg, kernel, p.x, p.x2, fin, M, H, p.residual, p.normWeight, p.eps, + p.wsUc, p.wsMc, p.flags, p.rank, p.world, p.outNormed, p.outResidual, p.outQuant, p.outSf, (H / 32 + 3) / 4, + p.prepushed, p.capOut, p.capStride, p.capWeight, p.capEps); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +// ws_uc / ws_mc are this rank's unicast and multicast views of ONE exchange buffer (they alias): no __restrict__. +// prepushed >= 2 contract: the producer (m3_mx_gemv's o_proj push, the m3_moe engine's finalize push) passes its own +// grid dependency, pushes its rows into the exchange, and only then releases this grid (griddepcontrol +// .launch_dependents); this kernel therefore skips griddepcontrol.wait and reads only the pushed (Lamport-polled) +// words, its parity flags and the residual / norm weight, which were written two or more grids earlier. +// +// The TP4 row kernel (4 ranks): one CTA of H / 8 = 768 threads per token row, used for every call but the MoE finalize. +// Its residual add + RMSNorm is the stock TP4 all-reduce fusion (allReduceFusionKernels.cu, oneshot Lamport), the fused +// norm after the all-reduce on this target, operation for operation at H 6144: thread j owns columns [8j, 8j + 8), its +// sum of squares is acc += v * v over them in order, then the warp butterfly 16..1, each group of 6 warps' sums through +// shared memory into a second 32-lane butterfly (lanes >= 6 zero), the 4 group sums added in order from 0.f (the stock +// oneshot kernel's 4-CTA cluster), rsqrtf(acc / H + eps) and (v * denom) * w. With the BF16 rank sum before the +// residual add, the normed rows, the updated residual and the MXFP8 copy are bitwise the stock ones. The exchange slots +// and the per-(row, slice) parity flags keep the cluster kernel's layout (slice = 96 threads), so producers that push +// (the o_proj GEMV) are unchanged. +constexpr int kSlices = 8; +constexpr int kSliceThreads = 96; +constexpr int kRowThreads = 768; +// The stock TP4 fused norm's launch at H 6144 (allreduce_fusion_kernel_launcher): 768 threads per token start as 8 +// CTAs of 96, then widen to CTAs of at least 128 threads: 4 CTAs of 192 per row cluster. +constexpr int kOneshotBlocks = 4; +constexpr int kOneshotThreads = kRowThreads / kOneshotBlocks; +static_assert(kOneshotThreads % 32 == 0 && kOneshotThreads / 32 <= 32, "one second-level butterfly per CTA"); + +// grid = M CTAs (one per row), THREADS = H / 8 threads. WORLD ranks (compile time). DRAFTER (opts bit 16): the Eagle3 +// drafter boundary, whose stock is a plain BF16 all-reduce then FlashInfer fused_add_rmsnorm: the RMS statistic and +// the normalization take the unrounded FP32 sum bf16(rank sum) + residual (only the stored residual is rounded). +// Its reduction is FlashInfer's CuTe-DSL FusedAddRMSNormKernel at H 6144 (what flashinfer.norm.fused_add_rmsnorm runs +// on SM 10.7 in this image: 128-thread CTAs, 64 threads per row, cluster 1): FI thread t sums its 96 columns +// 512 b + 8 t + e in (b, e) order from 0.f, a butterfly over its warp with offsets 1, 2, .., 16, the row's 2 warp sums +// through a second ascending butterfly (lanes >= 2 zero); rstd = rsqrt.approx(div.rn(sum, H) + eps), y = (h * rstd) * +// w. Bitwise to the stock boundary (normed and residual, eager and graph, 4 ranks, 128 calls). CARVE +// (opts bit 64): the same kernel, its own function with the max-smem carveout preference set, so the next kernel's CTAs +// (the QKV GEMV's big staged tiles) can co-reside on the SM while it runs (the cluster kernel's option 8). ws_uc / +// ws_mc alias (one exchange buffer): no __restrict__. +template +__global__ void __launch_bounds__(THREADS, 1) mc_allreduce_norm_row_kernel(__nv_bfloat16 const* __restrict__ x, + __nv_bfloat16 const* __restrict__ x2, int H, __nv_bfloat16 const* __restrict__ residual, + __nv_bfloat16 const* __restrict__ norm_w, float eps, int* ws_uc, int* ws_mc, int* __restrict__ flags, int rank, + int maxt, __nv_bfloat16* __restrict__ out_normed, __nv_bfloat16* __restrict__ out_residual, + uint8_t* __restrict__ out_q, uint8_t* __restrict__ out_sf, int sf_k_tiles, int prepushed, + __nv_bfloat16* __restrict__ cap_out, int64_t cap_stride, __nv_bfloat16 const* __restrict__ cap_w, float cap_eps) +{ +#if MINIMAX_M3_KERNELS_ENABLED + __shared__ float warp_sq[THREADS / 32]; + __shared__ float group_sq[kOneshotBlocks]; + extern __shared__ __align__(16) uint4 cap_row[]; // [THREADS] the updated row's chunks (cap_out, !DRAFTER only) + // DRAFTER: the row's FP32 sums h, regrouped into FlashInfer's per-thread column sets. + __shared__ float h_row[DRAFTER ? THREADS * 8 : 1]; + __shared__ float total_sh; + int const t = blockIdx.x; + int const j = threadIdx.x; + int const c0 = j * 8; + int const slice = j / kSliceThreads; + int64_t const half_words = static_cast(maxt) * WORLD * (H / 2); + if (prepushed < 2) + asm volatile("griddepcontrol.wait;" ::: "memory"); + int const parity = *reinterpret_cast(flags + t * kSlices + slice) & 1; + uint4 const rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + uint4 const wraw = *reinterpret_cast(norm_w + c0); + if (!prepushed) + { + uint4 raw = *reinterpret_cast(x + static_cast(t) * H + c0); + if (x2 != nullptr) + { + uint4 const raw2 = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162* a = reinterpret_cast<__nv_bfloat162*>(&raw); + __nv_bfloat162 const* b = reinterpret_cast<__nv_bfloat162 const*>(&raw2); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const fa = __bfloat1622float2(a[q]); + float2 const fb = __bfloat1622float2(b[q]); + a[q] = __floats2bfloat162_rn(fa.x + fb.x, fa.y + fb.y); + } + } + int64_t const slot = parity * half_words + (static_cast(t) * WORLD + rank) * (H / 2) + c0 / 2; + *reinterpret_cast(ws_mc + slot) + = make_int4(static_cast(canon_bf16x2(raw.x)), static_cast(canon_bf16x2(raw.y)), + static_cast(canon_bf16x2(raw.z)), static_cast(canon_bf16x2(raw.w))); + } + asm volatile("griddepcontrol.launch_dependents;"); + float s[8] = {}; + float sq = 0.f; + { + int* const p0 = ws_uc + parity * half_words + static_cast(t) * WORLD * (H / 2) + c0 / 2; + int* const p1 = p0 + 2 * half_words; // set B + bool const two = prepushed == 3; + int4 v[WORLD]; + int4 v2[WORLD]; + uint32_t pending = (1u << WORLD) - 1u; + uint32_t pending2 = two ? pending : 0u; + long long spins = 0; + while ((pending | pending2) != 0u) + { +#pragma unroll + for (int r = 0; r < WORLD; ++r) + { + if ((pending >> r) & 1u) + v[r] = ld_volatile_v4(p0 + static_cast(r) * (H / 2)); + if ((pending2 >> r) & 1u) + v2[r] = ld_volatile_v4(p1 + static_cast(r) * (H / 2)); + } +#pragma unroll + for (int r = 0; r < WORLD; ++r) + { + if (((pending >> r) & 1u) && v[r].x != kEmpty && v[r].y != kEmpty && v[r].z != kEmpty + && v[r].w != kEmpty) + pending &= ~(1u << r); + if (((pending2 >> r) & 1u) && v2[r].x != kEmpty && v2[r].y != kEmpty && v2[r].z != kEmpty + && v2[r].w != kEmpty) + pending2 &= ~(1u << r); + } + if (++spins > (1LL << 31)) + __trap(); + } +#pragma unroll + for (int r = 0; r < WORLD; ++r) + { + *reinterpret_cast(p0 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t ww[4] = {static_cast(v[r].x), static_cast(v[r].y), + static_cast(v[r].z), static_cast(v[r].w)}; + if (two) + { + *reinterpret_cast(p1 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t const w2[4] = {static_cast(v2[r].x), static_cast(v2[r].y), + static_cast(v2[r].z), static_cast(v2[r].w)}; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + __nv_bfloat162 const u + = __floats2bfloat162_rn(bf16_lo(ww[q]) + bf16_lo(w2[q]), bf16_hi(ww[q]) + bf16_hi(w2[q])); + ww[q] = *reinterpret_cast(&u); + } + } +#pragma unroll + for (int q = 0; q < 4; ++q) + { + s[2 * q] += bf16_lo(ww[q]); + s[2 * q + 1] += bf16_hi(ww[q]); + } + } + // Stock rounding points: the rank sum is a BF16 all-reduce output before the residual add. +#pragma unroll + for (int i = 0; i < 8; ++i) + s[i] = __bfloat162float(__float2bfloat16_rn(s[i])); + __nv_bfloat162 const* rr = reinterpret_cast<__nv_bfloat162 const*>(&rraw); + uint32_t upd[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const rf = __bfloat1622float2(rr[q]); + __nv_bfloat162 const u = __floats2bfloat162_rn(s[2 * q] + rf.x, s[2 * q + 1] + rf.y); + upd[q] = *reinterpret_cast(&u); + if constexpr (DRAFTER) + { + s[2 * q] += rf.x; + s[2 * q + 1] += rf.y; + } + else + { + float2 const uf = __bfloat1622float2(u); + s[2 * q] = uf.x; + s[2 * q + 1] = uf.y; + } + } +#pragma unroll + for (int e = 0; e < 8; ++e) + sq += s[e] * s[e]; + if constexpr (DRAFTER) + { +#pragma unroll + for (int e = 0; e < 8; ++e) + h_row[c0 + e] = s[e]; + } + *reinterpret_cast(out_residual + static_cast(t) * H + c0) + = make_uint4(upd[0], upd[1], upd[2], upd[3]); + if constexpr (!DRAFTER) + { + if (cap_out != nullptr) + cap_row[j] = make_uint4(upd[0], upd[1], upd[2], upd[3]); + } + } + // The stock TP4 all-reduce's fused RMSNorm (allReduceFusionKernels.cu, allreduce_fusion_kernel_oneshot_lamport at + // H 6144): a cluster of kOneshotBlocks CTAs x kOneshotThreads threads per row; each CTA reduces its warps with + // blockReduceSumV2 (warp butterfly 16..1, the CTA's warp sums into a second 32-lane butterfly, lanes past its warp + // count zero), then thread 0 sums the CTAs' totals in cluster-rank order from 0.f. + sq += __shfl_xor_sync(~0, sq, 16); + sq += __shfl_xor_sync(~0, sq, 8); + sq += __shfl_xor_sync(~0, sq, 4); + sq += __shfl_xor_sync(~0, sq, 2); + sq += __shfl_xor_sync(~0, sq, 1); + if ((j & 31) == 0) + warp_sq[j / 32] = sq; + __syncthreads(); + if constexpr (DRAFTER) + { + constexpr int kFiThreads = 64; // FlashInfer CuTe threads per row at H 6144 + constexpr int kFiBlocks = THREADS * 8 / (kFiThreads * 8); // 12 column blocks of 512 + if (j < kFiThreads) + { + float acc = 0.f; +#pragma unroll + for (int b = 0; b < kFiBlocks; ++b) + { +#pragma unroll + for (int e = 0; e < 8; ++e) + { + float const v = h_row[b * kFiThreads * 8 + j * 8 + e]; + acc = __fmaf_rn(v, v, acc); // the DSL's h * h + reduce contracts to FMA + } + } +#pragma unroll + for (int off = 1; off < 32; off <<= 1) + acc += __shfl_xor_sync(~0, acc, off); + if ((j & 31) == 0) + group_sq[j / 32] = acc; + } + __syncthreads(); + if (j < 32) + { + float val = j < kFiThreads / 32 ? group_sq[j] : 0.f; +#pragma unroll + for (int off = 1; off < 32; off <<= 1) + val += __shfl_xor_sync(~0, val, off); + if (j == 0) + total_sh = val; + } + } + else if (j < kOneshotBlocks * 32) + { + constexpr int kWarpsPerBlock = kOneshotThreads / 32; + int const g = j / 32; + int const lane = j & 31; + float val = lane < kWarpsPerBlock ? warp_sq[g * kWarpsPerBlock + lane] : 0.f; + val += __shfl_xor_sync(~0, val, 16); + val += __shfl_xor_sync(~0, val, 8); + val += __shfl_xor_sync(~0, val, 4); + val += __shfl_xor_sync(~0, val, 2); + val += __shfl_xor_sync(~0, val, 1); + if (lane == 0) + group_sq[g] = val; + } + __syncthreads(); + if (!DRAFTER && j == 0) + { + float acc = 0.f; +#pragma unroll + for (int g = 0; g < kOneshotBlocks; ++g) + acc += group_sq[g]; + total_sh = acc; + } + // The parity of this row's slices rotates once every thread has read the exchange (later grids only). + if (j % kSliceThreads == 0) + flags[t * kSlices + slice] = parity ^ 1; + __syncthreads(); + if constexpr (!DRAFTER) + { + if (cap_out != nullptr) + { + // The whole row is in this CTA: threads 0..255 run capture_norm's chains over the staged chunks. + __shared__ float cap_red[kCapWarps]; + if (j < kCapThreads) + { + float css = 0.f; +#pragma unroll + for (int it = 0; it < kCapIters; ++it) + css = cap_chunk_sq(cap_row[it * kCapThreads + j], css); + css = cap_warp_sum(css); + if ((j & 31) == 0) + cap_red[j / 32] = css; + } + __syncthreads(); + float tot = 0.f; +#pragma unroll + for (int w = 0; w < kCapWarps; ++w) + tot += cap_red[w]; + float const rs = rsqrtf(tot / H + cap_eps); + cap_store(s, rs, cap_w + c0, cap_out + static_cast(t) * cap_stride + c0); + } + } + float inv; + if constexpr (DRAFTER) + { + float const mean = __fdiv_rn(total_sh, static_cast(H)); + asm("rsqrt.approx.f32 %0, %1;" : "=f"(inv) : "f"(__fadd_rn(mean, eps))); + } + else + inv = rsqrtf(total_sh / H + eps); + __nv_bfloat162 const* wv = reinterpret_cast<__nv_bfloat162 const*>(&wraw); + uint32_t o[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const wf = __bfloat1622float2(wv[q]); + __nv_bfloat162 const n = __floats2bfloat162_rn(s[2 * q] * inv * wf.x, s[2 * q + 1] * inv * wf.y); + o[q] = *reinterpret_cast(&n); + float2 const nf = __bfloat1622float2(n); + s[2 * q] = nf.x; + s[2 * q + 1] = nf.y; + } + *reinterpret_cast(out_normed + static_cast(t) * H + c0) = make_uint4(o[0], o[1], o[2], o[3]); + if (out_q != nullptr) + { + float amax = 0.f; +#pragma unroll + for (int e = 0; e < 8; ++e) + amax = fmaxf(amax, fabsf(s[e])); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 1)); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 2)); + uint32_t const code = mxfp8_ue8m0(amax); + float const inv_scale = mxfp8_inv_scale(code); + uint32_t packed[2]; +#pragma unroll + for (int h = 0; h < 2; ++h) + { + packed[h] = mxfp8_e4m3x2(s[4 * h], s[4 * h + 1], inv_scale) + | (mxfp8_e4m3x2(s[4 * h + 2], s[4 * h + 3], inv_scale) << 16); + } + *reinterpret_cast(out_q + static_cast(t) * H + c0) = make_uint2(packed[0], packed[1]); + if ((j & 3) == 0) + { + int const kb = c0 / 32; + out_sf[static_cast(t / 128) * sf_k_tiles * 512 + (kb / 4) * 512 + (t % 32) * 16 + + ((t % 128) / 32) * 4 + kb % 4] + = static_cast(code); + } + } +#elif defined(__CUDA_ARCH__) + __trap(); // compiled for an architecture without the sm_10x features: never a silent no-op +#endif +} + +template +auto rowKernel() +{ + auto kernel = mc_allreduce_norm_row_kernel<4, kRowThreads, DRAFTER, CARVE>; + static bool const ok = [&]() + { + checkBinary(kernel, "mc_allreduce_norm_row_kernel"); + if constexpr (CARVE) + TLLM_CUDA_CHECK(cudaFuncSetAttribute( + kernel, cudaFuncAttributePreferredSharedMemoryCarveout, cudaSharedmemCarveoutMaxShared)); + return true; + }(); + (void) ok; + return kernel; +} + +void launchRow(MinimaxM3Sm107Tp4McAllreduceNormParams const& p, int32_t maxTokens, bool drafter, bool carve, bool pdl, + cudaStream_t stream) +{ + TLLM_CHECK_WITH_INFO(p.hidden == kRowThreads * 8 && p.world == 4, "the row kernel takes H 6144 at 4 ranks"); + TLLM_CHECK_WITH_INFO(p.numTokens >= 1 && p.numTokens <= maxTokens, "%d tokens do not fit the %d-token bucket", + p.numTokens, maxTokens); + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(p.numTokens); + cfg.blockDim = dim3(kRowThreads); + cfg.stream = stream; + if (p.capOut != nullptr) + { + TLLM_CHECK_WITH_INFO( + !drafter && p.hidden == kCapHidden, "the capture epilogue takes H %d, not the drafter", kCapHidden); + cfg.dynamicSmemBytes = static_cast(kRowThreads) * sizeof(uint4); + } + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + auto kernel = drafter ? (carve ? rowKernel() : rowKernel()) + : (carve ? rowKernel() : rowKernel()); + cudaError_t const err = cudaLaunchKernelEx(&cfg, kernel, p.x, p.x2, p.hidden, p.residual, p.normWeight, p.eps, + p.wsUc, p.wsMc, p.flags, p.rank, maxTokens, p.outNormed, p.outResidual, p.outQuant, p.outSf, + (p.hidden / 32 + 3) / 4, p.prepushed, p.capOut, p.capStride, p.capWeight, p.capEps); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +constexpr int kCluster = 8; +constexpr int kThreads = 128; + +} // namespace + +bool isMinimaxM3Sm107Tp4McAllreduceNormSupported(int32_t maxTokens, int32_t opts) +{ + opts &= ~(64 | 128); // the row kernel's carveout instance; 128 only routes the call (invoke) + return (maxTokens == 8 && (opts == 7 || opts == 15 || opts == 23 || opts == 31 || opts == 39 || opts == 47)) + || ((maxTokens == 16 || maxTokens == 48) && (opts == 15 || opts == 47)); +} + +// Option 128 routes a world-4 call to the 8-CTA cluster kernel instead of the row kernel: the drafter boundary's +// cluster form (opts 23 / 31 = 16 | 7 / 16 | 15, dispatch.DRAFT_MC_AR_CLUSTER). Deterministic and exact-class to the +// row kernel's FlashInfer order: the updated residual is bitwise, the RMS sum of squares is the cluster's (per-CTA +// warps, then the 8-CTA total), so a normed element can differ by one BF16 rounding step. +void invokeMinimaxM3Sm107Tp4McAllreduceNorm(MinimaxM3Sm107Tp4McAllreduceNormParams const& params, int32_t maxTokens, + int32_t opts, bool pdl, cudaStream_t stream) +{ + if (params.fc2 == nullptr && params.world == 4 && params.hidden == kRowThreads * 8 && (opts & (32 | 128)) == 0) + { + launchRow(params, maxTokens, (opts & 16) != 0, (opts & 64) != 0, pdl, stream); + return; + } + opts &= ~(64 | 128); + if (maxTokens == 8 && opts == 39) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 47) + { + launch(params, pdl, stream); + } + else if (maxTokens == 16 && opts == 47) + { + launch(params, pdl, stream); + } + else if (maxTokens == 48 && opts == 47) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 7) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 15) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 23) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 31) + { + launch(params, pdl, stream); + } + else if (maxTokens == 16 && opts == 15) + { + launch(params, pdl, stream); + } + else if (maxTokens == 48 && opts == 15) + { + launch(params, pdl, stream); + } + else + { + TLLM_THROW("unsupported (bucket %d, opts %d)", maxTokens, opts); + } +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.h new file mode 100644 index 000000000000..85938182c098 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.h @@ -0,0 +1,83 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Operands of the MiniMax-M3 multicast all-reduce + residual + RMSNorm kernel (decode, at most 48 tokens). +//! +//! Per rank the kernel forms the partial `bf16(x + x2)` (or, in finalize mode, +//! `bf16(bf16(sum_k scale * fc2[perm]) + x2)`, the routed sum rounded before the shared expert is added, as stock), +//! pushes it through the multicast mapping `wsMc` into every rank's exchange, polls its own unicast view `wsUc`, +//! sums the ranks in FP32 in rank order, adds the residual (BF16 round) and applies RMSNorm. With `outQuant` it +//! also writes the MXFP8 copy of the normed rows (E4M3 [numTokens, hidden] and swizzled UE8M0 scales). +struct MinimaxM3Sm107Tp4McAllreduceNormParams +{ + __nv_bfloat16 const* x{nullptr}; //!< [numTokens, hidden] partial; in finalize mode the FC2 rows [rows, hidden]. + __nv_bfloat16 const* x2{nullptr}; //!< Optional second addend [numTokens, hidden]. + __nv_bfloat16 const* fc2{nullptr}; //!< Finalize mode: the permuted FC2 output (same buffer as x); null otherwise. + int32_t const* perm{nullptr}; //!< Finalize mode: [numTokens, topk] expanded index -> permuted row, -1 dropped. + void const* scales{nullptr}; //!< Finalize mode: [numTokens, topk] routing weights (fp32 or bf16). + bool scaleFp32{false}; + int32_t topk{0}; + int32_t numTokens{0}; + int32_t hidden{0}; + __nv_bfloat16 const* residual{nullptr}; //!< [numTokens, hidden] + __nv_bfloat16 const* normWeight{nullptr}; //!< [hidden] + float eps{0.F}; + int32_t* wsUc{nullptr}; //!< This rank's unicast view of the exchange (int32 words, re-armed after reading). + int32_t* wsMc{nullptr}; //!< The multicast mapping of the exchange. + int32_t* flags{nullptr}; //!< Per-CTA call-parity counters, [maxTokens * 8]. + int32_t rank{0}; + int32_t world{0}; + __nv_bfloat16* outNormed{nullptr}; + __nv_bfloat16* outResidual{nullptr}; + uint8_t* outQuant{nullptr}; //!< Optional E4M3 [numTokens, hidden]. + uint8_t* outSf{nullptr}; //!< Optional UE8M0 scales, 128x4-swizzled. + //! 0: push x (+ x2) here; 2: a producer already pushed x and launched this grid after its pushes; 3: as 2, with x + //! in exchange set A and x2 in set B. + int32_t prepushed{0}; + //! Optional Eagle3 capture of a capture layer's boundary: capOut[t * capStride + c] = the m3_spec_tail + //! capture_norm of the updated residual row with weight capWeight [hidden] and capEps, bit for bit (hidden 6144, + //! not the drafter boundary). + __nv_bfloat16* capOut{nullptr}; + int64_t capStride{0}; + __nv_bfloat16 const* capWeight{nullptr}; + float capEps{0.F}; +}; + +//! The (bucket, epilogue options) pairs this launcher is built for: (8, 7), (8, 15), (16, 15) and (48, 15). +bool isMinimaxM3Sm107Tp4McAllreduceNormSupported(int32_t maxTokens, int32_t opts); + +//! Launch with 8-CTA clusters of 128 threads, one cluster per token. `maxTokens` selects the exchange bucket; +//! `opts` the epilogue variant (every variant is bit-identical). +void invokeMinimaxM3Sm107Tp4McAllreduceNorm(MinimaxM3Sm107Tp4McAllreduceNormParams const& params, int32_t maxTokens, + int32_t opts, bool pdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.cu new file mode 100644 index 000000000000..a0d5ac50025a --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.cu @@ -0,0 +1,340 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.h" + +#include +#include +#include + +// MiniMax-M3 NVFP4 QKV + indexer norm/RoPE/KV-insert producer with an early PDL trigger, at the SM107 TP4 heads. +// +// The SM107 TP4 target's copy of the SM107 TP2 producerPw2.cu (itself a copy of minimaxM3/producerPw.cu, which the +// sm_103 targets keep), for its pre-wait step inputs. Same math and +// outputs as minimaxM3Nvfp4QKVIndexerNormRopeKVInsertKernel (fusedQKNormRopeKernel.cu, built with --use_fast_math like +// the rest of the library), specialized to 16 query, 1 KV and 1 index head per rank (one index-K head), head_dim 128, +// rotary_dim 64, page 128, NVFP4 main KV and FP8 index-K. One warp per (token, packed head); K and V rows go to their +// head's block of the HND pools (the native kernel's head strides). +// +// The decode producer is latency bound: after griddepcontrol.wait the native kernel runs a chain of dependent global +// loads (packed row -> norm weight -> position id -> RoPE row -> cache slot -> quant scale). This kernel reads the +// constant norm weights and KV quant scale before griddepcontrol.wait and issues the packed row, position id and cache +// slot loads together after it, followed by the RoPE row; with `prewaitStepInputs` the position id, the slot and the +// RoPE row are read before the wait too, leaving only the packed row after it. With `early` it also calls +// griddepcontrol.launch_dependents on entry: consumers read its outputs only after their own griddepcontrol.wait, which +// waits for this grid to complete, so the early trigger only lets the next kernel launch and run its pre-wait prologue +// while this one runs. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +namespace producerPw4 +{ +namespace +{ + +constexpr int kHeadDim = 128; +constexpr int kRotaryDim = 64; +constexpr int kPageSize = 128; +constexpr int kElems = kHeadDim / 32; +constexpr int kBlock = 256; + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ uint32_t sf_group_mask4() +{ + uint32_t laneId; + asm("mov.u32 %0, %%laneid;" : "=r"(laneId)); + return 0xFu << (laneId & ~3u); +} + +__device__ __forceinline__ uint32_t fp32x8_to_e2m1(float (&a)[8]) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + uint32_t val; + asm volatile( + "{\n" + ".reg .b8 byte0;\n" + ".reg .b8 byte1;\n" + ".reg .b8 byte2;\n" + ".reg .b8 byte3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte0, %2, %1;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte1, %4, %3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte2, %6, %5;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte3, %8, %7;\n" + "mov.b32 %0, {byte0, byte1, byte2, byte3};\n" + "}" + : "=r"(val) + : "f"(a[0]), "f"(a[1]), "f"(a[2]), "f"(a[3]), "f"(a[4]), "f"(a[5]), "f"(a[6]), "f"(a[7])); + return val; +#else + return 0; +#endif +} + +__device__ __forceinline__ void store_fp8x4(__nv_fp8_e4m3* out, int64_t off, float const (&e)[kElems]) +{ + __nv_fp8x2_e4m3 const low(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const high(make_float2(e[2], e[3])); + *reinterpret_cast(out + off) = static_cast(low.__x) | (static_cast(high.__x) << 16); +} + +__device__ __forceinline__ uint16_t quantize_fp4x4(float const (&e)[4], float secondLevelScale, uint8_t* scaleOut) +{ + __nv_bfloat16 rounded[4]; + float localMax = 0.0F; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + rounded[i] = __float2bfloat16_rn(e[i]); + localMax = fmaxf(localMax, fabsf(__bfloat162float(rounded[i]))); + } + uint32_t const groupMask = sf_group_mask4(); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 1)); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 2)); + float const scaleValue = secondLevelScale * (localMax * rcp_approx_ftz(6.0F)); + __nv_fp8_e4m3 const encodedScale(scaleValue); + if (scaleOut != nullptr) + *scaleOut = encodedScale.__x; + float const narrowedScale = static_cast(encodedScale); + float const outputScale + = localMax != 0.0F ? rcp_approx_ftz(narrowedScale * rcp_approx_ftz(secondLevelScale)) : 0.0F; + float q[8] = {}; +#pragma unroll + for (int i = 0; i < 4; ++i) + q[i] = __bfloat162float(rounded[i]) * outputScale; + return static_cast(fp32x8_to_e2m1(q)); +} + +template +__global__ void __launch_bounds__(kBlock) m3_producer_kernel(Params p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + constexpr int kTotalHeads = kHeadsQ + 2 * kHeadsKV + kHeadsIndex + 1; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (kEarly) + { + asm volatile("griddepcontrol.launch_dependents;"); + } +#endif + int const warpId = threadIdx.x / 32; + int const laneId = threadIdx.x % 32; + int const globalWarp = blockIdx.x * (kBlock / 32) + warpId; + int const tokenIdx = globalWarp / kTotalHeads; + int const localHead = globalWarp % kTotalHeads; + if (tokenIdx >= p.numTokens) + return; + + constexpr int kBegin = kHeadsQ; + constexpr int vBegin = kBegin + kHeadsKV; + constexpr int indexQBegin = vBegin + kHeadsKV; + constexpr int indexKHead = indexQBegin + kHeadsIndex; + bool const isQ = localHead < kBegin; + bool const isK = localHead >= kBegin && localHead < vBegin; + bool const isV = localHead >= vBegin && localHead < indexQBegin; + bool const isIndexQ = localHead >= indexQBegin && localHead < indexKHead; + bool const isIndexK = localHead == indexKHead; + + // Constant data read before the wait: norm weights and the KV quant scale. + float w[kElems] = {}; + float quantScale = 0.0F; + __nv_bfloat16 const* normWeight + = isQ ? p.qWeight : (isK ? p.kWeight : (isIndexQ ? p.indexQWeight : p.indexKWeight)); + if (!isV) + { + uint2 const wv = *reinterpret_cast(normWeight + laneId * kElems); + float2 const w01 = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&wv.x)); + float2 const w23 = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&wv.y)); + w[0] = w01.x; + w[1] = w01.y; + w[2] = w23.x; + w[3] = w23.y; + } + if (isK || isV) + { + quantScale = p.kvQuantScale[isV ? 2 : 1]; + } + + int64_t const inputOffset = (static_cast(tokenIdx) * kTotalHeads + localHead) * kHeadDim + laneId * kElems; + bool const rotates = !isV && laneId * kElems < kRotaryDim; + bool const slotted = isK || isV || isIndexK; + int const coefficient = (laneId * kElems) % (kRotaryDim / 2); + // kPrewait: the position id, the cache slot and the RoPE row are read before the wait. They are per-step metadata + // written before the replay (host-to-device) or by plain-launched kernels earlier in the step (position_ids by the + // step's input preprocessing, the slots by msa_kv_update or its torch fallback), so they are final before this + // grid can start whatever its predecessors trigger; the RoPE table is constant. + int positionId = 0; + int slot = 0; + float4 c = make_float4(0.0F, 0.0F, 0.0F, 0.0F); + float4 s = c; + auto const loadStepInputs = [&]() + { + positionId = rotates ? p.positionIds[tokenIdx] : 0; + slot = slotted ? p.outCacheLoc[tokenIdx] : 0; + if (rotates) + { + int64_t const ropeRow = static_cast(positionId) * kRotaryDim; + c = *reinterpret_cast(p.rotaryCosSin + ropeRow + coefficient); + s = *reinterpret_cast(p.rotaryCosSin + ropeRow + kRotaryDim / 2 + coefficient); + } + }; + if constexpr (kPrewait) + { + loadStepInputs(); + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + // The packed row comes from the predecessor grid; without kPrewait the step inputs are issued with it, so the + // native kernel's serial chain of dependent loads becomes two round trips. + uint2 const packed = *reinterpret_cast(p.packed + inputOffset); + if constexpr (!kPrewait) + { + loadStepInputs(); + } + float cosine[kElems] = {c.x, c.y, c.z, c.w}; + float sine[kElems] = {s.x, s.y, s.z, s.w}; + + float e[kElems]; + float sumSquares = 0.0F; +#pragma unroll + for (int pair = 0; pair < 2; ++pair) + { + float2 const v = __bfloat1622float2( + *reinterpret_cast<__nv_bfloat162 const*>(reinterpret_cast(&packed) + pair)); + e[2 * pair] = v.x; + e[2 * pair + 1] = v.y; + if (!isV) + sumSquares += v.x * v.x + v.y * v.y; + } + + if (!isV) + { +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + sumSquares += __shfl_xor_sync(0xffffffff, sumSquares, mask, 32); + float const rmsReciprocal = rsqrtf(sumSquares / static_cast(kHeadDim) + p.eps); +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + e[i] *= rmsReciprocal * (1.0F + w[i]); + } + constexpr int kPairOffset = (kRotaryDim / 2) / kElems; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + int const dim = laneId * kElems + i; + float paired = __shfl_xor_sync(0xffffffff, e[i], kPairOffset); + if (dim < kRotaryDim) + { + if (dim < kRotaryDim / 2) + paired = -paired; + // The native kernel's contraction of e * cos + paired * sin; spelled out so this build matches it. + e[i] = __fmaf_rn(paired, sine[i], __fmul_rn(e[i], cosine[i])); + } + } + } + + if (isQ) + { + store_fp8x4(p.qOut, (static_cast(tokenIdx) * kHeadsQ + localHead) * kHeadDim + laneId * kElems, e); + return; + } + if (isIndexQ) + { + store_fp8x4(p.indexQOut, + (static_cast(tokenIdx) * kHeadsIndex + localHead - indexQBegin) * kHeadDim + laneId * kElems, e); + return; + } + + if (slot < 0) + return; + int const page = slot / kPageSize; + if (page >= p.numPages) + return; + int const withinPage = slot & (kPageSize - 1); + if (isIndexK) + { + store_fp8x4(p.indexK, + static_cast(page) * p.indexPageStride + static_cast(withinPage) * p.indexTokenStride + + laneId * kElems, + e); + return; + } + + int const head = isK ? localHead - kBegin : localHead - vBegin; + int const plane = isV ? 1 : 0; + int const scaleColumn = laneId / 4; + int64_t const scaleWithinPage = isV + ? static_cast(withinPage / 4) * 32 + static_cast(scaleColumn) * 4 + withinPage % 4 + : static_cast(withinPage) * 8 + scaleColumn; + int64_t const scaleOffset = static_cast(page) * p.scalePageStride + + static_cast(plane) * p.scalePlaneStride + static_cast(head) * p.scaleHeadStride + + scaleWithinPage; + uint8_t* scaleOut = laneId % 4 == 0 ? p.kvScale + scaleOffset : nullptr; + uint16_t const packedFp4 = quantize_fp4x4(e, quantScale, scaleOut); + int64_t const dataOffset = static_cast(page) * p.dataPageStride + + static_cast(plane) * p.dataPlaneStride + static_cast(head) * p.dataHeadStride + + static_cast(withinPage) * p.dataTokenStride + laneId * 2; + *reinterpret_cast(p.kvData + dataOffset) = packedFp4; +#endif +} + +template +cudaError_t launch(Params const& p, int numTokens, bool pdl, cudaStream_t stream) +{ + constexpr int kTotalHeads = kHeadsQ + 2 * kHeadsKV + kHeadsIndex + 1; + int const warps = numTokens * kTotalHeads; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3((warps + kBlock / 32 - 1) / (kBlock / 32)); + cfg.blockDim = dim3(kBlock); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, m3_producer_kernel, p); +} + +} // namespace + +cudaError_t invokeProducerPw4( + Params const& p, int numTokens, bool pdl, bool early, bool prewaitStepInputs, cudaStream_t stream) +{ + if (numTokens <= 0) + return cudaSuccess; + if (prewaitStepInputs) + { + return early ? launch(p, numTokens, pdl, stream) : launch(p, numTokens, pdl, stream); + } + return early ? launch(p, numTokens, pdl, stream) : launch(p, numTokens, pdl, stream); +} + +} // namespace producerPw4 +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.h new file mode 100644 index 000000000000..6f8aa54a812b --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.h @@ -0,0 +1,76 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +//! The MiniMax-M3 NVFP4 QKV + indexer RMSNorm / RoPE / KV-insert producer at the SM107 TP4 heads (16 query, 1 KV and +//! 1 index head per rank) with its constant loads before the PDL grid-dependency wait and an optional early dependents +//! trigger. Same outputs, bit for bit, as minimaxM3Nvfp4QKVIndexerNormRopeKVInsert. See producerPw4.cu. +namespace producerPw4 +{ + +struct Params +{ + __nv_bfloat16 const* packed; + __nv_fp8_e4m3* qOut; + __nv_fp8_e4m3* indexQOut; + uint8_t* kvData; + uint8_t* kvScale; + __nv_fp8_e4m3* indexK; + int const* outCacheLoc; + float const* kvQuantScale; + int64_t dataPageStride, dataPlaneStride, dataHeadStride, dataTokenStride; + int64_t scalePageStride, scalePlaneStride, scaleHeadStride; + int64_t indexPageStride, indexTokenStride, numPages; + int numTokens; + float eps; + __nv_bfloat16 const* qWeight; + __nv_bfloat16 const* kWeight; + __nv_bfloat16 const* indexQWeight; + __nv_bfloat16 const* indexKWeight; + float const* rotaryCosSin; + int const* positionIds; +}; + +//! The heads per rank this producer is built for. +constexpr int kHeadsQ = 16; +constexpr int kHeadsKV = 1; +constexpr int kHeadsIndex = 1; + +//! Launch over `numTokens` tokens, one warp per (token, packed head). `prewaitStepInputs` reads the position ids, the +//! cache slots and the RoPE rows before the grid-dependency wait (they must not be written by a PDL predecessor). +cudaError_t invokeProducerPw4( + Params const& p, int numTokens, bool pdl, bool early, bool prewaitStepInputs, cudaStream_t stream); + +} // namespace producerPw4 +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.cu new file mode 100644 index 000000000000..a9f4ae159af6 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.cu @@ -0,0 +1,944 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.h" + +#include "tensorrt_llm/common/assert.h" + +#include +#include + +// Fused sampled verification for the MiniMax-M3 Eagle3 target head (TP-sharded vocabulary, decode rows <= 48). +// +// sm_107 (Rubin) TP4 copy of kernels/minimaxM3Sm107Tp2/sampledAccept.cu: identical algorithm and arithmetic as the +// shared kernels/minimaxM3/sampledAccept.cu, with the TP2 copy's launch (a row's CTAs as one thread-block cluster) and +// bounded record polls. The vocab shard limit stays the shared kernel's (kMaxVs 51200 >= the TP4 shard 50016). +// +// Replaces, per decode step, the full-vocabulary all-gather of the target logits, the fused sampling kernel and the +// strict-acceptance ops for rows sampled with temperature and top-p (the AgentX contract). Rows with top-k or min-p +// active must stay on the generic path; the caller decides. +// +// Sampling is top-p rejection sampling, distributed over the TP ranks. With w = exp(logit / T - max) over the FULL +// vocabulary and Z = sum(w), a token is in the nucleus iff the mass strictly above its weight is at most top_p * Z +// (the fused sampling kernel's kept set, ties included). A candidate is drawn proportionally to w and accepted iff it +// is in the nucleus; a rejected candidate lies below the nucleus, so the next candidate is drawn over the tokens +// heavier than it. The accepted token is distributed as w restricted to the nucleus. The first candidate lands in the +// nucleus with probability >= top_p, so most rows decide on their second exchange. +// +// Per verify row, 8 CTAs (rows <= 32) or 4 CTAs (more rows) on every rank each own a slice of the rank's logits shard, +// staged in shared memory once. A candidate is drawn in two stages without per-token noise: each CTA draws a local +// candidate by inverse CDF over its slice, and the CTA whose log-mass plus Gumbel noise is largest supplies the +// candidate (CTA j wins with probability T_j / sum T, its token with w / T_j). Uniforms come from a counter-based +// Philox keyed by the request's seed and offset, the row, the attempt and the CTA. Each round every CTA multicasts one +// 4-word record (key, candidate logit, and max and mass in round 0 or the mass above the current candidate later) and +// reads the records of all CTAs of the row on all ranks. Every CTA takes the same decision from the same records in the +// same order, so all ranks agree without any intra-GPU handshake. Words carry the call's tag: a slot needs no clearing +// and two call halves keep a rank one call ahead from overwriting a slot still being read. +// +// SA-1R (Args.one_round_k 9 / 82; exact on 39,968 rows, eager + graph): each CTA +// also multicasts its largest bf16 logits with the round-0 record (3 per extra word). After round 0 every CTA knows M, +// the candidate logit cv and wc = __expf(cv - M); round 1 would accept iff S <= target, S = the fp32 sum (its tree) of +// w = __expf(__fmul_rn(x, tInv) - M) over every token with w > wc. x -> w is monotone (__fmul_rn by tInv > 0, fsub, +// __expf: checked exhaustively over all non-positive fp32 inputs), so if no CTA's unlisted value can beat wc (82: the +// bound tau; 9: the 9th listed), the listed w > wc are exactly round 1's summands, bitwise. Two fp32 sums of the same +// m non-negative terms differ by < m * 2^-23 * S, so accepting iff S_1r + m * 2^-21 * S_1r (round up) <= target +// implies round 1 accepts; m <= 1 is exact. Otherwise round 1 runs as before. An accepted row outputs the round-0 +// candidate, as the two-round kernel does, and consumes no Philox draw it would not. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +namespace sampledAccept +{ +namespace +{ +constexpr int kThreads = 256; +constexpr int kWarps = kThreads / 32; +constexpr int kMaxRec = kMaxWorld * kMaxSplits; +constexpr int kMaxAttempts = 16; // exchange slots per row, reused cyclically by the rounds (no round cap) +constexpr int kTagBits = 64 - 32 - kIdBits; +constexpr float kTempEpsilon = 1e-6f; // the fused sampling kernel's 1/T guard +// Record polls trap after this many spins (the multicast all-reduce's bound): a missing writer becomes a CUDA error, +// never a hang. +constexpr long long kMaxSpins = 1LL << 31; + +// SA-1R variant kTopK: 0 none; K exact top-K list; 10 * K + P pooled (each thread keeps its top P plus its (P + 1)-th +// as a bound; the CTA lists the pool's top K plus tau = max(K-th listed, max thread bound)). +__host__ __device__ constexpr int mode_k(int m) +{ + return m >= 10 ? m / 10 : m; +} + +__host__ __device__ constexpr int mode_pool(int m) +{ + return m >= 10 ? m % 10 : 0; +} + +__host__ __device__ constexpr int mode_vals(int m) +{ + return mode_k(m) + (mode_pool(m) ? 1 : 0); +} + +// Words per record: key, logit of key, f2, f3, then the round-0 list (3 bf16 + a 16-bit tag per word). +template +struct Rec +{ + static constexpr int kXw = (mode_vals(kTopK) + 2) / 3; + static constexpr int kEx = 4 + kXw; +}; + +// Multicast exchange, per call half: [kMaxAttempts][kMaxRows][world * kMaxSplits][kEx] u64. +template +__host__ __device__ inline long long half_words(int world) +{ + return static_cast(kMaxAttempts) * kMaxRows * world * kMaxSplits * Rec::kEx; +} + +__device__ __forceinline__ uint32_t ordered(float v) +{ + uint32_t const b = __float_as_uint(v); + return (b & 0x80000000u) ? ~b : (b | 0x80000000u); +} + +// bf16 bits <-> an order-preserving 16-bit key. +__device__ __forceinline__ uint32_t ord16(uint32_t b) +{ + return (b & 0x8000u) ? (~b & 0xFFFFu) : (b | 0x8000u); +} + +__device__ __forceinline__ uint32_t unord16(uint32_t k) +{ + return (k & 0x8000u) ? (k & 0x7FFFu) : (~k & 0xFFFFu); +} + +// Philox4x32-10. +__device__ __forceinline__ uint4 philox(uint4 c, uint2 k) +{ +#pragma unroll + for (int i = 0; i < 10; ++i) + { + uint32_t const hi0 = __umulhi(0xD2511F53u, c.x), lo0 = 0xD2511F53u * c.x; + uint32_t const hi1 = __umulhi(0xCD9E8D57u, c.z), lo1 = 0xCD9E8D57u * c.z; + c = make_uint4(hi1 ^ c.y ^ k.x, lo1, hi0 ^ c.w ^ k.y, lo0); + k.x += 0x9E3779B9u; + k.y += 0xBB67AE85u; + } + return c; +} + +// u in (0, 1): (k + 1/2) / 2^23, exact in fp32. +__device__ __forceinline__ float uniform(uint32_t bits) +{ + return (static_cast(bits >> 9) + 0.5f) * (1.0f / 8388608.0f); +} + +// Key on the wire: perturbed log-mass, then the candidate token id; 0 = no candidate. Tag bits are added on the wire. +__device__ __forceinline__ unsigned long long gkey(float lg, int id) +{ + return (static_cast(ordered(lg)) << kIdBits) + | static_cast((1u << kIdBits) - 1u - static_cast(id)); +} + +__device__ __forceinline__ int key_token(unsigned long long k) +{ + return static_cast((1u << kIdBits) - 1u - static_cast(k & ((1u << kIdBits) - 1u))); +} + +__device__ __forceinline__ void st_sys(unsigned long long* p, unsigned long long v) +{ + asm volatile("st.global.relaxed.sys.u64 [%0], %1;" ::"l"(p), "l"(v) : "memory"); +} + +__device__ __forceinline__ unsigned long long ld_sys(unsigned long long const* p) +{ + unsigned long long v; + asm volatile("ld.global.relaxed.sys.u64 %0, [%1];" : "=l"(v) : "l"(p) : "memory"); + return v; +} + +template +struct Shared +{ + __nv_bfloat16 row[kMaxVs / kSplits]; + float redf[kWarps]; + unsigned long long redk[kWarps]; + unsigned long long rk[kMaxRec]; + float rv[kMaxRec], r2[kMaxRec], r3[kMaxRec]; + float wsum[kWarps]; + int owner; + int pick; + float pickx; + bool last; + float dS[kWarps]; // SA-1R: per-warp partial mass above the candidate + int dM[kWarps]; // SA-1R: per-warp count | incomplete << 16 + uint32_t wtop[kWarps][kTopK > 0 ? mode_k(kTopK) : 1]; // SA-1R: per-warp top-K keys + uint32_t wbound[kWarps]; // SA-1R pooled: per-warp max thread bound + uint32_t ctop[kTopK > 0 ? mode_vals(kTopK) : 1]; // SA-1R: the CTA's top-K keys (descending), by rank + unsigned long long rx[kMaxRec][Rec::kXw > 0 ? Rec::kXw : 1]; +}; + +template +__device__ __forceinline__ float block_max(float v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, o)); + if ((threadIdx.x & 31) == 0) + s.redf[threadIdx.x >> 5] = v; + __syncthreads(); + float r = s.redf[0]; + for (int i = 1; i < kWarps; ++i) + r = fmaxf(r, s.redf[i]); + __syncthreads(); + return r; +} + +// Fixed-order block sum: identical result for identical inputs. +template +__device__ __forceinline__ float block_sum(float v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + if ((threadIdx.x & 31) == 0) + s.redf[threadIdx.x >> 5] = v; + __syncthreads(); + float r = 0.f; + for (int i = 0; i < kWarps; ++i) + r += s.redf[i]; + __syncthreads(); + return r; +} + +template +__device__ __forceinline__ unsigned long long block_max_u64(unsigned long long v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + unsigned long long const w = __shfl_xor_sync(0xffffffffu, v, o); + v = w > v ? w : v; + } + if ((threadIdx.x & 31) == 0) + s.redk[threadIdx.x >> 5] = v; + __syncthreads(); + unsigned long long r = s.redk[0]; + for (int i = 1; i < kWarps; ++i) + r = s.redk[i] > r ? s.redk[i] : r; + __syncthreads(); + return r; +} + +// Exclusive prefix of v over the threads in thread order, and the total. +template +__device__ __forceinline__ float block_excl_scan(float v, float& total, S& s) +{ + int const lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + float inc = v; +#pragma unroll + for (int o = 1; o < 32; o <<= 1) + { + float const t = __shfl_up_sync(0xffffffffu, inc, o); + if (lane >= o) + inc += t; + } + if (lane == 31) + s.wsum[warp] = inc; + __syncthreads(); + float base = 0.f; + total = 0.f; + for (int i = 0; i < kWarps; ++i) + { + base += i < warp ? s.wsum[i] : 0.f; + total += s.wsum[i]; + } + __syncthreads(); + return base + inc - v; +} + +// Warp-wide integer max / sum / or (the redux instructions from sm_80; a butterfly otherwise, same results). +__device__ __forceinline__ uint32_t warp_max_u32(uint32_t v) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 800) + return __reduce_max_sync(0xffffffffu, v); +#else +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + uint32_t const w = __shfl_xor_sync(0xffffffffu, v, o); + v = w > v ? w : v; + } + return v; +#endif +} + +__device__ __forceinline__ uint32_t warp_add_u32(uint32_t v) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 800) + return __reduce_add_sync(0xffffffffu, v); +#else +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + return v; +#endif +} + +__device__ __forceinline__ uint32_t warp_or_u32(uint32_t v) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 800) + return __reduce_or_sync(0xffffffffu, v); +#else +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v |= __shfl_xor_sync(0xffffffffu, v, o); + return v; +#endif +} + +// Insert key into a descending register list of K (fully unrolled: no local memory). +template +__device__ __forceinline__ void topk_insert(uint32_t (&a)[K], uint32_t key) +{ +#pragma unroll + for (int j = 0; j < K; ++j) + { + uint32_t const hi = key > a[j] ? key : a[j]; + key = key > a[j] ? a[j] : key; + a[j] = hi; + } +} + +template +__device__ __forceinline__ void topk_pop(uint32_t (&a)[K]) +{ +#pragma unroll + for (int j = 0; j + 1 < K; ++j) + a[j] = a[j + 1]; + a[K - 1] = 0u; +} + +// Per-thread insertion with an early reject: after the first few elements almost every key fails the K-th test. +template +__device__ __forceinline__ void thread_insert(uint32_t (&a)[K], uint32_t key) +{ + if (key > a[K - 1]) + topk_insert(a, key); +} + +// The warp's K largest keys (descending) into s.wtop[warp]; published by the caller's next block barrier. +template +__device__ __forceinline__ void warp_topk(uint32_t (&a)[K], S& s) +{ + int const lane = threadIdx.x & 31, warp = threadIdx.x >> 5; +#pragma unroll + for (int j = 0; j < K; ++j) + { + uint32_t const m = warp_max_u32(a[0]); + if (a[0] == m && m != 0u) + topk_pop(a); + if (lane == 0) + s.wtop[warp][j] = m; + } +} + +// Pooled: K rounds over each lane's P pool keys (descending) and the warp's max thread bound. +template +__device__ __forceinline__ void warp_pool(uint32_t (&a)[P + 1], S& s) +{ + int const lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + uint32_t const bound = warp_max_u32(a[P]); + uint32_t pl[P]; +#pragma unroll + for (int j = 0; j < P; ++j) + pl[j] = a[j]; +#pragma unroll + for (int j = 0; j < K; ++j) + { + uint32_t const m = warp_max_u32(pl[0]); + if (pl[0] == m && m != 0u) + topk_pop

(pl); + if (lane == 0) + s.wtop[warp][j] = m; + } + if (lane == 0) + s.wbound[warp] = bound; +} + +// A row's kSplits CTAs wait for each other's records, so they launch as one thread-block cluster: the hardware +// schedules a cluster's CTAs together, whatever other streams hold. Rows never wait on other rows. +template +__global__ void __launch_bounds__(kThreads, 2) sampled_accept_kernel(Args const a) +{ + using R = Rec; + extern __shared__ __align__(16) unsigned char smem[]; + Shared& s = *reinterpret_cast*>(smem); + float constexpr kNegInf = -__builtin_huge_valf(); + int const tid = threadIdx.x; + int const row = blockIdx.x / kSplits; + int const split = blockIdx.x % kSplits; + int const world = a.world; + int const nrec = world * kSplits; + // The row's sampling parameters are step inputs (the spec metadata, written before the target forward), not the + // predecessor head GEMM's output, so with prewait_params they load before the dependency wait and overlap the + // head's tail; the same values either way. + float temperature = 0.f, topP = 0.f; + long long seed_in = 0, offset_in = 0; + auto load_params = [&]() + { + temperature = a.temperatures[row]; + topP = a.top_ps[row]; + seed_in = a.seeds[row]; + offset_in = a.offsets[row]; + }; + if (a.prewait_params) + load_params(); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + if (!a.prewait_params) + load_params(); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + // early_trigger: the dependent grid (the first draft pass's input) may launch now and park at its own + // griddepcontrol.wait, which still returns only once this grid has completed and flushed; it skips the launch + // latency after this kernel ends. Otherwise dependents are released at the acceptance epilogue (the CTAs of + // splits 1.. exit without releasing). + if (a.early_trigger) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float const tempInv = 1.0f / (temperature + kTempEpsilon); + unsigned long long const seed = static_cast(seed_in); + unsigned long long const offset = static_cast(offset_in); + unsigned int const call = *reinterpret_cast(a.counters + 1); + // Wire tags: never 0 (fresh memory), different between calls of the same half and between consecutive laps of + // the slot ring (round r uses slot r % kMaxAttempts and lap parity (r / kMaxAttempts) & 1). A rank decides round + // r only after reading every record of round r, so no rank is more than one round ahead and a slot is rewritten + // only after every reader has consumed it; the lap bit tells the new record from the previous lap's. + auto ktag_of = [&](int r) -> unsigned long long + { + unsigned int const lap = static_cast(r / kMaxAttempts) & 1u; + return (((call >> 1) << 1) | lap) % ((1u << kTagBits) - 1u) + 1u; + }; + auto ftag_of = [&](int r) -> unsigned long long + { + unsigned int const lap = static_cast(r / kMaxAttempts) & 1u; + return static_cast(((((call >> 1) << 1) | lap) & 0x7FFFFFFFu) | 0x80000000u) << 32; + }; + // SA-1R round-0 list words: a 16-bit tag, never 0. + unsigned long long const xtag = (((call >> 1) << 1) & 0x7FFFu) | 0x8000u; + long long const rec_stride = static_cast(world) * kMaxSplits * R::kEx; + unsigned long long* const xu = a.ws_uc + static_cast(call & 1u) * half_words(world) + + static_cast(row) * rec_stride; + unsigned long long* const xm = a.ws_mc + static_cast(call & 1u) * half_words(world) + + static_cast(row) * rec_stride + (a.rank * kMaxSplits + split) * R::kEx; + long long const round_stride = static_cast(kMaxRows) * rec_stride; + + // ---- stage this CTA's slice and its max ------------------------------------------------------------------------- + __nv_bfloat16 const* src = a.logits + row * a.ld; + int const sl_len = ((a.Vs + kSplits - 1) / kSplits + 7) / 8 * 8; + int const s0 = min(split * sl_len, a.Vs); + int const n = min(s0 + sl_len, a.Vs) - s0; + int const id0 = a.vocab_offset + s0; + float tmax = kNegInf; + // SA-1R: each thread's largest (ord16(bf16) << 16 | ~index) keys while staging (only rows with a nucleus cut). + bool const want_x = kTopK > 0 && topP < 1.0f; + constexpr int kK = mode_k(kTopK), kPool = mode_pool(kTopK), kVals = mode_vals(kTopK); + constexpr int kTk = kPool ? kPool + 1 : (kK > 0 ? kK : 1); + uint32_t tk[kTk]; +#pragma unroll + for (int j = 0; j < kTk; ++j) + tk[j] = 0u; + bool const vec = (a.ld % 8) == 0 && (reinterpret_cast(a.logits) & 15) == 0; + int const nvec = vec ? n / 8 : 0; + for (int v = tid; v < nvec; v += kThreads) + { + uint4 const w = *reinterpret_cast(src + s0 + 8 * v); + *reinterpret_cast(&s.row[8 * v]) = w; + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&w); +#pragma unroll + for (int j = 0; j < 8; ++j) + tmax = fmaxf(tmax, __fmul_rn(__bfloat162float(e[j]), tempInv)); + if constexpr (kTopK > 0) + { + if (want_x) + { +#pragma unroll + for (int j = 0; j < 8; ++j) + { + uint32_t const key + = (ord16(__bfloat16_as_ushort(e[j])) << 16) | (0xFFFFu - static_cast(8 * v + j)); + if constexpr (kPool > 0) + topk_insert(tk, key); // branchless: P + 1 compare-swaps + else + thread_insert(tk, key); + } + } + } + } + for (int i = nvec * 8 + tid; i < n; i += kThreads) + { + s.row[i] = src[s0 + i]; + tmax = fmaxf(tmax, __fmul_rn(__bfloat162float(src[s0 + i]), tempInv)); + if constexpr (kTopK > 0) + { + if (want_x) + topk_insert( + tk, (ord16(__bfloat16_as_ushort(src[s0 + i])) << 16) | (0xFFFFu - static_cast(i))); + } + } + if constexpr (kTopK > 0) + { + if (tid < kVals) + s.ctop[tid] = 0u; // published with s.wtop by block_max's barrier + if (want_x) + { + if constexpr (kPool > 0) + warp_pool(tk, s); // s.wtop / s.wbound published by block_max's barrier + else + warp_topk(tk, s); // s.wtop published by block_max's barrier + } + } + float const smax = block_max(tmax, s); // the staged slice is visible to all threads after this barrier + if constexpr (kTopK > 0) + { + // SA-1R: the CTA's top K of the warp lists by rank (keys are unique): a lane quad per key, no serial merge; + // s.ctop is read after the round-0 draw's first barrier + constexpr int kN = kWarps * kK; + constexpr int kSp = kN * 4 <= kThreads ? 4 : 1; // threads per key + if (want_x && tid < kN * kSp) + { + uint32_t const* const wl = &s.wtop[0][0]; + int const part = tid % kSp; + uint32_t const key = wl[tid / kSp]; + int rank = 0; +#pragma unroll + for (int j = part; j < kN; j += kSp) + rank += wl[j] > key ? 1 : 0; + if constexpr (kSp == 4) + { + rank += __shfl_xor_sync(0xffffffffu, rank, 1); + rank += __shfl_xor_sync(0xffffffffu, rank, 2); + } + if (part == 0 && key != 0u && rank < kK) + s.ctop[rank] = key; + } + } + float M = 0.f, wc = 0.f, target = 0.f, cv = 0.f; // global max, candidate weight and logit, top_p * Z + unsigned long long ckey = 0ULL; + int const gid = a.rank * kMaxSplits + split; + for (int r = 0;; ++r) + { + // ---- local candidate by inverse CDF over the slice's tokens heavier than the candidate (w > 0 in round 0) + float const ref = r == 0 ? smax : M; + float const wmin = r == 0 ? 0.f : wc; + float ts = 0.f; + for (int i = tid; i < n; i += kThreads) + { + float const w = __expf(__fmul_rn(__bfloat162float(s.row[i]), tempInv) - ref); + ts += w > wmin ? w : 0.f; + } + float total; + float const excl = block_excl_scan(ts, total, s); + if constexpr (kTopK > 0) + { + // SA-1R round 0: the list words depend only on the staged slice, so they go out now (s.ctop / s.wbound are + // published by the barrier above), one lane per word, ahead of the main record: off the readers' path + if (r == 0 && want_x && tid < R::kXw) + { + uint32_t tau = 0u; + if constexpr (kPool > 0) + { + tau = s.ctop[kK - 1]; // the largest unlisted pool key is <= the K-th listed +#pragma unroll + for (int wq = 0; wq < kWarps; ++wq) + tau = s.wbound[wq] > tau ? s.wbound[wq] : tau; + } + unsigned long long w = xtag; +#pragma unroll + for (int t = 0; t < 3; ++t) + { + int const e = 3 * tid + t; + uint32_t const kv = e < kK ? s.ctop[e] : (kPool > 0 && e == kK ? tau : 0u); + // missing entries: -inf (bf16 0xFF80), w = 0 + uint32_t const b = kv != 0u ? unord16(kv >> 16) : 0xFF80u; + w |= static_cast(b) << (16 * (3 - t)); + } + st_sys(xm + 4 + tid, w); // round 0 = slot 0 + } + } + uint4 const rnd = philox(make_uint4(static_cast(gid), static_cast(row + 64 * r), + static_cast(offset), static_cast(offset >> 32)), + make_uint2(static_cast(seed), static_cast(seed >> 32))); + float const tgt = uniform(rnd.x) * total; + if (tid == 0) + s.owner = -1; + __syncthreads(); + if (ts > 0.f && excl <= tgt) + atomicMax(&s.owner, tid); + __syncthreads(); + if (tid == s.owner) + { + float acc = excl; + int pick = -1; + float px = 0.f; + if constexpr (kIlpWalk) + { + // SA-PW: the owner's strided tokens 8 at a time. The 8 x and w (the same expressions) are + // independent, so their MUFU work overlaps; the serial sum then adds them in the same order with + // the same early exit: the same acc sequence, pick and px as the one-token walk, bit for bit. + constexpr int kW = 8; + bool done = false; + for (int i0 = tid; i0 < n && !done; i0 += kW * kThreads) + { + float xs[kW], wv[kW]; +#pragma unroll + for (int k = 0; k < kW; ++k) + { + int const i = i0 + k * kThreads; + xs[k] = i < n ? __fmul_rn(__bfloat162float(s.row[i]), tempInv) : 0.f; + wv[k] = __expf(xs[k] - ref); + } +#pragma unroll + for (int k = 0; k < kW; ++k) + { + int const i = i0 + k * kThreads; + if (!done && i < n && wv[k] > wmin) + { + acc += wv[k]; + pick = i; + px = xs[k]; + if (acc > tgt) + done = true; + } + } + } + } + else + { + for (int i = tid; i < n; i += kThreads) + { + float const x = __fmul_rn(__bfloat162float(s.row[i]), tempInv); + float const w = __expf(x - ref); + if (w > wmin) + { + acc += w; + pick = i; + px = x; + if (acc > tgt) + break; + } + } + } + s.pick = pick; + s.pickx = px; + } + __syncthreads(); + unsigned long long sb = 0ULL; + if (total > 0.f && s.owner >= 0) + sb = gkey(__logf(total) + ref - logf(-logf(uniform(rnd.y))), id0 + s.pick); + float const smab = total; + unsigned long long const ktag = ktag_of(r); + unsigned long long const ftag = ftag_of(r); + unsigned long long* const xr = xu + (r % kMaxAttempts) * round_stride; + bool const with_x = want_x && r == 0; + if (tid == 0) + { + unsigned long long* const xw = xm + (r % kMaxAttempts) * round_stride; + st_sys(xw + 0, (sb << kTagBits) | ktag); + st_sys(xw + 1, ftag | __float_as_uint(sb != 0ULL ? s.pickx : 0.f)); + st_sys(xw + 2, ftag | __float_as_uint(r == 0 ? smax : smab)); + st_sys(xw + 3, ftag | __float_as_uint(r == 0 ? smab : 0.f)); + } + // ---- every record of the row, every rank --------------------------------------------------------------- + int const nw = with_x ? R::kEx : 4; + for (int t = tid; t < nrec * nw; t += kThreads) + { + int const q = t / nw, f = t % nw; + unsigned long long const* const p = xr + (q / kSplits * kMaxSplits + q % kSplits) * R::kEx + f; + unsigned long long v; + long long spins = 0; + if (f == 0) + { + do + { + v = ld_sys(p); + if (++spins > kMaxSpins) + __trap(); + } while ((v & ((1ULL << kTagBits) - 1ULL)) != ktag); + s.rk[q] = v >> kTagBits; + } + else if (f < 4) + { + do + { + v = ld_sys(p); + if (++spins > kMaxSpins) + __trap(); + } while ((v & 0xFFFFFFFF00000000ULL) != ftag); + float const fv = __uint_as_float(static_cast(v)); + (f == 1 ? s.rv : f == 2 ? s.r2 : s.r3)[q] = fv; + } + else + { + do + { + v = ld_sys(p); + if (++spins > kMaxSpins) + __trap(); + } while ((v & 0xFFFFULL) != xtag); + s.rx[q][f - 4] = v; + } + } + __syncthreads(); + // ---- the decision, identical in every CTA of the row on every rank: fixed record order ----------------- + unsigned long long gk = 0ULL; + float gv = 0.f; + for (int q = 0; q < nrec; ++q) + { + if (s.rk[q] > gk) + { + gk = s.rk[q]; + gv = s.rv[q]; + } + } + bool accept; + if (r == 0) + { + M = kNegInf; + for (int q = 0; q < nrec; ++q) + M = fmaxf(M, s.r2[q]); + float Z = 0.f; + for (int q = 0; q < nrec; ++q) + Z += s.r2[q] == kNegInf ? 0.f : s.r3[q] * __expf(s.r2[q] - M); + target = topP * Z; + accept = !(topP < 1.0f); // no nucleus cut: the first draw stands + ckey = gk; + cv = gv; + if constexpr (kTopK > 0) + { + // SA-1R: every thread takes (record, entry) items; the per-warp partials are published by the round's + // barrier below and every thread combines them in warp order after it (no new barrier) + if (with_x) + { + float const wcand = __expf(cv - M); // = round 1's wmin + float part = 0.f; + int cnt = 0, inc = 0; + for (int t = tid; t < nrec * kVals; t += kThreads) + { + int const q = t / kVals, e = t % kVals; + uint32_t const b = static_cast(s.rx[q][e / 3] >> (16 * (3 - e % 3))) & 0xFFFFu; + float const w = __expf( + __fmul_rn(__bfloat162float(__ushort_as_bfloat16(static_cast(b))), tempInv) + - M); // w bitwise as round 1 computes it + if (w > wcand) + { + if (kPool > 0 && e == kK) + inc = 1; // tau (the largest unlisted value) is heavier than the candidate + else + { + part += w; + ++cnt; + if (kPool == 0 && e == kK - 1) + inc = 1; // the K-th is heavier: unlisted heavier tokens may exist + } + } + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + part += __shfl_xor_sync(0xffffffffu, part, o); // any order: the margin covers it + cnt = static_cast(warp_add_u32(static_cast(cnt))); + inc = static_cast(warp_or_u32(static_cast(inc))); + if ((tid & 31) == 0) + { + s.dS[tid >> 5] = part; + s.dM[tid >> 5] = cnt | (inc << 16); + } + } + } + } + else + { + float S = 0.f; // mass strictly above the current candidate + for (int q = 0; q < nrec; ++q) + S += s.r2[q]; + accept = S <= target || gk == 0ULL; + if (!accept) + { + ckey = gk; + cv = gv; + } + } + wc = __expf(cv - M); + __syncthreads(); // records consumed before the next round's writes to s + bool decided = false; + if constexpr (kTopK > 0) + { + if (r == 0 && with_x) + { + float S1 = 0.f; + int m = 0, inc = 0; + for (int wq = 0; wq < kWarps; ++wq) + { + S1 += s.dS[wq]; + m += s.dM[wq] & 0xFFFF; + inc |= s.dM[wq] >> 16; + } + if (!inc) + { + float const margin = m <= 1 ? 0.f : S1 * static_cast(m) * (1.0f / 2097152.0f); + decided = __fadd_ru(S1, margin) <= target; + accept = accept || decided; + } + } + } + if (accept) + { + if (split != 0) + return; + if (tid == 0) + { + a.tokens[row] = key_token(ckey); + if (a.extra_attempts != nullptr && r > 1) + atomicAdd(a.extra_attempts, r - 1); + if (a.decided != nullptr && decided) + atomicAdd(a.decided, 1); + } + break; + } + } + + // ---- split 0: the last row to finish runs the strict acceptance ---------------------------------------------- +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (!a.early_trigger) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + __threadfence(); + __syncthreads(); + if (tid == 0) + s.last = atomicAdd(a.counters, 1u) == static_cast(a.rows) - 1; + __syncthreads(); + if (!s.last) + return; + __threadfence(); + if (tid == 0) + { + a.counters[0] = 0u; + a.counters[1] = call + 1u; + } + if (tid < a.nreq) + { + int const* tok = a.tokens; + int const b = tid; + int const k1 = a.draft_len + 1; + int nacc = 1; + bool run = true; + for (int k = 0; k < a.draft_len; ++k) + { + int const tt = *reinterpret_cast(tok + b * k1 + k); + a.acc_tokens[b * k1 + k] = tt; + run = run && a.draft[b * a.draft_len + k] == tt; + nacc += run ? 1 : 0; + } + a.acc_tokens[b * k1 + a.draft_len] = *reinterpret_cast(tok + b * k1 + a.draft_len); + if (a.force_vec != nullptr) + nacc = a.force_vec[b]; + else if (a.force_total > 0) + nacc = a.force_total; + a.num_acc[b] = nacc; + a.gather_ids[b] = static_cast(a.batch_idx[b]) * k1 + nacc - 1; + if (a.next_new != nullptr) + a.next_new[b * a.next_stride] + = *reinterpret_cast(tok + a.batch_idx[b] * k1 + nacc - 1); + } +} + +template +cudaError_t launch(Args const& a, bool use_pdl, cudaStream_t stream) +{ + static int max_clusters = 0; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(a.rows * kSplits); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = sizeof(Shared); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kSplits; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + if (max_clusters == 0) + { + cudaError_t err = cudaFuncSetAttribute(sampled_accept_kernel, + cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(sizeof(Shared))); + if (err == cudaSuccess) + { + err = cudaOccupancyMaxActiveClusters(&max_clusters, sampled_accept_kernel, &cfg); + } + if (err != cudaSuccess) + { + return err; + } + } + TLLM_CHECK_WITH_INFO(max_clusters >= 1, "m3_sampled_accept: a %d-CTA cluster does not fit on one GPC", kSplits); + return cudaLaunchKernelEx(&cfg, sampled_accept_kernel, a); +} + +template +cudaError_t launchRows(Args const& a, bool use_pdl, cudaStream_t stream) +{ + // TP4 caps 2 / 4 / 8 (8 / 16 / 32 rows) and TP8 cap 1 (4 rows): 8 CTAs per row, at most 256 CTAs (2 per SM); + // more rows: 4 CTAs per row. + if (a.ilp_walk) + { + return a.rows <= 32 ? launch<8, kTopK, true>(a, use_pdl, stream) : launch<4, kTopK, true>(a, use_pdl, stream); + } + return a.rows <= 32 ? launch<8, kTopK, false>(a, use_pdl, stream) : launch<4, kTopK, false>(a, use_pdl, stream); +} + +} // namespace + +long long sampledAcceptWsWords(int world, int oneRoundK) +{ + TLLM_CHECK_WITH_INFO( + oneRoundK == 0 || oneRoundK == 9 || oneRoundK == 82, "SA-1R variant %d is not built", oneRoundK); + return 2 * (oneRoundK == 9 ? half_words<9>(world) : oneRoundK == 82 ? half_words<82>(world) : half_words<0>(world)); +} + +cudaError_t invokeSampledAccept(Args const& a, bool usePdl, cudaStream_t stream) +{ + switch (a.one_round_k) + { + case 0: return launchRows<0>(a, usePdl, stream); + case 9: return launchRows<9>(a, usePdl, stream); + case 82: return launchRows<82>(a, usePdl, stream); + default: TLLM_CHECK_WITH_INFO(false, "SA-1R variant %d is not built", a.one_round_k); + } + return cudaErrorInvalidValue; +} + +} // namespace sampledAccept +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.h new file mode 100644 index 000000000000..0ebd153992d0 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.h @@ -0,0 +1,87 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ +//! Fused sampled (temperature + top-p) verification of the MiniMax-M3 Eagle3 target head over the TP-sharded +//! vocabulary, decode rows <= kMaxRows; see sampledAccept.cu. +namespace sampledAccept +{ + +constexpr int kMaxSplits = 8; // CTAs per verify row (at most) +constexpr int kMaxRows = 48; // verify rows per call +constexpr int kMaxWorld = 8; // TP ranks +constexpr int kMaxVs = 51200; // vocab shard width (TP4: 200064 / 4 = 50016) +constexpr int kIdBits = 18; // token ids < 2^18 + +struct Args +{ + __nv_bfloat16 const* logits; // [rows, Vs] this rank's shard + long long ld; + int rows, Vs, vocab_offset; + float const* temperatures; + float const* top_ps; + long long const* seeds; + long long const* offsets; + unsigned long long* ws_uc; + unsigned long long* ws_mc; + unsigned int* counters; // [0] finished rows, [1] call + int* tokens; // [rows] sampled token per row + int* extra_attempts; // optional [1]: rounds beyond the second, summed over rows + int rank, world; + int const* draft; + int const* batch_idx; + int* acc_tokens; + int* num_acc; + long long* gather_ids; + int draft_len, nreq, force_total; + int* next_new; + int next_stride; + int const* force_vec; + int prewait_params; // nonzero: the row's sampling parameters load before the PDL dependency wait + int early_trigger; // nonzero: dependents are released right after the dependency wait, not at the epilogue + // SA-1R (one-round decision): 0 the two-round kernel; 9 each CTA's top-9 logits ride its round-0 record; 82 pooled + // (each thread's top 2, the CTA lists the pool's top 8 plus the bound of everything unlisted). A row whose first + // candidate is provably in the nucleus then decides in round 0, with the same token as the two-round kernel. + int one_round_k; + int* decided; // optional [1]: rows decided in round 0 (SA-1R), summed + int ilp_walk; // nonzero: the owner walk takes its tokens 8 at a time (same weights, same summation order) +}; + +//! Words (uint64) of the multicast exchange workspace for `world` ranks and SA-1R variant `oneRoundK`: two call halves. +long long sampledAcceptWsWords(int world, int oneRoundK = 0); + +//! Launch the sampled verification over a.rows * (8 or 4) co-resident CTAs (8 per row up to 32 rows). +cudaError_t invokeSampledAccept(Args const& a, bool usePdl, cudaStream_t stream); + +} // namespace sampledAccept +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.cu new file mode 100644 index 000000000000..c93f5089bf2b --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.cu @@ -0,0 +1,312 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Top-16 block select for MiniMax-M3 decode rows of 129 to 9216 blocks. +// +// One 1024-thread CTA per (query, KV head) row holds the row's ordered score keys in registers (2, 4 or 9 per +// thread, by the row's valid count). The weakest of the 16 maxima of the interleaved groups `block % 16` is a lower +// bound on the 16th-largest key, so the top 16 is among the keys at or above it; those are compacted to shared +// memory and ranked exactly by (key, block). More than 1024 keys tied at the bound (degenerate rows) rank every +// candidate against the whole row (re-read). Rows of at most 64 candidates (the common case) skip the all-thread rank: +// warp 0 sorts them in registers (a 64-key bitonic network), keeps the 16 smallest (key, block) pairs and emits them, +// and the other warps exit. The row's valid count is a step input and is read before the grid dependency wait. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.h" + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +namespace +{ + +constexpr float kInitScore = 1.0e30f; +constexpr float kLocalScore = 1.0e29f; +constexpr uint32_t kNegInfKey = 0xff800000u; +constexpr int kItems = 9; // keys per thread: rows up to 9216 blocks (1M tokens + draft tokens) +constexpr int kWarpCandidates = 64; // candidates warp 0 ranks alone (2 per lane) + +// Descending order key: a smaller key is a larger score (NaN above +inf), as in the reference op. +__device__ __forceinline__ uint32_t orderedKey(float x) +{ + uint32_t const b = __float_as_uint(x); + return (b & 0x80000000u) ? b : (~b & 0x7fffffffu); +} + +__device__ __forceinline__ float applyForcing(float raw, int block, int initBlocks, long long localStart) +{ + if (block >= localStart) + return kLocalScore; + if (block < initBlocks) + return kInitScore; + return raw; +} + +struct SelectSmem +{ + uint32_t groupMin[16]; + unsigned long long cand[1024]; + int numCand; + unsigned long long picked[16]; +}; + +// One row with up to kItems * 1024 valid blocks; the caller picks kItems from the row's valid count. +// Returns -1 when sm.picked holds the row's picks, else the candidate count (at most kWarpCandidates, compacted to +// sm.cand and not yet ranked). The value is uniform across the CTA. +template +__device__ __forceinline__ int selectRow(SelectSmem& sm, float const* rowBase, long long blockStride, int valid, + int initBlocks, long long localStart, bool loadedTrigger) +{ + int const tid = threadIdx.x, lane = tid & 31; + uint32_t key[kItems]; + uint32_t localMin = 0xffffffffu; +#pragma unroll + for (int s = 0; s < kItems; ++s) + { + int const block = s * 1024 + tid; + key[s] = block < valid ? orderedKey(applyForcing(rowBase[block * blockStride], block, initBlocks, localStart)) + : 0xffffffffu; + localMin = min(localMin, key[s]); + } + if (valid <= 16) + { + if (tid < valid) + sm.picked[tid] = (static_cast(key[0]) << 32) | static_cast(tid); + if (loadedTrigger) + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + return -1; + } + // Group g = block % 16 = tid % 16: lanes l and l ^ 16 of every warp. + localMin = min(localMin, __shfl_xor_sync(0xffffffffu, localMin, 16)); + if (lane < 16) + atomicMin(&sm.groupMin[lane], localMin); + __syncthreads(); + // Every score of the row is loaded: release the sparse decode, so its launch and its pre-wait prologue overlap + // the rank below. This comes after this grid's own dependency wait, which the sparse decode relies on: it reads + // q and the seq_lens / block-table row before its own wait (sparseDecodeLc2.cu), safe only while every kernel + // between the producer and it releases its dependents after its own wait. + if (loadedTrigger) + asm volatile("griddepcontrol.launch_dependents;" ::: "memory"); + uint32_t bound = 0; +#pragma unroll + for (int g = 0; g < 16; ++g) + bound = max(bound, sm.groupMin[g]); +#pragma unroll + for (int s = 0; s < kItems; ++s) + { + int const block = s * 1024 + tid; + bool const isCand = block < valid && key[s] <= bound; + unsigned const ballot = __ballot_sync(0xffffffffu, isCand); + int base = 0; + if (lane == 0 && ballot) + base = atomicAdd(&sm.numCand, __popc(ballot)); + base = __shfl_sync(0xffffffffu, base, 0); + if (isCand) + { + int const i = base + __popc(ballot & ((1u << lane) - 1)); + if (i < 1024) + sm.cand[i] = (static_cast(key[s]) << 32) | static_cast(block); + } + } + __syncthreads(); + int const n = sm.numCand; + if (kFast && n <= kWarpCandidates) + return n; + if (n <= 1024) + { + for (int i = tid; i < n; i += 1024) + { + unsigned long long const mine = sm.cand[i]; + int rank = 0; + for (int j = 0; j < n; ++j) + rank += sm.cand[j] < mine; + if (rank < 16) + sm.picked[rank] = mine; + } + return -1; + } + // More than 1024 keys tied at the bound (degenerate rows): rank every candidate against the whole row. +#pragma unroll + for (int s = 0; s < kItems; ++s) + { + int const block = s * 1024 + tid; + if (block < valid && key[s] <= bound) + { + int rank = 0; + for (int j = 0; j < valid; ++j) + { + uint32_t const other = orderedKey(applyForcing(rowBase[j * blockStride], j, initBlocks, localStart)); + rank += other < key[s] || (other == key[s] && j < block); + } + if (rank < 16) + sm.picked[rank] = (static_cast(key[s]) << 32) | static_cast(block); + } + } + return -1; +} + +// Warp 0: the n <= kWarpCandidates candidates of sm.cand sorted ascending as 64 keys (element r * 32 + lane in +// register r); returns this lane's element (lanes 0..15 then hold the 16 smallest). +__device__ __forceinline__ unsigned long long warpSmallest(SelectSmem const& sm, int n, int lane) +{ + unsigned long long a = lane < n ? sm.cand[lane] : ~0ull; + unsigned long long b = lane + 32 < n ? sm.cand[lane + 32] : ~0ull; +#pragma unroll + for (int size = 2; size <= 64; size <<= 1) + { +#pragma unroll + for (int stride = size >> 1; stride > 0; stride >>= 1) + { + if (stride == 32) + { + unsigned long long const lo = min(a, b), hi = max(a, b); + a = lo; + b = hi; + } + else + { + unsigned long long const oa = __shfl_xor_sync(0xffffffffu, a, stride); + unsigned long long const ob = __shfl_xor_sync(0xffffffffu, b, stride); + bool const lower = (lane & stride) == 0; + bool const ascA = (lane & size) == 0, ascB = ((lane + 32) & size) == 0; + a = (lower == ascA) ? min(a, oa) : max(a, oa); + b = (lower == ascB) ? min(b, ob) : max(b, ob); + } + } + } + return a; +} + +// Warp 0: the picks (lane < 16 holds pick `lane`, ~0ull when absent) as ascending block ids, -1 for forced -inf picks +// and missing picks, padded at the tail. +__device__ __forceinline__ void emitPicks(unsigned long long c, int lane, int* outRow) +{ + uint32_t blockKey = ~0u; + if (lane < 16 && c != ~0ull && static_cast(c >> 32) != kNegInfKey) + blockKey = static_cast(c & 0xffffffffu); +#pragma unroll + for (int size = 2; size <= 32; size <<= 1) + { +#pragma unroll + for (int stride = size >> 1; stride > 0; stride >>= 1) + { + uint32_t const other = __shfl_xor_sync(0xffffffffu, blockKey, stride); + bool const ascending = (lane & size) == 0, lower = (lane & stride) == 0; + blockKey = (lower == ascending) ? min(blockKey, other) : max(blockKey, other); + } + } + if (lane < 16) + outRow[lane] = blockKey == ~0u ? -1 : static_cast(blockKey); +} + +// kFast = false is the select before the warp-0 tail (dispatch.SELECT_LONG_SL3 = False, the kill value). +template +__global__ void __launch_bounds__(1024) m3_select_long2_kernel(float const* __restrict__ scores, long long headStride, + long long blockStride, long long queryStride, int const* __restrict__ nValid, int* __restrict__ out, + long long outQueryStride, long long outHeadStride, int numKvHeads, int numBlocks, int initBlocks, int localBlocks, + bool earlyTrigger, bool loadedTrigger) +{ + __shared__ SelectSmem sm; + int const row = blockIdx.x, query = row / numKvHeads, head = row % numKvHeads; + int const tid = threadIdx.x, lane = tid & 31; + if (tid < 16) + { + sm.groupMin[tid] = 0xffffffffu; + sm.picked[tid] = ~0ull; + } + if (tid == 0) + sm.numCand = 0; + // With kFast the row's valid count is read before this grid's dependency wait: a step input (written between + // replays or at the step head), final before the wait, as for the sparse decode's pre-wait seq_lens / block-table + // reads. + int raw = 0; + if (kFast) + { + raw = nValid[query]; + __syncthreads(); + } +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + cudaGridDependencySynchronize(); + // The sparse decode that follows stages its table-independent inputs before its own grid wait; past this grid's + // wait those inputs are final, so it may launch now rather than at this grid's exit. + if (earlyTrigger) + cudaTriggerProgrammaticLaunchCompletion(); +#endif + if (!kFast) + { + raw = nValid[query]; + __syncthreads(); + } + int const valid = max(0, min(raw, numBlocks)); + long long const localStart = max(static_cast(raw) - localBlocks, 0LL); + float const* rowBase = scores + head * headStride + query * queryStride; + int n; + if (valid <= 2048) + n = selectRow<2, kFast>(sm, rowBase, blockStride, valid, initBlocks, localStart, loadedTrigger); + else if (valid <= 4096) + n = selectRow<4, kFast>(sm, rowBase, blockStride, valid, initBlocks, localStart, loadedTrigger); + else + n = selectRow(sm, rowBase, blockStride, valid, initBlocks, localStart, loadedTrigger); + int* const outRow = out + query * outQueryStride + head * outHeadStride; + if (n >= 0) + { + if (tid >= 32) + return; + emitPicks(warpSmallest(sm, n, lane), lane, outRow); + return; + } + __syncthreads(); + if (tid >= 32) + return; + emitPicks(lane < 16 ? sm.picked[lane] : ~0ull, lane, outRow); +} + +} // namespace + +cudaError_t invokeSelectLong2(float const* scores, int64_t headStride, int64_t blockStride, int64_t queryStride, + int32_t const* nValidBlocks, int32_t* out, int64_t outQueryStride, int64_t outHeadStride, int32_t numKvHeads, + int32_t numBlocks, int32_t totalQ, int32_t initBlocks, int32_t localBlocks, bool usePdl, bool earlyTrigger, + int32_t triggerPhase, bool fastTail, cudaStream_t stream) +{ + if (numBlocks > kSelectLong2MaxBlocks || triggerPhase < 0 || triggerPhase > 1) + { + return cudaErrorInvalidValue; + } + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(totalQ * numKvHeads); + cfg.blockDim = dim3(1024); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = usePdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, fastTail ? m3_select_long2_kernel : m3_select_long2_kernel, scores, + static_cast(headStride), static_cast(blockStride), static_cast(queryStride), + nValidBlocks, out, static_cast(outQueryStride), static_cast(outHeadStride), numKvHeads, + numBlocks, initBlocks, localBlocks, earlyTrigger, !earlyTrigger && triggerPhase == 1); +} + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.h new file mode 100644 index 000000000000..1e3683a3e7d1 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.h @@ -0,0 +1,52 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +//! Largest row (blocks per query) the long-row top-16 select covers: 9 keys per thread of one 1024-thread CTA. +constexpr int32_t kSelectLong2MaxBlocks = 9216; + +//! MiniMax-M3 top-16 block select over per-KV-head scores [numKvHeads, numBlocks, totalQ] (element strides given), +//! one 1024-thread CTA per (query, KV head). Same contract as minimax_m3_select_blocks: init / local forcing, the +//! per-query valid-block count, ties to the smaller block, forced -inf picks emitted as -1, ascending ids with -1 +//! tail padding. Writes out[query * outQueryStride + head * outHeadStride + 0..15]. The SM107 TP4 copy of +//! minimaxM3/selectLong with a choice of where the dependent grid (the sparse decode) is released: at this grid's exit +//! (triggerPhase 0), right after every score of the row is loaded (1), or, with earlyTrigger, right after this grid's +//! dependency wait. Every release point follows this grid's own dependency wait. fastTail ranks rows of at most 64 +//! candidates on warp 0 and reads the valid count before the wait (the same picks); false keeps the all-thread rank +//! (dispatch.SELECT_LONG_SL3 = False). +cudaError_t invokeSelectLong2(float const* scores, int64_t headStride, int64_t blockStride, int64_t queryStride, + int32_t const* nValidBlocks, int32_t* out, int64_t outQueryStride, int64_t outHeadStride, int32_t numKvHeads, + int32_t numBlocks, int32_t totalQ, int32_t initBlocks, int32_t localBlocks, bool usePdl, bool earlyTrigger, + int32_t triggerPhase, bool fastTail, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.cu new file mode 100644 index 000000000000..970b2aa21cf4 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.cu @@ -0,0 +1,89 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +namespace +{ + +constexpr int kThreads = 128; + +// One packed bf16 pair of gate and up values -> its two bf16 results (the low half from the low halves). The +// instruction sequence of the Triton kernel's PTX at its defaults (host/silu/silu_and_mul_plain.ptx): g * -log2(e) as +// one product, ex2.approx, + 1, div.full 1 / that, g * s, (u + 0.0) as an IEEE add, the product, one rounding to bf16. +__device__ __forceinline__ uint32_t siluMulBf16x2(uint32_t gate, uint32_t up) +{ + uint32_t r; + asm("{\n\t.reg .b32 gl, gh, ul, uh, t0, t1, e0, e1, s0, s1, p0, p1, q0, q1, r0, r1, one, zero;\n\t" + ".reg .b16 hl, hh;\n\t" + "mov.b32 {hl, hh}, %1;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, %2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mul.rn.f32 t0, gl, 0fBFB8AA3B;\n\tmul.rn.f32 t1, gh, 0fBFB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 p0, gl, s0;\n\tmul.rn.f32 p1, gh, s1;\n\t" + "mov.b32 zero, 0f00000000;\n\tadd.rn.f32 q0, ul, zero;\n\tadd.rn.f32 q1, uh, zero;\n\t" + "mul.rn.f32 r0, q0, p0;\n\tmul.rn.f32 r1, q1, p1;\n\t" + "cvt.rn.bf16x2.f32 %0, r1, r0;\n\t}" + : "=r"(r) + : "r"(gate), "r"(up)); + return r; +} + +// Eight outputs per thread: 16-byte loads of the gate and the up half, one 16-byte store. +__global__ void __launch_bounds__(kThreads) + siluAndMulKernel(uint4 const* __restrict__ x, uint4* __restrict__ out, int64_t rows, int64_t vecs) +{ + int64_t const row = blockIdx.y; + int64_t const v = static_cast(blockIdx.x) * kThreads + threadIdx.x; + if (row >= rows || v >= vecs) + { + return; + } + uint4 const g = x[row * 2 * vecs + v]; + uint4 const u = x[row * 2 * vecs + vecs + v]; + uint4 o; + o.x = siluMulBf16x2(g.x, u.x); + o.y = siluMulBf16x2(g.y, u.y); + o.z = siluMulBf16x2(g.z, u.z); + o.w = siluMulBf16x2(g.w, u.w); + out[row * vecs + v] = o; +} + +} // namespace + +cudaError_t invokeSiluAndMul(uint16_t const* x, uint16_t* out, int64_t rows, int64_t d, cudaStream_t stream) +{ + int64_t const vecs = d / 8; + dim3 const grid(static_cast((vecs + kThreads - 1) / kThreads), static_cast(rows)); + siluAndMulKernel<<>>( + reinterpret_cast(x), reinterpret_cast(out), rows, vecs); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.h new file mode 100644 index 000000000000..dccbb44572ef --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.h @@ -0,0 +1,40 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +//! out[m, j] = bf16(silu(g) * (u + 0)) with g = x[m, j], u = x[m, d + j] (x bf16 [rows, 2d], out bf16 [rows, d], both +//! contiguous, d a multiple of 8, 16-byte aligned rows): bit for bit the Triton silu_and_mul kernel at its defaults +//! (no limit, alpha 1, beta 0), the plain SwiGLU of the Eagle3 drafter MLP. +cudaError_t invokeSiluAndMul(uint16_t const* x, uint16_t* out, int64_t rows, int64_t d, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.cu new file mode 100644 index 000000000000..e26804fb47fa --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.cu @@ -0,0 +1,743 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3 long-context sparse decode attention in one launch. +// +// Grid: one cluster of 16 CTAs per query token; CTA `r` attends the token's `r`-th selected block (the Triton +// producer's chunk `r` at 16 chunks), keeps the partial in shared memory, and after a cluster barrier merges head +// `r` from all 16 partials over DSMEM, writing the BF16 row and, optionally, its MXFP8 copy and swizzled UE8M0 +// scales exactly as the Triton merge's `_store_mxfp8_head` lays them out. +// +// Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and +// PV on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +// log-domain recurrence. +// +// ONE_DIV merge (MSA_ONE_DIV_MERGE): each CTA pushes its unnormalized partial with its row max m and sum l, and the +// merge forms o = sum_c w_c acc_c / sum_c w_c l_c with w_c = 2^(m_c - max m), one division per output, instead of +// normalizing every partial and weighting by its LSE. Deterministic (fixed chunk order, no atomics); a different +// rounding sequence from the Triton merge, inside the same band of the float64 reference. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +namespace +{ + +constexpr int kHeads = 16; +constexpr int kDim = 128; +constexpr int kPage = 128; +constexpr int kTopk = 16; +constexpr int kThreads = 128; +constexpr int kPitch = kDim + 8; // halves per smem row: 272 B, conflict-free ldmatrix +constexpr float kNegInf = -__builtin_huge_valf(); + +using Params = MinimaxM3SparseDecodeLcParams; + +struct Smem +{ + __half qs[kHeads * kPitch]; + __half vs[kPage * kPitch]; + __half ps[kHeads * kPitch]; + float recv[kTopk][kDim]; // head `rank` partials pushed by the 16 CTAs, chunk-major + float recv_lse[kTopk]; // the chunk LSEs (log2); with ONE_DIV, the chunk row maxima + float recv_l[kTopk]; // ONE_DIV: the chunk row sums + float wgt[kTopk]; // the 16 merge weights, computed once per CTA + float red[4][kHeads]; + alignas(8) unsigned long long rbar; // the split-K partials of head `rank` landed (complete_tx) +}; + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t e2m1x2_to_f16x2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 s;\n cvt.u8.u32 s, %1;\n cvt.rn.f16x2.e2m1x2 %0, s;\n}" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t e4m3x2_to_f16x2(uint16_t pair) +{ + uint32_t out; + asm("cvt.rn.f16x2.e4m3x2 %0, %1;" : "=r"(out) : "h"(pair)); + return out; +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ void ldsm_x4(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x4_t(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma(float (&c)[4], uint32_t const (&a)[4], uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};" + : "+f"(c[0]), "+f"(c[1]), "+f"(c[2]), "+f"(c[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} + +__device__ __forceinline__ uint32_t map_cluster(void const* local, uint32_t rank) +{ + uint32_t remote; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(remote) : "r"(smem_u32(local)), "r"(rank)); + return remote; +} + +// Posted DSMEM stores that complete_tx on the receiving CTA's mbarrier `bar`. +__device__ __forceinline__ void st_async_v2(uint32_t addr, float a, float b, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.v2.f32 [%0], {%1, %2}, [%3];" ::"r"(addr), + "f"(a), "f"(b), "r"(bar) + : "memory"); +} + +__device__ __forceinline__ void st_async_f32(uint32_t addr, float a, uint32_t bar) +{ + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(addr), + "r"(__float_as_uint(a)), "r"(bar) + : "memory"); +} + +// One packed NVFP4 row (64 bytes, 128 values) with its 8 E4M3 scales -> 128 fp16 values in smem. +__device__ __forceinline__ void dequant_row(__half* dst, uint8_t const* packed, uint8_t const (&sc)[8], bool valid) +{ + uint4 const* src = reinterpret_cast(packed); + uint32_t* d = reinterpret_cast(dst); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + uint32_t words[4] = {w.x, w.y, w.z, w.w}; + // 16 bytes = 32 values = scale columns 2c and 2c + 1. +#pragma unroll + for (int h = 0; h < 2; ++h) + { + uint8_t const s = valid ? sc[2 * c + h] : 0; + uint32_t const sh = e4m3x2_to_f16x2(static_cast(s | (s << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); +#pragma unroll + for (int j = 0; j < 2; ++j) + { + uint32_t const word = words[2 * h + j]; +#pragma unroll + for (int b = 0; b < 4; ++b) + { + uint32_t const v2 = e2m1x2_to_f16x2((word >> (8 * b)) & 0xFF); + __half2 const x = __hmul2(*reinterpret_cast<__half2 const*>(&v2), scale); + d[c * 16 + h * 8 + j * 4 + b] = *reinterpret_cast(&x); + } + } + } + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +// This CTA's 16-byte-aligned slice of [base, base + bytes) into L2, at most 16 KiB per bulk prefetch. +__device__ __forceinline__ void prefetch_l2_slice(uint8_t const* base, int64_t bytes) +{ + if (base == nullptr || bytes <= 0) + return; + int64_t const n = gridDim.x; + int64_t const slice = ((bytes + n - 1) / n + 15) & ~int64_t{15}; + int64_t const off = static_cast(blockIdx.x) * slice; + if (off >= bytes) + return; + int64_t const len = min(slice, bytes - off) & ~int64_t{15}; + for (int64_t o = 0; o < len; o += 16384) + { + uint32_t const sz = static_cast(min(int64_t{16384}, len - o)); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(base + off + o), "r"(sz) : "memory"); + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +// BT_PREWAIT: the request's block-table row is staged in dynamic shared memory before the grid dependency (at most +// kBtCap entries, 4096 x 128 = 512K tokens; longer rows keep the global load), so the post-wait page lookup is a shared +// memory hit instead of a dependent L2 load. 52 + 16 KB per CTA still fits beside the o_proj GEMV's 154 KB on one SM. +// The table is written only before the step's graph runs or by the step-input prologue (a plain launch, PRE-WAIT +// WRITER CONTRACT), so it is complete when this grid launches; the values are the same, so is the output. +constexpr int kBtCap = 4096; + +template +constexpr size_t smemBytes() +{ + return BTPW ? sizeof(Smem) + kBtCap * sizeof(int) : sizeof(Smem); +} + +template +__global__ void __launch_bounds__(kThreads) m3_lc_attn_kernel(__grid_constant__ Params const p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(128) uint8_t smem_raw[]; + Smem& s = *reinterpret_cast(smem_raw); + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const tok = blockIdx.x / kTopk; + uint32_t rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(rank)); + + if (tid == 0) + { + // One arrival (here) plus every cluster CTA's partial row (kDim floats) and LSE (ONE_DIV: max and sum) for + // head `rank`. + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(smem_u32(&s.rbar)) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(smem_u32(&s.rbar)), + "r"(kTopk * (kDim + (ONE_DIV ? 2 : 1)) * 4) + : "memory"); + } + // Every peer must be running before its shared memory takes a DSMEM store: arrive now, wait before pushing. + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + // The o_proj weights are read-only for the whole step: start pulling them into L2 now, while HBM is idle in the + // latency-bound attention phase (one lane of warps 0 and 1; nothing waits on these). + if (tid == 0) + prefetch_l2_slice(p.pf0, p.pf0_bytes); + else if (tid == 32) + prefetch_l2_slice(p.pf1, p.pf1_bytes); + + // Everything that does not depend on the top-16 table is read before the grid dependency: the request's length + // (written before the step's graph runs) and, behind a select, the query rows (written by the producer, which + // completed before the select ahead of this grid could release it: the select triggers its dependents only after + // its own grid wait). Identity steps have no select (see the q load below). + int const req = tok / p.dql; + int const qoff = tok - req * p.dql; + int const seq = p.seq_lens[req]; + int const kv_len = max(seq - p.dql + qoff + 1, 0); + int const nblocks = (kv_len + kPage - 1) / kPage; + int const real_topk = min(kTopk, nblocks); + // The next sparse layer's block that its CTA (tok, rank) selected at the previous step (a decode step selects + // mostly the blocks it selected one step earlier): its K/V pages and scales into L2 for that layer, from one lane + // of warp 2. The id buffer was written by a kernel of the previous step, long complete; a recorded id at or past + // this request's block count (another request's row) or a page outside the pool is skipped. + if (p.n1_next != nullptr && tid == 64) + { + int const nb = p.n1_next[tok * kTopk + rank]; + if (nb >= 0 && nb < nblocks) + { + int64_t const pg = p.block_table[req * p.bt_stride + nb]; + if (pg >= 0 && pg < p.n1_pages) + { + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.n1_k + pg * p.k_blk), "r"(kPage * 64) + : "memory"); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.n1_v + pg * p.v_blk), "r"(kPage * 64) + : "memory"); + asm volatile( + "cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.n1_ksc + pg * p.ksc_blk), "r"(kPage * 8) + : "memory"); + asm volatile( + "cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.n1_vsc + pg * p.vsc_blk), "r"(kPage * 8) + : "memory"); + } + } + } + // q: 16 heads x 128 e4m3 -> fp16; thread = (head, 16-value chunk). + auto const load_q = [&]() + { + int const h = tid >> 3; + int const c = (tid & 7) * 16; + uint4 const w = *reinterpret_cast(p.q + tok * p.q_tok + h * p.q_head + c); + uint32_t const words[4] = {w.x, w.y, w.z, w.w}; + uint32_t* d = reinterpret_cast(s.qs + h * kPitch + c); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + d[2 * i] = e4m3x2_to_f16x2(static_cast(words[i] & 0xFFFF)); + d[2 * i + 1] = e4m3x2_to_f16x2(static_cast(words[i] >> 16)); + } + }; + // Before the wait only behind the select: an identity step has no scorer / select, so this grid's predecessor is + // the producer itself, which releases its dependents at entry, before it writes q. There q is read after the wait, + // whether the blocks come from the lengths (identity) or from the staged table. + if (!p.identity && !p.identity_step) + load_q(); + int const* const s_bt = reinterpret_cast(smem_raw + sizeof(Smem)); + bool bt_smem = false; + if constexpr (BTPW) + { + int const nbt = (max(seq, 0) + kPage - 1) / kPage; + bt_smem = nbt <= kBtCap; + if (bt_smem) + { + int* const dst = reinterpret_cast(smem_raw + sizeof(Smem)); + for (int i = tid; i < nbt; i += kThreads) + { + dst[i] = __ldg(p.block_table + req * p.bt_stride + i); + } + } + __syncthreads(); // the staged row is complete before any thread's lookup + } + + asm volatile("griddepcontrol.wait;" ::: "memory"); + if constexpr (EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } + if (p.identity || p.identity_step) + load_q(); + int blk = static_cast(rank) >= real_topk ? -1 + : p.identity ? static_cast(rank) + : p.topk[tok * p.t_tok + rank * p.t_k]; + // Fail closed on an id past the row (a stale or corrupt table), as the stock Triton producer does: the block is + // skipped. Read through the block table it would be another request's page, every token masked, and its softmax + // ex2(-inf - (-inf)) = NaN. + if (blk >= nblocks) + blk = -1; + bool const active = blk >= 0; + // Record this step's selection for this layer's next-step prefetch (read by the previous sparse layer's CTAs). + if (p.n1_self != nullptr && tid == 0) + p.n1_self[tok * kTopk + rank] = blk; + int64_t const page_pw = !active ? 0 : (bt_smem ? s_bt[blk] : p.block_table[req * p.bt_stride + blk]); + + float m_row[2] = {kNegInf, kNegInf}; + float acc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + acc[j][i] = 0.F; + float l_row[2] = {0.F, 0.F}; + + // K stays packed in registers: the MMA B fragment of token n = lane / 4 at k-step s is the f16x2 pair of + // packed bytes 8s + q and 8s + 4 + q (q = lane % 4), both under scale column s. + uint32_t kw[4][16]; + uint2 ksc[4]; + int64_t page = 0; + if (active) + { + page = page_pw; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const n = warp * 32 + j * 8 + (lane >> 2); + bool const valid = blk * kPage + n < kv_len; + uint4 const* src = reinterpret_cast(p.k + page * p.k_blk + n * 64); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 const w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + kw[j][4 * c] = w.x; + kw[j][4 * c + 1] = w.y; + kw[j][4 * c + 2] = w.z; + kw[j][4 * c + 3] = w.w; + } + ksc[j] = valid ? __ldg(reinterpret_cast(p.ksc + page * p.ksc_blk + n * 8)) : make_uint2(0, 0); + } + bool const valid = blk * kPage + tid < kv_len; + uint8_t vsc[8]; + if (valid) + { + uint8_t const* vb = p.vsc + page * p.vsc_blk + (tid >> 2) * 32 + (tid & 3); +#pragma unroll + for (int c = 0; c < 8; ++c) + vsc[c] = __ldg(vb + 4 * c); + } + dequant_row(s.vs + tid * kPitch, p.v + page * p.v_blk + tid * 64, vsc, valid); + } + __syncthreads(); + + if (active) + { + float const scale_log2 = p.sm_scale * 1.4426950409F * __ldg(p.kgs); + int const q4 = (lane & 3) * 8; + // S = q K^T for this warp's 32 tokens: rows g / g + 8, columns 2 * (lane % 4) + {0, 1} of each n-tile. + float sacc[4][4]; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + sacc[j][i] = 0.F; +#pragma unroll + for (int ks = 0; ks < kDim / 16; ++ks) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.qs + (lane & 15) * kPitch + ks * 16 + (lane >> 4) * 8)); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + uint32_t const scw = ks < 4 ? ksc[j].x : ksc[j].y; + uint8_t const sc = static_cast(scw >> (8 * (ks & 3))); + uint32_t const sh = e4m3x2_to_f16x2(static_cast(sc | (sc << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); + uint32_t const r0 = e2m1x2_to_f16x2((kw[j][2 * ks] >> q4) & 0xFF); + uint32_t const r1 = e2m1x2_to_f16x2((kw[j][2 * ks + 1] >> q4) & 0xFF); + __half2 const h0 = __hmul2(*reinterpret_cast<__half2 const*>(&r0), scale); + __half2 const h1 = __hmul2(*reinterpret_cast<__half2 const*>(&r1), scale); + mma(sacc[j], a, *reinterpret_cast(&h0), *reinterpret_cast(&h1)); + } + } + int const g = lane >> 2; + float mloc[2] = {kNegInf, kNegInf}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const t = warp * 32 + j * 8 + 2 * (lane & 3) + (i & 1); + float const v = blk * kPage + t < kv_len ? 0.F + sacc[j][i] * scale_log2 : kNegInf; + sacc[j][i] = v; + mloc[i >> 1] = fmaxf(mloc[i >> 1], v); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 1)); + mloc[r] = fmaxf(mloc[r], __shfl_xor_sync(0xffffffff, mloc[r], 2)); + } + if ((lane & 3) == 0) + { + s.red[warp][g] = mloc[0]; + s.red[warp][g + 8] = mloc[1]; + } + __syncthreads(); + // Every thread reduces its two rows' four warp maxima itself (the same expression, so the same bits): no + // shared row-max stage and its second barrier. +#pragma unroll + for (int r = 0; r < 2; ++r) + { + int const row = g + r * 8; + m_row[r] = fmaxf(fmaxf(s.red[0][row], s.red[1][row]), fmaxf(s.red[2][row], s.red[3][row])); + } + // A row with no valid token has m = -inf: its probabilities are ex2(-inf - 0) = 0, never NaN. + float const m_use[2] = {m_row[0] == kNegInf ? 0.F : m_row[0], m_row[1] == kNegInf ? 0.F : m_row[1]}; + float lloc[2] = {0.F, 0.F}; +#pragma unroll + for (int j = 0; j < 4; ++j) +#pragma unroll + for (int i = 0; i < 4; i += 2) + { + float const p0 = ex2(sacc[j][i] - m_use[i >> 1]); + float const p1 = ex2(sacc[j][i + 1] - m_use[i >> 1]); + lloc[i >> 1] += p0 + p1; + int const row = g + (i >> 1) * 8; + int const t = warp * 32 + j * 8 + 2 * (lane & 3); + *reinterpret_cast<__half2*>(s.ps + row * kPitch + t) = __floats2half2_rn(p0, p1); + } +#pragma unroll + for (int r = 0; r < 2; ++r) + { + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 1); + lloc[r] += __shfl_xor_sync(0xffffffff, lloc[r], 2); + } + __syncthreads(); // row-max reads of red done before red is reused; ps complete + if ((lane & 3) == 0) + { + s.red[warp][g] = lloc[0]; + s.red[warp][g + 8] = lloc[1]; + } + // O[:, warp's 32 dims] = P V over all 128 tokens. +#pragma unroll + for (int kk = 0; kk < kPage; kk += 16) + { + uint32_t a[4]; + ldsm_x4(a, smem_u32(s.ps + (lane & 15) * kPitch + kk + (lane >> 4) * 8)); +#pragma unroll + for (int jp = 0; jp < 2; ++jp) + { + uint32_t b[4]; + int const t = kk + (lane & 7) + ((lane >> 3) & 1) * 8; + int const d = warp * 32 + jp * 16 + (lane >> 4) * 8; + ldsm_x4_t(b, smem_u32(s.vs + t * kPitch + d)); + mma(acc[2 * jp], a, b[0], b[1]); + mma(acc[2 * jp + 1], a, b[2], b[3]); + } + } + __syncthreads(); + // Likewise the row sums, in the same order. red is next written by the merge, after this CTA's rbar wait, + // which completes only once every warp here has pushed its partial (formed from these sums). +#pragma unroll + for (int r = 0; r < 2; ++r) + { + int const row = g + r * 8; + l_row[r] = (s.red[0][row] + s.red[1][row]) + (s.red[2][row] + s.red[3][row]); + } + } + + // Normalize the partial and push head h's row with its LSE (log2) into CTA h's recv[rank] slot, as the Triton + // producer stores chunk `rank`; st.async stores that complete_tx on CTA h's barrier. + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); + { + int const g = lane >> 2; + float const vgs = __ldg(p.vgs); +#pragma unroll + for (int r = 0; r < 2; ++r) + { + float scale, lse; + if constexpr (ONE_DIV) + { + // Unnormalized: the merge divides once. An inactive CTA pushes m = -inf, l = 0 (weight 0). + scale = 1.F; + lse = m_row[r]; + } + else if constexpr (LINEAR) + { + scale = l_row[r] > 0.F ? 1.0F / l_row[r] : 0.F; + lse = l_row[r] > 0.F ? m_row[r] + lg2(l_row[r]) : kNegInf; + } + else + { + float const lse_i + = active && m_row[r] > kNegInf ? m_row[r] + lg2(ex2(kNegInf - m_row[r]) + l_row[r]) : kNegInf; + scale = lse_i > kNegInf ? ex2(m_row[r] - lse_i) : 0.F; + lse = lse_i; + } + scale = scale * vgs; + int const row = g + r * 8; + uint32_t const dst = map_cluster(&s.recv[rank][0], row); + uint32_t const rb = map_cluster(&s.rbar, row); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const d = warp * 32 + j * 8 + 2 * (lane & 3); + st_async_v2(dst + d * 4, acc[j][2 * r] * scale, acc[j][2 * r + 1] * scale, rb); + } + if (warp == 0 && (lane & 3) == 0) + { + st_async_f32(map_cluster(&s.recv_lse[rank], row), lse, rb); + if constexpr (ONE_DIV) + { + st_async_f32(map_cluster(&s.recv_l[rank], row), l_row[r], rb); + } + } + } + } + { + // Every CTA's partial row and LSE for head `rank` landed (complete_tx on this CTA's barrier). + uint32_t done = 0; + while (!done) + { + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}" + : "=r"(done) + : "r"(smem_u32(&s.rbar)) + : "memory"); + } + } + + // CTA `rank` merges head `rank` across the 16 partials; thread = output dim. + int const h = static_cast(rank); + float o = 0.F; + if constexpr (ONE_DIV) + { + // Thread c < 16 takes chunk c's weight 2^(m_c - max m) and its weighted row sum; every thread then sums the + // weighted partials and the weighted row sums in chunk order and divides once. + if (tid < kTopk) + { + float m_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + m_max = fmaxf(m_max, s.recv_lse[c]); + m_max = m_max == kNegInf ? 0.F : m_max; + float const w = ex2(s.recv_lse[tid] - m_max); + s.wgt[tid] = w; + s.red[0][tid] = w * s.recv_l[tid]; + } + __syncthreads(); + float num = 0.F; + float den = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + { + num += s.recv[c][tid] * s.wgt[c]; + den += s.red[0][c]; + } + o = den > 0.F ? __fdiv_rn(num, den) : 0.F; + } + else + { + // The weights are the Triton merge's (exp2 of each chunk's LSE minus the largest, divided by their sum in + // chunk order), formed once per CTA: thread c < 16 takes chunk c's exponential, then divides it by the sum + // every one of those threads forms in the same order. + if (tid < kTopk) + { + float lse_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + lse_max = fmaxf(lse_max, s.recv_lse[c]); + lse_max = lse_max == kNegInf ? 0.F : lse_max; + s.red[0][tid] = ex2(s.recv_lse[tid] - lse_max); + } + __syncthreads(); + if (tid < kTopk) + { + float denom = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + denom += s.red[0][c]; + denom = denom > 0.F ? denom : 1.F; + s.wgt[tid] = __fdiv_rn(s.red[0][tid], denom); + } + __syncthreads(); +#pragma unroll + for (int c = 0; c < kTopk; ++c) + o += s.recv[c][tid] * s.wgt[c]; + } + __nv_bfloat16 const ob = __float2bfloat16_rn(o); + p.out[tok * p.out_stride + h * kDim + tid] = ob; + + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + amax = fmaxf(amax, __shfl_xor_sync(0xffffffff, amax, off)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + float const xq = x * qs; + uint16_t q2; + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2) : "f"(0.F), "f"(xq)); + p.q8[static_cast(tok) * (p.num_sf_k_tiles * 128) + h * kDim + tid] = static_cast(q2 & 0xFF); + if (lane == 0) + { + int const k_sf = h * 4 + warp; + int64_t const off = static_cast(tok / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (tok % 32) * 16 + ((tok % 128) / 32) * 4 + k_sf % 4; + p.sf[off] = static_cast(e); + } + } + if constexpr (!EARLY) + { + asm volatile("griddepcontrol.launch_dependents;"); + } +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +} + +template +cudaError_t configure() +{ + auto kernel = m3_lc_attn_kernel; + cudaError_t err = cudaFuncSetAttribute( + kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smemBytes())); + if (err != cudaSuccess) + { + return err; + } + return cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1); +} + +template +cudaError_t launch(Params const& prm, int num_tokens, bool use_pdl, cudaStream_t stream) +{ + // Kernel attributes are set once per process, on the first launch (outside CUDA-graph capture: warmup runs + // every variant eagerly first). + static cudaError_t const configured = configure(); + if (configured != cudaSuccess) + { + return configured; + } + auto kernel = m3_lc_attn_kernel; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(num_tokens * kTopk); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smemBytes(); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kTopk; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +template +cudaError_t launchBt(Params const& prm, int num_tokens, bool use_pdl, cudaStream_t stream) +{ + return prm.bt_prewait ? launch(prm, num_tokens, use_pdl, stream) + : launch(prm, num_tokens, use_pdl, stream); +} + +} // namespace + +cudaError_t invokeSparseDecodeLc4(MinimaxM3SparseDecodeLcParams const& params, int32_t numTokens, bool linearSoftmax, + bool mxfp8, bool earlyTrigger, bool usePdl, cudaStream_t stream) +{ + if (!earlyTrigger) + { + return cudaErrorNotSupported; + } + if (params.one_div) + { + if (linearSoftmax) + { + return mxfp8 ? launchBt(params, numTokens, usePdl, stream) + : launchBt(params, numTokens, usePdl, stream); + } + return mxfp8 ? launchBt(params, numTokens, usePdl, stream) + : launchBt(params, numTokens, usePdl, stream); + } + if (linearSoftmax) + { + return mxfp8 ? launchBt(params, numTokens, usePdl, stream) + : launchBt(params, numTokens, usePdl, stream); + } + return mxfp8 ? launchBt(params, numTokens, usePdl, stream) + : launchBt(params, numTokens, usePdl, stream); +} + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.h new file mode 100644 index 000000000000..6fea5ee87498 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.h @@ -0,0 +1,104 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +//! SM107 TP4 copy of kernels/minimaxM3/sparseDecodeLc (the sm_103 targets keep that one), with the split-K merge +//! weights formed once per CTA. Operands of the MiniMax-M3 one-launch long-context sparse decode. Strides are in +//! elements of the pointed-to type. Geometry is fixed: 16 query heads on one KV head, head_dim 128, page 128, top-k 16. +struct MinimaxM3SparseDecodeLcParams +{ + uint8_t const* q; // e4m3 [T, 16, 128] + int64_t q_tok, q_head; + uint8_t const* k; // packed e2m1 [pages, 1, 128, 64] + uint8_t const* v; + int64_t k_blk, v_blk; + uint8_t const* ksc; // e4m3 [pages, 1, 128, 8], token-major + uint8_t const* vsc; // e4m3, 4x4 token-quad order + int64_t ksc_blk, vsc_blk; + float const* kgs; + float const* vgs; + int const* topk; // [T, 16] view (kv head 0) + int64_t t_tok, t_k; + int const* block_table; + int64_t bt_stride; + int const* seq_lens; + int dql; + float sm_scale; + __nv_bfloat16* out; // [T, 2048] + int64_t out_stride; + uint8_t* q8; // e4m3 [rows, num_sf_k_tiles * 128] + uint8_t* sf; + int num_sf_k_tiles; + // Optional L2 prefetch of two read-only ranges (the layer's o_proj weight and scales), split across the grid's + // CTAs and issued before the grid dependency: a hint, so the output does not depend on it. nullptr = none. + uint8_t const* pf0; + int64_t pf0_bytes; + uint8_t const* pf1; + int64_t pf1_bytes; + // Optional next-layer KV prefetch: CTA (token, r) records its selected block id in this layer's [rows, 16] id + // buffer (n1_self) and pulls into L2 the K/V pages and scales of the block the next sparse layer's CTA (token, r) + // selected at the previous step (n1_next, from that layer's pools n1_k / n1_v / n1_ksc / n1_vsc, this layer's + // strides). Hints only. nullptr = off. + int* n1_self; + int const* n1_next; + uint8_t const* n1_k; + uint8_t const* n1_v; + uint8_t const* n1_ksc; + uint8_t const* n1_vsc; + int64_t n1_pages; // pages in the next layer's pool; a recorded block outside it is skipped + // Identity selection (short-sequence steps, every request within 16 blocks): CTA r attends block r of its + // token's request when r < min(16, blocks), from the device sequence length; topk is not read. + int identity; + // The step is an identity step (short sequences: no scorer / select ran this step), whether or not this kernel + // derives its blocks from the lengths (identity) or reads the staged table: q is then read after the grid wait, + // because the predecessor is the producer, which releases its dependents at entry, before it writes q. + int identity_step; + // One-division merge (MSA_ONE_DIV_MERGE): the CTAs push unnormalized partials with their row max and sum, and the + // merge divides once per output. Deterministic, a different rounding sequence from the per-chunk normalization. + int one_div; + // BT_PREWAIT: stage the request's block-table row (up to 4096 entries) in shared memory before the grid + // dependency, so the post-wait page lookup reads shared memory. Same output. + int bt_prewait; +}; + +//! One cluster of 16 CTAs per query token: CTA r attends the token's r-th selected block, then the cluster merges +//! the 16 partials over DSMEM into the BF16 row (and, when mxfp8, its MXFP8 copy with swizzled UE8M0 scales). +//! +//! Instantiated variants: linearSoftmax in {false, true} x mxfp8 in {false, true}, with earlyTrigger (release PDL +//! dependents at entry). Returns cudaErrorNotSupported for any other variant. The device code is SM100-family only; +//! on other architectures the kernel compiles to an empty body, so callers check the device first. +cudaError_t invokeSparseDecodeLc4(MinimaxM3SparseDecodeLcParams const& params, int32_t numTokens, bool linearSoftmax, + bool mxfp8, bool earlyTrigger, bool usePdl, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.cu new file mode 100644 index 000000000000..679bcb002d1d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.cu @@ -0,0 +1,58 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +namespace +{ + +constexpr int kThreads = 256; + +__global__ void __launch_bounds__(kThreads) + specOffsetsKernel(int32_t* __restrict__ pos, int32_t const* __restrict__ posOff, int64_t nPos, + int32_t* __restrict__ kv, int32_t const* __restrict__ kvOff, int64_t nKv) +{ + for (int64_t i = threadIdx.x; i < nPos; i += kThreads) + { + pos[i] = static_cast(static_cast(pos[i]) + static_cast(posOff[i])); + } + for (int64_t j = threadIdx.x; j < nKv; j += kThreads) + { + kv[j] = static_cast(static_cast(kv[j]) + static_cast(kvOff[j])); + } +} + +} // namespace + +cudaError_t invokeSpecOffsets(int32_t* pos, int32_t const* posOff, int64_t nPos, int32_t* kv, int32_t const* kvOff, + int64_t nKv, cudaStream_t stream) +{ + specOffsetsKernel<<<1, kThreads, 0, stream>>>(pos, posOff, nPos, kv, kvOff, nKv); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.h new file mode 100644 index 000000000000..878ca0197e9e --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.h @@ -0,0 +1,41 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +//! The overlap scheduler's speculative step correction in one launch: pos[i] += posOff[i] for i < nPos and +//! kv[j] += kvOff[j] for j < nKv (int32, wrapping as the ATen adds do). A plain launch: it is a step-input writer that +//! pre-wait readers rely on (rubin-20261015/PREWAIT_AUDIT.md item b). +cudaError_t invokeSpecOffsets(int32_t* pos, int32_t const* posOff, int64_t nPos, int32_t* kv, int32_t const* kvOff, + int64_t nKv, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.cu new file mode 100644 index 000000000000..42361b5ef47f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.cu @@ -0,0 +1,111 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +namespace +{ + +constexpr int kThreads = 512; +constexpr int kWarps = kThreads / 32; + +// One warp per copy. Plain (coherent) loads: a device copy may read bytes another warp of this CTA just stored, which +// the CTA barrier between the two kinds makes visible; a non-coherent load could return the stale line. +__device__ __forceinline__ void copyWarp(uint8_t* dst, uint8_t const* src, int64_t bytes, int lane) +{ + auto const d = reinterpret_cast(dst); + auto const s = reinterpret_cast(src); + if (((d | s | static_cast(bytes)) & 15) == 0) + { + auto* d16 = reinterpret_cast(dst); + auto const* s16 = reinterpret_cast(src); + for (int64_t i = lane; i < bytes / 16; i += 32) + { + d16[i] = s16[i]; + } + } + else if (((d | s | static_cast(bytes)) & 3) == 0) + { + auto* d4 = reinterpret_cast(dst); + auto const* s4 = reinterpret_cast(src); + for (int64_t i = lane; i < bytes / 4; i += 32) + { + d4[i] = s4[i]; + } + } + else + { + for (int64_t i = lane; i < bytes; i += 32) + { + dst[i] = src[i]; + } + } +} + +// One CTA: a step's window is a few dozen copies of a few hundred bytes, so the launch is the cost; a CTA barrier +// orders the host copies before the device copies where the parallel scatter needed a second launch. +__global__ void __launch_bounds__(kThreads) stepPrologueKernel(uint8_t* window, int64_t* epoch, int64_t maxDesc) +{ + auto* header = reinterpret_cast(window); + int64_t const numHost = min(header[0], maxDesc); + int64_t const numDevice = min(header[1], maxDesc); + int64_t const* desc = header + 2; + int const warp = threadIdx.x / 32; + int const lane = threadIdx.x % 32; + if (epoch != nullptr && threadIdx.x == 0) + { + *epoch += 1; + } + for (int64_t i = warp; i < numHost; i += kWarps) + { + int64_t const* c = desc + 3 * i; + copyWarp(reinterpret_cast(c[0]), window + c[1], c[2], lane); + } + __syncthreads(); + for (int64_t i = warp; i < numDevice; i += kWarps) + { + // The device copies' slots follow all header[0] host copies, as the parallel scatter indexes them. + int64_t const* c = desc + 3 * (header[0] + i); + copyWarp(reinterpret_cast(c[0]), reinterpret_cast(c[1]), c[2], lane); + } + // Every thread has read the header and the descriptors: consume the window. + __syncthreads(); + if (threadIdx.x < 2) + { + header[threadIdx.x] = 0; + } +} + +} // namespace + +cudaError_t invokeStepPrologue(uint8_t* window, int64_t* epoch, int64_t maxDesc, cudaStream_t stream) +{ + stepPrologueKernel<<<1, kThreads, 0, stream>>>(window, epoch, maxDesc); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.h new file mode 100644 index 000000000000..c934decff161 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.h @@ -0,0 +1,47 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +//! The decode graph's step-input prologue in one launch (step_inputs.StepInputPack with early_copy): the work of the +//! two parallel scatter launches and the header clear that follow them, in that order. +//! +//! The window is the pack's staging layout (kernels/stepInputsScatter.h): an int64 header `[numHostCopies, +//! numDeviceCopies]`, then one `(dst, src, bytes)` int64 triple per copy, host copies first; a host copy's `src` is a +//! byte offset into the window, a device copy's a device address. Up to `maxDesc` copies of each kind are written (the +//! slots the parallel scatter's grid covers), every host copy before any device copy, so a device copy may read a +//! destination a host copy filled; the copies of one kind must not overlap (the pack issues the window only then). +//! Afterwards the header is zeroed, so a replay with no new upload writes nothing. With `epoch` non-null, it is +//! incremented once. +cudaError_t invokeStepPrologue(uint8_t* window, int64_t* epoch, int64_t maxDesc, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.cu new file mode 100644 index 000000000000..306801896ff9 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.cu @@ -0,0 +1,100 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +namespace +{ + +constexpr int kThreads = 256; + +// One packed bf16 pair of gate and up values -> its two bf16 results (the low half from the low halves): the +// instruction sequence of the Triton kernel at limit 7, alpha 1.702, beta 1 (bitwise on SM107): the +// gate clamp as min.bf16x2 before the widening, the up clamp as min.xorsign.abs, (u + 1), (g * -1.702) * log2(e), +// ex2.approx, + 1, div.full 1 / that, s * g, the product, one rounding to bf16. +__device__ __forceinline__ uint32_t swigluOaiBf16x2(uint32_t gate, uint32_t up) +{ + uint32_t r; + asm("{\n\t.reg .b32 g, glim, gl, gh, ul, uh, one, lim, t0, t1, e0, e1, s0, s1, r0, r1;\n\t.reg .b16 hl, hh;\n\t" + "mov.b32 glim, 0x40E040E0;\n\tmin.bf16x2 g, %1, glim;\n\t" + "mov.b32 {hl, hh}, g;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, %2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mov.b32 lim, 0f40E00000;\n\tmin.xorsign.abs.f32 ul, ul, lim;\n\tmin.xorsign.abs.f32 uh, uh, lim;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 ul, ul, one;\n\tadd.rn.f32 uh, uh, one;\n\t" + "mul.rn.f32 t0, gl, 0fBFD9DB23;\n\tmul.rn.f32 t1, gh, 0fBFD9DB23;\n\t" + "mul.rn.f32 t0, t0, 0f3FB8AA3B;\n\tmul.rn.f32 t1, t1, 0f3FB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "add.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 r0, s0, gl;\n\tmul.rn.f32 r1, s1, gh;\n\t" + "mul.rn.f32 r0, ul, r0;\n\tmul.rn.f32 r1, uh, r1;\n\t" + "cvt.rn.bf16x2.f32 %0, r1, r0;\n\t}" + : "=r"(r) + : "r"(gate), "r"(up)); + return r; +} + +template +__global__ void __launch_bounds__(kThreads) + swigluOaiKernel(uint32_t const* __restrict__ x, uint32_t* __restrict__ out, int64_t rows, int64_t halfD) +{ + if constexpr (kPdl) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + int64_t const i = blockIdx.x * static_cast(kThreads) + threadIdx.x; + if (i >= rows * halfD) + { + return; + } + int64_t const row = i / halfD; + int64_t const col = i - row * halfD; + uint32_t const* xr = x + row * 2 * halfD; + out[i] = swigluOaiBf16x2(xr[col], xr[halfD + col]); +} + +} // namespace + +cudaError_t invokeSwigluOai( + uint32_t const* x, uint32_t* out, int64_t rows, int64_t halfD, bool pdl, cudaStream_t stream) +{ + int64_t const n = rows * halfD; + cudaLaunchConfig_t cfg{}; + cfg.gridDim = dim3(static_cast((n + kThreads - 1) / kThreads)); + cfg.blockDim = dim3(kThreads); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return pdl ? cudaLaunchKernelEx(&cfg, swigluOaiKernel, x, out, rows, halfD) + : cudaLaunchKernelEx(&cfg, swigluOaiKernel, x, out, rows, halfD); +} + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.h new file mode 100644 index 000000000000..f2898e100006 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.h @@ -0,0 +1,43 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp4 +{ + +//! The dense MLP's swigluoai activation: out[m, j] = bf16(min(g, 7) * sigmoid(1.702 * min(g, 7)) * (clamp(u, -7, 7) + +//! 1)) with g = x[m, j], u = x[m, d + j] (x bf16 [rows, 2d], out bf16 [rows, d], both contiguous, d even): bit for bit +//! the Triton silu_and_mul kernel at swiglu_limit 7, swiglu_alpha 1.702, swiglu_beta 1. With `pdl` the launch allows +//! programmatic serialization and the grid waits on its dependency, then releases its dependents before it computes +//! (the down GEMV stages its weights during the activation and reads it only after its own grid wait). +cudaError_t invokeSwigluOai( + uint32_t const* x, uint32_t* out, int64_t rows, int64_t halfD, bool pdl, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp4 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.cu new file mode 100644 index 000000000000..fb28ce95250b --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.cu @@ -0,0 +1,711 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.h" + +#include +#include +#include + +// One pass of the fused greedy Eagle3 draft loop of MiniMax-M3 (GB300, TP4, <= 8 requests per pass). +// +// One kernel replaces the draft lm_head GEMM, the logits all-gather / float cast / argmax of the in-tree greedy draft +// sampling and the draft loop's per-pass bookkeeping, at the in-tree precision: +// +// * The vocab-sharded BF16 lm_head weight is stored in mma.m16n8k16 A-fragment order (the same values): per 16-row +// tile and k16 step, 512 bytes; lane (g, j) holds 16 bytes = {row g k 2j..2j+1, row g+8 k 2j.., row g k 2j+8.., +// row g+8 k 2j+8..}. +// * Or, by default, in a lossless 12-bit coding of those fragments (75% of the bytes; the target head's coding, see +// targetHead.cu): per 16-row tile and unit of 4 k16 steps a lane reads three 16-byte words, the sign/mantissa bytes +// of steps 0-1, of steps 2-3, then 4 code words (exponent - base, 4 bits; 15 = an exception). The decoded values +// are the checkpoint's BF16 bits; an exception decodes to +0 and the segment's flush adds the exact product x * w +// from the tile's list (products of BF16 values are exact in FP32). +// * A persistent grid (one 16-warp CTA per SM) streams whole tiles over the full K with BF16 MMAs and FP32 +// accumulation; the BF16 head input rows sit in shared memory in B-fragment order, unchanged. +// * Each logit is rounded to BF16 (the in-tree logits are the BF16 GEMM output) and becomes a 64-bit key +// ordered(value) << 32 | ~global_index, with -0 taken as +0: the larger value wins and ties pick the lowest index, as +// torch.max / torch.argmax over the gathered logits do. Keys are max-reduced per row: warp shuffles, then shared +// memory, then atomicMax. +// * The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping +// (Lamport: 0 = empty, a key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the +// slots, and runs the pass's bookkeeping. Call n uses slot half n % 2. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +namespace draftStep +{ +namespace +{ +constexpr int kUnroll = 8; // k16 chunks in flight per warp +constexpr int kChunkBytes = 512; // one tile's k16 step: 32 lanes x 16 bytes +constexpr long long kMaxSpins = 1LL << 31; // cross-rank key poll bound + +// Coded units (4 k16 steps, 3 x 16 bytes per lane) in flight per warp: 3 up to 4 rows, 2 at 8 rows. +template +constexpr int codedUnroll() +{ + return MP >= 8 ? 2 : 3; +} + +__device__ __forceinline__ void mma_bf16( + float (&d)[4], uint32_t a0, uint32_t a1, uint32_t a2, uint32_t a3, uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.bf16.bf16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};\n" + : "+f"(d[0]), "+f"(d[1]), "+f"(d[2]), "+f"(d[3]) + : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(b0), "r"(b1)); +} + +// BF16 pair p (values 2p, 2p + 1) of a lane's 8 coded values: sign/mantissa bytes sm (4 per word), codes word cw. +__device__ __forceinline__ uint32_t coded_decode_pair(uint32_t sm_word, int byte0, uint32_t cw, int p, uint32_t base) +{ + uint32_t const b = __byte_perm(sm_word, 0u, byte0 == 0 ? 0x4140 : 0x4342); // {b0, 0, b1, 0} + uint32_t const q = (cw >> (4 * p)) & 0x000f000fu; + uint32_t const e = q * 128u + base; // exponent field of both halves + // Code 15 (an exception, sign/mantissa byte 0) decodes to +0. + return ((b & 0x007f007fu) | ((b << 8) & 0x80008000u) | e) & __vcmpne2(q, 0x000f000fu); +} + +// One coded unit (4 k16 steps) against the BF16 B fragments: steps alternate between the two accumulators, as the +// k16 steps of the plain BF16 head do. +template +__device__ __forceinline__ void chunk_mma_coded(uint4 const& sm01, uint4 const& sm23, uint4 const& cw, uint32_t base, + uint2 const* xs, int unit, int j, int g, bool b_active, float (&da)[4], float (&db)[4]) +{ + uint32_t const smw[8] = {sm01.x, sm01.y, sm01.z, sm01.w, sm23.x, sm23.y, sm23.z, sm23.w}; + uint32_t const cws[4] = {cw.x, cw.y, cw.z, cw.w}; +#pragma unroll + for (int ci = 0; ci < 4; ++ci) + { + uint2 b = make_uint2(0u, 0u); + if (b_active) + b = xs[((unit * 4 + ci) * 4 + j) * MP + g]; + uint32_t const w0 = coded_decode_pair(smw[2 * ci], 0, cws[ci], 0, base); + uint32_t const w1 = coded_decode_pair(smw[2 * ci], 2, cws[ci], 1, base); + uint32_t const w2 = coded_decode_pair(smw[2 * ci + 1], 0, cws[ci], 2, base); + uint32_t const w3 = coded_decode_pair(smw[2 * ci + 1], 2, cws[ci], 3, base); + if (ci & 1) + mma_bf16(db, w0, w1, w2, w3, b.x, b.y); + else + mma_bf16(da, w0, w1, w2, w3, b.x, b.y); + } +} + +// Exceptions of `tile` with k in [kb, ke): v += x * w (exact) for the owner lanes (row g or g + 8; tokens 2j, +// 2j + 1 < M), x from the staged B fragments. Warp-uniform (shuffles). +template +__device__ __forceinline__ void coded_exceptions( + CodedArgs const& cd, int tile, int kb, int ke, uint2 const* xs, int M, int lane, float (&v)[4]) +{ + int const g = lane >> 2, j = lane & 3; + int const e0 = __ldg(cd.exc_off + tile), e1 = __ldg(cd.exc_off + tile + 1); + for (int b0 = e0; b0 < e1; b0 += 32) + { + uint32_t pos = 0xffffffffu; + float2 wv = make_float2(0.f, 0.f); + if (b0 + lane < e1) + { + pos = __ldg(cd.exc_pos + b0 + lane); + wv = __ldg(cd.exc_w + b0 + lane); + } + int const cnt = min(32, e1 - b0); + for (int q = 0; q < cnt; ++q) + { + uint32_t const pq = __shfl_sync(0xffffffffu, pos, q); + float const w0 = __shfl_sync(0xffffffffu, wv.x, q); + int const r = static_cast(pq >> 16), k = static_cast(pq & 0xffffu); + if (k >= kb && k < ke && (r & 7) == g) + { + int const st = k >> 4, rr = k & 15; + bool const hi = r >= 8; +#pragma unroll + for (int q2 = 0; q2 < 2; ++q2) + { + int const n = 2 * j + q2; + if (n < M) + { + uint2 const bw = xs[(st * 4 + ((rr & 7) >> 1)) * MP + n]; + uint32_t const word = rr >= 8 ? bw.y : bw.x; + float const xv = __uint_as_float((rr & 1) ? (word & 0xffff0000u) : (word << 16)); + float const d = __fmul_rn(xv, w0); + if (hi) + v[2 + q2] += d; + else + v[q2] += d; + } + } + } + __syncwarp(); + } + } +} + +__device__ __forceinline__ uint4 ldg_stream(uint4 const* p) +{ + uint4 r; + asm volatile("ld.global.nc.L1::no_allocate.L2::256B.v4.u32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(r.x), "=r"(r.y), "=r"(r.z), "=r"(r.w) + : "l"(p)); + return r; +} + +// The argmax value of a logit: its BF16 rounding (the in-tree logits' dtype), -0 as +0. +__device__ __forceinline__ float key_value(float v) +{ + float const r = __bfloat162float(__float2bfloat16(v)); + return r == 0.f ? 0.f : r; +} + +__device__ __forceinline__ unsigned long long make_key(float v, uint32_t index) +{ + uint32_t u = __float_as_uint(v); + u = (u & 0x80000000u) ? ~u : (u | 0x80000000u); + return (static_cast(u) << 32) | static_cast(0xFFFFFFFFu - index); +} + +__device__ __forceinline__ unsigned long long shfl_max(unsigned long long k, int o) +{ + unsigned long long const ok = __shfl_xor_sync(0xffffffffu, k, o); + return ok > k ? ok : k; +} + +// One k16 step of a tile: the lane's A fragment w against the activation B fragment of step `chunk` (rows g < MP). +// Even and odd steps accumulate separately (two independent MMA chains); the tile's sum is da + db. +template +__device__ __forceinline__ void chunk_mma( + uint4 const& w, uint2 const* xs, int chunk, int j, int g, bool b_active, float (&da)[4], float (&db)[4]) +{ + uint2 b = make_uint2(0u, 0u); + if (b_active) + b = xs[(chunk * 4 + j) * MP + g]; + if (chunk & 1) + mma_bf16(db, w.x, w.y, w.z, w.w, b.x, b.y); + else + mma_bf16(da, w.x, w.y, w.z, w.w, b.x, b.y); +} + +// A warp's pieces of the tiles it shares with other warps (its first and last local tile), kept for the warp-order sum. +struct Pieces +{ + float4 head = make_float4(0.f, 0.f, 0.f, 0.f); + float4 tail = make_float4(0.f, 0.f, 0.f, 0.f); +}; + +// Whether the warp's chunk range [q_begin, q_end) covers the CTA's whole range of local tile tl (the CTA streams +// chunks [lo, lo + span) of its local tiles). +__device__ __forceinline__ bool covers_tile(int tl, int kchunks, int lo, int span, int q_begin, int q_end) +{ + return q_begin <= max(tl * kchunks, lo) && q_end >= min((tl + 1) * kchunks, lo + span); +} + +// Local tile t's sum (D-FIX): the accumulator when one warp streamed the CTA's whole range of it, else its pieces +// from the warps that streamed it, added in warp order from 0 (each warp's first tile is its head piece, any later +// one its tail piece). Deterministic: the order is fixed by the launch shape, and with two pieces it is the same +// sum as the atomics' (two addends from 0 commute). +__device__ __forceinline__ float4 ordered_tile( + float4 const* acc_t, float4 const* stage, int t, int kchunks, int lo, int span, int lane) +{ + int const r_lo = max(t * kchunks, lo); + int const r_last = min((t + 1) * kchunks, lo + span) - 1; + int w_lo = 0, w_hi = 0; + for (int w = 1; w < kWarps; ++w) + { + int const qb = lo + w * span / kWarps; + if (qb <= r_lo) + w_lo = w; + if (qb <= r_last) + w_hi = w; + } + if (w_lo == w_hi) + return acc_t[lane]; + float4 s = make_float4(0.f, 0.f, 0.f, 0.f); + for (int w = w_lo; w <= w_hi; ++w) + { + int const qb = lo + w * span / kWarps; + float4 const x = stage[(w * 2 + (qb / kchunks == t ? 0 : 1)) * 32 + lane]; + s.x += x.x; + s.y += x.y; + s.z += x.z; + s.w += x.w; + } + return s; +} + +// Add a warp's segment of a tile into the CTA's accumulator (with a coded head, the segment's exceptions first); +// with `fixed` (D-FIX) a shared tile's piece goes to `pc` instead (ordered_tile adds it). +template +__device__ __forceinline__ void flush_tile(float4* acc, float (&da)[4], float (&db)[4], Params const& p, int tile, + int kb, int ke, uint2 const* xs, int lane, Pieces& pc, bool fixed, bool whole, bool first) +{ + float v[4] = {da[0] + db[0], da[1] + db[1], da[2] + db[2], da[3] + db[3]}; + if constexpr (CODED) + coded_exceptions(p.cd, tile, kb, ke, xs, p.M, lane, v); + if (fixed) + { + // A piece covering the CTA's whole range of the tile is its only addend (0 + v, as the atomic gave); a + // warp's other pieces (at most its first and last tile) wait in registers for the warp-order sum. + if (whole) + *acc = make_float4(0.f + v[0], 0.f + v[1], 0.f + v[2], 0.f + v[3]); + else if (first) + pc.head = make_float4(v[0], v[1], v[2], v[3]); + else + pc.tail = make_float4(v[0], v[1], v[2], v[3]); + } + else + { + float* a = reinterpret_cast(acc); + atomicAdd(a + 0, v[0]); + atomicAdd(a + 1, v[1]); + atomicAdd(a + 2, v[2]); + atomicAdd(a + 3, v[3]); + } +#pragma unroll + for (int e = 0; e < 4; ++e) + { + da[e] = 0.f; + db[e] = 0.f; + } +} + +// Balanced streaming GEMV over the fragment-ordered weight, persistent grid of 16-warp CTAs. CTA c owns tiles +// [t_begin, t_end); its (tile, chunk) space is split into 16 equal contiguous ranges, one per warp, so every warp +// streams the same number of bytes. A tile straddles at most two warps; their partial sums are added into shared +// memory (two addends: the order does not change the sum). MP = rows padded to 1 / 2 / 4 / 8. A chunk is one k16 +// step of the BF16 head, or one coded unit (4 k16 steps) of the coded head (CODED: no shared-memory prologue). +template +__global__ void __launch_bounds__(kWarps * 32, 1) + draft_step_kernel(Params const p, int kchunks, int pf_chunks, int pf_off) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(16) uint2 xs[]; // [K/16 steps][4 (j)][MP (g)], then the weight prologue + __shared__ float4 acc_s[kMaxTilesPerCta + 1][32]; // + 1: an equal-chunk span can touch one more tile + __shared__ unsigned long long red[kWarps][kMaxTokens]; + __shared__ __align__(8) unsigned long long pf_bar[kWarps]; + __shared__ bool s_last; + __shared__ int s_tok[kMaxTokens]; + + int const warp = threadIdx.x >> 5; + int const lane = threadIdx.x & 31; + int const g = lane >> 2; + int const j = lane & 3; + constexpr int kChunkK = CODED ? 64 : 16; // k per chunk + constexpr int kWords = CODED ? 3 : 1; // 16-byte words per lane and chunk + constexpr int KU = CODED ? codedUnroll() : kUnroll; + int const K = kchunks * kChunkK; + int const M = p.M; + int const r_lo = p.tile_lo; + int const r_n = (p.tile_hi > 0 ? p.tile_hi : p.n_tiles) - r_lo; + bool const eq = p.bnd != nullptr; + // Whole tiles: CTA c owns tiles [t_begin, t_end). Equal chunks: CTA c streams chunks [c0, c1) of the range, which + // start off0 chunks into tile t_begin (its first and last tiles may be shared with the neighbouring CTAs). + long long const r_chunks = static_cast(r_n) * kchunks; + long long const c0 = eq ? (static_cast(blockIdx.x) * r_chunks) / gridDim.x : 0; + long long const c1 = eq ? (static_cast(blockIdx.x + 1) * r_chunks) / gridDim.x : 0; + int const t_begin = r_lo + + (eq ? static_cast(c0 / kchunks) + : static_cast((static_cast(blockIdx.x) * r_n) / gridDim.x)); + int const t_end = r_lo + + (eq ? static_cast((c1 + kchunks - 1) / kchunks) + : static_cast((static_cast(blockIdx.x + 1) * r_n) / gridDim.x)); + int const n_local = t_end - t_begin; + int const off0 = eq ? static_cast(c0 - static_cast(t_begin - r_lo) * kchunks) : 0; + int const span = eq ? static_cast(c1 - c0) : n_local * kchunks; + int const q_begin = off0 + warp * span / kWarps; + int const q_end = off0 + (warp + 1) * span / kWarps; + // Equal chunks: whether the first / last local tile is shared with the previous / next CTA of the launch. + bool const shared_first = eq && off0 != 0; + bool const shared_last = eq && ((c1 % kchunks) != 0); + long long const base = static_cast(t_begin) * kchunks; // first chunk of this CTA + uint4 const* const wf = p.wf; + + // Weights are static: start this warp's stream before waiting on the producer of x. The warp's first pf_chunks + // chunks come in by one bulk copy into shared memory (dynamic offset pf_off), which fills the grid-dependency + // wait and the staging; the register ring starts after them. + int const n_pf = CODED ? 0 : min(pf_chunks, q_end - q_begin); + uint4* const wpf = reinterpret_cast(reinterpret_cast(xs) + pf_off) + warp * pf_chunks * 32; + if (n_pf > 0 && lane == 0) + { + uint32_t const bar = static_cast(__cvta_generic_to_shared(&pf_bar[warp])); + uint32_t const bytes = static_cast(n_pf) * kChunkBytes; + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(bar) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(bar), "r"(bytes) : "memory"); + asm volatile("cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes [%0], [%1], %2, [%3];" ::"r"( + static_cast(__cvta_generic_to_shared(wpf))), + "l"(wf + (base + q_begin) * 32), "r"(bytes), "r"(bar) + : "memory"); + } + __syncwarp(); + int const q_reg = q_begin + n_pf; + uint4 buf[KU][kWords]; +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q_reg + u < q_end) + { +#pragma unroll + for (int l = 0; l < kWords; ++l) + buf[u][l] = ldg_stream(wf + ((base + q_reg + u) * kWords + l) * 32 + lane); + } + } + for (int i = threadIdx.x; i < n_local * 32; i += blockDim.x) + acc_s[i / 32][i % 32] = make_float4(0.f, 0.f, 0.f, 0.f); + + asm volatile("griddepcontrol.wait;" ::: "memory"); + + // Stage the head input rows in B-fragment order: item (k16 step S, row gg) = two 16-byte loads of x, four + // fragment words {x[2jj], x[2jj + 1]}, {x[2jj + 8], x[2jj + 9]} (BF16 pairs, moved unchanged). Rows >= M are 0. + constexpr int kSU = MP >= 4 ? 4 : 2; + int const n_items = (K / 16) * MP; + for (int i0 = threadIdx.x; i0 < n_items; i0 += kSU * blockDim.x) + { + uint4 xv[kSU][2]; +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + int const gg = i % MP; + int const S = i / MP; + xv[u][0] = make_uint4(0u, 0u, 0u, 0u); + xv[u][1] = make_uint4(0u, 0u, 0u, 0u); + if (i < n_items && gg < M) + { + long long const xr = p.x_rows != nullptr ? p.x_rows[gg] : gg; + uint4 const* xp = reinterpret_cast(p.x + xr * K + 16 * S); + xv[u][0] = xp[0]; + xv[u][1] = xp[1]; + } + } +#pragma unroll + for (int u = 0; u < kSU; ++u) + { + int const i = i0 + u * blockDim.x; + if (i >= n_items) + continue; + int const gg = i % MP; + int const S = i / MP; + uint32_t const lo[4] = {xv[u][0].x, xv[u][0].y, xv[u][0].z, xv[u][0].w}; + uint32_t const hi[4] = {xv[u][1].x, xv[u][1].y, xv[u][1].z, xv[u][1].w}; +#pragma unroll + for (int jj = 0; jj < 4; ++jj) + xs[(S * 4 + jj) * MP + gg] = make_uint2(lo[jj], hi[jj]); + } + } + __syncthreads(); + + bool const b_active = g < MP; + float da[4] = {0.f, 0.f, 0.f, 0.f}; + float db[4] = {0.f, 0.f, 0.f, 0.f}; + int tl = q_begin / kchunks; // local tile of the current accumulator + int c = q_begin - tl * kchunks; // chunk within that tile + int cs = c; // first chunk of the warp's segment of that tile + int const tl0 = tl; // the warp's first local tile (its head piece) + Pieces pc; + if (n_pf > 0) + { + uint32_t const bar = static_cast(__cvta_generic_to_shared(&pf_bar[warp])); + asm volatile( + "{\n .reg .pred p;\n PF_WAIT%=:\n mbarrier.try_wait.parity.shared::cta.b64 p, [%0], 0;\n" + " @!p bra PF_WAIT%=;\n}\n" ::"r"(bar) + : "memory"); + for (int q = q_begin; q < q_reg; ++q) + { + chunk_mma(wpf[(q - q_begin) * 32 + lane], xs, c, j, g, b_active, da, db); + if (++c == kchunks) + { + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, K, xs, lane, pc, + p.fixedOrder, covers_tile(tl, kchunks, off0, span, q_begin, q_end), tl == tl0); + c = 0; + cs = 0; + ++tl; + } + } + } + for (int q = q_reg; q < q_end; q += KU) + { + uint4 nxt[KU][kWords]; +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q + KU + u < q_end) + { +#pragma unroll + for (int l = 0; l < kWords; ++l) + nxt[u][l] = ldg_stream(wf + ((base + q + KU + u) * kWords + l) * 32 + lane); + } + } +#pragma unroll + for (int u = 0; u < KU; ++u) + { + if (q + u < q_end) + { + if constexpr (CODED) + chunk_mma_coded(buf[u][0], buf[u][kWords > 1 ? 1 : 0], buf[u][kWords > 2 ? 2 : 0], p.cd.base, + xs, c, j, g, b_active, da, db); + else + chunk_mma(buf[u][0], xs, c, j, g, b_active, da, db); + if (++c == kchunks) + { + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, K, xs, lane, pc, + p.fixedOrder, covers_tile(tl, kchunks, off0, span, q_begin, q_end), tl == tl0); + c = 0; + cs = 0; + ++tl; + } + } + } +#pragma unroll + for (int u = 0; u < KU; ++u) +#pragma unroll + for (int l = 0; l < kWords; ++l) + buf[u][l] = nxt[u][l]; + } + if (c != 0) + flush_tile(&acc_s[tl][lane], da, db, p, t_begin + tl, cs * kChunkK, c * kChunkK, xs, lane, pc, + p.fixedOrder, covers_tile(tl, kchunks, off0, span, q_begin, q_end), tl == tl0); + __syncthreads(); + // D-FIX: stage every warp's head / tail piece. xs and the weight prologue are dead after the barrier above (the + // launcher sizes them to hold the 16 x 2 x 32 pieces). + float4* const stage = reinterpret_cast(xs); + if (p.fixedOrder) + { + stage[(warp * 2) * 32 + lane] = pc.head; + stage[(warp * 2 + 1) * 32 + lane] = pc.tail; + __syncthreads(); + } + + // Accumulator lane (g, j) of a tile: rows g / g + 8 (x, y / z, w) of tokens 2j / 2j + 1 (x, z / y, w). + unsigned long long best0 = 0ull, best1 = 0ull; // tokens 2j, 2j + 1 + for (int t = warp; t < n_local; t += kWarps) + { + float4 v = p.fixedOrder ? ordered_tile(acc_s[t], stage, t, kchunks, off0, span, lane) : acc_s[t][lane]; + if ((t == 0 && shared_first) || (t == n_local - 1 && shared_last)) + { + // A tile shared with a neighbouring CTA: add this CTA's partial into bnd; the second CTA to arrive reads + // the sum (two addends), keys the tile and restores bnd and its counter to zero for the next call. + int const tile = t_begin + t; + float* const bp = reinterpret_cast(p.bnd + static_cast(tile) * 32 + lane); + atomicAdd(bp + 0, v.x); + atomicAdd(bp + 1, v.y); + atomicAdd(bp + 2, v.z); + atomicAdd(bp + 3, v.w); + __threadfence(); + __syncwarp(); + unsigned int prev = 0; + if (lane == 0) + prev = atomicAdd(p.bnd_cnt + tile, 1u); + prev = __shfl_sync(0xffffffffu, prev, 0); + if (prev == 0) + continue; // the neighbour keys it + __threadfence(); + v = __ldcg(p.bnd + static_cast(tile) * 32 + lane); + p.bnd[static_cast(tile) * 32 + lane] = make_float4(0.f, 0.f, 0.f, 0.f); + if (lane == 0) + p.bnd_cnt[tile] = 0u; + } + int const r0 = (t_begin + t) * 16 + g; + uint32_t const i0 = static_cast(p.vocab_offset + r0); + unsigned long long k0 = make_key(key_value(v.x), i0); + unsigned long long const k0b = make_key(key_value(v.z), i0 + 8); + unsigned long long k1 = make_key(key_value(v.y), i0); + unsigned long long const k1b = make_key(key_value(v.w), i0 + 8); + k0 = k0b > k0 ? k0b : k0; + k1 = k1b > k1 ? k1b : k1; + best0 = k0 > best0 ? k0 : best0; + best1 = k1 > best1 ? k1 : best1; + } +#pragma unroll + for (int o = 4; o < 32; o <<= 1) + { + best0 = shfl_max(best0, o); + best1 = shfl_max(best1, o); + } + asm volatile("griddepcontrol.launch_dependents;"); + if (lane < 4) + { + red[warp][2 * lane] = best0; + red[warp][2 * lane + 1] = best1; + } + __syncthreads(); + if (threadIdx.x < M) + { + unsigned long long b = 0ull; +#pragma unroll + for (int w = 0; w < kWarps; ++w) + b = red[w][threadIdx.x] > b ? red[w][threadIdx.x] : b; + if (b != 0ull) + atomicMax(p.keys + threadIdx.x, b); + } + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) + { + unsigned int const prev = atomicAdd(p.counters, 1u); + s_last = prev == (p.total_ctas != 0u ? p.total_ctas : gridDim.x) - 1u; + } + __syncthreads(); + if (!s_last) + return; + + __threadfence(); + unsigned int const call = *reinterpret_cast(p.counters + 1); + int const half = static_cast(call & 1u); + if (threadIdx.x < M) + { + int const t = threadIdx.x; + unsigned long long const mine = atomicExch(p.keys + t, 0ull); + unsigned long long* slot_mc = p.ws_mc + (half * kMaxTokens + t) * p.world; + unsigned long long* slot_uc = p.ws_uc + (half * kMaxTokens + t) * p.world; + slot_mc[p.rank] = mine; + unsigned long long best = 0ull; + for (int r = 0; r < p.world; ++r) + { + unsigned long long v; + long long spins = 0; + do + { + v = *reinterpret_cast(slot_uc + r); + if (++spins > kMaxSpins) + __trap(); // a missing peer key becomes a CUDA error, never a hang (the mc all-reduce's bound) + } while (v == 0ull); + best = v > best ? v : best; + slot_uc[r] = 0ull; + } + s_tok[t] = static_cast(0xFFFFFFFFu - static_cast(best & 0xFFFFFFFFull)); + if (p.out_keys != nullptr) + p.out_keys[t] = best; + } + __syncthreads(); + if (threadIdx.x == 0) + { + p.counters[0] = 0u; + p.counters[1] = call + 1u; + } + if (threadIdx.x < M) + { + StepArgs const& st = p.st; + int const b = threadIdx.x; + int const tok = s_tok[b]; + st.draft_out[b * st.draft_stride + st.step] = tok; + st.ids_next[b] = tok; + long long const r = p.x_rows != nullptr ? p.x_rows[b] : b; + st.pos_next[b] = st.pos_src[r] + 1; + if (st.step == 0 && st.num_acc != nullptr) + { + st.kv_lens[b] -= st.draft_len - st.num_acc[b]; + st.seq_lens[b] = 1; + } + else if (st.step > 0) + { + st.kv_lens[b] += 1; + } + } + if (p.hasFold) + { + // The fold tail (H-R2-3) after this pass's tokens and bookkeeping: the barrier makes this CTA's writes above + // visible to warp 0; every earlier operand was written by grids this one waited on. All CTAs triggered their + // dependents before the exchange, which is safe: a dependent's griddepcontrol.wait waits for this whole grid + // (this tail included) to complete, so only a dependent that read the tail's outputs before its own wait + // could race, and none does (the next step's readers run in the next graph replay). + __syncthreads(); + if (threadIdx.x < 32) + { + foldTailWarp(p.fold, static_cast(threadIdx.x)); + } + } +#endif +} + +template +cudaError_t launch(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + auto kern = draft_step_kernel; + size_t const xs_bytes = static_cast(K / 16) * 4 * MP * sizeof(uint2); + // Shared-memory weight prologue: chunks per warp that fit next to the kernel's static + dynamic shared memory. + static int const sStaticSmem = [] + { + cudaFuncAttributes fa{}; + return cudaFuncGetAttributes(&fa, draft_step_kernel) == cudaSuccess + ? static_cast(fa.sharedSizeBytes) + : -1; + }(); + static int const sMaxSmemPerBlock = [] + { + int dev = 0, maxSmem = 0; + cudaGetDevice(&dev); + cudaDeviceGetAttribute(&maxSmem, cudaDevAttrMaxSharedMemoryPerBlockOptin, dev); + return maxSmem; + }(); + int pf = 0; + if (!CODED && sStaticSmem >= 0) + { + long long const room + = static_cast(sMaxSmemPerBlock) - sStaticSmem - static_cast(xs_bytes) - 1024; + long long const want = static_cast(kPrefetchKb) * 1024; + pf = static_cast(std::max(0ll, std::min(room, want)) / (kWarps * kChunkBytes)); + } + // At least the D-FIX piece staging (16 warps x head / tail x 32 lanes of float4), which reuses this space. + size_t const smem = std::max(xs_bytes + static_cast(pf) * kWarps * kChunkBytes, + static_cast(kWarps) * 2 * 32 * sizeof(float4)); + cudaError_t err = cudaFuncSetAttribute(kern, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smem)); + if (err != cudaSuccess) + return err; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(static_cast(grid)); + cfg.blockDim = dim3(kWarps * 32); + cfg.dynamicSmemBytes = smem; + cfg.stream = stream; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, kern, p, K / (CODED ? 64 : 16), pf, static_cast(xs_bytes)); +} + +template +cudaError_t dispatchRows(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + if (p.M <= 1) + return launch<1, CODED>(p, K, grid, pdl, stream); + if (p.M <= 2) + return launch<2, CODED>(p, K, grid, pdl, stream); + if (p.M <= 4) + return launch<4, CODED>(p, K, grid, pdl, stream); + return launch<8, CODED>(p, K, grid, pdl, stream); +} + +} // namespace + +cudaError_t invokeDraftStep(Params const& p, int K, int grid, bool pdl, cudaStream_t stream) +{ + bool const coded = p.cd.exc_off != nullptr; + if (p.M < 1 || p.M > kMaxTokens || K % (coded ? 64 : 16) != 0 || grid < 1) + return cudaErrorInvalidValue; + return coded ? dispatchRows(p, K, grid, pdl, stream) : dispatchRows(p, K, grid, pdl, stream); +} + +} // namespace draftStep +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.h new file mode 100644 index 000000000000..485f383732fe --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.h @@ -0,0 +1,119 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +//! One pass of the fused greedy Eagle3 draft loop of the MiniMax-M3 NVFP4 TP4 decode path (sm_103a): the BF16 +//! vocab-sharded draft head, the cross-rank argmax and the draft loop's per-pass bookkeeping in one kernel. See +//! draftStep.cu. +namespace draftStep +{ + +constexpr int kMaxTokens = 8; // head rows (requests) of one launch +constexpr int kWarps = 16; // warps per CTA +constexpr int kMaxTilesPerCta = 32; // 16-row weight tiles one CTA can own +//! Shared-memory weight prologue per CTA, in KB (capped by the free shared memory). +constexpr int kPrefetchKb = 112; + +// Draft-pass epilogue (last CTA, one thread per request row b): stores the token as draft token `step`, stages the +// next pass's input id and position, and advances the draft KV lengths the way the in-tree draft loop does between +// passes (pass 0 with num_acc: drop the rejected verify rows and switch to one token per request; later passes: +1). +struct StepArgs +{ + int* draft_out; // [nreq, draft_stride]: column `step` receives the token + int draft_stride; + int step; + int* ids_next; // [nreq] + int const* pos_src; // positions of the rows the head read (indexed like x) + int* pos_next; // [nreq] = pos_src[x_rows[b] or b] + 1 + int* kv_lens; // [nreq] draft KV lengths, updated in place + int const* num_acc; // pass 0: accepted counts (nullptr: no rollback) + int* seq_lens; // pass 0: set to 1 + int draft_len; +}; + +// The 12-bit coded head (see draftStep.cu): the per-tile exception lists and the exponent base. exc_off == nullptr: +// the head is the plain BF16 fragment-ordered copy. +struct CodedArgs +{ + int const* exc_off; // [n_tiles + 1] exception list offsets + uint32_t const* exc_pos; // row in tile << 16 | k, per exception + float2 const* exc_w; // {w, 0} per exception: the exact weight + uint32_t base; // exponent base << 7 in both BF16 halves (base << 7 | base << 23) +}; + +// Everything a draft-step launch takes (see invokeDraftStep). +struct Params +{ + __nv_bfloat16 const* x; // [*, K] head input rows + long long const* x_rows; // optional [M]: head input row of request b (pass 0: the last accepted verify row) + int M; // requests, <= kMaxTokens + uint4 const* wf; // [n_tiles][K / 16][32 lanes] 16-byte BF16 A fragments, or the 12-bit coded stream + // [n_tiles][K / 64][3][32 lanes] when cd.exc_off is set (see draftStep.cu) + int n_tiles; + int vocab_offset; // global id of this rank's first head row + unsigned long long* keys; + unsigned int* counters; + unsigned long long* ws_uc; // this rank's exchange slots [2][kMaxTokens][world] + unsigned long long* ws_mc; // their multicast address + int rank; + int world; + unsigned long long* out_keys; // optional [M]: the winning global key of each row + StepArgs st; + CodedArgs cd; + // The step's forward-tail store + next-step staging (foldTail.h), run by the last CTA after this pass's + // tokens and bookkeeping: set on the drafter's last pass only (dispatch.DRAFT_STEP_FOLD_TAIL). + FoldTailParams fold{}; + bool hasFold = false; + // D-FIX: sum the pieces of a tile that several warps stream in warp order (deterministic) instead of by + // shared-memory atomics in arrival order (dispatch.DRAFT_STEP_FIXED_ORDER). + bool fixedOrder = false; + // A pass split over several launches (one per locality domain, each on that domain's SMs): this launch streams + // tiles [tile_lo, tile_hi) (tile_hi 0 = n_tiles), and the last of total_ctas CTAs over all of the pass's launches + // runs the exchange and the bookkeeping (0 = this launch's grid: one launch per pass). + int tile_lo = 0; + int tile_hi = 0; + unsigned int total_ctas = 0; + // Equal-chunk split (set both, zero-filled [n_tiles][32] float4 / [n_tiles], restored to zero by the kernel): each + // CTA of a launch streams an equal share of its range's (tile, chunk) space instead of whole tiles, so a tile at a + // CTA boundary gets one partial sum from each of its two CTAs, added in bnd (two addends: the order does not change + // the sum) and keyed by the second to arrive (bnd_cnt). nullptr: whole tiles per CTA. + float4* bnd = nullptr; + unsigned int* bnd_cnt = nullptr; +}; + +//! Launch the draft step over `grid` 512-thread CTAs (K = the hidden size, a multiple of 16; of 64 for a coded head). +cudaError_t invokeDraftStep(Params const& p, int K, int grid, bool pdl, cudaStream_t stream); + +} // namespace draftStep +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.cu new file mode 100644 index 000000000000..68279dc3bfaa --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.cu @@ -0,0 +1,189 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// The TP8 drafter layer's generation preprocess in one launch. The rotation is applyBiasRopeUpdateKVCacheV2's GPT-NeoX +// path at BF16 (apply_rotary_embedding_gptneox: cos * x + sin * pair in fp32, the first rotary half's sin negated, +// dims past the rotary width with (cos, sin) = (1, 0), rounded to BF16); the K / V store is its unit-scale E4M3 +// store (__nv_fp8x2_e4m3 from BF16 pairs), which is also the query's E4M3: torch's CUDA cast of the BF16 query +// saturates the same way (448, not NaN). + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +namespace drafterRopeFp8Kv +{ +namespace +{ + +constexpr int kElemsPerThread = kHeadDim / 32; +constexpr int kBlockSize = 256; + +template +__device__ __forceinline__ float rotate(float c, float x, float s, float pair) +{ + if constexpr (kRopeMode == 0) + { + return __fadd_rn(__fmul_rn(c, x), __fmul_rn(s, pair)); + } + else if constexpr (kRopeMode == 1) + { + return __fmaf_rn(c, x, __fmul_rn(s, pair)); + } + else + { + return __fmaf_rn(s, pair, __fmul_rn(c, x)); + } +} + +__device__ __forceinline__ void storeE4m3Pairs(__nv_fp8_e4m3* dst, __nv_bfloat16 const (&v)[kElemsPerThread]) +{ +#pragma unroll + for (int i = 0; i < kElemsPerThread; i += 2) + { + __nv_bfloat162 pair; + pair.x = v[i]; + pair.y = v[i + 1]; + __nv_fp8x2_e4m3 const packed(pair); + reinterpret_cast<__nv_fp8x2_storage_t*>(dst)[i / 2] = packed.__x; + } +} + +template +__global__ void __launch_bounds__(kBlockSize) drafterRopeFp8KvKernel(Params const p) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); +#endif + int const lane = threadIdx.x % 32; + int const warp = (blockIdx.x * blockDim.x + threadIdx.x) / 32; + int const slots = p.numHeadsQ + p.numHeadsKv; + int const token = warp / slots; + if (token >= p.numTokens) + { + return; + } + int const slot = warp % slots; + bool const isQ = slot < p.numHeadsQ; + int const head = isQ ? slot : slot - p.numHeadsQ; + int const request = token / p.qLen; + int const row = token % p.qLen; + int const cacheLen = p.seqLens[request]; + int const kvPos = cacheLen - p.qLen + row; + int const ropePos + = p.specPositionOffsets != nullptr ? p.specPositionOffsets[request * p.qLen + row] + cacheLen - p.qLen : kvPos; + int const rowWidth = (p.numHeadsQ + 2 * p.numHeadsKv) * kHeadDim; + __nv_bfloat16 const* src + = p.qkv + static_cast(token) * rowWidth + (isQ ? head * kHeadDim : (p.numHeadsQ + head) * kHeadDim); + int const half = p.rotaryDim / 2; + float2 const* coef = p.cosSin + static_cast(ropePos) * half; + + __nv_bfloat16 rotated[kElemsPerThread]; +#pragma unroll + for (int i = 0; i < kElemsPerThread; ++i) + { + int const d = lane * kElemsPerThread + i; + bool const firstHalf = d < half; + float const x = __bfloat162float(src[d]); + float const pair = __bfloat162float(src[firstHalf ? d + half : d - half]); + float2 const cs = d < p.rotaryDim ? coef[firstHalf ? d : d - half] : make_float2(1.0f, 0.0f); + float const s = firstHalf ? -cs.y : cs.y; + rotated[i] = __float2bfloat16(rotate(cs.x, x, s, pair)); + } + + if (isQ) + { + storeE4m3Pairs( + p.qOut + (static_cast(token) * p.numHeadsQ + head) * kHeadDim + lane * kElemsPerThread, rotated); + return; + } + + int const block = kvPos / p.tokensPerBlock; + long long const inPage = (static_cast(head) * p.tokensPerBlock + kvPos % p.tokensPerBlock) * kHeadDim + + lane * kElemsPerThread; + long long const pageElems = static_cast(p.numHeadsKv) * p.tokensPerBlock * kHeadDim; + int const* table = p.blockTable + request * p.blockTableRequestStride; + long long const kPage = table[block]; + long long const vPage = table[p.blockTableKvStride + block]; + storeE4m3Pairs(p.pool + kPage * pageElems + inPage, rotated); + + __nv_bfloat16 const* vSrc = p.qkv + static_cast(token) * rowWidth + + (p.numHeadsQ + p.numHeadsKv + head) * kHeadDim + lane * kElemsPerThread; + __nv_bfloat16 v[kElemsPerThread]; +#pragma unroll + for (int i = 0; i < kElemsPerThread; ++i) + { + v[i] = vSrc[i]; + } + storeE4m3Pairs(p.pool + vPage * pageElems + inPage, v); +} + +//! With PDL and maxSharedCarveout, the maximum shared-memory carveout lets the full-context decode's large-smem CTAs +//! (the dependent this kernel releases right after its wait) co-reside on the SMs this small kernel occupies. Set once +//! per instantiation. +template +cudaError_t launchWithMaxSharedCarveout( + cudaLaunchConfig_t const& config, Params const& params, bool pdl, bool maxSharedCarveout) +{ + auto const kernel = drafterRopeFp8KvKernel; + if (pdl && maxSharedCarveout) + { + static cudaError_t const carveout = cudaFuncSetAttribute( + kernel, cudaFuncAttributePreferredSharedMemoryCarveout, cudaSharedmemCarveoutMaxShared); + if (carveout != cudaSuccess) + { + return carveout; + } + } + return cudaLaunchKernelEx(&config, kernel, params); +} + +} // namespace + +cudaError_t launch(Params const& params, bool pdl, bool maxSharedCarveout, cudaStream_t stream) +{ + int const warps = params.numTokens * (params.numHeadsQ + params.numHeadsKv); + int const warpsPerBlock = kBlockSize / 32; + cudaLaunchConfig_t config = {}; + config.gridDim = dim3((warps + warpsPerBlock - 1) / warpsPerBlock); + config.blockDim = dim3(kBlockSize); + config.dynamicSmemBytes = 0; + config.stream = stream; + cudaLaunchAttribute attribute[1]; + attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attribute[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + config.attrs = attribute; + config.numAttrs = 1; + switch (params.ropeMode) + { + case 0: return launchWithMaxSharedCarveout<0>(config, params, pdl, maxSharedCarveout); + case 1: return launchWithMaxSharedCarveout<1>(config, params, pdl, maxSharedCarveout); + case 2: return launchWithMaxSharedCarveout<2>(config, params, pdl, maxSharedCarveout); + default: return cudaErrorInvalidValue; + } +} + +} // namespace drafterRopeFp8Kv +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.h new file mode 100644 index 000000000000..f5316504784a --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.h @@ -0,0 +1,79 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +namespace drafterRopeFp8Kv +{ + +constexpr int kHeadDim = 128; + +struct Params +{ + //! [tokens, (numHeadsQ + 2 * numHeadsKv) * 128] BF16: the drafter layer's packed Q, K, V rows. + __nv_bfloat16 const* qkv; + //! [positions, rotaryDim / 2] (cos, sin) pairs: the layer's rotary_cos_sin table. + float2 const* cosSin; + //! [requests] cache lengths including this step's rows. + int const* seqLens; + //! [requests, qLen] spec-decoding position offsets of a multi-token pass, or null. + int const* specPositionOffsets; + //! [tokens, numHeadsQ, 128] E4M3 query out. + __nv_fp8_e4m3* qOut; + //! Base of the layer's K / V sub-pages, [*, numHeadsKv, tokensPerBlock, 128] E4M3. + __nv_fp8_e4m3* pool; + //! [requests, 2, maxBlocks] sub-page indices: K in row 0, V in row 1. + int const* blockTable; + long long blockTableRequestStride; + long long blockTableKvStride; + int numTokens; + int qLen; + int numHeadsQ; + int numHeadsKv; + int rotaryDim; + int tokensPerBlock; + //! How the rotation's two products are summed: 0 two products then the add, 1 fma(cos, x, sin * pair), + //! 2 fma(sin, pair, cos * x). + int ropeMode; +}; + +//! The MiniMax-M3 TP8 Eagle3 drafter layer's generation preprocess: GPT-NeoX RoPE from the cos/sin table on Q and K, +//! rounded to BF16 (the in-tree preprocess's q output and cache input), then Q, K and V to saturating E4M3: K / V +//! as the in-tree preprocess stores them at unit KV scale, Q as torch's CUDA cast of the BF16 query. Rows map to +//! (request token / qLen, row token % qLen) at cache position seqLen - qLen + row. +//! +//! PDL (pdl): the launch is programmatic; griddepcontrol.wait is the kernel's first instruction and the kernel +//! releases its dependent right after it. The dependent may read only step-invariant data before its own wait. +cudaError_t launch(Params const& params, bool pdl, bool maxSharedCarveout, cudaStream_t stream); + +} // namespace drafterRopeFp8Kv +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.cu new file mode 100644 index 000000000000..be98efebeb5d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.cu @@ -0,0 +1,64 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +namespace +{ + +// Row i of the step (lane c = column), as the Triton fold tail's program i: the store's padded rows, then the next +// step's inputs read from them (zero past each source width). +__global__ void __launch_bounds__(32) foldTailKernel(FoldTailParams const p) +{ + int const i = blockIdx.x; + if (i >= *p.ctrl) + { + return; + } + foldTailRow(p, i, static_cast(threadIdx.x)); +} + +__global__ void __launch_bounds__(32) foldTailOneWarpKernel(FoldTailParams const p) +{ + foldTailWarp(p, static_cast(threadIdx.x)); +} + +} // namespace + +cudaError_t invokeFoldTail(FoldTailParams const& p, int32_t rows, cudaStream_t stream) +{ + foldTailKernel<<>>(p); + return cudaGetLastError(); +} + +cudaError_t invokeFoldTailOneWarp(FoldTailParams const& p, cudaStream_t stream) +{ + foldTailOneWarpKernel<<<1, 32, 0, stream>>>(p); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.h new file mode 100644 index 000000000000..86572b786a81 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.h @@ -0,0 +1,129 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +//! Operands of the speculative step's forward-tail store and next-step staging (kernels/m3_fold_tail.py). Every tensor +//! is int32 except `slots` (int64); strides are in elements. The store views are [slots, width], the sources +//! [rows, width]. +struct FoldTailParams +{ + int32_t const* ctrl; // ctrl[0]: the rows the step armed (0: write nothing) + int64_t const* slots; + int32_t const* srcNew; + int64_t sn0, sn1; + int32_t snW; + int32_t* dstNew; + int64_t dn0, dn1; + int32_t dnW; + int32_t const* srcNext; + int64_t sx0, sx1; + int32_t sxW; + int32_t* dstNext; + int64_t dx0, dx1; + int32_t dxW; + int32_t const* srcDraft; + int64_t sd0, sd1; + int32_t sdW; + int32_t* dstDraft; + int64_t dd0, dd1; + int32_t ddW; + int32_t const* srcLens; + int32_t* dstLens; + int32_t* ids; // [rows * T] + int32_t* draftIds; // [rows * (T - 1)] + int32_t* posOff; // [rows * T] + int32_t* kvOff; // [rows] + int32_t tokensPerStep; +}; + +//! One CTA of 32 lanes per row, `rows` CTAs; the first ctrl[0] rows write. Widths and tokensPerStep must be <= 32. +cudaError_t invokeFoldTail(FoldTailParams const& p, int32_t rows, cudaStream_t stream); + +//! The same writes from ONE warp (one CTA of 32 lanes looping over the rows, foldTailWarp): the in-kernel form a caller +//! such as the drafter's last-pass draft_step runs in its last CTA; launched alone it is that form's oracle. +cudaError_t invokeFoldTailOneWarp(FoldTailParams const& p, cudaStream_t stream); + +#ifdef __CUDACC__ +//! Row i of the tail at lane c (0..31), the work of foldTailKernel's CTA i, for a caller that runs the tail inside its +//! own kernel (e.g. the drafter's last-pass draft_step in its last CTA, after its final tokens are written and fenced). +//! The caller runs rows i < *p.ctrl (one warp per row, or one warp looping over the rows); same code as the kernel, so +//! the writes are bit for bit the standalone tail's. +__device__ __forceinline__ void foldTailRow(FoldTailParams const& p, int i, int c) +{ + int64_t const s = p.slots[i]; + int32_t const nw = c < p.snW ? p.srcNew[i * p.sn0 + c * p.sn1] : 0; + if (c < p.dnW) + { + p.dstNew[s * p.dn0 + c * p.dn1] = nw; + } + int32_t const nxt = c < p.sxW ? p.srcNext[i * p.sx0 + c * p.sx1] : 0; + if (c < p.dxW) + { + p.dstNext[s * p.dx0 + c * p.dx1] = nxt; + } + int32_t const drf = c < p.sdW ? p.srcDraft[i * p.sd0 + c * p.sd1] : 0; + if (c < p.ddW) + { + p.dstDraft[s * p.dd0 + c * p.dd1] = drf; + } + int32_t const ln = p.srcLens[i]; + int32_t const t = p.tokensPerStep; + int32_t const d = t - 1; + if (c == 0) + { + p.dstLens[s] = ln; + p.kvOff[i] = ln - t; + } + if (c < t) + { + p.ids[i * t + c] = c < p.dxW ? nxt : 0; + p.posOff[i * t + c] = ln; + } + if (c < d) + { + p.draftIds[i * d + c] = c < p.ddW ? drf : 0; + } +} + +//! The whole tail from one warp of a running kernel (lane = threadIdx.x % 32): rows i < *p.ctrl in order. +__device__ __forceinline__ void foldTailWarp(FoldTailParams const& p, int lane) +{ + int const rows = *p.ctrl; + for (int i = 0; i < rows; ++i) + { + foldTailRow(p, i, lane); + } +} +#endif + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.cu new file mode 100644 index 000000000000..e7505fa48e13 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.cu @@ -0,0 +1,268 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// The split-KV merge for the MiniMax-M3 NVFP4 TP2 (Rubin) full-context decode. The producer is the PrimTS split-KV +// decode with its separate-reduction scratch: per (request, KV head, split, row) one normalized 16-bit partial O and +// one fp32 log2-LSE. Active splits per row follow the producer's split ranges (the causal union of the row's grouped +// Q CTA, instruction-group aligned): splits past them hold nothing and are not read. +// +// One CTA per (request, KV head, kRowsPerCta rows); per row kGroups split groups x kThreadsPerRow threads x 8 dims. +// A thread issues the loads of every split it owns (strided over the groups) before using any, folds them under its +// group's max, and the groups merge through SMEM with one max rescale; the row is normalized once and rounded to +// BF16 once. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +namespace fullCtxReduce +{ +namespace +{ + +constexpr int kGroups = 8; +constexpr int kThreadsPerRow = kHeadDim / 8; +constexpr int kRowsPerCta = 1; +constexpr int kSplitsPerGroup = kMaxSplits / kGroups; +constexpr int kThreads = kGroups * kThreadsPerRow * kRowsPerCta; + +__device__ int activeSplits(Params const& p, int b, int row) +{ + int const seqKv = p.seqLensKv[b]; + int visible = seqKv; + if (p.causal) + { + int const token = row / p.headsPerKv; + int const base = (token / p.qTokensPerCta) * p.qTokensPerCta; + int const end = min(base + p.qTokensPerCta, p.seqLenQ); + visible = min(max(seqKv - p.seqLenQ + end, 0), seqKv); + } + int const tiles = (visible + p.tileKv - 1) / p.tileKv; + int const tilesPerGroup = p.splits * p.instsKv; + int const groups = (tiles + tilesPerGroup - 1) / tilesPerGroup; + int const local = max(groups * p.instsKv, p.instsKv); + return (tiles + local - 1) / local; +} + +template +__device__ float2 unpack(uint32_t word) +{ + if constexpr (kBf16Partial) + { + return __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&word)); + } + else + { + return __half22float2(*reinterpret_cast<__half2 const*>(&word)); + } +} + +template +__global__ void __launch_bounds__(kThreads) fullCtxReduceKernel(Params const p) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + asm volatile("griddepcontrol.wait;" ::: "memory"); + // The dependent (the o_proj activation quantization / GEMV) reads `out` only after its own griddepcontrol.wait, + // which waits for this whole grid, so it is released here and runs its weight-only prologue under the merge. + asm volatile("griddepcontrol.launch_dependents;"); +#endif + int const b = blockIdx.z; + int const h = blockIdx.y; + int const localRow = threadIdx.x / (kGroups * kThreadsPerRow); + int const row = blockIdx.x * kRowsPerCta + localRow; + int const g = (threadIdx.x / kThreadsPerRow) % kGroups; + int const t = threadIdx.x % kThreadsPerRow; + bool const validRow = row < p.rows; + int const active = validRow ? activeSplits(p, b, row) : 0; + + __shared__ float sMax[kRowsPerCta][kGroups]; + __shared__ float sSum[kRowsPerCta][kGroups]; + __shared__ float sAcc[kRowsPerCta][kGroups][kHeadDim]; + + uint4 const* partialO = static_cast(p.partialO); + size_t const kvIdx = static_cast(b) * p.numKvHeads + h; + float lse[kSplitsPerGroup]; + uint4 vec[kSplitsPerGroup]; +#pragma unroll + for (int i = 0; i < kSplitsPerGroup; ++i) + { + int const s = g + i * kGroups; + if (s < active) + { + size_t const r = (kvIdx * p.splits + s) * p.rows + row; + lse[i] = __ldcg(p.partialStats + r); + vec[i] = __ldcg(partialO + r * kThreadsPerRow + t); + } + } + float groupMax = -CUDART_INF_F; +#pragma unroll + for (int i = 0; i < kSplitsPerGroup; ++i) + { + if (g + i * kGroups < active) + { + groupMax = fmaxf(groupMax, lse[i]); + } + } + // A split of the CTA's causal union can hold no key of this row (log2-LSE -inf). Such a split, and a group of + // only such splits, is skipped rather than weighted 0: its partial O carries no defined value. + float acc[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; + float sum = 0.f; +#pragma unroll + for (int i = 0; i < kSplitsPerGroup; ++i) + { + if (g + i * kGroups < active && lse[i] != -CUDART_INF_F) + { + float const w = exp2f(lse[i] - groupMax); + sum += w; + uint32_t const* words = reinterpret_cast(&vec[i]); +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float2 const f = unpack(words[j]); + acc[2 * j] += w * f.x; + acc[2 * j + 1] += w * f.y; + } + } + } + if (t == 0) + { + sMax[localRow][g] = groupMax; + sSum[localRow][g] = sum; + } +#pragma unroll + for (int j = 0; j < 8; ++j) + { + sAcc[localRow][g][t * 8 + j] = acc[j]; + } + __syncthreads(); + if (g != 0 || !validRow) + { + return; + } + float rowMax = -CUDART_INF_F; +#pragma unroll + for (int k = 0; k < kGroups; ++k) + { + rowMax = fmaxf(rowMax, sMax[localRow][k]); + } + float total = 0.f; + float o[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; +#pragma unroll + for (int k = 0; k < kGroups; ++k) + { + if (sMax[localRow][k] == -CUDART_INF_F) + { + continue; + } + float const scale = exp2f(sMax[localRow][k] - rowMax); + total += scale * sSum[localRow][k]; +#pragma unroll + for (int j = 0; j < 8; ++j) + { + o[j] += scale * sAcc[localRow][k][t * 8 + j]; + } + } + float const inv = total > 0.f ? 1.f / total : 0.f; + int const token = row / p.headsPerKv; + int const head = row % p.headsPerKv; + __nv_bfloat162 packed[4]; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + packed[j] = __floats2bfloat162_rn(o[2 * j] * inv, o[2 * j + 1] * inv); + } + size_t const outRow + = (static_cast(b) * p.seqLenQ + token) * (p.numKvHeads * p.headsPerKv) + h * p.headsPerKv + head; + reinterpret_cast(p.out + outRow * kHeadDim)[t] = *reinterpret_cast(packed); + if (p.q8 == nullptr) + { + return; + } + // MXFP8 copy of the BF16 row: a 32-column block is 4 neighbouring threads x 8 columns (lanes 0..15 of warp 0 + // hold the row). Block scale = the UE8M0 exponent of ceil(amax / 448), values = satfinite E4M3 of x * 2^-scale. + float x[8]; + float amax = 0.f; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + float2 const f = __bfloat1622float2(packed[j]); + x[2 * j] = f.x; + x[2 * j + 1] = f.y; + amax = fmaxf(amax, fmaxf(fabsf(f.x), fabsf(f.y))); + } + amax = fmaxf(amax, __shfl_xor_sync(0x0000ffffu, amax, 1)); + amax = fmaxf(amax, __shfl_xor_sync(0x0000ffffu, amax, 2)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + uint16_t q2[4]; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2[j]) : "f"(x[2 * j + 1] * qs), "f"(x[2 * j] * qs)); + } + int const tok = b * p.seqLenQ + token; + int const col = (h * p.headsPerKv + head) * kHeadDim + t * 8; + *reinterpret_cast(p.q8 + static_cast(tok) * (p.numSfKTiles * 128) + col) + = *reinterpret_cast(q2); + if (t % 4 == 0) + { + int const kSf = col / 32; + int64_t const off = static_cast(tok / 128) * p.numSfKTiles * 512 + (kSf / 4) * 512 + (tok % 32) * 16 + + ((tok % 128) / 32) * 4 + kSf % 4; + p.sf[off] = static_cast(e); + } +} + +} // namespace + +cudaError_t invokeFullCtxReduce(Params const& params, bool pdl, cudaStream_t stream) +{ + if (params.splits < 1 || params.splits > kMaxSplits || params.rows < 1) + { + return cudaErrorInvalidValue; + } + cudaLaunchConfig_t config = {}; + config.gridDim = dim3((params.rows + kRowsPerCta - 1) / kRowsPerCta, params.numKvHeads, params.batch); + config.blockDim = dim3(kThreads); + config.stream = stream; + cudaLaunchAttribute attribute[1]; + attribute[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attribute[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + config.attrs = attribute; + config.numAttrs = 1; + return params.bf16Partial ? cudaLaunchKernelEx(&config, fullCtxReduceKernel, params) + : cudaLaunchKernelEx(&config, fullCtxReduceKernel, params); +} + +} // namespace fullCtxReduce +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.h new file mode 100644 index 000000000000..1095ec4e020d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.h @@ -0,0 +1,77 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +//! The split-KV merge of the MiniMax-M3 TP2 full-context decode (PrimTS split-KV decode, separate-reduction scratch) +//! straight to BF16. See fullCtxReduce.cu. +namespace fullCtxReduce +{ + +constexpr int kHeadDim = 128; +constexpr int kMaxSplits = 128; + +struct Params +{ + void const* partialO; // [B, Hkv, S, R, 128] normalized 16-bit partial O per split + float const* partialStats; // [B, Hkv, S, R] log2-LSE per split + int const* seqLensKv; // [B] + __nv_bfloat16* out; // [B, SQ, Hkv * hR, 128] + int batch; + int numKvHeads; + int splits; // S, the planned split capacity (<= kMaxSplits) + int rows; // R = hR * SQ, rows [token, local head] + int seqLenQ; // SQ + int headsPerKv; // hR + int qTokensPerCta; // query tokens one producer CTA owns (grouped Q) + int tileKv; // producer K/V tile + int instsKv; // producer K/V instructions per loop step + bool causal; + bool bf16Partial; + uint8_t* q8 = nullptr; // optional E4M3 copy of out [B * SQ, Hkv * hR * 128] (the o_proj MXFP8 hand-off) + uint8_t* sf = nullptr; // its UE8M0 block scales, 128x4-swizzled as trtllm::mxfp8_quantize(x, True) lays them out + int numSfKTiles = 0; // Hkv * hR * 128 / 128 +}; + +//! Merges the active splits of every row in fp32 with one max rescale and writes BF16; with q8 set, also the MXFP8 +//! copy of the BF16 rows (bit for bit trtllm::mxfp8_quantize of out). With pdl the launch is +//! programmatic: the grid waits on the producer (griddepcontrol.wait) before reading its scratch. +//! +//! PDL contract: griddepcontrol.wait is the kernel's first instruction, so nothing (scratch, lengths, out) is read or +//! written before every prerequisite grid has completed and flushed; a producer's early trigger +//! (griddepcontrol.launch_dependents) only lets this grid launch and wait sooner. The kernel triggers its own +//! dependents right after its wait: the next kernel's griddepcontrol.wait still waits for this whole grid, so its +//! post-wait code sees all of out, and its pre-wait prologue must not read out. +cudaError_t invokeFullCtxReduce(Params const& params, bool pdl, cudaStream_t stream); + +} // namespace fullCtxReduce +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.cu new file mode 100644 index 000000000000..4239255b7450 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.cu @@ -0,0 +1,1096 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/tllmException.h" + +#include +#include +#include +#include + +// The sm_107 TP8 target's own copy of minimaxM3Sm107Tp2/mcAllreduceNormTp2.cu (the version with the Eagle3 capture +// epilogue, cap_out), verbatim but for the names: the TP8 ext carries it, so the TP8 target no longer needs an image +// whose TP2 op has cap_out. The TP2 file's notes follow unchanged. +// +// SM107 (Rubin) copy of minimaxM3/mcAllreduceNorm.cu for the sm_107 TP8 target. Two changes in the arithmetic: the +// rank sum rounds to BF16 before the residual add, the rounding points of the stock all-reduce + fused residual +// RMSNorm on this target (bf16 all-reduce output, then bf16(residual + sum)); and every call but the MoE finalize runs +// the row kernel below, whose RMS statistic is the stock fused norm's reduction tree (bitwise the stock outputs). +// +// The kernels use sm_100-family features (thread-block clusters, cp.async.bulk, st.async, mbarrier transaction +// counts, block-scaled conversions): they are compiled for every sm_10x target of the build, the sm_100f family target +// included (which runs on SM107). A device pass for any other architecture compiles them to a trap, never to a silent +// empty body, and the host entry point checks that the kernel binary it launches holds sm_10x code (see +// checkBinary). +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000 +#define MINIMAX_M3_KERNELS_ENABLED 1 +#else +#define MINIMAX_M3_KERNELS_ENABLED 0 +#endif + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +namespace cg = cooperative_groups; + +namespace +{ + +constexpr int kEmpty = static_cast(0x80000000u); +constexpr int kFinMaxTopk = 8; // MoE finalize: top-k rows loaded together up to this k + +// The Eagle3 capture written by a capture layer's boundary (cap_out, optional): rmsnorm(updated residual) * fc_norm[i], +// bit for bit the m3_spec_tail capture_norm kernel (kernels/minimaxM3/specTail.cu at H 6144: 256 threads, thread c +// sums the squares of the 8-column chunks c, c + 256 and c + 512 in that order, the warp butterfly 16..1, the 8 warp +// sums added in warp order, rsqrtf(sum / H + eps), bf16(v * rs * w)). Only its layout is reproduced here; its +// arithmetic is the same source. +constexpr int kCapThreads = 256; +constexpr int kCapIters = 3; +constexpr int kCapWarps = kCapThreads / 32; +constexpr int kCapHidden = kCapThreads * kCapIters * 8; + +// capture_norm's per-thread chain over one 8-column chunk (continues `ss`). +__device__ __forceinline__ float cap_chunk_sq(uint4 const raw, float ss) +{ + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&raw); +#pragma unroll + for (int q = 0; q < 8; ++q) + { + float const v = __bfloat162float(e[q]); + ss += v * v; + } + return ss; +} + +__device__ __forceinline__ float cap_warp_sum(float v) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + return v; +} + +// bf16(v * rs * w) for one 8-column chunk of the updated residual (v as floats) into cap. +__device__ __forceinline__ void cap_store( + float const (&v)[8], float rs, __nv_bfloat16 const* __restrict__ w, __nv_bfloat16* __restrict__ cap) +{ + uint4 const wr = *reinterpret_cast(w); + __nv_bfloat16 const* we = reinterpret_cast<__nv_bfloat16 const*>(&wr); + uint4 o; + __nv_bfloat16* oe = reinterpret_cast<__nv_bfloat16*>(&o); +#pragma unroll + for (int q = 0; q < 8; ++q) + oe[q] = __float2bfloat16(v[q] * rs * __bfloat162float(we[q])); + *reinterpret_cast(cap) = o; +} + +__device__ __forceinline__ uint32_t canon_bf16x2(uint32_t w) +{ + if ((w & 0xFFFFu) == 0x8000u) + w &= 0xFFFF0000u; + if ((w >> 16) == 0x8000u) + w &= 0x0000FFFFu; + return w; +} + +__device__ __forceinline__ float bf16_lo(uint32_t w) +{ + return __uint_as_float(w << 16); +} + +// The MXFP8 publish quantizes exactly as FlashInfer's CuTe DSL mxfp8 quantize (MXFP8QuantizeRunner.CUTE_DSL, MX-TAC's +// pinned decode quantize; quantization_cute_dsl_utils.py), the three helpers below being its PTX verbatim. It equals +// trtllm::mxfp8_quantize bit for bit except on blocks with 0 < amax < 448 * 2^-127 (UE8M0 byte 0), where the +// TRT-LLM kernel's rcp.approx.ftz of the denormal scale gives inf (NaN / garbage E4M3) and this flushes the block to 0. +__device__ __forceinline__ uint32_t mxfp8_ue8m0(float amax) +{ + float value; + asm("mul.rn.f32 %0, %1, %2;" : "=f"(value) : "f"(amax), "f"(1.0f / 448.0f)); + uint32_t result; + asm("{\n" + " .reg .pred p_zero, p_has_mant, p_exp_zero, p_tiny_sub, p_ovf;\n" + " .reg .u32 bits, exp_biased, mantissa, bump, r;\n" + " setp.le.f32 p_zero, %1, 0f00000000;\n" + " mov.b32 bits, %1;\n" + " shr.b32 exp_biased, bits, 23;\n" + " and.b32 exp_biased, exp_biased, 255;\n" + " and.b32 mantissa, bits, 0x7FFFFF;\n" + " setp.ne.u32 p_has_mant, mantissa, 0;\n" + " selp.u32 bump, 1, 0, p_has_mant;\n" + " setp.eq.u32 p_exp_zero, exp_biased, 0;\n" + " setp.le.u32 p_tiny_sub, mantissa, 0x400000;\n" + " and.pred p_tiny_sub, p_exp_zero, p_tiny_sub;\n" + " @p_tiny_sub mov.u32 bump, 0;\n" + " add.u32 r, exp_biased, bump;\n" + " setp.gt.u32 p_ovf, r, 254;\n" + " selp.u32 r, 254, r, p_ovf;\n" + " selp.u32 %0, 0, r, p_zero;\n" + "}\n" + : "=r"(result) + : "f"(value)); + return result; +} + +__device__ __forceinline__ float mxfp8_inv_scale(uint32_t ue8m0) +{ + float inv; + asm("{\n" + " .reg .s32 new_exp;\n" + " .reg .b32 float_bits;\n" + " .reg .pred p_zero;\n" + " setp.eq.u32 p_zero, %1, 0;\n" + " sub.s32 new_exp, 254, %1;\n" + " max.s32 new_exp, new_exp, 0;\n" + " shl.b32 float_bits, new_exp, 23;\n" + " mov.b32 %0, float_bits;\n" + " @p_zero mov.b32 %0, 0;\n" + "}\n" + : "=f"(inv) + : "r"(ue8m0)); + return inv; +} + +// Two BF16-valued floats (lo, hi) times inv_scale -> two E4M3 bytes (lo in the low byte), saturating to +/-448. +__device__ __forceinline__ uint32_t mxfp8_e4m3x2(float lo, float hi, float inv) +{ + uint16_t pair; + asm("{\n" + " .reg .f32 f0, f1;\n" + " mul.f32 f0, %1, %3;\n" + " mul.f32 f1, %2, %3;\n" + " min.f32 f0, f0, 0f43E00000;\n" + " max.f32 f0, f0, 0fC3E00000;\n" + " min.f32 f1, f1, 0f43E00000;\n" + " max.f32 f1, f1, 0fC3E00000;\n" + " cvt.rn.satfinite.e4m3x2.f32 %0, f1, f0;\n" + "}\n" + : "=h"(pair) + : "f"(lo), "f"(hi), "f"(inv)); + return pair; +} + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ float bf16_hi(uint32_t w) +{ + return __uint_as_float(w & 0xFFFF0000u); +} + +__device__ __forceinline__ int4 ld_volatile_v4(int const* p) +{ + int4 v; + asm volatile("ld.volatile.global.v4.s32 {%0,%1,%2,%3}, [%4];\n" + : "=r"(v.x), "=r"(v.y), "=r"(v.z), "=r"(v.w) + : "l"(p) + : "memory"); + return v; +} + +struct FinalizeArgs +{ + __nv_bfloat16 const* fc2; // [permuted rows, H] routed-expert FC2 output (null: plain input x) + int const* perm; // [M, topk] expanded index -> permuted row, -1 = dropped + void const* scales; // [M, topk] routing weights (fp32 or bf16) + int scale_fp32; + int topk; +}; + +// grid = min(M, MAXT) * CLUSTER CTAs, cluster (CLUSTER, 1, 1), THREADS threads. +// OPTS (each bit-identical): 1 residual + norm weight loaded at entry (before the poll); 2 cluster RMS exchange by +// st.async + mbarrier complete_tx instead of cluster.sync; 4 call parity rotated right after the poll; 8 max-smem +// carveout so the next kernel's CTAs co-reside early. +// OPTS 16 (NOT bit-identical with the others; a different boundary): the Eagle3 drafter's boundary, whose stock is a +// plain BF16 all-reduce, then FlashInfer fused_add_rmsnorm: the RMS statistic and the normalization take the +// unrounded FP32 sum bf16(rank sum) + residual (only the stored residual is rounded to BF16). +// CAP: the Eagle3 capture epilogue (cap_out) is compiled in only where a call passes cap_out, so every other call +// runs the kernel without the capture branches, shared arrays or parameters in its body. +template +__global__ void __launch_bounds__(THREADS) mc_allreduce_norm_kernel(__nv_bfloat16 const* __restrict__ x, + __nv_bfloat16 const* __restrict__ x2, FinalizeArgs fin, int M, int H, __nv_bfloat16 const* __restrict__ residual, + __nv_bfloat16 const* __restrict__ norm_w, float eps, int* ws_uc, int* ws_mc, int* __restrict__ flags, int rank, + int world, __nv_bfloat16* __restrict__ out_normed, __nv_bfloat16* __restrict__ out_residual, + uint8_t* __restrict__ out_q, uint8_t* __restrict__ out_sf, int sf_k_tiles, int prepushed, + __nv_bfloat16* __restrict__ cap_out, int64_t cap_stride, __nv_bfloat16 const* __restrict__ cap_w, float cap_eps) +{ +#if MINIMAX_M3_KERNELS_ENABLED + __shared__ float warp_sq[THREADS / 32]; + __shared__ alignas(16) float cluster_sq[CLUSTER]; + extern __shared__ __align__(16) uint4 cap_slice[]; // [cols / 8] this CTA's updated chunks (cap_out only) + __shared__ alignas(8) unsigned long long xbar; + cg::cluster_group cluster = cg::this_cluster(); + if constexpr ((OPTS & 2) != 0) + { + if (threadIdx.x == 0) + { + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + asm volatile("mbarrier.init.shared::cta.b64 [%0], 1;" ::"r"(b) : "memory"); + asm volatile("fence.mbarrier_init.release.cluster;" ::: "memory"); + } + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + } + int const crank = static_cast(cluster.block_rank()); + int const t = blockIdx.x / CLUSTER; + int const cols = H / CLUSTER; + int const c0 = crank * cols + threadIdx.x * 8; + bool const active = t < M && threadIdx.x * 8 < cols; + int64_t const half_words = static_cast(MAXT) * world * (H / 2); + + // prepushed 1: the producer (m3_mx_gemv) already pushed this rank's rows into the exchange slots; 2: and it + // launched this grid only after its pushes (its grid wait passed), so the slots are polled without a grid wait; + // 3: as 2, with x in set A and x2 in set B. +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (prepushed < 2) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + int const parity = *reinterpret_cast(flags + blockIdx.x) & 1; + uint4 rraw = make_uint4(0u, 0u, 0u, 0u); + uint4 wraw = make_uint4(0u, 0u, 0u, 0u); + if constexpr ((OPTS & 1) != 0) + { + // the residual row was written grids earlier (prepushed >= 2: the producer launched this grid only after its + // own grid wait; otherwise the grid wait above) + if (active) + { + rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + wraw = *reinterpret_cast(norm_w + c0); + } + } + // Push this rank's slice into every rank's buffer. + if (active && !prepushed) + { + uint4 raw; + if (fin.fc2 != nullptr) + { + // MoE finalize: fp32 sum of the routing-weighted top-k expert rows plus the shared expert, rounded once. + float f[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f}; + if (fin.topk <= kFinMaxTopk) + { + // All top-k indices and scales, then every expert-row load in flight at once, then the same in-order + // FP32 accumulation as the loop below (one memory round trip instead of topk; bit-identical). + int prow[kFinMaxTopk]; + float sc[kFinMaxTopk]; +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + prow[k] = -1; + sc[k] = 0.f; + if (k < fin.topk) + { + int const e = t * fin.topk + k; + prow[k] = fin.perm[e]; + sc[k] = fin.scale_fp32 ? static_cast(fin.scales)[e] + : __bfloat162float(static_cast<__nv_bfloat16 const*>(fin.scales)[e]); + } + } + uint4 rv[kFinMaxTopk]; +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + if (prow[k] >= 0) + rv[k] = *reinterpret_cast(fin.fc2 + static_cast(prow[k]) * H + c0); + } +#pragma unroll + for (int k = 0; k < kFinMaxTopk; ++k) + { + if (prow[k] < 0) + continue; + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv[k]); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] += v.x * sc[k]; + f[2 * q + 1] += v.y * sc[k]; + } + } + } + else + for (int k = 0; k < fin.topk; ++k) + { + int const e = t * fin.topk + k; + int const prow = fin.perm[e]; + if (prow < 0) + continue; + float const sc = fin.scale_fp32 + ? static_cast(fin.scales)[e] + : __bfloat162float(static_cast<__nv_bfloat16 const*>(fin.scales)[e]); + uint4 const rv = *reinterpret_cast(fin.fc2 + static_cast(prow) * H + c0); + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] += v.x * sc; + f[2 * q + 1] += v.y * sc; + } + } + if (x2 != nullptr) + { + // As stock: the routed output is rounded to BF16 first, then the shared expert is added (one more + // BF16 rounding of the FP32 sum). + uint4 const rv = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162 const* h = reinterpret_cast<__nv_bfloat162 const*>(&rv); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const routed = __bfloat1622float2(__floats2bfloat162_rn(f[2 * q], f[2 * q + 1])); + float2 const v = __bfloat1622float2(h[q]); + f[2 * q] = routed.x + v.x; + f[2 * q + 1] = routed.y + v.y; + } + } + __nv_bfloat162* o = reinterpret_cast<__nv_bfloat162*>(&raw); +#pragma unroll + for (int q = 0; q < 4; ++q) + o[q] = __floats2bfloat162_rn(f[2 * q], f[2 * q + 1]); + } + else + { + raw = *reinterpret_cast(x + static_cast(t) * H + c0); + } + if (fin.fc2 == nullptr && x2 != nullptr) + { + uint4 const raw2 = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162* a = reinterpret_cast<__nv_bfloat162*>(&raw); + __nv_bfloat162 const* b = reinterpret_cast<__nv_bfloat162 const*>(&raw2); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const fa = __bfloat1622float2(a[q]); + float2 const fb = __bfloat1622float2(b[q]); + a[q] = __floats2bfloat162_rn(fa.x + fb.x, fa.y + fb.y); + } + } + int64_t const slot = parity * half_words + (static_cast(t) * world + rank) * (H / 2) + c0 / 2; + *reinterpret_cast(ws_mc + slot) + = make_int4(static_cast(canon_bf16x2(raw.x)), static_cast(canon_bf16x2(raw.y)), + static_cast(canon_bf16x2(raw.z)), static_cast(canon_bf16x2(raw.w))); + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float s[8] = {}; + float sq = 0.f; + if (active) + { + // The residual row (written grids earlier) and every rank's slot are loaded together; slots still empty are + // re-polled. Ranks are then summed in rank order, as before. + if constexpr ((OPTS & 1) == 0) + rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + constexpr int kMaxWorld = 8; + int* const p0 = ws_uc + parity * half_words + static_cast(t) * world * (H / 2) + c0 / 2; + int4 v[kMaxWorld]; + int4 v2[kMaxWorld]; + bool const two = prepushed == 3; + int* const p1 = p0 + 2 * half_words; // set B + uint32_t pending = (1u << world) - 1u; + uint32_t pending2 = two ? pending : 0u; + long long spins = 0; + while ((pending | pending2) != 0u) + { +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if ((pending >> r) & 1u) + v[r] = ld_volatile_v4(p0 + static_cast(r) * (H / 2)); + if ((pending2 >> r) & 1u) + v2[r] = ld_volatile_v4(p1 + static_cast(r) * (H / 2)); + } +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if (((pending >> r) & 1u) && v[r].x != kEmpty && v[r].y != kEmpty && v[r].z != kEmpty + && v[r].w != kEmpty) + pending &= ~(1u << r); + if (((pending2 >> r) & 1u) && v2[r].x != kEmpty && v2[r].y != kEmpty && v2[r].z != kEmpty + && v2[r].w != kEmpty) + pending2 &= ~(1u << r); + } + if (++spins > (1LL << 31)) + __trap(); + } +#pragma unroll + for (int r = 0; r < kMaxWorld; ++r) + { + if (r < world) + { + *reinterpret_cast(p0 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t ww[4] = {static_cast(v[r].x), static_cast(v[r].y), + static_cast(v[r].z), static_cast(v[r].w)}; + if (two) + { + *reinterpret_cast(p1 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t const w2[4] = {static_cast(v2[r].x), static_cast(v2[r].y), + static_cast(v2[r].z), static_cast(v2[r].w)}; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + __nv_bfloat162 const u + = __floats2bfloat162_rn(bf16_lo(ww[q]) + bf16_lo(w2[q]), bf16_hi(ww[q]) + bf16_hi(w2[q])); + ww[q] = *reinterpret_cast(&u); + } + } +#pragma unroll + for (int q = 0; q < 4; ++q) + { + s[2 * q] += bf16_lo(ww[q]); + s[2 * q + 1] += bf16_hi(ww[q]); + } + } + } + // Stock rounding points: the rank sum is a BF16 all-reduce output before the residual add. +#pragma unroll + for (int i = 0; i < 8; ++i) + s[i] = __bfloat162float(__float2bfloat16_rn(s[i])); + __nv_bfloat162 const* rr = reinterpret_cast<__nv_bfloat162 const*>(&rraw); + uint32_t upd[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const rf = __bfloat1622float2(rr[q]); + __nv_bfloat162 const u = __floats2bfloat162_rn(s[2 * q] + rf.x, s[2 * q + 1] + rf.y); + upd[q] = *reinterpret_cast(&u); + if constexpr ((OPTS & 16) != 0) + { + float const x0 = s[2 * q] + rf.x, x1 = s[2 * q + 1] + rf.y; + s[2 * q] = x0; + s[2 * q + 1] = x1; + sq += x0 * x0 + x1 * x1; + } + else + { + float2 const uf = __bfloat1622float2(u); + s[2 * q] = uf.x; + s[2 * q + 1] = uf.y; + sq += uf.x * uf.x + uf.y * uf.y; + } + } + *reinterpret_cast(out_residual + static_cast(t) * H + c0) + = make_uint4(upd[0], upd[1], upd[2], upd[3]); + if constexpr ((OPTS & 16) == 0) + { + if constexpr (CAP) + cap_slice[threadIdx.x] = make_uint4(upd[0], upd[1], upd[2], upd[3]); + } + } + if constexpr ((OPTS & 4) != 0) + { + // every CTA (active or not) rotates its call parity once its own poll is done (read only by later grids) + if (threadIdx.x == 0) + flags[blockIdx.x] = parity ^ 1; + } +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + sq += __shfl_xor_sync(0xffffffffu, sq, off); + if ((threadIdx.x & 31) == 0) + warp_sq[threadIdx.x / 32] = sq; + __syncthreads(); + if constexpr ((OPTS & 2) != 0) + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); // peers' xbar initialized (long since) + if (threadIdx.x == 0) + { + float total = 0.f; +#pragma unroll + for (int w = 0; w < THREADS / 32; ++w) + total += warp_sq[w]; + if constexpr ((OPTS & 2) == 0) + for (int c = 0; c < CLUSTER; ++c) + *cluster.map_shared_rank(&cluster_sq[crank], c) = total; + else + { + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + uint32_t const d = static_cast(__cvta_generic_to_shared(&cluster_sq[crank])); + asm volatile("mbarrier.arrive.expect_tx.shared::cta.b64 _, [%0], %1;" ::"r"(b), "r"(CLUSTER * 4) + : "memory"); + for (int c = 0; c < CLUSTER; ++c) + { + uint32_t rd, rb; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rd) : "r"(d), "r"(c)); + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(rb) : "r"(b), "r"(c)); + asm volatile("st.async.shared::cluster.mbarrier::complete_tx::bytes.b32 [%0], %1, [%2];" ::"r"(rd), + "r"(__float_as_uint(total)), "r"(rb) + : "memory"); + } + } + } + if constexpr ((OPTS & 2) == 0) + cluster.sync(); + else + { + // every peer's sum landed (st.async complete_tx on this CTA's barrier); no cluster-wide release-arrive + uint32_t const b = static_cast(__cvta_generic_to_shared(&xbar)); + uint32_t done = 0; + while (!done) + asm volatile( + "{\n .reg .pred p;\n mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 p, [%1], 0;\n" + " selp.u32 %0, 1, 0, p;\n}" + : "=r"(done) + : "r"(b) + : "memory"); + } + if constexpr ((OPTS & 16) == 0) + { + if constexpr (CAP) + { + // The row's chunks are spread over the cluster (CTA c holds chunks [c * per, (c + 1) * per)); every CTA + // forms capture_norm's statistic itself, its 128 threads standing in for capture threads ct and ct + 128. + __shared__ float cap_red[kCapWarps]; + int const per = cols / 8; + asm volatile("barrier.cluster.arrive.release.aligned;" ::: "memory"); + asm volatile("barrier.cluster.wait.acquire.aligned;" ::: "memory"); + float ca = 0.f, cb = 0.f; +#pragma unroll + for (int it = 0; it < kCapIters; ++it) + { + int const ka = it * kCapThreads + static_cast(threadIdx.x); + int const kb = ka + kCapThreads / 2; + ca = cap_chunk_sq(*cluster.map_shared_rank(&cap_slice[ka % per], ka / per), ca); + cb = cap_chunk_sq(*cluster.map_shared_rank(&cap_slice[kb % per], kb / per), cb); + } + // Every peer read is done before any CTA moves on (and later exits). + asm volatile("barrier.cluster.arrive.release.aligned;" ::: "memory"); + asm volatile("barrier.cluster.wait.acquire.aligned;" ::: "memory"); + ca = cap_warp_sum(ca); + cb = cap_warp_sum(cb); + if ((threadIdx.x & 31) == 0) + { + cap_red[threadIdx.x / 32] = ca; + cap_red[threadIdx.x / 32 + kCapWarps / 2] = cb; + } + __syncthreads(); + float tot = 0.f; +#pragma unroll + for (int w = 0; w < kCapWarps; ++w) + tot += cap_red[w]; + float const rs = rsqrtf(tot / H + cap_eps); + if (active) + cap_store(s, rs, cap_w + c0, cap_out + static_cast(t) * cap_stride + c0); + } + } + if (active) + { + float total = 0.f; +#pragma unroll + for (int c = 0; c < CLUSTER; ++c) + total += cluster_sq[c]; + float const inv = rsqrtf(total / static_cast(H) + eps); + if constexpr ((OPTS & 1) == 0) + wraw = *reinterpret_cast(norm_w + c0); + __nv_bfloat162 const* wv = reinterpret_cast<__nv_bfloat162 const*>(&wraw); + uint32_t o[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const wf = __bfloat1622float2(wv[q]); + __nv_bfloat162 const n = __floats2bfloat162_rn(s[2 * q] * inv * wf.x, s[2 * q + 1] * inv * wf.y); + o[q] = *reinterpret_cast(&n); + float2 const nf = __bfloat1622float2(n); + s[2 * q] = nf.x; + s[2 * q + 1] = nf.y; + } + *reinterpret_cast(out_normed + static_cast(t) * H + c0) = make_uint4(o[0], o[1], o[2], o[3]); + } + if (out_q != nullptr) + { + // MXFP8 copy of the normed row for the next GEMM, as the CuTe DSL mxfp8 quantize of `normed` (the tree's + // decode quantize, MX-TAC; see mxfp8_ue8m0): four threads hold one 32-column scale block. + float amax = 0.f; + if (active) + { +#pragma unroll + for (int e = 0; e < 8; ++e) + amax = fmaxf(amax, fabsf(s[e])); + } + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 1)); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 2)); + if (active) + { + uint32_t const code = mxfp8_ue8m0(amax); + float const inv_scale = mxfp8_inv_scale(code); + uint32_t packed[2]; +#pragma unroll + for (int h = 0; h < 2; ++h) + { + packed[h] = mxfp8_e4m3x2(s[4 * h], s[4 * h + 1], inv_scale) + | (mxfp8_e4m3x2(s[4 * h + 2], s[4 * h + 3], inv_scale) << 16); + } + *reinterpret_cast(out_q + static_cast(t) * H + c0) = make_uint2(packed[0], packed[1]); + if ((threadIdx.x & 3) == 0) + { + int const kb = c0 / 32; + out_sf[static_cast(t / 128) * sf_k_tiles * 512 + (kb / 4) * 512 + (t % 32) * 16 + + ((t % 128) / 32) * 4 + kb % 4] + = static_cast(code); + } + } + } + if constexpr ((OPTS & 4) == 0) + { + __syncthreads(); + if (threadIdx.x == 0) + flags[blockIdx.x] = parity ^ 1; + } +#elif defined(__CUDA_ARCH__) + __trap(); // compiled for an architecture without the sm_10x features: never a silent no-op +#endif +} + +// Refuses (raises) to launch a kernel whose loaded binary holds no sm_10x code (PTX and SASS version >= 100): the +// device passes for other architectures compile the kernels to a trap, and a build that left out every sm_10x target +// must not reach the GPU at all. +template +bool checkBinary(Kernel kernel, char const* name) +{ + cudaFuncAttributes attr{}; + TLLM_CUDA_CHECK(cudaFuncGetAttributes(&attr, reinterpret_cast(kernel))); + TLLM_CHECK_WITH_INFO(attr.ptxVersion >= 100 && attr.binaryVersion >= 100, + "%s: the loaded binary holds no sm_10x code (PTX %d, SASS %d): rebuild with an sm_100f / sm_107a target", name, + attr.ptxVersion, attr.binaryVersion); + return true; +} + +template +void launch(MinimaxM3Sm107Tp8McAllreduceNormParams const& p, bool pdl, cudaStream_t stream) +{ + int const M = p.numTokens, H = p.hidden; + TLLM_CHECK_WITH_INFO(H % (CLUSTER * 8) == 0 && H / CLUSTER / 8 <= THREADS, "unsupported hidden size %d", H); + TLLM_CHECK_WITH_INFO(M >= 1 && M <= MAXT, "%d tokens do not fit the %d-token bucket", M, MAXT); + auto kernel = p.capOut != nullptr ? mc_allreduce_norm_kernel + : mc_allreduce_norm_kernel; + static bool const binaryOk + = checkBinary(mc_allreduce_norm_kernel, "mc_allreduce_norm_kernel") + && checkBinary( + mc_allreduce_norm_kernel, "mc_allreduce_norm_kernel (capture)"); + (void) binaryOk; + // Both instantiations (with and without the capture epilogue) get the same attributes, whichever runs first. + static bool const configured = []() + { + auto setup = [](auto k) + { + bool ok = CLUSTER <= 8 + || cudaFuncSetAttribute(k, cudaFuncAttributeNonPortableClusterSizeAllowed, 1) == cudaSuccess; + if ((OPTS & 8) != 0) + ok = ok + && cudaFuncSetAttribute( + k, cudaFuncAttributePreferredSharedMemoryCarveout, cudaSharedmemCarveoutMaxShared) + == cudaSuccess; + return ok; + }; + bool const ok = setup(mc_allreduce_norm_kernel) + && setup(mc_allreduce_norm_kernel); + if (!ok) + cudaGetLastError(); + return ok; + }(); + TLLM_CHECK_WITH_INFO(configured, "cluster size not supported"); + FinalizeArgs const fin{p.fc2, p.perm, p.scales, p.scaleFp32 ? 1 : 0, p.topk}; + cudaLaunchConfig_t cfg = {}; + // The grid is sized to the call's rows: min(M, MAXT) clusters. Each (row, slice) slot keeps its own parity flag, + // read by the producers before they push and rotated only by the slot's CTA after it polled: a slot that a call + // does not use keeps its parity, and every rank runs the same calls, so the parities stay consistent when M + // changes between calls. + cfg.gridDim = dim3((M < MAXT ? M : MAXT) * CLUSTER); + cfg.blockDim = dim3(THREADS); + cfg.stream = stream; + if (p.capOut != nullptr) + { + TLLM_CHECK_WITH_INFO((OPTS & 16) == 0 && H == kCapHidden && 2 * THREADS == kCapThreads, + "the capture epilogue takes H %d on the 128-thread cluster kernel, not the drafter boundary", kCapHidden); + cfg.dynamicSmemBytes = static_cast(H / CLUSTER / 8) * sizeof(uint4); + } + cudaLaunchAttribute attr[2]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + attr[1].id = cudaLaunchAttributeClusterDimension; + attr[1].val.clusterDim.x = CLUSTER; + attr[1].val.clusterDim.y = 1; + attr[1].val.clusterDim.z = 1; + cfg.attrs = attr; + cfg.numAttrs = 2; + cudaError_t const err = cudaLaunchKernelEx(&cfg, kernel, p.x, p.x2, fin, M, H, p.residual, p.normWeight, p.eps, + p.wsUc, p.wsMc, p.flags, p.rank, p.world, p.outNormed, p.outResidual, p.outQuant, p.outSf, (H / 32 + 3) / 4, + p.prepushed, p.capOut, p.capStride, p.capWeight, p.capEps); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +// ws_uc / ws_mc are this rank's unicast and multicast views of ONE exchange buffer (they alias): no __restrict__. +// prepushed >= 2 contract: the producer (m3_mx_gemv's o_proj push, the m3_moe engine's finalize push) passes its own +// grid dependency, pushes its rows into the exchange, and only then releases this grid (griddepcontrol +// .launch_dependents); this kernel therefore skips griddepcontrol.wait and reads only the pushed (Lamport-polled) +// words, its parity flags and the residual / norm weight, which were written two or more grids earlier. +// +// The TP8 row kernel: one CTA of H / 8 = 768 threads per token row, used for every call but the MoE finalize. Its +// residual add + RMSNorm is TRT-LLM's rms_norm_kernel (customAllReduceKernels.cu), the stock fused norm after the +// all-reduce on this target, operation for operation at H 6144: thread j owns columns [8j, 8j + 8), its sum of squares +// is acc += v * v over them in order, then the warp butterfly 16..1, the 24 warp sums through shared memory into a +// second 32-lane butterfly (lanes >= 24 zero), rsqrtf(acc / H + eps) and (v * denom) * w. With the BF16 rank sum +// before the residual add, the normed rows, the updated residual and the MXFP8 copy are bitwise the stock ones. The +// exchange slots and the per-(row, slice) parity flags keep the cluster kernel's layout (slice = 96 threads), so +// producers that push (the o_proj GEMV) are unchanged. +constexpr int kSlices = 8; +constexpr int kSliceThreads = 96; +constexpr int kRowThreads = 768; + +// grid = M CTAs (one per row), THREADS = H / 8 threads. WORLD ranks (compile time). DRAFTER (opts bit 16): the Eagle3 +// drafter boundary, whose stock is a plain BF16 all-reduce then FlashInfer fused_add_rmsnorm: the RMS statistic and +// the normalization take the unrounded FP32 sum bf16(rank sum) + residual (only the stored residual is rounded). +// ws_uc / ws_mc alias (one exchange buffer): no __restrict__. +template +__global__ void __launch_bounds__(THREADS, 1) mc_allreduce_norm_row_kernel(__nv_bfloat16 const* __restrict__ x, + __nv_bfloat16 const* __restrict__ x2, int H, __nv_bfloat16 const* __restrict__ residual, + __nv_bfloat16 const* __restrict__ norm_w, float eps, int* ws_uc, int* ws_mc, int* __restrict__ flags, int rank, + int maxt, __nv_bfloat16* __restrict__ out_normed, __nv_bfloat16* __restrict__ out_residual, + uint8_t* __restrict__ out_q, uint8_t* __restrict__ out_sf, int sf_k_tiles, int prepushed, + __nv_bfloat16* __restrict__ cap_out, int64_t cap_stride, __nv_bfloat16 const* __restrict__ cap_w, float cap_eps) +{ +#if MINIMAX_M3_KERNELS_ENABLED + __shared__ float warp_sq[THREADS / 32]; + __shared__ float total_sh; + extern __shared__ __align__(16) uint4 cap_row[]; // [THREADS] the updated row's chunks (cap_out only) + int const t = blockIdx.x; + int const j = threadIdx.x; + int const c0 = j * 8; + int const slice = j / kSliceThreads; + int64_t const half_words = static_cast(maxt) * WORLD * (H / 2); + if (prepushed < 2) + asm volatile("griddepcontrol.wait;" ::: "memory"); + int const parity = *reinterpret_cast(flags + t * kSlices + slice) & 1; + uint4 const rraw = *reinterpret_cast(residual + static_cast(t) * H + c0); + uint4 const wraw = *reinterpret_cast(norm_w + c0); + if (!prepushed) + { + uint4 raw = *reinterpret_cast(x + static_cast(t) * H + c0); + if (x2 != nullptr) + { + uint4 const raw2 = *reinterpret_cast(x2 + static_cast(t) * H + c0); + __nv_bfloat162* a = reinterpret_cast<__nv_bfloat162*>(&raw); + __nv_bfloat162 const* b = reinterpret_cast<__nv_bfloat162 const*>(&raw2); +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const fa = __bfloat1622float2(a[q]); + float2 const fb = __bfloat1622float2(b[q]); + a[q] = __floats2bfloat162_rn(fa.x + fb.x, fa.y + fb.y); + } + } + int64_t const slot = parity * half_words + (static_cast(t) * WORLD + rank) * (H / 2) + c0 / 2; + *reinterpret_cast(ws_mc + slot) + = make_int4(static_cast(canon_bf16x2(raw.x)), static_cast(canon_bf16x2(raw.y)), + static_cast(canon_bf16x2(raw.z)), static_cast(canon_bf16x2(raw.w))); + } + asm volatile("griddepcontrol.launch_dependents;"); + float s[8] = {}; + float sq = 0.f; + { + int* const p0 = ws_uc + parity * half_words + static_cast(t) * WORLD * (H / 2) + c0 / 2; + int* const p1 = p0 + 2 * half_words; // set B + bool const two = prepushed == 3; + int4 v[WORLD]; + int4 v2[WORLD]; + uint32_t pending = (1u << WORLD) - 1u; + uint32_t pending2 = two ? pending : 0u; + long long spins = 0; + while ((pending | pending2) != 0u) + { +#pragma unroll + for (int r = 0; r < WORLD; ++r) + { + if ((pending >> r) & 1u) + v[r] = ld_volatile_v4(p0 + static_cast(r) * (H / 2)); + if ((pending2 >> r) & 1u) + v2[r] = ld_volatile_v4(p1 + static_cast(r) * (H / 2)); + } +#pragma unroll + for (int r = 0; r < WORLD; ++r) + { + if (((pending >> r) & 1u) && v[r].x != kEmpty && v[r].y != kEmpty && v[r].z != kEmpty + && v[r].w != kEmpty) + pending &= ~(1u << r); + if (((pending2 >> r) & 1u) && v2[r].x != kEmpty && v2[r].y != kEmpty && v2[r].z != kEmpty + && v2[r].w != kEmpty) + pending2 &= ~(1u << r); + } + if (++spins > (1LL << 31)) + __trap(); + } +#pragma unroll + for (int r = 0; r < WORLD; ++r) + { + *reinterpret_cast(p0 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t ww[4] = {static_cast(v[r].x), static_cast(v[r].y), + static_cast(v[r].z), static_cast(v[r].w)}; + if (two) + { + *reinterpret_cast(p1 + static_cast(r) * (H / 2)) + = make_int4(kEmpty, kEmpty, kEmpty, kEmpty); + uint32_t const w2[4] = {static_cast(v2[r].x), static_cast(v2[r].y), + static_cast(v2[r].z), static_cast(v2[r].w)}; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + __nv_bfloat162 const u + = __floats2bfloat162_rn(bf16_lo(ww[q]) + bf16_lo(w2[q]), bf16_hi(ww[q]) + bf16_hi(w2[q])); + ww[q] = *reinterpret_cast(&u); + } + } +#pragma unroll + for (int q = 0; q < 4; ++q) + { + s[2 * q] += bf16_lo(ww[q]); + s[2 * q + 1] += bf16_hi(ww[q]); + } + } + // Stock rounding points: the rank sum is a BF16 all-reduce output before the residual add. +#pragma unroll + for (int i = 0; i < 8; ++i) + s[i] = __bfloat162float(__float2bfloat16_rn(s[i])); + __nv_bfloat162 const* rr = reinterpret_cast<__nv_bfloat162 const*>(&rraw); + uint32_t upd[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const rf = __bfloat1622float2(rr[q]); + __nv_bfloat162 const u = __floats2bfloat162_rn(s[2 * q] + rf.x, s[2 * q + 1] + rf.y); + upd[q] = *reinterpret_cast(&u); + if constexpr (DRAFTER) + { + s[2 * q] += rf.x; + s[2 * q + 1] += rf.y; + } + else + { + float2 const uf = __bfloat1622float2(u); + s[2 * q] = uf.x; + s[2 * q + 1] = uf.y; + } + } +#pragma unroll + for (int e = 0; e < 8; ++e) + sq += s[e] * s[e]; + *reinterpret_cast(out_residual + static_cast(t) * H + c0) + = make_uint4(upd[0], upd[1], upd[2], upd[3]); + if constexpr (!DRAFTER) + { + if constexpr (CAP) + cap_row[j] = make_uint4(upd[0], upd[1], upd[2], upd[3]); + } + } + // TRT-LLM reduce_fusion::block_reduce_sum. + sq += __shfl_xor_sync(~0, sq, 16); + sq += __shfl_xor_sync(~0, sq, 8); + sq += __shfl_xor_sync(~0, sq, 4); + sq += __shfl_xor_sync(~0, sq, 2); + sq += __shfl_xor_sync(~0, sq, 1); + if ((j & 31) == 0) + warp_sq[j / 32] = sq; + __syncthreads(); + { + float val = (j & 31) < THREADS / 32 ? warp_sq[j & 31] : 0.f; + val += __shfl_xor_sync(~0, val, 16); + val += __shfl_xor_sync(~0, val, 8); + val += __shfl_xor_sync(~0, val, 4); + val += __shfl_xor_sync(~0, val, 2); + val += __shfl_xor_sync(~0, val, 1); + if (j == 0) + total_sh = val; + } + // The parity of this row's slices rotates once every thread has read the exchange (later grids only). + if (j % kSliceThreads == 0) + flags[t * kSlices + slice] = parity ^ 1; + __syncthreads(); + if constexpr (!DRAFTER) + { + if constexpr (CAP) + { + // The whole row is in this CTA: threads 0..255 run capture_norm's chains over the staged chunks. + __shared__ float cap_red[kCapWarps]; + if (j < kCapThreads) + { + float css = 0.f; +#pragma unroll + for (int it = 0; it < kCapIters; ++it) + css = cap_chunk_sq(cap_row[it * kCapThreads + j], css); + css = cap_warp_sum(css); + if ((j & 31) == 0) + cap_red[j / 32] = css; + } + __syncthreads(); + float tot = 0.f; +#pragma unroll + for (int w = 0; w < kCapWarps; ++w) + tot += cap_red[w]; + float const rs = rsqrtf(tot / H + cap_eps); + cap_store(s, rs, cap_w + c0, cap_out + static_cast(t) * cap_stride + c0); + } + } + float const inv = rsqrtf(total_sh / H + eps); + __nv_bfloat162 const* wv = reinterpret_cast<__nv_bfloat162 const*>(&wraw); + uint32_t o[4]; +#pragma unroll + for (int q = 0; q < 4; ++q) + { + float2 const wf = __bfloat1622float2(wv[q]); + __nv_bfloat162 const n = __floats2bfloat162_rn(s[2 * q] * inv * wf.x, s[2 * q + 1] * inv * wf.y); + o[q] = *reinterpret_cast(&n); + float2 const nf = __bfloat1622float2(n); + s[2 * q] = nf.x; + s[2 * q + 1] = nf.y; + } + *reinterpret_cast(out_normed + static_cast(t) * H + c0) = make_uint4(o[0], o[1], o[2], o[3]); + if (out_q != nullptr) + { + float amax = 0.f; +#pragma unroll + for (int e = 0; e < 8; ++e) + amax = fmaxf(amax, fabsf(s[e])); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 1)); + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, 2)); + uint32_t const code = mxfp8_ue8m0(amax); + float const inv_scale = mxfp8_inv_scale(code); + uint32_t packed[2]; +#pragma unroll + for (int h = 0; h < 2; ++h) + { + packed[h] = mxfp8_e4m3x2(s[4 * h], s[4 * h + 1], inv_scale) + | (mxfp8_e4m3x2(s[4 * h + 2], s[4 * h + 3], inv_scale) << 16); + } + *reinterpret_cast(out_q + static_cast(t) * H + c0) = make_uint2(packed[0], packed[1]); + if ((j & 3) == 0) + { + int const kb = c0 / 32; + out_sf[static_cast(t / 128) * sf_k_tiles * 512 + (kb / 4) * 512 + (t % 32) * 16 + + ((t % 128) / 32) * 4 + kb % 4] + = static_cast(code); + } + } +#elif defined(__CUDA_ARCH__) + __trap(); // compiled for an architecture without the sm_10x features: never a silent no-op +#endif +} + +void launchRow( + MinimaxM3Sm107Tp8McAllreduceNormParams const& p, int32_t maxTokens, bool drafter, bool pdl, cudaStream_t stream) +{ + TLLM_CHECK_WITH_INFO(p.hidden == kRowThreads * 8 && p.world == 2, "the row kernel takes H 6144 at 2 ranks"); + static bool const binaryOk + = checkBinary(mc_allreduce_norm_row_kernel<2, kRowThreads, false, false>, "mc_allreduce_norm_row_kernel") + && checkBinary( + mc_allreduce_norm_row_kernel<2, kRowThreads, true, false>, "mc_allreduce_norm_row_kernel (drafter)") + && checkBinary( + mc_allreduce_norm_row_kernel<2, kRowThreads, false, true>, "mc_allreduce_norm_row_kernel (capture)"); + (void) binaryOk; + TLLM_CHECK_WITH_INFO(p.numTokens >= 1 && p.numTokens <= maxTokens, "%d tokens do not fit the %d-token bucket", + p.numTokens, maxTokens); + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(p.numTokens); + cfg.blockDim = dim3(kRowThreads); + cfg.stream = stream; + if (p.capOut != nullptr) + { + TLLM_CHECK_WITH_INFO( + !drafter && p.hidden == kCapHidden, "the capture epilogue takes H %d, not the drafter", kCapHidden); + cfg.dynamicSmemBytes = static_cast(kRowThreads) * sizeof(uint4); + } + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + // The capture epilogue never runs on the drafter boundary (checked above), so there are three instantiations. + auto kernel = drafter ? mc_allreduce_norm_row_kernel<2, kRowThreads, true, false> + : p.capOut != nullptr ? mc_allreduce_norm_row_kernel<2, kRowThreads, false, true> + : mc_allreduce_norm_row_kernel<2, kRowThreads, false, false>; + cudaError_t const err = cudaLaunchKernelEx(&cfg, kernel, p.x, p.x2, p.hidden, p.residual, p.normWeight, p.eps, + p.wsUc, p.wsMc, p.flags, p.rank, maxTokens, p.outNormed, p.outResidual, p.outQuant, p.outSf, + (p.hidden / 32 + 3) / 4, p.prepushed, p.capOut, p.capStride, p.capWeight, p.capEps); + if (err != cudaSuccess) + { + cudaGetLastError(); + TLLM_CUDA_CHECK(err); + } +} + +constexpr int kCluster = 8; +constexpr int kThreads = 128; + +} // namespace + +bool isMinimaxM3Sm107Tp8McAllreduceNormSupported(int32_t maxTokens, int32_t opts) +{ + return (maxTokens == 8 && (opts == 7 || opts == 15 || opts == 23 || opts == 31)) || (maxTokens == 16 && opts == 15) + || (maxTokens == 48 && opts == 15); +} + +void invokeMinimaxM3Sm107Tp8McAllreduceNorm(MinimaxM3Sm107Tp8McAllreduceNormParams const& params, int32_t maxTokens, + int32_t opts, bool pdl, cudaStream_t stream) +{ + if (params.fc2 == nullptr && params.world == 2 && params.hidden == kRowThreads * 8) + { + launchRow(params, maxTokens, (opts & 16) != 0, pdl, stream); + return; + } + if (maxTokens == 8 && opts == 7) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 15) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 23) + { + launch(params, pdl, stream); + } + else if (maxTokens == 8 && opts == 31) + { + launch(params, pdl, stream); + } + else if (maxTokens == 16 && opts == 15) + { + launch(params, pdl, stream); + } + else if (maxTokens == 48 && opts == 15) + { + launch(params, pdl, stream); + } + else + { + TLLM_THROW("unsupported (bucket %d, opts %d)", maxTokens, opts); + } +} + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.h new file mode 100644 index 000000000000..4b36daf66813 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.h @@ -0,0 +1,83 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ + +//! Operands of the MiniMax-M3 multicast all-reduce + residual + RMSNorm kernel (decode, at most 48 tokens). +//! +//! Per rank the kernel forms the partial `bf16(x + x2)` (or, in finalize mode, +//! `bf16(bf16(sum_k scale * fc2[perm]) + x2)`, the routed sum rounded before the shared expert is added, as stock), +//! pushes it through the multicast mapping `wsMc` into every rank's exchange, polls its own unicast view `wsUc`, +//! sums the ranks in FP32 in rank order, adds the residual (BF16 round) and applies RMSNorm. With `outQuant` it +//! also writes the MXFP8 copy of the normed rows (E4M3 [numTokens, hidden] and swizzled UE8M0 scales). +struct MinimaxM3Sm107Tp8McAllreduceNormParams +{ + __nv_bfloat16 const* x{nullptr}; //!< [numTokens, hidden] partial; in finalize mode the FC2 rows [rows, hidden]. + __nv_bfloat16 const* x2{nullptr}; //!< Optional second addend [numTokens, hidden]. + __nv_bfloat16 const* fc2{nullptr}; //!< Finalize mode: the permuted FC2 output (same buffer as x); null otherwise. + int32_t const* perm{nullptr}; //!< Finalize mode: [numTokens, topk] expanded index -> permuted row, -1 dropped. + void const* scales{nullptr}; //!< Finalize mode: [numTokens, topk] routing weights (fp32 or bf16). + bool scaleFp32{false}; + int32_t topk{0}; + int32_t numTokens{0}; + int32_t hidden{0}; + __nv_bfloat16 const* residual{nullptr}; //!< [numTokens, hidden] + __nv_bfloat16 const* normWeight{nullptr}; //!< [hidden] + float eps{0.F}; + int32_t* wsUc{nullptr}; //!< This rank's unicast view of the exchange (int32 words, re-armed after reading). + int32_t* wsMc{nullptr}; //!< The multicast mapping of the exchange. + int32_t* flags{nullptr}; //!< Per-CTA call-parity counters, [maxTokens * 8]. + int32_t rank{0}; + int32_t world{0}; + __nv_bfloat16* outNormed{nullptr}; + __nv_bfloat16* outResidual{nullptr}; + uint8_t* outQuant{nullptr}; //!< Optional E4M3 [numTokens, hidden]. + uint8_t* outSf{nullptr}; //!< Optional UE8M0 scales, 128x4-swizzled. + //! 0: push x (+ x2) here; 2: a producer already pushed x and launched this grid after its pushes; 3: as 2, with x + //! in exchange set A and x2 in set B. + int32_t prepushed{0}; + //! Optional Eagle3 capture of a capture layer's boundary: capOut[t * capStride + c] = the m3_spec_tail + //! capture_norm of the updated residual row with weight capWeight [hidden] and capEps, bit for bit (hidden 6144, + //! not the drafter boundary). + __nv_bfloat16* capOut{nullptr}; + int64_t capStride{0}; + __nv_bfloat16 const* capWeight{nullptr}; + float capEps{0.F}; +}; + +//! The (bucket, epilogue options) pairs this launcher is built for: (8, 7), (8, 15), (16, 15) and (48, 15). +bool isMinimaxM3Sm107Tp8McAllreduceNormSupported(int32_t maxTokens, int32_t opts); + +//! Launch with 8-CTA clusters of 128 threads, one cluster per token. `maxTokens` selects the exchange bucket; +//! `opts` the epilogue variant (every variant is bit-identical). +void invokeMinimaxM3Sm107Tp8McAllreduceNorm(MinimaxM3Sm107Tp8McAllreduceNormParams const& params, int32_t maxTokens, + int32_t opts, bool pdl, cudaStream_t stream); + +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.cu new file mode 100644 index 000000000000..3062080fac2c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.cu @@ -0,0 +1,340 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.h" + +#include +#include +#include + +// MiniMax-M3 NVFP4 QKV + indexer norm/RoPE/KV-insert producer with an early PDL trigger, at the SM107 TP8 heads. +// +// The SM107 TP8 target's copy of the SM107 TP2 producerPw2.cu (itself a copy of minimaxM3/producerPw.cu, which the +// sm_103 targets keep), for its pre-wait step inputs. Same math and +// outputs as minimaxM3Nvfp4QKVIndexerNormRopeKVInsertKernel (fusedQKNormRopeKernel.cu, built with --use_fast_math like +// the rest of the library), specialized to 8 query, 1 KV and 1 index head per rank (one index-K head), head_dim 128, +// rotary_dim 64, page 128, NVFP4 main KV and FP8 index-K. One warp per (token, packed head); K and V rows go to their +// head's block of the HND pools (the native kernel's head strides). +// +// The decode producer is latency bound: after griddepcontrol.wait the native kernel runs a chain of dependent global +// loads (packed row -> norm weight -> position id -> RoPE row -> cache slot -> quant scale). This kernel reads the +// constant norm weights and KV quant scale before griddepcontrol.wait and issues the packed row, position id and cache +// slot loads together after it, followed by the RoPE row; with `prewaitStepInputs` the position id, the slot and the +// RoPE row are read before the wait too, leaving only the packed row after it. With `early` it also calls +// griddepcontrol.launch_dependents on entry: consumers read its outputs only after their own griddepcontrol.wait, which +// waits for this grid to complete, so the early trigger only lets the next kernel launch and run its pre-wait prologue +// while this one runs. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +namespace producerPw8 +{ +namespace +{ + +constexpr int kHeadDim = 128; +constexpr int kRotaryDim = 64; +constexpr int kPageSize = 128; +constexpr int kElems = kHeadDim / 32; +constexpr int kBlock = 256; + +__device__ __forceinline__ float rcp_approx_ftz(float a) +{ + float b; + asm volatile("rcp.approx.ftz.f32 %0, %1;\n" : "=f"(b) : "f"(a)); + return b; +} + +__device__ __forceinline__ uint32_t sf_group_mask4() +{ + uint32_t laneId; + asm("mov.u32 %0, %%laneid;" : "=r"(laneId)); + return 0xFu << (laneId & ~3u); +} + +__device__ __forceinline__ uint32_t fp32x8_to_e2m1(float (&a)[8]) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + uint32_t val; + asm volatile( + "{\n" + ".reg .b8 byte0;\n" + ".reg .b8 byte1;\n" + ".reg .b8 byte2;\n" + ".reg .b8 byte3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte0, %2, %1;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte1, %4, %3;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte2, %6, %5;\n" + "cvt.rn.satfinite.e2m1x2.f32 byte3, %8, %7;\n" + "mov.b32 %0, {byte0, byte1, byte2, byte3};\n" + "}" + : "=r"(val) + : "f"(a[0]), "f"(a[1]), "f"(a[2]), "f"(a[3]), "f"(a[4]), "f"(a[5]), "f"(a[6]), "f"(a[7])); + return val; +#else + return 0; +#endif +} + +__device__ __forceinline__ void store_fp8x4(__nv_fp8_e4m3* out, int64_t off, float const (&e)[kElems]) +{ + __nv_fp8x2_e4m3 const low(make_float2(e[0], e[1])); + __nv_fp8x2_e4m3 const high(make_float2(e[2], e[3])); + *reinterpret_cast(out + off) = static_cast(low.__x) | (static_cast(high.__x) << 16); +} + +__device__ __forceinline__ uint16_t quantize_fp4x4(float const (&e)[4], float secondLevelScale, uint8_t* scaleOut) +{ + __nv_bfloat16 rounded[4]; + float localMax = 0.0F; +#pragma unroll + for (int i = 0; i < 4; ++i) + { + rounded[i] = __float2bfloat16_rn(e[i]); + localMax = fmaxf(localMax, fabsf(__bfloat162float(rounded[i]))); + } + uint32_t const groupMask = sf_group_mask4(); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 1)); + localMax = fmaxf(localMax, __shfl_xor_sync(groupMask, localMax, 2)); + float const scaleValue = secondLevelScale * (localMax * rcp_approx_ftz(6.0F)); + __nv_fp8_e4m3 const encodedScale(scaleValue); + if (scaleOut != nullptr) + *scaleOut = encodedScale.__x; + float const narrowedScale = static_cast(encodedScale); + float const outputScale + = localMax != 0.0F ? rcp_approx_ftz(narrowedScale * rcp_approx_ftz(secondLevelScale)) : 0.0F; + float q[8] = {}; +#pragma unroll + for (int i = 0; i < 4; ++i) + q[i] = __bfloat162float(rounded[i]) * outputScale; + return static_cast(fp32x8_to_e2m1(q)); +} + +template +__global__ void __launch_bounds__(kBlock) m3_producer_kernel(Params p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + constexpr int kTotalHeads = kHeadsQ + 2 * kHeadsKV + kHeadsIndex + 1; +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if constexpr (kEarly) + { + asm volatile("griddepcontrol.launch_dependents;"); + } +#endif + int const warpId = threadIdx.x / 32; + int const laneId = threadIdx.x % 32; + int const globalWarp = blockIdx.x * (kBlock / 32) + warpId; + int const tokenIdx = globalWarp / kTotalHeads; + int const localHead = globalWarp % kTotalHeads; + if (tokenIdx >= p.numTokens) + return; + + constexpr int kBegin = kHeadsQ; + constexpr int vBegin = kBegin + kHeadsKV; + constexpr int indexQBegin = vBegin + kHeadsKV; + constexpr int indexKHead = indexQBegin + kHeadsIndex; + bool const isQ = localHead < kBegin; + bool const isK = localHead >= kBegin && localHead < vBegin; + bool const isV = localHead >= vBegin && localHead < indexQBegin; + bool const isIndexQ = localHead >= indexQBegin && localHead < indexKHead; + bool const isIndexK = localHead == indexKHead; + + // Constant data read before the wait: norm weights and the KV quant scale. + float w[kElems] = {}; + float quantScale = 0.0F; + __nv_bfloat16 const* normWeight + = isQ ? p.qWeight : (isK ? p.kWeight : (isIndexQ ? p.indexQWeight : p.indexKWeight)); + if (!isV) + { + uint2 const wv = *reinterpret_cast(normWeight + laneId * kElems); + float2 const w01 = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&wv.x)); + float2 const w23 = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162 const*>(&wv.y)); + w[0] = w01.x; + w[1] = w01.y; + w[2] = w23.x; + w[3] = w23.y; + } + if (isK || isV) + { + quantScale = p.kvQuantScale[isV ? 2 : 1]; + } + + int64_t const inputOffset = (static_cast(tokenIdx) * kTotalHeads + localHead) * kHeadDim + laneId * kElems; + bool const rotates = !isV && laneId * kElems < kRotaryDim; + bool const slotted = isK || isV || isIndexK; + int const coefficient = (laneId * kElems) % (kRotaryDim / 2); + // kPrewait: the position id, the cache slot and the RoPE row are read before the wait. They are per-step metadata + // written before the replay (host-to-device) or by plain-launched kernels earlier in the step (position_ids by the + // step's input preprocessing, the slots by msa_kv_update or its torch fallback), so they are final before this + // grid can start whatever its predecessors trigger; the RoPE table is constant. + int positionId = 0; + int slot = 0; + float4 c = make_float4(0.0F, 0.0F, 0.0F, 0.0F); + float4 s = c; + auto const loadStepInputs = [&]() + { + positionId = rotates ? p.positionIds[tokenIdx] : 0; + slot = slotted ? p.outCacheLoc[tokenIdx] : 0; + if (rotates) + { + int64_t const ropeRow = static_cast(positionId) * kRotaryDim; + c = *reinterpret_cast(p.rotaryCosSin + ropeRow + coefficient); + s = *reinterpret_cast(p.rotaryCosSin + ropeRow + kRotaryDim / 2 + coefficient); + } + }; + if constexpr (kPrewait) + { + loadStepInputs(); + } +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + // The packed row comes from the predecessor grid; without kPrewait the step inputs are issued with it, so the + // native kernel's serial chain of dependent loads becomes two round trips. + uint2 const packed = *reinterpret_cast(p.packed + inputOffset); + if constexpr (!kPrewait) + { + loadStepInputs(); + } + float cosine[kElems] = {c.x, c.y, c.z, c.w}; + float sine[kElems] = {s.x, s.y, s.z, s.w}; + + float e[kElems]; + float sumSquares = 0.0F; +#pragma unroll + for (int pair = 0; pair < 2; ++pair) + { + float2 const v = __bfloat1622float2( + *reinterpret_cast<__nv_bfloat162 const*>(reinterpret_cast(&packed) + pair)); + e[2 * pair] = v.x; + e[2 * pair + 1] = v.y; + if (!isV) + sumSquares += v.x * v.x + v.y * v.y; + } + + if (!isV) + { +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) + sumSquares += __shfl_xor_sync(0xffffffff, sumSquares, mask, 32); + float const rmsReciprocal = rsqrtf(sumSquares / static_cast(kHeadDim) + p.eps); +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + e[i] *= rmsReciprocal * (1.0F + w[i]); + } + constexpr int kPairOffset = (kRotaryDim / 2) / kElems; +#pragma unroll + for (int i = 0; i < kElems; ++i) + { + int const dim = laneId * kElems + i; + float paired = __shfl_xor_sync(0xffffffff, e[i], kPairOffset); + if (dim < kRotaryDim) + { + if (dim < kRotaryDim / 2) + paired = -paired; + // The native kernel's contraction of e * cos + paired * sin; spelled out so this build matches it. + e[i] = __fmaf_rn(paired, sine[i], __fmul_rn(e[i], cosine[i])); + } + } + } + + if (isQ) + { + store_fp8x4(p.qOut, (static_cast(tokenIdx) * kHeadsQ + localHead) * kHeadDim + laneId * kElems, e); + return; + } + if (isIndexQ) + { + store_fp8x4(p.indexQOut, + (static_cast(tokenIdx) * kHeadsIndex + localHead - indexQBegin) * kHeadDim + laneId * kElems, e); + return; + } + + if (slot < 0) + return; + int const page = slot / kPageSize; + if (page >= p.numPages) + return; + int const withinPage = slot & (kPageSize - 1); + if (isIndexK) + { + store_fp8x4(p.indexK, + static_cast(page) * p.indexPageStride + static_cast(withinPage) * p.indexTokenStride + + laneId * kElems, + e); + return; + } + + int const head = isK ? localHead - kBegin : localHead - vBegin; + int const plane = isV ? 1 : 0; + int const scaleColumn = laneId / 4; + int64_t const scaleWithinPage = isV + ? static_cast(withinPage / 4) * 32 + static_cast(scaleColumn) * 4 + withinPage % 4 + : static_cast(withinPage) * 8 + scaleColumn; + int64_t const scaleOffset = static_cast(page) * p.scalePageStride + + static_cast(plane) * p.scalePlaneStride + static_cast(head) * p.scaleHeadStride + + scaleWithinPage; + uint8_t* scaleOut = laneId % 4 == 0 ? p.kvScale + scaleOffset : nullptr; + uint16_t const packedFp4 = quantize_fp4x4(e, quantScale, scaleOut); + int64_t const dataOffset = static_cast(page) * p.dataPageStride + + static_cast(plane) * p.dataPlaneStride + static_cast(head) * p.dataHeadStride + + static_cast(withinPage) * p.dataTokenStride + laneId * 2; + *reinterpret_cast(p.kvData + dataOffset) = packedFp4; +#endif +} + +template +cudaError_t launch(Params const& p, int numTokens, bool pdl, cudaStream_t stream) +{ + constexpr int kTotalHeads = kHeadsQ + 2 * kHeadsKV + kHeadsIndex + 1; + int const warps = numTokens * kTotalHeads; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3((warps + kBlock / 32 - 1) / (kBlock / 32)); + cfg.blockDim = dim3(kBlock); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return cudaLaunchKernelEx(&cfg, m3_producer_kernel, p); +} + +} // namespace + +cudaError_t invokeProducerPw8( + Params const& p, int numTokens, bool pdl, bool early, bool prewaitStepInputs, cudaStream_t stream) +{ + if (numTokens <= 0) + return cudaSuccess; + if (prewaitStepInputs) + { + return early ? launch(p, numTokens, pdl, stream) : launch(p, numTokens, pdl, stream); + } + return early ? launch(p, numTokens, pdl, stream) : launch(p, numTokens, pdl, stream); +} + +} // namespace producerPw8 +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.h new file mode 100644 index 000000000000..5b269b94587c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.h @@ -0,0 +1,76 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +//! The MiniMax-M3 NVFP4 QKV + indexer RMSNorm / RoPE / KV-insert producer at the SM107 TP8 heads (8 query, 1 KV and +//! 1 index head per rank) with its constant loads before the PDL grid-dependency wait and an optional early dependents +//! trigger. Same outputs, bit for bit, as minimaxM3Nvfp4QKVIndexerNormRopeKVInsert. See producerPw8.cu. +namespace producerPw8 +{ + +struct Params +{ + __nv_bfloat16 const* packed; + __nv_fp8_e4m3* qOut; + __nv_fp8_e4m3* indexQOut; + uint8_t* kvData; + uint8_t* kvScale; + __nv_fp8_e4m3* indexK; + int const* outCacheLoc; + float const* kvQuantScale; + int64_t dataPageStride, dataPlaneStride, dataHeadStride, dataTokenStride; + int64_t scalePageStride, scalePlaneStride, scaleHeadStride; + int64_t indexPageStride, indexTokenStride, numPages; + int numTokens; + float eps; + __nv_bfloat16 const* qWeight; + __nv_bfloat16 const* kWeight; + __nv_bfloat16 const* indexQWeight; + __nv_bfloat16 const* indexKWeight; + float const* rotaryCosSin; + int const* positionIds; +}; + +//! The heads per rank this producer is built for. +constexpr int kHeadsQ = 8; +constexpr int kHeadsKV = 1; +constexpr int kHeadsIndex = 1; + +//! Launch over `numTokens` tokens, one warp per (token, packed head). `prewaitStepInputs` reads the position ids, the +//! cache slots and the RoPE rows before the grid-dependency wait (they must not be written by a PDL predecessor). +cudaError_t invokeProducerPw8( + Params const& p, int numTokens, bool pdl, bool early, bool prewaitStepInputs, cudaStream_t stream); + +} // namespace producerPw8 +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.cu new file mode 100644 index 000000000000..ff62f0d15253 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.cu @@ -0,0 +1,946 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.h" + +#include "tensorrt_llm/common/assert.h" + +#include +#include + +// Fused sampled verification for the MiniMax-M3 Eagle3 target head (TP-sharded vocabulary, decode rows <= 48). +// +// sm_107 (Rubin) TP8-owned copy of kernels/minimaxM3Sm107Tp2/sampledAccept.cu (bit for bit the same kernel: the +// seam for the TP8 SA-1R round-0 decision), itself the TP2 copy of kernels/minimaxM3/sampledAccept.cu: identical +// algorithm and arithmetic; the vocab shard limit is raised to the TP2 shard (200064 / 2 = 100032 -> kMaxVs 100352), so +// a CTA stages up to kMaxVs / kSplits BF16 logits (25 KB at 8 splits, 50 KB at 4) and the launch opts in to that +// dynamic shared memory. +// +// Replaces, per decode step, the full-vocabulary all-gather of the target logits, the fused sampling kernel and the +// strict-acceptance ops for rows sampled with temperature and top-p (the AgentX contract). Rows with top-k or min-p +// active must stay on the generic path; the caller decides. +// +// Sampling is top-p rejection sampling, distributed over the TP ranks. With w = exp(logit / T - max) over the FULL +// vocabulary and Z = sum(w), a token is in the nucleus iff the mass strictly above its weight is at most top_p * Z +// (the fused sampling kernel's kept set, ties included). A candidate is drawn proportionally to w and accepted iff it +// is in the nucleus; a rejected candidate lies below the nucleus, so the next candidate is drawn over the tokens +// heavier than it. The accepted token is distributed as w restricted to the nucleus. The first candidate lands in the +// nucleus with probability >= top_p, so most rows decide on their second exchange. +// +// Per verify row, 8 CTAs (rows <= 32) or 4 CTAs (more rows) on every rank each own a slice of the rank's logits shard, +// staged in shared memory once. A candidate is drawn in two stages without per-token noise: each CTA draws a local +// candidate by inverse CDF over its slice, and the CTA whose log-mass plus Gumbel noise is largest supplies the +// candidate (CTA j wins with probability T_j / sum T, its token with w / T_j). Uniforms come from a counter-based +// Philox keyed by the request's seed and offset, the row, the attempt and the CTA. Each round every CTA multicasts one +// 4-word record (key, candidate logit, and max and mass in round 0 or the mass above the current candidate later) and +// reads the records of all CTAs of the row on all ranks. Every CTA takes the same decision from the same records in the +// same order, so all ranks agree without any intra-GPU handshake. Words carry the call's tag: a slot needs no clearing +// and two call halves keep a rank one call ahead from overwriting a slot still being read. +// +// SA-1R (Args.one_round_k 9 / 82; exact on 39,968 rows, eager + graph): each CTA +// also multicasts its largest bf16 logits with the round-0 record (3 per extra word). After round 0 every CTA knows M, +// the candidate logit cv and wc = __expf(cv - M); round 1 would accept iff S <= target, S = the fp32 sum (its tree) of +// w = __expf(__fmul_rn(x, tInv) - M) over every token with w > wc. x -> w is monotone (__fmul_rn by tInv > 0, fsub, +// __expf: checked exhaustively over all non-positive fp32 inputs), so if no CTA's unlisted value can beat wc (82: the +// bound tau; 9: the 9th listed), the listed w > wc are exactly round 1's summands, bitwise. Two fp32 sums of the same +// m non-negative terms differ by < m * 2^-23 * S, so accepting iff S_1r + m * 2^-21 * S_1r (round up) <= target +// implies round 1 accepts; m <= 1 is exact. Otherwise round 1 runs as before. An accepted row outputs the round-0 +// candidate, as the two-round kernel does, and consumes no Philox draw it would not. + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +namespace sampledAccept +{ +namespace +{ +constexpr int kThreads = 256; +constexpr int kWarps = kThreads / 32; +constexpr int kMaxRec = kMaxWorld * kMaxSplits; +constexpr int kMaxAttempts = 16; // exchange slots per row, reused cyclically by the rounds (no round cap) +constexpr int kTagBits = 64 - 32 - kIdBits; +constexpr float kTempEpsilon = 1e-6f; // the fused sampling kernel's 1/T guard +// Record polls trap after this many spins (the multicast all-reduce's bound): a missing writer becomes a CUDA error, +// never a hang. +constexpr long long kMaxSpins = 1LL << 31; + +// SA-1R variant kTopK: 0 none; K exact top-K list; 10 * K + P pooled (each thread keeps its top P plus its (P + 1)-th +// as a bound; the CTA lists the pool's top K plus tau = max(K-th listed, max thread bound)). +__host__ __device__ constexpr int mode_k(int m) +{ + return m >= 10 ? m / 10 : m; +} + +__host__ __device__ constexpr int mode_pool(int m) +{ + return m >= 10 ? m % 10 : 0; +} + +__host__ __device__ constexpr int mode_vals(int m) +{ + return mode_k(m) + (mode_pool(m) ? 1 : 0); +} + +// Words per record: key, logit of key, f2, f3, then the round-0 list (3 bf16 + a 16-bit tag per word). +template +struct Rec +{ + static constexpr int kXw = (mode_vals(kTopK) + 2) / 3; + static constexpr int kEx = 4 + kXw; +}; + +// Multicast exchange, per call half: [kMaxAttempts][kMaxRows][world * kMaxSplits][kEx] u64. +template +__host__ __device__ inline long long half_words(int world) +{ + return static_cast(kMaxAttempts) * kMaxRows * world * kMaxSplits * Rec::kEx; +} + +__device__ __forceinline__ uint32_t ordered(float v) +{ + uint32_t const b = __float_as_uint(v); + return (b & 0x80000000u) ? ~b : (b | 0x80000000u); +} + +// bf16 bits <-> an order-preserving 16-bit key. +__device__ __forceinline__ uint32_t ord16(uint32_t b) +{ + return (b & 0x8000u) ? (~b & 0xFFFFu) : (b | 0x8000u); +} + +__device__ __forceinline__ uint32_t unord16(uint32_t k) +{ + return (k & 0x8000u) ? (k & 0x7FFFu) : (~k & 0xFFFFu); +} + +// Philox4x32-10. +__device__ __forceinline__ uint4 philox(uint4 c, uint2 k) +{ +#pragma unroll + for (int i = 0; i < 10; ++i) + { + uint32_t const hi0 = __umulhi(0xD2511F53u, c.x), lo0 = 0xD2511F53u * c.x; + uint32_t const hi1 = __umulhi(0xCD9E8D57u, c.z), lo1 = 0xCD9E8D57u * c.z; + c = make_uint4(hi1 ^ c.y ^ k.x, lo1, hi0 ^ c.w ^ k.y, lo0); + k.x += 0x9E3779B9u; + k.y += 0xBB67AE85u; + } + return c; +} + +// u in (0, 1): (k + 1/2) / 2^23, exact in fp32. +__device__ __forceinline__ float uniform(uint32_t bits) +{ + return (static_cast(bits >> 9) + 0.5f) * (1.0f / 8388608.0f); +} + +// Key on the wire: perturbed log-mass, then the candidate token id; 0 = no candidate. Tag bits are added on the wire. +__device__ __forceinline__ unsigned long long gkey(float lg, int id) +{ + return (static_cast(ordered(lg)) << kIdBits) + | static_cast((1u << kIdBits) - 1u - static_cast(id)); +} + +__device__ __forceinline__ int key_token(unsigned long long k) +{ + return static_cast((1u << kIdBits) - 1u - static_cast(k & ((1u << kIdBits) - 1u))); +} + +__device__ __forceinline__ void st_sys(unsigned long long* p, unsigned long long v) +{ + asm volatile("st.global.relaxed.sys.u64 [%0], %1;" ::"l"(p), "l"(v) : "memory"); +} + +__device__ __forceinline__ unsigned long long ld_sys(unsigned long long const* p) +{ + unsigned long long v; + asm volatile("ld.global.relaxed.sys.u64 %0, [%1];" : "=l"(v) : "l"(p) : "memory"); + return v; +} + +template +struct Shared +{ + __nv_bfloat16 row[kMaxVs / kSplits]; + float redf[kWarps]; + unsigned long long redk[kWarps]; + unsigned long long rk[kMaxRec]; + float rv[kMaxRec], r2[kMaxRec], r3[kMaxRec]; + float wsum[kWarps]; + int owner; + int pick; + float pickx; + bool last; + float dS[kWarps]; // SA-1R: per-warp partial mass above the candidate + int dM[kWarps]; // SA-1R: per-warp count | incomplete << 16 + uint32_t wtop[kWarps][kTopK > 0 ? mode_k(kTopK) : 1]; // SA-1R: per-warp top-K keys + uint32_t wbound[kWarps]; // SA-1R pooled: per-warp max thread bound + uint32_t ctop[kTopK > 0 ? mode_vals(kTopK) : 1]; // SA-1R: the CTA's top-K keys (descending), by rank + unsigned long long rx[kMaxRec][Rec::kXw > 0 ? Rec::kXw : 1]; +}; + +template +__device__ __forceinline__ float block_max(float v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, o)); + if ((threadIdx.x & 31) == 0) + s.redf[threadIdx.x >> 5] = v; + __syncthreads(); + float r = s.redf[0]; + for (int i = 1; i < kWarps; ++i) + r = fmaxf(r, s.redf[i]); + __syncthreads(); + return r; +} + +// Fixed-order block sum: identical result for identical inputs. +template +__device__ __forceinline__ float block_sum(float v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + if ((threadIdx.x & 31) == 0) + s.redf[threadIdx.x >> 5] = v; + __syncthreads(); + float r = 0.f; + for (int i = 0; i < kWarps; ++i) + r += s.redf[i]; + __syncthreads(); + return r; +} + +template +__device__ __forceinline__ unsigned long long block_max_u64(unsigned long long v, S& s) +{ +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + unsigned long long const w = __shfl_xor_sync(0xffffffffu, v, o); + v = w > v ? w : v; + } + if ((threadIdx.x & 31) == 0) + s.redk[threadIdx.x >> 5] = v; + __syncthreads(); + unsigned long long r = s.redk[0]; + for (int i = 1; i < kWarps; ++i) + r = s.redk[i] > r ? s.redk[i] : r; + __syncthreads(); + return r; +} + +// Exclusive prefix of v over the threads in thread order, and the total. +template +__device__ __forceinline__ float block_excl_scan(float v, float& total, S& s) +{ + int const lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + float inc = v; +#pragma unroll + for (int o = 1; o < 32; o <<= 1) + { + float const t = __shfl_up_sync(0xffffffffu, inc, o); + if (lane >= o) + inc += t; + } + if (lane == 31) + s.wsum[warp] = inc; + __syncthreads(); + float base = 0.f; + total = 0.f; + for (int i = 0; i < kWarps; ++i) + { + base += i < warp ? s.wsum[i] : 0.f; + total += s.wsum[i]; + } + __syncthreads(); + return base + inc - v; +} + +// Warp-wide integer max / sum / or (the redux instructions from sm_80; a butterfly otherwise, same results). +__device__ __forceinline__ uint32_t warp_max_u32(uint32_t v) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 800) + return __reduce_max_sync(0xffffffffu, v); +#else +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + { + uint32_t const w = __shfl_xor_sync(0xffffffffu, v, o); + v = w > v ? w : v; + } + return v; +#endif +} + +__device__ __forceinline__ uint32_t warp_add_u32(uint32_t v) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 800) + return __reduce_add_sync(0xffffffffu, v); +#else +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v += __shfl_xor_sync(0xffffffffu, v, o); + return v; +#endif +} + +__device__ __forceinline__ uint32_t warp_or_u32(uint32_t v) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 800) + return __reduce_or_sync(0xffffffffu, v); +#else +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + v |= __shfl_xor_sync(0xffffffffu, v, o); + return v; +#endif +} + +// Insert key into a descending register list of K (fully unrolled: no local memory). +template +__device__ __forceinline__ void topk_insert(uint32_t (&a)[K], uint32_t key) +{ +#pragma unroll + for (int j = 0; j < K; ++j) + { + uint32_t const hi = key > a[j] ? key : a[j]; + key = key > a[j] ? a[j] : key; + a[j] = hi; + } +} + +template +__device__ __forceinline__ void topk_pop(uint32_t (&a)[K]) +{ +#pragma unroll + for (int j = 0; j + 1 < K; ++j) + a[j] = a[j + 1]; + a[K - 1] = 0u; +} + +// Per-thread insertion with an early reject: after the first few elements almost every key fails the K-th test. +template +__device__ __forceinline__ void thread_insert(uint32_t (&a)[K], uint32_t key) +{ + if (key > a[K - 1]) + topk_insert(a, key); +} + +// The warp's K largest keys (descending) into s.wtop[warp]; published by the caller's next block barrier. +template +__device__ __forceinline__ void warp_topk(uint32_t (&a)[K], S& s) +{ + int const lane = threadIdx.x & 31, warp = threadIdx.x >> 5; +#pragma unroll + for (int j = 0; j < K; ++j) + { + uint32_t const m = warp_max_u32(a[0]); + if (a[0] == m && m != 0u) + topk_pop(a); + if (lane == 0) + s.wtop[warp][j] = m; + } +} + +// Pooled: K rounds over each lane's P pool keys (descending) and the warp's max thread bound. +template +__device__ __forceinline__ void warp_pool(uint32_t (&a)[P + 1], S& s) +{ + int const lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + uint32_t const bound = warp_max_u32(a[P]); + uint32_t pl[P]; +#pragma unroll + for (int j = 0; j < P; ++j) + pl[j] = a[j]; +#pragma unroll + for (int j = 0; j < K; ++j) + { + uint32_t const m = warp_max_u32(pl[0]); + if (pl[0] == m && m != 0u) + topk_pop

(pl); + if (lane == 0) + s.wtop[warp][j] = m; + } + if (lane == 0) + s.wbound[warp] = bound; +} + +// A row's kSplits CTAs wait for each other's records, so they launch as one thread-block cluster: the hardware +// schedules a cluster's CTAs together, whatever other streams hold. Rows never wait on other rows. +template +__global__ void __launch_bounds__(kThreads, 2) sampled_accept_kernel(Args const a) +{ + using R = Rec; + extern __shared__ __align__(16) unsigned char smem[]; + Shared& s = *reinterpret_cast*>(smem); + float constexpr kNegInf = -__builtin_huge_valf(); + int const tid = threadIdx.x; + int const row = blockIdx.x / kSplits; + int const split = blockIdx.x % kSplits; + int const world = a.world; + int const nrec = world * kSplits; + // The row's sampling parameters are step inputs (the spec metadata, written before the target forward), not the + // predecessor head GEMM's output, so with prewait_params they load before the dependency wait and overlap the + // head's tail; the same values either way. + float temperature = 0.f, topP = 0.f; + long long seed_in = 0, offset_in = 0; + auto load_params = [&]() + { + temperature = a.temperatures[row]; + topP = a.top_ps[row]; + seed_in = a.seeds[row]; + offset_in = a.offsets[row]; + }; + if (a.prewait_params) + load_params(); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + asm volatile("griddepcontrol.wait;" ::: "memory"); +#endif + if (!a.prewait_params) + load_params(); +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + // early_trigger: the dependent grid (the first draft pass's input) may launch now and park at its own + // griddepcontrol.wait, which still returns only once this grid has completed and flushed; it skips the launch + // latency after this kernel ends. Otherwise dependents are released at the acceptance epilogue (the CTAs of + // splits 1.. exit without releasing). + if (a.early_trigger) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + float const tempInv = 1.0f / (temperature + kTempEpsilon); + unsigned long long const seed = static_cast(seed_in); + unsigned long long const offset = static_cast(offset_in); + unsigned int const call = *reinterpret_cast(a.counters + 1); + // Wire tags: never 0 (fresh memory), different between calls of the same half and between consecutive laps of + // the slot ring (round r uses slot r % kMaxAttempts and lap parity (r / kMaxAttempts) & 1). A rank decides round + // r only after reading every record of round r, so no rank is more than one round ahead and a slot is rewritten + // only after every reader has consumed it; the lap bit tells the new record from the previous lap's. + auto ktag_of = [&](int r) -> unsigned long long + { + unsigned int const lap = static_cast(r / kMaxAttempts) & 1u; + return (((call >> 1) << 1) | lap) % ((1u << kTagBits) - 1u) + 1u; + }; + auto ftag_of = [&](int r) -> unsigned long long + { + unsigned int const lap = static_cast(r / kMaxAttempts) & 1u; + return static_cast(((((call >> 1) << 1) | lap) & 0x7FFFFFFFu) | 0x80000000u) << 32; + }; + // SA-1R round-0 list words: a 16-bit tag, never 0. + unsigned long long const xtag = (((call >> 1) << 1) & 0x7FFFu) | 0x8000u; + long long const rec_stride = static_cast(world) * kMaxSplits * R::kEx; + unsigned long long* const xu = a.ws_uc + static_cast(call & 1u) * half_words(world) + + static_cast(row) * rec_stride; + unsigned long long* const xm = a.ws_mc + static_cast(call & 1u) * half_words(world) + + static_cast(row) * rec_stride + (a.rank * kMaxSplits + split) * R::kEx; + long long const round_stride = static_cast(kMaxRows) * rec_stride; + + // ---- stage this CTA's slice and its max ------------------------------------------------------------------------- + __nv_bfloat16 const* src = a.logits + row * a.ld; + int const sl_len = ((a.Vs + kSplits - 1) / kSplits + 7) / 8 * 8; + int const s0 = min(split * sl_len, a.Vs); + int const n = min(s0 + sl_len, a.Vs) - s0; + int const id0 = a.vocab_offset + s0; + float tmax = kNegInf; + // SA-1R: each thread's largest (ord16(bf16) << 16 | ~index) keys while staging (only rows with a nucleus cut). + bool const want_x = kTopK > 0 && topP < 1.0f; + constexpr int kK = mode_k(kTopK), kPool = mode_pool(kTopK), kVals = mode_vals(kTopK); + constexpr int kTk = kPool ? kPool + 1 : (kK > 0 ? kK : 1); + uint32_t tk[kTk]; +#pragma unroll + for (int j = 0; j < kTk; ++j) + tk[j] = 0u; + bool const vec = (a.ld % 8) == 0 && (reinterpret_cast(a.logits) & 15) == 0; + int const nvec = vec ? n / 8 : 0; + for (int v = tid; v < nvec; v += kThreads) + { + uint4 const w = *reinterpret_cast(src + s0 + 8 * v); + *reinterpret_cast(&s.row[8 * v]) = w; + __nv_bfloat16 const* e = reinterpret_cast<__nv_bfloat16 const*>(&w); +#pragma unroll + for (int j = 0; j < 8; ++j) + tmax = fmaxf(tmax, __fmul_rn(__bfloat162float(e[j]), tempInv)); + if constexpr (kTopK > 0) + { + if (want_x) + { +#pragma unroll + for (int j = 0; j < 8; ++j) + { + uint32_t const key + = (ord16(__bfloat16_as_ushort(e[j])) << 16) | (0xFFFFu - static_cast(8 * v + j)); + if constexpr (kPool > 0) + topk_insert(tk, key); // branchless: P + 1 compare-swaps + else + thread_insert(tk, key); + } + } + } + } + for (int i = nvec * 8 + tid; i < n; i += kThreads) + { + s.row[i] = src[s0 + i]; + tmax = fmaxf(tmax, __fmul_rn(__bfloat162float(src[s0 + i]), tempInv)); + if constexpr (kTopK > 0) + { + if (want_x) + topk_insert( + tk, (ord16(__bfloat16_as_ushort(src[s0 + i])) << 16) | (0xFFFFu - static_cast(i))); + } + } + if constexpr (kTopK > 0) + { + if (tid < kVals) + s.ctop[tid] = 0u; // published with s.wtop by block_max's barrier + if (want_x) + { + if constexpr (kPool > 0) + warp_pool(tk, s); // s.wtop / s.wbound published by block_max's barrier + else + warp_topk(tk, s); // s.wtop published by block_max's barrier + } + } + float const smax = block_max(tmax, s); // the staged slice is visible to all threads after this barrier + if constexpr (kTopK > 0) + { + // SA-1R: the CTA's top K of the warp lists by rank (keys are unique): a lane quad per key, no serial merge; + // s.ctop is read after the round-0 draw's first barrier + constexpr int kN = kWarps * kK; + constexpr int kSp = kN * 4 <= kThreads ? 4 : 1; // threads per key + if (want_x && tid < kN * kSp) + { + uint32_t const* const wl = &s.wtop[0][0]; + int const part = tid % kSp; + uint32_t const key = wl[tid / kSp]; + int rank = 0; +#pragma unroll + for (int j = part; j < kN; j += kSp) + rank += wl[j] > key ? 1 : 0; + if constexpr (kSp == 4) + { + rank += __shfl_xor_sync(0xffffffffu, rank, 1); + rank += __shfl_xor_sync(0xffffffffu, rank, 2); + } + if (part == 0 && key != 0u && rank < kK) + s.ctop[rank] = key; + } + } + float M = 0.f, wc = 0.f, target = 0.f, cv = 0.f; // global max, candidate weight and logit, top_p * Z + unsigned long long ckey = 0ULL; + int const gid = a.rank * kMaxSplits + split; + for (int r = 0;; ++r) + { + // ---- local candidate by inverse CDF over the slice's tokens heavier than the candidate (w > 0 in round 0) + float const ref = r == 0 ? smax : M; + float const wmin = r == 0 ? 0.f : wc; + float ts = 0.f; + for (int i = tid; i < n; i += kThreads) + { + float const w = __expf(__fmul_rn(__bfloat162float(s.row[i]), tempInv) - ref); + ts += w > wmin ? w : 0.f; + } + float total; + float const excl = block_excl_scan(ts, total, s); + if constexpr (kTopK > 0) + { + // SA-1R round 0: the list words depend only on the staged slice, so they go out now (s.ctop / s.wbound are + // published by the barrier above), one lane per word, ahead of the main record: off the readers' path + if (r == 0 && want_x && tid < R::kXw) + { + uint32_t tau = 0u; + if constexpr (kPool > 0) + { + tau = s.ctop[kK - 1]; // the largest unlisted pool key is <= the K-th listed +#pragma unroll + for (int wq = 0; wq < kWarps; ++wq) + tau = s.wbound[wq] > tau ? s.wbound[wq] : tau; + } + unsigned long long w = xtag; +#pragma unroll + for (int t = 0; t < 3; ++t) + { + int const e = 3 * tid + t; + uint32_t const kv = e < kK ? s.ctop[e] : (kPool > 0 && e == kK ? tau : 0u); + // missing entries: -inf (bf16 0xFF80), w = 0 + uint32_t const b = kv != 0u ? unord16(kv >> 16) : 0xFF80u; + w |= static_cast(b) << (16 * (3 - t)); + } + st_sys(xm + 4 + tid, w); // round 0 = slot 0 + } + } + uint4 const rnd = philox(make_uint4(static_cast(gid), static_cast(row + 64 * r), + static_cast(offset), static_cast(offset >> 32)), + make_uint2(static_cast(seed), static_cast(seed >> 32))); + float const tgt = uniform(rnd.x) * total; + if (tid == 0) + s.owner = -1; + __syncthreads(); + if (ts > 0.f && excl <= tgt) + atomicMax(&s.owner, tid); + __syncthreads(); + if (tid == s.owner) + { + float acc = excl; + int pick = -1; + float px = 0.f; + if constexpr (kIlpWalk) + { + // SA-PW: the owner's strided tokens 8 at a time. The 8 x and w (the same expressions) are + // independent, so their MUFU work overlaps; the serial sum then adds them in the same order with + // the same early exit: the same acc sequence, pick and px as the one-token walk, bit for bit. + constexpr int kW = 8; + bool done = false; + for (int i0 = tid; i0 < n && !done; i0 += kW * kThreads) + { + float xs[kW], wv[kW]; +#pragma unroll + for (int k = 0; k < kW; ++k) + { + int const i = i0 + k * kThreads; + xs[k] = i < n ? __fmul_rn(__bfloat162float(s.row[i]), tempInv) : 0.f; + wv[k] = __expf(xs[k] - ref); + } +#pragma unroll + for (int k = 0; k < kW; ++k) + { + int const i = i0 + k * kThreads; + if (!done && i < n && wv[k] > wmin) + { + acc += wv[k]; + pick = i; + px = xs[k]; + if (acc > tgt) + done = true; + } + } + } + } + else + { + for (int i = tid; i < n; i += kThreads) + { + float const x = __fmul_rn(__bfloat162float(s.row[i]), tempInv); + float const w = __expf(x - ref); + if (w > wmin) + { + acc += w; + pick = i; + px = x; + if (acc > tgt) + break; + } + } + } + s.pick = pick; + s.pickx = px; + } + __syncthreads(); + unsigned long long sb = 0ULL; + if (total > 0.f && s.owner >= 0) + sb = gkey(__logf(total) + ref - logf(-logf(uniform(rnd.y))), id0 + s.pick); + float const smab = total; + unsigned long long const ktag = ktag_of(r); + unsigned long long const ftag = ftag_of(r); + unsigned long long* const xr = xu + (r % kMaxAttempts) * round_stride; + bool const with_x = want_x && r == 0; + if (tid == 0) + { + unsigned long long* const xw = xm + (r % kMaxAttempts) * round_stride; + st_sys(xw + 0, (sb << kTagBits) | ktag); + st_sys(xw + 1, ftag | __float_as_uint(sb != 0ULL ? s.pickx : 0.f)); + st_sys(xw + 2, ftag | __float_as_uint(r == 0 ? smax : smab)); + st_sys(xw + 3, ftag | __float_as_uint(r == 0 ? smab : 0.f)); + } + // ---- every record of the row, every rank --------------------------------------------------------------- + int const nw = with_x ? R::kEx : 4; + for (int t = tid; t < nrec * nw; t += kThreads) + { + int const q = t / nw, f = t % nw; + unsigned long long const* const p = xr + (q / kSplits * kMaxSplits + q % kSplits) * R::kEx + f; + unsigned long long v; + long long spins = 0; + if (f == 0) + { + do + { + v = ld_sys(p); + if (++spins > kMaxSpins) + __trap(); + } while ((v & ((1ULL << kTagBits) - 1ULL)) != ktag); + s.rk[q] = v >> kTagBits; + } + else if (f < 4) + { + do + { + v = ld_sys(p); + if (++spins > kMaxSpins) + __trap(); + } while ((v & 0xFFFFFFFF00000000ULL) != ftag); + float const fv = __uint_as_float(static_cast(v)); + (f == 1 ? s.rv : f == 2 ? s.r2 : s.r3)[q] = fv; + } + else + { + do + { + v = ld_sys(p); + if (++spins > kMaxSpins) + __trap(); + } while ((v & 0xFFFFULL) != xtag); + s.rx[q][f - 4] = v; + } + } + __syncthreads(); + // ---- the decision, identical in every CTA of the row on every rank: fixed record order ----------------- + unsigned long long gk = 0ULL; + float gv = 0.f; + for (int q = 0; q < nrec; ++q) + { + if (s.rk[q] > gk) + { + gk = s.rk[q]; + gv = s.rv[q]; + } + } + bool accept; + if (r == 0) + { + M = kNegInf; + for (int q = 0; q < nrec; ++q) + M = fmaxf(M, s.r2[q]); + float Z = 0.f; + for (int q = 0; q < nrec; ++q) + Z += s.r2[q] == kNegInf ? 0.f : s.r3[q] * __expf(s.r2[q] - M); + target = topP * Z; + accept = !(topP < 1.0f); // no nucleus cut: the first draw stands + ckey = gk; + cv = gv; + if constexpr (kTopK > 0) + { + // SA-1R: every thread takes (record, entry) items; the per-warp partials are published by the round's + // barrier below and every thread combines them in warp order after it (no new barrier) + if (with_x) + { + float const wcand = __expf(cv - M); // = round 1's wmin + float part = 0.f; + int cnt = 0, inc = 0; + for (int t = tid; t < nrec * kVals; t += kThreads) + { + int const q = t / kVals, e = t % kVals; + uint32_t const b = static_cast(s.rx[q][e / 3] >> (16 * (3 - e % 3))) & 0xFFFFu; + float const w = __expf( + __fmul_rn(__bfloat162float(__ushort_as_bfloat16(static_cast(b))), tempInv) + - M); // w bitwise as round 1 computes it + if (w > wcand) + { + if (kPool > 0 && e == kK) + inc = 1; // tau (the largest unlisted value) is heavier than the candidate + else + { + part += w; + ++cnt; + if (kPool == 0 && e == kK - 1) + inc = 1; // the K-th is heavier: unlisted heavier tokens may exist + } + } + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) + part += __shfl_xor_sync(0xffffffffu, part, o); // any order: the margin covers it + cnt = static_cast(warp_add_u32(static_cast(cnt))); + inc = static_cast(warp_or_u32(static_cast(inc))); + if ((tid & 31) == 0) + { + s.dS[tid >> 5] = part; + s.dM[tid >> 5] = cnt | (inc << 16); + } + } + } + } + else + { + float S = 0.f; // mass strictly above the current candidate + for (int q = 0; q < nrec; ++q) + S += s.r2[q]; + accept = S <= target || gk == 0ULL; + if (!accept) + { + ckey = gk; + cv = gv; + } + } + wc = __expf(cv - M); + __syncthreads(); // records consumed before the next round's writes to s + bool decided = false; + if constexpr (kTopK > 0) + { + if (r == 0 && with_x) + { + float S1 = 0.f; + int m = 0, inc = 0; + for (int wq = 0; wq < kWarps; ++wq) + { + S1 += s.dS[wq]; + m += s.dM[wq] & 0xFFFF; + inc |= s.dM[wq] >> 16; + } + if (!inc) + { + float const margin = m <= 1 ? 0.f : S1 * static_cast(m) * (1.0f / 2097152.0f); + decided = __fadd_ru(S1, margin) <= target; + accept = accept || decided; + } + } + } + if (accept) + { + if (split != 0) + return; + if (tid == 0) + { + a.tokens[row] = key_token(ckey); + if (a.extra_attempts != nullptr && r > 1) + atomicAdd(a.extra_attempts, r - 1); + if (a.decided != nullptr && decided) + atomicAdd(a.decided, 1); + } + break; + } + } + + // ---- split 0: the last row to finish runs the strict acceptance ---------------------------------------------- +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) + if (!a.early_trigger) + asm volatile("griddepcontrol.launch_dependents;"); +#endif + __threadfence(); + __syncthreads(); + if (tid == 0) + s.last = atomicAdd(a.counters, 1u) == static_cast(a.rows) - 1; + __syncthreads(); + if (!s.last) + return; + __threadfence(); + if (tid == 0) + { + a.counters[0] = 0u; + a.counters[1] = call + 1u; + } + if (tid < a.nreq) + { + int const* tok = a.tokens; + int const b = tid; + int const k1 = a.draft_len + 1; + int nacc = 1; + bool run = true; + for (int k = 0; k < a.draft_len; ++k) + { + int const tt = *reinterpret_cast(tok + b * k1 + k); + a.acc_tokens[b * k1 + k] = tt; + run = run && a.draft[b * a.draft_len + k] == tt; + nacc += run ? 1 : 0; + } + a.acc_tokens[b * k1 + a.draft_len] = *reinterpret_cast(tok + b * k1 + a.draft_len); + if (a.force_vec != nullptr) + nacc = a.force_vec[b]; + else if (a.force_total > 0) + nacc = a.force_total; + a.num_acc[b] = nacc; + a.gather_ids[b] = static_cast(a.batch_idx[b]) * k1 + nacc - 1; + if (a.next_new != nullptr) + a.next_new[b * a.next_stride] + = *reinterpret_cast(tok + a.batch_idx[b] * k1 + nacc - 1); + } +} + +template +cudaError_t launch(Args const& a, bool use_pdl, cudaStream_t stream) +{ + static int max_clusters = 0; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(a.rows * kSplits); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = sizeof(Shared); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kSplits; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + if (max_clusters == 0) + { + cudaError_t err = cudaFuncSetAttribute(sampled_accept_kernel, + cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(sizeof(Shared))); + if (err == cudaSuccess) + { + err = cudaOccupancyMaxActiveClusters(&max_clusters, sampled_accept_kernel, &cfg); + } + if (err != cudaSuccess) + { + return err; + } + } + TLLM_CHECK_WITH_INFO(max_clusters >= 1, "m3_sampled_accept: a %d-CTA cluster does not fit on one GPC", kSplits); + return cudaLaunchKernelEx(&cfg, sampled_accept_kernel, a); +} + +template +cudaError_t launchRows(Args const& a, bool use_pdl, cudaStream_t stream) +{ + // TP4 caps 2 / 4 / 8 (8 / 16 / 32 rows) and TP8 cap 1 (4 rows): 8 CTAs per row, at most 256 CTAs (2 per SM); + // more rows: 4 CTAs per row. + if (a.ilp_walk) + { + return a.rows <= 32 ? launch<8, kTopK, true>(a, use_pdl, stream) : launch<4, kTopK, true>(a, use_pdl, stream); + } + return a.rows <= 32 ? launch<8, kTopK, false>(a, use_pdl, stream) : launch<4, kTopK, false>(a, use_pdl, stream); +} + +} // namespace + +long long sampledAcceptWsWords(int world, int oneRoundK) +{ + TLLM_CHECK_WITH_INFO( + oneRoundK == 0 || oneRoundK == 9 || oneRoundK == 82, "SA-1R variant %d is not built", oneRoundK); + return 2 * (oneRoundK == 9 ? half_words<9>(world) : oneRoundK == 82 ? half_words<82>(world) : half_words<0>(world)); +} + +cudaError_t invokeSampledAccept(Args const& a, bool usePdl, cudaStream_t stream) +{ + switch (a.one_round_k) + { + case 0: return launchRows<0>(a, usePdl, stream); + case 9: return launchRows<9>(a, usePdl, stream); + case 82: return launchRows<82>(a, usePdl, stream); + default: TLLM_CHECK_WITH_INFO(false, "SA-1R variant %d is not built", a.one_round_k); + } + return cudaErrorInvalidValue; +} + +} // namespace sampledAccept +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.h new file mode 100644 index 000000000000..aa52cf4a0559 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.h @@ -0,0 +1,87 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ +//! Fused sampled (temperature + top-p) verification of the MiniMax-M3 Eagle3 target head over the TP-sharded +//! vocabulary, decode rows <= kMaxRows; see sampledAccept.cu. +namespace sampledAccept +{ + +constexpr int kMaxSplits = 8; // CTAs per verify row (at most) +constexpr int kMaxRows = 48; // verify rows per call +constexpr int kMaxWorld = 8; // TP ranks +constexpr int kMaxVs = 100352; // vocab shard width (TP2: 200064 / 2 = 100032, padded to a multiple of 512) +constexpr int kIdBits = 18; // token ids < 2^18 + +struct Args +{ + __nv_bfloat16 const* logits; // [rows, Vs] this rank's shard + long long ld; + int rows, Vs, vocab_offset; + float const* temperatures; + float const* top_ps; + long long const* seeds; + long long const* offsets; + unsigned long long* ws_uc; + unsigned long long* ws_mc; + unsigned int* counters; // [0] finished rows, [1] call + int* tokens; // [rows] sampled token per row + int* extra_attempts; // optional [1]: rounds beyond the second, summed over rows + int rank, world; + int const* draft; + int const* batch_idx; + int* acc_tokens; + int* num_acc; + long long* gather_ids; + int draft_len, nreq, force_total; + int* next_new; + int next_stride; + int const* force_vec; + int prewait_params; // nonzero: the row's sampling parameters load before the PDL dependency wait + int early_trigger; // nonzero: dependents are released right after the dependency wait, not at the epilogue + // SA-1R (one-round decision): 0 the two-round kernel; 9 each CTA's top-9 logits ride its round-0 record; 82 pooled + // (each thread's top 2, the CTA lists the pool's top 8 plus the bound of everything unlisted). A row whose first + // candidate is provably in the nucleus then decides in round 0, with the same token as the two-round kernel. + int one_round_k; + int* decided; // optional [1]: rows decided in round 0 (SA-1R), summed + int ilp_walk; // nonzero: the owner walk takes its tokens 8 at a time (same weights, same summation order) +}; + +//! Words (uint64) of the multicast exchange workspace for `world` ranks and SA-1R variant `oneRoundK`: two call halves. +long long sampledAcceptWsWords(int world, int oneRoundK = 0); + +//! Launch the sampled verification over a.rows * (8 or 4) co-resident CTAs (8 per row up to 32 rows). +cudaError_t invokeSampledAccept(Args const& a, bool usePdl, cudaStream_t stream); + +} // namespace sampledAccept +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.cu new file mode 100644 index 000000000000..8cbf5d90175c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.cu @@ -0,0 +1,89 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +namespace +{ + +constexpr int kThreads = 128; + +// One packed bf16 pair of gate and up values -> its two bf16 results (the low half from the low halves). The +// instruction sequence of the Triton kernel's PTX at its defaults (host/silu/silu_and_mul_plain.ptx): g * -log2(e) as +// one product, ex2.approx, + 1, div.full 1 / that, g * s, (u + 0.0) as an IEEE add, the product, one rounding to bf16. +__device__ __forceinline__ uint32_t siluMulBf16x2(uint32_t gate, uint32_t up) +{ + uint32_t r; + asm("{\n\t.reg .b32 gl, gh, ul, uh, t0, t1, e0, e1, s0, s1, p0, p1, q0, q1, r0, r1, one, zero;\n\t" + ".reg .b16 hl, hh;\n\t" + "mov.b32 {hl, hh}, %1;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, %2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mul.rn.f32 t0, gl, 0fBFB8AA3B;\n\tmul.rn.f32 t1, gh, 0fBFB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 p0, gl, s0;\n\tmul.rn.f32 p1, gh, s1;\n\t" + "mov.b32 zero, 0f00000000;\n\tadd.rn.f32 q0, ul, zero;\n\tadd.rn.f32 q1, uh, zero;\n\t" + "mul.rn.f32 r0, q0, p0;\n\tmul.rn.f32 r1, q1, p1;\n\t" + "cvt.rn.bf16x2.f32 %0, r1, r0;\n\t}" + : "=r"(r) + : "r"(gate), "r"(up)); + return r; +} + +// Eight outputs per thread: 16-byte loads of the gate and the up half, one 16-byte store. +__global__ void __launch_bounds__(kThreads) + siluAndMulKernel(uint4 const* __restrict__ x, uint4* __restrict__ out, int64_t rows, int64_t vecs) +{ + int64_t const row = blockIdx.y; + int64_t const v = static_cast(blockIdx.x) * kThreads + threadIdx.x; + if (row >= rows || v >= vecs) + { + return; + } + uint4 const g = x[row * 2 * vecs + v]; + uint4 const u = x[row * 2 * vecs + vecs + v]; + uint4 o; + o.x = siluMulBf16x2(g.x, u.x); + o.y = siluMulBf16x2(g.y, u.y); + o.z = siluMulBf16x2(g.z, u.z); + o.w = siluMulBf16x2(g.w, u.w); + out[row * vecs + v] = o; +} + +} // namespace + +cudaError_t invokeSiluAndMul(uint16_t const* x, uint16_t* out, int64_t rows, int64_t d, cudaStream_t stream) +{ + int64_t const vecs = d / 8; + dim3 const grid(static_cast((vecs + kThreads - 1) / kThreads), static_cast(rows)); + siluAndMulKernel<<>>( + reinterpret_cast(x), reinterpret_cast(out), rows, vecs); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.h new file mode 100644 index 000000000000..669e33eb5cf5 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.h @@ -0,0 +1,40 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +//! out[m, j] = bf16(silu(g) * (u + 0)) with g = x[m, j], u = x[m, d + j] (x bf16 [rows, 2d], out bf16 [rows, d], both +//! contiguous, d a multiple of 8, 16-byte aligned rows): bit for bit the Triton silu_and_mul kernel at its defaults +//! (no limit, alpha 1, beta 0), the plain SwiGLU of the Eagle3 drafter MLP. +cudaError_t invokeSiluAndMul(uint16_t const* x, uint16_t* out, int64_t rows, int64_t d, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.cu new file mode 100644 index 000000000000..d51138cc15d2 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.cu @@ -0,0 +1,746 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// MiniMax-M3 TP8 long-context sparse decode attention in one launch (8 query heads on one KV head). +// +// Grid: one cluster of 16 CTAs per query token; CTA `r` attends the token's `r`-th selected block. The QK and PV +// MMAs are transposed (S^T = K q^T, O^T = V^T P^T) so the 8 heads fill the MMA n dimension. Each CTA pushes its +// normalized partial over DSMEM so that CTA `r` holds (head r / 2, dims (r % 2) * 64 + [0, 64)) of all 16 partials, +// merges them, and writes the BF16 row and, optionally, its MXFP8 copy and swizzled UE8M0 scales. +// +// Per-block math: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and PV on FP16 tensor cores with +// FP32 accumulation, log2-domain softmax with either the linear denominator or the log-domain recurrence. +// +// PREWAIT stages everything that does not depend on the top-16 table before the grid dependency wait: the request's +// seq_lens entry, the q rows and the request's block-table row (in dynamic shared memory after Smem8, when it has at +// most kBtCap entries). The arithmetic is the same in both variants. +// +// ONE_DIV merge (MSA_ONE_DIV_MERGE): each CTA pushes its unnormalized partial with its row max m and sum l, and the +// merge forms o = sum_c w_c acc_c / sum_c w_c l_c with w_c = 2^(m_c - max m), one division per output, instead of +// normalizing every partial and weighting by its LSE. Deterministic (fixed chunk order, no atomics); a different +// rounding sequence from the Triton merge, inside the same band of the float64 reference. + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +namespace +{ + +constexpr int kDim = 128; +constexpr int kPage = 128; +constexpr int kTopk = 16; +constexpr int kThreads = 128; +constexpr int kPitch = kDim + 8; // halves per smem row: 272 B, conflict-free ldmatrix +constexpr float kNegInf = -__builtin_huge_valf(); +constexpr int kHeads8 = 8; +constexpr int kHalfDim = kDim / 2; +constexpr int kBtCap = 8192; // block-table entries staged in shared memory (8192 x 128 = 1M tokens) + +using Params = MinimaxM3Tp8SparseDecodeLc8Params; + +struct Smem8 +{ + __half qs[kHeads8 * kPitch]; + __half vs[kPage * kPitch]; + __half ps[kHeads8 * kPitch]; // P^T: [head][token] + float recv[kTopk][kHalfDim]; // (head rank / 2, dims (rank % 2) * 64 + [0, 64)) partials of the 16 CTAs + float recv_lse[kTopk]; // the chunk LSEs (log2); with ONE_DIV, the chunk row maxima + float recv_l[kTopk]; // ONE_DIV: the chunk row sums + float part[kHalfDim]; + float ex[kTopk]; // the 16 chunk exponentials (ONE_DIV: weighted row sums), formed once per CTA + float wgt[kTopk]; // the 16 merge weights, formed once per CTA + float red[4][kHeads8]; +}; + +template +constexpr size_t smemBytes() +{ + return PREWAIT ? sizeof(Smem8) + kBtCap * sizeof(int) : sizeof(Smem8); +} + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +__device__ __forceinline__ uint32_t smem_u32(void const* p) +{ + return static_cast(__cvta_generic_to_shared(p)); +} + +__device__ __forceinline__ uint32_t e2m1x2_to_f16x2(uint32_t byte) +{ + uint32_t out; + asm("{\n .reg .b8 s;\n cvt.u8.u32 s, %1;\n cvt.rn.f16x2.e2m1x2 %0, s;\n}" : "=r"(out) : "r"(byte)); + return out; +} + +__device__ __forceinline__ uint32_t e4m3x2_to_f16x2(uint16_t pair) +{ + uint32_t out; + asm("cvt.rn.f16x2.e4m3x2 %0, %1;" : "=r"(out) : "h"(pair)); + return out; +} + +__device__ __forceinline__ float ex2(float x) +{ + float y; + asm("ex2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ float lg2(float x) +{ + float y; + asm("lg2.approx.f32 %0, %1;" : "=f"(y) : "f"(x)); + return y; +} + +__device__ __forceinline__ void ldsm_x4(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void ldsm_x4_t(uint32_t (&r)[4], uint32_t addr) +{ + asm volatile("ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%0,%1,%2,%3}, [%4];" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"(addr)); +} + +__device__ __forceinline__ void mma(float (&c)[4], uint32_t const (&a)[4], uint32_t b0, uint32_t b1) +{ + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, " + "{%0,%1,%2,%3};" + : "+f"(c[0]), "+f"(c[1]), "+f"(c[2]), "+f"(c[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} + +__device__ __forceinline__ void cluster_sync() +{ + asm volatile("barrier.cluster.arrive.release.aligned;\nbarrier.cluster.wait.acquire.aligned;" ::: "memory"); +} + +__device__ __forceinline__ uint32_t map_cluster(void const* local, uint32_t rank) +{ + uint32_t remote; + asm volatile("mapa.shared::cluster.u32 %0, %1, %2;" : "=r"(remote) : "r"(smem_u32(local)), "r"(rank)); + return remote; +} + +__device__ __forceinline__ void st_cluster_f32(uint32_t addr, float a) +{ + asm volatile("st.shared::cluster.f32 [%0], %1;" ::"r"(addr), "f"(a) : "memory"); +} + +// One packed NVFP4 row (64 bytes, 128 values) with its 8 E4M3 scales -> 128 fp16 values in smem. +__device__ __forceinline__ void dequant_row(__half* dst, uint8_t const* packed, uint8_t const (&sc)[8], bool valid) +{ + uint4 const* src = reinterpret_cast(packed); + uint32_t* d = reinterpret_cast(dst); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + uint32_t words[4] = {w.x, w.y, w.z, w.w}; + // 16 bytes = 32 values = scale columns 2c and 2c + 1. +#pragma unroll + for (int h = 0; h < 2; ++h) + { + uint8_t const s = valid ? sc[2 * c + h] : 0; + uint32_t const sh = e4m3x2_to_f16x2(static_cast(s | (s << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); +#pragma unroll + for (int j = 0; j < 2; ++j) + { + uint32_t const word = words[2 * h + j]; +#pragma unroll + for (int b = 0; b < 4; ++b) + { + uint32_t const v2 = e2m1x2_to_f16x2((word >> (8 * b)) & 0xFF); + __half2 const x = __hmul2(*reinterpret_cast<__half2 const*>(&v2), scale); + d[c * 16 + h * 8 + j * 4 + b] = *reinterpret_cast(&x); + } + } + } + } +} + +// q: 8 heads x 128 e4m3 -> fp16; threads 0..63 = (head, 16-value chunk). +__device__ __forceinline__ void load_q(Smem8& s, Params const& p, int tok, int tid) +{ + if (tid < kHeads8 * 8) + { + int const h = tid >> 3; + int const c = (tid & 7) * 16; + uint4 const w = *reinterpret_cast(p.q + tok * p.q_tok + h * p.q_head + c); + uint32_t const words[4] = {w.x, w.y, w.z, w.w}; + uint32_t* d = reinterpret_cast(s.qs + h * kPitch + c); +#pragma unroll + for (int i = 0; i < 4; ++i) + { + d[2 * i] = e4m3x2_to_f16x2(static_cast(words[i] & 0xFFFF)); + d[2 * i + 1] = e4m3x2_to_f16x2(static_cast(words[i] >> 16)); + } + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +// This CTA's 16-byte-aligned slice of [base, base + bytes) into L2, at most 16 KiB per bulk prefetch. +__device__ __forceinline__ void prefetch_l2_slice(uint8_t const* base, int64_t bytes) +{ + if (base == nullptr || bytes <= 0) + return; + int64_t const n = gridDim.x; + int64_t const slice = ((bytes + n - 1) / n + 15) & ~int64_t{15}; + int64_t const off = static_cast(blockIdx.x) * slice; + if (off >= bytes) + return; + int64_t const len = min(slice, bytes - off) & ~int64_t{15}; + for (int64_t o = 0; o < len; o += 16384) + { + uint32_t const sz = static_cast(min(int64_t{16384}, len - o)); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(base + off + o), "r"(sz) : "memory"); + } +} +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + +template +__global__ void __launch_bounds__(kThreads) m3_tp8_lc8_attn_kernel(__grid_constant__ Params const p) +{ +#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) + extern __shared__ __align__(128) uint8_t smem_raw[]; + Smem8& s = *reinterpret_cast(smem_raw); + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int const g = lane >> 2; + int const qd = lane & 3; + int const tok = blockIdx.x / kTopk; + uint32_t rank; + asm volatile("mov.u32 %0, %%cluster_ctarank;" : "=r"(rank)); + + asm volatile("barrier.cluster.arrive.relaxed.aligned;" ::: "memory"); + // The o_proj weights are read-only for the whole step: start pulling them into L2 now, while HBM is idle in the + // latency-bound attention phase (one lane of warps 0 and 1; nothing waits on these). + if (tid == 0) + prefetch_l2_slice(p.pf0, p.pf0_bytes); + else if (tid == 32) + prefetch_l2_slice(p.pf1, p.pf1_bytes); + if constexpr (!PREWAIT) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + + int const req = tok / p.dql; + int const qoff = tok - req * p.dql; + int const seq = p.seq_lens[req]; + int const kv_len = max(seq - p.dql + qoff + 1, 0); + int const nblocks = (kv_len + kPage - 1) / kPage; + int const real_topk = min(kTopk, nblocks); + // The next sparse layer's block that its CTA (tok, rank) selected at the previous step (a decode step selects + // mostly the blocks it selected one step earlier): its K/V pages and scales into L2 for that layer, from one lane + // of warp 2. The id buffer was written by a kernel of the previous step, long complete; a recorded id at or past + // this request's block count (another request's row) or a page outside the pool is skipped. + if (p.n1_next != nullptr && tid == 64) + { + int const nb = p.n1_next[tok * kTopk + rank]; + if (nb >= 0 && nb < nblocks) + { + int64_t const pg = p.block_table[req * p.bt_stride + nb]; + if (pg >= 0 && pg < p.n1_pages) + { + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.n1_k + pg * p.k_blk), "r"(kPage * 64) + : "memory"); + asm volatile("cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.n1_v + pg * p.v_blk), "r"(kPage * 64) + : "memory"); + asm volatile( + "cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.n1_ksc + pg * p.ksc_blk), "r"(kPage * 8) + : "memory"); + asm volatile( + "cp.async.bulk.prefetch.L2.global [%0], %1;" ::"l"(p.n1_vsc + pg * p.vsc_blk), "r"(kPage * 8) + : "memory"); + } + } + } + + // The request's block-table row, staged before the grid dependency (written before the graph runs). + int* const s_bt = reinterpret_cast(smem_raw + sizeof(Smem8)); + bool bt_smem = false; + if constexpr (PREWAIT) + { + int const nbt = (max(seq, 0) + kPage - 1) / kPage; + bt_smem = nbt <= kBtCap; + if (bt_smem) + { + for (int i = tid; i < nbt; i += kThreads) + { + s_bt[i] = __ldg(p.block_table + req * p.bt_stride + i); + } + } + // Before the wait only behind the select: an identity step has no scorer / select, so this grid's + // predecessor is the producer itself, which releases its dependents at entry, before it writes q. There q is + // read after the wait, whether the blocks come from the lengths (identity) or from the staged table. + if (!p.identity && !p.identity_step) + load_q(s, p, tok, tid); + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + int blk = static_cast(rank) >= real_topk ? -1 + : p.identity ? static_cast(rank) + : p.topk[tok * p.t_tok + rank * p.t_k]; + // Fail closed on an id past the row (a stale or corrupt table), as the stock Triton producer does: the block is + // skipped. Read through the block table it would be another request's page, every token masked, and its softmax + // ex2(-inf - (-inf)) = NaN. + if (blk >= nblocks) + blk = -1; + bool const active = blk >= 0; + // Record this step's selection for this layer's next-step prefetch (read by the previous sparse layer's CTAs). + if (p.n1_self != nullptr && tid == 0) + p.n1_self[tok * kTopk + rank] = blk; + if constexpr (PREWAIT) + { + if (p.identity || p.identity_step) + load_q(s, p, tok, tid); + __syncthreads(); // s_bt and the q rows are complete + } + else + { + load_q(s, p, tok, tid); + } + + // Lane holds heads 2 * qd + {0, 1}; acc[mt]: dims warp * 32 + mt * 16 + g + {0, 8}. + float m_row[2] = {kNegInf, kNegInf}; + float l_row[2] = {0.F, 0.F}; + float acc[2][4]; +#pragma unroll + for (int j = 0; j < 2; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + acc[j][i] = 0.F; + + // K rows n = warp * 32 + j * 8 + g stay packed in registers. As the MMA A operand, m-tile mt takes rows + // j = 2 mt (row g) and 2 mt + 1 (row g + 8); k-step ks takes packed bytes 8 ks + qd and 8 ks + 4 + qd. + uint32_t kw[4][16]; + uint2 ksc[4]; + if (active) + { + int64_t const page = bt_smem ? s_bt[blk] : p.block_table[req * p.bt_stride + blk]; +#pragma unroll + for (int j = 0; j < 4; ++j) + { + int const n = warp * 32 + j * 8 + g; + bool const valid = blk * kPage + n < kv_len; + uint4 const* src = reinterpret_cast(p.k + page * p.k_blk + n * 64); +#pragma unroll + for (int c = 0; c < 4; ++c) + { + uint4 const w = valid ? __ldg(src + c) : make_uint4(0, 0, 0, 0); + kw[j][4 * c] = w.x; + kw[j][4 * c + 1] = w.y; + kw[j][4 * c + 2] = w.z; + kw[j][4 * c + 3] = w.w; + } + ksc[j] = valid ? __ldg(reinterpret_cast(p.ksc + page * p.ksc_blk + n * 8)) : make_uint2(0, 0); + } + bool const valid = blk * kPage + tid < kv_len; + uint8_t vsc[8]; + if (valid) + { + uint8_t const* vb = p.vsc + page * p.vsc_blk + (tid >> 2) * 32 + (tid & 3); +#pragma unroll + for (int c = 0; c < 8; ++c) + vsc[c] = __ldg(vb + 4 * c); + } + dequant_row(s.vs + tid * kPitch, p.v + page * p.v_blk + tid * 64, vsc, valid); + } + __syncthreads(); + + if (active) + { + float const scale_log2 = p.sm_scale * 1.4426950409F * __ldg(p.kgs); + int const q4 = qd * 8; + // S^T = K q^T for this warp's 32 tokens: rows (tokens) g / g + 8 of each m-tile, columns (heads) 2 qd + {0, 1}. + float sacc[2][4]; +#pragma unroll + for (int j = 0; j < 2; ++j) +#pragma unroll + for (int i = 0; i < 4; ++i) + sacc[j][i] = 0.F; +#pragma unroll + for (int ks2 = 0; ks2 < kDim / 16; ks2 += 2) + { + uint32_t bq[4]; + ldsm_x4(bq, smem_u32(s.qs + (lane & 7) * kPitch + ks2 * 16 + (lane >> 3) * 8)); +#pragma unroll + for (int u = 0; u < 2; ++u) + { + int const ks = ks2 + u; +#pragma unroll + for (int mt = 0; mt < 2; ++mt) + { + uint32_t a[4]; +#pragma unroll + for (int hi = 0; hi < 2; ++hi) + { + int const j = 2 * mt + hi; + uint32_t const scw = ks < 4 ? ksc[j].x : ksc[j].y; + uint8_t const sc = static_cast(scw >> (8 * (ks & 3))); + uint32_t const sh = e4m3x2_to_f16x2(static_cast(sc | (sc << 8))); + __half2 const scale = *reinterpret_cast<__half2 const*>(&sh); + uint32_t const r0 = e2m1x2_to_f16x2((kw[j][2 * ks] >> q4) & 0xFF); + uint32_t const r1 = e2m1x2_to_f16x2((kw[j][2 * ks + 1] >> q4) & 0xFF); + __half2 const h0 = __hmul2(*reinterpret_cast<__half2 const*>(&r0), scale); + __half2 const h1 = __hmul2(*reinterpret_cast<__half2 const*>(&r1), scale); + a[hi] = *reinterpret_cast(&h0); + a[2 + hi] = *reinterpret_cast(&h1); + } + mma(sacc[mt], a, bq[2 * u], bq[2 * u + 1]); + } + } + } + float mloc[2] = {kNegInf, kNegInf}; +#pragma unroll + for (int mt = 0; mt < 2; ++mt) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const t = warp * 32 + mt * 16 + g + 8 * (i >> 1); + float const v = blk * kPage + t < kv_len ? 0.F + sacc[mt][i] * scale_log2 : kNegInf; + sacc[mt][i] = v; + mloc[i & 1] = fmaxf(mloc[i & 1], v); + } +#pragma unroll + for (int e = 0; e < 2; ++e) + { + mloc[e] = fmaxf(mloc[e], __shfl_xor_sync(0xffffffff, mloc[e], 4)); + mloc[e] = fmaxf(mloc[e], __shfl_xor_sync(0xffffffff, mloc[e], 8)); + mloc[e] = fmaxf(mloc[e], __shfl_xor_sync(0xffffffff, mloc[e], 16)); + } + if (g == 0) + { + s.red[warp][2 * qd] = mloc[0]; + s.red[warp][2 * qd + 1] = mloc[1]; + } + __syncthreads(); + // Every thread reduces its two heads' four warp maxima itself (the same expression, so the same bits): no + // shared row-max stage and its second barrier. +#pragma unroll + for (int e = 0; e < 2; ++e) + { + int const row = 2 * qd + e; + m_row[e] = fmaxf(fmaxf(s.red[0][row], s.red[1][row]), fmaxf(s.red[2][row], s.red[3][row])); + } + // A row with no valid token has m = -inf: its probabilities are ex2(-inf - 0) = 0, never NaN. + float const m_use[2] = {m_row[0] == kNegInf ? 0.F : m_row[0], m_row[1] == kNegInf ? 0.F : m_row[1]}; + float lloc[2] = {0.F, 0.F}; +#pragma unroll + for (int mt = 0; mt < 2; ++mt) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + float const pv = ex2(sacc[mt][i] - m_use[i & 1]); + lloc[i & 1] += pv; + int const t = warp * 32 + mt * 16 + g + 8 * (i >> 1); + s.ps[(2 * qd + (i & 1)) * kPitch + t] = __float2half_rn(pv); + } +#pragma unroll + for (int e = 0; e < 2; ++e) + { + lloc[e] += __shfl_xor_sync(0xffffffff, lloc[e], 4); + lloc[e] += __shfl_xor_sync(0xffffffff, lloc[e], 8); + lloc[e] += __shfl_xor_sync(0xffffffff, lloc[e], 16); + } + __syncthreads(); // row-max reads of red done before red is reused; ps complete + if (g == 0) + { + s.red[warp][2 * qd] = lloc[0]; + s.red[warp][2 * qd + 1] = lloc[1]; + } + // O^T[warp's 32 dims, 8 heads] = V^T P^T over all 128 tokens. +#pragma unroll + for (int kk = 0; kk < kPage; kk += 32) + { + uint32_t b[4]; + ldsm_x4(b, smem_u32(s.ps + (lane & 7) * kPitch + kk + (lane >> 3) * 8)); +#pragma unroll + for (int u = 0; u < 2; ++u) + { +#pragma unroll + for (int mt = 0; mt < 2; ++mt) + { + uint32_t a[4]; + int const t = kk + u * 16 + (lane & 7) + ((lane >> 4) & 1) * 8; + int const d = warp * 32 + mt * 16 + ((lane >> 3) & 1) * 8; + ldsm_x4_t(a, smem_u32(s.vs + t * kPitch + d)); + mma(acc[mt], a, b[2 * u], b[2 * u + 1]); + } + } + } + __syncthreads(); + // Likewise the row sums, in the same order (red is not written again: the merge uses ex / wgt). +#pragma unroll + for (int e = 0; e < 2; ++e) + { + int const row = 2 * qd + e; + l_row[e] = (s.red[0][row] + s.red[1][row]) + (s.red[2][row] + s.red[3][row]); + } + } + + // Normalize; push (head, dim) to CTA 2 * head + dim / 64, slot recv[rank][dim % 64], plus each head's LSE (log2) + // to both of its CTAs. Posted DSMEM stores, made visible by the cluster barrier. + asm volatile("barrier.cluster.wait.aligned;" ::: "memory"); + { + float const vgs = __ldg(p.vgs); + float scale[2], lse[2]; +#pragma unroll + for (int e = 0; e < 2; ++e) + { + if constexpr (ONE_DIV) + { + // Unnormalized: the merge divides once. An inactive CTA pushes m = -inf, l = 0 (weight 0). + scale[e] = 1.F; + lse[e] = m_row[e]; + } + else if constexpr (LINEAR) + { + scale[e] = l_row[e] > 0.F ? 1.0F / l_row[e] : 0.F; + lse[e] = l_row[e] > 0.F ? m_row[e] + lg2(l_row[e]) : kNegInf; + } + else + { + float const lse_i + = active && m_row[e] > kNegInf ? m_row[e] + lg2(ex2(kNegInf - m_row[e]) + l_row[e]) : kNegInf; + scale[e] = lse_i > kNegInf ? ex2(m_row[e] - lse_i) : 0.F; + lse[e] = lse_i; + } + scale[e] = scale[e] * vgs; + } + uint32_t dst[2]; +#pragma unroll + for (int e = 0; e < 2; ++e) + dst[e] = map_cluster(&s.recv[rank][0], (2 * qd + e) * 2 + (warp >> 1)); +#pragma unroll + for (int mt = 0; mt < 2; ++mt) +#pragma unroll + for (int i = 0; i < 4; ++i) + { + int const dl = (warp & 1) * 32 + mt * 16 + g + 8 * (i >> 1); + st_cluster_f32(dst[i & 1] + dl * 4, acc[mt][i] * scale[i & 1]); + } + if (warp == 0 && g == 0) + { +#pragma unroll + for (int e = 0; e < 2; ++e) +#pragma unroll + for (int hf = 0; hf < 2; ++hf) + { + st_cluster_f32(map_cluster(&s.recv_lse[rank], (2 * qd + e) * 2 + hf), lse[e]); + if constexpr (ONE_DIV) + { + st_cluster_f32(map_cluster(&s.recv_l[rank], (2 * qd + e) * 2 + hf), l_row[e]); + } + } + } + } + cluster_sync(); + + // CTA `rank` merges head rank / 2, dims (rank % 2) * 64 + dl; thread = (chunk half, dl). + int const h = static_cast(rank) >> 1; + int const dh = static_cast(rank) & 1; + int const dl = tid & (kHalfDim - 1); + int const hc = tid >> 6; + if constexpr (ONE_DIV) + { + // Thread c < 16 takes chunk c's weight 2^(m_c - max m) (wgt) and its weighted row sum (ex); the output is + // the weighted partials over the weighted row sums, both summed in chunk order, one division. + if (tid < kTopk) + { + float m_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + m_max = fmaxf(m_max, s.recv_lse[c]); + m_max = m_max == kNegInf ? 0.F : m_max; + float const w = ex2(s.recv_lse[tid] - m_max); + s.wgt[tid] = w; + s.ex[tid] = w * s.recv_l[tid]; + } + __syncthreads(); + } + else + { + // The merge weights (exp2 of each chunk's LSE minus the largest, divided by their sum in chunk order) are + // formed once per CTA: thread c < 16 takes chunk c's exponential, then divides it by the sum each of those + // threads forms in the same order. Same values in the same order as every thread forming all 16: bitwise the + // same rows. + if (tid < kTopk) + { + float lse_max = kNegInf; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + lse_max = fmaxf(lse_max, s.recv_lse[c]); + lse_max = lse_max == kNegInf ? 0.F : lse_max; + s.ex[tid] = ex2(s.recv_lse[tid] - lse_max); + } + __syncthreads(); + if (tid < kTopk) + { + float denom = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + denom += s.ex[c]; + denom = denom > 0.F ? denom : 1.F; + s.wgt[tid] = __fdiv_rn(s.ex[tid], denom); + } + __syncthreads(); + } + float o = 0.F; +#pragma unroll + for (int c = 0; c < kTopk / 2; ++c) + { + int const cc = hc * (kTopk / 2) + c; + o += s.recv[cc][dl] * s.wgt[cc]; + } + if (hc) + s.part[dl] = o; + __syncthreads(); + if (!hc) + { + o += s.part[dl]; + if constexpr (ONE_DIV) + { + float den = 0.F; +#pragma unroll + for (int c = 0; c < kTopk; ++c) + den += s.ex[c]; + o = den > 0.F ? __fdiv_rn(o, den) : 0.F; + } + int const col = h * kDim + dh * kHalfDim + dl; + __nv_bfloat16 const ob = __float2bfloat16_rn(o); + p.out[tok * p.out_stride + col] = ob; + if constexpr (MXFP8) + { + float const x = __bfloat162float(ob); + float amax = fabsf(x); +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + amax = fmaxf(amax, __shfl_xor_sync(0xffffffff, amax, off)); + float rcp; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(rcp) : "f"(448.0F)); + float const sfv = amax * rcp; + int const bits = __float_as_int(sfv); + int e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0); + e = min(e, 254); + float const qs = amax != 0.F ? __int_as_float((254 - e) << 23) : 0.F; + float const xq = x * qs; + uint16_t q2; + asm("cvt.rn.satfinite.e4m3x2.f32 %0, %1, %2;" : "=h"(q2) : "f"(0.F), "f"(xq)); + p.q8[static_cast(tok) * (p.num_sf_k_tiles * 128) + col] = static_cast(q2 & 0xFF); + if (lane == 0) + { + int const k_sf = col / 32; + int64_t const off = static_cast(tok / 128) * p.num_sf_k_tiles * 512 + (k_sf / 4) * 512 + + (tok % 32) * 16 + ((tok % 128) / 32) * 4 + k_sf % 4; + p.sf[off] = static_cast(e); + } + } + } +#endif // defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 1000) +} + +template +cudaError_t configure() +{ + auto kernel = m3_tp8_lc8_attn_kernel; + cudaError_t err = cudaFuncSetAttribute( + kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(smemBytes())); + if (err != cudaSuccess) + { + return err; + } + return cudaFuncSetAttribute(kernel, cudaFuncAttributeNonPortableClusterSizeAllowed, 1); +} + +template +cudaError_t launch(Params const& prm, int num_tokens, bool use_pdl, cudaStream_t stream) +{ + // Kernel attributes are set once per process, on the first launch (outside CUDA-graph capture: warmup runs + // every variant eagerly first). + static cudaError_t const configured = configure(); + if (configured != cudaSuccess) + { + return configured; + } + auto kernel = m3_tp8_lc8_attn_kernel; + cudaLaunchConfig_t cfg = {}; + cfg.gridDim = dim3(num_tokens * kTopk); + cfg.blockDim = dim3(kThreads); + cfg.dynamicSmemBytes = smemBytes(); + cfg.stream = stream; + cudaLaunchAttribute attrs[2]; + attrs[0].id = cudaLaunchAttributeClusterDimension; + attrs[0].val.clusterDim.x = kTopk; + attrs[0].val.clusterDim.y = 1; + attrs[0].val.clusterDim.z = 1; + attrs[1].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attrs[1].val.programmaticStreamSerializationAllowed = use_pdl ? 1 : 0; + cfg.attrs = attrs; + cfg.numAttrs = 2; + return cudaLaunchKernelEx(&cfg, kernel, prm); +} + +template +cudaError_t launchPrewait(Params const& prm, int num_tokens, bool prewait, bool use_pdl, cudaStream_t stream) +{ + if (prm.one_div) + { + return prewait ? launch(prm, num_tokens, use_pdl, stream) + : launch(prm, num_tokens, use_pdl, stream); + } + return prewait ? launch(prm, num_tokens, use_pdl, stream) + : launch(prm, num_tokens, use_pdl, stream); +} + +} // namespace + +cudaError_t invokeSparseDecodeLc8m(MinimaxM3Tp8SparseDecodeLc8Params const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, cudaStream_t stream) +{ + if (linearSoftmax) + { + return mxfp8 ? launchPrewait(params, numTokens, prewait, usePdl, stream) + : launchPrewait(params, numTokens, prewait, usePdl, stream); + } + return mxfp8 ? launchPrewait(params, numTokens, prewait, usePdl, stream) + : launchPrewait(params, numTokens, prewait, usePdl, stream); +} + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.h new file mode 100644 index 000000000000..7f59a0f7ff59 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.h @@ -0,0 +1,105 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +//! Operands of the MiniMax-M3 TP8 one-launch long-context sparse decode. Strides are in elements of the pointed-to +//! type. Geometry is fixed: 8 query heads on one KV head, head_dim 128, page 128, top-k 16. +struct MinimaxM3Tp8SparseDecodeLc8Params +{ + uint8_t const* q; // e4m3 [T, 8, 128] + int64_t q_tok, q_head; + uint8_t const* k; // packed e2m1 [pages, 1, 128, 64] + uint8_t const* v; + int64_t k_blk, v_blk; + uint8_t const* ksc; // e4m3 [pages, 1, 128, 8], token-major + uint8_t const* vsc; // e4m3, 4x4 token-quad order + int64_t ksc_blk, vsc_blk; + float const* kgs; + float const* vgs; + int const* topk; // [T, 16] view (kv head 0) + int64_t t_tok, t_k; + int const* block_table; + int64_t bt_stride; + int const* seq_lens; + int dql; + float sm_scale; + __nv_bfloat16* out; // [T, 1024] + int64_t out_stride; + uint8_t* q8; // e4m3 [rows, num_sf_k_tiles * 128] + uint8_t* sf; + int num_sf_k_tiles; + // Optional L2 prefetch of two read-only ranges (the layer's o_proj weight and scales), split across the grid's + // CTAs and issued before the grid dependency: a hint, so the output does not depend on it. nullptr = none. + uint8_t const* pf0; + int64_t pf0_bytes; + uint8_t const* pf1; + int64_t pf1_bytes; + // Optional next-layer KV prefetch: CTA (token, r) records its selected block id in this layer's [rows, 16] id + // buffer (n1_self) and pulls into L2 the K/V pages and scales of the block the next sparse layer's CTA (token, r) + // selected at the previous step (n1_next, from that layer's pools n1_k / n1_v / n1_ksc / n1_vsc, this layer's + // strides). Hints only. nullptr = off. + int* n1_self; + int const* n1_next; + uint8_t const* n1_k; + uint8_t const* n1_v; + uint8_t const* n1_ksc; + uint8_t const* n1_vsc; + int64_t n1_pages; // pages in the next layer's pool; a recorded block outside it is skipped + // Identity selection (short-sequence steps, every request within 16 blocks): CTA r attends block r of its + // token's request when r < min(16, blocks), from the device sequence length; topk is not read. + int identity; + // The step is an identity step (short sequences: no scorer / select ran this step), whether or not this kernel + // derives its blocks from the lengths (identity) or reads the staged table: q is then read after the grid wait, + // because the predecessor is the producer, which releases its dependents at entry, before it writes q. + int identity_step; + // One-division merge (MSA_ONE_DIV_MERGE): the CTAs push unnormalized partials with their row max and sum, and the + // merge divides once per output. Deterministic, a different rounding sequence from the per-chunk normalization. + int one_div; +}; + +//! One cluster of 16 CTAs per query token: CTA r attends the token's r-th selected block with transposed MMAs (the 8 +//! heads fill the MMA n dimension), then CTA r merges (head r / 2, dims (r % 2) * 64 + [0, 64)) of the 16 partials +//! over DSMEM into the BF16 [T, 1024] row (and, when mxfp8, its MXFP8 copy with swizzled UE8M0 scales). Dependents +//! are released right after the grid dependency wait. +//! +//! With prewait, the request's seq_lens entry, its q rows (widened to FP16) and its block-table row (up to 8192 +//! entries, in dynamic shared memory; longer rows are read from global memory) are staged before the grid dependency +//! wait; outputs are bit-identical to prewait=false. +//! +//! Instantiated variants: linearSoftmax x mxfp8 x prewait, all in {false, true}. The device code is SM100-family +//! only; on other architectures the kernel compiles to an empty body, so callers check the device first. +cudaError_t invokeSparseDecodeLc8m(MinimaxM3Tp8SparseDecodeLc8Params const& params, int32_t numTokens, + bool linearSoftmax, bool mxfp8, bool prewait, bool usePdl, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.cu new file mode 100644 index 000000000000..fd101e7f521c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.cu @@ -0,0 +1,58 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +namespace +{ + +constexpr int kThreads = 256; + +__global__ void __launch_bounds__(kThreads) + specOffsetsKernel(int32_t* __restrict__ pos, int32_t const* __restrict__ posOff, int64_t nPos, + int32_t* __restrict__ kv, int32_t const* __restrict__ kvOff, int64_t nKv) +{ + for (int64_t i = threadIdx.x; i < nPos; i += kThreads) + { + pos[i] = static_cast(static_cast(pos[i]) + static_cast(posOff[i])); + } + for (int64_t j = threadIdx.x; j < nKv; j += kThreads) + { + kv[j] = static_cast(static_cast(kv[j]) + static_cast(kvOff[j])); + } +} + +} // namespace + +cudaError_t invokeSpecOffsets(int32_t* pos, int32_t const* posOff, int64_t nPos, int32_t* kv, int32_t const* kvOff, + int64_t nKv, cudaStream_t stream) +{ + specOffsetsKernel<<<1, kThreads, 0, stream>>>(pos, posOff, nPos, kv, kvOff, nKv); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.h new file mode 100644 index 000000000000..3cfd05b73369 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.h @@ -0,0 +1,41 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +//! The overlap scheduler's speculative step correction in one launch: pos[i] += posOff[i] for i < nPos and +//! kv[j] += kvOff[j] for j < nKv (int32, wrapping as the ATen adds do). A plain launch: it is a step-input writer that +//! pre-wait readers rely on (rubin-20261015/PREWAIT_AUDIT.md item b). +cudaError_t invokeSpecOffsets(int32_t* pos, int32_t const* posOff, int64_t nPos, int32_t* kv, int32_t const* kvOff, + int64_t nKv, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.cu new file mode 100644 index 000000000000..b1ab46a36088 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.cu @@ -0,0 +1,111 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +namespace +{ + +constexpr int kThreads = 512; +constexpr int kWarps = kThreads / 32; + +// One warp per copy. Plain (coherent) loads: a device copy may read bytes another warp of this CTA just stored, which +// the CTA barrier between the two kinds makes visible; a non-coherent load could return the stale line. +__device__ __forceinline__ void copyWarp(uint8_t* dst, uint8_t const* src, int64_t bytes, int lane) +{ + auto const d = reinterpret_cast(dst); + auto const s = reinterpret_cast(src); + if (((d | s | static_cast(bytes)) & 15) == 0) + { + auto* d16 = reinterpret_cast(dst); + auto const* s16 = reinterpret_cast(src); + for (int64_t i = lane; i < bytes / 16; i += 32) + { + d16[i] = s16[i]; + } + } + else if (((d | s | static_cast(bytes)) & 3) == 0) + { + auto* d4 = reinterpret_cast(dst); + auto const* s4 = reinterpret_cast(src); + for (int64_t i = lane; i < bytes / 4; i += 32) + { + d4[i] = s4[i]; + } + } + else + { + for (int64_t i = lane; i < bytes; i += 32) + { + dst[i] = src[i]; + } + } +} + +// One CTA: a step's window is a few dozen copies of a few hundred bytes, so the launch is the cost; a CTA barrier +// orders the host copies before the device copies where the parallel scatter needed a second launch. +__global__ void __launch_bounds__(kThreads) stepPrologueKernel(uint8_t* window, int64_t* epoch, int64_t maxDesc) +{ + auto* header = reinterpret_cast(window); + int64_t const numHost = min(header[0], maxDesc); + int64_t const numDevice = min(header[1], maxDesc); + int64_t const* desc = header + 2; + int const warp = threadIdx.x / 32; + int const lane = threadIdx.x % 32; + if (epoch != nullptr && threadIdx.x == 0) + { + *epoch += 1; + } + for (int64_t i = warp; i < numHost; i += kWarps) + { + int64_t const* c = desc + 3 * i; + copyWarp(reinterpret_cast(c[0]), window + c[1], c[2], lane); + } + __syncthreads(); + for (int64_t i = warp; i < numDevice; i += kWarps) + { + // The device copies' slots follow all header[0] host copies, as the parallel scatter indexes them. + int64_t const* c = desc + 3 * (header[0] + i); + copyWarp(reinterpret_cast(c[0]), reinterpret_cast(c[1]), c[2], lane); + } + // Every thread has read the header and the descriptors: consume the window. + __syncthreads(); + if (threadIdx.x < 2) + { + header[threadIdx.x] = 0; + } +} + +} // namespace + +cudaError_t invokeStepPrologue(uint8_t* window, int64_t* epoch, int64_t maxDesc, cudaStream_t stream) +{ + stepPrologueKernel<<<1, kThreads, 0, stream>>>(window, epoch, maxDesc); + return cudaGetLastError(); +} + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.h new file mode 100644 index 000000000000..c125dede4361 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.h @@ -0,0 +1,47 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +//! The decode graph's step-input prologue in one launch (step_inputs.StepInputPack with early_copy): the work of the +//! two parallel scatter launches and the header clear that follow them, in that order. +//! +//! The window is the pack's staging layout (kernels/stepInputsScatter.h): an int64 header `[numHostCopies, +//! numDeviceCopies]`, then one `(dst, src, bytes)` int64 triple per copy, host copies first; a host copy's `src` is a +//! byte offset into the window, a device copy's a device address. Up to `maxDesc` copies of each kind are written (the +//! slots the parallel scatter's grid covers), every host copy before any device copy, so a device copy may read a +//! destination a host copy filled; the copies of one kind must not overlap (the pack issues the window only then). +//! Afterwards the header is zeroed, so a replay with no new upload writes nothing. With `epoch` non-null, it is +//! incremented once. +cudaError_t invokeStepPrologue(uint8_t* window, int64_t* epoch, int64_t maxDesc, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.cu b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.cu new file mode 100644 index 000000000000..71c7f49f7b80 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.cu @@ -0,0 +1,100 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +namespace +{ + +constexpr int kThreads = 256; + +// One packed bf16 pair of gate and up values -> its two bf16 results (the low half from the low halves): the +// instruction sequence of the Triton kernel at limit 7, alpha 1.702, beta 1 (bitwise on SM107): the +// gate clamp as min.bf16x2 before the widening, the up clamp as min.xorsign.abs, (u + 1), (g * -1.702) * log2(e), +// ex2.approx, + 1, div.full 1 / that, s * g, the product, one rounding to bf16. +__device__ __forceinline__ uint32_t swigluOaiBf16x2(uint32_t gate, uint32_t up) +{ + uint32_t r; + asm("{\n\t.reg .b32 g, glim, gl, gh, ul, uh, one, lim, t0, t1, e0, e1, s0, s1, r0, r1;\n\t.reg .b16 hl, hh;\n\t" + "mov.b32 glim, 0x40E040E0;\n\tmin.bf16x2 g, %1, glim;\n\t" + "mov.b32 {hl, hh}, g;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, %2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mov.b32 lim, 0f40E00000;\n\tmin.xorsign.abs.f32 ul, ul, lim;\n\tmin.xorsign.abs.f32 uh, uh, lim;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 ul, ul, one;\n\tadd.rn.f32 uh, uh, one;\n\t" + "mul.rn.f32 t0, gl, 0fBFD9DB23;\n\tmul.rn.f32 t1, gh, 0fBFD9DB23;\n\t" + "mul.rn.f32 t0, t0, 0f3FB8AA3B;\n\tmul.rn.f32 t1, t1, 0f3FB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "add.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 r0, s0, gl;\n\tmul.rn.f32 r1, s1, gh;\n\t" + "mul.rn.f32 r0, ul, r0;\n\tmul.rn.f32 r1, uh, r1;\n\t" + "cvt.rn.bf16x2.f32 %0, r1, r0;\n\t}" + : "=r"(r) + : "r"(gate), "r"(up)); + return r; +} + +template +__global__ void __launch_bounds__(kThreads) + swigluOaiKernel(uint32_t const* __restrict__ x, uint32_t* __restrict__ out, int64_t rows, int64_t halfD) +{ + if constexpr (kPdl) + { + asm volatile("griddepcontrol.wait;" ::: "memory"); + asm volatile("griddepcontrol.launch_dependents;"); + } + int64_t const i = blockIdx.x * static_cast(kThreads) + threadIdx.x; + if (i >= rows * halfD) + { + return; + } + int64_t const row = i / halfD; + int64_t const col = i - row * halfD; + uint32_t const* xr = x + row * 2 * halfD; + out[i] = swigluOaiBf16x2(xr[col], xr[halfD + col]); +} + +} // namespace + +cudaError_t invokeSwigluOai( + uint32_t const* x, uint32_t* out, int64_t rows, int64_t halfD, bool pdl, cudaStream_t stream) +{ + int64_t const n = rows * halfD; + cudaLaunchConfig_t cfg{}; + cfg.gridDim = dim3(static_cast((n + kThreads - 1) / kThreads)); + cfg.blockDim = dim3(kThreads); + cfg.stream = stream; + cudaLaunchAttribute attr[1]; + attr[0].id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr[0].val.programmaticStreamSerializationAllowed = pdl ? 1 : 0; + cfg.attrs = attr; + cfg.numAttrs = 1; + return pdl ? cudaLaunchKernelEx(&cfg, swigluOaiKernel, x, out, rows, halfD) + : cudaLaunchKernelEx(&cfg, swigluOaiKernel, x, out, rows, halfD); +} + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.h b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.h new file mode 100644 index 000000000000..78033e018997 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.h @@ -0,0 +1,43 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM3Sm107Tp8 +{ + +//! The dense MLP's swigluoai activation: out[m, j] = bf16(min(g, 7) * sigmoid(1.702 * min(g, 7)) * (clamp(u, -7, 7) + +//! 1)) with g = x[m, j], u = x[m, d + j] (x bf16 [rows, 2d], out bf16 [rows, d], both contiguous, d even): bit for bit +//! the Triton silu_and_mul kernel at swiglu_limit 7, swiglu_alpha 1.702, swiglu_beta 1. With `pdl` the launch allows +//! programmatic serialization and the grid waits on its dependency, then releases its dependents before it computes +//! (the down GEMV stages its weights during the activation and reads it only after its own grid wait). +cudaError_t invokeSwigluOai( + uint32_t const* x, uint32_t* out, int64_t rows, int64_t halfD, bool pdl, cudaStream_t stream); + +} // namespace minimaxM3Sm107Tp8 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/moe/trtllmGen/runner.cu b/cpp/tensorrt_llm/kernels/moe/trtllmGen/runner.cu index 01ec7600681c..269394b97c0e 100644 --- a/cpp/tensorrt_llm/kernels/moe/trtllmGen/runner.cu +++ b/cpp/tensorrt_llm/kernels/moe/trtllmGen/runner.cu @@ -780,6 +780,15 @@ std::tuple Runner::getWorkspaceSizeInBytes(MoERunnerArgs const return std::make_tuple(workspace_size_fc1, workspace_size_fc2); } +std::string Runner::getConfigName(int64_t configIndex) const +{ + TLLM_CHECK_WITH_INFO(configIndex >= 0 && configIndex < static_cast(mPassingConfigs.size()), + "MoE config index %ld is out of range [0, %zu).", static_cast(configIndex), mPassingConfigs.size()); + auto const& config = mPassingConfigs[configIndex]; + return mPermuteGemm1.getKernelNameFromConfigIndex(static_cast(config.gemm1Config)) + " | " + + mGemm2.getKernelNameFromConfigIndex(static_cast(config.gemm2Config)); +} + std::vector Runner::getValidConfigIndices(int32_t topK, int32_t hiddenSize, int32_t intermediateSize, int32_t numLocalExperts, int32_t numTokens, int32_t validHiddenSize, int32_t validIntermediateSize) const { diff --git a/cpp/tensorrt_llm/kernels/moe/trtllmGen/runner.h b/cpp/tensorrt_llm/kernels/moe/trtllmGen/runner.h index 453a29671f4a..bea86497bbd6 100644 --- a/cpp/tensorrt_llm/kernels/moe/trtllmGen/runner.h +++ b/cpp/tensorrt_llm/kernels/moe/trtllmGen/runner.h @@ -430,6 +430,9 @@ class Runner int32_t numLocalExperts, int32_t numTokens, int32_t validHiddenSize = -1, int32_t validIntermediateSize = -1) const; + //! The kernel names of config ``configIndex`` (its gemm1 and gemm2 kernels): a stable name for tactic caches. + [[nodiscard]] std::string getConfigName(int64_t configIndex) const; + private: void setOpsData(MoERunnerArgs const& args, MoEWorkspace const& workspace, moe::dev::convertsf::Data& convertSfData, moe::dev::activation::Data& activationData, moe::dev::finalize::Data& finalizeData); diff --git a/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp b/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp index 3ec67108cfd8..30c30c17af68 100644 --- a/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp @@ -26,6 +26,7 @@ #include #include +#include #include #include #include @@ -363,7 +364,7 @@ void initBindings(nb::module_& m) .def( "__init__", [](tb::LlmRequest* self, tb::LlmRequest::RequestIdType request_id, - tb::LlmRequest::SizeType32 max_new_tokens, std::vector input_tokens, + tb::LlmRequest::SizeType32 max_new_tokens, nb::handle input_tokens_obj, executor::SamplingConfig sampling_config, bool is_streaming, std::optional prompt_embedding_table, std::optional prompt_vocab_size, @@ -405,6 +406,24 @@ void initBindings(nb::module_& m) auto lora_config_tensor_ptr = makeOptionalTensor(lora_config); auto encoder_input_features_tensor_ptr = makeOptionalTensor(encoder_input_features); + // input_tokens: a 1-D contiguous int32 ndarray is memcpy'd (no per-element + // PyLong cast, O(ISL) under the GIL); anything else uses the sequence cast. + std::vector input_tokens; + nb::ndarray, nb::c_contig> input_tokens_arr; + if (nb::try_cast(input_tokens_obj, input_tokens_arr, /*convert=*/false)) + { + input_tokens.resize(input_tokens_arr.shape(0)); + if (input_tokens_arr.shape(0) > 0) + { + std::memcpy( + input_tokens.data(), input_tokens_arr.data(), input_tokens_arr.shape(0) * sizeof(int32_t)); + } + } + else + { + input_tokens = nb::cast>(input_tokens_obj); + } + new (self) tb::LlmRequest{request_id, max_new_tokens, input_tokens, sampling_config, is_streaming, prompt_embedding_table_tensor_ptr, prompt_vocab_size, multimodal_hashes, multimodal_positions, multimodal_lengths, multimodal_uuids, lora_task_id, lora_weights_tensor_ptr, lora_config_tensor_ptr, diff --git a/cpp/tensorrt_llm/nanobind/executor/request.cpp b/cpp/tensorrt_llm/nanobind/executor/request.cpp index 2275dabe4cea..a906923e6057 100644 --- a/cpp/tensorrt_llm/nanobind/executor/request.cpp +++ b/cpp/tensorrt_llm/nanobind/executor/request.cpp @@ -691,6 +691,20 @@ void initRequestBindings(nb::module_& m) nb::arg("cache_salt") = nb::none() ) // clang-format on .def_prop_ro("input_token_ids", &tle::Request::getInputTokenIds) + .def_prop_ro( + "input_token_ids_np", + [](tle::Request const& self) + { + // Owned int32 copy (one memcpy) for LlmRequest(input_tokens=...); + // skips the list[int] round trip of input_token_ids. + auto const& ids = self.getInputTokenIds(); + auto* buf = new tle::VecTokens(ids); + nb::capsule owner(buf, [](void* p) noexcept { delete static_cast(p); }); + return nb::ndarray>(buf->data(), {buf->size()}, owner); + }, + // The array owns its copy via the capsule; the property default + // (reference_internal) cannot be applied to an owned ndarray. + nb::rv_policy::take_ownership) .def_prop_ro("num_input_tokens", &tle::Request::getNumInputTokens) .def_prop_ro("max_tokens", &tle::Request::getMaxTokens) .def_prop_rw("streaming", &tle::Request::getStreaming, &tle::Request::setStreaming) diff --git a/cpp/tensorrt_llm/thop/CMakeLists.txt b/cpp/tensorrt_llm/thop/CMakeLists.txt index 5b23408cf16c..5f29b51649d8 100644 --- a/cpp/tensorrt_llm/thop/CMakeLists.txt +++ b/cpp/tensorrt_llm/thop/CMakeLists.txt @@ -76,14 +76,23 @@ add_th_common_family( nvfp4MlaKvCacheGatherOp.cpp trtllmGenQKVProcessOp.cpp) -# MiniMax-M3.1 GB300 native dependencies; explicit to avoid duplicate +# MiniMax-M3.1 GB300 / Rubin native dependencies; explicit to avoid duplicate # registrations. target_sources( th_common_attention PRIVATE minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp + minimaxM31/minimaxM31DsparkGlueOp.cpp + minimaxM31/minimaxM31F3ArOp.cpp + minimaxM31/minimaxM31FcShardGemmOp.cpp + minimaxM31/minimaxM31MarkovEpilogueOp.cpp + minimaxM31/minimaxM31MoeFinalizeOp.cpp + minimaxM31/minimaxM31S1bFoldOp.cpp + minimaxM31/minimaxM31S2MoeTailOp.cpp minimaxM31/minimaxM31Sm103CtxNormMxfp8Op.cpp minimaxM31/minimaxM31Sm103SelectPrefillTileOp.cpp minimaxM31/minimaxM31SoftmaxMcOp.cpp + minimaxM31/minimaxM31SoftmaxPathBOp.cpp + minimaxM31/minimaxM31SwigluOaiVecOp.cpp minimaxM31/minimaxM31ToppRenormDetOp.cpp minimaxM31/minimaxM31ToppRenormDetV3Op.cpp minimaxM31/minimaxM31Tp4SparseDecodeLcOp.cpp @@ -108,6 +117,51 @@ target_sources( minimaxM3/minimaxM3Tp8SparseDecodeLc8Op.cpp minimaxM3/stepInputsScatterOp.cpp) +# MiniMax-M3.1 Rubin (sm_107) TP2 / TP4 / TP8 decode ops; their kernels are the +# sm_107-only minimaxM3Sm107Tp{2,4,8} object library. +target_sources( + th_common_attention + PRIVATE minimaxM3/minimaxM3Sm107Tp2DenseQkNormRopeFp8Op.cpp + minimaxM3/minimaxM3Sm107Tp2DraftStepOp.cpp + minimaxM3/minimaxM3Sm107Tp2DrafterRopeFp8KvOp.cpp + minimaxM3/minimaxM3Sm107Tp2FoldTailOp.cpp + minimaxM3/minimaxM3Sm107Tp2FullCtxReduceOp.cpp + minimaxM3/minimaxM3Sm107Tp2McAllreduceNormOp.cpp + minimaxM3/minimaxM3Sm107Tp2ProducerPwOp.cpp + minimaxM3/minimaxM3Sm107Tp2SampledAcceptOp.cpp + minimaxM3/minimaxM3Sm107Tp2SelectLongOp.cpp + minimaxM3/minimaxM3Sm107Tp2SiluAndMulOp.cpp + minimaxM3/minimaxM3Sm107Tp2SparseDecodeLc2Op.cpp + minimaxM3/minimaxM3Sm107Tp2SpecOffsetsOp.cpp + minimaxM3/minimaxM3Sm107Tp2StepPrologueOp.cpp + minimaxM3/minimaxM3Sm107Tp2SwigluOaiOp.cpp + minimaxM3/minimaxM3Sm107Tp4DenseQkNormRopeFp8Op.cpp + minimaxM3/minimaxM3Sm107Tp4DraftStepOp.cpp + minimaxM3/minimaxM3Sm107Tp4DrafterRopeFp8KvOp.cpp + minimaxM3/minimaxM3Sm107Tp4FoldTailOp.cpp + minimaxM3/minimaxM3Sm107Tp4FullCtxReduceOp.cpp + minimaxM3/minimaxM3Sm107Tp4McAllreduceNormOp.cpp + minimaxM3/minimaxM3Sm107Tp4ProducerPwOp.cpp + minimaxM3/minimaxM3Sm107Tp4SampledAcceptOp.cpp + minimaxM3/minimaxM3Sm107Tp4SelectLongOp.cpp + minimaxM3/minimaxM3Sm107Tp4SiluAndMulOp.cpp + minimaxM3/minimaxM3Sm107Tp4SparseDecodeLcOp.cpp + minimaxM3/minimaxM3Sm107Tp4SpecOffsetsOp.cpp + minimaxM3/minimaxM3Sm107Tp4StepPrologueOp.cpp + minimaxM3/minimaxM3Sm107Tp4SwigluOaiOp.cpp + minimaxM3/minimaxM3Sm107Tp8DraftStepOp.cpp + minimaxM3/minimaxM3Sm107Tp8DrafterRopeFp8KvOp.cpp + minimaxM3/minimaxM3Sm107Tp8FoldTailOp.cpp + minimaxM3/minimaxM3Sm107Tp8FullCtxReduceOp.cpp + minimaxM3/minimaxM3Sm107Tp8McAllreduceNormOp.cpp + minimaxM3/minimaxM3Sm107Tp8ProducerPwOp.cpp + minimaxM3/minimaxM3Sm107Tp8SampledAcceptOp.cpp + minimaxM3/minimaxM3Sm107Tp8SiluAndMulOp.cpp + minimaxM3/minimaxM3Sm107Tp8SparseDecodeLc8Op.cpp + minimaxM3/minimaxM3Sm107Tp8SpecOffsetsOp.cpp + minimaxM3/minimaxM3Sm107Tp8StepPrologueOp.cpp + minimaxM3/minimaxM3Sm107Tp8SwigluOaiOp.cpp) + add_th_common_family( th_common_gemm_quant cutlassScaledMM.cpp diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3RouterGemvOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3RouterGemvOp.cpp index 059d29e43d85..6e7a6f1fbef9 100644 --- a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3RouterGemvOp.cpp +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3RouterGemvOp.cpp @@ -43,6 +43,14 @@ void checkSmVersion(char const* op) TORCH_CHECK(sm == 100 || sm == 103, op, " requires SM100 or SM103, got SM", sm); } +// The TF32 tensor-core router also runs on SM107 (Rubin; its kernels are sm_100f-family cubins), there only with +// 16-CTA clusters: one reduction order on every rank (minimaxM3RouterTcCluster16 lets callers fall back otherwise). +void checkSmVersionTc(char const* op) +{ + int const sm = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sm == 100 || sm == 103 || sm == 107, op, " requires SM100, SM103 or SM107, got SM", sm); +} + void checkActivation(torch::Tensor const& x, torch::Tensor const& w, int64_t maxM) { TORCH_CHECK(x.is_cuda() && x.scalar_type() == at::kBFloat16 && x.dim() == 2 && x.stride(1) == 1, @@ -96,7 +104,7 @@ std::vector minimaxM3RouterGemvFp4( std::vector minimaxM3RouterTc( torch::Tensor const& x, torch::Tensor const& w, std::optional const& sfScale, bool pdl) { - checkSmVersion("minimax_m3_router_tc"); + checkSmVersionTc("minimax_m3_router_tc"); TORCH_CHECK(x.is_cuda() && x.scalar_type() == at::kBFloat16 && x.dim() == 2 && x.stride(1) == 1, "x must be a 2-D bf16 CUDA tensor with unit column stride"); int64_t const M = x.size(0), K = x.size(1); @@ -105,6 +113,10 @@ std::vector minimaxM3RouterTc( "minimax_m3_router_tc takes 1..48 rows and a contiguous fp32 [128, K] weight"); TORCH_CHECK(K % 1024 == 0 && (x.stride(0) * 2) % 16 == 0, "K must be a multiple of 1024 with 16-byte rows"); c10::cuda::CUDAGuard const deviceGuard{x.device()}; + TORCH_CHECK(tensorrt_llm::common::getSMVersion() != 107 + || tensorrt_llm::kernels::minimaxM3RouterTcCluster(static_cast(K)) == 16, + "minimax_m3_router_tc on SM107 runs only with 16-CTA clusters; this device cannot co-schedule them (use the " + "stock router)"); auto out = torch::empty({M, w.size(0)}, x.options().dtype(at::kFloat)); torch::Tensor q, sf; uint8_t* qp = nullptr; @@ -130,6 +142,14 @@ std::vector minimaxM3RouterTc( return {out}; } +//! Whether `minimax_m3_router_tc` runs on this device with 16-CTA clusters for hidden size `k` (device-static; the +//! SM107 caller falls back to the stock router otherwise). +bool minimaxM3RouterTcCluster16(int64_t k) +{ + checkSmVersionTc("minimax_m3_router_tc_cluster16"); + return tensorrt_llm::kernels::minimaxM3RouterTcCluster(static_cast(k)) == 16; +} + } // namespace torch_ext TRTLLM_NAMESPACE_END @@ -139,6 +159,7 @@ TORCH_LIBRARY_FRAGMENT(trtllm, m) m.def("minimax_m3_router_gemv(Tensor x, Tensor w, int rows, bool pdl) -> Tensor"); m.def("minimax_m3_router_gemv_fp4(Tensor x, Tensor w, Tensor sf_scale, bool pdl) -> Tensor[]"); m.def("minimax_m3_router_tc(Tensor x, Tensor w, Tensor? sf_scale, bool pdl) -> Tensor[]"); + m.def("minimax_m3_router_tc_cluster16(int k) -> bool", &tensorrt_llm::torch_ext::minimaxM3RouterTcCluster16); } TORCH_LIBRARY_IMPL(trtllm, CUDA, m) diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DenseQkNormRopeFp8Op.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DenseQkNormRopeFp8Op.cpp new file mode 100644 index 000000000000..99c182799cab --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DenseQkNormRopeFp8Op.cpp @@ -0,0 +1,125 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/denseQkNormRopeFp8.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace dqk = tensorrt_llm::kernels::minimaxM3Sm107Tp2::denseQkNormRopeFp8; + +//! QK RMSNorm + partial RoPE of a packed BF16 [tokens, (q + k + v) * 128] row of the MiniMax-M3 TP2 dense layers, +//! returned as a new FP8 E4M3 tensor of the same shape (V copy-cast; Q/K rounded to BF16 before E4M3, as stock). +torch::Tensor minimaxM3Sm107Tp2DenseQkNormRopeFp8(torch::Tensor const& qkv, int64_t numHeadsQ, int64_t numHeadsK, + int64_t numHeadsV, int64_t rotaryDim, double eps, torch::Tensor const& qWeight, torch::Tensor const& kWeight, + double base, bool isNeox, torch::Tensor const& positionIds, bool useGemma) +{ + TORCH_CHECK(qkv.is_cuda() && qkv.is_contiguous() && qkv.dim() == 2 && qkv.scalar_type() == at::kBFloat16); + TORCH_CHECK(qkv.size(1) == (numHeadsQ + numHeadsK + numHeadsV) * dqk::kHeadDim); + TORCH_CHECK(qWeight.scalar_type() == at::kBFloat16 && kWeight.scalar_type() == at::kBFloat16); + TORCH_CHECK(qWeight.numel() == dqk::kHeadDim && kWeight.numel() == dqk::kHeadDim); + TORCH_CHECK(positionIds.scalar_type() == at::kInt && positionIds.numel() == qkv.size(0)); + TORCH_CHECK(rotaryDim > 0 && rotaryDim <= dqk::kHeadDim && rotaryDim % 2 == 0, "unsupported rotary_dim"); + TORCH_CHECK(!isNeox || (rotaryDim * 16) % dqk::kHeadDim == 0, "unsupported rotary_dim for the NeoX pairing"); + at::cuda::CUDAGuard guard(qkv.device()); + auto out = torch::empty_like(qkv, qkv.options().dtype(torch::kFloat8_e4m3fn)); + dqk::launch(qkv.data_ptr(), out.data_ptr(), static_cast(qkv.size(0)), static_cast(numHeadsQ), + static_cast(numHeadsK), static_cast(numHeadsV), static_cast(rotaryDim), static_cast(eps), + qWeight.data_ptr(), kWeight.data_ptr(), static_cast(base), !isNeox, positionIds.data_ptr(), + useGemma, nullptr, at::cuda::getCurrentCUDAStream(qkv.get_device())); + return out; +} + +//! As minimaxM3Sm107Tp2DenseQkNormRopeFp8, and the first numLiveTokens rows' K and V are also stored into the layer's +//! FP8 pool kvBuffers (the HND view [pages, 2, kv heads, tokens per block, 128]) at their int32 cache slots; with +//! qCompact, every row's Q is also stored there compact ([tokens, q heads, 128]). +torch::Tensor minimaxM3Sm107Tp2DenseQkNormRopeFp8Kv(torch::Tensor const& qkv, int64_t numHeadsQ, int64_t numHeadsK, + int64_t numHeadsV, int64_t rotaryDim, double eps, torch::Tensor const& qWeight, torch::Tensor const& kWeight, + double base, bool isNeox, torch::Tensor const& positionIds, bool useGemma, torch::Tensor const& kvBuffers, + torch::Tensor const& outCacheLoc, int64_t numLiveTokens, std::optional const& qCompact) +{ + TORCH_CHECK(qkv.is_cuda() && qkv.is_contiguous() && qkv.dim() == 2 && qkv.scalar_type() == at::kBFloat16); + TORCH_CHECK(qkv.size(1) == (numHeadsQ + numHeadsK + numHeadsV) * dqk::kHeadDim); + TORCH_CHECK(numHeadsK == numHeadsV, "the pool keeps one head count for K and V"); + TORCH_CHECK(qWeight.scalar_type() == at::kBFloat16 && kWeight.scalar_type() == at::kBFloat16); + TORCH_CHECK(qWeight.numel() == dqk::kHeadDim && kWeight.numel() == dqk::kHeadDim); + TORCH_CHECK(positionIds.scalar_type() == at::kInt && positionIds.numel() == qkv.size(0)); + TORCH_CHECK(rotaryDim > 0 && rotaryDim <= dqk::kHeadDim && rotaryDim % 2 == 0, "unsupported rotary_dim"); + TORCH_CHECK(!isNeox || (rotaryDim * 16) % dqk::kHeadDim == 0, "unsupported rotary_dim for the NeoX pairing"); + TORCH_CHECK(kvBuffers.is_cuda() && kvBuffers.scalar_type() == at::kFloat8_e4m3fn && kvBuffers.dim() == 5); + TORCH_CHECK(kvBuffers.size(1) == 2 && kvBuffers.size(2) == numHeadsK && kvBuffers.size(4) == dqk::kHeadDim + && kvBuffers.stride(4) == 1, + "kv_buffers must be the HND pool view [pages, 2, kv heads, tokens per block, 128]"); + TORCH_CHECK(outCacheLoc.is_cuda() && outCacheLoc.scalar_type() == at::kInt && outCacheLoc.is_contiguous()); + TORCH_CHECK(numLiveTokens >= 0 && numLiveTokens <= qkv.size(0) && numLiveTokens <= outCacheLoc.numel()); + at::cuda::CUDAGuard guard(qkv.device()); + auto out = torch::empty_like(qkv, qkv.options().dtype(torch::kFloat8_e4m3fn)); + dqk::KvCacheView view; + view.base = static_cast<__nv_fp8_e4m3*>(kvBuffers.data_ptr()); + view.outCacheLoc = outCacheLoc.data_ptr(); + view.pageStride = kvBuffers.stride(0); + view.kvStride = kvBuffers.stride(1); + view.headStride = kvBuffers.stride(2); + view.tokenStride = kvBuffers.stride(3); + view.tokensPerBlock = static_cast(kvBuffers.size(3)); + view.numLiveTokens = static_cast(numLiveTokens); + if (qCompact.has_value()) + { + TORCH_CHECK(qCompact->is_cuda() && qCompact->is_contiguous() && qCompact->scalar_type() == at::kFloat8_e4m3fn + && qCompact->numel() == qkv.size(0) * numHeadsQ * dqk::kHeadDim, + "q_compact must be a contiguous FP8 [tokens, q heads, 128] tensor"); + view.qCompact = static_cast<__nv_fp8_e4m3*>(qCompact->data_ptr()); + } + dqk::launch(qkv.data_ptr(), out.data_ptr(), static_cast(qkv.size(0)), static_cast(numHeadsQ), + static_cast(numHeadsK), static_cast(numHeadsV), static_cast(rotaryDim), static_cast(eps), + qWeight.data_ptr(), kWeight.data_ptr(), static_cast(base), !isNeox, positionIds.data_ptr(), + useGemma, &view, at::cuda::getCurrentCUDAStream(qkv.get_device())); + return out; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8(Tensor qkv, int num_heads_q, int num_heads_k, int num_heads_v, " + "int rotary_dim, float eps, Tensor q_weight, Tensor k_weight, float base, bool is_neox, Tensor position_ids, " + "bool use_gemma) -> Tensor"); + m.def( + "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv(Tensor qkv, int num_heads_q, int num_heads_k, int num_heads_v, " + "int rotary_dim, float eps, Tensor q_weight, Tensor k_weight, float base, bool is_neox, Tensor position_ids, " + "bool use_gemma, Tensor(a!) kv_buffers, Tensor out_cache_loc, int num_live_tokens, Tensor(b!)? q_compact=None) " + "-> Tensor"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl( + "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2DenseQkNormRopeFp8); + m.impl("minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv", + &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2DenseQkNormRopeFp8Kv); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DraftStepOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DraftStepOp.cpp new file mode 100644 index 000000000000..cc452ef4c776 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DraftStepOp.cpp @@ -0,0 +1,220 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/draftStep.h" + +#include +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace ds = tensorrt_llm::kernels::minimaxM3Sm107Tp2::draftStep; + +using OptTensor = std::optional; + +void checkArch(torch::Tensor const& t) +{ + TORCH_CHECK(t.is_cuda(), "minimax_m3_sm107tp2_draft_step expects CUDA tensors"); + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_sm107tp2_draft_step requires SM100 or newer"); +} + +void check(cudaError_t err) +{ + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, cudaGetErrorString(err)); + } +} + +void checkInt(torch::Tensor const& t, int64_t n, char const* name) +{ + TORCH_CHECK(t.scalar_type() == at::kInt && t.is_contiguous() && t.numel() >= n, "draft_step: ", name, + " must be contiguous int32 with at least ", n, " elements"); +} + +} // namespace + +//! One fused greedy draft pass over `rows` <= 8 requests: the BF16 head (this rank's fragment-ordered shard wf +//! [N / 16, K / 16, 256], or with excOff / excPos / excW / expBase its 12-bit coding, wf uint8 [N * K * 3 / 2]; +//! global ids vocabOffset..) over x[xRows[b]] (or x[b]), the global argmax across the TP ranks through the multicast +//! slots wsMcPtr / wsUc, then draftOut[b, step] = idsNext[b] = token, posNext[b] = posSrc[xRows[b] or b] + 1 and the +//! KV-length bookkeeping (see draftStep.h). +void minimaxM3Sm107Tp2DraftStep(torch::Tensor const& x, OptTensor const& xRows, int64_t rows, torch::Tensor const& wf, + int64_t vocabOffset, torch::Tensor& keys, torch::Tensor& counters, torch::Tensor& wsUc, int64_t wsMcPtr, + int64_t rank, int64_t world, torch::Tensor& draftOut, int64_t step, torch::Tensor& idsNext, + torch::Tensor const& posSrc, torch::Tensor& posNext, torch::Tensor& kvLens, OptTensor const& numAcc, + OptTensor const& seqLens, int64_t draftLen, int64_t numCtas, bool pdl, OptTensor const& outKeys, + OptTensor const& excOff, OptTensor const& excPos, OptTensor const& excW, int64_t expBase, int64_t tileLo, + int64_t tileHi, int64_t totalCtas, OptTensor const& bnd, OptTensor const& bndCnt) +{ + checkArch(x); + for (torch::Tensor const* t : std::initializer_list{ + &wf, &keys, &counters, &wsUc, &draftOut, &idsNext, &posSrc, &posNext, &kvLens}) + { + TORCH_CHECK(t->device() == x.device(), "draft_step: every tensor must be on x's device"); + } + for (OptTensor const* t : std::initializer_list{ + &xRows, &numAcc, &seqLens, &outKeys, &excOff, &excPos, &excW, &bnd, &bndCnt}) + { + TORCH_CHECK(!t->has_value() || (*t)->device() == x.device(), "draft_step: every tensor must be on x's device"); + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + TORCH_CHECK(x.scalar_type() == at::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "draft_step: x must be contiguous BF16 [rows, K]"); + int const K = static_cast(x.size(1)); + TORCH_CHECK(K % 16 == 0, "draft_step: K must be a multiple of 16"); + TORCH_CHECK(rows >= 1 && rows <= ds::kMaxTokens, "draft_step: 1..", ds::kMaxTokens, " rows"); + bool const coded = excOff.has_value(); + TORCH_CHECK(coded == excPos.has_value() && coded == excW.has_value(), + "draft_step: exc_off, exc_pos and exc_w come together"); + int nTiles = 0; + if (coded) + { + TORCH_CHECK(K % 64 == 0 && K / 16 <= 0xFFFF, "draft_step: a coded head needs K a multiple of 64"); + TORCH_CHECK(excOff->scalar_type() == at::kInt && excOff->is_contiguous() && excOff->numel() >= 2, + "draft_step: exc_off must be int32 [tiles + 1]"); + nTiles = static_cast(excOff->numel() - 1); + TORCH_CHECK(wf.scalar_type() == at::kByte && wf.is_contiguous() + && wf.numel() == static_cast(nTiles) * 16 * K * 3 / 2, + "draft_step: a coded wf must be the contiguous uint8 stream of [tiles * 16, K]"); + TORCH_CHECK(excPos->scalar_type() == at::kInt && excPos->is_contiguous(), "draft_step: exc_pos must be int32"); + TORCH_CHECK(excW->scalar_type() == at::kFloat && excW->is_contiguous() && excW->numel() == 2 * excPos->numel(), + "draft_step: exc_w must be fp32 [E, 2]"); + TORCH_CHECK(expBase >= 0 && expBase <= 240, "draft_step: exponent base out of range"); + } + else + { + TORCH_CHECK(wf.scalar_type() == at::kBFloat16 && wf.dim() == 3 && wf.is_contiguous() && wf.size(1) == K / 16 + && wf.size(2) == 256, + "draft_step: wf must be the fragment-ordered BF16 weight [N / 16, K / 16, 256]"); + nTiles = static_cast(wf.size(0)); + } + int64_t const rangeHi = tileHi > 0 ? tileHi : nTiles; + TORCH_CHECK(tileLo >= 0 && tileLo < rangeHi && rangeHi <= nTiles, "draft_step: tile range"); + TORCH_CHECK(nTiles >= 1 && numCtas >= 1 && (rangeHi - tileLo + numCtas - 1) / numCtas <= ds::kMaxTilesPerCta, + "draft_step: too many weight tiles per CTA"); + TORCH_CHECK(totalCtas == 0 || totalCtas >= numCtas, "draft_step: total_ctas"); + TORCH_CHECK(bnd.has_value() == bndCnt.has_value(), "draft_step: bnd and bnd_cnt come together"); + if (bnd.has_value()) + { + TORCH_CHECK(bnd->scalar_type() == at::kFloat && bnd->is_contiguous() && bnd->numel() >= int64_t{nTiles} * 128, + "draft_step: bnd must be zero-filled fp32 [tiles, 32, 4]"); + TORCH_CHECK(bndCnt->scalar_type() == at::kInt && bndCnt->is_contiguous() && bndCnt->numel() >= nTiles, + "draft_step: bnd_cnt must be zero-filled int32 [tiles]"); + // At least one tile per CTA: a tile is then shared by at most two CTAs (two addends in bnd). + TORCH_CHECK(rangeHi - tileLo >= numCtas, "draft_step: an equal-chunk split needs a tile per CTA"); + } + TORCH_CHECK(vocabOffset >= 0 && vocabOffset + 16LL * nTiles <= 0xFFFFFFFFLL, "draft_step: vocab offset"); + if (xRows.has_value()) + { + TORCH_CHECK(xRows->scalar_type() == at::kLong && xRows->is_contiguous() && xRows->numel() >= rows, + "draft_step: x_rows must be contiguous int64 [rows]"); + } + else + { + TORCH_CHECK(x.size(0) >= rows, "draft_step: x must hold a row per request"); + } + TORCH_CHECK(world >= 1 && rank >= 0 && rank < world, "draft_step: rank / world"); + TORCH_CHECK(keys.scalar_type() == at::kLong && keys.numel() >= rows, "draft_step: keys must be int64 [>= rows]"); + TORCH_CHECK(counters.scalar_type() == at::kInt && counters.numel() >= 2, "draft_step: counters must be int32 [2]"); + TORCH_CHECK(wsUc.scalar_type() == at::kLong && wsUc.numel() >= 2 * ds::kMaxTokens * world, + "draft_step: ws_uc must hold [2][8][world] slots"); + TORCH_CHECK(draftOut.scalar_type() == at::kInt && draftOut.dim() == 2 && draftOut.stride(1) == 1 + && draftOut.size(0) >= rows && step >= 0 && step < draftOut.size(1), + "draft_step: draft_out must be int32 [rows, > step] with unit column stride"); + checkInt(idsNext, rows, "ids_next"); + checkInt(posNext, rows, "pos_next"); + checkInt(kvLens, rows, "kv_lens"); + TORCH_CHECK(posSrc.scalar_type() == at::kInt && posSrc.is_contiguous(), "draft_step: pos_src must be int32"); + TORCH_CHECK(numAcc.has_value() == seqLens.has_value(), "draft_step: num_acc and seq_lens come together"); + TORCH_CHECK(!numAcc.has_value() || step == 0, "draft_step: the KV rollback (num_acc) is pass 0's"); + if (numAcc.has_value()) + { + checkInt(*numAcc, rows, "num_acc"); + checkInt(*seqLens, rows, "seq_lens"); + } + if (outKeys.has_value()) + { + TORCH_CHECK(outKeys->scalar_type() == at::kLong && outKeys->is_contiguous() && outKeys->numel() >= rows, + "draft_step: out_keys must be contiguous int64 [rows]"); + } + ds::Params p{}; + p.x = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.x_rows = xRows.has_value() ? reinterpret_cast(xRows->data_ptr()) : nullptr; + p.M = static_cast(rows); + p.wf = reinterpret_cast(wf.data_ptr()); + p.n_tiles = nTiles; + p.vocab_offset = static_cast(vocabOffset); + p.keys = reinterpret_cast(keys.data_ptr()); + p.counters = reinterpret_cast(counters.data_ptr()); + p.ws_uc = reinterpret_cast(wsUc.data_ptr()); + p.ws_mc = reinterpret_cast(wsMcPtr); + p.rank = static_cast(rank); + p.world = static_cast(world); + p.out_keys = outKeys.has_value() ? reinterpret_cast(outKeys->data_ptr()) : nullptr; + p.st = ds::StepArgs{draftOut.data_ptr(), static_cast(draftOut.stride(0)), static_cast(step), + idsNext.data_ptr(), posSrc.data_ptr(), posNext.data_ptr(), kvLens.data_ptr(), + numAcc.has_value() ? numAcc->data_ptr() : nullptr, + seqLens.has_value() ? seqLens->data_ptr() : nullptr, static_cast(draftLen)}; + p.cd = ds::CodedArgs{coded ? excOff->data_ptr() : nullptr, + coded ? reinterpret_cast(excPos->data_ptr()) : nullptr, + coded ? reinterpret_cast(excW->data_ptr()) : nullptr, + static_cast((expBase << 7) | (expBase << 23))}; + p.tile_lo = static_cast(tileLo); + p.tile_hi = static_cast(tileHi); + p.total_ctas = static_cast(totalCtas); + p.bnd = bnd.has_value() ? reinterpret_cast(bnd->data_ptr()) : nullptr; + p.bnd_cnt = bnd.has_value() ? reinterpret_cast(bndCnt->data_ptr()) : nullptr; + check(ds::invokeDraftStep(p, K, static_cast(numCtas), pdl, at::cuda::getCurrentCUDAStream(x.get_device()))); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +// Mutations through the raw multicast address ws_mc_ptr are not visible to the schema: that buffer is owned by the +// Python workspace and never aliased by the op's tensor arguments. +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107tp2_draft_step(Tensor x, Tensor? x_rows, int rows, Tensor wf, int vocab_offset, Tensor(a!) " + "keys, " + "Tensor(b!) counters, Tensor(c!) ws_uc, int ws_mc_ptr, int rank, int world, Tensor(d!) draft_out, int step, " + "Tensor(e!) ids_next, Tensor pos_src, Tensor(f!) pos_next, Tensor(g!) kv_lens, Tensor? num_acc, " + "Tensor(h!)? seq_lens, int draft_len, int num_ctas, bool pdl, Tensor(i!)? out_keys, Tensor? exc_off=None, " + "Tensor? exc_pos=None, Tensor? exc_w=None, int exp_base=0, int tile_lo=0, int tile_hi=0, int total_ctas=0, " + "Tensor(j!)? bnd=None, Tensor(k!)? bnd_cnt=None) " + "-> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp2_draft_step", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2DraftStep); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DrafterRopeFp8KvOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DrafterRopeFp8KvOp.cpp new file mode 100644 index 000000000000..6b154d20d3f2 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2DrafterRopeFp8KvOp.cpp @@ -0,0 +1,103 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace drk = tensorrt_llm::kernels::minimaxM3Sm107Tp2::drafterRopeFp8Kv; + +//! The MiniMax-M3 TP2 drafter layer's generation preprocess (kernels/minimaxM3Sm107Tp2/drafterRopeFp8Kv.h): RoPE on +//! the packed BF16 qkv rows, the E4M3 query into q_out, and the step's K / V rows into the paged E4M3 cache kv_pages +//! ([sub-pages, Hkv, tokens_per_block, 128]) at the sub-pages block_tables ([requests, 2, max blocks]) names. +void minimaxM3Sm107Tp2DrafterRopeFp8Kv(torch::Tensor const& qkv, torch::Tensor const& cosSin, + torch::Tensor const& seqLens, std::optional const& specPositionOffsets, torch::Tensor& kvPages, + torch::Tensor const& blockTables, torch::Tensor& qOut, int64_t qLen, int64_t numHeadsQ, int64_t numHeadsKv, + int64_t rotaryDim, int64_t tokensPerBlock, int64_t ropeMode, bool pdl, bool maxSharedCarveout) +{ + TORCH_CHECK(qkv.is_cuda() && qkv.is_contiguous() && qkv.dim() == 2 && qkv.scalar_type() == at::kBFloat16); + TORCH_CHECK(qkv.size(1) == (numHeadsQ + 2 * numHeadsKv) * drk::kHeadDim, "qkv is not [tokens, (Hq + 2 Hkv) * 128]"); + TORCH_CHECK(qLen >= 1 && qkv.size(0) % qLen == 0, "the rows are not whole passes of q_len rows"); + TORCH_CHECK(rotaryDim > 0 && rotaryDim <= drk::kHeadDim && rotaryDim % 16 == 0, "unsupported rotary_dim"); + TORCH_CHECK(cosSin.is_cuda() && cosSin.is_contiguous() && cosSin.scalar_type() == at::kFloat + && cosSin.numel() % rotaryDim == 0, + "cos_sin must be the fp32 [positions, rotary_dim / 2] (cos, sin) table"); + int64_t const requests = qkv.size(0) / qLen; + TORCH_CHECK(seqLens.is_cuda() && seqLens.scalar_type() == at::kInt && seqLens.numel() >= requests); + if (specPositionOffsets.has_value()) + { + TORCH_CHECK(specPositionOffsets->is_cuda() && specPositionOffsets->is_contiguous() + && specPositionOffsets->scalar_type() == at::kInt && specPositionOffsets->numel() >= requests * qLen); + } + TORCH_CHECK(kvPages.is_cuda() && kvPages.is_contiguous() && kvPages.dim() == 4 + && kvPages.scalar_type() == at::kFloat8_e4m3fn && kvPages.size(1) == numHeadsKv + && kvPages.size(2) == tokensPerBlock && kvPages.size(3) == drk::kHeadDim, + "kv_pages must be the E4M3 [sub-pages, Hkv, tokens_per_block, 128] cache"); + TORCH_CHECK(blockTables.is_cuda() && blockTables.scalar_type() == at::kInt && blockTables.dim() == 3 + && blockTables.size(0) >= requests && blockTables.size(1) == 2 && blockTables.stride(2) == 1, + "block_tables must be the int32 [requests, 2, max blocks] K / V sub-page table"); + TORCH_CHECK(qOut.is_cuda() && qOut.is_contiguous() && qOut.scalar_type() == at::kFloat8_e4m3fn + && qOut.numel() == qkv.size(0) * numHeadsQ * drk::kHeadDim); + c10::cuda::CUDAGuard const deviceGuard{qkv.device()}; + drk::Params params{}; + params.qkv = reinterpret_cast<__nv_bfloat16 const*>(qkv.data_ptr()); + params.cosSin = reinterpret_cast(cosSin.data_ptr()); + params.seqLens = seqLens.data_ptr(); + params.specPositionOffsets = specPositionOffsets.has_value() ? specPositionOffsets->data_ptr() : nullptr; + params.qOut = reinterpret_cast<__nv_fp8_e4m3*>(qOut.data_ptr()); + params.pool = reinterpret_cast<__nv_fp8_e4m3*>(kvPages.data_ptr()); + params.blockTable = blockTables.data_ptr(); + params.blockTableRequestStride = blockTables.stride(0); + params.blockTableKvStride = blockTables.stride(1); + params.numTokens = static_cast(qkv.size(0)); + params.qLen = static_cast(qLen); + params.numHeadsQ = static_cast(numHeadsQ); + params.numHeadsKv = static_cast(numHeadsKv); + params.rotaryDim = static_cast(rotaryDim); + params.tokensPerBlock = static_cast(tokensPerBlock); + params.ropeMode = static_cast(ropeMode); + cudaError_t const err + = drk::launch(params, pdl, maxSharedCarveout, at::cuda::getCurrentCUDAStream(qkv.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107tp2_drafter_rope_fp8_kv: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107tp2_drafter_rope_fp8_kv(Tensor qkv, Tensor cos_sin, Tensor seq_lens, " + "Tensor? spec_position_offsets, Tensor(a!) kv_pages, Tensor block_tables, Tensor(b!) q_out, int q_len, " + "int num_heads_q, int num_heads_kv, int rotary_dim, int tokens_per_block, int rope_mode, bool pdl, bool " + "max_shared_carveout=True) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp2_drafter_rope_fp8_kv", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2DrafterRopeFp8Kv); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2FoldTailOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2FoldTailOp.cpp new file mode 100644 index 000000000000..de42edcc82c0 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2FoldTailOp.cpp @@ -0,0 +1,130 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/foldTail.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kMaxWidth = 32; + +void check2d(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt32 && t.dim() == 2 && t.size(1) <= kMaxWidth, + "minimax_m3_sm107_tp2_fold_tail: ", name, " must be a 2-D int32 CUDA view at most 32 wide"); +} + +void check1d(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt32 && t.dim() == 1 && t.stride(0) == 1, + "minimax_m3_sm107_tp2_fold_tail: ", name, " must be a contiguous 1-D int32 CUDA tensor"); +} + +} // namespace + +//! The speculative step's forward-tail store and next-step staging in one launch (the C++ form of the Triton +//! kernels/m3_fold_tail.py, the same writes). The caller checks the row counts against the buffers (fold_tail). +void minimaxM3Sm107Tp2FoldTail(torch::Tensor const& ctrl, torch::Tensor const& slots, int64_t rows, + torch::Tensor const& srcNew, torch::Tensor& dstNew, torch::Tensor const& srcNext, torch::Tensor& dstNext, + torch::Tensor const& srcDraft, torch::Tensor& dstDraft, torch::Tensor const& srcLens, torch::Tensor& dstLens, + torch::Tensor& ids, torch::Tensor& draftIds, torch::Tensor& posOff, torch::Tensor& kvOff, int64_t tokensPerStep) +{ + TORCH_CHECK(ctrl.is_cuda() && ctrl.scalar_type() == torch::kInt32 && ctrl.numel() >= 1, + "minimax_m3_sm107_tp2_fold_tail: ctrl must be an int32 CUDA tensor"); + TORCH_CHECK(slots.is_cuda() && slots.scalar_type() == torch::kInt64 && slots.dim() == 1 && slots.stride(0) == 1, + "minimax_m3_sm107_tp2_fold_tail: slots must be a contiguous 1-D int64 CUDA tensor"); + TORCH_CHECK(rows > 0 && tokensPerStep > 0 && tokensPerStep <= kMaxWidth, + "minimax_m3_sm107_tp2_fold_tail: rows must be positive and tokens_per_step in [1, 32]"); + check2d(srcNew, "src_new"); + check2d(dstNew, "dst_new"); + check2d(srcNext, "src_next"); + check2d(dstNext, "dst_next"); + check2d(srcDraft, "src_draft"); + check2d(dstDraft, "dst_draft"); + check1d(srcLens, "src_lens"); + check1d(dstLens, "dst_lens"); + check1d(ids, "ids"); + check1d(draftIds, "draft_ids"); + check1d(posOff, "pos_off"); + check1d(kvOff, "kv_off"); + c10::cuda::CUDAGuard const deviceGuard{ctrl.device()}; + + tensorrt_llm::kernels::minimaxM3Sm107Tp2::FoldTailParams p{}; + p.ctrl = ctrl.data_ptr(); + p.slots = slots.data_ptr(); + p.srcNew = srcNew.data_ptr(); + p.sn0 = srcNew.stride(0); + p.sn1 = srcNew.stride(1); + p.snW = static_cast(srcNew.size(1)); + p.dstNew = dstNew.data_ptr(); + p.dn0 = dstNew.stride(0); + p.dn1 = dstNew.stride(1); + p.dnW = static_cast(dstNew.size(1)); + p.srcNext = srcNext.data_ptr(); + p.sx0 = srcNext.stride(0); + p.sx1 = srcNext.stride(1); + p.sxW = static_cast(srcNext.size(1)); + p.dstNext = dstNext.data_ptr(); + p.dx0 = dstNext.stride(0); + p.dx1 = dstNext.stride(1); + p.dxW = static_cast(dstNext.size(1)); + p.srcDraft = srcDraft.data_ptr(); + p.sd0 = srcDraft.stride(0); + p.sd1 = srcDraft.stride(1); + p.sdW = static_cast(srcDraft.size(1)); + p.dstDraft = dstDraft.data_ptr(); + p.dd0 = dstDraft.stride(0); + p.dd1 = dstDraft.stride(1); + p.ddW = static_cast(dstDraft.size(1)); + p.srcLens = srcLens.data_ptr(); + p.dstLens = dstLens.data_ptr(); + p.ids = ids.data_ptr(); + p.draftIds = draftIds.data_ptr(); + p.posOff = posOff.data_ptr(); + p.kvOff = kvOff.data_ptr(); + p.tokensPerStep = static_cast(tokensPerStep); + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp2::invokeFoldTail( + p, static_cast(rows), at::cuda::getCurrentCUDAStream(ctrl.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp2_fold_tail launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp2_fold_tail(Tensor ctrl, Tensor slots, int rows, Tensor src_new, Tensor(a!) dst_new, " + "Tensor src_next, Tensor(b!) dst_next, Tensor src_draft, Tensor(c!) dst_draft, Tensor src_lens, " + "Tensor(d!) dst_lens, Tensor(e!) ids, Tensor(f!) draft_ids, Tensor(g!) pos_off, Tensor(h!) kv_off, " + "int tokens_per_step) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_fold_tail", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2FoldTail); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2FullCtxReduceOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2FullCtxReduceOp.cpp new file mode 100644 index 000000000000..8db408008dc2 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2FullCtxReduceOp.cpp @@ -0,0 +1,74 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/fullCtxReduce.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace fcr = tensorrt_llm::kernels::minimaxM3Sm107Tp2::fullCtxReduce; + +//! Merges the PrimTS split-KV decode scratch (partial_o [B, Hkv, S, R, 128] 16-bit, partial_stats [B, Hkv, S, R] +//! fp32 log2-LSE) of the MiniMax-M3 TP2 full-context decode into out (BF16 [B * SQ, Hkv * hR, 128]). +void minimaxM3Sm107Tp2FullCtxReduce(torch::Tensor const& partialO, torch::Tensor const& partialStats, + torch::Tensor const& seqLensKv, torch::Tensor& out, int64_t seqLenQ, int64_t qTokensPerCta, int64_t tileKv, + int64_t instsKv, bool causal, bool pdl) +{ + TORCH_CHECK(partialO.is_cuda() && partialO.is_contiguous() && partialO.dim() == 5); + TORCH_CHECK(partialO.scalar_type() == at::kHalf || partialO.scalar_type() == at::kBFloat16); + TORCH_CHECK(partialO.size(4) == fcr::kHeadDim && partialO.size(2) <= fcr::kMaxSplits); + TORCH_CHECK(partialStats.scalar_type() == at::kFloat && partialStats.is_contiguous() + && partialStats.sizes() == partialO.sizes().slice(0, 4)); + TORCH_CHECK(seqLensKv.scalar_type() == at::kInt && seqLensKv.numel() >= partialO.size(0)); + int const rows = static_cast(partialO.size(3)); + TORCH_CHECK(seqLenQ >= 1 && rows % seqLenQ == 0); + int const headsPerKv = rows / static_cast(seqLenQ); + TORCH_CHECK(out.scalar_type() == at::kBFloat16 && out.is_contiguous() + && out.numel() == partialO.size(0) * seqLenQ * partialO.size(1) * headsPerKv * fcr::kHeadDim); + c10::cuda::CUDAGuard const deviceGuard{partialO.device()}; + fcr::Params params{partialO.data_ptr(), partialStats.data_ptr(), seqLensKv.data_ptr(), + reinterpret_cast<__nv_bfloat16*>(out.data_ptr()), static_cast(partialO.size(0)), + static_cast(partialO.size(1)), static_cast(partialO.size(2)), rows, static_cast(seqLenQ), + headsPerKv, static_cast(qTokensPerCta), static_cast(tileKv), static_cast(instsKv), causal, + partialO.scalar_type() == at::kBFloat16}; + cudaError_t const err + = fcr::invokeFullCtxReduce(params, pdl, at::cuda::getCurrentCUDAStream(partialO.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp2_fullctx_reduce: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp2_fullctx_reduce(Tensor partial_o, Tensor partial_stats, Tensor seq_lens_kv, " + "Tensor(a!) out, int seq_len_q, int q_tokens_per_cta, int tile_kv, int insts_kv, bool causal, bool pdl) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_fullctx_reduce", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2FullCtxReduce); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2McAllreduceNormOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2McAllreduceNormOp.cpp new file mode 100644 index 000000000000..a0f27263a152 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2McAllreduceNormOp.cpp @@ -0,0 +1,181 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/mcAllreduceNormTp2.h" + +#include +#include +#include + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kMaxTokens = 48; +constexpr int64_t kCluster = 8; +constexpr int64_t kMaxWorld = 8; + +} // namespace + +//! MiniMax-M3 SM107 TP2 multicast all-reduce (stock rounding points: BF16 rank sum, then the residual add) + residual + +//! RMSNorm (+ optional MoE finalize, + optional MXFP8 output). +//! +//! @param x BF16 [M, H] partial, or with `perm` the permuted FC2 output [rows, H]. +//! @param x2 Optional BF16 [M, H] second addend (the shared expert in finalize mode). +//! @param residual BF16 [M, H], 1 <= M <= 48. +//! @param normWeight BF16 [H]. +//! @param wsUc This rank's unicast view of the exchange workspace (int32 words); re-armed in place. +//! @param wsMcPtr Device address of the workspace's multicast mapping. +//! @param flags Int32 per-CTA call-parity counters, at least maxt * 8; updated in place. +//! @param cluster Cluster size; 8 is the only one built. +//! @param perm, scales Finalize mode: int32 [M, topk] permuted-row index and fp32 / bf16 [M, topk] routing weights. +//! @param maxt Token capacity of the workspace bucket (8, 16 or 48). +//! @param quant Also return the MXFP8 copy of the normed rows (E4M3 [M, H], swizzled UE8M0 scales). +//! @param prepushed 0, or 2 / 3 when producers already pushed this rank's rows (see the kernel header). +//! @param opts Epilogue variant; every variant is bit-identical. +//! @param capOut Optional BF16 [>= M, H] view (row stride a multiple of 8, 16-byte aligned) that receives the Eagle3 +//! capture of the updated residual: m3_spec_tail capture_norm with capW [H] and capEps, bit for bit (H 6144; not the +//! drafter boundary, opts bit 16). +//! @return [normed, updated residual] or, with `quant`, [normed, updated residual, E4M3 rows, scale bytes]. +std::vector minimaxM3Sm107Tp2McAllreduceNorm(torch::Tensor const& x, + std::optional const& x2, torch::Tensor const& residual, torch::Tensor const& normWeight, double eps, + torch::Tensor& wsUc, int64_t wsMcPtr, torch::Tensor& flags, int64_t rank, int64_t world, int64_t cluster, bool pdl, + std::optional const& perm, std::optional const& scales, int64_t maxt, bool quant, + int64_t prepushed, int64_t opts, std::optional const& capOut, + std::optional const& capW, double capEps) +{ + int const sm = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sm == 107, "minimax_m3_sm107_tp2_mc_allreduce_norm requires SM107, got SM", sm); + TORCH_CHECK(cluster == kCluster, "minimax_m3_sm107_tp2_mc_allreduce_norm: only cluster 8 is built, got ", cluster); + TORCH_CHECK(tensorrt_llm::kernels::isMinimaxM3Sm107Tp2McAllreduceNormSupported( + static_cast(maxt), static_cast(opts)), + "minimax_m3_sm107_tp2_mc_allreduce_norm: unsupported (bucket ", maxt, ", opts ", opts, ")"); + TORCH_CHECK( + world >= 1 && world <= kMaxWorld && rank >= 0 && rank < world, "invalid rank ", rank, " / world ", world); + + tensorrt_llm::kernels::MinimaxM3Sm107Tp2McAllreduceNormParams p; + if (x2.has_value()) + { + TORCH_CHECK(x2->scalar_type() == at::kBFloat16 && x2->is_contiguous() && x2->sizes() == residual.sizes(), + "x2 must be a contiguous bf16 tensor shaped like residual"); + p.x2 = reinterpret_cast<__nv_bfloat16 const*>(x2->data_ptr()); + } + if (perm.has_value()) + { + // Finalize mode: x is the permuted FC2 output [rows, H]. + TORCH_CHECK(scales.has_value() && perm->scalar_type() == at::kInt && perm->is_contiguous() && perm->dim() == 2 + && perm->size(0) == residual.size(0), + "perm must be a contiguous int32 [M, topk] tensor, with scales"); + TORCH_CHECK(scales->is_contiguous() && scales->numel() == perm->numel() + && (scales->scalar_type() == at::kFloat || scales->scalar_type() == at::kBFloat16), + "scales must be a contiguous fp32 or bf16 tensor with one weight per perm entry"); + TORCH_CHECK(x.dim() == 2 && x.size(1) == residual.size(1), "fc2 must be [rows, H]"); + p.fc2 = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.perm = perm->data_ptr(); + p.scales = scales->data_ptr(); + p.scaleFp32 = scales->scalar_type() == at::kFloat; + p.topk = static_cast(perm->size(1)); + } + TORCH_CHECK(x.is_cuda() && x.scalar_type() == at::kBFloat16 && x.is_contiguous() && x.dim() == 2, + "x must be a contiguous 2-D bf16 CUDA tensor"); + TORCH_CHECK(residual.dim() == 2 && residual.size(0) >= 1 && residual.size(0) <= std::min(kMaxTokens, maxt), + "residual must hold 1..min(48, bucket) rows"); + TORCH_CHECK(residual.scalar_type() == at::kBFloat16 && residual.is_contiguous() + && (perm.has_value() || residual.sizes() == x.sizes()), + "residual must be a contiguous bf16 tensor shaped like x"); + TORCH_CHECK( + normWeight.scalar_type() == at::kBFloat16 && normWeight.numel() == residual.size(1), "norm_w must be bf16 [H]"); + TORCH_CHECK(wsUc.scalar_type() == at::kInt && wsUc.is_cuda(), "ws_uc must be an int32 CUDA tensor"); + TORCH_CHECK(flags.scalar_type() == at::kInt && flags.is_contiguous() && flags.numel() >= maxt * kCluster, + "flags must be a contiguous int32 tensor of at least bucket * 8 counters"); + + int64_t const m = residual.size(0), h = residual.size(1); + TORCH_CHECK(!quant || h % 128 == 0, "MXFP8 output needs H % 128 == 0"); + c10::cuda::CUDAGuard const deviceGuard{residual.device()}; + auto outNormed = torch::empty_like(residual); + auto outResidual = torch::empty_like(residual); + auto outQ = quant ? torch::empty({m, h}, residual.options().dtype(at::kFloat8_e4m3fn)) : torch::Tensor(); + auto outSf = quant ? torch::empty({(m + 127) / 128 * 128 * (h / 32)}, residual.options().dtype(at::kByte)) + : torch::Tensor(); + + p.x = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.numTokens = static_cast(m); + p.hidden = static_cast(h); + p.residual = reinterpret_cast<__nv_bfloat16 const*>(residual.data_ptr()); + p.normWeight = reinterpret_cast<__nv_bfloat16 const*>(normWeight.data_ptr()); + p.eps = static_cast(eps); + p.wsUc = wsUc.data_ptr(); + p.wsMc = reinterpret_cast(wsMcPtr); + p.flags = flags.data_ptr(); + p.rank = static_cast(rank); + p.world = static_cast(world); + p.outNormed = reinterpret_cast<__nv_bfloat16*>(outNormed.data_ptr()); + p.outResidual = reinterpret_cast<__nv_bfloat16*>(outResidual.data_ptr()); + p.outQuant = quant ? reinterpret_cast(outQ.data_ptr()) : nullptr; + p.outSf = quant ? outSf.data_ptr() : nullptr; + p.prepushed = static_cast(prepushed); + if (capOut.has_value()) + { + TORCH_CHECK((opts & 16) == 0, "the capture epilogue is not built for the drafter boundary (opts bit 16)"); + TORCH_CHECK(capOut->scalar_type() == at::kBFloat16 && capOut->dim() == 2 && capOut->size(0) >= m + && capOut->size(1) == h && capOut->stride(1) == 1 && capOut->stride(0) % 8 == 0 + && reinterpret_cast(capOut->data_ptr()) % 16 == 0 && capOut->device() == residual.device(), + "cap_out must be a bf16 [>= M, H] view with unit column stride, a row stride that is a multiple of 8 and " + "16-byte alignment"); + TORCH_CHECK( + capW.has_value() && capW->scalar_type() == at::kBFloat16 && capW->is_contiguous() && capW->numel() == h, + "cap_w must be a contiguous bf16 [H] tensor"); + p.capOut = reinterpret_cast<__nv_bfloat16*>(capOut->data_ptr()); + p.capStride = capOut->stride(0); + p.capWeight = reinterpret_cast<__nv_bfloat16 const*>(capW->data_ptr()); + p.capEps = static_cast(capEps); + } + tensorrt_llm::kernels::invokeMinimaxM3Sm107Tp2McAllreduceNorm(p, static_cast(maxt), + static_cast(opts), pdl, at::cuda::getCurrentCUDAStream(residual.get_device())); + if (quant) + { + return {outNormed, outResidual, outQ, outSf}; + } + return {outNormed, outResidual}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp2_mc_allreduce_norm(Tensor x, Tensor? x2, Tensor residual, Tensor norm_w, float eps, " + "Tensor(a!) ws_uc, int ws_mc_ptr, Tensor(b!) flags, int rank, int world, int cluster, bool pdl, " + "Tensor? perm, Tensor? scales, int maxt, bool quant, int prepushed, int opts, Tensor(c!)? cap_out=None, " + "Tensor? cap_w=None, float cap_eps=0.) -> Tensor[]"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_mc_allreduce_norm", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2McAllreduceNorm); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2ProducerPwOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2ProducerPwOp.cpp new file mode 100644 index 000000000000..ed50e6d10ba2 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2ProducerPwOp.cpp @@ -0,0 +1,185 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/producerPw2.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace pw = tensorrt_llm::kernels::minimaxM3Sm107Tp2::producerPw2; + +constexpr int64_t kHeadDim = 128; +constexpr int64_t kPageSize = 128; + +void checkCuda(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda(), "sm107_tp2_producer_pw: ", name, " must be a CUDA tensor"); +} + +} // namespace + +//! The SM107 TP2 NVFP4 QKV + indexer norm/RoPE/KV-insert producer (same arguments and outputs as +//! minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert at 32 query, 2 KV and 2 index heads, head_dim 128, rotary_dim 64): +//! returns (q FP8 [T, 32, 128], index_q FP8 [T, 2, 128]) and writes the NVFP4 K/V of both KV heads and the FP8 index-K +//! of each token's cache slot. `early` triggers the dependents on entry; `prewait_step_inputs` reads the position ids, +//! the slots and the RoPE rows before the grid-dependency wait (none of them may be written by a PDL predecessor). +std::vector minimaxM3Sm107Tp2ProducerPw(torch::Tensor const& packed, torch::Tensor& kvData, + torch::Tensor& kvScale, torch::Tensor& indexK, torch::Tensor const& outCacheLoc, torch::Tensor const& kvQuantScale, + double eps, torch::Tensor const& qWeight, torch::Tensor const& kWeight, torch::Tensor const& indexQWeight, + torch::Tensor const& indexKWeight, torch::Tensor const& rotaryCosSin, torch::Tensor const& positionIds, bool pdl, + bool early, bool prewaitStepInputs) +{ + checkCuda(packed, "packed"); + for (torch::Tensor const* t : std::initializer_list{&kvData, &kvScale, &indexK, &outCacheLoc, + &kvQuantScale, &qWeight, &kWeight, &indexQWeight, &indexKWeight, &rotaryCosSin, &positionIds}) + { + TORCH_CHECK(t->is_cuda() && t->get_device() == packed.get_device(), + "sm107_tp2_producer_pw: every operand must be on packed's CUDA device"); + } + TORCH_CHECK(reinterpret_cast(packed.data_ptr()) % 8 == 0 + && reinterpret_cast(kvData.data_ptr()) % 2 == 0 + && reinterpret_cast(indexK.data_ptr()) % 4 == 0, + "sm107_tp2_producer_pw: packed and the caches must start at the vector-load / store alignment"); + c10::cuda::CUDAGuard const deviceGuard{packed.device()}; + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_sm107_tp2_producer_pw requires SM100 or newer"); + int64_t const headsQ = pw::kHeadsQ; + int64_t const headsKV = pw::kHeadsKV; + int64_t const headsIndex = pw::kHeadsIndex; + int64_t const totalHeads = headsQ + 2 * headsKV + headsIndex + 1; + TORCH_CHECK(packed.scalar_type() == at::kBFloat16 && packed.dim() == 2 && packed.size(1) == totalHeads * kHeadDim + && packed.is_contiguous(), + "sm107_tp2_producer_pw: packed must be contiguous BF16 [T, 39 * 128]"); + // The kernel's layout (the native producer's checkMinimaxM3Nvfp4HndKVPools at two KV heads): HND pools + // [pages, 2 planes, 2 heads, 128 tokens, 64 data / 8 scale bytes], dense token rows and head blocks, planes and + // pages that do not overlap. + TORCH_CHECK(kvData.scalar_type() == at::kByte && kvData.dim() == 5 && kvData.size(0) > 0 && kvData.size(1) == 2 + && kvData.size(2) == headsKV && kvData.size(3) == kPageSize && kvData.size(4) == kHeadDim / 2, + "sm107_tp2_producer_pw: kv_data must be the NVFP4 pool [pages, 2, 2, 128, 64] uint8"); + TORCH_CHECK(kvScale.scalar_type() == at::kByte && kvScale.dim() == 5 && kvScale.size(0) == kvData.size(0) + && kvScale.size(1) == 2 && kvScale.size(2) == headsKV && kvScale.size(3) == kPageSize + && kvScale.size(4) == kHeadDim / 16, + "sm107_tp2_producer_pw: kv_scale must be the E4M3 scale pool [pages, 2, 2, 128, 8] uint8"); + TORCH_CHECK(kvData.stride(4) == 1 && kvData.stride(3) == kvData.size(4) + && kvData.stride(2) == kvData.size(3) * kvData.stride(3), + "sm107_tp2_producer_pw: kv_data must have dense token rows and head blocks"); + TORCH_CHECK(kvScale.stride(4) == 1 && kvScale.stride(3) == kvScale.size(4) + && kvScale.stride(2) == kvScale.size(3) * kvScale.stride(3), + "sm107_tp2_producer_pw: kv_scale must have dense token rows and head blocks"); + TORCH_CHECK(kvData.stride(1) >= kvData.size(2) * kvData.stride(2) && kvData.stride(0) >= 2 * kvData.stride(1) + && kvScale.stride(1) >= kvScale.size(2) * kvScale.stride(2) && kvScale.stride(0) >= 2 * kvScale.stride(1), + "sm107_tp2_producer_pw: pool planes or pages overlap"); + TORCH_CHECK(kvData.stride(0) % 2 == 0 && kvData.stride(1) % 2 == 0, + "sm107_tp2_producer_pw: kv_data page and plane strides must keep the uint16 stores aligned"); + TORCH_CHECK(indexK.scalar_type() == at::kFloat8_e4m3fn && indexK.dim() == 4 && indexK.size(0) == kvData.size(0) + && indexK.size(1) == 1 && indexK.size(2) == kPageSize && indexK.size(3) == kHeadDim && indexK.stride(3) == 1 + && indexK.stride(2) == kHeadDim && indexK.stride(0) >= kPageSize * kHeadDim && indexK.stride(0) % 4 == 0, + "sm107_tp2_producer_pw: index_k must be FP8 [pages, 1, 128, 128] with a page stride keeping the 32-bit stores " + "aligned"); + int64_t const numTokens = packed.size(0); + TORCH_CHECK( + outCacheLoc.scalar_type() == at::kInt && outCacheLoc.is_contiguous() && outCacheLoc.numel() >= numTokens, + "sm107_tp2_producer_pw: out_cache_loc must be int32 [>= T]"); + TORCH_CHECK( + positionIds.scalar_type() == at::kInt && positionIds.is_contiguous() && positionIds.numel() >= numTokens, + "sm107_tp2_producer_pw: position_ids must be int32 [>= T]"); + TORCH_CHECK(kvQuantScale.scalar_type() == at::kFloat && kvQuantScale.is_contiguous() && kvQuantScale.numel() >= 3, + "sm107_tp2_producer_pw: kv_quant_scale must be fp32 [>= 3]"); + TORCH_CHECK(rotaryCosSin.scalar_type() == at::kFloat && rotaryCosSin.is_contiguous() && rotaryCosSin.dim() == 3 + && rotaryCosSin.size(1) == 2 && rotaryCosSin.size(2) == 32 + && reinterpret_cast(rotaryCosSin.data_ptr()) % 16 == 0, + "sm107_tp2_producer_pw: rotary_cos_sin must be fp32 [positions, 2, 32], 16-byte aligned (read as float4)"); + for (auto const* w : {&qWeight, &kWeight, &indexQWeight, &indexKWeight}) + { + TORCH_CHECK(w->scalar_type() == at::kBFloat16 && w->is_contiguous() && w->numel() == kHeadDim + && reinterpret_cast(w->data_ptr()) % 8 == 0, + "sm107_tp2_producer_pw: norm weights must be BF16 [128], 8-byte aligned (read as 4-value words)"); + } + auto const fp8 = packed.options().dtype(at::kFloat8_e4m3fn); + auto qOut = torch::empty({numTokens, headsQ, kHeadDim}, fp8); + auto indexQOut = torch::empty({numTokens, headsIndex, kHeadDim}, fp8); + if (numTokens == 0) + { + return {qOut, indexQOut}; + } + pw::Params p{}; + p.packed = reinterpret_cast<__nv_bfloat16 const*>(packed.data_ptr()); + p.qOut = reinterpret_cast<__nv_fp8_e4m3*>(qOut.data_ptr()); + p.indexQOut = reinterpret_cast<__nv_fp8_e4m3*>(indexQOut.data_ptr()); + p.kvData = kvData.data_ptr(); + p.kvScale = kvScale.data_ptr(); + p.indexK = reinterpret_cast<__nv_fp8_e4m3*>(indexK.data_ptr()); + p.outCacheLoc = outCacheLoc.data_ptr(); + p.kvQuantScale = kvQuantScale.data_ptr(); + p.dataPageStride = kvData.stride(0); + p.dataPlaneStride = kvData.stride(1); + p.dataHeadStride = kvData.stride(2); + p.dataTokenStride = kvData.stride(3); + p.scalePageStride = kvScale.stride(0); + p.scalePlaneStride = kvScale.stride(1); + p.scaleHeadStride = kvScale.stride(2); + p.indexPageStride = indexK.stride(0); + p.indexTokenStride = indexK.stride(2); + p.numPages = kvData.size(0); + p.numTokens = static_cast(numTokens); + p.eps = static_cast(eps); + p.qWeight = reinterpret_cast<__nv_bfloat16 const*>(qWeight.data_ptr()); + p.kWeight = reinterpret_cast<__nv_bfloat16 const*>(kWeight.data_ptr()); + p.indexQWeight = reinterpret_cast<__nv_bfloat16 const*>(indexQWeight.data_ptr()); + p.indexKWeight = reinterpret_cast<__nv_bfloat16 const*>(indexKWeight.data_ptr()); + p.rotaryCosSin = rotaryCosSin.data_ptr(); + p.positionIds = positionIds.data_ptr(); + cudaError_t const err = pw::invokeProducerPw2(p, static_cast(numTokens), pdl, early, prewaitStepInputs, + at::cuda::getCurrentCUDAStream(packed.get_device())); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "sm107_tp2_producer_pw launch: ", cudaGetErrorString(err)); + } + return {qOut, indexQOut}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp2_producer_pw(Tensor packed, Tensor(a!) kv_data, Tensor(b!) kv_scale, Tensor(c!) index_k, " + "Tensor out_cache_loc, Tensor kv_quant_scale, float eps, Tensor q_weight, Tensor k_weight, " + "Tensor index_q_weight, Tensor index_k_weight, Tensor rotary_cos_sin, Tensor position_ids, bool pdl, " + "bool early, bool prewait_step_inputs=False) -> Tensor[]"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_producer_pw", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2ProducerPw); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SampledAcceptOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SampledAcceptOp.cpp new file mode 100644 index 000000000000..415042d5c082 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SampledAcceptOp.cpp @@ -0,0 +1,110 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/sampledAccept.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace sa = tensorrt_llm::kernels::minimaxM3Sm107Tp2::sampledAccept; + +//! uint64 words of the sampled-accept exchange workspace for `world` ranks. +int64_t minimaxM3Sm107Tp2SampledAcceptWsWords(int64_t world) +{ + return sa::sampledAcceptWsWords(static_cast(world)); +} + +//! Sampled (temperature + top-p) verification from this rank's BF16 logits shard [rows, Vs], rows = nreq * +//! (draft_len + 1) <= 48: the sampled token of every row (tokens), then the strict acceptance against draft (accTokens, +//! numAcc, gatherIds, optional nextNew column 0). Per-row Philox (seeds, offsets); the exchange goes through the +//! multicast workspace wsMcPtr / wsUc with the (finished rows, call) counters. +void minimaxM3Sm107Tp2SampledAccept(torch::Tensor const& logits, int64_t vocabOffset, torch::Tensor const& temperatures, + torch::Tensor const& topPs, torch::Tensor const& seeds, torch::Tensor const& offsets, torch::Tensor& wsUc, + int64_t wsMcPtr, torch::Tensor& counters, torch::Tensor& tokens, int64_t rank, int64_t world, + torch::Tensor const& draft, torch::Tensor const& batchIdx, torch::Tensor& accTokens, torch::Tensor& numAcc, + torch::Tensor& gatherIds, int64_t forceTotal, std::optional const& nextNew, + std::optional const& forceVec, std::optional const& extraAttempts, bool usePdl, + bool prewaitParams, bool earlyTrigger, bool ilpWalk) +{ + c10::cuda::CUDAGuard const deviceGuard{logits.device()}; + sa::Args a; + a.logits = reinterpret_cast<__nv_bfloat16 const*>(logits.data_ptr()); + a.ld = logits.stride(0); + a.rows = static_cast(logits.size(0)); + a.Vs = static_cast(logits.size(1)); + TORCH_CHECK(a.Vs <= sa::kMaxVs && a.rows <= sa::kMaxRows && world <= sa::kMaxWorld + && static_cast(vocabOffset) + a.Vs <= (1LL << sa::kIdBits)); + a.vocab_offset = static_cast(vocabOffset); + a.temperatures = temperatures.data_ptr(); + a.top_ps = topPs.data_ptr(); + a.seeds = reinterpret_cast(seeds.data_ptr()); + a.offsets = reinterpret_cast(offsets.data_ptr()); + a.ws_uc = reinterpret_cast(wsUc.data_ptr()); + a.ws_mc = reinterpret_cast(wsMcPtr); + a.counters = reinterpret_cast(counters.data_ptr()); + a.tokens = tokens.data_ptr(); + a.extra_attempts = extraAttempts.has_value() ? extraAttempts->data_ptr() : nullptr; + a.rank = static_cast(rank); + a.world = static_cast(world); + a.draft = draft.data_ptr(); + a.batch_idx = batchIdx.data_ptr(); + a.acc_tokens = accTokens.data_ptr(); + a.num_acc = numAcc.data_ptr(); + a.gather_ids = reinterpret_cast(gatherIds.data_ptr()); + a.draft_len = static_cast(draft.size(1)); + a.nreq = static_cast(draft.size(0)); + a.force_total = static_cast(forceTotal); + a.next_new = nextNew.has_value() ? nextNew->data_ptr() : nullptr; + a.next_stride = nextNew.has_value() ? static_cast(nextNew->stride(0)) : 0; + a.force_vec = forceVec.has_value() ? forceVec->data_ptr() : nullptr; + a.prewait_params = prewaitParams ? 1 : 0; + a.early_trigger = earlyTrigger ? 1 : 0; + a.ilp_walk = ilpWalk ? 1 : 0; + cudaError_t const err = sa::invokeSampledAccept(a, usePdl, at::cuda::getCurrentCUDAStream(logits.get_device())); + TORCH_CHECK(err == cudaSuccess, "m3_sampled_accept launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +// ws_mc_ptr is the multicast address of ws_uc's mapping: the op writes through it, which the schema cannot show. +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + // No tensor arguments, so no dispatch key: registered with the schema as a catch-all kernel. + m.def("minimax_m3_sm107tp2_sampled_accept_ws_words(int world) -> int", + &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2SampledAcceptWsWords); + m.def( + "minimax_m3_sm107tp2_sampled_accept(Tensor logits, int vocab_offset, Tensor temperatures, Tensor top_ps, " + "Tensor seeds, Tensor offsets, Tensor(a!) ws_uc, int ws_mc_ptr, Tensor(b!) counters, Tensor(c!) tokens, " + "int rank, int world, Tensor draft, Tensor batch_idx, Tensor(d!) acc_tokens, Tensor(e!) num_acc, " + "Tensor(f!) gather_ids, int force_total, Tensor(g!)? next_new, Tensor? force_vec, " + "Tensor(h!)? extra_attempts, bool use_pdl, bool prewait_params=True, bool early_trigger=False, " + "bool ilp_walk=False) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp2_sampled_accept", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2SampledAccept); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SelectLongOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SelectLongOp.cpp new file mode 100644 index 000000000000..b54249840c93 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SelectLongOp.cpp @@ -0,0 +1,85 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/selectLong2.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! Top-16 block select for MiniMax-M3 decode rows of up to 9216 blocks, into a caller-owned table. +//! +//! @param scores Float32 CUDA tensor [num_kv_heads, num_blocks, total_q], any strides. +//! @param nValidBlocks Contiguous int32 CUDA tensor with at least total_q elements. +//! @param out Int32 CUDA tensor of logical shape [total_q, num_kv_heads, 16] with unit stride in the last dim (the +//! head-major backing the target hands over is a permuted view). Every entry is written. +//! @param triggerPhase Where the dependent grid is released without earlyTrigger: 0 at this grid's exit, 1 right +//! after every score of the row is loaded (both after this grid's dependency wait). +void minimaxM3Sm107Tp2SelectLong(torch::Tensor const& scores, torch::Tensor const& nValidBlocks, torch::Tensor& out, + int64_t initBlocks, int64_t localBlocks, bool usePdl, bool earlyTrigger, int64_t triggerPhase) +{ + TORCH_CHECK(scores.is_cuda() && scores.scalar_type() == torch::kFloat32 && scores.dim() == 3, + "minimax_m3_sm107_tp2_select_long: scores must be a 3-D float32 CUDA tensor [num_kv_heads, num_blocks, " + "total_q]"); + TORCH_CHECK(nValidBlocks.is_cuda() && nValidBlocks.scalar_type() == torch::kInt32 && nValidBlocks.is_contiguous() + && nValidBlocks.numel() >= scores.size(2), + "minimax_m3_sm107_tp2_select_long: n_valid_blocks must be contiguous int32 CUDA with at least total_q " + "elements"); + int64_t const numKvHeads = scores.size(0); + int64_t const numBlocks = scores.size(1); + int64_t const totalQ = scores.size(2); + TORCH_CHECK(numBlocks <= tensorrt_llm::kernels::minimaxM3Sm107Tp2::kSelectLong2MaxBlocks, + "minimax_m3_sm107_tp2_select_long supports at most ", + tensorrt_llm::kernels::minimaxM3Sm107Tp2::kSelectLong2MaxBlocks, " blocks, got ", numBlocks); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kInt32 && out.dim() == 3 && out.size(0) == totalQ + && out.size(1) == numKvHeads && out.size(2) == 16 && out.stride(2) == 1, + "minimax_m3_sm107_tp2_select_long: out must be int32 [total_q, num_kv_heads, 16] with unit last-dim stride"); + + c10::cuda::CUDAGuard const deviceGuard{scores.device()}; + auto const stream = at::cuda::getCurrentCUDAStream(scores.get_device()); + cudaError_t const err = tensorrt_llm::kernels::minimaxM3Sm107Tp2::invokeSelectLong2(scores.data_ptr(), + scores.stride(0), scores.stride(1), scores.stride(2), nValidBlocks.data_ptr(), out.data_ptr(), + out.stride(0), out.stride(1), static_cast(numKvHeads), static_cast(numBlocks), + static_cast(totalQ), static_cast(initBlocks), static_cast(localBlocks), usePdl, + earlyTrigger, static_cast(triggerPhase), stream); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "minimax_m3_sm107_tp2_select_long launch failed: ", cudaGetErrorString(err)); + } +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp2_select_long(Tensor scores, Tensor n_valid_blocks, Tensor(a!) out, int init_blocks, " + "int local_blocks, bool use_pdl=False, bool early_trigger=False, int trigger_phase=0) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_select_long", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2SelectLong); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SiluAndMulOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SiluAndMulOp.cpp new file mode 100644 index 000000000000..52f60de1b7ca --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SiluAndMulOp.cpp @@ -0,0 +1,66 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/siluAndMul.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The drafter MLP's plain SwiGLU (see kernels/minimaxM3Sm107Tp2/siluAndMul.h): x bf16 [rows, 2d] -> out bf16 [rows, +//! d]. +void minimaxM3Sm107Tp2SiluAndMul(torch::Tensor const& x, torch::Tensor& out) +{ + TORCH_CHECK(x.is_cuda() && x.scalar_type() == torch::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "minimax_m3_sm107_tp2_silu_and_mul: x must be a contiguous 2-D bfloat16 CUDA tensor"); + int64_t const rows = x.size(0); + int64_t const d = x.size(1) / 2; + TORCH_CHECK(x.size(1) % 16 == 0 && reinterpret_cast(x.data_ptr()) % 16 == 0, + "minimax_m3_sm107_tp2_silu_and_mul: x rows must be 16-byte aligned with a half width a multiple of 8"); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.is_contiguous() + && out.size(0) == rows && out.size(1) == d && reinterpret_cast(out.data_ptr()) % 16 == 0, + "minimax_m3_sm107_tp2_silu_and_mul: out must be a contiguous, 16-byte aligned bfloat16 [rows, d]"); + TORCH_CHECK(rows <= 65535, "minimax_m3_sm107_tp2_silu_and_mul: at most 65535 rows"); + if (rows == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto const err + = tensorrt_llm::kernels::minimaxM3Sm107Tp2::invokeSiluAndMul(reinterpret_cast(x.data_ptr()), + reinterpret_cast(out.data_ptr()), rows, d, at::cuda::getCurrentCUDAStream(x.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp2_silu_and_mul launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp2_silu_and_mul(Tensor x, Tensor(a!) out) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_silu_and_mul", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2SiluAndMul); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SparseDecodeLc2Op.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SparseDecodeLc2Op.cpp new file mode 100644 index 000000000000..1103d72f5e21 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SparseDecodeLc2Op.cpp @@ -0,0 +1,298 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/sparseDecodeLc2Split.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kHeadsPerKv = 16; +constexpr int64_t kHeadDim = 128; +constexpr int64_t kTopK = 16; + +void checkBytes(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, "minimax_m3_sm107_tp2_sparse_decode_lc: ", name, + " must be a one-byte CUDA view (uint8 / e4m3 / packed e2m1)"); +} + +// TP2WIDE_LC2_SPLIT: LC2 without its thread-block cluster. The per-(row, block) +// grid stores LC2's normalized partials and LSEs to a workspace and a second grid merges them in LC2's slot order: +// the same rows, scales and N1 ids bit for bit, -0.63 ms per K7 c56 step and -0.74 per D192 step at 60 layers. +// Default ON; TRTLLM_M31_DISABLE_TP2WIDE_LC2_SPLIT=1 runs the cluster kernel. Read once per process. +bool lc2SplitEnabled() +{ + static bool const enabled = []() + { + char const* v = std::getenv("TRTLLM_M31_DISABLE_TP2WIDE_LC2_SPLIT"); + return !(v != nullptr && v[0] == '1' && v[1] == '\0'); + }(); + return enabled; +} + +// The split's workspace, one per device, sized for the widest LC2 step (448 rows) or the step, whichever is larger. +// It only grows outside CUDA-graph capture, and every buffer stays alive for the process (captured graphs keep their +// pointers); a captured step that would need more than is allocated runs the cluster kernel (the same values). +void* lc2SplitWorkspace(int device, int64_t numTokens, int64_t kvHeads, size_t* bytes, cudaStream_t stream) +{ + constexpr int64_t kWidestStep = 448; + size_t const need = tensorrt_llm::kernels::minimaxM3Sm107Tp2SparseDecodeLcSplitWorkspaceBytes( + static_cast(numTokens), static_cast(kvHeads)); + static std::mutex mutex; + static std::unordered_map> buffers; + std::lock_guard const lock(mutex); + auto& held = buffers[device]; + if (!held.empty() && static_cast(held.back().numel()) >= need) + { + *bytes = static_cast(held.back().numel()); + return held.back().data_ptr(); + } + cudaStreamCaptureStatus status = cudaStreamCaptureStatusNone; + if (cudaStreamIsCapturing(stream, &status) != cudaSuccess || status != cudaStreamCaptureStatusNone) + { + return nullptr; + } + size_t const size = std::max(need, + tensorrt_llm::kernels::minimaxM3Sm107Tp2SparseDecodeLcSplitWorkspaceBytes( + static_cast(kWidestStep), static_cast(kvHeads))); + held.push_back( + torch::empty({static_cast(size)}, torch::dtype(torch::kUInt8).device(torch::kCUDA, device))); + *bytes = size; + return held.back().data_ptr(); +} + +} // namespace + +//! One-launch MiniMax-M3 long-context sparse decode over the selected NVFP4 blocks, SM107 TP2 geometry (16 query +//! heads per KV head, 2 KV heads per rank, head_dim 128, page 128, top-k 16). +//! +//! @param q E4M3 bytes [T, 16 * H, 128], unit stride in the last dim. +//! @param k, v Packed E2M1 bytes [pages, H, 128, 64] (the per-layer HND views). +//! @param kBlockScale, vBlockScale E4M3 bytes [pages, H, 128, 8]: K token-major, V in 4x4 token-quad order. +//! @param kGlobalScale, vGlobalScale One-element float32 per-tensor dequant scales. +//! @param topk Int32 [H, T, 16] ascending selected block ids with -1 tail (the head-major selector table). +//! @param out BF16 [T, H * 2048], written. q8 / sf, when given, receive the MXFP8 copy and its swizzled UE8M0 scales. +//! @param n1Ids Optional int32 [H, rows, 16] (unit last-dim stride): this layer's selected block ids at its previous +//! decode step, read to prefetch those blocks into L2 before the grid dependency and overwritten with this step's; +//! query rows at or past `rows` skip it. Initialize to -1. A hint: `out` does not depend on it. +void sparseDecodeLc2Impl(bool allowSplit, torch::Tensor const& q, torch::Tensor const& k, torch::Tensor const& v, + torch::Tensor const& kBlockScale, torch::Tensor const& vBlockScale, torch::Tensor const& kGlobalScale, + torch::Tensor const& vGlobalScale, torch::Tensor const& topk, torch::Tensor const& blockTable, + torch::Tensor const& seqLens, torch::Tensor& out, std::optional q8, std::optional sf, + int64_t numSfKTiles, int64_t decodeQueryLen, double smScale, bool linearSoftmax, bool usePdl, bool prewait, + std::optional n1Ids) +{ + TORCH_CHECK(q.is_cuda(), "minimax_m3_sm107_tp2_sparse_decode_lc: q must be a CUDA tensor"); + c10::cuda::CUDAGuard const deviceGuard{q.device()}; + TORCH_CHECK(at::cuda::getCurrentDeviceProperties()->major >= 10, + "minimax_m3_sm107_tp2_sparse_decode_lc requires an SM100-family GPU"); + checkBytes(q, "q"); + checkBytes(k, "k"); + checkBytes(v, "v"); + checkBytes(kBlockScale, "k_block_scale"); + checkBytes(vBlockScale, "v_block_scale"); + TORCH_CHECK(k.dim() == 4 && k.size(2) == 128 && k.size(3) == kHeadDim / 2 && k.stride(3) == 1 + && v.sizes() == k.sizes() && v.strides() == k.strides(), + "minimax_m3_sm107_tp2_sparse_decode_lc: k and v must be [pages, H, 128, 64] with matching strides"); + int64_t const kvHeads = k.size(1); + TORCH_CHECK(kBlockScale.dim() == 4 && kBlockScale.size(1) == kvHeads && kBlockScale.size(3) == 8 + && kBlockScale.stride(3) == 1 && kBlockScale.stride(2) == 8 && vBlockScale.sizes() == kBlockScale.sizes() + && vBlockScale.strides() == kBlockScale.strides(), + "minimax_m3_sm107_tp2_sparse_decode_lc: block scales must be contiguous [pages, H, 128, 8] per page"); + TORCH_CHECK(q.dim() == 3 && q.size(1) == kHeadsPerKv * kvHeads && q.size(2) == kHeadDim && q.stride(2) == 1, + "minimax_m3_sm107_tp2_sparse_decode_lc: q must be [T, 16 * H, 128] with unit last-dim stride"); + TORCH_CHECK(kGlobalScale.scalar_type() == torch::kFloat32 && vGlobalScale.scalar_type() == torch::kFloat32, + "minimax_m3_sm107_tp2_sparse_decode_lc: global scales must be float32"); + TORCH_CHECK(topk.scalar_type() == torch::kInt32 && topk.dim() == 3 && topk.size(0) == kvHeads + && topk.size(1) == q.size(0) && topk.size(2) == kTopK, + "minimax_m3_sm107_tp2_sparse_decode_lc: topk must be int32 [H, T, 16]"); + TORCH_CHECK(blockTable.scalar_type() == torch::kInt32 && seqLens.scalar_type() == torch::kInt32, + "minimax_m3_sm107_tp2_sparse_decode_lc: block_table and seq_lens must be int32"); + TORCH_CHECK(out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.size(0) == q.size(0) + && out.size(1) == kvHeads * kHeadsPerKv * kHeadDim && out.stride(1) == 1, + "minimax_m3_sm107_tp2_sparse_decode_lc: out must be bfloat16 [T, H * 2048] with unit last-dim stride"); + TORCH_CHECK( + q8.has_value() == sf.has_value(), "minimax_m3_sm107_tp2_sparse_decode_lc: pass both q8 and sf, or neither"); + bool const mxfp8 = q8.has_value(); + if (mxfp8) + { + checkBytes(*q8, "q8"); + checkBytes(*sf, "sf"); + } + + tensorrt_llm::kernels::MinimaxM3Sm107Tp2SparseDecodeLcParams prm{}; + prm.q = reinterpret_cast(q.data_ptr()); + prm.q_tok = q.stride(0); + prm.q_head = q.stride(1); + prm.k = reinterpret_cast(k.data_ptr()); + prm.v = reinterpret_cast(v.data_ptr()); + prm.k_blk = k.stride(0); + prm.v_blk = v.stride(0); + prm.k_h = k.stride(1); + prm.v_h = v.stride(1); + prm.ksc = reinterpret_cast(kBlockScale.data_ptr()); + prm.vsc = reinterpret_cast(vBlockScale.data_ptr()); + prm.ksc_blk = kBlockScale.stride(0); + prm.vsc_blk = vBlockScale.stride(0); + prm.ksc_h = kBlockScale.stride(1); + prm.vsc_h = vBlockScale.stride(1); + prm.kgs = kGlobalScale.data_ptr(); + prm.vgs = vGlobalScale.data_ptr(); + prm.topk = topk.data_ptr(); + prm.t_h = topk.stride(0); + prm.t_tok = topk.stride(1); + prm.t_k = topk.stride(2); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.seq_lens = seqLens.data_ptr(); + prm.dql = static_cast(decodeQueryLen); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(out.data_ptr()); + prm.out_stride = out.stride(0); + prm.q8 = mxfp8 ? reinterpret_cast(q8->data_ptr()) : nullptr; + prm.sf = mxfp8 ? reinterpret_cast(sf->data_ptr()) : nullptr; + prm.num_sf_k_tiles = static_cast(numSfKTiles); + prm.num_kv_heads = static_cast(kvHeads); + if (n1Ids.has_value()) + { + TORCH_CHECK(n1Ids->is_cuda() && n1Ids->scalar_type() == torch::kInt32 && n1Ids->dim() == 3 + && n1Ids->size(0) == kvHeads && n1Ids->size(2) == kTopK && n1Ids->stride(2) == 1 + && n1Ids->stride(1) == kTopK, + "minimax_m3_sm107_tp2_sparse_decode_lc: n1_ids must be int32 [H, rows, 16] with dense rows"); + prm.n1_ids = n1Ids->data_ptr(); + prm.n1_h = n1Ids->stride(0); + prm.n1_rows = static_cast(n1Ids->size(1)); + prm.n1_pages = k.size(0); + } + + auto const stream = at::cuda::getCurrentCUDAStream(q.get_device()); + if (allowSplit && lc2SplitEnabled() && q.size(0) >= 33 && q.size(0) <= 448) + { + size_t bytes = 0; + if (void* ws = lc2SplitWorkspace(q.get_device(), q.size(0), kvHeads, &bytes, stream)) + { + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM3Sm107Tp2SparseDecodeLcSplit( + prm, static_cast(q.size(0)), linearSoftmax, mxfp8, prewait, usePdl, ws, bytes, stream); + TORCH_CHECK(err == cudaSuccess, + "minimax_m3_sm107_tp2_sparse_decode_lc (split) launch failed: ", cudaGetErrorString(err)); + static std::atomic reported{false}; + if (!reported.exchange(true)) + { + std::fprintf(stderr, "[M31 LC2_SPLIT] taken device=%d rows=%lld workspace_bytes=%zu\n", q.get_device(), + static_cast(q.size(0)), bytes); + } + return; + } + } + cudaError_t const err = tensorrt_llm::kernels::invokeMinimaxM3Sm107Tp2SparseDecodeLc( + prm, static_cast(q.size(0)), linearSoftmax, mxfp8, prewait, usePdl, stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp2_sparse_decode_lc launch failed: ", cudaGetErrorString(err)); + static std::atomic reported{false}; + if (!reported.exchange(true)) + { + std::fprintf( + stderr, "[M31 LC2_CLUSTER] taken device=%d rows=%lld\n", q.get_device(), static_cast(q.size(0))); + } +} + +//! Reserve persistent scratch before the executor profiles memory and sizes its KV pool. +int64_t prepareSparseDecodeLcSplit(torch::Tensor const& anchor, int64_t kvHeads) +{ + TORCH_CHECK(anchor.is_cuda(), "LC2 workspace anchor must be CUDA"); + TORCH_CHECK(kvHeads == 2, "LC2 split requires the TP2 geometry"); + if (!lc2SplitEnabled()) + { + return 0; + } + c10::cuda::CUDAGuard guard(anchor.device()); + auto const stream = at::cuda::getCurrentCUDAStream(anchor.get_device()); + size_t bytes = 0; + TORCH_CHECK(lc2SplitWorkspace(anchor.get_device(), 448, kvHeads, &bytes, stream) != nullptr, + "LC2 split workspace must be prepared outside graph capture"); + std::fprintf(stderr, "[M31 LC2_SPLIT] reserved device=%d workspace_bytes=%zu\n", anchor.get_device(), bytes); + return static_cast(bytes); +} + +//! The serving op: LC2-split unless TRTLLM_M31_DISABLE_TP2WIDE_LC2_SPLIT=1 (or a captured step outgrows the +//! workspace), else the cluster kernel. Same arguments, outputs and N1 contract either way. +void minimaxM3Sm107Tp2SparseDecodeLc(torch::Tensor const& q, torch::Tensor const& k, torch::Tensor const& v, + torch::Tensor const& kBlockScale, torch::Tensor const& vBlockScale, torch::Tensor const& kGlobalScale, + torch::Tensor const& vGlobalScale, torch::Tensor const& topk, torch::Tensor const& blockTable, + torch::Tensor const& seqLens, torch::Tensor& out, std::optional q8, std::optional sf, + int64_t numSfKTiles, int64_t decodeQueryLen, double smScale, bool linearSoftmax, bool usePdl, bool prewait, + std::optional n1Ids) +{ + sparseDecodeLc2Impl(true, q, k, v, kBlockScale, vBlockScale, kGlobalScale, vGlobalScale, topk, blockTable, seqLens, + out, q8, sf, numSfKTiles, decodeQueryLen, smScale, linearSoftmax, usePdl, prewait, n1Ids); +} + +//! The cluster kernel (LC2) whatever the switch: the reference the split is tested against. +void minimaxM3Sm107Tp2SparseDecodeLcCluster(torch::Tensor const& q, torch::Tensor const& k, torch::Tensor const& v, + torch::Tensor const& kBlockScale, torch::Tensor const& vBlockScale, torch::Tensor const& kGlobalScale, + torch::Tensor const& vGlobalScale, torch::Tensor const& topk, torch::Tensor const& blockTable, + torch::Tensor const& seqLens, torch::Tensor& out, std::optional q8, std::optional sf, + int64_t numSfKTiles, int64_t decodeQueryLen, double smScale, bool linearSoftmax, bool usePdl, bool prewait, + std::optional n1Ids) +{ + sparseDecodeLc2Impl(false, q, k, v, kBlockScale, vBlockScale, kGlobalScale, vGlobalScale, topk, blockTable, seqLens, + out, q8, sf, numSfKTiles, decodeQueryLen, smScale, linearSoftmax, usePdl, prewait, n1Ids); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp2_sparse_decode_lc_prepare(Tensor anchor, int kv_heads) -> int"); + m.def( + "minimax_m3_sm107_tp2_sparse_decode_lc(Tensor q, Tensor k, Tensor v, Tensor k_block_scale, " + "Tensor v_block_scale, Tensor k_global_scale, Tensor v_global_scale, Tensor topk, Tensor block_table, " + "Tensor seq_lens, Tensor(a!) out, Tensor(b!)? q8, Tensor(c!)? sf, int num_sf_k_tiles, int decode_query_len, " + "float sm_scale, bool linear_softmax, bool use_pdl, bool prewait, Tensor(d!)? n1_ids=None) -> ()"); + m.def( + "minimax_m3_sm107_tp2_sparse_decode_lc_cluster(Tensor q, Tensor k, Tensor v, Tensor k_block_scale, " + "Tensor v_block_scale, Tensor k_global_scale, Tensor v_global_scale, Tensor topk, Tensor block_table, " + "Tensor seq_lens, Tensor(a!) out, Tensor(b!)? q8, Tensor(c!)? sf, int num_sf_k_tiles, int decode_query_len, " + "float sm_scale, bool linear_softmax, bool use_pdl, bool prewait, Tensor(d!)? n1_ids=None) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_sparse_decode_lc_prepare", &tensorrt_llm::torch_ext::prepareSparseDecodeLcSplit); + m.impl("minimax_m3_sm107_tp2_sparse_decode_lc", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2SparseDecodeLc); + m.impl("minimax_m3_sm107_tp2_sparse_decode_lc_cluster", + &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2SparseDecodeLcCluster); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SpecOffsetsOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SpecOffsetsOp.cpp new file mode 100644 index 000000000000..87b6ae599742 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SpecOffsetsOp.cpp @@ -0,0 +1,74 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/specOffsets.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +void check(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt32 && t.dim() == 1 && t.stride(0) == 1, + "minimax_m3_sm107_tp2_spec_offsets: ", name, " must be a contiguous 1-D int32 CUDA tensor"); +} + +} // namespace + +//! pos += pos_off and kv += kv_off (equal lengths per pair) in one plain launch +//! (kernels/minimaxM3Sm107Tp2/specOffsets.h). +void minimaxM3Sm107Tp2SpecOffsets( + torch::Tensor& pos, torch::Tensor const& posOff, torch::Tensor& kv, torch::Tensor const& kvOff) +{ + check(pos, "pos"); + check(posOff, "pos_off"); + check(kv, "kv"); + check(kvOff, "kv_off"); + TORCH_CHECK(pos.numel() == posOff.numel() && kv.numel() == kvOff.numel(), + "minimax_m3_sm107_tp2_spec_offsets: each tensor and its offsets must have the same length"); + if (pos.numel() == 0 && kv.numel() == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{pos.device()}; + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp2::invokeSpecOffsets(pos.data_ptr(), + posOff.data_ptr(), pos.numel(), kv.data_ptr(), kvOff.data_ptr(), kv.numel(), + at::cuda::getCurrentCUDAStream(pos.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp2_spec_offsets launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp2_spec_offsets(Tensor(a!) pos, Tensor pos_off, Tensor(b!) kv, Tensor kv_off) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_spec_offsets", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2SpecOffsets); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2StepPrologueOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2StepPrologueOp.cpp new file mode 100644 index 000000000000..9e8229ba9e16 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2StepPrologueOp.cpp @@ -0,0 +1,68 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/stepPrologue.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The decode graph's step-input prologue in one launch (see kernels/minimaxM3Sm107Tp2/stepPrologue.h). +//! +//! @param window The prologue's device window: contiguous uint8, at least the header and `maxDesc` descriptors (the +//! pack's descriptor area; the header's two counts add up to at most that). Its destinations are raw device addresses +//! the caller guarantees are live. +//! @param epoch Optional one-element int64 step counter, incremented once. +void minimaxM3Sm107Tp2StepPrologue(torch::Tensor& window, std::optional epoch, int64_t maxDesc) +{ + TORCH_CHECK(window.is_cuda() && window.scalar_type() == torch::kUInt8 && window.is_contiguous(), + "minimax_m3_sm107_tp2_step_prologue: window must be a contiguous uint8 CUDA tensor"); + TORCH_CHECK(maxDesc > 0 && window.numel() >= 16 + 24 * maxDesc, + "minimax_m3_sm107_tp2_step_prologue: window is shorter than its header and max_desc descriptors"); + c10::cuda::CUDAGuard const deviceGuard{window.device()}; + int64_t* epochPtr = nullptr; + if (epoch.has_value()) + { + TORCH_CHECK(epoch->is_cuda() && epoch->scalar_type() == torch::kInt64 && epoch->numel() == 1 + && epoch->device() == window.device(), + "minimax_m3_sm107_tp2_step_prologue: epoch must be a one-element int64 tensor on the window's device"); + epochPtr = epoch->data_ptr(); + } + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp2::invokeStepPrologue( + window.data_ptr(), epochPtr, maxDesc, at::cuda::getCurrentCUDAStream(window.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp2_step_prologue launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp2_step_prologue(Tensor(a!) window, Tensor(b!)? epoch, int max_desc) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_step_prologue", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2StepPrologue); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SwigluOaiOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SwigluOaiOp.cpp new file mode 100644 index 000000000000..2839f6036ffc --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp2SwigluOaiOp.cpp @@ -0,0 +1,65 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp2/swigluOai.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The dense MLP's swigluoai activation (see kernels/minimaxM3Sm107Tp2/swigluOai.h): x bf16 [rows, 2d] -> out bf16 +//! [rows, d], PDL-launched with an early release when `pdl`. +void minimaxM3Sm107Tp2SwigluOai(torch::Tensor const& x, torch::Tensor& out, bool pdl) +{ + TORCH_CHECK(x.is_cuda() && x.scalar_type() == torch::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "minimax_m3_sm107_tp2_swiglu_oai: x must be a contiguous 2-D bfloat16 CUDA tensor"); + int64_t const rows = x.size(0); + int64_t const d = x.size(1) / 2; + TORCH_CHECK(x.size(1) % 4 == 0 && reinterpret_cast(x.data_ptr()) % 4 == 0, + "minimax_m3_sm107_tp2_swiglu_oai: x rows must be 4-byte aligned with an even half width"); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.is_contiguous() + && out.size(0) == rows && out.size(1) == d && reinterpret_cast(out.data_ptr()) % 4 == 0, + "minimax_m3_sm107_tp2_swiglu_oai: out must be a contiguous, 4-byte aligned bfloat16 [rows, d]"); + if (rows == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp2::invokeSwigluOai( + reinterpret_cast(x.data_ptr()), reinterpret_cast(out.data_ptr()), rows, d / 2, pdl, + at::cuda::getCurrentCUDAStream(x.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp2_swiglu_oai launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp2_swiglu_oai(Tensor x, Tensor(a!) out, bool pdl) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp2_swiglu_oai", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp2SwigluOai); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DenseQkNormRopeFp8Op.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DenseQkNormRopeFp8Op.cpp new file mode 100644 index 000000000000..d92f2c1cd29b --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DenseQkNormRopeFp8Op.cpp @@ -0,0 +1,128 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/denseQkNormRopeFp8.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace dqk = tensorrt_llm::kernels::minimaxM3Sm107Tp4::denseQkNormRopeFp8; + +//! QK RMSNorm + partial RoPE of a packed BF16 [tokens, (q + k + v) * 128] row of the MiniMax-M3 TP4 dense layers, +//! returned as a new FP8 E4M3 tensor of the same shape (V copy-cast; Q/K rounded to BF16 before E4M3, as stock). +torch::Tensor minimaxM3Sm107Tp4DenseQkNormRopeFp8(torch::Tensor const& qkv, int64_t numHeadsQ, int64_t numHeadsK, + int64_t numHeadsV, int64_t rotaryDim, double eps, torch::Tensor const& qWeight, torch::Tensor const& kWeight, + double base, bool isNeox, torch::Tensor const& positionIds, bool useGemma) +{ + TORCH_CHECK(qkv.is_cuda() && qkv.is_contiguous() && qkv.dim() == 2 && qkv.scalar_type() == at::kBFloat16); + TORCH_CHECK(qkv.size(1) == (numHeadsQ + numHeadsK + numHeadsV) * dqk::kHeadDim); + TORCH_CHECK(qWeight.scalar_type() == at::kBFloat16 && kWeight.scalar_type() == at::kBFloat16); + TORCH_CHECK(qWeight.numel() == dqk::kHeadDim && kWeight.numel() == dqk::kHeadDim); + TORCH_CHECK(positionIds.scalar_type() == at::kInt && positionIds.numel() == qkv.size(0)); + TORCH_CHECK(rotaryDim > 0 && rotaryDim <= dqk::kHeadDim && rotaryDim % 2 == 0, "unsupported rotary_dim"); + TORCH_CHECK(!isNeox || (rotaryDim * 16) % dqk::kHeadDim == 0, "unsupported rotary_dim for the NeoX pairing"); + at::cuda::CUDAGuard guard(qkv.device()); + auto out = torch::empty_like(qkv, qkv.options().dtype(torch::kFloat8_e4m3fn)); + cudaError_t const err + = dqk::launch(qkv.data_ptr(), out.data_ptr(), static_cast(qkv.size(0)), static_cast(numHeadsQ), + static_cast(numHeadsK), static_cast(numHeadsV), static_cast(rotaryDim), + static_cast(eps), qWeight.data_ptr(), kWeight.data_ptr(), static_cast(base), !isNeox, + positionIds.data_ptr(), useGemma, nullptr, at::cuda::getCurrentCUDAStream(qkv.get_device())); + TORCH_CHECK(err == cudaSuccess, "dense_qk_norm_rope_fp8 launch: ", cudaGetErrorString(err)); + return out; +} + +//! As minimaxM3Sm107Tp4DenseQkNormRopeFp8, and the first numLiveTokens rows' K and V are also stored into the layer's +//! FP8 pool kvBuffers (the HND view [pages, 2, kv heads, tokens per block, 128]) at their int32 cache slots; with +//! qCompact, every row's Q is also stored there compact ([tokens, q heads, 128]). +torch::Tensor minimaxM3Sm107Tp4DenseQkNormRopeFp8Kv(torch::Tensor const& qkv, int64_t numHeadsQ, int64_t numHeadsK, + int64_t numHeadsV, int64_t rotaryDim, double eps, torch::Tensor const& qWeight, torch::Tensor const& kWeight, + double base, bool isNeox, torch::Tensor const& positionIds, bool useGemma, torch::Tensor const& kvBuffers, + torch::Tensor const& outCacheLoc, int64_t numLiveTokens, std::optional const& qCompact) +{ + TORCH_CHECK(qkv.is_cuda() && qkv.is_contiguous() && qkv.dim() == 2 && qkv.scalar_type() == at::kBFloat16); + TORCH_CHECK(qkv.size(1) == (numHeadsQ + numHeadsK + numHeadsV) * dqk::kHeadDim); + TORCH_CHECK(numHeadsK == numHeadsV, "the pool keeps one head count for K and V"); + TORCH_CHECK(qWeight.scalar_type() == at::kBFloat16 && kWeight.scalar_type() == at::kBFloat16); + TORCH_CHECK(qWeight.numel() == dqk::kHeadDim && kWeight.numel() == dqk::kHeadDim); + TORCH_CHECK(positionIds.scalar_type() == at::kInt && positionIds.numel() == qkv.size(0)); + TORCH_CHECK(rotaryDim > 0 && rotaryDim <= dqk::kHeadDim && rotaryDim % 2 == 0, "unsupported rotary_dim"); + TORCH_CHECK(!isNeox || (rotaryDim * 16) % dqk::kHeadDim == 0, "unsupported rotary_dim for the NeoX pairing"); + TORCH_CHECK(kvBuffers.is_cuda() && kvBuffers.scalar_type() == at::kFloat8_e4m3fn && kvBuffers.dim() == 5); + TORCH_CHECK(kvBuffers.size(1) == 2 && kvBuffers.size(2) == numHeadsK && kvBuffers.size(4) == dqk::kHeadDim + && kvBuffers.stride(4) == 1, + "kv_buffers must be the HND pool view [pages, 2, kv heads, tokens per block, 128]"); + TORCH_CHECK(outCacheLoc.is_cuda() && outCacheLoc.scalar_type() == at::kInt && outCacheLoc.is_contiguous()); + TORCH_CHECK(numLiveTokens >= 0 && numLiveTokens <= qkv.size(0) && numLiveTokens <= outCacheLoc.numel()); + at::cuda::CUDAGuard guard(qkv.device()); + auto out = torch::empty_like(qkv, qkv.options().dtype(torch::kFloat8_e4m3fn)); + dqk::KvCacheView view; + view.base = static_cast<__nv_fp8_e4m3*>(kvBuffers.data_ptr()); + view.outCacheLoc = outCacheLoc.data_ptr(); + view.pageStride = kvBuffers.stride(0); + view.kvStride = kvBuffers.stride(1); + view.headStride = kvBuffers.stride(2); + view.tokenStride = kvBuffers.stride(3); + view.tokensPerBlock = static_cast(kvBuffers.size(3)); + view.numLiveTokens = static_cast(numLiveTokens); + if (qCompact.has_value()) + { + TORCH_CHECK(qCompact->is_cuda() && qCompact->is_contiguous() && qCompact->scalar_type() == at::kFloat8_e4m3fn + && qCompact->numel() == qkv.size(0) * numHeadsQ * dqk::kHeadDim, + "q_compact must be a contiguous FP8 [tokens, q heads, 128] tensor"); + view.qCompact = static_cast<__nv_fp8_e4m3*>(qCompact->data_ptr()); + } + cudaError_t const err + = dqk::launch(qkv.data_ptr(), out.data_ptr(), static_cast(qkv.size(0)), static_cast(numHeadsQ), + static_cast(numHeadsK), static_cast(numHeadsV), static_cast(rotaryDim), + static_cast(eps), qWeight.data_ptr(), kWeight.data_ptr(), static_cast(base), !isNeox, + positionIds.data_ptr(), useGemma, &view, at::cuda::getCurrentCUDAStream(qkv.get_device())); + TORCH_CHECK(err == cudaSuccess, "dense_qk_norm_rope_fp8_kv launch: ", cudaGetErrorString(err)); + return out; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107tp4_dense_qk_norm_rope_fp8(Tensor qkv, int num_heads_q, int num_heads_k, int num_heads_v, " + "int rotary_dim, float eps, Tensor q_weight, Tensor k_weight, float base, bool is_neox, Tensor position_ids, " + "bool use_gemma) -> Tensor"); + m.def( + "minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv(Tensor qkv, int num_heads_q, int num_heads_k, int num_heads_v, " + "int rotary_dim, float eps, Tensor q_weight, Tensor k_weight, float base, bool is_neox, Tensor position_ids, " + "bool use_gemma, Tensor(a!) kv_buffers, Tensor out_cache_loc, int num_live_tokens, Tensor(b!)? q_compact=None) " + "-> Tensor"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp4_dense_qk_norm_rope_fp8", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4DenseQkNormRopeFp8); + m.impl("minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv", + &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4DenseQkNormRopeFp8Kv); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DraftStepOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DraftStepOp.cpp new file mode 100644 index 000000000000..b0a49e8a29fb --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DraftStepOp.cpp @@ -0,0 +1,215 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/draftStep.h" +#include "tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailParams.h" + +#include +#include +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace ds = tensorrt_llm::kernels::minimaxM3Sm107Tp4::draftStep; + +using OptTensor = std::optional; + +void checkArch(torch::Tensor const& t) +{ + TORCH_CHECK(t.is_cuda(), "minimax_m3_sm107tp4_draft_step expects CUDA tensors"); + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_sm107tp4_draft_step requires SM100 or newer"); +} + +void check(cudaError_t err) +{ + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, cudaGetErrorString(err)); + } +} + +void checkInt(torch::Tensor const& t, int64_t n, char const* name) +{ + TORCH_CHECK(t.scalar_type() == at::kInt && t.is_contiguous() && t.numel() >= n, "draft_step: ", name, + " must be contiguous int32 with at least ", n, " elements"); +} + +} // namespace + +//! One fused greedy draft pass over `rows` <= 8 requests: the BF16 head (this rank's fragment-ordered shard wf +//! [N / 16, K / 16, 256], or with excOff / excPos / excW / expBase its 12-bit coding, wf uint8 [N * K * 3 / 2]; +//! global ids vocabOffset..) over x[xRows[b]] (or x[b]), the global argmax across the TP ranks through the multicast +//! slots wsMcPtr / wsUc, then draftOut[b, step] = idsNext[b] = token, posNext[b] = posSrc[xRows[b] or b] + 1 and the +//! KV-length bookkeeping (see draftStep.h). +void minimaxM3Sm107Tp4DraftStep(torch::Tensor const& x, OptTensor const& xRows, int64_t rows, torch::Tensor const& wf, + int64_t vocabOffset, torch::Tensor& keys, torch::Tensor& counters, torch::Tensor& wsUc, int64_t wsMcPtr, + int64_t rank, int64_t world, torch::Tensor& draftOut, int64_t step, torch::Tensor& idsNext, + torch::Tensor const& posSrc, torch::Tensor& posNext, torch::Tensor& kvLens, OptTensor const& numAcc, + OptTensor const& seqLens, int64_t draftLen, int64_t numCtas, bool pdl, OptTensor const& outKeys, + OptTensor const& excOff, OptTensor const& excPos, OptTensor const& excW, int64_t expBase, + std::optional> fold, int64_t foldRows, int64_t foldTokensPerStep, bool fixedOrder) +{ + checkArch(x); + for (torch::Tensor const* t : std::initializer_list{ + &wf, &keys, &counters, &wsUc, &draftOut, &idsNext, &posSrc, &posNext, &kvLens}) + { + TORCH_CHECK(t->device() == x.device(), "draft_step: every tensor must be on x's device"); + } + for (OptTensor const* t : + std::initializer_list{&xRows, &numAcc, &seqLens, &outKeys, &excOff, &excPos, &excW}) + { + TORCH_CHECK(!t->has_value() || (*t)->device() == x.device(), "draft_step: every tensor must be on x's device"); + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + TORCH_CHECK(x.scalar_type() == at::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "draft_step: x must be contiguous BF16 [rows, K]"); + int const K = static_cast(x.size(1)); + TORCH_CHECK(K % 16 == 0, "draft_step: K must be a multiple of 16"); + TORCH_CHECK(rows >= 1 && rows <= ds::kMaxTokens, "draft_step: 1..", ds::kMaxTokens, " rows"); + bool const coded = excOff.has_value(); + TORCH_CHECK(coded == excPos.has_value() && coded == excW.has_value(), + "draft_step: exc_off, exc_pos and exc_w come together"); + int nTiles = 0; + if (coded) + { + TORCH_CHECK(K % 64 == 0 && K / 16 <= 0xFFFF, "draft_step: a coded head needs K a multiple of 64"); + TORCH_CHECK(excOff->scalar_type() == at::kInt && excOff->is_contiguous() && excOff->numel() >= 2, + "draft_step: exc_off must be int32 [tiles + 1]"); + nTiles = static_cast(excOff->numel() - 1); + TORCH_CHECK(wf.scalar_type() == at::kByte && wf.is_contiguous() + && wf.numel() == static_cast(nTiles) * 16 * K * 3 / 2, + "draft_step: a coded wf must be the contiguous uint8 stream of [tiles * 16, K]"); + TORCH_CHECK(excPos->scalar_type() == at::kInt && excPos->is_contiguous(), "draft_step: exc_pos must be int32"); + TORCH_CHECK(excW->scalar_type() == at::kFloat && excW->is_contiguous() && excW->numel() == 2 * excPos->numel(), + "draft_step: exc_w must be fp32 [E, 2]"); + TORCH_CHECK(expBase >= 0 && expBase <= 240, "draft_step: exponent base out of range"); + } + else + { + TORCH_CHECK(wf.scalar_type() == at::kBFloat16 && wf.dim() == 3 && wf.is_contiguous() && wf.size(1) == K / 16 + && wf.size(2) == 256, + "draft_step: wf must be the fragment-ordered BF16 weight [N / 16, K / 16, 256]"); + nTiles = static_cast(wf.size(0)); + } + TORCH_CHECK(nTiles >= 1 && numCtas >= 1 && (nTiles + numCtas - 1) / numCtas <= ds::kMaxTilesPerCta, + "draft_step: too many weight tiles per CTA"); + TORCH_CHECK(vocabOffset >= 0 && vocabOffset + 16LL * nTiles <= 0xFFFFFFFFLL, "draft_step: vocab offset"); + if (xRows.has_value()) + { + TORCH_CHECK(xRows->scalar_type() == at::kLong && xRows->is_contiguous() && xRows->numel() >= rows, + "draft_step: x_rows must be contiguous int64 [rows]"); + } + else + { + TORCH_CHECK(x.size(0) >= rows, "draft_step: x must hold a row per request"); + } + TORCH_CHECK(world >= 1 && rank >= 0 && rank < world, "draft_step: rank / world"); + TORCH_CHECK(keys.scalar_type() == at::kLong && keys.numel() >= rows, "draft_step: keys must be int64 [>= rows]"); + TORCH_CHECK(counters.scalar_type() == at::kInt && counters.numel() >= 2, "draft_step: counters must be int32 [2]"); + TORCH_CHECK(wsUc.scalar_type() == at::kLong && wsUc.numel() >= 2 * ds::kMaxTokens * world, + "draft_step: ws_uc must hold [2][8][world] slots"); + TORCH_CHECK(draftOut.scalar_type() == at::kInt && draftOut.dim() == 2 && draftOut.stride(1) == 1 + && draftOut.size(0) >= rows && step >= 0 && step < draftOut.size(1), + "draft_step: draft_out must be int32 [rows, > step] with unit column stride"); + checkInt(idsNext, rows, "ids_next"); + checkInt(posNext, rows, "pos_next"); + checkInt(kvLens, rows, "kv_lens"); + TORCH_CHECK(posSrc.scalar_type() == at::kInt && posSrc.is_contiguous(), "draft_step: pos_src must be int32"); + TORCH_CHECK(numAcc.has_value() == seqLens.has_value(), "draft_step: num_acc and seq_lens come together"); + TORCH_CHECK(!numAcc.has_value() || step == 0, "draft_step: the KV rollback (num_acc) is pass 0's"); + if (numAcc.has_value()) + { + checkInt(*numAcc, rows, "num_acc"); + checkInt(*seqLens, rows, "seq_lens"); + } + if (outKeys.has_value()) + { + TORCH_CHECK(outKeys->scalar_type() == at::kLong && outKeys->is_contiguous() && outKeys->numel() >= rows, + "draft_step: out_keys must be contiguous int64 [rows]"); + } + ds::Params p{}; + p.x = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.x_rows = xRows.has_value() ? reinterpret_cast(xRows->data_ptr()) : nullptr; + p.M = static_cast(rows); + p.wf = reinterpret_cast(wf.data_ptr()); + p.n_tiles = nTiles; + p.vocab_offset = static_cast(vocabOffset); + p.keys = reinterpret_cast(keys.data_ptr()); + p.counters = reinterpret_cast(counters.data_ptr()); + p.ws_uc = reinterpret_cast(wsUc.data_ptr()); + p.ws_mc = reinterpret_cast(wsMcPtr); + p.rank = static_cast(rank); + p.world = static_cast(world); + p.out_keys = outKeys.has_value() ? reinterpret_cast(outKeys->data_ptr()) : nullptr; + p.st = ds::StepArgs{draftOut.data_ptr(), static_cast(draftOut.stride(0)), static_cast(step), + idsNext.data_ptr(), posSrc.data_ptr(), posNext.data_ptr(), kvLens.data_ptr(), + numAcc.has_value() ? numAcc->data_ptr() : nullptr, + seqLens.has_value() ? seqLens->data_ptr() : nullptr, static_cast(draftLen)}; + p.cd = ds::CodedArgs{coded ? excOff->data_ptr() : nullptr, + coded ? reinterpret_cast(excPos->data_ptr()) : nullptr, + coded ? reinterpret_cast(excW->data_ptr()) : nullptr, + static_cast((expBase << 7) | (expBase << 23))}; + if (fold.has_value()) + { + // The 14 fold-tail tensors in buildParams' order (ctrl, slots, src_new, dst_new, src_next, dst_next, + // src_draft, dst_draft, src_lens, dst_lens, ids, draft_ids, pos_off, kv_off): the in-kernel tail's operands, + // packed exactly as the standalone fold tail packs them. + std::vector f = *fold; + TORCH_CHECK(f.size() == 14, "minimax_m3_sm107tp4_draft_step: fold takes 14 tensors, got ", f.size()); + p.fold = minimaxM3Sm107Tp4FoldTailParams::buildParams(f[0], f[1], foldRows, f[2], f[3], f[4], f[5], f[6], f[7], + f[8], f[9], f[10], f[11], f[12], f[13], foldTokensPerStep); + p.hasFold = true; + } + p.fixedOrder = fixedOrder; + check(ds::invokeDraftStep(p, K, static_cast(numCtas), pdl, at::cuda::getCurrentCUDAStream(x.get_device()))); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +// Mutations through the raw multicast address ws_mc_ptr are not visible to the schema: that buffer is owned by the +// Python workspace and never aliased by the op's tensor arguments. +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107tp4_draft_step(Tensor x, Tensor? x_rows, int rows, Tensor wf, int vocab_offset, " + "Tensor(a!) keys, " + "Tensor(b!) counters, Tensor(c!) ws_uc, int ws_mc_ptr, int rank, int world, Tensor(d!) draft_out, int step, " + "Tensor(e!) ids_next, Tensor pos_src, Tensor(f!) pos_next, Tensor(g!) kv_lens, Tensor? num_acc, " + "Tensor(h!)? seq_lens, int draft_len, int num_ctas, bool pdl, Tensor(i!)? out_keys, Tensor? exc_off=None, " + "Tensor? exc_pos=None, Tensor? exc_w=None, int exp_base=0" + ", Tensor(j!)[]? fold=None, int fold_rows=0, int fold_tokens_per_step=0, bool fixed_order=False) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp4_draft_step", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4DraftStep); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DrafterRopeFp8KvOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DrafterRopeFp8KvOp.cpp new file mode 100644 index 000000000000..034521d29796 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4DrafterRopeFp8KvOp.cpp @@ -0,0 +1,103 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace drk = tensorrt_llm::kernels::minimaxM3Sm107Tp4::drafterRopeFp8Kv; + +//! The MiniMax-M3 TP4 drafter layer's generation preprocess (kernels/minimaxM3Sm107Tp4/drafterRopeFp8Kv.h): RoPE on +//! the packed BF16 qkv rows, the E4M3 query into q_out, and the step's K / V rows into the paged E4M3 cache kv_pages +//! ([sub-pages, Hkv, tokens_per_block, 128]) at the sub-pages block_tables ([requests, 2, max blocks]) names. +void minimaxM3Sm107Tp4DrafterRopeFp8Kv(torch::Tensor const& qkv, torch::Tensor const& cosSin, + torch::Tensor const& seqLens, std::optional const& specPositionOffsets, torch::Tensor& kvPages, + torch::Tensor const& blockTables, torch::Tensor& qOut, int64_t qLen, int64_t numHeadsQ, int64_t numHeadsKv, + int64_t rotaryDim, int64_t tokensPerBlock, int64_t ropeMode, bool pdl, bool maxSharedCarveout) +{ + TORCH_CHECK(qkv.is_cuda() && qkv.is_contiguous() && qkv.dim() == 2 && qkv.scalar_type() == at::kBFloat16); + TORCH_CHECK(qkv.size(1) == (numHeadsQ + 2 * numHeadsKv) * drk::kHeadDim, "qkv is not [tokens, (Hq + 2 Hkv) * 128]"); + TORCH_CHECK(qLen >= 1 && qkv.size(0) % qLen == 0, "the rows are not whole passes of q_len rows"); + TORCH_CHECK(rotaryDim > 0 && rotaryDim <= drk::kHeadDim && rotaryDim % 16 == 0, "unsupported rotary_dim"); + TORCH_CHECK(cosSin.is_cuda() && cosSin.is_contiguous() && cosSin.scalar_type() == at::kFloat + && cosSin.numel() % rotaryDim == 0, + "cos_sin must be the fp32 [positions, rotary_dim / 2] (cos, sin) table"); + int64_t const requests = qkv.size(0) / qLen; + TORCH_CHECK(seqLens.is_cuda() && seqLens.scalar_type() == at::kInt && seqLens.numel() >= requests); + if (specPositionOffsets.has_value()) + { + TORCH_CHECK(specPositionOffsets->is_cuda() && specPositionOffsets->is_contiguous() + && specPositionOffsets->scalar_type() == at::kInt && specPositionOffsets->numel() >= requests * qLen); + } + TORCH_CHECK(kvPages.is_cuda() && kvPages.is_contiguous() && kvPages.dim() == 4 + && kvPages.scalar_type() == at::kFloat8_e4m3fn && kvPages.size(1) == numHeadsKv + && kvPages.size(2) == tokensPerBlock && kvPages.size(3) == drk::kHeadDim, + "kv_pages must be the E4M3 [sub-pages, Hkv, tokens_per_block, 128] cache"); + TORCH_CHECK(blockTables.is_cuda() && blockTables.scalar_type() == at::kInt && blockTables.dim() == 3 + && blockTables.size(0) >= requests && blockTables.size(1) == 2 && blockTables.stride(2) == 1, + "block_tables must be the int32 [requests, 2, max blocks] K / V sub-page table"); + TORCH_CHECK(qOut.is_cuda() && qOut.is_contiguous() && qOut.scalar_type() == at::kFloat8_e4m3fn + && qOut.numel() == qkv.size(0) * numHeadsQ * drk::kHeadDim); + c10::cuda::CUDAGuard const deviceGuard{qkv.device()}; + drk::Params params{}; + params.qkv = reinterpret_cast<__nv_bfloat16 const*>(qkv.data_ptr()); + params.cosSin = reinterpret_cast(cosSin.data_ptr()); + params.seqLens = seqLens.data_ptr(); + params.specPositionOffsets = specPositionOffsets.has_value() ? specPositionOffsets->data_ptr() : nullptr; + params.qOut = reinterpret_cast<__nv_fp8_e4m3*>(qOut.data_ptr()); + params.pool = reinterpret_cast<__nv_fp8_e4m3*>(kvPages.data_ptr()); + params.blockTable = blockTables.data_ptr(); + params.blockTableRequestStride = blockTables.stride(0); + params.blockTableKvStride = blockTables.stride(1); + params.numTokens = static_cast(qkv.size(0)); + params.qLen = static_cast(qLen); + params.numHeadsQ = static_cast(numHeadsQ); + params.numHeadsKv = static_cast(numHeadsKv); + params.rotaryDim = static_cast(rotaryDim); + params.tokensPerBlock = static_cast(tokensPerBlock); + params.ropeMode = static_cast(ropeMode); + cudaError_t const err + = drk::launch(params, pdl, maxSharedCarveout, at::cuda::getCurrentCUDAStream(qkv.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107tp4_drafter_rope_fp8_kv: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107tp4_drafter_rope_fp8_kv(Tensor qkv, Tensor cos_sin, Tensor seq_lens, " + "Tensor? spec_position_offsets, Tensor(a!) kv_pages, Tensor block_tables, Tensor(b!) q_out, int q_len, " + "int num_heads_q, int num_heads_kv, int rotary_dim, int tokens_per_block, int rope_mode, bool pdl, bool " + "max_shared_carveout=True) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp4_drafter_rope_fp8_kv", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4DrafterRopeFp8Kv); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailOp.cpp new file mode 100644 index 000000000000..3df36817814d --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailOp.cpp @@ -0,0 +1,89 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.h" +#include "tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailParams.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The speculative step's forward-tail store and next-step staging in one launch (the C++ form of the Triton +//! kernels/m3_fold_tail.py, the same writes). The caller checks the row counts against the buffers (fold_tail). +static void foldTailImpl(bool oneWarp, torch::Tensor const& ctrl, torch::Tensor const& slots, int64_t rows, + torch::Tensor const& srcNew, torch::Tensor& dstNew, torch::Tensor const& srcNext, torch::Tensor& dstNext, + torch::Tensor const& srcDraft, torch::Tensor& dstDraft, torch::Tensor const& srcLens, torch::Tensor& dstLens, + torch::Tensor& ids, torch::Tensor& draftIds, torch::Tensor& posOff, torch::Tensor& kvOff, int64_t tokensPerStep) +{ + c10::cuda::CUDAGuard const deviceGuard{ctrl.device()}; + auto const p = minimaxM3Sm107Tp4FoldTailParams::buildParams(ctrl, slots, rows, srcNew, dstNew, srcNext, dstNext, + srcDraft, dstDraft, srcLens, dstLens, ids, draftIds, posOff, kvOff, tokensPerStep); + auto const stream = at::cuda::getCurrentCUDAStream(ctrl.get_device()); + auto const err = oneWarp + ? tensorrt_llm::kernels::minimaxM3Sm107Tp4::invokeFoldTailOneWarp(p, stream) + : tensorrt_llm::kernels::minimaxM3Sm107Tp4::invokeFoldTail(p, static_cast(rows), stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp4_fold_tail launch failed: ", cudaGetErrorString(err)); +} + +//! The tail as its standalone kernel (one CTA per row). +void minimaxM3Sm107Tp4FoldTail(torch::Tensor const& ctrl, torch::Tensor const& slots, int64_t rows, + torch::Tensor const& srcNew, torch::Tensor& dstNew, torch::Tensor const& srcNext, torch::Tensor& dstNext, + torch::Tensor const& srcDraft, torch::Tensor& dstDraft, torch::Tensor const& srcLens, torch::Tensor& dstLens, + torch::Tensor& ids, torch::Tensor& draftIds, torch::Tensor& posOff, torch::Tensor& kvOff, int64_t tokensPerStep) +{ + foldTailImpl(false, ctrl, slots, rows, srcNew, dstNew, srcNext, dstNext, srcDraft, dstDraft, srcLens, dstLens, ids, + draftIds, posOff, kvOff, tokensPerStep); +} + +//! The tail's in-kernel form (one warp over the rows, foldTailWarp), the oracle of a caller that folds it in. +void minimaxM3Sm107Tp4FoldTailOneWarp(torch::Tensor const& ctrl, torch::Tensor const& slots, int64_t rows, + torch::Tensor const& srcNew, torch::Tensor& dstNew, torch::Tensor const& srcNext, torch::Tensor& dstNext, + torch::Tensor const& srcDraft, torch::Tensor& dstDraft, torch::Tensor const& srcLens, torch::Tensor& dstLens, + torch::Tensor& ids, torch::Tensor& draftIds, torch::Tensor& posOff, torch::Tensor& kvOff, int64_t tokensPerStep) +{ + foldTailImpl(true, ctrl, slots, rows, srcNew, dstNew, srcNext, dstNext, srcDraft, dstDraft, srcLens, dstLens, ids, + draftIds, posOff, kvOff, tokensPerStep); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp4_fold_tail(Tensor ctrl, Tensor slots, int rows, Tensor src_new, Tensor(a!) dst_new, " + "Tensor src_next, Tensor(b!) dst_next, Tensor src_draft, Tensor(c!) dst_draft, Tensor src_lens, " + "Tensor(d!) dst_lens, Tensor(e!) ids, Tensor(f!) draft_ids, Tensor(g!) pos_off, Tensor(h!) kv_off, " + "int tokens_per_step) -> ()"); + m.def( + "minimax_m3_sm107_tp4_fold_tail_one_warp(Tensor ctrl, Tensor slots, int rows, Tensor src_new, Tensor(a!) " + "dst_new, Tensor src_next, Tensor(b!) dst_next, Tensor src_draft, Tensor(c!) dst_draft, Tensor src_lens, " + "Tensor(d!) dst_lens, Tensor(e!) ids, Tensor(f!) draft_ids, Tensor(g!) pos_off, Tensor(h!) kv_off, " + "int tokens_per_step) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp4_fold_tail", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4FoldTail); + m.impl("minimax_m3_sm107_tp4_fold_tail_one_warp", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4FoldTailOneWarp); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailParams.h b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailParams.h new file mode 100644 index 000000000000..b73820b6fc54 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FoldTailParams.h @@ -0,0 +1,111 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/foldTail.h" + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ +namespace minimaxM3Sm107Tp4FoldTailParams +{ + +constexpr int64_t kMaxWidth = 32; + +inline void check2d(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt32 && t.dim() == 2 && t.size(1) <= kMaxWidth, + "minimax_m3_sm107_tp4_fold_tail: ", name, " must be a 2-D int32 CUDA view at most 32 wide"); +} + +inline void check1d(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt32 && t.dim() == 1 && t.stride(0) == 1, + "minimax_m3_sm107_tp4_fold_tail: ", name, " must be a contiguous 1-D int32 CUDA tensor"); +} + +//! The fold tail's operands checked and packed (the standalone fold tail op and any op that runs the tail in-kernel, +//! e.g. the drafter's last-pass draft_step, build them identically). +inline tensorrt_llm::kernels::minimaxM3Sm107Tp4::FoldTailParams buildParams(torch::Tensor const& ctrl, + torch::Tensor const& slots, int64_t rows, torch::Tensor const& srcNew, torch::Tensor& dstNew, + torch::Tensor const& srcNext, torch::Tensor& dstNext, torch::Tensor const& srcDraft, torch::Tensor& dstDraft, + torch::Tensor const& srcLens, torch::Tensor& dstLens, torch::Tensor& ids, torch::Tensor& draftIds, + torch::Tensor& posOff, torch::Tensor& kvOff, int64_t tokensPerStep) +{ + TORCH_CHECK(ctrl.is_cuda() && ctrl.scalar_type() == torch::kInt32 && ctrl.numel() >= 1, + "minimax_m3_sm107_tp4_fold_tail: ctrl must be an int32 CUDA tensor"); + TORCH_CHECK(slots.is_cuda() && slots.scalar_type() == torch::kInt64 && slots.dim() == 1 && slots.stride(0) == 1, + "minimax_m3_sm107_tp4_fold_tail: slots must be a contiguous 1-D int64 CUDA tensor"); + TORCH_CHECK(rows > 0 && tokensPerStep > 0 && tokensPerStep <= kMaxWidth, + "minimax_m3_sm107_tp4_fold_tail: rows must be positive and tokens_per_step in [1, 32]"); + check2d(srcNew, "src_new"); + check2d(dstNew, "dst_new"); + check2d(srcNext, "src_next"); + check2d(dstNext, "dst_next"); + check2d(srcDraft, "src_draft"); + check2d(dstDraft, "dst_draft"); + check1d(srcLens, "src_lens"); + check1d(dstLens, "dst_lens"); + check1d(ids, "ids"); + check1d(draftIds, "draft_ids"); + check1d(posOff, "pos_off"); + check1d(kvOff, "kv_off"); + tensorrt_llm::kernels::minimaxM3Sm107Tp4::FoldTailParams p{}; + p.ctrl = ctrl.data_ptr(); + p.slots = slots.data_ptr(); + p.srcNew = srcNew.data_ptr(); + p.sn0 = srcNew.stride(0); + p.sn1 = srcNew.stride(1); + p.snW = static_cast(srcNew.size(1)); + p.dstNew = dstNew.data_ptr(); + p.dn0 = dstNew.stride(0); + p.dn1 = dstNew.stride(1); + p.dnW = static_cast(dstNew.size(1)); + p.srcNext = srcNext.data_ptr(); + p.sx0 = srcNext.stride(0); + p.sx1 = srcNext.stride(1); + p.sxW = static_cast(srcNext.size(1)); + p.dstNext = dstNext.data_ptr(); + p.dx0 = dstNext.stride(0); + p.dx1 = dstNext.stride(1); + p.dxW = static_cast(dstNext.size(1)); + p.srcDraft = srcDraft.data_ptr(); + p.sd0 = srcDraft.stride(0); + p.sd1 = srcDraft.stride(1); + p.sdW = static_cast(srcDraft.size(1)); + p.dstDraft = dstDraft.data_ptr(); + p.dd0 = dstDraft.stride(0); + p.dd1 = dstDraft.stride(1); + p.ddW = static_cast(dstDraft.size(1)); + p.srcLens = srcLens.data_ptr(); + p.dstLens = dstLens.data_ptr(); + p.ids = ids.data_ptr(); + p.draftIds = draftIds.data_ptr(); + p.posOff = posOff.data_ptr(); + p.kvOff = kvOff.data_ptr(); + p.tokensPerStep = static_cast(tokensPerStep); + return p; +} + +} // namespace minimaxM3Sm107Tp4FoldTailParams +} // namespace torch_ext + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FullCtxReduceOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FullCtxReduceOp.cpp new file mode 100644 index 000000000000..e2f65f63ae33 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4FullCtxReduceOp.cpp @@ -0,0 +1,92 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/fullCtxReduce.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace fcr = tensorrt_llm::kernels::minimaxM3Sm107Tp4::fullCtxReduce; + +//! Merges the PrimTS split-KV decode scratch (partial_o [B, Hkv, S, R, 128] 16-bit, partial_stats [B, Hkv, S, R] +//! fp32 log2-LSE) of the MiniMax-M3 TP4 full-context decode into out (BF16 [B * SQ, Hkv * hR, 128]); with q8 / sf +//! also its MXFP8 copy (E4M3 [>= B * SQ, Hkv * hR * 128] + 128x4-swizzled UE8M0 scales, as mxfp8_quantize(out, True)). +void minimaxM3Sm107Tp4FullCtxReduce(torch::Tensor const& partialO, torch::Tensor const& partialStats, + torch::Tensor const& seqLensKv, torch::Tensor& out, int64_t seqLenQ, int64_t qTokensPerCta, int64_t tileKv, + int64_t instsKv, bool causal, bool pdl, std::optional const& q8, + std::optional const& sf) +{ + TORCH_CHECK(partialO.is_cuda() && partialO.is_contiguous() && partialO.dim() == 5); + TORCH_CHECK(partialO.scalar_type() == at::kHalf || partialO.scalar_type() == at::kBFloat16); + TORCH_CHECK(partialO.size(4) == fcr::kHeadDim && partialO.size(2) <= fcr::kMaxSplits); + TORCH_CHECK(partialStats.scalar_type() == at::kFloat && partialStats.is_contiguous() + && partialStats.sizes() == partialO.sizes().slice(0, 4)); + TORCH_CHECK(seqLensKv.scalar_type() == at::kInt && seqLensKv.numel() >= partialO.size(0)); + int const rows = static_cast(partialO.size(3)); + TORCH_CHECK(seqLenQ >= 1 && rows % seqLenQ == 0); + int const headsPerKv = rows / static_cast(seqLenQ); + TORCH_CHECK(out.scalar_type() == at::kBFloat16 && out.is_contiguous() + && out.numel() == partialO.size(0) * seqLenQ * partialO.size(1) * headsPerKv * fcr::kHeadDim); + c10::cuda::CUDAGuard const deviceGuard{partialO.device()}; + fcr::Params params{partialO.data_ptr(), partialStats.data_ptr(), seqLensKv.data_ptr(), + reinterpret_cast<__nv_bfloat16*>(out.data_ptr()), static_cast(partialO.size(0)), + static_cast(partialO.size(1)), static_cast(partialO.size(2)), rows, static_cast(seqLenQ), + headsPerKv, static_cast(qTokensPerCta), static_cast(tileKv), static_cast(instsKv), causal, + partialO.scalar_type() == at::kBFloat16}; + TORCH_CHECK(q8.has_value() == sf.has_value()); + if (q8.has_value()) + { + int64_t const width = partialO.size(1) * headsPerKv * fcr::kHeadDim; + int64_t const tokens = partialO.size(0) * seqLenQ; + int64_t const tileRows = (tokens + 127) / 128 * 128; + TORCH_CHECK(q8->is_cuda() && q8->is_contiguous() && q8->scalar_type() == at::kFloat8_e4m3fn && q8->dim() == 2 + && q8->size(1) == width && q8->size(0) >= tokens); + TORCH_CHECK(sf->is_cuda() && sf->is_contiguous() && sf->scalar_type() == at::kByte + && sf->numel() >= tileRows * (width / 32)); + params.q8 = reinterpret_cast(q8->data_ptr()); + params.sf = sf->data_ptr(); + params.numSfKTiles = static_cast(width / 128); + } + cudaError_t const err + = fcr::invokeFullCtxReduce(params, pdl, at::cuda::getCurrentCUDAStream(partialO.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107tp4_fullctx_reduce: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107tp4_fullctx_reduce(Tensor partial_o, Tensor partial_stats, Tensor seq_lens_kv, " + "Tensor(a!) out, int seq_len_q, int q_tokens_per_cta, int tile_kv, int insts_kv, bool causal, bool pdl, " + "Tensor(b!)? q8=None, Tensor(c!)? sf=None) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp4_fullctx_reduce", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4FullCtxReduce); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4McAllreduceNormOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4McAllreduceNormOp.cpp new file mode 100644 index 000000000000..27442555f3b4 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4McAllreduceNormOp.cpp @@ -0,0 +1,181 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/mcAllreduceNormTp4.h" + +#include +#include +#include + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kMaxTokens = 48; +constexpr int64_t kCluster = 8; +constexpr int64_t kMaxWorld = 8; + +} // namespace + +//! MiniMax-M3 SM107 TP4 multicast all-reduce (stock rounding points: BF16 rank sum, then the residual add) + residual + +//! RMSNorm (+ optional MoE finalize, + optional MXFP8 output). +//! +//! @param x BF16 [M, H] partial, or with `perm` the permuted FC2 output [rows, H]. +//! @param x2 Optional BF16 [M, H] second addend (the shared expert in finalize mode). +//! @param residual BF16 [M, H], 1 <= M <= 48. +//! @param normWeight BF16 [H]. +//! @param wsUc This rank's unicast view of the exchange workspace (int32 words); re-armed in place. +//! @param wsMcPtr Device address of the workspace's multicast mapping. +//! @param flags Int32 per-CTA call-parity counters, at least maxt * 8; updated in place. +//! @param cluster Cluster size; 8 is the only one built. +//! @param perm, scales Finalize mode: int32 [M, topk] permuted-row index and fp32 / bf16 [M, topk] routing weights. +//! @param maxt Token capacity of the workspace bucket (8, 16 or 48). +//! @param quant Also return the MXFP8 copy of the normed rows (E4M3 [M, H], swizzled UE8M0 scales). +//! @param prepushed 0, or 2 / 3 when producers already pushed this rank's rows (see the kernel header). +//! @param opts Epilogue variant; every variant is bit-identical. +//! @param capOut Optional BF16 [>= M, H] view (row stride a multiple of 8, 16-byte aligned) that receives the Eagle3 +//! capture of the updated residual: m3_spec_tail capture_norm with capW [H] and capEps, bit for bit (H 6144; not the +//! drafter boundary, opts bit 16). +//! @return [normed, updated residual] or, with `quant`, [normed, updated residual, E4M3 rows, scale bytes]. +std::vector minimaxM3Sm107Tp4McAllreduceNorm(torch::Tensor const& x, + std::optional const& x2, torch::Tensor const& residual, torch::Tensor const& normWeight, double eps, + torch::Tensor& wsUc, int64_t wsMcPtr, torch::Tensor& flags, int64_t rank, int64_t world, int64_t cluster, bool pdl, + std::optional const& perm, std::optional const& scales, int64_t maxt, bool quant, + int64_t prepushed, int64_t opts, std::optional const& capOut, + std::optional const& capW, double capEps) +{ + int const sm = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sm == 107, "minimax_m3_sm107_tp4_mc_allreduce_norm requires SM107, got SM", sm); + TORCH_CHECK(cluster == kCluster, "minimax_m3_sm107_tp4_mc_allreduce_norm: only cluster 8 is built, got ", cluster); + TORCH_CHECK(tensorrt_llm::kernels::isMinimaxM3Sm107Tp4McAllreduceNormSupported( + static_cast(maxt), static_cast(opts)), + "minimax_m3_sm107_tp4_mc_allreduce_norm: unsupported (bucket ", maxt, ", opts ", opts, ")"); + TORCH_CHECK( + world >= 1 && world <= kMaxWorld && rank >= 0 && rank < world, "invalid rank ", rank, " / world ", world); + + tensorrt_llm::kernels::MinimaxM3Sm107Tp4McAllreduceNormParams p; + if (x2.has_value()) + { + TORCH_CHECK(x2->scalar_type() == at::kBFloat16 && x2->is_contiguous() && x2->sizes() == residual.sizes(), + "x2 must be a contiguous bf16 tensor shaped like residual"); + p.x2 = reinterpret_cast<__nv_bfloat16 const*>(x2->data_ptr()); + } + if (perm.has_value()) + { + // Finalize mode: x is the permuted FC2 output [rows, H]. + TORCH_CHECK(scales.has_value() && perm->scalar_type() == at::kInt && perm->is_contiguous() && perm->dim() == 2 + && perm->size(0) == residual.size(0), + "perm must be a contiguous int32 [M, topk] tensor, with scales"); + TORCH_CHECK(scales->is_contiguous() && scales->numel() == perm->numel() + && (scales->scalar_type() == at::kFloat || scales->scalar_type() == at::kBFloat16), + "scales must be a contiguous fp32 or bf16 tensor with one weight per perm entry"); + TORCH_CHECK(x.dim() == 2 && x.size(1) == residual.size(1), "fc2 must be [rows, H]"); + p.fc2 = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.perm = perm->data_ptr(); + p.scales = scales->data_ptr(); + p.scaleFp32 = scales->scalar_type() == at::kFloat; + p.topk = static_cast(perm->size(1)); + } + TORCH_CHECK(x.is_cuda() && x.scalar_type() == at::kBFloat16 && x.is_contiguous() && x.dim() == 2, + "x must be a contiguous 2-D bf16 CUDA tensor"); + TORCH_CHECK(residual.dim() == 2 && residual.size(0) >= 1 && residual.size(0) <= std::min(kMaxTokens, maxt), + "residual must hold 1..min(48, bucket) rows"); + TORCH_CHECK(residual.scalar_type() == at::kBFloat16 && residual.is_contiguous() + && (perm.has_value() || residual.sizes() == x.sizes()), + "residual must be a contiguous bf16 tensor shaped like x"); + TORCH_CHECK( + normWeight.scalar_type() == at::kBFloat16 && normWeight.numel() == residual.size(1), "norm_w must be bf16 [H]"); + TORCH_CHECK(wsUc.scalar_type() == at::kInt && wsUc.is_cuda(), "ws_uc must be an int32 CUDA tensor"); + TORCH_CHECK(flags.scalar_type() == at::kInt && flags.is_contiguous() && flags.numel() >= maxt * kCluster, + "flags must be a contiguous int32 tensor of at least bucket * 8 counters"); + + int64_t const m = residual.size(0), h = residual.size(1); + TORCH_CHECK(!quant || h % 128 == 0, "MXFP8 output needs H % 128 == 0"); + c10::cuda::CUDAGuard const deviceGuard{residual.device()}; + auto outNormed = torch::empty_like(residual); + auto outResidual = torch::empty_like(residual); + auto outQ = quant ? torch::empty({m, h}, residual.options().dtype(at::kFloat8_e4m3fn)) : torch::Tensor(); + auto outSf = quant ? torch::empty({(m + 127) / 128 * 128 * (h / 32)}, residual.options().dtype(at::kByte)) + : torch::Tensor(); + + p.x = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.numTokens = static_cast(m); + p.hidden = static_cast(h); + p.residual = reinterpret_cast<__nv_bfloat16 const*>(residual.data_ptr()); + p.normWeight = reinterpret_cast<__nv_bfloat16 const*>(normWeight.data_ptr()); + p.eps = static_cast(eps); + p.wsUc = wsUc.data_ptr(); + p.wsMc = reinterpret_cast(wsMcPtr); + p.flags = flags.data_ptr(); + p.rank = static_cast(rank); + p.world = static_cast(world); + p.outNormed = reinterpret_cast<__nv_bfloat16*>(outNormed.data_ptr()); + p.outResidual = reinterpret_cast<__nv_bfloat16*>(outResidual.data_ptr()); + p.outQuant = quant ? reinterpret_cast(outQ.data_ptr()) : nullptr; + p.outSf = quant ? outSf.data_ptr() : nullptr; + p.prepushed = static_cast(prepushed); + if (capOut.has_value()) + { + TORCH_CHECK((opts & 16) == 0, "the capture epilogue is not built for the drafter boundary (opts bit 16)"); + TORCH_CHECK(capOut->scalar_type() == at::kBFloat16 && capOut->dim() == 2 && capOut->size(0) >= m + && capOut->size(1) == h && capOut->stride(1) == 1 && capOut->stride(0) % 8 == 0 + && reinterpret_cast(capOut->data_ptr()) % 16 == 0 && capOut->device() == residual.device(), + "cap_out must be a bf16 [>= M, H] view with unit column stride, a row stride that is a multiple of 8 and " + "16-byte alignment"); + TORCH_CHECK( + capW.has_value() && capW->scalar_type() == at::kBFloat16 && capW->is_contiguous() && capW->numel() == h, + "cap_w must be a contiguous bf16 [H] tensor"); + p.capOut = reinterpret_cast<__nv_bfloat16*>(capOut->data_ptr()); + p.capStride = capOut->stride(0); + p.capWeight = reinterpret_cast<__nv_bfloat16 const*>(capW->data_ptr()); + p.capEps = static_cast(capEps); + } + tensorrt_llm::kernels::invokeMinimaxM3Sm107Tp4McAllreduceNorm(p, static_cast(maxt), + static_cast(opts), pdl, at::cuda::getCurrentCUDAStream(residual.get_device())); + if (quant) + { + return {outNormed, outResidual, outQ, outSf}; + } + return {outNormed, outResidual}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp4_mc_allreduce_norm(Tensor x, Tensor? x2, Tensor residual, Tensor norm_w, float eps, " + "Tensor(a!) ws_uc, int ws_mc_ptr, Tensor(b!) flags, int rank, int world, int cluster, bool pdl, " + "Tensor? perm, Tensor? scales, int maxt, bool quant, int prepushed, int opts, Tensor(c!)? cap_out=None, " + "Tensor? cap_w=None, float cap_eps=0.) -> Tensor[]"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp4_mc_allreduce_norm", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4McAllreduceNorm); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4ProducerPwOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4ProducerPwOp.cpp new file mode 100644 index 000000000000..3ae68e7b4681 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4ProducerPwOp.cpp @@ -0,0 +1,185 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/producerPw4.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace pw = tensorrt_llm::kernels::minimaxM3Sm107Tp4::producerPw4; + +constexpr int64_t kHeadDim = 128; +constexpr int64_t kPageSize = 128; + +void checkCuda(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda(), "sm107_tp4_producer_pw: ", name, " must be a CUDA tensor"); +} + +} // namespace + +//! The SM107 TP4 NVFP4 QKV + indexer norm/RoPE/KV-insert producer (same arguments and outputs as +//! minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert at 16 query, 1 KV and 1 index head, head_dim 128, rotary_dim 64): +//! returns (q FP8 [T, 8, 128], index_q FP8 [T, 1, 128]) and writes the NVFP4 K/V of the KV head and the FP8 index-K +//! of each token's cache slot. `early` triggers the dependents on entry; `prewait_step_inputs` reads the position ids, +//! the slots and the RoPE rows before the grid-dependency wait (none of them may be written by a PDL predecessor). +std::vector minimaxM3Sm107Tp4ProducerPw(torch::Tensor const& packed, torch::Tensor& kvData, + torch::Tensor& kvScale, torch::Tensor& indexK, torch::Tensor const& outCacheLoc, torch::Tensor const& kvQuantScale, + double eps, torch::Tensor const& qWeight, torch::Tensor const& kWeight, torch::Tensor const& indexQWeight, + torch::Tensor const& indexKWeight, torch::Tensor const& rotaryCosSin, torch::Tensor const& positionIds, bool pdl, + bool early, bool prewaitStepInputs) +{ + checkCuda(packed, "packed"); + for (torch::Tensor const* t : std::initializer_list{&kvData, &kvScale, &indexK, &outCacheLoc, + &kvQuantScale, &qWeight, &kWeight, &indexQWeight, &indexKWeight, &rotaryCosSin, &positionIds}) + { + TORCH_CHECK(t->is_cuda() && t->get_device() == packed.get_device(), + "sm107_tp4_producer_pw: every operand must be on packed's CUDA device"); + } + TORCH_CHECK(reinterpret_cast(packed.data_ptr()) % 8 == 0 + && reinterpret_cast(kvData.data_ptr()) % 2 == 0 + && reinterpret_cast(indexK.data_ptr()) % 4 == 0, + "sm107_tp4_producer_pw: packed and the caches must start at the vector-load / store alignment"); + c10::cuda::CUDAGuard const deviceGuard{packed.device()}; + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_sm107_tp4_producer_pw requires SM100 or newer"); + int64_t const headsQ = pw::kHeadsQ; + int64_t const headsKV = pw::kHeadsKV; + int64_t const headsIndex = pw::kHeadsIndex; + int64_t const totalHeads = headsQ + 2 * headsKV + headsIndex + 1; + TORCH_CHECK(packed.scalar_type() == at::kBFloat16 && packed.dim() == 2 && packed.size(1) == totalHeads * kHeadDim + && packed.is_contiguous(), + "sm107_tp4_producer_pw: packed must be contiguous BF16 [T, 20 * 128]"); + // The kernel's layout (the native producer's checkMinimaxM3Nvfp4HndKVPools at one KV head): HND pools + // [pages, 2 planes, 1 head, 128 tokens, 64 data / 8 scale bytes], dense token rows and head blocks, planes and + // pages that do not overlap. + TORCH_CHECK(kvData.scalar_type() == at::kByte && kvData.dim() == 5 && kvData.size(0) > 0 && kvData.size(1) == 2 + && kvData.size(2) == headsKV && kvData.size(3) == kPageSize && kvData.size(4) == kHeadDim / 2, + "sm107_tp4_producer_pw: kv_data must be the NVFP4 pool [pages, 2, 1, 128, 64] uint8"); + TORCH_CHECK(kvScale.scalar_type() == at::kByte && kvScale.dim() == 5 && kvScale.size(0) == kvData.size(0) + && kvScale.size(1) == 2 && kvScale.size(2) == headsKV && kvScale.size(3) == kPageSize + && kvScale.size(4) == kHeadDim / 16, + "sm107_tp4_producer_pw: kv_scale must be the E4M3 scale pool [pages, 2, 1, 128, 8] uint8"); + TORCH_CHECK(kvData.stride(4) == 1 && kvData.stride(3) == kvData.size(4) + && kvData.stride(2) == kvData.size(3) * kvData.stride(3), + "sm107_tp4_producer_pw: kv_data must have dense token rows and head blocks"); + TORCH_CHECK(kvScale.stride(4) == 1 && kvScale.stride(3) == kvScale.size(4) + && kvScale.stride(2) == kvScale.size(3) * kvScale.stride(3), + "sm107_tp4_producer_pw: kv_scale must have dense token rows and head blocks"); + TORCH_CHECK(kvData.stride(1) >= kvData.size(2) * kvData.stride(2) && kvData.stride(0) >= 2 * kvData.stride(1) + && kvScale.stride(1) >= kvScale.size(2) * kvScale.stride(2) && kvScale.stride(0) >= 2 * kvScale.stride(1), + "sm107_tp4_producer_pw: pool planes or pages overlap"); + TORCH_CHECK(kvData.stride(0) % 2 == 0 && kvData.stride(1) % 2 == 0, + "sm107_tp4_producer_pw: kv_data page and plane strides must keep the uint16 stores aligned"); + TORCH_CHECK(indexK.scalar_type() == at::kFloat8_e4m3fn && indexK.dim() == 4 && indexK.size(0) == kvData.size(0) + && indexK.size(1) == 1 && indexK.size(2) == kPageSize && indexK.size(3) == kHeadDim && indexK.stride(3) == 1 + && indexK.stride(2) == kHeadDim && indexK.stride(0) >= kPageSize * kHeadDim && indexK.stride(0) % 4 == 0, + "sm107_tp4_producer_pw: index_k must be FP8 [pages, 1, 128, 128] with a page stride keeping the 32-bit stores " + "aligned"); + int64_t const numTokens = packed.size(0); + TORCH_CHECK( + outCacheLoc.scalar_type() == at::kInt && outCacheLoc.is_contiguous() && outCacheLoc.numel() >= numTokens, + "sm107_tp4_producer_pw: out_cache_loc must be int32 [>= T]"); + TORCH_CHECK( + positionIds.scalar_type() == at::kInt && positionIds.is_contiguous() && positionIds.numel() >= numTokens, + "sm107_tp4_producer_pw: position_ids must be int32 [>= T]"); + TORCH_CHECK(kvQuantScale.scalar_type() == at::kFloat && kvQuantScale.is_contiguous() && kvQuantScale.numel() >= 3, + "sm107_tp4_producer_pw: kv_quant_scale must be fp32 [>= 3]"); + TORCH_CHECK(rotaryCosSin.scalar_type() == at::kFloat && rotaryCosSin.is_contiguous() && rotaryCosSin.dim() == 3 + && rotaryCosSin.size(1) == 2 && rotaryCosSin.size(2) == 32 + && reinterpret_cast(rotaryCosSin.data_ptr()) % 16 == 0, + "sm107_tp4_producer_pw: rotary_cos_sin must be fp32 [positions, 2, 32], 16-byte aligned (read as float4)"); + for (auto const* w : {&qWeight, &kWeight, &indexQWeight, &indexKWeight}) + { + TORCH_CHECK(w->scalar_type() == at::kBFloat16 && w->is_contiguous() && w->numel() == kHeadDim + && reinterpret_cast(w->data_ptr()) % 8 == 0, + "sm107_tp4_producer_pw: norm weights must be BF16 [128], 8-byte aligned (read as 4-value words)"); + } + auto const fp8 = packed.options().dtype(at::kFloat8_e4m3fn); + auto qOut = torch::empty({numTokens, headsQ, kHeadDim}, fp8); + auto indexQOut = torch::empty({numTokens, headsIndex, kHeadDim}, fp8); + if (numTokens == 0) + { + return {qOut, indexQOut}; + } + pw::Params p{}; + p.packed = reinterpret_cast<__nv_bfloat16 const*>(packed.data_ptr()); + p.qOut = reinterpret_cast<__nv_fp8_e4m3*>(qOut.data_ptr()); + p.indexQOut = reinterpret_cast<__nv_fp8_e4m3*>(indexQOut.data_ptr()); + p.kvData = kvData.data_ptr(); + p.kvScale = kvScale.data_ptr(); + p.indexK = reinterpret_cast<__nv_fp8_e4m3*>(indexK.data_ptr()); + p.outCacheLoc = outCacheLoc.data_ptr(); + p.kvQuantScale = kvQuantScale.data_ptr(); + p.dataPageStride = kvData.stride(0); + p.dataPlaneStride = kvData.stride(1); + p.dataHeadStride = kvData.stride(2); + p.dataTokenStride = kvData.stride(3); + p.scalePageStride = kvScale.stride(0); + p.scalePlaneStride = kvScale.stride(1); + p.scaleHeadStride = kvScale.stride(2); + p.indexPageStride = indexK.stride(0); + p.indexTokenStride = indexK.stride(2); + p.numPages = kvData.size(0); + p.numTokens = static_cast(numTokens); + p.eps = static_cast(eps); + p.qWeight = reinterpret_cast<__nv_bfloat16 const*>(qWeight.data_ptr()); + p.kWeight = reinterpret_cast<__nv_bfloat16 const*>(kWeight.data_ptr()); + p.indexQWeight = reinterpret_cast<__nv_bfloat16 const*>(indexQWeight.data_ptr()); + p.indexKWeight = reinterpret_cast<__nv_bfloat16 const*>(indexKWeight.data_ptr()); + p.rotaryCosSin = rotaryCosSin.data_ptr(); + p.positionIds = positionIds.data_ptr(); + cudaError_t const err = pw::invokeProducerPw4(p, static_cast(numTokens), pdl, early, prewaitStepInputs, + at::cuda::getCurrentCUDAStream(packed.get_device())); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "sm107_tp4_producer_pw launch: ", cudaGetErrorString(err)); + } + return {qOut, indexQOut}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp4_producer_pw(Tensor packed, Tensor(a!) kv_data, Tensor(b!) kv_scale, Tensor(c!) index_k, " + "Tensor out_cache_loc, Tensor kv_quant_scale, float eps, Tensor q_weight, Tensor k_weight, " + "Tensor index_q_weight, Tensor index_k_weight, Tensor rotary_cos_sin, Tensor position_ids, bool pdl, " + "bool early, bool prewait_step_inputs=False) -> Tensor[]"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp4_producer_pw", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4ProducerPw); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SampledAcceptOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SampledAcceptOp.cpp new file mode 100644 index 000000000000..4f7fda5cac90 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SampledAcceptOp.cpp @@ -0,0 +1,113 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace sa = tensorrt_llm::kernels::minimaxM3Sm107Tp4::sampledAccept; + +//! uint64 words of the sampled-accept exchange workspace for `world` ranks and SA-1R variant `oneRoundK`. +int64_t minimaxM3Sm107Tp4SampledAcceptWsWords(int64_t world, int64_t oneRoundK) +{ + return sa::sampledAcceptWsWords(static_cast(world), static_cast(oneRoundK)); +} + +//! Sampled (temperature + top-p) verification from this rank's BF16 logits shard [rows, Vs], rows = nreq * +//! (draft_len + 1) <= 48: the sampled token of every row (tokens), then the strict acceptance against draft (accTokens, +//! numAcc, gatherIds, optional nextNew column 0). Per-row Philox (seeds, offsets); the exchange goes through the +//! multicast workspace wsMcPtr / wsUc with the (finished rows, call) counters. +void minimaxM3Sm107Tp4SampledAccept(torch::Tensor const& logits, int64_t vocabOffset, torch::Tensor const& temperatures, + torch::Tensor const& topPs, torch::Tensor const& seeds, torch::Tensor const& offsets, torch::Tensor& wsUc, + int64_t wsMcPtr, torch::Tensor& counters, torch::Tensor& tokens, int64_t rank, int64_t world, + torch::Tensor const& draft, torch::Tensor const& batchIdx, torch::Tensor& accTokens, torch::Tensor& numAcc, + torch::Tensor& gatherIds, int64_t forceTotal, std::optional const& nextNew, + std::optional const& forceVec, std::optional const& extraAttempts, bool usePdl, + bool prewaitParams, bool earlyTrigger, int64_t oneRoundK, std::optional const& decided, bool ilpWalk) +{ + c10::cuda::CUDAGuard const deviceGuard{logits.device()}; + sa::Args a; + a.logits = reinterpret_cast<__nv_bfloat16 const*>(logits.data_ptr()); + a.ld = logits.stride(0); + a.rows = static_cast(logits.size(0)); + a.Vs = static_cast(logits.size(1)); + TORCH_CHECK(a.Vs <= sa::kMaxVs && a.rows <= sa::kMaxRows && world <= sa::kMaxWorld + && static_cast(vocabOffset) + a.Vs <= (1LL << sa::kIdBits)); + a.vocab_offset = static_cast(vocabOffset); + a.temperatures = temperatures.data_ptr(); + a.top_ps = topPs.data_ptr(); + a.seeds = reinterpret_cast(seeds.data_ptr()); + a.offsets = reinterpret_cast(offsets.data_ptr()); + a.ws_uc = reinterpret_cast(wsUc.data_ptr()); + a.ws_mc = reinterpret_cast(wsMcPtr); + a.counters = reinterpret_cast(counters.data_ptr()); + a.tokens = tokens.data_ptr(); + a.extra_attempts = extraAttempts.has_value() ? extraAttempts->data_ptr() : nullptr; + a.rank = static_cast(rank); + a.world = static_cast(world); + a.draft = draft.data_ptr(); + a.batch_idx = batchIdx.data_ptr(); + a.acc_tokens = accTokens.data_ptr(); + a.num_acc = numAcc.data_ptr(); + a.gather_ids = reinterpret_cast(gatherIds.data_ptr()); + a.draft_len = static_cast(draft.size(1)); + a.nreq = static_cast(draft.size(0)); + a.force_total = static_cast(forceTotal); + a.next_new = nextNew.has_value() ? nextNew->data_ptr() : nullptr; + a.next_stride = nextNew.has_value() ? static_cast(nextNew->stride(0)) : 0; + a.force_vec = forceVec.has_value() ? forceVec->data_ptr() : nullptr; + a.prewait_params = prewaitParams ? 1 : 0; + a.early_trigger = earlyTrigger ? 1 : 0; + TORCH_CHECK(oneRoundK == 0 || oneRoundK == 9 || oneRoundK == 82, "one_round_k must be 0, 9 or 82, got ", oneRoundK); + a.one_round_k = static_cast(oneRoundK); + a.decided = decided.has_value() ? decided->data_ptr() : nullptr; + a.ilp_walk = ilpWalk ? 1 : 0; + cudaError_t const err = sa::invokeSampledAccept(a, usePdl, at::cuda::getCurrentCUDAStream(logits.get_device())); + TORCH_CHECK(err == cudaSuccess, "m3_sampled_accept launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +// ws_mc_ptr is the multicast address of ws_uc's mapping: the op writes through it, which the schema cannot show. +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + // No tensor arguments, so no dispatch key: registered with the schema as a catch-all kernel. + m.def("minimax_m3_sm107tp4_sampled_accept_ws_words(int world, int one_round_k=0) -> int", + &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4SampledAcceptWsWords); + m.def( + "minimax_m3_sm107tp4_sampled_accept(Tensor logits, int vocab_offset, Tensor temperatures, Tensor top_ps, " + "Tensor seeds, Tensor offsets, Tensor(a!) ws_uc, int ws_mc_ptr, Tensor(b!) counters, Tensor(c!) tokens, " + "int rank, int world, Tensor draft, Tensor batch_idx, Tensor(d!) acc_tokens, Tensor(e!) num_acc, " + "Tensor(f!) gather_ids, int force_total, Tensor(g!)? next_new, Tensor? force_vec, " + "Tensor(h!)? extra_attempts, bool use_pdl, bool prewait_params=True, bool early_trigger=False, " + "int one_round_k=0, Tensor(i!)? decided=None, bool ilp_walk=False) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp4_sampled_accept", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4SampledAccept); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SelectLongOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SelectLongOp.cpp new file mode 100644 index 000000000000..b1cf6be45569 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SelectLongOp.cpp @@ -0,0 +1,87 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/selectLong2.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! Top-16 block select for MiniMax-M3 decode rows of up to 9216 blocks, into a caller-owned table. +//! +//! @param scores Float32 CUDA tensor [num_kv_heads, num_blocks, total_q], any strides. +//! @param nValidBlocks Contiguous int32 CUDA tensor with at least total_q elements. +//! @param out Int32 CUDA tensor of logical shape [total_q, num_kv_heads, 16] with unit stride in the last dim (the +//! head-major backing the target hands over is a permuted view). Every entry is written. +//! @param triggerPhase Where the dependent grid is released without earlyTrigger: 0 at this grid's exit, 1 right +//! after every score of the row is loaded (both after this grid's dependency wait). +//! @param fastTail Rank rows of at most 64 candidates on warp 0 and read the valid count before the wait (same picks). +void minimaxM3Sm107Tp4SelectLong(torch::Tensor const& scores, torch::Tensor const& nValidBlocks, torch::Tensor& out, + int64_t initBlocks, int64_t localBlocks, bool usePdl, bool earlyTrigger, int64_t triggerPhase, bool fastTail) +{ + TORCH_CHECK(scores.is_cuda() && scores.scalar_type() == torch::kFloat32 && scores.dim() == 3, + "minimax_m3_sm107_tp4_select_long: scores must be a 3-D float32 CUDA tensor [num_kv_heads, num_blocks, " + "total_q]"); + TORCH_CHECK(nValidBlocks.is_cuda() && nValidBlocks.scalar_type() == torch::kInt32 && nValidBlocks.is_contiguous() + && nValidBlocks.numel() >= scores.size(2), + "minimax_m3_sm107_tp4_select_long: n_valid_blocks must be contiguous int32 CUDA with at least total_q " + "elements"); + int64_t const numKvHeads = scores.size(0); + int64_t const numBlocks = scores.size(1); + int64_t const totalQ = scores.size(2); + TORCH_CHECK(numBlocks <= tensorrt_llm::kernels::minimaxM3Sm107Tp4::kSelectLong2MaxBlocks, + "minimax_m3_sm107_tp4_select_long supports at most ", + tensorrt_llm::kernels::minimaxM3Sm107Tp4::kSelectLong2MaxBlocks, " blocks, got ", numBlocks); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kInt32 && out.dim() == 3 && out.size(0) == totalQ + && out.size(1) == numKvHeads && out.size(2) == 16 && out.stride(2) == 1, + "minimax_m3_sm107_tp4_select_long: out must be int32 [total_q, num_kv_heads, 16] with unit last-dim stride"); + + c10::cuda::CUDAGuard const deviceGuard{scores.device()}; + auto const stream = at::cuda::getCurrentCUDAStream(scores.get_device()); + cudaError_t const err = tensorrt_llm::kernels::minimaxM3Sm107Tp4::invokeSelectLong2(scores.data_ptr(), + scores.stride(0), scores.stride(1), scores.stride(2), nValidBlocks.data_ptr(), out.data_ptr(), + out.stride(0), out.stride(1), static_cast(numKvHeads), static_cast(numBlocks), + static_cast(totalQ), static_cast(initBlocks), static_cast(localBlocks), usePdl, + earlyTrigger, static_cast(triggerPhase), fastTail, stream); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "minimax_m3_sm107_tp4_select_long launch failed: ", cudaGetErrorString(err)); + } +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp4_select_long(Tensor scores, Tensor n_valid_blocks, Tensor(a!) out, int init_blocks, " + "int local_blocks, bool use_pdl=False, bool early_trigger=False, int trigger_phase=0, bool fast_tail=True) -> " + "()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp4_select_long", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4SelectLong); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SiluAndMulOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SiluAndMulOp.cpp new file mode 100644 index 000000000000..b312cf79a715 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SiluAndMulOp.cpp @@ -0,0 +1,66 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/siluAndMul.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The drafter MLP's plain SwiGLU (see kernels/minimaxM3Sm107Tp4/siluAndMul.h): x bf16 [rows, 2d] -> out bf16 [rows, +//! d]. +void minimaxM3Sm107Tp4SiluAndMul(torch::Tensor const& x, torch::Tensor& out) +{ + TORCH_CHECK(x.is_cuda() && x.scalar_type() == torch::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "minimax_m3_sm107_tp4_silu_and_mul: x must be a contiguous 2-D bfloat16 CUDA tensor"); + int64_t const rows = x.size(0); + int64_t const d = x.size(1) / 2; + TORCH_CHECK(x.size(1) % 16 == 0 && reinterpret_cast(x.data_ptr()) % 16 == 0, + "minimax_m3_sm107_tp4_silu_and_mul: x rows must be 16-byte aligned with a half width a multiple of 8"); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.is_contiguous() + && out.size(0) == rows && out.size(1) == d && reinterpret_cast(out.data_ptr()) % 16 == 0, + "minimax_m3_sm107_tp4_silu_and_mul: out must be a contiguous, 16-byte aligned bfloat16 [rows, d]"); + TORCH_CHECK(rows <= 65535, "minimax_m3_sm107_tp4_silu_and_mul: at most 65535 rows"); + if (rows == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto const err + = tensorrt_llm::kernels::minimaxM3Sm107Tp4::invokeSiluAndMul(reinterpret_cast(x.data_ptr()), + reinterpret_cast(out.data_ptr()), rows, d, at::cuda::getCurrentCUDAStream(x.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp4_silu_and_mul launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp4_silu_and_mul(Tensor x, Tensor(a!) out) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp4_silu_and_mul", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4SiluAndMul); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SparseDecodeLcOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SparseDecodeLcOp.cpp new file mode 100644 index 000000000000..53121e3b1e7a --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SparseDecodeLcOp.cpp @@ -0,0 +1,191 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/sparseDecodeLc4.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kHeads = 16; +constexpr int64_t kHeadDim = 128; +constexpr int64_t kTopK = 16; + +void checkBytes(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, "minimax_m3_sm107_tp4_sparse_decode_lc: ", name, + " must be a one-byte CUDA view (uint8 / e4m3 / packed e2m1)"); +} + +} // namespace + +//! One-launch MiniMax-M3 long-context sparse decode over the selected NVFP4 blocks (TP4 geometry: 16 query heads on +//! one KV head, head_dim 128, page 128, top-k 16). +//! +//! @param q E4M3 bytes [T, 16, 128], unit stride in the last dim. +//! @param k, v Packed E2M1 bytes [pages, 1, 128, 64] (the per-layer HND views). +//! @param kBlockScale, vBlockScale E4M3 bytes [pages, 1, 128, 8]: K token-major, V in 4x4 token-quad order. +//! @param kGlobalScale, vGlobalScale One-element float32 per-tensor dequant scales. +//! @param topk Int32 [T, 16] ascending selected block ids with -1 tail (kv head 0 of the head-major table). +//! @param out BF16 [T, 2048], written. q8 / sf, when given, receive the MXFP8 copy and its swizzled UE8M0 scales. +void minimaxM3Sm107Tp4SparseDecodeLc(torch::Tensor const& q, torch::Tensor const& k, torch::Tensor const& v, + torch::Tensor const& kBlockScale, torch::Tensor const& vBlockScale, torch::Tensor const& kGlobalScale, + torch::Tensor const& vGlobalScale, torch::Tensor const& topk, torch::Tensor const& blockTable, + torch::Tensor const& seqLens, torch::Tensor& out, std::optional q8, std::optional sf, + int64_t numSfKTiles, int64_t decodeQueryLen, double smScale, bool linearSoftmax, bool usePdl, bool earlyTrigger, + std::optional prefetchA, std::optional prefetchB, std::optional n1Self, + std::optional n1Next, std::optional n1K, std::optional n1V, + std::optional n1KScale, std::optional n1VScale, bool identity, bool oneDiv, + bool identityStep, bool btPrewait) +{ + TORCH_CHECK(earlyTrigger, "minimax_m3_sm107_tp4_sparse_decode_lc is instantiated with early_trigger=True only"); + TORCH_CHECK(q.is_cuda(), "minimax_m3_sm107_tp4_sparse_decode_lc: q must be a CUDA tensor"); + c10::cuda::CUDAGuard const deviceGuard{q.device()}; + TORCH_CHECK(at::cuda::getCurrentDeviceProperties()->major >= 10, + "minimax_m3_sm107_tp4_sparse_decode_lc requires an SM100-family GPU"); + checkBytes(q, "q"); + checkBytes(k, "k"); + checkBytes(v, "v"); + checkBytes(kBlockScale, "k_block_scale"); + checkBytes(vBlockScale, "v_block_scale"); + TORCH_CHECK(q.dim() == 3 && q.size(1) == kHeads && q.size(2) == kHeadDim && q.stride(2) == 1, + "minimax_m3_sm107_tp4_sparse_decode_lc: q must be [T, 16, 128] with unit last-dim stride"); + TORCH_CHECK(kGlobalScale.scalar_type() == torch::kFloat32 && vGlobalScale.scalar_type() == torch::kFloat32, + "minimax_m3_sm107_tp4_sparse_decode_lc: global scales must be float32"); + TORCH_CHECK(topk.scalar_type() == torch::kInt32 && topk.dim() == 2 && topk.size(1) == kTopK, + "minimax_m3_sm107_tp4_sparse_decode_lc: topk must be int32 [T, 16]"); + TORCH_CHECK(blockTable.scalar_type() == torch::kInt32 && seqLens.scalar_type() == torch::kInt32, + "minimax_m3_sm107_tp4_sparse_decode_lc: block_table and seq_lens must be int32"); + TORCH_CHECK(out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.size(0) == q.size(0) + && out.size(1) == kHeads * kHeadDim && out.stride(1) == 1, + "minimax_m3_sm107_tp4_sparse_decode_lc: out must be bfloat16 [T, 2048] with unit last-dim stride"); + TORCH_CHECK( + q8.has_value() == sf.has_value(), "minimax_m3_sm107_tp4_sparse_decode_lc: pass both q8 and sf, or neither"); + bool const mxfp8 = q8.has_value(); + if (mxfp8) + { + checkBytes(*q8, "q8"); + checkBytes(*sf, "sf"); + } + + tensorrt_llm::kernels::minimaxM3Sm107Tp4::MinimaxM3SparseDecodeLcParams prm{}; + prm.q = reinterpret_cast(q.data_ptr()); + prm.q_tok = q.stride(0); + prm.q_head = q.stride(1); + prm.k = reinterpret_cast(k.data_ptr()); + prm.v = reinterpret_cast(v.data_ptr()); + prm.k_blk = k.stride(0); + prm.v_blk = v.stride(0); + prm.ksc = reinterpret_cast(kBlockScale.data_ptr()); + prm.vsc = reinterpret_cast(vBlockScale.data_ptr()); + prm.ksc_blk = kBlockScale.stride(0); + prm.vsc_blk = vBlockScale.stride(0); + prm.kgs = kGlobalScale.data_ptr(); + prm.vgs = vGlobalScale.data_ptr(); + prm.topk = topk.data_ptr(); + prm.t_tok = topk.stride(0); + prm.t_k = topk.stride(1); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.seq_lens = seqLens.data_ptr(); + prm.dql = static_cast(decodeQueryLen); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(out.data_ptr()); + prm.out_stride = out.stride(0); + prm.q8 = mxfp8 ? reinterpret_cast(q8->data_ptr()) : nullptr; + prm.sf = mxfp8 ? reinterpret_cast(sf->data_ptr()) : nullptr; + prm.num_sf_k_tiles = static_cast(numSfKTiles); + prm.identity = identity ? 1 : 0; + prm.identity_step = (identity || identityStep) ? 1 : 0; + prm.one_div = oneDiv ? 1 : 0; + prm.bt_prewait = btPrewait ? 1 : 0; + auto const prefetchRange = [](std::optional const& t, uint8_t const*& ptr, int64_t& bytes) + { + ptr = nullptr; + bytes = 0; + if (t.has_value()) + { + TORCH_CHECK(t->is_cuda() && t->is_contiguous(), + "minimax_m3_sm107_tp4_sparse_decode_lc: a prefetch range must be a contiguous CUDA tensor"); + ptr = reinterpret_cast(t->data_ptr()); + bytes = t->numel() * static_cast(t->element_size()); + } + }; + prefetchRange(prefetchA, prm.pf0, prm.pf0_bytes); + prefetchRange(prefetchB, prm.pf1, prm.pf1_bytes); + bool const n1 = n1Self.has_value(); + TORCH_CHECK(n1 == n1Next.has_value() && n1 == n1K.has_value() && n1 == n1V.has_value() && n1 == n1KScale.has_value() + && n1 == n1VScale.has_value(), + "minimax_m3_sm107_tp4_sparse_decode_lc: pass all six next-layer prefetch tensors, or none"); + if (n1) + { + for (auto const* t : {&*n1Self, &*n1Next}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == torch::kInt32 && t->is_contiguous() && t->dim() == 2 + && t->size(1) == kTopK && t->size(0) >= q.size(0), + "minimax_m3_sm107_tp4_sparse_decode_lc: the id buffers must be contiguous int32 [>= T, 16]"); + } + TORCH_CHECK(n1K->stride(0) == k.stride(0) && n1V->stride(0) == v.stride(0) + && n1KScale->stride(0) == kBlockScale.stride(0) && n1VScale->stride(0) == vBlockScale.stride(0) + && n1K->size(0) == k.size(0), + "minimax_m3_sm107_tp4_sparse_decode_lc: the next layer's pools must have this layer's page strides"); + prm.n1_self = n1Self->data_ptr(); + prm.n1_next = n1Next->data_ptr(); + prm.n1_k = reinterpret_cast(n1K->data_ptr()); + prm.n1_v = reinterpret_cast(n1V->data_ptr()); + prm.n1_ksc = reinterpret_cast(n1KScale->data_ptr()); + prm.n1_vsc = reinterpret_cast(n1VScale->data_ptr()); + prm.n1_pages = n1K->size(0); + } + + auto const stream = at::cuda::getCurrentCUDAStream(q.get_device()); + cudaError_t const err = tensorrt_llm::kernels::minimaxM3Sm107Tp4::invokeSparseDecodeLc4( + prm, static_cast(q.size(0)), linearSoftmax, mxfp8, earlyTrigger, usePdl, stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp4_sparse_decode_lc launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp4_sparse_decode_lc(Tensor q, Tensor k, Tensor v, Tensor k_block_scale, Tensor " + "v_block_scale, " + "Tensor k_global_scale, Tensor v_global_scale, Tensor topk, Tensor block_table, Tensor seq_lens, " + "Tensor(a!) out, Tensor(b!)? q8, Tensor(c!)? sf, int num_sf_k_tiles, int decode_query_len, float sm_scale, " + "bool linear_softmax, bool use_pdl, bool early_trigger, Tensor? prefetch_a=None, Tensor? prefetch_b=None, " + "Tensor(d!)? n1_self=None, Tensor? n1_next=None, Tensor? n1_k=None, Tensor? n1_v=None, Tensor? " + "n1_k_scale=None, " + "Tensor? n1_v_scale=None, bool identity=False, bool one_div=False, bool identity_step=False, " + "bool bt_prewait=False) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp4_sparse_decode_lc", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4SparseDecodeLc); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SpecOffsetsOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SpecOffsetsOp.cpp new file mode 100644 index 000000000000..2142baa18599 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SpecOffsetsOp.cpp @@ -0,0 +1,74 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/specOffsets.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +void check(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt32 && t.dim() == 1 && t.stride(0) == 1, + "minimax_m3_sm107_tp4_spec_offsets: ", name, " must be a contiguous 1-D int32 CUDA tensor"); +} + +} // namespace + +//! pos += pos_off and kv += kv_off (equal lengths per pair) in one plain launch +//! (kernels/minimaxM3Sm107Tp4/specOffsets.h). +void minimaxM3Sm107Tp4SpecOffsets( + torch::Tensor& pos, torch::Tensor const& posOff, torch::Tensor& kv, torch::Tensor const& kvOff) +{ + check(pos, "pos"); + check(posOff, "pos_off"); + check(kv, "kv"); + check(kvOff, "kv_off"); + TORCH_CHECK(pos.numel() == posOff.numel() && kv.numel() == kvOff.numel(), + "minimax_m3_sm107_tp4_spec_offsets: each tensor and its offsets must have the same length"); + if (pos.numel() == 0 && kv.numel() == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{pos.device()}; + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp4::invokeSpecOffsets(pos.data_ptr(), + posOff.data_ptr(), pos.numel(), kv.data_ptr(), kvOff.data_ptr(), kv.numel(), + at::cuda::getCurrentCUDAStream(pos.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp4_spec_offsets launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp4_spec_offsets(Tensor(a!) pos, Tensor pos_off, Tensor(b!) kv, Tensor kv_off) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp4_spec_offsets", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4SpecOffsets); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4StepPrologueOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4StepPrologueOp.cpp new file mode 100644 index 000000000000..59009e461d33 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4StepPrologueOp.cpp @@ -0,0 +1,68 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/stepPrologue.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The decode graph's step-input prologue in one launch (see kernels/minimaxM3Sm107Tp4/stepPrologue.h). +//! +//! @param window The prologue's device window: contiguous uint8, at least the header and `maxDesc` descriptors (the +//! pack's descriptor area; the header's two counts add up to at most that). Its destinations are raw device addresses +//! the caller guarantees are live. +//! @param epoch Optional one-element int64 step counter, incremented once. +void minimaxM3Sm107Tp4StepPrologue(torch::Tensor& window, std::optional epoch, int64_t maxDesc) +{ + TORCH_CHECK(window.is_cuda() && window.scalar_type() == torch::kUInt8 && window.is_contiguous(), + "minimax_m3_sm107_tp4_step_prologue: window must be a contiguous uint8 CUDA tensor"); + TORCH_CHECK(maxDesc > 0 && window.numel() >= 16 + 24 * maxDesc, + "minimax_m3_sm107_tp4_step_prologue: window is shorter than its header and max_desc descriptors"); + c10::cuda::CUDAGuard const deviceGuard{window.device()}; + int64_t* epochPtr = nullptr; + if (epoch.has_value()) + { + TORCH_CHECK(epoch->is_cuda() && epoch->scalar_type() == torch::kInt64 && epoch->numel() == 1 + && epoch->device() == window.device(), + "minimax_m3_sm107_tp4_step_prologue: epoch must be a one-element int64 tensor on the window's device"); + epochPtr = epoch->data_ptr(); + } + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp4::invokeStepPrologue( + window.data_ptr(), epochPtr, maxDesc, at::cuda::getCurrentCUDAStream(window.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp4_step_prologue launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp4_step_prologue(Tensor(a!) window, Tensor(b!)? epoch, int max_desc) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp4_step_prologue", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4StepPrologue); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SwigluOaiOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SwigluOaiOp.cpp new file mode 100644 index 000000000000..1172d809a817 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp4SwigluOaiOp.cpp @@ -0,0 +1,65 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp4/swigluOai.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The dense MLP's swigluoai activation (see kernels/minimaxM3Sm107Tp4/swigluOai.h): x bf16 [rows, 2d] -> out bf16 +//! [rows, d], PDL-launched with an early release when `pdl`. +void minimaxM3Sm107Tp4SwigluOai(torch::Tensor const& x, torch::Tensor& out, bool pdl) +{ + TORCH_CHECK(x.is_cuda() && x.scalar_type() == torch::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "minimax_m3_sm107_tp4_swiglu_oai: x must be a contiguous 2-D bfloat16 CUDA tensor"); + int64_t const rows = x.size(0); + int64_t const d = x.size(1) / 2; + TORCH_CHECK(x.size(1) % 4 == 0 && reinterpret_cast(x.data_ptr()) % 4 == 0, + "minimax_m3_sm107_tp4_swiglu_oai: x rows must be 4-byte aligned with an even half width"); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.is_contiguous() + && out.size(0) == rows && out.size(1) == d && reinterpret_cast(out.data_ptr()) % 4 == 0, + "minimax_m3_sm107_tp4_swiglu_oai: out must be a contiguous, 4-byte aligned bfloat16 [rows, d]"); + if (rows == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp4::invokeSwigluOai( + reinterpret_cast(x.data_ptr()), reinterpret_cast(out.data_ptr()), rows, d / 2, pdl, + at::cuda::getCurrentCUDAStream(x.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp4_swiglu_oai launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp4_swiglu_oai(Tensor x, Tensor(a!) out, bool pdl) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp4_swiglu_oai", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp4SwigluOai); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8DraftStepOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8DraftStepOp.cpp new file mode 100644 index 000000000000..bee82725b41f --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8DraftStepOp.cpp @@ -0,0 +1,236 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/draftStep.h" +#include "tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailParams.h" + +#include +#include +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace ds = tensorrt_llm::kernels::minimaxM3Sm107Tp8::draftStep; + +using OptTensor = std::optional; + +void checkArch(torch::Tensor const& t) +{ + TORCH_CHECK(t.is_cuda(), "minimax_m3_sm107tp8_draft_step expects CUDA tensors"); + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_sm107tp8_draft_step requires SM100 or newer"); +} + +void check(cudaError_t err) +{ + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, cudaGetErrorString(err)); + } +} + +void checkInt(torch::Tensor const& t, int64_t n, char const* name) +{ + TORCH_CHECK(t.scalar_type() == at::kInt && t.is_contiguous() && t.numel() >= n, "draft_step: ", name, + " must be contiguous int32 with at least ", n, " elements"); +} + +} // namespace + +//! One fused greedy draft pass over `rows` <= 8 requests: the BF16 head (this rank's fragment-ordered shard wf +//! [N / 16, K / 16, 256], or with excOff / excPos / excW / expBase its 12-bit coding, wf uint8 [N * K * 3 / 2]; +//! global ids vocabOffset..) over x[xRows[b]] (or x[b]), the global argmax across the TP ranks through the multicast +//! slots wsMcPtr / wsUc, then draftOut[b, step] = idsNext[b] = token, posNext[b] = posSrc[xRows[b] or b] + 1 and the +//! KV-length bookkeeping (see draftStep.h). +void minimaxM3Sm107Tp8DraftStep(torch::Tensor const& x, OptTensor const& xRows, int64_t rows, torch::Tensor const& wf, + int64_t vocabOffset, torch::Tensor& keys, torch::Tensor& counters, torch::Tensor& wsUc, int64_t wsMcPtr, + int64_t rank, int64_t world, torch::Tensor& draftOut, int64_t step, torch::Tensor& idsNext, + torch::Tensor const& posSrc, torch::Tensor& posNext, torch::Tensor& kvLens, OptTensor const& numAcc, + OptTensor const& seqLens, int64_t draftLen, int64_t numCtas, bool pdl, OptTensor const& outKeys, + OptTensor const& excOff, OptTensor const& excPos, OptTensor const& excW, int64_t expBase, int64_t tileLo, + int64_t tileHi, int64_t totalCtas, OptTensor const& bnd, OptTensor const& bndCnt, + std::optional> fold, int64_t foldRows, int64_t foldTokensPerStep, bool fixedOrder) +{ + checkArch(x); + for (torch::Tensor const* t : std::initializer_list{ + &wf, &keys, &counters, &wsUc, &draftOut, &idsNext, &posSrc, &posNext, &kvLens}) + { + TORCH_CHECK(t->device() == x.device(), "draft_step: every tensor must be on x's device"); + } + for (OptTensor const* t : std::initializer_list{ + &xRows, &numAcc, &seqLens, &outKeys, &excOff, &excPos, &excW, &bnd, &bndCnt}) + { + TORCH_CHECK(!t->has_value() || (*t)->device() == x.device(), "draft_step: every tensor must be on x's device"); + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + TORCH_CHECK(x.scalar_type() == at::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "draft_step: x must be contiguous BF16 [rows, K]"); + int const K = static_cast(x.size(1)); + TORCH_CHECK(K % 16 == 0, "draft_step: K must be a multiple of 16"); + TORCH_CHECK(rows >= 1 && rows <= ds::kMaxTokens, "draft_step: 1..", ds::kMaxTokens, " rows"); + bool const coded = excOff.has_value(); + TORCH_CHECK(coded == excPos.has_value() && coded == excW.has_value(), + "draft_step: exc_off, exc_pos and exc_w come together"); + int nTiles = 0; + if (coded) + { + TORCH_CHECK(K % 64 == 0 && K / 16 <= 0xFFFF, "draft_step: a coded head needs K a multiple of 64"); + TORCH_CHECK(excOff->scalar_type() == at::kInt && excOff->is_contiguous() && excOff->numel() >= 2, + "draft_step: exc_off must be int32 [tiles + 1]"); + nTiles = static_cast(excOff->numel() - 1); + TORCH_CHECK(wf.scalar_type() == at::kByte && wf.is_contiguous() + && wf.numel() == static_cast(nTiles) * 16 * K * 3 / 2, + "draft_step: a coded wf must be the contiguous uint8 stream of [tiles * 16, K]"); + TORCH_CHECK(excPos->scalar_type() == at::kInt && excPos->is_contiguous(), "draft_step: exc_pos must be int32"); + TORCH_CHECK(excW->scalar_type() == at::kFloat && excW->is_contiguous() && excW->numel() == 2 * excPos->numel(), + "draft_step: exc_w must be fp32 [E, 2]"); + TORCH_CHECK(expBase >= 0 && expBase <= 240, "draft_step: exponent base out of range"); + } + else + { + TORCH_CHECK(wf.scalar_type() == at::kBFloat16 && wf.dim() == 3 && wf.is_contiguous() && wf.size(1) == K / 16 + && wf.size(2) == 256, + "draft_step: wf must be the fragment-ordered BF16 weight [N / 16, K / 16, 256]"); + nTiles = static_cast(wf.size(0)); + } + int64_t const rangeHi = tileHi > 0 ? tileHi : nTiles; + TORCH_CHECK(tileLo >= 0 && tileLo < rangeHi && rangeHi <= nTiles, "draft_step: tile range"); + TORCH_CHECK(nTiles >= 1 && numCtas >= 1 && (rangeHi - tileLo + numCtas - 1) / numCtas <= ds::kMaxTilesPerCta, + "draft_step: too many weight tiles per CTA"); + TORCH_CHECK(totalCtas == 0 || totalCtas >= numCtas, "draft_step: total_ctas"); + TORCH_CHECK(bnd.has_value() == bndCnt.has_value(), "draft_step: bnd and bnd_cnt come together"); + if (bnd.has_value()) + { + TORCH_CHECK(bnd->scalar_type() == at::kFloat && bnd->is_contiguous() && bnd->numel() >= int64_t{nTiles} * 128, + "draft_step: bnd must be zero-filled fp32 [tiles, 32, 4]"); + TORCH_CHECK(bndCnt->scalar_type() == at::kInt && bndCnt->is_contiguous() && bndCnt->numel() >= nTiles, + "draft_step: bnd_cnt must be zero-filled int32 [tiles]"); + // At least one tile per CTA: a tile is then shared by at most two CTAs (two addends in bnd). + TORCH_CHECK(rangeHi - tileLo >= numCtas, "draft_step: an equal-chunk split needs a tile per CTA"); + } + TORCH_CHECK(vocabOffset >= 0 && vocabOffset + 16LL * nTiles <= 0xFFFFFFFFLL, "draft_step: vocab offset"); + if (xRows.has_value()) + { + TORCH_CHECK(xRows->scalar_type() == at::kLong && xRows->is_contiguous() && xRows->numel() >= rows, + "draft_step: x_rows must be contiguous int64 [rows]"); + } + else + { + TORCH_CHECK(x.size(0) >= rows, "draft_step: x must hold a row per request"); + } + TORCH_CHECK(world >= 1 && rank >= 0 && rank < world, "draft_step: rank / world"); + TORCH_CHECK(keys.scalar_type() == at::kLong && keys.numel() >= rows, "draft_step: keys must be int64 [>= rows]"); + TORCH_CHECK(counters.scalar_type() == at::kInt && counters.numel() >= 2, "draft_step: counters must be int32 [2]"); + TORCH_CHECK(wsUc.scalar_type() == at::kLong && wsUc.numel() >= 2 * ds::kMaxTokens * world, + "draft_step: ws_uc must hold [2][8][world] slots"); + TORCH_CHECK(draftOut.scalar_type() == at::kInt && draftOut.dim() == 2 && draftOut.stride(1) == 1 + && draftOut.size(0) >= rows && step >= 0 && step < draftOut.size(1), + "draft_step: draft_out must be int32 [rows, > step] with unit column stride"); + checkInt(idsNext, rows, "ids_next"); + checkInt(posNext, rows, "pos_next"); + checkInt(kvLens, rows, "kv_lens"); + TORCH_CHECK(posSrc.scalar_type() == at::kInt && posSrc.is_contiguous(), "draft_step: pos_src must be int32"); + TORCH_CHECK(numAcc.has_value() == seqLens.has_value(), "draft_step: num_acc and seq_lens come together"); + TORCH_CHECK(!numAcc.has_value() || step == 0, "draft_step: the KV rollback (num_acc) is pass 0's"); + if (numAcc.has_value()) + { + checkInt(*numAcc, rows, "num_acc"); + checkInt(*seqLens, rows, "seq_lens"); + } + if (outKeys.has_value()) + { + TORCH_CHECK(outKeys->scalar_type() == at::kLong && outKeys->is_contiguous() && outKeys->numel() >= rows, + "draft_step: out_keys must be contiguous int64 [rows]"); + } + ds::Params p{}; + p.x = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.x_rows = xRows.has_value() ? reinterpret_cast(xRows->data_ptr()) : nullptr; + p.M = static_cast(rows); + p.wf = reinterpret_cast(wf.data_ptr()); + p.n_tiles = nTiles; + p.vocab_offset = static_cast(vocabOffset); + p.keys = reinterpret_cast(keys.data_ptr()); + p.counters = reinterpret_cast(counters.data_ptr()); + p.ws_uc = reinterpret_cast(wsUc.data_ptr()); + p.ws_mc = reinterpret_cast(wsMcPtr); + p.rank = static_cast(rank); + p.world = static_cast(world); + p.out_keys = outKeys.has_value() ? reinterpret_cast(outKeys->data_ptr()) : nullptr; + p.st = ds::StepArgs{draftOut.data_ptr(), static_cast(draftOut.stride(0)), static_cast(step), + idsNext.data_ptr(), posSrc.data_ptr(), posNext.data_ptr(), kvLens.data_ptr(), + numAcc.has_value() ? numAcc->data_ptr() : nullptr, + seqLens.has_value() ? seqLens->data_ptr() : nullptr, static_cast(draftLen)}; + p.cd = ds::CodedArgs{coded ? excOff->data_ptr() : nullptr, + coded ? reinterpret_cast(excPos->data_ptr()) : nullptr, + coded ? reinterpret_cast(excW->data_ptr()) : nullptr, + static_cast((expBase << 7) | (expBase << 23))}; + p.tile_lo = static_cast(tileLo); + p.tile_hi = static_cast(tileHi); + p.total_ctas = static_cast(totalCtas); + p.bnd = bnd.has_value() ? reinterpret_cast(bnd->data_ptr()) : nullptr; + p.bnd_cnt = bnd.has_value() ? reinterpret_cast(bndCnt->data_ptr()) : nullptr; + if (fold.has_value()) + { + // The 14 fold-tail tensors in buildParams' order (ctrl, slots, src_new, dst_new, src_next, dst_next, + // src_draft, dst_draft, src_lens, dst_lens, ids, draft_ids, pos_off, kv_off): the in-kernel tail's operands, + // packed exactly as the standalone fold tail packs them. + std::vector f = *fold; + TORCH_CHECK(f.size() == 14, "minimax_m3_sm107tp8_draft_step: fold takes 14 tensors, got ", f.size()); + p.fold = minimaxM3Sm107Tp8FoldTailParams::buildParams(f[0], f[1], foldRows, f[2], f[3], f[4], f[5], f[6], f[7], + f[8], f[9], f[10], f[11], f[12], f[13], foldTokensPerStep); + p.hasFold = true; + } + p.fixedOrder = fixedOrder; + check(ds::invokeDraftStep(p, K, static_cast(numCtas), pdl, at::cuda::getCurrentCUDAStream(x.get_device()))); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +// Mutations through the raw multicast address ws_mc_ptr are not visible to the schema: that buffer is owned by the +// Python workspace and never aliased by the op's tensor arguments. +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107tp8_draft_step(Tensor x, Tensor? x_rows, int rows, Tensor wf, int vocab_offset, Tensor(a!) " + "keys, " + "Tensor(b!) counters, Tensor(c!) ws_uc, int ws_mc_ptr, int rank, int world, Tensor(d!) draft_out, int step, " + "Tensor(e!) ids_next, Tensor pos_src, Tensor(f!) pos_next, Tensor(g!) kv_lens, Tensor? num_acc, " + "Tensor(h!)? seq_lens, int draft_len, int num_ctas, bool pdl, Tensor(i!)? out_keys, Tensor? exc_off=None, " + "Tensor? exc_pos=None, Tensor? exc_w=None, int exp_base=0, int tile_lo=0, int tile_hi=0, int total_ctas=0, " + "Tensor(j!)? bnd=None, Tensor(k!)? bnd_cnt=None" + ", Tensor(l!)[]? fold=None, int fold_rows=0, int fold_tokens_per_step=0, bool fixed_order=False) " + "-> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp8_draft_step", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8DraftStep); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8DrafterRopeFp8KvOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8DrafterRopeFp8KvOp.cpp new file mode 100644 index 000000000000..ec93068fc374 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8DrafterRopeFp8KvOp.cpp @@ -0,0 +1,103 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace drk = tensorrt_llm::kernels::minimaxM3Sm107Tp8::drafterRopeFp8Kv; + +//! The MiniMax-M3 TP8 drafter layer's generation preprocess (kernels/minimaxM3Sm107Tp8/drafterRopeFp8Kv.h): RoPE on +//! the packed BF16 qkv rows, the E4M3 query into q_out, and the step's K / V rows into the paged E4M3 cache kv_pages +//! ([sub-pages, Hkv, tokens_per_block, 128]) at the sub-pages block_tables ([requests, 2, max blocks]) names. +void minimaxM3Sm107Tp8DrafterRopeFp8Kv(torch::Tensor const& qkv, torch::Tensor const& cosSin, + torch::Tensor const& seqLens, std::optional const& specPositionOffsets, torch::Tensor& kvPages, + torch::Tensor const& blockTables, torch::Tensor& qOut, int64_t qLen, int64_t numHeadsQ, int64_t numHeadsKv, + int64_t rotaryDim, int64_t tokensPerBlock, int64_t ropeMode, bool pdl, bool maxSharedCarveout) +{ + TORCH_CHECK(qkv.is_cuda() && qkv.is_contiguous() && qkv.dim() == 2 && qkv.scalar_type() == at::kBFloat16); + TORCH_CHECK(qkv.size(1) == (numHeadsQ + 2 * numHeadsKv) * drk::kHeadDim, "qkv is not [tokens, (Hq + 2 Hkv) * 128]"); + TORCH_CHECK(qLen >= 1 && qkv.size(0) % qLen == 0, "the rows are not whole passes of q_len rows"); + TORCH_CHECK(rotaryDim > 0 && rotaryDim <= drk::kHeadDim && rotaryDim % 16 == 0, "unsupported rotary_dim"); + TORCH_CHECK(cosSin.is_cuda() && cosSin.is_contiguous() && cosSin.scalar_type() == at::kFloat + && cosSin.numel() % rotaryDim == 0, + "cos_sin must be the fp32 [positions, rotary_dim / 2] (cos, sin) table"); + int64_t const requests = qkv.size(0) / qLen; + TORCH_CHECK(seqLens.is_cuda() && seqLens.scalar_type() == at::kInt && seqLens.numel() >= requests); + if (specPositionOffsets.has_value()) + { + TORCH_CHECK(specPositionOffsets->is_cuda() && specPositionOffsets->is_contiguous() + && specPositionOffsets->scalar_type() == at::kInt && specPositionOffsets->numel() >= requests * qLen); + } + TORCH_CHECK(kvPages.is_cuda() && kvPages.is_contiguous() && kvPages.dim() == 4 + && kvPages.scalar_type() == at::kFloat8_e4m3fn && kvPages.size(1) == numHeadsKv + && kvPages.size(2) == tokensPerBlock && kvPages.size(3) == drk::kHeadDim, + "kv_pages must be the E4M3 [sub-pages, Hkv, tokens_per_block, 128] cache"); + TORCH_CHECK(blockTables.is_cuda() && blockTables.scalar_type() == at::kInt && blockTables.dim() == 3 + && blockTables.size(0) >= requests && blockTables.size(1) == 2 && blockTables.stride(2) == 1, + "block_tables must be the int32 [requests, 2, max blocks] K / V sub-page table"); + TORCH_CHECK(qOut.is_cuda() && qOut.is_contiguous() && qOut.scalar_type() == at::kFloat8_e4m3fn + && qOut.numel() == qkv.size(0) * numHeadsQ * drk::kHeadDim); + c10::cuda::CUDAGuard const deviceGuard{qkv.device()}; + drk::Params params{}; + params.qkv = reinterpret_cast<__nv_bfloat16 const*>(qkv.data_ptr()); + params.cosSin = reinterpret_cast(cosSin.data_ptr()); + params.seqLens = seqLens.data_ptr(); + params.specPositionOffsets = specPositionOffsets.has_value() ? specPositionOffsets->data_ptr() : nullptr; + params.qOut = reinterpret_cast<__nv_fp8_e4m3*>(qOut.data_ptr()); + params.pool = reinterpret_cast<__nv_fp8_e4m3*>(kvPages.data_ptr()); + params.blockTable = blockTables.data_ptr(); + params.blockTableRequestStride = blockTables.stride(0); + params.blockTableKvStride = blockTables.stride(1); + params.numTokens = static_cast(qkv.size(0)); + params.qLen = static_cast(qLen); + params.numHeadsQ = static_cast(numHeadsQ); + params.numHeadsKv = static_cast(numHeadsKv); + params.rotaryDim = static_cast(rotaryDim); + params.tokensPerBlock = static_cast(tokensPerBlock); + params.ropeMode = static_cast(ropeMode); + cudaError_t const err + = drk::launch(params, pdl, maxSharedCarveout, at::cuda::getCurrentCUDAStream(qkv.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107tp8_drafter_rope_fp8_kv: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107tp8_drafter_rope_fp8_kv(Tensor qkv, Tensor cos_sin, Tensor seq_lens, " + "Tensor? spec_position_offsets, Tensor(a!) kv_pages, Tensor block_tables, Tensor(b!) q_out, int q_len, " + "int num_heads_q, int num_heads_kv, int rotary_dim, int tokens_per_block, int rope_mode, bool pdl, bool " + "max_shared_carveout=True) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp8_drafter_rope_fp8_kv", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8DrafterRopeFp8Kv); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailOp.cpp new file mode 100644 index 000000000000..e4d5ca2f6b54 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailOp.cpp @@ -0,0 +1,89 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.h" +#include "tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailParams.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The speculative step's forward-tail store and next-step staging in one launch (the C++ form of the Triton +//! kernels/m3_fold_tail.py, the same writes). The caller checks the row counts against the buffers (fold_tail). +static void foldTailImpl(bool oneWarp, torch::Tensor const& ctrl, torch::Tensor const& slots, int64_t rows, + torch::Tensor const& srcNew, torch::Tensor& dstNew, torch::Tensor const& srcNext, torch::Tensor& dstNext, + torch::Tensor const& srcDraft, torch::Tensor& dstDraft, torch::Tensor const& srcLens, torch::Tensor& dstLens, + torch::Tensor& ids, torch::Tensor& draftIds, torch::Tensor& posOff, torch::Tensor& kvOff, int64_t tokensPerStep) +{ + c10::cuda::CUDAGuard const deviceGuard{ctrl.device()}; + auto const p = minimaxM3Sm107Tp8FoldTailParams::buildParams(ctrl, slots, rows, srcNew, dstNew, srcNext, dstNext, + srcDraft, dstDraft, srcLens, dstLens, ids, draftIds, posOff, kvOff, tokensPerStep); + auto const stream = at::cuda::getCurrentCUDAStream(ctrl.get_device()); + auto const err = oneWarp + ? tensorrt_llm::kernels::minimaxM3Sm107Tp8::invokeFoldTailOneWarp(p, stream) + : tensorrt_llm::kernels::minimaxM3Sm107Tp8::invokeFoldTail(p, static_cast(rows), stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp8_fold_tail launch failed: ", cudaGetErrorString(err)); +} + +//! The tail as its standalone kernel (one CTA per row). +void minimaxM3Sm107Tp8FoldTail(torch::Tensor const& ctrl, torch::Tensor const& slots, int64_t rows, + torch::Tensor const& srcNew, torch::Tensor& dstNew, torch::Tensor const& srcNext, torch::Tensor& dstNext, + torch::Tensor const& srcDraft, torch::Tensor& dstDraft, torch::Tensor const& srcLens, torch::Tensor& dstLens, + torch::Tensor& ids, torch::Tensor& draftIds, torch::Tensor& posOff, torch::Tensor& kvOff, int64_t tokensPerStep) +{ + foldTailImpl(false, ctrl, slots, rows, srcNew, dstNew, srcNext, dstNext, srcDraft, dstDraft, srcLens, dstLens, ids, + draftIds, posOff, kvOff, tokensPerStep); +} + +//! The tail's in-kernel form (one warp over the rows, foldTailWarp), the oracle of a caller that folds it in. +void minimaxM3Sm107Tp8FoldTailOneWarp(torch::Tensor const& ctrl, torch::Tensor const& slots, int64_t rows, + torch::Tensor const& srcNew, torch::Tensor& dstNew, torch::Tensor const& srcNext, torch::Tensor& dstNext, + torch::Tensor const& srcDraft, torch::Tensor& dstDraft, torch::Tensor const& srcLens, torch::Tensor& dstLens, + torch::Tensor& ids, torch::Tensor& draftIds, torch::Tensor& posOff, torch::Tensor& kvOff, int64_t tokensPerStep) +{ + foldTailImpl(true, ctrl, slots, rows, srcNew, dstNew, srcNext, dstNext, srcDraft, dstDraft, srcLens, dstLens, ids, + draftIds, posOff, kvOff, tokensPerStep); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp8_fold_tail(Tensor ctrl, Tensor slots, int rows, Tensor src_new, Tensor(a!) dst_new, " + "Tensor src_next, Tensor(b!) dst_next, Tensor src_draft, Tensor(c!) dst_draft, Tensor src_lens, " + "Tensor(d!) dst_lens, Tensor(e!) ids, Tensor(f!) draft_ids, Tensor(g!) pos_off, Tensor(h!) kv_off, " + "int tokens_per_step) -> ()"); + m.def( + "minimax_m3_sm107_tp8_fold_tail_one_warp(Tensor ctrl, Tensor slots, int rows, Tensor src_new, Tensor(a!) " + "dst_new, Tensor src_next, Tensor(b!) dst_next, Tensor src_draft, Tensor(c!) dst_draft, Tensor src_lens, " + "Tensor(d!) dst_lens, Tensor(e!) ids, Tensor(f!) draft_ids, Tensor(g!) pos_off, Tensor(h!) kv_off, " + "int tokens_per_step) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp8_fold_tail", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8FoldTail); + m.impl("minimax_m3_sm107_tp8_fold_tail_one_warp", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8FoldTailOneWarp); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailParams.h b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailParams.h new file mode 100644 index 000000000000..cc5cde28c9d9 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FoldTailParams.h @@ -0,0 +1,111 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/foldTail.h" + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ +namespace minimaxM3Sm107Tp8FoldTailParams +{ + +constexpr int64_t kMaxWidth = 32; + +inline void check2d(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt32 && t.dim() == 2 && t.size(1) <= kMaxWidth, + "minimax_m3_sm107_tp8_fold_tail: ", name, " must be a 2-D int32 CUDA view at most 32 wide"); +} + +inline void check1d(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt32 && t.dim() == 1 && t.stride(0) == 1, + "minimax_m3_sm107_tp8_fold_tail: ", name, " must be a contiguous 1-D int32 CUDA tensor"); +} + +//! The fold tail's operands checked and packed (the standalone fold tail op and any op that runs the tail in-kernel, +//! e.g. the drafter's last-pass draft_step, build them identically). +inline tensorrt_llm::kernels::minimaxM3Sm107Tp8::FoldTailParams buildParams(torch::Tensor const& ctrl, + torch::Tensor const& slots, int64_t rows, torch::Tensor const& srcNew, torch::Tensor& dstNew, + torch::Tensor const& srcNext, torch::Tensor& dstNext, torch::Tensor const& srcDraft, torch::Tensor& dstDraft, + torch::Tensor const& srcLens, torch::Tensor& dstLens, torch::Tensor& ids, torch::Tensor& draftIds, + torch::Tensor& posOff, torch::Tensor& kvOff, int64_t tokensPerStep) +{ + TORCH_CHECK(ctrl.is_cuda() && ctrl.scalar_type() == torch::kInt32 && ctrl.numel() >= 1, + "minimax_m3_sm107_tp8_fold_tail: ctrl must be an int32 CUDA tensor"); + TORCH_CHECK(slots.is_cuda() && slots.scalar_type() == torch::kInt64 && slots.dim() == 1 && slots.stride(0) == 1, + "minimax_m3_sm107_tp8_fold_tail: slots must be a contiguous 1-D int64 CUDA tensor"); + TORCH_CHECK(rows > 0 && tokensPerStep > 0 && tokensPerStep <= kMaxWidth, + "minimax_m3_sm107_tp8_fold_tail: rows must be positive and tokens_per_step in [1, 32]"); + check2d(srcNew, "src_new"); + check2d(dstNew, "dst_new"); + check2d(srcNext, "src_next"); + check2d(dstNext, "dst_next"); + check2d(srcDraft, "src_draft"); + check2d(dstDraft, "dst_draft"); + check1d(srcLens, "src_lens"); + check1d(dstLens, "dst_lens"); + check1d(ids, "ids"); + check1d(draftIds, "draft_ids"); + check1d(posOff, "pos_off"); + check1d(kvOff, "kv_off"); + tensorrt_llm::kernels::minimaxM3Sm107Tp8::FoldTailParams p{}; + p.ctrl = ctrl.data_ptr(); + p.slots = slots.data_ptr(); + p.srcNew = srcNew.data_ptr(); + p.sn0 = srcNew.stride(0); + p.sn1 = srcNew.stride(1); + p.snW = static_cast(srcNew.size(1)); + p.dstNew = dstNew.data_ptr(); + p.dn0 = dstNew.stride(0); + p.dn1 = dstNew.stride(1); + p.dnW = static_cast(dstNew.size(1)); + p.srcNext = srcNext.data_ptr(); + p.sx0 = srcNext.stride(0); + p.sx1 = srcNext.stride(1); + p.sxW = static_cast(srcNext.size(1)); + p.dstNext = dstNext.data_ptr(); + p.dx0 = dstNext.stride(0); + p.dx1 = dstNext.stride(1); + p.dxW = static_cast(dstNext.size(1)); + p.srcDraft = srcDraft.data_ptr(); + p.sd0 = srcDraft.stride(0); + p.sd1 = srcDraft.stride(1); + p.sdW = static_cast(srcDraft.size(1)); + p.dstDraft = dstDraft.data_ptr(); + p.dd0 = dstDraft.stride(0); + p.dd1 = dstDraft.stride(1); + p.ddW = static_cast(dstDraft.size(1)); + p.srcLens = srcLens.data_ptr(); + p.dstLens = dstLens.data_ptr(); + p.ids = ids.data_ptr(); + p.draftIds = draftIds.data_ptr(); + p.posOff = posOff.data_ptr(); + p.kvOff = kvOff.data_ptr(); + p.tokensPerStep = static_cast(tokensPerStep); + return p; +} + +} // namespace minimaxM3Sm107Tp8FoldTailParams +} // namespace torch_ext + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FullCtxReduceOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FullCtxReduceOp.cpp new file mode 100644 index 000000000000..29b9a0ab391b --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8FullCtxReduceOp.cpp @@ -0,0 +1,92 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/fullCtxReduce.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace fcr = tensorrt_llm::kernels::minimaxM3Sm107Tp8::fullCtxReduce; + +//! Merges the PrimTS split-KV decode scratch (partial_o [B, Hkv, S, R, 128] 16-bit, partial_stats [B, Hkv, S, R] +//! fp32 log2-LSE) of the MiniMax-M3 TP8 full-context decode into out (BF16 [B * SQ, Hkv * hR, 128]); with q8 / sf +//! also its MXFP8 copy (E4M3 [>= B * SQ, Hkv * hR * 128] + 128x4-swizzled UE8M0 scales, as mxfp8_quantize(out, True)). +void minimaxM3Sm107Tp8FullCtxReduce(torch::Tensor const& partialO, torch::Tensor const& partialStats, + torch::Tensor const& seqLensKv, torch::Tensor& out, int64_t seqLenQ, int64_t qTokensPerCta, int64_t tileKv, + int64_t instsKv, bool causal, bool pdl, std::optional const& q8, + std::optional const& sf) +{ + TORCH_CHECK(partialO.is_cuda() && partialO.is_contiguous() && partialO.dim() == 5); + TORCH_CHECK(partialO.scalar_type() == at::kHalf || partialO.scalar_type() == at::kBFloat16); + TORCH_CHECK(partialO.size(4) == fcr::kHeadDim && partialO.size(2) <= fcr::kMaxSplits); + TORCH_CHECK(partialStats.scalar_type() == at::kFloat && partialStats.is_contiguous() + && partialStats.sizes() == partialO.sizes().slice(0, 4)); + TORCH_CHECK(seqLensKv.scalar_type() == at::kInt && seqLensKv.numel() >= partialO.size(0)); + int const rows = static_cast(partialO.size(3)); + TORCH_CHECK(seqLenQ >= 1 && rows % seqLenQ == 0); + int const headsPerKv = rows / static_cast(seqLenQ); + TORCH_CHECK(out.scalar_type() == at::kBFloat16 && out.is_contiguous() + && out.numel() == partialO.size(0) * seqLenQ * partialO.size(1) * headsPerKv * fcr::kHeadDim); + c10::cuda::CUDAGuard const deviceGuard{partialO.device()}; + fcr::Params params{partialO.data_ptr(), partialStats.data_ptr(), seqLensKv.data_ptr(), + reinterpret_cast<__nv_bfloat16*>(out.data_ptr()), static_cast(partialO.size(0)), + static_cast(partialO.size(1)), static_cast(partialO.size(2)), rows, static_cast(seqLenQ), + headsPerKv, static_cast(qTokensPerCta), static_cast(tileKv), static_cast(instsKv), causal, + partialO.scalar_type() == at::kBFloat16}; + TORCH_CHECK(q8.has_value() == sf.has_value()); + if (q8.has_value()) + { + int64_t const width = partialO.size(1) * headsPerKv * fcr::kHeadDim; + int64_t const tokens = partialO.size(0) * seqLenQ; + int64_t const tileRows = (tokens + 127) / 128 * 128; + TORCH_CHECK(q8->is_cuda() && q8->is_contiguous() && q8->scalar_type() == at::kFloat8_e4m3fn && q8->dim() == 2 + && q8->size(1) == width && q8->size(0) >= tokens); + TORCH_CHECK(sf->is_cuda() && sf->is_contiguous() && sf->scalar_type() == at::kByte + && sf->numel() >= tileRows * (width / 32)); + params.q8 = reinterpret_cast(q8->data_ptr()); + params.sf = sf->data_ptr(); + params.numSfKTiles = static_cast(width / 128); + } + cudaError_t const err + = fcr::invokeFullCtxReduce(params, pdl, at::cuda::getCurrentCUDAStream(partialO.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107tp8_fullctx_reduce: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107tp8_fullctx_reduce(Tensor partial_o, Tensor partial_stats, Tensor seq_lens_kv, " + "Tensor(a!) out, int seq_len_q, int q_tokens_per_cta, int tile_kv, int insts_kv, bool causal, bool pdl, " + "Tensor(b!)? q8=None, Tensor(c!)? sf=None) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp8_fullctx_reduce", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8FullCtxReduce); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8McAllreduceNormOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8McAllreduceNormOp.cpp new file mode 100644 index 000000000000..eb3759b28a58 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8McAllreduceNormOp.cpp @@ -0,0 +1,181 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/mcAllreduceNormTp8.h" + +#include +#include +#include + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kMaxTokens = 48; +constexpr int64_t kCluster = 8; +constexpr int64_t kMaxWorld = 8; + +} // namespace + +//! MiniMax-M3 SM107 TP8 multicast all-reduce (stock rounding points: BF16 rank sum, then the residual add) + residual + +//! RMSNorm (+ optional MoE finalize, + optional MXFP8 output). +//! +//! @param x BF16 [M, H] partial, or with `perm` the permuted FC2 output [rows, H]. +//! @param x2 Optional BF16 [M, H] second addend (the shared expert in finalize mode). +//! @param residual BF16 [M, H], 1 <= M <= 48. +//! @param normWeight BF16 [H]. +//! @param wsUc This rank's unicast view of the exchange workspace (int32 words); re-armed in place. +//! @param wsMcPtr Device address of the workspace's multicast mapping. +//! @param flags Int32 per-CTA call-parity counters, at least maxt * 8; updated in place. +//! @param cluster Cluster size; 8 is the only one built. +//! @param perm, scales Finalize mode: int32 [M, topk] permuted-row index and fp32 / bf16 [M, topk] routing weights. +//! @param maxt Token capacity of the workspace bucket (8, 16 or 48). +//! @param quant Also return the MXFP8 copy of the normed rows (E4M3 [M, H], swizzled UE8M0 scales). +//! @param prepushed 0, or 2 / 3 when producers already pushed this rank's rows (see the kernel header). +//! @param opts Epilogue variant; every variant is bit-identical. +//! @param capOut Optional BF16 [>= M, H] view (row stride a multiple of 8, 16-byte aligned) that receives the Eagle3 +//! capture of the updated residual: m3_spec_tail capture_norm with capW [H] and capEps, bit for bit (H 6144; not the +//! drafter boundary, opts bit 16). +//! @return [normed, updated residual] or, with `quant`, [normed, updated residual, E4M3 rows, scale bytes]. +std::vector minimaxM3Sm107Tp8McAllreduceNorm(torch::Tensor const& x, + std::optional const& x2, torch::Tensor const& residual, torch::Tensor const& normWeight, double eps, + torch::Tensor& wsUc, int64_t wsMcPtr, torch::Tensor& flags, int64_t rank, int64_t world, int64_t cluster, bool pdl, + std::optional const& perm, std::optional const& scales, int64_t maxt, bool quant, + int64_t prepushed, int64_t opts, std::optional const& capOut, + std::optional const& capW, double capEps) +{ + int const sm = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sm == 107, "minimax_m3_sm107_tp8_mc_allreduce_norm requires SM107, got SM", sm); + TORCH_CHECK(cluster == kCluster, "minimax_m3_sm107_tp8_mc_allreduce_norm: only cluster 8 is built, got ", cluster); + TORCH_CHECK(tensorrt_llm::kernels::isMinimaxM3Sm107Tp8McAllreduceNormSupported( + static_cast(maxt), static_cast(opts)), + "minimax_m3_sm107_tp8_mc_allreduce_norm: unsupported (bucket ", maxt, ", opts ", opts, ")"); + TORCH_CHECK( + world >= 1 && world <= kMaxWorld && rank >= 0 && rank < world, "invalid rank ", rank, " / world ", world); + + tensorrt_llm::kernels::MinimaxM3Sm107Tp8McAllreduceNormParams p; + if (x2.has_value()) + { + TORCH_CHECK(x2->scalar_type() == at::kBFloat16 && x2->is_contiguous() && x2->sizes() == residual.sizes(), + "x2 must be a contiguous bf16 tensor shaped like residual"); + p.x2 = reinterpret_cast<__nv_bfloat16 const*>(x2->data_ptr()); + } + if (perm.has_value()) + { + // Finalize mode: x is the permuted FC2 output [rows, H]. + TORCH_CHECK(scales.has_value() && perm->scalar_type() == at::kInt && perm->is_contiguous() && perm->dim() == 2 + && perm->size(0) == residual.size(0), + "perm must be a contiguous int32 [M, topk] tensor, with scales"); + TORCH_CHECK(scales->is_contiguous() && scales->numel() == perm->numel() + && (scales->scalar_type() == at::kFloat || scales->scalar_type() == at::kBFloat16), + "scales must be a contiguous fp32 or bf16 tensor with one weight per perm entry"); + TORCH_CHECK(x.dim() == 2 && x.size(1) == residual.size(1), "fc2 must be [rows, H]"); + p.fc2 = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.perm = perm->data_ptr(); + p.scales = scales->data_ptr(); + p.scaleFp32 = scales->scalar_type() == at::kFloat; + p.topk = static_cast(perm->size(1)); + } + TORCH_CHECK(x.is_cuda() && x.scalar_type() == at::kBFloat16 && x.is_contiguous() && x.dim() == 2, + "x must be a contiguous 2-D bf16 CUDA tensor"); + TORCH_CHECK(residual.dim() == 2 && residual.size(0) >= 1 && residual.size(0) <= std::min(kMaxTokens, maxt), + "residual must hold 1..min(48, bucket) rows"); + TORCH_CHECK(residual.scalar_type() == at::kBFloat16 && residual.is_contiguous() + && (perm.has_value() || residual.sizes() == x.sizes()), + "residual must be a contiguous bf16 tensor shaped like x"); + TORCH_CHECK( + normWeight.scalar_type() == at::kBFloat16 && normWeight.numel() == residual.size(1), "norm_w must be bf16 [H]"); + TORCH_CHECK(wsUc.scalar_type() == at::kInt && wsUc.is_cuda(), "ws_uc must be an int32 CUDA tensor"); + TORCH_CHECK(flags.scalar_type() == at::kInt && flags.is_contiguous() && flags.numel() >= maxt * kCluster, + "flags must be a contiguous int32 tensor of at least bucket * 8 counters"); + + int64_t const m = residual.size(0), h = residual.size(1); + TORCH_CHECK(!quant || h % 128 == 0, "MXFP8 output needs H % 128 == 0"); + c10::cuda::CUDAGuard const deviceGuard{residual.device()}; + auto outNormed = torch::empty_like(residual); + auto outResidual = torch::empty_like(residual); + auto outQ = quant ? torch::empty({m, h}, residual.options().dtype(at::kFloat8_e4m3fn)) : torch::Tensor(); + auto outSf = quant ? torch::empty({(m + 127) / 128 * 128 * (h / 32)}, residual.options().dtype(at::kByte)) + : torch::Tensor(); + + p.x = reinterpret_cast<__nv_bfloat16 const*>(x.data_ptr()); + p.numTokens = static_cast(m); + p.hidden = static_cast(h); + p.residual = reinterpret_cast<__nv_bfloat16 const*>(residual.data_ptr()); + p.normWeight = reinterpret_cast<__nv_bfloat16 const*>(normWeight.data_ptr()); + p.eps = static_cast(eps); + p.wsUc = wsUc.data_ptr(); + p.wsMc = reinterpret_cast(wsMcPtr); + p.flags = flags.data_ptr(); + p.rank = static_cast(rank); + p.world = static_cast(world); + p.outNormed = reinterpret_cast<__nv_bfloat16*>(outNormed.data_ptr()); + p.outResidual = reinterpret_cast<__nv_bfloat16*>(outResidual.data_ptr()); + p.outQuant = quant ? reinterpret_cast(outQ.data_ptr()) : nullptr; + p.outSf = quant ? outSf.data_ptr() : nullptr; + p.prepushed = static_cast(prepushed); + if (capOut.has_value()) + { + TORCH_CHECK((opts & 16) == 0, "the capture epilogue is not built for the drafter boundary (opts bit 16)"); + TORCH_CHECK(capOut->scalar_type() == at::kBFloat16 && capOut->dim() == 2 && capOut->size(0) >= m + && capOut->size(1) == h && capOut->stride(1) == 1 && capOut->stride(0) % 8 == 0 + && reinterpret_cast(capOut->data_ptr()) % 16 == 0 && capOut->device() == residual.device(), + "cap_out must be a bf16 [>= M, H] view with unit column stride, a row stride that is a multiple of 8 and " + "16-byte alignment"); + TORCH_CHECK( + capW.has_value() && capW->scalar_type() == at::kBFloat16 && capW->is_contiguous() && capW->numel() == h, + "cap_w must be a contiguous bf16 [H] tensor"); + p.capOut = reinterpret_cast<__nv_bfloat16*>(capOut->data_ptr()); + p.capStride = capOut->stride(0); + p.capWeight = reinterpret_cast<__nv_bfloat16 const*>(capW->data_ptr()); + p.capEps = static_cast(capEps); + } + tensorrt_llm::kernels::invokeMinimaxM3Sm107Tp8McAllreduceNorm(p, static_cast(maxt), + static_cast(opts), pdl, at::cuda::getCurrentCUDAStream(residual.get_device())); + if (quant) + { + return {outNormed, outResidual, outQ, outSf}; + } + return {outNormed, outResidual}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp8_mc_allreduce_norm(Tensor x, Tensor? x2, Tensor residual, Tensor norm_w, float eps, " + "Tensor(a!) ws_uc, int ws_mc_ptr, Tensor(b!) flags, int rank, int world, int cluster, bool pdl, " + "Tensor? perm, Tensor? scales, int maxt, bool quant, int prepushed, int opts, Tensor(c!)? cap_out=None, " + "Tensor? cap_w=None, float cap_eps=0.) -> Tensor[]"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp8_mc_allreduce_norm", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8McAllreduceNorm); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8ProducerPwOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8ProducerPwOp.cpp new file mode 100644 index 000000000000..acd7061eadbb --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8ProducerPwOp.cpp @@ -0,0 +1,185 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/producerPw8.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +namespace pw = tensorrt_llm::kernels::minimaxM3Sm107Tp8::producerPw8; + +constexpr int64_t kHeadDim = 128; +constexpr int64_t kPageSize = 128; + +void checkCuda(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda(), "sm107_tp8_producer_pw: ", name, " must be a CUDA tensor"); +} + +} // namespace + +//! The SM107 TP8 NVFP4 QKV + indexer norm/RoPE/KV-insert producer (same arguments and outputs as +//! minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert at 8 query, 1 KV and 1 index head, head_dim 128, rotary_dim 64): +//! returns (q FP8 [T, 8, 128], index_q FP8 [T, 1, 128]) and writes the NVFP4 K/V of the KV head and the FP8 index-K +//! of each token's cache slot. `early` triggers the dependents on entry; `prewait_step_inputs` reads the position ids, +//! the slots and the RoPE rows before the grid-dependency wait (none of them may be written by a PDL predecessor). +std::vector minimaxM3Sm107Tp8ProducerPw(torch::Tensor const& packed, torch::Tensor& kvData, + torch::Tensor& kvScale, torch::Tensor& indexK, torch::Tensor const& outCacheLoc, torch::Tensor const& kvQuantScale, + double eps, torch::Tensor const& qWeight, torch::Tensor const& kWeight, torch::Tensor const& indexQWeight, + torch::Tensor const& indexKWeight, torch::Tensor const& rotaryCosSin, torch::Tensor const& positionIds, bool pdl, + bool early, bool prewaitStepInputs) +{ + checkCuda(packed, "packed"); + for (torch::Tensor const* t : std::initializer_list{&kvData, &kvScale, &indexK, &outCacheLoc, + &kvQuantScale, &qWeight, &kWeight, &indexQWeight, &indexKWeight, &rotaryCosSin, &positionIds}) + { + TORCH_CHECK(t->is_cuda() && t->get_device() == packed.get_device(), + "sm107_tp8_producer_pw: every operand must be on packed's CUDA device"); + } + TORCH_CHECK(reinterpret_cast(packed.data_ptr()) % 8 == 0 + && reinterpret_cast(kvData.data_ptr()) % 2 == 0 + && reinterpret_cast(indexK.data_ptr()) % 4 == 0, + "sm107_tp8_producer_pw: packed and the caches must start at the vector-load / store alignment"); + c10::cuda::CUDAGuard const deviceGuard{packed.device()}; + static int const sSmVersion = tensorrt_llm::common::getSMVersion(); + TORCH_CHECK(sSmVersion >= 100, "minimax_m3_sm107_tp8_producer_pw requires SM100 or newer"); + int64_t const headsQ = pw::kHeadsQ; + int64_t const headsKV = pw::kHeadsKV; + int64_t const headsIndex = pw::kHeadsIndex; + int64_t const totalHeads = headsQ + 2 * headsKV + headsIndex + 1; + TORCH_CHECK(packed.scalar_type() == at::kBFloat16 && packed.dim() == 2 && packed.size(1) == totalHeads * kHeadDim + && packed.is_contiguous(), + "sm107_tp8_producer_pw: packed must be contiguous BF16 [T, 12 * 128]"); + // The kernel's layout (the native producer's checkMinimaxM3Nvfp4HndKVPools at one KV head): HND pools + // [pages, 2 planes, 1 head, 128 tokens, 64 data / 8 scale bytes], dense token rows and head blocks, planes and + // pages that do not overlap. + TORCH_CHECK(kvData.scalar_type() == at::kByte && kvData.dim() == 5 && kvData.size(0) > 0 && kvData.size(1) == 2 + && kvData.size(2) == headsKV && kvData.size(3) == kPageSize && kvData.size(4) == kHeadDim / 2, + "sm107_tp8_producer_pw: kv_data must be the NVFP4 pool [pages, 2, 1, 128, 64] uint8"); + TORCH_CHECK(kvScale.scalar_type() == at::kByte && kvScale.dim() == 5 && kvScale.size(0) == kvData.size(0) + && kvScale.size(1) == 2 && kvScale.size(2) == headsKV && kvScale.size(3) == kPageSize + && kvScale.size(4) == kHeadDim / 16, + "sm107_tp8_producer_pw: kv_scale must be the E4M3 scale pool [pages, 2, 1, 128, 8] uint8"); + TORCH_CHECK(kvData.stride(4) == 1 && kvData.stride(3) == kvData.size(4) + && kvData.stride(2) == kvData.size(3) * kvData.stride(3), + "sm107_tp8_producer_pw: kv_data must have dense token rows and head blocks"); + TORCH_CHECK(kvScale.stride(4) == 1 && kvScale.stride(3) == kvScale.size(4) + && kvScale.stride(2) == kvScale.size(3) * kvScale.stride(3), + "sm107_tp8_producer_pw: kv_scale must have dense token rows and head blocks"); + TORCH_CHECK(kvData.stride(1) >= kvData.size(2) * kvData.stride(2) && kvData.stride(0) >= 2 * kvData.stride(1) + && kvScale.stride(1) >= kvScale.size(2) * kvScale.stride(2) && kvScale.stride(0) >= 2 * kvScale.stride(1), + "sm107_tp8_producer_pw: pool planes or pages overlap"); + TORCH_CHECK(kvData.stride(0) % 2 == 0 && kvData.stride(1) % 2 == 0, + "sm107_tp8_producer_pw: kv_data page and plane strides must keep the uint16 stores aligned"); + TORCH_CHECK(indexK.scalar_type() == at::kFloat8_e4m3fn && indexK.dim() == 4 && indexK.size(0) == kvData.size(0) + && indexK.size(1) == 1 && indexK.size(2) == kPageSize && indexK.size(3) == kHeadDim && indexK.stride(3) == 1 + && indexK.stride(2) == kHeadDim && indexK.stride(0) >= kPageSize * kHeadDim && indexK.stride(0) % 4 == 0, + "sm107_tp8_producer_pw: index_k must be FP8 [pages, 1, 128, 128] with a page stride keeping the 32-bit stores " + "aligned"); + int64_t const numTokens = packed.size(0); + TORCH_CHECK( + outCacheLoc.scalar_type() == at::kInt && outCacheLoc.is_contiguous() && outCacheLoc.numel() >= numTokens, + "sm107_tp8_producer_pw: out_cache_loc must be int32 [>= T]"); + TORCH_CHECK( + positionIds.scalar_type() == at::kInt && positionIds.is_contiguous() && positionIds.numel() >= numTokens, + "sm107_tp8_producer_pw: position_ids must be int32 [>= T]"); + TORCH_CHECK(kvQuantScale.scalar_type() == at::kFloat && kvQuantScale.is_contiguous() && kvQuantScale.numel() >= 3, + "sm107_tp8_producer_pw: kv_quant_scale must be fp32 [>= 3]"); + TORCH_CHECK(rotaryCosSin.scalar_type() == at::kFloat && rotaryCosSin.is_contiguous() && rotaryCosSin.dim() == 3 + && rotaryCosSin.size(1) == 2 && rotaryCosSin.size(2) == 32 + && reinterpret_cast(rotaryCosSin.data_ptr()) % 16 == 0, + "sm107_tp8_producer_pw: rotary_cos_sin must be fp32 [positions, 2, 32], 16-byte aligned (read as float4)"); + for (auto const* w : {&qWeight, &kWeight, &indexQWeight, &indexKWeight}) + { + TORCH_CHECK(w->scalar_type() == at::kBFloat16 && w->is_contiguous() && w->numel() == kHeadDim + && reinterpret_cast(w->data_ptr()) % 8 == 0, + "sm107_tp8_producer_pw: norm weights must be BF16 [128], 8-byte aligned (read as 4-value words)"); + } + auto const fp8 = packed.options().dtype(at::kFloat8_e4m3fn); + auto qOut = torch::empty({numTokens, headsQ, kHeadDim}, fp8); + auto indexQOut = torch::empty({numTokens, headsIndex, kHeadDim}, fp8); + if (numTokens == 0) + { + return {qOut, indexQOut}; + } + pw::Params p{}; + p.packed = reinterpret_cast<__nv_bfloat16 const*>(packed.data_ptr()); + p.qOut = reinterpret_cast<__nv_fp8_e4m3*>(qOut.data_ptr()); + p.indexQOut = reinterpret_cast<__nv_fp8_e4m3*>(indexQOut.data_ptr()); + p.kvData = kvData.data_ptr(); + p.kvScale = kvScale.data_ptr(); + p.indexK = reinterpret_cast<__nv_fp8_e4m3*>(indexK.data_ptr()); + p.outCacheLoc = outCacheLoc.data_ptr(); + p.kvQuantScale = kvQuantScale.data_ptr(); + p.dataPageStride = kvData.stride(0); + p.dataPlaneStride = kvData.stride(1); + p.dataHeadStride = kvData.stride(2); + p.dataTokenStride = kvData.stride(3); + p.scalePageStride = kvScale.stride(0); + p.scalePlaneStride = kvScale.stride(1); + p.scaleHeadStride = kvScale.stride(2); + p.indexPageStride = indexK.stride(0); + p.indexTokenStride = indexK.stride(2); + p.numPages = kvData.size(0); + p.numTokens = static_cast(numTokens); + p.eps = static_cast(eps); + p.qWeight = reinterpret_cast<__nv_bfloat16 const*>(qWeight.data_ptr()); + p.kWeight = reinterpret_cast<__nv_bfloat16 const*>(kWeight.data_ptr()); + p.indexQWeight = reinterpret_cast<__nv_bfloat16 const*>(indexQWeight.data_ptr()); + p.indexKWeight = reinterpret_cast<__nv_bfloat16 const*>(indexKWeight.data_ptr()); + p.rotaryCosSin = rotaryCosSin.data_ptr(); + p.positionIds = positionIds.data_ptr(); + cudaError_t const err = pw::invokeProducerPw8(p, static_cast(numTokens), pdl, early, prewaitStepInputs, + at::cuda::getCurrentCUDAStream(packed.get_device())); + if (err != cudaSuccess) + { + cudaGetLastError(); + TORCH_CHECK(false, "sm107_tp8_producer_pw launch: ", cudaGetErrorString(err)); + } + return {qOut, indexQOut}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp8_producer_pw(Tensor packed, Tensor(a!) kv_data, Tensor(b!) kv_scale, Tensor(c!) index_k, " + "Tensor out_cache_loc, Tensor kv_quant_scale, float eps, Tensor q_weight, Tensor k_weight, " + "Tensor index_q_weight, Tensor index_k_weight, Tensor rotary_cos_sin, Tensor position_ids, bool pdl, " + "bool early, bool prewait_step_inputs=False) -> Tensor[]"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp8_producer_pw", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8ProducerPw); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SampledAcceptOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SampledAcceptOp.cpp new file mode 100644 index 000000000000..f6200c9aa906 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SampledAcceptOp.cpp @@ -0,0 +1,113 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/sampledAccept.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace sa = tensorrt_llm::kernels::minimaxM3Sm107Tp8::sampledAccept; + +//! uint64 words of the sampled-accept exchange workspace for `world` ranks and SA-1R variant `oneRoundK`. +int64_t minimaxM3Sm107Tp8SampledAcceptWsWords(int64_t world, int64_t oneRoundK) +{ + return sa::sampledAcceptWsWords(static_cast(world), static_cast(oneRoundK)); +} + +//! Sampled (temperature + top-p) verification from this rank's BF16 logits shard [rows, Vs], rows = nreq * +//! (draft_len + 1) <= 48: the sampled token of every row (tokens), then the strict acceptance against draft (accTokens, +//! numAcc, gatherIds, optional nextNew column 0). Per-row Philox (seeds, offsets); the exchange goes through the +//! multicast workspace wsMcPtr / wsUc with the (finished rows, call) counters. +void minimaxM3Sm107Tp8SampledAccept(torch::Tensor const& logits, int64_t vocabOffset, torch::Tensor const& temperatures, + torch::Tensor const& topPs, torch::Tensor const& seeds, torch::Tensor const& offsets, torch::Tensor& wsUc, + int64_t wsMcPtr, torch::Tensor& counters, torch::Tensor& tokens, int64_t rank, int64_t world, + torch::Tensor const& draft, torch::Tensor const& batchIdx, torch::Tensor& accTokens, torch::Tensor& numAcc, + torch::Tensor& gatherIds, int64_t forceTotal, std::optional const& nextNew, + std::optional const& forceVec, std::optional const& extraAttempts, bool usePdl, + bool prewaitParams, bool earlyTrigger, int64_t oneRoundK, std::optional const& decided, bool ilpWalk) +{ + c10::cuda::CUDAGuard const deviceGuard{logits.device()}; + sa::Args a; + a.logits = reinterpret_cast<__nv_bfloat16 const*>(logits.data_ptr()); + a.ld = logits.stride(0); + a.rows = static_cast(logits.size(0)); + a.Vs = static_cast(logits.size(1)); + TORCH_CHECK(a.Vs <= sa::kMaxVs && a.rows <= sa::kMaxRows && world <= sa::kMaxWorld + && static_cast(vocabOffset) + a.Vs <= (1LL << sa::kIdBits)); + a.vocab_offset = static_cast(vocabOffset); + a.temperatures = temperatures.data_ptr(); + a.top_ps = topPs.data_ptr(); + a.seeds = reinterpret_cast(seeds.data_ptr()); + a.offsets = reinterpret_cast(offsets.data_ptr()); + a.ws_uc = reinterpret_cast(wsUc.data_ptr()); + a.ws_mc = reinterpret_cast(wsMcPtr); + a.counters = reinterpret_cast(counters.data_ptr()); + a.tokens = tokens.data_ptr(); + a.extra_attempts = extraAttempts.has_value() ? extraAttempts->data_ptr() : nullptr; + a.rank = static_cast(rank); + a.world = static_cast(world); + a.draft = draft.data_ptr(); + a.batch_idx = batchIdx.data_ptr(); + a.acc_tokens = accTokens.data_ptr(); + a.num_acc = numAcc.data_ptr(); + a.gather_ids = reinterpret_cast(gatherIds.data_ptr()); + a.draft_len = static_cast(draft.size(1)); + a.nreq = static_cast(draft.size(0)); + a.force_total = static_cast(forceTotal); + a.next_new = nextNew.has_value() ? nextNew->data_ptr() : nullptr; + a.next_stride = nextNew.has_value() ? static_cast(nextNew->stride(0)) : 0; + a.force_vec = forceVec.has_value() ? forceVec->data_ptr() : nullptr; + a.prewait_params = prewaitParams ? 1 : 0; + a.early_trigger = earlyTrigger ? 1 : 0; + TORCH_CHECK(oneRoundK == 0 || oneRoundK == 9 || oneRoundK == 82, "one_round_k must be 0, 9 or 82, got ", oneRoundK); + a.one_round_k = static_cast(oneRoundK); + a.decided = decided.has_value() ? decided->data_ptr() : nullptr; + a.ilp_walk = ilpWalk ? 1 : 0; + cudaError_t const err = sa::invokeSampledAccept(a, usePdl, at::cuda::getCurrentCUDAStream(logits.get_device())); + TORCH_CHECK(err == cudaSuccess, "m3_sampled_accept launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +// ws_mc_ptr is the multicast address of ws_uc's mapping: the op writes through it, which the schema cannot show. +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + // No tensor arguments, so no dispatch key: registered with the schema as a catch-all kernel. + m.def("minimax_m3_sm107tp8_sampled_accept_ws_words(int world, int one_round_k=0) -> int", + &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8SampledAcceptWsWords); + m.def( + "minimax_m3_sm107tp8_sampled_accept(Tensor logits, int vocab_offset, Tensor temperatures, Tensor top_ps, " + "Tensor seeds, Tensor offsets, Tensor(a!) ws_uc, int ws_mc_ptr, Tensor(b!) counters, Tensor(c!) tokens, " + "int rank, int world, Tensor draft, Tensor batch_idx, Tensor(d!) acc_tokens, Tensor(e!) num_acc, " + "Tensor(f!) gather_ids, int force_total, Tensor(g!)? next_new, Tensor? force_vec, " + "Tensor(h!)? extra_attempts, bool use_pdl, bool prewait_params=True, bool early_trigger=False, " + "int one_round_k=0, Tensor(i!)? decided=None, bool ilp_walk=False) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107tp8_sampled_accept", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8SampledAccept); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SiluAndMulOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SiluAndMulOp.cpp new file mode 100644 index 000000000000..a6cda645ba7f --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SiluAndMulOp.cpp @@ -0,0 +1,66 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/siluAndMul.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The drafter MLP's plain SwiGLU (see kernels/minimaxM3Sm107Tp8/siluAndMul.h): x bf16 [rows, 2d] -> out bf16 [rows, +//! d]. +void minimaxM3Sm107Tp8SiluAndMul(torch::Tensor const& x, torch::Tensor& out) +{ + TORCH_CHECK(x.is_cuda() && x.scalar_type() == torch::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "minimax_m3_sm107_tp8_silu_and_mul: x must be a contiguous 2-D bfloat16 CUDA tensor"); + int64_t const rows = x.size(0); + int64_t const d = x.size(1) / 2; + TORCH_CHECK(x.size(1) % 16 == 0 && reinterpret_cast(x.data_ptr()) % 16 == 0, + "minimax_m3_sm107_tp8_silu_and_mul: x rows must be 16-byte aligned with a half width a multiple of 8"); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.is_contiguous() + && out.size(0) == rows && out.size(1) == d && reinterpret_cast(out.data_ptr()) % 16 == 0, + "minimax_m3_sm107_tp8_silu_and_mul: out must be a contiguous, 16-byte aligned bfloat16 [rows, d]"); + TORCH_CHECK(rows <= 65535, "minimax_m3_sm107_tp8_silu_and_mul: at most 65535 rows"); + if (rows == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto const err + = tensorrt_llm::kernels::minimaxM3Sm107Tp8::invokeSiluAndMul(reinterpret_cast(x.data_ptr()), + reinterpret_cast(out.data_ptr()), rows, d, at::cuda::getCurrentCUDAStream(x.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp8_silu_and_mul launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp8_silu_and_mul(Tensor x, Tensor(a!) out) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp8_silu_and_mul", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8SiluAndMul); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SparseDecodeLc8Op.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SparseDecodeLc8Op.cpp new file mode 100644 index 000000000000..dc80380b48bb --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SparseDecodeLc8Op.cpp @@ -0,0 +1,189 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/sparseDecodeLc8m.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int64_t kHeads = 8; +constexpr int64_t kHeadDim = 128; +constexpr int64_t kTopK = 16; + +void checkBytes(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.element_size() == 1, "minimax_m3_sm107_tp8_sparse_decode_lc8: ", name, + " must be a one-byte CUDA view (uint8 / e4m3 / packed e2m1)"); +} + +} // namespace + +//! One-launch MiniMax-M3 TP8 long-context sparse decode over the selected NVFP4 blocks (8 query heads on one KV head, +//! head_dim 128, page 128, top-k 16). Dependents are released right after the grid dependency wait. +//! +//! @param q E4M3 bytes [T, 8, 128], unit stride in the last dim. +//! @param k, v Packed E2M1 bytes [pages, 1, 128, 64] (the per-layer HND views). +//! @param kBlockScale, vBlockScale E4M3 bytes [pages, 1, 128, 8]: K token-major, V in 4x4 token-quad order. +//! @param kGlobalScale, vGlobalScale One-element float32 per-tensor dequant scales. +//! @param topk Int32 [T, 16] ascending selected block ids with -1 tail (kv head 0 of the head-major table). +//! @param out BF16 [T, 1024], written. q8 / sf, when given, receive the MXFP8 copy and its swizzled UE8M0 scales. +//! @param prewait Stage seq_lens, q and the request's block-table row before the grid dependency wait (same output). +void minimaxM3Sm107Tp8SparseDecodeLc8(torch::Tensor const& q, torch::Tensor const& k, torch::Tensor const& v, + torch::Tensor const& kBlockScale, torch::Tensor const& vBlockScale, torch::Tensor const& kGlobalScale, + torch::Tensor const& vGlobalScale, torch::Tensor const& topk, torch::Tensor const& blockTable, + torch::Tensor const& seqLens, torch::Tensor& out, std::optional q8, std::optional sf, + int64_t numSfKTiles, int64_t decodeQueryLen, double smScale, bool linearSoftmax, bool usePdl, bool prewait, + std::optional prefetchA, std::optional prefetchB, std::optional n1Self, + std::optional n1Next, std::optional n1K, std::optional n1V, + std::optional n1KScale, std::optional n1VScale, bool identity, bool oneDiv, + bool identityStep) +{ + TORCH_CHECK(q.is_cuda(), "minimax_m3_sm107_tp8_sparse_decode_lc8: q must be a CUDA tensor"); + c10::cuda::CUDAGuard const deviceGuard{q.device()}; + TORCH_CHECK(at::cuda::getCurrentDeviceProperties()->major >= 10, + "minimax_m3_sm107_tp8_sparse_decode_lc8 requires an SM100-family GPU"); + checkBytes(q, "q"); + checkBytes(k, "k"); + checkBytes(v, "v"); + checkBytes(kBlockScale, "k_block_scale"); + checkBytes(vBlockScale, "v_block_scale"); + TORCH_CHECK(q.dim() == 3 && q.size(1) == kHeads && q.size(2) == kHeadDim && q.stride(2) == 1, + "minimax_m3_sm107_tp8_sparse_decode_lc8: q must be [T, 8, 128] with unit last-dim stride"); + TORCH_CHECK(kGlobalScale.scalar_type() == torch::kFloat32 && vGlobalScale.scalar_type() == torch::kFloat32, + "minimax_m3_sm107_tp8_sparse_decode_lc8: global scales must be float32"); + TORCH_CHECK(topk.scalar_type() == torch::kInt32 && topk.dim() == 2 && topk.size(1) == kTopK, + "minimax_m3_sm107_tp8_sparse_decode_lc8: topk must be int32 [T, 16]"); + TORCH_CHECK(blockTable.scalar_type() == torch::kInt32 && seqLens.scalar_type() == torch::kInt32, + "minimax_m3_sm107_tp8_sparse_decode_lc8: block_table and seq_lens must be int32"); + TORCH_CHECK(out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.size(0) == q.size(0) + && out.size(1) == kHeads * kHeadDim && out.stride(1) == 1, + "minimax_m3_sm107_tp8_sparse_decode_lc8: out must be bfloat16 [T, 1024] with unit last-dim stride"); + TORCH_CHECK( + q8.has_value() == sf.has_value(), "minimax_m3_sm107_tp8_sparse_decode_lc8: pass both q8 and sf, or neither"); + bool const mxfp8 = q8.has_value(); + if (mxfp8) + { + checkBytes(*q8, "q8"); + checkBytes(*sf, "sf"); + } + + tensorrt_llm::kernels::minimaxM3Sm107Tp8::MinimaxM3Tp8SparseDecodeLc8Params prm{}; + prm.q = reinterpret_cast(q.data_ptr()); + prm.q_tok = q.stride(0); + prm.q_head = q.stride(1); + prm.k = reinterpret_cast(k.data_ptr()); + prm.v = reinterpret_cast(v.data_ptr()); + prm.k_blk = k.stride(0); + prm.v_blk = v.stride(0); + prm.ksc = reinterpret_cast(kBlockScale.data_ptr()); + prm.vsc = reinterpret_cast(vBlockScale.data_ptr()); + prm.ksc_blk = kBlockScale.stride(0); + prm.vsc_blk = vBlockScale.stride(0); + prm.kgs = kGlobalScale.data_ptr(); + prm.vgs = vGlobalScale.data_ptr(); + prm.topk = topk.data_ptr(); + prm.t_tok = topk.stride(0); + prm.t_k = topk.stride(1); + prm.block_table = blockTable.data_ptr(); + prm.bt_stride = blockTable.stride(0); + prm.seq_lens = seqLens.data_ptr(); + prm.dql = static_cast(decodeQueryLen); + prm.sm_scale = static_cast(smScale); + prm.out = reinterpret_cast<__nv_bfloat16*>(out.data_ptr()); + prm.out_stride = out.stride(0); + prm.q8 = mxfp8 ? reinterpret_cast(q8->data_ptr()) : nullptr; + prm.sf = mxfp8 ? reinterpret_cast(sf->data_ptr()) : nullptr; + prm.num_sf_k_tiles = static_cast(numSfKTiles); + prm.identity = identity ? 1 : 0; + prm.identity_step = (identity || identityStep) ? 1 : 0; + prm.one_div = oneDiv ? 1 : 0; + auto const prefetchRange = [](std::optional const& t, uint8_t const*& ptr, int64_t& bytes) + { + ptr = nullptr; + bytes = 0; + if (t.has_value()) + { + TORCH_CHECK(t->is_cuda() && t->is_contiguous(), + "minimax_m3_sm107_tp8_sparse_decode_lc8: a prefetch range must be a contiguous CUDA tensor"); + ptr = reinterpret_cast(t->data_ptr()); + bytes = t->numel() * static_cast(t->element_size()); + } + }; + prefetchRange(prefetchA, prm.pf0, prm.pf0_bytes); + prefetchRange(prefetchB, prm.pf1, prm.pf1_bytes); + bool const n1 = n1Self.has_value(); + TORCH_CHECK(n1 == n1Next.has_value() && n1 == n1K.has_value() && n1 == n1V.has_value() && n1 == n1KScale.has_value() + && n1 == n1VScale.has_value(), + "minimax_m3_sm107_tp8_sparse_decode_lc8: pass all six next-layer prefetch tensors, or none"); + if (n1) + { + for (auto const* t : {&*n1Self, &*n1Next}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == torch::kInt32 && t->is_contiguous() && t->dim() == 2 + && t->size(1) == kTopK && t->size(0) >= q.size(0), + "minimax_m3_sm107_tp8_sparse_decode_lc8: the id buffers must be contiguous int32 [>= T, 16]"); + } + TORCH_CHECK(n1K->stride(0) == k.stride(0) && n1V->stride(0) == v.stride(0) + && n1KScale->stride(0) == kBlockScale.stride(0) && n1VScale->stride(0) == vBlockScale.stride(0) + && n1K->size(0) == k.size(0), + "minimax_m3_sm107_tp8_sparse_decode_lc8: the next layer's pools must have this layer's page strides"); + prm.n1_self = n1Self->data_ptr(); + prm.n1_next = n1Next->data_ptr(); + prm.n1_k = reinterpret_cast(n1K->data_ptr()); + prm.n1_v = reinterpret_cast(n1V->data_ptr()); + prm.n1_ksc = reinterpret_cast(n1KScale->data_ptr()); + prm.n1_vsc = reinterpret_cast(n1VScale->data_ptr()); + prm.n1_pages = n1K->size(0); + } + + auto const stream = at::cuda::getCurrentCUDAStream(q.get_device()); + cudaError_t const err = tensorrt_llm::kernels::minimaxM3Sm107Tp8::invokeSparseDecodeLc8m( + prm, static_cast(q.size(0)), linearSoftmax, mxfp8, prewait, usePdl, stream); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp8_sparse_decode_lc8 launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m3_sm107_tp8_sparse_decode_lc8(Tensor q, Tensor k, Tensor v, Tensor k_block_scale, Tensor " + "v_block_scale, " + "Tensor k_global_scale, Tensor v_global_scale, Tensor topk, Tensor block_table, Tensor seq_lens, " + "Tensor(a!) out, Tensor(b!)? q8, Tensor(c!)? sf, int num_sf_k_tiles, int decode_query_len, float sm_scale, " + "bool linear_softmax, bool use_pdl, bool prewait, Tensor? prefetch_a=None, Tensor? prefetch_b=None, " + "Tensor(d!)? n1_self=None, Tensor? n1_next=None, Tensor? n1_k=None, Tensor? n1_v=None, Tensor? " + "n1_k_scale=None, Tensor? n1_v_scale=None, bool identity=False, bool one_div=False, bool identity_step=False) " + "-> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp8_sparse_decode_lc8", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8SparseDecodeLc8); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SpecOffsetsOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SpecOffsetsOp.cpp new file mode 100644 index 000000000000..401aa5cc61fd --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SpecOffsetsOp.cpp @@ -0,0 +1,74 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/specOffsets.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +void check(torch::Tensor const& t, char const* name) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt32 && t.dim() == 1 && t.stride(0) == 1, + "minimax_m3_sm107_tp8_spec_offsets: ", name, " must be a contiguous 1-D int32 CUDA tensor"); +} + +} // namespace + +//! pos += pos_off and kv += kv_off (equal lengths per pair) in one plain launch +//! (kernels/minimaxM3Sm107Tp8/specOffsets.h). +void minimaxM3Sm107Tp8SpecOffsets( + torch::Tensor& pos, torch::Tensor const& posOff, torch::Tensor& kv, torch::Tensor const& kvOff) +{ + check(pos, "pos"); + check(posOff, "pos_off"); + check(kv, "kv"); + check(kvOff, "kv_off"); + TORCH_CHECK(pos.numel() == posOff.numel() && kv.numel() == kvOff.numel(), + "minimax_m3_sm107_tp8_spec_offsets: each tensor and its offsets must have the same length"); + if (pos.numel() == 0 && kv.numel() == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{pos.device()}; + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp8::invokeSpecOffsets(pos.data_ptr(), + posOff.data_ptr(), pos.numel(), kv.data_ptr(), kvOff.data_ptr(), kv.numel(), + at::cuda::getCurrentCUDAStream(pos.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp8_spec_offsets launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp8_spec_offsets(Tensor(a!) pos, Tensor pos_off, Tensor(b!) kv, Tensor kv_off) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp8_spec_offsets", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8SpecOffsets); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8StepPrologueOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8StepPrologueOp.cpp new file mode 100644 index 000000000000..f340b50dc7dd --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8StepPrologueOp.cpp @@ -0,0 +1,68 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/stepPrologue.h" + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The decode graph's step-input prologue in one launch (see kernels/minimaxM3Sm107Tp8/stepPrologue.h). +//! +//! @param window The prologue's device window: contiguous uint8, at least the header and `maxDesc` descriptors (the +//! pack's descriptor area; the header's two counts add up to at most that). Its destinations are raw device addresses +//! the caller guarantees are live. +//! @param epoch Optional one-element int64 step counter, incremented once. +void minimaxM3Sm107Tp8StepPrologue(torch::Tensor& window, std::optional epoch, int64_t maxDesc) +{ + TORCH_CHECK(window.is_cuda() && window.scalar_type() == torch::kUInt8 && window.is_contiguous(), + "minimax_m3_sm107_tp8_step_prologue: window must be a contiguous uint8 CUDA tensor"); + TORCH_CHECK(maxDesc > 0 && window.numel() >= 16 + 24 * maxDesc, + "minimax_m3_sm107_tp8_step_prologue: window is shorter than its header and max_desc descriptors"); + c10::cuda::CUDAGuard const deviceGuard{window.device()}; + int64_t* epochPtr = nullptr; + if (epoch.has_value()) + { + TORCH_CHECK(epoch->is_cuda() && epoch->scalar_type() == torch::kInt64 && epoch->numel() == 1 + && epoch->device() == window.device(), + "minimax_m3_sm107_tp8_step_prologue: epoch must be a one-element int64 tensor on the window's device"); + epochPtr = epoch->data_ptr(); + } + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp8::invokeStepPrologue( + window.data_ptr(), epochPtr, maxDesc, at::cuda::getCurrentCUDAStream(window.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp8_step_prologue launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp8_step_prologue(Tensor(a!) window, Tensor(b!)? epoch, int max_desc) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp8_step_prologue", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8StepPrologue); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SwigluOaiOp.cpp b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SwigluOaiOp.cpp new file mode 100644 index 000000000000..8b5e59751f24 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM3/minimaxM3Sm107Tp8SwigluOaiOp.cpp @@ -0,0 +1,65 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM3Sm107Tp8/swigluOai.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! The dense MLP's swigluoai activation (see kernels/minimaxM3Sm107Tp8/swigluOai.h): x bf16 [rows, 2d] -> out bf16 +//! [rows, d], PDL-launched with an early release when `pdl`. +void minimaxM3Sm107Tp8SwigluOai(torch::Tensor const& x, torch::Tensor& out, bool pdl) +{ + TORCH_CHECK(x.is_cuda() && x.scalar_type() == torch::kBFloat16 && x.dim() == 2 && x.is_contiguous(), + "minimax_m3_sm107_tp8_swiglu_oai: x must be a contiguous 2-D bfloat16 CUDA tensor"); + int64_t const rows = x.size(0); + int64_t const d = x.size(1) / 2; + TORCH_CHECK(x.size(1) % 4 == 0 && reinterpret_cast(x.data_ptr()) % 4 == 0, + "minimax_m3_sm107_tp8_swiglu_oai: x rows must be 4-byte aligned with an even half width"); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kBFloat16 && out.dim() == 2 && out.is_contiguous() + && out.size(0) == rows && out.size(1) == d && reinterpret_cast(out.data_ptr()) % 4 == 0, + "minimax_m3_sm107_tp8_swiglu_oai: out must be a contiguous, 4-byte aligned bfloat16 [rows, d]"); + if (rows == 0) + { + return; + } + c10::cuda::CUDAGuard const deviceGuard{x.device()}; + auto const err = tensorrt_llm::kernels::minimaxM3Sm107Tp8::invokeSwigluOai( + reinterpret_cast(x.data_ptr()), reinterpret_cast(out.data_ptr()), rows, d / 2, pdl, + at::cuda::getCurrentCUDAStream(x.get_device())); + TORCH_CHECK(err == cudaSuccess, "minimax_m3_sm107_tp8_swiglu_oai launch failed: ", cudaGetErrorString(err)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m3_sm107_tp8_swiglu_oai(Tensor x, Tensor(a!) out, bool pdl) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m3_sm107_tp8_swiglu_oai", &tensorrt_llm::torch_ext::minimaxM3Sm107Tp8SwigluOai); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp index aa2e7e5fdd68..83776d92cfe5 100644 --- a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ChainSpecSamplingMcOp.cpp @@ -114,12 +114,104 @@ std::tuple minimaxM31ChainSpecSamplingMc(torch::Te return {outputTokenIds, outputNumTokens}; } +//! minimax_m31_chain_speculative_sampling_mc with slot-indexed draft rows (VR REJ_SLOTS): the draft distributions are +//! draft_buf[slot_ids[r], :num_draft, :vocab] of the spec metadata's [slots, maxDraft, >= vocab] float32 buffer, read +//! in place instead of from a gathered contiguous copy (the same floats, so bitwise the gathered call's results). +//! slot_ids: int64 [batch] on the device; each must index a row of draft_buf (not checked on the host: the caller's +//! gather would index the same rows). +std::tuple minimaxM31ChainSpecSamplingMcSlots(torch::Tensor const& draftBuf, + torch::Tensor const& slotIds, int64_t numDraft, torch::Tensor const& draftTokenIds, + torch::Tensor const& targetProbs, torch::Tensor const& seed, torch::Tensor const& offset, torch::Tensor& workspace, + int64_t ctasPerRow) +{ + TORCH_CHECK(draftBuf.dim() == 3 && draftBuf.is_cuda() && draftBuf.is_contiguous() + && draftBuf.scalar_type() == torch::kFloat32, + "minimax_m31_chain_speculative_sampling_mc_slots: draft_buf must be a contiguous float32 CUDA [slots, " + "maxDraft, " + "vocab_stride] tensor"); + TORCH_CHECK(slotIds.dim() == 1 && slotIds.is_cuda() && slotIds.is_contiguous() + && slotIds.scalar_type() == torch::kInt64 && slotIds.get_device() == draftBuf.get_device(), + "minimax_m31_chain_speculative_sampling_mc_slots: slot_ids must be a contiguous int64 CUDA [batch] tensor"); + TORCH_CHECK(targetProbs.dim() == 3, "minimax_m31_chain_speculative_sampling_mc_slots: target_probs must be 3-D"); + auto const batchSize = slotIds.size(0); + auto const vocabSize = targetProbs.size(2); + TORCH_CHECK( + numDraft >= 0 && numDraft <= draftBuf.size(1) && vocabSize <= draftBuf.size(2) && draftBuf.size(2) % 4 == 0, + "minimax_m31_chain_speculative_sampling_mc_slots: num_draft / vocab exceed draft_buf (or its row is not a " + "multiple of 4 floats)"); + TORCH_CHECK(draftTokenIds.dim() == 2 && draftTokenIds.size(0) == batchSize && draftTokenIds.size(1) == numDraft, + "minimax_m31_chain_speculative_sampling_mc_slots: draft_token_ids must be [batch, num_draft]"); + TORCH_CHECK(targetProbs.size(0) == batchSize && targetProbs.size(1) == numDraft + 1, + "minimax_m31_chain_speculative_sampling_mc_slots: target_probs must be [batch, num_draft + 1, vocab]"); + for (auto const* t : {&draftTokenIds, &targetProbs, &seed, &offset}) + { + TORCH_CHECK(t->is_cuda() && t->is_contiguous() && t->get_device() == draftBuf.get_device(), + "minimax_m31_chain_speculative_sampling_mc_slots: inputs must be contiguous CUDA tensors on one device"); + } + TORCH_CHECK(targetProbs.scalar_type() == torch::kFloat32 && draftTokenIds.scalar_type() == torch::kInt32 + && seed.scalar_type() == torch::kInt64 && offset.scalar_type() == torch::kInt64 && seed.numel() == batchSize + && offset.numel() == batchSize, + "minimax_m31_chain_speculative_sampling_mc_slots: dtypes / counts as " + "minimax_m31_chain_speculative_sampling_mc"); + TORCH_CHECK(vocabSize % 4 == 0 && vocabSize <= tensorrt_llm::kernels::minimaxM31::kChainSpecMcMaxVocab + && reinterpret_cast(draftBuf.data_ptr()) % 16 == 0 + && reinterpret_cast(targetProbs.data_ptr()) % 16 == 0, + "minimax_m31_chain_speculative_sampling_mc_slots: 16-byte aligned probs, vocab divisible by 4, at most ", + tensorrt_llm::kernels::minimaxM31::kChainSpecMcMaxVocab); + TORCH_CHECK(batchSize <= 65535 && ctasPerRow >= 1 && ctasPerRow <= 1024 && 1024 % ctasPerRow == 0, + "minimax_m31_chain_speculative_sampling_mc_slots: batch <= 65535; ctas_per_row must divide 1024"); + auto const wsBytesPerRow + = static_cast(tensorrt_llm::kernels::minimaxM31::chainSpecSamplingMcWorkspaceBytesPerRow()); + TORCH_CHECK(workspace.is_cuda() && workspace.is_contiguous() && workspace.get_device() == draftBuf.get_device() + && reinterpret_cast(workspace.data_ptr()) % 16 == 0 + && workspace.numel() * workspace.element_size() >= wsBytesPerRow * batchSize, + "minimax_m31_chain_speculative_sampling_mc_slots: workspace too small or misaligned"); + + c10::cuda::CUDAGuard const deviceGuard{draftBuf.device()}; + TORCH_CHECK( + ctasPerRow == 1 || batchSize * ctasPerRow <= at::cuda::getCurrentDeviceProperties()->multiProcessorCount, + "minimax_m31_chain_speculative_sampling_mc_slots: batch x ctas_per_row must not exceed the SM count"); + auto outputTokenIds = torch::empty({batchSize, numDraft + 1}, draftTokenIds.options()); + auto outputNumTokens = torch::empty({batchSize}, draftTokenIds.options()); + if (batchSize == 0) + { + return {outputTokenIds, outputNumTokens}; + } + tensorrt_llm::kernels::speculative_decoding::ChainSpeculativeSamplingParams params; + params.draftProbs = draftBuf.const_data_ptr(); + params.draftTokenIds = draftTokenIds.const_data_ptr(); + params.targetProbs = targetProbs.const_data_ptr(); + params.seed = seed.const_data_ptr(); + params.offset = offset.const_data_ptr(); + params.outputTokenIds = outputTokenIds.data_ptr(); + params.outputNumTokens = outputNumTokens.data_ptr(); + params.batchSize = static_cast(batchSize); + params.numDraftTokens = static_cast(numDraft); + params.vocabSize = static_cast(vocabSize); + tensorrt_llm::kernels::minimaxM31::ChainSpecMcDraftRows rows; + rows.rowIds = slotIds.const_data_ptr(); + rows.rowStride = draftBuf.stride(0); + rows.posStride = draftBuf.stride(1); + auto const err = tensorrt_llm::kernels::minimaxM31::invokeChainSpecSamplingMc(params, static_cast(ctasPerRow), + workspace.data_ptr(), at::cuda::getCurrentCUDAStream(draftBuf.get_device()), rows); + TORCH_CHECK( + err == cudaSuccess, "minimax_m31_chain_speculative_sampling_mc_slots launch failed: ", cudaGetErrorString(err)); + return {outputTokenIds, outputNumTokens}; +} + //! Workspace bytes per request of minimax_m31_chain_speculative_sampling_mc. int64_t minimaxM31ChainSpecSamplingMcWsBytes() { return static_cast(tensorrt_llm::kernels::minimaxM31::chainSpecSamplingMcWorkspaceBytesPerRow()); } +//! The kernel minimax_m31_chain_speculative_sampling_mc launches: 3 (REJ v3, the default) or 2 +//! (TRTLLM_M31_DISABLE_REJ_V3=1). +int64_t minimaxM31ChainSpecSamplingMcVersion() +{ + return static_cast(tensorrt_llm::kernels::minimaxM31::chainSpecSamplingMcKernelVersion()); +} + } // namespace torch_ext TRTLLM_NAMESPACE_END @@ -129,12 +221,20 @@ TORCH_LIBRARY_FRAGMENT(trtllm, m) m.def( "minimax_m31_chain_speculative_sampling_mc(Tensor draft_probs, Tensor draft_token_ids, Tensor target_probs, " "Tensor seed, Tensor offset, Tensor(a!) workspace, int ctas_per_row) -> (Tensor, Tensor)"); + m.def( + "minimax_m31_chain_speculative_sampling_mc_slots(Tensor draft_buf, Tensor slot_ids, int num_draft, " + "Tensor draft_token_ids, Tensor target_probs, Tensor seed, Tensor offset, Tensor(a!) workspace, " + "int ctas_per_row) -> (Tensor, Tensor)"); // No tensor arguments: a catch-all kernel (as moe_a2a_get_timeout), callable without a device. m.def("minimax_m31_chain_speculative_sampling_mc_ws_bytes() -> int", &tensorrt_llm::torch_ext::minimaxM31ChainSpecSamplingMcWsBytes); + m.def("minimax_m31_chain_speculative_sampling_mc_version() -> int", + &tensorrt_llm::torch_ext::minimaxM31ChainSpecSamplingMcVersion); } TORCH_LIBRARY_IMPL(trtllm, CUDA, m) { m.impl("minimax_m31_chain_speculative_sampling_mc", &tensorrt_llm::torch_ext::minimaxM31ChainSpecSamplingMc); + m.impl("minimax_m31_chain_speculative_sampling_mc_slots", + &tensorrt_llm::torch_ext::minimaxM31ChainSpecSamplingMcSlots); } diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31DsparkGlueOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31DsparkGlueOp.cpp new file mode 100644 index 000000000000..261b2e1cf471 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31DsparkGlueOp.cpp @@ -0,0 +1,133 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/dsparkGlueSm107.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +bool isIndex(torch::Tensor const& t) +{ + return t.scalar_type() == torch::kInt32 || t.scalar_type() == torch::kInt64; +} + +void checkI64Vec(torch::Tensor const& t, char const* what) +{ + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kInt64 && t.dim() == 1 && t.is_contiguous(), + "minimax_m31_dspark_advance: ", what, " must be a contiguous int64 1-D CUDA tensor"); +} + +} // namespace + +//! DSPARK_GLUE_CPP (kernels/minimaxM31/dsparkGlueSm107.h): the drafter's context advance + read view, in place. +//! slots / rows: int64 [n]; nacc: int32 / int64 [n]; ctx_len: int64 [slots] (updated); counts: int64 [rows]; +//! base_pages: int64 [slots]; tables / read_view: int32 [rows, width], unit column stride (read_view written); +//! out: int64 [n] (written). +void minimaxM31DsparkAdvance(torch::Tensor const& slots, torch::Tensor const& rows, torch::Tensor const& nacc, + torch::Tensor& ctxLen, torch::Tensor const& counts, torch::Tensor const& basePages, torch::Tensor const& tables, + torch::Tensor& readView, torch::Tensor& out, int64_t maxCtx, int64_t dummySlot, int64_t page, int64_t blockSize) +{ + checkI64Vec(slots, "slots"); + checkI64Vec(rows, "rows"); + checkI64Vec(ctxLen, "ctx_len"); + checkI64Vec(counts, "counts"); + checkI64Vec(basePages, "base_pages"); + checkI64Vec(out, "out"); + int64_t const n = slots.size(0); + TORCH_CHECK(rows.size(0) == n && out.size(0) == n, "minimax_m31_dspark_advance: rows / out must have n entries"); + TORCH_CHECK(nacc.is_cuda() && isIndex(nacc) && nacc.dim() == 1 && nacc.size(0) == n && nacc.is_contiguous(), + "minimax_m31_dspark_advance: num_accepted must be a contiguous int32 / int64 [n] CUDA tensor"); + TORCH_CHECK(tables.is_cuda() && tables.scalar_type() == torch::kInt32 && tables.dim() == 2 && tables.stride(1) == 1 + && readView.is_cuda() && readView.scalar_type() == torch::kInt32 && readView.sizes() == tables.sizes() + && readView.stride(1) == 1, + "minimax_m31_dspark_advance: tables / read_view must be int32 [rows, width] with unit column stride"); + TORCH_CHECK(page >= 1 && n <= std::numeric_limits::max(), "minimax_m31_dspark_advance: page must be >= 1"); + at::cuda::CUDAGuard const guard(slots.device()); + tensorrt_llm::kernels::minimaxM31::invokeDsparkAdvance(slots.data_ptr(), rows.data_ptr(), + nacc.data_ptr(), nacc.scalar_type() == torch::kInt64, ctxLen.data_ptr(), counts.data_ptr(), + basePages.data_ptr(), tables.data_ptr(), readView.data_ptr(), + out.data_ptr(), n, tables.size(1), tables.stride(0), readView.stride(0), maxCtx, dummySlot, page, + blockSize, at::cuda::getCurrentCUDAStream(slots.get_device())); +} + +//! DSPARK_GLUE_CPP: the paged dflash_forward's index prelude (seven int32 outputs written in place). +//! num_ctx / batch_idx: int32 / int64 [B]; qpos: int32 / int64 [B, block] (any strides). +void minimaxM31DsparkIndices(torch::Tensor const& numCtx, torch::Tensor const& batchIdx, torch::Tensor const& qpos, + torch::Tensor& seqlens, torch::Tensor& batchOut, torch::Tensor& after, torch::Tensor& indptr, torch::Tensor& last, + torch::Tensor& append, torch::Tensor& qflat, int64_t block, int64_t pagesPerSlot, int64_t pageSize) +{ + TORCH_CHECK(numCtx.is_cuda() && isIndex(numCtx) && numCtx.dim() == 1 && numCtx.is_contiguous(), + "minimax_m31_dspark_indices: num_ctx must be a contiguous int32 / int64 [B] CUDA tensor"); + int64_t const B = numCtx.size(0); + TORCH_CHECK(isIndex(batchIdx) && batchIdx.dim() == 1 && batchIdx.size(0) == B && batchIdx.is_contiguous(), + "minimax_m31_dspark_indices: batch_idx must be a contiguous int32 / int64 [B] tensor"); + TORCH_CHECK(isIndex(qpos) && qpos.dim() == 2 && qpos.size(0) == B && qpos.size(1) == block, + "minimax_m31_dspark_indices: query_positions must be int32 / int64 [B, block]"); + TORCH_CHECK( + block >= 1 && pagesPerSlot >= 1 && pageSize >= 1 && B >= 1 && B * block <= std::numeric_limits::max(), + "minimax_m31_dspark_indices: invalid sizes"); + for (torch::Tensor const* t : {&seqlens, &batchOut, &after, &indptr, &last, &append, &qflat}) + { + TORCH_CHECK(t->is_cuda() && t->scalar_type() == torch::kInt32 && t->is_contiguous(), + "minimax_m31_dspark_indices: outputs must be contiguous int32 CUDA tensors"); + } + TORCH_CHECK(seqlens.numel() >= B && batchOut.numel() >= B && after.numel() >= B && indptr.numel() >= B + 1 + && last.numel() >= B && append.numel() >= B * block && qflat.numel() >= B * block, + "minimax_m31_dspark_indices: an output is too small"); + at::cuda::CUDAGuard const guard(numCtx.device()); + tensorrt_llm::kernels::minimaxM31::invokeDsparkIndices(numCtx.data_ptr(), numCtx.scalar_type() == torch::kInt64, + batchIdx.data_ptr(), batchIdx.scalar_type() == torch::kInt64, qpos.data_ptr(), + qpos.scalar_type() == torch::kInt64, qpos.stride(0), qpos.stride(1), seqlens.data_ptr(), + batchOut.data_ptr(), after.data_ptr(), indptr.data_ptr(), last.data_ptr(), + append.data_ptr(), qflat.data_ptr(), static_cast(B), static_cast(block), + static_cast(pagesPerSlot), static_cast(pageSize), + at::cuda::getCurrentCUDAStream(numCtx.get_device())); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_dspark_advance(Tensor slots, Tensor rows, Tensor num_accepted, Tensor(a!) ctx_len, " + "Tensor counts, Tensor base_pages, Tensor tables, Tensor(b!) read_view, Tensor(c!) out, int max_ctx, " + "int dummy_slot, int page, int block_size) -> ()"); + m.def( + "minimax_m31_dspark_indices(Tensor num_ctx, Tensor batch_idx, Tensor query_positions, Tensor(a!) seqlens, " + "Tensor(b!) batch_out, Tensor(c!) after, Tensor(d!) indptr, Tensor(e!) last, Tensor(f!) append, " + "Tensor(g!) qflat, int block, int pages_per_slot, int page_size) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_dspark_advance", &tensorrt_llm::torch_ext::minimaxM31DsparkAdvance); + m.impl("minimax_m31_dspark_indices", &tensorrt_llm::torch_ext::minimaxM31DsparkIndices); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31F3ArOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31F3ArOp.cpp new file mode 100644 index 000000000000..54d75360ac03 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31F3ArOp.cpp @@ -0,0 +1,107 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/f3ArPrepushedSm107.h" + +#include +#include +#include + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +//! The shipped libm31_f3_ar.so's launch rule: the programmatic launch only with TRTLLM_ENABLE_PDL=1 (read once). +bool f3PdlEnabled() +{ + static char const* e = std::getenv("TRTLLM_ENABLE_PDL"); + return e != nullptr && e[0] == '1'; +} + +} // namespace + +//! F3 attention boundary (TP2 decode): the one-shot Lamport all-reduce + residual + RMSNorm over the rows the o_proj's +//! F3 GEMM epilogue already pushed into the AllReduce workspace's data slots (no push loop), + optionally the MXFP8 of +//! norm_out (mx_out e4m3 [rows, hidden], mx_sf swizzled UE8M0 of pad128(rows) rows). allreduce_in is shape-only (its +//! bytes are not read). Bitwise to the stock one-shot allreduce_fusion RESIDUAL_RMS_NORM (+ mxfp8_quantize). +void minimaxM31F3ArPrepushed(int64_t rank, torch::Tensor workspace, torch::Tensor allreduceIn, torch::Tensor residualIn, + torch::Tensor residualOut, torch::Tensor normOut, torch::Tensor gamma, double eps, bool triggerCompletionAtEnd, + std::optional mxOut, std::optional mxSf) +{ + TORCH_CHECK(allreduceIn.scalar_type() == at::kBFloat16 && allreduceIn.is_contiguous()); + tensorrt_llm::kernels::minimaxM31::F3ArParams p; + p.rank = static_cast(rank); + p.size = static_cast(allreduceIn.numel()); + p.hidden_dim = static_cast(allreduceIn.size(-1)); + p.workspace = reinterpret_cast(workspace.data_ptr()); + p.allreduce_in = allreduceIn.data_ptr(); + p.residual_in = residualIn.data_ptr(); + p.residual_out = residualOut.data_ptr(); + p.norm_out = normOut.data_ptr(); + p.rms_gamma = gamma.data_ptr(); + p.rms_eps = static_cast(eps); + p.mx_out = nullptr; + p.mx_sf = nullptr; + if (mxOut.has_value()) + { + TORCH_CHECK(mxSf.has_value() && mxOut->is_contiguous() && mxSf->is_contiguous()); + int64_t const rows = p.size / p.hidden_dim; + TORCH_CHECK(mxOut->numel() * mxOut->element_size() == rows * p.hidden_dim, "mx_out must be [rows, H] e4m3"); + TORCH_CHECK(mxSf->numel() * mxSf->element_size() == (rows + 127) / 128 * 128 * (p.hidden_dim / 32), + "mx_sf must be the swizzled [pad128(rows), H / 32] UE8M0 scales"); + p.mx_out = mxOut->data_ptr(); + p.mx_sf = mxSf->data_ptr(); + } + TORCH_CHECK(p.size % p.hidden_dim == 0 && p.hidden_dim % 8 == 0); + at::cuda::CUDAGuard const guard(allreduceIn.device()); + C10_CUDA_CHECK(tensorrt_llm::kernels::minimaxM31::invokeF3ArPrepushed( + p, triggerCompletionAtEnd, f3PdlEnabled(), at::cuda::getCurrentCUDAStream(allreduceIn.get_device()))); +} + +//! 8 bytes at a device address (the Lamport workspace's comm size lives in device memory); setup only, never in a +//! CUDA graph. +int64_t minimaxM31F3ReadI64(int64_t addr) +{ + int64_t v = 0; + C10_CUDA_CHECK(cudaMemcpy(&v, reinterpret_cast(addr), 8, cudaMemcpyDeviceToHost)); + return v; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m31_f3_read_i64(int addr) -> int", &tensorrt_llm::torch_ext::minimaxM31F3ReadI64); + m.def( + "minimax_m31_f3_ar_prepushed(int rank, Tensor workspace, Tensor allreduce_in, Tensor residual_in, " + "Tensor(a!) residual_out, Tensor(b!) norm_out, Tensor gamma, float eps, bool trigger_completion_at_end, " + "Tensor(c!)? mx_out, Tensor(d!)? mx_sf) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_f3_ar_prepushed", &tensorrt_llm::torch_ext::minimaxM31F3ArPrepushed); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31FcShardGemmOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31FcShardGemmOp.cpp new file mode 100644 index 000000000000..e028628bbf57 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31FcShardGemmOp.cpp @@ -0,0 +1,172 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// FC_SHARD (MiniMax-M3.1 VR TP8): y[M, N] = x[M, K] @ w[N, K]^T in bf16 (fp32 accumulate) through cuBLASLt with a +// caller-given algo config (ALGO_ID, TILE_ID, SPLITK_NUM, REDUCTION_SCHEME, CTA_SWIZZLING, CUSTOM_OPTION, STAGES_ID, +// INNER_SHAPE_ID, CLUSTER_SHAPE_ID). Pinning a column shard of the DSpark drafter's fc to the algo cuBLASLt picks for +// the full fc makes every output column the same sum of the same K-split partials as the full GEMM, so the shard is +// bit for bit the matching columns of torch.nn.functional.linear(x, w_full). + +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +namespace +{ + +constexpr int64_t kWorkspaceBytes = 32LL << 20; + +void checkLt(cublasStatus_t st, char const* what) +{ + TORCH_CHECK(st == CUBLAS_STATUS_SUCCESS, "minimax_m31_fc_shard_gemm: ", what, " failed with cuBLAS status ", + static_cast(st)); +} + +struct DeviceState +{ + cublasLtHandle_t handle{nullptr}; + cublasLtMatmulDesc_t desc{nullptr}; + at::Tensor workspace; + // (m, n, k, config) -> the initialized algo after a passing AlgoCheck + std::map>, cublasLtMatmulAlgo_t> algos; +}; + +std::mutex gMutex; +std::map gStates; + +DeviceState& stateFor(at::Device const& device) +{ + auto& st = gStates[device.index()]; + if (st.handle == nullptr) + { + checkLt(cublasLtCreate(&st.handle), "cublasLtCreate"); + checkLt(cublasLtMatmulDescCreate(&st.desc, CUBLAS_COMPUTE_32F, CUDA_R_32F), "cublasLtMatmulDescCreate"); + cublasOperation_t const ta = CUBLAS_OP_T; + cublasOperation_t const tb = CUBLAS_OP_N; + checkLt(cublasLtMatmulDescSetAttribute(st.desc, CUBLASLT_MATMUL_DESC_TRANSA, &ta, sizeof(ta)), "transa"); + checkLt(cublasLtMatmulDescSetAttribute(st.desc, CUBLASLT_MATMUL_DESC_TRANSB, &tb, sizeof(tb)), "transb"); + st.workspace = at::empty({kWorkspaceBytes}, at::TensorOptions().dtype(at::kByte).device(device)); + } + return st; +} + +struct Layouts +{ + cublasLtMatrixLayout_t a{nullptr}, b{nullptr}, c{nullptr}; + + Layouts(int64_t m, int64_t n, int64_t k) + { + // op(A) = W (row-major [m, k] = column-major k x m, transposed); B = x (row-major [n, k] = column-major + // k x n); D = column-major m x n = row-major [n, m]. + checkLt(cublasLtMatrixLayoutCreate(&a, CUDA_R_16BF, k, m, k), "layout A"); + checkLt(cublasLtMatrixLayoutCreate(&b, CUDA_R_16BF, k, n, k), "layout B"); + checkLt(cublasLtMatrixLayoutCreate(&c, CUDA_R_16BF, m, n, m), "layout D"); + } + + ~Layouts() + { + cublasLtMatrixLayoutDestroy(a); + cublasLtMatrixLayoutDestroy(b); + cublasLtMatrixLayoutDestroy(c); + } +}; + +template +void setCfg(cublasLtMatmulAlgo_t& algo, cublasLtMatmulAlgoConfigAttributes_t attr, int64_t value, char const* what) +{ + T const v = static_cast(value); + checkLt(cublasLtMatmulAlgoConfigSetAttribute(&algo, attr, &v, sizeof(v)), what); +} + +} // namespace + +namespace torch_ext +{ + +//! x: bf16 [M, K] contiguous; w: bf16 [N, K] contiguous (a row slice of the full weight); cfg: the 9 config values +//! above. Returns bf16 [M, N]. Throws if cuBLASLt rejects the config for this shape (the caller keeps the stock fc). +at::Tensor minimaxM31FcShardGemm(at::Tensor const& x, at::Tensor const& w, std::vector const& cfg) +{ + TORCH_CHECK(x.is_cuda() && w.is_cuda() && x.get_device() == w.get_device(), + "minimax_m31_fc_shard_gemm: x and w must be CUDA tensors on one device"); + TORCH_CHECK( + x.scalar_type() == at::kBFloat16 && w.scalar_type() == at::kBFloat16, "minimax_m31_fc_shard_gemm: bf16 only"); + TORCH_CHECK(x.dim() == 2 && w.dim() == 2 && x.is_contiguous() && w.is_contiguous() && x.size(1) == w.size(1), + "minimax_m31_fc_shard_gemm: x [M, K] and w [N, K], contiguous"); + TORCH_CHECK(cfg.size() == 9, "minimax_m31_fc_shard_gemm: cfg must hold 9 values"); + c10::cuda::CUDAGuard guard(x.device()); + int64_t const n = x.size(0); + int64_t const k = x.size(1); + int64_t const m = w.size(0); + auto y = at::empty({n, m}, x.options()); + if (n == 0) + { + return y; + } + std::lock_guard lock(gMutex); + auto& st = stateFor(x.device()); + Layouts lay(m, n, k); + auto key = std::make_tuple(m, n, k, cfg); + auto it = st.algos.find(key); + if (it == st.algos.end()) + { + cublasLtMatmulAlgo_t algo; + checkLt(cublasLtMatmulAlgoInit(st.handle, CUBLAS_COMPUTE_32F, CUDA_R_32F, CUDA_R_16BF, CUDA_R_16BF, CUDA_R_16BF, + CUDA_R_16BF, static_cast(cfg[0]), &algo), + "cublasLtMatmulAlgoInit"); + setCfg(algo, CUBLASLT_ALGO_CONFIG_TILE_ID, cfg[1], "TILE_ID"); + setCfg(algo, CUBLASLT_ALGO_CONFIG_SPLITK_NUM, cfg[2], "SPLITK_NUM"); + setCfg(algo, CUBLASLT_ALGO_CONFIG_REDUCTION_SCHEME, cfg[3], "REDUCTION_SCHEME"); + setCfg(algo, CUBLASLT_ALGO_CONFIG_CTA_SWIZZLING, cfg[4], "CTA_SWIZZLING"); + setCfg(algo, CUBLASLT_ALGO_CONFIG_CUSTOM_OPTION, cfg[5], "CUSTOM_OPTION"); + setCfg(algo, CUBLASLT_ALGO_CONFIG_STAGES_ID, cfg[6], "STAGES_ID"); + setCfg(algo, CUBLASLT_ALGO_CONFIG_INNER_SHAPE_ID, cfg[7], "INNER_SHAPE_ID"); + setCfg(algo, CUBLASLT_ALGO_CONFIG_CLUSTER_SHAPE_ID, cfg[8], "CLUSTER_SHAPE_ID"); + cublasLtMatmulHeuristicResult_t result{}; + checkLt(cublasLtMatmulAlgoCheck(st.handle, st.desc, lay.a, lay.b, lay.c, lay.c, &algo, &result), + "cublasLtMatmulAlgoCheck"); + TORCH_CHECK(result.workspaceSize <= static_cast(kWorkspaceBytes), + "minimax_m31_fc_shard_gemm: the pinned algo needs more workspace than ", kWorkspaceBytes); + it = st.algos.emplace(key, algo).first; + } + float const alpha = 1.0f; + float const beta = 0.0f; + checkLt(cublasLtMatmul(st.handle, st.desc, &alpha, w.data_ptr(), lay.a, x.data_ptr(), lay.b, &beta, y.data_ptr(), + lay.c, y.data_ptr(), lay.c, &it->second, st.workspace.data_ptr(), kWorkspaceBytes, + at::cuda::getCurrentCUDAStream(x.get_device()).stream()), + "cublasLtMatmul"); + return y; +} + +} // namespace torch_ext + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m31_fc_shard_gemm(Tensor x, Tensor w, int[] cfg) -> Tensor"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_fc_shard_gemm", &torch_ext::minimaxM31FcShardGemm); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31MarkovEpilogueOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31MarkovEpilogueOp.cpp new file mode 100644 index 000000000000..291226758bc0 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31MarkovEpilogueOp.cpp @@ -0,0 +1,261 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/markovEpilogueSm107.h" +#include "tensorrt_llm/kernels/minimaxM31/scatterRowsSm107.h" +#include "tensorrt_llm/kernels/minimaxM31/tailBk3Sm107.h" +#include "tensorrt_llm/kernels/minimaxM31/zeroPaddingRowsSm107.h" + +#include +#include +#include + +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +bool aligned(void const* p, uintptr_t bytes) +{ + return reinterpret_cast(p) % bytes == 0; +} + +} // namespace + +//! S4FE (see kernels/minimaxM31/markovEpilogueSm107.h): one position k of the in-tree Markov chain's tail. +//! base / out: fp32 [rows, K, d] (out may be base's same-shape output buffer); bias: bf16 [rows, d] (the in-tree +//! F.linear output); prev: int64 [rows] (written); w1: bf16 [vocab, rank]; emb_out: bf16 [rows, rank] or None; +//! partials: a CUDA buffer of >= rows * splits * 8 bytes; counters: int32 [>= rows], zero (left zero). +void minimaxM31MarkovEpilogue(torch::Tensor const& base, int64_t k, torch::Tensor const& bias, torch::Tensor& out, + torch::Tensor& prev, torch::Tensor const& w1, std::optional embOut, torch::Tensor& partials, + torch::Tensor& counters, int64_t splits) +{ + TORCH_CHECK(base.is_cuda() && base.scalar_type() == torch::kFloat32 && base.dim() == 3 && base.is_contiguous(), + "minimax_m31_markov_epilogue: base must be a contiguous float32 [rows, K, d] CUDA tensor"); + int64_t const rows = base.size(0); + int64_t const block = base.size(1); + int64_t const d = base.size(2); + TORCH_CHECK(k >= 0 && k < block, "minimax_m31_markov_epilogue: position k out of range"); + TORCH_CHECK(d % 4 == 0 && rows >= 1 && rows <= 65535 && rows * block * d <= std::numeric_limits::max() / 4 + && d <= std::numeric_limits::max(), + "minimax_m31_markov_epilogue: width must be divisible by 4 and rows in 1..65535"); + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kFloat32 && out.sizes() == base.sizes() + && out.is_contiguous() && out.get_device() == base.get_device(), + "minimax_m31_markov_epilogue: out must be a contiguous float32 tensor shaped like base on its device"); + TORCH_CHECK(bias.is_cuda() && bias.scalar_type() == torch::kBFloat16 && bias.dim() == 2 && bias.is_contiguous() + && bias.size(0) == rows && bias.size(1) == d && bias.get_device() == base.get_device(), + "minimax_m31_markov_epilogue: bias must be a contiguous bf16 [rows, d] tensor on base's device"); + TORCH_CHECK(prev.is_cuda() && prev.scalar_type() == torch::kInt64 && prev.dim() == 1 && prev.size(0) == rows + && prev.is_contiguous(), + "minimax_m31_markov_epilogue: prev must be a contiguous int64 [rows] tensor"); + TORCH_CHECK(w1.is_cuda() && w1.scalar_type() == torch::kBFloat16 && w1.dim() == 2 && w1.is_contiguous() + && w1.size(0) >= d && w1.size(1) >= 1 && w1.size(1) <= 1024, + "minimax_m31_markov_epilogue: w1 must be a contiguous bf16 [vocab >= d, rank <= 1024] tensor"); + __nv_bfloat16* emb = nullptr; + if (embOut.has_value()) + { + auto& e = embOut.value(); + TORCH_CHECK(e.is_cuda() && e.scalar_type() == torch::kBFloat16 && e.is_contiguous() && e.dim() == 2 + && e.size(0) == rows && e.size(1) == w1.size(1), + "minimax_m31_markov_epilogue: emb_out must be a contiguous bf16 [rows, rank] tensor"); + emb = reinterpret_cast<__nv_bfloat16*>(e.data_ptr()); + } + TORCH_CHECK(splits >= 1 && splits <= 65535, "minimax_m31_markov_epilogue: splits must be in 1..65535"); + TORCH_CHECK(partials.is_cuda() && partials.nbytes() >= static_cast(rows * splits * 8), + "minimax_m31_markov_epilogue: partials is too small"); + TORCH_CHECK(counters.is_cuda() && counters.scalar_type() == torch::kInt32 && counters.numel() >= rows, + "minimax_m31_markov_epilogue: counters must be an int32 tensor of >= rows entries"); + float const* basePtr = base.data_ptr() + k * d; + float* outPtr = out.data_ptr() + k * d; + TORCH_CHECK(aligned(basePtr, 16) && aligned(outPtr, 16) && aligned(bias.data_ptr(), 8), + "minimax_m31_markov_epilogue: base / out rows must be 16-byte aligned and bias rows 8-byte aligned"); + + at::cuda::CUDAGuard const guard(base.device()); + tensorrt_llm::kernels::minimaxM31::invokeMarkovEpilogue(basePtr, block * d, + reinterpret_cast<__nv_bfloat16 const*>(bias.data_ptr()), outPtr, block * d, prev.data_ptr(), + reinterpret_cast<__nv_bfloat16 const*>(w1.data_ptr()), static_cast(w1.size(1)), emb, + reinterpret_cast(partials.data_ptr()), + counters.data_ptr(), static_cast(rows), static_cast(d), static_cast(splits), + at::cuda::getCurrentCUDAStream(base.get_device())); + C10_CUDA_KERNEL_LAUNCH_CHECK(); +} + +//! The kernel's default split count for rows x d on this device. +int64_t minimaxM31MarkovEpilogueSplits(int64_t rows, int64_t d) +{ + int const sms = at::cuda::getCurrentDeviceProperties()->multiProcessorCount; + return tensorrt_llm::kernels::minimaxM31::markovEpilogueSplits( + static_cast(rows), static_cast(d), sms); +} + +//! TAIL16-G2 (see kernels/minimaxM31/zeroPaddingRowsSm107.h): in place, logits[r] = 0 for every row r whose request +//! slotIds[r / rows_per_request] is dummy_slot_row. logits: fp32 [rows, d] with unit column stride; slot_ids: int64 +//! [rows / rows_per_request] (the generation requests' slot ids). +void minimaxM31ZeroPaddingRows( + torch::Tensor& logits, torch::Tensor const& slotIds, int64_t dummySlotRow, int64_t rowsPerRequest) +{ + TORCH_CHECK( + logits.is_cuda() && logits.scalar_type() == torch::kFloat32 && logits.dim() == 2 && logits.stride(1) == 1, + "minimax_m31_zero_padding_rows: logits must be a 2-D float32 CUDA tensor with unit column stride"); + TORCH_CHECK(rowsPerRequest >= 1 && logits.size(0) % rowsPerRequest == 0 && logits.size(0) <= 65535, + "minimax_m31_zero_padding_rows: rows must be a multiple of rows_per_request (<= 65535 rows)"); + TORCH_CHECK(slotIds.is_cuda() && slotIds.scalar_type() == torch::kInt64 && slotIds.dim() == 1 + && slotIds.is_contiguous() && slotIds.size(0) == logits.size(0) / rowsPerRequest + && slotIds.get_device() == logits.get_device(), + "minimax_m31_zero_padding_rows: slot_ids must be a contiguous int64 [rows / rows_per_request] tensor"); + TORCH_CHECK(logits.size(1) <= std::numeric_limits::max(), "minimax_m31_zero_padding_rows: too wide"); + if (logits.size(0) == 0) + { + return; + } + at::cuda::CUDAGuard const guard(logits.device()); + tensorrt_llm::kernels::minimaxM31::invokeZeroPaddingRows(logits.data_ptr(), logits.stride(0), + slotIds.data_ptr(), dummySlotRow, static_cast(logits.size(0)), + static_cast(rowsPerRequest), static_cast(logits.size(1)), + at::cuda::getCurrentCUDAStream(logits.get_device())); + C10_CUDA_KERNEL_LAUNCH_CHECK(); +} + +//! TAIL16-G3 (see kernels/minimaxM31/scatterRowsSm107.h): dst[slot_ids[i], :K, :v] = src[i] for src fp32 [n, K, v] +//! contiguous; dst fp32 [slots, >= K, >= v] with a unit last-dim stride. slot_ids int64 [n] on dst's device. +void minimaxM31ScatterRows(torch::Tensor& dst, torch::Tensor const& slotIds, torch::Tensor const& src) +{ + TORCH_CHECK(src.is_cuda() && src.scalar_type() == torch::kFloat32 && src.dim() == 3 && src.is_contiguous(), + "minimax_m31_scatter_rows: src must be a contiguous float32 [n, K, v] CUDA tensor"); + TORCH_CHECK(dst.is_cuda() && dst.scalar_type() == torch::kFloat32 && dst.dim() == 3 && dst.stride(2) == 1 + && dst.size(1) >= src.size(1) && dst.size(2) >= src.size(2) && dst.get_device() == src.get_device(), + "minimax_m31_scatter_rows: dst must be a float32 [slots, >= K, >= v] tensor with unit last-dim stride"); + TORCH_CHECK(slotIds.is_cuda() && slotIds.scalar_type() == torch::kInt64 && slotIds.dim() == 1 + && slotIds.is_contiguous() && slotIds.size(0) == src.size(0) && slotIds.get_device() == src.get_device(), + "minimax_m31_scatter_rows: slot_ids must be a contiguous int64 [n] tensor"); + TORCH_CHECK(src.size(0) * src.size(1) <= 65535 && src.size(2) <= std::numeric_limits::max(), + "minimax_m31_scatter_rows: too many rows"); + if (src.numel() == 0) + { + return; + } + at::cuda::CUDAGuard const guard(src.device()); + tensorrt_llm::kernels::minimaxM31::invokeScatterRows(dst.data_ptr(), dst.stride(0), dst.stride(1), + src.data_ptr(), slotIds.data_ptr(), static_cast(src.size(0)), + static_cast(src.size(1)), static_cast(src.size(2)), + at::cuda::getCurrentCUDAStream(src.get_device())); + C10_CUDA_KERNEL_LAUNCH_CHECK(); +} + +//! TAIL_BK3 (B): emb = w1[prev] with out-of-range ids as +0 rows (bitwise the in-tree masked embedding). +//! prev: int64 [n]; w1: contiguous bf16 [vocab, rank]; returns bf16 [n, rank]. +torch::Tensor minimaxM31MarkovPrevEmbed(torch::Tensor const& prev, torch::Tensor const& w1) +{ + TORCH_CHECK(prev.is_cuda() && prev.scalar_type() == torch::kInt64 && prev.dim() == 1 && prev.is_contiguous(), + "minimax_m31_markov_prev_embed: prev must be a contiguous int64 [n] CUDA tensor"); + TORCH_CHECK(w1.is_cuda() && w1.scalar_type() == torch::kBFloat16 && w1.dim() == 2 && w1.is_contiguous() + && w1.get_device() == prev.get_device() && w1.size(1) >= 1 + && w1.size(1) <= std::numeric_limits::max(), + "minimax_m31_markov_prev_embed: w1 must be a contiguous bf16 [vocab, rank] tensor on prev's device"); + TORCH_CHECK(prev.size(0) <= std::numeric_limits::max(), "minimax_m31_markov_prev_embed: too many rows"); + auto out = torch::empty({prev.size(0), w1.size(1)}, w1.options()); + if (prev.size(0) == 0) + { + return out; + } + at::cuda::CUDAGuard const guard(prev.device()); + tensorrt_llm::kernels::minimaxM31::invokeMarkovPrevEmbed(prev.data_ptr(), + static_cast(prev.size(0)), reinterpret_cast<__nv_bfloat16 const*>(w1.data_ptr()), w1.size(0), + static_cast(w1.size(1)), reinterpret_cast<__nv_bfloat16*>(out.data_ptr()), + at::cuda::getCurrentCUDAStream(prev.get_device())); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + return out; +} + +//! TAIL_BK3 (C): the block sampler's [n * K] parameters (bitwise repeat_interleave / _rng_state_per_request). +//! Inputs: the request slices temps f32, top_ks i32, top_ps f32, min_ps f32, seeds i64, offsets i64, each contiguous +//! [n]; slot_offset = slot * span. Returns (temps, top_ks, top_ps, min_ps, seeds, offsets), each [n * K]. +std::vector minimaxM31ExpandBlockSampling(torch::Tensor const& temps, torch::Tensor const& topKs, + torch::Tensor const& topPs, torch::Tensor const& minPs, torch::Tensor const& seeds, torch::Tensor const& offsets, + int64_t K, int64_t slotOffset, int64_t span) +{ + auto check = [&](torch::Tensor const& t, torch::ScalarType dt, char const* name) + { + TORCH_CHECK(t.is_cuda() && t.scalar_type() == dt && t.dim() == 1 && t.is_contiguous() + && t.size(0) == temps.size(0) && t.get_device() == temps.get_device(), + "minimax_m31_expand_block_sampling: ", name, " must be a contiguous 1-D CUDA tensor of n entries"); + }; + check(temps, torch::kFloat32, "temps"); + check(topKs, torch::kInt32, "top_ks"); + check(topPs, torch::kFloat32, "top_ps"); + check(minPs, torch::kFloat32, "min_ps"); + check(seeds, torch::kInt64, "seeds"); + check(offsets, torch::kInt64, "offsets"); + int64_t const n = temps.size(0); + TORCH_CHECK(K >= 1 && n * K <= std::numeric_limits::max(), "minimax_m31_expand_block_sampling: bad K"); + int64_t const total = n * K; + auto tOut = torch::empty({total}, temps.options()); + auto kOut = torch::empty({total}, topKs.options()); + auto pOut = torch::empty({total}, topPs.options()); + auto mOut = torch::empty({total}, minPs.options()); + auto sOut = torch::empty({total}, seeds.options()); + auto oOut = torch::empty({total}, offsets.options()); + if (total > 0) + { + at::cuda::CUDAGuard const guard(temps.device()); + tensorrt_llm::kernels::minimaxM31::invokeExpandBlockSampling(temps.data_ptr(), topKs.data_ptr(), + topPs.data_ptr(), minPs.data_ptr(), seeds.data_ptr(), offsets.data_ptr(), + static_cast(n), static_cast(K), slotOffset, span, tOut.data_ptr(), + kOut.data_ptr(), pOut.data_ptr(), mOut.data_ptr(), sOut.data_ptr(), + oOut.data_ptr(), at::cuda::getCurrentCUDAStream(temps.get_device())); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + } + return {tOut, kOut, pOut, mOut, sOut, oOut}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_markov_epilogue(Tensor base, int k, Tensor bias, Tensor(a!) out, Tensor(b!) prev, Tensor w1, " + "Tensor(c!)? emb_out, Tensor(d!) partials, Tensor(e!) counters, int splits) -> ()"); + m.def( + "minimax_m31_zero_padding_rows(Tensor(a!) logits, Tensor slot_ids, int dummy_slot_row, int rows_per_request) " + "-> ()"); + m.def("minimax_m31_scatter_rows(Tensor(a!) dst, Tensor slot_ids, Tensor src) -> ()"); + m.def("minimax_m31_markov_prev_embed(Tensor prev, Tensor w1) -> Tensor"); + m.def( + "minimax_m31_expand_block_sampling(Tensor temps, Tensor top_ks, Tensor top_ps, Tensor min_ps, Tensor seeds, " + "Tensor offsets, int K, int slot_offset, int span) -> Tensor[]"); + m.def("minimax_m31_markov_epilogue_splits(int rows, int d) -> int", + &tensorrt_llm::torch_ext::minimaxM31MarkovEpilogueSplits); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_markov_epilogue", &tensorrt_llm::torch_ext::minimaxM31MarkovEpilogue); + m.impl("minimax_m31_zero_padding_rows", &tensorrt_llm::torch_ext::minimaxM31ZeroPaddingRows); + m.impl("minimax_m31_scatter_rows", &tensorrt_llm::torch_ext::minimaxM31ScatterRows); + m.impl("minimax_m31_markov_prev_embed", &tensorrt_llm::torch_ext::minimaxM31MarkovPrevEmbed); + m.impl("minimax_m31_expand_block_sampling", &tensorrt_llm::torch_ext::minimaxM31ExpandBlockSampling); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31MoeFinalizeOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31MoeFinalizeOp.cpp new file mode 100644 index 000000000000..3e36b91edfe6 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31MoeFinalizeOp.cpp @@ -0,0 +1,82 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/moeFinalizeSm107.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! NATIVE_FINALIZE: the routed experts' finalize from the trtllm-gen runner's unfinalized parts (fc2 [P, H] bf16 rows, +//! weights [T, K] bf16 / fp32, idx = expanded -> permuted int32 [T, K], -1 = dropped), + optionally the shared expert. +//! Bit for bit the Triton kernels/m3_moe_finalize. cfg -1 = table entry 0. +torch::Tensor minimaxM31MoeFinalize( + torch::Tensor fc2, torch::Tensor weights, torch::Tensor idx, std::optional shared, int64_t cfg) +{ + TORCH_CHECK(fc2.dim() == 2 && fc2.scalar_type() == at::kBFloat16 && fc2.stride(1) == 1, "fc2: bf16 [P, H] rows"); + TORCH_CHECK(weights.dim() == 2 && weights.is_contiguous() && idx.sizes() == weights.sizes() + && idx.scalar_type() == at::kInt && idx.is_contiguous(), + "weights / idx: contiguous [T, K], idx int32"); + TORCH_CHECK(weights.scalar_type() == at::kBFloat16 || weights.scalar_type() == at::kFloat, "weights bf16 / fp32"); + int64_t const tokens = weights.size(0), topk = weights.size(1), hidden = fc2.size(1); + TORCH_CHECK(topk >= 1 && topk <= 10, "top_k must be in 1..10"); + TORCH_CHECK(cfg >= -1 && cfg <= 7, "cfg must be -1 or 0..7"); + TORCH_CHECK(hidden % 8 == 0 && fc2.stride(0) % 8 == 0 && reinterpret_cast(fc2.data_ptr()) % 16 == 0, + "fc2: 16-byte aligned rows"); + if (shared.has_value()) + { + TORCH_CHECK(shared->scalar_type() == at::kBFloat16 && shared->is_contiguous() && shared->size(0) == tokens + && shared->size(1) == hidden && reinterpret_cast(shared->data_ptr()) % 16 == 0, + "shared: contiguous bf16 [T, H]"); + } + auto out = torch::empty({tokens, hidden}, fc2.options()); + if (tokens == 0) + { + return out; + } + at::cuda::CUDAGuard const guard(fc2.device()); + cudaError_t const e + = tensorrt_llm::kernels::minimaxM31::invokeMoeFinalize(reinterpret_cast<__nv_bfloat16 const*>(fc2.data_ptr()), + fc2.stride(0), weights.data_ptr(), weights.scalar_type() == at::kFloat, idx.data_ptr(), + shared.has_value() ? reinterpret_cast<__nv_bfloat16 const*>(shared->data_ptr()) : nullptr, + reinterpret_cast<__nv_bfloat16*>(out.data_ptr()), tokens, static_cast(topk), static_cast(hidden), + static_cast(cfg), at::cuda::getCurrentCUDAStream(fc2.get_device())); + TORCH_CHECK(e == cudaSuccess, "minimax_m31_moe_finalize launch: ", cudaGetErrorString(e)); + return out; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_moe_finalize(Tensor fc2, Tensor weights, Tensor idx, Tensor? shared=None, int cfg=-1) -> Tensor"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_moe_finalize", &tensorrt_llm::torch_ext::minimaxM31MoeFinalize); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31S1bFoldOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31S1bFoldOp.cpp new file mode 100644 index 000000000000..33818b0b3fdd --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31S1bFoldOp.cpp @@ -0,0 +1,81 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/s1bFoldSm107.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! S1b fold (see kernels/minimaxM31/s1bFoldSm107.h): partials fp32 [S, M, 128] (S 8 or 16), bias fp32 [128] -> +//! ids int32 [M, 4] and weights bf16 [M, 4] (written), logits_out fp32 [M, 128] or None. +void minimaxM31S1bFold(torch::Tensor const& partials, torch::Tensor const& bias, std::optional logitsOut, + torch::Tensor& ids, torch::Tensor& weights, double routeScale, bool pdl) +{ + TORCH_CHECK(partials.is_cuda() && partials.scalar_type() == torch::kFloat32 && partials.dim() == 3 + && partials.is_contiguous() && partials.size(2) == 128 && (partials.size(0) == 8 || partials.size(0) == 16), + "minimax_m31_s1b_fold: partials must be a contiguous float32 [8 | 16, M, 128] CUDA tensor"); + int64_t const M = partials.size(1); + TORCH_CHECK(M <= 65535 * 8, "minimax_m31_s1b_fold: too many rows"); + TORCH_CHECK(bias.is_cuda() && bias.scalar_type() == torch::kFloat32 && bias.is_contiguous() && bias.numel() == 128 + && bias.get_device() == partials.get_device(), + "minimax_m31_s1b_fold: bias must be a contiguous float32 [128] tensor on partials' device"); + TORCH_CHECK(ids.is_cuda() && ids.scalar_type() == torch::kInt32 && ids.is_contiguous() && ids.numel() == M * 4, + "minimax_m31_s1b_fold: ids must be a contiguous int32 [M, 4] tensor"); + TORCH_CHECK(weights.is_cuda() && weights.scalar_type() == torch::kBFloat16 && weights.is_contiguous() + && weights.numel() == M * 4, + "minimax_m31_s1b_fold: weights must be a contiguous bf16 [M, 4] tensor"); + float* lo = nullptr; + if (logitsOut.has_value()) + { + auto& t = logitsOut.value(); + TORCH_CHECK(t.is_cuda() && t.scalar_type() == torch::kFloat32 && t.is_contiguous() && t.numel() == M * 128, + "minimax_m31_s1b_fold: logits_out must be a contiguous float32 [M, 128] tensor"); + lo = t.data_ptr(); + } + at::cuda::CUDAGuard const guard(partials.device()); + auto e = tensorrt_llm::kernels::minimaxM31::invokeS1bFold(partials.data_ptr(), + static_cast(partials.size(0)), static_cast(M), bias.data_ptr(), + static_cast(routeScale), lo, ids.data_ptr(), + reinterpret_cast<__nv_bfloat16*>(weights.data_ptr()), pdl, + at::cuda::getCurrentCUDAStream(partials.get_device())); + TORCH_CHECK(e == cudaSuccess, "minimax_m31_s1b_fold: ", cudaGetErrorString(e)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_s1b_fold(Tensor partials, Tensor bias, Tensor(a!)? logits_out, Tensor(b!) ids, Tensor(c!) " + "weights, " + "float route_scale, bool pdl) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_s1b_fold", &tensorrt_llm::torch_ext::minimaxM31S1bFold); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31S2MoeTailOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31S2MoeTailOp.cpp new file mode 100644 index 000000000000..df62edbca799 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31S2MoeTailOp.cpp @@ -0,0 +1,87 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +#include "tensorrt_llm/common/config.h" +#include "tensorrt_llm/kernels/minimaxM31/s2MoeTailSm107.h" + +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +std::tuple m31S2MoeTail(torch::Tensor const& fc2, + torch::Tensor const& scales, torch::Tensor const& perm, torch::Tensor const& shared, torch::Tensor const& residual, + torch::Tensor const& norm_weight, torch::Tensor const& workspace, int64_t rank, int64_t nranks, double eps, + bool trigger_completion_at_end) +{ + TORCH_CHECK(fc2.is_cuda() && fc2.scalar_type() == at::kBFloat16 && fc2.dim() == 2 && fc2.stride(1) == 1); + int64_t const hidden = fc2.size(1); + int64_t const tokens = shared.size(0); + TORCH_CHECK(shared.is_contiguous() && shared.scalar_type() == at::kBFloat16 && shared.size(1) == hidden); + TORCH_CHECK( + residual.is_contiguous() && residual.scalar_type() == at::kBFloat16 && residual.sizes() == shared.sizes()); + TORCH_CHECK( + norm_weight.is_contiguous() && norm_weight.scalar_type() == at::kBFloat16 && norm_weight.numel() == hidden); + TORCH_CHECK(perm.is_contiguous() && perm.scalar_type() == at::kInt && perm.dim() == 2 && perm.size(0) == tokens); + TORCH_CHECK(scales.is_contiguous() && scales.sizes() == perm.sizes() + && (scales.scalar_type() == at::kBFloat16 || scales.scalar_type() == at::kFloat)); + TORCH_CHECK(tokens >= 1 && tokens <= tensorrt_llm::kernels::minimaxM31Sm107::s2tail::kMaxTokens, + "one-shot Lamport rows only (<= 256)"); + TORCH_CHECK(hidden % 256 == 0 && (fc2.stride(0) % 8) == 0); + TORCH_CHECK(nranks == 2, "TP2 only"); + at::cuda::CUDAGuard const guard(fc2.device()); + auto const opts = shared.options(); + torch::Tensor norm_out = torch::empty_like(shared); + torch::Tensor residual_out = torch::empty_like(residual); + torch::Tensor q = torch::empty({tokens, hidden}, opts.dtype(at::ScalarType::Float8_e4m3fn)); + int64_t const sf_size = (tokens + 127) / 128 * 128 * ((hidden / 32 + 3) / 4 * 4); + torch::Tensor sf = torch::empty({sf_size}, opts.dtype(at::ScalarType::Byte)); + tensorrt_llm::kernels::minimaxM31Sm107::s2tail::Params p{}; + p.nranks = static_cast(nranks); + p.rank = static_cast(rank); + p.size = static_cast(tokens * hidden); + p.hidden_dim = static_cast(hidden); + p.top_k = static_cast(perm.size(1)); + p.ld_fc2 = static_cast(fc2.stride(0)); + p.workspace = reinterpret_cast(workspace.data_ptr()); + p.fc2 = reinterpret_cast<__nv_bfloat16 const*>(fc2.data_ptr()); + p.scales = scales.data_ptr(); + p.perm = perm.data_ptr(); + p.shared = reinterpret_cast<__nv_bfloat16 const*>(shared.data_ptr()); + p.residual_in = reinterpret_cast<__nv_bfloat16 const*>(residual.data_ptr()); + p.rms_gamma = reinterpret_cast<__nv_bfloat16 const*>(norm_weight.data_ptr()); + p.rms_eps = static_cast(eps); + p.residual_out = reinterpret_cast<__nv_bfloat16*>(residual_out.data_ptr()); + p.norm_out = reinterpret_cast<__nv_bfloat16*>(norm_out.data_ptr()); + p.quant_out = reinterpret_cast(q.data_ptr()); + p.scale_out = reinterpret_cast(sf.data_ptr()); + auto stream = at::cuda::getCurrentCUDAStream(fc2.get_device()); + tensorrt_llm::kernels::minimaxM31Sm107::s2tail::invokeS2MoeTail( + p, scales.scalar_type() == at::kBFloat16, trigger_completion_at_end, stream); + return {norm_out, residual_out, q, sf}; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "minimax_m31_s2_moe_tail(Tensor fc2, Tensor scales, Tensor perm, Tensor shared, Tensor residual, " + "Tensor norm_weight, Tensor workspace, int rank, int nranks, float eps, bool trigger_completion_at_end=True) " + "-> (Tensor, Tensor, Tensor, Tensor)"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_s2_moe_tail", &tensorrt_llm::torch_ext::m31S2MoeTail); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SoftmaxPathBOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SoftmaxPathBOp.cpp new file mode 100644 index 000000000000..4cd943390151 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SoftmaxPathBOp.cpp @@ -0,0 +1,81 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/softmaxPathBSm107.h" +#include +#include +#include +#include +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ +namespace +{ +bool overlaps(torch::Tensor const& lhs, torch::Tensor const& rhs) +{ + auto const lhsBegin = reinterpret_cast(lhs.data_ptr()); + auto const rhsBegin = reinterpret_cast(rhs.data_ptr()); + return lhsBegin < rhsBegin + rhs.nbytes() && rhsBegin < lhsBegin + lhs.nbytes(); +} +} // namespace + +void minimaxM31SoftmaxPathB(torch::Tensor const& logits, std::optional const& temperature, + double temperatureValue, torch::Tensor& output) +{ + using namespace tensorrt_llm::kernels::minimaxM31; + TORCH_CHECK( + logits.is_cuda() && logits.scalar_type() == torch::kFloat32 && logits.dim() == 2 && logits.is_contiguous(), + "softmax_pathb: expected contiguous fp32 CUDA matrix"); + int64_t const rows = logits.size(0); + TORCH_CHECK(rows >= kSoftmaxPathBMinRows && rows <= kSoftmaxPathBMaxRows && logits.size(1) == kSoftmaxPathBVocab + && reinterpret_cast(logits.data_ptr()) % 16 == 0, + "softmax_pathb: expected aligned [129..448, 200064] logits"); + TORCH_CHECK(output.device() == logits.device() && output.scalar_type() == torch::kFloat32 + && output.sizes() == logits.sizes() && output.is_contiguous() + && reinterpret_cast(output.data_ptr()) % 16 == 0 && !overlaps(logits, output), + "softmax_pathb: output must match logits and not overlap it"); + c10::cuda::CUDAGuard const deviceGuard{logits.device()}; + auto const* properties = at::cuda::getDeviceProperties(logits.get_device()); + TORCH_CHECK(properties->major == 10 && properties->minor == 7, "softmax_pathb: SM107 required"); + float* temperatureArray = nullptr; + if (temperature.has_value()) + { + auto const& t = temperature.value(); + TORCH_CHECK(t.device() == logits.device() && t.scalar_type() == torch::kFloat32 && t.dim() == 1 + && t.size(0) == rows && t.is_contiguous() && !overlaps(t, output), + "softmax_pathb: temperature must be nonoverlapping contiguous fp32 [rows] on logits device"); + temperatureArray = t.data_ptr(); + } + auto const error = invokeSoftmaxPathB(logits.data_ptr(), output.data_ptr(), temperatureArray, + static_cast(temperatureValue), static_cast(rows), + at::cuda::getCurrentCUDAStream(logits.get_device())); + TORCH_CHECK(error == cudaSuccess, "softmax_pathb launch failed: ", cudaGetErrorString(error)); +} +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m31_softmax_pathb(Tensor logits, Tensor? temperature, float temperature_val, Tensor(a!) out) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_softmax_pathb", &tensorrt_llm::torch_ext::minimaxM31SoftmaxPathB); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SwigluOaiVecOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SwigluOaiVecOp.cpp new file mode 100644 index 000000000000..9bf3bd6f3120 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31SwigluOaiVecOp.cpp @@ -0,0 +1,62 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/swigluOaiVecSm107.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! NATIVE_SWIGLU: x = [gate | up] contiguous bf16 [rows, 2d] -> bf16 [rows, d], bit for bit the +//! Triton silu_and_mul at limit 7 / alpha 1.702 / beta 1. nv: 16-byte output vectors per thread (1 / 2 / 4). +torch::Tensor minimaxM31SwigluOaiVec(torch::Tensor x, bool pdl, int64_t nv) +{ + TORCH_CHECK(x.dim() == 2 && x.scalar_type() == at::kBFloat16 && x.is_contiguous() && x.size(1) % 2 == 0, + "x: contiguous bf16 [rows, 2d]"); + int64_t const rows = x.size(0), d = x.size(1) / 2; + TORCH_CHECK(d % 2 == 0, "d must be even (packed pairs)"); + TORCH_CHECK(nv == 1 || nv == 2 || nv == 4, "nv must be 1, 2 or 4"); + auto out = torch::empty({rows, d}, x.options()); + if (rows == 0) + { + return out; + } + at::cuda::CUDAGuard const guard(x.device()); + cudaError_t const e = tensorrt_llm::kernels::minimaxM31::invokeSwigluOaiVec(x.data_ptr(), out.data_ptr(), rows, d, + static_cast(nv), pdl, at::cuda::getCurrentCUDAStream(x.get_device())); + TORCH_CHECK(e == cudaSuccess, "minimax_m31_swiglu_oai_vec launch: ", cudaGetErrorString(e)); + return out; +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m31_swiglu_oai_vec(Tensor x, bool pdl=True, int nv=1) -> Tensor"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_swiglu_oai_vec", &tensorrt_llm::torch_ext::minimaxM31SwigluOaiVec); +} diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetV3Op.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetV3Op.cpp index d125c4b79c0e..d43a07c9ea7a 100644 --- a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetV3Op.cpp +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31ToppRenormDetV3Op.cpp @@ -33,8 +33,10 @@ namespace torch_ext //! output, with the redundant per-CTA selection and an optional programmatic dependent launch. workspace: a //! zero-initialized byte buffer of at least rows x minimax_m31_topp_renorm_det_v3_ws_bytes(), reused across calls; //! ctas_per_row divides 1024 and rows x ctas_per_row must not exceed the SM count when ctas_per_row > 1. +//! list_path (V4) / wide_hist (K8): the one-sync list path and 8 privatized pass histograms; both bitwise the same. +//! Both default to off, so a caller that does not pass them runs the v3 kernel unchanged. void minimaxM31ToppRenormDetV3(torch::Tensor const& probs, torch::Tensor const& topP, torch::Tensor& workspace, - torch::Tensor& out, int64_t ctasPerRow, bool pdl, bool candidateList) + torch::Tensor& out, int64_t ctasPerRow, bool pdl, bool candidateList, bool listPath, bool wideHist) { TORCH_CHECK(probs.is_cuda() && probs.scalar_type() == torch::kFloat32 && probs.dim() == 2 && probs.is_contiguous(), "minimax_m31_topp_renorm_det_v3: probs must be a contiguous 2-D float32 CUDA tensor"); @@ -65,10 +67,10 @@ void minimaxM31ToppRenormDetV3(torch::Tensor const& probs, torch::Tensor const& return; } c10::cuda::CUDAGuard const deviceGuard{probs.device()}; - auto const err - = tensorrt_llm::kernels::minimaxM31::invokeToppRenormDetV3(probs.data_ptr(), out.data_ptr(), - topP.data_ptr(), static_cast(rows), static_cast(d), static_cast(ctasPerRow), - workspace.data_ptr(), nullptr, pdl, candidateList, at::cuda::getCurrentCUDAStream(probs.get_device())); + auto const err = tensorrt_llm::kernels::minimaxM31::invokeToppRenormDetV3(probs.data_ptr(), + out.data_ptr(), topP.data_ptr(), static_cast(rows), static_cast(d), + static_cast(ctasPerRow), workspace.data_ptr(), nullptr, pdl, candidateList, listPath, wideHist, + at::cuda::getCurrentCUDAStream(probs.get_device())); TORCH_CHECK(err == cudaSuccess, "minimax_m31_topp_renorm_det_v3 launch failed: ", cudaGetErrorString(err)); } @@ -86,7 +88,7 @@ TORCH_LIBRARY_FRAGMENT(trtllm, m) { m.def( "minimax_m31_topp_renorm_det_v3(Tensor probs, Tensor top_p, Tensor(a!) workspace, Tensor(b!) out, " - "int ctas_per_row, bool pdl, bool candidate_list) -> ()"); + "int ctas_per_row, bool pdl, bool candidate_list, bool list_path=False, bool wide_hist=False) -> ()"); // No tensor arguments: a catch-all kernel (as moe_a2a_get_timeout), callable without a device. m.def( "minimax_m31_topp_renorm_det_v3_ws_bytes() -> int", &tensorrt_llm::torch_ext::minimaxM31ToppRenormDetV3WsBytes); diff --git a/cpp/tensorrt_llm/thop/moe/trtllmGen/fp4BlockScaleMoe.cpp b/cpp/tensorrt_llm/thop/moe/trtllmGen/fp4BlockScaleMoe.cpp index af7aae8807fd..b8c492143e1a 100644 --- a/cpp/tensorrt_llm/thop/moe/trtllmGen/fp4BlockScaleMoe.cpp +++ b/cpp/tensorrt_llm/thop/moe/trtllmGen/fp4BlockScaleMoe.cpp @@ -554,6 +554,13 @@ class FP4BlockScaleMoeRunner : public torch::CustomClassHolder return *runner; } + //! The kernel names of tactic (tileN, config) as getValidConfigs returns it (the non-fine-grained runner of that + //! tile): a stable name for tactic caches. + [[nodiscard]] std::string getConfigName(int64_t tileN, int64_t config) + { + return getRunner(static_cast(tileN), /* useFineGrained */ false).getConfigName(config); + } + // Tuned over the non-fine-grained tile set only: the fine-grained kernels are never profiled, // so they reuse the tuned tile with their own default config. [[nodiscard]] std::vector> getValidConfigs( @@ -760,6 +767,7 @@ TORCH_LIBRARY_FRAGMENT(trtllm, m) m.class_("FP4BlockScaleMoERunner") .def(torch::init()) .def("get_valid_configs", &tensorrt_llm::torch_ext::FP4BlockScaleMoeRunner::getValidConfigs) + .def("get_config_name", &tensorrt_llm::torch_ext::FP4BlockScaleMoeRunner::getConfigName) .def("run_moe", &tensorrt_llm::torch_ext::FP4BlockScaleMoeRunner::run); m.class_("FP8FP4BlockScaleMoERunner") .def(torch::init()) diff --git a/cpp/tensorrt_llm/thop/mxfp8Gemm.cpp b/cpp/tensorrt_llm/thop/mxfp8Gemm.cpp index 576f88a1f833..129f6cd83b9b 100644 --- a/cpp/tensorrt_llm/thop/mxfp8Gemm.cpp +++ b/cpp/tensorrt_llm/thop/mxfp8Gemm.cpp @@ -29,6 +29,7 @@ #include #include #include +#include #include #include @@ -298,6 +299,12 @@ class MXFP8GemmRunner : public torch::CustomClassHolder return static_cast(mConfigs.size()); } + //! A stable description of native tactic ``configIdx`` (its CUTLASS config), for name-keyed tactic caches. + std::string getConfigName(int64_t configIdx) const + { + return getConfig(configIdx).toString(); + } + private: tkc::CutlassGemmConfig const& getConfig(int64_t const configIdx) const { @@ -320,6 +327,7 @@ TORCH_LIBRARY_FRAGMENT(trtllm, m) .def(torch::init()) .def("run_gemm", &tensorrt_llm::torch_ext::MXFP8GemmRunner::runGemm) .def("get_num_configs", &tensorrt_llm::torch_ext::MXFP8GemmRunner::getNumConfigs) + .def("get_config_name", &tensorrt_llm::torch_ext::MXFP8GemmRunner::getConfigName) .def("register_tactic", &tensorrt_llm::torch_ext::MXFP8GemmRunner::registerTactic) .def("get_cached_tactic", &tensorrt_llm::torch_ext::MXFP8GemmRunner::getCachedTactic) .def("clear_tactic_cache", &tensorrt_llm::torch_ext::MXFP8GemmRunner::clearTacticCache); diff --git a/requirements.txt b/requirements.txt index 0e3070694b06..694110796576 100644 --- a/requirements.txt +++ b/requirements.txt @@ -89,7 +89,7 @@ tiktoken blobfile openai-harmony==0.0.4 # 4.8.0.dev0 is the first release exposing Rubin/SM107. -nvidia-cutlass-dsl[cu13]==4.8.0.dev0; python_version >= "3.10" # required by PrimTS task-scheduled kernels and Rubin CuTe DSL kernels +nvidia-cutlass-dsl[cu13]==4.8.0; python_version >= "3.10" # required by PrimTS task-scheduled kernels and Rubin CuTe DSL kernels nvidia-matmul-heuristics==0.1.0.27; python_version >= "3.10" # analytic GEMM heuristics for CuTe DSL autotuner tactic pruning quack-kernels==0.5.0; python_version >= "3.10" # required for MiniMax-M3 MSA and FlashAttention 4; only published release compatible with CUTLASS DSL >= 4.7 jinja2 # required for MinimaxM3 MSA diff --git a/setup.py b/setup.py index 80181402552d..5cbc425bd441 100644 --- a/setup.py +++ b/setup.py @@ -203,6 +203,8 @@ def has_ext_modules(self): 'bindings/**/*.pyi', 'evaluate/lm_eval_tasks/**/*', 'usage/schemas/*.json', + '_torch/_experimental/modeling_v2/models/minimax_m3/m31_tuning/**/*.json', + '_torch/cute_aot_data/**/*.o', 'grpc/openengine/_generated/*.pyi', 'grpc/openengine/proto/manifest.json', 'grpc/openengine/proto/openengine/v1/*.proto', diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/_op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/_op.py index 3dc75761b4a3..4baee1ba64b6 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/_op.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/_op.py @@ -88,6 +88,7 @@ class Arch(str, enum.Enum): """ SM_103 = "sm_103" + SM_107 = "sm_107" @dataclass(frozen=True) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast.py index 24d10fae21bc..50e84a85b339 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/allgather_cast.py @@ -20,10 +20,13 @@ import importlib import os import threading +from pathlib import Path from typing import Callable, Dict, Optional, Sequence, Tuple import torch +from tensorrt_llm._torch import cute_aot + from .._op import Arch, Cell, OpWrapper MAX_ROWS = 64 # capacity: fixed: the workspace's rows per call (K3 verify rows at batch 16) @@ -58,14 +61,42 @@ def _kernel_module(): # world size, the rows and the device only, so every rank picks the same grid. GRID_BY_ROWS = ((4, 96), (8, 128), (12, 152), (16, 128), (32, 152)) GRID_BY_WORLD_ROWS = {4: ((4, 96), (8, 128), (32, 152)), 8: GRID_BY_ROWS} +# Rubin (sm_107, compute capability 10.7) overrides by world size, measured at 96 / 128 / 152 CTAs, us per call. +# TP8 over two trays: 4 rows 14.53 / 14.51 / 14.75, 8 rows 18.99 / 16.20 / 16.27, 16 rows 23.62 / 21.57 / 22.18, +# 24 rows 30.61 / 26.54 / 25.59, 32 rows 33.92 / 33.63 / 29.61, 40 rows 41.34 / 35.96 / 36.09, 48 rows 43.76 / 40.06 / +# 39.27, 56 rows 50.02 / 46.92 / 42.70, 64 rows 53.02 / 50.12 / 48.51. TP4 on one tray: 4 rows 14.46 / 13.33 / 13.41, +# 8 rows 16.66 / 15.78 / 15.84, 16 rows 21.61 / 21.54 / 19.97, 24 rows 26.78 / 24.93 / 24.20, 32 rows 31.26 / 29.63 / +# 28.96, 40 rows 36.60 / 34.14 / 33.17, 48 rows 41.00 / 39.50 / 37.89, 56 rows 47.02 / 43.53 / 41.05, 64 rows 49.86 / +# 47.30 / 46.15. Every grid is bitwise at every row count; the fastest grid per row count is kept, ties within 0.5 us +# going to the smaller grid. World sizes without a Rubin row keep the tables above. +GRID_BY_WORLD_ROWS_SM107 = {4: ((8, 128), (64, 152)), 8: ((16, 128), (64, 152))} + + +def configure_capacity(rows: int) -> None: + """Set the workspace's rows per call for this process (``MAX_ROWS``; ``MAX_ROWS_FP32`` follows at half): a + target whose verify steps exceed 64 rows (the TP2 agg steps) raises it before its first ``workspace_args``. + Every rank of the TP group must call it with the same value; once a workspace exists the capacity is fixed (its + slot layout and the compiled shapes depend on it), and a different value raises.""" + global MAX_ROWS, MAX_ROWS_FP32 + rows = int(rows) + if rows <= 0 or rows % 2: + raise ValueError(f"allgather_cast: capacity must be a positive even row count, got {rows}") + if rows == MAX_ROWS: + return + if _workspaces: + raise RuntimeError( + f"allgather_cast: capacity {MAX_ROWS} is fixed once a workspace exists (asked {rows})" + ) + MAX_ROWS, MAX_ROWS_FP32 = rows, rows // 2 def pick_grid(rows: int, device=None, world: int = 8) -> int: """CTAs for a call of ``rows`` rows over ``world`` ranks (capped by the SM count and MAX_GRID).""" - sms = torch.cuda.get_device_properties( - device if device is not None else torch.cuda.current_device() - ).multi_processor_count + dev = device if device is not None else torch.cuda.current_device() + sms = torch.cuda.get_device_properties(dev).multi_processor_count table = GRID_BY_WORLD_ROWS.get(world, GRID_BY_ROWS) + if tuple(torch.cuda.get_device_capability(dev)) == (10, 7): + table = GRID_BY_WORLD_ROWS_SM107.get(world, table) grid = next((g for bound, g in table if rows <= bound), MAX_GRID) return min(MAX_GRID, sms, grid) @@ -219,8 +250,12 @@ def _kernel_for( with _lock: fn = _compiled.get(key) if fn is None: - fn = _compiled[key] = cute.compile( - kern.allgather_cast, *args, int(rank), *consts, use_pdl, stream + fn = _compiled[key] = cute_aot.compile_or_load( + f"allgather_cast {key}", + lambda: cute.compile( + kern.allgather_cast, *args, int(rank), *consts, use_pdl, stream + ), + sources=cute_aot.py_files(str(Path(__file__).resolve().parent)), ) return fn, args, stream @@ -379,7 +414,7 @@ class AllgatherCast(OpWrapper): rows, S and grid, in the same order. """ - ARCHS = frozenset({Arch.SM_103}) + ARCHS = frozenset({Arch.SM_103, Arch.SM_107}) # MiniMax-M3.1's target LM head at TP8 over two GB300 trays: vocabulary 200064 (shard 25008), K3 verify rows # B * (K + 1): K3 at batches 1..16, K7 at batches 1..8. @@ -471,7 +506,7 @@ class AllgatherF32(OpWrapper): (0 picks). ``pdl``: as ``allgather_cast``'s. A collective, as ``allgather_cast``. """ - ARCHS = frozenset({Arch.SM_103}) + ARCHS = frozenset({Arch.SM_103, Arch.SM_107}) # MiniMax-M3.1's DSpark drafter at TP8 over two GB300 trays: the rejection sampler's draft logits, B * K rows of # the 25008-column vocabulary shard. diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/tp2ctx_ipc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/tp2ctx_ipc.py new file mode 100644 index 000000000000..412e35d740bd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/comm/tp2ctx_ipc.py @@ -0,0 +1,347 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2CTX_SP / AR1 workspace on MPI IPC: two-rank prefill boundary collectives without torch.distributed. + +The VR tp2ctx CTX workers are MPI-launched and have no torch process group, so the peer buffers come from TRT's +``IpcMemory`` (cudaIpc handles exchanged over the TP communicator) and the cross-rank sync is a one-CTA device-flag +barrier (``torch.ops.trtllm.minimax_m31_tp2ctx_ipc_barrier``, ~4.8 us on VR). Microbench (MPI only, T x 6144 bf16): +the sequence-parallel boundary (reduce-scatter of the owned rows + all-gather of the packed MXFP8 rows) is 91.9 vs +127.5 us at 8K and 163.2 vs 233.7 us at 16K against the NCCL ring all-reduce, bit for bit (the two-operand bf16 sum +is commutative). + +Row ownership is fixed for a step: rank 0 owns rows [0, S), rank 1 [S, T), where S is the 128-row boundary nearest +T / 2 (S = T / 2 whenever T % 256 == 0). Rank 0's rows are then whole 128-row MXFP8 scale-factor tiles and rank 1's +block starts on a tile, so each rank's swizzled scale bytes are one contiguous range of the full step's +(``own_sf_slice``); rank 1's last tile carries the same zero pad rows as the full step's quantize. Per boundary: +the producer writes its partial into ``input_slot(T)``; ``reduce_scatter_owned`` (barrier, owned-row peer add); the +caller writes its owned outputs into ``gather_view(kind, T)`` at ``own_rows(T)`` (the scale factors at +``own_sf_slice(T)``); ``push_owned(kinds, T)`` pushes them into the peer's views and barriers. +Two barriers per boundary make single-buffered slots safe: a peer's read of buffer n finishes before +either rank passes the next barrier, and every overwrite of buffer n + 1 comes after it. Eager prefill only. Kill +switch: TRTLLM_M31_DISABLE_TP2CTX_SP=1. +""" + +from __future__ import annotations + +import os + +import torch + +_DEFAULT_FUSION_WORKSPACE_BYTES = 64 * 1024 * 1024 +_SF_VEC = 32 # MXFP8 scale-factor block +_SF_TILE_ROWS = 128 # swizzled scale-factor tile: 128 rows x 4 scale columns + + +def enabled() -> bool: + """Default ON; TRTLLM_M31_DISABLE_TP2CTX_SP=1 restores the all-reduce boundary.""" + return os.environ.get("TRTLLM_M31_DISABLE_TP2CTX_SP", "0") != "1" + + +# TP2CTX_ROUTER_ZC (prefill): the pre-MoE boundary writes its fp32 router rows straight into this +# rank's rows of the router's full-step buffer (``router_owned_input``), and ``route_owned_full_shape`` no longer +# zeroes the [T, K] buffer or copies the owned rows in per call (a 165K-token prefill: 570 x 201 MB copies + 570 +# x 402 MB zero fills per request). The rows outside this rank's own hold earlier router inputs (or the zeros of +# the allocation): each output row of the router GEMM depends only on its own input row, and the call's shape and +# algorithm are unchanged, so the owned logits stay bit for bit. Default ON; TRTLLM_M31_DISABLE_TP2CTX_ROUTER_ZC=1 +# restores the per-call zero + copy. Read once at import. +_ROUTER_ZC = os.environ.get("TRTLLM_M31_DISABLE_TP2CTX_ROUTER_ZC", "0") != "1" + + +def router_zc_enabled() -> bool: + return _ROUTER_ZC + + +def _nccl_fallback_bytes() -> int: + env = os.environ.get("TRTLLM_ALLREDUCE_FUSION_WORKSPACE_SIZE") + return int(env) if env else _DEFAULT_FUSION_WORKSPACE_BYTES + + +def _view(ptr: int, shape, dtype: torch.dtype, device: torch.device) -> torch.Tensor: + """A tensor over raw device memory owned by an IpcMemory (no copy, no ownership).""" + nbytes = 1 + for d in shape: + nbytes *= d + nbytes *= torch.empty((), dtype=dtype).element_size() + + class _Iface: + __cuda_array_interface__ = { + "shape": (nbytes,), + "typestr": "|u1", + "data": (ptr, False), + "version": 3, + } + + return torch.as_tensor(_Iface(), device=device).view(dtype).view(shape) + + +# kind -> (bytes per row, layout); every layout is row-major in whole rows except the swizzled MXFP8 scale +# factors, whose 128-row tiles are contiguous blocks (owned-half quantize == the full step's block: +# UT test_owned_half_quantize_...). +_KINDS = { + "mxfp8": "hidden", # e4m3 data [T, hidden] + "mxfp8_sf": "sf_swizzled", # UE8M0, swizzled 128x4, K32 + "nvfp4": "hidden_half", # packed e2m1 [T, hidden / 2] + "nvfp4_sf": "nvfp4_sf", # E4M3, linear [T, hidden / 16] + "topk_ids": "topk", # int32 [T, 4] + "topk_w": "topk", # fp32 [T, 4] + "bf16": "bf16", # [T, hidden] + "logits": "logits", # fp32 [T, 128]: router logits (TP2AGG_SP, whose trtllm-gen experts route in-kernel) +} +_LOGITS_COLS = 128 + + +class Tp2ctxIpcWorkspace: + """Per-process two-rank IPC workspace; build it collectively (both TP ranks) at model init, outside capture.""" + + def __init__(self, mapping, max_tokens: int, hidden: int, device: torch.device): + from tensorrt_llm._ipc_utils import IpcMemory + + if mapping.tp_size != 2: + raise ValueError(f"Tp2ctxIpcWorkspace needs TP2, got {mapping.tp_size}") + self.rank = mapping.tp_rank + self.peer = 1 - self.rank + self.max_tokens, self.hidden, self.device = max_tokens, hidden, device + sf_cols_pad = (hidden // _SF_VEC + 3) // 4 * 4 + self._row_bytes = { + "hidden": hidden, + "sf_swizzled": sf_cols_pad, + "hidden_half": hidden // 2, + "nvfp4_sf": hidden // 16, + "topk": 16, + "bf16": hidden * 2, + "logits": _LOGITS_COLS * 4, + } + rows_pad = (max_tokens + _SF_TILE_ROWS - 1) // _SF_TILE_ROWS * _SF_TILE_ROWS + sizes = {"slot": max_tokens * hidden * 2} + for kind, layout in _KINDS.items(): + sizes[kind] = (rows_pad if layout == "sf_swizzled" else max_tokens) * self._row_bytes[ + layout + ] + self._mem = {k: IpcMemory(mapping, v) for k, v in sizes.items()} + self._flags = IpcMemory(mapping, 4096) # zeroed by IpcMemory + self._state = torch.zeros(4, dtype=torch.int32, device=device) + self._router_buf = None + self._side = None + self._pending_push = None + + def _ptr(self, kind: str, peer: bool = False) -> int: + m = self._mem[kind] + return m.peer_ptrs[self.peer] if peer else m.local_ptr + + def supports(self, num_tokens: int, *, nccl_only: bool = True) -> bool: + """Storage capacity, optionally restricted to rows above the fused all-reduce crossover.""" + return 2 * _SF_TILE_ROWS <= num_tokens <= self.max_tokens and ( + not nccl_only or num_tokens * self.hidden * 2 > _nccl_fallback_bytes() + ) + + @staticmethod + def split_row(num_tokens: int) -> int: + """The first row of rank 1: the 128-row boundary nearest T / 2 (exactly T / 2 when T % 256 == 0).""" + split = (num_tokens // 2 + _SF_TILE_ROWS // 2) // _SF_TILE_ROWS * _SF_TILE_ROWS + return min(max(split, _SF_TILE_ROWS), num_tokens - 1) + + def own_rows(self, num_tokens: int): + """(first owned row, owned row count): rank 0 owns [0, S), rank 1 [S, T) (S = ``split_row(T)``).""" + split = self.split_row(num_tokens) + return (0, split) if self.rank == 0 else (split, num_tokens - split) + + def own_sf_slice(self, num_tokens: int) -> slice: + """This rank's byte range of the swizzled MXFP8 scale factors in ``gather_view("mxfp8_sf", T)``: the 128-row + tiles of its rows (rank 1's last tile includes the pad rows up to the full step's 128-row multiple), i.e. + exactly the bytes its own rows' quantize writes.""" + per = self._row_bytes["sf_swizzled"] + split = self.split_row(num_tokens) + rows_pad = (num_tokens + _SF_TILE_ROWS - 1) // _SF_TILE_ROWS * _SF_TILE_ROWS + return slice(0, split * per) if self.rank == 0 else slice(split * per, rows_pad * per) + + def barrier(self) -> None: + self.wait_pending_push() + torch.ops.trtllm.minimax_m31_tp2ctx_ipc_barrier( + self._flags.local_ptr, self._flags.peer_ptrs[self.peer], self._state, self.rank + ) + + def wait_pending_push(self) -> None: + """Make the current stream wait for an outstanding ``push_owned_async`` (its pushes and barrier).""" + event, self._pending_push = self._pending_push, None + if event is not None: + torch.cuda.current_stream(self.device).wait_event(event) + + def peer_sf_slice(self, num_tokens: int) -> slice: + """The peer's byte range of the swizzled MXFP8 scale factors (the complement of ``own_sf_slice``).""" + own = self.own_sf_slice(num_tokens) + rows_pad = (num_tokens + _SF_TILE_ROWS - 1) // _SF_TILE_ROWS * _SF_TILE_ROWS + full = rows_pad * self._row_bytes["sf_swizzled"] + return slice(own.stop, full) if own.start == 0 else slice(0, own.start) + + def push_owned_async( + self, kinds, num_tokens: int, engine: str = "ce", barrier: bool = True + ) -> torch.cuda.Event: + """``push_owned`` off the current stream: after the current stream's work so far, a side stream copies this + rank's owned rows of each kind into the peer's views, by copy engine (``engine="ce"``, no SMs) or by the peer + push kernel (``"sm"``), and runs the barrier unless ``barrier=False`` (then a later ``push_owned`` on the + current stream ends the boundary with it). Returns the event a consumer of the gathered rows waits on; the next + ``barrier`` / ``push_owned`` on the current stream waits on it too, so the barrier epochs stay in order.""" + if engine not in ("ce", "sm"): + raise ValueError(f"push_owned_async: engine must be 'ce' or 'sm', got {engine!r}") + from cuda.bindings import driver as cuda_driver + + self.wait_pending_push() + main = torch.cuda.current_stream(self.device) + if self._side is None: + self._side = torch.cuda.Stream(device=self.device) + side = self._side + side.wait_stream(main) + first, count = self.own_rows(num_tokens) + for kind in kinds: + if _KINDS[kind] == "sf_swizzled": + sl = self.own_sf_slice(num_tokens) + off, nbytes = sl.start, sl.stop - sl.start + else: + per_row = self._row_bytes[_KINDS[kind]] + off, nbytes = first * per_row, count * per_row + if engine == "sm": + with torch.cuda.stream(side): + src = _view(self._ptr(kind) + off, (nbytes,), torch.uint8, self.device) + torch.ops.trtllm.minimax_m31_tp2ctx_peer_push( + src, self._ptr(kind, peer=True) + off + ) + continue + (err,) = cuda_driver.cuMemcpyAsync( + self._ptr(kind, peer=True) + off, self._ptr(kind) + off, nbytes, side.cuda_stream + ) + if err != cuda_driver.CUresult.CUDA_SUCCESS: + raise RuntimeError( + f"Tp2ctxIpcWorkspace.push_owned_async: cuMemcpyAsync({kind}) failed: {err}" + ) + if barrier: + with torch.cuda.stream(side): + torch.ops.trtllm.minimax_m31_tp2ctx_ipc_barrier( + self._flags.local_ptr, self._flags.peer_ptrs[self.peer], self._state, self.rank + ) + event = torch.cuda.Event() + event.record(side) + self._pending_push = event + return event + + def input_slot(self, num_tokens: int) -> torch.Tensor: + """The [T, hidden] bf16 buffer the producer writes its partial into (zero-copy).""" + return _view( + self._ptr("slot"), (self.max_tokens, self.hidden), torch.bfloat16, self.device + )[:num_tokens] + + def reduce_scatter_owned(self, slot: torch.Tensor, out: torch.Tensor = None) -> torch.Tensor: + """The owned rows of the two-rank sum of ``slot`` (bitwise the all-reduce's rows), [T / 2, hidden] bf16.""" + first, count = self.own_rows(slot.shape[0]) + off = first * self.hidden * 2 + if out is None: + out = torch.empty((count, self.hidden), dtype=torch.bfloat16, device=self.device) + self.barrier() + torch.ops.trtllm.minimax_m31_tp2ctx_peer_add( + self._ptr("slot") + off, self._ptr("slot", peer=True) + off, out, count * self.hidden + ) + return out + + def owned_slot_ptrs(self, num_tokens: int): + """(this rank's, the peer's) device addresses of the owned rows of the input slot: the operands of + ``reduce_scatter_owned``'s peer add, for an op that fuses the add into its consumer (call ``barrier()`` + first, as ``reduce_scatter_owned`` does).""" + first, _ = self.own_rows(num_tokens) + off = first * self.hidden * 2 + return self._ptr("slot") + off, self._ptr("slot", peer=True) + off + + def gather_view(self, kind: str, num_tokens: int) -> torch.Tensor: + """The full-T local view of ``kind`` (IPC memory): producers write their owned rows into it, consumers read + all of it after ``push_owned``.""" + layout = _KINDS[kind] + ptr = self._ptr(kind) + if layout == "sf_swizzled": + rows_pad = (num_tokens + _SF_TILE_ROWS - 1) // _SF_TILE_ROWS * _SF_TILE_ROWS + return _view(ptr, (rows_pad * self._row_bytes[layout],), torch.uint8, self.device) + if kind == "mxfp8": + return _view(ptr, (self.max_tokens, self.hidden), torch.uint8, self.device)[:num_tokens] + if kind == "nvfp4": + return _view(ptr, (self.max_tokens, self.hidden // 2), torch.uint8, self.device)[ + :num_tokens + ] + if kind == "nvfp4_sf": + return _view(ptr, (self.max_tokens, self.hidden // 16), torch.uint8, self.device)[ + :num_tokens + ] + if kind == "topk_ids": + return _view(ptr, (self.max_tokens, 4), torch.int32, self.device)[:num_tokens] + if kind == "topk_w": + return _view(ptr, (self.max_tokens, 4), torch.float32, self.device)[:num_tokens] + if kind == "logits": + return _view(ptr, (self.max_tokens, _LOGITS_COLS), torch.float32, self.device)[ + :num_tokens + ] + return _view(ptr, (self.max_tokens, self.hidden), torch.bfloat16, self.device)[:num_tokens] + + def push_owned(self, kinds, num_tokens: int) -> None: + """Copy this rank's owned rows of each kind into the peer's view, then one barrier.""" + self.wait_pending_push() + first, count = self.own_rows(num_tokens) + for kind in kinds: + if _KINDS[kind] == "sf_swizzled": + sl = self.own_sf_slice(num_tokens) + off, nbytes = sl.start, sl.stop - sl.start + else: + per_row = self._row_bytes[_KINDS[kind]] + off, nbytes = first * per_row, count * per_row + src = _view(self._ptr(kind) + off, (nbytes,), torch.uint8, self.device) + torch.ops.trtllm.minimax_m31_tp2ctx_peer_push(src, self._ptr(kind, peer=True) + off) + self.barrier() + + def _router_full(self, num_tokens: int, k: int) -> torch.Tensor: + """The router's full-step [T, K] fp32 input buffer (allocated zeroed, reused across calls).""" + buf = self._router_buf + if buf is None or buf.shape[1] != k or buf.shape[0] < num_tokens: + buf = self._router_buf = torch.zeros( + (self.max_tokens, k), dtype=torch.float32, device=self.device + ) + return buf[:num_tokens] + + def router_owned_input(self, num_tokens: int, k: int): + """TP2CTX_ROUTER_ZC: this rank's rows of the router's full-step input buffer ([own count, K] fp32, + contiguous), for the boundary to write its fp32 router rows into; None when the item is off.""" + if not _ROUTER_ZC: + return None + first, count = self.own_rows(num_tokens) + return self._router_full(num_tokens, k)[first : first + count] + + def route_owned_full_shape( + self, x_owned: torch.Tensor, weight: torch.Tensor, num_tokens: int + ) -> torch.Tensor: + """Router logits of the owned rows, bitwise those of today's full-step router call. + + The router is a cuBLAS fp32 / TF32 GEMM whose algorithm depends on M (split-K at small M), so running it at + M = T / 2 changes the K-reduction order. Instead the owned rows go to their own row offsets in a [T, K] + buffer and the SAME call as today's (F.linear at M = T) runs: identical shape, identical algorithm, identical + per-row result. The cost is today's full-step router GEMM. With TP2CTX_ROUTER_ZC an ``x_owned`` that already + is ``router_owned_input(T, K)`` is used in place, and the other rows are not re-zeroed (see _ROUTER_ZC).""" + first, count = self.own_rows(num_tokens) + full = self._router_full(num_tokens, x_owned.shape[1]) + owned = full[first : first + count] + if not _ROUTER_ZC: + full.zero_() + if x_owned.data_ptr() != owned.data_ptr(): + owned.copy_(x_owned) + return torch.nn.functional.linear(full, weight)[first : first + count] + + +_WS: dict = {} + + +def get_workspace( + mapping, max_tokens: int, hidden: int, device: torch.device +) -> Tp2ctxIpcWorkspace: + """Process-wide workspace (created collectively on first use, outside CUDA-graph capture).""" + key = (mapping.tp_rank, max_tokens, hidden, device.index) + ws = _WS.get(key) + if ws is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "Tp2ctxIpcWorkspace: create the workspace outside CUDA-graph capture" + ) + ws = _WS[key] = Tp2ctxIpcWorkspace(mapping, max_tokens, hidden, device) + return ws diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/m31_fake_ops.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/m31_fake_ops.py index 96c98d1b1afd..471874c25abb 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/m31_fake_ops.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/fakes/m31_fake_ops.py @@ -32,6 +32,17 @@ def _ctx_residual_rmsnorm_mxfp8_fake( return (norm_out, mxfp8, sf, fp4, fp4_sf) +def _ctx_residual_rmsnorm_mxfp8_f32_fake( + inter, residual, weight, eps, fp4_global_scale=None, enable_pdl=False +): + """The _f32 variant (TP2CTX_NORMQ_F32 / TP2AGG_ROUTER_F32): the base op's outputs with write_norm, plus the norm + widened to float32.""" + outs = _ctx_residual_rmsnorm_mxfp8_fake( + inter, residual, weight, eps, True, fp4_global_scale, enable_pdl + ) + return (*outs, torch.empty_like(inter, dtype=torch.float32)) + + def _select_prefill_tile_fake(scores, n_valid_blocks, out, init_blocks, local_blocks): """minimaxM31Sm103SelectPrefillTileOp.cpp: writes the block table into out; returns nothing.""" return None @@ -93,7 +104,25 @@ def _sparse_decode_lc8_fake( return None -def _topp_renorm_det_v3_fake(probs, top_p, workspace, out, ctas_per_row, pdl, candidate_list): +def _swiglu_oai_fake(x, out, pdl): + """m3_dense_swiglu: writes the [rows, d] bf16 activation into out in place; returns nothing.""" + return None + + +def _swiglu_oai_vec_fake(x, pdl=True, nv=1): + """minimaxM31SwigluOaiVecOp.cpp: the SwiGLU-OAI activation of bf16 x [rows, 2d] -> a new [rows, d].""" + return x.new_empty((x.shape[0], x.shape[1] // 2)) + + +def _moe_finalize_fake(fc2, weights, idx, shared=None, cfg=-1): + """minimaxM31MoeFinalizeOp.cpp: the routed experts' rows fc2 [P, H] reduced per token by weights / idx + [tokens, top_k] (+ shared) -> a new [tokens, H] of fc2's dtype.""" + return fc2.new_empty((weights.shape[0], fc2.shape[1])) + + +def _topp_renorm_det_v3_fake( + probs, top_p, workspace, out, ctas_per_row, pdl, candidate_list, list_path=True, wide_hist=True +): """m31_topp_det v3: as minimax_m31_topp_renorm_det (in place into out, workspace updated); returns nothing. minimax_m31_topp_renorm_det_v3_ws_bytes takes no tensor and needs no fake.""" return None @@ -105,12 +134,88 @@ def _topp_renorm_det_fake(probs, top_p, workspace, out, ctas_per_row): return None +def _softmax_pathb_fake(logits, temperature, temperature_val, out): + """Writes fp32 probabilities into out; no workspace and no return value.""" + return None + + def _softmax_mc_fake(logits, temperature, temperature_val, partials, out, write_ctas): """m31_softmax_mc (ST-01-SM): writes the [rows, d] fp32 probs into out in place (and overwrites the partials workspace); returns nothing.""" return None +def _markov_epilogue_fake(base, k, bias, out, prev, w1, emb_out, partials, counters, splits): + """m31_markov_epilogue (S4FE): writes position k of out, prev and emb_out in place; returns nothing.""" + return None + + +def _zero_padding_rows_fake(logits, slot_ids, dummy_slot_row, rows_per_request): + """m31_zero_padding_rows (TAIL16_G2): zeroes padding rows of logits in place; returns nothing.""" + return None + + +def _s1b_fold_fake(partials, bias, logits_out, ids, weights, route_scale, pdl): + """m31_s1b_fold (S1B_FOLD): writes ids / weights (and logits_out) in place; returns nothing.""" + return None + + +def _scatter_rows_fake(dst, slot_ids, src): + """Copies source rows into destination slots in place; returns nothing.""" + return None + + +def _dspark_advance_fake( + slots, + rows, + num_accepted, + ctx_len, + counts, + base_pages, + tables, + read_view, + out, + max_ctx, + dummy_slot, + page, + block_size, +): + """DSPARK_GLUE_CPP: updates ctx_len / read_view / out in place; returns nothing.""" + return None + + +def _dspark_indices_fake( + num_ctx, + batch_idx, + query_positions, + seqlens, + batch_out, + after, + indptr, + last, + append, + qflat, + block, + pages_per_slot, + page_size, +): + """DSPARK_GLUE_CPP: writes the seven int32 index outputs in place; returns nothing.""" + return None + + +def _markov_prev_embed_fake(prev, w1): + """m31_markov_prev_embed (TAIL_BK3 B): bf16 [n, rank].""" + return w1.new_empty((prev.shape[0], w1.shape[1])) + + +def _expand_block_sampling_fake( + temps, top_ks, top_ps, min_ps, seeds, offsets, K, slot_offset, span +): + """m31_expand_block_sampling (TAIL_BK3 C): six [n * K] tensors in the inputs' dtypes.""" + n = temps.shape[0] * K + return [t.new_empty((n,)) for t in (temps, top_ks, top_ps, min_ps, seeds, offsets)] + + def _chain_spec_sampling_mc_fake( draft_probs, draft_token_ids, target_probs, seed, offset, workspace, ctas_per_row ): @@ -124,16 +229,98 @@ def _chain_spec_sampling_mc_fake( ) -# op name -> fake impl (schemas as of img9, ONE_DIV included). +def _chain_spec_sampling_fake(draft_probs, draft_token_ids, target_probs, seed, offset): + """trtllm::chain_speculative_sampling: (output_token_ids [batch, numDraft + 1], num_tokens [batch]), int32.""" + batch_size, num_draft = draft_token_ids.shape + return ( + draft_token_ids.new_empty((batch_size, num_draft + 1)), + draft_token_ids.new_empty((batch_size,)), + ) + + +def _chain_spec_sampling_mc_slots_fake( + draft_buf, + slot_ids, + num_draft, + draft_token_ids, + target_probs, + seed, + offset, + workspace, + ctas_per_row, +): + """m31_chain_spec_mc REJ_SLOTS: as minimax_m31_chain_speculative_sampling_mc, draft rows read by slot.""" + batch_size = draft_token_ids.shape[0] + return ( + draft_token_ids.new_empty((batch_size, num_draft + 1)), + draft_token_ids.new_empty((batch_size,)), + ) + + +def _fc_shard_gemm_fake(x, w, cfg): + """m31_fc_shard: bf16 x [M, K] times w [N, K]^T -> a new [M, N].""" + return x.new_empty((x.shape[0], w.shape[0])) + + +def _s2_moe_tail_fake( + fc2, + scales, + perm, + shared, + residual, + norm_weight, + workspace, + rank, + nranks, + eps, + trigger_completion_at_end=True, +): + """m31_s2_moe_tail: (norm_out, residual_out) like shared / residual, the MXFP8 activation [tokens, hidden] and + its swizzled scale bytes (rows padded to 128, scale columns to 4).""" + tokens, hidden = shared.shape + sf_size = (tokens + 127) // 128 * 128 * ((hidden // 32 + 3) // 4 * 4) + return ( + torch.empty_like(shared), + torch.empty_like(residual), + shared.new_empty((tokens, hidden), dtype=torch.float8_e4m3fn), + shared.new_empty((sf_size,), dtype=torch.uint8), + ) + + +def _sparse_decode_lc_prepare_fake(anchor, kv_heads): + """m3_sparse_decode_lc_prepare reserves the split workspace once, before graph capture; traced, it reserves + nothing.""" + return 0 + + +# op name -> fake impl. FAKES = { + "minimax_m31_fc_shard_gemm": _fc_shard_gemm_fake, "minimax_m31_ctx_residual_rmsnorm_mxfp8": _ctx_residual_rmsnorm_mxfp8_fake, + "minimax_m31_ctx_residual_rmsnorm_mxfp8_f32": _ctx_residual_rmsnorm_mxfp8_f32_fake, "minimax_m31_select_prefill_tile": _select_prefill_tile_fake, "minimax_m31_tp4_sparse_decode_lc": _sparse_decode_lc_fake, "minimax_m31_tp8_sparse_decode_lc8": _sparse_decode_lc8_fake, + "minimax_m31_swiglu_oai": _swiglu_oai_fake, + "minimax_m31_swiglu_oai_vec": _swiglu_oai_vec_fake, + "minimax_m31_moe_finalize": _moe_finalize_fake, "minimax_m31_topp_renorm_det": _topp_renorm_det_fake, "minimax_m31_topp_renorm_det_v3": _topp_renorm_det_v3_fake, "minimax_m31_softmax_mc": _softmax_mc_fake, + "minimax_m31_softmax_pathb": _softmax_pathb_fake, + "minimax_m31_markov_epilogue": _markov_epilogue_fake, + "minimax_m31_zero_padding_rows": _zero_padding_rows_fake, + "minimax_m31_scatter_rows": _scatter_rows_fake, + "minimax_m31_dspark_advance": _dspark_advance_fake, + "minimax_m31_dspark_indices": _dspark_indices_fake, + "minimax_m31_s1b_fold": _s1b_fold_fake, + "minimax_m31_markov_prev_embed": _markov_prev_embed_fake, + "minimax_m31_expand_block_sampling": _expand_block_sampling_fake, + "chain_speculative_sampling": _chain_spec_sampling_fake, "minimax_m31_chain_speculative_sampling_mc": _chain_spec_sampling_mc_fake, + "minimax_m31_chain_speculative_sampling_mc_slots": _chain_spec_sampling_mc_slots_fake, + "minimax_m31_s2_moe_tail": _s2_moe_tail_fake, + "minimax_m3_sm107_tp2_sparse_decode_lc_prepare": _sparse_decode_lc_prepare_fake, } diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_advance.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_advance.py index 5d20f9f90217..3a55c6597776 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_advance.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_ctx_advance.py @@ -25,11 +25,67 @@ # read[row, j] = tables[row, min(j + rebase, width - 1)] (int32) # One program per generation request, so each request's advance is read back by the same program; rows on the dummy # slot (padding) skip the add and read 0, which the stock path's zeroing after the add makes them read too. +# +# ADV_PAR (``parallel=True``, chosen by the caller): at batch 1 the one program copies the whole read-view row +# (width = the pool's max blocks per sequence) alone. With it, the per-request program does the advance only and +# stores its rebase, and a second launch copies the row with one program per _COPY_BLOCK columns. The same values are +# stored (a pure index copy), so bit for bit the one-launch path; one launch cannot split the copy, since its programs +# would race the context-length store. + +import functools +import os +from typing import Optional import torch import triton import triton.language as tl +_adv_par_logged = False +_COPY_BLOCK = 256 + +# DSPARK_GLUE_CPP: the C++ op (torch.ops.trtllm.minimax_m31_dspark_advance, kernels/minimaxM31/dsparkGlueSm107.cu, +# bit for bit the one-launch Triton kernel) on SM107 when it is built, for the one-launch layout (parallel=False); the +# Triton kernels otherwise (other GPUs, parallel=True) or with TRTLLM_M31_DISABLE_DSPARK_GLUE_CPP=1 (read once, at +# import). +_GLUE_CPP_OFF = os.environ.get("TRTLLM_M31_DISABLE_DSPARK_GLUE_CPP", "").strip().lower() in ( + "1", + "true", + "on", + "yes", +) + + +@functools.lru_cache(maxsize=None) +def _is_sm107(device_index: int) -> bool: + return torch.cuda.get_device_capability(device_index) == (10, 7) + + +def _glue_cpp_op(name: str, device: torch.device): + """The C++ op, or None with the reason logged once (the Triton kernel runs).""" + if _GLUE_CPP_OFF: + return _glue_log(name, "killed (TRTLLM_M31_DISABLE_DSPARK_GLUE_CPP=1)") + if not _is_sm107(device.index if device.index is not None else torch.cuda.current_device()): + return _glue_log(name, "not an sm_107 device") + op = getattr(torch.ops.trtllm, name, None) + return op if op is not None else _glue_log(name, "the op is not registered") + + +_glue_logged = set() + + +def _glue_log(name: str, why: Optional[str] = None): + """Engagement line, once per (op, outcome) per process: "[M3 dispatch] DSPARK_GLUE_CPP taken: " when the C++ op + runs, "... not taken: : ; Triton kernel" when it falls back. Returns None.""" + if (name, why) not in _glue_logged: + _glue_logged.add((name, why)) + from tensorrt_llm.logger import logger + + if why is None: + logger.info(f"[M3 dispatch] DSPARK_GLUE_CPP taken: {name}") + else: + logger.info(f"[M3 dispatch] DSPARK_GLUE_CPP not taken: {name}: {why}; Triton kernel") + return None + @triton.jit def _advance_kernel( @@ -75,6 +131,60 @@ def _advance_kernel( tl.store(read_ptr + row * read_stride + j, v, mask=m) +@triton.jit +def _advance_only_kernel( + slots_ptr, # [n] int64 + rows_ptr, # [n] int64 + nacc_ptr, # [n] int + ctx_len_ptr, # [slots] int64, in / out + counts_ptr, # [rows] int64 + base_ptr, # [slots] int64 + out_ptr, # [n] int64 advertised context lengths + rebase_ptr, # [n] int64: each request's read-view rebase, for _read_view_kernel + max_ctx, + dummy_slot, + page, + block_size, +): + r = tl.program_id(0).to(tl.int64) + slot = tl.load(slots_ptr + r) + row = tl.load(rows_ptr + r) + is_dummy = slot == dummy_slot + old = tl.load(ctx_len_ptr + slot) + new = tl.minimum(old + tl.load(nacc_ptr + r).to(tl.int64), max_ctx) + if not is_dummy: + tl.store(ctx_len_ptr + slot, new) + if r == 0: + tl.store(ctx_len_ptr + dummy_slot, tl.zeros((), dtype=tl.int64)) + new = tl.where(is_dummy, 0, new) + allocated = tl.maximum(tl.load(counts_ptr + row) * page - block_size, 0) + num_ctx = tl.minimum(new, allocated) + base = tl.load(base_ptr + slot) + tl.store(rebase_ptr + r, tl.minimum(base, num_ctx // page)) + tl.store(out_ptr + r, tl.maximum(num_ctx - base * page, 0)) + + +@triton.jit +def _read_view_kernel( + rows_ptr, # [n] int64 + rebase_ptr, # [n] int64 + tables_ptr, # int32 [rows, width] + read_ptr, # int32 [rows, width] + width, + tables_stride, + read_stride, + BLOCK: tl.constexpr, +): + r = tl.program_id(0).to(tl.int64) + row = tl.load(rows_ptr + r) + rebase = tl.load(rebase_ptr + r) + j = tl.program_id(1) * BLOCK + tl.arange(0, BLOCK) + m = j < width + src = tl.minimum(j + rebase, width - 1) + v = tl.load(tables_ptr + row * tables_stride + src, mask=m) + tl.store(read_ptr + row * read_stride + j, v, mask=m) + + def advance_ctx_and_read_view( slots: torch.Tensor, rows: torch.Tensor, @@ -88,6 +198,7 @@ def advance_ctx_and_read_view( tables_read: torch.Tensor, page_size: int, block_size: int, + parallel: bool = False, ): """Advance the generation requests' context lengths and build their read view in one launch: returns the advertised context lengths [n] (int64) and writes ``tables_read[rows]``; None (nothing written) for a layout the @@ -113,10 +224,73 @@ def advance_ctx_and_read_view( if n == 0: ctx_len[dummy_slot] = 0 return out + op = None if parallel else _glue_cpp_op("minimax_m31_dspark_advance", ctx_len.device) + if op is not None and not all(t.is_contiguous() for t in (ctx_len, block_counts, base_pages)): + op = _glue_log( + "minimax_m31_dspark_advance", "non-contiguous ctx_len / block_counts / base_pages" + ) + if op is not None: + _glue_log("minimax_m31_dspark_advance") + op( + slots.to(torch.int64).contiguous(), + rows.to(torch.int64).contiguous(), + num_accepted.contiguous(), + ctx_len, + block_counts, + base_pages, + tables, + tables_read, + out, + int(max_ctx), + int(dummy_slot), + int(page_size), + int(block_size), + ) + return out width = tables.shape[1] + slots = slots.to(torch.int64).contiguous() + rows = rows.to(torch.int64).contiguous() + if parallel: + global _adv_par_logged + if not _adv_par_logged: + _adv_par_logged = True + from tensorrt_llm.logger import logger + + logger.info( + f"[M3 dispatch] ADV_PAR taken: ctx advance + parallel read-view copy (width {width}, " + f"{triton.cdiv(width, _COPY_BLOCK)} programs per request)" + ) + rebase = torch.empty(n, dtype=torch.int64, device=ctx_len.device) + _advance_only_kernel[(n,)]( + slots, + rows, + num_accepted.contiguous(), + ctx_len, + block_counts, + base_pages, + out, + rebase, + int(max_ctx), + int(dummy_slot), + int(page_size), + int(block_size), + num_warps=1, + ) + _read_view_kernel[(n, triton.cdiv(width, _COPY_BLOCK))]( + rows, + rebase, + tables, + tables_read, + width, + tables.stride(0), + tables_read.stride(0), + BLOCK=_COPY_BLOCK, + num_warps=4, + ) + return out _advance_kernel[(n,)]( - slots.to(torch.int64).contiguous(), - rows.to(torch.int64).contiguous(), + slots, + rows, num_accepted.contiguous(), ctx_len, block_counts, diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_forward_indices.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_forward_indices.py index 42448af2b144..7c793c9d2865 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_forward_indices.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_forward_indices.py @@ -26,10 +26,56 @@ # query_pos_i32 = query_positions.reshape(-1).to(int32) # The int64 -> int32 casts keep the low 32 bits, as torch's do. One program writes every output. +import functools +import os +from typing import Optional + import torch import triton import triton.language as tl +# DSPARK_GLUE_CPP: the C++ op (torch.ops.trtllm.minimax_m31_dspark_indices, kernels/minimaxM31/dsparkGlueSm107.cu, +# bit for bit this Triton kernel) on SM107 when it is built; the Triton kernel otherwise (other GPUs) or with +# TRTLLM_M31_DISABLE_DSPARK_GLUE_CPP=1 (read once, at import). +_GLUE_CPP_OFF = os.environ.get("TRTLLM_M31_DISABLE_DSPARK_GLUE_CPP", "").strip().lower() in ( + "1", + "true", + "on", + "yes", +) + + +@functools.lru_cache(maxsize=None) +def _is_sm107(device_index: int) -> bool: + return torch.cuda.get_device_capability(device_index) == (10, 7) + + +def _glue_cpp_op(name: str, device: torch.device): + """The C++ op, or None with the reason logged once (the Triton kernel runs).""" + if _GLUE_CPP_OFF: + return _glue_log(name, "killed (TRTLLM_M31_DISABLE_DSPARK_GLUE_CPP=1)") + if not _is_sm107(device.index if device.index is not None else torch.cuda.current_device()): + return _glue_log(name, "not an sm_107 device") + op = getattr(torch.ops.trtllm, name, None) + return op if op is not None else _glue_log(name, "the op is not registered") + + +_glue_logged = set() + + +def _glue_log(name: str, why: Optional[str] = None): + """Engagement line, once per (op, outcome) per process: "[M3 dispatch] DSPARK_GLUE_CPP taken: " when the C++ op + runs, "... not taken: : ; Triton kernel" when it falls back. Returns None.""" + if (name, why) not in _glue_logged: + _glue_logged.add((name, why)) + from tensorrt_llm.logger import logger + + if why is None: + logger.info(f"[M3 dispatch] DSPARK_GLUE_CPP taken: {name}") + else: + logger.info(f"[M3 dispatch] DSPARK_GLUE_CPP not taken: {name}: {why}; Triton kernel") + return None + @triton.jit def _indices_kernel( @@ -110,6 +156,14 @@ def forward_indices( last = torch.empty(B, **i32) append = torch.empty(B * block_size, **i32) qflat = torch.empty(B * block_size, **i32) + op = _glue_cpp_op("minimax_m31_dspark_indices", num_ctx.device) + if op is not None and not (num_ctx.is_contiguous() and batch_idx.is_contiguous()): + op = _glue_log("minimax_m31_dspark_indices", "non-contiguous num_ctx / batch_idx") + if op is not None: + _glue_log("minimax_m31_dspark_indices") + op(num_ctx, batch_idx, query_positions, seqlens, batch_out, after, indptr, last, append, qflat, block_size, + pages_per_slot, page_size) # fmt: skip + return seqlens, batch_out, after, indptr, last, append, qflat _indices_kernel[(1,)]( num_ctx, batch_idx, diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain.py index d32a58d50c15..a5bb6c1b8395 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain.py @@ -39,6 +39,9 @@ CERTIFIED_BATCH = ( 8 # the largest batch the cells drive (the kernel takes up to the kernel module's MAX_BATCH) ) +CERTIFIED_BATCH_TP2 = ( + 12 # the M3.1 TP2 cells' largest batch (that target serves up to 12 requests: S4CAP12) +) _lock = threading.Lock() _compiled: Dict[tuple, object] = {} @@ -149,18 +152,31 @@ def _sm_count(device) -> int: def pick_grid(shard: int, block: int, batch: int, device=None) -> int: - """CTAs for a (padded) shard: 128 when supported, else the largest supported count up to the SM count; 0 if - none.""" + """CTAs for a (padded) shard: 128 when supported with every markov_w2 row in shared memory, else the largest + such count up to the SM count; else (S4STREAM: a shard past the shared-memory capacity) the largest supported + count with streamed blocks (fewest rows per CTA); 0 if none.""" kern = _kernel_module() limit = min(MAX_GRID, _sm_count(device if device is not None else torch.cuda.current_device())) - if 128 <= limit and kern.supports(shard, 128, block, batch): + if 128 <= limit and kern.supports(shard, 128, block, batch) and kern.fits(shard, 128): return 128 + for grid in range(limit, 0, -1): + if kern.supports(shard, grid, block, batch) and kern.fits(shard, grid): + return grid for grid in range(limit, 0, -1): if kern.supports(shard, grid, block, batch): return grid return 0 +def _fits_all(rows: int, max_batch: int, device) -> bool: + kern = _kernel_module() + for b in (1, max_batch): + grid = pick_grid(rows, MAX_BLOCK, b, device) + if grid == 0 or not kern.fits(rows, grid): + return False + return True + + def padded_rows(shard: int, max_batch: int, device=None) -> int: """The smallest row count >= ``shard`` that the kernel splits for every batch <= ``max_batch`` and block <= MAX_BLOCK; 0 if none fits the shared-memory budget.""" @@ -169,10 +185,23 @@ def padded_rows(shard: int, max_batch: int, device=None) -> int: rows = -(-shard // step) * step limit = rows + MAX_GRID * step while rows <= limit: - if all(pick_grid(rows, MAX_BLOCK, b, device) > 0 for b in (1, max_batch)): + if _fits_all(rows, max_batch, device): return rows rows += step - return 0 + # S4STREAM: no padding keeps every row in shared memory: the padding with the fewest rows per CTA (then the + # fewest rows) among the even CTA counts the kernel takes, streaming the blocks past the budget. + sm = _sm_count(device if device is not None else torch.cuda.current_device()) + best = None + for grid in range(min(MAX_GRID, sm), 1, -1): + rc = -(-shard // (grid * step)) * step + cand = grid * rc + grids = [pick_grid(cand, MAX_BLOCK, b, device) for b in (1, max_batch)] + if 0 in grids: + continue + key = (max(kern.rows_per_cta(cand, g) for g in grids), cand) + if best is None or key < best: + best = key + return best[1] if best is not None else 0 def pad_w2_shard(w2_shard: torch.Tensor, rows: int) -> torch.Tensor: @@ -522,11 +551,12 @@ class DsparkMarkovChain(OpWrapper): (B, K), in the same order. """ - ARCHS = frozenset({Arch.SM_103}) + ARCHS = frozenset({Arch.SM_103, Arch.SM_107}) # MiniMax-M3.1's DSpark drafter: vocabulary 200064, Markov rank 256, K = 3 and 7 draft positions, the decode # batches the M3.1 targets serve with it (1..8). TP4 is one GB300 tray, TP8 two. _M31 = dict(vocab=200064, batches=tuple(range(1, CERTIFIED_BATCH + 1))) + _M31_TP2 = dict(vocab=200064, batches=tuple(range(1, CERTIFIED_BATCH_TP2 + 1))) CELLS: tuple[Cell, ...] = ( Cell( @@ -546,6 +576,12 @@ class DsparkMarkovChain(OpWrapper): spec=dict(world=8, block=3, **_M31), ), Cell(why="M3.1 TP8 over two trays, K = 7", spec=dict(world=8, block=7, **_M31)), + Cell( + why="M3.1 TP2, K = 7: the 100032-row shard, padded to 101376 = 144 CTAs x 704 rows; S4STREAM" + " (5 of 11 blocks per warp read from global memory)", + spec=dict(world=2, block=7, **_M31_TP2), + ), + Cell(why="M3.1 TP2, K = 3 (S4STREAM)", spec=dict(world=2, block=3, **_M31_TP2)), ) note = """ @@ -553,8 +589,8 @@ class DsparkMarkovChain(OpWrapper): any of those runs the in-tree chain. The kernel takes batches up to its module's MAX_BATCH (16) and blocks up to MAX_BLOCK (8); the cells certify - batches 1..8. Measured: at 16 requests the one-launch chain is 0.93x the in-tree chain's time, so the M3.1 - targets serve batches up to 8 only. + batches 1..8 (1..12 at TP2). Measured: at 16 requests the one-launch chain is 0.93x the in-tree chain's time, so + the M3.1 targets serve batches up to 8 only, except TP2 up to 12 (S4CAP12: it still wins at 12 on VR200). The kernel also computes next_new_tokens, the accepted-row bookkeeping and the target's KV-length rewind of the K3 fused verify; this entry passes none of them (one dummy word each) and returns only the tokens. diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel.py index a6bdf3861a22..33b3382d2457 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel.py @@ -82,6 +82,11 @@ NEG_INF_BITS = -8388608 # 0xFF800000 COPY_ROWS = 32 # rows per weight bulk copy (16 KB) SMEM_ROW_BUDGET = 400 # 200 KB of markov_w2 rows per CTA +# S4STREAM: a CTA keeps at most SMEM_BLOCKS 8-row blocks per warp of its markov_w2 rows in shared memory; a CTA with +# more rows (a shard past MAX_GRID x the budget, e.g. the M3.1 TP2 shard) reads the warp's remaining blocks from +# global memory with the same dot (same words, FMA order and xor tree: the same sums). MAX_ROWS_PER_CTA bounds that. +SMEM_BLOCKS = SMEM_ROW_BUDGET // (8 * 8) # 6 blocks per warp = 384 rows per CTA +MAX_ROWS_PER_CTA = 1024 SMEM_BYTES = 227 * 1024 @@ -90,14 +95,15 @@ def rows_per_cta(shard: int, grid: int) -> int: def supports(shard: int, grid: int, block: int, batch: int) -> bool: - """Shapes the kernel runs: whole 8-row blocks for every warp (RC a multiple of 64), the CTA's markov_w2 rows - within the shared-memory budget, block <= 16, batch <= 16, an even entry count per exchange.""" + """Shapes the kernel runs: whole 8-row blocks for every warp (RC a multiple of 64), at most MAX_ROWS_PER_CTA + rows per CTA (the part past the shared-memory budget streamed, see fits()), block <= 16, batch <= 16, an even + entry count per exchange.""" if grid <= 0 or shard % grid != 0: return False rc = rows_per_cta(shard, grid) return ( rc % (WARPS * BLOCK_ROWS) == 0 - and rc <= SMEM_ROW_BUDGET + and rc <= MAX_ROWS_PER_CTA and smem_bytes(rc, batch) <= SMEM_BYTES and grid % 2 == 0 and 0 < block <= MAX_BLOCK @@ -105,10 +111,20 @@ def supports(shard: int, grid: int, block: int, batch: int) -> bool: ) +def fits(shard: int, grid: int) -> bool: + """Whether every markov_w2 row of the CTA stays in shared memory (no streamed blocks).""" + return grid > 0 and shard % grid == 0 and rows_per_cta(shard, grid) <= SMEM_ROW_BUDGET + + +def smem_rows(rows: int) -> int: + """The CTA's markov_w2 rows held in shared memory (S4STREAM: SMEM_BLOCKS blocks per warp at most).""" + return min(rows, WARPS * SMEM_BLOCKS * BLOCK_ROWS) if rows > SMEM_ROW_BUDGET else rows + + def smem_bytes(rows: int, batch: int) -> int: - """Shared memory of a CTA: its markov_w2 rows, the markov_w1[prev] rows, the two [b][warp] reductions, the - barriers, flags and prev (with the allocator's alignment padding).""" - return rows * ROW_BYTES + batch * ROW_BYTES + 2 * batch * WARPS * 8 + 1024 + """Shared memory of a CTA: its markov_w2 rows held in shared memory, the markov_w1[prev] rows, the two [b][warp] + reductions, the barriers, flags and prev (with the allocator's alignment padding).""" + return smem_rows(rows) * ROW_BYTES + batch * ROW_BYTES + 2 * batch * WARPS * 8 + 1024 def buffer_words(block: int, batch: int, slots: int, grid: int) -> int: @@ -300,6 +316,9 @@ def k3_markov_kernel( ): rows_per_warp = rows // WARPS blocks_per_warp = rows_per_warp // BLOCK_ROWS + # S4STREAM: the warp's first smem_blocks blocks are in shared memory, the rest are read from global memory. + smem_blocks = smem_rows(rows) // (WARPS * BLOCK_ROWS) + streamed = smem_blocks < blocks_per_warp entries = slots * grid # (value, index) entries per (k, b) vectors = entries // 2 # 16-byte loads per (k, b) vec_per_thread = (vectors + THREADS - 1) // THREADS @@ -311,7 +330,7 @@ def k3_markov_kernel( lane = tx % cutlass.Int32(32) s_w2 = cutlass.Array( - cutlass.Int32, rows * ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128 + cutlass.Int32, smem_rows(rows) * ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128 ) # [b]: markov_w1[prev] of the position (one bulk copy per request, issued by the thread that learns prev). s_w1 = cutlass.Array( @@ -339,18 +358,36 @@ def k3_markov_kernel( # The CTA's markov_w2 rows: a weight, loaded before the grid dependency. if warp == 0: if prims.elect_sync(): - prims.mbarrier_arrive_expect_tx(s_mbar, rows * ROW_BYTES) - for c in cutlass.range_constexpr((rows + COPY_ROWS - 1) // COPY_ROWS): - n = min(COPY_ROWS, rows - c * COPY_ROWS) - prims.cp_async_bulk_shared_cluster_global( - s_w2.subview(c * COPY_ROWS * ROW_WORDS), - w2.subview( - cta * cutlass.Int32(rows * ROW_WORDS) - + cutlass.Int32(c * COPY_ROWS * ROW_WORDS) - ), - s_mbar, - n * ROW_BYTES, - ) + if cutlass.const_expr(not streamed): + prims.mbarrier_arrive_expect_tx(s_mbar, rows * ROW_BYTES) + for c in cutlass.range_constexpr((rows + COPY_ROWS - 1) // COPY_ROWS): + n = min(COPY_ROWS, rows - c * COPY_ROWS) + prims.cp_async_bulk_shared_cluster_global( + s_w2.subview(c * COPY_ROWS * ROW_WORDS), + w2.subview( + cta * cutlass.Int32(rows * ROW_WORDS) + + cutlass.Int32(c * COPY_ROWS * ROW_WORDS) + ), + s_mbar, + n * ROW_BYTES, + ) + else: + # S4STREAM: each warp's first smem_blocks blocks (its rows [w RPW, w RPW + smem_blocks 8)) land at + # [w smem_blocks 8, ...) of shared memory. + per_warp = smem_blocks * BLOCK_ROWS + prims.mbarrier_arrive_expect_tx(s_mbar, WARPS * per_warp * ROW_BYTES) + for w in cutlass.range_constexpr(WARPS): + for c in cutlass.range_constexpr((per_warp + COPY_ROWS - 1) // COPY_ROWS): + n = min(COPY_ROWS, per_warp - c * COPY_ROWS) + prims.cp_async_bulk_shared_cluster_global( + s_w2.subview((w * per_warp + c * COPY_ROWS) * ROW_WORDS), + w2.subview( + cta * cutlass.Int32(rows * ROW_WORDS) + + cutlass.Int32((w * rows_per_warp + c * COPY_ROWS) * ROW_WORDS) + ), + s_mbar, + n * ROW_BYTES, + ) # Dependents wait for this whole grid before reading its outputs. prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) @@ -437,7 +474,18 @@ def k3_markov_kernel( best_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) best_index = cutlass.Int32(INT_MAX) for g in cutlass.range_constexpr(blocks_per_warp): - dot = _block_dot(s_w2, row0 + cutlass.Int32(g * BLOCK_ROWS), lane, w1f) + if cutlass.const_expr(not streamed): + dot = _block_dot(s_w2, row0 + cutlass.Int32(g * BLOCK_ROWS), lane, w1f) + elif cutlass.const_expr(g < smem_blocks): + dot = _block_dot( + s_w2, + warp * cutlass.Int32(smem_blocks * BLOCK_ROWS) + + cutlass.Int32(g * BLOCK_ROWS), + lane, + w1f, + ) + else: # S4STREAM: a streamed block, its rows read from global memory by the same dot + dot = _block_dot(w2, shard_row0 + cutlass.Int32(g * BLOCK_ROWS), lane, w1f) bias = _lo_f32(_pack_bf16x2(dot, dot)) # fp32_sum (default): the in-tree chain's arithmetic. Its logits are fp32 (LogitsProcessor's # .float()) and the bf16 bias is widened, so the argmax compares the unrounded fp32 sum. Without diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1.py index 97c83e4c01be..b6f4ce351aa5 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_markov_chain_kernel_b1.py @@ -87,6 +87,11 @@ NEG_INF_BITS = -8388608 # 0xFF800000 COPY_ROWS = 32 # rows per weight bulk copy (16 KB) SMEM_ROW_BUDGET = 400 # 200 KB of markov_w2 rows per CTA +# S4STREAM: a CTA keeps at most SMEM_BLOCKS 8-row blocks per warp of its markov_w2 rows in shared memory; a CTA with +# more rows (a shard past MAX_GRID x the budget, e.g. the M3.1 TP2 shard) reads the warp's remaining blocks from +# global memory with the same dot (same words, FMA order and xor tree: the same sums). MAX_ROWS_PER_CTA bounds that. +SMEM_BLOCKS = SMEM_ROW_BUDGET // (8 * 8) # 6 blocks per warp = 384 rows per CTA +MAX_ROWS_PER_CTA = 1024 SMEM_BYTES = 227 * 1024 @@ -95,14 +100,15 @@ def rows_per_cta(shard: int, grid: int) -> int: def supports(shard: int, grid: int, block: int, batch: int) -> bool: - """Shapes the kernel runs: whole 8-row blocks for every warp (RC a multiple of 64), the CTA's markov_w2 rows - within the shared-memory budget, block <= 16, batch <= 16, an even entry count per exchange.""" + """Shapes the kernel runs: whole 8-row blocks for every warp (RC a multiple of 64), at most MAX_ROWS_PER_CTA + rows per CTA (the part past the shared-memory budget streamed, see fits()), block <= 16, batch <= 16, an even + entry count per exchange.""" if grid <= 0 or shard % grid != 0: return False rc = rows_per_cta(shard, grid) return ( rc % (WARPS * BLOCK_ROWS) == 0 - and rc <= SMEM_ROW_BUDGET + and rc <= MAX_ROWS_PER_CTA and smem_bytes(rc, batch) <= SMEM_BYTES and grid % 2 == 0 and 0 < block <= MAX_BLOCK @@ -110,10 +116,20 @@ def supports(shard: int, grid: int, block: int, batch: int) -> bool: ) +def fits(shard: int, grid: int) -> bool: + """Whether every markov_w2 row of the CTA stays in shared memory (no streamed blocks).""" + return grid > 0 and shard % grid == 0 and rows_per_cta(shard, grid) <= SMEM_ROW_BUDGET + + +def smem_rows(rows: int) -> int: + """The CTA's markov_w2 rows held in shared memory (S4STREAM: SMEM_BLOCKS blocks per warp at most).""" + return min(rows, WARPS * SMEM_BLOCKS * BLOCK_ROWS) if rows > SMEM_ROW_BUDGET else rows + + def smem_bytes(rows: int, batch: int) -> int: - """Shared memory of a CTA: its markov_w2 rows, the markov_w1[prev] rows, the two [b][warp] reductions, the - barriers, flags and prev (with the allocator's alignment padding).""" - return rows * ROW_BYTES + batch * ROW_BYTES + 2 * batch * WARPS * 8 + 1024 + """Shared memory of a CTA: its markov_w2 rows held in shared memory, the markov_w1[prev] rows, the two [b][warp] + reductions, the barriers, flags and prev (with the allocator's alignment padding).""" + return smem_rows(rows) * ROW_BYTES + batch * ROW_BYTES + 2 * batch * WARPS * 8 + 1024 def buffer_words(block: int, batch: int, slots: int, grid: int) -> int: @@ -305,6 +321,9 @@ def k3_markov_kernel( ): rows_per_warp = rows // WARPS blocks_per_warp = rows_per_warp // BLOCK_ROWS + # S4STREAM: the warp's first smem_blocks blocks are in shared memory, the rest are read from global memory. + smem_blocks = smem_rows(rows) // (WARPS * BLOCK_ROWS) + streamed = smem_blocks < blocks_per_warp entries = slots * grid # (value, index) entries per (k, b) vectors = entries // 2 # 16-byte loads per (k, b) vec_per_thread = (vectors + THREADS - 1) // THREADS @@ -316,7 +335,7 @@ def k3_markov_kernel( lane = tx % cutlass.Int32(32) s_w2 = cutlass.Array( - cutlass.Int32, rows * ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128 + cutlass.Int32, smem_rows(rows) * ROW_WORDS, space=cutlass.AddressSpace.smem, alignment=128 ) # [b]: markov_w1[prev] of the position (one bulk copy per request, issued by the thread that learns prev). s_w1 = cutlass.Array( @@ -344,18 +363,36 @@ def k3_markov_kernel( # The CTA's markov_w2 rows: a weight, loaded before the grid dependency. if warp == 0: if prims.elect_sync(): - prims.mbarrier_arrive_expect_tx(s_mbar, rows * ROW_BYTES) - for c in cutlass.range_constexpr((rows + COPY_ROWS - 1) // COPY_ROWS): - n = min(COPY_ROWS, rows - c * COPY_ROWS) - prims.cp_async_bulk_shared_cluster_global( - s_w2.subview(c * COPY_ROWS * ROW_WORDS), - w2.subview( - cta * cutlass.Int32(rows * ROW_WORDS) - + cutlass.Int32(c * COPY_ROWS * ROW_WORDS) - ), - s_mbar, - n * ROW_BYTES, - ) + if cutlass.const_expr(not streamed): + prims.mbarrier_arrive_expect_tx(s_mbar, rows * ROW_BYTES) + for c in cutlass.range_constexpr((rows + COPY_ROWS - 1) // COPY_ROWS): + n = min(COPY_ROWS, rows - c * COPY_ROWS) + prims.cp_async_bulk_shared_cluster_global( + s_w2.subview(c * COPY_ROWS * ROW_WORDS), + w2.subview( + cta * cutlass.Int32(rows * ROW_WORDS) + + cutlass.Int32(c * COPY_ROWS * ROW_WORDS) + ), + s_mbar, + n * ROW_BYTES, + ) + else: + # S4STREAM: each warp's first smem_blocks blocks (its rows [w RPW, w RPW + smem_blocks 8)) land at + # [w smem_blocks 8, ...) of shared memory. + per_warp = smem_blocks * BLOCK_ROWS + prims.mbarrier_arrive_expect_tx(s_mbar, WARPS * per_warp * ROW_BYTES) + for w in cutlass.range_constexpr(WARPS): + for c in cutlass.range_constexpr((per_warp + COPY_ROWS - 1) // COPY_ROWS): + n = min(COPY_ROWS, per_warp - c * COPY_ROWS) + prims.cp_async_bulk_shared_cluster_global( + s_w2.subview((w * per_warp + c * COPY_ROWS) * ROW_WORDS), + w2.subview( + cta * cutlass.Int32(rows * ROW_WORDS) + + cutlass.Int32((w * rows_per_warp + c * COPY_ROWS) * ROW_WORDS) + ), + s_mbar, + n * ROW_BYTES, + ) # Dependents wait for this whole grid before reading its outputs. prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) @@ -447,7 +484,18 @@ def k3_markov_kernel( best_value = cutlass.Int32(NEG_INF_BITS).bitcast(cutlass.Float32) best_index = cutlass.Int32(INT_MAX) for g in cutlass.range_constexpr(blocks_per_warp): - dot = _block_dot(s_w2, row0 + cutlass.Int32(g * BLOCK_ROWS), lane, w1f) + if cutlass.const_expr(not streamed): + dot = _block_dot(s_w2, row0 + cutlass.Int32(g * BLOCK_ROWS), lane, w1f) + elif cutlass.const_expr(g < smem_blocks): + dot = _block_dot( + s_w2, + warp * cutlass.Int32(smem_blocks * BLOCK_ROWS) + + cutlass.Int32(g * BLOCK_ROWS), + lane, + w1f, + ) + else: # S4STREAM: a streamed block, its rows read from global memory by the same dot + dot = _block_dot(w2, shard_row0 + cutlass.Int32(g * BLOCK_ROWS), lane, w1f) bias = _lo_f32(_pack_bf16x2(dot, dot)) # fp32_sum (default): the in-tree chain's arithmetic (fp32 logits from LogitsProcessor's .float() # plus the widened bf16 bias): compare the unrounded fp32 sum. Without fp32_sum: a bf16 reference diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_qk_norm_rope.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_qk_norm_rope.py index 08b6630400ab..3dea1c336594 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_qk_norm_rope.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_qk_norm_rope.py @@ -136,9 +136,14 @@ def _qk_norm_rope_kernel( i = tl.arange(0, HALF) cos = tl.load(cs_ptr + pos * cs_stride + i) sin = tl.load(cs_ptr + pos * cs_stride + HALF + i) - yf = y.to(tl.float32) - x1 = tl.sum(tl.where(d[None, :] == i[:, None], yf[None, :], 0.0), axis=1) - x2r = tl.sum(tl.where(d[None, :] == (i[:, None] + HALF), yf[None, :], 0.0), axis=1) + # The rotated pair y[i], y[i + HALF], recomputed elementwise from memory with y's own ops (same bits, sign of zero + # included; a masked tl.sum over y would add +0.0 terms and turn a -0 element into +0). + wa = tl.where(is_q, tl.load(wq_ptr + i), tl.load(wk_ptr + i)).to(tl.float32) + wb = tl.where(is_q, tl.load(wq_ptr + HALF + i), tl.load(wk_ptr + HALF + i)).to(tl.float32) + x1 = _mul(_mul(tl.load(base + i).to(tl.float32), rstd), wa).to(tl.bfloat16).to(tl.float32) + x2r = ( + _mul(_mul(tl.load(base + HALF + i).to(tl.float32), rstd), wb).to(tl.bfloat16).to(tl.float32) + ) o1 = _fma(x1, cos, _mul(-x2r, sin)) o2 = _fma(x2r, cos, _mul(x1, sin)) obase = tl.where(is_q, q_out + t * (HQ * HD) + h * HD, k_out + t * (HKV * HD) + (h - HQ) * HD) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept.py index eda460592842..9486a08a84ce 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/strict_accept.py @@ -319,7 +319,7 @@ class SpecStrictAccept(OpWrapper): the prefix of drafts equal to the target's tokens. No forced-acceptance override (the caller's). """ - ARCHS = frozenset({Arch.SM_103}) + ARCHS = frozenset({Arch.SM_103, Arch.SM_107}) CELLS: tuple[Cell, ...] = ( Cell( @@ -383,7 +383,7 @@ class SpecNextNewTokens(OpWrapper): ``accepted`` and ``next_draft`` (as ``torch.concat``). """ - ARCHS = frozenset({Arch.SM_103}) + ARCHS = frozenset({Arch.SM_103, Arch.SM_107}) CELLS: tuple[Cell, ...] = ( Cell( diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/fake_ops.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/fake_ops.py index 106f61daffe6..0177364f6c92 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/fake_ops.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/fake_ops.py @@ -110,6 +110,90 @@ def _minimax_m3_mc_allreduce_norm_fake( } +# The MiniMax-M3 sm_107 TP2 / TP4 / TP8 decode ops: one producer and one all-reduce + norm op per TP width (Q and +# index-Q heads per rank from the kernels' producerPw{2,4,8}.h), and the dense QK-norm + RoPE ops. +def _minimax_m3_sm107_producer_pw_fake(heads_q: int, heads_index: int): + def fake( + packed, + kv_data, + kv_scale, + index_k, + out_cache_loc, + kv_quant_scale, + eps, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + rotary_cos_sin, + position_ids, + pdl, + early, + prewait_step_inputs=False, + ): + # [Q, index Q] in FP8; the K / V / index-K rows go into the caches in place. + num_tokens = packed.shape[0] + return [ + packed.new_empty((num_tokens, heads_q, 128), dtype=torch.float8_e4m3fn), + packed.new_empty((num_tokens, heads_index, 128), dtype=torch.float8_e4m3fn), + ] + + return fake + + +def _minimax_m3_sm107_mc_allreduce_norm_fake( + x, + x2, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + perm, + scales, + maxt, + quant, + prepushed, + opts, + cap_out=None, + cap_w=None, + cap_eps=0.0, +): + # [normed, updated residual] (+ [E4M3 rows, swizzled scale bytes] with quant); cap_out is written in place. + outputs = [torch.empty_like(residual), torch.empty_like(residual)] + if quant: + m, h = residual.shape + outputs += [ + residual.new_empty((m, h), dtype=torch.float8_e4m3fn), + residual.new_empty(((m + 127) // 128 * 128 * (h // 32),), dtype=torch.uint8), + ] + return outputs + + +def _minimax_m3_sm107_dense_qk_norm_rope_fp8_fake(qkv, *args, **kwargs): + # A new FP8 tensor shaped like qkv; the KV variant also stores K / V (and the compact Q) in place. + return torch.empty_like(qkv, dtype=torch.float8_e4m3fn) + + +for _tp, _heads_q, _heads_index in ((2, 32, 2), (4, 16, 1), (8, 8, 1)): + FAKES[f"minimax_m3_sm107_tp{_tp}_producer_pw"] = _minimax_m3_sm107_producer_pw_fake( + _heads_q, _heads_index + ) + FAKES[f"minimax_m3_sm107_tp{_tp}_mc_allreduce_norm"] = _minimax_m3_sm107_mc_allreduce_norm_fake +for _name in ( + "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8", + "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv", + "minimax_m3_sm107tp4_dense_qk_norm_rope_fp8", + "minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv", +): + FAKES[_name] = _minimax_m3_sm107_dense_qk_norm_rope_fp8_fake + + def register(): """Register available native ops, preserving existing Meta kernels or Python fakes.""" for name, fake in FAKES.items(): diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_pins.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_pins.py new file mode 100644 index 000000000000..cbb7f958afae --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_pins.py @@ -0,0 +1,322 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tactic pins shipped with the MiniMax-M3.1 sm_107 targets. + +Each target names its topology (``tp2``, ``tp4``, ``tp8``, ...). Before the warmup tuning pass, ``install`` loads +``m31_tuning/sm107//rank_.json`` (format ``m31_pin_v1``) from this package: + +* ``autotuner``: the TensorRT-LLM AutoTuner entries, each keyed by op, runner, runner id and tuning shapes and + carrying its tactic by NAME (``TunableRunner.tactic_name``). On the first lookup of an entry the AutoTuner checks + that name against the runner: a tactic of that name is served, otherwise the entry is dropped with an + ``[M31_PIN] unresolved`` line and tuned again. Every shape the warmup tunes that the bundle does not cover logs + ``[M31_PIN] retune``. Both are counted per rank and reported at exit. An entry of a runner without tactic names + (``tactic_name`` null) is index-keyed: it is served only when the bundle's native guard (the md5s of the native + libraries it was tuned with) matches the installed ones; otherwise every such entry is dropped (one + ``[M31_PIN] unresolved native guard`` line) and its shapes are tuned again. +* ``flashinfer``: FlashInfer's own per-rank autotune cache (it keys its configs by index, so it cannot be keyed by + name here). It is served, load only, when its guard (the FlashInfer and CuTe DSL versions it was tuned with) + matches the installed ones; otherwise one ``[M31_PIN] unresolved flashinfer guard`` line and FlashInfer keeps its + own selection. + +``TLLM_AUTOTUNER_CACHE_PATH`` (an explicit AutoTuner cache) takes precedence over the bundle; the path +``tensorrt_llm`` derives from ``TRTLLM_CACHE_DIR`` is not explicit, so the bundle is installed and the AutoTuner then +loads and saves that cache file over it. ``TRTLLM_M31_PIN_DIR`` replaces the bundled directory. +``TRTLLM_M31_DISABLE_BUNDLED_PINS=1`` loads nothing. + +``python -m tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.m31_pins`` converts a tuning run's +standard files (an AutoTuner cache file and FlashInfer's per-rank files) into the bundle format; it needs the native +runners, so it runs on the target GPU with the wheel installed. +""" + +import argparse +import ast +import atexit +import json +import os +import tempfile +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +FORMAT = "m31_pin_v1" +DISABLE_ENV = "TRTLLM_M31_DISABLE_BUNDLED_PINS" +DIR_ENV = "TRTLLM_M31_PIN_DIR" +AUTOTUNER_CACHE_ENV = "TLLM_AUTOTUNER_CACHE_PATH" +UNIFIED_CACHE_ENV = "TRTLLM_CACHE_DIR" +BUNDLE_ROOT = Path(__file__).resolve().parent / "m31_tuning" / "sm107" + + +def _disabled() -> bool: + return os.environ.get(DISABLE_ENV, "0").strip().lower() in ("1", "true", "on", "yes") + + +def _explicit_autotuner_cache() -> Optional[str]: + """The AutoTuner cache path set for this process, or None when there is none or it is the unified cache's + default (``tensorrt_llm`` sets it at import from ``TRTLLM_CACHE_DIR``).""" + path = os.environ.get(AUTOTUNER_CACHE_ENV) + root = os.environ.get(UNIFIED_CACHE_ENV) + if path and root: + from tensorrt_llm._bootstrap import _unified_cache_defaults + + if path == _unified_cache_defaults(root)[AUTOTUNER_CACHE_ENV]: + return None + return path or None + + +def pin_file(topology: str, rank: int) -> Path: + """This rank's bundle file of ``topology`` (it may not exist).""" + root = Path(os.environ[DIR_ENV]) if os.environ.get(DIR_ENV) else BUNDLE_ROOT + return root / topology / f"rank_{rank}.json" + + +def _info(message: str) -> None: + from tensorrt_llm.logger import logger + + logger.info(message) + + +class PinReporter: + """Counts and logs the bundled entries no tactic matches and the shapes the bundle does not cover.""" + + def __init__(self, topology: str, rank: int): + self.topology = topology + self.rank = rank + self.unresolved_count = 0 + self.retune_count = 0 + self._final_logged = False + + def unresolved(self, cache_key: Tuple, name: str) -> None: + self.unresolved_count += 1 + _info( + f"[M31_PIN] unresolved rank={self.rank} op={cache_key[0]} runner={cache_key[1]} " + f"shapes={cache_key[3]} tactic={name!r}" + ) + + def unresolved_guard(self, what: str, want: Dict[str, Any], have: Dict[str, Any]) -> None: + self.unresolved_count += 1 + _info( + f"[M31_PIN] unresolved {what} guard rank={self.rank}: bundle {want}, installed {have}" + ) + + def retune(self, custom_op: str, shapes: Any) -> None: + self.retune_count += 1 + _info(f"[M31_PIN] retune rank={self.rank} op={custom_op} shapes={shapes}") + + def final(self) -> None: + if self._final_logged: + return + self._final_logged = True + _info( + f"[M31_PIN] final rank={self.rank} topology={self.topology}: unresolved {self.unresolved_count} " + f"retune {self.retune_count}" + ) + + +def _key(entry: Dict[str, Any]) -> Tuple: + """The AutoTuner cache key of a bundle entry: (op, runner class, runner id, shapes as nested tuples).""" + shapes = tuple(tuple(int(d) for d in dims) for dims in entry["shapes"]) + return (entry["op"], entry["runner"], entry["runner_key"], shapes) + + +def _flashinfer_versions() -> Dict[str, Optional[str]]: + import importlib.metadata as md + + versions: Dict[str, Optional[str]] = {} + for dist in ("flashinfer-python", "nvidia-cutlass-dsl"): + try: + versions[dist] = md.version(dist) + except md.PackageNotFoundError: + versions[dist] = None + return versions + + +_NATIVE_LIBS = ("libtensorrt_llm.so", "libth_common.so") +_native_md5s: Dict[str, Optional[str]] = {} + + +def _native_guard() -> Dict[str, Optional[str]]: + """The md5s of the native libraries whose tactic lists index-keyed entries refer to (computed once).""" + if not _native_md5s: + import hashlib + + import tensorrt_llm + + libs = Path(tensorrt_llm.__file__).resolve().parent / "libs" + for name in _NATIVE_LIBS: + path = libs / name + if path.is_file(): + digest = hashlib.md5() + with open(path, "rb") as f: + for block in iter(lambda: f.read(1 << 24), b""): + digest.update(block) + _native_md5s[name] = digest.hexdigest() + else: + _native_md5s[name] = None + return dict(_native_md5s) + + +def install(topology: str, rank: int, tuner: Optional[Any] = None) -> Optional[PinReporter]: + """Load this rank's bundled pins into the AutoTuner (and FlashInfer's autotune cache). Returns the reporter, or + None when nothing was loaded (switched off, an explicit AutoTuner cache, or no bundle file).""" + if _disabled(): + _info(f"[M31_PIN] bundled pins off ({DISABLE_ENV}=1) rank={rank}") + return None + if _explicit_autotuner_cache(): + _info(f"[M31_PIN] bundled pins not loaded rank={rank}: {AUTOTUNER_CACHE_ENV} is set") + return None + path = pin_file(topology, rank) + if not path.is_file(): + _info(f"[M31_PIN] no bundle rank={rank} topology={topology} ({path})") + return None + data = json.loads(path.read_text()) + if data.get("format") != FORMAT: + raise ValueError(f"{path}: format {data.get('format')!r}, expected {FORMAT!r}") + + if tuner is None: + from tensorrt_llm._torch.autotuner import AutoTuner + + tuner = AutoTuner.get() + reporter = PinReporter(topology, rank) + cache = tuner.profiling_cache + entries = data.get("autotuner", []) + unnamed = [e for e in entries if e.get("tactic_name") is None] + if unnamed: + have = _native_guard() + if data.get("native_guard") != have: + reporter.unresolved_guard( + f"native ({len(unnamed)} index-keyed entries dropped)", + data.get("native_guard"), + have, + ) + entries = [e for e in entries if e.get("tactic_name") is not None] + for entry in entries: + key = _key(entry) + cache.cache[key] = ( + int(entry.get("runner_id", 0)), + entry["tactic"], + float(entry.get("min_time", 0.0)), + ) + if entry.get("tactic_name") is not None: + cache.bundled_names[key] = entry["tactic_name"] + tuner.bundle_reporter = reporter + + flashinfer = data.get("flashinfer") + fi_state = "none" + if flashinfer: + have = _flashinfer_versions() + if flashinfer.get("guard") == have: + from tensorrt_llm._torch import flashinfer_autotune_cache + + fi_dir = Path(tempfile.mkdtemp(prefix=f"m31_pins_rank{rank}_")) + fi_path = fi_dir / f"rank{rank}.json" + fi_path.write_text(json.dumps(flashinfer["cache"])) + flashinfer_autotune_cache.use_bundled_file(str(fi_path)) + fi_state = f"{len(flashinfer['cache']) - ('_metadata' in flashinfer['cache'])} shapes" + else: + reporter.unresolved_guard("flashinfer", flashinfer.get("guard"), have) + fi_state = "guard mismatch" + atexit.register(reporter.final) + _info( + f"[M31_PIN] loaded rank={rank} topology={topology}: {len(entries)} AutoTuner entries, " + f"FlashInfer {fi_state} ({path.name})" + ) + return reporter + + +# --------------------------------------------------------------------------------------------------------------- +# Converter: a tuning run's standard files -> the bundle format (run on the target GPU with the wheel installed) +# --------------------------------------------------------------------------------------------------------------- + + +def _mxfp8_name(runner_key: str, tactic: Any) -> str: + import torch + + if tactic == -1: + return "fallback" + dtype_name = runner_key.strip("()").split(",")[0].strip() + dtype = getattr(torch, dtype_name.replace("torch.", "")) + return torch.classes.trtllm.MXFP8GemmRunner(dtype).get_config_name(int(tactic)) + + +def _fp4_moe_name(runner_key: str, tactic: Any) -> str: + import torch + + tile_n, config = (int(v) for v in tactic) + if tile_n == -1 or config == -1: + return "fallback" + act_type = int(runner_key.strip("()").split(",")[-1]) + return torch.classes.trtllm.FP4BlockScaleMoERunner(act_type).get_config_name(tile_n, config) + + +TACTIC_NAMERS = {"MXFP8GemmRunner": _mxfp8_name, "FP4BlockScaleMoERunner": _fp4_moe_name} + + +def convert_autotuner(cache_file: Path, rank: int) -> List[Dict[str, Any]]: + """One rank's entries of an AutoTuner cache file (shared + rank_), each with its tactic's name, or a null + name for a runner without one (index-keyed; ``install`` serves it only under the bundle's native guard).""" + import torch # noqa: F401 (loads the native runners) + + import tensorrt_llm # noqa: F401 + + data = json.loads(Path(cache_file).read_text()) + entries = [] + for section in ("shared", f"rank_{rank}"): + for key_str, value in data.get(section, {}).items(): + op, runner, runner_key, shapes = ast.literal_eval(key_str) + tactic = ast.literal_eval(value["tactic"]) + namer = TACTIC_NAMERS.get(runner) + entries.append( + { + "op": op, + "runner": runner, + "runner_key": runner_key, + "shapes": [list(dims) for dims in shapes], + "runner_id": int(value.get("runner_id", 0)), + "tactic": tactic, + "tactic_name": namer(runner_key, tactic) if namer is not None else None, + "min_time": float(value.get("min_time", 0.0)), + } + ) + return entries + + +def convert( + cache_file: Optional[Path], fi_dir: Optional[Path], out_dir: Path, ranks: int +) -> List[Path]: + """Write ``out_dir/rank_.json`` for ranks 0..ranks-1 from an AutoTuner cache file and FlashInfer rank files.""" + out_dir.mkdir(parents=True, exist_ok=True) + guard = _flashinfer_versions() + written = [] + for rank in range(ranks): + bundle: Dict[str, Any] = {"format": FORMAT, "autotuner": [], "flashinfer": None} + if cache_file is not None: + bundle["autotuner"] = convert_autotuner(cache_file, rank) + if any(e["tactic_name"] is None for e in bundle["autotuner"]): + bundle["native_guard"] = _native_guard() + if fi_dir is not None: + fi_file = Path(fi_dir) / f"rank{rank}.json" + if not fi_file.is_file(): + raise FileNotFoundError(f"{fi_file}: every rank needs its FlashInfer file") + bundle["flashinfer"] = {"guard": guard, "cache": json.loads(fi_file.read_text())} + path = out_dir / f"rank_{rank}.json" + path.write_text(json.dumps(bundle, indent=1, sort_keys=True) + "\n") + written.append(path) + return written + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument( + "--autotuner-cache", type=Path, help="TLLM_AUTOTUNER_CACHE_PATH file of the tuning run" + ) + parser.add_argument( + "--flashinfer-dir", type=Path, help="TLLM_FLASHINFER_AUTOTUNE_CACHE_DIR of the tuning run" + ) + parser.add_argument("--ranks", type=int, required=True) + parser.add_argument("--out", type=Path, required=True, help="m31_tuning/sm107/") + args = parser.parse_args() + for path in convert(args.autotuner_cache, args.flashinfer_dir, args.out, args.ranks): + print(path) + + +if __name__ == "__main__": + main() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_tuning/sm107/tp2/rank_0.json b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_tuning/sm107/tp2/rank_0.json new file mode 100644 index 000000000000..6ead6b58a5c0 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_tuning/sm107/tp2/rank_0.json @@ -0,0 +1,1644 @@ +{ + "autotuner": [ + { + "min_time": 0.011449600011110306, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 1, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 346 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.10818239450454711, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 1024, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 128, + 23 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.07978240251541138, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 128, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 16, + 236 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.049420800805091855, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 16, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 323 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.5331935882568359, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 16384, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 256, + 7 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x128_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x128_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.016697600483894348, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 2, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 255 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.14368640184402465, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 2048, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 128, + 23 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.09130880236625671, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 256, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 16, + 228 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f" + }, + { + "min_time": 0.06027519702911377, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 32, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 176 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f" + }, + { + "min_time": 0.02431039959192276, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 4, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 143 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.19847999811172484, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 4096, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 128, + 22 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.11694079637527466, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 512, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 64, + 2 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.06737599968910217, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 64, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 16, + 234 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.037254399061203, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 8, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 191 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.32629759311676027, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 8192, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 256, + 7 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x128_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x128_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.061312001943588254, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 16384, + 1536 + ], + [ + -1 + ], + [ + 6144, + 1536 + ], + [ + 294912 + ], + [ + 1 + ] + ], + "tactic": 2, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 4x2x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.19057600498199462, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 16384, + 6144 + ], + [ + -1 + ], + [ + 4992, + 6144 + ], + [ + 958464 + ], + [ + 1 + ] + ], + "tactic": 2, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 4x2x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.007670400291681289, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8, + 4096 + ], + [ + -1 + ], + [ + 6144, + 4096 + ], + [ + 786432 + ], + [ + 1 + ] + ], + "tactic": 3, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 2x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.016451199352741242, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8, + 6144 + ], + [ + -1 + ], + [ + 12288, + 6144 + ], + [ + 2359296 + ], + [ + 1 + ] + ], + "tactic": 3, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 2x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.00897279977798462, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8, + 6144 + ], + [ + -1 + ], + [ + 4992, + 6144 + ], + [ + 958464 + ], + [ + 1 + ] + ], + "tactic": 3, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 2x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.010239999741315842, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8, + 6144 + ], + [ + -1 + ], + [ + 6144, + 6144 + ], + [ + 1179648 + ], + [ + 1 + ] + ], + "tactic": 3, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 2x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.034483200311660765, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8192, + 1536 + ], + [ + -1 + ], + [ + 6144, + 1536 + ], + [ + 294912 + ], + [ + 1 + ] + ], + "tactic": 4, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 4x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.10131839513778687, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8192, + 6144 + ], + [ + -1 + ], + [ + 4992, + 6144 + ], + [ + 958464 + ], + [ + 1 + ] + ], + "tactic": 4, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 4x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + } + ], + "flashinfer": { + "cache": { + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 21 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 4096), (4096, 6144), (-1,), (786432,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 6144), (6144, 12288), (-1,), (2359296,), (0,), (-1, 12288), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 1 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 6144), (6144, 6144), (-1,), (1179648,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1024, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 12 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1024, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((128, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 6 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((128, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 5 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1280, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 22 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1280, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1536, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 4 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1536, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 23 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((16, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 22 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((16, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((16384, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 18 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((16384, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 18 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1792, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1792, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 21 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 4096), (4096, 6144), (-1,), (786432,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 6144), (6144, 12288), (-1,), (2359296,), (0,), (-1, 12288), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 1 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 6144), (6144, 6144), (-1,), (1179648,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2048, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2048, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 18 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((256, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 20 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((256, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 8 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2560, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2560, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 19 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((3072, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 17 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((3072, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((32, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((32, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 8 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((3584, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((3584, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 17 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 4096), (4096, 6144), (-1,), (786432,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 6 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 6144), (6144, 12288), (-1,), (2359296,), (0,), (-1, 12288), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 1 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 6144), (6144, 6144), (-1,), (1179648,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4096, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4096, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((512, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 4 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((512, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 23 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((64, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 6 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((64, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 9 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((768, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 13 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((768, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 4096), (4096, 6144), (-1,), (786432,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 6144), (6144, 12288), (-1,), (2359296,), (0,), (-1, 12288), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 1 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 6144), (6144, 6144), (-1,), (1179648,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8192, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 18 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8192, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 18 + ], + "_generation": "49984fba36679af08bf96ce9bcfab83e78b0570985c165b915f1fac8aec8e550", + "_metadata": { + "cublas_version": "13.8.0", + "cuda_version": "13.4", + "cudnn_frontend_version": "1.27.0", + "cudnn_version": "92600", + "flashinfer_version": "0.6.18", + "gpu": "NVIDIA Graphics Device" + } + }, + "guard": { + "flashinfer-python": "0.6.18", + "nvidia-cutlass-dsl": "4.8.0" + } + }, + "format": "m31_pin_v1", + "native_guard": { + "libtensorrt_llm.so": "267a6b9ba701948539a79f5234a26550", + "libth_common.so": "78cc74478f0bd79ca835dcf0390197da" + } +} diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_tuning/sm107/tp2/rank_1.json b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_tuning/sm107/tp2/rank_1.json new file mode 100644 index 000000000000..d50bc1e6e55e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/m31_tuning/sm107/tp2/rank_1.json @@ -0,0 +1,1644 @@ +{ + "autotuner": [ + { + "min_time": 0.011449600011110306, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 1, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 346 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.10818239450454711, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 1024, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 128, + 23 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.07978240251541138, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 128, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 16, + 236 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.049420800805091855, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 16, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 323 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.5331935882568359, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 16384, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 256, + 7 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x128_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x128_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.016697600483894348, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 2, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 255 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.14368640184402465, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 2048, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 128, + 23 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.09130880236625671, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 256, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 16, + 228 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f" + }, + { + "min_time": 0.06027519702911377, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 32, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 176 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f" + }, + { + "min_time": 0.02431039959192276, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 4, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 143 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.19847999811172484, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 4096, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 128, + 22 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.11694079637527466, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 512, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 64, + 2 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.06737599968910217, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 64, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 16, + 234 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x128_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.037254399061203, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 8, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 8, + 191 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x128_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.32629759311676027, + "op": "trtllm::fp4_block_scale_moe_runner", + "runner": "FP4BlockScaleMoERunner", + "runner_id": 0, + "runner_key": "(4, 1536, 128, 0)", + "shapes": [ + [ + -1, + 128 + ], + [ + 128 + ], + [ + 8192, + 3072 + ], + [ + -1 + ], + [ + 128, + 3072, + 3072 + ], + [ + 128, + 3072, + 384 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128, + 6144, + 768 + ], + [ + 128, + 6144, + 96 + ], + [ + 0 + ], + [ + 128 + ], + [ + 128 + ], + [ + 128 + ], + [ + 0 + ], + [ + 0 + ] + ], + "tactic": [ + 256, + 7 + ], + "tactic_name": "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x128_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm107a | bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x128_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm107a" + }, + { + "min_time": 0.061312001943588254, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 16384, + 1536 + ], + [ + -1 + ], + [ + 6144, + 1536 + ], + [ + 294912 + ], + [ + 1 + ] + ], + "tactic": 2, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 4x2x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.19057600498199462, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 16384, + 6144 + ], + [ + -1 + ], + [ + 4992, + 6144 + ], + [ + 958464 + ], + [ + 1 + ] + ], + "tactic": 2, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 4x2x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.007670400291681289, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8, + 4096 + ], + [ + -1 + ], + [ + 6144, + 4096 + ], + [ + 786432 + ], + [ + 1 + ] + ], + "tactic": 3, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 2x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.016451199352741242, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8, + 6144 + ], + [ + -1 + ], + [ + 12288, + 6144 + ], + [ + 2359296 + ], + [ + 1 + ] + ], + "tactic": 3, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 2x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.00897279977798462, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8, + 6144 + ], + [ + -1 + ], + [ + 4992, + 6144 + ], + [ + 958464 + ], + [ + 1 + ] + ], + "tactic": 3, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 2x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.010239999741315842, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8, + 6144 + ], + [ + -1 + ], + [ + 6144, + 6144 + ], + [ + 1179648 + ], + [ + 1 + ] + ], + "tactic": 3, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 2x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.034483200311660765, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8192, + 1536 + ], + [ + -1 + ], + [ + 6144, + 1536 + ], + [ + 294912 + ], + [ + 1 + ] + ], + "tactic": 4, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 4x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + }, + { + "min_time": 0.10131839513778687, + "op": "trtllm::mxfp8_mxfp8_gemm_autotuned::gemm", + "runner": "MXFP8GemmRunner", + "runner_id": 0, + "runner_key": "(torch.bfloat16, 107)", + "shapes": [ + [ + 8192, + 6144 + ], + [ + -1 + ], + [ + 4992, + 6144 + ], + [ + 958464 + ], + [ + 1 + ] + ], + "tactic": 4, + "tactic_name": "Cutlass GEMM Tactic\n\tstyle=TMA Warp Specialized\n\tsm: 107\n\ttile shape ID: 128x128x256\n\tcluster shape ID: 4x4x1\n\tdynamic cluster shape ID: undefined\n\tfallback cluster shape ID: undefined\n\tmainloop sched: 0\n\tepi sched: 0\n\tenable cuda kernel: false\n\tepilogue fusion type: 0\n\tswap_ab: false\n" + } + ], + "flashinfer": { + "cache": { + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 17 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 4096), (4096, 6144), (-1,), (786432,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 23 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 6144), (6144, 12288), (-1,), (2359296,), (0,), (-1, 12288), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 1 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1, 6144), (6144, 6144), (-1,), (1179648,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1024, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 19 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1024, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 17 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((128, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 12 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((128, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1280, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 23 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1280, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 17 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1536, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1536, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 23 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((16, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 1 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((16, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((16384, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 18 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((16384, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1792, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 22 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((1792, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 6 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 4096), (4096, 6144), (-1,), (786432,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 23 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 6144), (6144, 12288), (-1,), (2359296,), (0,), (-1, 12288), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 1 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2, 6144), (6144, 6144), (-1,), (1179648,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2048, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2048, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 4 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((256, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 17 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((256, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 9 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2560, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((2560, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((3072, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((3072, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 17 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((32, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 4 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((32, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((3584, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 18 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((3584, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 23 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 4096), (4096, 6144), (-1,), (786432,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 1 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 6144), (6144, 12288), (-1,), (2359296,), (0,), (-1, 12288), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4, 6144), (6144, 6144), (-1,), (1179648,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4096, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 15 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((4096, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 19 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((512, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 20 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((512, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 23 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((64, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 20 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((64, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 9 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((768, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 13 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((768, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 27 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 4096), (4096, 6144), (-1,), (786432,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 22 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 6144), (6144, 12288), (-1,), (2359296,), (0,), (-1, 12288), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8, 6144), (6144, 6144), (-1,), (1179648,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 3 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8192, 1536), (1536, 6144), (-1,), (294912,), (0,), (-1, 6144), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 18 + ], + "('mxfp8_gemm', 'CutlassMxfp8GemmRunner', ((8192, 6144), (6144, 4992), (-1,), (958464,), (0,), (-1, 4992), (33554432,)), ())": [ + "CutlassMxfp8GemmRunner", + 14 + ], + "_generation": "c393399e8cdffc33fdbd31f6f9ad4052182ddd01a6f61a9ed7de6dcaace37434", + "_metadata": { + "cublas_version": "13.8.0", + "cuda_version": "13.4", + "cudnn_frontend_version": "1.27.0", + "cudnn_version": "92600", + "flashinfer_version": "0.6.18", + "gpu": "NVIDIA Graphics Device" + } + }, + "guard": { + "flashinfer-python": "0.6.18", + "nvidia-cutlass-dsl": "4.8.0" + } + }, + "format": "m31_pin_v1", + "native_guard": { + "libtensorrt_llm.so": "267a6b9ba701948539a79f5234a26550", + "libth_common.so": "78cc74478f0bd79ca835dcf0390197da" + } +} diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/__init__.py new file mode 100644 index 000000000000..a9cc1287cdcf --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 NVFP4 / sm_107 / tp2 (decode; candidate: M3.1 tp4 + TP8 levers + TP-width edits).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/dspark.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/dspark.py new file mode 100644 index 000000000000..bf94071a4f9e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/dspark.py @@ -0,0 +1,1065 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-model DSpark on this target (phase W0): the in-tree DSpark drafter and worker, unchanged. + +The target keeps its own forward; the shell hands the step to ``M3DSparkWorker``, which is the in-tree +``DSparkWorker`` plus the five calls the shell makes on any worker (``capture_hidden_states``, +``arm_capture_norm``, ``tail_eligible``, ``prebuild``, ``__call__``). In W0 none of them takes a fused path: the +captures are the in-tree DFlash captures, the target logits go through the logits processor and the in-tree +strict accept verifies them, so the speculation is the in-tree one by construction. + +The drafter is the in-tree standalone DSpark drafter for the checkpoint's architecture, built through the draft +model registry. Its context K/V lives in the separate draft KV cache, which must keep the drafter's projection +dtype (BF16): a pool of another dtype is not bound and the worker falls back to a private arena sized by the +served sequence length. So the drafter never takes the target's KV quantization: neither the FP8 draft-KV +override (an Eagle3 need) nor the target's own KV algo (NVFP4 under ``kv_cache_config.dtype="nvfp4"``). +""" + +from __future__ import annotations + +import functools +import os +import time +from typing import Optional, Set, Tuple + +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.comm import ( + allgather_cast as _allgather_cast, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + strict_accept as _strict_accept, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_block_tables import ( + enable_parallel_refresh, + refresh_block_tables, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_advance import ( + advance_ctx_and_read_view, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_store import ( + fold_k_norm_weight, + fused_ctx_store, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_drafter_inputs import ( + drafter_inputs as _drafter_inputs, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_forward_indices import ( + forward_indices, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_qk_norm_rope import ( + qk_norm_rope, +) +from tensorrt_llm._torch.distributed import AllReduce, AllReduceStrategy, allgather +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + external_drafter_config_kwargs, + get_draft_model, +) +from tensorrt_llm._torch.modules.linear import Linear, TensorParallelMode +from tensorrt_llm._torch.speculative.dspark import DSparkWorker +from tensorrt_llm._torch.speculative.interface import should_use_separate_draft_kv_cache +from tensorrt_llm.logger import logger + +from .dspark_s4 import MAX_FAST_BATCH, TAILBK_SPEC, S4WorkerMixin +from .kernels import m3_mc_allreduce, m3_report +from .m31_switches import m31_disabled, m31_enabled +from .mxdraft import route_mx_linears +from .ops.embedding.m3_embedding import m3_embedding + +# S9b-VR-a: the drafter's row-parallel all-reduces (each layer's o_proj and MLP down projection) of up to this many +# rows run the one-shot all-reduce instead of AUTO. On this target AUTO resolves to NCCL at every row count: the AR +# tuner's profiling of the custom tactics fails here ("Total failed profiling tactics: 14"), so the +# decode-step drafter ARs ran as the NCCL ring. VR TP4 [rows, 6144] BF16, graph of 10 calls: 4 rows 16.46 -> +# 5.64 us per call, 32 rows 16.66 -> 6.46; error vs the FP32 sum 2.0e-3 against NCCL's 3.9e-3 (a different +# reduction order: not bitwise against NCCL). Wider calls (the drafter's context passes) keep AUTO. Never the two-shot +# all-reduce: 430 us at 4 rows here. TRTLLM_M31_DISABLE_S9BVR=1 restores AUTO at every row count. +# TP2: ON with the TP8 levers (MXDRAFT, S9b); the switch is parsed strictly (m31_switches). +S9BVR = not m31_disabled("S9BVR") +DRAFT_AR_ONESHOT_MAX_ROWS = 32 if S9BVR else 0 # capacity: fixed: the K3 c8 verify rows + + +class DraftOneshotAllReduce(nn.Module): + """A row-parallel linear's all-reduce: the one-shot all-reduce up to ``max_rows`` rows, its original (AUTO) + all-reduce above. The output buffer choice stays the original module's, so wider calls are unchanged.""" + + def __init__(self, auto: AllReduce, dtype: torch.dtype, max_rows: int, name: str): + super().__init__() + self.auto = auto + self.oneshot = AllReduce( + mapping=auto.mapping, strategy=AllReduceStrategy.ONESHOT, dtype=dtype + ) + self.max_rows = max_rows + self.name = name + self._logged: Set[int] = set() + + def uses_nccl_symmetric_memory_window(self) -> bool: + return self.auto.uses_nccl_symmetric_memory_window() + + def forward(self, input: torch.Tensor, *args, **kwargs): + rows = input.shape[0] if input.dim() > 1 else 1 + if rows > self.max_rows: + return self.auto(input, *args, **kwargs) + if rows not in self._logged: + self._logged.add(rows) + logger.info( + f"[M3 dispatch] rank {self.auto.mapping.tp_rank}: S9b-VR-a drafter all-reduce ONESHOT taken at " + f"{rows} rows ({self.name})" + ) + m3_report.work( + "S9BVR", f"{rows} rows, world {self.auto.mapping.tp_size}, ONESHOT ({self.name})" + ) + return self.oneshot(input, *args, **kwargs) + + +def install_draft_oneshot_allreduce(draft: nn.Module, max_rows: int) -> int: + """Wrap every row-parallel linear's AUTO all-reduce in the drafter (S9b-VR-a); returns how many were wrapped.""" + if max_rows <= 0: + return 0 + n = 0 + for name, mod in draft.named_modules(): + ar = getattr(mod, "all_reduce", None) + if ( + isinstance(mod, Linear) + and mod.tp_mode == TensorParallelMode.ROW + and isinstance(ar, AllReduce) + and ar.strategy == AllReduceStrategy.AUTO + and ar.mapping.tp_size > 1 + ): + mod.all_reduce = DraftOneshotAllReduce(ar, mod.dtype, max_rows, name) + n += 1 + return n + + +# BUNDLE_OFF_NOTE: S5 and S2 (prebuild below) are OFF by default (S9b is on at this TP again, see +# S9B_DEFAULT_ON) with the K3 bundle {S9b, S5, S2, FUSED_CAPTURE}: its +# powered K7 natural-acceptance gate (n 1,776) failed (mean d +# -0.032 AL, CI [-0.059, -0.005], median -0.022, sign p 0.037 against), which applies to this sm_107 target +# too, pending a per-member bisect. Opt in with TRTLLM_M31_ENABLE_=1; TRTLLM_M31_DISABLE_=1 still wins. +# FUSED_CAPTURE does not exist in this target. + +# S9b (fused drafter all-reduce + norm) by default at this TP; TRTLLM_M31_DISABLE_S9B still turns it off. +# measured: TP8 -0.4% / step. +# ON again at this target (S9b only; S5 and S2 stay off per BUNDLE_OFF_NOTE): S9b alone passed its own powered K7 +# natural-acceptance gate here (1,904 pairs, mean d +0.020 AL, CI [-0.003, +0.043], median +0.018). +# TRTLLM_M31_DISABLE_S9B still turns it off. +S9B_DEFAULT_ON = True + +# MXDRAFT (TP2: the TP8 default, ON): the drafter's qkv / o / gate_up / down GEMMs of the decode +# passes on this target's MXFP8 GEMV (mxdraft.py; drafter-only precision, gated on natural AL). TP4 measured it slower +# ON, TP8 faster; TP2 takes the TP8 default pending its own kill A/B. Kill switch +# TRTLLM_M31_DISABLE_MXDRAFT=1, read at import. +MXDRAFT = not m31_disabled("MXDRAFT") + +# S9B-MC: S9b's fused drafter all-reduces + norms on the target's multicast all-reduce + norm (its 8-token bucket, the +# drafter-norm form: the unrounded fp32 sum normalized, opts bit 16) instead of the stock fused all-reduce; not +# bitwise, as S9b (a different summation order). The K7 c1 drafter block (8 rows) takes it; other row counts keep S9b's. +# TP2 (with MXDRAFT and S9BVR): ON by default. At world 2 the target's multicast op runs the +# drafter form on its 8-CTA cluster kernel (opts 23 / 31 = 16 | 7 / 16 | 15; the TP4 row kernel takes world 4 only): +# deterministic, the updated residual exact, the RMS statistic the cluster's sum (exact-class to FlashInfer's order, +# as the TP4 option-128 form). Rows past S9B_MC_MAX_ROWS keep S9b's fused all-reduce (S9BVR's one-shot below 33 rows). +# Kill switch TRTLLM_M31_DISABLE_S9B_MC=1 (S9b's fused path at every row count). Read at import. +S9B_MC = not m31_disabled("S9B_MC") +S9B_MC_MAX_ROWS = 8 # capacity: fixed: the drafter-norm form exists for the 8-token bucket only + + +def mcag_unreachable(worker) -> Optional[str]: + """Why no vocab-sharded draft logits reach the rejection sampler's all-gather that MCAG replaces (None: some can). + Only the S4-logits path hands the stock sampler a rank's vocab shard of the draft logits; without it the drafter's + head gathers its logits itself (gather_output), they arrive full-vocab, and the gather is never called (no MCAG + work at TP2).""" + if getattr(worker, "_s4", None) is None: + return ( + "no vocab-sharded draft logits reach the rejection sampler: S4 is not set up, the drafter's gathered head " + "returns full-vocab logits" + ) + if not getattr(worker, "_s4_logits_on", False): + return ( + "no vocab-sharded draft logits reach the rejection sampler: S4-logits is off, sampled steps take the " + "in-tree chain on the gathered head's full-vocab logits" + ) + return None + + +def _counted(item: str, fn, *extra): + """``fn`` (a fused hook a DSpark worker installs) with a gate-e work counter: each call that did the work (returned + neither None nor False) counts ``item`` and every ``(item, enabled)`` of ``extra`` whose flag is set. None stays + None (the hook is not installed).""" + if fn is None: + return None + + @functools.wraps(fn) + def run(*args, **kwargs): + out = fn(*args, **kwargs) + if out is not None and out is not False: + m3_report.work(item, f"{getattr(fn, '__name__', item)} call") + for name, on in extra: + if on: + m3_report.work(name, f"inside {item}") + return out + + return run + + +def _s9b_count_op(mapping, inner): + """The fused AR + norm op installed whenever S9b is on: ``inner`` (S9B-MC's multicast op, or None) first; counts + S9b's fused all-reduce + norm on every call (the stock fused path when ``inner`` declines) and S9B-MC's when it + took the operands. Shapes decide, so every rank agrees.""" + + def op(partial, residual, norm_weight, eps): + out = inner(partial, residual, norm_weight, eps) if inner is not None else None + rows = partial.shape[0] if partial.dim() == 2 else 0 + m3_report.work("S9B", f"{rows} rows, world {mapping.tp_size}") + if out is not None: + m3_report.work( + "S9B_MC", + f"{rows} rows, world {mapping.tp_size}, drafter form (opts 16) on the " + + ("8-CTA cluster kernel" if mapping.tp_size != 4 else "row kernel"), + ) + return out + + return op + + +def _s9b_mc_op(mapping): + """The fused AR + norm op S9B-MC installs: (normed, residual + reduced) from the multicast all-reduce + norm, or + None (the stock fused all-reduce runs) for operands it does not take. Shapes and the collectively created workspace + decide, so every TP rank agrees.""" + + def op(partial, residual, norm_weight, eps): + if partial.dim() != 2 or not 1 <= partial.shape[0] <= S9B_MC_MAX_ROWS: + return None + workspace = m3_mc_allreduce.get_workspace(mapping, partial.shape[-1]) + if workspace is None: + return None + return m3_mc_allreduce.m3_mc_allreduce_norm( + partial.contiguous(), + residual.contiguous(), + norm_weight, + eps, + workspace, + use_pdl=get_env_enable_pdl(), + drafter_norm=True, + ) + + return op + + +# S5: the largest decode batch the window-capped max_seq_len applies to, by draft length; 0 = off. Per (TP, K) on +# VR: the largest batch at which the window-capped drafter attention is faster than the stock call at every batch up +# to it, in two replicates of the per-call microbench (L = 400K). TP8: K3 shows no win +# at c1 (+0.04 / +0.004 us per call), so off; K7 c1 / c2 win, c4 loses +1.5. +S5_MAX_BATCH_BY_K = {3: 0, 7: 2} + +# S10 (lean drafter glue) applies at every draft length on VR TP8: token-identical to the eager glue at K3 +# and at K7 with the AllReduce autotune pinned (43 / 43 requests over c1 / c2 / c4, seeded T 1.0; an +# earlier K7 divergence ran with the autotune unpinned). TRTLLM_M31_DISABLE_S10=1 turns it off. +S10_MAX_DRAFT_LEN = 63 + + +def s10_applies(num_draft_tokens: int) -> bool: + return num_draft_tokens <= S10_MAX_DRAFT_LEN + + +def s5_max_batch_for(num_draft_tokens: int) -> int: + """K <= 3 takes the K3 entry, K >= 7 the K7 entry, and K in between the smaller of the two.""" + if num_draft_tokens <= 3: + return S5_MAX_BATCH_BY_K[3] + if num_draft_tokens >= 7: + return S5_MAX_BATCH_BY_K[7] + return min(S5_MAX_BATCH_BY_K.values()) + + +def is_standalone_dspark(spec_config) -> bool: + """Whether the run speculates with a standalone DSpark drafter (its own checkpoint, ``decoding_type: DSpark``).""" + return bool( + spec_config is not None + and spec_config.spec_dec_mode.use_one_engine() + and spec_config.spec_dec_mode.is_dspark() + and not getattr(spec_config, "draft_is_embedded_in_target", False) + ) + + +class M3EmbedTokens(nn.Module): + """The target's token embedding as the module the in-tree drafter borrows (``load_weights_from_target_model``). + + The drafter embeds the bonus and mask tokens through ``embed_tokens(ids)``; the target keeps its replicated + table as a raw tensor and gathers it with ``m3_embedding``, so this module calls the same gather on the same + table (an exact gather, so the drafter sees the rows the target embeds). + """ + + def __init__(self, core: nn.Module) -> None: + super().__init__() + self._core = [core] # not a submodule: the table belongs to the target + + @property + def weight(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def forward(self, ids: torch.Tensor) -> torch.Tensor: + return m3_embedding(ids, self.weight) + + +def s10c_c_drafter_inputs( + embed_tokens, + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + mask_token_id, + trained_mask_embedding, + block_size, + tokens_per_request, +): + """S10c-C: the drafter's step inputs in one launch (catalog/spec_decode/dspark_drafter_inputs), for this target's + embedding only: ``M3EmbedTokens`` is an exact gather of the replicated table, so the noise block is table rows (the + trained mask row if the drafter keeps one). None for any other embedding: the stock path runs.""" + if not isinstance(embed_tokens, M3EmbedTokens): + return None + table = embed_tokens.weight + mask_row = ( + trained_mask_embedding if trained_mask_embedding is not None else table[int(mask_token_id)] + ) + return _drafter_inputs( + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + table, + mask_row, + block_size, + tokens_per_request, + ) + + +# N3 A2 (switch TRTLLM_M31_DISABLE_N3A): the target's strict acceptance and next-step input tokens in one launch each +# (catalog/spec_decode/strict_accept), bitwise to the in-tree torch path. Read once, at import. +N3A_DISABLED = m31_disabled("N3A") +# N3A-KV (switch TRTLLM_M31_DISABLE_N3A_KV, and off with N3A): DFlash's draft-forward KV bookkeeping (the rewind amount +# 1 - num_accepted and kv_lens += 1) in one launch where the in-tree lines run, after the forced-acceptance override, +# bitwise (integer arithmetic). Read once, at import. +N3A_KV_DISABLED = N3A_DISABLED or m31_disabled("N3A_KV") +# Gate-only cross-check (TRTLLM_M31_CHECK_N3A_KV=1; never in a perf arm): every N3A-KV call is compared with the stock +# lines on copies (graph-safe device counters; reported on eager steps). +N3A_KV_CHECK = os.environ.get("TRTLLM_M31_CHECK_N3A_KV", "").strip() not in ("", "0") +# N3 P-gid + P-cat (switch TRTLLM_M31_DISABLE_N3P): the verify-row target logits without the identity gather_ids copy +# and with the vocab concat and the fp32 cast in one strided copy, bitwise to the logits processor. Read at import. +N3P_DISABLED = m31_disabled("N3P") +# N3 A1 (switch TRTLLM_M31_DISABLE_A1): the verify logits' vocab all-gather, concat and fp32 cast in one launch over a +# multicast NVLink workspace (catalog/comm/allgather_cast), bitwise to the NCCL all-gather + strided copy below. The +# verify rows (B x (K + 1), up to 64 rows: K3 batches 1..16, K7 batches 1..8). Read once, at import. Default on +# on VR since the VR op matrix + grid sweep certified it bitwise; the switch turns it off. +A1_DISABLED = m31_disabled("A1") +# A1_WIDE (agg TP2: K3 c46 / c56 / c64 = 184 / 224 / 256 verify rows): the A1 workspace holds 256 rows +# (allgather_cast.configure_capacity before the first workspace; 307 MB of multicast memory per rank at TP2). VR +# microbench (world 2, shard 100032): bitwise at 64 / 128 / 184 / 224 / 256 rows and 95..188 us per call faster than the +# NCCL all-gather + concat + cast past 64 rows (e.g. 256 rows 393.4 -> 204.7 us). TRTLLM_M31_DISABLE_A1_WIDE=1 keeps +# 64 rows (the NCCL all-gather above them). Read once, at import. +A1_WIDE = not m31_disabled("A1_WIDE") +# The existing all-gather/cast also passed the 448-row capacity gate. +# Default ON, about 538 MB/rank at TP2. The new kill restores 256 rows; A1_WIDE still restores 64. +A1_WIDE_448 = A1_WIDE and not m31_disabled("A1_WIDE_448") +A1_MAX_ROWS = 448 if A1_WIDE_448 else (256 if A1_WIDE else 64) +# Draft-head gathering reuses the prebuilt verify workspace; the kill keeps stock routing. +DRAFT_A1 = not m31_disabled("DRAFT_A1") +# MCAG (multicast all-gather): the rejection sampler's gather of the vocab-sharded fp32 draft logits (stock: the NCCL +# all-gather + a concat copy) as one launch on the A1 multicast workspace (allgather_cast.allgather_f32: copied as is, +# bit for bit). VR TP8 K7 c1, [1, 7, 25008] fp32: 32.30 -> 20.15 us (bitwise eager and graph). Taken +# for the B * K draft rows up to the fp32 capacity while A1 is set up; other shapes keep the stock all-gather. OFF by +# default at this target, as at TP8: with it on, two runs of the same requests (every tactic cache pinned on all ranks) +# sample different tokens on a few long requests, and with it off they do not; the stock all-gather returns the same +# tensor bit for bit. TRTLLM_M31_ENABLE_MCAG=1 opts in; TRTLLM_M31_DISABLE_MCAG=1 still wins. +MCAG = m31_enabled("MCAG") and not m31_disabled("MCAG") +# A1 at draft lengths other than 3 (K7 included; switch TRTLLM_M31_DISABLE_A1_K7, K3 keeps A1). +A1_K7_DISABLED = m31_disabled("A1_K7") +_n3a_logged = False + + +def _n3a_log_once() -> None: + global _n3a_logged + if not _n3a_logged: + _n3a_logged = True + logger.info( + "[M3 dispatch] n3_strict_accept taken: fused strict acceptance + next-step input tokens" + ) + + +_n3p_logged = False +_n3a_kv_logged = False + + +def _n3a_kv_log_once() -> None: + global _n3a_kv_logged + if not _n3a_kv_logged: + _n3a_kv_logged = True + logger.info( + "[M3 dispatch] n3a_kv_rewind taken: draft-forward KV rewind + kv_lens bump in one launch" + ) + + +_a1_logged = set() + + +def _a1_log_once(key, message: str) -> None: + if key not in _a1_logged: + _a1_logged.add(key) + logger.info(message) + + +def _n3p_log_once() -> None: + global _n3p_logged + if not _n3p_logged: + _n3p_logged = True + logger.info( + "[M3 dispatch] n3p_target_logits taken: verify-row logits without the gather_ids copy and the concat" + ) + + +class _DraftA1Logits(nn.Module): + """Gather the full-vocabulary draft head through a prebuilt A1 workspace.""" + + def __init__(self, inner, head, workspace, rows: Tuple[int, ...]) -> None: + super().__init__() + self.inner = inner + # The model owns its head; avoid registering it a second time here. + self._head = [head] + self._workspace = workspace + self._rows = frozenset(rows) + + def forward(self, hidden_states, lm_head, attn_metadata, return_context_logits=False): + if ( + lm_head is not self._head[0] + or not return_context_logits + or hidden_states.dim() != 2 + or hidden_states.dtype != torch.bfloat16 + or hidden_states.shape[0] not in self._rows + or attn_metadata is None + or attn_metadata.num_contexts != 0 + ): + return self.inner(hidden_states, lm_head, attn_metadata, return_context_logits) + shard = lm_head.apply_linear(hidden_states, None) + m3_report.work("DRAFT_A1", f"{shard.shape[0]} draft rows, world {self._workspace['world']}") + return _allgather_cast.allgather_cast(shard, **self._workspace) + + +class M3DSparkWorker(S4WorkerMixin, DSparkWorker): + """The in-tree ``DSparkWorker`` with the calls this target's shell makes on its worker. W1: the one-launch Markov + chain (``S4WorkerMixin``, switch ``TRTLLM_M31_DRAFT_DISABLE_S4``). N3 A2: the strict acceptance and the next-step + input tokens in one launch each (switch ``TRTLLM_M31_DISABLE_N3A``). N3 P / A1: the verify logits (switches + ``TRTLLM_M31_DISABLE_N3P`` / ``TRTLLM_M31_DISABLE_A1``). Everything else is the in-tree path.""" + + def _static_arange(self, start: int, end: int) -> torch.Tensor: + """The in-tree worker's cached arange; under S10 (lean glue) each call counts the S10 work counter (the lean + glue's cached-index site, one of its strided-view sites).""" + if getattr(self, "_lean_glue", False): + m3_report.work("S10", f"lean glue arange [{start}, {end})") + return super()._static_arange(start, end) + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None + ) -> None: + """The in-tree DFlash capture: layer output (hidden + residual) into the static capture buffer.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model + ) -> None: + """No capture norm: DSpark's fc reads the raw captured layers.""" + return None + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Never: the target logits always go through the logits processor and the in-tree strict accept.""" + return False + + def prebuild(self, lm_head: nn.Module, draft_model) -> None: + """The one-launch Markov chain's shards, buffers, exchange workspace and compiles (outside capture, every + rank); no fused verify, no coded target head, no fused draft loop.""" + rank = getattr(getattr(self, "mapping", None), "tp_rank", None) + # S10: the drafter's strided-view glue (bitwise; kill switch TRTLLM_M31_DISABLE_S10, read at import). + k = getattr(getattr(self, "spec_config", None), "max_draft_len", 0) or 0 + lean = not m31_disabled("S10") and s10_applies(k) + self.enable_lean_glue(draft_model, lean) + # S5: window-capped max_seq_len for the drafter's sliding-window decode (not bitwise: the TRTLLM-Gen tile / KV + # split changes, not the keys read). OFF by default (K3 bundle, see BUNDLE_OFF_NOTE): opt in with + # TRTLLM_M31_ENABLE_S5=1; TRTLLM_M31_DISABLE_S5 still wins. + s5_max_batch = s5_max_batch_for(k) if m31_enabled("S5") and not m31_disabled("S5") else 0 + self.enable_window_capped_decode(draft_model, s5_max_batch) + self.enable_full_slot_arena() + # S2: the drafter's replicated fc split row-parallel with one fp32 all-reduce (not bitwise: the partial-sum + # order changes). Collective. OFF by default (K3 bundle): opt in with TRTLLM_M31_ENABLE_S2=1; + # TRTLLM_M31_DISABLE_S2 still wins. + s2 = self.enable_fc_tp_split(draft_model, m31_enabled("S2") and not m31_disabled("S2")) + split = getattr(draft_model, "_dflash_fc_split", None) + ar = split[3] if split is not None else None + ar_path = ( + "MNNVL" + if getattr(ar, "mnnvl_allreduce", None) is not None + else getattr(ar, "strategy", None) + ) + logger.info(f"M3.1 DSpark S2: fc TP split {'on' if s2 else 'off'} on rank {rank} (all-reduce " + f"{ar_path})") # fmt: skip + logger.info(f"M3.1 DSpark S5: window-capped draft decode for batches <= {s5_max_batch} on rank " + f"{rank}") # fmt: skip + logger.info(f"M3.1 DSpark S10: lean drafter glue {'on' if lean else 'off'} on rank {rank}") + # S9b: each drafter layer's all-reduce fused with the add + RMSNorm after it (not bitwise: the sum is not + # rounded before the add). On by default where S9B_DEFAULT_ON; kill switch TRTLLM_M31_DISABLE_S9B, read at + # import. + s9b = self.enable_fused_ar_norm( + draft_model, (S9B_DEFAULT_ON or m31_enabled("S9B")) and not m31_disabled("S9B") + ) + logger.info( + f"M3.1 DSpark S9b: fused drafter all-reduce + norm {'on' if s9b else 'off'} on rank {rank}" + ) + # S9B-MC: the multicast workspace is the target's (created collectively at its first eager boundary, the + # warmup forward, before any capture); until it exists the op declines on every rank alike. + installed = self.enable_fused_ar_norm_op( + draft_model, + _s9b_count_op(self.mapping, _s9b_mc_op(self.mapping) if S9B_MC else None) + if s9b + else None, + ) + s9b_mc = bool(installed) and S9B_MC + logger.info( + f"M3.1 DSpark S9B-MC: drafter all-reduce + norm on the multicast kernel (<= {S9B_MC_MAX_ROWS} rows) " + f"{'on' if s9b_mc else 'off'} on rank {rank}" + ) + # S10c-A: the drafter's context K / V epilogue + accept mask + paged store in one launch after the K / V GEMM + # (bitwise). Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_A, read at import. + # TAILBK_SPEC F: the store kernel also runs the k_norm weight multiply (bitwise torch's bf16 multiply; one + # launch fewer per step). Kill switch TRTLLM_M31_DISABLE_TAILBK_SPEC, read at import. + s10c_a_on = not (m31_disabled("S10C") or m31_disabled("S10C_A")) + tailbk_f = fold_k_norm_weight(s10c_a_on and TAILBK_SPEC) + s10c_a = self.enable_fused_ctx_store( + _counted("S10C_A", fused_ctx_store, ("TAILBK_SPEC", tailbk_f)) if s10c_a_on else None + ) + if not s10c_a: + tailbk_f = fold_k_norm_weight(False) + logger.info( + f"M3.1 DSpark S10c-A: fused drafter ctx store {'on' if s10c_a else 'off'}; TAILBK_SPEC F k_norm weight " + f"in the store kernel {'on' if tailbk_f else 'off'} on rank {rank}" + ) + # S10c-C: the drafter's step inputs (bonus, positions, noise block) in one launch (bitwise: integers and row + # copies). Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_C, read at import. + s10c_c = self.enable_fused_drafter_inputs( + None + if m31_disabled("S10C") or m31_disabled("S10C_C") + else _counted("S10C_C", s10c_c_drafter_inputs) + ) + logger.info( + f"M3.1 DSpark S10c-C: fused drafter step inputs {'on' if s10c_c else 'off'} on rank {rank}" + ) + # S10c-D: the drafter's context advance + read view in one launch (bitwise, integer). Kill switches + # TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_D, read at import. + s10c_d = self.enable_fused_ctx_advance( + None + if m31_disabled("S10C") or m31_disabled("S10C_D") + else _counted("S10C_D", advance_ctx_and_read_view) + ) + logger.info( + f"M3.1 DSpark S10c-D: fused drafter context advance {'on' if s10c_d else 'off'} on rank {rank}" + ) + # S10c-B: the drafter's block-table refresh (decode + placeholder count) in one launch (bitwise, integer). + # Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_B, read at import. + # S10c-B2: that refresh on a (row, column-chunk) grid (bitwise; flat at short tables, removes the serial + # kernel's growth past ~11K blocks per sequence). Kill switch TRTLLM_M31_DISABLE_S10C_B2, read at import. + s10c_b_on = not (m31_disabled("S10C") or m31_disabled("S10C_B")) + s10c_b2 = enable_parallel_refresh(s10c_b_on and not m31_disabled("S10C_B2")) + s10c_b = self.enable_fused_block_refresh( + _counted("S10C_B", refresh_block_tables, ("S10C_B2", s10c_b2)) if s10c_b_on else None + ) + if not s10c_b: + s10c_b2 = enable_parallel_refresh(False) + logger.info( + f"M3.1 DSpark S10c-B: fused drafter block-table refresh {'on' if s10c_b else 'off'} on rank {rank}" + f" (S10c-B2 2-D grid {'on' if s10c_b2 else 'off'})" + ) + # S10c-E: the draft model's paged block-decode index prelude in one launch (bitwise, integer). Kill switches + # TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_E, read at import. + s10c_e = self.enable_fused_forward_indices( + draft_model, + None + if m31_disabled("S10C") or m31_disabled("S10C_E") + else _counted("S10C_E", forward_indices), + ) + logger.info( + f"M3.1 DSpark S10c-E: fused drafter forward indices {'on' if s10c_e else 'off'} on rank {rank}" + ) + # EX-11: the drafter block's q / k RMSNorm + partial RoPE in one launch (bitwise to flashinfer's three). Kill + # switch TRTLLM_M31_DISABLE_EX11, read at import. + ex11 = self.enable_fused_qk_norm_rope( + draft_model, None if m31_disabled("EX11") else _counted("EX11", qk_norm_rope) + ) + logger.info( + f"M3.1 DSpark EX-11: fused drafter q/k norm + RoPE {'on' if ex11 else 'off'} on rank {rank}" + ) + # MXDRAFT: the drafter's qkv / o / gate_up / down GEMMs on this target's MXFP8 GEMV for the decode passes' + # rows (B x K, B <= MAX_FAST_BATCH, up to the GEMV's own row bound), from the exact MXFP8 form of the shipped + # weights (drafter-only precision, gated on natural AL). Kill switch TRTLLM_M31_DISABLE_MXDRAFT, read + # at import. + mx = ( + 0 + if not MXDRAFT or k <= 0 + else route_mx_linears(draft_model, [b * k for b in range(1, MAX_FAST_BATCH + 1)]) + ) + logger.info(f"M3.1 DSpark MXDRAFT: MXFP8 drafter GEMV on {mx} projections on rank {rank}") + self.prebuild_s4(lm_head, draft_model) + # N3 A2 compiles (rank-local), then the N3 A1 workspace and compiles (collective), outside capture. + if not N3A_DISABLED and lm_head is not None: + _strict_accept.precompile((self.max_draft_len,), lm_head.weight.device) + self._a1 = self._prebuild_a1(lm_head) if lm_head is not None else None + if DRAFT_A1 and self._a1 is not None: + self._prebuild_draft_a1(lm_head, draft_model) + # The rank-agreed S4 set-up and the S4-logits switch decide whether any vocab-sharded draft logits reach the + # gather MCAG replaces; when none can, MCAG is not compiled and its line says why (every rank alike). + mcag_off = mcag_unreachable(self) if self._a1 is not None and MCAG else None + self._mcag_cols = ( + self._prebuild_mcag(lm_head) + if self._a1 is not None and MCAG and mcag_off is None + else 0 + ) + mcag = self.enable_draft_logits_gather_op(self._mcag_gather if self._mcag_cols else None) + logger.info( + f"M3.1 MCAG: draft-logits all-gather on the multicast kernel {'on' if mcag else 'off'} on rank {rank}" + + (f" ({mcag_off})" if mcag_off else "") + ) + n3a_kv = False + if not N3A_KV_DISABLED and lm_head is not None: + _strict_accept.precompile_kv_rewind(lm_head.weight.device) + self._n3a_kv_buf = torch.empty( + _strict_accept.max_requests(), dtype=torch.int32, device=lm_head.weight.device + ) + n3a_kv = self.enable_fused_kv_rewind(self._n3a_kv_rewind) + logger.info( + f"M3.1 N3A-KV: one-launch draft-forward KV bookkeeping {'on' if n3a_kv else 'off'} on rank " + f"{getattr(getattr(self, 'mapping', None), 'tp_rank', None)}" + ) + + _n3a_kv_buf = None # the persistent rewind amounts of N3A-KV (int32, MAX_REQUESTS) + _n3a_kv_counts = None # gate-only cross-check counters (calls, mismatches) + + def _n3a_kv_rewind(self, num_accepted, kv_lens, nc: int, bs: int): + """N3A-KV for the DFlash hook: ``1 - num_accepted[nc:bs]`` (a view of the persistent buffer) with + ``kv_lens[nc:bs] += 1`` in one launch, or None (the stock lines) for operands or a variant it does not take.""" + n = bs - nc + buf = self._n3a_kv_buf + if buf is None or not _strict_accept.supports_kv_rewind(num_accepted, kv_lens, buf, nc, n): + return None + if not _strict_accept.is_compiled_kv_rewind(num_accepted, kv_lens): + return None + _n3a_kv_log_once() + m3_report.work("N3A_KV", f"{n} requests") + if not N3A_KV_CHECK: + return _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + want_rewind = 1 - num_accepted[nc:bs] + want_kv = kv_lens[nc:bs] + 1 + out = _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + if self._n3a_kv_counts is None: + self._n3a_kv_counts = torch.zeros(2, dtype=torch.int64, device=kv_lens.device) + bad = (out != want_rewind).any() | (kv_lens[nc:bs] != want_kv).any() + self._n3a_kv_counts[0] += 1 + self._n3a_kv_counts[1] += bad.to(torch.int64) + if not torch.cuda.is_current_stream_capturing(): + calls, mism = self._n3a_kv_counts.tolist() + logger.info(f"M3.1 N3A-KV check: {calls} calls, {mism} mismatches") + return out + + _a1 = None # the N3 A1 workspace (allgather_cast's arguments) once set up + + def _a1_all_ok(self, local: bool) -> bool: + """Whether ``local`` holds on every rank of the TP group: A1 is set up on every rank or on none (a rank that + is not eligible, or whose compile failed, turns it off everywhere).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + return _mnnvl_workspace_all_succeeded(_get_mnnvl_workspace_comm(self.mapping), local) + + def _prebuild_a1(self, lm_head: nn.Module): + """The N3 A1 workspace and its K3 verify-row compiles, or None (every rank alike): off by its switch, at + another draft length, without a column-parallel bf16 head over a TP group, or when a compile fails on any + rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + mapping = self.mapping + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return None + cols = int(getattr(lm_head, "out_features", 0)) + sms = torch.cuda.get_device_properties(lm_head.weight.device).multi_processor_count + # Why this rank would not set A1 up (empty: it would). The grid is a function of rows and the SM count: + # requiring every SM up to the workspace's MAX_GRID makes it a function of rows alone, the same on every rank. + off = [ + why + for why, bad in ( + ( + "TRTLLM_M31_DISABLE_A1 set", + A1_DISABLED, + ), + ( + f"draft length {int(self.max_draft_len)} with TRTLLM_M31_DISABLE_A1_K7 set", + int(self.max_draft_len) != 3 and A1_K7_DISABLED, + ), + ( + f"draft length {int(self.max_draft_len)}: more than {A1_MAX_ROWS} verify rows at batch 1", + int(self.max_draft_len) + 1 > A1_MAX_ROWS, + ), + ( + "not a column-parallel, gathered, unpadded, bias-free head", + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or getattr(lm_head, "bias", None) is not None, + ), + (f"head dtype {lm_head.weight.dtype}", lm_head.weight.dtype != torch.bfloat16), + ( + f"{cols} shard columns (whole 16-byte vectors needed)", + cols <= 0 or cols % 8 != 0, + ), + ( + f"{sms} SMs (fewer than {_allgather_cast.MAX_GRID})", + sms < _allgather_cast.MAX_GRID, + ), + ) + if bad + ] + if not self._a1_all_ok(not off): + logger.info( + f"M3.1 N3 A1: off on rank {mapping.tp_rank} ({'; '.join(off) or 'off on another rank'}); " + "the verify logits use the NCCL all-gather" + ) + return None + k1 = int(self.max_draft_len) + 1 + rows = tuple(b * k1 for b in range(1, A1_MAX_ROWS // k1 + 1)) + start = time.perf_counter() + _allgather_cast.configure_capacity(A1_MAX_ROWS) + if not _allgather_cast.precompile( + lm_head.mapping, cols, rows, lm_head.weight.device, agree=self._a1_all_ok + ): + logger.warning( + "M3.1 N3 A1: a compile failed on some rank; every rank keeps the NCCL all-gather" + ) + return None + ws = _allgather_cast.workspace_args(lm_head.mapping, cols) + # The engagement line: one per rank, at server start. + logger.info( + f"M3.1 N3 A1: one-launch verify all-gather set up on rank {mapping.tp_rank}/{mapping.tp_size} " + f"(shard {cols} columns, rows {rows}, workspace {ws['ws_uc'].numel() * 4} bytes/rank, " + f"set-up {time.perf_counter() - start:.2f} s)" + ) + return ws + + def _prebuild_draft_a1(self, target_head: nn.Module, draft_model) -> None: + """Reuse the verify workspace for matching draft shards, compiling B*K shapes before capture.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor + + head = draft_model.lm_head + inner = draft_model.logits_processor + if isinstance(inner, _DraftA1Logits): + inner = inner.inner + k = int(self.max_draft_len) + cols = int(getattr(head, "out_features", 0)) + eligible = ( + type(inner) is LogitsProcessor + and k > 0 + and head.tp_mode == TensorParallelMode.COLUMN + and head.gather_output + and head.gather_output_sizes is None + and head.padding_size == 0 + and head.bias is None + and head.weight.dtype == torch.bfloat16 + and head.weight.device == target_head.weight.device + and cols == target_head.out_features + and head.mapping.tp_group == target_head.mapping.tp_group + and head.mapping.tp_rank == target_head.mapping.tp_rank + ) + if not self._a1_all_ok(eligible): + logger.info( + "M3.1 DRAFT_A1: incompatible draft head or custom logits processor; keeping stock" + ) + return + rows = tuple(b * k for b in range(1, A1_MAX_ROWS // k + 1) if b * k > 64) + if not rows or not _allgather_cast.precompile( + target_head.mapping, cols, rows, head.weight.device, agree=self._a1_all_ok + ): + logger.warning("M3.1 DRAFT_A1: draft-shape compilation unavailable; keeping stock") + return + draft_model.logits_processor = _DraftA1Logits(inner, head, self._a1, rows) + logger.info( + f"M3.1 DRAFT_A1: draft-head all-gather set up on rank {self.mapping.tp_rank}, rows {rows}" + ) + + def _prebuild_mcag(self, lm_head: nn.Module) -> int: + """Compile the fp32 draft-logits gather for B * K rows up to its capacity, on every rank (A1's workspace and + agreement); its shard columns, or 0 (every rank alike) when a compile fails anywhere.""" + cols = int(getattr(lm_head, "out_features", 0)) + k = int(self.max_draft_len) + if k <= 0 or cols % 4 != 0: + return 0 + rows = tuple(b * k for b in range(1, _allgather_cast.MAX_ROWS_FP32 // k + 1)) + if not _allgather_cast.precompile_f32( + lm_head.mapping, cols, rows, lm_head.weight.device, agree=self._a1_all_ok + ): + logger.warning( + "M3.1 MCAG: a compile failed on some rank; every rank keeps the NCCL all-gather" + ) + return 0 + return cols + + def _mcag_gather(self, logits: torch.Tensor): + """The draft logits gathered in one launch on the A1 workspace, or None (the stock all-gather) for a shape + that is not compiled; the decision depends on the shape only, the same on every rank.""" + rows = logits.numel() // max(int(logits.shape[-1]), 1) + if ( + logits.dtype != torch.float32 + or int(logits.shape[-1]) != self._mcag_cols + or not _allgather_cast.is_compiled( + rows, self._mcag_cols, self._a1["world"], device=logits.device, fp32=True + ) + ): + _a1_log_once( + ("mcag", rows, logits.dtype), + f"[M3 dispatch] MCAG draft-logits all-gather not taken: {rows} rows {logits.dtype}; NCCL all-gather", + ) + return None + _a1_log_once( + "mcag", + "[M3 dispatch] MCAG draft-logits all-gather taken: one launch on the A1 workspace", + ) + m3_report.work("MCAG", f"{rows} draft rows, world {self._a1['world']}") + return _allgather_cast.gather_last_dim_f32(logits.contiguous(), **self._a1) + + def _n3a_eligible( + self, logits, draft_tokens, num_contexts: int, batch_size: int, spec_metadata + ) -> bool: + """Whether this step's strict acceptance takes the fused op: a decode-only, non-ragged step whose logits hold + exactly batch_size x (D + 1) verify rows and whose drafts the op takes. Host state and shapes only, so it is + fixed per captured graph and the same on every rank. Decided before the target tokens are sampled, so the + sampler runs exactly once either way.""" + if N3A_DISABLED or spec_metadata is None or spec_metadata.is_ragged_verify: + return False + if num_contexts != 0 or batch_size <= 0 or draft_tokens.dim() != 2: + return False + n_req, draft_len = draft_tokens.shape + if n_req != batch_size or draft_len <= 0 or n_req > _strict_accept.max_requests(): + return False + rows = logits.shape[0] if logits.dim() > 1 else 1 + return rows == batch_size * (draft_len + 1) and draft_tokens.is_contiguous() + + def _sample_and_accept_draft_tokens_base( + self, logits, draft_tokens, num_contexts, batch_size, spec_metadata + ): + """The in-tree strict acceptance with its epilogue in one launch (N3 A2): the same target tokens (the in-tree + sampler, called once), the same accepted tokens and counts, then the in-tree forced-acceptance override (its + RNG pool and counters advance as on an in-tree step).""" + if not self._n3a_eligible(logits, draft_tokens, num_contexts, batch_size, spec_metadata): + return super()._sample_and_accept_draft_tokens_base( + logits, draft_tokens, num_contexts, batch_size, spec_metadata + ) + if logits.dim() == 1: + logits = logits.unsqueeze(0) + target_tokens = self._sample_tokens_for_batch( + logits, spec_metadata, num_contexts, batch_size + ) + if _strict_accept.supports_strict_accept( + target_tokens, draft_tokens + ) and _strict_accept.is_compiled_strict_accept(target_tokens, draft_tokens): + accepted, num_accepted = _strict_accept.spec_strict_accept(target_tokens, draft_tokens) + _n3a_log_once() + m3_report.work("N3A", f"strict accept, {draft_tokens.shape[0]} requests") + else: + # The sampler already ran: finish with the in-tree epilogue (the op's reference is that code). + accepted, num_accepted = _strict_accept.spec_strict_accept.reference( + target_tokens, draft_tokens + ) + num_accepted = self._apply_force_accepted_tokens( + num_accepted, num_contexts, draft_tokens.shape[-1], spec_metadata=spec_metadata + ) + return accepted, num_accepted + + def target_logits(self, hidden_states, attn_metadata, spec_metadata, lm_head: nn.Module): + """The verify rows' gathered fp32 target logits (N3 P-gid + P-cat), or None for the logits processor. + + On a decode-only, non-ragged step every row is a verify row and ``gather_ids`` is ``arange(num_tokens)`` (each + generation request extends it with the next ``1 + D`` row indices), so the rows are a slice, not a gather. + The processor's head is the column-parallel BF16 GEMM, an all-gather along the vocab (NCCL, then a concat) + and ``.float()``; here the same all-gather runs along dim 0 (no concat) and one strided copy writes the + rank-major shards into the fp32 output: the same bytes, an exact widening. Shapes and host state only, so + the decision is fixed per captured graph and the same on every rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + if ( + N3P_DISABLED + or spec_metadata is None + or getattr(spec_metadata, "is_ragged_verify", False) + ): + return None + if attn_metadata.num_contexts != 0 or spec_metadata.gather_ids is None: + return None + rows = hidden_states.shape[0] + if spec_metadata.gather_ids.shape[0] != rows or rows == 0: + return None + if ( + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or getattr(lm_head, "bias", None) is not None + ): + return None + shard = lm_head.apply_linear(hidden_states, None) + _n3p_log_once() + m3_report.work("N3P", f"{rows} verify rows") + tp = lm_head.tp_size + if tp == 1: + return shard.float() + # N3 A1: one launch for the all-gather, the concat and the cast. The decision depends on the rows and the + # set-up only (the same on every rank, fixed per captured graph). + if self._a1 is not None: + if rows <= A1_MAX_ROWS and _allgather_cast.is_compiled( + rows, shard.shape[-1], tp, 0, shard.device + ): + _a1_log_once( + "taken", + "[M3 dispatch] n3_a1 allgather_cast taken: verify logits gathered in one launch", + ) + m3_report.work("A1", f"{rows} verify rows, world {tp}") + if int(self.max_draft_len) != 3: + m3_report.work( + "A1_K7", f"draft length {int(self.max_draft_len)}, {rows} verify rows" + ) + return _allgather_cast.allgather_cast(shard, **self._a1) + _a1_log_once( + rows, + f"[M3 dispatch] n3_a1 allgather_cast not taken: {rows} verify rows (compiled: verify rows up to " + f"{A1_MAX_ROWS}); NCCL all-gather", + ) + gathered = allgather(shard, lm_head.mapping, dim=0) + vocab_shard = shard.shape[-1] + out = torch.empty((rows, tp * vocab_shard), dtype=torch.float32, device=shard.device) + out.view(rows, tp, vocab_shard).copy_(gathered.view(tp, rows, vocab_shard).transpose(0, 1)) + return out + + def _prepare_next_new_tokens( + self, + accepted_tokens, + next_draft_tokens, + batch_indices_cuda, + batch_size, + num_accepted_tokens, + ): + """The next step's input tokens in one launch (N3 A2), else the in-tree index + concat.""" + batch_idx = batch_indices_cuda[:batch_size] + if ( + not N3A_DISABLED + and next_draft_tokens.dim() == 2 + and next_draft_tokens.shape[0] == batch_size + and _strict_accept.supports_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + and _strict_accept.is_compiled_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + ): + m3_report.work("N3A", f"next-step input tokens, batch {batch_size}") + return _strict_accept.spec_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + return super()._prepare_next_new_tokens( + accepted_tokens, next_draft_tokens, batch_indices_cuda, batch_size, num_accepted_tokens + ) + + +def build_dspark( + shell, model_config: ModelConfig +) -> Tuple[ModelConfig, nn.Module, M3DSparkWorker, bool]: + """The drafter config, the drafter and the worker of one-model standalone DSpark (mirrors + ``SpecDecOneEngineForCausalLM`` for an external drafter).""" + spec = model_config.spec_config + if not is_standalone_dspark(spec): + raise ValueError("build_dspark serves the standalone DSpark drafter only") + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, **external_drafter_config_kwargs(model_config, spec) + ) + # The drop is safe only because draft_config owns its QuantConfig: from_pretrained reads the drafter checkpoint + # and takes no quant_config from the target. Never pass the target's quant_config in here. + keep_drafter_kv_dtype(draft_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = get_draft_model(model_config, draft_config, shell.lm_head, shell.model) + # A parallel drafter (DSpark) projects its block rows through the target's logits processor, as the one-engine + wrapped = install_draft_oneshot_allreduce(draft, DRAFT_AR_ONESHOT_MAX_ROWS) + logger.info( + f"[M3 prebuild] S9b-VR-a: {wrapped} drafter row-parallel all-reduce(s) take ONESHOT up to " + f"{DRAFT_AR_ONESHOT_MAX_ROWS} rows" + + ("" if DRAFT_AR_ONESHOT_MAX_ROWS else " (off: TRTLLM_M31_DISABLE_S9BVR)") + ) + # construction sets it. + draft.logits_processor = shell.logits_processor + worker = M3DSparkWorker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + logger.info( + f"M3.1 target DSpark: {type(worker).__name__} over {type(draft).__name__}, K={spec.max_draft_len}, " + f"separate draft KV cache={use_separate}" + ) + return draft_config, draft, worker, use_separate + + +def load_dspark_draft_weights(shell, weights, weight_mapper=None) -> None: + """The drafter's own checkpoint, then the target's embedding and head (mirrors + ``SpecDecOneEngineForCausalLM.load_draft_weights`` for DSpark). + + The drafter borrows the target's embedding and head, and ``M3EmbedTokens.weight`` IS the target's table, so a + drafter checkpoint that ships either would load into a module the target owns or be shadowed by it: refused. + """ + borrowed = sorted( + k for k in weights.keys() if "embed_tokens." in k or k.split(".")[-2:-1] == ["lm_head"] + ) + if borrowed: + raise ValueError( + f"the DSpark drafter checkpoint ships tensors this target lends it (embedding / head): {borrowed[:4]}" + ) + shell.draft_model.load_weights(weights=weights, weight_mapper=weight_mapper) + shell.draft_model.load_weights_from_target_model(shell) + if shell.draft_model.lm_head is not shell.lm_head or ( + shell.draft_model.draft_model_full.model.embed_tokens is not shell.model.embed_tokens + ): + raise RuntimeError("the DSpark drafter does not borrow this target's embedding and head") + own = sorted( + n for n, _ in shell.named_parameters() if not n.startswith(("model.w.", "draft_model.")) + ) + logger.info(f"M3.1 target DSpark: shell parameters outside the core and the drafter: {own}") + + +def keep_drafter_kv_dtype(draft_config: ModelConfig) -> None: + """Drop any KV quantization from the drafter's config, so its context pool is allocated in its projection dtype. + + The one-engine construction copies the target's KV algo onto the drafter; on this target that is NVFP4, which + allocates a packed int8 pool the worker refuses to bind (it falls back to its private arena, and a reused + prefix's drafter K/V is lost). The cached quant modes are dropped with it, as the draft KV budget does. + """ + for quant in [draft_config.quant_config, *(draft_config.quant_config_dict or {}).values()]: + quant.kv_cache_quant_algo = None + quant.__dict__.pop("quant_mode", None) + quant.__dict__.pop("layer_quant_mode", None) + + +def draft_kv_override_applies(spec_config) -> bool: + """The target's FP8 draft-KV override is an Eagle3 need; a DSpark drafter keeps its BF16 context pool.""" + return not is_standalone_dspark(spec_config) + + +__all__ = [ + "M3DSparkWorker", + "M3EmbedTokens", + "build_dspark", + "draft_kv_override_applies", + "is_standalone_dspark", + "keep_drafter_kv_dtype", + "load_dspark_draft_weights", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/dspark_s4.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/dspark_s4.py new file mode 100644 index 000000000000..9e7fe155a72a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/dspark_s4.py @@ -0,0 +1,765 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""S4: the catalog's one-launch Markov chain (``catalog/spec_decode/dspark_markov_chain``) in this target's DSpark +worker (each M3.1 target keeps its own copy of this glue: targets share only the catalog). + +``S4WorkerMixin`` goes before ``DSparkWorker`` in the target's worker. For a generation step it serves (switch not +set, 1 <= generation requests <= ``MAX_FAST_BATCH``, greedy drafting, no d2t / guided decoder / DFlash-2 path, a +vocab-sharded borrowed head): + +- the draft head computes this rank's vocab shard of the block logits (the head's local GEMM, no gather) and copies + it into a padded buffer whose tail holds -inf (the step's logits processor, swapped in for the step); +- ``_refine_block_logits`` runs the catalog op over the shards in one launch and keeps the tokens; +- ``sample_draft_tokens`` returns them. + +The tokens are bitwise the in-tree chain's on the gathered logits: the gathered logits are the concatenation of the +same per-rank GEMMs, and the op reproduces the chain's bf16 bias, bf16 add and first-maximum argmax. When the op +does not support a step after the logits were sharded, the in-tree sharded path (``vocab_slice`` + the TP-aware +argmax) runs instead, with the same tokens. + +Setup (``prebuild_s4``, from the worker's ``prebuild``, outside capture, on every rank): the padded ``markov_w2`` +shard, one padded logits buffer per (batch, block), the exchange workspace (collective) and the compiles. + +Every routing decision is Python, so it is fixed into each decode graph when the graph is captured: +``TRTLLM_M31_DRAFT_DISABLE_S4`` is read once, at import, not a switch that can be toggled at run time. The set-up +decision must be the same on every rank (the workspace is collective); it follows from the config and the +environment, and ``prebuild_s4`` checks that the ranks agree before the collective. + +Gate only: with ``TRTLLM_M31_DSPARK_GATE_DUMP=

`` set at prebuild (eager runs), each served step also runs the +in-tree chain on the gathered logits and logs both token sets per rank to ``/rank.jsonl``. It synchronizes; +never set it in a perf arm. +""" + +from __future__ import annotations + +import atexit +import json +import os +from typing import Dict, Optional, Tuple + +import torch +import torch.nn.functional as F +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_bias as markov_bias, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_chain as chain, +) +from tensorrt_llm._torch.models.modeling_speculative import dspark_markov_step_bias +from tensorrt_llm.logger import logger + +from .kernels import m3_report +from .m31_switches import m31_disabled +from .ops.spec.m31_markov_epilogue import m31_markov_chain_fused, markov_epilogue_op + +MARKOV_RANK = 256 +# S4CAP12: generation steps of up to 12 requests take the chain (the kernel takes up to 16). Measured on VR200 TP2 +# (net of the S4-logits rebuild): it saves 115 / 42 us per step over the in-tree chain +# at 8 / 12 requests and loses 46.5 at 16. Kill switch TRTLLM_M31_DISABLE_S4CAP12 (back to 8 requests), read at import. +S4CAP12 = not m31_disabled("S4CAP12") +MAX_FAST_BATCH = 12 if S4CAP12 else 8 # capacity: measured (above) +DISABLE_ENV = "TRTLLM_M31_DRAFT_DISABLE_S4" +S4_COUNT_LOG_EVERY = 1000 # generation steps between the per-rank "served x/y" engagement lines +GATE_DUMP_ENV = "TRTLLM_M31_DSPARK_GATE_DUMP" # gate-only: a directory; never set in a perf arm (it synchronizes) + +# TAILBK_SPEC (draft-tail bookkeeping, DSpark-worker sites): bookkeeping launches +# of the draft tail removed exactly. Here, site C: the chain op's three read-only accepted inputs (ones, arange, +# zeros; 3 launches per step) come from a per-shape cache filled at precompile. Site H: the target's KV-length rewind +# after the draft forward (kv_lens[nc:bs] -= rewind; clamp_(min=0): 2 launches per step) runs in the chain's launch +# (its rewind path, after the grid-dependency wait; nothing between the chain and the stock rewind reads kv_lens). +# Site S4L (S4-logits, the advanced-sampling contract path): the biased block logits' glue around the bias GEMM -- the +# prevs cat / casts, get_masked_input_and_mask, F.embedding, masked_fill, the two widening copies and the add (14 +# launches per step) -- in two launches (catalog dspark_markov_bias; the GEMM unchanged). +# Bitwise the same tokens, kv_lens and biased logits. Default ON; kill switch TRTLLM_M31_DISABLE_TAILBK_SPEC=1, read +# at import. +TAILBK_SPEC = not m31_disabled("TAILBK_SPEC") +# S4FE: the steps the chain declines (above MAX_FAST_BATCH requests; the agg TP2 +# c46 / c56 / c64 steps) run the in-tree full-vocab Markov chain with its elementwise tail fused per position +# (ops/spec/m31_markov_epilogue.py): the same embedding and F.linear, then one C++ pass for the add, the corrected +# write, torch's argmax and the next embedding. Bitwise the in-tree corrected logits. VR TP2 K3: the +# in-tree chain costs 357 / 401 / 444 us per step at 46 / 56 / 64 requests, about 30-45 us of it the GEMM. +# Default ON; kill switch TRTLLM_M31_DISABLE_S4FE=1, read at import. +S4FE = not m31_disabled("S4FE") +# Zero uniform CUDA-graph padding rows in place. Real rows are untouched; padding rows become +0.0. +# Ragged verify layouts retain the stock path. Default ON; kill switch TRTLLM_M31_DISABLE_TAIL16_G2=1. +TAIL16_G2 = not m31_disabled("TAIL16_G2") +_TAIL16_G2_LOGGED: set = set() # (rows, rows per request) already logged +# Copy block draft probabilities into request slots. Default ON; kill switch TRTLLM_M31_DISABLE_TAIL16_G3=1. +TAIL16_G3 = not m31_disabled("TAIL16_G3") +_TAIL16_G3_LOGGED: set = set() +# TAIL_BK3: post-layer bookkeeping in fewer launches, bitwise the stock ops: +# (B) the S4FE position-0 Markov embedding as one gather (ops/spec/m31_markov_epilogue.py), (C) the block sampler's +# per-row parameters and Philox state as one kernel (_block_sampling_params), (D) no int32 -> int64 -> int32 round trip +# of the block tokens. Default ON; kill switch TRTLLM_M31_DISABLE_TAIL_BK3=1, read at import. +TAIL_BK3 = not m31_disabled("TAIL_BK3") +_TAIL_BK3_LOGGED: set = set() + +# Read once, at import: every rank and every step of a process sees the same setting. Any value but "" or "0" turns +# the item off, as the targets' m31_switches reads its switches. +_DISABLED = os.environ.get(DISABLE_ENV, "").strip() not in ("", "0") + + +def disabled() -> bool: + return _DISABLED + + +class _S4State: + def __init__(self, mapping, rows: int, shard: int, offset: int, w1: torch.Tensor, w2s: torch.Tensor, + block: int, ws: dict) -> None: # fmt: skip + self.mapping = mapping + self.ws = ws + self.rows = rows + self.shard = shard + self.offset = offset + self.w1 = w1 + self.w2s = w2s + self.block = block + self.bufs: Dict[Tuple[int, int], torch.Tensor] = {} + + def buffer(self, batch: int) -> torch.Tensor: + return self.bufs[(batch, self.block)] + + +# S4-logits bias form: True = all K positions' bias in one GEMM over B*K rows (S4LC: bitwise the per-step GEMMs); +# False = the in-tree per-step loop over the chain's tokens (exact by construction). +S4_LOGITS_BATCHED = True + + +def _biased_shard_logits( + shard_logits: torch.Tensor, + first_prev: torch.Tensor, + tokens: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + fp32: bool, +) -> torch.Tensor: + """The in-tree chain's biased block logits on this rank's vocab shard, given the chain's tokens: step k adds + w2 @ w1[prev_k] (dspark_markov_step_bias) with prev_0 = the anchor and prev_k = token k-1, cast to the in-tree + logits dtype: fp32 under the S4 fp32 sum (LogitsProcessor returns logits.float()), else the shard's bf16.""" + if ( + fp32 + and S4_LOGITS_BATCHED + and TAILBK_SPEC + and markov_bias.supports(shard_logits, first_prev, tokens, w1) + ): + # TAILBK_SPEC S4L: the same prevs, embedding rows, GEMM and fp32 sum, in two launches around the GEMM. + B, K, N = shard_logits.shape + bias = F.linear(markov_bias.prev_embeddings(first_prev, tokens, w1), w2_shard) + return markov_bias.widen_add(shard_logits, bias.view(B, K, N)) + B, K, N = shard_logits.shape + base = shard_logits.float() if fp32 else shard_logits + prevs = torch.cat([first_prev.view(B, 1), tokens[:, : K - 1].long()], dim=1) + if S4_LOGITS_BATCHED: + bias = dspark_markov_step_bias(prevs.reshape(-1), w1, w2_shard).view(B, K, N) + return base + bias.to(base.dtype) + out = torch.empty_like(base) + for k in range(K): + out[:, k] = base[:, k] + dspark_markov_step_bias(prevs[:, k], w1, w2_shard).to(base.dtype) + return out + + +class _ShardedDraftLogits(nn.Module): + """The step's draft logits processor: this rank's vocab shard of ``lm_head`` over the gathered block rows, + copied into the padded buffer; any other call goes to the drafter's own processor.""" + + def __init__(self, inner: nn.Module, state: _S4State, batch: int) -> None: + super().__init__() + self.inner = inner + self._state = [state] + self.batch = batch + + def forward(self, hidden_states, lm_head, attn_metadata, return_context_logits: bool = False): + state = self._state[0] + if hidden_states.dim() != 2 or hidden_states.shape[0] != self.batch * state.block: + return self.inner(hidden_states, lm_head, attn_metadata, return_context_logits) + logits = lm_head.apply_linear(hidden_states, None) + buf = state.buffer(self.batch) + buf[:, :, : state.shard].copy_(logits.view(self.batch, state.block, state.shard)) + return logits + + +def _head_is_sharded(lm_head: nn.Module, vocab: int, tp: int) -> bool: + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + return ( + tp > 1 + and getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and getattr(lm_head, "padding_size", 0) == 0 + and getattr(lm_head, "bias", None) is None + and lm_head.out_features * tp == vocab + and lm_head.weight.dtype == torch.bfloat16 + ) + + +def _capacity_reason(shard: int) -> str: + """The decline reason when no padded shard fits the chain kernel: its CTAs keep their markov_w2 rows in shared + memory up to SMEM_ROW_BUDGET rows and read the rest from global memory (S4STREAM), up to MAX_ROWS_PER_CTA rows per + CTA, and must all be co-resident (the TP2 shard of 100,032 rows is padded to 101,376 = 144 CTAs x 704 rows).""" + kern = chain._kernel_module() + budget = int(getattr(kern, "MAX_ROWS_PER_CTA", getattr(kern, "SMEM_ROW_BUDGET", 0))) + return ( + f"vocab shard {shard} rows past the chain kernel's capacity of {chain.MAX_GRID} co-resident CTAs x {budget} " + f"markov_w2 rows = {chain.MAX_GRID * budget} rows" + ) + + +class S4WorkerMixin: + """Mixed into the target's ``DSparkWorker`` subclass (before it in the bases).""" + + _block_tokens_keep_int32: bool = TAIL_BK3 + _s4: Optional[_S4State] = None + _s4_step: Optional[_S4State] = None + _s4_tokens: Optional[torch.Tensor] = None + _s4_step_meta: Optional[Tuple[int, int, object]] = None + _s4_dump_step: int = 0 + _s4_dump_dir: Optional[str] = None + _s4_draft: Optional[nn.Module] = None + _s4_served: int = 0 + _s4_gen_steps: int = 0 + _s4_step_attn = ( + None # the step's attention metadata (the KV-length rewind's kv_lens), while S4 serves it + ) + _tailbk_rewound: bool = ( + False # TAILBK_SPEC H: this step's chain launch applied the KV-length rewind + ) + _tailbk_h_logged: bool = False + _tailbk_s4l_logged: bool = False + _tailbk_c_logged: bool = False + + def prebuild_s4(self, lm_head: nn.Module, draft_model) -> bool: + """Set up the one-launch chain for this worker; False (and nothing set up) when it does not apply.""" + self._s4 = None + self._s4_dump_dir = os.environ.get(GATE_DUMP_ENV) or None + if disabled(): + logger.info(f"M3.1 DSpark S4: off ({DISABLE_ENV} set); the in-tree Markov chain runs") + return False + if not getattr(draft_model, "has_markov_head", False): + return self._s4_decline("the drafter has no Markov head") + w1 = draft_model.markov_w1 + w2 = draft_model.markov_w2 + mapping = self.mapping + tp = mapping.tp_size + vocab = w1.shape[0] + # Config and weights only, the same on every rank: no rank goes on to the collective agreement below alone. + off = [ + why + for why, bad in ( + (f"markov_w1 {w1.dtype} / markov_w2 {w2.dtype}, not bf16", + w1.dtype != torch.bfloat16 or w2.dtype != torch.bfloat16), + (f"Markov rank {w1.shape[1]}, not {MARKOV_RANK}", w1.shape[1] != MARKOV_RANK), + (f"markov_w2 {tuple(w2.shape)} is not markov_w1's {tuple(w1.shape)}", + tuple(w2.shape) != tuple(w1.shape)), + ("attention DP", bool(mapping.enable_attention_dp)), + ("the draft head is not a vocab-sharded, unpadded, bias-free bf16 column-parallel head", + not _head_is_sharded(lm_head, vocab, tp)), + ) + if bad + ] # fmt: skip + if off: + return self._s4_decline("; ".join(off)) + block = int(self.max_draft_len) + shard = vocab // tp + rows = chain.padded_rows(shard, MAX_FAST_BATCH, w1.device) + local = rows != 0 and block <= chain.MAX_BLOCK + if not self._s4_ranks_agree(local): + if rows == 0: + return self._s4_decline(_capacity_reason(shard)) + if block > chain.MAX_BLOCK: + return self._s4_decline(f"draft length {block} past the chain's {chain.MAX_BLOCK}") + return self._s4_decline("off on another rank") + offset = mapping.tp_rank * shard + w2s = chain.pad_w2_shard(w2[offset : offset + shard].contiguous(), rows) + state = _S4State( + mapping, rows, shard, offset, w1, w2s, block, chain.workspace_args(mapping) + ) + # One padded buffer for every batch: batch b takes its leading b rows (contiguous, at fixed addresses). + whole = chain.pad_logits_buffer(MAX_FAST_BATCH, block, rows, w1.device) + for batch in range(1, MAX_FAST_BATCH + 1): + state.bufs[(batch, block)] = whole[:batch] + # S4 fp32 sum: the chain compares base + bias as the in-tree chain does, the unrounded fp32 sum (its logits are + # fp32 from LogitsProcessor's .float()). Kill switch TRTLLM_M31_DISABLE_S4_FP32SUM (bf16-rounded sums). + self._s4_fp32 = chain.enable_fp32_sum(not m31_disabled("S4_FP32SUM")) + # S4B8: batch-1 calls take the chain's batch-1 kernel (bitwise the same tokens; faster at one request only). + # Kill switch TRTLLM_M31_DISABLE_S4B8, read at import. + s4b8 = chain.enable_batch1_variant(not m31_disabled("S4B8")) + self._s4b8 = bool(s4b8) + # TAILBK_SPEC site C: the chain's read-only accepted inputs from a per-shape cache (bitwise; see TAILBK_SPEC). + tailbk_c = chain.enable_cached_inputs(TAILBK_SPEC) + # S4-logits: under advanced (rejection) draft sampling the chain still computes the greedy chain's tokens; the + # biased block logits are rebuilt from them (_biased_shard_logits) and the stock sampler gathers and samples + # them as it would the in-tree chain's. ON by default with MCAG off (the TP8 recipe: the stock all-gather of the + # shard logits keeps two runs of the same requests identical). Kill switch TRTLLM_M31_DISABLE_S4_LOGITS. + self._s4_logits_on = not m31_disabled("S4_LOGITS") + chain.precompile(mapping, rows, w1, w2s, offset, blocks=(block,), max_batch=MAX_FAST_BATCH) + self._s4 = state + self._s4_draft = draft_model + atexit.register(self._s4_log_count) + # The engagement line: one per rank, at server start. + logger.info( + f"M3.1 DSpark S4: one-launch Markov chain set up on rank {mapping.tp_rank}/{tp} " + f"(shard {shard} rows padded to {rows}, K={block}, batches 1..{MAX_FAST_BATCH}; fp32 sum " + f"{'on' if self._s4_fp32 else 'off'}; S4-logits {'on' if self._s4_logits_on else 'off'}; " + f"S4B8 batch-1 kernel {'on' if s4b8 else 'off'}; TAILBK_SPEC cached inputs {'on' if tailbk_c else 'off'})" + ) + return True + + def _s4_decline(self, reason: str) -> bool: + """The S4 decline line (one per rank, at prebuild), then False: the in-tree Markov chain runs, with the + TAILBK_SPEC sites the chain launch carries (C, H, S4L) not taken.""" + mapping = getattr(self, "mapping", None) + logger.info( + f"M3.1 DSpark S4: one-launch Markov chain off on rank {getattr(mapping, 'tp_rank', None)}/" + f"{getattr(mapping, 'tp_size', None)} ({reason}); the in-tree Markov chain runs (TAILBK_SPEC chain sites " + "C / H / S4L not taken)" + ) + return False + + def _s4_ranks_agree(self, local: bool) -> bool: + """``local`` if every rank of the TP group decided the same; raises when they differ (a rank going on into + the collective workspace alone would hang its peers).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + comm = _get_mnnvl_workspace_comm(self.mapping) + all_on = _mnnvl_workspace_all_succeeded(comm, local) + all_off = _mnnvl_workspace_all_succeeded(comm, not local) + if not (all_on or all_off): + raise RuntimeError( + "m31 DSpark S4: the TP ranks disagree on setting up the one-launch Markov chain" + ) + return all_on + + def _s4_serves(self, num_gens: int, spec_metadata, draft_model) -> bool: + return self._s4_mode(num_gens, spec_metadata, draft_model) is not None + + def _s4_mode(self, num_gens: int, spec_metadata, draft_model): + """ "tokens" (greedy drafting: the chain's tokens are the draft), "logits" (advanced draft sampling with + S4-logits on: the chain's corrected logits go to the stock sampler), or None (the in-tree chain runs).""" + if not self._s4_common(num_gens, draft_model): + return None + if not getattr(spec_metadata, "wants_advanced_draft_sampling", False): + return "tokens" + return "logits" if getattr(self, "_s4_logits_on", False) else None + + def _s4_common(self, num_gens: int, draft_model) -> bool: + return ( + self._s4 is not None + and 1 <= num_gens <= MAX_FAST_BATCH + and getattr(self, "guided_decoder", None) is None + and getattr(self, "_d2t", None) is None + and getattr(draft_model, "lilicorr", None) is None + and not getattr(draft_model, "has_candidate_selector", False) + ) + + def _host_step_hook(self, spec_metadata) -> None: + """The engagement counter (host only, no CUDA): called by the spec metadata's prepare() on every iteration, + outside any captured graph, it evaluates the same eligibility the captured step was routed by and counts + the generation steps the chain serves. One line per rank every S4_COUNT_LOG_EVERY generation steps and at + exit: "M3.1 DSpark S4: served / generation steps". The counts start at server start, so they include + warmup and padding steps: read engagement as the difference of two lines logged after the server is ready.""" + if self._s4 is None: + return + num_gens = int(getattr(spec_metadata, "num_generations", 0) or 0) + if num_gens <= 0: + return + self._s4_gen_steps += 1 + self._s4_served += int(self._s4_serves(num_gens, spec_metadata, self._s4_draft)) + if self._s4_gen_steps % S4_COUNT_LOG_EVERY == 0: + self._s4_log_count() + + def _s4_log_count(self) -> None: + if self._s4 is not None and self._s4_gen_steps: + logger.info( + f"M3.1 DSpark S4: served {self._s4_served}/{self._s4_gen_steps} generation steps on rank " + f"{self.mapping.tp_rank}" + ) + + def _forward_impl(self, input_ids, position_ids, hidden_states, logits, attn_metadata, spec_metadata, draft_model, + resource_manager=None): # fmt: skip + num_gens = attn_metadata.num_seqs - attn_metadata.num_contexts + mode = self._s4_mode(num_gens, spec_metadata, draft_model) + served = mode is not None + if self._s4_dump_dir and not torch.cuda.is_current_stream_capturing(): + # The gate's independent denominator: every generation step, its eligibility by batch and sampling + # alone, and whether the chain served it. + eligible = 1 <= num_gens <= MAX_FAST_BATCH and not getattr( + spec_metadata, "wants_advanced_draft_sampling", False + ) + self._s4_dump_line({"kind": "step", "num_gens": num_gens, "eligible": bool(eligible), + "served": bool(served)}) # fmt: skip + if not served: + return super()._forward_impl(input_ids, position_ids, hidden_states, logits, attn_metadata, + spec_metadata, draft_model, resource_manager) # fmt: skip + inner = draft_model.logits_processor + draft_model.logits_processor = _ShardedDraftLogits(inner, self._s4, num_gens) + self._s4_step = self._s4 + self._s4_step_mode = mode + self._s4_step_meta = (attn_metadata.num_contexts, attn_metadata.num_seqs, spec_metadata) + self._s4_step_attn = attn_metadata + try: + return super()._forward_impl(input_ids, position_ids, hidden_states, logits, attn_metadata, + spec_metadata, draft_model, resource_manager) # fmt: skip + finally: + draft_model.logits_processor = inner + self._s4_step = None + self._s4_step_mode = None + self._s4_tokens = None + self._s4_step_meta = None + self._s4_step_attn = None + self._tailbk_rewound = False + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + state = self._s4_step + if state is not None and gen_logits.dim() == 3 and gen_logits.shape[-1] == state.shard: + base = state.buffer(gen_logits.shape[0]) + if ( + chain.supports(base, state.w1, state.w2s) + and getattr(self, "_s4_step_mode", None) == "logits" + ): + # S4-logits: the chain's tokens are the in-tree greedy chain's (bitwise); the biased shard logits are + # rebuilt from them with the in-tree bias arithmetic and handed to the stock advanced sampler. + first_prev = inputs["first_prev_tokens"].long() + tokens = self._s4_chain(state, base, first_prev, spec_metadata) + biased = _biased_shard_logits( + gen_logits, + first_prev, + tokens, + state.w1, + state.w2s[: state.shard], + getattr(self, "_s4_fp32", True), + ) + self._s4l_log(gen_logits, first_prev, tokens, state.w1) + if self._s4_dump_dir and not torch.cuda.is_current_stream_capturing(): + self._s4_logits_gate_dump( + draft_model, gen_logits, inputs, spec_metadata, biased + ) + return biased + if chain.supports(base, state.w1, state.w2s): + tokens = self._s4_chain( + state, base, inputs["first_prev_tokens"].long(), spec_metadata + ) + self._s4_tokens = tokens + if self._s4_dump_dir and not torch.cuda.is_current_stream_capturing(): + self._s4_gate_dump(draft_model, gen_logits, inputs, spec_metadata, tokens) + return gen_logits + if S4FE and state is None: + fused = self._s4fe(draft_model, gen_logits, inputs, spec_metadata) + if fused is not None: + return fused + return super()._refine_block_logits(draft_model, gen_logits, inputs, spec_metadata) + + @staticmethod + def _store_block_draft_probs(draft_probs, slot_ids, probs): + """TAIL16_G3: the stock store as one vectorized row copy; any other layout keeps the stock index_put.""" + loaded = markov_epilogue_op() if TAIL16_G3 else None + op = ( + getattr(torch.ops.trtllm, "minimax_m31_scatter_rows", None) + if loaded is not None + else None + ) + if ( + op is not None + and draft_probs is not None + and draft_probs.is_cuda + and probs.is_cuda + and slot_ids.is_cuda + and draft_probs.dtype == torch.float32 + and probs.dtype == torch.float32 + and draft_probs.dim() == 3 + and probs.dim() == 3 + and draft_probs.stride(2) == 1 + and draft_probs.shape[1] >= probs.shape[1] + and draft_probs.shape[2] >= probs.shape[2] + and slot_ids.dtype == torch.int64 + and slot_ids.dim() == 1 + and slot_ids.shape[0] == probs.shape[0] + and draft_probs.device == probs.device == slot_ids.device + and probs.shape[0] * probs.shape[1] <= 65535 + and probs.shape[2] <= 2147483647 + ): + op(draft_probs, slot_ids.contiguous(), probs.contiguous()) + key = (int(probs.shape[0]), int(probs.shape[1])) + if key not in _TAIL16_G3_LOGGED: + _TAIL16_G3_LOGGED.add(key) + logger.info(f"[M3.1] TAIL16_G3 taken at {key[0]} requests ({key[1]} rows)") + return + from tensorrt_llm._torch.speculative.interface import SpecWorkerBase + + SpecWorkerBase._store_block_draft_probs(draft_probs, slot_ids, probs) + + def _block_sampling_params(self, spec_metadata, num_contexts: int, batch_size: int, K: int): + """TAIL_BK3 (C): the stock per-row repeat_interleave of the request sampling parameters and the Philox state + (_rng_state_per_request, repeat=K) in one launch; any other layout keeps the stock ops.""" + from tensorrt_llm._torch.speculative.interface import _RNG_SLOT_SPAN + + loaded = markov_epilogue_op() if TAIL_BK3 else None + op = ( + getattr(torch.ops.trtllm, "minimax_m31_expand_block_sampling", None) + if loaded is not None + else None + ) + bufs = tuple( + getattr(spec_metadata, name, None) + for name in ( + "request_temperatures", + "request_top_ks", + "request_top_ps", + "request_min_ps", + "request_seeds", + "request_offsets", + ) + ) + dtypes = ( + torch.float32, + torch.int32, + torch.float32, + torch.float32, + torch.int64, + torch.int64, + ) + if ( + op is not None + and K >= 1 + and batch_size > num_contexts + and all(b is not None and b.is_cuda and b.dim() == 1 and b.stride(0) == 1 for b in bufs) + and all(b.dtype == dt for b, dt in zip(bufs, dtypes)) + and len({b.device for b in bufs}) == 1 + ): + slot = int(spec_metadata.rng_draft_slot(0)) + out = op( + *(b[num_contexts:batch_size] for b in bufs), + int(K), + slot * _RNG_SLOT_SPAN, + _RNG_SLOT_SPAN, + ) + key = (int(batch_size - num_contexts), int(K)) + if key not in _TAIL_BK3_LOGGED: + _TAIL_BK3_LOGGED.add(key) + logger.info( + f"[M3.1] TAIL_BK3 sampling params taken at {key[0]} requests (K{key[1]})" + ) + return tuple(out) + return super()._block_sampling_params(spec_metadata, num_contexts, batch_size, K) + + @staticmethod + def _zero_padding_rows( + logits, spec_metadata, num_contexts, batch_size, rows_per_request, *, packed_verify=False + ): + """TAIL16_G2: the stock zeroing of the CUDA-graph padding requests' rows, in place and in one launch, for the + uniform layout; every other case (ragged verify, a layout mismatch, the op or switch off) is the stock code.""" + slot_ids = getattr(spec_metadata, "batch_slot_ids", None) + dummy = getattr(spec_metadata, "dummy_slot_row", 0) + num_gens = batch_size - num_contexts + op = markov_epilogue_op() if TAIL16_G2 else None + if ( + op is not None + and slot_ids is not None + and dummy > 0 + and num_gens > 0 + and not (packed_verify and spec_metadata.is_ragged_verify) + and logits.dim() == 2 + and logits.shape[0] == num_gens * rows_per_request + and logits.dtype == torch.float32 + and logits.stride(1) == 1 + and slot_ids.dtype == torch.int64 + ): + torch.ops.trtllm.minimax_m31_zero_padding_rows( + logits, + slot_ids[num_contexts:batch_size].contiguous(), + int(dummy), + int(rows_per_request), + ) + key = (int(logits.shape[0]), int(rows_per_request)) + if key not in _TAIL16_G2_LOGGED: + _TAIL16_G2_LOGGED.add(key) + logger.info(f"[M3.1] TAIL16_G2 taken at {key[0]} rows ({key[1]} per request)") + return logits + from tensorrt_llm._torch.speculative.interface import SpecWorkerBase + + return SpecWorkerBase._zero_padding_rows( + logits, + spec_metadata, + num_contexts, + batch_size, + rows_per_request, + packed_verify=packed_verify, + ) + + def _s4fe(self, draft_model, gen_logits, inputs, spec_metadata): + """S4FE: the in-tree chain's corrected logits with the fused tail, or None (the in-tree path runs): no Markov + head, logits not the full vocab or TP-sharded (the in-tree argmax would gather), or outside the op.""" + if not getattr(draft_model, "has_markov_head", False) or gen_logits.dim() != 3: + return None + if gen_logits.shape[-1] != draft_model.markov_w2.shape[0] or self._draft_logits_are_sharded( + gen_logits[:, 0], spec_metadata + ): + return None + out = m31_markov_chain_fused( + gen_logits, inputs["first_prev_tokens"], draft_model.markov_w1, draft_model.markov_w2 + ) + key = (tuple(gen_logits.shape[:2]), out is not None) + logged = self.__dict__.setdefault("_s4fe_logged", set()) + if key not in logged: + logged.add(key) + what = "taken" if out is not None else "stock" + logger.info( + f"[M3.1] S4FE {what} at {gen_logits.shape[0]} requests (K{gen_logits.shape[1]}) on rank " + f"{self.mapping.tp_rank}" + ) + return out + + def _s4_chain( + self, state: _S4State, base: torch.Tensor, first_prev: torch.Tensor, spec_metadata + ): + """The chain launch; under TAILBK_SPEC (site H) it also applies the step's deferred KV-length rewind when + the stock _apply_kv_rewind_after_draft would (a real step: not the eager warmup of a graph batch, whose + kv_lens is restored wholesale), and marks it done for that method.""" + rewind = self._tailbk_rewind_args(base.shape[0], spec_metadata) + self._s4_work(base.shape[0], rewind is not None) + if rewind is None: + tokens = chain.dspark_markov_chain( + base, first_prev, state.w1, state.w2s, state.offset, **state.ws + ) + self._tailbk_c_log() + return tokens + kv_lens, amount, first = rewind + tokens = chain.dspark_markov_chain_rewind(base, first_prev, state.w1, state.w2s, state.offset, kv_lens, + amount, first, **state.ws) # fmt: skip + self._tailbk_rewound = True + self._tailbk_c_log() + if not self._tailbk_h_logged: + self._tailbk_h_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC H: KV-length rewind in the Markov chain launch " + f"on rank {self.mapping.tp_rank}" + ) + return tokens + + def _s4_work(self, batch: int, rewind: bool) -> None: + """Gate e: one chain launch's work counters (S4, and the variants and TAILBK_SPEC sites it carries).""" + m3_report.work("S4", f"Markov chain launch, batch {batch}, rank {self.mapping.tp_rank}") + if getattr(self, "_s4_fp32", False): + m3_report.work("S4_FP32SUM", f"batch {batch}") + if getattr(self, "_s4b8", False) and batch == 1: + m3_report.work("S4B8", "batch-1 chain kernel") + if TAILBK_SPEC and (rewind or getattr(chain, "cached_inputs_taken", lambda: False)()): + m3_report.work("TAILBK_SPEC", "chain site " + ("H (KV rewind) + C" if rewind else "C")) + + def _tailbk_c_log(self) -> None: + """The TAILBK_SPEC C engagement line: once per rank, after the first chain call that took its accepted inputs + from the precompile cache (host only; a captured step logs it while it is captured).""" + if self._tailbk_c_logged or not getattr(chain, "cached_inputs_taken", lambda: False)(): + return + self._tailbk_c_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC C: chain accepted inputs from precompile cache (taken) " + f"on rank {self.mapping.tp_rank}" + ) + + def _tailbk_rewind_args(self, batch: int, spec_metadata): + """(kv_lens, rewind amounts, first row) for the chain to apply, or None (the stock rewind runs).""" + if not TAILBK_SPEC or not getattr(self, "_kv_rewind_pending", False): + return None + amount = getattr(self, "_kv_rewind_amount", None) + attn = self._s4_step_attn + if amount is None or attn is None or not hasattr(attn, "kv_lens_cuda"): + return None + if spec_metadata.is_cuda_graph and not torch.cuda.is_current_stream_capturing(): + return None # the warmup: the stock method skips the rewind (kv_lens is restored wholesale) + nc, bs = self._kv_rewind_nc, self._kv_rewind_bs + if bs - nc != batch or amount.numel() != batch: + return None + kv_lens = attn.kv_lens_cuda + if not chain.supports_rewind(kv_lens, amount, nc): + return None + return kv_lens, amount, nc + + def _apply_kv_rewind_after_draft(self, attn_metadata, spec_metadata): + if self._tailbk_rewound: + # TAILBK_SPEC H: the chain launch applied it (see _s4_chain). + self._tailbk_rewound = False + self._kv_rewind_pending = False + return + super()._apply_kv_rewind_after_draft(attn_metadata, spec_metadata) + + def _s4l_log(self, shard_logits, first_prev, tokens, w1) -> None: + """The TAILBK_SPEC S4L engagement line: once per rank, on the first step whose biased logits took the + two-launch glue (host only; a captured step logs it while it is captured).""" + if self._tailbk_s4l_logged or not ( + getattr(self, "_s4_fp32", True) and S4_LOGITS_BATCHED and TAILBK_SPEC + ): + return + if markov_bias.supports(shard_logits, first_prev, tokens, w1): + self._tailbk_s4l_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC S4L: biased block logits glue in two launches " + f"on rank {self.mapping.tp_rank}" + ) + + def _s4_gate_dump(self, draft_model, shard_logits, inputs, spec_metadata, tokens) -> None: + """Gate-only (eager): the in-tree chain on the gathered logits of this same step, logged beside the + one-launch chain's tokens, one JSON line per step and rank.""" + from tensorrt_llm._torch.distributed import allgather + + full = allgather(shard_logits.contiguous(), self.mapping, dim=-1) + if getattr(self, "_s4_fp32", True): + full = full.float() # the in-tree chain's input: LogitsProcessor returns logits.float() + corrected = super()._refine_block_logits(draft_model, full, inputs, spec_metadata) + stock = torch.argmax(corrected, dim=-1) + num_contexts, num_seqs, meta = self._s4_step_meta or (0, tokens.shape[0], None) + ids = getattr(meta, "request_ids", None) + ids = list(ids[num_contexts:num_seqs]) if ids is not None else None + self._s4_dump_line({"kind": "tokens", "step": self._s4_dump_step, "request_ids": ids, "s4": tokens.tolist(), + "stock": stock.tolist(), "equal": bool(torch.equal(tokens.long(), stock))}) # fmt: skip + self._s4_dump_step += 1 + + def _s4_logits_gate_dump( + self, draft_model, shard_logits, inputs, spec_metadata, biased + ) -> None: + """Gate-only (eager): the in-tree chain on the gathered logits of this same step (full-vocab bias), compared + bitwise with S4-logits' gathered biased logits, one JSON line per step and rank.""" + from tensorrt_llm._torch.distributed import allgather + + full = allgather(shard_logits.contiguous(), self.mapping, dim=-1) + if getattr(self, "_s4_fp32", True): + full = full.float() # the in-tree chain's input: LogitsProcessor returns logits.float() + stock = super()._refine_block_logits(draft_model, full, inputs, spec_metadata) + ours = allgather(biased.contiguous(), self.mapping, dim=-1) + num_contexts, num_seqs, meta = self._s4_step_meta or (0, biased.shape[0], None) + ids = getattr(meta, "request_ids", None) + ids = list(ids[num_contexts:num_seqs]) if ids is not None else None + mism = int((ours != stock).sum()) + self._s4_dump_line({"kind": "logits", "step": self._s4_dump_step, "request_ids": ids, + "shape": list(ours.shape), "dtype": str(ours.dtype), "mismatch": mism, + "equal": mism == 0 and ours.dtype == stock.dtype, + "batched": S4_LOGITS_BATCHED}) # fmt: skip + self._s4_dump_step += 1 + + def _s4_dump_line(self, rec: dict) -> None: + rec["rank"] = self.mapping.tp_rank + path = os.path.join(self._s4_dump_dir, f"rank{self.mapping.tp_rank}.jsonl") + with open(path, "a") as f: + f.write(json.dumps(rec) + "\n") + + def sample_draft_tokens(self, logits, spec_metadata, batch_size, **kwargs): + tokens = self._s4_tokens + if tokens is not None and logits.dim() == 3 and tokens.shape == logits.shape[:2]: + self._s4_tokens = None + return tokens if tokens.dtype == torch.int32 else tokens.to(torch.int32) + return super().sample_draft_tokens(logits, spec_metadata, batch_size, **kwargs) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/__init__.py new file mode 100644 index 000000000000..91d4ff843d5e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Kernels of the MiniMax-M3 NVFP4 / sm_107 / tp4 target.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/__init__.py new file mode 100644 index 000000000000..4c6fc92cd49c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 decode embedding gather + first-layer RMSNorm as one CuTe DSL kernel (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py new file mode 100644 index 000000000000..78563b4db40d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fixed M3 H6144 gather/raw residual/RMS, reusing pinned K3 norm kernel. + +128 threads x 8 BF16 x 6 iterations covers the row. Unlike the original +K3 wrapper this target does not promise FlashInfer's bit-identical reduction. +""" + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass.experimental import primitives as prims + +NORM_THREADS = 128 +NORM_VEC = 8 +NORM_WARPS = 4 # capacity: fixed: the CTA's warps + + +@cute.kernel +def m3_embed_norm_kernel( + ids: cute.Tensor, # int32 or int64 [N] + table: cute.Tensor, # bf16 [V, H] + weight: cute.Tensor, # bf16 [H] + raw: cute.Tensor, # bf16 [N, H], out: the rows + out: cute.Tensor, # bf16 [N, H], out: the normed rows + vocab: cutlass.Int64, + eps: cutlass.Float32, + tv_layout: cute.Layout, + tiler_mn: cute.Shape, + hidden: cutlass.Constexpr[int], +): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + prims.griddepcontrol(prims.GridDepAction.WAIT) + + smem = cutlass.utils.SmemAllocator() + reduction_buffer = smem.allocate_tensor( + cutlass.Float32, cute.make_layout((1, NORM_WARPS)), byte_alignment=4 + ) + + tok = cutlass.Int64(ids[bidx]) + valid = (tok >= cutlass.Int64(0)) & (tok < vocab) + row = cutlass.Int64(cutlass.select_(valid, tok, cutlass.Int64(0))) + + gX = cute.local_tile(table, tiler_mn, (row, 0)) + gR = cute.local_tile(raw, tiler_mn, (bidx, 0)) + gY = cute.local_tile(out, tiler_mn, (bidx, 0)) + w_layout = cute.prepend(weight.layout, cute.make_layout((tiler_mn[0],), stride=(0,))) + gW = cute.local_tile(cute.make_tensor(weight.iterator, w_layout), tiler_mn, (0, 0)) + + copy_atom_load = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), table.element_type, num_bits_per_copy=128 + ) + copy_atom_store = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), out.element_type, num_bits_per_copy=128 + ) + thr_copy_X = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_W = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_O = cute.make_tiled_copy(copy_atom_store, tv_layout, tiler_mn).get_slice(tidx) + + tXgX = thr_copy_X.partition_S(gX) + tXrX = cute.make_fragment_like(tXgX) + tWgW = thr_copy_W.partition_S(gW) + tWrW = cute.make_fragment_like(tWgW) + tXrW = thr_copy_X.retile(tWrW) + tXgR = thr_copy_O.partition_D(gR) + tXgO = thr_copy_O.partition_D(gY) + tXrO = cute.make_fragment_like(tXgO) + + # The row (zero for an id outside [0, V)), straight to the snapshot slot; the weight. + tXrX.store(cute.zeros_like(tXrX, dtype=table.element_type)) + if valid: + cute.copy(copy_atom_load, tXgX, tXrX) + cute.copy(copy_atom_load, tWgW, tWrW) + cute.copy(copy_atom_store, tXrX, tXgR) + + # flashinfer's RMSNormKernel arithmetic, in its order (see the header). + x = tXrX.load().to(cutlass.Float32) + x_sq = x * x + sum_sq = x_sq.reduce(cute.ReductionOp.ADD, init_val=cutlass.Float32(0.0), reduction_profile=0) + for i in cutlass.range_constexpr(5): + sum_sq = sum_sq + cute.arch.shuffle_sync_bfly(sum_sq, offset=1 << i) + lane = cute.arch.lane_idx() + warp = cute.arch.warp_idx() + if lane == 0: + reduction_buffer[0, warp] = sum_sq + cute.arch.barrier() + total = cutlass.Float32(0.0) + if lane < NORM_WARPS: + total = reduction_buffer[0, lane] + for i in cutlass.range_constexpr(5): + total = total + cute.arch.shuffle_sync_bfly(total, offset=1 << i) + mean_sq = total / cutlass.Float32(hidden) + rstd = cute.math.rsqrt(mean_sq + eps, fastmath=True) + w = tXrW.load().to(cutlass.Float32) + y = x * rstd * (w + cutlass.Float32(0.0)) + tXrO.store(y.to(out.element_type)) + cute.copy(copy_atom_store, tXrO, tXgO) + + +@cute.jit +def m3_embed_norm( + ids: cute.Tensor, + table: cute.Tensor, + weight: cute.Tensor, + raw: cute.Tensor, + out: cute.Tensor, + vocab: cutlass.Int64, + eps: cutlass.Float32, + n_tokens: cutlass.Constexpr[int], + hidden: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + # flashinfer's thread-value layout for one row: thread t, value (v, k) -> column 8 t + v + 1024 k. + blocks = hidden // (NORM_VEC * NORM_THREADS) + tv_layout = cute.make_layout(((NORM_THREADS, 1), (NORM_VEC, blocks)), + stride=((NORM_VEC, 1), (1, NORM_VEC * NORM_THREADS))) # fmt: skip + m3_embed_norm_kernel( + ids, table, weight, raw, out, vocab, eps, tv_layout, (1, hidden), hidden + ).launch( + grid=[n_tokens, 1, 1], + block=[NORM_THREADS, 1, 1], + smem=NORM_WARPS * 4, + stream=stream, + use_pdl=use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/op.py new file mode 100644 index 000000000000..c12bd2500445 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/embedding_firstnorm/op.py @@ -0,0 +1,157 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Host side of the MiniMax-M3 decode embedding + first-layer RMSNorm (m3_embedding_firstnorm_kernel.py): H 6144, +1..8 rows, into caller-owned buffers. + +Compiled on the first call per (rows, id dtype, table shape, PDL), which must happen outside CUDA-graph capture +(:class:`DecodeEmbedding` compiles at construction). +""" + +import threading +from pathlib import Path +from typing import Dict, Tuple + +import cuda.bindings.driver as cuda_driver +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import from_dlpack + +from tensorrt_llm._torch import cute_aot + +from .m3_embedding_firstnorm_kernel import m3_embed_norm as _m3_embed_norm_jit + +HIDDEN = 6144 # capacity: fixed: the checkpoint's hidden size +MAX_ROWS = 8 # capacity: fixed: the kernel's built rows (decode steps) + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + + +def arg(tensor): + return from_dlpack( + tensor.detach(), + assumed_align=tensor.element_size() + if tensor.ndim == 1 and tensor.dtype in (torch.int32, torch.int64) + else 16, + ) + + +def _check(ids, table, weight, raw, out) -> None: + rows = ids.shape[0] if ids.dim() == 1 else 0 + if not ( + 1 <= rows <= MAX_ROWS + and ids.dtype in (torch.int32, torch.int64) + and ids.is_contiguous() + and table.dtype == torch.bfloat16 + and table.dim() == 2 + and table.shape[1] == HIDDEN + and table.is_contiguous() + and weight.dtype == torch.bfloat16 + and tuple(weight.shape) == (HIDDEN,) + and all( + t.dtype == torch.bfloat16 and tuple(t.shape) == (rows, HIDDEN) and t.is_contiguous() + for t in (raw, out) + ) + ): + raise ValueError( + "m3_embed_norm: expects ids int32/int64 [1..8], a contiguous bf16 table [V, 6144], bf16 weight [6144] " + "and contiguous bf16 raw / out [rows, 6144]" + ) + + +def _compile(ids, table, weight, eps, raw, out, use_pdl: bool): + key = (ids.shape[0], ids.dtype, tuple(table.shape), table.device, bool(use_pdl)) + fn = _compiled.get(key) + if fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_embed_norm compiles on its first call per shape: call it once before capture" + ) + with _lock: + fn = _compiled.get(key) + if fn is None: + stream = cuda_driver.CUstream(torch.cuda.current_stream(table.device).cuda_stream) + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_embed_norm {key[0]} {key[1]} {key[2]} {key[4]} eps={float(eps)!r}", + lambda: cute.compile( + _m3_embed_norm_jit, + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + ids.shape[0], + HIDDEN, + bool(use_pdl), + stream, + ), + sources=_aot_sources(), + ) + return fn + + +def m3_embed_norm( + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """Write ``table[ids]`` into ``raw`` (a zero row for an id outside ``[0, V)``) and its RMSNorm with ``weight`` / + ``eps`` into ``out``; returns ``(out, raw)``.""" + _check(ids, table, weight, raw, out) + fn = _compile(ids, table, weight, eps, raw, out, use_pdl) + stream = cuda_driver.CUstream(torch.cuda.current_stream(ids.device).cuda_stream) + fn( + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + stream, + ) + return out, raw + + +class DecodeEmbedding: + """Caller-owned buffers of one (rows, id dtype) decode embedding: ``raw`` (the gathered rows, the first residual) + and ``output`` (their RMSNorm), over ``table`` and the first layer's plain (non-Gemma) input norm ``weight`` / + ``eps``. Built (and compiled) before CUDA-graph capture.""" + + def __init__(self, table, weight, eps, rows, id_dtype, use_pdl=True): + assert not torch.cuda.is_current_stream_capturing() + assert table.dtype == torch.bfloat16 and table.is_contiguous() and table.shape[1] == HIDDEN + assert weight.dtype == torch.bfloat16 and weight.shape == (HIDDEN,) + assert 1 <= rows <= MAX_ROWS and id_dtype in (torch.int32, torch.int64) + self.table, self.weight, self.rows, self.id_dtype, self.use_pdl = ( + table, + weight, + rows, + id_dtype, + use_pdl, + ) + self.eps = float(eps) + self.raw = torch.empty(rows, HIDDEN, device=table.device, dtype=torch.bfloat16) + self.output = torch.empty_like(self.raw) + ids = torch.empty(rows, device=table.device, dtype=id_dtype) + _compile(ids, table, weight, self.eps, self.raw, self.output, use_pdl) + + def __call__(self, ids): + assert ids.shape == (self.rows,) and ids.dtype == self.id_dtype and ids.is_contiguous() + return m3_embed_norm( + ids, self.table, self.weight, self.eps, self.raw, self.output, self.use_pdl + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/lc2_mixed.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/lc2_mixed.py new file mode 100644 index 000000000000..c1357ca13638 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/lc2_mixed.py @@ -0,0 +1,77 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""LC2_MIXED (not bitwise): a mixed ctx + gen step's sparse generation rows (the +MsaDecodeFmha sparse_decode_hook) run LC2 (LC2-split: the C++ one-launch sparse decode) instead of GQAMIX's Triton +copy: 1.3-1.9x faster at 16..224 rows, in the same error band vs float64 as the Triton route. No MXFP8 hand-off: the +mixed step's o_proj input also holds the context rows. Operands the kernel declines (``m3_attn_lc.lc_applies``) keep +the Triton copy. Kill switch: TRTLLM_M31_DISABLE_LC2_MIXED=1, read at import. +""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm.logger import logger + +from ..m31_switches import m31_disabled +from . import m3_attn_lc, m3_report +from .triton_sparse_decode import TP2WIDE_LC2 + +LC2_MIXED = TP2WIDE_LC2 and not m31_disabled("LC2_MIXED") +_LOGGED: set = set() + + +def lc2_mixed_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + kv_scale=None, + k_block_scale=None, + v_block_scale=None, + k_global_scale=None, + v_global_scale=None, + num_topk_chunks=None, + identity_blocks=False, +) -> bool: + """Run ``minimax_m3_sparse_attn_decode``'s contract through LC2 when the operands fit; return whether it ran.""" + if not LC2_MIXED or kv_scale is not None or num_topk_chunks is not None: + return False + if any(t is None for t in (k_block_scale, v_block_scale, k_global_scale, v_global_scale)): + return False + if q.dtype != torch.float8_e4m3fn or int(q.shape[0]) == 0: + return False + if not m3_attn_lc.lc_applies( + q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output + ): + return False # operands outside LC2's layout: GQAMIX's Triton copy runs them + m3_attn_lc.m3_lc_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + mxfp8=None, + use_pdl=get_env_enable_pdl(), + ) + rows = int(q.shape[0]) + m3_report.work("LC2_MIXED", "mixed-step sparse generation rows on LC2") + if rows not in _LOGGED: + _LOGGED.add(rows) + logger.info(f"[M3.1 tp2] LC2_MIXED taken: {rows} mixed-step generation rows") + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/long_select.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/long_select.py new file mode 100644 index 000000000000..193cc96e72bf --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/long_select.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 block select for MiniMax-M3 decode rows of 129 to 9216 blocks. + +Same contract and bit-identical output as ``torch.ops.trtllm.minimax_m3_select_blocks`` on per-KV-head scores +[num_kv_heads, n_blocks, total_q]: init / local forcing, per-query valid-block count, ties to the smaller block index, +forced -inf picks emitted as -1, ascending block ids with -1 tail padding. + +One 1024-thread CTA per (query, KV head) row holds the row's ordered score keys in registers (2, 4 or 9 per +thread, by the row's valid count). The weakest of the 16 maxima of the interleaved groups ``block % 16`` is a lower +bound on the 16th-largest key, so the top 16 is among the keys at or above it; those are compacted to shared memory +and ranked exactly by (key, block). More than 1024 keys tied at the bound (degenerate rows) rank every candidate +against the whole row (re-read). + +The kernel is ``torch.ops.trtllm.minimax_m3_sm107_tp4_select_long`` (this target's copy of ``minimax_m3_select_long`` +with a choice of where it releases the sparse decode), called through this target's ``m3_select_long`` entry. +""" + +from typing import Optional + +import torch + +from ..ops.attention.m3_select_long import MAX_BLOCKS, m3_select_long +from ..ops.attention.m3_select_long_dsl import m3_select_long_dsl + + +def long_select_blocks( + max_score_kv: torch.Tensor, + *, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + dsl: bool = False, +) -> Optional[torch.Tensor]: + """Top-16 select over per-KV-head scores [num_kv_heads, n_blocks, total_q] with 129..9216 blocks. + + Returns the head-major backed [total_q, num_kv_heads, 16] int32 table, or None outside this kernel's shapes + (callers keep the reference op). ``early_trigger`` releases the dependent sparse decode right after the select's + grid wait (``SELECT_EARLY_MAX_ROWS`` in modeling.py); otherwise ``trigger_phase`` 1 releases it once every score + of the row is loaded and 0 at the select's exit (``SELECT_LONG_TRIGGER_PHASE``). ``fast_tail`` ranks rows of at + most 64 candidates on warp 0 (``SELECT_LONG_SL3``; the same picks). ``dsl`` runs the 384-thread kernel + (``SELECT_LONG_DSL``; the same table; ``fast_tail`` does not apply). + """ + if ( + max_score_kv.dtype != torch.float32 + or not max_score_kv.is_cuda + or max_score_kv.dim() != 3 + or not 128 < max_score_kv.shape[1] <= MAX_BLOCKS + or n_valid_blocks.dtype != torch.int32 + or not n_valid_blocks.is_cuda + or n_valid_blocks.shape[0] < max_score_kv.shape[2] + ): + return None + num_kv_heads, _, total_q = max_score_kv.shape + out = torch.empty( + (num_kv_heads, total_q, 16), dtype=torch.int32, device=max_score_kv.device + ).permute(1, 0, 2) + return (m3_select_long_dsl if dsl else m3_select_long)( + max_score_kv, + n_valid_blocks.contiguous(), + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + fast_tail, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m31_f3_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m31_f3_gemm.py new file mode 100644 index 000000000000..5ac7790f0055 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m31_f3_gemm.py @@ -0,0 +1,1910 @@ +# ruff: noqa: E731, E741 +# Retain upstream CuTe DSL helper expressions and dimension names for fork comparison. +# m31_f3_gemm (F3): GENERATED kernel (gen_f3_gemm.py from FlashInfer's dense_blockscaled_gemm_sm100.py +# (md5 99892be6); do not edit. +# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: BSD-3-Clause + +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: + +# 1. Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. + +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. + +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. + +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# This file is ported from TensorRT-LLM's dense_blockscaled_gemm_persistent.py +# with modifications for FlashInfer integration. +# Original: https://github.com/NVIDIA/TensorRT-LLM + +import hashlib as _hashlib +import os as _os +from pathlib import Path +from typing import Optional, Tuple, Type, Union + +import cuda.bindings.driver as cuda +import cutlass +import cutlass.cute as cute +import cutlass.pipeline as pipeline +import cutlass.utils as utils +import cutlass.utils.blackwell_helpers as sm100_utils +import cutlass.utils.blockscaled_layout as blockscaled_utils +import torch as _torch +from cutlass.cute.arch import griddepcontrol_launch_dependents, griddepcontrol_wait +from cutlass.cute.nvgpu import cpasync, tcgen05 +from cutlass.pipeline import PipelineTmaUmma, PipelineUmmaAsync +from flashinfer.gemm.kernels.dense_blockscaled_gemm_sm100_common import _Sm100BlockScaledGemmCommon + +from tensorrt_llm._torch import cute_aot + + +class F3LamportGemmKernel(_Sm100BlockScaledGemmCommon): + """Implements batched matrix multiplication (C = A x SFA x B x SFB) with support for various data types + and Blackwell GPU architectural features, including persistent tile scheduling and warp specialization. + + Notes: + - In the current version, tensors A and B must have the same data type. + For example, using Float8E4M3FN for A and Float8E5M2 for B is not supported. + - Supported combinations of A/B data types, scale factor (SF) data types, and SF vector size: + * MXF8: A/B: Float8E5M2/Float8E4M3FN, SF: Float8E8M0FNU, sf_vec_size: 32 + * MXF4: A/B: Float4E2M1FN, SF: Float8E8M0FNU, sf_vec_size: 32 + * NVF4: A/B: Float4E2M1FN, SF: Float8E8M0FNU/Float8E4M3FN, sf_vec_size: 16 + - Supported accumulator data types: + * Float32 + - Supported C data types: + * Float32 + * Float16/BFloat16 + * Float8E4M3FN/Float8E5M2 + - Constraints: + * MMA tiler M must be 128 or 256 (use_2cta_instrs). + * MMA tiler N must be 64/128/192/256 + * Cluster shape M must be a multiple of 2 if MMA tiler M is 256. + * Cluster shape M/N must be positive and a power of 2, with total cluster size <= 16. + * Cluster shape M/N must be <= 4 for scale factor multicasts due to limited scale factor size. + + Example: + >>> gemm = Sm100BlockScaledPersistentDenseGemmKernel( + ... sf_vec_size=16, mma_tiler_mn=(256, 128), cluster_shape_mn=(2, 1) + ... ) + >>> gemm(a_tensor, b_tensor, sfa_tensor, sfb_tensor, c_tensor, max_active_clusters, stream) + """ + + def __init__( + self, + sf_vec_size: int, + mma_tiler_mn: Tuple[int, int], + cluster_shape_mn: Tuple[int, int], + use_prefetch: bool = False, + enable_pdl: bool = True, + ): + """Initializes the configuration for a Blackwell dense GEMM kernel. + + This configuration includes several key aspects: + + 1. MMA Instruction Settings (tcgen05): + - acc_dtype: Data types for MMA accumulator, always set to Float32 + - sf_vec_size: Scalefactor A/B vector size. + - mma_tiler_mn: The (M, N) shape of the MMA instruction tiler. + + 2. Cluster Shape: + - cluster_shape_mn: The (ClusterM, ClusterN) shape of the CTA cluster. + + Args: + sf_vec_size (int): Scalefactor vector size. + mma_tiler_mn (Tuple[int, int]): Shape of the Matrix Multiply-Accumulate (MMA) tile (M, N). + cluster_shape_mn (Tuple[int, int]): Cluster dimensions (M, N) for parallel processing. + """ + + self.acc_dtype = cutlass.Float32 + self.sf_vec_size = sf_vec_size + self.use_2cta_instrs = mma_tiler_mn[0] == 256 + self.cluster_shape_mn = cluster_shape_mn + # K dimension is deferred in _setup_attributes + self.mma_tiler = (*mma_tiler_mn, 1) + self.use_prefetch = use_prefetch + self.enable_pdl = enable_pdl + self.cta_group = tcgen05.CtaGroup.TWO if self.use_2cta_instrs else tcgen05.CtaGroup.ONE + + self.occupancy = 1 + # Set specialized warp ids + self.epilog_warp_id = ( + 0, + 1, + 2, + 3, + ) + self.mma_warp_id = 4 + self.tma_warp_id = 5 + self.threads_per_cta = 32 * len((self.mma_warp_id, self.tma_warp_id, *self.epilog_warp_id)) + # Set barrier id for cta sync, epilogue sync and tmem ptr sync + self.cta_sync_bar_id = 0 + self.epilog_sync_bar_id = 1 + self.tmem_ptr_sync_bar_id = 2 + self.smem_capacity = utils.get_smem_capacity_in_bytes("sm_100") + SM100_TMEM_CAPACITY_COLUMNS = 512 + self.num_tmem_alloc_cols = SM100_TMEM_CAPACITY_COLUMNS + + def _setup_attributes(self): + """Set up configurations that are dependent on GEMM inputs + + This method configures various attributes based on the input tensor properties + (data types, leading dimensions) and kernel settings: + - Configuring tiled MMA + - Computing MMA/cluster/tile shapes + - Computing cluster layout + - Computing multicast CTAs for A/B/SFA/SFB + - Computing epilogue subtile + - Setting up A/B/SFA/SFB/C stage counts in shared memory + - Computing A/B/SFA/SFB/C shared memory layout + - Computing tensor memory allocation columns + """ + # Compute mma instruction shapes + mma_inst_bits_k = 256 + # (MMA_Tile_Shape_M, MMA_Tile_Shape_N, MMA_Inst_Shape_K) + self.mma_inst_shape_mnk = ( + self.mma_tiler[0], + self.mma_tiler[1], + mma_inst_bits_k // self.a_dtype.width, + ) + # (CTA_Tile_Shape_M, Round_Up(MMA_Tile_Shape_N, 128), MMA_Inst_Shape_K) + self.mma_inst_shape_mnk_sfb = ( + self.mma_inst_shape_mnk[0] // (2 if self.use_2cta_instrs else 1), + cute.round_up(self.mma_inst_shape_mnk[1], 128), + self.mma_inst_shape_mnk[2], + ) + + tiled_mma = sm100_utils.make_blockscaled_trivial_tiled_mma( + self.a_dtype, + self.b_dtype, + self.a_major_mode, + self.b_major_mode, + self.sf_dtype, + self.sf_vec_size, + self.cta_group, + self.mma_inst_shape_mnk[:2], + ) + + tiled_mma_sfb = sm100_utils.make_blockscaled_trivial_tiled_mma( + self.a_dtype, + self.b_dtype, + self.a_major_mode, + self.b_major_mode, + self.sf_dtype, + self.sf_vec_size, + cute.nvgpu.tcgen05.CtaGroup.ONE, + self.mma_inst_shape_mnk_sfb[:2], + ) + + # Compute mma/cluster/tile shapes + mma_inst_tile_k = 4 + self.mma_tiler = ( + self.mma_inst_shape_mnk[0], + self.mma_inst_shape_mnk[1], + self.mma_inst_shape_mnk[2] * mma_inst_tile_k, + ) + self.mma_tiler_sfb = ( + self.mma_inst_shape_mnk_sfb[0], + self.mma_inst_shape_mnk_sfb[1], + self.mma_inst_shape_mnk_sfb[2] * mma_inst_tile_k, + ) + self.cta_tile_shape_mnk = ( + self.mma_tiler[0] // cute.size(tiled_mma.thr_id.shape), + self.mma_tiler[1], + self.mma_tiler[2], + ) + self.cta_tile_shape_mnk_sfb = ( + self.mma_tiler_sfb[0] // cute.size(tiled_mma.thr_id.shape), + self.mma_tiler_sfb[1], + self.mma_tiler_sfb[2], + ) + + # Compute cluster layout + self.cluster_layout_vmnk = cute.tiled_divide( + cute.make_layout((*self.cluster_shape_mn, 1)), + (tiled_mma.thr_id.shape,), + ) + self.cluster_layout_sfb_vmnk = cute.tiled_divide( + cute.make_layout((*self.cluster_shape_mn, 1)), + (tiled_mma_sfb.thr_id.shape,), + ) + + # Compute number of multicast CTAs for A/B + self.num_mcast_ctas_a = cute.size(self.cluster_layout_vmnk.shape[2]) + self.num_mcast_ctas_b = cute.size(self.cluster_layout_vmnk.shape[1]) + self.num_mcast_ctas_sfb = cute.size(self.cluster_layout_sfb_vmnk.shape[1]) + self.is_a_mcast = self.num_mcast_ctas_a > 1 + self.is_b_mcast = self.num_mcast_ctas_b > 1 + self.is_sfb_mcast = self.num_mcast_ctas_sfb > 1 + + # Compute epilogue subtile + self.epi_tile = sm100_utils.compute_epilogue_tile_shape( + self.cta_tile_shape_mnk, + self.use_2cta_instrs, + self.c_layout, + self.c_dtype, + ) + + self.epi_tile_n = cute.size(self.epi_tile[1]) + + # Setup A/B/C stage count in shared memory and ACC stage count in tensor memory + self.num_acc_stage, self.num_ab_stage, self.num_c_stage = self._compute_stages( + tiled_mma, + self.mma_tiler, + self.a_dtype, + self.a_major_mode, + self.b_dtype, + self.b_major_mode, + self.epi_tile, + self.c_dtype, + self.c_layout, + self.sf_dtype, + self.sf_vec_size, + self.smem_capacity, + self.occupancy, + 1024, + ) + + # Compute A/B/SFA/SFB/C shared memory layout + self.a_smem_layout_staged = sm100_utils.make_smem_layout_a( + tiled_mma, + self.mma_tiler, + self.a_dtype, + self.num_ab_stage, + ) + self.b_smem_layout_staged = sm100_utils.make_smem_layout_b( + tiled_mma, + self.mma_tiler, + self.b_dtype, + self.num_ab_stage, + ) + self.sfa_smem_layout_staged = blockscaled_utils.make_smem_layout_sfa( + tiled_mma, + self.mma_tiler, + self.sf_vec_size, + self.num_ab_stage, + ) + self.sfb_smem_layout_staged = blockscaled_utils.make_smem_layout_sfb( + tiled_mma, + self.mma_tiler, + self.sf_vec_size, + self.num_ab_stage, + ) + self.c_smem_layout_staged = sm100_utils.make_smem_layout_epi( + self.c_dtype, + self.c_layout, + self.epi_tile, + self.num_c_stage, + ) + + self.overlapping_accum = self.num_acc_stage == 1 + sf_atom_mn = 32 + self.num_sfa_tmem_cols = (self.cta_tile_shape_mnk[0] // sf_atom_mn) * mma_inst_tile_k + self.num_sfb_tmem_cols = (self.cta_tile_shape_mnk_sfb[1] // sf_atom_mn) * mma_inst_tile_k + self.num_sf_tmem_cols = self.num_sfa_tmem_cols + self.num_sfb_tmem_cols + self.num_accumulator_tmem_cols = ( + self.cta_tile_shape_mnk[1] * self.num_acc_stage + if not self.overlapping_accum + else self.cta_tile_shape_mnk[1] * 2 - self.num_sf_tmem_cols + ) + + # Only when overlapping_accum is enabled, we need to release accumulator buffer early in epilogue + self.iter_acc_early_release_in_epilogue = self.num_sf_tmem_cols // self.epi_tile_n + + # TODO: [alel] Currently set prefetch dist to num_ab_stage, we may have more options for prefetch dist auto + # tuning + self.prefetch_dist = self.num_ab_stage + + @cute.jit + def __call__( + self, + a_tensor: cute.Tensor, + b_tensor: cute.Tensor, + sfa_tensor: cute.Tensor, + sfb_tensor: cute.Tensor, + c_tensor: cute.Tensor, + alpha: cute.Tensor, # Single-element tensor containing alpha value + max_active_clusters: cutlass.Constexpr, + stream: cuda.CUstream, + c_s0l: cute.Tensor, + c_s0p: cute.Tensor, + c_s1l: cute.Tensor, + c_s1p: cute.Tensor, + c_s2l: cute.Tensor, + c_s2p: cute.Tensor, + flag_tensor: cute.Tensor, + epilogue_op: cutlass.Constexpr = lambda x: x, + ): + """Execute the GEMM operation in steps: + - Setup static attributes before smem/grid/tma computation + - Setup TMA load/store atoms and tensors + - Compute grid size with regard to hardware constraints + - Define shared storage for kernel + - Launch the kernel synchronously + + Args: + a_tensor (cute.Tensor): Input tensor A + b_tensor (cute.Tensor): Input tensor B + sfa_tensor (cute.Tensor): Scale factor tensor A + sfb_tensor (cute.Tensor): Scale factor tensor B + c_tensor (cute.Tensor): Output tensor C + max_active_clusters (cutlass.Constexpr): Maximum number of active clusters + stream (cuda.CUstream): CUDA stream for asynchronous execution + epilogue_op (cutlass.Constexpr): Optional elementwise lambda function to apply to the output tensor + + Raises: + TypeError: If input data types are incompatible with the MMA instruction. + """ + ( + tiled_mma, + tiled_mma_sfb, + tma_atom_a, + tma_tensor_a, + tma_atom_b, + tma_tensor_b, + tma_atom_sfa, + tma_tensor_sfa, + tma_atom_sfb, + tma_tensor_sfb, + tma_atom_c, + tma_tensor_c, + ) = self._prepare_tma( + a_tensor, + b_tensor, + sfa_tensor, + sfb_tensor, + c_tensor, + ) + + # F3: one TMA store atom per Lamport data slot (3 rotating slots x local / peer section of this rank) + f3_epi_smem_layout = cute.slice_(self.c_smem_layout_staged, (None, None, 0)) + f3_atoms = [] + for f3_t in (c_s0l, c_s0p, c_s1l, c_s1p, c_s2l, c_s2p): + f3_atom, _ = cpasync.make_tiled_tma_atom( + cpasync.CopyBulkTensorTileS2GOp(), f3_t, f3_epi_smem_layout, self.epi_tile + ) + f3_atoms.append(f3_atom) + + # Compute grid size + self.tile_sched_params, grid = self._compute_grid( + c_tensor, + self.cta_tile_shape_mnk, + self.cluster_shape_mn, + max_active_clusters, + ) + + self.buffer_align_bytes = 1024 + + # Define shared storage for kernel + @cute.struct + class SharedStorage: + ab_full_mbar_ptr: cute.struct.MemRange[cutlass.Int64, self.num_ab_stage] + ab_empty_mbar_ptr: cute.struct.MemRange[cutlass.Int64, self.num_ab_stage] + acc_full_mbar_ptr: cute.struct.MemRange[cutlass.Int64, self.num_acc_stage] + acc_empty_mbar_ptr: cute.struct.MemRange[cutlass.Int64, self.num_acc_stage] + tmem_dealloc_mbar_ptr: cutlass.Int64 + tmem_holding_buf: cutlass.Int32 + # (EPI_TILE_M, EPI_TILE_N, STAGE) + sC: cute.struct.Align[ + cute.struct.MemRange[ + self.c_dtype, + cute.cosize(self.c_smem_layout_staged.outer), + ], + self.buffer_align_bytes, + ] + # (MMA, MMA_M, MMA_K, STAGE) + sA: cute.struct.Align[ + cute.struct.MemRange[self.a_dtype, cute.cosize(self.a_smem_layout_staged.outer)], + self.buffer_align_bytes, + ] + # (MMA, MMA_N, MMA_K, STAGE) + sB: cute.struct.Align[ + cute.struct.MemRange[self.b_dtype, cute.cosize(self.b_smem_layout_staged.outer)], + self.buffer_align_bytes, + ] + # (MMA, MMA_M, MMA_K, STAGE) + sSFA: cute.struct.Align[ + cute.struct.MemRange[self.sf_dtype, cute.cosize(self.sfa_smem_layout_staged)], + self.buffer_align_bytes, + ] + # (MMA, MMA_N, MMA_K, STAGE) + sSFB: cute.struct.Align[ + cute.struct.MemRange[self.sf_dtype, cute.cosize(self.sfb_smem_layout_staged)], + self.buffer_align_bytes, + ] + + self.shared_storage = SharedStorage + + # Launch the kernel synchronously + self.kernel( + tiled_mma, + tiled_mma_sfb, + tma_atom_a, + tma_tensor_a, + tma_atom_b, + tma_tensor_b, + tma_atom_sfa, + tma_tensor_sfa, + tma_atom_sfb, + tma_tensor_sfb, + tma_atom_c, + tma_tensor_c, + self.cluster_layout_vmnk, + self.cluster_layout_sfb_vmnk, + self.a_smem_layout_staged, + self.b_smem_layout_staged, + self.sfa_smem_layout_staged, + self.sfb_smem_layout_staged, + self.c_smem_layout_staged, + self.epi_tile, + self.tile_sched_params, + epilogue_op, + alpha, + f3_atoms[0], + f3_atoms[1], + f3_atoms[2], + f3_atoms[3], + f3_atoms[4], + f3_atoms[5], + flag_tensor, + ).launch( + grid=grid, + block=[self.threads_per_cta, 1, 1], + cluster=(*self.cluster_shape_mn, 1), + smem=self.shared_storage.size_in_bytes(), # type: ignore[attr-defined] + min_blocks_per_mp=1, + stream=stream, + use_pdl=self.enable_pdl, + ) + return + + # GPU device kernel + @cute.kernel + def kernel( + self, + tiled_mma: cute.TiledMma, + tiled_mma_sfb: cute.TiledMma, + tma_atom_a: cute.CopyAtom, + mA_mkl: cute.Tensor, + tma_atom_b: cute.CopyAtom, + mB_nkl: cute.Tensor, + tma_atom_sfa: cute.CopyAtom, + mSFA_mkl: cute.Tensor, + tma_atom_sfb: cute.CopyAtom, + mSFB_nkl: cute.Tensor, + tma_atom_c: Optional[cute.CopyAtom], + mC_mnl: cute.Tensor, + cluster_layout_vmnk: cute.Layout, + cluster_layout_sfb_vmnk: cute.Layout, + a_smem_layout_staged: cute.ComposedLayout, + b_smem_layout_staged: cute.ComposedLayout, + sfa_smem_layout_staged: cute.Layout, + sfb_smem_layout_staged: cute.Layout, + c_smem_layout_staged: Union[cute.Layout, cute.ComposedLayout, None], + epi_tile: cute.Tile, + tile_sched_params: utils.PersistentTileSchedulerParams, + epilogue_op: cutlass.Constexpr, + alpha: cute.Tensor, + f3_s0l: cute.CopyAtom, + f3_s0p: cute.CopyAtom, + f3_s1l: cute.CopyAtom, + f3_s1p: cute.CopyAtom, + f3_s2l: cute.CopyAtom, + f3_s2p: cute.CopyAtom, + f3_flag: cute.Tensor, + ): + """ + GPU device kernel performing the Persistent batched GEMM computation. + """ + # Keep alpha in FP32 for precision: the accumulator is in FP32 and alpha + # may be a very small scaling factor. Converting to c_dtype (e.g., FP16) + # before multiplication could cause overflow when acc values are large. + alpha_value = alpha[0].to(cutlass.Float32) + + warp_idx = cute.arch.warp_idx() + warp_idx = cute.arch.make_warp_uniform(warp_idx) + + # + # Prefetch tma desc + # + if warp_idx == self.tma_warp_id: + cpasync.prefetch_descriptor(tma_atom_a) + cpasync.prefetch_descriptor(tma_atom_b) + cpasync.prefetch_descriptor(tma_atom_sfa) + cpasync.prefetch_descriptor(tma_atom_sfb) + cpasync.prefetch_descriptor(tma_atom_c) + cpasync.prefetch_descriptor(f3_s0l) + cpasync.prefetch_descriptor(f3_s0p) + cpasync.prefetch_descriptor(f3_s1l) + cpasync.prefetch_descriptor(f3_s1p) + cpasync.prefetch_descriptor(f3_s2l) + cpasync.prefetch_descriptor(f3_s2p) + + use_2cta_instrs = cute.size(tiled_mma.thr_id.shape) == 2 + + # + # Setup cta/thread coordinates + # + # Coords inside cluster + bidx, bidy, bidz = cute.arch.block_idx() + mma_tile_coord_v = bidx % cute.size(tiled_mma.thr_id.shape) + is_leader_cta = mma_tile_coord_v == 0 + cta_rank_in_cluster = cute.arch.make_warp_uniform(cute.arch.block_idx_in_cluster()) + block_in_cluster_coord_vmnk = cluster_layout_vmnk.get_flat_coord(cta_rank_in_cluster) + block_in_cluster_coord_sfb_vmnk = cluster_layout_sfb_vmnk.get_flat_coord( + cta_rank_in_cluster + ) + # Coord inside cta + tidx, _, _ = cute.arch.thread_idx() + + # + # Alloc and init: a+b full/empty, accumulator full/empty, tensor memory dealloc barrier + # + smem = utils.SmemAllocator() + storage = smem.allocate(self.shared_storage) + + tmem_dealloc_mbar_ptr = storage.tmem_dealloc_mbar_ptr.ptr + tmem_holding_buf = storage.tmem_holding_buf.ptr + + # Initialize mainloop ab_pipeline (barrier) and states + ab_pipeline_producer_group = pipeline.CooperativeGroup(pipeline.Agent.Thread) + num_tma_producer = self.num_mcast_ctas_a + self.num_mcast_ctas_b - 1 + ab_pipeline_consumer_group = pipeline.CooperativeGroup( + pipeline.Agent.Thread, num_tma_producer + ) + ab_pipeline = PipelineTmaUmma.create( + barrier_storage=storage.ab_full_mbar_ptr.data_ptr(), + num_stages=self.num_ab_stage, + producer_group=ab_pipeline_producer_group, + consumer_group=ab_pipeline_consumer_group, + tx_count=self.num_tma_load_bytes, + cta_layout_vmnk=cluster_layout_vmnk, + ) + + # Initialize acc_pipeline (barrier) and states + acc_pipeline_producer_group = pipeline.CooperativeGroup(pipeline.Agent.Thread) + num_acc_consumer_threads = len(self.epilog_warp_id) * (2 if use_2cta_instrs else 1) + acc_pipeline_consumer_group = pipeline.CooperativeGroup( + pipeline.Agent.Thread, num_acc_consumer_threads + ) + acc_pipeline = PipelineUmmaAsync.create( + barrier_storage=storage.acc_full_mbar_ptr.data_ptr(), + num_stages=self.num_acc_stage, + producer_group=acc_pipeline_producer_group, + consumer_group=acc_pipeline_consumer_group, + cta_layout_vmnk=cluster_layout_vmnk, + ) + + # Tensor memory dealloc barrier init + if use_2cta_instrs: + if warp_idx == self.tma_warp_id: + num_tmem_dealloc_threads = 32 + with cute.arch.elect_one(): + cute.arch.mbarrier_init(tmem_dealloc_mbar_ptr, num_tmem_dealloc_threads) + cute.arch.mbarrier_init_fence() + + # Cluster arrive after barrier init + if cute.size(self.cluster_shape_mn) > 1: + cute.arch.cluster_arrive_relaxed(aligned=True) + + # + # Setup smem tensor A/B/SFA/SFB/C + # + # (EPI_TILE_M, EPI_TILE_N, STAGE) + sC = storage.sC.get_tensor(c_smem_layout_staged.outer, swizzle=c_smem_layout_staged.inner) + # (MMA, MMA_M, MMA_K, STAGE) + sA = storage.sA.get_tensor(a_smem_layout_staged.outer, swizzle=a_smem_layout_staged.inner) + # (MMA, MMA_N, MMA_K, STAGE) + sB = storage.sB.get_tensor(b_smem_layout_staged.outer, swizzle=b_smem_layout_staged.inner) + # (MMA, MMA_M, MMA_K, STAGE) + sSFA = storage.sSFA.get_tensor(sfa_smem_layout_staged) + # (MMA, MMA_N, MMA_K, STAGE) + sSFB = storage.sSFB.get_tensor(sfb_smem_layout_staged) + + # + # Compute multicast mask for A/B/SFA/SFB buffer full + # + a_full_mcast_mask = None + b_full_mcast_mask = None + sfa_full_mcast_mask = None + sfb_full_mcast_mask = None + if cutlass.const_expr(self.is_a_mcast or self.is_b_mcast or use_2cta_instrs): + a_full_mcast_mask = cpasync.create_tma_multicast_mask( + cluster_layout_vmnk, block_in_cluster_coord_vmnk, mcast_mode=2 + ) + b_full_mcast_mask = cpasync.create_tma_multicast_mask( + cluster_layout_vmnk, block_in_cluster_coord_vmnk, mcast_mode=1 + ) + sfa_full_mcast_mask = cpasync.create_tma_multicast_mask( + cluster_layout_vmnk, block_in_cluster_coord_vmnk, mcast_mode=2 + ) + sfb_full_mcast_mask = cpasync.create_tma_multicast_mask( + cluster_layout_sfb_vmnk, block_in_cluster_coord_sfb_vmnk, mcast_mode=1 + ) + + # + # Local_tile partition global tensors + # + # (bM, bK, RestM, RestK, RestL) + gA_mkl = cute.local_tile( + mA_mkl, cute.slice_(self.mma_tiler, (None, 0, None)), (None, None, None) + ) + # (bN, bK, RestN, RestK, RestL) + gB_nkl = cute.local_tile( + mB_nkl, cute.slice_(self.mma_tiler, (0, None, None)), (None, None, None) + ) + # (bM, bK, RestM, RestK, RestL) + gSFA_mkl = cute.local_tile( + mSFA_mkl, cute.slice_(self.mma_tiler, (None, 0, None)), (None, None, None) + ) + # (bN, bK, RestN, RestK, RestL) + gSFB_nkl = cute.local_tile( + mSFB_nkl, + cute.slice_(self.mma_tiler_sfb, (0, None, None)), + (None, None, None), + ) + # (bM, bN, RestM, RestN, RestL) + gC_mnl = cute.local_tile( + mC_mnl, cute.slice_(self.mma_tiler, (None, None, 0)), (None, None, None) + ) + k_block_cnt = cutlass.Int32(cute.size(gA_mkl, mode=[3])) + + # + # Partition global tensor for TiledMMA_A/B/C + # + thr_mma = tiled_mma.get_slice(mma_tile_coord_v) + thr_mma_sfb = tiled_mma_sfb.get_slice(mma_tile_coord_v) + # (MMA, MMA_M, MMA_K, RestM, RestK, RestL) + tCgA = thr_mma.partition_A(gA_mkl) + # (MMA, MMA_N, MMA_K, RestN, RestK, RestL) + tCgB = thr_mma.partition_B(gB_nkl) + # (MMA, MMA_M, MMA_K, RestM, RestK, RestL) + tCgSFA = thr_mma.partition_A(gSFA_mkl) + # (MMA, MMA_N, MMA_K, RestN, RestK, RestL) + tCgSFB = thr_mma_sfb.partition_B(gSFB_nkl) + # (MMA, MMA_M, MMA_N, RestM, RestN, RestL) + tCgC = thr_mma.partition_C(gC_mnl) + + # + # Partition global/shared tensor for TMA load A/B + # + # TMA load A partition_S/D + a_cta_layout = cute.make_layout(cute.slice_(cluster_layout_vmnk, (0, 0, None, 0)).shape) + # ((atom_v, rest_v), STAGE) + # ((atom_v, rest_v), RestM, RestK, RestL) + tAsA, tAgA = cpasync.tma_partition( + tma_atom_a, + block_in_cluster_coord_vmnk[2], + a_cta_layout, + cute.group_modes(sA, 0, 3), + cute.group_modes(tCgA, 0, 3), + ) + # TMA load B partition_S/D + b_cta_layout = cute.make_layout(cute.slice_(cluster_layout_vmnk, (0, None, 0, 0)).shape) + # ((atom_v, rest_v), STAGE) + # ((atom_v, rest_v), RestN, RestK, RestL) + tBsB, tBgB = cpasync.tma_partition( + tma_atom_b, + block_in_cluster_coord_vmnk[1], + b_cta_layout, + cute.group_modes(sB, 0, 3), + cute.group_modes(tCgB, 0, 3), + ) + + # TMA load SFA partition_S/D + sfa_cta_layout = a_cta_layout + # ((atom_v, rest_v), STAGE) + # ((atom_v, rest_v), RestM, RestK, RestL) + tAsSFA, tAgSFA = cute.nvgpu.cpasync.tma_partition( + tma_atom_sfa, + block_in_cluster_coord_vmnk[2], + sfa_cta_layout, + cute.group_modes(sSFA, 0, 3), + cute.group_modes(tCgSFA, 0, 3), + ) + tAsSFA = cute.filter_zeros(tAsSFA) + tAgSFA = cute.filter_zeros(tAgSFA) + + # TMA load SFB partition_S/D + sfb_cta_layout = cute.make_layout( + cute.slice_(cluster_layout_sfb_vmnk, (0, None, 0, 0)).shape + ) + # ((atom_v, rest_v), STAGE) + # ((atom_v, rest_v), RestN, RestK, RestL) + tBsSFB, tBgSFB = cute.nvgpu.cpasync.tma_partition( + tma_atom_sfb, + block_in_cluster_coord_sfb_vmnk[1], + sfb_cta_layout, + cute.group_modes(sSFB, 0, 3), + cute.group_modes(tCgSFB, 0, 3), + ) + tBsSFB = cute.filter_zeros(tBsSFB) + tBgSFB = cute.filter_zeros(tBgSFB) + + # + # Partition shared/tensor memory tensor for TiledMMA_A/B/C + # + # (MMA, MMA_M, MMA_K, STAGE) + tCrA = tiled_mma.make_fragment_A(sA) + # (MMA, MMA_N, MMA_K, STAGE) + tCrB = tiled_mma.make_fragment_B(sB) + # (MMA, MMA_M, MMA_N) + acc_shape = tiled_mma.partition_shape_C(self.mma_tiler[:2]) + # (MMA, MMA_M, MMA_N, STAGE) + if cutlass.const_expr(self.overlapping_accum): + num_acc_stage_overlapped = 2 + tCtAcc_fake = tiled_mma.make_fragment_C( + cute.append(acc_shape, num_acc_stage_overlapped) + ) + # (MMA, MMA_M, MMA_N, STAGE) + tCtAcc_fake = cute.make_tensor( + tCtAcc_fake.iterator, + cute.make_layout( + tCtAcc_fake.shape, + stride=( + tCtAcc_fake.stride[0], + tCtAcc_fake.stride[1], + tCtAcc_fake.stride[2], + (256 - self.num_sf_tmem_cols) * tCtAcc_fake.stride[0][1], + ), + ), + ) + else: + # (MMA, MMA_M, MMA_N, STAGE) + tCtAcc_fake = tiled_mma.make_fragment_C(cute.append(acc_shape, self.num_acc_stage)) + + # + # Cluster wait before tensor memory alloc + # + if cute.size(self.cluster_shape_mn) > 1: + cute.arch.cluster_wait() + else: + cute.arch.barrier( + barrier_id=self.cta_sync_bar_id, number_of_threads=self.threads_per_cta + ) + + # PDL bookend: griddepcontrol_wait is always emitted; the actual PDL + # behavior is controlled by use_pdl= in .launch(). The griddepcontrol + # instructions are effectively no-ops when PDL is not enabled at + # the driver level. + griddepcontrol_wait() + + # + # Specialized TMA load warp + # + if warp_idx == self.tma_warp_id: + # + # Persistent tile scheduling loop + # + tile_sched = utils.StaticPersistentTileScheduler.create( + tile_sched_params, cute.arch.block_idx(), cute.arch.grid_dim() + ) + work_tile = tile_sched.initial_work_tile_info() + + ab_producer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.Producer, self.num_ab_stage + ) + + while work_tile.is_valid_tile: + # Get tile coord from tile scheduler + cur_tile_coord = work_tile.tile_idx + mma_tile_coord_mnl = ( + cur_tile_coord[0] // cute.size(tiled_mma.thr_id.shape), + cur_tile_coord[1], + cur_tile_coord[2], + ) + + # + # Slice to per mma tile index + # + # ((atom_v, rest_v), RestK) + tAgA_slice = tAgA[(None, mma_tile_coord_mnl[0], None, mma_tile_coord_mnl[2])] + # ((atom_v, rest_v), RestK) + tBgB_slice = tBgB[(None, mma_tile_coord_mnl[1], None, mma_tile_coord_mnl[2])] + + # ((atom_v, rest_v), RestK) + tAgSFA_slice = tAgSFA[(None, mma_tile_coord_mnl[0], None, mma_tile_coord_mnl[2])] + slice_n = mma_tile_coord_mnl[1] + if cutlass.const_expr(self.cta_tile_shape_mnk[1] == 64): + slice_n = mma_tile_coord_mnl[1] // 2 + # ((atom_v, rest_v), RestK) + tBgSFB_slice = tBgSFB[(None, slice_n, None, mma_tile_coord_mnl[2])] + + if cutlass.const_expr(self.use_prefetch): + # Prefetch both A and B (default behavior) + for pf_k_block in cutlass.range( + 0, min(self.prefetch_dist, k_block_cnt), unroll=1 + ): + cute.prefetch( + tma_atom_a, + tAgA_slice[(None, pf_k_block)], + ) + cute.prefetch( + tma_atom_b, + tBgB_slice[(None, pf_k_block)], + ) + cute.prefetch( + tma_atom_sfa, + tAgSFA_slice[(None, pf_k_block)], + ) + cute.prefetch( + tma_atom_sfb, + tBgSFB_slice[(None, pf_k_block)], + ) + + # Peek (try_wait) AB buffer empty for k_block = prefetch_k_block_cnt + ab_producer_state.reset_count() + peek_ab_empty_status = cutlass.Boolean(1) + if ab_producer_state.count < k_block_cnt: + peek_ab_empty_status = ab_pipeline.producer_try_acquire(ab_producer_state) + # + # Tma load loop + # + for k_block in cutlass.range(0, k_block_cnt, 1, unroll=1): + # Conditionally wait for AB buffer empty + ab_pipeline.producer_acquire(ab_producer_state, peek_ab_empty_status) + + # TMA load A/B/SFA/SFB + cute.copy( + tma_atom_a, + tAgA_slice[(None, ab_producer_state.count)], + tAsA[(None, ab_producer_state.index)], + tma_bar_ptr=ab_pipeline.producer_get_barrier(ab_producer_state), + mcast_mask=a_full_mcast_mask, + ) + cute.copy( + tma_atom_b, + tBgB_slice[(None, ab_producer_state.count)], + tBsB[(None, ab_producer_state.index)], + tma_bar_ptr=ab_pipeline.producer_get_barrier(ab_producer_state), + mcast_mask=b_full_mcast_mask, + ) + cute.copy( + tma_atom_sfa, + tAgSFA_slice[(None, ab_producer_state.count)], + tAsSFA[(None, ab_producer_state.index)], + tma_bar_ptr=ab_pipeline.producer_get_barrier(ab_producer_state), + mcast_mask=sfa_full_mcast_mask, + ) + cute.copy( + tma_atom_sfb, + tBgSFB_slice[(None, ab_producer_state.count)], + tBsSFB[(None, ab_producer_state.index)], + tma_bar_ptr=ab_pipeline.producer_get_barrier(ab_producer_state), + mcast_mask=sfb_full_mcast_mask, + ) + + # Prefetch: Rolling prefetch for next tiles + if cutlass.const_expr(self.use_prefetch): + if k_block < k_block_cnt - self.prefetch_dist: + future_k_block = ab_producer_state.count + self.prefetch_dist + # Prefetch both A and B (default behavior) + cute.prefetch( + tma_atom_a, + tAgA_slice[(None, future_k_block)], + ) + cute.prefetch( + tma_atom_b, + tBgB_slice[(None, future_k_block)], + ) + cute.prefetch( + tma_atom_sfa, + tAgSFA_slice[(None, future_k_block)], + ) + cute.prefetch( + tma_atom_sfb, + tBgSFB_slice[(None, future_k_block)], + ) + + # Peek (try_wait) AB buffer empty for k_block = prefetch_k_block_cnt + k_block + 1 + ab_producer_state.advance() + peek_ab_empty_status = cutlass.Boolean(1) + if ab_producer_state.count < k_block_cnt: + peek_ab_empty_status = ab_pipeline.producer_try_acquire(ab_producer_state) + + # + # Advance to next tile + # + tile_sched.advance_to_next_work() + work_tile = tile_sched.get_current_work() + + # + # Wait A/B buffer empty + # + ab_pipeline.producer_tail(ab_producer_state) + + # + # Specialized MMA warp + # + if warp_idx == self.mma_warp_id: + # + # Bar sync for retrieve tensor memory ptr from shared mem + # + tmem_ptr_read_threads = 32 * len((self.mma_warp_id, *self.epilog_warp_id)) + cute.arch.barrier( + barrier_id=self.tmem_ptr_sync_bar_id, + number_of_threads=tmem_ptr_read_threads, + ) + + # + # Retrieving tensor memory ptr and make accumulator/SFA/SFB tensor + # + # Make accumulator tmem tensor + acc_tmem_ptr = cute.arch.retrieve_tmem_ptr( + self.acc_dtype, + alignment=16, + ptr_to_buffer_holding_addr=tmem_holding_buf, + ) + # (MMA, MMA_M, MMA_N, STAGE) + tCtAcc_base = cute.make_tensor(acc_tmem_ptr, tCtAcc_fake.layout) + + # Make SFA tmem tensor + sfa_tmem_ptr = cute.recast_ptr( + acc_tmem_ptr + self.num_accumulator_tmem_cols, + dtype=self.sf_dtype, + ) + # (MMA, MMA_M, MMA_K) + tCtSFA_layout = blockscaled_utils.make_tmem_layout_sfa( + tiled_mma, + self.mma_tiler, + self.sf_vec_size, + cute.slice_(sfa_smem_layout_staged, (None, None, None, 0)), + ) + tCtSFA = cute.make_tensor(sfa_tmem_ptr, tCtSFA_layout) + + # Make SFB tmem tensor + sfb_tmem_ptr = cute.recast_ptr( + acc_tmem_ptr + self.num_accumulator_tmem_cols + self.num_sfa_tmem_cols, + dtype=self.sf_dtype, + ) + # (MMA, MMA_N, MMA_K) + tCtSFB_layout = blockscaled_utils.make_tmem_layout_sfb( + tiled_mma, + self.mma_tiler, + self.sf_vec_size, + cute.slice_(sfb_smem_layout_staged, (None, None, None, 0)), + ) + tCtSFB = cute.make_tensor(sfb_tmem_ptr, tCtSFB_layout) + # + # Partition for S2T copy of SFA/SFB + # + tiled_copy_s2t_sfa, tCsSFA_compact_s2t, tCtSFA_compact_s2t = ( + self.mainloop_s2t_copy_and_partition(sSFA, tCtSFA) + ) + tiled_copy_s2t_sfb, tCsSFB_compact_s2t, tCtSFB_compact_s2t = ( + self.mainloop_s2t_copy_and_partition(sSFB, tCtSFB) + ) + + # + # Persistent tile scheduling loop + # + tile_sched = utils.StaticPersistentTileScheduler.create( + tile_sched_params, cute.arch.block_idx(), cute.arch.grid_dim() + ) + work_tile = tile_sched.initial_work_tile_info() + + ab_consumer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.Consumer, self.num_ab_stage + ) + acc_producer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.Producer, self.num_acc_stage + ) + + while work_tile.is_valid_tile: + # Get tile coord from tile scheduler + cur_tile_coord = work_tile.tile_idx + mma_tile_coord_mnl = ( + cur_tile_coord[0] // cute.size(tiled_mma.thr_id.shape), + cur_tile_coord[1], + cur_tile_coord[2], + ) + + if cutlass.const_expr(self.overlapping_accum): + acc_stage_index = acc_producer_state.phase ^ 1 + else: + acc_stage_index = acc_producer_state.index + + # Set tensor memory buffer for current tile + # (MMA, MMA_M, MMA_N) + tCtAcc = tCtAcc_base[(None, None, None, acc_stage_index)] + + # Peek (try_wait) AB buffer full for k_block = 0 + ab_consumer_state.reset_count() + peek_ab_full_status = cutlass.Boolean(1) + if ab_consumer_state.count < k_block_cnt and is_leader_cta: + peek_ab_full_status = ab_pipeline.consumer_try_wait(ab_consumer_state) + + # + # Wait for accumulator buffer empty + # + if is_leader_cta: + acc_pipeline.producer_acquire(acc_producer_state) + + tCtSFB_mma = tCtSFB + if cutlass.const_expr(self.cta_tile_shape_mnk[1] == 192): + # If this is an ODD tile, shift the TMEM start address for cta_tile_shape_n=192 case by two + # words (ignores first 64 columns of SFB) + offset = ( + cutlass.Int32(2) if mma_tile_coord_mnl[1] % 2 == 1 else cutlass.Int32(0) + ) + shifted_ptr = cute.recast_ptr( + acc_tmem_ptr + + self.num_accumulator_tmem_cols + + self.num_sfa_tmem_cols + + offset, + dtype=self.sf_dtype, + ) + tCtSFB_mma = cute.make_tensor(shifted_ptr, tCtSFB_layout) + elif cutlass.const_expr(self.cta_tile_shape_mnk[1] == 64): + # Move in increments of 64 columns of SFB + offset = cutlass.Int32((mma_tile_coord_mnl[1] % 2) * 2) + shifted_ptr = cute.recast_ptr( + acc_tmem_ptr + + self.num_accumulator_tmem_cols + + self.num_sfa_tmem_cols + + offset, + dtype=self.sf_dtype, + ) + tCtSFB_mma = cute.make_tensor(shifted_ptr, tCtSFB_layout) + + # + # Reset the ACCUMULATE field for each tile + # + tiled_mma.set(tcgen05.Field.ACCUMULATE, False) + + # + # Mma mainloop + # + for _k_block in range(k_block_cnt): + if is_leader_cta: + # Conditionally wait for AB buffer full + ab_pipeline.consumer_wait(ab_consumer_state, peek_ab_full_status) + + # Copy SFA/SFB from smem to tmem + s2t_stage_coord = ( + None, + None, + None, + None, + ab_consumer_state.index, + ) + tCsSFA_compact_s2t_staged = tCsSFA_compact_s2t[s2t_stage_coord] + tCsSFB_compact_s2t_staged = tCsSFB_compact_s2t[s2t_stage_coord] + cute.copy( + tiled_copy_s2t_sfa, + tCsSFA_compact_s2t_staged, + tCtSFA_compact_s2t, + ) + cute.copy( + tiled_copy_s2t_sfb, + tCsSFB_compact_s2t_staged, + tCtSFB_compact_s2t, + ) + + # tCtAcc += tCrA * tCrSFA * tCrB * tCrSFB + num_kphases = cute.size(tCrA, mode=[2]) + for kphase_idx in cutlass.range(num_kphases, unroll_full=True): + kphase_coord = ( + None, + None, + kphase_idx, + ab_consumer_state.index, + ) + + # Set SFA/SFB tensor to tiled_mma + sf_kphase_coord = (None, None, kphase_idx) + tiled_mma.set( + tcgen05.Field.SFA, + tCtSFA[sf_kphase_coord].iterator, + ) + tiled_mma.set( + tcgen05.Field.SFB, + tCtSFB_mma[sf_kphase_coord].iterator, + ) + + cute.gemm( + tiled_mma, + tCtAcc, + tCrA[kphase_coord], + tCrB[kphase_coord], + tCtAcc, + ) + + # Enable accumulate on tCtAcc after first kphase + tiled_mma.set(tcgen05.Field.ACCUMULATE, True) + + # Async arrive AB buffer empty + ab_pipeline.consumer_release(ab_consumer_state) + + # Peek (try_wait) AB buffer full for k_block = k_block + 1 + ab_consumer_state.advance() + peek_ab_full_status = cutlass.Boolean(1) + if ab_consumer_state.count < k_block_cnt: + if is_leader_cta: + peek_ab_full_status = ab_pipeline.consumer_try_wait(ab_consumer_state) + + # + # Async arrive accumulator buffer full + # + if is_leader_cta: + acc_pipeline.producer_commit(acc_producer_state) + acc_producer_state.advance() + + # + # Advance to next tile + # + tile_sched.advance_to_next_work() + work_tile = tile_sched.get_current_work() + + # + # Wait for accumulator buffer empty + # + acc_pipeline.producer_tail(acc_producer_state) + # + # Specialized epilogue warps + # + if warp_idx < self.mma_warp_id: + # + # Alloc tensor memory buffer + # + if warp_idx == self.epilog_warp_id[0]: + cute.arch.alloc_tmem( + self.num_tmem_alloc_cols, + tmem_holding_buf, + is_two_cta=use_2cta_instrs, + ) + + # + # Bar sync for retrieve tensor memory ptr from shared memory + # + tmem_ptr_read_threads = 32 * len((self.mma_warp_id, *self.epilog_warp_id)) + cute.arch.barrier( + barrier_id=self.tmem_ptr_sync_bar_id, + number_of_threads=tmem_ptr_read_threads, + ) + + # + # Retrieving tensor memory ptr and make accumulator tensor + # + acc_tmem_ptr = cute.arch.retrieve_tmem_ptr( + self.acc_dtype, + alignment=16, + ptr_to_buffer_holding_addr=tmem_holding_buf, + ) + # (MMA, MMA_M, MMA_N, STAGE) + tCtAcc_base = cute.make_tensor(acc_tmem_ptr, tCtAcc_fake.layout) + + # + # Partition for epilogue + # + epi_tidx = tidx + tiled_copy_t2r, tTR_tAcc_base, tTR_rAcc = self.epilog_tmem_copy_and_partition( + epi_tidx, tCtAcc_base, tCgC, epi_tile, use_2cta_instrs + ) + + tTR_rC = cute.make_rmem_tensor(tTR_rAcc.shape, self.c_dtype) + tiled_copy_r2s, tRS_rC, tRS_sC = self.epilog_smem_copy_and_partition( + tiled_copy_t2r, tTR_rC, epi_tidx, sC + ) + tma_atom_c, bSG_sC, bSG_gC_partitioned = self.epilog_gmem_copy_and_partition( + epi_tidx, tma_atom_c, tCgC, epi_tile, sC + ) + # F3: the Lamport slot of this call, and the C staging buffer as 32-bit words + f3_slot = f3_flag[0] % 3 + f3_stage_words = cute.cosize(c_smem_layout_staged.outer) // (2 * self.num_c_stage) + f3_words = cute.make_tensor( + cute.recast_ptr(storage.sC.data_ptr(), dtype=cutlass.Int32), + cute.make_layout(f3_stage_words * self.num_c_stage), + ) + f3_epi_threads = 32 * len(self.epilog_warp_id) + + # + # Persistent tile scheduling loop + # + tile_sched = utils.StaticPersistentTileScheduler.create( + tile_sched_params, cute.arch.block_idx(), cute.arch.grid_dim() + ) + work_tile = tile_sched.initial_work_tile_info() + + acc_consumer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.Consumer, self.num_acc_stage + ) + + # Threads/warps participating in tma store pipeline + c_producer_group = pipeline.CooperativeGroup( + pipeline.Agent.Thread, + 32 * len(self.epilog_warp_id), + ) + c_pipeline = pipeline.PipelineTmaStore.create( + num_stages=self.num_c_stage, + producer_group=c_producer_group, + ) + + while work_tile.is_valid_tile: + # Get tile coord from tile scheduler + cur_tile_coord = work_tile.tile_idx + mma_tile_coord_mnl = ( + cur_tile_coord[0] // cute.size(tiled_mma.thr_id.shape), + cur_tile_coord[1], + cur_tile_coord[2], + ) + + # + # Slice to per mma tile index + # + # ((ATOM_V, REST_V), EPI_M, EPI_N) + bSG_gC = bSG_gC_partitioned[ + ( + None, + None, + None, + *mma_tile_coord_mnl, + ) + ] + + if cutlass.const_expr(self.overlapping_accum): + acc_stage_index = acc_consumer_state.phase + reverse_subtile = ( + cutlass.Boolean(True) if acc_stage_index == 0 else cutlass.Boolean(False) + ) + else: + acc_stage_index = acc_consumer_state.index + + # Set tensor memory buffer for current tile + # (T2R, T2R_M, T2R_N, EPI_M, EPI_M) + tTR_tAcc = tTR_tAcc_base[(None, None, None, None, None, acc_stage_index)] + + # + # Wait for accumulator buffer full + # + acc_pipeline.consumer_wait(acc_consumer_state) + + tTR_tAcc = cute.group_modes(tTR_tAcc, 3, cute.rank(tTR_tAcc)) + bSG_gC = cute.group_modes(bSG_gC, 1, cute.rank(bSG_gC)) + + # + # Store accumulator to global memory in sub-tiles + # + subtile_cnt = cute.size(tTR_tAcc.shape, mode=[3]) + num_prev_subtiles = tile_sched.num_tiles_executed * subtile_cnt + + for subtile_idx in cutlass.range(subtile_cnt): + real_subtile_idx = subtile_idx + if cutlass.const_expr(self.overlapping_accum): + if reverse_subtile: + real_subtile_idx = ( + self.cta_tile_shape_mnk[1] // self.epi_tile_n - 1 - subtile_idx + ) + # + # Load accumulator from tensor memory buffer to register + # + tTR_tAcc_mn = tTR_tAcc[(None, None, None, real_subtile_idx)] + cute.copy(tiled_copy_t2r, tTR_tAcc_mn, tTR_rAcc) + + # + # Async arrive accumulator buffer empty earlier when overlapping_accum is enabled + # + if cutlass.const_expr(self.overlapping_accum): + if subtile_idx == self.iter_acc_early_release_in_epilogue: + # Fence for TMEM load + cute.arch.fence_view_async_tmem_load() + with cute.arch.elect_one(): + acc_pipeline.consumer_release(acc_consumer_state) + acc_consumer_state.advance() + # + # Convert to C type + # + acc_vec = tiled_copy_r2s.retile(tTR_rAcc).load() + # Multiply alpha in FP32 before converting to c_dtype to + # avoid overflow when c_dtype is FP16 and acc values are large. + acc_vec = epilogue_op((alpha_value * acc_vec).to(self.c_dtype)) + tRS_rC.store(acc_vec) + + # + # Store C to shared memory + # + c_buffer = (num_prev_subtiles + subtile_idx) % self.num_c_stage + cute.copy( + tiled_copy_r2s, + tRS_rC, + tRS_sC[(None, None, None, c_buffer)], + ) + # F3: the stock Lamport push's -0 word rule on the whole staged subtile (other threads' words too) + cute.arch.barrier( + barrier_id=self.epilog_sync_bar_id, + number_of_threads=f3_epi_threads, + ) + for f3_i in cutlass.range(epi_tidx, f3_stage_words, f3_epi_threads): + f3_j = c_buffer * f3_stage_words + f3_i + if f3_words[f3_j] == cutlass.Int32(-2147483648): + f3_words[f3_j] = cutlass.Int32(0) + # Fence and barrier to make sure shared memory store is visible to TMA store + cute.arch.fence_proxy( + "async.shared", + space="cta", + ) + epilog_threads = 32 * len(self.epilog_warp_id) + cute.arch.barrier( + barrier_id=self.epilog_sync_bar_id, + number_of_threads=epilog_threads, + ) + + # + # TMA store C to global memory + # + if warp_idx == self.epilog_warp_id[0]: + if f3_slot == 0: + cute.copy( + f3_s0l, bSG_sC[(None, c_buffer)], bSG_gC[(None, real_subtile_idx)] + ) + cute.copy( + f3_s0p, bSG_sC[(None, c_buffer)], bSG_gC[(None, real_subtile_idx)] + ) + elif f3_slot == 1: + cute.copy( + f3_s1l, bSG_sC[(None, c_buffer)], bSG_gC[(None, real_subtile_idx)] + ) + cute.copy( + f3_s1p, bSG_sC[(None, c_buffer)], bSG_gC[(None, real_subtile_idx)] + ) + else: + cute.copy( + f3_s2l, bSG_sC[(None, c_buffer)], bSG_gC[(None, real_subtile_idx)] + ) + cute.copy( + f3_s2p, bSG_sC[(None, c_buffer)], bSG_gC[(None, real_subtile_idx)] + ) + # Fence and barrier to make sure shared memory store is visible to TMA store + c_pipeline.producer_commit() + c_pipeline.producer_acquire() + cute.arch.barrier( + barrier_id=self.epilog_sync_bar_id, + number_of_threads=epilog_threads, + ) + + # + # Async arrive accumulator buffer empty + # + if cutlass.const_expr(not self.overlapping_accum): + with cute.arch.elect_one(): + acc_pipeline.consumer_release(acc_consumer_state) + acc_consumer_state.advance() + + # + # Advance to next tile + # + tile_sched.advance_to_next_work() + work_tile = tile_sched.get_current_work() + + # + # Dealloc the tensor memory buffer + # + if warp_idx == self.epilog_warp_id[0]: + cute.arch.relinquish_tmem_alloc_permit(is_two_cta=use_2cta_instrs) + epilog_threads = 32 * len(self.epilog_warp_id) + cute.arch.barrier(barrier_id=self.epilog_sync_bar_id, number_of_threads=epilog_threads) + if warp_idx == self.epilog_warp_id[0]: + if use_2cta_instrs: + cute.arch.mbarrier_arrive(tmem_dealloc_mbar_ptr, cta_rank_in_cluster ^ 1) + cute.arch.mbarrier_wait(tmem_dealloc_mbar_ptr, 0) + cute.arch.dealloc_tmem( + acc_tmem_ptr, self.num_tmem_alloc_cols, is_two_cta=use_2cta_instrs + ) + # + # Wait for C store complete + # + c_pipeline.producer_tail() + + griddepcontrol_launch_dependents() + + @staticmethod + def _compute_grid( + c: cute.Tensor, + cta_tile_shape_mnk: Tuple[int, int, int], + cluster_shape_mn: Tuple[int, int], + max_active_clusters: cutlass.Constexpr, + ) -> Tuple[utils.PersistentTileSchedulerParams, Tuple[int, int, int]]: + """Computes the grid size for the kernel launch. + + This method uses a persistent tile scheduler to determine the grid + dimensions based on the output tensor shape and hardware constraints. + + Args: + c (cute.Tensor): The output tensor C. + cta_tile_shape_mnk (Tuple[int, int, int]): The shape (M, N, K) of the + CTA tile. + cluster_shape_mn (Tuple[int, int]): The shape of a CTA cluster. + max_active_clusters (cutlass.Constexpr): The maximum number of + active clusters supported by the hardware. + + Returns: + A tuple containing the tile scheduler parameters and the computed + grid shape. + """ + c_shape = cute.slice_(cta_tile_shape_mnk, (None, None, 0)) + gc = cute.zipped_divide(c, tiler=c_shape) + num_ctas_mnl = gc[(0, (None, None, None))].shape + cluster_shape_mnl = (*cluster_shape_mn, 1) + + tile_sched_params = utils.PersistentTileSchedulerParams(num_ctas_mnl, cluster_shape_mnl) + grid = utils.StaticPersistentTileScheduler.get_grid_shape( + tile_sched_params, max_active_clusters + ) + + return tile_sched_params, grid + + @staticmethod + def is_valid_dtypes_and_scale_factor_vec_size( + ab_dtype: Type[cutlass.Numeric], + sf_dtype: Type[cutlass.Numeric], + sf_vec_size: int, + c_dtype: Type[cutlass.Numeric], + ) -> bool: + """Checks if the data types and scale factor vector size are a valid combination. + + Args: + ab_dtype (Type[cutlass.Numeric]): Data type of operands A and B. + sf_dtype (Type[cutlass.Numeric]): Data type of the scale factors. + sf_vec_size (int): Vector size of the scale factors. + c_dtype (Type[cutlass.Numeric]): Data type of the output tensor C. + + Returns: + bool: True if the combination is valid, False otherwise. + """ + is_valid = True + + # Check valid ab_dtype + if ab_dtype not in { + cutlass.Float4E2M1FN, + cutlass.Float8E5M2, + cutlass.Float8E4M3FN, + }: + is_valid = False + + # Check valid sf_vec_size + if sf_vec_size not in {16, 32}: + is_valid = False + + # Check valid sf_dtype + if sf_dtype not in {cutlass.Float8E8M0FNU, cutlass.Float8E4M3FN}: + is_valid = False + + # Check valid sf_dtype and sf_vec_size combinations + if sf_dtype == cutlass.Float8E4M3FN and sf_vec_size == 32: + is_valid = False + if ab_dtype in {cutlass.Float8E5M2, cutlass.Float8E4M3FN} and sf_vec_size == 16: + is_valid = False + + # Check valid c_dtype + if c_dtype not in { + cutlass.Float32, + cutlass.Float16, + cutlass.BFloat16, + cutlass.Float8E5M2, + cutlass.Float8E4M3FN, + }: + is_valid = False + + return is_valid + + @staticmethod + def is_valid_layouts( + ab_dtype: Type[cutlass.Numeric], + c_dtype: Type[cutlass.Numeric], + a_major: str, + b_major: str, + c_major: str, + ) -> bool: + """Checks if the tensor layouts are valid for the given data types. + + Args: + ab_dtype (Type[cutlass.Numeric]): Data type of operands A and B. + c_dtype (Type[cutlass.Numeric]): Data type of the output tensor C. + a_major (str): The major layout of tensor A ('k' or 'm'). + b_major (str): The major layout of tensor B ('k' or 'n'). + c_major (str): The major layout of tensor C ('n' or 'm'). + + Returns: + bool: True if the layouts are valid, False otherwise. + """ + is_valid = True + + if ab_dtype is cutlass.Float4E2M1FN and not (a_major == "k" and b_major == "k"): + is_valid = False + return is_valid + + @staticmethod + def is_valid_mma_tiler_and_cluster_shape( + mma_tiler_mn: Tuple[int, int], + cluster_shape_mn: Tuple[int, int], + ) -> bool: + """Checks if the MMA tiler and cluster shape are a valid combination. + + Args: + mma_tiler_mn (Tuple[int, int]): The (M, N) shape of the MMA tiler. + cluster_shape_mn (Tuple[int, int]): The (M, N) shape of the CTA cluster. + + Returns: + bool: True if the combination is valid, False otherwise. + """ + is_valid = True + # Skip invalid mma tile shape + if mma_tiler_mn[0] not in [128, 256]: + is_valid = False + if mma_tiler_mn[1] not in [8, 16, 32, 64, 128, 192, 256]: + is_valid = False + # Skip illegal cluster shape + if cluster_shape_mn[0] % (2 if mma_tiler_mn[0] == 256 else 1) != 0: + is_valid = False + # Skip invalid cluster shape + _is_power_of_2 = lambda x: x > 0 and (x & (x - 1)) == 0 + if ( + cluster_shape_mn[0] * cluster_shape_mn[1] > 16 + or cluster_shape_mn[0] <= 0 + or cluster_shape_mn[1] <= 0 + # Special cluster shape check for scale factor multicasts. + # Due to limited size of scale factors, we can't multicast among more than 4 CTAs. + or cluster_shape_mn[0] > 4 + or cluster_shape_mn[1] > 4 + or not _is_power_of_2(cluster_shape_mn[0]) + or not _is_power_of_2(cluster_shape_mn[1]) + ): + is_valid = False + return is_valid + + @staticmethod + def is_valid_tensor_alignment( + m: cutlass.Int64, + n: cutlass.Int64, + k: cutlass.Int64, + l: cutlass.Int64, + ab_dtype: Type[cutlass.Numeric], + c_dtype: Type[cutlass.Numeric], + a_major: str, + b_major: str, + c_major: str, + ) -> bool: + """Checks if the tensor dimensions are valid for memory alignment. + + Args: + m (cutlass.Int64): The M dimension of the GEMM problem. + n (cutlass.Int64): The N dimension of the GEMM problem. + k (cutlass.Int64): The K dimension of the GEMM problem. + l (cutlass.Int64): The batch dimension (L) of the GEMM problem. + ab_dtype (Type[cutlass.Numeric]): Data type of operands A and B. + c_dtype (Type[cutlass.Numeric]): Data type of the output tensor C. + a_major (str): The major layout of tensor A ('k' or 'm'). + b_major (str): The major layout of tensor B ('k' or 'n'). + c_major (str): The major layout of tensor C ('n' or 'm'). + + Returns: + bool: True if the tensor alignment is valid, False otherwise. + """ + is_valid = True + + def check_contigous_16B_alignment(dtype, is_mode0_major, tensor_shape): + major_mode_idx = 0 if is_mode0_major else 1 + num_major_elements = tensor_shape[major_mode_idx] + num_contiguous_elements = 16 * 8 // dtype.width + return num_major_elements % num_contiguous_elements == 0 + + if ( + not check_contigous_16B_alignment(ab_dtype, a_major == "m", (m, k, l)) + or not check_contigous_16B_alignment(ab_dtype, b_major == "n", (n, k, l)) + or not check_contigous_16B_alignment(c_dtype, c_major == "m", (m, n, l)) + ): + is_valid = False + return is_valid + + @classmethod + def can_implement( + cls, + ab_dtype: Type[cutlass.Numeric], + sf_dtype: Type[cutlass.Numeric], + sf_vec_size: int, + c_dtype: Type[cutlass.Numeric], + mma_tiler_mn: Tuple[int, int], + cluster_shape_mn: Tuple[int, int], + m: cutlass.Int64, + n: cutlass.Int64, + k: cutlass.Int64, + l: cutlass.Int64, + a_major: str, + b_major: str, + c_major: str, + ) -> bool: + """Checks if the kernel can implement the given GEMM problem. + + This method aggregates checks for data types, layouts, tile shapes, + and alignment to determine if the configuration is supported. + + Args: + ab_dtype (Type[cutlass.Numeric]): Data type of operands A and B. + sf_dtype (Type[cutlass.Numeric]): Data type of the scale factors. + sf_vec_size (int): Vector size of the scale factors. + c_dtype (Type[cutlass.Numeric]): Data type of the output tensor C. + mma_tiler_mn (Tuple[int, int]): The (M, N) shape of the MMA tiler. + cluster_shape_mn (Tuple[int, int]): The (M, N) shape of the CTA + cluster. + m (cutlass.Int64): The M dimension of the GEMM problem. + n (cutlass.Int64): The N dimension of the GEMM problem. + k (cutlass.Int64): The K dimension of the GEMM problem. + l (cutlass.Int64): The batch dimension (L) of the GEMM problem. + a_major (str): The major layout of tensor A ('k' or 'm'). + b_major (str): The major layout of tensor B ('k' or 'n'). + c_major (str): The major layout of tensor C ('n' or 'm'). + + Returns: + bool: True if the configuration is supported, False otherwise. + """ + can_implement = True + # Skip unsupported types + if not cls.is_valid_dtypes_and_scale_factor_vec_size( + ab_dtype, sf_dtype, sf_vec_size, c_dtype + ): + can_implement = False + # Skip unsupported layouts + if not cls.is_valid_layouts(ab_dtype, c_dtype, a_major, b_major, c_major): + can_implement = False + # Skip invalid mma tile shape and cluster shape + if not cls.is_valid_mma_tiler_and_cluster_shape(mma_tiler_mn, cluster_shape_mn): + can_implement = False + # Skip illegal problem shape for load/store alignment + if not cls.is_valid_tensor_alignment( + m, n, k, l, ab_dtype, c_dtype, a_major, b_major, c_major + ): + can_implement = False + + if mma_tiler_mn[1] < 64 and (n > mma_tiler_mn[1] or cluster_shape_mn[1] > 1): + can_implement = False + return can_implement + + @cute.jit + def f3_wrapper( + self, + mA: cute.Tensor, + mB: cute.Tensor, + mC: cute.Tensor, + sf_m: cutlass.Int64, + sf_n: cutlass.Int64, + sf_k: cutlass.Int64, + l: cutlass.Constexpr, + a_sf_ptr: cute.Pointer, + b_sf_ptr: cute.Pointer, + alpha_tensor: cute.Tensor, + p_s0l: cute.Pointer, + p_s0p: cute.Pointer, + p_s1l: cute.Pointer, + p_s1p: cute.Pointer, + p_s2l: cute.Pointer, + p_s2p: cute.Pointer, + p_flag: cute.Pointer, + max_active_clusters: cutlass.Constexpr, + current_stream, + swap_ab: cutlass.Constexpr = False, + epilogue_op: cutlass.Constexpr = lambda x: x, + ): + """The common wrapper (MXFP8 path) plus 6 Lamport slot tensors with C's layout and the flag tensor.""" + m = cute.size(mA, mode=[0]) + k = cute.size(mA, mode=[1]) + n = cute.size(mB, mode=[0]) + a_tensor = cute.make_tensor( + mA.iterator, layout=cute.make_ordered_layout((m, k, l), order=(1, 0, 2)) + ) + b_tensor = cute.make_tensor( + mB.iterator, layout=cute.make_ordered_layout((n, k, l), order=(1, 0, 2)) + ) + if cutlass.const_expr(swap_ab): + c_order = (0, 1, 2) + else: + c_order = (1, 0, 2) + c_tensor = cute.make_tensor( + mC.iterator, layout=cute.make_ordered_layout((m, n, l), order=c_order) + ) + slots = [ + cute.make_tensor(p, layout=cute.make_ordered_layout((m, n, l), order=c_order)) + for p in (p_s0l, p_s0p, p_s1l, p_s1p, p_s2l, p_s2p) + ] + flag_tensor = cute.make_tensor(p_flag, layout=cute.make_layout(1)) + sfa_tensor = cute.make_tensor( + a_sf_ptr, + layout=cute.make_ordered_layout((32, 4, sf_m, 4, sf_k, l), order=(2, 1, 4, 0, 3, 5)), + ) + sfb_tensor = cute.make_tensor( + b_sf_ptr, + layout=cute.make_ordered_layout((32, 4, sf_n, 4, sf_k, l), order=(2, 1, 4, 0, 3, 5)), + ) + self( + a_tensor, + b_tensor, + sfa_tensor, + sfb_tensor, + c_tensor, + alpha_tensor, + max_active_clusters, + current_stream, + slots[0], + slots[1], + slots[2], + slots[3], + slots[4], + slots[5], + flag_tensor, + epilogue_op, + ) + + +# --------------------------------------------------------------------------------------------------------------------- +# F3: the launcher the TP2 modeling uses. The same compile / launch as the standalone microbenchmarks. +# One compiled kernel per pinned tactic (mxfp8_cute_pinned_tactics; non-split-K only); the 6 Lamport slot +# pointers (3 rotating slots x local / peer section of this rank) and the flag address come from the AUTO all-reduce's +# workspace and are plain kernel params (graph-capture-safe: the workspace never moves; the slot rotates on the device). +# --------------------------------------------------------------------------------------------------------------------- + + +FI_BASE_MD5 = "99892be684e2b5af81662ed0768ce9bb" + + +def fi_base_ok() -> bool: + """The installed FlashInfer dense GEMM file is the fork's base (else F3 declines).""" + try: + import flashinfer + + p = _os.path.join( + _os.path.dirname(flashinfer.__file__), + "gemm", + "kernels", + "dense_blockscaled_gemm_sm100.py", + ) + return _hashlib.md5(open(p, "rb").read()).hexdigest() == FI_BASE_MD5 + except Exception: # noqa: BLE001 + return False + + +class F3GemmLauncher: + NR = 2 + + def __init__(self, workspace: _torch.Tensor, rank: int, hidden: int, k: int, peek_i64): + """``workspace``: get_allreduce_workspace(mapping) (int64 device pointers); ``peek_i64``: reads 8 bytes at a + device address (the comm size lives in device memory).""" + self.rank, self.hidden, self.k = rank, hidden, k + self.wsp = [int(v) for v in workspace.cpu().tolist()] + self.comm_size = int(peek_i64(self.wsp[self.NR * 3 + 1] + 8)) + self.flag_addr = self.wsp[self.NR * 3] + 8 # int index 2 = the Lamport flag + self._compiled = {} + self._alpha = {} + + def slots(self, rows: int): + base = self.rank * rows * self.hidden * 2 + return [ + self.wsp[2 * self.NR + d] + s * self.comm_size + base + for s in range(3) + for d in (self.rank, 1 - self.rank) + ] + + def compile(self, tactic): + import cutlass + import cutlass.cute as cute + from cutlass.cute.runtime import make_ptr + from flashinfer.cute_dsl.utils import get_max_active_clusters + + mma_tiler_mn, cluster_mn, swap_ab, use_prefetch, split_k = tactic + assert split_k == 1 + key = (tuple(mma_tiler_mn), tuple(cluster_mn), bool(swap_ab), bool(use_prefetch)) + if key in self._compiled: + return self._compiled[key] + gemm = F3LamportGemmKernel( + 32, tuple(mma_tiler_mn), tuple(cluster_mn), bool(use_prefetch), True + ) + mac = get_max_active_clusters(cluster_mn[0] * cluster_mn[1]) + sm, sk, sn = cute.sym_int(), cute.sym_int(), cute.sym_int() + a_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Float8E4M3FN, (sm, sk), stride_order=(1, 0), assumed_align=16 + ) + b_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Float8E4M3FN, (sn, sk), stride_order=(1, 0), assumed_align=16 + ) + if swap_ab: + c_fake = cute.runtime.make_fake_compact_tensor( + cutlass.BFloat16, (sn, sm), stride_order=(0, 1), assumed_align=16 + ) + else: + c_fake = cute.runtime.make_fake_compact_tensor( + cutlass.BFloat16, (sm, sn), stride_order=(1, 0), assumed_align=16 + ) + sfp = [make_ptr(cutlass.Float8E8M0FNU, 16, cute.AddressSpace.gmem, 16) for _ in range(2)] + cps = [make_ptr(cutlass.BFloat16, 16, cute.AddressSpace.gmem, 16) for _ in range(6)] + fp = make_ptr(cutlass.Int32, 16, cute.AddressSpace.gmem, 4) + alpha_fake = cute.runtime.make_fake_compact_tensor(cutlass.Float32, (1,), assumed_align=4) + stream_fake = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True) + # The kernel class comes from FlashInfer, so its version is part of the shipped object's key. + fn = cute_aot.compile_or_load( + f"m31_f3_gemm {key} mac={mac} flashinfer={_flashinfer_version()}", + lambda: cute.compile( + gemm.f3_wrapper, + a_fake, + b_fake, + c_fake, + 1, + 1, + 1, + 1, + sfp[0], + sfp[1], + alpha_fake, + *cps, + fp, + mac, + stream_fake, + swap_ab, + options="--opt-level 2 --enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + self._compiled[key] = fn + return fn + + def run(self, tactic, act, act_scale, w, w_scale, dummy_out, alpha=None): + """``act @ w^T`` (MXFP8, the pinned tactic) stored into this rank's section of the current Lamport slot on + both ranks, with the stock push's -0 word rule. ``dummy_out``: a [rows, hidden] bf16 tensor (shape only).""" + fn = self.compile(tactic) + rows = act.shape[0] + swap_ab = tactic[2] + if swap_ab: + km, kn, ka, kb, kasf, kbsf = self.hidden, rows, w, act, w_scale, act_scale + else: + km, kn, ka, kb, kasf, kbsf = rows, self.hidden, act, w, act_scale, w_scale + sf_m, sf_n, sf_k = (km + 127) // 128, (kn + 127) // 128, (self.k // 32 + 3) // 4 + out = ( + dummy_out.as_strided(dummy_out.shape, (1, dummy_out.shape[0])) if swap_ab else dummy_out + ) + if alpha is None: + alpha = self._alpha.get(act.device) + if alpha is None: + alpha = self._alpha[act.device] = _torch.ones( + 1, dtype=_torch.float32, device=act.device + ) + fn( + ka, + kb, + out, + sf_m, + sf_n, + sf_k, + kasf.data_ptr(), + kbsf.data_ptr(), + alpha, + *self.slots(rows), + self.flag_addr, + ) + + +def _flashinfer_version() -> str: + import importlib.metadata as md + + return md.version("flashinfer-python") + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_attn_lc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_attn_lc.py new file mode 100644 index 000000000000..79c131e5f719 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_attn_lc.py @@ -0,0 +1,182 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 long-context sparse decode attention in one launch (LC2: the M3 sm_107 TP2 target's kernel, ported). + +Replaces the generic chain's split-K producer (``_gqa_sparse_decode_kernel``, one block per chunk) and the LSE +merge (``_merge_topk_attn_out_kernel``, with its MXFP8 copy for the o_proj) once the top-k block table exists. + +Grid: one cluster of 16 CTAs per (query token, KV head); CTA ``r`` attends the token's ``r``-th selected block of +that head (the Triton producer's chunk ``r`` at 16 chunks), keeps the partial in shared memory, and after a cluster +barrier merges head ``r`` of the KV head's group from all 16 partials over DSMEM, writing the BF16 row and, +optionally, its MXFP8 copy and swizzled UE8M0 scales, bitwise ``torch.ops.trtllm.mxfp8_quantize(out, True)``. + +Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and +PV on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +log-domain recurrence. Accumulation order inside the MMAs and the reductions differs, so results match to rounding, +not bit for bit. + +Schedule: the request length and the query rows (FP8 widened to FP16 in shared memory) are read before the PDL grid +wait, which overlaps them with the select ahead when it releases this grid early (modeling's ``SELECT_EARLY_MAX_ROWS``); +the split-K partials travel by ``st.async`` into each receiving CTA's own mbarrier instead of a cluster-wide barrier. +Neither changes a value or the merge order. + +Geometry: 16 query heads per KV head, two KV heads per rank (TP2), head_dim 128, page 128, top-k 16. The kernel +is ``torch.ops.trtllm.minimax_m3_sm107_tp2_sparse_decode_lc``, called through this target's ``m3_sparse_decode_lc`` +entry. + +M3.1 TP2 (TP2WIDE_LC2): ``triton_sparse_decode`` routes only pure generation steps with a selected top-16 table at +``LC_MIN_ROWS``..``LC_MAX_ROWS`` (448) query rows here; narrower steps keep the Triton producer + merge. +""" + +import torch + +from tensorrt_llm.logger import logger + +from ..m31_switches import m31_disabled +from ..ops.attention.m3_sparse_decode_lc import m3_sparse_decode_lc + + +class N1Slot: + """The sparse layer's N1 id buffer (modeling's ``SPARSE_DECODE_LC_PREFETCH_SELF_KV``), set by the layer around its + decode call and taken by the long-context decode; empty means no prefetch.""" + + def __init__(self) -> None: + self.ids = None + + def take(self): + ids, self.ids = self.ids, None + return ids + + +# int32 [2, N1_MAX_ROWS (= LC_MAX_ROWS), 16]: the layer's selected block ids per (KV head, query row, slot) at its +# previous decode step. +N1_IDS = N1Slot() + +# Carried from the TP4 base (LC4's one-division split-K merge), with its switch contract: up to this many query rows +# the merge would divide once per output. TRTLLM_M31_DISABLE_ONEDIV_CAP=1 keeps ONE_DIV at every row count; +# TRTLLM_M31_DISABLE_LC_ONEDIV=1 turns it off at every row count, and wins over the cap switch. Inert at TP2: the LC2 +# op (minimax_m3_sm107_tp2_sparse_decode_lc) has no one-division variant, so ``m3_lc_attn`` never reads it. +ONE_DIV_MAX_ROWS = ( + 0 if m31_disabled("LC_ONEDIV") else 1 << 30 if m31_disabled("ONEDIV_CAP") else 16 +) # capacity: fixed: measured crossover (TP4) + + +def one_div_applies(total_q: int) -> bool: + """Whether the (TP4) merge divides once at ``total_q`` query rows.""" + return total_q <= ONE_DIV_MAX_ROWS + + +def lc_applies(q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output) -> bool: + """Whether the step's operands fit the kernel: TP2 geometry, packed NVFP4 pages, FP8 q, BF16 [T, 32, 128] out.""" + if q.dim() != 3 or tuple(q.shape[1:]) != (32, 128) or q.dtype != torch.float8_e4m3fn: + return False + if q.stride(2) != 1 or q.stride(1) % 16 or q.stride(0) % 16 or q.data_ptr() % 16: + return False + if ( + int(k_paged.shape[1]) != 2 + or topk_idx.dim() != 3 + or int(topk_idx.shape[0]) != 2 + or int(topk_idx.shape[1]) != int(q.shape[0]) + or int(topk_idx.shape[-1]) != 16 + or topk_idx.dtype != torch.int32 + or topk_idx.stride(2) != 1 + ): + return False + if v_paged.stride() != k_paged.stride(): + return False + for t in (k_paged, v_paged): + if ( + t.stride(3) != 1 + or t.stride(2) != 64 + or int(t.shape[3]) != 64 + or t.data_ptr() % 16 + or t.stride(0) % 16 + or t.stride(1) % 16 + ): + return False + for t in (k_block_scale, v_block_scale): + if ( + t is None + or int(t.shape[1]) != 2 + or t.stride(3) != 1 + or t.stride(2) != 8 + or t.data_ptr() % 8 + or t.stride(0) % 8 + or t.stride(1) % 8 + ): + return False + if v_block_scale.stride() != k_block_scale.stride(): + return False + return ( + output.dtype == torch.bfloat16 + and output.dim() == 3 + and output.is_contiguous() + and tuple(output.shape) == tuple(q.shape) + ) + + +def m3_lc_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + linear_softmax=False, + mxfp8=None, + use_pdl=True, + prewait=False, + n1_ids=None, +): + """Drop-in for ``minimax_m3_sparse_attn_decode`` on the TP2 NVFP4 shape; callers check ``lc_applies`` first. + + ``topk_idx`` is the head-major [2, total_q, 16] table the Triton producer takes. ``mxfp8`` is ``(q8, sf)`` from + ``MERGE_MXFP8.buffers(4096, dev)`` to also emit the o_proj hand-off. ``n1_ids`` is the layer's N1 id buffer + (``N1_IDS``): the decode prefetches last step's blocks into L2 and records this step's. + """ + total_q = int(q.shape[0]) + q8, sf = ( + (mxfp8[0].view(torch.uint8), mxfp8[1].view(torch.uint8)) + if mxfp8 is not None + else (None, None) + ) + m3_sparse_decode_lc( + q.view(torch.uint8), + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + output.view(total_q, 4096), + q8, + sf, + 4096 // 128, + int(decode_query_len), + float(sm_scale), + bool(linear_softmax), + bool(use_pdl), + bool(prewait), + n1_ids, + ) + key = (total_q, q8 is not None, n1_ids is not None) + if key not in _engaged: + _engaged.add(key) + logger.info( + f"[M3.1 sm_107 tp2] sparse decode Lc engaged: T={total_q}, kv_heads=2, mxfp8_handoff={q8 is not None}, " + f"n1_self_kv={n1_ids is not None}" + ) + + +_engaged = set() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_draft_lowm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_draft_lowm.py new file mode 100644 index 000000000000..00c6c6887575 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_draft_lowm.py @@ -0,0 +1,143 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter's BF16 linears on FlashInfer's fixed-tactic low-M kernels (``DRAFT_LOWM``). + +The drafter's bias-free BF16 projections (qkv, o_proj, gate_up, down, the fc; not the head) call ``draft_linear`` +first: a covered row count runs ``ops/gemm/m3_draft_lowm`` at the tactic of its (N, K) table, every other call the +plain GEMM. ``prebuild`` compiles every covered call before the executor's warmup; a forward never compiles one, and a +covered call it finds uncompiled is a miss. +""" + +from __future__ import annotations + +from typing import FrozenSet, Iterable, Optional, Set, Tuple + +import torch + +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm.logger import logger + +from ..ops.gemm.m3_draft_lowm import m3_draft_lowm, tactic_for +from . import m3_report + +PATH = "m3_draft_lowm" + + +# ((N, K), M) of every call the target covers (its DRAFT_LOWM keys), set by prebuild. +_KEYS: FrozenSet[Tuple[Tuple[int, int], int]] = frozenset() +# (N, K, M, PDL) of every call prebuild compiled. +_BUILT: Set[Tuple[int, int, int, bool]] = set() +# (N, K, M) of every call that logged its engagement line. +_ENGAGED: Set[Tuple[int, int, int]] = set() +# DRAFT_AR_PUSH: the weight whose next low-M split-K call also pushes its rows into a boundary all-reduce +# (data pointer, the bucket's multicast words, its flags, rank, world, max tokens), set by arm_push; and whether the +# last armed call pushed: the data pointer of the output it pushed, 0 if none (read once by take_pushed). Host state +# set and read in one forward's issue order. +_PUSH_ARM: Optional[tuple] = None +_PUSHED = 0 + + +def arm_push(weight: torch.Tensor, workspace, rows: int) -> None: + """Arm ``weight``'s next covered split-K call to push into ``workspace``'s bucket for ``rows`` (see _PUSH_ARM).""" + global _PUSH_ARM, _PUSHED + from . import m3_mc_allreduce + + bucket = m3_mc_allreduce._bucket(workspace, rows) + flags = None if bucket is None else bucket.flags_for(8) + _PUSHED = 0 + _PUSH_ARM = None + if bucket is not None and flags is not None and bucket.max_tokens == 8: + _PUSH_ARM = ( + weight.data_ptr(), + bucket.mc.data_ptr(), + flags.data_ptr(), + bucket.rank, + bucket.world_size, + bucket.max_tokens, + ) + + +def take_pushed(x: torch.Tensor) -> bool: + """Whether the armed call pushed its rows; disarms. ``x`` is what the boundary all-reduce is about to reduce: it + must be the pushing call's own output (the prepushed all-reduce reads only the pushed words, never ``x``), so an + operation between that call and the all-reduce raises instead of silently reducing the wrong rows.""" + global _PUSH_ARM, _PUSHED + pushed, _PUSHED, _PUSH_ARM = _PUSHED, 0, None + if pushed and x.data_ptr() != pushed: + raise RuntimeError( + "the drafter's pushed rows are not the boundary all-reduce's input (an operation ran in between)" + ) + return bool(pushed) + + +def draft_linear( + weight: torch.Tensor, x: torch.Tensor, bias: Optional[torch.Tensor] +) -> Optional[torch.Tensor]: + """x [M, K] @ weight^T through the low-M kernel when the target covers it (``prebuild``'s keys); None for the + plain GEMM. The split-K ('s') tactics run this target's copy of FlashInfer's kernel (kernels/m3_lowm_splitk, bit + for bit the same output at every certified cell), which also carries the drafter-boundary push epilogue.""" + if bias is not None or x.dim() != 2: + return None + n, k = weight.shape + m = int(x.shape[0]) + if ((n, k), m) not in _KEYS: + m3_report.uncovered(PATH, f"the drafter's {n}x{k} linear past its tactic table's rows") + return None + if not x.is_contiguous(): + # The first pass's fc may read a column window of the captured hidden states; the plain GEMM takes strides. + m3_report.uncovered(PATH, "a strided activation (the plain GEMM reads it in place)") + return None + use_pdl = get_env_enable_pdl() + if x.dtype != torch.bfloat16 or weight.dtype != torch.bfloat16: + m3_report.missed(PATH, m, "the activation or weight is not BF16") + return None + if (n, k, m, use_pdl) not in _BUILT: + m3_report.missed(PATH, m, f"the {n}x{k} kernel at {m} rows was not compiled before warmup") + return None + kind, tactic = tactic_for(n, k, m) + if (n, k, m) not in _ENGAGED: + _ENGAGED.add((n, k, m)) + logger.info( + f"[M3 sm_107 tp4] fast path engaged: m3_draft_lowm {n}x{k} at {m} rows (device {x.device.index})" + ) + out = torch.empty((m, n), dtype=x.dtype, device=x.device) + global _PUSHED + ar = None + if _PUSH_ARM is not None and _PUSH_ARM[0] == weight.data_ptr() and kind == "s": + ar = _PUSH_ARM[1:] + _PUSHED = out.data_ptr() + return m3_draft_lowm(x, weight, out, kind, tactic, use_pdl, True, ar) + + +def installed(weight: torch.Tensor) -> bool: + """Whether the target covers ``weight`` (bias-free BF16 [N, K]) at some row count (``prebuild``'s keys).""" + if weight.dtype != torch.bfloat16 or weight.dim() != 2: + return False + return any(nk == tuple(weight.shape) for nk, _ in _KEYS) + + +def prebuild(weights: Iterable[torch.Tensor], keys: Iterable[Tuple[Tuple[int, int], int]]) -> int: + """Take the target's covered ((N, K), M) ``keys``, then compile (and run once) every covered row count of every + covered drafter weight, outside capture, on the current stream. Returns the number of calls compiled.""" + global _KEYS + _KEYS = frozenset(keys) + use_pdl = get_env_enable_pdl() + built = 0 + shapes = {tuple(weight.shape): weight for weight in weights if installed(weight)} + for (n, k), weight in sorted(shapes.items()): + for nk, m in sorted(key for key in _KEYS if key[0] == (n, k)): + if (n, k, m, use_pdl) in _BUILT: + continue + kind, tactic = tactic_for(n, k, m) + x = torch.zeros(m, k, dtype=torch.bfloat16, device=weight.device) + out = torch.empty(m, n, dtype=torch.bfloat16, device=weight.device) + m3_draft_lowm(x, weight, out, kind, tactic, use_pdl, True) + if kind == "s": + # The push variant a boundary all-reduce may arm (o_proj / down): compiled only, never run here + # (its pushes need the all-reduce's exchange and a matching call). + from .m3_lowm_splitk import compile_push + + compile_push(x, weight.t(), out, use_pdl, tactic) + _BUILT.add((n, k, m, use_pdl)) + built += 1 + return built diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_draft_step.py new file mode 100644 index 000000000000..45a64dfe9210 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_draft_step.py @@ -0,0 +1,222 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 draft pass for the MiniMax-M3 drafter's vocab-sharded BF16 head (TP decode, <= 8 requests). + +One kernel replaces a draft pass's head and the draft loop's per-pass bookkeeping, at the in-tree precision: + +* The drafter's BF16 lm_head shard is kept a second time, built by ``prepare_head`` before the executor's warmup: in + the lossless 12-bit coding of the target head (``ops.gemm.m3_target_head``; 75% of the bytes, kept only when it + decodes to the BF16 shard bit for bit), else in ``mma.m16n8k16`` A-fragment order (``bf16_fragments``, the same + values). The kernel streams it with BF16 MMAs and FP32 accumulation over the BF16 head input rows; a coded + exception decodes to +0 and its exact product x * w is added at its tile's flush. +* Each logit is rounded to BF16 (the in-tree logits are the BF16 head GEMM's output) and keyed + ``ordered(value) << 32 | ~global_index`` with -0 as +0, so the larger value wins and ties take the lowest index, as + the in-tree ``torch.max`` / ``torch.argmax`` over the gathered logits (``greedy_sample_draft_with_tp_gather``). +* The last CTA exchanges each row's key across the TP ranks through a multicast mapping (Lamport slots, see + ``m3_shard_accept``), then stores the draft token, the next pass's input id and position, and advances the draft KV + lengths as the in-tree draft loop does between passes. + +The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/draftStep.cu) and called through the +catalog entry ``ops.spec.m3_draft_step``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +import torch + +from ..ops.gemm.m3_target_head import encode_chunked, round_trip_ok +from ..ops.spec.m3_draft_step import MAX_TILES_PER_CTA, MAX_TOKENS, bf16_fragments, max_tile_pieces +from ..ops.spec.m3_draft_step import m3_draft_step as _draft_step_op +from . import m3_report +from .m3_shard_accept import AcceptWorkspace, create_workspace + +# The draft step launches this many CTAs instead of one per SM (3126 tiles: 200 CTAs x 15 + 9 x 14, at most 2 warp +# pieces per tile), so the CTAs own near-equal whole-tile counts and the stream does not end on the tiles' straggler +# tail (-1.04 / -1.18 us per pass at 1 / 4 rows vs 212 on Rubin TP4). A different static warp segmentation: +# exact-class. +GRID = 209 # capacity: fixed: 3126 head tiles of the TP4 shard, at most 2 warp pieces per tile on 209 CTAs +# The draft head kernel sums a tile that several warps stream in warp order (fixed by the launch shape) instead of by +# shared-memory atomics in arrival order: with fewer than 16 tiles per CTA a tile can take 3-4 pieces and the atomics' +# order changed the fp32 sum run to run; with 2 pieces the sum is the atomics' bit for bit. +FIXED_ORDER_PATH = "m3_draft_step_fixed_order" + + +@dataclass(eq=False) +class DraftHead: + """The streamed copy of one rank's draft lm_head shard and its first global id: ``wf`` is the 12-bit coded stream + with its exception lists (``coded``), or the fragment-ordered BF16 copy (``bf16_fragments``).""" + + weight: torch.Tensor + wf: torch.Tensor + vocab_offset: int + coded: bool = False + exc_off: Optional[torch.Tensor] = None + exc_pos: Optional[torch.Tensor] = None + exc_w: Optional[torch.Tensor] = None + exp_base: int = 0 + + +def head_supported(lm_head) -> bool: + """A bias-free BF16 head whose local shard the kernel can own: whole 16-row tiles, K a multiple of 16, at most + MAX_TILES_PER_CTA tiles per CTA of a one-CTA-per-SM grid.""" + weight = getattr(lm_head, "weight", None) + if weight is None or weight.dim() != 2 or weight.dtype != torch.bfloat16: + return False + if getattr(lm_head, "bias", None) is not None: + return False + n, k = weight.shape + if n % 16 or k % 16: + return False + return (n // 16 + _grid() - 1) // _grid() <= MAX_TILES_PER_CTA + + +def prepare_head(lm_head, tp_rank: int, coded: bool) -> DraftHead: + """The streamed copy of ``lm_head``'s local shard (global ids from tp_rank * rows): its 12-bit coding when + ``coded``, K is a multiple of 64 and the coding decodes to the shard bit for bit, else (fail + closed) the fragment-ordered BF16 copy. + Built once, eagerly, before the executor's warmup (``M3Eagle3Worker.prebuild_draft_head``).""" + weight = lm_head.weight + offset = tp_rank * weight.shape[0] + with torch.no_grad(): + w = weight.detach() + if coded and w.shape[1] % 64 == 0: + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + if round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w): + return DraftHead( + weight=weight, + wf=packed, + vocab_offset=offset, + coded=True, + exc_off=exc_off, + exc_pos=exc_pos, + exc_w=exc_w.view(-1), + exp_base=base, + ) + del packed, exc_off, exc_pos, exc_w + wf = bf16_fragments(w) + return DraftHead(weight=weight, wf=wf, vocab_offset=offset) + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _grid() -> int: + """The draft step's CTA count: ``GRID`` (at most one per SM).""" + return min(GRID, _num_sms()) + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace ([2][MAX_TOKENS][world] slots), created collectively on the first eager call; None + during a CUDA-graph capture that finds it missing.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = create_workspace(mapping, device, MAX_TOKENS) + return _WS + + +def _fixed_order( + rows: int, head: DraftHead, x: torch.Tensor, num_ctas: int, split: bool = False +) -> bool: + """The fixed warp-order tile sum where the launch lets a tile take more than 2 warp pieces (max_tile_pieces; a + split or equal-chunk launch always), with its engagement line; False keeps the atomics, deterministic on this + shape.""" + eq = getattr(head, "bnd", None) is not None + if not (split or eq): + chunks = x.shape[1] // (16 if head.exc_off is None else 64) + if ( + max_tile_pieces( + head.wf.shape[0] if head.exc_off is None else head.exc_off.numel() - 1, + num_ctas, + chunks, + ) + <= 2 + ): + return False + m3_report.taken( + FIXED_ORDER_PATH, + rows, + "the draft head sums shared tiles in warp order", + ) + return True + + +def draft_step( + head: DraftHead, + x: torch.Tensor, + rows: int, + x_rows: Optional[torch.Tensor], + mapping, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, +) -> bool: + """One fused greedy draft pass over ``rows`` requests (see the module docstring); False when the exchange + workspace is missing during a capture. Collective over the TP group. + + Args: + head: the prepared head of this rank (``prepare_head``), coded or BF16 fragments. + x: [*, K] BF16 head input rows (the drafter's final-norm output); row ``x_rows[b]`` (int64) or ``b`` is + request b's. + draft_out: int32 [rows, >= step + 1], unit column stride: column ``step`` receives the tokens. + ids_next / pos_next: int32 [rows]: the tokens, and ``pos_src[x_rows[b] or b] + 1`` (pos_src int32). + kv_lens: int32 draft KV lengths: ``-= draft_len - num_acc`` with ``seq_lens = 1`` on pass 0 when ``num_acc`` + is given, ``+= 1`` on later passes. + """ + ws = workspace(mapping, x.device) + if ws is None: + return False + _draft_step_op( + x, + x_rows, + rows, + head.wf, + head.vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + _grid(), + True, + None, + head.exc_off, + head.exc_pos, + head.exc_w, + head.exp_base, + None, # no fold tail: the forward's tail runs the standalone one + fixed_order=_fixed_order(rows, head, x, _grid()), + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_drafter_attn.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_drafter_attn.py new file mode 100644 index 000000000000..aff8d752364f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_drafter_attn.py @@ -0,0 +1,472 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter layer's decode attention on Rubin: ``M3DrafterFullCtxFmha``. + +The drafter layer is a plain ``TrtllmAttention`` over its own FP8 paged cache (the full context, every draft pass). +On SM 10.7 its generation phase runs ``thop.attention``'s in-tree trtllm-gen runner, whose split is capped at 8 CTAs +per (request, KV head): at TP4 (1 KV head per rank) 45.4 us per pass at 165k tokens on 212 SMs. + +This library keeps the in-tree preprocessing -- ``thop.trtllm_gen_generation_preprocess``: RoPE, the FP8 K/V append +into the paged cache and the FP8 query, as ``FlashInferTrtllmGenFmha`` runs it -- and attends with the full-context +PrimTS split-KV decode of ``kernels/m3_fullctx_decode`` at the same pinned recipes as the dense layers. + +With ``fused_rope_mode`` set, the preprocess and the query cast run as one launch instead +(``minimax_m3_sm107tp4_drafter_rope_fp8_kv``, the same bytes into the query and the cache), the layer's sub-page +pool and block table taken from the KV cache pool pointers and mapping the preprocess reads. + +It serves generation-only steps of the covered (query rows per request, requests) shapes; the target installs it +first in the drafter layer's FMHA list (``install``), so every other step falls through to the stock libraries. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Mapping, Optional, Tuple + +import torch + +from tensorrt_llm._torch.attention.backends.fmha.fallback import FallbackFmha +from tensorrt_llm._torch.attention.backends.fmha.flashinfer_trtllm_gen import ( + FlashInferTrtllmGenFmha, +) +from tensorrt_llm._torch.attention.backends.fmha.interface import FmhaPhase +from tensorrt_llm._torch.attention.backends.fmha.phased import FmhaParams +from tensorrt_llm._torch.attention.backends.fmha.utils import ( + get_attention_chunk_size, + get_bmm1_scale, +) +from tensorrt_llm._torch.attention.backends.interface import AttentionInputType +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) +from tensorrt_llm._torch.flashinfer_utils import IS_FLASHINFER_AVAILABLE +from tensorrt_llm._utils import TensorWrapper, convert_to_torch_tensor, get_sm_version +from tensorrt_llm.bindings.internal import thop +from tensorrt_llm.functional import PositionEmbeddingType +from tensorrt_llm.quantization.mode import QuantMode + +from . import m3_fullctx_decode as fullctx +from . import m3_report + +# With PDL, the fused RoPE launch takes the maximum shared-memory carveout so the drafter decode's large-smem CTAs +# co-reside with it (timing only). +ROPE_CARVEOUT = True + + +def fwd_has_no_mrope(fwd) -> bool: + return getattr(fwd, "mrope_position_deltas", None) is None + + +if TYPE_CHECKING: + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + +class M3DrafterFullCtxFmha(FlashInferTrtllmGenFmha): + """Generation phase of the drafter layer: in-tree preprocess, PrimTS split-KV decode.""" + + def __init__( + self, + attn: "TrtllmAttention", + recipes: Mapping[Tuple[int, int], Mapping], + max_kv_len: int, + own_reducer: bool, + on_taken, + on_uncovered, + on_missed, + fused_rope_mode: Optional[int] = None, + pdl: bool = False, + on_rope_taken=None, + ) -> None: + super().__init__(attn) + self._recipes = recipes + self._max_kv_len = max_kv_len + self._own_reducer = own_reducer + self._fused_rope_mode = fused_rope_mode + self._pdl = pdl + self._on_rope_taken = on_rope_taken + self._on_taken = on_taken + self._on_uncovered = on_uncovered + self._on_missed = on_missed + + def _shape(self, q: torch.Tensor, metadata) -> Optional[Tuple[int, int]]: + nreq = int(metadata.num_generations) + if nreq <= 0 or int(q.shape[0]) % nreq: + return None + return int(q.shape[0]) // nreq, nreq + + def _is_supported(self, q, k, v, metadata, forward_args, *, phase=None) -> bool: + if phase == FmhaPhase.CONTEXT or metadata.num_contexts > 0: + return False + if phase is None and forward_args.attention_input_type == AttentionInputType.context_only: + return False + attn = self.attn + if ( + attn.num_heads != fullctx.HEADS + or attn.num_kv_heads != fullctx.KV_HEADS + or attn.head_dim != fullctx.HEAD_DIM + or not QuantMode(attn.quant_mode).has_fp8_kv_cache() + ): + self._on_uncovered("not the TP4 FP8 drafter geometry") + return False + # The arithmetic matches the stock decode only at unit KV scales (bmm1 and bmm2 carry none here); the layer's + # own scales are checked to be 1.0 at install, so any other per-step scale falls through to the stock path. + if ( + forward_args.kv_scale_orig_quant is not attn.kv_scale_orig_quant + or forward_args.kv_scale_quant_orig is not attn.kv_scale_quant_orig + ): + self._on_uncovered("per-step KV cache scales") + return False + if int(metadata.max_seq_len) > self._max_kv_len: + self._on_uncovered("the cache length bound is beyond the plans' K/V capacity") + return False + shape = self._shape(q, metadata) + if shape is None or shape not in self._recipes: + self._on_uncovered(f"{shape} is not a covered (query rows, requests) shape") + return False + # Compiled before warmup (``install``); the forward never compiles one. + if fullctx.plan_for(q.device, *shape, self._own_reducer) is None: + self._on_missed(shape[0] * shape[1], "the PrimTS decode plan is not prebuilt") + return False + return super()._is_supported(q, k, v, metadata, forward_args, phase=FmhaPhase.GENERATION) + + def _fused_rope_applies(self, params: FmhaParams, attention_input: torch.Tensor) -> bool: + attn = params.attn + width = (attn.num_heads + 2 * attn.num_kv_heads) * attn.head_dim + rotary_dim = int(attn.rope_params.dim) + return ( + self._fused_rope_mode is not None + and params.qkv_input is not None + and attention_input.dim() == 2 + and attention_input.dtype == torch.bfloat16 + and attention_input.is_contiguous() + and int(attention_input.shape[1]) == width + and not params.is_cross + and int(attn.position_embedding_type) == int(PositionEmbeddingType.rope_gpt_neox) + and attn.rotary_cos_sin is not None + and attn.rotary_cos_sin.dtype == torch.float32 + and fwd_has_no_mrope(params.fwd) + and 0 < rotary_dim <= attn.head_dim + and rotary_dim % 16 == 0 + and int(params.num_tokens) == int(params.batch_size) * int(params.input_seq_length) + # A one-request multi-token pass: the in-tree preprocess does not build its token map (buildDecoderInfo + # runs for several requests only) and reads the workspace's, so it keeps that path. + and not (self._multi_token(params) and int(params.batch_size) == 1) + ) + + @staticmethod + def _multi_token(params: FmhaParams) -> bool: + return ( + params.spec_decoding_generation_lengths is not None + and int(params.attn.predicted_tokens_per_seq) > 1 + ) + + def _run_fused_rope(self, params: FmhaParams, attention_input: torch.Tensor) -> bool: + """The pass with the one-launch preprocess; False (nothing run) where it does not apply.""" + if not self._fused_rope_applies(params, attention_input): + return False + attn = params.attn + meta = params.meta + qlen, nreq = int(params.input_seq_length), int(params.batch_size) + plan = fullctx.plan_for(attention_input.device, qlen, nreq, self._own_reducer) + if plan is None: + return False + # The preprocess's view of the layer's cache (buildFlashinferTrtllmGenPagedKvCacheBuffers): the pool's primary + # pointer plus the layer's offset inside its pool, sub-pages of Hkv x tokens_per_block x 128 E4M3 bytes; the + # block table is the pool's [requests, 2, blocks] K / V sub-page indices. + pool_index, layer_in_pool = ( + int(x) for x in meta.host_kv_cache_pool_mapping[attn.local_layer_idx, :2] + ) + tpb = int(params.tokens_per_block) + sub_page_bytes = attn.num_kv_heads * tpb * attn.head_dim + base = ( + int(meta.host_kv_cache_pool_pointers[pool_index, 0]) + + layer_in_pool * int(params.kv_factor) * sub_page_bytes + ) + pages = (int(params.total_num_blocks) - 1) * int(params.kv_factor) + 2 + kv_pages = convert_to_torch_tensor( + TensorWrapper(base, torch.float8_e4m3fn, [pages, attn.num_kv_heads, tpb, attn.head_dim]) + ) + tables = meta.kv_cache_block_offsets[ + pool_index, params.seq_offset : params.seq_offset + nreq + ] + multi = self._multi_token(params) + if not torch.cuda.is_current_stream_capturing(): + # The decode reads V page p as sub-page p + 1 of K's (kv_pages[1:] with the K table), as the preprocess + # path checks; the op itself writes V at the table's V sub-page. + tpb_held = (params.sequence_lengths[:nreq].to(torch.long) + tpb - 1) // tpb + cols = torch.arange(tables.shape[-1], device=tables.device) + in_use = cols[None, :] < tpb_held[:, None] + offsets = (tables[:, 1] - tables[:, 0])[in_use].unique() + if offsets.numel() != 1 or int(offsets) != 1: + raise RuntimeError( + f"the drafter cache does not keep V one sub-page after K: {offsets.tolist()}" + ) + q = torch.empty( + int(params.num_tokens), + attn.num_heads, + attn.head_dim, + dtype=torch.float8_e4m3fn, + device=attention_input.device, + ) + + torch.ops.trtllm.minimax_m3_sm107tp4_drafter_rope_fp8_kv( + attention_input, + attn.rotary_cos_sin, + params.sequence_lengths, + params.spec_decoding_position_offsets if multi else None, + kv_pages, + tables, + q, + qlen, + attn.num_heads, + attn.num_kv_heads, + int(attn.rope_params.dim), + tpb, + self._fused_rope_mode, + self._pdl, + ROPE_CARVEOUT, + ) + plan.run( + q, + kv_pages[:-1], + kv_pages[1:], + tables[:, 0], + params.sequence_lengths[:nreq], + get_bmm1_scale(attn), + params.output.view(params.num_tokens, attn.num_heads, attn.head_dim), + ) + if self._on_rope_taken is not None: + self._on_rope_taken(params.num_tokens, "spec" if multi else "plain") + self._on_taken(params.num_tokens) + return True + + def run_generation(self, params: FmhaParams) -> None: + attention_input = params.qkv_input if params.qkv_input is not None else params.query_input + if attention_input is None: + raise RuntimeError("the drafter decode requires QKV or query input.") + attn = params.attn + meta = params.meta + fwd = params.fwd + rope_params = attn.rope_params + if self._run_fused_rope(params, attention_input): + return + # The BF16 post-RoPE query, rounded to E4M3 below. At the unit KV scales this layer runs with, that is the FP8 + # query the stock decode gets (the preprocess's FP8 variant is e4m3(q * kv_scale_orig_quant)). + fp8_context_fmha = False + ( + q_processed, + kv_pool, + block_tables, + _kv_scale_pool, + _bmm1_scale, + _bmm2_scale, + _fmha_workspace, + _cu_seqlens, + _max_q_len, + _max_kv_len, + _window_left, + is_multi_token_gen, + ) = thop.trtllm_gen_generation_preprocess( + attention_input, + params.workspace, + params.sequence_lengths, + params.spec_decoding_generation_lengths, + params.spec_decoding_position_offsets, + meta.kv_cache_block_offsets, + meta.host_kv_cache_pool_pointers, + meta.host_kv_cache_pool_mapping, + fwd.kv_scale_orig_quant, + fwd.kv_scale_quant_orig, + fwd.out_scale, + attn.rotary_inv_freq, + attn.rotary_cos_sin, + fwd.mrope_position_deltas, + attn.local_layer_idx, + params.seq_offset, + attn.num_heads, + attn.num_kv_heads, + attn.head_dim, + params.tokens_per_block, + attn.quant_mode, + params.max_attention_window_size, + params.cyclic_attention_window_size, + params.num_tokens, + params.batch_size, + params.input_seq_length, + params.max_past_kv_length, + rope_params.dim, + rope_params.theta, + int(rope_params.scale_type), + rope_params.scale, + rope_params.max_positions, + attn.position_embedding_type, + get_bmm1_scale(attn), + 1.0, + fp8_context_fmha, + attn.predicted_tokens_per_seq, + get_attention_chunk_size(attn), + self._multi_processor_count, + params.total_num_blocks, + params.kv_factor, + True, + params.is_cross, + ) + qlen, nreq = params.input_seq_length, params.batch_size + plan = fullctx.plan_for(attention_input.device, qlen, nreq, self._own_reducer) + # A first draft pass arrives as qlen rows per request either as a plain multi-row generation step (the + # drafter's own, input_seq_length = qlen) or with spec-decoding lengths (multi-token generation); a 1-row + # pass is never multi-token. + if plan is None or (is_multi_token_gen and qlen == 1): + raise RuntimeError( + f"the drafter decode was selected for {qlen} query rows x {nreq} requests without a uniform plan" + ) + rows = params.num_tokens * attn.num_heads * attn.head_dim + q = fp8_e4m3_saturate(q_processed.flatten()[:rows]) + q = q.view(params.num_tokens, attn.num_heads, attn.head_dim) + # K pages in row 0 of the table, V in row 1; this cache keeps V one sub-page after K, so V page p is page p + 1 + # of the pool (the two views have the same page count, as the kernel requires). + if not torch.cuda.is_current_stream_capturing(): + # The plan runs every request at qlen rows; a multi-token pass must give each request exactly that many. + lengths = params.spec_decoding_generation_lengths + if is_multi_token_gen and not bool((lengths[:nreq] == qlen).all()): + raise RuntimeError( + f"the drafter pass rows per request are not uniform: {lengths[:nreq].tolist()}" + ) + # Only the blocks a request holds are laid out; the table's tail entries are unset. + tpb = int(params.tokens_per_block) + held = (params.sequence_lengths[:nreq].to(torch.long) + tpb - 1) // tpb + cols = torch.arange(block_tables.shape[-1], device=block_tables.device) + in_use = cols[None, :] < held[:, None] + offsets = (block_tables[:nreq, 1] - block_tables[:nreq, 0])[in_use].unique() + if offsets.numel() != 1 or int(offsets) != 1: + raise RuntimeError( + f"the drafter cache does not keep V one sub-page after K: {offsets.tolist()}" + ) + # The preprocess sizes kv_pool by blocks; its table indexes the K and V sub-pages of every block. + # The view stops two sub-pages past the last block's K, as get_kv_subpage_pool's does. + pages = (int(params.total_num_blocks) - 1) * int(params.kv_factor) + 2 + kv_pages = convert_to_torch_tensor( + TensorWrapper(kv_pool.data_ptr(), kv_pool.dtype, [pages, *kv_pool.shape[1:]]) + ) + plan.run( + q, + kv_pages[:-1], + kv_pages[1:], + block_tables[:nreq, 0], + params.sequence_lengths[:nreq], + get_bmm1_scale(attn), + params.output.view(params.num_tokens, attn.num_heads, attn.head_dim), + ) + self._on_taken(params.num_tokens) + + +def install( + draft_model: torch.nn.Module, + recipes, + max_kv_len: int, + own_reducer: bool, + pdl: bool, + on_taken, + on_uncovered, + on_missed, + fused_rope_mode: Optional[int] = None, + on_rope_taken=None, +) -> int: + """Prebuild the plans of ``recipes`` and put ``M3DrafterFullCtxFmha`` first in the FMHA list of the TrtllmAttention + backend of every attention module of ``draft_model``. Call before warmup. Returns the number of attention layers it + was installed on.""" + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + count = 0 + device = next(draft_model.parameters()).device + # The backend is not a module: each attention module of the drafter holds it as ``attn``. + for module in draft_model.modules(): + backend = getattr(module, "attn", None) + if not isinstance(backend, TrtllmAttention) or not hasattr(backend, "_fmha_manager"): + continue + if not bool((backend.kv_cache_scaling_factor == 1.0).all()): + on_uncovered("a scaled FP8 KV cache") + continue + fullctx.prebuild(device, recipes, max_kv_len, own_reducer, pdl) + libs = backend._fmha_manager.fmha_libs + if not any(isinstance(lib, M3DrafterFullCtxFmha) for lib in libs): + libs.insert( + 0, + M3DrafterFullCtxFmha( + backend, + recipes, + max_kv_len, + own_reducer, + on_taken, + on_uncovered, + on_missed, + fused_rope_mode=fused_rope_mode, + pdl=pdl, + on_rope_taken=on_rope_taken, + ), + ) + count += 1 + return count + + +class M3DrafterSlidingFiFmha(FlashInferTrtllmGenFmha): + """FlashInfer's trtllm-gen FMHA for the standalone DSpark drafter's attention on SM 10.7. + + The stock library declines SM 10.7, so the drafter's sliding-window decode falls through to ``thop.attention``'s + in-tree trtllm-gen runner, which picks the SM107a Q64 KeepsAb kernel (9.7 us per call at 7 rows, window 127, grid + 9). FlashInfer's own selector runs its SM100f kernels on SM 10.7 and picks the Q8 SwapsAb one (4.0 us per call on + the same shape). Everything else is the stock library.""" + + @classmethod + def is_environment_available(cls) -> bool: + if not IS_FLASHINFER_AVAILABLE or cls._missing_fused_nanobind_ops(): + return False + return get_sm_version() == 107 + + def run_generation(self, *args, **kwargs): + out = super().run_generation(*args, **kwargs) + m3_report.work( + "VR_DRAFT_FMHA_SM100F", "drafter generation attention on FlashInfer trtllm-gen" + ) + return out + + def run_context(self, *args, **kwargs): + out = super().run_context(*args, **kwargs) + m3_report.work("VR_DRAFT_FMHA_SM100F", "drafter context attention on FlashInfer trtllm-gen") + return out + + +def sliding_fi_bypass_reason(draft_model: torch.nn.Module) -> Optional[str]: + """Why ``M3DrafterSlidingFiFmha`` would do no work on ``draft_model`` (None: its decode reads the attention modules' + FMHA list). A DFlash-family drafter (the standalone DSpark drafter) runs its block decode on its own attention ops + (``dflash_attention_backend``: FlashInfer's trtllm_batch_decode_with_kv_cache for TRTLLM) and never calls its + attention modules' backend, so a library installed in their FMHA lists is never selected.""" + for module in draft_model.modules(): + backend = getattr(module, "dflash_attention_backend", None) + if backend is not None: + return ( + f"the standalone DSpark drafter's block decode runs its own attention ops (DFlash attention backend " + f"{backend}), never its attention modules' FMHA list, so the library would do no work there " + "(deferred: the lever needs the drafter's own decode call site)" + ) + return None + + +def install_sliding_fi(draft_model: torch.nn.Module) -> int: + """Put ``M3DrafterSlidingFiFmha`` ahead of the fallback in the FMHA list of every attention module of + ``draft_model`` it can serve. Call before warmup. Returns the number of attention layers it was installed on.""" + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + count = 0 + for module in draft_model.modules(): + backend = getattr(module, "attn", None) + if not isinstance(backend, TrtllmAttention) or not hasattr(backend, "_fmha_manager"): + continue + libs = backend._fmha_manager.fmha_libs + if any(isinstance(lib, M3DrafterSlidingFiFmha) for lib in libs): + continue + if not M3DrafterSlidingFiFmha.is_available(backend): + continue + at = next((i for i, lib in enumerate(libs) if isinstance(lib, FallbackFmha)), len(libs)) + libs.insert(at, M3DrafterSlidingFiFmha(backend)) + count += 1 + return count diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_fold_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_fold_tail.py new file mode 100644 index 000000000000..79aff5cb80ce --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_fold_tail.py @@ -0,0 +1,172 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The speculative step's store and next-step staging at the tail of the forward (one Triton launch). + +For each of the first ``n`` rows (``n`` read from device memory, so a captured graph whose step was not armed writes +nothing), with ``s = slots[i]``: + +* the spec sampler's per-slot store, exactly as ``m3_spec_tail.slot_scatter`` writes it (zero-padded to each store + width): ``new[s][c]``, ``next_new[s][c]``, ``draft[s][c]`` and ``lens[s]``; +* the engine's next-step inputs, exactly as ``m3_spec_tail.step_gather`` writes them from that store for a step whose + rows are these rows in this order (no new tokens, first previous row 0): ``ids[i*T + t] = next_new[s][t]``, + ``draft_ids[i*D + d] = draft[s][d]``, ``pos_off[i*T + t] = lens[s]``, ``kv_off[i] = lens[s] - T``. + +Integer copies only: the values are the ones the sampler's and the engine's own copies write. + +With ``cpp`` the same writes run as this target's C++ op (trtllm::minimax_m3_sm107_tp4_fold_tail, +minimaxM3Sm107Tp4/foldTail.cu: one 32-lane CTA per row) instead of the Triton kernel, for widths up to 32. +""" + +import torch +import triton +import triton.language as tl + + +@triton.jit +def _fold_tail_kernel( + ctrl, + slots, + src_new, + sn_s0, + sn_s1, + SN_W: tl.constexpr, + dst_new, + dn_s0, + dn_s1, + DN_W: tl.constexpr, + src_next, + sx_s0, + sx_s1, + SX_W: tl.constexpr, + dst_next, + dx_s0, + dx_s1, + DX_W: tl.constexpr, + src_draft, + sd_s0, + sd_s1, + SD_W: tl.constexpr, + dst_draft, + dd_s0, + dd_s1, + DD_W: tl.constexpr, + src_lens, + dst_lens, + ids, + draft_ids, + pos_off, + kv_off, + T: tl.constexpr, + D: tl.constexpr, + BLOCK: tl.constexpr, +): + i = tl.program_id(0) + n = tl.load(ctrl) + if i < n: + s = tl.load(slots + i).to(tl.int64) + c = tl.arange(0, BLOCK) + new = tl.load(src_new + i * sn_s0 + c * sn_s1, mask=c < SN_W, other=0) + tl.store(dst_new + s * dn_s0 + c * dn_s1, new, mask=c < DN_W) + nxt = tl.load(src_next + i * sx_s0 + c * sx_s1, mask=c < SX_W, other=0) + tl.store(dst_next + s * dx_s0 + c * dx_s1, nxt, mask=c < DX_W) + drf = tl.load(src_draft + i * sd_s0 + c * sd_s1, mask=c < SD_W, other=0) + tl.store(dst_draft + s * dd_s0 + c * dd_s1, drf, mask=c < DD_W) + ln = tl.load(src_lens + i) + tl.store(dst_lens + s, ln) + # The next step's inputs read the store's padded rows: zero past each source width. + tl.store(ids + i * T + c, tl.where(c < DX_W, nxt, 0), mask=c < T) + tl.store(draft_ids + i * D + c, tl.where(c < DD_W, drf, 0), mask=c < D) + tl.store(pos_off + i * T + c, tl.zeros([BLOCK], dtype=tl.int32) + ln, mask=c < T) + tl.store(kv_off + i, ln - T) + + +def fold_tail( + ctrl: torch.Tensor, + slots: torch.Tensor, + rows: int, + src_new: torch.Tensor, + dst_new: torch.Tensor, + src_next: torch.Tensor, + dst_next: torch.Tensor, + src_draft: torch.Tensor, + dst_draft: torch.Tensor, + src_lens: torch.Tensor, + dst_lens: torch.Tensor, + ids: torch.Tensor, + draft_ids: torch.Tensor, + pos_off: torch.Tensor, + kv_off: torch.Tensor, + tokens_per_step: int, + cpp: bool = False, +) -> bool: + """Launch over ``rows`` programs (the first ``ctrl[0]`` write); False, having launched nothing, when the inputs + are outside what the kernel covers. Store views are 2-D [slots, width] (any strides), sources [rows, width].""" + two = (src_new, dst_new, src_next, dst_next, src_draft, dst_draft) + one = (src_lens, dst_lens, ids, draft_ids, pos_off, kv_off) + if any(t.dim() != 2 for t in two) or any(t.dim() != 1 or t.stride(0) != 1 for t in one): + return False + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in two + one + (ctrl,)) + or slots.dtype != torch.long + ): + return False + t_step = tokens_per_step + d_step = t_step - 1 + if ( + rows <= 0 + or slots.numel() < rows + or any(t.shape[0] < rows for t in (src_new, src_next, src_draft, src_lens)) + or ids.numel() < rows * t_step + or pos_off.numel() < rows * t_step + or draft_ids.numel() < rows * d_step + or kv_off.numel() < rows + ): + return False + widths = (src_new.shape[1], dst_new.shape[1], src_next.shape[1], dst_next.shape[1]) + widths += (src_draft.shape[1], dst_draft.shape[1], t_step) + if cpp and max(widths) <= 32: + torch.ops.trtllm.minimax_m3_sm107_tp4_fold_tail( + ctrl, slots, rows, src_new, dst_new, src_next, dst_next, src_draft, dst_draft, + src_lens, dst_lens, ids, draft_ids, pos_off, kv_off, t_step, + ) # fmt: skip + return True + block = triton.next_power_of_2(max(widths)) + _fold_tail_kernel[(rows,)]( + ctrl, + slots, + src_new, + src_new.stride(0), + src_new.stride(1), + src_new.shape[1], + dst_new, + dst_new.stride(0), + dst_new.stride(1), + dst_new.shape[1], + src_next, + src_next.stride(0), + src_next.stride(1), + src_next.shape[1], + dst_next, + dst_next.stride(0), + dst_next.stride(1), + dst_next.shape[1], + src_draft, + src_draft.stride(0), + src_draft.stride(1), + src_draft.shape[1], + dst_draft, + dst_draft.stride(0), + dst_draft.stride(1), + dst_draft.shape[1], + src_lens, + dst_lens, + ids, + draft_ids, + pos_off, + kv_off, + t_step, + d_step, + block, + num_warps=1, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_force_vec.py new file mode 100644 index 000000000000..be04f26f9aff --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_force_vec.py @@ -0,0 +1,64 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance (the in-tree speculative worker's +``force_num_accepted_tokens``), in one launch. + +Triton: ``force_vec_kernel``: one CTA bumps the shared step counter and draws each request's count from the fixed-seed +pool with the in-tree (counter, slot) hash, exactly as ``_apply_force_accepted_tokens`` does. ``force_vec_req_kernel`` +(``FORCE_AL_PER_REQUEST``): each request draws from the same pool at its own key, its KV length at this verify. +Reached through the ``ops/spec`` entries ``m3_force_vec`` and ``m3_force_vec_req``. + +One compiled variant for every batch and accept count: specializing them (== 1, divisible by 16) would compile +mid-serving the first time a step reaches a new one, past the warm-up. +""" + +import triton +import triton.language as tl + + +@triton.jit(do_not_specialize=["batch_size", "base_total", "max_total"]) +def force_vec_kernel( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK: tl.constexpr, +): + c = tl.load(counter) + 1 + slots = tl.arange(0, BLOCK) + tl.store(counter, c) + idx = (c * counter_stride + slots.to(tl.int64) * slot_stride) & pool_mask + live = slots < batch_size + r = tl.load(pool + idx, mask=live, other=1.0) + v = tl.minimum(base_total + (r < frac).to(tl.int32), max_total) + tl.store(out + slots, v, mask=live) + + +@triton.jit(do_not_specialize=["batch_size", "base_total", "max_total"]) +def force_vec_req_kernel( + pool, + keys, + out, + batch_size, + base_total, + max_total, + frac, + key_stride, + pool_mask, + BLOCK: tl.constexpr, +): + # One draw per request from its own key (its KV length at this verify): no shared counter, so a request's forced + # counts depend only on its own prompt and progress, not on what else the server ran before or beside it. + slots = tl.arange(0, BLOCK) + live = slots < batch_size + k = tl.load(keys + slots, mask=live, other=0).to(tl.int64) + idx = (k * key_stride) & pool_mask + r = tl.load(pool + idx, mask=live, other=1.0) + v = tl.minimum(base_total + (r < frac).to(tl.int32), max_total) + tl.store(out + slots, v, mask=live) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_fullctx_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_fullctx_decode.py new file mode 100644 index 000000000000..ef6b9ee0d1ad --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_fullctx_decode.py @@ -0,0 +1,313 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 full-context FP8 decode attention on Rubin: ``fullctx_decode``. + +The dense layers (0-2) attend every token of their FP8 E4M3 (unscaled) paged KV cache. On SM 10.7 the stock call +(flashinfer trtllm-gen, ``minimax_m3_trtllm_gen_dense_decode``) caps its split at 16 cluster CTAs per (request, KV +head): at TP4 (16 query heads and 1 KV head per rank) one request runs on 16 CTAs of 212 and takes 33.4-33.8 us +per call at 165k tokens. + +This runs the same arithmetic -- FP8 Q (the same E4M3 rounding of q), FP8 K/V, P quantized to E4M3 per split -- on +the vendored PrimTS decode kernel (``attention/backends/prims_ts``, SM 10.7 through the sm_100f family) at a pinned +split-KV recipe, which fans one request out over 96-108 CTAs. The split partials are merged by +``torch.ops.trtllm.minimax_m3_sm107tp4_fullctx_reduce`` (``cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/ +fullCtxReduce.cu``, head-count generic, reused by name): unnormalized fp32 accumulation over the active splits, one +final division, one BF16 rounding, the merge form of stock trtllm-gen. Only the split changes against stock +(``modeling.FULLCTX_DECODE_RECIPES``). + +A plan is compiled per (device, query rows per request, requests) by ``prebuild`` before the executor's warmup; the +forward only reads plans (``plan_for``). The plan is built from PrimTS's own config, compile and workspace helpers at +an explicit recipe; the PrimTS wrapper's planning policy is not used, so no PrimTS state is replaced. With +``own_reducer`` off the PrimTS reducer merges into FP16 and the result is cast to BF16. +""" + +from __future__ import annotations + +import threading +from typing import Dict, Mapping, Optional, Tuple + +import torch + +# The TP4 per-rank geometry: 16 query heads over 1 KV head. +HEADS = 16 +KV_HEADS = 1 # capacity: fixed: the tp4 KV heads per rank +HEAD_DIM = 128 # capacity: fixed: the checkpoint's head dim +PAGE = 128 +# The per-request K/V length PrimTS's config selection sizes its policy for (Q/KV tiles, launch mode); the pinned +# recipes were measured with it. The compiled capacity (``max_kv_len``) may be larger. +POLICY_KV_LEN = ( + 1 << 20 +) # capacity: fixed: the PrimTS policy's K/V length bound (the plans' compile shape) +HEADS_PER_KV = HEADS // KV_HEADS # capacity: fixed: the tp4 query heads per KV head + +_plans: Dict[Tuple[int, int, int, bool], "FullCtxPlan"] = {} +_lock = threading.Lock() + + +class FullCtxPlan: + """One compiled PrimTS split-KV decode at a pinned recipe, its workspace and its FP16 staging output.""" + + def __init__( + self, + device: torch.device, + qlen: int, + nreq: int, + max_kv_len: int, + recipe: Mapping, + own_reducer: bool, + pdl: bool, + ): + from tensorrt_llm._torch.attention.backends.prims_ts import decode as tsdec + from tensorrt_llm._torch.attention.backends.prims_ts.kernels.fmha_decode.fmha_decode_config import ( + get_max_active_clusters_for_cluster_size, + make_decode_config, + ) + + self._tsdec = tsdec + self.qlen = qlen + self.nreq = nreq + self.pdl = pdl + self.recipe = dict(recipe) + splits = int(self.recipe["splits"]) + fp8 = tsdec._cutlass_dtype("float8_e4m3fn") + # PDL contract of the main kernel (prims_ts fmha_decode_kernel.py): every CTA acquires its producer + # (griddepcontrol.wait) before any task reads q, lengths or tables, and triggers its dependents + # (launch_dependents) only after its own tail: an active CTA after its last store and TMEM dealloc, an + # inactive split or zero-work CTA right after its wait. The trigger lets the merge launch before the + # slowest CTA ends, so a dependent may read only step-invariant data (weights) before its own + # griddepcontrol.wait, never this step's partial O/LSE, output, q, lengths or tables; the merge + # (minimax_m3_sm107tp4_fullctx_reduce) waits as its first instruction. + cfg = make_decode_config( + headdim=HEAD_DIM, + args={"use_pdl": pdl, **self.recipe.get("config", {})}, + seq_len_q=qlen, + seq_len_kv=min(max_kv_len, POLICY_KV_LEN), + batch_size=nreq, + num_heads_q=HEADS, + num_heads_kv=KV_HEADS, + q_dtype=fp8, + k_dtype=fp8, + v_dtype=fp8, + o_dtype=tsdec._cutlass_dtype("float16"), + qkv_layout="pagedKv", + num_tokens_per_page=PAGE, + storage_tokens_per_page=PAGE, + split_kv_mode="gmem_reduction_with_separate_kernel", + splits_kv=splits, + max_splits_kv=splits, + min_loop_iters_per_split=1, + mask_type="causal" if qlen > 1 else "dense", + auto_tuner=True, + split_kv=True, + ) + _check_layout(cfg, qlen, splits) + with torch.cuda.device(device): + max_active = get_max_active_clusters_for_cluster_size(1) + spec = tsdec._decode_launch_spec_from_config( + cfg, + batch_size=nreq, + num_qo_heads=HEADS, + num_kv_heads=KV_HEADS, + head_dim=HEAD_DIM, + seq_len_q=qlen, + max_active_clusters=int(max_active), + ) + compile_spec = tsdec._make_decode_compile_spec( + spec, + device_index=_index(device), + num_qo_heads=HEADS, + num_kv_heads=KV_HEADS, + head_dim=HEAD_DIM, + page_size=PAGE, + max_kv_len=max_kv_len, + seq_len_q=qlen, + q_dtype_key="float8_e4m3fn", + k_dtype_key="float8_e4m3fn", + v_dtype_key="float8_e4m3fn", + output_dtype_key="float16", + use_packed_q=False, + kv_prefix_mode="dynamic", + kv_lengths_mode="dynamic", + ) + self._main, self._primts_reducer = tsdec._get_compiled_decode(compile_spec) + layout = tsdec._make_decode_workspace_layout( + spec.scratch_shapes, + torch.float16, + use_separate_reduction_kernel=True, + use_split_kv=True, + ) + self._workspace_buffer = torch.empty(layout.total_bytes, device=device, dtype=torch.int8) + self.workspace = tsdec._bind_decode_workspace(self._workspace_buffer, layout) + self.workspace.split_kv_counter.zero_() + self.workspace.cu_seqlens_q.zero_() + self.workspace.attention_sinks.zero_() + rows = HEADS_PER_KV * qlen + if tuple(self.workspace.partial_o.shape) != (nreq, KV_HEADS, splits, rows, HEAD_DIM): + raise RuntimeError( + f"PrimTS split scratch {tuple(self.workspace.partial_o.shape)} is not the " + f"[requests, KV heads, splits, {rows} rows, {HEAD_DIM}] layout the BF16 merge reads" + ) + self.own_reducer = own_reducer + self.q_tokens_per_cta = max(int(cfg.q_tokens_per_cta), 1) + self.tile_kv = int(cfg.tile_size_kv) + self.insts_kv = int(cfg.num_insts_kv) + self.causal = qlen > 1 + shape = (nreq, qlen, HEADS, HEAD_DIM) if qlen > 1 else (nreq, HEADS, HEAD_DIM) + self.staging = torch.empty(shape, dtype=torch.float16, device=device) + + def run( + self, + q: torch.Tensor, + k_pages: torch.Tensor, + v_pages: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + mxfp8: Optional[Tuple[torch.Tensor, torch.Tensor]] = None, + ) -> bool: + """Decodes into output; with mxfp8 (E4M3 rows, UE8M0 scales) and a merge op that takes them, the merge + also writes the MXFP8 copy of output. Returns whether it did.""" + # The kernel takes a compact query; a column view of the fused QKV projection is copied once. + q = q.reshape(self.staging.shape).contiguous() + runtime = self._tsdec._prepare_decode_runtime_unchecked( + q, + (k_pages, v_pages), + None, + output_dtype=torch.float16, + bmm1_scale=sm_scale, + bmm2_scale=1.0, + out=self.staging, + ) + ws = self.workspace + self._main( + runtime.q, + runtime.k_cache, + runtime.v_cache, + runtime.k_sf_cache, + runtime.v_sf_cache, + runtime.out, + seq_lens, + ws.cu_seqlens_q, + block_table, + block_table, # Dense decode constexpr-elides the membership table. + ws.partial_o, + ws.partial_stats, + ws.split_kv_counter, + ws.attention_sinks, + runtime.num_physical_pages, + runtime.k_page_stride, + runtime.k_head_stride, + runtime.k_token_stride, + runtime.v_page_stride, + runtime.v_head_stride, + runtime.v_token_stride, + runtime.bmm1_scale, + runtime.bmm2_scale, + (), + ) + if self.own_reducer: + args = ( + ws.partial_o, + ws.partial_stats, + seq_lens, + output, + self.qlen, + self.q_tokens_per_cta, + self.tile_kv, + self.insts_kv, + self.causal, + self.pdl, + ) + if mxfp8 is not None and emits_mxfp8(): + torch.ops.trtllm.minimax_m3_sm107tp4_fullctx_reduce(*args, mxfp8[0], mxfp8[1]) + return True + torch.ops.trtllm.minimax_m3_sm107tp4_fullctx_reduce(*args) + return False + self._primts_reducer( + runtime.out, + seq_lens, + ws.cu_seqlens_q, + ws.partial_o, + ws.partial_stats, + ws.attention_sinks, + runtime.bmm1_scale, + runtime.bmm2_scale, + (), + ) + output.view(self.staging.shape).copy_(self.staging) + return False + + +def _check_layout(cfg, qlen: int, splits: int) -> None: + """The scratch layout and split ranges the BF16 merge (``fullCtxReduce.cu`` ``activeSplits``) assumes. + + Rows are [token, local head]; a row's split ranges are those of its grouped Q CTA, whose first token is + ``(token // q_tokens_per_cta) * q_tokens_per_cta``; tiles are grouped ``num_insts_kv`` per loop step; no sliding + window, attention sinks or packed Q. + """ + problems = [] + if not (cfg.use_split_kv and cfg.use_separate_reduction_kernel): + problems.append("not split-KV with the separate GMEM reducer") + if int(cfg.splits_kv) != splits or int(cfg.max_splits_kv) != splits: + problems.append(f"splits {cfg.splits_kv}/{cfg.max_splits_kv}, recipe {splits}") + if cfg.use_cluster_smem_reduction or cfg.use_persistent_scheduler: + problems.append("cluster-SMEM reduction or a persistent scheduler") + if cfg.use_variable_seqlens_q or cfg.use_sliding_window_causal or cfg.use_attention_sinks: + problems.append("packed Q, a sliding window or attention sinks") + if int(cfg.heads_q_per_kv) not in (0, HEADS_PER_KV): + problems.append(f"{cfg.heads_q_per_kv} query heads per KV head") + if qlen > 1: + if not (cfg.groups_tokens_heads_q and cfg.uses_nontrivial_grouped_q_layout): + problems.append("multi-token Q without the grouped [token, head] tile layout") + elif int(cfg.q_tokens_per_cta) * HEADS_PER_KV != int(cfg.tile_size_q): + problems.append( + f"{cfg.q_tokens_per_cta} tokens per CTA in a {cfg.tile_size_q}-row tile" + ) + if problems: + raise RuntimeError("PrimTS recipe outside the BF16 merge's layout: " + "; ".join(problems)) + + +def emits_mxfp8() -> bool: + """Whether the loaded merge op can also write the o_proj MXFP8 hand-off (its schema takes q8 / sf).""" + op = getattr(torch.ops.trtllm, "minimax_m3_sm107tp4_fullctx_reduce", None) + return op is not None and "q8" in str(op.default._schema) + + +def has_own_reducer() -> bool: + """Whether this build carries the BF16 split-KV merge op.""" + return hasattr(torch.ops.trtllm, "minimax_m3_sm107tp4_fullctx_reduce") + + +def plan_for( + device: torch.device, qlen: int, nreq: int, own_reducer: bool = True +) -> Optional[FullCtxPlan]: + """The prebuilt plan for this shape, or None.""" + return _plans.get((_index(device), qlen, nreq, own_reducer)) + + +def prebuild( + device: torch.device, + recipes: Mapping[Tuple[int, int], Mapping], + max_kv_len: int, + own_reducer: bool, + pdl: bool, +) -> int: + """Compile the plan of every (query rows per request, requests) in ``recipes``. Call before warmup.""" + built = 0 + for (qlen, nreq), recipe in recipes.items(): + key = (_index(device), qlen, nreq, own_reducer) + with _lock: + if key not in _plans: + _plans[key] = FullCtxPlan(device, qlen, nreq, max_kv_len, recipe, own_reducer, pdl) + built += 1 + return built + + +def kv_page_views(pool: torch.Tensor, subpages_per_slot: int) -> Tuple[torch.Tensor, torch.Tensor]: + """K and V page views of a flat sub-page pool (K of slot s at sub-page s * f, V one later), indexed by slot.""" + return pool[0::subpages_per_slot], pool[1::subpages_per_slot] + + +def _index(device: torch.device) -> int: + return device.index if device.index is not None else torch.cuda.current_device() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_index_score.py new file mode 100644 index 000000000000..51803e7057eb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_index_score.py @@ -0,0 +1,661 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py +"""MiniMax-M3 index decode block scorer (CuTe DSL, Blackwell) with its pre-wait schedule. + +The kernel of ``tensorrt_llm/_torch/attention/kernels/blackwell/minimax_m3_index_decode_score.py`` (the same scores, bit +for bit) with two schedule options this target compiles in: + +* ``prefetch_old``: before the grid-dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that hold only + tokens older than this step (the producer ahead writes only this step's tokens), up to ``prefetch_depth`` blocks per + CTA (0 = all of them). +* ``early_issue``: the TMA warp reads a ``seq_lens`` hint and its first page id before the wait and issues the Q tile + and the first K tile right after it. The tiles are read after the wait, so their contents are final; the block table + is only written between graph replays, so the early page id is the right one whenever the block is valid. A CTA + whose early tiles lie past the request's true length drains them before it exits. + +Every kernel is compiled before the executor's warmup (``layers.prebuild_index_scorer``) through ``IndexScoreRunner`` +and called through this target's ``m3_index_score`` entry. +""" + +from pathlib import Path + +import cutlass +import torch +from cuda.bindings.driver import CUstream +from cutlass import Float8E4M3FN, Float16, Float32, Int32, Int64, Uint32, cute +from cutlass._mlir.dialects import llvm +from cutlass.cute.nvgpu import cpasync, warp +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + EVICT_FIRST, + fp8x4_to_fp16x4, + mma_sync, + simple_tma_copy, +) +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + TRTLLM_ENABLE_PDL, + griddepcontrol_wait, +) + +__all__ = ["IndexDecodeScoreKernel", "IndexScoreRunner"] + + +@dsl_user_op +def _l2_prefetch_bulk(addr: Int64, size: Int32, *, loc=None, ip=None) -> None: + """Prefetch ``size`` contiguous bytes (multiple of 16) at global ``addr`` into L2.""" + llvm.inline_asm( + res=None, + operands_=[Int64(addr).ir_value(loc=loc, ip=ip), Int32(size).ir_value(loc=loc, ip=ip)], + asm_string="cp.async.bulk.prefetch.L2.global [$0], $1;", + constraints="l,r", + has_side_effects=True, + asm_dialect=llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@cute.jit +def _fp8_to_f16_mma_fragments(src: cute.Tensor): + """Split an FP8 ldmatrix fragment into the two FP16 k-fragments MMA wants.""" + src_elems = cute.size(src) + src_u32 = cute.recast_tensor(src, Uint32) + src_f16 = cute.make_rmem_tensor(src_elems, Float16) + src_f16_u32 = cute.recast_tensor(src_f16, Uint32) + # Packed conversion; faster and fewer SASS instructions than + # src.load().to(Float16). + for i in cutlass.range_constexpr(src_elems // 4): + converted = fp8x4_to_fp16x4(src_u32[i]) + src_f16_u32[i * 2] = converted[0] + src_f16_u32[i * 2 + 1] = converted[1] + lower = cute.make_rmem_tensor(src_elems // 2, Float16) + upper = cute.make_rmem_tensor(src_elems // 2, Float16) + + # FP8 ldmatrix yields four consecutive values along K. Split each group + # into the lower two and upper two for the two FP16 MMA k-fragments. + for i in cutlass.range_constexpr(src_elems // 2): + lower[i] = src_f16[(i // 2) * 4 + i % 2] + upper[i] = src_f16[(i // 2) * 4 + 2 + i % 2] + return lower, upper + + +class IndexDecodeScoreKernel: + """Per-block max index score for one decode step. + + Grid is (batch, split_k): CTA (b, s) walks KV blocks + s, s + split_k, s + 2 * split_k, ... of request b and writes each block's + score directly, so no cross-CTA reduction is needed. CTAs whose split_id + exceeds the request's block count exit immediately. + """ + + BLOCK_K = 128 + BAR_MMA = 1 + num_stages = 2 + + def __init__( + self, + dtype: type[cutlass.Numeric], + num_heads: int, + max_decode_query_len: int, + split_k: int, + head_dim: int = 128, + prefetch_old: bool = False, + early_issue: bool = False, + prefetch_depth: int = 0, + ): + # prefetch_old: before the grid dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that + # hold only tokens older than this step (the predecessor writes only this step's tokens). + self.prefetch_old = prefetch_old + # early_issue: the TMA warp reads a seq_lens hint and this CTA's first page id before the grid dependency + # wait, then issues the Q tile and the first K tile right after it, overlapping the post-wait seq_lens load. + # The tiles are read after the wait, so their contents are final; the block table is only written between + # graph replays, so the early page id is the right one whenever the block is valid. A CTA whose early tiles + # turn out to lie past the request's true length drains them before exiting. + self.early_issue = early_issue + # prefetch_depth: blocks per CTA that prefetch_old covers (0 = all of them). Prefetching every older page at + # large batches floods L2; the first few per CTA are the ones whose latency the pipeline cannot hide. + self.prefetch_depth = prefetch_depth + self.dtype = dtype + self.num_heads = num_heads + self.max_decode_query_len = max_decode_query_len + self.split_k = split_k + self.head_dim = head_dim + + def _swizzle_elems(self) -> int: + """Elements spanned by one 128-byte swizzle atom.""" + return 128 * 8 // self.dtype.width + + def _sq_layout(self): + """Composed SMEM layout for the Q tile, shared by the descriptor and the kernel.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_q = self.num_heads * self.max_decode_query_len + layout = cute.make_layout( + (self.max_decode_query_len, self.num_heads, (elems, head_dim // elems)), + stride=(elems, self.max_decode_query_len * elems, (1, block_q * elems)), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + def _sk_layout(self): + """Composed SMEM layout for the pipelined K tiles.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_k = self.BLOCK_K + layout = cute.make_layout( + (1, block_k, (elems, head_dim // elems), self.num_stages), + stride=(0, elems, (1, block_k * elems), block_k * head_dim), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + @cute.jit + def __call__( + self, + gQ: cute.Tensor, # [bs * runtime_decode_query_len, num_heads, head_dim] + gK_cache: cute.Tensor, # [num_pages, page_size, head_dim] + block_table: cute.Tensor, # [bs, max_pages] + score: cute.Tensor, # [num_heads, bs * runtime_decode_query_len, max_pages] + seq_lens: cute.Tensor, # [bs] + stream: CUstream, + ): + num_heads = self.num_heads + head_dim = self.head_dim + MAX_DQL = self.max_decode_query_len + assert num_heads * MAX_DQL <= 32 + + batch = seq_lens.shape[0] + decode_query_len = gQ.shape[0] // batch + grid = (batch, self.split_k, 1) + block = (32 * 5, 1, 1) + + tma_g2s = cpasync.CopyBulkTensorTileG2SOp() + elems = self._swizzle_elems() + + q_tma_atom, q_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gQ, (None, None, elems)), + self._sq_layout(), + (MAX_DQL, num_heads, head_dim), + ) + k_tma_atom, k_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gK_cache, (None, None, elems)), + self._sk_layout(), + (1, self.BLOCK_K, head_dim), + ) + + self.kernel( + q_tma_atom, + q_tma_tensor, + k_tma_atom, + k_tma_tensor, + gK_cache, + block_table, + score, + seq_lens, + decode_query_len, + ).launch(grid=grid, block=block, stream=stream, use_pdl=TRTLLM_ENABLE_PDL) + + @cute.kernel + def kernel( + self, + q_tma_atom: cute.CopyAtom, + q_tma_tensor: cute.Tensor, + k_tma_atom: cute.CopyAtom, + k_tma_tensor: cute.Tensor, + gK_cache: cute.Tensor, + block_table: cute.Tensor, + score: cute.Tensor, + seq_lens: cute.Tensor, + decode_query_len, + ): + batch_id, split_id, _ = cute.arch.block_idx() + _, split_k, _ = cute.arch.grid_dim() + warp_id = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane_id = cute.arch.lane_idx() + + NUM_HEADS = self.num_heads + MAX_DQL = self.max_decode_query_len + BLOCK_Q = NUM_HEADS * MAX_DQL + BLOCK_K = self.BLOCK_K + head_dim = self.head_dim + dtype = self.dtype + MMA_N = 8 + num_stages = self.num_stages + Q_TILES = cute.ceil_div(BLOCK_Q, MMA_N) + EPI_Q = Q_TILES * MMA_N + + sq_layout = self._sq_layout() + sk_layout = self._sk_layout() + + smem = SmemAllocator() + sK = smem.allocate_tensor( + dtype, + sk_layout.outer, + byte_alignment=128, + swizzle=sk_layout.inner, + )[0, None, None, None] + # sQ aliases the first K stage: Q is consumed into registers before the + # first K tile is needed, so the two never overlap in time. + sQ_tma = cute.make_tensor(sK[None, None, 0].iterator, layout=sq_layout.outer) + # TMA sees Q as (query, head, dim) while ldmatrix consumes a flattened Q + # column mode. The target profile keeps the rank-2 view even for + # degenerate shapes such as MAX_DQL == 1. + q_tma_elems = self._swizzle_elems() + sQ = cute.coalesce( + cute.group_modes(sQ_tma, 0, 2), + target_profile=(BLOCK_Q, (q_tma_elems, head_dim // q_tma_elems)), + ) + epi_buffer = smem.allocate_tensor(Float32, cute.make_layout((EPI_Q, 4))) + + tma_full_mbar = smem.allocate_array(Int64, num_stages) + tma_empty_mbar = smem.allocate_array(Int64, num_stages) + + # Barrier init and descriptor prefetch do not depend on the predecessor + # grid, so they are the work PDL can overlap. Every CTA runs them, even + # the ones that own no blocks, since ownership is not known until + # seq_lens is read below. + if warp_id == 0: + with cute.arch.elect_one(): + for i in cutlass.range_constexpr(num_stages): + cute.arch.mbarrier_init(tma_full_mbar + i, 1) + cute.arch.mbarrier_init(tma_empty_mbar + i, 128) + cute.arch.mbarrier_init_fence() + elif warp_id == 1: + cpasync.prefetch_descriptor(q_tma_atom) + cpasync.prefetch_descriptor(k_tma_atom) + cute.arch.sync_threads() + + if cutlass.const_expr(self.prefetch_old): + if warp_id == 4: + with cute.arch.elect_one(): + seqlen0 = seq_lens[batch_id] + num_blocks0 = cute.ceil_div(seqlen0, BLOCK_K) + stable = (seqlen0 - decode_query_len) // BLOCK_K + if stable > num_blocks0: + stable = num_blocks0 + if cutlass.const_expr(self.prefetch_depth > 0): + if stable > split_id + self.prefetch_depth * split_k: + stable = split_id + self.prefetch_depth * split_k + for block_id in range(split_id, stable, split_k): + page = block_table[batch_id, block_id] + _l2_prefetch_bulk( + gK_cache[page, None, None].iterator.toint().to(Int64), + Int32(BLOCK_K * head_dim * (dtype.width // 8)), + ) + + # No explicit launch_dependents: the only consumer is the block selector, + # which reduces over every block this grid writes, so triggering it early + # would leave it resident and spinning in its own wait. The implicit + # trigger as the CTAs exit still gives it the launch overlap. + early_q = False + early_page = Int32(0) + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + hint_blocks = cute.ceil_div(seq_lens[batch_id], BLOCK_K) + if hint_blocks > block_table.shape[1]: + hint_blocks = block_table.shape[1] + if split_id < hint_blocks: + early_q = True + early_page = block_table[batch_id, split_id] + + griddepcontrol_wait() + + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + gQ_early = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar, BLOCK_Q * head_dim * (dtype.width // 8) + ) + simple_tma_copy(q_tma_atom, gQ_early, sQ_tma, tma_full_mbar) + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, BLOCK_K * head_dim * (dtype.width // 8) + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + # seq_lens must be read after the wait. It is a graph-stable buffer that + # on_update_kv_lens patches every step, and num_blocks bounds an unmasked + # block_table load and an unmasked score store, so a stale length is an + # out-of-bounds write rather than just a wrong answer. + seqlen = seq_lens[batch_id] + num_blocks = cute.ceil_div(seqlen, BLOCK_K) + + if split_id < num_blocks: + if warp_id == 4: + # TMA warp + tma_stage = 0 + tma_parity = 1 + first_block = split_id + + if not early_q: + gQ_tile = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, tma_parity) + with cute.arch.elect_one(): + Q_size = BLOCK_Q * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(tma_full_mbar, Q_size) + # TMA bounds-checks rows when the runtime decode_query_len is + # below MAX_DQL; padded Q columns are masked before the stores. + simple_tma_copy(q_tma_atom, gQ_tile, sQ_tma, tma_full_mbar) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + if early_q: + # The first K tile was issued right after the wait. + first_block = split_id + split_k + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + for block_id in range(first_block, num_blocks, split_k): + page_id = block_table[batch_id, block_id] + gK_tile = k_tma_tensor[page_id, None, None] + k_mbar = tma_full_mbar + tma_stage + + cute.arch.mbarrier_wait(tma_empty_mbar + tma_stage, tma_parity) + with cute.arch.elect_one(): + K_size = BLOCK_K * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(k_mbar, K_size) + simple_tma_copy( + k_tma_atom, + gK_tile, + sK[None, None, tma_stage], + k_mbar, + cache_policy=EVICT_FIRST, + ) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + else: + # MMA warps; each handles K[32, head_dim] @ Q[BLOCK_Q, head_dim].T + sK_warp = cute.local_tile(sK, (32, head_dim, num_stages), (warp_id, 0, 0)) + q_start = seqlen - decode_query_len + + elems = 128 // dtype.width # 16B + MMA_K = 32 * 8 // dtype.width # 32B + + # Pre-compute ldmatrix addresses. + # sK loads a [16 x 16B] tile: + # ((16, (16B, 2), 1), (32 / 16, head_dim / 32B, num_stages)) + # sQ loads an [8 x 32B] tile: + # ((8, (16B, 4)), (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = cute.zipped_divide(sK_warp, (16, cute.make_layout((elems, 2)), 1)) + sQ_ldsm = cute.zipped_divide(sQ, (MMA_N, cute.make_layout((elems, 4)))) + + # sK: (16B, (32 / 16, head_dim / 32B, num_stages)) + # sQ: (16B, (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = sK_ldsm[(lane_id % 16, (None, lane_id // 16), 0), None] + sQ_ldsm = sQ_ldsm[(lane_id % MMA_N, (None, lane_id // 8)), None] + + ldsm_op = warp.LdMatrix8x8x16bOp(num_matrices=4) + ldsm_atom = cute.make_copy_atom(ldsm_op, dtype) + + rQ = cute.make_rmem_tensor( + ((elems // 2, 2), head_dim // (MMA_K * 2), Q_TILES), dtype + ) + rK = cute.make_rmem_tensor((elems, 2, head_dim // MMA_K), dtype) + rC = cute.make_rmem_tensor((4, 2, Q_TILES), Float32) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + for q in cutlass.range_constexpr(Q_TILES): + cute.copy(ldsm_atom, sQ_ldsm[None, (q, None)], rQ[None, None, q]) + cute.arch.mbarrier_arrive(tma_empty_mbar) + + tma_stage = 1 % self.num_stages + tma_parity = 0 + if tma_stage == 0: + tma_parity ^= 1 + + if cutlass.const_expr(dtype is Float8E4M3FN): + rQ_f16 = cute.make_rmem_tensor((4, head_dim // MMA_K, Q_TILES, 2), Float16) + q_lower, q_upper = _fp8_to_f16_mma_fragments(rQ) + rQ_f16[None, None, None, 0].store(q_lower.load()) + rQ_f16[None, None, None, 1].store(q_upper.load()) + + for block_id in range(split_id, num_blocks, split_k): + rC.fill(0.0) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar + tma_stage, tma_parity) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + for k in cutlass.range_constexpr(head_dim // MMA_K): + cute.copy( + ldsm_atom, + sK_ldsm[None, (None, k, tma_stage)], + rK[None, None, k], + ) + for m in cutlass.range_constexpr(2): + if cutlass.const_expr(dtype is Float8E4M3FN): + rK_lower, rK_upper = _fp8_to_f16_mma_fragments(rK[None, m, k]) + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK_lower, + rQ_f16[None, k, n, 0], + rC[None, m, n], + ) + rC[None, m, n] = mma_sync( + rK_upper, + rQ_f16[None, k, n, 1], + rC[None, m, n], + ) + else: + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK[None, m, k], + rQ[(None, k % 2), k // 2, n], + rC[None, m, n], + ) + + cute.arch.mbarrier_arrive(tma_empty_mbar + tma_stage) + + k_start = block_id * BLOCK_K + warp_id * 32 + + for q in cutlass.range_constexpr(Q_TILES): + for i in cutlass.range_constexpr(4): + for j in cutlass.range_constexpr(2): + col = q * 8 + (lane_id % 4) * 2 + j + q_local_pos = col % MAX_DQL + q_pos = q_start + q_local_pos + k_pos = k_start + i * 8 + lane_id // 4 + rC[q * 8 + i * 2 + j] = ( + rC[q * 8 + i * 2 + j] if q_pos >= k_pos else float("-inf") + ) + + for q in cutlass.range_constexpr(Q_TILES): + # Thread-local reduction along the BLOCK_K dim. + rScore = cute.make_rmem_tensor(2, Float32) + rScore.fill(float("-inf")) + for i in cutlass.range_constexpr(4): + rScore[0] = cute.arch.fmax(rScore[0], rC[i * 2 + 0 + q * 8]) + rScore[1] = cute.arch.fmax(rScore[1], rC[i * 2 + 1 + q * 8]) + + # Warp reduction among lanes 0, 4, 8, 12, ... + for i in cutlass.range_constexpr(3): + offset = 4 << i + other0 = cute.arch.shuffle_sync_bfly( + rScore[0], offset=offset, mask=-1, mask_and_clamp=31 + ) + other1 = cute.arch.shuffle_sync_bfly( + rScore[1], offset=offset, mask=-1, mask_and_clamp=31 + ) + rScore[0] = cute.arch.fmax(rScore[0], other0) + rScore[1] = cute.arch.fmax(rScore[1], other1) + + if lane_id * 2 < MMA_N: + epi_buffer[q * MMA_N + lane_id * 2 + 0, warp_id] = rScore[0] + epi_buffer[q * MMA_N + lane_id * 2 + 1, warp_id] = rScore[1] + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + head_id = lane_id // MAX_DQL + q_local_pos = lane_id - head_id * MAX_DQL + valid_q = head_id < NUM_HEADS and q_local_pos < decode_query_len + if lane_id < BLOCK_Q and valid_q: + final_score = epi_buffer[lane_id, 0] + for i in cutlass.range_constexpr(1, 4): + final_score = cute.arch.fmax(final_score, epi_buffer[lane_id, i]) + + t = batch_id * decode_query_len + q_local_pos + score[head_id, t, block_id] = final_score + + tma_stage = (tma_stage + 1) % self.num_stages + if tma_stage == 0: + tma_parity ^= 1 + else: + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + # Early tiles past the request's true length: let them land before the CTA exits. + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.mbarrier_wait(tma_full_mbar + 1 % num_stages, 0) + + +class IndexScoreRunner: + """Compiled ``IndexDecodeScoreKernel`` variants, keyed by (index dtype, index heads, decode query length, head dim, + split_k, early issue). Batch, query-token count, page count and block-table width stay symbolic, so one compile + serves every decode step of its key. ``prefetch_old`` is always on; ``early`` also caps the prefetch at one block + per CTA.""" + + kernel_cache: dict = {} + SUPPORTED_HEAD_DIM = 128 # capacity: fixed: the checkpoint's index head dim + SUPPORTED_PAGE_SIZE = IndexDecodeScoreKernel.BLOCK_K + # BLOCK_Q must fit one warp's worth of epilogue lanes. + MAX_BLOCK_Q = 32 # capacity: fixed: the scorer's query tile + EARLY_PREFETCH_DEPTH = 1 + + @classmethod + def is_supported( + cls, + *, + q_dtype: torch.dtype, + num_heads: int, + head_dim: int, + page_size: int, + max_decode_query_len: int, + ) -> bool: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + from tensorrt_llm._utils import is_sm_100f + + return ( + is_sm_100f() + and q_dtype in TORCH_TO_CUTE_DTYPE + and head_dim == cls.SUPPORTED_HEAD_DIM + and page_size == cls.SUPPORTED_PAGE_SIZE + and num_heads * max_decode_query_len <= cls.MAX_BLOCK_Q + and max_decode_query_len >= 1 + ) + + @classmethod + def _compile( + cls, + q_dtype: torch.dtype, + num_heads: int, + max_decode_query_len: int, + head_dim: int, + split_k: int, + early: bool, + ) -> None: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + + key = (q_dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + if key in cls.kernel_cache: + return + cute_dtype = TORCH_TO_CUTE_DTYPE[q_dtype] + page_size = cls.SUPPORTED_PAGE_SIZE + sym_total_tokens = cute.sym_int() + sym_batch = cute.sym_int() + + # 16-element divisibility on every non-innermost stride is what the TMA descriptors for Q and K assume. + def _sym_stride(): + return cute.sym_int64(divisibility=16) + + q_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (sym_total_tokens, num_heads, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + # The index-K pool may be coalesced with the main K/V cache (per-page stride above page_size * head_dim). + k_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (cute.sym_int(), page_size, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + bt_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch, cute.sym_int()), stride_order=(1, 0) + ) + # The target passes a transposed view of the [heads, blocks, tokens] selector buffer. + score_fake = cute.runtime.make_fake_tensor( + cutlass.Float32, + (num_heads, sym_total_tokens, cute.sym_int()), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + sl_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch,), stride_order=(0,) + ) + fake_stream = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True) + kernel = IndexDecodeScoreKernel( + cute_dtype, + num_heads, + max_decode_query_len, + split_k, + head_dim, + prefetch_old=True, + early_issue=early, + prefetch_depth=cls.EARLY_PREFETCH_DEPTH if early else 0, + ) + cls.kernel_cache[key] = cute_aot.compile_or_load( + f"m3_index_score {key}", + lambda: cute.compile( + kernel, + q_fake, + k_fake, + bt_fake, + score_fake, + sl_fake, + fake_stream, + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_index_score_t2sc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_index_score_t2sc.py new file mode 100644 index 000000000000..ebb916cab068 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_index_score_t2sc.py @@ -0,0 +1,702 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py +"""MiniMax-M3 index decode block scorer (CuTe DSL, Blackwell) with its pre-wait schedule: T2SC, this target's copy of +the sm_107 TP2 / TP8 targets' scorer (``modeling.INDEX_SCORE_T2SC``). The sm_103-derived ``kernels/m3_index_score`` +stays the kill path. + +The kernel of ``tensorrt_llm/_torch/attention/kernels/blackwell/minimax_m3_index_decode_score.py`` (the same scores, bit +for bit) with three schedule options this target compiles in: + +* ``prefetch_old``: before the grid-dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that hold only + tokens older than this step (the producer ahead writes only this step's tokens), up to ``prefetch_depth`` blocks per + CTA (0 = all of them). +* ``early_issue``: the TMA warp reads a ``seq_lens`` hint and its first page id before the wait and issues the Q tile + and the first K tile right after it. The tiles are read after the wait, so their contents are final; the block table + is only written between graph replays, so the early page id is the right one whenever the block is valid. A CTA + whose early tiles lie past the request's true length drains them before it exits. +* ``prestage_old`` (with ``early_issue``, ``modeling.INDEX_SCORE_T2SC_PRESTAGE_OLD``): when the CTA's first block holds + only tokens older than this step (block < (seq_lens hint - decode query length) / 128), its K tile is TMA-loaded into + its stage before the wait; Q (and a first block that is not all old) is still loaded after it. + +Every kernel is compiled before the executor's warmup (``layers.prebuild_index_scorer_t2sc``) through +``IndexScoreRunner`` and called through this target's ``m3_index_score_t2sc`` entry. +""" + +from pathlib import Path + +import cutlass +import torch +from cuda.bindings.driver import CUstream +from cutlass import Float8E4M3FN, Float16, Float32, Int32, Int64, Uint32, cute +from cutlass._mlir.dialects import llvm +from cutlass.cute.nvgpu import cpasync, warp +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + EVICT_FIRST, + fp8x4_to_fp16x4, + mma_sync, + simple_tma_copy, +) +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + TRTLLM_ENABLE_PDL, + griddepcontrol_launch_dependents, + griddepcontrol_wait, +) + +__all__ = ["IndexDecodeScoreKernel", "IndexScoreRunner"] + + +@dsl_user_op +def _l2_prefetch_bulk(addr: Int64, size: Int32, *, loc=None, ip=None) -> None: + """Prefetch ``size`` contiguous bytes (multiple of 16) at global ``addr`` into L2.""" + llvm.inline_asm( + res=None, + operands_=[Int64(addr).ir_value(loc=loc, ip=ip), Int32(size).ir_value(loc=loc, ip=ip)], + asm_string="cp.async.bulk.prefetch.L2.global [$0], $1;", + constraints="l,r", + has_side_effects=True, + asm_dialect=llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@cute.jit +def _fp8_to_f16_mma_fragments(src: cute.Tensor): + """Split an FP8 ldmatrix fragment into the two FP16 k-fragments MMA wants.""" + src_elems = cute.size(src) + src_u32 = cute.recast_tensor(src, Uint32) + src_f16 = cute.make_rmem_tensor(src_elems, Float16) + src_f16_u32 = cute.recast_tensor(src_f16, Uint32) + # Packed conversion; faster and fewer SASS instructions than + # src.load().to(Float16). + for i in cutlass.range_constexpr(src_elems // 4): + converted = fp8x4_to_fp16x4(src_u32[i]) + src_f16_u32[i * 2] = converted[0] + src_f16_u32[i * 2 + 1] = converted[1] + lower = cute.make_rmem_tensor(src_elems // 2, Float16) + upper = cute.make_rmem_tensor(src_elems // 2, Float16) + + # FP8 ldmatrix yields four consecutive values along K. Split each group + # into the lower two and upper two for the two FP16 MMA k-fragments. + for i in cutlass.range_constexpr(src_elems // 2): + lower[i] = src_f16[(i // 2) * 4 + i % 2] + upper[i] = src_f16[(i // 2) * 4 + 2 + i % 2] + return lower, upper + + +class IndexDecodeScoreKernel: + """Per-block max index score for one decode step. + + Grid is (batch, split_k): CTA (b, s) walks KV blocks + s, s + split_k, s + 2 * split_k, ... of request b and writes each block's + score directly, so no cross-CTA reduction is needed. CTAs whose split_id + exceeds the request's block count exit immediately. + """ + + BLOCK_K = 128 # capacity: fixed: the MSA block size of the checkpoint + BAR_MMA = 1 + num_stages = 2 + + def __init__( + self, + dtype: type[cutlass.Numeric], + num_heads: int, + max_decode_query_len: int, + split_k: int, + head_dim: int = 128, + prefetch_old: bool = False, + early_issue: bool = False, + prefetch_depth: int = 0, + prestage_old: bool = False, + early_trigger: bool = False, + ): + # prefetch_old: before the grid dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that + # hold only tokens older than this step (the predecessor writes only this step's tokens). + self.prefetch_old = prefetch_old + # early_issue: the TMA warp reads a seq_lens hint and this CTA's first page id before the grid dependency + # wait, then issues the Q tile and the first K tile right after it, overlapping the post-wait seq_lens load. + # The tiles are read after the wait, so their contents are final; the block table is only written between + # graph replays, so the early page id is the right one whenever the block is valid. A CTA whose early tiles + # turn out to lie past the request's true length drains them before exiting. + self.early_issue = early_issue + # prefetch_depth: blocks per CTA that prefetch_old covers (0 = all of them). Prefetching every older page at + # large batches floods L2; the first few per CTA are the ones whose latency the pipeline cannot hide. + self.prefetch_depth = prefetch_depth + # prestage_old (with early_issue): when this CTA's first block holds only tokens older than this step, its K + # tile is TMA-loaded into its stage BEFORE the grid dependency wait (the predecessor writes only this step's + # tokens; the block table and the seq_lens hint are final before the grid starts). Q stays after the wait. + self.prestage_old = prestage_old and early_issue + # early_trigger (on for this target): release the dependent selector right after this CTA's + # setup, so its launch and residency overlap the scoring; it still waits for this whole grid. + self.early_trigger = early_trigger + self.dtype = dtype + self.num_heads = num_heads + self.max_decode_query_len = max_decode_query_len + self.split_k = split_k + self.head_dim = head_dim + + def _swizzle_elems(self) -> int: + """Elements spanned by one 128-byte swizzle atom.""" + return 128 * 8 // self.dtype.width + + def _sq_layout(self): + """Composed SMEM layout for the Q tile, shared by the descriptor and the kernel.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_q = self.num_heads * self.max_decode_query_len + layout = cute.make_layout( + (self.max_decode_query_len, self.num_heads, (elems, head_dim // elems)), + stride=(elems, self.max_decode_query_len * elems, (1, block_q * elems)), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + def _sk_layout(self): + """Composed SMEM layout for the pipelined K tiles.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_k = self.BLOCK_K + layout = cute.make_layout( + (1, block_k, (elems, head_dim // elems), self.num_stages), + stride=(0, elems, (1, block_k * elems), block_k * head_dim), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + @cute.jit + def __call__( + self, + gQ: cute.Tensor, # [bs * runtime_decode_query_len, num_heads, head_dim] + gK_cache: cute.Tensor, # [num_pages, page_size, head_dim] + block_table: cute.Tensor, # [bs, max_pages] + score: cute.Tensor, # [num_heads, bs * runtime_decode_query_len, max_pages] + seq_lens: cute.Tensor, # [bs] + stream: CUstream, + ): + num_heads = self.num_heads + head_dim = self.head_dim + MAX_DQL = self.max_decode_query_len + assert num_heads * MAX_DQL <= 32 + + batch = seq_lens.shape[0] + decode_query_len = gQ.shape[0] // batch + grid = (batch, self.split_k, 1) + block = (32 * 5, 1, 1) + + tma_g2s = cpasync.CopyBulkTensorTileG2SOp() + elems = self._swizzle_elems() + + q_tma_atom, q_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gQ, (None, None, elems)), + self._sq_layout(), + (MAX_DQL, num_heads, head_dim), + ) + k_tma_atom, k_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gK_cache, (None, None, elems)), + self._sk_layout(), + (1, self.BLOCK_K, head_dim), + ) + + self.kernel( + q_tma_atom, + q_tma_tensor, + k_tma_atom, + k_tma_tensor, + gK_cache, + block_table, + score, + seq_lens, + decode_query_len, + ).launch(grid=grid, block=block, stream=stream, use_pdl=TRTLLM_ENABLE_PDL) + + @cute.kernel + def kernel( + self, + q_tma_atom: cute.CopyAtom, + q_tma_tensor: cute.Tensor, + k_tma_atom: cute.CopyAtom, + k_tma_tensor: cute.Tensor, + gK_cache: cute.Tensor, + block_table: cute.Tensor, + score: cute.Tensor, + seq_lens: cute.Tensor, + decode_query_len, + ): + batch_id, split_id, _ = cute.arch.block_idx() + _, split_k, _ = cute.arch.grid_dim() + warp_id = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane_id = cute.arch.lane_idx() + + NUM_HEADS = self.num_heads + MAX_DQL = self.max_decode_query_len + BLOCK_Q = NUM_HEADS * MAX_DQL + BLOCK_K = self.BLOCK_K + head_dim = self.head_dim + dtype = self.dtype + MMA_N = 8 + num_stages = self.num_stages + Q_TILES = cute.ceil_div(BLOCK_Q, MMA_N) + EPI_Q = Q_TILES * MMA_N + + sq_layout = self._sq_layout() + sk_layout = self._sk_layout() + + smem = SmemAllocator() + sK = smem.allocate_tensor( + dtype, + sk_layout.outer, + byte_alignment=128, + swizzle=sk_layout.inner, + )[0, None, None, None] + # sQ aliases the first K stage: Q is consumed into registers before the + # first K tile is needed, so the two never overlap in time. + sQ_tma = cute.make_tensor(sK[None, None, 0].iterator, layout=sq_layout.outer) + # TMA sees Q as (query, head, dim) while ldmatrix consumes a flattened Q + # column mode. The target profile keeps the rank-2 view even for + # degenerate shapes such as MAX_DQL == 1. + q_tma_elems = self._swizzle_elems() + sQ = cute.coalesce( + cute.group_modes(sQ_tma, 0, 2), + target_profile=(BLOCK_Q, (q_tma_elems, head_dim // q_tma_elems)), + ) + epi_buffer = smem.allocate_tensor(Float32, cute.make_layout((EPI_Q, 4))) + + tma_full_mbar = smem.allocate_array(Int64, num_stages) + tma_empty_mbar = smem.allocate_array(Int64, num_stages) + + # Barrier init and descriptor prefetch do not depend on the predecessor + # grid, so they are the work PDL can overlap. Every CTA runs them, even + # the ones that own no blocks, since ownership is not known until + # seq_lens is read below. + if warp_id == 0: + with cute.arch.elect_one(): + for i in cutlass.range_constexpr(num_stages): + cute.arch.mbarrier_init(tma_full_mbar + i, 1) + cute.arch.mbarrier_init(tma_empty_mbar + i, 128) + cute.arch.mbarrier_init_fence() + elif warp_id == 1: + cpasync.prefetch_descriptor(q_tma_atom) + cpasync.prefetch_descriptor(k_tma_atom) + cute.arch.sync_threads() + if cutlass.const_expr(self.early_trigger): + griddepcontrol_launch_dependents() + + if cutlass.const_expr(self.prefetch_old): + if warp_id == 4: + with cute.arch.elect_one(): + seqlen0 = seq_lens[batch_id] + num_blocks0 = cute.ceil_div(seqlen0, BLOCK_K) + stable = (seqlen0 - decode_query_len) // BLOCK_K + if stable > num_blocks0: + stable = num_blocks0 + if cutlass.const_expr(self.prefetch_depth > 0): + if stable > split_id + self.prefetch_depth * split_k: + stable = split_id + self.prefetch_depth * split_k + for block_id in range(split_id, stable, split_k): + page = block_table[batch_id, block_id] + _l2_prefetch_bulk( + gK_cache[page, None, None].iterator.toint().to(Int64), + Int32(BLOCK_K * head_dim * (dtype.width // 8)), + ) + + # Without early_trigger the selector launches at this grid's implicit trigger (the CTAs' exit). With it, the + # selector is already resident in its own grid-dependency wait, which still returns only once every block + # this grid writes is complete. + early_q = False + early_k = False + early_page = Int32(0) + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + seqlen_hint = seq_lens[batch_id] + hint_blocks = cute.ceil_div(seqlen_hint, BLOCK_K) + if hint_blocks > block_table.shape[1]: + hint_blocks = block_table.shape[1] + if split_id < hint_blocks: + early_q = True + early_page = block_table[batch_id, split_id] + if cutlass.const_expr(self.prestage_old): + # Stage the first K tile now when every token of the block predates this step. + if split_id < (seqlen_hint - decode_query_len) // BLOCK_K: + early_k = True + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, + BLOCK_K * head_dim * (dtype.width // 8), + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + griddepcontrol_wait() + + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + gQ_early = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar, BLOCK_Q * head_dim * (dtype.width // 8) + ) + simple_tma_copy(q_tma_atom, gQ_early, sQ_tma, tma_full_mbar) + if not early_k: + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, + BLOCK_K * head_dim * (dtype.width // 8), + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + # The length that bounds this grid's work is read after the wait (the + # pre-wait reads above are prefetch / early-issue hints only, clamped to + # the table width). seq_lens is a graph-stable buffer that on_update_kv_lens + # patches every step, and num_blocks bounds an unmasked block_table load + # and an unmasked score store, so a stale length is an out-of-bounds write + # rather than just a wrong answer. + seqlen = seq_lens[batch_id] + num_blocks = cute.ceil_div(seqlen, BLOCK_K) + + if split_id < num_blocks: + if warp_id == 4: + # TMA warp + tma_stage = 0 + tma_parity = 1 + first_block = split_id + + if not early_q: + gQ_tile = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, tma_parity) + with cute.arch.elect_one(): + Q_size = BLOCK_Q * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(tma_full_mbar, Q_size) + # TMA bounds-checks rows when the runtime decode_query_len is + # below MAX_DQL; padded Q columns are masked before the stores. + simple_tma_copy(q_tma_atom, gQ_tile, sQ_tma, tma_full_mbar) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + if early_q: + # The first K tile was issued right after the wait. + first_block = split_id + split_k + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + for block_id in range(first_block, num_blocks, split_k): + page_id = block_table[batch_id, block_id] + gK_tile = k_tma_tensor[page_id, None, None] + k_mbar = tma_full_mbar + tma_stage + + cute.arch.mbarrier_wait(tma_empty_mbar + tma_stage, tma_parity) + with cute.arch.elect_one(): + K_size = BLOCK_K * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(k_mbar, K_size) + simple_tma_copy( + k_tma_atom, + gK_tile, + sK[None, None, tma_stage], + k_mbar, + cache_policy=EVICT_FIRST, + ) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + else: + # MMA warps; each handles K[32, head_dim] @ Q[BLOCK_Q, head_dim].T + sK_warp = cute.local_tile(sK, (32, head_dim, num_stages), (warp_id, 0, 0)) + q_start = seqlen - decode_query_len + + elems = 128 // dtype.width # 16B + MMA_K = 32 * 8 // dtype.width # 32B + + # Pre-compute ldmatrix addresses. + # sK loads a [16 x 16B] tile: + # ((16, (16B, 2), 1), (32 / 16, head_dim / 32B, num_stages)) + # sQ loads an [8 x 32B] tile: + # ((8, (16B, 4)), (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = cute.zipped_divide(sK_warp, (16, cute.make_layout((elems, 2)), 1)) + sQ_ldsm = cute.zipped_divide(sQ, (MMA_N, cute.make_layout((elems, 4)))) + + # sK: (16B, (32 / 16, head_dim / 32B, num_stages)) + # sQ: (16B, (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = sK_ldsm[(lane_id % 16, (None, lane_id // 16), 0), None] + sQ_ldsm = sQ_ldsm[(lane_id % MMA_N, (None, lane_id // 8)), None] + + ldsm_op = warp.LdMatrix8x8x16bOp(num_matrices=4) + ldsm_atom = cute.make_copy_atom(ldsm_op, dtype) + + rQ = cute.make_rmem_tensor( + ((elems // 2, 2), head_dim // (MMA_K * 2), Q_TILES), dtype + ) + rK = cute.make_rmem_tensor((elems, 2, head_dim // MMA_K), dtype) + rC = cute.make_rmem_tensor((4, 2, Q_TILES), Float32) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + for q in cutlass.range_constexpr(Q_TILES): + cute.copy(ldsm_atom, sQ_ldsm[None, (q, None)], rQ[None, None, q]) + cute.arch.mbarrier_arrive(tma_empty_mbar) + + tma_stage = 1 % self.num_stages + tma_parity = 0 + if tma_stage == 0: + tma_parity ^= 1 + + if cutlass.const_expr(dtype is Float8E4M3FN): + rQ_f16 = cute.make_rmem_tensor((4, head_dim // MMA_K, Q_TILES, 2), Float16) + q_lower, q_upper = _fp8_to_f16_mma_fragments(rQ) + rQ_f16[None, None, None, 0].store(q_lower.load()) + rQ_f16[None, None, None, 1].store(q_upper.load()) + + for block_id in range(split_id, num_blocks, split_k): + rC.fill(0.0) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar + tma_stage, tma_parity) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + for k in cutlass.range_constexpr(head_dim // MMA_K): + cute.copy( + ldsm_atom, + sK_ldsm[None, (None, k, tma_stage)], + rK[None, None, k], + ) + for m in cutlass.range_constexpr(2): + if cutlass.const_expr(dtype is Float8E4M3FN): + rK_lower, rK_upper = _fp8_to_f16_mma_fragments(rK[None, m, k]) + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK_lower, + rQ_f16[None, k, n, 0], + rC[None, m, n], + ) + rC[None, m, n] = mma_sync( + rK_upper, + rQ_f16[None, k, n, 1], + rC[None, m, n], + ) + else: + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK[None, m, k], + rQ[(None, k % 2), k // 2, n], + rC[None, m, n], + ) + + cute.arch.mbarrier_arrive(tma_empty_mbar + tma_stage) + + k_start = block_id * BLOCK_K + warp_id * 32 + + for q in cutlass.range_constexpr(Q_TILES): + for i in cutlass.range_constexpr(4): + for j in cutlass.range_constexpr(2): + col = q * 8 + (lane_id % 4) * 2 + j + q_local_pos = col % MAX_DQL + q_pos = q_start + q_local_pos + k_pos = k_start + i * 8 + lane_id // 4 + rC[q * 8 + i * 2 + j] = ( + rC[q * 8 + i * 2 + j] if q_pos >= k_pos else float("-inf") + ) + + for q in cutlass.range_constexpr(Q_TILES): + # Thread-local reduction along the BLOCK_K dim. + rScore = cute.make_rmem_tensor(2, Float32) + rScore.fill(float("-inf")) + for i in cutlass.range_constexpr(4): + rScore[0] = cute.arch.fmax(rScore[0], rC[i * 2 + 0 + q * 8]) + rScore[1] = cute.arch.fmax(rScore[1], rC[i * 2 + 1 + q * 8]) + + # Warp reduction among lanes 0, 4, 8, 12, ... + for i in cutlass.range_constexpr(3): + offset = 4 << i + other0 = cute.arch.shuffle_sync_bfly( + rScore[0], offset=offset, mask=-1, mask_and_clamp=31 + ) + other1 = cute.arch.shuffle_sync_bfly( + rScore[1], offset=offset, mask=-1, mask_and_clamp=31 + ) + rScore[0] = cute.arch.fmax(rScore[0], other0) + rScore[1] = cute.arch.fmax(rScore[1], other1) + + if lane_id * 2 < MMA_N: + epi_buffer[q * MMA_N + lane_id * 2 + 0, warp_id] = rScore[0] + epi_buffer[q * MMA_N + lane_id * 2 + 1, warp_id] = rScore[1] + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + head_id = lane_id // MAX_DQL + q_local_pos = lane_id - head_id * MAX_DQL + valid_q = head_id < NUM_HEADS and q_local_pos < decode_query_len + if lane_id < BLOCK_Q and valid_q: + final_score = epi_buffer[lane_id, 0] + for i in cutlass.range_constexpr(1, 4): + final_score = cute.arch.fmax(final_score, epi_buffer[lane_id, i]) + + t = batch_id * decode_query_len + q_local_pos + score[head_id, t, block_id] = final_score + + tma_stage = (tma_stage + 1) % self.num_stages + if tma_stage == 0: + tma_parity ^= 1 + else: + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + # Early tiles past the request's true length: let them land before the CTA exits. + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.mbarrier_wait(tma_full_mbar + 1 % num_stages, 0) + + +class IndexScoreRunner: + """Compiled ``IndexDecodeScoreKernel`` variants, keyed by (index dtype, index heads, decode query length, head dim, + split_k, early issue). Batch, query-token count, page count and block-table width stay symbolic, so one compile + serves every decode step of its key. ``prefetch_old`` is always on; ``early`` also caps the prefetch at one block + per CTA.""" + + kernel_cache: dict = {} + SUPPORTED_HEAD_DIM = 128 # capacity: fixed: the checkpoint's index head dim + SUPPORTED_PAGE_SIZE = IndexDecodeScoreKernel.BLOCK_K + # BLOCK_Q must fit one warp's worth of epilogue lanes. + MAX_BLOCK_Q = 32 # capacity: fixed: the scorer's query tile + EARLY_PREFETCH_DEPTH = 1 + + @classmethod + def is_supported( + cls, + *, + q_dtype: torch.dtype, + num_heads: int, + head_dim: int, + page_size: int, + max_decode_query_len: int, + ) -> bool: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + from tensorrt_llm._utils import is_sm_100f + + return ( + is_sm_100f() + and q_dtype in TORCH_TO_CUTE_DTYPE + and head_dim == cls.SUPPORTED_HEAD_DIM + and page_size == cls.SUPPORTED_PAGE_SIZE + and num_heads * max_decode_query_len <= cls.MAX_BLOCK_Q + and max_decode_query_len >= 1 + ) + + @classmethod + def _compile( + cls, + q_dtype: torch.dtype, + num_heads: int, + max_decode_query_len: int, + head_dim: int, + split_k: int, + early: bool, + ) -> None: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + + key = (q_dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + if key in cls.kernel_cache: + return + cute_dtype = TORCH_TO_CUTE_DTYPE[q_dtype] + page_size = cls.SUPPORTED_PAGE_SIZE + sym_total_tokens = cute.sym_int() + sym_batch = cute.sym_int() + + # 16-element divisibility on every non-innermost stride is what the TMA descriptors for Q and K assume. + def _sym_stride(): + return cute.sym_int64(divisibility=16) + + q_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (sym_total_tokens, num_heads, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + # The index-K pool may be coalesced with the main K/V cache (per-page stride above page_size * head_dim). + k_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (cute.sym_int(), page_size, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + bt_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch, cute.sym_int()), stride_order=(1, 0) + ) + # The target passes a transposed view of the [heads, blocks, tokens] selector buffer. + score_fake = cute.runtime.make_fake_tensor( + cutlass.Float32, + (num_heads, sym_total_tokens, cute.sym_int()), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + sl_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch,), stride_order=(0,) + ) + fake_stream = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True) + kernel = IndexDecodeScoreKernel( + cute_dtype, + num_heads, + max_decode_query_len, + split_k, + head_dim, + prefetch_old=True, + early_issue=early, + prefetch_depth=cls.EARLY_PREFETCH_DEPTH if early else 0, + prestage_old=early, + early_trigger=True, + ) + cls.kernel_cache[key] = cute_aot.compile_or_load( + f"m3_index_score_t2sc {key}", + lambda: cute.compile( + kernel, + q_fake, + k_fake, + bt_fake, + score_fake, + sl_fake, + fake_stream, + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_lowm_splitk.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_lowm_splitk.py new file mode 100644 index 000000000000..53cdbd383ad8 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_lowm_splitk.py @@ -0,0 +1,1259 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-License-Identifier: Apache-2.0 +"""Blackwell low-M BF16/FP16 GEMM with an in-kernel cluster split-K reduction. + +This target's copy of FlashInfer's ``flashinfer/gemm/kernels/dense_bf16_gemm_sm100_splitk.py`` (the Eagle3 +drafter's low-M linears, ``kernels/m3_draft_lowm``): the same kernel and the same split order at every tactic, +bit for bit FlashInfer's output (spec/test_modeling_v2_m3_lowm_splitk.py). The one change: the owner rank's final +cast and store is one method, ``_store_tile``, the seam for the drafter-boundary push epilogue and, on the A side, the +SwiGLU prologue that follow. + +Each cluster rank accumulates an exact K slice in FP32. Peers publish partials +to rank 0 through DSMEM; rank 0 reduces, casts, and stores once. The public +``A[M, K] @ B[K, N]`` problem is swapped internally, so tile dimensions below +use kernel coordinates: kernel-M carries public N and kernel-N carries public M. +""" + +from __future__ import annotations + +import dataclasses +from pathlib import Path + +import cuda.bindings.driver as _cuda +import cutlass +import cutlass.cute as cute +import cutlass.utils as utils +import cutlass.utils.blackwell_helpers as sm100_utils +from cutlass import Int32 +from cutlass._mlir.dialects import llvm +from cutlass.cute import experimental as cute_ext +from cutlass.cute.nvgpu import tcgen05 +from cutlass.cute.runtime import from_dlpack +from cutlass.cutlass_dsl import T, dsl_user_op + +from tensorrt_llm._torch import cute_aot + +#: Per-CTA SMEM capacity reported by CuTeDSL on SM100/SM103. +_SMEM_CAPACITY_BYTES = utils.get_smem_capacity_in_bytes("sm_100") + +#: K extent of one CTA tile. +_CTA_K = 128 # capacity: fixed: a kernel tile constant + +#: Kernel-M tiles; 64 increases CTA count for low-M decode shapes. +_SUPPORTED_MMA_M = (64, 128) + +#: Kernel-N carries public M, which is limited to 32. +_SUPPORTED_MMA_N = (8, 16, 32) + +#: Physical cluster-K sizes; split 1 compiles out the DSMEM path. +_SUPPORTED_SPLIT_K = (1, 2, 3, 4) + +#: Largest public M this low-M policy serves. +_MAX_M = 32 # capacity: fixed: the copied FlashInfer kernel's row bound + +#: Bytes per FP32 partial exchanged through DSMEM. +_FP32_BYTES = 4 + +#: DSMEM mailbox base alignment, in bytes. +_MAILBOX_ALIGN_BYTES = 128 + +#: Size and alignment of one mbarrier, in bytes. +_MBARRIER_BYTES = 8 + +#: Size and alignment of the TMEM base pointer slot. +_TMEM_POINTER_BYTES = 4 + +#: Bytes per BF16/FP16 element. +_AB_ELEMENT_BYTES = 2 + +#: Alignment of the A/B shared-memory buffers. +_AB_BUFFER_ALIGN_BYTES = 1024 # capacity: fixed: an alignment + +#: A/B pipeline stage bounds. One stage is useful only for a single K tile. +_MIN_AB_STAGES = 1 # capacity: fixed: pipeline stages +_DEFAULT_MIN_AB_STAGES = 2 +_MAX_AB_STAGES = 12 # capacity: fixed: pipeline stages + + +@dataclasses.dataclass(frozen=True) +class SplitKTactic: + """One specialization; mma_m carries public N and mma_n carries public M.""" + + mma_m: int + mma_n: int + split_k: int + ab_stages: int + + +def _align_up(value: int, alignment: int) -> int: + return ((value + alignment - 1) // alignment) * alignment + + +def _smem_bytes( + tactic: SplitKTactic, + ab_stages: int, +) -> int: + """Mirror the device allocator's shared-memory layout.""" + cursor = ( + _align_up( + tactic.mma_m * _CTA_K * _AB_ELEMENT_BYTES * ab_stages, + _AB_BUFFER_ALIGN_BYTES, + ) + + tactic.mma_n * _CTA_K * _AB_ELEMENT_BYTES * ab_stages + ) + + cursor = _align_up(cursor, _MBARRIER_BYTES) + cursor += 2 * ab_stages * _MBARRIER_BYTES + cursor += 3 * _MBARRIER_BYTES + cursor = _align_up(cursor, _TMEM_POINTER_BYTES) + cursor += _TMEM_POINTER_BYTES + + if tactic.split_k == 1: + return cursor + + return ( + _align_up( + _align_up(cursor, _MAILBOX_ALIGN_BYTES) + + (tactic.split_k - 1) * tactic.mma_m * tactic.mma_n * _FP32_BYTES, + _MBARRIER_BYTES, + ) + + _MBARRIER_BYTES + ) + + +def _max_ab_stages_for( + tactic: SplitKTactic, + smem_capacity: int, +) -> int: + return next( + ( + stages + for stages in range(_MAX_AB_STAGES, -1, -1) + if _smem_bytes(tactic, stages) <= smem_capacity + ), + 0, + ) + + +def validate_tactic( + tactic: SplitKTactic, + m: int, + n: int, + k: int, + *, + smem_capacity: int = _SMEM_CAPACITY_BYTES, +) -> None: + """Reject a tactic that cannot serve ``(m, n, k)``.""" + if tactic.mma_m not in _SUPPORTED_MMA_M: + raise ValueError(f"unsupported mma_m={tactic.mma_m}") + if tactic.mma_n not in _SUPPORTED_MMA_N: + raise ValueError(f"unsupported mma_n={tactic.mma_n}") + if tactic.split_k not in _SUPPORTED_SPLIT_K: + raise ValueError(f"unsupported split_k={tactic.split_k}") + if not _MIN_AB_STAGES <= tactic.ab_stages <= _MAX_AB_STAGES: + raise ValueError( + f"ab_stages must be in [{_MIN_AB_STAGES}, {_MAX_AB_STAGES}], got {tactic.ab_stages}" + ) + if not 1 <= m <= _MAX_M: + raise ValueError(f"this low-M policy requires 1 <= M <= {_MAX_M}, got {m}") + if n <= 0: + raise ValueError(f"N must be positive, got {n}") + if k <= 0 or k % _CTA_K or (k // _CTA_K) % tactic.split_k: + raise ValueError( + f"K={k} with CTA_K={_CTA_K} does not divide evenly across split_k={tactic.split_k}" + ) + smem_bytes = _smem_bytes(tactic, tactic.ab_stages) + if smem_bytes > smem_capacity: + raise ValueError( + f"tactic {tactic} needs {smem_bytes} B of shared memory but only " + f"{smem_capacity} B are available; max ab_stages is " + f"{_max_ab_stages_for(tactic, smem_capacity)}" + ) + + +def autotune_tactics( + m: int, + n: int, + k: int, + *, + smem_capacity: int = _SMEM_CAPACITY_BYTES, +) -> list[SplitKTactic]: + """Return valid tactics in the shape-specific stage window.""" + tactics: list[SplitKTactic] = [] + for mma_m in _SUPPORTED_MMA_M: + for mma_n in _SUPPORTED_MMA_N: + for split_k in _SUPPORTED_SPLIT_K: + min_stages = _MIN_AB_STAGES if k == _CTA_K else _DEFAULT_MIN_AB_STAGES + base = SplitKTactic(mma_m, mma_n, split_k, min_stages) + try: + validate_tactic(base, m, n, k, smem_capacity=smem_capacity) + except ValueError: + continue + max_stages = _max_ab_stages_for(base, smem_capacity) + # Short K favors shallow pipelines; long K stays near the cap. + tactics.extend( + dataclasses.replace(base, ab_stages=ab_stages) + for ab_stages in ( + range(min_stages, min(max_stages, 6) + 1) + if k <= 4 * _CTA_K + else range( + min(max(5, max_stages - 2), max_stages), + max_stages + 1, + ) + ) + ) + return tactics + + +def default_tactic(m: int, n: int, k: int) -> SplitKTactic: + """Choose the default occupancy-oriented tactic.""" + if n <= 512: + mma_m = 64 + mma_n = 16 if n == 512 and m > 24 else 8 + requested_split = 4 + else: + mma_n = 8 if m <= 8 else 16 if m <= 16 else 32 + if n <= 3072: + mma_m = 128 if m <= 16 else 64 + requested_split = 4 if m <= 16 else 2 + elif n < 8192: + mma_m = 64 + requested_split = 2 + else: + mma_m = 128 if k <= 1024 and m <= 24 else 64 + requested_split = 1 + + if k <= 4 * _CTA_K: + requested_split = 1 + split_k = next( + split_k + for split_k in reversed(_SUPPORTED_SPLIT_K) + if split_k <= requested_split and (k // _CTA_K) % split_k == 0 + ) + tactic = SplitKTactic(mma_m, mma_n, split_k, _MIN_AB_STAGES) + max_stages = _max_ab_stages_for(tactic, _SMEM_CAPACITY_BYTES) + tactic = dataclasses.replace( + tactic, + ab_stages=( + _MIN_AB_STAGES + if k == _CTA_K + else _DEFAULT_MIN_AB_STAGES + if k <= 2 * _CTA_K and m > 8 + else min(max_stages, 6) + if k <= 4 * _CTA_K + else max_stages + ), + ) + validate_tactic(tactic, m, n, k) + return tactic + + +__all__ = [ + "SplitKTactic", + "autotune_tactics", + "default_tactic", + "run_splitk_dense", +] + + +@dsl_user_op +def _map_shared_rank( + smem_ptr: "cute.Pointer", + peer_cta_rank_in_cluster: "Int32", + *, + loc=None, + ip=None, +) -> "Int32": + """Map an SMEM pointer into a peer CTA's address space.""" + return Int32( + llvm.inline_asm( + T.i32(), + [ + smem_ptr.toint(loc=loc, ip=ip).ir_value(), + peer_cta_rank_in_cluster.ir_value(), + ], + "mapa.shared::cluster.u32 $0, $1, $2;", + "=r,r,r", + has_side_effects=False, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +@dsl_user_op +def _store_shared_remote_v4( + value0, + value1, + value2, + value3, + smem_ptr: "cute.Pointer", + mbar_ptr: "cute.Pointer", + peer_cta_rank_in_cluster: "Int32", + *, + loc=None, + ip=None, +) -> None: + """Publish four FP32 partials into a peer's SMEM, crediting 16 bytes.""" + llvm.inline_asm( + None, + [ + _map_shared_rank(smem_ptr, peer_cta_rank_in_cluster, loc=loc, ip=ip).ir_value(), + value0.bitcast(Int32).ir_value(loc=loc, ip=ip), + value1.bitcast(Int32).ir_value(loc=loc, ip=ip), + value2.bitcast(Int32).ir_value(loc=loc, ip=ip), + value3.bitcast(Int32).ir_value(loc=loc, ip=ip), + _map_shared_rank(mbar_ptr, peer_cta_rank_in_cluster, loc=loc, ip=ip).ir_value(), + ], + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 " + "[$0], {$1, $2, $3, $4}, [$5];", + "r,r,r,r,r,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + + +#: Rank that gathers partials and stores the output. +OWNER_RANK = 0 # capacity: fixed: a split index, not a size + + +# The drafter-boundary push (``ar_push``, modeling.DRAFT_AR_PUSH): the exchange words of +# kernels/m3_mc_allreduce, as m3_mx_gemv's ``ar`` push stores them. +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return Int32( + llvm.inline_asm( + T.i32(), + [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], + "ld.volatile.global.s32 $0, [$1];", + "=r,l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +@dsl_user_op +def _st_global_b32(addr, value, *, loc=None, ip=None): + llvm.inline_asm( + None, + [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), Int32(value).ir_value(loc=loc, ip=ip)], + "st.global.b32 [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return Int32( + llvm.inline_asm( + T.i32(), + [ + cutlass.Float32(hi).ir_value(loc=loc, ip=ip), + cutlass.Float32(lo).ir_value(loc=loc, ip=ip), + ], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", + "=r,f,f", + has_side_effects=False, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +class SplitKDenseGemmKernel: + """Standalone BF16/FP16 GEMM with a cluster-local split-K reduction.""" + + def __init__( + self, + *, + tactic: SplitKTactic, + use_pdl: bool, + has_bias: bool, + ar_push: bool = False, + ) -> None: + self.ar_push = ar_push + self.acc_dtype = cutlass.Float32 + self.cta_m = tactic.mma_m + self.cta_n = tactic.mma_n + self.cta_k = _CTA_K + self.num_ab_stage = tactic.ab_stages + self.split_k = tactic.split_k + self.use_pdl = use_pdl + self.has_bias = has_bias + + self.threads_per_cta = 256 + self.epilog_threads = 128 + self.mma_tiler_mn = (tactic.mma_m, tactic.mma_n) + self.cta_group = tcgen05.CtaGroup.ONE + self.tma_op = cute_ext.OperationTypeEnum.SM90_TMA_LOAD + self.cluster_shape = (1, tactic.split_k, 1) + + values_per_thread = (tactic.mma_m * tactic.mma_n) // self.epilog_threads + if values_per_thread % 4: + raise ValueError( + f"CTA tile ({tactic.mma_m}, {tactic.mma_n}) gives " + f"{values_per_thread} " + "values per epilogue thread; remote stores require a multiple of 4" + ) + self.mailbox_elements = (tactic.split_k - 1) * self.epilog_threads * values_per_thread + self.expected_transaction_bytes = self.mailbox_elements * _FP32_BYTES + + @cute.experimental.jit + def __call__( + self, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + bias: cute.Tensor, + stream: _cuda.CUstream, + ar_ws: cutlass.Int64 = cutlass.Int64(0), + ar_flags: cutlass.Int64 = cutlass.Int64(0), + ar_rank: Int32 = Int32(0), + ar_world: Int32 = Int32(1), + ar_maxt: Int32 = Int32(8), + ar_tokens: Int32 = Int32(0), + ar_n: Int32 = Int32(0), + ): + # Grid-y packs output-N tile and cluster rank. + self.kernel( + a, b, c, bias, ar_ws, ar_flags, ar_rank, ar_world, ar_maxt, ar_tokens, ar_n + ).launch( + grid=( + cute.ceil_div(c.layout.shape[0], self.cta_m), + cute.ceil_div(c.layout.shape[1], self.cta_n) * self.split_k, + c.layout.shape[2], + ), + block=(self.threads_per_cta, 1, 1), + cluster=self.cluster_shape, + smem=cute.Int64(utils.get_smem_capacity_in_bytes("sm_100")), + stream=stream, + use_pdl=self.use_pdl, + ) + + @cute.experimental.kernel + def kernel( + self, + mA: cute.Tensor, # (Gemm_M, Gemm_K, Gemm_L), K-major + mB: cute.Tensor, # (Gemm_N, Gemm_K, Gemm_L), K-major + mC: cute.Tensor, # (Gemm_M, Gemm_N, Gemm_L), M-major + mBias: cute.Tensor, # Broadcast bias; dead when has_bias=False + ar_ws: cutlass.Int64, # ar_push: the boundary all-reduce's multicast exchange words (int32) + ar_flags: cutlass.Int64, # ar_push: its per-(token, cluster CTA) call flags (parity = flag & 1) + ar_rank: Int32, + ar_world: Int32, + ar_maxt: Int32, # the bucket's tokens + ar_tokens: Int32, # rows pushed (public M) + ar_n: Int32, # public N (the hidden size): the exchange row width + ): + """Allocate storage and dispatch the specialized warps.""" + stages = self.num_ab_stage + + ab_dtype = mA.element_type + tiled_mma = sm100_utils.make_trivial_tiled_mma( + ab_dtype, + ab_dtype, + utils.LayoutEnum.from_tensor(mA).mma_major_mode(), + utils.LayoutEnum.from_tensor(mB).mma_major_mode(), + self.acc_dtype, + self.cta_group, + self.mma_tiler_mn, + ) + + mnk_tiler = (self.mma_tiler_mn[0], self.mma_tiler_mn[1], self.cta_k) + block_idx = cute.arch.block_idx() + bidx = block_idx[0] + split_rank = cute.arch.make_warp_uniform(cute.arch.block_idx_in_cluster()) + n_idx = block_idx[1] // self.split_k + l_idx = block_idx[2] + warp_idx = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + + sA = cute_ext.allocate( + ab_dtype, + cute.AddressSpace.smem, + sm100_utils.make_smem_layout_a(tiled_mma, mnk_tiler, ab_dtype, stages), + alignment=_AB_BUFFER_ALIGN_BYTES, + ) + sB = cute_ext.allocate( + ab_dtype, + cute.AddressSpace.smem, + sm100_utils.make_smem_layout_b(tiled_mma, mnk_tiler, ab_dtype, stages), + alignment=_AB_BUFFER_ALIGN_BYTES, + ) + + acc_layout = cute_ext.make_tmem_layout_acc(tiled_mma, self.mma_tiler_mn, acc_stage=1) + c_tiler_mn = (self.cta_m, self.cta_n) + + bar_full = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(stages), + alignment=_MBARRIER_BYTES, + ).iterator + bar_empty = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(stages), + alignment=_MBARRIER_BYTES, + ).iterator + bar_tma_epilog = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + bar_mma_epilog = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + bar_tmem_alloc = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + tmem_base_ptr = cute_ext.allocate( + cutlass.Int32, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_TMEM_POINTER_BYTES, + ).iterator + # ar_push: the owner's BF16 tile (kernel-M = output column, kernel-N = token), M-major. + sD = cute_ext.allocate( + mC.element_type, + cute.AddressSpace.smem, + cute.make_layout((self.cta_m, self.cta_n), stride=(1, self.cta_m)), + alignment=16, + ) + + if cutlass.const_expr(self.split_k > 1): + mailbox = cute_ext.allocate( + cutlass.Float32, + cute.AddressSpace.smem, + cute.make_layout(self.mailbox_elements), + alignment=_MAILBOX_ALIGN_BYTES, + ) + bar_reduce = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + else: + # Dummy operands for the compile-time-elided reduction. + mailbox = sA + bar_reduce = bar_mma_epilog + + if warp_idx == 0: + with cute.arch.elect_one(): + for i in range(stages): + cute.arch.mbarrier_init(bar_full + i, 2) + cute.arch.mbarrier_init(bar_empty + i, 1) + cute.arch.mbarrier_init(bar_tma_epilog, 32) + cute.arch.mbarrier_init(bar_mma_epilog, 1) + cute.arch.mbarrier_init(bar_tmem_alloc, 160) + + if cutlass.const_expr(self.split_k > 1): + # Owner arrival plus peer transaction-byte credits. + cute.arch.mbarrier_init(bar_reduce, 1) + + cute.arch.mbarrier_init_fence() + if cutlass.const_expr(self.split_k > 1): + # Publish peer barriers before cross-CTA stores. + cute.arch.cluster_arrive_relaxed() + else: + cute.arch.barrier() + + # Host validation guarantees an equal, tail-free K partition. + k_tile_count = cute.size(mA, mode=[1]) // self.cta_k // self.split_k + k_tile_start = split_rank * k_tile_count + + if cutlass.const_expr(self.split_k > 1): + cute.arch.cluster_wait() + + # Warp 3 is idle; warps 4-7 run the epilogue. + if warp_idx == 0: + self.dma_warp( + bar_full, + bar_empty, + bar_tma_epilog, + cute.local_tile(mA, (self.cta_m, self.cta_k), (bidx, None, l_idx)), + sA, + cute_ext.get_cta_v_map_ab(mA, mnk_tiler, tiled_mma, "A"), + k_tile_start, + k_tile_count, + True, + ) + elif warp_idx == 1: + self.dma_warp( + bar_full, + bar_empty, + bar_tma_epilog, + cute.local_tile(mB, (self.cta_n, self.cta_k), (n_idx, None, l_idx)), + sB, + cute_ext.get_cta_v_map_ab(mB, mnk_tiler, tiled_mma, "B"), + k_tile_start, + k_tile_count, + False, + ) + elif warp_idx == 2: + self.mma_warp( + bar_full, + bar_empty, + bar_mma_epilog, + bar_tmem_alloc, + tiled_mma, + sA, + sB, + tmem_base_ptr, + acc_layout, + self.cta_k // cute.size(tiled_mma.shape_mnk, mode=[2]), + k_tile_count, + ) + elif warp_idx >= 4: + self.epilog_warp( + bar_tma_epilog, + bar_mma_epilog, + bar_tmem_alloc, + tmem_base_ptr, + acc_layout, + cute.local_tile(mC, c_tiler_mn, (bidx, n_idx, l_idx)), + cute.local_tile(mBias, c_tiler_mn, (bidx, n_idx, l_idx)), + cute.arch.thread_idx()[0] - 128, + mC.element_type, + utils.LayoutEnum.from_tensor(mC), + mailbox, + bar_reduce, + split_rank, + sD, + bidx, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + @cute.experimental.jit + def dma_warp( + self, + bar_full, + bar_empty, + bar_tma_epilog, + g_tile: cute.Tensor, + s_tile: cute.Tensor, + cta_v_map: cute.Layout, + k_tile_start: cutlass.Int32, + k_tile_count: cutlass.Int32, + is_a: cutlass.Constexpr, + ): + stages = self.num_ab_stage + if cutlass.const_expr(not is_a and self.use_pdl): + cute.arch.griddepcontrol_wait() + + empty_phase = cutlass.Int32(1) + for k_tile in cutlass.range(k_tile_count, unroll=1): + stage = k_tile % stages + cute.arch.mbarrier_wait(bar_empty + stage, empty_phase) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + bar_full + stage, + cute.size_in_bytes( + s_tile.element_type, + cute.slice_(s_tile.layout, (None, None, None, 0)), + ), + ) + cute_ext.tma_load( + g_tile[None, None, k_tile_start + k_tile], + s_tile[None, None, None, stage], + (bar_full + stage).value, + cta_v_map=cta_v_map, + tma_operation_type=self.tma_op, + update_expect_tx=False, + ) + if stage == stages - 1: + empty_phase = empty_phase ^ 1 + + if cutlass.const_expr(is_a and self.use_pdl and not self.ar_push): + # ar_push: the dependent is a prepushed all-reduce, which skips its grid wait: it is released only after + # this grid's own grid wait and its pushes, in the owner's _store_tile (the prepushed >= 2 contract). + cute.arch.griddepcontrol_launch_dependents() + if cutlass.const_expr(not is_a and self.has_bias): + cute.arch.mbarrier_arrive(bar_tma_epilog) + self._drain_producer(bar_empty, empty_phase, k_tile_count) + + @cute.experimental.jit + def _drain_producer( + self, + bar_empty, + empty_phase: cutlass.Int32, + k_tile_count: cutlass.Int32, + ): + stages = self.num_ab_stage + for tail in cutlass.range(stages, unroll=1): + stage = (tail + k_tile_count) % stages + cute.arch.mbarrier_wait(bar_empty + stage, empty_phase) + if stage == stages - 1: + empty_phase = empty_phase ^ 1 + + @cute.experimental.jit + def mma_warp( + self, + bar_full, + bar_empty, + bar_mma_epilog, + bar_tmem_alloc, + tiled_mma: cute.TiledMma, + sA: cute.Tensor, + sB: cute.Tensor, + tmem_base_ptr, + acc_layout: cutlass.Constexpr, + mma_inst_tile_k: cutlass.Constexpr, + k_tile_count: cutlass.Int32, + ): + num_tmem_cols = 256 + cute.arch.alloc_tmem(num_tmem_cols, tmem_base_ptr, is_two_cta=False) + cute.arch.mbarrier_arrive(bar_tmem_alloc) + cute.arch.relinquish_tmem_alloc_permit(is_two_cta=False) + + tmem_ptr = cute.arch.retrieve_tmem_ptr(self.acc_dtype, 16, tmem_base_ptr) + accumulator = cute.make_tensor(tmem_ptr, acc_layout)[None, None, None, 0] + mma_atom = cute.make_mma_atom(tiled_mma.op) + full_phase = cutlass.Int32(0) + for k_tile in cutlass.range(k_tile_count, unroll=1): + stage = k_tile % self.num_ab_stage + cute.arch.mbarrier_wait(bar_full + stage, full_phase) + for k_block in range(mma_inst_tile_k): + if k_block == 0: + mma_atom.set(tcgen05.Field.ACCUMULATE, k_tile != 0) + else: + mma_atom.set(tcgen05.Field.ACCUMULATE, True) + cute_ext.dot( + mma_atom, + cute.append_ones(sA[None, None, k_block, stage], up_to_rank=3), + cute.append_ones(sB[None, None, k_block, stage], up_to_rank=3), + accumulator, + ) + with cute.arch.elect_one(): + tcgen05.commit(bar_empty + stage, None, self.cta_group) + if stage == self.num_ab_stage - 1: + full_phase = full_phase ^ 1 + + with cute.arch.elect_one(): + tcgen05.commit(bar_mma_epilog, None, self.cta_group) + cute.arch.mbarrier_arrive(bar_tmem_alloc) + cute.arch.mbarrier_wait(bar_tmem_alloc, 1) + cute.arch.dealloc_tmem(tmem_ptr, num_tmem_cols, is_two_cta=False) + + @cute.experimental.jit + def epilog_warp( + self, + bar_tma_epilog, + bar_mma_epilog, + bar_tmem_alloc, + tmem_base_ptr, + acc_layout: cutlass.Constexpr, + gD_tile: cute.Tensor, + gBias_tile: cute.Tensor, + epi_tid: cutlass.Int32, + c_dtype: cutlass.Constexpr, + d_layout: cutlass.Constexpr, + mailbox, + bar_reduce, + split_rank: cutlass.Int32, + sD: cute.Tensor, + bidx: cutlass.Int32, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_world: cutlass.Int32, + ar_maxt: cutlass.Int32, + ar_tokens: cutlass.Int32, + ar_n: cutlass.Int32, + ): + # Wait until MMA publishes the TMEM base pointer. + cute.arch.mbarrier_arrive(bar_tmem_alloc) + cute.arch.mbarrier_wait(bar_tmem_alloc, 0) + + acc_view = cute.make_tensor( + cute.arch.retrieve_tmem_ptr(self.acc_dtype, 16, tmem_base_ptr), + acc_layout, + )[((None, None), 0, 0, 0)] + + epi_tile = (self.cta_m, self.cta_n) + tiled_copy_t2r = cute.nvgpu.tcgen05.make_tmem_copy( + sm100_utils.get_tmem_load_op( + (self.cta_m, self.cta_n, self.cta_k), + d_layout, + c_dtype, + self.acc_dtype, + epi_tile, + False, + ), + acc_view, + ) + gD_epi = cute.flat_divide(gD_tile, epi_tile) + + # Match each epilogue thread's TMEM partition in RMEM. + rmem_layout = cute_ext.make_t2r_rmem_layout(tiled_copy_t2r, gD_epi, epi_tid) + rAcc = cute_ext.allocate( + self.acc_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + rD = cute_ext.allocate( + c_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + thr_t2r = tiled_copy_t2r.get_slice(epi_tid) + + if cutlass.const_expr(self.has_bias): + bias_dtype = gBias_tile.element_type + rBias = cute_ext.allocate( + bias_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + rBiasAcc = cute_ext.allocate( + self.acc_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + if split_rank == OWNER_RANK: + cute.arch.mbarrier_wait(bar_tma_epilog, 0) + cute_ext.partition_and_copy( + cute.make_tiled_copy_D( + cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), bias_dtype), + tiled_copy_t2r, + ).get_slice(epi_tid), + cute.flat_divide(gBias_tile, epi_tile)[None, None, 0, 0], + rBias, + ) + rBiasAcc.store(rBias.load().to(self.acc_dtype)) + + cute.arch.mbarrier_wait(bar_mma_epilog, 0) + cute_ext.partition_and_copy(thr_t2r, acc_view, rAcc) + # Make tcgen05.ld visible before TMEM release and RMEM use. + cute.arch.fence_view_async_tmem_load() + cute.arch.mbarrier_arrive(bar_tmem_alloc) + + # Peers publish FP32 partials; only rank 0 reduces and stores. + if cutlass.const_expr(self.split_k > 1): + assert cute.size(rmem_layout) == self.mailbox_elements // ( + (self.split_k - 1) * self.epilog_threads + ) + values_per_thread = cutlass.const_expr(cute.size(rmem_layout)) + values_per_peer = cutlass.const_expr(self.epilog_threads * values_per_thread) + if split_rank != OWNER_RANK: + for value_idx in cutlass.range_constexpr(0, values_per_thread, 4): + _store_shared_remote_v4( + rAcc[value_idx], + rAcc[value_idx + 1], + rAcc[value_idx + 2], + rAcc[value_idx + 3], + mailbox.iterator + + (split_rank - Int32(1)) * values_per_peer + + epi_tid * values_per_thread + + value_idx, + bar_reduce, + Int32(OWNER_RANK), + ) + else: + if epi_tid == 0: + cute.arch.mbarrier_arrive_and_expect_tx( + bar_reduce, self.expected_transaction_bytes + ) + cute.arch.mbarrier_wait(bar_reduce, 0) + for peer in cutlass.range_constexpr(self.split_k - 1): + for value_idx in cutlass.range_constexpr(values_per_thread): + rAcc[value_idx] = ( + rAcc[value_idx] + + mailbox[ + peer * values_per_peer + epi_tid * values_per_thread + value_idx + ] + ) + + if split_rank == OWNER_RANK: + if cutlass.const_expr(self.has_bias): + rAcc.store(rAcc.load() + rBiasAcc.load()) + self._store_tile(thr_t2r, rAcc, rD, gD_epi, c_dtype) + if cutlass.const_expr(self.ar_push): + self._push_tile( + thr_t2r, + rD, + sD, + epi_tid, + bidx, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + # The reduction mbarrier covers remote stores; no cluster barrier needed. + + @cute.experimental.jit + def _store_tile(self, thr_t2r, rAcc, rD, gD_epi, c_dtype: cutlass.Constexpr): + """The owner rank's reduced FP32 tile: one cast to ``c_dtype`` and the store to global memory.""" + rD.store(rAcc.load().to(c_dtype)) + # Preserve TMEM coordinates; the copy predicates output tails. + cute_ext.partition_and_copy(thr_t2r, rD, gD_epi[None, None, 0, 0]) + + @cute.experimental.jit + def _push_tile( + self, + thr_t2r, + rD, + sD, + epi_tid: cutlass.Int32, + bidx: cutlass.Int32, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_world: cutlass.Int32, + ar_maxt: cutlass.Int32, + ar_tokens: cutlass.Int32, + ar_n: cutlass.Int32, + ): + """ar_push: the owner's tile rows into the boundary all-reduce's exchange, as its own push of the output + would store them (kernels/m3_mc_allreduce: word (t, column pair) at the call parity of the (t, column / (N / 8)) + cluster CTA, -0.0 halves as +0.0), then the release of the dependents (the prepushed all-reduce).""" + cute_ext.partition_and_copy(thr_t2r, rD, sD) + cute.arch.barrier(barrier_id=1, number_of_threads=self.epilog_threads) + if cutlass.const_expr(self.use_pdl): + cute.arch.griddepcontrol_wait() + pairs = self.cta_m // 2 + col0 = bidx * self.cta_m + half = cutlass.Int64(ar_maxt) * cutlass.Int64(ar_world) * cutlass.Int64(ar_n // 2) + for w0 in cutlass.range_constexpr(0, pairs * self.cta_n, self.epilog_threads): + word = epi_tid + w0 + t = word // pairs + p = word % pairs + if t < ar_tokens: + n = col0 + 2 * p + par = _ld_volatile_s32( + ar_flags + cutlass.Int64(4) * cutlass.Int64(t * 8 + n // (ar_n // 8)) + ) & Int32(1) + off = ( + cutlass.Int64(par) * half + + cutlass.Int64(t * ar_world + ar_rank) * cutlass.Int64(ar_n // 2) + + cutlass.Int64(n // 2) + ) + _st_global_b32( + ar_ws + off * cutlass.Int64(4), + _bf16x2_word( + sD[2 * p, t].to(cutlass.Float32), sD[2 * p + 1, t].to(cutlass.Float32) + ), + ) + if cutlass.const_expr(self.use_pdl): + cute.arch.griddepcontrol_launch_dependents() + + +import torch as _torch # noqa: E402 (as in FlashInfer) + +_SUPPORTED_TORCH_DTYPES = (_torch.bfloat16, _torch.float16) + + +@cute.experimental.jit +def _bmm_no_bias( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + stream: _cuda.CUstream, +): + c = cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])) + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + c, + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[0, 1, 2])), + stream, + ) + + +@cute.experimental.jit +def _bmm_push( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + stream: _cuda.CUstream, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: Int32, + ar_world: Int32, + ar_maxt: Int32, + ar_tokens: Int32, + ar_n: Int32, +): + c = cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])) + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + c, + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[0, 1, 2])), + stream, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + +@cute.experimental.jit +def _bmm_bias( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + bias: cute.Tensor, + stream: _cuda.CUstream, +): + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])), + cute.make_tensor(bias.iterator, cute.select(bias.layout, mode=[1, 2, 0])), + stream, + ) + + +def _from_dlpack_dynamic(tensor, leading_dim: int, assumed_align: int = 32): + return from_dlpack(tensor, assumed_align=assumed_align).mark_layout_dynamic( + leading_dim=leading_dim + ) + + +def _detect_leading_dim(tensor: _torch.Tensor) -> int: + # Ignore synthetic batch stride, including 1x1. + for dim, stride in enumerate(tensor.stride()[1:], start=1): + if stride == 1: + return dim + raise ValueError("tensor has no stride-1 dimension") + + +def _make_layout_tensor( + shape: tuple[int, ...], dtype: _torch.dtype, leading_dim: int +) -> _torch.Tensor: + permutation = [dim for dim in range(len(shape)) if dim != leading_dim] + [leading_dim] + return _torch.empty( + tuple(shape[dim] for dim in permutation), dtype=dtype, device="cuda" + ).permute([permutation.index(dim) for dim in range(len(shape))]) + + +def _make_compile_repr_tensors( + dtype: _torch.dtype, + has_bias: bool, + a_leading: int, + b_leading: int, + c_leading: int, +): + m, n, k, batch = 64, 8, _CTA_K, 1 + tensors = tuple( + _from_dlpack_dynamic(_make_layout_tensor(shape, dtype, leading_dim), leading_dim) + for shape, leading_dim in ( + ((batch, n, k), a_leading), + ((batch, k, m), b_leading), + ((batch, n, m), c_leading), + ) + ) + if not has_bias: + return (*tensors, None) + return ( + *tensors, + _from_dlpack_dynamic( + _torch.empty((n,), dtype=dtype, device="cuda").as_strided( + size=(batch, n, m), stride=(0, 1, 0) + ), + 1, + 2, + ), + ) + + +def _to_cute_swap(a, b, out, bias): + a_swap = b.unsqueeze(0).transpose(-2, -1) + b_swap = a.unsqueeze(0).transpose(-2, -1) + c_swap = out.unsqueeze(0).transpose(-2, -1) + leading_dims = tuple(_detect_leading_dim(tensor) for tensor in (a_swap, b_swap, c_swap)) + cute_tensors = tuple( + _from_dlpack_dynamic(tensor, leading_dim) + for tensor, leading_dim in zip((a_swap, b_swap, c_swap), leading_dims, strict=True) + ) + if bias is None: + return (*cute_tensors, None, leading_dims) + return ( + *cute_tensors, + _from_dlpack_dynamic( + bias.as_strided(size=(1, c_swap.shape[1], c_swap.shape[2]), stride=(0, 1, 0)), + 1, + 2, + ), + leading_dims, + ) + + +@cute_aot.cached_compile("m3_lowm_splitk._get_compiled_splitk_kernel", lambda: _aot_sources()) +def _get_compiled_splitk_kernel( + dtype, + tactic: SplitKTactic, + use_pdl: bool, + has_bias: bool, + leading_dims: tuple[int, int, int], +): + if dtype not in _SUPPORTED_TORCH_DTYPES: + raise ValueError(f"split-K dense GEMM supports {_SUPPORTED_TORCH_DTYPES}; got {dtype}") + + kernel = SplitKDenseGemmKernel( + tactic=tactic, + use_pdl=use_pdl, + has_bias=has_bias, + ) + compile_tensors = _make_compile_repr_tensors(dtype, has_bias, *leading_dims) + stream = _cuda.CUstream(_torch.cuda.current_stream().cuda_stream) + if has_bias: + compiled = cute_ext.compile(_bmm_bias, kernel, *compile_tensors, stream) + else: + compiled = cute_ext.compile(_bmm_no_bias, kernel, *compile_tensors[:3], stream) + return compiled + + +def _validate_runtime_tensors(a, b, bias, out) -> tuple[int, int, int]: + tensors = (a, b, out) + ((bias,) if bias is not None else ()) + if any(not isinstance(tensor, _torch.Tensor) for tensor in tensors): + raise ValueError("a, b, out, and bias must be torch tensors") + if a.ndim != 2 or b.ndim != 2 or out.ndim != 2: + raise ValueError("split-K dense GEMM accepts only 2D tensors") + if a.device.type != "cuda" or any(tensor.device != a.device for tensor in tensors): + raise ValueError("all tensors must be on the same CUDA device") + if a.dtype not in _SUPPORTED_TORCH_DTYPES or any(tensor.dtype != a.dtype for tensor in tensors): + raise ValueError("a, b, out, and bias must share BF16 or FP16 dtype") + if any(not (tensor.is_contiguous() or tensor.t().is_contiguous()) for tensor in (a, b, out)): + raise ValueError("a, b, and out must be dense row-major or column-major matrices") + if any(tensor.data_ptr() % 32 for tensor in (a, b, out)): + raise ValueError("a, b, and out must be 32-byte aligned") + + m, k = a.shape + if b.shape[0] != k: + raise ValueError(f"incompatible shapes: a is {tuple(a.shape)}, b is {tuple(b.shape)}") + n = b.shape[1] + if out.shape != (m, n): + raise ValueError(f"out must have shape {(m, n)}, got {tuple(out.shape)}") + if bias is not None and (bias.ndim != 1 or bias.shape[0] != n or not bias.is_contiguous()): + raise ValueError( + f"bias must be contiguous with shape {(n,)}, " + f"got shape {tuple(bias.shape)} and stride {bias.stride()}" + ) + + return m, n, k + + +def run_splitk_dense( + a, + b, + bias, + out, + pdl: bool, + tactic: SplitKTactic, +): + """Run ``A[M,K] @ B[K,N]`` with the ``mm_bf16`` layouts.""" + validate_tactic(tactic, *_validate_runtime_tensors(a, b, bias, out)) + has_bias = bias is not None + cute_tensors = _to_cute_swap(a, b, out, bias) + compiled = _get_compiled_splitk_kernel( + dtype=a.dtype, + tactic=tactic, + use_pdl=pdl, + has_bias=has_bias, + leading_dims=cute_tensors[4], + ) + stream = _cuda.CUstream(_torch.cuda.current_stream(a.device).cuda_stream) + if has_bias: + compiled(*cute_tensors[:4], stream) + else: + compiled(*cute_tensors[:3], stream) + return out + + +@cute_aot.cached_compile("m3_lowm_splitk._get_compiled_push", lambda: _aot_sources()) +def _get_compiled_push( + dtype, tactic: SplitKTactic, use_pdl: bool, leading_dims: tuple[int, int, int] +): + kernel = SplitKDenseGemmKernel(tactic=tactic, use_pdl=use_pdl, has_bias=False, ar_push=True) + compile_tensors = _make_compile_repr_tensors(dtype, False, *leading_dims) + stream = _cuda.CUstream(_torch.cuda.current_stream().cuda_stream) + z64, z32 = cutlass.Int64(0), Int32(0) + return cute_ext.compile( + _bmm_push, + kernel, + *compile_tensors[:3], + stream, + z64, + z64, + z32, + Int32(1), + Int32(8), + z32, + Int32(6144), + ) + + +def compile_push(a, b, out, pdl: bool, tactic) -> None: + """Compile (only) the push variant of ``run_splitk_dense_push`` for these operands' shapes and layouts.""" + cute_tensors = _to_cute_swap(a, b, out, None) + _get_compiled_push(a.dtype, SplitKTactic(*tactic), pdl, cute_tensors[4]) + + +def run_splitk_dense_push(a, b, out, pdl: bool, tactic: SplitKTactic, ar: tuple): + """``run_splitk_dense`` without bias that also pushes ``out``'s rows into a drafter boundary all-reduce's exchange; + ``ar`` = (multicast words address, flags address, rank, world, bucket tokens) of the 8-token bucket.""" + validate_tactic(tactic, *_validate_runtime_tensors(a, b, None, out)) + cute_tensors = _to_cute_swap(a, b, out, None) + compiled = _get_compiled_push(a.dtype, tactic, pdl, cute_tensors[4]) + stream = _cuda.CUstream(_torch.cuda.current_stream(a.device).cuda_stream) + ws, flags, rank, world, maxt = ar + compiled( + *cute_tensors[:3], + stream, + cutlass.Int64(ws), + cutlass.Int64(flags), + Int32(rank), + Int32(world), + Int32(maxt), + Int32(out.shape[0]), + Int32(out.shape[1]), + ) + return out + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_mc_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_mc_allreduce.py new file mode 100644 index 000000000000..142bcb9a9133 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_mc_allreduce.py @@ -0,0 +1,422 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Low-latency TP all-reduce + residual + RMSNorm for decode (M <= 8 tokens) over a multicast mapping. + + partial = bf16(x + x2) (optional second input, e.g. shared + routed experts) + updated = bf16(residual + sum_r partial_r) (ranks summed in FP32, rank order) + normed = bf16(updated * rsqrt(mean(updated^2) + eps) * w) + +One cluster of CLUSTER CTAs per token; CTA c owns columns [c * H / CLUSTER, ...), one 16-byte chunk (8 BF16) +per thread. Each CTA pushes its slice of this rank's partial through the multicast mapping (one store lands in +every rank's buffer), releases dependents, polls its own buffer until all ranks' slices arrived, sums them, adds +the residual and reduces the RMS statistic across the cluster through DSMEM. + +Exchange protocol as in the K3 sandwich: int32 words, 0x80000000 = empty (pushed -0.0 halves become +0.0), each +CTA keeps a call counter whose parity selects the buffer half, readers re-arm the words they read. All ranks of +the TP group must issue the same calls in the same stream order. + +The workspace holds two exchange sets. With prepushed 3 two producers pushed this rank's x (set A) and x2 (set B), +e.g. the routed experts' finalize and the shared expert's down projection, and the kernel forms bf16(x + x2) per +rank from the two slots, exactly as the unpushed path does. + +Producers that push (the o_proj / shared-expert GEMV ``ar`` tuple, the routed-expert engine's ``arp`` descriptor) are +pointed at a bucket's exchange by its ``mc`` address, ``flags_for(8)`` counters, ``rank``, ``world_size`` and +``max_tokens``; ``get_workspace`` and ``_bucket`` select it. + +The kernel is ``torch.ops.trtllm.minimax_m3_sm107_tp4_mc_allreduce_norm``, reached through the ``ops/comm`` entries +``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm``; this module owns the workspaces, the bucket and +epilogue-option choice and the MXFP8 publication. +""" + +from dataclasses import dataclass +from typing import Any + +import torch + +from ..ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm as _allreduce_norm_op +from ..ops.comm.m3_mc_finalize_allreduce_norm import ( + m3_mc_finalize_allreduce_norm as _finalize_allreduce_norm_op, +) + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +BUCKETS = (8, 16, 48) # token capacity of each workspace (grid = bucket * cluster CTAs) +EMPTY_WORD = -2147483648 + +# Attention / MoE boundary epilogue options (each bit-identical): 1 residual + norm weight loaded at entry (before +# the poll); 2 cluster RMS exchange by st.async + mbarrier complete_tx instead of cluster.sync; 4 call parity rotated +# right after the poll (no trailing __syncthreads); 8 max-smem carveout so the next kernel's CTAs co-reside early. +# The 8-token bucket, by token count 1..8 (GB300 TP4 tuning). +AR_OPTS_BY_M = (7, 7, 15, 15, 15, 15, 7, 7) +# The 16- and 48-token buckets. +AR_OPTS_BY_BUCKET = {16: 15, 48: 15} +# The MoE-finalize all-reduce, per bucket. +AR_FIN_OPTS_BY_BUCKET = {8: 7, 16: 15, 48: 15} + + +# Option 32: the 8-CTA cluster kernel in the stock TP4 oneshot reduction order instead of the row kernel (not finalize). +_OPT_ONESHOT_CLUSTER = 32 + +# Option 64: the row kernel's max-smem-carveout instance (as option 8 is for the cluster kernel). +_OPT_ROW_CARVEOUT = 64 + +# Option 128: the drafter boundary (option 16) on the 8-CTA cluster kernel instead of the world-4 row kernel (routing +# only; mcAllreduceNormTp4.cu invoke). Exact-class to the row kernel's FlashInfer order: the RMS statistic is the +# cluster's sum (modeling.DRAFT_MC_AR_CLUSTER). +_OPT_DRAFTER_CLUSTER = 128 + +# Option 2 is also the safe ordering of the cluster RMS exchange: its stores into peer CTAs' shared memory follow a +# cluster barrier arrive / wait pair. Without it they run before any cluster barrier, when a peer need not have +# started, so every call keeps it. +_OPT_CLUSTER_BARRIER = 2 + + +def _ar_opts(m: int, bucket: int, finalize: bool = False, oneshot_cluster: bool = False) -> int: + """Epilogue options for an m-row call on the given bucket; every option is bit-identical. A bucket missing from + the tables gets 0, and every result keeps _OPT_CLUSTER_BARRIER. ``oneshot_cluster`` adds _OPT_ONESHOT_CLUSTER.""" + if oneshot_cluster and not finalize: + return _ar_opts(m, bucket) | _OPT_ONESHOT_CLUSTER + if finalize: + opts = AR_FIN_OPTS_BY_BUCKET.get(bucket, 0) + elif bucket == 8: + opts = AR_OPTS_BY_M[m - 1] + else: + opts = AR_OPTS_BY_BUCKET.get(bucket, 0) + return opts | _OPT_CLUSTER_BARRIER + + +@dataclass(eq=False) +class McAllReduceWorkspace: + """Two halves of [max_tokens][world][H] BF16 (int32 words) per rank behind one multicast mapping, plus one call + counter per CTA per cluster size. Create collectively and eagerly on every TP rank.""" + + uc: torch.Tensor + mc: torch.Tensor + flags: dict + rank: int + world_size: int + hidden: int + handle: Any + comm: Any + max_tokens: int = 8 + + @classmethod + def create(cls, mapping, hidden: int, max_tokens: int = 8) -> "McAllReduceWorkspace": + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + words = 4 * max_tokens * mapping.tp_size * (hidden // 2) # two exchange sets of two halves + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "McAllReduceWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 4, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("McAllReduceWorkspace.create failed on another rank") + return cls( + uc=uc, + mc=mc, + flags={}, + rank=mapping.tp_rank, + world_size=mapping.tp_size, + hidden=hidden, + handle=handle, + comm=comm, + max_tokens=max_tokens, + ) + + def flags_for(self, cluster: int): + """This cluster size's per-CTA call counters, or None when they do not exist yet during a capture. + get_workspace creates the cluster-8 counters with the workspace, so for cluster 8 (the only size built) + that None never happens and cannot differ between ranks.""" + flags = self.flags.get(cluster) + if flags is None: + if torch.cuda.is_current_stream_capturing(): + return None + flags = torch.zeros(self.max_tokens * cluster, dtype=torch.int32, device=self.uc.device) + self.flags[cluster] = flags + return flags + + +class McBuckets: + """One workspace per token bucket (8 / 16 / 48): calls of different sizes never share exchange slots.""" + + def __init__(self, workspaces): + self.workspaces = workspaces + self.hidden = workspaces[0].hidden + + def for_tokens(self, m: int): + for ws in self.workspaces: + if m <= ws.max_tokens: + return ws + return None + + +def _bucket(workspace, m): + return workspace.for_tokens(m) if isinstance(workspace, McBuckets) else workspace + + +def decline_reason(x, residual, workspace, cluster=8, x2=None) -> str: + """Why ``m3_mc_allreduce_norm`` returned None for these operands (for the dispatch's miss message).""" + if workspace is None: + return "no multicast workspace: it is created collectively on the first eager decode step" + bucket = _bucket(workspace, x.shape[0]) + if bucket is None: + return f"no workspace bucket holds {x.shape[0]} tokens (buckets {BUCKETS})" + if bucket.flags.get(cluster) is None: + return f"no {cluster}-CTA call counters for the {bucket.max_tokens}-token bucket" + return ( + f"operands outside the kernel: x {tuple(x.shape)} {x.dtype} contiguous={x.is_contiguous()}, " + f"residual contiguous={residual.is_contiguous()}, hidden {bucket.hidden}" + + ("" if x2 is None else f", x2 {tuple(x2.shape)} contiguous={x2.is_contiguous()}") + ) + + +def m3_mc_allreduce_norm( + x, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + x2=None, + quant=False, + prepushed=0, + oneshot_cluster=False, + drafter_norm=False, + carveout=False, + capture=None, + drafter_cluster=False, +): + """Return (normed, updated_residual) or None when unsupported. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM. ``capture`` = (out [>= M, H] view, fc_norm weight + [H], eps): the boundary also writes the Eagle3 capture of the updated residual, bit for bit + m3_spec_tail.capture_norm (CAPTURE_NORM_IN_BOUNDARY; not with ``drafter_norm``). ``drafter_cluster``: with + ``drafter_norm``, the 8-CTA cluster kernel's drafter form (option 128; its carveout is option 8) instead of the row + kernel's. + + A None makes the caller take another collective, so it must be the same on every TP rank. It is: it depends only + on the shapes, dtypes and layouts of the operands (the same code builds them on every rank) and on whether the + workspace's flags exist, which ``get_workspace`` creates collectively, before any capture, for cluster 8.""" + workspace = _bucket(workspace, x.shape[0]) + if workspace is None: + return None + if ( + x.dim() != 2 + or x.shape[0] > MAX_TOKENS + or x.dtype != torch.bfloat16 + or x.shape[1] != workspace.hidden + or not x.is_contiguous() + or not residual.is_contiguous() + or (x2 is not None and (not x2.is_contiguous() or x2.shape != x.shape)) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + # The Eagle3 drafter boundary (opts bit 16, the row kernel's FlashInfer fused_add_rmsnorm order): 8-token bucket, + # x only, no MXFP8 copy, never the cluster kernel. + if drafter_norm and ( + workspace.max_tokens != 8 or x2 is not None or quant or prepushed not in (0, 2) + ): + return None + if drafter_norm and drafter_cluster: + opts = ( + (16 | _ar_opts(x.shape[0], workspace.max_tokens)) & ~8 + | (8 if carveout else 0) + | _OPT_DRAFTER_CLUSTER + ) + elif drafter_norm: + opts = ( + 16 | _ar_opts(x.shape[0], workspace.max_tokens) | (_OPT_ROW_CARVEOUT if carveout else 0) + ) + else: + opts = _ar_opts(x.shape[0], workspace.max_tokens, oneshot_cluster=oneshot_cluster) + if carveout: + # the max-smem carveout on whichever kernel runs: option 8 on the cluster kernel, 64 on the row kernel + opts |= 8 if oneshot_cluster else _OPT_ROW_CARVEOUT + outs = _allreduce_norm_op( + x, + x2, + residual, + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and x.shape[1] % 128 == 0), + int(prepushed), + opts, + *_capture_args(capture, drafter_norm), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +def _capture_args(capture, drafter_norm: bool) -> tuple: + """(cap_out, cap_w, cap_eps) for the op, from ``capture`` = (out, fc_norm weight, eps) or None.""" + if capture is None: + return (None, None, 0.0) + assert not drafter_norm, "the capture epilogue is not built for the drafter boundary" + out, weight, eps = capture + return (out, weight, float(eps)) + + +def m3_mc_finalize_allreduce_norm( + fc2, + perm, + scales, + shared, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + quant=False, +): + """MoE finalize (sum_k scale * fc2[perm] + shared) + all-reduce + residual + RMSNorm in one kernel. + Returns (normed, updated_residual) or None when unsupported. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM. (It runs the shared GB300 op, which has no capture + epilogue: a capture layer finalized here keeps the capture_norm kernel.)""" + m = residual.shape[0] + workspace = _bucket(workspace, m) + if workspace is None: + return None + if ( + residual.dim() != 2 + or m > MAX_TOKENS + or residual.shape[1] != workspace.hidden + or fc2.dtype != torch.bfloat16 + or not fc2.is_contiguous() + or fc2.dim() != 2 + or perm.dtype != torch.int32 + or perm.dim() != 2 + or perm.shape[0] != m + or not perm.is_contiguous() + or scales.dtype not in (torch.float32, torch.bfloat16) + or not scales.is_contiguous() + or (shared is not None and (shared.shape != residual.shape or not shared.is_contiguous())) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + outs = _finalize_allreduce_norm_op( + fc2, + perm, + scales, + shared, + residual.contiguous(), + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and residual.shape[1] % 128 == 0), + _ar_opts(m, workspace.max_tokens, finalize=True), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +_WORKSPACE: dict = {} +_AGREE_COMMS: dict = {} + + +def _tp_comm(mapping): + """A communicator over mapping's TP group, created once (collective over that group only).""" + key = tuple(mapping.tp_group) + comm = _AGREE_COMMS.get(key) + if comm is None: + from tensorrt_llm._utils import mpi_comm, mpi_disabled + + if mpi_disabled(): + comm = mapping.tp_group_pg + else: + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + _AGREE_COMMS[key] = comm + return comm + + +def tp_agree(mapping, ok: bool) -> bool: + """Whether ``ok`` holds on every rank of mapping's TP group (one blocking host collective). + + For a fast path whose availability is decided from rank-local state (a kernel build that can fail on one rank): + every rank must take the same path, so that they issue the same collectives with the same exchange protocol and + the same kernels. Call it once per decision, at build time, at the same point on every rank, outside CUDA-graph + capture; never per step. With one rank, or attention DP (whose ranks run different batches), it returns ``ok``. + """ + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return ok + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("tp_agree must run outside CUDA-graph capture") + from tensorrt_llm._torch.distributed.ops import _mnnvl_workspace_all_succeeded + + return _mnnvl_workspace_all_succeeded(_tp_comm(mapping), bool(ok)) + + +def tp_agree_built(mapping, built, what: str): + """``built`` when it is not None on every TP rank, else None on every rank (see ``tp_agree``).""" + if tp_agree(mapping, built is not None) or built is None: + return built + from tensorrt_llm.logger import logger + + logger.warning(f"{what}: built here but not on another TP rank; every rank falls back") + return None + + +def get_workspace(mapping, hidden: int): + """Process-wide workspace (created collectively on first eager use); None while capturing before creation. + + The first eager call for a hidden size (the warmup forward, before any CUDA-graph capture) creates one + McAllReduceWorkspace per bucket in BUCKETS on every TP rank (MPI collective over the TP group) plus each + bucket's 8-CTA-cluster call counters; later calls, captured or not, return the same McBuckets. The workspaces + live until process exit. + """ + ws = _WORKSPACE.get(hidden) + if ws is None: + if torch.cuda.is_current_stream_capturing(): + return None + wss = [McAllReduceWorkspace.create(mapping, hidden, b) for b in BUCKETS] + for w in wss: + w.flags_for(8) + ws = McBuckets(wss) + _WORKSPACE[hidden] = ws + return ws diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/__init__.py new file mode 100644 index 000000000000..f818a0e6c9f1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/__init__.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 MoE layer of the sm_107 TP4 target for low-batch decode as a CuTe DSL kernel: ``m3_moe`` for 1..8 tokens +(op.py) at the TP4 routed and shared intermediate of 768 per rank.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/m3_moe16_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/m3_moe16_kernel.py new file mode 100644 index 000000000000..ff609e78ef29 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/m3_moe16_kernel.py @@ -0,0 +1,3871 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 routed + shared experts for 9..16 decode tokens on one TP rank: ``m3_moe16`` (v2). + +Same math as the 1..8-token engine (m3_moe_kernel.py), operation for operation for the routed part (bitwise equal to +TRTLLM-Gen's fused MoE on the fp32 router-GEMV logits); a different schedule for 9..16 rows: + +- Front: the hidden states land in smem by bulk copies in K chunks (the idle ring); all 16 warps run the router + GEMV (m3_router_gemv's per-thread order) and the NVFP4 quantization of B1 from smem; the logits cross CTAs as + tagged words; top-k one warp per token. +- Ring: FC1 / FC2 weight stages by TMA; their scale words go by 4-byte cp.async straight into the instruction atom + layout (SFA_CP; no raw scale box, no splicer warp), completing on the stage's full barrier. +- FC1 -> FC2 hand-off (B2IMG): the FC1 epilogue writes each unit's FP4 values and scale byte into a per-group global + image laid out as the FC2 B tile + its scale atoms; one counter per group (release add per finished unit, every + counter gains exactly 192 per call); warp 5 loads a completed image with two bulk copies. +- Shared expert on NSH more CTAs (MXFP8 tcgen05), as the 1..8 engine. +Configuration via M3_CONFIG (m_max, num_ctas, stages, nb2_cap, lag, shared, sfa_cp, b2buf, ...). +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +_CFG = globals().get("M3_CONFIG") or {} + + +def _cfg(key: str, default): + return type(default)(_CFG.get(key, default)) + + +H = 6144 # capacity: fixed: the checkpoint's hidden size +I_TP = 768 +TWO_I = 2 * I_TP +E_LOCAL = 128 +TOP_K = 4 # capacity: fixed: the checkpoint's top-k +ROUTE_SCALE = 2.0 +SUM_EPS = 1e-20 +M_MAX = _cfg("m_max", 12) +assert 9 <= M_MAX <= 16, M_MAX +NCTA = _cfg("num_ctas", 128) +FUSED_ROUTER = True +NROUTE = E_LOCAL +NGEMV = (NROUTE + NCTA - 1) // NCTA +assert NGEMV == 1, "m3_moe16: one router expert per CTA (num_ctas >= 128)" +TIMING = bool(_cfg("timing", 0)) +HANG_DBG = bool(_cfg("hang_dbg", 0)) +LAG = _cfg("lag", 3) +G_CAP = TOP_K * M_MAX +NG_CAP = M_MAX +THREADS = 512 # capacity: fixed: the CTA size +NWARPS = THREADS // 32 +GEMV_THREADS = 256 +SHARED = bool(_cfg("shared", 0)) +LATE_TRIGGER = bool(_cfg("late_trigger", 1)) +SFA_CP = bool( + _cfg("sfa_cp", 1) +) # A scales by 4-byte cp.async into the atom layout (else raw box + splicer warp) +B2IMG = True +PERM = bool(_cfg("perm", 1)) +PERM_MIN_NG = max( + 3, _cfg("perm_min_ng", 5) +) # (measured: helps at random routing G >= 23, hurts at spec G 13-14) +# a partial last slot group becomes group 1 (the plan must match: op16) +DBG_DELAY = bool( + _cfg("dbg_delay", 0) +) # (stress builds) pseudo-random producer / epilogue / release stalls +DBG_NOSTORE = bool(_cfg("dbg_nostore", 0)) # (timing experiment only: wrong results) +EPI_FENCE = bool( + _cfg("epi_fence", 0) +) # (rel_warp 0) writer lanes' fence.acq_rel.gpu before the release add +EPI_PFENCE = bool( + _cfg("epi_pfence", 1) +) # producer-side fence.proxy.async.global before the release add +REL_WARP = bool( + _cfg("rel_warp", 1) +) # warp 8 does the per-group release adds (off the epilogue's path) +NSH = 24 if SHARED else 0 +SH_I = 768 +SH_KB = H // 32 # capacity: fixed: a kernel tile constant +SH_DKB = SH_I // 32 # capacity: fixed: a kernel tile constant +SH_SPLIT = 2 +SH_GT = SH_KB // 4 // SH_SPLIT # capacity: fixed: a kernel tile constant +SH_STAGES_GU = SH_GT +SH_STAGES = SH_GT + 2 * (SH_I // 128) # capacity: fixed: pipeline stages +SHX_BAR = 6 +SH_PRETILE = False +SH_STAGE_BYTES = 16384 + 512 +MMA_M = 128 +KT_BYTES = 128 +INC_K = 2 +NUM_KB = 4 +# FC1 +N1 = _cfg("n1", 48 if _cfg("m_max", 12) <= 12 else 64) # B rows QS q + t +N1_LD = 1 << (N1 - 1).bit_length() # tcgen05.ld .num (a power of two) covering N1 columns +QS = N1 // 4 +assert QS >= M_MAX +R1 = 32 +C1 = 4 +U1 = TWO_I // R1 +K1_TILES = H // 256 +S1 = K1_TILES // C1 # capacity: fixed: pipeline stages +NUM_BYTES_A = MMA_M * KT_BYTES # capacity: fixed: a stage's operand bytes +NUM_BYTES_BOX1 = R1 * KT_BYTES +SF1_KA = H // 64 +SF1_RB = TWO_I // 128 +NUM_BYTES_SFA_RAW = 16 * 512 +NUM_WORDS_SFA_RAW = NUM_BYTES_SFA_RAW // 4 +NUM_BYTES_SFA = NUM_KB * 512 +NUM_WORDS_SFA = NUM_BYTES_SFA // 4 +NUM_BYTES_B1 = N1 * KT_BYTES +NUM_SFB1_ATOMS = S1 * NUM_KB +# FC2 +R2 = 32 +NB2 = H // R2 +KT2 = I_TP // 256 +SF2_KA = I_TP // 64 +SF2_RB = H // 128 +N2 = _cfg("n2", (4 * M_MAX + 15) // 16 * 16) # B rows j M + t +NUM_BYTES_B2 = N2 * KT_BYTES +IMG_B2 = KT2 * NUM_BYTES_B2 # one group's B2 tile image (global and smem) +IMG_SF = KT2 * NUM_KB * 512 # its scale atoms +IMG_BYTES = IMG_B2 + IMG_SF +CNT_PER_CALL = 4 * (TWO_I // R1) # every group counter gains 192 per call +B1IMG_OFF = None # set below (after N1): the activation image after the NG_CAP group images +TK_WARPS = [0, 1, 2, 3, 6, 8, 9, 10, 11, 12, 13, 14, 5, 7, 15, 4][ + : _cfg("m_max", 12) +] # top-k warp of token t (4 last: early0) +B1_WARP = 15 if _cfg("m_max", 12) <= 14 else -1 # (-1: warp 0 after the routing) +NB2_CAP = _cfg("nb2_cap", 2) +B1IMG_OFF = M_MAX * IMG_BYTES +B1IMG_BYTES = (S1 * N1 * 128 + M_MAX * (H // 16) + 15) // 16 * 16 +IMG_TOTAL = B1IMG_OFF + B1IMG_BYTES +UNITS_ROW = TWO_I // R1 +PARTIAL_EMPTY = -1 # capacity: fixed: a sentinel, not a size +EPI_THREADS = 128 +EPI_BAR = 1 +ROUTE_BAR = 2 +GEMV_BAR = 3 +TOPK_BAR = 4 +ALL_BAR = 5 +POLL_NS = _cfg("poll_ns", 150) +B2_POLL_NS = _cfg("b2_poll_ns", 16) +LG_REPL = _cfg("lg_repl", 64) +LG_STRIDE = M_MAX * E_LOCAL + 32 +W_WARP = 4 # capacity: fixed: a warp role index +X_WARP = 5 # B2 image loader; slots + plan # capacity: fixed: a warp role index +S_WARP = 6 # (SFA_CP = 0) scale splicer # capacity: fixed: a warp role index +MMA_WARP = 7 # capacity: fixed: a warp role index +EARLY0 = bool(_cfg("early0", 1)) +EVICT_FIRST = 0x12F0000000000000 +EVICT_LAST = 0x14F0000000000000 +_LOG2E = 1.4426950408889634 +NB2BUF = _cfg("b2buf", 2) +SPB = 1 +# x staging: K chunks of XCK elements, NXB smem buffers (rotating if fewer than the chunks) +XCK = _cfg("xck", 2048) +NXC = H // XCK +XCB = M_MAX * XCK * 2 # bytes of one chunk (all tokens) +STAGE_A = NUM_BYTES_A +STAGE_SF = NUM_BYTES_SFA if SFA_CP else NUM_BYTES_SFA_RAW + NUM_BYTES_SFA +_FIXED_SMEM = ( + S1 * NUM_BYTES_B1 + + NB2BUF * IMG_B2 + + IMG_SF + + NB2_CAP * M_MAX * TOP_K * R2 * 2 + + SPB * M_MAX * C1 * R1 * 4 + + (G_CAP + 1) * (NG_CAP + 1) * 4 + + 6 * 1024 +) +_STAGES_FIT = (227 * 1024 - _FIXED_SMEM - (128 if SHARED else 0)) // (STAGE_A + STAGE_SF) +ACC1_COL = 0 +ACC2_COL = 2 * N1 +ACC2_STRIDE = 1 << (N2 - 1).bit_length() +SFB1_COL = ACC2_COL + 2 * ACC2_STRIDE +SFB1_WSTEP = 1 +SFB2_COL = ( + SFB1_COL + NUM_SFB1_ATOMS * 4 +) # one TMEM slot (loaded after the previous group's FC2 MMAs) +SFA1_COL = SFB2_COL + KT2 * NUM_KB * 4 +STAGES = min(_cfg("stages", 8), _STAGES_FIT, (512 - SFA1_COL) // 16) +TMEM_COLS = 512 +assert STAGES >= 3, (STAGES, _STAGES_FIT, SFA1_COL) +# smem: [ring A | ring SF | B2 bufs | B2 SF staging | B1]; x chunks alias [0, _OFF_B1) during the front +_OFF_SFA = STAGES * NUM_BYTES_A +_OFF_B2 = (_OFF_SFA + STAGES * STAGE_SF + 1023) // 1024 * 1024 +_OFF_SFB2 = _OFF_B2 + NB2BUF * IMG_B2 +_OFF_B1 = (_OFF_SFB2 + IMG_SF + 1023) // 1024 * 1024 +BIG = _OFF_B1 + S1 * NUM_BYTES_B1 +NXB = min(NXC, _OFF_B1 // XCB) +assert NXB >= 2, (NXB, _OFF_B1, XCB) +# shared CTA layout in BIG: one pass over all M tokens; B rows SHN q + t (gate_up, 2 K slices), t (down) +SHN = 16 # capacity: fixed: a kernel tile constant +SH_NPASS = 1 +SH_HINT = EVICT_FIRST +SH_PF_POST = bool( + _cfg("sh_pf_post", 1) +) # L2 prefetch of the shared weights after the grid dependency (else before) +_SB_A = 0 +_SH_FIX = ( + SH_GT * 2 * SHN * 128 + + 6 * SHN * 128 + + M_MAX * SH_KB + + M_MAX * SH_DKB + + 64 * SHN * 4 + + 2 * 32 * SHN * 4 + + 2048 +) +SH_NST = min(8, (BIG - _SH_FIX) // (NUM_BYTES_A + 4 * 512 + 512)) if SHARED else 1 +_SB_RAW = SH_NST * NUM_BYTES_A +_SB_SPL = _SB_RAW + SH_NST * 2048 +_SB_BX = (_SB_SPL + SH_NST * 512 + 1023) // 1024 * 1024 +_SB_BH = _SB_BX + SH_GT * 2 * SHN * 128 +_SB_XC = _SB_BH + 6 * SHN * 128 +_SB_HC = _SB_XC + M_MAX * SH_KB +_SB_XCH = (_SB_HC + M_MAX * SH_DKB + 15) // 16 * 16 +_SB_GU = _SB_XCH + 64 * SHN * 4 +_SB_END = _SB_GU + 2 * 32 * SHN * 4 +SH_ACC_GU = 0 +SH_ACC_DN = 2 * SHN +SH_SFA = SH_ACC_DN + 2 * SHN +SH_SFB_GU = SH_SFA + 4 * 8 +SH_SFB_DN = SH_SFB_GU + 4 * SH_GT # capacity: fixed: a kernel tile constant +SH_TMEM = 256 # capacity: fixed: TMEM columns +if SHARED: + assert SH_STAGES - SH_NST <= 64, SH_STAGES + assert SH_NST >= 3 and _SB_END <= BIG and SH_SFB_DN + 24 <= SH_TMEM, (SH_NST, _SB_END, BIG) +TD_STRIDE = 64 +PFD = _cfg("pfd", 0) # L2 prefetch distance in ring stages (0: off) +PF_WARP = 12 # capacity: fixed: a warp role index +TICKET = bool(_cfg("ticket", 0)) # CTA roles by entry order +# priority tokens: token 0 and PT1 (M / 2: the second request's first row at spec c2) get their logits published first +NPT = _cfg("npt", 0) +PT1 = M_MAX // 2 +PF_PRIO = bool( + _cfg("pf_prio", 0) +) # L2 prefetch of the priority tokens' experts (this CTA's 1/NCTA of each) +PF_W1 = TWO_I * (H // 2) // NCTA +PF_S1 = TWO_I * (H // 16) // NCTA +PF_W2 = H * (I_TP // 2) // NCTA +PF_S2 = H * (I_TP // 16) // NCTA +assert PF_W1 % 16 == 0 and PF_S1 % 16 == 0 and PF_W2 % 16 == 0 and PF_S2 % 16 == 0 + + +# ============================================================================= +# Inline-PTX helpers. +# ============================================================================= +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +# ---- all-reduce push (the MoE boundary's multicast exchange, set A): +# arp = int64 [enable, mc, flags, rank, world, maxt] +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32_if(addr, value, pred, *, loc=None, ip=None): + """st.global.b32 when pred != 0.""" + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip), + cutlass.Int32(pred).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, $2, 0;\n\t@p st.global.b32 [$0], $1;\n\t}", "l,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _ar_push(arp, v, low, t, cl): + """Push bf16 channel pair (cl, cl + 1) of token t (v = channel cl's bf16 value on this lane, its neighbour's on + lane ^ 8) into this rank's set-A slot of the exchange; the slot half is the all-reduce CTA (t, cl // 768)'s call + parity. Only lanes with ``low`` store; every lane of the warp must call it (shuffle).""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + world = arp.load(idx=4) + parity = cutlass.Int64( + _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cs // cutlass.Int32(H // 8)) + ) + & cutlass.Int32(1) + ) + off = ( + parity * arp.load(idx=5) * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + arp.load(idx=3)) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(arp.load(idx=1) + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +@dsl_user_op +def _bar_arrive(bar_id: int, threads: int, *, loc=None, ip=None): + """bar.arrive: signal a named barrier without waiting (its memory ordering is bar.sync's).""" + _llvm.inline_asm( + None, + [], + f"bar.arrive {bar_id}, {threads};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _st_relaxed_u32(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "st.relaxed.gpu.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _st_relaxed_u64(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int64(val)], + "st.relaxed.gpu.global.u64 [$0], $1;", + "l,l", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm( + _T.i64(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u64 $0, [$1];", + "=l,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _nanosleep(ns: int, *, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + f"nanosleep.u32 {ns};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _gtime(*, loc=None, ip=None): + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), + [], + "mov.u64 $0, %globaltimer;", + "=l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _smid(*, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [], + "mov.u32 $0, %smid;", + "=r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@cute.jit +def _stamp(tdbg: cutlass.Array, bx, i: cutlass.Constexpr): + if cutlass.const_expr(TIMING): + tdbg.store(_gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(i)) + + +@dsl_user_op +def _rcp(a, *, loc=None, ip=None): + """rcp.approx.ftz.f32 (the reciprocal TRT-LLM's FP4 quantization uses).""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _div_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _sub_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "sub.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _mul_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3_bits(x, *, loc=None, ip=None): + """e4m3 (round to nearest even, saturating) of x as an int 0..255.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, 0f00000000, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3_to_f32(b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(b)], + "{\n.reg .b16 t, l, u;\n.reg .b32 h;\ncvt.u16.u32 t, $1;\ncvt.rn.f16x2.e4m3x2 h, t;\nmov.b32 {l, u}, h;\n" + "cvt.f32.f16 $0, l;\n}", + "=f,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e2m1x2(hi, lo, *, loc=None, ip=None): + """One byte of two E2M1 values (lo in the low nibble), round to nearest even, saturating.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b8 t;\ncvt.rn.satfinite.e2m1x2.f32 t, $1, $2;\ncvt.u32.u8 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2(x, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _bf16_round(x, *, loc=None, ip=None): + """x rounded to bf16 (nearest even), as f32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\nmov.b32 $0, {0, t};\n}", + "=f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _redux_max_u32(v, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(v)], + "redux.sync.max.u32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16 (nearest even) bits of x in the low 16 bits.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ue8m0_rp(x, *, loc=None, ip=None): + """UE8M0 code of x rounded toward +inf, saturating (__nv_cvt_float_to_e8m0(x, SATFINITE, RoundPosInf)).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rp.satfinite.ue8m0x2.f32 t, $1, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3x2(hi, lo, *, loc=None, ip=None): + """Two e4m3 (nearest even, saturating) in 16 bits, lo in the low byte.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, $1, $2;\ncvt.u32.u16 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _min_xorsign_abs(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "min.xorsign.abs.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _div_full(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.full.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2_full(x, *, loc=None, ip=None): + """ex2.approx.f32 (no flush to zero), Triton's tl.exp.""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _l2_prefetch(addr, nbytes, *, loc=None, ip=None): + """cp.async.bulk.prefetch.L2 of nbytes (a multiple of 16) at global address addr (16-byte aligned).""" + _llvm.inline_asm( + None, + [ + cutlass.Int64(addr).ir_value(loc=loc, ip=ip), + cutlass.Int32(nbytes).ir_value(loc=loc, ip=ip), + ], + "cp.async.bulk.prefetch.L2.global [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +def _mx_osc(amax): + """mxfp8 quantization of a block with absolute max amax: (UE8M0 code, the element multiplier).""" + code = _ue8m0_rp(_mul_rn(amax, _rcp(cutlass.Float32(448.0)))) + cf = cutlass.Int32( + cutlass.select_( + code == cutlass.Int32(255), + cutlass.Int32(0x7FFFFFFF), + cutlass.select_( + code == cutlass.Int32(0), cutlass.Int32(0x00400000), code << cutlass.Int32(23) + ), + ) + ).bitcast(cutlass.Float32) + return code, cutlass.select_(amax != cutlass.Float32(0.0), _rcp(cf), cutlass.Float32(0.0)) + + +def _mx_words(v, osc): + """e4m3 words of len(v) (a multiple of 4) values times osc, in element order.""" + words = [] + for t in range(len(v) // 4): + lo = _e4m3x2(_mul_rn(v[4 * t + 1], osc), _mul_rn(v[4 * t], osc)) + hi = _e4m3x2(_mul_rn(v[4 * t + 3], osc), _mul_rn(v[4 * t + 2], osc)) + words.append(lo | (hi << cutlass.Int32(16))) + return words + + +def _twiddle(score): + """Float bits mapped so that unsigned order is float order (cub TwiddleIn).""" + bits = ( + score.bitcast(cutlass.Int32) + if hasattr(score, "bitcast") + else cutlass.Float32(score).bitcast(cutlass.Int32) + ) + return cutlass.select_(bits < cutlass.Int32(0), ~bits, bits | cutlass.Int32(-2147483648)) + + +@cute.jit +def _wait(bar: cutlass.Array, parity): + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +def _tptr(base_row, col, dtype=cutlass.Int32): + return cutlass.inttoptr((base_row << 16) | col, 6, dtype) + + +@dsl_user_op +def _match_any(value, *, loc=None, ip=None): + """The mask of the warp's lanes holding the same 32-bit value.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(value)], + "match.any.sync.b32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@cute.jit +def _waitd(bar: cutlass.Array, parity, tdbg: cutlass.Array, site: cutlass.Constexpr): + """_wait, or (HANG_DBG) give up after ~2^22 polls and record the site in tdbg[bx, 24 + warp].""" + if cutlass.const_expr(HANG_DBG): + n = cutlass.Int32(0) + while (not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity)) & ( + n < cutlass.Int32(1 << 22) + ): + n = n + cutlass.Int32(1) + if n >= cutlass.Int32(1 << 22): + bx, _, _ = cute.arch.block_idx() + w = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + tdbg.store( + cutlass.Int64(site), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + w + ) + else: + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +@cute.jit +def _adv(stage, phase): + """The next ring position's (stage, phase).""" + stage = stage + cutlass.Int32(1) + wrap = stage == cutlass.Int32(STAGES) + return cutlass.select_(wrap, cutlass.Int32(0), stage), cutlass.select_( + wrap, phase ^ cutlass.Int32(1), phase + ) + + +@cute.jit +def _seg(i0, i1, k): + """Segment k of the FC1 item range [i0, i1) (items = unit-major stages): (unit, first stage, end stage). Order: + the last unit (its head, the part a split unit's next CTA does not cover), then the first unit, then the rest.""" + uf = i0 // cutlass.Int32(S1) + ul = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) + unit = uf + k - cutlass.Int32(1) + unit = cutlass.select_(k == cutlass.Int32(0), ul, unit) + unit = cutlass.select_(ul == uf, uf, unit) + j0 = i0 - unit * cutlass.Int32(S1) + j0 = cutlass.select_(j0 < cutlass.Int32(0), cutlass.Int32(0), j0) + j1 = i1 - unit * cutlass.Int32(S1) + j1 = cutlass.select_(j1 > cutlass.Int32(S1), cutlass.Int32(S1), j1) + return unit, j0, j1 + + +@cute.jit +def _nsegs(i0, i1): + n = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) - i0 // cutlass.Int32(S1) + cutlass.Int32(1) + return cutlass.select_(i1 > i0, n, cutlass.Int32(0)) + + +@cute.jit +def _sfb1_to_tmem(warp, lane, base_row, base_col, src: cutlass.Array, b1_ready: cutlass.Array): + """B1 scales to TMEM: instruction (j, kb)'s column holds, at lane n = QS q + t, token t's 4 scales of k-tile + 4 j + q, K 64 kb .. (src: int32 words [M, 96] of the token scales); each warp writes its sub-partition.""" + # B rows n >= 32 (N1 = 64) go to the atom's next 32-bit column (row n: lane n % 32, column n // 32) + for w in cutlass.range_constexpr((N1 + 31) // 32): + n = lane + cutlass.Int32(32 * w) + q = n // cutlass.Int32(QS) + tl = n - q * cutlass.Int32(QS) + sf_ok = (n < cutlass.Int32(N1)) & (tl < cutlass.Int32(M_MAX)) + sfv = [] + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + j = a // NUM_KB + kb = a % NUM_KB + wsf = src.load( + idx=cutlass.select_( + sf_ok, + tl * cutlass.Int32(H // 64) + + (cutlass.Int32(4 * j) + q) * cutlass.Int32(4) + + cutlass.Int32(kb), + cutlass.Int32(0), + ) + ) + sfv.append(cutlass.select_(sf_ok, cutlass.Int32(wsf), cutlass.Int32(0))) + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + prims.tcgen05_st( + "32x32b", + _tptr( + base_row + warp * cutlass.Int32(32), + base_col + SFB1_COL + a * 4 + w * SFB1_WSTEP, + ), + sfv[a], + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(b1_ready) + + +@cute.jit +def _b1_addr(t, kb16): + """Word index in the resident B tiles of the 8 FP4 bytes of token t's 16-value block kb16.""" + kt = kb16 >> cutlass.Int32(4) + j = kt >> cutlass.Int32(2) + n = (kt & cutlass.Int32(3)) * cutlass.Int32(QS) + t + cc = (kb16 >> cutlass.Int32(1)) & cutlass.Int32(7) + return ( + j * cutlass.Int32(NUM_BYTES_B1) + + n * cutlass.Int32(KT_BYTES) + + (cc ^ (n & cutlass.Int32(7))) * cutlass.Int32(16) + + (kb16 & cutlass.Int32(1)) * cutlass.Int32(8) + ) // cutlass.Int32(4) + + +@cute.jit +def _swiglu_quant(xv, gate_raw, is_up, a, c, alpha, beta, lim): + """TRTLLM-Gen's FC1 epilogue (the swiGlu Bmm E2m1 output), operation for operation: lane value xv is the up row, + gate_raw its gate row (fp32 sums). Returns (q, the scale's e4m3 bits) of the 16-row NVFP4 block (lanes r, r ^ 16 + and r ^ 8 cover it; amax over the warp's up lanes of a half). a = fc31_alpha, c = fc31_scale_c.""" + neg_max = cutlass.Float32(-3.40282346638528859812e38) + gc = cutlass.select_(gate_raw > lim, lim, gate_raw) + gc = cutlass.select_(gate_raw < neg_max, neg_max, gc) + uc = cutlass.select_(xv > lim, lim, xv) + uc = cutlass.select_(xv < -lim, -lim, uc) + c1 = _mul_rn(a, _mul_rn(alpha, cutlass.Float32(1.4426950216293334961))) + sgm = _rcp(_add_rn(_ex2(-_mul_rn(gc, c1)), cutlass.Float32(1.0))) + gs = _mul_rn(gc, sgm) + h = _mul_rn(_fma_rn(uc, a, _mul_rn(a, beta)), gs) + h = cutlass.select_(is_up, h, cutlass.Float32(0.0)) + amax = cute.arch.fmax(_mul_rn(cute.math.abs(h), c), cutlass.Float32(0.0)) + for off in (16, 8, 4, 2, 1): + amax = cute.arch.fmax(amax, cute.arch.shuffle_sync_bfly(amax, off)) + sfb = _e4m3_bits(_mul_rn(amax, cutlass.Float32(0.16666667163372039795))) + sff = _e4m3_to_f32(sfb) + osc = cutlass.select_(sff != cutlass.Float32(0.0), _rcp(sff), cutlass.Float32(0.0)) + return _mul_rn(_mul_rn(h, osc), c), sfb + + +@cute.jit +def _plan_range(s_plan: cutlass.Array, p): + w = s_plan.load(idx=p) + return w & cutlass.Int32(0xFFFF), w >> cutlass.Int32(16) + + +@cute.jit +def _sh_issue( + lane, + sid, + g, + s_big: cutlass.Array, + sh_full: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, + gpos, +): + """Stage g of the shared CTA's stream into ring slot g % SH_NST (warp 0). g < SH_GT: gate_up k-tile g of both + slices: quadrant Q = 2 q + u (q slice, u 0 gate / 1 up) = rows 32 sid .. (+ 768 for up) at k-tile SH_GT q + g, its + raw scale atom at raw slot Q. Else down tile j = (g - SH_GT) / 6 (rows 256 sid + 128 j ..), k-tile + (g - SH_GT) % 6.""" + st = gpos % cutlass.Int32(SH_NST) # ring position (stage g of this pass) + bar = sh_full.subview(st) + if cutlass.const_expr(SH_PRETILE): + if lane == 0: + src = (sid * cutlass.Int32(SH_STAGES) + g) * cutlass.Int32(SH_STAGE_BYTES) + prims.mbarrier_arrive_expect_tx(bar, SH_STAGE_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), gusf.subview(src), bar, 16384 + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + gusf.subview(src + cutlass.Int32(16384)), + bar, + 512, + ) + cute.arch.sync_warp() + else: + if g < cutlass.Int32(SH_GT): + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 4 * 512) + cute.arch.sync_warp() + if lane < 4: + q = lane >> cutlass.Int32(1) + u = lane & cutlass.Int32(1) + row0 = sid * cutlass.Int32(32) + u * cutlass.Int32(SH_I) + kt = q * cutlass.Int32(SH_GT) + g + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A) + lane * cutlass.Int32(4096)), + tma_gu, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=SH_HINT, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview( + cutlass.Int32(_SB_RAW) + + st * cutlass.Int32(2048) + + lane * cutlass.Int32(512) + ), + gusf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_KB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(6) + kt = gd - jt * cutlass.Int32(6) + row0 = sid * cutlass.Int32(256) + jt * cutlass.Int32(128) + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 512) + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), + tma_dn, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=SH_HINT, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_RAW) + st * cutlass.Int32(2048)), + dnsf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_DKB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + cute.arch.sync_warp() + + +@cute.jit +def _shared_cta( + tidx, + warp, + lane, + sid, + s_big: cutlass.Array, + sh_bar: cutlass.Array, + epochs: cutlass.Array, + tdbg: cutlass.Array, + bx, + xb32: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, + tmem_ptr_i32: cutlass.Array, + gu8: cutlass.Array, + dn8: cutlass.Array, +): + """Shared CTA sid (see SHARED): tcgen05 MXF8 (E4M3 x E4M3, UE8M0 per 32 K), m3_mx_gemv's tile order. gate_up: one + 128-row MMA tile = 2 K slices x (32 gate + 32 up rows), B rows 8 q + t = token t's slice-q k-tile; down: 2 tiles + of 128 rows, full K. Warps: 0 weights (TMA), 1 TMEM + scale splice + MMA, 4-7 epilogue; 2.. activation quant.""" + big32 = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Int32, alignment=16) + bigf = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Float32, alignment=16) + big8 = cutlass.Array(s_big.data_ptr(0), shape=(BIG,), dtype=cutlass.Int8, alignment=16) + sh_full = sh_bar # [0, SH_NST): stage full + sh_empty = cutlass.Array(sh_bar.data_ptr(8), shape=(8,), dtype=cutlass.Int64, alignment=8) + b_gu_rdy = sh_bar.subview(16) + b_dn_rdy = sh_bar.subview(17) + acc_gu = sh_bar.subview(18) + acc_dn = sh_bar.subview(19) + if warp == 0: + prims.prefetch_tensormap(tma_gu) + prims.prefetch_tensormap(tma_dn) + if lane < SH_NST: + prims.mbarrier_init(sh_full.subview(lane), 1) + prims.mbarrier_init( + sh_empty.subview(lane), 33 + ) # the MMAs' commit + each splicer lane's reads done + if lane == 0: + prims.mbarrier_init(b_gu_rdy, 1) + prims.mbarrier_init(b_dn_rdy, 1) + prims.mbarrier_init(acc_gu, 1) + prims.mbarrier_init(acc_dn, 1) + if warp == 1: + prims.tcgen05_alloc(tmem_ptr_i32, SH_TMEM) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + if warp == 0: + for g in cutlass.range_constexpr(SH_NST): # layer constants: before the grid dependency + _sh_issue( + lane, + sid, + cutlass.Int32(g), + s_big, + sh_full, + tma_gu, + tma_dn, + gusf, + dnsf, + cutlass.Int32(g), + ) + # the rest of this CTA's weights into L2 (64 gate / up rows of 6 KB, 256 down rows of 768 B) + if cutlass.const_expr(SH_PRETILE): + # this CTA's stages beyond the ring, 4 per lane + for r in cutlass.range_constexpr(2): + pg = cutlass.Int32(SH_NST) + lane * cutlass.Int32(2) + cutlass.Int32(r) + if pg < cutlass.Int32(SH_STAGES): + _l2_prefetch( + gusf.subview( + (sid * cutlass.Int32(SH_STAGES) + pg) * cutlass.Int32(SH_STAGE_BYTES) + ) + .data_ptr() + .toint(), + SH_STAGE_BYTES, + ) + elif cutlass.const_expr(not SH_PF_POST): + for r in cutlass.range_constexpr(2): + prw = sid * cutlass.Int32(32) + cutlass.Int32(r * SH_I) + lane + _l2_prefetch(gu8.subview(prw * cutlass.Int32(H)).data_ptr().toint(), H) + if lane == 0: + _l2_prefetch( + dn8.subview(sid * cutlass.Int32(256 * SH_I)).data_ptr().toint(), 256 * SH_I + ) + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(SH_PF_POST): + if warp == 0: # after the grid dependency: DRAM is idle during the routed front + for r in cutlass.range_constexpr(2): + prw = sid * cutlass.Int32(32) + cutlass.Int32(r * SH_I) + lane + _l2_prefetch(gu8.subview(prw * cutlass.Int32(H)).data_ptr().toint(), H) + if lane == 0: + _l2_prefetch( + dn8.subview(sid * cutlass.Int32(256 * SH_I)).data_ptr().toint(), 256 * SH_I + ) + if cutlass.const_expr(not LATE_TRIGGER): + if arp.load(idx=0) == cutlass.Int64(0): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + NQ = THREADS - 64 # activation quant threads (warps 2 ..) + for ps in cutlass.range_constexpr(SH_NPASS): + TT0 = 0 # one pass: B rows SHN q + t (gate_up), t (down) + MS = M_MAX + GB = ps * SH_STAGES # ring positions before this pass + PAR = ps & 1 # parity of the per-pass barriers (B ready, accumulators) + if warp == 0: + for g in cutlass.range(SH_NST if ps == 0 else 0, SH_STAGES, unroll=1): + gg = g + cutlass.Int32(GB) # ring position over the passes + _wait( + sh_empty.subview(gg % cutlass.Int32(SH_NST)), + ((gg // cutlass.Int32(SH_NST)) - cutlass.Int32(1)) & cutlass.Int32(1), + ) + _sh_issue(lane, sid, g, s_big, sh_full, tma_gu, tma_dn, gusf, dnsf, gg) + elif warp == 1: + # ---- MMA: gate_up (scale atoms spliced to the tile's row order), then down + idesc_gu = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=2 * SHN, + m_dim=128, + ) + idesc_dn = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=SHN, + m_dim=128, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + desc_a0 = prims.Tcgen05SmemDesc.build( + s_big.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bx = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BX), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bh = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BH), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + sq = sid & cutlass.Int32(3) + _wait(b_gu_rdy, PAR) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for g in cutlass.range(SH_STAGES, unroll=1): + gg = g + cutlass.Int32(GB) + st = gg % cutlass.Int32(SH_NST) + if g == cutlass.Int32(SH_GT): + _wait(b_dn_rdy, PAR) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + _wait(sh_full.subview(st), (gg // cutlass.Int32(SH_NST)) & cutlass.Int32(1)) + raw = cutlass.Int32(_SB_RAW // 4) + st * cutlass.Int32(512) + if (g < cutlass.Int32(SH_GT)) & cutlass.Boolean(not SH_PRETILE): + # MMA row 32 Q + i = quadrant Q's row i: word 4 i + Q <- raw atom Q's word 4 i + sid % 4 + w4 = [ + big32.load(idx=raw + cutlass.Int32(128 * qq) + lane * cutlass.Int32(4) + sq) + for qq in range(4) + ] + big32.store( + ( + cutlass.Int32(w4[0]), + cutlass.Int32(w4[1]), + cutlass.Int32(w4[2]), + cutlass.Int32(w4[3]), + ), + idx=cutlass.Int32(_SB_SPL // 4) + + st * cutlass.Int32(128) + + lane * cutlass.Int32(4), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + prims.mbarrier_arrive( + sh_empty.subview(st) + ) # this lane's generic reads of the slot are done + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + if g < cutlass.Int32(SH_GT): + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bx + cutlass.Int64(g * cutlass.Int32((2 * SHN * 128) >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr(tcol + SH_ACC_GU, 6, cutlass.Float32), + da + 2 * kb, + db + 2 * kb, + idesc_gu.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (g > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_GU + g * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_GT - 1): + prims.tcgen05_commit(acc_gu) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(6) + kt = gd - jt * cutlass.Int32(6) + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview( + cutlass.Int32(_SB_SPL if SH_PRETILE else _SB_RAW) + + st * cutlass.Int32(512 if SH_PRETILE else 2048) + ), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bh + cutlass.Int64(kt * cutlass.Int32((SHN * 128) >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr( + tcol + SH_ACC_DN + jt * cutlass.Int32(SHN), 6, cutlass.Float32 + ), + da + 2 * kb, + db + 2 * kb, + idesc_dn.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (kt > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_DN + kt * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_STAGES - 1): + prims.tcgen05_commit(acc_dn) + cute.arch.sync_warp() + if warp >= 2: + if cutlass.const_expr(ps > 0): + _wait( + acc_dn, PAR ^ 1 + ) # the previous pass's MMAs are done: its B tiles and TMEM scales are free + # ---- the activation, MXFP8 per 32-K block (native mxfp8_quantize), into B (row 8 q + t of stage tile i = + # token t's k-tile SH_GT q + i, 128-byte swizzle) and its codes + qt = tidx - cutlass.Int32(64) + for r in cutlass.range_constexpr((MS * SH_KB + NQ - 1) // NQ): + q_ = qt + cutlass.Int32(r * NQ) + if q_ < cutlass.Int32(MS * SH_KB): + t = q_ // cutlass.Int32(SH_KB) + b = q_ - t * cutlass.Int32(SH_KB) + wv = [ + xb32.load( + idx=(t + cutlass.Int32(TT0)) * cutlass.Int32(H // 2) + + b * cutlass.Int32(16) + + cutlass.Int32(4 * z), + vector_size=4, + alignment=16, + ) + for z in range(4) + ] + v = [] + for z in cutlass.range_constexpr(4): + for zz in cutlass.range_constexpr(4): + w = cutlass.Int32(wv[z][zz]) + v.append((w << cutlass.Int32(16)).bitcast(cutlass.Float32)) + v.append((w & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(v[z])) + code, osc = _mx_osc(amax) + words = _mx_words(v, osc) + kt = b >> cutlass.Int32(2) + sl = kt // cutlass.Int32(SH_GT) + it = kt - sl * cutlass.Int32(SH_GT) + n = sl * cutlass.Int32(SHN) + t + c0 = (b & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BX) + + it * cutlass.Int32(2 * SHN * 128) + + n * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + + (((c0 + cutlass.Int32(1)) ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_XC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR, number_of_threads=NQ) + if warp >= 4: + if warp < 8: + qd = warp - cutlass.Int32(4) + # B scales of gate_up to TMEM: column of stage i, lane n = 8 q + t: token t's codes of + # k-tile SH_GT q + i + n = lane + t = n & cutlass.Int32(SHN - 1) + ok = (n < cutlass.Int32(2 * SHN)) & (t < cutlass.Int32(MS)) + for i in cutlass.range_constexpr(SH_GT): + kt = (n // cutlass.Int32(SHN)) * cutlass.Int32(SH_GT) + cutlass.Int32(i) + wsf = big32.load( + idx=( + cutlass.Int32(_SB_XC) + + cutlass.select_(ok, t, cutlass.Int32(0)) * cutlass.Int32(SH_KB) + + kt * cutlass.Int32(4) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_GU + i * 4), + cutlass.select_(ok, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + et = tidx - cutlass.Int32(128) + if et == 0: + prims.mbarrier_arrive(b_gu_rdy) + _stamp(tdbg, bx, 54) + # ---- gate_up out: quadrant qd lane l = MMA row 32 qd + l; slice q = qd / 2 holds token t + # at column 8 q + t + _wait(acc_gu, PAR) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + av = prims.tcgen05_ld( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_ACC_GU, cutlass.Float32), + num=2 * SHN, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(2): + for tt in cutlass.range_constexpr(MS): + bigf.store( + cutlass.Float32(av[SHN + tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(2)) * cutlass.Int32(32) + lane) + * cutlass.Int32(SHN) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd < cutlass.Int32(2): + # m3_mx_gemv's owner sum: (0 + rank 0) + rank 1, then bf16 + for tt in cutlass.range_constexpr(MS): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(SHN) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(SHN) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + # h = silu_and_mul(gate, up) of positions 32 sid + p, published under the call's tag + for r in cutlass.range_constexpr((32 * MS + 127) // 128): + pq = et + cutlass.Int32(r * 128) + if pq < cutlass.Int32(32 * MS): + p = pq // cutlass.Int32(MS) + tt = pq - p * cutlass.Int32(MS) + ga = bigf.load(idx=cutlass.Int32(_SB_GU // 4) + p * cutlass.Int32(SHN) + tt) + up = bigf.load( + idx=cutlass.Int32(_SB_GU // 4) + + (cutlass.Int32(32) + p) * cutlass.Int32(SHN) + + tt + ) + q_a = cute.arch.fmin(cutlass.Float32(ga), cutlass.Float32(7.0)) + q_bu = _min_xorsign_abs(cutlass.Float32(up), cutlass.Float32(7.0)) + q_ub = _add_rn(q_bu, cutlass.Float32(1.0)) + q_e2 = _ex2_full( + _mul_rn( + _mul_rn(q_a, cutlass.Float32(-1.70200002193450927734)), + cutlass.Float32(1.4426950216293334961), + ) + ) + q_sg = _div_full(cutlass.Float32(1.0), _add_rn(q_e2, cutlass.Float32(1.0))) + q_h = _mul_rn(q_ub, _mul_rn(q_sg, q_a)) + _st_relaxed_u32( + hx.subview( + (tt + cutlass.Int32(TT0)) * cutlass.Int32(SH_I) + + sid * cutlass.Int32(32) + + p + ) + .data_ptr() + .toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(q_h), + ) + if et == 0: + _stamp(tdbg, bx, 50) + # every shared CTA's h: (token, K block) per thread, 32 tagged words, MXFP8 into B (h) and its codes + for rr in cutlass.range_constexpr((MS * SH_DKB + 127) // 128): + q_ = et + cutlass.Int32(rr * 128) + if q_ < cutlass.Int32(MS * SH_DKB): + t = q_ // cutlass.Int32(SH_DKB) + kb = q_ - t * cutlass.Int32(SH_DKB) + base = (t + cutlass.Int32(TT0)) * cutlass.Int32(SH_I) + kb * cutlass.Int32( + 32 + ) + hv = [cutlass.Int32(0) for _ in range(32)] + pend = cutlass.Int32(1) + while pend != cutlass.Int32(0): + for q4 in cutlass.range_constexpr(8): + v4 = prims.load_ext( + hx.subview(base + cutlass.Int32(4 * q4)), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for z in cutlass.range_constexpr(4): + hv[4 * q4 + z] = cutlass.Int32(v4[z]) + okw = cutlass.Boolean(True) + for z in cutlass.range_constexpr(32): + okw = okw & ( + ((hv[z] >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) == tag16 + ) + pend = cutlass.select_(okw, cutlass.Int32(0), cutlass.Int32(1)) + vals = [ + (hv[z] << cutlass.Int32(16)).bitcast(cutlass.Float32) for z in range(32) + ] + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(vals[z])) + code, osc = _mx_osc(amax) + words = _mx_words(vals, osc) + kt = kb >> cutlass.Int32(2) + c0 = (kb & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BH) + + kt * cutlass.Int32(SHN * 128) + + t * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ (t & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + + ( + ((c0 + cutlass.Int32(1)) ^ (t & cutlass.Int32(7))) + << cutlass.Int32(2) + ), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_HC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + okh = lane < cutlass.Int32(MS) + for kt in cutlass.range_constexpr(6): + wsf = big32.load( + idx=( + cutlass.Int32(_SB_HC) + + cutlass.select_(okh, lane, cutlass.Int32(0)) * cutlass.Int32(SH_DKB) + + cutlass.Int32(4 * kt) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_DN + kt * 4), + cutlass.select_(okh, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if et == 0: + prims.mbarrier_arrive(b_dn_rdy) + _stamp(tdbg, bx, 51) + # ---- down out: tile j row 32 qd + l -> y_sh row 256 sid + 128 j + 32 qd + l + _wait(acc_dn, PAR) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for jt in cutlass.range_constexpr(2): + dv = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_DN + jt * SHN, + cutlass.Float32, + ), + num=SHN, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + rowo = ( + sid * cutlass.Int32(256) + + cutlass.Int32(128 * jt) + + qd * cutlass.Int32(32) + + lane + ) + for tt in cutlass.range_constexpr(MS): + _st_relaxed_u32( + dnx.subview(cutlass.Int32((tt + TT0) * H) + rowo).data_ptr().toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(cutlass.Float32(dv[tt])), + ) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if et == 0: + _stamp(tdbg, bx, 52) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if warp == 1: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc(cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), SH_TMEM) + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + + +# ============================================================================= +# v2 helpers: global release / acquire, proxy fences. +# ============================================================================= +@dsl_user_op +def _red_release_add(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "red.release.gpu.global.add.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _atom_add_u64(addr, val, *, loc=None, ip=None): + return cutlass.Int64( + _asm( + _T.i64(), + [cutlass.Int64(addr), cutlass.Int64(val)], + "atom.relaxed.gpu.global.add.u64 $0, [$1], $2;", + "=l,l,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ld_acquire_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.acquire.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fence_proxy_async_global(*, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + "fence.proxy.async.global;", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _fence_acq_rel_gpu(*, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + "fence.acq_rel.gpu;", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@cute.jit +def _dbg_delay(salt): + """(DBG_DELAY stress builds) a pseudo-random ~0..4 us stall: protocol robustness under skew.""" + if cutlass.const_expr(DBG_DELAY): + r = cutlass.Int32(_gtime() >> cutlass.Int64(6)) * cutlass.Int32(1103515245) + salt + if ((r >> cutlass.Int32(12)) & cutlass.Int32(7)) == cutlass.Int32(0): + for _ in cutlass.range((r >> cutlass.Int32(16)) & cutlass.Int32(31), unroll=1): + _nanosleep(128) + + +@cute.jit +def _waitt(bar: cutlass.Array, parity, acc): + """_wait that (TIMING) adds the waited ns to acc.""" + t0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + if cutlass.const_expr(TIMING): + acc = acc + (_gtime() - t0) + return acc + + +@cute.jit +def _tstore(tdbg: cutlass.Array, bx, i: cutlass.Constexpr, v): + if cutlass.const_expr(TIMING): + tdbg.store(cutlass.Int64(v), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(i)) + + +@cute.jit +def _issue_fc1( + lane, + stage, + e, + c, + j, + sA: cutlass.Array, + sSF: cutlass.Array, + ab_full: cutlass.Array, + tma_a1, + tma_s1, + s1w: cutlass.Array, +): + """FC1 stage j of unit c of expert e into ring stage `stage` (the caller waited its empty barrier): 4 TMA boxes + (rows 32 c .., k-tiles 4 j + q) and the stage's scale atoms (SFA_CP: word [r][q] of atom kb <- raw atom + (k-tile 4 j + q, kb) word [r][c % 4], by this warp's 32 lanes).""" + row = c * cutlass.Int32(R1) + kx = j * cutlass.Int32(C1 * KT_BYTES) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview(stage * cutlass.Int32(NUM_BYTES_A) + cutlass.Int32(q * NUM_BYTES_BOX1)), + tma_a1, + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) * cutlass.Int32(SF1_KA) + + j * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + prims.cp_async_bulk_tensor_shared_cta_global( + sSF.subview(stage * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1, + (cutlass.Int32(0), sf_line), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + atom0 = (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) * cutlass.Int32( + SF1_KA + ) + j * cutlass.Int32(16) + src0 = atom0 * cutlass.Int32(128) + lane * cutlass.Int32(4) + (c & cutlass.Int32(3)) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for q in cutlass.range_constexpr(C1): + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSF.subview(dst0 + cutlass.Int32(kb * 128 + q)), + s1w.subview(src0 + cutlass.Int32((4 * q + kb) * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + # one group per ring stage (racecheck: the issuer waits it before the slot's reuse) + prims.cp_async_commit_group() + cute.arch.sync_warp() + + +@cute.jit +def _issue_fc2( + lane, + stage, + g, + blk, + t2: cutlass.Constexpr, + n_here, + s_el: cutlass.Array, + sA: cutlass.Array, + sSF: cutlass.Array, + ab_full: cutlass.Array, + tma_a2, + tma_s2, + s2w: cutlass.Array, +): + """FC2 k-tile t2 of output block blk for the group's n_here slots: box jj = rows 32 blk .. of slot jj's w2 at + k-tile t2; scale word [r][jj] of atom kb <- raw atom (rows / 128, K block 4 t2 + kb) word [r][blk % 4].""" + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + n_here + * cutlass.Int32(NUM_BYTES_BOX1 if SFA_CP else NUM_BYTES_BOX1 + NUM_BYTES_SFA_RAW // 4), + ) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + cutlass.Int32(jj * NUM_BYTES_BOX1) + ), + tma_a2, + (cutlass.Int32(t2 * KT_BYTES), blk * cutlass.Int32(R2), e), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSF.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + cutlass.Int32(jj * NUM_WORDS_SFA_RAW // 4) + ), + tma_s2, + ( + cutlass.Int32(0), + ( + (e * cutlass.Int32(SF2_RB) + (blk >> cutlass.Int32(2))) + * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(4), + ), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + src0 = ( + ( + (e * cutlass.Int32(SF2_RB) + (blk >> cutlass.Int32(2))) + * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (blk & cutlass.Int32(3)) + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSF.subview(dst0 + cutlass.Int32(kb * 128 + jj)), + s2w.subview(src0 + cutlass.Int32(kb * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + # one group per ring stage (racecheck: the issuer waits it before the slot's reuse) + prims.cp_async_commit_group() + cute.arch.sync_warp() + + +@cute.jit +def _quant_words(w, sc, rsc, rsix): + """NVFP4 of one 16-value block given as 8 bf16-pair words: (lo word, hi word, scale e4m3 bits).""" + xs = [] + for z in cutlass.range_constexpr(8): + xs.append((w[z] << cutlass.Int32(16)).bitcast(cutlass.Float32)) + xs.append((w[z] & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + vmax = cute.math.abs(xs[0]) + for z in cutlass.range_constexpr(1, 16): + vmax = cute.arch.fmax(vmax, cute.math.abs(xs[z])) + sfb = _e4m3_bits(_mul_rn(sc, _mul_rn(vmax, rsix))) + osc = cutlass.select_( + vmax != cutlass.Float32(0.0), _rcp(_mul_rn(_e4m3_to_f32(sfb), rsc)), cutlass.Float32(0.0) + ) + by = [ + _e2m1x2(_mul_rn(xs[2 * z + 1], osc), _mul_rn(xs[2 * z], osc)) & cutlass.Int32(0xFF) + for z in range(8) + ] + lo = ( + by[0] + | (by[1] << cutlass.Int32(8)) + | (by[2] << cutlass.Int32(16)) + | (by[3] << cutlass.Int32(24)) + ) + hi = ( + by[4] + | (by[5] << cutlass.Int32(8)) + | (by[6] << cutlass.Int32(16)) + | (by[7] << cutlass.Int32(24)) + ) + return lo, hi, sfb + + +@cute.jit +def _quant_block(xs32: cutlass.Array, widx, sc, rsc, rsix): + """NVFP4 of one 16-value block (8 bf16 words at widx): (lo word, hi word, scale e4m3 bits), fp4_quantize's ops.""" + w = [] + for h in cutlass.range_constexpr(2): + v4 = xs32.load(idx=widx + cutlass.Int32(4 * h), vector_size=4, alignment=16) + for z in cutlass.range_constexpr(4): + w.append(cutlass.Int32(v4[z])) + xs = [] + for z in cutlass.range_constexpr(8): + xs.append((w[z] << cutlass.Int32(16)).bitcast(cutlass.Float32)) + xs.append((w[z] & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + vmax = cute.math.abs(xs[0]) + for z in cutlass.range_constexpr(1, 16): + vmax = cute.arch.fmax(vmax, cute.math.abs(xs[z])) + sfb = _e4m3_bits(_mul_rn(sc, _mul_rn(vmax, rsix))) + osc = cutlass.select_( + vmax != cutlass.Float32(0.0), _rcp(_mul_rn(_e4m3_to_f32(sfb), rsc)), cutlass.Float32(0.0) + ) + by = [ + _e2m1x2(_mul_rn(xs[2 * z + 1], osc), _mul_rn(xs[2 * z], osc)) & cutlass.Int32(0xFF) + for z in range(8) + ] + lo = ( + by[0] + | (by[1] << cutlass.Int32(8)) + | (by[2] << cutlass.Int32(16)) + | (by[3] << cutlass.Int32(24)) + ) + hi = ( + by[4] + | (by[5] << cutlass.Int32(8)) + | (by[6] << cutlass.Int32(16)) + | (by[7] << cutlass.Int32(24)) + ) + return lo, hi, sfb + + +@cute.kernel +def m3_moe_kernel( + tma_a1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_a2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_gu_desc: cutlass.GridConstant[cuda.TensorMap], + tma_dn_desc: cutlass.GridConstant[cuda.TensorMap], + bias: cutlass.Array, # f32 [128] + escal: cutlass.Array, # f32 [6, 128] + plan: cutlass.Array, # int32 [4 M + 1, NCTA, M + 1] + partials: cutlass.Array, # int32 bits of f32 [G_CAP * 48, M, 32]: split units' head sums + img32: cutlass.Array, # int32 [NG_CAP, IMG_BYTES / 4]: per-group FC2 B image (tile + scale atoms) + cnt: cutlass.Array, # int32 [NG_CAP * 8]: per-group unit counters (index 8 g) + s1w: cutlass.Array, # int32 words of w3_w1_weight_scale + s2w: cutlass.Array, # int32 words of w2_weight_scale + w1b: cutlass.Array, # int8 bytes of w3_w1_weight (L2 prefetch) + w2b: cutlass.Array, # int8 bytes of w2_weight (L2 prefetch) + epochs: cutlass.Array, # int32 [NCTA + NSH] + out: cutlass.Array, # bf16 [M, 6144] + route_dbg: cutlass.Array, # int32 [M, 8] + tdbg: cutlass.Array, # int64 [NCTA + NSH, 64] + xb32: cutlass.Array, # int32 words (bf16 pairs) of the hidden states [M, 6144] + rw: cutlass.Array, # f32 router weight [128, 6144] + sfsc: cutlass.Array, # f32 [1] fc31_input_scale + lgbuf: cutlass.Array, # int64 [LG_REPL, LG_STRIDE] + guw: cutlass.Array, + gusf: cutlass.Array, + dnw: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, +): + tidx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tidx % 32 + if cutlass.const_expr(TICKET): + # role by entry order: the CTAs that become resident last (behind the predecessor's CTAs) take the shared + # expert roles (most slack), the first NCTA the routed ones (the router GEMV / logits exchange) + s_tk = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + if tidx == 0: + old = _atom_add_u64( + cnt.subview(cutlass.Int32(NG_CAP * 8 + 8)).data_ptr().toint(), cutlass.Int64(1) + ) + s_tk.store(cutlass.Int32(old % cutlass.Int64(NCTA + NSH)), idx=0) + prims.barrier_cta_sync(0) + bx = s_tk.load(idx=0) + if tidx == 0: + _stamp(tdbg, bx, 0) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(_smid()), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(63) + ) + + s_big = cutlass.Array(cutlass.Int8, BIG, space=cutlass.AddressSpace.smem, alignment=1024) + sA = cutlass.Array( + s_big.data_ptr(0), shape=(NUM_BYTES_A * STAGES,), dtype=cutlass.Int8, alignment=1024 + ) + # ring scale words: SFA_CP: the instruction atoms [STAGES][4 x 512 B]; else raw boxes [STAGES][8 KB] + spliced + sSF = cutlass.Array( + s_big.data_ptr(_OFF_SFA), + shape=(STAGES * STAGE_SF // 4,), + dtype=cutlass.Int32, + alignment=1024, + ) + sSFA = cutlass.Array( + s_big.data_ptr(_OFF_SFA + (0 if SFA_CP else STAGES * NUM_BYTES_SFA_RAW)), + shape=(NUM_WORDS_SFA * STAGES,), + dtype=cutlass.Int32, + alignment=512, + ) + sB1 = cutlass.Array( + s_big.data_ptr(_OFF_B1), shape=(NUM_BYTES_B1 * S1,), dtype=cutlass.Int8, alignment=1024 + ) + sB2 = cutlass.Array( + s_big.data_ptr(_OFF_B2), shape=(NB2BUF * IMG_B2,), dtype=cutlass.Int8, alignment=1024 + ) + sSFB2 = cutlass.Array( + s_big.data_ptr(_OFF_SFB2), shape=(IMG_SF // 4,), dtype=cutlass.Int32, alignment=1024 + ) + ys = cutlass.Array( + cutlass.BFloat16, + NB2_CAP * M_MAX * TOP_K * R2, + space=cutlass.AddressSpace.smem, + alignment=16, + ) # bf16 values + s_part = cutlass.Array( + cutlass.Float32, SPB * M_MAX * C1 * R1, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_bias = cutlass.Array(cutlass.Float32, E_LOCAL, space=cutlass.AddressSpace.smem, alignment=16) + s_red = cutlass.Array( + cutlass.Float32, 9 * M_MAX + 16, space=cutlass.AddressSpace.smem, alignment=16 + ) + # the activation's scale codes [M, 384] until they are in TMEM: aliases ys (first written by the FC2 epilogue) + s_xsf = cutlass.Array( + ys.data_ptr(0), shape=(M_MAX * (H // 64),), dtype=cutlass.Int32, alignment=16 + ) + ab_full = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + ab_empty = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + sfa_ready = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + acc_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_ready = cutlass.Array(cutlass.Int64, NB2BUF, space=cutlass.AddressSpace.smem, alignment=8) + b2_free = cutlass.Array(cutlass.Int64, NB2BUF, space=cutlass.AddressSpace.smem, alignment=8) + img_bar = cutlass.Array(cutlass.Int64, NB2BUF, space=cutlass.AddressSpace.smem, alignment=8) + sfb2_free = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + b1_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + x_full = cutlass.Array(cutlass.Int64, NXB, space=cutlass.AddressSpace.smem, alignment=8) + x_empty = cutlass.Array(cutlass.Int64, NXB, space=cutlass.AddressSpace.smem, alignment=8) + route_bar = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + ys_done = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + grp_done = cutlass.Array(cutlass.Int64, 4, space=cutlass.AddressSpace.smem, alignment=8) + b1_load = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + topk_bar = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + tok0_bar = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + px_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + s_prog = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + s_el = cutlass.Array(cutlass.Int32, G_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + s_kpos = cutlass.Array( + cutlass.Int32, G_CAP * M_MAX, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_w = cutlass.Array( + cutlass.Float32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_ids = cutlass.Array( + cutlass.Int32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_plan = cutlass.Array(cutlass.Int32, NG_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + s_gn = cutlass.Array( + cutlass.Int32, NG_CAP, space=cutlass.AddressSpace.smem, alignment=16 + ) # slots per group + s_emap = cutlass.Array(cutlass.Int32, E_LOCAL, space=cutlass.AddressSpace.smem, alignment=16) + s_misc = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_plan_all = cutlass.Array( + cutlass.Int32, (G_CAP + 1) * (NG_CAP + 1), space=cutlass.AddressSpace.smem, alignment=16 + ) + s_esc = cutlass.Array(cutlass.Float32, 8 * G_CAP, space=cutlass.AddressSpace.smem, alignment=32) + + is_routed = cutlass.Boolean(True) + if cutlass.const_expr(SHARED): + sh_full = cutlass.Array(cutlass.Int64, 32, space=cutlass.AddressSpace.smem, alignment=8) + is_routed = bx < cutlass.Int32(NCTA) + if bx >= cutlass.Int32(NCTA): + _shared_cta( + tidx, + warp, + lane, + bx - cutlass.Int32(NCTA), + s_big, + sh_full, + epochs, + tdbg, + bx, + xb32, + tma_gu_desc.get_ptr(), + tma_dn_desc.get_ptr(), + gusf, + dnsf, + hx, + dnx, + arp, + tmem_ptr_i32, + guw, + dnw, + ) + if is_routed: + # ---- before the grid dependency + if warp == 0: + if tidx < STAGES: + prims.mbarrier_init(ab_full.subview(tidx), 33 if SFA_CP else 1) + prims.mbarrier_init(ab_empty.subview(tidx), 1) + prims.mbarrier_init(sfa_ready.subview(tidx), 1) + if tidx < 2: + prims.mbarrier_init(acc_full.subview(tidx), 1) + prims.mbarrier_init(acc_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(acc2_full.subview(tidx), 1) + prims.mbarrier_init(acc2_empty.subview(tidx), EPI_THREADS) + if tidx < NB2BUF: + prims.mbarrier_init(b2_ready.subview(tidx), 1) + prims.mbarrier_init(b2_free.subview(tidx), 1) + prims.mbarrier_init(img_bar.subview(tidx), 1) + if tidx < NXB: + prims.mbarrier_init(x_full.subview(tidx), 1) + prims.mbarrier_init(x_empty.subview(tidx), NWARPS) + if tidx == 0: + prims.mbarrier_init(b1_ready.subview(0), EPI_THREADS) + prims.mbarrier_init(sfb2_free.subview(0), 1) + prims.mbarrier_init(route_bar.subview(0), 32) + prims.mbarrier_init(ys_done.subview(0), EPI_THREADS) + prims.mbarrier_init(b1_load.subview(0), 1) + prims.mbarrier_init(topk_bar.subview(0), M_MAX * TOP_K) + prims.mbarrier_init(tok0_bar.subview(0), TOP_K) + prims.mbarrier_init(px_full.subview(0), 1) + if tidx < 4: + prims.mbarrier_init(grp_done.subview(tidx), EPI_THREADS) + if tidx == 0: + s_prog.store(cutlass.Int32(0), idx=0) + if warp == X_WARP: # slot tables: empty (the slot warp fills them after the top-k) + for z in cutlass.range_constexpr((G_CAP * M_MAX + 31) // 32): + if lane + cutlass.Int32(32 * z) < cutlass.Int32(G_CAP * M_MAX): + s_kpos.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + for z in cutlass.range_constexpr(E_LOCAL // 32): + s_emap.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + if warp == MMA_WARP: + prims.tcgen05_alloc(tmem_ptr_i32, TMEM_COLS) + prims.tcgen05_relinquish_alloc_permit() + if warp == W_WARP: + prims.prefetch_tensormap(tma_a1_desc.get_ptr()) + prims.prefetch_tensormap(tma_a2_desc.get_ptr()) + if cutlass.const_expr(not SFA_CP): + prims.prefetch_tensormap(tma_s1_desc.get_ptr()) + prims.prefetch_tensormap(tma_s2_desc.get_ptr()) + for i in cutlass.range_constexpr(E_LOCAL // 32): + s_bias.store(cutlass.Float32(bias.load(idx=lane + i * 32)), idx=lane + i * 32) + for i in cutlass.range_constexpr(((G_CAP + 1) * (NG_CAP + 1) + 31) // 32): + q = lane + cutlass.Int32(32 * i) + if q < cutlass.Int32((G_CAP + 1) * (NG_CAP + 1)): + gq = q // cutlass.Int32(NG_CAP + 1) + s_plan_all.store( + plan.load( + idx=(gq * cutlass.Int32(NCTA) + bx) * cutlass.Int32(NG_CAP + 1) + + q + - gq * cutlass.Int32(NG_CAP + 1) + ), + idx=q, + ) + # router weights (expert bx) in registers: thread tg = tid % 256 owns elements 4 tg + 1024 j + tg = tidx % cutlass.Int32(GEMV_THREADS) + half = tidx // cutlass.Int32(GEMV_THREADS) + rwv = [ + rw.load( + idx=bx * cutlass.Int32(H) + tg * cutlass.Int32(4) + cutlass.Int32(1024 * j), + vector_size=4, + alignment=16, + ) + for j in range(H // 1024) + ] + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + + cute.arch.griddepcontrol_wait() + # ---- x chunks into the idle ring (warp 6): chunk c = K [XCK c, XCK (c + 1)) of every token + if warp == S_WARP: + if cutlass.const_expr(NPT > 0): + # the priority tokens' rows first (their logits are published ahead of the others) + if lane == 0: + prims.mbarrier_arrive_expect_tx(px_full.subview(0), NPT * H * 2) + cute.arch.sync_warp() + if lane < NPT: + prims.cp_async_bulk_shared_cluster_global( + sB1.subview(lane * cutlass.Int32(H * 2)), + xb32.subview( + cutlass.select_( + lane == cutlass.Int32(0), cutlass.Int32(0), cutlass.Int32(PT1) + ) + * cutlass.Int32(H // 2) + ), + px_full.subview(0), + H * 2, + ) + for c in cutlass.range_constexpr(NXB): + if lane == 0: + prims.mbarrier_arrive_expect_tx(x_full.subview(c), XCB) + cute.arch.sync_warp() + if lane < M_MAX: + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(c * XCB) + lane * cutlass.Int32(XCK * 2)), + xb32.subview(lane * cutlass.Int32(H // 2) + cutlass.Int32(c * XCK // 2)), + x_full.subview(c), + XCK * 2, + ) + ar_on = arp.load(idx=0) != cutlass.Int64(0) + if cutlass.const_expr(not LATE_TRIGGER): + if arp.load(idx=0) == cutlass.Int64(0): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + ltag = cutlass.Int64(ep + cutlass.Int32(1)) << cutlass.Int64(32) + cnt_target = (ep + cutlass.Int32(1)) * cutlass.Int32(CNT_PER_CALL) + + sc = cutlass.Float32(sfsc.load(idx=0)) + rsc = _rcp(sc) + rsix = _rcp(cutlass.Float32(6.0)) + img8f = cutlass.Array( + img32.data_ptr(0), shape=(IMG_TOTAL,), dtype=cutlass.Int8, alignment=16 + ) + if cutlass.const_expr(NPT > 0): + # ---- priority tokens (0 and PT1): their logits first (m3_router_gemv's per-thread / warp order) + _wait(px_full.subview(0), 0) + ph = cutlass.select_(half < cutlass.Int32(NPT), half, cutlass.Int32(0)) + pacc = cutlass.Float32(0.0) + px32 = cutlass.Array( + sB1.data_ptr(0), shape=(NPT * H // 2,), dtype=cutlass.Int32, alignment=16 + ) + for j in cutlass.range_constexpr(H // 1024): + pxw = px32.load( + idx=ph * cutlass.Int32(H // 2) + cutlass.Int32(512 * j) + tg * cutlass.Int32(2), + vector_size=2, + alignment=8, + ) + px0 = cutlass.Int32(pxw[0]) + px1 = cutlass.Int32(pxw[1]) + pacc = _fma_rn( + (px0 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[j][0]), + pacc, + ) + pacc = _fma_rn( + (px0 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[j][1]), + pacc, + ) + pacc = _fma_rn( + (px1 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[j][2]), + pacc, + ) + pacc = _fma_rn( + (px1 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[j][3]), + pacc, + ) + for off in (16, 8, 4, 2, 1): + pacc = _add_rn(pacc, cute.arch.shuffle_sync_bfly(pacc, off)) + if lane == 0: + s_red.store(pacc, idx=cutlass.Int32(9 * M_MAX) + warp) # [half][8 warps] + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ALL_BAR, number_of_threads=THREADS) + if tidx < cutlass.Int32(NPT * LG_REPL): + hh = tidx // cutlass.Int32(LG_REPL) + rr = tidx - hh * cutlass.Int32(LG_REPL) + sacc = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(8): + sacc = _add_rn( + sacc, s_red.load(idx=cutlass.Int32(9 * M_MAX + w) + hh * cutlass.Int32(8)) + ) + tp = cutlass.select_(hh == cutlass.Int32(0), cutlass.Int32(0), cutlass.Int32(PT1)) + _st_relaxed_u64( + lgbuf.subview(rr * cutlass.Int32(LG_STRIDE) + tp * cutlass.Int32(E_LOCAL) + bx) + .data_ptr() + .toint(), + ltag | (cutlass.Int64(sacc.bitcast(cutlass.Int32)) & cutlass.Int64(0xFFFFFFFF)), + ) + if tidx == 0: + _stamp(tdbg, bx, 43) + # ---- this CTA's share of the NVFP4 activation (QB blocks, from global) for the global B1 image: loads first + QB = (M_MAX * (H // 16) + NCTA - 1) // NCTA + qb = bx * cutlass.Int32(QB) + tidx + q_ok = (tidx < cutlass.Int32(QB)) & (qb < cutlass.Int32(M_MAX * (H // 16))) + qb = cutlass.select_(q_ok, qb, cutlass.Int32(0)) + qtok = qb // cutlass.Int32(H // 16) + qkb = qb - qtok * cutlass.Int32(H // 16) + xq_v = [ + xb32.load( + idx=qtok * cutlass.Int32(H // 2) + qkb * cutlass.Int32(8) + cutlass.Int32(4 * h), + vector_size=4, + alignment=16, + ) + for h in range(2) + ] + # ---- router GEMV from the staged chunks, all 512 threads (rolled: the front runs once per call) + MH = (M_MAX + 1) // 2 # tokens per GEMV half + acc_r = cute.make_rmem_tensor((MH,), cutlass.Float32) + for i in cutlass.range_constexpr(MH): + acc_r[i] = cutlass.Float32(0.0) + big32 = cutlass.Array( + s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Int32, alignment=16 + ) + JPC = XCK // 1024 # router j-steps per chunk + for xc in cutlass.range(NXC, unroll=1): + buf = xc % cutlass.Int32(NXB) + _wait(x_full.subview(buf), (xc // cutlass.Int32(NXB)) & cutlass.Int32(1)) + if (tidx == 0) & (xc == cutlass.Int32(0)): + _stamp(tdbg, bx, 41) + xw0 = buf * cutlass.Int32(XCB // 4) # chunk word base in s_big + for jj in cutlass.range_constexpr(JPC): + jg = xc * cutlass.Int32(JPC) + cutlass.Int32(jj) + wsel = [cutlass.Float32(rwv[0][z]) for z in range(4)] + for jx in cutlass.range_constexpr(1, H // 1024): + for z in cutlass.range_constexpr(4): + wsel[z] = cutlass.select_( + jg == cutlass.Int32(jx), cutlass.Float32(rwv[jx][z]), wsel[z] + ) + for i in cutlass.range_constexpr(MH): + t = half * cutlass.Int32(MH) + cutlass.Int32(i) + t = cutlass.select_(t < cutlass.Int32(M_MAX), t, cutlass.Int32(M_MAX - 1)) + xw = big32.load( + idx=xw0 + + t * cutlass.Int32(XCK // 2) + + cutlass.Int32(512 * jj) + + tg * cutlass.Int32(2), + vector_size=2, + alignment=8, + ) + x0 = cutlass.Int32(xw[0]) + x1 = cutlass.Int32(xw[1]) + acc_g = acc_r[i] + acc_g = _fma_rn( + (x0 << cutlass.Int32(16)).bitcast(cutlass.Float32), wsel[0], acc_g + ) + acc_g = _fma_rn( + (x0 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), wsel[1], acc_g + ) + acc_g = _fma_rn( + (x1 << cutlass.Int32(16)).bitcast(cutlass.Float32), wsel[2], acc_g + ) + acc_g = _fma_rn( + (x1 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), wsel[3], acc_g + ) + acc_r[i] = acc_g + if cutlass.const_expr(NXB < NXC): + if xc + cutlass.Int32(NXB) < cutlass.Int32(NXC): + # every thread's reads of this chunk are done (CTA barrier, then a generic -> async proxy fence by + # the refilling warp: the bulk copy overwrites what the generic proxy read); racecheck-clean + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ALL_BAR, number_of_threads=THREADS) + if warp == S_WARP: + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + if lane == 0: + prims.mbarrier_arrive_expect_tx(x_full.subview(buf), XCB) + cute.arch.sync_warp() + if lane < M_MAX: + prims.cp_async_bulk_shared_cluster_global( + s_big.subview( + buf * cutlass.Int32(XCB) + lane * cutlass.Int32(XCK * 2) + ), + xb32.subview( + lane * cutlass.Int32(H // 2) + + (xc + cutlass.Int32(NXB)) * cutlass.Int32(XCK // 2) + ), + x_full.subview(buf), + XCK * 2, + ) + if tidx == 0: + _stamp(tdbg, bx, 42) + for i in cutlass.range_constexpr(MH): + v = acc_r[i] + for off in (16, 8, 4, 2, 1): + v = _add_rn(v, cute.arch.shuffle_sync_bfly(v, off)) + t = half * cutlass.Int32(MH) + cutlass.Int32(i) + if (lane == 0) & (t < cutlass.Int32(M_MAX)): + s_red.store(v, idx=(warp % cutlass.Int32(8)) * cutlass.Int32(M_MAX) + t) + # this CTA's quantization blocks into the global B1 image (B1 tile bytes, then the [M, 384] scale codes) + if q_ok: + qw = [] + for h in cutlass.range_constexpr(2): + for z in cutlass.range_constexpr(4): + qw.append(cutlass.Int32(xq_v[h][z])) + qlo, qhi, qsf = _quant_words(qw, sc, rsc, rsix) + img32.store( + (qlo, qhi), idx=cutlass.Int32(B1IMG_OFF // 4) + _b1_addr(qtok, qkb), alignment=8 + ) + img8f.store(cutlass.Int8(qsf), idx=cutlass.Int32(B1IMG_OFF + S1 * NUM_BYTES_B1) + qb) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ALL_BAR, number_of_threads=THREADS) + if warp == cutlass.Int32(0): + _dbg_delay(bx * cutlass.Int32(3)) + if tidx < LG_REPL: + for t in cutlass.range_constexpr(M_MAX): + sacc = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(8): + sacc = _add_rn(sacc, s_red.load(idx=cutlass.Int32(w * M_MAX + t))) + lw = ltag | (cutlass.Int64(sacc.bitcast(cutlass.Int32)) & cutlass.Int64(0xFFFFFFFF)) + _st_relaxed_u64( + lgbuf.subview(tidx * cutlass.Int32(LG_STRIDE) + cutlass.Int32(t * E_LOCAL) + bx) + .data_ptr() + .toint(), + lw, + ) + if tidx == 0: + _stamp(tdbg, bx, 21) + if tidx == cutlass.Int32( + THREADS - 32 + ): # B1 share released after the logits (its gpu fence stalls warp 15 only) + if cutlass.const_expr(EPI_PFENCE): + _fence_proxy_async_global() # once, after the join (ALL_BAR) of the writer threads + _red_release_add( + cnt.subview(cutlass.Int32(NG_CAP * 8)).data_ptr().toint(), cutlass.Int32(1) + ) + # ---- the B1 image (every CTA's share) into smem (warp 13; the top-k warps are 0 .. M - 1) + if cutlass.const_expr(B1_WARP >= 0): + if warp == cutlass.Int32(B1_WARP): + if lane == 0: + b1addr = cnt.subview(cutlass.Int32(NG_CAP * 8)).data_ptr().toint() + b1tgt = (ep + cutlass.Int32(1)) * cutlass.Int32(NCTA) + cv = _ld_acquire_u32(b1addr) + while (cv - b1tgt) < cutlass.Int32(0): + _nanosleep(64) + cv = _ld_acquire_u32(b1addr) + _fence_proxy_async_global() + prims.mbarrier_arrive_expect_tx( + b1_load.subview(0), S1 * NUM_BYTES_B1 + M_MAX * (H // 16) + ) + prims.cp_async_bulk_shared_cluster_global( + sB1.subview(0), + img32.subview(cutlass.Int32(B1IMG_OFF // 4)), + b1_load.subview(0), + S1 * NUM_BYTES_B1, + ) + prims.cp_async_bulk_shared_cluster_global( + s_xsf.subview(0), + img32.subview(cutlass.Int32((B1IMG_OFF + S1 * NUM_BYTES_B1) // 4)), + b1_load.subview(0), + M_MAX * (H // 16), + ) + _stamp(tdbg, bx, 23) + cute.arch.sync_warp() + + # ---- top-k: token t on warp t (keys sigmoid + bias, ties to the lower id), TRTLLM-Gen MiniMax2 ops + tk_t = cutlass.Int32(-1) # this warp's top-k token (TK_WARPS[t] = warp) + for i_tk in cutlass.range_constexpr(M_MAX): + tk_t = cutlass.select_(warp == cutlass.Int32(TK_WARPS[i_tk]), cutlass.Int32(i_tk), tk_t) + if tk_t >= cutlass.Int32(0): + t = tk_t + want = ltag >> cutlass.Int64(32) + rbase = ( + (bx % cutlass.Int32(LG_REPL)) * cutlass.Int32(LG_STRIDE) + + t * cutlass.Int32(E_LOCAL) + + lane + ) + done = cutlass.Int32(0) + res = [cutlass.Int64(0) for _ in range(4)] + while done == cutlass.Int32(0): + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(4): + res[z] = _ld_relaxed_u64( + lgbuf.subview(cutlass.Int32(z * 32) + rbase).data_ptr().toint() + ) + ok = ok & ((res[z] >> cutlass.Int64(32)) == want) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32(-1): + done = cutlass.Int32(1) + else: + _nanosleep(POLL_NS) + if (lane == 0) & (t == cutlass.Int32(0)): + _stamp(tdbg, bx, 22) + lgv = [cutlass.Int32(res[z]).bitcast(cutlass.Float32) for z in range(4)] + keys = [] + for i in cutlass.range_constexpr(4): + sg = _fma_rn( + cutlass.Float32(0.5), + cute.math.tanh(_mul_rn(cutlass.Float32(0.5), lgv[i])), + cutlass.Float32(0.5), + ) + keys.append(_add_rn(sg, s_bias.load(idx=i * 32 + lane))) + hk = [_twiddle(keys[i]) for i in range(4)] + taken = [cutlass.Boolean(False) for _ in range(4)] + ids_t = [] + kt_t = [] + for k in cutlass.range_constexpr(TOP_K): + bh = cutlass.Int32(0) + bl = cutlass.Int32(0) + for i in cutlass.range_constexpr(4): + lo_i = cutlass.Int32(65535 - i * 32) - lane + hv = cutlass.select_(taken[i], cutlass.Int32(0), hk[i]) + lv = cutlass.select_(taken[i], cutlass.Int32(0), lo_i) + better = (cutlass.Uint32(hv) > cutlass.Uint32(bh)) | ( + (hv == bh) & (cutlass.Uint32(lv) > cutlass.Uint32(bl)) + ) + bh = cutlass.select_(better, hv, bh) + bl = cutlass.select_(better, lv, bl) + mh = _redux_max_u32(bh) + ml = _redux_max_u32(cutlass.select_(bh == mh, bl, cutlass.Int32(0))) + e_k = cutlass.Int32(65535) - ml + for i in cutlass.range_constexpr(4): + taken[i] = taken[i] | (e_k == cutlass.Int32(i * 32) + lane) + ids_t.append(e_k) + kv = cutlass.Float32(0.0) + for i in cutlass.range_constexpr(4): + kv = cutlass.select_(e_k == cutlass.Int32(i * 32) + lane, keys[i], kv) + kt_t.append(cute.arch.shuffle_sync(kv, e_k % cutlass.Int32(32))) + sig = cutlass.Float32(0.0) + e_l = cutlass.Int32(0) + for k in cutlass.range_constexpr(TOP_K): + sk = _sub_rn(kt_t[k], s_bias.load(idx=ids_t[k])) + sig = cutlass.select_(lane == cutlass.Int32(k), sk, sig) + e_l = cutlass.select_(lane == cutlass.Int32(k), ids_t[k], e_l) + total = sig + for off in (16, 8, 4, 2, 1): + total = _add_rn(total, cute.arch.shuffle_sync_bfly(total, off)) + wv = _bf16_round( + _div_rn( + _mul_rn(sig, cutlass.Float32(ROUTE_SCALE)), + _add_rn(total, cutlass.Float32(SUM_EPS)), + ) + ) + if lane < TOP_K: + s_w.store(wv, idx=t * cutlass.Int32(TOP_K) + lane) + s_ids.store(e_l, idx=t * cutlass.Int32(TOP_K) + lane) + if bx == 0: + route_dbg.store( + wv.bitcast(cutlass.Int32), + idx=t * cutlass.Int32(8) + cutlass.Int32(4) + lane, + ) + if (lane == 0) & (t == cutlass.Int32(0)): + _stamp(tdbg, bx, 20) + if cutlass.const_expr(PF_PRIO): + if (t == cutlass.Int32(0)) | (t == cutlass.Int32(PT1)): + if lane < TOP_K: + # this CTA's 1/NCTA of each of the token's experts into L2 (DRAM is idle until the routing is + # complete; every byte is streamed later anyway) + _l2_prefetch( + w1b.subview( + e_l * cutlass.Int32(TWO_I * H // 2) + bx * cutlass.Int32(PF_W1) + ) + .data_ptr() + .toint(), + PF_W1, + ) + _l2_prefetch( + s1w.subview( + (e_l * cutlass.Int32(TWO_I * H // 16) + bx * cutlass.Int32(PF_S1)) + // cutlass.Int32(4) + ) + .data_ptr() + .toint(), + PF_S1, + ) + _l2_prefetch( + w2b.subview( + e_l * cutlass.Int32(H * I_TP // 2) + bx * cutlass.Int32(PF_W2) + ) + .data_ptr() + .toint(), + PF_W2, + ) + _l2_prefetch( + s2w.subview( + (e_l * cutlass.Int32(H * I_TP // 16) + bx * cutlass.Int32(PF_S2)) + // cutlass.Int32(4) + ) + .data_ptr() + .toint(), + PF_S2, + ) + cute.arch.sync_warp() + if lane < TOP_K: # every writer lane arrives (its own release of s_ids / s_w) + prims.mbarrier_arrive(topk_bar.subview(0)) # this token's ids / weights are in smem + if t == cutlass.Int32(0): + prims.mbarrier_arrive( + tok0_bar.subview(0) + ) # token 0's experts: the early group-0 stages + cute.arch.sync_warp() + + if cutlass.const_expr(EARLY0): + if warp == W_WARP: + # slots 0..3 = token 0's four distinct experts; group 0's stage ranges are the same for every G + _wait(tok0_bar.subview(0), 0) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(STAGES): + _issue_fc1( + lane, + n0, + e, + c, + j, + sA, + sSF, + ab_full, + tma_a1_desc.get_ptr(), + tma_s1_desc.get_ptr(), + s1w, + ) + n0 = n0 + cutlass.Int32(1) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + if warp == X_WARP: + # slots: entry i < 4 M is (token i / 4, top-k i % 4); distinct experts in entry order + NCH = (TOP_K * M_MAX + 31) // 32 + _wait(topk_bar.subview(0), 0) + gcnt = cutlass.Int32(0) + rec = [] # per chunk: (first, rank, slot rank of the entry, expert, entry valid) + for ch in cutlass.range_constexpr(NCH): + ei = lane + cutlass.Int32(32 * ch) + valid = ei < cutlass.Int32(TOP_K * M_MAX) + my_e = cutlass.select_( + valid, + s_ids.load(idx=cutlass.select_(valid, ei, cutlass.Int32(0))), + cutlass.Int32(-1) - lane, + ) + known = cutlass.Boolean(False) + prev = cutlass.Int32(-1) + key = my_e + if cutlass.const_expr(ch > 0): + prev = cutlass.select_( + valid, + s_emap.load(idx=cutlass.select_(valid, my_e, cutlass.Int32(0))), + cutlass.Int32(-1), + ) + known = prev >= cutlass.Int32(0) + key = cutlass.select_(known, cutlass.Int32(-1000) - lane, my_e) + cute.arch.sync_warp() # every lane's s_emap read before any lane's write + same = _match_any(key) + leader = cute.arch.popc((same & (cutlass.Int32(0) - same)) - cutlass.Int32(1)) + first = valid & (known == cutlass.Boolean(False)) & (lane == leader) + bal = prims.vote_sync(0xFFFFFFFF, first, prims.VoteSync.BALLOT) + rank = gcnt + cute.arch.popc(bal & cutlass.Int32(cute.arch.lanemask_lt())) + slot_l = cute.arch.shuffle_sync(rank, leader) + slot_l = cutlass.select_(known, prev, slot_l) + if first: + s_emap.store(rank, idx=my_e) + if valid: + if bx == 0: + route_dbg.store( + my_e, + idx=(ei // cutlass.Int32(TOP_K)) * cutlass.Int32(8) + + ei % cutlass.Int32(TOP_K), + ) + rec.append((first, rank, slot_l, my_e, valid)) + cute.arch.sync_warp() + gcnt = gcnt + cute.arch.popc(bal) + # slot of rank r: with a partial last group (tail < 4) and >= 3 groups, that group becomes group 1 (its + # FC2 then is not the last hand-off behind a short FC1 phase); the full groups 1.. shift up by one + ngr = (gcnt + cutlass.Int32(3)) >> cutlass.Int32(2) + tail = gcnt - (ngr - cutlass.Int32(1)) * cutlass.Int32(4) + do_perm = ( + cutlass.Boolean(PERM) + & (tail < cutlass.Int32(4)) + & (ngr >= cutlass.Int32(PERM_MIN_NG)) + ) + for z in cutlass.range_constexpr((G_CAP + 31) // 32): + if lane + cutlass.Int32(32 * z) < cutlass.Int32(G_CAP): + s_el.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + cute.arch.sync_warp() + for ch in cutlass.range_constexpr(NCH): + first, rank, slot_r, my_e, valid = rec[ch] + ei = lane + cutlass.Int32(32 * ch) + for which in cutlass.range_constexpr(2): + r = rank if which == 0 else slot_r + sl_m = cutlass.select_( + r < cutlass.Int32(4), + r, + cutlass.select_( + r >= (ngr - cutlass.Int32(1)) * cutlass.Int32(4), + r - (ngr - cutlass.Int32(2)) * cutlass.Int32(4), + r + cutlass.Int32(4), + ), + ) + sl_m = cutlass.select_(do_perm, sl_m, r) + if cutlass.const_expr(which == 0): + if first: + s_el.store(my_e, idx=sl_m) + else: + if valid: + s_kpos.store( + ei % cutlass.Int32(TOP_K), + idx=sl_m * cutlass.Int32(M_MAX) + ei // cutlass.Int32(TOP_K), + ) + # slots per group + if lane < cutlass.Int32(NG_CAP): + nh = gcnt - lane * cutlass.Int32(4) + nh = cutlass.select_(nh > cutlass.Int32(4), cutlass.Int32(4), nh) + nh = cutlass.select_(nh < cutlass.Int32(0), cutlass.Int32(0), nh) + nh = cutlass.select_(do_perm & (lane == cutlass.Int32(1)), tail, nh) + nh = cutlass.select_( + do_perm & (lane == ngr - cutlass.Int32(1)), cutlass.Int32(4), nh + ) + s_gn.store(nh, idx=lane) + cute.arch.sync_warp() + if lane == 0: + s_el.store(gcnt, idx=G_CAP) + if lane < cutlass.Int32(NG_CAP + 1): + s_plan.store(s_plan_all.load(idx=gcnt * cutlass.Int32(NG_CAP + 1) + lane), idx=lane) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 24) + prims.mbarrier_arrive( + route_bar.subview(0) + ) # every lane: its slots / kpos / plan-row writes are in smem + cute.arch.sync_warp() + # CTA 0 pads every group counter to exactly CNT_PER_CALL for this call (absent slots, unused groups) + if (bx == 0) & (lane < cutlass.Int32(NG_CAP)): + nh = s_gn.load(idx=lane) + pad = cutlass.Int32(CNT_PER_CALL) - nh * cutlass.Int32(UNITS_ROW) + if pad > cutlass.Int32(0): + _red_release_add(cnt.subview(lane * cutlass.Int32(8)).data_ptr().toint(), pad) + else: + _wait(route_bar.subview(0), 0) + if tidx == 0: + _stamp(tdbg, bx, 2) + if cutlass.const_expr(B1_WARP < 0): + if warp == cutlass.Int32(0): + if lane == 0: + b1addr = cnt.subview(cutlass.Int32(NG_CAP * 8)).data_ptr().toint() + b1tgt = (ep + cutlass.Int32(1)) * cutlass.Int32(NCTA) + cv = _ld_acquire_u32(b1addr) + while (cv - b1tgt) < cutlass.Int32(0): + _nanosleep(64) + cv = _ld_acquire_u32(b1addr) + _fence_proxy_async_global() + prims.mbarrier_arrive_expect_tx( + b1_load.subview(0), S1 * NUM_BYTES_B1 + M_MAX * (H // 16) + ) + prims.cp_async_bulk_shared_cluster_global( + sB1.subview(0), + img32.subview(cutlass.Int32(B1IMG_OFF // 4)), + b1_load.subview(0), + S1 * NUM_BYTES_B1, + ) + prims.cp_async_bulk_shared_cluster_global( + s_xsf.subview(0), + img32.subview(cutlass.Int32((B1IMG_OFF + S1 * NUM_BYTES_B1) // 4)), + b1_load.subview(0), + M_MAX * (H // 16), + ) + _stamp(tdbg, bx, 23) + cute.arch.sync_warp() + if warp < 4: # B1 scales to TMEM (the MMA needs them only with the first ring stage) + _wait(b1_load.subview(0), 0) + _sfb1_to_tmem(warp, lane, base_row, base_col, s_xsf, b1_ready) + + g_n = s_el.load(idx=G_CAP) + ng = (g_n + cutlass.Int32(3)) >> cutlass.Int32(2) + pr2 = s_plan.load(idx=NG_CAP) + blk0 = pr2 & cutlass.Int32(0xFFFF) + nb = pr2 >> cutlass.Int32(16) + + # ---- weights producer (warp 4): phase p = FC1 stages of group p, then FC2 tiles of group p - LAG + if warp == W_WARP: + stage = cutlass.Int32(0) + phase = cutlass.Int32(1) + skip = cutlass.Int32(0) + n_iss = cutlass.Int32( + 0 + ) # stages issued (early ones included): the L2-prefetch warp's throttle + w_emp = cutlass.Int64(0) + if cutlass.const_expr(EARLY0): + skip = s_misc.load(idx=0) + stage = cutlass.select_(skip == cutlass.Int32(STAGES), cutlass.Int32(0), skip) + phase = cutlass.select_( + skip == cutlass.Int32(STAGES), cutlass.Int32(0), cutlass.Int32(1) + ) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_el.load(idx=slot) + for j in cutlass.range(j0, j1, unroll=1): + if skip > cutlass.Int32(0): + skip = skip - cutlass.Int32(1) + else: + w_emp = _waitt(ab_empty.subview(stage), phase, w_emp) + prims.cp_async_wait_group(STAGES - 1) + _dbg_delay(bx) + _issue_fc1( + lane, + stage, + e, + c, + j, + sA, + sSF, + ab_full, + tma_a1_desc.get_ptr(), + tma_s1_desc.get_ptr(), + s1w, + ) + stage, phase = _adv(stage, phase) + if cutlass.const_expr(PFD > 0): + n_iss = n_iss + cutlass.Int32(1) + if lane == 0: + prims.store_ext( + n_iss.ir_value(), + s_prog.subview(0), + order="relaxed", + scope="cta", + ) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + n_here = s_gn.load(idx=g) + for bb in range(nb): + blk = blk0 + bb + for t2 in cutlass.range_constexpr(KT2): + w_emp = _waitt(ab_empty.subview(stage), phase, w_emp) + prims.cp_async_wait_group(STAGES - 1) + _issue_fc2( + lane, + stage, + g, + blk, + t2, + n_here, + s_el, + sA, + sSF, + ab_full, + tma_a2_desc.get_ptr(), + tma_s2_desc.get_ptr(), + s2w, + ) + stage, phase = _adv(stage, phase) + if cutlass.const_expr(PFD > 0): + n_iss = n_iss + cutlass.Int32(1) + if lane == 0: + prims.store_ext( + n_iss.ir_value(), + s_prog.subview(0), + order="relaxed", + scope="cta", + ) + if lane == 0: + _stamp(tdbg, bx, 3) + _tstore(tdbg, bx, 30, w_emp) + + # ---- (SFA_CP = 0) scale splicer (warp 6): atom kb word [r][q] = raw atom 4 q + kb word [r][m1] + if cutlass.const_expr(not SFA_CP): + if warp == S_WARP: + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + for p in range(ng + cutlass.Int32(LAG)): + i0 = cutlass.Int32(0) + i1 = cutlass.Int32(0) + for part in cutlass.range_constexpr(2): + cnt_ = cutlass.Int32(0) + m1 = cutlass.Int32(0) + nk = cutlass.Int32(0) + if cutlass.const_expr(part == 0): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + nk = _nsegs(i0, i1) + else: + if p >= cutlass.Int32(LAG): + nk = nb + for k in range(nk): + if cutlass.const_expr(part == 0): + ug, j0, j1 = _seg(i0, i1, k) + m1 = ( + ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + ) & cutlass.Int32(3) + cnt_ = j1 - j0 + else: + m1 = (blk0 + k) & cutlass.Int32(3) + cnt_ = cutlass.Int32(KT2) + for _j in cutlass.range(cnt_, unroll=1): + _waitd(ab_full.subview(stage), phase, tdbg, 3) + raw0 = ( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + lane * cutlass.Int32(4) + + m1 + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32( + 4 + ) + for kb in cutlass.range_constexpr(NUM_KB): + w0 = sSF.load(idx=raw0 + cutlass.Int32((0 * 4 + kb) * 128)) + w1 = sSF.load(idx=raw0 + cutlass.Int32((1 * 4 + kb) * 128)) + w2 = sSF.load(idx=raw0 + cutlass.Int32((2 * 4 + kb) * 128)) + w3 = sSF.load(idx=raw0 + cutlass.Int32((3 * 4 + kb) * 128)) + sSFA.store( + ( + cutlass.Int32(w0), + cutlass.Int32(w1), + cutlass.Int32(w2), + cutlass.Int32(w3), + ), + idx=dst0 + cutlass.Int32(kb * 128), + alignment=16, + ) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + cute.arch.sync_warp() + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + prims.tcgen05_commit(sfa_ready.subview(stage)) + stage, phase = _adv(stage, phase) + + # ---- FC2 B loader (warp 5): group g's image once its counter is complete + if warp == X_WARP: + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + for g in range(ng): + buf = g % cutlass.Int32(NB2BUF) + if g >= cutlass.Int32(NB2BUF): + _waitd( + b2_free.subview(buf), + ((g // cutlass.Int32(NB2BUF)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 29) + caddr = cnt.subview(g * cutlass.Int32(8)).data_ptr().toint() + if lane == 0: + cv = _ld_acquire_u32(caddr) + while (cv - cnt_target) < cutlass.Int32(0): + _nanosleep(B2_POLL_NS) + cv = _ld_acquire_u32(caddr) + cute.arch.sync_warp() + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 51) + _fence_proxy_async_global() + if g >= cutlass.Int32(1): + _waitd( + sfb2_free.subview(0), (g - cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 107 + ) + if lane == 0: + prims.mbarrier_arrive_expect_tx(img_bar.subview(buf), IMG_BYTES) + prims.cp_async_bulk_shared_cluster_global( + sB2.subview(buf * cutlass.Int32(IMG_B2)), + img32.subview(g * cutlass.Int32(IMG_BYTES // 4)), + img_bar.subview(buf), + IMG_B2, + ) + prims.cp_async_bulk_shared_cluster_global( + sSFB2.subview(0), + img32.subview( + g * cutlass.Int32(IMG_BYTES // 4) + cutlass.Int32(IMG_B2 // 4) + ), + img_bar.subview(buf), + IMG_SF, + ) + cute.arch.sync_warp() + _waitd( + img_bar.subview(buf), (g // cutlass.Int32(NB2BUF)) & cutlass.Int32(1), tdbg, 108 + ) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 52) + if g >= cutlass.Int32(1): + # the single TMEM scale slot: the previous group's FC2 MMAs are done + gp = g - cutlass.Int32(1) + _waitd( + b2_free.subview(gp % cutlass.Int32(NB2BUF)), + (gp // cutlass.Int32(NB2BUF)) & cutlass.Int32(1), + tdbg, + 109, + ) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 53) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for aa in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SFB2_COL + aa * 4), + sd0 + cutlass.Int64(aa * (512 >> 4)), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + prims.tcgen05_commit(sfb2_free.subview(0)) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 25) + + # ---- (PFD) L2 prefetch warp (12): the same stage sequence as the producer, PFD stages ahead of it + if cutlass.const_expr(PFD > 0): + if warp == cutlass.Int32(PF_WARP): + pf = cutlass.Int32(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_el.load(idx=slot) + for j in cutlass.range(j0, j1, unroll=1): + if lane == 0: + prg = cutlass.Int32( + prims.load_ext( + s_prog.subview(0), + dtype=cutlass.Int32, + order="relaxed", + scope="cta", + ) + ) + while prg + cutlass.Int32(PFD) <= pf: + _nanosleep(100) + prg = cutlass.Int32( + prims.load_ext( + s_prog.subview(0), + dtype=cutlass.Int32, + order="relaxed", + scope="cta", + ) + ) + if pf >= prg: + kx = j * cutlass.Int32(C1 * KT_BYTES) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_prefetch( + tma_a1_desc.get_ptr(), + [ + kx + cutlass.Int32(q * KT_BYTES), + c * cutlass.Int32(R1), + e, + ], + [], + ) + atom0 = ( + e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2)) + ) * cutlass.Int32(SF1_KA) + j * cutlass.Int32(16) + _l2_prefetch( + s1w.subview(atom0 * cutlass.Int32(128)) + .data_ptr() + .toint(), + 8192, + ) + cute.arch.sync_warp() + pf = pf + cutlass.Int32(1) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + n_here = s_gn.load(idx=g) + for bb in range(nb): + blk = blk0 + bb + for t2 in cutlass.range_constexpr(KT2): + if lane == 0: + prg = cutlass.Int32( + prims.load_ext( + s_prog.subview(0), + dtype=cutlass.Int32, + order="relaxed", + scope="cta", + ) + ) + while prg + cutlass.Int32(PFD) <= pf: + _nanosleep(100) + prg = cutlass.Int32( + prims.load_ext( + s_prog.subview(0), + dtype=cutlass.Int32, + order="relaxed", + scope="cta", + ) + ) + if pf >= prg: + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load( + idx=g * cutlass.Int32(4) + cutlass.Int32(jj) + ) + prims.cp_async_bulk_tensor_prefetch( + tma_a2_desc.get_ptr(), + [ + cutlass.Int32(t2 * KT_BYTES), + blk * cutlass.Int32(R2), + e, + ], + [], + ) + a2 = ( + e * cutlass.Int32(SF2_RB) + + (blk >> cutlass.Int32(2)) + ) * cutlass.Int32(SF2_KA) + cutlass.Int32(4 * t2) + _l2_prefetch( + s2w.subview(a2 * cutlass.Int32(128)) + .data_ptr() + .toint(), + 2048, + ) + cute.arch.sync_warp() + pf = pf + cutlass.Int32(1) + + # ---- (REL_WARP) group releases (warp 8): this CTA's finished units of group p, one release add per phase + if cutlass.const_expr(REL_WARP): + if warp == cutlass.Int32(8): + for p in range(ng): + i0, i1 = _plan_range(s_plan, p) + n_done = cutlass.Int32(0) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + n_done = n_done + cutlass.select_( + (j0 != cutlass.Int32(0)) | (j1 == cutlass.Int32(S1)), + cutlass.Int32(1), + cutlass.Int32(0), + ) + _wait( + grp_done.subview(p & cutlass.Int32(3)), + (p >> cutlass.Int32(2)) & cutlass.Int32(1), + ) + _dbg_delay(bx * cutlass.Int32(13) + p) + if (lane == 0) & (n_done > cutlass.Int32(0)): + if cutlass.const_expr(EPI_PFENCE): + _fence_proxy_async_global() + _red_release_add( + cnt.subview(p * cutlass.Int32(8)).data_ptr().toint(), n_done + ) + if (lane == 0) & (p == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 54) + cute.arch.sync_warp() + + # ---- MMA (warp 7) + if warp == MMA_WARP: + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + idesc1 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N1, + m_dim=MMA_M, + sparsity_version=0, + ) + idesc2 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N2, + m_dim=MMA_M, + sparsity_version=0, + ) + desc_a0 = prims.Tcgen05SmemDesc.build( + sA.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b10 = prims.Tcgen05SmemDesc.build( + sB1.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b20 = prims.Tcgen05SmemDesc.build( + sB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + _waitd(b1_ready, 0, tdbg, 5) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + m_b2 = cutlass.Int64(0) + m_full = cutlass.Int64(0) + m_acc = cutlass.Int64(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + m_acc = _waitt( + acc_empty.subview(ab), + ((k1 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + m_acc, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc_ptr = cutlass.inttoptr( + tcol + ACC1_COL + ab * cutlass.Int32(N1), 6, cutlass.Float32 + ) + for j in cutlass.range(j0, j1, unroll=1): + if cutlass.const_expr(SFA_CP): + m_full = _waitt(ab_full.subview(stage), phase, m_full) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 6) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b10 + cutlass.Int64(j * cutlass.Int32(NUM_BYTES_B1 >> 4)) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = tcol + SFB1_COL + j * cutlass.Int32(NUM_KB * 4) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc1, + (j > j0) | cutlass.Boolean(kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + cute.arch.sync_warp() + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc_full.subview(ab)) + cute.arch.sync_warp() + k1 = k1 + cutlass.Int32(1) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + buf = g % cutlass.Int32(NB2BUF) + m_b2_0 = m_b2 + m_b2 = _waitt( + b2_ready.subview(buf), (g // cutlass.Int32(NB2BUF)) & cutlass.Int32(1), m_b2 + ) + if cutlass.const_expr(TIMING): + if (lane == 0) & (g < cutlass.Int32(4)): + tdbg.store( + m_b2 - m_b2_0, + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(44) + g, + ) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 4) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + m_acc = _waitt( + acc2_empty.subview(a2), + ((k2 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + m_acc, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc2_ptr = cutlass.inttoptr( + tcol + ACC2_COL + a2 * cutlass.Int32(ACC2_STRIDE), 6, cutlass.Float32 + ) + for t2 in cutlass.range_constexpr(KT2): + if cutlass.const_expr(SFA_CP): + m_full = _waitt(ab_full.subview(stage), phase, m_full) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 7) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b20 + cutlass.Int64( + (buf * cutlass.Int32(KT2) + cutlass.Int32(t2)) + * cutlass.Int32(NUM_BYTES_B2 >> 4) + ) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = tcol + SFB2_COL + cutlass.Int32(t2 * NUM_KB * 4) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc2_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc2, + cutlass.Boolean(t2 > 0 or kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + cute.arch.sync_warp() + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc2_full.subview(a2)) + cute.arch.sync_warp() + k2 = k2 + cutlass.Int32(1) + if prims.elect_sync(): + prims.tcgen05_commit( + b2_free.subview(buf) + ) # this group's B2 tile and TMEM scales are free + cute.arch.sync_warp() + if lane == 0: + _tstore(tdbg, bx, 26, m_b2) + _tstore(tdbg, bx, 27, m_full) + _tstore(tdbg, bx, 28, m_acc) + + # ---- epilogue warps (0-3) + if warp < 4: + row_w = base_row + warp * 32 + img8 = cutlass.Array( + img32.data_ptr(0), shape=(NG_CAP * IMG_BYTES,), dtype=cutlass.Int8, alignment=16 + ) + for r in cutlass.range_constexpr((6 * G_CAP + EPI_THREADS - 1) // EPI_THREADS): + qi = tidx + cutlass.Int32(r * EPI_THREADS) + sl_e = qi // cutlass.Int32(6) + if qi < cutlass.Int32(6 * G_CAP): + e_sl = s_el.load(idx=sl_e) + if e_sl >= cutlass.Int32(0): + fld = qi - sl_e * cutlass.Int32(6) + s_esc.store( + cutlass.Float32(escal.load(idx=fld * cutlass.Int32(E_LOCAL) + e_sl)), + idx=sl_e * cutlass.Int32(8) + fld, + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + e_wait = cutlass.Int64(0) + e_busy = cutlass.Int64(0) + e_a = cutlass.Int64(0) + e_pw = cutlass.Int64(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + cblk = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + unit = slot * cutlass.Int32(U1) + cblk + unit_done = (j0 != cutlass.Int32(0)) | (j1 == cutlass.Int32(S1)) + e_wait = _waitt( + acc_full.subview(ab), + (k1 >> cutlass.Int32(1)) & cutlass.Int32(1), + e_wait, + ) + tb0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, base_col + ACC1_COL + ab * cutlass.Int32(N1), cutlass.Float32 + ), + num=N1_LD, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc_empty.subview(ab)) + for t in cutlass.range_constexpr(M_MAX): + part = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(4): + part = cutlass.select_( + warp == cutlass.Int32(w), cutlass.Float32(acc[QS * w + t]), part + ) + s_part.store(part, idx=(t * C1 + warp) * R1 + lane) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + tc0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(TIMING): + e_a = e_a + (tc0 - tb0) + g_u = slot >> cutlass.Int32(2) + jrow = slot & cutlass.Int32(3) + # this warp's tokens t = warp + 4 tr, all in flight together (latency-bound otherwise) + NT = (M_MAX + 3) // 4 + xr = cute.make_rmem_tensor((NT,), cutlass.Float32) + for tr in cutlass.range_constexpr(NT): + tq = cutlass.select_( + warp + cutlass.Int32(4 * tr) < cutlass.Int32(M_MAX), + warp + cutlass.Int32(4 * tr), + cutlass.Int32(0), + ) + xr[tr] = _add_rn( + _add_rn( + _add_rn( + s_part.load(idx=(tq * C1 + 0) * R1 + lane), + s_part.load(idx=(tq * C1 + 1) * R1 + lane), + ), + s_part.load(idx=(tq * C1 + 2) * R1 + lane), + ), + s_part.load(idx=(tq * C1 + 3) * R1 + lane), + ) + if (j0 == cutlass.Int32(0)) & (j1 != cutlass.Int32(S1)): + # head of a unit split with the next CTA: its partial sums + for tr in cutlass.range_constexpr(NT): + t = warp + cutlass.Int32(4 * tr) + if t < cutlass.Int32(M_MAX): + _st_relaxed_u32( + partials.subview( + (unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane + ) + .data_ptr() + .toint(), + xr[tr].bitcast(cutlass.Int32), + ) + else: + if j0 != cutlass.Int32(0): + # tail: the head's partial sums (head first, as TRTLLM-Gen's K order) + for tr in cutlass.range_constexpr(NT): + t = warp + cutlass.Int32(4 * tr) + if t < cutlass.Int32(M_MAX): + paddr = ( + partials.subview( + (unit * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(R1) + + lane + ) + .data_ptr() + .toint() + ) + pbits = _ld_relaxed_u32(paddr) + while pbits == cutlass.Int32(PARTIAL_EMPTY): + pbits = _ld_relaxed_u32(paddr) + _st_relaxed_u32(paddr, cutlass.Int32(PARTIAL_EMPTY)) + xr[tr] = _add_rn(pbits.bitcast(cutlass.Float32), xr[tr]) + cute.arch.sync_warp() + _dbg_delay(bx * cutlass.Int32(7) + warp) + es = slot * cutlass.Int32(8) + ea = s_esc.load(idx=es) + ec = s_esc.load(idx=es + cutlass.Int32(1)) + eal = s_esc.load(idx=es + cutlass.Int32(2)) + ebe = s_esc.load(idx=es + cutlass.Int32(3)) + eli = s_esc.load(idx=es + cutlass.Int32(4)) + is_up = ( + (lane >> cutlass.Int32(3)) & cutlass.Int32(1) + ) == cutlass.Int32(0) + n0 = jrow * cutlass.Int32(M_MAX) + t2 = cblk >> cutlass.Int32(4) + cc = (cblk >> cutlass.Int32(1)) & cutlass.Int32(7) + gbase = g_u * cutlass.Int32(IMG_BYTES) + sfoff = ( + gbase + + cutlass.Int32(IMG_B2) + + ( + ( + t2 * cutlass.Int32(NUM_KB) + + ((cblk & cutlass.Int32(15)) >> cutlass.Int32(2)) + ) + * cutlass.Int32(512) + ) + + (cblk & cutlass.Int32(3)) + ) + for tr in cutlass.range_constexpr(NT): + # (skip) only tokens routed to this slot (warp-uniform); other B2 rows are never used + if s_kpos.load( + idx=slot * cutlass.Int32(M_MAX) + + cutlass.select_( + warp + cutlass.Int32(4 * tr) < cutlass.Int32(M_MAX), + warp + cutlass.Int32(4 * tr), + cutlass.Int32(0), + ) + ) >= cutlass.Int32(0): + xv = xr[tr] + gate_raw = cute.arch.shuffle_sync_bfly(xv, 8) + qv, sfb = _swiglu_quant( + xv, gate_raw, is_up, ea, ec, eal, ebe, eli + ) + byte = _e2m1x2(cute.arch.shuffle_sync_down(qv, 16), qv) + bt = [ + cute.arch.shuffle_sync(byte, i) & cutlass.Int32(0xFF) + for i in range(8) + ] + lo = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | (bt[3] << cutlass.Int32(24)) + ) + hi = ( + bt[4] + | (bt[5] << cutlass.Int32(8)) + | (bt[6] << cutlass.Int32(16)) + | (bt[7] << cutlass.Int32(24)) + ) + t = warp + cutlass.Int32(4 * tr) + if ( + (lane == 0) + & (t < cutlass.Int32(M_MAX)) + & cutlass.Boolean(not DBG_NOSTORE) + ): + # B2 row n = j M + t, k-tile c / 16, 16-byte chunk (c / 2) % 8 + # (swizzled), half c % 2; scale: atom (t2, kb = (c % 16) / 4), row n + # at byte 16 (n % 32) + 4 (n / 32), byte c % 4 + n = n0 + t + img32.store( + (lo, hi), + idx=( + gbase + + t2 * cutlass.Int32(NUM_BYTES_B2) + + n * cutlass.Int32(KT_BYTES) + + (cc ^ (n & cutlass.Int32(7))) * cutlass.Int32(16) + + (cblk & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + img8.store( + cutlass.Int8(sfb & cutlass.Int32(0xFF)), + idx=sfoff + + (n & cutlass.Int32(31)) * cutlass.Int32(16) + + (n >> cutlass.Int32(5)) * cutlass.Int32(4), + ) + tc1 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(TIMING): + e_pw = e_pw + (tc1 - tc0) + if cutlass.const_expr(not REL_WARP): + if unit_done: + if lane == 0: + if cutlass.const_expr(EPI_PFENCE): + _fence_proxy_async_global() + if cutlass.const_expr(EPI_FENCE): + _fence_acq_rel_gpu() + cute.arch.sync_warp() + cute.arch.barrier( + barrier_id=EPI_BAR, number_of_threads=EPI_THREADS + ) # s_part reads done + if cutlass.const_expr(not REL_WARP): + if unit_done & (tidx == 0): + _red_release_add( + cnt.subview(g_u * cutlass.Int32(8)).data_ptr().toint(), + cutlass.Int32(1), + ) + if cutlass.const_expr(TIMING): + e_busy = e_busy + (_gtime() - tb0) + k1 = k1 + cutlass.Int32(1) + if cutlass.const_expr(REL_WARP): + prims.mbarrier_arrive( + grp_done.subview(p & cutlass.Int32(3)) + ) # this CTA's group-p units are out + if tidx == 0: + _stamp(tdbg, bx, 10) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + e_wait = _waitt( + acc2_full.subview(a2), + (k2 >> cutlass.Int32(1)) & cutlass.Int32(1), + e_wait, + ) + if cutlass.const_expr(TIMING): + if ( + (tidx == 0) + & (g == ng - cutlass.Int32(1)) + & (bb == cutlass.Int32(0)) + ): + _stamp(tdbg, bx, 8) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + yv = [] + ya = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, + base_col + ACC2_COL + a2 * cutlass.Int32(ACC2_STRIDE), + cutlass.Float32, + ), + num=ACC2_STRIDE, + ) + for t in cutlass.range_constexpr(M_MAX): + v = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(4): + v = cutlass.select_( + warp == cutlass.Int32(w), cutlass.Float32(ya[w * M_MAX + t]), v + ) + yv.append(v) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc2_empty.subview(a2)) + sl = g * cutlass.Int32(4) + warp + if warp < s_gn.load(idx=g): + a2s = s_esc.load(idx=sl * cutlass.Int32(8) + cutlass.Int32(5)) + for t in cutlass.range_constexpr(M_MAX): + kp = s_kpos.load(idx=sl * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + if kp >= cutlass.Int32(0): + ys.store( + cutlass.BFloat16( + _bf16_round(_mul_rn(cutlass.Float32(yv[t]), a2s)) + ), + idx=( + (bb * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + * cutlass.Int32(TOP_K) + + kp + ) + * cutlass.Int32(R2) + + lane, + ) + k2 = k2 + cutlass.Int32(1) + if tidx == 0: + _stamp(tdbg, bx, 6) + _tstore(tdbg, bx, 31, e_wait) + _tstore(tdbg, bx, 34, e_busy) + _tstore(tdbg, bx, 36, e_a) + _tstore(tdbg, bx, 38, e_pw) + prims.mbarrier_arrive(ys_done.subview(0)) # this thread's ys values are in smem + + # ---- finalize on all 16 warps: (block bb, token t) pairs, the fma chain over t's top-4 in order, + y_sh + NPAIR = NB2_CAP * M_MAX + NPW = (NPAIR + NWARPS - 1) // NWARPS + wyl = [] + for bt in cutlass.range_constexpr(NPW): + pair = warp + cutlass.Int32(bt * NWARPS) + bb = pair // cutlass.Int32(M_MAX) + t = pair - bb * cutlass.Int32(M_MAX) + ok = (pair < cutlass.Int32(NPAIR)) & (bb < nb) + ch = cutlass.select_( + ok, + t * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + cutlass.Int32(0), + ) + wyl.append( + _ld_relaxed_u32(dnx.subview(ch).data_ptr().toint()) + if cutlass.const_expr(SHARED) + else cutlass.Int32(0) + ) + _wait(ys_done.subview(0), 0) + if tidx == 0: + _stamp(tdbg, bx, 19) + for bt in cutlass.range_constexpr(NPW): + pair = warp + cutlass.Int32(bt * NWARPS) + bb = pair // cutlass.Int32(M_MAX) + t = pair - bb * cutlass.Int32(M_MAX) + if (pair < cutlass.Int32(NPAIR)) & (bb < nb): + acc_o = cutlass.Float32(0.0) + for k in cutlass.range_constexpr(TOP_K): + acc_o = _fma_rn( + s_w.load(idx=t * cutlass.Int32(TOP_K) + cutlass.Int32(k)), + cutlass.Float32( + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ) + ), + acc_o, + ) + ch = ( + t * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(SHARED): + wy = wyl[bt] + while ((wy >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) != tag16: + wy = _ld_relaxed_u32(dnx.subview(ch).data_ptr().toint()) + acc_o = _add_rn( + (wy << cutlass.Int32(16)).bitcast(cutlass.Float32), _bf16_round(acc_o) + ) + out.store(cutlass.BFloat16(acc_o), idx=ch) + if ar_on: + cl = ( + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + _ar_push( + arp, + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + + # ---- teardown + if tidx == 0: + _stamp(tdbg, bx, 7) + epochs.store(ep + cutlass.Int32(1), idx=bx) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if ar_on | cutlass.Boolean(LATE_TRIGGER): + cute.arch.griddepcontrol_launch_dependents() + if warp == MMA_WARP: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Float32), TMEM_COLS + ) + + +@cute.jit +def m3_moe( + a1_tensor: cute.Tensor, + sfa1_tensor: cute.Tensor, + a2_tensor: cute.Tensor, + sfa2_tensor: cute.Tensor, + bias: cute.Tensor, + escal: cute.Tensor, + plan: cute.Tensor, + partials: cute.Tensor, + img32: cute.Tensor, + cnt: cute.Tensor, + s1w: cute.Tensor, + s2w: cute.Tensor, + w1b: cute.Tensor, + w2b: cute.Tensor, + epochs: cute.Tensor, + out: cute.Tensor, + route_dbg: cute.Tensor, + tdbg: cute.Tensor, + xb32: cute.Tensor, + rw: cute.Tensor, + sfsc: cute.Tensor, + lgbuf: cute.Tensor, + guw: cute.Tensor, + gusf: cute.Tensor, + dnw: cute.Tensor, + dnsf: cute.Tensor, + hx: cute.Tensor, + dnx: cute.Tensor, + arp: cute.Tensor, + stream: cuda_driver.CUstream, +): + tma_a1_desc = cuda.create_tensor_map_tiled( + global_address=a1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H // 2, TWO_I, E_LOCAL], + global_strides=[H // 2 // 16, TWO_I * (H // 2) // 16], + box_dims=(KT_BYTES, R1, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_a2_desc = cuda.create_tensor_map_tiled( + global_address=a2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[I_TP // 2, H, E_LOCAL], + global_strides=[I_TP // 2 // 16, H * (I_TP // 2) // 16], + box_dims=(KT_BYTES, R2, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_s1_desc = cuda.create_tensor_map_tiled( + global_address=sfa1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * TWO_I * (H // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_s2_desc = cuda.create_tensor_map_tiled( + global_address=sfa2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * H * (I_TP // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 4 // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_gu_desc = tma_a1_desc + tma_dn_desc = tma_a2_desc + if cutlass.const_expr(SHARED): + tma_gu_desc = cuda.create_tensor_map_tiled( + global_address=guw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H, 2 * SH_I], + global_strides=[H // 16], + box_dims=(KT_BYTES, 32), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_dn_desc = cuda.create_tensor_map_tiled( + global_address=dnw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[SH_I, H], + global_strides=[SH_I // 16], + box_dims=(KT_BYTES, 128), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + m3_moe_kernel( + tma_a1_desc, + tma_a2_desc, + tma_s1_desc, + tma_s2_desc, + tma_gu_desc, + tma_dn_desc, + bias, + escal, + plan, + partials, + img32, + cnt, + s1w, + s2w, + w1b, + w2b, + epochs, + out, + route_dbg, + tdbg, + xb32, + rw, + sfsc, + lgbuf, + guw, + gusf, + dnw, + dnsf, + hx, + dnx, + arp, + ).launch( + grid=(NCTA + NSH, 1, 1), + block=(THREADS, 1, 1), + cluster=(1, 1, 1), + stream=stream, + use_pdl=True, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/m3_moe_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/m3_moe_kernel.py new file mode 100644 index 000000000000..81122a06c18c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/m3_moe_kernel.py @@ -0,0 +1,3914 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 routed experts for up to 8 decode tokens on one TP4 rank (routed and shared intermediate 768): +``m3_moe`` (group-phased weight stream). + +One persistent kernel replaces TRTLLM-Gen's MiniMax2 routing, FC1 (NVFP4 x NVFP4 + SwiGLU-OAI + NVFP4 requantization), +FC2 and finalize, optionally with the router GEMV and the activation's NVFP4 quantization folded in (FUSED_ROUTER). It +writes the rank's routed partial [M, 6144] bf16, bit for bit what the TRTLLM-Gen op returns. + +- Routing (warps 4..7, token t on warp 4 + t % 4): TRTLLM-Gen's MiniMax2 top-4, operation for operation; warp 4 then + makes the distinct experts slots in (token, top-k) order (match.any), slot s in group s / 4. +- Work: the groups' FC1 stages and FC2 tiles stream through one ring per CTA in phases: phase p = FC1 of group p (a + stream-K range of the group's unit-major stages), then FC2 of group p - LAG (this CTA's 32-row output blocks, 4 + slots' slices stacked in one 128-row MMA, B row j M + t). LAG phases of streaming hide the hand-off of group + p - LAG's intermediate between CTAs. +- FC1 stage: a 32-row unit's rows at 4 consecutive 256-K tiles (rows 32 q + r) against resident B rows QS q + t. + FC2 stage: 4 slots' 32 rows at one 256-K tile. Both carry 8 KB of raw scale atoms, spliced into the 4 instruction + atoms by warp 6 with one formula (word [r][q] <- raw atom 4 q + kb, word [r][m1]). +- FC1 epilogue: k-quarters summed in order, SwiGLU-OAI, NVFP4 requant into tagged 16-byte records (3 words of 3 data + bytes + the call's tag); a unit split between two CTAs is summed by its tail's holder (Lamport slot). +- Warps 5 and 8.. (XW warps) build each group's FC2 B (tiles + scale atoms) from the records once they carry the + call's tag; warp 5 copies the scales to TMEM. +- Finalize: bf16(fc2_alpha * acc) per (slot, token) into its top-k position, then per token the fma chain over its + top-4 in order, bf16 out. With an all-reduce descriptor, every output word is also pushed to the exchange + (AR_PRE: call parities and descriptor loaded during the last group's FC2). + +Configuration (trace-time constants) via M3_CONFIG = {"m_max", "num_ctas", "stages", "nb2_cap", "fused_router", "lag", +"xw", ...}; the plan (op.make_phase_plan) must be built for the same lag. +Launched with PDL: barrier setup, TMEM allocation, tensormap prefetch, bias / plan / router weight loads before +griddepcontrol.wait; every producer-output read and every global write after it. +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +_CFG = globals().get("M3_CONFIG") or {} + + +def _cfg(key: str, default): + return type(default)(_CFG.get(key, default)) + + +# ============================================================================= +# Problem shape (trace-time constants). +# ============================================================================= +H = 6144 # capacity: fixed: the checkpoint's hidden size +# routed intermediate per rank: 1536 at TP2 (this target: FC2 K = 6 full 256-K tiles), 768 at TP4, 384 at TP8 (FC2 K +# = 1.5 256-K tiles: the last tile's A box is half out of bounds, zero-filled by TMA, and only its first KB2_LAST K-64 +# MMAs / scale atoms are issued; no padding, no extra bytes) +I_TP = _cfg("i_tp", 768) +TWO_I = 2 * I_TP +assert I_TP % 128 == 0 and 128 <= I_TP <= 1536, I_TP +E_LOCAL = 128 +TOP_K = 4 # capacity: fixed: the checkpoint's top-k +ROUTE_SCALE = 2.0 +SUM_EPS = 1e-20 +M_MAX = _cfg("m_max", 1) +NCTA = _cfg("num_ctas", 152) +FUSED_ROUTER = bool(_cfg("fused_router", 0)) +NROUTE = E_LOCAL if FUSED_ROUTER else 0 # experts whose logits the router GEMV computes +NGEMV = (NROUTE + NCTA - 1) // NCTA # experts per CTA: bx, bx + NCTA, .. +TIMING = bool(_cfg("timing", 0)) +HANG_DBG = bool(_cfg("hang_dbg", 0)) +XVDUMP = bool( + _cfg("xvdump", 0) +) # debug only (needs a plan without split units): FC1 sums into `partials` +YDUMP = bool( + _cfg("ydump", 0) +) # debug only: FC2 outputs per (token, k) into `partials` (breaks its protocol) +LAG = _cfg("lag", 2) # phase p runs FC1 of group p and FC2 of group p - LAG +G_CAP = TOP_K * M_MAX # distinct experts per call +NG_CAP = M_MAX # groups of 4 slots +XW = _cfg("xw", 5 if M_MAX > 4 else (4 if M_MAX > 2 else 1)) # FC2 B builder warps: 5, then 8 .. +THREADS = 256 + 32 * (XW - 1) +GEMV_THREADS = 256 # capacity: fixed: the CTA size +# SHARED: the MoE's shared expert (MXFP8 gate_up [1536, 6144] -> SwiGLU-OAI -> MXFP8 down [6144, 768]) on NSH more +# CTAs (warps 0-3), its output added in finalize: out = bf16(shared + routed) (shared_output.add_(routed_output)). +# The math is the low-M MXFP8 GEMM's (native MXFP8 activation quant, mma.sync m16n8k32 e4m3 per 32-K block, block scales +# as one fma, its partial-sum order, bf16) and the Triton silu_and_mul's, operation for operation. Shared CTA s owns +# intermediate positions 32 s .. 32 s + 31 (gate and up rows: MMA tiles = warps 0-3) and down rows 256 s .. + 255; +# 16-row x 768-byte weight chunks (+ their scale atoms) stream through NBW buffers per warp, the first NBW before +# the grid dependency. h crosses the shared CTAs, y_sh reaches the routed CTAs' finalize (both tagged words). +SHARED = bool(_cfg("shared", 0)) +# LATE_TRIGGER: release the dependents at teardown, not after the grid dependency. An early release lets the next +# kernels' CTAs (the all-reduce, then the attention it releases at entry) become resident during this kernel, packed +# onto the SMs it leaves free, where the attention then runs on a fraction of the GPU. +LATE_TRIGGER = bool(_cfg("late_trigger", 1)) +# TAIL_TRIGGER (with LATE_TRIGGER, no all-reduce push): the CTAs that own more FC2 blocks than the rest (the first +# H / 32 mod NCTA, which end last) release the dependents right after the grid dependency, so the dependents launch +# when the other CTAs tear down, onto the SMs they leave, instead of after the last of the heavy CTAs. +TAIL_TRIGGER = bool(_cfg("tail_trigger", 1)) +NHEAVY = (H // 32) % NCTA if NCTA < H // 32 else 0 +# AR_PRE: the finalize's all-reduce push uses the call parities and descriptor fields loaded while the last group's +# FC2 runs (not two dependent global loads per push on the kernel's tail). +AR_PRE = bool(_cfg("ar_pre", 1)) +# XSTAGE (fused router): the hidden states land in the (still idle) ring by one bulk copy per token right after the +# grid dependency; the NVFP4 quantization of B1 reads them from shared memory (one HBM / L2 round trip, not ~8). +XSTAGE = bool(_cfg("xstage", 1)) and bool(_cfg("fused_router", 0)) +NSH = 24 if SHARED else 0 +# the shared intermediate per rank: 1536 (TP2), 768 (TP4), 384 (TP8). A shared CTA owns SH_ROWS intermediate +# positions: its gate_up MMA tile is SH_SPLIT K slices x (SH_ROWS gate + SH_ROWS up rows) = 128 rows (TP2: 1 x (64 + +# 64), full K in one accumulator, the order of the stock tcgen05 MXFP8 GEMM; TP4: 2 x (32 + 32), the order of +# m3_mx_gemv split 2; TP8: 4 x (16 + 16), m3_mx_gemv split 4), and 6144 / NSH = 256 down rows over the full K. +SH_I = _cfg("sh_i", 768) +SH_ROWS = SH_I // 24 +# gate_up K slices (= m3_mx_gemv's cluster ranks, contiguous K ranges) +SH_SPLIT = 128 // (2 * SH_ROWS) +SH_NB = 8 * SH_SPLIT # gate_up B rows (8 q + t: token t's slice-q k-tile) = its MMA N +SH_KB = H // 32 # capacity: fixed: a kernel tile constant (gate_up K blocks) (gate_up K blocks) +SH_DKB = SH_I // 32 # down K blocks +SH_DKT = SH_I // 128 # down k-tiles +SH_GT = SH_KB // 4 // SH_SPLIT # gate_up k-tiles (128 K) per slice +SH_STAGES_GU = SH_GT +SH_STAGES = SH_GT + 2 * SH_DKT # + 2 down tiles x SH_DKT k-tiles +SHX_BAR = 6 +# SH_PRETILE: the shared weights come pre-tiled (op._pretile_shared): per shared CTA its SH_STAGES stages, each the +# 16 KB shared-memory image of its 128-row MMA tile (128-byte swizzle) followed by the stage's 512-byte instruction +# scale atom, so a stage is two contiguous bulk copies (not 128 row segments of 128 bytes) and needs no splicing. +SH_PRETILE = bool( + _cfg("sh_pretile", 1 if M_MAX <= 2 or SH_SPLIT != 2 else 0) +) # (in-chain: a win at M <= 2, not 4..8) +if SHARED: + assert SH_I in (384, 768, 1536), SH_I + assert SH_PRETILE or SH_SPLIT == 2, ( + "the un-pretiled shared stream is the 32-row (TP4) layout only" + ) +SH_STAGE_BYTES = 16384 + 512 +MMA_M = 128 +KT_BYTES = 128 +INC_K = 2 +NUM_KB = 4 +# FC1 +N1 = 16 if M_MAX <= 4 else 32 # B rows: QS q + t (token t at the stage's k-tile 4 j + q) +QS = N1 // 4 +R1 = 32 +C1 = 4 +U1 = TWO_I // R1 # 48 units per slot +K1_TILES = H // 256 +S1 = K1_TILES // C1 # 6 stages per unit # capacity: fixed: pipeline stages +NUM_BYTES_A = MMA_M * KT_BYTES # capacity: fixed: a stage's operand bytes +NUM_BYTES_BOX1 = R1 * KT_BYTES +SF1_KA = H // 64 +SF1_RB = TWO_I // 128 +NUM_BYTES_SFA_RAW = 16 * 512 +NUM_WORDS_SFA_RAW = NUM_BYTES_SFA_RAW // 4 +NUM_BYTES_SFA = NUM_KB * 512 # capacity: fixed: a stage's scale bytes +NUM_WORDS_SFA = NUM_BYTES_SFA // 4 +NUM_BYTES_B1 = N1 * KT_BYTES +NUM_SFB1_ATOMS = S1 * NUM_KB +# FC2 +R2 = 32 +NB2 = H // R2 # 192 +KT2 = (I_TP + 255) // 256 # 3 (TP4), 2 (TP8) +# K-64 instructions of the last FC2 k-tile: 4 (TP4), 2 (TP8) +KB2_LAST = (I_TP - 256 * (KT2 - 1)) // 64 + + +def _nkb2(t2: int) -> int: + """K-64 MMA instructions (= scale atoms) of FC2 k-tile t2.""" + return NUM_KB if t2 < KT2 - 1 else KB2_LAST + + +SF2_KA = I_TP // 64 +SF2_RB = H // 128 +N2 = max(8, (4 * M_MAX + 7) // 8 * 8) # B rows j M + t +NUM_BYTES_B2 = N2 * KT_BYTES +NB2_CAP = _cfg("nb2_cap", 2) +UNITS_ROW = TWO_I // R1 # 48 records per (slot, token) row +H_ROW = UNITS_ROW * 16 +# HB_REPL: the tagged FC1 -> FC2 records written to HB_REPL copies (HB_STRIDE int32 words apart); CTA bx polls copy +# bx % HB_REPL. Every routed CTA polls every record, so with one copy the hand-off latency depends on where the few +# hot lines land in L2 (TP8 M=1: 1.5 KB of records, +-1 us across allocations); copies spread the polls. +HB_REPL = _cfg("hb_repl", 1) +assert 1 <= HB_REPL <= 32, HB_REPL # (copy i is written by lane i) +HB_STRIDE = (G_CAP * M_MAX * UNITS_ROW * 4 + 255) // 256 * 256 + 32 +PARTIAL_EMPTY = -1 # capacity: fixed: a sentinel, not a size +EPI_THREADS = 128 +EPI_BAR = 1 +ROUTE_BAR = 2 +GEMV_BAR = 3 +B2_BAR = 5 +TD_STRIDE = 64 # tdbg columns per CTA +TOPK_BAR = 4 +B1_AFTER_TOPK = bool( + _cfg("b1_after_topk", 0) +) # (experiment) the B1 build waits for the routing warps' top-k +TOPK2_BAR = 7 +EARLY0 = bool( + _cfg("early0", 1) +) # warp 4 streams group 0 (token 0's experts) right after its own top-k +EARLY0_N = _cfg("early0_n", 99) # stages issued early (capped at STAGES) +# routing warps: 4..7, then the FC2 B builder warps 8.. (idle until the routing is done) up to one warp per token +RW_EXTRA = max(0, min(M_MAX - 4, (THREADS // 32) - 8)) if bool(_cfg("rw_extra", 1)) else 0 +NRW0 = ( + min(4, M_MAX) if bool(_cfg("rw_trim", 1)) else 4 +) # warps 4.. with a token (no idle pollers in the barrier) +NRW = NRW0 + RW_EXTRA +LG_REPL = _cfg("lg_repl", 32 if M_MAX > 4 else 128) +GEMV_SMEM = bool( + _cfg("gemv_smem", 1) +) # (XSTAGE) the router GEMV reads the staged hidden states from smem +LG_STRIDE = M_MAX * E_LOCAL + 32 +POLL_NS = _cfg("poll_ns", 150) +# Hang guard: every cross-CTA spin (the kernel needs all its CTAs resident at once) traps after this long, so a lost +# co-residency or a missing producer surfaces as a device exception instead of a silent hang. The timer is read only +# inside a spin iteration; the arithmetic is unchanged. +HANG_NS = _cfg("hang_ms", 2000) * 1000000 +B2_POLL_NS = _cfg("b2_poll_ns", 200) +# B2IMG (needs SFA_CP: warp 6 is free): the FC1 epilogue writes each finished unit's FP4 bytes and scale byte straight +# into a per-group global image laid out as the FC2 B tile + its scale atoms (exactly what the builder warps assembled +# from the tagged records); per group, warp 6 joins the epilogue (fc1_done), fences the async proxy and does one +# release add of this CTA's finished units to the group counter; warp 5 acquires the counter (every group gains +# CNT_PER_CALL per call, CTA 0 pads absent slots), fences, and bulk-copies the image into the B2 buffer. +B2IMG = bool(_cfg("b2img", 0)) and bool(_cfg("sfa_cp", 0)) +IMG_B2 = KT2 * NUM_BYTES_B2 +IMG_SF = KT2 * NUM_KB * 512 +IMG_BYTES = IMG_B2 + IMG_SF +CNT_PER_CALL = 4 * (TWO_I // R1) +B2IMG_NS = _cfg("b2img_ns", 16) # (B2IMG) counter poll back-off +B2_DLY = _cfg( + "b2_dly", 0 +) # (stress) the release warp spins up to this many ns (CTA- and group-dependent) first +W_WARP = 4 # capacity: fixed: a warp role index +X_WARP = 5 # capacity: fixed: a warp role index +S_WARP = 6 # capacity: fixed: a warp role index +MMA_WARP = 7 # capacity: fixed: a warp role index +# E0_M4: at M = 4 the MMA warp routes +# token 3 (NRW0 = 4), so the early group-0 stages go to FC2 B builder warp 8 instead (idle until the routing is done; +# RW_EXTRA = 0 at M = 4). Issue order only: exact. +E0_M4 = ( + bool(_cfg("e0_m4", 1)) and NRW0 == 4 and RW_EXTRA == 0 and THREADS // 32 > 8 +) # M == 4 only: at M > 4 warp 8 routes (RW_EXTRA) +E0_WARP = ( + 8 if E0_M4 else MMA_WARP +) # (M <= 3) the early stages are issued by the idle MMA warp, not by warp 4 (routing's critical path) +E0_BAR = 8 +E1_BAR = 9 +E0_OFFLOAD = EARLY0 and (NRW0 <= 3 or E0_M4) and bool(_cfg("e0_off", 1)) +# B1_AFTER_TOPK holds warps 0..3 (the B1 build, which arrives x_bar[1] when XSTAGE) on TOPK2_BAR until every routing +# warp arrives; with EARLY0 and no E0 offload, warp 4 first waits on x_bar[1] to fill the ring before it arrives, so the +# two wait on each other. Named barriers have no HANG_NS trap, so the kernel would spin forever: refuse the config. +if B1_AFTER_TOPK and XSTAGE and EARLY0 and not E0_OFFLOAD: + raise ValueError( + "m3_moe: b1_after_topk=1 with xstage, early0 and no early-stage offload deadlocks (warp 4 waits on x_bar[1] " + "before TOPK2_BAR, the B1 build waits on TOPK2_BAR before x_bar[1]); set b1_after_topk=0 or early0=0" + ) +EVICT_FIRST = 0x12F0000000000000 +_LOG2E = 1.4426950408889634 +LD_POW2 = ( + 1 if M_MAX == 1 else 2 if M_MAX == 2 else 4 if M_MAX <= 4 else 8 +) # tcgen05.ld .num covering M columns + +_FIXED_SMEM = ( + S1 * NUM_BYTES_B1 + + 2 * KT2 * NUM_BYTES_B2 + + 2 * KT2 * NUM_KB * 512 + + NB2_CAP * M_MAX * TOP_K * R2 * 4 + + 2 * M_MAX * C1 * R1 * 4 + + (M_MAX * (H // 16) if FUSED_ROUTER else 0) + + 6 * 1024 +) +# SFA_CP: the A scale words go by 4-byte cp.async straight into the instruction atom layout (no raw scale box, no +# splicer warp; the MMA warp tcgen05.cp's them), completing on the stage's full barrier: 18 KB stages instead of 26 KB +SFA_CP = bool(_cfg("sfa_cp", 0)) +_RAW_STAGE = 0 if SFA_CP else NUM_BYTES_SFA_RAW +_STAGE_SMEM = NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA +_STAGES_FIT = (227 * 1024 - _FIXED_SMEM - (128 if SHARED else 0)) // _STAGE_SMEM +ACC1_COL = 0 +ACC2_COL = 2 * N1 +SFB1_COL = ACC2_COL + 2 * N2 +SFB2_COL = SFB1_COL + NUM_SFB1_ATOMS * 4 +SFA1_COL = SFB2_COL + 2 * KT2 * NUM_KB * 4 +STAGES = min(_cfg("stages", 12 if SFA_CP else 8), _STAGES_FIT, (512 - SFA1_COL) // 16) +TMEM_COLS = 512 +assert 1 <= M_MAX <= 8, M_MAX +assert STAGES >= 3, (STAGES, _STAGES_FIT, SFA1_COL) +# TP2 at 8 tokens: the 98 KB of hidden states fit no 5-stage ring; the router GEMV then reads them from global (the +# kernel's non-XSTAGE path, same math) +if XSTAGE and M_MAX * H * 2 > STAGES * (NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA): + XSTAGE = False +assert not XSTAGE or M_MAX * H * 2 <= STAGES * (NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA), ( + "x staging must fit the ring" +) +# one shared-memory region for the ring and the FC1 / FC2 B operands (a shared CTA reuses all of it) +_OFF_SFAR = STAGES * NUM_BYTES_A +_OFF_SFA = _OFF_SFAR + STAGES * _RAW_STAGE +_OFF_B1 = _OFF_SFA + STAGES * NUM_BYTES_SFA +_OFF_B2 = _OFF_B1 + S1 * NUM_BYTES_B1 +_OFF_SFB2 = _OFF_B2 + 2 * KT2 * NUM_BYTES_B2 +BIG = _OFF_SFB2 + 2 * KT2 * NUM_KB * 512 +# shared CTA layout in BIG: weight buffers, x quarters (2), x scales [M, 192], gate / up [4, 16, M], h, h scales +# shared CTA layout in BIG: A ring (16 KB stages), raw scale atoms (4 per stage), spliced atoms, B (x: [stage][16 rows] +# [128 B]; h: [6][8][128 B]), x / h scale codes, gate_up partial exchange +_SB_A = 0 +_SB_RAW = None +# slices 1.. gate / up partials [slice - 1][u][p][8 tokens] f32 +_SH_XCH_BYTES = (SH_SPLIT - 1) * 2 * SH_ROWS * 8 * 4 +_SH_FIX = ( + SH_GT * SH_NB * 128 + + SH_DKT * 8 * 128 + + M_MAX * SH_KB + + M_MAX * SH_DKB + + _SH_XCH_BYTES + + 2 * SH_ROWS * 8 * 4 + + 1024 +) +SH_NST = min(8, (BIG - _SH_FIX) // (NUM_BYTES_A + 4 * 512 + 512)) if SHARED else 1 +_SB_RAW = SH_NST * NUM_BYTES_A +_SB_SPL = _SB_RAW + SH_NST * 2048 +_SB_BX = (_SB_SPL + SH_NST * 512 + 1023) // 1024 * 1024 +_SB_BH = _SB_BX + SH_GT * SH_NB * 128 +_SB_XC = _SB_BH + SH_DKT * 8 * 128 +_SB_HC = _SB_XC + M_MAX * SH_KB +_SB_XCH = (_SB_HC + M_MAX * SH_DKB + 15) // 16 * 16 +_SB_GU = _SB_XCH + _SH_XCH_BYTES +_SB_END = _SB_GU + 2 * SH_ROWS * 8 * 4 +# shared CTA TMEM columns: gate_up accumulator (N SH_NB), down accumulators (2 x N 8), A scales per stage, B scales +SH_ACC_GU = 0 +SH_ACC_DN = SH_NB +SH_SFA = SH_ACC_DN + 16 +SH_SFB_GU = SH_SFA + 4 * 8 +SH_SFB_DN = SH_SFB_GU + 4 * SH_GT +# pre-tiled stages beyond the ring that warp 0 prefetches into L2, per lane (TP2: 72 stages) +SH_PF = max(2, (SH_STAGES - SH_NST + 31) // 32) if SHARED else 2 +# TP2 (SH_GT 48 gate_up k-tiles, SH_DKT 12) needs more than 256 columns: allocate the next power of two +SH_TMEM = 256 if SH_SFB_DN + 4 * SH_DKT <= 256 else 512 +if SHARED: + assert FUSED_ROUTER, "shared expert: fused router only" + assert SH_STAGES - SH_NST <= 32 * SH_PF, SH_STAGES + assert SH_NST >= 3 and _SB_END <= BIG and SH_SFB_DN + 4 * SH_DKT <= SH_TMEM, ( + SH_NST, + _SB_END, + BIG, + ) + + +# ============================================================================= +# Inline-PTX helpers. +# ============================================================================= +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +# ---- all-reduce push (the MoE boundary's multicast exchange, set A): +# arp = int64 [enable, mc, flags, rank, world, maxt] +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32_if(addr, value, pred, *, loc=None, ip=None): + """st.global.b32 when pred != 0.""" + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip), + cutlass.Int32(pred).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, $2, 0;\n\t@p st.global.b32 [$0], $1;\n\t}", "l,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _ar_push(arp, v, low, t, cl): + """Push bf16 channel pair (cl, cl + 1) of token t (v = channel cl's bf16 value on this lane, its neighbour's on + lane ^ 8) into this rank's set-A slot of the exchange; the slot half is the all-reduce CTA (t, cl // 768)'s call + parity. Only lanes with ``low`` store; every lane of the warp must call it (shuffle).""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + world = arp.load(idx=4) + parity = cutlass.Int64( + _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cs // cutlass.Int32(H // 8)) + ) + & cutlass.Int32(1) + ) + off = ( + parity * arp.load(idx=5) * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + arp.load(idx=3)) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(arp.load(idx=1) + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +def _ar_parity(arp, t, cl): + """The call parity _ar_push reads for token t's channel cl (fixed between this kernel's grid dependency and the + next all-reduce, which this kernel releases at teardown): loaded ahead of the finalize (AR_PRE).""" + return _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cl // cutlass.Int32(H // 8)) + ) + + +def _ar_push_pre(ard, parity32, v, low, t, cl): + """_ar_push with the parity (_ar_parity) and the descriptor fields ard = (mc, rank, world, maxt) preloaded.""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + mc, rk, world, maxt = ard + parity = cutlass.Int64(parity32 & cutlass.Int32(1)) + off = ( + parity * maxt * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + rk) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(mc + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +@dsl_user_op +def _bar_arrive(bar_id: int, threads: int, *, loc=None, ip=None): + """bar.arrive: signal a named barrier without waiting (its memory ordering is bar.sync's).""" + _llvm.inline_asm( + None, + [], + f"bar.arrive {bar_id}, {threads};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _st_relaxed_u32(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "st.relaxed.gpu.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _st_relaxed_u64(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int64(val)], + "st.relaxed.gpu.global.u64 [$0], $1;", + "l,l", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm( + _T.i64(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u64 $0, [$1];", + "=l,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _nanosleep(ns: int, *, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + f"nanosleep.u32 {ns};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _gtime(*, loc=None, ip=None): + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), + [], + "mov.u64 $0, %globaltimer;", + "=l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _trap_after(elapsed, *, loc=None, ip=None): + """trap when elapsed (ns, int64) exceeds HANG_NS.""" + _llvm.inline_asm( + None, + [cutlass.Int64(elapsed).ir_value(loc=loc, ip=ip)], + f"{{ .reg .pred p; setp.gt.s64 p, $0, {HANG_NS}; @p trap; }}", + "l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +def _hang_tick(t0): + """One spin iteration of a cross-CTA wait: the wait's start (latched on the first call; t0 = 0 before it), after + trapping once the wait has lasted longer than HANG_NS.""" + now = _gtime() + t1 = cutlass.select_(t0 == cutlass.Int64(0), now, t0) + _trap_after(now - t1) + return t1 + + +@dsl_user_op +def _smid(*, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [], + "mov.u32 $0, %smid;", + "=r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@cute.jit +def _stamp(tdbg: cutlass.Array, bx, i: cutlass.Constexpr): + if cutlass.const_expr(TIMING): + tdbg.store(_gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(i)) + + +@dsl_user_op +def _rcp(a, *, loc=None, ip=None): + """rcp.approx.ftz.f32 (the reciprocal TRT-LLM's FP4 quantization uses).""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _div_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _sub_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "sub.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _mul_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3_bits(x, *, loc=None, ip=None): + """e4m3 (round to nearest even, saturating) of x as an int 0..255.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, 0f00000000, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3_to_f32(b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(b)], + "{\n.reg .b16 t, l, u;\n.reg .b32 h;\ncvt.u16.u32 t, $1;\ncvt.rn.f16x2.e4m3x2 h, t;\nmov.b32 {l, u}, h;\n" + "cvt.f32.f16 $0, l;\n}", + "=f,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e2m1x2(hi, lo, *, loc=None, ip=None): + """One byte of two E2M1 values (lo in the low nibble), round to nearest even, saturating.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b8 t;\ncvt.rn.satfinite.e2m1x2.f32 t, $1, $2;\ncvt.u32.u8 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2(x, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _bf16_round(x, *, loc=None, ip=None): + """x rounded to bf16 (nearest even), as f32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\nmov.b32 $0, {0, t};\n}", + "=f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _redux_max_u32(v, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(v)], + "redux.sync.max.u32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16 (nearest even) bits of x in the low 16 bits.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ue8m0_rp(x, *, loc=None, ip=None): + """UE8M0 code of x rounded toward +inf, saturating (__nv_cvt_float_to_e8m0(x, SATFINITE, RoundPosInf)).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rp.satfinite.ue8m0x2.f32 t, $1, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3x2(hi, lo, *, loc=None, ip=None): + """Two e4m3 (nearest even, saturating) in 16 bits, lo in the low byte.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, $1, $2;\ncvt.u32.u16 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _min_xorsign_abs(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "min.xorsign.abs.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _div_full(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.full.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2_full(x, *, loc=None, ip=None): + """ex2.approx.f32 (no flush to zero), Triton's tl.exp.""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _l2_prefetch(addr, nbytes, *, loc=None, ip=None): + """cp.async.bulk.prefetch.L2 of nbytes (a multiple of 16) at global address addr (16-byte aligned).""" + _llvm.inline_asm( + None, + [ + cutlass.Int64(addr).ir_value(loc=loc, ip=ip), + cutlass.Int32(nbytes).ir_value(loc=loc, ip=ip), + ], + "cp.async.bulk.prefetch.L2.global [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +def _mx_osc(amax): + """mxfp8 quantization of a block with absolute max amax: (UE8M0 code, the element multiplier).""" + code = _ue8m0_rp(_mul_rn(amax, _rcp(cutlass.Float32(448.0)))) + cf = cutlass.Int32( + cutlass.select_( + code == cutlass.Int32(255), + cutlass.Int32(0x7FFFFFFF), + cutlass.select_( + code == cutlass.Int32(0), cutlass.Int32(0x00400000), code << cutlass.Int32(23) + ), + ) + ).bitcast(cutlass.Float32) + return code, cutlass.select_(amax != cutlass.Float32(0.0), _rcp(cf), cutlass.Float32(0.0)) + + +def _mx_words(v, osc): + """e4m3 words of len(v) (a multiple of 4) values times osc, in element order.""" + words = [] + for t in range(len(v) // 4): + lo = _e4m3x2(_mul_rn(v[4 * t + 1], osc), _mul_rn(v[4 * t], osc)) + hi = _e4m3x2(_mul_rn(v[4 * t + 3], osc), _mul_rn(v[4 * t + 2], osc)) + words.append(lo | (hi << cutlass.Int32(16))) + return words + + +def _twiddle(score): + """Float bits mapped so that unsigned order is float order (cub TwiddleIn).""" + bits = ( + score.bitcast(cutlass.Int32) + if hasattr(score, "bitcast") + else cutlass.Float32(score).bitcast(cutlass.Int32) + ) + return cutlass.select_(bits < cutlass.Int32(0), ~bits, bits | cutlass.Int32(-2147483648)) + + +@cute.jit +def _wait(bar: cutlass.Array, parity): + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +def _tptr(base_row, col, dtype=cutlass.Int32): + return cutlass.inttoptr((base_row << 16) | col, 6, dtype) + + +@dsl_user_op +def _match_any(value, *, loc=None, ip=None): + """The mask of the warp's lanes holding the same 32-bit value.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(value)], + "match.any.sync.b32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@cute.jit +def _waitd(bar: cutlass.Array, parity, tdbg: cutlass.Array, site: cutlass.Constexpr): + """_wait, or (HANG_DBG) give up after ~2^22 polls and record the site in tdbg[bx, 24 + warp].""" + if cutlass.const_expr(HANG_DBG): + n = cutlass.Int32(0) + while (not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity)) & ( + n < cutlass.Int32(1 << 22) + ): + n = n + cutlass.Int32(1) + if n >= cutlass.Int32(1 << 22): + bx, _, _ = cute.arch.block_idx() + w = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + tdbg.store( + cutlass.Int64(site), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + w + ) + else: + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +@cute.jit +def _adv(stage, phase): + """The next ring position's (stage, phase).""" + stage = stage + cutlass.Int32(1) + wrap = stage == cutlass.Int32(STAGES) + return cutlass.select_(wrap, cutlass.Int32(0), stage), cutlass.select_( + wrap, phase ^ cutlass.Int32(1), phase + ) + + +@cute.jit +def _seg(i0, i1, k): + """Segment k of the FC1 item range [i0, i1) (items = unit-major stages): (unit, first stage, end stage). Order: + the last unit (its head, the part a split unit's next CTA does not cover), then the first unit, then the rest.""" + uf = i0 // cutlass.Int32(S1) + ul = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) + unit = uf + k - cutlass.Int32(1) + unit = cutlass.select_(k == cutlass.Int32(0), ul, unit) + unit = cutlass.select_(ul == uf, uf, unit) + j0 = i0 - unit * cutlass.Int32(S1) + j0 = cutlass.select_(j0 < cutlass.Int32(0), cutlass.Int32(0), j0) + j1 = i1 - unit * cutlass.Int32(S1) + j1 = cutlass.select_(j1 > cutlass.Int32(S1), cutlass.Int32(S1), j1) + return unit, j0, j1 + + +@cute.jit +def _nsegs(i0, i1): + n = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) - i0 // cutlass.Int32(S1) + cutlass.Int32(1) + return cutlass.select_(i1 > i0, n, cutlass.Int32(0)) + + +@cute.jit +def _sfb1_to_tmem(warp, lane, base_row, base_col, src: cutlass.Array, b1_ready: cutlass.Array): + """B1 scales to TMEM: instruction (j, kb)'s column holds, at lane n = QS q + t, token t's 4 scales of k-tile + 4 j + q, K 64 kb .. (src: int32 words [M, 96] of the token scales); each warp writes its sub-partition.""" + q = lane // cutlass.Int32(QS) + tl = lane - q * cutlass.Int32(QS) + sf_ok = (lane < cutlass.Int32(N1)) & (tl < cutlass.Int32(M_MAX)) + sfv = [] + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + j = a // NUM_KB + kb = a % NUM_KB + wsf = src.load( + idx=cutlass.select_( + sf_ok, + tl * cutlass.Int32(H // 64) + + (cutlass.Int32(4 * j) + q) * cutlass.Int32(4) + + cutlass.Int32(kb), + cutlass.Int32(0), + ) + ) + sfv.append(cutlass.select_(sf_ok, cutlass.Int32(wsf), cutlass.Int32(0))) + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + prims.tcgen05_st( + "32x32b", + _tptr(base_row + warp * cutlass.Int32(32), base_col + SFB1_COL + a * 4), + sfv[a], + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(b1_ready) + + +@cute.jit +def _b1_addr(t, kb16): + """Word index in the resident B tiles of the 8 FP4 bytes of token t's 16-value block kb16.""" + kt = kb16 >> cutlass.Int32(4) + j = kt >> cutlass.Int32(2) + n = (kt & cutlass.Int32(3)) * cutlass.Int32(QS) + t + cc = (kb16 >> cutlass.Int32(1)) & cutlass.Int32(7) + return ( + j * cutlass.Int32(NUM_BYTES_B1) + + n * cutlass.Int32(KT_BYTES) + + (cc ^ (n & cutlass.Int32(7))) * cutlass.Int32(16) + + (kb16 & cutlass.Int32(1)) * cutlass.Int32(8) + ) // cutlass.Int32(4) + + +@cute.jit +def _build_b1( + tidx, + warp, + lane, + base_row, + base_col, + x32: cutlass.Array, + xsf32: cutlass.Array, + sB1: cutlass.Array, + b1_ready: cutlass.Array, +): + """B (the M tokens' whole K) into the resident tiles from the NVFP4 activation, its scales to TMEM.""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + for r in cutlass.range_constexpr((M_MAX * (H // 16) + EPI_THREADS - 1) // EPI_THREADS): + b = tidx + cutlass.Int32(r * EPI_THREADS) + if b < cutlass.Int32(M_MAX * (H // 16)): + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + v2 = x32.load(idx=b * cutlass.Int32(2), vector_size=2, alignment=8) + sB1_32.store( + (cutlass.Int32(v2[0]), cutlass.Int32(v2[1])), idx=_b1_addr(t, kb), alignment=8 + ) + _sfb1_to_tmem(warp, lane, base_row, base_col, xsf32, b1_ready) + + +@cute.jit +def _build_b1_fused( + tidx, + warp, + lane, + base_row, + base_col, + xb32: cutlass.Array, + sfsc: cutlass.Array, + sB1: cutlass.Array, + s_xsf: cutlass.Array, + b1_ready: cutlass.Array, +): + """_build_b1 from the bf16 hidden states, quantized exactly as fp4_quantize(x, sf_scale, 16).""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + s_xsf8 = cutlass.Array( + s_xsf.data_ptr(0), shape=(M_MAX * (H // 16),), dtype=cutlass.Int8, alignment=16 + ) + sc = cutlass.Float32(sfsc.load(idx=0)) + rsc = _rcp(sc) + rsix = _rcp(cutlass.Float32(6.0)) + NBLK = M_MAX * (H // 16) // EPI_THREADS + QB = ( + 3 if NBLK % 3 == 0 else (2 if NBLK % 2 == 0 else 1) + ) # blocks whose loads are in flight together + # a rolled loop over groups of QB blocks (a fully unrolled one is ~50 KB of SASS at M = 8: instruction fetch bound) + for rg in cutlass.range(NBLK // QB, unroll=1): + wall = [] + for rr in cutlass.range_constexpr(QB): + bq = tidx + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(EPI_THREADS) + for h in cutlass.range_constexpr(2): + wall.append( + xb32.load( + idx=bq * cutlass.Int32(8) + cutlass.Int32(4 * h), + vector_size=4, + alignment=16, + ) + ) + for rr in cutlass.range_constexpr(QB): + b = tidx + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(EPI_THREADS) + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + w = [] + for h in cutlass.range_constexpr(2): + v4 = wall[rr * 2 + h] + for z in cutlass.range_constexpr(4): + w.append(cutlass.Int32(v4[z])) + xs = [] + for z in cutlass.range_constexpr(8): + xs.append((w[z] << cutlass.Int32(16)).bitcast(cutlass.Float32)) + xs.append((w[z] & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + vmax = cute.math.abs(xs[0]) + for z in cutlass.range_constexpr(1, 16): + vmax = cute.arch.fmax(vmax, cute.math.abs(xs[z])) + sfb = _e4m3_bits(_mul_rn(sc, _mul_rn(vmax, rsix))) + osc = cutlass.select_( + vmax != cutlass.Float32(0.0), + _rcp(_mul_rn(_e4m3_to_f32(sfb), rsc)), + cutlass.Float32(0.0), + ) + by = [ + _e2m1x2(_mul_rn(xs[2 * z + 1], osc), _mul_rn(xs[2 * z], osc)) & cutlass.Int32(0xFF) + for z in range(8) + ] + lo = ( + by[0] + | (by[1] << cutlass.Int32(8)) + | (by[2] << cutlass.Int32(16)) + | (by[3] << cutlass.Int32(24)) + ) + hi = ( + by[4] + | (by[5] << cutlass.Int32(8)) + | (by[6] << cutlass.Int32(16)) + | (by[7] << cutlass.Int32(24)) + ) + sB1_32.store((lo, hi), idx=_b1_addr(t, kb), alignment=8) + s_xsf8.store(cutlass.Int8(sfb), idx=b) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + _sfb1_to_tmem(warp, lane, base_row, base_col, s_xsf, b1_ready) + + +@cute.jit +def _swiglu_quant(xv, gate_raw, is_up, a, c, alpha, beta, lim): + """TRTLLM-Gen's FC1 epilogue (the swiGlu Bmm E2m1 output), operation for operation: lane value xv is the up row, + gate_raw its gate row (fp32 sums). Returns (q, the scale's e4m3 bits) of the 16-row NVFP4 block (lanes r, r ^ 16 + and r ^ 8 cover it; amax over the warp's up lanes of a half). a = fc31_alpha, c = fc31_scale_c.""" + neg_max = cutlass.Float32(-3.40282346638528859812e38) + gc = cutlass.select_(gate_raw > lim, lim, gate_raw) + gc = cutlass.select_(gate_raw < neg_max, neg_max, gc) + uc = cutlass.select_(xv > lim, lim, xv) + uc = cutlass.select_(xv < -lim, -lim, uc) + c1 = _mul_rn(a, _mul_rn(alpha, cutlass.Float32(1.4426950216293334961))) + sgm = _rcp(_add_rn(_ex2(-_mul_rn(gc, c1)), cutlass.Float32(1.0))) + gs = _mul_rn(gc, sgm) + h = _mul_rn(_fma_rn(uc, a, _mul_rn(a, beta)), gs) + h = cutlass.select_(is_up, h, cutlass.Float32(0.0)) + amax = cute.arch.fmax(_mul_rn(cute.math.abs(h), c), cutlass.Float32(0.0)) + for off in (16, 8, 4, 2, 1): + amax = cute.arch.fmax(amax, cute.arch.shuffle_sync_bfly(amax, off)) + sfb = _e4m3_bits(_mul_rn(amax, cutlass.Float32(0.16666667163372039795))) + sff = _e4m3_to_f32(sfb) + osc = cutlass.select_(sff != cutlass.Float32(0.0), _rcp(sff), cutlass.Float32(0.0)) + return _mul_rn(_mul_rn(h, osc), c), sfb + + +@cute.jit +def _plan_range(s_plan: cutlass.Array, p): + w = s_plan.load(idx=p) + return w & cutlass.Int32(0xFFFF), w >> cutlass.Int32(16) + + +@cute.jit +def _sh_issue( + lane, + sid, + g, + s_big: cutlass.Array, + sh_full: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, +): + """Stage g of the shared CTA's stream into ring slot g % SH_NST (warp 0). g < SH_GT: gate_up k-tile g of both + slices: quadrant Q = 2 q + u (q slice, u 0 gate / 1 up) = rows 32 sid .. (+ 768 for up) at k-tile SH_GT q + g, its + raw scale atom at raw slot Q. Else down tile j = (g - SH_GT) / 6 (rows 256 sid + 128 j ..), k-tile + (g - SH_GT) % 6.""" + st = g % cutlass.Int32(SH_NST) + bar = sh_full.subview(st) + if cutlass.const_expr(SH_PRETILE): + if lane == 0: + src = (sid * cutlass.Int32(SH_STAGES) + g) * cutlass.Int32(SH_STAGE_BYTES) + prims.mbarrier_arrive_expect_tx(bar, SH_STAGE_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), gusf.subview(src), bar, 16384 + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + gusf.subview(src + cutlass.Int32(16384)), + bar, + 512, + ) + cute.arch.sync_warp() + else: + if g < cutlass.Int32(SH_GT): + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 4 * 512) + cute.arch.sync_warp() + if lane < 4: + q = lane >> cutlass.Int32(1) + u = lane & cutlass.Int32(1) + row0 = sid * cutlass.Int32(32) + u * cutlass.Int32(SH_I) + kt = q * cutlass.Int32(SH_GT) + g + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A) + lane * cutlass.Int32(4096)), + tma_gu, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview( + cutlass.Int32(_SB_RAW) + + st * cutlass.Int32(2048) + + lane * cutlass.Int32(512) + ), + gusf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_KB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(SH_DKT) + kt = gd - jt * cutlass.Int32(SH_DKT) + row0 = sid * cutlass.Int32(256) + jt * cutlass.Int32(128) + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 512) + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), + tma_dn, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_RAW) + st * cutlass.Int32(2048)), + dnsf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_DKB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + cute.arch.sync_warp() + + +@dsl_user_op +def _red_release_add(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "red.release.gpu.global.add.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_acquire_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.acquire.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fence_proxy_async_global(*, loc=None, ip=None): + _asm(None, [], "fence.proxy.async.global;", "", True, loc, ip) + + +@cute.jit +def _sfa_to_tmem(sSFA: cutlass.Array, stage, base_row, base_col): + """(SFA_CP, elected MMA thread) the stage's 4 scale atoms smem -> TMEM (what the splicer warp does otherwise).""" + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SFA1_COL + stage * cutlass.Int32(16) + kb * 4), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + + +@cute.jit +def _sf1_cp( + lane, stage, sf_line, c, sSFA: cutlass.Array, s1w: cutlass.Array, ab_full: cutlass.Array +): + """(SFA_CP) FC1 stage scale atoms: word [r][q] of atom kb <- raw atom 4 q + kb (raw line sf_line = 4 x atom) + word [r][c % 4] (what the splicer copies out of the raw box), 4-byte cp.async by the warp's 32 lanes; their + arrivals (noinc) complete the stage's full barrier together with the A boxes' bytes.""" + src0 = ( + (sf_line >> cutlass.Int32(2)) * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (c & cutlass.Int32(3)) + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for q in cutlass.range_constexpr(C1): + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + q)), + s1w.subview(src0 + cutlass.Int32((4 * q + kb) * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _sf2_cp( + lane, + stage, + g, + blk, + t2: cutlass.Constexpr, + n_here, + s_el: cutlass.Array, + sSFA: cutlass.Array, + s2w: cutlass.Array, + ab_full: cutlass.Array, +): + """(SFA_CP) FC2 k-tile t2 scale atoms: word [r][jj] of atom kb <- raw atom (rows / 128, K block 4 t2 + kb) of slot + jj's expert, word [r][blk % 4].""" + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + src0 = ( + ( + (e * cutlass.Int32(SF2_RB) + (blk >> cutlass.Int32(2))) * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (blk & cutlass.Int32(3)) + ) + # (a partial last k-tile has no atoms past I_TP) + for kb in cutlass.range_constexpr(_nkb2(t2)): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + jj)), + s2w.subview(src0 + cutlass.Int32(kb * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _shared_cta( + tidx, + warp, + lane, + sid, + s_big: cutlass.Array, + sh_bar: cutlass.Array, + epochs: cutlass.Array, + tdbg: cutlass.Array, + bx, + xb32: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, + tmem_ptr_i32: cutlass.Array, + gu8: cutlass.Array, + dn8: cutlass.Array, +): + """Shared CTA sid (see SHARED): tcgen05 MXF8 (E4M3 x E4M3, UE8M0 per 32 K), m3_mx_gemv's tile order. gate_up: one + 128-row MMA tile = 2 K slices x (32 gate + 32 up rows), B rows 8 q + t = token t's slice-q k-tile; down: 2 tiles + of 128 rows, full K. Warps: 0 weights (TMA), 1 TMEM + scale splice + MMA, 4-7 epilogue; 2.. activation quant.""" + big32 = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Int32, alignment=16) + bigf = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Float32, alignment=16) + big8 = cutlass.Array(s_big.data_ptr(0), shape=(BIG,), dtype=cutlass.Int8, alignment=16) + sh_full = sh_bar # [0, SH_NST): stage full + sh_empty = cutlass.Array(sh_bar.data_ptr(8), shape=(8,), dtype=cutlass.Int64, alignment=8) + b_gu_rdy = sh_bar.subview(16) + b_dn_rdy = sh_bar.subview(17) + acc_gu = sh_bar.subview(18) + acc_dn = sh_bar.subview(19) + if warp == 0: + prims.prefetch_tensormap(tma_gu) + prims.prefetch_tensormap(tma_dn) + if lane < SH_NST: + prims.mbarrier_init(sh_full.subview(lane), 1) + prims.mbarrier_init( + sh_empty.subview(lane), 33 + ) # the MMAs' commit + each splicer lane's reads done + if lane == 0: + prims.mbarrier_init(b_gu_rdy, 1) + prims.mbarrier_init(b_dn_rdy, 1) + prims.mbarrier_init(acc_gu, 1) + prims.mbarrier_init(acc_dn, 1) + if warp == 1: + prims.tcgen05_alloc(tmem_ptr_i32, SH_TMEM) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + if warp == 0: + for g in cutlass.range_constexpr(SH_NST): # layer constants: before the grid dependency + _sh_issue(lane, sid, cutlass.Int32(g), s_big, sh_full, tma_gu, tma_dn, gusf, dnsf) + # the rest of this CTA's weights into L2 (64 gate / up rows of 6 KB, 256 down rows of 768 B) + if cutlass.const_expr(SH_PRETILE): + # this CTA's stages beyond the ring, 4 per lane + for r in cutlass.range_constexpr(SH_PF): + pg = cutlass.Int32(SH_NST) + lane * cutlass.Int32(SH_PF) + cutlass.Int32(r) + if pg < cutlass.Int32(SH_STAGES): + _l2_prefetch( + gusf.subview( + (sid * cutlass.Int32(SH_STAGES) + pg) * cutlass.Int32(SH_STAGE_BYTES) + ) + .data_ptr() + .toint(), + SH_STAGE_BYTES, + ) + else: + for r in cutlass.range_constexpr(2): + prw = sid * cutlass.Int32(32) + cutlass.Int32(r * SH_I) + lane + _l2_prefetch(gu8.subview(prw * cutlass.Int32(H)).data_ptr().toint(), H) + if lane == 0: + _l2_prefetch( + dn8.subview(sid * cutlass.Int32(256 * SH_I)).data_ptr().toint(), 256 * SH_I + ) + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(not LATE_TRIGGER): + if arp.load(idx=0) == cutlass.Int64(0): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + NQ = THREADS - 64 # activation quant threads (warps 2 ..) + if warp == 0: + for g in cutlass.range(SH_NST, SH_STAGES, unroll=1): + _wait( + sh_empty.subview(g % cutlass.Int32(SH_NST)), + ((g // cutlass.Int32(SH_NST)) - cutlass.Int32(1)) & cutlass.Int32(1), + ) + _sh_issue(lane, sid, g, s_big, sh_full, tma_gu, tma_dn, gusf, dnsf) + elif warp == 1: + # ---- MMA: gate_up (scale atoms spliced to the tile's row order), then down + idesc_gu = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=SH_NB, + m_dim=128, + ) + idesc_dn = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=8, + m_dim=128, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + desc_a0 = prims.Tcgen05SmemDesc.build( + s_big.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bx = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BX), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bh = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BH), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + sq = sid & cutlass.Int32(3) + _wait(b_gu_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for g in cutlass.range(SH_STAGES, unroll=1): + st = g % cutlass.Int32(SH_NST) + if g == cutlass.Int32(SH_GT): + _wait(b_dn_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + _wait(sh_full.subview(st), (g // cutlass.Int32(SH_NST)) & cutlass.Int32(1)) + raw = cutlass.Int32(_SB_RAW // 4) + st * cutlass.Int32(512) + if (g < cutlass.Int32(SH_GT)) & cutlass.Boolean(not SH_PRETILE): + # MMA row 32 Q + i = quadrant Q's row i: word 4 i + Q <- raw atom Q's word 4 i + sid % 4 + w4 = [ + big32.load(idx=raw + cutlass.Int32(128 * qq) + lane * cutlass.Int32(4) + sq) + for qq in range(4) + ] + big32.store( + ( + cutlass.Int32(w4[0]), + cutlass.Int32(w4[1]), + cutlass.Int32(w4[2]), + cutlass.Int32(w4[3]), + ), + idx=cutlass.Int32(_SB_SPL // 4) + + st * cutlass.Int32(128) + + lane * cutlass.Int32(4), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + prims.mbarrier_arrive( + sh_empty.subview(st) + ) # this lane's generic reads of the slot are done + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + if g < cutlass.Int32(SH_GT): + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bx + cutlass.Int64(g * cutlass.Int32(SH_NB * 128 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr(tcol + SH_ACC_GU, 6, cutlass.Float32), + da + 2 * kb, + db + 2 * kb, + idesc_gu.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (g > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_GU + g * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_GT - 1): + prims.tcgen05_commit(acc_gu) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(SH_DKT) + kt = gd - jt * cutlass.Int32(SH_DKT) + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview( + cutlass.Int32(_SB_SPL if SH_PRETILE else _SB_RAW) + + st * cutlass.Int32(512 if SH_PRETILE else 2048) + ), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bh + cutlass.Int64(kt * cutlass.Int32(1024 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr( + tcol + SH_ACC_DN + jt * cutlass.Int32(8), 6, cutlass.Float32 + ), + da + 2 * kb, + db + 2 * kb, + idesc_dn.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (kt > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_DN + kt * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_STAGES - 1): + prims.tcgen05_commit(acc_dn) + cute.arch.sync_warp() + if warp >= 2: + # ---- the activation, MXFP8 per 32-K block (native mxfp8_quantize), into B (row 8 q + t of stage tile i = + # token t's k-tile SH_GT q + i, 128-byte swizzle) and its codes + qt = tidx - cutlass.Int32(64) + for r in cutlass.range_constexpr((M_MAX * SH_KB + NQ - 1) // NQ): + q_ = qt + cutlass.Int32(r * NQ) + if q_ < cutlass.Int32(M_MAX * SH_KB): + t = q_ // cutlass.Int32(SH_KB) + b = q_ - t * cutlass.Int32(SH_KB) + wv = [ + xb32.load( + idx=t * cutlass.Int32(H // 2) + + b * cutlass.Int32(16) + + cutlass.Int32(4 * z), + vector_size=4, + alignment=16, + ) + for z in range(4) + ] + v = [] + for z in cutlass.range_constexpr(4): + for zz in cutlass.range_constexpr(4): + w = cutlass.Int32(wv[z][zz]) + v.append((w << cutlass.Int32(16)).bitcast(cutlass.Float32)) + v.append((w & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(v[z])) + code, osc = _mx_osc(amax) + words = _mx_words(v, osc) + kt = b >> cutlass.Int32(2) + sl = kt // cutlass.Int32(SH_GT) + it = kt - sl * cutlass.Int32(SH_GT) + n = sl * cutlass.Int32(8) + t + c0 = (b & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BX) + it * cutlass.Int32(SH_NB * 128) + n * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + + (((c0 + cutlass.Int32(1)) ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_XC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR, number_of_threads=NQ) + if warp >= 4: + if warp < 8: + qd = warp - cutlass.Int32(4) + # B scales of gate_up to TMEM: column of stage i, lane n = 8 q + t: token t's codes of k-tile SH_GT q + i + n = lane + t = n & cutlass.Int32(7) + ok = (n < cutlass.Int32(SH_NB)) & (t < cutlass.Int32(M_MAX)) + for i in cutlass.range_constexpr(SH_GT): + kt = (n >> cutlass.Int32(3)) * cutlass.Int32(SH_GT) + cutlass.Int32(i) + wsf = big32.load( + idx=( + cutlass.Int32(_SB_XC) + + cutlass.select_(ok, t, cutlass.Int32(0)) * cutlass.Int32(SH_KB) + + kt * cutlass.Int32(4) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_GU + i * 4), + cutlass.select_(ok, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + et = tidx - cutlass.Int32(128) + if et == 0: + prims.mbarrier_arrive(b_gu_rdy) + _stamp(tdbg, bx, 54) + # ---- gate_up out: quadrant qd lane l = MMA row 32 qd + l; slice q = qd / 2 holds token t at column 8 q + t + _wait(acc_gu, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if cutlass.const_expr(SH_SPLIT == 1): + # TP2: one K slice; quadrant qd = rows 32 qd .. (gate positions 0..63, then up 0..63), no partials: + # the owner sum is (0 + acc), then bf16, at _SB_GU word (u SH_ROWS + p) 8 + t = (32 qd + l) 8 + t + av = prims.tcgen05_ld( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_ACC_GU, cutlass.Float32), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + elif cutlass.const_expr(SH_SPLIT == 2): + av = prims.tcgen05_ld( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_ACC_GU, cutlass.Float32), + num=16, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(2): + for tt in cutlass.range_constexpr(8): + bigf.store( + cutlass.Float32(av[8 + tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(2)) * cutlass.Int32(32) + lane) + * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd < cutlass.Int32(2): + # m3_mx_gemv's owner sum: (0 + rank 0) + rank 1, then bf16 + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + else: + # 16-row octants: warp qd = slice qd, lane l = (u = l / 16 gate / up, position l % 16); its tokens are + # accumulator columns 8 qd + t + av = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_GU + qd * cutlass.Int32(8), + cutlass.Float32, + ), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(1): + for tt in cutlass.range_constexpr(8): + bigf.store( + cutlass.Float32(av[tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(1)) * cutlass.Int32(32) + lane) + * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd == cutlass.Int32(0): + # m3_mx_gemv's owner sum: (((0 + rank 0) + rank 1) + rank 2) + rank 3, then bf16 + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + for s_ in cutlass.range_constexpr(1, SH_SPLIT): + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (cutlass.Int32((s_ - 1) * 32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + lane * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + # h = silu_and_mul(gate, up) of positions 32 sid + p, published under the call's tag + for r in cutlass.range_constexpr((SH_ROWS * M_MAX + 127) // 128): + pq = et + cutlass.Int32(r * 128) + if pq < cutlass.Int32(SH_ROWS * M_MAX): + p = pq // cutlass.Int32(M_MAX) + tt = pq - p * cutlass.Int32(M_MAX) + ga = bigf.load(idx=cutlass.Int32(_SB_GU // 4) + p * cutlass.Int32(8) + tt) + up = bigf.load( + idx=cutlass.Int32(_SB_GU // 4) + + (cutlass.Int32(SH_ROWS) + p) * cutlass.Int32(8) + + tt + ) + q_a = cute.arch.fmin(cutlass.Float32(ga), cutlass.Float32(7.0)) + q_bu = _min_xorsign_abs(cutlass.Float32(up), cutlass.Float32(7.0)) + q_ub = _add_rn(q_bu, cutlass.Float32(1.0)) + q_e2 = _ex2_full( + _mul_rn( + _mul_rn(q_a, cutlass.Float32(-1.70200002193450927734)), + cutlass.Float32(1.4426950216293334961), + ) + ) + q_sg = _div_full(cutlass.Float32(1.0), _add_rn(q_e2, cutlass.Float32(1.0))) + q_h = _mul_rn(q_ub, _mul_rn(q_sg, q_a)) + _st_relaxed_u32( + hx.subview(tt * cutlass.Int32(SH_I) + sid * cutlass.Int32(SH_ROWS) + p) + .data_ptr() + .toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(q_h), + ) + if et == 0: + _stamp(tdbg, bx, 50) + # every shared CTA's h: (token, K block) per thread, 32 tagged words, MXFP8 into B (h) and its codes + for rr in cutlass.range_constexpr((M_MAX * SH_DKB + 127) // 128): + q_ = et + cutlass.Int32(rr * 128) + if q_ < cutlass.Int32(M_MAX * SH_DKB): + t = q_ // cutlass.Int32(SH_DKB) + kb = q_ - t * cutlass.Int32(SH_DKB) + base = t * cutlass.Int32(SH_I) + kb * cutlass.Int32(32) + hv = [cutlass.Int32(0) for _ in range(32)] + pend = cutlass.Int32(1) + t0h = cutlass.Int64(0) + while pend != cutlass.Int32(0): + t0h = _hang_tick(t0h) + for q4 in cutlass.range_constexpr(8): + v4 = prims.load_ext( + hx.subview(base + cutlass.Int32(4 * q4)), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for z in cutlass.range_constexpr(4): + hv[4 * q4 + z] = cutlass.Int32(v4[z]) + okw = cutlass.Boolean(True) + for z in cutlass.range_constexpr(32): + okw = okw & ( + ((hv[z] >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) == tag16 + ) + pend = cutlass.select_(okw, cutlass.Int32(0), cutlass.Int32(1)) + vals = [ + (hv[z] << cutlass.Int32(16)).bitcast(cutlass.Float32) for z in range(32) + ] + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(vals[z])) + code, osc = _mx_osc(amax) + words = _mx_words(vals, osc) + kt = kb >> cutlass.Int32(2) + c0 = (kb & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BH) + kt * cutlass.Int32(1024) + t * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + (((c0 + cutlass.Int32(1)) ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_HC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + okh = lane < cutlass.Int32(M_MAX) + for kt in cutlass.range_constexpr(SH_DKT): + wsf = big32.load( + idx=( + cutlass.Int32(_SB_HC) + + cutlass.select_(okh, lane, cutlass.Int32(0)) * cutlass.Int32(SH_DKB) + + cutlass.Int32(4 * kt) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_DN + kt * 4), + cutlass.select_(okh, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if et == 0: + prims.mbarrier_arrive(b_dn_rdy) + _stamp(tdbg, bx, 51) + # ---- down out: tile j row 32 qd + l -> y_sh row 256 sid + 128 j + 32 qd + l + _wait(acc_dn, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for jt in cutlass.range_constexpr(2): + dv = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_DN + jt * 8, + cutlass.Float32, + ), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + rowo = ( + sid * cutlass.Int32(256) + + cutlass.Int32(128 * jt) + + qd * cutlass.Int32(32) + + lane + ) + for tt in cutlass.range_constexpr(M_MAX): + _st_relaxed_u32( + dnx.subview(cutlass.Int32(tt * H) + rowo).data_ptr().toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(cutlass.Float32(dv[tt])), + ) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if et == 0: + _stamp(tdbg, bx, 52) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if warp == 1: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc(cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), SH_TMEM) + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + + +@cute.kernel +def m3_moe_kernel( + tma_a1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_a2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s1_desc: cutlass.GridConstant[cuda.TensorMap], # w3_w1_weight_scale as 128-byte lines + tma_s2_desc: cutlass.GridConstant[cuda.TensorMap], # w2_weight_scale as 128-byte lines + tma_gu_desc: cutlass.GridConstant[ + cuda.TensorMap + ], # SHARED: gate_up e4m3 [1536, 6144], box 128 B x 32 rows + tma_dn_desc: cutlass.GridConstant[ + cuda.TensorMap + ], # SHARED: down e4m3 [6144, 768], box 128 B x 128 rows + logits: cutlass.Array, # f32 [M, 128] (not FUSED_ROUTER) + bias: cutlass.Array, # f32 [128] + x32: cutlass.Array, # int32 words of x_fp4 [M, 3072 B] (not FUSED_ROUTER) + xsf32: cutlass.Array, # int32 words of x_sf [M, 384 B] (not FUSED_ROUTER) + escal: cutlass.Array, # f32 [6, 128]: fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha + plan: cutlass.Array, # int32 [4 M + 1, NCTA, NG_CAP + 1] by G: per group i0 | i1 << 16, then b0 | nb << 16 + partials: cutlass.Array, # int32 bits of f32 [G_CAP * 48, M, 32]: split units' head sums + hbuf32: cutlass.Array, # int32 [G_CAP * M, 48, 4]: tagged intermediate records + epochs: cutlass.Array, # int32 [NCTA] + out: cutlass.Array, # bf16 [M, 6144] + route_dbg: cutlass.Array, # int32 [M, 8]: ids, weight bits (CTA 0) + tdbg: cutlass.Array, # int64 [NCTA, 32] timing stamps (TIMING builds) + xb32: cutlass.Array, # FUSED_ROUTER: int32 words (bf16 pairs) of the hidden states [M, 6144] + rw: cutlass.Array, # FUSED_ROUTER: f32 router weight [128, 6144] + sfsc: cutlass.Array, # FUSED_ROUTER: f32 [1] fc31_input_scale + lgbuf: cutlass.Array, # FUSED_ROUTER: int64 [LG_REPL, LG_STRIDE] logits: tag << 32 | f32 bits + guw: cutlass.Array, # SHARED: uint8 gate_up e4m3 [1536, 6144] + gusf: cutlass.Array, # SHARED: uint8 its swizzled UE8M0 scales + dnw: cutlass.Array, # SHARED: uint8 down e4m3 [6144, 768] + dnsf: cutlass.Array, # SHARED: uint8 its swizzled UE8M0 scales + hx: cutlass.Array, # SHARED: int32 [M, 768]: h as (call tag << 16 | bf16), tag 0 = never written + dnx: cutlass.Array, # SHARED: int32 [M, 6144]: y_sh as (call tag << 16 | bf16), tag 0 = never written + arp: cutlass.Array, # int64 [8]: all-reduce push (enable, mc, flags, rank, world, maxt); enable 0 = off + s1w: cutlass.Array, # SFA_CP: int32 words of w3_w1_weight_scale + s2w: cutlass.Array, # SFA_CP: int32 words of w2_weight_scale + img32: cutlass.Array, # B2IMG: int32 [NG_CAP, IMG_BYTES / 4] per-group FC2 B image (tile + scale atoms) + b2cnt: cutlass.Array, # B2IMG: int32 [NG_CAP * 8 + 8] per-group unit counters (index 8 g) +): + tidx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tidx % 32 + if tidx == 0: + _stamp(tdbg, bx, 0) + if cutlass.const_expr(TIMING): + if tidx == 0: + tdbg.store( + cutlass.Int64(_smid()), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(63) + ) + + s_big = cutlass.Array(cutlass.Int8, BIG, space=cutlass.AddressSpace.smem, alignment=1024) + sA = cutlass.Array( + s_big.data_ptr(0), shape=(NUM_BYTES_A * STAGES,), dtype=cutlass.Int8, alignment=1024 + ) + sSFAraw = cutlass.Array( + s_big.data_ptr(_OFF_SFAR), + shape=(max(1, _RAW_STAGE // 4) * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sSFA = cutlass.Array( + s_big.data_ptr(_OFF_SFA), + shape=(NUM_WORDS_SFA * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sB1 = cutlass.Array( + s_big.data_ptr(_OFF_B1), shape=(NUM_BYTES_B1 * S1,), dtype=cutlass.Int8, alignment=1024 + ) + sB2 = cutlass.Array( + s_big.data_ptr(_OFF_B2), shape=(2 * KT2 * NUM_BYTES_B2,), dtype=cutlass.Int8, alignment=1024 + ) + sSFB2 = cutlass.Array( + s_big.data_ptr(_OFF_SFB2), + shape=(2 * KT2 * NUM_KB * 128,), + dtype=cutlass.Int32, + alignment=1024, + ) + ys = cutlass.Array( + cutlass.Float32, NB2_CAP * M_MAX * TOP_K * R2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_part = cutlass.Array( + cutlass.Float32, 2 * M_MAX * C1 * R1, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_bias = cutlass.Array(cutlass.Float32, E_LOCAL, space=cutlass.AddressSpace.smem, alignment=16) + s_red = cutlass.Array(cutlass.Float32, 9 * M_MAX, space=cutlass.AddressSpace.smem, alignment=16) + s_xsf = cutlass.Array( + cutlass.Int32, + M_MAX * (H // 64) if FUSED_ROUTER else 4, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + ab_full = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + ab_empty = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + sfa_ready = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + acc_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_ready = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_free = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + # this CTA's FC1 share of group g is out: one barrier per group parity (a single barrier could complete twice + # while warp 5 still works on an earlier group, and its parity wait would then miss them) + fc1_done = cutlass.Array(cutlass.Int64, 4, space=cutlass.AddressSpace.smem, alignment=8) + b1_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + x_bar = cutlass.Array( + cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8 + ) # x landed, x consumed + img_bar = cutlass.Array( + cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8 + ) # (B2IMG) image landed + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + s_el = cutlass.Array( + cutlass.Int32, G_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16 + ) # slots, then G + s_kpos = cutlass.Array( + cutlass.Int32, G_CAP * M_MAX, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_w = cutlass.Array( + cutlass.Float32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_ids = cutlass.Array( + cutlass.Int32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_plan = cutlass.Array(cutlass.Int32, NG_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + # this CTA's plan rows for every G (static data: loaded before the grid dependency) + s_misc = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_plan_all = cutlass.Array( + cutlass.Int32, (G_CAP + 1) * (NG_CAP + 1), space=cutlass.AddressSpace.smem, alignment=16 + ) + s_esc = cutlass.Array( + cutlass.Float32, 8 * G_CAP, space=cutlass.AddressSpace.smem, alignment=32 + ) # slot s at 8 s + + is_routed = cutlass.Boolean(True) + if cutlass.const_expr(SHARED): + sh_full = cutlass.Array(cutlass.Int64, 32, space=cutlass.AddressSpace.smem, alignment=8) + is_routed = bx < cutlass.Int32(NCTA) + if bx >= cutlass.Int32(NCTA): + _shared_cta( + tidx, + warp, + lane, + bx - cutlass.Int32(NCTA), + s_big, + sh_full, + epochs, + tdbg, + bx, + xb32, + tma_gu_desc.get_ptr(), + tma_dn_desc.get_ptr(), + gusf, + dnsf, + hx, + dnx, + arp, + tmem_ptr_i32, + guw, + dnw, + ) + if is_routed: + # ---- before the grid dependency + if warp == 0: + if tidx < STAGES: + prims.mbarrier_init(ab_full.subview(tidx), 33 if SFA_CP else 1) + prims.mbarrier_init(ab_empty.subview(tidx), 1) + prims.mbarrier_init(sfa_ready.subview(tidx), 1) + if tidx < 2: + prims.mbarrier_init(acc_full.subview(tidx), 1) + prims.mbarrier_init(acc_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(acc2_full.subview(tidx), 1) + prims.mbarrier_init(acc2_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(b2_ready.subview(tidx), 1) + prims.mbarrier_init(b2_free.subview(tidx), EPI_THREADS) + if cutlass.const_expr(B2IMG): + prims.mbarrier_init(img_bar.subview(tidx), 1) + if tidx < 4: + prims.mbarrier_init(fc1_done.subview(tidx), EPI_THREADS) + if tidx == 0: + prims.mbarrier_init(b1_ready.subview(0), EPI_THREADS) + if cutlass.const_expr(XSTAGE): + prims.mbarrier_init(x_bar.subview(0), 1) + prims.mbarrier_init(x_bar.subview(1), EPI_THREADS) + if warp == MMA_WARP: + prims.tcgen05_alloc(tmem_ptr_i32, TMEM_COLS) + prims.tcgen05_relinquish_alloc_permit() + if warp == W_WARP: + prims.prefetch_tensormap(tma_a1_desc.get_ptr()) + prims.prefetch_tensormap(tma_a2_desc.get_ptr()) + prims.prefetch_tensormap(tma_s1_desc.get_ptr()) + prims.prefetch_tensormap(tma_s2_desc.get_ptr()) + for i in cutlass.range_constexpr(E_LOCAL // 32): + s_bias.store(cutlass.Float32(bias.load(idx=lane + i * 32)), idx=lane + i * 32) + for i in cutlass.range_constexpr(((G_CAP + 1) * (NG_CAP + 1) + 31) // 32): + q = lane + cutlass.Int32(32 * i) + if q < cutlass.Int32((G_CAP + 1) * (NG_CAP + 1)): + gq = q // cutlass.Int32(NG_CAP + 1) + s_plan_all.store( + plan.load( + idx=(gq * cutlass.Int32(NCTA) + bx) * cutlass.Int32(NG_CAP + 1) + + q + - gq * cutlass.Int32(NG_CAP + 1) + ), + idx=q, + ) + rwv = [] + if cutlass.const_expr(FUSED_ROUTER): + tg = cutlass.select_(tidx < cutlass.Int32(GEMV_THREADS), tidx, cutlass.Int32(0)) + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + rbx = cutlass.select_(eg < cutlass.Int32(NROUTE), eg, cutlass.Int32(0)) + rwv.append( + [ + rw.load( + idx=rbx * cutlass.Int32(H) + + tg * cutlass.Int32(4) + + cutlass.Int32(1024 * j), + vector_size=4, + alignment=16, + ) + for j in range(H // 1024) + ] + ) + if cutlass.const_expr(TIMING): + if tidx == 0: + s_el.store(cutlass.Int32(-7), idx=G_CAP) + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(XSTAGE): + if warp == 0: + if lane == 0: + prims.mbarrier_arrive_expect_tx(x_bar.subview(0), M_MAX * H * 2) + cute.arch.sync_warp() + if lane < M_MAX: + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(lane * cutlass.Int32(H * 2)), + xb32.subview(lane * cutlass.Int32(H // 2)), + x_bar.subview(0), + H * 2, + ) + ar_on = arp.load(idx=0) != cutlass.Int64(0) + ar_off = arp.load(idx=0) == cutlass.Int64(0) + if cutlass.const_expr(not LATE_TRIGGER): + if ar_off: + cute.arch.griddepcontrol_launch_dependents() + if cutlass.const_expr(LATE_TRIGGER and TAIL_TRIGGER and NHEAVY > 0): + if ar_off & (bx < cutlass.Int32(NHEAVY)): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + + ep = epochs.load(idx=bx) + tagv = ((ep & cutlass.Int32(0x7F)) | cutlass.Int32(0x80)) << cutlass.Int32(24) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + ltag = cutlass.Int64(ep + cutlass.Int32(1)) << cutlass.Int64(32) + if cutlass.const_expr(FUSED_ROUTER): + # ---- router GEMV (experts bx, bx + NCTA, .., threads 0..255), m3_router_gemv's order exactly + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + if (eg < cutlass.Int32(NROUTE)) & (tidx < cutlass.Int32(GEMV_THREADS)): + accs = [cutlass.Float32(0.0) for _ in range(M_MAX)] + xsrc = xb32 + if cutlass.const_expr(GEMV_SMEM and XSTAGE): + _wait(x_bar.subview(0), 0) + xsrc = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + for j in cutlass.range_constexpr(H // 1024): + for t in cutlass.range_constexpr(M_MAX): + xw = xsrc.load( + idx=cutlass.Int32(t * (H // 2) + 512 * j) + tidx * cutlass.Int32(2), + vector_size=2, + alignment=8, + ) + x0 = cutlass.Int32(xw[0]) + x1 = cutlass.Int32(xw[1]) + a = accs[t] + a = _fma_rn( + (x0 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][0]), + a, + ) + a = _fma_rn( + (x0 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][1]), + a, + ) + a = _fma_rn( + (x1 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][2]), + a, + ) + a = _fma_rn( + (x1 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][3]), + a, + ) + accs[t] = a + for t in cutlass.range_constexpr(M_MAX): + v = accs[t] + for off in (16, 8, 4, 2, 1): + v = _add_rn(v, cute.arch.shuffle_sync_bfly(v, off)) + if lane == 0: + s_red.store(v, idx=warp * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < M_MAX: + sacc = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(8): + sacc = _add_rn(sacc, s_red.load(idx=cutlass.Int32(w * M_MAX) + tidx)) + s_red.store(sacc, idx=cutlass.Int32(8 * M_MAX) + tidx) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < LG_REPL: + for t in cutlass.range_constexpr(M_MAX): + lw = ltag | ( + cutlass.Int64( + cutlass.Float32(s_red.load(idx=8 * M_MAX + t)).bitcast( + cutlass.Int32 + ) + ) + & cutlass.Int64(0xFFFFFFFF) + ) + _st_relaxed_u64( + lgbuf.subview( + tidx * cutlass.Int32(LG_STRIDE) + + cutlass.Int32(t * E_LOCAL) + + eg + ) + .data_ptr() + .toint(), + lw, + ) + if cutlass.const_expr(NGEMV > 1): + cute.arch.sync_warp() + cute.arch.barrier( + barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS + ) # s_red reused next + if tidx == 0: + _stamp(tdbg, bx, 21) + + # ---- routing (warps 4..7, token t on warp 4 + t % 4); the epilogue warps build B meanwhile + if ((warp >= W_WARP) & (warp < cutlass.Int32(W_WARP + NRW0))) | ( + (warp >= cutlass.Int32(8)) & (warp < cutlass.Int32(8 + RW_EXTRA)) + ): + rw_i = cutlass.select_( + warp < cutlass.Int32(8), warp - cutlass.Int32(W_WARP), warp - cutlass.Int32(4) + ) + # ILP: poll every token's logits first, then run the independent top-k chains back to back + lgv_all = [] + t_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = rw_i + cutlass.Int32(NRW * tr) + if cutlass.const_expr(M_MAX % NRW != 0): + t = cutlass.select_(t < cutlass.Int32(M_MAX), t, cutlass.Int32(M_MAX - 1)) + t_all.append(t) + lgv = [] + if cutlass.const_expr(FUSED_ROUTER): + want = ltag >> cutlass.Int64(32) + rbase = ( + (bx % cutlass.Int32(LG_REPL)) * cutlass.Int32(LG_STRIDE) + + t * cutlass.Int32(E_LOCAL) + + lane + ) + done = cutlass.Int32(0) + res = [cutlass.Int64(0) for _ in range(4)] + t0h = cutlass.Int64(0) + while done == cutlass.Int32(0): + t0h = _hang_tick(t0h) + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(4): + res[z] = _ld_relaxed_u64( + lgbuf.subview(cutlass.Int32(z * 32) + rbase).data_ptr().toint() + ) + ok = ok & ((res[z] >> cutlass.Int64(32)) == want) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32( + -1 + ): + done = cutlass.Int32(1) + else: + _nanosleep(POLL_NS) + for z in cutlass.range_constexpr(4): + lgv.append(cutlass.Int32(res[z]).bitcast(cutlass.Float32)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 22) + else: + for i in cutlass.range_constexpr(4): + lgv.append( + cutlass.Float32( + logits.load( + idx=t * cutlass.Int32(E_LOCAL) + cutlass.Int32(i * 32) + lane + ) + ) + ) + lgv_all.append(lgv) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 61) + res_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = t_all[tr] + lgv = lgv_all[tr] + own = (rw_i + cutlass.Int32(NRW * tr)) < cutlass.Int32(M_MAX) + # top-4 (keys sigmoid + bias, ties to the lower id) + keys = [] + for i in cutlass.range_constexpr(4): + sg = _fma_rn( + cutlass.Float32(0.5), + cute.math.tanh(_mul_rn(cutlass.Float32(0.5), lgv[i])), + cutlass.Float32(0.5), + ) + keys.append(_add_rn(sg, s_bias.load(idx=i * 32 + lane))) + hi = [_twiddle(keys[i]) for i in range(4)] + taken = [cutlass.Boolean(False) for _ in range(4)] + ids_t = [] + kt_t = [] + for k in cutlass.range_constexpr(TOP_K): + bh = cutlass.Int32(0) + bl = cutlass.Int32(0) + for i in cutlass.range_constexpr(4): + lo_i = cutlass.Int32(65535 - i * 32) - lane + hv = cutlass.select_(taken[i], cutlass.Int32(0), hi[i]) + lv = cutlass.select_(taken[i], cutlass.Int32(0), lo_i) + better = (cutlass.Uint32(hv) > cutlass.Uint32(bh)) | ( + (hv == bh) & (cutlass.Uint32(lv) > cutlass.Uint32(bl)) + ) + bh = cutlass.select_(better, hv, bh) + bl = cutlass.select_(better, lv, bl) + mh = _redux_max_u32(bh) + ml = _redux_max_u32(cutlass.select_(bh == mh, bl, cutlass.Int32(0))) + e_k = cutlass.Int32(65535) - ml + for i in cutlass.range_constexpr(4): + taken[i] = taken[i] | (e_k == cutlass.Int32(i * 32) + lane) + ids_t.append(e_k) + kv = cutlass.Float32(0.0) + for i in cutlass.range_constexpr(4): + kv = cutlass.select_(e_k == cutlass.Int32(i * 32) + lane, keys[i], kv) + kt_t.append(cute.arch.shuffle_sync(kv, e_k % cutlass.Int32(32))) + sig = cutlass.Float32(0.0) + e_l = cutlass.Int32(0) + for k in cutlass.range_constexpr(TOP_K): + sk = _sub_rn(kt_t[k], s_bias.load(idx=ids_t[k])) + sig = cutlass.select_(lane == cutlass.Int32(k), sk, sig) + e_l = cutlass.select_(lane == cutlass.Int32(k), ids_t[k], e_l) + total = sig + for off in (16, 8, 4, 2, 1): + total = _add_rn(total, cute.arch.shuffle_sync_bfly(total, off)) + wv = _bf16_round( + _div_rn( + _mul_rn(sig, cutlass.Float32(ROUTE_SCALE)), + _add_rn(total, cutlass.Float32(SUM_EPS)), + ) + ) + res_all.append((t, own, wv, e_l)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 62) + for tr in cutlass.range_constexpr( + (M_MAX + NRW - 1) // NRW + ): # stores after every chain (one branch-free block) + t, own, wv, e_l = res_all[tr] + if (lane < TOP_K) & own: + s_w.store(wv, idx=t * cutlass.Int32(TOP_K) + lane) + s_ids.store(e_l, idx=t * cutlass.Int32(TOP_K) + lane) + if bx == 0: + route_dbg.store( + wv.bitcast(cutlass.Int32), + idx=t * cutlass.Int32(8) + cutlass.Int32(4) + lane, + ) + cute.arch.sync_warp() + if cutlass.const_expr(EARLY0 and not E0_OFFLOAD): + if warp == cutlass.Int32(W_WARP): + # slots 0..3 are always token 0's four (distinct) experts and group 0's stage ranges are the same + # for every G (the plan guarantees it): fill the ring with this CTA's first group-0 stages now + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(W_WARP): + _bar_arrive( + E0_BAR, 64 + ) # token 0's experts are in s_ids: warp 7 issues the early stages + if cutlass.const_expr(B1_AFTER_TOPK): + _bar_arrive(TOPK2_BAR, 128 + 32 * NRW) + cute.arch.barrier(barrier_id=TOPK_BAR, number_of_threads=32 * NRW) + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(E0_WARP): + cute.arch.barrier(barrier_id=E0_BAR, number_of_threads=64) + # slots 0..3 are always token 0's four (distinct) experts and group 0's stage ranges are the same + # for every G (the plan guarantees it): fill the ring with this CTA's first group-0 stages now + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + _bar_arrive(E1_BAR, 64) + if warp == W_WARP: + if lane == 0: + _stamp(tdbg, bx, 20) + # slots: lane l < 4 M is (token l / 4, top-k l % 4); the distinct experts in lane order (match.any) + valid = lane < cutlass.Int32(TOP_K * M_MAX) + my_e = cutlass.select_( + valid, + s_ids.load(idx=cutlass.select_(valid, lane, cutlass.Int32(0))), + cutlass.Int32(-1) - lane, + ) + same = _match_any(my_e) + leader = cute.arch.popc((same & (cutlass.Int32(0) - same)) - cutlass.Int32(1)) + first = valid & (lane == leader) + bal = prims.vote_sync(0xFFFFFFFF, first, prims.VoteSync.BALLOT) + rank = cute.arch.popc(bal & cutlass.Int32(cute.arch.lanemask_lt())) + slot_l = cute.arch.shuffle_sync(rank, leader) + for z in cutlass.range_constexpr((G_CAP * M_MAX + 31) // 32): + if lane + cutlass.Int32(32 * z) < cutlass.Int32(G_CAP * M_MAX): + s_kpos.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + cute.arch.sync_warp() + if first: + s_el.store(my_e, idx=rank) + if valid: + s_kpos.store( + lane % cutlass.Int32(TOP_K), + idx=slot_l * cutlass.Int32(M_MAX) + lane // cutlass.Int32(TOP_K), + ) + if bx == 0: + route_dbg.store( + my_e, + idx=(lane // cutlass.Int32(TOP_K)) * cutlass.Int32(8) + + lane % cutlass.Int32(TOP_K), + ) + gcnt = cute.arch.popc(bal) + if lane == 0: + s_el.store(gcnt, idx=G_CAP) + if lane < cutlass.Int32(NG_CAP + 1): # the plan row of this G (preloaded) + s_plan.store(s_plan_all.load(idx=gcnt * cutlass.Int32(NG_CAP + 1) + lane), idx=lane) + cute.arch.sync_warp() + if cutlass.const_expr(B2IMG): + # CTA 0 pads every group counter to exactly CNT_PER_CALL for this call (absent slots, unused groups) + if (bx == 0) & (lane < cutlass.Int32(NG_CAP)): + nh = gcnt - lane * cutlass.Int32(4) + nh = cutlass.select_(nh > cutlass.Int32(4), cutlass.Int32(4), nh) + nh = cutlass.select_(nh < cutlass.Int32(0), cutlass.Int32(0), nh) + pad = cutlass.Int32(CNT_PER_CALL) - nh * cutlass.Int32(UNITS_ROW) + if pad > cutlass.Int32(0): + _red_release_add( + b2cnt.subview(lane * cutlass.Int32(8)).data_ptr().toint(), pad + ) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 24) + cute.arch.sync_warp() + _bar_arrive(ROUTE_BAR, THREADS) + if warp < 4: + if cutlass.const_expr(B1_AFTER_TOPK): + cute.arch.barrier(barrier_id=TOPK2_BAR, number_of_threads=128 + 32 * NRW) + if cutlass.const_expr(FUSED_ROUTER): + if tidx == 0: + _stamp(tdbg, bx, 57) + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(0), 0) + if tidx == 0: + _stamp(tdbg, bx, 58) + xs32 = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xs32, sfsc, sB1, s_xsf, b1_ready + ) + prims.mbarrier_arrive(x_bar.subview(1)) # the ring is free for the weights + else: + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xb32, sfsc, sB1, s_xsf, b1_ready + ) + if tidx == 0: + _stamp(tdbg, bx, 23) + else: + _build_b1(tidx, warp, lane, base_row, base_col, x32, xsf32, sB1, b1_ready) + if warp != W_WARP: # one sync site for every thread but warp 4's (which arrived) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ROUTE_BAR, number_of_threads=THREADS) + if tidx == 0: + _stamp(tdbg, bx, 2) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(s_el.load(idx=G_CAP)), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(28), + ) + + g_n = s_el.load(idx=G_CAP) + ng = (g_n + cutlass.Int32(3)) >> cutlass.Int32(2) + pr2 = s_plan.load(idx=NG_CAP) + blk0 = pr2 & cutlass.Int32(0xFFFF) + nb = pr2 >> cutlass.Int32(16) + + # ---- weights producer (warp 4): phase p = FC1 stages of group p, then FC2 tiles of group p - LAG + if warp == W_WARP: + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) # the staged hidden states are quantized + stage = cutlass.Int32(0) + phase = cutlass.Int32(1) + skip = cutlass.Int32(0) + if cutlass.const_expr(E0_OFFLOAD): + cute.arch.barrier(barrier_id=E1_BAR, number_of_threads=64) + if cutlass.const_expr(EARLY0): + skip = s_misc.load(idx=0) # group-0 stages already in the ring + stage = cutlass.select_(skip == cutlass.Int32(STAGES), cutlass.Int32(0), skip) + phase = cutlass.select_( + skip == cutlass.Int32(STAGES), cutlass.Int32(0), cutlass.Int32(1) + ) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_el.load(idx=slot) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if skip > cutlass.Int32(0): + skip = skip - cutlass.Int32(1) + else: + _waitd(ab_empty.subview(stage), phase, tdbg, 1) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + ), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, stage, sf_line, c, sSFA, s1w, ab_full) + stage, phase = _adv(stage, phase) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + for bb in range(nb): + blk = blk0 + bb + for t2 in cutlass.range_constexpr(KT2): + _waitd(ab_empty.subview(stage), phase, tdbg, 2) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + n_here + * cutlass.Int32( + NUM_BYTES_BOX1 + if SFA_CP + else NUM_BYTES_BOX1 + NUM_BYTES_SFA_RAW // 4 + ), + ) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(jj * NUM_BYTES_BOX1) + ), + tma_a2_desc.get_ptr(), + ( + cutlass.Int32(t2 * KT_BYTES), + blk * cutlass.Int32(R2), + e, + ), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + cutlass.Int32(jj * NUM_WORDS_SFA_RAW // 4) + ), + tma_s2_desc.get_ptr(), + ( + cutlass.Int32(0), + ( + ( + e * cutlass.Int32(SF2_RB) + + (blk >> cutlass.Int32(2)) + ) + * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(4), + ), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf2_cp(lane, stage, g, blk, t2, n_here, s_el, sSFA, s2w, ab_full) + stage, phase = _adv(stage, phase) + if lane == 0: + _stamp(tdbg, bx, 3) + + # ---- scale splicer (warp 6): every stage, atom kb word [r][q] = raw atom 4 q + kb word [r][m1] + if (warp == S_WARP) & cutlass.Boolean(not SFA_CP): # (SFA_CP: no splicer) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + for p in range(ng + cutlass.Int32(LAG)): + i0 = cutlass.Int32(0) + i1 = cutlass.Int32(0) + for part in cutlass.range_constexpr(2): + cnt = cutlass.Int32(0) + m1 = cutlass.Int32(0) + nk = cutlass.Int32(0) + if cutlass.const_expr(part == 0): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + nk = _nsegs(i0, i1) + else: + if p >= cutlass.Int32(LAG): + nk = nb + for k in range(nk): + if cutlass.const_expr(part == 0): + ug, j0, j1 = _seg(i0, i1, k) + m1 = ( + ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + ) & cutlass.Int32(3) + cnt = j1 - j0 + else: + m1 = (blk0 + k) & cutlass.Int32(3) + cnt = cutlass.Int32(KT2) + for _j in cutlass.range(cnt, unroll=1): + _waitd(ab_full.subview(stage), phase, tdbg, 3) + raw0 = ( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + lane * cutlass.Int32(4) + + m1 + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for kb in cutlass.range_constexpr(NUM_KB): + w0 = sSFAraw.load(idx=raw0 + cutlass.Int32((0 * 4 + kb) * 128)) + w1 = sSFAraw.load(idx=raw0 + cutlass.Int32((1 * 4 + kb) * 128)) + w2 = sSFAraw.load(idx=raw0 + cutlass.Int32((2 * 4 + kb) * 128)) + w3 = sSFAraw.load(idx=raw0 + cutlass.Int32((3 * 4 + kb) * 128)) + sSFA.store( + ( + cutlass.Int32(w0), + cutlass.Int32(w1), + cutlass.Int32(w2), + cutlass.Int32(w3), + ), + idx=dst0 + cutlass.Int32(kb * 128), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + prims.tcgen05_commit(sfa_ready.subview(stage)) + stage, phase = _adv(stage, phase) + + if cutlass.const_expr(B2IMG): + # ---- (B2IMG) group releases (warp 6): this CTA's finished units of group p, one release add per group + if warp == cutlass.Int32(S_WARP): + for p in range(ng): + i0, i1 = _plan_range(s_plan, p) + n_done = cutlass.Int32(0) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + n_done = n_done + cutlass.select_( + (j0 != cutlass.Int32(0)) | (j1 == cutlass.Int32(S1)), + cutlass.Int32(1), + cutlass.Int32(0), + ) + _wait( + fc1_done.subview(p & cutlass.Int32(3)), + (p >> cutlass.Int32(2)) & cutlass.Int32(1), + ) + if cutlass.const_expr(B2_DLY > 0): + t0d = _gtime() + dl = cutlass.Int64( + ( + (bx * cutlass.Int32(37) + p * cutlass.Int32(11) + ep) + & cutlass.Int32(63) + ) + * cutlass.Int32(B2_DLY // 64) + ) + while _gtime() - t0d < dl: + pass + if (lane == 0) & (n_done > cutlass.Int32(0)): + _fence_proxy_async_global() + _red_release_add( + b2cnt.subview(p * cutlass.Int32(8)).data_ptr().toint(), n_done + ) + cute.arch.sync_warp() + # ---- (B2IMG) FC2 B loader (warp 5): group g's image once its counter is complete + if warp == cutlass.Int32(X_WARP): + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + cnt_target = (ep + cutlass.Int32(1)) * cutlass.Int32(CNT_PER_CALL) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + ((g >> cutlass.Int32(1)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + if lane == 0: + _stamp(tdbg, bx, 29) + caddr = b2cnt.subview(g * cutlass.Int32(8)).data_ptr().toint() + if lane == 0: + cv = _ld_acquire_u32(caddr) + t0h = cutlass.Int64(0) + while (cv - cnt_target) < cutlass.Int32(0): + t0h = _hang_tick(t0h) + _nanosleep(B2IMG_NS) + cv = _ld_acquire_u32(caddr) + _fence_proxy_async_global() + prims.mbarrier_arrive_expect_tx(img_bar.subview(buf), IMG_BYTES) + prims.cp_async_bulk_shared_cluster_global( + sB2.subview(buf * cutlass.Int32(IMG_B2)), + img32.subview(g * cutlass.Int32(IMG_BYTES // 4)), + img_bar.subview(buf), + IMG_B2, + ) + prims.cp_async_bulk_shared_cluster_global( + sSFB2.subview(buf * cutlass.Int32(IMG_SF // 4)), + img32.subview( + g * cutlass.Int32(IMG_BYTES // 4) + cutlass.Int32(IMG_B2 // 4) + ), + img_bar.subview(buf), + IMG_SF, + ) + cute.arch.sync_warp() + _waitd( + img_bar.subview(buf), (g >> cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 108 + ) + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 25) + # ---- FC2 B builder (warps 5, 8 ..): group g's records (all 48 per (slot, token), the call's tag on + # every word) into B2 buffer g % 2 (row j M + t, k-tile t2 = record / 16) and its scale atoms, then + # tcgen05.cp to TMEM. + if ((warp == X_WARP) | (warp >= cutlass.Int32(8))) & cutlass.Boolean(not B2IMG): + xi = cutlass.select_( + warp == cutlass.Int32(X_WARP), cutlass.Int32(0), warp - cutlass.Int32(7) + ) + if cutlass.const_expr(G_CAP > TOP_K): + if (bx == 0) & (xi == cutlass.Int32(0)): + # rows of slots this call does not use: tag 0, never taken for a later call's + for i in range( + g_n * cutlass.Int32(M_MAX * UNITS_ROW) + lane, + cutlass.Int32(G_CAP * M_MAX * UNITS_ROW), + 32, + ): + for rr in cutlass.range_constexpr(HB_REPL): + hbuf32.store( + ( + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + ), + idx=i * cutlass.Int32(4) + + (cutlass.Int32(rr * HB_STRIDE) if rr else 0), + alignment=16, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sSFB2_8 = cutlass.Array( + sSFB2.data_ptr(0), shape=(2 * KT2 * NUM_KB * 512,), dtype=cutlass.Int8, alignment=16 + ) + sB2_32 = cutlass.Array( + sB2.data_ptr(0), + shape=(2 * KT2 * NUM_BYTES_B2 // 4,), + dtype=cutlass.Int32, + alignment=16, + ) + nrec_cap = 4 * M_MAX * UNITS_ROW # records of one group + hb_off = cutlass.Int32(0) + if cutlass.const_expr(HB_REPL > 1): + # this CTA's copy of the records + hb_off = (bx % cutlass.Int32(HB_REPL)) * cutlass.Int32(HB_STRIDE) + NZ = (nrec_cap + 32 * XW - 1) // (32 * XW) # records per lane: lane + 32 (XW z + xi) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + ((g >> cutlass.Int32(1)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + _waitd( + fc1_done.subview(g & cutlass.Int32(3)), + (g >> cutlass.Int32(2)) & cutlass.Int32(1), + tdbg, + 4, + ) # this CTA's own share of group g is out + if cutlass.const_expr(TIMING): + if (lane == 0) & (xi == cutlass.Int32(0)) & (g < cutlass.Int32(8)): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(32) + g + ) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + nrec = n_here * cutlass.Int32(M_MAX * UNITS_ROW) + rec0 = g * cutlass.Int32( + 4 * M_MAX * UNITS_ROW + ) # rows (4 g + j) M + t are contiguous + done = cutlass.Int32(0) + rspin = cutlass.Int32(0) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 29) + npoll = cutlass.Int32(0) + got = [[cutlass.Int32(0) for _ in range(3)] for _ in range(NZ)] + t0h = cutlass.Int64(0) + while done == cutlass.Int32(0): + t0h = _hang_tick(t0h) + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + v4 = prims.load_ext( + hbuf32.subview( + (rec0 + qi) * cutlass.Int32(4) + (hb_off if HB_REPL > 1 else 0) + ), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for wq in cutlass.range_constexpr(3): + got[z][wq] = cutlass.Int32(v4[wq]) + ok = ok & ( + ( + ((got[z][0] ^ tagv) | (got[z][1] ^ tagv) | (got[z][2] ^ tagv)) + & cutlass.Int32(-16777216) + ) + == cutlass.Int32(0) + ) + npoll = npoll + cutlass.Int32(1) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32(-1): + done = cutlass.Int32(1) + else: + _nanosleep(B2_POLL_NS) + if cutlass.const_expr(HANG_DBG): + rspin = rspin + cutlass.Int32(1) + if rspin > cutlass.Int32(1 << 14): + done = cutlass.Int32(1) + tdbg.store( + cutlass.Int64(200), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + warp, + ) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + w0, w1, w2 = got[z][0], got[z][1], got[z][2] + rr = qi // cutlass.Int32( + UNITS_ROW + ) # row within the group: j M + t = B row n + c = qi - rr * cutlass.Int32(UNITS_ROW) + t2 = c >> cutlass.Int32(4) + cc = (c >> cutlass.Int32(1)) & cutlass.Int32(7) + lo = (w0 & cutlass.Int32(0xFFFFFF)) | (w1 << cutlass.Int32(24)) + hi = ((w1 >> cutlass.Int32(8)) & cutlass.Int32(0xFFFF)) | ( + (w2 & cutlass.Int32(0xFFFF)) << cutlass.Int32(16) + ) + sB2_32.store( + (lo, hi), + idx=( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_BYTES_B2) + + rr * cutlass.Int32(KT_BYTES) + + (cc ^ (rr & cutlass.Int32(7))) * cutlass.Int32(16) + + (c & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + # scale of block c: atom (t2, kb = (c % 16) / 4), row rr, byte c % 4 + sSFB2_8.store( + cutlass.Int8((w2 >> cutlass.Int32(16)) & cutlass.Int32(0xFF)), + idx=( + ( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_KB) + + ((c & cutlass.Int32(15)) >> cutlass.Int32(2)) + ) + * cutlass.Int32(128) + + rr * cutlass.Int32(4) + ) + * cutlass.Int32(4) + + (c & cutlass.Int32(3)), + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + if cutlass.const_expr(XW > 1): + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=B2_BAR, number_of_threads=32 * XW) + cute.arch.sync_warp() + if (xi == cutlass.Int32(0)) & prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + SFB2_COL + buf * cutlass.Int32(KT2 * NUM_KB * 4) + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 25) + if cutlass.const_expr(TIMING): + if g < cutlass.Int32(8): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(11) + g + ) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(npoll), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(30), + ) + + # ---- MMA (warp 7) + if warp == MMA_WARP: + idesc1 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N1, + m_dim=MMA_M, + sparsity_version=0, + ) + idesc2 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N2, + m_dim=MMA_M, + sparsity_version=0, + ) + desc_a0 = prims.Tcgen05SmemDesc.build( + sA.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b10 = prims.Tcgen05SmemDesc.build( + sB1.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b20 = prims.Tcgen05SmemDesc.build( + sB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + _waitd(b1_ready, 0, tdbg, 5) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + acc_b2 = cutlass.Int64(0) + acc_sf = cutlass.Int64(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + _waitd( + acc_empty.subview(ab), + ((k1 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + tdbg, + 102, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc_ptr = cutlass.inttoptr( + tcol + ACC1_COL + ab * cutlass.Int32(N1), 6, cutlass.Float32 + ) + for j in cutlass.range(j0, j1, unroll=1): + tw0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 6) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 6) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b10 + cutlass.Int64(j * cutlass.Int32(NUM_BYTES_B1 >> 4)) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = tcol + SFB1_COL + j * cutlass.Int32(NUM_KB * 4) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc1, + (j > j0) | cutlass.Boolean(kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc_full.subview(ab)) + k1 = k1 + cutlass.Int32(1) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + buf = g & cutlass.Int32(1) + tw2 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + _waitd( + b2_ready.subview(buf), (g >> cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 103 + ) + if cutlass.const_expr(TIMING): + acc_b2 = acc_b2 + (_gtime() - tw2) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 4) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_empty.subview(a2), + ((k2 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + tdbg, + 104, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc2_ptr = cutlass.inttoptr( + tcol + ACC2_COL + a2 * cutlass.Int32(N2), 6, cutlass.Float32 + ) + for t2 in cutlass.range_constexpr(KT2): + tw1 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 7) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 7) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw1) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b20 + cutlass.Int64( + (buf * cutlass.Int32(KT2) + cutlass.Int32(t2)) + * cutlass.Int32(NUM_BYTES_B2 >> 4) + ) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = ( + tcol + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + cutlass.Int32(t2 * NUM_KB * 4) + ) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(_nkb2(t2)): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc2_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc2, + cutlass.Boolean(t2 > 0 or kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc2_full.subview(a2)) + k2 = k2 + cutlass.Int32(1) + if cutlass.const_expr(TIMING): + if lane == 0: + tdbg.store(acc_b2, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(26)) + tdbg.store(acc_sf, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(27)) + + # ---- epilogue warps (0-3) + if warp < 4: + row_w = base_row + warp * 32 + for r in cutlass.range_constexpr((6 * G_CAP + EPI_THREADS - 1) // EPI_THREADS): + qi = tidx + cutlass.Int32(r * EPI_THREADS) + sl_e = qi // cutlass.Int32(6) + if (qi < cutlass.Int32(6 * G_CAP)) & (sl_e < g_n): + fld = qi - sl_e * cutlass.Int32(6) + s_esc.store( + cutlass.Float32( + escal.load(idx=fld * cutlass.Int32(E_LOCAL) + s_el.load(idx=sl_e)) + ), + idx=sl_e * cutlass.Int32(8) + fld, + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + SH_ADD = SHARED + wys = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + pars = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + ard = (cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0)) + for p in range(ng + cutlass.Int32(LAG)): + if cutlass.const_expr(AR_PRE): + if (p == ng + cutlass.Int32(LAG - 1)) & ar_on: + # one 8-byte load each (the vectorizer merges adjacent arp.load()s into 16-byte reads + # at 8-byte offsets: misaligned) + ard = tuple( + cutlass.Int64(_ld_relaxed_u64(arp.subview(i_).data_ptr().toint())) + for i_ in (1, 3, 4, 5) + ) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_cl = cutlass.select_( + f_bb < nb, (blk0 + f_bb) * cutlass.Int32(R2), cutlass.Int32(0) + ) + pars[bt] = _ar_parity(arp, f_t, f_cl) + if cutlass.const_expr(SH_ADD): + if p == ng + cutlass.Int32(LAG - 1): + # this CTA's y_sh words, loaded while the last group's FC2 runs (re-polled in finalize) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_ch = cutlass.select_( + f_bb < nb, + f_t * cutlass.Int32(H) + + (blk0 + f_bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + cutlass.Int32(0), + ) + wys[bt] = _ld_relaxed_u32(dnx.subview(f_ch).data_ptr().toint()) + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + cblk = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + unit = slot * cutlass.Int32(U1) + cblk + _waitd( + acc_full.subview(ab), + (k1 >> cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 105, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, base_col + ACC1_COL + ab * cutlass.Int32(N1), cutlass.Float32 + ), + num=N1, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc_empty.subview(ab)) + pb = ab * cutlass.Int32(M_MAX * C1 * R1) + for t in cutlass.range_constexpr(M_MAX): + part = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(4): + part = cutlass.select_( + warp == cutlass.Int32(w), cutlass.Float32(acc[QS * w + t]), part + ) + s_part.store(part, idx=pb + (t * C1 + warp) * R1 + lane) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + for tr in cutlass.range_constexpr((M_MAX + 3) // 4): + t = warp + cutlass.Int32(4 * tr) + if t < cutlass.Int32(M_MAX): + xv = _add_rn( + _add_rn( + _add_rn( + s_part.load(idx=pb + (t * C1 + 0) * R1 + lane), + s_part.load(idx=pb + (t * C1 + 1) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 2) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 3) * R1 + lane), + ) + finish = cutlass.Int32(1) + if (j0 != cutlass.Int32(0)) | (j1 != cutlass.Int32(S1)): + paddr = ( + partials.subview( + (unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane + ) + .data_ptr() + .toint() + ) + if j0 == cutlass.Int32(0): + _st_relaxed_u32(paddr, xv.bitcast(cutlass.Int32)) + finish = cutlass.Int32(0) + else: + pbits = _ld_relaxed_u32(paddr) + spins = cutlass.Int32(0) + t0h = cutlass.Int64(0) + while pbits == cutlass.Int32(PARTIAL_EMPTY): + t0h = _hang_tick(t0h) + pbits = _ld_relaxed_u32(paddr) + spins = spins + cutlass.Int32(1) + _st_relaxed_u32(paddr, cutlass.Int32(PARTIAL_EMPTY)) + xv = _add_rn(pbits.bitcast(cutlass.Float32), xv) + cute.arch.sync_warp() + if cutlass.const_expr( + XVDUMP + ): # debug: FC1 sums into the (unused) partial slots + partials.store( + xv.bitcast(cutlass.Int32), + idx=(unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane, + ) + if finish == cutlass.Int32(1): + gate_raw = cute.arch.shuffle_sync_bfly(xv, 8) + is_up = ( + (lane >> cutlass.Int32(3)) & cutlass.Int32(1) + ) == cutlass.Int32(0) + es = slot * cutlass.Int32(8) + qv, sfb = _swiglu_quant( + xv, + gate_raw, + is_up, + s_esc.load(idx=es), + s_esc.load(idx=es + cutlass.Int32(1)), + s_esc.load(idx=es + cutlass.Int32(2)), + s_esc.load(idx=es + cutlass.Int32(3)), + s_esc.load(idx=es + cutlass.Int32(4)), + ) + byte = _e2m1x2(cute.arch.shuffle_sync_down(qv, 16), qv) + bt = [ + cute.arch.shuffle_sync(byte, i) & cutlass.Int32(0xFF) + for i in range(8) + ] + rw0 = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | tagv + ) + rw1 = ( + bt[3] + | (bt[4] << cutlass.Int32(8)) + | (bt[5] << cutlass.Int32(16)) + | tagv + ) + rw2 = ( + bt[6] + | (bt[7] << cutlass.Int32(8)) + | ((sfb & cutlass.Int32(0xFF)) << cutlass.Int32(16)) + | tagv + ) + if cutlass.const_expr(B2IMG): + if lane == 0: + # B2 row n = j M + t, k-tile c / 16, 16-byte chunk (c / 2) % 8 + # (swizzled), half c % 2; scale: atom (t2, kb = (c % 16) / 4), + # row n at byte 16 n, byte c % 4 + n = (ug // cutlass.Int32(U1)) * cutlass.Int32(M_MAX) + t + t2i = cblk >> cutlass.Int32(4) + cci = (cblk >> cutlass.Int32(1)) & cutlass.Int32(7) + gbase = p * cutlass.Int32(IMG_BYTES) + lo = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | (bt[3] << cutlass.Int32(24)) + ) + hi = ( + bt[4] + | (bt[5] << cutlass.Int32(8)) + | (bt[6] << cutlass.Int32(16)) + | (bt[7] << cutlass.Int32(24)) + ) + img32.store( + (lo, hi), + idx=( + gbase + + t2i * cutlass.Int32(NUM_BYTES_B2) + + n * cutlass.Int32(KT_BYTES) + + (cci ^ (n & cutlass.Int32(7))) + * cutlass.Int32(16) + + (cblk & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + img8 = cutlass.Array( + img32.data_ptr(0), + shape=(NG_CAP * IMG_BYTES,), + dtype=cutlass.Int8, + alignment=16, + ) + img8.store( + cutlass.Int8(sfb & cutlass.Int32(0xFF)), + idx=gbase + + cutlass.Int32(IMG_B2) + + ( + ( + t2i * cutlass.Int32(NUM_KB) + + ( + (cblk & cutlass.Int32(15)) + >> cutlass.Int32(2) + ) + ) + * cutlass.Int32(512) + ) + + n * cutlass.Int32(16) + + (cblk & cutlass.Int32(3)), + ) + else: + if cutlass.const_expr(HB_REPL == 1): + if lane == 0: + hbuf32.store( + (rw0, rw1, rw2, tagv), + idx=( + (slot * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(UNITS_ROW) + + cblk + ) + * cutlass.Int32(4), + alignment=16, + ) + else: + if lane < HB_REPL: # copy `lane` + hbuf32.store( + (rw0, rw1, rw2, tagv), + idx=( + (slot * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(UNITS_ROW) + + cblk + ) + * cutlass.Int32(4) + + lane * cutlass.Int32(HB_STRIDE), + alignment=16, + ) + k1 = k1 + cutlass.Int32(1) + prims.mbarrier_arrive( + fc1_done.subview(p & cutlass.Int32(3)) + ) # warp 5 may poll group p + if tidx == 0: + _stamp(tdbg, bx, 10) + if cutlass.const_expr(TIMING): + if p < cutlass.Int32(8): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(40) + p, + ) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_full.subview(a2), + (k2 >> cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 106, + ) + if cutlass.const_expr(TIMING): + if (tidx == 0) & (g == ng - cutlass.Int32(1)): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(8) + bb, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + yv = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, + base_col + + ACC2_COL + + a2 * cutlass.Int32(N2) + + warp * cutlass.Int32(M_MAX), + cutlass.Float32, + ), + num=LD_POW2, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc2_empty.subview(a2)) + sl = g * cutlass.Int32(4) + warp + if sl < g_n: + a2s = s_esc.load(idx=sl * cutlass.Int32(8) + cutlass.Int32(5)) + for t in cutlass.range_constexpr(M_MAX): + kp = s_kpos.load(idx=sl * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + if kp >= cutlass.Int32(0): + ys.store( + _bf16_round(_mul_rn(cutlass.Float32(yv[t]), a2s)), + idx=( + (bb * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + * cutlass.Int32(TOP_K) + + kp + ) + * cutlass.Int32(R2) + + lane, + ) + k2 = k2 + cutlass.Int32(1) + prims.mbarrier_arrive(b2_free.subview(g & cutlass.Int32(1))) + if tidx == 0: + _stamp(tdbg, bx, 6) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + if tidx == 0: + _stamp(tdbg, bx, 19) + # finalize: (block bb, token t) per warp round: the fma chain over t's top-4 in order + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + pair = warp + cutlass.Int32(bt * 4) + bb = pair // cutlass.Int32(M_MAX) + t = pair - bb * cutlass.Int32(M_MAX) + if bb < nb: + acc_o = cutlass.Float32(0.0) + for k in cutlass.range_constexpr(TOP_K): + acc_o = _fma_rn( + s_w.load(idx=t * cutlass.Int32(TOP_K) + cutlass.Int32(k)), + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ), + acc_o, + ) + if cutlass.const_expr( + YDUMP + ): # debug: y of (token, k) into the partials workspace + partials.store( + cutlass.Float32( + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ) + ).bitcast(cutlass.Int32), + idx=(t * cutlass.Int32(TOP_K) + cutlass.Int32(k)) * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + ) + ch = ( + t * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(SH_ADD): + # + y_sh (this row's shared output, from its shared CTA), as shared_output.add_(routed_output) + wy = wys[bt] + t0h = cutlass.Int64(0) + while ((wy >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) != tag16: + t0h = _hang_tick(t0h) + wy = _ld_relaxed_u32(dnx.subview(ch).data_ptr().toint()) + acc_o = _add_rn( + (wy << cutlass.Int32(16)).bitcast(cutlass.Float32), _bf16_round(acc_o) + ) + out.store(cutlass.BFloat16(acc_o), idx=ch) + if ar_on: + cl = ( + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(AR_PRE): + _ar_push_pre( + ard, + pars[bt], + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + else: + _ar_push( + arp, + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + + # ---- teardown + if tidx == 0: + _stamp(tdbg, bx, 7) + epochs.store(ep + cutlass.Int32(1), idx=bx) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if ar_on | cutlass.Boolean(LATE_TRIGGER): + # the all-reduce polls the pushed slots without a grid wait: release it after every push of this CTA + cute.arch.griddepcontrol_launch_dependents() + if warp == MMA_WARP: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Float32), TMEM_COLS + ) + + +@cute.jit +def m3_moe( + a1_tensor: cute.Tensor, + sfa1_tensor: cute.Tensor, + a2_tensor: cute.Tensor, + sfa2_tensor: cute.Tensor, + logits: cute.Tensor, + bias: cute.Tensor, + x32: cute.Tensor, + xsf32: cute.Tensor, + escal: cute.Tensor, + plan: cute.Tensor, + partials: cute.Tensor, + hbuf32: cute.Tensor, + epochs: cute.Tensor, + out: cute.Tensor, + route_dbg: cute.Tensor, + tdbg: cute.Tensor, + xb32: cute.Tensor, + rw: cute.Tensor, + sfsc: cute.Tensor, + lgbuf: cute.Tensor, + guw: cute.Tensor, + gusf: cute.Tensor, + dnw: cute.Tensor, + dnsf: cute.Tensor, + hx: cute.Tensor, + dnx: cute.Tensor, + arp: cute.Tensor, + s1w: cute.Tensor, + s2w: cute.Tensor, + img32: cute.Tensor, + b2cnt: cute.Tensor, + stream: cuda_driver.CUstream, +): + tma_a1_desc = cuda.create_tensor_map_tiled( + global_address=a1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H // 2, TWO_I, E_LOCAL], + global_strides=[H // 2 // 16, TWO_I * (H // 2) // 16], + box_dims=(KT_BYTES, R1, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_a2_desc = cuda.create_tensor_map_tiled( + global_address=a2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[I_TP // 2, H, E_LOCAL], + global_strides=[I_TP // 2 // 16, H * (I_TP // 2) // 16], + box_dims=(KT_BYTES, R2, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_s1_desc = cuda.create_tensor_map_tiled( + global_address=sfa1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * TWO_I * (H // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_s2_desc = cuda.create_tensor_map_tiled( + global_address=sfa2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * H * (I_TP // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 4 // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_gu_desc = tma_a1_desc + tma_dn_desc = tma_a2_desc + if cutlass.const_expr(SHARED and not SH_PRETILE): + tma_gu_desc = cuda.create_tensor_map_tiled( + global_address=guw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H, 2 * SH_I], + global_strides=[H // 16], + box_dims=(KT_BYTES, 32), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_dn_desc = cuda.create_tensor_map_tiled( + global_address=dnw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[SH_I, H], + global_strides=[SH_I // 16], + box_dims=(KT_BYTES, 128), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + m3_moe_kernel( + tma_a1_desc, + tma_a2_desc, + tma_s1_desc, + tma_s2_desc, + tma_gu_desc, + tma_dn_desc, + logits, + bias, + x32, + xsf32, + escal, + plan, + partials, + hbuf32, + epochs, + out, + route_dbg, + tdbg, + xb32, + rw, + sfsc, + lgbuf, + guw, + gusf, + dnw, + dnsf, + hx, + dnx, + arp, + s1w, + s2w, + img32, + b2cnt, + ).launch( + grid=(NCTA + NSH, 1, 1), + block=(THREADS, 1, 1), + cluster=(1, 1, 1), + stream=stream, + use_pdl=True, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/op.py new file mode 100644 index 000000000000..66f5b8e6a21c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/op.py @@ -0,0 +1,880 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Host side of the MiniMax-M3 routed-expert kernel on one TP4 rank (routed and shared intermediate 768 per rank), +over the TRTLLM-Gen NVFP4 buffers read in place. + +``m3_moe`` (m3_moe_kernel.py, :class:`M3MoeState`): 1..8 decode tokens (decode batches, spec-decode verify). The call +takes the bf16 hidden states and folds in the router GEMV (bit-exact with m3_router_gemv_fp4) and the activation's +NVFP4 quantization. With ``shared=True`` it also runs the shared expert on dedicated CTAs and returns +bf16(shared + routed); otherwise it returns the rank's routed partial, bit for bit what the stock fused MoE returns. + +Usage (build before CUDA-graph capture; every call is capture-safe afterwards):: + + state = M3MoeState( + device, num_tokens=6, shared=True + ) # workspace + compiled kernel, shared by all layers + layer = state.layer( + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=router_weight, + fc31_input_scale=fc31_input_scale, + shared_weights=(gate_up_w, gate_up_sf, down_w, down_sf), + ) + out = layer(hidden) # bf16 [M, 6144] +""" + +from __future__ import annotations + +import importlib.util +import os +import sys +from pathlib import Path +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled +from .. import m3_report + +HIDDEN = 6144 # capacity: fixed: the checkpoint's hidden size +INTER = 1536 # capacity: fixed: the routed experts' width per tp2 rank +NUM_EXPERTS = 128 +SHARED_INTER = 1536 # capacity: fixed: the shared expert's width per tp2 rank +SHARED_CTAS = 24 # capacity: fixed: an SM split, not a size (m3_moe's shared-expert CTAs (beside num_ctas routed CTAs)) +TOP_K = 4 +U1 = 2 * INTER // 32 # FC1 units per expert +# The router GEMV takes at most 2 experts per CTA (router weights held in registers). +MIN_CTAS = 64 # capacity: fixed: an SM split, not a size + +_UNIT_BYTES = 32 * (HIDDEN // 2 + HIDDEN // 16) # one FC1 unit's weight + scale bytes +_ROW2_BYTES = INTER // 2 + INTER // 16 # one FC2 weight row's bytes (values + scales) +_S1 = 6 # FC1 stages (items) per unit # capacity: fixed: pipeline stages +_ITEM_BYTES = _UNIT_BYTES // _S1 # capacity: fixed: a ring item's bytes +_modules: Dict[tuple, object] = {} +_compiled: Dict[tuple, object] = {} +# m3_moe trace-time options by token count 1..8 (index num_tokens - 1). +# ``sfa_cp``: 18 KB ring stages, the A scales copied by cp.async into the instruction layout (every token count). +SFA_CP_BY_M: Tuple[int, ...] = (1, 1, 1, 1, 1, 1, 1, 1) +# ``b2img``: FC1 -> FC2 hand-off through per-group global B images + release counters. Off at TP4 (at 4 / 8 tokens +# 0.5 / 0.8 us/call slower than the tagged records in the TP4 sweep). +B2IMG_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 0, 0) +# ``ar_pre``: the finalize's all-reduce push with its call parities / descriptor preloaded (all but 2 tokens). +AR_PRE_BY_M: Tuple[int, ...] = (1, 0, 1, 1, 1, 1, 1, 1) +# Ring ``stages`` (0 = the kernel's default, the most that fit). 5 at 8 tokens (with a 50 ns FC2 record poll below) +# measured 0.8 us/call faster than 6 in the TP4 sweep (same MMAs, bit-identical outputs); at 4 tokens the default is +# fastest with heavy_fc2_only. +STAGES_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 0, 5) +# The FC2 B builders' record poll back-off in ns (0 = the kernel's default, 200); 50 at 8 tokens. +B2_POLL_NS_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 0, 50) +# ``early0``: warp 4 streams group 0 right after its own top-k. Off at 8 tokens (0.6 us/layer faster in the chain: +# there it is on every routing warp's critical path and the ring holds only 5 stages). +EARLY0_BY_M: Tuple[int, ...] = (1, 1, 1, 1, 1, 1, 1, 0) +# The plan's ``heavy_fc2_only``: 192 FC2 output blocks over 188 routed CTAs leave 4 CTAs with two blocks, whose +# trailing FC2 after the last hand-off otherwise ends the kernel 3-7 us after the rest; they stream no FC1 instead +# (1.7 / 0.9 us/call faster at 4 / 8 tokens in the TP4 sweep; bit-identical outputs). +HEAVY_FC2_ONLY_BY_M: Tuple[int, ...] = (0, 0, 0, 1, 0, 0, 0, 1) +# ``hb_repl``: copies of the tagged FC1 -> FC2 records (applied when > 1); CTA bx polls copy bx % hb_repl, which +# spreads the polls of the few hot record lines over L2. 7 and 8 tokens hand off through the B images instead. +HB_REPL_BY_M: Tuple[int, ...] = ( + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, +) # off at TP4 (0.2 us/call slower at 4 tokens) +# Per-token-count overrides of the tables above: at 8 tokens the FC1 -> FC2 hand-off through tagged records instead of +# the B image (ported from the GB300 target: 0.15 us/call faster there, paired 4/4 with bit-identical outputs, +# MOE_CLK64_MR6C_READ; it also leaves warp 6 idle at 8 tokens). Kill switch ``TRTLLM_M31_DISABLE_MOE_M8CFG`` restores +# the tables' values. +M8_CFG_BY_M: Dict[int, Dict[str, int]] = {8: {"b2img": 0}} +# SH20: at SH20_TOKENS the shared expert's 24 work units run on +# SH20_CTAS shared CTAs (CTAs 0 .. 3 run two units, both gate_ups before either down), so the routed grid takes the +# 4 SMs this frees: num_ctas = SMs - SH20_CTAS = 192 gives every routed CTA exactly one of the 192 FC2 32-row blocks +# (188 left 4 CTAs with two: the call's stragglers). Bit-identical outputs (only the CTA -> unit mapping and the +# routed CTA count change). Measured (VR TP8, 8 tokens): -0.94 us/call. Kill switch TRTLLM_M31_DISABLE_MOE_SH20, +# read at import. +# TP2 port: BLOCKED until the TP4/TP2 kernel (SH_SPLIT == 1 shared branch) takes the SH20 unit loop +# (kernels/m3_moe/m3_moe_kernel.py is the TP4 kernel here). Held off; the switch name is kept. +SH20 = False and not m31_disabled("MOE_SH20") +SH20_CTAS = 20 # capacity: fixed: an SM split, not a size +SH20_TOKENS = frozenset( + {8} +) # capacity: fixed: the K7 c1 rows (the two-unit CTAs finish off the critical path there) + + +def shared_ctas_for(num_tokens: int, shared: bool) -> int: + """The shared-expert CTAs beside the routed ones of an engine built for ``num_tokens`` (0 without the shared + expert): SH20_CTAS at SH20_TOKENS while SH20 is on, else SHARED_CTAS.""" + if not shared: + return 0 + return SH20_CTAS if SH20 and num_tokens in SH20_TOKENS else SHARED_CTAS + + +# ``lg_repl``: replicas of the router logit exchange at up to LG_REPL_MAX_TOKENS tokens (fewer stores before the +# exchange; bit-identical outputs). Above that the kernel's default applies. +LG_REPL = 32 +LG_REPL_MAX_TOKENS = 0 # capacity: fixed: measured crossover (off at TP4: flat in the sweep) +# The TP4 FLAT stack: the TP2 FLAT values (no B2 images, 16 router-logit replicas, AR push parities loaded in the +# finalize, PDL release right after the grid wait) over the per-m tables above. Bitwise-identical output (m 1..8 x 4 +# routings, eager + graph); standalone m=4 -0.49 us/call (late_trigger). The values are part of the engine's module +# key. +FLAT_STACK = {"b2img": 0, "lg_repl": 16, "ar_pre": 0, "late_trigger": 0} +# FLAT only up to 4 tokens: at 8 tokens the per-m tables are faster (TP2: 2.96 us/call; TP4 in-model at 2 requests +# +175 us/step with FLAT). Bit-identical either way. +FLAT_MAX_TOKENS = 4 # capacity: fixed: measured crossover + + +def flat_applies(num_tokens: int) -> bool: + """Whether an engine state for ``num_tokens`` tokens runs the FLAT stack.""" + return num_tokens <= FLAT_MAX_TOKENS + + +# VRPOLL: 16 replicas instead of the kernel's 32 at 7 and 8 tokens (the K7 c1 rows): the exchange's tail CTAs poll +# fewer lines. Bit-identical outputs. Microbench (VR TP8, CUDA graph, 57 calls): -0.25 us/call at 8 tokens, -0.24 at 7. +# Kill switch TRTLLM_M31_DISABLE_MOE_VRPOLL, read at import. +VRPOLL = not m31_disabled("MOE_VRPOLL") +VRPOLL_LG_REPL = 16 # capacity: fixed: measured optimum (8 / 16 / 64 swept) +VRPOLL_TOKENS = frozenset({7, 8}) # capacity: fixed: the measured rows +# Byte-balanced FC1 ranges below one unit per CTA (:func:`_fine_ranges`) for groups too small to give every CTA a +# whole unit; False gives such groups whole units on the CTAs with the least FC2 work. +FINE_PLAN = True +# HALFPLAN: when every CTA's byte-balanced share of a group is the same and at most half a unit (the partial last FC1 +# group with SH20's 192 routed CTAs at 8 tokens), the fine ranges' boundaries sit on half units (S1 // 2 items), so a +# split unit is halved (3 + 3) instead of extended to its end (1 + 5) and no CTA gets more than half a unit; other +# groups keep their ranges. Bit-identical outputs (microbench: -0.18 us per call at m8 random). Kill switch +# TRTLLM_M31_DISABLE_MOE_HALFPLAN, read at import. +HALFPLAN = not m31_disabled("MOE_HALFPLAN") +# m3_moe phase lag: phase p streams FC1 of group p and FC2 of group p - M3_MOE_LAG (the kernel's default lag). +M3_MOE_LAG = 2 + + +_ENGINE_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "m3_moe_kernel.py") + + +def _kernel_module(config: dict, path: str): + key = (os.path.basename(path),) + tuple(sorted(config.items())) + mod = _modules.get(key) + if mod is None: + name = ( + f"{__name__}_" + + os.path.splitext(os.path.basename(path))[0] + + "_" + + "_".join(f"{k}{v}" for k, v in key[1:]) + ) + spec = importlib.util.spec_from_file_location(name, path) + mod = importlib.util.module_from_spec(spec) + mod.M3_CONFIG = dict(config) + sys.modules[name] = mod + spec.loader.exec_module(mod) + _modules[key] = mod + return mod + + +def _view(t: torch.Tensor, align: int): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align) + + +def is_supported(device: Optional[torch.device] = None) -> tuple: + if not torch.cuda.is_available(): + return False, "no CUDA" + major, _ = torch.cuda.get_device_capability(device) + if major != 10: + return False, "needs the sm_100 family (tcgen05 block-scaled NVFP4 MMA)" + try: + import cutlass # noqa: F401 + except ImportError: + return False, "CuTe DSL (nvidia-cutlass-dsl) is not installed" + return True, "" + + +def ar_push_desc( + mc_ptr: int, flags_ptr: int, rank: int, world: int, max_tokens: int, device +) -> torch.Tensor: + """The ``arp`` descriptor that makes a call also push its bf16 output into a multicast all-reduce exchange (set A + of m3_mc_allreduce's workspace at ``mc_ptr``, slot half = the all-reduce CTA's call parity in ``flags_ptr``). + With it the kernel releases its PDL dependents only after every push; the default (enable 0) keeps the old + behaviour.""" + return torch.tensor( + [1, mc_ptr, flags_ptr, rank, world, max_tokens, 0, 0], dtype=torch.int64, device=device + ) + + +def _shared_views(shared_weights, who): + """(gate_up, its scales, down, its scales) of the shared expert as flat uint8 views, checked.""" + if shared_weights is None: + raise ValueError(f"{who} (shared): shared_weights are required") + gu_w, gu_sf, dn_w, dn_sf = shared_weights + want = {"gate_up": (gu_w, (2 * SHARED_INTER, HIDDEN)), "down": (dn_w, (HIDDEN, SHARED_INTER))} + for name, (t, shape) in want.items(): + if t.dtype != torch.float8_e4m3fn or tuple(t.shape) != shape or not t.is_contiguous(): + raise ValueError(f"{who} (shared): {name} weight must be contiguous e4m3 {shape}") + if ( + gu_sf.numel() != 2 * SHARED_INTER * HIDDEN // 32 + or dn_sf.numel() != HIDDEN * SHARED_INTER // 32 + ): + raise ValueError( + f"{who} (shared): the UE8M0 scales must be the swizzled [N, K / 32] buffers" + ) + return ( + gu_w.view(torch.uint8).view(-1), + gu_sf.view(torch.uint8).reshape(-1), + dn_w.view(torch.uint8).view(-1), + dn_sf.view(torch.uint8).reshape(-1), + ) + + +def _pretile_shared(gu8, gusf8, dn8, dnsf8, n_cta: int = 24) -> torch.Tensor: + """The shared expert's weights as m3_moe's shared CTAs stream them (SH_PRETILE): per CTA s, per stage g, the 16 KB + shared-memory image (128-byte swizzle: 16-byte chunk j of row r at j ^ (r % 8)) of the stage's 128-row MMA tile and + its 512-byte instruction scale atom (row R's 4 codes at word 4 (R % 32) + R / 32). CTA s owns P = SHARED_INTER / 24 + = 16 positions in SPLIT = 128 / (2 P) = 4 K slices of GT = 48 / SPLIT = 12 k-tiles. Stages 0..GT-1: MMA row + R = P O + i (O = 2 q + u: slice q, u 0 gate / 1 up) = weight row u SHARED_INTER + P s + i at k-tile GT q + g; then + down tile j (rows 256 s + 128 j ..) at k-tile kt, j-major (SHARED_INTER / 128 k-tiles each). Built once per + layer.""" + dev = gu8.device + H_, I_ = HIDDEN, SHARED_INTER + P = I_ // n_cta + split = 128 // (2 * P) + gt = H_ // 128 // split + dkt = I_ // 128 + gu = gu8.view(2 * I_, H_) + dn = dn8.view(H_, I_) + gkb, dkb = H_ // 32, I_ // 32 + + def sf_bytes(sf, n, b, kbn): # code of (row n, K block b), broadcast + return sf[ + ((n >> 7) * (kbn // 4) + (b >> 2)) * 512 + (n & 31) * 16 + ((n >> 5) & 3) * 4 + (b & 3) + ] + + r = torch.arange(128, device=dev) + q = r >> 5 + ri = r & 31 + octant = r // P + chunk = torch.arange(8, device=dev) + swz = chunk[None, :] ^ (r[:, None] & 7) # source chunk of image chunk j: j ^ (r % 8) + out = torch.empty( + n_cta, gt + 2 * dkt, 16384 + 512, dtype=torch.uint8, device=dev + ) # capacity: fixed: a workspace of the kernel's fixed geometry + for s in range(n_cta): + # gate_up: MMA row R -> weight row, slice + nrow = (octant & 1) * I_ + P * s + r % P + sl = octant >> 1 + for g in range(gt): + kt = sl * gt + g # per row + tile = gu[ + nrow[:, None], (kt[:, None] * 128 + torch.arange(128, device=dev)[None, :]) + ] # [128, 128] + img = tile.view(128, 8, 16)[r[:, None], swz] # image chunk j = tile chunk j ^ (r % 8) + out[s, g, :16384] = img.reshape(-1) + kb = torch.arange(4, device=dev) + codes = sf_bytes(gusf8, nrow[:, None], kt[:, None] * 4 + kb[None, :], gkb) # [128, 4] + atom = torch.empty(512, dtype=torch.uint8, device=dev) + atom[(ri[:, None] * 16 + q[:, None] * 4 + kb[None, :]).reshape(-1)] = codes.reshape(-1) + out[s, g, 16384:] = atom + for gd in range(2 * dkt): + jt, kt = gd // dkt, gd % dkt + row0 = 256 * s + 128 * jt + tile = dn[row0 : row0 + 128, kt * 128 : (kt + 1) * 128] + img = tile.reshape(128, 8, 16)[r[:, None], swz] + out[s, gt + gd, :16384] = img.reshape(-1) + out[s, gt + gd, 16384:] = dnsf8[ + ((row0 >> 7) * (dkb // 4) + kt) * 512 : ((row0 >> 7) * (dkb // 4) + kt) * 512 + 512 + ] + return out.view(-1) + + +# --------------------------------------------------------------------------------------------------------------------- +# m3_moe (m3_moe_kernel.py): 1..8 tokens per call, group-phased weight stream. +# --------------------------------------------------------------------------------------------------------------------- +# Pre-tiled shared weights per layer, keyed by the source buffers (which the engines read in place): every token-count +# state of a layer streams the same copy. An entry holds its sources, so their addresses are not reused while cached. +_PRETILED: Dict[tuple, tuple] = {} + + +def _pretiled(gu8, gusf8, dn8, dnsf8) -> torch.Tensor: + key = (gu8.device, gu8.data_ptr(), gusf8.data_ptr(), dn8.data_ptr(), dnsf8.data_ptr()) + hit = _PRETILED.get(key) + if hit is None: + hit = _PRETILED[key] = ((gu8, gusf8, dn8, dnsf8), _pretile_shared(gu8, gusf8, dn8, dnsf8)) + return hit[1] + + +# Set by ``_fine_ranges`` when HALFPLAN's half-unit plan is taken; read (and reset) per engine state build. +_HALFPLAN_TAKEN = [False] + + +def _fine_ranges(weights, total: int): + """Contiguous FC1 item ranges [s_c, e_c) over the CTAs, sized as close to ``weights`` (ideal real-valued item + counts summing to ``total``) as the split-unit protocol allows: every unit (S1 items) has at most one CTA boundary + strictly inside it (its head and tail parts, summed by the tail's holder), so a range that starts inside a unit + runs at least to that unit's end. Returns the item counts.""" + n = len(weights) + items = [0] * n + s = 0 + acc = 0.0 + half = _S1 // 2 + if HALFPLAN and n and max(weights) == min(weights) and total % half == 0 and total <= half * n: + _HALFPLAN_TAKEN[0] = True + # equal shares of at most half a unit: boundaries on half units, so every CTA gets 0 or S1 // 2 items, every + # split unit is halved and no range needs extending + for c in range(n): + acc += weights[c] + e = total if c == n - 1 else min(total, max(s, half * int(acc / half + 0.5))) + items[c] = e - s + s = e + assert s == total + return items + for c in range(n): + acc += weights[c] + e = total if c == n - 1 else min(total, max(s, int(acc + 0.5))) + if s % _S1 and e < total: # started inside a unit: run to (at least) its end + e = max(e, min(total, (s // _S1 + 1) * _S1)) + items[c] = e - s + s = e + assert s == total + return items + + +def make_phase_plan( + num_ctas: int, + m_max: int, + lag: int = 2, + tail_w: float = 1.0, + early0: bool = False, + fine: bool = FINE_PLAN, + heavy_fc2_only: bool = False, +): + """The work split of ``m3_moe`` for each distinct-expert count G (4 .. 4 m_max): FC2 32-row output blocks in + contiguous per-CTA ranges (the same for every group), and per group p (slots 4 p ..) a range of its unit-major FC1 + stages per CTA, balancing the bytes each CTA streams in phase p (FC1 of group p plus FC2 of group p - lag, and the + trailing FC2s with the last group). Ranges are empty or >= 6 stages (a unit is split over at most two CTAs); a + group too small for every CTA gets whole units on the CTAs with the least FC2 work, or (``fine``) byte-balanced + ranges of any length with at most one CTA boundary inside a unit (:func:`_fine_ranges`). Returns int32 + [4 m_max + 1, num_ctas, m_max + 1] (per group i0 | i1 << 16, then b0 | nb << 16) and the largest block count.""" + gcap = TOP_K * m_max + nb2 = HIDDEN // 32 + plan = torch.zeros(gcap + 1, num_ctas, m_max + 1, dtype=torch.int32) + nbs = [nb2 // num_ctas + (1 if c < nb2 % num_ctas else 0) for c in range(num_ctas)] + b0s = [sum(nbs[:c]) for c in range(num_ctas)] + # (heavy_fc2_only) CTAs with more FC2 blocks than the minimum stream no FC1: they run each group's FC2 as soon as + # its hand-off lands, so their trailing FC2 after the last hand-off matches the others' two lagged groups + min_nb = min(nbs) + no_fc1 = [heavy_fc2_only and nbs[c] > min_nb for c in range(num_ctas)] + BIG_W = 1 << 40 + blk_bytes = 32 * _ROW2_BYTES # one block, one slot + for g in range(TOP_K, gcap + 1): + ng = (g + 3) // 4 + for c in range(num_ctas): + plan[g, c, m_max] = b0s[c] | (nbs[c] << 16) + for p in range(ng): + n_p = min(4, g - 4 * p) + total = n_p * U1 * _S1 + fc2 = [0] * num_ctas + for c in range(num_ctas): + if p >= lag: + fc2[c] += nbs[c] * min(4, g - 4 * (p - lag)) * blk_bytes + if p == ng - 1 and not ( + early0 and p == 0 and m_max >= 2 + ): # early0: group 0's ranges the same for every G + for q in range(max(0, ng - lag), ng): + fc2[c] += int(tail_w * nbs[c] * min(4, g - 4 * q) * blk_bytes) + for c in range(num_ctas): + if no_fc1[c]: + fc2[c] += BIG_W + items = [0] * num_ctas + if total >= _S1 * num_ctas: + + def counts_for(limit): + return [ + 0 if no_fc1[c] else max(_S1, (limit - f) // _ITEM_BYTES) + for c, f in enumerate(fc2) + ] + + lo, hi = 0, max(f for c, f in enumerate(fc2) if not no_fc1[c]) + total * _ITEM_BYTES + while lo < hi: + mid = (lo + hi) // 2 + if sum(counts_for(mid)) >= total: + hi = mid + else: + lo = mid + 1 + items = counts_for(lo) + extra = sum(items) - total + while extra > 0: + c = min( + (c for c in range(num_ctas) if items[c] > _S1), + key=lambda c: (-(fc2[c] + items[c] * _ITEM_BYTES), c), + ) + items[c] -= 1 + extra -= 1 + elif fine: + # water-fill: CTA c streams fc2[c] + w_c item bytes, equal where w_c > 0 + lo_f, hi_f = 0.0, float(max(fc2) + total * _ITEM_BYTES) + for _ in range(100): + mid = 0.5 * (lo_f + hi_f) + if sum(max(0.0, (mid - f) / _ITEM_BYTES) for f in fc2) >= total: + hi_f = mid + else: + lo_f = mid + wts = [max(0.0, (hi_f - f) / _ITEM_BYTES) for f in fc2] + sw = sum(wts) + items = _fine_ranges([w * total / sw for w in wts], total) + else: + chosen = sorted(sorted(range(num_ctas), key=lambda c: (fc2[c], c))[: total // _S1]) + for c in chosen: + items[c] = _S1 + i0 = 0 + for c in range(num_ctas): + if items[c]: + plan[g, c, p] = i0 | ((i0 + items[c]) << 16) + i0 += items[c] + assert i0 == total + return plan, max(nbs) + + +class M3MoeState: + """Workspace + compiled kernel of ``m3_moe`` (1..8 tokens per call) on one device, shared by the layers that run + on it in one stream order. The call takes the bf16 hidden states and computes the router logits (bit-exact with + m3_router_gemv_fp4) and the NVFP4 activation itself. ``shared``: the shared expert runs on SHARED_CTAS dedicated + CTAs beside the ``num_ctas`` routed ones and the call returns bf16(shared + routed). Build before CUDA-graph + capture; the first call compiles.""" + + fused_router = True + + def __init__( + self, + device: torch.device, + num_tokens: int, + num_ctas: Optional[int] = None, + shared: bool = False, + inter: int = INTER, + ): + """``inter``: the routed intermediate per rank; this engine is built for INTER (768) only.""" + if inter != INTER: + raise ValueError( + f"m3_moe (TP4): routed intermediate {inter} per rank is not supported (only {INTER})" + ) + self.inter = inter + logger.info(f"[M3 MoE] Rubin m3_moe engine (tp4 tables): state for {num_tokens} tokens") + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "M3MoeState allocates and compiles: build it before CUDA-graph capture" + ) + sms_all = torch.cuda.get_device_properties(device).multi_processor_count + sh_ctas = shared_ctas_for(num_tokens, shared) + if shared and (num_ctas or sms_all) + sh_ctas > sms_all: + raise ValueError( + f"m3_moe (shared): num_ctas + {sh_ctas} shared-expert CTAs must fit the {sms_all} SMs " + f"(all resident at once)" + ) + if not 1 <= num_tokens <= 8: + raise ValueError(f"m3_moe takes 1..8 tokens per call, not {num_tokens}") + self.device = torch.device(device) + ok, why = is_supported(self.device) + if not ok: + raise RuntimeError(f"m3_moe: {why}") + sms = torch.cuda.get_device_properties(self.device).multi_processor_count + self.num_ctas = num_ctas or sms + if not MIN_CTAS <= self.num_ctas <= sms: + raise ValueError( + f"m3_moe runs {MIN_CTAS} .. {sms} CTAs (all resident at once), not {self.num_ctas}" + ) + if self.num_ctas > HIDDEN // 32: + # a routed CTA that owns no 32-row FC2 output block deadlocks the FC2 hand-off on its second call + raise ValueError( + f"m3_moe runs at most {HIDDEN // 32} routed CTAs (one FC2 output block each), not {self.num_ctas}" + ) + self.m = num_tokens + # Gate e: the M3.1 items this engine state carries; ``run`` counts each per call. + self.items = set() + _HALFPLAN_TAKEN[0] = False + # One token: the trailing FC2s weigh nothing in the last phase's balance. Group 0's FC1 ranges are the same + # for every distinct-expert count (early0). + plan, nb_cap = make_phase_plan( + self.num_ctas, + num_tokens, + M3_MOE_LAG, + 0.0 if num_tokens == 1 else 1.0, + True, + heavy_fc2_only=bool(HEAVY_FC2_ONLY_BY_M[num_tokens - 1]), + ) + self.config = { + "m_max": num_tokens, + "num_ctas": self.num_ctas, + "nb2_cap": nb_cap, + "fused_router": 1, + "timing": 0, + "i_tp": INTER, + } + if HB_REPL_BY_M[num_tokens - 1] > 1: + self.config["hb_repl"] = HB_REPL_BY_M[num_tokens - 1] + if num_tokens <= LG_REPL_MAX_TOKENS: + self.config["lg_repl"] = LG_REPL + elif VRPOLL and num_tokens in VRPOLL_TOKENS: + self.config["lg_repl"] = VRPOLL_LG_REPL + self.items.add("MOE_VRPOLL") + from tensorrt_llm.logger import logger as _vrpoll_log + + _vrpoll_log.info( + f"MiniMax-M3 MoE engine VRPOLL (TP2): {VRPOLL_LG_REPL} router logit replicas, tokens={num_tokens}" + ) + if shared: + self.config["sh_i"] = SHARED_INTER + self.config["shared"] = 1 + if sh_ctas != SHARED_CTAS: + # The kernel here (the TP4 build) has no SH20 unit loop and ignores ``sh_ctas``: its grid is always + # num_ctas + NSH. Never build an engine whose host buffers assume another shared CTA count. + raise ValueError( + f"m3_moe (TP2): SH20 asks for {sh_ctas} shared CTAs, but this kernel runs the shared expert on " + f"{SHARED_CTAS} (no SH20 unit loop)" + ) + self.shared = shared + # 18 KB ring stages (A scales by cp.async into the instruction layout) per token count, unless configured + self.config["sfa_cp"] = SFA_CP_BY_M[num_tokens - 1] + # FC1 -> FC2 hand-off through per-group global B images + release counters, per token count + self.config["b2img"] = B2IMG_BY_M[num_tokens - 1] + # the finalize's all-reduce push with its call parities / descriptor preloaded, per token count + self.config["ar_pre"] = AR_PRE_BY_M[num_tokens - 1] + if STAGES_BY_M[num_tokens - 1] > 0: + self.config["stages"] = STAGES_BY_M[num_tokens - 1] + if B2_POLL_NS_BY_M[num_tokens - 1] > 0: + self.config["b2_poll_ns"] = B2_POLL_NS_BY_M[num_tokens - 1] + if not EARLY0_BY_M[num_tokens - 1]: + self.config["early0"] = 0 + if flat_applies(num_tokens): + self.config.update(FLAT_STACK) + # engagement line (every rank; once per engine state = once per token count) + from tensorrt_llm.logger import logger as _flog + + _flog.info( + "MiniMax-M3 MoE engine FLAT (TP2): " + + " ".join(f"{k}={self.config[k]}" for k in FLAT_STACK) + + f" tokens={num_tokens}" + ) + else: + from tensorrt_llm.logger import logger as _flog + + _flog.info( + f"MiniMax-M3 MoE engine FLAT (TP2) off above {FLAT_MAX_TOKENS} tokens (per-m tables): " + f"tokens={num_tokens}" + ) + if num_tokens in M8_CFG_BY_M and not m31_disabled("MOE_M8CFG"): + self.config.update(M8_CFG_BY_M[num_tokens]) + self.items.add("MOE_M8CFG") + if _HALFPLAN_TAKEN[0]: + self.items.add("MOE_HALFPLAN") + # E0@m4: at 4 tokens the early group-0 FC1 stages are issued by FC2 B builder warp 8 instead of the routing + # warp 4 (issue order only, bitwise; TP4 -0.19 / TP8 -0.21 us/call). + if num_tokens == 4: + from tensorrt_llm.logger import logger as _e0log + + _e0log.info( + "MiniMax-M3 MoE engine E0M4 (%s): early group-0 stages on warp 8, tokens=4" % "TP2" + ) + try: + self.mod = mod = _kernel_module(self.config, _ENGINE_PATH) + except AssertionError as err: + raise ValueError( + f"m3_moe: num_tokens={num_tokens}, num_ctas={self.num_ctas} does not fit the shared " + f"memory / TMEM budget (fewer CTAs means more FC2 blocks per CTA): {err}" + ) from err + # Every CTA waits on others (cross-CTA spins on partials, hand-off records, the shared expert's h and y): the + # grid must be resident at once, one CTA per SM. The module's shared-memory / TMEM budget assertion makes one + # CTA fit an SM; the grid must not exceed the SMs. (The kernel also traps after HANG_NS in any such spin.) + if self.num_ctas + mod.NSH > sms: + raise ValueError( + f"m3_moe: a grid of {self.num_ctas} + {mod.NSH} CTAs exceeds the {sms} SMs; its CTAs wait on each " + "other and must all be resident at once" + ) + kw = dict(device=self.device) + self.plan = plan.to(self.device) + self.partials = torch.full( + (mod.G_CAP * mod.U1 * num_tokens * 32,), -1, dtype=torch.int32, **kw + ) + # the FC1 -> FC2 records; with HB_REPL > 1 copies, HB_STRIDE int32 words apart + hb_words = mod.G_CAP * num_tokens * mod.U1 * 4 + if mod.HB_REPL > 1: + hb_words = mod.HB_REPL * mod.HB_STRIDE + self.hbuf32 = torch.zeros(hb_words, dtype=torch.int32, **kw) + # One epoch / debug row per CTA of the grid (num_ctas + NSH: the TP4 kernel has no SH20 ``NSHC``). + self.epochs = torch.zeros(self.num_ctas + mod.NSH, dtype=torch.int32, **kw) + self.route_dbg = torch.zeros(num_tokens, 8, dtype=torch.int32, **kw) + self.tdbg = torch.zeros(self.num_ctas + mod.NSH, 64, dtype=torch.int64, **kw) + self.lgbuf = torch.zeros(mod.LG_REPL * mod.LG_STRIDE, dtype=torch.int64, **kw) + self._dummy = torch.zeros(4, dtype=torch.int32, **kw) + self._ar_off = torch.zeros(8, dtype=torch.int64, **kw) + # B2IMG: per-group FC2 B images and their unit counters (int32; every group gains CNT_PER_CALL per call) + self.img32 = torch.zeros(max(1, mod.NG_CAP * mod.IMG_BYTES // 4), dtype=torch.int32, **kw) + self.cnt = torch.zeros(mod.NG_CAP * 8 + 8, dtype=torch.int32, **kw) + # shared expert: h and y_sh as (call tag << 16 | bf16), tag 0 = never written + self.hx = torch.zeros(num_tokens * SHARED_INTER, dtype=torch.int32, **kw) + self.dnx = torch.zeros(num_tokens * HIDDEN, dtype=torch.int32, **kw) + self._fn = None + + def layer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=None, + fc31_input_scale=None, + shared_weights=None, + ) -> "M3MoeLayer": + """One MoE layer's operands (the TRTLLM-Gen NVFP4 expert buffers, read in place, never copied). + ``shared_weights`` (a ``shared`` state): the shared expert's (gate_up e4m3 [3072, 6144], its swizzled UE8M0 + scales, down e4m3 [6144, 768], its swizzled UE8M0 scales).""" + layer = M3MoeLayer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight, + fc31_input_scale, + ) + if self.shared: + layer.shared_weights = _shared_views(shared_weights, "m3_moe") + layer.shared_mxfp8 = tuple(shared_weights) + if self.mod.SH_PRETILE: + # (gate_up, pre-tiled stages, down, dummy): the kernel streams the pre-tiled copy + gu8, _, dn8, _ = layer.shared_weights + layer.shared_weights = ( + gu8, + _pretiled(*layer.shared_weights), + dn8, + self._dummy.view(torch.uint8), + ) + return layer + + def run(self, layer, hidden, out=None, arp=None) -> torch.Tensor: + m = self.m + if ( + hidden is None + or hidden.dtype != torch.bfloat16 + or tuple(hidden.shape) != (m, HIDDEN) + or not hidden.is_contiguous() + ): + raise ValueError( + f"m3_moe (fused router): expects contiguous bf16 hidden states [{m}, {HIDDEN}]" + ) + fz = ( + hidden.view(-1).view(torch.int32), + layer.router_weight.view(-1), + layer.fc31_input_scale.view(-1), + self.lgbuf, + ) + lgv, xq, xs = self._dummy.view(torch.float32), self._dummy, self._dummy + if out is None: + out = torch.empty(m, HIDDEN, dtype=torch.bfloat16, device=self.device) + sh = layer.shared_weights if self.shared else (self._dummy.view(torch.uint8),) * 4 + args = ( + layer.w3_w1_weight, + layer.w3_w1_weight_scale.view(-1), + layer.w2_weight, + layer.w2_weight_scale.view(-1), + lgv, + layer.bias, + xq, + xs, + layer.escal, + self.plan.view(-1), + self.partials, + self.hbuf32, + self.epochs, + out.view(-1), + self.route_dbg.view(-1), + self.tdbg.view(-1), + *fz, + *sh, + self.hx, + self.dnx, + self._ar_off if arp is None else arp, + layer.w3_w1_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + layer.w2_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + self.img32, + self.cnt, + ) + if self._fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_moe compiles on its first call: call it once before capture") + import cutlass.cute as cute + + aligns = [ + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + ] + sig = [_view(t, a) for t, a in zip(args, aligns)] + # keyed on the config the kernel module was built with (not self.config), so a module built with any other + # trace-time config can never reuse another build's compiled kernel + key = (_ENGINE_PATH,) + tuple(sorted(self.mod.M3_CONFIG.items())) + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_moe {os.path.basename(key[0])} {key[1:]}", + lambda: cute.compile( + self.mod.m3_moe, + *sig, + cute.runtime.make_fake_stream(), + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + self._fn = fn + self._fn(*args, torch.cuda.current_stream(self.device).cuda_stream) + for item in sorted(self.items): + m3_report.work(item, f"MoE engine at {m} tokens, width {INTER}") + return out + + +class M3MoeLayer: + """One MoE layer on a :class:`M3MoeState`: ``layer(hidden)``. The expert buffers are read in + place, never copied; the per-expert scalars are packed once (layer constants: rebuild the layer after reloading + weights).""" + + shared_weights = None # the kernel's view of the shared expert (a shared state) + shared_mxfp8 = None # the shared expert's (gate_up, its scales, down, its scales) as given + + def __init__( + self, + state, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + bias, + router_weight=None, + fc31_input_scale=None, + ): + e = NUM_EXPERTS + expected = { + "w3_w1_weight": (w3_w1_weight, (e, 2 * INTER, HIDDEN // 2)), + "w3_w1_weight_scale": (w3_w1_weight_scale, (e, 2 * INTER, HIDDEN // 16)), + "w2_weight": (w2_weight, (e, HIDDEN, INTER // 2)), + "w2_weight_scale": (w2_weight_scale, (e, HIDDEN, INTER // 16)), + } + for name, (t, shape) in expected.items(): + if t.element_size() != 1 or tuple(t.shape) != shape or not t.is_contiguous(): + raise ValueError( + f"m3_moe: {name} is {t.dtype} {tuple(t.shape)}, expected contiguous 1-byte {shape}" + ) + self.state = state + self.w3_w1_weight = w3_w1_weight.view(torch.uint8) + self.w3_w1_weight_scale = w3_w1_weight_scale.view(torch.uint8) + self.w2_weight = w2_weight.view(torch.uint8) + self.w2_weight_scale = w2_weight_scale.view(torch.uint8) + self.escal = torch.stack( + [fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha] + ).float() + self.escal = self.escal.reshape(6, e).contiguous() + self.bias = bias.float().contiguous() + if ( + router_weight is None + or fc31_input_scale is None + or router_weight.dtype != torch.float32 + or tuple(router_weight.shape) != (e, HIDDEN) + or not router_weight.is_contiguous() + or fc31_input_scale.dtype != torch.float32 + or fc31_input_scale.numel() != 1 + ): + raise ValueError( + "m3_moe (fused router): needs router_weight f32 [128, 6144] and fc31_input_scale f32 [1]" + ) + self.router_weight = router_weight + self.fc31_input_scale = fc31_input_scale + + def __call__( + self, + hidden: torch.Tensor, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """``hidden``: contiguous bf16 [M, 6144]. ``arp`` (int64 [8], see :func:`ar_push_desc`): also push the output + into an all-reduce exchange.""" + return self.state.run(self, hidden, out, arp=arp) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/op16.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/op16.py new file mode 100644 index 000000000000..1702e8881128 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe/op16.py @@ -0,0 +1,304 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Host side of ``m3_moe16`` (m3_moe16_kernel.py): the MiniMax-M3 MoE layer (fused router + routed experts + shared +expert) for 9..16 decode tokens on one TP4 rank. The 1..8-token engine (op.M3MoeState) is untouched; select this +state only for num_tokens > 8. + +Output: bf16 [M, 6144] = bf16(shared + routed), the routed part bit for bit TRTLLM-Gen's fused MoE on the fp32 router +GEMV logits (m3_router_gemv's math), the shared part as the in-model m3_mx_gemv shared expert. + + state = M3Moe16State(device, num_tokens=12) # before CUDA-graph capture; one per token count + layer = state.layer(w3_w1_weight, w3_w1_weight_scale, w2_weight, w2_weight_scale, fc31_alpha, fc31_scale_c, + act_alpha, act_beta, act_clamp, fc2_alpha, e_score_correction_bias, + router_weight=router_weight, fc31_input_scale=fc31_input_scale, + shared_weights=(gate_up_w, gate_up_sf, down_w, down_sf)) + out = layer(hidden) # first call compiles: call once before capture + +Protocol state (per state object, shared by the layers that run on it in stream order): per-group FC2-B images and +unit counters, the B1 image + counter, per-CTA epochs. Every call adds exactly 192 to each group counter and NCTA to +the B1 counter, so nothing is reset between calls (graph replay safe). +""" + +from __future__ import annotations + +import os +from pathlib import Path +from typing import Optional + +import torch + +from tensorrt_llm._torch import cute_aot + +from .op import ( + HIDDEN, + SHARED_CTAS, + SHARED_INTER, + U1, + M3MoeLayer, + _compiled, + _kernel_module, + _shared_views, + _view, + is_supported, +) + + +def _group_slots(g: int, p: int, perm: bool, min_ng: int = 5) -> int: + """Slots of group p when the call has g distinct experts (the kernel's slot order).""" + ng = (g + 3) // 4 + tail = g - 4 * (ng - 1) + if p >= ng: + return 0 + if perm and tail < 4 and ng >= max(3, min_ng): + return tail if p == 1 else 4 + return min(4, g - 4 * p) + + +def make_phase_plan16(num_ctas: int, m_max: int, lag: int = 2, perm: bool = True, min_ng: int = 5): + """op.make_phase_plan (early0, tail_w 1) with the kernel's group sizes (PERM: a partial last group is group 1).""" + S1 = 6 + UNIT_BYTES = 32 * (HIDDEN // 2 + HIDDEN // 16) + ITEM = UNIT_BYTES // S1 + blk_bytes = 32 * (768 // 2 + 768 // 16) + gcap = 4 * m_max + nb2 = HIDDEN // 32 + plan = torch.zeros(gcap + 1, num_ctas, m_max + 1, dtype=torch.int32) + nbs = [nb2 // num_ctas + (1 if c < nb2 % num_ctas else 0) for c in range(num_ctas)] + b0s = [sum(nbs[:c]) for c in range(num_ctas)] + for g in range(4, gcap + 1): + ng = (g + 3) // 4 + for c in range(num_ctas): + plan[g, c, m_max] = b0s[c] | (nbs[c] << 16) + for p in range(ng): + n_p = _group_slots(g, p, perm, min_ng) + total = n_p * U1 * S1 + fc2 = [0] * num_ctas + for c in range(num_ctas): + if p >= lag: + fc2[c] += nbs[c] * _group_slots(g, p - lag, perm, min_ng) * blk_bytes + if p == ng - 1 and not (p == 0 and m_max >= 2): + for q in range(max(0, ng - lag), ng): + fc2[c] += nbs[c] * _group_slots(g, q, perm, min_ng) * blk_bytes + items = [0] * num_ctas + if total >= S1 * num_ctas: + + def counts_for(limit): + return [max(S1, (limit - f) // ITEM) for f in fc2] + + lo, hi = 0, max(fc2) + total * ITEM + while lo < hi: + mid = (lo + hi) // 2 + if sum(counts_for(mid)) >= total: + hi = mid + else: + lo = mid + 1 + items = counts_for(lo) + extra = sum(items) - total + while extra > 0: + c = min( + (c for c in range(num_ctas) if items[c] > S1), + key=lambda c: (-(fc2[c] + items[c] * ITEM), c), + ) + items[c] -= 1 + extra -= 1 + else: + for c in sorted(sorted(range(num_ctas), key=lambda c: (fc2[c], c))[: total // S1]): + items[c] = S1 + i0 = 0 + for c in range(num_ctas): + if items[c]: + plan[g, c, p] = i0 | ((i0 + items[c]) << 16) + i0 += items[c] + assert i0 == total + return plan, max(nbs) + + +_ENGINE16_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "m3_moe16_kernel.py") +M16_MIN_TOKENS, M16_MAX_TOKENS = 9, 16 +# Default routed CTAs (beside the SHARED_CTAS shared-expert CTAs). +M16_NUM_CTAS = 128 +# Phase lag: phase p streams FC1 of group p and FC2 of group p - M16_LAG (the kernel's default lag). +M16_LAG = 3 + + +class M3Moe16State: + """Workspace + compiled kernel of ``m3_moe16`` for ``num_tokens`` (9..16) tokens per call.""" + + fused_router = True + shared = True + + def __init__(self, device: torch.device, num_tokens: int, num_ctas: Optional[int] = None): + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "M3Moe16State allocates and compiles: build it before CUDA-graph capture" + ) + if not M16_MIN_TOKENS <= num_tokens <= M16_MAX_TOKENS: + raise ValueError( + f"m3_moe16 takes {M16_MIN_TOKENS}..{M16_MAX_TOKENS} tokens per call, not {num_tokens}" + ) + self.device = torch.device(device) + ok, why = is_supported(self.device) + if not ok: + raise RuntimeError(f"m3_moe16: {why}") + sms = torch.cuda.get_device_properties(self.device).multi_processor_count + self.num_ctas = num_ctas or M16_NUM_CTAS + if not 128 <= self.num_ctas <= sms - SHARED_CTAS: + raise ValueError( + f"m3_moe16 runs 128 .. {sms - SHARED_CTAS} routed CTAs + {SHARED_CTAS} shared (all resident)" + ) + self.m = num_tokens + plan, nb_cap = make_phase_plan16(self.num_ctas, num_tokens, M16_LAG) + self.config = { + "m_max": num_tokens, + "num_ctas": self.num_ctas, + "nb2_cap": nb_cap, + "fused_router": 1, + "timing": 0, + "shared": 1, + } + try: + self.mod = mod = _kernel_module(self.config, _ENGINE16_PATH) + except AssertionError as err: + raise ValueError( + f"m3_moe16: num_tokens={num_tokens} does not fit the smem / TMEM budget: {err}" + ) from err + kw = dict(device=self.device) + self.plan = plan.to(self.device) + # split units' head sums: every slot empty (0xFFFFFFFF) between calls + self.partials = torch.full((mod.G_CAP * U1 * num_tokens * 32,), -1, dtype=torch.int32, **kw) + # FC2-B images per group + the B1 image; counters: group g at 8 g, B1 at 8 NG_CAP, CTA tickets at 8 NG_CAP + 8 + self.img = torch.zeros(mod.IMG_TOTAL // 4, dtype=torch.int32, **kw) + self.cnt = torch.zeros(mod.NG_CAP * 8 + 16, dtype=torch.int32, **kw) + self.epochs = torch.zeros(self.num_ctas + mod.NSH, dtype=torch.int32, **kw) + self.route_dbg = torch.zeros(num_tokens, 8, dtype=torch.int32, **kw) + self.tdbg = torch.zeros(self.num_ctas + mod.NSH, 64, dtype=torch.int64, **kw) + self.lgbuf = torch.zeros( + mod.LG_REPL * mod.LG_STRIDE, dtype=torch.int64, **kw + ) # logits: tag << 32 | f32 + self._ar_off = torch.zeros(8, dtype=torch.int64, **kw) + # shared expert: h and y_sh as (call tag << 16 | bf16), tag 0 = never written + self.hx = torch.zeros(num_tokens * SHARED_INTER, dtype=torch.int32, **kw) + self.dnx = torch.zeros(num_tokens * HIDDEN, dtype=torch.int32, **kw) + self._fn = None + + def layer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=None, + fc31_input_scale=None, + shared_weights=None, + ) -> M3MoeLayer: + layer = M3MoeLayer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight, + fc31_input_scale, + ) + layer.shared_weights = _shared_views(shared_weights, "m3_moe16") + layer.shared_mxfp8 = tuple(shared_weights) + return layer + + def run(self, layer, hidden, out=None, arp=None) -> torch.Tensor: + m = self.m + if ( + hidden is None + or hidden.dtype != torch.bfloat16 + or tuple(hidden.shape) != (m, HIDDEN) + or not hidden.is_contiguous() + ): + raise ValueError(f"m3_moe16: expects contiguous bf16 hidden states [{m}, {HIDDEN}]") + if out is None: + out = torch.empty(m, HIDDEN, dtype=torch.bfloat16, device=self.device) + args = ( + layer.w3_w1_weight, + layer.w3_w1_weight_scale.view(-1), + layer.w2_weight, + layer.w2_weight_scale.view(-1), + layer.bias, + layer.escal, + self.plan.view(-1), + self.partials, + self.img, + self.cnt, + layer.w3_w1_weight_scale.reshape(-1).view(torch.int32), + layer.w2_weight_scale.reshape(-1).view(torch.int32), + layer.w3_w1_weight.reshape(-1).view(torch.int8), + layer.w2_weight.reshape(-1).view(torch.int8), + self.epochs, + out.view(-1), + self.route_dbg.view(-1), + self.tdbg.view(-1), + hidden.view(-1).view(torch.int32), + layer.router_weight.view(-1), + layer.fc31_input_scale.view(-1), + self.lgbuf, + *layer.shared_weights, + self.hx, + self.dnx, + self._ar_off if arp is None else arp, + ) + if self._fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_moe16 compiles on its first call: call it once before capture" + ) + import cutlass.cute as cute + + sig = [_view(t, 16 if t.element_size() * t.numel() % 16 == 0 else 4) for t in args] + key = (_ENGINE16_PATH,) + tuple(sorted(self.config.items())) + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_moe16 {os.path.basename(key[0])} {key[1:]}", + lambda: cute.compile( + self.mod.m3_moe, + *sig, + cute.runtime.make_fake_stream(), + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + self._fn = fn + self._fn(*args, torch.cuda.current_stream(self.device).cuda_stream) + return out + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe_finalize.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe_finalize.py new file mode 100644 index 000000000000..714eafc4661b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_moe_finalize.py @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Exact MoE finalize for the routed experts' unfinalized output, optionally with the shared expert added. + +The TRTLLM-Gen MoE runner with ``do_finalize=False`` returns the FC2 output in permuted order, the per-token top-k +routing weights and ``expanded_idx_to_permuted_idx``. Its own finalize (``finalizeKernelVecLoad``) computes, per token +and hidden element, ``acc = 0; acc += weight_k * fc2[permuted_idx_k]`` for k in order (skipping ``-1``) in float32 +and rounds to BF16 once, but issues the k gathers one after another, so at prefill sizes it runs ~2.6x its byte +floor (153 us per layer at 16K rows on SM107). This kernel issues the gathers together and keeps the per-element +order, so the result is bitwise the runner's: a BF16 x BF16 product is exact in float32, so fused and unfused +multiply-add agree as well. ``shared`` adds the shared expert as ``shared_output.add_(routed_output)`` does +(``bf16(float(shared) + float(routed))``), which removes that separate pass over the hidden states. +""" + +from typing import Optional + +import torch +import triton +import triton.language as tl + +#: (BLOCK_H, num_warps): the tile only splits the hidden dimension, so it never changes the bits. Tuned on SM107 +#: (2K / 4K / 16K rows: 11.2 / 20.5 / 74.9 us, 1.52 / 1.39 / 1.27x the byte floor; the runner's finalize +#: 23 / 40 / 153 us). +CONFIGS = ((2048, 4),) + + +@triton.jit +def _moe_finalize_kernel(fc2, idx, weights, shared, out, ld_fc2, H: tl.constexpr, TOPK: tl.constexpr, + BLOCK_H: tl.constexpr, SHARED: tl.constexpr): # fmt: skip + t = tl.program_id(0).to(tl.int64) + cols = tl.program_id(1) * BLOCK_H + tl.arange(0, BLOCK_H) + cmask = cols < H + acc = tl.zeros((BLOCK_H,), dtype=tl.float32) + for k in tl.static_range(TOPK): + p = tl.load(idx + t * TOPK + k) + w = tl.load(weights + t * TOPK + k).to(tl.float32) + valid = p >= 0 + x = tl.load( + fc2 + tl.maximum(p, 0).to(tl.int64) * ld_fc2 + cols, mask=cmask & valid, other=0.0 + ) + acc = tl.where(valid, tl.fma(w, x.to(tl.float32), acc), acc) + r = acc.to(tl.bfloat16) + if SHARED: + s = tl.load(shared + t * H + cols, mask=cmask, other=0.0) + r = (s.to(tl.float32) + r.to(tl.float32)).to(tl.bfloat16) + tl.store(out + t * H + cols, r, mask=cmask) + + +def m3_moe_finalize( + fc2: torch.Tensor, + weights: torch.Tensor, + expanded_idx_to_permuted_idx: torch.Tensor, + shared: Optional[torch.Tensor] = None, +) -> Optional[torch.Tensor]: + """BF16 [tokens, H] = sum_k weights[:, k] * fc2[expanded_idx_to_permuted_idx[:, k]] (+ ``shared``), bitwise to the + TRTLLM-Gen finalize (and to ``shared.add_(that)``); None when the operands are not the runner's layout.""" + if ( + fc2.dim() != 2 + or fc2.dtype != torch.bfloat16 + or fc2.stride(1) != 1 + or weights.dim() != 2 + or expanded_idx_to_permuted_idx.shape != weights.shape + or expanded_idx_to_permuted_idx.dtype != torch.int32 + or not weights.is_contiguous() + or not expanded_idx_to_permuted_idx.is_contiguous() + or ( + shared is not None + and ( + shared.dtype != torch.bfloat16 + or not shared.is_contiguous() + or shared.shape != (weights.shape[0], fc2.shape[1]) + ) + ) + ): + return None + tokens, topk = weights.shape + hidden = fc2.shape[1] + out = torch.empty((tokens, hidden), dtype=torch.bfloat16, device=fc2.device) + if tokens == 0: + return out + block_h, warps = CONFIGS[0] + _moe_finalize_kernel[(tokens, triton.cdiv(hidden, block_h))]( + fc2, + expanded_idx_to_permuted_idx, + weights, + shared if shared is not None else out, + out, + fc2.stride(0), + hidden, + topk, + block_h, + shared is not None, + num_warps=warps, + ) + return out diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_nvfp4_index_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_nvfp4_index_decode.py new file mode 100644 index 000000000000..040354778413 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_nvfp4_index_decode.py @@ -0,0 +1,121 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""AGG_NVFP4_INDEXK decode reader: the pure-generation index scores of an NVFP4 index-K cache. + +The vendored Q8KV4 decode indexer (``fmha_sm100_q8kv8``: E4M3 index-Q x NVFP4 index-K, the CuTe DSL kernel at two +index heads per rank) scores each request's history pages and writes them straight into the select layout +[index heads, pages, tokens] (``run_scores_hpr``), the layout of the FP8 path's ``max_score``; the served selector +then takes the top 16 exactly as on FP8 pages. The wrapper's ``plan()`` refuses CUDA graph capture, so its device +part (the per-request history-page prefix sum) runs here, inside the captured step, and the kernel is compiled before +warmup (``prebuild``). Scores are raw Q.K (no scale), as the FP8 scorers'. +""" + +from __future__ import annotations + +from typing import Dict, Tuple + +import torch + +PAGE = 128 + + +def _interface(): + from fmha_sm100_q8kv8 import q8_indexer_interface as qi + + return qi + + +class Nvfp4IndexDecodeScorer: + """One per target; holds a wrapper per device and nothing step-dependent.""" + + def __init__(self, num_heads: int): + self.num_heads = int(num_heads) + self._wrappers: Dict[torch.device, object] = {} + + def available(self) -> bool: + try: + qi = _interface() + except ImportError: + return False + return hasattr(qi.BatchDecodeIndexerQ8KV4Wrapper, "run_scores_hpr") and hasattr( + qi, "q8kv4_hpr_buffer" + ) + + def _wrapper(self, device: torch.device): + w = self._wrappers.get(device) + if w is None: + w = _interface().BatchDecodeIndexerQ8KV4Wrapper(num_heads=self.num_heads) + self._wrappers[device] = w + return w + + def score( + self, + idx_q: torch.Tensor, + cache: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + query_len: int, + workspace: torch.Tensor = None, + ) -> torch.Tensor: + """The scores of one decode step, [heads, C, B * query_len] fp32. + + ``idx_q`` [B * query_len, heads, 128] E4M3; ``cache`` [pages, 1, 128, 72] uint8; ``block_table`` [B, C] int32; + ``seq_lens`` [B] int32. History pages are written; a row's local and later pages are left as allocated, which + the selector never ranks. + """ + qi = _interface() + batch, cols = int(block_table.shape[0]), int(block_table.shape[1]) + device = idx_q.device + _, view, kernel_out = qi.q8kv4_hpr_buffer( + batch, cols, self.num_heads, int(query_len), device + ) + if workspace is None: + workspace = self.history_workspace(seq_lens, cols) + w = self._wrapper(device) + w._block_table = block_table + w._seq_lens = seq_lens + w._workspace = workspace + w._query_len = int(query_len) + w._scores = torch.empty( + (batch, 0, 0), dtype=torch.float32, device=device + ) # run_scores_hpr's row count + w.run_scores_hpr( + idx_q, cache.reshape(int(cache.shape[0]), PAGE, int(cache.shape[-1])), kernel_out + ) + return view + + @staticmethod + def history_workspace(seq_lens: torch.Tensor, cols: int) -> torch.Tensor: + """The plan's device part: each request's history-page count (pages before its local one), prefix-summed, as + the kernel's scheduler workspace (uint8 [(B + 1) * 4], int32 view). It depends on the step only (``seq_lens`` + and the table width), so a caller scoring every layer of one step may build it once and pass it to ``score`` + (NVFP4_STEP_CACHE); the values are the same as building it per call.""" + batch = int(seq_lens.shape[0]) + workspace = torch.empty(((batch + 1) * 4,), dtype=torch.uint8, device=seq_lens.device) + scheduler = workspace.view(torch.int32) + history = torch.div(seq_lens - 1, PAGE, rounding_mode="floor").clamp_(min=0, max=cols) + scheduler[:1].zero_() + torch.cumsum(history, dim=0, out=scheduler[1:]) + return workspace + + def prebuild(self, device: torch.device, query_lens) -> Tuple[int, ...]: + """Compile the kernel for every decode query length before warmup and capture; returns them. + + The query length is the compile key (batch and pages are runtime); each runs once on a two-request step. + """ + built = [] + for qlen in sorted({int(q) for q in query_lens}): + batch, pages = 2, 4 + cache = torch.zeros((pages * batch, 1, PAGE, 72), dtype=torch.uint8, device=device) + table = torch.arange(pages * batch, dtype=torch.int32, device=device).view(batch, pages) + lens = torch.full((batch,), pages * PAGE, dtype=torch.int32, device=device) + q = torch.zeros((batch * qlen, self.num_heads, 128), device=device).to( + torch.float8_e4m3fn + ) + self.score(q, cache, table, lens, qlen) + built.append(qlen) + torch.cuda.synchronize(device) + return tuple(built) + + +__all__ = ["Nvfp4IndexDecodeScorer"] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_prequant.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_prequant.py new file mode 100644 index 000000000000..48a7827f59b3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_prequant.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Hand-off of an MXFP8-quantized activation from the kernel that produced it to the next MXFP8 GEMM. + +A producer that already holds a BF16 row in registers (the multicast all-reduce + RMSNorm, the split-K attention +merge) also writes its MXFP8 form (E4M3 rows + UE8M0 scales in the 128x4-swizzled layout of +``trtllm::mxfp8_quantize(x, True)``) and publishes it here keyed by the BF16 tensor. The MXFP8 linear whose input +is that tensor takes it and skips its own quantize kernel. Every take clears the slot, so a hand-off is only ever +consumed by the first MXFP8 linear that runs after the producer. + +Module state shared across files (one instance each per process): + +* ``SLOT`` (PrequantSlot): published by ``m3_mc_allreduce`` (``quant=True``), the sparse-decode merge and the fused + decode attention (MXFP8 output); cleared by the sparse decode before it runs; taken by the MXFP8 linear dispatch + and by the one-launch QKV + attention + o_proj path (which also peeks ``pending``). +* ``AR_PUSH`` (ArPushSlot): armed by the fast forward around the attention call; the o_proj GEMV takes the target + and marks ``pushed``; the fast forward checks ``take_pushed`` before the attention boundary all-reduce. +* ``MOE_PUSH`` (MoePushSlot): armed by the fast forward before the MoE; the routed-expert engine takes ``engine`` + and the shared expert's down projection takes ``shared``, each marks its pushed output; the fast forward + checks ``take_pushed`` before the MoE boundary all-reduce. +""" + + +class PrequantSlot: + def __init__(self): + self.pending = None # (data_ptr, rows, width, e4m3 [rows, width], scales) + + def publish(self, x, xq, xsf): + self.pending = (x.data_ptr(), x.shape[0], x.shape[-1], xq, xsf) + + def clear(self): + self.pending = None + + def take(self, x): + """(e4m3 [x.shape[0], width], scales) if x is the published tensor, else None.""" + pending, self.pending = self.pending, None + if ( + pending is None + or x.dim() != 2 + or x.data_ptr() != pending[0] + or x.shape[0] != pending[1] + or x.shape[1] != pending[2] + ): + return None + return pending[3], pending[4] + + +SLOT = PrequantSlot() + + +class ArPushSlot: + """The attention boundary's all-reduce exchange for the o_proj GEMV to push into. + + The fast forward arms ``target`` (m3_mx_gemv ``ar`` tuple) around the attention call; the o_proj marks the output + it pushed in ``pushed``; the boundary then runs the all-reduce with ``prepushed`` = the ``ar`` tuple's mode.""" + + def __init__(self): + self.target = None + self.pushed = None # data_ptr of the pushed output + + def arm(self, target): + self.target = target + self.pushed = None + + def take_target(self): + target, self.target = self.target, None + return target + + def take_pushed(self, x): + pushed, self.pushed = self.pushed, None + return pushed is not None and x.data_ptr() == pushed + + +AR_PUSH = ArPushSlot() + + +class MoePushSlot: + """The MoE boundary's all-reduce exchange for its two producers to push into: the routed-expert engine (set A, + ``engine`` = its ``arp`` descriptor) and the shared expert's down projection (set B, ``shared`` = m3_mx_gemv + ``ar`` tuple). Each marks the output it pushed; the boundary then runs the all-reduce with ``prepushed`` = 3.""" + + def __init__(self): + self.engine = None + self.shared = None + self.engine_pushed = None # data_ptr of the pushed routed output + self.shared_pushed = None # data_ptr of the pushed shared output + + def arm(self, engine, shared): + self.engine, self.shared = engine, shared + self.engine_pushed = self.shared_pushed = None + + def disarm(self): + self.engine = self.shared = None + + def take_engine(self): + engine, self.engine = self.engine, None + return engine + + def take_shared(self): + shared, self.shared = self.shared, None + return shared + + def take_pushed(self, shared, routed): + """(routed pushed, shared pushed) for these outputs; clears the marks.""" + e = routed is not None and self.engine_pushed == routed.data_ptr() + s = shared is not None and self.shared_pushed == shared.data_ptr() + self.engine_pushed = self.shared_pushed = None + return e, s + + +MOE_PUSH = MoePushSlot() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_report.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_report.py new file mode 100644 index 000000000000..e2696e608e1f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_report.py @@ -0,0 +1,108 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The fast-path reports a kernel module makes from inside a call: ``taken`` (an engagement line), ``missed`` (a covered +shape that cannot run: raises under ``modeling_v2='require'``), ``uncovered`` and ``capture_missed``. + +The policy (logging once, raising under require) is the target's: ``modeling.py`` installs its functions here at import, +so a kernel reports without importing the target module. Until then every report is a no-op. + +``work`` is the gate-e work-done counter (no install needed): the code that does an item's work calls +``work(item, detail)`` from its host code each time it does it. Per rank, the first call logs the item's engagement +line and every power-of-two count logs the running total; the totals are logged once more at process exit. Under CUDA +graphs the host code runs at capture and on eager steps only, so the count is captures + eager calls (replays are not +counted), and every line says so. +""" + +from __future__ import annotations + +import atexit +import threading +from typing import Callable, Dict, Optional + + +def _noop(*args, **kwargs) -> None: + return None + + +taken: Callable[..., None] = _noop +missed: Callable[..., None] = _noop +uncovered: Callable[..., None] = _noop +capture_missed: Callable[..., None] = _noop + + +def install( + *, + on_taken: Callable[[str, Optional[int], str], None], + on_missed: Callable[[str, Optional[int], str], None], + on_uncovered: Callable[[str, str], None], + on_capture_missed: Callable[[str, Optional[int], str], None], +) -> None: + """Route this module's reports to the target's policy.""" + global taken, missed, uncovered, capture_missed + taken, missed, uncovered, capture_missed = on_taken, on_missed, on_uncovered, on_capture_missed + + +#: The tag every work line carries (the eng kit greps it with the item id). +TAG = "[M3 sm107 tp2]" +#: What a count counts (said in every line). +HOST_COUNT = "host count: CUDA-graph captures + eager calls, replays not counted" + +_work_lock = threading.Lock() +_work_counts: Dict[str, int] = {} +_work_exit_registered = [False] + + +def _rank() -> int: + from tensorrt_llm._utils import mpi_rank + + return mpi_rank() + + +def _info(message: str) -> None: + from tensorrt_llm.logger import logger + + logger.info(message) + + +def work(item: str, detail: str = "") -> int: + """Count one unit of ``item``'s work on this rank and return the running count. Logs the engagement line at the + first count (with ``detail``) and the running total at every power of two.""" + with _work_lock: + count = _work_counts.get(item, 0) + 1 + _work_counts[item] = count + if not _work_exit_registered[0]: + _work_exit_registered[0] = True + atexit.register(log_work_totals) + if count & (count - 1) == 0: + if count == 1: + what = f" ({detail})" if detail else "" + _info(f"{TAG} rank {_rank()}: {item} engaged{what}; work 1 ({HOST_COUNT})") + else: + _info(f"{TAG} rank {_rank()}: {item} work {count} ({HOST_COUNT})") + return count + + +def work_count(item: str) -> int: + """This rank's running count of ``item`` (0 if it never ran).""" + with _work_lock: + return _work_counts.get(item, 0) + + +def work_totals() -> Dict[str, int]: + """Every item's running count on this rank.""" + with _work_lock: + return dict(_work_counts) + + +def log_work_totals() -> None: + """The final per-rank counts, one line (registered at exit by the first ``work`` call).""" + totals = work_totals() + if totals: + body = ", ".join(f"{k}={v}" for k, v in sorted(totals.items())) + _info(f"{TAG} rank {_rank()}: work-done final ({HOST_COUNT}): {body}") + + +def reset_work() -> None: + """Forget every count (tests).""" + with _work_lock: + _work_counts.clear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_sampled_accept.py new file mode 100644 index 000000000000..dbeb5c8850a6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_sampled_accept.py @@ -0,0 +1,174 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused sampled verification for the MiniMax-M3 Eagle3 target head (TP-sharded vocabulary, decode rows <= 48). + +Replaces, per decode step, the full-vocabulary all-gather of the target logits, the fused sampling kernel and the +strict-acceptance ops for rows sampled with temperature and top-p (the AgentX contract). Rows with top-k or min-p +active must stay on the generic path; the caller decides. + +Sampling is top-p rejection sampling, distributed over the TP ranks. With w = exp(logit / T - max) over the FULL +vocabulary and Z = sum(w), a token is in the nucleus iff the mass strictly above its weight is at most top_p * Z +(the fused sampling kernel's kept set, ties included). A candidate is drawn proportionally to w and accepted iff it +is in the nucleus; a rejected candidate lies below the nucleus, so the next candidate is drawn over the tokens heavier +than it. The accepted token is distributed as w restricted to the nucleus. The first candidate lands in the nucleus +with probability >= top_p, so most rows decide on their second exchange. + +Per verify row, 8 CTAs (rows <= 32) or 4 CTAs (more rows) on every rank each own a slice of the rank's logits shard, +staged in shared memory once. A candidate is drawn in two stages without per-token noise: each CTA draws a local +candidate by inverse CDF over its slice, and the CTA whose log-mass plus Gumbel noise is largest supplies the +candidate (CTA j wins with probability T_j / sum T, its token with w / T_j). Uniforms come from a counter-based Philox +keyed by the request's seed and offset, the row, the attempt and the CTA. Each round every CTA multicasts one 4-word +record (key, candidate logit, and max and mass in round 0 or the mass above the current candidate later) and reads +the records of all CTAs of the row on all ranks. Every CTA takes the same decision from the same records in the same +order, so all ranks agree without any intra-GPU handshake. Words carry the call's tag: a slot needs no clearing and +two call halves keep a rank one call ahead from overwriting a slot still being read. +""" + +from typing import Optional + +import torch + +from ..ops.spec.m3_sampled_accept import m3_sampled_accept as _sampled_accept_op +from ..ops.spec.m3_sampled_accept_ws_words import m3_sampled_accept_ws_words as _ws_words_op + +# The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.cu: a row's CTAs +# launch as one thread-block cluster, bounded record polls) and reached through the ops/spec entry m3_sampled_accept. +MAX_ROWS = 48 # capacity: fixed: the kernel's built rows (decode steps) + + +class SampledAcceptWorkspace: + """Exchange slots behind one multicast mapping (two call halves), the (finished rows, call) counters, the per-row + token scratch, an extra-attempt counter and the SA-1R round-0 decided counter. Created collectively, outside + CUDA-graph capture.""" + + def __init__(self, uc, mc_ptr, rank, world, handle=None, comm=None): + self.uc, self.mc_ptr, self.rank, self.world = uc, mc_ptr, rank, world + self.handle, self.comm = handle, comm + dev = uc.device + self.counters = torch.zeros(2, dtype=torch.int32, device=dev) + self.tokens = torch.zeros(MAX_ROWS, dtype=torch.int32, device=dev) + self.extra_attempts = torch.zeros(1, dtype=torch.int32, device=dev) + self.decided = torch.zeros(1, dtype=torch.int32, device=dev) + + @classmethod + def create(cls, mapping, device, one_round_k: int) -> "SampledAcceptWorkspace": + # SA-1R's records carry the round-0 list words: the size depends on the variant (``one_round_k``). + words = int(_ws_words_op(max(mapping.tp_size, 1), one_round_k)) + if mapping.tp_size == 1: + uc = torch.zeros(words, dtype=torch.int64, device=device) + return cls(uc, uc.data_ptr(), 0, 1) + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "SampledAcceptWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 8, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("SampledAcceptWorkspace.create failed on another rank") + return cls(uc, mc.data_ptr(), mapping.tp_rank, mapping.tp_size, handle, comm) + + +_WS: Optional[SampledAcceptWorkspace] = None + + +def workspace(mapping, device, one_round_k: int) -> Optional[SampledAcceptWorkspace]: + """The process-wide workspace for the SA-1R variant ``one_round_k`` (the same on every call of a process).""" + global _WS + if _WS is None: + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = SampledAcceptWorkspace.create(mapping, device, one_round_k) + return _WS + + +def sampled_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + temperatures: torch.Tensor, + top_ks: torch.Tensor, + top_ps: torch.Tensor, + min_ps: Optional[torch.Tensor], + seeds: torch.Tensor, + offsets: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, + ws: Optional[SampledAcceptWorkspace] = None, + use_pdl: bool = True, + *, + one_round_k: int, + prewait_params: bool, + early_trigger: bool, + ilp_walk: bool, +): + """Sampled verification (temperature + top-p) from this rank's bf16 logits shard [nreq * (draft_len + 1), V / tp]: + outputs as shard_accept (accepted [nreq, D+1], num_accepted [nreq], gather_ids [nreq]); None + when unavailable. Per-row sampling parameters and Philox (seed, offset) are the spec metadata's, one per row. + top_ks / min_ps are not applied: rows with top-k or min-p active must be routed to the generic path by the + caller. ``one_round_k`` (SA-1R: 0, 9 or 82), ``prewait_params``, ``early_trigger`` and ``ilp_walk`` select the + kernel variant (the target's constants).""" + del top_ks, min_ps + nreq, draft_len = draft_tokens.shape + rows = nreq * (draft_len + 1) + if logits.dim() != 2 or logits.shape[0] != rows or rows > MAX_ROWS or logits.shape[1] > 51200: + return None + if logits.dtype != torch.bfloat16 or logits.stride(1) != 1: + return None + ws = ws if ws is not None else workspace(mapping, logits.device, one_round_k) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + _sampled_accept_op( + logits, + int(vocab_offset), + temperatures[:rows].float().contiguous(), + top_ps[:rows].float().contiguous(), + seeds[:rows].long().contiguous(), + offsets[:rows].long().contiguous(), + ws.uc, + ws.mc_ptr, + ws.counters, + ws.tokens, + ws.rank, + ws.world, + draft_tokens.int().contiguous(), + batch_idx.int().contiguous(), + accepted, + num_accepted, + gather_ids, + int(force_total), + next_new, + force_vec, + ws.extra_attempts, + bool(use_pdl), + prewait_params=bool(prewait_params), + early_trigger=bool(early_trigger), + one_round_k=int(one_round_k), + decided=ws.decided, + ilp_walk=bool(ilp_walk), + ) + return accepted, num_accepted, gather_ids diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_select_long_dsl.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_select_long_dsl.py new file mode 100644 index 000000000000..106a0ab95862 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_select_long_dsl.py @@ -0,0 +1,382 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 block select for MiniMax-M3 decode rows of up to 9216 blocks, one row per 384-thread CTA (CuTe DSL). + +Bit for bit the selection of this target's ``m3_select_long`` (``torch.ops.trtllm.minimax_m3_sm107_tp4_select_long``, +one 1024-thread CTA per row), from a CTA of 384 threads: + +* Every block's score is forced (the first ``init_blocks`` and the last ``local_blocks`` of the row's valid blocks) and + mapped to an order key held in registers: thread t owns blocks t, t + 384, ... (4, 11 or 24 of them by the row's + valid count). Larger scores get smaller keys; NaN sorts above +inf. The key is the 1024-thread kernel's unsigned + key with its top bit flipped, so signed Int32 order here is that kernel's order. Every thread loads at a clamped + block and masks the result, so a thread's loads issue back to back. +* 384 is a multiple of 16, so the 16 interleaved groups ``block % 16`` are the lanes' ``lane % 16``. The weakest of + the 16 group maxima bounds the 16th-largest score, so the top 16 is among the keys at or above it. +* The candidates are compacted by a CTA prefix count, each as one int64 (key << 32 | block), and ranked exactly by + that value (one signed compare per pair); more than 1024 of them (keys tied at the bound) rank every candidate + against the whole row instead. +* Warp 0 emits the 16 picked block ids sorted ascending, -1 for an empty or -inf pick. + +The kernel waits on its grid dependency first; it releases the dependent sparse decode right after that wait +(``early_trigger``), once every score of its row is loaded (trigger phase 1) or at its exit (phase 0). Kernels are +compiled per (PDL launch, early trigger, trigger phase) with symbolic shapes and strides; ``prebuild`` compiles and +loads them before the executor's warmup, so a decode step never compiles one. +""" + +from pathlib import Path +from typing import Dict, Tuple + +import cutlass +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream, make_fake_tensor +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + griddepcontrol_launch_dependents, + griddepcontrol_wait, +) + +SELECT_THREADS = 384 +SELECT_WARPS = SELECT_THREADS // 32 +# 1M tokens + draft tokens at 128 tokens per block. +MAX_BLOCKS = 9216 # capacity: fixed: the kernel's built blocks per row +MAX_CAND = 1024 # capacity: fixed: the kernel's candidate staging +TOPK = 16 # capacity: fixed: MSA's top-k +NO_KEY = 0x7FFFFFFF # capacity: fixed: a sentinel, not a size (an unused key slot (the reference's 0xffffffff)) +# The key of -inf (the reference's 0xff800000): never emitted. +NEG_INF_KEY = 0x7F800000 # capacity: fixed: a key encoding, not a size +INIT_SCORE = 1.0e30 +LOCAL_SCORE = 1.0e29 +# (keys per thread, rows up to this many valid blocks) +ITEM_CLASSES = ((4, 4 * SELECT_THREADS), (11, 11 * SELECT_THREADS), (24, MAX_BLOCKS)) + +# Shared memory of the phase, int32 words: candidates [MAX_CAND] int64 (key << 32 | block, so one signed compare +# orders them by (key, block)), picked keys / blocks [TOPK] each, per-warp group minima [SELECT_WARPS][16], group +# minima [16], per-warp candidate counts [SELECT_WARPS]. +_CAND = 0 +_PICK_KEY = _CAND + 2 * MAX_CAND +_PICK_BLOCK = _PICK_KEY + TOPK +_WARP_MIN = _PICK_BLOCK + TOPK +_GROUP_MIN = _WARP_MIN + SELECT_WARPS * 16 +_WARP_CNT = _GROUP_MIN + 16 +SMEM_WORDS = _WARP_CNT + SELECT_WARPS +SMEM_BYTES = 4 * SMEM_WORDS + + +def _key(score): + """Order key of a float score: smaller key = larger score, NaN first; signed order = the reference's unsigned.""" + bits = cutlass.Float32(score).bitcast(cutlass.Int32) + return cutlass.select_(bits < cutlass.Int32(0), bits & cutlass.Int32(0x7FFFFFFF), ~bits) + + +def _imin(a, b): + return cutlass.select_(b < a, b, a) + + +def _imax(a, b): + return cutlass.select_(b > a, b, a) + + +def _forced_key(scores, head, block, query, init_blocks, local_start): + raw = scores[head, block, query] + s = cutlass.select_( + block >= local_start, + cutlass.Float32(LOCAL_SCORE), + cutlass.select_(block < init_blocks, cutlass.Float32(INIT_SCORE), cutlass.Float32(raw)), + ) + return _key(s) + + +@cute.jit +def _select_row( + row, + scores: cute.Tensor, # float32 [num_kv_heads, num_blocks, total_q], any strides + n_valid: cute.Tensor, # int32 [>= total_q] + out: cute.Tensor, # int32 [total_q, num_kv_heads, 16] + init_blocks, + local_blocks, + num_kv_heads, + smem: cute.Tensor, # int32 [SMEM_WORDS], shared memory + bar_id: cutlass.Constexpr, + loaded_trigger: cutlass.Constexpr, + raw_prewait, + n_valid_prewait: cutlass.Constexpr, +): + """Select row ``row`` (= query * num_kv_heads + head) with this CTA's SELECT_THREADS threads. Uses named barrier + ``bar_id`` over SELECT_THREADS threads. ``loaded_trigger`` releases the dependent grid once every score of the + row is loaded (trigger phase 1). With ``n_valid_prewait`` the row's valid-block count is ``raw_prewait``, read by + the caller before its grid wait; otherwise it is read here.""" + tid, _, _ = cute.arch.thread_idx() + lane = tid % cutlass.Int32(32) + warp = tid // cutlass.Int32(32) + query = row // num_kv_heads + head = row % num_kv_heads + if cutlass.const_expr(n_valid_prewait): + raw = raw_prewait + else: + raw = n_valid[query] + valid = _imax(_imin(raw, cutlass.Int32(scores.shape[1])), cutlass.Int32(0)) + local_start = _imax(raw - local_blocks, cutlass.Int32(0)) + cand = cute.make_tensor( + cute.recast_ptr(smem.iterator + _CAND, dtype=cutlass.Int64), cute.make_layout(MAX_CAND) + ) + + if tid < TOPK: + smem[_PICK_KEY + tid] = cutlass.Int32(NO_KEY) + smem[_PICK_BLOCK + tid] = cutlass.Int32(0) + + if valid <= TOPK: + # Every valid block is picked (thread t holds block t). + if tid < valid: + smem[_PICK_KEY + tid] = _forced_key(scores, head, tid, query, init_blocks, local_start) + smem[_PICK_BLOCK + tid] = tid + if cutlass.const_expr(loaded_trigger): + griddepcontrol_launch_dependents() + for ci in cutlass.range_constexpr(len(ITEM_CLASSES)): + items = ITEM_CLASSES[ci][0] + lo = TOPK if ci == 0 else ITEM_CLASSES[ci - 1][1] + hi = ITEM_CLASSES[ci][1] + if (valid > lo) & (valid <= hi): + key = cute.make_rmem_tensor(items, cutlass.Int32) + local_min = cutlass.Int32(NO_KEY) + for s in cutlass.range_constexpr(items): + # Every thread loads (a clamped block past the row) so the loads issue back to back, unbranched. + block = tid + cutlass.Int32(s * SELECT_THREADS) + k = _forced_key( + scores, head, _imin(block, valid - 1), query, init_blocks, local_start + ) + key[s] = cutlass.select_(block < valid, k, cutlass.Int32(NO_KEY)) + local_min = _imin(local_min, key[s]) + local_min = _imin(local_min, cute.arch.shuffle_sync_bfly(local_min, 16)) + if lane < 16: + smem[_WARP_MIN + warp * cutlass.Int32(16) + lane] = local_min + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + if cutlass.const_expr(loaded_trigger): + griddepcontrol_launch_dependents() + if tid < 16: + g = cutlass.Int32(NO_KEY) + for w in cutlass.range_constexpr(SELECT_WARPS): + g = _imin(g, smem[_WARP_MIN + w * 16 + tid]) + smem[_GROUP_MIN + tid] = g + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + bound = cutlass.Int32( + -2147483648 + ) # the smallest key: signed keys, unlike the reference's unsigned ones + for g in cutlass.range_constexpr(16): + bound = _imax(bound, smem[_GROUP_MIN + g]) + + # Candidates (key <= bound; unused slots hold NO_KEY > any bound): a CTA prefix count gives the slots. + cnt = cutlass.Int32(0) + for s in cutlass.range_constexpr(items): + cnt += cutlass.select_(key[s] <= bound, cutlass.Int32(1), cutlass.Int32(0)) + incl = cnt + for d in cutlass.range_constexpr(5): + up = cute.arch.shuffle_sync( + incl, _imax(lane - cutlass.Int32(1 << d), cutlass.Int32(0)) + ) + incl += cutlass.select_(lane >= cutlass.Int32(1 << d), up, cutlass.Int32(0)) + if lane == 31: + smem[_WARP_CNT + warp] = incl + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + before = cutlass.Int32(0) + total = cutlass.Int32(0) + for w in cutlass.range_constexpr(SELECT_WARPS): + c = smem[_WARP_CNT + w] + before += cutlass.select_(cutlass.Int32(w) < warp, c, cutlass.Int32(0)) + total += c + slot = before + incl - cnt + for s in cutlass.range_constexpr(items): + if key[s] <= bound: + if slot < MAX_CAND: + cand[slot] = (cutlass.Int64(key[s]) << cutlass.Int64(32)) | cutlass.Int64( + tid + cutlass.Int32(s * SELECT_THREADS) + ) + slot += 1 + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + + if total <= MAX_CAND: + for i in cutlass.range(tid, total, SELECT_THREADS, unroll=1): + mine = cand[i] + rank = cutlass.Int32(0) + for j in cutlass.range(total, unroll=8): + rank += cutlass.select_(cand[j] < mine, cutlass.Int32(1), cutlass.Int32(0)) + if rank < TOPK: + smem[_PICK_KEY + rank] = cutlass.Int32(mine >> cutlass.Int64(32)) + smem[_PICK_BLOCK + rank] = cutlass.Int32(mine & cutlass.Int64(0xFFFFFFFF)) + else: + # More than MAX_CAND keys tied at the bound: rank each candidate against the whole row. + for s in cutlass.range_constexpr(items): + if key[s] <= bound: + block = tid + cutlass.Int32(s * SELECT_THREADS) + rank = cutlass.Int32(0) + for j in cutlass.range(valid, unroll=1): + kj = _forced_key(scores, head, j, query, init_blocks, local_start) + less = (kj < key[s]) | ((kj == key[s]) & (j < block)) + rank += cutlass.select_(less, cutlass.Int32(1), cutlass.Int32(0)) + if rank < TOPK: + smem[_PICK_KEY + rank] = key[s] + smem[_PICK_BLOCK + rank] = block + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + + if warp == 0: + # The 16 picked block ids ascending (NO_KEY = empty or -inf, emitted as -1), bitonic over the warp. + v = cutlass.Int32(NO_KEY) + if lane < TOPK: + pk = smem[_PICK_KEY + lane] + if (pk != cutlass.Int32(NO_KEY)) & (pk != cutlass.Int32(NEG_INF_KEY)): + v = smem[_PICK_BLOCK + lane] + for size_log in cutlass.range_constexpr(1, 6): + for stride_log in cutlass.range_constexpr(size_log - 1, -1, -1): + other = cute.arch.shuffle_sync_bfly(v, 1 << stride_log) + ascending = (lane & cutlass.Int32(1 << size_log)) == cutlass.Int32(0) + lower = (lane & cutlass.Int32(1 << stride_log)) == cutlass.Int32(0) + v = cutlass.select_(lower == ascending, _imin(v, other), _imax(v, other)) + if lane < TOPK: + out[query, head, lane] = cutlass.select_( + v == cutlass.Int32(NO_KEY), cutlass.Int32(-1), v + ) + + +class _SelectLongDslKernel: + """One row per CTA: the grid wait, then ``_select_row``. ``n_valid_prewait`` reads the row's valid-block count + before the grid wait, so that dependent load leaves the post-wait critical path. That is exact because + ``n_valid`` is written only by ``on_update_kv_lens`` and the host copy in ``prepare()``, both before the step's + first layer and behind a plain (non-PDL) launch, so no grid this select can overlap writes it + (``attention/test_modeling_v2_m3_n_valid_contract.py``).""" + + def __init__( + self, use_pdl: bool, early_trigger: bool, trigger_phase: int, n_valid_prewait: bool + ): + self.use_pdl = use_pdl + self.early_trigger = early_trigger + self.loaded_trigger = (not early_trigger) and trigger_phase == 1 + self.n_valid_prewait = n_valid_prewait + + @cute.kernel + def kernel(self, scores, n_valid, out, init_blocks, local_blocks, num_kv_heads): + row, _, _ = cute.arch.block_idx() + smem = SmemAllocator() + words = smem.allocate_tensor(cutlass.Int32, cute.make_layout(SMEM_WORDS), byte_alignment=16) + raw_prewait = cutlass.Int32(0) + if cutlass.const_expr(self.n_valid_prewait): + raw_prewait = n_valid[row // num_kv_heads] + griddepcontrol_wait() + if cutlass.const_expr(self.early_trigger): + griddepcontrol_launch_dependents() + _select_row( + row, + scores, + n_valid, + out, + init_blocks, + local_blocks, + num_kv_heads, + words, + bar_id=1, + loaded_trigger=self.loaded_trigger, + raw_prewait=raw_prewait, + n_valid_prewait=self.n_valid_prewait, + ) + + @cute.jit + def __call__(self, scores, n_valid, out, init_blocks, local_blocks, stream): + num_kv_heads = scores.shape[0] + self.kernel(scores, n_valid, out, init_blocks, local_blocks, num_kv_heads).launch( + grid=(scores.shape[2] * num_kv_heads, 1, 1), + block=(SELECT_THREADS, 1, 1), + stream=stream, + use_pdl=self.use_pdl, + ) + + +# The select reads its row's valid-block count before its grid wait instead of after it, which takes one dependent +# global load off the post-wait path; the selection is unchanged bit for bit. Exact because the count +# (MsaMetadata.msa_n_valid_blocks) is written only before the step's first layer, behind a plain launch +# (attention/test_modeling_v2_m3_n_valid_contract.py). +N_VALID_PREWAIT = True +#: Compiled kernels by (PDL launch, early trigger, trigger phase, valid count read before the wait). +kernel_cache: Dict[Tuple[bool, bool, int, bool], object] = {} + + +def kernel_key( + use_pdl: bool, early_trigger: bool, trigger_phase: int +) -> Tuple[bool, bool, int, bool]: + """The compile key of a launch (the trigger phase does not matter with the early trigger). The kernel reads its + row's valid-block count before its grid wait (``N_VALID_PREWAIT``).""" + return ( + bool(use_pdl), + bool(early_trigger), + 0 if early_trigger else int(trigger_phase), + N_VALID_PREWAIT, + ) + + +def _compile(key: Tuple[bool, bool, int, bool]): + if key not in kernel_cache: + heads, blocks, rows = cute.sym_int(), cute.sym_int(), cute.sym_int() + scores = make_fake_tensor( + cutlass.Float32, + (heads, blocks, rows), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + n_valid = make_fake_compact_tensor(cutlass.Int32, (cute.sym_int(),), stride_order=(0,)) + out = make_fake_tensor( + cutlass.Int32, (rows, heads, TOPK), stride=(cute.sym_int64(), cute.sym_int64(), 1) + ) + kernel_cache[key] = cute_aot.compile_or_load( + f"m3_select_long_dsl {key}", + lambda: cute.compile( + _SelectLongDslKernel(*key), + scores, + n_valid, + out, + cutlass.Int32(0), + cutlass.Int32(0), + make_fake_stream(use_tvm_ffi_env_stream=True), + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + return kernel_cache[key] + + +def select_long_dsl( + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool, + early_trigger: bool, + trigger_phase: int, +) -> None: + """Launch the select on the current stream (compiling its key first if ``prebuild`` did not).""" + _compile(kernel_key(use_pdl, early_trigger, trigger_phase))( + scores, n_valid_blocks, out, init_blocks, local_blocks + ) + + +def prebuild(keys, device) -> list: + """Compile every key and launch each once on a small dummy row (loading its module). Returns the keys.""" + scores = torch.zeros( + 1, 2048, 4, dtype=torch.float32, device=device + ) # capacity: fixed: a dummy prebuild row, not a size + n_valid = torch.full((4,), 2048, dtype=torch.int32, device=device) + out = torch.empty(1, 4, TOPK, dtype=torch.int32, device=device).permute(1, 0, 2) + built = [] + for key in keys: + key = kernel_key(*key) + _compile(key)(scores, n_valid, out, 0, 1) + built.append(key) + return built + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_shard_accept.py new file mode 100644 index 000000000000..2b7d05ad920f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_shard_accept.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 verification for the MiniMax-M3 target head (vocab-sharded TP head, verify rows <= 48). + +Replaces the all-gather of the target logits, the argmax and the strict-acceptance ops of a greedy verify step with +one kernel over this rank's BF16 logits shard (the target lm_head's own local GEMM output): + +* Every logit becomes a 64-bit key ``ordered(value) << 32 | ~global_index`` (ties pick the lowest index, as + ``torch.argmax``), max-reduced per row: warp shuffles, then ``atomicMax``. +* The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping + (Lamport: 0 = empty, the key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the + slots and runs the strict acceptance. Call n uses slot half n % 2. + +The argmax is taken over the BF16 logits without arithmetic, so the tokens are those of the in-tree path's argmax +over the gathered logits. The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/ +shardAccept.cu) and reached through the ``ops/spec`` entry ``m3_shard_accept``. +""" + +from dataclasses import dataclass +from typing import Any, Optional + +import torch + +from ..ops.spec.m3_shard_accept import m3_shard_accept as _shard_accept_op + +#: Verify rows per call, and exchange rows per call half of the workspace. +SHARD_ROWS = 48 # capacity: fixed: DECODE_MAX_TOKENS verify rows + + +@dataclass(eq=False) +class AcceptWorkspace: + """Per-rank exchange slots ([2 halves][SHARD_ROWS rows][world] uint64) behind one multicast mapping, the local + key accumulators and the (finished CTAs, call) counters.""" + + uc: torch.Tensor + mc_ptr: int + keys: torch.Tensor + counters: torch.Tensor + rank: int + world_size: int + handle: Any = None + comm: Any = None + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def create_workspace(mapping, device, slot_rows: int) -> AcceptWorkspace: + """Exchange slots ([2 halves][slot_rows rows][world] uint64) behind one multicast mapping over the TP group, the + local key accumulators [slot_rows] and the (finished CTAs, call) counters. Collective over the TP group; call + outside CUDA-graph capture, in the same order on every rank.""" + words = 2 * slot_rows * max(mapping.tp_size, 1) + if mapping.tp_size > 1: + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + if not _mnnvl_workspace_all_succeeded(comm, True): + comm.Free() + raise RuntimeError("exchange workspace creation failed on another rank") + handle = _make_mnnvl_mcast_buffer( + comm, max(words * 8, 4096), mapping, mapping.is_multi_node() + ) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("exchange workspace creation failed on another rank") + mc_ptr, rank = mc.data_ptr(), mapping.tp_rank + else: + uc = torch.zeros(words, dtype=torch.int64, device=device) + mc_ptr, rank, handle, comm = uc.data_ptr(), 0, None, None + return AcceptWorkspace( + uc=uc, + mc_ptr=mc_ptr, + keys=torch.zeros(slot_rows, dtype=torch.int64, device=uc.device), + counters=torch.zeros(2, dtype=torch.int32, device=uc.device), + rank=rank, + world_size=max(mapping.tp_size, 1), + handle=handle, + comm=comm, + ) + + +def _workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace, created collectively outside CUDA-graph capture on first use.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = create_workspace(mapping, device, SHARD_ROWS) + return _WS + + +def shard_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, +): + """Greedy verification from this rank's BF16 logits shard. + + Args: + logits: [nreq * (draft_len + 1) <= SHARD_ROWS, V / tp] BF16, unit column stride: this rank's columns + (global ids ``vocab_offset ..``) of the verify rows, request-major. + draft_tokens: [nreq, draft_len] int32, the drafts being verified. + batch_idx: [nreq] int32, each request's row in the verify layout. + force_total / force_vec: forced acceptance (see ``m3_shard_accept``). + next_new: optional int32 [nreq, >= 1]; column 0 receives the last accepted token. + + Returns: + (accepted_tokens [nreq, draft_len + 1] int32, num_accepted [nreq] int32, first-pass gather ids [nreq] + int64), or None when the shape is not served or the workspace is missing during a capture. Collective over + the TP group. + """ + nreq, draft_len = draft_tokens.shape + if ( + logits.dim() != 2 + or logits.shape[0] != nreq * (draft_len + 1) + or logits.shape[0] > SHARD_ROWS + or logits.dtype != torch.bfloat16 + or logits.stride(1) != 1 + ): + return None + ws = _workspace(mapping, logits.device) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + _shard_accept_op( + logits, + vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_tokens.contiguous(), + batch_idx, + accepted, + num_accepted, + gather_ids, + force_total, + next_new, + _num_sms(), + force_vec, + ) + return accepted, num_accepted, gather_ids diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_spec_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_spec_tail.py new file mode 100644 index 000000000000..0d4dc9cd1b64 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_spec_tail.py @@ -0,0 +1,116 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Small fused kernels of the MiniMax-M3 Eagle3 step (GB300, decode). + +``chunk_norm`` / ``capture_norm`` apply the drafter's per-layer fc_norm RMSNorms to the captured target hidden states +(the in-tree ``apply_eagle3_fc`` norms, same fp32 arithmetic); ``draft_input`` is a fused draft pass's input (the +embedding lookup, the drafter layer's input_layernorm / hidden_norm and their concat, same arithmetic); +``step_gather`` / ``slot_scatter`` move the step's integer outputs between the spec sampler's store and the engine's +input buffers. +""" + +from typing import Optional, Tuple + +import torch + +# The kernels are compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu) and reached through the +# ops/spec entries. +from ..ops.spec.m3_capture_norm import m3_capture_norm as _capture_norm_op +from ..ops.spec.m3_chunk_norm import m3_chunk_norm as _chunk_norm_op +from ..ops.spec.m3_draft_input import m3_draft_input as _draft_input_op +from ..ops.spec.m3_slot_scatter import m3_slot_scatter as _slot_scatter_op +from ..ops.spec.m3_step_gather import m3_step_gather as _step_gather_op + + +def draft_input( + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + eps_embed: float, + w_hidden: torch.Tensor, + eps_hidden: float, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """(cat([rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden], -1) [M, 2H], hidden[rows] [M, H]) for + int32 ids [M] and int64 rows [M] (rows None: hidden[:M]); ids outside the table embed to zero.""" + m, h = ids.shape[0], table.shape[1] + out = torch.empty((m, 2 * h), dtype=table.dtype, device=table.device) + residual = torch.empty((m, h), dtype=table.dtype, device=table.device) + return _draft_input_op( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, use_pdl + ) + + +def chunk_norm( + src: torch.Tensor, weights: torch.Tensor, eps: float, use_pdl: bool = True +) -> torch.Tensor: + """Per-chunk RMSNorm of src [M, >= C*H] (row stride may exceed C*H) with weights [C, H] -> [M, C*H].""" + c, h = weights.shape + out = torch.empty((src.shape[0], c * h), dtype=src.dtype, device=src.device) + return _chunk_norm_op(src, weights, eps, out, use_pdl) + + +# The overlap scheduler's per-step gathers and the spec sampler's slot stores as one kernel each (step_gather / +# slot_scatter) instead of ~10 and 4 small torch ops; called by the target's SpecStepFold. + + +def step_gather( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 +) -> bool: + """See step_gather_kernel; False when the inputs are outside what the kernel covers (caller falls back).""" + ts = (new_tok, next_draft, lens, slots, ppi, ids, draft, pos_off, kv_off) + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or next_draft.dim() != 2 + or new_tok.dim() < 2 + ): + return False + if (new_tok.dim() == 3 and new_tok.shape[2] != 1) or not ( + lens.is_contiguous() + and ids.is_contiguous() + and draft.is_contiguous() + and pos_off.is_contiguous() + and kv_off.is_contiguous() + ): + return False + _step_gather_op( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 + ) + return True + + +def slot_scatter( + slots, src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft +) -> bool: + """See slot_scatter_kernel (2-D int32 views, zero padding); False to fall back.""" + ts = (src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft) + if any(t.dtype != torch.int32 or not t.is_cuda for t in ts) or slots.dtype != torch.long: + return False + two = (src_new, dst_new, src_next_new, dst_next_new, src_draft, dst_draft) + if any(t.dim() != 2 for t in two) or src_lens.dim() != 1 or dst_lens.dim() != 1: + return False + _slot_scatter_op( + slots, + src_new, + dst_new, + src_next_new, + dst_next_new, + src_lens, + dst_lens, + src_draft, + dst_draft, + ) + return True + + +def capture_norm( + h: torch.Tensor, + residual: Optional[torch.Tensor], + weight: torch.Tensor, + eps: float, + dst: torch.Tensor, +) -> None: + """dst [M, H] (row stride may exceed H) = rmsnorm(h + residual) * weight with chunk_norm's rounding.""" + _capture_norm_op(h, residual, weight, eps, dst) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_target_head.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_target_head.py new file mode 100644 index 000000000000..f5a1b229c777 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_target_head.py @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The target lm_head shard of a verify step of at most 8 rows, streamed from a lossless 12-bit coding of its BF16 +weight (75% of the bytes; ``ops.gemm.m3_target_head`` has the format). + +``prepare_head`` codes this rank's shard once, before the executor's warmup, and keeps it only when it decodes to the +BF16 weight bit for bit (``round_trip_ok``); a shard that does not keeps the in-tree BF16 GEMM. ``logits`` runs the +kernel: BF16 x BF16 MMAs with fp32 accumulation over the full K and one BF16 rounding per logit, the in-tree head's +arithmetic in another summation order. The kernel is compiled into TensorRT-LLM +(cpp/tensorrt_llm/kernels/minimaxM3/targetHead.cu) and called through the catalog entry ``ops.gemm.m3_target_head``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +import torch + +from ..ops.gemm.m3_target_head import CTA_WARPS, MAX_ROWS, encode_chunked, round_trip_ok +from ..ops.gemm.m3_target_head import m3_target_head as _target_head_op + + +@dataclass(eq=False) +class TargetHead: + """One rank's target lm_head shard in the 12-bit coding, and the kernel's split-tile scratch. + + ``coded`` is False when the coding did not reproduce the weight bit for bit (the in-tree GEMM then runs).""" + + weight: torch.Tensor + coded: bool + w: Optional[torch.Tensor] = None + base: int = 0 + exc_off: Optional[torch.Tensor] = None + exc_pos: Optional[torch.Tensor] = None + exc_w: Optional[torch.Tensor] = None + part: Optional[torch.Tensor] = None + counter: Optional[torch.Tensor] = None + grid: int = 0 + + +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def head_supported(lm_head) -> bool: + """A bias-free 2-D BF16 head of whole 16-row tiles whose K the coding covers (a multiple of 64).""" + weight = getattr(lm_head, "weight", None) + if weight is None or weight.dim() != 2 or weight.dtype != torch.bfloat16: + return False + return ( + getattr(lm_head, "bias", None) is None + and weight.shape[0] % 16 == 0 + and weight.shape[1] % 64 == 0 + ) + + +def prepare_head(lm_head, grid: Optional[int] = None) -> TargetHead: + """Code ``lm_head``'s local shard and check it decodes to the BF16 weight bit for bit (fails closed: a shard that + does not is returned with ``coded=False`` and nothing kept). Built once, eagerly, before the executor's warmup + (``M3Eagle3Worker.prebuild_target_head``).""" + weight = lm_head.weight + with torch.no_grad(): + w = weight.detach() + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + if not round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w): + return TargetHead(weight=weight, coded=False) + if grid is None: + grid = _num_sms() + n = weight.shape[0] + return TargetHead( + weight=weight, + coded=True, + w=packed, + base=base, + exc_off=exc_off, + exc_pos=exc_pos, + exc_w=exc_w.view(-1), + part=torch.zeros(grid * CTA_WARPS * 2 * 4 * 32, dtype=torch.float32, device=weight.device), + counter=torch.zeros((n + 15) // 16, dtype=torch.int32, device=weight.device), + grid=grid, + ) + + +def logits(head: TargetHead, x: torch.Tensor, use_pdl: bool = True) -> torch.Tensor: + """This rank's BF16 logits shard [rows, n] of the rows x [rows <= MAX_ROWS, K] (BF16) from the coded shard.""" + assert head.coded and 1 <= x.shape[0] <= MAX_ROWS, (head.coded, x.shape) + if x.stride(1) != 1 or x.stride(0) % 8 or x.data_ptr() % 16: + x = x.contiguous() + n = head.weight.shape[0] + out = torch.empty((x.shape[0], n), dtype=torch.bfloat16, device=x.device) + return _target_head_op( + head.w, + x, + out, + head.part, + head.counter, + head.exc_off, + head.exc_pos, + head.exc_w, + head.base, + n, + head.grid, + use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_tokmap_guard.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_tokmap_guard.py new file mode 100644 index 000000000000..db0077117fe3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/m3_tokmap_guard.py @@ -0,0 +1,185 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The token map of ``thop.trtllm_gen_generation_preprocess`` for one request with several tokens. + +For multi-token generation the op's QKV preprocessing places every row by the workspace token map (``tokensInfo``, +(request, token) per row), but builds that map only when ``isBuildDecoderInfoKernelNeeded()`` holds, which at one +request with the default RoPE it does not. The op then applies whatever map the last multi-request call left in the +workspace: a row can land on another request's length and pages, and its RoPE position and K/V slot are wrong. This +module wraps the op so that, for one request and several tokens, the map the build would write, (0, t) for each row t, +is written into the op's own map slice of the workspace (the offset from the op's layout function) just before the +call. Every other call passes through unchanged. The rows' positions still come from the request's generation length, +exactly as with a freshly built map. + +The write is a device copy from a map prebuilt at install (no allocation, no host sync), so it is captured into the +decode graphs with the call it precedes. +""" + +from __future__ import annotations + +from typing import Callable, Dict, Optional, Tuple + +import torch + +from tensorrt_llm.bindings.internal import thop +from tensorrt_llm.logger import logger + +#: Argument names of ``thop.trtllm_gen_generation_preprocess`` in order (the nanobind binding). +_ARG_NAMES = ( + "qkv_input", + "workspace", + "sequence_lengths", + "spec_decoding_generation_lengths", + "spec_decoding_position_offsets", + "kv_cache_block_offsets", + "host_kv_cache_pool_pointers", + "host_kv_cache_pool_mapping", + "kv_scale_orig_quant", + "kv_scale_quant_orig", + "attention_output_orig_quant", + "rotary_inv_freq", + "rotary_cos_sin", + "mrope_position_deltas", + "layer_idx", + "seq_offset", + "num_heads", + "num_kv_heads", + "head_size", + "tokens_per_block", + "kv_cache_quant_mode", + "max_attention_window_size", + "cyclic_attention_window_size", + "num_tokens", + "batch_beam", + "input_seq_length", + "max_past_kv_length", + "rotary_embedding_dim", + "rotary_embedding_base", + "rotary_embedding_scale_type", + "rotary_embedding_scale", + "rotary_embedding_max_positions", + "position_embedding_type", + "bmm1_scale", + "bmm2_scale", + "fp8_context_fmha", + "predicted_tokens_per_seq", + "attention_chunk_size", + "multi_processor_count", + "total_num_blocks", + "kv_factor", + "need_build_kv_cache_metadata", + "cross_attention", + "skip_fmha_workspace", +) +_INT2_BYTES = 8 # capacity: fixed: an element size + +_original: Optional[Callable] = None +_max_rows = 0 +_maps: Dict[torch.device, torch.Tensor] = {} +_offsets: Dict[Tuple, int] = {} + + +def _no_taken(rows: int) -> None: + pass + + +def _no_missed(rows: int, reason: str) -> None: + pass + + +_on_taken: Callable[[int], None] = _no_taken +_on_missed: Callable[[int, str], None] = _no_missed + + +def _indexed(device: torch.device) -> torch.device: + """``device`` with its index (an unindexed ``cuda`` means the current device), the key ``_maps`` is built under.""" + if device.type == "cuda" and device.index is None: + return torch.device("cuda", torch.cuda.current_device()) + return device + + +def _one_request_map(device: torch.device, rows: int) -> torch.Tensor: + """The prebuilt (0, t) map's first ``rows`` rows on ``device``.""" + device = _indexed(device) + m = _maps.get(device) + if m is None or m.shape[0] < rows: + reason = f"no {rows}-row token map was prebuilt on {device} (install() builds it before the warmup)" + _on_missed(rows, reason) + raise RuntimeError(f"m3_tokmap_guard: {reason}") + return m[:rows] + + +def _map_offset(args: dict, rows: int) -> int: + x = args["qkv_input"] + skip = bool(args.get("skip_fmha_workspace", False)) + key = ( + x.dtype, + rows, + int(args["num_heads"]), + int(args["head_size"]), + int(args["rotary_embedding_dim"]), + int(args["num_kv_heads"]), + skip, + ) + off = _offsets.get(key) + if off is None: + layout = thop.get_trtllm_gen_generation_workspace_layout( + x.dtype, 1, rows, key[2], key[3], key[4], key[5], 0, False, skip + ) + off, size = int(layout["tokens_info_offset"]), int(layout["tokens_info_size"]) + if off < 0 or off % 4 != 0 or size < _INT2_BYTES * rows: + raise RuntimeError( + f"m3_tokmap_guard: the op's generation layout has no {rows}-row token map " + f"(offset {off}, {size} bytes): the layout changed under this guard" + ) + _offsets[key] = off + return off + + +def _guarded(*args, **kwargs): + a = dict(zip(_ARG_NAMES, args)) + a.update(kwargs) + rows = int(a["num_tokens"]) + if ( + a.get("spec_decoding_generation_lengths") is not None + and int(a["predicted_tokens_per_seq"]) > 1 + and int(a["batch_beam"]) == 1 + and rows > 1 + and not bool(a.get("cross_attention", False)) + ): + ws = a["workspace"] + src = _one_request_map(ws.device, rows) + off = _map_offset(a, rows) + # view (never reshape: a copy would leave the op's own workspace unwritten); raises on a strided workspace. + dst = ws.view(-1).view(torch.uint8)[off : off + _INT2_BYTES * rows] + dst.view(torch.int32).view(rows, 2).copy_(src) + _on_taken(rows) + return _original(*args, **kwargs) + + +def install( + device: torch.device, + max_rows: int, + on_taken: Callable[[int], None], + on_missed: Callable[[int, str], None] = _no_missed, +) -> bool: + """Wrap ``thop.trtllm_gen_generation_preprocess`` (once per process) and prebuild the (0, t) map for up to + ``max_rows`` rows on ``device`` (a one-request call past it is ``on_missed`` and raises). Returns whether this + call installed the wrapper.""" + global _original, _max_rows, _on_taken, _on_missed + device = _indexed(device) + if device not in _maps or _maps[device].shape[0] < max_rows: + t = torch.arange(max_rows, dtype=torch.int32, device=device) + _maps[device] = torch.stack([torch.zeros_like(t), t], dim=1).contiguous() + _max_rows = max(_max_rows, max_rows) + _on_taken = on_taken + _on_missed = on_missed + if _original is not None: + return False + _original = thop.trtllm_gen_generation_preprocess + thop.trtllm_gen_generation_preprocess = _guarded + logger.info( + "[M3 prebuild] m3_tokmap_guard installed: one-request multi-token generations get their (0, t) token map " + f"(up to {max_rows} rows)" + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/msa_kv_update.py new file mode 100644 index 000000000000..8c2a67356d3f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/msa_kv_update.py @@ -0,0 +1,89 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One kernel for MSA's device-side KV-length correction (MsaAttentionMetadata.on_update_kv_lens). + +The torch formulation is ~23 one-CTA kernels (min, cumsum, gathers, div / clamp / remainder, the block-table lookup, +the valid-block counts) and runs at the head of every captured decode step, twice. This kernel computes the same +int32 results in one launch: + + kv_true[b] = min(kv_lens[b], staged[b]) -> seq_lens_out[b], cu_kv[b + 1] = prefix sum (cu_kv[0] = 0) + pos[t] = kv_true[r] - qo[r] + intra[t] (r = q_batch_row[t]) + out_loc[t] = table[r, min(max(pos, 0) // page, ncols - 1)] * page + max(pos, 0) % page + n_valid[t] = (max(pos + 1, 0) + page - 1) // page + +``msa_kv_update`` is the target's ``MsaKvUpdate`` (``tensorrt_llm._torch.pyexecutor.runtime_hooks``): the MSA +metadata calls it with its own buffers and falls back to the torch formulation when it returns False. The kernel is +``torch.ops.trtllm.minimax_m3_msa_kv_update``, called through this target's ``msa_kv_update`` entry. +""" + +from typing import Optional + +import torch + +from ..ops.attention.msa_kv_update import msa_kv_update as _kv_update_op + + +def msa_kv_update( + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], +) -> bool: + """The fused correction (see module doc); False, with nothing written, when an operand is outside what the + kernel covers. Arguments in ``MsaKvUpdate`` order; ``seq_lens`` are the per-request query lengths (``qo``).""" + ts = [ + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + msa_seq_lens, + out_cache_loc, + ] + ts += [t for t in (cu_kv_lens, n_valid_blocks) if t is not None] + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or block_table.dim() != 2 + or block_table.stride(1) != 1 + ): + return False + if any(not t.is_contiguous() for t in ts if t is not block_table): + return False + if ( + kv_lens.numel() < batch + or kv_lens_staged.numel() < batch + or seq_lens.numel() < batch + or msa_seq_lens.numel() < batch + or q_batch_row.numel() < total_q + or q_intra.numel() < total_q + or out_cache_loc.numel() < total_q + or (cu_kv_lens is not None and cu_kv_lens.numel() < batch + 1) + or (n_valid_blocks is not None and n_valid_blocks.numel() < total_q) + ): + return False + _kv_update_op( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/__init__.py new file mode 100644 index 000000000000..ef3a4c5847ea --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) as a CuTe DSL kernel (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/m3_mx_gemv_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/m3_mx_gemv_kernel.py new file mode 100644 index 000000000000..d32ddbde3510 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/m3_mx_gemv_kernel.py @@ -0,0 +1,1474 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 decode GEMV on tcgen05: ``y[M, N] = x[M, K] @ W[N, K]^T`` for M <= 8, block-scaled E4M3 x E4M3. + +The weight is the MMA's A operand (128 output features per CTA), the activation is B (8 token columns), +fp32 accumulation in TMEM. Both operands come in their MXFP8 storage: E4M3 values (K contiguous) and UE8M0 +scales in the 128x4-swizzled layout, whose 512-byte atom per (128 rows, 128 K) is exactly the source of a +``tcgen05.cp`` 32x128b.warpx4 scale copy, so both scale tensors are bulk-copied unchanged. For an activation of at +most 32 rows the atom of k-tile ``k`` sits at byte ``512 k``. + +A cluster of SPLIT CTAs shares one 128-row weight tile; rank r streams the interleaved k-tiles r, r + SPLIT, ... +through a RING-stage ring. Before the grid dependency each CTA issues its first RING weight tiles (and their scale +atoms) and prefetches the rest into L2; after it, the activation slice of the CTA (values and scale atoms) lands in +one buffer. Rank r owns rows [r * 128 / SPLIT, (r + 1) * 128 / SPLIT) of the tile: the other ranks store their fp32 +partials of those rows into its shared memory and arrive on its barrier (release, cluster scope); the owner adds the +SPLIT partials in rank order and stores bf16. + +``prod`` (QKV_PRODUCER_FOLD, the sm_107 TP8 target's #117: the sparse QKV + index projection at split 8, at most 8 +tokens): instead of storing the packed row, the owners stage their bf16 rows, the CTAs exchange them by st.async so +that rank t holds token t's 128-dim row of this tile's packed head, and rank t's warp 4 runs that head's producer +(``phase_producer.producer_head``: RMSNorm, RoPE, FP8 q / index-q / index-K and NVFP4 K / V stores, at the TP2 heads) +on it, bit for bit what the separate producer kernel does with the stored row. + +Warps: 0 weight TMA, 1 activation TMA, 2 TMEM allocation + scale copies + MMA, 3 idle, 4-7 epilogue (warp 4 + i +reads TMEM lanes 32 i .. 32 i + 31). +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass import dsl_user_op +from cutlass.experimental import primitives as prims + +from .phase_producer import ProducerArgs, bf16_bits_to_f32, producer_head + +CTA_M = 128 # weight rows per CTA = the MMA's M # capacity: fixed: a kernel tile constant +MMA_N = 8 # token columns +CTA_K = 128 # one k-tile: 128 E4M3 bytes per row = one 128-byte swizzle row # capacity: fixed: a kernel tile constant +MMA_INST_K = 32 +K_BLOCKS = ( + CTA_K // MMA_INST_K +) # 4 MMAs per k-tile, one per scale byte of the atom # capacity: fixed: a kernel tile constant +THREADS = 256 +SF_ATOM_BYTES = 512 +SF_ATOM_U16 = SF_ATOM_BYTES // 2 +SF_COLS = 4 # TMEM columns of one scale atom +W_TILE_BYTES = CTA_M * CTA_K +X_TILE_BYTES = MMA_N * CTA_K +EVICT_FIRST = 0x12F0000000000000 +A_COLS_PER_MMA = MMA_INST_K // 4 # TMEM columns of one 128 x K32 E4M3 granule of A + +a_dtype = cutlass.Float8E4M3FN +b_dtype = cutlass.Float8E4M3FN + + +def num_k_tiles(k_in: int) -> int: + return k_in // CTA_K + + +def acc_cols(nacc: int, mma_n: int = MMA_N) -> int: + # nacc independent accumulators of MMA_N columns: MMA j of the CTA accumulates into j % nacc, so the M128 x N8 + # MMAs do not form one serial dependency chain; the epilogue adds them. + return max(16, nacc * mma_n) + + +def a_col0(nacc: int, my_tiles: int, mma_n: int = MMA_N) -> int: + # a_tmem: [acc | SFA per k-tile | one SFB slot (reused in issue order) | A: 32 columns (128 x 128 E4M3) per k-tile] + return (acc_cols(nacc, mma_n) + (my_tiles + 1) * SF_COLS + 31) // 32 * 32 + + +def tmem_cols( + ring: int, nacc: int, a_tmem: bool = False, my_tiles: int = 0, mma_n: int = MMA_N +) -> int: + need = acc_cols(nacc, mma_n) + 2 * ring * SF_COLS + if a_tmem: + need = a_col0(nacc, my_tiles, mma_n) + my_tiles * K_BLOCKS * A_COLS_PER_MMA + cols = 32 + while cols < need: + cols *= 2 + return cols + + +@dsl_user_op +def read_globaltimer(*, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), [], "mov.u64 $0, %globaltimer;", "=l", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _mapa_u32(smem_ptr, peer, *, loc=None, ip=None): + """The shared::cluster address of this CTA's shared-memory location in cluster CTA ``peer``.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [smem_ptr.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(peer).ir_value(loc=loc, ip=ip)], + "mapa.shared::cluster.u32 $0, $1, $2;", "=r,r,r", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_async_v4(dst, w0, w1, w2, w3, mbar, *, loc=None, ip=None): + """st.async of four int32 words to a shared::cluster address, completing ``mbar`` there by 16 bytes.""" + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, + [cutlass.Int32(dst).ir_value(loc=loc, ip=ip), cutlass.Int32(w0).ir_value(loc=loc, ip=ip), + cutlass.Int32(w1).ir_value(loc=loc, ip=ip), cutlass.Int32(w2).ir_value(loc=loc, ip=ip), + cutlass.Int32(w3).ir_value(loc=loc, ip=ip), cutlass.Int32(mbar).ir_value(loc=loc, ip=ip)], + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [$0], {$1, $2, $3, $4}, [$5];", "r,r,r,r,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _ld_shared_v4_f32(smem_ptr, *, loc=None, ip=None): + """ld.shared.v4.f32 of four fp32 at a 16-byte-aligned shared-memory pointer.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + out = _llvm.inline_asm( + _llvm.StructType.get_literal([_T.f32()] * 4), [smem_ptr.toint(loc=loc, ip=ip).ir_value()], + "ld.shared.v4.f32 {$0, $1, $2, $3}, [$4];", "=f,=f,=f,=f,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + return [ + cutlass.Float32(_llvm.extractvalue(_T.f32(), out, [i], loc=loc, ip=ip)) for i in range(4) + ] + + +@dsl_user_op +def _ld_shared_v4_b32(smem_ptr, *, loc=None, ip=None): + """ld.shared.v4.b32 of four int32 words at a 16-byte-aligned shared-memory pointer.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + out = _llvm.inline_asm( + _llvm.StructType.get_literal([_T.i32()] * 4), [smem_ptr.toint(loc=loc, ip=ip).ir_value()], + "ld.shared.v4.b32 {$0, $1, $2, $3}, [$4];", "=r,=r,=r,=r,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + return [cutlass.Int32(_llvm.extractvalue(_T.i32(), out, [i], loc=loc, ip=ip)) for i in range(4)] + + +@dsl_user_op +def _test_wait_cluster(mbar, parity, *, loc=None, ip=None): + """mbarrier.test_wait.parity.acquire.cluster on a barrier of this CTA completed by other CTAs' st.async.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + done = cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [mbar.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(parity).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tmbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [$1], $2;\n\t" + "selp.u32 $0, 1, 0, p;\n\t}", "=r,r,r,~{memory}", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + return done != cutlass.Int32(0) + + +@dsl_user_op +def _absmax(acc, v, *, loc=None, ip=None): + """max(acc, |v|).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(acc).ir_value(loc=loc, ip=ip), cutlass.Float32(v).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .f32 a;\n\tabs.f32 a, $2;\n\tmax.f32 $0, $1, a;\n\t}", "=f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _rcp_approx_ftz(x, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _e4m3x4(v0, v1, v2, v3, *, loc=None, ip=None): + """Four f32 to E4M3 (RN, satfinite), packed little-endian into one int32 word (v0 in the low byte).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (v0, v1, v2, v3)], + "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32(addr, value, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip)], + "st.global.b32 [$0], $1;", "l,r", has_side_effects=True, is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16(x) (RN) as the fp32 bit pattern of the rounded value.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b16 h, z;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _bf16_round(x): + return _bf16_bits(x).bitcast(cutlass.Float32) + + +# The swigluoai activation of MiniMax-M3's dense MLP, gate * sigmoid(1.702 * gate) * (up + 1) with gate clamped to <= 7 +# and up to [-7, 7], in exactly the instruction sequence the Triton silu_and_mul kernel compiles to (modules/swiglu.py +# on SM107): the gate clamp on the packed bf16 pair, the up clamp as min.xorsign.abs, two rounded multiplies into +# ex2.approx, div.full for the reciprocal, two rounded multiplies, round to bf16. No multiply feeds an add, so the +# explicit .rn forms compile to what the Triton kernel's contractable forms do. +SWIGLU_OAI_ALPHA = 1.702 +SWIGLU_OAI_LIMIT = 7.0 + + +@dsl_user_op +def _swiglu_oai_bf16x2(gate_word, up_word, *, loc=None, ip=None): + """The activation of one packed bf16 pair of gate and up values: its two bf16 results as one int32 word (the low + half from the low halves).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [cutlass.Int32(w).ir_value(loc=loc, ip=ip) for w in (gate_word, up_word)], + "{\n\t.reg .b32 g, glim, gl, gh, ul, uh, one, lim, t0, t1, e0, e1, s0, s1, r0, r1;\n\t.reg .b16 hl, hh;\n\t" + "mov.b32 glim, 0x40E040E0;\n\tmin.bf16x2 g, $1, glim;\n\t" + "mov.b32 {hl, hh}, g;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, $2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mov.b32 lim, 0f40E00000;\n\tmin.xorsign.abs.f32 ul, ul, lim;\n\tmin.xorsign.abs.f32 uh, uh, lim;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 ul, ul, one;\n\tadd.rn.f32 uh, uh, one;\n\t" + "mul.rn.f32 t0, gl, 0fBFD9DB23;\n\tmul.rn.f32 t1, gh, 0fBFD9DB23;\n\t" + "mul.rn.f32 t0, t0, 0f3FB8AA3B;\n\tmul.rn.f32 t1, t1, 0f3FB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "add.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 r0, s0, gl;\n\tmul.rn.f32 r1, s1, gh;\n\t" + "mul.rn.f32 r0, ul, r0;\n\tmul.rn.f32 r1, uh, r1;\n\t" + "cvt.rn.bf16x2.f32 $0, r1, r0;\n\t}", "=r,r,r", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _rope_bits(c, a, s, b, *, loc=None, ip=None): + """bf16(c * a + s * b) with separately rounded products (no FMA), as fp32 bits: the stock RoPE arithmetic.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (c, a, s, b)], + "{\n\t.reg .f32 p, q, r;\n\t.reg .b16 h, z;\n\tmul.rn.f32 p, $1, $2;\n\tmul.rn.f32 q, $3, $4;\n\t" + "add.rn.f32 r, p, q;\n\tcvt.rn.bf16.f32 h, r;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _din_load4(table, hidden, hidden_half, word_idx): + """Four int32 words at word_idx of the hidden rows (hidden_half == 1) or the embedding table: one CTA-uniform + base-address select instead of a branch.""" + addr = cutlass.select_( + hidden_half == cutlass.Int32(1), hidden.data_ptr(0).toint(), table.data_ptr(0).toint() + ) + ptr = cutlass.inttoptr( + addr + cutlass.Int64(word_idx) * cutlass.Int64(4), mem_space=1, dtype=cutlass.Int32 + ) + return ptr.load(count=4, alignment=16) + + +def _stamp(timers, timed: bool, bx, slot: int): + # Diagnostic: %globaltimer of milestone `slot` of CTA bx (timers [grid, 8] int64). + if timed: + timers.store(read_globaltimer(), idx=bx * cutlass.Int32(8) + cutlass.Int32(slot)) + + +def _k_tile(rank, i: int, split: int, my_tiles: int, contig: bool): + # contig: rank r streams k-tiles [r * my_tiles, (r + 1) * my_tiles); else the interleaved r, r + split, ... + if contig: + return rank * cutlass.Int32(my_tiles) + cutlass.Int32(i) + return rank + cutlass.Int32(i * split) + + +@cute.kernel +def m3_mx_gemv_kernel( + tma_desc_w: cutlass.GridConstant[cuda.TensorMap], # W [N, K] E4M3 as bytes, box 128 x 128 + tma_desc_wsf: cutlass.GridConstant[ + cuda.TensorMap + ], # W scale atoms [N / 128 * K / 128, 256] u16 + tma_desc_x: cutlass.GridConstant[cuda.TensorMap], # x [M, K] E4M3 as bytes, box 128 x 8 + tma_desc_xsf: cutlass.GridConstant[cuda.TensorMap], # x scale atoms [K / 128, 256] u16 + y: cutlass.Array, # [M * N] bf16, token-major + x_words: cutlass.Array, # x_bf16: the bf16 activation [M, K] as int32 words + timers: cutlass.Array, # [grid, 8] int64 milestones when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, # ar_push: the all-reduce workspace's multicast mapping (int32 words) + ar_flags: cutlass.Int64, # ar_push: its call flags (parity of the next call = flags[0] & 1) + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[ + int + ], # 1: push the output rows into the next all-reduce; 2: and trigger after it + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], # token columns of the MMA (8, 16 or 48): M <= mma_n + din_i: cutlass.Array, # din: int32 token ids [M] + din_table: cutlass.Array, # din: embedding [V, K / 2] bf16 as int32 words + din_hidden: cutlass.Array, # din: hidden rows [M, K / 2] bf16 as int32 words + din_w: cutlass.Array, # din: [embedding norm | hidden norm] weights [K] bf16 as int32 words + din_f: cutlass.Array, # din: fp32 [eps_e, eps_h] + din_vocab: cutlass.Int32, + din_g: cutlass.Array, # din, flag 1: int64 hidden row of each token, as int32 words + din_res: cutlass.Array, # din, flag 2: the gathered hidden rows out [M, K / 2] bf16 as int32 words + din_flags: cutlass.Int32, # din: 1 = gather the hidden rows, 2 = write them to din_res + rope_cs: cutlass.Array, # rope: fp32 (cos, sin) [positions][64][2] + rope_i: cutlass.Array, # rope: int32 cache lengths [M / T] + rope_bt: cutlass.Array, # rope: int32 K / V page tables [M / T][2][max_blocks] + rope_q: cutlass.Array, # rope: E4M3 Q out [M][hq][128] as uint8 + rope_pool: cutlass.Array, # rope: paged E4M3 cache [pages][hkv][tokens per page][128] as uint8 + rope_scale: cutlass.Array, # rope: fp32 [1] E4M3 quantization scale + rope_dims: cutlass.Array, # rope: int32 [max_blocks, tokens-per-page log2, hq, hkv, T] + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], + x_swiglu: cutlass.Constexpr[ + bool + ], # x_bf16: x_words is [M, 2K] gate | up; the activation is swigluoai(gate, up) + # prod: the producer's addresses and strides (ProducerArgs fields; `packed` is unused), then num_pages and eps. + pr_q: cutlass.Int64, + pr_iq: cutlass.Int64, + pr_ik: cutlass.Int64, + pr_kvd: cutlass.Int64, + pr_kvs: cutlass.Int64, + pr_loc: cutlass.Int64, + pr_quant: cutlass.Int64, + pr_pos: cutlass.Int64, + pr_rope: cutlass.Int64, + pr_qw: cutlass.Int64, + pr_kw: cutlass.Int64, + pr_iqw: cutlass.Int64, + pr_ikw: cutlass.Int64, + pr_ips: cutlass.Int64, + pr_its: cutlass.Int64, + pr_dps: cutlass.Int64, + pr_dls: cutlass.Int64, + pr_dhs: cutlass.Int64, + pr_dts: cutlass.Int64, + pr_sps: cutlass.Int64, + pr_sls: cutlass.Int64, + pr_shs: cutlass.Int64, + pr_pages: cutlass.Int32, + pr_eps: cutlass.Float32, + prod: cutlass.Constexpr[bool], +): + k_tiles = num_k_tiles(k_in) + x_tile_bytes = mma_n * CTA_K + my_tiles = k_tiles // split + rows_per_rank = CTA_M // split + tx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp_id = cute.arch.warp_idx() + rank = cutlass.Int32(0) + if cutlass.const_expr(split > 1): + rank = cute.arch.block_idx_in_cluster() + tma_ptr_w = tma_desc_w.get_ptr() + tma_ptr_wsf = tma_desc_wsf.get_ptr() + tma_ptr_x = tma_desc_x.get_ptr() + tma_ptr_xsf = tma_desc_xsf.get_ptr() + n_tile = bx // cutlass.Int32(split) + m_offset = n_tile * cutlass.Int32(CTA_M) + + smem_a = cutlass.Array( + cutlass.Uint8, ring * W_TILE_BYTES, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_b = cutlass.Array( + cutlass.Uint8, my_tiles * x_tile_bytes, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_sfa = cutlass.Array( + cutlass.Uint8, ring * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + smem_sfb = cutlass.Array( + cutlass.Uint8, my_tiles * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + w_full = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + mma_done = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + act_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + acc_done = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + reduce_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + # Partials of the rows this rank owns: [source rank][row][token] fp32 (its own slot stays unused). Rows are padded + # to mma_n + 4 words: the owner's 16-byte row reads (lane = row) then hit all 32 banks once per 8 lanes; at a + # stride of mma_n they were 8- to 16-way conflicted (4.3 us of the 48-token QKV tail). + mb_stride = mma_n + 4 + mailbox = cutlass.Array( + cutlass.Float32, + split * rows_per_rank * mb_stride, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + # prod: this CTA's bf16 owner rows by token ([8][rows_per_rank]), and the packed-head row of token `rank` gathered + # from the cluster (128 bf16 as 64 words, completed on pr_full by 8 x rows_per_rank * 2 bytes of st.async). + pr_stage = None + pr_row = None + pr_full = None + if cutlass.const_expr(prod): + pr_stage = cutlass.Array( + cutlass.Uint16, 8 * rows_per_rank, space=cutlass.AddressSpace.smem, alignment=16 + ) + pr_row = cutlass.Array(cutlass.Int32, 64, space=cutlass.AddressSpace.smem, alignment=16) + pr_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + # din: per-token RMS factor of the half of x this CTA's k-tiles lie in. + din_rs = None + if cutlass.const_expr(din): + din_rs = cutlass.Array(cutlass.Float32, 8, space=cutlass.AddressSpace.smem, alignment=16) + + if warp_id == 0: + prims.prefetch_tensormap(tma_ptr_w) + prims.prefetch_tensormap(tma_ptr_wsf) + prims.prefetch_tensormap(tma_ptr_x) + prims.prefetch_tensormap(tma_ptr_xsf) + if prims.elect_sync(): + for s in cutlass.range_constexpr(ring): + prims.mbarrier_init(w_full.subview(s), 1) + prims.mbarrier_init(mma_done.subview(s), 1) + prims.mbarrier_init(act_full, 1) + prims.mbarrier_init(acc_done, 1) + if cutlass.const_expr(split > 1 and st_async): + # Completed by the peers' st.async bytes; this CTA's own arrival carries the expected count. + prims.mbarrier_init(reduce_ready, 1) + prims.mbarrier_arrive_expect_tx( + reduce_ready, (split - 1) * rows_per_rank * mma_n * 4 + ) + elif cutlass.const_expr(split > 1): + prims.mbarrier_init(reduce_ready, (split - 1) * rows_per_rank) + if cutlass.const_expr(prod): + # Completed by the 8 ranks' st.async (this one's included) only on a rank that receives a token row. + prims.mbarrier_init(pr_full, 1) + if rank < num_tokens: + prims.mbarrier_arrive_expect_tx(pr_full, split * rows_per_rank * 2) + if warp_id == 2: + prims.tcgen05_alloc(tmem_ptr_i32, tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n)) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + if cutlass.const_expr(split > 1): + prims.barrier_cluster_arrive_relaxed() + prims.barrier_cluster_wait() + prims.barrier_cta_sync(0) + if tx == 0: + _stamp(timers, timed, bx, 0) + + if warp_id == 0: + if prims.elect_sync(): + for i in cutlass.range_constexpr(ring, my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [k_global * cutlass.Int32(CTA_K), m_offset], [] + ) + if cutlass.const_expr((n_out // CTA_M) * split >= 80): + # Wide grids launch in part only after the predecessor (e.g. the fused attention) frees its SMs, which + # loses their pre-wait weight prefetch: the CTAs of the first half also stage their partner's + # (bx + grid / 2) weights into L2. + if bx < cutlass.Int32((n_out // CTA_M) * split // 2): + pb = bx + cutlass.Int32((n_out // CTA_M) * split // 2) + p_rank = pb % cutlass.Int32(split) + p_m_offset = (pb // cutlass.Int32(split)) * cutlass.Int32(CTA_M) + for i in cutlass.range_constexpr(my_tiles): + p_k = _k_tile(p_rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [p_k * cutlass.Int32(CTA_K), p_m_offset], [] + ) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + if cutlass.const_expr(i >= ring): + while not cute.arch.mbarrier_try_wait( + mma_done.subview(stage).data_ptr(), (i // ring - 1) % 2 + ): + pass + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.mbarrier_arrive_expect_tx(w_full.subview(stage), W_TILE_BYTES + SF_ATOM_BYTES) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_a.subview(stage * W_TILE_BYTES), + tma_ptr_w, + (k_global * cutlass.Int32(CTA_K), m_offset), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfa.subview(stage * SF_ATOM_BYTES), + tma_ptr_wsf, + (cutlass.Int32(0), n_tile * cutlass.Int32(k_tiles) + k_global), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + elif warp_id == 1: + prims.griddepcontrol(prims.GridDepAction.WAIT) + if cutlass.const_expr(trigger_early): + # Dependents launch once this grid's own dependency is satisfied (not at entry): consumers that read + # step metadata or caches before their own grid wait rely on every grid ahead of them having passed its + # wait, so the chain of releases never runs ahead of a producer still in flight. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if tx == 32: + _stamp(timers, timed, bx, 1) + if cutlass.const_expr(not x_bf16): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(act_full, my_tiles * (x_tile_bytes + SF_ATOM_BYTES)) + for i in cutlass.range_constexpr(my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_b.subview(i * x_tile_bytes), + tma_ptr_x, + (k_global * cutlass.Int32(CTA_K), cutlass.Int32(0)), + act_full, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfb.subview(i * SF_ATOM_BYTES), + tma_ptr_xsf, + (cutlass.Int32(0), k_global), + act_full, + ) + elif warp_id == 2: + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + ring * SF_COLS + idesc = prims.Tcgen05MxInstrDesc.build( + a_dtype=a_dtype, b_dtype=b_dtype, scale_format=1, n_dim=mma_n, m_dim=CTA_M + ) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + if cutlass.const_expr(a_tmem): + # A staged in TMEM: before the grid wait every weight k-tile and its scale atom go on into TMEM as they + # land (the copies of a stage refilled later are committed to its mma_done); once the activation lands, + # its scale atoms and the MMAs, A read from TMEM. + base_row = tmem_raw >> 16 + base_col = tmem_raw & 0xFFFF + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + my_tiles * SF_COLS + a_base = tmem_raw + cutlass.Int32(a_col0(nacc, my_tiles, mma_n)) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + if prims.elect_sync(): + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_cp( + prims.Tcgen05CpShape.SHAPE_128X256B, + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), + 6, + cutlass.Int32, + ), + desc_a + ((MMA_INST_K >> 4) * kb), + group=prims.CTAGroup.CTA_1, + ) + prims.tcgen05_cp( + s2t_shape, + cutlass.inttoptr((base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32), + desc_sfa, multicast=s2t_multicast, + ) # fmt: skip + if cutlass.const_expr(i + ring < my_tiles): + prims.tcgen05_commit(mma_done.subview(stage)) + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + for i in cutlass.range_constexpr(my_tiles): + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + sfa_tmem = cutlass.inttoptr( + (base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32 + ) + sfb_tmem = cutlass.inttoptr((base_row << 16) | sfb_col0, 6, cutlass.Int32) + prims.tcgen05_cp(s2t_shape, sfb_tmem, desc_sfb, multicast=s2t_multicast) + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), 6, cutlass.Int32 + ), + desc_b + ((MMA_INST_K >> 4) * kb), idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + i > 0 or kb > 0, + sfa_tmem, sfb_tmem, + ) # fmt: skip + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + else: + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + sfa_tmem = (base_row << 16) | (sfa_col0 + stage * SF_COLS) + sfb_tmem = (base_row << 16) | (sfb_col0 + stage * SF_COLS) + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + if prims.elect_sync(): + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), desc_sfa, + multicast=s2t_multicast) # fmt: skip + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), desc_sfb, + multicast=s2t_multicast) # fmt: skip + for kb in cutlass.range_constexpr(K_BLOCKS): + inc = ((MMA_INST_K * 8 // 8) >> 4) * kb + j = i * K_BLOCKS + kb + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw + cutlass.Int32((j % nacc) * mma_n), 6, cutlass.Float32), + desc_a + inc, desc_b + inc, idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), j >= nacc, + cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), + ) # fmt: skip + prims.tcgen05_commit(mma_done.subview(stage)) + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + elif warp_id >= 4: + lane = tx % 32 + row = (warp_id % 4) * 32 + lane + if cutlass.const_expr(x_bf16): + # The bf16 activation of this CTA's k-tiles to MXFP8 in shared memory, as + # torch.ops.trtllm.mxfp8_quantize(x, True) does (UE8M0 = round-up(qz_amax * rcp(448)), + # E4M3 RN-satfinite): one 32-element block (token qz_t, k-tile qz_i, block qz_kb) per thread and round, + # the E4M3 bytes into the 128-byte-swizzled B tile, the qz_scale byte into the tile's qz_scale atom. + qz_tid = tx - cutlass.Int32(128) + prims.griddepcontrol(prims.GridDepAction.WAIT) + din_half_k = k_in // 2 + din_hidden_half = cutlass.Int32(0) + if cutlass.const_expr(din): + # Draft input x = [rmsnorm(table[ids]) * w_e, rmsnorm(hidden) * w_h] (an out-of-range id embeds to + # zero): this CTA's k-tiles lie in one half, so it needs that half's statistic of every token. 16 + # threads per token, 16-byte loads, all of a thread's loads in flight at once. + din_hidden_half = cutlass.select_( + rank * cutlass.Int32(my_tiles * CTA_K) >= cutlass.Int32(din_half_k), + cutlass.Int32(1), + cutlass.Int32(0), + ) + din_tok = qz_tid // cutlass.Int32(16) + din_part = qz_tid % cutlass.Int32(16) + din_tok_ld = cutlass.select_(din_tok < num_tokens, din_tok, cutlass.Int32(0)) + din_id = din_i.load(idx=din_tok_ld) + din_valid = (din_id >= cutlass.Int32(0)) & (din_id < din_vocab) + din_hrow = din_tok_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + din_hrow = din_g.load(idx=din_tok_ld * cutlass.Int32(2)) + din_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + din_hrow, + cutlass.select_(din_valid, din_id, cutlass.Int32(0)), + ) + # The first hidden-half CTA of tile 0 reads whole hidden rows here: it also writes them out. + din_copy = ( + ((din_flags & cutlass.Int32(2)) != cutlass.Int32(0)) + & (bx == cutlass.Int32(split // 2)) + & (din_hidden_half == cutlass.Int32(1)) + & (din_tok < num_tokens) + ) + din_base = din_row * cutlass.Int32(din_half_k // 2) + din_ss = cutlass.Float32(0.0) + din_vecs_per_part = din_half_k // 8 // 16 + for din_round in cutlass.range_constexpr(0, din_vecs_per_part, 16): + din_n = min(16, din_vecs_per_part - din_round) + din_loaded = [] + for din_j in cutlass.range_constexpr(din_n): + din_widx = din_base + ( + din_part + cutlass.Int32((din_round + din_j) * 16) + ) * cutlass.Int32(4) + din_loaded.append( + _din_load4(din_table, din_hidden, din_hidden_half, din_widx) + ) + if din_copy: + for din_j in cutlass.range_constexpr(din_n): + din_v = din_loaded[din_j] + din_res.store( + (din_v[0], din_v[1], din_v[2], din_v[3]), + idx=din_tok * cutlass.Int32(din_half_k // 2) + + (din_part + cutlass.Int32((din_round + din_j) * 16)) + * cutlass.Int32(4), + alignment=16, + ) + for din_j in cutlass.range_constexpr(din_n): + for din_e in cutlass.range_constexpr(4): + din_word = cutlass.Int32(din_loaded[din_j][din_e]) + din_lo = (din_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + din_hi = (din_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + din_ss = din_ss + din_lo * din_lo + din_hi * din_hi + for din_o in cutlass.range_constexpr(4): + din_ss = din_ss + cute.arch.shuffle_sync_bfly(din_ss, offset=1 << din_o) + if din_part == cutlass.Int32(0): + din_eps = din_f.load(idx=din_hidden_half) + din_r = cute.math.rsqrt( + din_ss / cutlass.Float32(din_half_k) + din_eps, fastmath=True + ) + din_keep = din_valid | (din_hidden_half == cutlass.Int32(1)) + din_rs.store( + cutlass.select_(din_keep, din_r, cutlass.Float32(0.0)), idx=din_tok + ) + prims.barrier_cta_sync(3, thread_count=128) + qz_b_words = cutlass.Array(smem_b.data_ptr(0), shape=(my_tiles * x_tile_bytes // 4,), dtype=cutlass.Int32, + alignment=16) # fmt: skip + qz_items = my_tiles * K_BLOCKS * mma_n + qz_rounds = (qz_items + 127) // 128 + # A group's loads first (clamped to a valid row), then its math: one L2 round trip per group of + # QZ_GROUP rounds instead of one per round, without holding every round's data in registers. + QZ_GROUP = qz_rounds if qz_rounds <= 8 else 4 + qz_loaded = [] + for qz_r in cutlass.range_constexpr(qz_rounds): + qz_item = qz_tid + cutlass.Int32(qz_r * 128) + qz_t = qz_item % cutlass.Int32(mma_n) + qz_kb = (qz_item // cutlass.Int32(mma_n)) % cutlass.Int32(K_BLOCKS) + qz_i = qz_item // cutlass.Int32(mma_n * K_BLOCKS) + if cutlass.const_expr(qz_r * 128 + 127 >= qz_items): + qz_i = cutlass.select_( + qz_item < cutlass.Int32(qz_items), qz_i, cutlass.Int32(0) + ) + qz_k_global = rank + qz_i * cutlass.Int32(split) + if cutlass.const_expr(contig): + qz_k_global = rank * cutlass.Int32(my_tiles) + qz_i + qz_t_ld = cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + qz_x_base = ( + qz_t_ld * cutlass.Int32(k_in if x_swiglu else k_in // 2) + + qz_k_global * cutlass.Int32(CTA_K // 2) + + qz_kb * cutlass.Int32(MMA_INST_K // 2) + ) + qz_wvecs = None + if cutlass.const_expr(din): + # The draft input's raw values (table row of the id / hidden row) and its norm weights. + qz_col = qz_k_global * cutlass.Int32(CTA_K) + qz_kb * cutlass.Int32(MMA_INST_K) + qz_hcol = qz_col - din_hidden_half * cutlass.Int32(din_half_k) + qz_id = din_i.load(idx=qz_t_ld) + qz_hrow = qz_t_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + qz_hrow = din_g.load(idx=qz_t_ld * cutlass.Int32(2)) + qz_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + qz_hrow, + cutlass.select_( + (qz_id >= cutlass.Int32(0)) & (qz_id < din_vocab), + qz_id, + cutlass.Int32(0), + ), + ) + qz_src_w = qz_row * cutlass.Int32(din_half_k // 2) + qz_hcol // cutlass.Int32(2) + qz_vecs = [ + _din_load4( + din_table, + din_hidden, + din_hidden_half, + qz_src_w + cutlass.Int32(qz_v * 4), + ) + for qz_v in range(4) + ] + qz_wvecs = [ + din_w.load( + idx=qz_col // cutlass.Int32(2) + cutlass.Int32(qz_v * 4), + vector_size=4, + alignment=16, + ) + for qz_v in range(4) + ] + else: + qz_vecs = [ + x_words.load( + idx=qz_x_base + cutlass.Int32(qz_v * 4), vector_size=4, alignment=16 + ) + for qz_v in range(4) + ] + if cutlass.const_expr(x_swiglu): + # The up half of the same columns, K values after the gate in the row. + qz_wvecs = [ + x_words.load( + idx=qz_x_base + cutlass.Int32(k_in // 2 + qz_v * 4), + vector_size=4, + alignment=16, + ) + for qz_v in range(4) + ] + qz_loaded.append((qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs)) + if cutlass.const_expr((qz_r + 1) % QZ_GROUP == 0 or qz_r + 1 == qz_rounds): + for qz_g in cutlass.range_constexpr(len(qz_loaded)): + qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs = qz_loaded[qz_g] + qz_vals = [] + qz_amax = cutlass.Float32(0.0) + if cutlass.const_expr(din): + qz_rs = din_rs.load( + idx=cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + ) + for qz_v in cutlass.range_constexpr(4): + for qz_e4 in cutlass.range_constexpr(4): + qz_word = cutlass.Int32(qz_vecs[qz_v][qz_e4]) + if cutlass.const_expr(x_swiglu): + qz_word = _swiglu_oai_bf16x2( + qz_word, cutlass.Int32(qz_wvecs[qz_v][qz_e4]) + ) + qz_lo = (qz_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + qz_hi = (qz_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + if cutlass.const_expr(din): + # bf16(x * rs * w), the draft-input kernel's rounding. + qz_wword = cutlass.Int32(qz_wvecs[qz_v][qz_e4]) + qz_lo = _bf16_round( + qz_lo + * qz_rs + * (qz_wword << cutlass.Int32(16)).bitcast(cutlass.Float32) + ) + qz_hi = _bf16_round( + qz_hi + * qz_rs + * (qz_wword & cutlass.Int32(-65536)).bitcast( + cutlass.Float32 + ) + ) + qz_vals.append(qz_lo) + qz_vals.append(qz_hi) + qz_amax = _absmax(_absmax(qz_amax, qz_lo), qz_hi) + qz_bits = (qz_amax * _rcp_approx_ftz(cutlass.Float32(448.0))).bitcast( + cutlass.Int32 + ) + qz_e = ( + (qz_bits >> cutlass.Int32(23)) & cutlass.Int32(0xFF) + ) + cutlass.select_( + (qz_bits & cutlass.Int32(0x7FFFFF)) != cutlass.Int32(0), + cutlass.Int32(1), + cutlass.Int32(0), + ) + qz_e = cutlass.select_(qz_e > cutlass.Int32(254), cutlass.Int32(254), qz_e) + qz_scale = ((cutlass.Int32(254) - qz_e) << cutlass.Int32(23)).bitcast( + cutlass.Float32 + ) + qz_scale = cutlass.select_( + qz_amax != cutlass.Float32(0.0), qz_scale, cutlass.Float32(0.0) + ) + qz_q = [_e4m3x4(qz_vals[4 * qz_j] * qz_scale, qz_vals[4 * qz_j + 1] * qz_scale, + qz_vals[4 * qz_j + 2] * qz_scale, qz_vals[4 * qz_j + 3] * qz_scale) + for qz_j in range(8)] # fmt: skip + if qz_item < cutlass.Int32(qz_items): + if qz_t < num_tokens: + qz_row_w = qz_i * cutlass.Int32( + x_tile_bytes // 4 + ) + qz_t * cutlass.Int32(CTA_K // 4) + qz_c0 = qz_kb * cutlass.Int32(2) + qz_b_words.store( + (qz_q[0], qz_q[1], qz_q[2], qz_q[3]), + idx=qz_row_w + + (qz_c0 ^ (qz_t & cutlass.Int32(7))) * cutlass.Int32(4), + alignment=16, + ) + qz_b_words.store( + (qz_q[4], qz_q[5], qz_q[6], qz_q[7]), + idx=qz_row_w + + ((qz_c0 + cutlass.Int32(1)) ^ (qz_t & cutlass.Int32(7))) + * cutlass.Int32(4), + alignment=16, + ) + smem_sfb.store( + cutlass.Uint8(qz_e), + idx=qz_i * cutlass.Int32(SF_ATOM_BYTES) + + (qz_t % cutlass.Int32(32)) * cutlass.Int32(16) + + (qz_t // cutlass.Int32(32)) * cutlass.Int32(4) + + qz_kb, + ) + qz_loaded = [] + # Generic-proxy stores made visible to the MMA's async-proxy reads, then one arrival for all. + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.barrier_cta_sync(2, thread_count=128) + if qz_tid == cutlass.Int32(0): + prims.mbarrier_arrive(act_full) + if cutlass.const_expr(st_async): + while not cute.arch.mbarrier_test_wait(acc_done.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(acc_done.data_ptr(), 0): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + # tcgen05.ld widths are powers of two: the accumulator columns in descending power-of-two pieces. + acc_all = [] + ld_col = 0 + for ld_w in (128, 64, 32, 16, 8): + if cutlass.const_expr((mma_n * nacc - ld_col) >= ld_w): + piece = prims.tcgen05_ld( + "32x32b", + cutlass.inttoptr( + tmem_ptr_i32.load() + cutlass.Int32(ld_col), 6, cutlass.Float32 + ), + num=ld_w, + ) + acc_all = acc_all + [piece[c] for c in range(ld_w)] + ld_col += ld_w + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + acc = [] + for t in cutlass.range_constexpr(mma_n): + v = cutlass.Float32(acc_all[t]) + for a in cutlass.range_constexpr(1, nacc): + v = v + cutlass.Float32(acc_all[a * mma_n + t]) + acc.append(v) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if tx == 128: + _stamp(timers, timed, bx, 4) + n = m_offset + row + if cutlass.const_expr(split == 1): + for t in cutlass.range_constexpr(mma_n): + if cutlass.Int32(t) < num_tokens: + y.store( + cutlass.BFloat16(cutlass.Float32(acc[t])), idx=cutlass.Int32(t * n_out) + n + ) + if cutlass.const_expr(ar_push > 0): + prims.griddepcontrol(prims.GridDepAction.WAIT) + # The parity of each (token, column slice) slot: the all-reduce's CTA t * 8 + slice flag. All loads are + # issued before the first store so they overlap (one L2 round trip, not one per token). + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_v = cutlass.Float32(acc[t]) + ar_hi = cute.arch.shuffle_sync_bfly(ar_v, offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), _bf16x2_word(ar_v, ar_hi) + ) + else: + owner = row // cutlass.Int32(rows_per_rank) + local = row % cutlass.Int32(rows_per_rank) + if cutlass.const_expr(rope): + # Rank r owns head dims [r * P, (r + 1) * P) and their rotation partners 64 + ..., P = rows / 2. + pairs = rows_per_rank // 2 + owner = (row % cutlass.Int32(64)) // cutlass.Int32(pairs) + local = (row // cutlass.Int32(64)) * cutlass.Int32(pairs) + ( + row % cutlass.Int32(64) + ) % cutlass.Int32(pairs) + if owner != rank: + base = (rank * cutlass.Int32(rows_per_rank) + local) * cutlass.Int32(mb_stride) + if cutlass.const_expr(st_async): + dst = _mapa_u32(mailbox.subview(base).data_ptr(), owner) + mb = _mapa_u32(reduce_ready.data_ptr(), owner) + words = [cutlass.Float32(acc[t]).bitcast(cutlass.Int32) for t in range(mma_n)] + for g in cutlass.range_constexpr(mma_n // 4): + _st_async_v4( + dst + cutlass.Int32(16 * g), + words[4 * g], + words[4 * g + 1], + words[4 * g + 2], + words[4 * g + 3], + mb, + ) + else: + for t in cutlass.range_constexpr(mma_n): + prims.mapa(mailbox.subview(base + cutlass.Int32(t)), owner).store( + cutlass.Float32(acc[t]) + ) + prims.mbarrier_arrive( + prims.mapa(reduce_ready, owner), scope=prims.MemScope.CLUSTER + ) + else: + if cutlass.const_expr(st_async): + while not _test_wait_cluster(reduce_ready.data_ptr(), 0): + pass + else: + while not prims.mbarrier_try_wait_parity( + reduce_ready, 0, scope=prims.MBarrierScope.CLUSTER + ): + pass + if lane == 0: + if row // cutlass.Int32(rows_per_rank) == rank: + if (row % cutlass.Int32(rows_per_rank)) == 0: + _stamp(timers, timed, bx, 5) + totals = [] + for g in cutlass.range_constexpr(mma_n // 4): + sums = [cutlass.Float32(0.0)] * 4 + for q in cutlass.range_constexpr(split): + parts = _ld_shared_v4_f32( + mailbox.subview( + (cutlass.Int32(q * rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g) + ).data_ptr() + ) + sums = [sums[j] + cutlass.select_(rank == cutlass.Int32(q), cutlass.Float32(acc[4 * g + j]), + parts[j]) for j in range(4)] # fmt: skip + for j in cutlass.range_constexpr(4): + totals.append(sums[j]) + if cutlass.const_expr(rope): + # The own-rank mailbox slot is unused: it carries the totals to the RoPE stage. + mailbox.store( + sums[j], + idx=(rank * cutlass.Int32(rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g + j), + ) + elif cutlass.const_expr(prod): + # The bf16 the packed row would hold (RN, as the y store below), staged by token. + if cutlass.Int32(4 * g + j) < num_tokens: + pr_stage.store( + cutlass.Uint16( + (_bf16_bits(sums[j]) >> cutlass.Int32(16)) + & cutlass.Int32(0xFFFF) + ), + idx=cutlass.Int32((4 * g + j) * rows_per_rank) + local, + ) + else: + if cutlass.Int32(4 * g + j) < num_tokens: + y.store( + cutlass.BFloat16(sums[j]), + idx=cutlass.Int32((4 * g + j) * n_out) + n, + ) + if cutlass.const_expr(ar_push > 0): + # Every lane of an owner warp is an owner (rows_per_rank >= 32), so lane pairs hold row pairs. + prims.griddepcontrol(prims.GridDepAction.WAIT) + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_hi = cute.arch.shuffle_sync_bfly(totals[t], offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), + _bf16x2_word(totals[t], ar_hi), + ) + prims.barrier_cta_sync(1, thread_count=128) + if cutlass.const_expr(prod): + # Transpose: thread e < 2 T sends half e % 2 (16 bytes = 8 dims) of token e / 2's staged rows to cluster + # rank e / 2, at dims [rank * rows_per_rank + 8 (e % 2), ...) of that rank's packed-head row. + pr_e = tx - cutlass.Int32(128) + if pr_e < num_tokens * cutlass.Int32(2): + pr_t = pr_e // cutlass.Int32(2) + pr_h = pr_e % cutlass.Int32(2) + pr_w = _ld_shared_v4_b32( + pr_stage.subview( + pr_t * cutlass.Int32(rows_per_rank) + pr_h * cutlass.Int32(8) + ).data_ptr() + ) + _st_async_v4( + _mapa_u32( + pr_row.subview( + rank * cutlass.Int32(rows_per_rank // 2) + pr_h * cutlass.Int32(4) + ).data_ptr(), + pr_t, + ), + pr_w[0], + pr_w[1], + pr_w[2], + pr_w[3], + _mapa_u32(pr_full.data_ptr(), pr_t), + ) + if warp_id == 4: + if rank < num_tokens: + # The producer reads step inputs (positions, slots) and writes the caches: ordered after the + # upstream grids by this thread's own grid wait (already satisfied: the activation landed). + prims.griddepcontrol(prims.GridDepAction.WAIT) + while not _test_wait_cluster(pr_full.data_ptr(), 0): + pass + pr_lo = pr_row.load(idx=lane * cutlass.Int32(2)) + pr_hi = pr_row.load(idx=lane * cutlass.Int32(2) + cutlass.Int32(1)) + mask = cutlass.Int32(0xFFFF) + producer_head( + rank, + n_tile, + lane, + bf16_bits_to_f32(pr_lo & mask), + bf16_bits_to_f32((pr_lo >> cutlass.Int32(16)) & mask), + bf16_bits_to_f32(pr_hi & mask), + bf16_bits_to_f32((pr_hi >> cutlass.Int32(16)) & mask), + ProducerArgs( + packed=cutlass.Int64(0), + q_out=pr_q, + index_q_out=pr_iq, + index_k=pr_ik, + kv_data=pr_kvd, + kv_scale=pr_kvs, + out_cache_loc=pr_loc, + kv_quant_scale=pr_quant, + position_ids=pr_pos, + rotary_cos_sin=pr_rope, + q_weight=pr_qw, + k_weight=pr_kw, + index_q_weight=pr_iqw, + index_k_weight=pr_ikw, + eps=pr_eps, + num_pages=pr_pages, + index_page_stride=pr_ips, + index_token_stride=pr_its, + data_page_stride=pr_dps, + data_plane_stride=pr_dls, + data_head_stride=pr_dhs, + data_token_stride=pr_dts, + scale_page_stride=pr_sps, + scale_plane_stride=pr_sls, + scale_head_stride=pr_shs, + ), + ) + if cutlass.const_expr(rope): + # Token t of request t / T sits at cache position kv_len - T + t % T. Q / K heads: GPT-NeoX RoPE over + # the 128-wide head, bf16(cos * x -/+ sin * pair) on the bf16 projection; then E4M3 of x * scale, Q to + # rope_q, K / V into the page of that position. + pairs = rows_per_rank // 2 + rp_e = tx - cutlass.Int32(128) + rp_p = rp_e % cutlass.Int32(pairs) + rp_t = rp_e // cutlass.Int32(pairs) + if rp_t < num_tokens: + rp_max_blocks = rope_dims.load(idx=0) + rp_tpb_log2 = rope_dims.load(idx=1) + rp_hq = rope_dims.load(idx=2) + rp_hkv = rope_dims.load(idx=3) + rp_T = rope_dims.load(idx=4) + rp_d = rank * cutlass.Int32(pairs) + rp_p + rp_a = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + rp_p) * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_b = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + cutlass.Int32(pairs) + rp_p) + * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_req = rp_t // rp_T + rp_tok = rope_i.load(idx=rp_req) - rp_T + (rp_t - rp_req * rp_T) + rp_head = n_tile + if rp_head < rp_hq + rp_hkv: + rp_c = rope_cs.load(idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2)) + rp_s = rope_cs.load( + idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2) + + cutlass.Int32(1) + ) + rp_ra = _rope_bits(rp_c, rp_a, -rp_s, rp_b).bitcast(cutlass.Float32) + rp_rb = _rope_bits(rp_c, rp_b, rp_s, rp_a).bitcast(cutlass.Float32) + rp_a = rp_ra + rp_b = rp_rb + rp_qs = rope_scale.load(idx=0) + rp_w = _e4m3x4( + rp_a * rp_qs, rp_b * rp_qs, cutlass.Float32(0.0), cutlass.Float32(0.0) + ) + rp_lo = cutlass.Uint8(rp_w & cutlass.Int32(0xFF)) + rp_hi = cutlass.Uint8((rp_w >> cutlass.Int32(8)) & cutlass.Int32(0xFF)) + if rp_head < rp_hq: + rp_q = (rp_t * rp_hq + rp_head) * cutlass.Int32(128) + rp_d + rope_q.store(rp_lo, idx=rp_q) + rope_q.store(rp_hi, idx=rp_q + cutlass.Int32(64)) + else: + rp_kv = cutlass.select_( + rp_head < rp_hq + rp_hkv, cutlass.Int32(0), cutlass.Int32(1) + ) + rp_kh = rp_head - rp_hq - rp_kv * rp_hkv + rp_page = rope_bt.load( + idx=(rp_req * cutlass.Int32(2) + rp_kv) * rp_max_blocks + + (rp_tok >> rp_tpb_log2) + ) + # 64-bit byte offset: the pool view can exceed 2^31 bytes. + rp_dst = ( + (cutlass.Int64(rp_page) * cutlass.Int64(rp_hkv) + cutlass.Int64(rp_kh)) + << cutlass.Int64(rp_tpb_log2 + cutlass.Int32(7)) + ) + cutlass.Int64( + (rp_tok & ((cutlass.Int32(1) << rp_tpb_log2) - cutlass.Int32(1))) + * cutlass.Int32(128) + + rp_d + ) + rp_addr = rope_pool.data_ptr(0).toint() + rp_dst + cutlass.inttoptr(rp_addr, mem_space=1, dtype=cutlass.Uint8).store(rp_lo) + cutlass.inttoptr( + rp_addr + cutlass.Int64(64), mem_space=1, dtype=cutlass.Uint8 + ).store(rp_hi) + if tx == 128: + _stamp(timers, timed, bx, 6) + if cutlass.const_expr(ar_push == 2): + # The all-reduce polls the pushed words; it launches after this CTA's rows are out, not at grid end. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if warp_id == 4: + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Int32), + tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n), + ) + + +def _bytes_map(t, cols, rows, box_rows, swizzle): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[cols, rows], + global_strides=[cols // 16], + box_dims=[CTA_K, box_rows], + swizzle=swizzle, + ) + + +def _atom_map(t, atoms): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint16, + global_dims=[SF_ATOM_U16, atoms], + global_strides=[SF_ATOM_BYTES // 16], + box_dims=[SF_ATOM_U16, 1], + swizzle=cuda.TensorMapSwizzle.none, + ) + + +@cute.jit +def m3_mx_gemv( + w: cute.Tensor, # [N, K] E4M3 as uint8, K contiguous + wsf: cute.Tensor, # weight scales, 128x4-swizzled, as uint8 + x: cute.Tensor, # [M, K] E4M3 as uint8, M <= 8 + xsf: cute.Tensor, # activation scales, 128x4-swizzled (atom k at byte 512 k), as uint8 + y: cute.Tensor, # [M * N] bf16 + x_words: cute.Tensor, # x_bf16: bf16 x [M, K] as int32 words (x / xsf are then unused placeholders) + timers: cute.Tensor, # [grid * 8] int64, written when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[int], + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, + din_i: cute.Tensor, # din / rope inputs: see m3_mx_gemv_kernel (placeholders when off) + din_table: cute.Tensor, + din_hidden: cute.Tensor, + din_w: cute.Tensor, + din_f: cute.Tensor, + din_vocab: cutlass.Int32, + din_g: cute.Tensor, + din_res: cute.Tensor, + din_flags: cutlass.Int32, + rope_cs: cute.Tensor, + rope_i: cute.Tensor, + rope_bt: cute.Tensor, + rope_q: cute.Tensor, + rope_pool: cute.Tensor, + rope_scale: cute.Tensor, + rope_dims: cute.Tensor, + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], + x_swiglu: cutlass.Constexpr[bool], + # prod: see m3_mx_gemv_kernel (zeros when off). + pr_q: cutlass.Int64, + pr_iq: cutlass.Int64, + pr_ik: cutlass.Int64, + pr_kvd: cutlass.Int64, + pr_kvs: cutlass.Int64, + pr_loc: cutlass.Int64, + pr_quant: cutlass.Int64, + pr_pos: cutlass.Int64, + pr_rope: cutlass.Int64, + pr_qw: cutlass.Int64, + pr_kw: cutlass.Int64, + pr_iqw: cutlass.Int64, + pr_ikw: cutlass.Int64, + pr_ips: cutlass.Int64, + pr_its: cutlass.Int64, + pr_dps: cutlass.Int64, + pr_dls: cutlass.Int64, + pr_dhs: cutlass.Int64, + pr_dts: cutlass.Int64, + pr_sps: cutlass.Int64, + pr_sls: cutlass.Int64, + pr_shs: cutlass.Int64, + pr_pages: cutlass.Int32, + pr_eps: cutlass.Float32, + prod: cutlass.Constexpr[bool], +) -> None: + # x_swiglu's up half rides the slot din uses for its norm weights, and it activates the bf16 x. + assert not (din and x_swiglu) and (x_bf16 or not x_swiglu), ( + "x_swiglu needs x_bf16 and excludes din" + ) + k_tiles = num_k_tiles(k_in) + tma_desc_w = _bytes_map(w, k_in, n_out, CTA_M, cuda.TensorMapSwizzle.s128b) + tma_desc_wsf = _atom_map(wsf, (n_out // CTA_M) * k_tiles) + tma_desc_x = _bytes_map(x, k_in, num_tokens, mma_n, cuda.TensorMapSwizzle.s128b) + tma_desc_xsf = _atom_map(xsf, k_tiles) + m3_mx_gemv_kernel( + tma_desc_w, tma_desc_wsf, tma_desc_x, tma_desc_xsf, y, x_words, timers, num_tokens, n_out, k_in, split, ring, + trigger_early, contig, timed, nacc, a_tmem, st_async, x_bf16, ar_ws, ar_flags, ar_rank, ar_push, ar_world, + ar_maxt, mma_n, + din_i, din_table, din_hidden, din_w, din_f, din_vocab, din_g, din_res, din_flags, rope_cs, + rope_i, rope_bt, rope_q, rope_pool, rope_scale, rope_dims, din, rope, x_swiglu, + pr_q, pr_iq, pr_ik, pr_kvd, pr_kvs, pr_loc, pr_quant, pr_pos, pr_rope, pr_qw, pr_kw, pr_iqw, pr_ikw, pr_ips, + pr_its, pr_dps, pr_dls, pr_dhs, pr_dts, pr_sps, pr_sls, pr_shs, pr_pages, + pr_eps, prod, + ).launch( + grid=((n_out // CTA_M) * split, 1, 1), + block=(THREADS, 1, 1), + cluster=(split, 1, 1), + stream=stream, + use_pdl=use_pdl, + ) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/op.py new file mode 100644 index 000000000000..ecf4f86fa7dd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/op.py @@ -0,0 +1,683 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) for already-quantized activations of at most 8 tokens. + +Compiled on the first call per (N, K, split, ring, early trigger, PDL, push, MMA width), which must happen outside +CUDA-graph capture, or ahead of any call by ``precompile``; once ``freeze`` closes compiles, a call whose kernel is not +compiled declines instead. +""" + +from __future__ import annotations + +import threading +from pathlib import Path +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + +from ...m31_switches import m31_disabled +from .. import m3_report + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +MMA_WIDTHS = ( + 8, + 16, + 32, + 48, +) # token columns of the compiled MMA; a call uses the smallest that holds M +_SMEM_LIMIT = 220 * 1024 # capacity: fixed: the device's shared-memory limit + + +def _mma_n(m: int) -> int: + return next(w for w in MMA_WIDTHS if m <= w) + + +def _smem_bytes(ring: int, tiles: int, split: int, mma_n: int) -> int: + return ( + ring * (128 * 128 + 512) + + tiles * (mma_n * 128 + 512) + + split * (128 // split) * (mma_n + 4) * 4 + + 1024 + ) + + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + +# (N, K) -> (split, ring): GB300 exposed time behind a 4.25 us PDL predecessor at M 1 / 8, vs the fused low-M GEMM: +# QKV 2.78 / 3.16 vs 4.58 / 6.21 us, o_proj 2.23 / 2.41 vs 4.40 / 5.09, shared gate_up 2.43 / 2.74 vs 3.56 / 4.30, +# shared down 1.74 / 1.66 vs 2.92 / 3.38. A config is (split, ring[, contig, nacc, a_tmem, st_async]); the optional +# fields default to _CONTIG, _NACC, _A_TMEM, _ST_ASYNC. +# TP2 port: the TP2 per-rank projections' (split, ring), M3 tp2's sm_107 table (M3 tp2 kernels/mx_gemv/op.py); the +# TP4 keys are not TP2 shapes ((6144, 6144) is TP4's dense gate_up but TP2's dense down). Retune on the M3.1 TP2 +# GEMV (MB) before booking. +_CONFIGS: Dict[Tuple[int, int], tuple] = { + (4992, 6144): (4, 12), # sparse-layer QKV + index + # The DSpark drafter's QKV (MXDRAFT): the sparse QKV's config, unswept at this N. + (5120, 6144): (4, 12), + (4608, 6144): (4, 12), # dense-attention QKV (unused by M3.1) + (6144, 4096): (2, 12), # o_proj + (3072, 6144): (4, 12), # shared gate_up + (6144, 1536): (2, 12), # shared down + (12288, 6144): (2, 12), # dense gate_up + (6144, 6144): (2, 12), # dense down +} +# Per-token-count configs (N, K, M): the grid that suits the concurrent routed-expert engine depends on how long +# that engine runs. At 2 tokens the shared-expert gate_up splits K over 4 CTAs. +# TP4 had {(1536, 6144, 2): ...}, not a TP2 shape. +# TP2_GEMVCFG (agg TP2 K3, sweep: SM107, mclk locked, weights rotated past L2, 2 processes): the shared down +# (6144, 1536) at 9..16 rows takes ring 2 instead of 12 with the same split 2 and contiguous k-tiles (the same fp32 +# order, bit for bit the table's output): 3.81 -> 3.07 us at 12 rows, 4.52 -> 3.10 at 16. In-model (16 +# rows): busy -20 us / step on 6 / 6 windows; the routed experts beside it hide most of it. +# The swept split-8 shared gate_up at 17..32 rows (8.76 -> 7.58 us alone at 32 rows) is not taken: in-model it slowed +# the 32-row step by 228 us, its 8x CTAs crowding the concurrent routed experts. +# Kill switch TRTLLM_M31_DISABLE_TP2_GEMVCFG=1 (the table's configs), read at import. +TP2_GEMVCFG = not m31_disabled("TP2_GEMVCFG") +_CONFIGS_BY_M: Dict[Tuple[int, int, int], tuple] = ( + {(6144, 1536, m): (2, 2, 1) for m in range(9, 17)} if TP2_GEMVCFG else {} +) # capacity: fixed: the swept rows per projection +# Above 8 tokens (spec-decode verify rows): the shared-expert gate_up splits K over 4 CTAs. +_CONFIGS_WIDE: Dict[Tuple[int, int], tuple] = { + (1536, 6144): (4, 12, 1, 1, 0, 1), +} +# VRCFG-SWI: the swiglu-prologue calls (bf16 gate | up in) at up to SWIGLU_CFG_MAX_ROWS rows take their own ring: the +# prologue's per-k-tile quantize shifts the best ring depth. Same split and k-tile order as the table, so bit for bit +# its output. SM107 sweep (8 rows, PDL back-to-back, weights rotated past L2): dense down 4.66 vs 5.86 us. Kill +# switch TRTLLM_M31_DISABLE_MXGEMV_VRCFG, read at import. +MXGEMV_VRCFG = not m31_disabled("MXGEMV_VRCFG") +SWIGLU_CFG_MAX_ROWS = 8 # capacity: fixed: the swept rows +# TP2: (6144, 1536) is the SHARED down here (TP8: the dense down); empty until a TP2 sweep. +_CONFIGS_SWIGLU: Dict[Tuple[int, int], tuple] = {} # TP8: {(6144, 1536): (2, 3, 0)} + +# QKV_PRODUCER_FOLD (the sm_107 TP8 target's #117): the producer epilogue serves the sparse-layer QKV + index +# projection, 39 packed heads of 128 (32 q, 2 k, 2 v, 2 index-q, index-k), one head per split-8 cluster. +PROD_HEADS_Q, PROD_HEADS_KV, PROD_HEADS_INDEX = ( + 32, + 2, + 2, +) # capacity: fixed: the TP2 producer's heads per rank +PROD_N = ( + PROD_HEADS_Q + 2 * PROD_HEADS_KV + PROD_HEADS_INDEX + 1 +) * 128 # capacity: fixed: the packed heads +# The kernel's producer arguments on a call without the producer epilogue (22 addresses / strides, pages, eps). +PROD_OFF = (0,) * 23 + (0.0,) + +# Defaults of the optional config fields: contiguous k-tile ranges per split rank, one TMEM accumulator, the A +# operand staged in TMEM (when its columns fit), st.async split-K partial exchange. +_CONTIG = True +_NACC = 1 +_A_TMEM = True +_ST_ASYNC = True +_ensured = set() + + +_DUMMY: Dict[torch.device, torch.Tensor] = {} + + +def _dummy_timers(device) -> torch.Tensor: + """Placeholder for the kernel's timer buffer, which it only writes when compiled with timers on.""" + t = _DUMMY.get(device) + if t is None: + t = _DUMMY[device] = torch.zeros(8, dtype=torch.int64, device=device) + return t + + +def _arg(t: torch.Tensor, align: int = 16): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align).mark_layout_dynamic(leading_dim=t.dim() - 1) + + +def get_config(n: int, k: int, m: Optional[int] = None) -> Optional[tuple]: + cfg = _CONFIGS_BY_M.get((n, k, m)) if m is not None else None + if cfg is None and m is not None and m > 8: + cfg = _CONFIGS_WIDE.get((n, k)) + cfg = cfg or _CONFIGS.get((n, k)) + # The split-K epilogue gives each of the split CTAs 128 // split rows of a 128-row output tile. + if cfg is None or n % 128 != 0 or k % 128 != 0 or (k // 128) % cfg[0] != 0 or 128 % cfg[0] != 0: + return None + return cfg + + +def _declines(n: int, k: int, m: int, x_bf16: bool, cfg: tuple) -> bool: + """Whether a (split, ring, ...) config declines an [M, K] x [N, K]^T call on shape grounds.""" + split = cfg[0] + if not 0 < m <= MAX_TOKENS or 128 % split != 0 or (k // 128) % split != 0: + return True + tiles = k // 128 // split + mma_n = _mma_n(m) + # GB300 exposed time vs the FlashInfer prequantized GEMM (behind a PDL predecessor), at 24 / 48 tokens: QKV 3.76 / + # 4.63 vs 7.46 / 11.44 us, o_proj 2.74 / 3.33 vs 4.46 / 5.84, shared gate_up 4.57 / 5.71 vs 7.16 / 11.25. Behind + # only for K = 768 (unsplit shared down) at 17-32 tokens: 3.1 vs 2.9. + # The in-kernel bf16 -> MXFP8 quantization (x_bf16) costs one 32-element block per thread and round on 128 threads; + # beyond 8 rounds per CTA, mxfp8_quantize + this GEMV (or the fallback GEMM) is faster. + if (16 < m <= 32 and k < 1024) or (x_bf16 and tiles * 4 * mma_n > 8 * 128): + return True + return _smem_bytes(min(cfg[1], tiles, 2), tiles, split, mma_n) > _SMEM_LIMIT + + +def serves(n: int, k: int, m: int, x_bf16: bool) -> bool: + """Whether the GEMV with its default config takes an [M, K] activation (bf16 or MXFP8) against an [N, K] weight.""" + cfg = get_config(n, k, m) + return cfg is not None and not _declines(n, k, m, x_bf16, cfg) + + +def _resolve( + m: int, + n: int, + k: int, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple], + x_swiglu: bool = False, + prod: bool = False, +) -> Optional[tuple]: + """The compile key of an [M, K] x [N, K]^T call and its runtime push words, or None when the call is declined. + + ``ar`` is the push tuple of ``_run``; only its (world, bucket tokens, mode) enter the key, the addresses and the + rank are runtime arguments.""" + if cfg is None and x_swiglu and m <= SWIGLU_CFG_MAX_ROWS: + cfg = _CONFIGS_SWIGLU.get((n, k)) + if cfg is not None: + m3_report.work("MXGEMV_VRCFG", f"swiglu-prologue config [{n}, {k}] at {m} rows") + cfg = cfg or get_config(n, k, m) + if cfg is None or _declines(n, k, m, x_bf16, cfg): + return None + split, ring = cfg[0], cfg[1] + contig = bool(cfg[2]) if len(cfg) > 2 else _CONTIG + nacc = int(cfg[3]) if len(cfg) > 3 else _NACC + a_tmem = bool(cfg[4]) if len(cfg) > 4 else _A_TMEM + st_async = bool(cfg[5]) if len(cfg) > 5 else _ST_ASYNC + if prod and not ( + n == PROD_N + and split == 8 + and m <= split + and st_async + and not x_bf16 + and not x_swiglu + and ar is None + ): + # The producer epilogue: the sparse QKV + index projection (one 128-row packed head per cluster of 8), at most + # one token per cluster rank, the st.async exchange, MXFP8 input, no all-reduce push. + return None + if nacc != 1: + # More than one TMEM accumulator is broken on SM107 (nondeterministic or garbage + # outputs). No shipped table uses it; decline (the caller's fallback runs) until the kernel path is fixed. + return None + ring = min(ring, k // 128 // split) + tiles = k // 128 // split + mma_n = _mma_n(m) + while ring > 2 and _smem_bytes(ring, tiles, split, mma_n) > _SMEM_LIMIT: + ring -= 1 + if a_tmem: + from .m3_mx_gemv_kernel import tmem_cols + + if tmem_cols(ring, nacc, True, tiles, mma_n) > 512: + a_tmem = False + # ar = (multicast workspace address, flags address, rank, world, bucket tokens, mode): push the output rows into + # the next m3_mc_allreduce call's exchange slots (mode 2 also defers the dependents' launch until after the push). + ar_ws, ar_flags, ar_rank, ar_mode, ar_world, ar_maxt = 0, 0, 0, 0, 1, 8 + if ar is not None and split <= 4 and n % 2 == 0: + ar_ws, ar_flags, ar_rank, ar_world, ar_maxt, ar_mode = ar + if ar_mode == 2: + trigger_early = False + if use_pdl is None: + use_pdl = get_env_enable_pdl() + # The kernel's ``timed`` flag (per-CTA milestone timers) is always off. + key = ( + n, + k, + split, + ring, + trigger_early, + contig, + False, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + x_swiglu, + prod, + ) + return key, (ar_ws, ar_flags, ar_rank) + + +def _compile( + key: tuple, args: tuple, m: int, runtime_ar: tuple, stream, extra: tuple, prod_args: tuple +): + """cute.compile the kernel of ``key`` (see ``_resolve``) on these argument views; cached.""" + import cutlass.cute as cute + + from . import m3_mx_gemv_kernel as kernel + + ( + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + x_swiglu, + prod, + ) = key + ar_ws, ar_flags, ar_rank = runtime_ar + with _lock: + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_mx_gemv {key}", + lambda: cute.compile( + kernel.m3_mx_gemv, + *args, + m, + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_ws, + ar_flags, + ar_rank, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + stream, + *extra, + False, + False, + x_swiglu, + *prod_args, + prod, + ), + sources=_aot_sources(), + ) + return fn + + +def _views( + x: torch.Tensor, + xsf: Optional[torch.Tensor], + weight, + weight_scale, + y, + x_bf16: bool, + x_swiglu: bool = False, +) -> tuple: + # x_swiglu: x is [M, 2K] (gate | up); the GEMV's K is half its width. + k = x.shape[1] // 2 if x_swiglu else x.shape[1] + if x_bf16: + x_bytes = x.view(torch.uint8) + xsf = _placeholder_scales(k, x.device) + x_words = x.view(torch.int32) + else: + x_bytes = x.view(torch.uint8) + x_words = _placeholder_words(x.device) + return ( + _arg(weight.view(torch.uint8)), + _arg(weight_scale.view(torch.uint8).reshape(-1)), + _arg(x_bytes), + _arg(xsf.view(torch.uint8).reshape(-1)), + _arg(y.view(-1) if y.numel() else _placeholder_words(x.device).view(torch.bfloat16)), + _arg(x_words.reshape(-1)), + _arg(_dummy_timers(x.device)), + ) + + +def _run( + x: torch.Tensor, + xsf: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple] = None, + x_swiglu: bool = False, + prod_args: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + global last_pushed, last_uncompiled + last_pushed = False + last_uncompiled = None + import cuda.bindings.driver as cuda_driver + + m, k = x.shape + if x_swiglu: + k //= 2 + n = weight.shape[0] + if weight.shape[1] != k: + return None + prod = prod_args is not None + resolved = _resolve(m, n, k, x_bf16, trigger_early, cfg, use_pdl, ar, x_swiglu, prod) + if resolved is None: + return None + key, runtime_ar = resolved + if not prod: + prod_args = PROD_OFF + fn = _compiled.get(key) + if fn is None: + if _frozen[0]: + # Compiles are closed (the model built every kernel it dispatches to before warmup): decline, the caller + # reports the miss. + last_uncompiled = key + return None + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_mx_gemv must run once per shape outside CUDA-graph capture first" + ) + # prod: no packed row is stored (the producer epilogue consumes it); y is the placeholder. + y = torch.empty(0 if prod else m, n, dtype=torch.bfloat16, device=x.device) + args = _views(x, xsf, weight, weight_scale, y, x_bf16, x_swiglu) + extra = _extra_args(x.device) + stream = cuda_driver.CUstream(torch.cuda.current_stream(x.device).cuda_stream) + if fn is None: + fn = _compile(key, args, m, runtime_ar, stream, extra, prod_args) + ar_ws, ar_flags, ar_rank = runtime_ar + fn(*args, m, ar_ws, ar_flags, ar_rank, stream, *extra, *prod_args) + last_pushed = key[11] > 0 + return y + + +def precompile( + weight: torch.Tensor, + weight_scale: torch.Tensor, + m: int, + x_bf16: bool, + use_pdl: Optional[bool] = None, + ar: Optional[Tuple[int, int, int]] = None, + x_swiglu: bool = False, + prod: bool = False, +) -> Optional[tuple]: + """Compile, without launching, the kernel an M-row call against ``weight`` takes; returns its key, or None when + such a call is declined. ``ar`` = (world, bucket tokens, mode) of a pushing call (its addresses and rank are + runtime arguments, so none is needed here). Runs outside capture, even while compiles are frozen.""" + import cuda.bindings.driver as cuda_driver + + n, k = weight.shape + push = None if ar is None else (0, 0, 0, *ar) + resolved = _resolve(m, n, k, x_bf16, True, None, use_pdl, push, x_swiglu, prod) + if resolved is None: + return None + key, runtime_ar = resolved + if key in _compiled: + return key + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_mx_gemv.precompile allocates and compiles: call it outside capture") + dev = weight.device + x = torch.zeros( + m, + 2 * k if x_swiglu else k, + dtype=torch.bfloat16 if x_bf16 else torch.float8_e4m3fn, + device=dev, + ) + xsf = None if x_bf16 else torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=dev) + y = torch.empty(m, n, dtype=torch.bfloat16, device=dev) + args = _views(x, xsf, weight, weight_scale, y, x_bf16, x_swiglu) + stream = cuda_driver.CUstream(torch.cuda.current_stream(dev).cuda_stream) + _compile(key, args, m, runtime_ar, stream, _extra_args(dev), PROD_OFF) + return key + + +def freeze(frozen: bool = True) -> None: + """Close (or reopen) compiles: while frozen, a call whose kernel is not compiled yet declines (returns None and + sets ``last_uncompiled``) instead of compiling. ``precompile`` and ``ensure_compiled`` still compile.""" + _frozen[0] = bool(frozen) + + +def frozen() -> bool: + return _frozen[0] + + +def is_compiled(key: tuple) -> bool: + return key in _compiled + + +def _extra_args(device) -> tuple: + """Placeholder arguments of the kernel's draft-input prologue and RoPE / KV epilogue (compiled off).""" + i32 = _placeholder_words(device) + f32 = i32.view(torch.float32) + u8 = i32.view(torch.uint8) + d = (_arg(i32), _arg(i32), _arg(i32), _arg(i32), _arg(f32), 0, _arg(i32), _arg(i32), 0) + r = (_arg(f32), _arg(i32), _arg(i32), _arg(u8), _arg(u8), _arg(f32), _arg(i32)) + return d + r + + +def m3_mx_gemv( + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """``xq`` E4M3 [M <= 8, K] with ``xsf`` its 128x4-swizzled UE8M0 scales; ``weight`` E4M3 [N, K] with + ``weight_scale`` swizzled. Returns bf16 [M, N], or None when the shape is not served.""" + return _run(xq, xsf, weight, weight_scale, False, trigger_early, cfg, use_pdl, ar) + + +def m3_mx_gemv_bf16( + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """The same GEMV for a bf16 ``x`` [M <= 8, K] (contiguous), quantized to MXFP8 inside the kernel exactly as + torch.ops.trtllm.mxfp8_quantize(x, True) does.""" + if x.dtype != torch.bfloat16 or x.dim() != 2 or not x.is_contiguous() or x.data_ptr() % 16 != 0: + return None + return _run(x, None, weight, weight_scale, True, trigger_early, cfg, use_pdl, ar) + + +def m3_mx_gemv_swiglu( + gate_up: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """``m3_mx_gemv_bf16(swiglu(gate_up), ...)`` in one launch: ``gate_up`` bf16 [M <= 8, 2K] (contiguous, gate then + up) is activated in the kernel's quantize prologue with the swigluoai arithmetic of the Triton silu_and_mul kernel + (alpha 1.702, limit 7, the (up + 1) offset; bit for bit), then quantized as the bf16 path does.""" + if ( + gate_up.dtype != torch.bfloat16 + or gate_up.dim() != 2 + or not gate_up.is_contiguous() + or gate_up.data_ptr() % 16 != 0 + or gate_up.shape[1] % 2 != 0 + ): + return None + return _run(gate_up, None, weight, weight_scale, True, trigger_early, cfg, use_pdl, ar, True) + + +def m3_mx_gemv_producer( + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + q: torch.Tensor, + index_q: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + cfg: Optional[tuple] = None, + trigger_early: bool = True, + use_pdl: Optional[bool] = None, +) -> bool: + """QKV_PRODUCER_FOLD: ``m3_producer(m3_mx_gemv(xq, xsf, weight, weight_scale, cfg=cfg), ...)`` in one launch, bit + for bit: the QKV + index GEMV of the sparse layer (``weight`` [39 x 128, K]; ``cfg`` the (split, ring) override the + plain call takes, which must resolve to split 8) whose owner epilogue hands each token's packed head to the + producer (``phase_producer.producer_head``). Writes ``q`` FP8 [T, 32, 128] and ``index_q`` FP8 [T, 2, 128] and the + token's cache slot (NVFP4 K / V + scales of both KV heads, FP8 index-K) exactly as the producer does; the caches + and tensors are those ``minimax_m3_sm107_tp2_producer_pw`` takes (kv_data [pages, 2, 2, 128, 64] u8, kv_scale + [pages, 2, 2, 128, 8] u8, index_k [pages, 1, 128, 128] e4m3). Returns False (nothing launched) when the call is + declined. + """ + m = xq.shape[0] + if ( + m > 8 + or weight.shape[0] != PROD_N + or q.shape != (m, PROD_HEADS_Q, 128) + or index_q.shape != (m, PROD_HEADS_INDEX, 128) + or not (q.is_contiguous() and index_q.is_contiguous()) + or kv_data.dim() != 5 + or tuple(kv_data.shape[1:]) != (2, PROD_HEADS_KV, 128, 64) + or kv_data.stride()[2:] != (128 * 64, 64, 1) + or kv_scale.dim() != 5 + or tuple(kv_scale.shape[1:]) != (2, PROD_HEADS_KV, 128, 8) + or kv_scale.stride()[2:] != (128 * 8, 8, 1) + or kv_scale.shape[0] != kv_data.shape[0] + or index_k.dim() != 4 + ): + return False + tensors = ( + q, + index_q, + index_k, + kv_data, + kv_scale, + out_cache_loc, + kv_quant_scale, + position_ids, + rotary_cos_sin, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + ) + prod_args = ( + *(t.data_ptr() for t in tensors), + index_k.stride(0), + index_k.stride(2), + kv_data.stride(0), + kv_data.stride(1), + kv_data.stride(2), + kv_data.stride(3), + kv_scale.stride(0), + kv_scale.stride(1), + kv_scale.stride(2), + kv_data.shape[0], + float(eps), + ) + out = _run( + xq, xsf, weight, weight_scale, False, trigger_early, cfg, use_pdl, None, False, prod_args + ) + return out is not None + + +_PLACEHOLDERS: Dict[tuple, torch.Tensor] = {} +# Whether the last call pushed its output into an all-reduce (see _run's ``ar``). +last_pushed = False +# The key of the last call declined because compiles are frozen and its kernel was not compiled (see ``freeze``). +last_uncompiled: Optional[tuple] = None +_frozen = [False] + + +def _placeholder_scales(k: int, device) -> torch.Tensor: + key = ("sf", k, device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(k // 128 * 512, dtype=torch.uint8, device=device) + return t + + +def _placeholder_words(device) -> torch.Tensor: + key = ("w", device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(16, dtype=torch.int32, device=device) + return t + + +def ensure_compiled(weight: torch.Tensor, weight_scale: torch.Tensor, use_pdl: bool) -> None: + """Compiles the kernel of this weight's shape outside capture (a one-token dummy call).""" + if ( + weight.dim() != 2 + or weight.dtype != torch.float8_e4m3fn + or get_config(*weight.shape) is None + ): + return + n, k = weight.shape + if (n, k, use_pdl) in _ensured: + return + _ensured.add((n, k, use_pdl)) + xq = torch.zeros(1, k, dtype=torch.float8_e4m3fn, device=weight.device) + xsf = torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=weight.device) + m3_mx_gemv(xq, xsf, weight, weight_scale, use_pdl=use_pdl) + for width in MMA_WIDTHS[1:]: + m3_mx_gemv( + torch.zeros(width, k, dtype=torch.float8_e4m3fn, device=weight.device), + xsf, + weight, + weight_scale, + use_pdl=use_pdl, + ) + m3_mx_gemv_bf16( + torch.zeros(1, k, dtype=torch.bfloat16, device=weight.device), + weight, + weight_scale, + use_pdl=use_pdl, + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/phase_producer.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/phase_producer.py new file mode 100644 index 000000000000..66e6420f1e4d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/mx_gemv/phase_producer.py @@ -0,0 +1,508 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode producer of one packed head, for the QKV GEMV's producer epilogue (``QKV_PRODUCER_FOLD``): q / k / +index-q / index-k RMSNorm + NeoX partial RoPE, NVFP4 K / V and FP8 index-K cache insert, FP8 q / index-q out. + +Bit for bit ``torch.ops.trtllm.minimax_m3_sm107_tp2_producer_pw`` (cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/ +producerPw2.cu, m3_producer_kernel) at the TP2 geometry: 32 query heads, 2 KV heads, 2 index-q heads and one index-K +head, head_dim 128, rotary_dim 64, page 128. The sm_107 TP8 target's m3_layer.phase_producer at these heads: lane l of +the calling warp holds elements [4l, 4l + 4) of the head exactly as the C++ kernel's warp, so every reduction (the RMS +sum of squares, the RoPE pair exchange, the NVFP4 block max) runs the same shuffle tree in the same order. Every +floating-point operation is written as its PTX instruction (mul.rn / add.rn / fma.rn / rsqrt.approx / rcp.approx.ftz / +cvt) so the DSL compiler cannot contract or reassociate differently from the library build (--use_fast_math). The +places where that build contracted are marked CONTRACT and follow the shipped SASS (msa/results/sassdump-*). + +Packed row layout per token: [32 q | 2 k | 2 v | 2 index_q | index_k] x 128 BF16 (the QKV GEMV output rounded to +BF16). +""" + +from __future__ import annotations + +import cutlass +import cutlass.cute as cute +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op + +HEADS_Q = 32 # capacity: fixed: the tp2 query heads per rank +HEADS_KV = 2 # capacity: fixed: the tp2 KV heads per rank +HEADS_INDEX = 2 # capacity: fixed: the tp2 index-q heads per rank +PACKED_HEADS = ( + HEADS_Q + 2 * HEADS_KV + HEADS_INDEX + 1 +) # q x 32, k x 2, v x 2, index_q x 2, index_k +HEAD_DIM = 128 +ROTARY_DIM = 64 # capacity: fixed: the checkpoint's rotary dim +PAGE = 128 +ELEMS = HEAD_DIM // 32 # capacity: fixed: elements per lane (per lane) +# The first packed head of each kind (producerPw2.cu kBegin / vBegin / indexQBegin / indexKHead). +K_BEGIN = HEADS_Q +V_BEGIN = K_BEGIN + HEADS_KV +IQ_BEGIN = V_BEGIN + HEADS_KV +IK_HEAD = IQ_BEGIN + HEADS_INDEX +# Head kinds (``producer_head``'s ``kind``). +KIND_Q, KIND_K, KIND_V, KIND_IQ, KIND_IK = 0, 1, 2, 3, 4 + + +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +# ---- exact float arithmetic (no contraction) --------------------------------------------------------------------- +@dsl_user_op +def f_mul(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_add(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_fma(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_max(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "max.f32 $0, $1, $2;", + "=f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_abs(a, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "abs.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def f_neg(a, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "neg.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def f_rsqrt(a, *, loc=None, ip=None): + # NOT ftz: both producer builds this one is held to (the library producerPw and the target's own) run + # rsqrt.approx.f32 with denormal scaling; .ftz differed on a denormal RMS argument (eps 0). + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rsqrt.approx.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def f_rcp(a, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def bf16_bits_to_f32(bits16, *, loc=None, ip=None): + """The float value of a BF16 given in the low 16 bits of an Int32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(bits16)], + "{\n\t.reg .b32 t;\n\tshl.b32 t, $1, 16;\n\tmov.b32 $0, t;\n\t}", + "=f,r", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def round_bf16(a, *, loc=None, ip=None): + """__bfloat162float(__float2bfloat16_rn(a)).""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a)], + "{\n\t.reg .b16 h;\n\t.reg .b32 t;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b32 t, {0, h};\n\tmov.b32 $0, t;\n\t}", + "=f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def e4m3x4_word(e0, e1, e2, e3, *, loc=None, ip=None): + """store_fp8x4: bytes [e0, e1, e2, e3] (cvt.rn.satfinite.e4m3x2.f32 puts its first source in the high byte).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(e0), cutlass.Float32(e1), cutlass.Float32(e2), cutlass.Float32(e3)], + "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}", + "=r,f,f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def e4m3_byte_and_value(x, *, loc=None, ip=None): + """(byte, float value) of __nv_fp8_e4m3(x): the encoded scale and static_cast of it.""" + byte = cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n\t.reg .b16 p;\n\t.reg .b32 w;\n\tcvt.rn.satfinite.e4m3x2.f32 p, 0f00000000, $1;\n\t" + "cvt.u32.u16 w, p;\n\tand.b32 $0, w, 0xFF;\n\t}", + "=r,f", + loc=loc, + ip=ip, + ) + ) + value = cutlass.Float32( + _asm( + _T.f32(), + [byte], + "{\n\t.reg .b16 p;\n\t.reg .b32 h2;\n\t.reg .b16 h;\n\tcvt.u16.u32 p, $1;\n\t" + "cvt.rn.f16x2.e4m3x2 h2, p;\n\tmov.b32 {h, _}, h2;\n\tcvt.f32.f16 $0, h;\n\t}", + "=f,r", + loc=loc, + ip=ip, + ) + ) + return byte, value + + +@dsl_user_op +def e2m1x4_halfword(q0, q1, q2, q3, *, loc=None, ip=None): + """The low 16 bits of fp32x8_to_e2m1 for q[0..3] (q[4..7] = 0): byte0 = (q1, q0), byte1 = (q3, q2).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(q0), cutlass.Float32(q1), cutlass.Float32(q2), cutlass.Float32(q3)], + "{\n\t.reg .b8 b0, b1;\n\t.reg .b32 w;\n\tcvt.rn.satfinite.e2m1x2.f32 b0, $2, $1;\n\t" + "cvt.rn.satfinite.e2m1x2.f32 b1, $4, $3;\n\tmov.b32 w, {b0, b1, 0, 0};\n\tmov.b32 $0, w;\n\t}", + "=r,f,f,f,f", + loc=loc, + ip=ip, + ) + ) + + +# ---- global memory ----------------------------------------------------------------------------------------------- +@dsl_user_op +def ld_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm(_T.i64(), [cutlass.Int64(addr)], "ld.global.u64 $0, [$1];", "=l,l", True, loc, ip) + ) + + +@dsl_user_op +def ld_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm(_T.i32(), [cutlass.Int64(addr)], "ld.global.s32 $0, [$1];", "=r,l", True, loc, ip) + ) + + +@dsl_user_op +def ld_f32(addr, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Int64(addr)], "ld.global.f32 $0, [$1];", "=f,l", True, loc, ip) + ) + + +@dsl_user_op +def st_u32(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(v)], + "st.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def st_u16(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(v)], + "{\n\t.reg .b16 h;\n\tcvt.u16.u32 h, $1;\n\tst.global.u16 [$0], h;\n\t}", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def st_u8(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(v)], + "{\n\t.reg .b16 h;\n\tcvt.u16.u32 h, $1;\n\tst.global.u8 [$0], h;\n\t}", + "l,r", + True, + loc, + ip, + ) + + +# ---- CONTRACT points (follow the shipped SASS; see the module docstring) ------------------------------------------- +def sumsq_pair(acc, x, y): + """sumSquares += x * x + y * y (C++: one pair of the packed row). CONTRACT: fma(y, y, x * x) then add.""" + return f_add(acc, f_fma(y, y, f_mul(x, x))) + + +def rms_reciprocal(sum_squares, eps): + """rsqrtf(sumSquares / 128 + eps). CONTRACT: the / 128 is an exact * 2^-7; fused with + eps into one fma.""" + return f_rsqrt(f_fma(sum_squares, cutlass.Float32(1.0 / HEAD_DIM), eps)) + + +class ProducerArgs: + """Raw device addresses (int64) + strides, as the C++ Params (producerPw2.h) carries them.""" + + def __init__( + self, + packed, + q_out, + index_q_out, + index_k, + kv_data, + kv_scale, + out_cache_loc, + kv_quant_scale, + position_ids, + rotary_cos_sin, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + eps, + num_pages, + index_page_stride, + index_token_stride, + data_page_stride, + data_plane_stride, + data_head_stride, + data_token_stride, + scale_page_stride, + scale_plane_stride, + scale_head_stride, + ): + self.__dict__.update(locals()) + del self.__dict__["self"] + + +@cute.jit +def producer_head(tok, head, lane, x0, x1, x2, x3, a: ProducerArgs): + """Everything after the packed-row load for packed head ``head`` (warp-uniform; dynamic Int32 or Constexpr) of + decode token ``tok``, on the calling warp: lane ``lane`` holds the head's elements [4 lane, 4 lane + 4) as the + float values x0..x3 of the BF16 packed row (the QKV GEMV output rounded to BF16). Norm weights, KV quant scale, + position and RoPE loads, the RMS tree, RoPE, and the q / index-q / index-K / K-V NVFP4 stores, at the TP2 heads + (K and V rows into their head's block of the HND pools, index-q into its head of ``index_q_out``). + + Data flow is branch-free (every load is in bounds for every head: V reads the q norm weight, a non-rotating lane + reads its token's RoPE row, and the results are then discarded with select); only the stores branch, on + warp-uniform conditions, so every shuffle runs on the full warp as in the C++ kernel.""" + l4 = cutlass.Int64(lane * ELEMS) + # The head's kind: the packed row's heads run q, k, v, index-q, index-k in order. + kind = cutlass.select_( + head >= IK_HEAD, + cutlass.Int32(KIND_IK), + cutlass.select_( + head >= IQ_BEGIN, + cutlass.Int32(KIND_IQ), + cutlass.select_( + head >= V_BEGIN, + cutlass.Int32(KIND_V), + cutlass.select_(head >= K_BEGIN, cutlass.Int32(KIND_K), cutlass.Int32(KIND_Q)), + ), + ), + ) + is_v = kind == KIND_V + zero = cutlass.Float32(0.0) + + # Norm weights (BF16): q / k / index-q / index-k weight by head kind (V: q's, unused). + w_base = cutlass.select_( + kind == KIND_K, + a.k_weight, + cutlass.select_( + kind == KIND_IQ, + a.index_q_weight, + cutlass.select_(kind == KIND_IK, a.index_k_weight, a.q_weight), + ), + ) + ww = ld_u64(w_base + l4 * 2) + w0 = bf16_bits_to_f32(cutlass.Int32(ww & cutlass.Int64(0xFFFF))) + w1 = bf16_bits_to_f32(cutlass.Int32((ww >> cutlass.Int64(16)) & cutlass.Int64(0xFFFF))) + w2 = bf16_bits_to_f32(cutlass.Int32((ww >> cutlass.Int64(32)) & cutlass.Int64(0xFFFF))) + w3 = bf16_bits_to_f32(cutlass.Int32((ww >> cutlass.Int64(48)) & cutlass.Int64(0xFFFF))) + quant = ld_f32(a.kv_quant_scale + cutlass.select_(is_v, cutlass.Int64(8), cutlass.Int64(4))) + + # Position id, RoPE row. + pos = ld_s32(a.position_ids + cutlass.Int64(tok) * 4) + row = cutlass.Int64(pos) * ROTARY_DIM + coef = cutlass.Int64((lane * ELEMS) % (ROTARY_DIM // 2)) + c0 = ld_f32(a.rotary_cos_sin + (row + coef) * 4) + c1 = ld_f32(a.rotary_cos_sin + (row + coef + 1) * 4) + c2 = ld_f32(a.rotary_cos_sin + (row + coef + 2) * 4) + c3 = ld_f32(a.rotary_cos_sin + (row + coef + 3) * 4) + s0 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef) * 4) + s1 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef + 1) * 4) + s2 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef + 2) * 4) + s3 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef + 3) * 4) + + # RMSNorm (all heads compute it; V keeps x). + ss = sumsq_pair(zero, x0, x1) + ss = sumsq_pair(ss, x2, x3) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=16)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=8)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=4)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=2)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=1)) + r = rms_reciprocal(ss, a.eps) + one = cutlass.Float32(1.0) + n0 = f_mul(x0, f_mul(r, f_add(one, w0))) + n1 = f_mul(x1, f_mul(r, f_add(one, w1))) + n2 = f_mul(x2, f_mul(r, f_add(one, w2))) + n3 = f_mul(x3, f_mul(r, f_add(one, w3))) + # NeoX partial RoPE: lane l's dim 4l + i pairs with lane l ^ 8 (dim +-32) within the first 64 dims. + pair_off = (ROTARY_DIM // 2) // ELEMS + p0 = cute.arch.shuffle_sync_bfly(n0, offset=pair_off) + p1 = cute.arch.shuffle_sync_bfly(n1, offset=pair_off) + p2 = cute.arch.shuffle_sync_bfly(n2, offset=pair_off) + p3 = cute.arch.shuffle_sync_bfly(n3, offset=pair_off) + rot = lane * ELEMS < ROTARY_DIM + low = lane * ELEMS < ROTARY_DIM // 2 + p0 = cutlass.select_(low, f_neg(p0), p0) + p1 = cutlass.select_(low, f_neg(p1), p1) + p2 = cutlass.select_(low, f_neg(p2), p2) + p3 = cutlass.select_(low, f_neg(p3), p3) + r0 = cutlass.select_(rot, f_fma(p0, s0, f_mul(n0, c0)), n0) + r1 = cutlass.select_(rot, f_fma(p1, s1, f_mul(n1, c1)), n1) + r2 = cutlass.select_(rot, f_fma(p2, s2, f_mul(n2, c2)), n2) + r3 = cutlass.select_(rot, f_fma(p3, s3, f_mul(n3, c3)), n3) + e0 = cutlass.select_(is_v, x0, r0) + e1 = cutlass.select_(is_v, x1, r1) + e2 = cutlass.select_(is_v, x2, r2) + e3 = cutlass.select_(is_v, x3, r3) + + if kind == KIND_Q: + st_u32( + a.q_out + (cutlass.Int64(tok) * HEADS_Q + cutlass.Int64(head)) * HEAD_DIM + l4, + e4m3x4_word(e0, e1, e2, e3), + ) + if kind == KIND_IQ: + st_u32( + a.index_q_out + + (cutlass.Int64(tok) * HEADS_INDEX + cutlass.Int64(head - IQ_BEGIN)) * HEAD_DIM + + l4, + e4m3x4_word(e0, e1, e2, e3), + ) + if kind != KIND_Q: + if kind != KIND_IQ: + slot = ld_s32(a.out_cache_loc + cutlass.Int64(tok) * 4) + page = slot // PAGE + if slot >= 0: + if page < a.num_pages: + within = slot & (PAGE - 1) + if kind == KIND_IK: + st_u32( + a.index_k + + cutlass.Int64(page) * a.index_page_stride + + cutlass.Int64(within) * a.index_token_stride + + l4, + e4m3x4_word(e0, e1, e2, e3), + ) + else: + plane = cutlass.select_(is_v, cutlass.Int64(1), cutlass.Int64(0)) + kv_head = cutlass.Int64( + cutlass.select_(is_v, head - V_BEGIN, head - K_BEGIN) + ) + col = lane // 4 + sw = cutlass.select_( + is_v, + cutlass.Int64((within // 4) * 32 + col * 4 + within % 4), + cutlass.Int64(within * 8 + col), + ) + s_off = ( + cutlass.Int64(page) * a.scale_page_stride + + plane * a.scale_plane_stride + + kv_head * a.scale_head_stride + + sw + ) + # quantize_fp4x4: BF16-rounded values, 16-element block max over the 4-lane group. + q0 = round_bf16(e0) + q1 = round_bf16(e1) + q2 = round_bf16(e2) + q3 = round_bf16(e3) + lmax = f_max( + f_max(f_max(f_max(zero, f_abs(q0)), f_abs(q1)), f_abs(q2)), f_abs(q3) + ) + lmax = f_max(lmax, cute.arch.shuffle_sync_bfly(lmax, offset=1)) + lmax = f_max(lmax, cute.arch.shuffle_sync_bfly(lmax, offset=2)) + sbyte, sval = e4m3_byte_and_value( + f_mul(quant, f_mul(lmax, f_rcp(cutlass.Float32(6.0)))) + ) + if lane % 4 == 0: + st_u8(a.kv_scale + s_off, sbyte) + outs = cutlass.select_(lmax != zero, f_rcp(f_mul(sval, f_rcp(quant))), zero) + hw = e2m1x4_halfword( + f_mul(q0, outs), f_mul(q1, outs), f_mul(q2, outs), f_mul(q3, outs) + ) + d_off = ( + cutlass.Int64(page) * a.data_page_stride + + plane * a.data_plane_stride + + kv_head * a.data_head_stride + + cutlass.Int64(within) * a.data_token_stride + + cutlass.Int64(lane * 2) + ) + st_u16(a.kv_data + d_off, hw) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/__init__.py new file mode 100644 index 000000000000..52a7a9daf028 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/clc_tile_scheduler.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/clc_tile_scheduler.py new file mode 100644 index 000000000000..b7e09877c83e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/clc_tile_scheduler.py @@ -0,0 +1,559 @@ +# Copyright (c) 2025 - 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: BSD-3-Clause + +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: + +# 1. Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. + +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. + +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. + +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import inspect +from typing import Any, Optional, Tuple + +import cutlass +import cutlass.cute as cute +from cutlass._mlir import ir +from cutlass.cutlass_dsl import ( + Boolean, + Int32, + Integer, + dsl_user_op, + extract_mlir_values, + new_from_mlir_values, +) +from cutlass.utils.static_persistent_tile_scheduler import WorkTileInfo + + +class ClcDynamicPersistentTileSchedulerParams: + """A class to represent parameters for a dynamic persistent tile scheduler. + + This class is designed to manage and compute the layout of clusters and tiles + in a batched gemm problem. + + :ivar cluster_shape_mn: Shape of the cluster in (m, n) dimensions (K dimension cta count must be 1). + :type cluster_shape_mn: tuple + """ + + @dsl_user_op + def __init__( + self, + problem_shape_ntile_mnl: cute.Shape, + cluster_shape_mnk: cute.Shape, + swizzle_size: int = 1, + raster_along_m: bool = True, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, + ) -> None: + """ + Initializes the ClcDynamicPersistentTileSchedulerParams with the given parameters. + + :param problem_shape_ntile_mnl: The shape of the problem in terms of + number of CTA (Cooperative Thread Array) in (m, n, l) dimensions. + :type problem_shape_ntile_mnl: cute.Shape + :param cluster_shape_mnk: The shape of the cluster in (m, n) dimensions. + :type cluster_shape_mnk: cute.Shape + :param swizzle_size: Swizzling size in the unit of cluster. 1 means no swizzle + :type swizzle_size: int + :param raster_along_m: Rasterization order of clusters. Only used when swizzle_size > 1. + True means along M, false means along N. + :type raster_along_m: bool + + :raises ValueError: If cluster_shape_k is not 1. + """ + + if cluster_shape_mnk[2] != 1: # type: ignore[index] + raise ValueError(f"unsupported cluster_shape_k {cluster_shape_mnk[2]}") # type: ignore[index] + if swizzle_size < 1: + raise ValueError(f"expect swizzle_size >= 1, but get {swizzle_size}") + + self.problem_shape_ntile_mnl = problem_shape_ntile_mnl + # cluster_shape_mnk is kept for reconstruction + self._cluster_shape_mnk = cluster_shape_mnk + self.cluster_shape_mn = cluster_shape_mnk[:2] # type: ignore[index] + self.swizzle_size = swizzle_size + self._raster_along_m = raster_along_m + self.cluster_shape_major_fdd = None + self.cluster_shape_minor_fdd = None + self._loc = loc + + # By default, we follow m major (col-major) raster order, so make a col-major layout + self.problem_layout_ncluster_mnl = cute.make_layout( + cute.ceil_div( + self.problem_shape_ntile_mnl, + cluster_shape_mnk[:2], # type: ignore[index] + loc=loc, + ip=ip, + ), + loc=loc, + ip=ip, + ) + + # Apply swizzle if swizzle_size > 1 + if swizzle_size > 1: + problem_shape_ncluster_mnl = cute.round_up( + self.problem_layout_ncluster_mnl.shape, + (1, swizzle_size, 1) if raster_along_m else (swizzle_size, 1, 1), + ) + + if raster_along_m: + self.problem_layout_ncluster_mnl = cute.make_layout( + ( + problem_shape_ncluster_mnl[0], # type: ignore[index] + (swizzle_size, problem_shape_ncluster_mnl[1] // swizzle_size), # type: ignore[index, operator] + problem_shape_ncluster_mnl[2], # type: ignore[index] + ), + stride=( + swizzle_size, + (1, swizzle_size * problem_shape_ncluster_mnl[0]), # type: ignore[index] + problem_shape_ncluster_mnl[0] * problem_shape_ncluster_mnl[1], # type: ignore[index, operator] + ), + loc=loc, + ip=ip, + ) + else: + self.problem_layout_ncluster_mnl = cute.make_layout( + ( + (swizzle_size, problem_shape_ncluster_mnl[0] // swizzle_size), # type: ignore[index, operator] + problem_shape_ncluster_mnl[1], # type: ignore[index] + problem_shape_ncluster_mnl[2], # type: ignore[index] + ), + stride=( + (1, swizzle_size * problem_shape_ncluster_mnl[1]), # type: ignore[index] + swizzle_size, + problem_shape_ncluster_mnl[0] * problem_shape_ncluster_mnl[1], # type: ignore[index, operator] + ), + loc=loc, + ip=ip, + ) + elif not raster_along_m: + cluster_count_major = self.problem_layout_ncluster_mnl.shape[1] + cluster_count_minor = self.problem_layout_ncluster_mnl.shape[0] + self.cluster_shape_major_fdd = cute.fast_divmod_create_divisor( + cluster_count_major, loc=loc, ip=ip + ) + self.cluster_shape_minor_fdd = cute.fast_divmod_create_divisor( + cluster_count_minor, loc=loc, ip=ip + ) + + def __extract_mlir_values__(self) -> list[ir.Value]: + values, self._values_pos = [], [] + for obj in [ + self.problem_shape_ntile_mnl, + self._cluster_shape_mnk, + self.swizzle_size, + self._raster_along_m, + ]: + obj_values = extract_mlir_values(obj) + values += obj_values + self._values_pos.append(len(obj_values)) + + # Add FastDivmod divisors to MLIR values for Host->Device transfer + # Only add non-None values to avoid MLIR type errors + fastdivmod_values = [] + fastdivmod_indices = [] # Track which FastDivmod objects are present + + for i, (fdd_name, fdd_obj) in enumerate( + [ + ("cluster_shape_major_fdd", self.cluster_shape_major_fdd), + ("cluster_shape_minor_fdd", self.cluster_shape_minor_fdd), + ] + ): + if fdd_obj is not None: + # Extract MLIR values from FastDivmodDivisor objects + fdd_values = extract_mlir_values(fdd_obj) + fastdivmod_values.extend(fdd_values) + fastdivmod_indices.append(i) + + values += fastdivmod_values + self._values_pos.append( + len(fastdivmod_indices) + ) # Store count of FastDivmod objects, not values + self._fastdivmod_indices = fastdivmod_indices # Store for reconstruction + + return values + + def __new_from_mlir_values__( + self, values: list[ir.Value] + ) -> "ClcDynamicPersistentTileSchedulerParams": + obj_list = [] + values_copy = list(values) # Make a copy to avoid modifying original + + # Reconstruct original objects from MLIR values + for obj, n_items in zip( + [ + self.problem_shape_ntile_mnl, + self._cluster_shape_mnk, + self.swizzle_size, + self._raster_along_m, + ], + self._values_pos[:-1], # Exclude FastDivmod count + ): + obj_list.append(new_from_mlir_values(obj, values_copy[:n_items])) + values_copy = values_copy[n_items:] + + new_params = ClcDynamicPersistentTileSchedulerParams(*(tuple(obj_list)), loc=self._loc) + + # Restore FastDivmod divisors from remaining values + fdd_names = [ + "cluster_shape_major_fdd", + "cluster_shape_minor_fdd", + ] + + if hasattr(self, "_fastdivmod_indices") and len(self._fastdivmod_indices) > 0: + # Override the FastDivmod divisors created by __init__ with reconstructed ones + for j, original_index in enumerate(self._fastdivmod_indices): + fdd_name = fdd_names[original_index] + # Get the original FastDivmodDivisor object + original_fdd = getattr(self, fdd_name) + if original_fdd is not None and j < len(values_copy): + # Each FastDivmodDivisor has 1 MLIR value + reconstructed_fdd = new_from_mlir_values(original_fdd, [values_copy[j]]) + setattr(new_params, fdd_name, reconstructed_fdd) + + return new_params + + @dsl_user_op + def get_grid_shape( + self, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, + ) -> Tuple[Integer, Integer, Integer]: + """ + Computes the grid shape based on the problem shape and cluster shape. + + :return: the grid is the CTA numbers that has aligned with cluster shape. + """ + + problem_ceiling_cta_mnl = cute.round_up( + self.problem_shape_ntile_mnl, self._cluster_shape_mnk + ) + + if self.swizzle_size == 1 and self._raster_along_m: + return problem_ceiling_cta_mnl # type: ignore[return-value] + elif self.swizzle_size > 1: + # The swizzled layout spans the cluster counts rounded up to the + # swizzle size; launch every cluster of it (the padding tiles + # fall outside the problem and are skipped by the kernel), or the + # linear indices past the true count would never be decoded. + return ( + *self.cluster_shape_mn, + cute.size(self.problem_layout_ncluster_mnl.shape, loc=loc, ip=ip), + ) + else: + # If swizzling is enabled or raster_along_n, + # we are going to map from a linear idx to tile id manually. + num_problem_cta_count = cute.size(problem_ceiling_cta_mnl, loc=loc, ip=ip) + num_ctas_per_cluster = cute.size(self.cluster_shape_mn, loc=loc, ip=ip) + return ( + *self.cluster_shape_mn, + num_problem_cta_count // num_ctas_per_cluster, + ) + + +# Set explicit signature for Sphinx documentation to avoid issues with @dsl_user_op decorator +ClcDynamicPersistentTileSchedulerParams.__init__.__signature__ = inspect.Signature( # type: ignore[attr-defined] + [ + inspect.Parameter("self", inspect.Parameter.POSITIONAL_OR_KEYWORD), + ] +) + + +class ClcDynamicPersistentTileScheduler: + """A scheduler for dynamic persistent tile execution in CUTLASS/CuTe kernels. + + :ivar params: Tile schedule related params, including cluster shape. + :type params: ClcDynamicPersistentTileSchedulerParams + :ivar cta_id_in_cluster: ID of the CTA within its cluster + :type cta_id_in_cluster: cute.Coord + :ivar _num_tiles_executed: Counter for executed tiles + :type _num_tiles_executed: Int32 + """ + + def __init__( + self, + params: ClcDynamicPersistentTileSchedulerParams, + cta_id_in_cluster: cute.Coord, + num_tiles_executed: Int32, + clc_response_ptr: cute.Pointer, + block_idx: Tuple[Integer, Integer, Integer], + insert_fence: bool = True, + ): + """ + Initializes the ClcDynamicPersistentTileScheduler with the given parameters. + + :param params: Tile schedule related params, including cluster shape. + :type params: ClcDynamicPersistentTileSchedulerParams + :param cta_id_in_cluster: ID of the CTA within its cluster. + :type cta_id_in_cluster: cute.Coord + :param num_tiles_executed: Counter for executed tiles. + :type num_tiles_executed: Int32 + :param clc_response_ptr: Pointer of the clc rsponse. + :type clc_response_ptr: cute.Pointer + :param block_idx: The block index. + :type block_idx: Tuple[Integer, Integer, Integer] + :param insert_fence: Whether to insert a fence to ensure generic-async proxy order. + CLC issue is in async proxy while loading the response from shared memory is in + generic proxy. A cross-proxy fence is needed to ensure producer's next issue + won't race with consumer's current loading. Therefore the scheduler inserts a + fence by default after loading the response. + Developers may insert the fence in pipeline acquire/release functions. In that case, + the fence here can be omitted. + :type insert_fence: bool + """ + self.params = params + self.cta_id_in_cluster = cta_id_in_cluster + self._num_tiles_executed = num_tiles_executed + self._clc_response_ptr = clc_response_ptr + self._block_idx = block_idx + self.insert_fence = insert_fence + + def __extract_mlir_values__(self) -> list[ir.Value]: + values = extract_mlir_values(self.cta_id_in_cluster) + values.extend(extract_mlir_values(self._num_tiles_executed)) + values.extend(extract_mlir_values(self._clc_response_ptr)) + values.extend(extract_mlir_values(self._block_idx)) + return values + + def __new_from_mlir_values__( + self, values: list[ir.Value] + ) -> "ClcDynamicPersistentTileScheduler": + assert len(values) == 8 + new_cta_id_in_cluster = new_from_mlir_values(self.cta_id_in_cluster, values[0:3]) + new_num_tiles_executed = new_from_mlir_values(self._num_tiles_executed, [values[3]]) + new_clc_response_ptr = new_from_mlir_values(self._clc_response_ptr, [values[4]]) + new_block_idx = new_from_mlir_values(self._block_idx, values[5:8]) + return ClcDynamicPersistentTileScheduler( + self.params, + new_cta_id_in_cluster, + new_num_tiles_executed, + new_clc_response_ptr, + new_block_idx, + ) + + @dsl_user_op + @staticmethod + def create( + params: ClcDynamicPersistentTileSchedulerParams, + block_idx: Tuple[Integer, Integer, Integer], + grid_dim: Tuple[Integer, Integer, Integer], + clc_response_ptr: cute.Pointer, + insert_fence: bool = True, + physical_cluster_shape_mn: Optional[Tuple[int, int]] = None, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, + ) -> "ClcDynamicPersistentTileScheduler": + """Initialize the dynamic persistent tile scheduler. + + :param params: Parameters for the persistent + tile scheduler. + :type params: ClcDynamicPersistentTileSchedulerParams + :param block_idx: The 3d block index in the format (bidx, bidy, bidz). + :type block_idx: Tuple[Integer, Integer, Integer] + :param grid_dim: The 3d grid dimensions for kernel launch. + :type grid_dim: Tuple[Integer, Integer, Integer] + + :return: A ClcDynamicPersistentTileScheduler object. + :rtype: ClcDynamicPersistentTileScheduler + """ + params = params + + bidx, bidy, bidz = block_idx + + # CTA id in the cluster. This is a PHYSICAL property: it must + # be taken modulo the cluster the hardware actually formed, which + # under mixed CGA is not the params (preferred) shape. A + # fallback body passes physical_cluster_shape_mn so it can keep + # decoding with the preferred params (see _swizzle_and_rasterize) + # while offsetting by its own shape. + cluster_mn = ( + params.cluster_shape_mn + if physical_cluster_shape_mn is None + else physical_cluster_shape_mn + ) + cta_id_in_cluster = ( + Int32(bidx % cluster_mn[0]), + Int32(bidy % cluster_mn[1]), + Int32(0), + ) + + # Initialize number of tiles executed to zero + num_tiles_executed = Int32(0) + # Initialize clc response pointer + clc_response_ptr = clc_response_ptr + # The block index + block_idx = block_idx + + return ClcDynamicPersistentTileScheduler( + params, + cta_id_in_cluster, + num_tiles_executed, + clc_response_ptr, + block_idx, + insert_fence, + ) + + # called by host + @dsl_user_op + def get_grid_shape( + params: ClcDynamicPersistentTileSchedulerParams, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, + ) -> Tuple[Integer, Integer, Integer]: + """Calculates the grid shape to be launched on GPU using problem shape, + threadblock shape, and active cluster size. + + :param params: Parameters for grid shape calculation. + :type params: ClcDynamicPersistentTileSchedulerParams + + :return: The calculated 3d grid shape. + :rtype: Tuple[Integer, Integer, Integer] + """ + + return params.get_grid_shape(loc=loc, ip=ip) + + @cute.jit + def _swizzle_and_rasterize( + self, + x_idx: Int32, + y_idx: Int32, + z_idx: Int32, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, + ) -> Tuple[Int32, Int32, Int32]: + """Swizzle and rasterize the given coordinates for leader CTA of the cluster. + + x_idx / y_idx are the leader's offsets INSIDE the params cluster + footprint and z_idx selects the footprint. The upstream version + dropped x_idx / y_idx on the two z-form paths, which is sound only + when every launched cluster fills the footprint (then they are + provably 0). MIXED CGA breaks that: a z slice that cannot form the + preferred cluster is tiled by several smaller fallback clusters + whose leaders differ ONLY in x/y, so dropping them aliases them all + onto the same tiles and orphans the rest (silicon: intermittent + 256-row zero blocks and out-of-bounds tile ids). Adding the offsets + back is a no-op for the non-mixed case. + """ + if cutlass.const_expr(self.params.swizzle_size == 1): + if cutlass.const_expr(self.params._raster_along_m): + return x_idx, y_idx, z_idx + else: + # Decode linear index using FastDivmod objects. + # First, get cluster_major using cluster_shape_major_fdd + cluster_minor_batch, cluster_major = divmod( # type: ignore[operator] + z_idx, self.params.cluster_shape_major_fdd + ) + # Then decode cluster_minor_batch to get cluster_minor and batch_l using FastDivmod + batch_l, cluster_minor = divmod( + cluster_minor_batch, self.params.cluster_shape_minor_fdd + ) + cluster_m = cluster_minor + cluster_n = cluster_major + + return ( + cluster_m * self.params.cluster_shape_mn[0] + x_idx, + cluster_n * self.params.cluster_shape_mn[1] + y_idx, + batch_l, + ) + else: + cluster_coord = self.params.problem_layout_ncluster_mnl.get_flat_coord( + z_idx, loc=loc, ip=ip + ) + return ( + cluster_coord[0] * self.params.cluster_shape_mn[0] + x_idx, + cluster_coord[1] * self.params.cluster_shape_mn[1] + y_idx, + cluster_coord[2], + ) + + @dsl_user_op + def work_tile_info_from_clc_response( + self, + result_addr: cute.Pointer, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, + ) -> WorkTileInfo: + """ + Simulates parsing CLC response data in Python. + result_addr: 16-byte response data (simulating shared memory access) + """ + m_idx, n_idx, l_idx, vld = cute.arch.clc_response(result_addr, loc=loc, ip=ip) + + # Cross-proxy fence to ensure generic proxy access are seen by later + # async proxy operations. + if self.insert_fence: + cute.arch.fence_proxy("async.shared", space="cta", loc=loc, ip=ip) + + m_idx, n_idx, l_idx = self._swizzle_and_rasterize(m_idx, n_idx, l_idx, loc=loc, ip=ip) + cta_idx_in_cluster, cta_idy_in_cluster, _ = self.cta_id_in_cluster # type: ignore[misc] + cur_tile_coord = (m_idx + cta_idx_in_cluster, n_idx + cta_idy_in_cluster, l_idx) + + return WorkTileInfo(cur_tile_coord, vld) + + @dsl_user_op + @cute.jit + def get_current_work( + self, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, + ) -> WorkTileInfo: + smem_addr = self._clc_response_ptr + work_tile = self.work_tile_info_from_clc_response(smem_addr, loc=loc, ip=ip) + return work_tile + + @dsl_user_op + @cute.jit + def initial_work_tile_info(self, *, loc: Any = None, ip: Any = None) -> WorkTileInfo: + bidx, bidy, bidz = self._block_idx + # Subtract cta_id_in_cluster from block_idx because swizzle_and_rasterize expects coordinates to be + # those of the leader CTA in the cluster. + cta_idx_in_cluster, cta_idy_in_cluster, _ = self.cta_id_in_cluster # type: ignore[misc] + m_idx = bidx - cta_idx_in_cluster + n_idx = bidy - cta_idy_in_cluster + l_idx = bidz + m_idx, n_idx, l_idx = self._swizzle_and_rasterize(m_idx, n_idx, l_idx, loc=loc, ip=ip) + cur_tile_coord = (m_idx + cta_idx_in_cluster, n_idx + cta_idy_in_cluster, l_idx) + return WorkTileInfo(cur_tile_coord, Boolean(True)) + + @dsl_user_op + @cute.jit + def advance_to_next_work( + self, + mbarrier_addr: cute.Pointer, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, + ) -> None: + # Query new work tile + with cute.arch.elect_one(): + cute.arch.issue_clc_query(mbarrier_addr, self._clc_response_ptr, loc=loc, ip=ip) + self._num_tiles_executed += Int32(1) + + @property + @cute.jit + def num_tiles_executed(self) -> Int32: + return self._num_tiles_executed diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/kernel.py new file mode 100644 index 000000000000..5e313ef413a7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/qkv_fusion/kernel.py @@ -0,0 +1,1996 @@ +# Copyright (c) 2025 - 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: BSD-3-Clause + +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: + +# 1. Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. + +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. + +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. + +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# ruff: noqa: E501,E731,E741 + +"""SM107 persistent MXFP8 QKV GEMM fused with the MiniMax-M3 QKV-index producer. + +Per output row m (token) and 128-wide head slot h of the packed +[q (Hq) | K (Hkv) | V (Hkv) | index-q (Hi) | index-K (1)] projection:: + + x = bf16_rn(alpha * (act[m, :] @ weight[h*128:(h+1)*128, :].T)) + # MXFP8 block-scaled GEMM (E4M3 operands, UE8M0 K32 + # scales), FP32 accumulation; x is re-read as fp32 + V : no norm, no rope + otherwise: RMSNorm with the per-slot-type bf16 weight w (1 + w + scaling) over the 128 dims, then NeoX RoPE on dims 0..63 + with (cos, sin) = rotary_cos_sin[position_ids[m]] + q -> E4M3 q_out[m, h, :] + index-q -> E4M3 index_q_out[m, h - first index-q slot, :] + index-K -> E4M3 paged index-K cache row out_cache_loc[m] + K, V -> NVFP4 (16-element E4M3 block scales, second-level scale + kv_quant_scale[1] / [2]) into the paged HND K/V caches + +Every floating-point step after the GEMM is the op sequence that +``minimaxM3Nvfp4QKVIndexerNormRopeKVInsert{,Rows}Kernel`` compiles to, so +the outputs are bitwise those of running that producer on this kernel's +own bf16 GEMM output (``debug_gemm_out=True`` writes that bf16 tensor +instead of the producer outputs): + +* sum of squares per 4-dim lane group l: fma(x0, x0, x1*x1) + + fma(x2, x2, x3*x3), then the xor-butterfly 16, 8, 4, 2, 1 over the 32 + lane partials, replayed in-thread; +* inv_rms = rsqrt.approx.f32(fma(sum, 1/128, eps)); +* e = x * (inv_rms * (1 + w)); +* rope: out = fma(+-paired, sin, e * cos), paired = e[d ^ 32], negated for + d < 32; +* E4M3 via cvt.rn.satfinite.e4m3x2.f32; NVFP4 via the bf16-rounded + values, the 16-element abs-max, scale = sl * (max * 0x3E2AAAAB), its + E4M3 encoding, output scale rcp.approx.ftz(e4m3(scale) * + rcp.approx.ftz(sl)) (0 for an all-zero group) and + cvt.rn.satfinite.e2m1x2.f32. + +Schedule: a persistent warp-specialized SM107 block-scaled GEMM (CLC tile +scheduler; warps 0-3 epilogue, 4 MMA, 5 TMA, 6 scheduler, 7 idle). One +work tile is (``mma_m`` rows, ``heads_per_tile`` head slots of 128 +columns); ``mma_m=256`` runs the 2-CTA MMA with each CTA of the pair +owning 128 rows. A tile wider than one MMA (N > ``mma_n``) issues its N +sets back to back per k-block with A held in the operand collector. The +accumulator has as many TMEM stages as fit (``num_acc_stage``) so the +epilogue of one tile overlaps the mainloop of the next. ``epi_wgs`` +epilogue warpgroups split a tile's heads between them, so with a single +accumulator stage the MMA of the next tile waits for only one warpgroup's +share of producer math. Each epilogue thread owns one tile row (one +token) and whole 128-column heads, so the row reduction needs no +shuffles. Rows >= M and head slots past the last are +never stored; cache stores are skipped for out_cache_loc < 0 or +page >= num_pages. + +Defaults (4 head slots per tile, 2-CTA M256, A reused across two N256 +MMAs, one epilogue warpgroup) are the fastest measured on SM107 at +M=8192..16384: SMEM operand reads per SM, not L2, bound the mainloop, so +wider tiles win; a second epilogue warpgroup caps the epilogue at 208 +registers and runs slower. +""" + +from __future__ import annotations + +import os +from typing import NamedTuple, Tuple + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.pipeline as pipeline +import cutlass.utils as utils +import cutlass.utils.blackwell_helpers as sm100_utils +import cutlass.utils.blockscaled_layout as blockscaled_utils +import cutlass.utils.rubin_helpers as sm107_utils +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.dialects import nvvm +from cutlass._mlir.extras import types as _T +from cutlass.cute.nvgpu import OperandMajorMode, cpasync, tcgen05 +from cutlass.cute.nvgpu.tcgen05.mma import CollectorOp +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.pipeline import pipeline_init_arrive, pipeline_init_wait +from cutlass.utils.gemm import sm100 as gemm_sm100 + +from .clc_tile_scheduler import ( + ClcDynamicPersistentTileScheduler, + ClcDynamicPersistentTileSchedulerParams, +) + +HEAD_DIM = 128 +ROTARY_DIM = 64 +PAGE_SIZE = 128 +# float32(rcp.approx.ftz(6.0f)) as the producer's compiled FMUL immediate. +RCP_SIX_BITS = 0x3E2AAAAB + + +# --------------------------------------------------------------------------- +# Inline-PTX helpers. Every float op is an explicitly rounded PTX op so that +# neither LLVM nor ptxas contracts or reassociates it. +# --------------------------------------------------------------------------- + + +def _asm(res_type, args, text, constraints, side_effects, loc, ip): + return _llvm.inline_asm( + res_type, + [a.ir_value(loc=loc, ip=ip) for a in args], + text, + constraints, + has_side_effects=side_effects, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _mul_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _rsqrt_approx(x, *, loc=None, ip=None): + """rsqrtf without fast-math: rsqrt.approx.f32 (denormal-preserving).""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(x)], + "rsqrt.approx.f32 $0, $1;", + "=f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _rcp_approx_ftz(x, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(x)], + "rcp.approx.ftz.f32 $0, $1;", + "=f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _absmax(acc, v, *, loc=None, ip=None): + """fmaxf(acc, fabsf(v)) (max.f32 ignores a NaN operand like fmaxf).""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(acc), cutlass.Float32(v)], + "{\n\t.reg .f32 a;\n\tabs.f32 a, $2;\n\tmax.f32 $0, $1, a;\n\t}", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _bf16x2_rn(lo, hi, *, loc=None, ip=None): + """bf16_rn(lo) | bf16_rn(hi) << 16.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "cvt.rn.bf16x2.f32 $0, $1, $2;", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3x4(v0, v1, v2, v3, *, loc=None, ip=None): + """Four f32 to E4M3 (RN, satfinite), v0 in the low byte.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(v) for v in (v0, v1, v2, v3)], + "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}", + "=r,f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3_byte(x, *, loc=None, ip=None): + """__nv_fp8_e4m3(x).__x: cvt.rn.satfinite.e4m3x2.f32 of (0, x), low byte.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n\t.reg .b16 h;\n\t.reg .f32 z;\n\tmov.f32 z, 0f00000000;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 h, z, $1;\n\tcvt.u32.u16 $0, h;\n\t" + "and.b32 $0, $0, 255;\n\t}", + "=r,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3_byte_to_f32(b, *, loc=None, ip=None): + """static_cast(__nv_fp8_e4m3) of the byte b (exact).""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(b)], + "{\n\t.reg .b16 h, lo, hi;\n\t.reg .b32 r;\n\tcvt.u16.u32 h, $1;\n\t" + "cvt.rn.f16x2.e4m3x2 r, h;\n\tmov.b32 {lo, hi}, r;\n\tcvt.f32.f16 $0, lo;\n\t}", + "=f,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e2m1x8(v, *, loc=None, ip=None): + """fp32_vec_to_e2m1 of eight f32: element 2i in the low nibble of byte i.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x) for x in v], + "{\n\t.reg .b8 byte0, byte1, byte2, byte3;\n\t" + "cvt.rn.satfinite.e2m1x2.f32 byte0, $2, $1;\n\t" + "cvt.rn.satfinite.e2m1x2.f32 byte1, $4, $3;\n\t" + "cvt.rn.satfinite.e2m1x2.f32 byte2, $6, $5;\n\t" + "cvt.rn.satfinite.e2m1x2.f32 byte3, $8, $7;\n\t" + "mov.b32 $0, {byte0, byte1, byte2, byte3};\n\t}", + "=r,f,f,f,f,f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ldg_v4_f32(addr, *, loc=None, ip=None): + out = _asm( + _llvm.StructType.get_literal([_T.f32()] * 4), + [cutlass.Int64(addr)], + "ld.global.nc.v4.f32 {$0, $1, $2, $3}, [$4];", + "=f,=f,=f,=f,l", + False, + loc, + ip, + ) + return [ + cutlass.Float32(_llvm.extractvalue(_T.f32(), out, [i], loc=loc, ip=ip)) for i in range(4) + ] + + +@dsl_user_op +def _ldg_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.global.nc.s32 $0, [$1];", + "=r,l", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ldg_u16(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "{\n\t.reg .b16 h;\n\tld.global.nc.b16 h, [$1];\n\tcvt.u32.u16 $0, h;\n\t}", + "=r,l", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _stg_v4(addr, w0, w1, w2, w3, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr)] + [cutlass.Int32(w) for w in (w0, w1, w2, w3)], + "st.global.v4.b32 [$0], {$1, $2, $3, $4};", + "l,r,r,r,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _stg_v2(addr, w0, w1, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(w0), cutlass.Int32(w1)], + "st.global.v2.b32 [$0], {$1, $2};", + "l,r,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _stg_u8(addr, w, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(w)], + "st.global.u8 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _sts_f32(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int32(addr), cutlass.Float32(v)], + "st.shared.f32 [$0], $1;", + "r,f", + True, + loc, + ip, + ) + + +@dsl_user_op +def _lds_v4_f32(addr, *, loc=None, ip=None): + out = _asm( + _llvm.StructType.get_literal([_T.f32()] * 4), + [cutlass.Int32(addr)], + "ld.shared.v4.f32 {$0, $1, $2, $3}, [$4];", + "=f,=f,=f,=f,r", + True, + loc, + ip, + ) + return [ + cutlass.Float32(_llvm.extractvalue(_T.f32(), out, [i], loc=loc, ip=ip)) for i in range(4) + ] + + +def _lo_f32(w): + """The bf16 in the low half of w, as fp32.""" + return (w << cutlass.Int32(16)).bitcast(cutlass.Float32) + + +def _hi_f32(w): + """The bf16 in the high half of w, as fp32.""" + return (w & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + + +class S2TCopyBundle(NamedTuple): + """Tiled copy + partitioned tensors for one SMEM-to-TMEM SF copy.""" + + tiled_copy: cute.TiledCopy + sSF_compact: cute.Tensor + tSF_compact: cute.Tensor + + +class M3QkvGemmNormRopeKvInsert: + """Persistent SM107 MXFP8 QKV GEMM with the M3 producer as its epilogue.""" + + def __init__( + self, + *, + num_heads_q: int = 32, + num_heads_kv: int = 2, + num_heads_index: int = 2, + heads_per_tile: int = 4, + mma_m: int = 256, + mma_n: int | None = None, + num_acc_stage: int | None = None, + cluster_shape_mn: Tuple[int, int] | None = None, + swizzle_size: int = 1, + raster_along_m: bool = True, + debug_gemm_out: bool = False, + enable_pdl: bool = True, + epi_wgs: int = 1, + ): + if mma_m not in (128, 256): + raise ValueError("mma_m must be 128 (1-CTA) or 256 (2-CTA pair)") + if heads_per_tile not in (1, 2, 3, 4): + raise ValueError("heads_per_tile must be 1..4") + self.num_heads_q = int(num_heads_q) + self.num_heads_kv = int(num_heads_kv) + self.num_heads_index = int(num_heads_index) + self.k_begin = self.num_heads_q + self.v_begin = self.k_begin + self.num_heads_kv + self.iq_begin = self.v_begin + self.num_heads_kv + self.ik_slot = self.iq_begin + self.num_heads_index + self.num_slots = self.ik_slot + 1 + self.n_total = self.num_slots * HEAD_DIM + + self.heads_per_tile = int(heads_per_tile) + self.tile_n = HEAD_DIM * self.heads_per_tile + inst_n = min(self.tile_n, 256) if mma_n is None else int(mma_n) + if inst_n not in (128, 256) or self.tile_n % inst_n: + raise ValueError("mma_n must be 128 or 256 and divide the tile N") + self.n_sets = self.tile_n // inst_n + self.mma_inst_shape = (mma_m, inst_n, 64) + self.mma_tiler = (mma_m, self.tile_n, 128) + self.use_2cta_instrs = mma_m == 256 + self.cta_group = tcgen05.CtaGroup.TWO if self.use_2cta_instrs else tcgen05.CtaGroup.ONE + self.atom_thr_size = 2 if self.use_2cta_instrs else 1 + + if cluster_shape_mn is None: + cluster_shape_mn = (self.atom_thr_size, 1) + self.cluster_shape_mn = tuple(cluster_shape_mn) + cm, cn = self.cluster_shape_mn + if cm < 1 or cn < 1 or (cm & (cm - 1)) or (cn & (cn - 1)) or cm * cn > 16: + raise ValueError("cluster dims must be powers of two, <= 16 CTAs") + if cm % self.atom_thr_size: + raise ValueError("a 2-CTA MMA needs an even cluster M") + self.swizzle_size = int(swizzle_size) + self.raster_along_m = bool(raster_along_m) + self.debug_gemm_out = bool(debug_gemm_out) + self.enable_pdl = bool(enable_pdl) + if epi_wgs not in (1, 2) or self.heads_per_tile % epi_wgs: + raise ValueError("epi_wgs must be 1 or 2 and divide heads_per_tile") + self.epi_wgs = int(epi_wgs) + self.heads_per_wg = self.heads_per_tile // self.epi_wgs + + self.acc_dtype = cutlass.Float32 + self.sf_vec_size = 32 + self.occupancy = 1 + # Warpgroup g of the epilogue is warps 4g..4g+3 (TMEM lane quadrant + # = warp % 4); the MMA / TMA / scheduler / idle warps follow. + num_epi_warps = 4 * self.epi_wgs + self.epilog_warp_id = tuple(range(num_epi_warps)) + self.mma_warp_id = num_epi_warps + self.tma_warp_id = num_epi_warps + 1 + self.sched_warp_id = num_epi_warps + 2 + self.pad_warp_id = num_epi_warps + 3 + self.threads_per_warp = 32 + self.threads_per_cta = 32 * (num_epi_warps + 4) + self.epilog_reg_count = 256 if self.epi_wgs == 1 else 208 + self.mainloop_reg_count = 88 + self.num_clc_stage = 1 + self.num_clc_response_bytes = 16 + self.epilog_sync_barrier = pipeline.NamedBarrier( + barrier_id=1, num_threads=32 * len(self.epilog_warp_id) + ) + self.tmem_alloc_barrier = pipeline.NamedBarrier( + barrier_id=2, num_threads=32 * (1 + len(self.epilog_warp_id)) + ) + self.arch = "sm_107" + self.smem_capacity = utils.get_smem_capacity_in_bytes(self.arch) + self.num_tmem_alloc_cols = cute.arch.get_max_tmem_alloc_cols(self.arch) + self.requested_acc_stage = num_acc_stage + self.epi_chunk = 32 + self.chunks_per_head = HEAD_DIM // self.epi_chunk + + # ------------------------------------------------------------------ + def _make_tiled_mmas(self): + def make(a_op, b_op): + mma = sm107_utils.make_blockscaled_trivial_tiled_mma( + self.a_dtype, + self.b_dtype, + self.a_major_mode, + self.b_major_mode, + self.sf_dtype, + self.sf_vec_size, + self.cta_group, + self.mma_inst_shape, + a_collector_op=a_op, + b_collector_op=b_op, + atom_layout_mnk=(1, 1, 1), + permutation_mnk=(1, 1, 1), + ) + mma.set(tcgen05.Field.NEGATE_A, False) + mma.set(tcgen05.Field.NEGATE_B, False) + return mma + + # A stays in the collector across the tile's N sets of one k-block. + if self.n_sets == 1: + tiled_mma_first = make(CollectorOp.DISCARD, CollectorOp.DISCARD) + tiled_mma_mid = tiled_mma_first + tiled_mma_last = tiled_mma_first + else: + tiled_mma_first = make(CollectorOp.FILL, CollectorOp.DISCARD) + tiled_mma_mid = make(CollectorOp.USE, CollectorOp.DISCARD) + tiled_mma_last = make(CollectorOp.LASTUSE, CollectorOp.DISCARD) + tiled_mma_sfb = sm107_utils.make_blockscaled_trivial_tiled_mma( + self.a_dtype, + self.b_dtype, + self.a_major_mode, + self.b_major_mode, + self.sf_dtype, + self.sf_vec_size, + tcgen05.CtaGroup.ONE, + self.mma_inst_shape_sfb, + a_collector_op=CollectorOp.DISCARD, + b_collector_op=CollectorOp.DISCARD, + ) + tiled_mma_sfb.set(tcgen05.Field.NEGATE_A, False) + tiled_mma_sfb.set(tcgen05.Field.NEGATE_B, False) + return (tiled_mma_first, tiled_mma_mid, tiled_mma_last), tiled_mma_sfb + + def _cluster_layouts(self, tiled_mma, tiled_mma_sfb): + vmnk = cute.tiled_divide( + cute.make_layout((*self.cluster_shape_mn, 1)), (tiled_mma.thr_id.shape,) + ) + sfb_vmnk = cute.tiled_divide( + cute.make_layout((*self.cluster_shape_mn, 1)), + (tiled_mma_sfb.thr_id.shape,), + ) + is_a_mcast = cute.size(vmnk.shape[2]) > 1 + is_b_mcast = cute.size(vmnk.shape[1]) > 1 + return vmnk, sfb_vmnk, is_a_mcast, is_b_mcast + + def _setup_attributes(self): + self.mma_inst_shape_sfb = ( + self.mma_inst_shape[0] // self.atom_thr_size, + cute.round_up(self.mma_inst_shape[1], 128), + self.mma_inst_shape[2], + ) + self.mma_tiler_sfb = ( + self.mma_inst_shape_sfb[0], + self.mma_tiler[1], + self.mma_tiler[2], + ) + (tiled_mma, _, _), _ = self._make_tiled_mmas() + self.cta_tile_shape_mnk = ( + self.mma_tiler[0] // cute.size(tiled_mma.thr_id.shape), + self.mma_tiler[1], + self.mma_tiler[2], + ) + a_one = sm100_utils.make_smem_layout_a(tiled_mma, self.mma_tiler, self.a_dtype, 1) + b_one = sm100_utils.make_smem_layout_b(tiled_mma, self.mma_tiler, self.b_dtype, 1) + sfa_one = blockscaled_utils.make_smem_layout_sfa( + tiled_mma, self.mma_tiler, self.sf_vec_size, 1 + ) + sfb_one = blockscaled_utils.make_smem_layout_sfb( + tiled_mma, self.mma_tiler, self.sf_vec_size, 1 + ) + stage_bytes = ( + cute.size_in_bytes(self.a_dtype, a_one) + + cute.size_in_bytes(self.b_dtype, b_one) + + cute.size_in_bytes(self.sf_dtype, sfa_one) + + cute.size_in_bytes(self.sf_dtype, sfb_one) + ) + # barriers, CLC response, (1 + w) table (4 x 128 fp32) and slack + helpers_bytes = 1024 + 4 * HEAD_DIM * 4 + 1024 + self.num_ab_stage = (self.smem_capacity // self.occupancy - helpers_bytes) // stage_bytes + + self.a_smem_layout_staged = sm100_utils.make_smem_layout_a( + tiled_mma, self.mma_tiler, self.a_dtype, self.num_ab_stage + ) + self.b_smem_layout_staged = sm100_utils.make_smem_layout_b( + tiled_mma, self.mma_tiler, self.b_dtype, self.num_ab_stage + ) + self.sfa_smem_layout_staged = blockscaled_utils.make_smem_layout_sfa( + tiled_mma, self.mma_tiler, self.sf_vec_size, self.num_ab_stage + ) + self.sfb_smem_layout_staged = blockscaled_utils.make_smem_layout_sfb( + tiled_mma, self.mma_tiler, self.sf_vec_size, self.num_ab_stage + ) + self.tCtSFA_layout = blockscaled_utils.make_tmem_layout_sfa( + tiled_mma, + self.mma_tiler, + self.sf_vec_size, + cute.slice_(self.sfa_smem_layout_staged, (None, None, None, 0)), + ) + self.tCtSFB_layout = blockscaled_utils.make_tmem_layout_sfb( + tiled_mma, + self.mma_tiler, + self.sf_vec_size, + cute.slice_(self.sfb_smem_layout_staged, (None, None, None, 0)), + ) + self.num_sfa_tmem_cols = ( + cute.cosize(cute.recast_layout(32, self.sf_dtype.width, self.tCtSFA_layout)) + & 0x0000FFFF + ) + self.num_sfb_tmem_cols = ( + cute.cosize(cute.recast_layout(32, self.sf_dtype.width, self.tCtSFB_layout)) + & 0x0000FFFF + ) + sf_cols = self.num_sfa_tmem_cols + self.num_sfb_tmem_cols + max_acc_stage = (self.num_tmem_alloc_cols - sf_cols) // self.tile_n + if self.requested_acc_stage is None: + self.num_acc_stage = max(1, min(4, max_acc_stage)) + else: + self.num_acc_stage = int(self.requested_acc_stage) + if self.num_acc_stage < 1 or self.num_acc_stage > max_acc_stage: + raise ValueError( + f"num_acc_stage={self.num_acc_stage} does not fit TMEM " + f"(max {max_acc_stage} for tile_n={self.tile_n})" + ) + self.num_accumulator_tmem_cols = self.tile_n * self.num_acc_stage + if os.environ.get("CUTE_DSL_DUMP_STAGES"): + print( + f"[stages] num_ab_stage={self.num_ab_stage} stage_bytes={stage_bytes} " + f"num_acc_stage={self.num_acc_stage} sf_cols={sf_cols}", + flush=True, + ) + + # ------------------------------------------------------------------ + @cute.jit + def __call__( + self, + activation: cute.Tensor, # (M, K) e4m3 + activation_sf: cute.Tensor, # 1-D swizzled UE8M0 K32 + weight: cute.Tensor, # (N, K) e4m3 + weight_sf: cute.Tensor, # 1-D swizzled UE8M0 K32 + alpha: cute.Tensor, # (1,) f32 + q_out: cute.Tensor, # (M * Hq * 128,) u8 (E4M3 bytes) + index_q_out: cute.Tensor, # (M * Hi * 128,) u8 + kv_data_cache: cute.Tensor, # u8, flat + kv_scale_cache: cute.Tensor, # u8, flat + index_k_cache: cute.Tensor, # u8 (E4M3 bytes), flat + out_cache_loc: cute.Tensor, # (>= M,) i32 + kv_quant_scale: cute.Tensor, # (3,) f32 + rotary_cos_sin: cute.Tensor, # (max_pos * 64,) f32 + position_ids: cute.Tensor, # (M,) i32 + norm_weights: cute.Tensor, # (4 * 128,) bf16: q, K, index-q, index-K + gemm_out: cute.Tensor, # (M * N,) bf16, debug only + data_page_stride: cutlass.Int64, + data_plane_stride: cutlass.Int64, + data_head_stride: cutlass.Int64, + data_token_stride: cutlass.Int64, + scale_page_stride: cutlass.Int64, + scale_plane_stride: cutlass.Int64, + scale_head_stride: cutlass.Int64, + index_page_stride: cutlass.Int64, + index_token_stride: cutlass.Int64, + num_pages: cutlass.Int64, + eps: cutlass.Float32, + stream: cuda_driver.CUstream, + ): + self.a_dtype = activation.element_type + self.b_dtype = weight.element_type + self.sf_dtype = activation_sf.element_type + self.a_major_mode = OperandMajorMode.K + self.b_major_mode = OperandMajorMode.K + + m = activation.shape[0] + k = activation.shape[1] + n = weight.shape[0] + + self._setup_attributes() + tiled_mmas, tiled_mma_sfb = self._make_tiled_mmas() + tiled_mma = tiled_mmas[0] + + a_tensor = cute.make_tensor( + activation.iterator, cute.make_ordered_layout((m, k, 1), order=(1, 0, 2)) + ) + b_tensor = cute.make_tensor( + weight.iterator, cute.make_ordered_layout((n, k, 1), order=(1, 0, 2)) + ) + sfa_tensor = cute.make_tensor( + activation_sf.iterator, + blockscaled_utils.tile_atom_to_shape_SF(a_tensor.shape, self.sf_vec_size), + ) + sfb_tensor = cute.make_tensor( + weight_sf.iterator, + blockscaled_utils.tile_atom_to_shape_SF(b_tensor.shape, self.sf_vec_size), + ) + tma_atoms, tma_tensors = self._make_tma_atoms( + tiled_mma, tiled_mma_sfb, a_tensor, b_tensor, sfa_tensor, sfb_tensor + ) + + a_smem_layout = cute.slice_(self.a_smem_layout_staged, (None, None, None, 0)) + b_smem_layout = cute.slice_(self.b_smem_layout_staged, (None, None, None, 0)) + sfa_smem_layout = cute.slice_(self.sfa_smem_layout_staged, (None, None, None, 0)) + sfb_smem_layout = cute.slice_(self.sfb_smem_layout_staged, (None, None, None, 0)) + self.num_tma_load_bytes = ( + cute.size_in_bytes(self.a_dtype, a_smem_layout) + + cute.size_in_bytes(self.sf_dtype, sfa_smem_layout) + + cute.size_in_bytes(self.b_dtype, b_smem_layout) + + cute.size_in_bytes(self.sf_dtype, sfb_smem_layout) + ) * self.atom_thr_size + + num_ctas_mnl = ( + cute.ceil_div(m, self.cta_tile_shape_mnk[0]), + (self.num_slots + self.heads_per_tile - 1) // self.heads_per_tile, + 1, + ) + tile_sched_params = ClcDynamicPersistentTileSchedulerParams( + num_ctas_mnl, + (*self.cluster_shape_mn, 1), + self.swizzle_size, + self.raster_along_m, + ) + grid = ClcDynamicPersistentTileScheduler.get_grid_shape(tile_sched_params) + + self.buffer_align_bytes = 1024 + + @cute.struct + class SharedStorage: + ab_full_mbar_ptr: cute.struct.MemRange[cutlass.Int64, self.num_ab_stage] + ab_empty_mbar_ptr: cute.struct.MemRange[cutlass.Int64, self.num_ab_stage] + acc_full_mbar_ptr: cute.struct.MemRange[cutlass.Int64, self.num_acc_stage] + acc_empty_mbar_ptr: cute.struct.MemRange[cutlass.Int64, self.num_acc_stage] + clc_mbar_ptr: cute.struct.MemRange[cutlass.Int64, self.num_clc_stage * 2] + clc_response: cute.struct.Align[ + cute.struct.MemRange[cutlass.Int32, self.num_clc_stage * 4], 16 + ] + sOnePlusW: cute.struct.Align[cute.struct.MemRange[cutlass.Float32, 4 * HEAD_DIM], 16] + tmem_dealloc_mbar: cutlass.Int64 + tmem_holding_buf: cutlass.Int32 + sA: cute.struct.Align[ + cute.struct.MemRange[self.a_dtype, cute.cosize(self.a_smem_layout_staged.outer)], + self.buffer_align_bytes, + ] + sB: cute.struct.Align[ + cute.struct.MemRange[self.b_dtype, cute.cosize(self.b_smem_layout_staged.outer)], + self.buffer_align_bytes, + ] + sSFA: cute.struct.Align[ + cute.struct.MemRange[self.sf_dtype, cute.cosize(self.sfa_smem_layout_staged)], + self.buffer_align_bytes, + ] + sSFB: cute.struct.Align[ + cute.struct.MemRange[self.sf_dtype, cute.cosize(self.sfb_smem_layout_staged)], + self.buffer_align_bytes, + ] + + self.shared_storage = SharedStorage + + self.kernel( + tiled_mmas[0], + tiled_mmas[1], + tiled_mmas[2], + tiled_mma_sfb, + tma_atoms, + tma_tensors, + alpha, + q_out, + index_q_out, + kv_data_cache, + kv_scale_cache, + index_k_cache, + out_cache_loc, + kv_quant_scale, + rotary_cos_sin, + position_ids, + norm_weights, + gemm_out, + ( + data_page_stride, + data_plane_stride, + data_head_stride, + data_token_stride, + scale_page_stride, + scale_plane_stride, + scale_head_stride, + index_page_stride, + index_token_stride, + ), + num_pages, + tile_sched_params, + (m, n), + eps, + ).launch( + grid=grid, + block=[self.threads_per_cta, 1, 1], + cluster=(*self.cluster_shape_mn, 1), + stream=stream, + min_blocks_per_mp=1, + smem_merge_branch_allocs=True, + ) + return + + @cute.jit + def _make_tma_atoms(self, tiled_mma, tiled_mma_sfb, a_tensor, b_tensor, sfa_tensor, sfb_tensor): + vmnk, sfb_vmnk, _, _ = self._cluster_layouts(tiled_mma, tiled_mma_sfb) + a_smem_layout = cute.slice_(self.a_smem_layout_staged, (None, None, None, 0)) + b_smem_layout = cute.slice_(self.b_smem_layout_staged, (None, None, None, 0)) + sfa_smem_layout = cute.slice_(self.sfa_smem_layout_staged, (None, None, None, 0)) + sfb_smem_layout = cute.slice_(self.sfb_smem_layout_staged, (None, None, None, 0)) + tma_atom_a, tma_tensor_a = cute.nvgpu.make_tiled_tma_atom_A( + sm100_utils.cluster_shape_to_tma_atom_A(self.cluster_shape_mn, tiled_mma.thr_id), + a_tensor, + a_smem_layout, + self.mma_tiler, + tiled_mma, + vmnk.shape, + ) + tma_atom_b, tma_tensor_b = cute.nvgpu.make_tiled_tma_atom_B( + sm100_utils.cluster_shape_to_tma_atom_B(self.cluster_shape_mn, tiled_mma.thr_id), + b_tensor, + b_smem_layout, + self.mma_tiler, + tiled_mma, + vmnk.shape, + ) + tma_atom_sfa, tma_tensor_sfa = cute.nvgpu.make_tiled_tma_atom_A( + sm100_utils.cluster_shape_to_tma_atom_A(self.cluster_shape_mn, tiled_mma.thr_id), + sfa_tensor, + sfa_smem_layout, + self.mma_tiler, + tiled_mma, + vmnk.shape, + internal_type=cutlass.Int16, + ) + tma_atom_sfb, tma_tensor_sfb = cute.nvgpu.make_tiled_tma_atom_B( + sm100_utils.cluster_shape_to_tma_atom_SFB(self.cluster_shape_mn, tiled_mma.thr_id), + sfb_tensor, + sfb_smem_layout, + self.mma_tiler_sfb, + tiled_mma_sfb, + sfb_vmnk.shape, + internal_type=cutlass.Int16, + ) + return ( + (tma_atom_a, tma_atom_b, tma_atom_sfa, tma_atom_sfb), + (tma_tensor_a, tma_tensor_b, tma_tensor_sfa, tma_tensor_sfb), + ) + + # ------------------------------------------------------------------ + @cute.kernel + def kernel( + self, + tiled_mma: cute.TiledMma, + tiled_mma_mid: cute.TiledMma, + tiled_mma_last: cute.TiledMma, + tiled_mma_sfb: cute.TiledMma, + tma_atoms: Tuple, + tma_tensors: Tuple, + alpha: cute.Tensor, + q_out: cute.Tensor, + index_q_out: cute.Tensor, + kv_data_cache: cute.Tensor, + kv_scale_cache: cute.Tensor, + index_k_cache: cute.Tensor, + out_cache_loc: cute.Tensor, + kv_quant_scale: cute.Tensor, + rotary_cos_sin: cute.Tensor, + position_ids: cute.Tensor, + norm_weights: cute.Tensor, + gemm_out: cute.Tensor, + cache_strides: Tuple, + num_pages: cutlass.Int64, + tile_sched_params: ClcDynamicPersistentTileSchedulerParams, + problem_mn: Tuple, + eps: cutlass.Float32, + ): + self._setup_attributes() + tma_atom_a, tma_atom_b, tma_atom_sfa, tma_atom_sfb = tma_atoms + mA_mkl, mB_nkl, mSFA_mkl, mSFB_nkl = tma_tensors + m, n = problem_mn + m_i32 = cutlass.Int32(m) + warp_idx = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + + ( + cluster_layout_vmnk, + cluster_layout_sfb_vmnk, + is_a_mcast, + is_b_mcast, + ) = self._cluster_layouts(tiled_mma, tiled_mma_sfb) + + if warp_idx == self.tma_warp_id: + cpasync.prefetch_descriptor(tma_atom_a) + cpasync.prefetch_descriptor(tma_atom_b) + cpasync.prefetch_descriptor(tma_atom_sfa) + cpasync.prefetch_descriptor(tma_atom_sfb) + + bidx, bidy, bidz = cute.arch.block_idx() + mma_tile_coord_v = bidx % cute.size(tiled_mma.thr_id.shape) + is_leader_cta = mma_tile_coord_v == 0 + cta_rank_in_cluster = cute.arch.make_warp_uniform(cute.arch.block_idx_in_cluster()) + block_in_cluster_coord_vmnk = cluster_layout_vmnk.get_flat_coord(cta_rank_in_cluster) + block_in_cluster_coord_sfb_vmnk = cluster_layout_sfb_vmnk.get_flat_coord( + cta_rank_in_cluster + ) + tidx, _, _ = cute.arch.thread_idx() + + smem = utils.SmemAllocator() + storage = smem.allocate(self.shared_storage) + + num_mcast_ctas_a = cute.size(cluster_layout_vmnk.shape[2]) + num_mcast_ctas_b = cute.size(cluster_layout_vmnk.shape[1]) + num_tma_producers = num_mcast_ctas_a + num_mcast_ctas_b - 1 + ab_pipeline = pipeline.PipelineTmaUmma.create( + barrier_storage=storage.ab_full_mbar_ptr.data_ptr(), + num_stages=self.num_ab_stage, + producer_group=pipeline.CooperativeGroup(pipeline.Agent.Thread), + consumer_group=pipeline.CooperativeGroup(pipeline.Agent.Thread, num_tma_producers), + tx_count=self.num_tma_load_bytes, + cta_layout_vmnk=cluster_layout_vmnk, + defer_sync=True, + ) + acc_pipeline = pipeline.PipelineUmmaAsync.create( + barrier_storage=storage.acc_full_mbar_ptr.data_ptr(), + num_stages=self.num_acc_stage, + producer_group=pipeline.CooperativeGroup(pipeline.Agent.Thread), + consumer_group=pipeline.CooperativeGroup( + pipeline.Agent.Thread, 32 * len(self.epilog_warp_id) * self.atom_thr_size + ), + cta_layout_vmnk=cluster_layout_vmnk, + defer_sync=True, + ) + cluster_size = cute.size(cluster_layout_vmnk.shape) + num_clc_consumer_threads = 32 * (1 + cluster_size * (1 + len(self.epilog_warp_id) + 1)) + clc_pipeline = pipeline.PipelineClcFetchAsync.create( + barrier_storage=storage.clc_mbar_ptr.data_ptr(), + num_stages=self.num_clc_stage, + producer_group=pipeline.CooperativeGroup(pipeline.Agent.Thread), + consumer_group=pipeline.CooperativeGroup( + pipeline.Agent.Thread, num_clc_consumer_threads + ), + tx_count=self.num_clc_response_bytes, + cta_layout_vmnk=cluster_layout_vmnk, + defer_sync=True, + ) + tmem = utils.TmemAllocator( + storage.tmem_holding_buf.ptr, + barrier_for_retrieve=self.tmem_alloc_barrier, + allocator_warp_id=self.epilog_warp_id[0], + is_two_cta=self.use_2cta_instrs, + two_cta_tmem_dealloc_mbar_ptr=storage.tmem_dealloc_mbar.ptr, + arch=self.arch, + ) + pipeline_init_arrive(cluster_shape_mn=self.cluster_shape_mn, is_relaxed=True) + + sA = storage.sA.get_tensor( + self.a_smem_layout_staged.outer, swizzle=self.a_smem_layout_staged.inner + ) + sB = storage.sB.get_tensor( + self.b_smem_layout_staged.outer, swizzle=self.b_smem_layout_staged.inner + ) + sSFA = storage.sSFA.get_tensor(self.sfa_smem_layout_staged) + sSFB = storage.sSFB.get_tensor(self.sfb_smem_layout_staged) + + a_full_mcast_mask = None + b_full_mcast_mask = None + sfa_full_mcast_mask = None + sfb_full_mcast_mask = None + if cutlass.const_expr(is_a_mcast or is_b_mcast or self.use_2cta_instrs): + a_full_mcast_mask = cpasync.create_tma_multicast_mask( + cluster_layout_vmnk, block_in_cluster_coord_vmnk, mcast_mode=2 + ) + b_full_mcast_mask = cpasync.create_tma_multicast_mask( + cluster_layout_vmnk, block_in_cluster_coord_vmnk, mcast_mode=1 + ) + sfa_full_mcast_mask = a_full_mcast_mask + sfb_full_mcast_mask = cpasync.create_tma_multicast_mask( + cluster_layout_sfb_vmnk, block_in_cluster_coord_sfb_vmnk, mcast_mode=1 + ) + + gA_mkl = cute.local_tile( + mA_mkl, cute.slice_(self.mma_tiler, (None, 0, None)), (None, None, None) + ) + gB_nkl = cute.local_tile( + mB_nkl, cute.slice_(self.mma_tiler, (0, None, None)), (None, None, None) + ) + gSFA_mkl = cute.local_tile( + mSFA_mkl, cute.slice_(self.mma_tiler, (None, 0, None)), (None, None, None) + ) + gSFB_nkl = cute.local_tile( + mSFB_nkl, cute.slice_(self.mma_tiler_sfb, (0, None, None)), (None, None, None) + ) + k_tile_cnt = cute.size(gA_mkl, mode=[3]) + + thr_mma = tiled_mma.get_slice(mma_tile_coord_v) + thr_mma_sfb = tiled_mma_sfb.get_slice(mma_tile_coord_v) + tCgA = thr_mma.partition_A(gA_mkl) + tCgB = thr_mma.partition_B(gB_nkl) + tCgSFA = thr_mma.partition_A(gSFA_mkl) + tCgSFB = thr_mma_sfb.partition_B(gSFB_nkl) + + a_cta_layout = cute.make_layout(cute.slice_(cluster_layout_vmnk, (0, 0, None, 0)).shape) + tAsA, tAgA = cpasync.tma_partition( + tma_atom_a, + block_in_cluster_coord_vmnk[2], + a_cta_layout, + cute.group_modes(sA, 0, 3), + cute.group_modes(tCgA, 0, 3), + ) + b_cta_layout = cute.make_layout(cute.slice_(cluster_layout_vmnk, (0, None, 0, 0)).shape) + tBsB, tBgB = cpasync.tma_partition( + tma_atom_b, + block_in_cluster_coord_vmnk[1], + b_cta_layout, + cute.group_modes(sB, 0, 3), + cute.group_modes(tCgB, 0, 3), + ) + tAsSFA, tAgSFA = cpasync.tma_partition( + tma_atom_sfa, + block_in_cluster_coord_vmnk[2], + a_cta_layout, + cute.group_modes(sSFA, 0, 3), + cute.group_modes(tCgSFA, 0, 3), + ) + tAsSFA = cute.filter_zeros(tAsSFA) + tAgSFA = cute.filter_zeros(tAgSFA) + sfb_cta_layout = cute.make_layout( + cute.slice_(cluster_layout_sfb_vmnk, (0, None, 0, 0)).shape + ) + tBsSFB, tBgSFB = cpasync.tma_partition( + tma_atom_sfb, + block_in_cluster_coord_sfb_vmnk[1], + sfb_cta_layout, + cute.group_modes(sSFB, 0, 3), + cute.group_modes(tCgSFB, 0, 3), + ) + tBsSFB = cute.filter_zeros(tBsSFB) + tBgSFB = cute.filter_zeros(tBgSFB) + + tCrA = tiled_mma.make_fragment_A(sA) + tCrB = tiled_mma.make_fragment_B(sB) + acc_shape = tiled_mma.partition_shape_C(self.mma_tiler[:2]) + tCtAcc_fake = tiled_mma.make_fragment_C(cute.append(acc_shape, self.num_acc_stage)) + + pipeline_init_wait(cluster_shape_mn=self.cluster_shape_mn) + + clc_response_ptr = storage.clc_response.data_ptr() + clc_consumer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.Consumer, self.num_clc_stage + ) + is_first_cta_in_cluster = cta_rank_in_cluster == 0 + tile_sched = ClcDynamicPersistentTileScheduler.create( + tile_sched_params, + cute.arch.block_idx(), + cute.arch.grid_dim(), + clc_response_ptr, + physical_cluster_shape_mn=self.cluster_shape_mn, + ) + work_tile = tile_sched.initial_work_tile_info() + + # ============================================================== + # Epilogue warps + # ============================================================== + if warp_idx < self.mma_warp_id: + cute.arch.setmaxregister_increase(self.epilog_reg_count) + if cutlass.const_expr(self.enable_pdl): + cute.arch.griddepcontrol_wait() + tmem.allocate(self.num_tmem_alloc_cols) + tmem.wait_for_alloc() + acc_tmem_ptr = tmem.retrieve_ptr(self.acc_dtype) + + s_w_addr = cutlass.Int32(storage.sOnePlusW.data_ptr().toint()) + if cutlass.const_expr(not self.debug_gemm_out): + # (1 + w) per (slot type, dim), as the producer's FADD. + nw_addr = norm_weights.iterator.toint().to(cutlass.Int64) + for r in cutlass.range_constexpr(4 // self.epi_wgs): + j = tidx + cutlass.Int32(32 * len(self.epilog_warp_id) * r) + w_bits = _ldg_u16(nw_addr + j.to(cutlass.Int64) * cutlass.Int64(2)) + w_f = (w_bits << cutlass.Int32(16)).bitcast(cutlass.Float32) + _sts_f32(s_w_addr + j * cutlass.Int32(4), _add_rn(w_f, cutlass.Float32(1.0))) + self.epilog_sync_barrier.arrive_and_wait() + + alpha_v = alpha[0] + sl_k = kv_quant_scale[1] + sl_v = kv_quant_scale[2] + rcp_six = cutlass.Int32(RCP_SIX_BITS).bitcast(cutlass.Float32) + ( + data_page_stride, + data_plane_stride, + data_head_stride, + data_token_stride, + scale_page_stride, + scale_plane_stride, + scale_head_stride, + index_page_stride, + index_token_stride, + ) = cache_strides + q_addr = q_out.iterator.toint().to(cutlass.Int64) + iq_addr = index_q_out.iterator.toint().to(cutlass.Int64) + kvd_addr = kv_data_cache.iterator.toint().to(cutlass.Int64) + kvs_addr = kv_scale_cache.iterator.toint().to(cutlass.Int64) + ik_addr = index_k_cache.iterator.toint().to(cutlass.Int64) + cs_addr = rotary_cos_sin.iterator.toint().to(cutlass.Int64) + pos_addr = position_ids.iterator.toint().to(cutlass.Int64) + loc_addr = out_cache_loc.iterator.toint().to(cutlass.Int64) + g_addr = gemm_out.iterator.toint().to(cutlass.Int64) + + # TMEM -> register copy: thread t owns tile row t, 32 columns per load. + tCtAcc_all = cute.make_tensor(acc_tmem_ptr, tCtAcc_fake.layout) + tCtAcc_x = gemm_sm100.transform_partitioned_tensor_layout(tCtAcc_all) + epi_tile = (self.cta_tile_shape_mnk[0], self.epi_chunk) + tAcc_epi = cute.flat_divide(tCtAcc_x, epi_tile) + copy_atom_t2r = cute.make_copy_atom( + tcgen05.copy.Ld32x32bOp(tcgen05.copy.Repetition(self.epi_chunk)), + self.acc_dtype, + ) + tiled_copy_t2r = tcgen05.make_tmem_copy(copy_atom_t2r, tAcc_epi[(None, None, 0, 0, 0)]) + row_in_cta = tidx % cutlass.Int32(128) + epi_wg = cute.arch.make_warp_uniform(warp_idx // cutlass.Int32(4)) + thr_copy_t2r = tiled_copy_t2r.get_slice(row_in_cta) + tTR_tAcc = thr_copy_t2r.partition_S(tAcc_epi) + cC = cute.make_identity_tensor(epi_tile) + tTR_cC = thr_copy_t2r.partition_D(cC) + tTR_rAcc = cute.make_rmem_tensor(tTR_cC.shape, self.acc_dtype) + tTR_rAcc_flat = cute.make_tensor(tTR_rAcc.iterator, cute.make_layout((self.epi_chunk,))) + + acc_consumer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.Consumer, self.num_acc_stage + ) + + while work_tile.is_valid_tile: + cur_tile_coord = work_tile.tile_idx + row = cur_tile_coord[0] * cutlass.Int32(self.cta_tile_shape_mnk[0]) + row_in_cta + head0 = cur_tile_coord[1] * cutlass.Int32(self.heads_per_tile) + row_ok = row < m_i32 + row_c = cutlass.Int32(cutlass.select_(row_ok, row, m_i32 - cutlass.Int32(1))) + row64 = row_c.to(cutlass.Int64) + + if cutlass.const_expr(not self.debug_gemm_out): + position = _ldg_s32(pos_addr + row64 * cutlass.Int64(4)) + slot = _ldg_s32(loc_addr + row64 * cutlass.Int64(4)) + cs_row = cs_addr + position.to(cutlass.Int64) * cutlass.Int64(ROTARY_DIM * 4) + cs = [] + for qd in cutlass.range_constexpr(ROTARY_DIM // 4): + cs.extend(_ldg_v4_f32(cs_row + cutlass.Int64(16 * qd))) + slot_ok = slot >= cutlass.Int32(0) + slot_c = cutlass.Int32(cutlass.select_(slot_ok, slot, 0)) + page = (slot_c // cutlass.Int32(PAGE_SIZE)).to(cutlass.Int64) + within = (slot_c & cutlass.Int32(PAGE_SIZE - 1)).to(cutlass.Int64) + cache_ok = row_ok & slot_ok & (page < num_pages) + + acc_pipeline.consumer_wait(acc_consumer_state) + + def load_head(h): + # TMEM -> registers, bf16_rn(alpha * acc) packed in pairs. + words = [] + for c in cutlass.range_constexpr(self.chunks_per_head): + cute.copy( + tiled_copy_t2r, + tTR_tAcc[ + ( + None, + None, + None, + 0, + h * cutlass.Int32(self.chunks_per_head) + cutlass.Int32(c), + acc_consumer_state.index, + ) + ], + tTR_rAcc, + ) + for p in cutlass.range_constexpr(self.epi_chunk // 2): + a0, a1 = cute.arch.mul_packed_f32x2( + (tTR_rAcc_flat[2 * p], tTR_rAcc_flat[2 * p + 1]), + (alpha_v, alpha_v), + ) + words.append(_bf16x2_rn(a0, a1)) + return words + + def release_acc(): + cute.arch.fence_view_async_tmem_load() + nvvm.tcgen05_fence(nvvm.Tcgen05FenceKind.BEFORE_THREAD_SYNC) + acc_pipeline.consumer_release(acc_consumer_state) + + for hl in cutlass.range_constexpr(self.heads_per_wg): + h = epi_wg * cutlass.Int32(self.heads_per_wg) + cutlass.Int32(hl) + head = head0 + h + words = load_head(h) + if cutlass.const_expr(hl == self.heads_per_wg - 1): + release_acc() + + if cutlass.const_expr(self.debug_gemm_out): + if row_ok & (head < cutlass.Int32(self.num_slots)): + base = g_addr + ( + row64 * cutlass.Int64(self.n_total) + + head.to(cutlass.Int64) * cutlass.Int64(HEAD_DIM) + ) * cutlass.Int64(2) + for v in cutlass.range_constexpr(HEAD_DIM // 8): + _stg_v4( + base + cutlass.Int64(16 * v), + words[4 * v], + words[4 * v + 1], + words[4 * v + 2], + words[4 * v + 3], + ) + else: + valid = head < cutlass.Int32(self.num_slots) + is_v = (head >= cutlass.Int32(self.v_begin)) & ( + head < cutlass.Int32(self.iq_begin) + ) + not_v = (head < cutlass.Int32(self.v_begin)) | ( + head >= cutlass.Int32(self.iq_begin) + ) + if valid & is_v: + dw, sb = self._nvfp4_pack(self._unpack(words), False, sl_v, rcp_six) + if cache_ok: + self._store_kv( + dw, + sb, + False, + kvd_addr, + kvs_addr, + (head - cutlass.Int32(self.v_begin)).to(cutlass.Int64), + page, + within, + cache_strides, + ) + if valid & not_v: + e = self._norm_rope(words, head, cs, s_w_addr, eps) + if (head >= cutlass.Int32(self.k_begin)) & ( + head < cutlass.Int32(self.v_begin) + ): + dw, sb = self._nvfp4_pack(e, True, sl_k, rcp_six) + if cache_ok: + self._store_kv( + dw, + sb, + True, + kvd_addr, + kvs_addr, + (head - cutlass.Int32(self.k_begin)).to(cutlass.Int64), + page, + within, + cache_strides, + ) + else: + packed = [] + for i in cutlass.range_constexpr(HEAD_DIM // 4): + packed.append( + _e4m3x4(e[4 * i], e[4 * i + 1], e[4 * i + 2], e[4 * i + 3]) + ) + is_q = head < cutlass.Int32(self.k_begin) + is_iq = head < cutlass.Int32(self.ik_slot) + q_dst = q_addr + ( + row64 * cutlass.Int64(self.num_heads_q) + head.to(cutlass.Int64) + ) * cutlass.Int64(HEAD_DIM) + iq_dst = iq_addr + ( + row64 * cutlass.Int64(self.num_heads_index) + + (head - cutlass.Int32(self.iq_begin)).to(cutlass.Int64) + ) * cutlass.Int64(HEAD_DIM) + ik_dst = ( + ik_addr + page * index_page_stride + within * index_token_stride + ) + dst = cutlass.Int64( + cutlass.select_( + is_q, + q_dst, + cutlass.Int64(cutlass.select_(is_iq, iq_dst, ik_dst)), + ) + ) + ok = cutlass.Boolean(cutlass.select_(is_iq, row_ok, cache_ok)) + if ok: + for v in cutlass.range_constexpr(HEAD_DIM // 16): + _stg_v4( + dst + cutlass.Int64(16 * v), + packed[4 * v], + packed[4 * v + 1], + packed[4 * v + 2], + packed[4 * v + 3], + ) + + acc_consumer_state.advance() + clc_pipeline.consumer_wait(clc_consumer_state) + work_tile = tile_sched.get_current_work() + clc_pipeline.consumer_release(clc_consumer_state) + clc_consumer_state.advance() + + tmem.relinquish_alloc_permit() + # Every epilogue warp is done reading TMEM before the allocator frees it. + self.epilog_sync_barrier.arrive_and_wait() + tmem.free(acc_tmem_ptr) + + # ============================================================== + # TMA warp + # ============================================================== + if warp_idx == self.tma_warp_id: + cute.arch.setmaxregister_decrease(self.mainloop_reg_count) + if cutlass.const_expr(self.enable_pdl): + cute.arch.griddepcontrol_wait() + a_cache_policy = cutlass.Int64(0x14F0000000000000) # L2 evict_last + ab_producer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.Producer, self.num_ab_stage + ) + while work_tile.is_valid_tile: + cur_tile_coord = work_tile.tile_idx + mma_tile_coord_m = cur_tile_coord[0] // cute.size(tiled_mma.thr_id.shape) + n_tile = cur_tile_coord[1] + tAgA_slice = tAgA[(None, mma_tile_coord_m, None, 0)] + tBgB_slice = tBgB[(None, n_tile, None, 0)] + tAgSFA_slice = tAgSFA[(None, mma_tile_coord_m, None, 0)] + tBgSFB_slice = tBgSFB[(None, n_tile, None, 0)] + + ab_producer_state.reset_count() + peek_ab_empty_status = cutlass.Boolean(1) + if ab_producer_state.count < k_tile_cnt: + peek_ab_empty_status = ab_pipeline.producer_try_acquire(ab_producer_state) + for k_tile in cutlass.range(0, k_tile_cnt, 1, unroll=1): + ab_pipeline.producer_acquire(ab_producer_state, peek_ab_empty_status) + tma_bar = ab_pipeline.producer_get_barrier(ab_producer_state) + cute.copy( + tma_atom_a, + tAgA_slice[(None, ab_producer_state.count)], + tAsA[(None, ab_producer_state.index)], + tma_bar_ptr=tma_bar, + mcast_mask=a_full_mcast_mask, + cache_policy=a_cache_policy, + ) + cute.copy( + tma_atom_sfa, + tAgSFA_slice[(None, ab_producer_state.count)], + tAsSFA[(None, ab_producer_state.index)], + tma_bar_ptr=tma_bar, + mcast_mask=sfa_full_mcast_mask, + cache_policy=a_cache_policy, + ) + cute.copy( + tma_atom_b, + tBgB_slice[(None, ab_producer_state.count)], + tBsB[(None, ab_producer_state.index)], + tma_bar_ptr=tma_bar, + mcast_mask=b_full_mcast_mask, + ) + cute.copy( + tma_atom_sfb, + tBgSFB_slice[(None, ab_producer_state.count)], + tBsSFB[(None, ab_producer_state.index)], + tma_bar_ptr=tma_bar, + mcast_mask=sfb_full_mcast_mask, + ) + ab_producer_state.advance() + peek_ab_empty_status = cutlass.Boolean(1) + if ab_producer_state.count < k_tile_cnt: + peek_ab_empty_status = ab_pipeline.producer_try_acquire(ab_producer_state) + clc_pipeline.consumer_wait(clc_consumer_state) + work_tile = tile_sched.get_current_work() + clc_pipeline.consumer_release(clc_consumer_state) + clc_consumer_state.advance() + if cutlass.const_expr(self.enable_pdl): + cute.arch.griddepcontrol_launch_dependents() + ab_pipeline.producer_tail(ab_producer_state) + + # ============================================================== + # MMA warp + # ============================================================== + if warp_idx == self.mma_warp_id: + cute.arch.setmaxregister_decrease(self.mainloop_reg_count) + tmem.wait_for_alloc() + acc_tmem_ptr = tmem.retrieve_ptr(self.acc_dtype) + tCtAcc_base = cute.make_tensor(acc_tmem_ptr, tCtAcc_fake.layout) + sfa_tmem_ptr = cute.recast_ptr( + acc_tmem_ptr + self.num_accumulator_tmem_cols, dtype=self.sf_dtype + ) + tCtSFA = cute.make_tensor(sfa_tmem_ptr, self.tCtSFA_layout) + sfb_tmem_ptr = cute.recast_ptr( + acc_tmem_ptr + self.num_accumulator_tmem_cols + self.num_sfa_tmem_cols, + dtype=self.sf_dtype, + ) + tCtSFB = cute.make_tensor(sfb_tmem_ptr, self.tCtSFB_layout) + + sfa_s2t = self._mainloop_s2t_copy_and_partition(sSFA, tCtSFA) + sfb_s2t = self._mainloop_s2t_copy_and_partition(sSFB, tCtSFB) + + ab_consumer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.Consumer, self.num_ab_stage + ) + acc_producer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.Producer, self.num_acc_stage + ) + while work_tile.is_valid_tile: + ab_consumer_state.reset_count() + peek_ab_full_status = cutlass.Boolean(1) + if ab_consumer_state.count < k_tile_cnt and is_leader_cta: + peek_ab_full_status = ab_pipeline.consumer_try_wait(ab_consumer_state) + if is_leader_cta: + acc_pipeline.producer_acquire(acc_producer_state) + tCtAcc_stage = tCtAcc_base[(None, None, None, acc_producer_state.index)] + + for k_tile in cutlass.range(0, k_tile_cnt, 1, unroll=1): + if is_leader_cta: + ab_pipeline.consumer_wait(ab_consumer_state, peek_ab_full_status) + self._mainloop_s2t_copies(ab_consumer_state.index, sfa_s2t, sfb_s2t) + num_kblocks = cute.size(tCrA, mode=[2]) + for k_block in cutlass.range(num_kblocks, unroll_full=True): + a_kblk_crd = (None, 0, k_block, ab_consumer_state.index) + sfa_kblk_crd = (None, 0, k_block) + for n_set in cutlass.range_constexpr(self.n_sets): + b_kblk_crd = (None, n_set, k_block, ab_consumer_state.index) + sfb_kblk_crd = (None, n_set, k_block) + tCtAcc_set = tCtAcc_stage[(None, 0, n_set)] + mma_op = ( + tiled_mma + if n_set == 0 + else ( + tiled_mma_last + if n_set == self.n_sets - 1 + else tiled_mma_mid + ) + ) + mma_op.set(tcgen05.Field.ACCUMULATE, k_tile != 0 or k_block != 0) + cute.gemm( + mma_op, + tCtAcc_set, + [tCrA[a_kblk_crd], tCtSFA[sfa_kblk_crd]], + [tCrB[b_kblk_crd], tCtSFB[sfb_kblk_crd]], + tCtAcc_set, + ) + ab_pipeline.consumer_release(ab_consumer_state) + ab_consumer_state.advance() + peek_ab_full_status = cutlass.Boolean(1) + if ab_consumer_state.count < k_tile_cnt: + if is_leader_cta: + peek_ab_full_status = ab_pipeline.consumer_try_wait(ab_consumer_state) + if is_leader_cta: + acc_pipeline.producer_commit(acc_producer_state) + acc_producer_state.advance() + clc_pipeline.consumer_wait(clc_consumer_state) + work_tile = tile_sched.get_current_work() + clc_pipeline.consumer_release(clc_consumer_state) + clc_consumer_state.advance() + acc_pipeline.producer_tail(acc_producer_state) + + # ============================================================== + # CLC scheduler warp + # ============================================================== + if warp_idx == self.sched_warp_id: + cute.arch.setmaxregister_decrease(self.mainloop_reg_count) + if is_first_cta_in_cluster: + clc_producer_state = pipeline.make_pipeline_state( + pipeline.PipelineUserType.ProducerConsumer, self.num_clc_stage + ) + while work_tile.is_valid_tile: + clc_pipeline.producer_acquire(clc_producer_state) + mbarrier_addr = clc_pipeline.producer_get_barrier(clc_producer_state) + tile_sched.advance_to_next_work(mbarrier_addr) + clc_producer_state.advance() + clc_pipeline.consumer_wait(clc_consumer_state) + work_tile = tile_sched.get_current_work() + clc_pipeline.consumer_release(clc_consumer_state) + clc_consumer_state.advance() + clc_pipeline.producer_tail(clc_producer_state) + + if warp_idx == self.pad_warp_id: + cute.arch.setmaxregister_decrease(self.mainloop_reg_count) + + # ------------------------------------------------------------------ + # Straight-line epilogue helpers (plain Python: traced inline, no + # dynamic control flow inside). + # ------------------------------------------------------------------ + @staticmethod + def _unpack(words): + x = [] + for w in words: + x.append(_lo_f32(w)) + x.append(_hi_f32(w)) + return x + + def _norm_rope(self, words, head, cs, s_w_addr, eps): + """RMSNorm (1 + w) + NeoX rope of one head row in the producer's op order.""" + x = self._unpack(words) + # Lane partials: (0 + fma(x0, x0, x1*x1)) + fma(x2, x2, x3*x3); the + # leading 0 + is exact (the partial is never -0). + q = [] + for l in range(32): + yy0, yy1 = cute.arch.mul_packed_f32x2( + (x[4 * l + 1], x[4 * l + 3]), (x[4 * l + 1], x[4 * l + 3]) + ) + q.append( + cute.arch.fma_packed_f32x2( + (x[4 * l], x[4 * l + 2]), (x[4 * l], x[4 * l + 2]), (yy0, yy1) + ) + ) + lane = [] + for l in range(16): + s0, s1 = cute.arch.add_packed_f32x2( + (q[2 * l][0], q[2 * l + 1][0]), (q[2 * l][1], q[2 * l + 1][1]) + ) + lane.append(s0) + lane.append(s1) + # warpReduceSum (xor 16, 8, 4, 2, 1) as computed by lane 0. + half = 16 + while half > 1: + nxt = [] + for l in range(half // 2): + a0, a1 = cute.arch.add_packed_f32x2( + (lane[2 * l], lane[2 * l + 1]), (lane[2 * l + half], lane[2 * l + 1 + half]) + ) + nxt.append(a0) + nxt.append(a1) + lane = nxt + half //= 2 + total = _add_rn(lane[0], lane[1]) + inv_rms = _rsqrt_approx(_fma_rn(total, cutlass.Float32(1.0 / HEAD_DIM), eps)) + + is_q = head < cutlass.Int32(self.k_begin) + is_k = head < cutlass.Int32(self.v_begin) + is_iq = head < cutlass.Int32(self.ik_slot) + w_type = cutlass.Int32( + cutlass.select_( + is_q, + cutlass.Int32(0), + cutlass.Int32( + cutlass.select_( + is_k, + cutlass.Int32(1), + cutlass.Int32(cutlass.select_(is_iq, cutlass.Int32(2), cutlass.Int32(3))), + ) + ), + ) + ) + w_base = s_w_addr + w_type * cutlass.Int32(HEAD_DIM * 4) + e = [] + for qd in range(HEAD_DIM // 4): + opw = _lds_v4_f32(w_base + cutlass.Int32(16 * qd)) + t0, t1 = cute.arch.mul_packed_f32x2((inv_rms, inv_rms), (opw[0], opw[1])) + t2, t3 = cute.arch.mul_packed_f32x2((inv_rms, inv_rms), (opw[2], opw[3])) + e0, e1 = cute.arch.mul_packed_f32x2((x[4 * qd], x[4 * qd + 1]), (t0, t1)) + e2, e3 = cute.arch.mul_packed_f32x2((x[4 * qd + 2], x[4 * qd + 3]), (t2, t3)) + e.extend([e0, e1, e2, e3]) + half_rot = ROTARY_DIM // 2 + out = list(e) + for d in range(half_rot): + cos_v = cs[d] + sin_v = cs[half_rot + d] + lo = e[d] + hi = e[d + half_rot] + m0, m1 = cute.arch.mul_packed_f32x2((lo, hi), (cos_v, cos_v)) + r0, r1 = cute.arch.fma_packed_f32x2((-hi, lo), (sin_v, sin_v), (m0, m1)) + out[d] = r0 + out[d + half_rot] = r1 + return out + + @staticmethod + def _nvfp4_pack(vals, round_bf16, second_level, rcp_six): + """quantizeMinimaxM3Fp4x4 over 8 groups of 16: (16 data words, 8 scale bytes).""" + if round_bf16: + v = [] + for i in range(HEAD_DIM // 2): + w = _bf16x2_rn(vals[2 * i], vals[2 * i + 1]) + v.append(_lo_f32(w)) + v.append(_hi_f32(w)) + else: + v = vals + rcp_sl = _rcp_approx_ftz(second_level) + data_words = [] + scale_bytes = [] + for g in range(HEAD_DIM // 16): + amax = cutlass.Float32(0.0) + for i in range(16): + amax = _absmax(amax, v[16 * g + i]) + enc = _e4m3_byte(_mul_rn(second_level, _mul_rn(amax, rcp_six))) + out_scale = _rcp_approx_ftz(_mul_rn(_e4m3_byte_to_f32(enc), rcp_sl)) + out_scale = cutlass.Float32( + cutlass.select_(amax != cutlass.Float32(0.0), out_scale, cutlass.Float32(0.0)) + ) + qv = [] + for i in range(8): + p0, p1 = cute.arch.mul_packed_f32x2( + (v[16 * g + 2 * i], v[16 * g + 2 * i + 1]), (out_scale, out_scale) + ) + qv.append(p0) + qv.append(p1) + data_words.append(_e2m1x8(qv[0:8])) + data_words.append(_e2m1x8(qv[8:16])) + scale_bytes.append(enc) + return data_words, scale_bytes + + @staticmethod + def _store_kv( + data_words, scale_bytes, is_k, kvd_addr, kvs_addr, head_kv, page, within, cache_strides + ): + ( + data_page_stride, + data_plane_stride, + data_head_stride, + data_token_stride, + scale_page_stride, + scale_plane_stride, + scale_head_stride, + _, + _, + ) = cache_strides + plane = cutlass.Int64(0 if is_k else 1) + data_dst = ( + kvd_addr + + page * data_page_stride + + plane * data_plane_stride + + head_kv * data_head_stride + + within * data_token_stride + ) + for i in range(4): + _stg_v4( + data_dst + cutlass.Int64(16 * i), + data_words[4 * i], + data_words[4 * i + 1], + data_words[4 * i + 2], + data_words[4 * i + 3], + ) + scale_base = ( + kvs_addr + + page * scale_page_stride + + plane * scale_plane_stride + + head_kv * scale_head_stride + ) + if is_k: + # K: the 8 group scales of a token are contiguous. + words = [] + for wi in range(2): + b = scale_bytes[4 * wi : 4 * wi + 4] + words.append( + b[0] + | (b[1] << cutlass.Int32(8)) + | (b[2] << cutlass.Int32(16)) + | (b[3] << cutlass.Int32(24)) + ) + _stg_v2(scale_base + within * cutlass.Int64(8), words[0], words[1]) + else: + # V: four tokens interleaved per 32-byte group block. + vbase = ( + scale_base + + (within // cutlass.Int64(4)) * cutlass.Int64(32) + + (within % cutlass.Int64(4)) + ) + for g in range(HEAD_DIM // 16): + _stg_u8(vbase + cutlass.Int64(4 * g), scale_bytes[g]) + + # ------------------------------------------------------------------ + @cute.jit + def _mainloop_s2t_copy_and_partition(self, sSF: cute.Tensor, tSF: cute.Tensor) -> S2TCopyBundle: + tCsSF_compact = cute.filter_zeros(sSF) + tCtSF_compact = cute.filter_zeros(tSF) + copy_atom_s2t = cute.make_copy_atom(tcgen05.Cp4x32x128bOp(self.cta_group), self.sf_dtype) + tiled_copy_s2t = tcgen05.make_s2t_copy(copy_atom_s2t, tCtSF_compact) + thr_copy_s2t = tiled_copy_s2t.get_slice(0) + + def append_mn_broadcast_mode(smem_layout: cute.Layout): + mn_dim = cute.get(smem_layout, mode=[0, 0]) + mn_dim = cute.append(mn_dim, cute.make_layout((4), stride=(0))) + layout = cute.append(cute.group_modes(mn_dim, 0), cute.get(smem_layout, mode=[0, 1])) + layout = cute.append(cute.group_modes(layout, 0), cute.get(smem_layout, mode=[1])) + layout = cute.append(layout, cute.get(smem_layout, mode=[2])) + layout = cute.append(layout, cute.get(smem_layout, mode=[3])) + return layout + + tCsSF_compact_bcast = cute.make_tensor( + tCsSF_compact.iterator, append_mn_broadcast_mode(tCsSF_compact.layout) + ) + tCsSF_compact_s2t_ = thr_copy_s2t.partition_S(tCsSF_compact_bcast) + tCsSF_compact_s2t = tcgen05.get_s2t_smem_desc_tensor(tiled_copy_s2t, tCsSF_compact_s2t_) + tCtSF_compact_s2t = thr_copy_s2t.partition_D(tCtSF_compact) + return S2TCopyBundle(tiled_copy_s2t, tCsSF_compact_s2t, tCtSF_compact_s2t) + + @cute.jit + def _mainloop_s2t_copies(self, stage_idx, sfa_s2t_bundle, sfb_s2t_bundle): + s2t_stage_coord = (None, None, None, None, stage_idx) + cute.copy( + sfa_s2t_bundle.tiled_copy, + sfa_s2t_bundle.sSF_compact[s2t_stage_coord], + sfa_s2t_bundle.tSF_compact, + ) + cute.copy( + sfb_s2t_bundle.tiled_copy, + sfb_s2t_bundle.sSF_compact[s2t_stage_coord], + sfb_s2t_bundle.tSF_compact, + ) + + +# --------------------------------------------------------------------------- +# Host side +# --------------------------------------------------------------------------- + +_COMPILE_CACHE: dict = {} + + +def compile( + *, + num_heads_q: int = 32, + num_heads_kv: int = 2, + num_heads_index: int = 2, + heads_per_tile: int = 4, + mma_m: int = 256, + mma_n: int | None = None, + num_acc_stage: int | None = None, + cluster_shape_mn: Tuple[int, int] | None = None, + swizzle_size: int = 1, + raster_along_m: bool = True, + debug_gemm_out: bool = False, + enable_pdl: bool = True, + epi_wgs: int = 1, +): + """Compile one handle serving every M (and any cache geometry).""" + arch = os.environ.get("CUTE_DSL_ARCH") + if arch is not None and arch != "sm_107a": + raise RuntimeError("requires CUTE_DSL_ARCH=sm_107a") + key = ( + num_heads_q, + num_heads_kv, + num_heads_index, + heads_per_tile, + mma_m, + mma_n, + num_acc_stage, + None if cluster_shape_mn is None else tuple(cluster_shape_mn), + swizzle_size, + raster_along_m, + debug_gemm_out, + enable_pdl, + epi_wgs, + ) + if key in _COMPILE_CACHE: + return _COMPILE_CACHE[key] + op = M3QkvGemmNormRopeKvInsert( + num_heads_q=num_heads_q, + num_heads_kv=num_heads_kv, + num_heads_index=num_heads_index, + heads_per_tile=heads_per_tile, + mma_m=mma_m, + mma_n=mma_n, + num_acc_stage=num_acc_stage, + cluster_shape_mn=cluster_shape_mn, + swizzle_size=swizzle_size, + raster_along_m=raster_along_m, + debug_gemm_out=debug_gemm_out, + enable_pdl=enable_pdl, + epi_wgs=epi_wgs, + ) + + from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream + + def sym(): + return cute.sym_int64() + + sym_m, sym_k = sym(), sym() + fake_act = make_fake_compact_tensor( + cutlass.Float8E4M3FN, (sym_m, sym_k), stride_order=(1, 0), assumed_align=16 + ) + fake_w = make_fake_compact_tensor( + cutlass.Float8E4M3FN, (op.n_total, sym_k), stride_order=(1, 0), assumed_align=16 + ) + fake_asf = make_fake_compact_tensor(cutlass.Float8E8M0FNU, (sym(),), assumed_align=16) + fake_wsf = make_fake_compact_tensor(cutlass.Float8E8M0FNU, (sym(),), assumed_align=16) + fake_alpha = make_fake_compact_tensor(cutlass.Float32, (1,), assumed_align=4) + fake_q = make_fake_compact_tensor(cutlass.Uint8, (sym(),), assumed_align=16) + fake_iq = make_fake_compact_tensor(cutlass.Uint8, (sym(),), assumed_align=16) + fake_kvd = make_fake_compact_tensor(cutlass.Uint8, (sym(),), assumed_align=16) + fake_kvs = make_fake_compact_tensor(cutlass.Uint8, (sym(),), assumed_align=8) + fake_ik = make_fake_compact_tensor(cutlass.Uint8, (sym(),), assumed_align=16) + fake_loc = make_fake_compact_tensor(cutlass.Int32, (sym(),), assumed_align=4) + fake_kvq = make_fake_compact_tensor(cutlass.Float32, (sym(),), assumed_align=4) + fake_cs = make_fake_compact_tensor(cutlass.Float32, (sym(),), assumed_align=16) + fake_pos = make_fake_compact_tensor(cutlass.Int32, (sym(),), assumed_align=4) + fake_nw = make_fake_compact_tensor(cutlass.BFloat16, (4 * HEAD_DIM,), assumed_align=16) + fake_g = make_fake_compact_tensor(cutlass.BFloat16, (sym(),), assumed_align=16) + compiled = cute.compile( + op, + fake_act, + fake_asf, + fake_w, + fake_wsf, + fake_alpha, + fake_q, + fake_iq, + fake_kvd, + fake_kvs, + fake_ik, + fake_loc, + fake_kvq, + fake_cs, + fake_pos, + fake_nw, + fake_g, + *(cutlass.Int64(0) for _ in range(9)), + cutlass.Int64(0), + cutlass.Float32(1e-6), + make_fake_stream(), + options="--enable-tvm-ffi", + ) + _COMPILE_CACHE[key] = (op, compiled) + return _COMPILE_CACHE[key] + + +def _cache_strides(kv_data_cache, kv_scale_cache, index_k_cache): + """(data page, plane, head, token; scale page, plane, head; index page, token) strides in bytes.""" + return ( + int(kv_data_cache.stride(0)), + int(kv_data_cache.stride(1)), + int(kv_data_cache.stride(2)), + int(kv_data_cache.stride(3)), + int(kv_scale_cache.stride(0)), + int(kv_scale_cache.stride(1)), + int(kv_scale_cache.stride(2)), + int(index_k_cache.stride(0)), + int(index_k_cache.stride(2)), + ) + + +class M3QkvFusedRunner: + """Torch-callable wrapper: builds the TVM-FFI dlpack arguments once per call. + + Tensors (all CUDA, contiguous unless noted): + act e4m3 [M, K], act_sf / weight_sf 1-D swizzled UE8M0 K32 scales + (``torch.ops.trtllm.mxfp8_quantize(x, True)``), weight e4m3 [N, K], + alpha fp32 [1], q_out e4m3 [M, Hq, 128], index_q_out e4m3 [M, Hi, 128], + kv_data_cache uint8 [P, 2, Hkv, 128, 64], kv_scale_cache uint8 + [P, 2, Hkv, 128, 8], index_k_cache e4m3 [P, 1, 128, 128] (strided + pools allowed as long as rows stay contiguous and 16-byte aligned), + out_cache_loc int32 [>= M], kv_quant_scale fp32 [3], rotary_cos_sin + fp32 [max_pos, 2, 32], position_ids int32 [M], norm weights bf16 [128] + each (or one pre-concatenated [4 * 128] q|K|index-q|index-K tensor). + ``debug_gemm_out=True`` handles write only the bf16 GEMM output to + ``gemm_out`` [M, N] and leave every producer output untouched. + """ + + def __init__(self, **cfg): + self.op, self.compiled = compile(**cfg) + self.debug = bool(cfg.get("debug_gemm_out", False)) + + def make_args( + self, + act, + act_sf, + weight, + weight_sf, + alpha, + q_out, + index_q_out, + kv_data_cache, + kv_scale_cache, + index_k_cache, + out_cache_loc, + kv_quant_scale, + rotary_cos_sin, + position_ids, + norm_weights, + eps, + gemm_out=None, + ): + import torch + from cutlass.cute.runtime import from_dlpack + + op = self.op + assert act.dim() == 2 and weight.shape == (op.n_total, act.shape[1]) + assert act.shape[1] % 128 == 0 + m = act.shape[0] + assert q_out.shape == (m, op.num_heads_q, HEAD_DIM) + assert index_q_out.shape == (m, op.num_heads_index, HEAD_DIM) + assert kv_data_cache.dim() == 5 and kv_data_cache.shape[1:] == ( + 2, + op.num_heads_kv, + PAGE_SIZE, + HEAD_DIM // 2, + ) + assert kv_scale_cache.dim() == 5 and kv_scale_cache.shape[1:] == ( + 2, + op.num_heads_kv, + PAGE_SIZE, + HEAD_DIM // 16, + ) + assert index_k_cache.dim() == 4 and index_k_cache.shape[1:] == (1, PAGE_SIZE, HEAD_DIM) + assert kv_data_cache.stride(4) == 1 and kv_data_cache.stride(3) == HEAD_DIM // 2 + assert kv_scale_cache.stride(4) == 1 and kv_scale_cache.stride(3) == HEAD_DIM // 16 + assert index_k_cache.stride(3) == 1 and index_k_cache.stride(2) == HEAD_DIM + strides = _cache_strides(kv_data_cache, kv_scale_cache, index_k_cache) + assert all(s % 16 == 0 for s in strides[0:4] + strides[7:9]), strides + assert all(s % 8 == 0 for s in strides[4:7]), strides + for t, al in ((kv_data_cache, 16), (kv_scale_cache, 8), (index_k_cache, 16)): + assert t.data_ptr() % al == 0 + if isinstance(norm_weights, (list, tuple)): + norm_weights = torch.cat([w.reshape(-1) for w in norm_weights]) + assert norm_weights.dtype == torch.bfloat16 and norm_weights.numel() == 4 * HEAD_DIM + if gemm_out is None: + assert not self.debug + gemm_out = torch.empty(8, dtype=torch.bfloat16, device=act.device) + else: + assert gemm_out.shape == (m, op.n_total) and gemm_out.dtype == torch.bfloat16 + + def dl(t, al=16): + return from_dlpack(t, assumed_align=al, enable_tvm_ffi=True) + + def flat_u8(t): + return t.view(torch.uint8) if t.dtype != torch.uint8 else t + + # Caches are passed as flat byte views spanning their storage extent. + def span_u8(t): + base = flat_u8(t) + extent = 1 + sum((s - 1) * st for s, st in zip(base.shape, base.stride())) + return torch.as_strided(base, (extent,), (1,)) + + return ( + dl(act), + dl(act_sf.view(torch.float8_e8m0fnu).reshape(-1)), + dl(weight), + dl(weight_sf.view(torch.float8_e8m0fnu).reshape(-1)), + dl(alpha, 4), + dl(flat_u8(q_out).reshape(-1)), + dl(flat_u8(index_q_out).reshape(-1)), + dl(span_u8(kv_data_cache)), + dl(span_u8(kv_scale_cache), 8), + dl(span_u8(index_k_cache)), + dl(out_cache_loc, 4), + dl(kv_quant_scale, 4), + dl(rotary_cos_sin.reshape(-1)), + dl(position_ids, 4), + dl(norm_weights.reshape(-1)), + dl(gemm_out.reshape(-1)), + *strides, + int(kv_data_cache.shape[0]), + float(eps), + ) + + def run_args(self, args, stream=None): + import torch + + if stream is None: + stream = torch.cuda.current_stream().cuda_stream + self.compiled(*args, cuda_driver.CUstream(stream)) + + def __call__(self, *tensors, **kw): + self.run_args(self.make_args(*tensors, **kw)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/triton_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/triton_sparse_decode.py new file mode 100644 index 000000000000..87312508faeb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/triton_sparse_decode.py @@ -0,0 +1,1829 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. + +This is the MiniMax-M3.1 NVFP4 / sm_107 / tp2 target's copy. On top of the +in-tree decode it routes long-context TP2 steps of ``LC_MIN_ROWS``..``LC_MAX_ROWS`` +query rows to the one-launch ``m3_attn_lc`` kernel (LC2, TP2WIDE_LC2), short-sequence verify steps of at least +``MQ_MIN_TOKENS`` rows to the multi-query producer, releases PDL dependents +early, and has the merge emit the MXFP8 o_proj hand-off (``m3_prequant.SLOT``). +""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers +from tensorrt_llm.logger import logger + +from ..m31_switches import m31_disabled, m31_enabled +from . import m3_attn_lc, m3_prequant, m3_report + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 # capacity: fixed: the MSA block size of the checkpoint + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 # capacity: fixed: a launch-grid target, not a size + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +# Fewest query rows for which the merge writes the MXFP8 o_proj hand-off: every decode step, so the MXFP8 o_proj +# never runs its own quantize kernel after attention. +MERGE_MXFP8_MIN_TOKENS = 1 # capacity: fixed: a row lower bound + + +class MergeMxfp8: + """MXFP8 copy of the split-K merge output for the o_proj GEMM. + + From ``MERGE_MXFP8_MIN_TOKENS`` to ``TRITON_MAX_ROWS`` rows the Triton merge kernel (and up to ``LC_MAX_ROWS`` rows + the LC2 kernel) writes the E4M3 rows and swizzled UE8M0 scales next to the BF16 output and publishes them to + ``m3_prequant.SLOT``, so the MXFP8 o_proj skips its own quantization. One ``MAX_ROWS``-row buffer per width serves + every layer: each o_proj consumes it before the next layer's decode rewrites it, in stream order. + """ + + # TP2WIDE_LC2_448: the buffers hold the widest LC2 step (448 rows: K7 at 56 requests, K3 at 112); + # the Triton merge's own hand-off keeps its 128-row bound (TRITON_MAX_ROWS), so only LC2 steps use the extra rows. + MAX_ROWS = 448 # capacity: fixed: the hand-off buffers' rows (the widest LC2 decode step) + TRITON_MAX_ROWS = ( + 128 # capacity: fixed: the Triton merge / fullctx hand-off's rows (decode steps) + ) + + def __init__(self): + self._buffers = {} + + @classmethod + def scale_bytes(cls, rows: int, width: int) -> int: + """The swizzled UE8M0 scales of ``rows`` rows: whole 128-row tiles (a partial last tile is laid out in full).""" + return (rows + 127) // 128 * 128 * (width // 32) + + def buffers(self, width: int, device: torch.device): + key = (width, device) + if key not in self._buffers: + self._buffers[key] = ( + torch.empty((self.MAX_ROWS, width), dtype=torch.float8_e4m3fn, device=device), + # Whole scale tiles: MAX_ROWS (448) is not a multiple of 128; the kernels write the last tile in + # full. + torch.zeros( + (self.scale_bytes(self.MAX_ROWS, width),), dtype=torch.uint8, device=device + ), + ) + return self._buffers[key] + + @staticmethod + def handoff(mxfp8, rows: int): + """The hand-off of a ``rows``-row step: its E4M3 rows and the 128-row scale tiles that cover them. The o_proj + GEMM checks the swizzled scales against the step's padded rows, so the step gets the tiles that cover its rows, + never the whole buffer.""" + q8, sf = mxfp8 + width = q8.shape[-1] + return q8[:rows], sf[: MergeMxfp8.scale_bytes(rows, width)] + + +MERGE_MXFP8 = MergeMxfp8() + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +# Mixed ctx+gen steps launch on [num_ctx:] slices (unaligned seq_lens / block_table) with arbitrary +# total_q; specializing on those compiled a new variant mid-serving (~650 ms TTFT stall at c8). +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + out_ptr, # final [total_q, num_heads, head_dim] for FUSED_MERGE, else unused + counter_ptr, # int32 [total_q, num_kv_heads] arrival counters for FUSED_MERGE, zero at rest + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + num_bt_cols, # block-table width: a top-k id at or past it is not a block of the request + num_pages, # KV pool pages: a page id outside [0, num_pages) is not read + stride_out_n, + stride_out_h, + stride_out_d, + num_kv_heads, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge + FUSED_MERGE: tl.constexpr, # the last chunk to finish merges the partials in place + EARLY_TRIGGER: tl.constexpr, # release PDL dependents (weight prefetch) at entry + IDENTITY: tl.constexpr = False, # identity selection: slot i is block i (t_ptr is not read) +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + # Dependents still wait for this grid to complete; triggering here only + # lets them start their input-independent prologue (e.g. weight loads). + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for it in tl.range(chunk_start, chunk_end): + if IDENTITY: + # Slot `it` of the identity selection, bounded by this row's own kv extent above. + blk = it + else: + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + # Fail closed: a top-k id outside the request's block-table row, or the page it names outside the pool, + # contributes nothing instead of a read through a wrong page (valid inputs are unaffected). + blk_ok = (blk >= 0) & (blk < num_bt_cols) + page = tl.load(bt_row + blk, mask=blk_ok, other=0).to(tl.int64) + blk_ok = blk_ok & (page >= 0) & (page < num_pages) + page = tl.where(blk_ok, page, 0) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + # A slot that failed closed is skipped whole: no score, no rescale (an all-masked first block + # would make exp2(-inf - -inf) = NaN). + if blk_ok: + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + if FUSED_MERGE: + # Split-K serial reduction: every chunk publishes its partial, and the + # chunk that arrives last for this (token, KV head) merges all of them + # exactly as _merge_topk_attn_out_kernel does, then re-arms the counter. + tl.debug_barrier() + counter = counter_ptr + pid_b * num_kv_heads + pid_kh + arrived = tl.atomic_add(counter, 1, sem="acq_rel", scope="gpu") + if arrived == NUM_TOPK_CHUNKS - 1: + tl.debug_barrier() + o_rows = ( + o_ptr + + pid_b * stride_o_b + + (pid_h + off_h[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d + ) + l_rows = lse_ptr + pid_b * stride_l_b + (pid_h + off_h) * stride_l_h + lse_max = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + lse_max = tl.maximum(lse_max, lse_c) + # All chunks empty (a CUDA-graph padding row): zero, never NaN. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + denom = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + denom += tl.exp2(lse_c - lse_max) + denom = tl.where(denom > 0, denom, 1.0) + merged = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + w_c = tl.exp2(lse_c - lse_max) / denom + o_c = tl.load( + o_rows + c * stride_o_c, mask=hd_mask, other=0.0, cache_modifier=".cg" + ).to(tl.float32) + merged += o_c * w_c[:, None] + tl.store( + out_ptr + + pid_b * stride_out_n + + (pid_h + off_h[:, None]) * stride_out_h + + off_d[None, :] * stride_out_d, + merged.to(out_ptr.dtype.element_ty), + mask=hd_mask, + ) + tl.atomic_xchg(counter, 0, sem="relaxed", scope="gpu") + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + num_bt_cols, # block-table width: a top-k id at or past it is not a block of the request + num_pages, # KV pool pages: a page id outside [0, num_pages) is not read + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge + IDENTITY: tl.constexpr = False, # identity selection: slot i is block i (t_ptr is not read) +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + if IDENTITY: + blocks_0 = tl.where(chunk_offsets < count_0, chunk_start + chunk_offsets, sentinel) + blocks_1 = tl.where(chunk_offsets < count_1, chunk_start + chunk_offsets, sentinel) + else: + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for cand in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + if IDENTITY: + blk = chunk_start + cand + else: + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + if IDENTITY: + blk_0 = tl.where(valid_0, idx_0, sentinel) + blk_1 = tl.where(valid_1, idx_1, sentinel) + else: + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + if IDENTITY: + blk = chunk_start + cand + else: + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + # Fail closed as in _gqa_sparse_decode_kernel: an id outside the row, or a page outside the + # pool, reads nothing. + blk_ok = (blk >= 0) & (blk < num_bt_cols) + page = tl.load(bt_row + blk, mask=blk_ok, other=0).to(tl.int64) + blk_ok = blk_ok & (page >= 0) & (page < num_pages) + page = tl.where(blk_ok, page, 0) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + # A slot that failed closed is skipped whole: no score, no rescale (an all-masked first block + # would make exp2(-inf - -inf) = NaN). + if blk_ok: + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.jit +def _rcp_approx_ftz(x): + return tl.inline_asm_elementwise( + "rcp.approx.ftz.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _store_mxfp8_head(x, q8_ptr, sf_ptr, row, head, num_sf_k_tiles, HEAD_DIM: tl.constexpr): + """Quantize one row's [head * HEAD_DIM, (head + 1) * HEAD_DIM) slice to MXFP8 as + torch.ops.trtllm.mxfp8_quantize(x, True) does: UE8M0 scale = round-up(amax * rcp(448)), E4M3 RN-satfinite, + scales in the 128x4-swizzled layout.""" + NUM_SF: tl.constexpr = HEAD_DIM // 32 + xr = tl.reshape(x.to(tl.float32), (NUM_SF, 32)) + amax = tl.max(tl.abs(xr), axis=1) + sfv = amax * _rcp_approx_ftz(tl.full((NUM_SF,), 448.0, tl.float32)) + bits = sfv.to(tl.int32, bitcast=True) + e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(tl.int32) + e = tl.minimum(e, 254) + scale = ((254 - e) << 23).to(tl.float32, bitcast=True) + scale = tl.where(amax != 0, scale, 0.0) + q = (xr * scale[:, None]).to(tl.float8e4nv) + off_q = tl.reshape(tl.arange(0, HEAD_DIM), (NUM_SF, 32)) + tl.store(q8_ptr + row * (num_sf_k_tiles * 128) + head * HEAD_DIM + off_q, q) + k_sf = head * NUM_SF + tl.arange(0, NUM_SF) + off_sf = ( + (row // 128) * num_sf_k_tiles * 512 + + (k_sf // 4) * 512 + + (row % 32) * 16 + + ((row % 128) // 32) * 4 + + k_sf % 4 + ) + tl.store(sf_ptr + off_sf, e.to(tl.uint8)) + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr = False, + q8_ptr=None, # MXFP8 copy of out: [total_q, num_heads * head_dim] E4M3 + sf_ptr=None, # its UE8M0 block scales, 128x4-swizzled + num_sf_k_tiles=0, # ceil(num_heads * head_dim / 128) + MXFP8_OUT: tl.constexpr = False, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + if EARLY_TRIGGER: + # Release the dependent (the o_proj GEMM) before waiting on the attention producer, so its weight + # prefetch overlaps the attention kernel as well as this merge; its griddepcontrol.wait still waits + # for this grid (and therefore the producer) to complete. + tl.extra.cuda.gdc_launch_dependents() + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + if MXFP8_OUT: + _store_mxfp8_head( + o_merged.to(out_ptr.dtype.element_ty), + q8_ptr, + sf_ptr, + pid_b, + pid_h, + num_sf_k_tiles, + BLOCK_SIZE_D, + ) + + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on the datacenter Blackwell and Rubin targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3), (10, 7)) +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +# Multi-query producer (triton_sparse_decode_mq) for identity-selection verify steps with at least this many query +# tokens; below it the per-token producer is as fast. +MQ_MIN_TOKENS = 17 # capacity: fixed: the multi-query producer's crossover +# Warps per multi-query producer program. +MQ_NUM_WARPS = 8 +# The long-context decode (``_longattn_lc_decode``) in its engagement line (modeling's SPARSE_DECODE_LC path; the +# rows it takes are LC_MIN_ROWS..LC_MAX_ROWS, see TP2WIDE_LC2). +LC_PATH = "m3_sparse_decode_lc" +# LC_TP2 (OFF by default): the long-context rows run this Triton sparse decode at TP2, never the C++ +# ``sparse_decode_lc`` op. The TP4 build (LC4) is compiled for 16 query heads on one KV head per rank and the TP2 build +# (LC2) lacks the TP4 arguments (prefetch, next-KV, identity, one-div, pre-wait, early trigger), so neither serves the +# TP2 geometry (32 query heads, 2 KV heads per rank) as ported. OFF: the LC dispatch is not attempted at all; ON +# (``TRTLLM_M31_ENABLE_LC_TP2=1``, for an LC2 port) tries it, and ``m3_attn_lc.lc_applies`` still declines every +# TP2 shape until that port lands. ``TRTLLM_M31_DISABLE_LC_TP2=1`` always wins. Read at import. +# +# Superseded by TP2WIDE_LC2 below: the dispatch never asks ``lc_applies`` under LC_TP2 any more (it would now accept +# the TP2 geometry). LC_TP2 is kept for its switch contract and for the engagement line of the Triton route. +LC_TP2 = m31_enabled("LC_TP2") and not m31_disabled("LC_TP2") +# The engagement path of the LC_TP2-off route: an LC-shaped step (packed NVFP4 pages, FP8 q, one top-k chunk) that runs +# the Triton decode because LC_TP2 is off (``m3_report.taken``, one line per rank and row count). +LC_TRITON_PATH = "m3_sparse_decode_lc_tp2_triton" +# TP2WIDE_LC2 (TP2 WIDE rule, ported from the M3 sm_107 TP2 target): pure generation steps with a selected top-16 table +# (not the identity selection) at LC_MIN_ROWS..LC_MAX_ROWS query rows run LC2 +# (``torch.ops.trtllm.minimax_m3_sm107_tp2_sparse_decode_lc`` through ``m3_attn_lc``): attention, split-K merge and +# the MXFP8 o_proj hand-off in one launch, with the layer's N1 self-KV prefetch (modeling's +# SPARSE_DECODE_LC_PREFETCH_SELF_KV). Steps of at most 32 rows keep this target's Triton producer + merge exactly as +# before (same code path, bit for bit). The M3 target's LC2 settings: log-domain recurrence (``linear_softmax=False``), +# no pre-wait block-table staging (``LC_PREWAIT``), PDL from the environment, MXFP8 hand-off on. +# TRTLLM_M31_DISABLE_TP2WIDE_LC2=1 restores the Triton route at every width (LC_KEYS = frozenset()). Read at import. +TP2WIDE_LC2 = not m31_disabled("TP2WIDE_LC2") +# The fewest query rows LC2 takes: past the 32-row band, where the Triton route stays. +LC_MIN_ROWS = 33 # capacity: fixed: TP2 WIDE rule floor (rows <= 32 keep the Triton route) +# The long-context decode's shapes (modeling's SPARSE_DECODE_LC keys: "mxfp8" = a step of LC_MIN_ROWS.. +# LC_MAX_ROWS rows, the hand-off written and published to m3_prequant.SLOT; "plain", a wider step, is not covered). +# frozenset() routes every step back to the Triton producer + merge. +LC_KEYS = frozenset({"mxfp8"}) if TP2WIDE_LC2 else frozenset() +# LC2_ALLROWS (not bitwise): LC2 also takes the steps outside the LC band, from one row +# (1.03x the Triton route at 8 rows) and past LC_MAX_ROWS (1.31x at 512 rows), in the same error band vs float64. +# Steps past LC_MAX_ROWS are the "plain" key: no MXFP8 hand-off (the o_proj quantizes its input as usual). +# TRTLLM_M31_DISABLE_LC2_ALLROWS=1 restores the lc_min_rows..LC_MAX_ROWS band. Read at import. +LC2_ALLROWS = TP2WIDE_LC2 and not m31_disabled("LC2_ALLROWS") +if LC2_ALLROWS: + LC_KEYS = LC_KEYS | {"plain"} +# LC2_IDENTITY (not bitwise): an identity-selection step (every request within 16 blocks) also runs LC2, +# on the step's explicit identity table (2.0-2.4x the Triton route, the same band). TRTLLM_M31_DISABLE_LC2_IDENTITY=1 +# keeps those steps on the Triton route. Read at import. +LC2_IDENTITY = TP2WIDE_LC2 and not m31_disabled("LC2_IDENTITY") +# TP2WIDE_LC2_448 (exact-class as the 33..128 band): LC2 takes up to 448 query rows (agg TP2 c46 / c56 / +# c64 at K3 = 184 / 224 / 256 rows; K7 c46 / c56 = 368 / 448) in one launch with the MXFP8 hand-off, instead of the +# Triton producer + merge (no hand-off past 128 rows) there. TRTLLM_M31_DISABLE_TP2WIDE_LC2_448=1 restores the 128-row +# bound (wider steps back on the Triton route). Read at import. +LC_MAX_ROWS = ( # capacity: fixed: LC2's widest step + MergeMxfp8.TRITON_MAX_ROWS if m31_disabled("TP2WIDE_LC2_448") else MergeMxfp8.MAX_ROWS +) + +# K7 uses LC2 from 9 rows (not bitwise; passed its quality gates). K3 remains opt-in via +# TRTLLM_M31_ENABLE_TP2_LC2_SMALL=1; TRTLLM_M31_DISABLE_TP2_LC2_SMALL=1 kills both routes. +# Other query lengths keep the 33-row lower bound. Read once at import. +TP2_LC2_SMALL = not m31_disabled("TP2_LC2_SMALL") +LC2_SMALL_QLENS = frozenset({4, 8} if m31_enabled("TP2_LC2_SMALL") else {8}) +LC2_SMALL_MIN_ROWS = 9 # capacity: fixed: measured crossover (12 rows, the first K3 step past 8) + + +def lc_min_rows(decode_query_len: int) -> int: + """The fewest query rows LC2 takes for a step of ``decode_query_len`` query rows per request.""" + if LC2_ALLROWS: + return 1 + if TP2_LC2_SMALL and int(decode_query_len) in LC2_SMALL_QLENS: + return LC2_SMALL_MIN_ROWS + return LC_MIN_ROWS + + +# LC2 stages the request's block-table row before its PDL grid wait: off on the M3 target (measured +1.6..+1.8 us at 32 +# rows; flat to -0.2 us below), up to LC_PREWAIT_MAX_ROWS query rows when on (never at LC_MIN_ROWS and above). +LC_PREWAIT = False +LC_PREWAIT_MAX_ROWS = 16 # capacity: fixed: measured occupancy crossover +# The once-per-row-count engagement line of TP2WIDE_LC2 (per process; every GEN rank logs its own). +_TP2WIDE_LC2_LOGGED: set = set() + + +def _log_tp2wide_lc2(rows: int) -> None: + if rows not in _TP2WIDE_LC2_LOGGED: + _TP2WIDE_LC2_LOGGED.add(rows) + logger.info(f"[M3.1 tp2] TP2WIDE_LC2 taken: {rows} rows") + + +def _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, +) -> None: + """minimax_m3_sparse_attn_decode through m3_attn_lc (LC2): same table, output and MXFP8 hand-off contract.""" + use_pdl = get_env_enable_pdl() + total_q, num_heads, head_dim = q.shape + prewait = bool(LC_PREWAIT) and total_q <= LC_PREWAIT_MAX_ROWS + mxfp8 = None + m3_prequant.SLOT.clear() + if MERGE_MXFP8_MIN_TOKENS <= total_q <= LC_MAX_ROWS: + mxfp8 = MERGE_MXFP8.buffers(num_heads * head_dim, output.device) + # The early trigger lets the o_proj launch and stream its weights while attention runs. + m3_attn_lc.m3_lc_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + linear_softmax=False, + mxfp8=mxfp8, + use_pdl=use_pdl, + prewait=prewait, + n1_ids=m3_attn_lc.N1_IDS.take(), + ) + if prewait: + m3_report.taken(LC_PREWAIT_PATH, total_q) + if mxfp8 is not None: + m3_prequant.SLOT.publish( + output.view(total_q, num_heads * head_dim), *MergeMxfp8.handoff(mxfp8, total_q) + ) + + +# Engagement path of the identity-selection producers reading their blocks from the kv lengths (IDENTITY). +DEVICE_IDENTITY_PATH = "sparse_decode_device_identity" +# Engagement path of LC2's pre-wait block-table staging (LC_PREWAIT, up to its row cap). +LC_PREWAIT_PATH = "sparse_decode_lc_prewait" + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + if kv_nvfp4: + capability = torch.cuda.get_device_capability(q.device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103/SM107; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + # Long-context TP2 steps (TP2WIDE_LC2): attention, split-K merge and MXFP8 hand-off in one launch. A selected table + # on an NVFP4 cache with FP8 q at the TP2 heads and lc_min_rows(decode_query_len)..LC_MAX_ROWS rows is the + # kernel's (modeling's SPARSE_DECODE_LC); operands it declines are a miss. Every other step (narrower, identity + # selection, wider than LC_MAX_ROWS, or the kill switch) takes the unchanged Triton route below. + lc_shaped = kv_nvfp4 and widen_q and num_topk_chunks is None + lc_key = "mxfp8" if MERGE_MXFP8_MIN_TOKENS <= total_q <= LC_MAX_ROWS else "plain" + lc_covered = ( + lc_shaped + and (not identity_blocks or LC2_IDENTITY) + and total_q >= lc_min_rows(decode_query_len) + and num_heads == 32 + and num_kv_heads == 2 + and lc_key in LC_KEYS + ) + if lc_covered and not m3_attn_lc.lc_applies( + q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output + ): + m3_report.missed( + LC_PATH, + total_q, + "the step's q / KV pages / scales / top-k table / output are not the layout the kernel is built for", + ) + lc_covered = False + if lc_covered: + _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + ) + m3_report.taken(LC_PATH, total_q) + _log_tp2wide_lc2(total_q) + return + if not LC_TP2 and lc_shaped: + m3_report.taken(LC_TRITON_PATH, total_q, "LC rows on the Triton sparse decode (LC_TP2 off)") + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else 1 + ) + # The stock in-tree decode's split-K chunking at every row count (bit for bit its rows), as the SM107 TP2 target. + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + # Every query token of a request reads the same blocks under identity selection (short-sequence mode): one + # program per (request, block) covers all of them and dequantizes each block once. + use_mq = ( + identity_blocks + and total_q >= MQ_MIN_TOKENS + and kv_nvfp4 + and num_kv_heads == 1 + and decode_query_len > 1 + and head_dim == 128 + and gqa_group_size * decode_query_len <= 128 + ) + if identity_blocks and kv_nvfp4 and widen_q: + # The identity-selection route is modeling.SPARSE_DECODE_PRODUCER's. + m3_report.taken( + "sparse_decode_producer", + total_q, + "multi_query" if use_mq else "per_row", + ) + if use_mq: + num_topk_chunks = max_topk + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = get_env_enable_pdl() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + device_identity = identity_blocks + if device_identity and not use_mq: + m3_report.taken(DEVICE_IDENTITY_PATH, total_q) + single_kernel_args = ( + { + # No split-K arrival counters: the separate merge kernel combines the chunks. + "merge": (output, None), + "tail": (output.stride(0), output.stride(1), output.stride(2), num_kv_heads), + # An early release pays off when the dependent is the consumer of the + # final output (it can prefetch), or a merge grid that itself releases + # its dependent at entry: the o_proj then launches during this kernel and + # streams its weights while attention runs. Above 8 query rows the + # concurrent weight stream slows the larger attention grid more than the + # prefetch saves (GB300 TP4: -1.1% at 48 rows). + "options": {"FUSED_MERGE": False, "EARLY_TRIGGER": direct_output or total_q <= 8}, + } + if query_group_size == 1 + else {"merge": (), "tail": (), "options": {}} + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + if use_mq: + from .triton_sparse_decode_mq import launch_mq_producer + + launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + sm_scale=sm_scale, + decode_query_len=decode_query_len, + use_pdl=use_pdl, + early_trigger=False, + num_warps=MQ_NUM_WARPS, + ) + else: + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + *single_kernel_args["merge"], + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + block_table.shape[1], + k_paged.shape[0], + *single_kernel_args["tail"], + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + # Identity selection from the lengths this kernel reads, not the staged table (dispatch kill switch). + IDENTITY=device_identity, + **single_kernel_args["options"], + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + m3_prequant.SLOT.clear() + if not direct_output: + width = num_heads * head_dim + mxfp8 = ( + MERGE_MXFP8.buffers(width, output.device) + if MERGE_MXFP8_MIN_TOKENS <= total_q <= MergeMxfp8.TRITON_MAX_ROWS + and head_dim == 128 + and output.dim() == 3 + and output.is_contiguous() + and output.dtype == torch.bfloat16 + else None + ) + mxfp8_args = ( + { + "q8_ptr": mxfp8[0], + "sf_ptr": mxfp8[1], + "num_sf_k_tiles": width // 128, + "MXFP8_OUT": True, + } + if mxfp8 is not None + else {} + ) + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + # Release the o_proj at the merge's entry so it streams its weights during the merge. + EARLY_TRIGGER=True, + **mxfp8_args, + **pdl_launch, + **merge_launch_options, + ) + if mxfp8 is not None: + m3_prequant.SLOT.publish( + output.view(total_q, width), *MergeMxfp8.handoff(mxfp8, total_q) + ) + + +__all__ = [ + "MERGE_MXFP8", + "MERGE_MXFP8_MIN_TOKENS", + "MergeMxfp8", + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/triton_sparse_decode_mq.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/triton_sparse_decode_mq.py new file mode 100644 index 000000000000..048d31b82d68 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/kernels/triton_sparse_decode_mq.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multi-query block-sparse decode producer for MiniMax-M3 short-sequence verify steps. + +In short-sequence mode every request attends at most ``topk`` KV blocks and block selection is the identity, so +all ``decode_query_len`` query tokens of a request read the same blocks. The per-token producer +(``_gqa_sparse_decode_kernel``) then dequantizes every block once per query token. This producer runs one program +per (request, block) for all of the request's query tokens: rows are (token, head) pairs, each block is +dequantized once, and the causal limit is a per-row mask. + +It writes the same per-block partials (output scaled by 1/l and the V global scale, log2 LSE) as the per-token +producer with one block per chunk, so ``_merge_topk_attn_out_kernel`` merges them unchanged. +""" + +import triton +import triton.language as tl + +from .triton_sparse_decode import _dequant_nvfp4_rows + + +@triton.jit( + do_not_specialize=["num_reqs"], do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"] +) +def _gqa_sparse_decode_mq_kernel( + q_ptr, # [num_reqs * QLEN, num_heads, head_dim], FP8 or FP16/BF16 + k_ptr, # packed NVFP4 K: [num_pages, 1, page_size, head_dim / 2] uint8 + v_ptr, # packed NVFP4 V: same layout + k_sc_ptr, # K block scales [num_pages, 1, page_size, head_dim / 16], token-linear + v_sc_ptr, # V block scales, 4x4 token-quad order + k_gs_ptr, + v_gs_ptr, + t_ptr, # topk table [1, total_q, topk]; the request's last token's row is read + o_ptr, # partials [topk, total_q, num_heads, head_dim] fp32 + lse_ptr, # partials [topk, total_q, num_heads] fp32 (log2) + block_table_ptr, + seq_lens, + num_reqs, + sm_scale, + stride_qn, + stride_qh, + stride_k_blk, + stride_k_pos, + stride_v_blk, + stride_v_pos, + stride_ksc_blk, + stride_vsc_blk, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + QLEN: tl.constexpr, + GROUP: tl.constexpr, # query heads per KV head (16) + BLOCK_R: tl.constexpr, # next power of two of QLEN * GROUP + HEAD_DIM: tl.constexpr, + BLOCK_SIZE_K: tl.constexpr, + SCALE_COLS: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr, +): + pid_r = tl.program_id(0) + pid_c = tl.program_id(1) # top-k position == block index under identity selection + sm_scale_log2e = sm_scale * 1.4426950409 * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + + off_r = tl.arange(0, BLOCK_R) + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, HEAD_DIM) + off_dp = tl.arange(0, HEAD_DIM // 2) + off_s = tl.arange(0, SCALE_COLS) + row_t = off_r // GROUP + row_h = off_r % GROUP + r_mask = off_r < QLEN * GROUP + tok = pid_r * QLEN + row_t + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + pid_r) + # Token t of the request sits at seq_len - QLEN + t and attends positions [0, kv_len_t). + kv_len = tl.maximum(seq_len - QLEN + row_t + 1, 0) + blk = tl.load(t_ptr + (pid_r * QLEN + QLEN - 1) * stride_tn + pid_c * stride_tk) + blk_ok = blk >= 0 + blk = tl.where(blk_ok, blk, 0) + page = tl.load(block_table_ptr + pid_r * stride_bt_b + blk, mask=blk_ok, other=0).to(tl.int64) + # Rows past the last token's position are never read: their cache bytes may be stale (0 * NaN in the PV dot). + n_valid = tl.where( + blk_ok, tl.minimum(tl.maximum(seq_len - blk * BLOCK_SIZE_K, 0), BLOCK_SIZE_K), 0 + ) + n_mask = off_n < n_valid + + q = tl.load( + q_ptr + tok[:, None] * stride_qn + row_h[:, None] * stride_qh + off_d[None, :], + mask=r_mask[:, None], + other=0.0, + ).to(tl.float16) + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + page * stride_k_blk + off_n[:, None] * stride_k_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + page * stride_ksc_blk + off_n[:, None] * SCALE_COLS + off_s[None, :], + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ) + valid = ( + (blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None]) & n_mask[None, :] & r_mask[:, None] + ) + qk = tl.dot(q, tl.trans(k_deq).to(tl.float16)) * sm_scale_log2e + qk = tl.where(valid, qk, float("-inf")) + m = tl.max(qk, axis=1) + m_safe = tl.where(m == float("-inf"), 0.0, m) + p = tl.exp2(qk - m_safe[:, None]) + p_sum = tl.sum(p, axis=1) + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + page * stride_v_blk + off_n[:, None] * stride_v_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ).to(tl.float16) + acc = tl.dot(p.to(tl.float16), v) + # An empty partial stores zero with LSE -inf, so the merge gives it weight 0. + scale = tl.where(p_sum > 0.0, 1.0 / p_sum, 0.0) * v_global_scale + lse = tl.where(p_sum > 0.0, m + tl.log2(p_sum), float("-inf")) + acc = acc * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + tok[:, None] * stride_o_b + + row_h[:, None] * stride_o_h + + off_d[None, :], + acc, + mask=r_mask[:, None], + ) + tl.store(lse_ptr + pid_c * stride_l_c + tok * stride_l_b + row_h * stride_l_h, lse, mask=r_mask) + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + *, + sm_scale, + decode_query_len, + use_pdl, + early_trigger, + num_warps=8, + num_stages=1, +): + """Launch the multi-query producer; arguments as minimax_m3_sparse_attn_decode after its NVFP4 views.""" + total_q, num_heads, head_dim = q.shape + num_reqs = total_q // decode_query_len + max_topk = int(topk_idx.shape[-1]) + rows = decode_query_len * num_heads + pdl_launch = {"launch_pdl": True} if use_pdl else {} + _gqa_sparse_decode_mq_kernel[(num_reqs, max_topk)]( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + num_reqs, + sm_scale, + q.stride(0), + q.stride(1), + k_paged.stride(0), + k_paged.stride(2), + v_paged.stride(0), + v_paged.stride(2), + k_block_scale.stride(0), + v_block_scale.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + o_arg.stride(0), + o_arg.stride(1), + o_arg.stride(2), + lse_arg.stride(0), + lse_arg.stride(1), + lse_arg.stride(2), + block_table.stride(0), + QLEN=decode_query_len, + GROUP=num_heads, + BLOCK_R=triton.next_power_of_2(rows), + HEAD_DIM=head_dim, + BLOCK_SIZE_K=int(k_paged.shape[2]), + SCALE_COLS=head_dim // 16, + USE_PDL=use_pdl, + EARLY_TRIGGER=early_trigger, + num_warps=num_warps, + num_stages=num_stages, + **pdl_launch, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/m31_switches.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/m31_switches.py new file mode 100644 index 000000000000..51f966c595fd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/m31_switches.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Kill switches of this MiniMax-M3.1 target's default-on items (each target keeps its own copy: targets share +only the catalog). + +Every item is on by default; ``TRTLLM_M31_DISABLE_=1`` turns item ```` off and restores the path it replaced. +The environment is read once, at import, so every rank and every step of a process sees the same setting (a step must +not change its graph mid-run). Each item lists its id and what the switch restores where it reads it. + +Values (case-insensitive, surrounding blanks ignored): ``1`` / ``true`` / ``on`` / ``yes`` set a switch; empty / ``0`` / +``false`` / ``off`` / ``no`` leave it unset; anything else raises at import, naming the variable, so a typo fails loudly +instead of silently turning an item on or off. +""" + +import os +from typing import FrozenSet + +_PREFIX = "TRTLLM_M31_DISABLE_" +_SET = frozenset(("1", "true", "on", "yes")) +_UNSET = frozenset(("", "0", "false", "off", "no")) + + +def _is_set(name: str, value: str) -> bool: + """Whether the switch variable ``name`` is set to ``value``; raises on a value of neither kind.""" + word = value.strip().lower() + if word in _SET: + return True + if word in _UNSET: + return False + raise ValueError( + f"{name}={value!r}: expected 1 / true / on / yes (set) or 0 / false / off / no / empty (unset)" + ) + + +# The item ids switched off in this process, read once at import. +_DISABLED: FrozenSet[str] = frozenset( + name[len(_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_PREFIX) and _is_set(name, value) +) + + +def m31_disabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_DISABLE_`` turned the item off for this process.""" + return item_id in _DISABLED + + +# Items that are OFF by default (rule 1b without a passing gate c, or a failed gate c): ``TRTLLM_M31_ENABLE_=1`` +# opts in; the item's ``TRTLLM_M31_DISABLE_`` still wins. +_ENABLE_PREFIX = "TRTLLM_M31_ENABLE_" + +_ENABLED: FrozenSet[str] = frozenset( + name[len(_ENABLE_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_ENABLE_PREFIX) and _is_set(name, value) +) + + +def m31_enabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_ENABLE_`` opted a default-off item in for this process.""" + return item_id in _ENABLED diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py new file mode 100644 index 000000000000..2153e950db70 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py @@ -0,0 +1,9237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: MiniMax-M3.1 (MXFP8 + packed NVFP4 experts) / sm_107 / tp2 -- self-contained modeling code. + +MiniMax-M3.1's language model has MiniMax-M3's shape (60 layers, hidden 6144, 64 query heads and 4 KV heads of dim +128, 128 routed experts top 4 plus one shared expert) with a 1e7 RoPE base and MSA block-sparse attention on EVERY +layer (an index branch scores 128-token blocks and the top 16 are attended); layers 0-2 keep the dense SwiGLU-OAI MLP, +layers 3-59 the MoE. The checkpoint is an MXFP8 export whose routed experts are packed NVFP4 (compressed-tensors +names, global scales stored as multipliers; ``weights`` takes their FP32 reciprocals); the KV cache is NVFP4 with an +FP8 index-K on every layer. The layer kinds are read from the config (``sparse_attention_freq``, ``moe_layer_freq``), +so this code is the MiniMax-M3 target's with no dense-attention layer in use. + +``tp4`` is four Rubin GPUs (SM 10.7): head-split attention (16 query heads, one KV head and one index head per rank) and +tensor-parallel experts (every expert on every rank, intermediate 768 per rank), no attention DP. Speculative +decoding, when configured, is one-model Eagle3 (draft length 3): the BF16 drafter (``M3Eagle3Draft``) and the BF16 +target head, both at checkpoint precision; decode steps of up to 8 requests verify and draft through fused kernels +at that precision (``M3Eagle3Worker``). + +**The core** (``MiniMaxM3Tp2ModelingV2``) holds the weights (``weights.MODEL_WEIGHTS``) and the runtime resources the +steps use -- one MSA attention backend per layer, the TP all-reduce communicator, the RoPE table, the MXFP8 GEMM +dispatch state the engine tunes -- and dispatches each step, on the host, from its context count alone +(``_select_target``): + +* ``DecodeTarget`` takes every pure generation step. Up to ``DECODE_MAX_TOKENS`` tokens (up to 8 requests, each one + token or a draft verify of 1 + 3 rows) with 1-D input ids it runs the decode-step kernels and picks each layer + boundary's collective by the step's token bucket (``DECODE_PLANS``); it runs its own generic layer for the first + layer when the fused embedding does not take the step, for every layer whose output Eagle3 captures, and for every + layer of a wider step. +* ``PrefillTarget`` takes every step with context rows: prefill chunks and mixed steps, every layer in its generic + form over the MSA backend. + +The two targets share no body: each owns its layer, attention and MoE compositions and its ``step_args``, the one +projection of the step metadata its forward reads. What both call are the module-level op adapters, each picking the +kernel for one call from shapes and layouts (``_linear``, ``_qk_norm_rope``, ``_sparse_qkv_producer``, the router and +MoE-runner calls, the all-reduce + norm calls), which read no step metadata themselves. + +Every fast path the decode step can take, the rows each covers (``FAST_PATHS``, each paired with the entry whose +``CELLS`` certify it) and what a step that cannot take one does are below: **uncovered** (outside every covered +range: log once, run the generic path) or **missed** (inside a covered range but a precondition is missing: raise +``FastPathUnavailable`` under ``modeling_v2='require'``, else log once and fall back). + +**Hand-offs between kernels.** A producer that already holds an activation in registers writes its MXFP8 form for the +next MXFP8 projection: the decode boundary all-reduce for the next layer's QKV, the sparse-decode merge and the +long-context decode for the o_proj. It publishes it in ``kernels/m3_prequant.SLOT`` keyed by the BF16 tensor; the next +MXFP8 projection on a decode graph takes it (``_linear``), and every take clears the slot. The decode boundaries' +push targets (the o_proj / shared-expert GEMVs and the MoE engine writing straight into the boundary all-reduce's +exchange) are passed explicitly from the layer to the producer and back. + +**What this target asserts rather than adapts**: SM 10.3, the topology and checkpoint (``routing.py``); the +MiniMax-M3 MSA sparse attention runtime with the fused five-way QKV/index projection, an FP8 indexer and an FP8 or +NVFP4 KV cache, no torch.compile and no piecewise prefill graphs (``_validate_runtime``); text-only input (the vision +tower is not built and a step carrying image input raises). +""" + +from __future__ import annotations + +import atexit +import copy +import dataclasses +import os +import signal +import sys +import threading +import time +from dataclasses import dataclass +from types import MappingProxyType +from typing import ( + Any, + Callable, + Dict, + FrozenSet, + Hashable, + Iterable, + List, + Mapping, + NamedTuple, + Optional, + Tuple, + Union, +) + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch._experimental.modeling_v2._router_index import ( + ModelingV2Mode, + resolved_mode, + step_contract_enabled, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops, # noqa: F401 (registers the fake impls of the M3.1 torch ops) +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( # noqa: E501 + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_rmsnorm import ( + flashinfer_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.quantization.fp4_quantize import ( + fp4_quantize, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 import m31_pins +from tensorrt_llm._torch.attention.backends import AttentionMetadata +from tensorrt_llm._torch.attention.backends.fmha.msa_prefill import _aligned_nvfp4_dequant_scales +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionInputType, + PositionalEmbeddingParams, + RopeParams, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import MiniMaxM3MsaSparseAttention +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import ( + nvfp4_index_k as _nvfp4_index_k, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + MSA_REQUIRED_TOPK, + msa_paged_kv, + require_msa_module, + write_msa_phase_kv, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) +from tensorrt_llm._torch.attention.backends.utils import create_attention +from tensorrt_llm._torch.autotuner import AutoTuner +from tensorrt_llm._torch.custom_ops.trtllm_gen_custom_ops import FP4BlockScaleMoERunner +from tensorrt_llm._torch.distributed import ( + AllReduce, + AllReduceFusionOp, + AllReduceParams, + AllReduceStrategy, +) +from tensorrt_llm._torch.distributed.ops import ( + MNNVLAllReduce, + get_or_scale_allreduce_mnnvl_workspace, +) +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + _ensure_draft_vocab_size, + _get_requested_draft_moe_backend, + _set_draft_kv_cache_quant_algo, + _slice_spec_position_ids, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.modules.embedding import LMHead +from tensorrt_llm._torch.modules.linear import ( + MXFP8LinearMethod, + TensorParallelMode, + flashinfer_mxfp8_autotune_active, + flashinfer_mxfp8_decode_graph_capture_active, +) +from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor +from tensorrt_llm._torch.modules.multi_stream_utils import maybe_execute_in_parallel +from tensorrt_llm._torch.moe.fused_moe.routing import RoutingMethodType +from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph import ( + eager_on_graph, + is_in_breakable_cuda_graph, +) +from tensorrt_llm._torch.pyexecutor.h2d_cache import copy_h2d_if_changed +from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import MambaHybridCacheManager +from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy, RuntimeHooks +from tensorrt_llm._torch.pyexecutor.sampler.ops.flashinfer import ( + set_softmax_override, + set_top_p_renorm_override, +) +from tensorrt_llm._torch.speculative import SpecMetadata +from tensorrt_llm._torch.speculative.eagle3 import Eagle3OneModelWorker +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + set_chain_speculative_sampling_override, + set_chain_speculative_sampling_slot_override, + should_use_separate_draft_kv_cache, +) +from tensorrt_llm._torch.utils import ActType_TrtllmGen, get_model_extra_attrs, is_torch_compiling +from tensorrt_llm._utils import maybe_pin_memory, mpi_rank, prefer_pinned +from tensorrt_llm.functional import PositionEmbeddingType +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.mode import QuantAlgo + +from . import dspark as _dspark +from . import weights as _weights +from .kernels import lc2_mixed as _lc2_mixed +from .kernels import m3_attn_lc, m3_fold_tail, m3_mc_allreduce, m3_prequant, m3_report, m3_spec_tail +from .kernels import m3_draft_lowm as _draft_lowm_kernel +from .kernels import m3_draft_step as m3_draft_step_kernel +from .kernels import m3_drafter_attn as _drafter_attn_kernel +from .kernels import m3_fullctx_decode as _fullctx_decode_kernel +from .kernels import m3_index_score as _index_score_kernel +from .kernels import m3_index_score_t2sc as _index_score_t2sc_kernel +from .kernels import m3_sampled_accept as m3_sampled_accept_kernel +from .kernels import m3_select_long_dsl as _select_long_dsl_kernel +from .kernels import m3_shard_accept as m3_shard_accept_kernel +from .kernels import m3_target_head as m3_target_head_kernel +from .kernels import m3_tokmap_guard as _tokmap_guard +from .kernels import msa_kv_update as _msa_kv_update_kernel +from .kernels import triton_sparse_decode as _sparse_decode_kernel +from .kernels.long_select import long_select_blocks +from .kernels.m3_moe.op import INTER as _MOE_INTER +from .kernels.m3_moe_finalize import m3_moe_finalize +from .kernels.m3_nvfp4_index_decode import Nvfp4IndexDecodeScorer +from .kernels.mx_gemv import op as mx_gemv +from .kernels.triton_sparse_decode import MERGE_MXFP8, MERGE_MXFP8_MIN_TOKENS +from .m31_switches import m31_disabled, m31_enabled +from .ops import gqamix_sparse_decode as _gqamix_kernel +from .ops.activation.m3_silu_and_mul import m3_silu_and_mul +from .ops.activation.m3_swq import m3_swq +from .ops.attention.m3_dense_decode import m3_dense_decode +from .ops.attention.m3_dense_qk_norm_rope_fp8 import m3_dense_qk_norm_rope_fp8 +from .ops.attention.m3_draft_attention import m3_draft_attention +from .ops.attention.m3_fp8_indexer_qk_norm_rope import m3_fp8_indexer_qk_norm_rope +from .ops.attention.m3_fullctx_decode import m3_fullctx_decode +from .ops.attention.m3_index_score import SPLIT_K as INDEX_SCORE_SPLIT_K +from .ops.attention.m3_index_score import m3_index_score +from .ops.attention.m3_index_score_t2sc import m3_index_score_t2sc +from .ops.attention.m3_msa_attention import m3_msa_attention +from .ops.attention.m3_producer import m3_producer +from .ops.attention.m3_qk_norm_rope_fp8 import m3_qk_norm_rope_fp8 +from .ops.attention.m3_qkv_insert import m3_qkv_insert +from .ops.attention.m3_select_blocks import m3_select_blocks +from .ops.attention.m3_select_long import m3_select_long +from .ops.attention.m3_select_long_dsl import m3_select_long_dsl +from .ops.attention.m3_sparse_decode import m3_sparse_decode +from .ops.attention.m3_sparse_decode_lc import m3_sparse_decode_lc, prepare_lc2_split_workspace +from .ops.comm.m3_allreduce import m3_allreduce +from .ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm +from .ops.comm.m3_mc_finalize_allreduce_norm import m3_mc_finalize_allreduce_norm +from .ops.embedding.m3_embedding import m3_embedding +from .ops.gemm import m3_mxfp8_cublaslt as mxfp8_cublaslt +from .ops.gemm import m31_agg_prefill_cublaslt as agg_prefill_cublaslt +from .ops.gemm.m3_bf16_linear import m3_bf16_linear +from .ops.gemm.m3_dense_swiglu import m3_dense_swiglu +from .ops.gemm.m3_draft_lowm import ROWS as DRAFT_LOWM_TACTIC_ROWS +from .ops.gemm.m3_draft_lowm import m3_draft_lowm +from .ops.gemm.m3_fp32_linear import m3_fp32_linear +from .ops.gemm.m3_mx_gemv import ( + DENSE_DOWN, + DENSE_GATE_UP, + GEMV48_MIN_TOKENS, + GEMV48_SHAPES, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, + m3_mx_gemv, +) +from .ops.gemm.m3_mx_gemv_bf16 import m3_mx_gemv_bf16 +from .ops.gemm.m3_mxfp8_cublaslt import m3_mxfp8_cublaslt +from .ops.gemm.m3_mxfp8_gemm import m3_mxfp8_gemm +from .ops.gemm.m3_mxfp8_linear import m3_mxfp8_linear +from .ops.gemm.m3_mxfp8_pinned_gemm import m3_mxfp8_pinned_gemm +from .ops.gemm.m3_mxfp8_pinned_quantize import m3_mxfp8_pinned_quantize +from .ops.gemm.m3_target_head import MAX_ROWS as TARGET_HEAD_KERNEL_MAX_ROWS +from .ops.gemm.m3_target_head import m3_target_head +from .ops.moe import m31_s1b_fold +from .ops.moe.m3_moe16_engine import M3Moe16State, m3_moe16_engine +from .ops.moe.m3_moe_engine import M3MoeState, ar_push_desc, m3_moe_engine, shared_ctas_for +from .ops.moe.m3_moe_tactic_pin import run_pinned as _run_pinned_moe +from .ops.moe.m3_router_tc import m3_router_tc +from .ops.norm.m3_embedding_firstnorm import DecodeEmbedding, m3_embedding_firstnorm +from .ops.sampling import m31_chain_spec_mc as _chain_spec_mc +from .ops.sampling import m31_softmax_mc as _softmax_mc +from .ops.sampling import m31_topp_det as _topp_det +from .ops.spec.m3_draft_input import m3_draft_input +from .ops.spec.m3_draft_silu import m3_draft_silu +from .ops.spec.m3_draft_step import MAX_TOKENS as DRAFT_STEP_KERNEL_MAX_TOKENS +from .ops.spec.m3_draft_step import m3_draft_step +from .ops.spec.m3_force_vec import m3_force_vec +from .ops.spec.m3_force_vec_req import m3_force_vec_req +from .ops.spec.m3_shard_accept import m3_shard_accept +from .ops.spec.m3_spec_offsets import m3_spec_offsets +from .ops.spec.m3_step_prologue import m3_step_prologue + +_SM = (10, 7) # capacity: fixed: the device capability, not a size +# PROXYPB: compile the MSA indexer proxy's fmha_sm100 variants before warmup (``_prebuild_msa_proxy``). Compile and +# load only, no numerics change. TRTLLM_M31_DISABLE_PROXYPB=1: each variant compiles on its first call instead (which +# can be after the server is ready). +MSA_PROXY_PREBUILD = os.environ.get("TRTLLM_M31_DISABLE_PROXYPB", "0").strip() in ("", "0") +# R6: the dense-MLP layer boundary also publishes the next layer's MXFP8 QKV activation (as the MoE boundary does): +# M3.1's layers 1-3 follow a dense MLP but have the sparse five-way QKV. TRTLLM_M31_DISABLE_R6=1 restores the BF16 +# hand-off (the next QKV quantizes its own input). +# VR: OFF by default. On sm_107 the hand-off's QKV output is not bitwise against the BF16 path, and R6 has not passed +# the natural-acceptance gate (with R6 off, that gate passes). +# TRTLLM_M31_ENABLE_R6=1 turns it on; TRTLLM_M31_DISABLE_R6=1 still forces it off. +_R6_ENABLED = os.environ.get("TRTLLM_M31_ENABLE_R6", "0").strip() not in ("", "0") +_R6_DISABLED = os.environ.get("TRTLLM_M31_DISABLE_R6", "0").strip() not in ("", "0") +R6_DENSE_PREQUANT = _R6_ENABLED and not _R6_DISABLED +# CTXNORMQ: a prefill layer boundary of at least ``CTXNORMQ_MIN_ROWS`` rows runs the plain all-reduce, then +# one kernel for the residual add + RMSNorm that also writes the next MXFP8 projection's activation (and, at a MoE +# boundary, the routed experts' NVFP4 input), instead of the fused all-reduce + norm followed by the consumers' own +# quantize passes. The quantized outputs equal mxfp8_quantize / fp4_quantize of the norm bit for bit; the norm itself +# is not bitwise against the fused all-reduce's (reduction order). Opt-in: TRTLLM_M31_ENABLE_CTXNORMQ=1; +# TRTLLM_M31_DISABLE_CTXNORMQ=1 wins. +CTXNORMQ = m31_enabled("CTXNORMQ") and not m31_disabled("CTXNORMQ") +# TP2AGG_ROUTER_F32 (the tp2ctx NORMQ_F32 router fold): a MoE layer's CTXNORMQ boundary runs the op's _f32 +# variant, which also writes the normed rows widened to fp32 (bit for bit norm.float()); the prefill router reads them +# instead of m3_fp32_linear's own x.to(float32) cast (~300 us at 16K rows on sm_107). Same GEMM operands, so the +# logits are bitwise today's. The op comes from the build or the overlay tensorrt_llm/libs/libm31_agg_ctxnormq_f32.so; +# without it the boundary keeps the base op. Kill switch: TRTLLM_M31_DISABLE_TP2AGG_ROUTER_F32=1. +ROUTER_F32 = not m31_disabled("TP2AGG_ROUTER_F32") +# F3: on TP2 decode-graph steps of F3_MIN_ROWS..F3_MAX_ROWS rows, the o_proj's MX-TAC pinned GEMM (FlashInfer's +# dense blockscaled GEMM, epilogue-only fork kernels/m31_f3_gemm) stores its rows straight into the attention +# boundary's Lamport one-shot slots (both ranks, the stock push's -0 word rule), and the boundary runs the one-shot AR +# + residual + RMSNorm without its push loop, also writing the MXFP8 of the norm (trtllm::minimax_m31_f3_ar_prepushed, +# the stock quantize's own device functions); the next pinned GENERIC MXFP8 linear (the shared expert / dense +# gate_up) takes it instead of quantizing (_F3_MX). Bitwise to the stock chain. Kill switch: TRTLLM_M31_DISABLE_F3=1. +# TRTLLM_M31_F3_CHECK=1: each F3 boundary also runs the stock GEMM + AR + quantize in the graph and counts byte +# mismatches on the device (reported at exit / on eager steps). +TP2_F3 = not m31_disabled("F3") +# Native CUDA ops for the SwiGLU-OAI activation and the MoE finalize, bit for bit the Triton kernels they replace on +# every band. NATIVE_SWIGLU: trtllm::minimax_m31_swiglu_oai_vec (16-byte vectors, the in-tree swigluOai pair sequence +# verbatim); kill switch TRTLLM_M31_DISABLE_NATIVE_SWIGLU=1. NATIVE_FINALIZE: trtllm::minimax_m31_moe_finalize (the +# Triton kernel's per-element fma chain); kill switch TRTLLM_M31_DISABLE_NATIVE_FINALIZE=1. +NATIVE_SWIGLU = not m31_disabled("NATIVE_SWIGLU") +NATIVE_FINALIZE = not m31_disabled("NATIVE_FINALIZE") +# True serves the native ops on every band, including the few where they measured slightly slower (the dense SwiGLU +# at 88..2048 rows, the finalize at 49..448 and 2K+ rows); False keeps the Triton kernels there. +NATIVE_ALL_BANDS = True +_NATIVE_STATE: dict = {} + + +def _native_op(name: str): + """``torch.ops.trtllm.``, or None when this build lacks it; decided once per process.""" + if name not in _NATIVE_STATE: + op = getattr(torch.ops.trtllm, name, None) + if op is None: + logger.info(f"[M3.1] native op {name} not available: the Triton kernel serves") + _NATIVE_STATE[name] = op + return _NATIVE_STATE[name] + + +def _native_dispatch(item: str, site: str, why: str = "") -> None: + """The backend line of ``item`` (NATIVE_SWIGLU / NATIVE_FINALIZE) at ``site``, once per (item, site, outcome) per + process: "[M3 dispatch] rank r: taken ()" when the native op serves, "... not taken (): " + when the caller's fallback does.""" + outcome = f"not taken ({site}): {why}" if why else f"taken ({site})" + _log_once(logger.info, item, outcome, f"[M3 dispatch] rank {mpi_rank()}: {item} {outcome}") + + +F3_CHECK = os.environ.get("TRTLLM_M31_F3_CHECK", "0") == "1" +F3_MIN_ROWS = 49 # capacity: fixed: the o_proj GEMV serves 1..48 rows (no pinned GEMM there) +F3_MAX_ROWS = 256 # capacity: fixed: the pinned MXFP8 table's 256 bucket (tested at 49..224) +F3_N, F3_K = 6144, 4096 # capacity: fixed: the TP2 o_proj shape the fork is tested on + +# TP2W_S2_TAIL (megakernel stage S2): a generic MoE layer's decode boundary of S2_TAIL_MIN_ROWS..S2_TAIL_MAX_ROWS +# rows runs ONE kernel (trtllm::minimax_m31_s2_moe_tail) for the +# routed finalize + shared add (m3_moe_finalize's math), the one-shot Lamport all-reduce + residual + RMSNorm +# (allreduce_fusion_kernel_oneshot_lamport, its launch configuration) and the next layer's QKV MXFP8 activation +# (mxfp8_quantize's math), instead of three kernels. Microbench: bitwise to that chain at 1..224 rows, -5.6..-6.5 us +# per layer. The activation is consumed only where the QKV projection would run +# mxfp8_quantize + the same GEMM (bit-identical operands). TRTLLM_M31_TP2W_S2_CHECK=1 (debug, never timed) also runs +# the stock boundary with the all-reduce the step would use and logs the bit differences. Enabled by default; +# TRTLLM_M31_DISABLE_TP2W_S2_TAIL=1 forces it off. +TP2W_S2_TAIL = not m31_disabled("TP2W_S2_TAIL") +S2_TAIL_MIN_ROWS = 49 # capacity: fixed: the Triton finalize's rows (past the MC finalize's 48) +S2_TAIL_MAX_ROWS = ( + 224 # capacity: measured: microbench rows (the in-model one-shot Lamport takes up to 256) +) +S2_CHECK = os.environ.get("TRTLLM_M31_TP2W_S2_CHECK", "0") == "1" +# TP2AGG_QKV_A1 (the tp2ctx TP2CTX_QKV_FUSED A1 kernel, kernels/qkv_fusion verbatim): an eager sparse-layer step of +# more than A1_MIN_ROWS rows whose QKV activation CTXNORMQ published runs one CuTe DSL kernel for the QKV + index MXFP8 +# GEMM with the horizontal producer in its epilogue, instead of the GEMM + m3_producer. The producer math is the stock +# producer's on the kernel's own bf16 GEMM output; the GEMM is deterministic but not bitwise against the stock MXFP8 +# GEMM (agg microbench: <= 59 q bytes of 67M differ, one E4M3 step; -88 / -134 / -180 us per call at 8K / 12K / 16K +# rows). Not bitwise, so OFF until its det + GSM8K McNemar gates pass; TRTLLM_M31_ENABLE_TP2AGG_QKV_A1=1 opts in, +# TRTLLM_M31_DISABLE_TP2AGG_QKV_A1=1 forces it off. +QKV_A1 = m31_enabled("TP2AGG_QKV_A1") and not m31_disabled("TP2AGG_QKV_A1") +A1_MIN_ROWS = 1024 # capacity: tp2ctx _CTX_QKV_FUSED_MIN_ROWS (steps above it) +# TP2AGG_SP: the tp2ctx TP2CTX_SP recipe on agg's eager steps of +# 4096 < T <= 16384 rows. Each CTXNORMQ boundary becomes an IPC reduce-scatter of the partial to this rank's owned rows +# (rank 0 [0, S), rank 1 [S, T), S the 128-row boundary nearest T / 2), the residual add + RMSNorm + quantize on the +# owned rows written into the workspace's gather views, then a push of the owned products to the peer; the gathered +# full-T products are published to the CTXNORMQ slots, so the consumers are unchanged. The residual stream is +# owned-only inside such a step and is unsharded before an Eagle3 capture layer and the final norm. At the pre-MoE +# boundary the router logits of the owned rows (ROUTER_ZC, the fp32 input from TP2AGG_ROUTER_F32) are gathered as a +# "logits" kind (128 fp32 = 512 B per row) instead of tp2ctx's top-k, since trtllm-gen routes in-kernel. The workspace +# is catalog/comm/tp2ctx_ipc with the peer ops of libm31_tp2ctx_peer.so, built at weight load (_sp_init, both ranks); +# without them every step keeps the ordinary boundary ("not taken" logged once). Opt-in: TRTLLM_M31_ENABLE_TP2AGG_SP=1; +# TRTLLM_M31_DISABLE_TP2AGG_SP=1 or TRTLLM_M31_DISABLE_AGG_SP=1 disables both boundaries. +# TRTLLM_M31_DISABLE_AGG_SP_PREMOE=1 keeps only post-MoE SP, gathering the residual before pre-MoE. +# The per-item ids stay tp2ctx's (TP2CTX_ROUTER_ZC / _SP_ADDNORM / _SP_OVL / _SP_PMOVL). +TP2AGG_SP = ( + m31_enabled("TP2AGG_SP") and not m31_disabled("TP2AGG_SP") and not m31_disabled("AGG_SP") +) +SP_PREMOE = m31_enabled("AGG_SP_PREMOE") and not m31_disabled("AGG_SP_PREMOE") +SP_MIN_ROWS = 4097 # capacity: eager agg SP band; the workspace also covers the Lamport overlap +SP_MAX_ROWS = 16384 # capacity: max_num_tokens of the agg configs +SP_SF_TILE_ROWS = 128 # rank 1's rows start on a 128-row MXFP8 scale-factor swizzle tile +SP_LOGITS_ROW_BYTES = 128 * 4 # the "logits" gather kind: [T, 128] fp32 +_SP_STATE: dict = {} +# TRTLLM_M31_TP2CTX_SP_CHECK=1 (debug, never timed): each owned reduce-scatter is bit-compared with the all-reduce's +# rows, and the keyed bf16 view (no data) is filled with NaN, so a consumer reading it instead of the products shows. +SP_CHECK = os.environ.get("TRTLLM_M31_TP2CTX_SP_CHECK", "0") == "1" +_SP_LIB = "libm31_tp2ctx_peer.so" +_SP_NORMQ_OUT_LIB = "libm31_ctx_normq_out.so" +_SP_ADDNORM_LIB = "libm31_sp_addnorm.so" +# SP_ADDNORM: the owned-row peer add fused into the NORMQ out= launch (bitwise the two kernels). SP_PMOVL: the pre-MoE +# products pushed by the peer push kernel on a side stream under the router GEMM (scheduling only). SP_OVL: before a +# sparse layer whose QKV runs TP2AGG_QKV_A1, the owned MXFP8 rows go by copy engine on a side stream while A1 runs on +# the own rows (A1 is row-split bitwise; the stock MXFP8 GEMM is not guaranteed, so it waits for the push). +# Each is opt-in (TRTLLM_M31_ENABLE_=1); TRTLLM_M31_DISABLE_=1 wins. +SP_ADDNORM = m31_enabled("TP2CTX_SP_ADDNORM") and not m31_disabled("TP2CTX_SP_ADDNORM") +SP_PMOVL = m31_enabled("TP2CTX_SP_PMOVL") and not m31_disabled("TP2CTX_SP_PMOVL") +SP_PMOVL_MIN_ROWS = 12288 # capacity: tp2ctx measurement (A8K +1.6% below it) +SP_OVL = m31_enabled("TP2CTX_SP_OVL") and not m31_disabled("TP2CTX_SP_OVL") +SP_OVL_MIN_ROWS = 12288 # capacity: tp2ctx _SP_OVL_MIN_ROWS +# GQAMIX: a mixed ctx+gen step's sparse generation rows (MsaDecodeFmha) run the Triton block-sparse decode +# through ops/gqamix_sparse_decode, a verbatim copy of the in-tree kernel module whose two decode kernels do not +# specialize on the step's row count or on the [num_ctx:] slices' alignment, registered on MsaDecodeFmha's +# sparse_decode_hook. Those specializations compiled new variants after the server was ready. Kill switch: +# TRTLLM_M31_DISABLE_GQAMIX=1. +GQAMIX = not m31_disabled("GQAMIX") +# Eager prefill tile selection uses a deterministic score-descending, block-ascending +# tie order. Opt-in: TRTLLM_M31_ENABLE_AGG_TSEL=1; TRTLLM_M31_DISABLE_AGG_TSEL=1 restores the built-in selector. +AGG_TSEL = m31_enabled("AGG_TSEL") and not m31_disabled("AGG_TSEL") +AGG_TSEL_MIN_ROWS = 224 +AGG_TSEL_MAX_BLOCKS = 5856 # the op's shared-memory bound (kMinimaxM3SelectPrefillTileMaxBlocks) +AGG_TSEL_ORACLE = m31_enabled("AGG_TSEL_ORACLE") +# TP2WIDE_PREFILL_ATTN (ported from the tp2ctx target's TP2CTX_SPARSE_OUT / TP2CTX_SCHED_LPT): the NVFP4 sparse +# prefill's MSA combine writes the attention output in place (msa_prefill: msa_sparse_out_in_place; bitwise the copied +# result), and its k2q CSR schedule runs the work items heaviest first (msa_prefill: msa_k2q_schedule_order "lpt"; the +# order only changes which CTA runs which item, never what an item computes or where it writes). Each applies only when +# the loaded MSA build accepts it. With the shared msa_prefill's E4M3X forward (kill switch +# TRTLLM_M31_DISABLE_TP2CTX_E4M3X=1), the agg 16K chunk at a 96K prefix runs 909.7 -> 530.1 us per call, output bitwise +# (microbench). Kill switches: TRTLLM_M31_DISABLE_TP2WIDE_SPARSE_OUT=1, TRTLLM_M31_DISABLE_TP2WIDE_SCHED_LPT=1. +TP2WIDE_SPARSE_OUT = not m31_disabled("TP2WIDE_SPARSE_OUT") +TP2WIDE_SCHED_LPT = not m31_disabled("TP2WIDE_SCHED_LPT") +# NVFP4_STEP_CACHE (AGG_NVFP4_INDEXK): the NVFP4 decode reader's history scheduler is built once per step, not per +# sparse layer (same values). Kill switch: TRTLLM_M31_DISABLE_NVFP4_STEP_CACHE=1. +NVFP4_STEP_CACHE = not m31_disabled("NVFP4_STEP_CACHE") +# TP2WIDE_COMBINE_MXFP8 (ported from the tp2ctx target's TP2CTX_COMBINE_MXFP8, LongAttn): with the in-place +# output, the sparse prefill's MSA combine also writes o_proj's MXFP8 input (msa_prefill: msa_sparse_out_mxfp8_sink; +# bitwise mxfp8_quantize(out, True)) when the loaded build takes o_fp8 / sf_out (the E4M3X / BF16X fork's combine, or +# the old NVFP4 sparse func), and publishes it in the CTXNORMQ slot, so o_proj's prefill MXFP8 linear skips its own +# quantize (the GEMM runs on the same operands). Calls under _COMBINE_MXFP8_MIN_ROWS rows keep the stock quantize. +# Needs CTXNORMQ (the slot's consumer). Kill switch: TRTLLM_M31_DISABLE_TP2WIDE_COMBINE_MXFP8=1. +TP2WIDE_COMBINE_MXFP8 = not m31_disabled("TP2WIDE_COMBINE_MXFP8") + + +def _prefill_attention_opt_ins(backends) -> None: + """Set the MSA prefill opt-ins (TP2WIDE_SPARSE_OUT, TP2WIDE_SCHED_LPT, TP2WIDE_COMBINE_MXFP8) on every layer's + attention backend.""" + for backend in backends: + if TP2WIDE_SPARSE_OUT: + backend.msa_sparse_out_in_place = True + if TP2WIDE_COMBINE_MXFP8 and CTXNORMQ: + backend.msa_sparse_out_mxfp8_sink = _combine_mxfp8_sink + if TP2WIDE_SCHED_LPT: + backend.msa_k2q_schedule_order = "lpt" + + +# DSFOLD: under the standalone DSpark worker, the engine's previous-batch input gathers (input ids, draft tokens, +# position / KV-length offsets) run as one m3_spec_tail.step_gather launch instead of ~a dozen eager index and copy +# launches at the step boundary. Integer copies of the same values into the same buffers; the sampler keeps its +# in-tree slot stores. Kill switch: TRTLLM_M31_DISABLE_DSFOLD=1. +DSFOLD = not m31_disabled("DSFOLD") +CTXNORMQ_MIN_ROWS = ( + 1024 # capacity: NORMQ microbench (GB300): the fused pass wins at 1160 and 16384 rows +) +# -------------------------------------------------------------------------------------------------------------------- +# Row ranges +# -------------------------------------------------------------------------------------------------------------------- + +# The largest generation step the decode target takes: 8 requests verifying 1 + 3 drafts and their padding rows. +# R1VR8 (the sm_107 TP8 target's row-bound kill, kept at TP2): the decode layer takes pure generation steps up to +# DECODE_MAX_TOKENS rows. TRTLLM_M31_DISABLE_R1VR8=1 restores the TP2-lineage 8-row bound (TP2_ROWS): every wider step +# runs the generic layers (and VR_BANDS is off with it), as at TP8. Read once, at import. +TP2_ROWS = 8 # capacity: fixed: the TP2-lineage paths' certified rows +R1VR8 = not m31_disabled("R1VR8") +DECODE_MAX_TOKENS = 48 if R1VR8 else TP2_ROWS # capacity: fixed: the decode kernels' step bound +# VR-BANDS (MIDCONC): the decode step bound is 64 rows (8 requests verifying 1 + 7 drafts, the K7 c8 step) while every +# op keeps its own certified row range (DECODE_MAX_TOKENS and the per-op tables; the shard accept keeps its 48 rows): +# an op past its range runs its generic form and the rest of the step stays on the decode layer. Steps of 49..64 rows +# take the "wide" plan (the built-in all-reduce at both boundaries). TRTLLM_M31_DISABLE_VR_BANDS=1 restores the +# 48-row step bound. Read once, at import. +VR_BANDS = R1VR8 and not m31_disabled("VR_BANDS") +TP2WIDE_DECODE = VR_BANDS and not m31_disabled("TP2WIDE_DECODE") +DECODE_STEP_MAX = ( + 448 if TP2WIDE_DECODE else 64 if VR_BANDS else DECODE_MAX_TOKENS +) # capacity: fixed: the step bound; ops keep their own rows +# SWQ (the sm_107 TP8 lever, ported): on the shared-expert down projection's rows the decode GEMV does not serve +# (SHARED_DOWN (6144, 1536) at 17..32 rows, MX_GEMV_ROWS below) the SwiGLU-OAI writes the down projection's MXFP8 +# activation directly (ops/activation/m3_swq.py: bitwise to silu_and_mul + mxfp8_quantize on every block that quantize +# emits finite; a scale-code-0 block, which it emits as inf / NaN, is kept finite), taken by ``_linear``'s prequant +# branch into the pinned MXFP8 GEMM. Compiled before warmup (prebuild). Kill switch TRTLLM_M31_DISABLE_SWQ=1 (the bf16 +# SwiGLU + quantize). Read at import. +SWQ = not m31_disabled("SWQ") +SWQ_SHARED_DOWN = (6144, 1536) # the shared expert's down projection at the TP2 width 1536 +SWQ_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + {SWQ_SHARED_DOWN: frozenset(range(17, min(32, DECODE_MAX_TOKENS) + 1))} if SWQ else {} +) # capacity: fixed: the shared down's rows past its GEMV (17..32) +_SWQ_LOGGED: set = set() + + +@dataclass(frozen=True) +class DecodePlan: + """The layer-boundary collectives of one decode token bucket (GB300 tp4 measurements in the comments).""" + + name: str + max_tokens: int + # The attention boundary's multicast all-reduce + residual + RMSNorm (5.05 vs 5.27 us native at 1 token). + attn_mc_ar: bool + # The same kernel at the MoE boundary with the shared + routed add folded in: wins up to 48 tokens. + moe_mc_ar: bool + # The MoE-boundary all-reduce also emits the next QKV activation in MXFP8 for the decode GEMV / GEMM. + qkv_prequant: bool + + +# tiny: every step up to 8 tokens (c1..c8 without speculation, the c1 verify, every draft pass); small: the c2 +# verify; verify: the c4 / c8 verifies. Each is one multicast all-reduce bucket (kernels/m3_mc_allreduce.BUCKETS). +# SM 10.7 TP4: one bucket. MC_AR_TP4 puts every decode boundary on the multicast all-reduce + residual + RMSNorm +# (trtllm::minimax_m3_sm107_tp4_mc_allreduce_norm, its row kernel: bitwise to the stock TP4 oneshot all-reduce fusion +# on the residual and the normed rows) with the MXFP8 copy for the next QKV; False = the built-in all-reduce. +MC_AR_TP4 = True +# Up to this many rows the decode all-reduces (not the MoE finalize) take the 8-CTA cluster kernel instead of the row +# kernel: the same stock oneshot order (opts bit 32: per-column sum of squares, warp butterflies, 6-warp groups, 4 group +# sums from 0.f), so bitwise the row kernel's output, with each row's exchange spread over 8 CTAs. Rubin TP4 synth, both +# orders: c1 (4 rows) -12, c2 (8 rows) -38 us/step, but c4 / c8 (16 / 32 rows) +5 / +45, where +# 8x the CTAs per row cost more than they save. 0 = the row kernel at every row count. +# TP2 port (world-2 microbench): oneshot_cluster off at every row count (bit 32 is 0.4 us slower at world 2); +# the call still runs the 8-CTA cluster kernel because the TP4 op's row kernel takes world 4 only. +MC_AR_CLUSTER_MAX_ROWS = 0 # capacity: fixed: TP2: no oneshot-cluster rows +# Every decode / drafter boundary all-reduce launches with the max-smem carveout preference (the cluster kernel's +# option 8 at every row count, the row kernel's carveout instance), so the next kernel's CTAs (the QKV GEMV's staged +# tiles) can co-reside early. Launch attribute only: bitwise unchanged. Kill: False (the GB option table). +MC_AR_CARVEOUT = True +# The fused draft loop's boundary all-reduces (M3Eagle3Worker._m3_draft_ar_norm) launch WITHOUT that carveout: the +# kernel after the drafter's last boundary is the draft step (one CTA per SM, ~145 KB of shared memory) rather than a +# staged GEMV, and TP4 alone of the sm_107 targets ran its draft pass ~74-80 us in-model vs ~57 us standalone at world 4 +# in the step census. Launch attribute only: bitwise unchanged. Kill: MC_AR_CARVEOUT (the shared setting). +DRAFT_MC_AR_CARVEOUT = False +# The fused draft loop's two all-reduce boundaries per pass (o_proj -> post-attention norm, down -> final norm) of up +# to this many rows run the multicast all-reduce + residual + RMSNorm's row kernel in its drafter form (opts bit 16: +# FlashInfer fused_add_rmsnorm's reduction order on the unrounded FP32 sum, so bitwise the drafter's stock plain +# all-reduce + RMSNorm) instead of the built-in all-reduce and the separate norm. Kill: 0. +DRAFT_MC_AR_MAX_ROWS = ( + 8 if MC_AR_TP4 else 0 +) # capacity: fixed: the drafter boundary kernel's certified rows +# Those drafter boundaries run the 8-CTA cluster kernel's drafter form (opts bit 128; TP8's drafter boundary kernel) +# instead of the 1-CTA row kernel. Exact-class: deterministic, the updated residual bitwise, the RMS sum of +# squares in the cluster's order instead of FlashInfer's, so a normed row can differ by one BF16 rounding step (7/64 +# rows at 1 row, 14/256 at 4). Rubin world 4, both orders: 4.39 -> 3.75 us per boundary at 1 row, +# 4.79 -> 4.14 at 4 rows, ~ -2..-3.5 us/step. Kill: False (the row kernel, bitwise the stock boundary). +# OFF: in-model timing was flat and natural acceptance length was grey (lower bound -1.16% vs the -1% line). +DRAFT_MC_AR_CLUSTER = False +# The drafter's o_proj and down projection, on the copied split-K kernel (kernels/m3_lowm_splitk), also push their rows +# into the drafter boundary all-reduce's exchange (the row kernel's slots: the canonical words its own push would store, +# at the (token, 768-column slice) call parity), which then runs prepushed 2 instead of pushing x itself; the copy +# releases its dependents only after its own grid wait and the pushes (the prepushed >= 2 contract). Bitwise. The TP8 +# port: ~ -4 us/step at world 8. Kill: False. +DRAFT_AR_PUSH = True +# The drafter boundary all-reduce fed by its projection's push (DRAFT_AR_PUSH): its own taken / not-taken lines. +DRAFT_AR_PUSH_PATH = "m3_draft_ar_push" +# TP2 port: the M3 sm_107 tp2 target's row-bucket skeleton (tiny 1..8 / b16 9..16 / b48 17..DECODE_MAX_TOKENS, the +# multicast all-reduce's workspace buckets) with the TP4 base's one setting on every bucket, so the plan (and every +# boundary) is the TP4 base's at every row count; the TP2 WIDE stage sets each bucket's flags / cells on its own. +TP2_BUCKETS = tuple( + b for b in (8, 16, 48) if b <= DECODE_MAX_TOKENS +) # capacity: fixed: the plan buckets (MC buckets) +DECODE_PLANS = tuple( + DecodePlan(name, rows, attn_mc_ar=MC_AR_TP4, moe_mc_ar=MC_AR_TP4, qkv_prequant=MC_AR_TP4) + for name, rows in zip(("tiny", "b16", "b48"), TP2_BUCKETS) +) +if DECODE_STEP_MAX > DECODE_MAX_TOKENS: + # VR-BANDS: 49..64-row steps (K7 c7..c8) with the built-in all-reduce at both boundaries. + DECODE_PLANS += (DecodePlan("wide", 64, attn_mc_ar=False, moe_mc_ar=False, qkv_prequant=False),) + +if TP2WIDE_DECODE: + # Separate bucket preserves every existing plan through 64 rows. Each op keeps its own dispatch range. + DECODE_PLANS += ( + DecodePlan("wide448", 448, attn_mc_ar=False, moe_mc_ar=False, qkv_prequant=False), + ) + +# The dense layers' decode attention (trtllm-gen, msa_decode.dense_decode_max_seq_len): the max_seq_len it is handed +# per request count. It only sizes the multi-CTA KV split (every CTA still covers its share of the actual length; the +# result changes only by the split-merge order), and the model maximum over-splits long-context caches at one +# request: GB300 TP4, 16 heads, FP8 KV, 64K context: 15.1 us at 16K vs 28.6 us at 1M. Two requests are neutral (the +# grid is unchanged) and keep the step's bound. +# The generic layer's boundaries (prefill, mixed steps, Eagle3 capture layers): the most rows that take the multicast +# all-reduce + residual + RMSNorm, alone and with the shared + routed add folded in. TP4 skeleton: none (the built-in +# all-reduce) until the multicast all-reduce is certified on Rubin. +GENERIC_MC_AR_MAX_TOKENS = 0 # capacity: fixed: off at TP4 (the built-in all-reduce) +GENERIC_MC_AR_FOLDED_MAX_TOKENS = 0 # capacity: fixed: off at TP4 (the built-in all-reduce) +# An Eagle3 capture layer whose capture norm is armed (CAPTURE_NORM) has its fused feed-forward boundary all-reduce +# (the 8-CTA cluster kernel up to MC_AR_CLUSTER_MAX_ROWS, the row kernel above) also write the capture: +# rmsnorm(updated residual) * fc_norm[i], bit for bit the capture_norm kernel, which then does not run. The MoE +# finalize runs the shared GB300 op and keeps the capture_norm kernel. Wired at the generic layer's fused boundary +# (layers.M3DecoderLayer._feed_forward_boundary) and through M3DecodeTarget._layer's boundaries (``capture``) for a +# capture layer on the fast decode layer. Kill: False (the capture_norm kernel after the boundary). +CAPTURE_NORM_IN_BOUNDARY = True + +# TP4 skeleton: no cap (the built-in split) until a TP4 SM 10.7 cap is measured and certified. +DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH: Mapping[int, int] = MappingProxyType({}) + +# The dense layers' decode attention on PrimTS split-KV decode (kernels/m3_fullctx_decode) instead of the trtllm-gen +# call, per (query rows per request, requests): the Q tile (``config``) and the split-KV fanout (``splits``) with the +# BF16 merge. At TP4 (16 query heads, 1 KV head per rank) the stock split is capped at 16 cluster CTAs per (request, +# KV head), i.e. 16 of 212 CTAs at one request: stock dense fmhaSm107a 33.4-33.8 us per call and the stock drafter +# fmhaSm100f 45.4 us per pass at 165k. Rubin TP4, FP8 KV, 165k, chained us per call +# (microbench), every recipe passing the fp32 oracle: (4, 1) 13.70 (stock 33.6), (1, 1) 11.52. At one +# request the K/V stream is latency-bound, so both deepen the K/V load pipeline (the load pipeline only; the same +# bits): 12 stages at one query row, 8 at four (the long-context sweep: (4, 1) 12.95 / 17.00 / 20.98 us +# per call at 165k / 323k / 512k with 8 stages vs 13.03 / 17.15 / 21.30 with 12; (1, 1) is fastest with 12). Two +# requests (microbench; stock at c2: dense fmhaSm107a 26.5 us, drafter fmhaSm100f 44.0-44.6 us per pass, +# census): (4, 2) 17.34, (1, 2) 14.16. +# K/V stages in flight of the (4, 1) recipe (12 was the previous depth; the same bits at any depth). +FULLCTX_DECODE_Q4_KV_STAGES = 8 # capacity: fixed: a pipeline depth, not a size +FULLCTX_DECODE_RECIPES: Mapping[Tuple[int, int], Mapping[str, Any]] = MappingProxyType( + { + (4, 1): MappingProxyType( + { + "config": { + "tile_size_q": 32, + "groups_tokens_heads_q": True, + "kv_stages": FULLCTX_DECODE_Q4_KV_STAGES, + }, + "splits": 96, + } + ), + (1, 1): MappingProxyType({"config": {"kv_stages": 12}, "splits": 108}), + (4, 2): MappingProxyType( + {"config": {"tile_size_q": 32, "groups_tokens_heads_q": True}, "splits": 48} + ), + (1, 2): MappingProxyType({"config": {}, "splits": 80}), + } +) # capacity: fixed: pinned recipes per certified (rows, requests) +# Kill constant: False puts the dense layers back on the trtllm-gen call at every shape. +FULLCTX_DECODE_ENABLED = True +# The dense layers' FP8 q/k/v (decode steps of the hybrid cache): the op minimax_m3_sm107tp4_dense_qk_norm_rope_fp8 +# (head-count generic) rounds Q/K to BF16 and then to E4M3 in its store, the bits of the BF16 fused_qk_norm_rope output +# cast to FP8. False (or a build without the op) keeps the single-rounding fused_qk_norm_rope FP8 output. +DENSE_QKV_FP8_VIA_BF16 = True +# On a full-context decode step that op also stores the new K/V rows into the layer's paged pool (the bytes +# write_msa_phase_kv's indexed copies store, one launch instead of about ten). False writes them with +# write_msa_phase_kv. +DENSE_KV_INSERT_FUSED = True +# The engagement line of that fused store (taken, one per row count and rank). +DENSE_KV_INSERT_PATH = "m3_dense_kv_insert" +# The drafter MLP's C++ SwiGLU (DRAFT_SILU_CPP, M3Eagle3Draft.mlp): its own taken / not-taken lines. +DRAFT_SILU_PATH = "m3_draft_silu" +# The split-KV merge on the BF16 op (minimax_m3_sm107tp4_fullctx_reduce, head-count generic) instead of PrimTS's FP16 +# reducer and a cast. +FULLCTX_DECODE_OWN_REDUCER = True +# The merge also writes the MXFP8 copy of its BF16 rows (E4M3 + 128x4-swizzled UE8M0, bit for bit +# trtllm::mxfp8_quantize) into the MERGE_MXFP8 buffers and publishes it to m3_prequant.SLOT, so the dense layer's +# MXFP8 o_proj skips its own quantization. Kill: False leaves the o_proj to quantize its BF16 input. +FULLCTX_DECODE_MXFP8 = True +FULLCTX_DECODE_MXFP8_PATH = "m3_fullctx_merge_mxfp8" +# The K/V length bound the full-context plans are compiled for; a step whose cache length bound is longer is a miss. It +# covers the long-context serving config's max_seq_len (1 << 20) plus the speculative rows the cache reserves past it. +FULLCTX_DECODE_MAX_KV_LEN = ( + 1 << 21 +) # capacity: fixed: the plans' KV-length bound (1M context + spec rows) +# The Eagle3 drafter layer's generation phase on the same full-context decode (kernels/m3_drafter_attn: the in-tree +# RoPE + FP8 K/V append, then the pinned PrimTS recipe) instead of thop.attention's in-tree runner, which caps its +# split at 8 CTAs per (request, KV head) on SM 10.7: 45.4 us per draft pass at 165k. Kill constant. +DRAFTER_FULLCTX_ENABLED = True +# The standalone DSpark drafter's attention on FlashInfer's trtllm-gen library (kernels/m3_drafter_attn +# M3DrafterSlidingFiFmha): its SM100f Q8 SwapsAb decode kernel instead of thop.attention's in-tree SM107a Q64 KeepsAb +# one (4.0 vs 9.7 us per call, 5 calls per step at c1). A different kernel, so not bitwise the in-tree output. +# Kill switch: TRTLLM_M31_DISABLE_VR_DRAFT_FMHA_SM100F=1 (the in-tree runner), read at import. +# Installed only where the drafter's decode reads its attention modules' FMHA list. The standalone DSpark drafter's +# block decode (DFlash) calls its own attention ops (FlashInfer trtllm_batch_decode_with_kv_cache for TRTLLM), so the +# library installed there did no work (no VR_DRAFT_FMHA_SM100F count at TP2; a TP8 kill A/B was flat); +# the prebuild logs "not installed" with that reason instead (the lever needs the drafter's own call site: deferred). +VR_DRAFT_FMHA_SM100F = not m31_disabled("VR_DRAFT_FMHA_SM100F") +# On those passes the drafter's RoPE, FP8 query and FP8 K/V append run as one launch (the op +# minimax_m3_sm107tp4_drafter_rope_fp8_kv: the in-tree preprocess's GPT-NeoX rotation and unit-scale E4M3 store, and +# the torch E4M3 cast of its BF16 query) instead of thop.trtllm_gen_generation_preprocess and the cast. Kill constant. +DRAFTER_ROPE_FUSED = True +# The engagement line of that launch (taken, one per row count and rank). +DRAFTER_ROPE_PATH = "m3_drafter_rope" +# thop.trtllm_gen_generation_preprocess builds its workspace token map only for several requests, yet a one-request +# multi-token generation (the drafter's first pass at one request, any stock trtllm-gen fallback) places its rows by +# that map: without a rebuild it applies the last multi-request call's map (rows on another request's length and pages, +# token-0 drafts). kernels/m3_tokmap_guard writes the one-request map (0, t) into the op's map slice right before such +# calls: what the build writes (padded rows included), so bitwise the op with a fresh map. Kill: False. +PREPROCESS_TOKMAP_FIX = True +PREPROCESS_TOKMAP_PATH = "m3_preprocess_tokmap" +# How the rotation's products are summed, as the in-tree preprocess build sums them (0: two products then the add, +# 1: fma(cos, x, sin * pair), 2: fma(sin, pair, cos * x)); the op's bitwise test pins it. +DRAFTER_ROPE_MODE = 1 # capacity: fixed: a launch mode, not a size + +# The sparse layers' NVFP4 QKV + indexer producer (ops/attention/m3_producer, every step its geometry fits) triggers its +# dependent kernel on entry, so the index scorer launches and stages its pre-wait loads while the producer runs (its +# outputs are read only after the scorer's own grid wait). +PRODUCER_EARLY_TRIGGER = True +# Steps of up to this many rows keep the producer's trigger at its exit instead. On Rubin TP4 the early-launched scorer +# (its grid doubles with the requests, its pre-wait index-K prefetch running) is resident through the producer's whole +# post-QKV segment and slows the producer's dependent loads; from 16 rows the overlap pays. In-model, locked memory +# clock, tactic- and FlashInfer-pinned, A and B in one job: -75 / -176 us per decode step at 1 / 2 requests, +# +67 at 4 with the trigger at the exit. Timing only: the scorer reads the producer's outputs after its own grid wait. +# Kill: 0 (the early trigger at every row count). Set to 0 on the shipped stack: with the long-row select's late release +# and post-load trigger, the scorer's pre-wait overlap with the producer is worth more than the producer's own +# segment (both orders, double-pinned: -12 / -42 us at 1 / 2 requests with 0). +PRODUCER_LATE_TRIGGER_MAX_ROWS = 0 # capacity: fixed: measured trigger crossover (0: off) +# The producer reads its per-step inputs (position ids, cache slots, RoPE rows) before its grid wait, leaving only the +# packed QKV row after it: this target's copy of the producer (trtllm::minimax_m3_sm107_tp2_producer_pw, +# minimaxM3Sm107Tp4/producerPw4.cu, the TP2 producerPw2 at 16 query heads). Their writers are host-to-device copies +# before the replay and plain-launched kernels earlier in the step (the input preprocessing, msa_kv_update or its torch +# fallback), so they are final before the producer can start; a PDL-launched writer of any of them would have to turn +# this off. The call site's packed.contiguous() and position-id copies are plain launches too. Bit-identical outputs. +# Kill: False (the shared minimax_m3_producer_pw at 16 query heads, which reads them after its wait). +PRODUCER_PREWAIT_STEP_INPUTS = True +# QKV_PRODUCER_FOLD (the sm_107 TP8 target's #117 at the TP2 heads): on a decode step of at most 8 rows whose QKV GEMV +# runs split 8 (``mx_gemv_cfg``) the sparse layer's QKV + index GEMV runs the producer in its owner epilogue +# (kernels/mx_gemv m3_mx_gemv_producer, kernels/mx_gemv/phase_producer.producer_head): one launch and one PDL edge +# fewer per sparse layer, bit for bit the GEMV + m3_producer chain (q, index-q and every cache byte). The QKV linear +# defers its GEMV (``_linear``'s ``defer``: a ``_DeferredGemv``) and the producer call site runs the fused kernel, or +# the plain GEMV when the producer does not take the step. TP2 c1 microbench (60-layer graphs, cold L2, 2 processes): +# producer removed -1.358 us/layer, fold epilogue +1.052, net -0.306 us/layer. Kill switch +# TRTLLM_M31_DISABLE_TP2_QKV_PRODUCER_FOLD=1 (the GEMV, then m3_producer), read at import. +QKV_PRODUCER_FOLD = not m31_disabled("TP2_QKV_PRODUCER_FOLD") +QKV_PRODUCER_FOLD_PATH = "m3_qkv_producer_fold" + +# The fused embedding + first RMSNorm kernel: one owner per row count and index dtype. +EMBED_FIRSTNORM_MAX_TOKENS = 8 # capacity: fixed: built rows; wider steps embed generically +# The o_proj pushes its rows into the attention boundary's multicast all-reduce (prepushed mode 2) up to this many +# tokens; wider steps run the GEMV and a plain multicast all-reduce. +ATTN_PUSH_MAX_TOKENS = 8 if MC_AR_TP4 else 0 # capacity: fixed: measured push crossover +# The dense layers' MLP down projection pushes its rows into the post-MLP boundary's multicast all-reduce (prepushed +# mode 2), as the o_proj does into the attention boundary (TP2: those hops end 5.5-5.8 us after the down +# GEMV unpushed vs 2.4-2.5 us pushed). Bitwise equal to the unpushed boundary (the same bf16 words). Kill: False. +DENSE_PUSH_TP4 = True +DENSE_PUSH_MAX_TOKENS = ( + 8 if (MC_AR_TP4 and DENSE_PUSH_TP4) else 0 +) # capacity: fixed: the GEMV push rows +AR_PUSH_MODE = 2 # capacity: fixed: an exchange mode, not a size +# The 1..8-token MoE engine (router, routed experts and the shared expert in one launch). It pushes its sum into the +# MoE boundary's multicast all-reduce (prepushed mode 2) on the rows MOE_PUSH covers (with MC_AR_TP4). +MOE_ENGINE_MAX_TOKENS = 8 # capacity: fixed: the engine's built rows +MOE_PUSH_MAX_TOKENS = ( + MOE_ENGINE_MAX_TOKENS if MC_AR_TP4 else 0 +) # capacity: fixed: the engine's rows +# The 9..13-row engine (router, routed and shared experts in one kernel); its bf16 sum goes to the boundary. +# MOE16_TP2: the 9..13-row engine is the TP4 kernel, built and certified at the TP4 expert +# width (768); at the TP2 width (INTER 1536) it is untested, so OFF in the baseline: 9..13-row steps run the generic +# MoE inside the decode layer (no engine is built, MOE16_ENGINE covers no rows). TRTLLM_M31_ENABLE_MOE16_TP2=1 opts in +# (after its UT at 1536 and the natural-acceptance gate); TRTLLM_M31_DISABLE_MOE16_TP2=1 always wins. Read at import. +MOE16_TP2 = m31_enabled("MOE16_TP2") and not m31_disabled("MOE16_TP2") +# The width the 9..13-row engine's kernel is compiled for (kernels/m3_moe/m3_moe16_kernel.py I_TP / SH_I, SH_SPLIT 2): +# an opted-in build at another expert width fails (the generic MoE runs) until a TP2-width kernel exists. +MOE16_KERNEL_WIDTH = 768 # capacity: fixed: the TP4-width build of m3_moe16_kernel +MOE16_MIN_TOKENS = 9 # capacity: fixed: the 9..13-row engine's lower bound +MOE16_MAX_TOKENS = 13 if MOE16_TP2 else 0 # capacity: fixed: built rows (0: off at TP2) +# From this many tokens the MoE runs unfinalized through the TRTLLM-Gen runner and its finalize (top-k gather, +# routing weights, shared expert add) runs inside the boundary all-reduce. 9..13 take it only when their engine is +# missing (outside require mode). TP4 skeleton: no rows (the built-in MoE finalize and all-reduce). +# Fold the unfinalized MoE output into the boundary all-reduce within its certified 48-row bucket. +# The parent kill switch disables the fold at every row count. +TP2W_MC_FINALIZE = not m31_disabled("TP2W_MC_FINALIZE") and DECODE_MAX_TOKENS >= 48 +# Include rows past the MoE engines by default; the small-row kill restores the 33-row lower bound. +TP2_MC_FINALIZE_SMALL = not m31_disabled("TP2_MC_FINALIZE_SMALL") +# capacity: fixed: past the MoE engines' rows (or the WIDE rows' 33 with the kill) .. the finalize op's 48-token bucket +MC_FINALIZE_MIN_TOKENS = ( + (max(9, MOE16_MAX_TOKENS + 1) if TP2_MC_FINALIZE_SMALL else 33) if TP2W_MC_FINALIZE else 9 +) +MC_FINALIZE_MAX_TOKENS = 48 if TP2W_MC_FINALIZE else 0 +# TP2W_MOE_FINALIZE (TP2 WIDE; port of the M3 tp2 target's MOE_TRITON_FINALIZE): a finalized MoE forward of at least +# MOE_TRITON_FINALIZE_MIN_TOKENS rows in one runner call (the decode "wide" plan's 49..64 rows, the generic layers past +# them) takes the trtllm-gen runner's FC2 output unfinalized and kernels/m3_moe_finalize finalizes it with the shared +# expert added, bitwise to the runner's finalize + shared.add_ (CTX microbench at 48..192 rows, dropped slots, graph == +# eager; 8.3 -> 3.1 us per layer at 64 rows). Kill switch TRTLLM_M31_DISABLE_TP2W_MOE_FINALIZE=1 (the runner finalizes). +TP2W_MOE_FINALIZE = not m31_disabled("TP2W_MOE_FINALIZE") +# Reuse the CTXNORMQ boundary's NVFP4 product for ordinary BF16 prefill rows instead of re-quantizing it. +# SP keyed views always consume the slot, independently of this switch, because they contain no BF16 payload. +# The debug oracle skips graph captures and SP keyed views. Kill: TRTLLM_M31_DISABLE_TP2W_FP4SLOT_FINALIZE=1. +TP2W_FP4SLOT_FINALIZE = not m31_disabled("TP2W_FP4SLOT_FINALIZE") +_FP4SLOT_CHECK = os.environ.get("TRTLLM_M31_TP2W_FP4SLOT_CHECK", "0") == "1" +_FP4SLOT_STATE = {"taken": 0, "checked": 0, "diff": 0} +MOE_TRITON_FINALIZE_MIN_TOKENS = 49 # capacity: fixed: past the MC finalize's 48 rows +# A greedy Eagle3 verify step of up to this many requests verifies with the fused shard argmax + acceptance kernel +# over the target lm_head's BF16 shard logits (kernels/m3_shard_accept), at the configured draft length. +SHARD_ACCEPT_DRAFT_LEN = 3 # capacity: fixed: the built Eagle3 draft length +SHARD_ACCEPT_MAX_REQUESTS = DECODE_MAX_TOKENS // ( + SHARD_ACCEPT_DRAFT_LEN + 1 +) # capacity: fixed: decode rows / (draft length + 1) +# A sampled (temperature + top-p, advanced_sampling_mode=no_topk) Eagle3 verify step of up to this many requests +# verifies with the fused sampled acceptance (kernels/m3_sampled_accept) over this rank's logits shard: every request +# count a decode step holds. +SAMPLED_VERIFY_MAX_REQUESTS = SHARD_ACCEPT_MAX_REQUESTS # capacity: fixed: a decode step's requests +# The sampled verify runs the sm_107 tp4 copy of the TP2 sampled-accept kernel +# (trtllm::minimax_m3_sm107tp4_sampled_accept: a row's CTAs launch as one thread-block cluster, bounded record polls). +# Same records and tokens. Kill: False (the shared kernel, trtllm::minimax_m3_sampled_accept: every CTA of the grid +# co-resident, unbounded polls). +SAMPLED_ACCEPT_SM107TP4 = True +# The sm_107 tp4 sampled verify loads each row's sampling parameters (temperature, top-p, Philox seed and offset: step +# inputs, not the head's output) before its PDL dependency wait, overlapping the head GEMM's tail; the same values and +# tokens. Kill: False (they load after the wait; the shared kernel always loads them after). Rubin TP2 microbench: +# verify end 20.48 -> 19.97 us at top-p 0.95. +SAMPLED_ACCEPT_PREWAIT_PARAMS = True +# The sm_107 tp4 sampled verify releases its dependents (the first draft pass's input) right after its dependency wait +# instead of at the acceptance epilogue: the dependent launches early and parks at its own griddepcontrol.wait, which +# returns only once the verify has completed, so the values are the same; it saves the launch latency after the verify +# (TP2: draft_input started 2.0-2.4 us after the verify / fc join). Kill: False. Off until its sampled-column +# in-model pairs read. +SAMPLED_ACCEPT_EARLY_TRIGGER = False +# SA-1R: the sm_107 tp4 sampled verify also multicasts each CTA's largest logits with its round-0 record (82: each +# thread's top 2 pooled, the CTA's top 8 plus the bound of everything it does not list; 9: an exact top-9 list), so a +# row whose first candidate is provably in the nucleus decides in round 0 instead of exchanging a second round. The +# same tokens, accepted counts and Philox draws as the two-round kernel, bit for bit (a fp32 margin m * 2^-21 * S +# covers the summation order; any doubt runs round 1). The prototype was exact on 39,968 rows; real +# long-context serving logits decide all 4 rows in round 0 on 79.5% (82) / 80.8% (9) of steps. Off (0, the two-round +# kernel): exact, but the untimed decode step reads +47 / +65 us slower with it on at TP4 (the earlier dependent +# launch it enables costs more than the saved round). +SAMPLED_ACCEPT_1R_K = 0 # capacity: fixed: a kernel variant, not a size +# The sm_107 tp4 sampled verify's owner walk (the one thread that finds its row's drawn token by a serial +# prefix over its strided tokens) takes those tokens 8 at a time: their weights are computed independently, then +# summed in the same order with the same early exit, so the draw, tokens and Philox draws are the one-token walk's, +# bit for bit. Rubin TP4 bench (one request, top_p 0.95): -1.06 / -1.07 us per call at SA-1R K 0 / 82. Kill: False +# (the one-token walk, the stock instantiation; the shared kernel has no 8-token walk). +# Off until its in-model TP4 pair passes. +SAMPLED_ACCEPT_ILP_WALK = False +# A step verified by the in-tree path takes the fused first drafter inputs (the capture norms and the first pass's +# gather) up to this many requests: every request count a decode step holds. +FIRST_DRAFTER_INPUTS_MAX_REQUESTS = SHARD_ACCEPT_MAX_REQUESTS # capacity: fixed: step requests +# A fused-verify step of up to this many verify rows (one or two requests at draft length 3) computes the target +# logits shard from the 12-bit coded copy of the BF16 head (kernels/m3_target_head; the same BF16 x BF16 -> fp32 +# arithmetic, 75% of the bytes) instead of the in-tree head GEMM; wider verifies keep the in-tree GEMM. +# Off at TP4 (as at TP2): at the TP4 shard (50,016 x 6144) the in-tree BF16 cuBLAS head is faster than the coded kernel, +# 47.9 vs 56.8 us at 4 and 8 rows (TP2: 93 vs 109). True re-enables the coded head. +TARGET_HEAD_CODED = False +TARGET_HEAD_MAX_ROWS = ( + TARGET_HEAD_KERNEL_MAX_ROWS if TARGET_HEAD_CODED else 0 +) # capacity: fixed: the coded head's built rows (off at TP4) +# On a fused-verify step of up to this many verify rows the drafter's first fc is forked onto the aux stream right +# after the last captured target layer, overlapping the remaining target layers and the head (M3Eagle3Worker). +FC_EARLY_MAX_ROWS = 0 # capacity: fixed: measured fork crossover (0: off) +# A sampled fused verify of up to this many verify rows on the in-tree head GEMM forks the drafter's first fc after the +# head instead of beside it (M3Eagle3Worker), as the coded head's verify already does: the fc's split-K grid otherwise +# holds the SMs first and the head GEMM starts late (TP2: 17.4 us late). Kill: 0. +FC_AFTER_HEAD_MAX_ROWS = 8 # capacity: fixed: measured fork-after-head crossover +# A decode-only Eagle3 step of up to this many requests drafts with the fused draft loop (M3Eagle3Worker): per pass the +# draft input in one launch (kernels/m3_spec_tail.draft_input), the drafter layer's in-tree attention, MLP and +# all-reduces, and one kernel for the draft head (its 12-bit coded copy, or the BF16 fragments when the coding does +# not decode bit for bit), the cross-rank argmax and the pass's bookkeeping (kernels/m3_draft_step), at the +# configured draft length. The head kernel takes one row per request, at most 8. +# The fused draft loop streams the 12-bit coded copy of the draft head (when it decodes bit for bit); False keeps the +# BF16 fragments. +DRAFT_HEAD_CODED = False +DRAFT_STEP_DRAFT_LEN = SHARD_ACCEPT_DRAFT_LEN # capacity: fixed: the built draft length +DRAFT_STEP_MAX_REQUESTS = DRAFT_STEP_KERNEL_MAX_TOKENS +# The Eagle3 capture layers (decoder layers whose output the drafter reads) run the fast decode layer, with its +# o_proj / MoE / dense-down pushes into the boundary all-reduces, and capture the MoE / dense boundary's reduced +# residual sum (the same BF16 words the generic layer's fused boundary captures). Kill: False (the generic layer, +# no boundary pushes). Rubin TP2: the capture layers' hops end 4.4-5.3 us after their producing +# GEMV unpushed vs 2.4-2.5 us pushed. +CAPTURE_LAYERS_FAST = True +# An Eagle3 capture layer (one with a next norm) ends at the fused boundary (all-reduce + residual + next RMSNorm in +# one kernel) and captures that kernel's reduced residual sum. Kill: False keeps the feed-forward module's own +# all-reduce, captures hidden + residual and runs the next norm separately, as the built-in model does; it also +# turns CAPTURE_LAYERS_FAST off, which captures the fused boundary's sum. +CAPTURE_FUSED_BOUNDARY = True +# The Eagle3 drafter's BF16 linears (qkv, o_proj, gate_up, down, fc) run FlashInfer's fixed-tactic low-M kernels +# (ops/gemm/m3_draft_lowm, the TP2 target's port with the TP4 table) at the rows its table is certified for, up to this +# many rows; other calls keep the in-tree cuBLAS GEMM (split-K + reduce). Kill: 0. Rubin TP4 sweep (locked memory +# clock) at 1 row: qkv 8.15 -> 5.90, o 5.36 -> 3.40, gate_up 12.93 -> 9.93, down 7.68 -> 6.16 us. +DRAFT_LOWM_MAX_ROWS = 4 # capacity: fixed: the low-M tactic table's certified rows +# The drafter MLP's activation (the plain SwiGLU of its in-tree GatedMLP) runs this target's C++ op +# (ops/spec/m3_draft_silu, minimaxM3Sm107Tp4/siluAndMul.cu) instead of the Triton silu_and_mul: bit for bit, one launch +# either way (no Triton on the generation decode path). The fused draft loop's MLP (M3Eagle3Draft.mlp) routes the +# call; the in-tree draft path keeps the Triton op. Kill: False. +DRAFT_SILU_CPP = True +DRAFT_LOWM_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = { + nk: frozenset(m for m in rows if m <= DRAFT_LOWM_MAX_ROWS) + for nk, rows in DRAFT_LOWM_TACTIC_ROWS.items() +} +# The identity-selection sparse decode moves from the per-row producer to the multi-query producer at this many rows +# (kernels/triton_sparse_decode.MQ_MIN_TOKENS). +SPARSE_DECODE_MQ_MIN_TOKENS = 17 # capacity: fixed: the multi-query crossover +# The long-row top-16 selector takes engines of 129..LONG_SELECT_MAX_BLOCKS KV blocks of 128 tokens; it holds 2, 4 or +# 9 keys per thread by the row's valid-block count. At most 128 blocks the built-in selector runs. +LONG_SELECT_MIN_BLOCKS = 129 # capacity: fixed: past the built-in selector's 128 +LONG_SELECT_MAX_BLOCKS = 0 # capacity: fixed: off at TP4 (the DSL select takes the long rows) +LONG_SELECT_KEY_WIDTHS = (2, 4, 9) # capacity: fixed: compiled keys per thread +# The long-row select releases the sparse decode after it right after its own grid wait (the decode stages q and the +# request length before its wait) up to this many query rows. On Rubin TP4 the early-launched attention CTAs crowd the +# select and the o_proj GEMV already at 2 requests (8 rows), so the early release is off at every row count: in-model, +# locked memory clock, pinned tactics, long-context draws, 2 instances vs the stored baseline, the replay period with +# the trigger phase below is -209 us per step at 2 requests (mean of draws), -39 at 4 and -12 at 8. +SELECT_EARLY_MAX_ROWS = 0 # capacity: fixed: measured early-release crossover (0: off) +# N1 at TP2 (with TP2WIDE_LC2, ported from the M3 sm_107 TP2 target): the long-context sparse decode (LC2, +# SPARSE_DECODE_LC) records each step's selected block ids per sparse layer and, before its grid wait (while the select +# ranks), pulls into L2 the K/V pages + scales of the blocks the SAME layer selected one step back (65-68% of a layer's +# selected blocks recur one step later, measured). Same layer, not the next one as at TP4: the MoE between two +# sparse layers streams ~250 MB through the 126 MB L2, so an earlier issue would be evicted. Hints only (no value +# changes); up to N1_MAX_ROWS query rows. Replaces the TP4 base's LC4 knobs (merge-once op, o_proj / next-layer KV +# prefetch, identity steps), which never ran at TP2 (LC_TP2 off; LC4 declines the TP2 heads). Kill: False. +SPARSE_DECODE_LC_PREFETCH_SELF_KV = True +N1_MAX_ROWS = ( + _sparse_decode_kernel.LC_MAX_ROWS +) # capacity: fixed: the N1 prefetch's row bound (LC2's widest step) +# Up to SELECT_LONG_TRIGGER_MAX_ROWS query rows and without the early release, the long-row select releases the sparse +# decode once every score of its row is loaded (SELECT_LONG_TRIGGER_PHASE 1, before the exact rank of the candidates) +# instead of at its exit (0, the kill value); wider steps keep the exit. The release still follows the select's own +# grid wait, which the sparse decode's pre-wait q / length / block-table reads rely on. +SELECT_LONG_TRIGGER_PHASE = 1 # capacity: fixed: a trigger phase, not a size +SELECT_LONG_TRIGGER_MAX_ROWS = 8 # capacity: fixed: measured trigger crossover +# The long-row select ranks a row's candidates on warp 0 when they number at most 64 (a 64-key bitonic network; the +# other warps exit) and reads the row's valid count before its grid wait; the picks are the same. Kill: False (the +# all-thread rank). In-model, locked memory clock, double-pinned, A and B in one job, both orders: flat within +# noise (mean of draws -5 / -60 us at 1 / 2 requests; replay period at the 323K request +14 / +54); a FLAT-but-exact +# stack item (post-wait select time 1.76 -> 1.54 us per layer at 1 request, 165K tokens). OFF by default: at TP2 the +# warp-0 tail runs ~0.95 us/layer slower in-model next to the early-launched sparse decode (+53 us/step); +# on at TP4 only after an in-model A,B shows a gain. +SELECT_LONG_SL3 = False +# The long-row select runs as kernels/m3_select_long_dsl (ops/attention/m3_select_long_dsl): one 384-thread CTA per row +# instead of one 1024-thread CTA, the same selection bit for bit, with 4, 11 or 24 keys per thread +# (LONG_SELECT_DSL_KEY_WIDTHS) and the same trigger choices (SELECT_LONG_SL3 does not apply). TP4 has the TP8 target's +# per-rank select geometry (1 KV head, 4 rows per request), where on the Rubin decode chain (57 layers behind the +# producer stream, 1 / 2 requests) the select is -0.15 / -0.12 us per layer at 165K tokens, 0.00 / -0.01 at 323K and +# -0.04 / -0.04 at 500K against the 1024-thread kernel. Kill: False (the 1024-thread kernel). +SELECT_LONG_DSL = True +LONG_SELECT_DSL_KEY_WIDTHS = (4, 11, 24) # capacity: fixed: compiled keys per thread +# The MSA index scorer (ops/attention/m3_index_score) prefetches the older index-K pages into L2 before its grid wait +# at every batch; up to this many requests it also issues Q and the first K tile right after the wait and caps that +# prefetch at one block per CTA (larger batches are bandwidth bound, where the extra early tiles only crowd the +# prefetch). On Rubin TP4 the 2-request step keeps the early schedule: with the producer's trigger at its exit +# (PRODUCER_LATE_TRIGGER_MAX_ROWS) the scorer no longer overlaps the producer, and the capped prefetch shortens its +# exposed pre-wait stream. In-model, locked memory clock, tactic- and FlashInfer-pinned, A and B in one job, both +# orders: -260 us per decode step at 2 requests (replay period at the 323K P90 request; -82 mean of draws), flat at +# 1 / 4 / 8. Timing only: the tiles are read after the wait. Kill: 1. +INDEX_SCORE_EARLY_MAX_REQUESTS = 2 # capacity: fixed: measured early-issue crossover +# T2SC: up to 2 requests the sparse decode scores with this target's copy of the sm_107 TP2 / TP8 scorer +# (kernels/m3_index_score_t2sc; ops/attention/m3_index_score_t2sc), whose split is chosen per request count and which +# TMA-loads an all-old first block before its grid wait; the scores are bit-identical +# to the in-tree scorer. TP4 has the TP8 target's per-rank indexer geometry (1 index head over the replicated index-K +# of its KV group), where this schedule is ~100 us per decode step faster at 1 request than split 512 without the +# prestage (TP8 in-model, greedy early column, both orders). Wider steps keep the sm_103-derived scorer above. +# Kill: False (the sm_103-derived scorer at every request count). +INDEX_SCORE_T2SC = True +# At 1 request the split is one wave of CTAs (1272 = 212 SMs x 6 resident at this CTA's shared memory): at the +# 1-index-head geometry (57-layer chain behind the producer, scores and the selection bitwise) -0.03 / -0.27 / +0.06 us +# per layer at 165K / 323K / 500K tokens against 1536. Kill: 1536. +INDEX_SCORE_T2SC_SPLIT_K: Mapping[int, int] = MappingProxyType( + {1: 1272, 2: 768} +) # capacity: fixed: measured splits per request count +# TP2 WIDE: from INDEX_SCORE_INTREE_MIN_REQUESTS requests on (40+ rows at K7) the sparse decode scores with the +# in-tree CuTe DSL scorer (``cute_dsl_minimax_m3_index_decode_score``, its 256-CTA split) instead of the sm_103-derived +# split-512 scorer: the same scores bit for bit. VR microbench (60-call graphs, 2 index heads, qlen 8, PDL): faster at +# every request count 5..24 and context 16K / 64K / 128K / mixed, e.g. mixed -0.97 / -1.51 / -3.28 / -3.86 / -5.98 us +# per layer at 5 / 8 / 12 / 16 / 24 requests. Steps of <= 32 rows (any request count) keep their scorer. Kill: +# TRTLLM_M31_DISABLE_TP2WIDE_INTREE_SCORER=1. Read once, at import. +INDEX_SCORE_INTREE = not m31_disabled("TP2WIDE_INTREE_SCORER") +INDEX_SCORE_INTREE_MIN_REQUESTS = ( + 5 # capacity: fixed: measured crossover (4 requests = 32 rows stays as is) +) +TP2_WIDE_MIN_ROWS = ( + 33 # capacity: fixed: TP2 WIDE rule: steps of <= 32 rows keep the baseline dispatch bit for bit +) +_INDEX_SCORE_INTREE_LOGGED: set = set() + + +def _in_tree_scorer(): + """The in-tree CuTe DSL index scorer's runner (importing it registers ``cute_dsl_minimax_m3_index_decode_score``); + imported on first use, as the CuTe DSL ops are.""" + from tensorrt_llm._torch.custom_ops.cute_dsl_custom_ops import ( + CuteDSLMiniMaxM3IndexDecodeScoreRunner, + ) + + return CuteDSLMiniMaxM3IndexDecodeScoreRunner + + +# The decode MXFP8 GEMVs, per projection (N, K): the rows each serves. On an activation its producer quantized +# (the MoE boundary for the QKV, the attention merge for the o_proj) the GEMV serves every row up to 48 but the +# unsplit K=768 shared down projection at 17..32 rows, where FlashInfer's GEMM is faster. +# The QKV MXFP8 GEMVs' (split, ring) at decode rows up to GEMV_QKV_CFG_MAX_ROWS: a TP4-owned override of the shared +# GB300-tuned table (kernels/mx_gemv/op.py). SM107 TP4 sweep +# (4 rows): sparse-layer QKV (8, 8) 2.70 us vs (4, 12) 3.01, dense-attention QKV (8, 8) 2.60 vs 2.83. Split 8 +# sums K in a different fp32 split-K order than split 4: exact-class, not bitwise to the shared config. Kill: False. +GEMV_QKV_SPLIT8_TP4 = True +GEMV_QKV_CFG_MAX_ROWS = 8 # capacity: fixed: the measured override's rows +MX_GEMV_CFG: Dict[Tuple[int, int], tuple] = { + QKV_SPARSE: (8, 8), + QKV: (8, 8), +} # capacity: fixed: (split, ring) per projection +# (exact-class, deterministic) The o_proj GEMV at decode rows up to GEMV_QKV_CFG_MAX_ROWS with strided k-tiles per split +# rank (contig 0: rank r owns k-tiles r, r + split, ...) instead of contiguous ranges, the shared (2, 8): another +# fixed fp32 order. SM107 TP4 4 rows: 2.513 / 2.533 vs 2.550 / 2.563 us; deterministic (3 runs bitwise) and +# max error vs the fp64 product equal to the contiguous order's. Kill: False. +GEMV_OPROJ_STRIDED_TP4 = True +MX_GEMV_CFG_STRIDED: Dict[Tuple[int, int], tuple] = { + O_PROJ: (2, 8, 0) +} # capacity: fixed: (split, ring, contig) per projection + + +def mx_gemv_cfg(n: int, k: int, rows: int) -> Optional[tuple]: + """The TP4 GEMV config override for an [rows, k] x [n, k]^T MXFP8 GEMV call, or None for the shared table's.""" + if rows > GEMV_QKV_CFG_MAX_ROWS: + return None + if GEMV_QKV_SPLIT8_TP4 and (n, k) in MX_GEMV_CFG: + return MX_GEMV_CFG[(n, k)] + if GEMV_OPROJ_STRIDED_TP4: + return MX_GEMV_CFG_STRIDED.get((n, k)) + return None + + +# Synth only (TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS, fractional): the fused verify's forced accepted counts are +# drawn per request, keyed by the request's KV length at the verify (its prompt plus what it has accepted), instead of +# from the server-wide counter, so a request's forced counts do not depend on what the server ran before (A/B draws +# become history-independent). Production sampling never forces counts. Kill: False (the shared counter, the in-tree +# sequence). +FORCE_AL_PER_REQUEST = True + +_ALL_ROWS = frozenset(range(1, DECODE_MAX_TOKENS + 1)) # capacity: decode rows +# TP2WIDE_GEMM (TP2 WIDE, microbench: SM107, mclk locked, MXFP8 activation, weights rotated past L2, 2 processes): +# at 33..48 rows the dense MLP's MXFP8 projections take the pinned FlashInfer GEMM (MX-TAC, the armed tactic table) +# instead of the GEMV: dense gate_up 15.05 -> 10.10 us/call, dense down 13.57 -> 8.51 (M3 #20623's tp2 wide method, +# which keeps the dense MLP on the GEMM). Every other projection keeps the GEMV there (QKV / shared expert 1.7..2.7 us +# faster; o_proj even). Rows <= 32 unchanged. Kill switch TRTLLM_M31_DISABLE_TP2WIDE_GEMM=1 (the GEMV at 33..48 and the +# VR_GEMV48 / LINEAR_CUBLASLT row tables below), read at import. +TP2WIDE_GEMM = not m31_disabled("TP2WIDE_GEMM") +_DENSE_GEMM_ROWS = ( + frozenset(range(33, DECODE_MAX_TOKENS + 1)) if TP2WIDE_GEMM else frozenset() +) # capacity: fixed: the measured GEMM band of the dense MLP +MX_GEMV_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = { + QKV_SPARSE: _ALL_ROWS, + QKV: _ALL_ROWS, + O_PROJ: _ALL_ROWS, + SHARED_GATE_UP: _ALL_ROWS, + SHARED_DOWN: _ALL_ROWS - frozenset(range(17, 33)), + DENSE_GATE_UP: _ALL_ROWS - _DENSE_GEMM_ROWS, + DENSE_DOWN: _ALL_ROWS - _DENSE_GEMM_ROWS, +} # capacity: decode rows per projection +# On a bf16 activation the GEMV quantizes it in its prologue, which pays only up to 8 32-element blocks per thread: +# the most rows each projection takes. +# TP2: the GEMV's own prologue bound (kernels/mx_gemv/op._declines: tiles * 4 * mma_n <= 1024 at each projection's +# config) caps o_proj (K 4096, split 2: 16 k-tiles) at 16 rows and the dense down (K 6144, split 2: 24 k-tiles) at 8; +# the TP4 table's 32 / 16 would be covered rows the GEMV declines (a miss, raised under modeling_v2='require'). +MX_GEMV_BF16_MAX_TOKENS: Dict[Tuple[int, int], int] = { + QKV_SPARSE: 16, + QKV: 16, + O_PROJ: 16, + SHARED_GATE_UP: 16, + SHARED_DOWN: 16, + DENSE_GATE_UP: 8, + DENSE_DOWN: 8, +} # capacity: fixed: measured bf16-GEMV crossover per projection +# MX-TAC (the TP2 target's, ported): the decode-graph MXFP8 projections the GEMVs do not serve (SHARED_DOWN at 17..32 +# rows, rows over DECODE_MAX_TOKENS or a bf16 activation over MX_GEMV_BF16_MAX_TOKENS, or a GEMV decline) take one fixed +# decision instead of the AutoTuner's per-start, per-rank choice: FlashInfer's CuTe DSL quantize and CuTe DSL GEMM. All +# of stock's candidates (quantize {TRT-LLM, CuTe DSL} x GEMM {CUTLASS, CuTe DSL}) give the same bits at the TP4 +# projections (gemm/test_modeling_v2_m3_pinned_gemm.py), so the pin is stock's output on every rank and start. Kill +# constant: False returns these projections to the AutoTuner. +MX_TAC_TP4 = True +# The tactics below are MXFP8QuantizeRunner.CUTE_DSL / FlashInferMXFP8GemmRunner.CUTE_DSL (0). +MXFP8_DECODE_QUANT_TACTIC = 0 # capacity: fixed: a tactic id, not a size +MXFP8_DECODE_GEMM_TACTIC = 0 # capacity: fixed: a tactic id, not a size +# TP2 port: ported from the sm_107 TP8 target as is. Its rows start past DECODE_MAX_TOKENS (48 here), so at K7 +# c1..c4 (8..32 rows) it never engages; its shape keys (m3_mx_gemv GEMV48_SHAPES, m3_mxfp8_cublaslt.SHAPES) +# are the per-rank shapes of this target's m3_mx_gemv constants. +# Generation steps of 33..64 rows (K7 c5..c8) run past the decode bound, so their MXFP8 projections took the quantize + +# FlashInfer CuTe DSL GEMM, which steps from ~8 to ~11.8 us/call above 32 rows on the K = 6144 projections (SM107). +# Two existing kernels take those rows instead, on the same quantized activation (not bitwise to that GEMM): +# * VR_GEMV48 (switch TRTLLM_M31_DISABLE_VR_GEMV48): the GEMV (it serves up to 48 rows) on GEMV48_SHAPES at 33..48; +# * LINEAR_CUBLASLT (switch TRTLLM_M31_DISABLE_LINEAR_CUBLASLT, shared with the sm_103 targets): cuBLASLt's MXFP8 GEMM +# (ops/gemm/m3_mxfp8_cublaslt) on its SHAPES at every other row up to 64. +# A bf16 activation is quantized first by the CuTe DSL quantize (the bytes of mxfp8_quantize). Measured us/step over the +# 240 calls of a K7 step: -583..-592 at 33..48 rows, -494..-497 at 56..64. Off: back to FlashInfer's GEMM. +VR_GEMV48 = not m31_disabled("VR_GEMV48") +LINEAR_CUBLASLT = not m31_disabled("LINEAR_CUBLASLT") +GEMV48_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + { + nk: frozenset(range(max(GEMV48_MIN_TOKENS, DECODE_MAX_TOKENS + 1), mx_gemv.MAX_TOKENS + 1)) + for nk in GEMV48_SHAPES + } + if VR_GEMV48 and not TP2WIDE_GEMM + else {} +) +# TP2WIDE_GEMM: ``_linear`` has no branch for GEMV48_PATH / CUBLASLT_GEMM_PATH at TP2, so their rows ran the pinned +# FlashInfer GEMM (MX-TAC) under those names; with TP2WIDE_GEMM the tables are empty and ``linear_path`` names that +# GEMM (the same kernels: a microbench found cuBLASLt flat or slower than it at 49..64 on every TP2 projection). +LINEAR_CUBLASLT_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + { + nk: frozenset( + m + for m in range( + max(mxfp8_cublaslt.MIN_TOKENS, DECODE_MAX_TOKENS + 1), mxfp8_cublaslt.MAX_TOKENS + 1 + ) + if m not in GEMV48_ROWS.get(nk, ()) + ) + for nk in mxfp8_cublaslt.SHAPES + } + if LINEAR_CUBLASLT and not TP2WIDE_GEMM + else {} +) + +Key = Hashable + + +@dataclass(frozen=True) +class RowRange: + """The shapes one fast path takes, and the catalog entry whose cells certify each of them. + + ``keys`` are the covered shapes: row counts for most paths, (query rows per request, requests) for the attention + kernels, a variant name for thresholds. ``certifies`` maps a cell's spec of ``entry`` to the keys that cell + drives (empty for a cell of another variant). + """ + + path: str + keys: FrozenSet[Key] + entry: OpWrapper + certifies: Callable[[Mapping[str, Any]], Iterable[Key]] + + def covers(self, key: Key) -> bool: + return key in self.keys + + def certified(self) -> FrozenSet[Key]: + return frozenset(k for cell in self.entry.CELLS for k in self.certifies(cell.spec)) + + +def _rows(lo: int, hi: int) -> FrozenSet[int]: + return frozenset(range(lo, hi + 1)) + + +def _spec_rows(field: str, **match: Any) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + """A cell's ``spec[field]`` when the cell's other fields match ``match``.""" + + def certifies(spec: Mapping[str, Any]) -> Iterable[Key]: + if any(spec.get(name) != value for name, value in match.items()): + return () + return (spec[field],) + + return certifies + + +def _const(*keys: Key) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + return lambda spec: keys + + +def _sparse_decode_producer(spec: Mapping[str, Any]) -> Iterable[Key]: + if not spec["identity"]: + return () + rows = spec["decode_query_len"] * len(spec["seq_lens"]) + return ("multi_query" if rows >= SPARSE_DECODE_MQ_MIN_TOKENS else "per_row",) + + +def _long_select_widths(spec: Mapping[str, Any]) -> Iterable[Key]: + """The key widths a cell's rows reach: every width up to its blocks' when rows of mixed lengths are drawn.""" + blocks = spec["num_blocks"] + widest = 2 if blocks <= 2048 else 4 if blocks <= 4096 else 9 + if spec["valid"] == "mixed": + return tuple(w for w in LONG_SELECT_KEY_WIDTHS if w <= widest) + return (widest,) + + +EMBED_FIRSTNORM = RowRange( + "embed_firstnorm", + _rows(1, EMBED_FIRSTNORM_MAX_TOKENS), + m3_embedding_firstnorm, + _spec_rows("rows"), +) +SPARSE_DECODE_PRODUCER = RowRange( + "sparse_decode_producer", + frozenset({"per_row", "multi_query"}), + m3_sparse_decode, + _sparse_decode_producer, +) + + +def _sparse_decode_lc_keys(spec: Mapping[str, Any]) -> Iterable[Key]: + """A cell certifies the LC2 key it drives when its step lies in the rows TP2WIDE_LC2 takes.""" + rows = spec["decode_query_len"] * len(spec["seq_lens"]) + lo = _sparse_decode_kernel.lc_min_rows(spec["decode_query_len"]) + if not (lo <= rows <= _sparse_decode_kernel.LC_MAX_ROWS): + return () + if spec["linear_softmax"] or spec["prewait"]: + return () + return ("mxfp8" if spec["mxfp8"] else "plain",) + + +# TP2WIDE_LC2: the long-context sparse decode at the TP2 heads (32 query / 2 KV heads, LC2): attention, split-K merge +# and the MXFP8 o_proj hand-off in one launch, for pure generation steps with a selected top-16 table at +# LC_MIN_ROWS (33)..LC_MAX_ROWS (448; 128 with TP2WIDE_LC2_448 disabled) rows. TP2_LC2_SMALL opts K3 steps +# into LC2 from 9 rows. The MXFP8 hand-off is published to m3_prequant.SLOT. Narrower steps keep the Triton +# producer + merge. frozenset() +# (TRTLLM_M31_DISABLE_TP2WIDE_LC2=1) routes every step back to the Triton producer + merge. +SPARSE_DECODE_LC = RowRange( + _sparse_decode_kernel.LC_PATH, + _sparse_decode_kernel.LC_KEYS, + m3_sparse_decode_lc, + _sparse_decode_lc_keys, +) +# Every token count of a sparse layer on an NVFP4 KV cache at the TP2 heads (prefill and decode): the producer takes the +# geometry, not a row range. +PRODUCER = RowRange("m3_producer", frozenset({"nvfp4_tp2"}), m3_producer, _const("nvfp4_tp2")) +# The per-rank (query, KV, index) heads ``m3_producer`` is built for (ops/attention/m3_producer HEADS_Q / HEADS_KV / +# HEADS_INDEX: MiniMax-M3's 64 / 4 / 4 heads over two ranks). C6: the gate held the TP4 heads (16, 1, 1) from the copied +# TP4 target, so no TP2 layer ever took the producer (the generic m3_qkv_insert ran, uncounted and unlogged). +PRODUCER_HEADS = (32, 2, 2) +INDEX_SCORE = RowRange( + "m3_index_score", + frozenset({"early", "prefetch"}), + m3_index_score, + lambda spec: ("early" if spec["early"] else "prefetch",), +) + + +def index_score_t2sc_schedule(requests: int) -> Tuple[int, bool]: + """T2SC's (CTAs per request, early issue) at this many requests (a request count in INDEX_SCORE_T2SC_SPLIT_K).""" + return INDEX_SCORE_T2SC_SPLIT_K[requests], requests <= INDEX_SCORE_EARLY_MAX_REQUESTS + + +# The index heads per rank this target's scorer runs (MiniMax-M3.1: 4 index heads over 2 ranks). +INDEX_HEADS_TP2 = 2 # capacity: fixed: the per-rank index heads at TP2 + + +def _index_score_t2sc_requests(spec: Mapping[str, Any]) -> Iterable[Key]: + """A cell certifies its request count when it drives this target's 2 index heads with that count's schedule.""" + requests = len(spec["seq_lens"]) + if spec.get("heads") != INDEX_HEADS_TP2 or requests not in INDEX_SCORE_T2SC_SPLIT_K: + return () + if (spec["split_k"], spec["early"]) != index_score_t2sc_schedule(requests): + return () + return (requests,) + + +INDEX_SCORE_T2SC_PATH = RowRange( + "m3_index_score_t2sc", + frozenset(INDEX_SCORE_T2SC_SPLIT_K) if INDEX_SCORE_T2SC else frozenset(), + m3_index_score_t2sc, + _index_score_t2sc_requests, +) +LONG_SELECT = RowRange("long_select", frozenset(), m3_select_long, _long_select_widths) + + +def _long_select_dsl_widths(spec: Mapping[str, Any]) -> Iterable[Key]: + """The 384-thread select's key widths a cell's rows reach (every width up to its blocks' for mixed rows).""" + blocks = spec["num_blocks"] + widest = 4 if blocks <= 4 * 384 else 11 if blocks <= 11 * 384 else 24 + if spec["valid"] == "mixed": + return tuple(w for w in LONG_SELECT_DSL_KEY_WIDTHS if w <= widest) + return (widest,) + + +LONG_SELECT_DSL = RowRange( + "long_select_dsl", + frozenset(LONG_SELECT_DSL_KEY_WIDTHS) if SELECT_LONG_DSL else frozenset(), + m3_select_long_dsl, + _long_select_dsl_widths, +) +MC_ALLREDUCE = RowRange( + "mc_allreduce_norm", + _rows(1, DECODE_MAX_TOKENS) if MC_AR_TP4 else frozenset(), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=0, drafter=None), +) +# Rows of a fused-draft-loop all-reduce boundary (drafter hidden 6144) on the multicast all-reduce's drafter form. +DRAFT_MC_AR = RowRange( + "m3_draft_mc_allreduce_norm", + _rows(1, DRAFT_MC_AR_MAX_ROWS), + m3_mc_allreduce_norm, + _spec_rows("m", drafter=True), +) +DENSE_PUSH = RowRange( + "dense_push", + _rows(1, DENSE_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +ATTN_PUSH = RowRange( + "attn_push", + _rows(1, ATTN_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +MOE_PUSH = RowRange( + "moe_push", + _rows(1, MOE_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +MC_FINALIZE = RowRange( + "mc_finalize_allreduce_norm", + _rows(MC_FINALIZE_MIN_TOKENS, MC_FINALIZE_MAX_TOKENS), + m3_mc_finalize_allreduce_norm, + _spec_rows("m"), +) +MOE_ENGINE = RowRange( + "m3_moe_engine", _rows(1, MOE_ENGINE_MAX_TOKENS), m3_moe_engine, _spec_rows("tokens") +) +MOE16_ENGINE = RowRange( + "m3_moe16_engine", + _rows(MOE16_MIN_TOKENS, MOE16_MAX_TOKENS), + m3_moe16_engine, + _spec_rows("tokens"), +) +# The router of a pure generation step of 9..16 rows when the MoE runs unfused (above the MoE engine's 8; at 9..13 an +# MoE16 engine miss, 14..16 always): the TF32 tensor-core router (minimax_m3_router_tc, 16-CTA clusters, one reduction +# order), the arithmetic the built-in gate takes on SM107 at these rows. Devices that cannot co-schedule 16-CTA +# clusters keep the fp32 router. Context steps never take it. +ROUTER_TC_GEN_MIN_ROWS = 9 # capacity: fixed: past the MoE engine's 8 rows +# ROUTER48 (ported from the sm_107 TP8 target; switch TRTLLM_M31_DISABLE_ROUTER48): the TF32 tensor-core +# router up to the op's certified 48 rows (K7 c3 / c4: 24 / 32 rows). The router weights are replicated, so +# the op's cells are TP-independent; a microbench measured -116 / -103 us per step at 24 / 32 rows (TP8). +ROUTER48 = not m31_disabled("ROUTER48") +# Reuse the native router NVFP4 side output at its certified small-row shapes. +X1 = not m31_disabled("X1") +ROUTER_TC_GEN_MAX_ROWS = 48 if ROUTER48 else 16 # capacity: fixed: the op's certified rows +# S1B_FOLD (ops/moe/m31_s1b_fold.py): generation steps above the TC router's rows route from top-k ids (bmm over +# K-slices + one reduce / MiniMax2-select kernel; bitwise the stock router + routing). Pinned per row count at +# prebuild up to the WIDE decode cap. Kill switch TRTLLM_M31_DISABLE_S1B_FOLD=1. +S1B_MAX_ROWS = 448 # capacity: fixed: the WIDE decode rows (A1_WIDE_448) +# Diagnostics (default off): TRTLLM_M31_DISABLE_S1B_FOLD_PDL=1 launches the fold without PDL; +# TRTLLM_M31_ENABLE_S1B_FOLD_CHECK=1 also runs the stock route + MoE on every folded call, keeps the STOCK output and +# counts bitwise mismatches of the fold's output on device (logged by a host thread; ``_S1bCheck``). +S1B_FOLD_PDL = not m31_disabled("S1B_FOLD_PDL") +S1B_FOLD_CHECK = m31_enabled("S1B_FOLD_CHECK") +ROUTER_TC_GEN = RowRange( + "m3_router_tc", + _rows(ROUTER_TC_GEN_MIN_ROWS, ROUTER_TC_GEN_MAX_ROWS), + m3_router_tc, + _spec_rows("m"), # every cell checks the logits, with or without the NVFP4 activation +) +# The GEMM paths are keyed (projection (N, K), rows): each projection runs its own split-K / ring / MMA +# configuration, so a row count is certified per projection. +MX_GEMV = RowRange( + "m3_mx_gemv", + frozenset((nk, m) for nk, rows in MX_GEMV_ROWS.items() for m in rows), + m3_mx_gemv, + lambda spec: ((spec["nk"], spec["m"]),), +) +MX_GEMV_BF16 = RowRange( + "m3_mx_gemv_bf16", + frozenset((nk, m) for nk, hi in MX_GEMV_BF16_MAX_TOKENS.items() for m in _rows(1, hi)), + m3_mx_gemv_bf16, + lambda spec: ((spec["nk"], spec["m"]),), +) +MX_GEMV48 = RowRange( + "m3_mx_gemv48", + frozenset((nk, m) for nk, rows in GEMV48_ROWS.items() for m in rows), + m3_mx_gemv, + lambda spec: ((spec["nk"], spec["m"]),), +) +LINEAR_CUBLASLT_GEMM = RowRange( + "m3_mxfp8_cublaslt", + frozenset((nk, m) for nk, rows in LINEAR_CUBLASLT_ROWS.items() for m in rows), + m3_mxfp8_cublaslt, + lambda spec: (((spec["n"], spec["k"]), spec["m"]),), +) +# (draft length, requests) of a greedy verify step. +SHARD_ACCEPT = RowRange( + "m3_shard_accept", + frozenset((SHARD_ACCEPT_DRAFT_LEN, n) for n in range(1, SHARD_ACCEPT_MAX_REQUESTS + 1)), + m3_shard_accept, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) + +# Requests of a decode step whose dense-layer attention runs at a capped KV split. +DENSE_DECODE_SPLIT = RowRange( + "m3_dense_decode", + frozenset(DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH), + m3_dense_decode, + lambda spec: (len(spec["kv_lens"]),) if spec["capped"] else (), +) + +# (query rows per request, requests) of a decode step whose dense-layer attention runs on PrimTS split-KV decode. +FULLCTX_DECODE = RowRange( + "m3_fullctx_decode", + frozenset(FULLCTX_DECODE_RECIPES) if FULLCTX_DECODE_ENABLED else frozenset(), + m3_fullctx_decode, + lambda spec: ((spec["qlen"], len(spec["kv_lens"])),), +) + +# (query rows per request, requests) of a generation-only drafter pass that runs on the full-context decode. +DRAFTER_FULLCTX = RowRange( + "m3_drafter_fullctx", + frozenset(FULLCTX_DECODE_RECIPES) if DRAFTER_FULLCTX_ENABLED else frozenset(), + m3_fullctx_decode, + lambda spec: ((spec["qlen"], len(spec["kv_lens"])),), +) + +# Verify rows of a fused-verify step whose target logits come from the coded head. +TARGET_HEAD = RowRange( + "m3_target_head", + _rows(1, TARGET_HEAD_MAX_ROWS), + m3_target_head, + _spec_rows("m"), +) + +# (draft length, requests) of a fused draft loop, and the rows of its passes' draft input: the first pass's +# requests x (draft length + 1), one per request on the later passes. +DRAFT_STEP = RowRange( + "m3_draft_step", + frozenset((DRAFT_STEP_DRAFT_LEN, n) for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1)), + m3_draft_step, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) +# ((N, K), rows) of a drafter BF16 linear on its low-M kernel. +DRAFT_LOWM = RowRange( + "m3_draft_lowm", + frozenset((nk, m) for nk, rows in DRAFT_LOWM_ROWS.items() for m in rows), + m3_draft_lowm, + lambda spec: ((spec["nk"], spec["m"]),), +) +DRAFT_INPUT = RowRange( + "m3_draft_input", + frozenset( + rows + for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1) + for rows in (n, n * (DRAFT_STEP_DRAFT_LEN + 1)) + ), + m3_draft_input, + _spec_rows("m"), +) + +# The step-loop hooks (RuntimeHooks), one switch each. They move a decode step's input preparation +# off the GPU timeline between step graphs. +STEP_PACK_INPUTS = True +# The pack's host copies go out in one parallel scatter (a program per copy) when their destinations do not overlap: the +# serial one-CTA scatter pays one dependent global round trip per copy (SM107: 12.5 / 15.6 / 31.7 us at 16 / 24 / 40 +# copies vs 6.5 / 6.5 / 6.9 us parallel). +STEP_PACK_PARALLEL_SCATTER = True +# The pack is issued by the decode graph's own prologue (a copy of a pinned window + the parallel scatter), so no +# operation is left between the step graphs (needs STEP_PACK_PARALLEL_SCATTER). +STEP_PACK_IN_GRAPH = True +# The window reaches the prologue's device buffer on a side copy stream while the previous graph runs; the prologue +# waits on that copy and scatters device to device, so the replay no longer starts with a host copy. Needs +# STEP_PACK_IN_GRAPH. +STEP_PACK_EARLY_COPY = True +# The step epoch (step_inputs.step_epoch): one int64 device counter the decode graph's prologue increments once per +# replay (program 0 of its first scatter launch, no extra kernel) and never resets. The per-layer megakernel pieces +# tag their per-step device flags with it (contract v1: tag = epoch << 8 | layer). Needs STEP_PACK_IN_GRAPH. +STEP_PACK_EPOCH = True +# The prologue is one launch of this target's op (ops/spec/m3_step_prologue, minimaxM3Sm107Tp4/stepPrologue.cu): the +# host copies, a CTA barrier, the device copies, the header clear and the epoch, where the default prologue runs two +# Triton scatter launches and an ATen fill (3 graph kernels -> 1, no Triton at the head of the decode graph). Same bytes +# written. Needs STEP_PACK_EARLY_COPY. Kill: False. +STEP_PACK_CPP_PROLOGUE = True +# PRE-WAIT WRITER CONTRACT: the kernels that read step inputs (seq_lens, +# block tables, positions, cache slots, the sampled-accept pre-wait parameters) before their grid-dependency wait rely +# on every writer of them being a host-to-device copy before the replay or a PLAIN launch earlier in the step: +# msa_kv_update, the step-input prologue (Triton scatter or the C++ op), the spec fold tail (Triton or C++), the pack +# scatter and any step-head op (the spec step correction, STEP_SPEC_OFFSETS_ONE_LAUNCH). Never give one of them a PDL +# attribute with an early trigger. +STEP_SKIP_UNCHANGED_H2D = True +# A changed packed 2-D metadata upload stages only each row's changed column span. On a step where a request crosses +# into a new page the MSA block table changed by one entry, but the whole padded table (32 KB a request) was staged, +# overflowing the 16 KB in-graph window into an eager upload and scatter between the step graphs (57-69 us at the +# ~1 in 19 block-crossing steps; TTFT inter-graph scan). Kill: False. +STEP_H2D_CHANGED_SPAN = True +# A speculative overlap step re-derives the MSA KV-length metadata once, after the forward's KV-length correction, +# instead of also before it (RuntimeHooks.kv_lens_update_once): the first pass (a fill, the out_cache_loc tail fill and +# an msa_kv_update at the head of the decode graph, 2.13 us at TP4) is rewritten in full by the second. Kill: False. +STEP_KV_LENS_UPDATE_ONCE = True +# The overlap scheduler's speculative step correction (position_ids += offsets, kv_lens += offsets) runs as this +# target's one plain launch (RuntimeHooks.spec_offsets_op, ops/spec/m3_spec_offsets) instead of two ATen adds at the +# head of the decode graph: the same int32 adds. TP8 graph bench (the work is per token, not per rank; 64 steps +# per graph, a predecessor kernel each): -1.27 us/step at 4 tokens / 1 request, -0.72..-0.78 at 8-32 tokens. +# Kill: False. +STEP_SPEC_OFFSETS_ONE_LAUNCH = True +STEP_LOOP_ON_EXECUTION_STREAM = True +STEP_MERGE_TUNED_TACTICS = True +STEP_SKIP_GREEDY_RNG_UPLOAD = True +# Sampled speculative batches stage their four per-step Philox seed/offset uploads into the step-input pack +# (STEP_PACK_INPUTS) instead of issuing four copies between the step graphs. +STEP_PACK_SPEC_RNG_UPLOADS = True +# The spec step's small uploads: the accepted-draft counts ride the step-input pack, and the sampler's slot ids stay in +# a device buffer re-uploaded only when they change (a pageable upload every step otherwise). +STEP_SPEC_SMALL_H2D = True +# SKIP_ACCEPTED_UPLOAD (KIMI-HO-09, RuntimeHooks.skip_accepted_draft_upload): the step's per-request accepted-draft +# counts are no longer uploaded and spec_metadata.num_accepted_draft_tokens is None. Nothing on this target reads that +# tensor (modeling.py, dspark.py, dspark_s4.py, mxdraft.py, kernels/, ops/; the in-tree DFlash / DSpark workers and +# the Eagle3 path: tests/unittest/_torch/executor/test_skip_accepted_draft_upload.py scans for readers). Kill switch: +# TRTLLM_M31_DISABLE_SKIP_ACCEPTED_UPLOAD=1 (the stock per-step upload, packed under spec_small_h2d). +STEP_SKIP_ACCEPTED_UPLOAD = not m31_disabled("SKIP_ACCEPTED_UPLOAD") +# BATCH_CHANGED_ONLY (KIMI-HO-03 / ST-18(d), RuntimeHooks.spec_batch_map_changed_only): the standalone DSpark (DFlash) +# drafter's batch_indices and batch-to-slot map are uploaded only when they change (both constant in steady decode); +# the buffers hold the same values as with the stock every-step copies. Kill switch: +# TRTLLM_M31_DISABLE_BATCH_CHANGED_ONLY=1 (both copied every step). +STEP_BATCH_CHANGED_ONLY = not m31_disabled("BATCH_CHANGED_ONLY") +STEP_SPEC_FOLD = True +# The forward of a decode-only step stores its outputs into the spec sampler's store and stages the next step's inputs +# at its tail (one launch inside the step graph), so neither the sampler's store nor the next step's gathers run +# between the step graphs (M3SpecStepFold.in_forward). +STEP_SPEC_FOLD_IN_FORWARD = True +# The forward's tail (STEP_SPEC_FOLD_IN_FORWARD) runs as this target's C++ op (kernels/m3_fold_tail cpp=, +# minimaxM3Sm107Tp4/foldTail.cu) instead of the Triton kernel: the same integer copies, one launch either way (no +# Triton on the generation decode path). Kill: False. +STEP_FOLD_TAIL_CPP = True +# With STEP_SPEC_FOLD_IN_FORWARD the spec sampler's host copies of its store (new tokens, lengths, draft tokens) still +# ran on the main stream between the step graphs (the forward, not the sampler, wrote the store, and it rewrites it at +# the next replay's tail). They run on the D2H side stream instead, and the forward's tail waits on an event the +# sampler records after them before it rewrites the store (M3SpecStepFold.store_copies_done). +STEP_SPEC_STORE_COPY_SIDE = True +# The spec sampler copies its per-slot store to the host on its D2H side stream, so the copies overlap the next +# step's input staging instead of sitting between the step graphs. +STEP_SAMPLER_D2H_SIDE_STREAM = True +# The iteration log times the device step with one event per iteration boundary on a side stream, so no timing +# record sits between two step graphs on the main stream (the reported step times are the same). +STEP_ONE_TIMING_EVENT = True +MSA_SHORT_SEQUENCE_IDENTITY = True +# The sparse decode (the LC, the Triton merge) also writes the MXFP8 o_proj hand-off (bit +# for bit torch.ops.trtllm.mxfp8_quantize of its BF16 rows) and publishes it to m3_prequant.SLOT. Kill: False publishes +# nothing, so the o_proj quantizes its BF16 input itself as the built-in path does. +MERGE_MXFP8_HANDOFF = True +MSA_KV_UPDATE = True +MSA_SKIP_UNCHANGED_BLOCK_OFFSETS = True +MSA_REUSE_POOL_VIEWS = True +# Step-input preparation fills the MSA block-id table by slice-copying each request's page-index view (MSA +# RuntimeHooks.block_ids_from_views) instead of building a Python list of the whole padded row; the table is the same. +MSA_BLOCK_IDS_FROM_VIEWS = True +# The prefill indexer proxy can use tile 128 and a row/KV-size split ladder. The planner's tile is retained at +# <=128 rows; larger spans require maxscore_qo_tile support. Both routes are opt-in until the in-model oracle gate: +# TRTLLM_M31_ENABLE_AGG_PROXY_Q128 / _AGG_PROXY_SPLITS. Matching DISABLE switches override the opt-ins. +# TRTLLM_M31_ENABLE_AGG_PROXY_ORACLE compares valid score bits and the full selected table against the plain plan. +MSA_PROXY_Q128 = m31_enabled("AGG_PROXY_Q128") and not m31_disabled("AGG_PROXY_Q128") +MSA_PROXY_SPLITS = m31_enabled("AGG_PROXY_SPLITS") and not m31_disabled("AGG_PROXY_SPLITS") +MSA_PROXY_ORACLE = m31_enabled("AGG_PROXY_ORACLE") + +FAST_PATHS: Tuple[RowRange, ...] = ( + EMBED_FIRSTNORM, + DENSE_DECODE_SPLIT, + FULLCTX_DECODE, + DRAFTER_FULLCTX, + SPARSE_DECODE_PRODUCER, + SPARSE_DECODE_LC, + PRODUCER, + INDEX_SCORE, + INDEX_SCORE_T2SC_PATH, + LONG_SELECT, + LONG_SELECT_DSL, + MC_ALLREDUCE, + DRAFT_MC_AR, + ATTN_PUSH, + DENSE_PUSH, + MOE_PUSH, + MC_FINALIZE, + MOE_ENGINE, + MOE16_ENGINE, + ROUTER_TC_GEN, + MX_GEMV, + MX_GEMV_BF16, + MX_GEMV48, + LINEAR_CUBLASLT_GEMM, + SHARD_ACCEPT, + TARGET_HEAD, + DRAFT_STEP, + DRAFT_LOWM, + DRAFT_INPUT, +) + +# -------------------------------------------------------------------------------------------------------------------- +# Selection: pure functions of the step shape +# -------------------------------------------------------------------------------------------------------------------- + + +def plan_for(num_tokens: int) -> DecodePlan: + for plan in DECODE_PLANS: + if num_tokens <= plan.max_tokens: + return plan + raise AssertionError(f"no decode plan covers {num_tokens} tokens") + + +def is_decode_step(num_tokens: int) -> bool: + """Whether a pure generation step of ``num_tokens`` rows runs the decode kernels (else generic layers).""" + return num_tokens <= DECODE_STEP_MAX + + +def moe_path(num_tokens: int) -> Optional[str]: + """The MoE kernel of a decode step: an engine, the unfinalized MoE + finalize all-reduce, or None (generic).""" + if MOE_ENGINE.covers(num_tokens): + return MOE_ENGINE.path + if MOE16_ENGINE.covers(num_tokens): + return MOE16_ENGINE.path + if MC_FINALIZE.covers(num_tokens): + return MC_FINALIZE.path + return None + + +MX_GEMV_PATH = MX_GEMV.path +MX_GEMV_BF16_PATH = MX_GEMV_BF16.path +# The dense layers' swigluoai activation (where the fold above would apply) as this target's C++ op +# (ops/gemm/m3_dense_swiglu, minimaxM3Sm107Tp8/swigluOai.cu: bit for bit the Triton silu_and_mul), PDL-launched with +# an early release so the down GEMV stages its weights during it; the down GEMV reads the activation only after its own +# grid wait. TP2 chain bench: -0.69 us per dense layer vs the Triton kernel. Kill constant: False +# runs the Triton kernel. +DENSE_SWIGLU_CPP = True +DENSE_SWIGLU_CPP_PATH = "m3_dense_swiglu" +# FlashInfer's CuTe DSL MXFP8 GEMM on the producer's MXFP8 activation (rows the GEMV does not serve). +PREQUANT_GEMM_PATH = "flashinfer_mxfp8_gemm" +# The MXFP8 quantize (tuned backend) and FlashInfer's CuTe DSL GEMM: every row the decode kernels do not serve. +GENERIC_GEMM_PATH = "mxfp8_gemm" +# Past the decode bound: the GEMV (VR_GEMV48) or cuBLASLt's MXFP8 GEMM (LINEAR_CUBLASLT), on either activation. +GEMV48_PATH = MX_GEMV48.path +CUBLASLT_GEMM_PATH = LINEAR_CUBLASLT_GEMM.path + + +def linear_path(n: int, k: int, num_tokens: int, prequant: bool, low_m_gemm: bool) -> str: + """The decode-graph MXFP8 GEMM of an [N, K] projection on ``num_tokens`` rows. + + ``prequant``: the producer published the activation in MXFP8. ``low_m_gemm`` is accepted for the call sites' + symmetry: no low-M GEMM configuration is certified at TP4, so it selects nothing. + """ + if num_tokens in GEMV48_ROWS.get((n, k), ()): + return GEMV48_PATH + if num_tokens in LINEAR_CUBLASLT_ROWS.get((n, k), ()): + return CUBLASLT_GEMM_PATH + if prequant: + if num_tokens in MX_GEMV_ROWS.get((n, k), ()): + return MX_GEMV_PATH + return PREQUANT_GEMM_PATH + if num_tokens <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0): + return MX_GEMV_BF16_PATH + return GENERIC_GEMM_PATH + + +# -------------------------------------------------------------------------------------------------------------------- +# Fallback policy +# -------------------------------------------------------------------------------------------------------------------- + + +class FastPathUnavailable(RuntimeError): + """A covered decode shape could not take its fast path under ``modeling_v2='require'``.""" + + +_lock = threading.Lock() +_require = [False] +_logged: set = set() + + +def set_require(require: bool) -> None: + """Whether a miss raises (``modeling_v2='require'``); set by the target at construction.""" + _require[0] = bool(require) + + +def require_mode() -> bool: + return _require[0] + + +def _log_once(level: Callable[[str], None], path: str, reason: str, message: str) -> None: + key = (path, reason) + with _lock: + if key in _logged: + return + _logged.add(key) + level(message) + + +def long_select_dsl_keys() -> Tuple[Tuple[bool, bool, int], ...]: + """The (PDL launch, early trigger, trigger phase) keys of the 384-thread select a decode step can launch, compiled + before warmup; none when ``SELECT_LONG_DSL`` is off.""" + if not LONG_SELECT_DSL.keys: + return () + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + pdl = get_env_enable_pdl() + keys = [(pdl, False, SELECT_LONG_TRIGGER_PHASE), (pdl, False, 0)] + if SELECT_EARLY_MAX_ROWS: + keys.append((pdl, True, 0)) + return tuple(dict.fromkeys(keys)) + + +def missed(path: str, rows: Optional[int], reason: str) -> None: + """``path`` covers this step but cannot run: raise under require, else log once; the caller falls back.""" + at = "" if rows is None else f" at {rows} rows" + if _require[0]: + raise FastPathUnavailable( + f"modeling_v2='require': the MiniMax-M3 decode path {path!r} covers this step{at} but cannot run: " + f"{reason}" + ) + _log_once( + logger.warning, + path, + reason, + f"[M3 dispatch] {path}{at}: {reason}; falling back (logged once per reason)", + ) + + +def capture_missed(path: str, rows: Optional[int], what: str) -> None: + """``what`` is built on eager steps but a CUDA-graph capture found it missing (see ``missed``).""" + missed(path, rows, f"{what} is not built and a CUDA graph is being captured") + + +def taken(path: str, rows: Optional[int], what: str = "") -> None: + """``path`` ran on this rank at ``rows``: one rank-tagged info line per (path, rows, what), the engagement line + every GEN rank prints for each fast path it takes (``what``: an optional detail, e.g. the projection).""" + at = "" if rows is None else f" at {rows} rows" + detail = f" ({what})" if what else "" + _log_once( + logger.info, + path, + f"taken: {rows}" + (f": {what}" if what else ""), + f"[M3 dispatch] rank {mpi_rank()}: {path} taken{at}{detail}", + ) + # Gate e: every taken path is also a work-done counter under its path name (kernels/m3_report.work). + m3_report.work( + path, (f"{rows} rows" if rows is not None else "") + (f": {what}" if what else "") + ) + + +def _mc_taken(mapping, rows: int, what: str) -> None: + """The multicast all-reduce + norm's engagement line with its world size, kernel and this rank's call count (the + count is the ``mc_allreduce_norm`` work counter: captures + eager calls). At world 4 above + ``MC_AR_CLUSTER_MAX_ROWS`` the TP4 op runs its row kernel; at any other world size (TP2: world 2) it runs the 8-CTA + cluster kernel. Logged once per (rows, detail) per rank, like ``taken``.""" + world = int(mapping.tp_size) + kernel = ( + "row kernel" if world == 4 and rows > MC_AR_CLUSTER_MAX_ROWS else "8-CTA cluster kernel" + ) + calls = m3_report.work(MC_ALLREDUCE.path, f"{rows} rows: {what}, world {world}, {kernel}") + _log_once( + logger.info, + MC_ALLREDUCE.path, + f"taken: {rows}: {what}", + f"[M3 dispatch] rank {mpi_rank()}: {MC_ALLREDUCE.path} taken at {rows} rows ({what}, world {world}, " + f"{kernel}, call {calls})", + ) + + +def uncovered(path: str, reason: str) -> None: + """The step is outside ``path``'s covered shapes: log once per (path, reason); the generic path runs.""" + _log_once(logger.info, path, reason, f"[M3 dispatch] {path} not taken: {reason} (logged once)") + + +def reset_logged() -> None: + """Forget which (path, reason) pairs were logged.""" + with _lock: + _logged.clear() + + +# Kernel modules report from inside their calls through kernels/m3_report: the same policy. +m3_report.install( + on_taken=taken, on_missed=missed, on_uncovered=uncovered, on_capture_missed=capture_missed +) + + +# -------------------------------------------------------------------------------------------------------------------- +# Target constants +# -------------------------------------------------------------------------------------------------------------------- + +# The shared expert folds into the routed engine from one token (dedicated shared-expert CTAs). +MOE_FOLD_MIN_TOKENS = 1 # capacity: fixed: a row lower bound +# SMs left to the concurrent kernels while a persistent routed engine runs. +MOE_ENGINE_RESERVED_SMS = 24 # capacity: fixed: an SM split, not a size +# Id of the target's final hidden state in Eagle3's ``layers_to_capture``. +FINAL_HIDDEN_LAYER_ID = -1 # capacity: fixed: an Eagle3 layer id, not a size +# The trtllm-gen MoE runner's routing kind for MiniMax-M3 (sigmoid + correction bias, renormalized, scaled) and its +# gated activation (SwiGLU with the per-expert alpha / beta / clamp operands). +ROUTING_MINIMAX2 = int(RoutingMethodType.MiniMax2) +ACT_SWIGLU = int(ActType_TrtllmGen.SwiGlu) + +# -------------------------------------------------------------------------------------------------------------------- +# Streaming: also emit a response on decoding iterations 2..FIRST_CONTENT_EMIT_ITERS (RuntimeHooks), so a request's +# first content leaves one step after its first token. 1: the stock schedule. +FIRST_CONTENT_EMIT_ITERS = 2 + +# The runtime-value dump (TRTLLM_MODELING_V2_VALUE_DUMP=1): the metadata of every fast-path and entry decision a step +# takes, recorded on eager steps and captures (replay runs no Python), printed once at exit. +# -------------------------------------------------------------------------------------------------------------------- + +_VALUE_DUMP = os.environ.get("TRTLLM_MODELING_V2_VALUE_DUMP") == "1" +_VALUES: set = set() + + +def _record(*meta: Hashable) -> None: + if _VALUE_DUMP: + _VALUES.add(meta) + + +def _print_values() -> None: + if _VALUES: + rank = os.environ.get("OMPI_COMM_WORLD_RANK", os.environ.get("RANK", "0")) + for meta in sorted(_VALUES, key=repr): + print(f"[modeling_v2 value dump] rank {rank}: {meta}", flush=True) + + +if _VALUE_DUMP: + atexit.register(_print_values) + +# -------------------------------------------------------------------------------------------------------------------- +# Configuration: the text tower and the runtime the target assumes + +# -------------------------------------------------------------------------------------------------------------------- + + +def _is_vl_config(pretrained_config: PretrainedConfig) -> bool: + model_type = getattr(pretrained_config, "model_type", None) + if model_type == "minimax_m3_vl": + return True + if model_type == "minimax_m3": + return False + return any( + "ForConditionalGeneration" in a + for a in (getattr(pretrained_config, "architectures", None) or []) + ) + + +def _text_model_config(model_config: ModelConfig) -> ModelConfig: + """The model config with the language model's ``text_config`` as its pretrained config. + + The checkpoint declares the vision-language wrapper; its ``text_config`` arrives as a plain dict (no typed + sub-config is registered), so it is materialized into a ``PretrainedConfig`` with the wrapper's dtype and + embedding tie carried over. A text config is returned unchanged. + """ + if not _is_vl_config(model_config.pretrained_config): + return model_config + cfg = copy.deepcopy(model_config) + text = getattr(cfg.pretrained_config, "text_config", None) + if text is None: + raise ValueError("MiniMax-M3 VL config has no `text_config`") + if isinstance(text, dict): + out = PretrainedConfig() + for k, v in text.items(): + setattr(out, k, v) + text = out + if ( + getattr(text, "torch_dtype", None) is None + and getattr(cfg.pretrained_config, "torch_dtype", None) is not None + ): + text.torch_dtype = cfg.pretrained_config.torch_dtype + if ( + getattr(text, "tie_word_embeddings", None) is None + and getattr(cfg.pretrained_config, "tie_word_embeddings", None) is not None + ): + text.tie_word_embeddings = cfg.pretrained_config.tie_word_embeddings + return dataclasses.replace(cfg, pretrained_config=text) + + +def _validate_runtime(model_config: ModelConfig) -> None: + """The runtime this target is written for (it is not in the checkpoint): routing declines every other one + (``routing._runtime_mismatch``), so this is the backstop for a construction that did not go through routing. + + The MiniMax-M3 MSA backend (it owns the step metadata and the index-K cache), the fused five-way QKV/index + projection or the separate QKV and index projections, an FP8 indexer and an FP8 or NVFP4 main KV cache. + """ + sparse = model_config.sparse_attention_config + if ( + sparse is None + or getattr(sparse, "algorithm", None) != "minimax_m3" + or sparse.implementation != "msa" + ): + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.algorithm='minimax_m3' with implementation='msa'" + ) + if getattr(sparse, "indexer_kv_dtype", None) not in ("fp8", "nvfp4"): + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.indexer_kv_dtype='fp8' (or the agg opt-in " + "'nvfp4', AGG_NVFP4_INDEXK)" + ) + mode = model_config.quant_config.quant_mode + if mode is None or not (mode.has_fp8_kv_cache() or mode.has_fp4_kv_cache()): + raise ValueError("this MiniMax-M3 target requires an FP8 or NVFP4 KV cache") + + +def _sparse_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = cfg.sparse_attention_config["sparse_attention_freq"] + return [i for i, f in enumerate(freq) if int(f) != 0] + + +def _moe_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = getattr(cfg, "moe_layer_freq", None) + if freq is None: + return list(range(cfg.num_hidden_layers)) + return [i for i, f in enumerate(freq) if int(f) != 0] + + +class _Mxfp8GemmState(nn.Module): + """The MXFP8 GEMM backend state the engine's warmup drives. + + The engine finds MXFP8 projections by a module's ``quant_method`` and, for every one, opts it into native GEMM + tuning and (on decode CUDA graphs) the auto-dispatched FlashInfer GEMM, then marks each tuned. Every projection of + this target dispatches identically, so one method object carries that state for all of them; the forward reads it + at each call (``_mx_mode``) and passes the resolved backend to the GEMM entries. + """ + + def __init__(self) -> None: + super().__init__() + self.quant_method = MXFP8LinearMethod() + + +@dataclass(frozen=True) +class MxMode: + """The MXFP8 GEMM dispatch of one call, resolved from ``_Mxfp8GemmState`` and the engine's contexts.""" + + decode_graph: bool # inside a decode CUDA-graph warmup or capture + autotune: bool # inside the engine's FlashInfer MXFP8 tuning pass + tuned_quantize_gemm: ( + bool # per-bucket tuned quantize + GEMM (decode graphs, tune_decode_graph_backends) + ) + flashinfer: bool # FlashInfer's MXFP8 GEMM on the quantized activation + native_tuning: ( + bool # the native GEMM through the autotuner (warmup, before its tactics are registered) + ) + + +def _mx_mode(state: _Mxfp8GemmState) -> MxMode: + method = state.quant_method + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + autotune = flashinfer_mxfp8_autotune_active() + return MxMode( + decode_graph=decode_graph, + autotune=autotune, + tuned_quantize_gemm=method.tune_decode_graph_backends and decode_graph, + flashinfer=method.backend == "flashinfer" + or ( + method.backend == "auto" + and (autotune or (method._flashinfer_autotuned and decode_graph)) + ), + native_tuning=method.needs_native_autotune, + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# Projections: the MXFP8 GEMM dispatch (``linear_path``) and its hand-offs +# -------------------------------------------------------------------------------------------------------------------- + + +def _mxfp8_linear( + core, x: torch.Tensor, w: torch.Tensor, s: torch.Tensor, mode: MxMode +) -> torch.Tensor: + """One MXFP8 projection on the engine-tuned path: MXFP8 quantize + GEMM, backend per ``mode``.""" + _record( + "m3_mxfp8_linear", + tuple(w.shape), + mode.tuned_quantize_gemm, + mode.flashinfer, + mode.native_tuning, + ) + return m3_mxfp8_linear( + x.contiguous(), + w, + s, + core.mx_alpha, + torch.bfloat16, + mode.tuned_quantize_gemm, + mode.autotune, + mode.flashinfer, + mode.native_tuning, + ) + + +def _gemv_variants(n: int, k: int, push: Optional[str], world: int): + """Every (rows, bf16 activation, push (world, bucket tokens, mode)) call the decode dispatch routes to an [N, K] + projection's GEMV: an MXFP8 activation on the rows ``MX_GEMV_ROWS`` gives it, a bf16 one up to + ``MX_GEMV_BF16_MAX_TOKENS``; each pushing when the projection's boundary push covers the rows.""" + push_rows = {"attn": ATTN_PUSH.keys, "dense": DENSE_PUSH.keys, "moe": MOE_PUSH.keys}.get( + push, frozenset() + ) + variants = [] + for rows in range(1, DECODE_MAX_TOKENS + 1): + for x_bf16, served in ( + (False, rows in MX_GEMV_ROWS.get((n, k), ())), + (True, rows <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0)), + ): + if not served: + continue + variants.append((rows, x_bf16, None)) + if rows in push_rows: + bucket = next(b for b in m3_mc_allreduce.BUCKETS if rows <= b) + variants.append((rows, x_bf16, (world, bucket, AR_PUSH_MODE))) + # VR_GEMV48: the generation steps past the decode bound, on the (quantized) MXFP8 activation, never pushing. + variants.extend((rows, False, None) for rows in sorted(GEMV48_ROWS.get((n, k), ()))) + return variants + + +def _precompile_cfg( + weight: torch.Tensor, + weight_scale: torch.Tensor, + m: int, + cfg: tuple, + use_pdl: bool, + ar: Optional[Tuple[int, int, int]] = None, + prod: bool = False, +) -> Optional[tuple]: + """``mx_gemv.precompile`` for a call with this target's (split, ring) override ``cfg`` (``mx_gemv_cfg``): the + shared precompile resolves the shared table's config only. ``prod``: the call with the producer epilogue + (``QKV_PRODUCER_FOLD``). Compiles without launching, outside capture, also while compiles are frozen; returns the + kernel's key, or None when such a call is declined.""" + import cuda.bindings.driver as cuda_driver + + n, k = weight.shape + push = None if ar is None else (0, 0, 0, *ar) + resolved = mx_gemv._resolve(m, n, k, False, True, cfg, use_pdl, push, False, prod) + if resolved is None: + return None + key, runtime_ar = resolved + if mx_gemv.is_compiled(key): + return key + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("_precompile_cfg allocates and compiles: call it outside capture") + dev = weight.device + x = torch.zeros(m, k, dtype=torch.float8_e4m3fn, device=dev) + xsf = torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=dev) + y = torch.empty(m, n, dtype=torch.bfloat16, device=dev) + args = mx_gemv._views(x, xsf, weight, weight_scale, y, False) + stream = cuda_driver.CUstream(torch.cuda.current_stream(dev).cuda_stream) + mx_gemv._compile(key, args, m, runtime_ar, stream, mx_gemv._extra_args(dev), mx_gemv.PROD_OFF) + return key + + +_MX_TAC_ENGAGED: set = set() + + +def _note_mx_tac(kind: str, n: int, k: int, rows: int) -> None: + """MX-TAC's engagement line, once per (activation kind, projection, rows).""" + key = (kind, n, k, rows) + if key not in _MX_TAC_ENGAGED: + _MX_TAC_ENGAGED.add(key) + logger.info( + f"[M3 sm_107 tp4] MX-TAC engaged: pinned MXFP8 decision (quantize tactic " + f"{MXFP8_DECODE_QUANT_TACTIC}, GEMM tactic {MXFP8_DECODE_GEMM_TACTIC}) " + f"for the [{n}, {k}] projection at {rows} rows, {kind} activation" + ) + + +def _uncompiled_note() -> str: + key = mx_gemv.last_uncompiled + return "" if key is None else f" (its kernel {key} was not compiled before warmup)" + + +def _fold_cfg(n: int, k: int, rows: int) -> Optional[tuple]: + """The QKV GEMV config of a call ``QKV_PRODUCER_FOLD`` takes (the sparse QKV + index projection at <= 8 rows on + its split-8 override), or None when the producer epilogue does not serve it.""" + cfg = mx_gemv_cfg(n, k, rows) + if not QKV_PRODUCER_FOLD or n != mx_gemv.PROD_N or rows > 8 or cfg is None or cfg[0] != 8: + return None + return cfg + + +class _DeferredGemv: + """``QKV_PRODUCER_FOLD``: a sparse layer's QKV + index MXFP8 GEMV, recorded instead of launched. + + ``_linear(..., defer=True)`` returns one on the decode GEMV path of the 39-head packed projection at the rows + ``_fold_cfg`` serves, with an MXFP8 hand-off. The producer call site either runs the GEMV with the producer in its + epilogue (``_qkv_producer_fold``) or calls ``materialize`` (the plain GEMV) before anything reads the packed row; + ``placeholder`` (unwritten, the packed shape) stands for that row until then and is never read.""" + + def __init__(self, act, act_scale, weight, weight_scale, cfg, use_pdl): + self.placeholder = torch.empty( + act.shape[0], weight.shape[0], dtype=torch.bfloat16, device=act.device + ) + self.act, self.act_scale = act, act_scale + self.weight, self.weight_scale = weight, weight_scale + self.cfg = cfg + self.use_pdl = use_pdl + # Set once the record ran, fused (the producer GEMV) or plain (``materialize``). + self.done = False + + def materialize(self) -> torch.Tensor: + """Launch the plain GEMV this record stands for; returns the packed row ``_linear`` would have returned.""" + self.done = True + rows, (n, k) = self.act.shape[0], self.weight.shape + output = m3_mx_gemv( + self.act, + self.act_scale, + self.weight, + self.weight_scale, + True, + self.cfg, + self.use_pdl, + None, + ) + if output is None: + raise RuntimeError( + f"the deferred [{n}, {k}] GEMV declined at {rows} rows" + _uncompiled_note() + ) + taken(MX_GEMV_PATH, rows, f"[{n}, {k}] mxfp8 in, split {self.cfg[0]}") + return output + + +def _linear( + core, + x: torch.Tensor, + wk: str, + sk: str, + low_m_gemm: bool, + push: Optional[tuple] = None, + defer: bool = False, +) -> Tuple[Any, bool]: + """One MXFP8 projection ``x @ w[wk]^T``; returns (output, whether its GEMV pushed into ``push``). + + Outside decode CUDA graphs: the engine-tuned MXFP8 quantize + GEMM. Inside them the projection's (N, K) and the + step's rows pick one path (``linear_path``): + + 1. an activation its producer already quantized to MXFP8 (``m3_prequant.SLOT``, taken here): the tcgen05 decode + GEMV on the rows it serves, else the decode-graph MXFP8 GEMM on that activation (AutoTuner-selected backend + per token bucket); + 2. a bf16 activation on the rows the GEMV serves quantizing in its prologue; + 3. otherwise the engine-tuned MXFP8 quantize + GEMM. + + A kernel declining rows its path covers is a miss; the chain then continues down the list. ``push`` (an + ``m3_mx_gemv`` ``ar`` tuple) is the boundary all-reduce exchange the GEMV writes its rows into. ``low_m_gemm`` + is accepted for the call sites' symmetry (``linear_path``): no TP4 low-M GEMM configuration is built. ``defer`` + (``QKV_PRODUCER_FOLD``, the sparse layers' packed QKV + index projection): on the rows ``_fold_cfg`` serves the + GEMV is not launched; a ``_DeferredGemv`` is returned for the producer call site. + """ + w, s = core.w[wk], core.w[sk] + x = x.contiguous() + mode = _mx_mode(core.mxfp8) + # F3: an F3Push target is used only by the MX-TAC pinned GEMM branches below; no other path sees it. + f3 = push if isinstance(push, F3Push) else None + if f3 is not None: + push = None + # F3: the attention boundary's MXFP8 norm, taken where the stock path is the pinned GENERIC quantize + GEMM. + f3mx = _F3_MX.take(x) if TP2_F3 and _F3_MX.pending is not None else None + # TP2W_S2_TAIL: the fused MoE tail's MXFP8 activation of x, used where the stock path quantizes x itself before + # the same GEMM (bit-identical operands). Decode graphs (tuned_quantize_gemm) take it too: their pinned MX-TAC + # branch below quantizes with m3_mxfp8_pinned_quantize, the bytes the fused tail writes. + s2 = ( + _S2_MX.take(x) + if TP2W_S2_TAIL and (mode.decode_graph or not mode.tuned_quantize_gemm) + else None + ) + if not mode.decode_graph: + prequant = _CTXNORMQ_MX.take(x) if CTXNORMQ and not mode.tuned_quantize_gemm else None + if prequant is not None: + _sp_ovl_wait(x) + if x.data_ptr() == _COMBINE_MXFP8_STATE.get("src"): + _COMBINE_MXFP8_STATE["src"] = None + m3_report.work( + "TP2WIDE_COMBINE_MXFP8", + f"o_proj took the combine's MXFP8 input, {x.shape[0]} rows", + ) + if "taken" not in _COMBINE_MXFP8_STATE: + _COMBINE_MXFP8_STATE["taken"] = True + logger.info( + "[M3.1] TP2WIDE_COMBINE_MXFP8 taken: o_proj consumed the combine's MXFP8 input " + f"({x.shape[0]} rows)" + ) + # Rule-1b agg prefill port: opt-in until its agg quality gates pass. The same + # band dispatch consumes either the existing prequant pair or ordinary BF16. + if agg_prefill_cublaslt.ENABLED and not is_in_breakable_cuda_graph(): + pinned = agg_prefill_cublaslt.try_gemm(x, w, s, core.mx_alpha, prequant) + if pinned is not None: + m3_report.work( + "AGG_PREFILL_CUBLASLT", f"N={w.shape[0]} K={w.shape[1]} M={x.shape[0]}" + ) + return pinned, False + if prequant is not None: + return _ctxnormq_mxfp8_gemm(core, prequant, w, s, mode), False + if s2 is not None: + return _ctxnormq_mxfp8_gemm(core, s2, w, s, mode), False + return _mxfp8_linear(core, x, w, s, mode), False + use_pdl = get_env_enable_pdl() + rows = x.shape[0] + n, k = w.shape + prequant = m3_prequant.SLOT.take(x) + if ( + f3mx is not None + and s2 is None + and prequant is None + and MX_TAC_TP4 + and core.mxfp8.quant_method.tune_decode_graph_backends + and linear_path(n, k, rows, False, low_m_gemm) == GENERIC_GEMM_PATH + ): + # The stock bf16 branch below is m3_mxfp8_pinned_quantize + this GEMM: the boundary wrote those bytes. + _record("linear_f3", (n, k), rows) + output = m3_mxfp8_pinned_gemm( + f3mx[0], f3mx[1], w, s, torch.bfloat16, MXFP8_DECODE_GEMM_TACTIC + ) + _note_mx_tac("bf16", n, k, rows) + cnt = _F3_STATE["consumed"] = _F3_STATE.get("consumed", 0) + 1 + if cnt & (cnt - 1) == 0: + logger.info( + f"[M3.1] TP2_F3 slot taken n={cnt}: [{n}, {k}] GEMM on the boundary's MXFP8 ({rows} rows)" + ) + return output, False + if ( + s2 is not None + and prequant is None + and MX_TAC_TP4 + and core.mxfp8.quant_method.tune_decode_graph_backends + and linear_path(n, k, rows, False, low_m_gemm) == GENERIC_GEMM_PATH + ): + # The stock bf16 branch below is the pinned MX-TAC quantize + GEMM: the same GEMM on the fused tail's bytes. + # Any other decode-graph path keeps its own quantize (the engine-tuned GEMM may not be the pinned one). + _record("linear_s2", (n, k), rows) + output = m3_mxfp8_pinned_gemm(s2[0], s2[1], w, s, torch.bfloat16, MXFP8_DECODE_GEMM_TACTIC) + _note_mx_tac("bf16", n, k, rows) + cnt = _S2_STATE["consumed"] = _S2_STATE.get("consumed", 0) + 1 + if cnt & (cnt - 1) == 0: + logger.info( + f"[M3.1 agg] TP2W_S2_TAIL consumed n={cnt}: [{n}, {k}] GEMM on the fused tail's MXFP8 ({rows} rows)" + ) + return output, False + if prequant is None and x.dtype != torch.bfloat16: + path = GENERIC_GEMM_PATH + else: + path = linear_path(n, k, rows, prequant is not None, low_m_gemm) + _record("linear", (n, k), rows, path, push is not None) + output = None + # MX-TAC: the decode-graph MXFP8 quantize and GEMM at the target's fixed tactics, no AutoTuner. + pinned = MX_TAC_TP4 and core.mxfp8.quant_method.tune_decode_graph_backends + if prequant is not None: + act, act_scale = prequant + fold_cfg = _fold_cfg(n, k, rows) if defer and path == MX_GEMV_PATH else None + if fold_cfg is not None: + return _DeferredGemv(act, act_scale, w, s, fold_cfg, use_pdl), False + if path == MX_GEMV_PATH: + cfg = mx_gemv_cfg(n, k, rows) + output = m3_mx_gemv(act, act_scale, w, s, True, cfg, use_pdl, push) + if output is not None: + split = f", split {cfg[0]}" if cfg is not None else "" + taken(path, rows, f"[{n}, {k}] mxfp8 in{split}") + return output, push is not None and mx_gemv.last_pushed + missed( + path, + rows, + f"the GEMV declined the [{n}, {k}] projection's MXFP8 activation" + + _uncompiled_note(), + ) + if path == PREQUANT_GEMM_PATH: + uncovered(MX_GEMV_PATH, f"the [{n}, {k}] projection at rows the GEMV does not serve") + if pinned: + if f3 is not None and f3.accepts(n, k, rows): + f3.run(act, act_scale, w, s) + _note_mx_tac("mxfp8", n, k, rows) + return f3.out, True + output = m3_mxfp8_pinned_gemm( + act, act_scale, w, s, torch.bfloat16, MXFP8_DECODE_GEMM_TACTIC + ) + _note_mx_tac("mxfp8", n, k, rows) + return output, False + return m3_mxfp8_gemm(act, act_scale, w, s, torch.bfloat16, mode.autotune), False + if path == MX_GEMV_BF16_PATH: + output = m3_mx_gemv_bf16(x, w, s, True, None, use_pdl, push) + if output is not None: + taken(path, rows, f"[{n}, {k}] bf16 in") + return output, push is not None and mx_gemv.last_pushed + missed( + path, rows, f"the GEMV declined the [{n}, {k}] projection's input" + _uncompiled_note() + ) + if path == GENERIC_GEMM_PATH: + uncovered( + MX_GEMV_BF16_PATH, + f"the [{n}, {k}] projection's bf16 input beyond the decode GEMV's rows", + ) + if pinned and x.dtype == torch.bfloat16: + act, act_scale = m3_mxfp8_pinned_quantize(x, MXFP8_DECODE_QUANT_TACTIC) + if f3 is not None and f3.accepts(n, k, rows): + f3.run(act, act_scale, w, s) + _note_mx_tac("bf16", n, k, rows) + return f3.out, True + output = m3_mxfp8_pinned_gemm( + act, act_scale, w, s, torch.bfloat16, MXFP8_DECODE_GEMM_TACTIC + ) + _note_mx_tac("bf16", n, k, rows) + return output, False + # Stock's decode-graph dispatch for this shape (the tuned quantize + GEMM). + return _mxfp8_linear(core, x, w, s, mode), False + + +def _native_swiglu(core, gu: torch.Tensor, dense: bool) -> Optional[torch.Tensor]: + """NATIVE_SWIGLU: the SwiGLU-OAI activation of ``gu`` [rows, 2d] by the native op, or None (the caller's path). + Served for the model's constants (limit 7, alpha 1.702, beta 1), contiguous bf16, outside torch.compile; the dense + MLP's 88..2048-row band only with NATIVE_ALL_BANDS. The vector count nv is chosen by rows.""" + site = "dense" if dense else "shared" + if not NATIVE_SWIGLU: + return _native_dispatch( + "NATIVE_SWIGLU", site, "killed (TRTLLM_M31_DISABLE_NATIVE_SWIGLU=1)" + ) + if is_torch_compiling(): + return _native_dispatch("NATIVE_SWIGLU", site, "under torch.compile") + if (core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) != (7.0, 1.702, 1.0): + return _native_dispatch("NATIVE_SWIGLU", site, "SwiGLU constants other than (7, 1.702, 1)") + if gu.dim() != 2 or gu.dtype != torch.bfloat16 or not gu.is_contiguous(): + return _native_dispatch("NATIVE_SWIGLU", site, "input is not a contiguous 2-D bf16 tensor") + rows = int(gu.shape[0]) + if dense and not NATIVE_ALL_BANDS and 49 < rows < 4096: + return _native_dispatch( + "NATIVE_SWIGLU", site, "the 50..4095-row band (NATIVE_ALL_BANDS off)" + ) + op = _native_op("minimax_m31_swiglu_oai_vec") + if op is None: + return _native_dispatch("NATIVE_SWIGLU", site, "the op is not registered") + nv = 1 if rows <= 2048 else (2 if rows <= 8192 else 4) + _native_dispatch("NATIVE_SWIGLU", site) + _record("m31_native_swiglu", rows, dense) + taken("m31_native_swiglu", rows, f"{'dense' if dense else 'shared'} nv {nv}") + return op(gu, get_env_enable_pdl(), nv) + + +def _dense_swiglu(core, gu: torch.Tensor) -> torch.Tensor: + """A dense MLP's SwiGLU-OAI activation: this target's C++ op (``DENSE_SWIGLU_CPP``, bit for bit the Triton kernel, + PDL-launched with an early release) up to its certified rows, in any graph or eager step; else the Triton + kernel.""" + native = _native_swiglu(core, gu, True) + if native is not None: + return native + rows = int(gu.shape[0]) + path = DENSE_SWIGLU_CPP_PATH + max_rows = max(cell.spec["rows"] for cell in m3_dense_swiglu.CELLS) + if DENSE_SWIGLU_CPP and not is_torch_compiling(): + if rows > max_rows: + uncovered(path, f"the dense swigluoai op: {rows} rows, past its certified {max_rows}") + else: + act = m3_dense_swiglu( + gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta, get_env_enable_pdl() + ) + if act is not None: + taken(path, rows) + return act + uncovered(path, "the gate_up layout") + return m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + + +# -------------------------------------------------------------------------------------------------------------------- +# Layer boundaries: all-reduce + residual + RMSNorm +# -------------------------------------------------------------------------------------------------------------------- + + +def _can_mc(x, folded: bool = False) -> bool: + """Whether the generic boundary takes the multicast all-reduce + residual + RMSNorm (``GENERIC_MC_AR_MAX_TOKENS``, + with the shared + routed add folded in ``GENERIC_MC_AR_FOLDED_MAX_TOKENS``).""" + return ( + isinstance(x, torch.Tensor) + and x.dim() == 2 + and x.shape[0] <= (GENERIC_MC_AR_FOLDED_MAX_TOKENS if folded else GENERIC_MC_AR_MAX_TOKENS) + and x.dtype == torch.bfloat16 + and x.is_contiguous() + ) + + +def _mc_allreduce_norm(core, x, x2, residual, norm_w, *, quant=False, prepushed=0, capture=None): + """The multicast all-reduce + optional second addend + residual + RMSNorm (``m3_mc_allreduce_norm``), or None + without a workspace or for operands the kernel does not take (the same on every rank). With ``quant`` the normed + rows are also published, MXFP8-quantized, to ``m3_prequant.SLOT`` for the next MXFP8 projection; ``capture`` = + (out, fc_norm weight, eps): the boundary also writes the Eagle3 capture norm of the updated residual. Up to + ``MC_AR_CLUSTER_MAX_ROWS`` rows the one-shot 8-CTA cluster form runs; ``MC_AR_CARVEOUT`` sets its shared-memory + carveout.""" + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + if workspace is None: + return None + _record("m3_mc_allreduce_norm", x.shape[0], x2 is not None, quant, prepushed) + return m3_mc_allreduce.m3_mc_allreduce_norm( + x, + residual.contiguous(), + norm_w, + core.rms_eps, + workspace, + use_pdl=get_env_enable_pdl(), + x2=x2, + quant=quant, + prepushed=prepushed, + oneshot_cluster=x.shape[0] <= MC_AR_CLUSTER_MAX_ROWS, + carveout=MC_AR_CARVEOUT, + capture=capture, + ) + + +# The CTXNORMQ hand-offs: the boundary's MXFP8 activation for the first prefill MXFP8 projection reading the normed +# rows, and its NVFP4 form for the routed experts. Separate from ``m3_prequant.SLOT`` so only a CTXNORMQ boundary +# feeds the prefill path. +_CTXNORMQ_MX = m3_prequant.PrequantSlot() +_CTXNORMQ_FP4 = m3_prequant.PrequantSlot() +_CTXNORMQ_STATE = {} +# TP2AGG_ROUTER_F32: the boundary's (normed, fp32 copy) for the router of the same MoE layer. +_CTXNORMQ_F32 = m3_prequant.PrequantSlot() +# TP2AGG_SP: the pre-MoE boundary's gathered full-T router logits for the keyed view. +_SP_LOGITS = m3_prequant.PrequantSlot() + + +def _ctxnormq_op(): + """``torch.ops.trtllm.minimax_m31_ctx_residual_rmsnorm_mxfp8``, or None. A build without the op in its library + loads it from ``tensorrt_llm/libs/libm31_ctxnormq.so`` when that file exists (an overlay); otherwise the boundary + keeps the fused all-reduce. Decided once per process.""" + if "op" not in _CTXNORMQ_STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_ctx_residual_rmsnorm_mxfp8", None) + if op is None: + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_ctxnormq.so") + if os.path.exists(lib): + torch.ops.load_library(lib) + op = getattr(torch.ops.trtllm, "minimax_m31_ctx_residual_rmsnorm_mxfp8", None) + _CTXNORMQ_STATE["op"] = op + logger.info( + f"[M3 dispatch] CTXNORMQ prefill boundary norm + quantize: " + f"{'engaged' if op is not None else 'not taken: the op is not built'} (rows >= {CTXNORMQ_MIN_ROWS})" + ) + return _CTXNORMQ_STATE["op"] + + +def _ctxnormq_f32_op(): + """``torch.ops.trtllm.minimax_m31_ctx_residual_rmsnorm_mxfp8_f32`` (TP2AGG_ROUTER_F32), or None: switched off, or + neither the build nor the overlay ``tensorrt_llm/libs/libm31_agg_ctxnormq_f32.so`` has it. Decided once per + process.""" + if "f32" not in _CTXNORMQ_STATE: + op = ( + getattr(torch.ops.trtllm, "minimax_m31_ctx_residual_rmsnorm_mxfp8_f32", None) + if ROUTER_F32 + else None + ) + if ROUTER_F32 and op is None: + import tensorrt_llm + + lib = os.path.join( + os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_agg_ctxnormq_f32.so" + ) + if os.path.exists(lib): + torch.ops.load_library(lib) + op = getattr(torch.ops.trtllm, "minimax_m31_ctx_residual_rmsnorm_mxfp8_f32", None) + m31_fake_ops.register() + _CTXNORMQ_STATE["f32"] = op + why = "switched off" if not ROUTER_F32 else "the op is not built" + logger.info( + f"[M3 dispatch] TP2AGG_ROUTER_F32 router fp32 input from the CTXNORMQ boundary: " + f"{'engaged' if op is not None else 'not taken: ' + why}" + ) + return _CTXNORMQ_STATE["f32"] + + +def _sp_supports(num_tokens: int) -> bool: + """Whether a step of ``num_tokens`` rows is in TP2AGG_SP's band (the workspace still has to exist).""" + return TP2AGG_SP and SP_MIN_ROWS <= num_tokens <= SP_MAX_ROWS + + +def _sp_own_rows(num_tokens: int, rank: int) -> Tuple[int, int]: + """(offset, count) of the rows ``rank`` (0 or 1) owns in a TP2AGG_SP step: rank 0 [0, S), rank 1 [S, T), S the + 128-row boundary nearest T / 2 (tp2ctx_ipc ``split_row``; T / 2 when T % 256 == 0).""" + tile = SP_SF_TILE_ROWS + split = min(max((num_tokens // 2 + tile // 2) // tile * tile, tile), num_tokens - 1) + return (0, split) if rank == 0 else (split, num_tokens - split) + + +def _sp_own_sf_bytes(num_tokens: int, rank: int, hidden: int) -> Tuple[int, int]: + """(start, end) of ``rank``'s bytes in the swizzled MXFP8 scale factors of a T-row step: the 128-row tiles of its + rows; rank 1's last tile includes the pad rows up to the step's 128-row multiple (tp2ctx_ipc ``own_sf_slice``).""" + per = (hidden // 32 + 3) // 4 * 4 + split = _sp_own_rows(num_tokens, 0)[1] + rows_pad = (num_tokens + SP_SF_TILE_ROWS - 1) // SP_SF_TILE_ROWS * SP_SF_TILE_ROWS + return (0, split * per) if rank == 0 else (split * per, rows_pad * per) + + +def _sp_logits_bytes(num_tokens: int, rank: int) -> Tuple[int, int]: + """(start, end) of ``rank``'s owned rows in the "logits" gather view ([T, 128] fp32, whole rows).""" + off, count = _sp_own_rows(num_tokens, rank) + return off * SP_LOGITS_ROW_BYTES, (off + count) * SP_LOGITS_ROW_BYTES + + +class _SpOvl(NamedTuple): + """An SP boundary's asynchronous push (SP_OVL): the event the peer's rows wait on and the two row / scale ranges.""" + + event: Any + own_rows: Tuple[int, int] + peer_rows: Tuple[int, int] + own_sf: slice + peer_sf: slice + + +def _sp_once(key, message: str) -> None: + """One info line per key (engagement lines a gate job greps).""" + noted = _SP_STATE.setdefault("noted", set()) + if key not in noted: + noted.add(key) + logger.info(message) + + +def _sp_load(op_name: str, lib_name: str): + """``torch.ops.trtllm.``, loading ``tensorrt_llm/libs/`` first when the build lacks it.""" + if not hasattr(torch.ops.trtllm, op_name): + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", lib_name) + if os.path.exists(lib): + torch.ops.load_library(lib) + return getattr(torch.ops.trtllm, op_name, None) + + +def _sp_init(core) -> None: + """Load the peer ops and build the IPC workspace (collective: both TP ranks, at weight load, outside capture, + before the KV cache is sized), plus the zero-copy NORMQ out= and SP_ADDNORM ops. Once per process.""" + if not TP2AGG_SP or "ws" in _SP_STATE: + return + ws, why = None, None + if getattr(core.mapping, "tp_size", 1) != 2: + why = f"tp_size {getattr(core.mapping, 'tp_size', None)} != 2" + elif _sp_load("minimax_m31_tp2ctx_ipc_barrier", _SP_LIB) is None: + why = f"no peer ops (tensorrt_llm/libs/{_SP_LIB})" + else: + try: + from tensorrt_llm._torch._experimental.modeling_v2.catalog.comm import tp2ctx_ipc + + ws = tp2ctx_ipc.get_workspace( + core.mapping, + max_tokens=SP_MAX_ROWS, + hidden=core.hidden, + device=torch.device("cuda", torch.cuda.current_device()), + ) + except (ImportError, AttributeError) as e: + why = f"no IPC workspace ({type(e).__name__}: {str(e)[:120]})" + _SP_STATE["ws"] = ws + if ws is None: + logger.info(f"[M3.1 agg] TP2AGG_SP not taken: {why}") + return + normq_out = _sp_load("minimax_m31_ctx_residual_rmsnorm_mxfp8_out", _SP_NORMQ_OUT_LIB) + addnorm = None + if SP_ADDNORM and normq_out is not None and not SP_CHECK: + addnorm = _sp_load("minimax_m31_tp2ctx_peer_add_normq_out", _SP_ADDNORM_LIB) + _SP_STATE["normq_out"], _SP_STATE["addnorm"] = normq_out, addnorm + logger.info( + f"[M3.1 agg] TP2AGG_SP workspace ready: NORMQ out= {'on' if normq_out is not None else 'off'}, " + f"SP_ADDNORM {'on' if addnorm is not None else 'off'}, SP_CHECK {'on' if SP_CHECK else 'off'}" + ) + + +def _sp_ws(x: torch.Tensor): + """The workspace when a boundary over ``x``'s rows runs sequence-parallel (eager, rows in the band), else None.""" + ws = _SP_STATE.get("ws") + if ( + ws is None + or x.dim() != 2 + or not _sp_supports(x.shape[0]) + or not ws.supports(x.shape[0], nccl_only=False) + or torch.cuda.is_current_stream_capturing() + or is_torch_compiling() + or is_in_breakable_cuda_graph() + ): + return None + return ws + + +def _sp_consumer_ws(core, x): + """SP publishes a keyed placeholder, so every consumer must accept its prequantized products.""" + if not core.fused_projection: + return None + mode = _mx_mode(core.mxfp8) + if mode.decode_graph or mode.tuned_quantize_gemm: + return None + return _sp_ws(x) + + +def _sp_count(rows: int, owned: int, kind: str) -> None: + """Engagement: per step-size band, logged at boundary counts 1, 2, 4, ... (read the last line per band).""" + band = "<=8K" if rows <= 8192 else "<=12K" if rows <= 12288 else ">12K" + counts = _SP_STATE.setdefault("counts", {}) + counts[band] = n = counts.get(band, 0) + 1 + if n & (n - 1) == 0: + logger.info( + f"[M3.1 agg] TP2AGG_SP taken band={band} n={n} (rows {rows}, owned {owned}, {kind})" + ) + m3_report.work("TP2AGG_SP", f"{rows} rows, {kind}") + + +def _sp_put(ws, kind: str, rows: int, owned: torch.Tensor) -> torch.Tensor: + """Copy this rank's owned rows of ``kind`` into the full-T gather view; returns the view.""" + first, count = ws.own_rows(rows) + view = ws.gather_view(kind, rows) + if kind == "mxfp8_sf": + sl = ws.own_sf_slice(rows) + src = owned.view(torch.uint8).reshape(-1) + if src.numel() != sl.stop - sl.start: + raise RuntimeError( + f"TP2AGG_SP: MXFP8 scale factors {src.numel()} bytes != own_sf_slice {sl}" + ) + view[sl].copy_(src) + elif kind == "logits": + view[first : first + count].copy_(owned) + else: + view[first : first + count].copy_(owned.view(torch.uint8).view(count, -1)) + return view + + +def _sp_unshard(residual, x): + """The full-T residual: an owned-only one (inside an SP step) is gathered from both ranks first (through the + workspace's bf16 view, then cloned: the next gather reuses the view). Anything else is returned as is.""" + ws = _SP_STATE.get("ws") + if ( + ws is None + or not isinstance(residual, torch.Tensor) + or not isinstance(x, torch.Tensor) + or residual.dim() != 2 + or x.dim() != 2 + or residual.shape[0] == x.shape[0] + or residual.shape[0] != ws.own_rows(x.shape[0])[1] + ): + return residual + rows = x.shape[0] + first, count = ws.own_rows(rows) + view = ws.gather_view("bf16", rows) + view[first : first + count].copy_(residual) + ws.push_owned(("bf16",), rows) + _sp_once( + ("unshard", rows), f"[M3.1 agg] TP2AGG_SP residual unshard: rows {rows} (owned {count})" + ) + return view.clone() + + +def _sp_ovl_take(x: torch.Tensor): + """The SP_OVL push pending for the keyed tensor ``x`` (cleared), or None.""" + pending, _SP_STATE["ovl"] = _SP_STATE.get("ovl"), None + if pending is None or x.dim() != 2 or pending[0] != x.data_ptr(): + if pending is not None: + torch.cuda.current_stream().wait_event(pending[1].event) + return None + return pending[1] + + +def _sp_ovl_wait(x: torch.Tensor) -> None: + """A consumer that reads the whole gathered activation: wait for a pending SP_OVL push first.""" + if _SP_STATE.get("ovl") is not None: + ovl = _sp_ovl_take(x) + if ovl is not None: + torch.cuda.current_stream().wait_event(ovl.event) + + +def _sp_boundary(core, ws, op, x, residual, norm_w, fp4_scale, allreduce, router_w, qkv_a1_next): + """The CTXNORMQ boundary on this rank's owned rows: (the keyed bf16 view, the owned updated residual). The + gathered MXFP8 (+ NVFP4) products and, at a pre-MoE boundary, the gathered router logits are published for the + keyed view, so the consumers take them as today.""" + rows = x.shape[0] + first, count = ws.own_rows(rows) + pre_moe = fp4_scale is not None + slot = ws.input_slot(rows) + ref = None + if SP_CHECK: + ref = m3_allreduce(x.clone(), core.allreduce if allreduce is None else allreduce, None)[ + first : first + count + ] + if x.data_ptr() != slot.data_ptr(): + slot.copy_(x) + normq_out = _SP_STATE.get("normq_out") + addnorm = _SP_STATE.get("addnorm") if normq_out is not None else None + if addnorm is not None: + owned = torch.empty((count, x.shape[1]), dtype=x.dtype, device=x.device) + else: + owned = ws.reduce_scatter_owned(slot) + if ref is not None and addnorm is None: + # The reduce-scatter's own rows (SP_ADDNORM fuses the peer add into its norm instead: its bitwise oracle is + # mb_addnorm, so a check run kills SP_ADDNORM). + diff = int((owned.view(torch.int16) != ref.view(torch.int16)).sum()) + checked = _SP_STATE["checked"] = _SP_STATE.get("checked", 0) + 1 + if diff or checked <= 4: + logger.info( + f"[M3.1 agg] TP2AGG_SP check: rows {rows} owned reduce-scatter differs from the all-reduce in " + f"{diff} elements (boundaries checked {checked})" + ) + res = (residual[first : first + count] if residual.shape[0] == rows else residual).contiguous() + kinds = ["mxfp8", "mxfp8_sf"] + (["nvfp4", "nvfp4_sf"] if pre_moe else []) + pushed = False + x32 = None + fp4 = fp4_sf = None + if normq_out is not None: + # Zero-copy: the NORMQ op writes this rank's products straight into the gather views at the owned rows. + gq, gsf = ws.gather_view("mxfp8", rows), ws.gather_view("mxfp8_sf", rows) + g4 = ws.gather_view("nvfp4", rows) if pre_moe else None + g4sf = ws.gather_view("nvfp4_sf", rows) if pre_moe else None + normed = torch.empty_like(owned) if pre_moe else None + if pre_moe and ROUTER_F32: + # ROUTER_ZC: the fp32 router rows land in the router's full-step buffer (no copy, no zero fill). + x32 = ws.router_owned_input(rows, owned.shape[1]) + _sp_once( + ("router_zc", x32 is not None, rows), + f"[M3.1 agg] TP2AGG_ROUTER_ZC {'taken' if x32 is not None else 'not taken'}: rows {rows}", + ) + if x32 is None: + x32 = torch.empty(owned.shape, dtype=torch.float32, device=owned.device) + outs_args = ( + res, + norm_w, + core.rms_eps, + normed, + gq[first : first + count], + gsf[ws.own_sf_slice(rows)], + None if g4 is None else g4[first : first + count], + None if g4sf is None else g4sf[first : first + count], + fp4_scale, + x32, + ) + if addnorm is not None: + local_ptr, peer_ptr = ws.owned_slot_ptrs(rows) + ws.barrier() + addnorm(local_ptr, peer_ptr, owned, *outs_args, 1, False) + _sp_once( + ("addnorm", rows, pre_moe), + f"[M3.1 agg] TP2AGG_SP_ADDNORM taken: {'pre-MoE' if pre_moe else 'boundary'} rows {rows}", + ) + else: + normq_out(owned, *outs_args, False) + xq, xsf = gq.view(torch.float8_e4m3fn), gsf + if pre_moe: + fp4, fp4_sf = g4, g4sf.reshape(-1) + if x32 is None: + x32 = normed.float() + if SP_PMOVL and rows >= SP_PMOVL_MIN_ROWS: + # SP_PMOVL: the products go now, under the router GEMM; the logits and the barrier follow. + ws.push_owned_async(kinds, rows, engine="sm", barrier=False) + pushed = True + _sp_once(("pmovl", rows), f"[M3.1 agg] TP2AGG_SP_PMOVL taken: pre-MoE rows {rows}") + else: + f32_op = _ctxnormq_f32_op() if pre_moe else None + pdl = get_env_enable_pdl() + if f32_op is not None: + outs = f32_op(owned, res, norm_w, core.rms_eps, fp4_scale, pdl) + else: + outs = op(owned, res, norm_w, core.rms_eps, True, fp4_scale, pdl) + xq = _sp_put(ws, "mxfp8", rows, outs[1]).view(outs[1].dtype) + xsf = _sp_put(ws, "mxfp8_sf", rows, outs[2]).view(outs[2].dtype) + if pre_moe: + fp4 = _sp_put(ws, "nvfp4", rows, outs[3]).view(outs[3].dtype) + fp4_sf = _sp_put(ws, "nvfp4_sf", rows, outs[4]).view(outs[4].dtype).reshape(-1) + x32 = outs[5] if f32_op is not None else outs[0].float() + logits = None + if pre_moe: + # The owned rows' logits from the same full-T router call as today (bitwise per row), gathered as a kind. + logits = _sp_put(ws, "logits", rows, ws.route_owned_full_shape(x32, router_w, rows)) + kinds = ["logits"] if pushed else kinds + ["logits"] + ovl = None + if ( + not pre_moe + and qkv_a1_next + and QKV_A1 + and SP_OVL + and rows >= SP_OVL_MIN_ROWS + and normq_out is not None + ): + event = ws.push_owned_async(kinds, rows) + peer_first = first + count if first == 0 else 0 + ovl = _SpOvl( + event, + (first, count), + (peer_first, rows - count), + ws.own_sf_slice(rows), + ws.peer_sf_slice(rows), + ) + else: + ws.push_owned(kinds, rows) + keyed = ws.gather_view("bf16", rows) + if SP_CHECK: + if ovl is not None: + ws.wait_pending_push() + ovl = None + keyed.fill_(float("nan")) + _CTXNORMQ_MX.publish(keyed, xq, xsf) + _SP_STATE["ovl"] = (keyed.data_ptr(), ovl) if ovl is not None else None + if pre_moe: + _CTXNORMQ_FP4.publish(keyed, fp4, fp4_sf) + _SP_LOGITS.publish(keyed, logits, None) + else: + _CTXNORMQ_FP4.clear() + _SP_LOGITS.clear() + _CTXNORMQ_F32.clear() + _SP_STATE["keyed"] = keyed.data_ptr() + _record("m31_tp2agg_sp", rows, pre_moe, normq_out is not None) + _sp_count(rows, count, "pre-MoE" if pre_moe else "boundary") + return keyed, owned + + +def _sp_keyed(h: torch.Tensor) -> bool: + """Whether ``h`` is the keyed view of this step's last SP boundary (its bf16 rows hold no data).""" + return h.dim() == 2 and _SP_STATE.get("keyed") == h.data_ptr() + + +def _ctxnormq_allreduce_norm( + core, + x, + residual, + norm_w, + fp4_scale, + allreduce=None, + router_w=None, + qkv_a1_next=False, + pre_moe=False, +): + """A prefill boundary as the plain all-reduce + the fused residual add, RMSNorm and quantize, or None (switched + off, fewer than ``CTXNORMQ_MIN_ROWS`` rows, op not built). Returns (normed, updated) like the fused all-reduce; + the MXFP8 activation (and with ``fp4_scale`` the NVFP4 one) wait in the CTXNORMQ slots for their consumers. + ``allreduce``: the all-reduce to run (default the target's AUTO one). TP2AGG_SP: a step in its band runs the + boundary sequence-parallel (``_sp_boundary``; a pre-MoE one needs ``router_w``, the layer's router weight; + ``qkv_a1_next``: the next projection is a sparse layer's QKV, for SP_OVL) and returns (keyed view, owned + residual).""" + if not CTXNORMQ or x.dim() != 2 or x.shape[0] < CTXNORMQ_MIN_ROWS: + return None + op = _ctxnormq_op() + if op is None: + return None + ws = ( + _sp_consumer_ws(core, x) + if TP2AGG_SP and (not pre_moe or SP_PREMOE) and (fp4_scale is None or router_w is not None) + else None + ) + if ws is not None: + return _sp_boundary( + core, ws, op, x, residual, norm_w, fp4_scale, allreduce, router_w, qkv_a1_next + ) + residual = _sp_unshard(residual, x) + reduced = m3_allreduce(x, core.allreduce if allreduce is None else allreduce, None) + _record("m31_ctxnormq", x.shape[0], fp4_scale is not None) + f32_op = _ctxnormq_f32_op() if fp4_scale is not None else None + if f32_op is not None: + normed, xq, xsf, xfp4, xfp4_sf, normed_f32 = f32_op( + reduced, residual.contiguous(), norm_w, core.rms_eps, fp4_scale, get_env_enable_pdl() + ) + _CTXNORMQ_F32.publish(normed, normed_f32, None) + else: + normed, xq, xsf, xfp4, xfp4_sf = op( + reduced, + residual.contiguous(), + norm_w, + core.rms_eps, + True, + fp4_scale, + get_env_enable_pdl(), + ) + _CTXNORMQ_F32.clear() + _CTXNORMQ_MX.publish(normed, xq, xsf) + if fp4_scale is not None: + _CTXNORMQ_FP4.publish(normed, xfp4, xfp4_sf) + else: + _CTXNORMQ_FP4.clear() + m3_report.work( + "CTXNORMQ", f"{x.shape[0]} rows, NVFP4 {'on' if fp4_scale is not None else 'off'}" + ) + return normed, reduced + + +def _ctxnormq_mxfp8_gemm(core, prequant, w, s, mode): + """The GEMM ``m3_mxfp8_linear`` runs after its own quantize, on the CTXNORMQ activation (bit-identical operands).""" + act, act_scale = prequant + _record("m31_ctxnormq_gemm", tuple(w.shape), act.shape[0], mode.flashinfer, mode.native_tuning) + if mode.flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8(act, act_scale, w, s, torch.bfloat16) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if mode.native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, w, s, core.mx_alpha, torch.bfloat16) + + +# TP2WIDE_COMBINE_MXFP8: the combine's MXFP8 copy is written only for calls of at least this many rows (msa_prefill +# reads the sink's ``min_rows``); at 8K rows the extra epilogue costs more than o_proj's quantize saves (tp2ctx +# A/B). The agg target's prefill chunks are 16K rows. +_COMBINE_MXFP8_MIN_ROWS = 12288 +_COMBINE_MXFP8_STATE: dict = {} +_COMBINE_MXFP8_CHECK = os.environ.get("TRTLLM_M31_TP2WIDE_COMBINE_MXFP8_CHECK") == "1" + + +def _combine_mxfp8_allocate(out: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor] | None: + """Give the context combine views into the armed full-step quantized buffers.""" + pending = _COMBINE_MXFP8_STATE.get("pending") + if pending is None: + return None + output, context_rows = pending["output"], pending["context_rows"] + if out.data_ptr() != output.data_ptr() or out.shape != (context_rows, output.shape[1]): + raise RuntimeError("TP2WIDE_COMBINE_MXFP8 context output differs from the armed step") + rows, width = output.shape + pair = ( + torch.empty((rows, width), dtype=torch.uint8, device=output.device), + torch.zeros( + (((rows + 127) // 128 * 128) * width // 32,), dtype=torch.uint8, device=output.device + ), + ) + pending["pair"] = pair + return pair[0][:context_rows], pair[1][: ((context_rows + 127) // 128 * 128) * width // 32] + + +def _combine_mxfp8_sink(out: torch.Tensor, xq: torch.Tensor, xsf: torch.Tensor) -> None: + """Record the context combine; publish only after generation and partial-tile repair.""" + pending = _COMBINE_MXFP8_STATE.get("pending") + if pending is None: + return + output, context_rows = pending["output"], pending["context_rows"] + pair = pending.get("pair") + if ( + pair is None + or out.data_ptr() != output.data_ptr() + or out.shape != (context_rows, output.shape[1]) + or xq.data_ptr() != pair[0].data_ptr() + or xsf.data_ptr() != pair[1].data_ptr() + ): + raise RuntimeError("TP2WIDE_COMBINE_MXFP8 combine did not use the caller's buffers") + pending["fired"] = True + + +def _combine_mxfp8_finish(output: torch.Tensor) -> None: + """Repair the last context tile plus generation rows, then publish the full C+G identity.""" + pending = _COMBINE_MXFP8_STATE.pop("pending", None) + if pending is None or not pending.get("fired", False): + return + if output is not pending["output"]: + raise RuntimeError("TP2WIDE_COMBINE_MXFP8 output changed before publication") + context_rows = pending["context_rows"] + xq, xsf = pending["pair"] + tail_start = context_rows // 128 * 128 + if tail_start < output.shape[0]: + tail_q, tail_sf = torch.ops.trtllm.mxfp8_quantize(output[tail_start:], True) + xq[tail_start:].copy_(tail_q.view(torch.uint8)) + offset = tail_start * output.shape[1] // 32 + xsf[offset : offset + tail_sf.numel()].copy_(tail_sf.view(torch.uint8).flatten()) + if _COMBINE_MXFP8_CHECK: + ref_q, ref_sf = torch.ops.trtllm.mxfp8_quantize(output, True) + if not torch.equal(xq, ref_q.view(torch.uint8)) or not torch.equal( + xsf, ref_sf.view(torch.uint8).flatten() + ): + raise RuntimeError("TP2WIDE_COMBINE_MXFP8_CHECK full-step quantization mismatch") + if "engaged" not in _COMBINE_MXFP8_STATE: + _COMBINE_MXFP8_STATE["engaged"] = True + logger.info( + f"[M3.1] TP2WIDE_COMBINE_MXFP8 engaged: context {context_rows}, total {output.shape[0]} rows" + ) + _COMBINE_MXFP8_STATE["src"] = output.data_ptr() + _CTXNORMQ_MX.publish(output, xq.view(torch.float8_e4m3fn), xsf) + + +_combine_mxfp8_sink.min_rows = _COMBINE_MXFP8_MIN_ROWS +_combine_mxfp8_sink.allocate = _combine_mxfp8_allocate + + +_GQAMIX_STATE: dict = {} + + +def _gqamix_decode(*args, **kwargs) -> None: + """``minimax_m3_sparse_attn_decode`` for MsaDecodeFmha's sparse generation rows, through the target-local copy.""" + if _lc2_mixed.lc2_mixed_decode(*args, **kwargs): + return + if not _GQAMIX_STATE.get("logged"): + _GQAMIX_STATE["logged"] = True + logger.info( + "[M3 dispatch] GQAMIX: mixed-step sparse decode through ops/gqamix_sparse_decode" + ) + _gqamix_kernel.minimax_m3_sparse_attn_decode(*args, **kwargs) + m3_report.work("GQAMIX", "mixed-step sparse generation rows") + + +_TSEL_STATE: dict = {} +_TSEL_ORACLE = { + "calls": 0, "rows": 0, "rows_diff": 0, "rows_tie_only": 0, "rows_non_tie": 0, + "t_last": 0.0, "t_call": 0.0, "flushed_calls": 0, +} # fmt: skip +_TSEL_ORACLE_IDLE_S = 15.0 + + +def _tsel_op(): + """``torch.ops.trtllm.minimax_m31_select_prefill_tile`` on sm_107, or None. A build without the op loads it from + ``tensorrt_llm/libs/libm31_tsel.so`` when that file exists (an overlay). Decided once per process.""" + if "op" not in _TSEL_STATE: + op = None + if torch.cuda.get_device_capability() == (10, 7): + op = getattr(torch.ops.trtllm, "minimax_m31_select_prefill_tile", None) + if op is None: + import tensorrt_llm + + lib = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_tsel.so") + if os.path.exists(lib): + torch.ops.load_library(lib) + op = getattr(torch.ops.trtllm, "minimax_m31_select_prefill_tile", None) + _TSEL_STATE["op"] = op + print( + f"[M3 dispatch] AGG_TSEL prefill tile select rank {mpi_rank()}: " + f"{'engaged' if op is not None else 'not taken: not sm_107 or the op is not built'} " + f"(rows >= {AGG_TSEL_MIN_ROWS}, blocks <= {AGG_TSEL_MAX_BLOCKS}, oracle {int(AGG_TSEL_ORACLE)})", + file=sys.stderr, + flush=True, + ) + return _TSEL_STATE["op"] + + +def _tsel_oracle_flush(final: bool = False) -> None: + st = _TSEL_ORACLE + st["flushed_calls"] = st["calls"] + print( + f"AGG_TSEL_ORACLE{'_FINAL' if final else ''} rank {mpi_rank()} calls {st['calls']} rows {st['rows']} " + f"rows_diff {st['rows_diff']} tie_only {st['rows_tie_only']} non_tie {st['rows_non_tie']}", + file=sys.stderr, + flush=True, + ) + + +def _tsel_oracle_check(max_score_kv, nvb, out, init_blocks, local_blocks) -> None: + """Compare the tile table with minimax_m3_select_blocks'; a differing row is tie-only when both tables select the + same multiset of score bit patterns (only the order among equal keys differs), else non-tie.""" + st = _TSEL_ORACLE + ref = torch.ops.trtllm.minimax_m3_select_blocks( + max_score_kv, nvb, MSA_REQUIRED_TOPK, init_blocks, local_blocks, True + ) + total_q = int(out.shape[0]) + st["calls"] += 1 + st["rows"] += total_q * int(out.shape[1]) + diff = (out != ref).any(dim=-1) # [total_q, num_kv_heads] + n_diff = int(diff.sum()) + if n_diff: + q_idx, h_idx = diff.nonzero(as_tuple=True) + bits = max_score_kv.view(torch.int32) # [heads, blocks, total_q] + + def picked(table): + rows = table[q_idx, h_idx].long() # [n, topk] + vals = bits[h_idx[:, None], rows.clamp(min=0), q_idx[:, None]] + vals = torch.where(rows >= 0, vals, torch.full_like(vals, -(2**31))) + return vals.sort(dim=-1).values + + tie = (picked(out) == picked(ref)).all(dim=-1) + n_tie = int(tie.sum()) + st["rows_diff"] += n_diff + st["rows_tie_only"] += n_tie + st["rows_non_tie"] += n_diff - n_tie + if n_diff - n_tie and st["rows_non_tie"] == n_diff - n_tie: + print( + f"AGG_TSEL_ORACLE first non-tie rank {mpi_rank()}: {n_diff - n_tie} rows; " + f"blocks {max_score_kv.shape[1]}, " + f"rows {total_q}, init {init_blocks}, local {local_blocks}", + file=sys.stderr, + flush=True, + ) + now = time.monotonic() + st["t_call"] = now + if now - st["t_last"] >= 30.0: + st["t_last"] = now + _tsel_oracle_flush() + + +def _tsel_oracle_idle_watch() -> None: + """Daemon: flush once the tile select has been idle for 15 s with unflushed calls (a server killed at teardown + never reaches atexit, so its last interval lands here).""" + st = _TSEL_ORACLE + while True: + time.sleep(5.0) + if ( + st["calls"] != st["flushed_calls"] + and time.monotonic() - st["t_call"] >= _TSEL_ORACLE_IDLE_S + ): + _tsel_oracle_flush() + + +def _tsel_oracle_install_flushes() -> None: + """Once per process: the final flush at exit, on SIGTERM (then the previous handler), and after 15 s idle.""" + atexit.register(_tsel_oracle_flush, True) + threading.Thread( + target=_tsel_oracle_idle_watch, name="agg_tsel_oracle_flush", daemon=True + ).start() + try: + prev = signal.getsignal(signal.SIGTERM) + + def _on_term(signum, frame): + _tsel_oracle_flush(True) + if callable(prev): + prev(signum, frame) + elif prev == signal.SIG_DFL: + signal.signal(signal.SIGTERM, signal.SIG_DFL) + os.kill(os.getpid(), signal.SIGTERM) + + signal.signal(signal.SIGTERM, _on_term) + except ValueError: # not the main thread: atexit and the idle flush still cover it + pass + + +def _tsel_select(max_score_kv, n_valid_blocks, init_blocks, local_blocks): + """The MSA indexer's selector hook: the head-major [total_q, num_kv_heads, 16] table of + ``select_blocks_from_maxscore``, or None (outside the tile select's rows / blocks range, or no op) to keep it.""" + if ( + torch.cuda.is_current_stream_capturing() # the host-side rows / blocks gate is not graph-stable + or max_score_kv.dtype != torch.float32 + or max_score_kv.dim() != 3 + or max_score_kv.stride(2) != 1 + or max_score_kv.shape[2] < AGG_TSEL_MIN_ROWS + or max_score_kv.shape[1] > AGG_TSEL_MAX_BLOCKS + ): + return None + op = _tsel_op() + if op is None: + return None + num_kv_heads, num_blocks, total_q = max_score_kv.shape + _record("m31_agg_tsel", num_blocks, total_q >= 1024) + out = torch.empty( + (num_kv_heads, total_q, MSA_REQUIRED_TOPK), dtype=torch.int32, device=max_score_kv.device + ).permute(1, 0, 2) + nvb = n_valid_blocks.to(device=max_score_kv.device, dtype=torch.int32).contiguous() + op(max_score_kv, nvb, out, init_blocks, local_blocks) + if not _TSEL_STATE.get("taken"): + _TSEL_STATE["taken"] = True + print( + f"AGG_TSEL taken rank {mpi_rank()}: blocks {num_blocks}, rows {total_q}", + file=sys.stderr, + flush=True, + ) + if AGG_TSEL_ORACLE: + _tsel_oracle_check(max_score_kv, nvb, out, init_blocks, local_blocks) + return out + + +def _tsel_register(backend) -> None: + """Register ``_tsel_select`` on the layer backend's MSA indexer (an indexer without the hook is left alone).""" + indexer = getattr(backend, "indexer", None) + if indexer is not None and getattr(indexer, "selector_hook", False) is None: + indexer.selector_hook = _tsel_select + if AGG_TSEL_ORACLE and not _TSEL_STATE.get("flushes"): + _TSEL_STATE["flushes"] = True + _tsel_oracle_install_flushes() + + +def _gqamix_register(backend) -> None: + """Register ``_gqamix_decode`` on the layer backend's MsaDecodeFmha (a library without the hook is left alone).""" + manager = getattr(backend, "_fmha_manager", None) + for fmha in getattr(manager, "fmha_libs", ()): + if getattr(fmha, "sparse_decode_hook", False) is None: + fmha.sparse_decode_hook = _gqamix_decode + + +def _native_allreduce_norm(core, x, residual, norm_w, allreduce=None): + """The built-in fused all-reduce + residual + RMSNorm (``allreduce``: default the target's AUTO one). An + owned-only TP2AGG_SP residual (the final norm, a fallback) is unsharded first.""" + residual = _sp_unshard(residual, x) + _record("m3_allreduce", x.shape[0]) + return m3_allreduce( + x, + core.allreduce if allreduce is None else allreduce, + AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual, + norm_weight=norm_w, + eps=core.rms_eps, + trigger_completion_at_end=False, + ), + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# The MoE: router, routed experts (trtllm-gen NVFP4), shared expert +# -------------------------------------------------------------------------------------------------------------------- + + +@dataclass +class _MoeScalars: + """The routed experts' kernel scalars of one layer, derived from the checkpoint's per-tensor scales once the + weights are real (``MiniMaxM3Tp2ModelingV2._derive_moe``).""" + + fc31_input_scale: torch.Tensor + fc2_input_scale: torch.Tensor + fc31_alpha: torch.Tensor + fc2_alpha: torch.Tensor + fc31_scale_c: torch.Tensor + act_alpha: torch.Tensor + act_beta: torch.Tensor + act_clamp: torch.Tensor + + +_S2_MX = m3_prequant.PrequantSlot() +_S2_STATE: dict = {} + + +def _s2_op(): + """``torch.ops.trtllm.minimax_m31_s2_moe_tail`` (TP2W_S2_TAIL), or None: switched off, or the build does not + carry it. Decided once per process.""" + if "op" not in _S2_STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_s2_moe_tail", None) if TP2W_S2_TAIL else None + _S2_STATE["op"] = op + why = "switched off" if not TP2W_S2_TAIL else "the op is not built" + logger.info( + f"[M3.1 agg] TP2W_S2_TAIL fused MoE tail: {'engaged' if op is not None else 'not taken: ' + why}" + ) + return _S2_STATE["op"] + + +def _s2_sf_valid(rows: int, hidden: int, device) -> torch.Tensor: + """TP2W_S2_CHECK: which bytes of a swizzled (128x4) K32 scale tensor of ``rows`` x ``hidden`` hold valid rows (the + rest are the pad rows up to the 128-row tile), as a bool mask over its bytes (get_sf_out_offset_128x4).""" + key = (rows, hidden, str(device)) + mask = _S2_STATE.get(key) + if mask is None: + cols = hidden // 32 + padded = (rows + 127) // 128 * 128 + r = torch.arange(padded, device=device).view(-1, 1).expand(padded, cols) + c = torch.arange(cols, device=device).view(1, -1).expand(padded, cols) + off = ( + (r // 128) * ((cols + 3) // 4 * 512) + + (c // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + c % 4 + ) + mask = torch.zeros(padded * ((cols + 3) // 4 * 4), dtype=torch.bool, device=device) + mask[off.reshape(-1)] = (r < rows).reshape(-1) + _S2_STATE[key] = mask + return mask + + +def _s2_boundary(core, rows: int, parts, shared, residual, norm_w, stock_ar): + """TP2W_S2_TAIL: the MoE boundary of ``rows`` rows in one launch from the routed experts' unfinalized ``parts`` + and the shared expert's output. Returns (normed, updated) like the fused all-reduce and publishes the next QKV's + MXFP8 activation (``_S2_MX``), or None (nothing launched) when the operands are not the op's. ``stock_ar`` is the + all-reduce the boundary would run (the decode layer's AUTO one; a prefill-target boundary's _prefill_allreduce).""" + op = _s2_op() + if op is None or not (isinstance(parts, (list, tuple)) and len(parts) == 3): + return None + fc2, scales, perm = parts + lamport = core.allreduce + ws = getattr(lamport, "workspace", None) + if ( + ws is None + or fc2.dtype != torch.bfloat16 + or fc2.dim() != 2 + or fc2.stride(1) != 1 + or shared is None + or shared.dtype != torch.bfloat16 + or not shared.is_contiguous() + or residual.shape != shared.shape + or residual.dtype != torch.bfloat16 + ): + return None + scales = scales.reshape(rows, -1).contiguous() + perm = perm.view(rows, -1).to(torch.int32).contiguous() + residual = residual.contiguous() + if "ar" not in _S2_STATE: + _S2_STATE["ar"] = True + logger.info( + f"[M3.1 agg] TP2W_S2_TAIL replaces finalize + {type(stock_ar).__name__}(" + f"{getattr(stock_ar, 'strategy', '?')}) + quantize at {rows} rows (rank {mpi_rank()})" + ) + ref = None + # The check reads its counts on the host: eager steps only (a captured decode graph is checked by an eager run). + if S2_CHECK and not torch.cuda.is_current_stream_capturing(): + x = m3_moe_finalize(fc2, scales, perm, shared) + ref_n, ref_r = _native_allreduce_norm(core, x, residual, norm_w, stock_ar) + ref = (ref_n, ref_r) + tuple(torch.ops.trtllm.mxfp8_quantize(ref_n, True)) + _record("m31_s2_moe_tail", rows) + normed, updated, q, sf = op( + fc2, + scales, + perm, + shared, + residual, + norm_w, + ws, + lamport.mapping.tp_rank, + lamport.mapping.tp_size, + core.rms_eps, + False, + ) + if ref is not None: + # All four outputs: normed, residual, MXFP8 bytes, and the scales the next QKV consumes (the valid rows' + # bytes bit for bit; the pad rows' bytes 0 in the fused output, as mxfp8_quantize writes them). + diffs = [ + int((a.reshape(-1).view(torch.uint8) != b.reshape(-1).view(torch.uint8)).sum()) + for a, b in zip(ref[:3], (normed, updated, q)) + ] + valid = _s2_sf_valid(rows, normed.shape[1], sf.device) + diffs.append(int((ref[3].reshape(-1)[valid] != sf.reshape(-1)[valid]).sum())) + diffs.append(int((sf.reshape(-1)[~valid] != 0).sum())) + # The decode graph's pinned QKV path (MX-TAC) quantizes with m3_mxfp8_pinned_quantize: the same bytes? + pq, psf = m3_mxfp8_pinned_quantize(ref[0], MXFP8_DECODE_QUANT_TACTIC) + pinned = [ + int((pq.reshape(-1).view(torch.uint8) != q.reshape(-1).view(torch.uint8)).sum()), + int((psf.reshape(-1)[valid] != sf.reshape(-1)[valid]).sum()), + ] + n = _S2_STATE["checked"] = _S2_STATE.get("checked", 0) + 1 + if any(diffs) or any(pinned) or n <= 4: + logger.info( + f"[M3.1 agg] TP2W_S2_TAIL check: rows {rows} normed / residual / mxfp8 / sf / sf_pad differ from " + f"the stock boundary in {' / '.join(str(d) for d in diffs)} bytes; vs the pinned quantize " + f"{pinned[0]} / {pinned[1]} (boundaries checked {n})" + ) + _S2_MX.publish(normed, q, sf) + n = _S2_STATE["taken"] = _S2_STATE.get("taken", 0) + 1 + if n & (n - 1) == 0: + logger.info(f"[M3.1 agg] TP2W_S2_TAIL taken n={n} (rows {rows})") + return normed, updated + + +def _router_logits(core, i: int, h: torch.Tensor) -> torch.Tensor: + """The fp32 router logits of MoE layer ``i`` when the MoE runs unfused (the decode steps' engines route + in-kernel): ``F.linear(float(h), w)``, the built-in gate's arithmetic outside its tensor-core router's rows. The + experts quantize their bf16 input themselves.""" + # TP2AGG_SP: the boundary gathered this step's logits (h is its keyed view). + sp = _SP_LOGITS.take(h) + if sp is not None: + _record("m31_tp2agg_sp_logits", h.shape[0]) + return sp[0] + # TP2AGG_ROUTER_F32: the boundary already wrote float(h). + f32 = _CTXNORMQ_F32.take(h) + if f32 is not None: + _record("m31_router_f32", h.shape[0]) + return torch.nn.functional.linear(f32[0], core.w[f"l{i}_router"]) + _record("m3_fp32_linear", h.shape[0] if h.dim() == 2 else 0) + return m3_fp32_linear(h, core.w[f"l{i}_router"]) + + +def _decode_router_logits( + core, i: int, h: torch.Tensor, fp4: Optional[List[torch.Tensor]] = None +) -> torch.Tensor: + """The router logits of a pure generation step: ``ROUTER_TC_GEN``'s rows take the TF32 tensor-core router when + the device co-schedules 16-CTA clusters (decided at prebuild) and the operands are the built-in gate's (2-D bf16, + unit column stride, 16-byte aligned rows); every other step the fp32 router (``_router_logits``).""" + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + ROUTER_TC_GEN.covers(rows) + and core.router_tc_cluster16 + and h.dtype == torch.bfloat16 + and h.stride(1) == 1 + and (h.stride(0) * 2) % 16 == 0 + ): + taken(ROUTER_TC_GEN.path, rows) + if rows > 16: + m3_report.work("ROUTER48", f"{rows} rows past the TP4 router's 16") + _record("m3_router_tc", rows) + scale = core.moe[i].fc31_input_scale if fp4 is not None else None + outputs = m3_router_tc(h, core.w[f"l{i}_router"], scale, get_env_enable_pdl()) + if fp4 is not None: + fp4.extend(outputs[1:]) + return outputs[0] + return _router_logits(core, i, h) + + +class _S1bCheck: + """S1B_FOLD_CHECK counters: per process, device int64 [calls, mismatching calls, mismatching elements], mirrored to + pinned host memory by a copy inside the step (captured with it) and logged by a daemon thread when they change.""" + + _dev = None + _host = None + + @classmethod + def record(cls, got: torch.Tensor, ref: torch.Tensor, ident=None): + if cls._dev is None: + cls._dev = torch.zeros(3, dtype=torch.int64, device=got.device) + cls._host = maybe_pin_memory(torch.zeros(3, dtype=torch.int64)) + threading.Thread(target=cls._log, daemon=True).start() + bad = (got.contiguous().view(torch.int16) != ref.contiguous().view(torch.int16)).sum() + cls._dev.add_(torch.stack([torch.ones_like(bad), (bad > 0).to(torch.int64), bad])) + cls._host.copy_(cls._dev, non_blocking=True) + if ident is not None and not torch.cuda.is_current_stream_capturing(): + n = int(bad) + if n: + logger.info(f"[M3.1] S1B_FOLD_CHECK_BAD {ident} elements={n}") + + @classmethod + def _log(cls): + import time + + last = None + while True: + time.sleep(20.0) + now = tuple(int(v) for v in cls._host.tolist()) + if now != last: + last = now + logger.info( + f"[M3.1] S1B_FOLD_CHECK calls={now[0]} mismatching_calls={now[1]} mismatching_elements={now[2]}" + ) + + +def _s1b_ident(core, i: int, h: torch.Tensor, do_finalize: bool) -> str: + """S1B_FOLD_CHECK per-call identity: layer, rows, the tactic the fold's MoE resolved, the tuning mode.""" + rows = h.shape[0] + tactic = core.__dict__.get("_s1b_moe_tactics_seen", {}).get( + (h.device, rows, do_finalize), "pinned" + ) + return ( + f"layer={i} rows={rows} finalize={do_finalize} fold_tactic={tactic} " + f"tuning={AutoTuner.get().is_tuning_mode} rank={core.mapping.tp_rank}" + ) + + +def _decode_route(core, i: int, h: torch.Tensor, fp4: Optional[List[torch.Tensor]] = None): + """The routing of a pure generation step's MoE: ``("topk", (ids, weights))`` from ``S1B_FOLD`` at its pinned row + counts above the TC router's rows, else ``("logits", router_logits)`` exactly as ``_decode_router_logits``. Keyed + sources keep their own path: SP logits route from those logits; the boundary's fp32 rows feed the fold's GEMM in + place of ``float(h)`` (the same operand ``m3_fp32_linear`` casts).""" + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + not ( + core.s1b_pins + and rows in core.s1b_pins + and not ROUTER_TC_GEN.covers(rows) + and h.dim() == 2 + ) + or AutoTuner.get().is_tuning_mode + ): + return "logits", _decode_router_logits(core, i, h, fp4) + sp = _SP_LOGITS.take(h) + if sp is not None: + _record("m31_tp2agg_sp_logits", rows) + return "logits", sp[0] + f32 = _CTXNORMQ_F32.take(h) + x = f32[0] if f32 is not None else h.to(torch.float32) + w = core.w[f"l{i}_router"] + topk = m31_s1b_fold.route( + x, + w, + core.w[f"l{i}_router_bias"], + float(core.routed_scaling_factor), + get_env_enable_pdl() and S1B_FOLD_PDL, + ) + if topk is None: + _record("m31_router_f32" if f32 is not None else "m3_fp32_linear", rows) + return "logits", torch.nn.functional.linear(x, w) + _record("m31_s1b_fold", rows) + taken("m31_s1b_fold", rows, f"S {core.s1b_pins[rows]}") + return "topk", topk + + +def _triton_finalize(rows: int, routed, shared: Optional[torch.Tensor]) -> torch.Tensor: + """The finalized routed output (+ ``shared``) from the runner's unfinalized parts via kernels/m3_moe_finalize + (``TP2W_MOE_FINALIZE``). Raises if the parts are not the trtllm-gen layout (the runner did not finalize them: + no fallback).""" + out = None + if isinstance(routed, (list, tuple)) and len(routed) == 3: + fc2, scales, perm = routed + scales, perm = scales.reshape(rows, -1), perm.view(rows, -1).to(torch.int32) + native = _native_finalize(rows, fc2, scales, perm, shared) + if native is not None: + return native + out = m3_moe_finalize(fc2, scales, perm, shared) + if out is None: + raise RuntimeError( + f"the routed experts' unfinalized output at {rows} rows is not the trtllm-gen layout" + ) + _record("m3_moe_finalize", rows, shared is not None) + taken( + "m3_moe_finalize", rows, "Triton finalize" + (" + shared add" if shared is not None else "") + ) + return out + + +def _native_finalize(rows: int, fc2, scales, perm, shared) -> Optional[torch.Tensor]: + """NATIVE_FINALIZE: the finalize by the native op, or None (the Triton kernel). The 49..448-row and 2K+-row + (without shared) bands only with NATIVE_ALL_BANDS. cfg by rows: 3 = (128, 1) up to 224 rows, 6 = (96, 2) above, + 4 = (64, 2) at 2K+ rows with shared.""" + site = "shared add" if shared is not None else "routed" + if not NATIVE_FINALIZE: + return _native_dispatch( + "NATIVE_FINALIZE", site, "killed (TRTLLM_M31_DISABLE_NATIVE_FINALIZE=1)" + ) + if is_torch_compiling(): + return _native_dispatch("NATIVE_FINALIZE", site, "under torch.compile") + if not NATIVE_ALL_BANDS and (rows < 2048 or shared is None): + return _native_dispatch( + "NATIVE_FINALIZE", site, "outside the served bands (NATIVE_ALL_BANDS off)" + ) + if ( + fc2.dim() != 2 + or fc2.dtype != torch.bfloat16 + or fc2.stride(1) != 1 + or fc2.shape[1] % 8 != 0 + or fc2.stride(0) % 8 != 0 + or fc2.data_ptr() % 16 != 0 + or not scales.is_contiguous() + or not perm.is_contiguous() + or (shared is not None and (not shared.is_contiguous() or shared.data_ptr() % 16 != 0)) + ): + return _native_dispatch("NATIVE_FINALIZE", site, "operand layout or alignment") + op = _native_op("minimax_m31_moe_finalize") + if op is None: + return _native_dispatch("NATIVE_FINALIZE", site, "the op is not registered") + cfg = 3 if rows <= 224 else (4 if rows >= 2048 and shared is not None else 6) + _native_dispatch("NATIVE_FINALIZE", site) + _record("m31_native_finalize", rows, shared is not None) + taken( + "m31_native_finalize", rows, f"cfg {cfg}" + (" + shared add" if shared is not None else "") + ) + return op(fc2, scales, perm, shared, cfg) + + +def _triton_finalize_rows(core, h: torch.Tensor) -> bool: + """Whether a finalized MoE forward of ``h`` takes ``TP2W_MOE_FINALIZE``: at least its rows, in one runner call.""" + rows = h.shape[0] if h.dim() == 2 else 0 + return ( + TP2W_MOE_FINALIZE + and rows >= MOE_TRITON_FINALIZE_MIN_TOKENS + and rows <= core.moe_max_num_tokens + ) + + +def _moe_tactic_as_logits(core, inputs, do_finalize: bool): + """(runner, tactic) the stock runner resolves for these rows with ROUTING LOGITS: the same AutoTuner key as + ``fp4_block_scale_moe_runner``'s (an fp32 [rows, experts] logits operand in slot 0, no top-k tensors), so a + top-k-ids call (``S1B_FOLD``) runs the GEMM tactic the logits path runs (the pinned / tuned one) instead of the + tuner's miss default: the same kernels, hence the same output bits. Resolved on EVERY call, as the stock op does + (a tactic resolved before the tuning cache is loaded would otherwise stay the miss default); only + the runner object is cached per (rows, finalize). Logged when a row count's tactic first resolves or changes.""" + rows = inputs[2].shape[0] + key = (inputs[2].device, rows, do_finalize) + runners = core.__dict__.setdefault("_s1b_moe_runners", {}) + if key not in runners: + runners[key] = FP4BlockScaleMoERunner( + core.num_experts, + core.top_k, + None, + None, + core.moe_inter, + 0, + core.num_experts, + core.routed_scaling_factor, + ROUTING_MINIMAX2, + do_finalize, + ACT_SWIGLU, + tune_max_num_tokens=core.max_num_tokens, + use_dp=False, + ) + runner = runners[key] + logits_shape = torch.empty(rows, core.num_experts, dtype=torch.float32, device=inputs[2].device) + runner, tactic = AutoTuner.get().choose_one( + "trtllm::fp4_block_scale_moe_runner", + [runner], + runner.tuning_config, + [logits_shape, *inputs[1:], None, None], + ) + tactic = [-1, -1] if tactic == -1 else tactic + seen = core.__dict__.setdefault("_s1b_moe_tactics_seen", {}) + if seen.get(key) != list(tactic): + seen[key] = list(tactic) + logger.info( + f"[M3.1] S1B_FOLD MoE tactic at {rows} rows (finalize={do_finalize}): {list(tactic)}" + ) + return runner, tactic + + +def _routed_experts(core, i: int, x, x_sf, router_logits, do_finalize: bool, topk=None): + """The trtllm-gen NVFP4 MoE runner over this rank's tensor-parallel slice of every expert, routing in-kernel. + + ``x_sf`` None means ``x`` is the bf16 activation, quantized here with the experts' global input scale. ``topk`` + (int32 ids, bf16 weights) routes from those (``S1B_FOLD``; ``router_logits`` None) instead of the logits. Returns + the finalized rows, or the unfinalized ``[gemm2 out, expert scales, expanded->permuted index]``. + """ + moe = core.moe[i] + if x_sf is None: + x, x_sf = fp4_quantize(x, moe.fc31_input_scale, 16, False, False) + rows = x.shape[0] + x_sf = x_sf.view(rows, -1).flatten() + _record("fp4_block_scale_moe_runner", rows, do_finalize) + inputs = ( + router_logits, + core.w[f"l{i}_router_bias"], + x, + x_sf.view(torch.float8_e4m3fn), + core.w[f"l{i}_e_w31"], + core.w[f"l{i}_e_s31"].view(torch.float8_e4m3fn), + None, + moe.act_alpha, + moe.act_beta, + moe.act_clamp, + core.w[f"l{i}_e_w2"], + core.w[f"l{i}_e_s2"].view(torch.float8_e4m3fn), + None, + moe.fc31_scale_c, + moe.fc31_alpha, + moe.fc2_alpha, + ) + outs = None + if _mx_mode(core.mxfp8).decode_graph: + outs = _run_pinned_moe(core, inputs, do_finalize, ROUTING_MINIMAX2, ACT_SWIGLU, topk=topk) + if outs is None and topk is not None: + runner, tactic = _moe_tactic_as_logits(core, inputs, do_finalize) + outs = runner([*inputs, topk[1], topk[0]], tactic=tactic) + if outs is None: + outs = fp4_block_scale_moe_runner( + *inputs, + core.num_experts, + core.top_k, + None, + None, + core.moe_inter, + 0, + core.num_experts, + core.routed_scaling_factor, + ROUTING_MINIMAX2, + do_finalize, + act_type=ACT_SWIGLU, + topk_weights=None if topk is None else topk[1], + topk_ids=None if topk is None else topk[0], + output=None, + tune_max_num_tokens=core.max_num_tokens, + use_dp=False, + ) + if do_finalize: + return outs[0] + if topk is not None: + # The thop's unfinalized return hands back its own expert_weights buffer, which a top-k-ids call never writes + # (routing and the in-kernel finalize use topk_weights): the finalize's scales are the weights passed in. + outs = [outs[0], topk[1], outs[2]] + return outs + + +def _split_chunk(rows: int, num_chunks: int) -> List[int]: + """The MoE scheduler's even split: the first ``rows % num_chunks`` chunks take one more row.""" + base, extra = divmod(rows, num_chunks) + return [base + 1] * extra + [base] * (num_chunks - extra) + + +# -------------------------------------------------------------------------------------------------------------------- +# Attention +# -------------------------------------------------------------------------------------------------------------------- + + +def _nvfp4_layer(manager, i: int) -> bool: + return bool(getattr(manager, "is_nvfp4_layer", lambda _layer_idx: False)(i)) + + +def _producer_covered(core) -> bool: + """Whether ``PRODUCER`` covers this target's sparse layers: an NVFP4 KV cache at the TP2 heads (``PRODUCER_HEADS``: + 32 query, two KV and two index heads per rank), with the range on. A covered layer that cannot take ``m3_producer`` + is a miss.""" + return ( + core.main_kv_is_nvfp4 + and (core.heads, core.kv_heads, core.index_heads) == PRODUCER_HEADS + and PRODUCER.covers("nvfp4_tp2") + ) + + +def _producer_layout(kv_data, kv_scale) -> bool: + """Whether a sparse layer's NVFP4 pools are the layout ``m3_producer`` is built for (TP2: two KV heads): + [pages, 2, 2, 128, 64] with its scale pool [pages, 2, 2, 128, 8].""" + return ( + kv_data is not None + and kv_scale is not None + and kv_data.dim() == 5 + and tuple(kv_data.shape[1:]) == (2, 2, 128, 64) + and kv_data.stride()[2:] == (128 * 64, 64, 1) + and kv_scale.dim() == 5 + and tuple(kv_scale.shape[1:]) == (2, 2, 128, 8) + and kv_scale.stride()[2:] == (128 * 8, 8, 1) + and kv_scale.shape[0] == kv_data.shape[0] + ) + + +def _nvfp4_fused_write_layout(data: torch.Tensor, scales: torch.Tensor, num_heads: int) -> bool: + """Reject invalid HND pools; select the fused producer only for its packed layout.""" + if ( + data.dim() != 5 + or scales.dim() != 5 + or data.shape[0] <= 0 + or data.shape[0] != scales.shape[0] + or tuple(data.shape[1:]) != (2, num_heads, 128, 64) + or tuple(scales.shape[1:]) != (2, num_heads, 128, 8) + or data.dtype not in (torch.int8, torch.uint8) + or scales.dtype != torch.uint8 + ): + raise ValueError("MiniMax-M3 NVFP4 data and scale caches must have matching HND geometry") + for cache in (data, scales): + if ( + cache.stride(4) != 1 + or cache.stride(3) != cache.shape[4] + or cache.stride(2) < 128 * cache.stride(3) + or cache.stride(1) < num_heads * cache.stride(2) + or cache.stride(0) < 2 * cache.stride(1) + ): + raise ValueError( + "MiniMax-M3 NVFP4 cache rows must be packed and heads/planes/pages must not overlap" + ) + if scales.stride(2) != 128 * 8: + raise ValueError("MiniMax-M3 NVFP4 MSA scale cache must have packed head blocks") + return data.stride(2) == 128 * 64 and data.stride(0) % 2 == 0 and data.stride(1) % 2 == 0 + + +def _producer_miss(covered: bool, rows: Optional[int], reason: str) -> None: + if covered: + missed(PRODUCER.path, rows, reason) + return None + + +def _sparse_qkv_producer( + core, + i: int, + packed, + position_ids, + kv_cache_manager, + msa_idx_k_cache, + msa_out_cache_loc, + deferred: Optional[_DeferredGemv] = None, +): + """The horizontal producer of a sparse layer: Q / index-Q (FP8) out, K / V / index-K written into the paged + caches (NVFP4 or FP8 main cache), for every step type (token-major). An NVFP4 cache at the TP2 heads takes + ``m3_producer`` (``PRODUCER``); a covered layer that cannot is a miss. Returns (q, index_q), or None for caches + neither kernel takes. + + ``kv_cache_manager``, ``msa_idx_k_cache`` (the per-layer index-K cache getter) and ``msa_out_cache_loc`` are the + calling target's ``step_args`` entries. ``deferred``: the QKV GEMV ``_linear`` recorded instead of launching + (``QKV_PRODUCER_FOLD``; ``packed`` is then its unwritten placeholder). The ``m3_producer`` branch runs it with the + producer in its epilogue (``_qkv_producer_fold``); any other branch launches it first (``materialize``). A None + return leaves it unlaunched.""" + covered = _producer_covered(core) + rows = int(packed.shape[0]) + if packed.dtype != torch.bfloat16 or position_ids is None: + return _producer_miss( + covered, + rows, + "the packed QKV dtype, position ids, head dims, norm or RoPE configuration", + ) + manager = kv_cache_manager + if manager is None: + return _producer_miss(covered, rows, "the step has no KV cache manager") + buffers = manager.get_buffers(i, kv_layout="HND") + index_k_cache = msa_idx_k_cache(i) + out_cache_loc = msa_out_cache_loc + # AGG_NVFP4_INDEXK: the producer writes E4M3 index-K into the cache manager's FP8 sink (the main pool's pages, one + # for every layer); the layer's NVFP4 index-K pages are packed from it right after (_pack_nvfp4_index_k). + nvfp4_index_cache = None + if _nvfp4_index_k.is_nvfp4_index_cache(index_k_cache): + sink = getattr(manager, "get_nvfp4_index_k_sink", lambda: None)() + if sink is None: + return _producer_miss( + covered, rows, "an NVFP4 index-K cache without the manager's FP8 sink" + ) + nvfp4_index_cache, index_k_cache = index_k_cache, sink + layer_uses_nvfp4 = _nvfp4_layer(manager, i) + scale_buffers = ( + manager.get_block_scale_buffers(i, kv_layout="HND") if layer_uses_nvfp4 else None + ) + kv_quant_scale = core.inv_kv_scales[i] if layer_uses_nvfp4 else None + if layer_uses_nvfp4: + supported_main = ( + buffers is not None + and scale_buffers is not None + and _nvfp4_fused_write_layout(buffers, scale_buffers, core.kv_heads) + and buffers.is_cuda + and scale_buffers.is_cuda + and scale_buffers.device == buffers.device + and kv_quant_scale.is_cuda + and kv_quant_scale.dtype == torch.float32 + and kv_quant_scale.numel() >= 3 + and kv_quant_scale.is_contiguous() + ) + else: + supported_main = ( + buffers is not None + and buffers.is_cuda + and buffers.dtype == torch.float8_e4m3fn + and buffers.dim() == 5 + and tuple(buffers.shape[1:]) == (2, core.kv_heads, 128, 128) + and buffers.stride(4) == 1 + and buffers.stride(3) == 128 + and buffers.stride(2) == 128 * 128 + and buffers.stride(1) >= core.kv_heads * buffers.stride(2) + and buffers.stride(0) >= 2 * buffers.stride(1) + and buffers.stride(0) % 4 == 0 + and buffers.stride(1) % 4 == 0 + ) + supported_index = ( + index_k_cache.is_cuda + and index_k_cache.dtype == torch.float8_e4m3fn + and index_k_cache.dim() == 4 + and tuple(index_k_cache.shape[1:]) == (1, 128, 128) + and index_k_cache.stride(3) == 1 + and index_k_cache.stride(2) == 128 + and index_k_cache.stride(1) >= 128 * 128 + and index_k_cache.stride(0) >= index_k_cache.stride(1) + and index_k_cache.stride(0) % 4 == 0 + and index_k_cache.stride(1) % 4 == 0 + and buffers is not None + and index_k_cache.shape[0] == buffers.shape[0] + ) + cos_sin = core.rotary_cos_sin + supported_rope = ( + cos_sin.is_cuda + and cos_sin.dtype == torch.float32 + and cos_sin.is_contiguous() + and cos_sin.dim() == 3 + and tuple(cos_sin.shape[1:]) == (2, 32) + ) + if ( + not supported_main + or not supported_index + or not supported_rope + or out_cache_loc is None + or not out_cache_loc.is_cuda + or out_cache_loc.dtype != torch.int32 + or not out_cache_loc.is_contiguous() + or out_cache_loc.numel() < rows + ): + return _producer_miss( + covered and layer_uses_nvfp4, + rows, + "the layer's NVFP4 / index caches, RoPE table, KV quant scale or cache slots are not the layout the " + "producer is built for", + ) + w = core.w + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + if ( + layer_uses_nvfp4 + and PRODUCER.covers("nvfp4_tp2") + and (core.heads, core.kv_heads, core.index_heads) == PRODUCER_HEADS + and _producer_layout(buffers, scale_buffers) + ): + if deferred is not None: + folded = _qkv_producer_fold( + core, + i, + deferred, + buffers, + scale_buffers, + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + cos_sin, + pos, + ) + if folded is not None: + _pack_nvfp4_index_k(nvfp4_index_cache, index_k_cache, out_cache_loc, rows) + return folded + packed = deferred.materialize() + early_trigger = PRODUCER_EARLY_TRIGGER and rows > PRODUCER_LATE_TRIGGER_MAX_ROWS + _record("m3_producer", rows, early_trigger) + q, index_q = m3_producer( + packed.contiguous(), + buffers.view(torch.uint8), + scale_buffers.view(torch.uint8), + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + core.heads, + get_env_enable_pdl(), + early_trigger, + PRODUCER_PREWAIT_STEP_INPUTS, + ) + # Every token count takes the producer: rows past a decode step log once as one line. + decode_rows = is_decode_step(rows) + if decode_rows: + # The target tag line: once per rank, the first decode producer call. + _log_once( + logger.info, + "m3_sm107_tp2_target", + "engaged", + f"[M3 sm107 tp2] fast path engaged: m3_producer (decode, rank {mpi_rank()}, {rows} rows, nvfp4_tp2: " + "32 query / 2 KV / 2 index heads per rank)", + ) + taken( + PRODUCER.path, + rows if decode_rows else None, + f"nvfp4_tp2, {'entry' if early_trigger else 'exit'} trigger, " + f"prewait_step_inputs={PRODUCER_PREWAIT_STEP_INPUTS}" + if decode_rows + else "nvfp4_tp2, past the decode rows", + ) + else: + if layer_uses_nvfp4 and covered: + missed( + PRODUCER.path, + rows, + f"the NVFP4 pool {tuple(buffers.shape)} / scale pool {tuple(scale_buffers.shape)} is not the " + "producer's layout", + ) + if deferred is not None: + packed = deferred.materialize() + _record("m3_qkv_insert", rows, layer_uses_nvfp4) + q, index_q = m3_qkv_insert( + packed.contiguous(), + buffers.view(torch.uint8) if layer_uses_nvfp4 else buffers, + scale_buffers, + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.heads, + core.kv_heads, + core.index_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + ) + _pack_nvfp4_index_k(nvfp4_index_cache, index_k_cache, out_cache_loc, rows) + return q.flatten(1), index_q.flatten(1) + + +_NVFP4_INDEXK_DECODE_LOGGED: set = set() + + +def _pack_nvfp4_index_k(nvfp4_index_cache, sink, out_cache_loc, rows: int) -> None: + """AGG_NVFP4_INDEXK: pack the step's index-K rows from the producer's FP8 sink into the layer's NVFP4 pages (the + first ``rows`` slots; negative slots skipped). No-op on an FP8 index cache.""" + if nvfp4_index_cache is None: + return + _nvfp4_index_k.write_from_sink(sink, out_cache_loc, rows, nvfp4_index_cache) + if not torch.cuda.is_current_stream_capturing() and not _NVFP4_INDEXK_LOGGED: + _NVFP4_INDEXK_LOGGED.append(True) + logger.info( + f"[M3.1 tp2] AGG_NVFP4_INDEXK writer taken: producer sink -> NVFP4 index-K pages ({rows} rows)" + ) + + +_NVFP4_INDEXK_LOGGED: list = [] + + +class _DeferredA1: + """``TP2AGG_QKV_A1``: a sparse layer's packed QKV + index MXFP8 GEMM on its CTXNORMQ activation, recorded instead + of launched. The producer call site runs it fused (``fold``: the A1 kernel, GEMM + producer) or calls + ``materialize`` (the stock GEMM ``_linear`` would have run on that activation). ``placeholder`` has the packed + shape and no storage; it is never read. ``ovl`` (TP2AGG_SP SP_OVL): the own rows run first, under the push of + this rank's rows to the peer; the peer's rows after its push landed (A1 is row-split bitwise).""" + + def __init__(self, core, act, act_scale, weight, weight_scale, mode, ovl=None): + rows, n = act.shape[0], weight.shape[0] + self.placeholder = torch.empty(1, 1, dtype=torch.bfloat16, device=act.device).expand( + rows, n + ) + self.core, self.mode = core, mode + self.act, self.act_scale = act, act_scale + self.weight, self.weight_scale = weight, weight_scale + self.ovl = ovl + self.done = False + + def _wait_ovl(self) -> None: + if self.ovl is not None: + torch.cuda.current_stream().wait_event(self.ovl.event) + + def materialize(self) -> torch.Tensor: + self.done = True + self._wait_ovl() + return _ctxnormq_mxfp8_gemm( + self.core, (self.act, self.act_scale), self.weight, self.weight_scale, self.mode + ) + + def fold( + self, i, buffers, scale_buffers, index_k_cache, out_cache_loc, kv_quant_scale, cos_sin, pos + ): + """The A1 kernel for layer ``i``: (q, index_q) as the producer returns them, the caches written; None (nothing + launched) when the caches are outside the kernel's alignment contract.""" + core = self.core + rows = int(self.act.shape[0]) + runner = _a1_runner(core.heads, core.kv_heads, core.index_heads) + device = self.act.device + q = torch.empty(rows, core.heads, core.head_dim, dtype=torch.float8_e4m3fn, device=device) + index_q = torch.empty( + rows, core.index_heads, core.head_dim, dtype=torch.float8_e4m3fn, device=device + ) + norms = _A1_NORMS.get(i) + if norms is None: + w = core.w + norms = _A1_NORMS[i] = torch.cat( + [w[f"l{i}_q_norm"], w[f"l{i}_k_norm"], w[f"l{i}_idx_q_norm"], w[f"l{i}_idx_k_norm"]] + ).contiguous() + act = self.act.view(torch.float8_e4m3fn).view(rows, -1) + ovl = self.ovl + parts = ( + [((0, rows), slice(None))] + if ovl is None + else [(ovl.own_rows, ovl.own_sf), (ovl.peer_rows, ovl.peer_sf)] + ) + try: + part_args = [] + for (r0, n), sf_range in parts: + rr = slice(r0, r0 + n) + part_args.append( + runner.make_args( + act[rr], + self.act_scale[sf_range], + self.weight.view(torch.float8_e4m3fn), + self.weight_scale, + core.mx_alpha, + q[rr], + index_q[rr], + buffers.view(torch.uint8), + scale_buffers.view(torch.uint8), + index_k_cache, + out_cache_loc[rr], + kv_quant_scale, + cos_sin, + pos[rr], + norms, + core.rms_eps, + ) + ) + except AssertionError as e: + _log_once( + logger.warning, + "tp2agg_qkv_a1", + "declined", + f"[M3.1 agg] TP2AGG_QKV_A1 declined: {e}", + ) + return None + _record("m31_qkv_a1", rows, ovl is not None) + runner.run_args(part_args[0]) + if ovl is not None: + self._wait_ovl() + runner.run_args(part_args[1]) + _sp_once( + ("ovl", rows), + f"[M3.1 agg] TP2AGG_SP_OVL taken: A1 rows {rows} as own {ovl.own_rows[1]} + peer " + f"{ovl.peer_rows[1]} (push by copy engine under the own rows)", + ) + self.done = True + _a1_count(rows) + return q.flatten(1), index_q.flatten(1) + + +_A1_RUNNERS: dict = {} +_A1_NORMS: dict = {} +_A1_COUNTS: dict = {} + + +def _a1_runner(num_heads_q: int, num_heads_kv: int, num_heads_index: int): + """The compiled A1 runner for one per-rank head geometry (compiled on the first take, eager: the warmup's first + long step).""" + key = (num_heads_q, num_heads_kv, num_heads_index) + runner = _A1_RUNNERS.get(key) + if runner is None: + from .kernels.qkv_fusion.kernel import M3QkvFusedRunner + + runner = _A1_RUNNERS[key] = M3QkvFusedRunner( + num_heads_q=num_heads_q, num_heads_kv=num_heads_kv, num_heads_index=num_heads_index + ) + logger.info(f"[M3.1 agg] TP2AGG_QKV_A1 runner compiled for heads {key}") + return runner + + +def _a1_count(rows: int) -> None: + """Per-call A1 engagement by step-size band, logged at counts 1, 2, 4, ... per band.""" + band = "<=2K" if rows <= 2048 else "<=4K" if rows <= 4096 else "<=8K" if rows <= 8192 else ">8K" + _A1_COUNTS[band] = n = _A1_COUNTS.get(band, 0) + 1 + if n & (n - 1) == 0: + logger.info(f"[M3.1 agg] TP2AGG_QKV_A1 taken band={band} n={n} (rows {rows})") + m3_report.work("TP2AGG_QKV_A1", f"{rows} rows") + + +def _a1_defer(core, i: int, hidden: torch.Tensor) -> Optional[_DeferredA1]: + """A ``_DeferredA1`` for layer ``i``'s packed QKV projection when A1 takes the step (switched on, eager, + not capturing, not tuning, more than A1_MIN_ROWS rows, the CTXNORMQ activation published for ``hidden``), else + None (the caller runs ``_linear``; the CTXNORMQ slot is then still published unless this took it).""" + if not QKV_A1 or not CTXNORMQ or hidden.dim() != 2 or hidden.shape[0] <= A1_MIN_ROWS: + return None + mode = _mx_mode(core.mxfp8) + if ( + mode.decode_graph + or mode.tuned_quantize_gemm + or is_torch_compiling() + or torch.cuda.is_current_stream_capturing() + or (core.heads, core.kv_heads, core.index_heads) != PRODUCER_HEADS + ): + return None + from tensorrt_llm._torch.autotuner import AutoTuner + + if AutoTuner.get().is_tuning_mode: + # The stock QKV GEMM profiles its shape in the tuning pass (the steps A1 does not take run it). + return None + w, s = core.w[f"l{i}_qkv_w"], core.w[f"l{i}_qkv_s"] + prequant = _CTXNORMQ_MX.take(hidden.contiguous()) + if prequant is None: + return None + return _DeferredA1(core, prequant[0], prequant[1], w, s, mode, _sp_ovl_take(hidden)) + + +def _qkv_producer_fold( + core, + i: int, + deferred: _DeferredGemv, + buffers, + scale_buffers, + index_k_cache, + out_cache_loc, + kv_quant_scale, + cos_sin, + pos, +) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """``QKV_PRODUCER_FOLD``: the deferred QKV GEMV with layer ``i``'s producer in its epilogue + (``mx_gemv.m3_mx_gemv_producer``), the same outputs and cache bytes as the GEMV + ``m3_producer``; None (nothing + launched) when the kernel declines. A ``_DeferredA1`` record runs the A1 kernel instead.""" + if isinstance(deferred, _DeferredA1): + return deferred.fold( + i, buffers, scale_buffers, index_k_cache, out_cache_loc, kv_quant_scale, cos_sin, pos + ) + rows = int(deferred.act.shape[0]) + device = deferred.act.device + q = torch.empty(rows, core.heads, core.head_dim, dtype=torch.float8_e4m3fn, device=device) + index_q = torch.empty( + rows, core.index_heads, core.head_dim, dtype=torch.float8_e4m3fn, device=device + ) + w = core.w + _record("m3_qkv_producer_fold", rows) + ok = mx_gemv.m3_mx_gemv_producer( + deferred.act, + deferred.act_scale, + deferred.weight, + deferred.weight_scale, + q, + index_q, + buffers.view(torch.uint8), + scale_buffers.view(torch.uint8), + index_k_cache, + out_cache_loc, + kv_quant_scale, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + cfg=deferred.cfg, + use_pdl=deferred.use_pdl, + ) + if not ok: + missed( + QKV_PRODUCER_FOLD_PATH, + rows, + "the producer GEMV declined the step" + _uncompiled_note(), + ) + return None + deferred.done = True + n, k = deferred.weight.shape + taken(MX_GEMV_PATH, rows, f"[{n}, {k}] mxfp8 in, split {deferred.cfg[0]}, producer epilogue") + taken(QKV_PRODUCER_FOLD_PATH, rows, f"[{n}, {k}] mxfp8 in + producer") + # The producer ran (in the GEMV's epilogue): its engagement and the target tag line, as the m3_producer branch. + _log_once( + logger.info, + "m3_sm107_tp2_target", + "engaged", + f"[M3 sm107 tp2] fast path engaged: m3_producer (decode, rank {mpi_rank()}, {rows} rows, nvfp4_tp2: " + "32 query / 2 KV / 2 index heads per rank; in the QKV GEMV epilogue)", + ) + taken(PRODUCER.path, rows, "nvfp4_tp2, in the QKV GEMV epilogue (QKV_PRODUCER_FOLD)") + return q.flatten(1), index_q.flatten(1) + + +def _qk_norm_rope( + core, i: int, qkv: torch.Tensor, position_ids, out_fp8: bool, kv_dest: Optional[tuple] = None +) -> torch.Tensor: + """The per-head Gemma RMSNorm of Q and K and the partial NeoX RoPE of a layer's packed [Q | K | V], in one kernel: + in place, or an FP8 copy (Q / K normed and roped, V cast) when ``out_fp8``. A dense layer's FP8 copy carries the + BF16 output's bits cast to E4M3 (``m3_dense_qk_norm_rope_fp8``, ``DENSE_QKV_FP8_VIA_BF16``); with ``kv_dest`` = + (the layer's HND pool, the step's cache slots, live tokens, a [tokens, q heads, 128] FP8 buffer) that kernel also + stores the live tokens' K / V rows into the pool and every row's Q into the buffer.""" + qkv = qkv.contiguous() + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + if ( + out_fp8 + and DENSE_QKV_FP8_VIA_BF16 + and i not in core.sparse_layers + and qkv.dtype == torch.bfloat16 + and core.head_dim == 128 + ): + _record("m3_dense_qk_norm_rope_fp8", qkv.shape[0], kv_dest is not None) + return m3_dense_qk_norm_rope_fp8( + qkv, + core.heads, + core.kv_heads, + core.kv_heads, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_q_norm"], + core.w[f"l{i}_k_norm"], + core.rope_theta, + True, + pos, + True, + *(kv_dest or ()), + ) + assert kv_dest is None, "only the dense FP8 kernel stores K / V" + args = ( + qkv, + core.heads, + core.kv_heads, + core.kv_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_q_norm"], + core.w[f"l{i}_k_norm"], + core.rope_theta, + True, + pos, + 1.0, # factor: no YARN + 0.0, # low + 0.0, # high + 1.0, # attention_factor + True, # is_qk_norm + True, # use_gemma + False, # use_mrope + 0, + 0, + ) + _record("qk_norm_rope", qkv.shape[0], out_fp8) + if out_fp8: + return m3_qk_norm_rope_fp8(*args) + return fused_qk_norm_rope(*args) + + +def _fp8_index_qk_norm_rope( + core, i: int, idx_qk: torch.Tensor, position_ids, msa_idx_k_cache, msa_out_cache_loc +) -> torch.Tensor: + """The index branch of an unfused sparse layer: index-Q in FP8 out, index-K into the FP8 index-K cache, normed and + roped in one kernel (the built-in ``_fused_fp8_index_qk_norm_rope``). ``msa_idx_k_cache`` / ``msa_out_cache_loc`` + are the calling target's ``step_args`` entries.""" + if position_ids is None or idx_qk.dtype != torch.bfloat16: + raise NotImplementedError( + "MiniMax-M3 fused FP8 indexer requires BF16 activations and position_ids." + ) + index_k_cache = msa_idx_k_cache(i) + if index_k_cache.dtype != torch.float8_e4m3fn: + raise ValueError( + "MiniMax-M3 fused FP8 indexer requires an E4M3 index-K cache, " + f"got {index_k_cache.dtype}." + ) + rows = int(idx_qk.shape[0]) + _record("m3_fp8_indexer_qk_norm_rope", rows) + return m3_fp8_indexer_qk_norm_rope( + idx_qk.contiguous(), + index_k_cache, + msa_out_cache_loc[:rows], + core.index_heads, + core.index_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_idx_q_norm"], + core.w[f"l{i}_idx_k_norm"], + core.rope_theta, + position_ids.reshape(-1).contiguous().to(torch.int32), + ).flatten(1) + + +# -------------------------------------------------------------------------------------------------------------------- +# Layer helpers both targets' bodies call: the Eagle3 capture hook and the norm after a layer +# -------------------------------------------------------------------------------------------------------------------- + + +def _plain_capture(spec_metadata, layer_id: int, hidden_states, residual) -> None: + """The in-tree Eagle3 hidden-state capture.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + +def _next_norm(core, i: int) -> torch.Tensor: + """The norm the boundary after layer ``i`` applies: the next layer's input norm, the final norm after the last.""" + return core.w[f"l{i + 1}_in_norm"] if i + 1 < core.num_layers else core.w["final_norm"] + + +# -------------------------------------------------------------------------------------------------------------------- +# The core +# -------------------------------------------------------------------------------------------------------------------- + +# Engine states (compiled kernel + workspace) per (device, token count[, shared fold]), shared by every layer: a step +# runs the layers one after another on one stream. +_ENGINE_STATES: Dict[tuple, object] = {} +_MOE16_STATES: Dict[tuple, object] = {} + + +# AR-DET: AllReduce AUTO's per-process timing autotuner made VR runs differ across server starts +# (with it off, 0 / 888 same-node and 0 / 296 across node pairs differ, bitwise to the +# modal autotuned output). The target turns it off for the whole process at init (AUTO then takes the C++ size / world +# heuristic, a pure function of the shape), unless TRTLLM_M31_ENABLE_AR_AUTOTUNE=1. An explicit +# TLLM_DISABLE_ALLREDUCE_AUTOTUNE is left as given. Read once, at import. +AR_AUTOTUNE_ENABLED = m31_enabled("AR_AUTOTUNE") # the m31_switches value contract (1/true/on/yes) + + +def _pin_allreduce_heuristic(rank: int) -> bool: + """Default the AllReduce autotune off for this process (see AR-DET); returns whether it is off. Logs one line.""" + if not AR_AUTOTUNE_ENABLED: + os.environ.setdefault("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "1") + off = os.environ.get("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "0") == "1" + if off: + m3_report.work("AR_DET", "AllReduce autotune off for this process") + logger.info( + f"M3.1 AR-DET: AllReduce autotune {'off (C++ size / world heuristic)' if off else 'on'} on rank {rank}" + ) + return off + + +# PREFILL-MNNVL: the built-in all-reduce of a prefill or mixed step's layer boundaries (the rows the multicast kernel +# does not take) runs MNNVL instead of AllReduce(AUTO)'s pick. On VR TP4 every strategy is cross-process deterministic +# there and MNNVL is the fastest, 1.7..29 us per call under AUTO at 128..8192 rows (plain and with the fused residual + +# RMSNorm). Up to PREFILL_AR_MNNVL_MAX_ROWS rows (MNNVL's workspace grows with the rows; larger steps keep AUTO); +# generation steps keep AUTO. TRTLLM_M31_DISABLE_PREFILL_AR_MNNVL=1 restores AUTO. Read once, at import. +PREFILL_AR_MNNVL = not m31_disabled("PREFILL_AR_MNNVL") +PREFILL_AR_MNNVL_MAX_ROWS = 8192 # capacity: measured: the largest step the strategy table covers + + +def _prefill_allreduce(core, rows: int): + """The built-in all-reduce of a prefill / mixed-step boundary of ``rows`` rows: MNNVL (PREFILL-MNNVL) up to its + rows, else the target's AUTO one.""" + if core.allreduce_mnnvl is not None and rows <= PREFILL_AR_MNNVL_MAX_ROWS: + taken("prefill_ar_mnnvl", None) + return core.allreduce_mnnvl + return core.allreduce + + +class MiniMaxM3Tp2ModelingV2(DecoderModel): + """The MiniMax-M3 text model of this target: its weights, its runtime resources, and the per-step dispatch.""" + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + _validate_runtime(model_config) + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + _pin_allreduce_heuristic(mapping.tp_rank) + self.mapping = mapping + self.vocab_size = cfg.vocab_size + d = _weights.MODEL_WEIGHTS.dims(self) + self.num_layers = d.num_layers + self.hidden = d.hidden + self.heads = d.heads + self.kv_heads = d.kv_heads + self.head_dim = d.head_dim + self.index_heads = d.index_heads + self.index_dim = d.index_dim + self.q_size = d.heads * d.head_dim + self.kv_size = d.kv_heads * d.head_dim + self.moe_inter = d.moe_inter + self.num_experts = d.num_experts + self.top_k = int(cfg.num_experts_per_tok) + self.routed_scaling_factor = float(getattr(cfg, "routed_scaling_factor", 1.0)) + self.rms_eps = float(cfg.rms_norm_eps) + self.swiglu_alpha = float(getattr(cfg, "swiglu_alpha", 1.702)) + self.swiglu_beta = 1.0 # the (up + 1) offset of SwiGLU-OAI + self.swiglu_limit = float(getattr(cfg, "swiglu_limit", 7.0)) + self.max_num_tokens = model_config.max_num_tokens + self.moe_max_num_tokens = model_config.moe_max_num_tokens + self.sparse_layers = frozenset(_sparse_layer_ids(cfg)) + self.moe_layers = frozenset(_moe_layer_ids(cfg)) + mode = model_config.quant_config.quant_mode + self.main_kv_is_fp8 = bool(mode is not None and mode.has_fp8_kv_cache()) + self.main_kv_is_nvfp4 = bool(mode is not None and mode.has_fp4_kv_cache()) + # The sparse layers' QKV and index projections: one five-way packed projection, or the QKV projection and a + # separate MXFP8 index projection (fuse_qkv_index_projection=False) run on two streams. + self.fused_projection = _weights.fused_projection(model_config.sparse_attention_config) + + self.w = _weights.MODEL_WEIGHTS.declare(d) + + # The engine reads these two off the module tree: the MXFP8 GEMM state it tunes, and the opt-in to the + # automatic FlashInfer MXFP8 path on decode graphs. + self.mxfp8 = _Mxfp8GemmState() + self._use_flashinfer_mxfp8_decode_graph_default = True + + # The built-in fused all-reduce + residual + RMSNorm of the layer boundaries the multicast kernel does not take. + self.allreduce = AllReduce( + mapping=mapping, strategy=model_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + # PREFILL-MNNVL: the prefill / mixed-step boundaries' all-reduce (None: switched off). + self.allreduce_mnnvl = ( + AllReduce(mapping=mapping, strategy=AllReduceStrategy.MNNVL, dtype=cfg.torch_dtype) + if PREFILL_AR_MNNVL + else None + ) + if self.allreduce_mnnvl is not None and self.allreduce_mnnvl.mnnvl_allreduce is not None: + # Size MNNVL's shared workspace for the largest step it takes now, so the growth lands before the KV cache + # is sized (the memory-estimation step runs more rows than the pin covers and takes AUTO). + get_or_scale_allreduce_mnnvl_workspace( + mapping, + cfg.torch_dtype, + buffer_size_bytes=MNNVLAllReduce.get_required_workspace_size( + PREFILL_AR_MNNVL_MAX_ROWS, cfg.hidden_size, mapping.tp_size, cfg.torch_dtype + ), + ) + + # Streams: the shared / routed expert overlap and, when the MoE chunks its input, the chunking overlap. + self.aux_stream_moe = torch.cuda.Stream() + chunked = self.moe_max_num_tokens < model_config.max_num_tokens * mapping.dp_size + self.aux_stream_moe_chunk = torch.cuda.Stream() if chunked else None + self.moe_events = {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers} + self.moe_chunk_events = { + i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers + } + # The unfused sparse layers' index branch overlaps the QKV branch on its own stream. + self.aux_stream_attn = None if self.fused_projection else torch.cuda.Stream() + self.attn_events = ( + {} + if self.fused_projection + else {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.sparse_layers} + ) + + # Partial NeoX RoPE (rotary_dim of head_dim): the cos/sin table every attention kernel reads. + rope = RopeParams.from_config(cfg) + self.rope_theta = rope.theta + self.rotary_dim = int(rope.dim) + self.rotary_cos_sin = rope.create_rope_const_params(interleave=False)[1].reshape( + rope.max_positions, 2, -1 + ) + + # One MSA attention backend per layer (dense and sparse; the step metadata is the backend's). + quant_config = model_config.get_quant_config() + self.attn: List[MiniMaxM3MsaSparseAttention] = [] + for i in range(self.num_layers): + backend = create_attention( + model_config.attn_backend, + i, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=None, + quant_config=quant_config, + skip_create_weights_in_init=model_config.skip_create_weights_in_init, + q_scaling=1.0, + sparse_params=model_config.sparse_attention_config.to_sparse_params( + pretrained_config=cfg, layer_idx=i + ), + ) + backend.update_quant_config(quant_config) + if not isinstance(backend, MiniMaxM3MsaSparseAttention): + raise ValueError( + f"layer {i}: expected the MiniMax-M3 MSA backend, got {type(backend).__name__}" + ) + if i not in self.sparse_layers: + # The KV split trtllm-gen dense decode runs at, per request count. + backend.dense_decode_max_seq_len_by_batch = DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH + self.attn.append(backend) + _prefill_attention_opt_ins(self.attn) + + # Built once the weights are real (``post_load_weights``): the KV (Q, K, V) scales, the MoE scalars, the + # decode engines and workspaces, the two targets. + self.kv_scales: Dict[int, torch.Tensor] = {} + self.inv_kv_scales: Dict[int, torch.Tensor] = {} + self.moe: Dict[int, _MoeScalars] = {} + self.mx_alpha: Optional[torch.Tensor] = None + self.moe_engines: Dict[int, Dict[Tuple[int, bool, bool], Optional[object]]] = {} + self.moe16_engines: Dict[int, Dict[int, Optional[object]]] = {} + self.embed_owners: Dict[tuple, DecodeEmbedding] = {} + # Whether the 9..16-row generation router (ROUTER_TC_GEN) runs; decided at prebuild. + self.router_tc_cluster16 = False + # S1B_FOLD's pinned split factor per generation row count (prebuild_kernels; empty = the stock router). + self.s1b_pins: Dict[int, int] = {} + # Each sparse layer's selected block ids per (KV head, query row, slot) at its last long-context decode, which + # that decode prefetches into L2 before its grid wait (``SPARSE_DECODE_LC_PREFETCH_SELF_KV``, up to + # ``N1_MAX_ROWS`` query rows). Allocated once here (stable addresses for CUDA-graph capture); none when + # TP2WIDE_LC2 is killed. + self.n1_ids: Dict[int, torch.Tensor] = ( + { + i: torch.full( + (self.kv_heads, N1_MAX_ROWS, MSA_REQUIRED_TOPK), + -1, + dtype=torch.int32, + device=torch.device("cuda", torch.cuda.current_device()), + ) + for i in sorted(self.sparse_layers) + } + if SPARSE_DECODE_LC_PREFETCH_SELF_KV and _sparse_decode_kernel.LC_KEYS + else {} + ) + self._built_for: Optional[tuple] = None + self._prefill: Optional[PrefillTarget] = None + self._decode: Optional[DecodeTarget] = None + + def sm_scale(self, i: int) -> float: + return (self.head_dim**-0.5) / float(self.attn[i].q_scaling) + + # ---------------------------------------------------------------------------------------------- after the load + + def post_load_weights(self) -> None: + """Derive the kernels' state from the loaded weights, then build both targets and every decode kernel. + + Idempotent per set of weight storages: the loader's post-load walk reaches this module directly and through + the shell, and a second call on the same storages does nothing. + """ + identity = ( + self.w["embed"].data_ptr(), + self.w["l0_in_norm"].data_ptr(), + self.w["final_norm"].data_ptr(), + ) + if self._built_for == identity: + return + device = self.w["embed"].device + # MetaInitMode redirects even an explicit CUDA empty to meta. Reserve only + # after materialization, still before executor KV capacity profiling. + if self.main_kv_is_nvfp4: + with torch.cuda.device(device): + prepare_lc2_split_workspace(self.kv_heads) + derived = nn.Module() + derived.register_buffer( + "mx_alpha", torch.ones(1, dtype=torch.float32, device=device), persistent=False + ) + for i in range(self.num_layers): + # No KV calibration in the checkpoint: unit (Q, K, V) scales and their reciprocals. + derived.register_buffer( + f"l{i}_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + derived.register_buffer( + f"l{i}_inv_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + for i in sorted(self.moe_layers): + for name, value in dataclasses.asdict(self._derive_moe(i)).items(): + derived.register_buffer(f"l{i}_{name}", value, persistent=False) + self.derived = derived + self.mx_alpha = derived.mx_alpha + self.kv_scales = {i: getattr(derived, f"l{i}_kv_scales") for i in range(self.num_layers)} + self.inv_kv_scales = { + i: getattr(derived, f"l{i}_inv_kv_scales") for i in range(self.num_layers) + } + self.moe = { + i: _MoeScalars( + **{ + f.name: getattr(derived, f"l{i}_{f.name}") + for f in dataclasses.fields(_MoeScalars) + } + ) + for i in sorted(self.moe_layers) + } + self._build_embed_owners() + self.prebuild_kernels() + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + _sp_init(self) + self._built_for = identity + + def cache_derived_state(self) -> None: + """The staged loaders' entry to the same build (``post_load_weights``).""" + self.post_load_weights() + + def _derive_moe(self, i: int) -> _MoeScalars: + """The routed experts' kernel scalars from the checkpoint's per-tensor scales (trtllm-gen NVFP4 layout). + + Global input scales: the reciprocal of the largest per-expert input scale (gate and up must agree). Per expert: + ``alpha = 1 / (input_scale * (1 / weight_scale_2))`` for the gate (fc31_alpha), the up half (its GEMM2-input + rescale ``fc31_scale_c = fc2_input_scale * up_alpha``) and the down projection; SwiGLU-OAI's activation + constants per expert, with beta and the clamp in the pre-dequant GEMM output domain (divided by fc31_alpha). + """ + w = self.w + isc1, isc3, isc2 = w[f"l{i}_e_isc1"].data, w[f"l{i}_e_isc3"].data, w[f"l{i}_e_isc2"].data + if not torch.allclose(isc1, isc3): + raise ValueError(f"layer {i}: the routed experts' w1 and w3 input scales differ") + fc31_input_scale = isc1.max().reciprocal() + fc2_input_scale = isc2.max().reciprocal() + ws2_1, ws2_3, ws2_2 = ( + w[f"l{i}_e_ws2_1"].data, + w[f"l{i}_e_ws2_3"].data, + w[f"l{i}_e_ws2_2"].data, + ) + fc31_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_1)) + up_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_3)) + fc2_alpha = 1.0 / (fc2_input_scale * (1.0 / ws2_2)) + fc31_scale_c = fc2_input_scale * up_alpha + n = self.num_experts + device = fc31_alpha.device + act_alpha = torch.full((n,), self.swiglu_alpha, dtype=torch.float32, device=device) + act_beta = torch.full((n,), self.swiglu_beta, dtype=torch.float32, device=device) + act_clamp = torch.full((n,), self.swiglu_limit, dtype=torch.float32, device=device) + act_beta.div_(fc31_alpha) + act_clamp.div_(fc31_alpha) + return _MoeScalars( + fc31_input_scale=fc31_input_scale, + fc2_input_scale=fc2_input_scale, + fc31_alpha=fc31_alpha, + fc2_alpha=fc2_alpha, + fc31_scale_c=fc31_scale_c, + act_alpha=act_alpha, + act_beta=act_beta, + act_clamp=act_clamp, + ) + + def _build_embed_owners(self) -> None: + """The fused embedding + first RMSNorm's caller-owned buffers, one per (rows, id dtype) of a decode step.""" + table, weight = self.w["embed"], self.w["l0_in_norm"] + self.embed_owners = { + (rows, dtype): DecodeEmbedding(table, weight, self.rms_eps, rows, dtype) + for rows in range(1, EMBED_FIRSTNORM_MAX_TOKENS + 1) + for dtype in (torch.int32, torch.int64) + } + + def projections(self) -> List[Tuple[str, str, Optional[str]]]: + """Every MXFP8 projection in layer order: (weight key, scale key, the boundary its GEMV pushes into).""" + out = [] + for i in range(self.num_layers): + out.append((f"l{i}_qkv_w", f"l{i}_qkv_s", None)) + out.append((f"l{i}_o_w", f"l{i}_o_s", "attn")) + if i in self.moe_layers: + out.append((f"l{i}_sh_gu_w", f"l{i}_sh_gu_s", None)) + out.append((f"l{i}_sh_dn_w", f"l{i}_sh_dn_s", "moe")) + else: + out.append((f"l{i}_mlp_gu_w", f"l{i}_mlp_gu_s", None)) + out.append((f"l{i}_mlp_dn_w", f"l{i}_mlp_dn_s", "dense")) + return out + + def decode_query_lens(self) -> List[int]: + """The query rows per request a decode step can have: 1, and 1 + each draft length speculation runs.""" + lens = {1} + spec = getattr(self.model_config, "spec_config", None) + if spec is not None: + # The verify width the config itself declares, whatever it names its draft count. + per_step = getattr(spec, "tokens_per_gen_step", None) + if isinstance(per_step, int) and per_step > 1: + lens.add(per_step) + for drafts in ( + getattr(spec, "max_draft_len", None), + getattr(spec, "max_total_draft_tokens", None), + *(getattr(spec, "draft_len_schedule", None) or {}).values(), + ): + if drafts: + lens.add(1 + int(drafts)) + return sorted(lens) + + def prebuild_kernels(self) -> Dict[str, object]: + """Compile (and load) every kernel a decode forward dispatches to that would otherwise compile on its first + call: each projection's decode GEMVs (every row count, activation and boundary push), the MoE engines of + ``MOE_ENGINE`` / ``MOE16_ENGINE``'s token counts, and the index scorer at the decode query lengths. Runs on the + current (main) stream before the executor's warmup, in the same order on every rank, then synchronizes; after + it, GEMV compiles are frozen and the forward only looks kernels up (a missing one is a miss). It also builds + the T2SC scorer (``INDEX_SCORE_T2SC_PATH``), the 384-thread long-row select (``LONG_SELECT_DSL``) and the dense + layers' full-context decode plans (``FULLCTX_DECODE``). Returns the seconds each part took and the scorer + keys.""" + times: Dict[str, object] = {} + # F3: the o_proj fork's tactics and the boundary op, before warmup / capture, on the real workspace. + _f3_t0 = time.perf_counter() + _f3_init(self) + times["f3"] = round(time.perf_counter() - _f3_t0, 3) + # The 9..16-row generation router: decided once, from the device, never inside a CUDA-graph capture. + # The built-in gate's conditions, mirrored exactly: 128 experts, a hidden size in 1024s, 16-CTA clusters. + self.router_tc_cluster16 = ( + self.num_experts == 128 + and self.hidden % 1024 == 0 + and bool(torch.ops.trtllm.minimax_m3_router_tc_cluster16(self.hidden)) + ) + logger.info( + f"[M3 prebuild] rank {mpi_rank()}: TF32 tensor-core router for {ROUTER_TC_GEN_MIN_ROWS}.." + f"{ROUTER_TC_GEN_MAX_ROWS}-row generation steps: " + + ( + "engaged (16-CTA clusters)" + if self.router_tc_cluster16 + else "not used (no 16-CTA clusters)" + ) + ) + # S1B_FOLD: pin the router's split factor per generation row count above the TC router's (bitwise probe of this + # device's cuBLASLt, outside capture); row counts without a pin keep the stock router + routing. + start_s1b = time.perf_counter() + self.s1b_pins = m31_s1b_fold.prebuild( + [self.w[f"l{i}_router"] for i in sorted(self.moe_layers)], + range(ROUTER_TC_GEN_MAX_ROWS + 1, S1B_MAX_ROWS + 1), + ) + times["s1b_fold_pins_s"] = round(time.perf_counter() - start_s1b, 3) + if not self.mxfp8.quant_method.use_cutlass: + raise ValueError("the MiniMax-M3 decode dispatch needs the CUTLASS MXFP8 GEMM") + if PREPROCESS_TOKMAP_FIX: + # Every path that can run the stock generation preprocess (the drafter's full-context decode or its + # fallback, any trtllm-gen fallback), so it is installed whatever else is enabled. + _tokmap_guard.install( + torch.device("cuda", torch.cuda.current_device()), + DECODE_MAX_TOKENS, + lambda rows: taken(PREPROCESS_TOKMAP_PATH, rows, "one-request token map"), + lambda rows, reason: missed(PREPROCESS_TOKMAP_PATH, rows, reason), + ) + mx_gemv.freeze(False) + use_pdl = get_env_enable_pdl() + world = int(self.mapping.tp_size) + start = time.perf_counter() + for wk, sk, push in self.projections(): + weight, scale = self.w[wk], self.w[sk] + mx_gemv.ensure_compiled(weight, scale, use_pdl) + n, k = weight.shape + for rows, x_bf16, ar in _gemv_variants(n, k, push, world): + # An MXFP8 call this target gives a (split, ring) override (``mx_gemv_cfg``, as ``_linear`` passes + # it) compiles that config. + cfg = None if x_bf16 else mx_gemv_cfg(n, k, rows) + if cfg is not None: + _precompile_cfg(weight, scale, rows, cfg, use_pdl, ar=ar) + else: + mx_gemv.precompile(weight, scale, rows, x_bf16, use_pdl=use_pdl, ar=ar) + # QKV_PRODUCER_FOLD: the packed QKV + index projection also takes these rows with the producer epilogue. + fold_cfg = None if x_bf16 or ar is not None else _fold_cfg(n, k, rows) + if fold_cfg is not None: + _precompile_cfg(weight, scale, rows, fold_cfg, use_pdl, prod=True) + # SWQ (Triton) compiles on first use, which must not happen inside a graph capture: one call per down + # projection shape it serves covers every row count. + swq_shapes = {tuple(self.w[wk].shape) for wk, _, _ in self.projections()} & set(SWQ_ROWS) + for n, k in sorted(swq_shapes): + if not SWQ_ROWS[(n, k)]: + continue + gu = torch.zeros( + max(SWQ_ROWS[(n, k)]), 2 * k, dtype=torch.bfloat16, device=self.w["embed"].device + ) + m3_swq(gu, self.swiglu_limit, self.swiglu_alpha, self.swiglu_beta) + logger.info(f"[M3 prebuild] SWQ: compiled for intermediate {k}") + torch.cuda.synchronize() + times["gemv_s"] = time.perf_counter() - start + start = time.perf_counter() + for i in sorted(self.moe_layers): + self._prebuild_engines(i) + torch.cuda.synchronize() + times["moe_s"] = time.perf_counter() - start + start = time.perf_counter() + times["scorer_keys"] = self._prebuild_index_scorer( + self.decode_query_lens(), self.w["embed"].device + ) + times["scorer_t2sc_keys"] = self._prebuild_index_scorer_t2sc( + self.decode_query_lens(), self.w["embed"].device + ) + times["scorer_intree_keys"] = self._prebuild_index_scorer_intree( + self.decode_query_lens(), self.w["embed"].device + ) + if self.sparse_layers and self.attn[min(self.sparse_layers)].indexer_kv_dtype == "nvfp4": + # AGG_NVFP4_INDEXK: the Q8KV4 decode reader at every decode query length (DecodeTarget scores through it). + reader = Nvfp4IndexDecodeScorer(self.index_heads) + if not reader.available(): + raise RuntimeError( + "indexer_kv_dtype='nvfp4' needs the vendored fmha_sm100_q8kv8 package with run_scores_hpr (v3h); " + "TRTLLM_M31_DISABLE_AGG_NVFP4_INDEXK=1 falls back to FP8 index-K" + ) + times["nvfp4_reader_qlens"] = reader.prebuild( + self.w["embed"].device, self.decode_query_lens() + ) + logger.info( + f"[M3 prebuild] AGG_NVFP4_INDEXK decode reader compiled: qlen {times['nvfp4_reader_qlens']}" + ) + torch.cuda.synchronize() + times["scorer_s"] = time.perf_counter() - start + start = time.perf_counter() + times["select_keys"] = [] + if long_select_dsl_keys(): + # The 384-thread long-row select (LONG_SELECT_DSL): one kernel per launch key. + times["select_keys"] = _select_long_dsl_kernel.prebuild( + long_select_dsl_keys(), self.w["embed"].device + ) + torch.cuda.synchronize() + times["select_s"] = time.perf_counter() - start + start = time.perf_counter() + times["fullctx_plans"] = 0 + if FULLCTX_DECODE.keys: + # The dense layers' full-context decode: one compiled PrimTS plan per covered (rows, requests). + times["fullctx_plans"] = _fullctx_decode_kernel.prebuild( + self.w["embed"].device, + {key: FULLCTX_DECODE_RECIPES[key] for key in FULLCTX_DECODE.keys}, + FULLCTX_DECODE_MAX_KV_LEN, + FULLCTX_DECODE_OWN_REDUCER and _fullctx_decode_kernel.has_own_reducer(), + get_env_enable_pdl(), + ) + torch.cuda.synchronize() + times["fullctx_s"] = time.perf_counter() - start + start = time.perf_counter() + times["proxy_variants"] = self._prebuild_msa_proxy() if MSA_PROXY_PREBUILD else [] + if times["proxy_variants"]: + m3_report.work("PROXYPB", f"{len(times['proxy_variants'])} proxy variants built") + times["proxy_s"] = time.perf_counter() - start + mx_gemv.freeze(True) + logger.info( + "[M3 prebuild] decode kernels built before warmup: GEMVs %.1f s, MoE engines %.1f s, index scorer " + "%.1f s (%d variants), long-row select %.1f s (%d variants), full-context decode %.1f s (%d plans), " + "MSA indexer proxy %.1f s (%d variants)" + % ( + times["gemv_s"], + times["moe_s"], + times["scorer_s"], + len(times["scorer_keys"]), + times["select_s"], + len(times["select_keys"]), + times["fullctx_s"], + times["fullctx_plans"], + times["proxy_s"], + len(times["proxy_variants"]), + ) + ) + return times + + # ------------------------------------------------------------------------------------------ the MoE engines + + def _expert_operands(self, i: int): + """The routed experts' operands the engines read (trtllm-gen NVFP4 SwiGLU-OAI layout), and the input scale.""" + w, m = self.w, self.moe[i] + tensors = [ + w[f"l{i}_e_w31"].data, + w[f"l{i}_e_s31"].data, + w[f"l{i}_e_w2"].data, + w[f"l{i}_e_s2"].data, + m.fc31_alpha, + m.fc31_scale_c, + m.act_alpha, + m.act_beta, + m.act_clamp, + m.fc2_alpha, + ] + return tensors, m.fc31_input_scale + + def _shared_weights(self, i: int): + w = self.w + return ( + w[f"l{i}_sh_gu_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_gu_s"].data, + w[f"l{i}_sh_dn_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_dn_s"].data, + ) + + def engine(self, i: int, num_tokens: int, shared: bool) -> Optional[object]: + """Layer ``i``'s prebuilt routed-expert engine for ``num_tokens`` (1..8), router folded in (and the shared + expert with ``shared``). None after a miss: the build failed or was never made (a forward never builds one).""" + return self._lookup( + i, self.moe_engines[i], (num_tokens, True, shared), MOE_ENGINE.path, num_tokens + ) + + def moe16_engine(self, i: int, num_tokens: int) -> Optional[object]: + """Layer ``i``'s prebuilt 9..13-row engine (router, routed and shared experts in one kernel), or None.""" + if not MOE16_MIN_TOKENS <= num_tokens <= MOE16_MAX_TOKENS: + return None + return self._lookup(i, self.moe16_engines[i], num_tokens, MOE16_ENGINE.path, num_tokens) + + def _lookup(self, i: int, engines: dict, key, path: str, num_tokens: int) -> Optional[object]: + if key not in engines: + what = f"layer {i}'s engine" + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, what) + else: + missed(path, num_tokens, f"{what} was not built before warmup") + return None + engine = engines[key] + if engine is None: + missed(path, num_tokens, "its build failed (logged)") + return engine + + def _prebuild_engines(self, i: int) -> None: + """Build (compile, on first use of each shared state, and warm) every engine a decode step of layer ``i`` can + take: ``MOE_ENGINE``'s token counts with the shared expert folded in, and ``MOE16_ENGINE``'s. Whether an + engine is kept is agreed across the TP group (one host collective per engine): a build that fails on one rank + drops that engine on every rank, so the ranks never mix an engine (and its boundary push) with the generic + MoE.""" + engines = self.moe_engines.setdefault(i, {}) + engines16 = self.moe16_engines.setdefault(i, {}) + for num_tokens in sorted(MOE_ENGINE.keys): + key = (num_tokens, True, True) + if key not in engines: + engines[key] = m3_mc_allreduce.tp_agree_built( + self.mapping, + self._try_build( + i, MOE_ENGINE.path, num_tokens, self._build_engine, shared=True + ), + f"[M3 dispatch] {MOE_ENGINE.path} at {num_tokens} rows: layer {i}'s engine", + ) + for num_tokens in sorted(MOE16_ENGINE.keys): + if num_tokens not in engines16: + engines16[num_tokens] = m3_mc_allreduce.tp_agree_built( + self.mapping, + self._try_build(i, MOE16_ENGINE.path, num_tokens, self._build_moe16_engine), + f"[M3 dispatch] {MOE16_ENGINE.path} at {num_tokens} rows: layer {i}'s engine", + ) + + def _try_build(self, i: int, path: str, num_tokens: int, build, **kwargs) -> Optional[object]: + try: + return build(i, num_tokens, **kwargs) + except (ImportError, RuntimeError, ValueError) as err: + logger.warning( + f"[M3 dispatch] {path} at {num_tokens} rows: layer {i}'s engine failed to build: {err}" + ) + return None + + def _build_engine(self, i: int, num_tokens: int, shared: bool) -> object: + tensors, scale = self._expert_operands(i) + device = tensors[0].device + sms = torch.cuda.get_device_properties(device).multi_processor_count + kwargs = {} + if shared: + kwargs["shared_weights"] = self._shared_weights(i) + state = _ENGINE_STATES.get((str(device), num_tokens, shared)) + if state is None: + state = _ENGINE_STATES[(str(device), num_tokens, shared)] = M3MoeState( + device, + num_tokens=num_tokens, + num_ctas=sms + - (shared_ctas_for(num_tokens, shared) if shared else MOE_ENGINE_RESERVED_SMS), + shared=shared, + ) + engine = state.layer( + *tensors, + self.w[f"l{i}_router_bias"].data, + router_weight=self.w[f"l{i}_router"].data, + fc31_input_scale=scale.reshape(1).float(), + **kwargs, + ) + # Compile (first use of the state) and warm. + m3_moe_engine( + torch.zeros(num_tokens, tensors[0].shape[-1] * 2, dtype=torch.bfloat16, device=device), + engine, + ) + return engine + + def _build_moe16_engine(self, i: int, num_tokens: int) -> object: + if MOE16_KERNEL_WIDTH != _MOE_INTER: + # MOE16_TP2 opted in: the engine's kernel is the TP4-width build; never run it on TP2-width weights. + raise RuntimeError( + f"the 9..13-row engine is built for expert width {MOE16_KERNEL_WIDTH}, this target's is {_MOE_INTER} " + "(no TP2-width build yet); the generic MoE runs" + ) + tensors, scale = self._expert_operands(i) + device = tensors[0].device + state = _MOE16_STATES.get((str(device), num_tokens)) + if state is None: + state = _MOE16_STATES[(str(device), num_tokens)] = M3Moe16State( + device, num_tokens=num_tokens + ) + engine = state.layer( + *tensors, + self.w[f"l{i}_router_bias"].data, + router_weight=self.w[f"l{i}_router"].data, + fc31_input_scale=scale.reshape(1).float(), + shared_weights=self._shared_weights(i), + ) + # Compile (first use of the state) and warm. + m3_moe16_engine( + torch.zeros(num_tokens, tensors[0].shape[-1] * 2, dtype=torch.bfloat16, device=device), + engine, + ) + return engine + + # ------------------------------------------------------------------------------------------ the index scorer + + def _scorer_variants(self, query_lens) -> list: + """The scorer keys a decode step can take: every sparse layer's (index dtype, index heads per rank, index dim) + at each decode query length, with the early-issue schedule and without it, where the kernel serves it.""" + runner = _index_score_kernel.IndexScoreRunner + keys = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + config = backend.m3_config + dtype = ( + torch.float8_e4m3fn + if backend.indexer_kv_dtype in ("fp8", "nvfp4") + else torch.bfloat16 + ) + for qlen in sorted(set(query_lens)): + if not runner.is_supported( + q_dtype=dtype, + num_heads=config.num_index_heads, + head_dim=config.sparse_index_dim, + page_size=runner.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ): + continue + for early in (True, False): + key = ( + dtype, + int(config.num_index_heads), + int(qlen), + int(config.sparse_index_dim), + int(INDEX_SCORE_SPLIT_K), + bool(early), + ) + if key not in keys: + keys.append(key) + return keys + + def _prebuild_msa_proxy(self) -> list: + """Compile and load every fmha_sm100 variant the MSA indexer's proxy pass can pick (OnlyScore over the paged + index-K: one KV head, page 128, no split-KV), so no context chunk compiles one after the server is ready (a + 65..128-token chunk's variant did, stalling the steps behind it by 0.8-1.6 ms). The variant follows only the + longest query row: fmha_sm100 packs a row of up to 32 tokens by its pack factor, then takes the 128- or 256-row + query tile and the single-warpgroup kernel from the packed length, so query lengths 1..129 reach every variant. + Compile and load only, nothing runs: no numerics change. fmha_sm100 keys its cache by its installed sources, so + a source overlay builds its own variants. Returns the variant names.""" + require_msa_module() + from fmha_sm100 import api as fmha_api + from fmha_sm100 import jit as fmha_jit + + only_score, page = 2, 128 + names = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + dtype = ( + torch.float8_e4m3fn + if backend.indexer_kv_dtype in ("fp8", "nvfp4") + else torch.bfloat16 + ) + heads = int(backend.m3_config.num_index_heads) + for qlen in range(1, 130): + pack = fmha_api._compute_pack_factor(qlen, heads, 1) + packed = qlen * pack + key = ( + fmha_jit._dlpack_dtype_code(dtype), + 128 if packed <= 128 else 256, + packed <= 64, + only_score, + page, + False, + pack, + ) + name, _ = fmha_jit._variant_key_from_runtime(*key) + if name not in names: + fmha_jit.get_fmha_variant(*key) + names.append(name) + return names + + def _prebuild_index_scorer(self, query_lens, device) -> list: + """Compile every scorer key and launch each once on real-shaped dummy operands (loading its module), on the + current stream. The kernel's batch, token, page and block counts are symbolic: its compile needs no step + metadata. Returns the keys.""" + runner = _index_score_kernel.IndexScoreRunner + keys = self._scorer_variants(query_lens) + page = runner.SUPPORTED_PAGE_SIZE + for key in keys: + dtype, heads, qlen, dim, split_k, early = key + runner._compile(dtype, heads, qlen, dim, split_k, early) + rows = 1 if early else INDEX_SCORE_EARLY_MAX_REQUESTS + 1 + blocks = 2 + tokens = rows * qlen + idx_q = torch.zeros(tokens, heads, dim, dtype=torch.bfloat16, device=device).to(dtype) + idx_k = torch.zeros( + rows * blocks, 1, page, dim, dtype=torch.bfloat16, device=device + ).to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device=device).view( + rows, blocks + ) + seq_lens = torch.full((rows,), blocks * page, dtype=torch.int32, device=device) + score = torch.empty(heads, blocks, tokens, dtype=torch.float32, device=device) + runner.kernel_cache[key]( + idx_q, idx_k.squeeze(1), block_table, score.transpose(1, 2), seq_lens + ) + return keys + + def _prebuild_index_scorer_t2sc(self, query_lens, device) -> list: + """T2SC (``INDEX_SCORE_T2SC_PATH``): compile this target's copy of the TP2 / TP8 scorer at every covered request + count's schedule (``index_score_t2sc_schedule``) for every sparse layer geometry and decode query length, and + launch each once on real-shaped dummy operands (loading its module). Returns the keys.""" + runner = _index_score_t2sc_kernel.IndexScoreRunner + keys = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + config = backend.m3_config + dtype = ( + torch.float8_e4m3fn + if backend.indexer_kv_dtype in ("fp8", "nvfp4") + else torch.bfloat16 + ) + for qlen in sorted(set(query_lens)): + if not runner.is_supported( + q_dtype=dtype, + num_heads=config.num_index_heads, + head_dim=config.sparse_index_dim, + page_size=runner.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ): + continue + for requests in sorted(INDEX_SCORE_T2SC_PATH.keys): + split_k, early = index_score_t2sc_schedule(requests) + key = ( + dtype, + int(config.num_index_heads), + int(qlen), + int(config.sparse_index_dim), + int(split_k), + bool(early), + ) + if all(key != seen for seen, _ in keys): + keys.append((key, requests)) + page = runner.SUPPORTED_PAGE_SIZE + for key, rows in keys: + dtype, heads, qlen, dim, split_k, early = key + runner._compile(dtype, heads, qlen, dim, split_k, early) + blocks = 2 + tokens = rows * qlen + idx_q = torch.zeros(tokens, heads, dim, dtype=torch.bfloat16, device=device).to(dtype) + idx_k = torch.zeros( + rows * blocks, 1, page, dim, dtype=torch.bfloat16, device=device + ).to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device=device).view( + rows, blocks + ) + seq_lens = torch.full((rows,), blocks * page, dtype=torch.int32, device=device) + score = torch.empty(heads, blocks, tokens, dtype=torch.float32, device=device) + runner.kernel_cache[key]( + idx_q, idx_k.squeeze(1), block_table, score.transpose(1, 2), seq_lens + ) + return [key for key, _ in keys] + + def _prebuild_index_scorer_intree(self, query_lens, device) -> list: + """TP2 WIDE (``INDEX_SCORE_INTREE``): compile the in-tree CuTe DSL scorer at every sparse layer geometry and + decode query length a step of ``INDEX_SCORE_INTREE_MIN_REQUESTS`` or more requests can take, and launch each + once on real-shaped dummy operands (loading its module). Returns the keys.""" + if not INDEX_SCORE_INTREE: + return [] + in_tree = _in_tree_scorer() + keys = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + config = backend.m3_config + dtype = ( + torch.float8_e4m3fn + if backend.indexer_kv_dtype in ("fp8", "nvfp4") + else torch.bfloat16 + ) + for qlen in sorted(set(query_lens)): + if max( + qlen * INDEX_SCORE_INTREE_MIN_REQUESTS, TP2_WIDE_MIN_ROWS + ) > DECODE_STEP_MAX or not in_tree.is_supported( + q_dtype=dtype, + num_heads=config.num_index_heads, + head_dim=config.sparse_index_dim, + page_size=in_tree.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ): + continue + key = ( + dtype, + int(config.num_index_heads), + int(qlen), + int(config.sparse_index_dim), + int(in_tree.SPLIT_K), + ) + if key not in keys: + keys.append(key) + page = in_tree.SUPPORTED_PAGE_SIZE + for key in keys: + dtype, heads, qlen, dim, split_k = key + in_tree._compile(dtype, heads, qlen, dim, split_k) + rows, blocks = 1, 2 + tokens = rows * qlen + idx_q = torch.zeros(tokens, heads, dim, dtype=torch.bfloat16, device=device).to(dtype) + idx_k = torch.zeros( + rows * blocks, 1, page, dim, dtype=torch.bfloat16, device=device + ).to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device=device).view( + rows, blocks + ) + seq_lens = torch.full((rows,), blocks * page, dtype=torch.int32, device=device) + score = torch.empty(heads, blocks, tokens, dtype=torch.float32, device=device) + in_tree.kernel_cache[key]( + idx_q, idx_k.squeeze(1), block_table, score.transpose(1, 2), seq_lens + ) + return keys + + # --------------------------------------------------------------------------------------------- the dispatch + + def _select_target(self, attn_metadata: AttentionMetadata): + """Which target runs this step: one host-side integer, fixed per CUDA-graph capture. A step with context rows + (a mixed step included) is prefill's; every pure generation step is decode's, whatever its width.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.IntTensor] = None, + position_ids: Optional[torch.IntTensor] = None, + inputs_embeds: Optional[torch.FloatTensor] = None, + spec_metadata: Optional[SpecMetadata] = None, + capture: Callable[..., None] = _plain_capture, + capture_target: Optional[Callable[..., Any]] = None, + **kwargs, + ) -> torch.Tensor: + """``capture_target``: the worker's capture-norm target (``CAPTURE_NORM_IN_BOUNDARY``) for a capture layer + on the decode fast path; None: the capture runs its own norm.""" + if (input_ids is None) ^ (inputs_embeds is not None): + raise ValueError("You must specify exactly one of input_ids or inputs_embeds") + if kwargs.get("lora_params") is not None: + raise ValueError("this MiniMax-M3 target serves no LoRA adapters") + return self._select_target(attn_metadata).forward( + attn_metadata, + input_ids, + position_ids, + inputs_embeds, + spec_metadata, + capture, + capture_target, + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# The targets +# -------------------------------------------------------------------------------------------------------------------- + + +# PREFILL_BRK (ported from the M3 sm_107 tp2 target): a step with context rows runs under breakable +# (piecewise) prefill CUDA graphs, as the aggregated deployment captures prefill and mixed steps. The projections, +# norms, MoE and boundaries are captured per padded row bucket; the attention boundary (the horizontal producer over +# the live rows, then the MSA backend) runs eagerly at every replay (``_prefill_attention_eager``), reading the live +# row count from the step's metadata, as the built-in model's MiniMax-M3 attention does. Without a breakable graph the +# step runs as before. The separate (unfused) index projection still declines breakable graphs; pure generation steps +# never run under one. Kill switch TRTLLM_M31_DISABLE_PREFILL_BRK=1 (the PrefillTarget raises under a breakable +# graph, as before the port). Read at import. +PREFILL_BRK = not m31_disabled("PREFILL_BRK") +PREFILL_BRK_PATH = "m3_prefill_brk_attention" +# PREFILL_OUT_CLONE (determinism): a prefill / mixed step returns a copy of its final +# hidden, taken on the current stream right after the last boundary. Without it a later write from another stream lands +# in the returned hidden's memory before the logits processor gathers the verify rows (K7, 8 x ~50K-token decodes with a +# 3840-row mixed step: up to 8 of 272 sampled requests differ run to run; with the copy 0 of 272). The copy is bitwise +# the hidden. Opt-in: TRTLLM_M31_ENABLE_PREFILL_OUT_CLONE=1; TRTLLM_M31_DISABLE_PREFILL_OUT_CLONE=1 wins. Read at +# import. +PREFILL_OUT_CLONE = m31_enabled("PREFILL_OUT_CLONE") and not m31_disabled("PREFILL_OUT_CLONE") +PREFILL_OUT_CLONE_PATH = "m31_prefill_out_clone" + +#: The cores the breakable prefill graphs' eager attention boundary looks up by key: an ``eager_on_graph`` callable +#: replays with weak references to its arguments, so they may be tensors or scalars only. +_BCG_CORES: Dict[str, "MiniMaxM3Tp2ModelingV2"] = {} + + +def _msa_attention_live(core, i: int, q, k, v, idx_q, md, output) -> None: + """The MSA backend over a step's live rows (``md.num_tokens``; a padded step's pad rows of ``output`` are left as + allocated): the KV write unless a producer wrote the caches, the sparse layers' top-k block selection, the FMHA + over the context rows and any generation rows beside them.""" + live = int(md.num_tokens) + sparse = i in core.sparse_layers + nvfp4 = core.main_kv_is_nvfp4 + if GQAMIX and sparse: + _gqamix_register(core.attn[i]) + if AGG_TSEL and sparse: + _tsel_register(core.attn[i]) + _record("m3_msa_attention", sparse, k is not None) + m3_msa_attention( + core.attn[i], + q[:live], + k[:live] if k is not None else None, + v[:live] if v is not None else None, + idx_q[:live] if idx_q is not None else None, + md, + output[:live], + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + + +def _prefill_attention_boundary( + packed, position_ids, q, k, v, idx_q, output, core_key: str, i: int +) -> None: + """The attention boundary of a prefill layer under a breakable prefill CUDA graph, run eagerly at every replay (as + the built-in model's MiniMax-M3 attention boundary is): everything that depends on the step's live row count. A + sparse layer passes its ``packed`` five-way projection: the horizontal producer runs here over the live rows + (K / V / index-K into the paged caches, Q / index-Q out), then the MSA backend; a dense layer passes q / k / v.""" + core = _BCG_CORES[core_key] + md = get_model_extra_attrs()["attention_metadata"]() + live = int(md.num_tokens) + if packed is not None: + horizontal = _sparse_qkv_producer( + core, + i, + packed[:live], + position_ids[..., :live] if position_ids is not None else None, + md.kv_cache_manager, + md.msa_idx_k_cache, + md.msa_out_cache_loc, + ) + if horizontal is None: + raise RuntimeError( + f"MiniMax-M3 sparse layer {i}: no horizontal producer takes this step's caches (an FP8 or NVFP4 HND " + "pool with an FP8 index-K cache)" + ) + q, idx_q = horizontal + k = v = None + _msa_attention_live(core, i, q, k, v, idx_q, md, output) + + +_prefill_attention_eager = eager_on_graph(_prefill_attention_boundary) + + +class PrefillTarget: + """A step with context rows: prefill chunks, and the mixed steps that carry generation rows beside them. Every + layer runs in its generic form; the attention core is the MSA backend's for every row. + + Under a breakable (piecewise) prefill CUDA graph (``PREFILL_BRK``) the projections, norms, MoE and boundaries are + captured per padded row bucket and the attention boundary runs eagerly at every replay + (``_prefill_attention_eager``: the live-row horizontal producer and the MSA backend), as the built-in model's + MiniMax-M3 attention does.""" + + def __init__(self, core: MiniMaxM3Tp2ModelingV2) -> None: + self.core = core + self._bcg_key = f"m31_tp2_core_{id(core)}" + _BCG_CORES[self._bcg_key] = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it; read once here rather than per forward. + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + Counts and per-step host scalars are read here; device tensors are the metadata's persistent buffers, held + by reference; ``msa_idx_k_cache`` is the metadata's per-layer index-K cache getter. ``metadata`` is the + prepared object itself: the MSA backend's step (``m3_msa_attention``) takes it as its argument. + """ + return dict( + num_contexts=md.num_contexts, + num_ctx_tokens=md.num_ctx_tokens, + num_tokens=md.num_tokens, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + metadata=md, + ) + + def forward( + self, + attn_metadata, + input_ids, + position_ids, + inputs_embeds, + spec_metadata, + capture, + capture_target, + ): + """``capture_target`` is unused: the generic boundary takes no multicast all-reduce at this TP + (``GENERIC_MC_AR_MAX_TOKENS`` 0), so a capture layer always runs its own capture norm.""" + core = self.core + if is_in_breakable_cuda_graph(): + if not PREFILL_BRK: + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) prefill CUDA graphs " + "(TRTLLM_M31_DISABLE_PREFILL_BRK is set)" + ) + taken(PREFILL_BRK_PATH, None, "attention boundary eager at every replay") + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + _record("prefill", int(step["num_tokens"]), int(step["num_contexts"])) + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + hidden, residual = self._layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + if PREFILL_OUT_CLONE: + taken(PREFILL_OUT_CLONE_PATH, None, "final hidden copied after the last boundary") + hidden = hidden.clone() + return hidden + + def _swiglu_mlp( + self, h: torch.Tensor, prefix: str, low_m_gemm: bool, dense: bool = False + ) -> torch.Tensor: + """A gated MXFP8 MLP (the dense MLP with ``dense``, the shared expert): gate_up, SwiGLU-OAI, down, no + reduction.""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s", low_m_gemm) + if dense: + act = _dense_swiglu(core, gu) + else: + act = m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + out, _ = _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm) + return out + + def _routed_chunks(self, i: int, x, router_logits): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: + the MoE scheduler's even split, even chunks on the MoE chunking stream.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + # CTXNORMQ: the boundary already wrote this activation's NVFP4 form. + fp4 = _CTXNORMQ_FP4.take(x) if CTXNORMQ else None + if num_chunks <= 1: + if fp4 is not None: + return _routed_experts(core, i, fp4[0], fp4[1], router_logits, True) + return _routed_experts(core, i, x, None, router_logits, True) + sizes = _split_chunk(rows, num_chunks) + xs, logits = x.split(sizes), router_logits.split(sizes) + sfs = [None] * num_chunks + if fp4 is not None: + xs = fp4[0].split(sizes) + sfs = [t.flatten() for t in fp4[1].view(rows, -1).split(sizes)] + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + else: + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor): + """Router + routed experts, finalized: the fp32 router logits and the experts quantizing their bf16 input + (chunked).""" + return self._routed_chunks(i, h, _router_logits(self.core, i, h)) + + def _moe(self, i: int, h: torch.Tensor, *, return_parts: bool): + """The MoE block of layer ``i`` without its reduction. Returns ``(out, routed)``: with ``return_parts`` the + shared and routed parts (routed None when the engine folded the shared expert in), else their sum in + ``out``.""" + core = self.core + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + h.dim() == 2 + and h.dtype == torch.bfloat16 + and 1 <= rows <= MOE_ENGINE_MAX_TOKENS + and rows >= MOE_FOLD_MIN_TOKENS + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + _record("m3_moe_engine", rows, False) + folded = m3_moe_engine(h.contiguous(), engine, None, None) + return folded, None + if not return_parts and _triton_finalize_rows(core, h): + # TP2AGG_SP: the keyed view holds no bf16 rows; the experts take the gathered NVFP4 products. + keyed = _sp_keyed(h) + fp4 = _CTXNORMQ_FP4.take(h) if keyed or (CTXNORMQ and TP2W_FP4SLOT_FINALIZE) else None + if fp4 is not None and not keyed: + _FP4SLOT_STATE["taken"] += 1 + if _FP4SLOT_STATE["taken"] in (1, 1000): + logger.info( + f"[M3.1 tp2] TP2W_FP4SLOT_FINALIZE taken n={_FP4SLOT_STATE['taken']} (rows {rows})" + ) + if _FP4SLOT_CHECK and not torch.cuda.is_current_stream_capturing(): + ref, ref_sf = fp4_quantize(h, core.moe[i].fc31_input_scale, 16, False, False) + d = int((ref.view(torch.uint8) != fp4[0].view(torch.uint8)).sum()) + int( + ( + ref_sf.view(torch.uint8).flatten() != fp4[1].view(torch.uint8).flatten() + ).sum() + ) + _FP4SLOT_STATE["checked"] += 1 + _FP4SLOT_STATE["diff"] += d + if d or _FP4SLOT_STATE["checked"] <= 2 or _FP4SLOT_STATE["checked"] % 500 == 0: + logger.info( + f"[M3.1 tp2] TP2W_FP4SLOT_CHECK rows {rows} diff {d} (checked " + f"{_FP4SLOT_STATE['checked']}, total diff {_FP4SLOT_STATE['diff']})" + ) + x_in, x_sf = (fp4[0], fp4[1]) if fp4 is not None else (h, None) + routed_parts, shared_out = maybe_execute_in_parallel( + lambda: _routed_experts(core, i, x_in, x_sf, _router_logits(core, i, h), False), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False), + core.moe_events[i][0], + core.moe_events[i][1], + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + return _triton_finalize(rows, routed_parts, shared_out), None + routed_out, shared_out = maybe_execute_in_parallel( + lambda: self._routed(i, h), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False), + core.moe_events[i][0], + core.moe_events[i][1], + # The warmup tuning pass profiles the shared expert's GEMMs: keep it on the main stream, between the + # same collectives on every rank (the capture pass and serving still overlap it on the aux stream). + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if return_parts: + return shared_out, routed_out + # In-place add into the shared output to avoid allocating a temporary. + return shared_out.add_(routed_out), None + + def _generic_boundary(self, x, x2, residual, norm_w, qkv_a1_next: bool = False): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce. ``qkv_a1_next``: the next layer is + sparse (TP2AGG_SP's SP_OVL may overlap its push with A1).""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + if norm_w is not core.w["final_norm"]: + fused = _ctxnormq_allreduce_norm( + core, + x, + residual, + norm_w, + None, + _prefill_allreduce(core, x.shape[0]), + qkv_a1_next=qkv_a1_next, + ) + if fused is not None: + return fused + return _native_allreduce_norm( + core, x, residual, norm_w, _prefill_allreduce(core, x.shape[0]) + ) + + def _layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm( + core, + hidden, + residual, + core.w[f"l{i}_post_norm"], + _prefill_allreduce(core, hidden.shape[0]), + ) + ) + else: + fused = _ctxnormq_allreduce_norm( + core, + hidden, + residual, + core.w[f"l{i}_post_norm"], + core.moe[i].fc31_input_scale if i in core.moe_layers else None, + _prefill_allreduce(core, hidden.shape[0]), + router_w=core.w[f"l{i}_router"] if i in core.moe_layers else None, + pre_moe=True, + ) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm( + core, + hidden, + residual, + core.w[f"l{i}_post_norm"], + _prefill_allreduce(core, hidden.shape[0]), + ) + ) + if i in core.moe_layers: + out, routed = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out = self._swiglu_mlp(hidden, f"l{i}_mlp", True, dense=True) + routed = None + normed, updated = self._generic_boundary( + out, routed, residual, _next_norm(core, i), qkv_a1_next=(i + 1) in core.sparse_layers + ) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + # TP2AGG_SP: the capture reads the full residual (gathered from an owned-only one). + updated = _sp_unshard(updated, normed) + capture(spec_metadata, i, updated, None) + return normed, updated + + def _attention(self, i: int, position_ids, hidden, step): + """Attention of layer ``i`` up to its o_proj (no reduction). Returns (o_proj output, pushed): a prefill step + arms no boundary push, so pushed is always False.""" + core = self.core + if i not in core.sparse_layers: + # Dense layers 0-2: QKV, the fused norm + RoPE (FP8 q/k/v for an FP8 cache, else in place in bf16), the + # MSA backend (it writes the layer's cache), o_proj. + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + if not core.fused_projection: + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target runs a breakable prefill CUDA graph only with the five-way projection " + "(fuse_qkv_index_projection)" + ) + return self._unfused_sparse_attention(i, position_ids, hidden, step) + in_bcg = is_in_breakable_cuda_graph() + deferred = None if in_bcg else _a1_defer(core, i, hidden) + if deferred is not None: + packed = deferred.placeholder + else: + packed, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + if in_bcg: + output = packed.new_empty( + (packed.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16 + ) + _prefill_attention_eager( + packed, position_ids, None, None, None, None, output, self._bcg_key, i + ) + return _linear(core, output, f"l{i}_o_w", f"l{i}_o_s", True) + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + deferred, + ) + if horizontal is None: + raise RuntimeError( + f"MiniMax-M3 sparse layer {i}: no horizontal producer takes this step's caches (an FP8 or NVFP4 HND " + "pool with an FP8 index-K cache)" + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output.""" + core = self.core + if _producer_covered(core): + # Not a miss: the producer packs the five-way projection's output, which this configuration lacks. + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): the MSA + backend (``m3_msa_attention``: the KV write unless a producer wrote the caches, the sparse layers' top-k block + selection, the FMHA over the context rows and any generation rows beside them).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + if is_in_breakable_cuda_graph(): + # The live row count is a host value of the step: read at every replay inside the eager boundary. + _prefill_attention_eager(None, None, q, k, v, idx_q, output, self._bcg_key, i) + else: + _COMBINE_MXFP8_STATE.pop("pending", None) + _COMBINE_MXFP8_STATE.pop("src", None) + _CTXNORMQ_MX.clear() + if ( + TP2WIDE_COMBINE_MXFP8 + and TP2WIDE_SPARSE_OUT + and CTXNORMQ + and core.main_kv_is_nvfp4 + and i in core.sparse_layers + and int(step["num_ctx_tokens"]) >= _COMBINE_MXFP8_MIN_ROWS + and output.shape[0] == int(step["num_tokens"]) + and not torch.cuda.is_current_stream_capturing() + and not torch.compiler.is_compiling() + ): + _COMBINE_MXFP8_STATE["pending"] = { + "output": output, + "context_rows": int(step["num_ctx_tokens"]), + } + try: + _msa_attention_live(core, i, q, k, v, idx_q, step["metadata"], output) + _combine_mxfp8_finish(output) + finally: + _COMBINE_MXFP8_STATE.pop("pending", None) + return output + + +# F3: the attention boundary's MXFP8 norm for the next pinned GENERIC MXFP8 linear (shared / dense gate_up). +_F3_MX = m3_prequant.PrequantSlot() +_F3_STATE: dict = {} + + +class F3Push: + """The o_proj's F3 target (decided before the attention from the row count and import / prebuild state only, so + every rank agrees): the launcher, the pinned tactic and the boundary's allreduce_in-shaped tensor (shape only).""" + + def __init__(self, launcher, tactic, rows: int, hidden: int, device): + self.launcher, self.tactic, self.rows = launcher, tactic, rows + self.out = torch.empty((rows, hidden), dtype=torch.bfloat16, device=device) + self.ref = None # (act, act_scale, w, s): F3_CHECK's stock re-run + + def accepts(self, n: int, k: int, rows: int) -> bool: + return (n, k) == (F3_N, F3_K) and rows == self.rows + + def run(self, act, act_scale, w, s): + self.launcher.run(self.tactic, act, act_scale, w, s, self.out) + if F3_CHECK: + self.ref = (act, act_scale, w, s) + + +def _f3_op(): + """``torch.ops.trtllm.minimax_m31_f3_ar_prepushed`` when the build carries it and F3 is on, else None. Decided + once per process.""" + if "op" not in _F3_STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_f3_ar_prepushed", None) if TP2_F3 else None + _F3_STATE["op"] = op + return _F3_STATE["op"] + + +def _f3_bucket(rows: int) -> int: + return max(64, 1 << (rows - 1).bit_length()) # mxfp8_cute_pinned_tactics._bucket + + +def _f3_init(core) -> None: + """Prebuild (before warmup and capture, every rank, real weights): load the op, build the launcher on + core.allreduce's own Lamport workspace, compile every pinned o_proj tactic of F3_MIN_ROWS..F3_MAX_ROWS and its + alpha. Leaves core._f3_launcher None (F3 never taken) with one reason line otherwise.""" + core._f3_launcher = None + why = None + ar = getattr(core, "allreduce", None) + ws = getattr(ar, "workspace", None) if ar is not None else None + if not TP2_F3: + why = "switched off (TRTLLM_M31_DISABLE_F3)" + elif core.mapping.tp_size != 2: + why = f"tp_size {core.mapping.tp_size} != 2" + elif ( + ar.strategy not in (AllReduceStrategy.AUTO, AllReduceStrategy.ONESHOT) + or getattr(ar, "mnnvl_allreduce", None) is not None + or getattr(ar, "symm_mem_allreduce", None) is not None + ): + why = f"all-reduce strategy {ar.strategy} is not the one-shot Lamport" + elif ws is None or ws.dtype != torch.int64 or not ws.is_cuda or ws.numel() < 7: + why = "no Lamport workspace" + elif _f3_op() is None: + why = "the minimax_m31_f3_ar_prepushed op is not built" + else: + from tensorrt_llm._torch.custom_ops import mxfp8_cute_pinned_tactics as _pin + + from .kernels import m31_f3_gemm as _g + + first = min(core.attn_layers) if hasattr(core, "attn_layers") else 0 + n, k = core.w[f"l{first}_o_w"].shape + if not _g.fi_base_ok(): + why = "FlashInfer dense_blockscaled_gemm_sm100.py is not the fork base 99892be6" + elif (n, k) != (F3_N, F3_K): + why = f"o_proj [{n}, {k}] is not [{F3_N}, {F3_K}]" + else: + launcher = _g.F3GemmLauncher( + ws, core.mapping.tp_rank, F3_N, F3_K, torch.ops.trtllm.minimax_m31_f3_read_i64 + ) + tactics = sorted( + { + _pin.TACTICS.get((F3_N, F3_K, _f3_bucket(r))) + for r in range(F3_MIN_ROWS, F3_MAX_ROWS + 1) + } + - {None} + ) + if not tactics or any(t[4] != 1 for t in tactics): + why = f"pinned tactics {tactics} are not all non-split-K" + else: + for t in tactics: + launcher.compile(t) + launcher._alpha[ws.device] = torch.ones(1, dtype=torch.float32, device=ws.device) + core._f3_launcher = launcher + _F3_STATE["bad"] = torch.zeros(1, dtype=torch.int64, device=ws.device) + _F3_STATE["checked"] = torch.zeros(1, dtype=torch.int64, device=ws.device) + if F3_CHECK: + import atexit + import signal + + atexit.register(_f3_report, core, True) + # Serving harnesses end the server with a signal, which skips atexit; the final + # report also runs on SIGTERM / SIGINT, then the previous handler (or the default action). + for _sig in (signal.SIGTERM, signal.SIGINT): + try: + _prev = signal.getsignal(_sig) + + def _f3_on_signal(signum, frame, _prev=_prev): + try: + _f3_report(core, True) + except Exception: # noqa: BLE001 + pass + if callable(_prev): + _prev(signum, frame) + elif _prev == signal.SIG_IGN: + return + else: + signal.signal(signum, signal.SIG_DFL) + os.kill(os.getpid(), signum) + + signal.signal(_sig, _f3_on_signal) + except ValueError: # not the main thread: atexit only + logger.info( + f"[M3.1] TP2_F3 check: no {_sig.name} hook (not the main thread)" + ) + logger.info( + f"[M3.1] TP2_F3 ready rank={core.mapping.tp_rank}: rows {F3_MIN_ROWS}..{F3_MAX_ROWS}, tactics " + f"{tactics}, check={F3_CHECK}" + + ( + ", device assert armed (every boundary, replays included)" + if F3_CHECK + else "" + ) + ) + if why is not None: + logger.info(f"[M3.1] TP2_F3 not taken rank={core.mapping.tp_rank}: {why}") + + +def _f3_check(core, f3, residual, norm_w, normed, updated, q, sf) -> None: + """F3_CHECK: the stock GEMM + AR + quantize on the same operands (one more collective round, in the graph), byte + mismatches added on the device.""" + if f3.ref is None: + _F3_STATE["bad"].add_(1) + torch._assert_async(_F3_STATE["bad"] == 0, "TP2_F3 check: missing stock reference") + return + ref = m3_mxfp8_pinned_gemm(*f3.ref, torch.bfloat16, MXFP8_DECODE_GEMM_TACTIC) + rn, rr = _native_allreduce_norm(core, ref, residual, norm_w) + rq, rsf = m3_mxfp8_pinned_quantize(rn, MXFP8_DECODE_QUANT_TACTIC) + bad = (normed.view(torch.int16) != rn.view(torch.int16)).sum() + bad = bad + (updated.view(torch.int16) != rr.view(torch.int16)).sum() + bad = bad + (q.view(torch.uint8) != rq.view(torch.uint8)).sum() + bad = bad + (sf.view(torch.uint8).reshape(-1) != rsf.view(torch.uint8).reshape(-1)).sum() + _F3_STATE["bad"].add_(bad) + _F3_STATE["checked"].add_(1) + # A device-side assert on every checked boundary, replays included (capture-safe). A mismatch on + # any replay aborts the server, so a run that completes proves 0 bad without any end-of-run read. + torch._assert_async(bad == 0, "TP2_F3 check: F3 boundary bytes differ from the stock chain") + + +def _f3_report(core, final: bool = False) -> None: + """Log the F3 counters (host side: capture-time boundaries and slot takes; device side: F3_CHECK).""" + if "bad" not in _F3_STATE or torch.cuda.is_current_stream_capturing(): + return + if final and F3_CHECK: + # A signal runs on the main thread; graph replay uses the executor stream. + # Wait for its writes before reading the diagnostic counters on this thread. + torch.cuda.synchronize(_F3_STATE["bad"].device) + msg = ( + f"[M3.1] TP2_F3 {'final ' if final else ''}report rank={core.mapping.tp_rank}: captured boundaries " + f"{_F3_STATE.get('taken', 0)}, slot taken {_F3_STATE.get('consumed', 0)}" + ) + if F3_CHECK: + msg += f", check boundaries {int(_F3_STATE['checked'].item())} mismatched bytes {int(_F3_STATE['bad'].item())}" + logger.info(msg) + + +class DecodeTarget: + """A pure generation step (no context rows), of any width. + + Up to ``DECODE_MAX_TOKENS`` tokens with 1-D input ids, per layer: attention (o_proj all-reduce deferred, its rows + pushed into the boundary all-reduce up to 16 tokens) -> attention boundary (all-reduce + residual + post-attention + RMSNorm) -> MoE (the 1..8 / 9..13-row engines, the first pushing its sum into the boundary all-reduce; the + unfinalized MoE with the finalize in the all-reduce from 14 rows) or dense MLP -> MoE boundary (all-reduce + + shared/routed add + residual + the next layer's RMSNorm, emitting the next QKV's MXFP8 activation). A layer whose + output Eagle3 captures runs the generic layer, as does the first layer when the fused embedding does not take the + step. A wider step (or one carrying embeddings) runs every layer in its generic form, with this target's attention + (the decode selection and sparse decode where the producer wrote the caches). + """ + + def __init__(self, core: MiniMaxM3Tp2ModelingV2) -> None: + self.core = core + self._moe_push_descs: Dict[tuple, Any] = {} + # Set by a fused feed-forward boundary that wrote the layer's Eagle3 capture (see _layer's ``capture``). + self._capture_written = False + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it; read once here rather than per forward. + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + States the two context counts instead of reading them back: this target runs only when ``num_contexts`` is + zero, so reading them would ask the metadata to confirm what the routing already decided. + + Graph classes: counts, the generation span, the identity-selection flag and the worst-case score extent are + host values fixed per capture (prepare() stages them before the forward); the block table, lengths, cache + slots, identity table and valid-block counts are the metadata's persistent buffers, held by reference and + refreshed in place every step; ``msa_idx_k_cache`` / ``msa_proxy_max_score_view`` are the metadata's getters + of the per-layer index-K cache and of the score scratch view. ``metadata`` is the prepared object itself: the + MSA backend's step (``m3_msa_attention``) takes it as its argument. + """ + return dict( + num_contexts=0, + num_ctx_tokens=0, + num_tokens=md.num_tokens, + num_seqs=md.num_seqs, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + msa_decode_span=md.msa_decode_span, + msa_block_table=md.msa_block_table, + msa_seq_lens_cuda=md.msa_seq_lens_cuda, + msa_short_ok=md._msa_short_ok, + msa_short_table=md.msa_short_table, + msa_proxy_max_score_view=md.msa_proxy_max_score_view, + msa_worst_case_max_k_tiles=md.msa_worst_case_max_k_tiles, + msa_n_valid_blocks=md.msa_n_valid_blocks, + max_seq_len=md.max_seq_len, + metadata=md, + ) + + def forward( + self, + attn_metadata, + input_ids, + position_ids, + inputs_embeds, + spec_metadata, + capture, + capture_target, + ): + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + # The decode kernels' rows (a per-capture constant): up to DECODE_STEP_MAX tokens of 1-D input ids. + kernels = ( + inputs_embeds is None and input_ids.ndim == 1 and is_decode_step(input_ids.numel()) + ) + if kernels: + num_tokens = int(input_ids.numel()) + plan = plan_for(num_tokens) + _record("decode", num_tokens, plan.name) + if num_tokens > TP2_ROWS: + # R1VR8: a step past the TP2-lineage 8 rows on the decode layer. + m3_report.work("R1VR8", f"{num_tokens}-row step on the decode layer ({plan.name})") + if plan.name == "wide": + taken("VR-BANDS wide decode step", num_tokens) + elif plan.name == "wide448": + taken("TP2WIDE_DECODE", num_tokens, f"rank {core.mapping.tp_rank}") + hidden, residual = self._embed(input_ids, num_tokens) + else: + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) CUDA graphs" + ) + plan = None + _record("decode", int(step["num_tokens"]), "generic") + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + is_capture = spec_metadata is not None and spec_metadata.is_layer_capture(i) + if ( + kernels + and residual is not None + and is_capture + and CAPTURE_LAYERS_FAST + and CAPTURE_FUSED_BOUNDARY + ): + # The capture layer on the fast path (its boundary pushes); the capture reads the MoE / dense + # boundary's reduced residual sum. With CAPTURE_NORM_IN_BOUNDARY the boundary all-reduce writes the + # capture norm itself. + capture_out = ( + capture_target(spec_metadata, i, num_tokens) + if capture_target is not None and CAPTURE_NORM_IN_BOUNDARY + else None + ) + self._capture_written = False + hidden, residual = self._layer( + i, plan, position_ids, hidden, residual, step, capture=capture_out + ) + taken("capture_layer_fast", num_tokens, f"layer {i}") + if capture_out is not None and self._capture_written: + taken("capture_norm_in_boundary", num_tokens, f"layer {i}") + capture(spec_metadata, i, residual, None, written=True) + else: + capture(spec_metadata, i, residual, None) + elif not kernels or residual is None or is_capture: + hidden, residual = self._generic_layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + else: + hidden, residual = self._layer(i, plan, position_ids, hidden, residual, step) + return hidden + + def _embed(self, input_ids: torch.Tensor, num_tokens: int): + """(hidden, residual) of the fused embedding + first RMSNorm; (embedding, None) when it does not run, so the + first layer runs its own input norm.""" + core = self.core + if TP2WIDE_DECODE and 64 < num_tokens <= DECODE_STEP_MAX: + # Preserve the generic first layer's embedding and FlashInfer norm arithmetic. + # These graph-owned tensors avoid preallocating an owner for every possible row count. + raw = m3_embedding(input_ids, core.w["embed"]) + normed = flashinfer_rmsnorm(raw, core.w["l0_in_norm"], core.rms_eps) + taken("TP2WIDE_INPUT", num_tokens, f"rank {core.mapping.tp_rank}") + return normed, raw + if not EMBED_FIRSTNORM.covers(num_tokens): + uncovered(EMBED_FIRSTNORM.path, f"more than {EMBED_FIRSTNORM_MAX_TOKENS} tokens") + return m3_embedding(input_ids, core.w["embed"]), None + owner = core.embed_owners.get((num_tokens, input_ids.dtype)) + if owner is None or not input_ids.is_contiguous(): + missed( + EMBED_FIRSTNORM.path, + num_tokens, + f"no embedding owner for {input_ids.dtype} ids (built for int32 / int64 after the load)" + if owner is None + else "the input ids are not contiguous", + ) + return m3_embedding(input_ids, core.w["embed"]), None + _record("m3_embedding_firstnorm", num_tokens, input_ids.dtype) + taken(EMBED_FIRSTNORM.path, num_tokens, str(input_ids.dtype)) + return m3_embedding_firstnorm( + input_ids, owner.table, owner.weight, owner.eps, owner.raw, owner.output, owner.use_pdl + ) + + def _bucket_flags(self, num_tokens: int): + """(bucket workspace, its 8-CTA call counters) of the boundary all-reduce for ``num_tokens``, or None.""" + workspace = m3_mc_allreduce.get_workspace(self.core.mapping, self.core.hidden) + workspace = None if workspace is None else m3_mc_allreduce._bucket(workspace, num_tokens) + flags = None if workspace is None else workspace.flags_for(8) + return None if flags is None else (workspace, flags) + + @staticmethod + def _no_bucket(path: str, num_tokens: int) -> None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, "the multicast all-reduce workspace") + else: + missed(path, num_tokens, "no multicast all-reduce bucket holds the step") + + def _attn_push_target(self, num_tokens: int): + """The o_proj's push tuple into the attention boundary's all-reduce, or None (after a miss). + + None comes only from the token count or from workspace state created collectively in the same eager call on + every rank, so all ranks agree. (A lone non-pushing rank would still be safe: its all-reduce pushes the same + bf16 rows into the same set A.)""" + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(ATTN_PUSH.path, num_tokens) + return None + workspace, flags = found + return ( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + AR_PUSH_MODE, + ) + + def _moe_push_targets(self, num_tokens: int): + """(engine push descriptor, shared down-projection push tuple) for the MoE boundary's exchange sets, or None + (after a miss). + + This decision must match across ranks: prepushed 3 reads set B of every rank, which a non-pushing rank never + fills. None comes only from the token count or from state created in the same eager call on every rank + (workspace, flags, the descriptor cache), and the engine it is armed for is agreed at build time + (``_prebuild_engines``), so it does.""" + hidden = self.core.hidden + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(MOE_PUSH.path, num_tokens) + return None + workspace, flags = found + key = (workspace.mc.data_ptr(), flags.data_ptr()) + desc = self._moe_push_descs.get(key) + if desc is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(MOE_PUSH.path, num_tokens, "the MoE engine's push descriptor") + return None + desc = self._moe_push_descs[key] = ar_push_desc( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + flags.device, + ) + set_b = workspace.mc.data_ptr() + 2 * 4 * workspace.max_tokens * workspace.world_size * ( + hidden // 2 + ) + return desc, ( + set_b, + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + 2, + ) + + def _boundary(self, use_mc: bool, x, residual, norm_w, x2=None, quant=False, capture=None): + """All-reduce + (second addend) + residual + RMSNorm: the multicast kernel, else the built-in all-reduce. + ``capture``: the Eagle3 capture the multicast kernel also writes (``_capture_written`` records it).""" + core = self.core + if use_mc: + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w, quant=quant, capture=capture) + if fused is not None: + if capture is not None: + self._capture_written = True + rows = x.shape[0] + _mc_taken( + core.mapping, + rows, + ("x + x2" if x2 is not None else "x") + + (", MXFP8 copy" if quant else "") + + (", cluster" if rows <= MC_AR_CLUSTER_MAX_ROWS else ""), + ) + return fused + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + missed( + MC_ALLREDUCE.path, + x.shape[0], + m3_mc_allreduce.decline_reason(x, residual, workspace, x2=x2), + ) + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _prepushed(self, i: int, x, residual, norm_w, quant, prepushed, x2=None, capture=None): + fused = _mc_allreduce_norm( + self.core, x, x2, residual, norm_w, quant=quant, prepushed=prepushed, capture=capture + ) + if fused is None: + raise RuntimeError(f"layer {i}: rows were pushed into an all-reduce that did not run") + if capture is not None: + self._capture_written = True + rows = x.shape[0] + _mc_taken( + self.core.mapping, + rows, + f"prepushed {prepushed}" + (", cluster" if rows <= MC_AR_CLUSTER_MAX_ROWS else ""), + ) + return fused + + def _f3_push(self, num_tokens: int, residual) -> Optional[F3Push]: + """F3's o_proj target for this decode-graph step, or None: from the row count, the residual's shape and + prebuild state only (every rank agrees).""" + core = self.core + launcher = getattr(core, "_f3_launcher", None) + if launcher is None or not F3_MIN_ROWS <= num_tokens <= F3_MAX_ROWS: + return None + if ( + residual is None + or residual.dim() != 2 + or residual.shape != (num_tokens, F3_N) + or residual.dtype != torch.bfloat16 + or not residual.is_contiguous() + ): + return None + from tensorrt_llm._torch.custom_ops import mxfp8_cute_pinned_tactics as _pin + + tactic = _pin.tactic_for(num_tokens, F3_N, F3_K) + if tactic is None or tactic[4] != 1: + return None + return F3Push(launcher, tactic, num_tokens, F3_N, residual.device) + + def _f3_boundary(self, f3: F3Push, residual, norm_w): + """The attention boundary after an F3 o_proj: the one-shot AR without its push loop + residual + RMSNorm, and + the norm's MXFP8 (published in _F3_MX for the next pinned GENERIC MXFP8 linear).""" + core = self.core + rows, hidden = f3.rows, F3_N + normed = torch.empty_like(residual) + updated = torch.empty_like(residual) + q = torch.empty((rows, hidden), dtype=torch.float8_e4m3fn, device=residual.device) + sf = torch.empty( + ((rows + 127) // 128 * 128) * (hidden // 32), dtype=torch.uint8, device=residual.device + ) + _f3_op()( + core.mapping.tp_rank, + core.allreduce.workspace, + f3.out, + residual, + updated, + normed, + norm_w, + core.rms_eps, + False, + q, + sf, + ) + if F3_CHECK: + _f3_check(core, f3, residual, norm_w, normed, updated, q, sf) + _F3_MX.publish(normed, q, sf) + _record("m31_f3_boundary", rows) + cnt = _F3_STATE["taken"] = _F3_STATE.get("taken", 0) + 1 + if cnt & (cnt - 1) == 0: + logger.info( + f"[M3.1] TP2_F3 taken n={cnt} rows={rows} rank={core.mapping.tp_rank} tactic={f3.tactic}" + ) + return normed, updated + + def _layer(self, i: int, plan: DecodePlan, position_ids, hidden, residual, step, capture=None): + """``capture``: (out, fc_norm weight, eps) for the feed-forward boundary to write the layer's Eagle3 capture + norm (CAPTURE_NORM_IN_BOUNDARY, for a capture layer on this path); a fused boundary that writes it sets + ``_capture_written``.""" + core = self.core + num_tokens = hidden.shape[0] + # Padded rows: the boundary all-reduce polls and re-arms rows t < M = residual.shape[0], CUDA-graph padding + # rows included. Every producer pushes all rows of the same tensor: the o_proj / shared down GEMV t < m = + # x.shape[0] (its row count comes from the host shape, not a live count), the MoE engine every row of the + # num_tokens it was built for (= num_tokens here). The all-reduce checks x.sizes() == residual.sizes(), so + # no padded row is left with a stale flag. + # The GEMV dispatch (and with it every push into a boundary all-reduce) is a decode-graph one. + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + if not decode_graph: + uncovered( + "decode_graph", + "an eager decode step: the decode-graph GEMV dispatch and the boundary pushes are off", + ) + push = plan.attn_mc_ar and decode_graph and ATTN_PUSH.covers(num_tokens) + armed = self._attn_push_target(num_tokens) if push else None + f3 = ( + self._f3_push(num_tokens, residual) + if armed is None and not plan.attn_mc_ar and decode_graph + else None + ) + if not decode_graph and i == 0: + _f3_report(core) + hidden, attn_pushed = self._attention( + i, position_ids, hidden, step, armed if f3 is None else f3 + ) + post_norm = core.w[f"l{i}_post_norm"] + if f3 is not None and attn_pushed: + hidden, residual = self._f3_boundary(f3, residual, post_norm) + elif armed is not None and attn_pushed: + taken(ATTN_PUSH.path, num_tokens) + hidden, residual = self._prepushed(i, hidden, residual, post_norm, False, AR_PUSH_MODE) + else: + if armed is not None: + missed( + ATTN_PUSH.path, + num_tokens, + "the o_proj did not push its rows into the armed boundary all-reduce", + ) + hidden, residual = self._boundary(plan.attn_mc_ar, hidden, residual, post_norm) + + norm_w = _next_norm(core, i) + if i not in core.moe_layers: + # A dense layer: its MLP's down projection pushes into the post-MLP boundary's all-reduce, as the o_proj + # does into the attention one (the same slot; the gate_up projection never pushes). + # R6: the next layer's QKV takes its MXFP8 activation from this boundary, as after a MoE layer. + quant = R6_DENSE_PREQUANT and plan.qkv_prequant and i + 1 < core.num_layers + if plan.qkv_prequant and i + 1 < core.num_layers: + _log_once( + logger.info, + "r6_dense_prequant", + "on" if quant else "off", + "[M3 dispatch] r6_dense_prequant " + + ( + "taken: the dense-MLP boundary hands the next QKV its MXFP8 activation" + if quant + else "off (VR default; TRTLLM_M31_ENABLE_R6=1 turns it on), BF16 hand-off" + ) + + " (logged once)", + ) + dense_push = plan.attn_mc_ar and decode_graph and DENSE_PUSH.covers(num_tokens) + armed = self._attn_push_target(num_tokens) if dense_push else None + out, dense_pushed = self._swiglu_mlp(hidden, f"l{i}_mlp", True, armed, dense=True) + if armed is not None and dense_pushed: + taken(DENSE_PUSH.path, num_tokens) + return self._prepushed( + i, out, residual, norm_w, quant, AR_PUSH_MODE, capture=capture + ) + if armed is not None: + missed( + DENSE_PUSH.path, + num_tokens, + "the dense MLP's down projection did not push its rows into the armed boundary all-reduce", + ) + return self._boundary( + plan.attn_mc_ar, out, residual, norm_w, quant=quant, capture=capture + ) + + quant = plan.qkv_prequant and i + 1 < core.num_layers + path = moe_path(num_tokens) + moe_push = None + if path == MOE_ENGINE.path: + # The engine runs inside the MoE; looked up (or found missing) here first so a push is armed only for an + # engine that will take it. + engine = core.engine(i, num_tokens, shared=True) + if ( + engine is not None + and plan.moe_mc_ar + and decode_graph + and MOE_PUSH.covers(num_tokens) + ): + moe_push = self._moe_push_targets(num_tokens) + elif path == MOE16_ENGINE.path and decode_graph: + # Router, routed and shared experts in one kernel; its bf16 sum goes to the boundary all-reduce. + engine = core.moe16_engine(i, num_tokens) + if engine is not None: + _record("m3_moe16_engine", num_tokens) + out = m3_moe16_engine(hidden.contiguous(), engine) + taken(MOE16_ENGINE.path, num_tokens) + return self._boundary( + plan.moe_mc_ar, out, residual, norm_w, quant=quant, capture=capture + ) + if path in (MOE16_ENGINE.path, MC_FINALIZE.path) and plan.moe_mc_ar: + if self._bucket_flags(num_tokens) is None: + self._no_bucket(MC_FINALIZE.path, num_tokens) + else: + (shared, fc2, scales, perm), _, _, _ = self._moe( + i, hidden, return_parts=False, do_finalize=False + ) + _record("m3_mc_finalize_allreduce_norm", num_tokens, quant) + fused = m3_mc_allreduce.m3_mc_finalize_allreduce_norm( + fc2, + perm.view(num_tokens, -1).to(torch.int32), + scales.reshape(num_tokens, -1), + shared, + residual, + norm_w, + core.rms_eps, + m3_mc_allreduce.get_workspace(core.mapping, core.hidden), + use_pdl=get_env_enable_pdl(), + quant=quant, + ) + if fused is None: + raise RuntimeError( + "the MC finalize all-reduce rejected the unfinalized MoE outputs" + ) + taken(MC_FINALIZE.path, num_tokens) + return fused + if ( + TP2W_S2_TAIL + and not plan.moe_mc_ar + and moe_push is None + and capture is None + and not quant + and i in core.moe_layers + ): + s2 = self._s2_tail(i, hidden, residual, norm_w) + if s2 is not None: + return s2 + # The parts only where the boundary folds the add (the multicast all-reduce) or a push is armed; the built-in + # all-reduce boundary ("wide" plan) adds them first anyway, so there the MoE returns the sum + # (TP2W_MOE_FINALIZE). + shared, routed, engine_pushed, shared_pushed = self._moe( + i, hidden, return_parts=plan.moe_mc_ar or moe_push is not None, push=moe_push + ) + if moe_push is not None: + if routed is None: + # The engine folded the shared expert in and pushed routed + shared into set A. + if engine_pushed: + taken(MOE_PUSH.path, num_tokens, "engine, routed + shared") + return self._prepushed(i, shared, residual, norm_w, quant, 2, capture=capture) + else: + if engine_pushed and shared_pushed: + taken(MOE_PUSH.path, num_tokens, "routed, shared") + return self._prepushed( + i, shared, residual, norm_w, quant, 3, x2=routed, capture=capture + ) + if engine_pushed or shared_pushed: + raise RuntimeError("only one MoE part pushed into the boundary all-reduce") + missed( + MOE_PUSH.path, + num_tokens, + "the MoE did not push its output into the armed boundary all-reduce", + ) + return self._boundary( + plan.moe_mc_ar, shared, residual, norm_w, x2=routed, quant=quant, capture=capture + ) + + def _s2_tail(self, i: int, h: torch.Tensor, residual, norm_w): + """TP2W_S2_TAIL on the decode layer's built-in all-reduce boundary ("wide" plans): MoE layer ``i`` with its + boundary as one fused kernel, or None when the step is not the stage's (rows outside + S2_TAIL_MIN_ROWS..S2_TAIL_MAX_ROWS, not the Triton-finalize MoE, the final norm). A declined fused boundary + finalizes and reduces as today.""" + core = self.core + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + h.dtype != torch.bfloat16 + or not S2_TAIL_MIN_ROWS <= rows <= S2_TAIL_MAX_ROWS + or not _triton_finalize_rows(core, h) + or norm_w is core.w["final_norm"] + or is_torch_compiling() + or _s2_op() is None + ): + return None + (shared, fc2, scales, perm), _, _, _ = self._moe( + i, h, return_parts=False, do_finalize=False + ) + fused = _s2_boundary( + core, rows, (fc2, scales, perm), shared, residual, norm_w, core.allreduce + ) + if fused is not None: + return fused + return _native_allreduce_norm( + core, _triton_finalize(rows, (fc2, scales, perm), shared), residual, norm_w + ) + + def _swiglu_mlp( + self, + h: torch.Tensor, + prefix: str, + low_m_gemm: bool, + push: Optional[tuple] = None, + dense: bool = False, + ): + """A gated MXFP8 MLP (the dense MLP with ``dense``, the shared expert): gate_up, SwiGLU-OAI, down, no + reduction. Returns (output, whether the down projection's GEMV pushed into ``push``).""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s", low_m_gemm) + n, k = core.w[f"{prefix}_dn_w"].shape + rows = gu.shape[0] + if not dense and rows in SWQ_ROWS.get((n, k), ()) and _mx_mode(core.mxfp8).decode_graph: + # The MXFP8 activation is published keyed by ``gu`` and taken by the down projection's prequant branch. + act, act_scale = m3_swq(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + m3_prequant.SLOT.publish(gu, act, act_scale) + if rows not in _SWQ_LOGGED: + _SWQ_LOGGED.add(rows) + logger.info(f"[M3 dispatch] SWQ: SwiGLU + MXFP8 producer rows={rows} [{n}, {k}]") + m3_report.work("SWQ", f"{rows} rows [{n}, {k}]") + return _linear(core, gu, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm, push) + if dense: + act = _dense_swiglu(core, gu) + else: + act = _native_swiglu(core, gu, False) + if act is None: + act = m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + return _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm, push) + + def _routed_chunks(self, i: int, x, router_logits, topk=None): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: the MoE + scheduler's even split, even chunks on the MoE chunking stream. ``topk`` (S1B_FOLD) splits with the rows.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + if num_chunks <= 1: + return _routed_experts(core, i, x, None, router_logits, True, topk) + sizes = _split_chunk(rows, num_chunks) + xs = x.split(sizes) + if topk is None: + logits, tks = router_logits.split(sizes), [None] * num_chunks + else: + logits, tks = [None] * num_chunks, list(zip(topk[0].split(sizes), topk[1].split(sizes))) + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True, tks[c])) + else: + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True, tks[c])) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor, do_finalize: bool): + """Router + routed experts: the router logits of a generation step (``_decode_router_logits``) and the experts + quantizing their bf16 input.""" + core = self.core + rows = h.shape[0] if h.dim() == 2 else 0 + fp4 = [] if X1 and 9 <= rows <= min(48, core.moe_max_num_tokens) else None + kind, route = _decode_route(core, i, h, fp4) + router_logits, topk = (route, None) if kind == "logits" else (None, route) + if fp4: + x, x_sf = fp4 + _record("m31_x1_router_fp4", rows) + taken("X1 router_fp4", rows, "native router payload consumed by routed experts") + return _routed_experts(core, i, x, x_sf, router_logits, do_finalize) + if S1B_FOLD_CHECK and topk is not None: + ref_logits = m3_fp32_linear(h, core.w[f"l{i}_router"]) + if not do_finalize: + rows = h.shape[0] + got = _routed_experts(core, i, h, None, None, False, topk) + ref = _routed_experts(core, i, h, None, ref_logits, False) + _S1bCheck.record( + _triton_finalize(rows, got, None), + _triton_finalize(rows, ref, None), + _s1b_ident(core, i, h, False), + ) + return ref + got = self._routed_chunks(i, h, None, topk) + ref = self._routed_chunks(i, h, ref_logits) + _S1bCheck.record(got, ref, _s1b_ident(core, i, h, True)) + return ref + if not do_finalize: + return _routed_experts(core, i, h, None, router_logits, False, topk) + return self._routed_chunks(i, h, router_logits, topk) + + def _moe( + self, i: int, h: torch.Tensor, *, return_parts: bool, do_finalize: bool = True, push=None + ): + """The MoE block of layer ``i`` without its reduction. + + ``push`` = (the engine's ``arp`` descriptor, the shared down projection's ``ar`` tuple) the decode layer armed + for the MoE boundary's exchange sets, or None. Returns ``(out, routed, engine_pushed, shared_pushed)``: with + ``return_parts`` the shared and routed parts (routed None when the engine folded the shared expert in), else + their sum in ``out``; with ``do_finalize=False`` ``out`` is ``[shared, fc2 out, expert scales, permuted + index]``. + """ + core = self.core + engine_arp, shared_ar = push if push is not None else (None, None) + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + do_finalize + and h.dim() == 2 + and h.dtype == torch.bfloat16 + and 1 <= rows <= MOE_ENGINE_MAX_TOKENS + and rows >= MOE_FOLD_MIN_TOKENS + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + _record("m3_moe_engine", rows, engine_arp is not None) + folded = m3_moe_engine(h.contiguous(), engine, None, engine_arp) + taken(MOE_ENGINE.path, rows) + return folded, None, engine_arp is not None, False + if do_finalize and not return_parts and push is None and _triton_finalize_rows(core, h): + routed_parts, (shared_out, _) = maybe_execute_in_parallel( + lambda: self._routed(i, h, False), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False), + core.moe_events[i][0], + core.moe_events[i][1], + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + return _triton_finalize(rows, routed_parts, shared_out), None, False, False + routed_out, (shared_out, shared_pushed) = maybe_execute_in_parallel( + lambda: self._routed(i, h, do_finalize), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False, shared_ar), + core.moe_events[i][0], + core.moe_events[i][1], + # The warmup tuning pass profiles the shared expert's GEMMs: keep it on the main stream, between the same + # collectives on every rank (the capture pass and serving still overlap it on the aux stream). + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if not do_finalize: + return [shared_out, *routed_out], None, False, shared_pushed + if return_parts: + return shared_out, routed_out, False, shared_pushed + # In-place add into the shared output to avoid allocating a temporary. + return shared_out.add_(routed_out), None, False, shared_pushed + + def _generic_boundary(self, x, x2, residual, norm_w): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce.""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _generic_layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + else: + hidden, residual = _native_allreduce_norm( + core, hidden, residual, core.w[f"l{i}_post_norm"] + ) + if i in core.moe_layers: + out, routed, _, _ = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out, _ = self._swiglu_mlp(hidden, f"l{i}_mlp", True, dense=True) + routed = None + normed, updated = self._generic_boundary(out, routed, residual, _next_norm(core, i)) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + capture(spec_metadata, i, updated, None) + return normed, updated + + def _attention(self, i: int, position_ids, hidden, step, push: Optional[tuple] = None): + """Attention of layer ``i`` up to its o_proj (no reduction). ``push`` is the attention boundary's exchange the + o_proj may write into. Returns (o_proj output, pushed).""" + core = self.core + if i not in core.sparse_layers: + # Dense layers 0-2: QKV, the fused norm + RoPE (FP8 q/k/v: an FP8 cache, or the FP8 pool the dense layers + # sit in within a hybrid NVFP4 cache on a decode step), the attention, o_proj. A covered (query rows, + # requests) shape attends on the full-context PrimTS decode (``FULLCTX_DECODE``), the norm/RoPE kernel then + # storing the new K / V rows itself (``DENSE_KV_INSERT_FUSED``); every other step the MSA backend. + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + out_fp8 = core.main_kv_is_fp8 or core.main_kv_is_nvfp4 + plan = self._fullctx_plan(i, qkv.device, step) + kv_dest = None + if ( + plan is not None + and out_fp8 + and DENSE_KV_INSERT_FUSED + and DENSE_QKV_FP8_VIA_BF16 + and qkv.dtype == torch.bfloat16 + and core.head_dim == 128 + ): + kv_dest = ( + step["kv_cache_manager"].get_buffers(i, kv_layout="HND"), + step["msa_out_cache_loc"], + plan[1], + torch.empty( + (qkv.shape[0], core.heads, core.head_dim), + dtype=torch.float8_e4m3fn, + device=qkv.device, + ), + ) + fused = _qk_norm_rope(core, i, qkv, position_ids, out_fp8, kv_dest) + if kv_dest is not None: + taken(DENSE_KV_INSERT_PATH, kv_dest[2]) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + if kv_dest is not None: + q = kv_dest[3].view(q.shape[0], -1) + o = None + if plan is not None: + o = self._fullctx_decode(i, q, k, v, step, plan, kv_written=kv_dest is not None) + if o is None: + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + if not core.fused_projection: + return self._unfused_sparse_attention(i, position_ids, hidden, step, push) + # QKV_PRODUCER_FOLD: the QKV GEMV may be deferred to the producer call site. + packed, _ = _linear( + core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True, defer=QKV_PRODUCER_FOLD + ) + deferred = None + if isinstance(packed, _DeferredGemv): + deferred, packed = packed, packed.placeholder + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + deferred, + ) + if horizontal is None: + raise RuntimeError( + f"MiniMax-M3 sparse layer {i}: no horizontal producer takes this step's caches (an FP8 or NVFP4 HND " + "pool with an FP8 index-K cache)" + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + + def _fullctx_plan(self, i: int, device: torch.device, step): + """(the PrimTS split-KV decode plan, the step's query tokens) where dense layer ``i``'s decode attention runs on + kernels/m3_fullctx_decode: a covered (query rows per request, requests) shape over the layer's FP8 pool. None + otherwise (the MSA backend runs).""" + path = FULLCTX_DECODE.path + if is_torch_compiling(): + uncovered(path, "not a pure generation step on the MSA backend") + return None + span = step["msa_decode_span"] + if span is None or span.row_first != 0: + uncovered(path, "no decode span starting at row 0") + return None + nreq = int(step["num_seqs"]) + qlen = int(span.query_len) + if not FULLCTX_DECODE.covers((qlen, nreq)): + uncovered(path, f"{qlen} query rows x {nreq} requests is not a covered shape") + return None + if int(step["max_seq_len"]) > FULLCTX_DECODE_MAX_KV_LEN: + uncovered(path, "the cache length bound is beyond the plans' K/V capacity") + return None + num_tokens = nreq * qlen + pool, _ = step["kv_cache_manager"].get_kv_subpage_pool(i, "HND") + if pool.dtype != torch.float8_e4m3fn: + missed(path, num_tokens, "the dense layer's KV pool is not FP8") + return None + own_reducer = FULLCTX_DECODE_OWN_REDUCER + if own_reducer and not _fullctx_decode_kernel.has_own_reducer(): + missed(path, num_tokens, "the BF16 split-KV merge op is not built") + return None + # Compiled before warmup (``prebuild_kernels``); the forward never compiles one. + plan = _fullctx_decode_kernel.plan_for(device, qlen, nreq, own_reducer) + if plan is None: + missed(path, num_tokens, "the PrimTS decode plan is not prebuilt") + return None + return plan, num_tokens + + def _fullctx_decode( + self, i: int, q, k, v, step, plan_tokens, *, kv_written: bool + ) -> torch.Tensor: + """Dense layer ``i``'s decode attention on the PrimTS plan of ``_fullctx_plan``: the step's new K / V go into + the paged cache as the MSA backend writes them (``write_msa_phase_kv``; already there when ``kv_written``), + then every query row attends its request's full causal context. The merge also emits the o_proj's MXFP8 + input (``FULLCTX_DECODE_MXFP8``) when the plan supports it.""" + core = self.core + backend = core.attn[i] + plan, num_tokens = plan_tokens + nreq = int(step["num_seqs"]) + if not kv_written: + write_msa_phase_kv( + backend, + k[:num_tokens], + v[:num_tokens], + step["metadata"], + AttentionInputType.mixed, + token_offset=0, + ) + pool, subpages = step["kv_cache_manager"].get_kv_subpage_pool(i, "HND") + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + k_pages, v_pages = _fullctx_decode_kernel.kv_page_views(pool, subpages) + mxfp8 = None + if ( + FULLCTX_DECODE_MXFP8 + and output.shape[0] == num_tokens + and _fullctx_decode_kernel.emits_mxfp8() + and MERGE_MXFP8_MIN_TOKENS <= num_tokens <= MERGE_MXFP8.TRITON_MAX_ROWS + ): + mxfp8 = MERGE_MXFP8.buffers(output.shape[1], output.device) + _record("m3_fullctx_decode", num_tokens, kv_written, mxfp8 is not None) + emitted = plan.run( + fp8_e4m3_saturate(q[:num_tokens].reshape(num_tokens, core.heads, core.head_dim)), + k_pages, + v_pages, + step["msa_block_table"][:nreq], + step["msa_seq_lens_cuda"][:nreq], + core.sm_scale(i), + output[:num_tokens], + mxfp8=mxfp8, + ) + if emitted: + m3_prequant.SLOT.publish(output, *MERGE_MXFP8.handoff(mxfp8, num_tokens)) + taken(FULLCTX_DECODE_MXFP8_PATH, num_tokens) + taken(FULLCTX_DECODE.path, num_tokens) + return output + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step, push): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output. ``push`` is the attention + boundary exchange the o_proj GEMV writes its rows into, as in the five-way path.""" + core = self.core + if _producer_covered(core): + # Not a miss: the producer packs the five-way projection's output, which this configuration lacks. + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): a sparse + layer whose caches the producer wrote runs this target's selection and sparse decode when the generation span + starts at row 0; everything else the MSA backend (``m3_msa_attention``: the KV write unless a producer wrote the + caches, the sparse layers' top-k block selection, the FMHA).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + live = int(step["num_tokens"]) + q_live, out_live = q[:live], output[:live] + k_live = k[:live] if k is not None else None + v_live = v[:live] if v is not None else None + idx_q_live = idx_q[:live] if idx_q is not None else None + span = step["msa_decode_span"] + sparse = i in core.sparse_layers + if sparse and k_live is None and span is not None and span.row_first == 0: + self._decode_sparse_core(i, q_live, idx_q_live, step, out_live) + return output + nvfp4 = core.main_kv_is_nvfp4 + _record("m3_msa_attention", sparse, k_live is not None) + m3_msa_attention( + core.attn[i], + q_live, + k_live, + v_live, + idx_q_live, + step["metadata"], + out_live, + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + return output + + @property + def _nvfp4_index_scorer(self) -> Nvfp4IndexDecodeScorer: + scorer = getattr(self, "_nvfp4_index_scorer_obj", None) + if scorer is None: + scorer = self._nvfp4_index_scorer_obj = Nvfp4IndexDecodeScorer(self.core.index_heads) + return scorer + + def _decode_sparse_core(self, i: int, q, idx_q, step, output) -> None: + """Top-16 block selection and the block-sparse decode of one pure generation step of a sparse layer whose caches + the producer already wrote. + + The selection is the identity when the metadata staged it (every request within 16 blocks). Otherwise the CuTe + DSL scorer writes the per-index-head block maxima, reduced to the KV head, and the long-row selector takes the + top 16 (the built-in selector beyond its shapes). A selected table at 33..128 rows on an NVFP4 layer runs LC2 + (TP2WIDE_LC2, ``SPARSE_DECODE_LC``), which also prefetches into L2, before its grid wait, the K/V pages of the + blocks the same layer selected one step back (``SPARSE_DECODE_LC_PREFETCH_SELF_KV``; hints only). + """ + core = self.core + backend = core.attn[i] + config = backend.m3_config + num_tokens = int(q.shape[0]) + rows = int(step["num_seqs"]) + qlen = int(step["msa_decode_span"].query_len) + block_table = step["msa_block_table"][:rows] + seq_lens = step["msa_seq_lens_cuda"][:rows] + num_kv_heads = int(backend.indexer.config.num_kv_heads) + identity = step["msa_short_ok"] and step["msa_short_table"] is not None + if identity: + topk = step["msa_short_table"][: num_kv_heads * num_tokens * MSA_REQUIRED_TOPK].view( + num_kv_heads, num_tokens, MSA_REQUIRED_TOPK + ) + else: + idx_q_view = idx_q.view(num_tokens, config.num_index_heads, config.sparse_index_dim) + idx_k_cache = step["msa_idx_k_cache"](i) + max_score = step["msa_proxy_max_score_view"]( + config.num_index_heads, step["msa_worst_case_max_k_tiles"], num_tokens + ) + if _nvfp4_index_k.is_nvfp4_index_cache(idx_k_cache): + # AGG_NVFP4_INDEXK decode reader: the Q8KV4 indexer scores the NVFP4 pages in the select layout, at + # the FP8 path's tile width (the same selector geometry). + tiles = int(max_score.shape[1]) + # NVFP4_STEP_CACHE: the reader's history scheduler depends on the step only; build it for the first + # sparse layer of the step (this step dict is per forward) and reuse it. + workspace = None + if NVFP4_STEP_CACHE: + key = ("nvfp4_hpr_ws", tiles) + workspace = step.get(key) + if workspace is None: + workspace = step[key] = Nvfp4IndexDecodeScorer.history_workspace( + seq_lens, tiles + ) + max_score = self._nvfp4_index_scorer.score( + idx_q_view, idx_k_cache, block_table[:, :tiles], seq_lens, qlen, workspace + ) + if rows not in _NVFP4_INDEXK_DECODE_LOGGED: + _NVFP4_INDEXK_DECODE_LOGGED.add(rows) + logger.info( + f"[M3.1 tp2] AGG_NVFP4_INDEXK decode reader taken: {rows} requests x {qlen}, Q8KV4 HPR" + ) + elif INDEX_SCORE_T2SC_PATH.covers(rows): + # T2SC: this target's copy of the TP2 / TP8 scorer at the request count's split and prestage. + split_k, early = index_score_t2sc_schedule(rows) + key = ( + idx_q_view.dtype, + int(config.num_index_heads), + int(qlen), + int(idx_q_view.shape[-1]), + int(split_k), + bool(early), + ) + if key not in _index_score_t2sc_kernel.IndexScoreRunner.kernel_cache: + missed( + INDEX_SCORE_T2SC_PATH.path, + num_tokens, + f"the T2SC index scorer {key} was not compiled before warmup", + ) + _record("m3_index_score_t2sc", key) + m3_index_score_t2sc( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens, + max_score.transpose(1, 2), + qlen, + split_k, + early, + ) + taken(INDEX_SCORE_T2SC_PATH.path, rows) + elif ( + INDEX_SCORE_INTREE + and rows >= INDEX_SCORE_INTREE_MIN_REQUESTS + and num_tokens > TP2_WIDE_MIN_ROWS - 1 + ): + # TP2 WIDE: the in-tree scorer (same scores), compiled before warmup (_prebuild_index_scorer_intree). + in_tree = _in_tree_scorer() + key = ( + idx_q_view.dtype, + int(config.num_index_heads), + int(qlen), + int(idx_q_view.shape[-1]), + int(in_tree.SPLIT_K), + ) + if key not in in_tree.kernel_cache: + missed( + INDEX_SCORE.path, + num_tokens, + f"the in-tree index scorer {key} was not compiled before warmup", + ) + if rows not in _INDEX_SCORE_INTREE_LOGGED: + _INDEX_SCORE_INTREE_LOGGED.add(rows) + logger.info( + f"[M3.1 tp2] TP2WIDE_INTREE_SCORER taken: {rows} requests, " + "the in-tree CuTe DSL scorer (split 256)" + ) + _record("cute_dsl_minimax_m3_index_decode_score", rows, qlen) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens, + max_score.transpose(1, 2), + qlen, + ) + else: + early = rows <= INDEX_SCORE_EARLY_MAX_REQUESTS + key = ( + idx_q_view.dtype, + int(config.num_index_heads), + int(qlen), + int(idx_q_view.shape[-1]), + int(INDEX_SCORE_SPLIT_K), + bool(early), + ) + if key not in _index_score_kernel.IndexScoreRunner.kernel_cache: + # Outside require mode the op below compiles it (the only scorer this step has). + missed( + INDEX_SCORE.path, + num_tokens, + f"the index scorer {key} was not compiled before warmup", + ) + _record("m3_index_score", key) + m3_index_score( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens, + max_score.transpose(1, 2), + qlen, + INDEX_SCORE_SPLIT_K, + early, + ) + taken(INDEX_SCORE.path, rows) + n_valid_blocks = step["msa_n_valid_blocks"][:num_tokens] + group = config.num_index_heads // config.num_kv_heads + max_score_kv = ( + max_score.view( + config.num_kv_heads, group, max_score.shape[1], max_score.shape[2] + ).amax(dim=1) + if group > 1 + else max_score + ) + early_trigger = num_tokens <= SELECT_EARLY_MAX_ROWS + trigger_phase = ( + SELECT_LONG_TRIGGER_PHASE if num_tokens <= SELECT_LONG_TRIGGER_MAX_ROWS else 0 + ) + dsl = False + if LONG_SELECT_DSL.keys and 128 < max_score_kv.shape[1]: + # Compiled before warmup (prebuild_kernels); a missing kernel falls back to the 1024-thread one. + key = _select_long_dsl_kernel.kernel_key( + get_env_enable_pdl(), early_trigger, trigger_phase + ) + dsl = key in _select_long_dsl_kernel.kernel_cache + if not dsl: + missed( + LONG_SELECT_DSL.path, + num_tokens, + f"the 384-thread select {key} was not compiled before warmup", + ) + table = long_select_blocks( + max_score_kv, + n_valid_blocks=n_valid_blocks, + init_blocks=config.init_blocks, + local_blocks=config.local_blocks, + use_pdl=get_env_enable_pdl(), + early_trigger=early_trigger, + trigger_phase=trigger_phase, + fast_tail=SELECT_LONG_SL3, + dsl=dsl, + ) + if table is not None: + taken((LONG_SELECT_DSL if dsl else LONG_SELECT).path, num_tokens) + else: + uncovered( + LONG_SELECT.path, + f"{max_score_kv.shape[1]} KV blocks, outside {LONG_SELECT_MIN_BLOCKS}..{LONG_SELECT_MAX_BLOCKS}: " + "the built-in selector", + ) + table = m3_select_blocks( + max_score_kv, + MSA_REQUIRED_TOPK, + n_valid_blocks, + config.init_blocks, + config.local_blocks, + True, + ) + topk = table.permute(1, 0, 2) + manager = step["kv_cache_manager"] + k_paged, v_paged = msa_paged_kv(manager, i) + nvfp4 = {} + if manager.is_nvfp4_layer(i): + k_scale, v_scale = _aligned_nvfp4_dequant_scales(backend, core.kv_scales[i]) + scales = manager.get_block_scale_buffers(i, "HND") + nvfp4 = dict( + k_block_scale=scales[:, 0], + v_block_scale=scales[:, 1], + k_global_scale=k_scale, + v_global_scale=v_scale, + ) + _record("m3_sparse_decode", num_tokens, qlen, identity) + # The long-context decode (if it runs: TP2WIDE_LC2) takes the layer's N1 id buffer; any other route (the Triton + # producer + merge, every step of at most 32 rows) leaves the slot unused, and it is cleared after the call + # either way. + n1_ids = core.n1_ids.get(i) + if n1_ids is not None and nvfp4 and num_tokens <= N1_MAX_ROWS: + m3_attn_lc.N1_IDS.ids = n1_ids + try: + m3_sparse_decode( + q.view(num_tokens, core.heads, core.head_dim), + k_paged, + v_paged, + topk, + block_table, + seq_lens, + sm_scale=core.sm_scale(i), + output=output.view(num_tokens, core.heads, core.head_dim), + decode_query_len=qlen, + identity_blocks=identity, + **nvfp4, + ) + finally: + m3_attn_lc.N1_IDS.ids = None + + +# -------------------------------------------------------------------------------------------------------------------- +# Speculative decoding: one-model Eagle3 at checkpoint precision +# -------------------------------------------------------------------------------------------------------------------- + + +class M3SpecStepFold: + """``SpecStepFold`` of this target. + + Between two decode graphs of a one-model Eagle3 step, the spec sampler stores the step's outputs into its per-slot + store and the engine gathers them back into its flat input buffers for the next step. This object does both in one + launch each (``m3_spec_tail.slot_scatter`` / ``step_gather``): integer copies only, the same values the engine's and + the sampler's own copies write. + + With ``in_forward`` (``STEP_SPEC_FOLD_IN_FORWARD``) the forward of a decode-only step also does both at its tail + (``store_in_forward``, one ``m3_fold_tail`` launch inside the step graph): it stores the step's outputs for the rows + ``begin_step`` armed and stages the next step's inputs for those rows in that order. The arming is a device-side row + count and slot list staged before the replay, so a captured graph whose step was not armed writes nothing. The + sampler then skips its store for exactly the armed rows (``sampler_rows_stored``), and the next step skips its + gathers when its rows are those rows (``begin_step``). + + One instance per target model; the target passes it to ``RuntimeHooks.spec_step_fold``. + """ + + def __init__( + self, in_forward: bool = False, store_copy_wait: bool = False, cpp_tail: bool = False + ) -> None: + self.store: Optional[object] = None + self.step_inputs: Optional[ + tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor] + ] = None + self.in_forward = in_forward + # The forward's tail as this target's C++ op instead of the Triton kernel (m3_fold_tail.fold_tail(cpp=)). + self.cpp_tail = in_forward and cpp_tail + # Device arming read by the forward's tail: ctrl[0] = armed rows (0 = write nothing), slots[:rows]. + self._ctrl: Optional[torch.Tensor] = None + self._slots: Optional[torch.Tensor] = None + # The rows (sequence slots) the next forward stores and stages for, as begin_step armed them. + self._armed: Optional[tuple[int, ...]] = None + # The rows the previous forward staged the next step's inputs for (confirmed by sampler_rows_stored). + self._staged_for: Optional[tuple[int, ...]] = None + # Output buffers the tail ran on: the captured graphs' static outputs (by address) and the last eager step's. + self._graph_outputs: set[int] = set() + self._eager_outputs: Optional[torch.Tensor] = None + self._logged = False + # store_copy_wait: the sampler records this (external) event on its D2H side stream after copying the store + # to the host, and the forward's tail waits on it before it rewrites the store; the sampler can then copy + # on the side stream on the steps this forward stored the rows (see SpecSamplerBase._copy_outputs_to_host). + self._store_copy_wait = in_forward and store_copy_wait + self.store_copies_done: Optional[torch.cuda.Event] = None + + def bind_engine_buffers( + self, + input_ids: torch.Tensor, + draft_tokens: torch.Tensor, + position_id_offsets: torch.Tensor, + kv_lens_offsets: torch.Tensor, + max_batch_size: int, + ) -> None: + self.step_inputs = (input_ids, draft_tokens, position_id_offsets, kv_lens_offsets) + if self._store_copy_wait: + # Recorded once now so a captured wait has a completed record to wait on before the first copy. + self.store_copies_done = torch.cuda.Event(external=True) + self.store_copies_done.record() + logger.info( + "[M3 host] spec store host copies on the D2H side stream, awaited by the forward's tail" + ) + if self.cpp_tail: + logger.info( + "[M3 host] spec fold tail: one C++ launch (no Triton) at the forward's tail" + ) + if self.in_forward: + self._ctrl = torch.zeros(1, dtype=torch.int32, device=input_ids.device) + self._slots = torch.zeros( + max(max_batch_size, 1), dtype=torch.long, device=input_ids.device + ) + + def bind_sampler_store(self, store: object) -> None: + self.store = store + + def begin_step(self, seq_slots: Optional[list[int]], num_rows: int) -> bool: + if not self.in_forward or self._ctrl is None: + return False + rows = tuple(seq_slots) if seq_slots is not None else None + staged = rows is not None and rows == self._staged_for + self._staged_for = None + armed = rows if rows is not None and 0 < len(rows) <= self._slots.numel() else None + # Read only by this step's forward, so both uploads may wait for the step-input pack's flush. + if armed is not None: + slots = torch.tensor(armed, dtype=torch.long, pin_memory=prefer_pinned()) + copy_h2d_if_changed(self._slots[: len(armed)], slots, pack=True) + ctrl = torch.tensor( + [len(armed) if armed is not None else 0], dtype=torch.int32, pin_memory=prefer_pinned() + ) + copy_h2d_if_changed(self._ctrl, ctrl, pack=True) + self._armed = armed + return staged + + # The launches the store wait may precede. store_copies_done is an external event: under graph capture its wait + # is a node of its own, so the kernel launched right after it loses its programmatic (PDL) edge and with it the + # overlap of its pre-wait part. Only a launch that waits for its predecessor's completion anyway may follow it: + # the fold tail (a plain launch, no griddepcontrol). + STORE_WAIT_BEFORE = frozenset({"fold_tail"}) + + def _wait_store_copies(self, before: str) -> None: + """Order the store's rewrite after the sampler's host copies of it; ``before`` names the launch that follows. + Under graph capture a launch outside STORE_WAIT_BEFORE fails the capture.""" + if self.store_copies_done is None: + return + if before not in self.STORE_WAIT_BEFORE and torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + f"M3SpecStepFold: the store wait would precede {before!r}, a launch outside " + f"{sorted(self.STORE_WAIT_BEFORE)}; an external-event wait node drops the next kernel's PDL edge" + ) + torch.cuda.current_stream().wait_event(self.store_copies_done) + + def store_in_forward(self, outputs: dict) -> None: + """At the forward's tail: store the armed rows' outputs and stage the next step's inputs (one launch).""" + if self._ctrl is None or self.store is None or self.step_inputs is None: + return + store = self.store + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets = self.step_inputs + new_tokens = outputs.get("new_tokens") + next_new_tokens = outputs.get("next_new_tokens") + new_tokens_lens = outputs.get("new_tokens_lens") + next_draft_tokens = outputs.get("next_draft_tokens") + if any( + t is None or not isinstance(t, torch.Tensor) + for t in (new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens) + ): + return + rows = new_tokens.shape[0] + # The previous step's host copies of the store run on the sampler's side stream. + self._wait_store_copies(before="fold_tail") + if not m3_fold_tail.fold_tail( + self._ctrl, + self._slots, + min(rows, self._slots.numel()), + new_tokens, + store.new_tokens.squeeze(-1).T, + next_new_tokens, + store.next_new_tokens.squeeze(-1).T, + next_draft_tokens, + store.next_draft_tokens, + new_tokens_lens, + store.new_tokens_lens, + input_ids, + draft_tokens, + position_id_offsets, + kv_lens_offsets, + next_new_tokens.shape[1], + cpp=self.cpp_tail, + ): + return + if torch.cuda.is_current_stream_capturing(): + self._graph_outputs.add(new_tokens.data_ptr()) + else: + self._eager_outputs = new_tokens + + def gather( + self, + new_tokens: torch.Tensor, + next_draft_tokens: torch.Tensor, + new_tokens_lens: torch.Tensor, + previous_slots: torch.Tensor, + previous_pos_indices: torch.Tensor, + previous_batch_len: int, + tokens_per_gen_step: int, + draft_token_buffer_width: int, + num_tokens: int, + num_draft_tokens: int, + first_previous_row: int, + ) -> bool: + if self.step_inputs is None: + return False + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets = self.step_inputs + return m3_spec_tail.step_gather( + new_tokens, + next_draft_tokens, + new_tokens_lens, + previous_slots, + previous_pos_indices, + previous_batch_len, + tokens_per_gen_step, + draft_token_buffer_width, + input_ids, + num_tokens, + draft_tokens, + num_draft_tokens, + position_id_offsets, + kv_lens_offsets, + first_previous_row, + ) + + def sampler_rows_stored(self, outputs: dict, num_skip: int, seq_slots: list[int]) -> bool: + armed, self._armed = self._armed, None + new_tokens = outputs.get("new_tokens") + ran = isinstance(new_tokens, torch.Tensor) and ( + new_tokens is self._eager_outputs or new_tokens.data_ptr() in self._graph_outputs + ) + self._eager_outputs = None + stored = ( + ran + and armed is not None + and num_skip == 0 + and tuple(seq_slots) == armed + and new_tokens.shape[0] == len(armed) + ) + self._staged_for = armed if stored else None + if stored and not self._logged: + self._logged = True + logger.info("[M3 host] spec step stored and next step staged in the forward tail") + return stored + + def scatter( + self, + seq_slots: torch.Tensor, + new_tokens: torch.Tensor, + next_new_tokens: torch.Tensor, + new_tokens_lens: torch.Tensor, + next_draft_tokens: torch.Tensor, + ) -> bool: + store = self.store + if store is None: + return False + return m3_spec_tail.slot_scatter( + seq_slots, + new_tokens, + store.new_tokens.squeeze(-1).T, + next_new_tokens, + store.next_new_tokens.squeeze(-1).T, + new_tokens_lens, + store.new_tokens_lens, + next_draft_tokens, + store.next_draft_tokens, + ) + + +class M3DSparkStepFold(M3SpecStepFold): + """The step fold under the standalone DSpark worker (DSFOLD): the engine's previous-batch gathers in one launch + (``step_gather``). The sampler's slot stores stay on its in-tree path, so ``scatter`` declines every step. + ``taken`` / ``declined`` count the steps the gather ran / fell back to the engine's own copies.""" + + def __init__(self) -> None: + super().__init__() + self.taken = 0 + self.declined = 0 + + def gather(self, *args, **kwargs) -> bool: + if not super().gather(*args, **kwargs): + self.declined += 1 + uncovered( + "DSFOLD", + "step_gather declined the inputs (dtype / layout); the engine's copies run", + ) + return False + if self.taken == 0: + logger.info( + "[M3 dispatch] DSFOLD DSpark step-input gather: engaged (one step_gather launch)" + ) + self.taken += 1 + m3_report.work("DSFOLD", f"step_gather (declined {self.declined})") + return True + + def scatter( + self, seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens + ) -> bool: + return False + + +class _DraftForward: + """``draft_model.model`` for the in-tree worker: the drafter layer's forward, called as + ``model(input_ids=, position_ids=, hidden_states=, attn_metadata=, spec_metadata=, all_rank_num_tokens=)`` and + returning (hidden, hidden to carry to the next pass). ``d2t`` None: the draft vocabulary is the target's.""" + + d2t = None + + def __init__(self, draft: "M3Eagle3Draft") -> None: + self._draft = draft + + def __call__(self, **kwargs): + return self._draft.forward_layer(**kwargs) + + +class M3Eagle3Draft(nn.Module): + """The one-layer BF16 Eagle3 drafter (``speculative_model``): its weights (``weights.DRAFT_WEIGHTS``), its + vocab-parallel BF16 head and logits processor, the drafter layer's TRTLLM attention backend (layer index = the + target's layer count, FP8 KV) and its TP all-reduce. + + What the worker reads: ``model`` (the layer forward), ``apply_eagle3_fc`` (the per-layer fc_norms + fc over the + captured target hidden states), ``lm_head`` and ``logits_processor``; the fused draft loop reads the layer's parts + (``draft_input_weights``, ``attention``, ``mlp``, ``ar_norm``). The drafter embeds with the target's table. + """ + + def __init__( + self, + core: MiniMaxM3Tp2ModelingV2, + draft_config: ModelConfig, + start_layer_idx: int, + capture_layers: int, + ) -> None: + super().__init__() + cfg = draft_config.pretrained_config + _ensure_draft_vocab_size(cfg) + eagle = getattr(cfg, "eagle_config", None) or {} + mapping = draft_config.mapping + self.draft_config = draft_config + # The engine's draft weight mapper binds to these two. + self.model_config = draft_config + self.config = cfg + self.capture_layers = int(capture_layers) + self.layer_idx = int(start_layer_idx) + self.dtype = cfg.torch_dtype + self.hidden = int(cfg.hidden_size) + self.hidden_size_in = int(getattr(cfg, "target_hidden_size", cfg.hidden_size)) + self.rms_eps = float(cfg.rms_norm_eps) + self.return_hidden_post_norm = bool( + eagle.get("return_hidden_post_norm", False) or getattr(cfg, "norm_output", False) + ) + if ( + int(cfg.num_hidden_layers) != 1 + or eagle.get("eh_proj_before_attn", False) + or eagle.get("norm_before_fc", False) + or not getattr(cfg, "fc_norm", False) + or getattr(cfg, "kv_lora_rank", None) + or self.capture_layers <= 1 + or self.hidden_size_in != self.hidden + or cfg.draft_vocab_size != cfg.vocab_size + or getattr(cfg, "attention_bias", False) + or getattr(cfg, "mlp_bias", False) + or getattr(cfg, "bias", False) + or getattr(cfg, "tie_word_embeddings", False) + ): + raise ValueError( + "this MiniMax-M3 target drafts with a one-layer Llama-style Eagle3 drafter with per-layer " + "fc_norms over " + "several captured layers, the target's hidden size and vocabulary and no biases" + ) + tp = mapping.tp_size + self.heads = cfg.num_attention_heads // tp + self.kv_heads = (cfg.num_key_value_heads + tp - 1) // tp + self.head_dim = int( + getattr(cfg, "head_dim", None) or cfg.hidden_size // cfg.num_attention_heads + ) + self.q_size = self.heads * self.head_dim + self.kv_size = self.kv_heads * self.head_dim + self.w = _weights.DRAFT_WEIGHTS.declare(_weights.DRAFT_WEIGHTS.dims(self)) + self.lm_head = LMHead( + cfg.draft_vocab_size, + cfg.hidden_size, + dtype=cfg.torch_dtype, + mapping=mapping, + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=draft_config.lm_head_gather_output, + reduce_output=False, + use_custom_cublas_mm=False, + quant_config=None, + ) + self.logits_processor = LogitsProcessor() + self.allreduce = AllReduce( + mapping=mapping, strategy=draft_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + quant_config = draft_config.get_quant_config() + self.attn = create_attention( + draft_config.attn_backend, + self.layer_idx, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=PositionalEmbeddingParams( + type=PositionEmbeddingType.rope_gpt_neox, rope=RopeParams.from_config(cfg) + ), + quant_config=quant_config, + skip_create_weights_in_init=draft_config.skip_create_weights_in_init, + q_scaling=1.0, + attention_chunk_size=None, + sparse_params=None, + ) + if not self.attn.support_fused_rope() or not self.attn.support_fused_qkv(): + raise ValueError( + f"the drafter's attention backend {type(self.attn).__name__} must fuse RoPE and QKV" + ) + self.attn.update_quant_config(quant_config) + # The drafter embeds with the target's table, read off the core on every call (the engine materializes the + # core's weights after construction). + self._core = [core] + self.model = _DraftForward(self) + # The projections (weight keys) on the fixed-tactic low-M kernels (``DRAFT_LOWM``), set by + # ``prebuild_linears``. + self._lowm_keys: frozenset = frozenset() + + def prebuild_linears(self) -> int: + """Put the drafter's BF16 projections ``DRAFT_LOWM`` covers on the fixed-tactic low-M kernels and compile + every covered call. Once the weights are real, before the executor's warmup, on every rank. Returns the + number of low-M calls compiled.""" + keys = ("fc", "qkv", "o", "gu", "dn") + built = _draft_lowm_kernel.prebuild([self.w[k] for k in keys], DRAFT_LOWM.keys) + self._lowm_keys = frozenset(k for k in keys if _draft_lowm_kernel.installed(self.w[k])) + return built + + def _linear(self, x: torch.Tensor, key: str) -> torch.Tensor: + """``x @ w[key]^T``: the low-M kernel at the rows ``DRAFT_LOWM`` covers (``prebuild_linears``), else the plain + GEMM.""" + w = self.w[key] + if key in self._lowm_keys: + out = _draft_lowm_kernel.draft_linear(w, x, None) + if out is not None: + return out + return m3_bf16_linear(x, w) + + @property + def embed_table(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def load_weights_from_target_model(self, target_model: nn.Module) -> None: + """Nothing to alias: the drafter reads the target's embedding table through the core and owns its head.""" + + # ------------------------------------------------------------------------------------------------- the layer + + def fc(self, hs: torch.Tensor) -> torch.Tensor: + return self._linear(hs, "fc") + + def apply_eagle3_fc(self, hidden_states: torch.Tensor) -> torch.Tensor: + """The captured target hidden states [T, layers x H] to the drafter's input width: each layer's fc_norm, + then the fc (the in-tree ``Eagle3ForCausalLM.apply_eagle3_fc``).""" + hidden_states = hidden_states.to(self.dtype) + if hidden_states.shape[-1] != self.hidden: + chunks = hidden_states.chunk(self.capture_layers, dim=-1) + hidden_states = torch.cat( + [ + flashinfer_rmsnorm(c, self.w[f"fc_norm{i}"], self.rms_eps) + for i, c in enumerate(chunks) + ], + dim=-1, + ) + hidden_states = self.fc(hidden_states) + return hidden_states + + def attention(self, x: torch.Tensor, md, reduce: bool) -> torch.Tensor: + """QKV, the backend's fused RoPE + KV append + causal attention, o_proj (all-reduced unless ``reduce`` is + False, where the caller fuses the reduction).""" + qkv = self._linear(x, "qkv") + _record("m3_draft_attention", int(md.num_tokens)) + o = m3_draft_attention(self.attn, qkv, md) + out = self._linear(o, "o") + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def mlp(self, h: torch.Tensor, reduce: bool, silu_cpp: bool = False) -> torch.Tensor: + """gate_up, the plain SwiGLU (with ``silu_cpp``, ``DRAFT_SILU_CPP``: this target's C++ op, bit for bit the + Triton one, in one launch), down.""" + gu = self._linear(h, "gu") + act = None + if silu_cpp and DRAFT_SILU_CPP: + act = m3_draft_silu(gu) + if act is None: + uncovered( + DRAFT_SILU_PATH, "the drafter SwiGLU input layout (the Triton op runs it)" + ) + else: + taken(DRAFT_SILU_PATH, int(gu.shape[0]), "drafter MLP SwiGLU, one C++ launch") + if act is None: + act = m3_silu_and_mul(gu, None, None, None) + out = self._linear(act, "dn") + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def ar_norm(self, x: torch.Tensor, residual: torch.Tensor, norm_key: str): + """One draft-layer boundary in the in-tree structure: the projection's plain all-reduce (BF16 out), then the + fused residual add + RMSNorm (FP32 add, norm of the FP32 sum). Returns (normed, residual sum).""" + return flashinfer_fused_add_rmsnorm( + m3_allreduce(x, self.allreduce, None), residual, self.w[norm_key], self.rms_eps + ) + + def forward_layer( + self, + input_ids: torch.Tensor, + position_ids: torch.Tensor, + hidden_states: torch.Tensor, + attn_metadata, + spec_metadata, + all_rank_num_tokens: Optional[List[int]] = None, + ): + """The drafter layer (the in-tree ``Eagle3DraftModel.forward`` for one layer): the embedded tokens and the + fc'd hidden states normed and concatenated, attention, the post-attention norm with the residual, the MLP and + the final norm. Returns (normed output, the hidden state the next pass carries).""" + previous = attn_metadata.all_rank_num_tokens + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = all_rank_num_tokens + try: + embeds = m3_embedding(input_ids, self.embed_table).to(self.dtype) + residual = hidden_states + h = flashinfer_rmsnorm(hidden_states, self.w["hidden_norm"], self.rms_eps) + e = flashinfer_rmsnorm(embeds, self.w["in_norm"], self.rms_eps) + h = torch.cat([e, h], dim=-1) + h = self.attention(h, attn_metadata, reduce=True) + h, residual = flashinfer_fused_add_rmsnorm( + h, residual, self.w["post_norm"], self.rms_eps + ) + h = self.mlp(h, reduce=True) + spec_metadata.maybe_capture_hidden_states(self.layer_idx, h, residual) + h, to_save = flashinfer_fused_add_rmsnorm( + h, residual, self.w["final_norm"], self.rms_eps + ) + return (h, h) if self.return_hidden_post_norm else (h, to_save) + finally: + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = previous + + +def _eagle3_capture(spec_metadata, layer_id: int, hidden_states, residual, capture_norm) -> None: + """The Eagle3 hidden-state capture of one target layer. + + Mirrors ``Eagle3OneModelSpecMetadata.maybe_capture_hidden_states``. With ``capture_norm`` = (the drafter's stacked + fc_norm weights [layers, H], eps), the capture writes rmsnorm(bf16(h + residual)) * fc_norm[i] in one kernel (the + in-tree capture's bf16 add, then the in-tree fc_norm arithmetic), which the fused first drafter inputs read as is; + otherwise it is the in-tree capture. + """ + if capture_norm is None: + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + return + for i, captured_layer_id in enumerate(spec_metadata.layers_to_capture): + if captured_layer_id == layer_id: + assert spec_metadata.hidden_states is not None + num_tokens = spec_metadata.num_capture_tokens + if num_tokens > spec_metadata.hidden_states.shape[0]: + raise RuntimeError( + "EAGLE3 hidden-state capture token count exceeds buffer capacity: " + f"num_tokens={num_tokens}, capacity={spec_metadata.hidden_states.shape[0]}" + ) + if num_tokens > hidden_states.shape[0]: + raise RuntimeError( + "EAGLE3 hidden-state capture token count exceeds available hidden states: " + f"num_tokens={num_tokens}, hidden_states={hidden_states.shape[0]}" + ) + hidden_size = spec_metadata.hidden_size + m3_spec_tail.capture_norm( + hidden_states[:num_tokens], + residual[:num_tokens] if residual is not None else None, + capture_norm[0][i], + capture_norm[1], + spec_metadata.hidden_states[:num_tokens, i * hidden_size : (i + 1) * hidden_size], + ) + break + + +def _sharded_head(lm_head: nn.Module) -> bool: + """A vocab-sharded TP head without padding (the fused kernels read the local shard).""" + return ( + getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and getattr(lm_head, "padding_size", 0) == 0 + and getattr(lm_head, "tp_size", 1) > 1 + ) + + +class M3Eagle3Worker(Eagle3OneModelWorker): + """``Eagle3OneModelWorker`` with the MiniMax-M3 fused verify and fused draft loop, at the in-tree numerics. + + * Verify: on decode-only batches whose verify rows the fused kernels cover (``tail_eligible``), the target's BF16 + lm_head computes this rank's vocabulary shard of the logits (the in-tree head's own local GEMM; at most + ``TARGET_HEAD_MAX_ROWS`` verify rows stream a lossless 12-bit coding of the same BF16 weight) and one kernel + replaces the logits all-gather, the argmax (or the temperature + top-p draw) and the strict acceptance. + * Draft: the drafter's first pass may take the per-layer fc_norms in one launch (or folded into the hidden-state + capture) and run its fc beside the target head GEMM (forked right after the last captured target layer at most + ``FC_EARLY_MAX_ROWS`` verify rows). On a decode-only step ``DRAFT_STEP`` covers, the draft loop is fused: each + pass's draft input in one launch, the drafter layer's attention and MLP with their all-reduces fused with the + residual add and the next RMSNorm, and one kernel for the BF16 head, the cross-rank argmax and the pass's + bookkeeping. + + Every other batch takes the in-tree paths. Methods whose docstring says "Mirrors" copy the named in-tree method + with the fused branches inserted. + """ + + def __init__(self, spec_config, mapping, use_separate_draft_kv_cache: bool = False) -> None: + super().__init__( + spec_config, mapping=mapping, use_separate_draft_kv_cache=use_separate_draft_kv_cache + ) + # Per-step state, reset at the top of every _forward_impl: the first draft pass's gather rows when this step + # takes the fused first drafter inputs, and the fc output a fork computed beside the target head. + self._m3_first_gather_ids: Optional[torch.Tensor] = None + self._m3_fc_hidden: Optional[torch.Tensor] = None + self._m3_target_lm_head: Optional[nn.Module] = None + # (fc_norm0 weight, stacked fc_norm weights [layers, H]), built eagerly on the first fused first pass. + self._m3_fc_norm_w: Optional[tuple] = None + # Armed per step by arm_capture_norm: (stacked fc_norm weights, eps) or None. + self._m3_capture_norm: Optional[tuple] = None + # Per step: the captured layers written with the capture norm (set by capture_hidden_states). + self._m3_capture_normed: set = set() + self._m3_aux_stream: Optional[torch.cuda.Stream] = None + # Armed per step by arm_capture_norm: the drafter whose first fc is forked at the last captured target layer, + # and the fork that did it, joined by _m3_target_accept. + self._m3_fc_early_drafter: Optional[M3Eagle3Draft] = None + self._m3_fc_early_stream: Optional[torch.cuda.Stream] = None + # The 12-bit coded copy of the target head shard the fused verify streams, built before the warmup. + self._m3_target_head = None + self._m3_force_vec_warm = False + # The streamed copy of the drafter's BF16 head shard the fused draft loop reads, built before the warmup. + self._m3_draft_head = None + # Per step: the next step's input tokens [nreq, D + 1], column 0 written by the fused verify (or the seed) and + # columns 1.. by the fused draft passes; set as _m3_next_new_tokens once the fused draft loop filled it. + self._m3_next_new: Optional[torch.Tensor] = None + self._m3_next_new_tokens: Optional[torch.Tensor] = None + # Per step: the first draft inputs are the fused ones (accepted tokens as they lie, fc output of every row). + self._m3_fused_first_inputs = False + + def capture_target( + self, spec_metadata, layer_id: int, num_tokens: int + ) -> Optional[Tuple[torch.Tensor, torch.Tensor, float]]: + """Where this step's capture norm writes layer ``layer_id``'s capture, for a boundary all-reduce that writes it + itself (``CAPTURE_NORM_IN_BOUNDARY``): (the capture buffer's [num_tokens, H] column block, the layer's + fc_norm weight, eps), or None (no capture norm armed, not a captured layer, or more rows than the buffer). + The boundary writes every row it reduces; rows past num_capture_tokens are unread.""" + capture_norm = self._m3_capture_norm + if capture_norm is None or spec_metadata.hidden_states is None: + return None + if num_tokens > spec_metadata.hidden_states.shape[0]: + return None + hidden_size = spec_metadata.hidden_size + for i, captured_layer_id in enumerate(spec_metadata.layers_to_capture): + if captured_layer_id == layer_id: + out = spec_metadata.hidden_states[ + :num_tokens, i * hidden_size : (i + 1) * hidden_size + ] + return out, capture_norm[0][i], capture_norm[1] + return None + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None, written: bool = False + ) -> None: + """The Eagle3 capture with this step's capture norm (``written``: the layer's boundary all-reduce already + wrote it, see ``capture_target``); on a step armed for the early fc fork (``arm_capture_norm``), the last + captured layer then forks the drafter's first fc onto the aux stream, where it overlaps the remaining target + layers and the head (joined by ``_m3_target_accept``).""" + if not written: + _eagle3_capture(spec_metadata, layer_id, hidden_states, residual, self._m3_capture_norm) + if self._m3_capture_norm is not None: + # Written with this step's capture norm: by the boundary all-reduce (written) or just above. + self._m3_capture_normed.add(layer_id) + drafter = self._m3_fc_early_drafter + layers = spec_metadata.layers_to_capture + if drafter is not None and layers and layer_id == layers[-1]: + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = self._m3_fc_fork( + spec_metadata, spec_metadata.num_capture_tokens, drafter + ) + + def prebuild(self, lm_head: nn.Module, draft: Optional[M3Eagle3Draft]) -> None: + """The fused verify's coded target head and the fused draft loop's head copy, from the loaded weights, + before the executor's warmup, on every rank.""" + start = time.perf_counter() + if self.prebuild_target_head(lm_head): + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] coded target head built before warmup: %.1f s" + % (time.perf_counter() - start) + ) + start = time.perf_counter() + if self.prebuild_draft_head(draft): + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] fused draft head built before warmup: %.1f s" + % (time.perf_counter() - start) + ) + + def _forward_impl( + self, + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + resource_manager=None, + ): + """Mirrors Eagle3OneModelWorker._forward_impl, with the fused verify branch.""" + runtime_draft_len = spec_metadata.runtime_draft_len + # skip the draft forward if the runtime draft length is 0 + if runtime_draft_len == 0: + return self.skip_drafting( + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + ) + + batch_size = attn_metadata.num_seqs + num_contexts = attn_metadata.num_contexts + num_gens = batch_size - num_contexts + + raw_logits = logits + + self._m3_first_gather_ids = None + self._m3_next_new = None + self._m3_next_new_tokens = None + self._m3_fused_first_inputs = False + if logits is None: + # tail_eligible() skipped the gathered target logits: verify from this rank's logits shard. + accepted_tokens, num_accepted_tokens, self._m3_first_gather_ids = ( + self._m3_target_accept( + hidden_states, + spec_metadata, + batch_size, + runtime_draft_len, + draft_model, + kv_lens=getattr(attn_metadata, "kv_lens_cuda", None), + ) + ) + else: + self._execute_guided_decoder_if_present(logits) + + # Sample and accept tokens. ``input_ids`` is required by the relaxed-acceptance path (scans for + # thinking-phase tokens); ignored otherwise. + accepted_tokens, num_accepted_tokens = self.sample_and_accept_draft_tokens( + input_ids, logits, attn_metadata, spec_metadata + ) + if self.draft_only_eligible(attn_metadata, spec_metadata, draft_model): + self._m3_seed_first_pass(num_accepted_tokens, spec_metadata, batch_size) + + # Mamba hybrid models need state updates after token acceptance (a no-op on this target's cache manager). + if self._is_mamba_hybrid_cache is None: + self._is_mamba_hybrid_cache = isinstance( + attn_metadata.kv_cache_manager, MambaHybridCacheManager + ) + if num_gens > 0 and self._is_mamba_hybrid_cache: + attn_metadata.kv_cache_manager.update_mamba_states( + attn_metadata=attn_metadata, + num_accepted_tokens=num_accepted_tokens, + state_indices=attn_metadata.mamba_metadata.state_indices, + ) + + sa_manager = getattr(spec_metadata.spec_resource_manager, "sa_manager", None) + if self.sa_enhancer is not None and sa_manager is not None: + self.sa_enhancer.extend_and_prepare( + sa_manager=sa_manager, + request_ids=spec_metadata.request_ids, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_gens=num_gens, + num_contexts=num_contexts, + max_draft_len=runtime_draft_len, + ) + + if self._uses_external_shared_target_kv: + next_draft_tokens = self._forward_external_shared_target_kv_draft_loop( + position_ids=position_ids, + hidden_states=hidden_states, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_contexts=num_contexts, + batch_size=batch_size, + ) + else: + # Save the old attn_metadata and spec_metadata + self._prepare_attn_metadata_for_spec_dec(attn_metadata) + + # Prepare inputs for the 1st draft model forward + position_ids = position_ids.squeeze(0) + inputs = self.prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + + # Predict draft tokens. ``original_all_rank_num_tokens`` is saved here so the post-loop restore (below) + # can put attn_metadata back into a state the target model expects. + original_all_rank_num_tokens = attn_metadata.all_rank_num_tokens + + # Get the draft KV cache manager if using separate layouts + draft_kv_cache_manager = self.get_draft_kv_cache_manager(resource_manager) + + next_draft_tokens = self._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + # restore attn_metadata to support cuda graph + self._restore_attn_metadata_from_spec_dec(attn_metadata) + # restore all_rank_num_tokens for attention DP + if original_all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = original_all_rank_num_tokens + + # prepare next new tokens to support overlap scheduler + if self._m3_next_new_tokens is not None: + # Filled in place by the fused verify or the seed (column 0) and the fused draft passes (columns 1..). + next_new_tokens = self._m3_next_new_tokens + else: + next_new_tokens = self._prepare_next_new_tokens( + accepted_tokens, + next_draft_tokens, + spec_metadata.batch_indices_cuda, + batch_size, + num_accepted_tokens, + ) + + attn_metadata.use_spec_decoding = True + + # Commit model-owned target state only after draft preparation has succeeded; SpecWorkerBase aborts the + # still-pending snapshots on any exception above this point. + if self._auxiliary_state_handlers: + self.commit_auxiliary_speculative_states( + num_accepted_tokens, + attn_metadata.mamba_metadata.state_indices[:batch_size], + num_contexts, + ) + + return { + "logits": raw_logits, + "new_tokens": accepted_tokens, + "new_tokens_lens": num_accepted_tokens, + "next_draft_tokens": next_draft_tokens, + "next_new_tokens": next_new_tokens, + } + + def prepare_1st_drafter_inputs( + self, + input_ids, + position_ids, + hidden_states, + accepted_tokens, + attn_metadata, + spec_metadata, + draft_model, + ): + """The fused first draft inputs on a step that verified with the fused kernels (or was seeded for them), + else the in-tree ones.""" + fused = self._m3_first_drafter_inputs( + position_ids, + accepted_tokens, + input_ids.shape[0], + attn_metadata.num_contexts, + spec_metadata, + draft_model, + ) + self._m3_fused_first_inputs = fused is not None + if fused is not None: + taken("m3_first_drafter_inputs", attn_metadata.num_seqs) + fused.update(attn_metadata=attn_metadata, spec_metadata=spec_metadata) + return fused + return super().prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + + def _m3_structural_ok(self, attn_metadata, spec_metadata) -> bool: + """The batch conditions shared by the fused verify and the fused first drafter inputs: a decode-only plain + Eagle3 step without attention DP, guided decoding, relaxed acceptance or a suffix-automaton enhancer, of at + most TP2_ROWS verify rows (the fused tail's certified rows; R1-VR-TP8 does not widen it). Host state only, so + the TP ranks agree.""" + if self.is_mtp_eagle or self.use_dynamic_tree: + return False + if self._uses_external_shared_target_kv or self.sa_enhancer is not None: + return False + if ( + self.guided_decoder is not None + or self.mapping is None + or self.mapping.enable_attention_dp + ): + return False + if getattr(self.spec_config, "use_relaxed_acceptance_for_thinking", False): + return False + if attn_metadata.num_contexts != 0 or attn_metadata.num_seqs == 0: + return False + draft_len = spec_metadata.runtime_draft_len + if draft_len == 0 or attn_metadata.num_seqs * (draft_len + 1) > DECODE_MAX_TOKENS: + return False + return not spec_metadata.wants_advanced_draft_sampling + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Whether this step verifies with the fused kernels instead of the gathered target logits. + + Decode-only Eagle3 batches (``_m3_structural_ok``) over an unpadded vocab-sharded bias-free target head, all + greedy at a request count ``SHARD_ACCEPT`` covers, or sampled under the temperature + top-p contract (see + _m3_sampled_verify_ok), with nothing else reading the target logits. Every input is host state shared by all + TP ranks, so the ranks agree. + """ + if not self._m3_structural_ok(attn_metadata, spec_metadata): + return False + if getattr(spec_metadata, "enable_penalty", False): + return False + if not _sharded_head(lm_head) or getattr(lm_head, "bias", None) is not None: + return False + draft_len = spec_metadata.runtime_draft_len + if spec_metadata.is_all_greedy_sample: + key = (draft_len, attn_metadata.num_seqs) + if not SHARD_ACCEPT.covers(key): + uncovered( + SHARD_ACCEPT.path, + f"draft length {draft_len} at {attn_metadata.num_seqs} requests", + ) + return False + elif attn_metadata.num_seqs > SAMPLED_VERIFY_MAX_REQUESTS: + uncovered("m3_sampled_accept", f"sampled verify at {attn_metadata.num_seqs} requests") + return False + elif not self._m3_sampled_verify_ok(spec_metadata): + return False + self._m3_target_lm_head = lm_head + return True + + def draft_only_eligible(self, attn_metadata, spec_metadata, draft_model) -> bool: + """Whether a step verified by the in-tree path can still take the fused first drafter inputs: the structural + conditions of tail_eligible, without the target-sampling ones, up to ``FIRST_DRAFTER_INPUTS_MAX_REQUESTS`` + requests.""" + if attn_metadata.num_seqs > FIRST_DRAFTER_INPUTS_MAX_REQUESTS: + uncovered( + "m3_first_drafter_inputs", f"in-tree verify at {attn_metadata.num_seqs} requests" + ) + return False + return self._m3_structural_ok(attn_metadata, spec_metadata) and draft_model is not None + + def _m3_seed_first_pass(self, num_accepted_tokens, spec_metadata, batch_size: int) -> None: + """The first draft pass's gather rows (the last accepted row of each request) from the in-tree acceptance, as + _m3_target_accept leaves them.""" + k1 = spec_metadata.runtime_draft_len + 1 + n = num_accepted_tokens[:batch_size].to(torch.int64) + rows = spec_metadata.batch_indices_cuda[:batch_size].to(torch.int64) + self._m3_first_gather_ids = rows * k1 + n - 1 + + def _m3_first_drafter_inputs( + self, + position_ids, + accepted_tokens, + num_tokens: int, + num_contexts: int, + spec_metadata, + draft_model: Optional[M3Eagle3Draft], + ): + """The in-tree first draft inputs of a decode-only step, with the per-layer fc_norm RMSNorms of + ``apply_eagle3_fc`` in one kernel (unless the capture already applied them) before the fc; the input ids are + the accepted tokens as they lie. None to use the in-tree path.""" + if ( + self._m3_first_gather_ids is None + or self.is_mtp_eagle + or num_contexts != 0 + or draft_model is None + ): + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + ): + return None + cache = self._m3_fc_norm_w + if cache is None or cache[0] is not draft_model.w["fc_norm0"]: + if torch.cuda.is_current_stream_capturing(): + capture_missed("spec_tail", None, "the stacked Eagle3 fc_norm weights") + return None + cache = ( + draft_model.w["fc_norm0"], + torch.stack( + [draft_model.w[f"fc_norm{i}"].detach() for i in range(layers)] + ).contiguous(), + ) + self._m3_fc_norm_w = cache + hidden = self._m3_fc_hidden + self._m3_fc_hidden = None + if hidden is None or hidden.shape[0] != num_tokens: + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + hs = m3_spec_tail.chunk_norm(hs, cache[1], draft_model.rms_eps) + else: + hs = self._m3_renorm_raw_captures(hs, spec_metadata, draft_model) + hidden = draft_model.fc(hs) + input_ids = accepted_tokens[:, : spec_metadata.runtime_draft_len + 1].reshape(-1) + return {"input_ids": input_ids, "position_ids": position_ids, "hidden_states": hidden} + + def _m3_renorm_raw_captures( + self, hs: torch.Tensor, spec_metadata, draft_model: M3Eagle3Draft + ) -> torch.Tensor: + """The captured rows ``hs`` [rows, layers * H] for a reader that skips the fc_norm (the capture norm is armed): + unchanged when every captured layer was written with this step's norm (capture_hidden_states records them), + else -- not expected on any path -- a copy with each layer captured raw normalized by its own fc_norm here, + logged as a miss, so a raw row never reaches the drafter's fc.""" + missing = [ + i + for i, layer_id in enumerate(spec_metadata.layers_to_capture) + if layer_id not in self._m3_capture_normed + ] + if not missing: + return hs + missed( + "m3_capture_norm", + int(hs.shape[0]), + f"captured layer(s) {[spec_metadata.layers_to_capture[i] for i in missing]} were written without the " + "capture norm; their fc_norm is applied before the drafter's fc", + ) + hs = hs.clone() + hidden = spec_metadata.hidden_size + for i in missing: + block = hs[:, i * hidden : (i + 1) * hidden] + block.copy_( + flashinfer_rmsnorm( + block.contiguous(), draft_model.w[f"fc_norm{i}"], draft_model.rms_eps + ) + ) + return hs + + def _m3_fc_fork( + self, + spec_metadata, + num_tokens: int, + draft_model: Optional[M3Eagle3Draft], + after: Optional[torch.cuda.Event] = None, + ) -> Optional[torch.cuda.Stream]: + """Launch the first draft pass's fc_norm + fc on a forked stream beside the target head GEMM (after the event + ``after`` when given, else after everything queued on the current stream); the result is picked up by + _m3_first_drafter_inputs. Returns the stream to join, or None when the fc path does not apply (the first + drafter inputs then compute it).""" + self._m3_fc_hidden = None + if draft_model is None: + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + or self._m3_fc_norm_w is None + ): + return None + aux = self._m3_aux_stream + if aux is None: + aux = self._m3_aux_stream = torch.cuda.Stream() + if after is not None: + aux.wait_event(after) + else: + aux.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(aux): + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + hs = m3_spec_tail.chunk_norm( + hs, self._m3_fc_norm_w[1], draft_model.rms_eps, use_pdl=False + ) + else: + hs = self._m3_renorm_raw_captures(hs, spec_metadata, draft_model) + self._m3_fc_hidden = draft_model.fc(hs) + return aux + + def _m3_fc_join(self, fork: torch.cuda.Stream) -> None: + """Join the fc fork onto the current stream and keep its output's memory alive for that stream. + + _m3_fc_hidden is allocated in fork (aux) stream order and read on the current stream. Without the record, + dropping it returns its block to the aux stream's pool, where later aux work can take it while the current + stream's reads are still queued (eager warm-up / mixed steps). Nothing is recorded under graph capture, as in + maybe_execute_in_parallel: a captured graph relies on the next fork's aux.wait_stream(main) to order the + reuse.""" + main = torch.cuda.current_stream() + main.wait_stream(fork) + hidden = self._m3_fc_hidden + if ( + hidden is not None + and hidden.is_cuda + and not (torch.compiler.is_compiling() or torch.cuda.is_current_stream_capturing()) + ): + hidden.record_stream(main) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model: Optional[M3Eagle3Draft] + ) -> None: + """Before every target forward: whether this step's Eagle3 captures apply the drafter's fc_norm themselves. + + On a fused-verify step whose fc_norm weights are already stacked (by an earlier fused first pass), the + capture writes rmsnorm(h + r) * fc_norm[i] (one kernel per captured layer instead of add + copy) and the fused + first drafter inputs read it without their own chunk_norm. A fused-verify step of at most + ``FC_EARLY_MAX_ROWS`` verify rows is also armed to fork the drafter's first fc at the last captured layer + (``capture_hidden_states``). + """ + self._m3_capture_norm = None + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = None + # The captured layers this step wrote with its capture norm (capture_hidden_states). + self._m3_capture_normed = set() + # A fork result no first draft pass consumed must not reach a later step's (same-size) first inputs. + self._m3_fc_hidden = None + if draft_model is None: + return + if not self.tail_eligible(attn_metadata, spec_metadata, lm_head): + return + # The fork reads every captured column at the last captured layer, so a final-state capture, which is written + # after the model returns, keeps the fc at its stock place. + if ( + attn_metadata.num_seqs * (spec_metadata.runtime_draft_len + 1) <= FC_EARLY_MAX_ROWS + and FINAL_HIDDEN_LAYER_ID not in spec_metadata.layers_to_capture + ): + self._m3_fc_early_drafter = draft_model + fc_norm_w = self._m3_fc_norm_w + if ( + fc_norm_w is not None + and fc_norm_w[0] is draft_model.w["fc_norm0"] + and len(spec_metadata.layers_to_capture) == fc_norm_w[1].shape[0] + ): + self._m3_capture_norm = (fc_norm_w[1], draft_model.rms_eps) + + @staticmethod + def _m3_sampled_verify_ok(spec_metadata) -> bool: + """Whether sampled verify rows can use the fused sampled op: only under advanced_sampling_mode=no_topk, the + deploy-time guarantee that no row uses top-k or min-p (that op applies neither).""" + mode = getattr(spec_metadata, "advanced_sampling_mode", None) + return mode is not None and mode.skips_top_k and not mode.skips_top_p + + def _m3_target_accept( + self, + hidden_states, + spec_metadata, + batch_size: int, + draft_len: int, + draft_model: Optional[M3Eagle3Draft] = None, + kv_lens: Optional[torch.Tensor] = None, + ): + """Fused verification (see tail_eligible); same outputs as the strict acceptance path. ``kv_lens``: the + requests' KV lengths at this verify (the per-request forced-acceptance key, ``FORCE_AL_PER_REQUEST``).""" + force_total = 0 + force_vec = None + if self.force_num_accepted_tokens != 0.0: + is_warmup = spec_metadata.is_cuda_graph and not torch.cuda.is_current_stream_capturing() + if not torch.cuda.is_current_stream_capturing(): + # The draw pool is built by an H2D copy and the draw kernel JIT-compiles per block width, so both + # happen here, before the first captured step (a scratch counter keeps the real draw sequence). + self._ensure_force_accept_rng_state(hidden_states.device) + if not self._m3_force_vec_warm: + scratch = torch.zeros(1, dtype=torch.int64, device=hidden_states.device) + dummy = torch.empty(64, dtype=torch.int32, device=hidden_states.device) + for block in (16, 32, 64): + m3_force_vec( + self._force_accept_rng_pool, + scratch, + dummy, + block, + 1, + 2, + 0.5, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + m3_force_vec_req( + self._force_accept_rng_pool, + dummy, + dummy, + block, + 1, + 2, + 0.5, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + self._m3_force_vec_warm = True + if not is_warmup: + force_total = min(int(self.force_num_accepted_tokens) + 1, draft_len + 1) + force_vec = self._m3_force_vec( + batch_size, force_total, draft_len + 1, hidden_states.device, kv_lens + ) + draft_tokens = spec_metadata.draft_tokens.view(batch_size, draft_len) + if draft_tokens.dtype != torch.int32: + draft_tokens = draft_tokens.to(torch.int32) + # Column 0 receives the last accepted token, the next step's first input (the fused draft loop adds the rest). + self._m3_next_new = torch.empty( + (batch_size, draft_len + 1), dtype=torch.int32, device=hidden_states.device + ) + rows = hidden_states[: batch_size * (draft_len + 1)] + lm_head = self._m3_target_lm_head + batch_idx = spec_metadata.batch_indices_cuda[:batch_size] + head = self._m3_coded_head(lm_head, rows.shape[0]) + # The Eagle3 fc on the aux stream: the fork made at the last captured layer, else one beside the head. A + # sampled verify's coded head holds every SM while it streams: its fc is forked after it instead (overlapping + # the sampled acceptance), where it does not delay the head's start. + fork = self._m3_fc_early_stream + self._m3_fc_early_stream = None + head_first = ( + fork is None + and (head is not None or rows.shape[0] <= FC_AFTER_HEAD_MAX_ROWS) + and not spec_metadata.is_all_greedy_sample + ) + if fork is None and not head_first: + fork = self._m3_fc_fork(spec_metadata, rows.shape[0], draft_model) + # This rank's vocabulary shard of the target logits: the head's own local GEMM (or its coded copy), whose + # all-gather the fused kernels replace (they exchange per-row records across ranks, not logits). + if head is not None: + taken(TARGET_HEAD.path, rows.shape[0]) + logits = m3_target_head_kernel.logits(head, rows) + else: + logits = lm_head.apply_linear(rows, None) + head_done = None + if head_first: + head_done = torch.cuda.Event() + head_done.record() + vocab_offset = self.mapping.tp_rank * lm_head.weight.shape[0] + if spec_metadata.is_all_greedy_sample: + result = m3_shard_accept_kernel.shard_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + force_total, + self._m3_next_new, + force_vec=force_vec, + ) + path = SHARD_ACCEPT.path + else: + n_rows = rows.shape[0] + seeds, offsets = self._rng_state_per_token(spec_metadata, n_rows) + result = m3_sampled_accept_kernel.sampled_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + spec_metadata.temperatures[:n_rows], + spec_metadata.top_ks[:n_rows], + spec_metadata.top_ps[:n_rows], + None, + seeds, + offsets, + force_total, + self._m3_next_new, + force_vec=force_vec, + one_round_k=SAMPLED_ACCEPT_1R_K if SAMPLED_ACCEPT_SM107TP4 else 0, + prewait_params=SAMPLED_ACCEPT_PREWAIT_PARAMS, + early_trigger=SAMPLED_ACCEPT_EARLY_TRIGGER, + ilp_walk=SAMPLED_ACCEPT_ILP_WALK, + ) + path = "spec_tail" + if head_done is not None: + # Enqueued after the sampled acceptance (which starts first) but waits on the head only. + fork = self._m3_fc_fork(spec_metadata, rows.shape[0], draft_model, after=head_done) + if fork is not None: + self._m3_fc_join(fork) + if result is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, None, "the fused verify's exchange workspace") + raise RuntimeError( + "M3 fused verify unavailable after tail_eligible() accepted the step" + ) + taken( + SHARD_ACCEPT.path if spec_metadata.is_all_greedy_sample else "m3_sampled_accept", + batch_size, + f"draft length {draft_len}" + + ( + "" + if spec_metadata.is_all_greedy_sample + else f", SA-1R one_round_k {SAMPLED_ACCEPT_1R_K if SAMPLED_ACCEPT_SM107TP4 else 0}" + ), + ) + return result + + def prebuild_target_head(self, lm_head: nn.Module) -> bool: + """Build the 12-bit coded copy of the target head shard the fused verify streams at <= TARGET_HEAD_MAX_ROWS + rows (rebuilt from the current weight values on every call), on every rank. True when the coded copy decodes + to the BF16 weight bit for bit; otherwise (or for a head the fused verify does not read) nothing is kept and + the head's own GEMM runs.""" + self._m3_target_head = None + if not _sharded_head(lm_head) or not m3_target_head_kernel.head_supported(lm_head): + return False + head = m3_target_head_kernel.prepare_head(lm_head) + self._m3_target_head = head + if not head.coded: + logger.warning( + f"[M3 target head] the coded shard {tuple(lm_head.weight.shape)} does not decode to the BF16 weight " + "bit for bit; the fused verify keeps the head's BF16 GEMM" + ) + return head.coded + + def _m3_coded_head(self, lm_head: nn.Module, rows: int): + """The coded target head for a fused verify of ``rows`` rows, or None for the head's own GEMM: above + ``TARGET_HEAD_MAX_ROWS`` rows (uncovered), on a shard whose coding failed its round trip (uncovered: the GEMM + is the exact path), and when the copy was not built for this weight (a miss).""" + path = TARGET_HEAD.path + if not TARGET_HEAD.covers(rows): + uncovered(path, f"{rows} verify rows (the head's own GEMM runs)") + return None + head = self._m3_target_head + if head is None or head.weight is not lm_head.weight: + missed( + path, + rows, + "the coded target head is not built for the head's current weight (prebuild)", + ) + return None + if not head.coded: + uncovered( + path, "the coded shard failed its bitwise round trip (the head's own GEMM runs)" + ) + return None + return head + + def _m3_force_vec( + self, + batch_size: int, + base_total: int, + max_total: int, + device, + kv_lens: Optional[torch.Tensor] = None, + ) -> Optional[torch.Tensor]: + """Per-request forced accepted counts for a fractional forced acceptance, else None. + + ``FORCE_AL_PER_REQUEST`` (with ``kv_lens``): each request draws from the fixed-seed pool at its own key, its + KV length at this verify (its prompt plus what it has accepted so far), so its draws do not depend on the + server's history or on the other requests; every TP rank holds the same lengths, so the ranks agree. Without + lengths: exactly as _apply_force_accepted_tokens draws for a decode-only batch (one shared counter bump per + step, the (counter, slot) hash), the generic path's sequence. + """ + frac = self.force_num_accepted_tokens - int(self.force_num_accepted_tokens) + if frac <= 0.0 or base_total >= max_total: + return None + self._ensure_force_accept_rng_state(device) + out = torch.empty(batch_size, dtype=torch.int32, device=device) + if FORCE_AL_PER_REQUEST and kv_lens is not None: + keys = kv_lens[:batch_size] + if keys.dtype != torch.int32: + keys = keys.to(torch.int32) + return m3_force_vec_req( + self._force_accept_rng_pool, + keys, + out, + batch_size, + base_total, + max_total, + float(frac), + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + max(16, 1 << (batch_size - 1).bit_length()), + ) + return m3_force_vec( + self._force_accept_rng_pool, + self._force_accept_rng_counter, + out, + batch_size, + base_total, + max_total, + float(frac), + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + max(16, 1 << (batch_size - 1).bit_length()), + ) + + # ---------------------------------------------------------------------------------------------- fused draft loop + + def prebuild_draft_head(self, draft_model: Optional[M3Eagle3Draft]) -> bool: + """Build the streamed copy of the drafter's BF16 head shard that the fused draft loop reads (rebuilt from the + current weight values on every call), on every rank. False when the drafter is not one the loop serves.""" + self._m3_draft_head = None + if draft_model is None or not self._m3_drafter_supported(draft_model): + return False + self._m3_draft_head = m3_draft_step_kernel.prepare_head( + draft_model.lm_head, self.mapping.tp_rank, DRAFT_HEAD_CODED + ) + return True + + def _m3_drafter_supported(self, draft_model: M3Eagle3Draft) -> bool: + """The drafter the fused draft loop runs (``M3Eagle3Draft`` validates its shape at construction) over this + worker's tensor-parallel group with a bias-free vocab-sharded BF16 head without padding.""" + if self.mapping is None or self.mapping.tp_size <= 1 or self.mapping.enable_attention_dp: + return False + if self._d2t is not None: + return False + lm_head = draft_model.lm_head + return _sharded_head(lm_head) and m3_draft_step_kernel.head_supported(lm_head) + + def _m3_draft_ar_norm( + self, + draft_model: M3Eagle3Draft, + x: torch.Tensor, + residual: torch.Tensor, + norm_key: str, + pushed: bool = False, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """One drafter boundary of the fused draft loop: the multicast all-reduce + residual + RMSNorm in its drafter + form (``DRAFT_MC_AR``; bitwise the stock boundary), else the built-in path (``M3Eagle3Draft.ar_norm``). The + choice depends on shapes and a collectively created workspace only, so the TP ranks agree. ``pushed``: the + projection already pushed its rows into the all-reduce (``DRAFT_AR_PUSH``).""" + rows = x.shape[0] + path = DRAFT_MC_AR.path + if DRAFT_MC_AR.covers(rows): + workspace = m3_mc_allreduce.get_workspace(self.mapping, x.shape[-1]) + out = None + if workspace is not None: + out = m3_mc_allreduce.m3_mc_allreduce_norm( + x.contiguous(), + residual.contiguous(), + draft_model.w[norm_key], + draft_model.rms_eps, + workspace, + use_pdl=get_env_enable_pdl(), + drafter_norm=True, + carveout=DRAFT_MC_AR_CARVEOUT, + prepushed=2 if pushed else 0, + drafter_cluster=DRAFT_MC_AR_CLUSTER, + ) + if out is not None: + taken( + path, + rows, + "drafter boundary (cluster kernel)" + if DRAFT_MC_AR_CLUSTER + else "drafter boundary (row kernel)", + ) + if pushed: + taken(DRAFT_AR_PUSH_PATH, rows, "the projection pushed its rows (prepushed 2)") + elif DRAFT_AR_PUSH: + uncovered( + DRAFT_AR_PUSH_PATH, + "the boundary's projection did not push (the AR pushes x)", + ) + return out + if pushed: + raise RuntimeError( + "the drafter's rows were pushed into a boundary all-reduce that did not run" + ) + missed(path, rows, "the multicast all-reduce workspace or operand layout") + return draft_model.ar_norm(x, residual, norm_key) + + def _m3_arm_draft_push(self, weight: torch.Tensor, rows: int) -> None: + """``DRAFT_AR_PUSH``: let the projection ``weight``'s next low-M split-K call push its rows into the boundary + all-reduce that ``_m3_draft_ar_norm`` then runs (the multicast path only; shapes and a collectively created + workspace decide it, so the TP ranks agree).""" + if not (DRAFT_AR_PUSH and DRAFT_MC_AR.covers(rows)): + return + workspace = m3_mc_allreduce.get_workspace(self.mapping, weight.shape[0]) + if workspace is not None: + _draft_lowm_kernel.arm_push(weight, workspace, rows) + + def _m3_fused_loop_ok( + self, attn_metadata, spec_metadata, batch_size: int, draft_model: Optional[M3Eagle3Draft] + ) -> bool: + """Whether this step drafts with the fused loop: a decode-only step the fused verify or the seed prepared + (``_m3_first_gather_ids``) with the fused first draft inputs, a supported drafter, a (draft length, requests) + ``DRAFT_STEP`` covers, the prebuilt head and the exchange workspace. Host state only, so the TP ranks + agree.""" + if ( + self._m3_first_gather_ids is None + or not self._m3_fused_first_inputs + or draft_model is None + ): + return False + path = DRAFT_STEP.path + if not self._m3_drafter_supported(draft_model) or not hasattr( + attn_metadata, "kv_lens_cuda" + ): + uncovered( + path, "the drafter or its attention metadata is not one the fused loop serves" + ) + return False + draft_len = spec_metadata.runtime_draft_len + if not DRAFT_STEP.covers((draft_len, batch_size)): + uncovered(path, f"draft length {draft_len} at {batch_size} requests") + return False + head = self._m3_draft_head + if head is None or head.weight is not draft_model.lm_head.weight: + missed( + path, + batch_size, + "the drafter head's streamed copy is not built for its current weight (prebuild)", + ) + return False + if m3_draft_step_kernel.workspace(self.mapping, head.wf.device) is None: + capture_missed(path, None, "the draft-step exchange workspace") + return False + return True + + def _forward_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ): + """The fused draft loop on a step it covers (``_m3_fused_loop_ok``), else the in-tree loop.""" + if self._m3_fused_loop_ok(attn_metadata, spec_metadata, batch_size, draft_model): + return self._m3_fused_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + batch_size, + num_accepted_tokens, + ) + return super()._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + + def _m3_fused_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model: M3Eagle3Draft, + draft_kv_cache_manager, + batch_size: int, + num_accepted_tokens, + ): + """Mirrors Eagle3OneModelWorker._forward_linear_draft_loop for a decode-only Eagle3 step, with each pass's + glue fused (see ``_m3_fused_loop_ok``): + + * the draft input -- the embedding lookup, the layer's input norm and hidden norm and their concat -- in one + launch (``m3_spec_tail.draft_input``; the second pass gathers the last accepted rows there); + * the drafter layer's attention and MLP, each followed by the all-reduce with the residual add and the next + RMSNorm (the post-attention norm, then the drafter's final norm); + * one kernel for the BF16 head over the head rows (pass 0: the last accepted row of each request), the + cross-rank argmax and the bookkeeping the in-tree loop does between passes: the draft token, the next pass's + input id and position, the draft KV lengths (and pass 0's sequence lengths) (``m3_draft_step``). + + The host metadata transitions are the in-tree loop's. The drafts land in columns 1.. of the step's next-input + tokens (column 0 is the last accepted token), which ``_forward_impl`` returns as they lie. + """ + draft_len = spec_metadata.runtime_draft_len + table = draft_model.embed_table + dev = table.device + head = self._m3_draft_head + gather_ids = self._m3_first_gather_ids + ids = inputs["input_ids"] + hidden = inputs["hidden_states"] + pos = inputs["position_ids"] + if ids.dtype != torch.int32: + ids = ids.to(torch.int32) + if pos.dtype != torch.int32: + pos = pos.to(torch.int32) + num_acc = num_accepted_tokens[:batch_size] + if num_acc.dtype != torch.int32: + num_acc = num_acc.to(torch.int32) + next_new = self._m3_next_new + if next_new is None: + # Seeded from the in-tree acceptance: the last accepted token is the first-pass input at the gather row. + next_new = torch.empty((batch_size, draft_len + 1), dtype=torch.int32, device=dev) + next_new[:, 0] = ids[gather_ids] + draft_tokens = next_new[:, 1:] + w = draft_model.w + eps = draft_model.rms_eps + + with self.draft_kv_cache_context(attn_metadata, draft_kv_cache_manager) as md: + for i in range(draft_len): + x, residual = m3_spec_tail.draft_input( + ids, + table, + hidden, + gather_ids if i == 1 else None, + w["in_norm"], + eps, + w["hidden_norm"], + eps, + ) + taken(DRAFT_INPUT.path, x.shape[0]) + rows = x.shape[0] + self._m3_arm_draft_push(w["o"], rows) + h = draft_model.attention(x, md, reduce=False) + h, residual = self._m3_draft_ar_norm( + draft_model, + h, + residual, + "post_norm", + pushed=_draft_lowm_kernel.take_pushed(h), + ) + self._m3_arm_draft_push(w["dn"], rows) + h = draft_model.mlp(h, reduce=False, silu_cpp=True) + h, to_save = self._m3_draft_ar_norm( + draft_model, + h, + residual, + "final_norm", + pushed=_draft_lowm_kernel.take_pushed(h), + ) + if draft_model.return_hidden_post_norm: + to_save = h + ids_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + pos_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + if not m3_draft_step_kernel.draft_step( + head, + h, + batch_size, + gather_ids if i == 0 else None, + self.mapping, + draft_tokens, + i, + ids_next, + pos, + pos_next, + md.kv_lens_cuda, + num_acc if i == 0 else None, + md._seq_lens_cuda if i == 0 else None, + draft_len, + ): + raise RuntimeError( + "M3 fused draft step unavailable after _m3_fused_loop_ok() accepted the step" + ) + taken(DRAFT_STEP.path, batch_size, f"draft length {draft_len}") + ids, pos, hidden = ids_next, pos_next, to_save + if i == 0: + # The draft step set the device sequence lengths to 1 and rolled the KV lengths back. + md._seq_lens[:batch_size].fill_(1) + md.on_update() + has_kv_cache = md.kv_cache_manager is not None + if has_kv_cache and hasattr(md, "host_request_types"): + md.host_request_types[: md.num_contexts].fill_(1) + if has_kv_cache: + md.num_contexts = 0 + self._prepare_flash_mla_generation_layout(md, 0, batch_size) + md.update_for_spec_dec() + md.use_spec_decoding = False + else: + md.update_for_spec_dec() + self._m3_next_new_tokens = next_new + return draft_tokens + + +def _build_speculation(shell, model_config: ModelConfig): + """The drafter config, the drafter and the worker of one-model Eagle3 (the in-tree one-engine construction for + the llama3 drafter architecture), or of one-model standalone DSpark (``dspark.build_dspark``: the in-tree drafter + and worker).""" + spec = model_config.spec_config + if _dspark.is_standalone_dspark(spec): + shell.model.embed_tokens = _dspark.M3EmbedTokens(shell.model) + return _dspark.build_dspark(shell, model_config) + if ( + not spec.spec_dec_mode.use_one_engine() + or not spec.spec_dec_mode.is_eagle3_one_model() + or spec.eagle3_model_arch != "llama3" + or getattr(spec, "use_dynamic_tree", False) + ): + raise ValueError( + "this MiniMax-M3 target speculates with linear one-model Eagle3 (llama3 drafter) only" + ) + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, + trust_remote_code=True, + attn_backend=model_config.attn_backend, + moe_backend=_get_requested_draft_moe_backend(model_config, spec), + mapping=model_config.mapping, + spec_config=model_config.spec_config, + max_num_tokens=model_config.max_num_tokens, + moe_max_num_tokens=model_config.moe_max_num_tokens, + ) + _set_draft_kv_cache_quant_algo(draft_config, model_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = M3Eagle3Draft( + shell.model, + draft_config, + model_config.pretrained_config.num_hidden_layers, + spec.num_capture_layers, + ) + worker = M3Eagle3Worker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + return draft_config, draft, worker, use_separate + + +# -------------------------------------------------------------------------------------------------------------------- +# Runtime hooks and the shell +# -------------------------------------------------------------------------------------------------------------------- + + +# TRTLLM_M31_DISABLE_HOSTTAIL=1 turns off the six items of the sm_103 targets' HOSTTAIL bundle together (HOST_D2H_SIDE, +# SMALL_H2D, BATCH_CHANGED_ONLY, SKIP_ACCEPTED_UPLOAD, PHILOX_PACK, KV_LENS_ONCE below), as there. +_HOSTTAIL_OFF = m31_disabled("HOSTTAIL") +# SKIP_ACCEPTED_UPLOAD (KIMI-HO-09, RuntimeHooks.skip_accepted_draft_upload): the step's per-request accepted-draft +# counts are no longer uploaded and spec_metadata.num_accepted_draft_tokens is None. Nothing on this target reads that +# tensor (modeling.py, dspark.py, dspark_s4.py, mxdraft.py, kernels/, ops/; the in-tree DFlash / DSpark workers and +# the Eagle3 path: tests/unittest/_torch/executor/test_skip_accepted_draft_upload.py scans for readers). Kill switch: +# TRTLLM_M31_DISABLE_SKIP_ACCEPTED_UPLOAD=1 (the stock per-step upload, packed under spec_small_h2d). +STEP_SKIP_ACCEPTED_UPLOAD = not (_HOSTTAIL_OFF or m31_disabled("SKIP_ACCEPTED_UPLOAD")) +# BATCH_CHANGED_ONLY (KIMI-HO-03 / ST-18(d), RuntimeHooks.spec_batch_map_changed_only): the standalone DSpark (DFlash) +# drafter's batch_indices and batch-to-slot map are uploaded only when they change (both constant in steady decode); +# the buffers hold the same values as with the stock every-step copies. Kill switch: +# TRTLLM_M31_DISABLE_BATCH_CHANGED_ONLY=1 (both copied every step). +STEP_BATCH_CHANGED_ONLY = not (_HOSTTAIL_OFF or m31_disabled("BATCH_CHANGED_ONLY")) +# The rest of the step loop's host-side items, each default on with its own kill switch; every one is exact (the same +# values reach the device, only the copies' stream, size, grouping or timing change): +# * HOST_D2H_SIDE (RuntimeHooks.sampler_d2h_side_stream): the spec sampler's store copies to the host on a side +# stream. TRTLLM_M31_DISABLE_HOST_D2H_SIDE=1: the stock copies on the main stream (the spec fold's store_copies_done +# event is then never re-recorded, so its tail wait is satisfied, and stream order keeps the copies ahead of it). +# * SMALL_H2D (spec_small_h2d): the sampler's slot ids stay in a device buffer re-uploaded only on change and the +# accepted counts ride in the step-input pack. TRTLLM_M31_DISABLE_SMALL_H2D=1: the stock per-step uploads. +# * PHILOX_PACK (pack_spec_rng_uploads): sampled batches stage their Philox seed/offset uploads into the step-input +# pack. TRTLLM_M31_DISABLE_PHILOX_PACK=1: the stock separate uploads. +# * KV_LENS_ONCE (kv_lens_update_once): a speculative overlap step re-derives the MSA KV-length metadata once, after the +# KV-length correction. TRTLLM_M31_DISABLE_KV_LENS_ONCE=1: both passes. +# * PACK_PSCATTER (pack_parallel_scatter): the pack's non-overlapping copies scatter with a program per copy. +# TRTLLM_M31_DISABLE_PACK_PSCATTER=1: the serial one-CTA scatter (and, as below, no in-graph pack). +# * PACK_IN_GRAPH (pack_in_graph; needs PACK_PSCATTER): the decode graph's prologue issues the pack (and, with it, the +# step epoch). TRTLLM_M31_DISABLE_PACK_IN_GRAPH=1: the eager upload and scatter between the graphs. +# * PACK_EARLY_COPY (pack_early_copy; needs PACK_IN_GRAPH): the prologue's window is uploaded on a side copy stream +# while the previous graph runs, and the prologue is this target's one launch (m3_step_prologue). +# TRTLLM_M31_DISABLE_PACK_EARLY_COPY=1: the prologue copies the pinned window itself (scatter launches + fill). +# * H2D_CHANGED_SPAN (h2d_changed_span): a changed packed 2-D upload stages only each row's changed column span. +# TRTLLM_M31_DISABLE_H2D_CHANGED_SPAN=1: the whole view. +STEP_HOST_D2H_SIDE = not (_HOSTTAIL_OFF or m31_disabled("HOST_D2H_SIDE")) +STEP_SMALL_H2D = not (_HOSTTAIL_OFF or m31_disabled("SMALL_H2D")) +STEP_PHILOX_PACK = not (_HOSTTAIL_OFF or m31_disabled("PHILOX_PACK")) +STEP_KV_LENS_ONCE = not (_HOSTTAIL_OFF or m31_disabled("KV_LENS_ONCE")) +STEP_PACK_PSCATTER = not m31_disabled("PACK_PSCATTER") +STEP_PACK_IN_GRAPH = STEP_PACK_PSCATTER and not m31_disabled("PACK_IN_GRAPH") +STEP_PACK_EARLY_COPY = STEP_PACK_IN_GRAPH and not m31_disabled("PACK_EARLY_COPY") +STEP_H2D_CHANGED_SPAN = not m31_disabled("H2D_CHANGED_SPAN") +# BLOCKOFF_GUARD (RuntimeHooks.skip_unchanged_draft_block_offsets): the standalone DSpark drafter's separate KV cache +# manager skips its per-step block-table copy (one launch between the step graphs, full max_seq_len width) when the +# copied rows' page indices are unchanged since its last copy into the same buffer; a skipped step leaves the bytes +# the copy would write. TRTLLM_M31_DISABLE_BLOCKOFF_GUARD=1 copies every step. +STEP_BLOCKOFF_GUARD = not m31_disabled("BLOCKOFF_GUARD") +# BLOCKOFF_PATCH (RuntimeHooks.patch_unchanged_draft_block_offsets; needs BLOCKOFF_GUARD): the drafter's sliding window +# changes one page index per row on about a third of the decode steps (a tail page allocated or re-allocated, a front +# page released); such a step rewrites the copied rows with the copy kernel's int32 transform done on the host and one +# host-to-device copy, instead of the full-width copy kernel. TRTLLM_M31_DISABLE_BLOCKOFF_PATCH=1 copies those steps. +STEP_BLOCKOFF_PATCH = STEP_BLOCKOFF_GUARD and not m31_disabled("BLOCKOFF_PATCH") +# SLOT_IDS_CHANGED (spec_slot_ids_changed_only; ported from the sm_103 targets): the spec +# metadata's per-request slot-id table (batch_slot_ids; rejection sampling) is re-uploaded only when the slot ids +# change, not copied every step. With BLOCKOFF_GUARD it empties the eager step-boundary stream the next decode graph +# waits on. OFF by default on this target: its token-identity gate failed here (259 / 272 requests identical, the +# rest diverging mid-sequence), so the skip misses a slot-id change on this step loop. Opt in with +# TRTLLM_M31_ENABLE_SLOT_IDS_CHANGED=1; TRTLLM_M31_DISABLE_SLOT_IDS_CHANGED=1 still wins. Read at import. +SLOT_IDS_CHANGED = m31_enabled("SLOT_IDS_CHANGED") and not m31_disabled("SLOT_IDS_CHANGED") + + +def _counted_pack_prologue(*args, **kwargs): + """``m3_step_prologue`` (the decode graph's step-input pack prologue) with its gate-e counters: each call is the + in-graph pack (PACK_IN_GRAPH) issued from the early-copied window (PACK_EARLY_COPY) by the parallel scatter + (PACK_PSCATTER), the setting it is installed under.""" + out = m3_step_prologue(*args, **kwargs) + m3_report.work("PACK_EARLY_COPY", "the C++ step-input prologue") + m3_report.work("PACK_IN_GRAPH", "pack issued by the decode graph's prologue") + m3_report.work("PACK_PSCATTER", "the prologue's parallel scatter") + return out + + +def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: + """The step-loop hooks this target opts into (each behind its constant above). + + * ``pack_step_inputs``: a decode step's small input copies go out as one upload and one scatter at the graph + replay instead of a chain of copies the next graph waits on (~30 us of idle GPU at one request). + * ``skip_unchanged_h2d``: steady decode re-uploads slot indices, query lengths and page tables that rarely change. + * ``loop_on_execution_stream``: the overlap loop's own stream work and the forward on one stream, so step N+1's + inputs wait on graph N in program order rather than through two cross-stream event hops. + * ``merge_tuned_tactics_across_ranks``: tensor-parallel peers capture the same GEMM tactics (rank-local profiling + noise otherwise splits them, and the slowest rank sets every step). + * ``skip_greedy_rng_upload``: all-greedy speculative batches never read the Philox seeds. + * ``spec_step_fold``: the spec sampler's slot stores and the engine's next-step input gathers in one launch each. + * ``pack_parallel_scatter`` / ``pack_in_graph`` / ``pack_early_copy``: the pack's copies scatter in parallel, issued + by the decode graph's own prologue from a window uploaded ahead of the replay on a side copy stream. + * ``pack_spec_rng_uploads`` / ``spec_small_h2d``: the spec step's seeds, accepted counts and slot ids ride the pack + or stay on the device. + * ``sampler_d2h_side_stream`` / ``iteration_timing_one_event``: the sampler's store copies and the iteration timing + leave the main stream. + * ``skip_accepted_draft_upload`` / ``spec_batch_map_changed_only`` (kill switches ``SKIP_ACCEPTED_UPLOAD`` / + ``BATCH_CHANGED_ONLY``, ``STEP_SKIP_ACCEPTED_UPLOAD`` / ``STEP_BATCH_CHANGED_ONLY``): no per-step upload of the + accepted-draft counts (no reader here), and the drafter's batch_indices / batch-to-slot map uploaded only on + change. + * MSA: identity selections for short sequences (and short/long graph families), the one-launch KV-length update, + unchanged block-table copies skipped, pool views built once, block-id tables copied from the page-index views. + """ + logger.info(f"[M3.1] SLOT_IDS_CHANGED (BP-3) {'on' if SLOT_IDS_CHANGED else 'off'}") + # Gate e: the step-loop items' work is done by the shared runtime (pyexecutor) from these flags; this target logs + # what it armed, per rank. Their work lines are the runtime's own ("[M3 host] ..."); the pack prologue below is + # this target's op and counts its calls. + armed = { + "HOST_D2H_SIDE": STEP_HOST_D2H_SIDE, + "SMALL_H2D": STEP_SMALL_H2D, + "PHILOX_PACK": STEP_PHILOX_PACK, + "KV_LENS_ONCE": STEP_KV_LENS_ONCE, + "BATCH_CHANGED_ONLY": STEP_BATCH_CHANGED_ONLY, + "SKIP_ACCEPTED_UPLOAD": STEP_SKIP_ACCEPTED_UPLOAD, + "PACK_PSCATTER": STEP_PACK_PSCATTER, + "PACK_IN_GRAPH": STEP_PACK_IN_GRAPH, + "PACK_EARLY_COPY": STEP_PACK_EARLY_COPY, + "H2D_CHANGED_SPAN": STEP_H2D_CHANGED_SPAN, + "BLOCKOFF_GUARD": STEP_BLOCKOFF_GUARD, + "BLOCKOFF_PATCH": STEP_BLOCKOFF_PATCH, + "SLOT_IDS_CHANGED": SLOT_IDS_CHANGED, + } + logger.info( + f"[M3 sm107 tp2] rank {mpi_rank()}: step-loop items armed (shared runtime does the work): " + + " ".join(f"{k}={'on' if v else 'off'}" for k, v in armed.items()) + ) + return RuntimeHooks( + fp8_saturating_casts=True, + msa_kv_cache_fixes=True, + draft_pool_fixes=True, + persist_flashinfer_mxfp8_tactics=True, + # Bundled name-keyed tactic pins (m31_pins); kill: TRTLLM_M31_DISABLE_BUNDLED_PINS=1. + install_tactic_pins=lambda rank: m31_pins.install("tp2", rank), + disagg_draft_kv=True, # kill: TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV=1 + first_content_emit_iters=FIRST_CONTENT_EMIT_ITERS, + mxfp8_pinned_tactics=True, + late_compile_detect=True, + pack_step_inputs=STEP_PACK_INPUTS, + pack_parallel_scatter=STEP_PACK_PSCATTER, + pack_in_graph=STEP_PACK_IN_GRAPH, + pack_early_copy=STEP_PACK_EARLY_COPY, + pack_step_epoch=STEP_PACK_EPOCH, + pack_prologue_op=_counted_pack_prologue if STEP_PACK_CPP_PROLOGUE else None, + skip_unchanged_h2d=STEP_SKIP_UNCHANGED_H2D, + h2d_changed_span=STEP_H2D_CHANGED_SPAN, + kv_lens_update_once=STEP_KV_LENS_ONCE, + spec_offsets_op=m3_spec_offsets if STEP_SPEC_OFFSETS_ONE_LAUNCH else None, + loop_on_execution_stream=STEP_LOOP_ON_EXECUTION_STREAM, + merge_tuned_tactics_across_ranks=STEP_MERGE_TUNED_TACTICS, + skip_greedy_rng_upload=STEP_SKIP_GREEDY_RNG_UPLOAD, + pack_spec_rng_uploads=STEP_PHILOX_PACK, + spec_small_h2d=STEP_SMALL_H2D, + skip_accepted_draft_upload=STEP_SKIP_ACCEPTED_UPLOAD, + spec_batch_map_changed_only=STEP_BATCH_CHANGED_ONLY, + spec_slot_ids_changed_only=SLOT_IDS_CHANGED, + spec_step_fold=spec_fold, + sampler_d2h_side_stream=STEP_HOST_D2H_SIDE, + skip_unchanged_draft_block_offsets=STEP_BLOCKOFF_GUARD, + patch_unchanged_draft_block_offsets=STEP_BLOCKOFF_PATCH, + iteration_timing_one_event=STEP_ONE_TIMING_EVENT, + msa=MsaRuntimePolicy( + forward_staging=True, + short_sequence_identity=MSA_SHORT_SEQUENCE_IDENTITY, + kv_update=_msa_kv_update_kernel.msa_kv_update if MSA_KV_UPDATE else None, + skip_unchanged_block_offsets=MSA_SKIP_UNCHANGED_BLOCK_OFFSETS, + reuse_pool_views=MSA_REUSE_POOL_VIEWS, + block_ids_from_views=MSA_BLOCK_IDS_FROM_VIEWS, + prefill_proxy_qo_tile=128 if MSA_PROXY_Q128 else None, + prefill_proxy_kv_splits=4 if MSA_PROXY_Q128 else None, + prefill_proxy_kv_splits_by_rows=( + ( + (128, 64, 1), + (2048, 32, 20), + (4096, 16, 20), + (16384, 16 if m31_disabled("AGG_PROXY_SPLITCAP5") else 8, 40), + ) + if MSA_PROXY_SPLITS + else None + ), + prefill_proxy_oracle=MSA_PROXY_ORACLE, + ), + ) + + +# TOPP_DET (ported from the sm_103 targets): the spec sampler's top-p renorm (FlashInfer +# top_p_renorm_probs in the no-top-k rejection path, the draft's and the verify's call each step) runs as one +# persistent C++ launch (ops/sampling/m31_topp_det, kernels/minimaxM31/toppRenormDetSm103.cu, built for sm_107 +# in-tree or overlaid as tensorrt_llm/libs/libm31_topp_det.so) through the sampler's top-p renorm override, instead of +# FlashInfer's five launches with a one-CTA-per-row apply (AirTopPRenormApply: 63 us per call, 126 us per step at TP8 +# K7 c1 on VR). +# Bitwise FlashInfer's deterministic renorm on rows with top_p < 1; rows with top_p >= 1 keep every token +# (a deliberate deviation: FlashInfer drops tail buckets there in ~43% of rows). Taken up to ``_topp_det.MAX_ROWS`` +# rows; larger calls, inputs outside the contract and builds without the op keep the stock call. Kill switch: +# TRTLLM_M31_DISABLE_TOPP_DET=1 (stock FlashInfer top_p_renorm_probs). +TOPP_DET = not m31_disabled("TOPP_DET") +_TOPP_DET_STATE: dict = {} + + +def _install_topp_det() -> None: + """Once per process: install ``_topp_det.top_p_renorm_det`` as the sampler's top-p renorm override when TOPP_DET + is on and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _TOPP_DET_STATE: + return + if not TOPP_DET: + state = "off (TRTLLM_M31_DISABLE_TOPP_DET=1); stock FlashInfer top_p_renorm_probs" + elif _topp_det.topp_det_op() is None: + state = "not taken: the op is not built; stock FlashInfer top_p_renorm_probs" + else: + set_top_p_renorm_override(_topp_det.top_p_renorm_det) + ctas = _topp_det.ctas_per_row(_topp_det.MAX_ROWS, _topp_det.sm_count(torch.device("cuda"))) + state = f"engaged (ctas_per_row={ctas}, rows <= {_topp_det.MAX_ROWS})" + _TOPP_DET_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01 TOPP_DET det top-p renorm: {state}") + + +def _prebuild_topp_det() -> None: + """The override's persistent workspace on this rank's device, allocated before warmup and any graph capture.""" + if _TOPP_DET_STATE.get("installed"): + _topp_det.prebuild() + + +# SOFTMAX_MC (ported from the sm_103 targets): the spec sampler's softmax (FlashInfer +# softmax(logits, temperature) in softmax_op and compute_probs_from_logits_op, the draft's and the verify's call each +# step) keeps FlashInfer's Map kernel and runs its Reduce kernel's write pass on ``_softmax_mc.WRITE_CTAS`` CTAs per row +# (ops/sampling/m31_softmax_mc, kernels/minimaxM31/softmaxMcSm103.cu, built for sm_107 in-tree or overlaid as +# tensorrt_llm/libs/libm31_softmax_mc.so) through the sampler's softmax override, instead of one 1024-thread CTA per +# row. Bitwise FlashInfer's softmax on its Path A shapes (rows <= 128, vocab >= 24576; GB300 microbench at +# write CTAs 4 / 8 / 16). GB300 per call at 7-8 rows: 9.2 vs 21.8 us. Taken up to ``_softmax_mc.MAX_ROWS`` rows; larger +# calls at 129..448 rows use the separate Path-B operation when built and enabled. +# Inputs outside both contracts keep the stock call. Kill switch: +# TRTLLM_M31_DISABLE_SOFTMAX_MC=1 (stock FlashInfer softmax). +SOFTMAX_MC = not m31_disabled("SOFTMAX_MC") +_SOFTMAX_MC_STATE: dict = {} + + +def _install_softmax_mc() -> None: + """Once per process: install ``_softmax_mc.softmax_mc`` as the sampler's softmax override when SOFTMAX_MC is on + and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _SOFTMAX_MC_STATE: + return + if not SOFTMAX_MC: + state = "off (TRTLLM_M31_DISABLE_SOFTMAX_MC=1); stock FlashInfer softmax" + elif _softmax_mc.softmax_mc_op() is None: + state = "not taken: the op is not built; stock FlashInfer softmax" + else: + set_softmax_override(_softmax_mc.softmax_mc) + rows = _softmax_mc.max_rows() + state = f"engaged (write_ctas={_softmax_mc.write_ctas(1)}..{_softmax_mc.write_ctas(rows)}, rows <= {rows})" + _SOFTMAX_MC_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01-SM softmax multi-CTA write: {state}") + + +def _prebuild_softmax_mc() -> None: + """The override's persistent partials workspace on this rank's device, allocated before warmup and any graph + capture.""" + if _SOFTMAX_MC_STATE.get("installed"): + _softmax_mc.prebuild() + + +# REJ_MC (ported from the sm_103 targets): the spec sampler's rejection step +# (trtllm::chain_speculative_sampling in speculative/interface.rejection_sampling_one_model, once per verify step) runs +# on ctas_per_row CTAs per request (ops/sampling/m31_chain_spec_mc, kernels/minimaxM31/chainSpecSamplingMcSm103.cu, +# built for sm_107 in-tree or overlaid as tensorrt_llm/libs/libm31_chain_spec_mc.so) through the rejection sampler's +# override, instead of one 1024-thread CTA per request. ctas_per_row is sized from this device's SM count (the CTAs +# of a request meet at global-memory barriers, so batch x ctas_per_row stays co-resident). Bitwise the stock op's +# tokens and counts (GB300 microbench: K7, batch 1 / 2 / 4 / 8 x 5 cases x 64 seeds at ctas_per_row +# 4 / 8 / 16, eager and graph). GB300 per call at batch 1: 31.2 vs 41.2 us. Taken up to ``_chain_spec_mc.MAX_BATCH`` +# requests; larger calls, inputs outside the contract and builds without the op keep the stock op. Kill switch: +# TRTLLM_M31_DISABLE_REJ_MC=1 (stock trtllm::chain_speculative_sampling). +REJ_MC = not m31_disabled("REJ_MC") +# REJ_SLOTS (VR only): on top of REJ_MC, the rejection step reads each request's draft +# distributions in place from the spec metadata's slot-indexed buffer (draft_probs[slot, k, :]) instead of the +# sampler first gathering them into a contiguous copy (one ~5.6 MB index_elementwise per step at c1). The same floats, +# so bitwise the gathered call. Enabled by default when the slot-indexed native op is available. +# TRTLLM_M31_DISABLE_REJ_SLOTS=1 restores the gather + REJ_MC path. +REJ_SLOTS = not m31_disabled("REJ_SLOTS") +_REJ_MC_STATE: dict = {} + + +def _install_rej_mc() -> None: + """Once per process: install ``_chain_spec_mc.chain_spec_mc`` as the rejection sampler's override when REJ_MC is + on and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _REJ_MC_STATE: + return + if not REJ_MC: + state = "off (TRTLLM_M31_DISABLE_REJ_MC=1); stock trtllm::chain_speculative_sampling" + elif _chain_spec_mc.chain_spec_mc_op() is None: + state = "not taken: the op is not built; stock trtllm::chain_speculative_sampling" + else: + set_chain_speculative_sampling_override(_chain_spec_mc.chain_spec_mc) + batch = _chain_spec_mc.max_batch() + ctas = _chain_spec_mc.ctas_per_row(batch, _chain_spec_mc.sm_count(torch.device("cuda"))) + state = f"engaged (ctas_per_row={ctas} at batch {batch}, batch <= {batch})" + slots = REJ_SLOTS and _chain_spec_mc.chain_spec_mc_slots_op() is not None + if slots: + set_chain_speculative_sampling_slot_override(_chain_spec_mc.chain_spec_mc_slots) + state += f"; REJ_SLOTS slot-indexed draft rows {'on' if slots else 'off'}" + _REJ_MC_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01-REJ chain speculative sampling multi-CTA: {state}") + + +def _prebuild_rej_mc() -> None: + """The override's persistent workspace on this rank's device, allocated before warmup and any graph capture.""" + if _REJ_MC_STATE.get("installed"): + _chain_spec_mc.prebuild() + + +class ModelingV2MiniMaxM31Nvfp4Sm107Tp2( + DecoderModelForCausalLM[MiniMaxM3Tp2ModelingV2, PretrainedConfig] +): + """MiniMax-M3 NVFP4 / sm_107 / tp2, text only, with one-model Eagle3 or no speculation. + + The shell owns the vocab-parallel BF16 ``lm_head`` and the logits processor (``DecoderModelForCausalLM``); with + speculation, the drafter (``M3Eagle3Draft``, its weights from the speculative checkpoint through + ``load_draft_weights``) and the worker (``M3Eagle3Worker``) that verifies and drafts each step. + """ + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + assert torch.cuda.get_device_capability() == _SM, ( + f"this target is built for sm_{_SM[0]}{_SM[1]}, the device is sm_" + f"{''.join(map(str, torch.cuda.get_device_capability()))}" + ) + # Under modeling_v2='require' a covered decode shape that cannot take its fast path raises. + set_require(resolved_mode(type(self)) is ModelingV2Mode.REQUIRE) + model_config = _text_model_config(model_config) + cfg = model_config.pretrained_config + spec_config = getattr(model_config, "spec_config", None) + if ( + model_config.quant_config.kv_cache_quant_algo == QuantAlgo.NVFP4 + and _dspark.draft_kv_override_applies(spec_config) + ): + # The one-model Eagle3 layer has no NVFP4 trtllm-gen decode kernel: keep its modules and the shared + # draft cache on FP8 while the target stays NVFP4. + model_config.extra_attrs["draft_kv_cache_quant_algo_override"] = QuantAlgo.FP8 + super().__init__( + MiniMaxM3Tp2ModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.spec_config = spec_config + # A standalone DSpark drafter takes none of the Eagle3 drafter's hooks (full-context attention, low-M + # linears, boundary capture-norm target). + self._standalone_dspark = _dspark.is_standalone_dspark(spec_config) + self.draft_model: Optional[nn.Module] = None + self.draft_config: Optional[ModelConfig] = None + self.spec_worker: Optional[Union[M3Eagle3Worker, _dspark.M3DSparkWorker]] = None + self.use_separate_draft_kv_cache = False + self.layer_idx = FINAL_HIDDEN_LAYER_ID + spec_fold = None + if spec_config is not None: + ( + self.draft_config, + self.draft_model, + self.spec_worker, + self.use_separate_draft_kv_cache, + ) = _build_speculation(self, model_config) + # The step fold serves the Eagle3 worker's sampler slots; the DSpark worker keeps the in-tree ones. + if STEP_SPEC_FOLD and not _dspark.is_standalone_dspark(spec_config): + spec_fold = M3SpecStepFold( + in_forward=STEP_SPEC_FOLD_IN_FORWARD, + store_copy_wait=STEP_SPEC_STORE_COPY_SIDE, + cpp_tail=STEP_FOLD_TAIL_CPP, + ) + elif DSFOLD and _dspark.is_standalone_dspark(spec_config): + # The DSpark worker keeps the in-tree sampler slot stores; only the engine's input gathers fold. + spec_fold = M3DSparkStepFold() + self._spec_fold = spec_fold + self.runtime_hooks = _runtime_hooks(spec_fold) + _install_topp_det() + _install_softmax_mc() + _install_rej_mc() + + def load_weights(self, weights: Dict, *args, **kwargs) -> None: + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_draft_weights(self, weights: Dict, weight_mapper=None) -> None: + """The drafter's checkpoint (``speculative_model``), loaded by the engine after the target's.""" + if _dspark.is_standalone_dspark(self.spec_config): + _dspark.load_dspark_draft_weights(self, weights, weight_mapper) + return + _weights.DRAFT_WEIGHTS.load(self.draft_model, weights) + + def set_guided_decoder(self, guided_decoder) -> bool: + """Mirrors ``SpecDecOneEngineForCausalLM.set_guided_decoder``: guided decoding with speculation is installed + on the worker.""" + if hasattr(self.spec_worker, "set_guided_decoder"): + return self.spec_worker.set_guided_decoder(guided_decoder) + return False + + def cache_derived_state(self) -> None: + """Build the targets and every decode kernel once the weights are real, before the executor's warmup; with + speculation also the drafter layer's full-context decode attention (``DRAFTER_FULLCTX``), the fused verify's + coded target head, the fused draft loop's head copy and the drafter's low-M GEMMs (``DRAFT_LOWM``).""" + super().cache_derived_state() + self.model.post_load_weights() + _prebuild_topp_det() + _prebuild_softmax_mc() + _prebuild_rej_mc() + draft = self.draft_model + eagle3_draft = draft is not None and not self._standalone_dspark + if eagle3_draft and DRAFTER_FULLCTX.keys: + path = DRAFTER_FULLCTX.path + installed = _drafter_attn_kernel.install( + draft, + {k: FULLCTX_DECODE_RECIPES[k] for k in DRAFTER_FULLCTX.keys}, + FULLCTX_DECODE_MAX_KV_LEN, + FULLCTX_DECODE_OWN_REDUCER and _fullctx_decode_kernel.has_own_reducer(), + get_env_enable_pdl(), + fused_rope_mode=( + DRAFTER_ROPE_MODE + if DRAFTER_ROPE_FUSED + and hasattr(torch.ops.trtllm, "minimax_m3_sm107tp4_drafter_rope_fp8_kv") + else None + ), + on_rope_taken=lambda rows, what: taken(DRAFTER_ROPE_PATH, rows, what), + on_taken=lambda rows: taken(path, rows), + on_uncovered=lambda reason: uncovered(path, reason), + on_missed=lambda rows, reason: missed(path, rows, reason), + ) + logger.info(f"[M3 prebuild] {path} installed on {installed} drafter attention layer(s)") + if self.draft_model is not None and self._standalone_dspark and VR_DRAFT_FMHA_SM100F: + bypass = _drafter_attn_kernel.sliding_fi_bypass_reason(self.draft_model) + if bypass is not None: + logger.info(f"[M3 prebuild] VR_DRAFT_FMHA_SM100F: not installed: {bypass}") + else: + installed = _drafter_attn_kernel.install_sliding_fi(self.draft_model) + logger.info( + f"[M3 prebuild] VR_DRAFT_FMHA_SM100F: FlashInfer trtllm-gen FMHA installed on {installed} " + "drafter attention layer(s)" + ) + if self.spec_worker is not None: + self.spec_worker.prebuild(self.lm_head, draft) + if eagle3_draft: + # The drafter linears' low-M kernels, before the warmup. + start = time.perf_counter() + built = draft.prebuild_linears() + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] drafter low-M GEMMs built before warmup: %d calls, %.1f s" + % (built, time.perf_counter() - start) + ) + + def post_load_weights(self) -> None: + super().post_load_weights() + self.model.post_load_weights() + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.Tensor] = None, + position_ids: Optional[torch.Tensor] = None, + inputs_embeds: Optional[torch.Tensor] = None, + return_context_logits: bool = False, + spec_metadata: Optional[SpecMetadata] = None, + resource_manager: Optional[Any] = None, + **kwargs, + ) -> Any: + """One engine step: the target forward, then the logits (or, with speculation, the worker's verify and + draft). Text only.""" + multimodal_params = kwargs.pop("multimodal_params", None) or [] + if any(p is not None and getattr(p, "multimodal_data", None) for p in multimodal_params): + raise ValueError( + "this MiniMax-M3 target is text only; the request carries multimodal input" + ) + worker = self.spec_worker + capture = _plain_capture + if worker is not None: + capture = worker.capture_hidden_states + if spec_metadata is not None: + worker.arm_capture_norm( + attn_metadata, spec_metadata, self.lm_head, self.draft_model + ) + hidden_states = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + capture=capture, + capture_target=( + worker.capture_target + if worker is not None and not self._standalone_dspark + else None + ), + **kwargs, + ) + if getattr(attn_metadata, "csa2_remote_tail_mode", None) == "source": + # A context worker stopped at the encoder/decoder boundary: a disposable token for the context-only + # completion, and no draft worker. + return self.logits_processor.forward(hidden_states, self.lm_head, attn_metadata, False) + if spec_metadata is not None and spec_metadata.is_layer_capture(FINAL_HIDDEN_LAYER_ID): + capture(spec_metadata, FINAL_HIDDEN_LAYER_ID, hidden_states, None) + if attn_metadata.padded_num_tokens is not None: + hidden_states = hidden_states[: attn_metadata.num_tokens] + if worker is None: + return self.logits_processor.forward( + hidden_states, self.lm_head, attn_metadata, return_context_logits + ) + # The fused verify computes this rank's logits shard itself and needs no gathered logits. + if worker.tail_eligible(attn_metadata, spec_metadata, self.lm_head): + logits = None + else: + # A worker may produce the gathered target logits itself (bitwise to the processor); None = the processor. + target_logits = getattr(worker, "target_logits", None) + logits = ( + target_logits(hidden_states, attn_metadata, spec_metadata, self.lm_head) + if target_logits is not None + else None + ) + if logits is None: + logits = self.logits_processor.forward( + hidden_states[spec_metadata.gather_ids], self.lm_head, attn_metadata, True + ) + spec_input_ids = input_ids if input_ids is not None else kwargs.get("orig_input_ids") + spec_position_ids = position_ids + if attn_metadata.padded_num_tokens is not None: + if spec_input_ids is not None: + spec_input_ids = spec_input_ids[: attn_metadata.num_tokens] + if position_ids is not None: + spec_position_ids = _slice_spec_position_ids(position_ids, attn_metadata.num_tokens) + outputs = worker( + input_ids=spec_input_ids, + position_ids=spec_position_ids, + hidden_states=hidden_states, + logits=logits, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=resource_manager, + ) + spec_fold = self._spec_fold + if ( + spec_fold is not None + and spec_fold.in_forward + and attn_metadata.num_contexts == 0 + and isinstance(outputs, dict) + ): + spec_fold.store_in_forward(outputs) + return outputs diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/mxdraft.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/mxdraft.py new file mode 100644 index 000000000000..4bf6bdc2e07c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/mxdraft.py @@ -0,0 +1,116 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MXDRAFT: the DSpark drafter's decode projections on this target's MXFP8 GEMV. + +The drafter checkpoint ships its linears as MXFP8 (E4M3 codes, one UE8M0 scale per 32 columns) and the stock drafter +dequantizes them to BF16 at load, so every decode pass streams twice the bytes it needs. MXDRAFT recovers the MXFP8 +form of each BF16 weight (exactly: the recovery is checked to dequantize back to the same BF16 bits, else the +projection is left alone) and routes the projection's GEMM to ``m3_mx_gemv_bf16`` for decode inputs it serves; any +other call runs the Linear unchanged. The activation is quantized to MXFP8 inside the GEMV, so the output is not the +stock BF16 GEMM's: drafter-only precision, gated on natural AL. Row-parallel projections keep +their all-reduce (only the Linear's GEMM is replaced). +""" + +from __future__ import annotations + +from typing import Iterable, Optional, Tuple + +import torch + +from .kernels import m3_report +from .kernels.mx_gemv import op as _gemv +from .ops.comm.m3_mc_allreduce_norm import swizzle_sf + +E4M3_MAX = 448.0 +BLOCK = 32 + + +def mxfp8_from_bf16(weight: torch.Tensor) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """The MXFP8 form (E4M3 codes [N, K], UE8M0 scale bytes [N, K / 32]) of a BF16 weight that is exactly an MXFP8 + dequantization, or None. Each block takes the smallest power-of-two scale that fits its amax in E4M3, which is at + most the shipped scale, so the codes are exact whenever the weight came from MXFP8.""" + if weight.dtype != torch.bfloat16 or weight.dim() != 2 or weight.shape[1] % BLOCK != 0: + return None + n, k = weight.shape + blocks = weight.float().view(n, k // BLOCK, BLOCK) + amax = blocks.abs().amax(-1) + exp = torch.ceil(torch.log2(amax.clamp_min(2.0**-126) / E4M3_MAX)).clamp(-127, 127) + scale = torch.exp2(exp).unsqueeze(-1) + codes = (blocks / scale).to(torch.float8_e4m3fn) + back = (codes.float() * scale).view(n, k).to(torch.bfloat16) + if not torch.equal(back, weight): + return None + return codes.view(n, k).contiguous(), (exp + 127).to(torch.uint8).contiguous() + + +def _projections(draft_model) -> Iterable[torch.nn.Module]: + for layer in getattr(getattr(draft_model, "model", None), "layers", []) or []: + attn, mlp = getattr(layer, "self_attn", None), getattr(layer, "mlp", None) + for lin in ( + getattr(attn, "qkv_proj", None), + getattr(attn, "o_proj", None), + getattr(mlp, "gate_up_proj", None), + getattr(mlp, "down_proj", None), + ): + if lin is not None: + yield lin + + +def route_mx_linears(draft_model, rows: Iterable[int]) -> int: + """Route each drafter projection whose BF16 weight is an exact MXFP8 dequantization to the MXFP8 GEMV for 2-D + contiguous bf16 inputs of the given row counts (the decode passes'); compiles those kernels now (outside capture). + Returns how many projections were routed.""" + rows = sorted({int(m) for m in rows if 0 < int(m) <= _gemv.MAX_TOKENS}) + n_routed = 0 + for lin in _projections(draft_model): + weight = getattr(lin, "weight", None) + if ( + weight is None + or getattr(lin, "bias", None) is not None + or getattr(lin, "_m31_mxdraft", None) is not None + ): + continue + rec = mxfp8_from_bf16(weight.data) + if rec is None: + continue + n, k = weight.shape + served = tuple(m for m in rows if _gemv.serves(n, k, m, True)) + if not served: + continue + w8, sf = rec[0], swizzle_sf(rec[1]) + for m in served: + _gemv.precompile(w8, sf, m, True) + original = lin.apply_linear + + def apply_linear( + input, + bias, + lora_params=None, + layer_idx=None, + *, + _o=original, + _w8=w8, + _sf=sf, + _served=served, + ): + if ( + bias is None + and not lora_params + and isinstance(input, torch.Tensor) + and input.dtype == torch.bfloat16 + and input.dim() == 2 + and input.shape[0] in _served + and input.is_contiguous() + ): + out = _gemv.m3_mx_gemv_bf16(input, _w8, _sf) + if out is not None: + m3_report.work( + "MXDRAFT", f"{input.shape[0]} rows [{_w8.shape[0]}, {_w8.shape[1]}]" + ) + return out + return _o(input, bias, lora_params, layer_idx) + + lin._m31_mxdraft = (w8, sf, served) + lin.apply_linear = apply_linear + n_routed += 1 + return n_routed diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/__init__.py new file mode 100644 index 000000000000..1059445198db --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Ops used only by this target. Shared ops stay in the catalog.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/__init__.py new file mode 100644 index 000000000000..b8c7426e3933 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Activation ops of this target: the gated MLPs' fused SwiGLU.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/m3_silu_and_mul.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/m3_silu_and_mul.py new file mode 100644 index 000000000000..5bcdee3fbb82 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/m3_silu_and_mul.py @@ -0,0 +1,83 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The gated MLPs' fused SwiGLU: ``silu_and_mul`` with MiniMax-M3's SwiGLU-OAI gain, offset and clamp.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SiluAndMul(OpWrapper): + """``out = g * sigmoid(alpha * g) * (u + beta)`` over ``x = [g | u]`` (bf16 [M, 2I] -> [M, I]), with + ``g = min(gate, limit)`` and ``u = clamp(up, -limit, limit)``; ``limit``, ``alpha`` and ``beta`` None give plain + SwiGLU ``silu(gate) * up``. + + The MXFP8 dense MLPs (layers 0-2, intermediate 3072 per rank) and the MoE's shared expert (intermediate 768 per + rank) run it at MiniMax-M3's (limit 7, alpha 1.702, beta 1); the BF16 Eagle3 drafter's MLP (intermediate 4608 per + rank) runs it plain. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert, one decode token", spec=dict(m=1, inter=768, oai=True)), + Cell(why="shared expert, a 48-row decode verify", spec=dict(m=48, inter=768, oai=True)), + Cell(why="dense MLP, a prefill chunk", spec=dict(m=2048, inter=3072, oai=True)), + Cell(why="dense MLP, one decode token", spec=dict(m=1, inter=3072, oai=True)), + Cell( + why="drafter MLP (plain SwiGLU), eight decode rows", + spec=dict(m=8, inter=4608, oai=False), + ), + Cell( + why="drafter MLP (plain SwiGLU), a prefill chunk", + spec=dict(m=1024, inter=4608, oai=False), + ), + ) + + note = """ + The kernel computes in fp32 and rounds once to bf16; `reference` does the same arithmetic in fp32, so the two + differ only by the fast sigmoid's last bits (the test's band). + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + + def reference( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + gate, up = x.float().chunk(2, dim=-1) + if limit is not None: + gate = gate.clamp(max=limit) + up = up.clamp(min=-limit, max=limit) + a = 1.0 if alpha is None else alpha + b = 0.0 if beta is None else beta + return (gate * torch.sigmoid(a * gate) * (up + b)).to(x.dtype) + + def is_valid( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> None: + assert x.stride(-1) == 1, "the gate/up halves are read from contiguous rows" + + +m3_silu_and_mul = M3SiluAndMul() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/m3_swq.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/m3_swq.py new file mode 100644 index 000000000000..b3b6adadfa27 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/activation/m3_swq.py @@ -0,0 +1,170 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SWQ: the shared expert's SwiGLU-OAI written directly as the down projection's MXFP8 activation.""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +@triton.jit +def _rcp_approx_ftz(x): + return tl.inline_asm_elementwise( + "rcp.approx.ftz.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _swq_kernel( + x_ptr, + x_stride, + q_ptr, + sf_ptr, + M, + INTER: tl.constexpr, + NB: tl.constexpr, + NB_PAD: tl.constexpr, + P2: tl.constexpr, + swiglu_limit: tl.constexpr, + swiglu_alpha: tl.constexpr, + swiglu_beta: tl.constexpr, + HAS_LIMIT: tl.constexpr, +): + # One program per row of the 128-row-padded scale layout; rows past M only zero their scale bytes. + r = tl.program_id(0) + blk = tl.arange(0, P2) + blk_ok = blk < NB_PAD + # The 128x4 swizzle of trtllm::mxfp8_quantize(x, True) (quantization.cuh get_sf_out_offset_128x4). + sf_off = ( + (blk % 4) + + (blk // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * NB_PAD) + ) + if r >= M: + tl.store(sf_ptr + sf_off, tl.zeros([P2], dtype=tl.uint8), mask=blk_ok) + return + cols = tl.arange(0, P2 * 32) + valid = cols < INTER + row = x_ptr + r.to(tl.int64) * x_stride + # silu_and_mul_kernel's arithmetic (modules/swiglu.py), then its bf16 store. + a = tl.load(row + cols, mask=valid, other=0.0).to(tl.float32) + b = tl.load(row + cols + INTER, mask=valid, other=0.0).to(tl.float32) + if HAS_LIMIT: + a = tl.minimum(a, swiglu_limit) + b = tl.clamp(b, -swiglu_limit, swiglu_limit) + v = (a * tl.sigmoid(swiglu_alpha * a) * (b + swiglu_beta)).to(tl.bfloat16).to(tl.float32) + # cvt_warp_fp16_to_mxfp8: SF = amax * rcp.approx(448), UE8M0 toward +inf, scale = rcp.approx(SF), E4M3 RN. + v2 = tl.reshape(v, (P2, 32)) + amax = tl.max(tl.abs(v2), axis=1) + sf = amax * _rcp_approx_ftz(tl.full([P2], 448.0, tl.float32)) + bits = sf.to(tl.int32, bitcast=True) + exp = (bits >> 23) & 0xFF + mant = bits & 0x7FFFFF + e = tl.where(exp > 0, exp + (mant != 0).to(tl.int32), tl.where(bits > 0x400000, 1, 0)) + e = tl.minimum(e, 254) + # At code 0 the scale 2^-127 is a float subnormal, so rcp.approx.ftz returns inf (native publishes + # inf/NaN there, found by a tiny-amax oracle). Use its exact reciprocal 2^127 instead; every other code keeps + # native's rcp.approx (exact on a normal power of two), so each block native emits finite stays bitwise. + pinv = tl.where( + e == 0, 1.7014118346046923e38, _rcp_approx_ftz((e << 23).to(tl.float32, bitcast=True)) + ) + inv = tl.where(amax != 0.0, pinv, 0.0) + q = tl.reshape((v2 * inv[:, None]).to(tl.float8e4nv), (P2 * 32,)) + tl.store(q_ptr + r.to(tl.int64) * INTER + cols, q, mask=valid) + tl.store(sf_ptr + sf_off, tl.where(blk < NB, e, 0).to(tl.uint8), mask=blk_ok) + + +class M3Swq(OpWrapper): + """``silu_and_mul`` (SwiGLU-OAI) over ``x = [g | u]`` (bf16 [M, 2I]) rounded to bf16, then MXFP8-quantized as + ``trtllm::mxfp8_quantize(h, True)``: (E4M3 [M, I], UE8M0 scales in the 128x4-swizzled layout), bit for bit, in one + launch. The TP8 shared expert's down projection (intermediate 384, no GEMV row table) reads it at ``SWQ_ROWS``.""" + + ARCHS = frozenset({Arch.SM_107}) + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert, the smallest row count it serves", spec=dict(m=9, inter=384)), + Cell( + why="shared expert, a 32-row decode verify (c8; the widest R1-VR-TP8 step)", + spec=dict(m=32, inter=384), + ), + Cell( + why="TP2 shared expert (width 1536), the first row past its down GEMV (17)", + spec=dict(m=17, inter=1536), + ), + Cell( + why="TP2 shared expert (width 1536), the K7 c4 step (32 rows)", + spec=dict(m=32, inter=1536), + ), + ) + + note = """ + Bitwise to the two-kernel path (silu_and_mul's fp32 arithmetic and bf16 rounding, then mxfp8_quantize's + rcp.approx scale rule): q bytes and every scale byte, padding included (checked on GB300 and VR). + One deliberate exception: a block whose scale code is 0 (amax / 448 <= 2^-127), where native's + rcp.approx.ftz of the subnormal scale is inf and its e4m3 bytes encode inf/NaN, is quantized with the exact + reciprocal 2^127 instead, so it stays finite; its scale byte is native's. + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> tuple[torch.Tensor, torch.Tensor]: + m, i = x.shape[0], x.shape[1] // 2 + nb = i // 32 + nb_pad = (nb + 3) // 4 * 4 + q = torch.empty(m, i, dtype=torch.float8_e4m3fn, device=x.device) + sf = torch.empty((m + 127) // 128 * 128 * nb_pad, dtype=torch.uint8, device=x.device) + _swq_kernel[((m + 127) // 128 * 128,)]( + x, + x.stride(0), + q, + sf, + m, + INTER=i, + NB=nb, + NB_PAD=nb_pad, + P2=triton.next_power_of_2(nb_pad), + swiglu_limit=limit or 0.0, + swiglu_alpha=alpha if alpha is not None else 1.0, + swiglu_beta=beta if beta is not None else 0.0, + HAS_LIMIT=limit is not None and limit > 0.0, + ) + return q, sf + + def reference( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> tuple[torch.Tensor, torch.Tensor]: + """The two kernels it replaces (bitwise is the contract).""" + h = torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + q, sf = torch.ops.trtllm.mxfp8_quantize(h, True) + return q.view(h.shape), sf + + def is_valid( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> None: + assert x.dtype == torch.bfloat16 and x.stride(-1) == 1, "bf16 [M, 2I] with contiguous rows" + assert (x.shape[1] // 2) % 32 == 0, "I must be a multiple of the 32-element MXFP8 block" + + +m3_swq = M3Swq() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/__init__.py new file mode 100644 index 000000000000..a7f7fee7a234 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Attention ops of this target: MSA block selection, the sparse decode and the KV-length update.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_dense_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_dense_decode.py new file mode 100644 index 000000000000..ba008f493b94 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_dense_decode.py @@ -0,0 +1,190 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' decode attention (trtllm-gen over the full page table) with the KV split this target sizes per +request count (``modeling.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH``).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.trtllm_gen_dense_decode import ( # noqa: E501 + minimax_m3_trtllm_gen_dense_decode, +) + +__all__ = ["m3_dense_decode"] + +#: The TP4 dense layers' geometry: 16 query heads and one KV head of dim 128 per rank, an FP8 KV pool. +HEADS = 16 +KV_HEADS = 1 +HEAD_DIM = 128 +#: The split this target hands trtllm-gen at one request (modeling.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH), and a model +#: maximum the in-tree step passes otherwise. +CAPPED = 16384 +MODEL_MAX = 1 << 20 + + +def dense_attention_reference( + q: torch.Tensor, + keys: torch.Tensor, + values: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, +) -> torch.Tensor: + """fp32 causal decode attention: query row ``b * qlen + i`` of request b attends its first + ``seq_lens[b] - qlen + i + 1`` keys. ``keys`` / ``values``: [batch, kv_heads, max_len, head_dim] (the request's + tokens in order). Returns fp32 [total_q, heads, head_dim].""" + batch, kv_heads, max_len, _ = keys.shape + heads = q.shape[1] + group = heads // kv_heads + out = torch.zeros(q.shape, dtype=torch.float32, device=q.device) + pos = torch.arange(max_len, device=q.device) + with true_fp32_matmul(): + for b in range(batch): + full = int(seq_lens[b]) + rows = q[b * decode_query_len : (b + 1) * decode_query_len].float() # [qlen, heads, d] + limit = full - decode_query_len + 1 + torch.arange(decode_query_len, device=q.device) + valid = pos[None, :] < limit[:, None] # [qlen, max_len] + for h in range(kv_heads): + qh = rows[:, h * group : (h + 1) * group] # [qlen, group, d] + logits = torch.einsum("qgd,kd->qgk", qh, keys[b, h].float()) * sm_scale + logits = logits.masked_fill(~valid[:, None, :], float("-inf")) + probs = torch.softmax(logits, dim=-1) + out[ + b * decode_query_len : (b + 1) * decode_query_len, h * group : (h + 1) * group + ] = torch.einsum("qgk,kd->qgd", probs, values[b, h].float()) + return out + + +def gather_paged_kv( + kv_cache_manager, layer_idx: int, block_table: torch.Tensor, seq_lens: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor]: + """The K and V of each request in token order, [batch, kv_heads, max_len, head_dim], from the flat sub-page pool + (``get_kv_subpage_pool``: K of slot s at sub-page s * subpages_per_slot, V one sub-page later).""" + pool, subpages = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + page = pool.shape[2] + pages = (int(seq_lens.max()) + page - 1) // page + slots = block_table[:, :pages].long() + k = pool[slots * subpages] # [batch, pages, kv_heads, page, d] + v = pool[slots * subpages + 1] + batch, _, kv_heads, _, d = k.shape + + def order(t): + return t.permute(0, 2, 1, 3, 4).reshape(batch, kv_heads, pages * page, d) + + return order(k), order(v) + + +class M3DenseDecode(OpWrapper): + """``minimax_m3_trtllm_gen_dense_decode`` as ``MsaDecodeFmha`` calls it for a dense layer, with the max_seq_len + ``msa_decode.dense_decode_max_seq_len`` resolves: this target's per-request-count cap, else the step's bound. + + Decode attention of every query row over its request's full causal context in the paged FP8 pool, written into + ``output`` (BF16 [T, heads, head_dim]) in place. ``max_seq_len`` only sizes trtllm-gen's multi-CTA KV split: + every CTA still covers its share of each request's actual length, which may exceed it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over a 20000-token context, the split sized for 16384", + spec=dict(kv_lens=(20000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request, one row, over a 65536-token context (an AgentX-length cache) at the capped split", + spec=dict(kv_lens=(65536,), qlen=1, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request verifying 1 + 3 rows over a 3000-token context: the cap above the length", + spec=dict(kv_lens=(3000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="two requests keep the model maximum (no cap at two requests)", + spec=dict(kv_lens=(20000, 5000), qlen=4, max_seq_len=MODEL_MAX, capped=False), + ), + ) + + note = """ + The split only reorders the merge of the CTAs' partial softmax sums, so the capped split's output is the uncapped + one's to a few BF16 ulps (the test holds it there), and both are held to the fp32 reference within the FP8-KV + band of the in-tree dense decode test. + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + minimax_m3_trtllm_gen_dense_decode( + q, + kv_cache_manager, + layer_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + max_seq_len=max_seq_len, + max_num_requests=max_num_requests, + ) + return output + + def reference( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + """fp32 attention over the pool's K / V (q rounded to the pool's dtype first, as the kernel runs it). + + ``output``, ``max_seq_len`` and ``max_num_requests`` do not change the result.""" + keys, values = gather_paged_kv(kv_cache_manager, layer_idx, block_table, seq_lens) + if keys.dtype == torch.float8_e4m3fn: + q = fp8_e4m3_saturate(q) + return dense_attention_reference(q, keys, values, seq_lens, decode_query_len, sm_scale) + + def is_valid( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> None: + # The multi-CTA KV counters are sized for max_num_requests: more requests overrun them. + assert seq_lens.shape[0] <= max_num_requests, "more requests than the counters cover" + # Each verify row attends a causal prefix of at least one key. + assert bool((seq_lens >= decode_query_len).all()), "a request shorter than its query rows" + + +m3_dense_decode = M3DenseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_dense_qk_norm_rope_fp8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_dense_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..d9e8ca93fff7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_dense_qk_norm_rope_fp8.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' per-head RMSNorm + partial NeoX RoPE of the packed [Q | K | V] into FP8, the bits of the BF16 +``fused_qk_norm_rope`` output cast to E4M3; optionally also storing the live tokens' K / V into the paged FP8 cache.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3DenseQkNormRopeFp8(OpWrapper): + """``qkv`` [T, (q + 2 kv) heads x 128] bf16 -> [T, same] E4M3: Q and K heads normed and roped in fp32, rounded to + BF16 and then to E4M3 (saturating), V cast; ``qkv`` untouched + (``trtllm::minimax_m3_sm107tp4_dense_qk_norm_rope_fp8``). + + With ``kv_buffers`` (the layer's HND FP8 pool [pages, 2, kv heads, 128, 128]), ``out_cache_loc`` (the step's int32 + cache slots) and ``num_live_tokens``, the kernel also stores the live tokens' K and V rows into the pool (the bytes + the MSA backend's KV write stores) and every row's Q into ``q_compact`` [T, q heads, 128] + (``..._dense_qk_norm_rope_fp8_kv``). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, tp4 heads (16 q, 1 kv)", spec=dict(m=1, q=16, kv=1)), + Cell( + why="a 4-request verify (16 rows) storing K / V", + spec=dict(m=16, q=16, kv=1, kv_write=True), + ), + Cell(why="a 48-row decode verify", spec=dict(m=48, q=16, kv=1)), + Cell(why="a prefill chunk on an FP8 cache", spec=dict(m=2048, q=16, kv=1)), + ) + + note = """ + Bit-exact: `reference` runs the catalog's in-place BF16 ``fused_qk_norm_rope`` on a copy and casts every element to + E4M3, which is this kernel's rounding sequence (fp32 -> BF16 -> E4M3) for Q and K and the cast for V. + """ + + def __call__( + self, + qkv: torch.Tensor, + num_heads_q: int, + num_heads_k: int, + num_heads_v: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + is_neox: bool, + position_ids: torch.Tensor, + use_gemma: bool, + kv_buffers: Optional[torch.Tensor] = None, + out_cache_loc: Optional[torch.Tensor] = None, + num_live_tokens: int = 0, + q_compact: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + args = ( + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + use_gemma, + ) + if kv_buffers is not None: + return torch.ops.trtllm.minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv( + *args, kv_buffers, out_cache_loc, num_live_tokens, q_compact + ) + return torch.ops.trtllm.minimax_m3_sm107tp4_dense_qk_norm_rope_fp8(*args) + + def reference( + self, + qkv: torch.Tensor, + num_heads_q: int, + num_heads_k: int, + num_heads_v: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + is_neox: bool, + position_ids: torch.Tensor, + use_gemma: bool, + kv_buffers: Optional[torch.Tensor] = None, + out_cache_loc: Optional[torch.Tensor] = None, + num_live_tokens: int = 0, + q_compact: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """The FP8 copy (the K / V store into ``kv_buffers`` is checked by the test against the reference rows).""" + head_dim = qkv.shape[-1] // (num_heads_q + num_heads_k + num_heads_v) + out = qkv.clone() + fused_qk_norm_rope( + out, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + 1.0, + 0.0, + 0.0, + 1.0, + True, + use_gemma, + False, + 0, + 0, + ) + return out.to(torch.float8_e4m3fn) + + +m3_dense_qk_norm_rope_fp8 = M3DenseQkNormRopeFp8() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_draft_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_draft_attention.py new file mode 100644 index 000000000000..ed4e3048d780 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_draft_attention.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter layer's attention on the TRTLLM backend: RoPE, KV-cache append and causal attention.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + PredefinedAttentionMask, +) +from tensorrt_llm.functional import PositionEmbeddingType, RotaryScalingType + + +class M3DraftAttention(OpWrapper): + """The drafter layer's attention over its fused QKV rows ``qkv`` [T, (q + 2 kv) heads x 128] (bf16): the + backend applies the drafter's RoPE (fused), appends K / V to the drafter layer's paged cache (FP8, the target's + NVFP4 cache keeps its draft layer in FP8) and runs causal attention over the step's live rows (``metadata`` + ``num_tokens``); returns [num_tokens, q heads x 128] bf16. + + ``backend`` (``TrtllmAttention``) and ``metadata`` are the runtime's objects for the drafter layer and step. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="first draft pass of a 4-request verify (4 x 4 rows)", spec=dict(reqs=4, qlen=4)), + Cell(why="a later draft pass, one row per request", spec=dict(reqs=8, qlen=1)), + Cell(why="a context chunk's draft rows", spec=dict(reqs=1, context=512)), + ) + + note = """ + `reference` is exact (float64) causal softmax attention: Q rotated by the backend's NeoX RoPE at each token's + position, over the drafter layer's K / V rows read back from its FP8 cache (dequantized by the backend's + ``kv_scale_quant_orig``). What remains between the two is the FMHA's own arithmetic (Q's rounding to the + kernel's input precision, its P rounding and accumulation order), which the test bounds. + """ + + def __call__(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + num_tokens = metadata.num_tokens + out = backend.forward( + qkv[:num_tokens, :], + None, + None, + metadata, + forward_args=AttentionForwardArgs(attention_mask=PredefinedAttentionMask.CAUSAL), + ) + return out[0] if isinstance(out, tuple) else out + + def reference(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + """The call's [num_tokens, q heads x 128] output, in ``qkv``'s dtype. + + Query token ``j`` of request ``r`` sits at position ``cached_r + j`` (the metadata's cached lengths) and + attends positions up to it. Its Q is rotated here; every K / V row, the step's own included, is read from + the drafter layer's paged cache at the slots the metadata's KV cache manager holds, so this is evaluated + after the call (whose roped, quantized append it then reads back; the test checks that append itself). + Mirrors one KV head per rank and un-scaled NeoX RoPE, the drafter's configuration. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + assert backend.num_kv_heads == 1, "mirrors the one-KV-head-per-rank layout only" + rope = backend.rope_params + assert backend.position_embedding_type == int(PositionEmbeddingType.rope_gpt_neox) + assert rope.scale_type == RotaryScalingType.none and rope.alpha == 1.0 + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + kv_scale = float(backend.kv_scale_quant_orig.reshape(-1)[0]) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = int(metadata.num_tokens) + assert sum(qlens) == total + device = qkv.device + + rot = int(rope.dim) + inv_freq = float(rope.theta) ** ( + -torch.arange(0, rot, 2, dtype=torch.float64, device=device) / rot + ) + pool = manager.get_buffers(layer, "HND") + q = qkv[:total, : heads * dim].to(torch.float64).reshape(total, heads, dim) + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][ + : (length + page - 1) // page + ], + dtype=torch.int64, + device=device, + ) + keys = pool[pages, 0, 0].to(torch.float64).reshape(-1, dim)[:length] * kv_scale + values = pool[pages, 1, 0].to(torch.float64).reshape(-1, dim)[:length] * kv_scale + qpos = cached[row] + torch.arange(n, device=device) + angles = qpos.to(torch.float64)[:, None] * inv_freq[None, :] + cos, sin = angles.cos()[:, None, :], angles.sin()[:, None, :] + rows = q[first : first + n] + x1, x2 = rows[..., : rot // 2], rows[..., rot // 2 : rot] + rows = torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin, rows[..., rot:]], dim=-1) + allowed = torch.arange(length, device=device)[None, :] <= qpos[:, None] + logits = torch.einsum("thd,kd->thk", rows, keys) * sm_scale + logits = logits.masked_fill(~allowed[:, None, :], float("-inf")) + out[first : first + n] = torch.softmax(logits, dim=-1) @ values + first += n + return out.reshape(total, heads * dim).to(qkv.dtype) + + +m3_draft_attention = M3DraftAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_fp8_indexer_qk_norm_rope.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_fp8_indexer_qk_norm_rope.py new file mode 100644 index 000000000000..48543be75675 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_fp8_indexer_qk_norm_rope.py @@ -0,0 +1,114 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The unfused sparse layers' index branch: per-head Gemma RMSNorm + partial NeoX RoPE of [index-Q | index-K], index-Q +out in FP8 E4M3 and index-K inserted into the FP8 index-K cache, in one kernel.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3Fp8IndexerQkNormRope(OpWrapper): + """``qk`` [T, (heads + 1) * 128] BF16 (the index projection's output): returns index-Q [T, heads, 128] E4M3 and + writes index-K (E4M3) at ``out_cache_loc`` of ``index_k_cache`` [pages, 1, 128, 128]. Gemma RMSNorm of each head, + then RoPE on its leading ``rotary_dim`` channels (NeoX). + + The unfused sparse layers (``fuse_qkv_index_projection=False``) run it on the index branch with an FP8 indexer; + the fused layers' producer writes the index cache itself. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, one index head per rank", spec=dict(m=1, heads=1)), + Cell(why="a 48-row decode verify", spec=dict(m=48, heads=1)), + Cell(why="a prefill chunk", spec=dict(m=2048, heads=1)), + ) + + note = """ + `reference` is the in-place fused norm + RoPE in BF16 followed by the E4M3 cast (the kernel rounds through BF16 + before the cast); its index-K is what the kernel writes to the cache. The two kernels may round a handful of + values differently: the test allows fewer than 0.1% of the bytes to differ, as the in-tree test does. + """ + + def __call__( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> torch.Tensor: + return torch.ops.trtllm.minimax_m3_fp8_indexer_qk_norm_rope( + qk, + index_k_cache, + out_cache_loc, + num_heads_q, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + position_ids, + ) + + def reference( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor]: + """(index-Q [T, heads, head_dim] E4M3, index-K [T, head_dim] E4M3): the catalog's in-place kernel on a copy, + then the E4M3 cast.""" + out = fused_qk_norm_rope( + qk.clone(), + num_heads_q, + 1, + 0, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + True, + position_ids, + 1.0, + 0.0, + 0.0, + 1.0, + True, + True, + False, + 0, + 0, + ) + q, k = out.split([num_heads_q * head_dim, head_dim], dim=-1) + return ( + q.reshape(q.shape[0], num_heads_q, head_dim).to(torch.float8_e4m3fn), + k.to(torch.float8_e4m3fn), + ) + + +m3_fp8_indexer_qk_norm_rope = M3Fp8IndexerQkNormRope() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_fullctx_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_fullctx_decode.py new file mode 100644 index 000000000000..ad3abcadaae9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_fullctx_decode.py @@ -0,0 +1,142 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' full-context decode attention at TP4 on Rubin (``kernels/m3_fullctx_decode``): PrimTS split-KV +decode at the recipe ``modeling.FULLCTX_DECODE_RECIPES`` pins per (query rows per request, requests).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) + +from ...kernels import m3_fullctx_decode as kernel +from .m3_dense_decode import dense_attention_reference, gather_paged_kv + +__all__ = ["m3_fullctx_decode"] + +HEADS = kernel.HEADS +KV_HEADS = kernel.KV_HEADS +HEAD_DIM = kernel.HEAD_DIM +#: The plan capacity (the per-request K/V length bound PrimTS sizes its policy for): dispatch's +#: FULLCTX_DECODE_MAX_KV_LEN, which the forward's prebuilt plans use. +MAX_KV_LEN = 1 << 21 + + +class M3FullCtxDecode(OpWrapper): + """Decode attention of every query row over its request's full causal context in the paged FP8 pool, written + into ``output`` (BF16 [T, heads, head_dim]) in place. + + ``recipe`` is ``modeling.FULLCTX_DECODE_RECIPES[(qlen, requests)]``: the PrimTS Q tile and the split-KV fanout. + ``own_reducer`` merges the splits on the BF16 op (``modeling.FULLCTX_DECODE_OWN_REDUCER``) rather than PrimTS's + FP16 reducer and a cast. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over an AgentX-length (165k) context", + spec=dict(kv_lens=(165525,), qlen=4), + ), + Cell( + why="one request, one draft-pass row, over a 165k context", + spec=dict(kv_lens=(165522,), qlen=1), + ), + Cell( + why="two requests verifying 1 + 3 rows at unequal lengths (165k, 120k)", + spec=dict(kv_lens=(165525, 120011), qlen=4), + ), + Cell( + why="two requests, one row each, at unequal lengths (165k, 120k)", + spec=dict(kv_lens=(165522, 120008), qlen=1), + ), + Cell( + why="one request verifying 1 + 3 rows over a short (4.1k) context: most splits are empty", + spec=dict(kv_lens=(4104,), qlen=4), + ), + ) + + note = """ + Same operands and arithmetic as the stock trtllm-gen call (E4M3 q, E4M3 K/V, P quantized to E4M3 per split, + fp32 accumulation): the split differs, so the result differs from stock as a stock split change does (the P + quantization is per split). Deterministic, and the same bits eager and in a CUDA graph. Held to the fp32 + reference within the FP8-KV band of the in-tree dense decode test. + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + recipe, + max_kv_len: int = MAX_KV_LEN, + own_reducer: bool = True, + ) -> torch.Tensor: + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + shape = (decode_query_len, int(seq_lens.shape[0])) + kernel.prebuild(q.device, {shape: recipe}, max_kv_len, own_reducer, get_env_enable_pdl()) + plan = kernel.plan_for(q.device, *shape, own_reducer) + pool, subpages = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + k_pages, v_pages = kernel.kv_page_views(pool, subpages) + plan.run(fp8_e4m3_saturate(q), k_pages, v_pages, block_table, seq_lens, sm_scale, output) + return output + + def reference( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + recipe, + max_kv_len: int = MAX_KV_LEN, + own_reducer: bool = True, + ) -> torch.Tensor: + """fp32 attention over the pool's K / V, q rounded to E4M3 first as the kernel runs it.""" + keys, values = gather_paged_kv(kv_cache_manager, layer_idx, block_table, seq_lens) + return dense_attention_reference( + fp8_e4m3_saturate(q), keys, values, seq_lens, decode_query_len, sm_scale + ) + + def is_valid( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + recipe, + max_kv_len: int = MAX_KV_LEN, + own_reducer: bool = True, + ) -> None: + pool, _ = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + assert pool.dtype == torch.float8_e4m3fn, "an FP8 E4M3 pool" + assert tuple(pool.shape[1:]) == (KV_HEADS, kernel.PAGE, HEAD_DIM), ( + "the TP4 dense-layer pool geometry" + ) + assert q.shape[1:] == (HEADS, HEAD_DIM), "16 query heads of 128 per rank" + assert int(q.shape[0]) == int(seq_lens.shape[0]) * decode_query_len, ( + "q covers exactly the batch" + ) + if torch.cuda.is_current_stream_capturing(): + return # the length checks read the device: eager calls only + assert bool((seq_lens >= decode_query_len).all()), "a request shorter than its query rows" + assert bool((seq_lens <= max_kv_len).all()), "a request longer than the plan capacity" + + +m3_fullctx_decode = M3FullCtxDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_index_score.py new file mode 100644 index 000000000000..f44d566310e7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_index_score.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# PyTorch oracle ported from vLLM (Apache-2.0), _reference_decode_index_score: +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/tests/kernels/attention/test_minimax_m3.py#L188 +"""The MSA index scorer of a sparse decode step: per (index head, query token, KV block) the causal max of Q . K over +the block's 128 index-K positions, with the pre-wait schedule of ``kernels/m3_index_score``.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_index_score"] + +PAGE = 128 +HEAD_DIM = 128 +#: CTAs per request (each scores blocks split, split + SPLIT_K, ...; the scores do not depend on it). +SPLIT_K = 512 + + +def compiled_scorer(key: tuple): + """The compiled kernel of ``key`` (dtype, heads, query length, head dim, split_k, early), compiling it on a miss + (the target compiles every key it can take before warmup: ``layers.prebuild_index_scorer``).""" + from ...kernels.m3_index_score import IndexScoreRunner + + if key not in IndexScoreRunner.kernel_cache: + IndexScoreRunner._compile(*key) + return IndexScoreRunner.kernel_cache[key] + + +class M3IndexScore(OpWrapper): + """``score[h, t, j]`` (fp32, a transposed view of the [heads, blocks, tokens] selector buffer) = the max over the + positions ``p`` of block ``j`` of request ``b = t // qlen`` with ``p <= seq_lens[b] - qlen + t % qlen`` of + ``idx_q[t, h] . index_k[block_table[b, j], p % 128]``, for ``j < ceil(seq_lens[b] / 128)`` (no other entry is + written). FP8 or BF16 q and index-K, fp32 products and sums. + + ``early`` (the target's one-request kernel) issues the Q and first K tiles right after the grid wait from a + pre-wait length hint and prefetches one older block per CTA into L2 before the wait; otherwise every older block of + the CTA is prefetched. Neither changes a score. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over 20000 tokens: the early-issue kernel", + spec=dict(seq_lens=(20000,), qlen=4, early=True), + ), + Cell( + why="one request, one row, 65536 tokens: early issue, 512 blocks on 512 CTAs", + spec=dict(seq_lens=(65536,), qlen=1, early=True), + ), + Cell( + why="one request of 129 tokens: the early tiles of most CTAs lie past the length and are drained", + spec=dict(seq_lens=(129,), qlen=4, early=True), + ), + Cell( + why="one request longer than 512 blocks (the CTAs loop): early issue", + spec=dict(seq_lens=(70000,), qlen=4, early=True), + ), + Cell( + why="two requests: the prefetching kernel", + spec=dict(seq_lens=(20000, 3000), qlen=4, early=False), + ), + Cell( + why="eight requests of mixed lengths: the prefetching kernel", + spec=dict( + seq_lens=(300, 1500, 2049, 4096, 5000, 7777, 8192, 33001), qlen=4, early=False + ), + ), + ) + + note = """ + The products of FP8 (or BF16) values are exact in fp32 and the kernel sums 128 of them in its k-split order, so it + differs from the reference matmul by a few fp32 ulps; it is held to the in-tree scorer bit for bit. + """ + + def __call__( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + _, num_heads, head_dim = idx_q.shape + key = (idx_q.dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + compiled_scorer(key)(idx_q, index_k, block_table, score, seq_lens) + return score + + def reference( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + """The fp32 [heads, total_q, blocks] scores (``-inf`` where nothing is written). ``score``, ``split_k`` and + ``early`` do not change them.""" + total_q, heads, _ = idx_q.shape + qlen = max_decode_query_len + out = torch.full( + (heads, total_q, score.shape[2]), + -float("inf"), + dtype=torch.float32, + device=idx_q.device, + ) + with true_fp32_matmul(): + for b, seq in enumerate(seq_lens.tolist()): + nblocks = (seq + PAGE - 1) // PAGE + if nblocks == 0: + continue + t0 = b * qlen + q = idx_q[t0 : t0 + qlen].float() + k = index_k[block_table[b, :nblocks].long()].reshape(nblocks * PAGE, -1).float() + s = torch.einsum("qhd,kd->hqk", q, k) + q_pos = seq - qlen + torch.arange(qlen, device=idx_q.device) + k_pos = torch.arange(k.shape[0], device=idx_q.device) + s.masked_fill_(k_pos[None, :] > q_pos[:, None], -float("inf")) + out[:, t0 : t0 + qlen, :nblocks] = s.reshape(heads, qlen, nblocks, PAGE).amax(dim=3) + return out + + def is_valid( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> None: + # The block-table row and the score row are read and written up to ceil(seq_len / 128) with no bound. + nblocks = (seq_lens + PAGE - 1) // PAGE + assert bool((nblocks <= block_table.shape[1]).all()), ( + "a request runs past its block-table row" + ) + assert bool((nblocks <= score.shape[2]).all()), "a request runs past the score buffer" + assert idx_q.shape[0] == seq_lens.shape[0] * max_decode_query_len, ( + "idx_q must hold max_decode_query_len rows per request" + ) + + +m3_index_score = M3IndexScore() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_index_score_t2sc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_index_score_t2sc.py new file mode 100644 index 000000000000..0c47017d8a87 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_index_score_t2sc.py @@ -0,0 +1,181 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# PyTorch oracle ported from vLLM (Apache-2.0), _reference_decode_index_score: +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/tests/kernels/attention/test_minimax_m3.py#L188 +"""The MSA index scorer of a sparse decode step: per (index head, query token, KV block) the causal max of Q . K over +the block's 128 index-K positions, with the pre-wait schedule of ``kernels/m3_index_score_t2sc`` (T2SC: the sm_107 +TP2 / TP8 scorer as this target's copy, at its 1 index head per rank; ``modeling.INDEX_SCORE_T2SC``).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_index_score_t2sc"] + +PAGE = 128 +HEAD_DIM = 128 + + +def compiled_scorer(key: tuple): + """The compiled kernel of ``key`` (dtype, heads, query length, head dim, split_k, early), compiling it on a miss + (the target compiles every key it can take before warmup: ``layers.prebuild_index_scorer_t2sc``).""" + from ...kernels.m3_index_score_t2sc import IndexScoreRunner + + if key not in IndexScoreRunner.kernel_cache: + IndexScoreRunner._compile(*key) + return IndexScoreRunner.kernel_cache[key] + + +class M3IndexScoreT2sc(OpWrapper): + """``score[h, t, j]`` (fp32, a transposed view of the [heads, blocks, tokens] selector buffer) = the max over the + positions ``p`` of block ``j`` of request ``b = t // qlen`` with ``p <= seq_lens[b] - qlen + t % qlen`` of + ``idx_q[t, h] . index_k[block_table[b, j], p % 128]``, for ``j < ceil(seq_lens[b] / 128)`` (no other entry is + written). FP8 or BF16 q and index-K, fp32 products and sums. + + ``early`` (the target's kernel at every request count it covers) issues the Q and first K tiles right after the + grid wait from a pre-wait length hint and prefetches one older block per CTA into L2 before the wait; otherwise + every older block of the CTA is prefetched. Neither, nor ``split_k``, changes a score. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request, 1 + 3 rows over 165521 tokens (AgentX ISL), 1 index head: 1536 CTAs, early issue", + spec=dict(seq_lens=(165521,), qlen=4, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request of 129 tokens: the early tiles of most of the 1536 CTAs lie past the length and drain", + spec=dict(seq_lens=(129,), qlen=4, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request of 400000 tokens: 3125 blocks, so the 1536 CTAs loop", + spec=dict(seq_lens=(400000,), qlen=4, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request decoding one row over 65536 tokens", + spec=dict(seq_lens=(65536,), qlen=1, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request, 1 + 3 rows over 165521 tokens, 1 index head (TP4): 1272 CTAs (one wave), early issue", + spec=dict(seq_lens=(165521,), qlen=4, heads=1, split_k=1272, early=True), + ), + Cell( + why="one request of 129 tokens at 1272 CTAs: the early tiles of most CTAs lie past the length and drain", + spec=dict(seq_lens=(129,), qlen=4, heads=1, split_k=1272, early=True), + ), + Cell( + why="one request of 400000 tokens at 1272 CTAs: 3125 blocks, so the CTAs loop", + spec=dict(seq_lens=(400000,), qlen=4, heads=1, split_k=1272, early=True), + ), + Cell( + why="two requests verifying 4 rows each over 165521 and 20000 tokens: 768 CTAs each, early issue", + spec=dict(seq_lens=(165521, 20000), qlen=4, heads=1, split_k=768, early=True), + ), + Cell( + why="two requests, one of 300 tokens and one of 262144 (2048 blocks: the 768 CTAs loop)", + spec=dict(seq_lens=(300, 262144), qlen=4, heads=1, split_k=768, early=True), + ), + # TP2 (MiniMax-M3.1 at 2 ranks): 2 index heads per rank, the K7 verify (1 + 7 rows) at the target's schedule. + Cell( + why="TP2, one request, 1 + 7 rows over 165521 tokens, 2 index heads: 1272 CTAs, early issue", + spec=dict(seq_lens=(165521,), qlen=8, heads=2, split_k=1272, early=True), + ), + Cell( + why="TP2, one request of 129 tokens, 2 index heads: most CTAs' early tiles lie past the length", + spec=dict(seq_lens=(129,), qlen=8, heads=2, split_k=1272, early=True), + ), + Cell( + why="TP2, two requests verifying 8 rows each over 165521 and 20000 tokens, 2 index heads: 768 CTAs", + spec=dict(seq_lens=(165521, 20000), qlen=8, heads=2, split_k=768, early=True), + ), + ) + + note = """ + The products of FP8 (or BF16) values are exact in fp32 and the kernel sums 128 of them in its k-split order, so it + differs from the reference matmul by a few fp32 ulps; it is held to the in-tree scorer bit for bit. + """ + + def __call__( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + _, num_heads, head_dim = idx_q.shape + key = (idx_q.dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + compiled_scorer(key)(idx_q, index_k, block_table, score, seq_lens) + return score + + def reference( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + """The fp32 [heads, total_q, blocks] scores (``-inf`` where nothing is written). ``score``, ``split_k`` and + ``early`` do not change them.""" + total_q, heads, _ = idx_q.shape + qlen = max_decode_query_len + out = torch.full( + (heads, total_q, score.shape[2]), + -float("inf"), + dtype=torch.float32, + device=idx_q.device, + ) + with true_fp32_matmul(): + for b, seq in enumerate(seq_lens.tolist()): + nblocks = (seq + PAGE - 1) // PAGE + if nblocks == 0: + continue + t0 = b * qlen + q = idx_q[t0 : t0 + qlen].float() + k = index_k[block_table[b, :nblocks].long()].reshape(nblocks * PAGE, -1).float() + s = torch.einsum("qhd,kd->hqk", q, k) + q_pos = seq - qlen + torch.arange(qlen, device=idx_q.device) + k_pos = torch.arange(k.shape[0], device=idx_q.device) + s.masked_fill_(k_pos[None, :] > q_pos[:, None], -float("inf")) + out[:, t0 : t0 + qlen, :nblocks] = s.reshape(heads, qlen, nblocks, PAGE).amax(dim=3) + return out + + def is_valid( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> None: + # The block-table row and the score row are read and written up to ceil(seq_len / 128) with no bound. + nblocks = (seq_lens + PAGE - 1) // PAGE + assert bool((nblocks <= block_table.shape[1]).all()), ( + "a request runs past its block-table row" + ) + assert bool((nblocks <= score.shape[2]).all()), "a request runs past the score buffer" + assert idx_q.shape[0] == seq_lens.shape[0] * max_decode_query_len, ( + "idx_q must hold max_decode_query_len rows per request" + ) + + +m3_index_score_t2sc = M3IndexScoreT2sc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_msa_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_msa_attention.py new file mode 100644 index 000000000000..0266dd804c17 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_msa_attention.py @@ -0,0 +1,249 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MiniMax-M3 MSA attention backend's step: KV write, sparse top-k block selection, FMHA.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + SparseBackendForwardArgs, +) + +from .m3_sparse_decode_lc import dequant_nvfp4_pages + + +class M3MsaAttention(OpWrapper): + """One layer's attention core on the MSA backend (``MiniMaxM3MsaSparseAttention``), written into ``output`` + [T, heads * 128] (bf16, the step's live rows). + + * ``k`` / ``v`` given (the dense layers): written into the layer's paged main cache first (NVFP4 with + ``kv_scale_orig_quant``, else FP8 / BF16 as the pool is). None: a producer already wrote the caches. + * ``sparse``: the indexer scores the layer's index-K cache (already written) with ``idx_q`` and publishes the + top-k blocks; the FMHA attends those blocks. Otherwise the FMHA attends the whole page table. + + The FMHA dispatches on the step's metadata (context rows: the MSA prefill FMHA; generation rows: the MSA decode + FMHA, the dense layers' trtllm-gen decode at their capped KV split). ``backend`` and ``metadata`` are the runtime's + objects for this layer and step. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="dense layer, a prefill chunk, FP8 pool", + spec=dict(layer="dense", phase="context", tokens=1024), + ), + Cell( + why="dense layer, a decode step of 4 requests", + spec=dict(layer="dense", phase="generation", reqs=4), + ), + Cell( + why="sparse layer, a prefill chunk beyond 16 blocks, NVFP4 pool", + spec=dict(layer="sparse", phase="context", tokens=4096), + ), + Cell( + why="sparse layer, a mixed step (context + generation rows)", + spec=dict(layer="sparse", phase="mixed", tokens=512, reqs=2), + ), + ) + + note = """ + `reference` is exact (float64) softmax attention over the cache rows the step attends, read back from the paged + cache the call wrote (or a producer wrote before it) and dequantized as the FMHA dequantizes them: FP8 rows as + E4M3, NVFP4 rows as E2M1 x E4M3 block scale x the per-tensor ``kv_scale_quant_orig``. A sparse layer's top-k blocks + are re-selected from ``idx_q`` and the index-K cache with the backend's rule (per-block max over the causal + keys, the query's own block forced in, ties to the lower block). On an FP8 pool the FMHA takes Q as a plain E4M3 + cast, which the reference applies too. What remains between the two is the FMHA's own arithmetic (its P + rounding and accumulation order), which the test bounds. + """ + + def __call__( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + if k is not None: + backend.write_layer_caches( + k, v, None, metadata, kv_scale_orig_quant=kv_scale_orig_quant + ) + if sparse: + blocks = backend.run_indexer(idx_q, None, metadata, idx_k_prewritten=True) + args = AttentionForwardArgs( + output=output, sparse_backend_args=SparseBackendForwardArgs(topk_indices=blocks) + ) + else: + args = AttentionForwardArgs(output=output) + args.kv_scale_orig_quant = kv_scale_orig_quant + args.kv_scale_quant_orig = kv_scale_quant_orig + backend.forward(q, None, None, metadata, forward_args=args) + return output + + def reference( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + """``output`` as the call is supposed to write it, [T, heads * 128] in ``output``'s dtype. + + Every attended row -- the step's own included -- is read from the layer's paged cache, at the slots the + metadata's KV cache manager holds for each request, so this is evaluated after the call (whose write of + ``k`` / ``v`` it then reads back). ``k``, ``v`` and ``kv_scale_orig_quant`` are therefore not read; the + test checks the write itself. Query token ``j`` of request ``r`` sits at position ``cached_r + j`` and + attends positions up to it; on a sparse layer only those in its top-k blocks. Mirrors grouped KV heads (query + head h reads KV head h // (heads / kv heads); two per rank at tp2: 32 query heads, 2 index heads), each KV head + selecting its own top-k blocks from its own index heads against the rank's single index-K head. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + kv_heads = int(backend.num_kv_heads) + assert heads % kv_heads == 0 + group = heads // kv_heads + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = sum(qlens) + assert q.shape[0] == total and output.shape[0] == total + + pool = manager.get_buffers(layer, "HND") + nvfp4 = manager.is_nvfp4_layer(layer) + if nvfp4: + scale_pool = manager.get_block_scale_buffers(layer, "HND") + k_global = float(kv_scale_quant_orig.reshape(-1)[1]) + v_global = float(kv_scale_quant_orig.reshape(-1)[2]) + q_rows = q.reshape(total, heads, dim) + if q_rows.dtype != torch.float8_e4m3fn and pool.dtype == torch.float8_e4m3fn: + # The FP8 FMHA shares one dtype across Q / K / V: Q is cast to E4M3, unscaled. + q_rows = q_rows.to(torch.float8_e4m3fn) + q64 = q_rows.to(torch.float64) + if sparse: + config = backend.m3_config + idx_cache = metadata.msa_idx_k_cache(layer) + idx64 = idx_q.reshape(total, config.num_index_heads, config.sparse_index_dim).to( + torch.float64 + ) + assert config.num_index_heads % kv_heads == 0 + idx_group = config.num_index_heads // kv_heads + + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=q.device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + blocks = (length + page - 1) // page + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][:blocks], + dtype=torch.int64, + device=q.device, + ) + qpos = cached[row] + torch.arange(n, device=q.device) + causal = torch.arange(length, device=q.device)[None, :] <= qpos[:, None] + for g in range(kv_heads): + if nvfp4: + keys = dequant_nvfp4_pages( + pool[:, 0].view(torch.uint8), + scale_pool[:, 0], + pages, + quad_scales=False, + kv_head=g, + ) + values = dequant_nvfp4_pages( + pool[:, 1].view(torch.uint8), + scale_pool[:, 1], + pages, + quad_scales=True, + kv_head=g, + ) + keys, values = keys * k_global, values * v_global + else: + keys = pool[pages, 0, g].to(torch.float64) + values = pool[pages, 1, g].to(torch.float64) + keys = keys.reshape(-1, dim)[:length] + values = values.reshape(-1, dim)[:length] + allowed = causal + if sparse: + # One index-K head per rank, shared by every index head (the cache is [pages, 1, page, dim]). + idx_keys = idx_cache[pages, 0].to(torch.float64).reshape(-1, dim)[:length] + chosen = _selected_blocks( + idx64[first : first + n, g * idx_group : (g + 1) * idx_group], + idx_keys, + causal, + qpos, + page, + config, + ) + allowed = causal & chosen.repeat_interleave(page, dim=1)[:, :length] + h0, h1 = g * group, (g + 1) * group + for lo in range(0, n, 256): + hi = min(n, lo + 256) + logits = ( + torch.einsum("thd,kd->thk", q64[first + lo : first + hi, h0:h1], keys) + * sm_scale + ) + logits = logits.masked_fill(~allowed[lo:hi, None, :], float("-inf")) + out[first + lo : first + hi, h0:h1] = torch.softmax(logits, dim=-1) @ values + first += n + return out.reshape(total, heads * dim).to(output.dtype) + + +def _selected_blocks( + idx_q: torch.Tensor, + idx_keys: torch.Tensor, + allowed: torch.Tensor, + qpos: torch.Tensor, + page: int, + config, +) -> torch.Tensor: + """The blocks each query token of one request attends, [n, blocks] bool. + + A block's score is the maximum, over the KV head's index heads and the block's causally visible keys, of + ``idx_q . idx_k``. Of the token's ``ceil((pos + 1) / page)`` valid blocks the first ``init_blocks`` and the last + ``local_blocks`` are forced in, and the ``topk`` highest-scoring are kept, ties to the lower block (the + backend's ``select_blocks_from_maxscore`` rule). The positive score scale the scorers apply does not change the + ranking, so none is applied. + """ + n, length = allowed.shape + blocks = (length + page - 1) // page + scores = torch.einsum("thd,kd->thk", idx_q, idx_keys).amax(dim=1) + scores = scores.masked_fill(~allowed, float("-inf")) + padded = torch.full((n, blocks * page), float("-inf"), dtype=scores.dtype, device=scores.device) + padded[:, :length] = scores + block_scores = padded.view(n, blocks, page).amax(dim=-1) + ids = torch.arange(blocks, device=scores.device)[None, :] + n_valid = torch.div(qpos + page, page, rounding_mode="floor")[:, None] + block_scores = block_scores.masked_fill(ids >= n_valid, float("-inf")) + forced = (ids < torch.clamp(n_valid, max=config.init_blocks)) | ( + (ids >= n_valid - config.local_blocks) & (ids < n_valid) + ) + block_scores = block_scores.masked_fill(forced, float("inf")) + order = torch.sort(block_scores, dim=1, descending=True, stable=True).indices[:, : config.topk] + keep = torch.arange(order.shape[1], device=scores.device)[None, :] < n_valid + chosen = torch.zeros(n, blocks, dtype=torch.bool, device=scores.device) + rows = torch.arange(n, device=scores.device)[:, None].expand_as(order) + chosen[rows[keep], order[keep]] = True + return chosen + + +m3_msa_attention = M3MsaAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_producer.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_producer.py new file mode 100644 index 000000000000..ff9a05c0233b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_producer.py @@ -0,0 +1,216 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' NVFP4 QKV + indexer RMSNorm / RoPE / KV-insert producer at the TP2 heads, with its constant loads +staged before the PDL grid-dependency wait and an early dependents trigger +(``kernels/minimaxM3Sm107Tp2/producerPw2.cu``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +__all__ = ["m3_producer"] + +HEAD_DIM = 128 +ROTARY_DIM = 64 +PAGE = 128 +#: Heads per rank at TP2 (MiniMax-M3: 64 query, 4 KV and 4 index heads in all; one index-K head per token). +HEADS_Q = 32 +HEADS_KV = 2 +HEADS_INDEX = 2 +#: Packed heads per token: q, k, v, index-q and the index-k head. +HEADS_PACKED = HEADS_Q + 2 * HEADS_KV + HEADS_INDEX + 1 +#: NVFP4 block: 16 values share one E4M3 scale. +FP4_BLOCK = 16 +E2M1_VALUES = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0) + + +def _norm_rope( + x: torch.Tensor, weight: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor, eps: float +): + """Gemma RMSNorm (``x * rsqrt(mean(x^2) + eps) * (1 + w)``) then NeoX RoPE on the first 64 dims, in fp32. + x [T, 128] fp32; cos / sin [T, 32].""" + y = x * torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) * (1.0 + weight.float()) + lo, hi, rest = y[:, :32], y[:, 32:64], y[:, 64:] + return torch.cat([lo * cos - hi * sin, hi * cos + lo * sin, rest], dim=-1) + + +def decode_nvfp4_token( + data: torch.Tensor, scales: torch.Tensor, quant_scale: float +) -> torch.Tensor: + """fp32 [128] of one token's packed E2M1 row (64 bytes, low nibble first) and its 8 E4M3 block scales.""" + table = torch.tensor(E2M1_VALUES, dtype=torch.float32, device=data.device) + nib = torch.stack([data & 0xF, data >> 4], dim=-1).reshape(-1).long() + vals = table[nib & 7] * torch.where(nib >= 8, -1.0, 1.0) + s = scales.view(torch.float8_e4m3fn).float() / quant_scale + return vals * s.repeat_interleave(FP4_BLOCK) + + +def scale_index(within_page: int, plane: int) -> torch.Tensor: + """Positions of a token's 8 block scales in its page's [128 * 8] scale plane: row-major for K, the 4-token + swizzle for V.""" + cols = torch.arange(8) + if plane == 0: + return within_page * 8 + cols + return (within_page // 4) * 32 + cols * 4 + within_page % 4 + + +class M3Producer(OpWrapper): + """The NVFP4 QKV + indexer producer of a sparse layer: ``minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert`` at the + TP2 heads (32 query, two KV and two index heads), head dim 128 and rotary dim 64, with the same arguments and + outputs bit for bit. + + ``packed`` [T, 39 * 128] BF16 holds each token's 32 q heads, 2 k heads, 2 v heads, 2 index-q heads and the + index-k head. Every head but v gets the Gemma RMSNorm with its weight and NeoX RoPE on its first 64 dims + (``rotary_cos_sin[position]``). Returns (q FP8 [T, 32, 128], index-q FP8 [T, 2, 128]); writes the token's slot + ``out_cache_loc[t]`` (skipped when negative or past the pool): each KV head's k and v as NVFP4 into its block of + the HND pools (16-value blocks, E4M3 scales against ``kv_quant_scale[1]`` / ``[2]``, the V scales in the 4-token + swizzle) and index-k as FP8. ``early`` triggers the dependent kernel on entry (its outputs are read only after the + dependent's own grid wait); ``prewait_step_inputs`` reads the position ids, the slots and the RoPE rows before the + kernel's own grid wait (none of them may be written by a PDL predecessor). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token (c1 without speculation)", spec=dict(tokens=1, scale=1.0)), + Cell(why="a c1 verify: 4 tokens", spec=dict(tokens=4, scale=3.0)), + Cell(why="a c4 verify: 16 tokens, small activations", spec=dict(tokens=16, scale=1e-3)), + Cell( + why="a c8 verify: 32 tokens, large activations, a skipped and an out-of-pool slot", + spec=dict(tokens=32, scale=40.0, bad_slots=True), + ), + Cell(why="the widest decode step: 48 tokens", spec=dict(tokens=48, scale=1.0)), + Cell( + why="a prefill chunk of 1000 tokens, early trigger off", + spec=dict(tokens=1000, scale=1.0, early=False), + ), + ) + + note = """ + The kernel is the native producer's arithmetic, compiled with the library's --use_fast_math, so it is held to that + kernel bit for bit on every output and cache byte. The torch reference states the math in fp32: the FP8 outputs are + held to it within one E4M3 step, and the NVFP4 K/V, decoded, within one E2M1 step of their block scale. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + prewait_step_inputs: bool = False, + ) -> tuple[torch.Tensor, torch.Tensor]: + q, index_q = torch.ops.trtllm.minimax_m3_sm107_tp2_producer_pw( + packed, + kv_data, + kv_scale, + index_k, + out_cache_loc, + kv_quant_scale, + eps, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + rotary_cos_sin, + position_ids, + pdl, + early, + prewait_step_inputs, + ) + return q, index_q + + def reference( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + prewait_step_inputs: bool = False, + ) -> dict[str, torch.Tensor]: + """fp32 values before their quantization: "q" [T, 32, 128], "index_q" [T, 2, 128], per token and KV head "k" + and "v" [T, 2, 128], and "index_k" [T, 128] (the values written to the token's slot). The caches, ``pdl``, + ``early`` and ``prewait_step_inputs`` do not change them.""" + t = packed.shape[0] + x = packed.float().view(t, HEADS_PACKED, HEAD_DIM) + rope = rotary_cos_sin[position_ids[:t].long()] + cos, sin = rope[:, 0], rope[:, 1] + + def heads(first, count, weight): + return torch.stack( + [_norm_rope(x[:, h], weight, cos, sin, eps) for h in range(first, first + count)], + dim=1, + ) + + k0 = HEADS_Q + v0 = k0 + HEADS_KV + iq0 = v0 + HEADS_KV + return { + "q": heads(0, HEADS_Q, q_weight), + "k": heads(k0, HEADS_KV, k_weight), + "v": x[:, v0 : v0 + HEADS_KV], + "index_q": heads(iq0, HEADS_INDEX, index_q_weight), + "index_k": _norm_rope(x[:, iq0 + HEADS_INDEX], index_k_weight, cos, sin, eps), + } + + def is_valid( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + prewait_step_inputs: bool = False, + ) -> None: + assert heads_q == HEADS_Q, f"the producer is built for {HEADS_Q} query heads, not {heads_q}" + t = packed.shape[0] + # The RoPE row is read at the position with no bound. + pos = position_ids[:t] + assert bool(((pos >= 0) & (pos < rotary_cos_sin.shape[0])).all()), ( + "position_ids must index rotary_cos_sin" + ) + # Two tokens writing one slot race. + slots = out_cache_loc[:t] + live = slots[(slots >= 0) & (slots < kv_data.shape[0] * PAGE)] + assert live.unique().numel() == live.numel(), "out_cache_loc must not repeat a slot" + + +m3_producer = M3Producer() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_qk_norm_rope_fp8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..f92a3e6efaa1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_qk_norm_rope_fp8.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' per-head Gemma RMSNorm + partial NeoX RoPE of the packed [Q | K | V], into a fresh FP8 copy.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3QkNormRopeFp8(OpWrapper): + """``fused_qk_norm_rope`` written out of place in FP8 E4M3: Q and K heads normed and roped in fp32 and rounded + once to E4M3 (saturating at +/-448), V cast, ``qkv`` left untouched. Same arguments as the catalog's in-place + entry. + + The dense layers (0-2) feed the FP8 main KV cache with it: on every step of an FP8 cache, and on the decode steps + of a hybrid NVFP4 cache (whose dense layers sit in its FP8 pool). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, tp4 heads (16 q, 1 kv)", spec=dict(m=1, q=16, kv=1)), + Cell(why="a 48-row decode verify", spec=dict(m=48, q=16, kv=1)), + Cell(why="a prefill chunk on an FP8 cache", spec=dict(m=2048, q=16, kv=1)), + ) + + note = """ + `reference` is the in-place kernel followed by a cast to E4M3. V matches it bit for bit; Q and K round once from + fp32 where the composition rounds through bf16 first, so they are within one E4M3 step of it. + """ + + def __call__( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + return torch.ops.trtllm.fused_qk_norm_rope_to_fp8( + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + + def reference( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + """The catalog's in-place kernel on a copy, then the E4M3 cast (V exact; Q and K within one E4M3 step).""" + out = fused_qk_norm_rope( + qkv.clone(), + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + return out.to(torch.float8_e4m3fn) + + +m3_qk_norm_rope_fp8 = M3QkNormRopeFp8() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_qkv_insert.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_qkv_insert.py new file mode 100644 index 000000000000..ef79579c0c21 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_qkv_insert.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' generic horizontal producer: packed QKV/index rows -> FP8 Q / index-Q, K / V / index-K into +the paged caches.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3QkvInsert(OpWrapper): + """The packed ``[Q | K | V | index-Q | index-K]`` bf16 rows of a sparse layer: per-head Gemma RMSNorm and the + partial NeoX RoPE of Q, K, index-Q and index-K, K / V written to the main paged cache at ``out_cache_loc`` and + index-K to the FP8 index cache; returns (Q, index-Q) in FP8 E4M3 [M, heads, 128]. + + ``kv_scale`` given: the main cache is NVFP4 (packed E2M1 data [pages, 2, kv, 128, 64] uint8 + E4M3 scale pool + [pages, 2, kv, 128, 8], ``kv_quant_scale`` the fp32 (Q, K, V) quantization scales); None: FP8 E4M3 + [pages, 2, kv, 128, 128]. Token-major and step-type agnostic. ``m3_producer`` takes the NVFP4 cache at the tp4 + heads; this entry takes the FP8 cache and NVFP4 pools outside that layout. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="FP8 main cache, a prefill chunk", spec=dict(m=1024, cache="fp8")), + Cell(why="FP8 main cache, one decode token", spec=dict(m=1, cache="fp8")), + ) + + note = """ + The arithmetic is the CUDA producer's (fp32 norm and RoPE with bf16 roundings, then the E4M3 casts); `reference` + returns the fp32 values, and the test holds the FP8 outputs and cache rows within one E4M3 step of them. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + common = ( + num_heads, + num_kv_heads, + num_index_heads, + head_dim, + rotary_dim, + eps, + q_norm, + k_norm, + index_q_norm, + index_k_norm, + rotary_cos_sin, + position_ids, + ) + if kv_scale is not None: + return torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, kv_scale, index_k_cache, out_cache_loc, kv_quant_scale, *common + ) + return torch.ops.trtllm.minimax_m3_fp8_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, index_k_cache, out_cache_loc, *common + ) + + def reference( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + """The fp32 values the producer rounds (a new dict, nothing written): Gemma RMSNorm + NeoX partial RoPE of + Q [M, heads, D], index-Q [M, index heads, D], K [M, kv heads, D] and index-K [M, D]; V [M, kv heads, D] as + packed. The cache rows at ``out_cache_loc`` and the returned FP8 Q / index-Q are these values cast.""" + m = packed.shape[0] + sizes = [ + num_heads * head_dim, + num_kv_heads * head_dim, + num_kv_heads * head_dim, + num_index_heads * head_dim, + head_dim, + ] + q, k, v, iq, ik = packed.float().split(sizes, dim=-1) + cos, sin = rotary_cos_sin[position_ids.long()].unbind(1) + half = rotary_dim // 2 + + def norm_rope(x, w, heads): + x = x.reshape(m, heads, head_dim) + x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * (1.0 + w.float()) + x1, x2, rest = x[..., :half], x[..., half:rotary_dim], x[..., rotary_dim:] + c, s = cos.unsqueeze(1), sin.unsqueeze(1) + return torch.cat([x1 * c - x2 * s, x2 * c + x1 * s, rest], dim=-1) + + return { + "q": norm_rope(q, q_norm, num_heads), + "index_q": norm_rope(iq, index_q_norm, num_index_heads), + "k": norm_rope(k, k_norm, num_kv_heads), + "v": v.reshape(m, num_kv_heads, head_dim), + "index_k": norm_rope(ik, index_k_norm, 1)[:, 0], + } + + +m3_qkv_insert = M3QkvInsert() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_blocks.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_blocks.py new file mode 100644 index 000000000000..76841cca732d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_blocks.py @@ -0,0 +1,82 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MSA top-k block selection from per-KV-head block scores (the built-in selector).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SelectBlocks(OpWrapper): + """Per query row the ``topk`` highest-scoring KV blocks of ``max_score_kv`` [kv_heads, n_blocks, total_q] (fp32), + with the first ``init_blocks`` and the last ``local_blocks`` valid blocks forced in and blocks at or beyond the + row's ``n_valid_blocks`` excluded; [total_q, kv_heads, topk] int32 ascending ids, -1 padded. ``head_major_output`` + backs the result head-major so ``permute(1, 0, 2)`` is contiguous. + + The decode dispatch runs it where the long-row selector does not apply: requests of 17..128 KV blocks. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="eight decode rows of 17..128 blocks", + spec=dict(total_q=8, n_blocks=128, mixed=True), + ), + Cell( + why="one decode row at the 128-block edge", + spec=dict(total_q=1, n_blocks=128, mixed=False), + ), + Cell(why="a 4-row verify at 40 blocks", spec=dict(total_q=4, n_blocks=40, mixed=False)), + ) + + note = """ + Exact: ties between equal scores break toward the lower block id; `reference` sorts with the same rule. + """ + + def __call__( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + nvb = n_valid_blocks.to(device=max_score_kv.device, dtype=torch.int32).contiguous() + return torch.ops.trtllm.minimax_m3_select_blocks( + max_score_kv, nvb, topk, init_blocks, local_blocks, head_major_output + ) + + def reference( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + """Plain torch selection; ``head_major_output`` changes the backing only.""" + heads, n_blocks, total_q = max_score_kv.shape + out = torch.full((total_q, heads, topk), -1, dtype=torch.int32, device=max_score_kv.device) + blocks = torch.arange(n_blocks, device=max_score_kv.device) + for t in range(total_q): + nv = int(n_valid_blocks[t]) + for h in range(heads): + s = max_score_kv[h, :, t].clone().double() + s[blocks >= nv] = float("-inf") + forced = (blocks < min(init_blocks, nv)) | ( + (blocks >= nv - local_blocks) & (blocks < nv) + ) + s[forced] = float("inf") + order = sorted(range(nv), key=lambda b: (-float(s[b]), b))[: min(topk, nv)] + ids = torch.tensor(sorted(order), dtype=torch.int32, device=out.device) + out[t, h, : ids.numel()] = ids + return out + + +m3_select_blocks = M3SelectBlocks() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_long.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_long.py new file mode 100644 index 000000000000..3d076c8181d6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_long.py @@ -0,0 +1,186 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 MSA block selection for decode rows of up to 9216 blocks.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Blocks per decode row the op covers (9 keys per thread of one 1024-thread CTA). +MAX_BLOCKS = 9216 +#: Blocks the selection keeps per (query, KV head). +TOPK = 16 +#: Scores the forcing substitutes: initial blocks rank above local ones, and both above any finite score. +INIT_SCORE = 1.0e30 +LOCAL_SCORE = 1.0e29 + +# M3's checkpoint: no forced initial blocks, the one block holding the query forced in. +_M3 = dict(num_kv_heads=1, init_blocks=0, local_blocks=1) + + +class M3SelectLong(OpWrapper): + """Fill ``out`` [total_q, num_kv_heads, 16] with each query's top-16 blocks. + + ``scores`` is the per-KV-head block maxima [num_kv_heads, num_blocks, + total_q] the CuTe DSL index scorer wrote (amax over the KV head's index + heads). Per (query, KV head) row the op keeps the ``n_valid_blocks[query]`` + leading blocks (clamped to ``[0, num_blocks]``), forces the last + ``local_blocks`` of them (counted from the *unclamped* valid count) to + ``LOCAL_SCORE`` and the first ``init_blocks`` to ``INIT_SCORE`` -- local wins + where both apply -- and selects the 16 largest, ties to the smaller block. + A pick whose forced score is -inf is emitted as -1. The ids are written + ascending with a -1 tail. + + Same contract as ``torch.ops.trtllm.minimax_m3_select_blocks`` (the in-tree + ``select_blocks_from_maxscore``) and bit-identical to it, except on a row + where more than 1024 valid blocks tie at the 16th-largest score: this op + breaks that tie to the smallest blocks as the contract says, the in-tree op + does not. The test checks both against ``reference``. Returns ``out``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="32K-context engine (257 worst-case k-tiles), Eagle3 verify at one request: 4 query rows", + spec=dict(num_blocks=257, total_q=4, valid="near_full", scores="random", **_M3), + ), + Cell( + why="decode without speculation at batch 8 on a 2048-block engine: the 2-key-per-thread path at its edge", + spec=dict(num_blocks=2048, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="1M-context engine at batch 8 x qlen 4: rows on all three key widths, short rows and padding rows", + spec=dict(num_blocks=8193, total_q=32, valid="mixed", scores="random", **_M3), + ), + Cell( + why="duplicated scores: ties broken to the smaller block, and > 1024 keys at the bound (re-read path)", + spec=dict(num_blocks=4500, total_q=4, valid="near_full", scores="ties", **_M3), + ), + Cell( + why="non-finite scores: +inf, NaN ranked above +inf, and rows whose picks run into -inf (emitted -1)", + spec=dict(num_blocks=600, total_q=8, valid="mixed", scores="nonfinite", **_M3), + ), + ) + + note = """ + Certified at one KV head per rank (TP4) with M3's forcing (`init_blocks=0`, + `local_blocks=1`). The kernel takes any head count and forcing; neither is + driven here. + + `scores` may be strided (the op reads it by stride); the target hands over + the scorer's contiguous view, which is what the cells drive. Rows of at + most 128 blocks are within the kernel's range, but the target keeps those on + the in-tree op, so they are not certified here. + + Ordering is by the descending-score key `orderedKey`: a positive NaN ranks + above +inf, +0 above -0, and a negative NaN below -inf -- which, unlike + -inf, is emitted as a block id rather than -1. The cells drive positive + NaN only. + """ + + def __call__( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_sm107_tp4_select_long( + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + fast_tail, + ) + return out + + def reference( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> torch.Tensor: + """The [total_q, num_kv_heads, 16] int32 table the op is supposed to write. + + Exact integer ranking, no floating point beyond the forcing: each + score's bit pattern maps to an unsigned key that is smaller for a larger + score (NaN above +inf), packed with the block id so one sort ranks by + (score descending, block ascending). ``out``, ``use_pdl``, ``early_trigger`` and + ``trigger_phase`` and ``fast_tail`` only say where and how the op writes; they do not enter the result. + """ + num_kv_heads, num_blocks, total_q = scores.shape + device = scores.device + blocks = torch.arange(num_blocks, device=device, dtype=torch.int64) + raw = n_valid_blocks[:total_q].to(torch.int64) + valid = raw.clamp(0, num_blocks) + local_start = (raw - local_blocks).clamp_min(0) + result = torch.full((total_q, num_kv_heads, TOPK), -1, dtype=torch.int32, device=device) + for head in range(num_kv_heads): + forced = scores[head].float().t().clone() # [total_q, num_blocks] + forced = torch.where(blocks[None, :] < init_blocks, INIT_SCORE, forced) + forced = torch.where(blocks[None, :] >= local_start[:, None], LOCAL_SCORE, forced) + bits = forced.view(torch.int32).to(torch.int64) & 0xFFFFFFFF + key = torch.where(bits >= 0x80000000, bits, (~bits) & 0x7FFFFFFF) + in_row = blocks[None, :] < valid[:, None] + packed = torch.where( + in_row, (key << 16) | blocks[None, :], torch.iinfo(torch.int64).max + ) + k = min(TOPK, num_blocks) + picked = torch.topk(packed, k, dim=1, largest=False).values + picked_key = picked >> 16 + keep = (picked != torch.iinfo(torch.int64).max) & (picked_key != 0xFF800000) + ids = torch.where(keep, picked & 0xFFFF, torch.iinfo(torch.int64).max) + ids = ids.sort(dim=1).values + ids = torch.where(ids == torch.iinfo(torch.int64).max, -1, ids) + result[:, head, :k] = ids.to(torch.int32) + return result + + def is_valid( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> None: + """The inputs this op takes and answers wrongly. + + Negative forcing counts are accepted and push the forced ranges past the + row instead of failing; the in-tree op this one stands in for refuses + them, so a caller relying on this one would silently diverge from it. + An ``out`` overlapping ``scores`` is overwritten while it is still read. + """ + assert init_blocks >= 0 and local_blocks >= 0, ( + f"init_blocks ({init_blocks}) and local_blocks ({local_blocks}) must be non-negative; the " + "op accepts negative counts and selects as if the forced ranges were elsewhere" + ) + storage = scores.untyped_storage() + assert not (storage.data_ptr() <= out.data_ptr() < storage.data_ptr() + storage.nbytes()), ( + "out must not alias scores: the op writes it while scores are still read" + ) + + +m3_select_long = M3SelectLong() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_long_dsl.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_long_dsl.py new file mode 100644 index 000000000000..9f79f4b69517 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_select_long_dsl.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 MSA block selection for decode rows of up to 9216 blocks, one row per 384-thread CTA (CuTe DSL).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_select_long import _M3, MAX_BLOCKS, M3SelectLong + + +class M3SelectLongDsl(OpWrapper): + """``m3_select_long``'s contract, cells and reference, run by ``kernels/m3_select_long_dsl``: the same selection + bit for bit as the 1024-thread ``minimax_m3_sm107_tp4_select_long`` (also on rows where more than 2048 blocks tie + at the 16th-largest score), from one 384-thread CTA per row. ``fast_tail`` (the 1024-thread kernel's warp-0 rank) + does not apply and is ignored. Returns ``out``.""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="32K-context engine (257 worst-case k-tiles), Eagle3 verify at one request: 4 query rows", + spec=dict(num_blocks=257, total_q=4, valid="near_full", scores="random", **_M3), + ), + Cell( + why="decode without speculation at batch 8 on a 2048-block engine: the 2-key-per-thread path at its edge", + spec=dict(num_blocks=2048, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="1M-context engine at batch 8 x qlen 4: rows on all three key widths, short rows and padding rows", + spec=dict(num_blocks=8193, total_q=32, valid="mixed", scores="random", **_M3), + ), + Cell( + why="duplicated scores: ties broken to the smaller block, and > 1024 keys at the bound (re-read path)", + spec=dict(num_blocks=4500, total_q=4, valid="near_full", scores="ties", **_M3), + ), + Cell( + why="non-finite scores: +inf, NaN ranked above +inf, and rows whose picks run into -inf (emitted -1)", + spec=dict(num_blocks=600, total_q=8, valid="mixed", scores="nonfinite", **_M3), + ), + ) + + def reference( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> torch.Tensor: + """``m3_select_long``'s reference: the selection is the same bit for bit.""" + return M3SelectLong.reference( + self, + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + fast_tail, + ) + + def is_valid( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> None: + """``m3_select_long``'s domain.""" + M3SelectLong.is_valid( + self, + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + fast_tail, + ) + + note = """ + Certified at one KV head per rank (TP4, Rubin) with M3's forcing + (`init_blocks=0`, `local_blocks=1`), and bit for bit against the + 1024-thread `m3_select_long` on every cell. The kernel takes any head count + and forcing; other forcing is not driven here. + + The kernel is compiled per (PDL launch, early trigger, trigger phase) with + symbolic shapes and strides; the target compiles its keys before warmup + (`kernels/m3_select_long_dsl.prebuild`), and a call outside them compiles + on first use. + """ + + def __call__( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> torch.Tensor: + from ...kernels.m3_select_long_dsl import select_long_dsl + + if scores.shape[1] > MAX_BLOCKS: + raise RuntimeError( + f"m3_select_long_dsl supports at most {MAX_BLOCKS} blocks, got {scores.shape[1]}" + ) + select_long_dsl( + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + ) + return out + + +m3_select_long_dsl = M3SelectLongDsl() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_sparse_decode.py new file mode 100644 index 000000000000..04d76f241b29 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_sparse_decode.py @@ -0,0 +1,215 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""This target's block-sparse decode attention (Triton, with the one-launch long-context route).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.triton_sparse_decode import minimax_m3_sparse_attn_decode +from .m3_sparse_decode_lc import PAGE, sparse_attention_reference + + +class M3SparseDecode(OpWrapper): + """``minimax_m3_sparse_attn_decode`` as ``M3Attention._decode_sparse_core`` calls it. + + Attention of each query token over the blocks ``topk_idx`` selected for it + (``[num_kv_heads, T, topk]``, ascending ids with a -1 tail), each block + truncated at the token's causal extent, written into ``output`` in place. + The target passes FP8 q, the packed NVFP4 sparse-layer cache with its block + and per-tensor scales, and BF16 ``output`` [T, 16, 128]. Returns ``output``. + + Which kernel runs is decided inside, per step: a selected (non-identity) + table on the TP4 geometry goes to the one-launch ``m3_sparse_decode_lc`` + kernel; an identity table (every request within 16 blocks, staged by the + metadata) runs the Triton producer, or from 17 query rows the multi-query + producer, then the LSE merge. Every route also publishes the MXFP8 o_proj + hand-off to ``m3_prequant.SLOT``; this entry certifies the attention output. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="identity selection, decode without speculation at batch 8: the Triton producer + merge", + spec=dict( + seq_lens=(1, 128, 129, 700, 2048, 1500, 64, 0), decode_query_len=1, identity=True + ), + ), + Cell( + why="identity selection, Eagle3 verify at one request (T=4): one block per producer program", + spec=dict(seq_lens=(1900,), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 4 (T=16): the last row count below the MQ producer", + spec=dict(seq_lens=(300, 2048, 4, 1000), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 8 (T=32): the multi-query producer, padding rows", + spec=dict( + seq_lens=(2048, 130, 5, 0, 1777, 900, 2000, 0), decode_query_len=4, identity=True + ), + ), + Cell( + why="selected blocks, Eagle3 verify at batch 2: routed to the one-launch long-context kernel", + spec=dict(seq_lens=(9000, 40000), decode_query_len=4, identity=False), + ), + Cell( + why="selected blocks, decode without speculation at batch 8: the long-context route at T=8", + spec=dict( + seq_lens=(3000, 2100, 9999, 2048, 2049, 4500, 16000, 2200), + decode_query_len=1, + identity=False, + ), + ), + ) + + note = """ + Certified only as the target calls it: one KV head of 16 query heads, + head_dim 128, FP8 q, packed NVFP4 cache, BF16 output, `num_topk_chunks` + left to the dispatcher. The function also takes BF16 / FP8 caches, other + head geometries and an explicit split-K factor (the in-tree tests drive + those); none is certified here. + + The MXFP8 hand-off each route publishes is certified on the long-context + kernel's own entry (`m3_sparse_decode_lc`); the Triton merge writes it with + the same recipe (`_store_mxfp8_head`). + """ + + def __call__( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + minimax_m3_sparse_attn_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + kv_scale=kv_scale, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + num_topk_chunks=num_topk_chunks, + identity_blocks=identity_blocks, + ) + return output + + def reference( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + """``output`` as the call is supposed to write it: float64 block-sparse attention over the + dequantized NVFP4 blocks (``sparse_attention_reference``), rounded once to ``output``'s dtype. + + Only the packed NVFP4 cache with one KV head is mirrored -- the target's + only configuration. ``kv_scale``, ``num_topk_chunks`` and + ``identity_blocks`` do not change what is computed. + """ + assert k_block_scale is not None and k_paged.shape[1] == 1, ( + "mirrors the one-KV-head NVFP4 cache only" + ) + attn = sparse_attention_reference( + q, + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx[0], + block_table, + seq_lens, + decode_query_len, + sm_scale, + ) + return attn.to(output.dtype) + + def is_valid( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> None: + """The inputs this call takes and answers wrongly. + + Every producer walks a token's first ``min(topk, ceil(kv_len / 128))`` + table entries and indexes the block table with them unchecked, so they + must be real blocks of the request. ``identity_blocks`` is a promise the + multi-query producer relies on: every token of a request reads the same + blocks, all of them. Reads ``topk_idx`` and ``seq_lens`` back to the + host: a test-only cost. + """ + total_q = q.shape[0] + seq = seq_lens.tolist() + max_topk = topk_idx.shape[-1] + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk_idx[:, tok, : min(max_topk, nblocks)] + assert bool(((ids >= 0) & (ids < min(nblocks, block_table.shape[1]))).all()), ( + f"token {tok}: the first min(topk, block count) selected ids must be blocks of the request; " + "the block table is indexed with them unchecked" + ) + if identity_blocks: + assert nblocks <= max_topk and bool( + (ids == torch.arange(nblocks, device=ids.device, dtype=ids.dtype)).all() + ), ( + f"token {tok}: identity_blocks promises the table holds every block of the request in order" + ) + + +m3_sparse_decode = M3SparseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_sparse_decode_lc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_sparse_decode_lc.py new file mode 100644 index 000000000000..1e556f2c61b1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/m3_sparse_decode_lc.py @@ -0,0 +1,607 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-launch long-context sparse decode over the selected NVFP4 blocks (16-CTA clusters), SM107 TP2 geometry. + +LC2 (``trtllm::minimax_m3_sm107_tp2_sparse_decode_lc``), ported from the M3 sm_107 TP2 target; the M3.1 TP2 target takes +it on pure generation steps with a selected table at 33..128 query rows (TP2WIDE_LC2, ``kernels/triton_sparse_decode``). + +Also the plain-torch block-sparse attention and MXFP8 hand-off mirrors the +Triton sparse decode entry (``m3_sparse_decode``) reuses: both compute the same +thing over the same cache. +""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The TP2 geometry the kernel is compiled for: 16 query heads on each of 2 KV heads per rank. +HEADS_PER_KV = 16 +KV_HEADS = 2 +NUM_HEADS = HEADS_PER_KV * KV_HEADS +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 +#: NVFP4 elements per E4M3 block scale. +SF_VEC = 16 +#: MXFP8 elements per UE8M0 scale, and the o_proj width the hand-off is laid out for. +MX_BLOCK = 32 +WIDTH = NUM_HEADS * HEAD_DIM + +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) + + +def prepare_lc2_split_workspace(kv_heads: int) -> int: + """Reserve LC2-split's persistent scratch before executor KV capacity profiling.""" + prepare = getattr(torch.ops.trtllm, "minimax_m3_sm107_tp2_sparse_decode_lc_prepare", None) + if prepare is None: + return 0 + anchor = torch.empty(0, dtype=torch.uint8, device="cuda") + return prepare(anchor, kv_heads) + + +def dequant_nvfp4_pages( + packed: torch.Tensor, + block_scale: torch.Tensor, + pages: torch.Tensor, + quad_scales: bool, + kv_head: int = 0, +) -> torch.Tensor: + """Dequantize KV head ``kv_head`` of the given pages of a packed NVFP4 cache to float64 [n, PAGE, HEAD_DIM]. + + ``packed`` is the [pages, kv_heads, PAGE, HEAD_DIM // 2] byte view (low + nibble = even element), ``block_scale`` the [pages, kv_heads, PAGE, + HEAD_DIM // 16] E4M3 bytes ``fused_write_layer_caches_nvfp4`` writes: token + ``t``'s column ``c`` at byte ``t * 8 + c`` of the page's region for K, and at + ``(t // 4) * 32 + c * 4 + t % 4`` for V (``quad_scales``). The per-tensor + global scale is not applied. Every product is exact in float64. + """ + table = torch.tensor(_E2M1, dtype=torch.float64, device=packed.device) + rows = packed[pages, kv_head].to(torch.int64) # [n, PAGE, HEAD_DIM // 2] + nibbles = torch.stack([rows & 0x0F, rows >> 4], dim=-1).reshape(rows.shape[0], PAGE, HEAD_DIM) + cols = HEAD_DIM // SF_VEC + region = block_scale[pages, kv_head].reshape(rows.shape[0], PAGE * cols) + t = torch.arange(PAGE, device=packed.device)[:, None] + c = torch.arange(cols, device=packed.device)[None, :] + index = (t // 4) * 32 + c * 4 + t % 4 if quad_scales else t * cols + c + scales = region[:, index.reshape(-1)].reshape(rows.shape[0], PAGE, cols) + scales = scales.view(torch.float8_e4m3fn).to(torch.float64) + return table[nibbles] * scales.repeat_interleave(SF_VEC, dim=-1) + + +def sparse_attention_reference( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, + kv_head: int = 0, +) -> torch.Tensor: + """Block-sparse decode attention over KV head ``kv_head`` of a packed NVFP4 cache, float64 [T, heads, HEAD_DIM]. + + Query token ``tok`` belongs to request ``tok // decode_query_len`` at offset + ``tok % decode_query_len`` and attends the first ``min(16, ceil(kv_len / + PAGE))`` entries of ``topk[tok]`` that are not -1, each block truncated at + ``kv_len = max(seq_len - decode_query_len + offset + 1, 0)``. A token that + attends nothing (a CUDA-graph padding row) is all zeros. ``q`` is E4M3 (or + its byte view); softmax and both products are float64. + """ + q64 = ( + q.view(torch.float8_e4m3fn).to(torch.float64) + if q.element_size() == 1 + else q.to(torch.float64) + ) + total_q, num_heads, _ = q64.shape + kgs = float(k_global_scale.reshape(-1)[0]) + vgs = float(v_global_scale.reshape(-1)[0]) + out = torch.zeros(total_q, num_heads, HEAD_DIM, dtype=torch.float64, device=q.device) + seq = seq_lens.tolist() + positions = torch.arange(PAGE, device=q.device) + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + real_topk = min(TOPK, (kv_len + PAGE - 1) // PAGE) + blocks = topk[tok, :real_topk] + blocks = blocks[blocks >= 0].to(torch.int64) + if blocks.numel() == 0: + continue + pages = block_table[req, blocks].to(torch.int64) + keys = ( + dequant_nvfp4_pages( + k, k_block_scale, pages, quad_scales=False, kv_head=kv_head + ).reshape(-1, HEAD_DIM) + * kgs + ) + values = ( + dequant_nvfp4_pages(v, v_block_scale, pages, quad_scales=True, kv_head=kv_head).reshape( + -1, HEAD_DIM + ) + * vgs + ) + valid = ((blocks[:, None] * PAGE + positions[None, :]) < kv_len).reshape(-1) + logits = (q64[tok] @ keys.T) * sm_scale + logits = logits.masked_fill(~valid[None, :], float("-inf")) + out[tok] = torch.softmax(logits, dim=-1) @ values + return out + + +def mxfp8_handoff_reference( + out: torch.Tensor, num_rows_alloc: int +) -> Tuple[torch.Tensor, torch.Tensor]: + """The MXFP8 o_proj hand-off the merge writes for BF16 rows ``out`` [T, WIDTH]: ``(q8 [T, WIDTH] uint8, + sf [num_rows_alloc * WIDTH // 32] uint8)``. + + Per 32-element block: ``e`` is the biased exponent of ``amax * rcp(448)`` + rounded up when its mantissa is non-zero (capped at 254), the elements are + scaled by ``2^(127 - e)`` and rounded to E4M3 (saturating), and ``e`` lands + in the 128x4-swizzled scale layout the MXFP8 GEMM reads. Entries of ``sf`` + for rows past ``T`` are zero. Computed from the op's own BF16 rows, so a + comparison against it is exact. + """ + total_q = out.shape[0] + x = out.float().reshape(total_q, WIDTH // MX_BLOCK, MX_BLOCK) + amax = x.abs().amax(dim=-1) + sfv = amax * torch.tensor(1.0 / 448.0, dtype=torch.float32, device=out.device) + bits = sfv.view(torch.int32) + e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + e = e.clamp(max=254) + qs = torch.where(amax != 0, ((254 - e) << 23).view(torch.float32), torch.zeros_like(amax)) + xq = (x * qs[..., None]).clamp(-448.0, 448.0) + q8 = xq.to(torch.float8_e4m3fn).view(torch.uint8).reshape(total_q, WIDTH) + num_sf_k_tiles = WIDTH // 128 + sf = torch.zeros(num_rows_alloc * (WIDTH // MX_BLOCK), dtype=torch.uint8, device=out.device) + tok = torch.arange(total_q, device=out.device)[:, None] + k_sf = torch.arange(WIDTH // MX_BLOCK, device=out.device)[None, :] + offset = ( + (tok // 128) * num_sf_k_tiles * 512 + + (k_sf // 4) * 512 + + (tok % 32) * 16 + + ((tok % 128) // 32) * 4 + + k_sf % 4 + ) + sf[offset.reshape(-1)] = e.to(torch.uint8).reshape(-1) + return q8, sf + + +class M3SparseDecodeLc(OpWrapper): + """Sparse decode attention, split-K merge and MXFP8 hand-off of one TP2 decode step in one launch. + + One cluster of 16 CTAs per (query token, KV head): CTA ``r`` attends the + token's ``r``-th selected block of that head (``topk[h, tok, r]``), and the + cluster merges the 16 partials over distributed shared memory into the + head's 16 BF16 rows of ``out[tok]``. With ``q8`` / ``sf`` it also writes + the row's MXFP8 copy and swizzled UE8M0 scales for the o_proj, bitwise + ``torch.ops.trtllm.mxfp8_quantize(out, True)``. All operands are the byte + views ``m3_attn_lc.m3_lc_attn`` hands over (E4M3 ``q``, packed E2M1 pages, + E4M3 block scales). Returns ``out``. + + Per block: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16 + (exact), QK and PV on FP16 tensor cores with FP32 accumulation, the softmax + probabilities rounded to FP16 for PV, log2-domain softmax with + ``ex2.approx``. ``linear_softmax`` picks the denominator recurrence (the two + agree to rounding); SM107 keeps the log-domain recurrence of the stock + Triton producer. ``prewait`` stages the request's block-table row in shared + memory before the grid dependency wait; it does not change a value. + ``n1_ids`` (int32 [2, rows, 16], dense rows, initialized to -1) holds the + layer's selected block ids one decode step back: each CTA prefetches its + slot's block into L2 before the grid dependency wait and overwrites the slot + with this step's id (-1 past the token's block count); rows at or past + ``rows`` skip both. A hint: it does not change a value. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="Eagle3 verify at one request, 165k context: T=4, log-domain recurrence, MXFP8 hand-off", + spec=dict( + seq_lens=(165521,), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="Eagle3 verify at two requests, long and medium context: T=8, no hand-off", + spec=dict( + seq_lens=(165521, 98765), + decode_query_len=4, + linear_softmax=False, + mxfp8=False, + prewait=False, + ), + ), + Cell( + why="Eagle3 verify at two requests: T=8 with the MXFP8 hand-off", + spec=dict( + seq_lens=(165521, 12345), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="two short requests with -1 tails and a zero-length padding request: T=8", + spec=dict( + seq_lens=(130, 0), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="decode without speculation at two requests: T=2", + spec=dict( + seq_lens=(165521, 5000), + decode_query_len=1, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="the wide target's largest hand-off step: 32 requests, T=128 (one full scale tile), mixed lengths " + "with tile edges and a zero-length padding request, MXFP8 hand-off", + spec=dict( + seq_lens=tuple((165521, 20000, 1281, 129)[i % 4] for i in range(31)) + (0,), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="K7 verify (qlen 8) at 5 requests: T=40, the narrowest TP2WIDE_LC2 step class, mixed lengths with " + "128-token edges inside the verify rows and a zero-length padding request, MXFP8 hand-off", + spec=dict( + seq_lens=(165521, 20000, 1285, 129, 0), + decode_query_len=8, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="K7 verify (qlen 8) at 8 requests: T=64 (the VR-BANDS step bound), long / medium / exactly 16 " + "blocks / tile edges / fewer tokens than verify rows / a zero-length padding request, MXFP8 hand-off", + spec=dict( + seq_lens=(165521, 98765, 20000, 2048, 1285, 136, 4, 0), + decode_query_len=8, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="K7 verify (qlen 8) at 12 requests: T=96, mixed lengths with tile edges and a zero-length padding " + "request, MXFP8 hand-off", + spec=dict( + seq_lens=(165521,) + + tuple((40000, 20000, 1285, 129, 2056)[i % 5] for i in range(10)) + + (0,), + decode_query_len=8, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="K7 verify (qlen 8) at 16 requests: T=128 (MergeMxfp8.TRITON_MAX_ROWS, one full scale tile), mixed " + "lengths " + "with tile edges and a zero-length padding request, MXFP8 hand-off", + spec=dict( + seq_lens=(165521,) + + tuple((40000, 20000, 1285, 129, 2056, 136)[i % 6] for i in range(14)) + + (0,), + decode_query_len=8, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + # TP2WIDE_LC2_448: the agg TP2 Y steps (K3, 4 rows / request) and the K7 c46 / c56 steps. + Cell( + why="K3 verify (qlen 4) at 46 requests: T=184 (one full scale tile + a 56-row partial tile), mixed " + "lengths, MXFP8 hand-off", + spec=dict( + seq_lens=(165521,) + + tuple((40000, 20000, 1285, 129, 2056, 136)[i % 6] for i in range(44)) + + (0,), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="K3 verify (qlen 4) at 56 requests: T=224 (agg TP2 c56), mixed lengths, MXFP8 hand-off", + spec=dict( + seq_lens=(165521,) + + tuple((40000, 20000, 1285, 129, 2056, 136)[i % 6] for i in range(54)) + + (0,), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="K3 verify (qlen 4) at 64 requests: T=256 (agg TP2 c64, two full scale tiles), mixed lengths, " + "MXFP8 hand-off", + spec=dict( + seq_lens=(165521,) + + tuple((40000, 20000, 1285, 129, 2056, 136)[i % 6] for i in range(62)) + + (0,), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="K7 verify (qlen 8) at 46 requests: T=368 (two full scale tiles + a 112-row partial tile), " + "MXFP8 hand-off", + spec=dict( + seq_lens=(165521,) + + tuple((40000, 20000, 1285, 129, 2056, 136)[i % 6] for i in range(44)) + + (0,), + decode_query_len=8, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="K7 verify (qlen 8) at 56 requests: T=448 (LC_MAX_ROWS, MergeMxfp8.MAX_ROWS), mixed lengths, " + "MXFP8 hand-off", + spec=dict( + seq_lens=(165521,) + + tuple((40000, 20000, 1285, 129, 2056, 136)[i % 6] for i in range(54)) + + (0,), + decode_query_len=8, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="the linear denominator and pre-wait staging variants, driven at T=8", + spec=dict( + seq_lens=(165521, 20000), + decode_query_len=4, + linear_softmax=True, + mxfp8=True, + prewait=True, + ), + ), + ) + + note = """ + Certified: the eight instantiated variants (`linear_softmax` x `prewait` x + hand-off on / off). The hand-off cells are certified bit-exactly against + `mxfp8_handoff_reference` applied to the op's own BF16 rows, which equals + `torch.ops.trtllm.mxfp8_quantize(out, True)`; the BF16 rows themselves within + the band the test derives. `prewait` and the hand-off do not change the BF16 + rows. + + `topk` is the head-major [2, T, 16] table; it is read for the first + `min(16, ceil(kv_len / 128))` entries of each token; a -1 among them skips + that CTA's block, entries past them are never read. `use_pdl` only changes + the launch attribute. The hand-off assumes `num_sf_k_tiles = 32` (a + 4096-wide o_proj input) and buffers with at least as many rows as the step; + `MERGE_MXFP8` sizes them at `MergeMxfp8.MAX_ROWS` (448) rows. + + The M3.1 TP2 target calls it at 33..LC_MAX_ROWS (448) query rows only + (TP2WIDE_LC2, TP2WIDE_LC2_448): the qlen-8 (K7) cells at 5 / 8 / 12 / 16 / + 46 / 56 requests (40 / 64 / 96 / 128 / 368 / 448 rows) and the qlen-4 (K3) + cells at 46 / 56 / 64 requests (184 / 224 / 256 rows) certify that range in + the variant it runs (log-domain recurrence, no pre-wait, MXFP8 hand-off on); + the narrower cells are kept from the M3 target. + """ + + def __call__( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + prewait: bool, + n1_ids: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_sm107_tp2_sparse_decode_lc( + q, + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + out, + q8, + sf, + num_sf_k_tiles, + decode_query_len, + sm_scale, + linear_softmax, + use_pdl, + prewait, + n1_ids, + ) + return out + + def reference( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + prewait: bool, + n1_ids: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """The BF16 ``out`` [T, 4096] the op is supposed to write: ``sparse_attention_reference`` per KV head in + float64, rounded once. ``linear_softmax``, ``use_pdl``, ``prewait`` and ``n1_ids`` do not change what is + computed; the hand-off buffers are checked separately with ``mxfp8_handoff_reference``. + """ + q3 = q.view(-1, NUM_HEADS, HEAD_DIM) + attn = torch.cat( + [ + sparse_attention_reference( + q3[:, h * HEADS_PER_KV : (h + 1) * HEADS_PER_KV], + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk[h], + block_table, + seq_lens, + decode_query_len, + sm_scale, + kv_head=h, + ) + for h in range(KV_HEADS) + ], + dim=1, + ) + return attn.reshape(out.shape).to(out.dtype) + + def is_valid( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + prewait: bool, + n1_ids: Optional[torch.Tensor] = None, + ) -> None: + """The inputs this op takes and answers wrongly (``m3_attn_lc.lc_applies`` is the target's form of the + layout half). + + The kernel addresses a page's token rows at a fixed 64-byte (data) and + 8-byte (scale) pitch and loads them 16 / 8 bytes at a time, so only the + page and KV-head strides are read from the views. It reads the selected + block ids, the block table and the hand-off buffers unchecked. Reads + ``topk`` and ``seq_lens`` back to the host: a test-only cost. + """ + for name, t in (("k", k), ("v", v)): + assert ( + t.dim() == 4 and t.shape[1] == KV_HEADS and t.shape[2:] == (PAGE, HEAD_DIM // 2) + ), f"{name} must be the [pages, 2, 128, 64] byte view of the rank's KV heads" + assert t.stride(3) == 1 and t.stride(2) == HEAD_DIM // 2, ( + f"{name} rows must be dense 64-byte rows: the kernel ignores the row stride" + ) + assert t.data_ptr() % 16 == 0 and t.stride(0) % 16 == 0 and t.stride(1) % 16 == 0, ( + f"{name} pages and heads must be 16-byte aligned" + ) + assert v.stride() == k.stride(), "k and v share their strides" + for name, t in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + assert ( + t.dim() == 4 + and t.shape[1] == KV_HEADS + and t.stride(3) == 1 + and t.stride(2) == HEAD_DIM // SF_VEC + ), f"{name} rows must be dense 8-byte rows: the kernel ignores the row stride" + assert t.data_ptr() % 8 == 0 and t.stride(0) % 8 == 0 and t.stride(1) % 8 == 0, ( + f"{name} pages and heads must be 8-byte aligned" + ) + assert q.dim() == 3 and q.shape[1:] == (NUM_HEADS, HEAD_DIM), "q must be [T, 32, 128]" + assert q.data_ptr() % 16 == 0 and q.stride(0) % 16 == 0 and q.stride(1) % 16 == 0, ( + "q rows are loaded 16 bytes at a time and must be 16-byte aligned" + ) + assert k_global_scale.numel() == 1 and v_global_scale.numel() == 1, ( + "global scales are single values" + ) + total_q = q.shape[0] + assert topk.dim() == 3 and topk.shape[0] == KV_HEADS and topk.shape[1] == total_q, ( + "topk is the head-major [2, T, 16] table" + ) + assert ( + total_q % decode_query_len == 0 and seq_lens.numel() >= total_q // decode_query_len + ), ( + "every query token must belong to a request: seq_lens is read at tok // decode_query_len unchecked" + ) + seq = seq_lens.tolist() + num_cols = block_table.shape[1] + for h in range(KV_HEADS): + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk[h, tok, : min(TOPK, nblocks)] + assert bool(((ids >= -1) & (ids < min(nblocks, num_cols))).all()), ( + f"head {h} token {tok}: selected block ids must be -1 or below the request's block count; " + "the block table is indexed with them unchecked" + ) + if q8 is not None: + assert num_sf_k_tiles == WIDTH // 128, ( + f"num_sf_k_tiles must be {WIDTH // 128}: the hand-off rows are {WIDTH} wide and a smaller " + "value overlaps them silently" + ) + assert ( + q8.numel() >= total_q * WIDTH + and sf is not None + and sf.numel() >= ((total_q + 127) // 128 * 128 * (WIDTH // MX_BLOCK)) + ), "the hand-off buffers must hold every row of the step: they are written unchecked" + + +m3_sparse_decode_lc = M3SparseDecodeLc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/msa_kv_update.py new file mode 100644 index 000000000000..43166b472eca --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/attention/msa_kv_update.py @@ -0,0 +1,200 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MSA KV-length correction in one launch (this target's ``MsaKvUpdate`` hook).""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The MSA pool's page: one sparse block is one 128-token KV page. +TOKENS_PER_BLOCK = 128 + + +class MsaKvUpdate(OpWrapper): + """Re-derive the MSA decode state after the overlap scheduler moved ``kv_lens``. + + ``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` calls this with its + own buffers at the head of every captured speculative decode step. One + single-CTA launch writes, for ``batch`` requests and ``total_q`` query + tokens (all int32): + + * ``msa_seq_lens[b] = min(kv_lens[b], kv_lens_staged[b])`` and, when given, + ``cu_kv_lens`` as its exclusive prefix sum (``batch + 1`` entries); + * ``out_cache_loc[t]``, the KV write slot of the position token ``t`` + attends up to, through ``block_table``; + * ``n_valid_blocks[t]``, when given, the block count top-k selection is + bounded by. + + It returns None; the cells read the four buffers. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode without speculation at one request: qo = 1, every buffer written", + spec=dict(batch=1, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="decode without speculation at the captured batch 8", + spec=dict(batch=8, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="Eagle3 verify (draft 3): qo = 4 per request, partial acceptance shortens kv_lens below staged", + spec=dict(batch=8, query_len=4, accept="partial", cu_kv=True, n_valid=True), + ), + Cell( + why="the metadata without cu_kv_lens (no extend_ctx promotion): the optional prefix sum is skipped", + spec=dict(batch=4, query_len=4, accept="partial", cu_kv=False, n_valid=True), + ), + Cell( + why="CUDA-graph padding rows (kv_lens 0, pos -1) and no n_valid_blocks buffer", + spec=dict(batch=8, query_len=4, accept="padded", cu_kv=True, n_valid=False), + ), + Cell( + why="positions crossing a page boundary: the slot moves to the previous page of the request", + spec=dict(batch=3, query_len=4, accept="page_edge", cu_kv=True, n_valid=True), + ), + ) + + note = """ + The kernel covers any batch up to the block size's loop (it strides over + 256 threads), and any page size; only `tokens_per_block = 128`, the MSA + pool's page, is certified. + + The op does not bounds-check the request index a token reads + (`q_batch_row`) or the block-table column it derives: both come from the + metadata's own staging, which keeps them in range by construction, and an + out-of-range value reads whatever memory sits there. `is_valid` states both. + """ + + def __call__( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + torch.ops.trtllm.minimax_m3_msa_kv_update( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + + def reference( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> Tuple[torch.Tensor, Optional[torch.Tensor], torch.Tensor, Optional[torch.Tensor]]: + """``(msa_seq_lens[:batch], cu_kv_lens[:batch + 1], out_cache_loc[:total_q], + n_valid_blocks[:total_q])`` as the op is supposed to write them. + + The in-tree metadata's own torch re-derivation + (``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` past its hook + call), statement for statement, returning new tensors rather than + writing the buffers. The buffer arguments only say which outputs exist: + a ``None`` buffer gives a ``None`` result. + """ + page = int(tokens_per_block) + kv_true = torch.minimum(kv_lens[:batch], kv_lens_staged[:batch]) + cu_kv = None + if cu_kv_lens is not None: + cu_kv = torch.zeros(batch + 1, dtype=torch.int32, device=kv_lens.device) + cu_kv[1:] = torch.cumsum(kv_true, 0) + qbr = q_batch_row[:total_q].to(torch.long) + qo_dev = seq_lens[:batch] + pos = kv_true[qbr] - qo_dev[qbr] + q_intra[:total_q] + pos_slot = pos.clamp_min(0) + block_col = torch.div(pos_slot, page, rounding_mode="floor").clamp( + max=int(block_table.shape[1]) - 1 + ) + slots = block_table[qbr, block_col.to(torch.long)] * page + torch.remainder(pos_slot, page) + n_valid = None + if n_valid_blocks is not None: + n_valid = torch.div( + (pos + 1).clamp_min(0) + (page - 1), page, rounding_mode="floor" + ).to(torch.int32) + return kv_true.to(torch.int32), cu_kv, slots.to(torch.int32), n_valid + + def is_valid( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + """The inputs this op takes and answers wrongly. + + Every operand but the block table is read as a dense run from its first + element, so a strided view reads the wrong elements. The request index of + each token and the table row it selects are not bounds-checked. Reads + ``q_batch_row`` back to the host: a test-only cost. + """ + for name, t in ( + ("kv_lens", kv_lens), + ("kv_lens_staged", kv_lens_staged), + ("seq_lens", seq_lens), + ("q_batch_row", q_batch_row), + ("q_intra", q_intra), + ("msa_seq_lens", msa_seq_lens), + ("out_cache_loc", out_cache_loc), + ("cu_kv_lens", cu_kv_lens), + ("n_valid_blocks", n_valid_blocks), + ): + assert t is None or t.is_contiguous(), ( + f"{name} must be contiguous: the kernel indexes it densely from data_ptr and ignores strides" + ) + rows = q_batch_row[:total_q] + assert total_q == 0 or ( + int(rows.min()) >= 0 and int(rows.max()) < min(batch, block_table.shape[0]) + ), ( + "q_batch_row must index a request below batch (and a block-table row): the op reads the " + "request's lengths and table row unchecked" + ) + + +msa_kv_update = MsaKvUpdate() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/__init__.py new file mode 100644 index 000000000000..5da86fe24bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collective-communication entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_allreduce.py new file mode 100644 index 000000000000..d67b040e8bfd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_allreduce.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The built-in TP all-reduce, fused with the residual add and the next RMSNorm.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.distributed import AllReduce, AllReduceParams + + +class M3AllReduce(OpWrapper): + """``comm(x, all_reduce_params=params)``: the runtime's ``AllReduce`` over the TP group. ``params`` None: the + plain sum over the ranks (the drafter's o_proj and down projection). With ``RESIDUAL_RMS_NORM`` params: + ``updated = residual + sum_ranks(x)``, ``normed = rmsnorm(updated) * norm_weight``; + returns (normed, updated). The strategy (``AllReduceStrategy.AUTO``: one-shot / two-shot Lamport, NCCL) is the + communicator's per call. + + The layer boundaries the multicast all-reduce does not take run it: prefill and mixed steps, and generation steps + beyond the multicast kernel's rows. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="tp4, a prefill chunk", spec=dict(world=4, m=2048, hidden=6144)), + Cell(why="tp4, a 64-row generation step", spec=dict(world=4, m=64, hidden=6144)), + Cell( + why="tp4, plain sum of eight drafter rows", + spec=dict(world=4, m=8, hidden=6144, plain=True), + ), + ) + + note = """ + The reduction order across ranks is the strategy's; `reference` sums the ranks' rows in fp32 in rank order, then + adds the residual and normalizes (one bf16 rounding of the sum and one of the norm). A strategy may round the sum + to bf16 before the residual add, so the residual sum agrees within one bf16 ulp of its operands' magnitude. + """ + + def __call__(self, x: torch.Tensor, comm: AllReduce, params: Optional[AllReduceParams]): + return comm(x, all_reduce_params=params) + + def reference(self, xs, comm: AllReduce, params: Optional[AllReduceParams]): + """``xs``: every rank's ``x`` in rank order (a collective's reference takes the group's inputs).""" + total = torch.stack([x.float() for x in xs]).sum(0) + if params is None: + return total.to(xs[0].dtype) + updated = (params.residual.float() + total).to(xs[0].dtype) + h = updated.float() + normed = ( + h + * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + params.eps) + * params.norm_weight.float() + ) + return normed.to(xs[0].dtype), updated + + +m3_allreduce = M3AllReduce() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_mc_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_mc_allreduce_norm.py new file mode 100644 index 000000000000..208cb25aa72e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_mc_allreduce_norm.py @@ -0,0 +1,329 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode layer boundary, 4 ranks.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +_E4M3_MAX = 448.0 + + +def mxfp8_rows(normed: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """MXFP8 of BF16 rows as ``trtllm::mxfp8_quantize(normed, True)``: E4M3 [M, H] and 128x4-swizzled UE8M0 bytes. + + One scale per 32 columns, the smallest power of two at or above ``amax / 448`` (biased exponent, round up), + data = row / scale rounded to nearest E4M3. The scale buffer is padded to 128 rows, padding bytes left as the + op leaves them (the comparison masks them out). + """ + m, h = normed.shape + blocks = normed.float().view(m, h // 32, 32) + amax = blocks.abs().amax(-1) + bits = (amax / _E4M3_MAX).view(torch.int32) + byte = (((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) > 0).to(torch.int32)).clamp(0, 254) + scale = torch.exp2(byte.float() - 127.0).unsqueeze(-1) + data = torch.where(amax.unsqueeze(-1) != 0, blocks / scale, torch.zeros_like(blocks)) + data = data.view(m, h).to(torch.float8_e4m3fn) + return data, swizzle_sf(byte.to(torch.uint8)) + + +def swizzle_sf(byte: torch.Tensor) -> torch.Tensor: + """Scatter [rows, H / 32] scale bytes into the 128x4-swizzled buffer the op writes (zero padding).""" + rows, cols = byte.shape + padded_rows = (rows + 127) // 128 * 128 + r = torch.arange(rows, device=byte.device).view(-1, 1) + c = torch.arange(cols, device=byte.device).view(1, -1) + index = ( + (c % 4) + (c // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + (r // 128) * (128 * cols) + ) + out = torch.zeros(padded_rows * cols, dtype=torch.uint8, device=byte.device) + out[index.reshape(-1)] = byte.reshape(-1) + return out + + +def sf_written(rows: int, cols: int, device) -> torch.Tensor: + """Mask of the swizzled scale bytes the op writes for ``rows`` rows (the padding rows are left unwritten).""" + return swizzle_sf(torch.ones(rows, cols, dtype=torch.uint8, device=device)).bool() + + +def boundary( + partials: Sequence[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + drafter: bool = False, +) -> tuple[torch.Tensor, ...]: + """The boundary arithmetic once each rank's BF16 partial is known (shared with the finalize entry). + + ``drafter``: the Eagle3 drafter boundary (opts bit 16), whose stock is a plain BF16 all-reduce, then FlashInfer + fused_add_rmsnorm: the RMS statistic and the normalization take the unrounded FP32 sum bf16(rank sum) + residual; + only the stored residual is rounded to BF16. + + The stock rounding points of this target (a BF16 all-reduce, then the fused residual add + RMSNorm): ranks are + summed in FP32 in rank order and rounded to BF16, the residual added and rounded to BF16 again; the RMS statistic + and the normalization are done in float64 and rounded to BF16 once. + """ + total = torch.zeros(partials[0].shape, dtype=torch.float32, device=residual.device) + for p in partials: + total = total + p.float() + pre = total.to(torch.bfloat16).float() + residual.float() + updated = pre.to(torch.bfloat16) + u = pre.double() if drafter else updated.double() + inv = torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + eps) + normed = (u * inv * norm_w.double()).to(torch.bfloat16) + if not quant: + return normed, updated + return (normed, updated, *mxfp8_rows(normed)) + + +class M3McAllreduceNorm(OpWrapper): + """One TP group's boundary: ``updated = bf16(residual + sum_r bf16(x_r + x2_r))``, ``normed = RMSNorm(updated)``. + + Every rank pushes its partial through the multicast mapping ``ws_mc_ptr`` into every rank's exchange, then polls + its own view ``ws_uc`` (re-arming the words it read) and rotates the per-CTA call parities in ``flags``; the + workspace is the ``McAllReduceWorkspace`` bucket of ``maxt`` tokens. All ranks must make the same calls in the + same stream order. ``prepushed`` 2 means a producer already pushed ``x`` (and launched this grid after its + pushes); 3 that two producers pushed ``x`` into exchange set A and ``x2`` into set B. Returns + ``[normed, updated]`` or, with ``quant``, also the MXFP8 copy of ``normed`` (E4M3 rows, swizzled UE8M0). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The tp4 decode boundaries (hidden 6144, 4 ranks, eps 1e-6). Buckets 8 / 16 / 48 are separate workspaces; the + # 8-token bucket runs epilogue option 7 at 1, 2, 7, 8 tokens and 15 at 3..6, the others 15. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one decode token at the attention boundary, x only, bucket 8 option 7", + spec=dict(m=1, maxt=8, opts=7, x2=False, quant=False, prepushed=0), + ), + Cell( + why="MoE boundary at 4 tokens: shared + routed summed in-kernel, MXFP8 copy for the next QKV, option 15", + spec=dict(m=4, maxt=8, opts=15, x2=True, quant=True, prepushed=0), + ), + Cell( + why="a full 8-token bucket with the second addend", + spec=dict(m=8, maxt=8, opts=7, x2=True, quant=False, prepushed=0), + ), + Cell( + why="attention boundary whose rows the o_proj GEMV already pushed (prepushed 2)", + spec=dict(m=8, maxt=8, opts=7, x2=False, quant=False, prepushed=2), + ), + Cell( + why="routed engine pushed set A, shared down projection set B (prepushed 3), with the MXFP8 copy", + spec=dict(m=6, maxt=8, opts=15, x2=True, quant=True, prepushed=3), + ), + Cell( + why="12 tokens pushed by the o_proj GEMV into the 16-token bucket", + spec=dict(m=12, maxt=16, opts=15, x2=False, quant=False, prepushed=2), + ), + Cell( + why="the 9..13-token MoE kernel's sum into the 16-token bucket, MXFP8 copy", + spec=dict(m=16, maxt=16, opts=15, x2=False, quant=True, prepushed=0), + ), + Cell( + why="the 48-token bucket at a partial fill", + spec=dict(m=30, maxt=48, opts=15, x2=False, quant=False, prepushed=0), + ), + Cell( + why="the 48-token bucket full, second addend and MXFP8 copy", + spec=dict(m=48, maxt=48, opts=15, x2=True, quant=True, prepushed=0), + ), + # Every other row count of the decode boundaries (modeling.MC_ALLREDUCE): the bucket and epilogue options the + # target picks for it (kernels/m3_mc_allreduce), the second addend and the MXFP8 copy alternating. + *( + Cell( + why=f"{m} tokens: a boundary row count between the ones above", + spec=dict( + m=m, + maxt=8 if m <= 8 else 16 if m <= 16 else 48, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1] if m <= 8 else 15, + x2=m % 2 == 0, + quant=m % 3 == 0, + prepushed=0, + ), + ) + for m in range(1, 49) + if m not in (1, 4, 8, 16, 30, 48) + ), + # Every other row count pushed by the o_proj (1..16) or the 1..8-token MoE engine (with the MXFP8 copy): + # modeling.ATTN_PUSH / modeling.MOE_PUSH. + *( + Cell( + why=f"{m} tokens prepushed: a pushed row count between the ones above", + spec=dict( + m=m, + maxt=8 if m <= 8 else 16, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1] if m <= 8 else 15, + x2=False, + quant=m <= 8, + prepushed=2, + ), + ) + for m in range(1, 17) + if m not in (8, 12) + ), + # The Eagle3 drafter boundaries (modeling.DRAFT_MC_AR): option 16 (the row kernel's drafter form) on 23 / 31 + # at every draft-pass row count 1..8 (pass 0: 4 x requests, later passes: requests), x only. + *( + Cell( + why=f"Eagle3 drafter boundary at {m} rows (opts {23 if m in (1, 2, 7, 8) else 31})", + spec=dict( + m=m, + maxt=8, + opts=23 if m in (1, 2, 7, 8) else 31, + x2=False, + quant=False, + prepushed=0, + drafter=True, + ), + ) + for m in range(1, 9) + ), + # The same boundaries on the 8-CTA cluster kernel (opts bit 128, modeling.DRAFT_MC_AR_CLUSTER, launched without + # the carveout: 23 | 128), alone and fed by the drafter projection's push (prepushed 2, DRAFT_AR_PUSH). + *( + Cell( + why=f"Eagle3 drafter boundary on the cluster kernel at {m} rows, prepushed {pre}", + spec=dict( + m=m, + maxt=8, + opts=23 | 128, + x2=False, + quant=False, + prepushed=pre, + drafter=True, + ), + ) + for pre in (0, 2) + for m in range(1, 9) + ), + ) + + note = """ + The epilogue option bits (1 early residual / weight load, 2 st.async cluster exchange, 4 early parity rotation, + 8 max carveout) are bit-identical by construction; only the four (bucket, option) pairs the target selects are + built. `prepushed` 1 (pushed, grid wait kept) is accepted by the op but no caller passes it. `pdl` changes only + launch overlap and is driven off in the cells. + + `normed` is gated to one BF16 rounding step: the kernel's RMS statistic is a warp / cluster tree of FP32 sums + and its `rsqrtf` is approximate, so a row whose scaled value lands next to a BF16 midpoint can round either + way. `updated` and the MXFP8 copy of the op's own `normed` are bit-exact. + """ + + def __call__( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + cap_out: Optional[torch.Tensor] = None, + cap_w: Optional[torch.Tensor] = None, + cap_eps: float = 0.0, + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_sm107_tp4_mc_allreduce_norm( + x, + x2, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + None, + None, + maxt, + quant, + prepushed, + opts, + cap_out, + cap_w, + cap_eps, + ) + + def reference( + self, + per_rank_x: Sequence[torch.Tensor], + per_rank_x2: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + drafter: bool = False, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute (``drafter``: the opts-16 drafter boundary). + + The documented collective exception: it takes every rank's ``x`` (and ``x2``) in rank order, not the calling + rank's. Each rank's partial is ``bf16(x + x2)`` (``x`` alone without ``x2``), whatever pushed it. + """ + partials = [] + for r, x in enumerate(per_rank_x): + if per_rank_x2 is None: + partials.append(x) + else: + partials.append((x.float() + per_rank_x2[r].float()).to(torch.bfloat16)) + return boundary(partials, residual, norm_w, eps, quant, drafter) + + def is_valid( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + cap_out: Optional[torch.Tensor] = None, + cap_w: Optional[torch.Tensor] = None, + cap_eps: float = 0.0, + ) -> None: + h = residual.shape[-1] + # The op checks norm_w's element count, not its layout: a strided view is read as packed memory. + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + # The workspace is addressed as [set][parity][maxt][world][H / 2] words and never bounds-checked: a smaller + # buffer (another bucket's, or one built for fewer ranks) is polled and re-armed out of range. + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + # Flags are per (token, cluster CTA) of the bucket; the op checks the count, not that they belong to the + # workspace. Flags of another bucket hold another call history, and the parity they select is not the one the + # peers push into: the poll then waits on the other half forever or reads last call's rows. + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + + +m3_mc_allreduce_norm = M3McAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_mc_finalize_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_mc_finalize_allreduce_norm.py new file mode 100644 index 000000000000..0d457022c38f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/comm/m3_mc_finalize_allreduce_norm.py @@ -0,0 +1,197 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MoE finalize + multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode MoE boundary, 4 ranks.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_mc_allreduce_norm import boundary + + +def finalize_partial( + fc2: torch.Tensor, perm: torch.Tensor, scales: torch.Tensor, shared: Optional[torch.Tensor] +) -> torch.Tensor: + """One rank's BF16 partial: ``bf16(bf16(sum_k scale_k * fc2[perm_k]) + shared)``, the routed sum in FP32 in k + order and rounded to BF16, then the shared expert added as stock adds it (one more BF16 rounding). + + The kernel's per-k step is a fused multiply-add; it is reproduced exactly by the float64 product (exact for two + FP32 operands) plus the FP32 accumulator, rounded back to FP32. Dropped entries (perm -1) are skipped. + """ + m, topk = perm.shape + acc = torch.zeros(m, fc2.shape[1], dtype=torch.float32, device=fc2.device) + sc = scales.reshape(m, topk).float() + for k in range(topk): + rows = perm[:, k].long() + keep = (rows >= 0).unsqueeze(-1) + gathered = fc2[rows.clamp_min(0)].double() + step = (gathered * sc[:, k : k + 1].double() + acc.double()).float() + acc = torch.where(keep, step, acc) + if shared is not None: + acc = acc.to(torch.bfloat16).float() + shared.float() + return acc.to(torch.bfloat16) + + +class M3McFinalizeAllreduceNorm(OpWrapper): + """The MoE boundary with the routed experts' finalize folded in. + + Per rank the partial is ``bf16(bf16(sum_k scales[t, k] * fc2[perm[t, k]]) + shared[t])`` (``perm`` -1 = dropped, + ``scales`` fp32 or bf16), then the group's all-reduce, residual add and RMSNorm exactly as + ``m3_mc_allreduce_norm``: same workspace, flags and call-order contract. Returns ``[normed, updated]`` or, with + ``quant``, also the MXFP8 copy of ``normed``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The tp4 target takes this path from 9 tokens (top-4 of 128 experts, fp32 routing weights, the shared expert + # always present): the 16- and 48-token buckets, epilogue option 15. + CELLS: tuple[Cell, ...] = ( + Cell( + why="the smallest finalize call (9 tokens, 16-token bucket) with the next QKV's MXFP8 copy", + spec=dict(m=9, maxt=16, opts=15, topk=4, quant=True, dropped=0), + ), + Cell( + why="a full 16-token bucket with dropped expert slots", + spec=dict(m=16, maxt=16, opts=15, topk=4, quant=False, dropped=5), + ), + Cell( + why="the 48-token bucket at a partial fill", + spec=dict(m=24, maxt=48, opts=15, topk=4, quant=False, dropped=0), + ), + Cell( + why="the 48-token bucket full, MXFP8 copy", + spec=dict(m=48, maxt=48, opts=15, topk=4, quant=True, dropped=3), + ), + # Every other row count of the unfinalized MoE (modeling.MC_FINALIZE): the bucket and options the target picks. + *( + Cell( + why=f"{m} tokens: a finalize row count between the ones above", + spec=dict( + m=m, + maxt=16 if m <= 16 else 48, + opts=15, + topk=4, + quant=m % 2 == 0, + dropped=m % 4, + ), + ) + for m in range(9, 49) + if m not in (9, 16, 24, 48) + ), + ) + + note = """ + BF16 routing weights and the 8-token bucket are accepted by the op (option 7 there) but the target only enters + this path at 9 tokens and up, with the built-in MoE's fp32 weights; neither is a cell. The rest is as in + `m3_mc_allreduce_norm`'s note. + """ + + def __call__( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_sm107_tp4_mc_allreduce_norm( + fc2, + shared, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + perm, + scales, + maxt, + quant, + 0, + opts, + ) + + def reference( + self, + per_rank_fc2: Sequence[torch.Tensor], + per_rank_perm: Sequence[torch.Tensor], + per_rank_scales: Sequence[torch.Tensor], + per_rank_shared: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute, from every rank's MoE outputs in rank order (the + collective exception, as in ``m3_mc_allreduce_norm.reference``).""" + partials = [ + finalize_partial( + fc2, + per_rank_perm[r], + per_rank_scales[r], + None if per_rank_shared is None else per_rank_shared[r], + ) + for r, fc2 in enumerate(per_rank_fc2) + ] + return boundary(partials, residual, norm_w, eps, quant) + + def is_valid( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + ) -> None: + h = residual.shape[-1] + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + # The kernel gathers fc2[perm] without a bound: an index past the FC2 rows reads whatever follows them. + lo, hi = int(perm.min()), int(perm.max()) + assert lo >= -1 and hi < fc2.shape[0], ( + f"perm spans [{lo}, {hi}]; entries must be -1 (dropped) or a row of fc2 (< {fc2.shape[0]})" + ) + + +m3_mc_finalize_allreduce_norm = M3McFinalizeAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/embedding/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/embedding/__init__.py new file mode 100644 index 000000000000..e9f4d8e20b94 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/embedding/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Embedding ops of this target: the generic token-embedding gather.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/embedding/m3_embedding.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/embedding/m3_embedding.py new file mode 100644 index 000000000000..3060c17acdff --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/embedding/m3_embedding.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The generic token-embedding gather over the replicated table (an id outside the table gives a zero row).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modules.embedding import pre_comm_embedding_ops + + +class M3Embedding(OpWrapper): + """``table[ids]`` with a zero row for an id outside ``[0, V)`` (the speculative sampler pads with -1). + + The table is the full (replicated) bf16 embedding [V, 6144]; ids int32 / int64 of any count. The fused decode + embedding (``m3_embedding_firstnorm``) takes decode steps of up to 8 rows; this entry takes every other step. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="a prefill chunk, int32 ids", spec=dict(rows=2048, id_dtype=torch.int32, pad=False) + ), + Cell( + why="a 48-row decode verify, int64 ids", + spec=dict(rows=48, id_dtype=torch.int64, pad=False), + ), + Cell( + why="draft rows padded with -1 ids", spec=dict(rows=32, id_dtype=torch.int32, pad=True) + ), + ) + + note = """ + An exact gather: `reference` must match bit for bit. + """ + + def __call__(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + return pre_comm_embedding_ops(ids, table, 1, 0, None, 0, table.shape[0], False, 0) + + def reference(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] + return rows * valid.unsqueeze(-1).to(table.dtype) + + +m3_embedding = M3Embedding() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/__init__.py new file mode 100644 index 000000000000..9cc374bdf1cc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GEMM entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m31_agg_prefill_cublaslt.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m31_agg_prefill_cublaslt.py new file mode 100644 index 000000000000..1d2ea94ebd68 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m31_agg_prefill_cublaslt.py @@ -0,0 +1,133 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Opt-in pinned cuBLASLt for agg prefill: not bitwise, pending agg quality gates. + +Pins and first-use checks reuse the sm_107 tp2ctx implementation. +The scaled_mm reference certifies cuBLASLt's own numerics, not FlashInfer parity. +""" + +import os +from typing import Callable + +import torch + +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled, m31_enabled + +ENABLED = m31_enabled("AGG_PREFILL_CUBLASLT") and not m31_disabled("AGG_PREFILL_CUBLASLT") +PINS = { + (6144, 4096): ( + (8192, 12287, (66, 513, 36, 1, 0, 0, 1, 0, 6)), + (12288, 16384, (66, 513, 36, 1, 0, 0, 1, 0, 9)), + ), + (12288, 6144): ((8192, 16384, (66, 513, 36, 1, 0, 0, 3, 0, 9)),), + (6144, 6144): ( + (8192, 12287, (66, 513, 36, 1, 0, 0, 1, 0, 6)), + (12288, 16384, (66, 513, 36, 1, 0, 0, 1, 0, 9)), + ), + (3072, 6144): ((8192, 16384, (66, 513, 36, 1, 0, 0, 1, 0, 6)),), +} +_STATE = {} + + +def _op() -> ( + Callable[[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, list[int]], torch.Tensor] + | None +): + """Load the handed native op once; absence preserves the original dispatch.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_mxfp8_cublaslt_gemm", None) + if op is None: + import tensorrt_llm + + lib = os.path.join( + os.path.dirname(tensorrt_llm.__file__), "libs", "libm31_mxfp8_cublaslt.so" + ) + if os.path.exists(lib): + try: + torch.ops.load_library(lib) + op = getattr(torch.ops.trtllm, "minimax_m31_mxfp8_cublaslt_gemm", None) + except (OSError, RuntimeError) as exc: + logger.warning(f"AGG_PREFILL_CUBLASLT load failed: {exc}") + _STATE["op"] = op + return _STATE["op"] + + +def self_check(weight: torch.Tensor, scale: torch.Tensor, band: tuple) -> str | None: + """Reuse tp2ctx's deterministic, scaled_mm-exact check at both band edges.""" + n, k = weight.shape + gen = torch.Generator(device=weight.device).manual_seed(n * 31 + k) + op = _op() + if op is None: + return "native op unavailable" + for rows in (band[0], band[1]): + x = torch.randn(rows, k, device=weight.device, dtype=torch.float32, generator=gen) + exponent = torch.randint( + -12, 13, (rows, k // 32, 1), device=x.device, generator=gen + ).float() + act, act_sf = torch.ops.trtllm.mxfp8_quantize( + (x.view(rows, k // 32, 32) * torch.exp2(exponent)).view(rows, k).to(torch.bfloat16), + True, + ) + del x, exponent + try: + pinned = op(act, act_sf, weight, scale, list(band[2])) + again = op(act, act_sf, weight, scale, list(band[2])) + except RuntimeError as exc: + return f"pinned launch failed at M={rows}: {exc}" + if not torch.equal(again.view(torch.int16), pinned.view(torch.int16)): + return f"repeated pinned calls differ at M={rows}" + reference = torch._scaled_mm( + act.view(torch.float8_e4m3fn), + weight.view(torch.float8_e4m3fn).t(), + act_sf.reshape(-1).view(torch.float8_e8m0fnu), + scale.reshape(-1).view(torch.float8_e8m0fnu), + out_dtype=torch.bfloat16, + ) + if not torch.equal(pinned.view(torch.int16), reference.view(torch.int16)): + return f"pinned output differs from scaled_mm at M={rows}" + return None + + +def try_gemm( + x: torch.Tensor, + weight: torch.Tensor, + scale: torch.Tensor, + alpha: torch.Tensor, + prequant: tuple[torch.Tensor, torch.Tensor] | None, +) -> torch.Tensor | None: + """Run an eligible eager band, or leave the original WIDE path unchanged.""" + if not ENABLED or x.dim() != 2 or x.dtype != torch.bfloat16: + return None + if torch.compiler.is_compiling() or torch.cuda.is_current_stream_capturing(): + return None + rows = x.shape[0] + n, k = weight.shape + band = next((b for b in PINS.get((n, k), ()) if b[0] <= rows <= b[1]), None) + if band is None: + return None + if torch.cuda.get_device_capability(weight.device) != (10, 7): + return None + alpha_key = ("alpha", alpha.device, alpha.data_ptr()) + if alpha_key not in _STATE: + _STATE[alpha_key] = float(alpha) == 1.0 + if not _STATE[alpha_key]: + return None + key = (weight.device, n, k, band[0], band[1]) + if key not in _STATE: + why = self_check(weight, scale, band) + _STATE[key] = why is None + if why is not None: + logger.warning(f"AGG_PREFILL_CUBLASLT not taken N={n} K={k} band={band[:2]}: {why}") + if not _STATE[key]: + return None + act, act_sf = ( + prequant if prequant is not None else torch.ops.trtllm.mxfp8_quantize(x.contiguous(), True) + ) + output = _op()(act, act_sf, weight, scale, list(band[2])) + logged = ("engaged", key) + if logged not in _STATE: + _STATE[logged] = True + logger.info(f"AGG_PREFILL_CUBLASLT engaged N={n} K={k} band={band[:2]} M={rows}") + return output diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_bf16_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_bf16_linear.py new file mode 100644 index 000000000000..4fb5af737dc0 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_bf16_linear.py @@ -0,0 +1,41 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The BF16 Eagle3 drafter's projections: ``x @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3Bf16Linear(OpWrapper): + """``F.linear(x, w)`` for bf16 ``x`` [M, K] and a bf16 weight [N, K] (cuBLAS, fp32 accumulation), bf16 out. + + The drafter's fc over the captured target layers (K = 3 x 6144), its fused QKV (K = 2 x 6144, this rank's heads), + o_proj (this rank's input columns), gate_up and down projections run it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="drafter fc, 32 verify rows", spec=dict(m=32, n=6144, k=18432)), + Cell(why="drafter QKV, eight draft rows", spec=dict(m=8, n=2304, k=12288)), + Cell(why="drafter o_proj, a prefill chunk", spec=dict(m=1024, n=6144, k=2048)), + Cell(why="drafter gate_up, one row", spec=dict(m=1, n=9216, k=6144)), + Cell(why="drafter down, eight rows", spec=dict(m=8, n=6144, k=4608)), + ) + + note = """ + cuBLAS's split and accumulation order: `reference` multiplies in fp32 (one bf16 rounding), within the test's band. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x, w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return (x.float() @ w.float().t()).to(x.dtype) + + +m3_bf16_linear = M3Bf16Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_dense_swiglu.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_dense_swiglu.py new file mode 100644 index 000000000000..1fb7a1d759ae --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_dense_swiglu.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense MLP's swigluoai activation as one C++ launch with PDL and an early release (DENSE_SWIGLU_CPP).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +# The dense layers' swigluoai parameters, which the kernel hard-codes (modeling_minimaxm3._build_swiglu_oai_dense_mlp). +LIMIT, ALPHA, BETA = 7.0, 1.702, 1.0 + + +class M3DenseSwiglu(OpWrapper): + """``min(g, 7) * sigmoid(1.702 * min(g, 7)) * (clamp(u, -7, 7) + 1)`` of a bf16 [rows, 2d] gate | up tensor, + rounded once to bf16: trtllm::minimax_m3_sm107_tp4_swiglu_oai (minimaxM3Sm107Tp4/swigluOai.cu), bit for bit the + Triton ``torch.ops.trtllm.silu_and_mul`` at swiglu_limit 7, swiglu_alpha 1.702, swiglu_beta 1. With ``use_pdl`` the + grid waits on its predecessor and releases its dependents before it computes, so the down GEMV stages its weights + during the activation (the GEMV reads the activation only after its own grid wait). Contiguous rows with an even + half width; ``__call__`` returns None otherwise or for other activation parameters (the caller runs the Triton + op).""" + + ARCHS = frozenset({Arch.SM_107}) + + # The TP4 dense MLP shard: gate | up 6144 wide (d = 3072), at the decode rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="the TP4 dense MLP at 1 row", spec=dict(rows=1, d=3072)), + Cell(why="the TP4 dense MLP at 4 rows (1 request x 4 tokens)", spec=dict(rows=4, d=3072)), + Cell(why="the TP4 dense MLP at 8 rows", spec=dict(rows=8, d=3072)), + ) + + note = """ + Elementwise, one rounding: bit for bit the Triton kernel (its instruction sequence: the clamps, (u + 1), the + alpha-scaled ex2.approx sigmoid, div.full), checked over every bf16 gate and every bf16 up pattern. + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float] = LIMIT, + alpha: Optional[float] = ALPHA, + beta: Optional[float] = BETA, + use_pdl: bool = True, + ) -> Optional[torch.Tensor]: + if ( + (limit, alpha, beta) != (LIMIT, ALPHA, BETA) + or x.dtype != torch.bfloat16 + or x.dim() != 2 + or not x.is_cuda + or not x.is_contiguous() + or x.shape[1] % 4 != 0 + or x.data_ptr() % 4 != 0 + ): + return None + out = torch.empty((x.shape[0], x.shape[1] // 2), dtype=x.dtype, device=x.device) + torch.ops.trtllm.minimax_m3_sm107_tp4_swiglu_oai(x, out, bool(use_pdl)) + return out + + def reference(self, x, limit=LIMIT, alpha=ALPHA, beta=BETA, use_pdl=True) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + + def is_valid(self, x, limit=LIMIT, alpha=ALPHA, beta=BETA, use_pdl=True) -> None: + assert x.dtype == torch.bfloat16 and x.dim() == 2, "x: bf16 [rows, 2d]" + + +m3_dense_swiglu = M3DenseSwiglu() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_draft_lowm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_draft_lowm.py new file mode 100644 index 000000000000..6fb8b1572d22 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_draft_lowm.py @@ -0,0 +1,158 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""BF16 x BF16 -> BF16 low-M GEMM of the Eagle3 drafter's linears at fixed FlashInfer tactics (SM10x).""" + +from __future__ import annotations + +from typing import Mapping, Optional, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: (N, K) of the TP4 drafter shards -> ((max rows, kind, tactic), ...), rows ascending. Kind "d" is FlashInfer's direct +#: SIMT kernel with tactic (block, outputs per block, rows per block); kind "s" is its split-K tensor-core kernel with +#: tactic (mma_m, mma_n, split_k, ab_stages). A row count is served by the first entry whose max rows holds it; the +#: rows modeling.DRAFT_LOWM covers are only the ones measured (the fastest tactic at no larger error vs fp64 than +#: cuBLAS). Rubin SM107 TP4 sweep (mclk locked, >= 3x L2 of rotating weights, CUDA-graph timing), us cuBLAS -> +#: low-M at 1 / 4 rows: qkv 8.15 -> 5.90 / 8.32 -> 6.71, o 5.36 -> 3.40 / 5.44 -> 3.66, gate_up 12.93 -> 9.93 / +#: 13.03 -> 12.05, down 7.68 -> 6.16 / 7.64 -> 6.15. The fc is replicated (the TP2 shape): its TP2 tactic and row. +TACTICS: Mapping[Tuple[int, int], tuple] = { + # qkv + (2304, 12288): ((1, "d", (256, 2, 1)), (4, "s", (64, 8, 4, 10))), + # o_proj + (6144, 2048): ((1, "d", (128, 4, 1)), (4, "s", (64, 8, 1, 10))), + # gate_up + (9216, 6144): ((1, "d", (128, 2, 1)), (4, "s", (64, 8, 1, 6))), + # down + (6144, 4608): ((4, "s", (64, 8, 2, 10)),), + # the Eagle3 fc over the three captured hidden states + (6144, 18432): ((1, "d", (384, 2, 1)),), +} +#: Rows each (N, K) is certified at (a subset of what its table would serve). +ROWS: Mapping[Tuple[int, int], Tuple[int, ...]] = { + (2304, 12288): (1, 4), + (6144, 2048): (1, 4), + (9216, 6144): (1, 4), + (6144, 4608): (1, 4), + (6144, 18432): (1,), +} + + +def tactic_for(n: int, k: int, m: int) -> Optional[Tuple[str, tuple]]: + """(kind, tactic) of an [m, k] x [n, k]^T call, or None when the table does not serve it.""" + for rows, kind, tactic in TACTICS.get((n, k), ()): + if m <= rows: + return kind, tactic + return None + + +class M3DraftLowM(OpWrapper): + """``out`` [M, N] BF16 = ``x`` [M, K] BF16 @ ``weight`` [N, K]^T BF16, FP32 accumulation. + + ``kind`` / ``tactic`` pick FlashInfer's kernel: "d" runs ``run_direct_dense`` (SIMT, one dot product per output in + K order), "s" runs ``run_splitk_dense`` (tcgen05 MMA, K split over ``split_k`` CTAs and summed in FP32). Both + round once to BF16. The kernel is compiled on its first call per (M, N, K, tactic, PDL); the target compiles every + covered call before warmup (kernels/m3_draft_lowm.prebuild). + """ + + ARCHS = frozenset({Arch.SM_107}) + CELLS: tuple[Cell, ...] = ( + Cell(why="drafter qkv, 1 rows: d tactic", spec=dict(nk=(2304, 12288), m=1)), + Cell(why="drafter qkv, 4 rows: s tactic", spec=dict(nk=(2304, 12288), m=4)), + Cell(why="drafter o_proj, 1 rows: d tactic", spec=dict(nk=(6144, 2048), m=1)), + Cell(why="drafter o_proj, 4 rows: s tactic", spec=dict(nk=(6144, 2048), m=4)), + Cell(why="drafter gate_up, 1 rows: d tactic", spec=dict(nk=(9216, 6144), m=1)), + Cell(why="drafter gate_up, 4 rows: s tactic", spec=dict(nk=(9216, 6144), m=4)), + Cell(why="drafter down, 1 rows: s tactic", spec=dict(nk=(6144, 4608), m=1)), + Cell(why="drafter down, 4 rows: s tactic", spec=dict(nk=(6144, 4608), m=4)), + Cell(why="drafter fc, 1 rows: d tactic", spec=dict(nk=(6144, 18432), m=1)), + ) + + note = """ + The table is the TP4 drafter's: its shapes are the per-rank shards of the Eagle3 layer at 4-way tensor + parallelism (32 query and 2 KV heads per rank, intermediate 9216 per rank) and the replicated fc, at the b1 + decode rows (1 row in draft passes 1-2, 4 in pass 0). Certified at those shapes and rows only; FlashInfer's + kernels take other shapes, which are not measured here. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + own_splitk: bool = False, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + if kind == "d": + from flashinfer.gemm.kernels.dense_bf16_gemm_direct import ( + DirectTactic, + run_direct_dense, + ) + + run_direct_dense(x, weight.t(), out, use_pdl, DirectTactic(*tactic)) + elif own_splitk: + # This target's copy of the split-K kernel (kernels/m3_lowm_splitk): bit for bit FlashInfer's; with ``ar`` + # it also pushes the rows into a drafter boundary all-reduce (modeling.DRAFT_AR_PUSH). + from ...kernels.m3_lowm_splitk import ( + SplitKTactic, + run_splitk_dense, + run_splitk_dense_push, + ) + + if ar is not None: + run_splitk_dense_push(x, weight.t(), out, use_pdl, SplitKTactic(*tactic), ar) + else: + run_splitk_dense(x, weight.t(), None, out, use_pdl, SplitKTactic(*tactic)) + else: + from flashinfer.gemm.kernels.dense_bf16_gemm_sm100_splitk import ( + SplitKTactic, + run_splitk_dense, + ) + + run_splitk_dense(x, weight.t(), None, out, use_pdl, SplitKTactic(*tactic)) + return out + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + own_splitk: bool = False, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """bf16(x @ weight^T) summed in fp64 and rounded once; ``out``, ``kind``, ``tactic`` and ``use_pdl`` only + choose where and how the op writes.""" + return (x.double() @ weight.double().t()).to(torch.bfloat16) + + def is_valid( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + own_splitk: bool = False, + ar: Optional[tuple] = None, + ) -> None: + """The kernels index ``x`` and ``out`` densely: a strided view is read as if it were packed.""" + assert x.dim() == 2 and x.is_contiguous(), "x must be a contiguous [M, K] matrix" + assert weight.dim() == 2 and weight.is_contiguous(), ( + "weight must be a contiguous [N, K] matrix" + ) + assert out.is_contiguous() and tuple(out.shape) == (x.shape[0], weight.shape[0]), ( + "out must be a contiguous [M, N] matrix" + ) + assert ar is None or (own_splitk and kind == "s" and len(ar) == 5), ( + "ar (the boundary all-reduce push) runs only on this target's split-K copy" + ) + + +m3_draft_lowm = M3DraftLowM() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_fp32_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_fp32_linear.py new file mode 100644 index 000000000000..9702cc0778a4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_fp32_linear.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The router's fp32 projection beyond the decode routers' rows: ``float(x) @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + + +class M3Fp32Linear(OpWrapper): + """fp32 router logits ``F.linear(float(x[M, 6144]), w[128, 6144])`` for prefill and steps above 48 rows (the + fp32 GEMV and the tensor-core router take 1..48). Whatever fp32 matmul precision the process runs with applies + (the engine leaves torch's default).""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="a prefill chunk", spec=dict(m=2048, e=128, k=6144)), + Cell( + why="a generation step above the tensor-core router's 48 rows", + spec=dict(m=64, e=128, k=6144), + ), + ) + + note = """ + Under torch's default fp32 matmul precision the product may be TF32: the test's band covers that rounding against + the true fp32 reference. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x.to(torch.float32), w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + with true_fp32_matmul(): + return x.float() @ w.t() + + +m3_fp32_linear = M3Fp32Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mx_gemv.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mx_gemv.py new file mode 100644 index 000000000000..b1bc51fed18f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mx_gemv.py @@ -0,0 +1,171 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 x MXFP8 GEMV (tcgen05, block-scaled, CuTe DSL) on an activation its producer already quantized.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.mx_gemv import op as _gemv +from ..moe.m3_moe_engine import mxfp8_dequant + +__all__ = ["PROJECTIONS", "gemv_is_valid", "m3_mx_gemv", "served_cells"] + +# (N, K) of this target's MXFP8 projections at tp4, the GEMV's tuned shapes. +QKV_SPARSE = (4992, 6144) +QKV = (4608, 6144) +O_PROJ = (6144, 4096) +SHARED_GATE_UP = (3072, 6144) +SHARED_DOWN = (6144, 1536) +DENSE_GATE_UP = (12288, 6144) +DENSE_DOWN = (6144, 6144) +PROJECTIONS = { + "sparse-layer QKV": QKV_SPARSE, + "dense-attention QKV": QKV, + "o_proj": O_PROJ, + "shared gate_up": SHARED_GATE_UP, + "shared down": SHARED_DOWN, + "dense gate_up": DENSE_GATE_UP, + "dense down": DENSE_DOWN, +} +# VR_GEMV48: the projections the GEMV also serves on 33..48-row generation steps, where it beats cuBLASLt's MXFP8 +# GEMM (SM107 us/call at 33..48 rows on the same MXFP8 activation: QKV_SPARSE 6.34..6.35 vs 6.94..6.95, +# SHARED_GATE_UP 5.87..5.88 vs 6.81..6.82; the dense-attention QKV is the sparse one's shape class). +GEMV48_SHAPES = (QKV_SPARSE, QKV, SHARED_GATE_UP) +GEMV48_MIN_TOKENS = 33 # capacity: fixed: past the R1-VR-TP8 decode bound (32 rows) + + +def served_cells(named: tuple[Cell, ...], x_bf16: bool, max_m: int) -> tuple[Cell, ...]: + """A cell for every (projection, rows) the GEMV serves up to ``max_m`` rows that ``named`` does not list. + + Certification is per projection: a projection's split-K, ring and MMA width differ, so each (N, K, M) the + decode dispatch can send here is its own cell. + """ + listed = {(cell.spec["nk"], cell.spec["m"]) for cell in named} + return tuple( + Cell(why=f"{name}, {m} rows", spec=dict(nk=nk, m=m)) + for name, nk in PROJECTIONS.items() + for m in range(1, max_m + 1) + if (nk, m) not in listed and _gemv.serves(nk[0], nk[1], m, x_bf16) + ) + + +def gemv_reference( + x: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor +) -> torch.Tensor: + """bf16(x @ dequant(weight)^T) for fp32 activation values x [M, K], summed in fp64 and rounded once.""" + return (x.double() @ mxfp8_dequant(weight, weight_scale).double().t()).to(torch.bfloat16) + + +def gemv_is_valid(weight: torch.Tensor, weight_scale: torch.Tensor) -> None: + n, k = weight.shape + # The kernel reads both from data_ptr() as dense buffers at the layout it was compiled for. + assert weight.dtype == torch.float8_e4m3fn and weight.is_contiguous(), ( + "weight must be contiguous e4m3 [N, K]" + ) + assert weight_scale.numel() == n * k // 32, ( + "weight_scale must be the 128x4-swizzled UE8M0 buffer [N * K / 32]" + ) + + +class M3MxGemv(OpWrapper): + """bf16 [M, N] = xq [M, K] (e4m3, 128x4-swizzled UE8M0 scales ``xsf``) x weight [N, K]^T (e4m3, swizzled + ``weight_scale``), M <= 48; None when the shape is not served (the caller falls back). + + ``trigger_early``: release the PDL dependents before the epilogue. ``cfg``: a (split, ring[, contig, nacc, a_tmem, + st_async]) tuning override; default the table's. ``use_pdl``: default the environment's. ``ar`` = (multicast + workspace address, flags address, rank, world, bucket tokens, mode): also push the rows into the next boundary + all-reduce's exchange (then ``op.last_pushed`` is True). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The projections' (N, K) at the token counts each build differs in: the MMA width (8 / 16 / 32 / 48 columns) a + # call uses is the smallest that holds M, and the shared gate_up takes another split at 2 tokens and above 8. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token (split 4)", spec=dict(nk=QKV_SPARSE, m=1)), + Cell(why="sparse-layer QKV, 8 tokens", spec=dict(nk=QKV_SPARSE, m=8)), + Cell( + why="sparse-layer QKV, 48 verify rows: the widest MMA", spec=dict(nk=QKV_SPARSE, m=48) + ), + Cell(why="dense-attention QKV, one token", spec=dict(nk=QKV, m=1)), + Cell(why="dense-attention QKV, 13 rows: the 16-column MMA", spec=dict(nk=QKV, m=13)), + Cell(why="o_proj, one token (split 2)", spec=dict(nk=O_PROJ, m=1)), + Cell(why="o_proj, 24 rows: the 32-column MMA", spec=dict(nk=O_PROJ, m=24)), + Cell( + why="shared gate_up, one token (A from shared memory)", + spec=dict(nk=SHARED_GATE_UP, m=1), + ), + Cell(why="shared gate_up, 2 tokens: its split-4 config", spec=dict(nk=SHARED_GATE_UP, m=2)), + Cell( + why="shared gate_up, 12 rows: the wide split-4 config", + spec=dict(nk=SHARED_GATE_UP, m=12), + ), + Cell(why="shared down, one token (unsplit, K 768)", spec=dict(nk=SHARED_DOWN, m=1)), + Cell(why="shared down, 8 tokens", spec=dict(nk=SHARED_DOWN, m=8)), + Cell(why="dense gate_up, one token", spec=dict(nk=DENSE_GATE_UP, m=1)), + Cell(why="dense down, 8 tokens", spec=dict(nk=DENSE_DOWN, m=8)), + ) + # Every other (projection, rows) the decode dispatch can send here (modeling.MX_GEMV_ROWS). + CELLS = CELLS + served_cells(CELLS, x_bf16=False, max_m=48) + + note = """ + The kernel sums in fp32 in split-K order where `reference` sums in fp64: the bf16 outputs agree within the band the + test derives. Not covered: the all-reduce push (`ar`), which needs a multicast exchange; `cfg` overrides beyond the + table (the target passes none). Shapes outside the table, M > 48, and 17..32 rows at K < 1024 return None. + """ + + def __call__( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv(xq, xsf, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The dequantized product; the launch options are ignored.""" + return gemv_reference(mxfp8_dequant(xq, xsf), weight, weight_scale) + + def is_valid( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + m, k = xq.shape + assert xq.dtype == torch.float8_e4m3fn and xq.is_contiguous(), ( + "xq must be contiguous e4m3 [M, K]" + ) + # A linear-layout [M, K / 32] scale buffer is read as the swizzled one: wrong scales, no error. + assert xsf.numel() >= (m + 127) // 128 * 128 * (k // 32), ( + "xsf must be the 128x4-swizzled scale buffer (rows padded to 128), not the linear one" + ) + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv = M3MxGemv() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mx_gemv_bf16.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mx_gemv_bf16.py new file mode 100644 index 000000000000..046be527589a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mx_gemv_bf16.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 GEMV (CuTe DSL) on a bf16 activation, quantized to MXFP8 in the kernel's prologue.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.mx_gemv import op as _gemv +from ..moe.m3_moe_engine import mxfp8_quant_dequant +from .m3_mx_gemv import ( + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, + gemv_is_valid, + gemv_reference, + served_cells, +) + +__all__ = ["m3_mx_gemv_bf16"] + + +class M3MxGemvBf16(OpWrapper): + """bf16 [M, N] = mxfp8_quantize(x) x weight [N, K]^T for a contiguous, 16-byte aligned bf16 ``x`` [M, K]; the + quantization is mxfp8_quantize's (UE8M0 scale per 32 = the smallest power of two >= amax / 448). None when the + shape is not served. The other arguments as ``m3_mx_gemv``'s. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The prologue quantizes one 32-element block per thread and round: the projections at the token counts the + # target sends here (no producer-quantized activation), up to the widest each serves. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token", spec=dict(nk=QKV_SPARSE, m=1)), + Cell( + why="sparse-layer QKV, 16 rows: the most its split-4 prologue serves", + spec=dict(nk=QKV_SPARSE, m=16), + ), + Cell(why="dense-attention QKV, 8 tokens", spec=dict(nk=QKV, m=8)), + Cell(why="o_proj, one token", spec=dict(nk=O_PROJ, m=1)), + # TP2: o_proj's K is 4096 (split 2: 16 k-tiles), so its prologue serves up to 16 rows (the TP4 cell's 32-column + # MMA at K 2048 is past the 8-round bound here; modeling.MX_GEMV_BF16_MAX_TOKENS caps o_proj at 16). + Cell( + why="o_proj, 16 rows: the most its split-2 prologue serves at K 4096", + spec=dict(nk=O_PROJ, m=16), + ), + Cell(why="shared gate_up, one token", spec=dict(nk=SHARED_GATE_UP, m=1)), + Cell(why="shared gate_up, 2 tokens: its split-4 config", spec=dict(nk=SHARED_GATE_UP, m=2)), + Cell(why="shared down, 8 tokens (K 768)", spec=dict(nk=SHARED_DOWN, m=8)), + Cell( + why="dense gate_up, 8 tokens: the most split 2 at K 6144 serves", + spec=dict(nk=DENSE_GATE_UP, m=8), + ), + Cell(why="dense down, one token", spec=dict(nk=DENSE_DOWN, m=1)), + ) + # Every other (projection, rows) the decode dispatch can send here (modeling.MX_GEMV_BF16_MAX_TOKENS). + CELLS = CELLS + served_cells(CELLS, x_bf16=True, max_m=32) + + note = """ + As `m3_mx_gemv`. The in-kernel quantization is mxfp8_quantize's bit for bit except where the kernel's + rcp.approx(448) and an exact 1/448 put a block's amax on different sides of a power of two (not observed on the + cells). Not covered: `ar`, `cfg` overrides. Returns None for rows beyond 8 rounds of the prologue per CTA. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv_bf16(x, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The MXFP8 round trip of x times the dequantized weight; the launch options are ignored.""" + return gemv_reference(mxfp8_quant_dequant(x), weight, weight_scale) + + def is_valid( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv_bf16 = M3MxGemvBf16() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_cublaslt.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_cublaslt.py new file mode 100644 index 000000000000..652343ac88cd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_cublaslt.py @@ -0,0 +1,127 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MXFP8 GEMM through cuBLASLt for generation steps wider than the decode kernels' rows. + +cuBLASLt's block-scaled MXFP8 matmul (``torch._scaled_mm`` with UE8M0 block scales), on an MXFP8 activation or on a +bf16 one it quantizes first. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner + +from .m3_mx_gemv import DENSE_DOWN, DENSE_GATE_UP, O_PROJ, QKV_SPARSE, SHARED_DOWN, SHARED_GATE_UP +from .m3_mxfp8_linear import mxfp8_reference + +# The projections the cuBLASLt GEMM takes, and its rows: past the R1-VR-TP8 decode bound (32) up to the K7 verify rows +# at batch 8 (64). SM107 microbench (us/call on the same MXFP8 activation, 33..64 rows): QKV_SPARSE +# 6.94..6.98 vs FlashInfer's 10.14..10.18, O_PROJ 3.06..3.10 vs 3.81..3.86, SHARED_GATE_UP 6.81..6.82 vs 10.04..10.08, +# SHARED_DOWN 2.49..2.51 vs 2.86..2.87, DENSE_GATE_UP 7.21..7.24 vs 10.46..10.47, DENSE_DOWN 3.62..3.77 vs 4.48..4.64. +SHAPES = (QKV_SPARSE, O_PROJ, SHARED_GATE_UP, SHARED_DOWN, DENSE_GATE_UP, DENSE_DOWN) +MIN_TOKENS = 33 # capacity: fixed: past the R1-VR-TP8 decode bound (32 rows) +MAX_TOKENS = 64 # capacity: fixed: the K7 verify rows at batch 8 + +_QUANTIZE = {} + + +def mxfp8_quantize(x: torch.Tensor): + """The bf16 activation's MXFP8 form by FlashInfer's CuTe DSL quantize. + + E4M3 [M, K] + 128x4-swizzled UE8M0 scales: the bytes of ``torch.ops.trtllm.mxfp8_quantize(x, True)``. + """ + runner = _QUANTIZE.get(x.dtype) + if runner is None: + runner = _QUANTIZE[x.dtype] = MXFP8QuantizeRunner(x.dtype) + act, act_scale = runner([x], tactic=MXFP8QuantizeRunner.CUTE_DSL) + return act.view(x.shape[0], x.shape[1]), act_scale + + +def _e4m3(t: torch.Tensor) -> torch.Tensor: + return t if t.dtype == torch.float8_e4m3fn else t.view(torch.float8_e4m3fn) + + +def _ue8m0(scale: torch.Tensor) -> torch.Tensor: + """The swizzled scale bytes as one flat UE8M0 vector (whatever element type the buffer was allocated with).""" + return scale.reshape(-1).view(torch.uint8).view(torch.float8_e8m0fnu) + + +class M3Mxfp8Cublaslt(OpWrapper): + """``act @ weight^T`` in bf16 for an MXFP8 weight [N, K] (swizzled scales), via cuBLASLt. + + The activation is MXFP8 (``act`` e4m3 [M, K] + ``act_scale`` swizzled) or, with ``act_scale`` None, bf16 and + quantized first. The GEMM is cuBLASLt's block-scaled MXFP8 matmul (VEC32 UE8M0 scales), reached through + ``torch._scaled_mm``; the algorithm is cuBLASLt's heuristic for the shape and the process's workspace (no + timing-based selection). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = tuple( + Cell(why=f"[{n}, {k}], {m} rows, {src}", spec=dict(m=m, n=n, k=k, prequant=src == "MXFP8")) + for (n, k) in SHAPES + for m in range(MIN_TOKENS, MAX_TOKENS + 1) + for src in ("MXFP8", "bf16") + ) + + note = """ + Exactly the MXFP8 product (fp32 accumulation, one bf16 rounding): `reference` dequantizes both operands and + multiplies in fp32; the test's band covers the accumulation order. Against FlashInfer's MXFP8 GEMM on the same + operands (SM107) the output was within 0.61 ULP[bf16] at every shape and row, bit-identical on most cells. + Not covered: shapes outside SHAPES, rows outside 33..64. + """ + + def __call__( + self, + act: torch.Tensor, + act_scale: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + ) -> torch.Tensor: + if act_scale is None: + act, act_scale = mxfp8_quantize(act) + return torch._scaled_mm( + _e4m3(act), + _e4m3(weight).t(), + scale_a=_ue8m0(act_scale), + scale_b=_ue8m0(weight_scale), + out_dtype=torch.bfloat16, + ) + + def reference( + self, + act: torch.Tensor, + act_scale: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + ) -> torch.Tensor: + """The dequantized product of the MXFP8 operands (a bf16 activation quantized first).""" + if act_scale is None: + act, act_scale = mxfp8_quantize(act) + return mxfp8_reference(act, act_scale, weight, weight_scale).to(torch.bfloat16) + + def is_valid( + self, + act: torch.Tensor, + act_scale: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + ) -> None: + m, k = act.shape + n = weight.shape[0] + assert (n, k) in SHAPES and MIN_TOKENS <= m <= MAX_TOKENS, ( + f"m3_mxfp8_cublaslt is certified for {SHAPES} at {MIN_TOKENS}..{MAX_TOKENS} rows, got [{n}, {k}] x {m}" + ) + assert act.is_contiguous() and weight.is_contiguous(), ( + "m3_mxfp8_cublaslt takes contiguous operands" + ) + assert act_scale is not None or act.dtype == torch.bfloat16, ( + "a bf16 activation, or an MXFP8 one + scales" + ) + + +m3_mxfp8_cublaslt = M3Mxfp8Cublaslt() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_gemm.py new file mode 100644 index 000000000000..0b422a7167e5 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_gemm.py @@ -0,0 +1,58 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 GEMM on an activation its producer already quantized (rows the GEMV does not serve).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import mxfp8_gemm_autotuned + +from .m3_mxfp8_linear import mxfp8_reference + + +class M3Mxfp8Gemm(OpWrapper): + """``act @ weight^T`` for an MXFP8 activation (e4m3 [M, K] + 128x4-swizzled UE8M0 scales) and an MXFP8 weight + [N, K] (swizzled scales), bf16 out. The GEMM backend per token bucket is the AutoTuner's (the same runner, op key + and tactic cache as the engine-tuned quantize + GEMM); ``tune`` profiles in the engine's tuning pass.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The decode dispatch hands it the boundary / merge MXFP8 activation on rows the GEMV does not serve: the + # unsplit shared down projection at 17..32 rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert down projection, 17 rows", spec=dict(m=17, n=6144, k=768)), + Cell(why="shared expert down projection, 32 rows", spec=dict(m=32, n=6144, k=768)), + ) + + note = """ + Exactly the MXFP8 product (fp32 accumulation, one bf16 rounding): `reference` dequantizes both operands and + multiplies in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + return mxfp8_gemm_autotuned(act, act_scale, weight, weight_scale, output_dtype, tune=tune) + + def reference( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + """The dequantized product; ``tune`` is ignored.""" + return mxfp8_reference(act, act_scale, weight, weight_scale).to(output_dtype) + + +m3_mxfp8_gemm = M3Mxfp8Gemm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_linear.py new file mode 100644 index 000000000000..d7a70d725189 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_linear.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One MXFP8 projection on the engine-tuned path: dynamic MXFP8 quantize of a bf16 activation + block-scaled GEMM.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import mxfp8_quantize_gemm_autotuned + +_BLOCK = 32 + + +def _unswizzle(scale: torch.Tensor, rows: int, cols: int) -> torch.Tensor: + """[rows, cols] UE8M0 bytes from the 128x4-swizzled layout (rows padded to 128, columns to 4).""" + pad_r = (rows + 127) // 128 * 128 + pad_c = (cols + 3) // 4 * 4 + s = scale.view(torch.uint8).reshape(pad_r // 128, pad_c // 4, 32, 4, 4) + return s.permute(0, 3, 2, 1, 4).reshape(pad_r, pad_c)[:rows, :cols] + + +def _dequant(x: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + rows, k = x.shape + s = _unswizzle(scale, rows, k // _BLOCK).to(torch.int32) + factor = torch.pow(2.0, (s - 127).to(torch.float32)) + return ( + x.view(torch.float8_e4m3fn).float().view(rows, k // _BLOCK, _BLOCK) * factor.unsqueeze(-1) + ).view(rows, k) + + +def mxfp8_reference(act, act_scale, weight, weight_scale) -> torch.Tensor: + """fp32 product of two dequantized MXFP8 operands (swizzled UE8M0 scales).""" + with true_fp32_matmul(): + return _dequant(act, act_scale) @ _dequant(weight, weight_scale).t() + + +class M3Mxfp8Linear(OpWrapper): + """``x @ weight^T`` for bf16 ``x`` [M, K] and an MXFP8 weight [N, K] (e4m3 + 128x4-swizzled UE8M0 scales), bf16 + out, on the backend the engine selected: + + * ``tuned_quantize_gemm`` (decode CUDA graphs once the engine tuned per-bucket backends): the AutoTuner's quantize + and GEMM per token bucket (``tune`` profiles them in the engine's tuning pass); + * otherwise the native ``mxfp8_quantize`` (swizzled scales) and then FlashInfer's MXFP8 GEMM (``flashinfer``) or + the CUTLASS block-scaled GEMM, through the autotuner while the engine tunes it (``native_tuning``), scaled by + ``alpha`` (fp32 [1], 1.0 for these checkpoints). + + Every MXFP8 projection of the target (the attention QKV and o_proj, the dense MLPs and the shared expert) runs it + outside the decode-step kernels. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="sparse-layer QKV, a prefill chunk, native GEMM", + spec=dict(m=2048, n=2560, k=6144, path="native"), + ), + Cell( + why="o_proj, a prefill chunk, native GEMM", + spec=dict(m=2048, n=6144, k=2048, path="native"), + ), + Cell( + why="dense down projection, eager decode, native GEMM", + spec=dict(m=4, n=6144, k=3072, path="native"), + ), + Cell( + why="shared gate_up, FlashInfer GEMM", + spec=dict(m=64, n=1536, k=6144, path="flashinfer"), + ), + Cell(why="o_proj, tuned quantize + GEMM", spec=dict(m=64, n=6144, k=2048, path="tuned")), + ) + + note = """ + Exactly the MXFP8 product of the MXFP8 round trip of x (fp32 accumulation, one bf16 rounding): `reference` + quantizes x with the same block rule (UE8M0 scale = the block's power-of-two ceiling over e4m3's 448) and + multiplies dequantized operands in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + if tuned_quantize_gemm: + return mxfp8_quantize_gemm_autotuned(x, weight, weight_scale, output_dtype, tune=tune) + act, act_scale = torch.ops.trtllm.mxfp8_quantize(x, True) + if flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8( + act, act_scale, weight, weight_scale, output_dtype + ) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, weight, weight_scale, alpha, output_dtype) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + """The MXFP8 round trip of ``x`` times the dequantized weight, times ``alpha``; the backend flags are + ignored.""" + m, k = x.shape + blocks = x.float().view(m, k // _BLOCK, _BLOCK) + amax = blocks.abs().amax(-1, keepdim=True) + exp = torch.ceil(torch.log2(torch.where(amax > 0, amax / 448.0, torch.ones_like(amax)))) + scale = torch.pow(2.0, exp) + q = (blocks / scale).to(torch.float8_e4m3fn).float() * scale + with true_fp32_matmul(): + out = q.view(m, k) @ _dequant(weight, weight_scale).t() + return (out * alpha.float()).to(output_dtype) + + +m3_mxfp8_linear = M3Mxfp8Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_pinned_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_pinned_gemm.py new file mode 100644 index 000000000000..ee781b16ff42 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_pinned_gemm.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 GEMM at a fixed FlashInfer tactic (MX-TAC): the same choice on every rank and every start.""" + +from __future__ import annotations + +from typing import Dict, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import FlashInferMXFP8GemmRunner + +from .m3_mxfp8_linear import mxfp8_reference + +# One runner per output dtype (a runner holds no per-call state). +_RUNNERS: Dict[Tuple[type, torch.dtype], object] = {} + + +def _runner(dtype: torch.dtype): + runner = _RUNNERS.get((FlashInferMXFP8GemmRunner, dtype)) + if runner is None: + runner = _RUNNERS[(FlashInferMXFP8GemmRunner, dtype)] = FlashInferMXFP8GemmRunner(dtype) + return runner + + +class M3Mxfp8PinnedGemm(OpWrapper): + """``act @ weight^T`` for an MXFP8 activation (e4m3 [M, K] + swizzled UE8M0 scales) and an MXFP8 weight [N, K] + (swizzled scales) on FlashInfer's MXFP8 GEMM at ``tactic``, no AutoTuner; ``output_dtype`` out.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The decode graph's MXFP8 GEMMs on rows the GEMVs do not serve: the shared down projection on its boundary + # activation (17..32 rows) and the bf16 projections past the GEMV rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert down projection, 17 rows", spec=dict(m=17, n=6144, k=768)), + Cell(why="packed QKV + index projection, 48 rows", spec=dict(m=48, n=4992, k=6144)), + ) + + note = """ + The MXFP8 product (fp32 accumulation, one rounding to the output dtype): `reference` dequantizes both operands and + multiplies in fp32; the test's band covers the accumulation order. The tactic only picks the kernel. + """ + + def __call__( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tactic: int, + ) -> torch.Tensor: + return _runner(output_dtype).forward([act, act_scale, weight, weight_scale], tactic=tactic) + + def reference( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tactic: int, + ) -> torch.Tensor: + """The dequantized product; ``tactic`` is ignored.""" + return mxfp8_reference(act, act_scale, weight, weight_scale).to(output_dtype) + + +m3_mxfp8_pinned_gemm = M3Mxfp8PinnedGemm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_pinned_quantize.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_pinned_quantize.py new file mode 100644 index 000000000000..3f35af65048a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_mxfp8_pinned_quantize.py @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 activation quantize at a fixed tactic (MX-TAC), ahead of ``m3_mxfp8_pinned_gemm``.""" + +from __future__ import annotations + +from typing import Dict, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner + +_RUNNERS: Dict[torch.dtype, object] = {} + + +def _runner(dtype: torch.dtype): + runner = _RUNNERS.get(dtype) + if runner is None: + runner = _RUNNERS[dtype] = MXFP8QuantizeRunner(dtype) + return runner + + +class M3Mxfp8PinnedQuantize(OpWrapper): + """A bf16 activation [M, K] to MXFP8: (e4m3 [M, K], swizzled UE8M0 scales), on the runner's ``tactic``.""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="packed QKV + index projection input, 48 rows", spec=dict(m=48, k=6144)), + Cell(why="shared expert down projection input, 17 rows", spec=dict(m=17, k=768)), + ) + + note = """ + Bit-exact: `reference` is the stock MXFP8 quantize (``torch.ops.trtllm.mxfp8_quantize``, swizzled scales) of the + same input; every tactic computes the same block maxima and roundings. + """ + + def __call__(self, x: torch.Tensor, tactic: int) -> Tuple[torch.Tensor, torch.Tensor]: + return _runner(x.dtype).forward([x], tactic=tactic) + + def reference(self, x: torch.Tensor, tactic: int) -> Tuple[torch.Tensor, torch.Tensor]: + """The stock quantize; ``tactic`` is ignored.""" + return torch.ops.trtllm.mxfp8_quantize(x, True) + + +m3_mxfp8_pinned_quantize = M3Mxfp8PinnedQuantize() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_target_head.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_target_head.py new file mode 100644 index 000000000000..f8deb858c843 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gemm/m3_target_head.py @@ -0,0 +1,346 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The target lm_head shard GEMV of a verify step of at most 8 rows over a lossless 12-bit coding of its BF16 weight, +and that coding (``encode_chunked`` / ``decode_reference`` / ``round_trip_ok``). + +The coding: the BF16 exponents of a trained weight sit almost all in a window of 15 consecutive values +``[base, base + 14]``. Each weight is one byte ``sign << 7 | mantissa`` plus a 4-bit code ``exponent - base``; code 15 +marks an exception, which decodes to +0 (it adds nothing to the MMA sums); its exact product ``x * w`` is added from a +per-tile list. The bytes are laid out in ``mma.m16n8k16`` A-fragment order: +per 16-row tile and unit of 4 k16 steps, each lane reads the sign/mantissa bytes of steps 0-1, of steps 2-3, then 4 +code words (``[tiles][K / 64][3][32 lanes][16 bytes]``). A coded weight is used only after ``round_trip_ok`` proved +it decodes to the BF16 weight bit for bit. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_target_head"] + +#: Verify rows per call (requests x (draft length + 1)). +MAX_ROWS = 8 +#: Warps per CTA (the split-tile scratch holds 2 x 4 x 32 fp32 per warp). +CTA_WARPS = 16 +#: The target lm_head's TP4 shard (200064-id vocabulary) and the hidden size. +TARGET_SHARD = 50016 +HIDDEN = 6144 +#: Codes per window: 0..14 decode, 15 marks an exception. +WINDOW = 15 +#: The largest exponent base whose window top plus one (base + 15) is still a finite BF16 exponent. +MAX_BASE = 254 - WINDOW +#: Rows per block of encode_chunked / round_trip_ok (bounded temporaries). +ROWS_PER_CHUNK = 4096 + + +def _pad_rows(weight: torch.Tensor) -> torch.Tensor: + n = weight.shape[0] + tiles = (n + 15) // 16 + if tiles * 16 == n: + return weight + return torch.cat([weight, weight.new_zeros(tiles * 16 - n, weight.shape[1])]) + + +def _fragment_view(w: torch.Tensor) -> torch.Tensor: + """[tiles * 16, K] -> [tiles, K / 16, 32 lanes, 8 values] in mma A-fragment order (a permutation of the values).""" + n, k = w.shape + # rows: (tile, rh, g); cols: (ks, ch, j, e) -> (tile, ks, g, j, ch, rh, e) + return ( + w.view(n // 16, 2, 8, k // 16, 2, 4, 2) + .permute(0, 3, 2, 5, 4, 1, 6) + .reshape(n // 16, k // 16, 32, 8) + ) + + +def _exponents(weight: torch.Tensor) -> torch.Tensor: + return (weight.contiguous().view(torch.int16).to(torch.int32) >> 7) & 0xFF + + +def _window_base(hist: torch.Tensor) -> int: + """The base of the 15-exponent window holding the most weights (the lowest such base).""" + window = torch.stack([hist[b : b + WINDOW].sum() for b in range(0, MAX_BASE + 1)]) + return int(window.argmax()) + + +def encode(weight: torch.Tensor, base: Optional[int] = None): + """Lossless 12-bit coding of a BF16 [N, K] weight (see the module docstring); ``base`` defaults to this weight's + own window. + + Returns (packed uint8 [tiles * K / 64 * 1536], base, exc_off int32 [tiles + 1], exc_pos int32 [E] (row in tile + << 16 | k), exc_w fp32 [E, 2] ({w, 0})). + """ + w = _pad_rows(weight) + n, k = w.shape + tiles, ks = n // 16, k // 16 + assert ks % 4 == 0, "K must be a multiple of 64" + bits = w.contiguous().view(torch.int16).to(torch.int32) & 0xFFFF + expo = (bits >> 7) & 0xFF + if base is None: + base = _window_base(torch.bincount(expo.flatten(), minlength=256)) + code = expo - base + exc = (code < 0) | (code >= WINDOW) + code = torch.where(exc, torch.full_like(code, WINDOW), code) + sm = ((bits >> 8) & 0x80) | (bits & 0x7F) + sm = torch.where(exc, torch.zeros_like(sm), sm) + # Exceptions: (tile, row in tile, k) in tile-major order; code 15 decodes to +0. + rows, cols = exc.nonzero(as_tuple=True) + order = torch.argsort((rows // 16).long() * (16 * k) + (rows % 16).long() * k + cols.long()) + rows, cols = rows[order], cols[order] + exc_w = torch.stack( + [ + w[rows, cols].float(), + torch.zeros((rows.numel(),), device=w.device), + ], + dim=1, + ) + exc_pos = (((rows % 16) << 16) | cols).to(torch.int32) + exc_off = torch.zeros(tiles + 1, dtype=torch.int32, device=w.device) + exc_off[1:] = torch.cumsum(torch.bincount((rows // 16).long(), minlength=tiles), 0).to( + torch.int32 + ) + # Fragment order, then per (tile, group of 4 k16 steps, lane): 32 sign/mantissa bytes and 4 code words. + smf = _fragment_view(sm.to(torch.uint8).view(n, k)).view(tiles, ks // 4, 4, 32, 8) + cf = ( + _fragment_view(code.to(torch.uint8).view(n, k)) + .view(tiles, ks // 4, 4, 32, 8) + .to(torch.int32) + ) + # Code word of a k16 step: value 2p -> bits 4p..4p+3, value 2p+1 -> bits 16+4p.. + cw = torch.zeros(tiles, ks // 4, 4, 32, dtype=torch.int64, device=w.device) + for p in range(4): + cw |= (cf[..., 2 * p].long() << (4 * p)) | (cf[..., 2 * p + 1].long() << (16 + 4 * p)) + cw = cw.to(torch.int32) # wraps for bit 31; the kernel reads it as uint32 + # Word 0 of a lane = the bytes of steps 0 and 1; word 1 = steps 2 and 3; word 2 = the code words of steps 0..3. + seg01 = ( + smf.permute(0, 1, 3, 2, 4).reshape(tiles, ks // 4, 32, 2, 16).permute(0, 1, 3, 2, 4) + ) # [.., 2, 32, 16] + seg2 = cw.permute(0, 1, 3, 2).contiguous().view(torch.uint8).view(tiles, ks // 4, 1, 32, 16) + packed = torch.cat([seg01.contiguous(), seg2], dim=2) # [tiles, ks / 4, 3, 32, 16] + return packed.contiguous().view(-1), base, exc_off, exc_pos, exc_w.contiguous() + + +def encode_chunked(weight: torch.Tensor, rows_per_chunk: int = ROWS_PER_CHUNK): + """``encode`` over row blocks (bounded temporaries), with the exponent window chosen on the whole weight.""" + n, _ = weight.shape + assert rows_per_chunk % 16 == 0 + hist = torch.zeros(256, dtype=torch.int64, device=weight.device) + for r in range(0, n, rows_per_chunk): + hist += torch.bincount(_exponents(weight[r : r + rows_per_chunk]).flatten(), minlength=256) + base = _window_base(hist) + packed, offs, poss, ws = [], [], [], [] + n_exc = 0 + for r in range(0, n, rows_per_chunk): + p, _, off, pos, ew = encode(weight[r : r + rows_per_chunk], base=base) + packed.append(p) + offs.append(off[:-1] + n_exc) + poss.append(pos) + ws.append(ew) + n_exc += pos.numel() + exc_off = torch.cat(offs + [torch.tensor([n_exc], dtype=torch.int32, device=weight.device)]) + return torch.cat(packed), base, exc_off, torch.cat(poss), torch.cat(ws) + + +def decode_reference(packed: torch.Tensor, base: int, n: int, k: int) -> torch.Tensor: + """BF16 [N, K] from the coded stream with the in-kernel rule (exceptions, code 15, decode to +0).""" + tiles, ks = (n + 15) // 16, k // 16 + p = packed.view(tiles, ks // 4, 3, 32, 16) + sm = ( + p[:, :, :2] + .permute(0, 1, 3, 2, 4) + .reshape(tiles, ks // 4, 32, 4, 8) + .permute(0, 1, 3, 2, 4) + .to(torch.int32) + ) + cw = ( + p[:, :, 2] + .contiguous() + .view(torch.int32) + .view(tiles, ks // 4, 32, 4) + .permute(0, 1, 3, 2) + .long() + & 0xFFFFFFFF + ) + codes = torch.stack([(cw >> (4 * (i // 2) + 16 * (i % 2))) & 0xF for i in range(8)], dim=-1).to( + torch.int32 + ) + bits = ((sm & 0x80) << 8) | ((codes + base) << 7) | (sm & 0x7F) + bits = torch.where(codes == WINDOW, torch.zeros_like(bits), bits) + frag = bits.reshape(tiles, ks, 32, 8) + # Inverse of _fragment_view: (tile, ks, g, j, ch, rh, e) -> (tile, rh, g, ks, ch, j, e). + w = frag.view(tiles, ks, 8, 4, 2, 2, 2).permute(0, 5, 2, 1, 4, 3, 6).reshape(tiles * 16, k) + return w.to(torch.int16).view(torch.bfloat16)[:n] + + +def decode( + packed: torch.Tensor, + base: int, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + n: int, + k: int, + row0: int = 0, + row1: Optional[int] = None, +) -> torch.Tensor: + """BF16 rows [row0, row1) of the coded weight [N, K] with the exceptions applied (what the kernel multiplies by). + ``row0`` must be a multiple of 16.""" + row1 = n if row1 is None else row1 + assert row0 % 16 == 0 + t0, t1 = row0 // 16, (row1 + 15) // 16 + unit_bytes = (k // 64) * 3 * 32 * 16 + dec = decode_reference(packed[t0 * unit_bytes : t1 * unit_bytes], base, row1 - row0, k).clone() + counts = (exc_off[1:] - exc_off[:-1]).long() + tile_of = torch.repeat_interleave(torch.arange(counts.numel(), device=packed.device), counts) + rows = tile_of * 16 + (exc_pos >> 16).long() + cols = (exc_pos & 0xFFFF).long() + sel = (rows >= row0) & (rows < row1) + dec[rows[sel] - row0, cols[sel]] = exc_w.view(-1, 2)[sel, 0].to(torch.bfloat16) + return dec + + +def round_trip_ok( + packed: torch.Tensor, + base: int, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + weight: torch.Tensor, + rows_per_chunk: int = ROWS_PER_CHUNK, +) -> bool: + """Whether the coded weight (code values + exception list) decodes to ``weight`` bit for bit, with its exponent + window below the BF16 infinity exponent.""" + n, k = weight.shape + if not 0 <= base <= MAX_BASE or k % 64: + return False + if ( + packed.numel() != ((n + 15) // 16) * 16 * k * 3 // 2 + or exc_off.numel() != (n + 15) // 16 + 1 + ): + return False + for r0 in range(0, n, rows_per_chunk): + r1 = min(n, r0 + rows_per_chunk) + dec = decode(packed, base, exc_off, exc_pos, exc_w, n, k, r0, r1) + if not torch.equal(dec.view(torch.int16), weight[r0:r1].contiguous().view(torch.int16)): + return False + return True + + +class M3TargetHead(OpWrapper): + """``out[:M, :n]`` (BF16) = x [M, K] (BF16, M <= 8) times the coded BF16 shard W [n, K] transposed: each logit + the fp32 sum of the exact BF16 products, rounded once to BF16 (the in-tree BF16 head GEMM's arithmetic, in + another summation order). + + ``w`` / ``exp_base`` / ``exc_off`` / ``exc_pos`` / ``exc_w`` are ``encode_chunked``'s outputs for W; ``part`` + (fp32 [grid * 16 * 256]) and ``counter`` (int32 [ceil(n / 16)], zero between calls, re-armed by the kernel) are + the split-tile scratch; ``grid`` is the number of 16-warp CTAs (one per SM). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The verify of a fused-verify step at most 8 rows wide: 1 + 3 drafts for one or two requests (4 / 8 rows), and + # every other row count the kernel's one n8 MMA tile serves; one cell with an exception in many tiles, one with + # a shard whose rows are not a multiple of 16. + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"{m} verify row(s) of the TP4 target head shard", + spec=dict(m=m, n=TARGET_SHARD, k=HIDDEN), + ) + for m in range(1, MAX_ROWS + 1) + ), + Cell( + why="4 rows with outliers and exact zeros in many tiles (the exception lists)", + spec=dict(m=4, n=TARGET_SHARD, k=HIDDEN, exceptions=True), + ), + Cell( + why="2 rows over a shard of 4000 rows (250 tiles), K 1024", + spec=dict(m=2, n=4000, k=1024, exceptions=True), + ), + ) + + note = """ + The weight the kernel multiplies by is the checkpoint's BF16 weight bit for bit (round_trip_ok); an exception + adds nothing to the MMA sums and its exact product x * w is added at its tile's flush. A logit therefore differs + from the in-tree head's only by the fp32 summation order before its single BF16 rounding (ulp-level); + integer-valued data, whose sums are exact in any order, is equal bit for bit. Split tiles are summed in warp + order: the result is deterministic. + """ + + def __call__( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_target_head( + w, x, out, part, counter, exc_off, exc_pos, exc_w, exp_base, n, grid, pdl + ) + return out + + def reference( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> torch.Tensor: + """The BF16 logits [M, n] of x against the decoded weight: the fp32 product, rounded once to BF16. + + ``out``, ``part``, ``counter``, ``grid`` and ``pdl`` do not change the result.""" + weight = decode(w, exp_base, exc_off, exc_pos, exc_w, n, x.shape[1]) + with true_fp32_matmul(): + logits = x.float() @ weight.float().t() + return logits.to(torch.bfloat16) + + def is_valid( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> None: + # A split tile's last finisher is the CTA whose arrival makes the count reach its owners: a stale count + # stores a partial sum. + assert not bool(counter[: (n + 15) // 16].any()), "counter must be zero between calls" + # The exceptions are read per tile in list order with no bound against the coded stream. + assert int(exc_off[-1].item()) == exc_pos.numel(), "exc_off must end at the exception count" + assert bool((exc_off[1:] >= exc_off[:-1]).all()), "exc_off must be non-decreasing" + assert out.data_ptr() != x.data_ptr(), "out must not alias x" + + +m3_target_head = M3TargetHead() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gqamix_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gqamix_sparse_decode.py new file mode 100644 index 000000000000..b415e7be538c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/gqamix_sparse_decode.py @@ -0,0 +1,1337 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. +""" + +from __future__ import annotations + +import os +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +def _pdl_enabled() -> bool: + return os.environ.get("TRTLLM_ENABLE_PDL", "1") == "1" + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for _ in tl.range(chunk_start, chunk_end): + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for _ in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on the datacenter Blackwell and Rubin targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3), (10, 7)) + + +# Devices already found to support the NVFP4 decode kernels, by capability. +_NVFP4_DEVICE_CAPABILITIES: dict[torch.device, tuple[int, int]] = {} + + +def _nvfp4_device_capability(device: torch.device) -> tuple[int, int]: + """Return the device's capability, checking NVFP4 support once per device.""" + capability = _NVFP4_DEVICE_CAPABILITIES.get(device) + if capability is None: + capability = torch.cuda.get_device_capability(device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103/SM107; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _NVFP4_DEVICE_CAPABILITIES[device] = capability + return capability + + +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + # Checked once per device; the shape heuristics below consult it per call. + capability = _nvfp4_device_capability(q.device) if kv_nvfp4 else None + if kv_nvfp4: + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else 1 + ) + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = _pdl_enabled() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + if not direct_output: + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + **pdl_launch, + **merge_launch_options, + ) + + +__all__ = [ + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/__init__.py new file mode 100644 index 000000000000..77d7b84d9bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mixture-of-experts entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m31_s1b_fold.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m31_s1b_fold.py new file mode 100644 index 000000000000..84f4e58207b4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m31_s1b_fold.py @@ -0,0 +1,173 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""S1b fold: the decode router above the tensor-core router's rows, routed from top-k ids. + +Stock: ``F.linear(float(h), w)`` (cuBLASLt split-K TF32 GEMM + its splitKreduce), then the trtllm-gen MiniMax2 routing +selects the top-4 from the logits. Here: ONE strided-batched TF32 GEMM over S K-slices (partials), then +``trtllm::minimax_m31_s1b_fold`` sums them in slice order (bitwise cuBLASLt's reduce at the split factor its heuristic +picks for the shape) and runs the routing library's own MiniMax2 selection (the production E128K8 instance), so the +top-4 ids (slot order) and bf16 weights are bitwise the stock routing's; the MoE runner then routes from them +(RoutingFromTopKIds). VR microbench: -1.7 / -2.6 us per layer at 88 / 176 rows. + +The split factor and the weights operand's layout (a strided view of the router weight, or a laid-out copy) are pinned +together per row count at prebuild, by a bitwise probe of this device's cuBLASLt; every pin is then re-validated with +its final operand before capture (a failure drops the row count, which keeps the stock path). Kill switch +TRTLLM_M31_DISABLE_S1B_FOLD=1. +""" + +from __future__ import annotations + +from typing import Dict, Optional, Tuple + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import m31_fake_ops +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled + +S1B_FOLD = not m31_disabled("S1B_FOLD") +SPLITS = ( + 16, + 8, +) # the fold kernel's built split factors (cuBLASLt's choices at 49..448 rows on sm_107) +HIDDEN, EXPERTS, TOP_K = 6144, 128, 4 +PROBES = 3 +_STATE: dict = {} +_PINS: Dict[int, int] = {} # row count -> split factor +_LAYOUT: Dict[ + int, bool +] = {} # row count -> True: strided view, False: laid-out copy (probed together with its S) +_LAID: Dict[Tuple[int, int], torch.Tensor] = {} +_LOGGED: set = set() + + +def fold_op(): + """``trtllm::minimax_m31_s1b_fold`` from the separately built library, never loaded during capture.""" + if "op" not in _STATE: + if torch.cuda.is_current_stream_capturing(): + return None + op = getattr(torch.ops.trtllm, "minimax_m31_s1b_fold", None) + if op is not None: + m31_fake_ops.register() + _STATE["op"] = op + return _STATE["op"] + + +def x_slices(x: torch.Tensor, s: int) -> torch.Tensor: + """fp32 [rows, 6144] -> [S, rows, 6144 / S] as a view (no copy).""" + return x.view(x.shape[0], s, HIDDEN // s).transpose(0, 1) + + +def w_strided(w: torch.Tensor, s: int) -> torch.Tensor: + """fp32 [128, 6144] -> [S, 6144 / S, 128] as a view (no copy).""" + return w.view(EXPERTS, s, HIDDEN // s).permute(1, 2, 0) + + +def _w_operand(w: torch.Tensor, s: int, strided: bool) -> torch.Tensor: + return w_strided(w, s) if strided else _LAID[(w.data_ptr(), s)] + + +def _slice_sum(p: torch.Tensor) -> torch.Tensor: + r = p[0].clone() + for k in range(1, p.shape[0]): + r = r + p[k] + return r + + +def _bitwise(rows: int, s: int, w: torch.Tensor, w_op: torch.Tensor, gen: torch.Generator) -> bool: + for _ in range(PROBES): + x = torch.randn(rows, HIDDEN, generator=gen, device=w.device).to(torch.bfloat16).float() + got = _slice_sum(torch.bmm(x_slices(x, s), w_op)) + if not torch.equal(got.view(torch.int32), F.linear(x, w).view(torch.int32)): + return False + return True + + +def prebuild(router_weights, rows_range) -> Dict[int, int]: + """Pin, for every decode row count in ``rows_range`` (outside capture, before warmup), a split factor S AND the + weights operand's layout whose bmm + slice-order sum is bitwise this device's F.linear on random inputs: the + strided view first, else a laid-out copy (one per layer and S, built only for the S some row count needs). Then + every pin is re-validated with exactly the operand it will execute (its own S and layout, the final copies); + a failure drops that row count. Returns row count -> S (empty: the fold is off or not built).""" + _PINS.clear() + _LAYOUT.clear() + _LAID.clear() + if not S1B_FOLD or fold_op() is None or not router_weights: + logger.info( + f"[M3.1] S1B_FOLD not taken: {'switched off' if not S1B_FOLD else 'library not built'}" + ) + return dict(_PINS) + w0 = router_weights[0] + gen = torch.Generator(device=w0.device).manual_seed(20261102) + rows_list = list(rows_range) + for rows in rows_list: + for s in SPLITS: + if _bitwise(rows, s, w0, w_strided(w0, s), gen): + _PINS[rows], _LAYOUT[rows] = s, True + break + for rows in rows_list: + if rows in _PINS: + continue + for s in SPLITS: + if _bitwise(rows, s, w0, w_strided(w0, s).contiguous(), gen): + _PINS[rows], _LAYOUT[rows] = s, False + break + for s in {v for r, v in _PINS.items() if not _LAYOUT[r]}: + for w in router_weights: + _LAID[(w.data_ptr(), s)] = w_strided(w, s).contiguous() + # Final validation with the operand each pin executes (no pin runs a layout it was not probed with). + dropped = [ + r + for r in sorted(_PINS) + if not _bitwise(r, _PINS[r], w0, _w_operand(w0, _PINS[r], _LAYOUT[r]), gen) + ] + for r in dropped: + del _PINS[r], _LAYOUT[r] + torch.cuda.synchronize() + groups = {} + for r, s in _PINS.items(): + groups.setdefault((s, _LAYOUT[r]), []).append(r) + logger.info( + "[M3.1] S1B_FOLD pinned " + + ", ".join( + f"S{s} {'strided' if st else 'laid out'}: {len(v)} row counts ({min(v)}..{max(v)})" + for (s, st), v in sorted(groups.items()) + ) + + f"; {len(rows_list) - len(_PINS)} row counts keep the stock router" + + (f" ({len(dropped)} dropped by the final validation)" if dropped else "") + ) + return dict(_PINS) + + +def route( + x_f32: torch.Tensor, w: torch.Tensor, bias: torch.Tensor, routed_scaling: float, pdl: bool +) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """(top-k ids int32 [rows, 4], weights bf16 [rows, 4]) for fp32 router input ``x_f32``, or None (no pin, or an + operand outside the fold's layout): the caller keeps the stock logits path.""" + rows = x_f32.shape[0] if x_f32.dim() == 2 else 0 + s = _PINS.get(rows) + if ( + s is None + or x_f32.dtype != torch.float32 + or x_f32.shape[1] != HIDDEN + or x_f32.stride(1) != 1 + or x_f32.stride(0) != HIDDEN + or w.dtype != torch.float32 + or tuple(w.shape) != (EXPERTS, HIDDEN) + or not w.is_contiguous() + or bias.dtype != torch.float32 + or bias.numel() != EXPERTS + ): + return None + partials = torch.bmm(x_slices(x_f32, s), _w_operand(w, s, _LAYOUT[rows])) + ids = torch.empty(rows, TOP_K, dtype=torch.int32, device=x_f32.device) + weights = torch.empty(rows, TOP_K, dtype=torch.bfloat16, device=x_f32.device) + torch.ops.trtllm.minimax_m31_s1b_fold( + partials, bias.contiguous(), None, ids, weights, routed_scaling, pdl + ) + if rows not in _LOGGED: + _LOGGED.add(rows) + logger.info(f"[M3.1] S1B_FOLD taken at {rows} rows (S {s})") + return ids, weights diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe16_engine.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe16_engine.py new file mode 100644 index 000000000000..871f6887b7a5 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe16_engine.py @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3's MoE layer for 9..13 decode rows in one CuTe DSL launch (``m3_moe16``): the router, the routed NVFP4 +experts and the shared MXFP8 expert, on caller-owned state (an ``M3Moe16State`` and one ``M3MoeLayer`` per MoE +layer).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.m3_moe.op import M3MoeLayer +from ...kernels.m3_moe.op16 import M3Moe16State +from .m3_moe_engine import moe_reference + +__all__ = ["M3Moe16State", "m3_moe16_engine"] + + +class M3Moe16Engine(OpWrapper): + """One MiniMax-M3 MoE layer for M (9..13 here; the state takes 9..16) decode rows: + ``layer.state.run(layer, hidden, out, arp)``, bf16(shared + routed) [M, 6144]. + + The same arguments and the same math as ``m3_moe_engine`` (the routed part operation for operation, the shared + expert always folded in) on an ``M3Moe16State``: a different schedule for more rows. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # One build per row count (m_max is a trace-time constant); the target routes 9..13 rows here (spec-decode verify + # of 2 x 6 rows lands at 12). + CELLS: tuple[Cell, ...] = ( + Cell(why="9 rows: the smallest build, a partial last slot group", spec=dict(tokens=9)), + Cell(why="10 rows: its own build (m_max is a trace-time constant)", spec=dict(tokens=10)), + Cell(why="11 rows: its own build, an odd row count", spec=dict(tokens=11)), + Cell(why="12 rows: two requests verifying 5-token drafts", spec=dict(tokens=12)), + Cell(why="13 rows: the largest the target routes here", spec=dict(tokens=13)), + ) + + note = """ + Bands as `m3_moe_engine` (same approximations, same quantization boundaries). Not covered: 14..16 rows, which the + state takes and this target never routes here; the all-reduce push (`arp`). No `is_valid`, as `m3_moe_engine`. + """ + + def __call__( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return layer.state.run(layer, hidden, out, arp=arp) + + def reference( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """``m3_moe_engine``'s reference with the shared expert; `out` and `arp` are ignored.""" + return moe_reference(hidden, layer, shared=True) + + +m3_moe16_engine = M3Moe16Engine() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe_engine.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe_engine.py new file mode 100644 index 000000000000..a819bb85fd92 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe_engine.py @@ -0,0 +1,328 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3's MoE layer for 1..8 decode tokens in one CuTe DSL launch (``m3_moe``): the router, the routed NVFP4 +experts and the shared MXFP8 expert, on caller-owned state (an ``M3MoeState`` and one ``M3MoeLayer`` per MoE layer).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.quantization.utils.fp4_utils import ( + get_reorder_rows_for_gated_act_gemm_row_indices, + get_shuffle_matrix_a_row_indices, +) + +from ...kernels.m3_moe.op import ( + HIDDEN, + INTER, + SHARED_INTER, + TOP_K, + M3MoeLayer, + M3MoeState, + ar_push_desc, + is_supported, + shared_ctas_for, +) + +__all__ = [ + "M3MoeLayer", + "M3MoeState", + "ar_push_desc", + "is_supported", + "m3_moe_engine", + "moe_reference", + "moe_route", + "mxfp8_dequant", + "mxfp8_quant_dequant", + "shared_ctas_for", + "unswizzle_sf", +] + +# The model's constants the kernels are built with: routed_scaling_factor, the shared expert's SwiGLU-OAI +# (swiglu_alpha, swiglu_limit, the (up + 1) offset). +ROUTED_SCALING = 2.0 +SHARED_ALPHA = 1.702 +SHARED_LIMIT = 7.0 + +_LOG2E = 1.4426950216293334961 +_RCP6 = 0.16666667163372039795 # fp32(1 / 6), the kernels' reciprocal of the E2M1 maximum +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) +_F32_MAX = 3.40282346638528859812e38 + + +def _f32(x: float) -> torch.Tensor: + return torch.tensor(x, dtype=torch.float32) + + +def _e2m1_decode(packed: torch.Tensor) -> torch.Tensor: + """fp32 values of packed E2M1 bytes [..., K / 2] (low nibble first) -> [..., K].""" + lut = torch.tensor(_E2M1, dtype=torch.float32, device=packed.device) + lo = lut[(packed & 0xF).long()] + hi = lut[(packed >> 4).long()] + return torch.stack((lo, hi), dim=-1).flatten(-2) + + +def _e2m1_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to the nearest E2M1 value, ties to even, saturating at 6 (cvt.rn.satfinite.e2m1x2).""" + a = x.abs() + q = torch.full_like(a, 6.0) + for hi, value, inclusive in ( + (5.0, 4.0, True), + (3.5, 3.0, False), + (2.5, 2.0, True), + (1.75, 1.5, False), + (1.25, 1.0, True), + (0.75, 0.5, False), + (0.25, 0.0, True), + ): + q = torch.where(a <= hi if inclusive else a < hi, torch.full_like(a, value), q) + return torch.where(x < 0, -q, q) + + +def _e4m3_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to E4M3, nearest even, saturating (cvt.rn.satfinite.e4m3x2), as fp32.""" + return x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn).float() + + +def _ue8m0_up(x: torch.Tensor) -> torch.Tensor: + """UE8M0 code of x >= 0 rounded toward +inf, saturating (cvt.rp.satfinite.ue8m0x2), int32.""" + bits = x.contiguous().view(torch.int32) + code = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + return code.clamp(0, 254) + + +def unswizzle_sf(sf: torch.Tensor, rows: int, blocks: int) -> torch.Tensor: + """The [rows, blocks] scale bytes of a 128x4-swizzled scale buffer (int64 indices, any byte dtype).""" + flat = sf.reshape(-1).view(torch.uint8) + r = torch.arange(rows, device=flat.device).view(-1, 1) + b = torch.arange(blocks, device=flat.device).view(1, -1) + padded_blocks = (blocks + 3) // 4 * 4 + idx = ( + (r // 128) * (128 * padded_blocks) + + (b // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (b % 4) + ) + return flat[idx] + + +def mxfp8_dequant(q: torch.Tensor, sf: torch.Tensor) -> torch.Tensor: + """fp32 values of e4m3 ``q`` [R, K] with its 128x4-swizzled UE8M0 scales (one per 32 elements).""" + rows, k = q.shape + codes = unswizzle_sf(sf, rows, k // 32).float() + return q.view(torch.float8_e4m3fn).float() * torch.exp2(codes - 127.0).repeat_interleave( + 32, dim=1 + ) + + +def mxfp8_quant_dequant(x: torch.Tensor) -> torch.Tensor: + """x [R, K] through MXFP8 (one UE8M0 scale per 32, the smallest power of two >= amax / 448; e4m3 data), back to + fp32 -- mxfp8_quantize's recipe, which every MXFP8 activation here reproduces.""" + rows, k = x.shape + blocks = x.float().view(rows, k // 32, 32) + code = _ue8m0_up(blocks.abs().amax(dim=-1) * _f32(1.0 / 448.0).to(x.device)) + scale = torch.exp2(code.float() - 127.0).unsqueeze(-1) + return (_e4m3_round(blocks / scale) * scale).view(rows, k) + + +def _bf16(x: torch.Tensor) -> torch.Tensor: + return x.to(torch.bfloat16).float() + + +def _gemm(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """a [M, K] @ b [N, K]^T, accumulated in fp64 and rounded once to fp32.""" + return (a.double() @ b.double().t()).float() + + +def _row_perm(rows: int, gated: bool) -> torch.Tensor: + """The TRTLLM-Gen row order: stored row j holds logical row perm[j] (the gated interleave, then the 32-row shuffle + of epilogue tile 128).""" + probe = torch.empty(rows, 1) + perm = get_shuffle_matrix_a_row_indices(probe, epilogue_tile_m=128) + if gated: + perm = get_reorder_rows_for_gated_act_gemm_row_indices(probe)[perm] + return perm + + +def _nvfp4_expert(w: torch.Tensor, sf: torch.Tensor, perm_inv: torch.Tensor) -> torch.Tensor: + """One expert's stored NVFP4 matrix (values [R, K / 2], swizzled e4m3 scales) as raw fp32 [R, K] in logical row + order (the global scale is not applied: it is the per-expert alpha).""" + rows, half = w.shape + codes = unswizzle_sf(sf, rows, 2 * half // 16).view(torch.float8_e4m3fn).float() + vals = _e2m1_decode(w) * codes.repeat_interleave(16, dim=1) + return vals[perm_inv] + + +def moe_route(x: torch.Tensor, router_weight: torch.Tensor, bias: torch.Tensor): + """MiniMax2 routing on the fp32 router GEMV: (top-4 expert ids [M, 4] int64, bf16-rounded weights [M, 4] fp32).""" + logits = _gemm(x, router_weight) + sig = (0.5 * torch.tanh(0.5 * logits).double() + 0.5).float() + keys = sig + bias + ids = torch.sort(keys, dim=-1, descending=True, stable=True).indices[:, :TOP_K] + sk = keys.gather(1, ids) - bias[ids] + total = (sk[:, 0] + sk[:, 2]) + (sk[:, 1] + sk[:, 3]) + w = (sk * _f32(ROUTED_SCALING).to(x.device)) / (total + _f32(1e-20).to(x.device)).unsqueeze(1) + return ids, _bf16(w) + + +def _nvfp4_act(x: torch.Tensor, sc: torch.Tensor) -> torch.Tensor: + """The activation's NVFP4 quantization (fp4_quantize(x, sc, 16)), as raw fp32 values (q x block scale).""" + rows, k = x.shape + blocks = x.view(rows, k // 16, 16) + vmax = blocks.abs().amax(dim=-1, keepdim=True) + sff = _e4m3_round(sc * (vmax * _f32(_RCP6).to(x.device))) + # The kernels' rcp.approx(sf x rcp.approx(sc)) lands where the correctly rounded sc / sf does on every element + # measured (the 1 / (sf x (1 / sc)) spelling differs on ~0.06% of them). + osc = torch.where(vmax != 0, sc / sff, torch.zeros_like(vmax)) + return (_e2m1_round(blocks * osc) * sff).view(rows, k) + + +def _fc1_epilogue(up: torch.Tensor, gate: torch.Tensor, esc: torch.Tensor) -> torch.Tensor: + """TRTLLM-Gen's SwiGLU-OAI FC1 epilogue on the raw FC1 sums (fp32 [I]) and its NVFP4 requantization: the raw FC2 + input values. ``esc`` = (fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha) of the expert, with + act_beta and act_clamp already divided by fc31_alpha (the loader's convention).""" + a, c, alpha, beta, lim = (esc[i] for i in range(5)) + gc = torch.where(gate > lim, lim, gate) + gc = torch.where(gate < -_F32_MAX, torch.full_like(gc, -_F32_MAX), gc) + uc = torch.minimum(torch.maximum(up, -lim), lim) + c1 = a * (alpha * _f32(_LOG2E).to(up.device)) + sgm = 1.0 / (torch.exp2(-(gc * c1)) + 1.0) + h = (uc.double() * a.double() + (a * beta).double()).float() * (gc * sgm) + blocks = h.view(-1, 16) + amax = (blocks.abs() * c).amax(dim=-1, keepdim=True) + sff = _e4m3_round(amax * _f32(_RCP6).to(up.device)) + osc = torch.where(sff != 0, 1.0 / sff, torch.zeros_like(sff)) + return (_e2m1_round((blocks * osc) * c) * sff).view(-1) + + +def _shared_expert(x: torch.Tensor, shared_mxfp8) -> torch.Tensor: + """The shared expert as the target runs it outside the engine: MXFP8 gate_up (bf16 out), SwiGLU-OAI with the + model's alpha / limit / (up + 1) in fp32 (bf16 out), MXFP8 down (bf16 out).""" + gu_w, gu_sf, dn_w, dn_sf = shared_mxfp8 + gu = _bf16(_gemm(mxfp8_quant_dequant(x), mxfp8_dequant(gu_w, gu_sf))) + gate, up = gu[:, :SHARED_INTER], gu[:, SHARED_INTER:] + a = torch.minimum(gate, torch.full_like(gate, SHARED_LIMIT)) + ub = up.clamp(-SHARED_LIMIT, SHARED_LIMIT) + 1.0 + sg = 1.0 / ( + torch.exp2((a * _f32(-SHARED_ALPHA).to(x.device)) * _f32(_LOG2E).to(x.device)) + 1.0 + ) + h = _bf16(ub * (sg * a)) + return _bf16(_gemm(mxfp8_quant_dequant(h), mxfp8_dequant(dn_w, dn_sf))) + + +def moe_reference(hidden: torch.Tensor, layer: M3MoeLayer, shared: bool) -> torch.Tensor: + """The MoE layer of the decode engines (both token ranges compute this), in plain torch: bf16 [M, 6144].""" + x = hidden.float() + m = x.shape[0] + ids, weights = moe_route(x, layer.router_weight, layer.bias) + xr = _nvfp4_act(x, layer.fc31_input_scale.reshape(()).float()) + inv31 = torch.argsort(_row_perm(2 * INTER, gated=True)).to(x.device) + inv2 = torch.argsort(_row_perm(HIDDEN, gated=False)).to(x.device) + y = torch.zeros(m, TOP_K, HIDDEN, dtype=torch.float32, device=x.device) + for e in torch.unique(ids).tolist(): + w31 = _nvfp4_expert(layer.w3_w1_weight[e], layer.w3_w1_weight_scale[e], inv31) + w2 = _nvfp4_expert(layer.w2_weight[e], layer.w2_weight_scale[e], inv2) + esc = layer.escal[:, e] + for t, k in (ids == e).nonzero().tolist(): + acc1 = _gemm(xr[t : t + 1], w31)[0] + # The buffer holds [w3 (up); w1 (gate)]. + h = _fc1_epilogue(acc1[:INTER], acc1[INTER:], esc) + y[t, k] = _bf16(_gemm(h.unsqueeze(0), w2)[0] * esc[5]) + acc = torch.zeros(m, HIDDEN, dtype=torch.float64, device=x.device) + for k in range(TOP_K): + acc = (acc + weights[:, k : k + 1].double() * y[:, k].double()).float().double() + routed = acc.float() + if not shared: + return routed.to(torch.bfloat16) + return (_shared_expert(x, layer.shared_mxfp8) + _bf16(routed)).to(torch.bfloat16) + + +class M3MoeEngine(OpWrapper): + """One MiniMax-M3 MoE layer for M (1..8) decode tokens: ``layer.state.run(layer, hidden, out, arp)``. + + ``hidden`` bf16 [M, 6144] contiguous, M = the state's token count. ``layer`` (from ``M3MoeState.layer``) carries the + TRTLLM-Gen NVFP4 expert buffers read in place, the per-expert scalars, the router weight / bias, the activation's + NVFP4 global scale and, on a ``shared`` state, the shared expert's MXFP8 weights; its state carries the compiled + kernel and the workspace every layer on it shares. Returns bf16 [M, 6144]: bf16(shared + routed) on a ``shared`` + state, else the routed partial. ``out``: write there instead of a new tensor. ``arp`` (``ar_push_desc``): also + push the rows into a multicast all-reduce exchange. + + The routed part is TRTLLM-Gen's NVFP4 MoE on the fp32 router logits: sigmoid (tanh form) + correction bias top-4 + of 128 (ties to the lower id), weights sigmoid / sum * 2.0 rounded to bf16; the activation quantized to NVFP4 with + the layer's global scale; per expert the raw FC1 sums, SwiGLU-OAI (alpha / beta / clamp), the NVFP4 requant with + fc31_scale_c, FC2, bf16(fc2_alpha x sum); the fma chain over the top-4 in order. The shared expert is the target's + MXFP8 SwiGLU-OAI MLP. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # One cell per token count: m_max is a trace-time constant, so each is its own build, and the target builds all + # eight (decode batches of 1..8 requests, spec-decode verify rows up to 8). The whys name what the build differs in. + CELLS: tuple[Cell, ...] = ( + Cell( + why="1 token: pre-tiled shared expert, last phase balanced without trailing FC2", + spec=dict(tokens=1), + ), + Cell( + why="2 tokens: pre-tiled shared expert, the one build without AR_PRE", + spec=dict(tokens=2), + ), + Cell( + why="3 tokens: 4 FC2-B builder warps, the shared expert from swizzled scales", + spec=dict(tokens=3), + ), + Cell( + why="4 tokens: one token per routing warp, logits replicated 128x", spec=dict(tokens=4) + ), + Cell( + why="5 tokens: 5 builder warps, a routing warp takes two tokens, logits 32x", + spec=dict(tokens=5), + ), + Cell(why="6 tokens: one request verifying a 5-token draft", spec=dict(tokens=6)), + Cell(why="7 tokens: the B2IMG global FC1 -> FC2 hand-off", spec=dict(tokens=7)), + Cell( + why="8 tokens: B2IMG with a 6-stage ring, 32 distinct experts at most", + spec=dict(tokens=8), + ), + ) + + note = """ + The kernel's elementwise math uses the PTX approximations (rcp.approx, ex2.approx, tanh) where `reference` uses + torch's correctly rounded ones, and its GEMMs sum in tensor-core order where `reference` sums in fp64: a + quantization boundary (NVFP4 activation / intermediate code, routing near-tie) can fall on the other side, so the + two agree within the band the test derives, not bit for bit. The routed part is bit for bit the stock TRTLLM-Gen + NVFP4 MoE (fp4_quantize + fp4_block_scale_moe_runner) on the same routing; the test checks that too. + + Not covered: the all-reduce push (`arp`), which needs a multicast exchange (the decode forward's boundary all-reduce + exercises it); the non-shared state (`shared=False`), which this target never builds. Every layer on one state + must run in one stream order: the workspace is shared and tagged per call, never reset. + + No `is_valid`: the compiled call checks every buffer's dtype, shape and alignment against its signature and + raises on a mismatch (a misaligned `hidden`, an `out` that is not bf16 [M, 6144], a malformed `arp`). + """ + + def __call__( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return layer.state.run(layer, hidden, out, arp=arp) + + def reference( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """Plain torch over the layer's buffers (see the class docstring); `out` and `arp` are ignored.""" + return moe_reference(hidden, layer, shared=layer.state.shared) + + +m3_moe_engine = M3MoeEngine() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe_tactic_pin.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe_tactic_pin.py new file mode 100644 index 000000000000..50bdd3ce741a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_moe_tactic_pin.py @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2 WIDE decode tactics on the existing TRTLLM-Gen NVFP4 MoE runner. + +The caller retains quantization, routing logits and finalization. Only the tactic +choice changes. Invalid configurations decline to the existing tuner path. +""" + +from tensorrt_llm._torch.custom_ops.trtllm_gen_custom_ops import FP4BlockScaleMoERunner +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled + +ENABLED = not m31_disabled("TP2W_MOE_TACTIC_PIN") + + +def tactic_for_rows(rows: int): + """The CTX v2 natural-routing table; no interpolation outside its bands.""" + if 96 <= rows <= 127: + return (16, 128) + if rows == 232: + return (32, 64) + if rows == 256: + return (32, 96) + return None + + +def run_pinned(core, inputs, do_finalize: bool, routing_method_type: int, act_type: int, topk=None): + """Run with a validated tactic, or return None so the caller keeps its tuner. + + inputs are the stock runner's first sixteen tensor arguments, in their + original order. Routing uses the logits (inputs[0]), or with ``topk`` (int32 ids, bf16 weights; S1B_FOLD, + inputs[0] None) routes from those ids. + The caller limits this adapter to decode; prefill keeps the original path. + """ + if not ENABLED: + return None + x = inputs[2] + rows = x.shape[0] + tactic = tactic_for_rows(rows) + if tactic is None: + return None + if (core.mapping.tp_size, core.hidden, core.moe_inter, core.num_experts, core.top_k) != ( + 2, + 6144, + 1536, + 128, + 4, + ): + return None + assert len(inputs) == 16 and (inputs[0] is not None) != (topk is not None) + assert x.shape[1] * 2 == core.hidden + config = ( + core.num_experts, + core.top_k, + None, + None, + core.moe_inter, + 0, + core.num_experts, + core.routed_scaling_factor, + routing_method_type, + do_finalize, + act_type, + ) + key = (x.device, rows, config, core.max_num_tokens) + cache = core.__dict__.setdefault("_tp2w_moe_tactic_pins", {}) + if key not in cache: + runner = FP4BlockScaleMoERunner( + *config, tune_max_num_tokens=core.max_num_tokens, use_dp=False + ) + valid = runner.get_runner().get_valid_configs( + core.top_k, core.hidden, core.moe_inter, core.num_experts, rows + ) + accepted = tactic in {tuple(v) for v in valid} + cache[key] = runner if accepted else None + status = "taken" if accepted else "declined" + logger.info( + f"[M3.1] TP2W_MOE_TACTIC_PIN {status} at {rows} rows tactic {list(tactic)} " + f"on rank {core.mapping.tp_rank} finalize={do_finalize}" + ) + runner = cache[key] + if runner is None: + return None + tw, ti = (None, None) if topk is None else (topk[1], topk[0]) + return runner.forward([*inputs, tw, ti], tactic=list(tactic)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_router_tc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_router_tc.py new file mode 100644 index 000000000000..bfc04b06d366 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/moe/m3_router_tc.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TF32 tensor-core router of the decode MoE for 2..48 tokens, optionally with the NVFP4 activation.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +# ROUTER48 at TP2: the sm_107 TP8 wrapper (cells 2..48) with the TP4 wrapper's inline NVFP4 reference quantize +# (this target has no m3_router_gemv module). +_E4M3_MAX = 448.0 +_E2M1_MAX = 6.0 +#: Midpoints between consecutive e2m1 magnitudes; ``_TIE_UP[i]``: round-to-nearest-even takes the upper code. +_E2M1_MIDPOINTS = (0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0) +_TIE_UP = (False, True, False, True, False, True, False) + + +def nvfp4_quantize( + x: torch.Tensor, sf_scale: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """``fp4_quantize(x, sf_scale, 16, False, False)``: (packed e2m1 [M, K / 2], linear E4M3 scales [M * K / 16], + the pre-rounding scaled values [M, K]). + + Per 16 columns: scale = e4m3(sf_scale * amax / 6), data = x * sf_scale / scale rounded to nearest-even e2m1. + """ + gs = sf_scale.float().reshape(()) + m, k = x.shape + blocks = x.float().view(m, k // 16, 16) + vecmax = blocks.abs().amax(-1) + sf = (gs * (vecmax / _E2M1_MAX)).clamp(max=_E4M3_MAX).to(torch.float8_e4m3fn) + out_scale = torch.where(vecmax != 0, gs / sf.float(), torch.zeros_like(vecmax)) + scaled = (blocks * out_scale.unsqueeze(-1)).view(m, k) + mid = torch.tensor(_E2M1_MIDPOINTS, device=x.device, dtype=torch.float32) + tie_up = torch.tensor(_TIE_UP, device=x.device, dtype=torch.bool) + a = scaled.abs().unsqueeze(-1) + code = ((a > mid).sum(-1) + ((a == mid) & tie_up).any(-1)).to(torch.uint8) + negative = (scaled < 0) | ((scaled == 0) & torch.signbit(scaled)) + code = code | (negative.to(torch.uint8) << 3) + packed = code[:, 0::2] | (code[:, 1::2] << 4) + return packed, sf.view(torch.uint8).reshape(-1), scaled + + +def tf32_rna(w: torch.Tensor) -> torch.Tensor: + """fp32 -> TF32 as ``cvt.rna.tf32.f32``: keep 10 mantissa bits, round to nearest with ties away from zero.""" + bits = w.contiguous().view(torch.int32) + rounded = (bits + 0x1000) & ~0x1FFF + finite = (bits & 0x7F800000) != 0x7F800000 + return torch.where(finite, rounded, bits).view(torch.float32) + + +class M3RouterTc(OpWrapper): + """Router logits on TF32 tensor cores for 128 experts and 1..48 tokens: ``x @ tf32(w)^T``, fp32 accumulate. + + ``x`` is BF16 (exact in TF32), ``w`` the fp32 router weight, rounded to TF32 in the kernel. The 16 K slices' + partials are summed in slice order whatever cluster size the device co-schedules (16 or 8 CTAs), so the result + does not depend on it. With ``sf_scale`` also the NVFP4 activation (``fp4_quantize``'s bytes): returns + ``[logits]`` or ``[logits, x_fp4, x_sf]``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The target routes 2..48 tokens here: plain logits on the unfused path, logits + NVFP4 on the fused-router path. + CELLS: tuple[Cell, ...] = ( + Cell( + why="two tokens, logits only (the smallest call the target makes here)", + spec=dict(m=2, k=6144, fp4=False), + ), + Cell( + why="8 tokens with the NVFP4 activation: one 8-token MMA tile, every expert tile quantizes one row", + spec=dict(m=8, k=6144, fp4=True), + ), + Cell( + why="13 tokens: a partial second token tile", + spec=dict(m=13, k=6144, fp4=True), + ), + Cell( + why="the 48-token maximum, six token tiles, logits only", + spec=dict(m=48, k=6144, fp4=False), + ), + Cell( + why="the 48-token maximum with the NVFP4 activation", + spec=dict(m=48, k=6144, fp4=True), + ), + # Every other row count the router takes (modeling's ROUTER_TC), with and without the NVFP4 activation. + *( + Cell( + why=f"{m} rows: a router row count between the ones above", + spec=dict(m=m, k=6144, fp4=m % 2 == 1), + ) + for m in range(2, 49) + if m not in (2, 8, 13, 48) + ), + ) + + note = """ + The kernel uses 16-CTA clusters when the device co-schedules eight of them and 8-CTA clusters otherwise; both + sum the same 16 slice partials in the same order. Only the one the test device picks is driven. + """ + + def __call__( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_router_tc(x, w, sf_scale, pdl) + + def reference( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> tuple[torch.Tensor, ...]: + """fp32 product of ``x`` and the TF32-rounded weight, plus the NVFP4 recipe with ``sf_scale``.""" + with true_fp32_matmul(): + logits = x.float() @ tf32_rna(w).t() + if sf_scale is None: + return (logits,) + packed, sf, _ = nvfp4_quantize(x, sf_scale) + return logits, packed, sf + + def is_valid( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> None: + assert x.data_ptr() % 16 == 0, "x must start 16-byte aligned (cp.async.bulk source)" + if sf_scale is not None: + value = float(sf_scale.reshape(())) + assert value > 0.0 and value != float("inf"), ( + f"sf_scale must be positive and finite, got {value}" + ) + + +m3_router_tc = M3RouterTc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/norm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/norm/__init__.py new file mode 100644 index 000000000000..acec8fdbb26f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/norm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Normalization entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/norm/m3_embedding_firstnorm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/norm/m3_embedding_firstnorm.py new file mode 100644 index 000000000000..7ba04c92eb61 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/norm/m3_embedding_firstnorm.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-step embedding gather and the first layer's input RMSNorm in one CuTe DSL launch, into caller buffers.""" + +from __future__ import annotations + +from typing import Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.embedding_firstnorm.op import DecodeEmbedding, m3_embed_norm + +__all__ = ["DecodeEmbedding", "m3_embedding_firstnorm"] + +VOCAB = 200064 # MiniMax-M3's vocabulary; the decode table is the full (replicated) embedding + + +class M3EmbeddingFirstnorm(OpWrapper): + """Write ``table[ids]`` into ``raw`` (bf16 [M, 6144], a zero row for an id outside ``[0, V)``) and + ``raw * rsqrt(mean(raw^2) + eps) * weight`` into ``out``; returns ``(out, raw)`` -- the first layer's normed input + and its residual. ``ids`` int32 or int64 [M], M = 1..8; ``table`` bf16 [V, 6144] contiguous; ``weight`` bf16 + [6144]. The first call per (M, id dtype, table shape, PDL) compiles (``DecodeEmbedding`` does it at build).""" + + ARCHS = frozenset({Arch.SM_107}) + + # The target builds every (rows 1..8, id dtype) owner; the kernel is one CTA per row, so the rows differ only in + # grid size: the ends of the range and both id dtypes. + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, int32 ids", spec=dict(rows=1, id_dtype=torch.int32)), + Cell(why="eight requests, int32 ids", spec=dict(rows=8, id_dtype=torch.int32)), + Cell(why="one request, int64 ids", spec=dict(rows=1, id_dtype=torch.int64)), + Cell( + why="six rows (one request verifying a draft), int64 ids", + spec=dict(rows=6, id_dtype=torch.int64), + ), + # Every other row count the decode dispatch takes (modeling.EMBED_FIRSTNORM), both id dtypes. + *( + Cell( + why=f"{rows} rows: a decode row count between the ones above", + spec=dict(rows=rows, id_dtype=torch.int32 if rows % 2 else torch.int64), + ) + for rows in (2, 3, 4, 5, 7) + ), + ) + + note = """ + The gathered rows are exact. The norm is flashinfer's RMSNorm arithmetic in its reduction order with a fast + rsqrt, where `reference` sums in fp64 and takes torch's rsqrt: within the band the test derives, not bit for bit. + An id outside [0, V) gives a zero row and a zero norm (torch's embedding would raise). No accepted input is known + to compute wrongly: the host side rejects any other dtype, shape or a strided buffer. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + return m3_embed_norm(ids, table, weight, eps, raw, out, use_pdl) + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """New tensors (``raw`` / ``out`` are not written); `use_pdl` is ignored.""" + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] * valid.unsqueeze(1).to( + table.dtype + ) + x = rows.double() + rstd = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) + return (x * rstd * weight.double()).to(torch.bfloat16), rows + + +m3_embedding_firstnorm = M3EmbeddingFirstnorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/__init__.py new file mode 100644 index 000000000000..b3ba19cee80e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampling entries this target alone calls: the spec sampler's top-p renorm override (TOPP_DET), its softmax +override (SOFTMAX_MC) and its chain speculative sampling override (REJ_MC).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_chain_spec_mc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_chain_spec_mc.py new file mode 100644 index 000000000000..2f7352664f0b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_chain_spec_mc.py @@ -0,0 +1,381 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""REJ_MC: the spec sampler's chain speculative sampling (rejection) on many CTAs per request, exact vs +``torch.ops.trtllm.chain_speculative_sampling`` (``trtllm::minimax_m31_chain_speculative_sampling_mc``, +kernels/minimaxM31/chainSpecSamplingMcSm103.cu). + +The target installs ``chain_spec_mc`` as the rejection sampler's override +(``speculative/interface.set_chain_speculative_sampling_override``); it returns None, and the sampler keeps the stock +op, whenever the op is not built or the input is outside the kernel's contract. + +Workspace: one zero-initialized byte buffer per device, ``WORKSPACE_MAX_BATCH`` requests of +``minimax_m31_chain_speculative_sampling_mc_ws_bytes()``, kept for the life of the process (CUDA graphs captured with +it keep its address). The kernel leaves it ready for the next launch, so it is zeroed once. ``prebuild`` allocates it +outside any CUDA-graph capture (the target calls it from ``cache_derived_state``, before the executor's warmup and +captures); a call that finds no workspace allocates one when the stream is not capturing, and inside a capture returns +None (the stock op for that call) rather than allocating from the graph's private pool. Calls that share it must be +stream-ordered (the sampler's are). +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels import m3_report +from ...m31_switches import m31_disabled + +# The largest CTA count per request (the grid is batch x ctas_per_row 1024-thread CTAs, all co-resident; ctas_per_row +# is sized from the device's own SM count, so the same entry holds on GB300 and VR). +MAX_CTAS_PER_ROW = 16 + +# The requests the gathered override takes: up to the agg TP2 c46 / c48 calls. The VR microbenchmarks +# (batch 8 / 12 / 16 / 24 / 32 and batch 32 / 40 / 46 / 48 / 56 / 64; x random / peaked x 8 seeds, every legal +# ctas_per_row) measured the tokens and counts equal to the stock op's everywhere and the production ctas_per_row at +# least 6 us per call faster than the stock op through batch 48 (e.g. batch 16: 41.0 -> 27.0 us, batch 48: 51.9 -> +# 40.1). At batch 56 / 64 the rule gives 2 CTAs per request, which is slower than the stock op (56.2 -> 64.2 us), so +# those calls keep it. A GB300 microbenchmark gated batch 1 / 2 / 4 / 8 (31.2 vs 41.2 us at batch 1). +MAX_BATCH = 48 +# The cap with REJ_MC_WIDE disabled: the TP8 / TP4 K7 calls up to c8. +NARROW_MAX_BATCH = 8 +WIDE = not m31_disabled("REJ_MC_WIDE") +# Slot-indexed calls avoid the caller's draft-probability gather. The compound path +# passed exactness and timing at measured K3 cells up to 112 and K7 up to 56. +SLOTS_WIDE = not m31_disabled("REJ_SLOTS_WIDE") +WORKSPACE_MAX_BATCH = 112 + +# The widest vocab the kernel takes (64 tiles of 4096; kChainSpecMcMaxVocab). +MAX_VOCAB = 64 * 4096 + +# The op when it is built, decided once per process. +_STATE: dict = {} +# Per device index: the persistent workspace and the SM count. +_WORKSPACES: dict = {} +_SMS: dict = {} + + +def chain_spec_mc_op(): + """``trtllm::minimax_m31_chain_speculative_sampling_mc`` if this build carries it, else + None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_chain_speculative_sampling_mc", None) + _STATE["op"] = op + _STATE["ws_bytes"] = ( + int(torch.ops.trtllm.minimax_m31_chain_speculative_sampling_mc_ws_bytes()) + if op is not None + else 0 + ) + return _STATE["op"] + + +def max_batch() -> int: + """The requests the override takes: MAX_BATCH, or NARROW_MAX_BATCH with REJ_MC_WIDE disabled.""" + return MAX_BATCH if WIDE else NARROW_MAX_BATCH + + +def max_slot_batch(num_draft: int, vocab: int) -> int: + """Slot-only capacity; unmeasured K/vocab and the kill switch retain 48.""" + if SLOTS_WIDE and vocab == 200064: + if num_draft == 3: + return 112 + if num_draft == 7: + return 56 + return MAX_BATCH + + +def ws_bytes_per_row() -> int: + """Workspace bytes per request (0 when the op is not built).""" + chain_spec_mc_op() + return _STATE["ws_bytes"] + + +def sm_count(device: torch.device) -> int: + """The device's SM count, cached per device.""" + idx = torch.device(device).index + idx = torch.cuda.current_device() if idx is None else idx + if idx not in _SMS: + _SMS[idx] = torch.cuda.get_device_properties(idx).multi_processor_count + return _SMS[idx] + + +def ctas_per_row(batch: int, sms: int) -> int: + """The largest power of two <= MAX_CTAS_PER_ROW with ctas x batch <= sms (1 past that: one CTA per request needs + no co-residency).""" + g = MAX_CTAS_PER_ROW + while g > 1 and g * batch > sms: + g //= 2 + return g + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate the device's zeroed WORKSPACE_MAX_BATCH workspace outside capture. + Returns it, or None when the op is not built or the stream is capturing.""" + if chain_spec_mc_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.zeros( + WORKSPACE_MAX_BATCH * ws_bytes_per_row(), dtype=torch.uint8, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31ChainSpecMc(OpWrapper): + """``trtllm::chain_speculative_sampling(draft_probs, draft_token_ids, target_probs, seed, offset)`` bit for bit + (accepted tokens [batch, D + 1] and token counts [batch]) in ONE launch of batch x ctas_per_row CTAs (stock: one + CTA per request). ``workspace`` is the persistent zeroed buffer of ``prebuild``. ``__call__`` returns None (the + caller keeps the stock op) for a non-fp32, non-contiguous or misaligned probs, a vocab not divisible by 4 or above + MAX_VOCAB, mismatched shapes, non-int32 draft tokens, non-int64 or mis-sized seed / offset, inputs on different + devices, more requests than the workspace holds, a batch x ctas_per_row above the SM count, or when the op is not + built.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The K7 calls the microbenchmark gated bitwise on GB300 (5 cases x 64 seeds at ctas_per_row 4 / 8 / 16); the + # sm_107 build of the same source is gated by the same microbenchmark on VR before the target takes it. + CELLS: tuple[Cell, ...] = ( + Cell(why="K7 at c1 (vocab 200064, 16 CTAs)", spec=dict(batch=1, k=7, v=200064)), + Cell(why="K7 at c2 (vocab 200064)", spec=dict(batch=2, k=7, v=200064)), + Cell(why="K7 at c4 (vocab 200064)", spec=dict(batch=4, k=7, v=200064)), + Cell(why="K7 at c8 (vocab 200064, 128 CTAs)", spec=dict(batch=8, k=7, v=200064)), + Cell(why="K7 at c12 (vocab 200064, 192 CTAs)", spec=dict(batch=12, k=7, v=200064)), + Cell(why="K7 at c24 (vocab 200064, 8 CTAs each)", spec=dict(batch=24, k=7, v=200064)), + Cell(why="K7 at c32 (vocab 200064, 4 CTAs each)", spec=dict(batch=32, k=7, v=200064)), + Cell(why="K3 at agg c46 (vocab 200064, 4 CTAs each)", spec=dict(batch=46, k=3, v=200064)), + Cell( + why="K7 at the 48-request cap (vocab 200064, 4 CTAs each)", + spec=dict(batch=48, k=7, v=200064), + ), + ) + + note = """ + Exact by construction: every CTA of a request runs the stock Philox acceptance loop, the (target, residual) mass is + the stock 1024 per-thread chains into the same cub BlockReduce, each tile sum is the stock per-tile + cub reduce, and one CTA runs the stock tile-order prefix and in-tile cub BlockScan search. Default flags, as the + stock kernel. + """ + + def __call__( + self, + draft_probs: torch.Tensor, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, + workspace: Optional[torch.Tensor], + ctas_per_row: int, + ) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + if ( + workspace is None + or not all( + isinstance(t, torch.Tensor) + for t in (draft_probs, draft_token_ids, target_probs, seed, offset) + ) + or not draft_probs.is_cuda + or draft_probs.dtype != torch.float32 + or target_probs.dtype != torch.float32 + or draft_token_ids.dtype != torch.int32 + or seed.dtype != torch.int64 + or offset.dtype != torch.int64 + or draft_probs.dim() != 3 + or target_probs.dim() != 3 + or draft_token_ids.dim() != 2 + ): + return None + batch, k, v = draft_probs.shape + if ( + batch < 1 + or tuple(draft_token_ids.shape) != (batch, k) + or tuple(target_probs.shape) != (batch, k + 1, v) + or seed.numel() != batch + or offset.numel() != batch + or v % 4 != 0 + or v > MAX_VOCAB + or any( + not t.is_contiguous() or t.device != draft_probs.device + for t in (draft_token_ids, target_probs, seed, offset, workspace) + ) + or not draft_probs.is_contiguous() + or draft_probs.data_ptr() % 16 != 0 + or target_probs.data_ptr() % 16 != 0 + or ctas_per_row < 1 + or 1024 % ctas_per_row != 0 + or (ctas_per_row > 1 and batch * ctas_per_row > sm_count(draft_probs.device)) + ): + return None + op = chain_spec_mc_op() + if op is None or workspace.numel() * workspace.element_size() < batch * ws_bytes_per_row(): + return None + tokens, num_tokens = op( + draft_probs, draft_token_ids, target_probs, seed, offset, workspace, int(ctas_per_row) + ) + return tokens, num_tokens + + def reference( + self, + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace=None, + ctas_per_row=None, + ) -> tuple[torch.Tensor, torch.Tensor]: + """The stock op (workspace and ctas_per_row are ignored).""" + return torch.ops.trtllm.chain_speculative_sampling( + draft_probs, draft_token_ids, target_probs, seed, offset + ) + + def is_valid( + self, + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace=None, + ctas_per_row=None, + **kwargs, + ) -> None: + assert draft_probs.dtype == torch.float32 and draft_probs.dim() == 3, ( + "draft_probs: fp32 [batch, D, vocab]" + ) + + +m31_chain_spec_mc = M31ChainSpecMc() + + +def chain_spec_mc( + draft_probs: torch.Tensor, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, +) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + """The rejection sampler's override: the stock op's (accepted tokens, token counts), or None (the sampler runs the + stock op) when the op is not built, the batch exceeds max_batch(), no workspace can be had (first use inside a + capture) or the input is outside the contract.""" + if ( + not isinstance(draft_probs, torch.Tensor) + or not draft_probs.is_cuda + or draft_probs.dim() != 3 + ): + return None + batch = draft_probs.shape[0] + if batch < 1 or batch > max_batch() or chain_spec_mc_op() is None: + return None + workspace = prebuild(draft_probs.device) + if workspace is None: + return None + out = m31_chain_spec_mc( + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace, + ctas_per_row(batch, sm_count(draft_probs.device)), + ) + if out is not None: + m3_report.work("REJ_MC", f"batch {batch}") + return out + + +def chain_spec_mc_slots_op(): + """``trtllm::minimax_m31_chain_speculative_sampling_mc_slots`` (REJ_SLOTS) if this build carries it, else None.""" + if chain_spec_mc_op() is None: + return None + return getattr(torch.ops.trtllm, "minimax_m31_chain_speculative_sampling_mc_slots", None) + + +def chain_spec_mc_slots( + draft_buf: torch.Tensor, + slot_ids: torch.Tensor, + num_draft: int, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, +) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + """The rejection sampler's slot-indexed override (REJ_SLOTS): ``chain_spec_mc`` of the gathered + ``draft_buf[slot_ids, :num_draft, :vocab]`` (vocab = target_probs' last dim), reading those rows in place instead of + from the copy -- the same floats, so bitwise the gathered call. None (the sampler gathers and runs the plain path) + when the op is not built or the input is outside the contract.""" + op = chain_spec_mc_slots_op() + if ( + op is None + or not all( + isinstance(t, torch.Tensor) + for t in (draft_buf, slot_ids, draft_token_ids, target_probs, seed, offset) + ) + or not draft_buf.is_cuda + or draft_buf.dtype != torch.float32 + or draft_buf.dim() != 3 + or not draft_buf.is_contiguous() + or draft_buf.data_ptr() % 16 != 0 + or draft_buf.shape[2] % 4 != 0 + or slot_ids.dtype != torch.int64 + or slot_ids.dim() != 1 + or not slot_ids.is_contiguous() + or target_probs.dtype != torch.float32 + or target_probs.dim() != 3 + or draft_token_ids.dtype != torch.int32 + or seed.dtype != torch.int64 + or offset.dtype != torch.int64 + ): + return None + batch = slot_ids.shape[0] + v = target_probs.shape[2] + if ( + batch < 1 + or batch > max_slot_batch(num_draft, v) + or num_draft < 0 + or num_draft > draft_buf.shape[1] + or v > draft_buf.shape[2] + or v % 4 != 0 + or v > MAX_VOCAB + or tuple(draft_token_ids.shape) != (batch, num_draft) + or tuple(target_probs.shape) != (batch, num_draft + 1, v) + or seed.numel() != batch + or offset.numel() != batch + or target_probs.data_ptr() % 16 != 0 + or any( + not t.is_contiguous() or t.device != draft_buf.device + for t in (slot_ids, draft_token_ids, target_probs, seed, offset) + ) + ): + return None + workspace = prebuild(draft_buf.device) + if ( + workspace is None + or workspace.numel() * workspace.element_size() < batch * ws_bytes_per_row() + ): + return None + g = ctas_per_row(batch, sm_count(draft_buf.device)) + if g > 1 and batch * g > sm_count(draft_buf.device): + return None + out = op( + draft_buf, + slot_ids, + int(num_draft), + draft_token_ids, + target_probs, + seed, + offset, + workspace, + int(g), + ) + m3_report.work("REJ_SLOTS", f"batch {batch}") + return out diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_softmax_mc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_softmax_mc.py new file mode 100644 index 000000000000..c1d7a0bb3ac4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_softmax_mc.py @@ -0,0 +1,240 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SOFTMAX_MC: the spec sampler's softmax with its write pass on many CTAs per row, exact vs +``flashinfer.sampling.softmax(logits, temperature)`` (``trtllm::minimax_m31_softmax_mc``, +kernels/minimaxM31/softmaxMcSm103.cu). + +Rows 129..448 at vocab 200064 use the separate ``m31_softmax_pathb`` operation when enabled. +Its child kill is ``TRTLLM_M31_DISABLE_SOFTMAX_PATHB``; SOFTMAX_MC_WIDE also suppresses it. + +The target installs ``softmax_mc`` as the sampler's softmax override (``sampler/ops/flashinfer.set_softmax_override``); +it returns None, and the sampler keeps the stock FlashInfer call, whenever the op is not built or the input is outside +the kernel's contract (FlashInfer's Path A shapes: rows <= 128 and vocab >= 24576; here also rows <= ``max_rows()``). +The write CTAs per row follow the row count (``write_ctas``): 16 up to 12 rows (every c1 call), 8 up to 24, 4 up to +128. ``TRTLLM_M31_DISABLE_SOFTMAX_MC_128`` restores the 112-row cap; +``TRTLLM_M31_DISABLE_SOFTMAX_MC_WIDE`` keeps 16 CTAs and the 8-row cap. + +Workspace: one partials buffer per device (``MAX_ROWS`` x ceil(``MAX_VOCAB`` / 8192) float2, 128 KiB), kept for the life +of the process (CUDA graphs captured with it keep its address). Every launch overwrites what it reads, so it needs no +initialization. ``prebuild`` allocates it outside any CUDA-graph capture (the target calls it from +``cache_derived_state``, before the executor's warmup and captures); a call that finds no workspace allocates one when +the stream is not capturing, and inside a capture returns None (stock FlashInfer for that call) rather than allocating +from the graph's private pool. Calls that share it must be stream-ordered (the sampler's are). +""" + +from __future__ import annotations + +from typing import Optional, Union + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.logger import logger + +from ...kernels import m3_report +from ...m31_switches import m31_disabled +from . import m31_softmax_pathb as _pathb + +# The write pass's CTAs per row (GB300 microbenchmark: 9.2 vs FlashInfer 21.8 us per call at 7-8 rows; +# bitwise at 4 / 8 / 16). The grid (rows x WRITE_CTAS) needs no co-residency, so the SM count does not bound it. +WRITE_CTAS = 16 + +# Cover FlashInfer's complete Path-A row range, including K7 c16 verify calls. The same four-CTA write path +# serves every row above 24; the item kill restores the 112-row workspace and dispatch cap. +MAX_ROWS = 112 if m31_disabled("SOFTMAX_MC_128") else 128 +# The cap with SOFTMAX_MC_WIDE disabled: the TP8 / TP4 K7 c1 calls (the draft's 7 rows and the verify's 8). +NARROW_MAX_ROWS = 8 +# (largest row count, write CTAs per row): fewer, longer write CTAs once rows x WRITE_CTAS outgrows the SMs. +WRITE_CTAS_BY_ROWS = ((12, WRITE_CTAS), (24, 8), (MAX_ROWS, 4)) +WIDE = not m31_disabled("SOFTMAX_MC_WIDE") + +# FlashInfer's Path A bounds (OnlineSoftmax: batch <= 128 and d >= 24576); outside them FlashInfer takes its +# single-block Path B, which this kernel does not reproduce. +PATH_A_MAX_ROWS = 128 +PATH_A_MIN_VOCAB = 24576 +# FlashInfer's Path A slice (DEFAULT_SLICE_SIZE) and the widest vocab the workspace holds. +SLICE = 8192 +MAX_VOCAB = 1 << 20 + +# The op when it is built, decided once per process. +_STATE: dict = {} +# Per device index: the persistent partials workspace. +_WORKSPACES: dict = {} + + +def softmax_mc_op(): + """``trtllm::minimax_m31_softmax_mc`` if this build carries it, else None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_softmax_mc", None) + _STATE["op"] = op + if op is not None: + logger.info( + "[M3.1] SOFTMAX_MC rows: " + + ( + f"<= {MAX_ROWS}, write CTAs 16 / 8 / 4 up to 12 / 24 / {MAX_ROWS} rows" + if WIDE + else f"<= {NARROW_MAX_ROWS}, write CTAs {WRITE_CTAS} (SOFTMAX_MC_WIDE off)" + ) + ) + return _STATE["op"] + + +def max_rows() -> int: + """The rows the override takes: MAX_ROWS, or NARROW_MAX_ROWS with SOFTMAX_MC_WIDE disabled.""" + return MAX_ROWS if WIDE else NARROW_MAX_ROWS + + +def write_ctas(rows: int) -> int: + """The write CTAs per row for a call of ``rows`` rows (WRITE_CTAS with SOFTMAX_MC_WIDE disabled).""" + if WIDE: + for top, ctas in WRITE_CTAS_BY_ROWS: + if rows <= top: + return ctas + return WRITE_CTAS + + +def partials_numel(rows: int, d: int) -> int: + """fp32 elements of the partials workspace for rows x d (one float2 per row and 8192-wide slice).""" + return 2 * rows * ((d + SLICE - 1) // SLICE) + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's partials workspace for MAX_ROWS rows of up to MAX_VOCAB; must run outside a + CUDA-graph capture. Returns it, or None when the op is not built or the stream is capturing.""" + if _pathb.ENABLED: + _pathb.softmax_pathb_op() + if softmax_mc_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.empty( + partials_numel(MAX_ROWS, MAX_VOCAB), dtype=torch.float32, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31SoftmaxMc(OpWrapper): + """FlashInfer's ``softmax(logits, temperature)`` of fp32 logits [rows, d], bit for bit, on FlashInfer's Path A + shapes (rows <= 128, d >= 24576): FlashInfer's Map kernel, then its Reduce kernel's prologue in each of + ``write_ctas`` CTAs per row, each writing its share of the row (FlashInfer: one CTA writes the whole row). + ``temperature`` is None (1.0), a Python number (one temperature) or an fp32 [rows] tensor (one per row), as + FlashInfer takes it. ``partials`` is the persistent workspace of ``prebuild``. ``__call__`` returns None (the caller + keeps the stock call) for a non-fp32, non-contiguous or misaligned input, a width not divisible by 4, a shape + outside Path A or the workspace, a temperature tensor that is not a contiguous fp32 [rows] tensor on logits' + device, or when the op is not built.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The c1 K7 calls the microbenchmark gated bitwise on GB300; the sm_107 build of the same source is gated by the + # same microbenchmark on VR before the target takes it. + CELLS: tuple[Cell, ...] = ( + Cell(why="the K7 draft's 7 rows at c1 (vocab 200064)", spec=dict(rows=7, d=200064)), + Cell(why="the K7 verify's 8 rows at c1 (vocab 200064)", spec=dict(rows=8, d=200064)), + Cell(why="one row (the smallest grid)", spec=dict(rows=1, d=200064)), + Cell(why="TP2 WIDE K7 c2: 16 rows, 8 write CTAs", spec=dict(rows=16, d=200064)), + Cell(why="TP2 WIDE K7 c5: 40 rows, 4 write CTAs", spec=dict(rows=40, d=200064)), + Cell(why="TP2 WIDE K7 c12: 96 rows, 4 write CTAs", spec=dict(rows=96, d=200064)), + Cell(why="K7 c14, 4 write CTAs", spec=dict(rows=112, d=200064)), + Cell(why="the 128-row cap (K7 c16), 4 write CTAs", spec=dict(rows=128, d=200064)), + ) + + note = """ + Exact by construction: the Map kernel is FlashInfer's, every write CTA reduces the same partials with the same cub + BlockReduce and Float2SoftmaxReduceOp as FlashInfer's single Reduce CTA, and the per-element write + keeps FlashInfer's source form. Built with --use_fast_math, as FlashInfer's JIT builds it. + """ + + def __call__( + self, + logits: torch.Tensor, + temperature: Union[torch.Tensor, float, None], + partials: Optional[torch.Tensor], + write_ctas: int, + ) -> Optional[torch.Tensor]: + if ( + partials is None + or not isinstance(logits, torch.Tensor) + or not logits.is_cuda + or logits.dtype != torch.float32 + or logits.dim() != 2 + or not logits.is_contiguous() + or logits.shape[0] < 1 + or logits.shape[0] > PATH_A_MAX_ROWS + or logits.shape[1] < PATH_A_MIN_VOCAB + or logits.shape[1] % 4 != 0 + or logits.data_ptr() % 16 != 0 + or partials.device != logits.device + or partials.numel() * partials.element_size() + < 4 * partials_numel(logits.shape[0], logits.shape[1]) + or write_ctas < 1 + or write_ctas > 65535 + ): + return None + if temperature is None: + temp_arr, temp_val = None, 1.0 + elif isinstance(temperature, torch.Tensor): + if ( + temperature.dtype != torch.float32 + or temperature.dim() != 1 + or temperature.shape[0] != logits.shape[0] + or not temperature.is_contiguous() + or temperature.device != logits.device + ): + return None + temp_arr, temp_val = temperature, 0.0 + elif isinstance(temperature, (int, float)) and not isinstance(temperature, bool): + temp_arr, temp_val = None, float(temperature) + else: + return None + op = softmax_mc_op() + if op is None: + return None + out = torch.empty_like(logits) + op(logits, temp_arr, temp_val, partials, out, int(write_ctas)) + return out + + def reference(self, logits, temperature, partials=None, write_ctas=None) -> torch.Tensor: + """FlashInfer's softmax (partials and write_ctas are ignored).""" + import flashinfer.sampling + + return flashinfer.sampling.softmax(logits, temperature) + + def is_valid(self, logits, temperature, partials=None, write_ctas=None, **kwargs) -> None: + assert logits.dtype == torch.float32 and logits.dim() == 2, "logits: fp32 [rows, d]" + + +m31_softmax_mc = M31SoftmaxMc() + + +def softmax_mc( + logits: torch.Tensor, temperature: Union[torch.Tensor, float, None] +) -> Optional[torch.Tensor]: + """The sampler's softmax override: FlashInfer's softmax of logits [rows, d] at temperature, or None (the sampler + runs stock FlashInfer) when the op is not built, rows exceed max_rows(), d exceeds MAX_VOCAB, no workspace can be + had (first use inside a capture) or the input is outside the contract.""" + if not isinstance(logits, torch.Tensor) or not logits.is_cuda or logits.dim() != 2: + return None + rows = logits.shape[0] + if _pathb.MIN_ROWS <= rows <= _pathb.MAX_ROWS and WIDE and _pathb.ENABLED: + out = _pathb.m31_softmax_pathb(logits, temperature) + if out is not None: + m3_report.work("SOFTMAX_PATHB", f"{rows} rows") + return out + if rows < 1 or rows > max_rows() or logits.shape[1] > MAX_VOCAB or softmax_mc_op() is None: + return None + partials = prebuild(logits.device) + if partials is None: + return None + out = m31_softmax_mc(logits, temperature, partials, write_ctas(rows)) + if out is not None: + m3_report.work("SOFTMAX_MC", f"{rows} rows") + return out diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_softmax_pathb.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_softmax_pathb.py new file mode 100644 index 000000000000..b9fd798aa228 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_softmax_pathb.py @@ -0,0 +1,102 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SM107 fixed-vocabulary softmax for FlashInfer Path B, rows 129 through 448. + +One CTA per row preserves the ordered denominator recurrence and final CUB sum. +Warp integer maxima replace the per-chunk CUB max. The library is loaded outside +capture; the operation needs only its output, with no persistent workspace. +""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import m31_fake_ops +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled + +MIN_ROWS = 129 +MAX_ROWS = 448 +VOCAB = 200064 +ENABLED = not any(m31_disabled(item) for item in ("SOFTMAX_MC", "SOFTMAX_MC_WIDE", "SOFTMAX_PATHB")) +_STATE: dict = {} + + +def softmax_pathb_op(): + """Resolve the separately built native library, never loading during capture.""" + if "op" not in _STATE: + if torch.cuda.is_current_stream_capturing(): + return None + op = getattr(torch.ops.trtllm, "minimax_m31_softmax_pathb", None) + if op is not None: + m31_fake_ops.register() + logger.info( + "[M3.1] SOFTMAX_PATHB available: rows 129..448, vocab 200064, one CTA per row" + ) + _STATE["op"] = op + return _STATE["op"] + + +class M31SoftmaxPathB(OpWrapper): + """Contiguous aligned fp32 [129..448, 200064] logits on SM107. + + Temperature is None, a scalar number, or contiguous fp32 [rows] on the + same device. Unsupported inputs decline to the caller's stock operation. + """ + + ARCHS = frozenset({Arch.SM_107}) + CELLS = tuple( + Cell(why=f"Path B row coverage: {rows}", spec=dict(rows=rows, d=VOCAB)) + for rows in range(MIN_ROWS, MAX_ROWS + 1) + ) + note = "Preserves FlashInfer's ordered denominator and output arithmetic; no PDL or workspace." + + def __call__( + self, logits: torch.Tensor, temperature: torch.Tensor | float | None + ) -> torch.Tensor | None: + if ( + not isinstance(logits, torch.Tensor) + or not logits.is_cuda + or logits.dtype != torch.float32 + or logits.dim() != 2 + or not logits.is_contiguous() + or not MIN_ROWS <= logits.shape[0] <= MAX_ROWS + or logits.shape[1] != VOCAB + or logits.data_ptr() % 16 != 0 + ): + return None + if temperature is None: + temp_arr, temp_val = None, 1.0 + elif isinstance(temperature, torch.Tensor): + if ( + temperature.dtype != torch.float32 + or temperature.dim() != 1 + or temperature.shape[0] != logits.shape[0] + or not temperature.is_contiguous() + or temperature.device != logits.device + ): + return None + temp_arr, temp_val = temperature, 0.0 + elif isinstance(temperature, (int, float)) and not isinstance(temperature, bool): + temp_arr, temp_val = None, float(temperature) + else: + return None + op = softmax_pathb_op() + if op is None: + return None + out = torch.empty_like(logits) + op(logits, temp_arr, temp_val, out) + return out + + def reference( + self, logits: torch.Tensor, temperature: torch.Tensor | float | None + ) -> torch.Tensor: + """The installed FlashInfer softmax with the same temperature.""" + import flashinfer.sampling + + return flashinfer.sampling.softmax(logits, temperature) + + +m31_softmax_pathb = M31SoftmaxPathB() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_topp_det.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_topp_det.py new file mode 100644 index 000000000000..038aa8c31b15 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/sampling/m31_topp_det.py @@ -0,0 +1,303 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TOPP_DET: the spec sampler's top-p renorm as one persistent C++ launch, exact vs FlashInfer's deterministic +``top_p_renorm_probs`` (``trtllm::minimax_m31_topp_renorm_det``, kernels/minimaxM31/toppRenormDetSm103.cu). + +TOPP_DET v3 (``trtllm::minimax_m31_topp_renorm_det_v3``, kernels/minimaxM31/toppRenormDetV3Sm103.cu) is the same +renorm, bitwise v1.2's output, with the redundant per-CTA selection and a programmatic dependent launch. It is taken +when it is built unless +``TRTLLM_M31_DISABLE_TOPPDET_V3`` is set; v1.2 otherwise. ``TRTLLM_M31_DISABLE_TOPPDET_T3`` keeps v3 on its global +radix passes (no candidate list). + +V4K8 (a library whose op takes ``list_path`` / ``wide_hist``, default on): the one-sync list path (pass 0 lists the +elements >= 2^-12; when the selected bucket lies at or above that edge every CTA finishes the selection, FlashInfer's +chain sums and the write from the list after one barrier) and 8 privatized pass histograms, both bitwise the same +output; ``TRTLLM_M31_DISABLE_TOPPDET_V4`` / ``TRTLLM_M31_DISABLE_TOPPDET_K8`` turn them off. + +The target installs ``top_p_renorm_det`` as the sampler's top-p renorm override +(``sampler/ops/flashinfer.set_top_p_renorm_override``); it returns None, and the sampler keeps the stock FlashInfer +call, whenever the op is not built or the input is outside the kernel's contract. + +Workspace: one zero-initialized byte buffer per device, ``max_rows()`` rows of +``minimax_m31_topp_renorm_det_ws_bytes()``, kept for the life of the process (CUDA graphs captured with it keep its +address). The kernel leaves it ready for the next launch, so it is zeroed once. ``prebuild`` allocates it outside any +CUDA-graph capture (the target calls it from ``cache_derived_state``, before the executor's warmup and captures); a +call that finds no workspace allocates one when the stream is not capturing, and inside a capture returns None +(stock FlashInfer for that call) rather than allocating from the graph's private pool. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.logger import logger + +from ...kernels import m3_report +from ...m31_switches import m31_disabled + +# The largest CTA count per row (the persistent grid is rows x ctas_per_row 1024-thread CTAs, all co-resident). +MAX_CTAS_PER_ROW = 16 + +# The rows the workspace holds and the override takes with v3 / V4K8: up to the agg TP2 K3 c64 calls (256 rows) and +# the K7 c56 calls (448 rows). The VR microbenchmarks (rows 9..192 and 192 / 256 / 320 / 368 / 448; +# x peaked / randn / near_uniform) measured v3 and V4K8 bitwise FlashInfer's det mode on every cell and V4K8 faster +# on every cell (by >= 67 us per call; e.g. 64 rows 208.5 -> 44.5 us peaked, 256 rows 629.1 -> 343.8 randn, 448 rows +# 1047.6 -> 528.5 randn). Past 106 rows each row runs on one CTA (no co-residency). The workspace is ~0.2 MB per row +# (~90 MB per device at 448 rows). +MAX_ROWS = 448 +# v1.2 keeps the TP8 / TP4 K7 c1 calls (the draft's 7 rows and the verify's 8), where GB300 microbenchmarks measured it +# faster than stock FlashInfer (54.3 vs 104.2 us per call at 7 peaked rows, 54.8 vs 105.1 at 8). At +# 56 / 64 rows v1 was slower than FlashInfer's default mode (340 vs 187 / 199 us) and v1.2 is not +# re-measured there, so larger calls keep the stock kernel when v3 is not engaged. +V12_MAX_ROWS = 8 + +# The op when it is built, decided once per process. +_STATE: dict = {} +# Per device index: the persistent workspace and the SM count. +_WORKSPACES: dict = {} +_SMS: dict = {} +# The row counts already logged as taken / stock (one engagement line per distinct row count and outcome). +_LOGGED: set = set() + + +def _load_op(name: str): + """``torch.ops.trtllm.`` when this build carries it, else None.""" + return getattr(torch.ops.trtllm, name, None) + + +def topp_det_op(): + """The renorm op this process uses, decided once: ``trtllm::minimax_m31_topp_renorm_det_v3`` when it is built + and TOPPDET_V3 is not disabled, else ``trtllm::minimax_m31_topp_renorm_det`` when built, else None.""" + if "op" not in _STATE: + op, ws, v3 = None, 0, False + if not m31_disabled("TOPPDET_V3"): + op = _load_op("minimax_m31_topp_renorm_det_v3") + if op is not None: + v3 = True + ws = int(torch.ops.trtllm.minimax_m31_topp_renorm_det_v3_ws_bytes()) + logger.info("[M3.1] TOPP_DET v3 private op engaged (in-tree)") + if op is None: + op = _load_op("minimax_m31_topp_renorm_det") + ws = ( + int(torch.ops.trtllm.minimax_m31_topp_renorm_det_ws_bytes()) + if op is not None + else 0 + ) + _STATE["op"], _STATE["ws_bytes"], _STATE["v3"] = op, ws, v3 + # v3's candidate-list radix passes (bitwise the same output) unless TOPPDET_T3 is disabled. + _STATE["cand_list"] = v3 and not m31_disabled("TOPPDET_T3") + # V4K8 (a library whose schema carries list_path / wide_hist): the one-sync list path and 8 privatized pass + # histograms, both bitwise the same output, unless TOPPDET_V4 / TOPPDET_K8 is disabled. + v4k8 = v3 and "list_path" in str(op.default._schema) + _STATE["v4k8"] = v4k8 + _STATE["list_path"] = v4k8 and not m31_disabled("TOPPDET_V4") + _STATE["wide_hist"] = v4k8 and not m31_disabled("TOPPDET_K8") + if v3: + logger.info( + "[M3.1] TOPP_DET v3 T3 candidate-list fused passes: " + f"{'on' if _STATE['cand_list'] else 'off'} (cap 4096)" + ) + logger.info( + f"[M3.1] TOPP_DET V4K8: list path {'on' if _STATE['list_path'] else 'off'}, " + f"8 pass histograms {'on' if _STATE['wide_hist'] else 'off'}" + f"{'' if v4k8 else ' (library without V4K8)'}" + ) + if op is not None: + logger.info(f"[M3.1] TOPP_DET row cap: {MAX_ROWS if v4k8 else V12_MAX_ROWS} rows") + return _STATE["op"] + + +def is_v3() -> bool: + """Whether the engaged op is TOPP_DET v3.""" + topp_det_op() + return _STATE["v3"] + + +def max_rows() -> int: + """The rows the workspace holds and the override takes: MAX_ROWS with v3 + V4K8, V12_MAX_ROWS otherwise.""" + # TP2: the wide cap needs the V4K8 library; V4K8 (not v3/T3 alone) measured faster than stock at + # 9..448 rows (VR microbenchmarks), so a v3/T3-only build keeps the 8-row cap. + topp_det_op() + return MAX_ROWS if _STATE.get("v4k8") else V12_MAX_ROWS + + +def ws_bytes_per_row() -> int: + """Workspace bytes per row (0 when the op is not built).""" + topp_det_op() + return _STATE["ws_bytes"] + + +def sm_count(device: torch.device) -> int: + """The device's SM count, cached per device.""" + idx = torch.device(device).index + idx = torch.cuda.current_device() if idx is None else idx + if idx not in _SMS: + _SMS[idx] = torch.cuda.get_device_properties(idx).multi_processor_count + return _SMS[idx] + + +def ctas_per_row(rows: int, sms: int) -> int: + """The largest power of two <= MAX_CTAS_PER_ROW with ctas x rows <= sms (1 past that: one CTA per row needs no + co-residency).""" + g = MAX_CTAS_PER_ROW + while g > 1 and g * rows > sms: + g //= 2 + return g + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's zeroed workspace for max_rows() rows; must run outside a CUDA-graph capture. + Returns it, or None when the op is not built or the stream is capturing.""" + if topp_det_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.zeros( + max_rows() * ws_bytes_per_row(), dtype=torch.uint8, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31ToppDet(OpWrapper): + """FlashInfer's ``top_p_renorm_probs(probs, top_p, is_deterministic=True)`` of fp32 probs [rows, d] with fp32 + per-row top_p [rows], bit for bit on every row with top_p < 1, in ONE launch of rows x ctas_per_row CTAs + (FlashInfer's: five launches, apply on one CTA per row, slow on high-entropy rows). Rows with top_p >= 1 keep every + element (FlashInfer drops tail buckets there in ~43% of rows, in both of its modes); their output is + ``ftz(ftz(p) * norm)``. ``workspace`` is the persistent zeroed buffer of ``prebuild``. ``__call__`` returns None + (the caller keeps the stock call) for a non-fp32, non-contiguous or misaligned input, a width not divisible by 4, + a top_p that is not a contiguous fp32 [rows] tensor on probs' device, more rows than the workspace holds, or when + the op is not built.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The c1 K7 calls the microbenchmark gated bitwise on GB300 (8 row classes x 4 top_p modes x 16 seeds); the + # sm_107 build of the same source is gated by the same microbenchmark on VR before the target takes it. + CELLS: tuple[Cell, ...] = ( + Cell(why="the K7 draft's 7 rows at c1 (vocab 200064)", spec=dict(rows=7, d=200064)), + Cell(why="the K7 verify's 8 rows at c1 (vocab 200064)", spec=dict(rows=8, d=200064)), + Cell(why="one row (the smallest grid, 16 CTAs)", spec=dict(rows=1, d=200064)), + Cell(why="TP2 WIDE K7 c8: 64 rows, 2 CTAs per row", spec=dict(rows=64, d=200064)), + Cell(why="TP2 WIDE K7 c24: 192 rows, 1 CTA per row", spec=dict(rows=192, d=200064)), + Cell(why="agg TP2 K3 c64: 256 rows, 1 CTA per row", spec=dict(rows=256, d=200064)), + Cell(why="the 448-row cap (K7 c56), 1 CTA per row", spec=dict(rows=448, d=200064)), + ) + + note = """ + Exact by construction: the det histograms are integer (any aggregation order is FlashInfer's), the per-pass + selection is FlashInfer's last-block code on them, and the kept sum is FlashInfer's 1024 per-thread chains into + the same cub BlockReduce. Built with --use_fast_math, as FlashInfer's JIT builds it. + """ + + def __call__( + self, + probs: torch.Tensor, + top_p: torch.Tensor, + workspace: Optional[torch.Tensor], + ctas_per_row: int, + ) -> Optional[torch.Tensor]: + if ( + not isinstance(top_p, torch.Tensor) + or workspace is None + or not probs.is_cuda + or probs.dtype != torch.float32 + or probs.dim() != 2 + or not probs.is_contiguous() + or probs.shape[0] < 1 + or probs.shape[1] % 4 != 0 + or probs.data_ptr() % 16 != 0 + or top_p.dtype != torch.float32 + or top_p.dim() != 1 + or top_p.shape[0] != probs.shape[0] + or not top_p.is_contiguous() + or top_p.device != probs.device + or workspace.device != probs.device + or ctas_per_row < 1 + or 1024 % ctas_per_row != 0 + ): + return None + op = topp_det_op() + if ( + op is None + or workspace.numel() * workspace.element_size() < probs.shape[0] * ws_bytes_per_row() + ): + return None + out = torch.empty_like(probs) + if is_v3(): + if _STATE["v4k8"]: + op( + probs, + top_p, + workspace, + out, + int(ctas_per_row), + True, + _STATE["cand_list"], + _STATE["list_path"], + _STATE["wide_hist"], + ) + else: + op(probs, top_p, workspace, out, int(ctas_per_row), True, _STATE["cand_list"]) + else: + op(probs, top_p, workspace, out, int(ctas_per_row)) + return out + + def reference(self, probs, top_p, workspace=None, ctas_per_row=None) -> torch.Tensor: + """FlashInfer's deterministic renorm (rows with top_p < 1; workspace and ctas_per_row are ignored).""" + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + def is_valid(self, probs, top_p, workspace=None, ctas_per_row=None, **kwargs) -> None: + assert probs.dtype == torch.float32 and probs.dim() == 2, "probs: fp32 [rows, d]" + + +m31_topp_det = M31ToppDet() + + +def top_p_renorm_det(probs: torch.Tensor, top_p: torch.Tensor) -> Optional[torch.Tensor]: + """The sampler's top-p renorm override: the det renorm of probs [rows, d] at per-row top_p, or None (the sampler + runs stock FlashInfer) when the op is not built, rows exceed max_rows(), no workspace can be had (first use inside + a capture) or the input is outside the contract.""" + if not isinstance(top_p, torch.Tensor) or not probs.is_cuda or probs.dim() != 2: + return None + rows = probs.shape[0] + if rows < 1 or topp_det_op() is None: + return None + if rows > max_rows(): + _log_once(rows, False, f"above the {max_rows()}-row cap") + return None + workspace = prebuild(probs.device) + if workspace is None: + return None + g = ctas_per_row(rows, sm_count(probs.device)) + out = m31_topp_det(probs, top_p, workspace, g) + if out is None: + _log_once(rows, False, "outside the kernel's contract") + else: + _log_once(rows, True, f"ctas_per_row {g}") + m3_report.work("TOPP_DET", f"{rows} rows") + if _STATE.get("v3"): + m3_report.work("TOPPDET_V3", f"{rows} rows") + if _STATE.get("cand_list"): + m3_report.work("TOPPDET_T3", f"{rows} rows") + return out + + +def _log_once(rows: int, taken: bool, detail: str) -> None: + """One engagement line per distinct (row count, outcome): "TOPP_DET taken at N rows" when the op served the call, + "TOPP_DET stock at N rows" when the sampler keeps FlashInfer.""" + if (rows, taken) not in _LOGGED: + _LOGGED.add((rows, taken)) + logger.info(f"[M3.1] TOPP_DET {'taken' if taken else 'stock'} at {rows} rows ({detail})") diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/__init__.py new file mode 100644 index 000000000000..d25c41dc9557 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Speculative-decoding entries this target alone calls: the Eagle3 verify ops, the capture / fc_norm RMSNorms and +the step-input staging.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m31_markov_epilogue.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m31_markov_epilogue.py new file mode 100644 index 000000000000..94468f0b7754 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m31_markov_epilogue.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""S4FE: the in-tree DSpark Markov chain (the steps S4's one-launch chain declines: above MAX_FAST_BATCH requests, full +vocab, replicated on both ranks) with its elementwise tail fused per position. + +The in-tree chain (models/modeling_speculative.py dspark_markov_chain, cast_bias_to_logits, the unsharded argmax of +interface.py greedy_sample_draft_with_tp_gather) runs per position k: bias = F.linear(markov_prev_embeddings(prev, w1), +w2) (bf16), then bias.to(fp32), base[:, k] + bias, argmax, and finally torch.cat of the corrected rows. Here the same +embedding (position 0) and the same F.linear run; ``trtllm::minimax_m31_markov_epilogue`` does the rest in one pass: +the fp32 add (one add.rn, as torch's), the corrected row written in place of the cat, torch's argmax order (NaN wins, +ties take the lower index) and the next position's embedding w1[prev] (prev is in vocab, so the mask is a no-op). +Bitwise the in-tree corrected logits by construction. + +Workspace: per device, a partials buffer (rows x splits x 8 B) and rows int32 counters that every launch leaves zero, +allocated once outside capture (``prebuild``).""" + +from __future__ import annotations + +from typing import Optional + +import torch +import torch.nn.functional as F + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.models.modeling_speculative import ( + dspark_markov_chain_logits, + markov_prev_embeddings, +) + +# Request capacity covers 448 verify rows at K3 (112 requests). +MAX_ROWS = 112 +# Splits per row are at most d / 4 / 1024 (markovEpilogueSplits); the workspace holds the 200064-vocab maximum. +MAX_SPLITS = 1024 + +_STATE: dict = {} +_BK3_LOGGED: set = set() +_WORKSPACES: dict = {} + + +def markov_epilogue_op(): + """``trtllm::minimax_m31_markov_epilogue`` if this build carries it, else None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_markov_epilogue", None) + _STATE["op"] = op + return _STATE["op"] + + +def _prev_embeddings(first_prev: torch.Tensor, w1: torch.Tensor) -> torch.Tensor: + """Position 0's embedding: TAIL_BK3 (B) ``trtllm::minimax_m31_markov_prev_embed`` (one gather, bitwise the in-tree + masked embedding) when on and in layout, else the in-tree ``markov_prev_embeddings``.""" + from ...dspark_s4 import TAIL_BK3 + + op = getattr(torch.ops.trtllm, "minimax_m31_markov_prev_embed", None) if TAIL_BK3 else None + prev = first_prev.long() + if ( + op is not None + and prev.dim() == 1 + and w1.dim() == 2 + and w1.dtype == torch.bfloat16 + and w1.is_contiguous() + ): + rows = int(prev.shape[0]) + if rows not in _BK3_LOGGED: + _BK3_LOGGED.add(rows) + from tensorrt_llm.logger import logger + + logger.info(f"[M3.1] TAIL_BK3 Markov embedding taken at {rows} requests") + return op(prev.contiguous(), w1) + return markov_prev_embeddings(prev, w1) + + +def prebuild(device: torch.device) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + """Allocate (once, outside capture) the device's (partials, counters); None when the op is not built.""" + if markov_epilogue_op() is None: + return None + index = device.index if device.index is not None else torch.cuda.current_device() + if index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + dev = torch.device("cuda", index) + _WORKSPACES[index] = ( + torch.empty(MAX_ROWS * MAX_SPLITS * 8, dtype=torch.uint8, device=dev), + torch.zeros(MAX_ROWS, dtype=torch.int32, device=dev), + ) + return _WORKSPACES[index] + + +def _splits(rows: int, d: int) -> int: + key = ("splits", rows, d) + if key not in _STATE: + _STATE[key] = int(torch.ops.trtllm.minimax_m31_markov_epilogue_splits(rows, d)) + return _STATE[key] + + +class M31MarkovChainFused(OpWrapper): + """The in-tree Markov chain's corrected logits [B, K, d] (full vocab), with the elementwise tail fused; None when + the op is not built, the step is wider than MAX_ROWS or the operands are outside the kernel's layout (the caller + keeps the in-tree chain).""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = tuple( + Cell(why=f"{b} requests at K{k} (agg TP2 above S4's cap)", spec=dict(b=b, k=k, d=200064)) + for k, rows in ((3, range(13, 113)), (7, range(13, 57))) + for b in rows + ) + + note = """ + The F.linear and the position-0 embedding are the in-tree calls themselves, so cuBLAS picks the same tactic; the + kernel's add, argmax order and embedding copy are exact, so the corrected logits are the in-tree bits. + """ + + def __call__( + self, base: torch.Tensor, first_prev: torch.Tensor, w1: torch.Tensor, w2: torch.Tensor + ) -> Optional[torch.Tensor]: + op = markov_epilogue_op() + if ( + op is None + or base.dim() != 3 + or base.dtype != torch.float32 + or not base.is_contiguous() + or not 1 <= base.shape[0] <= MAX_ROWS + or base.shape[2] % 4 != 0 + or w1.dtype != torch.bfloat16 + or w2.dtype != torch.bfloat16 + or not w1.is_contiguous() + or w2.shape[0] != base.shape[2] + or w1.shape[0] < base.shape[2] + ): + return None + ws = prebuild(base.device) + if ws is None: + return None + partials, counters = ws + rows, block, d = base.shape + splits = _splits(rows, d) + out = torch.empty_like(base) + prev = torch.empty(rows, dtype=torch.int64, device=base.device) + emb = _prev_embeddings(first_prev, w1) + for k in range(block): + bias = F.linear(emb, w2) + nxt = torch.empty_like(emb) if k + 1 < block else None + op(base, k, bias, out, prev, w1, nxt, partials, counters, splits) + emb = nxt + return out + + def reference(self, base, first_prev, w1, w2) -> torch.Tensor: + """The in-tree chain with the unsharded argmax (interface.py greedy_sample_draft_with_tp_gather).""" + return dspark_markov_chain_logits( + base, first_prev, w1, w2, + argmax_fn=lambda x: torch.argmax(x, dim=-1).type(torch.int32).long(), + ) # fmt: skip + + def is_valid(self, base, first_prev, w1, w2) -> None: + assert base.dtype == torch.float32 and base.dim() == 3, "base: fp32 [B, K, d]" + + +m31_markov_chain_fused = M31MarkovChainFused() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_capture_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_capture_norm.py new file mode 100644 index 000000000000..be5aa46fe774 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_capture_norm.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Eagle3 hidden-state capture with the drafter's fc_norm folded in (``m3_spec_tail.capture_norm``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_chunk_norm import rmsnorm_rows + + +class M3CaptureNorm(OpWrapper): + """``dst [M, H] = rmsnorm(bf16(h + r)) * w`` (``r`` optional), the rounding chunk_norm applies, into a capture + buffer slice whose row stride may exceed H. Returns ``dst``.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The target's captured layers (hidden 6144) into the 3-layer capture buffer (row stride 3 x 6144), over the + # verify rows of draft_len 3 (4 .. 32 rows); the residual is present for every captured decoder layer. + CELLS: tuple[Cell, ...] = ( + Cell( + why="4 verify rows of one request, layer slot 0, residual added", + spec=dict(m=4, slot=0, residual=True), + ), + Cell( + why="cap 8 (32 rows) into the last layer slot", spec=dict(m=32, slot=2, residual=True) + ), + Cell( + why="no residual (a capture point without one): plain RMSNorm", + spec=dict(m=8, slot=1, residual=False), + ), + ) + + note = """ + fp32 statistic (block sum, rsqrtf), one bf16 rounding after the bf16 sum h + r; the band states the occasional + one-ulp move against torch's statistic. + """ + + def __call__( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_capture_norm(h, r, w, eps, dst) + return dst + + def reference( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + """bf16 sum, then the normalized row; ``dst`` only says where the op writes.""" + x = (h.float() + r.float()).to(torch.bfloat16) if r is not None else h + return rmsnorm_rows(x, w, eps) + + def is_valid( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> None: + # The norm weight is read as H dense bf16 values; a strided view (a column of a stacked weight) is read + # past its elements without an error. + assert w.is_contiguous(), "w must be dense [H]" + + +m3_capture_norm = M3CaptureNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_chunk_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_chunk_norm.py new file mode 100644 index 000000000000..6699a55fc3e6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_chunk_norm.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 fc_norm over the captured layers in one launch: per-chunk RMSNorm of the concatenated hidden states +(``m3_spec_tail.chunk_norm``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3ChunkNorm(OpWrapper): + """``dst[r, c*H:(c+1)*H] = rmsnorm(src[r, c*H:(c+1)*H]) * w[c]`` for the C chunks of ``w [C, H]``; ``src`` may + have a row stride wider than C * H (the capture buffer). Writes ``dst [M, C*H]`` and returns it.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The Eagle3 capture buffer of the target (3 captured layers x hidden 6144, buffer rows wider than the layers + # when the final hidden is also captured), over the verify rows of draft_len 3: 4 / 8 / 32 rows. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request's 4 verify rows, dense [4, 3 x 6144]", + spec=dict(m=4, c=3, h=6144, extra_cols=0), + ), + Cell( + why="cap 8 (32 rows) read from the capture buffer with a fourth (final-hidden) slot in the row stride", + spec=dict(m=32, c=3, h=6144, extra_cols=6144), + ), + ) + + note = """ + fp32 statistic per chunk (block sum, rsqrtf), one bf16 rounding: an output moves by one bf16 ulp now and then + against torch's statistic; the cells' band states it. + """ + + def __call__( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + torch.ops.trtllm.minimax_m3_chunk_norm(src, w, eps, dst, pdl) + return dst + + def reference( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + """Each chunk normalized on its own; ``dst`` only says where the op writes, ``pdl`` shapes the launch.""" + c, h = w.shape + x = src[:, : c * h].reshape(src.shape[0], c, h) + return rmsnorm_rows(x, w, eps).reshape(src.shape[0], c * h) + + def is_valid( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ) -> None: + c, h = w.shape + assert w.is_contiguous(), "w must be dense [C, H] (chunk c read at c * H)" + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + + +m3_chunk_norm = M3ChunkNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_input.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_input.py new file mode 100644 index 000000000000..8f0f69ee812b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_input.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 draft pass's input in one launch: embedding lookup, both RMSNorms and the concat +(``m3_spec_tail.draft_input``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3DraftInput(OpWrapper): + """``out [M, 2H] = [rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden]`` and ``residual [M, H] = + hidden[rows]`` (rows default to 0..M-1) in one launch; ids outside the table embed to zero (the masked lookup of + a vocab-local embedding). Returns ``(out, residual)``.""" + + ARCHS = frozenset({Arch.SM_107}) + + # Every pass of the fused draft loop (modeling.DRAFT_INPUT) at draft_len 3, drafter hidden 6144, eps 1e-6: the + # first pass of 1..8 requests (4..32 rows, the carried hidden read in place) and the later passes of 1..8 requests + # (one row each; the second pass gathers the last accepted rows from the first pass's 4 rows per request). + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"first pass of {n} request(s): {4 * n} rows, hidden read in place", + spec=dict(m=4 * n, h=6144, gather=False, invalid_id=False), + ) + for n in range(1, 9) + ), + *( + Cell( + why=f"second pass of {n} request(s): the last accepted rows gathered from {4 * n} rows", + spec=dict(m=n, h=6144, gather=True, invalid_id=False), + ) + for n in range(1, 9) + ), + Cell( + why="third pass of 3 requests: one row each, hidden read in place", + spec=dict(m=3, h=6144, gather=False, invalid_id=False), + ), + Cell( + why="one row with an out-of-table id: the embedding half is zero", + spec=dict(m=1, h=6144, gather=False, invalid_id=True), + ), + ) + + note = """ + The arithmetic is the in-tree RMSNorm's (fp32 statistic, (x * rsqrt) * w, one bf16 rounding); the statistic's + summation order differs from FlashInfer's, which moves an output by one bf16 ulp now and then (the test's band, + also held against the in-tree modules). The out-of-table id is the masked lookup of a vocab-local embedding; the + drafter's embedding is the whole table, so its ids are always in range. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + torch.ops.trtllm.minimax_m3_draft_input( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, pdl + ) + return out, residual + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + """The two normalized halves (fp32 statistic, one bf16 rounding) and the gathered hidden rows. ``out`` / + ``residual`` only say where the op writes; ``pdl`` only shapes the launch.""" + valid = (ids >= 0) & (ids < table.shape[0]) + emb = rmsnorm_rows(table[torch.where(valid, ids, 0).long()], w_embed, eps_embed) + emb = torch.where(valid[:, None], emb, torch.zeros_like(emb)) + hid = hidden[rows.long()] if rows is not None else hidden[: ids.shape[0]] + ref = torch.cat([emb, rmsnorm_rows(hid, w_hidden, eps_hidden)], dim=-1) + return ref, (hid.clone() if residual is not None else None) + + def is_valid( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> None: + h = table.shape[1] + # Rows are read as hidden + rows[r] * H and table + id * H: both must be dense H-wide rows. + assert table.is_contiguous() and hidden.is_contiguous(), ( + "table / hidden must be dense [*, H]" + ) + if rows is not None: + assert bool(((rows >= 0) & (rows < hidden.shape[0])).all()), "rows must index hidden" + else: + assert hidden.shape[0] >= ids.shape[0], "hidden must hold a row per id" + # Vector loads of 16 bytes: H must be a multiple of 8 within the register tile, and the weights aligned. + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + assert residual is None or ( + residual.is_contiguous() and residual.shape == (ids.shape[0], h) + ), "residual must be contiguous [M, H]" + + +m3_draft_input = M3DraftInput() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_silu.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_silu.py new file mode 100644 index 000000000000..75626e570244 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_silu.py @@ -0,0 +1,58 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter MLP's plain SwiGLU as one C++ launch (``modeling.DRAFT_SILU_CPP``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3DraftSilu(OpWrapper): + """``silu(g) * u`` of a bf16 [rows, 2d] gate | up tensor, rounded once to bf16: + trtllm::minimax_m3_sm107_tp4_silu_and_mul (minimaxM3Sm107Tp4/siluAndMul.cu), bit for bit the Triton + ``torch.ops.trtllm.silu_and_mul`` at its defaults (no limit, alpha 1, beta 0), the activation the drafter's GatedMLP + runs. Contiguous, 16-byte aligned rows and d a multiple of 8; ``__call__`` returns None otherwise (the caller runs + the Triton op).""" + + ARCHS = frozenset({Arch.SM_107}) + + # The TP4 drafter MLP shard: intermediate 18432 / 4 = 4608, gate | up 9216 wide, at the drafter's decode rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="the TP4 drafter MLP at 1 row", spec=dict(rows=1, d=4608)), + Cell(why="the TP4 drafter MLP at 4 rows (1 request x 4 tokens)", spec=dict(rows=4, d=4608)), + Cell(why="the TP4 drafter MLP at 8 rows", spec=dict(rows=8, d=4608)), + ) + + note = """ + Elementwise, one rounding: bit for bit the Triton kernel (the same PTX instruction sequence), checked over every + bf16 gate and every bf16 up pattern. + """ + + def __call__(self, x: torch.Tensor) -> Optional[torch.Tensor]: + if ( + x.dtype != torch.bfloat16 + or x.dim() != 2 + or not x.is_cuda + or not x.is_contiguous() + or x.shape[1] % 16 != 0 + or x.data_ptr() % 16 != 0 + or x.shape[0] > 65535 + ): + return None + out = torch.empty((x.shape[0], x.shape[1] // 2), dtype=x.dtype, device=x.device) + torch.ops.trtllm.minimax_m3_sm107_tp4_silu_and_mul(x, out) + return out + + def reference(self, x: torch.Tensor) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul(x) + + def is_valid(self, x: torch.Tensor) -> None: + assert x.dtype == torch.bfloat16 and x.dim() == 2, "x: bf16 [rows, 2d]" + + +m3_draft_silu = M3DraftSilu() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_step.py new file mode 100644 index 000000000000..5dbb324a7d50 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_draft_step.py @@ -0,0 +1,417 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One pass of the fused greedy Eagle3 draft loop: the drafter's vocab-sharded BF16 head (streamed from its lossless +12-bit coding, ``ops.gemm.m3_target_head``, or as BF16 fragments), the cross-rank argmax and the loop's per-pass +bookkeeping in one kernel (``m3_draft_step.draft_step``).""" + +from __future__ import annotations + +import functools +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +from ..gemm.m3_target_head import decode as decode_coded +from .m3_shard_accept import exchange_tokens, local_argmax, make_keys + +#: Requests (head rows) per pass, and exchange rows per call half of the workspace. +MAX_TOKENS = 8 +#: 16-row weight tiles one CTA can own (the kernel's shared-memory accumulator). +MAX_TILES_PER_CTA = 32 +KERNEL_WARPS = 16 # warps per CTA (draftStep.h kWarps) + + +@functools.lru_cache(maxsize=64) +def max_tile_pieces(n_tiles: int, num_ctas: int, chunks_per_tile: int) -> int: + """The most warps that stream one weight tile under the kernel's whole-tile split: CTA c owns tiles + [c * n / grid, (c + 1) * n / grid), and its (tile, chunk) range is cut into KERNEL_WARPS equal contiguous + segments. The shared-memory atomics add a tile's pieces in arrival order: deterministic for at most 2 pieces (two + addends from 0 commute), not for more (fixed_order then sums them in warp order).""" + worst = 1 + for c in range(num_ctas): + lo, hi = c * n_tiles // num_ctas, (c + 1) * n_tiles // num_ctas + span = (hi - lo) * chunks_per_tile + bounds = [w * span // KERNEL_WARPS for w in range(KERNEL_WARPS + 1)] + for t in range(hi - lo): + a, b = t * chunks_per_tile, (t + 1) * chunks_per_tile + worst = max( + worst, sum(1 for w in range(KERNEL_WARPS) if bounds[w] < b and bounds[w + 1] > a) + ) + return worst + + +#: Draft length of the fused loop, and the drafter's TP4 head shard (200064-id vocabulary) and hidden size. +DRAFT_LEN = 3 +DRAFT_SHARD = 50016 +HIDDEN = 6144 + + +def bf16_fragments(weight: torch.Tensor) -> torch.Tensor: + """BF16 [N, K] -> the same values in A-fragment order [N / 16, K / 16, 256]. + + One lane's 16 bytes of a 16-row tile's k16 step are the m16n8k16 A fragment: element (row h * 8 + g, k ks * 16 + + hk * 8 + jj * 2 + e) goes to [tile][ks][g][jj][hk][h][e] (lane = g * 4 + jj). + """ + n, k = weight.shape + assert n % 16 == 0 and k % 16 == 0 and weight.dtype == torch.bfloat16, (n, k, weight.dtype) + q = weight.view(n // 16, 2, 8, k // 16, 2, 4, 2).permute(0, 3, 2, 5, 4, 1, 6).contiguous() + return q.view(n // 16, k // 16, 256) + + +def from_fragments(wf: torch.Tensor) -> torch.Tensor: + """The inverse of ``bf16_fragments``: [N / 16, K / 16, 256] -> [N, K].""" + tiles, ks, _ = wf.shape + return ( + wf.view(tiles, ks, 8, 4, 2, 2, 2).permute(0, 5, 2, 1, 4, 3, 6).reshape(tiles * 16, ks * 16) + ) + + +def head_weight( + wf: torch.Tensor, + k: int, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The BF16 head shard [N, K] the kernel multiplies by: the fragment-ordered copy, or (``exc_off`` given) the + 12-bit coded stream with its exception lists decoded.""" + if exc_off is None: + return from_fragments(wf) + n = (exc_off.numel() - 1) * 16 + return decode_coded(wf, exp_base, exc_off, exc_pos, exc_w, n, k) + + +def shard_logits( + x: torch.Tensor, + wf: torch.Tensor, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The in-tree head's BF16 logits of the rows x [M, K] against the shard (``head_weight``): the fp32 product, + rounded once to BF16 (the BF16 GEMM's output), as fp32.""" + w = head_weight(wf, x.shape[1], exc_off, exc_pos, exc_w, exp_base) + with true_fp32_matmul(): + logits = x.float() @ w.float().t() + return logits.to(torch.bfloat16).float() + + +class M3DraftStep(OpWrapper): + """One fused greedy draft pass over ``rows <= 8`` requests (one head row each). + + Head input ``x[x_rows[b]]`` (pass 0: the last accepted verify row of each request) or ``x[b]``. The head is this + rank's BF16 lm_head shard in fragment order (``m3_draft_step.bf16_fragments``, global ids ``vocab_offset ..``), + or with ``exc_off`` / ``exc_pos`` / ``exc_w`` / ``exp_base`` its lossless 12-bit coding (``wf`` = the uint8 stream + of ``m3_target_head.encode_chunked``, which must round-trip to the BF16 shard); each logit is the fp32 dot product + rounded to BF16 and the row's token is the global argmax over all ranks' + shards (ties, -0 / +0 included, to the lowest global id), exchanged through the multicast slots + ``[2][8][world]``. The last CTA then writes ``draft_out[b, step]`` and ``ids_next[b]`` (the token), ``pos_next[b] + = pos_src[x_rows[b] or b] + 1``, and advances the draft KV lengths as the in-tree draft loop does between passes + (pass 0 with ``num_acc``: ``kv_lens -= draft_len - num_acc`` and ``seq_lens = 1``; later passes: ``kv_lens += + 1``). ``out_keys`` (optional) receives each row's winning key. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # Every pass of the fused draft loop (modeling.DRAFT_STEP): draft_len 3 at 1..8 requests, the drafter's + # full-vocabulary BF16 head (50016 rows per rank of 4, hidden 6144). Pass 0 reads the verify rows' hidden (4 per + # request) through x_rows and rolls the KV back; the later passes read one row per request. + _PASSES: tuple[Cell, ...] = ( + *( + Cell( + why=f"pass 0 at {n} request(s): head rows gathered from {4 * n} verify rows, KV rollback, seq_lens = 1", + spec=dict( + nreq=n, + draft_len=DRAFT_LEN, + step=0, + rank=n % 4, + world=4, + vs=DRAFT_SHARD, + k=HIDDEN, + ), + ) + for n in range(1, MAX_TOKENS + 1) + ), + Cell( + why="pass 1 at two requests: one row each, kv_lens + 1, rank 3's ids", + spec=dict( + nreq=2, draft_len=DRAFT_LEN, step=1, rank=3, world=4, vs=DRAFT_SHARD, k=HIDDEN + ), + ), + Cell( + why="last pass (step 2) at 8 requests, rank 0", + spec=dict( + nreq=8, draft_len=DRAFT_LEN, step=2, rank=0, world=4, vs=DRAFT_SHARD, k=HIDDEN + ), + ), + Cell( + why="pass 2 at 5 requests with exact cross-rank ties (the lower global id wins)", + spec=dict( + nreq=5, + draft_len=DRAFT_LEN, + step=2, + rank=1, + world=4, + vs=DRAFT_SHARD, + k=HIDDEN, + ties=True, + ), + ), + ) + # The coded head (the fused draft loop's default) at every pass shape, and the BF16 fragments (the coding's + # fail-closed path) at the first and last pass shapes and with ties. + CELLS: tuple[Cell, ...] = ( + *(Cell(why=f"coded: {c.why}", spec=dict(c.spec, coded=True)) for c in _PASSES), + *( + Cell(why=f"bf16: {c.why}", spec=dict(c.spec, coded=False)) + for c in (_PASSES[0], _PASSES[MAX_TOKENS - 1], _PASSES[-1]) + ), + ) + + note = """ + The head is the in-tree BF16 head's arithmetic (BF16 weights and activations, fp32 accumulation, a BF16 logit; + the coded head's weight is the BF16 shard bit for bit; its exceptions decode to +0 in the MMAs and their exact + products are added at the tile's flush) + and the argmax is the in-tree one over BF16 logits, so a token differs from the in-tree path's only where the fp32 + sum, accumulated in another order, rounds to the other side of a BF16 boundary and that decides the row's maximum. + The cells' rows are decidable and the peers' keys sit 0.25 away (or tie exactly); the edge tests (forced BF16 + ties, -0 / +0, top-2 one fp32 step apart that round to one BF16 value, adjacent BF16 values) use integer data + whose fp32 sums are exact in any order and hold the kernel to the in-tree path bit for bit. The cross-rank + exchange is certified on one device with the peers' keys written into the slots (see m3_shard_accept). + """ + + def __call__( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + fold: Optional[tuple] = None, + fixed_order: bool = False, + ) -> None: + # fold: (the 14 fold-tail tensors in buildParams' order, rows, tokens per step) -- the step's forward-tail store + # and next-step staging, run by the pass's last CTA after its tokens (unused by this target, which passes None). + tensors, fold_rows, fold_t = ( + (list(fold[0]), int(fold[1]), int(fold[2])) if fold is not None else (None, 0, 0) + ) + torch.ops.trtllm.minimax_m3_sm107tp4_draft_step( + x, + x_rows, + rows, + wf, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + num_ctas, + pdl, + out_keys, + exc_off, + exc_pos, + exc_w, + exp_base, + tensors, + fold_rows, + fold_t, + fixed_order, + ) + + def reference( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + fold: Optional[tuple] = None, + fixed_order: bool = False, + ) -> dict[str, torch.Tensor]: + """Every buffer the pass writes, computed on copies: "logits" (this rank's BF16 logits as fp32), "local_keys" + (this rank's key per row), "keys" (the winning key), "tokens", "draft_out", "ids_next", "pos_next", "kv_lens" + and "seq_lens". + + Reads ``ws_uc``, ``counters`` and ``kv_lens`` before the call. ``keys``, ``ws_mc_ptr``, ``num_ctas``, ``pdl``, + ``out_keys`` and ``fixed_order`` do not change the result; ``fold`` adds the fold tail's writes (m3_fold_tail), + not these. + """ + h = x[x_rows.long()] if x_rows is not None else x[:rows] + logits = shard_logits(h, wf, exc_off, exc_pos, exc_w, exp_base) + best, idx = local_argmax(logits, vocab_offset) + local_keys = make_keys(best, idx) + tokens = exchange_tokens(local_keys, ws_uc, counters, rank, world, MAX_TOKENS) + half = int(counters[1].item()) & 1 + slots = ws_uc.view(torch.int64)[ + half * MAX_TOKENS * world : (half * MAX_TOKENS + rows) * world + ] + slots = slots.view(rows, world).clone() + slots[:, rank] = local_keys + sign = torch.iinfo(torch.int64).min + win = (slots ^ sign).max(dim=1).values ^ sign + d_out = draft_out.clone() + d_out[:rows, step] = tokens + src = x_rows.long() if x_rows is not None else torch.arange(rows, device=x.device) + lens = kv_lens.clone() + sl = seq_lens.clone() if seq_lens is not None else None + if step == 0 and num_acc is not None: + lens[:rows] -= draft_len - num_acc[:rows] + sl[:rows] = 1 + elif step > 0: + lens[:rows] += 1 + out = { + "logits": logits, + "local_keys": local_keys, + "keys": win, + "tokens": tokens, + "draft_out": d_out, + "ids_next": tokens.clone(), + "pos_next": pos_src[src] + 1, + "kv_lens": lens, + } + if sl is not None: + out["seq_lens"] = sl + return out + + def is_valid( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + fold: Optional[tuple] = None, + fixed_order: bool = False, + ) -> None: + assert not bool(keys[:rows].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + # The kernel reads the head row x + x_rows[b] * K and the position pos_src[x_rows[b]] with no bound. + if x_rows is not None: + assert bool(((x_rows[:rows] >= 0) & (x_rows[:rows] < x.shape[0])).all()), ( + "x_rows must index x" + ) + assert bool((x_rows[:rows] < pos_src.shape[0]).all()), ( + "pos_src must cover the head rows" + ) + else: + assert pos_src.shape[0] >= rows, "pos_src must cover the head rows" + # Each CTA accumulates at most MAX_TILES_PER_CTA tiles in shared memory. + tiles = wf.shape[0] if exc_off is None else exc_off.numel() - 1 + assert -(-tiles // num_ctas) <= MAX_TILES_PER_CTA, "too many weight tiles per CTA" + # The atomics' arrival-order sum is deterministic only while no tile takes more than 2 warp pieces. + assert ( + fixed_order + or max_tile_pieces(tiles, num_ctas, x.shape[1] // (16 if exc_off is None else 64)) <= 2 + ), ( + "a tile takes more than 2 warp pieces on this grid: the atomics' sum is order-dependent (fixed_order)" + ) + if exc_off is not None: + # The exceptions are read per tile in list order with no bound against the coded stream. + assert int(exc_off[-1].item()) == exc_pos.numel(), ( + "exc_off must end at the exception count" + ) + assert bool((exc_off[1:] >= exc_off[:-1]).all()), "exc_off must be non-decreasing" + # A NaN logit maps to a key above every finite one and wins the row silently. + h = x[x_rows.long()] if x_rows is not None else x[:rows] + assert not bool(torch.isnan(h).any()), "NaN head inputs win the argmax" + if fold is not None: + assert len(fold) == 3 and len(fold[0]) == 14, ( + "fold is (the 14 fold-tail tensors, rows, tokens per step)" + ) + if num_acc is not None: + na = num_acc[:rows] + assert bool(((na >= 1) & (na <= draft_len + 1)).all()), ( + "num_acc counts must lie in [1, draft_len + 1]" + ) + + +m3_draft_step = M3DraftStep() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_force_vec.py new file mode 100644 index 000000000000..fcb327e374e3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_force_vec.py @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance, drawn as the in-tree worker draws them.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.m3_force_vec import force_vec_kernel + + +class M3ForceVec(OpWrapper): + """Bump ``counter`` (int64 [1]) by one, then for each of the ``batch_size`` requests ``b`` write + ``min(base_total + (pool[(c * counter_stride + b * slot_stride) & pool_mask] < frac), max_total)`` into ``out`` + (int32), ``c`` the bumped counter: the in-tree ``_apply_force_accepted_tokens`` draw over its fixed-seed fp32 + ``pool``. ``block`` is the launch width (a power of two >= batch_size; 16, 32 or 64 are compiled at warmup). + + Only the forced-acceptance testing knob (``TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`` with a fractional part) + runs it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, fraction 0.5", spec=dict(batch=1, block=16, frac=0.5)), + Cell(why="eight requests, fraction 0.3", spec=dict(batch=8, block=16, frac=0.3)), + ) + + note = """ + Integer arithmetic and one fp32 comparison: `reference` must match exactly. + """ + + def __call__( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + force_vec_kernel[(1,)]( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK=block, + ) + return out + + def reference( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + """The counts (a new tensor; ``counter`` and ``out`` are not written).""" + c = int(counter.item()) + 1 + slots = torch.arange(batch_size, dtype=torch.int64, device=pool.device) + idx = (c * counter_stride + slots * slot_stride) & pool_mask + r = pool[idx] + return torch.clamp(base_total + (r < frac).to(torch.int32), max=max_total).to(torch.int32) + + +m3_force_vec = M3ForceVec() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_force_vec_req.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_force_vec_req.py new file mode 100644 index 000000000000..5a2ffd8b6627 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_force_vec_req.py @@ -0,0 +1,62 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance, each request drawn at its own key.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.m3_force_vec import force_vec_req_kernel + + +class M3ForceVecReq(OpWrapper): + """For each of the ``batch_size`` requests ``b`` write ``min(base_total + (pool[(keys[b] * key_stride) & + pool_mask] < frac), max_total)`` into ``out`` (int32): the forced-acceptance draw from the fixed-seed fp32 ``pool`` + at the request's own key (int32 ``keys``, its KV length at this verify), so a request's draws depend on neither the + server's history nor the other requests. ``block`` is the launch width (a power of two >= batch_size; 16, 32 or + 64 are compiled at warmup). + + Only the forced-acceptance testing knob (``TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`` with a fractional part) + runs it, with ``FORCE_AL_PER_REQUEST``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, fraction 0.5", spec=dict(batch=1, block=16, frac=0.5)), + Cell(why="eight requests, fraction 0.3", spec=dict(batch=8, block=16, frac=0.3)), + ) + + note = """ + Integer arithmetic and one fp32 comparison: `reference` must match exactly. + """ + + def __call__( + self, pool, keys, out, batch_size, base_total, max_total, frac, key_stride, pool_mask, block + ): + force_vec_req_kernel[(1,)]( + pool, + keys, + out, + batch_size, + base_total, + max_total, + frac, + key_stride, + pool_mask, + BLOCK=block, + ) + return out + + def reference( + self, pool, keys, out, batch_size, base_total, max_total, frac, key_stride, pool_mask, block + ): + """The counts (a new tensor; ``out`` is not written).""" + idx = (keys[:batch_size].to(torch.int64) * key_stride) & pool_mask + r = pool[idx] + return torch.clamp(base_total + (r < frac).to(torch.int32), max=max_total).to(torch.int32) + + +m3_force_vec_req = M3ForceVecReq() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_sampled_accept.py new file mode 100644 index 000000000000..7623f5fe582b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_sampled_accept.py @@ -0,0 +1,289 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampled (temperature + top-p) Eagle3 verification over the TP-sharded vocabulary in one kernel +(``m3_sampled_accept.sampled_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_shard_accept import strict_accept + +#: Verify rows per call. +MAX_ROWS = 48 +#: The fused sampling kernel's 1 / T guard. +TEMP_EPSILON = 1e-6 + + +def nucleus_probs( + logits: torch.Tensor, temperatures: torch.Tensor, top_ps: torch.Tensor +) -> torch.Tensor: + """Per-row sampling distribution [rows, Vs] (fp32): w = exp(x / (T + 1e-6) - max), kept iff the mass strictly + above its weight is at most top_p * sum(w) (ties included), renormalized over the kept tokens.""" + inv = 1.0 / (temperatures.float() + TEMP_EPSILON) + x = logits.float() * inv[:, None] + w = torch.exp(x - x.max(dim=1, keepdim=True).values).double() + z = w.sum(dim=1, keepdim=True) + order = torch.sort(w, dim=1, descending=True) + sw = order.values + # mass strictly above each sorted weight: the cumulative sum before the first entry of its tie group + csum = torch.cumsum(sw, dim=1) - sw + first = torch.ones_like(sw, dtype=torch.bool) + first[:, 1:] = sw[:, 1:] != sw[:, :-1] + group = torch.cumsum(first.long(), dim=1) - 1 + starts = torch.zeros_like(csum) + starts.scatter_reduce_(1, group, csum, reduce="amin", include_self=False) + above = torch.gather(starts, 1, group) + keep_sorted = above <= top_ps.double()[:, None] * z + keep = torch.zeros_like(keep_sorted) + keep.scatter_(1, order.indices, keep_sorted) + p = torch.where(keep, w, torch.zeros_like(w)) + return (p / p.sum(dim=1, keepdim=True)).float() + + +class M3SampledAccept(OpWrapper): + """Sampled verification of ``rows = nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard. + + Each row draws one token from its top-p nucleus of softmax(logits / T) over the full (all-rank) vocabulary by + distributed rejection sampling (Philox keyed by the row's seed / offset, the row, the attempt and the CTA), then + the strict acceptance runs against ``draft`` as in ``m3_shard_accept``: ``acc_tokens``, ``num_acc``, + ``gather_ids``, optional ``next_new[:, 0]``. ``tokens`` receives the sampled token per row; ``extra_attempts`` + accumulates exchange rounds beyond the second. The exchange goes through the multicast workspace (two call + halves of ``minimax_m3_sm107tp4_sampled_accept_ws_words(world)`` words) with the (finished rows, call) ``counters``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # draft_len 3 (T = 4 rows per request) at the TP4 caps: 1 / 2 / 8 requests (4 / 8 / 32 rows, 8 CTAs per row) + # and the 48-row maximum (4 CTAs per row); the 50016-column target shard. Single rank (see note). + CELLS: tuple[Cell, ...] = ( + Cell( + why="cap 1 (4 rows), top_p 0.8 over a 12-token head: the draws follow the nucleus distribution", + spec=dict( + nreq=1, vs=50016, vocab_offset=0, check="distribution", top_p=0.8, temperature=0.7 + ), + ), + Cell( + why="cap 8 (32 rows, 256 co-resident CTAs): top_p ~ 0 keeps only the argmax, token exact", + spec=dict( + nreq=8, + vs=50016, + vocab_offset=2 * 50016, + check="argmax", + top_p=1e-7, + temperature=1.0, + ), + ), + Cell( + why="cap 2 (8 rows) with top_p = 1 (no cut: the first draw stands), next_new and forced counts", + spec=dict( + nreq=2, + vs=50016, + vocab_offset=50016, + check="distribution", + top_p=1.0, + temperature=1.0, + force="vec", + ), + ), + Cell( + why="48 rows (4 CTAs per row): the second launch geometry, nucleus distribution at top_p 0.5", + spec=dict( + nreq=12, + vs=50016, + vocab_offset=3 * 50016, + check="distribution", + top_p=0.5, + temperature=1.3, + force="total", + ), + ), + ) + + note = """ + What is sampled is gated statistically, the rest exactly. Every drawn token must lie in the reference nucleus; + the empirical distribution over many calls (distinct Philox offsets) must sit within a stated total-variation + band of the reference's; the acceptance outputs must equal the strict acceptance of the drawn tokens exactly. + The Philox stream itself is not reproduced in torch: the kernel's draw uses the hardware exp / log + approximations (__expf, __logf) and a CTA-order inverse CDF, which torch does not reproduce bit for bit. + + Certified at world 1 only. The decision is replicated from every rank's per-round records, which another rank + computes from its own shard on the fly, so peers cannot be stood in for on one device; the cross-rank path + needs the multi-device job and is not covered here. Rows with top-k or min-p must not reach this op (it applies + neither); is_valid states it. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + prewait_params: bool = True, + early_trigger: bool = False, + one_round_k: int = 0, + decided: Optional[torch.Tensor] = None, + ilp_walk: bool = False, + ) -> None: + torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept( + logits, + vocab_offset, + temperatures, + top_ps, + seeds, + offsets, + ws_uc, + ws_mc_ptr, + counters, + tokens, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + force_vec, + extra_attempts, + use_pdl, + prewait_params, + early_trigger, + one_round_k, + decided, + ilp_walk, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + prewait_params: bool = True, + early_trigger: bool = False, + one_round_k: int = 0, + decided: Optional[torch.Tensor] = None, + ilp_walk: bool = False, + ) -> dict: + """{"probs": the per-row nucleus distribution over this shard [rows, Vs] (global id = vocab_offset + + column), "accept": tokens -> the strict acceptance outputs}. The draw itself is random; ``seeds`` / + ``offsets`` select it and do not change the distribution. World 1: this shard is the whole vocabulary.""" + rows = logits.shape[0] + probs = nucleus_probs(logits, temperatures[:rows], top_ps[:rows]) + + def accept(drawn: torch.Tensor) -> dict[str, torch.Tensor]: + return strict_accept(drawn, draft, batch_idx, force_total, next_new, force_vec) + + return {"probs": probs, "accept": accept} + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + prewait_params: bool = True, + early_trigger: bool = False, + one_round_k: int = 0, + decided: Optional[torch.Tensor] = None, + ilp_walk: bool = False, + ) -> None: + rows = logits.shape[0] + # top-k and min-p are not applied: a row that has them gets a token outside its real sampling set. + if top_ks is not None: + k = top_ks[:rows] + assert bool(((k <= 0) | (k >= logits.shape[1] * world)).all()), ( + "top-k rows must take the generic path" + ) + if min_ps is not None: + assert not bool((min_ps[:rows] > 0).any()), "min-p rows must take the generic path" + # Rows are request-major verify rows: tokens[b * (D + 1) + k]; a mismatched layout pairs drafts with the + # wrong rows. + assert rows == draft.shape[0] * (draft.shape[1] + 1), ( + "logits rows must be nreq * (draft_len + 1)" + ) + # Per-row parameters are read at the row index, unchecked. + for t in (temperatures, top_ps, seeds, offsets): + assert t.numel() >= rows and t.is_contiguous(), ( + "per-row parameters must cover every row" + ) + assert temperatures.dtype == torch.float32 and top_ps.dtype == torch.float32, ( + "fp32 sampling parameters" + ) + assert seeds.dtype == torch.int64 and offsets.dtype == torch.int64, ( + "int64 Philox seeds / offsets" + ) + # The finished-row counter elects the CTA that runs the acceptance. + assert int(counters[0].item()) == 0, ( + "counters[0] (finished rows) must be zero between calls" + ) + + +m3_sampled_accept = M3SampledAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_sampled_accept_ws_words.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_sampled_accept_ws_words.py new file mode 100644 index 000000000000..9193b3dce61d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_sampled_accept_ws_words.py @@ -0,0 +1,59 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Size of the sampled-verify exchange workspace (``m3_sampled_accept.SampledAcceptWorkspace.create``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Exchange rounds per row, verify rows per call, CTAs per row (at most), uint64 words per record. +MAX_ATTEMPTS = 16 +MAX_ROWS = 48 +MAX_SPLITS = 8 +RECORD_WORDS = 4 +#: SA-1R (one_round_k 9 / 82): 9 listed bf16 values ride each record, 3 per extra word. +LIST_WORDS = {0: 0, 9: 3, 82: 3} + + +class M3SampledAcceptWsWords(OpWrapper): + """uint64 words of the multicast workspace ``minimax_m3_sm107tp4_sampled_accept`` exchanges through for + ``world`` ranks: two call halves of ``[attempts][rows][world * splits][record words]``.""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="the tp4 target's workspace", spec=dict(world=4)), + Cell(why="a single rank (local workspace, no multicast)", spec=dict(world=1)), + Cell( + why="the tp4 target's workspace with SA-1R pooled (82)", + spec=dict(world=4, one_round_k=82), + ), + Cell( + why="the tp4 target's workspace with SA-1R top-9 (9)", spec=dict(world=4, one_round_k=9) + ), + ) + + note = """ + A host computation (no device work); it is an op so the workspace size and the kernel's indexing come from one + definition. + """ + + def __call__(self, world: int, one_round_k: int = 0) -> int: + return torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept_ws_words(world, one_round_k) + + def reference(self, world: int, one_round_k: int = 0) -> int: + """Two halves of attempts x rows x (world x splits) records x record words.""" + return ( + 2 + * MAX_ATTEMPTS + * MAX_ROWS + * world + * MAX_SPLITS + * (RECORD_WORDS + LIST_WORDS[one_round_k]) + ) + + +m3_sampled_accept_ws_words = M3SampledAcceptWsWords() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_shard_accept.py new file mode 100644 index 000000000000..8beba4373b02 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_shard_accept.py @@ -0,0 +1,284 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Greedy Eagle3 verification from this rank's logits shard: per-row argmax, the cross-rank (value, index) exchange +and the strict acceptance in one kernel (``m3_shard_accept.shard_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Verify rows per call, and exchange rows per call half of its workspace. +SHARD_ROWS = 48 + +_LOW32 = 0xFFFFFFFF + + +def ordered_bits(v: torch.Tensor) -> torch.Tensor: + """The kernel's order-preserving uint32 image of fp32 values, as int64 (-0.0 maps to +0.0's image: they tie).""" + b = v.float().contiguous().view(torch.int32).to(torch.int64) & _LOW32 + b = torch.where(b == 0x80000000, torch.zeros_like(b), b) + return torch.where(b >= 0x80000000, _LOW32 - b, b | 0x80000000) + + +def make_keys(values: torch.Tensor, index: torch.Tensor) -> torch.Tensor: + """Exchange keys of (fp32 value, global token index) pairs: larger value wins, ties take the lower index.""" + return (ordered_bits(values) << 32) | (_LOW32 - index.to(torch.int64)) + + +def unsigned_max(keys: torch.Tensor, dim: int) -> torch.Tensor: + """Max of int64-stored uint64 keys (the kernel compares them unsigned).""" + sign = torch.iinfo(torch.int64).min + return (keys ^ sign).max(dim=dim).values ^ sign + + +def key_token(key: torch.Tensor) -> torch.Tensor: + return (_LOW32 - (key & _LOW32)).to(torch.int32) + + +def local_argmax(logits: torch.Tensor, vocab_offset: int) -> tuple[torch.Tensor, torch.Tensor]: + """(max value, lowest global index reaching it) per row of a [M, Vs] fp32 shard.""" + best = logits.max(dim=1).values + idx = (logits == best[:, None]).float().argmax(dim=1) # first (lowest) index at the max + return best, idx.to(torch.int64) + vocab_offset + + +def exchange_tokens( + own_keys: torch.Tensor, + ws_uc: torch.Tensor, + counters: torch.Tensor, + rank: int, + world: int, + slot_rows: int, +) -> torch.Tensor: + """Global tokens of the cross-rank pick: this rank's keys against the other ranks' keys already in its slots. + + Slot (half * slot_rows + t) * world + r of ``ws_uc`` holds rank r's key of row t; half = call counter & 1. A + single-device run stands in for the other ranks by writing their keys there before the call. + """ + m = own_keys.shape[0] + half = int(counters[1].item()) & 1 + slots = ( + ws_uc.view(torch.int64)[half * slot_rows * world : (half * slot_rows + m) * world] + .view(m, world) + .clone() + ) + slots[:, rank] = own_keys + return key_token(unsigned_max(slots, dim=1)) + + +def strict_accept( + tokens: torch.Tensor, + draft: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], +) -> dict[str, torch.Tensor]: + """The strict acceptance epilogue from the verify-row tokens [nreq * (D + 1)] (request-major). + + accepted[b] = tokens of request b's rows; num_accepted[b] = 1 + the matching prefix of draft[b] (overridden by + force_vec[b], else by force_total > 0); gather_ids[b] = batch_idx[b] * (D + 1) + num_accepted[b] - 1, and + next_new[b, 0] (when given) = the token at that row. Returns the outputs plus ``next_new`` as written. + """ + nreq, d = draft.shape + k1 = d + 1 + tokens = tokens.to(torch.int32) + accepted = tokens.view(nreq, k1).clone() + match = (draft.to(torch.int32) == accepted[:, :d]).to(torch.int32) + num = 1 + match.cumprod(dim=1).sum(dim=1) + if force_vec is not None: + num = force_vec[:nreq].to(num.dtype) + elif force_total > 0: + num = torch.full_like(num, force_total) + gather = batch_idx.to(torch.int64) * k1 + num.to(torch.int64) - 1 + out = {"accepted": accepted, "num_accepted": num.to(torch.int32), "gather_ids": gather} + if next_new is not None: + nn = next_new.clone() + nn[:, 0] = tokens[gather] + out["next_new"] = nn + return out + + +class M3ShardAccept(OpWrapper): + """Greedy verification of ``nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard ``[rows, Vs]`` + (global ids ``vocab_offset ..``). + + Each row's token is its global argmax over all ranks' shards: keys ``ordered(value) << 32 | ~index`` (ties take + the lowest index, as ``torch.argmax``) are max-reduced over the shard, pushed into every rank's exchange slot + through the multicast slots ``[2][48][world]`` (Lamport: 0 = empty) and max-reduced over the ranks by the last + CTA, which then runs the strict acceptance: ``acc_tokens [nreq, D + 1]``, ``num_acc [nreq]`` (``force_vec`` / + ``force_total`` override the counts), ``gather_ids [nreq]`` (the first draft pass's row per request) and, when + given, ``next_new[:, 0]`` (the last accepted token). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # Every greedy verify step of the target (modeling.SHARD_ACCEPT): draft_len 3 at 1..12 requests (4..48 rows), the + # target lm_head's BF16 shard logits (50016 columns per rank at TP4). The logits come from the head GEMM, so the + # op only reduces. + CELLS: tuple[Cell, ...] = ( + Cell( + why="cap 1: 4 verify rows (1 request x T = 4), rank 0 of 4, next_new staged", + spec=dict(nreq=1, draft_len=3, vs=50016, world=4, rank=0, force="none", strided=False), + ), + Cell( + why="cap 2: 8 verify rows, rank 2, global ids offset by 2 x 50016", + spec=dict(nreq=2, draft_len=3, vs=50016, world=4, rank=2, force="none", strided=False), + ), + Cell( + why="cap 4: 16 verify rows (4 requests x T = 4), rank 1 of 4, next_new staged", + spec=dict(nreq=4, draft_len=3, vs=50016, world=4, rank=1, force="none", strided=False), + ), + Cell( + why="cap 8: 32 verify rows, rank 3, global ids offset by 3 x 50016", + spec=dict(nreq=8, draft_len=3, vs=50016, world=4, rank=3, force="none", strided=False), + ), + Cell( + why="12 rows (3 requests) read from a row-strided view of a wider logits buffer, forced counts", + spec=dict(nreq=3, draft_len=3, vs=50016, world=4, rank=0, force="vec", strided=True), + ), + Cell( + why="the 48-row maximum (12 requests at draft 3), force_total, rows with exact cross-rank ties", + spec=dict( + nreq=12, + draft_len=3, + vs=50016, + world=4, + rank=2, + force="total", + strided=False, + ties=True, + ), + ), + # Every other request count of the verify range, on the 4-rank 50016-column shards. + *( + Cell( + why=f"{n} requests ({4 * n} verify rows): a request count between the ones above", + spec=dict( + nreq=n, draft_len=3, vs=50016, world=4, rank=n % 4, force="none", strided=False + ), + ) + for n in (5, 6, 7, 9, 10, 11) + ), + ) + + note = """ + The op reduces BF16 logits exactly, so the reference is bit-exact, including cross-rank ties (lowest global + index wins). The cross-rank exchange is certified on one device: the other ranks' keys are written into this + rank's slots before the call, which is what their multicast pushes would have put there; the multicast mapping + itself (ws_mc_ptr = this rank's slots, as at world 1) is not exercised across devices here. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + torch.ops.trtllm.minimax_m3_shard_accept( + logits, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + num_ctas, + force_vec, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> dict[str, torch.Tensor]: + """Lowest-index maximum of each row's BF16 logits, the largest key among it and the peers' keys in + ``ws_uc``, then the strict acceptance. Reads ``ws_uc`` / ``counters`` before the call; the scratch and + output buffers and ``num_ctas`` do not change the result.""" + best, idx = local_argmax(logits.float(), vocab_offset) + tokens = exchange_tokens(make_keys(best, idx), ws_uc, counters, rank, world, SHARD_ROWS) + out = strict_accept(tokens, draft, batch_idx, force_total, next_new, force_vec) + out["tokens"] = tokens + return out + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + assert not bool(keys[: logits.shape[0]].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + assert ws_uc.numel() >= 2 * SHARD_ROWS * world, "ws_uc must hold [2][48][world] slots" + # A NaN logit maps to a key above every finite one and wins the row silently. + assert not bool(torch.isnan(logits).any()), "NaN logits win the argmax" + nreq = draft.shape[0] + assert bool(((batch_idx[:nreq] >= 0) & (batch_idx[:nreq] < nreq)).all()), ( + "batch_idx must index the nreq requests" + ) + if force_vec is not None: + fv = force_vec[:nreq] + assert bool(((fv >= 1) & (fv <= draft.shape[1] + 1)).all()), ( + "force_vec counts must lie in [1, D + 1]" + ) + + +m3_shard_accept = M3ShardAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_slot_scatter.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_slot_scatter.py new file mode 100644 index 000000000000..ff07c54de1f0 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_slot_scatter.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The spec sampler's per-slot stores of one step's outputs, in one launch (``M3SpecStepFold.scatter`` -> +``m3_spec_tail.slot_scatter``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +def padded_rows(src: torch.Tensor, width: int) -> torch.Tensor: + """src rows zero-padded (or cut) to ``width`` columns.""" + out = torch.zeros(src.shape[0], width, dtype=src.dtype, device=src.device) + w = min(width, src.shape[1]) + out[:, :w] = src[:, :w] + return out + + +class M3SlotScatter(OpWrapper): + """For i < len(slots), all int32 2-D strided views: ``da[slots[i]] = sa[i]``, ``db[slots[i]] = sb[i]``, + ``dd[slots[i]] = sd[i]`` (each zero-padded / cut to the destination width) and ``dl[slots[i]] = sl[i]``. The + destinations are the store's transposed [slots, width] views.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The sampler store at draft_len 3 (8 slots): new tokens / next new tokens [slots, 4] (transposed views of + # [4, slots, 1]), next draft tokens [slots, 3]; a worker output narrower than the store (dynamic draft length) + # is zero-padded. + CELLS: tuple[Cell, ...] = ( + Cell( + why="8 rows into shuffled slots, widths equal to the store's", + spec=dict(n=8, slots=8, src_w=4, src_d=3, store_w=4, store_d=3), + ), + Cell( + why="3 rows of a narrower step (runtime draft 2): zero padding to the store width", + spec=dict(n=3, slots=8, src_w=3, src_d=2, store_w=4, store_d=3), + ), + ) + + note = """ + Pure index copies: the reference is exact. Slots are not bounds-checked; is_valid states it. + """ + + def __call__(self, slots, sa, da, sb, db, sl, dl, sd, dd): + torch.ops.trtllm.minimax_m3_slot_scatter(slots, sa, da, sb, db, sl, dl, sd, dd) + + def reference(self, slots, sa, da, sb, db, sl, dl, sd, dd): + """The four destinations after the call, on copies.""" + s = slots.long() + out = {} + for name, src, dst in (("da", sa, da), ("db", sb, db), ("dd", sd, dd)): + r = dst.clone() + r[s] = padded_rows(src[: s.numel()], dst.shape[1]) + out[name] = r + lr = dl.clone() + lr[s] = sl[: s.numel()] + out["dl"] = lr + return out + + def is_valid(self, slots, sa, da, sb, db, sl, dl, sd, dd): + assert bool(((slots >= 0) & (slots < da.shape[0]) & (slots < dl.numel())).all()), ( + "slots must index the store" + ) + # Two rows into one slot race (which one lands is unordered). + assert slots.unique().numel() == slots.numel(), "slots must be distinct" + assert sa.shape[0] >= slots.numel() and sl.numel() >= slots.numel(), "a source row per slot" + + +m3_slot_scatter = M3SlotScatter() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_spec_offsets.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_spec_offsets.py new file mode 100644 index 000000000000..36e421412f30 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_spec_offsets.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The overlap scheduler's speculative step correction as one launch (``RuntimeHooks.spec_offsets_op``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.logger import logger + +# The engagement line is logged once per process (module state: __call__ reads no instance state). +_ENGAGED: list = [] + + +class M3SpecOffsets(OpWrapper): + """``positions += position_offsets`` and ``kv_lens += kv_lens_offsets`` (int32, contiguous 1-D views, equal lengths + per pair) in one plain launch: trtllm::minimax_m3_sm107_tp4_spec_offsets (minimaxM3Sm107Tp4/specOffsets.cu), the + same wrapping int32 adds the two ATen ops do. ``__call__`` returns False, having written nothing, for other + layouts (the caller runs the ATen adds).""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="1 request, 4 tokens (draft length 3)", spec=dict(tokens=4, seqs=1)), + Cell(why="8 requests, 32 tokens", spec=dict(tokens=32, seqs=8)), + Cell(why="wrap-around values", spec=dict(tokens=8, seqs=2, wrap=True)), + ) + + note = "Integer adds: the reference (the two in-place ATen adds) is exact." + + def __call__(self, pos, pos_off, kv, kv_off) -> bool: + ts = (pos, pos_off, kv, kv_off) + if ( + any( + t.dtype != torch.int32 or t.dim() != 1 or t.stride(0) != 1 or not t.is_cuda + for t in ts + ) + or pos.numel() != pos_off.numel() + or kv.numel() != kv_off.numel() + ): + return False + torch.ops.trtllm.minimax_m3_sm107_tp4_spec_offsets(pos, pos_off, kv, kv_off) + if not _ENGAGED: + _ENGAGED.append(True) + logger.info("[M3 host] spec step correction: one launch taken (spec_offsets_op)") + return True + + def reference(self, pos, pos_off, kv, kv_off) -> None: + pos += pos_off + kv += kv_off + + def is_valid(self, pos, pos_off, kv, kv_off) -> None: + assert pos.numel() == pos_off.numel() and kv.numel() == kv_off.numel(), ( + "equal lengths per pair" + ) + + +m3_spec_offsets = M3SpecOffsets() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_step_gather.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_step_gather.py new file mode 100644 index 000000000000..16c4000199da --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_step_gather.py @@ -0,0 +1,152 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The overlap scheduler's per-step input gathers from the previous step's device outputs, in one launch +(``M3SpecStepFold.gather`` -> ``m3_spec_tail.step_gather``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3StepGather(OpWrapper): + """For the ``P`` previous requests (sequence slots ``slots``, ``T`` tokens and ``D`` draft tokens each), all + int32:: + + ids[ids0 + p*T + t] = new_tok[t, slots[p]] (new_tok [T', slots, (1)]) + draft[d0 + p*D + d] = next_draft[slots[p], d] + pos_off[i] = lens[ppi[i - b0*T]] for i in [b0*T, (b0+P)*T), else 0 (every i < pos_off.numel()) + kv_off[i] = lens[slots[i - b0]] - T for i in [b0, b0+P), else 0 (every i < kv_off.numel()) + + Every other element of ``ids`` / ``draft`` is left as it was. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # draft_len 3: T = 4, D = 3; the store's [T, slots, 1] new tokens and [slots, D] next drafts (8 slots, the + # max batch); the engine's flat buffers sized for the max batch. + CELLS: tuple[Cell, ...] = ( + Cell( + why="an unchanged batch of 8 previous requests in shuffled slots, nothing before them (b0 = 0)", + spec=dict(p=8, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0), + ), + Cell( + why="2 previous requests after 3 new ones: offsets ids0 / d0 / b0 nonzero, zero-filled offsets outside", + spec=dict(p=2, t=4, d=3, slots=8, b0=3, ids0=12, d0=9, extra_rows=3), + ), + Cell( + why="one request, the store's 2-D view of new tokens (beam dim squeezed)", + spec=dict(p=1, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0, two_d=True), + ), + ) + + note = """ + Pure index copies: the reference is exact. The op does not bound slots, ppi or the destination offsets; is_valid + states them. + """ + + def __call__( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + torch.ops.trtllm.minimax_m3_step_gather( + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ) + + def reference( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + """The four destinations after the call, on copies (``ids`` / ``draft`` keep their other elements).""" + nt = new_tok.reshape(new_tok.shape[0], new_tok.shape[1]) + s = slots[:P].long() + ids_r, draft_r = ids.clone(), draft.clone() + ids_r[ids0 : ids0 + P * T] = nt[:T, s].t().reshape(-1) + draft_r[d0 : d0 + P * D] = next_draft[s, :D].reshape(-1) + pos_r = torch.zeros_like(pos_off) + pos_r[b0 * T : (b0 + P) * T] = lens[ppi[: P * T].long()] + kv_r = torch.zeros_like(kv_off) + kv_r[b0 : b0 + P] = lens[s] - T + return {"ids": ids_r, "draft": draft_r, "pos_off": pos_r, "kv_off": kv_r} + + def is_valid( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + # Reads through slots / ppi and writes at the offsets are unchecked: any of them out of range reads or + # writes another buffer. + s = slots[:P] + assert bool( + ( + (s >= 0) & (s < next_draft.shape[0]) & (s < new_tok.shape[1]) & (s < lens.numel()) + ).all() + ), "slots must index the store" + assert bool(((ppi[: P * T] >= 0) & (ppi[: P * T] < lens.numel())).all()), ( + "ppi must index lens" + ) + assert new_tok.shape[0] >= T and next_draft.shape[1] >= D, "store narrower than T / D" + assert ids.numel() >= ids0 + P * T and draft.numel() >= d0 + P * D, "destinations too short" + assert pos_off.numel() >= (b0 + P) * T and kv_off.numel() >= b0 + P, ( + "offset buffers too short" + ) + + +m3_step_gather = M3StepGather() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_step_prologue.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_step_prologue.py new file mode 100644 index 000000000000..122d43fb175a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/ops/spec/m3_step_prologue.py @@ -0,0 +1,76 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode graph's step-input prologue in one launch (this target's ``RuntimeHooks.pack_prologue_op``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3StepPrologue(OpWrapper): + """``StepInputPack``'s early-copy prologue as one single-CTA C++ launch + (trtllm::minimax_m3_sm107_tp4_step_prologue, minimaxM3Sm107Tp4/stepPrologue.cu). + + ``window`` is the pack's device window (``gdev``): an int64 header ``[host copies, device copies]``, then one + ``(dst, src, bytes)`` descriptor per copy, host copies first (a host copy's ``src`` is a byte offset into the + window, a device copy's a device address). The launch writes up to ``max_desc`` copies of each kind, every host + copy before any device copy, then zeroes the header; with ``epoch`` it increments ``epoch[0]`` once. The default + prologue does the same with two parallel-scatter launches (``step_inputs._graph_scatter_kernel``) and a header + fill.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The TP4 decode step's packs: a few dozen host copies of 4 to a few hundred bytes (positions, KV lengths, slots, + # the spec sampler's uploads), a handful of device copies (the graph runner's static inputs), some reading what a + # host copy of the same window wrote. + CELLS: tuple[Cell, ...] = ( + Cell(why="an empty window (a replay with no new upload)", spec=dict(h2d=0, d2d=0)), + Cell(why="host copies only, odd sizes and alignments", spec=dict(h2d=40, d2d=0)), + Cell( + why="host copies feeding device copies (a device copy reads a host copy's destination)", + spec=dict(h2d=24, d2d=8, chained=True), + ), + Cell( + why="every descriptor slot (128 + 128 of the pack's 256)", spec=dict(h2d=128, d2d=128) + ), + ) + + note = """ + Byte copies: the reference (the default prologue's Triton launches and fill on the same window) is exact. + Destinations are raw addresses; is_valid cannot check them. + """ + + def __call__(self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int) -> None: + torch.ops.trtllm.minimax_m3_sm107_tp4_step_prologue(window, epoch, max_desc) + + def reference(self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int) -> None: + """The default prologue on the same arguments: it writes the same destinations.""" + from tensorrt_llm._torch.pyexecutor.step_inputs import _graph_scatter_kernel + + stage_i64 = window.view(torch.int64) + for device_src in (False, True): + bump = epoch is not None and not device_src + _graph_scatter_kernel[(max_desc,)]( + window, + stage_i64, + epoch if bump else stage_i64, + DEVICE_SRC=device_src, + BLOCK=256, + BUMP_EPOCH=bump, + num_warps=1, + ) + window[:16].zero_() + + def is_valid(self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int) -> None: + assert window.dtype == torch.uint8 and window.is_contiguous(), "window: contiguous uint8" + assert window.numel() >= 16 + 24 * max_desc, "window: header and max_desc descriptors" + counts = window[:16].view(torch.int64).tolist() + assert counts[0] + counts[1] <= max_desc, "at most max_desc copies in all" + + +m3_step_prologue = M3StepPrologue() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/weights.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/weights.py new file mode 100644 index 000000000000..67682313ac5e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/weights.py @@ -0,0 +1,910 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight tables and loaders: MiniMax-M3.1 (MXFP8 + packed NVFP4 experts) / sm_107 / tp2. + +Two tables, one per checkpoint the target reads: + +* ``MODEL_WEIGHTS`` (``MiniMaxM3Tp2Weights``): the vision-language checkpoint's language model, + ``language_model.*``. Every rank is handed the whole checkpoint, so every tensor-parallel split is this + table's: the Q heads, the K/V/index-Q heads (one each per rank), the dense MLPs and the shared expert's + intermediate, the o_proj / down-projection input columns, every routed expert's intermediate (all 128 + experts live on every rank), and the vocabulary of the shell's ``lm_head``. The index-K head is replicated. +* ``DRAFT_WEIGHTS`` (``MiniMaxM3Tp2DraftWeights``): the one-layer BF16 Eagle3 drafter's own checkpoint + (``speculative_model``), loaded through the shell's ``load_draft_weights`` when speculation is on. + +Storage is HF ``[out, in]`` row-major. Five families need a transform, each a layout a kernel demands: + +* **MXFP8 block scales** (every attention projection, the dense MLPs and the shared expert): the checkpoint + stores UE8M0 bytes ``[N, K / 32]``; the GEMMs read the 128x4-swizzled order + (``trtllm::block_scale_interleave``). A fused projection concatenates its shards' scales first and swizzles + once. +* **The five-way sparse-layer projection** ``[Q | K | V | index-Q | index-K]`` and the three-way dense one + ``[Q | K | V]``: the per-rank shards concatenated along N. +* **Gated MLPs** (dense and shared expert): ``[gate ; up]`` rows. +* **NVFP4 routed experts** (layers 3..59): ``[up (w3) ; gate (w1)]`` rows, then the trtllm-gen row shuffle of + the weight and of its E4M3 scales (gated-activation reorder + 128-row epilogue tile shuffle, + ``trtllmgen_maybe_get_cached_*_permute_indices``) and the 128x4 swizzle of the scales. +* **Gemma boundary norms**: ``input_layernorm``, ``post_attention_layernorm`` and ``model.norm`` are stored as + Gemma norms, ``(1 + w) * x``; the fused all-reduce + RMSNorm kernels apply ``w * x``, so ``1`` is added + (in BF16) at load. The per-head Q/K/index norms stay Gemma: their kernels apply the offset themselves. + +The routed experts' per-tensor scalars (input scales and second-level weight scales, every expert) load raw; +``modeling`` derives the kernels' global scales, alphas and activation constants from them once the weights +are real. + +Loading contract: ``load(model, weights)`` consumes the engine-provided checkpoint dict, fills every declared +parameter exactly once and asserts coverage in both directions: every declared parameter written, and every +checkpoint key either consumed or in ``expected_unconsumed`` (the vision tower and its projector, which this +text-only target does not build, and the language model's multi-token-prediction head). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn + +from tensorrt_llm._torch.models.checkpoints.hf.minimaxm3_weight_mapper import ( + reciprocal_global_scale, +) +from tensorrt_llm._torch.moe.fused_moe.quantization import ( + trtllmgen_maybe_get_cached_w2_permute_indices, + trtllmgen_maybe_get_cached_w3_w1_permute_indices, +) + +#: ``layers=ALL_LAYERS`` is the common case: a weight every decoder layer carries. +ALL_LAYERS = object() + +_LM = "language_model." +#: Checkpoint key families this target does not build: the vision tower and its two projectors (text only), and the +#: language model's multi-token-prediction head (speculation, when on, is the separate Eagle3 drafter). +_UNBUILT_PREFIXES = ( + "vision_tower.", + "multi_modal_projector.", + "patch_merge_mlp.", + _LM + "model.mtp.", +) + +MX_BLOCK = 32 # MXFP8: one UE8M0 scale per 32 elements along K +NV_BLOCK = 16 # NVFP4: one E4M3 scale per 16 elements along K +EPILOGUE_TILE_M = ( + 128 # the trtllm-gen MoE runner's row tile: fixes its weight and scale row shuffles +) + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + ``shape`` takes the dimension bundle and returns the declared shape. ``src`` is a callable + ``(d, i) -> [(key or key tuple, index into the param | None), ...]`` with keys relative to the checkpoint + root; a key tuple hands the transform those tensors together. ``layers`` is ``ALL_LAYERS``, ``None`` (not + per-layer) or ``d -> layer indices``. ``transform(src, d)`` runs on the destination's device before the copy. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + +# ------------------------------------------------------------------------------------------------ shards + + +def fused_projection(sparse_config) -> bool: + """Whether the sparse layers run the five-way packed QKV + index projection. + + On by default: this target requires the MSA implementation and an FP8 indexer, which the packed projection needs. + An explicit ``fuse_qkv_index_projection`` in the deployment's sparse attention config wins, so setting it to False + is the kill switch (the QKV projection plus a separate MXFP8 index projection). + """ + if sparse_config is None: + return False + # A config that does not record which fields were set (a plain namespace or dataclass) is taken at its word. + fields_set = getattr(sparse_config, "model_fields_set", None) + if fields_set is None or "fuse_qkv_index_projection" in fields_set: + return bool(getattr(sparse_config, "fuse_qkv_index_projection", False)) + return True + + +def _col( + t: torch.Tensor, d: SimpleNamespace, tp: int | None = None, rank: int | None = None +) -> torch.Tensor: + """This rank's rows (column-parallel output split), even split, sliced on the host and moved to the + destination device (only this rank's bytes are read).""" + tp = d.tp if tp is None else tp + rank = d.rank if rank is None else rank + n = t.shape[0] // tp + return _dev(t[rank * n : (rank + 1) * n], d) + + +def _row(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """This rank's input columns (row-parallel input split), even split, on the destination device.""" + n = t.shape[1] // d.tp + return _dev(t[:, d.rank * n : (d.rank + 1) * n], d) + + +def _dev(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """``t`` on the destination device (``d.dev``, set per fill).""" + return t.to(d.dev, non_blocking=True) + + +def _swizzle(scale: torch.Tensor) -> torch.Tensor: + """128x4 swizzle of a row-major ``[N, K / 32]`` UE8M0 scale matrix, flat.""" + return torch.ops.trtllm.block_scale_interleave(scale.contiguous()) + + +# --------------------------------------------------------------------------------------------- transforms + + +def _gemma_fold(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A Gemma boundary norm as the plain norm the fused all-reduce epilogue applies: ``w + 1`` in BF16.""" + return t + 1.0 + + +def _f32(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return t.to(torch.float32) + + +def _divisor(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A stored global scale (multiplier) -> the modelopt divisor: its FP32 reciprocal.""" + return reciprocal_global_scale("expert global scale", t) + + +def _qkv_sparse_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[Q | K | V | index-Q | index-K]``: Q by attention heads, K/V/index-Q by KV heads, index-K replicated.""" + q, k, v, iq, ik = t + return torch.cat( + [_col(q, d), _col(k, d), _col(v, d), _col(iq, d), _dev(ik, d)], dim=0 + ).contiguous() + + +def _qkv_sparse_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v, iq, ik = t + return _swizzle( + torch.cat([_col(q, d), _col(k, d), _col(v, d), _col(iq, d), _dev(ik, d)], dim=0) + ) + + +def _qkv_dense_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + return torch.cat([_col(q, d), _col(k, d), _col(v, d)], dim=0).contiguous() + + +def _qkv_dense_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + return _swizzle(torch.cat([_col(q, d), _col(k, d), _col(v, d)], dim=0)) + + +def _index_qk_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[index-Q | index-K]`` as the checkpoint's MXFP8 (E4M3) values: this rank's index-Q heads and the replicated + index-K, the same rows the five-way projection packs after Q / K / V.""" + iq, ik = t + return torch.cat([_col(iq, d), _dev(ik, d)], dim=0).contiguous() + + +def _index_qk_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """The UE8M0 block scales of ``_index_qk_w``, 128x4-swizzled as every MXFP8 projection's.""" + iq, ik = t + return _swizzle(torch.cat([_col(iq, d), _dev(ik, d)], dim=0)) + + +def _gate_up_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return torch.cat([_col(gate, d), _col(up, d)], dim=0).contiguous() + + +def _gate_up_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return _swizzle(torch.cat([_col(gate, d), _col(up, d)], dim=0)) + + +def _row_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _row(t, d).contiguous() + + +def _row_s(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _swizzle(_row(t, d)) + + +def _col_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _col(t, d).contiguous() + + +#: Permute indices of the trtllm-gen expert relayouts, keyed by (shape, family, scale group): one expert's are +#: every expert's, so each is computed once per load. +_PERMUTE_CACHE: dict = {} + + +def _expert_w31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 operand: ``[up ; gate]`` packed-FP4 rows of this rank, then the gated-activation row + reorder and the epilogue-tile shuffle.""" + up, gate = t + w = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + w, _PERMUTE_CACHE, EPILOGUE_TILE_M, is_gated_act_gemm=True + ) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 E4M3 scales: ``[up ; gate]`` rows of this rank, the same row reorder at scale + granularity, then the 128x4 swizzle.""" + up, gate = t + s = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK, is_gated_act_gemm=True + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled.view(torch.uint8).reshape(shape)) + .view(torch.float8_e4m3fn) + .reshape(shape) + ) + + +def _expert_w2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 operand: this rank's input columns, then the epilogue-tile row shuffle.""" + w = _row(t, d).contiguous() + perm = trtllmgen_maybe_get_cached_w2_permute_indices(w, _PERMUTE_CACHE, EPILOGUE_TILE_M) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 E4M3 scales: this rank's columns, the row shuffle at scale granularity, the swizzle.""" + s = _row(t, d).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w2_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled).view(torch.float8_e4m3fn).reshape(shape) + ) + + +# ------------------------------------------------------------------------------------------------ sources + + +def _one(template: str): + return lambda d, i, t=template: [(_LM + t.format(i=i), None)] + + +def _many(*templates: str): + return lambda d, i, ts=templates: [(tuple(_LM + t.format(i=i) for t in ts), None)] + + +def _per_expert(*suffixes: str): + """One source per routed expert, written at its expert index (all experts are local at tp4).""" + + def build(d, i, sx=suffixes): + out = [] + for e in range(d.num_experts): + keys = tuple(_LM + f"model.layers.{i}.block_sparse_moe.experts.{e}.{s}" for s in sx) + out.append((keys if len(keys) > 1 else keys[0], (e,))) + return out + + return build + + +def _dense_mlp_layers(d): + """Layers with the dense SwiGLU-OAI MLP (``moe_layer_freq`` 0).""" + return d.dense_mlp + + +def _moe_layers(d): + return d.moe + + +def _sparse_attn_layers(d): + """Layers with MSA sparse attention (``sparse_attention_freq`` 1): every layer of MiniMax-M3.1.""" + return d.sparse_attn + + +def _three_way_qkv_layers(d): + """Layers whose QKV is the three-way [Q | K | V]: the dense-attention layers, and every layer without the + five-way packed projection (``fuse_qkv_index_projection=False``).""" + return d.dense_attn if d.fused else range(d.num_layers) + + +def _five_way_qkv_layers(d): + return d.sparse_attn if d.fused else range(0) + + +def _unfused_sparse_layers(d): + return range(0) if d.fused else d.sparse_attn + + +_ATTN = "model.layers.{i}.self_attn" +_MOE = "model.layers.{i}.block_sparse_moe" + + +class MiniMaxM3Tp2Weights: + """MiniMax-M3 NVFP4 / sm_107 / tp2: the language model's table and the passes over it. + + ``declare`` allocates every weight it names, ``manifest`` maps checkpoint keys onto them, ``load`` fills + them; ``_targets`` is the one place that decides which parameters exist. + """ + + #: Free the per-expert relayout scratch after each expert stack. + RELEASE_AFTER = "_e_s2" + + WEIGHTS: tuple[W, ...] = ( + # --- the two boundary norms of every layer (Gemma, folded) --- + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.input_layernorm.weight"), + transform=_gemma_fold, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.post_attention_layernorm.weight"), + transform=_gemma_fold, + ), + # --- attention: per-head Q/K norms (Gemma, unfolded), the packed QKV and the output projection --- + W("q_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".q_norm.weight")), + W("k_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".k_norm.weight")), + W( + "qkv_w", + shape=lambda d: (d.qkv_dense_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in "qkv")), + transform=_qkv_dense_w, + layers=_three_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_dense_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many(*(f"{_ATTN}.{p}_proj.weight_scale_inv" for p in "qkv")), + transform=_qkv_dense_s, + layers=_three_way_qkv_layers, + ), + W( + "qkv_w", + shape=lambda d: (d.qkv_sparse_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in ("q", "k", "v", "index_q", "index_k"))), + transform=_qkv_sparse_w, + layers=_five_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_sparse_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + *( + f"{_ATTN}.{p}_proj.weight_scale_inv" + for p in ("q", "k", "v", "index_q", "index_k") + ) + ), + transform=_qkv_sparse_s, + layers=_five_way_qkv_layers, + ), + # The separate index projection [index-Q | index-K] of the unfused sparse layers: the checkpoint's MXFP8 + # values with their block scales (the rows of the five-way projection's index part), run as an MXFP8 GEMM. + W( + "idx_w", + shape=lambda d: (d.index_heads * d.index_dim + d.index_dim, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(_ATTN + ".index_q_proj.weight", _ATTN + ".index_k_proj.weight"), + transform=_index_qk_w, + layers=_unfused_sparse_layers, + ), + W( + "idx_s", + shape=lambda d: ((d.index_heads * d.index_dim + d.index_dim) * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _ATTN + ".index_q_proj.weight_scale_inv", _ATTN + ".index_k_proj.weight_scale_inv" + ), + transform=_index_qk_s, + layers=_unfused_sparse_layers, + ), + W( + "idx_q_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_q_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "idx_k_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_k_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "o_w", + shape=lambda d: (d.hidden, d.q_width), + dtype=torch.float8_e4m3fn, + src=_one(_ATTN + ".o_proj.weight"), + transform=_row_w, + ), + W( + "o_s", + shape=lambda d: (d.hidden * (d.q_width // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_ATTN + ".o_proj.weight_scale_inv"), + transform=_row_s, + ), + # --- dense SwiGLU-OAI MLP (layers 0..2) --- + W( + "mlp_gu_w", + shape=lambda d: (2 * d.dense_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight", "model.layers.{i}.mlp.up_proj.weight" + ), + transform=_gate_up_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_gu_s", + shape=lambda d: (2 * d.dense_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight_scale_inv", + "model.layers.{i}.mlp.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_w", + shape=lambda d: (d.hidden, d.dense_inter), + dtype=torch.float8_e4m3fn, + src=_one("model.layers.{i}.mlp.down_proj.weight"), + transform=_row_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_s", + shape=lambda d: (d.hidden * (d.dense_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one("model.layers.{i}.mlp.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_dense_mlp_layers, + ), + # --- MoE (layers 3..59): the fp32 router and its bias, the shared expert --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + dtype=torch.float32, + src=_one(_MOE + ".gate.weight"), + transform=_f32, + layers=_moe_layers, + ), + W( + "router_bias", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_one(_MOE + ".e_score_correction_bias"), + transform=_f32, + layers=_moe_layers, + ), + W( + "sh_gu_w", + shape=lambda d: (2 * d.shared_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + _MOE + ".shared_experts.gate_proj.weight", _MOE + ".shared_experts.up_proj.weight" + ), + transform=_gate_up_w, + layers=_moe_layers, + ), + W( + "sh_gu_s", + shape=lambda d: (2 * d.shared_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _MOE + ".shared_experts.gate_proj.weight_scale_inv", + _MOE + ".shared_experts.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_moe_layers, + ), + W( + "sh_dn_w", + shape=lambda d: (d.hidden, d.shared_inter), + dtype=torch.float8_e4m3fn, + src=_one(_MOE + ".shared_experts.down_proj.weight"), + transform=_row_w, + layers=_moe_layers, + ), + W( + "sh_dn_s", + shape=lambda d: (d.hidden * (d.shared_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_MOE + ".shared_experts.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_moe_layers, + ), + # --- the routed experts, NVFP4 (trtllm-gen layouts), all 128 on every rank --- + W( + "e_w31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // 2), + dtype=torch.uint8, + src=_per_expert("w3.weight_packed", "w1.weight_packed"), + transform=_expert_w31, + layers=_moe_layers, + ), + W( + "e_w2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // 2), + dtype=torch.uint8, + src=_per_expert("w2.weight_packed"), + transform=_expert_w2, + layers=_moe_layers, + ), + W( + "e_s31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w3.weight_scale", "w1.weight_scale"), + transform=_expert_s31, + layers=_moe_layers, + ), + W( + "e_s2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w2.weight_scale"), + transform=_expert_s2, + layers=_moe_layers, + ), + # The per-tensor scalars of every expert: the checkpoint stores the multipliers into the quantized range + # (``*_global_scale``); the modelopt divisors modeling derives the kernels' scalars from are their FP32 + # reciprocals, taken by the same helper as the built-in loader. + *( + W( + f"e_{name}", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_per_expert(suffix), + transform=_divisor, + layers=_moe_layers, + ) + for name, suffix in ( + ("isc1", "w1.input_global_scale"), + ("isc3", "w3.input_global_scale"), + ("isc2", "w2.input_global_scale"), + ("ws2_1", "w1.weight_global_scale"), + ("ws2_3", "w3.weight_global_scale"), + ("ws2_2", "w2.weight_global_scale"), + ) + ), + # --- not per-layer --- + W( + "embed", + shape=lambda d: (d.vocab, d.hidden), + src=_one("model.embed_tokens.weight"), + layers=None, + ), + W( + "final_norm", + shape=lambda d: (d.hidden,), + src=_one("model.norm.weight"), + transform=_gemma_fold, + layers=None, + ), + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + mapping = core.model_config.mapping + sparse = cfg.sparse_attention_config + freq = [int(f) for f in sparse["sparse_attention_freq"]] + moe_freq = [ + int(f) for f in getattr(cfg, "moe_layer_freq", None) or [1] * cfg.num_hidden_layers + ] + tp = mapping.tp_size + head_dim = int(cfg.head_dim) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + index_heads = max(int(sparse["sparse_num_index_heads"]) // tp, 1) + index_dim = int(sparse["sparse_index_dim"]) + return SimpleNamespace( + fused=fused_projection(core.model_config.sparse_attention_config), + num_layers=cfg.num_hidden_layers, + sparse_attn=[i for i, f in enumerate(freq) if f], + dense_attn=[i for i, f in enumerate(freq) if not f], + moe=[i for i, f in enumerate(moe_freq) if f], + dense_mlp=[i for i, f in enumerate(moe_freq) if not f], + hidden=cfg.hidden_size, + head_dim=head_dim, + heads=heads, + kv_heads=kv_heads, + index_heads=index_heads, + index_dim=index_dim, + q_width=heads * head_dim, + qkv_dense_rows=(heads + 2 * kv_heads) * head_dim, + qkv_sparse_rows=(heads + 2 * kv_heads) * head_dim + index_heads * index_dim + index_dim, + dense_inter=int(cfg.dense_intermediate_size) // tp, + shared_inter=int(cfg.shared_intermediate_size) * int(cfg.n_shared_experts) // tp, + moe_inter=int(cfg.intermediate_size) // tp, + num_experts=int(cfg.num_local_experts), + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None)] + return [(f"l{i}_{entry.name}", i) for i in layers] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares (meta under the engine's meta init).""" + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, core) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + """param key -> [(checkpoint key or key tuple, index into the param | None, transform | None)].""" + d = self.dims(core) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, core, weights) -> set: + """The vision tower and its projectors, and the multi-token-prediction head: in the checkpoint, never built + by this target.""" + return {k for k in weights.keys() if k.startswith(_UNBUILT_PREFIXES)} + + def load(self, model, weights) -> None: + """Fill every declared weight and the shell's ``lm_head`` shard; assert coverage both ways.""" + core = model.model + d = self.dims(core) + manifest = self.manifest(core) + assert set(manifest) == set(core.w.keys()), ( + "manifest/parameter drift", + set(manifest) ^ set(core.w.keys()), + ) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(core, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(core.w[param_key], ckpt, index, transform, weights, d, consumed) + if param_key.endswith(self.RELEASE_AFTER): + torch.cuda.empty_cache() + # The shell owns the vocab-parallel head: this rank's rows of the full table. + _fill(model.lm_head.weight, _LM + "lm_head.weight", None, _col_w, weights, d, consumed) + shell = { + n for n, _ in model.named_parameters() if not n.startswith(("model.w.", "draft_model.")) + } + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + "checkpoint coverage: leftover keys are not exactly the unbuilt families (vision, MTP head); " + f"unexpected {sorted(leftover - expected)[:8]}, missed {sorted(expected - leftover)[:8]}" + ) + + +def _materialize(entry) -> torch.Tensor: + """``[:]`` on a lazy safetensors slice; a 0-dim tensor (the per-tensor scalars) rejects that index.""" + return entry if getattr(entry, "ndim", 1) == 0 else entry[:] + + +def _fill(param, ckpt, index, transform, weights, d, consumed) -> None: + keys = ckpt if isinstance(ckpt, tuple) else (ckpt,) + for key in keys: + assert key in weights, f"checkpoint key missing: {key}" + dst = param.data if index is None else param.data[index] + d.dev = dst.device + # Host tensors: a transform slices this rank's part on the host and moves only that to the device. + src_all = tuple(_materialize(weights[key]) for key in keys) + if transform is not None: + src = transform(src_all if isinstance(ckpt, tuple) else src_all[0], d) + else: + assert len(src_all) == 1, "a multi-key row needs a transform" + src = src_all[0] + if src.dim() == 0 and dst.dim() == 0: + pass + elif src.numel() == 1 and dst.numel() == 1: + src = src.reshape(dst.shape) + src = src.to(dst.device, non_blocking=True) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(keys) + _release(weights, keys) + + +def _release(weights, keys) -> None: + """Drop consumed checkpoint tensors from the engine's dict as the load goes (bounded host memory).""" + if hasattr(weights, "mark_consumed_keys"): + weights.mark_consumed_keys(keys) + else: + for key in keys: + weights.pop(key, None) + + +MODEL_WEIGHTS = MiniMaxM3Tp2Weights() + + +# -------------------------------------------------------------------------------------------- the drafter + + +def _draft_one(template: str): + return lambda d, i, t=template: [(t, None)] + + +def _draft_many(*templates: str): + return lambda d, i, ts=templates: [(tuple(ts), None)] + + +def _draft_fc_norms(d): + return range(d.capture_layers) + + +def _draft_qkv(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + return torch.cat([_col(q, d), _col(k, d), _col(v, d)], dim=0).contiguous() + + +class MiniMaxM3Tp2DraftWeights: + """The one-layer BF16 Eagle3 drafter (``speculative_model``), tp4. + + The fc over the three captured target layers and its per-layer fc_norms, the layer's three norms, its QKV + (input ``[embeds | hidden]``, so twice the hidden width; Q by heads, K/V by KV heads) and o_proj, its gated + MLP, the final norm and the full-vocabulary head (vocab-parallel). The checkpoint's ``embed_tokens`` is the + target's table (the drafter reads the target's); it is a predicted non-load. + """ + + WEIGHTS: tuple[W, ...] = ( + W( + "fc", + shape=lambda d: (d.hidden, d.capture_layers * d.hidden), + src=_draft_one("fc.weight"), + layers=None, + ), + W( + "fc_norm", + shape=lambda d: (d.hidden,), + src=lambda d, i: [(f"fc_norm.{i}.weight", None)], + layers=_draft_fc_norms, + ), + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.input_layernorm.weight"), + layers=None, + ), + W( + "hidden_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.hidden_norm.weight"), + layers=None, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.post_attention_layernorm.weight"), + layers=None, + ), + W( + "qkv", + shape=lambda d: (d.qkv_rows, 2 * d.hidden), + src=_draft_many(*(f"layers.0.self_attn.{p}_proj.weight" for p in "qkv")), + transform=_draft_qkv, + layers=None, + ), + W( + "o", + shape=lambda d: (d.hidden, d.q_width), + src=_draft_one("layers.0.self_attn.o_proj.weight"), + transform=_row_w, + layers=None, + ), + W( + "gu", + shape=lambda d: (2 * d.inter, d.hidden), + src=_draft_many("layers.0.mlp.gate_proj.weight", "layers.0.mlp.up_proj.weight"), + transform=_gate_up_w, + layers=None, + ), + W( + "dn", + shape=lambda d: (d.hidden, d.inter), + src=_draft_one("layers.0.mlp.down_proj.weight"), + transform=_row_w, + layers=None, + ), + W("final_norm", shape=lambda d: (d.hidden,), src=_draft_one("norm.weight"), layers=None), + ) + + def dims(self, draft) -> SimpleNamespace: + cfg = draft.draft_config.pretrained_config + mapping = draft.draft_config.mapping + tp = mapping.tp_size + head_dim = int(getattr(cfg, "head_dim", cfg.hidden_size // cfg.num_attention_heads)) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + return SimpleNamespace( + hidden=cfg.hidden_size, + capture_layers=draft.capture_layers, + head_dim=head_dim, + q_width=heads * head_dim, + qkv_rows=(heads + 2 * kv_heads) * head_dim, + inter=cfg.intermediate_size // tp, + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + if entry.layers is None: + return [(entry.name, None)] + return [(f"{entry.name}{i}", i) for i in entry.layers(d)] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, draft) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + d = self.dims(draft) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, draft, weights) -> set: + """The drafter checkpoint's own embedding: the drafter reads the target's table.""" + return {"embed_tokens.weight"} & set(weights.keys()) + + def load(self, draft, weights) -> None: + d = self.dims(draft) + manifest = self.manifest(draft) + assert set(manifest) == set(draft.w.keys()), set(manifest) ^ set(draft.w.keys()) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(draft, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(draft.w[param_key], ckpt, index, transform, weights, d, consumed) + _fill(draft.lm_head.weight, "lm_head.weight", None, _col_w, weights, d, consumed) + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + f"drafter checkpoint coverage: unexpected {sorted(leftover - expected)[:8]}, " + f"missed {sorted(expected - leftover)[:8]}" + ) + + +DRAFT_WEIGHTS = MiniMaxM3Tp2DraftWeights() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/__init__.py new file mode 100644 index 000000000000..36a53a834458 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 NVFP4 / sm_107 / tp4.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/dspark.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/dspark.py new file mode 100644 index 000000000000..760fe06cc6b0 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/dspark.py @@ -0,0 +1,823 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-model DSpark on this target (phase W0): the in-tree DSpark drafter and worker, unchanged but for S9b-VR-a +(the drafter's row-parallel all-reduces take the one-shot all-reduce at decode rows; ``DRAFT_AR_ONESHOT_MAX_ROWS``). + +The target keeps its own forward; the shell hands the step to ``M3DSparkWorker``, which is the in-tree +``DSparkWorker`` plus the five calls the shell makes on any worker (``capture_hidden_states``, +``arm_capture_norm``, ``tail_eligible``, ``prebuild``, ``__call__``). In W0 none of them takes a fused path: the +captures are the in-tree DFlash captures, the target logits go through the logits processor and the in-tree +strict accept verifies them, so the speculation is the in-tree one by construction. + +The drafter is the in-tree standalone DSpark drafter for the checkpoint's architecture, built through the draft +model registry. Its context K/V lives in the separate draft KV cache, which must keep the drafter's projection +dtype (BF16): a pool of another dtype is not bound and the worker falls back to a private arena sized by the +served sequence length. So the drafter never takes the target's KV quantization: neither the FP8 draft-KV +override (an Eagle3 need) nor the target's own KV algo (NVFP4 under ``kv_cache_config.dtype="nvfp4"``). +""" + +from __future__ import annotations + +import functools +import os +import time +from typing import Set, Tuple + +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.comm import ( + allgather_cast as _allgather_cast, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + strict_accept as _strict_accept, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_block_tables import ( + enable_parallel_refresh, + refresh_block_tables, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_advance import ( + advance_ctx_and_read_view, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_store import ( + fused_ctx_store, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_drafter_inputs import ( + drafter_inputs as _drafter_inputs, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_forward_indices import ( + forward_indices, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_qk_norm_rope import ( + qk_norm_rope, + qk_norm_rope_append, +) +from tensorrt_llm._torch.distributed import AllReduce, AllReduceStrategy, allgather +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + external_drafter_config_kwargs, + get_draft_model, +) +from tensorrt_llm._torch.modules.linear import Linear, TensorParallelMode +from tensorrt_llm._torch.speculative.dspark import DSparkWorker +from tensorrt_llm._torch.speculative.interface import should_use_separate_draft_kv_cache +from tensorrt_llm.logger import logger + +from .dspark_s4 import MAX_FAST_BATCH, S4WorkerMixin +from .m31_switches import m31_disabled, m31_enabled +from .mxdraft import route_mx_linears +from .ops.embedding.m3_embedding import m3_embedding +from .ops.gemm.m31_fc_shard import FcShard + +# S9b-VR-a: the drafter's row-parallel all-reduces (each layer's o_proj and MLP down projection) of up to this many +# rows run the one-shot all-reduce instead of AUTO. On this target AUTO resolves to NCCL at every row count: the AR +# tuner's profiling of the custom tactics fails here ("Total failed profiling tactics: 14"), so the +# decode-step drafter ARs ran as the NCCL ring. VR TP4 [rows, 6144] BF16, graph of 10 calls: 4 rows 16.46 -> +# 5.64 us per call, 32 rows 16.66 -> 6.46; error vs the FP32 sum 2.0e-3 against NCCL's 3.9e-3 (a different +# reduction order: not bitwise against NCCL). Wider calls (the drafter's context passes) keep AUTO. Never the two-shot +# all-reduce: 430 us at 4 rows here. TRTLLM_M31_DISABLE_S9BVR=1 restores AUTO at every row count. +DRAFT_AR_ONESHOT_MAX_ROWS = ( + 0 if os.environ.get("TRTLLM_M31_DISABLE_S9BVR", "0").strip() not in ("", "0") else 32 +) # capacity: fixed: the K3 c8 verify rows + + +# DRAFT_APPEND_FUSE: the drafter's per-layer flashinfer +# append_paged_kv_cache folded into the EX-11 q / k norm + RoPE launch (the same K bits, V copied; same pages / slots). +# Exact. Default ON; kill switch TRTLLM_M31_DISABLE_DRAFT_APPEND_FUSE=1, read at import. Taken only where the drafter +# appends through flashinfer's paged HND cache (append_supported); otherwise the per-layer append runs as before. +DRAFT_APPEND_FUSE = not m31_disabled("DRAFT_APPEND_FUSE") +# FC_SHARD: the drafter's fc (replicated on every rank) as this rank's output-column shard + an all-gather in +# project_target_hidden, bit for bit the full fc (the K + 1 verify rows at batch 1). Kill switch +# TRTLLM_M31_DISABLE_FC_SHARD, read at import. +FC_SHARD = not m31_disabled("FC_SHARD") +_APPEND_FUSE_TAKEN = [0] + + +def _qk_norm_rope_append_logged(*args): + """catalog qk_norm_rope_append, counting taken calls and logging the first (host calls: eager or captured).""" + out = qk_norm_rope_append(*args) + if out is not None: + _APPEND_FUSE_TAKEN[0] += 1 + if _APPEND_FUSE_TAKEN[0] == 1: + logger.info( + "[M3 dispatch] DRAFT_APPEND_FUSE taken: drafter K / V append in the EX-11 launch" + ) + return out + + +class DraftOneshotAllReduce(nn.Module): + """A row-parallel linear's all-reduce: the one-shot all-reduce up to ``max_rows`` rows, its original (AUTO) + all-reduce above. The output buffer choice stays the original module's, so wider calls are unchanged.""" + + def __init__(self, auto: AllReduce, dtype: torch.dtype, max_rows: int, name: str): + super().__init__() + self.auto = auto + self.oneshot = AllReduce( + mapping=auto.mapping, strategy=AllReduceStrategy.ONESHOT, dtype=dtype + ) + self.max_rows = max_rows + self.name = name + self._logged: Set[int] = set() + + def uses_nccl_symmetric_memory_window(self) -> bool: + return self.auto.uses_nccl_symmetric_memory_window() + + def forward(self, input: torch.Tensor, *args, **kwargs): + rows = input.shape[0] if input.dim() > 1 else 1 + if rows > self.max_rows: + return self.auto(input, *args, **kwargs) + if rows not in self._logged: + self._logged.add(rows) + logger.info( + f"[M3 dispatch] rank {self.auto.mapping.tp_rank}: S9b-VR-a drafter all-reduce ONESHOT taken at " + f"{rows} rows ({self.name})" + ) + return self.oneshot(input, *args, **kwargs) + + +def install_draft_oneshot_allreduce(draft: nn.Module, max_rows: int) -> int: + """Wrap every row-parallel linear's AUTO all-reduce in the drafter (S9b-VR-a); returns how many were wrapped.""" + if max_rows <= 0: + return 0 + n = 0 + for name, mod in draft.named_modules(): + ar = getattr(mod, "all_reduce", None) + if ( + isinstance(mod, Linear) + and mod.tp_mode == TensorParallelMode.ROW + and isinstance(ar, AllReduce) + and ar.strategy == AllReduceStrategy.AUTO + and ar.mapping.tp_size > 1 + ): + mod.all_reduce = DraftOneshotAllReduce(ar, mod.dtype, max_rows, name) + n += 1 + return n + + +# BUNDLE_OFF_NOTE: S9b, S5 and S2 (prebuild below) are OFF by default with the K3 bundle +# {S9b, S5, S2, FUSED_CAPTURE}: its powered K7 natural-acceptance gate (n 1,776) failed (mean d +# -0.032 AL, CI [-0.059, -0.005], median -0.022, sign p 0.037 against) (this applies to the sm_107 target +# too), pending the per-member bisect. Opt in with TRTLLM_M31_ENABLE_=1; TRTLLM_M31_DISABLE_=1 still wins. +# FUSED_CAPTURE does not exist in this target. + +# S9b (fused drafter all-reduce + norm) by default at this TP; TRTLLM_M31_DISABLE_S9B still turns it off. +# measured: TP4 +0.5% / step while its drafter all-reduce runs the untuned AUTO path +# re-enable after that tuning fix. +S9B_DEFAULT_ON = False + + +# S5: the largest decode batch the window-capped max_seq_len applies to, by draft length; 0 = off. Per (TP, K) on +# VR: the largest batch at which the window-capped drafter attention is faster than the stock call at every batch up +# to it, in two replicates of the per-call microbench (L = 400K). TP4: K3 c1 / c2 / c4 +# win -2.7 / -2.6 / -2.0 us per call, c8 loses +2.0; K7 c1 / c2 / c4 win. +S5_MAX_BATCH_BY_K = {3: 4, 7: 4} + +# S10 (lean drafter glue) is on only up to this draft length: bitwise at K3 on VR, not at K7 (it +# diverges from request 5 while K7 OFF / OFF is identical); K4..6 are unmeasured. +# Gate-only: TRTLLM_M31_ENABLE_S10_K7=1 lifts the bound to every draft length, for re-gating S10 at K7 under AR-DET +# (the K7 divergence predates it). Read once, at import. +S10_MAX_DRAFT_LEN = 63 if m31_enabled("S10_K7") else 3 + + +# MXDRAFT is default OFF on TP4: the TP4 K7 c1 step was slower with it on in two c1 kill A/Bs (4,009.1 -> +# 4,040.7 us; 4,012.3 -> 4,036.4 us) while TP8 gains (-1.3%). +# TRTLLM_M31_ENABLE_MXDRAFT=1 opts in; TRTLLM_M31_DISABLE_MXDRAFT=1 still wins. Read once, at import. +MXDRAFT = m31_enabled("MXDRAFT") and not m31_disabled("MXDRAFT") + + +def s10_applies(num_draft_tokens: int) -> bool: + return num_draft_tokens <= S10_MAX_DRAFT_LEN + + +def s5_max_batch_for(num_draft_tokens: int) -> int: + """K <= 3 takes the K3 entry, K >= 7 the K7 entry, and K in between the smaller of the two.""" + if num_draft_tokens <= 3: + return S5_MAX_BATCH_BY_K[3] + if num_draft_tokens >= 7: + return S5_MAX_BATCH_BY_K[7] + return min(S5_MAX_BATCH_BY_K.values()) + + +def is_standalone_dspark(spec_config) -> bool: + """Whether the run speculates with a standalone DSpark drafter (its own checkpoint, ``decoding_type: DSpark``).""" + return bool( + spec_config is not None + and spec_config.spec_dec_mode.use_one_engine() + and spec_config.spec_dec_mode.is_dspark() + and not getattr(spec_config, "draft_is_embedded_in_target", False) + ) + + +class M3EmbedTokens(nn.Module): + """The target's token embedding as the module the in-tree drafter borrows (``load_weights_from_target_model``). + + The drafter embeds the bonus and mask tokens through ``embed_tokens(ids)``; the target keeps its replicated + table as a raw tensor and gathers it with ``m3_embedding``, so this module calls the same gather on the same + table (an exact gather, so the drafter sees the rows the target embeds). + """ + + def __init__(self, core: nn.Module) -> None: + super().__init__() + self._core = [core] # not a submodule: the table belongs to the target + + @property + def weight(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def forward(self, ids: torch.Tensor) -> torch.Tensor: + return m3_embedding(ids, self.weight) + + +def s10c_c_drafter_inputs( + embed_tokens, + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + mask_token_id, + trained_mask_embedding, + block_size, + tokens_per_request, +): + """S10c-C: the drafter's step inputs in one launch (catalog/spec_decode/dspark_drafter_inputs), for this target's + embedding only: ``M3EmbedTokens`` is an exact gather of the replicated table, so the noise block is table rows (the + trained mask row if the drafter keeps one). None for any other embedding: the stock path runs.""" + if not isinstance(embed_tokens, M3EmbedTokens): + return None + table = embed_tokens.weight + mask_row = ( + trained_mask_embedding if trained_mask_embedding is not None else table[int(mask_token_id)] + ) + return _drafter_inputs( + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + table, + mask_row, + block_size, + tokens_per_request, + ) + + +# N3 A2 (switch TRTLLM_M31_DISABLE_N3A): the target's strict acceptance and next-step input tokens in one launch each +# (catalog/spec_decode/strict_accept), bitwise to the in-tree torch path. Read once, at import. +N3A_DISABLED = m31_disabled("N3A") +# N3A-KV (switch TRTLLM_M31_DISABLE_N3A_KV, and off with N3A): DFlash's draft-forward KV bookkeeping (the rewind amount +# 1 - num_accepted and kv_lens += 1) in one launch where the in-tree lines run, after the forced-acceptance override, +# bitwise (integer arithmetic). Read once, at import. +N3A_KV_DISABLED = N3A_DISABLED or m31_disabled("N3A_KV") +# Gate-only cross-check (TRTLLM_M31_CHECK_N3A_KV=1; never in a perf arm): every N3A-KV call is compared with the stock +# lines on copies (graph-safe device counters; reported on eager steps). +N3A_KV_CHECK = os.environ.get("TRTLLM_M31_CHECK_N3A_KV", "").strip() not in ("", "0") +# N3 P-gid + P-cat (switch TRTLLM_M31_DISABLE_N3P): the verify-row target logits without the identity gather_ids copy +# and with the vocab concat and the fp32 cast in one strided copy, bitwise to the logits processor. Read at import. +N3P_DISABLED = m31_disabled("N3P") +# N3 A1 (switch TRTLLM_M31_DISABLE_A1): the verify logits' vocab all-gather, concat and fp32 cast in one launch over a +# multicast NVLink workspace (catalog/comm/allgather_cast), bitwise to the NCCL all-gather + strided copy below. The +# verify rows (B x (K + 1), up to 64 rows: K3 batches 1..16, K7 batches 1..8). Read once, at import. Default on +# on VR since the VR op matrix + grid sweep certified it bitwise; the switch turns it off. +A1_DISABLED = m31_disabled("A1") +A1_MAX_ROWS = 64 # capacity: fixed: allgather_cast.MAX_ROWS (K3 at batch 16, K7 at batch 8) +# A1 at draft lengths other than 3 (K7 included; switch TRTLLM_M31_DISABLE_A1_K7, K3 keeps A1). +A1_K7_DISABLED = m31_disabled("A1_K7") +_n3a_logged = False + + +def _n3a_log_once() -> None: + global _n3a_logged + if not _n3a_logged: + _n3a_logged = True + logger.info( + "[M3 dispatch] n3_strict_accept taken: fused strict acceptance + next-step input tokens" + ) + + +_n3p_logged = False +_n3a_kv_logged = False + + +def _n3a_kv_log_once() -> None: + global _n3a_kv_logged + if not _n3a_kv_logged: + _n3a_kv_logged = True + logger.info( + "[M3 dispatch] n3a_kv_rewind taken: draft-forward KV rewind + kv_lens bump in one launch" + ) + + +_a1_logged = set() + + +def _a1_log_once(key, message: str) -> None: + if key not in _a1_logged: + _a1_logged.add(key) + logger.info(message) + + +def _n3p_log_once() -> None: + global _n3p_logged + if not _n3p_logged: + _n3p_logged = True + logger.info( + "[M3 dispatch] n3p_target_logits taken: verify-row logits without the gather_ids copy and the concat" + ) + + +class M3DSparkWorker(S4WorkerMixin, DSparkWorker): + """The in-tree ``DSparkWorker`` with the calls this target's shell makes on its worker. W1: the one-launch Markov + chain (``S4WorkerMixin``, switch ``TRTLLM_M31_DRAFT_DISABLE_S4``). N3 A2: the strict acceptance and the next-step + input tokens in one launch each (switch ``TRTLLM_M31_DISABLE_N3A``). N3 P / A1: the verify logits (switches + ``TRTLLM_M31_DISABLE_N3P`` / ``TRTLLM_M31_DISABLE_A1``). Everything else is the in-tree path.""" + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None + ) -> None: + """The in-tree DFlash capture: layer output (hidden + residual) into the static capture buffer.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model + ) -> None: + """No capture norm: DSpark's fc reads the raw captured layers.""" + return None + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Never: the target logits always go through the logits processor and the in-tree strict accept.""" + return False + + def prebuild(self, lm_head: nn.Module, draft_model) -> None: + """The one-launch Markov chain's shards, buffers, exchange workspace and compiles (outside capture, every + rank); no fused verify, no coded target head, no fused draft loop.""" + rank = getattr(getattr(self, "mapping", None), "tp_rank", None) + # S10: the drafter's strided-view glue (bitwise; kill switch TRTLLM_M31_DISABLE_S10, read at import). + k = getattr(getattr(self, "spec_config", None), "max_draft_len", 0) or 0 + lean = not m31_disabled("S10") and s10_applies(k) + self.enable_lean_glue(draft_model, lean) + # S5: window-capped max_seq_len for the drafter's sliding-window decode (not bitwise: the TRTLLM-Gen tile / KV + # split changes, not the keys read). OFF by default (K3 bundle, see BUNDLE_OFF_NOTE): opt in with + # TRTLLM_M31_ENABLE_S5=1; TRTLLM_M31_DISABLE_S5 still wins. + s5_max_batch = s5_max_batch_for(k) if m31_enabled("S5") and not m31_disabled("S5") else 0 + self.enable_window_capped_decode(draft_model, s5_max_batch) + # The context arena spans every seq slot (CUDA graph metadata narrows max_num_requests to its bucket). + self.enable_full_slot_arena() + # S2: the drafter's replicated fc split row-parallel with one fp32 all-reduce (not bitwise: the partial-sum + # order changes). Collective. OFF by default (K3 bundle): opt in with TRTLLM_M31_ENABLE_S2=1; + # TRTLLM_M31_DISABLE_S2 still wins. + s2 = self.enable_fc_tp_split(draft_model, m31_enabled("S2") and not m31_disabled("S2")) + split = getattr(draft_model, "_dflash_fc_split", None) + ar = split[3] if split is not None else None + ar_path = ( + "MNNVL" + if getattr(ar, "mnnvl_allreduce", None) is not None + else getattr(ar, "strategy", None) + ) + # FC_SHARD (bitwise; collective: every rank builds or every rank declines). S2's split, when on, wins. No + # drafter (a shell worker): nothing to set. + if draft_model is not None: + draft_model._dflash_fc_shard = ( + FcShard.build(draft_model.fc, self.mapping) + if FC_SHARD and not s2 and getattr(draft_model, "fc", None) is not None + else None + ) + logger.info(f"M3.1 DSpark S2: fc TP split {'on' if s2 else 'off'} on rank {rank} (all-reduce " + f"{ar_path})") # fmt: skip + logger.info(f"M3.1 DSpark S5: window-capped draft decode for batches <= {s5_max_batch} on rank " + f"{rank}") # fmt: skip + logger.info(f"M3.1 DSpark S10: lean drafter glue {'on' if lean else 'off'} on rank {rank}") + # S9b: each drafter layer's all-reduce fused with the add + RMSNorm after it (not bitwise: the sum is not + # rounded before the add). On by default where S9B_DEFAULT_ON; kill switch TRTLLM_M31_DISABLE_S9B, read at + # import. + s9b = self.enable_fused_ar_norm( + draft_model, (S9B_DEFAULT_ON or m31_enabled("S9B")) and not m31_disabled("S9B") + ) + logger.info( + f"M3.1 DSpark S9b: fused drafter all-reduce + norm {'on' if s9b else 'off'} on rank {rank}" + ) + # S10c-A: the drafter's context K / V epilogue + accept mask + paged store in one launch after the K / V GEMM + # (bitwise). Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_A, read at import. + s10c_a = self.enable_fused_ctx_store( + None if m31_disabled("S10C") or m31_disabled("S10C_A") else fused_ctx_store + ) + logger.info( + f"M3.1 DSpark S10c-A: fused drafter ctx store {'on' if s10c_a else 'off'} on rank {rank}" + ) + # S10c-C: the drafter's step inputs (bonus, positions, noise block) in one launch (bitwise: integers and row + # copies). Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_C, read at import. + s10c_c = self.enable_fused_drafter_inputs( + None if m31_disabled("S10C") or m31_disabled("S10C_C") else s10c_c_drafter_inputs + ) + logger.info( + f"M3.1 DSpark S10c-C: fused drafter step inputs {'on' if s10c_c else 'off'} on rank {rank}" + ) + # S10c-D: the drafter's context advance + read view in one launch (bitwise, integer), with ADV_PAR's parallel + # read-view copy. Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_D / + # TRTLLM_M31_DISABLE_ADV_PAR (the one-launch copy), read at import. + s10c_d = self.enable_fused_ctx_advance( + None + if m31_disabled("S10C") or m31_disabled("S10C_D") + else functools.partial(advance_ctx_and_read_view, parallel=not m31_disabled("ADV_PAR")) + ) + logger.info( + f"M3.1 DSpark S10c-D: fused drafter context advance {'on' if s10c_d else 'off'} on rank {rank}" + ) + # S10c-B: the drafter's block-table refresh (decode + placeholder count) in one launch (bitwise, integer). + # Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_B, read at import. + s10c_b = self.enable_fused_block_refresh( + None if m31_disabled("S10C") or m31_disabled("S10C_B") else refresh_block_tables + ) + # S10c-B2: that refresh on a (row, column-chunk) grid (bitwise; flat at short tables, removes the serial + # kernel's growth past ~11K blocks per sequence). Kill switch TRTLLM_M31_DISABLE_S10C_B2, read at import. + s10c_b2 = enable_parallel_refresh(s10c_b and not m31_disabled("S10C_B2")) + logger.info( + f"M3.1 DSpark S10c-B: fused drafter block-table refresh {'on' if s10c_b else 'off'} on rank {rank}" + f" (S10c-B2 2-D grid {'on' if s10c_b2 else 'off'})" + ) + # S10c-E: the draft model's paged block-decode index prelude in one launch (bitwise, integer). Kill switches + # TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_E, read at import. + s10c_e = self.enable_fused_forward_indices( + draft_model, None if m31_disabled("S10C") or m31_disabled("S10C_E") else forward_indices + ) + logger.info( + f"M3.1 DSpark S10c-E: fused drafter forward indices {'on' if s10c_e else 'off'} on rank {rank}" + ) + # EX-11: the drafter block's q / k RMSNorm + partial RoPE in one launch (bitwise to flashinfer's three). Kill + # switch TRTLLM_M31_DISABLE_EX11, read at import. + ex11 = self.enable_fused_qk_norm_rope( + draft_model, None if m31_disabled("EX11") else qk_norm_rope + ) + logger.info( + f"M3.1 DSpark EX-11: fused drafter q/k norm + RoPE {'on' if ex11 else 'off'} on rank {rank}" + ) + # DRAFT_APPEND_FUSE: on top of EX-11, the same launch also writes the block's K / V into the layer's paged + # cache (bitwise flashinfer's append), and the per-layer append launch is skipped. Kill switch + # TRTLLM_M31_DISABLE_DRAFT_APPEND_FUSE, read at import. + append_fuse = self.enable_fused_qk_norm_rope_append( + draft_model, _qk_norm_rope_append_logged if ex11 and DRAFT_APPEND_FUSE else None + ) + logger.info( + f"M3.1 DSpark DRAFT_APPEND_FUSE: drafter K / V append in the EX-11 launch {'on' if append_fuse else 'off'} " + f"on rank {rank}" + ) + # MXDRAFT: the drafter's qkv / o / gate_up / down GEMMs on this target's MXFP8 GEMV for the decode passes' + # rows (B x K, B <= MAX_FAST_BATCH, up to the GEMV's own row bound), from the exact MXFP8 form of the shipped + # weights (drafter-only precision, gated on natural AL). Default off on TP4 (``MXDRAFT``). + mx = ( + 0 + if not MXDRAFT or k <= 0 + else route_mx_linears(draft_model, [b * k for b in range(1, MAX_FAST_BATCH + 1)]) + ) + logger.info(f"M3.1 DSpark MXDRAFT: MXFP8 drafter GEMV on {mx} projections on rank {rank}") + self.prebuild_s4(lm_head, draft_model) + # N3 A2 compiles (rank-local), then the N3 A1 workspace and compiles (collective), outside capture. + if not N3A_DISABLED and lm_head is not None: + _strict_accept.precompile((self.max_draft_len,), lm_head.weight.device) + self._a1 = self._prebuild_a1(lm_head) if lm_head is not None else None + n3a_kv = False + if not N3A_KV_DISABLED and lm_head is not None: + _strict_accept.precompile_kv_rewind(lm_head.weight.device) + self._n3a_kv_buf = torch.empty( + _strict_accept.max_requests(), dtype=torch.int32, device=lm_head.weight.device + ) + n3a_kv = self.enable_fused_kv_rewind(self._n3a_kv_rewind) + logger.info( + f"M3.1 N3A-KV: one-launch draft-forward KV bookkeeping {'on' if n3a_kv else 'off'} on rank " + f"{getattr(getattr(self, 'mapping', None), 'tp_rank', None)}" + ) + + _n3a_kv_buf = None # the persistent rewind amounts of N3A-KV (int32, MAX_REQUESTS) + _n3a_kv_counts = None # gate-only cross-check counters (calls, mismatches) + + def _n3a_kv_rewind(self, num_accepted, kv_lens, nc: int, bs: int): + """N3A-KV for the DFlash hook: ``1 - num_accepted[nc:bs]`` (a view of the persistent buffer) with + ``kv_lens[nc:bs] += 1`` in one launch, or None (the stock lines) for operands or a variant it does not take.""" + n = bs - nc + buf = self._n3a_kv_buf + if buf is None or not _strict_accept.supports_kv_rewind(num_accepted, kv_lens, buf, nc, n): + return None + if not _strict_accept.is_compiled_kv_rewind(num_accepted, kv_lens): + return None + _n3a_kv_log_once() + if not N3A_KV_CHECK: + return _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + want_rewind = 1 - num_accepted[nc:bs] + want_kv = kv_lens[nc:bs] + 1 + out = _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + if self._n3a_kv_counts is None: + self._n3a_kv_counts = torch.zeros(2, dtype=torch.int64, device=kv_lens.device) + bad = (out != want_rewind).any() | (kv_lens[nc:bs] != want_kv).any() + self._n3a_kv_counts[0] += 1 + self._n3a_kv_counts[1] += bad.to(torch.int64) + if not torch.cuda.is_current_stream_capturing(): + calls, mism = self._n3a_kv_counts.tolist() + logger.info(f"M3.1 N3A-KV check: {calls} calls, {mism} mismatches") + return out + + _a1 = None # the N3 A1 workspace (allgather_cast's arguments) once set up + + def _a1_all_ok(self, local: bool) -> bool: + """Whether ``local`` holds on every rank of the TP group: A1 is set up on every rank or on none (a rank that + is not eligible, or whose compile failed, turns it off everywhere).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + return _mnnvl_workspace_all_succeeded(_get_mnnvl_workspace_comm(self.mapping), local) + + def _prebuild_a1(self, lm_head: nn.Module): + """The N3 A1 workspace and its K3 verify-row compiles, or None (every rank alike): off by its switch, at + another draft length, without a column-parallel bf16 head over a TP group, or when a compile fails on any + rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + mapping = self.mapping + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return None + cols = int(getattr(lm_head, "out_features", 0)) + sms = torch.cuda.get_device_properties(lm_head.weight.device).multi_processor_count + # Why this rank would not set A1 up (empty: it would). The grid is a function of rows and the SM count: + # requiring every SM up to the workspace's MAX_GRID makes it a function of rows alone, the same on every rank. + off = [ + why + for why, bad in ( + ( + "TRTLLM_M31_DISABLE_A1 set", + A1_DISABLED, + ), + ( + f"draft length {int(self.max_draft_len)} with TRTLLM_M31_DISABLE_A1_K7 set", + int(self.max_draft_len) != 3 and A1_K7_DISABLED, + ), + ( + f"draft length {int(self.max_draft_len)}: more than {A1_MAX_ROWS} verify rows at batch 1", + int(self.max_draft_len) + 1 > A1_MAX_ROWS, + ), + ( + "not a column-parallel, gathered, unpadded, bias-free head", + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or getattr(lm_head, "bias", None) is not None, + ), + (f"head dtype {lm_head.weight.dtype}", lm_head.weight.dtype != torch.bfloat16), + ( + f"{cols} shard columns (whole 16-byte vectors needed)", + cols <= 0 or cols % 8 != 0, + ), + ( + f"{sms} SMs (fewer than {_allgather_cast.MAX_GRID})", + sms < _allgather_cast.MAX_GRID, + ), + ) + if bad + ] + if not self._a1_all_ok(not off): + logger.info( + f"M3.1 N3 A1: off on rank {mapping.tp_rank} ({'; '.join(off) or 'off on another rank'}); " + "the verify logits use the NCCL all-gather" + ) + return None + k1 = int(self.max_draft_len) + 1 + rows = tuple(b * k1 for b in range(1, A1_MAX_ROWS // k1 + 1)) + start = time.perf_counter() + if not _allgather_cast.precompile( + lm_head.mapping, cols, rows, lm_head.weight.device, agree=self._a1_all_ok + ): + logger.warning( + "M3.1 N3 A1: a compile failed on some rank; every rank keeps the NCCL all-gather" + ) + return None + ws = _allgather_cast.workspace_args(lm_head.mapping, cols) + # The engagement line: one per rank, at server start. + logger.info( + f"M3.1 N3 A1: one-launch verify all-gather set up on rank {mapping.tp_rank}/{mapping.tp_size} " + f"(shard {cols} columns, rows {rows}, workspace {ws['ws_uc'].numel() * 4} bytes/rank, " + f"set-up {time.perf_counter() - start:.2f} s)" + ) + return ws + + def _n3a_eligible( + self, logits, draft_tokens, num_contexts: int, batch_size: int, spec_metadata + ) -> bool: + """Whether this step's strict acceptance takes the fused op: a decode-only, non-ragged step whose logits hold + exactly batch_size x (D + 1) verify rows and whose drafts the op takes. Host state and shapes only, so it is + fixed per captured graph and the same on every rank. Decided before the target tokens are sampled, so the + sampler runs exactly once either way.""" + if N3A_DISABLED or spec_metadata is None or spec_metadata.is_ragged_verify: + return False + if num_contexts != 0 or batch_size <= 0 or draft_tokens.dim() != 2: + return False + n_req, draft_len = draft_tokens.shape + if n_req != batch_size or draft_len <= 0 or n_req > _strict_accept.max_requests(): + return False + rows = logits.shape[0] if logits.dim() > 1 else 1 + return rows == batch_size * (draft_len + 1) and draft_tokens.is_contiguous() + + def _sample_and_accept_draft_tokens_base( + self, logits, draft_tokens, num_contexts, batch_size, spec_metadata + ): + """The in-tree strict acceptance with its epilogue in one launch (N3 A2): the same target tokens (the in-tree + sampler, called once), the same accepted tokens and counts, then the in-tree forced-acceptance override (its + RNG pool and counters advance as on an in-tree step).""" + if not self._n3a_eligible(logits, draft_tokens, num_contexts, batch_size, spec_metadata): + return super()._sample_and_accept_draft_tokens_base( + logits, draft_tokens, num_contexts, batch_size, spec_metadata + ) + if logits.dim() == 1: + logits = logits.unsqueeze(0) + target_tokens = self._sample_tokens_for_batch( + logits, spec_metadata, num_contexts, batch_size + ) + if _strict_accept.supports_strict_accept( + target_tokens, draft_tokens + ) and _strict_accept.is_compiled_strict_accept(target_tokens, draft_tokens): + accepted, num_accepted = _strict_accept.spec_strict_accept(target_tokens, draft_tokens) + _n3a_log_once() + else: + # The sampler already ran: finish with the in-tree epilogue (the op's reference is that code). + accepted, num_accepted = _strict_accept.spec_strict_accept.reference( + target_tokens, draft_tokens + ) + num_accepted = self._apply_force_accepted_tokens( + num_accepted, num_contexts, draft_tokens.shape[-1], spec_metadata=spec_metadata + ) + return accepted, num_accepted + + def target_logits(self, hidden_states, attn_metadata, spec_metadata, lm_head: nn.Module): + """The verify rows' gathered fp32 target logits (N3 P-gid + P-cat), or None for the logits processor. + + On a decode-only, non-ragged step every row is a verify row and ``gather_ids`` is ``arange(num_tokens)`` (each + generation request extends it with the next ``1 + D`` row indices), so the rows are a slice, not a gather. + The processor's head is the column-parallel BF16 GEMM, an all-gather along the vocab (NCCL, then a concat) + and ``.float()``; here the same all-gather runs along dim 0 (no concat) and one strided copy writes the + rank-major shards into the fp32 output: the same bytes, an exact widening. Shapes and host state only, so + the decision is fixed per captured graph and the same on every rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + if ( + N3P_DISABLED + or spec_metadata is None + or getattr(spec_metadata, "is_ragged_verify", False) + ): + return None + if attn_metadata.num_contexts != 0 or spec_metadata.gather_ids is None: + return None + rows = hidden_states.shape[0] + if spec_metadata.gather_ids.shape[0] != rows or rows == 0: + return None + if ( + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or getattr(lm_head, "bias", None) is not None + ): + return None + shard = lm_head.apply_linear(hidden_states, None) + _n3p_log_once() + tp = lm_head.tp_size + if tp == 1: + return shard.float() + # N3 A1: one launch for the all-gather, the concat and the cast. The decision depends on the rows and the + # set-up only (the same on every rank, fixed per captured graph). + if self._a1 is not None: + if rows <= A1_MAX_ROWS and _allgather_cast.is_compiled( + rows, shard.shape[-1], tp, 0, shard.device + ): + _a1_log_once( + "taken", + "[M3 dispatch] n3_a1 allgather_cast taken: verify logits gathered in one launch", + ) + return _allgather_cast.allgather_cast(shard, **self._a1) + _a1_log_once( + rows, + f"[M3 dispatch] n3_a1 allgather_cast not taken: {rows} verify rows (compiled: verify rows up to " + f"{A1_MAX_ROWS}); NCCL all-gather", + ) + gathered = allgather(shard, lm_head.mapping, dim=0) + vocab_shard = shard.shape[-1] + out = torch.empty((rows, tp * vocab_shard), dtype=torch.float32, device=shard.device) + out.view(rows, tp, vocab_shard).copy_(gathered.view(tp, rows, vocab_shard).transpose(0, 1)) + return out + + def _prepare_next_new_tokens( + self, + accepted_tokens, + next_draft_tokens, + batch_indices_cuda, + batch_size, + num_accepted_tokens, + ): + """The next step's input tokens in one launch (N3 A2), else the in-tree index + concat.""" + batch_idx = batch_indices_cuda[:batch_size] + if ( + not N3A_DISABLED + and next_draft_tokens.dim() == 2 + and next_draft_tokens.shape[0] == batch_size + and _strict_accept.supports_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + and _strict_accept.is_compiled_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + ): + return _strict_accept.spec_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + return super()._prepare_next_new_tokens( + accepted_tokens, next_draft_tokens, batch_indices_cuda, batch_size, num_accepted_tokens + ) + + +def build_dspark( + shell, model_config: ModelConfig +) -> Tuple[ModelConfig, nn.Module, M3DSparkWorker, bool]: + """The drafter config, the drafter and the worker of one-model standalone DSpark (mirrors + ``SpecDecOneEngineForCausalLM`` for an external drafter).""" + spec = model_config.spec_config + if not is_standalone_dspark(spec): + raise ValueError("build_dspark serves the standalone DSpark drafter only") + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, **external_drafter_config_kwargs(model_config, spec) + ) + # The drop is safe only because draft_config owns its QuantConfig: from_pretrained reads the drafter checkpoint + # and takes no quant_config from the target. Never pass the target's quant_config in here. + keep_drafter_kv_dtype(draft_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = get_draft_model(model_config, draft_config, shell.lm_head, shell.model) + # A parallel drafter (DSpark) projects its block rows through the target's logits processor, as the one-engine + # construction sets it. + draft.logits_processor = shell.logits_processor + wrapped = install_draft_oneshot_allreduce(draft, DRAFT_AR_ONESHOT_MAX_ROWS) + logger.info( + f"[M3 prebuild] S9b-VR-a: {wrapped} drafter row-parallel all-reduce(s) take ONESHOT up to " + f"{DRAFT_AR_ONESHOT_MAX_ROWS} rows" + + ("" if DRAFT_AR_ONESHOT_MAX_ROWS else " (off: TRTLLM_M31_DISABLE_S9BVR)") + ) + worker = M3DSparkWorker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + logger.info( + f"M3.1 target DSpark: {type(worker).__name__} over {type(draft).__name__}, K={spec.max_draft_len}, " + f"separate draft KV cache={use_separate}" + ) + return draft_config, draft, worker, use_separate + + +def load_dspark_draft_weights(shell, weights, weight_mapper=None) -> None: + """The drafter's own checkpoint, then the target's embedding and head (mirrors + ``SpecDecOneEngineForCausalLM.load_draft_weights`` for DSpark). + + The drafter borrows the target's embedding and head, and ``M3EmbedTokens.weight`` IS the target's table, so a + drafter checkpoint that ships either would load into a module the target owns or be shadowed by it: refused. + """ + borrowed = sorted( + k for k in weights.keys() if "embed_tokens." in k or k.split(".")[-2:-1] == ["lm_head"] + ) + if borrowed: + raise ValueError( + f"the DSpark drafter checkpoint ships tensors this target lends it (embedding / head): {borrowed[:4]}" + ) + shell.draft_model.load_weights(weights=weights, weight_mapper=weight_mapper) + shell.draft_model.load_weights_from_target_model(shell) + if shell.draft_model.lm_head is not shell.lm_head or ( + shell.draft_model.draft_model_full.model.embed_tokens is not shell.model.embed_tokens + ): + raise RuntimeError("the DSpark drafter does not borrow this target's embedding and head") + own = sorted( + n for n, _ in shell.named_parameters() if not n.startswith(("model.w.", "draft_model.")) + ) + logger.info(f"M3.1 target DSpark: shell parameters outside the core and the drafter: {own}") + + +def keep_drafter_kv_dtype(draft_config: ModelConfig) -> None: + """Drop any KV quantization from the drafter's config, so its context pool is allocated in its projection dtype. + + The one-engine construction copies the target's KV algo onto the drafter; on this target that is NVFP4, which + allocates a packed int8 pool the worker refuses to bind (it falls back to its private arena, and a reused + prefix's drafter K/V is lost). The cached quant modes are dropped with it, as the draft KV budget does. + """ + for quant in [draft_config.quant_config, *(draft_config.quant_config_dict or {}).values()]: + quant.kv_cache_quant_algo = None + quant.__dict__.pop("quant_mode", None) + quant.__dict__.pop("layer_quant_mode", None) + + +def draft_kv_override_applies(spec_config) -> bool: + """The target's FP8 draft-KV override is an Eagle3 need; a DSpark drafter keeps its BF16 context pool.""" + return not is_standalone_dspark(spec_config) + + +__all__ = [ + "M3DSparkWorker", + "M3EmbedTokens", + "build_dspark", + "draft_kv_override_applies", + "is_standalone_dspark", + "keep_drafter_kv_dtype", + "load_dspark_draft_weights", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/dspark_s4.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/dspark_s4.py new file mode 100644 index 000000000000..b0aa53afabcb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/dspark_s4.py @@ -0,0 +1,547 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""S4: the catalog's one-launch Markov chain (``catalog/spec_decode/dspark_markov_chain``) in this target's DSpark +worker (each M3.1 target keeps its own copy of this glue: targets share only the catalog). + +``S4WorkerMixin`` goes before ``DSparkWorker`` in the target's worker. For a generation step it serves (switch not +set, 1 <= generation requests <= ``MAX_FAST_BATCH``, greedy drafting, no d2t / guided decoder / DFlash-2 path, a +vocab-sharded borrowed head): + +- the draft head computes this rank's vocab shard of the block logits (the head's local GEMM, no gather) and copies + it into a padded buffer whose tail holds -inf (the step's logits processor, swapped in for the step); +- ``_refine_block_logits`` runs the catalog op over the shards in one launch and keeps the tokens; +- ``sample_draft_tokens`` returns them. + +The tokens are bitwise the in-tree chain's on the gathered logits: the gathered logits are the concatenation of the +same per-rank GEMMs, and the op reproduces the chain's bf16 bias, bf16 add and first-maximum argmax. When the op +does not support a step after the logits were sharded, the in-tree sharded path (``vocab_slice`` + the TP-aware +argmax) runs instead, with the same tokens. + +Setup (``prebuild_s4``, from the worker's ``prebuild``, outside capture, on every rank): the padded ``markov_w2`` +shard, one padded logits buffer per (batch, block), the exchange workspace (collective) and the compiles. + +Every routing decision is Python, so it is fixed into each decode graph when the graph is captured: +``TRTLLM_M31_DRAFT_DISABLE_S4`` is read once, at import, not a switch that can be toggled at run time. The set-up +decision must be the same on every rank (the workspace is collective); it follows from the config and the +environment, and ``prebuild_s4`` checks that the ranks agree before the collective. + +Gate only: with ``TRTLLM_M31_DSPARK_GATE_DUMP=`` set at prebuild (eager runs), each served step also runs the +in-tree chain on the gathered logits and logs both token sets per rank to ``/rank.jsonl``. It synchronizes; +never set it in a perf arm. +""" + +from __future__ import annotations + +import atexit +import json +import os +from typing import Dict, Optional, Tuple + +import torch +import torch.nn.functional as F +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_bias as markov_bias, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_chain as chain, +) +from tensorrt_llm._torch.models.modeling_speculative import dspark_markov_step_bias +from tensorrt_llm.logger import logger + +from .m31_switches import m31_disabled + +MARKOV_RANK = 256 +MAX_FAST_BATCH = ( + 8 # capacity: measured: above 8 requests the chain is within 10% of the reference (16: 0.93x) +) +DISABLE_ENV = "TRTLLM_M31_DRAFT_DISABLE_S4" +S4_COUNT_LOG_EVERY = 1000 # generation steps between the per-rank "served x/y" engagement lines +GATE_DUMP_ENV = "TRTLLM_M31_DSPARK_GATE_DUMP" # gate-only: a directory; never set in a perf arm (it synchronizes) + +# TAILBK_SPEC (DSpark-worker sites): bookkeeping launches +# of the draft tail removed exactly. Here, site C: the chain op's three read-only accepted inputs (ones, arange, +# zeros; 3 launches per step) come from a per-shape cache filled at precompile. Site H: the target's KV-length rewind +# after the draft forward (kv_lens[nc:bs] -= rewind; clamp_(min=0): 2 launches per step) runs in the chain's launch +# (its rewind path, after the grid-dependency wait; nothing between the chain and the stock rewind reads kv_lens). +# Site S4L (S4-logits, the advanced-sampling contract path): the biased block logits' glue around the bias GEMM -- the +# prevs cat / casts, get_masked_input_and_mask, F.embedding, masked_fill, the two widening copies and the add (14 +# launches per step) -- in two launches (catalog dspark_markov_bias; the GEMM unchanged). +# Bitwise the same tokens, kv_lens and biased logits. Default ON; kill switch TRTLLM_M31_DISABLE_TAILBK_SPEC=1, read +# at import. +TAILBK_SPEC = not m31_disabled("TAILBK_SPEC") +# MARKOV_LL (GB300 TP8 K7 b1 exchange 2.25 us + post 0.5 us per position +# against a ~1.25 us single-hop floor): the batch-1 kernel's per-position exchange as one 16-byte {value, tag, index, +# tag} multicast store per CTA (no fence, no flag word), polled and reduced by warp 0 alone (no CTA barrier after the +# poll). Bitwise the S4B8 kernel's tokens (same entries, same total argmax order). Default ON; kill switch +# TRTLLM_M31_DISABLE_MARKOV_LL=1, read at import. +MARKOV_LL = not m31_disabled("MARKOV_LL") + +# Read once, at import: every rank and every step of a process sees the same setting. Any value but "" or "0" turns +# the item off, as the targets' m31_switches reads its switches. +_DISABLED = os.environ.get(DISABLE_ENV, "").strip() not in ("", "0") + + +def disabled() -> bool: + return _DISABLED + + +class _S4State: + def __init__(self, mapping, rows: int, shard: int, offset: int, w1: torch.Tensor, w2s: torch.Tensor, + block: int, ws: dict) -> None: # fmt: skip + self.mapping = mapping + self.ws = ws + self.rows = rows + self.shard = shard + self.offset = offset + self.w1 = w1 + self.w2s = w2s + self.block = block + self.bufs: Dict[Tuple[int, int], torch.Tensor] = {} + + def buffer(self, batch: int) -> torch.Tensor: + return self.bufs[(batch, self.block)] + + +# S4-logits bias form: True = all K positions' bias in one GEMM over B*K rows (S4LC: bitwise the per-step GEMMs); +# False = the in-tree per-step loop over the chain's tokens (exact by construction). +S4_LOGITS_BATCHED = True + + +def _biased_shard_logits( + shard_logits: torch.Tensor, + first_prev: torch.Tensor, + tokens: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + fp32: bool, +) -> torch.Tensor: + """The in-tree chain's biased block logits on this rank's vocab shard, given the chain's tokens: step k adds + w2 @ w1[prev_k] (dspark_markov_step_bias) with prev_0 = the anchor and prev_k = token k-1, cast to the in-tree + logits dtype: fp32 under the S4 fp32 sum (LogitsProcessor returns logits.float()), else the shard's bf16.""" + if ( + fp32 + and S4_LOGITS_BATCHED + and TAILBK_SPEC + and markov_bias.supports(shard_logits, first_prev, tokens, w1) + ): + # TAILBK_SPEC S4L: the same prevs, embedding rows, GEMM and fp32 sum, in two launches around the GEMM. + B, K, N = shard_logits.shape + bias = F.linear(markov_bias.prev_embeddings(first_prev, tokens, w1), w2_shard) + return markov_bias.widen_add(shard_logits, bias.view(B, K, N)) + B, K, N = shard_logits.shape + base = shard_logits.float() if fp32 else shard_logits + prevs = torch.cat([first_prev.view(B, 1), tokens[:, : K - 1].long()], dim=1) + if S4_LOGITS_BATCHED: + bias = dspark_markov_step_bias(prevs.reshape(-1), w1, w2_shard).view(B, K, N) + return base + bias.to(base.dtype) + out = torch.empty_like(base) + for k in range(K): + out[:, k] = base[:, k] + dspark_markov_step_bias(prevs[:, k], w1, w2_shard).to(base.dtype) + return out + + +class _ShardedDraftLogits(nn.Module): + """The step's draft logits processor: this rank's vocab shard of ``lm_head`` over the gathered block rows, + copied into the padded buffer; any other call goes to the drafter's own processor.""" + + def __init__(self, inner: nn.Module, state: _S4State, batch: int) -> None: + super().__init__() + self.inner = inner + self._state = [state] + self.batch = batch + + def forward(self, hidden_states, lm_head, attn_metadata, return_context_logits: bool = False): + state = self._state[0] + if hidden_states.dim() != 2 or hidden_states.shape[0] != self.batch * state.block: + return self.inner(hidden_states, lm_head, attn_metadata, return_context_logits) + logits = lm_head.apply_linear(hidden_states, None) + buf = state.buffer(self.batch) + buf[:, :, : state.shard].copy_(logits.view(self.batch, state.block, state.shard)) + return logits + + +def _head_is_sharded(lm_head: nn.Module, vocab: int, tp: int) -> bool: + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + return ( + tp > 1 + and getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and getattr(lm_head, "padding_size", 0) == 0 + and getattr(lm_head, "bias", None) is None + and lm_head.out_features * tp == vocab + and lm_head.weight.dtype == torch.bfloat16 + ) + + +class S4WorkerMixin: + """Mixed into the target's ``DSparkWorker`` subclass (before it in the bases).""" + + _s4: Optional[_S4State] = None + _s4_step: Optional[_S4State] = None + _s4_tokens: Optional[torch.Tensor] = None + _s4_step_meta: Optional[Tuple[int, int, object]] = None + _s4_dump_step: int = 0 + _s4_dump_dir: Optional[str] = None + _s4_draft: Optional[nn.Module] = None + _s4_served: int = 0 + _s4_gen_steps: int = 0 + _s4_step_attn = ( + None # the step's attention metadata (the KV-length rewind's kv_lens), while S4 serves it + ) + _tailbk_rewound: bool = ( + False # TAILBK_SPEC H: this step's chain launch applied the KV-length rewind + ) + _tailbk_h_logged: bool = False + _tailbk_s4l_logged: bool = False + _tailbk_c_logged: bool = False + _s4_ll_logged: bool = False + + def prebuild_s4(self, lm_head: nn.Module, draft_model) -> bool: + """Set up the one-launch chain for this worker; False (and nothing set up) when it does not apply.""" + self._s4 = None + self._s4_dump_dir = os.environ.get(GATE_DUMP_ENV) or None + if disabled(): + logger.info(f"M3.1 DSpark S4: off ({DISABLE_ENV} set); the in-tree Markov chain runs") + return False + if not getattr(draft_model, "has_markov_head", False): + return False + w1 = draft_model.markov_w1 + w2 = draft_model.markov_w2 + mapping = self.mapping + tp = mapping.tp_size + vocab = w1.shape[0] + if ( + w1.dtype != torch.bfloat16 + or w2.dtype != torch.bfloat16 + or w1.shape[1] != MARKOV_RANK + or tuple(w2.shape) != tuple(w1.shape) + or mapping.enable_attention_dp + or not _head_is_sharded(lm_head, vocab, tp) + ): + return False + block = int(self.max_draft_len) + shard = vocab // tp + rows = chain.padded_rows(shard, MAX_FAST_BATCH, w1.device) + if not self._s4_ranks_agree(rows != 0 and block <= chain.MAX_BLOCK): + return False + offset = mapping.tp_rank * shard + w2s = chain.pad_w2_shard(w2[offset : offset + shard].contiguous(), rows) + state = _S4State( + mapping, rows, shard, offset, w1, w2s, block, chain.workspace_args(mapping) + ) + for batch in range(1, MAX_FAST_BATCH + 1): + state.bufs[(batch, block)] = chain.pad_logits_buffer(batch, block, rows, w1.device) + # S4 fp32 sum: the chain compares base + bias as the in-tree chain does, the unrounded fp32 sum (its logits are + # fp32 from LogitsProcessor's .float()). Kill switch TRTLLM_M31_DISABLE_S4_FP32SUM (bf16-rounded sums). + self._s4_fp32 = chain.enable_fp32_sum(not m31_disabled("S4_FP32SUM")) + # S4B8: batch-1 calls take the chain's batch-1 kernel (bitwise the same tokens; faster at one request only). + # Kill switch TRTLLM_M31_DISABLE_S4B8, read at import. + s4b8 = chain.enable_batch1_variant(not m31_disabled("S4B8")) + # TAILBK_SPEC site C: the chain's read-only accepted inputs from a per-shape cache (bitwise; see TAILBK_SPEC). + tailbk_c = chain.enable_cached_inputs(TAILBK_SPEC) + # MARKOV_LL: those batch-1 calls exchange through the LL flag-in-payload kernel (bitwise; see MARKOV_LL). + markov_ll = chain.enable_ll_exchange(MARKOV_LL) + # S4-logits: under advanced (rejection) draft sampling the chain still computes the greedy chain's tokens; the + # biased block logits are rebuilt from them (_biased_shard_logits) and the stock sampler gathers and samples + # them as it would the in-tree chain's. Kill switch TRTLLM_M31_DISABLE_S4_LOGITS, read at import. + self._s4_logits_on = not m31_disabled("S4_LOGITS") + chain.precompile(mapping, rows, w1, w2s, offset, blocks=(block,), max_batch=MAX_FAST_BATCH) + self._s4 = state + self._s4_draft = draft_model + atexit.register(self._s4_log_count) + # The engagement line: one per rank, at server start. + logger.info( + f"M3.1 DSpark S4: one-launch Markov chain set up on rank {mapping.tp_rank}/{tp} " + f"(shard {shard} rows padded to {rows}, K={block}, batches 1..{MAX_FAST_BATCH}; fp32 sum " + f"{'on' if self._s4_fp32 else 'off'}; S4-logits {'on' if self._s4_logits_on else 'off'}; " + f"S4B8 batch-1 kernel {'on' if s4b8 else 'off'}; MARKOV_LL {'on' if markov_ll and s4b8 else 'off'}; " + f"TAILBK_SPEC cached inputs {'on' if tailbk_c else 'off'})" + ) + return True + + def _s4_ranks_agree(self, local: bool) -> bool: + """``local`` if every rank of the TP group decided the same; raises when they differ (a rank going on into + the collective workspace alone would hang its peers).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + comm = _get_mnnvl_workspace_comm(self.mapping) + all_on = _mnnvl_workspace_all_succeeded(comm, local) + all_off = _mnnvl_workspace_all_succeeded(comm, not local) + if not (all_on or all_off): + raise RuntimeError( + "m31 DSpark S4: the TP ranks disagree on setting up the one-launch Markov chain" + ) + return all_on + + def _s4_serves(self, num_gens: int, spec_metadata, draft_model) -> bool: + return self._s4_mode(num_gens, spec_metadata, draft_model) is not None + + def _s4_mode(self, num_gens: int, spec_metadata, draft_model): + """ "tokens" (greedy drafting: the chain's tokens are the draft), "logits" (advanced draft sampling with + S4-logits on: the chain's corrected logits go to the stock sampler), or None (the in-tree chain runs).""" + if not self._s4_common(num_gens, draft_model): + return None + if not getattr(spec_metadata, "wants_advanced_draft_sampling", False): + return "tokens" + return "logits" if getattr(self, "_s4_logits_on", False) else None + + def _s4_common(self, num_gens: int, draft_model) -> bool: + return ( + self._s4 is not None + and 1 <= num_gens <= MAX_FAST_BATCH + and getattr(self, "guided_decoder", None) is None + and getattr(self, "_d2t", None) is None + and getattr(draft_model, "lilicorr", None) is None + and not getattr(draft_model, "has_candidate_selector", False) + ) + + def _host_step_hook(self, spec_metadata) -> None: + """The engagement counter (host only, no CUDA): called by the spec metadata's prepare() on every iteration, + outside any captured graph, it evaluates the same eligibility the captured step was routed by and counts + the generation steps the chain serves. One line per rank every S4_COUNT_LOG_EVERY generation steps and at + exit: "M3.1 DSpark S4: served / generation steps". The counts start at server start, so they include + warmup and padding steps: read engagement as the difference of two lines logged after the server is ready.""" + if self._s4 is None: + return + num_gens = int(getattr(spec_metadata, "num_generations", 0) or 0) + if num_gens <= 0: + return + self._s4_gen_steps += 1 + self._s4_served += int(self._s4_serves(num_gens, spec_metadata, self._s4_draft)) + if self._s4_gen_steps % S4_COUNT_LOG_EVERY == 0: + self._s4_log_count() + + def _s4_log_count(self) -> None: + if self._s4 is not None and self._s4_gen_steps: + logger.info( + f"M3.1 DSpark S4: served {self._s4_served}/{self._s4_gen_steps} generation steps on rank " + f"{self.mapping.tp_rank}" + ) + + def _forward_impl(self, input_ids, position_ids, hidden_states, logits, attn_metadata, spec_metadata, draft_model, + resource_manager=None): # fmt: skip + num_gens = attn_metadata.num_seqs - attn_metadata.num_contexts + mode = self._s4_mode(num_gens, spec_metadata, draft_model) + served = mode is not None + if self._s4_dump_dir and not torch.cuda.is_current_stream_capturing(): + # The gate's independent denominator: every generation step, its eligibility by batch and sampling + # alone, and whether the chain served it. + eligible = 1 <= num_gens <= MAX_FAST_BATCH and not getattr( + spec_metadata, "wants_advanced_draft_sampling", False + ) + self._s4_dump_line({"kind": "step", "num_gens": num_gens, "eligible": bool(eligible), + "served": bool(served)}) # fmt: skip + if not served: + return super()._forward_impl(input_ids, position_ids, hidden_states, logits, attn_metadata, + spec_metadata, draft_model, resource_manager) # fmt: skip + inner = draft_model.logits_processor + draft_model.logits_processor = _ShardedDraftLogits(inner, self._s4, num_gens) + self._s4_step = self._s4 + self._s4_step_mode = mode + self._s4_step_meta = (attn_metadata.num_contexts, attn_metadata.num_seqs, spec_metadata) + self._s4_step_attn = attn_metadata + try: + return super()._forward_impl(input_ids, position_ids, hidden_states, logits, attn_metadata, + spec_metadata, draft_model, resource_manager) # fmt: skip + finally: + draft_model.logits_processor = inner + self._s4_step = None + self._s4_step_mode = None + self._s4_tokens = None + self._s4_step_meta = None + self._s4_step_attn = None + self._tailbk_rewound = False + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + state = self._s4_step + if state is not None and gen_logits.dim() == 3 and gen_logits.shape[-1] == state.shard: + base = state.buffer(gen_logits.shape[0]) + if ( + chain.supports(base, state.w1, state.w2s) + and getattr(self, "_s4_step_mode", None) == "logits" + ): + # S4-logits: the chain's tokens are the in-tree greedy chain's (bitwise); the biased shard logits are + # rebuilt from them with the in-tree bias arithmetic and handed to the stock advanced sampler. + first_prev = inputs["first_prev_tokens"].long() + tokens = self._s4_chain(state, base, first_prev, spec_metadata) + self._s4_log_ll(base.shape[0]) + biased = _biased_shard_logits( + gen_logits, + first_prev, + tokens, + state.w1, + state.w2s[: state.shard], + getattr(self, "_s4_fp32", True), + ) + self._s4l_log(gen_logits, first_prev, tokens, state.w1) + if self._s4_dump_dir and not torch.cuda.is_current_stream_capturing(): + self._s4_logits_gate_dump( + draft_model, gen_logits, inputs, spec_metadata, biased + ) + return biased + if chain.supports(base, state.w1, state.w2s): + tokens = self._s4_chain( + state, base, inputs["first_prev_tokens"].long(), spec_metadata + ) + self._s4_log_ll(base.shape[0]) + self._s4_tokens = tokens + if self._s4_dump_dir and not torch.cuda.is_current_stream_capturing(): + self._s4_gate_dump(draft_model, gen_logits, inputs, spec_metadata, tokens) + return gen_logits + return super()._refine_block_logits(draft_model, gen_logits, inputs, spec_metadata) + + def _s4_chain( + self, state: _S4State, base: torch.Tensor, first_prev: torch.Tensor, spec_metadata + ): + """The chain launch; under TAILBK_SPEC (site H) it also applies the step's deferred KV-length rewind when + the stock _apply_kv_rewind_after_draft would (a real step: not the eager warmup of a graph batch, whose + kv_lens is restored wholesale), and marks it done for that method.""" + rewind = self._tailbk_rewind_args(base.shape[0], spec_metadata) + if rewind is None: + tokens = chain.dspark_markov_chain( + base, first_prev, state.w1, state.w2s, state.offset, **state.ws + ) + self._tailbk_c_log() + return tokens + kv_lens, amount, first = rewind + tokens = chain.dspark_markov_chain_rewind(base, first_prev, state.w1, state.w2s, state.offset, kv_lens, + amount, first, **state.ws) # fmt: skip + self._tailbk_rewound = True + self._tailbk_c_log() + if not self._tailbk_h_logged: + self._tailbk_h_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC H: KV-length rewind in the Markov chain launch " + f"on rank {self.mapping.tp_rank}" + ) + return tokens + + def _tailbk_c_log(self) -> None: + """The TAILBK_SPEC C engagement line: once per rank, after the first chain call that took its accepted inputs + from the precompile cache (host only; a captured step logs it while it is captured).""" + if self._tailbk_c_logged or not getattr(chain, "cached_inputs_taken", lambda: False)(): + return + self._tailbk_c_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC C: chain accepted inputs from precompile cache (taken) " + f"on rank {self.mapping.tp_rank}" + ) + + def _tailbk_rewind_args(self, batch: int, spec_metadata): + """(kv_lens, rewind amounts, first row) for the chain to apply, or None (the stock rewind runs).""" + if not TAILBK_SPEC or not getattr(self, "_kv_rewind_pending", False): + return None + amount = getattr(self, "_kv_rewind_amount", None) + attn = self._s4_step_attn + if amount is None or attn is None or not hasattr(attn, "kv_lens_cuda"): + return None + if spec_metadata.is_cuda_graph and not torch.cuda.is_current_stream_capturing(): + return None # the warmup: the stock method skips the rewind (kv_lens is restored wholesale) + nc, bs = self._kv_rewind_nc, self._kv_rewind_bs + if bs - nc != batch or amount.numel() != batch: + return None + kv_lens = attn.kv_lens_cuda + if not chain.supports_rewind(kv_lens, amount, nc): + return None + return kv_lens, amount, nc + + def _apply_kv_rewind_after_draft(self, attn_metadata, spec_metadata): + if self._tailbk_rewound: + # TAILBK_SPEC H: the chain launch applied it (see _s4_chain). + self._tailbk_rewound = False + self._kv_rewind_pending = False + return + super()._apply_kv_rewind_after_draft(attn_metadata, spec_metadata) + + def _s4l_log(self, shard_logits, first_prev, tokens, w1) -> None: + """The TAILBK_SPEC S4L engagement line: once per rank, on the first step whose biased logits took the + two-launch glue (host only; a captured step logs it while it is captured).""" + if self._tailbk_s4l_logged or not ( + getattr(self, "_s4_fp32", True) and S4_LOGITS_BATCHED and TAILBK_SPEC + ): + return + if markov_bias.supports(shard_logits, first_prev, tokens, w1): + self._tailbk_s4l_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC S4L: biased block logits glue in two launches " + f"on rank {self.mapping.tp_rank}" + ) + + def _s4_log_ll(self, batch: int) -> None: + """The MARKOV_LL engagement line: once per rank, on the first served step whose chain call took the LL + exchange kernel (host only; a captured step logs it while it is captured).""" + if not self._s4_ll_logged and chain.ll_exchange_engaged(batch): + self._s4_ll_logged = True + logger.info( + "[M3 dispatch] MARKOV_LL: k3_markov LL flag-in-payload exchange (batch 1) " + f"on rank {self.mapping.tp_rank}" + ) + + def _s4_gate_dump(self, draft_model, shard_logits, inputs, spec_metadata, tokens) -> None: + """Gate-only (eager): the in-tree chain on the gathered logits of this same step, logged beside the + one-launch chain's tokens, one JSON line per step and rank.""" + from tensorrt_llm._torch.distributed import allgather + + full = allgather(shard_logits.contiguous(), self.mapping, dim=-1) + if getattr(self, "_s4_fp32", True): + full = full.float() # the in-tree chain's input: LogitsProcessor returns logits.float() + corrected = super()._refine_block_logits(draft_model, full, inputs, spec_metadata) + stock = torch.argmax(corrected, dim=-1) + num_contexts, num_seqs, meta = self._s4_step_meta or (0, tokens.shape[0], None) + ids = getattr(meta, "request_ids", None) + ids = list(ids[num_contexts:num_seqs]) if ids is not None else None + self._s4_dump_line({"kind": "tokens", "step": self._s4_dump_step, "request_ids": ids, "s4": tokens.tolist(), + "stock": stock.tolist(), "equal": bool(torch.equal(tokens.long(), stock))}) # fmt: skip + self._s4_dump_step += 1 + + def _s4_logits_gate_dump( + self, draft_model, shard_logits, inputs, spec_metadata, biased + ) -> None: + """Gate-only (eager): the in-tree chain on the gathered logits of this same step (full-vocab bias), compared + bitwise with S4-logits' gathered biased logits, one JSON line per step and rank.""" + from tensorrt_llm._torch.distributed import allgather + + full = allgather(shard_logits.contiguous(), self.mapping, dim=-1) + if getattr(self, "_s4_fp32", True): + full = full.float() # the in-tree chain's input: LogitsProcessor returns logits.float() + stock = super()._refine_block_logits(draft_model, full, inputs, spec_metadata) + ours = allgather(biased.contiguous(), self.mapping, dim=-1) + num_contexts, num_seqs, meta = self._s4_step_meta or (0, biased.shape[0], None) + ids = getattr(meta, "request_ids", None) + ids = list(ids[num_contexts:num_seqs]) if ids is not None else None + mism = int((ours != stock).sum()) + self._s4_dump_line({"kind": "logits", "step": self._s4_dump_step, "request_ids": ids, + "shape": list(ours.shape), "dtype": str(ours.dtype), "mismatch": mism, + "equal": mism == 0 and ours.dtype == stock.dtype, + "batched": S4_LOGITS_BATCHED}) # fmt: skip + self._s4_dump_step += 1 + + def _s4_dump_line(self, rec: dict) -> None: + rec["rank"] = self.mapping.tp_rank + path = os.path.join(self._s4_dump_dir, f"rank{self.mapping.tp_rank}.jsonl") + with open(path, "a") as f: + f.write(json.dumps(rec) + "\n") + + def sample_draft_tokens(self, logits, spec_metadata, batch_size, **kwargs): + tokens = self._s4_tokens + if tokens is not None and logits.dim() == 3 and tokens.shape == logits.shape[:2]: + self._s4_tokens = None + return tokens if tokens.dtype == torch.int32 else tokens.to(torch.int32) + return super().sample_draft_tokens(logits, spec_metadata, batch_size, **kwargs) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/__init__.py new file mode 100644 index 000000000000..91d4ff843d5e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Kernels of the MiniMax-M3 NVFP4 / sm_107 / tp4 target.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/__init__.py new file mode 100644 index 000000000000..4c6fc92cd49c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 decode embedding gather + first-layer RMSNorm as one CuTe DSL kernel (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py new file mode 100644 index 000000000000..78563b4db40d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fixed M3 H6144 gather/raw residual/RMS, reusing pinned K3 norm kernel. + +128 threads x 8 BF16 x 6 iterations covers the row. Unlike the original +K3 wrapper this target does not promise FlashInfer's bit-identical reduction. +""" + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass.experimental import primitives as prims + +NORM_THREADS = 128 +NORM_VEC = 8 +NORM_WARPS = 4 # capacity: fixed: the CTA's warps + + +@cute.kernel +def m3_embed_norm_kernel( + ids: cute.Tensor, # int32 or int64 [N] + table: cute.Tensor, # bf16 [V, H] + weight: cute.Tensor, # bf16 [H] + raw: cute.Tensor, # bf16 [N, H], out: the rows + out: cute.Tensor, # bf16 [N, H], out: the normed rows + vocab: cutlass.Int64, + eps: cutlass.Float32, + tv_layout: cute.Layout, + tiler_mn: cute.Shape, + hidden: cutlass.Constexpr[int], +): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + prims.griddepcontrol(prims.GridDepAction.WAIT) + + smem = cutlass.utils.SmemAllocator() + reduction_buffer = smem.allocate_tensor( + cutlass.Float32, cute.make_layout((1, NORM_WARPS)), byte_alignment=4 + ) + + tok = cutlass.Int64(ids[bidx]) + valid = (tok >= cutlass.Int64(0)) & (tok < vocab) + row = cutlass.Int64(cutlass.select_(valid, tok, cutlass.Int64(0))) + + gX = cute.local_tile(table, tiler_mn, (row, 0)) + gR = cute.local_tile(raw, tiler_mn, (bidx, 0)) + gY = cute.local_tile(out, tiler_mn, (bidx, 0)) + w_layout = cute.prepend(weight.layout, cute.make_layout((tiler_mn[0],), stride=(0,))) + gW = cute.local_tile(cute.make_tensor(weight.iterator, w_layout), tiler_mn, (0, 0)) + + copy_atom_load = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), table.element_type, num_bits_per_copy=128 + ) + copy_atom_store = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), out.element_type, num_bits_per_copy=128 + ) + thr_copy_X = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_W = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_O = cute.make_tiled_copy(copy_atom_store, tv_layout, tiler_mn).get_slice(tidx) + + tXgX = thr_copy_X.partition_S(gX) + tXrX = cute.make_fragment_like(tXgX) + tWgW = thr_copy_W.partition_S(gW) + tWrW = cute.make_fragment_like(tWgW) + tXrW = thr_copy_X.retile(tWrW) + tXgR = thr_copy_O.partition_D(gR) + tXgO = thr_copy_O.partition_D(gY) + tXrO = cute.make_fragment_like(tXgO) + + # The row (zero for an id outside [0, V)), straight to the snapshot slot; the weight. + tXrX.store(cute.zeros_like(tXrX, dtype=table.element_type)) + if valid: + cute.copy(copy_atom_load, tXgX, tXrX) + cute.copy(copy_atom_load, tWgW, tWrW) + cute.copy(copy_atom_store, tXrX, tXgR) + + # flashinfer's RMSNormKernel arithmetic, in its order (see the header). + x = tXrX.load().to(cutlass.Float32) + x_sq = x * x + sum_sq = x_sq.reduce(cute.ReductionOp.ADD, init_val=cutlass.Float32(0.0), reduction_profile=0) + for i in cutlass.range_constexpr(5): + sum_sq = sum_sq + cute.arch.shuffle_sync_bfly(sum_sq, offset=1 << i) + lane = cute.arch.lane_idx() + warp = cute.arch.warp_idx() + if lane == 0: + reduction_buffer[0, warp] = sum_sq + cute.arch.barrier() + total = cutlass.Float32(0.0) + if lane < NORM_WARPS: + total = reduction_buffer[0, lane] + for i in cutlass.range_constexpr(5): + total = total + cute.arch.shuffle_sync_bfly(total, offset=1 << i) + mean_sq = total / cutlass.Float32(hidden) + rstd = cute.math.rsqrt(mean_sq + eps, fastmath=True) + w = tXrW.load().to(cutlass.Float32) + y = x * rstd * (w + cutlass.Float32(0.0)) + tXrO.store(y.to(out.element_type)) + cute.copy(copy_atom_store, tXrO, tXgO) + + +@cute.jit +def m3_embed_norm( + ids: cute.Tensor, + table: cute.Tensor, + weight: cute.Tensor, + raw: cute.Tensor, + out: cute.Tensor, + vocab: cutlass.Int64, + eps: cutlass.Float32, + n_tokens: cutlass.Constexpr[int], + hidden: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + # flashinfer's thread-value layout for one row: thread t, value (v, k) -> column 8 t + v + 1024 k. + blocks = hidden // (NORM_VEC * NORM_THREADS) + tv_layout = cute.make_layout(((NORM_THREADS, 1), (NORM_VEC, blocks)), + stride=((NORM_VEC, 1), (1, NORM_VEC * NORM_THREADS))) # fmt: skip + m3_embed_norm_kernel( + ids, table, weight, raw, out, vocab, eps, tv_layout, (1, hidden), hidden + ).launch( + grid=[n_tokens, 1, 1], + block=[NORM_THREADS, 1, 1], + smem=NORM_WARPS * 4, + stream=stream, + use_pdl=use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/op.py new file mode 100644 index 000000000000..c12bd2500445 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/embedding_firstnorm/op.py @@ -0,0 +1,157 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Host side of the MiniMax-M3 decode embedding + first-layer RMSNorm (m3_embedding_firstnorm_kernel.py): H 6144, +1..8 rows, into caller-owned buffers. + +Compiled on the first call per (rows, id dtype, table shape, PDL), which must happen outside CUDA-graph capture +(:class:`DecodeEmbedding` compiles at construction). +""" + +import threading +from pathlib import Path +from typing import Dict, Tuple + +import cuda.bindings.driver as cuda_driver +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import from_dlpack + +from tensorrt_llm._torch import cute_aot + +from .m3_embedding_firstnorm_kernel import m3_embed_norm as _m3_embed_norm_jit + +HIDDEN = 6144 # capacity: fixed: the checkpoint's hidden size +MAX_ROWS = 8 # capacity: fixed: the kernel's built rows (decode steps) + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + + +def arg(tensor): + return from_dlpack( + tensor.detach(), + assumed_align=tensor.element_size() + if tensor.ndim == 1 and tensor.dtype in (torch.int32, torch.int64) + else 16, + ) + + +def _check(ids, table, weight, raw, out) -> None: + rows = ids.shape[0] if ids.dim() == 1 else 0 + if not ( + 1 <= rows <= MAX_ROWS + and ids.dtype in (torch.int32, torch.int64) + and ids.is_contiguous() + and table.dtype == torch.bfloat16 + and table.dim() == 2 + and table.shape[1] == HIDDEN + and table.is_contiguous() + and weight.dtype == torch.bfloat16 + and tuple(weight.shape) == (HIDDEN,) + and all( + t.dtype == torch.bfloat16 and tuple(t.shape) == (rows, HIDDEN) and t.is_contiguous() + for t in (raw, out) + ) + ): + raise ValueError( + "m3_embed_norm: expects ids int32/int64 [1..8], a contiguous bf16 table [V, 6144], bf16 weight [6144] " + "and contiguous bf16 raw / out [rows, 6144]" + ) + + +def _compile(ids, table, weight, eps, raw, out, use_pdl: bool): + key = (ids.shape[0], ids.dtype, tuple(table.shape), table.device, bool(use_pdl)) + fn = _compiled.get(key) + if fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_embed_norm compiles on its first call per shape: call it once before capture" + ) + with _lock: + fn = _compiled.get(key) + if fn is None: + stream = cuda_driver.CUstream(torch.cuda.current_stream(table.device).cuda_stream) + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_embed_norm {key[0]} {key[1]} {key[2]} {key[4]} eps={float(eps)!r}", + lambda: cute.compile( + _m3_embed_norm_jit, + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + ids.shape[0], + HIDDEN, + bool(use_pdl), + stream, + ), + sources=_aot_sources(), + ) + return fn + + +def m3_embed_norm( + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """Write ``table[ids]`` into ``raw`` (a zero row for an id outside ``[0, V)``) and its RMSNorm with ``weight`` / + ``eps`` into ``out``; returns ``(out, raw)``.""" + _check(ids, table, weight, raw, out) + fn = _compile(ids, table, weight, eps, raw, out, use_pdl) + stream = cuda_driver.CUstream(torch.cuda.current_stream(ids.device).cuda_stream) + fn( + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + stream, + ) + return out, raw + + +class DecodeEmbedding: + """Caller-owned buffers of one (rows, id dtype) decode embedding: ``raw`` (the gathered rows, the first residual) + and ``output`` (their RMSNorm), over ``table`` and the first layer's plain (non-Gemma) input norm ``weight`` / + ``eps``. Built (and compiled) before CUDA-graph capture.""" + + def __init__(self, table, weight, eps, rows, id_dtype, use_pdl=True): + assert not torch.cuda.is_current_stream_capturing() + assert table.dtype == torch.bfloat16 and table.is_contiguous() and table.shape[1] == HIDDEN + assert weight.dtype == torch.bfloat16 and weight.shape == (HIDDEN,) + assert 1 <= rows <= MAX_ROWS and id_dtype in (torch.int32, torch.int64) + self.table, self.weight, self.rows, self.id_dtype, self.use_pdl = ( + table, + weight, + rows, + id_dtype, + use_pdl, + ) + self.eps = float(eps) + self.raw = torch.empty(rows, HIDDEN, device=table.device, dtype=torch.bfloat16) + self.output = torch.empty_like(self.raw) + ids = torch.empty(rows, device=table.device, dtype=id_dtype) + _compile(ids, table, weight, self.eps, self.raw, self.output, use_pdl) + + def __call__(self, ids): + assert ids.shape == (self.rows,) and ids.dtype == self.id_dtype and ids.is_contiguous() + return m3_embed_norm( + ids, self.table, self.weight, self.eps, self.raw, self.output, self.use_pdl + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/long_select.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/long_select.py new file mode 100644 index 000000000000..193cc96e72bf --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/long_select.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 block select for MiniMax-M3 decode rows of 129 to 9216 blocks. + +Same contract and bit-identical output as ``torch.ops.trtllm.minimax_m3_select_blocks`` on per-KV-head scores +[num_kv_heads, n_blocks, total_q]: init / local forcing, per-query valid-block count, ties to the smaller block index, +forced -inf picks emitted as -1, ascending block ids with -1 tail padding. + +One 1024-thread CTA per (query, KV head) row holds the row's ordered score keys in registers (2, 4 or 9 per +thread, by the row's valid count). The weakest of the 16 maxima of the interleaved groups ``block % 16`` is a lower +bound on the 16th-largest key, so the top 16 is among the keys at or above it; those are compacted to shared memory +and ranked exactly by (key, block). More than 1024 keys tied at the bound (degenerate rows) rank every candidate +against the whole row (re-read). + +The kernel is ``torch.ops.trtllm.minimax_m3_sm107_tp4_select_long`` (this target's copy of ``minimax_m3_select_long`` +with a choice of where it releases the sparse decode), called through this target's ``m3_select_long`` entry. +""" + +from typing import Optional + +import torch + +from ..ops.attention.m3_select_long import MAX_BLOCKS, m3_select_long +from ..ops.attention.m3_select_long_dsl import m3_select_long_dsl + + +def long_select_blocks( + max_score_kv: torch.Tensor, + *, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + dsl: bool = False, +) -> Optional[torch.Tensor]: + """Top-16 select over per-KV-head scores [num_kv_heads, n_blocks, total_q] with 129..9216 blocks. + + Returns the head-major backed [total_q, num_kv_heads, 16] int32 table, or None outside this kernel's shapes + (callers keep the reference op). ``early_trigger`` releases the dependent sparse decode right after the select's + grid wait (``SELECT_EARLY_MAX_ROWS`` in modeling.py); otherwise ``trigger_phase`` 1 releases it once every score + of the row is loaded and 0 at the select's exit (``SELECT_LONG_TRIGGER_PHASE``). ``fast_tail`` ranks rows of at + most 64 candidates on warp 0 (``SELECT_LONG_SL3``; the same picks). ``dsl`` runs the 384-thread kernel + (``SELECT_LONG_DSL``; the same table; ``fast_tail`` does not apply). + """ + if ( + max_score_kv.dtype != torch.float32 + or not max_score_kv.is_cuda + or max_score_kv.dim() != 3 + or not 128 < max_score_kv.shape[1] <= MAX_BLOCKS + or n_valid_blocks.dtype != torch.int32 + or not n_valid_blocks.is_cuda + or n_valid_blocks.shape[0] < max_score_kv.shape[2] + ): + return None + num_kv_heads, _, total_q = max_score_kv.shape + out = torch.empty( + (num_kv_heads, total_q, 16), dtype=torch.int32, device=max_score_kv.device + ).permute(1, 0, 2) + return (m3_select_long_dsl if dsl else m3_select_long)( + max_score_kv, + n_valid_blocks.contiguous(), + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + fast_tail, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_attn_lc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_attn_lc.py new file mode 100644 index 000000000000..d008355ebb9e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_attn_lc.py @@ -0,0 +1,163 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 long-context sparse decode attention in one launch. + +Replaces the generic chain's split-K producer (``_gqa_sparse_decode_kernel``, one block per chunk) and the LSE +merge (``_merge_topk_attn_out_kernel``, with its MXFP8 copy for the o_proj) once the top-k block table exists. + +Grid: one cluster of 16 CTAs per query token; CTA ``r`` attends the token's ``r``-th selected block (the Triton +producer's chunk ``r`` at 16 chunks), keeps the partial in shared memory, and after a cluster barrier merges head +``r`` from all 16 partials over DSMEM, writing the BF16 row and, optionally, its MXFP8 copy and swizzled UE8M0 +scales exactly as ``_store_mxfp8_head`` lays them out. + +Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and +PV on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +log-domain recurrence. Accumulation order inside the MMAs and the reductions differs, so results match to rounding, +not bit for bit. + +Schedule: the request length and the query rows (FP8 widened to FP16 in shared memory) are read before the PDL grid +wait, which overlaps them with the select ahead when it releases this grid early (modeling's +``SELECT_EARLY_MAX_ROWS``); +the split-K partials travel by ``st.async`` into each receiving CTA's own mbarrier instead of a cluster-wide barrier. +Neither changes a value or the merge order. + +Geometry: 16 query heads per KV head (TP4), one KV head, head_dim 128, page 128, top-k 16. The kernel is +``torch.ops.trtllm.minimax_m3_sparse_decode_lc``, called through this target's ``m3_sparse_decode_lc`` entry. +""" + +import torch + +from ..m31_switches import m31_disabled +from ..ops.attention.m3_sparse_decode_lc import m3_sparse_decode_lc + + +class PrefetchSlot: + """The read-only tensors the next long-context sparse decode prefetches into L2 (the attention layer's o_proj weight + and scales), set by the layer around its decode call and taken by the decode; empty means no prefetch.""" + + def __init__(self) -> None: + self.tensors = None + + def take(self): + tensors, self.tensors = self.tensors, None + return tensors + + +# Up to this many query rows the split-K merge divides once per output (VR TP4: ONE_DIV wins +# 15.7..19.9 us/step at 4 / 8 rows, ~0 at 16 and loses 2.2..2.3 us/step at 32, where the trigger phase changes). +# TRTLLM_M31_DISABLE_ONEDIV_CAP=1 keeps ONE_DIV at every row count; TRTLLM_M31_DISABLE_LC_ONEDIV=1 turns it off at every +# row count (the LSE-weighted merge), and wins over the cap switch. +ONE_DIV_MAX_ROWS = ( + 0 if m31_disabled("LC_ONEDIV") else 1 << 30 if m31_disabled("ONEDIV_CAP") else 16 +) # capacity: fixed: measured crossover + + +def one_div_applies(total_q: int) -> bool: + """Whether the merge divides once at ``total_q`` query rows.""" + return total_q <= ONE_DIV_MAX_ROWS + + +PREFETCH = PrefetchSlot() +# (this layer's id buffer, the next sparse layer's id buffer, its K / V pools, its K / V block scales) +NEXT_KV = PrefetchSlot() + + +def lc_applies(q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output) -> bool: + """Whether the step's operands fit the kernel: TP4 geometry, packed NVFP4 pages, FP8 q, BF16 [T, 16, 128] out.""" + if q.dim() != 3 or tuple(q.shape[1:]) != (16, 128) or q.dtype != torch.float8_e4m3fn: + return False + if q.stride(2) != 1 or q.stride(1) % 16 or q.stride(0) % 16 or q.data_ptr() % 16: + return False + if int(k_paged.shape[1]) != 1 or int(topk_idx.shape[0]) != 1 or int(topk_idx.shape[-1]) != 16: + return False + for t in (k_paged, v_paged): + if ( + t.stride(3) != 1 + or t.stride(2) != 64 + or int(t.shape[3]) != 64 + or t.data_ptr() % 16 + or t.stride(0) % 16 + ): + return False + for t in (k_block_scale, v_block_scale): + if t is None or t.stride(3) != 1 or t.stride(2) != 8 or t.data_ptr() % 8 or t.stride(0) % 8: + return False + return ( + output.dtype == torch.bfloat16 + and output.dim() == 3 + and output.is_contiguous() + and tuple(output.shape) == tuple(q.shape) + ) + + +def m3_lc_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + linear_softmax=True, + mxfp8=None, + use_pdl=True, + early_trigger=True, + merge_once=False, + prefetch=None, + next_kv=None, + identity=False, + one_div=False, + identity_step=False, +): + """Drop-in for ``minimax_m3_sparse_attn_decode`` on the TP4 NVFP4 shape; callers check ``lc_applies`` first. + + ``topk_idx`` is the head-major [1, total_q, 16] view the Triton producer takes. ``mxfp8`` is ``(q8, sf)`` from + ``MERGE_MXFP8.buffers(2048, dev)`` to also emit the o_proj hand-off. The op is built with the early PDL + trigger only (``early_trigger=True``). ``merge_once`` runs this target's op with the merge weights formed once + per CTA (bitwise the same output); with it, ``prefetch`` (up to two contiguous tensors) is pulled into L2 by the + op's CTAs before their grid dependency. + """ + total_q = int(q.shape[0]) + q8, sf = ( + (mxfp8[0].view(torch.uint8), mxfp8[1].view(torch.uint8)) + if mxfp8 is not None + else (None, None) + ) + m3_sparse_decode_lc( + q.view(torch.uint8), + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx[0], + block_table, + seq_lens, + output.view(total_q, 2048), + q8, + sf, + 2048 // 128, + int(decode_query_len), + float(sm_scale), + bool(linear_softmax), + bool(use_pdl), + bool(early_trigger), + merge_once=bool(merge_once), + identity=bool(identity) and bool(merge_once), + identity_step=bool(identity_step) and bool(merge_once), + one_div=bool(one_div) and bool(merge_once), + prefetch=prefetch if merge_once else None, + next_kv=( + tuple(next_kv[:2]) + tuple(t.view(torch.uint8) for t in next_kv[2:]) + if merge_once and next_kv is not None + else None + ), + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_draft_lowm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_draft_lowm.py new file mode 100644 index 000000000000..00c6c6887575 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_draft_lowm.py @@ -0,0 +1,143 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter's BF16 linears on FlashInfer's fixed-tactic low-M kernels (``DRAFT_LOWM``). + +The drafter's bias-free BF16 projections (qkv, o_proj, gate_up, down, the fc; not the head) call ``draft_linear`` +first: a covered row count runs ``ops/gemm/m3_draft_lowm`` at the tactic of its (N, K) table, every other call the +plain GEMM. ``prebuild`` compiles every covered call before the executor's warmup; a forward never compiles one, and a +covered call it finds uncompiled is a miss. +""" + +from __future__ import annotations + +from typing import FrozenSet, Iterable, Optional, Set, Tuple + +import torch + +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm.logger import logger + +from ..ops.gemm.m3_draft_lowm import m3_draft_lowm, tactic_for +from . import m3_report + +PATH = "m3_draft_lowm" + + +# ((N, K), M) of every call the target covers (its DRAFT_LOWM keys), set by prebuild. +_KEYS: FrozenSet[Tuple[Tuple[int, int], int]] = frozenset() +# (N, K, M, PDL) of every call prebuild compiled. +_BUILT: Set[Tuple[int, int, int, bool]] = set() +# (N, K, M) of every call that logged its engagement line. +_ENGAGED: Set[Tuple[int, int, int]] = set() +# DRAFT_AR_PUSH: the weight whose next low-M split-K call also pushes its rows into a boundary all-reduce +# (data pointer, the bucket's multicast words, its flags, rank, world, max tokens), set by arm_push; and whether the +# last armed call pushed: the data pointer of the output it pushed, 0 if none (read once by take_pushed). Host state +# set and read in one forward's issue order. +_PUSH_ARM: Optional[tuple] = None +_PUSHED = 0 + + +def arm_push(weight: torch.Tensor, workspace, rows: int) -> None: + """Arm ``weight``'s next covered split-K call to push into ``workspace``'s bucket for ``rows`` (see _PUSH_ARM).""" + global _PUSH_ARM, _PUSHED + from . import m3_mc_allreduce + + bucket = m3_mc_allreduce._bucket(workspace, rows) + flags = None if bucket is None else bucket.flags_for(8) + _PUSHED = 0 + _PUSH_ARM = None + if bucket is not None and flags is not None and bucket.max_tokens == 8: + _PUSH_ARM = ( + weight.data_ptr(), + bucket.mc.data_ptr(), + flags.data_ptr(), + bucket.rank, + bucket.world_size, + bucket.max_tokens, + ) + + +def take_pushed(x: torch.Tensor) -> bool: + """Whether the armed call pushed its rows; disarms. ``x`` is what the boundary all-reduce is about to reduce: it + must be the pushing call's own output (the prepushed all-reduce reads only the pushed words, never ``x``), so an + operation between that call and the all-reduce raises instead of silently reducing the wrong rows.""" + global _PUSH_ARM, _PUSHED + pushed, _PUSHED, _PUSH_ARM = _PUSHED, 0, None + if pushed and x.data_ptr() != pushed: + raise RuntimeError( + "the drafter's pushed rows are not the boundary all-reduce's input (an operation ran in between)" + ) + return bool(pushed) + + +def draft_linear( + weight: torch.Tensor, x: torch.Tensor, bias: Optional[torch.Tensor] +) -> Optional[torch.Tensor]: + """x [M, K] @ weight^T through the low-M kernel when the target covers it (``prebuild``'s keys); None for the + plain GEMM. The split-K ('s') tactics run this target's copy of FlashInfer's kernel (kernels/m3_lowm_splitk, bit + for bit the same output at every certified cell), which also carries the drafter-boundary push epilogue.""" + if bias is not None or x.dim() != 2: + return None + n, k = weight.shape + m = int(x.shape[0]) + if ((n, k), m) not in _KEYS: + m3_report.uncovered(PATH, f"the drafter's {n}x{k} linear past its tactic table's rows") + return None + if not x.is_contiguous(): + # The first pass's fc may read a column window of the captured hidden states; the plain GEMM takes strides. + m3_report.uncovered(PATH, "a strided activation (the plain GEMM reads it in place)") + return None + use_pdl = get_env_enable_pdl() + if x.dtype != torch.bfloat16 or weight.dtype != torch.bfloat16: + m3_report.missed(PATH, m, "the activation or weight is not BF16") + return None + if (n, k, m, use_pdl) not in _BUILT: + m3_report.missed(PATH, m, f"the {n}x{k} kernel at {m} rows was not compiled before warmup") + return None + kind, tactic = tactic_for(n, k, m) + if (n, k, m) not in _ENGAGED: + _ENGAGED.add((n, k, m)) + logger.info( + f"[M3 sm_107 tp4] fast path engaged: m3_draft_lowm {n}x{k} at {m} rows (device {x.device.index})" + ) + out = torch.empty((m, n), dtype=x.dtype, device=x.device) + global _PUSHED + ar = None + if _PUSH_ARM is not None and _PUSH_ARM[0] == weight.data_ptr() and kind == "s": + ar = _PUSH_ARM[1:] + _PUSHED = out.data_ptr() + return m3_draft_lowm(x, weight, out, kind, tactic, use_pdl, True, ar) + + +def installed(weight: torch.Tensor) -> bool: + """Whether the target covers ``weight`` (bias-free BF16 [N, K]) at some row count (``prebuild``'s keys).""" + if weight.dtype != torch.bfloat16 or weight.dim() != 2: + return False + return any(nk == tuple(weight.shape) for nk, _ in _KEYS) + + +def prebuild(weights: Iterable[torch.Tensor], keys: Iterable[Tuple[Tuple[int, int], int]]) -> int: + """Take the target's covered ((N, K), M) ``keys``, then compile (and run once) every covered row count of every + covered drafter weight, outside capture, on the current stream. Returns the number of calls compiled.""" + global _KEYS + _KEYS = frozenset(keys) + use_pdl = get_env_enable_pdl() + built = 0 + shapes = {tuple(weight.shape): weight for weight in weights if installed(weight)} + for (n, k), weight in sorted(shapes.items()): + for nk, m in sorted(key for key in _KEYS if key[0] == (n, k)): + if (n, k, m, use_pdl) in _BUILT: + continue + kind, tactic = tactic_for(n, k, m) + x = torch.zeros(m, k, dtype=torch.bfloat16, device=weight.device) + out = torch.empty(m, n, dtype=torch.bfloat16, device=weight.device) + m3_draft_lowm(x, weight, out, kind, tactic, use_pdl, True) + if kind == "s": + # The push variant a boundary all-reduce may arm (o_proj / down): compiled only, never run here + # (its pushes need the all-reduce's exchange and a matching call). + from .m3_lowm_splitk import compile_push + + compile_push(x, weight.t(), out, use_pdl, tactic) + _BUILT.add((n, k, m, use_pdl)) + built += 1 + return built diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_draft_step.py new file mode 100644 index 000000000000..45a64dfe9210 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_draft_step.py @@ -0,0 +1,222 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 draft pass for the MiniMax-M3 drafter's vocab-sharded BF16 head (TP decode, <= 8 requests). + +One kernel replaces a draft pass's head and the draft loop's per-pass bookkeeping, at the in-tree precision: + +* The drafter's BF16 lm_head shard is kept a second time, built by ``prepare_head`` before the executor's warmup: in + the lossless 12-bit coding of the target head (``ops.gemm.m3_target_head``; 75% of the bytes, kept only when it + decodes to the BF16 shard bit for bit), else in ``mma.m16n8k16`` A-fragment order (``bf16_fragments``, the same + values). The kernel streams it with BF16 MMAs and FP32 accumulation over the BF16 head input rows; a coded + exception decodes to +0 and its exact product x * w is added at its tile's flush. +* Each logit is rounded to BF16 (the in-tree logits are the BF16 head GEMM's output) and keyed + ``ordered(value) << 32 | ~global_index`` with -0 as +0, so the larger value wins and ties take the lowest index, as + the in-tree ``torch.max`` / ``torch.argmax`` over the gathered logits (``greedy_sample_draft_with_tp_gather``). +* The last CTA exchanges each row's key across the TP ranks through a multicast mapping (Lamport slots, see + ``m3_shard_accept``), then stores the draft token, the next pass's input id and position, and advances the draft KV + lengths as the in-tree draft loop does between passes. + +The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/draftStep.cu) and called through the +catalog entry ``ops.spec.m3_draft_step``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +import torch + +from ..ops.gemm.m3_target_head import encode_chunked, round_trip_ok +from ..ops.spec.m3_draft_step import MAX_TILES_PER_CTA, MAX_TOKENS, bf16_fragments, max_tile_pieces +from ..ops.spec.m3_draft_step import m3_draft_step as _draft_step_op +from . import m3_report +from .m3_shard_accept import AcceptWorkspace, create_workspace + +# The draft step launches this many CTAs instead of one per SM (3126 tiles: 200 CTAs x 15 + 9 x 14, at most 2 warp +# pieces per tile), so the CTAs own near-equal whole-tile counts and the stream does not end on the tiles' straggler +# tail (-1.04 / -1.18 us per pass at 1 / 4 rows vs 212 on Rubin TP4). A different static warp segmentation: +# exact-class. +GRID = 209 # capacity: fixed: 3126 head tiles of the TP4 shard, at most 2 warp pieces per tile on 209 CTAs +# The draft head kernel sums a tile that several warps stream in warp order (fixed by the launch shape) instead of by +# shared-memory atomics in arrival order: with fewer than 16 tiles per CTA a tile can take 3-4 pieces and the atomics' +# order changed the fp32 sum run to run; with 2 pieces the sum is the atomics' bit for bit. +FIXED_ORDER_PATH = "m3_draft_step_fixed_order" + + +@dataclass(eq=False) +class DraftHead: + """The streamed copy of one rank's draft lm_head shard and its first global id: ``wf`` is the 12-bit coded stream + with its exception lists (``coded``), or the fragment-ordered BF16 copy (``bf16_fragments``).""" + + weight: torch.Tensor + wf: torch.Tensor + vocab_offset: int + coded: bool = False + exc_off: Optional[torch.Tensor] = None + exc_pos: Optional[torch.Tensor] = None + exc_w: Optional[torch.Tensor] = None + exp_base: int = 0 + + +def head_supported(lm_head) -> bool: + """A bias-free BF16 head whose local shard the kernel can own: whole 16-row tiles, K a multiple of 16, at most + MAX_TILES_PER_CTA tiles per CTA of a one-CTA-per-SM grid.""" + weight = getattr(lm_head, "weight", None) + if weight is None or weight.dim() != 2 or weight.dtype != torch.bfloat16: + return False + if getattr(lm_head, "bias", None) is not None: + return False + n, k = weight.shape + if n % 16 or k % 16: + return False + return (n // 16 + _grid() - 1) // _grid() <= MAX_TILES_PER_CTA + + +def prepare_head(lm_head, tp_rank: int, coded: bool) -> DraftHead: + """The streamed copy of ``lm_head``'s local shard (global ids from tp_rank * rows): its 12-bit coding when + ``coded``, K is a multiple of 64 and the coding decodes to the shard bit for bit, else (fail + closed) the fragment-ordered BF16 copy. + Built once, eagerly, before the executor's warmup (``M3Eagle3Worker.prebuild_draft_head``).""" + weight = lm_head.weight + offset = tp_rank * weight.shape[0] + with torch.no_grad(): + w = weight.detach() + if coded and w.shape[1] % 64 == 0: + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + if round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w): + return DraftHead( + weight=weight, + wf=packed, + vocab_offset=offset, + coded=True, + exc_off=exc_off, + exc_pos=exc_pos, + exc_w=exc_w.view(-1), + exp_base=base, + ) + del packed, exc_off, exc_pos, exc_w + wf = bf16_fragments(w) + return DraftHead(weight=weight, wf=wf, vocab_offset=offset) + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _grid() -> int: + """The draft step's CTA count: ``GRID`` (at most one per SM).""" + return min(GRID, _num_sms()) + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace ([2][MAX_TOKENS][world] slots), created collectively on the first eager call; None + during a CUDA-graph capture that finds it missing.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = create_workspace(mapping, device, MAX_TOKENS) + return _WS + + +def _fixed_order( + rows: int, head: DraftHead, x: torch.Tensor, num_ctas: int, split: bool = False +) -> bool: + """The fixed warp-order tile sum where the launch lets a tile take more than 2 warp pieces (max_tile_pieces; a + split or equal-chunk launch always), with its engagement line; False keeps the atomics, deterministic on this + shape.""" + eq = getattr(head, "bnd", None) is not None + if not (split or eq): + chunks = x.shape[1] // (16 if head.exc_off is None else 64) + if ( + max_tile_pieces( + head.wf.shape[0] if head.exc_off is None else head.exc_off.numel() - 1, + num_ctas, + chunks, + ) + <= 2 + ): + return False + m3_report.taken( + FIXED_ORDER_PATH, + rows, + "the draft head sums shared tiles in warp order", + ) + return True + + +def draft_step( + head: DraftHead, + x: torch.Tensor, + rows: int, + x_rows: Optional[torch.Tensor], + mapping, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, +) -> bool: + """One fused greedy draft pass over ``rows`` requests (see the module docstring); False when the exchange + workspace is missing during a capture. Collective over the TP group. + + Args: + head: the prepared head of this rank (``prepare_head``), coded or BF16 fragments. + x: [*, K] BF16 head input rows (the drafter's final-norm output); row ``x_rows[b]`` (int64) or ``b`` is + request b's. + draft_out: int32 [rows, >= step + 1], unit column stride: column ``step`` receives the tokens. + ids_next / pos_next: int32 [rows]: the tokens, and ``pos_src[x_rows[b] or b] + 1`` (pos_src int32). + kv_lens: int32 draft KV lengths: ``-= draft_len - num_acc`` with ``seq_lens = 1`` on pass 0 when ``num_acc`` + is given, ``+= 1`` on later passes. + """ + ws = workspace(mapping, x.device) + if ws is None: + return False + _draft_step_op( + x, + x_rows, + rows, + head.wf, + head.vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + _grid(), + True, + None, + head.exc_off, + head.exc_pos, + head.exc_w, + head.exp_base, + None, # no fold tail: the forward's tail runs the standalone one + fixed_order=_fixed_order(rows, head, x, _grid()), + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_drafter_attn.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_drafter_attn.py new file mode 100644 index 000000000000..e5fa1bb5e945 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_drafter_attn.py @@ -0,0 +1,405 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter layer's decode attention on Rubin: ``M3DrafterFullCtxFmha``. + +The drafter layer is a plain ``TrtllmAttention`` over its own FP8 paged cache (the full context, every draft pass). +On SM 10.7 its generation phase runs ``thop.attention``'s in-tree trtllm-gen runner, whose split is capped at 8 CTAs +per (request, KV head): at TP4 (1 KV head per rank) 45.4 us per pass at 165k tokens on 212 SMs. + +This library keeps the in-tree preprocessing -- ``thop.trtllm_gen_generation_preprocess``: RoPE, the FP8 K/V append +into the paged cache and the FP8 query, as ``FlashInferTrtllmGenFmha`` runs it -- and attends with the full-context +PrimTS split-KV decode of ``kernels/m3_fullctx_decode`` at the same pinned recipes as the dense layers. + +With ``fused_rope_mode`` set, the preprocess and the query cast run as one launch instead +(``minimax_m3_sm107tp4_drafter_rope_fp8_kv``, the same bytes into the query and the cache), the layer's sub-page +pool and block table taken from the KV cache pool pointers and mapping the preprocess reads. + +It serves generation-only steps of the covered (query rows per request, requests) shapes; the target installs it +first in the drafter layer's FMHA list (``install``), so every other step falls through to the stock libraries. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Mapping, Optional, Tuple + +import torch + +from tensorrt_llm._torch.attention.backends.fmha.flashinfer_trtllm_gen import ( + FlashInferTrtllmGenFmha, +) +from tensorrt_llm._torch.attention.backends.fmha.interface import FmhaPhase +from tensorrt_llm._torch.attention.backends.fmha.phased import FmhaParams +from tensorrt_llm._torch.attention.backends.fmha.utils import ( + get_attention_chunk_size, + get_bmm1_scale, +) +from tensorrt_llm._torch.attention.backends.interface import AttentionInputType +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) +from tensorrt_llm._utils import TensorWrapper, convert_to_torch_tensor +from tensorrt_llm.bindings.internal import thop +from tensorrt_llm.functional import PositionEmbeddingType +from tensorrt_llm.quantization.mode import QuantMode + +from . import m3_fullctx_decode as fullctx + +# With PDL, the fused RoPE launch takes the maximum shared-memory carveout so the drafter decode's large-smem CTAs +# co-reside with it (timing only). +ROPE_CARVEOUT = True + + +def fwd_has_no_mrope(fwd) -> bool: + return getattr(fwd, "mrope_position_deltas", None) is None + + +if TYPE_CHECKING: + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + +class M3DrafterFullCtxFmha(FlashInferTrtllmGenFmha): + """Generation phase of the drafter layer: in-tree preprocess, PrimTS split-KV decode.""" + + def __init__( + self, + attn: "TrtllmAttention", + recipes: Mapping[Tuple[int, int], Mapping], + max_kv_len: int, + own_reducer: bool, + on_taken, + on_uncovered, + on_missed, + fused_rope_mode: Optional[int] = None, + pdl: bool = False, + on_rope_taken=None, + ) -> None: + super().__init__(attn) + self._recipes = recipes + self._max_kv_len = max_kv_len + self._own_reducer = own_reducer + self._fused_rope_mode = fused_rope_mode + self._pdl = pdl + self._on_rope_taken = on_rope_taken + self._on_taken = on_taken + self._on_uncovered = on_uncovered + self._on_missed = on_missed + + def _shape(self, q: torch.Tensor, metadata) -> Optional[Tuple[int, int]]: + nreq = int(metadata.num_generations) + if nreq <= 0 or int(q.shape[0]) % nreq: + return None + return int(q.shape[0]) // nreq, nreq + + def _is_supported(self, q, k, v, metadata, forward_args, *, phase=None) -> bool: + if phase == FmhaPhase.CONTEXT or metadata.num_contexts > 0: + return False + if phase is None and forward_args.attention_input_type == AttentionInputType.context_only: + return False + attn = self.attn + if ( + attn.num_heads != fullctx.HEADS + or attn.num_kv_heads != fullctx.KV_HEADS + or attn.head_dim != fullctx.HEAD_DIM + or not QuantMode(attn.quant_mode).has_fp8_kv_cache() + ): + self._on_uncovered("not the TP4 FP8 drafter geometry") + return False + # The arithmetic matches the stock decode only at unit KV scales (bmm1 and bmm2 carry none here); the layer's + # own scales are checked to be 1.0 at install, so any other per-step scale falls through to the stock path. + if ( + forward_args.kv_scale_orig_quant is not attn.kv_scale_orig_quant + or forward_args.kv_scale_quant_orig is not attn.kv_scale_quant_orig + ): + self._on_uncovered("per-step KV cache scales") + return False + if int(metadata.max_seq_len) > self._max_kv_len: + self._on_uncovered("the cache length bound is beyond the plans' K/V capacity") + return False + shape = self._shape(q, metadata) + if shape is None or shape not in self._recipes: + self._on_uncovered(f"{shape} is not a covered (query rows, requests) shape") + return False + # Compiled before warmup (``install``); the forward never compiles one. + if fullctx.plan_for(q.device, *shape, self._own_reducer) is None: + self._on_missed(shape[0] * shape[1], "the PrimTS decode plan is not prebuilt") + return False + return super()._is_supported(q, k, v, metadata, forward_args, phase=FmhaPhase.GENERATION) + + def _fused_rope_applies(self, params: FmhaParams, attention_input: torch.Tensor) -> bool: + attn = params.attn + width = (attn.num_heads + 2 * attn.num_kv_heads) * attn.head_dim + rotary_dim = int(attn.rope_params.dim) + return ( + self._fused_rope_mode is not None + and params.qkv_input is not None + and attention_input.dim() == 2 + and attention_input.dtype == torch.bfloat16 + and attention_input.is_contiguous() + and int(attention_input.shape[1]) == width + and not params.is_cross + and int(attn.position_embedding_type) == int(PositionEmbeddingType.rope_gpt_neox) + and attn.rotary_cos_sin is not None + and attn.rotary_cos_sin.dtype == torch.float32 + and fwd_has_no_mrope(params.fwd) + and 0 < rotary_dim <= attn.head_dim + and rotary_dim % 16 == 0 + and int(params.num_tokens) == int(params.batch_size) * int(params.input_seq_length) + # A one-request multi-token pass: the in-tree preprocess does not build its token map (buildDecoderInfo + # runs for several requests only) and reads the workspace's, so it keeps that path. + and not (self._multi_token(params) and int(params.batch_size) == 1) + ) + + @staticmethod + def _multi_token(params: FmhaParams) -> bool: + return ( + params.spec_decoding_generation_lengths is not None + and int(params.attn.predicted_tokens_per_seq) > 1 + ) + + def _run_fused_rope(self, params: FmhaParams, attention_input: torch.Tensor) -> bool: + """The pass with the one-launch preprocess; False (nothing run) where it does not apply.""" + if not self._fused_rope_applies(params, attention_input): + return False + attn = params.attn + meta = params.meta + qlen, nreq = int(params.input_seq_length), int(params.batch_size) + plan = fullctx.plan_for(attention_input.device, qlen, nreq, self._own_reducer) + if plan is None: + return False + # The preprocess's view of the layer's cache (buildFlashinferTrtllmGenPagedKvCacheBuffers): the pool's primary + # pointer plus the layer's offset inside its pool, sub-pages of Hkv x tokens_per_block x 128 E4M3 bytes; the + # block table is the pool's [requests, 2, blocks] K / V sub-page indices. + pool_index, layer_in_pool = ( + int(x) for x in meta.host_kv_cache_pool_mapping[attn.local_layer_idx, :2] + ) + tpb = int(params.tokens_per_block) + sub_page_bytes = attn.num_kv_heads * tpb * attn.head_dim + base = ( + int(meta.host_kv_cache_pool_pointers[pool_index, 0]) + + layer_in_pool * int(params.kv_factor) * sub_page_bytes + ) + pages = (int(params.total_num_blocks) - 1) * int(params.kv_factor) + 2 + kv_pages = convert_to_torch_tensor( + TensorWrapper(base, torch.float8_e4m3fn, [pages, attn.num_kv_heads, tpb, attn.head_dim]) + ) + tables = meta.kv_cache_block_offsets[ + pool_index, params.seq_offset : params.seq_offset + nreq + ] + multi = self._multi_token(params) + if not torch.cuda.is_current_stream_capturing(): + # The decode reads V page p as sub-page p + 1 of K's (kv_pages[1:] with the K table), as the preprocess + # path checks; the op itself writes V at the table's V sub-page. + tpb_held = (params.sequence_lengths[:nreq].to(torch.long) + tpb - 1) // tpb + cols = torch.arange(tables.shape[-1], device=tables.device) + in_use = cols[None, :] < tpb_held[:, None] + offsets = (tables[:, 1] - tables[:, 0])[in_use].unique() + if offsets.numel() != 1 or int(offsets) != 1: + raise RuntimeError( + f"the drafter cache does not keep V one sub-page after K: {offsets.tolist()}" + ) + q = torch.empty( + int(params.num_tokens), + attn.num_heads, + attn.head_dim, + dtype=torch.float8_e4m3fn, + device=attention_input.device, + ) + + torch.ops.trtllm.minimax_m3_sm107tp4_drafter_rope_fp8_kv( + attention_input, + attn.rotary_cos_sin, + params.sequence_lengths, + params.spec_decoding_position_offsets if multi else None, + kv_pages, + tables, + q, + qlen, + attn.num_heads, + attn.num_kv_heads, + int(attn.rope_params.dim), + tpb, + self._fused_rope_mode, + self._pdl, + ROPE_CARVEOUT, + ) + plan.run( + q, + kv_pages[:-1], + kv_pages[1:], + tables[:, 0], + params.sequence_lengths[:nreq], + get_bmm1_scale(attn), + params.output.view(params.num_tokens, attn.num_heads, attn.head_dim), + ) + if self._on_rope_taken is not None: + self._on_rope_taken(params.num_tokens, "spec" if multi else "plain") + self._on_taken(params.num_tokens) + return True + + def run_generation(self, params: FmhaParams) -> None: + attention_input = params.qkv_input if params.qkv_input is not None else params.query_input + if attention_input is None: + raise RuntimeError("the drafter decode requires QKV or query input.") + attn = params.attn + meta = params.meta + fwd = params.fwd + rope_params = attn.rope_params + if self._run_fused_rope(params, attention_input): + return + # The BF16 post-RoPE query, rounded to E4M3 below. At the unit KV scales this layer runs with, that is the FP8 + # query the stock decode gets (the preprocess's FP8 variant is e4m3(q * kv_scale_orig_quant)). + fp8_context_fmha = False + ( + q_processed, + kv_pool, + block_tables, + _kv_scale_pool, + _bmm1_scale, + _bmm2_scale, + _fmha_workspace, + _cu_seqlens, + _max_q_len, + _max_kv_len, + _window_left, + is_multi_token_gen, + ) = thop.trtllm_gen_generation_preprocess( + attention_input, + params.workspace, + params.sequence_lengths, + params.spec_decoding_generation_lengths, + params.spec_decoding_position_offsets, + meta.kv_cache_block_offsets, + meta.host_kv_cache_pool_pointers, + meta.host_kv_cache_pool_mapping, + fwd.kv_scale_orig_quant, + fwd.kv_scale_quant_orig, + fwd.out_scale, + attn.rotary_inv_freq, + attn.rotary_cos_sin, + fwd.mrope_position_deltas, + attn.local_layer_idx, + params.seq_offset, + attn.num_heads, + attn.num_kv_heads, + attn.head_dim, + params.tokens_per_block, + attn.quant_mode, + params.max_attention_window_size, + params.cyclic_attention_window_size, + params.num_tokens, + params.batch_size, + params.input_seq_length, + params.max_past_kv_length, + rope_params.dim, + rope_params.theta, + int(rope_params.scale_type), + rope_params.scale, + rope_params.max_positions, + attn.position_embedding_type, + get_bmm1_scale(attn), + 1.0, + fp8_context_fmha, + attn.predicted_tokens_per_seq, + get_attention_chunk_size(attn), + self._multi_processor_count, + params.total_num_blocks, + params.kv_factor, + True, + params.is_cross, + ) + qlen, nreq = params.input_seq_length, params.batch_size + plan = fullctx.plan_for(attention_input.device, qlen, nreq, self._own_reducer) + # A first draft pass arrives as qlen rows per request either as a plain multi-row generation step (the + # drafter's own, input_seq_length = qlen) or with spec-decoding lengths (multi-token generation); a 1-row + # pass is never multi-token. + if plan is None or (is_multi_token_gen and qlen == 1): + raise RuntimeError( + f"the drafter decode was selected for {qlen} query rows x {nreq} requests without a uniform plan" + ) + rows = params.num_tokens * attn.num_heads * attn.head_dim + q = fp8_e4m3_saturate(q_processed.flatten()[:rows]) + q = q.view(params.num_tokens, attn.num_heads, attn.head_dim) + # K pages in row 0 of the table, V in row 1; this cache keeps V one sub-page after K, so V page p is page p + 1 + # of the pool (the two views have the same page count, as the kernel requires). + if not torch.cuda.is_current_stream_capturing(): + # The plan runs every request at qlen rows; a multi-token pass must give each request exactly that many. + lengths = params.spec_decoding_generation_lengths + if is_multi_token_gen and not bool((lengths[:nreq] == qlen).all()): + raise RuntimeError( + f"the drafter pass rows per request are not uniform: {lengths[:nreq].tolist()}" + ) + # Only the blocks a request holds are laid out; the table's tail entries are unset. + tpb = int(params.tokens_per_block) + held = (params.sequence_lengths[:nreq].to(torch.long) + tpb - 1) // tpb + cols = torch.arange(block_tables.shape[-1], device=block_tables.device) + in_use = cols[None, :] < held[:, None] + offsets = (block_tables[:nreq, 1] - block_tables[:nreq, 0])[in_use].unique() + if offsets.numel() != 1 or int(offsets) != 1: + raise RuntimeError( + f"the drafter cache does not keep V one sub-page after K: {offsets.tolist()}" + ) + # The preprocess sizes kv_pool by blocks; its table indexes the K and V sub-pages of every block. + # The view stops two sub-pages past the last block's K, as get_kv_subpage_pool's does. + pages = (int(params.total_num_blocks) - 1) * int(params.kv_factor) + 2 + kv_pages = convert_to_torch_tensor( + TensorWrapper(kv_pool.data_ptr(), kv_pool.dtype, [pages, *kv_pool.shape[1:]]) + ) + plan.run( + q, + kv_pages[:-1], + kv_pages[1:], + block_tables[:nreq, 0], + params.sequence_lengths[:nreq], + get_bmm1_scale(attn), + params.output.view(params.num_tokens, attn.num_heads, attn.head_dim), + ) + self._on_taken(params.num_tokens) + + +def install( + draft_model: torch.nn.Module, + recipes, + max_kv_len: int, + own_reducer: bool, + pdl: bool, + on_taken, + on_uncovered, + on_missed, + fused_rope_mode: Optional[int] = None, + on_rope_taken=None, +) -> int: + """Prebuild the plans of ``recipes`` and put ``M3DrafterFullCtxFmha`` first in the FMHA list of the TrtllmAttention + backend of every attention module of ``draft_model``. Call before warmup. Returns the number of attention layers it + was installed on.""" + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + count = 0 + device = next(draft_model.parameters()).device + # The backend is not a module: each attention module of the drafter holds it as ``attn``. + for module in draft_model.modules(): + backend = getattr(module, "attn", None) + if not isinstance(backend, TrtllmAttention) or not hasattr(backend, "_fmha_manager"): + continue + if not bool((backend.kv_cache_scaling_factor == 1.0).all()): + on_uncovered("a scaled FP8 KV cache") + continue + fullctx.prebuild(device, recipes, max_kv_len, own_reducer, pdl) + libs = backend._fmha_manager.fmha_libs + if not any(isinstance(lib, M3DrafterFullCtxFmha) for lib in libs): + libs.insert( + 0, + M3DrafterFullCtxFmha( + backend, + recipes, + max_kv_len, + own_reducer, + on_taken, + on_uncovered, + on_missed, + fused_rope_mode=fused_rope_mode, + pdl=pdl, + on_rope_taken=on_rope_taken, + ), + ) + count += 1 + return count diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_fold_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_fold_tail.py new file mode 100644 index 000000000000..79aff5cb80ce --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_fold_tail.py @@ -0,0 +1,172 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The speculative step's store and next-step staging at the tail of the forward (one Triton launch). + +For each of the first ``n`` rows (``n`` read from device memory, so a captured graph whose step was not armed writes +nothing), with ``s = slots[i]``: + +* the spec sampler's per-slot store, exactly as ``m3_spec_tail.slot_scatter`` writes it (zero-padded to each store + width): ``new[s][c]``, ``next_new[s][c]``, ``draft[s][c]`` and ``lens[s]``; +* the engine's next-step inputs, exactly as ``m3_spec_tail.step_gather`` writes them from that store for a step whose + rows are these rows in this order (no new tokens, first previous row 0): ``ids[i*T + t] = next_new[s][t]``, + ``draft_ids[i*D + d] = draft[s][d]``, ``pos_off[i*T + t] = lens[s]``, ``kv_off[i] = lens[s] - T``. + +Integer copies only: the values are the ones the sampler's and the engine's own copies write. + +With ``cpp`` the same writes run as this target's C++ op (trtllm::minimax_m3_sm107_tp4_fold_tail, +minimaxM3Sm107Tp4/foldTail.cu: one 32-lane CTA per row) instead of the Triton kernel, for widths up to 32. +""" + +import torch +import triton +import triton.language as tl + + +@triton.jit +def _fold_tail_kernel( + ctrl, + slots, + src_new, + sn_s0, + sn_s1, + SN_W: tl.constexpr, + dst_new, + dn_s0, + dn_s1, + DN_W: tl.constexpr, + src_next, + sx_s0, + sx_s1, + SX_W: tl.constexpr, + dst_next, + dx_s0, + dx_s1, + DX_W: tl.constexpr, + src_draft, + sd_s0, + sd_s1, + SD_W: tl.constexpr, + dst_draft, + dd_s0, + dd_s1, + DD_W: tl.constexpr, + src_lens, + dst_lens, + ids, + draft_ids, + pos_off, + kv_off, + T: tl.constexpr, + D: tl.constexpr, + BLOCK: tl.constexpr, +): + i = tl.program_id(0) + n = tl.load(ctrl) + if i < n: + s = tl.load(slots + i).to(tl.int64) + c = tl.arange(0, BLOCK) + new = tl.load(src_new + i * sn_s0 + c * sn_s1, mask=c < SN_W, other=0) + tl.store(dst_new + s * dn_s0 + c * dn_s1, new, mask=c < DN_W) + nxt = tl.load(src_next + i * sx_s0 + c * sx_s1, mask=c < SX_W, other=0) + tl.store(dst_next + s * dx_s0 + c * dx_s1, nxt, mask=c < DX_W) + drf = tl.load(src_draft + i * sd_s0 + c * sd_s1, mask=c < SD_W, other=0) + tl.store(dst_draft + s * dd_s0 + c * dd_s1, drf, mask=c < DD_W) + ln = tl.load(src_lens + i) + tl.store(dst_lens + s, ln) + # The next step's inputs read the store's padded rows: zero past each source width. + tl.store(ids + i * T + c, tl.where(c < DX_W, nxt, 0), mask=c < T) + tl.store(draft_ids + i * D + c, tl.where(c < DD_W, drf, 0), mask=c < D) + tl.store(pos_off + i * T + c, tl.zeros([BLOCK], dtype=tl.int32) + ln, mask=c < T) + tl.store(kv_off + i, ln - T) + + +def fold_tail( + ctrl: torch.Tensor, + slots: torch.Tensor, + rows: int, + src_new: torch.Tensor, + dst_new: torch.Tensor, + src_next: torch.Tensor, + dst_next: torch.Tensor, + src_draft: torch.Tensor, + dst_draft: torch.Tensor, + src_lens: torch.Tensor, + dst_lens: torch.Tensor, + ids: torch.Tensor, + draft_ids: torch.Tensor, + pos_off: torch.Tensor, + kv_off: torch.Tensor, + tokens_per_step: int, + cpp: bool = False, +) -> bool: + """Launch over ``rows`` programs (the first ``ctrl[0]`` write); False, having launched nothing, when the inputs + are outside what the kernel covers. Store views are 2-D [slots, width] (any strides), sources [rows, width].""" + two = (src_new, dst_new, src_next, dst_next, src_draft, dst_draft) + one = (src_lens, dst_lens, ids, draft_ids, pos_off, kv_off) + if any(t.dim() != 2 for t in two) or any(t.dim() != 1 or t.stride(0) != 1 for t in one): + return False + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in two + one + (ctrl,)) + or slots.dtype != torch.long + ): + return False + t_step = tokens_per_step + d_step = t_step - 1 + if ( + rows <= 0 + or slots.numel() < rows + or any(t.shape[0] < rows for t in (src_new, src_next, src_draft, src_lens)) + or ids.numel() < rows * t_step + or pos_off.numel() < rows * t_step + or draft_ids.numel() < rows * d_step + or kv_off.numel() < rows + ): + return False + widths = (src_new.shape[1], dst_new.shape[1], src_next.shape[1], dst_next.shape[1]) + widths += (src_draft.shape[1], dst_draft.shape[1], t_step) + if cpp and max(widths) <= 32: + torch.ops.trtllm.minimax_m3_sm107_tp4_fold_tail( + ctrl, slots, rows, src_new, dst_new, src_next, dst_next, src_draft, dst_draft, + src_lens, dst_lens, ids, draft_ids, pos_off, kv_off, t_step, + ) # fmt: skip + return True + block = triton.next_power_of_2(max(widths)) + _fold_tail_kernel[(rows,)]( + ctrl, + slots, + src_new, + src_new.stride(0), + src_new.stride(1), + src_new.shape[1], + dst_new, + dst_new.stride(0), + dst_new.stride(1), + dst_new.shape[1], + src_next, + src_next.stride(0), + src_next.stride(1), + src_next.shape[1], + dst_next, + dst_next.stride(0), + dst_next.stride(1), + dst_next.shape[1], + src_draft, + src_draft.stride(0), + src_draft.stride(1), + src_draft.shape[1], + dst_draft, + dst_draft.stride(0), + dst_draft.stride(1), + dst_draft.shape[1], + src_lens, + dst_lens, + ids, + draft_ids, + pos_off, + kv_off, + t_step, + d_step, + block, + num_warps=1, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_force_vec.py new file mode 100644 index 000000000000..be04f26f9aff --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_force_vec.py @@ -0,0 +1,64 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance (the in-tree speculative worker's +``force_num_accepted_tokens``), in one launch. + +Triton: ``force_vec_kernel``: one CTA bumps the shared step counter and draws each request's count from the fixed-seed +pool with the in-tree (counter, slot) hash, exactly as ``_apply_force_accepted_tokens`` does. ``force_vec_req_kernel`` +(``FORCE_AL_PER_REQUEST``): each request draws from the same pool at its own key, its KV length at this verify. +Reached through the ``ops/spec`` entries ``m3_force_vec`` and ``m3_force_vec_req``. + +One compiled variant for every batch and accept count: specializing them (== 1, divisible by 16) would compile +mid-serving the first time a step reaches a new one, past the warm-up. +""" + +import triton +import triton.language as tl + + +@triton.jit(do_not_specialize=["batch_size", "base_total", "max_total"]) +def force_vec_kernel( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK: tl.constexpr, +): + c = tl.load(counter) + 1 + slots = tl.arange(0, BLOCK) + tl.store(counter, c) + idx = (c * counter_stride + slots.to(tl.int64) * slot_stride) & pool_mask + live = slots < batch_size + r = tl.load(pool + idx, mask=live, other=1.0) + v = tl.minimum(base_total + (r < frac).to(tl.int32), max_total) + tl.store(out + slots, v, mask=live) + + +@triton.jit(do_not_specialize=["batch_size", "base_total", "max_total"]) +def force_vec_req_kernel( + pool, + keys, + out, + batch_size, + base_total, + max_total, + frac, + key_stride, + pool_mask, + BLOCK: tl.constexpr, +): + # One draw per request from its own key (its KV length at this verify): no shared counter, so a request's forced + # counts depend only on its own prompt and progress, not on what else the server ran before or beside it. + slots = tl.arange(0, BLOCK) + live = slots < batch_size + k = tl.load(keys + slots, mask=live, other=0).to(tl.int64) + idx = (k * key_stride) & pool_mask + r = tl.load(pool + idx, mask=live, other=1.0) + v = tl.minimum(base_total + (r < frac).to(tl.int32), max_total) + tl.store(out + slots, v, mask=live) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_fullctx_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_fullctx_decode.py new file mode 100644 index 000000000000..ef6b9ee0d1ad --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_fullctx_decode.py @@ -0,0 +1,313 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 full-context FP8 decode attention on Rubin: ``fullctx_decode``. + +The dense layers (0-2) attend every token of their FP8 E4M3 (unscaled) paged KV cache. On SM 10.7 the stock call +(flashinfer trtllm-gen, ``minimax_m3_trtllm_gen_dense_decode``) caps its split at 16 cluster CTAs per (request, KV +head): at TP4 (16 query heads and 1 KV head per rank) one request runs on 16 CTAs of 212 and takes 33.4-33.8 us +per call at 165k tokens. + +This runs the same arithmetic -- FP8 Q (the same E4M3 rounding of q), FP8 K/V, P quantized to E4M3 per split -- on +the vendored PrimTS decode kernel (``attention/backends/prims_ts``, SM 10.7 through the sm_100f family) at a pinned +split-KV recipe, which fans one request out over 96-108 CTAs. The split partials are merged by +``torch.ops.trtllm.minimax_m3_sm107tp4_fullctx_reduce`` (``cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/ +fullCtxReduce.cu``, head-count generic, reused by name): unnormalized fp32 accumulation over the active splits, one +final division, one BF16 rounding, the merge form of stock trtllm-gen. Only the split changes against stock +(``modeling.FULLCTX_DECODE_RECIPES``). + +A plan is compiled per (device, query rows per request, requests) by ``prebuild`` before the executor's warmup; the +forward only reads plans (``plan_for``). The plan is built from PrimTS's own config, compile and workspace helpers at +an explicit recipe; the PrimTS wrapper's planning policy is not used, so no PrimTS state is replaced. With +``own_reducer`` off the PrimTS reducer merges into FP16 and the result is cast to BF16. +""" + +from __future__ import annotations + +import threading +from typing import Dict, Mapping, Optional, Tuple + +import torch + +# The TP4 per-rank geometry: 16 query heads over 1 KV head. +HEADS = 16 +KV_HEADS = 1 # capacity: fixed: the tp4 KV heads per rank +HEAD_DIM = 128 # capacity: fixed: the checkpoint's head dim +PAGE = 128 +# The per-request K/V length PrimTS's config selection sizes its policy for (Q/KV tiles, launch mode); the pinned +# recipes were measured with it. The compiled capacity (``max_kv_len``) may be larger. +POLICY_KV_LEN = ( + 1 << 20 +) # capacity: fixed: the PrimTS policy's K/V length bound (the plans' compile shape) +HEADS_PER_KV = HEADS // KV_HEADS # capacity: fixed: the tp4 query heads per KV head + +_plans: Dict[Tuple[int, int, int, bool], "FullCtxPlan"] = {} +_lock = threading.Lock() + + +class FullCtxPlan: + """One compiled PrimTS split-KV decode at a pinned recipe, its workspace and its FP16 staging output.""" + + def __init__( + self, + device: torch.device, + qlen: int, + nreq: int, + max_kv_len: int, + recipe: Mapping, + own_reducer: bool, + pdl: bool, + ): + from tensorrt_llm._torch.attention.backends.prims_ts import decode as tsdec + from tensorrt_llm._torch.attention.backends.prims_ts.kernels.fmha_decode.fmha_decode_config import ( + get_max_active_clusters_for_cluster_size, + make_decode_config, + ) + + self._tsdec = tsdec + self.qlen = qlen + self.nreq = nreq + self.pdl = pdl + self.recipe = dict(recipe) + splits = int(self.recipe["splits"]) + fp8 = tsdec._cutlass_dtype("float8_e4m3fn") + # PDL contract of the main kernel (prims_ts fmha_decode_kernel.py): every CTA acquires its producer + # (griddepcontrol.wait) before any task reads q, lengths or tables, and triggers its dependents + # (launch_dependents) only after its own tail: an active CTA after its last store and TMEM dealloc, an + # inactive split or zero-work CTA right after its wait. The trigger lets the merge launch before the + # slowest CTA ends, so a dependent may read only step-invariant data (weights) before its own + # griddepcontrol.wait, never this step's partial O/LSE, output, q, lengths or tables; the merge + # (minimax_m3_sm107tp4_fullctx_reduce) waits as its first instruction. + cfg = make_decode_config( + headdim=HEAD_DIM, + args={"use_pdl": pdl, **self.recipe.get("config", {})}, + seq_len_q=qlen, + seq_len_kv=min(max_kv_len, POLICY_KV_LEN), + batch_size=nreq, + num_heads_q=HEADS, + num_heads_kv=KV_HEADS, + q_dtype=fp8, + k_dtype=fp8, + v_dtype=fp8, + o_dtype=tsdec._cutlass_dtype("float16"), + qkv_layout="pagedKv", + num_tokens_per_page=PAGE, + storage_tokens_per_page=PAGE, + split_kv_mode="gmem_reduction_with_separate_kernel", + splits_kv=splits, + max_splits_kv=splits, + min_loop_iters_per_split=1, + mask_type="causal" if qlen > 1 else "dense", + auto_tuner=True, + split_kv=True, + ) + _check_layout(cfg, qlen, splits) + with torch.cuda.device(device): + max_active = get_max_active_clusters_for_cluster_size(1) + spec = tsdec._decode_launch_spec_from_config( + cfg, + batch_size=nreq, + num_qo_heads=HEADS, + num_kv_heads=KV_HEADS, + head_dim=HEAD_DIM, + seq_len_q=qlen, + max_active_clusters=int(max_active), + ) + compile_spec = tsdec._make_decode_compile_spec( + spec, + device_index=_index(device), + num_qo_heads=HEADS, + num_kv_heads=KV_HEADS, + head_dim=HEAD_DIM, + page_size=PAGE, + max_kv_len=max_kv_len, + seq_len_q=qlen, + q_dtype_key="float8_e4m3fn", + k_dtype_key="float8_e4m3fn", + v_dtype_key="float8_e4m3fn", + output_dtype_key="float16", + use_packed_q=False, + kv_prefix_mode="dynamic", + kv_lengths_mode="dynamic", + ) + self._main, self._primts_reducer = tsdec._get_compiled_decode(compile_spec) + layout = tsdec._make_decode_workspace_layout( + spec.scratch_shapes, + torch.float16, + use_separate_reduction_kernel=True, + use_split_kv=True, + ) + self._workspace_buffer = torch.empty(layout.total_bytes, device=device, dtype=torch.int8) + self.workspace = tsdec._bind_decode_workspace(self._workspace_buffer, layout) + self.workspace.split_kv_counter.zero_() + self.workspace.cu_seqlens_q.zero_() + self.workspace.attention_sinks.zero_() + rows = HEADS_PER_KV * qlen + if tuple(self.workspace.partial_o.shape) != (nreq, KV_HEADS, splits, rows, HEAD_DIM): + raise RuntimeError( + f"PrimTS split scratch {tuple(self.workspace.partial_o.shape)} is not the " + f"[requests, KV heads, splits, {rows} rows, {HEAD_DIM}] layout the BF16 merge reads" + ) + self.own_reducer = own_reducer + self.q_tokens_per_cta = max(int(cfg.q_tokens_per_cta), 1) + self.tile_kv = int(cfg.tile_size_kv) + self.insts_kv = int(cfg.num_insts_kv) + self.causal = qlen > 1 + shape = (nreq, qlen, HEADS, HEAD_DIM) if qlen > 1 else (nreq, HEADS, HEAD_DIM) + self.staging = torch.empty(shape, dtype=torch.float16, device=device) + + def run( + self, + q: torch.Tensor, + k_pages: torch.Tensor, + v_pages: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + mxfp8: Optional[Tuple[torch.Tensor, torch.Tensor]] = None, + ) -> bool: + """Decodes into output; with mxfp8 (E4M3 rows, UE8M0 scales) and a merge op that takes them, the merge + also writes the MXFP8 copy of output. Returns whether it did.""" + # The kernel takes a compact query; a column view of the fused QKV projection is copied once. + q = q.reshape(self.staging.shape).contiguous() + runtime = self._tsdec._prepare_decode_runtime_unchecked( + q, + (k_pages, v_pages), + None, + output_dtype=torch.float16, + bmm1_scale=sm_scale, + bmm2_scale=1.0, + out=self.staging, + ) + ws = self.workspace + self._main( + runtime.q, + runtime.k_cache, + runtime.v_cache, + runtime.k_sf_cache, + runtime.v_sf_cache, + runtime.out, + seq_lens, + ws.cu_seqlens_q, + block_table, + block_table, # Dense decode constexpr-elides the membership table. + ws.partial_o, + ws.partial_stats, + ws.split_kv_counter, + ws.attention_sinks, + runtime.num_physical_pages, + runtime.k_page_stride, + runtime.k_head_stride, + runtime.k_token_stride, + runtime.v_page_stride, + runtime.v_head_stride, + runtime.v_token_stride, + runtime.bmm1_scale, + runtime.bmm2_scale, + (), + ) + if self.own_reducer: + args = ( + ws.partial_o, + ws.partial_stats, + seq_lens, + output, + self.qlen, + self.q_tokens_per_cta, + self.tile_kv, + self.insts_kv, + self.causal, + self.pdl, + ) + if mxfp8 is not None and emits_mxfp8(): + torch.ops.trtllm.minimax_m3_sm107tp4_fullctx_reduce(*args, mxfp8[0], mxfp8[1]) + return True + torch.ops.trtllm.minimax_m3_sm107tp4_fullctx_reduce(*args) + return False + self._primts_reducer( + runtime.out, + seq_lens, + ws.cu_seqlens_q, + ws.partial_o, + ws.partial_stats, + ws.attention_sinks, + runtime.bmm1_scale, + runtime.bmm2_scale, + (), + ) + output.view(self.staging.shape).copy_(self.staging) + return False + + +def _check_layout(cfg, qlen: int, splits: int) -> None: + """The scratch layout and split ranges the BF16 merge (``fullCtxReduce.cu`` ``activeSplits``) assumes. + + Rows are [token, local head]; a row's split ranges are those of its grouped Q CTA, whose first token is + ``(token // q_tokens_per_cta) * q_tokens_per_cta``; tiles are grouped ``num_insts_kv`` per loop step; no sliding + window, attention sinks or packed Q. + """ + problems = [] + if not (cfg.use_split_kv and cfg.use_separate_reduction_kernel): + problems.append("not split-KV with the separate GMEM reducer") + if int(cfg.splits_kv) != splits or int(cfg.max_splits_kv) != splits: + problems.append(f"splits {cfg.splits_kv}/{cfg.max_splits_kv}, recipe {splits}") + if cfg.use_cluster_smem_reduction or cfg.use_persistent_scheduler: + problems.append("cluster-SMEM reduction or a persistent scheduler") + if cfg.use_variable_seqlens_q or cfg.use_sliding_window_causal or cfg.use_attention_sinks: + problems.append("packed Q, a sliding window or attention sinks") + if int(cfg.heads_q_per_kv) not in (0, HEADS_PER_KV): + problems.append(f"{cfg.heads_q_per_kv} query heads per KV head") + if qlen > 1: + if not (cfg.groups_tokens_heads_q and cfg.uses_nontrivial_grouped_q_layout): + problems.append("multi-token Q without the grouped [token, head] tile layout") + elif int(cfg.q_tokens_per_cta) * HEADS_PER_KV != int(cfg.tile_size_q): + problems.append( + f"{cfg.q_tokens_per_cta} tokens per CTA in a {cfg.tile_size_q}-row tile" + ) + if problems: + raise RuntimeError("PrimTS recipe outside the BF16 merge's layout: " + "; ".join(problems)) + + +def emits_mxfp8() -> bool: + """Whether the loaded merge op can also write the o_proj MXFP8 hand-off (its schema takes q8 / sf).""" + op = getattr(torch.ops.trtllm, "minimax_m3_sm107tp4_fullctx_reduce", None) + return op is not None and "q8" in str(op.default._schema) + + +def has_own_reducer() -> bool: + """Whether this build carries the BF16 split-KV merge op.""" + return hasattr(torch.ops.trtllm, "minimax_m3_sm107tp4_fullctx_reduce") + + +def plan_for( + device: torch.device, qlen: int, nreq: int, own_reducer: bool = True +) -> Optional[FullCtxPlan]: + """The prebuilt plan for this shape, or None.""" + return _plans.get((_index(device), qlen, nreq, own_reducer)) + + +def prebuild( + device: torch.device, + recipes: Mapping[Tuple[int, int], Mapping], + max_kv_len: int, + own_reducer: bool, + pdl: bool, +) -> int: + """Compile the plan of every (query rows per request, requests) in ``recipes``. Call before warmup.""" + built = 0 + for (qlen, nreq), recipe in recipes.items(): + key = (_index(device), qlen, nreq, own_reducer) + with _lock: + if key not in _plans: + _plans[key] = FullCtxPlan(device, qlen, nreq, max_kv_len, recipe, own_reducer, pdl) + built += 1 + return built + + +def kv_page_views(pool: torch.Tensor, subpages_per_slot: int) -> Tuple[torch.Tensor, torch.Tensor]: + """K and V page views of a flat sub-page pool (K of slot s at sub-page s * f, V one later), indexed by slot.""" + return pool[0::subpages_per_slot], pool[1::subpages_per_slot] + + +def _index(device: torch.device) -> int: + return device.index if device.index is not None else torch.cuda.current_device() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_index_score.py new file mode 100644 index 000000000000..51803e7057eb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_index_score.py @@ -0,0 +1,661 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py +"""MiniMax-M3 index decode block scorer (CuTe DSL, Blackwell) with its pre-wait schedule. + +The kernel of ``tensorrt_llm/_torch/attention/kernels/blackwell/minimax_m3_index_decode_score.py`` (the same scores, bit +for bit) with two schedule options this target compiles in: + +* ``prefetch_old``: before the grid-dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that hold only + tokens older than this step (the producer ahead writes only this step's tokens), up to ``prefetch_depth`` blocks per + CTA (0 = all of them). +* ``early_issue``: the TMA warp reads a ``seq_lens`` hint and its first page id before the wait and issues the Q tile + and the first K tile right after it. The tiles are read after the wait, so their contents are final; the block table + is only written between graph replays, so the early page id is the right one whenever the block is valid. A CTA + whose early tiles lie past the request's true length drains them before it exits. + +Every kernel is compiled before the executor's warmup (``layers.prebuild_index_scorer``) through ``IndexScoreRunner`` +and called through this target's ``m3_index_score`` entry. +""" + +from pathlib import Path + +import cutlass +import torch +from cuda.bindings.driver import CUstream +from cutlass import Float8E4M3FN, Float16, Float32, Int32, Int64, Uint32, cute +from cutlass._mlir.dialects import llvm +from cutlass.cute.nvgpu import cpasync, warp +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + EVICT_FIRST, + fp8x4_to_fp16x4, + mma_sync, + simple_tma_copy, +) +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + TRTLLM_ENABLE_PDL, + griddepcontrol_wait, +) + +__all__ = ["IndexDecodeScoreKernel", "IndexScoreRunner"] + + +@dsl_user_op +def _l2_prefetch_bulk(addr: Int64, size: Int32, *, loc=None, ip=None) -> None: + """Prefetch ``size`` contiguous bytes (multiple of 16) at global ``addr`` into L2.""" + llvm.inline_asm( + res=None, + operands_=[Int64(addr).ir_value(loc=loc, ip=ip), Int32(size).ir_value(loc=loc, ip=ip)], + asm_string="cp.async.bulk.prefetch.L2.global [$0], $1;", + constraints="l,r", + has_side_effects=True, + asm_dialect=llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@cute.jit +def _fp8_to_f16_mma_fragments(src: cute.Tensor): + """Split an FP8 ldmatrix fragment into the two FP16 k-fragments MMA wants.""" + src_elems = cute.size(src) + src_u32 = cute.recast_tensor(src, Uint32) + src_f16 = cute.make_rmem_tensor(src_elems, Float16) + src_f16_u32 = cute.recast_tensor(src_f16, Uint32) + # Packed conversion; faster and fewer SASS instructions than + # src.load().to(Float16). + for i in cutlass.range_constexpr(src_elems // 4): + converted = fp8x4_to_fp16x4(src_u32[i]) + src_f16_u32[i * 2] = converted[0] + src_f16_u32[i * 2 + 1] = converted[1] + lower = cute.make_rmem_tensor(src_elems // 2, Float16) + upper = cute.make_rmem_tensor(src_elems // 2, Float16) + + # FP8 ldmatrix yields four consecutive values along K. Split each group + # into the lower two and upper two for the two FP16 MMA k-fragments. + for i in cutlass.range_constexpr(src_elems // 2): + lower[i] = src_f16[(i // 2) * 4 + i % 2] + upper[i] = src_f16[(i // 2) * 4 + 2 + i % 2] + return lower, upper + + +class IndexDecodeScoreKernel: + """Per-block max index score for one decode step. + + Grid is (batch, split_k): CTA (b, s) walks KV blocks + s, s + split_k, s + 2 * split_k, ... of request b and writes each block's + score directly, so no cross-CTA reduction is needed. CTAs whose split_id + exceeds the request's block count exit immediately. + """ + + BLOCK_K = 128 + BAR_MMA = 1 + num_stages = 2 + + def __init__( + self, + dtype: type[cutlass.Numeric], + num_heads: int, + max_decode_query_len: int, + split_k: int, + head_dim: int = 128, + prefetch_old: bool = False, + early_issue: bool = False, + prefetch_depth: int = 0, + ): + # prefetch_old: before the grid dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that + # hold only tokens older than this step (the predecessor writes only this step's tokens). + self.prefetch_old = prefetch_old + # early_issue: the TMA warp reads a seq_lens hint and this CTA's first page id before the grid dependency + # wait, then issues the Q tile and the first K tile right after it, overlapping the post-wait seq_lens load. + # The tiles are read after the wait, so their contents are final; the block table is only written between + # graph replays, so the early page id is the right one whenever the block is valid. A CTA whose early tiles + # turn out to lie past the request's true length drains them before exiting. + self.early_issue = early_issue + # prefetch_depth: blocks per CTA that prefetch_old covers (0 = all of them). Prefetching every older page at + # large batches floods L2; the first few per CTA are the ones whose latency the pipeline cannot hide. + self.prefetch_depth = prefetch_depth + self.dtype = dtype + self.num_heads = num_heads + self.max_decode_query_len = max_decode_query_len + self.split_k = split_k + self.head_dim = head_dim + + def _swizzle_elems(self) -> int: + """Elements spanned by one 128-byte swizzle atom.""" + return 128 * 8 // self.dtype.width + + def _sq_layout(self): + """Composed SMEM layout for the Q tile, shared by the descriptor and the kernel.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_q = self.num_heads * self.max_decode_query_len + layout = cute.make_layout( + (self.max_decode_query_len, self.num_heads, (elems, head_dim // elems)), + stride=(elems, self.max_decode_query_len * elems, (1, block_q * elems)), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + def _sk_layout(self): + """Composed SMEM layout for the pipelined K tiles.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_k = self.BLOCK_K + layout = cute.make_layout( + (1, block_k, (elems, head_dim // elems), self.num_stages), + stride=(0, elems, (1, block_k * elems), block_k * head_dim), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + @cute.jit + def __call__( + self, + gQ: cute.Tensor, # [bs * runtime_decode_query_len, num_heads, head_dim] + gK_cache: cute.Tensor, # [num_pages, page_size, head_dim] + block_table: cute.Tensor, # [bs, max_pages] + score: cute.Tensor, # [num_heads, bs * runtime_decode_query_len, max_pages] + seq_lens: cute.Tensor, # [bs] + stream: CUstream, + ): + num_heads = self.num_heads + head_dim = self.head_dim + MAX_DQL = self.max_decode_query_len + assert num_heads * MAX_DQL <= 32 + + batch = seq_lens.shape[0] + decode_query_len = gQ.shape[0] // batch + grid = (batch, self.split_k, 1) + block = (32 * 5, 1, 1) + + tma_g2s = cpasync.CopyBulkTensorTileG2SOp() + elems = self._swizzle_elems() + + q_tma_atom, q_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gQ, (None, None, elems)), + self._sq_layout(), + (MAX_DQL, num_heads, head_dim), + ) + k_tma_atom, k_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gK_cache, (None, None, elems)), + self._sk_layout(), + (1, self.BLOCK_K, head_dim), + ) + + self.kernel( + q_tma_atom, + q_tma_tensor, + k_tma_atom, + k_tma_tensor, + gK_cache, + block_table, + score, + seq_lens, + decode_query_len, + ).launch(grid=grid, block=block, stream=stream, use_pdl=TRTLLM_ENABLE_PDL) + + @cute.kernel + def kernel( + self, + q_tma_atom: cute.CopyAtom, + q_tma_tensor: cute.Tensor, + k_tma_atom: cute.CopyAtom, + k_tma_tensor: cute.Tensor, + gK_cache: cute.Tensor, + block_table: cute.Tensor, + score: cute.Tensor, + seq_lens: cute.Tensor, + decode_query_len, + ): + batch_id, split_id, _ = cute.arch.block_idx() + _, split_k, _ = cute.arch.grid_dim() + warp_id = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane_id = cute.arch.lane_idx() + + NUM_HEADS = self.num_heads + MAX_DQL = self.max_decode_query_len + BLOCK_Q = NUM_HEADS * MAX_DQL + BLOCK_K = self.BLOCK_K + head_dim = self.head_dim + dtype = self.dtype + MMA_N = 8 + num_stages = self.num_stages + Q_TILES = cute.ceil_div(BLOCK_Q, MMA_N) + EPI_Q = Q_TILES * MMA_N + + sq_layout = self._sq_layout() + sk_layout = self._sk_layout() + + smem = SmemAllocator() + sK = smem.allocate_tensor( + dtype, + sk_layout.outer, + byte_alignment=128, + swizzle=sk_layout.inner, + )[0, None, None, None] + # sQ aliases the first K stage: Q is consumed into registers before the + # first K tile is needed, so the two never overlap in time. + sQ_tma = cute.make_tensor(sK[None, None, 0].iterator, layout=sq_layout.outer) + # TMA sees Q as (query, head, dim) while ldmatrix consumes a flattened Q + # column mode. The target profile keeps the rank-2 view even for + # degenerate shapes such as MAX_DQL == 1. + q_tma_elems = self._swizzle_elems() + sQ = cute.coalesce( + cute.group_modes(sQ_tma, 0, 2), + target_profile=(BLOCK_Q, (q_tma_elems, head_dim // q_tma_elems)), + ) + epi_buffer = smem.allocate_tensor(Float32, cute.make_layout((EPI_Q, 4))) + + tma_full_mbar = smem.allocate_array(Int64, num_stages) + tma_empty_mbar = smem.allocate_array(Int64, num_stages) + + # Barrier init and descriptor prefetch do not depend on the predecessor + # grid, so they are the work PDL can overlap. Every CTA runs them, even + # the ones that own no blocks, since ownership is not known until + # seq_lens is read below. + if warp_id == 0: + with cute.arch.elect_one(): + for i in cutlass.range_constexpr(num_stages): + cute.arch.mbarrier_init(tma_full_mbar + i, 1) + cute.arch.mbarrier_init(tma_empty_mbar + i, 128) + cute.arch.mbarrier_init_fence() + elif warp_id == 1: + cpasync.prefetch_descriptor(q_tma_atom) + cpasync.prefetch_descriptor(k_tma_atom) + cute.arch.sync_threads() + + if cutlass.const_expr(self.prefetch_old): + if warp_id == 4: + with cute.arch.elect_one(): + seqlen0 = seq_lens[batch_id] + num_blocks0 = cute.ceil_div(seqlen0, BLOCK_K) + stable = (seqlen0 - decode_query_len) // BLOCK_K + if stable > num_blocks0: + stable = num_blocks0 + if cutlass.const_expr(self.prefetch_depth > 0): + if stable > split_id + self.prefetch_depth * split_k: + stable = split_id + self.prefetch_depth * split_k + for block_id in range(split_id, stable, split_k): + page = block_table[batch_id, block_id] + _l2_prefetch_bulk( + gK_cache[page, None, None].iterator.toint().to(Int64), + Int32(BLOCK_K * head_dim * (dtype.width // 8)), + ) + + # No explicit launch_dependents: the only consumer is the block selector, + # which reduces over every block this grid writes, so triggering it early + # would leave it resident and spinning in its own wait. The implicit + # trigger as the CTAs exit still gives it the launch overlap. + early_q = False + early_page = Int32(0) + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + hint_blocks = cute.ceil_div(seq_lens[batch_id], BLOCK_K) + if hint_blocks > block_table.shape[1]: + hint_blocks = block_table.shape[1] + if split_id < hint_blocks: + early_q = True + early_page = block_table[batch_id, split_id] + + griddepcontrol_wait() + + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + gQ_early = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar, BLOCK_Q * head_dim * (dtype.width // 8) + ) + simple_tma_copy(q_tma_atom, gQ_early, sQ_tma, tma_full_mbar) + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, BLOCK_K * head_dim * (dtype.width // 8) + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + # seq_lens must be read after the wait. It is a graph-stable buffer that + # on_update_kv_lens patches every step, and num_blocks bounds an unmasked + # block_table load and an unmasked score store, so a stale length is an + # out-of-bounds write rather than just a wrong answer. + seqlen = seq_lens[batch_id] + num_blocks = cute.ceil_div(seqlen, BLOCK_K) + + if split_id < num_blocks: + if warp_id == 4: + # TMA warp + tma_stage = 0 + tma_parity = 1 + first_block = split_id + + if not early_q: + gQ_tile = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, tma_parity) + with cute.arch.elect_one(): + Q_size = BLOCK_Q * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(tma_full_mbar, Q_size) + # TMA bounds-checks rows when the runtime decode_query_len is + # below MAX_DQL; padded Q columns are masked before the stores. + simple_tma_copy(q_tma_atom, gQ_tile, sQ_tma, tma_full_mbar) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + if early_q: + # The first K tile was issued right after the wait. + first_block = split_id + split_k + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + for block_id in range(first_block, num_blocks, split_k): + page_id = block_table[batch_id, block_id] + gK_tile = k_tma_tensor[page_id, None, None] + k_mbar = tma_full_mbar + tma_stage + + cute.arch.mbarrier_wait(tma_empty_mbar + tma_stage, tma_parity) + with cute.arch.elect_one(): + K_size = BLOCK_K * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(k_mbar, K_size) + simple_tma_copy( + k_tma_atom, + gK_tile, + sK[None, None, tma_stage], + k_mbar, + cache_policy=EVICT_FIRST, + ) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + else: + # MMA warps; each handles K[32, head_dim] @ Q[BLOCK_Q, head_dim].T + sK_warp = cute.local_tile(sK, (32, head_dim, num_stages), (warp_id, 0, 0)) + q_start = seqlen - decode_query_len + + elems = 128 // dtype.width # 16B + MMA_K = 32 * 8 // dtype.width # 32B + + # Pre-compute ldmatrix addresses. + # sK loads a [16 x 16B] tile: + # ((16, (16B, 2), 1), (32 / 16, head_dim / 32B, num_stages)) + # sQ loads an [8 x 32B] tile: + # ((8, (16B, 4)), (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = cute.zipped_divide(sK_warp, (16, cute.make_layout((elems, 2)), 1)) + sQ_ldsm = cute.zipped_divide(sQ, (MMA_N, cute.make_layout((elems, 4)))) + + # sK: (16B, (32 / 16, head_dim / 32B, num_stages)) + # sQ: (16B, (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = sK_ldsm[(lane_id % 16, (None, lane_id // 16), 0), None] + sQ_ldsm = sQ_ldsm[(lane_id % MMA_N, (None, lane_id // 8)), None] + + ldsm_op = warp.LdMatrix8x8x16bOp(num_matrices=4) + ldsm_atom = cute.make_copy_atom(ldsm_op, dtype) + + rQ = cute.make_rmem_tensor( + ((elems // 2, 2), head_dim // (MMA_K * 2), Q_TILES), dtype + ) + rK = cute.make_rmem_tensor((elems, 2, head_dim // MMA_K), dtype) + rC = cute.make_rmem_tensor((4, 2, Q_TILES), Float32) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + for q in cutlass.range_constexpr(Q_TILES): + cute.copy(ldsm_atom, sQ_ldsm[None, (q, None)], rQ[None, None, q]) + cute.arch.mbarrier_arrive(tma_empty_mbar) + + tma_stage = 1 % self.num_stages + tma_parity = 0 + if tma_stage == 0: + tma_parity ^= 1 + + if cutlass.const_expr(dtype is Float8E4M3FN): + rQ_f16 = cute.make_rmem_tensor((4, head_dim // MMA_K, Q_TILES, 2), Float16) + q_lower, q_upper = _fp8_to_f16_mma_fragments(rQ) + rQ_f16[None, None, None, 0].store(q_lower.load()) + rQ_f16[None, None, None, 1].store(q_upper.load()) + + for block_id in range(split_id, num_blocks, split_k): + rC.fill(0.0) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar + tma_stage, tma_parity) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + for k in cutlass.range_constexpr(head_dim // MMA_K): + cute.copy( + ldsm_atom, + sK_ldsm[None, (None, k, tma_stage)], + rK[None, None, k], + ) + for m in cutlass.range_constexpr(2): + if cutlass.const_expr(dtype is Float8E4M3FN): + rK_lower, rK_upper = _fp8_to_f16_mma_fragments(rK[None, m, k]) + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK_lower, + rQ_f16[None, k, n, 0], + rC[None, m, n], + ) + rC[None, m, n] = mma_sync( + rK_upper, + rQ_f16[None, k, n, 1], + rC[None, m, n], + ) + else: + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK[None, m, k], + rQ[(None, k % 2), k // 2, n], + rC[None, m, n], + ) + + cute.arch.mbarrier_arrive(tma_empty_mbar + tma_stage) + + k_start = block_id * BLOCK_K + warp_id * 32 + + for q in cutlass.range_constexpr(Q_TILES): + for i in cutlass.range_constexpr(4): + for j in cutlass.range_constexpr(2): + col = q * 8 + (lane_id % 4) * 2 + j + q_local_pos = col % MAX_DQL + q_pos = q_start + q_local_pos + k_pos = k_start + i * 8 + lane_id // 4 + rC[q * 8 + i * 2 + j] = ( + rC[q * 8 + i * 2 + j] if q_pos >= k_pos else float("-inf") + ) + + for q in cutlass.range_constexpr(Q_TILES): + # Thread-local reduction along the BLOCK_K dim. + rScore = cute.make_rmem_tensor(2, Float32) + rScore.fill(float("-inf")) + for i in cutlass.range_constexpr(4): + rScore[0] = cute.arch.fmax(rScore[0], rC[i * 2 + 0 + q * 8]) + rScore[1] = cute.arch.fmax(rScore[1], rC[i * 2 + 1 + q * 8]) + + # Warp reduction among lanes 0, 4, 8, 12, ... + for i in cutlass.range_constexpr(3): + offset = 4 << i + other0 = cute.arch.shuffle_sync_bfly( + rScore[0], offset=offset, mask=-1, mask_and_clamp=31 + ) + other1 = cute.arch.shuffle_sync_bfly( + rScore[1], offset=offset, mask=-1, mask_and_clamp=31 + ) + rScore[0] = cute.arch.fmax(rScore[0], other0) + rScore[1] = cute.arch.fmax(rScore[1], other1) + + if lane_id * 2 < MMA_N: + epi_buffer[q * MMA_N + lane_id * 2 + 0, warp_id] = rScore[0] + epi_buffer[q * MMA_N + lane_id * 2 + 1, warp_id] = rScore[1] + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + head_id = lane_id // MAX_DQL + q_local_pos = lane_id - head_id * MAX_DQL + valid_q = head_id < NUM_HEADS and q_local_pos < decode_query_len + if lane_id < BLOCK_Q and valid_q: + final_score = epi_buffer[lane_id, 0] + for i in cutlass.range_constexpr(1, 4): + final_score = cute.arch.fmax(final_score, epi_buffer[lane_id, i]) + + t = batch_id * decode_query_len + q_local_pos + score[head_id, t, block_id] = final_score + + tma_stage = (tma_stage + 1) % self.num_stages + if tma_stage == 0: + tma_parity ^= 1 + else: + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + # Early tiles past the request's true length: let them land before the CTA exits. + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.mbarrier_wait(tma_full_mbar + 1 % num_stages, 0) + + +class IndexScoreRunner: + """Compiled ``IndexDecodeScoreKernel`` variants, keyed by (index dtype, index heads, decode query length, head dim, + split_k, early issue). Batch, query-token count, page count and block-table width stay symbolic, so one compile + serves every decode step of its key. ``prefetch_old`` is always on; ``early`` also caps the prefetch at one block + per CTA.""" + + kernel_cache: dict = {} + SUPPORTED_HEAD_DIM = 128 # capacity: fixed: the checkpoint's index head dim + SUPPORTED_PAGE_SIZE = IndexDecodeScoreKernel.BLOCK_K + # BLOCK_Q must fit one warp's worth of epilogue lanes. + MAX_BLOCK_Q = 32 # capacity: fixed: the scorer's query tile + EARLY_PREFETCH_DEPTH = 1 + + @classmethod + def is_supported( + cls, + *, + q_dtype: torch.dtype, + num_heads: int, + head_dim: int, + page_size: int, + max_decode_query_len: int, + ) -> bool: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + from tensorrt_llm._utils import is_sm_100f + + return ( + is_sm_100f() + and q_dtype in TORCH_TO_CUTE_DTYPE + and head_dim == cls.SUPPORTED_HEAD_DIM + and page_size == cls.SUPPORTED_PAGE_SIZE + and num_heads * max_decode_query_len <= cls.MAX_BLOCK_Q + and max_decode_query_len >= 1 + ) + + @classmethod + def _compile( + cls, + q_dtype: torch.dtype, + num_heads: int, + max_decode_query_len: int, + head_dim: int, + split_k: int, + early: bool, + ) -> None: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + + key = (q_dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + if key in cls.kernel_cache: + return + cute_dtype = TORCH_TO_CUTE_DTYPE[q_dtype] + page_size = cls.SUPPORTED_PAGE_SIZE + sym_total_tokens = cute.sym_int() + sym_batch = cute.sym_int() + + # 16-element divisibility on every non-innermost stride is what the TMA descriptors for Q and K assume. + def _sym_stride(): + return cute.sym_int64(divisibility=16) + + q_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (sym_total_tokens, num_heads, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + # The index-K pool may be coalesced with the main K/V cache (per-page stride above page_size * head_dim). + k_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (cute.sym_int(), page_size, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + bt_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch, cute.sym_int()), stride_order=(1, 0) + ) + # The target passes a transposed view of the [heads, blocks, tokens] selector buffer. + score_fake = cute.runtime.make_fake_tensor( + cutlass.Float32, + (num_heads, sym_total_tokens, cute.sym_int()), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + sl_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch,), stride_order=(0,) + ) + fake_stream = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True) + kernel = IndexDecodeScoreKernel( + cute_dtype, + num_heads, + max_decode_query_len, + split_k, + head_dim, + prefetch_old=True, + early_issue=early, + prefetch_depth=cls.EARLY_PREFETCH_DEPTH if early else 0, + ) + cls.kernel_cache[key] = cute_aot.compile_or_load( + f"m3_index_score {key}", + lambda: cute.compile( + kernel, + q_fake, + k_fake, + bt_fake, + score_fake, + sl_fake, + fake_stream, + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_index_score_t2sc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_index_score_t2sc.py new file mode 100644 index 000000000000..ebb916cab068 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_index_score_t2sc.py @@ -0,0 +1,702 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py +"""MiniMax-M3 index decode block scorer (CuTe DSL, Blackwell) with its pre-wait schedule: T2SC, this target's copy of +the sm_107 TP2 / TP8 targets' scorer (``modeling.INDEX_SCORE_T2SC``). The sm_103-derived ``kernels/m3_index_score`` +stays the kill path. + +The kernel of ``tensorrt_llm/_torch/attention/kernels/blackwell/minimax_m3_index_decode_score.py`` (the same scores, bit +for bit) with three schedule options this target compiles in: + +* ``prefetch_old``: before the grid-dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that hold only + tokens older than this step (the producer ahead writes only this step's tokens), up to ``prefetch_depth`` blocks per + CTA (0 = all of them). +* ``early_issue``: the TMA warp reads a ``seq_lens`` hint and its first page id before the wait and issues the Q tile + and the first K tile right after it. The tiles are read after the wait, so their contents are final; the block table + is only written between graph replays, so the early page id is the right one whenever the block is valid. A CTA + whose early tiles lie past the request's true length drains them before it exits. +* ``prestage_old`` (with ``early_issue``, ``modeling.INDEX_SCORE_T2SC_PRESTAGE_OLD``): when the CTA's first block holds + only tokens older than this step (block < (seq_lens hint - decode query length) / 128), its K tile is TMA-loaded into + its stage before the wait; Q (and a first block that is not all old) is still loaded after it. + +Every kernel is compiled before the executor's warmup (``layers.prebuild_index_scorer_t2sc``) through +``IndexScoreRunner`` and called through this target's ``m3_index_score_t2sc`` entry. +""" + +from pathlib import Path + +import cutlass +import torch +from cuda.bindings.driver import CUstream +from cutlass import Float8E4M3FN, Float16, Float32, Int32, Int64, Uint32, cute +from cutlass._mlir.dialects import llvm +from cutlass.cute.nvgpu import cpasync, warp +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + EVICT_FIRST, + fp8x4_to_fp16x4, + mma_sync, + simple_tma_copy, +) +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + TRTLLM_ENABLE_PDL, + griddepcontrol_launch_dependents, + griddepcontrol_wait, +) + +__all__ = ["IndexDecodeScoreKernel", "IndexScoreRunner"] + + +@dsl_user_op +def _l2_prefetch_bulk(addr: Int64, size: Int32, *, loc=None, ip=None) -> None: + """Prefetch ``size`` contiguous bytes (multiple of 16) at global ``addr`` into L2.""" + llvm.inline_asm( + res=None, + operands_=[Int64(addr).ir_value(loc=loc, ip=ip), Int32(size).ir_value(loc=loc, ip=ip)], + asm_string="cp.async.bulk.prefetch.L2.global [$0], $1;", + constraints="l,r", + has_side_effects=True, + asm_dialect=llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@cute.jit +def _fp8_to_f16_mma_fragments(src: cute.Tensor): + """Split an FP8 ldmatrix fragment into the two FP16 k-fragments MMA wants.""" + src_elems = cute.size(src) + src_u32 = cute.recast_tensor(src, Uint32) + src_f16 = cute.make_rmem_tensor(src_elems, Float16) + src_f16_u32 = cute.recast_tensor(src_f16, Uint32) + # Packed conversion; faster and fewer SASS instructions than + # src.load().to(Float16). + for i in cutlass.range_constexpr(src_elems // 4): + converted = fp8x4_to_fp16x4(src_u32[i]) + src_f16_u32[i * 2] = converted[0] + src_f16_u32[i * 2 + 1] = converted[1] + lower = cute.make_rmem_tensor(src_elems // 2, Float16) + upper = cute.make_rmem_tensor(src_elems // 2, Float16) + + # FP8 ldmatrix yields four consecutive values along K. Split each group + # into the lower two and upper two for the two FP16 MMA k-fragments. + for i in cutlass.range_constexpr(src_elems // 2): + lower[i] = src_f16[(i // 2) * 4 + i % 2] + upper[i] = src_f16[(i // 2) * 4 + 2 + i % 2] + return lower, upper + + +class IndexDecodeScoreKernel: + """Per-block max index score for one decode step. + + Grid is (batch, split_k): CTA (b, s) walks KV blocks + s, s + split_k, s + 2 * split_k, ... of request b and writes each block's + score directly, so no cross-CTA reduction is needed. CTAs whose split_id + exceeds the request's block count exit immediately. + """ + + BLOCK_K = 128 # capacity: fixed: the MSA block size of the checkpoint + BAR_MMA = 1 + num_stages = 2 + + def __init__( + self, + dtype: type[cutlass.Numeric], + num_heads: int, + max_decode_query_len: int, + split_k: int, + head_dim: int = 128, + prefetch_old: bool = False, + early_issue: bool = False, + prefetch_depth: int = 0, + prestage_old: bool = False, + early_trigger: bool = False, + ): + # prefetch_old: before the grid dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that + # hold only tokens older than this step (the predecessor writes only this step's tokens). + self.prefetch_old = prefetch_old + # early_issue: the TMA warp reads a seq_lens hint and this CTA's first page id before the grid dependency + # wait, then issues the Q tile and the first K tile right after it, overlapping the post-wait seq_lens load. + # The tiles are read after the wait, so their contents are final; the block table is only written between + # graph replays, so the early page id is the right one whenever the block is valid. A CTA whose early tiles + # turn out to lie past the request's true length drains them before exiting. + self.early_issue = early_issue + # prefetch_depth: blocks per CTA that prefetch_old covers (0 = all of them). Prefetching every older page at + # large batches floods L2; the first few per CTA are the ones whose latency the pipeline cannot hide. + self.prefetch_depth = prefetch_depth + # prestage_old (with early_issue): when this CTA's first block holds only tokens older than this step, its K + # tile is TMA-loaded into its stage BEFORE the grid dependency wait (the predecessor writes only this step's + # tokens; the block table and the seq_lens hint are final before the grid starts). Q stays after the wait. + self.prestage_old = prestage_old and early_issue + # early_trigger (on for this target): release the dependent selector right after this CTA's + # setup, so its launch and residency overlap the scoring; it still waits for this whole grid. + self.early_trigger = early_trigger + self.dtype = dtype + self.num_heads = num_heads + self.max_decode_query_len = max_decode_query_len + self.split_k = split_k + self.head_dim = head_dim + + def _swizzle_elems(self) -> int: + """Elements spanned by one 128-byte swizzle atom.""" + return 128 * 8 // self.dtype.width + + def _sq_layout(self): + """Composed SMEM layout for the Q tile, shared by the descriptor and the kernel.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_q = self.num_heads * self.max_decode_query_len + layout = cute.make_layout( + (self.max_decode_query_len, self.num_heads, (elems, head_dim // elems)), + stride=(elems, self.max_decode_query_len * elems, (1, block_q * elems)), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + def _sk_layout(self): + """Composed SMEM layout for the pipelined K tiles.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_k = self.BLOCK_K + layout = cute.make_layout( + (1, block_k, (elems, head_dim // elems), self.num_stages), + stride=(0, elems, (1, block_k * elems), block_k * head_dim), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + @cute.jit + def __call__( + self, + gQ: cute.Tensor, # [bs * runtime_decode_query_len, num_heads, head_dim] + gK_cache: cute.Tensor, # [num_pages, page_size, head_dim] + block_table: cute.Tensor, # [bs, max_pages] + score: cute.Tensor, # [num_heads, bs * runtime_decode_query_len, max_pages] + seq_lens: cute.Tensor, # [bs] + stream: CUstream, + ): + num_heads = self.num_heads + head_dim = self.head_dim + MAX_DQL = self.max_decode_query_len + assert num_heads * MAX_DQL <= 32 + + batch = seq_lens.shape[0] + decode_query_len = gQ.shape[0] // batch + grid = (batch, self.split_k, 1) + block = (32 * 5, 1, 1) + + tma_g2s = cpasync.CopyBulkTensorTileG2SOp() + elems = self._swizzle_elems() + + q_tma_atom, q_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gQ, (None, None, elems)), + self._sq_layout(), + (MAX_DQL, num_heads, head_dim), + ) + k_tma_atom, k_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gK_cache, (None, None, elems)), + self._sk_layout(), + (1, self.BLOCK_K, head_dim), + ) + + self.kernel( + q_tma_atom, + q_tma_tensor, + k_tma_atom, + k_tma_tensor, + gK_cache, + block_table, + score, + seq_lens, + decode_query_len, + ).launch(grid=grid, block=block, stream=stream, use_pdl=TRTLLM_ENABLE_PDL) + + @cute.kernel + def kernel( + self, + q_tma_atom: cute.CopyAtom, + q_tma_tensor: cute.Tensor, + k_tma_atom: cute.CopyAtom, + k_tma_tensor: cute.Tensor, + gK_cache: cute.Tensor, + block_table: cute.Tensor, + score: cute.Tensor, + seq_lens: cute.Tensor, + decode_query_len, + ): + batch_id, split_id, _ = cute.arch.block_idx() + _, split_k, _ = cute.arch.grid_dim() + warp_id = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane_id = cute.arch.lane_idx() + + NUM_HEADS = self.num_heads + MAX_DQL = self.max_decode_query_len + BLOCK_Q = NUM_HEADS * MAX_DQL + BLOCK_K = self.BLOCK_K + head_dim = self.head_dim + dtype = self.dtype + MMA_N = 8 + num_stages = self.num_stages + Q_TILES = cute.ceil_div(BLOCK_Q, MMA_N) + EPI_Q = Q_TILES * MMA_N + + sq_layout = self._sq_layout() + sk_layout = self._sk_layout() + + smem = SmemAllocator() + sK = smem.allocate_tensor( + dtype, + sk_layout.outer, + byte_alignment=128, + swizzle=sk_layout.inner, + )[0, None, None, None] + # sQ aliases the first K stage: Q is consumed into registers before the + # first K tile is needed, so the two never overlap in time. + sQ_tma = cute.make_tensor(sK[None, None, 0].iterator, layout=sq_layout.outer) + # TMA sees Q as (query, head, dim) while ldmatrix consumes a flattened Q + # column mode. The target profile keeps the rank-2 view even for + # degenerate shapes such as MAX_DQL == 1. + q_tma_elems = self._swizzle_elems() + sQ = cute.coalesce( + cute.group_modes(sQ_tma, 0, 2), + target_profile=(BLOCK_Q, (q_tma_elems, head_dim // q_tma_elems)), + ) + epi_buffer = smem.allocate_tensor(Float32, cute.make_layout((EPI_Q, 4))) + + tma_full_mbar = smem.allocate_array(Int64, num_stages) + tma_empty_mbar = smem.allocate_array(Int64, num_stages) + + # Barrier init and descriptor prefetch do not depend on the predecessor + # grid, so they are the work PDL can overlap. Every CTA runs them, even + # the ones that own no blocks, since ownership is not known until + # seq_lens is read below. + if warp_id == 0: + with cute.arch.elect_one(): + for i in cutlass.range_constexpr(num_stages): + cute.arch.mbarrier_init(tma_full_mbar + i, 1) + cute.arch.mbarrier_init(tma_empty_mbar + i, 128) + cute.arch.mbarrier_init_fence() + elif warp_id == 1: + cpasync.prefetch_descriptor(q_tma_atom) + cpasync.prefetch_descriptor(k_tma_atom) + cute.arch.sync_threads() + if cutlass.const_expr(self.early_trigger): + griddepcontrol_launch_dependents() + + if cutlass.const_expr(self.prefetch_old): + if warp_id == 4: + with cute.arch.elect_one(): + seqlen0 = seq_lens[batch_id] + num_blocks0 = cute.ceil_div(seqlen0, BLOCK_K) + stable = (seqlen0 - decode_query_len) // BLOCK_K + if stable > num_blocks0: + stable = num_blocks0 + if cutlass.const_expr(self.prefetch_depth > 0): + if stable > split_id + self.prefetch_depth * split_k: + stable = split_id + self.prefetch_depth * split_k + for block_id in range(split_id, stable, split_k): + page = block_table[batch_id, block_id] + _l2_prefetch_bulk( + gK_cache[page, None, None].iterator.toint().to(Int64), + Int32(BLOCK_K * head_dim * (dtype.width // 8)), + ) + + # Without early_trigger the selector launches at this grid's implicit trigger (the CTAs' exit). With it, the + # selector is already resident in its own grid-dependency wait, which still returns only once every block + # this grid writes is complete. + early_q = False + early_k = False + early_page = Int32(0) + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + seqlen_hint = seq_lens[batch_id] + hint_blocks = cute.ceil_div(seqlen_hint, BLOCK_K) + if hint_blocks > block_table.shape[1]: + hint_blocks = block_table.shape[1] + if split_id < hint_blocks: + early_q = True + early_page = block_table[batch_id, split_id] + if cutlass.const_expr(self.prestage_old): + # Stage the first K tile now when every token of the block predates this step. + if split_id < (seqlen_hint - decode_query_len) // BLOCK_K: + early_k = True + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, + BLOCK_K * head_dim * (dtype.width // 8), + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + griddepcontrol_wait() + + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + gQ_early = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar, BLOCK_Q * head_dim * (dtype.width // 8) + ) + simple_tma_copy(q_tma_atom, gQ_early, sQ_tma, tma_full_mbar) + if not early_k: + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, + BLOCK_K * head_dim * (dtype.width // 8), + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + # The length that bounds this grid's work is read after the wait (the + # pre-wait reads above are prefetch / early-issue hints only, clamped to + # the table width). seq_lens is a graph-stable buffer that on_update_kv_lens + # patches every step, and num_blocks bounds an unmasked block_table load + # and an unmasked score store, so a stale length is an out-of-bounds write + # rather than just a wrong answer. + seqlen = seq_lens[batch_id] + num_blocks = cute.ceil_div(seqlen, BLOCK_K) + + if split_id < num_blocks: + if warp_id == 4: + # TMA warp + tma_stage = 0 + tma_parity = 1 + first_block = split_id + + if not early_q: + gQ_tile = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, tma_parity) + with cute.arch.elect_one(): + Q_size = BLOCK_Q * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(tma_full_mbar, Q_size) + # TMA bounds-checks rows when the runtime decode_query_len is + # below MAX_DQL; padded Q columns are masked before the stores. + simple_tma_copy(q_tma_atom, gQ_tile, sQ_tma, tma_full_mbar) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + if early_q: + # The first K tile was issued right after the wait. + first_block = split_id + split_k + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + for block_id in range(first_block, num_blocks, split_k): + page_id = block_table[batch_id, block_id] + gK_tile = k_tma_tensor[page_id, None, None] + k_mbar = tma_full_mbar + tma_stage + + cute.arch.mbarrier_wait(tma_empty_mbar + tma_stage, tma_parity) + with cute.arch.elect_one(): + K_size = BLOCK_K * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(k_mbar, K_size) + simple_tma_copy( + k_tma_atom, + gK_tile, + sK[None, None, tma_stage], + k_mbar, + cache_policy=EVICT_FIRST, + ) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + else: + # MMA warps; each handles K[32, head_dim] @ Q[BLOCK_Q, head_dim].T + sK_warp = cute.local_tile(sK, (32, head_dim, num_stages), (warp_id, 0, 0)) + q_start = seqlen - decode_query_len + + elems = 128 // dtype.width # 16B + MMA_K = 32 * 8 // dtype.width # 32B + + # Pre-compute ldmatrix addresses. + # sK loads a [16 x 16B] tile: + # ((16, (16B, 2), 1), (32 / 16, head_dim / 32B, num_stages)) + # sQ loads an [8 x 32B] tile: + # ((8, (16B, 4)), (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = cute.zipped_divide(sK_warp, (16, cute.make_layout((elems, 2)), 1)) + sQ_ldsm = cute.zipped_divide(sQ, (MMA_N, cute.make_layout((elems, 4)))) + + # sK: (16B, (32 / 16, head_dim / 32B, num_stages)) + # sQ: (16B, (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = sK_ldsm[(lane_id % 16, (None, lane_id // 16), 0), None] + sQ_ldsm = sQ_ldsm[(lane_id % MMA_N, (None, lane_id // 8)), None] + + ldsm_op = warp.LdMatrix8x8x16bOp(num_matrices=4) + ldsm_atom = cute.make_copy_atom(ldsm_op, dtype) + + rQ = cute.make_rmem_tensor( + ((elems // 2, 2), head_dim // (MMA_K * 2), Q_TILES), dtype + ) + rK = cute.make_rmem_tensor((elems, 2, head_dim // MMA_K), dtype) + rC = cute.make_rmem_tensor((4, 2, Q_TILES), Float32) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + for q in cutlass.range_constexpr(Q_TILES): + cute.copy(ldsm_atom, sQ_ldsm[None, (q, None)], rQ[None, None, q]) + cute.arch.mbarrier_arrive(tma_empty_mbar) + + tma_stage = 1 % self.num_stages + tma_parity = 0 + if tma_stage == 0: + tma_parity ^= 1 + + if cutlass.const_expr(dtype is Float8E4M3FN): + rQ_f16 = cute.make_rmem_tensor((4, head_dim // MMA_K, Q_TILES, 2), Float16) + q_lower, q_upper = _fp8_to_f16_mma_fragments(rQ) + rQ_f16[None, None, None, 0].store(q_lower.load()) + rQ_f16[None, None, None, 1].store(q_upper.load()) + + for block_id in range(split_id, num_blocks, split_k): + rC.fill(0.0) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar + tma_stage, tma_parity) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + for k in cutlass.range_constexpr(head_dim // MMA_K): + cute.copy( + ldsm_atom, + sK_ldsm[None, (None, k, tma_stage)], + rK[None, None, k], + ) + for m in cutlass.range_constexpr(2): + if cutlass.const_expr(dtype is Float8E4M3FN): + rK_lower, rK_upper = _fp8_to_f16_mma_fragments(rK[None, m, k]) + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK_lower, + rQ_f16[None, k, n, 0], + rC[None, m, n], + ) + rC[None, m, n] = mma_sync( + rK_upper, + rQ_f16[None, k, n, 1], + rC[None, m, n], + ) + else: + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK[None, m, k], + rQ[(None, k % 2), k // 2, n], + rC[None, m, n], + ) + + cute.arch.mbarrier_arrive(tma_empty_mbar + tma_stage) + + k_start = block_id * BLOCK_K + warp_id * 32 + + for q in cutlass.range_constexpr(Q_TILES): + for i in cutlass.range_constexpr(4): + for j in cutlass.range_constexpr(2): + col = q * 8 + (lane_id % 4) * 2 + j + q_local_pos = col % MAX_DQL + q_pos = q_start + q_local_pos + k_pos = k_start + i * 8 + lane_id // 4 + rC[q * 8 + i * 2 + j] = ( + rC[q * 8 + i * 2 + j] if q_pos >= k_pos else float("-inf") + ) + + for q in cutlass.range_constexpr(Q_TILES): + # Thread-local reduction along the BLOCK_K dim. + rScore = cute.make_rmem_tensor(2, Float32) + rScore.fill(float("-inf")) + for i in cutlass.range_constexpr(4): + rScore[0] = cute.arch.fmax(rScore[0], rC[i * 2 + 0 + q * 8]) + rScore[1] = cute.arch.fmax(rScore[1], rC[i * 2 + 1 + q * 8]) + + # Warp reduction among lanes 0, 4, 8, 12, ... + for i in cutlass.range_constexpr(3): + offset = 4 << i + other0 = cute.arch.shuffle_sync_bfly( + rScore[0], offset=offset, mask=-1, mask_and_clamp=31 + ) + other1 = cute.arch.shuffle_sync_bfly( + rScore[1], offset=offset, mask=-1, mask_and_clamp=31 + ) + rScore[0] = cute.arch.fmax(rScore[0], other0) + rScore[1] = cute.arch.fmax(rScore[1], other1) + + if lane_id * 2 < MMA_N: + epi_buffer[q * MMA_N + lane_id * 2 + 0, warp_id] = rScore[0] + epi_buffer[q * MMA_N + lane_id * 2 + 1, warp_id] = rScore[1] + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + head_id = lane_id // MAX_DQL + q_local_pos = lane_id - head_id * MAX_DQL + valid_q = head_id < NUM_HEADS and q_local_pos < decode_query_len + if lane_id < BLOCK_Q and valid_q: + final_score = epi_buffer[lane_id, 0] + for i in cutlass.range_constexpr(1, 4): + final_score = cute.arch.fmax(final_score, epi_buffer[lane_id, i]) + + t = batch_id * decode_query_len + q_local_pos + score[head_id, t, block_id] = final_score + + tma_stage = (tma_stage + 1) % self.num_stages + if tma_stage == 0: + tma_parity ^= 1 + else: + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + # Early tiles past the request's true length: let them land before the CTA exits. + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.mbarrier_wait(tma_full_mbar + 1 % num_stages, 0) + + +class IndexScoreRunner: + """Compiled ``IndexDecodeScoreKernel`` variants, keyed by (index dtype, index heads, decode query length, head dim, + split_k, early issue). Batch, query-token count, page count and block-table width stay symbolic, so one compile + serves every decode step of its key. ``prefetch_old`` is always on; ``early`` also caps the prefetch at one block + per CTA.""" + + kernel_cache: dict = {} + SUPPORTED_HEAD_DIM = 128 # capacity: fixed: the checkpoint's index head dim + SUPPORTED_PAGE_SIZE = IndexDecodeScoreKernel.BLOCK_K + # BLOCK_Q must fit one warp's worth of epilogue lanes. + MAX_BLOCK_Q = 32 # capacity: fixed: the scorer's query tile + EARLY_PREFETCH_DEPTH = 1 + + @classmethod + def is_supported( + cls, + *, + q_dtype: torch.dtype, + num_heads: int, + head_dim: int, + page_size: int, + max_decode_query_len: int, + ) -> bool: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + from tensorrt_llm._utils import is_sm_100f + + return ( + is_sm_100f() + and q_dtype in TORCH_TO_CUTE_DTYPE + and head_dim == cls.SUPPORTED_HEAD_DIM + and page_size == cls.SUPPORTED_PAGE_SIZE + and num_heads * max_decode_query_len <= cls.MAX_BLOCK_Q + and max_decode_query_len >= 1 + ) + + @classmethod + def _compile( + cls, + q_dtype: torch.dtype, + num_heads: int, + max_decode_query_len: int, + head_dim: int, + split_k: int, + early: bool, + ) -> None: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + + key = (q_dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + if key in cls.kernel_cache: + return + cute_dtype = TORCH_TO_CUTE_DTYPE[q_dtype] + page_size = cls.SUPPORTED_PAGE_SIZE + sym_total_tokens = cute.sym_int() + sym_batch = cute.sym_int() + + # 16-element divisibility on every non-innermost stride is what the TMA descriptors for Q and K assume. + def _sym_stride(): + return cute.sym_int64(divisibility=16) + + q_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (sym_total_tokens, num_heads, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + # The index-K pool may be coalesced with the main K/V cache (per-page stride above page_size * head_dim). + k_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (cute.sym_int(), page_size, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + bt_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch, cute.sym_int()), stride_order=(1, 0) + ) + # The target passes a transposed view of the [heads, blocks, tokens] selector buffer. + score_fake = cute.runtime.make_fake_tensor( + cutlass.Float32, + (num_heads, sym_total_tokens, cute.sym_int()), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + sl_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch,), stride_order=(0,) + ) + fake_stream = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True) + kernel = IndexDecodeScoreKernel( + cute_dtype, + num_heads, + max_decode_query_len, + split_k, + head_dim, + prefetch_old=True, + early_issue=early, + prefetch_depth=cls.EARLY_PREFETCH_DEPTH if early else 0, + prestage_old=early, + early_trigger=True, + ) + cls.kernel_cache[key] = cute_aot.compile_or_load( + f"m3_index_score_t2sc {key}", + lambda: cute.compile( + kernel, + q_fake, + k_fake, + bt_fake, + score_fake, + sl_fake, + fake_stream, + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_lowm_splitk.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_lowm_splitk.py new file mode 100644 index 000000000000..53cdbd383ad8 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_lowm_splitk.py @@ -0,0 +1,1259 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-License-Identifier: Apache-2.0 +"""Blackwell low-M BF16/FP16 GEMM with an in-kernel cluster split-K reduction. + +This target's copy of FlashInfer's ``flashinfer/gemm/kernels/dense_bf16_gemm_sm100_splitk.py`` (the Eagle3 +drafter's low-M linears, ``kernels/m3_draft_lowm``): the same kernel and the same split order at every tactic, +bit for bit FlashInfer's output (spec/test_modeling_v2_m3_lowm_splitk.py). The one change: the owner rank's final +cast and store is one method, ``_store_tile``, the seam for the drafter-boundary push epilogue and, on the A side, the +SwiGLU prologue that follow. + +Each cluster rank accumulates an exact K slice in FP32. Peers publish partials +to rank 0 through DSMEM; rank 0 reduces, casts, and stores once. The public +``A[M, K] @ B[K, N]`` problem is swapped internally, so tile dimensions below +use kernel coordinates: kernel-M carries public N and kernel-N carries public M. +""" + +from __future__ import annotations + +import dataclasses +from pathlib import Path + +import cuda.bindings.driver as _cuda +import cutlass +import cutlass.cute as cute +import cutlass.utils as utils +import cutlass.utils.blackwell_helpers as sm100_utils +from cutlass import Int32 +from cutlass._mlir.dialects import llvm +from cutlass.cute import experimental as cute_ext +from cutlass.cute.nvgpu import tcgen05 +from cutlass.cute.runtime import from_dlpack +from cutlass.cutlass_dsl import T, dsl_user_op + +from tensorrt_llm._torch import cute_aot + +#: Per-CTA SMEM capacity reported by CuTeDSL on SM100/SM103. +_SMEM_CAPACITY_BYTES = utils.get_smem_capacity_in_bytes("sm_100") + +#: K extent of one CTA tile. +_CTA_K = 128 # capacity: fixed: a kernel tile constant + +#: Kernel-M tiles; 64 increases CTA count for low-M decode shapes. +_SUPPORTED_MMA_M = (64, 128) + +#: Kernel-N carries public M, which is limited to 32. +_SUPPORTED_MMA_N = (8, 16, 32) + +#: Physical cluster-K sizes; split 1 compiles out the DSMEM path. +_SUPPORTED_SPLIT_K = (1, 2, 3, 4) + +#: Largest public M this low-M policy serves. +_MAX_M = 32 # capacity: fixed: the copied FlashInfer kernel's row bound + +#: Bytes per FP32 partial exchanged through DSMEM. +_FP32_BYTES = 4 + +#: DSMEM mailbox base alignment, in bytes. +_MAILBOX_ALIGN_BYTES = 128 + +#: Size and alignment of one mbarrier, in bytes. +_MBARRIER_BYTES = 8 + +#: Size and alignment of the TMEM base pointer slot. +_TMEM_POINTER_BYTES = 4 + +#: Bytes per BF16/FP16 element. +_AB_ELEMENT_BYTES = 2 + +#: Alignment of the A/B shared-memory buffers. +_AB_BUFFER_ALIGN_BYTES = 1024 # capacity: fixed: an alignment + +#: A/B pipeline stage bounds. One stage is useful only for a single K tile. +_MIN_AB_STAGES = 1 # capacity: fixed: pipeline stages +_DEFAULT_MIN_AB_STAGES = 2 +_MAX_AB_STAGES = 12 # capacity: fixed: pipeline stages + + +@dataclasses.dataclass(frozen=True) +class SplitKTactic: + """One specialization; mma_m carries public N and mma_n carries public M.""" + + mma_m: int + mma_n: int + split_k: int + ab_stages: int + + +def _align_up(value: int, alignment: int) -> int: + return ((value + alignment - 1) // alignment) * alignment + + +def _smem_bytes( + tactic: SplitKTactic, + ab_stages: int, +) -> int: + """Mirror the device allocator's shared-memory layout.""" + cursor = ( + _align_up( + tactic.mma_m * _CTA_K * _AB_ELEMENT_BYTES * ab_stages, + _AB_BUFFER_ALIGN_BYTES, + ) + + tactic.mma_n * _CTA_K * _AB_ELEMENT_BYTES * ab_stages + ) + + cursor = _align_up(cursor, _MBARRIER_BYTES) + cursor += 2 * ab_stages * _MBARRIER_BYTES + cursor += 3 * _MBARRIER_BYTES + cursor = _align_up(cursor, _TMEM_POINTER_BYTES) + cursor += _TMEM_POINTER_BYTES + + if tactic.split_k == 1: + return cursor + + return ( + _align_up( + _align_up(cursor, _MAILBOX_ALIGN_BYTES) + + (tactic.split_k - 1) * tactic.mma_m * tactic.mma_n * _FP32_BYTES, + _MBARRIER_BYTES, + ) + + _MBARRIER_BYTES + ) + + +def _max_ab_stages_for( + tactic: SplitKTactic, + smem_capacity: int, +) -> int: + return next( + ( + stages + for stages in range(_MAX_AB_STAGES, -1, -1) + if _smem_bytes(tactic, stages) <= smem_capacity + ), + 0, + ) + + +def validate_tactic( + tactic: SplitKTactic, + m: int, + n: int, + k: int, + *, + smem_capacity: int = _SMEM_CAPACITY_BYTES, +) -> None: + """Reject a tactic that cannot serve ``(m, n, k)``.""" + if tactic.mma_m not in _SUPPORTED_MMA_M: + raise ValueError(f"unsupported mma_m={tactic.mma_m}") + if tactic.mma_n not in _SUPPORTED_MMA_N: + raise ValueError(f"unsupported mma_n={tactic.mma_n}") + if tactic.split_k not in _SUPPORTED_SPLIT_K: + raise ValueError(f"unsupported split_k={tactic.split_k}") + if not _MIN_AB_STAGES <= tactic.ab_stages <= _MAX_AB_STAGES: + raise ValueError( + f"ab_stages must be in [{_MIN_AB_STAGES}, {_MAX_AB_STAGES}], got {tactic.ab_stages}" + ) + if not 1 <= m <= _MAX_M: + raise ValueError(f"this low-M policy requires 1 <= M <= {_MAX_M}, got {m}") + if n <= 0: + raise ValueError(f"N must be positive, got {n}") + if k <= 0 or k % _CTA_K or (k // _CTA_K) % tactic.split_k: + raise ValueError( + f"K={k} with CTA_K={_CTA_K} does not divide evenly across split_k={tactic.split_k}" + ) + smem_bytes = _smem_bytes(tactic, tactic.ab_stages) + if smem_bytes > smem_capacity: + raise ValueError( + f"tactic {tactic} needs {smem_bytes} B of shared memory but only " + f"{smem_capacity} B are available; max ab_stages is " + f"{_max_ab_stages_for(tactic, smem_capacity)}" + ) + + +def autotune_tactics( + m: int, + n: int, + k: int, + *, + smem_capacity: int = _SMEM_CAPACITY_BYTES, +) -> list[SplitKTactic]: + """Return valid tactics in the shape-specific stage window.""" + tactics: list[SplitKTactic] = [] + for mma_m in _SUPPORTED_MMA_M: + for mma_n in _SUPPORTED_MMA_N: + for split_k in _SUPPORTED_SPLIT_K: + min_stages = _MIN_AB_STAGES if k == _CTA_K else _DEFAULT_MIN_AB_STAGES + base = SplitKTactic(mma_m, mma_n, split_k, min_stages) + try: + validate_tactic(base, m, n, k, smem_capacity=smem_capacity) + except ValueError: + continue + max_stages = _max_ab_stages_for(base, smem_capacity) + # Short K favors shallow pipelines; long K stays near the cap. + tactics.extend( + dataclasses.replace(base, ab_stages=ab_stages) + for ab_stages in ( + range(min_stages, min(max_stages, 6) + 1) + if k <= 4 * _CTA_K + else range( + min(max(5, max_stages - 2), max_stages), + max_stages + 1, + ) + ) + ) + return tactics + + +def default_tactic(m: int, n: int, k: int) -> SplitKTactic: + """Choose the default occupancy-oriented tactic.""" + if n <= 512: + mma_m = 64 + mma_n = 16 if n == 512 and m > 24 else 8 + requested_split = 4 + else: + mma_n = 8 if m <= 8 else 16 if m <= 16 else 32 + if n <= 3072: + mma_m = 128 if m <= 16 else 64 + requested_split = 4 if m <= 16 else 2 + elif n < 8192: + mma_m = 64 + requested_split = 2 + else: + mma_m = 128 if k <= 1024 and m <= 24 else 64 + requested_split = 1 + + if k <= 4 * _CTA_K: + requested_split = 1 + split_k = next( + split_k + for split_k in reversed(_SUPPORTED_SPLIT_K) + if split_k <= requested_split and (k // _CTA_K) % split_k == 0 + ) + tactic = SplitKTactic(mma_m, mma_n, split_k, _MIN_AB_STAGES) + max_stages = _max_ab_stages_for(tactic, _SMEM_CAPACITY_BYTES) + tactic = dataclasses.replace( + tactic, + ab_stages=( + _MIN_AB_STAGES + if k == _CTA_K + else _DEFAULT_MIN_AB_STAGES + if k <= 2 * _CTA_K and m > 8 + else min(max_stages, 6) + if k <= 4 * _CTA_K + else max_stages + ), + ) + validate_tactic(tactic, m, n, k) + return tactic + + +__all__ = [ + "SplitKTactic", + "autotune_tactics", + "default_tactic", + "run_splitk_dense", +] + + +@dsl_user_op +def _map_shared_rank( + smem_ptr: "cute.Pointer", + peer_cta_rank_in_cluster: "Int32", + *, + loc=None, + ip=None, +) -> "Int32": + """Map an SMEM pointer into a peer CTA's address space.""" + return Int32( + llvm.inline_asm( + T.i32(), + [ + smem_ptr.toint(loc=loc, ip=ip).ir_value(), + peer_cta_rank_in_cluster.ir_value(), + ], + "mapa.shared::cluster.u32 $0, $1, $2;", + "=r,r,r", + has_side_effects=False, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +@dsl_user_op +def _store_shared_remote_v4( + value0, + value1, + value2, + value3, + smem_ptr: "cute.Pointer", + mbar_ptr: "cute.Pointer", + peer_cta_rank_in_cluster: "Int32", + *, + loc=None, + ip=None, +) -> None: + """Publish four FP32 partials into a peer's SMEM, crediting 16 bytes.""" + llvm.inline_asm( + None, + [ + _map_shared_rank(smem_ptr, peer_cta_rank_in_cluster, loc=loc, ip=ip).ir_value(), + value0.bitcast(Int32).ir_value(loc=loc, ip=ip), + value1.bitcast(Int32).ir_value(loc=loc, ip=ip), + value2.bitcast(Int32).ir_value(loc=loc, ip=ip), + value3.bitcast(Int32).ir_value(loc=loc, ip=ip), + _map_shared_rank(mbar_ptr, peer_cta_rank_in_cluster, loc=loc, ip=ip).ir_value(), + ], + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 " + "[$0], {$1, $2, $3, $4}, [$5];", + "r,r,r,r,r,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + + +#: Rank that gathers partials and stores the output. +OWNER_RANK = 0 # capacity: fixed: a split index, not a size + + +# The drafter-boundary push (``ar_push``, modeling.DRAFT_AR_PUSH): the exchange words of +# kernels/m3_mc_allreduce, as m3_mx_gemv's ``ar`` push stores them. +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return Int32( + llvm.inline_asm( + T.i32(), + [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], + "ld.volatile.global.s32 $0, [$1];", + "=r,l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +@dsl_user_op +def _st_global_b32(addr, value, *, loc=None, ip=None): + llvm.inline_asm( + None, + [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), Int32(value).ir_value(loc=loc, ip=ip)], + "st.global.b32 [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return Int32( + llvm.inline_asm( + T.i32(), + [ + cutlass.Float32(hi).ir_value(loc=loc, ip=ip), + cutlass.Float32(lo).ir_value(loc=loc, ip=ip), + ], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", + "=r,f,f", + has_side_effects=False, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +class SplitKDenseGemmKernel: + """Standalone BF16/FP16 GEMM with a cluster-local split-K reduction.""" + + def __init__( + self, + *, + tactic: SplitKTactic, + use_pdl: bool, + has_bias: bool, + ar_push: bool = False, + ) -> None: + self.ar_push = ar_push + self.acc_dtype = cutlass.Float32 + self.cta_m = tactic.mma_m + self.cta_n = tactic.mma_n + self.cta_k = _CTA_K + self.num_ab_stage = tactic.ab_stages + self.split_k = tactic.split_k + self.use_pdl = use_pdl + self.has_bias = has_bias + + self.threads_per_cta = 256 + self.epilog_threads = 128 + self.mma_tiler_mn = (tactic.mma_m, tactic.mma_n) + self.cta_group = tcgen05.CtaGroup.ONE + self.tma_op = cute_ext.OperationTypeEnum.SM90_TMA_LOAD + self.cluster_shape = (1, tactic.split_k, 1) + + values_per_thread = (tactic.mma_m * tactic.mma_n) // self.epilog_threads + if values_per_thread % 4: + raise ValueError( + f"CTA tile ({tactic.mma_m}, {tactic.mma_n}) gives " + f"{values_per_thread} " + "values per epilogue thread; remote stores require a multiple of 4" + ) + self.mailbox_elements = (tactic.split_k - 1) * self.epilog_threads * values_per_thread + self.expected_transaction_bytes = self.mailbox_elements * _FP32_BYTES + + @cute.experimental.jit + def __call__( + self, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + bias: cute.Tensor, + stream: _cuda.CUstream, + ar_ws: cutlass.Int64 = cutlass.Int64(0), + ar_flags: cutlass.Int64 = cutlass.Int64(0), + ar_rank: Int32 = Int32(0), + ar_world: Int32 = Int32(1), + ar_maxt: Int32 = Int32(8), + ar_tokens: Int32 = Int32(0), + ar_n: Int32 = Int32(0), + ): + # Grid-y packs output-N tile and cluster rank. + self.kernel( + a, b, c, bias, ar_ws, ar_flags, ar_rank, ar_world, ar_maxt, ar_tokens, ar_n + ).launch( + grid=( + cute.ceil_div(c.layout.shape[0], self.cta_m), + cute.ceil_div(c.layout.shape[1], self.cta_n) * self.split_k, + c.layout.shape[2], + ), + block=(self.threads_per_cta, 1, 1), + cluster=self.cluster_shape, + smem=cute.Int64(utils.get_smem_capacity_in_bytes("sm_100")), + stream=stream, + use_pdl=self.use_pdl, + ) + + @cute.experimental.kernel + def kernel( + self, + mA: cute.Tensor, # (Gemm_M, Gemm_K, Gemm_L), K-major + mB: cute.Tensor, # (Gemm_N, Gemm_K, Gemm_L), K-major + mC: cute.Tensor, # (Gemm_M, Gemm_N, Gemm_L), M-major + mBias: cute.Tensor, # Broadcast bias; dead when has_bias=False + ar_ws: cutlass.Int64, # ar_push: the boundary all-reduce's multicast exchange words (int32) + ar_flags: cutlass.Int64, # ar_push: its per-(token, cluster CTA) call flags (parity = flag & 1) + ar_rank: Int32, + ar_world: Int32, + ar_maxt: Int32, # the bucket's tokens + ar_tokens: Int32, # rows pushed (public M) + ar_n: Int32, # public N (the hidden size): the exchange row width + ): + """Allocate storage and dispatch the specialized warps.""" + stages = self.num_ab_stage + + ab_dtype = mA.element_type + tiled_mma = sm100_utils.make_trivial_tiled_mma( + ab_dtype, + ab_dtype, + utils.LayoutEnum.from_tensor(mA).mma_major_mode(), + utils.LayoutEnum.from_tensor(mB).mma_major_mode(), + self.acc_dtype, + self.cta_group, + self.mma_tiler_mn, + ) + + mnk_tiler = (self.mma_tiler_mn[0], self.mma_tiler_mn[1], self.cta_k) + block_idx = cute.arch.block_idx() + bidx = block_idx[0] + split_rank = cute.arch.make_warp_uniform(cute.arch.block_idx_in_cluster()) + n_idx = block_idx[1] // self.split_k + l_idx = block_idx[2] + warp_idx = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + + sA = cute_ext.allocate( + ab_dtype, + cute.AddressSpace.smem, + sm100_utils.make_smem_layout_a(tiled_mma, mnk_tiler, ab_dtype, stages), + alignment=_AB_BUFFER_ALIGN_BYTES, + ) + sB = cute_ext.allocate( + ab_dtype, + cute.AddressSpace.smem, + sm100_utils.make_smem_layout_b(tiled_mma, mnk_tiler, ab_dtype, stages), + alignment=_AB_BUFFER_ALIGN_BYTES, + ) + + acc_layout = cute_ext.make_tmem_layout_acc(tiled_mma, self.mma_tiler_mn, acc_stage=1) + c_tiler_mn = (self.cta_m, self.cta_n) + + bar_full = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(stages), + alignment=_MBARRIER_BYTES, + ).iterator + bar_empty = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(stages), + alignment=_MBARRIER_BYTES, + ).iterator + bar_tma_epilog = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + bar_mma_epilog = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + bar_tmem_alloc = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + tmem_base_ptr = cute_ext.allocate( + cutlass.Int32, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_TMEM_POINTER_BYTES, + ).iterator + # ar_push: the owner's BF16 tile (kernel-M = output column, kernel-N = token), M-major. + sD = cute_ext.allocate( + mC.element_type, + cute.AddressSpace.smem, + cute.make_layout((self.cta_m, self.cta_n), stride=(1, self.cta_m)), + alignment=16, + ) + + if cutlass.const_expr(self.split_k > 1): + mailbox = cute_ext.allocate( + cutlass.Float32, + cute.AddressSpace.smem, + cute.make_layout(self.mailbox_elements), + alignment=_MAILBOX_ALIGN_BYTES, + ) + bar_reduce = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + else: + # Dummy operands for the compile-time-elided reduction. + mailbox = sA + bar_reduce = bar_mma_epilog + + if warp_idx == 0: + with cute.arch.elect_one(): + for i in range(stages): + cute.arch.mbarrier_init(bar_full + i, 2) + cute.arch.mbarrier_init(bar_empty + i, 1) + cute.arch.mbarrier_init(bar_tma_epilog, 32) + cute.arch.mbarrier_init(bar_mma_epilog, 1) + cute.arch.mbarrier_init(bar_tmem_alloc, 160) + + if cutlass.const_expr(self.split_k > 1): + # Owner arrival plus peer transaction-byte credits. + cute.arch.mbarrier_init(bar_reduce, 1) + + cute.arch.mbarrier_init_fence() + if cutlass.const_expr(self.split_k > 1): + # Publish peer barriers before cross-CTA stores. + cute.arch.cluster_arrive_relaxed() + else: + cute.arch.barrier() + + # Host validation guarantees an equal, tail-free K partition. + k_tile_count = cute.size(mA, mode=[1]) // self.cta_k // self.split_k + k_tile_start = split_rank * k_tile_count + + if cutlass.const_expr(self.split_k > 1): + cute.arch.cluster_wait() + + # Warp 3 is idle; warps 4-7 run the epilogue. + if warp_idx == 0: + self.dma_warp( + bar_full, + bar_empty, + bar_tma_epilog, + cute.local_tile(mA, (self.cta_m, self.cta_k), (bidx, None, l_idx)), + sA, + cute_ext.get_cta_v_map_ab(mA, mnk_tiler, tiled_mma, "A"), + k_tile_start, + k_tile_count, + True, + ) + elif warp_idx == 1: + self.dma_warp( + bar_full, + bar_empty, + bar_tma_epilog, + cute.local_tile(mB, (self.cta_n, self.cta_k), (n_idx, None, l_idx)), + sB, + cute_ext.get_cta_v_map_ab(mB, mnk_tiler, tiled_mma, "B"), + k_tile_start, + k_tile_count, + False, + ) + elif warp_idx == 2: + self.mma_warp( + bar_full, + bar_empty, + bar_mma_epilog, + bar_tmem_alloc, + tiled_mma, + sA, + sB, + tmem_base_ptr, + acc_layout, + self.cta_k // cute.size(tiled_mma.shape_mnk, mode=[2]), + k_tile_count, + ) + elif warp_idx >= 4: + self.epilog_warp( + bar_tma_epilog, + bar_mma_epilog, + bar_tmem_alloc, + tmem_base_ptr, + acc_layout, + cute.local_tile(mC, c_tiler_mn, (bidx, n_idx, l_idx)), + cute.local_tile(mBias, c_tiler_mn, (bidx, n_idx, l_idx)), + cute.arch.thread_idx()[0] - 128, + mC.element_type, + utils.LayoutEnum.from_tensor(mC), + mailbox, + bar_reduce, + split_rank, + sD, + bidx, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + @cute.experimental.jit + def dma_warp( + self, + bar_full, + bar_empty, + bar_tma_epilog, + g_tile: cute.Tensor, + s_tile: cute.Tensor, + cta_v_map: cute.Layout, + k_tile_start: cutlass.Int32, + k_tile_count: cutlass.Int32, + is_a: cutlass.Constexpr, + ): + stages = self.num_ab_stage + if cutlass.const_expr(not is_a and self.use_pdl): + cute.arch.griddepcontrol_wait() + + empty_phase = cutlass.Int32(1) + for k_tile in cutlass.range(k_tile_count, unroll=1): + stage = k_tile % stages + cute.arch.mbarrier_wait(bar_empty + stage, empty_phase) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + bar_full + stage, + cute.size_in_bytes( + s_tile.element_type, + cute.slice_(s_tile.layout, (None, None, None, 0)), + ), + ) + cute_ext.tma_load( + g_tile[None, None, k_tile_start + k_tile], + s_tile[None, None, None, stage], + (bar_full + stage).value, + cta_v_map=cta_v_map, + tma_operation_type=self.tma_op, + update_expect_tx=False, + ) + if stage == stages - 1: + empty_phase = empty_phase ^ 1 + + if cutlass.const_expr(is_a and self.use_pdl and not self.ar_push): + # ar_push: the dependent is a prepushed all-reduce, which skips its grid wait: it is released only after + # this grid's own grid wait and its pushes, in the owner's _store_tile (the prepushed >= 2 contract). + cute.arch.griddepcontrol_launch_dependents() + if cutlass.const_expr(not is_a and self.has_bias): + cute.arch.mbarrier_arrive(bar_tma_epilog) + self._drain_producer(bar_empty, empty_phase, k_tile_count) + + @cute.experimental.jit + def _drain_producer( + self, + bar_empty, + empty_phase: cutlass.Int32, + k_tile_count: cutlass.Int32, + ): + stages = self.num_ab_stage + for tail in cutlass.range(stages, unroll=1): + stage = (tail + k_tile_count) % stages + cute.arch.mbarrier_wait(bar_empty + stage, empty_phase) + if stage == stages - 1: + empty_phase = empty_phase ^ 1 + + @cute.experimental.jit + def mma_warp( + self, + bar_full, + bar_empty, + bar_mma_epilog, + bar_tmem_alloc, + tiled_mma: cute.TiledMma, + sA: cute.Tensor, + sB: cute.Tensor, + tmem_base_ptr, + acc_layout: cutlass.Constexpr, + mma_inst_tile_k: cutlass.Constexpr, + k_tile_count: cutlass.Int32, + ): + num_tmem_cols = 256 + cute.arch.alloc_tmem(num_tmem_cols, tmem_base_ptr, is_two_cta=False) + cute.arch.mbarrier_arrive(bar_tmem_alloc) + cute.arch.relinquish_tmem_alloc_permit(is_two_cta=False) + + tmem_ptr = cute.arch.retrieve_tmem_ptr(self.acc_dtype, 16, tmem_base_ptr) + accumulator = cute.make_tensor(tmem_ptr, acc_layout)[None, None, None, 0] + mma_atom = cute.make_mma_atom(tiled_mma.op) + full_phase = cutlass.Int32(0) + for k_tile in cutlass.range(k_tile_count, unroll=1): + stage = k_tile % self.num_ab_stage + cute.arch.mbarrier_wait(bar_full + stage, full_phase) + for k_block in range(mma_inst_tile_k): + if k_block == 0: + mma_atom.set(tcgen05.Field.ACCUMULATE, k_tile != 0) + else: + mma_atom.set(tcgen05.Field.ACCUMULATE, True) + cute_ext.dot( + mma_atom, + cute.append_ones(sA[None, None, k_block, stage], up_to_rank=3), + cute.append_ones(sB[None, None, k_block, stage], up_to_rank=3), + accumulator, + ) + with cute.arch.elect_one(): + tcgen05.commit(bar_empty + stage, None, self.cta_group) + if stage == self.num_ab_stage - 1: + full_phase = full_phase ^ 1 + + with cute.arch.elect_one(): + tcgen05.commit(bar_mma_epilog, None, self.cta_group) + cute.arch.mbarrier_arrive(bar_tmem_alloc) + cute.arch.mbarrier_wait(bar_tmem_alloc, 1) + cute.arch.dealloc_tmem(tmem_ptr, num_tmem_cols, is_two_cta=False) + + @cute.experimental.jit + def epilog_warp( + self, + bar_tma_epilog, + bar_mma_epilog, + bar_tmem_alloc, + tmem_base_ptr, + acc_layout: cutlass.Constexpr, + gD_tile: cute.Tensor, + gBias_tile: cute.Tensor, + epi_tid: cutlass.Int32, + c_dtype: cutlass.Constexpr, + d_layout: cutlass.Constexpr, + mailbox, + bar_reduce, + split_rank: cutlass.Int32, + sD: cute.Tensor, + bidx: cutlass.Int32, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_world: cutlass.Int32, + ar_maxt: cutlass.Int32, + ar_tokens: cutlass.Int32, + ar_n: cutlass.Int32, + ): + # Wait until MMA publishes the TMEM base pointer. + cute.arch.mbarrier_arrive(bar_tmem_alloc) + cute.arch.mbarrier_wait(bar_tmem_alloc, 0) + + acc_view = cute.make_tensor( + cute.arch.retrieve_tmem_ptr(self.acc_dtype, 16, tmem_base_ptr), + acc_layout, + )[((None, None), 0, 0, 0)] + + epi_tile = (self.cta_m, self.cta_n) + tiled_copy_t2r = cute.nvgpu.tcgen05.make_tmem_copy( + sm100_utils.get_tmem_load_op( + (self.cta_m, self.cta_n, self.cta_k), + d_layout, + c_dtype, + self.acc_dtype, + epi_tile, + False, + ), + acc_view, + ) + gD_epi = cute.flat_divide(gD_tile, epi_tile) + + # Match each epilogue thread's TMEM partition in RMEM. + rmem_layout = cute_ext.make_t2r_rmem_layout(tiled_copy_t2r, gD_epi, epi_tid) + rAcc = cute_ext.allocate( + self.acc_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + rD = cute_ext.allocate( + c_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + thr_t2r = tiled_copy_t2r.get_slice(epi_tid) + + if cutlass.const_expr(self.has_bias): + bias_dtype = gBias_tile.element_type + rBias = cute_ext.allocate( + bias_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + rBiasAcc = cute_ext.allocate( + self.acc_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + if split_rank == OWNER_RANK: + cute.arch.mbarrier_wait(bar_tma_epilog, 0) + cute_ext.partition_and_copy( + cute.make_tiled_copy_D( + cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), bias_dtype), + tiled_copy_t2r, + ).get_slice(epi_tid), + cute.flat_divide(gBias_tile, epi_tile)[None, None, 0, 0], + rBias, + ) + rBiasAcc.store(rBias.load().to(self.acc_dtype)) + + cute.arch.mbarrier_wait(bar_mma_epilog, 0) + cute_ext.partition_and_copy(thr_t2r, acc_view, rAcc) + # Make tcgen05.ld visible before TMEM release and RMEM use. + cute.arch.fence_view_async_tmem_load() + cute.arch.mbarrier_arrive(bar_tmem_alloc) + + # Peers publish FP32 partials; only rank 0 reduces and stores. + if cutlass.const_expr(self.split_k > 1): + assert cute.size(rmem_layout) == self.mailbox_elements // ( + (self.split_k - 1) * self.epilog_threads + ) + values_per_thread = cutlass.const_expr(cute.size(rmem_layout)) + values_per_peer = cutlass.const_expr(self.epilog_threads * values_per_thread) + if split_rank != OWNER_RANK: + for value_idx in cutlass.range_constexpr(0, values_per_thread, 4): + _store_shared_remote_v4( + rAcc[value_idx], + rAcc[value_idx + 1], + rAcc[value_idx + 2], + rAcc[value_idx + 3], + mailbox.iterator + + (split_rank - Int32(1)) * values_per_peer + + epi_tid * values_per_thread + + value_idx, + bar_reduce, + Int32(OWNER_RANK), + ) + else: + if epi_tid == 0: + cute.arch.mbarrier_arrive_and_expect_tx( + bar_reduce, self.expected_transaction_bytes + ) + cute.arch.mbarrier_wait(bar_reduce, 0) + for peer in cutlass.range_constexpr(self.split_k - 1): + for value_idx in cutlass.range_constexpr(values_per_thread): + rAcc[value_idx] = ( + rAcc[value_idx] + + mailbox[ + peer * values_per_peer + epi_tid * values_per_thread + value_idx + ] + ) + + if split_rank == OWNER_RANK: + if cutlass.const_expr(self.has_bias): + rAcc.store(rAcc.load() + rBiasAcc.load()) + self._store_tile(thr_t2r, rAcc, rD, gD_epi, c_dtype) + if cutlass.const_expr(self.ar_push): + self._push_tile( + thr_t2r, + rD, + sD, + epi_tid, + bidx, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + # The reduction mbarrier covers remote stores; no cluster barrier needed. + + @cute.experimental.jit + def _store_tile(self, thr_t2r, rAcc, rD, gD_epi, c_dtype: cutlass.Constexpr): + """The owner rank's reduced FP32 tile: one cast to ``c_dtype`` and the store to global memory.""" + rD.store(rAcc.load().to(c_dtype)) + # Preserve TMEM coordinates; the copy predicates output tails. + cute_ext.partition_and_copy(thr_t2r, rD, gD_epi[None, None, 0, 0]) + + @cute.experimental.jit + def _push_tile( + self, + thr_t2r, + rD, + sD, + epi_tid: cutlass.Int32, + bidx: cutlass.Int32, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_world: cutlass.Int32, + ar_maxt: cutlass.Int32, + ar_tokens: cutlass.Int32, + ar_n: cutlass.Int32, + ): + """ar_push: the owner's tile rows into the boundary all-reduce's exchange, as its own push of the output + would store them (kernels/m3_mc_allreduce: word (t, column pair) at the call parity of the (t, column / (N / 8)) + cluster CTA, -0.0 halves as +0.0), then the release of the dependents (the prepushed all-reduce).""" + cute_ext.partition_and_copy(thr_t2r, rD, sD) + cute.arch.barrier(barrier_id=1, number_of_threads=self.epilog_threads) + if cutlass.const_expr(self.use_pdl): + cute.arch.griddepcontrol_wait() + pairs = self.cta_m // 2 + col0 = bidx * self.cta_m + half = cutlass.Int64(ar_maxt) * cutlass.Int64(ar_world) * cutlass.Int64(ar_n // 2) + for w0 in cutlass.range_constexpr(0, pairs * self.cta_n, self.epilog_threads): + word = epi_tid + w0 + t = word // pairs + p = word % pairs + if t < ar_tokens: + n = col0 + 2 * p + par = _ld_volatile_s32( + ar_flags + cutlass.Int64(4) * cutlass.Int64(t * 8 + n // (ar_n // 8)) + ) & Int32(1) + off = ( + cutlass.Int64(par) * half + + cutlass.Int64(t * ar_world + ar_rank) * cutlass.Int64(ar_n // 2) + + cutlass.Int64(n // 2) + ) + _st_global_b32( + ar_ws + off * cutlass.Int64(4), + _bf16x2_word( + sD[2 * p, t].to(cutlass.Float32), sD[2 * p + 1, t].to(cutlass.Float32) + ), + ) + if cutlass.const_expr(self.use_pdl): + cute.arch.griddepcontrol_launch_dependents() + + +import torch as _torch # noqa: E402 (as in FlashInfer) + +_SUPPORTED_TORCH_DTYPES = (_torch.bfloat16, _torch.float16) + + +@cute.experimental.jit +def _bmm_no_bias( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + stream: _cuda.CUstream, +): + c = cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])) + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + c, + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[0, 1, 2])), + stream, + ) + + +@cute.experimental.jit +def _bmm_push( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + stream: _cuda.CUstream, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: Int32, + ar_world: Int32, + ar_maxt: Int32, + ar_tokens: Int32, + ar_n: Int32, +): + c = cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])) + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + c, + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[0, 1, 2])), + stream, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + +@cute.experimental.jit +def _bmm_bias( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + bias: cute.Tensor, + stream: _cuda.CUstream, +): + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])), + cute.make_tensor(bias.iterator, cute.select(bias.layout, mode=[1, 2, 0])), + stream, + ) + + +def _from_dlpack_dynamic(tensor, leading_dim: int, assumed_align: int = 32): + return from_dlpack(tensor, assumed_align=assumed_align).mark_layout_dynamic( + leading_dim=leading_dim + ) + + +def _detect_leading_dim(tensor: _torch.Tensor) -> int: + # Ignore synthetic batch stride, including 1x1. + for dim, stride in enumerate(tensor.stride()[1:], start=1): + if stride == 1: + return dim + raise ValueError("tensor has no stride-1 dimension") + + +def _make_layout_tensor( + shape: tuple[int, ...], dtype: _torch.dtype, leading_dim: int +) -> _torch.Tensor: + permutation = [dim for dim in range(len(shape)) if dim != leading_dim] + [leading_dim] + return _torch.empty( + tuple(shape[dim] for dim in permutation), dtype=dtype, device="cuda" + ).permute([permutation.index(dim) for dim in range(len(shape))]) + + +def _make_compile_repr_tensors( + dtype: _torch.dtype, + has_bias: bool, + a_leading: int, + b_leading: int, + c_leading: int, +): + m, n, k, batch = 64, 8, _CTA_K, 1 + tensors = tuple( + _from_dlpack_dynamic(_make_layout_tensor(shape, dtype, leading_dim), leading_dim) + for shape, leading_dim in ( + ((batch, n, k), a_leading), + ((batch, k, m), b_leading), + ((batch, n, m), c_leading), + ) + ) + if not has_bias: + return (*tensors, None) + return ( + *tensors, + _from_dlpack_dynamic( + _torch.empty((n,), dtype=dtype, device="cuda").as_strided( + size=(batch, n, m), stride=(0, 1, 0) + ), + 1, + 2, + ), + ) + + +def _to_cute_swap(a, b, out, bias): + a_swap = b.unsqueeze(0).transpose(-2, -1) + b_swap = a.unsqueeze(0).transpose(-2, -1) + c_swap = out.unsqueeze(0).transpose(-2, -1) + leading_dims = tuple(_detect_leading_dim(tensor) for tensor in (a_swap, b_swap, c_swap)) + cute_tensors = tuple( + _from_dlpack_dynamic(tensor, leading_dim) + for tensor, leading_dim in zip((a_swap, b_swap, c_swap), leading_dims, strict=True) + ) + if bias is None: + return (*cute_tensors, None, leading_dims) + return ( + *cute_tensors, + _from_dlpack_dynamic( + bias.as_strided(size=(1, c_swap.shape[1], c_swap.shape[2]), stride=(0, 1, 0)), + 1, + 2, + ), + leading_dims, + ) + + +@cute_aot.cached_compile("m3_lowm_splitk._get_compiled_splitk_kernel", lambda: _aot_sources()) +def _get_compiled_splitk_kernel( + dtype, + tactic: SplitKTactic, + use_pdl: bool, + has_bias: bool, + leading_dims: tuple[int, int, int], +): + if dtype not in _SUPPORTED_TORCH_DTYPES: + raise ValueError(f"split-K dense GEMM supports {_SUPPORTED_TORCH_DTYPES}; got {dtype}") + + kernel = SplitKDenseGemmKernel( + tactic=tactic, + use_pdl=use_pdl, + has_bias=has_bias, + ) + compile_tensors = _make_compile_repr_tensors(dtype, has_bias, *leading_dims) + stream = _cuda.CUstream(_torch.cuda.current_stream().cuda_stream) + if has_bias: + compiled = cute_ext.compile(_bmm_bias, kernel, *compile_tensors, stream) + else: + compiled = cute_ext.compile(_bmm_no_bias, kernel, *compile_tensors[:3], stream) + return compiled + + +def _validate_runtime_tensors(a, b, bias, out) -> tuple[int, int, int]: + tensors = (a, b, out) + ((bias,) if bias is not None else ()) + if any(not isinstance(tensor, _torch.Tensor) for tensor in tensors): + raise ValueError("a, b, out, and bias must be torch tensors") + if a.ndim != 2 or b.ndim != 2 or out.ndim != 2: + raise ValueError("split-K dense GEMM accepts only 2D tensors") + if a.device.type != "cuda" or any(tensor.device != a.device for tensor in tensors): + raise ValueError("all tensors must be on the same CUDA device") + if a.dtype not in _SUPPORTED_TORCH_DTYPES or any(tensor.dtype != a.dtype for tensor in tensors): + raise ValueError("a, b, out, and bias must share BF16 or FP16 dtype") + if any(not (tensor.is_contiguous() or tensor.t().is_contiguous()) for tensor in (a, b, out)): + raise ValueError("a, b, and out must be dense row-major or column-major matrices") + if any(tensor.data_ptr() % 32 for tensor in (a, b, out)): + raise ValueError("a, b, and out must be 32-byte aligned") + + m, k = a.shape + if b.shape[0] != k: + raise ValueError(f"incompatible shapes: a is {tuple(a.shape)}, b is {tuple(b.shape)}") + n = b.shape[1] + if out.shape != (m, n): + raise ValueError(f"out must have shape {(m, n)}, got {tuple(out.shape)}") + if bias is not None and (bias.ndim != 1 or bias.shape[0] != n or not bias.is_contiguous()): + raise ValueError( + f"bias must be contiguous with shape {(n,)}, " + f"got shape {tuple(bias.shape)} and stride {bias.stride()}" + ) + + return m, n, k + + +def run_splitk_dense( + a, + b, + bias, + out, + pdl: bool, + tactic: SplitKTactic, +): + """Run ``A[M,K] @ B[K,N]`` with the ``mm_bf16`` layouts.""" + validate_tactic(tactic, *_validate_runtime_tensors(a, b, bias, out)) + has_bias = bias is not None + cute_tensors = _to_cute_swap(a, b, out, bias) + compiled = _get_compiled_splitk_kernel( + dtype=a.dtype, + tactic=tactic, + use_pdl=pdl, + has_bias=has_bias, + leading_dims=cute_tensors[4], + ) + stream = _cuda.CUstream(_torch.cuda.current_stream(a.device).cuda_stream) + if has_bias: + compiled(*cute_tensors[:4], stream) + else: + compiled(*cute_tensors[:3], stream) + return out + + +@cute_aot.cached_compile("m3_lowm_splitk._get_compiled_push", lambda: _aot_sources()) +def _get_compiled_push( + dtype, tactic: SplitKTactic, use_pdl: bool, leading_dims: tuple[int, int, int] +): + kernel = SplitKDenseGemmKernel(tactic=tactic, use_pdl=use_pdl, has_bias=False, ar_push=True) + compile_tensors = _make_compile_repr_tensors(dtype, False, *leading_dims) + stream = _cuda.CUstream(_torch.cuda.current_stream().cuda_stream) + z64, z32 = cutlass.Int64(0), Int32(0) + return cute_ext.compile( + _bmm_push, + kernel, + *compile_tensors[:3], + stream, + z64, + z64, + z32, + Int32(1), + Int32(8), + z32, + Int32(6144), + ) + + +def compile_push(a, b, out, pdl: bool, tactic) -> None: + """Compile (only) the push variant of ``run_splitk_dense_push`` for these operands' shapes and layouts.""" + cute_tensors = _to_cute_swap(a, b, out, None) + _get_compiled_push(a.dtype, SplitKTactic(*tactic), pdl, cute_tensors[4]) + + +def run_splitk_dense_push(a, b, out, pdl: bool, tactic: SplitKTactic, ar: tuple): + """``run_splitk_dense`` without bias that also pushes ``out``'s rows into a drafter boundary all-reduce's exchange; + ``ar`` = (multicast words address, flags address, rank, world, bucket tokens) of the 8-token bucket.""" + validate_tactic(tactic, *_validate_runtime_tensors(a, b, None, out)) + cute_tensors = _to_cute_swap(a, b, out, None) + compiled = _get_compiled_push(a.dtype, tactic, pdl, cute_tensors[4]) + stream = _cuda.CUstream(_torch.cuda.current_stream(a.device).cuda_stream) + ws, flags, rank, world, maxt = ar + compiled( + *cute_tensors[:3], + stream, + cutlass.Int64(ws), + cutlass.Int64(flags), + Int32(rank), + Int32(world), + Int32(maxt), + Int32(out.shape[0]), + Int32(out.shape[1]), + ) + return out + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_mc_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_mc_allreduce.py new file mode 100644 index 000000000000..142bcb9a9133 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_mc_allreduce.py @@ -0,0 +1,422 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Low-latency TP all-reduce + residual + RMSNorm for decode (M <= 8 tokens) over a multicast mapping. + + partial = bf16(x + x2) (optional second input, e.g. shared + routed experts) + updated = bf16(residual + sum_r partial_r) (ranks summed in FP32, rank order) + normed = bf16(updated * rsqrt(mean(updated^2) + eps) * w) + +One cluster of CLUSTER CTAs per token; CTA c owns columns [c * H / CLUSTER, ...), one 16-byte chunk (8 BF16) +per thread. Each CTA pushes its slice of this rank's partial through the multicast mapping (one store lands in +every rank's buffer), releases dependents, polls its own buffer until all ranks' slices arrived, sums them, adds +the residual and reduces the RMS statistic across the cluster through DSMEM. + +Exchange protocol as in the K3 sandwich: int32 words, 0x80000000 = empty (pushed -0.0 halves become +0.0), each +CTA keeps a call counter whose parity selects the buffer half, readers re-arm the words they read. All ranks of +the TP group must issue the same calls in the same stream order. + +The workspace holds two exchange sets. With prepushed 3 two producers pushed this rank's x (set A) and x2 (set B), +e.g. the routed experts' finalize and the shared expert's down projection, and the kernel forms bf16(x + x2) per +rank from the two slots, exactly as the unpushed path does. + +Producers that push (the o_proj / shared-expert GEMV ``ar`` tuple, the routed-expert engine's ``arp`` descriptor) are +pointed at a bucket's exchange by its ``mc`` address, ``flags_for(8)`` counters, ``rank``, ``world_size`` and +``max_tokens``; ``get_workspace`` and ``_bucket`` select it. + +The kernel is ``torch.ops.trtllm.minimax_m3_sm107_tp4_mc_allreduce_norm``, reached through the ``ops/comm`` entries +``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm``; this module owns the workspaces, the bucket and +epilogue-option choice and the MXFP8 publication. +""" + +from dataclasses import dataclass +from typing import Any + +import torch + +from ..ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm as _allreduce_norm_op +from ..ops.comm.m3_mc_finalize_allreduce_norm import ( + m3_mc_finalize_allreduce_norm as _finalize_allreduce_norm_op, +) + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +BUCKETS = (8, 16, 48) # token capacity of each workspace (grid = bucket * cluster CTAs) +EMPTY_WORD = -2147483648 + +# Attention / MoE boundary epilogue options (each bit-identical): 1 residual + norm weight loaded at entry (before +# the poll); 2 cluster RMS exchange by st.async + mbarrier complete_tx instead of cluster.sync; 4 call parity rotated +# right after the poll (no trailing __syncthreads); 8 max-smem carveout so the next kernel's CTAs co-reside early. +# The 8-token bucket, by token count 1..8 (GB300 TP4 tuning). +AR_OPTS_BY_M = (7, 7, 15, 15, 15, 15, 7, 7) +# The 16- and 48-token buckets. +AR_OPTS_BY_BUCKET = {16: 15, 48: 15} +# The MoE-finalize all-reduce, per bucket. +AR_FIN_OPTS_BY_BUCKET = {8: 7, 16: 15, 48: 15} + + +# Option 32: the 8-CTA cluster kernel in the stock TP4 oneshot reduction order instead of the row kernel (not finalize). +_OPT_ONESHOT_CLUSTER = 32 + +# Option 64: the row kernel's max-smem-carveout instance (as option 8 is for the cluster kernel). +_OPT_ROW_CARVEOUT = 64 + +# Option 128: the drafter boundary (option 16) on the 8-CTA cluster kernel instead of the world-4 row kernel (routing +# only; mcAllreduceNormTp4.cu invoke). Exact-class to the row kernel's FlashInfer order: the RMS statistic is the +# cluster's sum (modeling.DRAFT_MC_AR_CLUSTER). +_OPT_DRAFTER_CLUSTER = 128 + +# Option 2 is also the safe ordering of the cluster RMS exchange: its stores into peer CTAs' shared memory follow a +# cluster barrier arrive / wait pair. Without it they run before any cluster barrier, when a peer need not have +# started, so every call keeps it. +_OPT_CLUSTER_BARRIER = 2 + + +def _ar_opts(m: int, bucket: int, finalize: bool = False, oneshot_cluster: bool = False) -> int: + """Epilogue options for an m-row call on the given bucket; every option is bit-identical. A bucket missing from + the tables gets 0, and every result keeps _OPT_CLUSTER_BARRIER. ``oneshot_cluster`` adds _OPT_ONESHOT_CLUSTER.""" + if oneshot_cluster and not finalize: + return _ar_opts(m, bucket) | _OPT_ONESHOT_CLUSTER + if finalize: + opts = AR_FIN_OPTS_BY_BUCKET.get(bucket, 0) + elif bucket == 8: + opts = AR_OPTS_BY_M[m - 1] + else: + opts = AR_OPTS_BY_BUCKET.get(bucket, 0) + return opts | _OPT_CLUSTER_BARRIER + + +@dataclass(eq=False) +class McAllReduceWorkspace: + """Two halves of [max_tokens][world][H] BF16 (int32 words) per rank behind one multicast mapping, plus one call + counter per CTA per cluster size. Create collectively and eagerly on every TP rank.""" + + uc: torch.Tensor + mc: torch.Tensor + flags: dict + rank: int + world_size: int + hidden: int + handle: Any + comm: Any + max_tokens: int = 8 + + @classmethod + def create(cls, mapping, hidden: int, max_tokens: int = 8) -> "McAllReduceWorkspace": + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + words = 4 * max_tokens * mapping.tp_size * (hidden // 2) # two exchange sets of two halves + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "McAllReduceWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 4, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("McAllReduceWorkspace.create failed on another rank") + return cls( + uc=uc, + mc=mc, + flags={}, + rank=mapping.tp_rank, + world_size=mapping.tp_size, + hidden=hidden, + handle=handle, + comm=comm, + max_tokens=max_tokens, + ) + + def flags_for(self, cluster: int): + """This cluster size's per-CTA call counters, or None when they do not exist yet during a capture. + get_workspace creates the cluster-8 counters with the workspace, so for cluster 8 (the only size built) + that None never happens and cannot differ between ranks.""" + flags = self.flags.get(cluster) + if flags is None: + if torch.cuda.is_current_stream_capturing(): + return None + flags = torch.zeros(self.max_tokens * cluster, dtype=torch.int32, device=self.uc.device) + self.flags[cluster] = flags + return flags + + +class McBuckets: + """One workspace per token bucket (8 / 16 / 48): calls of different sizes never share exchange slots.""" + + def __init__(self, workspaces): + self.workspaces = workspaces + self.hidden = workspaces[0].hidden + + def for_tokens(self, m: int): + for ws in self.workspaces: + if m <= ws.max_tokens: + return ws + return None + + +def _bucket(workspace, m): + return workspace.for_tokens(m) if isinstance(workspace, McBuckets) else workspace + + +def decline_reason(x, residual, workspace, cluster=8, x2=None) -> str: + """Why ``m3_mc_allreduce_norm`` returned None for these operands (for the dispatch's miss message).""" + if workspace is None: + return "no multicast workspace: it is created collectively on the first eager decode step" + bucket = _bucket(workspace, x.shape[0]) + if bucket is None: + return f"no workspace bucket holds {x.shape[0]} tokens (buckets {BUCKETS})" + if bucket.flags.get(cluster) is None: + return f"no {cluster}-CTA call counters for the {bucket.max_tokens}-token bucket" + return ( + f"operands outside the kernel: x {tuple(x.shape)} {x.dtype} contiguous={x.is_contiguous()}, " + f"residual contiguous={residual.is_contiguous()}, hidden {bucket.hidden}" + + ("" if x2 is None else f", x2 {tuple(x2.shape)} contiguous={x2.is_contiguous()}") + ) + + +def m3_mc_allreduce_norm( + x, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + x2=None, + quant=False, + prepushed=0, + oneshot_cluster=False, + drafter_norm=False, + carveout=False, + capture=None, + drafter_cluster=False, +): + """Return (normed, updated_residual) or None when unsupported. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM. ``capture`` = (out [>= M, H] view, fc_norm weight + [H], eps): the boundary also writes the Eagle3 capture of the updated residual, bit for bit + m3_spec_tail.capture_norm (CAPTURE_NORM_IN_BOUNDARY; not with ``drafter_norm``). ``drafter_cluster``: with + ``drafter_norm``, the 8-CTA cluster kernel's drafter form (option 128; its carveout is option 8) instead of the row + kernel's. + + A None makes the caller take another collective, so it must be the same on every TP rank. It is: it depends only + on the shapes, dtypes and layouts of the operands (the same code builds them on every rank) and on whether the + workspace's flags exist, which ``get_workspace`` creates collectively, before any capture, for cluster 8.""" + workspace = _bucket(workspace, x.shape[0]) + if workspace is None: + return None + if ( + x.dim() != 2 + or x.shape[0] > MAX_TOKENS + or x.dtype != torch.bfloat16 + or x.shape[1] != workspace.hidden + or not x.is_contiguous() + or not residual.is_contiguous() + or (x2 is not None and (not x2.is_contiguous() or x2.shape != x.shape)) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + # The Eagle3 drafter boundary (opts bit 16, the row kernel's FlashInfer fused_add_rmsnorm order): 8-token bucket, + # x only, no MXFP8 copy, never the cluster kernel. + if drafter_norm and ( + workspace.max_tokens != 8 or x2 is not None or quant or prepushed not in (0, 2) + ): + return None + if drafter_norm and drafter_cluster: + opts = ( + (16 | _ar_opts(x.shape[0], workspace.max_tokens)) & ~8 + | (8 if carveout else 0) + | _OPT_DRAFTER_CLUSTER + ) + elif drafter_norm: + opts = ( + 16 | _ar_opts(x.shape[0], workspace.max_tokens) | (_OPT_ROW_CARVEOUT if carveout else 0) + ) + else: + opts = _ar_opts(x.shape[0], workspace.max_tokens, oneshot_cluster=oneshot_cluster) + if carveout: + # the max-smem carveout on whichever kernel runs: option 8 on the cluster kernel, 64 on the row kernel + opts |= 8 if oneshot_cluster else _OPT_ROW_CARVEOUT + outs = _allreduce_norm_op( + x, + x2, + residual, + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and x.shape[1] % 128 == 0), + int(prepushed), + opts, + *_capture_args(capture, drafter_norm), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +def _capture_args(capture, drafter_norm: bool) -> tuple: + """(cap_out, cap_w, cap_eps) for the op, from ``capture`` = (out, fc_norm weight, eps) or None.""" + if capture is None: + return (None, None, 0.0) + assert not drafter_norm, "the capture epilogue is not built for the drafter boundary" + out, weight, eps = capture + return (out, weight, float(eps)) + + +def m3_mc_finalize_allreduce_norm( + fc2, + perm, + scales, + shared, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + quant=False, +): + """MoE finalize (sum_k scale * fc2[perm] + shared) + all-reduce + residual + RMSNorm in one kernel. + Returns (normed, updated_residual) or None when unsupported. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM. (It runs the shared GB300 op, which has no capture + epilogue: a capture layer finalized here keeps the capture_norm kernel.)""" + m = residual.shape[0] + workspace = _bucket(workspace, m) + if workspace is None: + return None + if ( + residual.dim() != 2 + or m > MAX_TOKENS + or residual.shape[1] != workspace.hidden + or fc2.dtype != torch.bfloat16 + or not fc2.is_contiguous() + or fc2.dim() != 2 + or perm.dtype != torch.int32 + or perm.dim() != 2 + or perm.shape[0] != m + or not perm.is_contiguous() + or scales.dtype not in (torch.float32, torch.bfloat16) + or not scales.is_contiguous() + or (shared is not None and (shared.shape != residual.shape or not shared.is_contiguous())) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + outs = _finalize_allreduce_norm_op( + fc2, + perm, + scales, + shared, + residual.contiguous(), + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and residual.shape[1] % 128 == 0), + _ar_opts(m, workspace.max_tokens, finalize=True), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +_WORKSPACE: dict = {} +_AGREE_COMMS: dict = {} + + +def _tp_comm(mapping): + """A communicator over mapping's TP group, created once (collective over that group only).""" + key = tuple(mapping.tp_group) + comm = _AGREE_COMMS.get(key) + if comm is None: + from tensorrt_llm._utils import mpi_comm, mpi_disabled + + if mpi_disabled(): + comm = mapping.tp_group_pg + else: + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + _AGREE_COMMS[key] = comm + return comm + + +def tp_agree(mapping, ok: bool) -> bool: + """Whether ``ok`` holds on every rank of mapping's TP group (one blocking host collective). + + For a fast path whose availability is decided from rank-local state (a kernel build that can fail on one rank): + every rank must take the same path, so that they issue the same collectives with the same exchange protocol and + the same kernels. Call it once per decision, at build time, at the same point on every rank, outside CUDA-graph + capture; never per step. With one rank, or attention DP (whose ranks run different batches), it returns ``ok``. + """ + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return ok + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("tp_agree must run outside CUDA-graph capture") + from tensorrt_llm._torch.distributed.ops import _mnnvl_workspace_all_succeeded + + return _mnnvl_workspace_all_succeeded(_tp_comm(mapping), bool(ok)) + + +def tp_agree_built(mapping, built, what: str): + """``built`` when it is not None on every TP rank, else None on every rank (see ``tp_agree``).""" + if tp_agree(mapping, built is not None) or built is None: + return built + from tensorrt_llm.logger import logger + + logger.warning(f"{what}: built here but not on another TP rank; every rank falls back") + return None + + +def get_workspace(mapping, hidden: int): + """Process-wide workspace (created collectively on first eager use); None while capturing before creation. + + The first eager call for a hidden size (the warmup forward, before any CUDA-graph capture) creates one + McAllReduceWorkspace per bucket in BUCKETS on every TP rank (MPI collective over the TP group) plus each + bucket's 8-CTA-cluster call counters; later calls, captured or not, return the same McBuckets. The workspaces + live until process exit. + """ + ws = _WORKSPACE.get(hidden) + if ws is None: + if torch.cuda.is_current_stream_capturing(): + return None + wss = [McAllReduceWorkspace.create(mapping, hidden, b) for b in BUCKETS] + for w in wss: + w.flags_for(8) + ws = McBuckets(wss) + _WORKSPACE[hidden] = ws + return ws diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/__init__.py new file mode 100644 index 000000000000..f818a0e6c9f1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/__init__.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 MoE layer of the sm_107 TP4 target for low-batch decode as a CuTe DSL kernel: ``m3_moe`` for 1..8 tokens +(op.py) at the TP4 routed and shared intermediate of 768 per rank.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/m3_moe16_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/m3_moe16_kernel.py new file mode 100644 index 000000000000..28e4ee3ab6f1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/m3_moe16_kernel.py @@ -0,0 +1,3900 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 routed + shared experts for 9..16 decode tokens on one TP rank: ``m3_moe16`` (v2). + +Same math as the 1..8-token engine (m3_moe_kernel.py), operation for operation for the routed part (bitwise equal to +TRTLLM-Gen's fused MoE on the fp32 router-GEMV logits); a different schedule for 9..16 rows: + +- Front: the hidden states land in smem by bulk copies in K chunks (the idle ring); all 16 warps run the router + GEMV (m3_router_gemv's per-thread order) and the NVFP4 quantization of B1 from smem; the logits cross CTAs as + tagged words; top-k one warp per token. +- Ring: FC1 / FC2 weight stages by TMA; their scale words go by 4-byte cp.async straight into the instruction atom + layout (SFA_CP; no raw scale box, no splicer warp), completing on the stage's full barrier. +- FC1 -> FC2 hand-off (B2IMG): the FC1 epilogue writes each unit's FP4 values and scale byte into a per-group global + image laid out as the FC2 B tile + its scale atoms; one counter per group (release add per finished unit, every + counter gains exactly 192 per call); warp 5 loads a completed image with two bulk copies. +- Shared expert on NSH more CTAs (MXFP8 tcgen05), as the 1..8 engine. +Configuration via M3_CONFIG (m_max, num_ctas, stages, nb2_cap, lag, shared, sfa_cp, b2buf, ...). +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +_CFG = globals().get("M3_CONFIG") or {} + + +def _cfg(key: str, default): + return type(default)(_CFG.get(key, default)) + + +H = 6144 # capacity: fixed: the checkpoint's hidden size +I_TP = 768 +TWO_I = 2 * I_TP +E_LOCAL = 128 +TOP_K = 4 # capacity: fixed: the checkpoint's top-k +ROUTE_SCALE = 2.0 +SUM_EPS = 1e-20 +M_MAX = _cfg("m_max", 12) +assert 9 <= M_MAX <= 16, M_MAX +NCTA = _cfg("num_ctas", 128) +FUSED_ROUTER = True +NROUTE = E_LOCAL +NGEMV = (NROUTE + NCTA - 1) // NCTA +assert NGEMV == 1, "m3_moe16: one router expert per CTA (num_ctas >= 128)" +TIMING = bool(_cfg("timing", 0)) +HANG_DBG = bool(_cfg("hang_dbg", 0)) +LAG = _cfg("lag", 3) +G_CAP = TOP_K * M_MAX +NG_CAP = M_MAX +THREADS = 512 # capacity: fixed: the CTA size +NWARPS = THREADS // 32 +GEMV_THREADS = 256 +SHARED = bool(_cfg("shared", 0)) +LATE_TRIGGER = bool(_cfg("late_trigger", 1)) +# OUT_SKIP (MOE-FINALIZE-DEADSTORE): on a call that pushes into the boundary all-reduce (arp enabled), +# the finalize does not also store its bf16 rows into ``out``: the pushed all-reduce (prepushed >= 2) never reads them +# and returns its own tensors, so the store is dead traffic. Calls without a push store as before. Bitwise: the pushed +# words are unchanged. +OUT_SKIP = bool(_cfg("out_skip", 0)) +# PROD_TRIGGER (scheduling only; the TP8 MOE_TAIL_TRIGGER): on a call that pushes into the boundary +# all-reduce, each routed CTA releases the dependents when its weights producer has issued its last stage (the +# post-stream tail: FC1 drain, hand-off, FC2, finalize), not at teardown. The pushed all-reduce reads only the pushed +# tagged words, without a grid wait. +PROD_TRIGGER = bool(_cfg("prod_trigger", 0)) +# With an early release the next kernels' completion no longer implies this kernel's, so the per-CTA call epoch -- read +# by the NEXT call on this state -- is stored once every thread of the CTA has read it, not at teardown. Every other +# workspace write of a call lands before the finalize's push. +EARLY_EPOCH = PROD_TRIGGER +SFA_CP = bool( + _cfg("sfa_cp", 1) +) # A scales by 4-byte cp.async into the atom layout (else raw box + splicer warp) +B2IMG = True +PERM = bool(_cfg("perm", 1)) +PERM_MIN_NG = max( + 3, _cfg("perm_min_ng", 5) +) # (measured: helps at random routing G >= 23, hurts at spec G 13-14) +# a partial last slot group becomes group 1 (the plan must match: op16) +DBG_DELAY = bool( + _cfg("dbg_delay", 0) +) # (stress builds) pseudo-random producer / epilogue / release stalls +DBG_NOSTORE = bool(_cfg("dbg_nostore", 0)) # (timing experiment only: wrong results) +EPI_FENCE = bool( + _cfg("epi_fence", 0) +) # (rel_warp 0) writer lanes' fence.acq_rel.gpu before the release add +EPI_PFENCE = bool( + _cfg("epi_pfence", 1) +) # producer-side fence.proxy.async.global before the release add +REL_WARP = bool( + _cfg("rel_warp", 1) +) # warp 8 does the per-group release adds (off the epilogue's path) +NSH = 24 if SHARED else 0 +SH_I = 768 +SH_KB = H // 32 # capacity: fixed: a kernel tile constant +SH_DKB = SH_I // 32 # capacity: fixed: a kernel tile constant +SH_SPLIT = 2 +SH_GT = SH_KB // 4 // SH_SPLIT # capacity: fixed: a kernel tile constant +SH_STAGES_GU = SH_GT +SH_STAGES = SH_GT + 2 * (SH_I // 128) # capacity: fixed: pipeline stages +SHX_BAR = 6 +SH_PRETILE = False +SH_STAGE_BYTES = 16384 + 512 +MMA_M = 128 +KT_BYTES = 128 +INC_K = 2 +NUM_KB = 4 +# FC1 +N1 = _cfg("n1", 48 if _cfg("m_max", 12) <= 12 else 64) # B rows QS q + t +N1_LD = 1 << (N1 - 1).bit_length() # tcgen05.ld .num (a power of two) covering N1 columns +QS = N1 // 4 +assert QS >= M_MAX +R1 = 32 +C1 = 4 +U1 = TWO_I // R1 +K1_TILES = H // 256 +S1 = K1_TILES // C1 # capacity: fixed: pipeline stages +NUM_BYTES_A = MMA_M * KT_BYTES # capacity: fixed: a stage's operand bytes +NUM_BYTES_BOX1 = R1 * KT_BYTES +SF1_KA = H // 64 +SF1_RB = TWO_I // 128 +NUM_BYTES_SFA_RAW = 16 * 512 +NUM_WORDS_SFA_RAW = NUM_BYTES_SFA_RAW // 4 +NUM_BYTES_SFA = NUM_KB * 512 +NUM_WORDS_SFA = NUM_BYTES_SFA // 4 +NUM_BYTES_B1 = N1 * KT_BYTES +NUM_SFB1_ATOMS = S1 * NUM_KB +# FC2 +R2 = 32 +NB2 = H // R2 +KT2 = I_TP // 256 +SF2_KA = I_TP // 64 +SF2_RB = H // 128 +N2 = _cfg("n2", (4 * M_MAX + 15) // 16 * 16) # B rows j M + t +NUM_BYTES_B2 = N2 * KT_BYTES +IMG_B2 = KT2 * NUM_BYTES_B2 # one group's B2 tile image (global and smem) +IMG_SF = KT2 * NUM_KB * 512 # its scale atoms +IMG_BYTES = IMG_B2 + IMG_SF +CNT_PER_CALL = 4 * (TWO_I // R1) # every group counter gains 192 per call +B1IMG_OFF = None # set below (after N1): the activation image after the NG_CAP group images +TK_WARPS = [0, 1, 2, 3, 6, 8, 9, 10, 11, 12, 13, 14, 5, 7, 15, 4][ + : _cfg("m_max", 12) +] # top-k warp of token t (4 last: early0) +B1_WARP = 15 if _cfg("m_max", 12) <= 14 else -1 # (-1: warp 0 after the routing) +NB2_CAP = _cfg("nb2_cap", 2) +B1IMG_OFF = M_MAX * IMG_BYTES +B1IMG_BYTES = (S1 * N1 * 128 + M_MAX * (H // 16) + 15) // 16 * 16 +IMG_TOTAL = B1IMG_OFF + B1IMG_BYTES +UNITS_ROW = TWO_I // R1 +PARTIAL_EMPTY = -1 # capacity: fixed: a sentinel, not a size +EPI_THREADS = 128 +EPI_BAR = 1 +ROUTE_BAR = 2 +GEMV_BAR = 3 +TOPK_BAR = 4 +ALL_BAR = 5 +POLL_NS = _cfg("poll_ns", 150) +B2_POLL_NS = _cfg("b2_poll_ns", 16) +LG_REPL = _cfg("lg_repl", 64) +LG_STRIDE = M_MAX * E_LOCAL + 32 +W_WARP = 4 # capacity: fixed: a warp role index +X_WARP = 5 # B2 image loader; slots + plan # capacity: fixed: a warp role index +S_WARP = 6 # (SFA_CP = 0) scale splicer # capacity: fixed: a warp role index +MMA_WARP = 7 # capacity: fixed: a warp role index +EARLY0 = bool(_cfg("early0", 1)) +EVICT_FIRST = 0x12F0000000000000 +EVICT_LAST = 0x14F0000000000000 +_LOG2E = 1.4426950408889634 +NB2BUF = _cfg("b2buf", 2) +SPB = 1 +# x staging: K chunks of XCK elements, NXB smem buffers (rotating if fewer than the chunks) +XCK = _cfg("xck", 2048) +NXC = H // XCK +XCB = M_MAX * XCK * 2 # bytes of one chunk (all tokens) +STAGE_A = NUM_BYTES_A +STAGE_SF = NUM_BYTES_SFA if SFA_CP else NUM_BYTES_SFA_RAW + NUM_BYTES_SFA +_FIXED_SMEM = ( + S1 * NUM_BYTES_B1 + + NB2BUF * IMG_B2 + + IMG_SF + + NB2_CAP * M_MAX * TOP_K * R2 * 2 + + SPB * M_MAX * C1 * R1 * 4 + + (G_CAP + 1) * (NG_CAP + 1) * 4 + + 6 * 1024 +) +_STAGES_FIT = (227 * 1024 - _FIXED_SMEM - (128 if SHARED else 0)) // (STAGE_A + STAGE_SF) +ACC1_COL = 0 +ACC2_COL = 2 * N1 +ACC2_STRIDE = 1 << (N2 - 1).bit_length() +SFB1_COL = ACC2_COL + 2 * ACC2_STRIDE +SFB1_WSTEP = 1 +SFB2_COL = ( + SFB1_COL + NUM_SFB1_ATOMS * 4 +) # one TMEM slot (loaded after the previous group's FC2 MMAs) +SFA1_COL = SFB2_COL + KT2 * NUM_KB * 4 +STAGES = min(_cfg("stages", 8), _STAGES_FIT, (512 - SFA1_COL) // 16) +TMEM_COLS = 512 +assert STAGES >= 3, (STAGES, _STAGES_FIT, SFA1_COL) +# smem: [ring A | ring SF | B2 bufs | B2 SF staging | B1]; x chunks alias [0, _OFF_B1) during the front +_OFF_SFA = STAGES * NUM_BYTES_A +_OFF_B2 = (_OFF_SFA + STAGES * STAGE_SF + 1023) // 1024 * 1024 +_OFF_SFB2 = _OFF_B2 + NB2BUF * IMG_B2 +_OFF_B1 = (_OFF_SFB2 + IMG_SF + 1023) // 1024 * 1024 +BIG = _OFF_B1 + S1 * NUM_BYTES_B1 +NXB = min(NXC, _OFF_B1 // XCB) +assert NXB >= 2, (NXB, _OFF_B1, XCB) +# shared CTA layout in BIG: one pass over all M tokens; B rows SHN q + t (gate_up, 2 K slices), t (down) +SHN = 16 # capacity: fixed: a kernel tile constant +SH_NPASS = 1 +SH_HINT = EVICT_FIRST +SH_PF_POST = bool( + _cfg("sh_pf_post", 1) +) # L2 prefetch of the shared weights after the grid dependency (else before) +_SB_A = 0 +_SH_FIX = ( + SH_GT * 2 * SHN * 128 + + 6 * SHN * 128 + + M_MAX * SH_KB + + M_MAX * SH_DKB + + 64 * SHN * 4 + + 2 * 32 * SHN * 4 + + 2048 +) +SH_NST = min(8, (BIG - _SH_FIX) // (NUM_BYTES_A + 4 * 512 + 512)) if SHARED else 1 +_SB_RAW = SH_NST * NUM_BYTES_A +_SB_SPL = _SB_RAW + SH_NST * 2048 +_SB_BX = (_SB_SPL + SH_NST * 512 + 1023) // 1024 * 1024 +_SB_BH = _SB_BX + SH_GT * 2 * SHN * 128 +_SB_XC = _SB_BH + 6 * SHN * 128 +_SB_HC = _SB_XC + M_MAX * SH_KB +_SB_XCH = (_SB_HC + M_MAX * SH_DKB + 15) // 16 * 16 +_SB_GU = _SB_XCH + 64 * SHN * 4 +_SB_END = _SB_GU + 2 * 32 * SHN * 4 +SH_ACC_GU = 0 +SH_ACC_DN = 2 * SHN +SH_SFA = SH_ACC_DN + 2 * SHN +SH_SFB_GU = SH_SFA + 4 * 8 +SH_SFB_DN = SH_SFB_GU + 4 * SH_GT # capacity: fixed: a kernel tile constant +SH_TMEM = 256 # capacity: fixed: TMEM columns +if SHARED: + assert SH_STAGES - SH_NST <= 64, SH_STAGES + assert SH_NST >= 3 and _SB_END <= BIG and SH_SFB_DN + 24 <= SH_TMEM, (SH_NST, _SB_END, BIG) +TD_STRIDE = 64 +PFD = _cfg("pfd", 0) # L2 prefetch distance in ring stages (0: off) +PF_WARP = 12 # capacity: fixed: a warp role index +TICKET = bool(_cfg("ticket", 0)) # CTA roles by entry order +# priority tokens: token 0 and PT1 (M / 2: the second request's first row at spec c2) get their logits published first +NPT = _cfg("npt", 0) +PT1 = M_MAX // 2 +PF_PRIO = bool( + _cfg("pf_prio", 0) +) # L2 prefetch of the priority tokens' experts (this CTA's 1/NCTA of each) +PF_W1 = TWO_I * (H // 2) // NCTA +PF_S1 = TWO_I * (H // 16) // NCTA +PF_W2 = H * (I_TP // 2) // NCTA +PF_S2 = H * (I_TP // 16) // NCTA +assert PF_W1 % 16 == 0 and PF_S1 % 16 == 0 and PF_W2 % 16 == 0 and PF_S2 % 16 == 0 + + +# ============================================================================= +# Inline-PTX helpers. +# ============================================================================= +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +# ---- all-reduce push (the MoE boundary's multicast exchange, set A): +# arp = int64 [enable, mc, flags, rank, world, maxt] +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32_if(addr, value, pred, *, loc=None, ip=None): + """st.global.b32 when pred != 0.""" + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip), + cutlass.Int32(pred).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, $2, 0;\n\t@p st.global.b32 [$0], $1;\n\t}", "l,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _ar_push(arp, v, low, t, cl): + """Push bf16 channel pair (cl, cl + 1) of token t (v = channel cl's bf16 value on this lane, its neighbour's on + lane ^ 8) into this rank's set-A slot of the exchange; the slot half is the all-reduce CTA (t, cl // 768)'s call + parity. Only lanes with ``low`` store; every lane of the warp must call it (shuffle).""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + world = arp.load(idx=4) + parity = cutlass.Int64( + _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cs // cutlass.Int32(H // 8)) + ) + & cutlass.Int32(1) + ) + off = ( + parity * arp.load(idx=5) * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + arp.load(idx=3)) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(arp.load(idx=1) + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +@dsl_user_op +def _bar_arrive(bar_id: int, threads: int, *, loc=None, ip=None): + """bar.arrive: signal a named barrier without waiting (its memory ordering is bar.sync's).""" + _llvm.inline_asm( + None, + [], + f"bar.arrive {bar_id}, {threads};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _st_relaxed_u32(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "st.relaxed.gpu.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _st_relaxed_u64(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int64(val)], + "st.relaxed.gpu.global.u64 [$0], $1;", + "l,l", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm( + _T.i64(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u64 $0, [$1];", + "=l,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _nanosleep(ns: int, *, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + f"nanosleep.u32 {ns};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _gtime(*, loc=None, ip=None): + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), + [], + "mov.u64 $0, %globaltimer;", + "=l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _smid(*, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [], + "mov.u32 $0, %smid;", + "=r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@cute.jit +def _stamp(tdbg: cutlass.Array, bx, i: cutlass.Constexpr): + if cutlass.const_expr(TIMING): + tdbg.store(_gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(i)) + + +@dsl_user_op +def _rcp(a, *, loc=None, ip=None): + """rcp.approx.ftz.f32 (the reciprocal TRT-LLM's FP4 quantization uses).""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _div_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _sub_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "sub.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _mul_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3_bits(x, *, loc=None, ip=None): + """e4m3 (round to nearest even, saturating) of x as an int 0..255.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, 0f00000000, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3_to_f32(b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(b)], + "{\n.reg .b16 t, l, u;\n.reg .b32 h;\ncvt.u16.u32 t, $1;\ncvt.rn.f16x2.e4m3x2 h, t;\nmov.b32 {l, u}, h;\n" + "cvt.f32.f16 $0, l;\n}", + "=f,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e2m1x2(hi, lo, *, loc=None, ip=None): + """One byte of two E2M1 values (lo in the low nibble), round to nearest even, saturating.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b8 t;\ncvt.rn.satfinite.e2m1x2.f32 t, $1, $2;\ncvt.u32.u8 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2(x, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _bf16_round(x, *, loc=None, ip=None): + """x rounded to bf16 (nearest even), as f32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\nmov.b32 $0, {0, t};\n}", + "=f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _redux_max_u32(v, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(v)], + "redux.sync.max.u32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16 (nearest even) bits of x in the low 16 bits.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ue8m0_rp(x, *, loc=None, ip=None): + """UE8M0 code of x rounded toward +inf, saturating (__nv_cvt_float_to_e8m0(x, SATFINITE, RoundPosInf)).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rp.satfinite.ue8m0x2.f32 t, $1, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3x2(hi, lo, *, loc=None, ip=None): + """Two e4m3 (nearest even, saturating) in 16 bits, lo in the low byte.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, $1, $2;\ncvt.u32.u16 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _min_xorsign_abs(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "min.xorsign.abs.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _div_full(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.full.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2_full(x, *, loc=None, ip=None): + """ex2.approx.f32 (no flush to zero), Triton's tl.exp.""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _l2_prefetch(addr, nbytes, *, loc=None, ip=None): + """cp.async.bulk.prefetch.L2 of nbytes (a multiple of 16) at global address addr (16-byte aligned).""" + _llvm.inline_asm( + None, + [ + cutlass.Int64(addr).ir_value(loc=loc, ip=ip), + cutlass.Int32(nbytes).ir_value(loc=loc, ip=ip), + ], + "cp.async.bulk.prefetch.L2.global [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +def _mx_osc(amax): + """mxfp8 quantization of a block with absolute max amax: (UE8M0 code, the element multiplier).""" + code = _ue8m0_rp(_mul_rn(amax, _rcp(cutlass.Float32(448.0)))) + cf = cutlass.Int32( + cutlass.select_( + code == cutlass.Int32(255), + cutlass.Int32(0x7FFFFFFF), + cutlass.select_( + code == cutlass.Int32(0), cutlass.Int32(0x00400000), code << cutlass.Int32(23) + ), + ) + ).bitcast(cutlass.Float32) + return code, cutlass.select_(amax != cutlass.Float32(0.0), _rcp(cf), cutlass.Float32(0.0)) + + +def _mx_words(v, osc): + """e4m3 words of len(v) (a multiple of 4) values times osc, in element order.""" + words = [] + for t in range(len(v) // 4): + lo = _e4m3x2(_mul_rn(v[4 * t + 1], osc), _mul_rn(v[4 * t], osc)) + hi = _e4m3x2(_mul_rn(v[4 * t + 3], osc), _mul_rn(v[4 * t + 2], osc)) + words.append(lo | (hi << cutlass.Int32(16))) + return words + + +def _twiddle(score): + """Float bits mapped so that unsigned order is float order (cub TwiddleIn).""" + bits = ( + score.bitcast(cutlass.Int32) + if hasattr(score, "bitcast") + else cutlass.Float32(score).bitcast(cutlass.Int32) + ) + return cutlass.select_(bits < cutlass.Int32(0), ~bits, bits | cutlass.Int32(-2147483648)) + + +@cute.jit +def _wait(bar: cutlass.Array, parity): + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +def _tptr(base_row, col, dtype=cutlass.Int32): + return cutlass.inttoptr((base_row << 16) | col, 6, dtype) + + +@dsl_user_op +def _match_any(value, *, loc=None, ip=None): + """The mask of the warp's lanes holding the same 32-bit value.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(value)], + "match.any.sync.b32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@cute.jit +def _waitd(bar: cutlass.Array, parity, tdbg: cutlass.Array, site: cutlass.Constexpr): + """_wait, or (HANG_DBG) give up after ~2^22 polls and record the site in tdbg[bx, 24 + warp].""" + if cutlass.const_expr(HANG_DBG): + n = cutlass.Int32(0) + while (not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity)) & ( + n < cutlass.Int32(1 << 22) + ): + n = n + cutlass.Int32(1) + if n >= cutlass.Int32(1 << 22): + bx, _, _ = cute.arch.block_idx() + w = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + tdbg.store( + cutlass.Int64(site), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + w + ) + else: + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +@cute.jit +def _adv(stage, phase): + """The next ring position's (stage, phase).""" + stage = stage + cutlass.Int32(1) + wrap = stage == cutlass.Int32(STAGES) + return cutlass.select_(wrap, cutlass.Int32(0), stage), cutlass.select_( + wrap, phase ^ cutlass.Int32(1), phase + ) + + +@cute.jit +def _seg(i0, i1, k): + """Segment k of the FC1 item range [i0, i1) (items = unit-major stages): (unit, first stage, end stage). Order: + the last unit (its head, the part a split unit's next CTA does not cover), then the first unit, then the rest.""" + uf = i0 // cutlass.Int32(S1) + ul = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) + unit = uf + k - cutlass.Int32(1) + unit = cutlass.select_(k == cutlass.Int32(0), ul, unit) + unit = cutlass.select_(ul == uf, uf, unit) + j0 = i0 - unit * cutlass.Int32(S1) + j0 = cutlass.select_(j0 < cutlass.Int32(0), cutlass.Int32(0), j0) + j1 = i1 - unit * cutlass.Int32(S1) + j1 = cutlass.select_(j1 > cutlass.Int32(S1), cutlass.Int32(S1), j1) + return unit, j0, j1 + + +@cute.jit +def _nsegs(i0, i1): + n = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) - i0 // cutlass.Int32(S1) + cutlass.Int32(1) + return cutlass.select_(i1 > i0, n, cutlass.Int32(0)) + + +@cute.jit +def _sfb1_to_tmem(warp, lane, base_row, base_col, src: cutlass.Array, b1_ready: cutlass.Array): + """B1 scales to TMEM: instruction (j, kb)'s column holds, at lane n = QS q + t, token t's 4 scales of k-tile + 4 j + q, K 64 kb .. (src: int32 words [M, 96] of the token scales); each warp writes its sub-partition.""" + # B rows n >= 32 (N1 = 64) go to the atom's next 32-bit column (row n: lane n % 32, column n // 32) + for w in cutlass.range_constexpr((N1 + 31) // 32): + n = lane + cutlass.Int32(32 * w) + q = n // cutlass.Int32(QS) + tl = n - q * cutlass.Int32(QS) + sf_ok = (n < cutlass.Int32(N1)) & (tl < cutlass.Int32(M_MAX)) + sfv = [] + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + j = a // NUM_KB + kb = a % NUM_KB + wsf = src.load( + idx=cutlass.select_( + sf_ok, + tl * cutlass.Int32(H // 64) + + (cutlass.Int32(4 * j) + q) * cutlass.Int32(4) + + cutlass.Int32(kb), + cutlass.Int32(0), + ) + ) + sfv.append(cutlass.select_(sf_ok, cutlass.Int32(wsf), cutlass.Int32(0))) + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + prims.tcgen05_st( + "32x32b", + _tptr( + base_row + warp * cutlass.Int32(32), + base_col + SFB1_COL + a * 4 + w * SFB1_WSTEP, + ), + sfv[a], + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(b1_ready) + + +@cute.jit +def _b1_addr(t, kb16): + """Word index in the resident B tiles of the 8 FP4 bytes of token t's 16-value block kb16.""" + kt = kb16 >> cutlass.Int32(4) + j = kt >> cutlass.Int32(2) + n = (kt & cutlass.Int32(3)) * cutlass.Int32(QS) + t + cc = (kb16 >> cutlass.Int32(1)) & cutlass.Int32(7) + return ( + j * cutlass.Int32(NUM_BYTES_B1) + + n * cutlass.Int32(KT_BYTES) + + (cc ^ (n & cutlass.Int32(7))) * cutlass.Int32(16) + + (kb16 & cutlass.Int32(1)) * cutlass.Int32(8) + ) // cutlass.Int32(4) + + +@cute.jit +def _swiglu_quant(xv, gate_raw, is_up, a, c, alpha, beta, lim): + """TRTLLM-Gen's FC1 epilogue (the swiGlu Bmm E2m1 output), operation for operation: lane value xv is the up row, + gate_raw its gate row (fp32 sums). Returns (q, the scale's e4m3 bits) of the 16-row NVFP4 block (lanes r, r ^ 16 + and r ^ 8 cover it; amax over the warp's up lanes of a half). a = fc31_alpha, c = fc31_scale_c.""" + neg_max = cutlass.Float32(-3.40282346638528859812e38) + gc = cutlass.select_(gate_raw > lim, lim, gate_raw) + gc = cutlass.select_(gate_raw < neg_max, neg_max, gc) + uc = cutlass.select_(xv > lim, lim, xv) + uc = cutlass.select_(xv < -lim, -lim, uc) + c1 = _mul_rn(a, _mul_rn(alpha, cutlass.Float32(1.4426950216293334961))) + sgm = _rcp(_add_rn(_ex2(-_mul_rn(gc, c1)), cutlass.Float32(1.0))) + gs = _mul_rn(gc, sgm) + h = _mul_rn(_fma_rn(uc, a, _mul_rn(a, beta)), gs) + h = cutlass.select_(is_up, h, cutlass.Float32(0.0)) + amax = cute.arch.fmax(_mul_rn(cute.math.abs(h), c), cutlass.Float32(0.0)) + for off in (16, 8, 4, 2, 1): + amax = cute.arch.fmax(amax, cute.arch.shuffle_sync_bfly(amax, off)) + sfb = _e4m3_bits(_mul_rn(amax, cutlass.Float32(0.16666667163372039795))) + sff = _e4m3_to_f32(sfb) + osc = cutlass.select_(sff != cutlass.Float32(0.0), _rcp(sff), cutlass.Float32(0.0)) + return _mul_rn(_mul_rn(h, osc), c), sfb + + +@cute.jit +def _plan_range(s_plan: cutlass.Array, p): + w = s_plan.load(idx=p) + return w & cutlass.Int32(0xFFFF), w >> cutlass.Int32(16) + + +@cute.jit +def _sh_issue( + lane, + sid, + g, + s_big: cutlass.Array, + sh_full: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, + gpos, +): + """Stage g of the shared CTA's stream into ring slot g % SH_NST (warp 0). g < SH_GT: gate_up k-tile g of both + slices: quadrant Q = 2 q + u (q slice, u 0 gate / 1 up) = rows 32 sid .. (+ 768 for up) at k-tile SH_GT q + g, its + raw scale atom at raw slot Q. Else down tile j = (g - SH_GT) / 6 (rows 256 sid + 128 j ..), k-tile + (g - SH_GT) % 6.""" + st = gpos % cutlass.Int32(SH_NST) # ring position (stage g of this pass) + bar = sh_full.subview(st) + if cutlass.const_expr(SH_PRETILE): + if lane == 0: + src = (sid * cutlass.Int32(SH_STAGES) + g) * cutlass.Int32(SH_STAGE_BYTES) + prims.mbarrier_arrive_expect_tx(bar, SH_STAGE_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), gusf.subview(src), bar, 16384 + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + gusf.subview(src + cutlass.Int32(16384)), + bar, + 512, + ) + cute.arch.sync_warp() + else: + if g < cutlass.Int32(SH_GT): + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 4 * 512) + cute.arch.sync_warp() + if lane < 4: + q = lane >> cutlass.Int32(1) + u = lane & cutlass.Int32(1) + row0 = sid * cutlass.Int32(32) + u * cutlass.Int32(SH_I) + kt = q * cutlass.Int32(SH_GT) + g + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A) + lane * cutlass.Int32(4096)), + tma_gu, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=SH_HINT, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview( + cutlass.Int32(_SB_RAW) + + st * cutlass.Int32(2048) + + lane * cutlass.Int32(512) + ), + gusf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_KB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(6) + kt = gd - jt * cutlass.Int32(6) + row0 = sid * cutlass.Int32(256) + jt * cutlass.Int32(128) + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 512) + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), + tma_dn, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=SH_HINT, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_RAW) + st * cutlass.Int32(2048)), + dnsf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_DKB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + cute.arch.sync_warp() + + +@cute.jit +def _shared_cta( + tidx, + warp, + lane, + sid, + s_big: cutlass.Array, + sh_bar: cutlass.Array, + epochs: cutlass.Array, + tdbg: cutlass.Array, + bx, + xb32: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, + tmem_ptr_i32: cutlass.Array, + gu8: cutlass.Array, + dn8: cutlass.Array, +): + """Shared CTA sid (see SHARED): tcgen05 MXF8 (E4M3 x E4M3, UE8M0 per 32 K), m3_mx_gemv's tile order. gate_up: one + 128-row MMA tile = 2 K slices x (32 gate + 32 up rows), B rows 8 q + t = token t's slice-q k-tile; down: 2 tiles + of 128 rows, full K. Warps: 0 weights (TMA), 1 TMEM + scale splice + MMA, 4-7 epilogue; 2.. activation quant.""" + big32 = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Int32, alignment=16) + bigf = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Float32, alignment=16) + big8 = cutlass.Array(s_big.data_ptr(0), shape=(BIG,), dtype=cutlass.Int8, alignment=16) + sh_full = sh_bar # [0, SH_NST): stage full + sh_empty = cutlass.Array(sh_bar.data_ptr(8), shape=(8,), dtype=cutlass.Int64, alignment=8) + b_gu_rdy = sh_bar.subview(16) + b_dn_rdy = sh_bar.subview(17) + acc_gu = sh_bar.subview(18) + acc_dn = sh_bar.subview(19) + if warp == 0: + prims.prefetch_tensormap(tma_gu) + prims.prefetch_tensormap(tma_dn) + if lane < SH_NST: + prims.mbarrier_init(sh_full.subview(lane), 1) + prims.mbarrier_init( + sh_empty.subview(lane), 33 + ) # the MMAs' commit + each splicer lane's reads done + if lane == 0: + prims.mbarrier_init(b_gu_rdy, 1) + prims.mbarrier_init(b_dn_rdy, 1) + prims.mbarrier_init(acc_gu, 1) + prims.mbarrier_init(acc_dn, 1) + if warp == 1: + prims.tcgen05_alloc(tmem_ptr_i32, SH_TMEM) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + if warp == 0: + for g in cutlass.range_constexpr(SH_NST): # layer constants: before the grid dependency + _sh_issue( + lane, + sid, + cutlass.Int32(g), + s_big, + sh_full, + tma_gu, + tma_dn, + gusf, + dnsf, + cutlass.Int32(g), + ) + # the rest of this CTA's weights into L2 (64 gate / up rows of 6 KB, 256 down rows of 768 B) + if cutlass.const_expr(SH_PRETILE): + # this CTA's stages beyond the ring, 4 per lane + for r in cutlass.range_constexpr(2): + pg = cutlass.Int32(SH_NST) + lane * cutlass.Int32(2) + cutlass.Int32(r) + if pg < cutlass.Int32(SH_STAGES): + _l2_prefetch( + gusf.subview( + (sid * cutlass.Int32(SH_STAGES) + pg) * cutlass.Int32(SH_STAGE_BYTES) + ) + .data_ptr() + .toint(), + SH_STAGE_BYTES, + ) + elif cutlass.const_expr(not SH_PF_POST): + for r in cutlass.range_constexpr(2): + prw = sid * cutlass.Int32(32) + cutlass.Int32(r * SH_I) + lane + _l2_prefetch(gu8.subview(prw * cutlass.Int32(H)).data_ptr().toint(), H) + if lane == 0: + _l2_prefetch( + dn8.subview(sid * cutlass.Int32(256 * SH_I)).data_ptr().toint(), 256 * SH_I + ) + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(SH_PF_POST): + if warp == 0: # after the grid dependency: DRAM is idle during the routed front + for r in cutlass.range_constexpr(2): + prw = sid * cutlass.Int32(32) + cutlass.Int32(r * SH_I) + lane + _l2_prefetch(gu8.subview(prw * cutlass.Int32(H)).data_ptr().toint(), H) + if lane == 0: + _l2_prefetch( + dn8.subview(sid * cutlass.Int32(256 * SH_I)).data_ptr().toint(), 256 * SH_I + ) + if cutlass.const_expr(not LATE_TRIGGER): + if arp.load(idx=0) == cutlass.Int64(0): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + if cutlass.const_expr(EARLY_EPOCH): + prims.barrier_cta_sync(0) # every thread holds this call's epoch + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + NQ = THREADS - 64 # activation quant threads (warps 2 ..) + for ps in cutlass.range_constexpr(SH_NPASS): + TT0 = 0 # one pass: B rows SHN q + t (gate_up), t (down) + MS = M_MAX + GB = ps * SH_STAGES # ring positions before this pass + PAR = ps & 1 # parity of the per-pass barriers (B ready, accumulators) + if warp == 0: + for g in cutlass.range(SH_NST if ps == 0 else 0, SH_STAGES, unroll=1): + gg = g + cutlass.Int32(GB) # ring position over the passes + _wait( + sh_empty.subview(gg % cutlass.Int32(SH_NST)), + ((gg // cutlass.Int32(SH_NST)) - cutlass.Int32(1)) & cutlass.Int32(1), + ) + _sh_issue(lane, sid, g, s_big, sh_full, tma_gu, tma_dn, gusf, dnsf, gg) + elif warp == 1: + # ---- MMA: gate_up (scale atoms spliced to the tile's row order), then down + idesc_gu = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=2 * SHN, + m_dim=128, + ) + idesc_dn = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=SHN, + m_dim=128, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + desc_a0 = prims.Tcgen05SmemDesc.build( + s_big.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bx = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BX), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bh = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BH), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + sq = sid & cutlass.Int32(3) + _wait(b_gu_rdy, PAR) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for g in cutlass.range(SH_STAGES, unroll=1): + gg = g + cutlass.Int32(GB) + st = gg % cutlass.Int32(SH_NST) + if g == cutlass.Int32(SH_GT): + _wait(b_dn_rdy, PAR) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + _wait(sh_full.subview(st), (gg // cutlass.Int32(SH_NST)) & cutlass.Int32(1)) + raw = cutlass.Int32(_SB_RAW // 4) + st * cutlass.Int32(512) + if (g < cutlass.Int32(SH_GT)) & cutlass.Boolean(not SH_PRETILE): + # MMA row 32 Q + i = quadrant Q's row i: word 4 i + Q <- raw atom Q's word 4 i + sid % 4 + w4 = [ + big32.load(idx=raw + cutlass.Int32(128 * qq) + lane * cutlass.Int32(4) + sq) + for qq in range(4) + ] + big32.store( + ( + cutlass.Int32(w4[0]), + cutlass.Int32(w4[1]), + cutlass.Int32(w4[2]), + cutlass.Int32(w4[3]), + ), + idx=cutlass.Int32(_SB_SPL // 4) + + st * cutlass.Int32(128) + + lane * cutlass.Int32(4), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + prims.mbarrier_arrive( + sh_empty.subview(st) + ) # this lane's generic reads of the slot are done + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + if g < cutlass.Int32(SH_GT): + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bx + cutlass.Int64(g * cutlass.Int32((2 * SHN * 128) >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr(tcol + SH_ACC_GU, 6, cutlass.Float32), + da + 2 * kb, + db + 2 * kb, + idesc_gu.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (g > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_GU + g * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_GT - 1): + prims.tcgen05_commit(acc_gu) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(6) + kt = gd - jt * cutlass.Int32(6) + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview( + cutlass.Int32(_SB_SPL if SH_PRETILE else _SB_RAW) + + st * cutlass.Int32(512 if SH_PRETILE else 2048) + ), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bh + cutlass.Int64(kt * cutlass.Int32((SHN * 128) >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr( + tcol + SH_ACC_DN + jt * cutlass.Int32(SHN), 6, cutlass.Float32 + ), + da + 2 * kb, + db + 2 * kb, + idesc_dn.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (kt > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_DN + kt * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_STAGES - 1): + prims.tcgen05_commit(acc_dn) + cute.arch.sync_warp() + if warp >= 2: + if cutlass.const_expr(ps > 0): + _wait( + acc_dn, PAR ^ 1 + ) # the previous pass's MMAs are done: its B tiles and TMEM scales are free + # ---- the activation, MXFP8 per 32-K block (native mxfp8_quantize), into B (row 8 q + t of stage tile i = + # token t's k-tile SH_GT q + i, 128-byte swizzle) and its codes + qt = tidx - cutlass.Int32(64) + for r in cutlass.range_constexpr((MS * SH_KB + NQ - 1) // NQ): + q_ = qt + cutlass.Int32(r * NQ) + if q_ < cutlass.Int32(MS * SH_KB): + t = q_ // cutlass.Int32(SH_KB) + b = q_ - t * cutlass.Int32(SH_KB) + wv = [ + xb32.load( + idx=(t + cutlass.Int32(TT0)) * cutlass.Int32(H // 2) + + b * cutlass.Int32(16) + + cutlass.Int32(4 * z), + vector_size=4, + alignment=16, + ) + for z in range(4) + ] + v = [] + for z in cutlass.range_constexpr(4): + for zz in cutlass.range_constexpr(4): + w = cutlass.Int32(wv[z][zz]) + v.append((w << cutlass.Int32(16)).bitcast(cutlass.Float32)) + v.append((w & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(v[z])) + code, osc = _mx_osc(amax) + words = _mx_words(v, osc) + kt = b >> cutlass.Int32(2) + sl = kt // cutlass.Int32(SH_GT) + it = kt - sl * cutlass.Int32(SH_GT) + n = sl * cutlass.Int32(SHN) + t + c0 = (b & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BX) + + it * cutlass.Int32(2 * SHN * 128) + + n * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + + (((c0 + cutlass.Int32(1)) ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_XC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR, number_of_threads=NQ) + if warp >= 4: + if warp < 8: + qd = warp - cutlass.Int32(4) + # B scales of gate_up to TMEM: column of stage i, lane n = 8 q + t: token t's codes of + # k-tile SH_GT q + i + n = lane + t = n & cutlass.Int32(SHN - 1) + ok = (n < cutlass.Int32(2 * SHN)) & (t < cutlass.Int32(MS)) + for i in cutlass.range_constexpr(SH_GT): + kt = (n // cutlass.Int32(SHN)) * cutlass.Int32(SH_GT) + cutlass.Int32(i) + wsf = big32.load( + idx=( + cutlass.Int32(_SB_XC) + + cutlass.select_(ok, t, cutlass.Int32(0)) * cutlass.Int32(SH_KB) + + kt * cutlass.Int32(4) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_GU + i * 4), + cutlass.select_(ok, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + et = tidx - cutlass.Int32(128) + if et == 0: + prims.mbarrier_arrive(b_gu_rdy) + _stamp(tdbg, bx, 54) + # ---- gate_up out: quadrant qd lane l = MMA row 32 qd + l; slice q = qd / 2 holds token t + # at column 8 q + t + _wait(acc_gu, PAR) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + av = prims.tcgen05_ld( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_ACC_GU, cutlass.Float32), + num=2 * SHN, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(2): + for tt in cutlass.range_constexpr(MS): + bigf.store( + cutlass.Float32(av[SHN + tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(2)) * cutlass.Int32(32) + lane) + * cutlass.Int32(SHN) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd < cutlass.Int32(2): + # m3_mx_gemv's owner sum: (0 + rank 0) + rank 1, then bf16 + for tt in cutlass.range_constexpr(MS): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(SHN) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(SHN) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + # h = silu_and_mul(gate, up) of positions 32 sid + p, published under the call's tag + for r in cutlass.range_constexpr((32 * MS + 127) // 128): + pq = et + cutlass.Int32(r * 128) + if pq < cutlass.Int32(32 * MS): + p = pq // cutlass.Int32(MS) + tt = pq - p * cutlass.Int32(MS) + ga = bigf.load(idx=cutlass.Int32(_SB_GU // 4) + p * cutlass.Int32(SHN) + tt) + up = bigf.load( + idx=cutlass.Int32(_SB_GU // 4) + + (cutlass.Int32(32) + p) * cutlass.Int32(SHN) + + tt + ) + q_a = cute.arch.fmin(cutlass.Float32(ga), cutlass.Float32(7.0)) + q_bu = _min_xorsign_abs(cutlass.Float32(up), cutlass.Float32(7.0)) + q_ub = _add_rn(q_bu, cutlass.Float32(1.0)) + q_e2 = _ex2_full( + _mul_rn( + _mul_rn(q_a, cutlass.Float32(-1.70200002193450927734)), + cutlass.Float32(1.4426950216293334961), + ) + ) + q_sg = _div_full(cutlass.Float32(1.0), _add_rn(q_e2, cutlass.Float32(1.0))) + q_h = _mul_rn(q_ub, _mul_rn(q_sg, q_a)) + _st_relaxed_u32( + hx.subview( + (tt + cutlass.Int32(TT0)) * cutlass.Int32(SH_I) + + sid * cutlass.Int32(32) + + p + ) + .data_ptr() + .toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(q_h), + ) + if et == 0: + _stamp(tdbg, bx, 50) + # every shared CTA's h: (token, K block) per thread, 32 tagged words, MXFP8 into B (h) and its codes + for rr in cutlass.range_constexpr((MS * SH_DKB + 127) // 128): + q_ = et + cutlass.Int32(rr * 128) + if q_ < cutlass.Int32(MS * SH_DKB): + t = q_ // cutlass.Int32(SH_DKB) + kb = q_ - t * cutlass.Int32(SH_DKB) + base = (t + cutlass.Int32(TT0)) * cutlass.Int32(SH_I) + kb * cutlass.Int32( + 32 + ) + hv = [cutlass.Int32(0) for _ in range(32)] + pend = cutlass.Int32(1) + while pend != cutlass.Int32(0): + for q4 in cutlass.range_constexpr(8): + v4 = prims.load_ext( + hx.subview(base + cutlass.Int32(4 * q4)), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for z in cutlass.range_constexpr(4): + hv[4 * q4 + z] = cutlass.Int32(v4[z]) + okw = cutlass.Boolean(True) + for z in cutlass.range_constexpr(32): + okw = okw & ( + ((hv[z] >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) == tag16 + ) + pend = cutlass.select_(okw, cutlass.Int32(0), cutlass.Int32(1)) + vals = [ + (hv[z] << cutlass.Int32(16)).bitcast(cutlass.Float32) for z in range(32) + ] + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(vals[z])) + code, osc = _mx_osc(amax) + words = _mx_words(vals, osc) + kt = kb >> cutlass.Int32(2) + c0 = (kb & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BH) + + kt * cutlass.Int32(SHN * 128) + + t * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ (t & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + + ( + ((c0 + cutlass.Int32(1)) ^ (t & cutlass.Int32(7))) + << cutlass.Int32(2) + ), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_HC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + okh = lane < cutlass.Int32(MS) + for kt in cutlass.range_constexpr(6): + wsf = big32.load( + idx=( + cutlass.Int32(_SB_HC) + + cutlass.select_(okh, lane, cutlass.Int32(0)) * cutlass.Int32(SH_DKB) + + cutlass.Int32(4 * kt) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_DN + kt * 4), + cutlass.select_(okh, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if et == 0: + prims.mbarrier_arrive(b_dn_rdy) + _stamp(tdbg, bx, 51) + # ---- down out: tile j row 32 qd + l -> y_sh row 256 sid + 128 j + 32 qd + l + _wait(acc_dn, PAR) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for jt in cutlass.range_constexpr(2): + dv = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_DN + jt * SHN, + cutlass.Float32, + ), + num=SHN, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + rowo = ( + sid * cutlass.Int32(256) + + cutlass.Int32(128 * jt) + + qd * cutlass.Int32(32) + + lane + ) + for tt in cutlass.range_constexpr(MS): + _st_relaxed_u32( + dnx.subview(cutlass.Int32((tt + TT0) * H) + rowo).data_ptr().toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(cutlass.Float32(dv[tt])), + ) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if et == 0: + _stamp(tdbg, bx, 52) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if warp == 1: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc(cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), SH_TMEM) + if cutlass.const_expr(not EARLY_EPOCH): + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + + +# ============================================================================= +# v2 helpers: global release / acquire, proxy fences. +# ============================================================================= +@dsl_user_op +def _red_release_add(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "red.release.gpu.global.add.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _atom_add_u64(addr, val, *, loc=None, ip=None): + return cutlass.Int64( + _asm( + _T.i64(), + [cutlass.Int64(addr), cutlass.Int64(val)], + "atom.relaxed.gpu.global.add.u64 $0, [$1], $2;", + "=l,l,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ld_acquire_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.acquire.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fence_proxy_async_global(*, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + "fence.proxy.async.global;", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _fence_acq_rel_gpu(*, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + "fence.acq_rel.gpu;", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@cute.jit +def _dbg_delay(salt): + """(DBG_DELAY stress builds) a pseudo-random ~0..4 us stall: protocol robustness under skew.""" + if cutlass.const_expr(DBG_DELAY): + r = cutlass.Int32(_gtime() >> cutlass.Int64(6)) * cutlass.Int32(1103515245) + salt + if ((r >> cutlass.Int32(12)) & cutlass.Int32(7)) == cutlass.Int32(0): + for _ in cutlass.range((r >> cutlass.Int32(16)) & cutlass.Int32(31), unroll=1): + _nanosleep(128) + + +@cute.jit +def _waitt(bar: cutlass.Array, parity, acc): + """_wait that (TIMING) adds the waited ns to acc.""" + t0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + if cutlass.const_expr(TIMING): + acc = acc + (_gtime() - t0) + return acc + + +@cute.jit +def _tstore(tdbg: cutlass.Array, bx, i: cutlass.Constexpr, v): + if cutlass.const_expr(TIMING): + tdbg.store(cutlass.Int64(v), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(i)) + + +@cute.jit +def _issue_fc1( + lane, + stage, + e, + c, + j, + sA: cutlass.Array, + sSF: cutlass.Array, + ab_full: cutlass.Array, + tma_a1, + tma_s1, + s1w: cutlass.Array, +): + """FC1 stage j of unit c of expert e into ring stage `stage` (the caller waited its empty barrier): 4 TMA boxes + (rows 32 c .., k-tiles 4 j + q) and the stage's scale atoms (SFA_CP: word [r][q] of atom kb <- raw atom + (k-tile 4 j + q, kb) word [r][c % 4], by this warp's 32 lanes).""" + row = c * cutlass.Int32(R1) + kx = j * cutlass.Int32(C1 * KT_BYTES) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview(stage * cutlass.Int32(NUM_BYTES_A) + cutlass.Int32(q * NUM_BYTES_BOX1)), + tma_a1, + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) * cutlass.Int32(SF1_KA) + + j * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + prims.cp_async_bulk_tensor_shared_cta_global( + sSF.subview(stage * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1, + (cutlass.Int32(0), sf_line), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + atom0 = (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) * cutlass.Int32( + SF1_KA + ) + j * cutlass.Int32(16) + src0 = atom0 * cutlass.Int32(128) + lane * cutlass.Int32(4) + (c & cutlass.Int32(3)) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for q in cutlass.range_constexpr(C1): + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSF.subview(dst0 + cutlass.Int32(kb * 128 + q)), + s1w.subview(src0 + cutlass.Int32((4 * q + kb) * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + # one group per ring stage (racecheck: the issuer waits it before the slot's reuse) + prims.cp_async_commit_group() + cute.arch.sync_warp() + + +@cute.jit +def _issue_fc2( + lane, + stage, + g, + blk, + t2: cutlass.Constexpr, + n_here, + s_el: cutlass.Array, + sA: cutlass.Array, + sSF: cutlass.Array, + ab_full: cutlass.Array, + tma_a2, + tma_s2, + s2w: cutlass.Array, +): + """FC2 k-tile t2 of output block blk for the group's n_here slots: box jj = rows 32 blk .. of slot jj's w2 at + k-tile t2; scale word [r][jj] of atom kb <- raw atom (rows / 128, K block 4 t2 + kb) word [r][blk % 4].""" + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + n_here + * cutlass.Int32(NUM_BYTES_BOX1 if SFA_CP else NUM_BYTES_BOX1 + NUM_BYTES_SFA_RAW // 4), + ) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + cutlass.Int32(jj * NUM_BYTES_BOX1) + ), + tma_a2, + (cutlass.Int32(t2 * KT_BYTES), blk * cutlass.Int32(R2), e), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSF.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + cutlass.Int32(jj * NUM_WORDS_SFA_RAW // 4) + ), + tma_s2, + ( + cutlass.Int32(0), + ( + (e * cutlass.Int32(SF2_RB) + (blk >> cutlass.Int32(2))) + * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(4), + ), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + src0 = ( + ( + (e * cutlass.Int32(SF2_RB) + (blk >> cutlass.Int32(2))) + * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (blk & cutlass.Int32(3)) + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSF.subview(dst0 + cutlass.Int32(kb * 128 + jj)), + s2w.subview(src0 + cutlass.Int32(kb * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + # one group per ring stage (racecheck: the issuer waits it before the slot's reuse) + prims.cp_async_commit_group() + cute.arch.sync_warp() + + +@cute.jit +def _quant_words(w, sc, rsc, rsix): + """NVFP4 of one 16-value block given as 8 bf16-pair words: (lo word, hi word, scale e4m3 bits).""" + xs = [] + for z in cutlass.range_constexpr(8): + xs.append((w[z] << cutlass.Int32(16)).bitcast(cutlass.Float32)) + xs.append((w[z] & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + vmax = cute.math.abs(xs[0]) + for z in cutlass.range_constexpr(1, 16): + vmax = cute.arch.fmax(vmax, cute.math.abs(xs[z])) + sfb = _e4m3_bits(_mul_rn(sc, _mul_rn(vmax, rsix))) + osc = cutlass.select_( + vmax != cutlass.Float32(0.0), _rcp(_mul_rn(_e4m3_to_f32(sfb), rsc)), cutlass.Float32(0.0) + ) + by = [ + _e2m1x2(_mul_rn(xs[2 * z + 1], osc), _mul_rn(xs[2 * z], osc)) & cutlass.Int32(0xFF) + for z in range(8) + ] + lo = ( + by[0] + | (by[1] << cutlass.Int32(8)) + | (by[2] << cutlass.Int32(16)) + | (by[3] << cutlass.Int32(24)) + ) + hi = ( + by[4] + | (by[5] << cutlass.Int32(8)) + | (by[6] << cutlass.Int32(16)) + | (by[7] << cutlass.Int32(24)) + ) + return lo, hi, sfb + + +@cute.jit +def _quant_block(xs32: cutlass.Array, widx, sc, rsc, rsix): + """NVFP4 of one 16-value block (8 bf16 words at widx): (lo word, hi word, scale e4m3 bits), fp4_quantize's ops.""" + w = [] + for h in cutlass.range_constexpr(2): + v4 = xs32.load(idx=widx + cutlass.Int32(4 * h), vector_size=4, alignment=16) + for z in cutlass.range_constexpr(4): + w.append(cutlass.Int32(v4[z])) + xs = [] + for z in cutlass.range_constexpr(8): + xs.append((w[z] << cutlass.Int32(16)).bitcast(cutlass.Float32)) + xs.append((w[z] & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + vmax = cute.math.abs(xs[0]) + for z in cutlass.range_constexpr(1, 16): + vmax = cute.arch.fmax(vmax, cute.math.abs(xs[z])) + sfb = _e4m3_bits(_mul_rn(sc, _mul_rn(vmax, rsix))) + osc = cutlass.select_( + vmax != cutlass.Float32(0.0), _rcp(_mul_rn(_e4m3_to_f32(sfb), rsc)), cutlass.Float32(0.0) + ) + by = [ + _e2m1x2(_mul_rn(xs[2 * z + 1], osc), _mul_rn(xs[2 * z], osc)) & cutlass.Int32(0xFF) + for z in range(8) + ] + lo = ( + by[0] + | (by[1] << cutlass.Int32(8)) + | (by[2] << cutlass.Int32(16)) + | (by[3] << cutlass.Int32(24)) + ) + hi = ( + by[4] + | (by[5] << cutlass.Int32(8)) + | (by[6] << cutlass.Int32(16)) + | (by[7] << cutlass.Int32(24)) + ) + return lo, hi, sfb + + +@cute.kernel +def m3_moe_kernel( + tma_a1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_a2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_gu_desc: cutlass.GridConstant[cuda.TensorMap], + tma_dn_desc: cutlass.GridConstant[cuda.TensorMap], + bias: cutlass.Array, # f32 [128] + escal: cutlass.Array, # f32 [6, 128] + plan: cutlass.Array, # int32 [4 M + 1, NCTA, M + 1] + partials: cutlass.Array, # int32 bits of f32 [G_CAP * 48, M, 32]: split units' head sums + img32: cutlass.Array, # int32 [NG_CAP, IMG_BYTES / 4]: per-group FC2 B image (tile + scale atoms) + cnt: cutlass.Array, # int32 [NG_CAP * 8]: per-group unit counters (index 8 g) + s1w: cutlass.Array, # int32 words of w3_w1_weight_scale + s2w: cutlass.Array, # int32 words of w2_weight_scale + w1b: cutlass.Array, # int8 bytes of w3_w1_weight (L2 prefetch) + w2b: cutlass.Array, # int8 bytes of w2_weight (L2 prefetch) + epochs: cutlass.Array, # int32 [NCTA + NSH] + out: cutlass.Array, # bf16 [M, 6144] + route_dbg: cutlass.Array, # int32 [M, 8] + tdbg: cutlass.Array, # int64 [NCTA + NSH, 64] + xb32: cutlass.Array, # int32 words (bf16 pairs) of the hidden states [M, 6144] + rw: cutlass.Array, # f32 router weight [128, 6144] + sfsc: cutlass.Array, # f32 [1] fc31_input_scale + lgbuf: cutlass.Array, # int64 [LG_REPL, LG_STRIDE] + guw: cutlass.Array, + gusf: cutlass.Array, + dnw: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, +): + tidx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tidx % 32 + if cutlass.const_expr(TICKET): + # role by entry order: the CTAs that become resident last (behind the predecessor's CTAs) take the shared + # expert roles (most slack), the first NCTA the routed ones (the router GEMV / logits exchange) + s_tk = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + if tidx == 0: + old = _atom_add_u64( + cnt.subview(cutlass.Int32(NG_CAP * 8 + 8)).data_ptr().toint(), cutlass.Int64(1) + ) + s_tk.store(cutlass.Int32(old % cutlass.Int64(NCTA + NSH)), idx=0) + prims.barrier_cta_sync(0) + bx = s_tk.load(idx=0) + if tidx == 0: + _stamp(tdbg, bx, 0) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(_smid()), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(63) + ) + + s_big = cutlass.Array(cutlass.Int8, BIG, space=cutlass.AddressSpace.smem, alignment=1024) + sA = cutlass.Array( + s_big.data_ptr(0), shape=(NUM_BYTES_A * STAGES,), dtype=cutlass.Int8, alignment=1024 + ) + # ring scale words: SFA_CP: the instruction atoms [STAGES][4 x 512 B]; else raw boxes [STAGES][8 KB] + spliced + sSF = cutlass.Array( + s_big.data_ptr(_OFF_SFA), + shape=(STAGES * STAGE_SF // 4,), + dtype=cutlass.Int32, + alignment=1024, + ) + sSFA = cutlass.Array( + s_big.data_ptr(_OFF_SFA + (0 if SFA_CP else STAGES * NUM_BYTES_SFA_RAW)), + shape=(NUM_WORDS_SFA * STAGES,), + dtype=cutlass.Int32, + alignment=512, + ) + sB1 = cutlass.Array( + s_big.data_ptr(_OFF_B1), shape=(NUM_BYTES_B1 * S1,), dtype=cutlass.Int8, alignment=1024 + ) + sB2 = cutlass.Array( + s_big.data_ptr(_OFF_B2), shape=(NB2BUF * IMG_B2,), dtype=cutlass.Int8, alignment=1024 + ) + sSFB2 = cutlass.Array( + s_big.data_ptr(_OFF_SFB2), shape=(IMG_SF // 4,), dtype=cutlass.Int32, alignment=1024 + ) + ys = cutlass.Array( + cutlass.BFloat16, + NB2_CAP * M_MAX * TOP_K * R2, + space=cutlass.AddressSpace.smem, + alignment=16, + ) # bf16 values + s_part = cutlass.Array( + cutlass.Float32, SPB * M_MAX * C1 * R1, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_bias = cutlass.Array(cutlass.Float32, E_LOCAL, space=cutlass.AddressSpace.smem, alignment=16) + s_red = cutlass.Array( + cutlass.Float32, 9 * M_MAX + 16, space=cutlass.AddressSpace.smem, alignment=16 + ) + # the activation's scale codes [M, 384] until they are in TMEM: aliases ys (first written by the FC2 epilogue) + s_xsf = cutlass.Array( + ys.data_ptr(0), shape=(M_MAX * (H // 64),), dtype=cutlass.Int32, alignment=16 + ) + ab_full = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + ab_empty = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + sfa_ready = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + acc_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_ready = cutlass.Array(cutlass.Int64, NB2BUF, space=cutlass.AddressSpace.smem, alignment=8) + b2_free = cutlass.Array(cutlass.Int64, NB2BUF, space=cutlass.AddressSpace.smem, alignment=8) + img_bar = cutlass.Array(cutlass.Int64, NB2BUF, space=cutlass.AddressSpace.smem, alignment=8) + sfb2_free = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + b1_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + x_full = cutlass.Array(cutlass.Int64, NXB, space=cutlass.AddressSpace.smem, alignment=8) + x_empty = cutlass.Array(cutlass.Int64, NXB, space=cutlass.AddressSpace.smem, alignment=8) + route_bar = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + ys_done = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + grp_done = cutlass.Array(cutlass.Int64, 4, space=cutlass.AddressSpace.smem, alignment=8) + b1_load = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + topk_bar = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + tok0_bar = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + px_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + s_prog = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + s_el = cutlass.Array(cutlass.Int32, G_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + s_kpos = cutlass.Array( + cutlass.Int32, G_CAP * M_MAX, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_w = cutlass.Array( + cutlass.Float32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_ids = cutlass.Array( + cutlass.Int32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_plan = cutlass.Array(cutlass.Int32, NG_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + s_gn = cutlass.Array( + cutlass.Int32, NG_CAP, space=cutlass.AddressSpace.smem, alignment=16 + ) # slots per group + s_emap = cutlass.Array(cutlass.Int32, E_LOCAL, space=cutlass.AddressSpace.smem, alignment=16) + s_misc = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_plan_all = cutlass.Array( + cutlass.Int32, (G_CAP + 1) * (NG_CAP + 1), space=cutlass.AddressSpace.smem, alignment=16 + ) + s_esc = cutlass.Array(cutlass.Float32, 8 * G_CAP, space=cutlass.AddressSpace.smem, alignment=32) + + is_routed = cutlass.Boolean(True) + if cutlass.const_expr(SHARED): + sh_full = cutlass.Array(cutlass.Int64, 32, space=cutlass.AddressSpace.smem, alignment=8) + is_routed = bx < cutlass.Int32(NCTA) + if bx >= cutlass.Int32(NCTA): + _shared_cta( + tidx, + warp, + lane, + bx - cutlass.Int32(NCTA), + s_big, + sh_full, + epochs, + tdbg, + bx, + xb32, + tma_gu_desc.get_ptr(), + tma_dn_desc.get_ptr(), + gusf, + dnsf, + hx, + dnx, + arp, + tmem_ptr_i32, + guw, + dnw, + ) + if is_routed: + # ---- before the grid dependency + if warp == 0: + if tidx < STAGES: + prims.mbarrier_init(ab_full.subview(tidx), 33 if SFA_CP else 1) + prims.mbarrier_init(ab_empty.subview(tidx), 1) + prims.mbarrier_init(sfa_ready.subview(tidx), 1) + if tidx < 2: + prims.mbarrier_init(acc_full.subview(tidx), 1) + prims.mbarrier_init(acc_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(acc2_full.subview(tidx), 1) + prims.mbarrier_init(acc2_empty.subview(tidx), EPI_THREADS) + if tidx < NB2BUF: + prims.mbarrier_init(b2_ready.subview(tidx), 1) + prims.mbarrier_init(b2_free.subview(tidx), 1) + prims.mbarrier_init(img_bar.subview(tidx), 1) + if tidx < NXB: + prims.mbarrier_init(x_full.subview(tidx), 1) + prims.mbarrier_init(x_empty.subview(tidx), NWARPS) + if tidx == 0: + prims.mbarrier_init(b1_ready.subview(0), EPI_THREADS) + prims.mbarrier_init(sfb2_free.subview(0), 1) + prims.mbarrier_init(route_bar.subview(0), 32) + prims.mbarrier_init(ys_done.subview(0), EPI_THREADS) + prims.mbarrier_init(b1_load.subview(0), 1) + prims.mbarrier_init(topk_bar.subview(0), M_MAX * TOP_K) + prims.mbarrier_init(tok0_bar.subview(0), TOP_K) + prims.mbarrier_init(px_full.subview(0), 1) + if tidx < 4: + prims.mbarrier_init(grp_done.subview(tidx), EPI_THREADS) + if tidx == 0: + s_prog.store(cutlass.Int32(0), idx=0) + if warp == X_WARP: # slot tables: empty (the slot warp fills them after the top-k) + for z in cutlass.range_constexpr((G_CAP * M_MAX + 31) // 32): + if lane + cutlass.Int32(32 * z) < cutlass.Int32(G_CAP * M_MAX): + s_kpos.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + for z in cutlass.range_constexpr(E_LOCAL // 32): + s_emap.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + if warp == MMA_WARP: + prims.tcgen05_alloc(tmem_ptr_i32, TMEM_COLS) + prims.tcgen05_relinquish_alloc_permit() + if warp == W_WARP: + prims.prefetch_tensormap(tma_a1_desc.get_ptr()) + prims.prefetch_tensormap(tma_a2_desc.get_ptr()) + if cutlass.const_expr(not SFA_CP): + prims.prefetch_tensormap(tma_s1_desc.get_ptr()) + prims.prefetch_tensormap(tma_s2_desc.get_ptr()) + for i in cutlass.range_constexpr(E_LOCAL // 32): + s_bias.store(cutlass.Float32(bias.load(idx=lane + i * 32)), idx=lane + i * 32) + for i in cutlass.range_constexpr(((G_CAP + 1) * (NG_CAP + 1) + 31) // 32): + q = lane + cutlass.Int32(32 * i) + if q < cutlass.Int32((G_CAP + 1) * (NG_CAP + 1)): + gq = q // cutlass.Int32(NG_CAP + 1) + s_plan_all.store( + plan.load( + idx=(gq * cutlass.Int32(NCTA) + bx) * cutlass.Int32(NG_CAP + 1) + + q + - gq * cutlass.Int32(NG_CAP + 1) + ), + idx=q, + ) + # router weights (expert bx) in registers: thread tg = tid % 256 owns elements 4 tg + 1024 j + tg = tidx % cutlass.Int32(GEMV_THREADS) + half = tidx // cutlass.Int32(GEMV_THREADS) + rwv = [ + rw.load( + idx=bx * cutlass.Int32(H) + tg * cutlass.Int32(4) + cutlass.Int32(1024 * j), + vector_size=4, + alignment=16, + ) + for j in range(H // 1024) + ] + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + + cute.arch.griddepcontrol_wait() + # ---- x chunks into the idle ring (warp 6): chunk c = K [XCK c, XCK (c + 1)) of every token + if warp == S_WARP: + if cutlass.const_expr(NPT > 0): + # the priority tokens' rows first (their logits are published ahead of the others) + if lane == 0: + prims.mbarrier_arrive_expect_tx(px_full.subview(0), NPT * H * 2) + cute.arch.sync_warp() + if lane < NPT: + prims.cp_async_bulk_shared_cluster_global( + sB1.subview(lane * cutlass.Int32(H * 2)), + xb32.subview( + cutlass.select_( + lane == cutlass.Int32(0), cutlass.Int32(0), cutlass.Int32(PT1) + ) + * cutlass.Int32(H // 2) + ), + px_full.subview(0), + H * 2, + ) + for c in cutlass.range_constexpr(NXB): + if lane == 0: + prims.mbarrier_arrive_expect_tx(x_full.subview(c), XCB) + cute.arch.sync_warp() + if lane < M_MAX: + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(c * XCB) + lane * cutlass.Int32(XCK * 2)), + xb32.subview(lane * cutlass.Int32(H // 2) + cutlass.Int32(c * XCK // 2)), + x_full.subview(c), + XCK * 2, + ) + ar_on = arp.load(idx=0) != cutlass.Int64(0) + if cutlass.const_expr(not LATE_TRIGGER): + if arp.load(idx=0) == cutlass.Int64(0): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + ltag = cutlass.Int64(ep + cutlass.Int32(1)) << cutlass.Int64(32) + cnt_target = (ep + cutlass.Int32(1)) * cutlass.Int32(CNT_PER_CALL) + + sc = cutlass.Float32(sfsc.load(idx=0)) + rsc = _rcp(sc) + rsix = _rcp(cutlass.Float32(6.0)) + img8f = cutlass.Array( + img32.data_ptr(0), shape=(IMG_TOTAL,), dtype=cutlass.Int8, alignment=16 + ) + if cutlass.const_expr(NPT > 0): + # ---- priority tokens (0 and PT1): their logits first (m3_router_gemv's per-thread / warp order) + _wait(px_full.subview(0), 0) + ph = cutlass.select_(half < cutlass.Int32(NPT), half, cutlass.Int32(0)) + pacc = cutlass.Float32(0.0) + px32 = cutlass.Array( + sB1.data_ptr(0), shape=(NPT * H // 2,), dtype=cutlass.Int32, alignment=16 + ) + for j in cutlass.range_constexpr(H // 1024): + pxw = px32.load( + idx=ph * cutlass.Int32(H // 2) + cutlass.Int32(512 * j) + tg * cutlass.Int32(2), + vector_size=2, + alignment=8, + ) + px0 = cutlass.Int32(pxw[0]) + px1 = cutlass.Int32(pxw[1]) + pacc = _fma_rn( + (px0 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[j][0]), + pacc, + ) + pacc = _fma_rn( + (px0 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[j][1]), + pacc, + ) + pacc = _fma_rn( + (px1 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[j][2]), + pacc, + ) + pacc = _fma_rn( + (px1 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[j][3]), + pacc, + ) + for off in (16, 8, 4, 2, 1): + pacc = _add_rn(pacc, cute.arch.shuffle_sync_bfly(pacc, off)) + if lane == 0: + s_red.store(pacc, idx=cutlass.Int32(9 * M_MAX) + warp) # [half][8 warps] + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ALL_BAR, number_of_threads=THREADS) + if tidx < cutlass.Int32(NPT * LG_REPL): + hh = tidx // cutlass.Int32(LG_REPL) + rr = tidx - hh * cutlass.Int32(LG_REPL) + sacc = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(8): + sacc = _add_rn( + sacc, s_red.load(idx=cutlass.Int32(9 * M_MAX + w) + hh * cutlass.Int32(8)) + ) + tp = cutlass.select_(hh == cutlass.Int32(0), cutlass.Int32(0), cutlass.Int32(PT1)) + _st_relaxed_u64( + lgbuf.subview(rr * cutlass.Int32(LG_STRIDE) + tp * cutlass.Int32(E_LOCAL) + bx) + .data_ptr() + .toint(), + ltag | (cutlass.Int64(sacc.bitcast(cutlass.Int32)) & cutlass.Int64(0xFFFFFFFF)), + ) + if tidx == 0: + _stamp(tdbg, bx, 43) + # ---- this CTA's share of the NVFP4 activation (QB blocks, from global) for the global B1 image: loads first + QB = (M_MAX * (H // 16) + NCTA - 1) // NCTA + qb = bx * cutlass.Int32(QB) + tidx + q_ok = (tidx < cutlass.Int32(QB)) & (qb < cutlass.Int32(M_MAX * (H // 16))) + qb = cutlass.select_(q_ok, qb, cutlass.Int32(0)) + qtok = qb // cutlass.Int32(H // 16) + qkb = qb - qtok * cutlass.Int32(H // 16) + xq_v = [ + xb32.load( + idx=qtok * cutlass.Int32(H // 2) + qkb * cutlass.Int32(8) + cutlass.Int32(4 * h), + vector_size=4, + alignment=16, + ) + for h in range(2) + ] + # ---- router GEMV from the staged chunks, all 512 threads (rolled: the front runs once per call) + MH = (M_MAX + 1) // 2 # tokens per GEMV half + acc_r = cute.make_rmem_tensor((MH,), cutlass.Float32) + for i in cutlass.range_constexpr(MH): + acc_r[i] = cutlass.Float32(0.0) + big32 = cutlass.Array( + s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Int32, alignment=16 + ) + JPC = XCK // 1024 # router j-steps per chunk + for xc in cutlass.range(NXC, unroll=1): + buf = xc % cutlass.Int32(NXB) + _wait(x_full.subview(buf), (xc // cutlass.Int32(NXB)) & cutlass.Int32(1)) + if (tidx == 0) & (xc == cutlass.Int32(0)): + _stamp(tdbg, bx, 41) + xw0 = buf * cutlass.Int32(XCB // 4) # chunk word base in s_big + for jj in cutlass.range_constexpr(JPC): + jg = xc * cutlass.Int32(JPC) + cutlass.Int32(jj) + wsel = [cutlass.Float32(rwv[0][z]) for z in range(4)] + for jx in cutlass.range_constexpr(1, H // 1024): + for z in cutlass.range_constexpr(4): + wsel[z] = cutlass.select_( + jg == cutlass.Int32(jx), cutlass.Float32(rwv[jx][z]), wsel[z] + ) + for i in cutlass.range_constexpr(MH): + t = half * cutlass.Int32(MH) + cutlass.Int32(i) + t = cutlass.select_(t < cutlass.Int32(M_MAX), t, cutlass.Int32(M_MAX - 1)) + xw = big32.load( + idx=xw0 + + t * cutlass.Int32(XCK // 2) + + cutlass.Int32(512 * jj) + + tg * cutlass.Int32(2), + vector_size=2, + alignment=8, + ) + x0 = cutlass.Int32(xw[0]) + x1 = cutlass.Int32(xw[1]) + acc_g = acc_r[i] + acc_g = _fma_rn( + (x0 << cutlass.Int32(16)).bitcast(cutlass.Float32), wsel[0], acc_g + ) + acc_g = _fma_rn( + (x0 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), wsel[1], acc_g + ) + acc_g = _fma_rn( + (x1 << cutlass.Int32(16)).bitcast(cutlass.Float32), wsel[2], acc_g + ) + acc_g = _fma_rn( + (x1 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), wsel[3], acc_g + ) + acc_r[i] = acc_g + if cutlass.const_expr(NXB < NXC): + if xc + cutlass.Int32(NXB) < cutlass.Int32(NXC): + # every thread's reads of this chunk are done (CTA barrier, then a generic -> async proxy fence by + # the refilling warp: the bulk copy overwrites what the generic proxy read); racecheck-clean + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ALL_BAR, number_of_threads=THREADS) + if warp == S_WARP: + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + if lane == 0: + prims.mbarrier_arrive_expect_tx(x_full.subview(buf), XCB) + cute.arch.sync_warp() + if lane < M_MAX: + prims.cp_async_bulk_shared_cluster_global( + s_big.subview( + buf * cutlass.Int32(XCB) + lane * cutlass.Int32(XCK * 2) + ), + xb32.subview( + lane * cutlass.Int32(H // 2) + + (xc + cutlass.Int32(NXB)) * cutlass.Int32(XCK // 2) + ), + x_full.subview(buf), + XCK * 2, + ) + if tidx == 0: + _stamp(tdbg, bx, 42) + for i in cutlass.range_constexpr(MH): + v = acc_r[i] + for off in (16, 8, 4, 2, 1): + v = _add_rn(v, cute.arch.shuffle_sync_bfly(v, off)) + t = half * cutlass.Int32(MH) + cutlass.Int32(i) + if (lane == 0) & (t < cutlass.Int32(M_MAX)): + s_red.store(v, idx=(warp % cutlass.Int32(8)) * cutlass.Int32(M_MAX) + t) + # this CTA's quantization blocks into the global B1 image (B1 tile bytes, then the [M, 384] scale codes) + if q_ok: + qw = [] + for h in cutlass.range_constexpr(2): + for z in cutlass.range_constexpr(4): + qw.append(cutlass.Int32(xq_v[h][z])) + qlo, qhi, qsf = _quant_words(qw, sc, rsc, rsix) + img32.store( + (qlo, qhi), idx=cutlass.Int32(B1IMG_OFF // 4) + _b1_addr(qtok, qkb), alignment=8 + ) + img8f.store(cutlass.Int8(qsf), idx=cutlass.Int32(B1IMG_OFF + S1 * NUM_BYTES_B1) + qb) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ALL_BAR, number_of_threads=THREADS) + if cutlass.const_expr(EARLY_EPOCH): + if tidx == 0: + epochs.store( + ep + cutlass.Int32(1), idx=bx + ) # every thread read it before this CTA barrier + if warp == cutlass.Int32(0): + _dbg_delay(bx * cutlass.Int32(3)) + if tidx < LG_REPL: + for t in cutlass.range_constexpr(M_MAX): + sacc = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(8): + sacc = _add_rn(sacc, s_red.load(idx=cutlass.Int32(w * M_MAX + t))) + lw = ltag | (cutlass.Int64(sacc.bitcast(cutlass.Int32)) & cutlass.Int64(0xFFFFFFFF)) + _st_relaxed_u64( + lgbuf.subview(tidx * cutlass.Int32(LG_STRIDE) + cutlass.Int32(t * E_LOCAL) + bx) + .data_ptr() + .toint(), + lw, + ) + if tidx == 0: + _stamp(tdbg, bx, 21) + if tidx == cutlass.Int32( + THREADS - 32 + ): # B1 share released after the logits (its gpu fence stalls warp 15 only) + if cutlass.const_expr(EPI_PFENCE): + _fence_proxy_async_global() # once, after the join (ALL_BAR) of the writer threads + _red_release_add( + cnt.subview(cutlass.Int32(NG_CAP * 8)).data_ptr().toint(), cutlass.Int32(1) + ) + # ---- the B1 image (every CTA's share) into smem (warp 13; the top-k warps are 0 .. M - 1) + if cutlass.const_expr(B1_WARP >= 0): + if warp == cutlass.Int32(B1_WARP): + if lane == 0: + b1addr = cnt.subview(cutlass.Int32(NG_CAP * 8)).data_ptr().toint() + b1tgt = (ep + cutlass.Int32(1)) * cutlass.Int32(NCTA) + cv = _ld_acquire_u32(b1addr) + while (cv - b1tgt) < cutlass.Int32(0): + _nanosleep(64) + cv = _ld_acquire_u32(b1addr) + _fence_proxy_async_global() + prims.mbarrier_arrive_expect_tx( + b1_load.subview(0), S1 * NUM_BYTES_B1 + M_MAX * (H // 16) + ) + prims.cp_async_bulk_shared_cluster_global( + sB1.subview(0), + img32.subview(cutlass.Int32(B1IMG_OFF // 4)), + b1_load.subview(0), + S1 * NUM_BYTES_B1, + ) + prims.cp_async_bulk_shared_cluster_global( + s_xsf.subview(0), + img32.subview(cutlass.Int32((B1IMG_OFF + S1 * NUM_BYTES_B1) // 4)), + b1_load.subview(0), + M_MAX * (H // 16), + ) + _stamp(tdbg, bx, 23) + cute.arch.sync_warp() + + # ---- top-k: token t on warp t (keys sigmoid + bias, ties to the lower id), TRTLLM-Gen MiniMax2 ops + tk_t = cutlass.Int32(-1) # this warp's top-k token (TK_WARPS[t] = warp) + for i_tk in cutlass.range_constexpr(M_MAX): + tk_t = cutlass.select_(warp == cutlass.Int32(TK_WARPS[i_tk]), cutlass.Int32(i_tk), tk_t) + if tk_t >= cutlass.Int32(0): + t = tk_t + want = ltag >> cutlass.Int64(32) + rbase = ( + (bx % cutlass.Int32(LG_REPL)) * cutlass.Int32(LG_STRIDE) + + t * cutlass.Int32(E_LOCAL) + + lane + ) + done = cutlass.Int32(0) + res = [cutlass.Int64(0) for _ in range(4)] + while done == cutlass.Int32(0): + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(4): + res[z] = _ld_relaxed_u64( + lgbuf.subview(cutlass.Int32(z * 32) + rbase).data_ptr().toint() + ) + ok = ok & ((res[z] >> cutlass.Int64(32)) == want) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32(-1): + done = cutlass.Int32(1) + else: + _nanosleep(POLL_NS) + if (lane == 0) & (t == cutlass.Int32(0)): + _stamp(tdbg, bx, 22) + lgv = [cutlass.Int32(res[z]).bitcast(cutlass.Float32) for z in range(4)] + keys = [] + for i in cutlass.range_constexpr(4): + sg = _fma_rn( + cutlass.Float32(0.5), + cute.math.tanh(_mul_rn(cutlass.Float32(0.5), lgv[i])), + cutlass.Float32(0.5), + ) + keys.append(_add_rn(sg, s_bias.load(idx=i * 32 + lane))) + hk = [_twiddle(keys[i]) for i in range(4)] + taken = [cutlass.Boolean(False) for _ in range(4)] + ids_t = [] + kt_t = [] + for k in cutlass.range_constexpr(TOP_K): + bh = cutlass.Int32(0) + bl = cutlass.Int32(0) + for i in cutlass.range_constexpr(4): + lo_i = cutlass.Int32(65535 - i * 32) - lane + hv = cutlass.select_(taken[i], cutlass.Int32(0), hk[i]) + lv = cutlass.select_(taken[i], cutlass.Int32(0), lo_i) + better = (cutlass.Uint32(hv) > cutlass.Uint32(bh)) | ( + (hv == bh) & (cutlass.Uint32(lv) > cutlass.Uint32(bl)) + ) + bh = cutlass.select_(better, hv, bh) + bl = cutlass.select_(better, lv, bl) + mh = _redux_max_u32(bh) + ml = _redux_max_u32(cutlass.select_(bh == mh, bl, cutlass.Int32(0))) + e_k = cutlass.Int32(65535) - ml + for i in cutlass.range_constexpr(4): + taken[i] = taken[i] | (e_k == cutlass.Int32(i * 32) + lane) + ids_t.append(e_k) + kv = cutlass.Float32(0.0) + for i in cutlass.range_constexpr(4): + kv = cutlass.select_(e_k == cutlass.Int32(i * 32) + lane, keys[i], kv) + kt_t.append(cute.arch.shuffle_sync(kv, e_k % cutlass.Int32(32))) + sig = cutlass.Float32(0.0) + e_l = cutlass.Int32(0) + for k in cutlass.range_constexpr(TOP_K): + sk = _sub_rn(kt_t[k], s_bias.load(idx=ids_t[k])) + sig = cutlass.select_(lane == cutlass.Int32(k), sk, sig) + e_l = cutlass.select_(lane == cutlass.Int32(k), ids_t[k], e_l) + total = sig + for off in (16, 8, 4, 2, 1): + total = _add_rn(total, cute.arch.shuffle_sync_bfly(total, off)) + wv = _bf16_round( + _div_rn( + _mul_rn(sig, cutlass.Float32(ROUTE_SCALE)), + _add_rn(total, cutlass.Float32(SUM_EPS)), + ) + ) + if lane < TOP_K: + s_w.store(wv, idx=t * cutlass.Int32(TOP_K) + lane) + s_ids.store(e_l, idx=t * cutlass.Int32(TOP_K) + lane) + if bx == 0: + route_dbg.store( + wv.bitcast(cutlass.Int32), + idx=t * cutlass.Int32(8) + cutlass.Int32(4) + lane, + ) + if (lane == 0) & (t == cutlass.Int32(0)): + _stamp(tdbg, bx, 20) + if cutlass.const_expr(PF_PRIO): + if (t == cutlass.Int32(0)) | (t == cutlass.Int32(PT1)): + if lane < TOP_K: + # this CTA's 1/NCTA of each of the token's experts into L2 (DRAM is idle until the routing is + # complete; every byte is streamed later anyway) + _l2_prefetch( + w1b.subview( + e_l * cutlass.Int32(TWO_I * H // 2) + bx * cutlass.Int32(PF_W1) + ) + .data_ptr() + .toint(), + PF_W1, + ) + _l2_prefetch( + s1w.subview( + (e_l * cutlass.Int32(TWO_I * H // 16) + bx * cutlass.Int32(PF_S1)) + // cutlass.Int32(4) + ) + .data_ptr() + .toint(), + PF_S1, + ) + _l2_prefetch( + w2b.subview( + e_l * cutlass.Int32(H * I_TP // 2) + bx * cutlass.Int32(PF_W2) + ) + .data_ptr() + .toint(), + PF_W2, + ) + _l2_prefetch( + s2w.subview( + (e_l * cutlass.Int32(H * I_TP // 16) + bx * cutlass.Int32(PF_S2)) + // cutlass.Int32(4) + ) + .data_ptr() + .toint(), + PF_S2, + ) + cute.arch.sync_warp() + if lane < TOP_K: # every writer lane arrives (its own release of s_ids / s_w) + prims.mbarrier_arrive(topk_bar.subview(0)) # this token's ids / weights are in smem + if t == cutlass.Int32(0): + prims.mbarrier_arrive( + tok0_bar.subview(0) + ) # token 0's experts: the early group-0 stages + cute.arch.sync_warp() + + if cutlass.const_expr(EARLY0): + if warp == W_WARP: + # slots 0..3 = token 0's four distinct experts; group 0's stage ranges are the same for every G + _wait(tok0_bar.subview(0), 0) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(STAGES): + _issue_fc1( + lane, + n0, + e, + c, + j, + sA, + sSF, + ab_full, + tma_a1_desc.get_ptr(), + tma_s1_desc.get_ptr(), + s1w, + ) + n0 = n0 + cutlass.Int32(1) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + if warp == X_WARP: + # slots: entry i < 4 M is (token i / 4, top-k i % 4); distinct experts in entry order + NCH = (TOP_K * M_MAX + 31) // 32 + _wait(topk_bar.subview(0), 0) + gcnt = cutlass.Int32(0) + rec = [] # per chunk: (first, rank, slot rank of the entry, expert, entry valid) + for ch in cutlass.range_constexpr(NCH): + ei = lane + cutlass.Int32(32 * ch) + valid = ei < cutlass.Int32(TOP_K * M_MAX) + my_e = cutlass.select_( + valid, + s_ids.load(idx=cutlass.select_(valid, ei, cutlass.Int32(0))), + cutlass.Int32(-1) - lane, + ) + known = cutlass.Boolean(False) + prev = cutlass.Int32(-1) + key = my_e + if cutlass.const_expr(ch > 0): + prev = cutlass.select_( + valid, + s_emap.load(idx=cutlass.select_(valid, my_e, cutlass.Int32(0))), + cutlass.Int32(-1), + ) + known = prev >= cutlass.Int32(0) + key = cutlass.select_(known, cutlass.Int32(-1000) - lane, my_e) + cute.arch.sync_warp() # every lane's s_emap read before any lane's write + same = _match_any(key) + leader = cute.arch.popc((same & (cutlass.Int32(0) - same)) - cutlass.Int32(1)) + first = valid & (known == cutlass.Boolean(False)) & (lane == leader) + bal = prims.vote_sync(0xFFFFFFFF, first, prims.VoteSync.BALLOT) + rank = gcnt + cute.arch.popc(bal & cutlass.Int32(cute.arch.lanemask_lt())) + slot_l = cute.arch.shuffle_sync(rank, leader) + slot_l = cutlass.select_(known, prev, slot_l) + if first: + s_emap.store(rank, idx=my_e) + if valid: + if bx == 0: + route_dbg.store( + my_e, + idx=(ei // cutlass.Int32(TOP_K)) * cutlass.Int32(8) + + ei % cutlass.Int32(TOP_K), + ) + rec.append((first, rank, slot_l, my_e, valid)) + cute.arch.sync_warp() + gcnt = gcnt + cute.arch.popc(bal) + # slot of rank r: with a partial last group (tail < 4) and >= 3 groups, that group becomes group 1 (its + # FC2 then is not the last hand-off behind a short FC1 phase); the full groups 1.. shift up by one + ngr = (gcnt + cutlass.Int32(3)) >> cutlass.Int32(2) + tail = gcnt - (ngr - cutlass.Int32(1)) * cutlass.Int32(4) + do_perm = ( + cutlass.Boolean(PERM) + & (tail < cutlass.Int32(4)) + & (ngr >= cutlass.Int32(PERM_MIN_NG)) + ) + for z in cutlass.range_constexpr((G_CAP + 31) // 32): + if lane + cutlass.Int32(32 * z) < cutlass.Int32(G_CAP): + s_el.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + cute.arch.sync_warp() + for ch in cutlass.range_constexpr(NCH): + first, rank, slot_r, my_e, valid = rec[ch] + ei = lane + cutlass.Int32(32 * ch) + for which in cutlass.range_constexpr(2): + r = rank if which == 0 else slot_r + sl_m = cutlass.select_( + r < cutlass.Int32(4), + r, + cutlass.select_( + r >= (ngr - cutlass.Int32(1)) * cutlass.Int32(4), + r - (ngr - cutlass.Int32(2)) * cutlass.Int32(4), + r + cutlass.Int32(4), + ), + ) + sl_m = cutlass.select_(do_perm, sl_m, r) + if cutlass.const_expr(which == 0): + if first: + s_el.store(my_e, idx=sl_m) + else: + if valid: + s_kpos.store( + ei % cutlass.Int32(TOP_K), + idx=sl_m * cutlass.Int32(M_MAX) + ei // cutlass.Int32(TOP_K), + ) + # slots per group + if lane < cutlass.Int32(NG_CAP): + nh = gcnt - lane * cutlass.Int32(4) + nh = cutlass.select_(nh > cutlass.Int32(4), cutlass.Int32(4), nh) + nh = cutlass.select_(nh < cutlass.Int32(0), cutlass.Int32(0), nh) + nh = cutlass.select_(do_perm & (lane == cutlass.Int32(1)), tail, nh) + nh = cutlass.select_( + do_perm & (lane == ngr - cutlass.Int32(1)), cutlass.Int32(4), nh + ) + s_gn.store(nh, idx=lane) + cute.arch.sync_warp() + if lane == 0: + s_el.store(gcnt, idx=G_CAP) + if lane < cutlass.Int32(NG_CAP + 1): + s_plan.store(s_plan_all.load(idx=gcnt * cutlass.Int32(NG_CAP + 1) + lane), idx=lane) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 24) + prims.mbarrier_arrive( + route_bar.subview(0) + ) # every lane: its slots / kpos / plan-row writes are in smem + cute.arch.sync_warp() + # CTA 0 pads every group counter to exactly CNT_PER_CALL for this call (absent slots, unused groups) + if (bx == 0) & (lane < cutlass.Int32(NG_CAP)): + nh = s_gn.load(idx=lane) + pad = cutlass.Int32(CNT_PER_CALL) - nh * cutlass.Int32(UNITS_ROW) + if pad > cutlass.Int32(0): + _red_release_add(cnt.subview(lane * cutlass.Int32(8)).data_ptr().toint(), pad) + else: + _wait(route_bar.subview(0), 0) + if tidx == 0: + _stamp(tdbg, bx, 2) + if cutlass.const_expr(B1_WARP < 0): + if warp == cutlass.Int32(0): + if lane == 0: + b1addr = cnt.subview(cutlass.Int32(NG_CAP * 8)).data_ptr().toint() + b1tgt = (ep + cutlass.Int32(1)) * cutlass.Int32(NCTA) + cv = _ld_acquire_u32(b1addr) + while (cv - b1tgt) < cutlass.Int32(0): + _nanosleep(64) + cv = _ld_acquire_u32(b1addr) + _fence_proxy_async_global() + prims.mbarrier_arrive_expect_tx( + b1_load.subview(0), S1 * NUM_BYTES_B1 + M_MAX * (H // 16) + ) + prims.cp_async_bulk_shared_cluster_global( + sB1.subview(0), + img32.subview(cutlass.Int32(B1IMG_OFF // 4)), + b1_load.subview(0), + S1 * NUM_BYTES_B1, + ) + prims.cp_async_bulk_shared_cluster_global( + s_xsf.subview(0), + img32.subview(cutlass.Int32((B1IMG_OFF + S1 * NUM_BYTES_B1) // 4)), + b1_load.subview(0), + M_MAX * (H // 16), + ) + _stamp(tdbg, bx, 23) + cute.arch.sync_warp() + if warp < 4: # B1 scales to TMEM (the MMA needs them only with the first ring stage) + _wait(b1_load.subview(0), 0) + _sfb1_to_tmem(warp, lane, base_row, base_col, s_xsf, b1_ready) + + g_n = s_el.load(idx=G_CAP) + ng = (g_n + cutlass.Int32(3)) >> cutlass.Int32(2) + pr2 = s_plan.load(idx=NG_CAP) + blk0 = pr2 & cutlass.Int32(0xFFFF) + nb = pr2 >> cutlass.Int32(16) + + # ---- weights producer (warp 4): phase p = FC1 stages of group p, then FC2 tiles of group p - LAG + if warp == W_WARP: + stage = cutlass.Int32(0) + phase = cutlass.Int32(1) + skip = cutlass.Int32(0) + n_iss = cutlass.Int32( + 0 + ) # stages issued (early ones included): the L2-prefetch warp's throttle + w_emp = cutlass.Int64(0) + if cutlass.const_expr(EARLY0): + skip = s_misc.load(idx=0) + stage = cutlass.select_(skip == cutlass.Int32(STAGES), cutlass.Int32(0), skip) + phase = cutlass.select_( + skip == cutlass.Int32(STAGES), cutlass.Int32(0), cutlass.Int32(1) + ) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_el.load(idx=slot) + for j in cutlass.range(j0, j1, unroll=1): + if skip > cutlass.Int32(0): + skip = skip - cutlass.Int32(1) + else: + w_emp = _waitt(ab_empty.subview(stage), phase, w_emp) + prims.cp_async_wait_group(STAGES - 1) + _dbg_delay(bx) + _issue_fc1( + lane, + stage, + e, + c, + j, + sA, + sSF, + ab_full, + tma_a1_desc.get_ptr(), + tma_s1_desc.get_ptr(), + s1w, + ) + stage, phase = _adv(stage, phase) + if cutlass.const_expr(PFD > 0): + n_iss = n_iss + cutlass.Int32(1) + if lane == 0: + prims.store_ext( + n_iss.ir_value(), + s_prog.subview(0), + order="relaxed", + scope="cta", + ) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + n_here = s_gn.load(idx=g) + for bb in range(nb): + blk = blk0 + bb + for t2 in cutlass.range_constexpr(KT2): + w_emp = _waitt(ab_empty.subview(stage), phase, w_emp) + prims.cp_async_wait_group(STAGES - 1) + _issue_fc2( + lane, + stage, + g, + blk, + t2, + n_here, + s_el, + sA, + sSF, + ab_full, + tma_a2_desc.get_ptr(), + tma_s2_desc.get_ptr(), + s2w, + ) + stage, phase = _adv(stage, phase) + if cutlass.const_expr(PFD > 0): + n_iss = n_iss + cutlass.Int32(1) + if lane == 0: + prims.store_ext( + n_iss.ir_value(), + s_prog.subview(0), + order="relaxed", + scope="cta", + ) + if lane == 0: + _stamp(tdbg, bx, 3) + _tstore(tdbg, bx, 30, w_emp) + if cutlass.const_expr(PROD_TRIGGER): + if ar_on: + cute.arch.griddepcontrol_launch_dependents() + + # ---- (SFA_CP = 0) scale splicer (warp 6): atom kb word [r][q] = raw atom 4 q + kb word [r][m1] + if cutlass.const_expr(not SFA_CP): + if warp == S_WARP: + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + for p in range(ng + cutlass.Int32(LAG)): + i0 = cutlass.Int32(0) + i1 = cutlass.Int32(0) + for part in cutlass.range_constexpr(2): + cnt_ = cutlass.Int32(0) + m1 = cutlass.Int32(0) + nk = cutlass.Int32(0) + if cutlass.const_expr(part == 0): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + nk = _nsegs(i0, i1) + else: + if p >= cutlass.Int32(LAG): + nk = nb + for k in range(nk): + if cutlass.const_expr(part == 0): + ug, j0, j1 = _seg(i0, i1, k) + m1 = ( + ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + ) & cutlass.Int32(3) + cnt_ = j1 - j0 + else: + m1 = (blk0 + k) & cutlass.Int32(3) + cnt_ = cutlass.Int32(KT2) + for _j in cutlass.range(cnt_, unroll=1): + _waitd(ab_full.subview(stage), phase, tdbg, 3) + raw0 = ( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + lane * cutlass.Int32(4) + + m1 + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32( + 4 + ) + for kb in cutlass.range_constexpr(NUM_KB): + w0 = sSF.load(idx=raw0 + cutlass.Int32((0 * 4 + kb) * 128)) + w1 = sSF.load(idx=raw0 + cutlass.Int32((1 * 4 + kb) * 128)) + w2 = sSF.load(idx=raw0 + cutlass.Int32((2 * 4 + kb) * 128)) + w3 = sSF.load(idx=raw0 + cutlass.Int32((3 * 4 + kb) * 128)) + sSFA.store( + ( + cutlass.Int32(w0), + cutlass.Int32(w1), + cutlass.Int32(w2), + cutlass.Int32(w3), + ), + idx=dst0 + cutlass.Int32(kb * 128), + alignment=16, + ) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + cute.arch.sync_warp() + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + prims.tcgen05_commit(sfa_ready.subview(stage)) + stage, phase = _adv(stage, phase) + + # ---- FC2 B loader (warp 5): group g's image once its counter is complete + if warp == X_WARP: + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + for g in range(ng): + buf = g % cutlass.Int32(NB2BUF) + if g >= cutlass.Int32(NB2BUF): + _waitd( + b2_free.subview(buf), + ((g // cutlass.Int32(NB2BUF)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 29) + caddr = cnt.subview(g * cutlass.Int32(8)).data_ptr().toint() + if lane == 0: + cv = _ld_acquire_u32(caddr) + while (cv - cnt_target) < cutlass.Int32(0): + _nanosleep(B2_POLL_NS) + cv = _ld_acquire_u32(caddr) + cute.arch.sync_warp() + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 51) + _fence_proxy_async_global() + if g >= cutlass.Int32(1): + _waitd( + sfb2_free.subview(0), (g - cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 107 + ) + if lane == 0: + prims.mbarrier_arrive_expect_tx(img_bar.subview(buf), IMG_BYTES) + prims.cp_async_bulk_shared_cluster_global( + sB2.subview(buf * cutlass.Int32(IMG_B2)), + img32.subview(g * cutlass.Int32(IMG_BYTES // 4)), + img_bar.subview(buf), + IMG_B2, + ) + prims.cp_async_bulk_shared_cluster_global( + sSFB2.subview(0), + img32.subview( + g * cutlass.Int32(IMG_BYTES // 4) + cutlass.Int32(IMG_B2 // 4) + ), + img_bar.subview(buf), + IMG_SF, + ) + cute.arch.sync_warp() + _waitd( + img_bar.subview(buf), (g // cutlass.Int32(NB2BUF)) & cutlass.Int32(1), tdbg, 108 + ) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 52) + if g >= cutlass.Int32(1): + # the single TMEM scale slot: the previous group's FC2 MMAs are done + gp = g - cutlass.Int32(1) + _waitd( + b2_free.subview(gp % cutlass.Int32(NB2BUF)), + (gp // cutlass.Int32(NB2BUF)) & cutlass.Int32(1), + tdbg, + 109, + ) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 53) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for aa in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SFB2_COL + aa * 4), + sd0 + cutlass.Int64(aa * (512 >> 4)), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + prims.tcgen05_commit(sfb2_free.subview(0)) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 25) + + # ---- (PFD) L2 prefetch warp (12): the same stage sequence as the producer, PFD stages ahead of it + if cutlass.const_expr(PFD > 0): + if warp == cutlass.Int32(PF_WARP): + pf = cutlass.Int32(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_el.load(idx=slot) + for j in cutlass.range(j0, j1, unroll=1): + if lane == 0: + prg = cutlass.Int32( + prims.load_ext( + s_prog.subview(0), + dtype=cutlass.Int32, + order="relaxed", + scope="cta", + ) + ) + while prg + cutlass.Int32(PFD) <= pf: + _nanosleep(100) + prg = cutlass.Int32( + prims.load_ext( + s_prog.subview(0), + dtype=cutlass.Int32, + order="relaxed", + scope="cta", + ) + ) + if pf >= prg: + kx = j * cutlass.Int32(C1 * KT_BYTES) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_prefetch( + tma_a1_desc.get_ptr(), + [ + kx + cutlass.Int32(q * KT_BYTES), + c * cutlass.Int32(R1), + e, + ], + [], + ) + atom0 = ( + e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2)) + ) * cutlass.Int32(SF1_KA) + j * cutlass.Int32(16) + _l2_prefetch( + s1w.subview(atom0 * cutlass.Int32(128)) + .data_ptr() + .toint(), + 8192, + ) + cute.arch.sync_warp() + pf = pf + cutlass.Int32(1) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + n_here = s_gn.load(idx=g) + for bb in range(nb): + blk = blk0 + bb + for t2 in cutlass.range_constexpr(KT2): + if lane == 0: + prg = cutlass.Int32( + prims.load_ext( + s_prog.subview(0), + dtype=cutlass.Int32, + order="relaxed", + scope="cta", + ) + ) + while prg + cutlass.Int32(PFD) <= pf: + _nanosleep(100) + prg = cutlass.Int32( + prims.load_ext( + s_prog.subview(0), + dtype=cutlass.Int32, + order="relaxed", + scope="cta", + ) + ) + if pf >= prg: + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load( + idx=g * cutlass.Int32(4) + cutlass.Int32(jj) + ) + prims.cp_async_bulk_tensor_prefetch( + tma_a2_desc.get_ptr(), + [ + cutlass.Int32(t2 * KT_BYTES), + blk * cutlass.Int32(R2), + e, + ], + [], + ) + a2 = ( + e * cutlass.Int32(SF2_RB) + + (blk >> cutlass.Int32(2)) + ) * cutlass.Int32(SF2_KA) + cutlass.Int32(4 * t2) + _l2_prefetch( + s2w.subview(a2 * cutlass.Int32(128)) + .data_ptr() + .toint(), + 2048, + ) + cute.arch.sync_warp() + pf = pf + cutlass.Int32(1) + + # ---- (REL_WARP) group releases (warp 8): this CTA's finished units of group p, one release add per phase + if cutlass.const_expr(REL_WARP): + if warp == cutlass.Int32(8): + for p in range(ng): + i0, i1 = _plan_range(s_plan, p) + n_done = cutlass.Int32(0) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + n_done = n_done + cutlass.select_( + (j0 != cutlass.Int32(0)) | (j1 == cutlass.Int32(S1)), + cutlass.Int32(1), + cutlass.Int32(0), + ) + _wait( + grp_done.subview(p & cutlass.Int32(3)), + (p >> cutlass.Int32(2)) & cutlass.Int32(1), + ) + _dbg_delay(bx * cutlass.Int32(13) + p) + if (lane == 0) & (n_done > cutlass.Int32(0)): + if cutlass.const_expr(EPI_PFENCE): + _fence_proxy_async_global() + _red_release_add( + cnt.subview(p * cutlass.Int32(8)).data_ptr().toint(), n_done + ) + if (lane == 0) & (p == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 54) + cute.arch.sync_warp() + + # ---- MMA (warp 7) + if warp == MMA_WARP: + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + idesc1 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N1, + m_dim=MMA_M, + sparsity_version=0, + ) + idesc2 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N2, + m_dim=MMA_M, + sparsity_version=0, + ) + desc_a0 = prims.Tcgen05SmemDesc.build( + sA.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b10 = prims.Tcgen05SmemDesc.build( + sB1.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b20 = prims.Tcgen05SmemDesc.build( + sB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + _waitd(b1_ready, 0, tdbg, 5) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + m_b2 = cutlass.Int64(0) + m_full = cutlass.Int64(0) + m_acc = cutlass.Int64(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + m_acc = _waitt( + acc_empty.subview(ab), + ((k1 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + m_acc, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc_ptr = cutlass.inttoptr( + tcol + ACC1_COL + ab * cutlass.Int32(N1), 6, cutlass.Float32 + ) + for j in cutlass.range(j0, j1, unroll=1): + if cutlass.const_expr(SFA_CP): + m_full = _waitt(ab_full.subview(stage), phase, m_full) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 6) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b10 + cutlass.Int64(j * cutlass.Int32(NUM_BYTES_B1 >> 4)) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = tcol + SFB1_COL + j * cutlass.Int32(NUM_KB * 4) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc1, + (j > j0) | cutlass.Boolean(kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + cute.arch.sync_warp() + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc_full.subview(ab)) + cute.arch.sync_warp() + k1 = k1 + cutlass.Int32(1) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + buf = g % cutlass.Int32(NB2BUF) + m_b2_0 = m_b2 + m_b2 = _waitt( + b2_ready.subview(buf), (g // cutlass.Int32(NB2BUF)) & cutlass.Int32(1), m_b2 + ) + if cutlass.const_expr(TIMING): + if (lane == 0) & (g < cutlass.Int32(4)): + tdbg.store( + m_b2 - m_b2_0, + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(44) + g, + ) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 4) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + m_acc = _waitt( + acc2_empty.subview(a2), + ((k2 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + m_acc, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc2_ptr = cutlass.inttoptr( + tcol + ACC2_COL + a2 * cutlass.Int32(ACC2_STRIDE), 6, cutlass.Float32 + ) + for t2 in cutlass.range_constexpr(KT2): + if cutlass.const_expr(SFA_CP): + m_full = _waitt(ab_full.subview(stage), phase, m_full) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 7) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b20 + cutlass.Int64( + (buf * cutlass.Int32(KT2) + cutlass.Int32(t2)) + * cutlass.Int32(NUM_BYTES_B2 >> 4) + ) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = tcol + SFB2_COL + cutlass.Int32(t2 * NUM_KB * 4) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc2_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc2, + cutlass.Boolean(t2 > 0 or kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + cute.arch.sync_warp() + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc2_full.subview(a2)) + cute.arch.sync_warp() + k2 = k2 + cutlass.Int32(1) + if prims.elect_sync(): + prims.tcgen05_commit( + b2_free.subview(buf) + ) # this group's B2 tile and TMEM scales are free + cute.arch.sync_warp() + if lane == 0: + _tstore(tdbg, bx, 26, m_b2) + _tstore(tdbg, bx, 27, m_full) + _tstore(tdbg, bx, 28, m_acc) + + # ---- epilogue warps (0-3) + if warp < 4: + row_w = base_row + warp * 32 + img8 = cutlass.Array( + img32.data_ptr(0), shape=(NG_CAP * IMG_BYTES,), dtype=cutlass.Int8, alignment=16 + ) + for r in cutlass.range_constexpr((6 * G_CAP + EPI_THREADS - 1) // EPI_THREADS): + qi = tidx + cutlass.Int32(r * EPI_THREADS) + sl_e = qi // cutlass.Int32(6) + if qi < cutlass.Int32(6 * G_CAP): + e_sl = s_el.load(idx=sl_e) + if e_sl >= cutlass.Int32(0): + fld = qi - sl_e * cutlass.Int32(6) + s_esc.store( + cutlass.Float32(escal.load(idx=fld * cutlass.Int32(E_LOCAL) + e_sl)), + idx=sl_e * cutlass.Int32(8) + fld, + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + e_wait = cutlass.Int64(0) + e_busy = cutlass.Int64(0) + e_a = cutlass.Int64(0) + e_pw = cutlass.Int64(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + cblk = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + unit = slot * cutlass.Int32(U1) + cblk + unit_done = (j0 != cutlass.Int32(0)) | (j1 == cutlass.Int32(S1)) + e_wait = _waitt( + acc_full.subview(ab), + (k1 >> cutlass.Int32(1)) & cutlass.Int32(1), + e_wait, + ) + tb0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, base_col + ACC1_COL + ab * cutlass.Int32(N1), cutlass.Float32 + ), + num=N1_LD, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc_empty.subview(ab)) + for t in cutlass.range_constexpr(M_MAX): + part = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(4): + part = cutlass.select_( + warp == cutlass.Int32(w), cutlass.Float32(acc[QS * w + t]), part + ) + s_part.store(part, idx=(t * C1 + warp) * R1 + lane) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + tc0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(TIMING): + e_a = e_a + (tc0 - tb0) + g_u = slot >> cutlass.Int32(2) + jrow = slot & cutlass.Int32(3) + # this warp's tokens t = warp + 4 tr, all in flight together (latency-bound otherwise) + NT = (M_MAX + 3) // 4 + xr = cute.make_rmem_tensor((NT,), cutlass.Float32) + for tr in cutlass.range_constexpr(NT): + tq = cutlass.select_( + warp + cutlass.Int32(4 * tr) < cutlass.Int32(M_MAX), + warp + cutlass.Int32(4 * tr), + cutlass.Int32(0), + ) + xr[tr] = _add_rn( + _add_rn( + _add_rn( + s_part.load(idx=(tq * C1 + 0) * R1 + lane), + s_part.load(idx=(tq * C1 + 1) * R1 + lane), + ), + s_part.load(idx=(tq * C1 + 2) * R1 + lane), + ), + s_part.load(idx=(tq * C1 + 3) * R1 + lane), + ) + if (j0 == cutlass.Int32(0)) & (j1 != cutlass.Int32(S1)): + # head of a unit split with the next CTA: its partial sums + for tr in cutlass.range_constexpr(NT): + t = warp + cutlass.Int32(4 * tr) + if t < cutlass.Int32(M_MAX): + _st_relaxed_u32( + partials.subview( + (unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane + ) + .data_ptr() + .toint(), + xr[tr].bitcast(cutlass.Int32), + ) + else: + if j0 != cutlass.Int32(0): + # tail: the head's partial sums (head first, as TRTLLM-Gen's K order) + for tr in cutlass.range_constexpr(NT): + t = warp + cutlass.Int32(4 * tr) + if t < cutlass.Int32(M_MAX): + paddr = ( + partials.subview( + (unit * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(R1) + + lane + ) + .data_ptr() + .toint() + ) + pbits = _ld_relaxed_u32(paddr) + while pbits == cutlass.Int32(PARTIAL_EMPTY): + pbits = _ld_relaxed_u32(paddr) + _st_relaxed_u32(paddr, cutlass.Int32(PARTIAL_EMPTY)) + xr[tr] = _add_rn(pbits.bitcast(cutlass.Float32), xr[tr]) + cute.arch.sync_warp() + _dbg_delay(bx * cutlass.Int32(7) + warp) + es = slot * cutlass.Int32(8) + ea = s_esc.load(idx=es) + ec = s_esc.load(idx=es + cutlass.Int32(1)) + eal = s_esc.load(idx=es + cutlass.Int32(2)) + ebe = s_esc.load(idx=es + cutlass.Int32(3)) + eli = s_esc.load(idx=es + cutlass.Int32(4)) + is_up = ( + (lane >> cutlass.Int32(3)) & cutlass.Int32(1) + ) == cutlass.Int32(0) + n0 = jrow * cutlass.Int32(M_MAX) + t2 = cblk >> cutlass.Int32(4) + cc = (cblk >> cutlass.Int32(1)) & cutlass.Int32(7) + gbase = g_u * cutlass.Int32(IMG_BYTES) + sfoff = ( + gbase + + cutlass.Int32(IMG_B2) + + ( + ( + t2 * cutlass.Int32(NUM_KB) + + ((cblk & cutlass.Int32(15)) >> cutlass.Int32(2)) + ) + * cutlass.Int32(512) + ) + + (cblk & cutlass.Int32(3)) + ) + for tr in cutlass.range_constexpr(NT): + # (skip) only tokens routed to this slot (warp-uniform); other B2 rows are never used + if s_kpos.load( + idx=slot * cutlass.Int32(M_MAX) + + cutlass.select_( + warp + cutlass.Int32(4 * tr) < cutlass.Int32(M_MAX), + warp + cutlass.Int32(4 * tr), + cutlass.Int32(0), + ) + ) >= cutlass.Int32(0): + xv = xr[tr] + gate_raw = cute.arch.shuffle_sync_bfly(xv, 8) + qv, sfb = _swiglu_quant( + xv, gate_raw, is_up, ea, ec, eal, ebe, eli + ) + byte = _e2m1x2(cute.arch.shuffle_sync_down(qv, 16), qv) + bt = [ + cute.arch.shuffle_sync(byte, i) & cutlass.Int32(0xFF) + for i in range(8) + ] + lo = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | (bt[3] << cutlass.Int32(24)) + ) + hi = ( + bt[4] + | (bt[5] << cutlass.Int32(8)) + | (bt[6] << cutlass.Int32(16)) + | (bt[7] << cutlass.Int32(24)) + ) + t = warp + cutlass.Int32(4 * tr) + if ( + (lane == 0) + & (t < cutlass.Int32(M_MAX)) + & cutlass.Boolean(not DBG_NOSTORE) + ): + # B2 row n = j M + t, k-tile c / 16, 16-byte chunk (c / 2) % 8 + # (swizzled), half c % 2; scale: atom (t2, kb = (c % 16) / 4), row n + # at byte 16 (n % 32) + 4 (n / 32), byte c % 4 + n = n0 + t + img32.store( + (lo, hi), + idx=( + gbase + + t2 * cutlass.Int32(NUM_BYTES_B2) + + n * cutlass.Int32(KT_BYTES) + + (cc ^ (n & cutlass.Int32(7))) * cutlass.Int32(16) + + (cblk & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + img8.store( + cutlass.Int8(sfb & cutlass.Int32(0xFF)), + idx=sfoff + + (n & cutlass.Int32(31)) * cutlass.Int32(16) + + (n >> cutlass.Int32(5)) * cutlass.Int32(4), + ) + tc1 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(TIMING): + e_pw = e_pw + (tc1 - tc0) + if cutlass.const_expr(not REL_WARP): + if unit_done: + if lane == 0: + if cutlass.const_expr(EPI_PFENCE): + _fence_proxy_async_global() + if cutlass.const_expr(EPI_FENCE): + _fence_acq_rel_gpu() + cute.arch.sync_warp() + cute.arch.barrier( + barrier_id=EPI_BAR, number_of_threads=EPI_THREADS + ) # s_part reads done + if cutlass.const_expr(not REL_WARP): + if unit_done & (tidx == 0): + _red_release_add( + cnt.subview(g_u * cutlass.Int32(8)).data_ptr().toint(), + cutlass.Int32(1), + ) + if cutlass.const_expr(TIMING): + e_busy = e_busy + (_gtime() - tb0) + k1 = k1 + cutlass.Int32(1) + if cutlass.const_expr(REL_WARP): + prims.mbarrier_arrive( + grp_done.subview(p & cutlass.Int32(3)) + ) # this CTA's group-p units are out + if tidx == 0: + _stamp(tdbg, bx, 10) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + e_wait = _waitt( + acc2_full.subview(a2), + (k2 >> cutlass.Int32(1)) & cutlass.Int32(1), + e_wait, + ) + if cutlass.const_expr(TIMING): + if ( + (tidx == 0) + & (g == ng - cutlass.Int32(1)) + & (bb == cutlass.Int32(0)) + ): + _stamp(tdbg, bx, 8) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + yv = [] + ya = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, + base_col + ACC2_COL + a2 * cutlass.Int32(ACC2_STRIDE), + cutlass.Float32, + ), + num=ACC2_STRIDE, + ) + for t in cutlass.range_constexpr(M_MAX): + v = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(4): + v = cutlass.select_( + warp == cutlass.Int32(w), cutlass.Float32(ya[w * M_MAX + t]), v + ) + yv.append(v) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc2_empty.subview(a2)) + sl = g * cutlass.Int32(4) + warp + if warp < s_gn.load(idx=g): + a2s = s_esc.load(idx=sl * cutlass.Int32(8) + cutlass.Int32(5)) + for t in cutlass.range_constexpr(M_MAX): + kp = s_kpos.load(idx=sl * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + if kp >= cutlass.Int32(0): + ys.store( + cutlass.BFloat16( + _bf16_round(_mul_rn(cutlass.Float32(yv[t]), a2s)) + ), + idx=( + (bb * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + * cutlass.Int32(TOP_K) + + kp + ) + * cutlass.Int32(R2) + + lane, + ) + k2 = k2 + cutlass.Int32(1) + if tidx == 0: + _stamp(tdbg, bx, 6) + _tstore(tdbg, bx, 31, e_wait) + _tstore(tdbg, bx, 34, e_busy) + _tstore(tdbg, bx, 36, e_a) + _tstore(tdbg, bx, 38, e_pw) + prims.mbarrier_arrive(ys_done.subview(0)) # this thread's ys values are in smem + + # ---- finalize on all 16 warps: (block bb, token t) pairs, the fma chain over t's top-4 in order, + y_sh + NPAIR = NB2_CAP * M_MAX + NPW = (NPAIR + NWARPS - 1) // NWARPS + wyl = [] + for bt in cutlass.range_constexpr(NPW): + pair = warp + cutlass.Int32(bt * NWARPS) + bb = pair // cutlass.Int32(M_MAX) + t = pair - bb * cutlass.Int32(M_MAX) + ok = (pair < cutlass.Int32(NPAIR)) & (bb < nb) + ch = cutlass.select_( + ok, + t * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + cutlass.Int32(0), + ) + wyl.append( + _ld_relaxed_u32(dnx.subview(ch).data_ptr().toint()) + if cutlass.const_expr(SHARED) + else cutlass.Int32(0) + ) + _wait(ys_done.subview(0), 0) + if tidx == 0: + _stamp(tdbg, bx, 19) + for bt in cutlass.range_constexpr(NPW): + pair = warp + cutlass.Int32(bt * NWARPS) + bb = pair // cutlass.Int32(M_MAX) + t = pair - bb * cutlass.Int32(M_MAX) + if (pair < cutlass.Int32(NPAIR)) & (bb < nb): + acc_o = cutlass.Float32(0.0) + for k in cutlass.range_constexpr(TOP_K): + acc_o = _fma_rn( + s_w.load(idx=t * cutlass.Int32(TOP_K) + cutlass.Int32(k)), + cutlass.Float32( + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ) + ), + acc_o, + ) + ch = ( + t * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(SHARED): + wy = wyl[bt] + while ((wy >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) != tag16: + wy = _ld_relaxed_u32(dnx.subview(ch).data_ptr().toint()) + acc_o = _add_rn( + (wy << cutlass.Int32(16)).bitcast(cutlass.Float32), _bf16_round(acc_o) + ) + if (ar_on == cutlass.Boolean(False)) | cutlass.Boolean(not OUT_SKIP): + out.store(cutlass.BFloat16(acc_o), idx=ch) + if ar_on: + cl = ( + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + _ar_push( + arp, + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + + # ---- teardown + if tidx == 0: + _stamp(tdbg, bx, 7) + if cutlass.const_expr(not EARLY_EPOCH): + epochs.store(ep + cutlass.Int32(1), idx=bx) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if ar_on | cutlass.Boolean(LATE_TRIGGER): + cute.arch.griddepcontrol_launch_dependents() + if warp == MMA_WARP: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Float32), TMEM_COLS + ) + + +@cute.jit +def m3_moe( + a1_tensor: cute.Tensor, + sfa1_tensor: cute.Tensor, + a2_tensor: cute.Tensor, + sfa2_tensor: cute.Tensor, + bias: cute.Tensor, + escal: cute.Tensor, + plan: cute.Tensor, + partials: cute.Tensor, + img32: cute.Tensor, + cnt: cute.Tensor, + s1w: cute.Tensor, + s2w: cute.Tensor, + w1b: cute.Tensor, + w2b: cute.Tensor, + epochs: cute.Tensor, + out: cute.Tensor, + route_dbg: cute.Tensor, + tdbg: cute.Tensor, + xb32: cute.Tensor, + rw: cute.Tensor, + sfsc: cute.Tensor, + lgbuf: cute.Tensor, + guw: cute.Tensor, + gusf: cute.Tensor, + dnw: cute.Tensor, + dnsf: cute.Tensor, + hx: cute.Tensor, + dnx: cute.Tensor, + arp: cute.Tensor, + stream: cuda_driver.CUstream, +): + tma_a1_desc = cuda.create_tensor_map_tiled( + global_address=a1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H // 2, TWO_I, E_LOCAL], + global_strides=[H // 2 // 16, TWO_I * (H // 2) // 16], + box_dims=(KT_BYTES, R1, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_a2_desc = cuda.create_tensor_map_tiled( + global_address=a2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[I_TP // 2, H, E_LOCAL], + global_strides=[I_TP // 2 // 16, H * (I_TP // 2) // 16], + box_dims=(KT_BYTES, R2, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_s1_desc = cuda.create_tensor_map_tiled( + global_address=sfa1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * TWO_I * (H // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_s2_desc = cuda.create_tensor_map_tiled( + global_address=sfa2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * H * (I_TP // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 4 // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_gu_desc = tma_a1_desc + tma_dn_desc = tma_a2_desc + if cutlass.const_expr(SHARED): + tma_gu_desc = cuda.create_tensor_map_tiled( + global_address=guw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H, 2 * SH_I], + global_strides=[H // 16], + box_dims=(KT_BYTES, 32), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_dn_desc = cuda.create_tensor_map_tiled( + global_address=dnw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[SH_I, H], + global_strides=[SH_I // 16], + box_dims=(KT_BYTES, 128), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + m3_moe_kernel( + tma_a1_desc, + tma_a2_desc, + tma_s1_desc, + tma_s2_desc, + tma_gu_desc, + tma_dn_desc, + bias, + escal, + plan, + partials, + img32, + cnt, + s1w, + s2w, + w1b, + w2b, + epochs, + out, + route_dbg, + tdbg, + xb32, + rw, + sfsc, + lgbuf, + guw, + gusf, + dnw, + dnsf, + hx, + dnx, + arp, + ).launch( + grid=(NCTA + NSH, 1, 1), + block=(THREADS, 1, 1), + cluster=(1, 1, 1), + stream=stream, + use_pdl=True, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/m3_moe_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/m3_moe_kernel.py new file mode 100644 index 000000000000..60694770577a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/m3_moe_kernel.py @@ -0,0 +1,3943 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 routed experts for up to 8 decode tokens on one TP4 rank (routed and shared intermediate 768): +``m3_moe`` (group-phased weight stream). + +One persistent kernel replaces TRTLLM-Gen's MiniMax2 routing, FC1 (NVFP4 x NVFP4 + SwiGLU-OAI + NVFP4 requantization), +FC2 and finalize, optionally with the router GEMV and the activation's NVFP4 quantization folded in (FUSED_ROUTER). It +writes the rank's routed partial [M, 6144] bf16, bit for bit what the TRTLLM-Gen op returns. + +- Routing (warps 4..7, token t on warp 4 + t % 4): TRTLLM-Gen's MiniMax2 top-4, operation for operation; warp 4 then + makes the distinct experts slots in (token, top-k) order (match.any), slot s in group s / 4. +- Work: the groups' FC1 stages and FC2 tiles stream through one ring per CTA in phases: phase p = FC1 of group p (a + stream-K range of the group's unit-major stages), then FC2 of group p - LAG (this CTA's 32-row output blocks, 4 + slots' slices stacked in one 128-row MMA, B row j M + t). LAG phases of streaming hide the hand-off of group + p - LAG's intermediate between CTAs. +- FC1 stage: a 32-row unit's rows at 4 consecutive 256-K tiles (rows 32 q + r) against resident B rows QS q + t. + FC2 stage: 4 slots' 32 rows at one 256-K tile. Both carry 8 KB of raw scale atoms, spliced into the 4 instruction + atoms by warp 6 with one formula (word [r][q] <- raw atom 4 q + kb, word [r][m1]). +- FC1 epilogue: k-quarters summed in order, SwiGLU-OAI, NVFP4 requant into tagged 16-byte records (3 words of 3 data + bytes + the call's tag); a unit split between two CTAs is summed by its tail's holder (Lamport slot). +- Warps 5 and 8.. (XW warps) build each group's FC2 B (tiles + scale atoms) from the records once they carry the + call's tag; warp 5 copies the scales to TMEM. +- Finalize: bf16(fc2_alpha * acc) per (slot, token) into its top-k position, then per token the fma chain over its + top-4 in order, bf16 out. With an all-reduce descriptor, every output word is also pushed to the exchange + (AR_PRE: call parities and descriptor loaded during the last group's FC2). + +Configuration (trace-time constants) via M3_CONFIG = {"m_max", "num_ctas", "stages", "nb2_cap", "fused_router", "lag", +"xw", ...}; the plan (op.make_phase_plan) must be built for the same lag. +Launched with PDL: barrier setup, TMEM allocation, tensormap prefetch, bias / plan / router weight loads before +griddepcontrol.wait; every producer-output read and every global write after it. +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +_CFG = globals().get("M3_CONFIG") or {} + + +def _cfg(key: str, default): + return type(default)(_CFG.get(key, default)) + + +# ============================================================================= +# Problem shape (trace-time constants). +# ============================================================================= +H = 6144 # capacity: fixed: the checkpoint's hidden size +# routed intermediate per rank: 1536 at TP2 (this target: FC2 K = 6 full 256-K tiles), 768 at TP4, 384 at TP8 (FC2 K +# = 1.5 256-K tiles: the last tile's A box is half out of bounds, zero-filled by TMA, and only its first KB2_LAST K-64 +# MMAs / scale atoms are issued; no padding, no extra bytes) +I_TP = _cfg("i_tp", 768) +TWO_I = 2 * I_TP +assert I_TP % 128 == 0 and 128 <= I_TP <= 1536, I_TP +E_LOCAL = 128 +TOP_K = 4 # capacity: fixed: the checkpoint's top-k +ROUTE_SCALE = 2.0 +SUM_EPS = 1e-20 +M_MAX = _cfg("m_max", 1) +NCTA = _cfg("num_ctas", 152) +FUSED_ROUTER = bool(_cfg("fused_router", 0)) +NROUTE = E_LOCAL if FUSED_ROUTER else 0 # experts whose logits the router GEMV computes +NGEMV = (NROUTE + NCTA - 1) // NCTA # experts per CTA: bx, bx + NCTA, .. +TIMING = bool(_cfg("timing", 0)) +HANG_DBG = bool(_cfg("hang_dbg", 0)) +XVDUMP = bool( + _cfg("xvdump", 0) +) # debug only (needs a plan without split units): FC1 sums into `partials` +YDUMP = bool( + _cfg("ydump", 0) +) # debug only: FC2 outputs per (token, k) into `partials` (breaks its protocol) +LAG = _cfg("lag", 2) # phase p runs FC1 of group p and FC2 of group p - LAG +G_CAP = TOP_K * M_MAX # distinct experts per call +NG_CAP = M_MAX # groups of 4 slots +XW = _cfg("xw", 5 if M_MAX > 4 else (4 if M_MAX > 2 else 1)) # FC2 B builder warps: 5, then 8 .. +THREADS = 256 + 32 * (XW - 1) +GEMV_THREADS = 256 # capacity: fixed: the CTA size +# SHARED: the MoE's shared expert (MXFP8 gate_up [1536, 6144] -> SwiGLU-OAI -> MXFP8 down [6144, 768]) on NSH more +# CTAs (warps 0-3), its output added in finalize: out = bf16(shared + routed) (shared_output.add_(routed_output)). +# The math is the low-M MXFP8 GEMM's (native MXFP8 activation quant, mma.sync m16n8k32 e4m3 per 32-K block, block scales +# as one fma, its partial-sum order, bf16) and the Triton silu_and_mul's, operation for operation. Shared CTA s owns +# intermediate positions 32 s .. 32 s + 31 (gate and up rows: MMA tiles = warps 0-3) and down rows 256 s .. + 255; +# 16-row x 768-byte weight chunks (+ their scale atoms) stream through NBW buffers per warp, the first NBW before +# the grid dependency. h crosses the shared CTAs, y_sh reaches the routed CTAs' finalize (both tagged words). +SHARED = bool(_cfg("shared", 0)) +# LATE_TRIGGER: release the dependents at teardown, not after the grid dependency. An early release lets the next +# kernels' CTAs (the all-reduce, then the attention it releases at entry) become resident during this kernel, packed +# onto the SMs it leaves free, where the attention then runs on a fraction of the GPU. +LATE_TRIGGER = bool(_cfg("late_trigger", 1)) +# TAIL_TRIGGER (with LATE_TRIGGER, no all-reduce push): the CTAs that own more FC2 blocks than the rest (the first +# H / 32 mod NCTA, which end last) release the dependents right after the grid dependency, so the dependents launch +# when the other CTAs tear down, onto the SMs they leave, instead of after the last of the heavy CTAs. +TAIL_TRIGGER = bool(_cfg("tail_trigger", 1)) +NHEAVY = (H // 32) % NCTA if NCTA < H // 32 else 0 +# PROD_TRIGGER (scheduling only; the TP8 MOE_TAIL_TRIGGER): on a call that pushes into the boundary +# all-reduce, each routed CTA releases the dependents when its weights producer has issued its last stage (the +# post-stream tail: FC1 drain, hand-off, FC2, finalize), not at teardown. The pushed all-reduce reads only the pushed +# tagged words, without a grid wait. +PROD_TRIGGER = bool(_cfg("prod_trigger", 0)) +# With an early release the next kernels' completion no longer implies this kernel's, so the per-CTA call epoch -- read +# by the NEXT call on this state -- is stored once every thread of the CTA has read it, not at teardown. Every other +# workspace write of a call lands before the finalize's push. +EARLY_EPOCH = PROD_TRIGGER +# AR_PRE: the finalize's all-reduce push uses the call parities and descriptor fields loaded while the last group's +# FC2 runs (not two dependent global loads per push on the kernel's tail). +AR_PRE = bool(_cfg("ar_pre", 1)) +# OUT_SKIP (MOE-FINALIZE-DEADSTORE): on a call that pushes into the boundary all-reduce (arp enabled), +# the finalize does not also store its bf16 rows into ``out``: the pushed all-reduce (prepushed >= 2) never reads them +# and returns its own tensors, so the store is dead traffic. Calls without a push store as before. Bitwise: the pushed +# words are unchanged. +OUT_SKIP = bool(_cfg("out_skip", 0)) +# XSTAGE (fused router): the hidden states land in the (still idle) ring by one bulk copy per token right after the +# grid dependency; the NVFP4 quantization of B1 reads them from shared memory (one HBM / L2 round trip, not ~8). +XSTAGE = bool(_cfg("xstage", 1)) and bool(_cfg("fused_router", 0)) +NSH = 24 if SHARED else 0 +# the shared intermediate per rank: 1536 (TP2), 768 (TP4), 384 (TP8). A shared CTA owns SH_ROWS intermediate +# positions: its gate_up MMA tile is SH_SPLIT K slices x (SH_ROWS gate + SH_ROWS up rows) = 128 rows (TP2: 1 x (64 + +# 64), full K in one accumulator, the order of the stock tcgen05 MXFP8 GEMM; TP4: 2 x (32 + 32), the order of +# m3_mx_gemv split 2; TP8: 4 x (16 + 16), m3_mx_gemv split 4), and 6144 / NSH = 256 down rows over the full K. +SH_I = _cfg("sh_i", 768) +SH_ROWS = SH_I // 24 +# gate_up K slices (= m3_mx_gemv's cluster ranks, contiguous K ranges) +SH_SPLIT = 128 // (2 * SH_ROWS) +SH_NB = 8 * SH_SPLIT # gate_up B rows (8 q + t: token t's slice-q k-tile) = its MMA N +SH_KB = H // 32 # capacity: fixed: a kernel tile constant (gate_up K blocks) (gate_up K blocks) +SH_DKB = SH_I // 32 # down K blocks +SH_DKT = SH_I // 128 # down k-tiles +SH_GT = SH_KB // 4 // SH_SPLIT # gate_up k-tiles (128 K) per slice +SH_STAGES_GU = SH_GT +SH_STAGES = SH_GT + 2 * SH_DKT # + 2 down tiles x SH_DKT k-tiles +SHX_BAR = 6 +# SH_PRETILE: the shared weights come pre-tiled (op._pretile_shared): per shared CTA its SH_STAGES stages, each the +# 16 KB shared-memory image of its 128-row MMA tile (128-byte swizzle) followed by the stage's 512-byte instruction +# scale atom, so a stage is two contiguous bulk copies (not 128 row segments of 128 bytes) and needs no splicing. +SH_PRETILE = bool( + _cfg("sh_pretile", 1 if M_MAX <= 2 or SH_SPLIT != 2 else 0) +) # (in-chain: a win at M <= 2, not 4..8) +if SHARED: + assert SH_I in (384, 768, 1536), SH_I + assert SH_PRETILE or SH_SPLIT == 2, ( + "the un-pretiled shared stream is the 32-row (TP4) layout only" + ) +SH_STAGE_BYTES = 16384 + 512 +MMA_M = 128 +KT_BYTES = 128 +INC_K = 2 +NUM_KB = 4 +# FC1 +N1 = 16 if M_MAX <= 4 else 32 # B rows: QS q + t (token t at the stage's k-tile 4 j + q) +QS = N1 // 4 +R1 = 32 +C1 = 4 +U1 = TWO_I // R1 # 48 units per slot +K1_TILES = H // 256 +S1 = K1_TILES // C1 # 6 stages per unit # capacity: fixed: pipeline stages +NUM_BYTES_A = MMA_M * KT_BYTES # capacity: fixed: a stage's operand bytes +NUM_BYTES_BOX1 = R1 * KT_BYTES +SF1_KA = H // 64 +SF1_RB = TWO_I // 128 +NUM_BYTES_SFA_RAW = 16 * 512 +NUM_WORDS_SFA_RAW = NUM_BYTES_SFA_RAW // 4 +NUM_BYTES_SFA = NUM_KB * 512 # capacity: fixed: a stage's scale bytes +NUM_WORDS_SFA = NUM_BYTES_SFA // 4 +NUM_BYTES_B1 = N1 * KT_BYTES +NUM_SFB1_ATOMS = S1 * NUM_KB +# FC2 +R2 = 32 +NB2 = H // R2 # 192 +KT2 = (I_TP + 255) // 256 # 3 (TP4), 2 (TP8) +# K-64 instructions of the last FC2 k-tile: 4 (TP4), 2 (TP8) +KB2_LAST = (I_TP - 256 * (KT2 - 1)) // 64 + + +def _nkb2(t2: int) -> int: + """K-64 MMA instructions (= scale atoms) of FC2 k-tile t2.""" + return NUM_KB if t2 < KT2 - 1 else KB2_LAST + + +SF2_KA = I_TP // 64 +SF2_RB = H // 128 +N2 = max(8, (4 * M_MAX + 7) // 8 * 8) # B rows j M + t +NUM_BYTES_B2 = N2 * KT_BYTES +NB2_CAP = _cfg("nb2_cap", 2) +UNITS_ROW = TWO_I // R1 # 48 records per (slot, token) row +H_ROW = UNITS_ROW * 16 +# HB_REPL: the tagged FC1 -> FC2 records written to HB_REPL copies (HB_STRIDE int32 words apart); CTA bx polls copy +# bx % HB_REPL. Every routed CTA polls every record, so with one copy the hand-off latency depends on where the few +# hot lines land in L2 (TP8 M=1: 1.5 KB of records, +-1 us across allocations); copies spread the polls. +HB_REPL = _cfg("hb_repl", 1) +assert 1 <= HB_REPL <= 32, HB_REPL # (copy i is written by lane i) +HB_STRIDE = (G_CAP * M_MAX * UNITS_ROW * 4 + 255) // 256 * 256 + 32 +PARTIAL_EMPTY = -1 # capacity: fixed: a sentinel, not a size +EPI_THREADS = 128 +EPI_BAR = 1 +ROUTE_BAR = 2 +GEMV_BAR = 3 +B2_BAR = 5 +TD_STRIDE = 64 # tdbg columns per CTA +TOPK_BAR = 4 +B1_AFTER_TOPK = bool( + _cfg("b1_after_topk", 0) +) # (experiment) the B1 build waits for the routing warps' top-k +TOPK2_BAR = 7 +EARLY0 = bool( + _cfg("early0", 1) +) # warp 4 streams group 0 (token 0's experts) right after its own top-k +EARLY0_N = _cfg("early0_n", 99) # stages issued early (capped at STAGES) +# routing warps: 4..7, then the FC2 B builder warps 8.. (idle until the routing is done) up to one warp per token +RW_EXTRA = max(0, min(M_MAX - 4, (THREADS // 32) - 8)) if bool(_cfg("rw_extra", 1)) else 0 +NRW0 = ( + min(4, M_MAX) if bool(_cfg("rw_trim", 1)) else 4 +) # warps 4.. with a token (no idle pollers in the barrier) +NRW = NRW0 + RW_EXTRA +LG_REPL = _cfg("lg_repl", 32 if M_MAX > 4 else 128) +GEMV_SMEM = bool( + _cfg("gemv_smem", 1) +) # (XSTAGE) the router GEMV reads the staged hidden states from smem +LG_STRIDE = M_MAX * E_LOCAL + 32 +POLL_NS = _cfg("poll_ns", 150) +# Hang guard: every cross-CTA spin (the kernel needs all its CTAs resident at once) traps after this long, so a lost +# co-residency or a missing producer surfaces as a device exception instead of a silent hang. The timer is read only +# inside a spin iteration; the arithmetic is unchanged. +HANG_NS = _cfg("hang_ms", 2000) * 1000000 +B2_POLL_NS = _cfg("b2_poll_ns", 200) +# B2IMG (needs SFA_CP: warp 6 is free): the FC1 epilogue writes each finished unit's FP4 bytes and scale byte straight +# into a per-group global image laid out as the FC2 B tile + its scale atoms (exactly what the builder warps assembled +# from the tagged records); per group, warp 6 joins the epilogue (fc1_done), fences the async proxy and does one +# release add of this CTA's finished units to the group counter; warp 5 acquires the counter (every group gains +# CNT_PER_CALL per call, CTA 0 pads absent slots), fences, and bulk-copies the image into the B2 buffer. +B2IMG = bool(_cfg("b2img", 0)) and bool(_cfg("sfa_cp", 0)) +IMG_B2 = KT2 * NUM_BYTES_B2 +IMG_SF = KT2 * NUM_KB * 512 +IMG_BYTES = IMG_B2 + IMG_SF +CNT_PER_CALL = 4 * (TWO_I // R1) +B2IMG_NS = _cfg("b2img_ns", 16) # (B2IMG) counter poll back-off +B2_DLY = _cfg( + "b2_dly", 0 +) # (stress) the release warp spins up to this many ns (CTA- and group-dependent) first +W_WARP = 4 # capacity: fixed: a warp role index +X_WARP = 5 # capacity: fixed: a warp role index +S_WARP = 6 # capacity: fixed: a warp role index +MMA_WARP = 7 # capacity: fixed: a warp role index +# E0_M4: at M = 4 the MMA warp routes +# token 3 (NRW0 = 4), so the early group-0 stages go to FC2 B builder warp 8 instead (idle until the routing is done; +# RW_EXTRA = 0 at M = 4). Issue order only: exact. +E0_M4 = ( + bool(_cfg("e0_m4", 1)) and NRW0 == 4 and RW_EXTRA == 0 and THREADS // 32 > 8 +) # M == 4 only: at M > 4 warp 8 routes (RW_EXTRA) +E0_WARP = ( + 8 if E0_M4 else MMA_WARP +) # (M <= 3) the early stages are issued by the idle MMA warp, not by warp 4 (routing's critical path) +E0_BAR = 8 +E1_BAR = 9 +E0_OFFLOAD = EARLY0 and (NRW0 <= 3 or E0_M4) and bool(_cfg("e0_off", 1)) +# B1_AFTER_TOPK holds warps 0..3 (the B1 build, which arrives x_bar[1] when XSTAGE) on TOPK2_BAR until every routing +# warp arrives; with EARLY0 and no E0 offload, warp 4 first waits on x_bar[1] to fill the ring before it arrives, so the +# two wait on each other. Named barriers have no HANG_NS trap, so the kernel would spin forever: refuse the config. +if B1_AFTER_TOPK and XSTAGE and EARLY0 and not E0_OFFLOAD: + raise ValueError( + "m3_moe: b1_after_topk=1 with xstage, early0 and no early-stage offload deadlocks (warp 4 waits on x_bar[1] " + "before TOPK2_BAR, the B1 build waits on TOPK2_BAR before x_bar[1]); set b1_after_topk=0 or early0=0" + ) +EVICT_FIRST = 0x12F0000000000000 +_LOG2E = 1.4426950408889634 +LD_POW2 = ( + 1 if M_MAX == 1 else 2 if M_MAX == 2 else 4 if M_MAX <= 4 else 8 +) # tcgen05.ld .num covering M columns + +_FIXED_SMEM = ( + S1 * NUM_BYTES_B1 + + 2 * KT2 * NUM_BYTES_B2 + + 2 * KT2 * NUM_KB * 512 + + NB2_CAP * M_MAX * TOP_K * R2 * 4 + + 2 * M_MAX * C1 * R1 * 4 + + (M_MAX * (H // 16) if FUSED_ROUTER else 0) + + 6 * 1024 +) +# SFA_CP: the A scale words go by 4-byte cp.async straight into the instruction atom layout (no raw scale box, no +# splicer warp; the MMA warp tcgen05.cp's them), completing on the stage's full barrier: 18 KB stages instead of 26 KB +SFA_CP = bool(_cfg("sfa_cp", 0)) +_RAW_STAGE = 0 if SFA_CP else NUM_BYTES_SFA_RAW +_STAGE_SMEM = NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA +_STAGES_FIT = (227 * 1024 - _FIXED_SMEM - (128 if SHARED else 0)) // _STAGE_SMEM +ACC1_COL = 0 +ACC2_COL = 2 * N1 +SFB1_COL = ACC2_COL + 2 * N2 +SFB2_COL = SFB1_COL + NUM_SFB1_ATOMS * 4 +SFA1_COL = SFB2_COL + 2 * KT2 * NUM_KB * 4 +STAGES = min(_cfg("stages", 12 if SFA_CP else 8), _STAGES_FIT, (512 - SFA1_COL) // 16) +TMEM_COLS = 512 +assert 1 <= M_MAX <= 8, M_MAX +assert STAGES >= 3, (STAGES, _STAGES_FIT, SFA1_COL) +# TP2 at 8 tokens: the 98 KB of hidden states fit no 5-stage ring; the router GEMV then reads them from global (the +# kernel's non-XSTAGE path, same math) +if XSTAGE and M_MAX * H * 2 > STAGES * (NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA): + XSTAGE = False +assert not XSTAGE or M_MAX * H * 2 <= STAGES * (NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA), ( + "x staging must fit the ring" +) +# one shared-memory region for the ring and the FC1 / FC2 B operands (a shared CTA reuses all of it) +_OFF_SFAR = STAGES * NUM_BYTES_A +_OFF_SFA = _OFF_SFAR + STAGES * _RAW_STAGE +_OFF_B1 = _OFF_SFA + STAGES * NUM_BYTES_SFA +_OFF_B2 = _OFF_B1 + S1 * NUM_BYTES_B1 +_OFF_SFB2 = _OFF_B2 + 2 * KT2 * NUM_BYTES_B2 +BIG = _OFF_SFB2 + 2 * KT2 * NUM_KB * 512 +# shared CTA layout in BIG: weight buffers, x quarters (2), x scales [M, 192], gate / up [4, 16, M], h, h scales +# shared CTA layout in BIG: A ring (16 KB stages), raw scale atoms (4 per stage), spliced atoms, B (x: [stage][16 rows] +# [128 B]; h: [6][8][128 B]), x / h scale codes, gate_up partial exchange +_SB_A = 0 +_SB_RAW = None +# slices 1.. gate / up partials [slice - 1][u][p][8 tokens] f32 +_SH_XCH_BYTES = (SH_SPLIT - 1) * 2 * SH_ROWS * 8 * 4 +_SH_FIX = ( + SH_GT * SH_NB * 128 + + SH_DKT * 8 * 128 + + M_MAX * SH_KB + + M_MAX * SH_DKB + + _SH_XCH_BYTES + + 2 * SH_ROWS * 8 * 4 + + 1024 +) +SH_NST = min(8, (BIG - _SH_FIX) // (NUM_BYTES_A + 4 * 512 + 512)) if SHARED else 1 +_SB_RAW = SH_NST * NUM_BYTES_A +_SB_SPL = _SB_RAW + SH_NST * 2048 +_SB_BX = (_SB_SPL + SH_NST * 512 + 1023) // 1024 * 1024 +_SB_BH = _SB_BX + SH_GT * SH_NB * 128 +_SB_XC = _SB_BH + SH_DKT * 8 * 128 +_SB_HC = _SB_XC + M_MAX * SH_KB +_SB_XCH = (_SB_HC + M_MAX * SH_DKB + 15) // 16 * 16 +_SB_GU = _SB_XCH + _SH_XCH_BYTES +_SB_END = _SB_GU + 2 * SH_ROWS * 8 * 4 +# shared CTA TMEM columns: gate_up accumulator (N SH_NB), down accumulators (2 x N 8), A scales per stage, B scales +SH_ACC_GU = 0 +SH_ACC_DN = SH_NB +SH_SFA = SH_ACC_DN + 16 +SH_SFB_GU = SH_SFA + 4 * 8 +SH_SFB_DN = SH_SFB_GU + 4 * SH_GT +# pre-tiled stages beyond the ring that warp 0 prefetches into L2, per lane (TP2: 72 stages) +SH_PF = max(2, (SH_STAGES - SH_NST + 31) // 32) if SHARED else 2 +# TP2 (SH_GT 48 gate_up k-tiles, SH_DKT 12) needs more than 256 columns: allocate the next power of two +SH_TMEM = 256 if SH_SFB_DN + 4 * SH_DKT <= 256 else 512 +if SHARED: + assert FUSED_ROUTER, "shared expert: fused router only" + assert SH_STAGES - SH_NST <= 32 * SH_PF, SH_STAGES + assert SH_NST >= 3 and _SB_END <= BIG and SH_SFB_DN + 4 * SH_DKT <= SH_TMEM, ( + SH_NST, + _SB_END, + BIG, + ) + + +# ============================================================================= +# Inline-PTX helpers. +# ============================================================================= +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +# ---- all-reduce push (the MoE boundary's multicast exchange, set A): +# arp = int64 [enable, mc, flags, rank, world, maxt] +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32_if(addr, value, pred, *, loc=None, ip=None): + """st.global.b32 when pred != 0.""" + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip), + cutlass.Int32(pred).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, $2, 0;\n\t@p st.global.b32 [$0], $1;\n\t}", "l,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _ar_push(arp, v, low, t, cl): + """Push bf16 channel pair (cl, cl + 1) of token t (v = channel cl's bf16 value on this lane, its neighbour's on + lane ^ 8) into this rank's set-A slot of the exchange; the slot half is the all-reduce CTA (t, cl // 768)'s call + parity. Only lanes with ``low`` store; every lane of the warp must call it (shuffle).""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + world = arp.load(idx=4) + parity = cutlass.Int64( + _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cs // cutlass.Int32(H // 8)) + ) + & cutlass.Int32(1) + ) + off = ( + parity * arp.load(idx=5) * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + arp.load(idx=3)) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(arp.load(idx=1) + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +def _ar_parity(arp, t, cl): + """The call parity _ar_push reads for token t's channel cl (fixed between this kernel's grid dependency and the + next all-reduce, which this kernel releases at teardown): loaded ahead of the finalize (AR_PRE).""" + return _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cl // cutlass.Int32(H // 8)) + ) + + +def _ar_push_pre(ard, parity32, v, low, t, cl): + """_ar_push with the parity (_ar_parity) and the descriptor fields ard = (mc, rank, world, maxt) preloaded.""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + mc, rk, world, maxt = ard + parity = cutlass.Int64(parity32 & cutlass.Int32(1)) + off = ( + parity * maxt * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + rk) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(mc + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +@dsl_user_op +def _bar_arrive(bar_id: int, threads: int, *, loc=None, ip=None): + """bar.arrive: signal a named barrier without waiting (its memory ordering is bar.sync's).""" + _llvm.inline_asm( + None, + [], + f"bar.arrive {bar_id}, {threads};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _st_relaxed_u32(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "st.relaxed.gpu.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _st_relaxed_u64(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int64(val)], + "st.relaxed.gpu.global.u64 [$0], $1;", + "l,l", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm( + _T.i64(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u64 $0, [$1];", + "=l,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _nanosleep(ns: int, *, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + f"nanosleep.u32 {ns};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _gtime(*, loc=None, ip=None): + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), + [], + "mov.u64 $0, %globaltimer;", + "=l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _trap_after(elapsed, *, loc=None, ip=None): + """trap when elapsed (ns, int64) exceeds HANG_NS.""" + _llvm.inline_asm( + None, + [cutlass.Int64(elapsed).ir_value(loc=loc, ip=ip)], + f"{{ .reg .pred p; setp.gt.s64 p, $0, {HANG_NS}; @p trap; }}", + "l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +def _hang_tick(t0): + """One spin iteration of a cross-CTA wait: the wait's start (latched on the first call; t0 = 0 before it), after + trapping once the wait has lasted longer than HANG_NS.""" + now = _gtime() + t1 = cutlass.select_(t0 == cutlass.Int64(0), now, t0) + _trap_after(now - t1) + return t1 + + +@dsl_user_op +def _smid(*, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [], + "mov.u32 $0, %smid;", + "=r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@cute.jit +def _stamp(tdbg: cutlass.Array, bx, i: cutlass.Constexpr): + if cutlass.const_expr(TIMING): + tdbg.store(_gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(i)) + + +@dsl_user_op +def _rcp(a, *, loc=None, ip=None): + """rcp.approx.ftz.f32 (the reciprocal TRT-LLM's FP4 quantization uses).""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _div_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _sub_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "sub.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _mul_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3_bits(x, *, loc=None, ip=None): + """e4m3 (round to nearest even, saturating) of x as an int 0..255.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, 0f00000000, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3_to_f32(b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(b)], + "{\n.reg .b16 t, l, u;\n.reg .b32 h;\ncvt.u16.u32 t, $1;\ncvt.rn.f16x2.e4m3x2 h, t;\nmov.b32 {l, u}, h;\n" + "cvt.f32.f16 $0, l;\n}", + "=f,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e2m1x2(hi, lo, *, loc=None, ip=None): + """One byte of two E2M1 values (lo in the low nibble), round to nearest even, saturating.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b8 t;\ncvt.rn.satfinite.e2m1x2.f32 t, $1, $2;\ncvt.u32.u8 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2(x, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _bf16_round(x, *, loc=None, ip=None): + """x rounded to bf16 (nearest even), as f32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\nmov.b32 $0, {0, t};\n}", + "=f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _redux_max_u32(v, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(v)], + "redux.sync.max.u32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16 (nearest even) bits of x in the low 16 bits.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ue8m0_rp(x, *, loc=None, ip=None): + """UE8M0 code of x rounded toward +inf, saturating (__nv_cvt_float_to_e8m0(x, SATFINITE, RoundPosInf)).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rp.satfinite.ue8m0x2.f32 t, $1, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3x2(hi, lo, *, loc=None, ip=None): + """Two e4m3 (nearest even, saturating) in 16 bits, lo in the low byte.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, $1, $2;\ncvt.u32.u16 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _min_xorsign_abs(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "min.xorsign.abs.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _div_full(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.full.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2_full(x, *, loc=None, ip=None): + """ex2.approx.f32 (no flush to zero), Triton's tl.exp.""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _l2_prefetch(addr, nbytes, *, loc=None, ip=None): + """cp.async.bulk.prefetch.L2 of nbytes (a multiple of 16) at global address addr (16-byte aligned).""" + _llvm.inline_asm( + None, + [ + cutlass.Int64(addr).ir_value(loc=loc, ip=ip), + cutlass.Int32(nbytes).ir_value(loc=loc, ip=ip), + ], + "cp.async.bulk.prefetch.L2.global [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +def _mx_osc(amax): + """mxfp8 quantization of a block with absolute max amax: (UE8M0 code, the element multiplier).""" + code = _ue8m0_rp(_mul_rn(amax, _rcp(cutlass.Float32(448.0)))) + cf = cutlass.Int32( + cutlass.select_( + code == cutlass.Int32(255), + cutlass.Int32(0x7FFFFFFF), + cutlass.select_( + code == cutlass.Int32(0), cutlass.Int32(0x00400000), code << cutlass.Int32(23) + ), + ) + ).bitcast(cutlass.Float32) + return code, cutlass.select_(amax != cutlass.Float32(0.0), _rcp(cf), cutlass.Float32(0.0)) + + +def _mx_words(v, osc): + """e4m3 words of len(v) (a multiple of 4) values times osc, in element order.""" + words = [] + for t in range(len(v) // 4): + lo = _e4m3x2(_mul_rn(v[4 * t + 1], osc), _mul_rn(v[4 * t], osc)) + hi = _e4m3x2(_mul_rn(v[4 * t + 3], osc), _mul_rn(v[4 * t + 2], osc)) + words.append(lo | (hi << cutlass.Int32(16))) + return words + + +def _twiddle(score): + """Float bits mapped so that unsigned order is float order (cub TwiddleIn).""" + bits = ( + score.bitcast(cutlass.Int32) + if hasattr(score, "bitcast") + else cutlass.Float32(score).bitcast(cutlass.Int32) + ) + return cutlass.select_(bits < cutlass.Int32(0), ~bits, bits | cutlass.Int32(-2147483648)) + + +@cute.jit +def _wait(bar: cutlass.Array, parity): + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +def _tptr(base_row, col, dtype=cutlass.Int32): + return cutlass.inttoptr((base_row << 16) | col, 6, dtype) + + +@dsl_user_op +def _match_any(value, *, loc=None, ip=None): + """The mask of the warp's lanes holding the same 32-bit value.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(value)], + "match.any.sync.b32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@cute.jit +def _waitd(bar: cutlass.Array, parity, tdbg: cutlass.Array, site: cutlass.Constexpr): + """_wait, or (HANG_DBG) give up after ~2^22 polls and record the site in tdbg[bx, 24 + warp].""" + if cutlass.const_expr(HANG_DBG): + n = cutlass.Int32(0) + while (not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity)) & ( + n < cutlass.Int32(1 << 22) + ): + n = n + cutlass.Int32(1) + if n >= cutlass.Int32(1 << 22): + bx, _, _ = cute.arch.block_idx() + w = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + tdbg.store( + cutlass.Int64(site), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + w + ) + else: + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +@cute.jit +def _adv(stage, phase): + """The next ring position's (stage, phase).""" + stage = stage + cutlass.Int32(1) + wrap = stage == cutlass.Int32(STAGES) + return cutlass.select_(wrap, cutlass.Int32(0), stage), cutlass.select_( + wrap, phase ^ cutlass.Int32(1), phase + ) + + +@cute.jit +def _seg(i0, i1, k): + """Segment k of the FC1 item range [i0, i1) (items = unit-major stages): (unit, first stage, end stage). Order: + the last unit (its head, the part a split unit's next CTA does not cover), then the first unit, then the rest.""" + uf = i0 // cutlass.Int32(S1) + ul = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) + unit = uf + k - cutlass.Int32(1) + unit = cutlass.select_(k == cutlass.Int32(0), ul, unit) + unit = cutlass.select_(ul == uf, uf, unit) + j0 = i0 - unit * cutlass.Int32(S1) + j0 = cutlass.select_(j0 < cutlass.Int32(0), cutlass.Int32(0), j0) + j1 = i1 - unit * cutlass.Int32(S1) + j1 = cutlass.select_(j1 > cutlass.Int32(S1), cutlass.Int32(S1), j1) + return unit, j0, j1 + + +@cute.jit +def _nsegs(i0, i1): + n = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) - i0 // cutlass.Int32(S1) + cutlass.Int32(1) + return cutlass.select_(i1 > i0, n, cutlass.Int32(0)) + + +@cute.jit +def _sfb1_to_tmem(warp, lane, base_row, base_col, src: cutlass.Array, b1_ready: cutlass.Array): + """B1 scales to TMEM: instruction (j, kb)'s column holds, at lane n = QS q + t, token t's 4 scales of k-tile + 4 j + q, K 64 kb .. (src: int32 words [M, 96] of the token scales); each warp writes its sub-partition.""" + q = lane // cutlass.Int32(QS) + tl = lane - q * cutlass.Int32(QS) + sf_ok = (lane < cutlass.Int32(N1)) & (tl < cutlass.Int32(M_MAX)) + sfv = [] + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + j = a // NUM_KB + kb = a % NUM_KB + wsf = src.load( + idx=cutlass.select_( + sf_ok, + tl * cutlass.Int32(H // 64) + + (cutlass.Int32(4 * j) + q) * cutlass.Int32(4) + + cutlass.Int32(kb), + cutlass.Int32(0), + ) + ) + sfv.append(cutlass.select_(sf_ok, cutlass.Int32(wsf), cutlass.Int32(0))) + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + prims.tcgen05_st( + "32x32b", + _tptr(base_row + warp * cutlass.Int32(32), base_col + SFB1_COL + a * 4), + sfv[a], + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(b1_ready) + + +@cute.jit +def _b1_addr(t, kb16): + """Word index in the resident B tiles of the 8 FP4 bytes of token t's 16-value block kb16.""" + kt = kb16 >> cutlass.Int32(4) + j = kt >> cutlass.Int32(2) + n = (kt & cutlass.Int32(3)) * cutlass.Int32(QS) + t + cc = (kb16 >> cutlass.Int32(1)) & cutlass.Int32(7) + return ( + j * cutlass.Int32(NUM_BYTES_B1) + + n * cutlass.Int32(KT_BYTES) + + (cc ^ (n & cutlass.Int32(7))) * cutlass.Int32(16) + + (kb16 & cutlass.Int32(1)) * cutlass.Int32(8) + ) // cutlass.Int32(4) + + +@cute.jit +def _build_b1( + tidx, + warp, + lane, + base_row, + base_col, + x32: cutlass.Array, + xsf32: cutlass.Array, + sB1: cutlass.Array, + b1_ready: cutlass.Array, +): + """B (the M tokens' whole K) into the resident tiles from the NVFP4 activation, its scales to TMEM.""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + for r in cutlass.range_constexpr((M_MAX * (H // 16) + EPI_THREADS - 1) // EPI_THREADS): + b = tidx + cutlass.Int32(r * EPI_THREADS) + if b < cutlass.Int32(M_MAX * (H // 16)): + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + v2 = x32.load(idx=b * cutlass.Int32(2), vector_size=2, alignment=8) + sB1_32.store( + (cutlass.Int32(v2[0]), cutlass.Int32(v2[1])), idx=_b1_addr(t, kb), alignment=8 + ) + _sfb1_to_tmem(warp, lane, base_row, base_col, xsf32, b1_ready) + + +@cute.jit +def _build_b1_fused( + tidx, + warp, + lane, + base_row, + base_col, + xb32: cutlass.Array, + sfsc: cutlass.Array, + sB1: cutlass.Array, + s_xsf: cutlass.Array, + b1_ready: cutlass.Array, +): + """_build_b1 from the bf16 hidden states, quantized exactly as fp4_quantize(x, sf_scale, 16).""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + s_xsf8 = cutlass.Array( + s_xsf.data_ptr(0), shape=(M_MAX * (H // 16),), dtype=cutlass.Int8, alignment=16 + ) + sc = cutlass.Float32(sfsc.load(idx=0)) + rsc = _rcp(sc) + rsix = _rcp(cutlass.Float32(6.0)) + NBLK = M_MAX * (H // 16) // EPI_THREADS + QB = ( + 3 if NBLK % 3 == 0 else (2 if NBLK % 2 == 0 else 1) + ) # blocks whose loads are in flight together + # a rolled loop over groups of QB blocks (a fully unrolled one is ~50 KB of SASS at M = 8: instruction fetch bound) + for rg in cutlass.range(NBLK // QB, unroll=1): + wall = [] + for rr in cutlass.range_constexpr(QB): + bq = tidx + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(EPI_THREADS) + for h in cutlass.range_constexpr(2): + wall.append( + xb32.load( + idx=bq * cutlass.Int32(8) + cutlass.Int32(4 * h), + vector_size=4, + alignment=16, + ) + ) + for rr in cutlass.range_constexpr(QB): + b = tidx + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(EPI_THREADS) + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + w = [] + for h in cutlass.range_constexpr(2): + v4 = wall[rr * 2 + h] + for z in cutlass.range_constexpr(4): + w.append(cutlass.Int32(v4[z])) + xs = [] + for z in cutlass.range_constexpr(8): + xs.append((w[z] << cutlass.Int32(16)).bitcast(cutlass.Float32)) + xs.append((w[z] & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + vmax = cute.math.abs(xs[0]) + for z in cutlass.range_constexpr(1, 16): + vmax = cute.arch.fmax(vmax, cute.math.abs(xs[z])) + sfb = _e4m3_bits(_mul_rn(sc, _mul_rn(vmax, rsix))) + osc = cutlass.select_( + vmax != cutlass.Float32(0.0), + _rcp(_mul_rn(_e4m3_to_f32(sfb), rsc)), + cutlass.Float32(0.0), + ) + by = [ + _e2m1x2(_mul_rn(xs[2 * z + 1], osc), _mul_rn(xs[2 * z], osc)) & cutlass.Int32(0xFF) + for z in range(8) + ] + lo = ( + by[0] + | (by[1] << cutlass.Int32(8)) + | (by[2] << cutlass.Int32(16)) + | (by[3] << cutlass.Int32(24)) + ) + hi = ( + by[4] + | (by[5] << cutlass.Int32(8)) + | (by[6] << cutlass.Int32(16)) + | (by[7] << cutlass.Int32(24)) + ) + sB1_32.store((lo, hi), idx=_b1_addr(t, kb), alignment=8) + s_xsf8.store(cutlass.Int8(sfb), idx=b) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + _sfb1_to_tmem(warp, lane, base_row, base_col, s_xsf, b1_ready) + + +@cute.jit +def _swiglu_quant(xv, gate_raw, is_up, a, c, alpha, beta, lim): + """TRTLLM-Gen's FC1 epilogue (the swiGlu Bmm E2m1 output), operation for operation: lane value xv is the up row, + gate_raw its gate row (fp32 sums). Returns (q, the scale's e4m3 bits) of the 16-row NVFP4 block (lanes r, r ^ 16 + and r ^ 8 cover it; amax over the warp's up lanes of a half). a = fc31_alpha, c = fc31_scale_c.""" + neg_max = cutlass.Float32(-3.40282346638528859812e38) + gc = cutlass.select_(gate_raw > lim, lim, gate_raw) + gc = cutlass.select_(gate_raw < neg_max, neg_max, gc) + uc = cutlass.select_(xv > lim, lim, xv) + uc = cutlass.select_(xv < -lim, -lim, uc) + c1 = _mul_rn(a, _mul_rn(alpha, cutlass.Float32(1.4426950216293334961))) + sgm = _rcp(_add_rn(_ex2(-_mul_rn(gc, c1)), cutlass.Float32(1.0))) + gs = _mul_rn(gc, sgm) + h = _mul_rn(_fma_rn(uc, a, _mul_rn(a, beta)), gs) + h = cutlass.select_(is_up, h, cutlass.Float32(0.0)) + amax = cute.arch.fmax(_mul_rn(cute.math.abs(h), c), cutlass.Float32(0.0)) + for off in (16, 8, 4, 2, 1): + amax = cute.arch.fmax(amax, cute.arch.shuffle_sync_bfly(amax, off)) + sfb = _e4m3_bits(_mul_rn(amax, cutlass.Float32(0.16666667163372039795))) + sff = _e4m3_to_f32(sfb) + osc = cutlass.select_(sff != cutlass.Float32(0.0), _rcp(sff), cutlass.Float32(0.0)) + return _mul_rn(_mul_rn(h, osc), c), sfb + + +@cute.jit +def _plan_range(s_plan: cutlass.Array, p): + w = s_plan.load(idx=p) + return w & cutlass.Int32(0xFFFF), w >> cutlass.Int32(16) + + +@cute.jit +def _sh_issue( + lane, + sid, + g, + s_big: cutlass.Array, + sh_full: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, +): + """Stage g of the shared CTA's stream into ring slot g % SH_NST (warp 0). g < SH_GT: gate_up k-tile g of both + slices: quadrant Q = 2 q + u (q slice, u 0 gate / 1 up) = rows 32 sid .. (+ 768 for up) at k-tile SH_GT q + g, its + raw scale atom at raw slot Q. Else down tile j = (g - SH_GT) / 6 (rows 256 sid + 128 j ..), k-tile + (g - SH_GT) % 6.""" + st = g % cutlass.Int32(SH_NST) + bar = sh_full.subview(st) + if cutlass.const_expr(SH_PRETILE): + if lane == 0: + src = (sid * cutlass.Int32(SH_STAGES) + g) * cutlass.Int32(SH_STAGE_BYTES) + prims.mbarrier_arrive_expect_tx(bar, SH_STAGE_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), gusf.subview(src), bar, 16384 + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + gusf.subview(src + cutlass.Int32(16384)), + bar, + 512, + ) + cute.arch.sync_warp() + else: + if g < cutlass.Int32(SH_GT): + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 4 * 512) + cute.arch.sync_warp() + if lane < 4: + q = lane >> cutlass.Int32(1) + u = lane & cutlass.Int32(1) + row0 = sid * cutlass.Int32(32) + u * cutlass.Int32(SH_I) + kt = q * cutlass.Int32(SH_GT) + g + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A) + lane * cutlass.Int32(4096)), + tma_gu, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview( + cutlass.Int32(_SB_RAW) + + st * cutlass.Int32(2048) + + lane * cutlass.Int32(512) + ), + gusf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_KB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(SH_DKT) + kt = gd - jt * cutlass.Int32(SH_DKT) + row0 = sid * cutlass.Int32(256) + jt * cutlass.Int32(128) + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 512) + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), + tma_dn, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_RAW) + st * cutlass.Int32(2048)), + dnsf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_DKB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + cute.arch.sync_warp() + + +@dsl_user_op +def _red_release_add(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "red.release.gpu.global.add.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_acquire_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.acquire.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fence_proxy_async_global(*, loc=None, ip=None): + _asm(None, [], "fence.proxy.async.global;", "", True, loc, ip) + + +@cute.jit +def _sfa_to_tmem(sSFA: cutlass.Array, stage, base_row, base_col): + """(SFA_CP, elected MMA thread) the stage's 4 scale atoms smem -> TMEM (what the splicer warp does otherwise).""" + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SFA1_COL + stage * cutlass.Int32(16) + kb * 4), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + + +@cute.jit +def _sf1_cp( + lane, stage, sf_line, c, sSFA: cutlass.Array, s1w: cutlass.Array, ab_full: cutlass.Array +): + """(SFA_CP) FC1 stage scale atoms: word [r][q] of atom kb <- raw atom 4 q + kb (raw line sf_line = 4 x atom) + word [r][c % 4] (what the splicer copies out of the raw box), 4-byte cp.async by the warp's 32 lanes; their + arrivals (noinc) complete the stage's full barrier together with the A boxes' bytes.""" + src0 = ( + (sf_line >> cutlass.Int32(2)) * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (c & cutlass.Int32(3)) + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for q in cutlass.range_constexpr(C1): + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + q)), + s1w.subview(src0 + cutlass.Int32((4 * q + kb) * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _sf2_cp( + lane, + stage, + g, + blk, + t2: cutlass.Constexpr, + n_here, + s_el: cutlass.Array, + sSFA: cutlass.Array, + s2w: cutlass.Array, + ab_full: cutlass.Array, +): + """(SFA_CP) FC2 k-tile t2 scale atoms: word [r][jj] of atom kb <- raw atom (rows / 128, K block 4 t2 + kb) of slot + jj's expert, word [r][blk % 4].""" + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + src0 = ( + ( + (e * cutlass.Int32(SF2_RB) + (blk >> cutlass.Int32(2))) * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (blk & cutlass.Int32(3)) + ) + # (a partial last k-tile has no atoms past I_TP) + for kb in cutlass.range_constexpr(_nkb2(t2)): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + jj)), + s2w.subview(src0 + cutlass.Int32(kb * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _shared_cta( + tidx, + warp, + lane, + sid, + s_big: cutlass.Array, + sh_bar: cutlass.Array, + epochs: cutlass.Array, + tdbg: cutlass.Array, + bx, + xb32: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, + tmem_ptr_i32: cutlass.Array, + gu8: cutlass.Array, + dn8: cutlass.Array, +): + """Shared CTA sid (see SHARED): tcgen05 MXF8 (E4M3 x E4M3, UE8M0 per 32 K), m3_mx_gemv's tile order. gate_up: one + 128-row MMA tile = 2 K slices x (32 gate + 32 up rows), B rows 8 q + t = token t's slice-q k-tile; down: 2 tiles + of 128 rows, full K. Warps: 0 weights (TMA), 1 TMEM + scale splice + MMA, 4-7 epilogue; 2.. activation quant.""" + big32 = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Int32, alignment=16) + bigf = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Float32, alignment=16) + big8 = cutlass.Array(s_big.data_ptr(0), shape=(BIG,), dtype=cutlass.Int8, alignment=16) + sh_full = sh_bar # [0, SH_NST): stage full + sh_empty = cutlass.Array(sh_bar.data_ptr(8), shape=(8,), dtype=cutlass.Int64, alignment=8) + b_gu_rdy = sh_bar.subview(16) + b_dn_rdy = sh_bar.subview(17) + acc_gu = sh_bar.subview(18) + acc_dn = sh_bar.subview(19) + if warp == 0: + prims.prefetch_tensormap(tma_gu) + prims.prefetch_tensormap(tma_dn) + if lane < SH_NST: + prims.mbarrier_init(sh_full.subview(lane), 1) + prims.mbarrier_init( + sh_empty.subview(lane), 33 + ) # the MMAs' commit + each splicer lane's reads done + if lane == 0: + prims.mbarrier_init(b_gu_rdy, 1) + prims.mbarrier_init(b_dn_rdy, 1) + prims.mbarrier_init(acc_gu, 1) + prims.mbarrier_init(acc_dn, 1) + if warp == 1: + prims.tcgen05_alloc(tmem_ptr_i32, SH_TMEM) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + if warp == 0: + for g in cutlass.range_constexpr(SH_NST): # layer constants: before the grid dependency + _sh_issue(lane, sid, cutlass.Int32(g), s_big, sh_full, tma_gu, tma_dn, gusf, dnsf) + # the rest of this CTA's weights into L2 (64 gate / up rows of 6 KB, 256 down rows of 768 B) + if cutlass.const_expr(SH_PRETILE): + # this CTA's stages beyond the ring, 4 per lane + for r in cutlass.range_constexpr(SH_PF): + pg = cutlass.Int32(SH_NST) + lane * cutlass.Int32(SH_PF) + cutlass.Int32(r) + if pg < cutlass.Int32(SH_STAGES): + _l2_prefetch( + gusf.subview( + (sid * cutlass.Int32(SH_STAGES) + pg) * cutlass.Int32(SH_STAGE_BYTES) + ) + .data_ptr() + .toint(), + SH_STAGE_BYTES, + ) + else: + for r in cutlass.range_constexpr(2): + prw = sid * cutlass.Int32(32) + cutlass.Int32(r * SH_I) + lane + _l2_prefetch(gu8.subview(prw * cutlass.Int32(H)).data_ptr().toint(), H) + if lane == 0: + _l2_prefetch( + dn8.subview(sid * cutlass.Int32(256 * SH_I)).data_ptr().toint(), 256 * SH_I + ) + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(not LATE_TRIGGER): + if arp.load(idx=0) == cutlass.Int64(0): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + if cutlass.const_expr(EARLY_EPOCH): + prims.barrier_cta_sync(0) # every thread holds this call's epoch + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + NQ = THREADS - 64 # activation quant threads (warps 2 ..) + if warp == 0: + for g in cutlass.range(SH_NST, SH_STAGES, unroll=1): + _wait( + sh_empty.subview(g % cutlass.Int32(SH_NST)), + ((g // cutlass.Int32(SH_NST)) - cutlass.Int32(1)) & cutlass.Int32(1), + ) + _sh_issue(lane, sid, g, s_big, sh_full, tma_gu, tma_dn, gusf, dnsf) + elif warp == 1: + # ---- MMA: gate_up (scale atoms spliced to the tile's row order), then down + idesc_gu = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=SH_NB, + m_dim=128, + ) + idesc_dn = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=8, + m_dim=128, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + desc_a0 = prims.Tcgen05SmemDesc.build( + s_big.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bx = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BX), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bh = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BH), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + sq = sid & cutlass.Int32(3) + _wait(b_gu_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for g in cutlass.range(SH_STAGES, unroll=1): + st = g % cutlass.Int32(SH_NST) + if g == cutlass.Int32(SH_GT): + _wait(b_dn_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + _wait(sh_full.subview(st), (g // cutlass.Int32(SH_NST)) & cutlass.Int32(1)) + raw = cutlass.Int32(_SB_RAW // 4) + st * cutlass.Int32(512) + if (g < cutlass.Int32(SH_GT)) & cutlass.Boolean(not SH_PRETILE): + # MMA row 32 Q + i = quadrant Q's row i: word 4 i + Q <- raw atom Q's word 4 i + sid % 4 + w4 = [ + big32.load(idx=raw + cutlass.Int32(128 * qq) + lane * cutlass.Int32(4) + sq) + for qq in range(4) + ] + big32.store( + ( + cutlass.Int32(w4[0]), + cutlass.Int32(w4[1]), + cutlass.Int32(w4[2]), + cutlass.Int32(w4[3]), + ), + idx=cutlass.Int32(_SB_SPL // 4) + + st * cutlass.Int32(128) + + lane * cutlass.Int32(4), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + prims.mbarrier_arrive( + sh_empty.subview(st) + ) # this lane's generic reads of the slot are done + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + if g < cutlass.Int32(SH_GT): + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bx + cutlass.Int64(g * cutlass.Int32(SH_NB * 128 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr(tcol + SH_ACC_GU, 6, cutlass.Float32), + da + 2 * kb, + db + 2 * kb, + idesc_gu.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (g > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_GU + g * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_GT - 1): + prims.tcgen05_commit(acc_gu) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(SH_DKT) + kt = gd - jt * cutlass.Int32(SH_DKT) + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview( + cutlass.Int32(_SB_SPL if SH_PRETILE else _SB_RAW) + + st * cutlass.Int32(512 if SH_PRETILE else 2048) + ), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bh + cutlass.Int64(kt * cutlass.Int32(1024 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr( + tcol + SH_ACC_DN + jt * cutlass.Int32(8), 6, cutlass.Float32 + ), + da + 2 * kb, + db + 2 * kb, + idesc_dn.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (kt > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_DN + kt * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_STAGES - 1): + prims.tcgen05_commit(acc_dn) + cute.arch.sync_warp() + if warp >= 2: + # ---- the activation, MXFP8 per 32-K block (native mxfp8_quantize), into B (row 8 q + t of stage tile i = + # token t's k-tile SH_GT q + i, 128-byte swizzle) and its codes + qt = tidx - cutlass.Int32(64) + for r in cutlass.range_constexpr((M_MAX * SH_KB + NQ - 1) // NQ): + q_ = qt + cutlass.Int32(r * NQ) + if q_ < cutlass.Int32(M_MAX * SH_KB): + t = q_ // cutlass.Int32(SH_KB) + b = q_ - t * cutlass.Int32(SH_KB) + wv = [ + xb32.load( + idx=t * cutlass.Int32(H // 2) + + b * cutlass.Int32(16) + + cutlass.Int32(4 * z), + vector_size=4, + alignment=16, + ) + for z in range(4) + ] + v = [] + for z in cutlass.range_constexpr(4): + for zz in cutlass.range_constexpr(4): + w = cutlass.Int32(wv[z][zz]) + v.append((w << cutlass.Int32(16)).bitcast(cutlass.Float32)) + v.append((w & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(v[z])) + code, osc = _mx_osc(amax) + words = _mx_words(v, osc) + kt = b >> cutlass.Int32(2) + sl = kt // cutlass.Int32(SH_GT) + it = kt - sl * cutlass.Int32(SH_GT) + n = sl * cutlass.Int32(8) + t + c0 = (b & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BX) + it * cutlass.Int32(SH_NB * 128) + n * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + + (((c0 + cutlass.Int32(1)) ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_XC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR, number_of_threads=NQ) + if warp >= 4: + if warp < 8: + qd = warp - cutlass.Int32(4) + # B scales of gate_up to TMEM: column of stage i, lane n = 8 q + t: token t's codes of k-tile SH_GT q + i + n = lane + t = n & cutlass.Int32(7) + ok = (n < cutlass.Int32(SH_NB)) & (t < cutlass.Int32(M_MAX)) + for i in cutlass.range_constexpr(SH_GT): + kt = (n >> cutlass.Int32(3)) * cutlass.Int32(SH_GT) + cutlass.Int32(i) + wsf = big32.load( + idx=( + cutlass.Int32(_SB_XC) + + cutlass.select_(ok, t, cutlass.Int32(0)) * cutlass.Int32(SH_KB) + + kt * cutlass.Int32(4) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_GU + i * 4), + cutlass.select_(ok, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + et = tidx - cutlass.Int32(128) + if et == 0: + prims.mbarrier_arrive(b_gu_rdy) + _stamp(tdbg, bx, 54) + # ---- gate_up out: quadrant qd lane l = MMA row 32 qd + l; slice q = qd / 2 holds token t at column 8 q + t + _wait(acc_gu, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if cutlass.const_expr(SH_SPLIT == 1): + # TP2: one K slice; quadrant qd = rows 32 qd .. (gate positions 0..63, then up 0..63), no partials: + # the owner sum is (0 + acc), then bf16, at _SB_GU word (u SH_ROWS + p) 8 + t = (32 qd + l) 8 + t + av = prims.tcgen05_ld( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_ACC_GU, cutlass.Float32), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + elif cutlass.const_expr(SH_SPLIT == 2): + av = prims.tcgen05_ld( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_ACC_GU, cutlass.Float32), + num=16, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(2): + for tt in cutlass.range_constexpr(8): + bigf.store( + cutlass.Float32(av[8 + tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(2)) * cutlass.Int32(32) + lane) + * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd < cutlass.Int32(2): + # m3_mx_gemv's owner sum: (0 + rank 0) + rank 1, then bf16 + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + else: + # 16-row octants: warp qd = slice qd, lane l = (u = l / 16 gate / up, position l % 16); its tokens are + # accumulator columns 8 qd + t + av = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_GU + qd * cutlass.Int32(8), + cutlass.Float32, + ), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(1): + for tt in cutlass.range_constexpr(8): + bigf.store( + cutlass.Float32(av[tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(1)) * cutlass.Int32(32) + lane) + * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd == cutlass.Int32(0): + # m3_mx_gemv's owner sum: (((0 + rank 0) + rank 1) + rank 2) + rank 3, then bf16 + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + for s_ in cutlass.range_constexpr(1, SH_SPLIT): + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (cutlass.Int32((s_ - 1) * 32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + lane * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + # h = silu_and_mul(gate, up) of positions 32 sid + p, published under the call's tag + for r in cutlass.range_constexpr((SH_ROWS * M_MAX + 127) // 128): + pq = et + cutlass.Int32(r * 128) + if pq < cutlass.Int32(SH_ROWS * M_MAX): + p = pq // cutlass.Int32(M_MAX) + tt = pq - p * cutlass.Int32(M_MAX) + ga = bigf.load(idx=cutlass.Int32(_SB_GU // 4) + p * cutlass.Int32(8) + tt) + up = bigf.load( + idx=cutlass.Int32(_SB_GU // 4) + + (cutlass.Int32(SH_ROWS) + p) * cutlass.Int32(8) + + tt + ) + q_a = cute.arch.fmin(cutlass.Float32(ga), cutlass.Float32(7.0)) + q_bu = _min_xorsign_abs(cutlass.Float32(up), cutlass.Float32(7.0)) + q_ub = _add_rn(q_bu, cutlass.Float32(1.0)) + q_e2 = _ex2_full( + _mul_rn( + _mul_rn(q_a, cutlass.Float32(-1.70200002193450927734)), + cutlass.Float32(1.4426950216293334961), + ) + ) + q_sg = _div_full(cutlass.Float32(1.0), _add_rn(q_e2, cutlass.Float32(1.0))) + q_h = _mul_rn(q_ub, _mul_rn(q_sg, q_a)) + _st_relaxed_u32( + hx.subview(tt * cutlass.Int32(SH_I) + sid * cutlass.Int32(SH_ROWS) + p) + .data_ptr() + .toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(q_h), + ) + if et == 0: + _stamp(tdbg, bx, 50) + # every shared CTA's h: (token, K block) per thread, 32 tagged words, MXFP8 into B (h) and its codes + for rr in cutlass.range_constexpr((M_MAX * SH_DKB + 127) // 128): + q_ = et + cutlass.Int32(rr * 128) + if q_ < cutlass.Int32(M_MAX * SH_DKB): + t = q_ // cutlass.Int32(SH_DKB) + kb = q_ - t * cutlass.Int32(SH_DKB) + base = t * cutlass.Int32(SH_I) + kb * cutlass.Int32(32) + hv = [cutlass.Int32(0) for _ in range(32)] + pend = cutlass.Int32(1) + t0h = cutlass.Int64(0) + while pend != cutlass.Int32(0): + t0h = _hang_tick(t0h) + for q4 in cutlass.range_constexpr(8): + v4 = prims.load_ext( + hx.subview(base + cutlass.Int32(4 * q4)), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for z in cutlass.range_constexpr(4): + hv[4 * q4 + z] = cutlass.Int32(v4[z]) + okw = cutlass.Boolean(True) + for z in cutlass.range_constexpr(32): + okw = okw & ( + ((hv[z] >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) == tag16 + ) + pend = cutlass.select_(okw, cutlass.Int32(0), cutlass.Int32(1)) + vals = [ + (hv[z] << cutlass.Int32(16)).bitcast(cutlass.Float32) for z in range(32) + ] + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(vals[z])) + code, osc = _mx_osc(amax) + words = _mx_words(vals, osc) + kt = kb >> cutlass.Int32(2) + c0 = (kb & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BH) + kt * cutlass.Int32(1024) + t * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + (((c0 + cutlass.Int32(1)) ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_HC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + okh = lane < cutlass.Int32(M_MAX) + for kt in cutlass.range_constexpr(SH_DKT): + wsf = big32.load( + idx=( + cutlass.Int32(_SB_HC) + + cutlass.select_(okh, lane, cutlass.Int32(0)) * cutlass.Int32(SH_DKB) + + cutlass.Int32(4 * kt) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_DN + kt * 4), + cutlass.select_(okh, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if et == 0: + prims.mbarrier_arrive(b_dn_rdy) + _stamp(tdbg, bx, 51) + # ---- down out: tile j row 32 qd + l -> y_sh row 256 sid + 128 j + 32 qd + l + _wait(acc_dn, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for jt in cutlass.range_constexpr(2): + dv = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_DN + jt * 8, + cutlass.Float32, + ), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + rowo = ( + sid * cutlass.Int32(256) + + cutlass.Int32(128 * jt) + + qd * cutlass.Int32(32) + + lane + ) + for tt in cutlass.range_constexpr(M_MAX): + _st_relaxed_u32( + dnx.subview(cutlass.Int32(tt * H) + rowo).data_ptr().toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(cutlass.Float32(dv[tt])), + ) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if et == 0: + _stamp(tdbg, bx, 52) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if warp == 1: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc(cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), SH_TMEM) + if cutlass.const_expr(not EARLY_EPOCH): + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + + +@cute.kernel +def m3_moe_kernel( + tma_a1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_a2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s1_desc: cutlass.GridConstant[cuda.TensorMap], # w3_w1_weight_scale as 128-byte lines + tma_s2_desc: cutlass.GridConstant[cuda.TensorMap], # w2_weight_scale as 128-byte lines + tma_gu_desc: cutlass.GridConstant[ + cuda.TensorMap + ], # SHARED: gate_up e4m3 [1536, 6144], box 128 B x 32 rows + tma_dn_desc: cutlass.GridConstant[ + cuda.TensorMap + ], # SHARED: down e4m3 [6144, 768], box 128 B x 128 rows + logits: cutlass.Array, # f32 [M, 128] (not FUSED_ROUTER) + bias: cutlass.Array, # f32 [128] + x32: cutlass.Array, # int32 words of x_fp4 [M, 3072 B] (not FUSED_ROUTER) + xsf32: cutlass.Array, # int32 words of x_sf [M, 384 B] (not FUSED_ROUTER) + escal: cutlass.Array, # f32 [6, 128]: fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha + plan: cutlass.Array, # int32 [4 M + 1, NCTA, NG_CAP + 1] by G: per group i0 | i1 << 16, then b0 | nb << 16 + partials: cutlass.Array, # int32 bits of f32 [G_CAP * 48, M, 32]: split units' head sums + hbuf32: cutlass.Array, # int32 [G_CAP * M, 48, 4]: tagged intermediate records + epochs: cutlass.Array, # int32 [NCTA] + out: cutlass.Array, # bf16 [M, 6144] + route_dbg: cutlass.Array, # int32 [M, 8]: ids, weight bits (CTA 0) + tdbg: cutlass.Array, # int64 [NCTA, 32] timing stamps (TIMING builds) + xb32: cutlass.Array, # FUSED_ROUTER: int32 words (bf16 pairs) of the hidden states [M, 6144] + rw: cutlass.Array, # FUSED_ROUTER: f32 router weight [128, 6144] + sfsc: cutlass.Array, # FUSED_ROUTER: f32 [1] fc31_input_scale + lgbuf: cutlass.Array, # FUSED_ROUTER: int64 [LG_REPL, LG_STRIDE] logits: tag << 32 | f32 bits + guw: cutlass.Array, # SHARED: uint8 gate_up e4m3 [1536, 6144] + gusf: cutlass.Array, # SHARED: uint8 its swizzled UE8M0 scales + dnw: cutlass.Array, # SHARED: uint8 down e4m3 [6144, 768] + dnsf: cutlass.Array, # SHARED: uint8 its swizzled UE8M0 scales + hx: cutlass.Array, # SHARED: int32 [M, 768]: h as (call tag << 16 | bf16), tag 0 = never written + dnx: cutlass.Array, # SHARED: int32 [M, 6144]: y_sh as (call tag << 16 | bf16), tag 0 = never written + arp: cutlass.Array, # int64 [8]: all-reduce push (enable, mc, flags, rank, world, maxt); enable 0 = off + s1w: cutlass.Array, # SFA_CP: int32 words of w3_w1_weight_scale + s2w: cutlass.Array, # SFA_CP: int32 words of w2_weight_scale + img32: cutlass.Array, # B2IMG: int32 [NG_CAP, IMG_BYTES / 4] per-group FC2 B image (tile + scale atoms) + b2cnt: cutlass.Array, # B2IMG: int32 [NG_CAP * 8 + 8] per-group unit counters (index 8 g) +): + tidx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tidx % 32 + if tidx == 0: + _stamp(tdbg, bx, 0) + if cutlass.const_expr(TIMING): + if tidx == 0: + tdbg.store( + cutlass.Int64(_smid()), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(63) + ) + + s_big = cutlass.Array(cutlass.Int8, BIG, space=cutlass.AddressSpace.smem, alignment=1024) + sA = cutlass.Array( + s_big.data_ptr(0), shape=(NUM_BYTES_A * STAGES,), dtype=cutlass.Int8, alignment=1024 + ) + sSFAraw = cutlass.Array( + s_big.data_ptr(_OFF_SFAR), + shape=(max(1, _RAW_STAGE // 4) * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sSFA = cutlass.Array( + s_big.data_ptr(_OFF_SFA), + shape=(NUM_WORDS_SFA * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sB1 = cutlass.Array( + s_big.data_ptr(_OFF_B1), shape=(NUM_BYTES_B1 * S1,), dtype=cutlass.Int8, alignment=1024 + ) + sB2 = cutlass.Array( + s_big.data_ptr(_OFF_B2), shape=(2 * KT2 * NUM_BYTES_B2,), dtype=cutlass.Int8, alignment=1024 + ) + sSFB2 = cutlass.Array( + s_big.data_ptr(_OFF_SFB2), + shape=(2 * KT2 * NUM_KB * 128,), + dtype=cutlass.Int32, + alignment=1024, + ) + ys = cutlass.Array( + cutlass.Float32, NB2_CAP * M_MAX * TOP_K * R2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_part = cutlass.Array( + cutlass.Float32, 2 * M_MAX * C1 * R1, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_bias = cutlass.Array(cutlass.Float32, E_LOCAL, space=cutlass.AddressSpace.smem, alignment=16) + s_red = cutlass.Array(cutlass.Float32, 9 * M_MAX, space=cutlass.AddressSpace.smem, alignment=16) + s_xsf = cutlass.Array( + cutlass.Int32, + M_MAX * (H // 64) if FUSED_ROUTER else 4, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + ab_full = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + ab_empty = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + sfa_ready = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + acc_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_ready = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_free = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + # this CTA's FC1 share of group g is out: one barrier per group parity (a single barrier could complete twice + # while warp 5 still works on an earlier group, and its parity wait would then miss them) + fc1_done = cutlass.Array(cutlass.Int64, 4, space=cutlass.AddressSpace.smem, alignment=8) + b1_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + x_bar = cutlass.Array( + cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8 + ) # x landed, x consumed + img_bar = cutlass.Array( + cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8 + ) # (B2IMG) image landed + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + s_el = cutlass.Array( + cutlass.Int32, G_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16 + ) # slots, then G + s_kpos = cutlass.Array( + cutlass.Int32, G_CAP * M_MAX, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_w = cutlass.Array( + cutlass.Float32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_ids = cutlass.Array( + cutlass.Int32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_plan = cutlass.Array(cutlass.Int32, NG_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + # this CTA's plan rows for every G (static data: loaded before the grid dependency) + s_misc = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_plan_all = cutlass.Array( + cutlass.Int32, (G_CAP + 1) * (NG_CAP + 1), space=cutlass.AddressSpace.smem, alignment=16 + ) + s_esc = cutlass.Array( + cutlass.Float32, 8 * G_CAP, space=cutlass.AddressSpace.smem, alignment=32 + ) # slot s at 8 s + + is_routed = cutlass.Boolean(True) + if cutlass.const_expr(SHARED): + sh_full = cutlass.Array(cutlass.Int64, 32, space=cutlass.AddressSpace.smem, alignment=8) + is_routed = bx < cutlass.Int32(NCTA) + if bx >= cutlass.Int32(NCTA): + _shared_cta( + tidx, + warp, + lane, + bx - cutlass.Int32(NCTA), + s_big, + sh_full, + epochs, + tdbg, + bx, + xb32, + tma_gu_desc.get_ptr(), + tma_dn_desc.get_ptr(), + gusf, + dnsf, + hx, + dnx, + arp, + tmem_ptr_i32, + guw, + dnw, + ) + if is_routed: + # ---- before the grid dependency + if warp == 0: + if tidx < STAGES: + prims.mbarrier_init(ab_full.subview(tidx), 33 if SFA_CP else 1) + prims.mbarrier_init(ab_empty.subview(tidx), 1) + prims.mbarrier_init(sfa_ready.subview(tidx), 1) + if tidx < 2: + prims.mbarrier_init(acc_full.subview(tidx), 1) + prims.mbarrier_init(acc_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(acc2_full.subview(tidx), 1) + prims.mbarrier_init(acc2_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(b2_ready.subview(tidx), 1) + prims.mbarrier_init(b2_free.subview(tidx), EPI_THREADS) + if cutlass.const_expr(B2IMG): + prims.mbarrier_init(img_bar.subview(tidx), 1) + if tidx < 4: + prims.mbarrier_init(fc1_done.subview(tidx), EPI_THREADS) + if tidx == 0: + prims.mbarrier_init(b1_ready.subview(0), EPI_THREADS) + if cutlass.const_expr(XSTAGE): + prims.mbarrier_init(x_bar.subview(0), 1) + prims.mbarrier_init(x_bar.subview(1), EPI_THREADS) + if warp == MMA_WARP: + prims.tcgen05_alloc(tmem_ptr_i32, TMEM_COLS) + prims.tcgen05_relinquish_alloc_permit() + if warp == W_WARP: + prims.prefetch_tensormap(tma_a1_desc.get_ptr()) + prims.prefetch_tensormap(tma_a2_desc.get_ptr()) + prims.prefetch_tensormap(tma_s1_desc.get_ptr()) + prims.prefetch_tensormap(tma_s2_desc.get_ptr()) + for i in cutlass.range_constexpr(E_LOCAL // 32): + s_bias.store(cutlass.Float32(bias.load(idx=lane + i * 32)), idx=lane + i * 32) + for i in cutlass.range_constexpr(((G_CAP + 1) * (NG_CAP + 1) + 31) // 32): + q = lane + cutlass.Int32(32 * i) + if q < cutlass.Int32((G_CAP + 1) * (NG_CAP + 1)): + gq = q // cutlass.Int32(NG_CAP + 1) + s_plan_all.store( + plan.load( + idx=(gq * cutlass.Int32(NCTA) + bx) * cutlass.Int32(NG_CAP + 1) + + q + - gq * cutlass.Int32(NG_CAP + 1) + ), + idx=q, + ) + rwv = [] + if cutlass.const_expr(FUSED_ROUTER): + tg = cutlass.select_(tidx < cutlass.Int32(GEMV_THREADS), tidx, cutlass.Int32(0)) + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + rbx = cutlass.select_(eg < cutlass.Int32(NROUTE), eg, cutlass.Int32(0)) + rwv.append( + [ + rw.load( + idx=rbx * cutlass.Int32(H) + + tg * cutlass.Int32(4) + + cutlass.Int32(1024 * j), + vector_size=4, + alignment=16, + ) + for j in range(H // 1024) + ] + ) + if cutlass.const_expr(TIMING): + if tidx == 0: + s_el.store(cutlass.Int32(-7), idx=G_CAP) + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(XSTAGE): + if warp == 0: + if lane == 0: + prims.mbarrier_arrive_expect_tx(x_bar.subview(0), M_MAX * H * 2) + cute.arch.sync_warp() + if lane < M_MAX: + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(lane * cutlass.Int32(H * 2)), + xb32.subview(lane * cutlass.Int32(H // 2)), + x_bar.subview(0), + H * 2, + ) + ar_on = arp.load(idx=0) != cutlass.Int64(0) + ar_off = arp.load(idx=0) == cutlass.Int64(0) + if cutlass.const_expr(not LATE_TRIGGER): + if ar_off: + cute.arch.griddepcontrol_launch_dependents() + if cutlass.const_expr(LATE_TRIGGER and TAIL_TRIGGER and NHEAVY > 0): + if ar_off & (bx < cutlass.Int32(NHEAVY)): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + + ep = epochs.load(idx=bx) + tagv = ((ep & cutlass.Int32(0x7F)) | cutlass.Int32(0x80)) << cutlass.Int32(24) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + ltag = cutlass.Int64(ep + cutlass.Int32(1)) << cutlass.Int64(32) + if cutlass.const_expr(FUSED_ROUTER): + # ---- router GEMV (experts bx, bx + NCTA, .., threads 0..255), m3_router_gemv's order exactly + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + if (eg < cutlass.Int32(NROUTE)) & (tidx < cutlass.Int32(GEMV_THREADS)): + accs = [cutlass.Float32(0.0) for _ in range(M_MAX)] + xsrc = xb32 + if cutlass.const_expr(GEMV_SMEM and XSTAGE): + _wait(x_bar.subview(0), 0) + xsrc = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + for j in cutlass.range_constexpr(H // 1024): + for t in cutlass.range_constexpr(M_MAX): + xw = xsrc.load( + idx=cutlass.Int32(t * (H // 2) + 512 * j) + tidx * cutlass.Int32(2), + vector_size=2, + alignment=8, + ) + x0 = cutlass.Int32(xw[0]) + x1 = cutlass.Int32(xw[1]) + a = accs[t] + a = _fma_rn( + (x0 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][0]), + a, + ) + a = _fma_rn( + (x0 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][1]), + a, + ) + a = _fma_rn( + (x1 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][2]), + a, + ) + a = _fma_rn( + (x1 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][3]), + a, + ) + accs[t] = a + for t in cutlass.range_constexpr(M_MAX): + v = accs[t] + for off in (16, 8, 4, 2, 1): + v = _add_rn(v, cute.arch.shuffle_sync_bfly(v, off)) + if lane == 0: + s_red.store(v, idx=warp * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < M_MAX: + sacc = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(8): + sacc = _add_rn(sacc, s_red.load(idx=cutlass.Int32(w * M_MAX) + tidx)) + s_red.store(sacc, idx=cutlass.Int32(8 * M_MAX) + tidx) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < LG_REPL: + for t in cutlass.range_constexpr(M_MAX): + lw = ltag | ( + cutlass.Int64( + cutlass.Float32(s_red.load(idx=8 * M_MAX + t)).bitcast( + cutlass.Int32 + ) + ) + & cutlass.Int64(0xFFFFFFFF) + ) + _st_relaxed_u64( + lgbuf.subview( + tidx * cutlass.Int32(LG_STRIDE) + + cutlass.Int32(t * E_LOCAL) + + eg + ) + .data_ptr() + .toint(), + lw, + ) + if cutlass.const_expr(NGEMV > 1): + cute.arch.sync_warp() + cute.arch.barrier( + barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS + ) # s_red reused next + if tidx == 0: + _stamp(tdbg, bx, 21) + + # ---- routing (warps 4..7, token t on warp 4 + t % 4); the epilogue warps build B meanwhile + if ((warp >= W_WARP) & (warp < cutlass.Int32(W_WARP + NRW0))) | ( + (warp >= cutlass.Int32(8)) & (warp < cutlass.Int32(8 + RW_EXTRA)) + ): + rw_i = cutlass.select_( + warp < cutlass.Int32(8), warp - cutlass.Int32(W_WARP), warp - cutlass.Int32(4) + ) + # ILP: poll every token's logits first, then run the independent top-k chains back to back + lgv_all = [] + t_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = rw_i + cutlass.Int32(NRW * tr) + if cutlass.const_expr(M_MAX % NRW != 0): + t = cutlass.select_(t < cutlass.Int32(M_MAX), t, cutlass.Int32(M_MAX - 1)) + t_all.append(t) + lgv = [] + if cutlass.const_expr(FUSED_ROUTER): + want = ltag >> cutlass.Int64(32) + rbase = ( + (bx % cutlass.Int32(LG_REPL)) * cutlass.Int32(LG_STRIDE) + + t * cutlass.Int32(E_LOCAL) + + lane + ) + done = cutlass.Int32(0) + res = [cutlass.Int64(0) for _ in range(4)] + t0h = cutlass.Int64(0) + while done == cutlass.Int32(0): + t0h = _hang_tick(t0h) + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(4): + res[z] = _ld_relaxed_u64( + lgbuf.subview(cutlass.Int32(z * 32) + rbase).data_ptr().toint() + ) + ok = ok & ((res[z] >> cutlass.Int64(32)) == want) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32( + -1 + ): + done = cutlass.Int32(1) + else: + _nanosleep(POLL_NS) + for z in cutlass.range_constexpr(4): + lgv.append(cutlass.Int32(res[z]).bitcast(cutlass.Float32)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 22) + else: + for i in cutlass.range_constexpr(4): + lgv.append( + cutlass.Float32( + logits.load( + idx=t * cutlass.Int32(E_LOCAL) + cutlass.Int32(i * 32) + lane + ) + ) + ) + lgv_all.append(lgv) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 61) + res_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = t_all[tr] + lgv = lgv_all[tr] + own = (rw_i + cutlass.Int32(NRW * tr)) < cutlass.Int32(M_MAX) + # top-4 (keys sigmoid + bias, ties to the lower id) + keys = [] + for i in cutlass.range_constexpr(4): + sg = _fma_rn( + cutlass.Float32(0.5), + cute.math.tanh(_mul_rn(cutlass.Float32(0.5), lgv[i])), + cutlass.Float32(0.5), + ) + keys.append(_add_rn(sg, s_bias.load(idx=i * 32 + lane))) + hi = [_twiddle(keys[i]) for i in range(4)] + taken = [cutlass.Boolean(False) for _ in range(4)] + ids_t = [] + kt_t = [] + for k in cutlass.range_constexpr(TOP_K): + bh = cutlass.Int32(0) + bl = cutlass.Int32(0) + for i in cutlass.range_constexpr(4): + lo_i = cutlass.Int32(65535 - i * 32) - lane + hv = cutlass.select_(taken[i], cutlass.Int32(0), hi[i]) + lv = cutlass.select_(taken[i], cutlass.Int32(0), lo_i) + better = (cutlass.Uint32(hv) > cutlass.Uint32(bh)) | ( + (hv == bh) & (cutlass.Uint32(lv) > cutlass.Uint32(bl)) + ) + bh = cutlass.select_(better, hv, bh) + bl = cutlass.select_(better, lv, bl) + mh = _redux_max_u32(bh) + ml = _redux_max_u32(cutlass.select_(bh == mh, bl, cutlass.Int32(0))) + e_k = cutlass.Int32(65535) - ml + for i in cutlass.range_constexpr(4): + taken[i] = taken[i] | (e_k == cutlass.Int32(i * 32) + lane) + ids_t.append(e_k) + kv = cutlass.Float32(0.0) + for i in cutlass.range_constexpr(4): + kv = cutlass.select_(e_k == cutlass.Int32(i * 32) + lane, keys[i], kv) + kt_t.append(cute.arch.shuffle_sync(kv, e_k % cutlass.Int32(32))) + sig = cutlass.Float32(0.0) + e_l = cutlass.Int32(0) + for k in cutlass.range_constexpr(TOP_K): + sk = _sub_rn(kt_t[k], s_bias.load(idx=ids_t[k])) + sig = cutlass.select_(lane == cutlass.Int32(k), sk, sig) + e_l = cutlass.select_(lane == cutlass.Int32(k), ids_t[k], e_l) + total = sig + for off in (16, 8, 4, 2, 1): + total = _add_rn(total, cute.arch.shuffle_sync_bfly(total, off)) + wv = _bf16_round( + _div_rn( + _mul_rn(sig, cutlass.Float32(ROUTE_SCALE)), + _add_rn(total, cutlass.Float32(SUM_EPS)), + ) + ) + res_all.append((t, own, wv, e_l)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 62) + for tr in cutlass.range_constexpr( + (M_MAX + NRW - 1) // NRW + ): # stores after every chain (one branch-free block) + t, own, wv, e_l = res_all[tr] + if (lane < TOP_K) & own: + s_w.store(wv, idx=t * cutlass.Int32(TOP_K) + lane) + s_ids.store(e_l, idx=t * cutlass.Int32(TOP_K) + lane) + if bx == 0: + route_dbg.store( + wv.bitcast(cutlass.Int32), + idx=t * cutlass.Int32(8) + cutlass.Int32(4) + lane, + ) + cute.arch.sync_warp() + if cutlass.const_expr(EARLY0 and not E0_OFFLOAD): + if warp == cutlass.Int32(W_WARP): + # slots 0..3 are always token 0's four (distinct) experts and group 0's stage ranges are the same + # for every G (the plan guarantees it): fill the ring with this CTA's first group-0 stages now + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(W_WARP): + _bar_arrive( + E0_BAR, 64 + ) # token 0's experts are in s_ids: warp 7 issues the early stages + if cutlass.const_expr(B1_AFTER_TOPK): + _bar_arrive(TOPK2_BAR, 128 + 32 * NRW) + cute.arch.barrier(barrier_id=TOPK_BAR, number_of_threads=32 * NRW) + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(E0_WARP): + cute.arch.barrier(barrier_id=E0_BAR, number_of_threads=64) + # slots 0..3 are always token 0's four (distinct) experts and group 0's stage ranges are the same + # for every G (the plan guarantees it): fill the ring with this CTA's first group-0 stages now + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + _bar_arrive(E1_BAR, 64) + if warp == W_WARP: + if lane == 0: + _stamp(tdbg, bx, 20) + # slots: lane l < 4 M is (token l / 4, top-k l % 4); the distinct experts in lane order (match.any) + valid = lane < cutlass.Int32(TOP_K * M_MAX) + my_e = cutlass.select_( + valid, + s_ids.load(idx=cutlass.select_(valid, lane, cutlass.Int32(0))), + cutlass.Int32(-1) - lane, + ) + same = _match_any(my_e) + leader = cute.arch.popc((same & (cutlass.Int32(0) - same)) - cutlass.Int32(1)) + first = valid & (lane == leader) + bal = prims.vote_sync(0xFFFFFFFF, first, prims.VoteSync.BALLOT) + rank = cute.arch.popc(bal & cutlass.Int32(cute.arch.lanemask_lt())) + slot_l = cute.arch.shuffle_sync(rank, leader) + for z in cutlass.range_constexpr((G_CAP * M_MAX + 31) // 32): + if lane + cutlass.Int32(32 * z) < cutlass.Int32(G_CAP * M_MAX): + s_kpos.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + cute.arch.sync_warp() + if first: + s_el.store(my_e, idx=rank) + if valid: + s_kpos.store( + lane % cutlass.Int32(TOP_K), + idx=slot_l * cutlass.Int32(M_MAX) + lane // cutlass.Int32(TOP_K), + ) + if bx == 0: + route_dbg.store( + my_e, + idx=(lane // cutlass.Int32(TOP_K)) * cutlass.Int32(8) + + lane % cutlass.Int32(TOP_K), + ) + gcnt = cute.arch.popc(bal) + if lane == 0: + s_el.store(gcnt, idx=G_CAP) + if lane < cutlass.Int32(NG_CAP + 1): # the plan row of this G (preloaded) + s_plan.store(s_plan_all.load(idx=gcnt * cutlass.Int32(NG_CAP + 1) + lane), idx=lane) + cute.arch.sync_warp() + if cutlass.const_expr(B2IMG): + # CTA 0 pads every group counter to exactly CNT_PER_CALL for this call (absent slots, unused groups) + if (bx == 0) & (lane < cutlass.Int32(NG_CAP)): + nh = gcnt - lane * cutlass.Int32(4) + nh = cutlass.select_(nh > cutlass.Int32(4), cutlass.Int32(4), nh) + nh = cutlass.select_(nh < cutlass.Int32(0), cutlass.Int32(0), nh) + pad = cutlass.Int32(CNT_PER_CALL) - nh * cutlass.Int32(UNITS_ROW) + if pad > cutlass.Int32(0): + _red_release_add( + b2cnt.subview(lane * cutlass.Int32(8)).data_ptr().toint(), pad + ) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 24) + cute.arch.sync_warp() + _bar_arrive(ROUTE_BAR, THREADS) + if warp < 4: + if cutlass.const_expr(B1_AFTER_TOPK): + cute.arch.barrier(barrier_id=TOPK2_BAR, number_of_threads=128 + 32 * NRW) + if cutlass.const_expr(FUSED_ROUTER): + if tidx == 0: + _stamp(tdbg, bx, 57) + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(0), 0) + if tidx == 0: + _stamp(tdbg, bx, 58) + xs32 = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xs32, sfsc, sB1, s_xsf, b1_ready + ) + prims.mbarrier_arrive(x_bar.subview(1)) # the ring is free for the weights + else: + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xb32, sfsc, sB1, s_xsf, b1_ready + ) + if tidx == 0: + _stamp(tdbg, bx, 23) + else: + _build_b1(tidx, warp, lane, base_row, base_col, x32, xsf32, sB1, b1_ready) + if warp != W_WARP: # one sync site for every thread but warp 4's (which arrived) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ROUTE_BAR, number_of_threads=THREADS) + if cutlass.const_expr(EARLY_EPOCH): + if tidx == 0: + epochs.store( + ep + cutlass.Int32(1), idx=bx + ) # every thread read it before the route barrier + if tidx == 0: + _stamp(tdbg, bx, 2) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(s_el.load(idx=G_CAP)), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(28), + ) + + g_n = s_el.load(idx=G_CAP) + ng = (g_n + cutlass.Int32(3)) >> cutlass.Int32(2) + pr2 = s_plan.load(idx=NG_CAP) + blk0 = pr2 & cutlass.Int32(0xFFFF) + nb = pr2 >> cutlass.Int32(16) + + # ---- weights producer (warp 4): phase p = FC1 stages of group p, then FC2 tiles of group p - LAG + if warp == W_WARP: + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) # the staged hidden states are quantized + stage = cutlass.Int32(0) + phase = cutlass.Int32(1) + skip = cutlass.Int32(0) + if cutlass.const_expr(E0_OFFLOAD): + cute.arch.barrier(barrier_id=E1_BAR, number_of_threads=64) + if cutlass.const_expr(EARLY0): + skip = s_misc.load(idx=0) # group-0 stages already in the ring + stage = cutlass.select_(skip == cutlass.Int32(STAGES), cutlass.Int32(0), skip) + phase = cutlass.select_( + skip == cutlass.Int32(STAGES), cutlass.Int32(0), cutlass.Int32(1) + ) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_el.load(idx=slot) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if skip > cutlass.Int32(0): + skip = skip - cutlass.Int32(1) + else: + _waitd(ab_empty.subview(stage), phase, tdbg, 1) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + ), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, stage, sf_line, c, sSFA, s1w, ab_full) + stage, phase = _adv(stage, phase) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + for bb in range(nb): + blk = blk0 + bb + for t2 in cutlass.range_constexpr(KT2): + _waitd(ab_empty.subview(stage), phase, tdbg, 2) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + n_here + * cutlass.Int32( + NUM_BYTES_BOX1 + if SFA_CP + else NUM_BYTES_BOX1 + NUM_BYTES_SFA_RAW // 4 + ), + ) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(jj * NUM_BYTES_BOX1) + ), + tma_a2_desc.get_ptr(), + ( + cutlass.Int32(t2 * KT_BYTES), + blk * cutlass.Int32(R2), + e, + ), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + cutlass.Int32(jj * NUM_WORDS_SFA_RAW // 4) + ), + tma_s2_desc.get_ptr(), + ( + cutlass.Int32(0), + ( + ( + e * cutlass.Int32(SF2_RB) + + (blk >> cutlass.Int32(2)) + ) + * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(4), + ), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf2_cp(lane, stage, g, blk, t2, n_here, s_el, sSFA, s2w, ab_full) + stage, phase = _adv(stage, phase) + if lane == 0: + _stamp(tdbg, bx, 3) + if cutlass.const_expr(PROD_TRIGGER): + if ar_on: + cute.arch.griddepcontrol_launch_dependents() + + # ---- scale splicer (warp 6): every stage, atom kb word [r][q] = raw atom 4 q + kb word [r][m1] + if (warp == S_WARP) & cutlass.Boolean(not SFA_CP): # (SFA_CP: no splicer) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + for p in range(ng + cutlass.Int32(LAG)): + i0 = cutlass.Int32(0) + i1 = cutlass.Int32(0) + for part in cutlass.range_constexpr(2): + cnt = cutlass.Int32(0) + m1 = cutlass.Int32(0) + nk = cutlass.Int32(0) + if cutlass.const_expr(part == 0): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + nk = _nsegs(i0, i1) + else: + if p >= cutlass.Int32(LAG): + nk = nb + for k in range(nk): + if cutlass.const_expr(part == 0): + ug, j0, j1 = _seg(i0, i1, k) + m1 = ( + ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + ) & cutlass.Int32(3) + cnt = j1 - j0 + else: + m1 = (blk0 + k) & cutlass.Int32(3) + cnt = cutlass.Int32(KT2) + for _j in cutlass.range(cnt, unroll=1): + _waitd(ab_full.subview(stage), phase, tdbg, 3) + raw0 = ( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + lane * cutlass.Int32(4) + + m1 + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for kb in cutlass.range_constexpr(NUM_KB): + w0 = sSFAraw.load(idx=raw0 + cutlass.Int32((0 * 4 + kb) * 128)) + w1 = sSFAraw.load(idx=raw0 + cutlass.Int32((1 * 4 + kb) * 128)) + w2 = sSFAraw.load(idx=raw0 + cutlass.Int32((2 * 4 + kb) * 128)) + w3 = sSFAraw.load(idx=raw0 + cutlass.Int32((3 * 4 + kb) * 128)) + sSFA.store( + ( + cutlass.Int32(w0), + cutlass.Int32(w1), + cutlass.Int32(w2), + cutlass.Int32(w3), + ), + idx=dst0 + cutlass.Int32(kb * 128), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + prims.tcgen05_commit(sfa_ready.subview(stage)) + stage, phase = _adv(stage, phase) + + if cutlass.const_expr(B2IMG): + # ---- (B2IMG) group releases (warp 6): this CTA's finished units of group p, one release add per group + if warp == cutlass.Int32(S_WARP): + for p in range(ng): + i0, i1 = _plan_range(s_plan, p) + n_done = cutlass.Int32(0) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + n_done = n_done + cutlass.select_( + (j0 != cutlass.Int32(0)) | (j1 == cutlass.Int32(S1)), + cutlass.Int32(1), + cutlass.Int32(0), + ) + _wait( + fc1_done.subview(p & cutlass.Int32(3)), + (p >> cutlass.Int32(2)) & cutlass.Int32(1), + ) + if cutlass.const_expr(B2_DLY > 0): + t0d = _gtime() + dl = cutlass.Int64( + ( + (bx * cutlass.Int32(37) + p * cutlass.Int32(11) + ep) + & cutlass.Int32(63) + ) + * cutlass.Int32(B2_DLY // 64) + ) + while _gtime() - t0d < dl: + pass + if (lane == 0) & (n_done > cutlass.Int32(0)): + _fence_proxy_async_global() + _red_release_add( + b2cnt.subview(p * cutlass.Int32(8)).data_ptr().toint(), n_done + ) + cute.arch.sync_warp() + # ---- (B2IMG) FC2 B loader (warp 5): group g's image once its counter is complete + if warp == cutlass.Int32(X_WARP): + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + cnt_target = (ep + cutlass.Int32(1)) * cutlass.Int32(CNT_PER_CALL) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + ((g >> cutlass.Int32(1)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + if lane == 0: + _stamp(tdbg, bx, 29) + caddr = b2cnt.subview(g * cutlass.Int32(8)).data_ptr().toint() + if lane == 0: + cv = _ld_acquire_u32(caddr) + t0h = cutlass.Int64(0) + while (cv - cnt_target) < cutlass.Int32(0): + t0h = _hang_tick(t0h) + _nanosleep(B2IMG_NS) + cv = _ld_acquire_u32(caddr) + _fence_proxy_async_global() + prims.mbarrier_arrive_expect_tx(img_bar.subview(buf), IMG_BYTES) + prims.cp_async_bulk_shared_cluster_global( + sB2.subview(buf * cutlass.Int32(IMG_B2)), + img32.subview(g * cutlass.Int32(IMG_BYTES // 4)), + img_bar.subview(buf), + IMG_B2, + ) + prims.cp_async_bulk_shared_cluster_global( + sSFB2.subview(buf * cutlass.Int32(IMG_SF // 4)), + img32.subview( + g * cutlass.Int32(IMG_BYTES // 4) + cutlass.Int32(IMG_B2 // 4) + ), + img_bar.subview(buf), + IMG_SF, + ) + cute.arch.sync_warp() + _waitd( + img_bar.subview(buf), (g >> cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 108 + ) + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 25) + # ---- FC2 B builder (warps 5, 8 ..): group g's records (all 48 per (slot, token), the call's tag on + # every word) into B2 buffer g % 2 (row j M + t, k-tile t2 = record / 16) and its scale atoms, then + # tcgen05.cp to TMEM. + if ((warp == X_WARP) | (warp >= cutlass.Int32(8))) & cutlass.Boolean(not B2IMG): + xi = cutlass.select_( + warp == cutlass.Int32(X_WARP), cutlass.Int32(0), warp - cutlass.Int32(7) + ) + if cutlass.const_expr(G_CAP > TOP_K): + if (bx == 0) & (xi == cutlass.Int32(0)): + # rows of slots this call does not use: tag 0, never taken for a later call's + for i in range( + g_n * cutlass.Int32(M_MAX * UNITS_ROW) + lane, + cutlass.Int32(G_CAP * M_MAX * UNITS_ROW), + 32, + ): + for rr in cutlass.range_constexpr(HB_REPL): + hbuf32.store( + ( + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + ), + idx=i * cutlass.Int32(4) + + (cutlass.Int32(rr * HB_STRIDE) if rr else 0), + alignment=16, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sSFB2_8 = cutlass.Array( + sSFB2.data_ptr(0), shape=(2 * KT2 * NUM_KB * 512,), dtype=cutlass.Int8, alignment=16 + ) + sB2_32 = cutlass.Array( + sB2.data_ptr(0), + shape=(2 * KT2 * NUM_BYTES_B2 // 4,), + dtype=cutlass.Int32, + alignment=16, + ) + nrec_cap = 4 * M_MAX * UNITS_ROW # records of one group + hb_off = cutlass.Int32(0) + if cutlass.const_expr(HB_REPL > 1): + # this CTA's copy of the records + hb_off = (bx % cutlass.Int32(HB_REPL)) * cutlass.Int32(HB_STRIDE) + NZ = (nrec_cap + 32 * XW - 1) // (32 * XW) # records per lane: lane + 32 (XW z + xi) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + ((g >> cutlass.Int32(1)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + _waitd( + fc1_done.subview(g & cutlass.Int32(3)), + (g >> cutlass.Int32(2)) & cutlass.Int32(1), + tdbg, + 4, + ) # this CTA's own share of group g is out + if cutlass.const_expr(TIMING): + if (lane == 0) & (xi == cutlass.Int32(0)) & (g < cutlass.Int32(8)): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(32) + g + ) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + nrec = n_here * cutlass.Int32(M_MAX * UNITS_ROW) + rec0 = g * cutlass.Int32( + 4 * M_MAX * UNITS_ROW + ) # rows (4 g + j) M + t are contiguous + done = cutlass.Int32(0) + rspin = cutlass.Int32(0) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 29) + npoll = cutlass.Int32(0) + got = [[cutlass.Int32(0) for _ in range(3)] for _ in range(NZ)] + t0h = cutlass.Int64(0) + while done == cutlass.Int32(0): + t0h = _hang_tick(t0h) + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + v4 = prims.load_ext( + hbuf32.subview( + (rec0 + qi) * cutlass.Int32(4) + (hb_off if HB_REPL > 1 else 0) + ), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for wq in cutlass.range_constexpr(3): + got[z][wq] = cutlass.Int32(v4[wq]) + ok = ok & ( + ( + ((got[z][0] ^ tagv) | (got[z][1] ^ tagv) | (got[z][2] ^ tagv)) + & cutlass.Int32(-16777216) + ) + == cutlass.Int32(0) + ) + npoll = npoll + cutlass.Int32(1) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32(-1): + done = cutlass.Int32(1) + else: + _nanosleep(B2_POLL_NS) + if cutlass.const_expr(HANG_DBG): + rspin = rspin + cutlass.Int32(1) + if rspin > cutlass.Int32(1 << 14): + done = cutlass.Int32(1) + tdbg.store( + cutlass.Int64(200), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + warp, + ) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + w0, w1, w2 = got[z][0], got[z][1], got[z][2] + rr = qi // cutlass.Int32( + UNITS_ROW + ) # row within the group: j M + t = B row n + c = qi - rr * cutlass.Int32(UNITS_ROW) + t2 = c >> cutlass.Int32(4) + cc = (c >> cutlass.Int32(1)) & cutlass.Int32(7) + lo = (w0 & cutlass.Int32(0xFFFFFF)) | (w1 << cutlass.Int32(24)) + hi = ((w1 >> cutlass.Int32(8)) & cutlass.Int32(0xFFFF)) | ( + (w2 & cutlass.Int32(0xFFFF)) << cutlass.Int32(16) + ) + sB2_32.store( + (lo, hi), + idx=( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_BYTES_B2) + + rr * cutlass.Int32(KT_BYTES) + + (cc ^ (rr & cutlass.Int32(7))) * cutlass.Int32(16) + + (c & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + # scale of block c: atom (t2, kb = (c % 16) / 4), row rr, byte c % 4 + sSFB2_8.store( + cutlass.Int8((w2 >> cutlass.Int32(16)) & cutlass.Int32(0xFF)), + idx=( + ( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_KB) + + ((c & cutlass.Int32(15)) >> cutlass.Int32(2)) + ) + * cutlass.Int32(128) + + rr * cutlass.Int32(4) + ) + * cutlass.Int32(4) + + (c & cutlass.Int32(3)), + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + if cutlass.const_expr(XW > 1): + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=B2_BAR, number_of_threads=32 * XW) + cute.arch.sync_warp() + if (xi == cutlass.Int32(0)) & prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + SFB2_COL + buf * cutlass.Int32(KT2 * NUM_KB * 4) + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 25) + if cutlass.const_expr(TIMING): + if g < cutlass.Int32(8): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(11) + g + ) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(npoll), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(30), + ) + + # ---- MMA (warp 7) + if warp == MMA_WARP: + idesc1 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N1, + m_dim=MMA_M, + sparsity_version=0, + ) + idesc2 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N2, + m_dim=MMA_M, + sparsity_version=0, + ) + desc_a0 = prims.Tcgen05SmemDesc.build( + sA.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b10 = prims.Tcgen05SmemDesc.build( + sB1.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b20 = prims.Tcgen05SmemDesc.build( + sB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + _waitd(b1_ready, 0, tdbg, 5) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + acc_b2 = cutlass.Int64(0) + acc_sf = cutlass.Int64(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + _waitd( + acc_empty.subview(ab), + ((k1 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + tdbg, + 102, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc_ptr = cutlass.inttoptr( + tcol + ACC1_COL + ab * cutlass.Int32(N1), 6, cutlass.Float32 + ) + for j in cutlass.range(j0, j1, unroll=1): + tw0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 6) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 6) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b10 + cutlass.Int64(j * cutlass.Int32(NUM_BYTES_B1 >> 4)) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = tcol + SFB1_COL + j * cutlass.Int32(NUM_KB * 4) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc1, + (j > j0) | cutlass.Boolean(kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc_full.subview(ab)) + k1 = k1 + cutlass.Int32(1) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + buf = g & cutlass.Int32(1) + tw2 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + _waitd( + b2_ready.subview(buf), (g >> cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 103 + ) + if cutlass.const_expr(TIMING): + acc_b2 = acc_b2 + (_gtime() - tw2) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 4) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_empty.subview(a2), + ((k2 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + tdbg, + 104, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc2_ptr = cutlass.inttoptr( + tcol + ACC2_COL + a2 * cutlass.Int32(N2), 6, cutlass.Float32 + ) + for t2 in cutlass.range_constexpr(KT2): + tw1 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 7) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 7) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw1) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b20 + cutlass.Int64( + (buf * cutlass.Int32(KT2) + cutlass.Int32(t2)) + * cutlass.Int32(NUM_BYTES_B2 >> 4) + ) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = ( + tcol + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + cutlass.Int32(t2 * NUM_KB * 4) + ) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(_nkb2(t2)): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc2_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc2, + cutlass.Boolean(t2 > 0 or kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc2_full.subview(a2)) + k2 = k2 + cutlass.Int32(1) + if cutlass.const_expr(TIMING): + if lane == 0: + tdbg.store(acc_b2, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(26)) + tdbg.store(acc_sf, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(27)) + + # ---- epilogue warps (0-3) + if warp < 4: + row_w = base_row + warp * 32 + for r in cutlass.range_constexpr((6 * G_CAP + EPI_THREADS - 1) // EPI_THREADS): + qi = tidx + cutlass.Int32(r * EPI_THREADS) + sl_e = qi // cutlass.Int32(6) + if (qi < cutlass.Int32(6 * G_CAP)) & (sl_e < g_n): + fld = qi - sl_e * cutlass.Int32(6) + s_esc.store( + cutlass.Float32( + escal.load(idx=fld * cutlass.Int32(E_LOCAL) + s_el.load(idx=sl_e)) + ), + idx=sl_e * cutlass.Int32(8) + fld, + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + SH_ADD = SHARED + wys = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + pars = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + ard = (cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0)) + for p in range(ng + cutlass.Int32(LAG)): + if cutlass.const_expr(AR_PRE): + if (p == ng + cutlass.Int32(LAG - 1)) & ar_on: + # one 8-byte load each (the vectorizer merges adjacent arp.load()s into 16-byte reads + # at 8-byte offsets: misaligned) + ard = tuple( + cutlass.Int64(_ld_relaxed_u64(arp.subview(i_).data_ptr().toint())) + for i_ in (1, 3, 4, 5) + ) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_cl = cutlass.select_( + f_bb < nb, (blk0 + f_bb) * cutlass.Int32(R2), cutlass.Int32(0) + ) + pars[bt] = _ar_parity(arp, f_t, f_cl) + if cutlass.const_expr(SH_ADD): + if p == ng + cutlass.Int32(LAG - 1): + # this CTA's y_sh words, loaded while the last group's FC2 runs (re-polled in finalize) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_ch = cutlass.select_( + f_bb < nb, + f_t * cutlass.Int32(H) + + (blk0 + f_bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + cutlass.Int32(0), + ) + wys[bt] = _ld_relaxed_u32(dnx.subview(f_ch).data_ptr().toint()) + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + cblk = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + unit = slot * cutlass.Int32(U1) + cblk + _waitd( + acc_full.subview(ab), + (k1 >> cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 105, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, base_col + ACC1_COL + ab * cutlass.Int32(N1), cutlass.Float32 + ), + num=N1, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc_empty.subview(ab)) + pb = ab * cutlass.Int32(M_MAX * C1 * R1) + for t in cutlass.range_constexpr(M_MAX): + part = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(4): + part = cutlass.select_( + warp == cutlass.Int32(w), cutlass.Float32(acc[QS * w + t]), part + ) + s_part.store(part, idx=pb + (t * C1 + warp) * R1 + lane) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + for tr in cutlass.range_constexpr((M_MAX + 3) // 4): + t = warp + cutlass.Int32(4 * tr) + if t < cutlass.Int32(M_MAX): + xv = _add_rn( + _add_rn( + _add_rn( + s_part.load(idx=pb + (t * C1 + 0) * R1 + lane), + s_part.load(idx=pb + (t * C1 + 1) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 2) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 3) * R1 + lane), + ) + finish = cutlass.Int32(1) + if (j0 != cutlass.Int32(0)) | (j1 != cutlass.Int32(S1)): + paddr = ( + partials.subview( + (unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane + ) + .data_ptr() + .toint() + ) + if j0 == cutlass.Int32(0): + _st_relaxed_u32(paddr, xv.bitcast(cutlass.Int32)) + finish = cutlass.Int32(0) + else: + pbits = _ld_relaxed_u32(paddr) + spins = cutlass.Int32(0) + t0h = cutlass.Int64(0) + while pbits == cutlass.Int32(PARTIAL_EMPTY): + t0h = _hang_tick(t0h) + pbits = _ld_relaxed_u32(paddr) + spins = spins + cutlass.Int32(1) + _st_relaxed_u32(paddr, cutlass.Int32(PARTIAL_EMPTY)) + xv = _add_rn(pbits.bitcast(cutlass.Float32), xv) + cute.arch.sync_warp() + if cutlass.const_expr( + XVDUMP + ): # debug: FC1 sums into the (unused) partial slots + partials.store( + xv.bitcast(cutlass.Int32), + idx=(unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane, + ) + if finish == cutlass.Int32(1): + gate_raw = cute.arch.shuffle_sync_bfly(xv, 8) + is_up = ( + (lane >> cutlass.Int32(3)) & cutlass.Int32(1) + ) == cutlass.Int32(0) + es = slot * cutlass.Int32(8) + qv, sfb = _swiglu_quant( + xv, + gate_raw, + is_up, + s_esc.load(idx=es), + s_esc.load(idx=es + cutlass.Int32(1)), + s_esc.load(idx=es + cutlass.Int32(2)), + s_esc.load(idx=es + cutlass.Int32(3)), + s_esc.load(idx=es + cutlass.Int32(4)), + ) + byte = _e2m1x2(cute.arch.shuffle_sync_down(qv, 16), qv) + bt = [ + cute.arch.shuffle_sync(byte, i) & cutlass.Int32(0xFF) + for i in range(8) + ] + rw0 = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | tagv + ) + rw1 = ( + bt[3] + | (bt[4] << cutlass.Int32(8)) + | (bt[5] << cutlass.Int32(16)) + | tagv + ) + rw2 = ( + bt[6] + | (bt[7] << cutlass.Int32(8)) + | ((sfb & cutlass.Int32(0xFF)) << cutlass.Int32(16)) + | tagv + ) + if cutlass.const_expr(B2IMG): + if lane == 0: + # B2 row n = j M + t, k-tile c / 16, 16-byte chunk (c / 2) % 8 + # (swizzled), half c % 2; scale: atom (t2, kb = (c % 16) / 4), + # row n at byte 16 n, byte c % 4 + n = (ug // cutlass.Int32(U1)) * cutlass.Int32(M_MAX) + t + t2i = cblk >> cutlass.Int32(4) + cci = (cblk >> cutlass.Int32(1)) & cutlass.Int32(7) + gbase = p * cutlass.Int32(IMG_BYTES) + lo = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | (bt[3] << cutlass.Int32(24)) + ) + hi = ( + bt[4] + | (bt[5] << cutlass.Int32(8)) + | (bt[6] << cutlass.Int32(16)) + | (bt[7] << cutlass.Int32(24)) + ) + img32.store( + (lo, hi), + idx=( + gbase + + t2i * cutlass.Int32(NUM_BYTES_B2) + + n * cutlass.Int32(KT_BYTES) + + (cci ^ (n & cutlass.Int32(7))) + * cutlass.Int32(16) + + (cblk & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + img8 = cutlass.Array( + img32.data_ptr(0), + shape=(NG_CAP * IMG_BYTES,), + dtype=cutlass.Int8, + alignment=16, + ) + img8.store( + cutlass.Int8(sfb & cutlass.Int32(0xFF)), + idx=gbase + + cutlass.Int32(IMG_B2) + + ( + ( + t2i * cutlass.Int32(NUM_KB) + + ( + (cblk & cutlass.Int32(15)) + >> cutlass.Int32(2) + ) + ) + * cutlass.Int32(512) + ) + + n * cutlass.Int32(16) + + (cblk & cutlass.Int32(3)), + ) + else: + if cutlass.const_expr(HB_REPL == 1): + if lane == 0: + hbuf32.store( + (rw0, rw1, rw2, tagv), + idx=( + (slot * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(UNITS_ROW) + + cblk + ) + * cutlass.Int32(4), + alignment=16, + ) + else: + if lane < HB_REPL: # copy `lane` + hbuf32.store( + (rw0, rw1, rw2, tagv), + idx=( + (slot * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(UNITS_ROW) + + cblk + ) + * cutlass.Int32(4) + + lane * cutlass.Int32(HB_STRIDE), + alignment=16, + ) + k1 = k1 + cutlass.Int32(1) + prims.mbarrier_arrive( + fc1_done.subview(p & cutlass.Int32(3)) + ) # warp 5 may poll group p + if tidx == 0: + _stamp(tdbg, bx, 10) + if cutlass.const_expr(TIMING): + if p < cutlass.Int32(8): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(40) + p, + ) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_full.subview(a2), + (k2 >> cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 106, + ) + if cutlass.const_expr(TIMING): + if (tidx == 0) & (g == ng - cutlass.Int32(1)): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(8) + bb, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + yv = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, + base_col + + ACC2_COL + + a2 * cutlass.Int32(N2) + + warp * cutlass.Int32(M_MAX), + cutlass.Float32, + ), + num=LD_POW2, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc2_empty.subview(a2)) + sl = g * cutlass.Int32(4) + warp + if sl < g_n: + a2s = s_esc.load(idx=sl * cutlass.Int32(8) + cutlass.Int32(5)) + for t in cutlass.range_constexpr(M_MAX): + kp = s_kpos.load(idx=sl * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + if kp >= cutlass.Int32(0): + ys.store( + _bf16_round(_mul_rn(cutlass.Float32(yv[t]), a2s)), + idx=( + (bb * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + * cutlass.Int32(TOP_K) + + kp + ) + * cutlass.Int32(R2) + + lane, + ) + k2 = k2 + cutlass.Int32(1) + prims.mbarrier_arrive(b2_free.subview(g & cutlass.Int32(1))) + if tidx == 0: + _stamp(tdbg, bx, 6) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + if tidx == 0: + _stamp(tdbg, bx, 19) + # finalize: (block bb, token t) per warp round: the fma chain over t's top-4 in order + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + pair = warp + cutlass.Int32(bt * 4) + bb = pair // cutlass.Int32(M_MAX) + t = pair - bb * cutlass.Int32(M_MAX) + if bb < nb: + acc_o = cutlass.Float32(0.0) + for k in cutlass.range_constexpr(TOP_K): + acc_o = _fma_rn( + s_w.load(idx=t * cutlass.Int32(TOP_K) + cutlass.Int32(k)), + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ), + acc_o, + ) + if cutlass.const_expr( + YDUMP + ): # debug: y of (token, k) into the partials workspace + partials.store( + cutlass.Float32( + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ) + ).bitcast(cutlass.Int32), + idx=(t * cutlass.Int32(TOP_K) + cutlass.Int32(k)) * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + ) + ch = ( + t * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(SH_ADD): + # + y_sh (this row's shared output, from its shared CTA), as shared_output.add_(routed_output) + wy = wys[bt] + t0h = cutlass.Int64(0) + while ((wy >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) != tag16: + t0h = _hang_tick(t0h) + wy = _ld_relaxed_u32(dnx.subview(ch).data_ptr().toint()) + acc_o = _add_rn( + (wy << cutlass.Int32(16)).bitcast(cutlass.Float32), _bf16_round(acc_o) + ) + if (ar_on == cutlass.Boolean(False)) | cutlass.Boolean(not OUT_SKIP): + out.store(cutlass.BFloat16(acc_o), idx=ch) + if ar_on: + cl = ( + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(AR_PRE): + _ar_push_pre( + ard, + pars[bt], + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + else: + _ar_push( + arp, + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + + # ---- teardown + if tidx == 0: + _stamp(tdbg, bx, 7) + if cutlass.const_expr(not EARLY_EPOCH): + epochs.store(ep + cutlass.Int32(1), idx=bx) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if ar_on | cutlass.Boolean(LATE_TRIGGER): + # the all-reduce polls the pushed slots without a grid wait: release it after every push of this CTA + cute.arch.griddepcontrol_launch_dependents() + if warp == MMA_WARP: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Float32), TMEM_COLS + ) + + +@cute.jit +def m3_moe( + a1_tensor: cute.Tensor, + sfa1_tensor: cute.Tensor, + a2_tensor: cute.Tensor, + sfa2_tensor: cute.Tensor, + logits: cute.Tensor, + bias: cute.Tensor, + x32: cute.Tensor, + xsf32: cute.Tensor, + escal: cute.Tensor, + plan: cute.Tensor, + partials: cute.Tensor, + hbuf32: cute.Tensor, + epochs: cute.Tensor, + out: cute.Tensor, + route_dbg: cute.Tensor, + tdbg: cute.Tensor, + xb32: cute.Tensor, + rw: cute.Tensor, + sfsc: cute.Tensor, + lgbuf: cute.Tensor, + guw: cute.Tensor, + gusf: cute.Tensor, + dnw: cute.Tensor, + dnsf: cute.Tensor, + hx: cute.Tensor, + dnx: cute.Tensor, + arp: cute.Tensor, + s1w: cute.Tensor, + s2w: cute.Tensor, + img32: cute.Tensor, + b2cnt: cute.Tensor, + stream: cuda_driver.CUstream, +): + tma_a1_desc = cuda.create_tensor_map_tiled( + global_address=a1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H // 2, TWO_I, E_LOCAL], + global_strides=[H // 2 // 16, TWO_I * (H // 2) // 16], + box_dims=(KT_BYTES, R1, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_a2_desc = cuda.create_tensor_map_tiled( + global_address=a2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[I_TP // 2, H, E_LOCAL], + global_strides=[I_TP // 2 // 16, H * (I_TP // 2) // 16], + box_dims=(KT_BYTES, R2, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_s1_desc = cuda.create_tensor_map_tiled( + global_address=sfa1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * TWO_I * (H // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_s2_desc = cuda.create_tensor_map_tiled( + global_address=sfa2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * H * (I_TP // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 4 // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_gu_desc = tma_a1_desc + tma_dn_desc = tma_a2_desc + if cutlass.const_expr(SHARED and not SH_PRETILE): + tma_gu_desc = cuda.create_tensor_map_tiled( + global_address=guw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H, 2 * SH_I], + global_strides=[H // 16], + box_dims=(KT_BYTES, 32), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_dn_desc = cuda.create_tensor_map_tiled( + global_address=dnw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[SH_I, H], + global_strides=[SH_I // 16], + box_dims=(KT_BYTES, 128), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + m3_moe_kernel( + tma_a1_desc, + tma_a2_desc, + tma_s1_desc, + tma_s2_desc, + tma_gu_desc, + tma_dn_desc, + logits, + bias, + x32, + xsf32, + escal, + plan, + partials, + hbuf32, + epochs, + out, + route_dbg, + tdbg, + xb32, + rw, + sfsc, + lgbuf, + guw, + gusf, + dnw, + dnsf, + hx, + dnx, + arp, + s1w, + s2w, + img32, + b2cnt, + ).launch( + grid=(NCTA + NSH, 1, 1), + block=(THREADS, 1, 1), + cluster=(1, 1, 1), + stream=stream, + use_pdl=True, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/op.py new file mode 100644 index 000000000000..0cc24dff28c2 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/op.py @@ -0,0 +1,850 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Host side of the MiniMax-M3 routed-expert kernel on one TP4 rank (routed and shared intermediate 768 per rank), +over the TRTLLM-Gen NVFP4 buffers read in place. + +``m3_moe`` (m3_moe_kernel.py, :class:`M3MoeState`): 1..8 decode tokens (decode batches, spec-decode verify). The call +takes the bf16 hidden states and folds in the router GEMV (bit-exact with m3_router_gemv_fp4) and the activation's +NVFP4 quantization. With ``shared=True`` it also runs the shared expert on dedicated CTAs and returns +bf16(shared + routed); otherwise it returns the rank's routed partial, bit for bit what the stock fused MoE returns. + +Usage (build before CUDA-graph capture; every call is capture-safe afterwards):: + + state = M3MoeState( + device, num_tokens=6, shared=True + ) # workspace + compiled kernel, shared by all layers + layer = state.layer( + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=router_weight, + fc31_input_scale=fc31_input_scale, + shared_weights=(gate_up_w, gate_up_sf, down_w, down_sf), + ) + out = layer(hidden) # bf16 [M, 6144] +""" + +from __future__ import annotations + +import importlib.util +import os +import sys +from pathlib import Path +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled + +HIDDEN = 6144 # capacity: fixed: the checkpoint's hidden size +INTER = 768 # capacity: fixed: the routed experts' width per tp4 rank +NUM_EXPERTS = 128 +SHARED_INTER = 768 # capacity: fixed: the shared expert's width per tp4 rank +SHARED_CTAS = 24 # capacity: fixed: an SM split, not a size (m3_moe's shared-expert CTAs (beside num_ctas routed CTAs)) +TOP_K = 4 +U1 = 2 * INTER // 32 # FC1 units per expert +# The router GEMV takes at most 2 experts per CTA (router weights held in registers). +MIN_CTAS = 64 # capacity: fixed: an SM split, not a size + +_UNIT_BYTES = 32 * (HIDDEN // 2 + HIDDEN // 16) # one FC1 unit's weight + scale bytes +_ROW2_BYTES = INTER // 2 + INTER // 16 # one FC2 weight row's bytes (values + scales) +_S1 = 6 # FC1 stages (items) per unit # capacity: fixed: pipeline stages +_ITEM_BYTES = _UNIT_BYTES // _S1 # capacity: fixed: a ring item's bytes +_modules: Dict[tuple, object] = {} +_compiled: Dict[tuple, object] = {} +# m3_moe trace-time options by token count 1..8 (index num_tokens - 1). +# ``sfa_cp``: 18 KB ring stages, the A scales copied by cp.async into the instruction layout (every token count). +SFA_CP_BY_M: Tuple[int, ...] = (1, 1, 1, 1, 1, 1, 1, 1) +# ``b2img``: FC1 -> FC2 hand-off through per-group global B images + release counters. Off at TP4 (at 4 / 8 tokens +# 0.5 / 0.8 us/call slower than the tagged records in the TP4 sweep). +B2IMG_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 0, 0) +# ``ar_pre``: the finalize's all-reduce push with its call parities / descriptor preloaded (all but 2 tokens). +AR_PRE_BY_M: Tuple[int, ...] = (1, 0, 1, 1, 1, 1, 1, 1) +# token counts whose OUT_SKIP engagement line was logged (once per process each) +_OUT_SKIP_LOGGED: set = set() +# Ring ``stages`` (0 = the kernel's default, the most that fit). 5 at 8 tokens (with a 50 ns FC2 record poll below) +# measured 0.8 us/call faster than 6 in the TP4 sweep (same MMAs, bit-identical outputs); at 4 tokens the default is +# fastest with heavy_fc2_only. +STAGES_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 0, 5) +# The FC2 B builders' record poll back-off in ns (0 = the kernel's default, 200); 50 at 8 tokens. +B2_POLL_NS_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 0, 50) +# ``early0``: warp 4 streams group 0 right after its own top-k. Off at 8 tokens (0.6 us/layer faster in the chain: +# there it is on every routing warp's critical path and the ring holds only 5 stages). +EARLY0_BY_M: Tuple[int, ...] = (1, 1, 1, 1, 1, 1, 1, 0) +# The plan's ``heavy_fc2_only``: 192 FC2 output blocks over 188 routed CTAs leave 4 CTAs with two blocks, whose +# trailing FC2 after the last hand-off otherwise ends the kernel 3-7 us after the rest; they stream no FC1 instead +# (1.7 / 0.9 us/call faster at 4 / 8 tokens in the TP4 sweep; bit-identical outputs). +HEAVY_FC2_ONLY_BY_M: Tuple[int, ...] = (0, 0, 0, 1, 0, 0, 0, 1) +# ``hb_repl``: copies of the tagged FC1 -> FC2 records (applied when > 1); CTA bx polls copy bx % hb_repl, which +# spreads the polls of the few hot record lines over L2. 7 and 8 tokens hand off through the B images instead. +HB_REPL_BY_M: Tuple[int, ...] = ( + 1, + 1, + 1, + 1, + 1, + 1, + 1, + 1, +) # off at TP4 (0.2 us/call slower at 4 tokens) +# ``lg_repl``: replicas of the router logit exchange at up to LG_REPL_MAX_TOKENS tokens (fewer stores before the +# exchange; bit-identical outputs). Above that the kernel's default applies. +LG_REPL = 32 +LG_REPL_MAX_TOKENS = 0 # capacity: fixed: measured crossover (off at TP4: flat in the sweep) +# The TP4 FLAT stack: the TP2 FLAT values (no B2 images, 16 router-logit replicas, AR push parities loaded in the +# finalize, PDL release right after the grid wait) over the per-m tables above. Bitwise-identical output (m 1..8 x 4 +# routings, eager + graph); standalone m=4 -0.49 us/call (late_trigger). The values are part of the engine's module +# key. +FLAT_STACK = {"b2img": 0, "lg_repl": 16, "ar_pre": 0, "late_trigger": 0} +# FLAT only up to 4 tokens: at 8 tokens the per-m tables are faster (TP2: 2.96 us/call; TP4 in-model at 2 requests +# +175 us/step with FLAT). Bit-identical either way. +FLAT_MAX_TOKENS = 4 # capacity: fixed: measured crossover + + +# OUT_SKIP work counter (pushing engine calls that skipped the out store, this process) +OUT_SKIP_TAKEN = [0] + + +def flat_applies(num_tokens: int) -> bool: + """Whether an engine state for ``num_tokens`` tokens runs the FLAT stack.""" + return num_tokens <= FLAT_MAX_TOKENS + + +# Byte-balanced FC1 ranges below one unit per CTA (:func:`_fine_ranges`) for groups too small to give every CTA a +# whole unit; False gives such groups whole units on the CTAs with the least FC2 work. +FINE_PLAN = True +# m3_moe phase lag: phase p streams FC1 of group p and FC2 of group p - M3_MOE_LAG (the kernel's default lag). +M3_MOE_LAG = 2 + + +_ENGINE_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "m3_moe_kernel.py") + + +def _kernel_module(config: dict, path: str): + key = (os.path.basename(path),) + tuple(sorted(config.items())) + mod = _modules.get(key) + if mod is None: + name = ( + f"{__name__}_" + + os.path.splitext(os.path.basename(path))[0] + + "_" + + "_".join(f"{k}{v}" for k, v in key[1:]) + ) + spec = importlib.util.spec_from_file_location(name, path) + mod = importlib.util.module_from_spec(spec) + mod.M3_CONFIG = dict(config) + sys.modules[name] = mod + spec.loader.exec_module(mod) + _modules[key] = mod + return mod + + +def _view(t: torch.Tensor, align: int): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align) + + +def is_supported(device: Optional[torch.device] = None) -> tuple: + if not torch.cuda.is_available(): + return False, "no CUDA" + major, _ = torch.cuda.get_device_capability(device) + if major != 10: + return False, "needs the sm_100 family (tcgen05 block-scaled NVFP4 MMA)" + try: + import cutlass # noqa: F401 + except ImportError: + return False, "CuTe DSL (nvidia-cutlass-dsl) is not installed" + return True, "" + + +def ar_push_desc( + mc_ptr: int, flags_ptr: int, rank: int, world: int, max_tokens: int, device +) -> torch.Tensor: + """The ``arp`` descriptor that makes a call also push its bf16 output into a multicast all-reduce exchange (set A + of m3_mc_allreduce's workspace at ``mc_ptr``, slot half = the all-reduce CTA's call parity in ``flags_ptr``). + With it the kernel releases its PDL dependents only after every push; the default (enable 0) keeps the old + behaviour.""" + return torch.tensor( + [1, mc_ptr, flags_ptr, rank, world, max_tokens, 0, 0], dtype=torch.int64, device=device + ) + + +def _shared_views(shared_weights, who): + """(gate_up, its scales, down, its scales) of the shared expert as flat uint8 views, checked.""" + if shared_weights is None: + raise ValueError(f"{who} (shared): shared_weights are required") + gu_w, gu_sf, dn_w, dn_sf = shared_weights + want = {"gate_up": (gu_w, (2 * SHARED_INTER, HIDDEN)), "down": (dn_w, (HIDDEN, SHARED_INTER))} + for name, (t, shape) in want.items(): + if t.dtype != torch.float8_e4m3fn or tuple(t.shape) != shape or not t.is_contiguous(): + raise ValueError(f"{who} (shared): {name} weight must be contiguous e4m3 {shape}") + if ( + gu_sf.numel() != 2 * SHARED_INTER * HIDDEN // 32 + or dn_sf.numel() != HIDDEN * SHARED_INTER // 32 + ): + raise ValueError( + f"{who} (shared): the UE8M0 scales must be the swizzled [N, K / 32] buffers" + ) + return ( + gu_w.view(torch.uint8).view(-1), + gu_sf.view(torch.uint8).reshape(-1), + dn_w.view(torch.uint8).view(-1), + dn_sf.view(torch.uint8).reshape(-1), + ) + + +def _pretile_shared(gu8, gusf8, dn8, dnsf8, n_cta: int = 24) -> torch.Tensor: + """The shared expert's weights as m3_moe's shared CTAs stream them (SH_PRETILE): per CTA s, per stage g, the 16 KB + shared-memory image (128-byte swizzle: 16-byte chunk j of row r at j ^ (r % 8)) of the stage's 128-row MMA tile and + its 512-byte instruction scale atom (row R's 4 codes at word 4 (R % 32) + R / 32). CTA s owns P = SHARED_INTER / 24 + = 16 positions in SPLIT = 128 / (2 P) = 4 K slices of GT = 48 / SPLIT = 12 k-tiles. Stages 0..GT-1: MMA row + R = P O + i (O = 2 q + u: slice q, u 0 gate / 1 up) = weight row u SHARED_INTER + P s + i at k-tile GT q + g; then + down tile j (rows 256 s + 128 j ..) at k-tile kt, j-major (SHARED_INTER / 128 k-tiles each). Built once per + layer.""" + dev = gu8.device + H_, I_ = HIDDEN, SHARED_INTER + P = I_ // n_cta + split = 128 // (2 * P) + gt = H_ // 128 // split + dkt = I_ // 128 + gu = gu8.view(2 * I_, H_) + dn = dn8.view(H_, I_) + gkb, dkb = H_ // 32, I_ // 32 + + def sf_bytes(sf, n, b, kbn): # code of (row n, K block b), broadcast + return sf[ + ((n >> 7) * (kbn // 4) + (b >> 2)) * 512 + (n & 31) * 16 + ((n >> 5) & 3) * 4 + (b & 3) + ] + + r = torch.arange(128, device=dev) + q = r >> 5 + ri = r & 31 + octant = r // P + chunk = torch.arange(8, device=dev) + swz = chunk[None, :] ^ (r[:, None] & 7) # source chunk of image chunk j: j ^ (r % 8) + out = torch.empty( + n_cta, gt + 2 * dkt, 16384 + 512, dtype=torch.uint8, device=dev + ) # capacity: fixed: a workspace of the kernel's fixed geometry + for s in range(n_cta): + # gate_up: MMA row R -> weight row, slice + nrow = (octant & 1) * I_ + P * s + r % P + sl = octant >> 1 + for g in range(gt): + kt = sl * gt + g # per row + tile = gu[ + nrow[:, None], (kt[:, None] * 128 + torch.arange(128, device=dev)[None, :]) + ] # [128, 128] + img = tile.view(128, 8, 16)[r[:, None], swz] # image chunk j = tile chunk j ^ (r % 8) + out[s, g, :16384] = img.reshape(-1) + kb = torch.arange(4, device=dev) + codes = sf_bytes(gusf8, nrow[:, None], kt[:, None] * 4 + kb[None, :], gkb) # [128, 4] + atom = torch.empty(512, dtype=torch.uint8, device=dev) + atom[(ri[:, None] * 16 + q[:, None] * 4 + kb[None, :]).reshape(-1)] = codes.reshape(-1) + out[s, g, 16384:] = atom + for gd in range(2 * dkt): + jt, kt = gd // dkt, gd % dkt + row0 = 256 * s + 128 * jt + tile = dn[row0 : row0 + 128, kt * 128 : (kt + 1) * 128] + img = tile.reshape(128, 8, 16)[r[:, None], swz] + out[s, gt + gd, :16384] = img.reshape(-1) + out[s, gt + gd, 16384:] = dnsf8[ + ((row0 >> 7) * (dkb // 4) + kt) * 512 : ((row0 >> 7) * (dkb // 4) + kt) * 512 + 512 + ] + return out.view(-1) + + +# --------------------------------------------------------------------------------------------------------------------- +# m3_moe (m3_moe_kernel.py): 1..8 tokens per call, group-phased weight stream. +# --------------------------------------------------------------------------------------------------------------------- +# Pre-tiled shared weights per layer, keyed by the source buffers (which the engines read in place): every token-count +# state of a layer streams the same copy. An entry holds its sources, so their addresses are not reused while cached. +_PRETILED: Dict[tuple, tuple] = {} + + +def _pretiled(gu8, gusf8, dn8, dnsf8) -> torch.Tensor: + key = (gu8.device, gu8.data_ptr(), gusf8.data_ptr(), dn8.data_ptr(), dnsf8.data_ptr()) + hit = _PRETILED.get(key) + if hit is None: + hit = _PRETILED[key] = ((gu8, gusf8, dn8, dnsf8), _pretile_shared(gu8, gusf8, dn8, dnsf8)) + return hit[1] + + +def _fine_ranges(weights, total: int): + """Contiguous FC1 item ranges [s_c, e_c) over the CTAs, sized as close to ``weights`` (ideal real-valued item + counts summing to ``total``) as the split-unit protocol allows: every unit (S1 items) has at most one CTA boundary + strictly inside it (its head and tail parts, summed by the tail's holder), so a range that starts inside a unit + runs at least to that unit's end. Returns the item counts.""" + n = len(weights) + items = [0] * n + s = 0 + acc = 0.0 + for c in range(n): + acc += weights[c] + e = total if c == n - 1 else min(total, max(s, int(acc + 0.5))) + if s % _S1 and e < total: # started inside a unit: run to (at least) its end + e = max(e, min(total, (s // _S1 + 1) * _S1)) + items[c] = e - s + s = e + assert s == total + return items + + +def make_phase_plan( + num_ctas: int, + m_max: int, + lag: int = 2, + tail_w: float = 1.0, + early0: bool = False, + fine: bool = FINE_PLAN, + heavy_fc2_only: bool = False, +): + """The work split of ``m3_moe`` for each distinct-expert count G (4 .. 4 m_max): FC2 32-row output blocks in + contiguous per-CTA ranges (the same for every group), and per group p (slots 4 p ..) a range of its unit-major FC1 + stages per CTA, balancing the bytes each CTA streams in phase p (FC1 of group p plus FC2 of group p - lag, and the + trailing FC2s with the last group). Ranges are empty or >= 6 stages (a unit is split over at most two CTAs); a + group too small for every CTA gets whole units on the CTAs with the least FC2 work, or (``fine``) byte-balanced + ranges of any length with at most one CTA boundary inside a unit (:func:`_fine_ranges`). Returns int32 + [4 m_max + 1, num_ctas, m_max + 1] (per group i0 | i1 << 16, then b0 | nb << 16) and the largest block count.""" + gcap = TOP_K * m_max + nb2 = HIDDEN // 32 + plan = torch.zeros(gcap + 1, num_ctas, m_max + 1, dtype=torch.int32) + nbs = [nb2 // num_ctas + (1 if c < nb2 % num_ctas else 0) for c in range(num_ctas)] + b0s = [sum(nbs[:c]) for c in range(num_ctas)] + # (heavy_fc2_only) CTAs with more FC2 blocks than the minimum stream no FC1: they run each group's FC2 as soon as + # its hand-off lands, so their trailing FC2 after the last hand-off matches the others' two lagged groups + min_nb = min(nbs) + no_fc1 = [heavy_fc2_only and nbs[c] > min_nb for c in range(num_ctas)] + BIG_W = 1 << 40 + blk_bytes = 32 * _ROW2_BYTES # one block, one slot + for g in range(TOP_K, gcap + 1): + ng = (g + 3) // 4 + for c in range(num_ctas): + plan[g, c, m_max] = b0s[c] | (nbs[c] << 16) + for p in range(ng): + n_p = min(4, g - 4 * p) + total = n_p * U1 * _S1 + fc2 = [0] * num_ctas + for c in range(num_ctas): + if p >= lag: + fc2[c] += nbs[c] * min(4, g - 4 * (p - lag)) * blk_bytes + if p == ng - 1 and not ( + early0 and p == 0 and m_max >= 2 + ): # early0: group 0's ranges the same for every G + for q in range(max(0, ng - lag), ng): + fc2[c] += int(tail_w * nbs[c] * min(4, g - 4 * q) * blk_bytes) + for c in range(num_ctas): + if no_fc1[c]: + fc2[c] += BIG_W + items = [0] * num_ctas + if total >= _S1 * num_ctas: + + def counts_for(limit): + return [ + 0 if no_fc1[c] else max(_S1, (limit - f) // _ITEM_BYTES) + for c, f in enumerate(fc2) + ] + + lo, hi = 0, max(f for c, f in enumerate(fc2) if not no_fc1[c]) + total * _ITEM_BYTES + while lo < hi: + mid = (lo + hi) // 2 + if sum(counts_for(mid)) >= total: + hi = mid + else: + lo = mid + 1 + items = counts_for(lo) + extra = sum(items) - total + while extra > 0: + c = min( + (c for c in range(num_ctas) if items[c] > _S1), + key=lambda c: (-(fc2[c] + items[c] * _ITEM_BYTES), c), + ) + items[c] -= 1 + extra -= 1 + elif fine: + # water-fill: CTA c streams fc2[c] + w_c item bytes, equal where w_c > 0 + lo_f, hi_f = 0.0, float(max(fc2) + total * _ITEM_BYTES) + for _ in range(100): + mid = 0.5 * (lo_f + hi_f) + if sum(max(0.0, (mid - f) / _ITEM_BYTES) for f in fc2) >= total: + hi_f = mid + else: + lo_f = mid + wts = [max(0.0, (hi_f - f) / _ITEM_BYTES) for f in fc2] + sw = sum(wts) + items = _fine_ranges([w * total / sw for w in wts], total) + else: + chosen = sorted(sorted(range(num_ctas), key=lambda c: (fc2[c], c))[: total // _S1]) + for c in chosen: + items[c] = _S1 + i0 = 0 + for c in range(num_ctas): + if items[c]: + plan[g, c, p] = i0 | ((i0 + items[c]) << 16) + i0 += items[c] + assert i0 == total + return plan, max(nbs) + + +# MOE_TAIL_TRIGGER (default on; TRTLLM_M31_DISABLE_MOE_TAIL_TRIGGER restores the teardown release): on a call that +# pushes into the boundary all-reduce, each routed CTA releases the PDL dependents when its weights producer is done +# (the kernels' PROD_TRIGGER, epoch stored early). Scheduling only. Engine states that logged the taken line, and +# pushing calls that release at the producer tail (counted at launch, so in eager mode only; a graph replays the +# captured call). +_PROD_TRIGGER_LOGGED = set() +TAIL_TRIGGER_TAKEN = [0] + + +def prod_trigger_config(config: dict, who: str, num_tokens: int) -> None: + """Turns the producer-tail release on in an engine config unless MOE_TAIL_TRIGGER is killed.""" + if m31_disabled("MOE_TAIL_TRIGGER"): + return + config["prod_trigger"] = 1 + if (who, num_tokens) not in _PROD_TRIGGER_LOGGED: + _PROD_TRIGGER_LOGGED.add((who, num_tokens)) + logger.info( + f"[M3 dispatch] MOE_TAIL_TRIGGER taken ({who}, tokens={num_tokens}, producer-tail release, " + "epoch stored early)" + ) + + +class M3MoeState: + """Workspace + compiled kernel of ``m3_moe`` (1..8 tokens per call) on one device, shared by the layers that run + on it in one stream order. The call takes the bf16 hidden states and computes the router logits (bit-exact with + m3_router_gemv_fp4) and the NVFP4 activation itself. ``shared``: the shared expert runs on SHARED_CTAS dedicated + CTAs beside the ``num_ctas`` routed ones and the call returns bf16(shared + routed). Build before CUDA-graph + capture; the first call compiles.""" + + fused_router = True + + def __init__( + self, + device: torch.device, + num_tokens: int, + num_ctas: Optional[int] = None, + shared: bool = False, + inter: int = INTER, + ): + """``inter``: the routed intermediate per rank; this engine is built for INTER (768) only.""" + if inter != INTER: + raise ValueError( + f"m3_moe (TP4): routed intermediate {inter} per rank is not supported (only {INTER})" + ) + self.inter = inter + logger.info(f"[M3 MoE] Rubin m3_moe engine (tp4 tables): state for {num_tokens} tokens") + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "M3MoeState allocates and compiles: build it before CUDA-graph capture" + ) + sms_all = torch.cuda.get_device_properties(device).multi_processor_count + if shared and (num_ctas or sms_all) + SHARED_CTAS > sms_all: + raise ValueError( + f"m3_moe (shared): num_ctas + {SHARED_CTAS} shared-expert CTAs must fit the {sms_all} SMs " + f"(all resident at once)" + ) + if not 1 <= num_tokens <= 8: + raise ValueError(f"m3_moe takes 1..8 tokens per call, not {num_tokens}") + self.device = torch.device(device) + ok, why = is_supported(self.device) + if not ok: + raise RuntimeError(f"m3_moe: {why}") + sms = torch.cuda.get_device_properties(self.device).multi_processor_count + self.num_ctas = num_ctas or sms + if not MIN_CTAS <= self.num_ctas <= sms: + raise ValueError( + f"m3_moe runs {MIN_CTAS} .. {sms} CTAs (all resident at once), not {self.num_ctas}" + ) + if self.num_ctas > HIDDEN // 32: + # a routed CTA that owns no 32-row FC2 output block deadlocks the FC2 hand-off on its second call + raise ValueError( + f"m3_moe runs at most {HIDDEN // 32} routed CTAs (one FC2 output block each), not {self.num_ctas}" + ) + self.m = num_tokens + # One token: the trailing FC2s weigh nothing in the last phase's balance. Group 0's FC1 ranges are the same + # for every distinct-expert count (early0). + plan, nb_cap = make_phase_plan( + self.num_ctas, + num_tokens, + M3_MOE_LAG, + 0.0 if num_tokens == 1 else 1.0, + True, + heavy_fc2_only=bool(HEAVY_FC2_ONLY_BY_M[num_tokens - 1]), + ) + self.config = { + "m_max": num_tokens, + "num_ctas": self.num_ctas, + "nb2_cap": nb_cap, + "fused_router": 1, + "timing": 0, + "i_tp": INTER, + } + if HB_REPL_BY_M[num_tokens - 1] > 1: + self.config["hb_repl"] = HB_REPL_BY_M[num_tokens - 1] + if num_tokens <= LG_REPL_MAX_TOKENS: + self.config["lg_repl"] = LG_REPL + if shared: + self.config["sh_i"] = SHARED_INTER + self.config["shared"] = 1 + self.shared = shared + # 18 KB ring stages (A scales by cp.async into the instruction layout) per token count, unless configured + self.config["sfa_cp"] = SFA_CP_BY_M[num_tokens - 1] + # FC1 -> FC2 hand-off through per-group global B images + release counters, per token count + self.config["b2img"] = B2IMG_BY_M[num_tokens - 1] + # the finalize's all-reduce push with its call parities / descriptor preloaded, per token count + self.config["ar_pre"] = AR_PRE_BY_M[num_tokens - 1] + # MOE-FINALIZE-DEADSTORE (OUT_SKIP): a pushing call skips the bf16 ``out`` store the pushed + # all-reduce never reads (exact: the pushed words are unchanged). Default ON; kill switch + # TRTLLM_M31_DISABLE_MOE_OUT_SKIP=1. + if not m31_disabled("MOE_OUT_SKIP"): + self.config["out_skip"] = 1 + if num_tokens not in _OUT_SKIP_LOGGED: + _OUT_SKIP_LOGGED.add(num_tokens) + from tensorrt_llm.logger import logger as _os_log + + _os_log.info( + f"MiniMax-M3.1 MoE engine OUT_SKIP: a pushing call skips the dead bf16 out store " + f"(tokens={num_tokens})" + ) + if STAGES_BY_M[num_tokens - 1] > 0: + self.config["stages"] = STAGES_BY_M[num_tokens - 1] + if B2_POLL_NS_BY_M[num_tokens - 1] > 0: + self.config["b2_poll_ns"] = B2_POLL_NS_BY_M[num_tokens - 1] + if not EARLY0_BY_M[num_tokens - 1]: + self.config["early0"] = 0 + if flat_applies(num_tokens): + self.config.update(FLAT_STACK) + # engagement line (every rank; once per engine state = once per token count) + from tensorrt_llm.logger import logger as _flog + + _flog.info( + "MiniMax-M3 MoE engine FLAT (TP4): " + + " ".join(f"{k}={self.config[k]}" for k in FLAT_STACK) + + f" tokens={num_tokens}" + ) + else: + from tensorrt_llm.logger import logger as _flog + + _flog.info( + f"MiniMax-M3 MoE engine FLAT (TP4) off above {FLAT_MAX_TOKENS} tokens (per-m tables): " + f"tokens={num_tokens}" + ) + prod_trigger_config(self.config, "TP4 MoE engine", num_tokens) + # E0@m4: at 4 tokens the early group-0 FC1 stages are issued by FC2 B builder warp 8 instead of the routing + # warp 4 (issue order only, bitwise; TP4 -0.19 / TP8 -0.21 us/call). + if num_tokens == 4: + from tensorrt_llm.logger import logger as _e0log + + _e0log.info( + "MiniMax-M3 MoE engine E0M4 (%s): early group-0 stages on warp 8, tokens=4" % "TP4" + ) + try: + self.mod = mod = _kernel_module(self.config, _ENGINE_PATH) + except AssertionError as err: + raise ValueError( + f"m3_moe: num_tokens={num_tokens}, num_ctas={self.num_ctas} does not fit the shared " + f"memory / TMEM budget (fewer CTAs means more FC2 blocks per CTA): {err}" + ) from err + # Every CTA waits on others (cross-CTA spins on partials, hand-off records, the shared expert's h and y): the + # grid must be resident at once, one CTA per SM. The module's shared-memory / TMEM budget assertion makes one + # CTA fit an SM; the grid must not exceed the SMs. (The kernel also traps after HANG_NS in any such spin.) + if self.num_ctas + mod.NSH > sms: + raise ValueError( + f"m3_moe: a grid of {self.num_ctas} + {mod.NSH} CTAs exceeds the {sms} SMs; its CTAs wait on each " + "other and must all be resident at once" + ) + kw = dict(device=self.device) + self.plan = plan.to(self.device) + self.partials = torch.full( + (mod.G_CAP * mod.U1 * num_tokens * 32,), -1, dtype=torch.int32, **kw + ) + # the FC1 -> FC2 records; with HB_REPL > 1 copies, HB_STRIDE int32 words apart + hb_words = mod.G_CAP * num_tokens * mod.U1 * 4 + if mod.HB_REPL > 1: + hb_words = mod.HB_REPL * mod.HB_STRIDE + self.hbuf32 = torch.zeros(hb_words, dtype=torch.int32, **kw) + self.epochs = torch.zeros(self.num_ctas + mod.NSH, dtype=torch.int32, **kw) + self.route_dbg = torch.zeros(num_tokens, 8, dtype=torch.int32, **kw) + self.tdbg = torch.zeros(self.num_ctas + mod.NSH, 64, dtype=torch.int64, **kw) + self.lgbuf = torch.zeros(mod.LG_REPL * mod.LG_STRIDE, dtype=torch.int64, **kw) + self._dummy = torch.zeros(4, dtype=torch.int32, **kw) + self._ar_off = torch.zeros(8, dtype=torch.int64, **kw) + # B2IMG: per-group FC2 B images and their unit counters (int32; every group gains CNT_PER_CALL per call) + self.img32 = torch.zeros(max(1, mod.NG_CAP * mod.IMG_BYTES // 4), dtype=torch.int32, **kw) + self.cnt = torch.zeros(mod.NG_CAP * 8 + 8, dtype=torch.int32, **kw) + # shared expert: h and y_sh as (call tag << 16 | bf16), tag 0 = never written + self.hx = torch.zeros(num_tokens * SHARED_INTER, dtype=torch.int32, **kw) + self.dnx = torch.zeros(num_tokens * HIDDEN, dtype=torch.int32, **kw) + self._fn = None + + def layer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=None, + fc31_input_scale=None, + shared_weights=None, + ) -> "M3MoeLayer": + """One MoE layer's operands (the TRTLLM-Gen NVFP4 expert buffers, read in place, never copied). + ``shared_weights`` (a ``shared`` state): the shared expert's (gate_up e4m3 [3072, 6144], its swizzled UE8M0 + scales, down e4m3 [6144, 768], its swizzled UE8M0 scales).""" + layer = M3MoeLayer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight, + fc31_input_scale, + ) + if self.shared: + layer.shared_weights = _shared_views(shared_weights, "m3_moe") + layer.shared_mxfp8 = tuple(shared_weights) + if self.mod.SH_PRETILE: + # (gate_up, pre-tiled stages, down, dummy): the kernel streams the pre-tiled copy + gu8, _, dn8, _ = layer.shared_weights + layer.shared_weights = ( + gu8, + _pretiled(*layer.shared_weights), + dn8, + self._dummy.view(torch.uint8), + ) + return layer + + def run(self, layer, hidden, out=None, arp=None) -> torch.Tensor: + m = self.m + if arp is not None and self.config.get("out_skip"): + # OUT_SKIP work counter: pushing calls issued with the out store skipped (host calls: eager or captured). + OUT_SKIP_TAKEN[0] += 1 + if OUT_SKIP_TAKEN[0] == 1: + from tensorrt_llm.logger import logger as _ost_log + + _ost_log.info( + f"[M3 dispatch] OUT_SKIP taken: a pushing MoE engine call skips its bf16 out store " + f"(tokens={m}; first of the counted calls)" + ) + if arp is not None and self.config.get("prod_trigger"): + TAIL_TRIGGER_TAKEN[0] += 1 + if ( + hidden is None + or hidden.dtype != torch.bfloat16 + or tuple(hidden.shape) != (m, HIDDEN) + or not hidden.is_contiguous() + ): + raise ValueError( + f"m3_moe (fused router): expects contiguous bf16 hidden states [{m}, {HIDDEN}]" + ) + fz = ( + hidden.view(-1).view(torch.int32), + layer.router_weight.view(-1), + layer.fc31_input_scale.view(-1), + self.lgbuf, + ) + lgv, xq, xs = self._dummy.view(torch.float32), self._dummy, self._dummy + if out is None: + out = torch.empty(m, HIDDEN, dtype=torch.bfloat16, device=self.device) + sh = layer.shared_weights if self.shared else (self._dummy.view(torch.uint8),) * 4 + args = ( + layer.w3_w1_weight, + layer.w3_w1_weight_scale.view(-1), + layer.w2_weight, + layer.w2_weight_scale.view(-1), + lgv, + layer.bias, + xq, + xs, + layer.escal, + self.plan.view(-1), + self.partials, + self.hbuf32, + self.epochs, + out.view(-1), + self.route_dbg.view(-1), + self.tdbg.view(-1), + *fz, + *sh, + self.hx, + self.dnx, + self._ar_off if arp is None else arp, + layer.w3_w1_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + layer.w2_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + self.img32, + self.cnt, + ) + if self._fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_moe compiles on its first call: call it once before capture") + import cutlass.cute as cute + + aligns = [ + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + ] + sig = [_view(t, a) for t, a in zip(args, aligns)] + # keyed on the config the kernel module was built with (not self.config), so a module built with any other + # trace-time config can never reuse another build's compiled kernel + key = (_ENGINE_PATH,) + tuple(sorted(self.mod.M3_CONFIG.items())) + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_moe {os.path.basename(key[0])} {key[1:]}", + lambda: cute.compile( + self.mod.m3_moe, + *sig, + cute.runtime.make_fake_stream(), + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + self._fn = fn + self._fn(*args, torch.cuda.current_stream(self.device).cuda_stream) + return out + + +class M3MoeLayer: + """One MoE layer on a :class:`M3MoeState`: ``layer(hidden)``. The expert buffers are read in + place, never copied; the per-expert scalars are packed once (layer constants: rebuild the layer after reloading + weights).""" + + shared_weights = None # the kernel's view of the shared expert (a shared state) + shared_mxfp8 = None # the shared expert's (gate_up, its scales, down, its scales) as given + + def __init__( + self, + state, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + bias, + router_weight=None, + fc31_input_scale=None, + ): + e = NUM_EXPERTS + expected = { + "w3_w1_weight": (w3_w1_weight, (e, 2 * INTER, HIDDEN // 2)), + "w3_w1_weight_scale": (w3_w1_weight_scale, (e, 2 * INTER, HIDDEN // 16)), + "w2_weight": (w2_weight, (e, HIDDEN, INTER // 2)), + "w2_weight_scale": (w2_weight_scale, (e, HIDDEN, INTER // 16)), + } + for name, (t, shape) in expected.items(): + if t.element_size() != 1 or tuple(t.shape) != shape or not t.is_contiguous(): + raise ValueError( + f"m3_moe: {name} is {t.dtype} {tuple(t.shape)}, expected contiguous 1-byte {shape}" + ) + self.state = state + self.w3_w1_weight = w3_w1_weight.view(torch.uint8) + self.w3_w1_weight_scale = w3_w1_weight_scale.view(torch.uint8) + self.w2_weight = w2_weight.view(torch.uint8) + self.w2_weight_scale = w2_weight_scale.view(torch.uint8) + self.escal = torch.stack( + [fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha] + ).float() + self.escal = self.escal.reshape(6, e).contiguous() + self.bias = bias.float().contiguous() + if ( + router_weight is None + or fc31_input_scale is None + or router_weight.dtype != torch.float32 + or tuple(router_weight.shape) != (e, HIDDEN) + or not router_weight.is_contiguous() + or fc31_input_scale.dtype != torch.float32 + or fc31_input_scale.numel() != 1 + ): + raise ValueError( + "m3_moe (fused router): needs router_weight f32 [128, 6144] and fc31_input_scale f32 [1]" + ) + self.router_weight = router_weight + self.fc31_input_scale = fc31_input_scale + + def __call__( + self, + hidden: torch.Tensor, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """``hidden``: contiguous bf16 [M, 6144]. ``arp`` (int64 [8], see :func:`ar_push_desc`): also push the output + into an all-reduce exchange.""" + return self.state.run(self, hidden, out, arp=arp) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/op16.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/op16.py new file mode 100644 index 000000000000..9c1e9a1e221f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_moe/op16.py @@ -0,0 +1,327 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Host side of ``m3_moe16`` (m3_moe16_kernel.py): the MiniMax-M3 MoE layer (fused router + routed experts + shared +expert) for 9..16 decode tokens on one TP4 rank. The 1..8-token engine (op.M3MoeState) is untouched; select this +state only for num_tokens > 8. + +Output: bf16 [M, 6144] = bf16(shared + routed), the routed part bit for bit TRTLLM-Gen's fused MoE on the fp32 router +GEMV logits (m3_router_gemv's math), the shared part as the in-model m3_mx_gemv shared expert. + + state = M3Moe16State(device, num_tokens=12) # before CUDA-graph capture; one per token count + layer = state.layer(w3_w1_weight, w3_w1_weight_scale, w2_weight, w2_weight_scale, fc31_alpha, fc31_scale_c, + act_alpha, act_beta, act_clamp, fc2_alpha, e_score_correction_bias, + router_weight=router_weight, fc31_input_scale=fc31_input_scale, + shared_weights=(gate_up_w, gate_up_sf, down_w, down_sf)) + out = layer(hidden) # first call compiles: call once before capture + +Protocol state (per state object, shared by the layers that run on it in stream order): per-group FC2-B images and +unit counters, the B1 image + counter, per-CTA epochs. Every call adds exactly 192 to each group counter and NCTA to +the B1 counter, so nothing is reset between calls (graph replay safe). +""" + +from __future__ import annotations + +import os +from pathlib import Path +from typing import Optional + +import torch + +from tensorrt_llm._torch import cute_aot + +from ...m31_switches import m31_disabled +from . import op as _op +from .op import ( + HIDDEN, + SHARED_CTAS, + SHARED_INTER, + U1, + M3MoeLayer, + _compiled, + _kernel_module, + _shared_views, + _view, + is_supported, + prod_trigger_config, +) + +# OUT_SKIP work counter (pushing engine calls that skipped the out store, this process) +OUT_SKIP_TAKEN = [0] + + +def _group_slots(g: int, p: int, perm: bool, min_ng: int = 5) -> int: + """Slots of group p when the call has g distinct experts (the kernel's slot order).""" + ng = (g + 3) // 4 + tail = g - 4 * (ng - 1) + if p >= ng: + return 0 + if perm and tail < 4 and ng >= max(3, min_ng): + return tail if p == 1 else 4 + return min(4, g - 4 * p) + + +def make_phase_plan16(num_ctas: int, m_max: int, lag: int = 2, perm: bool = True, min_ng: int = 5): + """op.make_phase_plan (early0, tail_w 1) with the kernel's group sizes (PERM: a partial last group is group 1).""" + S1 = 6 + UNIT_BYTES = 32 * (HIDDEN // 2 + HIDDEN // 16) + ITEM = UNIT_BYTES // S1 + blk_bytes = 32 * (768 // 2 + 768 // 16) + gcap = 4 * m_max + nb2 = HIDDEN // 32 + plan = torch.zeros(gcap + 1, num_ctas, m_max + 1, dtype=torch.int32) + nbs = [nb2 // num_ctas + (1 if c < nb2 % num_ctas else 0) for c in range(num_ctas)] + b0s = [sum(nbs[:c]) for c in range(num_ctas)] + for g in range(4, gcap + 1): + ng = (g + 3) // 4 + for c in range(num_ctas): + plan[g, c, m_max] = b0s[c] | (nbs[c] << 16) + for p in range(ng): + n_p = _group_slots(g, p, perm, min_ng) + total = n_p * U1 * S1 + fc2 = [0] * num_ctas + for c in range(num_ctas): + if p >= lag: + fc2[c] += nbs[c] * _group_slots(g, p - lag, perm, min_ng) * blk_bytes + if p == ng - 1 and not (p == 0 and m_max >= 2): + for q in range(max(0, ng - lag), ng): + fc2[c] += nbs[c] * _group_slots(g, q, perm, min_ng) * blk_bytes + items = [0] * num_ctas + if total >= S1 * num_ctas: + + def counts_for(limit): + return [max(S1, (limit - f) // ITEM) for f in fc2] + + lo, hi = 0, max(fc2) + total * ITEM + while lo < hi: + mid = (lo + hi) // 2 + if sum(counts_for(mid)) >= total: + hi = mid + else: + lo = mid + 1 + items = counts_for(lo) + extra = sum(items) - total + while extra > 0: + c = min( + (c for c in range(num_ctas) if items[c] > S1), + key=lambda c: (-(fc2[c] + items[c] * ITEM), c), + ) + items[c] -= 1 + extra -= 1 + else: + for c in sorted(sorted(range(num_ctas), key=lambda c: (fc2[c], c))[: total // S1]): + items[c] = S1 + i0 = 0 + for c in range(num_ctas): + if items[c]: + plan[g, c, p] = i0 | ((i0 + items[c]) << 16) + i0 += items[c] + assert i0 == total + return plan, max(nbs) + + +_ENGINE16_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "m3_moe16_kernel.py") +M16_MIN_TOKENS, M16_MAX_TOKENS = 9, 16 +# Default routed CTAs (beside the SHARED_CTAS shared-expert CTAs). +M16_NUM_CTAS = 128 +# Phase lag: phase p streams FC1 of group p and FC2 of group p - M16_LAG (the kernel's default lag). +M16_LAG = 3 + + +class M3Moe16State: + """Workspace + compiled kernel of ``m3_moe16`` for ``num_tokens`` (9..16) tokens per call.""" + + fused_router = True + shared = True + + def __init__(self, device: torch.device, num_tokens: int, num_ctas: Optional[int] = None): + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "M3Moe16State allocates and compiles: build it before CUDA-graph capture" + ) + if not M16_MIN_TOKENS <= num_tokens <= M16_MAX_TOKENS: + raise ValueError( + f"m3_moe16 takes {M16_MIN_TOKENS}..{M16_MAX_TOKENS} tokens per call, not {num_tokens}" + ) + self.device = torch.device(device) + ok, why = is_supported(self.device) + if not ok: + raise RuntimeError(f"m3_moe16: {why}") + sms = torch.cuda.get_device_properties(self.device).multi_processor_count + self.num_ctas = num_ctas or M16_NUM_CTAS + if not 128 <= self.num_ctas <= sms - SHARED_CTAS: + raise ValueError( + f"m3_moe16 runs 128 .. {sms - SHARED_CTAS} routed CTAs + {SHARED_CTAS} shared (all resident)" + ) + self.m = num_tokens + plan, nb_cap = make_phase_plan16(self.num_ctas, num_tokens, M16_LAG) + self.config = { + "m_max": num_tokens, + "num_ctas": self.num_ctas, + "nb2_cap": nb_cap, + "fused_router": 1, + "timing": 0, + "shared": 1, + } + # MOE-FINALIZE-DEADSTORE (OUT_SKIP), as op.py: a pushing call skips the dead bf16 out store. Kill switch + # TRTLLM_M31_DISABLE_MOE_OUT_SKIP=1. + if not m31_disabled("MOE_OUT_SKIP"): + self.config["out_skip"] = 1 + prod_trigger_config(self.config, "TP4 MoE16 engine", num_tokens) + try: + self.mod = mod = _kernel_module(self.config, _ENGINE16_PATH) + except AssertionError as err: + raise ValueError( + f"m3_moe16: num_tokens={num_tokens} does not fit the smem / TMEM budget: {err}" + ) from err + kw = dict(device=self.device) + self.plan = plan.to(self.device) + # split units' head sums: every slot empty (0xFFFFFFFF) between calls + self.partials = torch.full((mod.G_CAP * U1 * num_tokens * 32,), -1, dtype=torch.int32, **kw) + # FC2-B images per group + the B1 image; counters: group g at 8 g, B1 at 8 NG_CAP, CTA tickets at 8 NG_CAP + 8 + self.img = torch.zeros(mod.IMG_TOTAL // 4, dtype=torch.int32, **kw) + self.cnt = torch.zeros(mod.NG_CAP * 8 + 16, dtype=torch.int32, **kw) + self.epochs = torch.zeros(self.num_ctas + mod.NSH, dtype=torch.int32, **kw) + self.route_dbg = torch.zeros(num_tokens, 8, dtype=torch.int32, **kw) + self.tdbg = torch.zeros(self.num_ctas + mod.NSH, 64, dtype=torch.int64, **kw) + self.lgbuf = torch.zeros( + mod.LG_REPL * mod.LG_STRIDE, dtype=torch.int64, **kw + ) # logits: tag << 32 | f32 + self._ar_off = torch.zeros(8, dtype=torch.int64, **kw) + # shared expert: h and y_sh as (call tag << 16 | bf16), tag 0 = never written + self.hx = torch.zeros(num_tokens * SHARED_INTER, dtype=torch.int32, **kw) + self.dnx = torch.zeros(num_tokens * HIDDEN, dtype=torch.int32, **kw) + self._fn = None + + def layer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=None, + fc31_input_scale=None, + shared_weights=None, + ) -> M3MoeLayer: + layer = M3MoeLayer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight, + fc31_input_scale, + ) + layer.shared_weights = _shared_views(shared_weights, "m3_moe16") + layer.shared_mxfp8 = tuple(shared_weights) + return layer + + def run(self, layer, hidden, out=None, arp=None) -> torch.Tensor: + m = self.m + if arp is not None and self.config.get("out_skip"): + # OUT_SKIP work counter: pushing calls issued with the out store skipped (host calls: eager or captured). + OUT_SKIP_TAKEN[0] += 1 + if OUT_SKIP_TAKEN[0] == 1: + from tensorrt_llm.logger import logger as _ost_log + + _ost_log.info( + f"[M3 dispatch] OUT_SKIP taken: a pushing MoE engine call skips its bf16 out store " + f"(tokens={m}; first of the counted calls)" + ) + if arp is not None and self.config.get("prod_trigger"): + _op.TAIL_TRIGGER_TAKEN[0] += 1 + if ( + hidden is None + or hidden.dtype != torch.bfloat16 + or tuple(hidden.shape) != (m, HIDDEN) + or not hidden.is_contiguous() + ): + raise ValueError(f"m3_moe16: expects contiguous bf16 hidden states [{m}, {HIDDEN}]") + if out is None: + out = torch.empty(m, HIDDEN, dtype=torch.bfloat16, device=self.device) + args = ( + layer.w3_w1_weight, + layer.w3_w1_weight_scale.view(-1), + layer.w2_weight, + layer.w2_weight_scale.view(-1), + layer.bias, + layer.escal, + self.plan.view(-1), + self.partials, + self.img, + self.cnt, + layer.w3_w1_weight_scale.reshape(-1).view(torch.int32), + layer.w2_weight_scale.reshape(-1).view(torch.int32), + layer.w3_w1_weight.reshape(-1).view(torch.int8), + layer.w2_weight.reshape(-1).view(torch.int8), + self.epochs, + out.view(-1), + self.route_dbg.view(-1), + self.tdbg.view(-1), + hidden.view(-1).view(torch.int32), + layer.router_weight.view(-1), + layer.fc31_input_scale.view(-1), + self.lgbuf, + *layer.shared_weights, + self.hx, + self.dnx, + self._ar_off if arp is None else arp, + ) + if self._fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_moe16 compiles on its first call: call it once before capture" + ) + import cutlass.cute as cute + + sig = [_view(t, 16 if t.element_size() * t.numel() % 16 == 0 else 4) for t in args] + key = (_ENGINE16_PATH,) + tuple(sorted(self.config.items())) + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_moe16 {os.path.basename(key[0])} {key[1:]}", + lambda: cute.compile( + self.mod.m3_moe, + *sig, + cute.runtime.make_fake_stream(), + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + self._fn = fn + self._fn(*args, torch.cuda.current_stream(self.device).cuda_stream) + return out + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_prequant.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_prequant.py new file mode 100644 index 000000000000..48a7827f59b3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_prequant.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Hand-off of an MXFP8-quantized activation from the kernel that produced it to the next MXFP8 GEMM. + +A producer that already holds a BF16 row in registers (the multicast all-reduce + RMSNorm, the split-K attention +merge) also writes its MXFP8 form (E4M3 rows + UE8M0 scales in the 128x4-swizzled layout of +``trtllm::mxfp8_quantize(x, True)``) and publishes it here keyed by the BF16 tensor. The MXFP8 linear whose input +is that tensor takes it and skips its own quantize kernel. Every take clears the slot, so a hand-off is only ever +consumed by the first MXFP8 linear that runs after the producer. + +Module state shared across files (one instance each per process): + +* ``SLOT`` (PrequantSlot): published by ``m3_mc_allreduce`` (``quant=True``), the sparse-decode merge and the fused + decode attention (MXFP8 output); cleared by the sparse decode before it runs; taken by the MXFP8 linear dispatch + and by the one-launch QKV + attention + o_proj path (which also peeks ``pending``). +* ``AR_PUSH`` (ArPushSlot): armed by the fast forward around the attention call; the o_proj GEMV takes the target + and marks ``pushed``; the fast forward checks ``take_pushed`` before the attention boundary all-reduce. +* ``MOE_PUSH`` (MoePushSlot): armed by the fast forward before the MoE; the routed-expert engine takes ``engine`` + and the shared expert's down projection takes ``shared``, each marks its pushed output; the fast forward + checks ``take_pushed`` before the MoE boundary all-reduce. +""" + + +class PrequantSlot: + def __init__(self): + self.pending = None # (data_ptr, rows, width, e4m3 [rows, width], scales) + + def publish(self, x, xq, xsf): + self.pending = (x.data_ptr(), x.shape[0], x.shape[-1], xq, xsf) + + def clear(self): + self.pending = None + + def take(self, x): + """(e4m3 [x.shape[0], width], scales) if x is the published tensor, else None.""" + pending, self.pending = self.pending, None + if ( + pending is None + or x.dim() != 2 + or x.data_ptr() != pending[0] + or x.shape[0] != pending[1] + or x.shape[1] != pending[2] + ): + return None + return pending[3], pending[4] + + +SLOT = PrequantSlot() + + +class ArPushSlot: + """The attention boundary's all-reduce exchange for the o_proj GEMV to push into. + + The fast forward arms ``target`` (m3_mx_gemv ``ar`` tuple) around the attention call; the o_proj marks the output + it pushed in ``pushed``; the boundary then runs the all-reduce with ``prepushed`` = the ``ar`` tuple's mode.""" + + def __init__(self): + self.target = None + self.pushed = None # data_ptr of the pushed output + + def arm(self, target): + self.target = target + self.pushed = None + + def take_target(self): + target, self.target = self.target, None + return target + + def take_pushed(self, x): + pushed, self.pushed = self.pushed, None + return pushed is not None and x.data_ptr() == pushed + + +AR_PUSH = ArPushSlot() + + +class MoePushSlot: + """The MoE boundary's all-reduce exchange for its two producers to push into: the routed-expert engine (set A, + ``engine`` = its ``arp`` descriptor) and the shared expert's down projection (set B, ``shared`` = m3_mx_gemv + ``ar`` tuple). Each marks the output it pushed; the boundary then runs the all-reduce with ``prepushed`` = 3.""" + + def __init__(self): + self.engine = None + self.shared = None + self.engine_pushed = None # data_ptr of the pushed routed output + self.shared_pushed = None # data_ptr of the pushed shared output + + def arm(self, engine, shared): + self.engine, self.shared = engine, shared + self.engine_pushed = self.shared_pushed = None + + def disarm(self): + self.engine = self.shared = None + + def take_engine(self): + engine, self.engine = self.engine, None + return engine + + def take_shared(self): + shared, self.shared = self.shared, None + return shared + + def take_pushed(self, shared, routed): + """(routed pushed, shared pushed) for these outputs; clears the marks.""" + e = routed is not None and self.engine_pushed == routed.data_ptr() + s = shared is not None and self.shared_pushed == shared.data_ptr() + self.engine_pushed = self.shared_pushed = None + return e, s + + +MOE_PUSH = MoePushSlot() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_report.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_report.py new file mode 100644 index 000000000000..7f864bd240c6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_report.py @@ -0,0 +1,34 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The fast-path reports a kernel module makes from inside a call: ``taken`` (an engagement line), ``missed`` (a covered +shape that cannot run: raises under ``modeling_v2='require'``), ``uncovered`` and ``capture_missed``. + +The policy (logging once, raising under require) is the target's: ``modeling.py`` installs its functions here at import, +so a kernel reports without importing the target module. Until then every report is a no-op. +""" + +from __future__ import annotations + +from typing import Callable, Optional + + +def _noop(*args, **kwargs) -> None: + return None + + +taken: Callable[..., None] = _noop +missed: Callable[..., None] = _noop +uncovered: Callable[..., None] = _noop +capture_missed: Callable[..., None] = _noop + + +def install( + *, + on_taken: Callable[[str, Optional[int], str], None], + on_missed: Callable[[str, Optional[int], str], None], + on_uncovered: Callable[[str, str], None], + on_capture_missed: Callable[[str, Optional[int], str], None], +) -> None: + """Route this module's reports to the target's policy.""" + global taken, missed, uncovered, capture_missed + taken, missed, uncovered, capture_missed = on_taken, on_missed, on_uncovered, on_capture_missed diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_sampled_accept.py new file mode 100644 index 000000000000..dbeb5c8850a6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_sampled_accept.py @@ -0,0 +1,174 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused sampled verification for the MiniMax-M3 Eagle3 target head (TP-sharded vocabulary, decode rows <= 48). + +Replaces, per decode step, the full-vocabulary all-gather of the target logits, the fused sampling kernel and the +strict-acceptance ops for rows sampled with temperature and top-p (the AgentX contract). Rows with top-k or min-p +active must stay on the generic path; the caller decides. + +Sampling is top-p rejection sampling, distributed over the TP ranks. With w = exp(logit / T - max) over the FULL +vocabulary and Z = sum(w), a token is in the nucleus iff the mass strictly above its weight is at most top_p * Z +(the fused sampling kernel's kept set, ties included). A candidate is drawn proportionally to w and accepted iff it +is in the nucleus; a rejected candidate lies below the nucleus, so the next candidate is drawn over the tokens heavier +than it. The accepted token is distributed as w restricted to the nucleus. The first candidate lands in the nucleus +with probability >= top_p, so most rows decide on their second exchange. + +Per verify row, 8 CTAs (rows <= 32) or 4 CTAs (more rows) on every rank each own a slice of the rank's logits shard, +staged in shared memory once. A candidate is drawn in two stages without per-token noise: each CTA draws a local +candidate by inverse CDF over its slice, and the CTA whose log-mass plus Gumbel noise is largest supplies the +candidate (CTA j wins with probability T_j / sum T, its token with w / T_j). Uniforms come from a counter-based Philox +keyed by the request's seed and offset, the row, the attempt and the CTA. Each round every CTA multicasts one 4-word +record (key, candidate logit, and max and mass in round 0 or the mass above the current candidate later) and reads +the records of all CTAs of the row on all ranks. Every CTA takes the same decision from the same records in the same +order, so all ranks agree without any intra-GPU handshake. Words carry the call's tag: a slot needs no clearing and +two call halves keep a rank one call ahead from overwriting a slot still being read. +""" + +from typing import Optional + +import torch + +from ..ops.spec.m3_sampled_accept import m3_sampled_accept as _sampled_accept_op +from ..ops.spec.m3_sampled_accept_ws_words import m3_sampled_accept_ws_words as _ws_words_op + +# The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp4/sampledAccept.cu: a row's CTAs +# launch as one thread-block cluster, bounded record polls) and reached through the ops/spec entry m3_sampled_accept. +MAX_ROWS = 48 # capacity: fixed: the kernel's built rows (decode steps) + + +class SampledAcceptWorkspace: + """Exchange slots behind one multicast mapping (two call halves), the (finished rows, call) counters, the per-row + token scratch, an extra-attempt counter and the SA-1R round-0 decided counter. Created collectively, outside + CUDA-graph capture.""" + + def __init__(self, uc, mc_ptr, rank, world, handle=None, comm=None): + self.uc, self.mc_ptr, self.rank, self.world = uc, mc_ptr, rank, world + self.handle, self.comm = handle, comm + dev = uc.device + self.counters = torch.zeros(2, dtype=torch.int32, device=dev) + self.tokens = torch.zeros(MAX_ROWS, dtype=torch.int32, device=dev) + self.extra_attempts = torch.zeros(1, dtype=torch.int32, device=dev) + self.decided = torch.zeros(1, dtype=torch.int32, device=dev) + + @classmethod + def create(cls, mapping, device, one_round_k: int) -> "SampledAcceptWorkspace": + # SA-1R's records carry the round-0 list words: the size depends on the variant (``one_round_k``). + words = int(_ws_words_op(max(mapping.tp_size, 1), one_round_k)) + if mapping.tp_size == 1: + uc = torch.zeros(words, dtype=torch.int64, device=device) + return cls(uc, uc.data_ptr(), 0, 1) + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "SampledAcceptWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 8, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("SampledAcceptWorkspace.create failed on another rank") + return cls(uc, mc.data_ptr(), mapping.tp_rank, mapping.tp_size, handle, comm) + + +_WS: Optional[SampledAcceptWorkspace] = None + + +def workspace(mapping, device, one_round_k: int) -> Optional[SampledAcceptWorkspace]: + """The process-wide workspace for the SA-1R variant ``one_round_k`` (the same on every call of a process).""" + global _WS + if _WS is None: + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = SampledAcceptWorkspace.create(mapping, device, one_round_k) + return _WS + + +def sampled_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + temperatures: torch.Tensor, + top_ks: torch.Tensor, + top_ps: torch.Tensor, + min_ps: Optional[torch.Tensor], + seeds: torch.Tensor, + offsets: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, + ws: Optional[SampledAcceptWorkspace] = None, + use_pdl: bool = True, + *, + one_round_k: int, + prewait_params: bool, + early_trigger: bool, + ilp_walk: bool, +): + """Sampled verification (temperature + top-p) from this rank's bf16 logits shard [nreq * (draft_len + 1), V / tp]: + outputs as shard_accept (accepted [nreq, D+1], num_accepted [nreq], gather_ids [nreq]); None + when unavailable. Per-row sampling parameters and Philox (seed, offset) are the spec metadata's, one per row. + top_ks / min_ps are not applied: rows with top-k or min-p active must be routed to the generic path by the + caller. ``one_round_k`` (SA-1R: 0, 9 or 82), ``prewait_params``, ``early_trigger`` and ``ilp_walk`` select the + kernel variant (the target's constants).""" + del top_ks, min_ps + nreq, draft_len = draft_tokens.shape + rows = nreq * (draft_len + 1) + if logits.dim() != 2 or logits.shape[0] != rows or rows > MAX_ROWS or logits.shape[1] > 51200: + return None + if logits.dtype != torch.bfloat16 or logits.stride(1) != 1: + return None + ws = ws if ws is not None else workspace(mapping, logits.device, one_round_k) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + _sampled_accept_op( + logits, + int(vocab_offset), + temperatures[:rows].float().contiguous(), + top_ps[:rows].float().contiguous(), + seeds[:rows].long().contiguous(), + offsets[:rows].long().contiguous(), + ws.uc, + ws.mc_ptr, + ws.counters, + ws.tokens, + ws.rank, + ws.world, + draft_tokens.int().contiguous(), + batch_idx.int().contiguous(), + accepted, + num_accepted, + gather_ids, + int(force_total), + next_new, + force_vec, + ws.extra_attempts, + bool(use_pdl), + prewait_params=bool(prewait_params), + early_trigger=bool(early_trigger), + one_round_k=int(one_round_k), + decided=ws.decided, + ilp_walk=bool(ilp_walk), + ) + return accepted, num_accepted, gather_ids diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_select_long_dsl.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_select_long_dsl.py new file mode 100644 index 000000000000..106a0ab95862 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_select_long_dsl.py @@ -0,0 +1,382 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 block select for MiniMax-M3 decode rows of up to 9216 blocks, one row per 384-thread CTA (CuTe DSL). + +Bit for bit the selection of this target's ``m3_select_long`` (``torch.ops.trtllm.minimax_m3_sm107_tp4_select_long``, +one 1024-thread CTA per row), from a CTA of 384 threads: + +* Every block's score is forced (the first ``init_blocks`` and the last ``local_blocks`` of the row's valid blocks) and + mapped to an order key held in registers: thread t owns blocks t, t + 384, ... (4, 11 or 24 of them by the row's + valid count). Larger scores get smaller keys; NaN sorts above +inf. The key is the 1024-thread kernel's unsigned + key with its top bit flipped, so signed Int32 order here is that kernel's order. Every thread loads at a clamped + block and masks the result, so a thread's loads issue back to back. +* 384 is a multiple of 16, so the 16 interleaved groups ``block % 16`` are the lanes' ``lane % 16``. The weakest of + the 16 group maxima bounds the 16th-largest score, so the top 16 is among the keys at or above it. +* The candidates are compacted by a CTA prefix count, each as one int64 (key << 32 | block), and ranked exactly by + that value (one signed compare per pair); more than 1024 of them (keys tied at the bound) rank every candidate + against the whole row instead. +* Warp 0 emits the 16 picked block ids sorted ascending, -1 for an empty or -inf pick. + +The kernel waits on its grid dependency first; it releases the dependent sparse decode right after that wait +(``early_trigger``), once every score of its row is loaded (trigger phase 1) or at its exit (phase 0). Kernels are +compiled per (PDL launch, early trigger, trigger phase) with symbolic shapes and strides; ``prebuild`` compiles and +loads them before the executor's warmup, so a decode step never compiles one. +""" + +from pathlib import Path +from typing import Dict, Tuple + +import cutlass +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream, make_fake_tensor +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + griddepcontrol_launch_dependents, + griddepcontrol_wait, +) + +SELECT_THREADS = 384 +SELECT_WARPS = SELECT_THREADS // 32 +# 1M tokens + draft tokens at 128 tokens per block. +MAX_BLOCKS = 9216 # capacity: fixed: the kernel's built blocks per row +MAX_CAND = 1024 # capacity: fixed: the kernel's candidate staging +TOPK = 16 # capacity: fixed: MSA's top-k +NO_KEY = 0x7FFFFFFF # capacity: fixed: a sentinel, not a size (an unused key slot (the reference's 0xffffffff)) +# The key of -inf (the reference's 0xff800000): never emitted. +NEG_INF_KEY = 0x7F800000 # capacity: fixed: a key encoding, not a size +INIT_SCORE = 1.0e30 +LOCAL_SCORE = 1.0e29 +# (keys per thread, rows up to this many valid blocks) +ITEM_CLASSES = ((4, 4 * SELECT_THREADS), (11, 11 * SELECT_THREADS), (24, MAX_BLOCKS)) + +# Shared memory of the phase, int32 words: candidates [MAX_CAND] int64 (key << 32 | block, so one signed compare +# orders them by (key, block)), picked keys / blocks [TOPK] each, per-warp group minima [SELECT_WARPS][16], group +# minima [16], per-warp candidate counts [SELECT_WARPS]. +_CAND = 0 +_PICK_KEY = _CAND + 2 * MAX_CAND +_PICK_BLOCK = _PICK_KEY + TOPK +_WARP_MIN = _PICK_BLOCK + TOPK +_GROUP_MIN = _WARP_MIN + SELECT_WARPS * 16 +_WARP_CNT = _GROUP_MIN + 16 +SMEM_WORDS = _WARP_CNT + SELECT_WARPS +SMEM_BYTES = 4 * SMEM_WORDS + + +def _key(score): + """Order key of a float score: smaller key = larger score, NaN first; signed order = the reference's unsigned.""" + bits = cutlass.Float32(score).bitcast(cutlass.Int32) + return cutlass.select_(bits < cutlass.Int32(0), bits & cutlass.Int32(0x7FFFFFFF), ~bits) + + +def _imin(a, b): + return cutlass.select_(b < a, b, a) + + +def _imax(a, b): + return cutlass.select_(b > a, b, a) + + +def _forced_key(scores, head, block, query, init_blocks, local_start): + raw = scores[head, block, query] + s = cutlass.select_( + block >= local_start, + cutlass.Float32(LOCAL_SCORE), + cutlass.select_(block < init_blocks, cutlass.Float32(INIT_SCORE), cutlass.Float32(raw)), + ) + return _key(s) + + +@cute.jit +def _select_row( + row, + scores: cute.Tensor, # float32 [num_kv_heads, num_blocks, total_q], any strides + n_valid: cute.Tensor, # int32 [>= total_q] + out: cute.Tensor, # int32 [total_q, num_kv_heads, 16] + init_blocks, + local_blocks, + num_kv_heads, + smem: cute.Tensor, # int32 [SMEM_WORDS], shared memory + bar_id: cutlass.Constexpr, + loaded_trigger: cutlass.Constexpr, + raw_prewait, + n_valid_prewait: cutlass.Constexpr, +): + """Select row ``row`` (= query * num_kv_heads + head) with this CTA's SELECT_THREADS threads. Uses named barrier + ``bar_id`` over SELECT_THREADS threads. ``loaded_trigger`` releases the dependent grid once every score of the + row is loaded (trigger phase 1). With ``n_valid_prewait`` the row's valid-block count is ``raw_prewait``, read by + the caller before its grid wait; otherwise it is read here.""" + tid, _, _ = cute.arch.thread_idx() + lane = tid % cutlass.Int32(32) + warp = tid // cutlass.Int32(32) + query = row // num_kv_heads + head = row % num_kv_heads + if cutlass.const_expr(n_valid_prewait): + raw = raw_prewait + else: + raw = n_valid[query] + valid = _imax(_imin(raw, cutlass.Int32(scores.shape[1])), cutlass.Int32(0)) + local_start = _imax(raw - local_blocks, cutlass.Int32(0)) + cand = cute.make_tensor( + cute.recast_ptr(smem.iterator + _CAND, dtype=cutlass.Int64), cute.make_layout(MAX_CAND) + ) + + if tid < TOPK: + smem[_PICK_KEY + tid] = cutlass.Int32(NO_KEY) + smem[_PICK_BLOCK + tid] = cutlass.Int32(0) + + if valid <= TOPK: + # Every valid block is picked (thread t holds block t). + if tid < valid: + smem[_PICK_KEY + tid] = _forced_key(scores, head, tid, query, init_blocks, local_start) + smem[_PICK_BLOCK + tid] = tid + if cutlass.const_expr(loaded_trigger): + griddepcontrol_launch_dependents() + for ci in cutlass.range_constexpr(len(ITEM_CLASSES)): + items = ITEM_CLASSES[ci][0] + lo = TOPK if ci == 0 else ITEM_CLASSES[ci - 1][1] + hi = ITEM_CLASSES[ci][1] + if (valid > lo) & (valid <= hi): + key = cute.make_rmem_tensor(items, cutlass.Int32) + local_min = cutlass.Int32(NO_KEY) + for s in cutlass.range_constexpr(items): + # Every thread loads (a clamped block past the row) so the loads issue back to back, unbranched. + block = tid + cutlass.Int32(s * SELECT_THREADS) + k = _forced_key( + scores, head, _imin(block, valid - 1), query, init_blocks, local_start + ) + key[s] = cutlass.select_(block < valid, k, cutlass.Int32(NO_KEY)) + local_min = _imin(local_min, key[s]) + local_min = _imin(local_min, cute.arch.shuffle_sync_bfly(local_min, 16)) + if lane < 16: + smem[_WARP_MIN + warp * cutlass.Int32(16) + lane] = local_min + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + if cutlass.const_expr(loaded_trigger): + griddepcontrol_launch_dependents() + if tid < 16: + g = cutlass.Int32(NO_KEY) + for w in cutlass.range_constexpr(SELECT_WARPS): + g = _imin(g, smem[_WARP_MIN + w * 16 + tid]) + smem[_GROUP_MIN + tid] = g + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + bound = cutlass.Int32( + -2147483648 + ) # the smallest key: signed keys, unlike the reference's unsigned ones + for g in cutlass.range_constexpr(16): + bound = _imax(bound, smem[_GROUP_MIN + g]) + + # Candidates (key <= bound; unused slots hold NO_KEY > any bound): a CTA prefix count gives the slots. + cnt = cutlass.Int32(0) + for s in cutlass.range_constexpr(items): + cnt += cutlass.select_(key[s] <= bound, cutlass.Int32(1), cutlass.Int32(0)) + incl = cnt + for d in cutlass.range_constexpr(5): + up = cute.arch.shuffle_sync( + incl, _imax(lane - cutlass.Int32(1 << d), cutlass.Int32(0)) + ) + incl += cutlass.select_(lane >= cutlass.Int32(1 << d), up, cutlass.Int32(0)) + if lane == 31: + smem[_WARP_CNT + warp] = incl + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + before = cutlass.Int32(0) + total = cutlass.Int32(0) + for w in cutlass.range_constexpr(SELECT_WARPS): + c = smem[_WARP_CNT + w] + before += cutlass.select_(cutlass.Int32(w) < warp, c, cutlass.Int32(0)) + total += c + slot = before + incl - cnt + for s in cutlass.range_constexpr(items): + if key[s] <= bound: + if slot < MAX_CAND: + cand[slot] = (cutlass.Int64(key[s]) << cutlass.Int64(32)) | cutlass.Int64( + tid + cutlass.Int32(s * SELECT_THREADS) + ) + slot += 1 + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + + if total <= MAX_CAND: + for i in cutlass.range(tid, total, SELECT_THREADS, unroll=1): + mine = cand[i] + rank = cutlass.Int32(0) + for j in cutlass.range(total, unroll=8): + rank += cutlass.select_(cand[j] < mine, cutlass.Int32(1), cutlass.Int32(0)) + if rank < TOPK: + smem[_PICK_KEY + rank] = cutlass.Int32(mine >> cutlass.Int64(32)) + smem[_PICK_BLOCK + rank] = cutlass.Int32(mine & cutlass.Int64(0xFFFFFFFF)) + else: + # More than MAX_CAND keys tied at the bound: rank each candidate against the whole row. + for s in cutlass.range_constexpr(items): + if key[s] <= bound: + block = tid + cutlass.Int32(s * SELECT_THREADS) + rank = cutlass.Int32(0) + for j in cutlass.range(valid, unroll=1): + kj = _forced_key(scores, head, j, query, init_blocks, local_start) + less = (kj < key[s]) | ((kj == key[s]) & (j < block)) + rank += cutlass.select_(less, cutlass.Int32(1), cutlass.Int32(0)) + if rank < TOPK: + smem[_PICK_KEY + rank] = key[s] + smem[_PICK_BLOCK + rank] = block + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + + if warp == 0: + # The 16 picked block ids ascending (NO_KEY = empty or -inf, emitted as -1), bitonic over the warp. + v = cutlass.Int32(NO_KEY) + if lane < TOPK: + pk = smem[_PICK_KEY + lane] + if (pk != cutlass.Int32(NO_KEY)) & (pk != cutlass.Int32(NEG_INF_KEY)): + v = smem[_PICK_BLOCK + lane] + for size_log in cutlass.range_constexpr(1, 6): + for stride_log in cutlass.range_constexpr(size_log - 1, -1, -1): + other = cute.arch.shuffle_sync_bfly(v, 1 << stride_log) + ascending = (lane & cutlass.Int32(1 << size_log)) == cutlass.Int32(0) + lower = (lane & cutlass.Int32(1 << stride_log)) == cutlass.Int32(0) + v = cutlass.select_(lower == ascending, _imin(v, other), _imax(v, other)) + if lane < TOPK: + out[query, head, lane] = cutlass.select_( + v == cutlass.Int32(NO_KEY), cutlass.Int32(-1), v + ) + + +class _SelectLongDslKernel: + """One row per CTA: the grid wait, then ``_select_row``. ``n_valid_prewait`` reads the row's valid-block count + before the grid wait, so that dependent load leaves the post-wait critical path. That is exact because + ``n_valid`` is written only by ``on_update_kv_lens`` and the host copy in ``prepare()``, both before the step's + first layer and behind a plain (non-PDL) launch, so no grid this select can overlap writes it + (``attention/test_modeling_v2_m3_n_valid_contract.py``).""" + + def __init__( + self, use_pdl: bool, early_trigger: bool, trigger_phase: int, n_valid_prewait: bool + ): + self.use_pdl = use_pdl + self.early_trigger = early_trigger + self.loaded_trigger = (not early_trigger) and trigger_phase == 1 + self.n_valid_prewait = n_valid_prewait + + @cute.kernel + def kernel(self, scores, n_valid, out, init_blocks, local_blocks, num_kv_heads): + row, _, _ = cute.arch.block_idx() + smem = SmemAllocator() + words = smem.allocate_tensor(cutlass.Int32, cute.make_layout(SMEM_WORDS), byte_alignment=16) + raw_prewait = cutlass.Int32(0) + if cutlass.const_expr(self.n_valid_prewait): + raw_prewait = n_valid[row // num_kv_heads] + griddepcontrol_wait() + if cutlass.const_expr(self.early_trigger): + griddepcontrol_launch_dependents() + _select_row( + row, + scores, + n_valid, + out, + init_blocks, + local_blocks, + num_kv_heads, + words, + bar_id=1, + loaded_trigger=self.loaded_trigger, + raw_prewait=raw_prewait, + n_valid_prewait=self.n_valid_prewait, + ) + + @cute.jit + def __call__(self, scores, n_valid, out, init_blocks, local_blocks, stream): + num_kv_heads = scores.shape[0] + self.kernel(scores, n_valid, out, init_blocks, local_blocks, num_kv_heads).launch( + grid=(scores.shape[2] * num_kv_heads, 1, 1), + block=(SELECT_THREADS, 1, 1), + stream=stream, + use_pdl=self.use_pdl, + ) + + +# The select reads its row's valid-block count before its grid wait instead of after it, which takes one dependent +# global load off the post-wait path; the selection is unchanged bit for bit. Exact because the count +# (MsaMetadata.msa_n_valid_blocks) is written only before the step's first layer, behind a plain launch +# (attention/test_modeling_v2_m3_n_valid_contract.py). +N_VALID_PREWAIT = True +#: Compiled kernels by (PDL launch, early trigger, trigger phase, valid count read before the wait). +kernel_cache: Dict[Tuple[bool, bool, int, bool], object] = {} + + +def kernel_key( + use_pdl: bool, early_trigger: bool, trigger_phase: int +) -> Tuple[bool, bool, int, bool]: + """The compile key of a launch (the trigger phase does not matter with the early trigger). The kernel reads its + row's valid-block count before its grid wait (``N_VALID_PREWAIT``).""" + return ( + bool(use_pdl), + bool(early_trigger), + 0 if early_trigger else int(trigger_phase), + N_VALID_PREWAIT, + ) + + +def _compile(key: Tuple[bool, bool, int, bool]): + if key not in kernel_cache: + heads, blocks, rows = cute.sym_int(), cute.sym_int(), cute.sym_int() + scores = make_fake_tensor( + cutlass.Float32, + (heads, blocks, rows), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + n_valid = make_fake_compact_tensor(cutlass.Int32, (cute.sym_int(),), stride_order=(0,)) + out = make_fake_tensor( + cutlass.Int32, (rows, heads, TOPK), stride=(cute.sym_int64(), cute.sym_int64(), 1) + ) + kernel_cache[key] = cute_aot.compile_or_load( + f"m3_select_long_dsl {key}", + lambda: cute.compile( + _SelectLongDslKernel(*key), + scores, + n_valid, + out, + cutlass.Int32(0), + cutlass.Int32(0), + make_fake_stream(use_tvm_ffi_env_stream=True), + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + return kernel_cache[key] + + +def select_long_dsl( + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool, + early_trigger: bool, + trigger_phase: int, +) -> None: + """Launch the select on the current stream (compiling its key first if ``prebuild`` did not).""" + _compile(kernel_key(use_pdl, early_trigger, trigger_phase))( + scores, n_valid_blocks, out, init_blocks, local_blocks + ) + + +def prebuild(keys, device) -> list: + """Compile every key and launch each once on a small dummy row (loading its module). Returns the keys.""" + scores = torch.zeros( + 1, 2048, 4, dtype=torch.float32, device=device + ) # capacity: fixed: a dummy prebuild row, not a size + n_valid = torch.full((4,), 2048, dtype=torch.int32, device=device) + out = torch.empty(1, 4, TOPK, dtype=torch.int32, device=device).permute(1, 0, 2) + built = [] + for key in keys: + key = kernel_key(*key) + _compile(key)(scores, n_valid, out, 0, 1) + built.append(key) + return built + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_shard_accept.py new file mode 100644 index 000000000000..2b7d05ad920f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_shard_accept.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 verification for the MiniMax-M3 target head (vocab-sharded TP head, verify rows <= 48). + +Replaces the all-gather of the target logits, the argmax and the strict-acceptance ops of a greedy verify step with +one kernel over this rank's BF16 logits shard (the target lm_head's own local GEMM output): + +* Every logit becomes a 64-bit key ``ordered(value) << 32 | ~global_index`` (ties pick the lowest index, as + ``torch.argmax``), max-reduced per row: warp shuffles, then ``atomicMax``. +* The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping + (Lamport: 0 = empty, the key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the + slots and runs the strict acceptance. Call n uses slot half n % 2. + +The argmax is taken over the BF16 logits without arithmetic, so the tokens are those of the in-tree path's argmax +over the gathered logits. The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/ +shardAccept.cu) and reached through the ``ops/spec`` entry ``m3_shard_accept``. +""" + +from dataclasses import dataclass +from typing import Any, Optional + +import torch + +from ..ops.spec.m3_shard_accept import m3_shard_accept as _shard_accept_op + +#: Verify rows per call, and exchange rows per call half of the workspace. +SHARD_ROWS = 48 # capacity: fixed: DECODE_MAX_TOKENS verify rows + + +@dataclass(eq=False) +class AcceptWorkspace: + """Per-rank exchange slots ([2 halves][SHARD_ROWS rows][world] uint64) behind one multicast mapping, the local + key accumulators and the (finished CTAs, call) counters.""" + + uc: torch.Tensor + mc_ptr: int + keys: torch.Tensor + counters: torch.Tensor + rank: int + world_size: int + handle: Any = None + comm: Any = None + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def create_workspace(mapping, device, slot_rows: int) -> AcceptWorkspace: + """Exchange slots ([2 halves][slot_rows rows][world] uint64) behind one multicast mapping over the TP group, the + local key accumulators [slot_rows] and the (finished CTAs, call) counters. Collective over the TP group; call + outside CUDA-graph capture, in the same order on every rank.""" + words = 2 * slot_rows * max(mapping.tp_size, 1) + if mapping.tp_size > 1: + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + if not _mnnvl_workspace_all_succeeded(comm, True): + comm.Free() + raise RuntimeError("exchange workspace creation failed on another rank") + handle = _make_mnnvl_mcast_buffer( + comm, max(words * 8, 4096), mapping, mapping.is_multi_node() + ) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("exchange workspace creation failed on another rank") + mc_ptr, rank = mc.data_ptr(), mapping.tp_rank + else: + uc = torch.zeros(words, dtype=torch.int64, device=device) + mc_ptr, rank, handle, comm = uc.data_ptr(), 0, None, None + return AcceptWorkspace( + uc=uc, + mc_ptr=mc_ptr, + keys=torch.zeros(slot_rows, dtype=torch.int64, device=uc.device), + counters=torch.zeros(2, dtype=torch.int32, device=uc.device), + rank=rank, + world_size=max(mapping.tp_size, 1), + handle=handle, + comm=comm, + ) + + +def _workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace, created collectively outside CUDA-graph capture on first use.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + # A capture that finds it missing is the caller's miss (it is built on the eager steps). + return None + _WS = create_workspace(mapping, device, SHARD_ROWS) + return _WS + + +def shard_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, +): + """Greedy verification from this rank's BF16 logits shard. + + Args: + logits: [nreq * (draft_len + 1) <= SHARD_ROWS, V / tp] BF16, unit column stride: this rank's columns + (global ids ``vocab_offset ..``) of the verify rows, request-major. + draft_tokens: [nreq, draft_len] int32, the drafts being verified. + batch_idx: [nreq] int32, each request's row in the verify layout. + force_total / force_vec: forced acceptance (see ``m3_shard_accept``). + next_new: optional int32 [nreq, >= 1]; column 0 receives the last accepted token. + + Returns: + (accepted_tokens [nreq, draft_len + 1] int32, num_accepted [nreq] int32, first-pass gather ids [nreq] + int64), or None when the shape is not served or the workspace is missing during a capture. Collective over + the TP group. + """ + nreq, draft_len = draft_tokens.shape + if ( + logits.dim() != 2 + or logits.shape[0] != nreq * (draft_len + 1) + or logits.shape[0] > SHARD_ROWS + or logits.dtype != torch.bfloat16 + or logits.stride(1) != 1 + ): + return None + ws = _workspace(mapping, logits.device) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + _shard_accept_op( + logits, + vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_tokens.contiguous(), + batch_idx, + accepted, + num_accepted, + gather_ids, + force_total, + next_new, + _num_sms(), + force_vec, + ) + return accepted, num_accepted, gather_ids diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_spec_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_spec_tail.py new file mode 100644 index 000000000000..0d4dc9cd1b64 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_spec_tail.py @@ -0,0 +1,116 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Small fused kernels of the MiniMax-M3 Eagle3 step (GB300, decode). + +``chunk_norm`` / ``capture_norm`` apply the drafter's per-layer fc_norm RMSNorms to the captured target hidden states +(the in-tree ``apply_eagle3_fc`` norms, same fp32 arithmetic); ``draft_input`` is a fused draft pass's input (the +embedding lookup, the drafter layer's input_layernorm / hidden_norm and their concat, same arithmetic); +``step_gather`` / ``slot_scatter`` move the step's integer outputs between the spec sampler's store and the engine's +input buffers. +""" + +from typing import Optional, Tuple + +import torch + +# The kernels are compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu) and reached through the +# ops/spec entries. +from ..ops.spec.m3_capture_norm import m3_capture_norm as _capture_norm_op +from ..ops.spec.m3_chunk_norm import m3_chunk_norm as _chunk_norm_op +from ..ops.spec.m3_draft_input import m3_draft_input as _draft_input_op +from ..ops.spec.m3_slot_scatter import m3_slot_scatter as _slot_scatter_op +from ..ops.spec.m3_step_gather import m3_step_gather as _step_gather_op + + +def draft_input( + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + eps_embed: float, + w_hidden: torch.Tensor, + eps_hidden: float, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """(cat([rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden], -1) [M, 2H], hidden[rows] [M, H]) for + int32 ids [M] and int64 rows [M] (rows None: hidden[:M]); ids outside the table embed to zero.""" + m, h = ids.shape[0], table.shape[1] + out = torch.empty((m, 2 * h), dtype=table.dtype, device=table.device) + residual = torch.empty((m, h), dtype=table.dtype, device=table.device) + return _draft_input_op( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, use_pdl + ) + + +def chunk_norm( + src: torch.Tensor, weights: torch.Tensor, eps: float, use_pdl: bool = True +) -> torch.Tensor: + """Per-chunk RMSNorm of src [M, >= C*H] (row stride may exceed C*H) with weights [C, H] -> [M, C*H].""" + c, h = weights.shape + out = torch.empty((src.shape[0], c * h), dtype=src.dtype, device=src.device) + return _chunk_norm_op(src, weights, eps, out, use_pdl) + + +# The overlap scheduler's per-step gathers and the spec sampler's slot stores as one kernel each (step_gather / +# slot_scatter) instead of ~10 and 4 small torch ops; called by the target's SpecStepFold. + + +def step_gather( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 +) -> bool: + """See step_gather_kernel; False when the inputs are outside what the kernel covers (caller falls back).""" + ts = (new_tok, next_draft, lens, slots, ppi, ids, draft, pos_off, kv_off) + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or next_draft.dim() != 2 + or new_tok.dim() < 2 + ): + return False + if (new_tok.dim() == 3 and new_tok.shape[2] != 1) or not ( + lens.is_contiguous() + and ids.is_contiguous() + and draft.is_contiguous() + and pos_off.is_contiguous() + and kv_off.is_contiguous() + ): + return False + _step_gather_op( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 + ) + return True + + +def slot_scatter( + slots, src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft +) -> bool: + """See slot_scatter_kernel (2-D int32 views, zero padding); False to fall back.""" + ts = (src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft) + if any(t.dtype != torch.int32 or not t.is_cuda for t in ts) or slots.dtype != torch.long: + return False + two = (src_new, dst_new, src_next_new, dst_next_new, src_draft, dst_draft) + if any(t.dim() != 2 for t in two) or src_lens.dim() != 1 or dst_lens.dim() != 1: + return False + _slot_scatter_op( + slots, + src_new, + dst_new, + src_next_new, + dst_next_new, + src_lens, + dst_lens, + src_draft, + dst_draft, + ) + return True + + +def capture_norm( + h: torch.Tensor, + residual: Optional[torch.Tensor], + weight: torch.Tensor, + eps: float, + dst: torch.Tensor, +) -> None: + """dst [M, H] (row stride may exceed H) = rmsnorm(h + residual) * weight with chunk_norm's rounding.""" + _capture_norm_op(h, residual, weight, eps, dst) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_target_head.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_target_head.py new file mode 100644 index 000000000000..f5a1b229c777 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_target_head.py @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The target lm_head shard of a verify step of at most 8 rows, streamed from a lossless 12-bit coding of its BF16 +weight (75% of the bytes; ``ops.gemm.m3_target_head`` has the format). + +``prepare_head`` codes this rank's shard once, before the executor's warmup, and keeps it only when it decodes to the +BF16 weight bit for bit (``round_trip_ok``); a shard that does not keeps the in-tree BF16 GEMM. ``logits`` runs the +kernel: BF16 x BF16 MMAs with fp32 accumulation over the full K and one BF16 rounding per logit, the in-tree head's +arithmetic in another summation order. The kernel is compiled into TensorRT-LLM +(cpp/tensorrt_llm/kernels/minimaxM3/targetHead.cu) and called through the catalog entry ``ops.gemm.m3_target_head``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +import torch + +from ..ops.gemm.m3_target_head import CTA_WARPS, MAX_ROWS, encode_chunked, round_trip_ok +from ..ops.gemm.m3_target_head import m3_target_head as _target_head_op + + +@dataclass(eq=False) +class TargetHead: + """One rank's target lm_head shard in the 12-bit coding, and the kernel's split-tile scratch. + + ``coded`` is False when the coding did not reproduce the weight bit for bit (the in-tree GEMM then runs).""" + + weight: torch.Tensor + coded: bool + w: Optional[torch.Tensor] = None + base: int = 0 + exc_off: Optional[torch.Tensor] = None + exc_pos: Optional[torch.Tensor] = None + exc_w: Optional[torch.Tensor] = None + part: Optional[torch.Tensor] = None + counter: Optional[torch.Tensor] = None + grid: int = 0 + + +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def head_supported(lm_head) -> bool: + """A bias-free 2-D BF16 head of whole 16-row tiles whose K the coding covers (a multiple of 64).""" + weight = getattr(lm_head, "weight", None) + if weight is None or weight.dim() != 2 or weight.dtype != torch.bfloat16: + return False + return ( + getattr(lm_head, "bias", None) is None + and weight.shape[0] % 16 == 0 + and weight.shape[1] % 64 == 0 + ) + + +def prepare_head(lm_head, grid: Optional[int] = None) -> TargetHead: + """Code ``lm_head``'s local shard and check it decodes to the BF16 weight bit for bit (fails closed: a shard that + does not is returned with ``coded=False`` and nothing kept). Built once, eagerly, before the executor's warmup + (``M3Eagle3Worker.prebuild_target_head``).""" + weight = lm_head.weight + with torch.no_grad(): + w = weight.detach() + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + if not round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w): + return TargetHead(weight=weight, coded=False) + if grid is None: + grid = _num_sms() + n = weight.shape[0] + return TargetHead( + weight=weight, + coded=True, + w=packed, + base=base, + exc_off=exc_off, + exc_pos=exc_pos, + exc_w=exc_w.view(-1), + part=torch.zeros(grid * CTA_WARPS * 2 * 4 * 32, dtype=torch.float32, device=weight.device), + counter=torch.zeros((n + 15) // 16, dtype=torch.int32, device=weight.device), + grid=grid, + ) + + +def logits(head: TargetHead, x: torch.Tensor, use_pdl: bool = True) -> torch.Tensor: + """This rank's BF16 logits shard [rows, n] of the rows x [rows <= MAX_ROWS, K] (BF16) from the coded shard.""" + assert head.coded and 1 <= x.shape[0] <= MAX_ROWS, (head.coded, x.shape) + if x.stride(1) != 1 or x.stride(0) % 8 or x.data_ptr() % 16: + x = x.contiguous() + n = head.weight.shape[0] + out = torch.empty((x.shape[0], n), dtype=torch.bfloat16, device=x.device) + return _target_head_op( + head.w, + x, + out, + head.part, + head.counter, + head.exc_off, + head.exc_pos, + head.exc_w, + head.base, + n, + head.grid, + use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_tokmap_guard.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_tokmap_guard.py new file mode 100644 index 000000000000..db0077117fe3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/m3_tokmap_guard.py @@ -0,0 +1,185 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The token map of ``thop.trtllm_gen_generation_preprocess`` for one request with several tokens. + +For multi-token generation the op's QKV preprocessing places every row by the workspace token map (``tokensInfo``, +(request, token) per row), but builds that map only when ``isBuildDecoderInfoKernelNeeded()`` holds, which at one +request with the default RoPE it does not. The op then applies whatever map the last multi-request call left in the +workspace: a row can land on another request's length and pages, and its RoPE position and K/V slot are wrong. This +module wraps the op so that, for one request and several tokens, the map the build would write, (0, t) for each row t, +is written into the op's own map slice of the workspace (the offset from the op's layout function) just before the +call. Every other call passes through unchanged. The rows' positions still come from the request's generation length, +exactly as with a freshly built map. + +The write is a device copy from a map prebuilt at install (no allocation, no host sync), so it is captured into the +decode graphs with the call it precedes. +""" + +from __future__ import annotations + +from typing import Callable, Dict, Optional, Tuple + +import torch + +from tensorrt_llm.bindings.internal import thop +from tensorrt_llm.logger import logger + +#: Argument names of ``thop.trtllm_gen_generation_preprocess`` in order (the nanobind binding). +_ARG_NAMES = ( + "qkv_input", + "workspace", + "sequence_lengths", + "spec_decoding_generation_lengths", + "spec_decoding_position_offsets", + "kv_cache_block_offsets", + "host_kv_cache_pool_pointers", + "host_kv_cache_pool_mapping", + "kv_scale_orig_quant", + "kv_scale_quant_orig", + "attention_output_orig_quant", + "rotary_inv_freq", + "rotary_cos_sin", + "mrope_position_deltas", + "layer_idx", + "seq_offset", + "num_heads", + "num_kv_heads", + "head_size", + "tokens_per_block", + "kv_cache_quant_mode", + "max_attention_window_size", + "cyclic_attention_window_size", + "num_tokens", + "batch_beam", + "input_seq_length", + "max_past_kv_length", + "rotary_embedding_dim", + "rotary_embedding_base", + "rotary_embedding_scale_type", + "rotary_embedding_scale", + "rotary_embedding_max_positions", + "position_embedding_type", + "bmm1_scale", + "bmm2_scale", + "fp8_context_fmha", + "predicted_tokens_per_seq", + "attention_chunk_size", + "multi_processor_count", + "total_num_blocks", + "kv_factor", + "need_build_kv_cache_metadata", + "cross_attention", + "skip_fmha_workspace", +) +_INT2_BYTES = 8 # capacity: fixed: an element size + +_original: Optional[Callable] = None +_max_rows = 0 +_maps: Dict[torch.device, torch.Tensor] = {} +_offsets: Dict[Tuple, int] = {} + + +def _no_taken(rows: int) -> None: + pass + + +def _no_missed(rows: int, reason: str) -> None: + pass + + +_on_taken: Callable[[int], None] = _no_taken +_on_missed: Callable[[int, str], None] = _no_missed + + +def _indexed(device: torch.device) -> torch.device: + """``device`` with its index (an unindexed ``cuda`` means the current device), the key ``_maps`` is built under.""" + if device.type == "cuda" and device.index is None: + return torch.device("cuda", torch.cuda.current_device()) + return device + + +def _one_request_map(device: torch.device, rows: int) -> torch.Tensor: + """The prebuilt (0, t) map's first ``rows`` rows on ``device``.""" + device = _indexed(device) + m = _maps.get(device) + if m is None or m.shape[0] < rows: + reason = f"no {rows}-row token map was prebuilt on {device} (install() builds it before the warmup)" + _on_missed(rows, reason) + raise RuntimeError(f"m3_tokmap_guard: {reason}") + return m[:rows] + + +def _map_offset(args: dict, rows: int) -> int: + x = args["qkv_input"] + skip = bool(args.get("skip_fmha_workspace", False)) + key = ( + x.dtype, + rows, + int(args["num_heads"]), + int(args["head_size"]), + int(args["rotary_embedding_dim"]), + int(args["num_kv_heads"]), + skip, + ) + off = _offsets.get(key) + if off is None: + layout = thop.get_trtllm_gen_generation_workspace_layout( + x.dtype, 1, rows, key[2], key[3], key[4], key[5], 0, False, skip + ) + off, size = int(layout["tokens_info_offset"]), int(layout["tokens_info_size"]) + if off < 0 or off % 4 != 0 or size < _INT2_BYTES * rows: + raise RuntimeError( + f"m3_tokmap_guard: the op's generation layout has no {rows}-row token map " + f"(offset {off}, {size} bytes): the layout changed under this guard" + ) + _offsets[key] = off + return off + + +def _guarded(*args, **kwargs): + a = dict(zip(_ARG_NAMES, args)) + a.update(kwargs) + rows = int(a["num_tokens"]) + if ( + a.get("spec_decoding_generation_lengths") is not None + and int(a["predicted_tokens_per_seq"]) > 1 + and int(a["batch_beam"]) == 1 + and rows > 1 + and not bool(a.get("cross_attention", False)) + ): + ws = a["workspace"] + src = _one_request_map(ws.device, rows) + off = _map_offset(a, rows) + # view (never reshape: a copy would leave the op's own workspace unwritten); raises on a strided workspace. + dst = ws.view(-1).view(torch.uint8)[off : off + _INT2_BYTES * rows] + dst.view(torch.int32).view(rows, 2).copy_(src) + _on_taken(rows) + return _original(*args, **kwargs) + + +def install( + device: torch.device, + max_rows: int, + on_taken: Callable[[int], None], + on_missed: Callable[[int, str], None] = _no_missed, +) -> bool: + """Wrap ``thop.trtllm_gen_generation_preprocess`` (once per process) and prebuild the (0, t) map for up to + ``max_rows`` rows on ``device`` (a one-request call past it is ``on_missed`` and raises). Returns whether this + call installed the wrapper.""" + global _original, _max_rows, _on_taken, _on_missed + device = _indexed(device) + if device not in _maps or _maps[device].shape[0] < max_rows: + t = torch.arange(max_rows, dtype=torch.int32, device=device) + _maps[device] = torch.stack([torch.zeros_like(t), t], dim=1).contiguous() + _max_rows = max(_max_rows, max_rows) + _on_taken = on_taken + _on_missed = on_missed + if _original is not None: + return False + _original = thop.trtllm_gen_generation_preprocess + thop.trtllm_gen_generation_preprocess = _guarded + logger.info( + "[M3 prebuild] m3_tokmap_guard installed: one-request multi-token generations get their (0, t) token map " + f"(up to {max_rows} rows)" + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/msa_kv_update.py new file mode 100644 index 000000000000..8c2a67356d3f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/msa_kv_update.py @@ -0,0 +1,89 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One kernel for MSA's device-side KV-length correction (MsaAttentionMetadata.on_update_kv_lens). + +The torch formulation is ~23 one-CTA kernels (min, cumsum, gathers, div / clamp / remainder, the block-table lookup, +the valid-block counts) and runs at the head of every captured decode step, twice. This kernel computes the same +int32 results in one launch: + + kv_true[b] = min(kv_lens[b], staged[b]) -> seq_lens_out[b], cu_kv[b + 1] = prefix sum (cu_kv[0] = 0) + pos[t] = kv_true[r] - qo[r] + intra[t] (r = q_batch_row[t]) + out_loc[t] = table[r, min(max(pos, 0) // page, ncols - 1)] * page + max(pos, 0) % page + n_valid[t] = (max(pos + 1, 0) + page - 1) // page + +``msa_kv_update`` is the target's ``MsaKvUpdate`` (``tensorrt_llm._torch.pyexecutor.runtime_hooks``): the MSA +metadata calls it with its own buffers and falls back to the torch formulation when it returns False. The kernel is +``torch.ops.trtllm.minimax_m3_msa_kv_update``, called through this target's ``msa_kv_update`` entry. +""" + +from typing import Optional + +import torch + +from ..ops.attention.msa_kv_update import msa_kv_update as _kv_update_op + + +def msa_kv_update( + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], +) -> bool: + """The fused correction (see module doc); False, with nothing written, when an operand is outside what the + kernel covers. Arguments in ``MsaKvUpdate`` order; ``seq_lens`` are the per-request query lengths (``qo``).""" + ts = [ + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + msa_seq_lens, + out_cache_loc, + ] + ts += [t for t in (cu_kv_lens, n_valid_blocks) if t is not None] + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or block_table.dim() != 2 + or block_table.stride(1) != 1 + ): + return False + if any(not t.is_contiguous() for t in ts if t is not block_table): + return False + if ( + kv_lens.numel() < batch + or kv_lens_staged.numel() < batch + or seq_lens.numel() < batch + or msa_seq_lens.numel() < batch + or q_batch_row.numel() < total_q + or q_intra.numel() < total_q + or out_cache_loc.numel() < total_q + or (cu_kv_lens is not None and cu_kv_lens.numel() < batch + 1) + or (n_valid_blocks is not None and n_valid_blocks.numel() < total_q) + ): + return False + _kv_update_op( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/__init__.py new file mode 100644 index 000000000000..ef3a4c5847ea --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) as a CuTe DSL kernel (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/m3_mx_gemv_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/m3_mx_gemv_kernel.py new file mode 100644 index 000000000000..932971278c49 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/m3_mx_gemv_kernel.py @@ -0,0 +1,1299 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 decode GEMV on tcgen05: ``y[M, N] = x[M, K] @ W[N, K]^T`` for M <= 8, block-scaled E4M3 x E4M3. + +The weight is the MMA's A operand (128 output features per CTA), the activation is B (8 token columns), +fp32 accumulation in TMEM. Both operands come in their MXFP8 storage: E4M3 values (K contiguous) and UE8M0 +scales in the 128x4-swizzled layout, whose 512-byte atom per (128 rows, 128 K) is exactly the source of a +``tcgen05.cp`` 32x128b.warpx4 scale copy, so both scale tensors are bulk-copied unchanged. For an activation of at +most 32 rows the atom of k-tile ``k`` sits at byte ``512 k``. + +A cluster of SPLIT CTAs shares one 128-row weight tile; rank r streams the interleaved k-tiles r, r + SPLIT, ... +through a RING-stage ring. Before the grid dependency each CTA issues its first RING weight tiles (and their scale +atoms) and prefetches the rest into L2; after it, the activation slice of the CTA (values and scale atoms) lands in +one buffer. Rank r owns rows [r * 128 / SPLIT, (r + 1) * 128 / SPLIT) of the tile: the other ranks store their fp32 +partials of those rows into its shared memory and arrive on its barrier (release, cluster scope); the owner adds the +SPLIT partials in rank order and stores bf16. + +Warps: 0 weight TMA, 1 activation TMA, 2 TMEM allocation + scale copies + MMA, 3 idle, 4-7 epilogue (warp 4 + i +reads TMEM lanes 32 i .. 32 i + 31). +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass import dsl_user_op +from cutlass.experimental import primitives as prims + +CTA_M = 128 # weight rows per CTA = the MMA's M # capacity: fixed: a kernel tile constant +MMA_N = 8 # token columns +CTA_K = 128 # one k-tile: 128 E4M3 bytes per row = one 128-byte swizzle row # capacity: fixed: a kernel tile constant +MMA_INST_K = 32 +K_BLOCKS = ( + CTA_K // MMA_INST_K +) # 4 MMAs per k-tile, one per scale byte of the atom # capacity: fixed: a kernel tile constant +THREADS = 256 +SF_ATOM_BYTES = 512 +SF_ATOM_U16 = SF_ATOM_BYTES // 2 +SF_COLS = 4 # TMEM columns of one scale atom +W_TILE_BYTES = CTA_M * CTA_K +X_TILE_BYTES = MMA_N * CTA_K +EVICT_FIRST = 0x12F0000000000000 +A_COLS_PER_MMA = MMA_INST_K // 4 # TMEM columns of one 128 x K32 E4M3 granule of A + +a_dtype = cutlass.Float8E4M3FN +b_dtype = cutlass.Float8E4M3FN + + +def num_k_tiles(k_in: int) -> int: + return k_in // CTA_K + + +def acc_cols(nacc: int, mma_n: int = MMA_N) -> int: + # nacc independent accumulators of MMA_N columns: MMA j of the CTA accumulates into j % nacc, so the M128 x N8 + # MMAs do not form one serial dependency chain; the epilogue adds them. + return max(16, nacc * mma_n) + + +def a_col0(nacc: int, my_tiles: int, mma_n: int = MMA_N) -> int: + # a_tmem: [acc | SFA per k-tile | one SFB slot (reused in issue order) | A: 32 columns (128 x 128 E4M3) per k-tile] + return (acc_cols(nacc, mma_n) + (my_tiles + 1) * SF_COLS + 31) // 32 * 32 + + +def tmem_cols( + ring: int, nacc: int, a_tmem: bool = False, my_tiles: int = 0, mma_n: int = MMA_N +) -> int: + need = acc_cols(nacc, mma_n) + 2 * ring * SF_COLS + if a_tmem: + need = a_col0(nacc, my_tiles, mma_n) + my_tiles * K_BLOCKS * A_COLS_PER_MMA + cols = 32 + while cols < need: + cols *= 2 + return cols + + +@dsl_user_op +def read_globaltimer(*, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), [], "mov.u64 $0, %globaltimer;", "=l", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _mapa_u32(smem_ptr, peer, *, loc=None, ip=None): + """The shared::cluster address of this CTA's shared-memory location in cluster CTA ``peer``.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [smem_ptr.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(peer).ir_value(loc=loc, ip=ip)], + "mapa.shared::cluster.u32 $0, $1, $2;", "=r,r,r", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_async_v4(dst, w0, w1, w2, w3, mbar, *, loc=None, ip=None): + """st.async of four int32 words to a shared::cluster address, completing ``mbar`` there by 16 bytes.""" + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, + [cutlass.Int32(dst).ir_value(loc=loc, ip=ip), cutlass.Int32(w0).ir_value(loc=loc, ip=ip), + cutlass.Int32(w1).ir_value(loc=loc, ip=ip), cutlass.Int32(w2).ir_value(loc=loc, ip=ip), + cutlass.Int32(w3).ir_value(loc=loc, ip=ip), cutlass.Int32(mbar).ir_value(loc=loc, ip=ip)], + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [$0], {$1, $2, $3, $4}, [$5];", "r,r,r,r,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _ld_shared_v4_f32(smem_ptr, *, loc=None, ip=None): + """ld.shared.v4.f32 of four fp32 at a 16-byte-aligned shared-memory pointer.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + out = _llvm.inline_asm( + _llvm.StructType.get_literal([_T.f32()] * 4), [smem_ptr.toint(loc=loc, ip=ip).ir_value()], + "ld.shared.v4.f32 {$0, $1, $2, $3}, [$4];", "=f,=f,=f,=f,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + return [ + cutlass.Float32(_llvm.extractvalue(_T.f32(), out, [i], loc=loc, ip=ip)) for i in range(4) + ] + + +@dsl_user_op +def _test_wait_cluster(mbar, parity, *, loc=None, ip=None): + """mbarrier.test_wait.parity.acquire.cluster on a barrier of this CTA completed by other CTAs' st.async.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + done = cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [mbar.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(parity).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tmbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [$1], $2;\n\t" + "selp.u32 $0, 1, 0, p;\n\t}", "=r,r,r,~{memory}", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + return done != cutlass.Int32(0) + + +@dsl_user_op +def _absmax(acc, v, *, loc=None, ip=None): + """max(acc, |v|).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(acc).ir_value(loc=loc, ip=ip), cutlass.Float32(v).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .f32 a;\n\tabs.f32 a, $2;\n\tmax.f32 $0, $1, a;\n\t}", "=f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _rcp_approx_ftz(x, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _e4m3x4(v0, v1, v2, v3, *, loc=None, ip=None): + """Four f32 to E4M3 (RN, satfinite), packed little-endian into one int32 word (v0 in the low byte).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (v0, v1, v2, v3)], + "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32(addr, value, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip)], + "st.global.b32 [$0], $1;", "l,r", has_side_effects=True, is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16(x) (RN) as the fp32 bit pattern of the rounded value.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b16 h, z;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _bf16_round(x): + return _bf16_bits(x).bitcast(cutlass.Float32) + + +# The swigluoai activation of MiniMax-M3's dense MLP, gate * sigmoid(1.702 * gate) * (up + 1) with gate clamped to <= 7 +# and up to [-7, 7], in exactly the instruction sequence the Triton silu_and_mul kernel compiles to (modules/swiglu.py +# on SM107): the gate clamp on the packed bf16 pair, the up clamp as min.xorsign.abs, two rounded multiplies into +# ex2.approx, div.full for the reciprocal, two rounded multiplies, round to bf16. No multiply feeds an add, so the +# explicit .rn forms compile to what the Triton kernel's contractable forms do. +SWIGLU_OAI_ALPHA = 1.702 +SWIGLU_OAI_LIMIT = 7.0 + + +@dsl_user_op +def _swiglu_oai_bf16x2(gate_word, up_word, *, loc=None, ip=None): + """The activation of one packed bf16 pair of gate and up values: its two bf16 results as one int32 word (the low + half from the low halves).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [cutlass.Int32(w).ir_value(loc=loc, ip=ip) for w in (gate_word, up_word)], + "{\n\t.reg .b32 g, glim, gl, gh, ul, uh, one, lim, t0, t1, e0, e1, s0, s1, r0, r1;\n\t.reg .b16 hl, hh;\n\t" + "mov.b32 glim, 0x40E040E0;\n\tmin.bf16x2 g, $1, glim;\n\t" + "mov.b32 {hl, hh}, g;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, $2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mov.b32 lim, 0f40E00000;\n\tmin.xorsign.abs.f32 ul, ul, lim;\n\tmin.xorsign.abs.f32 uh, uh, lim;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 ul, ul, one;\n\tadd.rn.f32 uh, uh, one;\n\t" + "mul.rn.f32 t0, gl, 0fBFD9DB23;\n\tmul.rn.f32 t1, gh, 0fBFD9DB23;\n\t" + "mul.rn.f32 t0, t0, 0f3FB8AA3B;\n\tmul.rn.f32 t1, t1, 0f3FB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "add.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 r0, s0, gl;\n\tmul.rn.f32 r1, s1, gh;\n\t" + "mul.rn.f32 r0, ul, r0;\n\tmul.rn.f32 r1, uh, r1;\n\t" + "cvt.rn.bf16x2.f32 $0, r1, r0;\n\t}", "=r,r,r", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _rope_bits(c, a, s, b, *, loc=None, ip=None): + """bf16(c * a + s * b) with separately rounded products (no FMA), as fp32 bits: the stock RoPE arithmetic.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (c, a, s, b)], + "{\n\t.reg .f32 p, q, r;\n\t.reg .b16 h, z;\n\tmul.rn.f32 p, $1, $2;\n\tmul.rn.f32 q, $3, $4;\n\t" + "add.rn.f32 r, p, q;\n\tcvt.rn.bf16.f32 h, r;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _din_load4(table, hidden, hidden_half, word_idx): + """Four int32 words at word_idx of the hidden rows (hidden_half == 1) or the embedding table: one CTA-uniform + base-address select instead of a branch.""" + addr = cutlass.select_( + hidden_half == cutlass.Int32(1), hidden.data_ptr(0).toint(), table.data_ptr(0).toint() + ) + ptr = cutlass.inttoptr( + addr + cutlass.Int64(word_idx) * cutlass.Int64(4), mem_space=1, dtype=cutlass.Int32 + ) + return ptr.load(count=4, alignment=16) + + +def _stamp(timers, timed: bool, bx, slot: int): + # Diagnostic: %globaltimer of milestone `slot` of CTA bx (timers [grid, 8] int64). + if timed: + timers.store(read_globaltimer(), idx=bx * cutlass.Int32(8) + cutlass.Int32(slot)) + + +def _k_tile(rank, i: int, split: int, my_tiles: int, contig: bool): + # contig: rank r streams k-tiles [r * my_tiles, (r + 1) * my_tiles); else the interleaved r, r + split, ... + if contig: + return rank * cutlass.Int32(my_tiles) + cutlass.Int32(i) + return rank + cutlass.Int32(i * split) + + +@cute.kernel +def m3_mx_gemv_kernel( + tma_desc_w: cutlass.GridConstant[cuda.TensorMap], # W [N, K] E4M3 as bytes, box 128 x 128 + tma_desc_wsf: cutlass.GridConstant[ + cuda.TensorMap + ], # W scale atoms [N / 128 * K / 128, 256] u16 + tma_desc_x: cutlass.GridConstant[cuda.TensorMap], # x [M, K] E4M3 as bytes, box 128 x 8 + tma_desc_xsf: cutlass.GridConstant[cuda.TensorMap], # x scale atoms [K / 128, 256] u16 + y: cutlass.Array, # [M * N] bf16, token-major + x_words: cutlass.Array, # x_bf16: the bf16 activation [M, K] as int32 words + timers: cutlass.Array, # [grid, 8] int64 milestones when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, # ar_push: the all-reduce workspace's multicast mapping (int32 words) + ar_flags: cutlass.Int64, # ar_push: its call flags (parity of the next call = flags[0] & 1) + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[ + int + ], # 1: push the output rows into the next all-reduce; 2: and trigger after it + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], # token columns of the MMA (8, 16 or 48): M <= mma_n + din_i: cutlass.Array, # din: int32 token ids [M] + din_table: cutlass.Array, # din: embedding [V, K / 2] bf16 as int32 words + din_hidden: cutlass.Array, # din: hidden rows [M, K / 2] bf16 as int32 words + din_w: cutlass.Array, # din: [embedding norm | hidden norm] weights [K] bf16 as int32 words + din_f: cutlass.Array, # din: fp32 [eps_e, eps_h] + din_vocab: cutlass.Int32, + din_g: cutlass.Array, # din, flag 1: int64 hidden row of each token, as int32 words + din_res: cutlass.Array, # din, flag 2: the gathered hidden rows out [M, K / 2] bf16 as int32 words + din_flags: cutlass.Int32, # din: 1 = gather the hidden rows, 2 = write them to din_res + rope_cs: cutlass.Array, # rope: fp32 (cos, sin) [positions][64][2] + rope_i: cutlass.Array, # rope: int32 cache lengths [M / T] + rope_bt: cutlass.Array, # rope: int32 K / V page tables [M / T][2][max_blocks] + rope_q: cutlass.Array, # rope: E4M3 Q out [M][hq][128] as uint8 + rope_pool: cutlass.Array, # rope: paged E4M3 cache [pages][hkv][tokens per page][128] as uint8 + rope_scale: cutlass.Array, # rope: fp32 [1] E4M3 quantization scale + rope_dims: cutlass.Array, # rope: int32 [max_blocks, tokens-per-page log2, hq, hkv, T] + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], + x_swiglu: cutlass.Constexpr[ + bool + ], # x_bf16: x_words is [M, 2K] gate | up; the activation is swigluoai(gate, up) +): + k_tiles = num_k_tiles(k_in) + x_tile_bytes = mma_n * CTA_K + my_tiles = k_tiles // split + rows_per_rank = CTA_M // split + tx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp_id = cute.arch.warp_idx() + rank = cutlass.Int32(0) + if cutlass.const_expr(split > 1): + rank = cute.arch.block_idx_in_cluster() + tma_ptr_w = tma_desc_w.get_ptr() + tma_ptr_wsf = tma_desc_wsf.get_ptr() + tma_ptr_x = tma_desc_x.get_ptr() + tma_ptr_xsf = tma_desc_xsf.get_ptr() + n_tile = bx // cutlass.Int32(split) + m_offset = n_tile * cutlass.Int32(CTA_M) + + smem_a = cutlass.Array( + cutlass.Uint8, ring * W_TILE_BYTES, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_b = cutlass.Array( + cutlass.Uint8, my_tiles * x_tile_bytes, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_sfa = cutlass.Array( + cutlass.Uint8, ring * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + smem_sfb = cutlass.Array( + cutlass.Uint8, my_tiles * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + w_full = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + mma_done = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + act_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + acc_done = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + reduce_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + # Partials of the rows this rank owns: [source rank][row][token] fp32 (its own slot stays unused). Rows are padded + # to mma_n + 4 words: the owner's 16-byte row reads (lane = row) then hit all 32 banks once per 8 lanes; at a + # stride of mma_n they were 8- to 16-way conflicted (4.3 us of the 48-token QKV tail). + mb_stride = mma_n + 4 + mailbox = cutlass.Array( + cutlass.Float32, + split * rows_per_rank * mb_stride, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + # din: per-token RMS factor of the half of x this CTA's k-tiles lie in. + din_rs = None + if cutlass.const_expr(din): + din_rs = cutlass.Array(cutlass.Float32, 8, space=cutlass.AddressSpace.smem, alignment=16) + + if warp_id == 0: + prims.prefetch_tensormap(tma_ptr_w) + prims.prefetch_tensormap(tma_ptr_wsf) + prims.prefetch_tensormap(tma_ptr_x) + prims.prefetch_tensormap(tma_ptr_xsf) + if prims.elect_sync(): + for s in cutlass.range_constexpr(ring): + prims.mbarrier_init(w_full.subview(s), 1) + prims.mbarrier_init(mma_done.subview(s), 1) + prims.mbarrier_init(act_full, 1) + prims.mbarrier_init(acc_done, 1) + if cutlass.const_expr(split > 1 and st_async): + # Completed by the peers' st.async bytes; this CTA's own arrival carries the expected count. + prims.mbarrier_init(reduce_ready, 1) + prims.mbarrier_arrive_expect_tx( + reduce_ready, (split - 1) * rows_per_rank * mma_n * 4 + ) + elif cutlass.const_expr(split > 1): + prims.mbarrier_init(reduce_ready, (split - 1) * rows_per_rank) + if warp_id == 2: + prims.tcgen05_alloc(tmem_ptr_i32, tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n)) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + if cutlass.const_expr(split > 1): + prims.barrier_cluster_arrive_relaxed() + prims.barrier_cluster_wait() + prims.barrier_cta_sync(0) + if tx == 0: + _stamp(timers, timed, bx, 0) + + if warp_id == 0: + if prims.elect_sync(): + for i in cutlass.range_constexpr(ring, my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [k_global * cutlass.Int32(CTA_K), m_offset], [] + ) + if cutlass.const_expr((n_out // CTA_M) * split >= 80): + # Wide grids launch in part only after the predecessor (e.g. the fused attention) frees its SMs, which + # loses their pre-wait weight prefetch: the CTAs of the first half also stage their partner's + # (bx + grid / 2) weights into L2. + if bx < cutlass.Int32((n_out // CTA_M) * split // 2): + pb = bx + cutlass.Int32((n_out // CTA_M) * split // 2) + p_rank = pb % cutlass.Int32(split) + p_m_offset = (pb // cutlass.Int32(split)) * cutlass.Int32(CTA_M) + for i in cutlass.range_constexpr(my_tiles): + p_k = _k_tile(p_rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [p_k * cutlass.Int32(CTA_K), p_m_offset], [] + ) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + if cutlass.const_expr(i >= ring): + while not cute.arch.mbarrier_try_wait( + mma_done.subview(stage).data_ptr(), (i // ring - 1) % 2 + ): + pass + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.mbarrier_arrive_expect_tx(w_full.subview(stage), W_TILE_BYTES + SF_ATOM_BYTES) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_a.subview(stage * W_TILE_BYTES), + tma_ptr_w, + (k_global * cutlass.Int32(CTA_K), m_offset), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfa.subview(stage * SF_ATOM_BYTES), + tma_ptr_wsf, + (cutlass.Int32(0), n_tile * cutlass.Int32(k_tiles) + k_global), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + elif warp_id == 1: + prims.griddepcontrol(prims.GridDepAction.WAIT) + if cutlass.const_expr(trigger_early): + # Dependents launch once this grid's own dependency is satisfied (not at entry): consumers that read + # step metadata or caches before their own grid wait rely on every grid ahead of them having passed its + # wait, so the chain of releases never runs ahead of a producer still in flight. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if tx == 32: + _stamp(timers, timed, bx, 1) + if cutlass.const_expr(not x_bf16): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(act_full, my_tiles * (x_tile_bytes + SF_ATOM_BYTES)) + for i in cutlass.range_constexpr(my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_b.subview(i * x_tile_bytes), + tma_ptr_x, + (k_global * cutlass.Int32(CTA_K), cutlass.Int32(0)), + act_full, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfb.subview(i * SF_ATOM_BYTES), + tma_ptr_xsf, + (cutlass.Int32(0), k_global), + act_full, + ) + elif warp_id == 2: + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + ring * SF_COLS + idesc = prims.Tcgen05MxInstrDesc.build( + a_dtype=a_dtype, b_dtype=b_dtype, scale_format=1, n_dim=mma_n, m_dim=CTA_M + ) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + if cutlass.const_expr(a_tmem): + # A staged in TMEM: before the grid wait every weight k-tile and its scale atom go on into TMEM as they + # land (the copies of a stage refilled later are committed to its mma_done); once the activation lands, + # its scale atoms and the MMAs, A read from TMEM. + base_row = tmem_raw >> 16 + base_col = tmem_raw & 0xFFFF + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + my_tiles * SF_COLS + a_base = tmem_raw + cutlass.Int32(a_col0(nacc, my_tiles, mma_n)) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + if prims.elect_sync(): + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_cp( + prims.Tcgen05CpShape.SHAPE_128X256B, + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), + 6, + cutlass.Int32, + ), + desc_a + ((MMA_INST_K >> 4) * kb), + group=prims.CTAGroup.CTA_1, + ) + prims.tcgen05_cp( + s2t_shape, + cutlass.inttoptr((base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32), + desc_sfa, multicast=s2t_multicast, + ) # fmt: skip + if cutlass.const_expr(i + ring < my_tiles): + prims.tcgen05_commit(mma_done.subview(stage)) + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + for i in cutlass.range_constexpr(my_tiles): + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + sfa_tmem = cutlass.inttoptr( + (base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32 + ) + sfb_tmem = cutlass.inttoptr((base_row << 16) | sfb_col0, 6, cutlass.Int32) + prims.tcgen05_cp(s2t_shape, sfb_tmem, desc_sfb, multicast=s2t_multicast) + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), 6, cutlass.Int32 + ), + desc_b + ((MMA_INST_K >> 4) * kb), idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + i > 0 or kb > 0, + sfa_tmem, sfb_tmem, + ) # fmt: skip + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + else: + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + sfa_tmem = (base_row << 16) | (sfa_col0 + stage * SF_COLS) + sfb_tmem = (base_row << 16) | (sfb_col0 + stage * SF_COLS) + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + if prims.elect_sync(): + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), desc_sfa, + multicast=s2t_multicast) # fmt: skip + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), desc_sfb, + multicast=s2t_multicast) # fmt: skip + for kb in cutlass.range_constexpr(K_BLOCKS): + inc = ((MMA_INST_K * 8 // 8) >> 4) * kb + j = i * K_BLOCKS + kb + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw + cutlass.Int32((j % nacc) * mma_n), 6, cutlass.Float32), + desc_a + inc, desc_b + inc, idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), j >= nacc, + cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), + ) # fmt: skip + prims.tcgen05_commit(mma_done.subview(stage)) + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + elif warp_id >= 4: + lane = tx % 32 + row = (warp_id % 4) * 32 + lane + if cutlass.const_expr(x_bf16): + # The bf16 activation of this CTA's k-tiles to MXFP8 in shared memory, as + # torch.ops.trtllm.mxfp8_quantize(x, True) does (UE8M0 = round-up(qz_amax * rcp(448)), + # E4M3 RN-satfinite): one 32-element block (token qz_t, k-tile qz_i, block qz_kb) per thread and round, + # the E4M3 bytes into the 128-byte-swizzled B tile, the qz_scale byte into the tile's qz_scale atom. + qz_tid = tx - cutlass.Int32(128) + prims.griddepcontrol(prims.GridDepAction.WAIT) + din_half_k = k_in // 2 + din_hidden_half = cutlass.Int32(0) + if cutlass.const_expr(din): + # Draft input x = [rmsnorm(table[ids]) * w_e, rmsnorm(hidden) * w_h] (an out-of-range id embeds to + # zero): this CTA's k-tiles lie in one half, so it needs that half's statistic of every token. 16 + # threads per token, 16-byte loads, all of a thread's loads in flight at once. + din_hidden_half = cutlass.select_( + rank * cutlass.Int32(my_tiles * CTA_K) >= cutlass.Int32(din_half_k), + cutlass.Int32(1), + cutlass.Int32(0), + ) + din_tok = qz_tid // cutlass.Int32(16) + din_part = qz_tid % cutlass.Int32(16) + din_tok_ld = cutlass.select_(din_tok < num_tokens, din_tok, cutlass.Int32(0)) + din_id = din_i.load(idx=din_tok_ld) + din_valid = (din_id >= cutlass.Int32(0)) & (din_id < din_vocab) + din_hrow = din_tok_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + din_hrow = din_g.load(idx=din_tok_ld * cutlass.Int32(2)) + din_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + din_hrow, + cutlass.select_(din_valid, din_id, cutlass.Int32(0)), + ) + # The first hidden-half CTA of tile 0 reads whole hidden rows here: it also writes them out. + din_copy = ( + ((din_flags & cutlass.Int32(2)) != cutlass.Int32(0)) + & (bx == cutlass.Int32(split // 2)) + & (din_hidden_half == cutlass.Int32(1)) + & (din_tok < num_tokens) + ) + din_base = din_row * cutlass.Int32(din_half_k // 2) + din_ss = cutlass.Float32(0.0) + din_vecs_per_part = din_half_k // 8 // 16 + for din_round in cutlass.range_constexpr(0, din_vecs_per_part, 16): + din_n = min(16, din_vecs_per_part - din_round) + din_loaded = [] + for din_j in cutlass.range_constexpr(din_n): + din_widx = din_base + ( + din_part + cutlass.Int32((din_round + din_j) * 16) + ) * cutlass.Int32(4) + din_loaded.append( + _din_load4(din_table, din_hidden, din_hidden_half, din_widx) + ) + if din_copy: + for din_j in cutlass.range_constexpr(din_n): + din_v = din_loaded[din_j] + din_res.store( + (din_v[0], din_v[1], din_v[2], din_v[3]), + idx=din_tok * cutlass.Int32(din_half_k // 2) + + (din_part + cutlass.Int32((din_round + din_j) * 16)) + * cutlass.Int32(4), + alignment=16, + ) + for din_j in cutlass.range_constexpr(din_n): + for din_e in cutlass.range_constexpr(4): + din_word = cutlass.Int32(din_loaded[din_j][din_e]) + din_lo = (din_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + din_hi = (din_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + din_ss = din_ss + din_lo * din_lo + din_hi * din_hi + for din_o in cutlass.range_constexpr(4): + din_ss = din_ss + cute.arch.shuffle_sync_bfly(din_ss, offset=1 << din_o) + if din_part == cutlass.Int32(0): + din_eps = din_f.load(idx=din_hidden_half) + din_r = cute.math.rsqrt( + din_ss / cutlass.Float32(din_half_k) + din_eps, fastmath=True + ) + din_keep = din_valid | (din_hidden_half == cutlass.Int32(1)) + din_rs.store( + cutlass.select_(din_keep, din_r, cutlass.Float32(0.0)), idx=din_tok + ) + prims.barrier_cta_sync(3, thread_count=128) + qz_b_words = cutlass.Array(smem_b.data_ptr(0), shape=(my_tiles * x_tile_bytes // 4,), dtype=cutlass.Int32, + alignment=16) # fmt: skip + qz_items = my_tiles * K_BLOCKS * mma_n + qz_rounds = (qz_items + 127) // 128 + # A group's loads first (clamped to a valid row), then its math: one L2 round trip per group of + # QZ_GROUP rounds instead of one per round, without holding every round's data in registers. + QZ_GROUP = qz_rounds if qz_rounds <= 8 else 4 + qz_loaded = [] + for qz_r in cutlass.range_constexpr(qz_rounds): + qz_item = qz_tid + cutlass.Int32(qz_r * 128) + qz_t = qz_item % cutlass.Int32(mma_n) + qz_kb = (qz_item // cutlass.Int32(mma_n)) % cutlass.Int32(K_BLOCKS) + qz_i = qz_item // cutlass.Int32(mma_n * K_BLOCKS) + if cutlass.const_expr(qz_r * 128 + 127 >= qz_items): + qz_i = cutlass.select_( + qz_item < cutlass.Int32(qz_items), qz_i, cutlass.Int32(0) + ) + qz_k_global = rank + qz_i * cutlass.Int32(split) + if cutlass.const_expr(contig): + qz_k_global = rank * cutlass.Int32(my_tiles) + qz_i + qz_t_ld = cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + qz_x_base = ( + qz_t_ld * cutlass.Int32(k_in if x_swiglu else k_in // 2) + + qz_k_global * cutlass.Int32(CTA_K // 2) + + qz_kb * cutlass.Int32(MMA_INST_K // 2) + ) + qz_wvecs = None + if cutlass.const_expr(din): + # The draft input's raw values (table row of the id / hidden row) and its norm weights. + qz_col = qz_k_global * cutlass.Int32(CTA_K) + qz_kb * cutlass.Int32(MMA_INST_K) + qz_hcol = qz_col - din_hidden_half * cutlass.Int32(din_half_k) + qz_id = din_i.load(idx=qz_t_ld) + qz_hrow = qz_t_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + qz_hrow = din_g.load(idx=qz_t_ld * cutlass.Int32(2)) + qz_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + qz_hrow, + cutlass.select_( + (qz_id >= cutlass.Int32(0)) & (qz_id < din_vocab), + qz_id, + cutlass.Int32(0), + ), + ) + qz_src_w = qz_row * cutlass.Int32(din_half_k // 2) + qz_hcol // cutlass.Int32(2) + qz_vecs = [ + _din_load4( + din_table, + din_hidden, + din_hidden_half, + qz_src_w + cutlass.Int32(qz_v * 4), + ) + for qz_v in range(4) + ] + qz_wvecs = [ + din_w.load( + idx=qz_col // cutlass.Int32(2) + cutlass.Int32(qz_v * 4), + vector_size=4, + alignment=16, + ) + for qz_v in range(4) + ] + else: + qz_vecs = [ + x_words.load( + idx=qz_x_base + cutlass.Int32(qz_v * 4), vector_size=4, alignment=16 + ) + for qz_v in range(4) + ] + if cutlass.const_expr(x_swiglu): + # The up half of the same columns, K values after the gate in the row. + qz_wvecs = [ + x_words.load( + idx=qz_x_base + cutlass.Int32(k_in // 2 + qz_v * 4), + vector_size=4, + alignment=16, + ) + for qz_v in range(4) + ] + qz_loaded.append((qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs)) + if cutlass.const_expr((qz_r + 1) % QZ_GROUP == 0 or qz_r + 1 == qz_rounds): + for qz_g in cutlass.range_constexpr(len(qz_loaded)): + qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs = qz_loaded[qz_g] + qz_vals = [] + qz_amax = cutlass.Float32(0.0) + if cutlass.const_expr(din): + qz_rs = din_rs.load( + idx=cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + ) + for qz_v in cutlass.range_constexpr(4): + for qz_e4 in cutlass.range_constexpr(4): + qz_word = cutlass.Int32(qz_vecs[qz_v][qz_e4]) + if cutlass.const_expr(x_swiglu): + qz_word = _swiglu_oai_bf16x2( + qz_word, cutlass.Int32(qz_wvecs[qz_v][qz_e4]) + ) + qz_lo = (qz_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + qz_hi = (qz_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + if cutlass.const_expr(din): + # bf16(x * rs * w), the draft-input kernel's rounding. + qz_wword = cutlass.Int32(qz_wvecs[qz_v][qz_e4]) + qz_lo = _bf16_round( + qz_lo + * qz_rs + * (qz_wword << cutlass.Int32(16)).bitcast(cutlass.Float32) + ) + qz_hi = _bf16_round( + qz_hi + * qz_rs + * (qz_wword & cutlass.Int32(-65536)).bitcast( + cutlass.Float32 + ) + ) + qz_vals.append(qz_lo) + qz_vals.append(qz_hi) + qz_amax = _absmax(_absmax(qz_amax, qz_lo), qz_hi) + qz_bits = (qz_amax * _rcp_approx_ftz(cutlass.Float32(448.0))).bitcast( + cutlass.Int32 + ) + qz_e = ( + (qz_bits >> cutlass.Int32(23)) & cutlass.Int32(0xFF) + ) + cutlass.select_( + (qz_bits & cutlass.Int32(0x7FFFFF)) != cutlass.Int32(0), + cutlass.Int32(1), + cutlass.Int32(0), + ) + qz_e = cutlass.select_(qz_e > cutlass.Int32(254), cutlass.Int32(254), qz_e) + qz_scale = ((cutlass.Int32(254) - qz_e) << cutlass.Int32(23)).bitcast( + cutlass.Float32 + ) + qz_scale = cutlass.select_( + qz_amax != cutlass.Float32(0.0), qz_scale, cutlass.Float32(0.0) + ) + qz_q = [_e4m3x4(qz_vals[4 * qz_j] * qz_scale, qz_vals[4 * qz_j + 1] * qz_scale, + qz_vals[4 * qz_j + 2] * qz_scale, qz_vals[4 * qz_j + 3] * qz_scale) + for qz_j in range(8)] # fmt: skip + if qz_item < cutlass.Int32(qz_items): + if qz_t < num_tokens: + qz_row_w = qz_i * cutlass.Int32( + x_tile_bytes // 4 + ) + qz_t * cutlass.Int32(CTA_K // 4) + qz_c0 = qz_kb * cutlass.Int32(2) + qz_b_words.store( + (qz_q[0], qz_q[1], qz_q[2], qz_q[3]), + idx=qz_row_w + + (qz_c0 ^ (qz_t & cutlass.Int32(7))) * cutlass.Int32(4), + alignment=16, + ) + qz_b_words.store( + (qz_q[4], qz_q[5], qz_q[6], qz_q[7]), + idx=qz_row_w + + ((qz_c0 + cutlass.Int32(1)) ^ (qz_t & cutlass.Int32(7))) + * cutlass.Int32(4), + alignment=16, + ) + smem_sfb.store( + cutlass.Uint8(qz_e), + idx=qz_i * cutlass.Int32(SF_ATOM_BYTES) + + (qz_t % cutlass.Int32(32)) * cutlass.Int32(16) + + (qz_t // cutlass.Int32(32)) * cutlass.Int32(4) + + qz_kb, + ) + qz_loaded = [] + # Generic-proxy stores made visible to the MMA's async-proxy reads, then one arrival for all. + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.barrier_cta_sync(2, thread_count=128) + if qz_tid == cutlass.Int32(0): + prims.mbarrier_arrive(act_full) + if cutlass.const_expr(st_async): + while not cute.arch.mbarrier_test_wait(acc_done.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(acc_done.data_ptr(), 0): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + # tcgen05.ld widths are powers of two: the accumulator columns in descending power-of-two pieces. + acc_all = [] + ld_col = 0 + for ld_w in (128, 64, 32, 16, 8): + if cutlass.const_expr((mma_n * nacc - ld_col) >= ld_w): + piece = prims.tcgen05_ld( + "32x32b", + cutlass.inttoptr( + tmem_ptr_i32.load() + cutlass.Int32(ld_col), 6, cutlass.Float32 + ), + num=ld_w, + ) + acc_all = acc_all + [piece[c] for c in range(ld_w)] + ld_col += ld_w + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + acc = [] + for t in cutlass.range_constexpr(mma_n): + v = cutlass.Float32(acc_all[t]) + for a in cutlass.range_constexpr(1, nacc): + v = v + cutlass.Float32(acc_all[a * mma_n + t]) + acc.append(v) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if tx == 128: + _stamp(timers, timed, bx, 4) + n = m_offset + row + if cutlass.const_expr(split == 1): + for t in cutlass.range_constexpr(mma_n): + if cutlass.Int32(t) < num_tokens: + y.store( + cutlass.BFloat16(cutlass.Float32(acc[t])), idx=cutlass.Int32(t * n_out) + n + ) + if cutlass.const_expr(ar_push > 0): + prims.griddepcontrol(prims.GridDepAction.WAIT) + # The parity of each (token, column slice) slot: the all-reduce's CTA t * 8 + slice flag. All loads are + # issued before the first store so they overlap (one L2 round trip, not one per token). + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_v = cutlass.Float32(acc[t]) + ar_hi = cute.arch.shuffle_sync_bfly(ar_v, offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), _bf16x2_word(ar_v, ar_hi) + ) + else: + owner = row // cutlass.Int32(rows_per_rank) + local = row % cutlass.Int32(rows_per_rank) + if cutlass.const_expr(rope): + # Rank r owns head dims [r * P, (r + 1) * P) and their rotation partners 64 + ..., P = rows / 2. + pairs = rows_per_rank // 2 + owner = (row % cutlass.Int32(64)) // cutlass.Int32(pairs) + local = (row // cutlass.Int32(64)) * cutlass.Int32(pairs) + ( + row % cutlass.Int32(64) + ) % cutlass.Int32(pairs) + if owner != rank: + base = (rank * cutlass.Int32(rows_per_rank) + local) * cutlass.Int32(mb_stride) + if cutlass.const_expr(st_async): + dst = _mapa_u32(mailbox.subview(base).data_ptr(), owner) + mb = _mapa_u32(reduce_ready.data_ptr(), owner) + words = [cutlass.Float32(acc[t]).bitcast(cutlass.Int32) for t in range(mma_n)] + for g in cutlass.range_constexpr(mma_n // 4): + _st_async_v4( + dst + cutlass.Int32(16 * g), + words[4 * g], + words[4 * g + 1], + words[4 * g + 2], + words[4 * g + 3], + mb, + ) + else: + for t in cutlass.range_constexpr(mma_n): + prims.mapa(mailbox.subview(base + cutlass.Int32(t)), owner).store( + cutlass.Float32(acc[t]) + ) + prims.mbarrier_arrive( + prims.mapa(reduce_ready, owner), scope=prims.MemScope.CLUSTER + ) + else: + if cutlass.const_expr(st_async): + while not _test_wait_cluster(reduce_ready.data_ptr(), 0): + pass + else: + while not prims.mbarrier_try_wait_parity( + reduce_ready, 0, scope=prims.MBarrierScope.CLUSTER + ): + pass + if lane == 0: + if row // cutlass.Int32(rows_per_rank) == rank: + if (row % cutlass.Int32(rows_per_rank)) == 0: + _stamp(timers, timed, bx, 5) + totals = [] + for g in cutlass.range_constexpr(mma_n // 4): + sums = [cutlass.Float32(0.0)] * 4 + for q in cutlass.range_constexpr(split): + parts = _ld_shared_v4_f32( + mailbox.subview( + (cutlass.Int32(q * rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g) + ).data_ptr() + ) + sums = [sums[j] + cutlass.select_(rank == cutlass.Int32(q), cutlass.Float32(acc[4 * g + j]), + parts[j]) for j in range(4)] # fmt: skip + for j in cutlass.range_constexpr(4): + totals.append(sums[j]) + if cutlass.const_expr(rope): + # The own-rank mailbox slot is unused: it carries the totals to the RoPE stage. + mailbox.store( + sums[j], + idx=(rank * cutlass.Int32(rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g + j), + ) + elif cutlass.Int32(4 * g + j) < num_tokens: + y.store( + cutlass.BFloat16(sums[j]), + idx=cutlass.Int32((4 * g + j) * n_out) + n, + ) + if cutlass.const_expr(ar_push > 0): + # Every lane of an owner warp is an owner (rows_per_rank >= 32), so lane pairs hold row pairs. + prims.griddepcontrol(prims.GridDepAction.WAIT) + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_hi = cute.arch.shuffle_sync_bfly(totals[t], offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), + _bf16x2_word(totals[t], ar_hi), + ) + prims.barrier_cta_sync(1, thread_count=128) + if cutlass.const_expr(rope): + # Token t of request t / T sits at cache position kv_len - T + t % T. Q / K heads: GPT-NeoX RoPE over + # the 128-wide head, bf16(cos * x -/+ sin * pair) on the bf16 projection; then E4M3 of x * scale, Q to + # rope_q, K / V into the page of that position. + pairs = rows_per_rank // 2 + rp_e = tx - cutlass.Int32(128) + rp_p = rp_e % cutlass.Int32(pairs) + rp_t = rp_e // cutlass.Int32(pairs) + if rp_t < num_tokens: + rp_max_blocks = rope_dims.load(idx=0) + rp_tpb_log2 = rope_dims.load(idx=1) + rp_hq = rope_dims.load(idx=2) + rp_hkv = rope_dims.load(idx=3) + rp_T = rope_dims.load(idx=4) + rp_d = rank * cutlass.Int32(pairs) + rp_p + rp_a = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + rp_p) * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_b = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + cutlass.Int32(pairs) + rp_p) + * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_req = rp_t // rp_T + rp_tok = rope_i.load(idx=rp_req) - rp_T + (rp_t - rp_req * rp_T) + rp_head = n_tile + if rp_head < rp_hq + rp_hkv: + rp_c = rope_cs.load(idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2)) + rp_s = rope_cs.load( + idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2) + + cutlass.Int32(1) + ) + rp_ra = _rope_bits(rp_c, rp_a, -rp_s, rp_b).bitcast(cutlass.Float32) + rp_rb = _rope_bits(rp_c, rp_b, rp_s, rp_a).bitcast(cutlass.Float32) + rp_a = rp_ra + rp_b = rp_rb + rp_qs = rope_scale.load(idx=0) + rp_w = _e4m3x4( + rp_a * rp_qs, rp_b * rp_qs, cutlass.Float32(0.0), cutlass.Float32(0.0) + ) + rp_lo = cutlass.Uint8(rp_w & cutlass.Int32(0xFF)) + rp_hi = cutlass.Uint8((rp_w >> cutlass.Int32(8)) & cutlass.Int32(0xFF)) + if rp_head < rp_hq: + rp_q = (rp_t * rp_hq + rp_head) * cutlass.Int32(128) + rp_d + rope_q.store(rp_lo, idx=rp_q) + rope_q.store(rp_hi, idx=rp_q + cutlass.Int32(64)) + else: + rp_kv = cutlass.select_( + rp_head < rp_hq + rp_hkv, cutlass.Int32(0), cutlass.Int32(1) + ) + rp_kh = rp_head - rp_hq - rp_kv * rp_hkv + rp_page = rope_bt.load( + idx=(rp_req * cutlass.Int32(2) + rp_kv) * rp_max_blocks + + (rp_tok >> rp_tpb_log2) + ) + # 64-bit byte offset: the pool view can exceed 2^31 bytes. + rp_dst = ( + (cutlass.Int64(rp_page) * cutlass.Int64(rp_hkv) + cutlass.Int64(rp_kh)) + << cutlass.Int64(rp_tpb_log2 + cutlass.Int32(7)) + ) + cutlass.Int64( + (rp_tok & ((cutlass.Int32(1) << rp_tpb_log2) - cutlass.Int32(1))) + * cutlass.Int32(128) + + rp_d + ) + rp_addr = rope_pool.data_ptr(0).toint() + rp_dst + cutlass.inttoptr(rp_addr, mem_space=1, dtype=cutlass.Uint8).store(rp_lo) + cutlass.inttoptr( + rp_addr + cutlass.Int64(64), mem_space=1, dtype=cutlass.Uint8 + ).store(rp_hi) + if tx == 128: + _stamp(timers, timed, bx, 6) + if cutlass.const_expr(ar_push == 2): + # The all-reduce polls the pushed words; it launches after this CTA's rows are out, not at grid end. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if warp_id == 4: + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Int32), + tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n), + ) + + +def _bytes_map(t, cols, rows, box_rows, swizzle): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[cols, rows], + global_strides=[cols // 16], + box_dims=[CTA_K, box_rows], + swizzle=swizzle, + ) + + +def _atom_map(t, atoms): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint16, + global_dims=[SF_ATOM_U16, atoms], + global_strides=[SF_ATOM_BYTES // 16], + box_dims=[SF_ATOM_U16, 1], + swizzle=cuda.TensorMapSwizzle.none, + ) + + +@cute.jit +def m3_mx_gemv( + w: cute.Tensor, # [N, K] E4M3 as uint8, K contiguous + wsf: cute.Tensor, # weight scales, 128x4-swizzled, as uint8 + x: cute.Tensor, # [M, K] E4M3 as uint8, M <= 8 + xsf: cute.Tensor, # activation scales, 128x4-swizzled (atom k at byte 512 k), as uint8 + y: cute.Tensor, # [M * N] bf16 + x_words: cute.Tensor, # x_bf16: bf16 x [M, K] as int32 words (x / xsf are then unused placeholders) + timers: cute.Tensor, # [grid * 8] int64, written when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[int], + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, + din_i: cute.Tensor, # din / rope inputs: see m3_mx_gemv_kernel (placeholders when off) + din_table: cute.Tensor, + din_hidden: cute.Tensor, + din_w: cute.Tensor, + din_f: cute.Tensor, + din_vocab: cutlass.Int32, + din_g: cute.Tensor, + din_res: cute.Tensor, + din_flags: cutlass.Int32, + rope_cs: cute.Tensor, + rope_i: cute.Tensor, + rope_bt: cute.Tensor, + rope_q: cute.Tensor, + rope_pool: cute.Tensor, + rope_scale: cute.Tensor, + rope_dims: cute.Tensor, + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], + x_swiglu: cutlass.Constexpr[bool] = False, +) -> None: + # x_swiglu's up half rides the slot din uses for its norm weights, and it activates the bf16 x. + assert not (din and x_swiglu) and (x_bf16 or not x_swiglu), ( + "x_swiglu needs x_bf16 and excludes din" + ) + k_tiles = num_k_tiles(k_in) + tma_desc_w = _bytes_map(w, k_in, n_out, CTA_M, cuda.TensorMapSwizzle.s128b) + tma_desc_wsf = _atom_map(wsf, (n_out // CTA_M) * k_tiles) + tma_desc_x = _bytes_map(x, k_in, num_tokens, mma_n, cuda.TensorMapSwizzle.s128b) + tma_desc_xsf = _atom_map(xsf, k_tiles) + m3_mx_gemv_kernel( + tma_desc_w, tma_desc_wsf, tma_desc_x, tma_desc_xsf, y, x_words, timers, num_tokens, n_out, k_in, split, ring, + trigger_early, contig, timed, nacc, a_tmem, st_async, x_bf16, ar_ws, ar_flags, ar_rank, ar_push, ar_world, + ar_maxt, mma_n, + din_i, din_table, din_hidden, din_w, din_f, din_vocab, din_g, din_res, din_flags, rope_cs, + rope_i, rope_bt, rope_q, rope_pool, rope_scale, rope_dims, din, rope, x_swiglu, + ).launch( + grid=((n_out // CTA_M) * split, 1, 1), + block=(THREADS, 1, 1), + cluster=(split, 1, 1), + stream=stream, + use_pdl=use_pdl, + ) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/op.py new file mode 100644 index 000000000000..a0ccb1ab0215 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/mx_gemv/op.py @@ -0,0 +1,535 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) for already-quantized activations of at most 8 tokens. + +Compiled on the first call per (N, K, split, ring, early trigger, PDL, push, MMA width), which must happen outside +CUDA-graph capture, or ahead of any call by ``precompile``; once ``freeze`` closes compiles, a call whose kernel is not +compiled declines instead. +""" + +from __future__ import annotations + +import threading +from pathlib import Path +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +MMA_WIDTHS = ( + 8, + 16, + 32, + 48, +) # token columns of the compiled MMA; a call uses the smallest that holds M +_SMEM_LIMIT = 220 * 1024 # capacity: fixed: the device's shared-memory limit + + +def _mma_n(m: int) -> int: + return next(w for w in MMA_WIDTHS if m <= w) + + +def _smem_bytes(ring: int, tiles: int, split: int, mma_n: int) -> int: + return ( + ring * (128 * 128 + 512) + + tiles * (mma_n * 128 + 512) + + split * (128 // split) * (mma_n + 4) * 4 + + 1024 + ) + + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + +# (N, K) -> (split, ring): GB300 exposed time behind a 4.25 us PDL predecessor at M 1 / 8, vs the fused low-M GEMM: +# QKV 2.78 / 3.16 vs 4.58 / 6.21 us, o_proj 2.23 / 2.41 vs 4.40 / 5.09, shared gate_up 2.43 / 2.74 vs 3.56 / 4.30, +# shared down 1.74 / 1.66 vs 2.92 / 3.38. A config is (split, ring[, contig, nacc, a_tmem, st_async]); the optional +# fields default to _CONTIG, _NACC, _A_TMEM, _ST_ASYNC. +_CONFIGS: Dict[Tuple[int, int], tuple] = { + (2560, 6144): (4, 12), + (6144, 2048): (2, 8), + # Shared-expert gate_up: split 2, ring 12, single accumulator, A operand from shared memory. + (1536, 6144): (2, 12, 1, 1, 0, 1), + # Shared-expert down: unsplit, ring 3, single accumulator, A operand from shared memory. + (6144, 768): (1, 3, 1, 1, 0, 1), + (2304, 6144): (4, 12), + (6144, 3072): (2, 12), + # Dense-layer gate_up: 6.4 / 6.3 us at M 1 / 8 vs FlashInfer 8.8. + (6144, 6144): (2, 12), +} +# Per-token-count configs (N, K, M): the grid that suits the concurrent routed-expert engine depends on how long +# that engine runs. At 2 tokens the shared-expert gate_up splits K over 4 CTAs. +_CONFIGS_BY_M: Dict[Tuple[int, int, int], tuple] = { + (1536, 6144, 2): (4, 12, 1, 1, 0, 1), +} +# Above 8 tokens (spec-decode verify rows): the shared-expert gate_up splits K over 4 CTAs. +_CONFIGS_WIDE: Dict[Tuple[int, int], tuple] = { + (1536, 6144): (4, 12, 1, 1, 0, 1), +} + +# Defaults of the optional config fields: contiguous k-tile ranges per split rank, one TMEM accumulator, the A +# operand staged in TMEM (when its columns fit), st.async split-K partial exchange. +_CONTIG = True +_NACC = 1 +_A_TMEM = True +_ST_ASYNC = True +_ensured = set() + + +_DUMMY: Dict[torch.device, torch.Tensor] = {} + + +def _dummy_timers(device) -> torch.Tensor: + """Placeholder for the kernel's timer buffer, which it only writes when compiled with timers on.""" + t = _DUMMY.get(device) + if t is None: + t = _DUMMY[device] = torch.zeros(8, dtype=torch.int64, device=device) + return t + + +def _arg(t: torch.Tensor, align: int = 16): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align).mark_layout_dynamic(leading_dim=t.dim() - 1) + + +def get_config(n: int, k: int, m: Optional[int] = None) -> Optional[tuple]: + cfg = _CONFIGS_BY_M.get((n, k, m)) if m is not None else None + if cfg is None and m is not None and m > 8: + cfg = _CONFIGS_WIDE.get((n, k)) + cfg = cfg or _CONFIGS.get((n, k)) + # The split-K epilogue gives each of the split CTAs 128 // split rows of a 128-row output tile. + if cfg is None or n % 128 != 0 or k % 128 != 0 or (k // 128) % cfg[0] != 0 or 128 % cfg[0] != 0: + return None + return cfg + + +def _declines(n: int, k: int, m: int, x_bf16: bool, cfg: tuple) -> bool: + """Whether a (split, ring, ...) config declines an [M, K] x [N, K]^T call on shape grounds.""" + split = cfg[0] + if not 0 < m <= MAX_TOKENS or 128 % split != 0 or (k // 128) % split != 0: + return True + tiles = k // 128 // split + mma_n = _mma_n(m) + # GB300 exposed time vs the FlashInfer prequantized GEMM (behind a PDL predecessor), at 24 / 48 tokens: QKV 3.76 / + # 4.63 vs 7.46 / 11.44 us, o_proj 2.74 / 3.33 vs 4.46 / 5.84, shared gate_up 4.57 / 5.71 vs 7.16 / 11.25. Behind + # only for K = 768 (unsplit shared down) at 17-32 tokens: 3.1 vs 2.9. + # The in-kernel bf16 -> MXFP8 quantization (x_bf16) costs one 32-element block per thread and round on 128 threads; + # beyond 8 rounds per CTA, mxfp8_quantize + this GEMV (or the fallback GEMM) is faster. + if (16 < m <= 32 and k < 1024) or (x_bf16 and tiles * 4 * mma_n > 8 * 128): + return True + return _smem_bytes(min(cfg[1], tiles, 2), tiles, split, mma_n) > _SMEM_LIMIT + + +def serves(n: int, k: int, m: int, x_bf16: bool) -> bool: + """Whether the GEMV with its default config takes an [M, K] activation (bf16 or MXFP8) against an [N, K] weight.""" + cfg = get_config(n, k, m) + return cfg is not None and not _declines(n, k, m, x_bf16, cfg) + + +def _resolve( + m: int, + n: int, + k: int, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple], + x_swiglu: bool = False, +) -> Optional[tuple]: + """The compile key of an [M, K] x [N, K]^T call and its runtime push words, or None when the call is declined. + + ``ar`` is the push tuple of ``_run``; only its (world, bucket tokens, mode) enter the key, the addresses and the + rank are runtime arguments.""" + cfg = cfg or get_config(n, k, m) + if cfg is None or _declines(n, k, m, x_bf16, cfg): + return None + split, ring = cfg[0], cfg[1] + contig = bool(cfg[2]) if len(cfg) > 2 else _CONTIG + nacc = int(cfg[3]) if len(cfg) > 3 else _NACC + a_tmem = bool(cfg[4]) if len(cfg) > 4 else _A_TMEM + st_async = bool(cfg[5]) if len(cfg) > 5 else _ST_ASYNC + if nacc != 1: + # More than one TMEM accumulator is broken on SM107 (nondeterministic or garbage + # outputs). No shipped table uses it; decline (the caller's fallback runs) until the kernel path is fixed. + return None + ring = min(ring, k // 128 // split) + tiles = k // 128 // split + mma_n = _mma_n(m) + while ring > 2 and _smem_bytes(ring, tiles, split, mma_n) > _SMEM_LIMIT: + ring -= 1 + if a_tmem: + from .m3_mx_gemv_kernel import tmem_cols + + if tmem_cols(ring, nacc, True, tiles, mma_n) > 512: + a_tmem = False + # ar = (multicast workspace address, flags address, rank, world, bucket tokens, mode): push the output rows into + # the next m3_mc_allreduce call's exchange slots (mode 2 also defers the dependents' launch until after the push). + ar_ws, ar_flags, ar_rank, ar_mode, ar_world, ar_maxt = 0, 0, 0, 0, 1, 8 + if ar is not None and split <= 4 and n % 2 == 0: + ar_ws, ar_flags, ar_rank, ar_world, ar_maxt, ar_mode = ar + if ar_mode == 2: + trigger_early = False + if use_pdl is None: + use_pdl = get_env_enable_pdl() + # The kernel's ``timed`` flag (per-CTA milestone timers) is always off. + key = ( + n, + k, + split, + ring, + trigger_early, + contig, + False, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + x_swiglu, + ) + return key, (ar_ws, ar_flags, ar_rank) + + +def _compile(key: tuple, args: tuple, m: int, runtime_ar: tuple, stream, extra: tuple): + """cute.compile the kernel of ``key`` (see ``_resolve``) on these argument views; cached.""" + import cutlass.cute as cute + + from . import m3_mx_gemv_kernel as kernel + + ( + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + x_swiglu, + ) = key + ar_ws, ar_flags, ar_rank = runtime_ar + with _lock: + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_mx_gemv {key}", + lambda: cute.compile( + kernel.m3_mx_gemv, + *args, + m, + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_ws, + ar_flags, + ar_rank, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + stream, + *extra, + False, + False, + x_swiglu, + ), + sources=_aot_sources(), + ) + return fn + + +def _views( + x: torch.Tensor, + xsf: Optional[torch.Tensor], + weight, + weight_scale, + y, + x_bf16: bool, + x_swiglu: bool = False, +) -> tuple: + # x_swiglu: x is [M, 2K] (gate | up); the GEMV's K is half its width. + k = x.shape[1] // 2 if x_swiglu else x.shape[1] + if x_bf16: + x_bytes = x.view(torch.uint8) + xsf = _placeholder_scales(k, x.device) + x_words = x.view(torch.int32) + else: + x_bytes = x.view(torch.uint8) + x_words = _placeholder_words(x.device) + return ( + _arg(weight.view(torch.uint8)), + _arg(weight_scale.view(torch.uint8).reshape(-1)), + _arg(x_bytes), + _arg(xsf.view(torch.uint8).reshape(-1)), + _arg(y.view(-1) if y.numel() else _placeholder_words(x.device).view(torch.bfloat16)), + _arg(x_words.reshape(-1)), + _arg(_dummy_timers(x.device)), + ) + + +def _run( + x: torch.Tensor, + xsf: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple] = None, + x_swiglu: bool = False, +) -> Optional[torch.Tensor]: + global last_pushed, last_uncompiled + last_pushed = False + last_uncompiled = None + import cuda.bindings.driver as cuda_driver + + m, k = x.shape + if x_swiglu: + k //= 2 + n = weight.shape[0] + if weight.shape[1] != k: + return None + resolved = _resolve(m, n, k, x_bf16, trigger_early, cfg, use_pdl, ar, x_swiglu) + if resolved is None: + return None + key, runtime_ar = resolved + fn = _compiled.get(key) + if fn is None: + if _frozen[0]: + # Compiles are closed (the model built every kernel it dispatches to before warmup): decline, the caller + # reports the miss. + last_uncompiled = key + return None + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_mx_gemv must run once per shape outside CUDA-graph capture first" + ) + y = torch.empty(m, n, dtype=torch.bfloat16, device=x.device) + args = _views(x, xsf, weight, weight_scale, y, x_bf16, x_swiglu) + extra = _extra_args(x.device) + stream = cuda_driver.CUstream(torch.cuda.current_stream(x.device).cuda_stream) + if fn is None: + fn = _compile(key, args, m, runtime_ar, stream, extra) + ar_ws, ar_flags, ar_rank = runtime_ar + fn(*args, m, ar_ws, ar_flags, ar_rank, stream, *extra) + last_pushed = key[11] > 0 + return y + + +def precompile( + weight: torch.Tensor, + weight_scale: torch.Tensor, + m: int, + x_bf16: bool, + use_pdl: Optional[bool] = None, + ar: Optional[Tuple[int, int, int]] = None, + x_swiglu: bool = False, +) -> Optional[tuple]: + """Compile, without launching, the kernel an M-row call against ``weight`` takes; returns its key, or None when + such a call is declined. ``ar`` = (world, bucket tokens, mode) of a pushing call (its addresses and rank are + runtime arguments, so none is needed here). Runs outside capture, even while compiles are frozen.""" + import cuda.bindings.driver as cuda_driver + + n, k = weight.shape + push = None if ar is None else (0, 0, 0, *ar) + resolved = _resolve(m, n, k, x_bf16, True, None, use_pdl, push, x_swiglu) + if resolved is None: + return None + key, runtime_ar = resolved + if key in _compiled: + return key + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_mx_gemv.precompile allocates and compiles: call it outside capture") + dev = weight.device + x = torch.zeros( + m, + 2 * k if x_swiglu else k, + dtype=torch.bfloat16 if x_bf16 else torch.float8_e4m3fn, + device=dev, + ) + xsf = None if x_bf16 else torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=dev) + y = torch.empty(m, n, dtype=torch.bfloat16, device=dev) + args = _views(x, xsf, weight, weight_scale, y, x_bf16, x_swiglu) + stream = cuda_driver.CUstream(torch.cuda.current_stream(dev).cuda_stream) + _compile(key, args, m, runtime_ar, stream, _extra_args(dev)) + return key + + +def freeze(frozen: bool = True) -> None: + """Close (or reopen) compiles: while frozen, a call whose kernel is not compiled yet declines (returns None and + sets ``last_uncompiled``) instead of compiling. ``precompile`` and ``ensure_compiled`` still compile.""" + _frozen[0] = bool(frozen) + + +def frozen() -> bool: + return _frozen[0] + + +def is_compiled(key: tuple) -> bool: + return key in _compiled + + +def _extra_args(device) -> tuple: + """Placeholder arguments of the kernel's draft-input prologue and RoPE / KV epilogue (compiled off).""" + i32 = _placeholder_words(device) + f32 = i32.view(torch.float32) + u8 = i32.view(torch.uint8) + d = (_arg(i32), _arg(i32), _arg(i32), _arg(i32), _arg(f32), 0, _arg(i32), _arg(i32), 0) + r = (_arg(f32), _arg(i32), _arg(i32), _arg(u8), _arg(u8), _arg(f32), _arg(i32)) + return d + r + + +def m3_mx_gemv( + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """``xq`` E4M3 [M <= 8, K] with ``xsf`` its 128x4-swizzled UE8M0 scales; ``weight`` E4M3 [N, K] with + ``weight_scale`` swizzled. Returns bf16 [M, N], or None when the shape is not served.""" + return _run(xq, xsf, weight, weight_scale, False, trigger_early, cfg, use_pdl, ar) + + +def m3_mx_gemv_bf16( + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """The same GEMV for a bf16 ``x`` [M <= 8, K] (contiguous), quantized to MXFP8 inside the kernel exactly as + torch.ops.trtllm.mxfp8_quantize(x, True) does.""" + if x.dtype != torch.bfloat16 or x.dim() != 2 or not x.is_contiguous() or x.data_ptr() % 16 != 0: + return None + return _run(x, None, weight, weight_scale, True, trigger_early, cfg, use_pdl, ar) + + +def m3_mx_gemv_swiglu( + gate_up: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """``m3_mx_gemv_bf16(swiglu(gate_up), ...)`` in one launch: ``gate_up`` bf16 [M <= 8, 2K] (contiguous, gate then + up) is activated in the kernel's quantize prologue with the swigluoai arithmetic of the Triton silu_and_mul kernel + (alpha 1.702, limit 7, the (up + 1) offset; bit for bit), then quantized as the bf16 path does.""" + if ( + gate_up.dtype != torch.bfloat16 + or gate_up.dim() != 2 + or not gate_up.is_contiguous() + or gate_up.data_ptr() % 16 != 0 + or gate_up.shape[1] % 2 != 0 + ): + return None + return _run(gate_up, None, weight, weight_scale, True, trigger_early, cfg, use_pdl, ar, True) + + +_PLACEHOLDERS: Dict[tuple, torch.Tensor] = {} +# Whether the last call pushed its output into an all-reduce (see _run's ``ar``). +last_pushed = False +# The key of the last call declined because compiles are frozen and its kernel was not compiled (see ``freeze``). +last_uncompiled: Optional[tuple] = None +_frozen = [False] + + +def _placeholder_scales(k: int, device) -> torch.Tensor: + key = ("sf", k, device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(k // 128 * 512, dtype=torch.uint8, device=device) + return t + + +def _placeholder_words(device) -> torch.Tensor: + key = ("w", device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(16, dtype=torch.int32, device=device) + return t + + +def ensure_compiled(weight: torch.Tensor, weight_scale: torch.Tensor, use_pdl: bool) -> None: + """Compiles the kernel of this weight's shape outside capture (a one-token dummy call).""" + if ( + weight.dim() != 2 + or weight.dtype != torch.float8_e4m3fn + or get_config(*weight.shape) is None + ): + return + n, k = weight.shape + if (n, k, use_pdl) in _ensured: + return + _ensured.add((n, k, use_pdl)) + xq = torch.zeros(1, k, dtype=torch.float8_e4m3fn, device=weight.device) + xsf = torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=weight.device) + m3_mx_gemv(xq, xsf, weight, weight_scale, use_pdl=use_pdl) + for width in MMA_WIDTHS[1:]: + m3_mx_gemv( + torch.zeros(width, k, dtype=torch.float8_e4m3fn, device=weight.device), + xsf, + weight, + weight_scale, + use_pdl=use_pdl, + ) + m3_mx_gemv_bf16( + torch.zeros(1, k, dtype=torch.bfloat16, device=weight.device), + weight, + weight_scale, + use_pdl=use_pdl, + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/triton_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/triton_sparse_decode.py new file mode 100644 index 000000000000..ef5a7c7f039f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/triton_sparse_decode.py @@ -0,0 +1,1732 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. + +This is the MiniMax-M3 NVFP4 / sm_107 / tp4 target's copy. On top of the +in-tree decode it routes long-context TP4 steps to the one-launch +``m3_attn_lc`` kernel, short-sequence verify steps of at least +``MQ_MIN_TOKENS`` rows to the multi-query producer, releases PDL dependents +early, and has the merge emit the MXFP8 o_proj hand-off (``m3_prequant.SLOT``). +""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers + +from . import m3_attn_lc, m3_prequant, m3_report + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 # capacity: fixed: the MSA block size of the checkpoint + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 # capacity: fixed: a launch-grid target, not a size + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +# Fewest query rows for which the merge writes the MXFP8 o_proj hand-off: every decode step, so the MXFP8 o_proj +# never runs its own quantize kernel after attention. +MERGE_MXFP8_MIN_TOKENS = 1 # capacity: fixed: a row lower bound + + +class MergeMxfp8: + """MXFP8 copy of the split-K merge output for the o_proj GEMM. + + From ``MERGE_MXFP8_MIN_TOKENS`` to ``MAX_ROWS`` rows the merge kernel writes the E4M3 rows and swizzled UE8M0 + scales next to the BF16 output and publishes them to ``m3_prequant.SLOT``, so the MXFP8 o_proj skips its own + quantization. One 128-row buffer per width serves every layer: each o_proj consumes it before the next layer's + merge rewrites it, in stream order. + """ + + MAX_ROWS = 128 # capacity: fixed: the merge hand-off's rows (decode steps) + + def __init__(self): + self._buffers = {} + + def buffers(self, width: int, device: torch.device): + key = (width, device) + if key not in self._buffers: + self._buffers[key] = ( + torch.empty((self.MAX_ROWS, width), dtype=torch.float8_e4m3fn, device=device), + torch.zeros((self.MAX_ROWS * (width // 32),), dtype=torch.uint8, device=device), + ) + return self._buffers[key] + + +MERGE_MXFP8 = MergeMxfp8() + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +# Mixed ctx+gen steps launch on [num_ctx:] slices (unaligned seq_lens / block_table) with arbitrary +# total_q; specializing on those compiled a new variant mid-serving (~650 ms TTFT stall at c8). +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + out_ptr, # final [total_q, num_heads, head_dim] for FUSED_MERGE, else unused + counter_ptr, # int32 [total_q, num_kv_heads] arrival counters for FUSED_MERGE, zero at rest + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + num_bt_cols, # block-table width: a top-k id at or past it is not a block of the request + num_pages, # KV pool pages: a page id outside [0, num_pages) is not read + stride_out_n, + stride_out_h, + stride_out_d, + num_kv_heads, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge + FUSED_MERGE: tl.constexpr, # the last chunk to finish merges the partials in place + EARLY_TRIGGER: tl.constexpr, # release PDL dependents (weight prefetch) at entry + IDENTITY: tl.constexpr = False, # identity selection: slot i is block i (t_ptr is not read) +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + # Dependents still wait for this grid to complete; triggering here only + # lets them start their input-independent prologue (e.g. weight loads). + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for it in tl.range(chunk_start, chunk_end): + if IDENTITY: + # Slot `it` of the identity selection, bounded by this row's own kv extent above. + blk = it + else: + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + # Fail closed: a top-k id outside the request's block-table row, or the page it names outside the pool, + # contributes nothing instead of a read through a wrong page (valid inputs are unaffected). + blk_ok = (blk >= 0) & (blk < num_bt_cols) + page = tl.load(bt_row + blk, mask=blk_ok, other=0).to(tl.int64) + blk_ok = blk_ok & (page >= 0) & (page < num_pages) + page = tl.where(blk_ok, page, 0) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + # A slot that failed closed is skipped whole: no score, no rescale (an all-masked first block + # would make exp2(-inf - -inf) = NaN). + if blk_ok: + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + if FUSED_MERGE: + # Split-K serial reduction: every chunk publishes its partial, and the + # chunk that arrives last for this (token, KV head) merges all of them + # exactly as _merge_topk_attn_out_kernel does, then re-arms the counter. + tl.debug_barrier() + counter = counter_ptr + pid_b * num_kv_heads + pid_kh + arrived = tl.atomic_add(counter, 1, sem="acq_rel", scope="gpu") + if arrived == NUM_TOPK_CHUNKS - 1: + tl.debug_barrier() + o_rows = ( + o_ptr + + pid_b * stride_o_b + + (pid_h + off_h[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d + ) + l_rows = lse_ptr + pid_b * stride_l_b + (pid_h + off_h) * stride_l_h + lse_max = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + lse_max = tl.maximum(lse_max, lse_c) + # All chunks empty (a CUDA-graph padding row): zero, never NaN. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + denom = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + denom += tl.exp2(lse_c - lse_max) + denom = tl.where(denom > 0, denom, 1.0) + merged = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + w_c = tl.exp2(lse_c - lse_max) / denom + o_c = tl.load( + o_rows + c * stride_o_c, mask=hd_mask, other=0.0, cache_modifier=".cg" + ).to(tl.float32) + merged += o_c * w_c[:, None] + tl.store( + out_ptr + + pid_b * stride_out_n + + (pid_h + off_h[:, None]) * stride_out_h + + off_d[None, :] * stride_out_d, + merged.to(out_ptr.dtype.element_ty), + mask=hd_mask, + ) + tl.atomic_xchg(counter, 0, sem="relaxed", scope="gpu") + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + num_bt_cols, # block-table width: a top-k id at or past it is not a block of the request + num_pages, # KV pool pages: a page id outside [0, num_pages) is not read + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge + IDENTITY: tl.constexpr = False, # identity selection: slot i is block i (t_ptr is not read) +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + if IDENTITY: + blocks_0 = tl.where(chunk_offsets < count_0, chunk_start + chunk_offsets, sentinel) + blocks_1 = tl.where(chunk_offsets < count_1, chunk_start + chunk_offsets, sentinel) + else: + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for cand in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + if IDENTITY: + blk = chunk_start + cand + else: + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + if IDENTITY: + blk_0 = tl.where(valid_0, idx_0, sentinel) + blk_1 = tl.where(valid_1, idx_1, sentinel) + else: + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + if IDENTITY: + blk = chunk_start + cand + else: + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + # Fail closed as in _gqa_sparse_decode_kernel: an id outside the row, or a page outside the + # pool, reads nothing. + blk_ok = (blk >= 0) & (blk < num_bt_cols) + page = tl.load(bt_row + blk, mask=blk_ok, other=0).to(tl.int64) + blk_ok = blk_ok & (page >= 0) & (page < num_pages) + page = tl.where(blk_ok, page, 0) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + # A slot that failed closed is skipped whole: no score, no rescale (an all-masked first block + # would make exp2(-inf - -inf) = NaN). + if blk_ok: + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.jit +def _rcp_approx_ftz(x): + return tl.inline_asm_elementwise( + "rcp.approx.ftz.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _store_mxfp8_head(x, q8_ptr, sf_ptr, row, head, num_sf_k_tiles, HEAD_DIM: tl.constexpr): + """Quantize one row's [head * HEAD_DIM, (head + 1) * HEAD_DIM) slice to MXFP8 as + torch.ops.trtllm.mxfp8_quantize(x, True) does: UE8M0 scale = round-up(amax * rcp(448)), E4M3 RN-satfinite, + scales in the 128x4-swizzled layout.""" + NUM_SF: tl.constexpr = HEAD_DIM // 32 + xr = tl.reshape(x.to(tl.float32), (NUM_SF, 32)) + amax = tl.max(tl.abs(xr), axis=1) + sfv = amax * _rcp_approx_ftz(tl.full((NUM_SF,), 448.0, tl.float32)) + bits = sfv.to(tl.int32, bitcast=True) + e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(tl.int32) + e = tl.minimum(e, 254) + scale = ((254 - e) << 23).to(tl.float32, bitcast=True) + scale = tl.where(amax != 0, scale, 0.0) + q = (xr * scale[:, None]).to(tl.float8e4nv) + off_q = tl.reshape(tl.arange(0, HEAD_DIM), (NUM_SF, 32)) + tl.store(q8_ptr + row * (num_sf_k_tiles * 128) + head * HEAD_DIM + off_q, q) + k_sf = head * NUM_SF + tl.arange(0, NUM_SF) + off_sf = ( + (row // 128) * num_sf_k_tiles * 512 + + (k_sf // 4) * 512 + + (row % 32) * 16 + + ((row % 128) // 32) * 4 + + k_sf % 4 + ) + tl.store(sf_ptr + off_sf, e.to(tl.uint8)) + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr = False, + q8_ptr=None, # MXFP8 copy of out: [total_q, num_heads * head_dim] E4M3 + sf_ptr=None, # its UE8M0 block scales, 128x4-swizzled + num_sf_k_tiles=0, # ceil(num_heads * head_dim / 128) + MXFP8_OUT: tl.constexpr = False, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + if EARLY_TRIGGER: + # Release the dependent (the o_proj GEMM) before waiting on the attention producer, so its weight + # prefetch overlaps the attention kernel as well as this merge; its griddepcontrol.wait still waits + # for this grid (and therefore the producer) to complete. + tl.extra.cuda.gdc_launch_dependents() + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + if MXFP8_OUT: + _store_mxfp8_head( + o_merged.to(out_ptr.dtype.element_ty), + q8_ptr, + sf_ptr, + pid_b, + pid_h, + num_sf_k_tiles, + BLOCK_SIZE_D, + ) + + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on the datacenter Blackwell and Rubin targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3), (10, 7)) +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +# Multi-query producer (triton_sparse_decode_mq) for identity-selection verify steps with at least this many query +# tokens; below it the per-token producer is as fast. +MQ_MIN_TOKENS = 17 # capacity: fixed: the multi-query producer's crossover +# Warps per multi-query producer program. +MQ_NUM_WARPS = 8 +# The long-context decode (``_longattn_lc_decode``) in its engagement line (no row range: every step the LC +# applies to takes it). +LC_PATH = "m3_sparse_decode_lc" + + +def _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + identity=False, + identity_step=False, +) -> None: + """minimax_m3_sparse_attn_decode through m3_attn_lc: same table, output and MXFP8 hand-off contract. ``identity`` + takes the short-sequence identity selection from the device lengths (the table is not read); ``identity_step`` + marks a step with no select (the kernel reads q after its grid wait), also when the staged table is read.""" + + use_pdl = get_env_enable_pdl() + total_q, num_heads, head_dim = q.shape + linear = _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=1, + gqa_group_size=num_heads, + max_topk=int(topk_idx.shape[-1]), + decode_query_len=decode_query_len, + ) + mxfp8 = None + m3_prequant.SLOT.clear() + if MERGE_MXFP8_MIN_TOKENS <= total_q <= MergeMxfp8.MAX_ROWS: + mxfp8 = MERGE_MXFP8.buffers(num_heads * head_dim, output.device) + # The early trigger lets the o_proj launch and stream its weights while attention runs. + m3_attn_lc.m3_lc_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + linear_softmax=linear, + mxfp8=mxfp8, + use_pdl=use_pdl, + early_trigger=True, + # This target's op (sparseDecodeLc4): the merge weights formed once per CTA, and up to + # m3_attn_lc.ONE_DIV_MAX_ROWS rows the split-K merge dividing once per output (fixed order, no atomics; + # inside the same band of the float64 reference). + merge_once=True, + prefetch=m3_attn_lc.PREFETCH.take(), + next_kv=m3_attn_lc.NEXT_KV.take(), + identity=identity, + identity_step=identity_step, + one_div=m3_attn_lc.one_div_applies(total_q), + ) + if m3_attn_lc.one_div_applies(total_q): + m3_report.taken(ONE_DIV_MERGE_PATH, total_q) + if mxfp8 is not None: + m3_prequant.SLOT.publish( + output.view(total_q, num_heads * head_dim), mxfp8[0][:total_q], mxfp8[1] + ) + + +# Engagement path of the identity-selection producers reading their blocks from the kv lengths (IDENTITY). +DEVICE_IDENTITY_PATH = "sparse_decode_device_identity" +# Engagement path of the LC's one-division split-K merge. +ONE_DIV_MERGE_PATH = "sparse_decode_lc_one_div_merge" + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + if kv_nvfp4: + capability = torch.cuda.get_device_capability(q.device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103/SM107; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + # Long-context TP4 steps: attention, split-K merge and MXFP8 hand-off in one launch. + + if ( + kv_nvfp4 + and widen_q + and num_topk_chunks is None + and m3_attn_lc.lc_applies( + q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output + ) + ): + _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + # The identity selection from the device lengths (the staged table is not read). + identity=identity_blocks, + # Every identity step (no select ran): q after the grid wait. + identity_step=identity_blocks, + ) + m3_report.taken(LC_PATH, total_q) + return + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else 1 + ) + # The stock in-tree decode's split-K chunking at every row count (bit for bit its rows), as the SM107 TP2 target. + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + # Every query token of a request reads the same blocks under identity selection (short-sequence mode): one + # program per (request, block) covers all of them and dequantizes each block once. + use_mq = ( + identity_blocks + and total_q >= MQ_MIN_TOKENS + and kv_nvfp4 + and num_kv_heads == 1 + and decode_query_len > 1 + and head_dim == 128 + and gqa_group_size * decode_query_len <= 128 + ) + if identity_blocks and kv_nvfp4 and widen_q: + # The identity-selection route is modeling.SPARSE_DECODE_PRODUCER's. + m3_report.taken( + "sparse_decode_producer", + total_q, + "multi_query" if use_mq else "per_row", + ) + if use_mq: + num_topk_chunks = max_topk + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = get_env_enable_pdl() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + device_identity = identity_blocks + if device_identity and not use_mq: + m3_report.taken(DEVICE_IDENTITY_PATH, total_q) + single_kernel_args = ( + { + # No split-K arrival counters: the separate merge kernel combines the chunks. + "merge": (output, None), + "tail": (output.stride(0), output.stride(1), output.stride(2), num_kv_heads), + # An early release pays off when the dependent is the consumer of the + # final output (it can prefetch), or a merge grid that itself releases + # its dependent at entry: the o_proj then launches during this kernel and + # streams its weights while attention runs. Above 8 query rows the + # concurrent weight stream slows the larger attention grid more than the + # prefetch saves (GB300 TP4: -1.1% at 48 rows). + "options": {"FUSED_MERGE": False, "EARLY_TRIGGER": direct_output or total_q <= 8}, + } + if query_group_size == 1 + else {"merge": (), "tail": (), "options": {}} + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + if use_mq: + from .triton_sparse_decode_mq import launch_mq_producer + + launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + sm_scale=sm_scale, + decode_query_len=decode_query_len, + use_pdl=use_pdl, + early_trigger=False, + num_warps=MQ_NUM_WARPS, + ) + else: + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + *single_kernel_args["merge"], + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + block_table.shape[1], + k_paged.shape[0], + *single_kernel_args["tail"], + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + # Identity selection from the lengths this kernel reads, not the staged table (dispatch kill switch). + IDENTITY=device_identity, + **single_kernel_args["options"], + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + m3_prequant.SLOT.clear() + if not direct_output: + width = num_heads * head_dim + mxfp8 = ( + MERGE_MXFP8.buffers(width, output.device) + if MERGE_MXFP8_MIN_TOKENS <= total_q <= MergeMxfp8.MAX_ROWS + and head_dim == 128 + and output.dim() == 3 + and output.is_contiguous() + and output.dtype == torch.bfloat16 + else None + ) + mxfp8_args = ( + { + "q8_ptr": mxfp8[0], + "sf_ptr": mxfp8[1], + "num_sf_k_tiles": width // 128, + "MXFP8_OUT": True, + } + if mxfp8 is not None + else {} + ) + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + # Release the o_proj at the merge's entry so it streams its weights during the merge. + EARLY_TRIGGER=True, + **mxfp8_args, + **pdl_launch, + **merge_launch_options, + ) + if mxfp8 is not None: + m3_prequant.SLOT.publish(output.view(total_q, width), mxfp8[0][:total_q], mxfp8[1]) + + +__all__ = [ + "MERGE_MXFP8", + "MERGE_MXFP8_MIN_TOKENS", + "MergeMxfp8", + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/triton_sparse_decode_mq.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/triton_sparse_decode_mq.py new file mode 100644 index 000000000000..048d31b82d68 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/kernels/triton_sparse_decode_mq.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multi-query block-sparse decode producer for MiniMax-M3 short-sequence verify steps. + +In short-sequence mode every request attends at most ``topk`` KV blocks and block selection is the identity, so +all ``decode_query_len`` query tokens of a request read the same blocks. The per-token producer +(``_gqa_sparse_decode_kernel``) then dequantizes every block once per query token. This producer runs one program +per (request, block) for all of the request's query tokens: rows are (token, head) pairs, each block is +dequantized once, and the causal limit is a per-row mask. + +It writes the same per-block partials (output scaled by 1/l and the V global scale, log2 LSE) as the per-token +producer with one block per chunk, so ``_merge_topk_attn_out_kernel`` merges them unchanged. +""" + +import triton +import triton.language as tl + +from .triton_sparse_decode import _dequant_nvfp4_rows + + +@triton.jit( + do_not_specialize=["num_reqs"], do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"] +) +def _gqa_sparse_decode_mq_kernel( + q_ptr, # [num_reqs * QLEN, num_heads, head_dim], FP8 or FP16/BF16 + k_ptr, # packed NVFP4 K: [num_pages, 1, page_size, head_dim / 2] uint8 + v_ptr, # packed NVFP4 V: same layout + k_sc_ptr, # K block scales [num_pages, 1, page_size, head_dim / 16], token-linear + v_sc_ptr, # V block scales, 4x4 token-quad order + k_gs_ptr, + v_gs_ptr, + t_ptr, # topk table [1, total_q, topk]; the request's last token's row is read + o_ptr, # partials [topk, total_q, num_heads, head_dim] fp32 + lse_ptr, # partials [topk, total_q, num_heads] fp32 (log2) + block_table_ptr, + seq_lens, + num_reqs, + sm_scale, + stride_qn, + stride_qh, + stride_k_blk, + stride_k_pos, + stride_v_blk, + stride_v_pos, + stride_ksc_blk, + stride_vsc_blk, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + QLEN: tl.constexpr, + GROUP: tl.constexpr, # query heads per KV head (16) + BLOCK_R: tl.constexpr, # next power of two of QLEN * GROUP + HEAD_DIM: tl.constexpr, + BLOCK_SIZE_K: tl.constexpr, + SCALE_COLS: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr, +): + pid_r = tl.program_id(0) + pid_c = tl.program_id(1) # top-k position == block index under identity selection + sm_scale_log2e = sm_scale * 1.4426950409 * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + + off_r = tl.arange(0, BLOCK_R) + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, HEAD_DIM) + off_dp = tl.arange(0, HEAD_DIM // 2) + off_s = tl.arange(0, SCALE_COLS) + row_t = off_r // GROUP + row_h = off_r % GROUP + r_mask = off_r < QLEN * GROUP + tok = pid_r * QLEN + row_t + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + pid_r) + # Token t of the request sits at seq_len - QLEN + t and attends positions [0, kv_len_t). + kv_len = tl.maximum(seq_len - QLEN + row_t + 1, 0) + blk = tl.load(t_ptr + (pid_r * QLEN + QLEN - 1) * stride_tn + pid_c * stride_tk) + blk_ok = blk >= 0 + blk = tl.where(blk_ok, blk, 0) + page = tl.load(block_table_ptr + pid_r * stride_bt_b + blk, mask=blk_ok, other=0).to(tl.int64) + # Rows past the last token's position are never read: their cache bytes may be stale (0 * NaN in the PV dot). + n_valid = tl.where( + blk_ok, tl.minimum(tl.maximum(seq_len - blk * BLOCK_SIZE_K, 0), BLOCK_SIZE_K), 0 + ) + n_mask = off_n < n_valid + + q = tl.load( + q_ptr + tok[:, None] * stride_qn + row_h[:, None] * stride_qh + off_d[None, :], + mask=r_mask[:, None], + other=0.0, + ).to(tl.float16) + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + page * stride_k_blk + off_n[:, None] * stride_k_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + page * stride_ksc_blk + off_n[:, None] * SCALE_COLS + off_s[None, :], + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ) + valid = ( + (blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None]) & n_mask[None, :] & r_mask[:, None] + ) + qk = tl.dot(q, tl.trans(k_deq).to(tl.float16)) * sm_scale_log2e + qk = tl.where(valid, qk, float("-inf")) + m = tl.max(qk, axis=1) + m_safe = tl.where(m == float("-inf"), 0.0, m) + p = tl.exp2(qk - m_safe[:, None]) + p_sum = tl.sum(p, axis=1) + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + page * stride_v_blk + off_n[:, None] * stride_v_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ).to(tl.float16) + acc = tl.dot(p.to(tl.float16), v) + # An empty partial stores zero with LSE -inf, so the merge gives it weight 0. + scale = tl.where(p_sum > 0.0, 1.0 / p_sum, 0.0) * v_global_scale + lse = tl.where(p_sum > 0.0, m + tl.log2(p_sum), float("-inf")) + acc = acc * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + tok[:, None] * stride_o_b + + row_h[:, None] * stride_o_h + + off_d[None, :], + acc, + mask=r_mask[:, None], + ) + tl.store(lse_ptr + pid_c * stride_l_c + tok * stride_l_b + row_h * stride_l_h, lse, mask=r_mask) + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + *, + sm_scale, + decode_query_len, + use_pdl, + early_trigger, + num_warps=8, + num_stages=1, +): + """Launch the multi-query producer; arguments as minimax_m3_sparse_attn_decode after its NVFP4 views.""" + total_q, num_heads, head_dim = q.shape + num_reqs = total_q // decode_query_len + max_topk = int(topk_idx.shape[-1]) + rows = decode_query_len * num_heads + pdl_launch = {"launch_pdl": True} if use_pdl else {} + _gqa_sparse_decode_mq_kernel[(num_reqs, max_topk)]( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + num_reqs, + sm_scale, + q.stride(0), + q.stride(1), + k_paged.stride(0), + k_paged.stride(2), + v_paged.stride(0), + v_paged.stride(2), + k_block_scale.stride(0), + v_block_scale.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + o_arg.stride(0), + o_arg.stride(1), + o_arg.stride(2), + lse_arg.stride(0), + lse_arg.stride(1), + lse_arg.stride(2), + block_table.stride(0), + QLEN=decode_query_len, + GROUP=num_heads, + BLOCK_R=triton.next_power_of_2(rows), + HEAD_DIM=head_dim, + BLOCK_SIZE_K=int(k_paged.shape[2]), + SCALE_COLS=head_dim // 16, + USE_PDL=use_pdl, + EARLY_TRIGGER=early_trigger, + num_warps=num_warps, + num_stages=num_stages, + **pdl_launch, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/m31_switches.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/m31_switches.py new file mode 100644 index 000000000000..51f966c595fd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/m31_switches.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Kill switches of this MiniMax-M3.1 target's default-on items (each target keeps its own copy: targets share +only the catalog). + +Every item is on by default; ``TRTLLM_M31_DISABLE_=1`` turns item ```` off and restores the path it replaced. +The environment is read once, at import, so every rank and every step of a process sees the same setting (a step must +not change its graph mid-run). Each item lists its id and what the switch restores where it reads it. + +Values (case-insensitive, surrounding blanks ignored): ``1`` / ``true`` / ``on`` / ``yes`` set a switch; empty / ``0`` / +``false`` / ``off`` / ``no`` leave it unset; anything else raises at import, naming the variable, so a typo fails loudly +instead of silently turning an item on or off. +""" + +import os +from typing import FrozenSet + +_PREFIX = "TRTLLM_M31_DISABLE_" +_SET = frozenset(("1", "true", "on", "yes")) +_UNSET = frozenset(("", "0", "false", "off", "no")) + + +def _is_set(name: str, value: str) -> bool: + """Whether the switch variable ``name`` is set to ``value``; raises on a value of neither kind.""" + word = value.strip().lower() + if word in _SET: + return True + if word in _UNSET: + return False + raise ValueError( + f"{name}={value!r}: expected 1 / true / on / yes (set) or 0 / false / off / no / empty (unset)" + ) + + +# The item ids switched off in this process, read once at import. +_DISABLED: FrozenSet[str] = frozenset( + name[len(_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_PREFIX) and _is_set(name, value) +) + + +def m31_disabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_DISABLE_`` turned the item off for this process.""" + return item_id in _DISABLED + + +# Items that are OFF by default (rule 1b without a passing gate c, or a failed gate c): ``TRTLLM_M31_ENABLE_=1`` +# opts in; the item's ``TRTLLM_M31_DISABLE_`` still wins. +_ENABLE_PREFIX = "TRTLLM_M31_ENABLE_" + +_ENABLED: FrozenSet[str] = frozenset( + name[len(_ENABLE_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_ENABLE_PREFIX) and _is_set(name, value) +) + + +def m31_enabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_ENABLE_`` opted a default-off item in for this process.""" + return item_id in _ENABLED diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py new file mode 100644 index 000000000000..921c65470472 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py @@ -0,0 +1,6396 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: MiniMax-M3.1 (MXFP8 + packed NVFP4 experts) / sm_107 / tp4 -- self-contained modeling code. + +MiniMax-M3.1's language model has MiniMax-M3's shape (60 layers, hidden 6144, 64 query heads and 4 KV heads of dim +128, 128 routed experts top 4 plus one shared expert) with a 1e7 RoPE base and MSA block-sparse attention on EVERY +layer (an index branch scores 128-token blocks and the top 16 are attended); layers 0-2 keep the dense SwiGLU-OAI MLP, +layers 3-59 the MoE. The checkpoint is an MXFP8 export whose routed experts are packed NVFP4 (compressed-tensors +names, global scales stored as multipliers; ``weights`` takes their FP32 reciprocals); the KV cache is NVFP4 with an +FP8 index-K on every layer. The layer kinds are read from the config (``sparse_attention_freq``, ``moe_layer_freq``), +so this code is the MiniMax-M3 target's with no dense-attention layer in use. + +``tp4`` is four Rubin GPUs (SM 10.7): head-split attention (16 query heads, one KV head and one index head per rank) and +tensor-parallel experts (every expert on every rank, intermediate 768 per rank), no attention DP. Speculative +decoding, when configured, is one-model Eagle3 (draft length 3): the BF16 drafter (``M3Eagle3Draft``) and the BF16 +target head, both at checkpoint precision; decode steps of up to 8 requests verify and draft through fused kernels +at that precision (``M3Eagle3Worker``). + +**The core** (``MiniMaxM3Tp4ModelingV2``) holds the weights (``weights.MODEL_WEIGHTS``) and the runtime resources the +steps use -- one MSA attention backend per layer, the TP all-reduce communicator, the RoPE table, the MXFP8 GEMM +dispatch state the engine tunes -- and dispatches each step, on the host, from its context count alone +(``_select_target``): + +* ``DecodeTarget`` takes every pure generation step. Up to ``DECODE_MAX_TOKENS`` tokens (up to 8 requests, each one + token or a draft verify of 1 + 3 rows) with 1-D input ids it runs the decode-step kernels and picks each layer + boundary's collective by the step's token bucket (``DECODE_PLANS``); it runs its own generic layer for the first + layer when the fused embedding does not take the step, for every layer whose output Eagle3 captures, and for every + layer of a wider step. +* ``PrefillTarget`` takes every step with context rows: prefill chunks and mixed steps, every layer in its generic + form over the MSA backend. + +The two targets share no body: each owns its layer, attention and MoE compositions and its ``step_args``, the one +projection of the step metadata its forward reads. What both call are the module-level op adapters, each picking the +kernel for one call from shapes and layouts (``_linear``, ``_qk_norm_rope``, ``_sparse_qkv_producer``, the router and +MoE-runner calls, the all-reduce + norm calls), which read no step metadata themselves. + +Every fast path the decode step can take, the rows each covers (``FAST_PATHS``, each paired with the entry whose +``CELLS`` certify it) and what a step that cannot take one does are below: **uncovered** (outside every covered +range: log once, run the generic path) or **missed** (inside a covered range but a precondition is missing: raise +``FastPathUnavailable`` under ``modeling_v2='require'``, else log once and fall back). + +**Hand-offs between kernels.** A producer that already holds an activation in registers writes its MXFP8 form for the +next MXFP8 projection: the decode boundary all-reduce for the next layer's QKV, the sparse-decode merge and the +long-context decode for the o_proj. It publishes it in ``kernels/m3_prequant.SLOT`` keyed by the BF16 tensor; the next +MXFP8 projection on a decode graph takes it (``_linear``), and every take clears the slot. The decode boundaries' +push targets (the o_proj / shared-expert GEMVs and the MoE engine writing straight into the boundary all-reduce's +exchange) are passed explicitly from the layer to the producer and back. + +**What this target asserts rather than adapts**: SM 10.3, the topology and checkpoint (``routing.py``); the +MiniMax-M3 MSA sparse attention runtime with the fused five-way QKV/index projection, an FP8 indexer and an FP8 or +NVFP4 KV cache, no torch.compile and no piecewise prefill graphs (``_validate_runtime``); text-only input (the vision +tower is not built and a step carrying image input raises). +""" + +from __future__ import annotations + +import atexit +import copy +import dataclasses +import os +import threading +import time +from dataclasses import dataclass +from types import MappingProxyType +from typing import ( + Any, + Callable, + Dict, + FrozenSet, + Hashable, + Iterable, + List, + Mapping, + Optional, + Tuple, + Union, +) + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch._experimental.modeling_v2._router_index import ( + ModelingV2Mode, + resolved_mode, + step_contract_enabled, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops, # noqa: F401 (registers the fake impls of the M3.1 torch ops) +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( # noqa: E501 + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_rmsnorm import ( + flashinfer_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.quantization.fp4_quantize import ( + fp4_quantize, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 import m31_pins +from tensorrt_llm._torch.attention.backends import AttentionMetadata +from tensorrt_llm._torch.attention.backends.fmha.msa_prefill import _aligned_nvfp4_dequant_scales +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionInputType, + PositionalEmbeddingParams, + RopeParams, +) +from tensorrt_llm._torch.attention.backends.prims_ts import _worktile_scope +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import MiniMaxM3MsaSparseAttention +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + MSA_REQUIRED_TOPK, + msa_paged_kv, + require_msa_module, + write_msa_phase_kv, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) +from tensorrt_llm._torch.attention.backends.utils import create_attention +from tensorrt_llm._torch.distributed import ( + AllReduce, + AllReduceFusionOp, + AllReduceParams, + AllReduceStrategy, +) +from tensorrt_llm._torch.distributed.ops import ( + MNNVLAllReduce, + get_or_scale_allreduce_mnnvl_workspace, +) +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + _ensure_draft_vocab_size, + _get_requested_draft_moe_backend, + _set_draft_kv_cache_quant_algo, + _slice_spec_position_ids, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.modules.embedding import LMHead +from tensorrt_llm._torch.modules.linear import ( + MXFP8LinearMethod, + TensorParallelMode, + flashinfer_mxfp8_autotune_active, + flashinfer_mxfp8_decode_graph_capture_active, +) +from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor +from tensorrt_llm._torch.modules.multi_stream_utils import maybe_execute_in_parallel +from tensorrt_llm._torch.moe.fused_moe.routing import RoutingMethodType +from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph import is_in_breakable_cuda_graph +from tensorrt_llm._torch.pyexecutor.h2d_cache import copy_h2d_if_changed +from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import MambaHybridCacheManager +from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy, RuntimeHooks +from tensorrt_llm._torch.pyexecutor.sampler.ops.flashinfer import set_top_p_renorm_override +from tensorrt_llm._torch.speculative import SpecMetadata +from tensorrt_llm._torch.speculative.eagle3 import Eagle3OneModelWorker +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + should_use_separate_draft_kv_cache, +) +from tensorrt_llm._torch.utils import ActType_TrtllmGen, is_torch_compiling +from tensorrt_llm._utils import mpi_rank, prefer_pinned +from tensorrt_llm.functional import PositionEmbeddingType +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.mode import QuantAlgo + +from . import dspark as _dspark +from . import weights as _weights +from .kernels import m3_attn_lc, m3_fold_tail, m3_mc_allreduce, m3_prequant, m3_report, m3_spec_tail +from .kernels import m3_draft_lowm as _draft_lowm_kernel +from .kernels import m3_draft_step as m3_draft_step_kernel +from .kernels import m3_drafter_attn as _drafter_attn_kernel +from .kernels import m3_fullctx_decode as _fullctx_decode_kernel +from .kernels import m3_index_score as _index_score_kernel +from .kernels import m3_index_score_t2sc as _index_score_t2sc_kernel +from .kernels import m3_sampled_accept as m3_sampled_accept_kernel +from .kernels import m3_select_long_dsl as _select_long_dsl_kernel +from .kernels import m3_shard_accept as m3_shard_accept_kernel +from .kernels import m3_target_head as m3_target_head_kernel +from .kernels import m3_tokmap_guard as _tokmap_guard +from .kernels import msa_kv_update as _msa_kv_update_kernel +from .kernels.long_select import long_select_blocks +from .kernels.mx_gemv import op as mx_gemv +from .kernels.triton_sparse_decode import MERGE_MXFP8, MERGE_MXFP8_MIN_TOKENS +from .m31_switches import m31_disabled, m31_enabled +from .ops import gqamix_sparse_decode as _gqamix_kernel +from .ops.activation.m3_silu_and_mul import m3_silu_and_mul +from .ops.attention.m3_dense_decode import m3_dense_decode +from .ops.attention.m3_dense_qk_norm_rope_fp8 import m3_dense_qk_norm_rope_fp8 +from .ops.attention.m3_draft_attention import m3_draft_attention +from .ops.attention.m3_fp8_indexer_qk_norm_rope import m3_fp8_indexer_qk_norm_rope +from .ops.attention.m3_fullctx_decode import m3_fullctx_decode +from .ops.attention.m3_index_score import SPLIT_K as INDEX_SCORE_SPLIT_K +from .ops.attention.m3_index_score import m3_index_score +from .ops.attention.m3_index_score_t2sc import m3_index_score_t2sc +from .ops.attention.m3_msa_attention import m3_msa_attention +from .ops.attention.m3_producer import m3_producer +from .ops.attention.m3_qk_norm_rope_fp8 import m3_qk_norm_rope_fp8 +from .ops.attention.m3_qkv_insert import m3_qkv_insert +from .ops.attention.m3_select_blocks import m3_select_blocks +from .ops.attention.m3_select_long import m3_select_long +from .ops.attention.m3_select_long_dsl import m3_select_long_dsl +from .ops.attention.m3_sparse_decode import m3_sparse_decode +from .ops.comm.m3_allreduce import m3_allreduce +from .ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm +from .ops.comm.m3_mc_finalize_allreduce_norm import m3_mc_finalize_allreduce_norm +from .ops.embedding.m3_embedding import m3_embedding +from .ops.gemm.m3_bf16_linear import m3_bf16_linear +from .ops.gemm.m3_dense_swiglu import m3_dense_swiglu +from .ops.gemm.m3_draft_lowm import ROWS as DRAFT_LOWM_TACTIC_ROWS +from .ops.gemm.m3_draft_lowm import m3_draft_lowm +from .ops.gemm.m3_fp32_linear import m3_fp32_linear +from .ops.gemm.m3_mx_gemv import ( + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, + m3_mx_gemv, +) +from .ops.gemm.m3_mx_gemv_bf16 import m3_mx_gemv_bf16 +from .ops.gemm.m3_mxfp8_gemm import m3_mxfp8_gemm +from .ops.gemm.m3_mxfp8_linear import m3_mxfp8_linear +from .ops.gemm.m3_mxfp8_pinned_gemm import m3_mxfp8_pinned_gemm +from .ops.gemm.m3_mxfp8_pinned_quantize import m3_mxfp8_pinned_quantize +from .ops.gemm.m3_target_head import MAX_ROWS as TARGET_HEAD_KERNEL_MAX_ROWS +from .ops.gemm.m3_target_head import m3_target_head +from .ops.moe.m3_moe16_engine import M3Moe16State, m3_moe16_engine +from .ops.moe.m3_moe_engine import M3MoeState, ar_push_desc, m3_moe_engine +from .ops.moe.m3_router_tc import m3_router_tc +from .ops.norm.m3_embedding_firstnorm import DecodeEmbedding, m3_embedding_firstnorm +from .ops.sampling import m31_topp_det as _topp_det +from .ops.spec.m3_draft_input import m3_draft_input +from .ops.spec.m3_draft_silu import m3_draft_silu +from .ops.spec.m3_draft_step import MAX_TOKENS as DRAFT_STEP_KERNEL_MAX_TOKENS +from .ops.spec.m3_draft_step import m3_draft_step +from .ops.spec.m3_force_vec import m3_force_vec +from .ops.spec.m3_force_vec_req import m3_force_vec_req +from .ops.spec.m3_shard_accept import m3_shard_accept +from .ops.spec.m3_spec_offsets import m3_spec_offsets +from .ops.spec.m3_step_prologue import m3_step_prologue + +_SM = (10, 7) # capacity: fixed: the device capability, not a size +# PROXYPB: compile the MSA indexer proxy's fmha_sm100 variants before warmup (``_prebuild_msa_proxy``). Compile and +# load only, no numerics change. TRTLLM_M31_DISABLE_PROXYPB=1: each variant compiles on its first call instead (which +# can be after the server is ready). +MSA_PROXY_PREBUILD = os.environ.get("TRTLLM_M31_DISABLE_PROXYPB", "0").strip() in ("", "0") +# R6: the dense-MLP layer boundary also publishes the next layer's MXFP8 QKV activation (as the MoE boundary does): +# M3.1's layers 1-3 follow a dense MLP but have the sparse five-way QKV. TRTLLM_M31_DISABLE_R6=1 restores the BF16 +# hand-off (the next QKV quantizes its own input). +# VR: OFF by default. On sm_107 the hand-off's QKV output is not bitwise against the BF16 path and +# R6 has no passing natural-acceptance gate (with R6 off, the gate passes). +# TRTLLM_M31_ENABLE_R6=1 turns it on (for its quality gate); TRTLLM_M31_DISABLE_R6=1 still forces it off. +_R6_ENABLED = os.environ.get("TRTLLM_M31_ENABLE_R6", "0").strip() not in ("", "0") +_R6_DISABLED = os.environ.get("TRTLLM_M31_DISABLE_R6", "0").strip() not in ("", "0") +R6_DENSE_PREQUANT = _R6_ENABLED and not _R6_DISABLED +# CTXNORMQ: a prefill layer boundary of at least ``CTXNORMQ_MIN_ROWS`` rows runs the plain all-reduce, then +# one kernel for the residual add + RMSNorm that also writes the next MXFP8 projection's activation (and, at a MoE +# boundary, the routed experts' NVFP4 input), instead of the fused all-reduce + norm followed by the consumers' own +# quantize passes. The quantized outputs equal mxfp8_quantize / fp4_quantize of the norm bit for bit; the norm itself +# is not bitwise against the fused all-reduce's (reduction order). Kill switch: TRTLLM_M31_DISABLE_CTXNORMQ=1. + +# PrimTS (the full-context decode) and FA4 trace in one process: scope task scheduling's WorkTileInfo rebind to the +# PrimTS traces (attention/backends/prims_ts/_worktile_scope.py). +_worktile_scope.arm() + +CTXNORMQ = os.environ.get("TRTLLM_M31_DISABLE_CTXNORMQ", "0").strip() in ("", "0") +# GQAMIX: a mixed ctx+gen step's sparse generation rows (MsaDecodeFmha) run the Triton block-sparse decode +# through ops/gqamix_sparse_decode, a verbatim copy of the in-tree kernel module whose two decode kernels do not +# specialize on the step's row count or on the [num_ctx:] slices' alignment, registered on MsaDecodeFmha's +# sparse_decode_hook. Those specializations compiled new variants after the server was ready. Kill switch: +# TRTLLM_M31_DISABLE_GQAMIX=1. +GQAMIX = not m31_disabled("GQAMIX") +# DSFOLD: under the standalone DSpark worker, the engine's previous-batch input gathers (input ids, draft tokens, +# position / KV-length offsets) run as one m3_spec_tail.step_gather launch instead of ~a dozen eager index and copy +# launches at the step boundary. Integer copies of the same values into the same buffers; the sampler keeps its +# in-tree slot stores. Kill switch: TRTLLM_M31_DISABLE_DSFOLD=1. +DSFOLD = not m31_disabled("DSFOLD") +CTXNORMQ_MIN_ROWS = ( + 1024 # capacity: NORMQ microbench (GB300): the fused pass wins at 1160 and 16384 rows +) +# -------------------------------------------------------------------------------------------------------------------- +# Row ranges +# -------------------------------------------------------------------------------------------------------------------- + +# The largest generation step the decode target takes: 8 requests verifying 1 + 3 drafts and their padding rows. +DECODE_MAX_TOKENS = 48 # capacity: fixed: the decode kernels' step bound +# VR-BANDS (MIDCONC): the decode step bound is 64 rows (8 requests verifying 1 + 7 drafts, the K7 c8 step) while every +# op keeps its own certified row range (DECODE_MAX_TOKENS and the per-op tables; the shard accept keeps its 48 rows): +# an op past its range runs its generic form and the rest of the step stays on the decode layer. Steps of 49..64 rows +# take the "wide" plan (the built-in all-reduce at both boundaries). TRTLLM_M31_DISABLE_VR_BANDS=1 restores the +# 48-row step bound. Read once, at import. +VR_BANDS = not m31_disabled("VR_BANDS") +DECODE_STEP_MAX = ( + 64 if VR_BANDS else DECODE_MAX_TOKENS +) # capacity: fixed: the step bound; ops keep their own rows + + +@dataclass(frozen=True) +class DecodePlan: + """The layer-boundary collectives of one decode token bucket (GB300 tp4 measurements in the comments).""" + + name: str + max_tokens: int + # The attention boundary's multicast all-reduce + residual + RMSNorm (5.05 vs 5.27 us native at 1 token). + attn_mc_ar: bool + # The same kernel at the MoE boundary with the shared + routed add folded in: wins up to 48 tokens. + moe_mc_ar: bool + # The MoE-boundary all-reduce also emits the next QKV activation in MXFP8 for the decode GEMV / GEMM. + qkv_prequant: bool + + +# tiny: every step up to 8 tokens (c1..c8 without speculation, the c1 verify, every draft pass); small: the c2 +# verify; verify: the c4 / c8 verifies. Each is one multicast all-reduce bucket (kernels/m3_mc_allreduce.BUCKETS). +# SM 10.7 TP4: one bucket. MC_AR_TP4 puts every decode boundary on the multicast all-reduce + residual + RMSNorm +# (trtllm::minimax_m3_sm107_tp4_mc_allreduce_norm, its row kernel: bitwise to the stock TP4 oneshot all-reduce fusion +# on the residual and the normed rows) with the MXFP8 copy for the next QKV; False = the built-in all-reduce. +MC_AR_TP4 = True +# Up to this many rows the decode all-reduces (not the MoE finalize) take the 8-CTA cluster kernel instead of the row +# kernel: the same stock oneshot order (opts bit 32: per-column sum of squares, warp butterflies, 6-warp groups, 4 group +# sums from 0.f), so bitwise the row kernel's output, with each row's exchange spread over 8 CTAs. Rubin TP4 synth, both +# orders: c1 (4 rows) -12, c2 (8 rows) -38 us/step, but c4 / c8 (16 / 32 rows) +5 / +45, where +# 8x the CTAs per row cost more than they save. 0 = the row kernel at every row count. +MC_AR_CLUSTER_MAX_ROWS = 8 # capacity: fixed: measured cluster-kernel crossover +# Every decode / drafter boundary all-reduce launches with the max-smem carveout preference (the cluster kernel's +# option 8 at every row count, the row kernel's carveout instance), so the next kernel's CTAs (the QKV GEMV's staged +# tiles) can co-reside early. Launch attribute only: bitwise unchanged. Kill: False (the GB option table). +MC_AR_CARVEOUT = True +# The fused draft loop's boundary all-reduces (M3Eagle3Worker._m3_draft_ar_norm) launch WITHOUT that carveout: the +# kernel after the drafter's last boundary is the draft step (one CTA per SM, ~145 KB of shared memory) rather than a +# staged GEMV, and TP4 alone of the sm_107 targets ran its draft pass ~74-80 us in-model vs ~57 us standalone at world 4 +# in the step profile. Launch attribute only: bitwise unchanged. Kill: MC_AR_CARVEOUT (the shared setting). +DRAFT_MC_AR_CARVEOUT = False +# The fused draft loop's two all-reduce boundaries per pass (o_proj -> post-attention norm, down -> final norm) of up +# to this many rows run the multicast all-reduce + residual + RMSNorm's row kernel in its drafter form (opts bit 16: +# FlashInfer fused_add_rmsnorm's reduction order on the unrounded FP32 sum, so bitwise the drafter's stock plain +# all-reduce + RMSNorm) instead of the built-in all-reduce and the separate norm. Kill: 0. +DRAFT_MC_AR_MAX_ROWS = ( + 8 if MC_AR_TP4 else 0 +) # capacity: fixed: the drafter boundary kernel's certified rows +# Those drafter boundaries run the 8-CTA cluster kernel's drafter form (opts bit 128; TP8's drafter boundary kernel) +# instead of the 1-CTA row kernel. Exact-class: deterministic, the updated residual bitwise, the RMS sum of +# squares in the cluster's order instead of FlashInfer's, so a normed row can differ by one BF16 rounding step (7/64 +# rows at 1 row, 14/256 at 4). Rubin world 4, both orders: 4.39 -> 3.75 us per boundary at 1 row, +# 4.79 -> 4.14 at 4 rows, ~ -2..-3.5 us/step. Kill: False (the row kernel, bitwise the stock boundary). +# OFF: in-model timing was flat and natural acceptance length was grey (lower bound -1.16% vs the -1% line). +DRAFT_MC_AR_CLUSTER = False +# The drafter's o_proj and down projection, on the copied split-K kernel (kernels/m3_lowm_splitk), also push their rows +# into the drafter boundary all-reduce's exchange (the row kernel's slots: the canonical words its own push would store, +# at the (token, 768-column slice) call parity), which then runs prepushed 2 instead of pushing x itself; the copy +# releases its dependents only after its own grid wait and the pushes (the prepushed >= 2 contract). Bitwise. The TP8 +# port: ~ -4 us/step at world 8. Kill: False. +DRAFT_AR_PUSH = True +# The drafter boundary all-reduce fed by its projection's push (DRAFT_AR_PUSH): its own taken / not-taken lines. +DRAFT_AR_PUSH_PATH = "m3_draft_ar_push" +DECODE_PLANS = ( + DecodePlan( + "tiny", DECODE_MAX_TOKENS, attn_mc_ar=MC_AR_TP4, moe_mc_ar=MC_AR_TP4, qkv_prequant=MC_AR_TP4 + ), +) +if DECODE_STEP_MAX > DECODE_MAX_TOKENS: + # VR-BANDS: 49..64-row steps (K7 c7..c8) with the built-in all-reduce at both boundaries. + DECODE_PLANS += ( + DecodePlan("wide", DECODE_STEP_MAX, attn_mc_ar=False, moe_mc_ar=False, qkv_prequant=False), + ) + +# The dense layers' decode attention (trtllm-gen, msa_decode.dense_decode_max_seq_len): the max_seq_len it is handed +# per request count. It only sizes the multi-CTA KV split (every CTA still covers its share of the actual length; the +# result changes only by the split-merge order), and the model maximum over-splits long-context caches at one +# request: GB300 TP4, 16 heads, FP8 KV, 64K context: 15.1 us at 16K vs 28.6 us at 1M. Two requests are neutral (the +# grid is unchanged) and keep the step's bound. +# The generic layer's boundaries (prefill, mixed steps, Eagle3 capture layers): the most rows that take the multicast +# all-reduce + residual + RMSNorm, alone and with the shared + routed add folded in. TP4 skeleton: none (the built-in +# all-reduce) until the multicast all-reduce is certified on Rubin. +GENERIC_MC_AR_MAX_TOKENS = 0 # capacity: fixed: off at TP4 (the built-in all-reduce) +GENERIC_MC_AR_FOLDED_MAX_TOKENS = 0 # capacity: fixed: off at TP4 (the built-in all-reduce) +# An Eagle3 capture layer whose capture norm is armed (CAPTURE_NORM) has its fused feed-forward boundary all-reduce +# (the 8-CTA cluster kernel up to MC_AR_CLUSTER_MAX_ROWS, the row kernel above) also write the capture: +# rmsnorm(updated residual) * fc_norm[i], bit for bit the capture_norm kernel, which then does not run. The MoE +# finalize runs the shared GB300 op and keeps the capture_norm kernel. Wired at the generic layer's fused boundary +# (layers.M3DecoderLayer._feed_forward_boundary) and through M3DecodeTarget._layer's boundaries (``capture``) for a +# capture layer on the fast decode layer. Kill: False (the capture_norm kernel after the boundary). +CAPTURE_NORM_IN_BOUNDARY = True + +# TP4 skeleton: no cap (the built-in split) until a TP4 SM 10.7 cap is measured and certified. +DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH: Mapping[int, int] = MappingProxyType({}) + +# The dense layers' decode attention on PrimTS split-KV decode (kernels/m3_fullctx_decode) instead of the trtllm-gen +# call, per (query rows per request, requests): the Q tile (``config``) and the split-KV fanout (``splits``) with the +# BF16 merge. At TP4 (16 query heads, 1 KV head per rank) the stock split is capped at 16 cluster CTAs per (request, +# KV head), i.e. 16 of 212 CTAs at one request: stock dense fmhaSm107a 33.4-33.8 us per call and the stock drafter +# fmhaSm100f 45.4 us per pass at 165k. Rubin TP4, FP8 KV, 165k, chained us per call +# (microbench), every recipe passing the fp32 oracle: (4, 1) 13.70 (stock 33.6), (1, 1) 11.52. At one +# request the K/V stream is latency-bound, so both deepen the K/V load pipeline (the load pipeline only; the same +# bits): 12 stages at one query row, 8 at four (the long-context sweep: (4, 1) 12.95 / 17.00 / 20.98 us +# per call at 165k / 323k / 512k with 8 stages vs 13.03 / 17.15 / 21.30 with 12; (1, 1) is fastest with 12). Two +# requests (microbench; stock at c2: dense fmhaSm107a 26.5 us, drafter fmhaSm100f 44.0-44.6 us per pass, +# in-model): (4, 2) 17.34, (1, 2) 14.16. +# K/V stages in flight of the (4, 1) recipe (12 was the previous depth; the same bits at any depth). +FULLCTX_DECODE_Q4_KV_STAGES = 8 # capacity: fixed: a pipeline depth, not a size +FULLCTX_DECODE_RECIPES: Mapping[Tuple[int, int], Mapping[str, Any]] = MappingProxyType( + { + (4, 1): MappingProxyType( + { + "config": { + "tile_size_q": 32, + "groups_tokens_heads_q": True, + "kv_stages": FULLCTX_DECODE_Q4_KV_STAGES, + }, + "splits": 96, + } + ), + (1, 1): MappingProxyType({"config": {"kv_stages": 12}, "splits": 108}), + (4, 2): MappingProxyType( + {"config": {"tile_size_q": 32, "groups_tokens_heads_q": True}, "splits": 48} + ), + (1, 2): MappingProxyType({"config": {}, "splits": 80}), + } +) # capacity: fixed: pinned recipes per certified (rows, requests) +# Kill constant: False puts the dense layers back on the trtllm-gen call at every shape. +FULLCTX_DECODE_ENABLED = True +# The dense layers' FP8 q/k/v (decode steps of the hybrid cache): the op minimax_m3_sm107tp4_dense_qk_norm_rope_fp8 +# (head-count generic) rounds Q/K to BF16 and then to E4M3 in its store, the bits of the BF16 fused_qk_norm_rope output +# cast to FP8. False (or a build without the op) keeps the single-rounding fused_qk_norm_rope FP8 output. +DENSE_QKV_FP8_VIA_BF16 = True +# On a full-context decode step that op also stores the new K/V rows into the layer's paged pool (the bytes +# write_msa_phase_kv's indexed copies store, one launch instead of about ten). False writes them with +# write_msa_phase_kv. +DENSE_KV_INSERT_FUSED = True +# The engagement line of that fused store (taken, one per row count and rank). +DENSE_KV_INSERT_PATH = "m3_dense_kv_insert" +# The drafter MLP's C++ SwiGLU (DRAFT_SILU_CPP, M3Eagle3Draft.mlp): its own taken / not-taken lines. +DRAFT_SILU_PATH = "m3_draft_silu" +# The split-KV merge on the BF16 op (minimax_m3_sm107tp4_fullctx_reduce, head-count generic) instead of PrimTS's FP16 +# reducer and a cast. +FULLCTX_DECODE_OWN_REDUCER = True +# The merge also writes the MXFP8 copy of its BF16 rows (E4M3 + 128x4-swizzled UE8M0, bit for bit +# trtllm::mxfp8_quantize) into the MERGE_MXFP8 buffers and publishes it to m3_prequant.SLOT, so the dense layer's +# MXFP8 o_proj skips its own quantization. Kill: False leaves the o_proj to quantize its BF16 input. +FULLCTX_DECODE_MXFP8 = True +FULLCTX_DECODE_MXFP8_PATH = "m3_fullctx_merge_mxfp8" +# The K/V length bound the full-context plans are compiled for; a step whose cache length bound is longer is a miss. It +# covers the long-context serving config's max_seq_len (1 << 20) plus the speculative rows the cache reserves past it. +FULLCTX_DECODE_MAX_KV_LEN = ( + 1 << 21 +) # capacity: fixed: the plans' KV-length bound (1M context + spec rows) +# The Eagle3 drafter layer's generation phase on the same full-context decode (kernels/m3_drafter_attn: the in-tree +# RoPE + FP8 K/V append, then the pinned PrimTS recipe) instead of thop.attention's in-tree runner, which caps its +# split at 8 CTAs per (request, KV head) on SM 10.7: 45.4 us per draft pass at 165k. Kill constant. +DRAFTER_FULLCTX_ENABLED = True +# On those passes the drafter's RoPE, FP8 query and FP8 K/V append run as one launch (the op +# minimax_m3_sm107tp4_drafter_rope_fp8_kv: the in-tree preprocess's GPT-NeoX rotation and unit-scale E4M3 store, and +# the torch E4M3 cast of its BF16 query) instead of thop.trtllm_gen_generation_preprocess and the cast. Kill constant. +DRAFTER_ROPE_FUSED = True +# The engagement line of that launch (taken, one per row count and rank). +DRAFTER_ROPE_PATH = "m3_drafter_rope" +# thop.trtllm_gen_generation_preprocess builds its workspace token map only for several requests, yet a one-request +# multi-token generation (the drafter's first pass at one request, any stock trtllm-gen fallback) places its rows by +# that map: without a rebuild it applies the last multi-request call's map (rows on another request's length and pages, +# token-0 drafts). kernels/m3_tokmap_guard writes the one-request map (0, t) into the op's map slice right before such +# calls: what the build writes (padded rows included), so bitwise the op with a fresh map. Kill: False. +PREPROCESS_TOKMAP_FIX = True +PREPROCESS_TOKMAP_PATH = "m3_preprocess_tokmap" +# How the rotation's products are summed, as the in-tree preprocess build sums them (0: two products then the add, +# 1: fma(cos, x, sin * pair), 2: fma(sin, pair, cos * x)); the op's bitwise test pins it. +DRAFTER_ROPE_MODE = 1 # capacity: fixed: a launch mode, not a size + +# The sparse layers' NVFP4 QKV + indexer producer (ops/attention/m3_producer, every step its geometry fits) triggers its +# dependent kernel on entry, so the index scorer launches and stages its pre-wait loads while the producer runs (its +# outputs are read only after the scorer's own grid wait). +PRODUCER_EARLY_TRIGGER = True +# Steps of up to this many rows keep the producer's trigger at its exit instead. On Rubin TP4 the early-launched scorer +# (its grid doubles with the requests, its pre-wait index-K prefetch running) is resident through the producer's whole +# post-QKV segment and slows the producer's dependent loads; from 16 rows the overlap pays. In-model, locked memory +# clock, tactic- and FlashInfer-pinned, A and B in one job: -75 / -176 us per decode step at 1 / 2 requests, +# +67 at 4 with the trigger at the exit. Timing only: the scorer reads the producer's outputs after its own grid wait. +# Kill: 0 (the early trigger at every row count). Set to 0 on the shipped stack: with the long-row select's late release +# and post-load trigger, the scorer's pre-wait overlap with the producer is worth more than the producer's own +# segment (both orders, double-pinned: -12 / -42 us at 1 / 2 requests with 0). +PRODUCER_LATE_TRIGGER_MAX_ROWS = 0 # capacity: fixed: measured trigger crossover (0: off) +# The producer reads its per-step inputs (position ids, cache slots, RoPE rows) before its grid wait, leaving only the +# packed QKV row after it: this target's copy of the producer (trtllm::minimax_m3_sm107_tp4_producer_pw, +# minimaxM3Sm107Tp4/producerPw4.cu, the TP2 producerPw2 at 16 query heads). Their writers are host-to-device copies +# before the replay and plain-launched kernels earlier in the step (the input preprocessing, msa_kv_update or its torch +# fallback), so they are final before the producer can start; a PDL-launched writer of any of them would have to turn +# this off. The call site's packed.contiguous() and position-id copies are plain launches too. Bit-identical outputs. +# Kill: False (the shared minimax_m3_producer_pw at 16 query heads, which reads them after its wait). +PRODUCER_PREWAIT_STEP_INPUTS = True + +# The fused embedding + first RMSNorm kernel: one owner per row count and index dtype. +EMBED_FIRSTNORM_MAX_TOKENS = 8 # capacity: fixed: built rows; wider steps embed generically +# The o_proj pushes its rows into the attention boundary's multicast all-reduce (prepushed mode 2) up to this many +# tokens; wider steps run the GEMV and a plain multicast all-reduce. +ATTN_PUSH_MAX_TOKENS = 8 if MC_AR_TP4 else 0 # capacity: fixed: measured push crossover +# The dense layers' MLP down projection pushes its rows into the post-MLP boundary's multicast all-reduce (prepushed +# mode 2), as the o_proj does into the attention boundary (TP2: those hops end 5.5-5.8 us after the down +# GEMV unpushed vs 2.4-2.5 us pushed). Bitwise equal to the unpushed boundary (the same bf16 words). Kill: False. +DENSE_PUSH_TP4 = True +DENSE_PUSH_MAX_TOKENS = ( + 8 if (MC_AR_TP4 and DENSE_PUSH_TP4) else 0 +) # capacity: fixed: the GEMV push rows +AR_PUSH_MODE = 2 # capacity: fixed: an exchange mode, not a size +# The 1..8-token MoE engine (router, routed experts and the shared expert in one launch). It pushes its sum into the +# MoE boundary's multicast all-reduce (prepushed mode 2) on the rows MOE_PUSH covers (with MC_AR_TP4). +MOE_ENGINE_MAX_TOKENS = 8 # capacity: fixed: the engine's built rows +MOE_PUSH_MAX_TOKENS = ( + MOE_ENGINE_MAX_TOKENS if MC_AR_TP4 else 0 +) # capacity: fixed: the engine's rows +# The 9..13-row engine (router, routed and shared experts in one kernel); its bf16 sum goes to the boundary. +MOE16_MIN_TOKENS = 9 # capacity: fixed: the 9..13-row engine's lower bound +MOE16_MAX_TOKENS = 13 # capacity: fixed: the 9..13-row engine's built rows +# From this many tokens the MoE runs unfinalized through the TRTLLM-Gen runner and its finalize (top-k gather, +# routing weights, shared expert add) runs inside the boundary all-reduce. 9..13 take it only when their engine is +# missing (outside require mode). TP4 skeleton: no rows (the built-in MoE finalize and all-reduce). +MC_FINALIZE_MIN_TOKENS = 9 # capacity: fixed: the rows above the engines +MC_FINALIZE_MAX_TOKENS = 0 # capacity: fixed: off at TP4 +# A greedy Eagle3 verify step of up to this many requests verifies with the fused shard argmax + acceptance kernel +# over the target lm_head's BF16 shard logits (kernels/m3_shard_accept), at the configured draft length. +SHARD_ACCEPT_DRAFT_LEN = 3 # capacity: fixed: the built Eagle3 draft length +SHARD_ACCEPT_MAX_REQUESTS = DECODE_MAX_TOKENS // ( + SHARD_ACCEPT_DRAFT_LEN + 1 +) # capacity: fixed: decode rows / (draft length + 1) +# A sampled (temperature + top-p, advanced_sampling_mode=no_topk) Eagle3 verify step of up to this many requests +# verifies with the fused sampled acceptance (kernels/m3_sampled_accept) over this rank's logits shard: every request +# count a decode step holds. +SAMPLED_VERIFY_MAX_REQUESTS = SHARD_ACCEPT_MAX_REQUESTS # capacity: fixed: a decode step's requests +# The sampled verify runs the sm_107 tp4 copy of the TP2 sampled-accept kernel +# (trtllm::minimax_m3_sm107tp4_sampled_accept: a row's CTAs launch as one thread-block cluster, bounded record polls). +# Same records and tokens. Kill: False (the shared kernel, trtllm::minimax_m3_sampled_accept: every CTA of the grid +# co-resident, unbounded polls). +SAMPLED_ACCEPT_SM107TP4 = True +# The sm_107 tp4 sampled verify loads each row's sampling parameters (temperature, top-p, Philox seed and offset: step +# inputs, not the head's output) before its PDL dependency wait, overlapping the head GEMM's tail; the same values and +# tokens. Kill: False (they load after the wait; the shared kernel always loads them after). Rubin TP2 microbench: +# verify end 20.48 -> 19.97 us at top-p 0.95. +SAMPLED_ACCEPT_PREWAIT_PARAMS = True +# The sm_107 tp4 sampled verify releases its dependents (the first draft pass's input) right after its dependency wait +# instead of at the acceptance epilogue: the dependent launches early and parks at its own griddepcontrol.wait, which +# returns only once the verify has completed, so the values are the same; it saves the launch latency after the verify +# (TP2: draft_input started 2.0-2.4 us after the verify / fc join). Kill: False. Off until its sampled-column +# in-model pairs read. +SAMPLED_ACCEPT_EARLY_TRIGGER = False +# SA-1R: the sm_107 tp4 sampled verify also multicasts each CTA's largest logits with its round-0 record (82: each +# thread's top 2 pooled, the CTA's top 8 plus the bound of everything it does not list; 9: an exact top-9 list), so a +# row whose first candidate is provably in the nucleus decides in round 0 instead of exchanging a second round. The +# same tokens, accepted counts and Philox draws as the two-round kernel, bit for bit (a fp32 margin m * 2^-21 * S +# covers the summation order; any doubt runs round 1). The prototype was exact on 39,968 rows; real +# long-context serving logits decide all 4 rows in round 0 on 79.5% (82) / 80.8% (9) of steps. Off (0, the two-round +# kernel): exact, but the untimed decode step reads +47 / +65 us slower with it on at TP4 (the earlier dependent +# launch it enables costs more than the saved round). +SAMPLED_ACCEPT_1R_K = 0 # capacity: fixed: a kernel variant, not a size +# The sm_107 tp4 sampled verify's owner walk (the one thread that finds its row's drawn token by a serial +# prefix over its strided tokens) takes those tokens 8 at a time: their weights are computed independently, then +# summed in the same order with the same early exit, so the draw, tokens and Philox draws are the one-token walk's, +# bit for bit. Rubin TP4 bench (one request, top_p 0.95): -1.06 / -1.07 us per call at SA-1R K 0 / 82. Kill: False +# (the one-token walk, the stock instantiation; the shared kernel has no 8-token walk). +# Off until its in-model TP4 pair passes. +SAMPLED_ACCEPT_ILP_WALK = False +# A step verified by the in-tree path takes the fused first drafter inputs (the capture norms and the first pass's +# gather) up to this many requests: every request count a decode step holds. +FIRST_DRAFTER_INPUTS_MAX_REQUESTS = SHARD_ACCEPT_MAX_REQUESTS # capacity: fixed: step requests +# A fused-verify step of up to this many verify rows (one or two requests at draft length 3) computes the target +# logits shard from the 12-bit coded copy of the BF16 head (kernels/m3_target_head; the same BF16 x BF16 -> fp32 +# arithmetic, 75% of the bytes) instead of the in-tree head GEMM; wider verifies keep the in-tree GEMM. +# Off at TP4 (as at TP2): at the TP4 shard (50,016 x 6144) the in-tree BF16 cuBLAS head is faster than the coded kernel, +# 47.9 vs 56.8 us at 4 and 8 rows (TP2: 93 vs 109). True re-enables the coded head. +TARGET_HEAD_CODED = False +TARGET_HEAD_MAX_ROWS = ( + TARGET_HEAD_KERNEL_MAX_ROWS if TARGET_HEAD_CODED else 0 +) # capacity: fixed: the coded head's built rows (off at TP4) +# On a fused-verify step of up to this many verify rows the drafter's first fc is forked onto the aux stream right +# after the last captured target layer, overlapping the remaining target layers and the head (M3Eagle3Worker). +FC_EARLY_MAX_ROWS = 0 # capacity: fixed: measured fork crossover (0: off) +# A sampled fused verify of up to this many verify rows on the in-tree head GEMM forks the drafter's first fc after the +# head instead of beside it (M3Eagle3Worker), as the coded head's verify already does: the fc's split-K grid otherwise +# holds the SMs first and the head GEMM starts late (TP2: 17.4 us late). Kill: 0. +FC_AFTER_HEAD_MAX_ROWS = 8 # capacity: fixed: measured fork-after-head crossover +# A decode-only Eagle3 step of up to this many requests drafts with the fused draft loop (M3Eagle3Worker): per pass the +# draft input in one launch (kernels/m3_spec_tail.draft_input), the drafter layer's in-tree attention, MLP and +# all-reduces, and one kernel for the draft head (its 12-bit coded copy, or the BF16 fragments when the coding does +# not decode bit for bit), the cross-rank argmax and the pass's bookkeeping (kernels/m3_draft_step), at the +# configured draft length. The head kernel takes one row per request, at most 8. +# The fused draft loop streams the 12-bit coded copy of the draft head (when it decodes bit for bit); False keeps the +# BF16 fragments. +DRAFT_HEAD_CODED = False +DRAFT_STEP_DRAFT_LEN = SHARD_ACCEPT_DRAFT_LEN # capacity: fixed: the built draft length +DRAFT_STEP_MAX_REQUESTS = DRAFT_STEP_KERNEL_MAX_TOKENS +# The Eagle3 capture layers (decoder layers whose output the drafter reads) run the fast decode layer, with its +# o_proj / MoE / dense-down pushes into the boundary all-reduces, and capture the MoE / dense boundary's reduced +# residual sum (the same BF16 words the generic layer's fused boundary captures). Kill: False (the generic layer, +# no boundary pushes). Rubin TP2: the capture layers' hops end 4.4-5.3 us after their producing +# GEMV unpushed vs 2.4-2.5 us pushed. +CAPTURE_LAYERS_FAST = True +# An Eagle3 capture layer (one with a next norm) ends at the fused boundary (all-reduce + residual + next RMSNorm in +# one kernel) and captures that kernel's reduced residual sum. Kill: False keeps the feed-forward module's own +# all-reduce, captures hidden + residual and runs the next norm separately, as the built-in model does; it also +# turns CAPTURE_LAYERS_FAST off, which captures the fused boundary's sum. +CAPTURE_FUSED_BOUNDARY = True +# The Eagle3 drafter's BF16 linears (qkv, o_proj, gate_up, down, fc) run FlashInfer's fixed-tactic low-M kernels +# (ops/gemm/m3_draft_lowm, the TP2 target's port with the TP4 table) at the rows its table is certified for, up to this +# many rows; other calls keep the in-tree cuBLAS GEMM (split-K + reduce). Kill: 0. Rubin TP4 sweep (locked memory +# clock) at 1 row: qkv 8.15 -> 5.90, o 5.36 -> 3.40, gate_up 12.93 -> 9.93, down 7.68 -> 6.16 us. +DRAFT_LOWM_MAX_ROWS = 4 # capacity: fixed: the low-M tactic table's certified rows +# The drafter MLP's activation (the plain SwiGLU of its in-tree GatedMLP) runs this target's C++ op +# (ops/spec/m3_draft_silu, minimaxM3Sm107Tp4/siluAndMul.cu) instead of the Triton silu_and_mul: bit for bit, one launch +# either way (no Triton on the generation decode path). The fused draft loop's MLP (M3Eagle3Draft.mlp) routes the +# call; the in-tree draft path keeps the Triton op. Kill: False. +DRAFT_SILU_CPP = True +DRAFT_LOWM_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = { + nk: frozenset(m for m in rows if m <= DRAFT_LOWM_MAX_ROWS) + for nk, rows in DRAFT_LOWM_TACTIC_ROWS.items() +} +# The identity-selection sparse decode moves from the per-row producer to the multi-query producer at this many rows +# (kernels/triton_sparse_decode.MQ_MIN_TOKENS). +SPARSE_DECODE_MQ_MIN_TOKENS = 17 # capacity: fixed: the multi-query crossover +# The long-row top-16 selector takes engines of 129..LONG_SELECT_MAX_BLOCKS KV blocks of 128 tokens; it holds 2, 4 or +# 9 keys per thread by the row's valid-block count. At most 128 blocks the built-in selector runs. +LONG_SELECT_MIN_BLOCKS = 129 # capacity: fixed: past the built-in selector's 128 +LONG_SELECT_MAX_BLOCKS = 0 # capacity: fixed: off at TP4 (the DSL select takes the long rows) +LONG_SELECT_KEY_WIDTHS = (2, 4, 9) # capacity: fixed: compiled keys per thread +# The long-row select releases the sparse decode after it right after its own grid wait (the decode stages q and the +# request length before its wait) up to this many query rows. On Rubin TP4 the early-launched attention CTAs crowd the +# select and the o_proj GEMV already at 2 requests (8 rows), so the early release is off at every row count: in-model, +# locked memory clock, pinned tactics, long-context draws, 2 instances vs the stored baseline, the replay period with +# the trigger phase below is -209 us per step at 2 requests (mean of draws), -39 at 4 and -12 at 8. +SELECT_EARLY_MAX_ROWS = 0 # capacity: fixed: measured early-release crossover (0: off) +# The long-context sparse decode runs this target's copy of the op (trtllm::minimax_m3_sm107_tp4_sparse_decode_lc, +# minimaxM3Sm107Tp4/sparseDecodeLc4.cu): its split-K merge forms the 16 weights once per CTA (16 threads, through shared +# memory) instead of in each of the 128 threads. Same values in the same order: bitwise the image op's output. The +# SM107 TP2 sparse decode's -0.17 / -0.23 us per layer at 1 / 2 requests. Kill: False (the image op). +SPARSE_DECODE_LC_MERGE_ONCE = True +# With SPARSE_DECODE_LC_MERGE_ONCE, that op's CTAs also pull the layer's o_proj weight and scales into L2 before +# their grid dependency (cp.async.bulk.prefetch.L2, a hint: no value changes), so the o_proj GEMV after it finds them +# L2-warm. The o_proj's own pre-wait staging streams part of them already; the cold-vs-hot GEMV gap left is ~0.18 us +# per layer. Kill: False. +SPARSE_DECODE_LC_PREFETCH_OPROJ = True +# With SPARSE_DECODE_LC_MERGE_ONCE, that op also records each step's selected block ids per layer and pulls into L2 +# the K/V pages + scales of the blocks the NEXT sparse layer selected at the previous step (65-68% of a layer's +# distinct selected blocks recur one step later, measured on all 4 ranks), so that layer's sparse decode reads +# them L2-warm. Hints only; up to N1_MAX_ROWS query rows. Kill: False. +SPARSE_DECODE_LC_PREFETCH_NEXT_KV = True +N1_MAX_ROWS = 128 # capacity: fixed: the next-KV id buffer's rows +# With SPARSE_DECODE_LC_MERGE_ONCE, short-sequence identity steps (every request within 16 blocks, +# MSA_SHORT_SEQUENCE_IDENTITY) also run that op instead of the Triton producer: CTA r attends block r of its request +# when r < min(16, blocks), taken from the device sequence length (the table is not read, so a host-staged identity +# table that misses the in-flight accepted tokens cannot drop a block). Exact-class like every LC row (band vs the +# fp64 reference); removes the Triton decode from the GEN path at TP4. Kill: False (the Triton identity producer). +# Identity steps skip the select, so the producer's entry trigger used to release LC4's pre-wait q read before the +# producer wrote q (NUL tokens in GSM8K outputs). LC4 reads q after the grid wait, which clears that race (0 NUL). +# Matched ON-vs-OFF GSM8K pairs, greedy and sampled, show no accuracy change; acceptance length +0.024. +SPARSE_DECODE_LC_IDENTITY = True +# Up to SELECT_LONG_TRIGGER_MAX_ROWS query rows and without the early release, the long-row select releases the sparse +# decode once every score of its row is loaded (SELECT_LONG_TRIGGER_PHASE 1, before the exact rank of the candidates) +# instead of at its exit (0, the kill value); wider steps keep the exit. The release still follows the select's own +# grid wait, which the sparse decode's pre-wait q / length / block-table reads rely on. +SELECT_LONG_TRIGGER_PHASE = 1 # capacity: fixed: a trigger phase, not a size +SELECT_LONG_TRIGGER_MAX_ROWS = 8 # capacity: fixed: measured trigger crossover +# The long-row select ranks a row's candidates on warp 0 when they number at most 64 (a 64-key bitonic network; the +# other warps exit) and reads the row's valid count before its grid wait; the picks are the same. Kill: False (the +# all-thread rank). In-model, locked memory clock, double-pinned, A and B in one job, both orders: flat within +# noise (mean of draws -5 / -60 us at 1 / 2 requests; replay period at the 323K request +14 / +54); a FLAT-but-exact +# stack item (post-wait select time 1.76 -> 1.54 us per layer at 1 request, 165K tokens). OFF by default: at TP2 the +# warp-0 tail runs ~0.95 us/layer slower in-model next to the early-launched sparse decode (+53 us/step); +# on at TP4 only after an in-model A,B shows a gain. +SELECT_LONG_SL3 = False +# The long-row select runs as kernels/m3_select_long_dsl (ops/attention/m3_select_long_dsl): one 384-thread CTA per row +# instead of one 1024-thread CTA, the same selection bit for bit, with 4, 11 or 24 keys per thread +# (LONG_SELECT_DSL_KEY_WIDTHS) and the same trigger choices (SELECT_LONG_SL3 does not apply). TP4 has the TP8 target's +# per-rank select geometry (1 KV head, 4 rows per request), where on the Rubin decode chain (57 layers behind the +# producer stream, 1 / 2 requests) the select is -0.15 / -0.12 us per layer at 165K tokens, 0.00 / -0.01 at 323K and +# -0.04 / -0.04 at 500K against the 1024-thread kernel. Kill: False (the 1024-thread kernel). +SELECT_LONG_DSL = True +LONG_SELECT_DSL_KEY_WIDTHS = (4, 11, 24) # capacity: fixed: compiled keys per thread +# The MSA index scorer (ops/attention/m3_index_score) prefetches the older index-K pages into L2 before its grid wait +# at every batch; up to this many requests it also issues Q and the first K tile right after the wait and caps that +# prefetch at one block per CTA (larger batches are bandwidth bound, where the extra early tiles only crowd the +# prefetch). On Rubin TP4 the 2-request step keeps the early schedule: with the producer's trigger at its exit +# (PRODUCER_LATE_TRIGGER_MAX_ROWS) the scorer no longer overlaps the producer, and the capped prefetch shortens its +# exposed pre-wait stream. In-model, locked memory clock, tactic- and FlashInfer-pinned, A and B in one job, both +# orders: -260 us per decode step at 2 requests (replay period at the 323K P90 request; -82 mean of draws), flat at +# 1 / 4 / 8. Timing only: the tiles are read after the wait. Kill: 1. +INDEX_SCORE_EARLY_MAX_REQUESTS = 2 # capacity: fixed: measured early-issue crossover +# T2SC: up to 2 requests the sparse decode scores with this target's copy of the sm_107 TP2 / TP8 scorer +# (kernels/m3_index_score_t2sc; ops/attention/m3_index_score_t2sc), whose split is chosen per request count and which +# TMA-loads an all-old first block before its grid wait; the scores are bit-identical +# to the in-tree scorer. TP4 has the TP8 target's per-rank indexer geometry (1 index head over the replicated index-K +# of its KV group), where this schedule is ~100 us per decode step faster at 1 request than split 512 without the +# prestage (TP8 in-model, greedy early column, both orders). Wider steps keep the sm_103-derived scorer above. +# Kill: False (the sm_103-derived scorer at every request count). +INDEX_SCORE_T2SC = True +# At 1 request the split is one wave of CTAs (1272 = 212 SMs x 6 resident at this CTA's shared memory): at the +# 1-index-head geometry (57-layer chain behind the producer, scores and the selection bitwise) -0.03 / -0.27 / +0.06 us +# per layer at 165K / 323K / 500K tokens against 1536. Kill: 1536. +INDEX_SCORE_T2SC_SPLIT_K: Mapping[int, int] = MappingProxyType( + {1: 1272, 2: 768} +) # capacity: fixed: measured splits per request count + +# The decode MXFP8 GEMVs, per projection (N, K): the rows each serves. On an activation its producer quantized +# (the MoE boundary for the QKV, the attention merge for the o_proj) the GEMV serves every row up to 48 but the +# unsplit K=768 shared down projection at 17..32 rows, where FlashInfer's GEMM is faster. +# The QKV MXFP8 GEMVs' (split, ring) at decode rows up to GEMV_QKV_CFG_MAX_ROWS: a TP4-owned override of the shared +# GB300-tuned table (kernels/mx_gemv/op.py). SM107 TP4 sweep +# (4 rows): sparse-layer QKV (8, 8) 2.70 us vs (4, 12) 3.01, dense-attention QKV (8, 8) 2.60 vs 2.83. Split 8 +# sums K in a different fp32 split-K order than split 4: exact-class, not bitwise to the shared config. Kill: False. +GEMV_QKV_SPLIT8_TP4 = True +GEMV_QKV_CFG_MAX_ROWS = 8 # capacity: fixed: the measured override's rows +MX_GEMV_CFG: Dict[Tuple[int, int], tuple] = { + QKV_SPARSE: (8, 8), + QKV: (8, 8), +} # capacity: fixed: (split, ring) per projection +# (exact-class, deterministic) The o_proj GEMV at decode rows up to GEMV_QKV_CFG_MAX_ROWS with strided k-tiles per split +# rank (contig 0: rank r owns k-tiles r, r + split, ...) instead of contiguous ranges, the shared (2, 8): another +# fixed fp32 order. SM107 TP4 4 rows: 2.513 / 2.533 vs 2.550 / 2.563 us; deterministic (3 runs bitwise) and +# max error vs the fp64 product equal to the contiguous order's. Kill: False. +GEMV_OPROJ_STRIDED_TP4 = True +MX_GEMV_CFG_STRIDED: Dict[Tuple[int, int], tuple] = { + O_PROJ: (2, 8, 0) +} # capacity: fixed: (split, ring, contig) per projection + + +def mx_gemv_cfg(n: int, k: int, rows: int) -> Optional[tuple]: + """The TP4 GEMV config override for an [rows, k] x [n, k]^T MXFP8 GEMV call, or None for the shared table's.""" + if rows > GEMV_QKV_CFG_MAX_ROWS: + return None + if GEMV_QKV_SPLIT8_TP4 and (n, k) in MX_GEMV_CFG: + return MX_GEMV_CFG[(n, k)] + if GEMV_OPROJ_STRIDED_TP4: + return MX_GEMV_CFG_STRIDED.get((n, k)) + return None + + +# Synth only (TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS, fractional): the fused verify's forced accepted counts are +# drawn per request, keyed by the request's KV length at the verify (its prompt plus what it has accepted), instead of +# from the server-wide counter, so a request's forced counts do not depend on what the server ran before (A/B draws +# become history-independent). Production sampling never forces counts. Kill: False (the shared counter, the in-tree +# sequence). +FORCE_AL_PER_REQUEST = True + +_ALL_ROWS = frozenset(range(1, DECODE_MAX_TOKENS + 1)) # capacity: decode rows +MX_GEMV_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = { + QKV_SPARSE: _ALL_ROWS, + QKV: _ALL_ROWS, + O_PROJ: _ALL_ROWS, + SHARED_GATE_UP: _ALL_ROWS, + SHARED_DOWN: _ALL_ROWS - frozenset(range(17, 33)), + DENSE_GATE_UP: _ALL_ROWS, + DENSE_DOWN: _ALL_ROWS, +} # capacity: decode rows per projection +# On a bf16 activation the GEMV quantizes it in its prologue, which pays only up to 8 32-element blocks per thread: +# the most rows each projection takes. +MX_GEMV_BF16_MAX_TOKENS: Dict[Tuple[int, int], int] = { + QKV_SPARSE: 16, + QKV: 16, + O_PROJ: 32, + SHARED_GATE_UP: 16, + SHARED_DOWN: 16, + DENSE_GATE_UP: 8, + DENSE_DOWN: 16, +} # capacity: fixed: measured bf16-GEMV crossover per projection +# MX-TAC (the TP2 target's, ported): the decode-graph MXFP8 projections the GEMVs do not serve (SHARED_DOWN at 17..32 +# rows, rows over DECODE_MAX_TOKENS or a bf16 activation over MX_GEMV_BF16_MAX_TOKENS, or a GEMV decline) take one fixed +# decision instead of the AutoTuner's per-start, per-rank choice: FlashInfer's CuTe DSL quantize and CuTe DSL GEMM. All +# of stock's candidates (quantize {TRT-LLM, CuTe DSL} x GEMM {CUTLASS, CuTe DSL}) give the same bits at the TP4 +# projections (gemm/test_modeling_v2_m3_pinned_gemm.py), so the pin is stock's output on every rank and start. Kill +# constant: False returns these projections to the AutoTuner. +MX_TAC_TP4 = True +# The tactics below are MXFP8QuantizeRunner.CUTE_DSL / FlashInferMXFP8GemmRunner.CUTE_DSL (0). +MXFP8_DECODE_QUANT_TACTIC = 0 # capacity: fixed: a tactic id, not a size +MXFP8_DECODE_GEMM_TACTIC = 0 # capacity: fixed: a tactic id, not a size + +Key = Hashable + + +@dataclass(frozen=True) +class RowRange: + """The shapes one fast path takes, and the catalog entry whose cells certify each of them. + + ``keys`` are the covered shapes: row counts for most paths, (query rows per request, requests) for the attention + kernels, a variant name for thresholds. ``certifies`` maps a cell's spec of ``entry`` to the keys that cell + drives (empty for a cell of another variant). + """ + + path: str + keys: FrozenSet[Key] + entry: OpWrapper + certifies: Callable[[Mapping[str, Any]], Iterable[Key]] + + def covers(self, key: Key) -> bool: + return key in self.keys + + def certified(self) -> FrozenSet[Key]: + return frozenset(k for cell in self.entry.CELLS for k in self.certifies(cell.spec)) + + +def _rows(lo: int, hi: int) -> FrozenSet[int]: + return frozenset(range(lo, hi + 1)) + + +def _spec_rows(field: str, **match: Any) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + """A cell's ``spec[field]`` when the cell's other fields match ``match``.""" + + def certifies(spec: Mapping[str, Any]) -> Iterable[Key]: + if any(spec.get(name) != value for name, value in match.items()): + return () + return (spec[field],) + + return certifies + + +def _const(*keys: Key) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + return lambda spec: keys + + +def _sparse_decode_producer(spec: Mapping[str, Any]) -> Iterable[Key]: + if not spec["identity"]: + return () + rows = spec["decode_query_len"] * len(spec["seq_lens"]) + return ("multi_query" if rows >= SPARSE_DECODE_MQ_MIN_TOKENS else "per_row",) + + +def _long_select_widths(spec: Mapping[str, Any]) -> Iterable[Key]: + """The key widths a cell's rows reach: every width up to its blocks' when rows of mixed lengths are drawn.""" + blocks = spec["num_blocks"] + widest = 2 if blocks <= 2048 else 4 if blocks <= 4096 else 9 + if spec["valid"] == "mixed": + return tuple(w for w in LONG_SELECT_KEY_WIDTHS if w <= widest) + return (widest,) + + +EMBED_FIRSTNORM = RowRange( + "embed_firstnorm", + _rows(1, EMBED_FIRSTNORM_MAX_TOKENS), + m3_embedding_firstnorm, + _spec_rows("rows"), +) +SPARSE_DECODE_PRODUCER = RowRange( + "sparse_decode_producer", + frozenset({"per_row", "multi_query"}), + m3_sparse_decode, + _sparse_decode_producer, +) +# Every token count of a sparse layer on an NVFP4 KV cache at the TP4 heads (prefill and decode): the producer takes the +# geometry, not a row range. +PRODUCER = RowRange("m3_producer", frozenset({"nvfp4_tp4"}), m3_producer, _const("nvfp4_tp4")) +INDEX_SCORE = RowRange( + "m3_index_score", + frozenset({"early", "prefetch"}), + m3_index_score, + lambda spec: ("early" if spec["early"] else "prefetch",), +) + + +def index_score_t2sc_schedule(requests: int) -> Tuple[int, bool]: + """T2SC's (CTAs per request, early issue) at this many requests (a request count in INDEX_SCORE_T2SC_SPLIT_K).""" + return INDEX_SCORE_T2SC_SPLIT_K[requests], requests <= INDEX_SCORE_EARLY_MAX_REQUESTS + + +def _index_score_t2sc_requests(spec: Mapping[str, Any]) -> Iterable[Key]: + """A cell certifies its request count when it drives this target's 1 index head with that count's schedule.""" + requests = len(spec["seq_lens"]) + if spec.get("heads") != 1 or requests not in INDEX_SCORE_T2SC_SPLIT_K: + return () + if (spec["split_k"], spec["early"]) != index_score_t2sc_schedule(requests): + return () + return (requests,) + + +INDEX_SCORE_T2SC_PATH = RowRange( + "m3_index_score_t2sc", + frozenset(INDEX_SCORE_T2SC_SPLIT_K) if INDEX_SCORE_T2SC else frozenset(), + m3_index_score_t2sc, + _index_score_t2sc_requests, +) +LONG_SELECT = RowRange("long_select", frozenset(), m3_select_long, _long_select_widths) + + +def _long_select_dsl_widths(spec: Mapping[str, Any]) -> Iterable[Key]: + """The 384-thread select's key widths a cell's rows reach (every width up to its blocks' for mixed rows).""" + blocks = spec["num_blocks"] + widest = 4 if blocks <= 4 * 384 else 11 if blocks <= 11 * 384 else 24 + if spec["valid"] == "mixed": + return tuple(w for w in LONG_SELECT_DSL_KEY_WIDTHS if w <= widest) + return (widest,) + + +LONG_SELECT_DSL = RowRange( + "long_select_dsl", + frozenset(LONG_SELECT_DSL_KEY_WIDTHS) if SELECT_LONG_DSL else frozenset(), + m3_select_long_dsl, + _long_select_dsl_widths, +) +MC_ALLREDUCE = RowRange( + "mc_allreduce_norm", + _rows(1, DECODE_MAX_TOKENS) if MC_AR_TP4 else frozenset(), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=0, drafter=None), +) +# Rows of a fused-draft-loop all-reduce boundary (drafter hidden 6144) on the multicast all-reduce's drafter form. +DRAFT_MC_AR = RowRange( + "m3_draft_mc_allreduce_norm", + _rows(1, DRAFT_MC_AR_MAX_ROWS), + m3_mc_allreduce_norm, + _spec_rows("m", drafter=True), +) +DENSE_PUSH = RowRange( + "dense_push", + _rows(1, DENSE_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +ATTN_PUSH = RowRange( + "attn_push", + _rows(1, ATTN_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +MOE_PUSH = RowRange( + "moe_push", + _rows(1, MOE_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +MC_FINALIZE = RowRange( + "mc_finalize_allreduce_norm", + _rows(MC_FINALIZE_MIN_TOKENS, MC_FINALIZE_MAX_TOKENS), + m3_mc_finalize_allreduce_norm, + _spec_rows("m"), +) +MOE_ENGINE = RowRange( + "m3_moe_engine", _rows(1, MOE_ENGINE_MAX_TOKENS), m3_moe_engine, _spec_rows("tokens") +) +MOE16_ENGINE = RowRange( + "m3_moe16_engine", + _rows(MOE16_MIN_TOKENS, MOE16_MAX_TOKENS), + m3_moe16_engine, + _spec_rows("tokens"), +) +# The router of a pure generation step of 9..16 rows when the MoE runs unfused (above the MoE engine's 8; at 9..13 an +# MoE16 engine miss, 14..16 always): the TF32 tensor-core router (minimax_m3_router_tc, 16-CTA clusters, one reduction +# order), the arithmetic the built-in gate takes on SM107 at these rows. Devices that cannot co-schedule 16-CTA +# clusters keep the fp32 router. Context steps never take it. +ROUTER_TC_GEN_MIN_ROWS = 9 # capacity: fixed: past the MoE engine's 8 rows +ROUTER_TC_GEN_MAX_ROWS = 16 # capacity: fixed: the built-in gate's router_tc range +ROUTER_TC_GEN = RowRange( + "m3_router_tc", + _rows(ROUTER_TC_GEN_MIN_ROWS, ROUTER_TC_GEN_MAX_ROWS), + m3_router_tc, + _spec_rows("m"), # every cell checks the logits, with or without the NVFP4 activation +) +# The GEMM paths are keyed (projection (N, K), rows): each projection runs its own split-K / ring / MMA +# configuration, so a row count is certified per projection. +MX_GEMV = RowRange( + "m3_mx_gemv", + frozenset((nk, m) for nk, rows in MX_GEMV_ROWS.items() for m in rows), + m3_mx_gemv, + lambda spec: ((spec["nk"], spec["m"]),), +) +MX_GEMV_BF16 = RowRange( + "m3_mx_gemv_bf16", + frozenset((nk, m) for nk, hi in MX_GEMV_BF16_MAX_TOKENS.items() for m in _rows(1, hi)), + m3_mx_gemv_bf16, + lambda spec: ((spec["nk"], spec["m"]),), +) +# (draft length, requests) of a greedy verify step. +SHARD_ACCEPT = RowRange( + "m3_shard_accept", + frozenset((SHARD_ACCEPT_DRAFT_LEN, n) for n in range(1, SHARD_ACCEPT_MAX_REQUESTS + 1)), + m3_shard_accept, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) + +# Requests of a decode step whose dense-layer attention runs at a capped KV split. +DENSE_DECODE_SPLIT = RowRange( + "m3_dense_decode", + frozenset(DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH), + m3_dense_decode, + lambda spec: (len(spec["kv_lens"]),) if spec["capped"] else (), +) + +# (query rows per request, requests) of a decode step whose dense-layer attention runs on PrimTS split-KV decode. +FULLCTX_DECODE = RowRange( + "m3_fullctx_decode", + frozenset(FULLCTX_DECODE_RECIPES) if FULLCTX_DECODE_ENABLED else frozenset(), + m3_fullctx_decode, + lambda spec: ((spec["qlen"], len(spec["kv_lens"])),), +) + +# (query rows per request, requests) of a generation-only drafter pass that runs on the full-context decode. +DRAFTER_FULLCTX = RowRange( + "m3_drafter_fullctx", + frozenset(FULLCTX_DECODE_RECIPES) if DRAFTER_FULLCTX_ENABLED else frozenset(), + m3_fullctx_decode, + lambda spec: ((spec["qlen"], len(spec["kv_lens"])),), +) + +# Verify rows of a fused-verify step whose target logits come from the coded head. +TARGET_HEAD = RowRange( + "m3_target_head", + _rows(1, TARGET_HEAD_MAX_ROWS), + m3_target_head, + _spec_rows("m"), +) + +# (draft length, requests) of a fused draft loop, and the rows of its passes' draft input: the first pass's +# requests x (draft length + 1), one per request on the later passes. +DRAFT_STEP = RowRange( + "m3_draft_step", + frozenset((DRAFT_STEP_DRAFT_LEN, n) for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1)), + m3_draft_step, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) +# ((N, K), rows) of a drafter BF16 linear on its low-M kernel. +DRAFT_LOWM = RowRange( + "m3_draft_lowm", + frozenset((nk, m) for nk, rows in DRAFT_LOWM_ROWS.items() for m in rows), + m3_draft_lowm, + lambda spec: ((spec["nk"], spec["m"]),), +) +DRAFT_INPUT = RowRange( + "m3_draft_input", + frozenset( + rows + for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1) + for rows in (n, n * (DRAFT_STEP_DRAFT_LEN + 1)) + ), + m3_draft_input, + _spec_rows("m"), +) + +# The step-loop hooks (RuntimeHooks), one switch each. They move a decode step's input preparation +# off the GPU timeline between step graphs. +STEP_PACK_INPUTS = True +# The pack's host copies go out in one parallel scatter (a program per copy) when their destinations do not overlap: the +# serial one-CTA scatter pays one dependent global round trip per copy (SM107: 12.5 / 15.6 / 31.7 us at 16 / 24 / 40 +# copies vs 6.5 / 6.5 / 6.9 us parallel). +STEP_PACK_PARALLEL_SCATTER = True +# The pack is issued by the decode graph's own prologue (a copy of a pinned window + the parallel scatter), so no +# operation is left between the step graphs (needs STEP_PACK_PARALLEL_SCATTER). +STEP_PACK_IN_GRAPH = True +# The window reaches the prologue's device buffer on a side copy stream while the previous graph runs; the prologue +# waits on that copy and scatters device to device, so the replay no longer starts with a host copy. Needs +# STEP_PACK_IN_GRAPH. +STEP_PACK_EARLY_COPY = True +# The step epoch (step_inputs.step_epoch): one int64 device counter the decode graph's prologue increments once per +# replay (program 0 of its first scatter launch, no extra kernel) and never resets. The per-layer megakernel pieces +# tag their per-step device flags with it (contract v1: tag = epoch << 8 | layer). Needs STEP_PACK_IN_GRAPH. +STEP_PACK_EPOCH = True +# The prologue is one launch of this target's op (ops/spec/m3_step_prologue, minimaxM3Sm107Tp4/stepPrologue.cu): the +# host copies, a CTA barrier, the device copies, the header clear and the epoch, where the default prologue runs two +# Triton scatter launches and an ATen fill (3 graph kernels -> 1, no Triton at the head of the decode graph). Same bytes +# written. Needs STEP_PACK_EARLY_COPY. Kill: False. +STEP_PACK_CPP_PROLOGUE = True +# PRE-WAIT WRITER CONTRACT: the kernels that read step inputs (seq_lens, +# block tables, positions, cache slots, the sampled-accept pre-wait parameters) before their grid-dependency wait rely +# on every writer of them being a host-to-device copy before the replay or a PLAIN launch earlier in the step: +# msa_kv_update, the step-input prologue (Triton scatter or the C++ op), the spec fold tail (Triton or C++), the pack +# scatter and any step-head op (the spec step correction, STEP_SPEC_OFFSETS_ONE_LAUNCH). Never give one of them a PDL +# attribute with an early trigger. +STEP_SKIP_UNCHANGED_H2D = True +# A changed packed 2-D metadata upload stages only each row's changed column span. On a step where a request crosses +# into a new page the MSA block table changed by one entry, but the whole padded table (32 KB a request) was staged, +# overflowing the 16 KB in-graph window into an eager upload and scatter between the step graphs (57-69 us at the +# ~1 in 19 block-crossing steps; TTFT inter-graph scan). Kill: False. +STEP_H2D_CHANGED_SPAN = True +# A speculative overlap step re-derives the MSA KV-length metadata once, after the forward's KV-length correction, +# instead of also before it (RuntimeHooks.kv_lens_update_once): the first pass (a fill, the out_cache_loc tail fill and +# an msa_kv_update at the head of the decode graph, 2.13 us at TP4) is rewritten in full by the second. Kill: False. +STEP_KV_LENS_UPDATE_ONCE = True +# The overlap scheduler's speculative step correction (position_ids += offsets, kv_lens += offsets) runs as this +# target's one plain launch (RuntimeHooks.spec_offsets_op, ops/spec/m3_spec_offsets) instead of two ATen adds at the +# head of the decode graph: the same int32 adds. TP8 graph bench (the work is per token, not per rank; 64 steps +# per graph, a predecessor kernel each): -1.27 us/step at 4 tokens / 1 request, -0.72..-0.78 at 8-32 tokens. +# Kill: False. +STEP_SPEC_OFFSETS_ONE_LAUNCH = True +STEP_LOOP_ON_EXECUTION_STREAM = True +STEP_MERGE_TUNED_TACTICS = True +STEP_SKIP_GREEDY_RNG_UPLOAD = True +# Sampled speculative batches stage their four per-step Philox seed/offset uploads into the step-input pack +# (STEP_PACK_INPUTS) instead of issuing four copies between the step graphs. +STEP_PACK_SPEC_RNG_UPLOADS = True +# The spec step's small uploads: the accepted-draft counts ride the step-input pack, and the sampler's slot ids stay in +# a device buffer re-uploaded only when they change (a pageable upload every step otherwise). +STEP_SPEC_SMALL_H2D = True +# SKIP_ACCEPTED_UPLOAD (RuntimeHooks.skip_accepted_draft_upload): the step's per-request accepted-draft +# counts are no longer uploaded and spec_metadata.num_accepted_draft_tokens is None. Nothing on this target reads that +# tensor (modeling.py, dspark.py, dspark_s4.py, mxdraft.py, kernels/, ops/; the in-tree DFlash / DSpark workers and +# the Eagle3 path: tests/unittest/_torch/executor/test_skip_accepted_draft_upload.py scans for readers). Kill switch: +# TRTLLM_M31_DISABLE_SKIP_ACCEPTED_UPLOAD=1 (the stock per-step upload, packed under spec_small_h2d). +STEP_SKIP_ACCEPTED_UPLOAD = not m31_disabled("SKIP_ACCEPTED_UPLOAD") +# BATCH_CHANGED_ONLY (RuntimeHooks.spec_batch_map_changed_only): the standalone DSpark (DFlash) +# drafter's batch_indices and batch-to-slot map are uploaded only when they change (both constant in steady decode); +# the buffers hold the same values as with the stock every-step copies. Kill switch: +# TRTLLM_M31_DISABLE_BATCH_CHANGED_ONLY=1 (both copied every step). +STEP_BATCH_CHANGED_ONLY = not m31_disabled("BATCH_CHANGED_ONLY") +STEP_SPEC_FOLD = True +# The forward of a decode-only step stores its outputs into the spec sampler's store and stages the next step's inputs +# at its tail (one launch inside the step graph), so neither the sampler's store nor the next step's gathers run +# between the step graphs (M3SpecStepFold.in_forward). +STEP_SPEC_FOLD_IN_FORWARD = True +# The forward's tail (STEP_SPEC_FOLD_IN_FORWARD) runs as this target's C++ op (kernels/m3_fold_tail cpp=, +# minimaxM3Sm107Tp4/foldTail.cu) instead of the Triton kernel: the same integer copies, one launch either way (no +# Triton on the generation decode path). Kill: False. +STEP_FOLD_TAIL_CPP = True +# With STEP_SPEC_FOLD_IN_FORWARD the spec sampler's host copies of its store (new tokens, lengths, draft tokens) still +# ran on the main stream between the step graphs (the forward, not the sampler, wrote the store, and it rewrites it at +# the next replay's tail). They run on the D2H side stream instead, and the forward's tail waits on an event the +# sampler records after them before it rewrites the store (M3SpecStepFold.store_copies_done). +STEP_SPEC_STORE_COPY_SIDE = True +# The spec sampler copies its per-slot store to the host on its D2H side stream, so the copies overlap the next +# step's input staging instead of sitting between the step graphs. +STEP_SAMPLER_D2H_SIDE_STREAM = True +# The iteration log times the device step with one event per iteration boundary on a side stream, so no timing +# record sits between two step graphs on the main stream (the reported step times are the same). +STEP_ONE_TIMING_EVENT = True +MSA_SHORT_SEQUENCE_IDENTITY = True +# The sparse decode (the LC, the Triton merge) also writes the MXFP8 o_proj hand-off (bit +# for bit torch.ops.trtllm.mxfp8_quantize of its BF16 rows) and publishes it to m3_prequant.SLOT. Kill: False publishes +# nothing, so the o_proj quantizes its BF16 input itself as the built-in path does. +MERGE_MXFP8_HANDOFF = True +MSA_KV_UPDATE = True +MSA_SKIP_UNCHANGED_BLOCK_OFFSETS = True +MSA_REUSE_POOL_VIEWS = True +# Step-input preparation fills the MSA block-id table by slice-copying each request's page-index view (MSA +# RuntimeHooks.block_ids_from_views) instead of building a Python list of the whole padded row; the table is the same. +MSA_BLOCK_IDS_FROM_VIEWS = True + +FAST_PATHS: Tuple[RowRange, ...] = ( + EMBED_FIRSTNORM, + DENSE_DECODE_SPLIT, + FULLCTX_DECODE, + DRAFTER_FULLCTX, + SPARSE_DECODE_PRODUCER, + PRODUCER, + INDEX_SCORE, + INDEX_SCORE_T2SC_PATH, + LONG_SELECT, + LONG_SELECT_DSL, + MC_ALLREDUCE, + DRAFT_MC_AR, + ATTN_PUSH, + DENSE_PUSH, + MOE_PUSH, + MC_FINALIZE, + MOE_ENGINE, + MOE16_ENGINE, + ROUTER_TC_GEN, + MX_GEMV, + MX_GEMV_BF16, + SHARD_ACCEPT, + TARGET_HEAD, + DRAFT_STEP, + DRAFT_LOWM, + DRAFT_INPUT, +) + +# -------------------------------------------------------------------------------------------------------------------- +# Selection: pure functions of the step shape +# -------------------------------------------------------------------------------------------------------------------- + + +def plan_for(num_tokens: int) -> DecodePlan: + for plan in DECODE_PLANS: + if num_tokens <= plan.max_tokens: + return plan + raise AssertionError(f"no decode plan covers {num_tokens} tokens") + + +def is_decode_step(num_tokens: int) -> bool: + """Whether a pure generation step of ``num_tokens`` rows runs the decode kernels (else generic layers).""" + return num_tokens <= DECODE_STEP_MAX + + +def moe_path(num_tokens: int) -> Optional[str]: + """The MoE kernel of a decode step: an engine, the unfinalized MoE + finalize all-reduce, or None (generic).""" + if MOE_ENGINE.covers(num_tokens): + return MOE_ENGINE.path + if MOE16_ENGINE.covers(num_tokens): + return MOE16_ENGINE.path + if MC_FINALIZE.covers(num_tokens): + return MC_FINALIZE.path + return None + + +MX_GEMV_PATH = MX_GEMV.path +MX_GEMV_BF16_PATH = MX_GEMV_BF16.path +# The dense layers' swigluoai activation (where the fold above would apply) as this target's C++ op +# (ops/gemm/m3_dense_swiglu, minimaxM3Sm107Tp8/swigluOai.cu: bit for bit the Triton silu_and_mul), PDL-launched with +# an early release so the down GEMV stages its weights during it; the down GEMV reads the activation only after its own +# grid wait. TP2 chain bench: -0.69 us per dense layer vs the Triton kernel. Kill constant: False +# runs the Triton kernel. +DENSE_SWIGLU_CPP = True +DENSE_SWIGLU_CPP_PATH = "m3_dense_swiglu" +# FlashInfer's CuTe DSL MXFP8 GEMM on the producer's MXFP8 activation (rows the GEMV does not serve). +PREQUANT_GEMM_PATH = "flashinfer_mxfp8_gemm" +# The MXFP8 quantize (tuned backend) and FlashInfer's CuTe DSL GEMM: every row the decode kernels do not serve. +GENERIC_GEMM_PATH = "mxfp8_gemm" + + +def linear_path(n: int, k: int, num_tokens: int, prequant: bool, low_m_gemm: bool) -> str: + """The decode-graph MXFP8 GEMM of an [N, K] projection on ``num_tokens`` rows. + + ``prequant``: the producer published the activation in MXFP8. ``low_m_gemm`` is accepted for the call sites' + symmetry: no low-M GEMM configuration is certified at TP4, so it selects nothing. + """ + if prequant: + if num_tokens in MX_GEMV_ROWS.get((n, k), ()): + return MX_GEMV_PATH + return PREQUANT_GEMM_PATH + if num_tokens <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0): + return MX_GEMV_BF16_PATH + return GENERIC_GEMM_PATH + + +# -------------------------------------------------------------------------------------------------------------------- +# Fallback policy +# -------------------------------------------------------------------------------------------------------------------- + + +class FastPathUnavailable(RuntimeError): + """A covered decode shape could not take its fast path under ``modeling_v2='require'``.""" + + +_lock = threading.Lock() +_require = [False] +_logged: set = set() + + +def set_require(require: bool) -> None: + """Whether a miss raises (``modeling_v2='require'``); set by the target at construction.""" + _require[0] = bool(require) + + +def require_mode() -> bool: + return _require[0] + + +def _log_once(level: Callable[[str], None], path: str, reason: str, message: str) -> None: + key = (path, reason) + with _lock: + if key in _logged: + return + _logged.add(key) + level(message) + + +def long_select_dsl_keys() -> Tuple[Tuple[bool, bool, int], ...]: + """The (PDL launch, early trigger, trigger phase) keys of the 384-thread select a decode step can launch, compiled + before warmup; none when ``SELECT_LONG_DSL`` is off.""" + if not LONG_SELECT_DSL.keys: + return () + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + pdl = get_env_enable_pdl() + keys = [(pdl, False, SELECT_LONG_TRIGGER_PHASE), (pdl, False, 0)] + if SELECT_EARLY_MAX_ROWS: + keys.append((pdl, True, 0)) + return tuple(dict.fromkeys(keys)) + + +def missed(path: str, rows: Optional[int], reason: str) -> None: + """``path`` covers this step but cannot run: raise under require, else log once; the caller falls back.""" + at = "" if rows is None else f" at {rows} rows" + if _require[0]: + raise FastPathUnavailable( + f"modeling_v2='require': the MiniMax-M3 decode path {path!r} covers this step{at} but cannot run: " + f"{reason}" + ) + _log_once( + logger.warning, + path, + reason, + f"[M3 dispatch] {path}{at}: {reason}; falling back (logged once per reason)", + ) + + +def capture_missed(path: str, rows: Optional[int], what: str) -> None: + """``what`` is built on eager steps but a CUDA-graph capture found it missing (see ``missed``).""" + missed(path, rows, f"{what} is not built and a CUDA graph is being captured") + + +def taken(path: str, rows: Optional[int], what: str = "") -> None: + """``path`` ran on this rank at ``rows``: one rank-tagged info line per (path, rows, what), the engagement line + every GEN rank prints for each fast path it takes (``what``: an optional detail, e.g. the projection).""" + at = "" if rows is None else f" at {rows} rows" + detail = f" ({what})" if what else "" + _log_once( + logger.info, + path, + f"taken: {rows}" + (f": {what}" if what else ""), + f"[M3 dispatch] rank {mpi_rank()}: {path} taken{at}{detail}", + ) + + +def uncovered(path: str, reason: str) -> None: + """The step is outside ``path``'s covered shapes: log once per (path, reason); the generic path runs.""" + _log_once(logger.info, path, reason, f"[M3 dispatch] {path} not taken: {reason} (logged once)") + + +def reset_logged() -> None: + """Forget which (path, reason) pairs were logged.""" + with _lock: + _logged.clear() + + +# Kernel modules report from inside their calls through kernels/m3_report: the same policy. +m3_report.install( + on_taken=taken, on_missed=missed, on_uncovered=uncovered, on_capture_missed=capture_missed +) + + +# -------------------------------------------------------------------------------------------------------------------- +# Target constants +# -------------------------------------------------------------------------------------------------------------------- + +# The shared expert folds into the routed engine from one token (dedicated shared-expert CTAs). +MOE_FOLD_MIN_TOKENS = 1 # capacity: fixed: a row lower bound +# SMs left to the concurrent kernels while a persistent routed engine runs. +MOE_ENGINE_RESERVED_SMS = 24 # capacity: fixed: an SM split, not a size +# Id of the target's final hidden state in Eagle3's ``layers_to_capture``. +FINAL_HIDDEN_LAYER_ID = -1 # capacity: fixed: an Eagle3 layer id, not a size +# The trtllm-gen MoE runner's routing kind for MiniMax-M3 (sigmoid + correction bias, renormalized, scaled) and its +# gated activation (SwiGLU with the per-expert alpha / beta / clamp operands). +ROUTING_MINIMAX2 = int(RoutingMethodType.MiniMax2) +ACT_SWIGLU = int(ActType_TrtllmGen.SwiGlu) + +# -------------------------------------------------------------------------------------------------------------------- +# Streaming: also emit a response on decoding iterations 2..FIRST_CONTENT_EMIT_ITERS (RuntimeHooks), so a request's +# first content leaves one step after its first token. 1: the stock schedule. +FIRST_CONTENT_EMIT_ITERS = 2 + +# The runtime-value dump (TRTLLM_MODELING_V2_VALUE_DUMP=1): the metadata of every fast-path and entry decision a step +# takes, recorded on eager steps and captures (replay runs no Python), printed once at exit. +# -------------------------------------------------------------------------------------------------------------------- + +_VALUE_DUMP = os.environ.get("TRTLLM_MODELING_V2_VALUE_DUMP") == "1" +_VALUES: set = set() + + +def _record(*meta: Hashable) -> None: + if _VALUE_DUMP: + _VALUES.add(meta) + + +def _print_values() -> None: + if _VALUES: + rank = os.environ.get("OMPI_COMM_WORLD_RANK", os.environ.get("RANK", "0")) + for meta in sorted(_VALUES, key=repr): + print(f"[modeling_v2 value dump] rank {rank}: {meta}", flush=True) + + +if _VALUE_DUMP: + atexit.register(_print_values) + +# -------------------------------------------------------------------------------------------------------------------- +# Configuration: the text tower and the runtime the target assumes + +# -------------------------------------------------------------------------------------------------------------------- + + +def _is_vl_config(pretrained_config: PretrainedConfig) -> bool: + model_type = getattr(pretrained_config, "model_type", None) + if model_type == "minimax_m3_vl": + return True + if model_type == "minimax_m3": + return False + return any( + "ForConditionalGeneration" in a + for a in (getattr(pretrained_config, "architectures", None) or []) + ) + + +def _text_model_config(model_config: ModelConfig) -> ModelConfig: + """The model config with the language model's ``text_config`` as its pretrained config. + + The checkpoint declares the vision-language wrapper; its ``text_config`` arrives as a plain dict (no typed + sub-config is registered), so it is materialized into a ``PretrainedConfig`` with the wrapper's dtype and + embedding tie carried over. A text config is returned unchanged. + """ + if not _is_vl_config(model_config.pretrained_config): + return model_config + cfg = copy.deepcopy(model_config) + text = getattr(cfg.pretrained_config, "text_config", None) + if text is None: + raise ValueError("MiniMax-M3 VL config has no `text_config`") + if isinstance(text, dict): + out = PretrainedConfig() + for k, v in text.items(): + setattr(out, k, v) + text = out + if ( + getattr(text, "torch_dtype", None) is None + and getattr(cfg.pretrained_config, "torch_dtype", None) is not None + ): + text.torch_dtype = cfg.pretrained_config.torch_dtype + if ( + getattr(text, "tie_word_embeddings", None) is None + and getattr(cfg.pretrained_config, "tie_word_embeddings", None) is not None + ): + text.tie_word_embeddings = cfg.pretrained_config.tie_word_embeddings + return dataclasses.replace(cfg, pretrained_config=text) + + +def _validate_runtime(model_config: ModelConfig) -> None: + """The runtime this target is written for (it is not in the checkpoint): routing declines every other one + (``routing._runtime_mismatch``), so this is the backstop for a construction that did not go through routing. + + The MiniMax-M3 MSA backend (it owns the step metadata and the index-K cache), the fused five-way QKV/index + projection or the separate QKV and index projections, an FP8 indexer and an FP8 or NVFP4 main KV cache. + """ + sparse = model_config.sparse_attention_config + if ( + sparse is None + or getattr(sparse, "algorithm", None) != "minimax_m3" + or sparse.implementation != "msa" + ): + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.algorithm='minimax_m3' with implementation='msa'" + ) + if getattr(sparse, "indexer_kv_dtype", None) != "fp8": + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.indexer_kv_dtype='fp8'" + ) + mode = model_config.quant_config.quant_mode + if mode is None or not (mode.has_fp8_kv_cache() or mode.has_fp4_kv_cache()): + raise ValueError("this MiniMax-M3 target requires an FP8 or NVFP4 KV cache") + + +def _sparse_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = cfg.sparse_attention_config["sparse_attention_freq"] + return [i for i, f in enumerate(freq) if int(f) != 0] + + +def _moe_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = getattr(cfg, "moe_layer_freq", None) + if freq is None: + return list(range(cfg.num_hidden_layers)) + return [i for i, f in enumerate(freq) if int(f) != 0] + + +class _Mxfp8GemmState(nn.Module): + """The MXFP8 GEMM backend state the engine's warmup drives. + + The engine finds MXFP8 projections by a module's ``quant_method`` and, for every one, opts it into native GEMM + tuning and (on decode CUDA graphs) the auto-dispatched FlashInfer GEMM, then marks each tuned. Every projection of + this target dispatches identically, so one method object carries that state for all of them; the forward reads it + at each call (``_mx_mode``) and passes the resolved backend to the GEMM entries. + """ + + def __init__(self) -> None: + super().__init__() + self.quant_method = MXFP8LinearMethod() + + +@dataclass(frozen=True) +class MxMode: + """The MXFP8 GEMM dispatch of one call, resolved from ``_Mxfp8GemmState`` and the engine's contexts.""" + + decode_graph: bool # inside a decode CUDA-graph warmup or capture + autotune: bool # inside the engine's FlashInfer MXFP8 tuning pass + tuned_quantize_gemm: ( + bool # per-bucket tuned quantize + GEMM (decode graphs, tune_decode_graph_backends) + ) + flashinfer: bool # FlashInfer's MXFP8 GEMM on the quantized activation + native_tuning: ( + bool # the native GEMM through the autotuner (warmup, before its tactics are registered) + ) + + +def _mx_mode(state: _Mxfp8GemmState) -> MxMode: + method = state.quant_method + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + autotune = flashinfer_mxfp8_autotune_active() + return MxMode( + decode_graph=decode_graph, + autotune=autotune, + tuned_quantize_gemm=method.tune_decode_graph_backends and decode_graph, + flashinfer=method.backend == "flashinfer" + or ( + method.backend == "auto" + and (autotune or (method._flashinfer_autotuned and decode_graph)) + ), + native_tuning=method.needs_native_autotune, + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# Projections: the MXFP8 GEMM dispatch (``linear_path``) and its hand-offs +# -------------------------------------------------------------------------------------------------------------------- + + +def _mxfp8_linear( + core, x: torch.Tensor, w: torch.Tensor, s: torch.Tensor, mode: MxMode +) -> torch.Tensor: + """One MXFP8 projection on the engine-tuned path: MXFP8 quantize + GEMM, backend per ``mode``.""" + _record( + "m3_mxfp8_linear", + tuple(w.shape), + mode.tuned_quantize_gemm, + mode.flashinfer, + mode.native_tuning, + ) + return m3_mxfp8_linear( + x.contiguous(), + w, + s, + core.mx_alpha, + torch.bfloat16, + mode.tuned_quantize_gemm, + mode.autotune, + mode.flashinfer, + mode.native_tuning, + ) + + +def _gemv_variants(n: int, k: int, push: Optional[str], world: int): + """Every (rows, bf16 activation, push (world, bucket tokens, mode)) call the decode dispatch routes to an [N, K] + projection's GEMV: an MXFP8 activation on the rows ``MX_GEMV_ROWS`` gives it, a bf16 one up to + ``MX_GEMV_BF16_MAX_TOKENS``; each pushing when the projection's boundary push covers the rows.""" + push_rows = {"attn": ATTN_PUSH.keys, "dense": DENSE_PUSH.keys, "moe": MOE_PUSH.keys}.get( + push, frozenset() + ) + variants = [] + for rows in range(1, DECODE_MAX_TOKENS + 1): + for x_bf16, served in ( + (False, rows in MX_GEMV_ROWS.get((n, k), ())), + (True, rows <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0)), + ): + if not served: + continue + variants.append((rows, x_bf16, None)) + if rows in push_rows: + bucket = next(b for b in m3_mc_allreduce.BUCKETS if rows <= b) + variants.append((rows, x_bf16, (world, bucket, AR_PUSH_MODE))) + return variants + + +def _precompile_cfg( + weight: torch.Tensor, + weight_scale: torch.Tensor, + m: int, + cfg: tuple, + use_pdl: bool, + ar: Optional[Tuple[int, int, int]] = None, +) -> Optional[tuple]: + """``mx_gemv.precompile`` for a call with this target's (split, ring) override ``cfg`` (``mx_gemv_cfg``): the + shared precompile resolves the shared table's config only. Compiles without launching, outside capture, also while + compiles are frozen; returns the kernel's key, or None when such a call is declined.""" + import cuda.bindings.driver as cuda_driver + + n, k = weight.shape + push = None if ar is None else (0, 0, 0, *ar) + resolved = mx_gemv._resolve(m, n, k, False, True, cfg, use_pdl, push) + if resolved is None: + return None + key, runtime_ar = resolved + if mx_gemv.is_compiled(key): + return key + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("_precompile_cfg allocates and compiles: call it outside capture") + dev = weight.device + x = torch.zeros(m, k, dtype=torch.float8_e4m3fn, device=dev) + xsf = torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=dev) + y = torch.empty(m, n, dtype=torch.bfloat16, device=dev) + args = mx_gemv._views(x, xsf, weight, weight_scale, y, False) + stream = cuda_driver.CUstream(torch.cuda.current_stream(dev).cuda_stream) + mx_gemv._compile(key, args, m, runtime_ar, stream, mx_gemv._extra_args(dev)) + return key + + +_MX_TAC_ENGAGED: set = set() + + +def _note_mx_tac(kind: str, n: int, k: int, rows: int) -> None: + """MX-TAC's engagement line, once per (activation kind, projection, rows).""" + key = (kind, n, k, rows) + if key not in _MX_TAC_ENGAGED: + _MX_TAC_ENGAGED.add(key) + logger.info( + f"[M3 sm_107 tp4] MX-TAC engaged: pinned MXFP8 decision (quantize tactic " + f"{MXFP8_DECODE_QUANT_TACTIC}, GEMM tactic {MXFP8_DECODE_GEMM_TACTIC}) " + f"for the [{n}, {k}] projection at {rows} rows, {kind} activation" + ) + + +def _uncompiled_note() -> str: + key = mx_gemv.last_uncompiled + return "" if key is None else f" (its kernel {key} was not compiled before warmup)" + + +def _linear( + core, x: torch.Tensor, wk: str, sk: str, low_m_gemm: bool, push: Optional[tuple] = None +) -> Tuple[torch.Tensor, bool]: + """One MXFP8 projection ``x @ w[wk]^T``; returns (output, whether its GEMV pushed into ``push``). + + Outside decode CUDA graphs: the engine-tuned MXFP8 quantize + GEMM. Inside them the projection's (N, K) and the + step's rows pick one path (``linear_path``): + + 1. an activation its producer already quantized to MXFP8 (``m3_prequant.SLOT``, taken here): the tcgen05 decode + GEMV on the rows it serves, else the decode-graph MXFP8 GEMM on that activation (AutoTuner-selected backend + per token bucket); + 2. a bf16 activation on the rows the GEMV serves quantizing in its prologue; + 3. otherwise the engine-tuned MXFP8 quantize + GEMM. + + A kernel declining rows its path covers is a miss; the chain then continues down the list. ``push`` (an + ``m3_mx_gemv`` ``ar`` tuple) is the boundary all-reduce exchange the GEMV writes its rows into. ``low_m_gemm`` + is accepted for the call sites' symmetry (``linear_path``): no TP4 low-M GEMM configuration is built. + """ + w, s = core.w[wk], core.w[sk] + x = x.contiguous() + mode = _mx_mode(core.mxfp8) + if not mode.decode_graph: + prequant = _CTXNORMQ_MX.take(x) if CTXNORMQ and not mode.tuned_quantize_gemm else None + if prequant is not None: + return _ctxnormq_mxfp8_gemm(core, prequant, w, s, mode), False + return _mxfp8_linear(core, x, w, s, mode), False + use_pdl = get_env_enable_pdl() + rows = x.shape[0] + n, k = w.shape + prequant = m3_prequant.SLOT.take(x) + if prequant is None and x.dtype != torch.bfloat16: + path = GENERIC_GEMM_PATH + else: + path = linear_path(n, k, rows, prequant is not None, low_m_gemm) + _record("linear", (n, k), rows, path, push is not None) + output = None + # MX-TAC: the decode-graph MXFP8 quantize and GEMM at the target's fixed tactics, no AutoTuner. + pinned = MX_TAC_TP4 and core.mxfp8.quant_method.tune_decode_graph_backends + if prequant is not None: + act, act_scale = prequant + if path == MX_GEMV_PATH: + cfg = mx_gemv_cfg(n, k, rows) + output = m3_mx_gemv(act, act_scale, w, s, True, cfg, use_pdl, push) + if output is not None: + split = f", split {cfg[0]}" if cfg is not None else "" + taken(path, rows, f"[{n}, {k}] mxfp8 in{split}") + return output, push is not None and mx_gemv.last_pushed + missed( + path, + rows, + f"the GEMV declined the [{n}, {k}] projection's MXFP8 activation" + + _uncompiled_note(), + ) + if path == PREQUANT_GEMM_PATH: + uncovered(MX_GEMV_PATH, f"the [{n}, {k}] projection at rows the GEMV does not serve") + if pinned: + output = m3_mxfp8_pinned_gemm( + act, act_scale, w, s, torch.bfloat16, MXFP8_DECODE_GEMM_TACTIC + ) + _note_mx_tac("mxfp8", n, k, rows) + return output, False + return m3_mxfp8_gemm(act, act_scale, w, s, torch.bfloat16, mode.autotune), False + if path == MX_GEMV_BF16_PATH: + output = m3_mx_gemv_bf16(x, w, s, True, None, use_pdl, push) + if output is not None: + taken(path, rows, f"[{n}, {k}] bf16 in") + return output, push is not None and mx_gemv.last_pushed + missed( + path, rows, f"the GEMV declined the [{n}, {k}] projection's input" + _uncompiled_note() + ) + if path == GENERIC_GEMM_PATH: + uncovered( + MX_GEMV_BF16_PATH, + f"the [{n}, {k}] projection's bf16 input beyond the decode GEMV's rows", + ) + if pinned and x.dtype == torch.bfloat16: + act, act_scale = m3_mxfp8_pinned_quantize(x, MXFP8_DECODE_QUANT_TACTIC) + output = m3_mxfp8_pinned_gemm( + act, act_scale, w, s, torch.bfloat16, MXFP8_DECODE_GEMM_TACTIC + ) + _note_mx_tac("bf16", n, k, rows) + return output, False + # Stock's decode-graph dispatch for this shape (the tuned quantize + GEMM). + return _mxfp8_linear(core, x, w, s, mode), False + + +def _dense_swiglu(core, gu: torch.Tensor) -> torch.Tensor: + """A dense MLP's SwiGLU-OAI activation: this target's C++ op (``DENSE_SWIGLU_CPP``, bit for bit the Triton kernel, + PDL-launched with an early release) up to its certified rows, in any graph or eager step; else the Triton + kernel.""" + rows = int(gu.shape[0]) + path = DENSE_SWIGLU_CPP_PATH + max_rows = max(cell.spec["rows"] for cell in m3_dense_swiglu.CELLS) + if DENSE_SWIGLU_CPP and not is_torch_compiling(): + if rows > max_rows: + uncovered(path, f"the dense swigluoai op: {rows} rows, past its certified {max_rows}") + else: + act = m3_dense_swiglu( + gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta, get_env_enable_pdl() + ) + if act is not None: + taken(path, rows) + return act + uncovered(path, "the gate_up layout") + return m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + + +# -------------------------------------------------------------------------------------------------------------------- +# Layer boundaries: all-reduce + residual + RMSNorm +# -------------------------------------------------------------------------------------------------------------------- + + +def _can_mc(x, folded: bool = False) -> bool: + """Whether the generic boundary takes the multicast all-reduce + residual + RMSNorm (``GENERIC_MC_AR_MAX_TOKENS``, + with the shared + routed add folded in ``GENERIC_MC_AR_FOLDED_MAX_TOKENS``).""" + return ( + isinstance(x, torch.Tensor) + and x.dim() == 2 + and x.shape[0] <= (GENERIC_MC_AR_FOLDED_MAX_TOKENS if folded else GENERIC_MC_AR_MAX_TOKENS) + and x.dtype == torch.bfloat16 + and x.is_contiguous() + ) + + +def _mc_allreduce_norm(core, x, x2, residual, norm_w, *, quant=False, prepushed=0, capture=None): + """The multicast all-reduce + optional second addend + residual + RMSNorm (``m3_mc_allreduce_norm``), or None + without a workspace or for operands the kernel does not take (the same on every rank). With ``quant`` the normed + rows are also published, MXFP8-quantized, to ``m3_prequant.SLOT`` for the next MXFP8 projection; ``capture`` = + (out, fc_norm weight, eps): the boundary also writes the Eagle3 capture norm of the updated residual. Up to + ``MC_AR_CLUSTER_MAX_ROWS`` rows the one-shot 8-CTA cluster form runs; ``MC_AR_CARVEOUT`` sets its shared-memory + carveout.""" + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + if workspace is None: + return None + _record("m3_mc_allreduce_norm", x.shape[0], x2 is not None, quant, prepushed) + return m3_mc_allreduce.m3_mc_allreduce_norm( + x, + residual.contiguous(), + norm_w, + core.rms_eps, + workspace, + use_pdl=get_env_enable_pdl(), + x2=x2, + quant=quant, + prepushed=prepushed, + oneshot_cluster=x.shape[0] <= MC_AR_CLUSTER_MAX_ROWS, + carveout=MC_AR_CARVEOUT, + capture=capture, + ) + + +# The CTXNORMQ hand-offs: the boundary's MXFP8 activation for the first prefill MXFP8 projection reading the normed +# rows, and its NVFP4 form for the routed experts. Separate from ``m3_prequant.SLOT`` so only a CTXNORMQ boundary +# feeds the prefill path. +_CTXNORMQ_MX = m3_prequant.PrequantSlot() +_CTXNORMQ_FP4 = m3_prequant.PrequantSlot() +_CTXNORMQ_STATE = {} + + +def _ctxnormq_op(): + """The op for the CTXNORMQ prefill boundary, or None: ``trtllm::minimax_m31_ctx_residual_rmsnorm_mxfp8`` is + built for SM103 only and refuses this device, so the boundary keeps the fused all-reduce. Decided once per + process.""" + if "op" not in _CTXNORMQ_STATE: + _CTXNORMQ_STATE["op"] = None + logger.info( + "[M3 dispatch] CTXNORMQ prefill boundary norm + quantize: not taken: the op is not built for SM107 " + f"(rows >= {CTXNORMQ_MIN_ROWS})" + ) + return _CTXNORMQ_STATE["op"] + + +def _ctxnormq_allreduce_norm(core, x, residual, norm_w, fp4_scale, allreduce=None): + """A prefill boundary as the plain all-reduce + the fused residual add, RMSNorm and quantize, or None (switched + off, fewer than ``CTXNORMQ_MIN_ROWS`` rows, op not built). Returns (normed, updated) like the fused all-reduce; + the MXFP8 activation (and with ``fp4_scale`` the NVFP4 one) wait in the CTXNORMQ slots for their consumers. + ``allreduce``: the all-reduce to run (default the target's AUTO one).""" + if not CTXNORMQ or x.dim() != 2 or x.shape[0] < CTXNORMQ_MIN_ROWS: + return None + op = _ctxnormq_op() + if op is None: + return None + reduced = m3_allreduce(x, core.allreduce if allreduce is None else allreduce, None) + _record("m31_ctxnormq", x.shape[0], fp4_scale is not None) + normed, xq, xsf, xfp4, xfp4_sf = op( + reduced, residual.contiguous(), norm_w, core.rms_eps, True, fp4_scale, get_env_enable_pdl() + ) + _CTXNORMQ_MX.publish(normed, xq, xsf) + if fp4_scale is not None: + _CTXNORMQ_FP4.publish(normed, xfp4, xfp4_sf) + else: + _CTXNORMQ_FP4.clear() + return normed, reduced + + +def _ctxnormq_mxfp8_gemm(core, prequant, w, s, mode): + """The GEMM ``m3_mxfp8_linear`` runs after its own quantize, on the CTXNORMQ activation (bit-identical operands).""" + act, act_scale = prequant + _record("m31_ctxnormq_gemm", tuple(w.shape), act.shape[0], mode.flashinfer, mode.native_tuning) + if mode.flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8(act, act_scale, w, s, torch.bfloat16) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if mode.native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, w, s, core.mx_alpha, torch.bfloat16) + + +_GQAMIX_STATE: dict = {} + + +def _gqamix_decode(*args, **kwargs) -> None: + """``minimax_m3_sparse_attn_decode`` for MsaDecodeFmha's sparse generation rows, through the target-local copy.""" + if not _GQAMIX_STATE.get("logged"): + _GQAMIX_STATE["logged"] = True + logger.info( + "[M3 dispatch] GQAMIX: mixed-step sparse decode through ops/gqamix_sparse_decode" + ) + _gqamix_kernel.minimax_m3_sparse_attn_decode(*args, **kwargs) + + +def _gqamix_register(backend) -> None: + """Register ``_gqamix_decode`` on the layer backend's MsaDecodeFmha (a library without the hook is left alone).""" + manager = getattr(backend, "_fmha_manager", None) + for fmha in getattr(manager, "fmha_libs", ()): + if getattr(fmha, "sparse_decode_hook", False) is None: + fmha.sparse_decode_hook = _gqamix_decode + + +def _native_allreduce_norm(core, x, residual, norm_w, allreduce=None): + """The built-in fused all-reduce + residual + RMSNorm (``allreduce``: default the target's AUTO one).""" + _record("m3_allreduce", x.shape[0]) + return m3_allreduce( + x, + core.allreduce if allreduce is None else allreduce, + AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual, + norm_weight=norm_w, + eps=core.rms_eps, + trigger_completion_at_end=False, + ), + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# The MoE: router, routed experts (trtllm-gen NVFP4), shared expert +# -------------------------------------------------------------------------------------------------------------------- + + +@dataclass +class _MoeScalars: + """The routed experts' kernel scalars of one layer, derived from the checkpoint's per-tensor scales once the + weights are real (``MiniMaxM3Tp4ModelingV2._derive_moe``).""" + + fc31_input_scale: torch.Tensor + fc2_input_scale: torch.Tensor + fc31_alpha: torch.Tensor + fc2_alpha: torch.Tensor + fc31_scale_c: torch.Tensor + act_alpha: torch.Tensor + act_beta: torch.Tensor + act_clamp: torch.Tensor + + +def _router_logits(core, i: int, h: torch.Tensor) -> torch.Tensor: + """The fp32 router logits of MoE layer ``i`` when the MoE runs unfused (the decode steps' engines route + in-kernel): ``F.linear(float(h), w)``, the built-in gate's arithmetic outside its tensor-core router's rows. The + experts quantize their bf16 input themselves.""" + _record("m3_fp32_linear", h.shape[0] if h.dim() == 2 else 0) + return m3_fp32_linear(h, core.w[f"l{i}_router"]) + + +def _decode_router_logits(core, i: int, h: torch.Tensor) -> torch.Tensor: + """The router logits of a pure generation step: ``ROUTER_TC_GEN``'s rows take the TF32 tensor-core router when + the device co-schedules 16-CTA clusters (decided at prebuild) and the operands are the built-in gate's (2-D bf16, + unit column stride, 16-byte aligned rows); every other step the fp32 router (``_router_logits``).""" + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + ROUTER_TC_GEN.covers(rows) + and core.router_tc_cluster16 + and h.dtype == torch.bfloat16 + and h.stride(1) == 1 + and (h.stride(0) * 2) % 16 == 0 + ): + taken(ROUTER_TC_GEN.path, rows) + _record("m3_router_tc", rows) + return m3_router_tc(h, core.w[f"l{i}_router"], None, get_env_enable_pdl())[0] + return _router_logits(core, i, h) + + +def _routed_experts(core, i: int, x, x_sf, router_logits, do_finalize: bool): + """The trtllm-gen NVFP4 MoE runner over this rank's tensor-parallel slice of every expert, routing in-kernel. + + ``x_sf`` None means ``x`` is the bf16 activation, quantized here with the experts' global input scale. Returns + the finalized rows, or the unfinalized ``[gemm2 out, expert scales, expanded->permuted index]``. + """ + moe = core.moe[i] + if x_sf is None: + x, x_sf = fp4_quantize(x, moe.fc31_input_scale, 16, False, False) + rows = x.shape[0] + x_sf = x_sf.view(rows, -1).flatten() + _record("fp4_block_scale_moe_runner", rows, do_finalize) + outs = fp4_block_scale_moe_runner( + router_logits, + core.w[f"l{i}_router_bias"], + x, + x_sf.view(torch.float8_e4m3fn), + core.w[f"l{i}_e_w31"], + core.w[f"l{i}_e_s31"].view(torch.float8_e4m3fn), + None, + moe.act_alpha, + moe.act_beta, + moe.act_clamp, + core.w[f"l{i}_e_w2"], + core.w[f"l{i}_e_s2"].view(torch.float8_e4m3fn), + None, + moe.fc31_scale_c, + moe.fc31_alpha, + moe.fc2_alpha, + core.num_experts, + core.top_k, + None, + None, + core.moe_inter, + 0, + core.num_experts, + core.routed_scaling_factor, + ROUTING_MINIMAX2, + do_finalize, + act_type=ACT_SWIGLU, + topk_weights=None, + topk_ids=None, + output=None, + tune_max_num_tokens=core.max_num_tokens, + use_dp=False, + ) + return outs[0] if do_finalize else outs + + +def _split_chunk(rows: int, num_chunks: int) -> List[int]: + """The MoE scheduler's even split: the first ``rows % num_chunks`` chunks take one more row.""" + base, extra = divmod(rows, num_chunks) + return [base + 1] * extra + [base] * (num_chunks - extra) + + +# -------------------------------------------------------------------------------------------------------------------- +# Attention +# -------------------------------------------------------------------------------------------------------------------- + + +def _nvfp4_layer(manager, i: int) -> bool: + return bool(getattr(manager, "is_nvfp4_layer", lambda _layer_idx: False)(i)) + + +def _producer_covered(core) -> bool: + """Whether ``PRODUCER`` covers this target's sparse layers: an NVFP4 KV cache at the TP4 heads (16 query, one KV + and one index head), with the range on. A covered layer that cannot take ``m3_producer`` is a miss.""" + return ( + core.main_kv_is_nvfp4 + and (core.heads, core.kv_heads, core.index_heads) == (16, 1, 1) + and PRODUCER.covers("nvfp4_tp4") + ) + + +def _producer_layout(kv_data, kv_scale) -> bool: + """Whether a sparse layer's NVFP4 pools are the layout ``m3_producer`` is built for: [pages, 2, 1, 128, 64] with + its scale pool [pages, 2, 1, 128, 8].""" + return ( + kv_data is not None + and kv_scale is not None + and kv_data.dim() == 5 + and tuple(kv_data.shape[1:]) == (2, 1, 128, 64) + and kv_data.stride()[2:] == (128 * 64, 64, 1) + and kv_scale.dim() == 5 + and tuple(kv_scale.shape[1:]) == (2, 1, 128, 8) + and kv_scale.stride()[2:] == (128 * 8, 8, 1) + and kv_scale.shape[0] == kv_data.shape[0] + ) + + +def _nvfp4_fused_write_layout(data: torch.Tensor, scales: torch.Tensor, num_heads: int) -> bool: + """Reject invalid HND pools; select the fused producer only for its packed layout.""" + if ( + data.dim() != 5 + or scales.dim() != 5 + or data.shape[0] <= 0 + or data.shape[0] != scales.shape[0] + or tuple(data.shape[1:]) != (2, num_heads, 128, 64) + or tuple(scales.shape[1:]) != (2, num_heads, 128, 8) + or data.dtype not in (torch.int8, torch.uint8) + or scales.dtype != torch.uint8 + ): + raise ValueError("MiniMax-M3 NVFP4 data and scale caches must have matching HND geometry") + for cache in (data, scales): + if ( + cache.stride(4) != 1 + or cache.stride(3) != cache.shape[4] + or cache.stride(2) < 128 * cache.stride(3) + or cache.stride(1) < num_heads * cache.stride(2) + or cache.stride(0) < 2 * cache.stride(1) + ): + raise ValueError( + "MiniMax-M3 NVFP4 cache rows must be packed and heads/planes/pages must not overlap" + ) + if scales.stride(2) != 128 * 8: + raise ValueError("MiniMax-M3 NVFP4 MSA scale cache must have packed head blocks") + return data.stride(2) == 128 * 64 and data.stride(0) % 2 == 0 and data.stride(1) % 2 == 0 + + +def _producer_miss(covered: bool, rows: Optional[int], reason: str) -> None: + if covered: + missed(PRODUCER.path, rows, reason) + return None + + +def _sparse_qkv_producer( + core, i: int, packed, position_ids, kv_cache_manager, msa_idx_k_cache, msa_out_cache_loc +): + """The horizontal producer of a sparse layer: Q / index-Q (FP8) out, K / V / index-K written into the paged + caches (NVFP4 or FP8 main cache), for every step type (token-major). An NVFP4 cache at the TP4 heads takes + ``m3_producer`` (``PRODUCER``); a covered layer that cannot is a miss. Returns (q, index_q), or None for caches + neither kernel takes. + + ``kv_cache_manager``, ``msa_idx_k_cache`` (the per-layer index-K cache getter) and ``msa_out_cache_loc`` are the + calling target's ``step_args`` entries.""" + covered = _producer_covered(core) + rows = int(packed.shape[0]) + if packed.dtype != torch.bfloat16 or position_ids is None: + return _producer_miss( + covered, + rows, + "the packed QKV dtype, position ids, head dims, norm or RoPE configuration", + ) + manager = kv_cache_manager + if manager is None: + return _producer_miss(covered, rows, "the step has no KV cache manager") + buffers = manager.get_buffers(i, kv_layout="HND") + index_k_cache = msa_idx_k_cache(i) + out_cache_loc = msa_out_cache_loc + layer_uses_nvfp4 = _nvfp4_layer(manager, i) + scale_buffers = ( + manager.get_block_scale_buffers(i, kv_layout="HND") if layer_uses_nvfp4 else None + ) + kv_quant_scale = core.inv_kv_scales[i] if layer_uses_nvfp4 else None + if layer_uses_nvfp4: + supported_main = ( + buffers is not None + and scale_buffers is not None + and _nvfp4_fused_write_layout(buffers, scale_buffers, core.kv_heads) + and buffers.is_cuda + and scale_buffers.is_cuda + and scale_buffers.device == buffers.device + and kv_quant_scale.is_cuda + and kv_quant_scale.dtype == torch.float32 + and kv_quant_scale.numel() >= 3 + and kv_quant_scale.is_contiguous() + ) + else: + supported_main = ( + buffers is not None + and buffers.is_cuda + and buffers.dtype == torch.float8_e4m3fn + and buffers.dim() == 5 + and tuple(buffers.shape[1:]) == (2, core.kv_heads, 128, 128) + and buffers.stride(4) == 1 + and buffers.stride(3) == 128 + and buffers.stride(2) == 128 * 128 + and buffers.stride(1) >= core.kv_heads * buffers.stride(2) + and buffers.stride(0) >= 2 * buffers.stride(1) + and buffers.stride(0) % 4 == 0 + and buffers.stride(1) % 4 == 0 + ) + supported_index = ( + index_k_cache.is_cuda + and index_k_cache.dtype == torch.float8_e4m3fn + and index_k_cache.dim() == 4 + and tuple(index_k_cache.shape[1:]) == (1, 128, 128) + and index_k_cache.stride(3) == 1 + and index_k_cache.stride(2) == 128 + and index_k_cache.stride(1) >= 128 * 128 + and index_k_cache.stride(0) >= index_k_cache.stride(1) + and index_k_cache.stride(0) % 4 == 0 + and index_k_cache.stride(1) % 4 == 0 + and buffers is not None + and index_k_cache.shape[0] == buffers.shape[0] + ) + cos_sin = core.rotary_cos_sin + supported_rope = ( + cos_sin.is_cuda + and cos_sin.dtype == torch.float32 + and cos_sin.is_contiguous() + and cos_sin.dim() == 3 + and tuple(cos_sin.shape[1:]) == (2, 32) + ) + if ( + not supported_main + or not supported_index + or not supported_rope + or out_cache_loc is None + or not out_cache_loc.is_cuda + or out_cache_loc.dtype != torch.int32 + or not out_cache_loc.is_contiguous() + or out_cache_loc.numel() < rows + ): + return _producer_miss( + covered and layer_uses_nvfp4, + rows, + "the layer's NVFP4 / index caches, RoPE table, KV quant scale or cache slots are not the layout the " + "producer is built for", + ) + w = core.w + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + if ( + layer_uses_nvfp4 + and PRODUCER.covers("nvfp4_tp4") + and (core.heads, core.kv_heads, core.index_heads) == (16, 1, 1) + and _producer_layout(buffers, scale_buffers) + ): + early_trigger = PRODUCER_EARLY_TRIGGER and rows > PRODUCER_LATE_TRIGGER_MAX_ROWS + _record("m3_producer", rows, early_trigger) + q, index_q = m3_producer( + packed.contiguous(), + buffers.view(torch.uint8), + scale_buffers.view(torch.uint8), + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + core.heads, + get_env_enable_pdl(), + early_trigger, + PRODUCER_PREWAIT_STEP_INPUTS, + ) + # Every token count takes the producer: rows past a decode step log once as one line. + decode_rows = is_decode_step(rows) + taken( + PRODUCER.path, + rows if decode_rows else None, + f"nvfp4_tp4, {'entry' if early_trigger else 'exit'} trigger, " + f"prewait_step_inputs={PRODUCER_PREWAIT_STEP_INPUTS}" + if decode_rows + else "nvfp4_tp4, past the decode rows", + ) + else: + if layer_uses_nvfp4 and covered: + missed( + PRODUCER.path, + rows, + f"the NVFP4 pool {tuple(buffers.shape)} / scale pool {tuple(scale_buffers.shape)} is not the " + "producer's layout", + ) + _record("m3_qkv_insert", rows, layer_uses_nvfp4) + q, index_q = m3_qkv_insert( + packed.contiguous(), + buffers.view(torch.uint8) if layer_uses_nvfp4 else buffers, + scale_buffers, + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.heads, + core.kv_heads, + core.index_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + ) + return q.flatten(1), index_q.flatten(1) + + +def _qk_norm_rope( + core, i: int, qkv: torch.Tensor, position_ids, out_fp8: bool, kv_dest: Optional[tuple] = None +) -> torch.Tensor: + """The per-head Gemma RMSNorm of Q and K and the partial NeoX RoPE of a layer's packed [Q | K | V], in one kernel: + in place, or an FP8 copy (Q / K normed and roped, V cast) when ``out_fp8``. A dense layer's FP8 copy carries the + BF16 output's bits cast to E4M3 (``m3_dense_qk_norm_rope_fp8``, ``DENSE_QKV_FP8_VIA_BF16``); with ``kv_dest`` = + (the layer's HND pool, the step's cache slots, live tokens, a [tokens, q heads, 128] FP8 buffer) that kernel also + stores the live tokens' K / V rows into the pool and every row's Q into the buffer.""" + qkv = qkv.contiguous() + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + if ( + out_fp8 + and DENSE_QKV_FP8_VIA_BF16 + and i not in core.sparse_layers + and qkv.dtype == torch.bfloat16 + and core.head_dim == 128 + ): + _record("m3_dense_qk_norm_rope_fp8", qkv.shape[0], kv_dest is not None) + return m3_dense_qk_norm_rope_fp8( + qkv, + core.heads, + core.kv_heads, + core.kv_heads, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_q_norm"], + core.w[f"l{i}_k_norm"], + core.rope_theta, + True, + pos, + True, + *(kv_dest or ()), + ) + assert kv_dest is None, "only the dense FP8 kernel stores K / V" + args = ( + qkv, + core.heads, + core.kv_heads, + core.kv_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_q_norm"], + core.w[f"l{i}_k_norm"], + core.rope_theta, + True, + pos, + 1.0, # factor: no YARN + 0.0, # low + 0.0, # high + 1.0, # attention_factor + True, # is_qk_norm + True, # use_gemma + False, # use_mrope + 0, + 0, + ) + _record("qk_norm_rope", qkv.shape[0], out_fp8) + if out_fp8: + return m3_qk_norm_rope_fp8(*args) + return fused_qk_norm_rope(*args) + + +def _fp8_index_qk_norm_rope( + core, i: int, idx_qk: torch.Tensor, position_ids, msa_idx_k_cache, msa_out_cache_loc +) -> torch.Tensor: + """The index branch of an unfused sparse layer: index-Q in FP8 out, index-K into the FP8 index-K cache, normed and + roped in one kernel (the built-in ``_fused_fp8_index_qk_norm_rope``). ``msa_idx_k_cache`` / ``msa_out_cache_loc`` + are the calling target's ``step_args`` entries.""" + if position_ids is None or idx_qk.dtype != torch.bfloat16: + raise NotImplementedError( + "MiniMax-M3 fused FP8 indexer requires BF16 activations and position_ids." + ) + index_k_cache = msa_idx_k_cache(i) + if index_k_cache.dtype != torch.float8_e4m3fn: + raise ValueError( + "MiniMax-M3 fused FP8 indexer requires an E4M3 index-K cache, " + f"got {index_k_cache.dtype}." + ) + rows = int(idx_qk.shape[0]) + _record("m3_fp8_indexer_qk_norm_rope", rows) + return m3_fp8_indexer_qk_norm_rope( + idx_qk.contiguous(), + index_k_cache, + msa_out_cache_loc[:rows], + core.index_heads, + core.index_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_idx_q_norm"], + core.w[f"l{i}_idx_k_norm"], + core.rope_theta, + position_ids.reshape(-1).contiguous().to(torch.int32), + ).flatten(1) + + +# -------------------------------------------------------------------------------------------------------------------- +# Layer helpers both targets' bodies call: the Eagle3 capture hook and the norm after a layer +# -------------------------------------------------------------------------------------------------------------------- + + +def _plain_capture(spec_metadata, layer_id: int, hidden_states, residual) -> None: + """The in-tree Eagle3 hidden-state capture.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + +def _next_norm(core, i: int) -> torch.Tensor: + """The norm the boundary after layer ``i`` applies: the next layer's input norm, the final norm after the last.""" + return core.w[f"l{i + 1}_in_norm"] if i + 1 < core.num_layers else core.w["final_norm"] + + +# -------------------------------------------------------------------------------------------------------------------- +# The core +# -------------------------------------------------------------------------------------------------------------------- + +# Engine states (compiled kernel + workspace) per (device, token count[, shared fold]), shared by every layer: a step +# runs the layers one after another on one stream. +_ENGINE_STATES: Dict[tuple, object] = {} +_MOE16_STATES: Dict[tuple, object] = {} + + +# AR-DET: AllReduce AUTO's per-process timing autotuner made VR runs differ across server starts +# (0 / 888 same-node and 0 / 296 across node pairs with it off, bitwise to the +# modal autotuned output). The target turns it off for the whole process at init (AUTO then takes the C++ size / world +# heuristic, a pure function of the shape), unless TRTLLM_M31_ENABLE_AR_AUTOTUNE=1. An explicit +# TLLM_DISABLE_ALLREDUCE_AUTOTUNE is left as given. Read once, at import. +AR_AUTOTUNE_ENABLED = m31_enabled("AR_AUTOTUNE") # the m31_switches value contract (1/true/on/yes) + + +def _pin_allreduce_heuristic(rank: int) -> bool: + """Default the AllReduce autotune off for this process (see AR-DET); returns whether it is off. Logs one line.""" + if not AR_AUTOTUNE_ENABLED: + os.environ.setdefault("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "1") + off = os.environ.get("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "0") == "1" + if off: + logger.info(f"[M3.1] rank {rank}: AR_DET engaged (AllReduce autotune off for this process)") + logger.info( + f"M3.1 AR-DET: AllReduce autotune {'off (C++ size / world heuristic)' if off else 'on'} on rank {rank}" + ) + return off + + +# PREFILL-MNNVL: the built-in all-reduce of a prefill or mixed step's layer boundaries (the rows the multicast kernel +# does not take) runs MNNVL instead of AllReduce(AUTO)'s pick. On VR TP4 every strategy is cross-process deterministic +# there and MNNVL is the fastest, 1.7..29 us per call under AUTO at 128..8192 rows (plain and with the fused residual + +# RMSNorm). Up to PREFILL_AR_MNNVL_MAX_ROWS rows (MNNVL's workspace grows with the rows; larger steps keep AUTO); +# generation steps keep AUTO. TRTLLM_M31_DISABLE_PREFILL_AR_MNNVL=1 restores AUTO. Read once, at import. +PREFILL_AR_MNNVL = not m31_disabled("PREFILL_AR_MNNVL") +PREFILL_AR_MNNVL_MAX_ROWS = 8192 # capacity: measured: the largest step the strategy table covers + + +def _prefill_allreduce(core, rows: int): + """The built-in all-reduce of a prefill / mixed-step boundary of ``rows`` rows: MNNVL (PREFILL-MNNVL) up to its + rows, else the target's AUTO one.""" + if core.allreduce_mnnvl is not None and rows <= PREFILL_AR_MNNVL_MAX_ROWS: + taken("prefill_ar_mnnvl", None) + return core.allreduce_mnnvl + return core.allreduce + + +class MiniMaxM3Tp4ModelingV2(DecoderModel): + """The MiniMax-M3 text model of this target: its weights, its runtime resources, and the per-step dispatch.""" + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + _validate_runtime(model_config) + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + _pin_allreduce_heuristic(mapping.tp_rank) + self.mapping = mapping + self.vocab_size = cfg.vocab_size + d = _weights.MODEL_WEIGHTS.dims(self) + self.num_layers = d.num_layers + self.hidden = d.hidden + self.heads = d.heads + self.kv_heads = d.kv_heads + self.head_dim = d.head_dim + self.index_heads = d.index_heads + self.index_dim = d.index_dim + self.q_size = d.heads * d.head_dim + self.kv_size = d.kv_heads * d.head_dim + self.moe_inter = d.moe_inter + self.num_experts = d.num_experts + self.top_k = int(cfg.num_experts_per_tok) + self.routed_scaling_factor = float(getattr(cfg, "routed_scaling_factor", 1.0)) + self.rms_eps = float(cfg.rms_norm_eps) + self.swiglu_alpha = float(getattr(cfg, "swiglu_alpha", 1.702)) + self.swiglu_beta = 1.0 # the (up + 1) offset of SwiGLU-OAI + self.swiglu_limit = float(getattr(cfg, "swiglu_limit", 7.0)) + self.max_num_tokens = model_config.max_num_tokens + self.moe_max_num_tokens = model_config.moe_max_num_tokens + self.sparse_layers = frozenset(_sparse_layer_ids(cfg)) + self.moe_layers = frozenset(_moe_layer_ids(cfg)) + mode = model_config.quant_config.quant_mode + self.main_kv_is_fp8 = bool(mode is not None and mode.has_fp8_kv_cache()) + self.main_kv_is_nvfp4 = bool(mode is not None and mode.has_fp4_kv_cache()) + # The sparse layers' QKV and index projections: one five-way packed projection, or the QKV projection and a + # separate MXFP8 index projection (fuse_qkv_index_projection=False) run on two streams. + self.fused_projection = _weights.fused_projection(model_config.sparse_attention_config) + + self.w = _weights.MODEL_WEIGHTS.declare(d) + + # The engine reads these two off the module tree: the MXFP8 GEMM state it tunes, and the opt-in to the + # automatic FlashInfer MXFP8 path on decode graphs. + self.mxfp8 = _Mxfp8GemmState() + self._use_flashinfer_mxfp8_decode_graph_default = True + + # The built-in fused all-reduce + residual + RMSNorm of the layer boundaries the multicast kernel does not take. + self.allreduce = AllReduce( + mapping=mapping, strategy=model_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + # PREFILL-MNNVL: the prefill / mixed-step boundaries' all-reduce (None: switched off). + self.allreduce_mnnvl = ( + AllReduce(mapping=mapping, strategy=AllReduceStrategy.MNNVL, dtype=cfg.torch_dtype) + if PREFILL_AR_MNNVL + else None + ) + if self.allreduce_mnnvl is not None and self.allreduce_mnnvl.mnnvl_allreduce is not None: + # Size MNNVL's shared workspace for the largest step it takes now, so the growth lands before the KV cache + # is sized (the memory-estimation step runs more rows than the pin covers and takes AUTO). + get_or_scale_allreduce_mnnvl_workspace( + mapping, + cfg.torch_dtype, + buffer_size_bytes=MNNVLAllReduce.get_required_workspace_size( + PREFILL_AR_MNNVL_MAX_ROWS, cfg.hidden_size, mapping.tp_size, cfg.torch_dtype + ), + ) + + # Streams: the shared / routed expert overlap and, when the MoE chunks its input, the chunking overlap. + self.aux_stream_moe = torch.cuda.Stream() + chunked = self.moe_max_num_tokens < model_config.max_num_tokens * mapping.dp_size + self.aux_stream_moe_chunk = torch.cuda.Stream() if chunked else None + self.moe_events = {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers} + self.moe_chunk_events = { + i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers + } + # The unfused sparse layers' index branch overlaps the QKV branch on its own stream. + self.aux_stream_attn = None if self.fused_projection else torch.cuda.Stream() + self.attn_events = ( + {} + if self.fused_projection + else {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.sparse_layers} + ) + + # Partial NeoX RoPE (rotary_dim of head_dim): the cos/sin table every attention kernel reads. + rope = RopeParams.from_config(cfg) + self.rope_theta = rope.theta + self.rotary_dim = int(rope.dim) + self.rotary_cos_sin = rope.create_rope_const_params(interleave=False)[1].reshape( + rope.max_positions, 2, -1 + ) + + # One MSA attention backend per layer (dense and sparse; the step metadata is the backend's). + quant_config = model_config.get_quant_config() + self.attn: List[MiniMaxM3MsaSparseAttention] = [] + for i in range(self.num_layers): + backend = create_attention( + model_config.attn_backend, + i, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=None, + quant_config=quant_config, + skip_create_weights_in_init=model_config.skip_create_weights_in_init, + q_scaling=1.0, + sparse_params=model_config.sparse_attention_config.to_sparse_params( + pretrained_config=cfg, layer_idx=i + ), + ) + backend.update_quant_config(quant_config) + if not isinstance(backend, MiniMaxM3MsaSparseAttention): + raise ValueError( + f"layer {i}: expected the MiniMax-M3 MSA backend, got {type(backend).__name__}" + ) + if i not in self.sparse_layers: + # The KV split trtllm-gen dense decode runs at, per request count. + backend.dense_decode_max_seq_len_by_batch = DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH + self.attn.append(backend) + + # Built once the weights are real (``post_load_weights``): the KV (Q, K, V) scales, the MoE scalars, the + # decode engines and workspaces, the two targets. + self.kv_scales: Dict[int, torch.Tensor] = {} + self.inv_kv_scales: Dict[int, torch.Tensor] = {} + self.moe: Dict[int, _MoeScalars] = {} + self.mx_alpha: Optional[torch.Tensor] = None + self.moe_engines: Dict[int, Dict[Tuple[int, bool, bool], Optional[object]]] = {} + self.moe16_engines: Dict[int, Dict[int, Optional[object]]] = {} + self.embed_owners: Dict[tuple, DecodeEmbedding] = {} + # Whether the 9..16-row generation router (ROUTER_TC_GEN) runs; decided at prebuild. + self.router_tc_cluster16 = False + # Each sparse layer's selected block ids per query row at its last long-context decode, read by the previous + # sparse layer's decode to prefetch this layer's KV (``SPARSE_DECODE_LC_PREFETCH_NEXT_KV``). + self.n1_ids: Dict[int, torch.Tensor] = ( + { + i: torch.full( + (N1_MAX_ROWS, 16), + -1, + dtype=torch.int32, + device=torch.device("cuda", torch.cuda.current_device()), + ) + for i in sorted(self.sparse_layers) + } + if SPARSE_DECODE_LC_PREFETCH_NEXT_KV + else {} + ) + self._built_for: Optional[tuple] = None + self._prefill: Optional[PrefillTarget] = None + self._decode: Optional[DecodeTarget] = None + + def sm_scale(self, i: int) -> float: + return (self.head_dim**-0.5) / float(self.attn[i].q_scaling) + + # ---------------------------------------------------------------------------------------------- after the load + + def post_load_weights(self) -> None: + """Derive the kernels' state from the loaded weights, then build both targets and every decode kernel. + + Idempotent per set of weight storages: the loader's post-load walk reaches this module directly and through + the shell, and a second call on the same storages does nothing. + """ + identity = ( + self.w["embed"].data_ptr(), + self.w["l0_in_norm"].data_ptr(), + self.w["final_norm"].data_ptr(), + ) + if self._built_for == identity: + return + device = self.w["embed"].device + derived = nn.Module() + derived.register_buffer( + "mx_alpha", torch.ones(1, dtype=torch.float32, device=device), persistent=False + ) + for i in range(self.num_layers): + # No KV calibration in the checkpoint: unit (Q, K, V) scales and their reciprocals. + derived.register_buffer( + f"l{i}_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + derived.register_buffer( + f"l{i}_inv_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + for i in sorted(self.moe_layers): + for name, value in dataclasses.asdict(self._derive_moe(i)).items(): + derived.register_buffer(f"l{i}_{name}", value, persistent=False) + self.derived = derived + self.mx_alpha = derived.mx_alpha + self.kv_scales = {i: getattr(derived, f"l{i}_kv_scales") for i in range(self.num_layers)} + self.inv_kv_scales = { + i: getattr(derived, f"l{i}_inv_kv_scales") for i in range(self.num_layers) + } + self.moe = { + i: _MoeScalars( + **{ + f.name: getattr(derived, f"l{i}_{f.name}") + for f in dataclasses.fields(_MoeScalars) + } + ) + for i in sorted(self.moe_layers) + } + self._build_embed_owners() + self.prebuild_kernels() + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + self._built_for = identity + + def cache_derived_state(self) -> None: + """The staged loaders' entry to the same build (``post_load_weights``).""" + self.post_load_weights() + + def _derive_moe(self, i: int) -> _MoeScalars: + """The routed experts' kernel scalars from the checkpoint's per-tensor scales (trtllm-gen NVFP4 layout). + + Global input scales: the reciprocal of the largest per-expert input scale (gate and up must agree). Per expert: + ``alpha = 1 / (input_scale * (1 / weight_scale_2))`` for the gate (fc31_alpha), the up half (its GEMM2-input + rescale ``fc31_scale_c = fc2_input_scale * up_alpha``) and the down projection; SwiGLU-OAI's activation + constants per expert, with beta and the clamp in the pre-dequant GEMM output domain (divided by fc31_alpha). + """ + w = self.w + isc1, isc3, isc2 = w[f"l{i}_e_isc1"].data, w[f"l{i}_e_isc3"].data, w[f"l{i}_e_isc2"].data + if not torch.allclose(isc1, isc3): + raise ValueError(f"layer {i}: the routed experts' w1 and w3 input scales differ") + fc31_input_scale = isc1.max().reciprocal() + fc2_input_scale = isc2.max().reciprocal() + ws2_1, ws2_3, ws2_2 = ( + w[f"l{i}_e_ws2_1"].data, + w[f"l{i}_e_ws2_3"].data, + w[f"l{i}_e_ws2_2"].data, + ) + fc31_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_1)) + up_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_3)) + fc2_alpha = 1.0 / (fc2_input_scale * (1.0 / ws2_2)) + fc31_scale_c = fc2_input_scale * up_alpha + n = self.num_experts + device = fc31_alpha.device + act_alpha = torch.full((n,), self.swiglu_alpha, dtype=torch.float32, device=device) + act_beta = torch.full((n,), self.swiglu_beta, dtype=torch.float32, device=device) + act_clamp = torch.full((n,), self.swiglu_limit, dtype=torch.float32, device=device) + act_beta.div_(fc31_alpha) + act_clamp.div_(fc31_alpha) + return _MoeScalars( + fc31_input_scale=fc31_input_scale, + fc2_input_scale=fc2_input_scale, + fc31_alpha=fc31_alpha, + fc2_alpha=fc2_alpha, + fc31_scale_c=fc31_scale_c, + act_alpha=act_alpha, + act_beta=act_beta, + act_clamp=act_clamp, + ) + + def _build_embed_owners(self) -> None: + """The fused embedding + first RMSNorm's caller-owned buffers, one per (rows, id dtype) of a decode step.""" + table, weight = self.w["embed"], self.w["l0_in_norm"] + self.embed_owners = { + (rows, dtype): DecodeEmbedding(table, weight, self.rms_eps, rows, dtype) + for rows in range(1, EMBED_FIRSTNORM_MAX_TOKENS + 1) + for dtype in (torch.int32, torch.int64) + } + + def projections(self) -> List[Tuple[str, str, Optional[str]]]: + """Every MXFP8 projection in layer order: (weight key, scale key, the boundary its GEMV pushes into).""" + out = [] + for i in range(self.num_layers): + out.append((f"l{i}_qkv_w", f"l{i}_qkv_s", None)) + out.append((f"l{i}_o_w", f"l{i}_o_s", "attn")) + if i in self.moe_layers: + out.append((f"l{i}_sh_gu_w", f"l{i}_sh_gu_s", None)) + out.append((f"l{i}_sh_dn_w", f"l{i}_sh_dn_s", "moe")) + else: + out.append((f"l{i}_mlp_gu_w", f"l{i}_mlp_gu_s", None)) + out.append((f"l{i}_mlp_dn_w", f"l{i}_mlp_dn_s", "dense")) + return out + + def decode_query_lens(self) -> List[int]: + """The query rows per request a decode step can have: 1, and 1 + each draft length speculation runs.""" + lens = {1} + spec = getattr(self.model_config, "spec_config", None) + if spec is not None: + # The verify width the config itself declares, whatever it names its draft count. + per_step = getattr(spec, "tokens_per_gen_step", None) + if isinstance(per_step, int) and per_step > 1: + lens.add(per_step) + for drafts in ( + getattr(spec, "max_draft_len", None), + getattr(spec, "max_total_draft_tokens", None), + *(getattr(spec, "draft_len_schedule", None) or {}).values(), + ): + if drafts: + lens.add(1 + int(drafts)) + return sorted(lens) + + def prebuild_kernels(self) -> Dict[str, object]: + """Compile (and load) every kernel a decode forward dispatches to that would otherwise compile on its first + call: each projection's decode GEMVs (every row count, activation and boundary push), the MoE engines of + ``MOE_ENGINE`` / ``MOE16_ENGINE``'s token counts, and the index scorer at the decode query lengths. Runs on the + current (main) stream before the executor's warmup, in the same order on every rank, then synchronizes; after + it, GEMV compiles are frozen and the forward only looks kernels up (a missing one is a miss). It also builds + the T2SC scorer (``INDEX_SCORE_T2SC_PATH``), the 384-thread long-row select (``LONG_SELECT_DSL``) and the dense + layers' full-context decode plans (``FULLCTX_DECODE``). Returns the seconds each part took and the scorer + keys.""" + times: Dict[str, object] = {} + # The 9..16-row generation router: decided once, from the device, never inside a CUDA-graph capture. + # The built-in gate's conditions, mirrored exactly: 128 experts, a hidden size in 1024s, 16-CTA clusters. + self.router_tc_cluster16 = ( + self.num_experts == 128 + and self.hidden % 1024 == 0 + and bool(torch.ops.trtllm.minimax_m3_router_tc_cluster16(self.hidden)) + ) + logger.info( + f"[M3 prebuild] rank {mpi_rank()}: TF32 tensor-core router for {ROUTER_TC_GEN_MIN_ROWS}.." + f"{ROUTER_TC_GEN_MAX_ROWS}-row generation steps: " + + ( + "engaged (16-CTA clusters)" + if self.router_tc_cluster16 + else "not used (no 16-CTA clusters)" + ) + ) + if not self.mxfp8.quant_method.use_cutlass: + raise ValueError("the MiniMax-M3 decode dispatch needs the CUTLASS MXFP8 GEMM") + if PREPROCESS_TOKMAP_FIX: + # Every path that can run the stock generation preprocess (the drafter's full-context decode or its + # fallback, any trtllm-gen fallback), so it is installed whatever else is enabled. + _tokmap_guard.install( + torch.device("cuda", torch.cuda.current_device()), + DECODE_MAX_TOKENS, + lambda rows: taken(PREPROCESS_TOKMAP_PATH, rows, "one-request token map"), + lambda rows, reason: missed(PREPROCESS_TOKMAP_PATH, rows, reason), + ) + mx_gemv.freeze(False) + use_pdl = get_env_enable_pdl() + world = int(self.mapping.tp_size) + start = time.perf_counter() + for wk, sk, push in self.projections(): + weight, scale = self.w[wk], self.w[sk] + mx_gemv.ensure_compiled(weight, scale, use_pdl) + n, k = weight.shape + for rows, x_bf16, ar in _gemv_variants(n, k, push, world): + # An MXFP8 call this target gives a (split, ring) override (``mx_gemv_cfg``, as ``_linear`` passes + # it) compiles that config. + cfg = None if x_bf16 else mx_gemv_cfg(n, k, rows) + if cfg is not None: + _precompile_cfg(weight, scale, rows, cfg, use_pdl, ar=ar) + else: + mx_gemv.precompile(weight, scale, rows, x_bf16, use_pdl=use_pdl, ar=ar) + torch.cuda.synchronize() + times["gemv_s"] = time.perf_counter() - start + start = time.perf_counter() + for i in sorted(self.moe_layers): + self._prebuild_engines(i) + torch.cuda.synchronize() + times["moe_s"] = time.perf_counter() - start + start = time.perf_counter() + times["scorer_keys"] = self._prebuild_index_scorer( + self.decode_query_lens(), self.w["embed"].device + ) + times["scorer_t2sc_keys"] = self._prebuild_index_scorer_t2sc( + self.decode_query_lens(), self.w["embed"].device + ) + torch.cuda.synchronize() + times["scorer_s"] = time.perf_counter() - start + start = time.perf_counter() + times["select_keys"] = [] + if long_select_dsl_keys(): + # The 384-thread long-row select (LONG_SELECT_DSL): one kernel per launch key. + times["select_keys"] = _select_long_dsl_kernel.prebuild( + long_select_dsl_keys(), self.w["embed"].device + ) + torch.cuda.synchronize() + times["select_s"] = time.perf_counter() - start + start = time.perf_counter() + times["fullctx_plans"] = 0 + if FULLCTX_DECODE.keys: + # The dense layers' full-context decode: one compiled PrimTS plan per covered (rows, requests). + times["fullctx_plans"] = _fullctx_decode_kernel.prebuild( + self.w["embed"].device, + {key: FULLCTX_DECODE_RECIPES[key] for key in FULLCTX_DECODE.keys}, + FULLCTX_DECODE_MAX_KV_LEN, + FULLCTX_DECODE_OWN_REDUCER and _fullctx_decode_kernel.has_own_reducer(), + get_env_enable_pdl(), + ) + torch.cuda.synchronize() + times["fullctx_s"] = time.perf_counter() - start + start = time.perf_counter() + times["proxy_variants"] = self._prebuild_msa_proxy() if MSA_PROXY_PREBUILD else [] + times["proxy_s"] = time.perf_counter() - start + mx_gemv.freeze(True) + logger.info( + "[M3 prebuild] decode kernels built before warmup: GEMVs %.1f s, MoE engines %.1f s, index scorer " + "%.1f s (%d variants), long-row select %.1f s (%d variants), full-context decode %.1f s (%d plans), " + "MSA indexer proxy %.1f s (%d variants)" + % ( + times["gemv_s"], + times["moe_s"], + times["scorer_s"], + len(times["scorer_keys"]), + times["select_s"], + len(times["select_keys"]), + times["fullctx_s"], + times["fullctx_plans"], + times["proxy_s"], + len(times["proxy_variants"]), + ) + ) + return times + + # ------------------------------------------------------------------------------------------ the MoE engines + + def _expert_operands(self, i: int): + """The routed experts' operands the engines read (trtllm-gen NVFP4 SwiGLU-OAI layout), and the input scale.""" + w, m = self.w, self.moe[i] + tensors = [ + w[f"l{i}_e_w31"].data, + w[f"l{i}_e_s31"].data, + w[f"l{i}_e_w2"].data, + w[f"l{i}_e_s2"].data, + m.fc31_alpha, + m.fc31_scale_c, + m.act_alpha, + m.act_beta, + m.act_clamp, + m.fc2_alpha, + ] + return tensors, m.fc31_input_scale + + def _shared_weights(self, i: int): + w = self.w + return ( + w[f"l{i}_sh_gu_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_gu_s"].data, + w[f"l{i}_sh_dn_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_dn_s"].data, + ) + + def engine(self, i: int, num_tokens: int, shared: bool) -> Optional[object]: + """Layer ``i``'s prebuilt routed-expert engine for ``num_tokens`` (1..8), router folded in (and the shared + expert with ``shared``). None after a miss: the build failed or was never made (a forward never builds one).""" + return self._lookup( + i, self.moe_engines[i], (num_tokens, True, shared), MOE_ENGINE.path, num_tokens + ) + + def moe16_engine(self, i: int, num_tokens: int) -> Optional[object]: + """Layer ``i``'s prebuilt 9..13-row engine (router, routed and shared experts in one kernel), or None.""" + if not MOE16_MIN_TOKENS <= num_tokens <= MOE16_MAX_TOKENS: + return None + return self._lookup(i, self.moe16_engines[i], num_tokens, MOE16_ENGINE.path, num_tokens) + + def _lookup(self, i: int, engines: dict, key, path: str, num_tokens: int) -> Optional[object]: + if key not in engines: + what = f"layer {i}'s engine" + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, what) + else: + missed(path, num_tokens, f"{what} was not built before warmup") + return None + engine = engines[key] + if engine is None: + missed(path, num_tokens, "its build failed (logged)") + return engine + + def _prebuild_engines(self, i: int) -> None: + """Build (compile, on first use of each shared state, and warm) every engine a decode step of layer ``i`` can + take: ``MOE_ENGINE``'s token counts with the shared expert folded in, and ``MOE16_ENGINE``'s. Whether an + engine is kept is agreed across the TP group (one host collective per engine): a build that fails on one rank + drops that engine on every rank, so the ranks never mix an engine (and its boundary push) with the generic + MoE.""" + engines = self.moe_engines.setdefault(i, {}) + engines16 = self.moe16_engines.setdefault(i, {}) + for num_tokens in sorted(MOE_ENGINE.keys): + key = (num_tokens, True, True) + if key not in engines: + engines[key] = m3_mc_allreduce.tp_agree_built( + self.mapping, + self._try_build( + i, MOE_ENGINE.path, num_tokens, self._build_engine, shared=True + ), + f"[M3 dispatch] {MOE_ENGINE.path} at {num_tokens} rows: layer {i}'s engine", + ) + for num_tokens in sorted(MOE16_ENGINE.keys): + if num_tokens not in engines16: + engines16[num_tokens] = m3_mc_allreduce.tp_agree_built( + self.mapping, + self._try_build(i, MOE16_ENGINE.path, num_tokens, self._build_moe16_engine), + f"[M3 dispatch] {MOE16_ENGINE.path} at {num_tokens} rows: layer {i}'s engine", + ) + + def _try_build(self, i: int, path: str, num_tokens: int, build, **kwargs) -> Optional[object]: + try: + return build(i, num_tokens, **kwargs) + except (ImportError, RuntimeError, ValueError) as err: + logger.warning( + f"[M3 dispatch] {path} at {num_tokens} rows: layer {i}'s engine failed to build: {err}" + ) + return None + + def _build_engine(self, i: int, num_tokens: int, shared: bool) -> object: + tensors, scale = self._expert_operands(i) + device = tensors[0].device + sms = torch.cuda.get_device_properties(device).multi_processor_count + kwargs = {} + if shared: + kwargs["shared_weights"] = self._shared_weights(i) + state = _ENGINE_STATES.get((str(device), num_tokens, shared)) + if state is None: + state = _ENGINE_STATES[(str(device), num_tokens, shared)] = M3MoeState( + device, num_tokens=num_tokens, num_ctas=sms - MOE_ENGINE_RESERVED_SMS, shared=shared + ) + engine = state.layer( + *tensors, + self.w[f"l{i}_router_bias"].data, + router_weight=self.w[f"l{i}_router"].data, + fc31_input_scale=scale.reshape(1).float(), + **kwargs, + ) + # Compile (first use of the state) and warm. + m3_moe_engine( + torch.zeros(num_tokens, tensors[0].shape[-1] * 2, dtype=torch.bfloat16, device=device), + engine, + ) + return engine + + def _build_moe16_engine(self, i: int, num_tokens: int) -> object: + tensors, scale = self._expert_operands(i) + device = tensors[0].device + state = _MOE16_STATES.get((str(device), num_tokens)) + if state is None: + state = _MOE16_STATES[(str(device), num_tokens)] = M3Moe16State( + device, num_tokens=num_tokens + ) + engine = state.layer( + *tensors, + self.w[f"l{i}_router_bias"].data, + router_weight=self.w[f"l{i}_router"].data, + fc31_input_scale=scale.reshape(1).float(), + shared_weights=self._shared_weights(i), + ) + # Compile (first use of the state) and warm. + m3_moe16_engine( + torch.zeros(num_tokens, tensors[0].shape[-1] * 2, dtype=torch.bfloat16, device=device), + engine, + ) + return engine + + # ------------------------------------------------------------------------------------------ the index scorer + + def _scorer_variants(self, query_lens) -> list: + """The scorer keys a decode step can take: every sparse layer's (index dtype, index heads per rank, index dim) + at each decode query length, with the early-issue schedule and without it, where the kernel serves it.""" + runner = _index_score_kernel.IndexScoreRunner + keys = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + config = backend.m3_config + dtype = torch.float8_e4m3fn if backend.indexer_kv_dtype == "fp8" else torch.bfloat16 + for qlen in sorted(set(query_lens)): + if not runner.is_supported( + q_dtype=dtype, + num_heads=config.num_index_heads, + head_dim=config.sparse_index_dim, + page_size=runner.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ): + continue + for early in (True, False): + key = ( + dtype, + int(config.num_index_heads), + int(qlen), + int(config.sparse_index_dim), + int(INDEX_SCORE_SPLIT_K), + bool(early), + ) + if key not in keys: + keys.append(key) + return keys + + def _prebuild_msa_proxy(self) -> list: + """Compile and load every fmha_sm100 variant the MSA indexer's proxy pass can pick (OnlyScore over the paged + index-K: one KV head, page 128, no split-KV), so no context chunk compiles one after the server is ready (a + 65..128-token chunk's variant did, stalling the steps behind it by 0.8-1.6 ms). The variant follows only the + longest query row: fmha_sm100 packs a row of up to 32 tokens by its pack factor, then takes the 128- or 256-row + query tile and the single-warpgroup kernel from the packed length, so query lengths 1..129 reach every variant. + Compile and load only, nothing runs: no numerics change. fmha_sm100 keys its cache by its installed sources, so + a source overlay builds its own variants. Returns the variant names.""" + require_msa_module() + from fmha_sm100 import api as fmha_api + from fmha_sm100 import jit as fmha_jit + + only_score, page = 2, 128 + names = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + dtype = torch.float8_e4m3fn if backend.indexer_kv_dtype == "fp8" else torch.bfloat16 + heads = int(backend.m3_config.num_index_heads) + for qlen in range(1, 130): + pack = fmha_api._compute_pack_factor(qlen, heads, 1) + packed = qlen * pack + key = ( + fmha_jit._dlpack_dtype_code(dtype), + 128 if packed <= 128 else 256, + packed <= 64, + only_score, + page, + False, + pack, + ) + name, _ = fmha_jit._variant_key_from_runtime(*key) + if name not in names: + fmha_jit.get_fmha_variant(*key) + names.append(name) + return names + + def _prebuild_index_scorer(self, query_lens, device) -> list: + """Compile every scorer key and launch each once on real-shaped dummy operands (loading its module), on the + current stream. The kernel's batch, token, page and block counts are symbolic: its compile needs no step + metadata. Returns the keys.""" + runner = _index_score_kernel.IndexScoreRunner + keys = self._scorer_variants(query_lens) + page = runner.SUPPORTED_PAGE_SIZE + for key in keys: + dtype, heads, qlen, dim, split_k, early = key + runner._compile(dtype, heads, qlen, dim, split_k, early) + rows = 1 if early else INDEX_SCORE_EARLY_MAX_REQUESTS + 1 + blocks = 2 + tokens = rows * qlen + idx_q = torch.zeros(tokens, heads, dim, dtype=torch.bfloat16, device=device).to(dtype) + idx_k = torch.zeros( + rows * blocks, 1, page, dim, dtype=torch.bfloat16, device=device + ).to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device=device).view( + rows, blocks + ) + seq_lens = torch.full((rows,), blocks * page, dtype=torch.int32, device=device) + score = torch.empty(heads, blocks, tokens, dtype=torch.float32, device=device) + runner.kernel_cache[key]( + idx_q, idx_k.squeeze(1), block_table, score.transpose(1, 2), seq_lens + ) + return keys + + def _prebuild_index_scorer_t2sc(self, query_lens, device) -> list: + """T2SC (``INDEX_SCORE_T2SC_PATH``): compile this target's copy of the TP2 / TP8 scorer at every covered request + count's schedule (``index_score_t2sc_schedule``) for every sparse layer geometry and decode query length, and + launch each once on real-shaped dummy operands (loading its module). Returns the keys.""" + runner = _index_score_t2sc_kernel.IndexScoreRunner + keys = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + config = backend.m3_config + dtype = torch.float8_e4m3fn if backend.indexer_kv_dtype == "fp8" else torch.bfloat16 + for qlen in sorted(set(query_lens)): + if not runner.is_supported( + q_dtype=dtype, + num_heads=config.num_index_heads, + head_dim=config.sparse_index_dim, + page_size=runner.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ): + continue + for requests in sorted(INDEX_SCORE_T2SC_PATH.keys): + split_k, early = index_score_t2sc_schedule(requests) + key = ( + dtype, + int(config.num_index_heads), + int(qlen), + int(config.sparse_index_dim), + int(split_k), + bool(early), + ) + if all(key != seen for seen, _ in keys): + keys.append((key, requests)) + page = runner.SUPPORTED_PAGE_SIZE + for key, rows in keys: + dtype, heads, qlen, dim, split_k, early = key + runner._compile(dtype, heads, qlen, dim, split_k, early) + blocks = 2 + tokens = rows * qlen + idx_q = torch.zeros(tokens, heads, dim, dtype=torch.bfloat16, device=device).to(dtype) + idx_k = torch.zeros( + rows * blocks, 1, page, dim, dtype=torch.bfloat16, device=device + ).to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device=device).view( + rows, blocks + ) + seq_lens = torch.full((rows,), blocks * page, dtype=torch.int32, device=device) + score = torch.empty(heads, blocks, tokens, dtype=torch.float32, device=device) + runner.kernel_cache[key]( + idx_q, idx_k.squeeze(1), block_table, score.transpose(1, 2), seq_lens + ) + return [key for key, _ in keys] + + # --------------------------------------------------------------------------------------------- the dispatch + + def _select_target(self, attn_metadata: AttentionMetadata): + """Which target runs this step: one host-side integer, fixed per CUDA-graph capture. A step with context rows + (a mixed step included) is prefill's; every pure generation step is decode's, whatever its width.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.IntTensor] = None, + position_ids: Optional[torch.IntTensor] = None, + inputs_embeds: Optional[torch.FloatTensor] = None, + spec_metadata: Optional[SpecMetadata] = None, + capture: Callable[..., None] = _plain_capture, + capture_target: Optional[Callable[..., Any]] = None, + **kwargs, + ) -> torch.Tensor: + """``capture_target``: the worker's capture-norm target (``CAPTURE_NORM_IN_BOUNDARY``) for a capture layer + on the decode fast path; None: the capture runs its own norm.""" + if (input_ids is None) ^ (inputs_embeds is not None): + raise ValueError("You must specify exactly one of input_ids or inputs_embeds") + if kwargs.get("lora_params") is not None: + raise ValueError("this MiniMax-M3 target serves no LoRA adapters") + return self._select_target(attn_metadata).forward( + attn_metadata, + input_ids, + position_ids, + inputs_embeds, + spec_metadata, + capture, + capture_target, + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# The targets +# -------------------------------------------------------------------------------------------------------------------- + + +class PrefillTarget: + """A step with context rows: prefill chunks, and the mixed steps that carry generation rows beside them. Every + layer runs in its generic form; the attention core is the MSA backend's for every row.""" + + def __init__(self, core: MiniMaxM3Tp4ModelingV2) -> None: + self.core = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it; read once here rather than per forward. + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + Counts and per-step host scalars are read here; device tensors are the metadata's persistent buffers, held + by reference; ``msa_idx_k_cache`` is the metadata's per-layer index-K cache getter. ``metadata`` is the + prepared object itself: the MSA backend's step (``m3_msa_attention``) takes it as its argument. + """ + return dict( + num_contexts=md.num_contexts, + num_ctx_tokens=md.num_ctx_tokens, + num_tokens=md.num_tokens, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + metadata=md, + ) + + def forward( + self, + attn_metadata, + input_ids, + position_ids, + inputs_embeds, + spec_metadata, + capture, + capture_target, + ): + """``capture_target`` is unused: the generic boundary takes no multicast all-reduce at this TP + (``GENERIC_MC_AR_MAX_TOKENS`` 0), so a capture layer always runs its own capture norm.""" + core = self.core + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) prefill CUDA graphs" + ) + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + _record("prefill", int(step["num_tokens"]), int(step["num_contexts"])) + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + hidden, residual = self._layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + return hidden + + def _swiglu_mlp( + self, h: torch.Tensor, prefix: str, low_m_gemm: bool, dense: bool = False + ) -> torch.Tensor: + """A gated MXFP8 MLP (the dense MLP with ``dense``, the shared expert): gate_up, SwiGLU-OAI, down, no + reduction.""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s", low_m_gemm) + if dense: + act = _dense_swiglu(core, gu) + else: + act = m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + out, _ = _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm) + return out + + def _routed_chunks(self, i: int, x, router_logits): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: + the MoE scheduler's even split, even chunks on the MoE chunking stream.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + # CTXNORMQ: the boundary already wrote this activation's NVFP4 form. + fp4 = _CTXNORMQ_FP4.take(x) if CTXNORMQ else None + if num_chunks <= 1: + if fp4 is not None: + return _routed_experts(core, i, fp4[0], fp4[1], router_logits, True) + return _routed_experts(core, i, x, None, router_logits, True) + sizes = _split_chunk(rows, num_chunks) + xs, logits = x.split(sizes), router_logits.split(sizes) + sfs = [None] * num_chunks + if fp4 is not None: + xs = fp4[0].split(sizes) + sfs = [t.flatten() for t in fp4[1].view(rows, -1).split(sizes)] + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + else: + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor): + """Router + routed experts, finalized: the fp32 router logits and the experts quantizing their bf16 input + (chunked).""" + return self._routed_chunks(i, h, _router_logits(self.core, i, h)) + + def _moe(self, i: int, h: torch.Tensor, *, return_parts: bool): + """The MoE block of layer ``i`` without its reduction. Returns ``(out, routed)``: with ``return_parts`` the + shared and routed parts (routed None when the engine folded the shared expert in), else their sum in + ``out``.""" + core = self.core + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + h.dim() == 2 + and h.dtype == torch.bfloat16 + and 1 <= rows <= MOE_ENGINE_MAX_TOKENS + and rows >= MOE_FOLD_MIN_TOKENS + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + _record("m3_moe_engine", rows, False) + folded = m3_moe_engine(h.contiguous(), engine, None, None) + return folded, None + routed_out, shared_out = maybe_execute_in_parallel( + lambda: self._routed(i, h), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False), + core.moe_events[i][0], + core.moe_events[i][1], + # The warmup tuning pass profiles the shared expert's GEMMs: keep it on the main stream, between the + # same collectives on every rank (the capture pass and serving still overlap it on the aux stream). + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if return_parts: + return shared_out, routed_out + # In-place add into the shared output to avoid allocating a temporary. + return shared_out.add_(routed_out), None + + def _generic_boundary(self, x, x2, residual, norm_w): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce.""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + if norm_w is not core.w["final_norm"]: + fused = _ctxnormq_allreduce_norm( + core, x, residual, norm_w, None, _prefill_allreduce(core, x.shape[0]) + ) + if fused is not None: + return fused + return _native_allreduce_norm( + core, x, residual, norm_w, _prefill_allreduce(core, x.shape[0]) + ) + + def _layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm( + core, + hidden, + residual, + core.w[f"l{i}_post_norm"], + _prefill_allreduce(core, hidden.shape[0]), + ) + ) + else: + fused = _ctxnormq_allreduce_norm( + core, + hidden, + residual, + core.w[f"l{i}_post_norm"], + core.moe[i].fc31_input_scale if i in core.moe_layers else None, + _prefill_allreduce(core, hidden.shape[0]), + ) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm( + core, + hidden, + residual, + core.w[f"l{i}_post_norm"], + _prefill_allreduce(core, hidden.shape[0]), + ) + ) + if i in core.moe_layers: + out, routed = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out = self._swiglu_mlp(hidden, f"l{i}_mlp", True, dense=True) + routed = None + normed, updated = self._generic_boundary(out, routed, residual, _next_norm(core, i)) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + capture(spec_metadata, i, updated, None) + return normed, updated + + def _attention(self, i: int, position_ids, hidden, step): + """Attention of layer ``i`` up to its o_proj (no reduction). Returns (o_proj output, pushed): a prefill step + arms no boundary push, so pushed is always False.""" + core = self.core + if i not in core.sparse_layers: + # Dense layers 0-2: QKV, the fused norm + RoPE (FP8 q/k/v for an FP8 cache, else in place in bf16), the + # MSA backend (it writes the layer's cache), o_proj. + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + if not core.fused_projection: + return self._unfused_sparse_attention(i, position_ids, hidden, step) + packed, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + ) + if horizontal is None: + raise RuntimeError( + f"MiniMax-M3 sparse layer {i}: no horizontal producer takes this step's caches (an FP8 or NVFP4 HND " + "pool with an FP8 index-K cache)" + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output.""" + core = self.core + if _producer_covered(core): + # Not a miss: the producer packs the five-way projection's output, which this configuration lacks. + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): the MSA + backend (``m3_msa_attention``: the KV write unless a producer wrote the caches, the sparse layers' top-k block + selection, the FMHA over the context rows and any generation rows beside them).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + live = int(step["num_tokens"]) + q_live, out_live = q[:live], output[:live] + k_live = k[:live] if k is not None else None + v_live = v[:live] if v is not None else None + idx_q_live = idx_q[:live] if idx_q is not None else None + sparse = i in core.sparse_layers + nvfp4 = core.main_kv_is_nvfp4 + if GQAMIX and sparse: + _gqamix_register(core.attn[i]) + _record("m3_msa_attention", sparse, k_live is not None) + m3_msa_attention( + core.attn[i], + q_live, + k_live, + v_live, + idx_q_live, + step["metadata"], + out_live, + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + return output + + +class DecodeTarget: + """A pure generation step (no context rows), of any width. + + Up to ``DECODE_MAX_TOKENS`` tokens with 1-D input ids, per layer: attention (o_proj all-reduce deferred, its rows + pushed into the boundary all-reduce up to 16 tokens) -> attention boundary (all-reduce + residual + post-attention + RMSNorm) -> MoE (the 1..8 / 9..13-row engines, the first pushing its sum into the boundary all-reduce; the + unfinalized MoE with the finalize in the all-reduce from 14 rows) or dense MLP -> MoE boundary (all-reduce + + shared/routed add + residual + the next layer's RMSNorm, emitting the next QKV's MXFP8 activation). A layer whose + output Eagle3 captures runs the generic layer, as does the first layer when the fused embedding does not take the + step. A wider step (or one carrying embeddings) runs every layer in its generic form, with this target's attention + (the decode selection and sparse decode where the producer wrote the caches). + """ + + def __init__(self, core: MiniMaxM3Tp4ModelingV2) -> None: + self.core = core + self._moe_push_descs: Dict[tuple, Any] = {} + # Set by a fused feed-forward boundary that wrote the layer's Eagle3 capture (see _layer's ``capture``). + self._capture_written = False + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it; read once here rather than per forward. + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + States the two context counts instead of reading them back: this target runs only when ``num_contexts`` is + zero, so reading them would ask the metadata to confirm what the routing already decided. + + Graph classes: counts, the generation span, the identity-selection flag and the worst-case score extent are + host values fixed per capture (prepare() stages them before the forward); the block table, lengths, cache + slots, identity table and valid-block counts are the metadata's persistent buffers, held by reference and + refreshed in place every step; ``msa_idx_k_cache`` / ``msa_proxy_max_score_view`` are the metadata's getters + of the per-layer index-K cache and of the score scratch view. ``metadata`` is the prepared object itself: the + MSA backend's step (``m3_msa_attention``) takes it as its argument. + """ + return dict( + num_contexts=0, + num_ctx_tokens=0, + num_tokens=md.num_tokens, + num_seqs=md.num_seqs, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + msa_decode_span=md.msa_decode_span, + msa_block_table=md.msa_block_table, + msa_seq_lens_cuda=md.msa_seq_lens_cuda, + msa_short_ok=md._msa_short_ok, + msa_short_table=md.msa_short_table, + msa_proxy_max_score_view=md.msa_proxy_max_score_view, + msa_worst_case_max_k_tiles=md.msa_worst_case_max_k_tiles, + msa_n_valid_blocks=md.msa_n_valid_blocks, + max_seq_len=md.max_seq_len, + metadata=md, + ) + + def forward( + self, + attn_metadata, + input_ids, + position_ids, + inputs_embeds, + spec_metadata, + capture, + capture_target, + ): + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + # The decode kernels' rows (a per-capture constant): up to DECODE_STEP_MAX tokens of 1-D input ids. + kernels = ( + inputs_embeds is None and input_ids.ndim == 1 and is_decode_step(input_ids.numel()) + ) + if kernels: + num_tokens = int(input_ids.numel()) + plan = plan_for(num_tokens) + _record("decode", num_tokens, plan.name) + if plan.name == "wide": + taken("VR-BANDS wide decode step", num_tokens) + hidden, residual = self._embed(input_ids, num_tokens) + else: + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) CUDA graphs" + ) + plan = None + _record("decode", int(step["num_tokens"]), "generic") + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + is_capture = spec_metadata is not None and spec_metadata.is_layer_capture(i) + if ( + kernels + and residual is not None + and is_capture + and CAPTURE_LAYERS_FAST + and CAPTURE_FUSED_BOUNDARY + ): + # The capture layer on the fast path (its boundary pushes); the capture reads the MoE / dense + # boundary's reduced residual sum. With CAPTURE_NORM_IN_BOUNDARY the boundary all-reduce writes the + # capture norm itself. + capture_out = ( + capture_target(spec_metadata, i, num_tokens) + if capture_target is not None and CAPTURE_NORM_IN_BOUNDARY + else None + ) + self._capture_written = False + hidden, residual = self._layer( + i, plan, position_ids, hidden, residual, step, capture=capture_out + ) + taken("capture_layer_fast", num_tokens, f"layer {i}") + if capture_out is not None and self._capture_written: + taken("capture_norm_in_boundary", num_tokens, f"layer {i}") + capture(spec_metadata, i, residual, None, written=True) + else: + capture(spec_metadata, i, residual, None) + elif not kernels or residual is None or is_capture: + hidden, residual = self._generic_layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + else: + hidden, residual = self._layer(i, plan, position_ids, hidden, residual, step) + return hidden + + def _embed(self, input_ids: torch.Tensor, num_tokens: int): + """(hidden, residual) of the fused embedding + first RMSNorm; (embedding, None) when it does not run, so the + first layer runs its own input norm.""" + core = self.core + if not EMBED_FIRSTNORM.covers(num_tokens): + uncovered(EMBED_FIRSTNORM.path, f"more than {EMBED_FIRSTNORM_MAX_TOKENS} tokens") + return m3_embedding(input_ids, core.w["embed"]), None + owner = core.embed_owners.get((num_tokens, input_ids.dtype)) + if owner is None or not input_ids.is_contiguous(): + missed( + EMBED_FIRSTNORM.path, + num_tokens, + f"no embedding owner for {input_ids.dtype} ids (built for int32 / int64 after the load)" + if owner is None + else "the input ids are not contiguous", + ) + return m3_embedding(input_ids, core.w["embed"]), None + _record("m3_embedding_firstnorm", num_tokens, input_ids.dtype) + taken(EMBED_FIRSTNORM.path, num_tokens, str(input_ids.dtype)) + return m3_embedding_firstnorm( + input_ids, owner.table, owner.weight, owner.eps, owner.raw, owner.output, owner.use_pdl + ) + + def _bucket_flags(self, num_tokens: int): + """(bucket workspace, its 8-CTA call counters) of the boundary all-reduce for ``num_tokens``, or None.""" + workspace = m3_mc_allreduce.get_workspace(self.core.mapping, self.core.hidden) + workspace = None if workspace is None else m3_mc_allreduce._bucket(workspace, num_tokens) + flags = None if workspace is None else workspace.flags_for(8) + return None if flags is None else (workspace, flags) + + @staticmethod + def _no_bucket(path: str, num_tokens: int) -> None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, "the multicast all-reduce workspace") + else: + missed(path, num_tokens, "no multicast all-reduce bucket holds the step") + + def _attn_push_target(self, num_tokens: int): + """The o_proj's push tuple into the attention boundary's all-reduce, or None (after a miss). + + None comes only from the token count or from workspace state created collectively in the same eager call on + every rank, so all ranks agree. (A lone non-pushing rank would still be safe: its all-reduce pushes the same + bf16 rows into the same set A.)""" + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(ATTN_PUSH.path, num_tokens) + return None + workspace, flags = found + return ( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + AR_PUSH_MODE, + ) + + def _moe_push_targets(self, num_tokens: int): + """(engine push descriptor, shared down-projection push tuple) for the MoE boundary's exchange sets, or None + (after a miss). + + This decision must match across ranks: prepushed 3 reads set B of every rank, which a non-pushing rank never + fills. None comes only from the token count or from state created in the same eager call on every rank + (workspace, flags, the descriptor cache), and the engine it is armed for is agreed at build time + (``_prebuild_engines``), so it does.""" + hidden = self.core.hidden + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(MOE_PUSH.path, num_tokens) + return None + workspace, flags = found + key = (workspace.mc.data_ptr(), flags.data_ptr()) + desc = self._moe_push_descs.get(key) + if desc is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(MOE_PUSH.path, num_tokens, "the MoE engine's push descriptor") + return None + desc = self._moe_push_descs[key] = ar_push_desc( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + flags.device, + ) + set_b = workspace.mc.data_ptr() + 2 * 4 * workspace.max_tokens * workspace.world_size * ( + hidden // 2 + ) + return desc, ( + set_b, + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + 2, + ) + + def _boundary(self, use_mc: bool, x, residual, norm_w, x2=None, quant=False, capture=None): + """All-reduce + (second addend) + residual + RMSNorm: the multicast kernel, else the built-in all-reduce. + ``capture``: the Eagle3 capture the multicast kernel also writes (``_capture_written`` records it).""" + core = self.core + if use_mc: + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w, quant=quant, capture=capture) + if fused is not None: + if capture is not None: + self._capture_written = True + rows = x.shape[0] + taken( + MC_ALLREDUCE.path, + rows, + ("x + x2" if x2 is not None else "x") + + (", MXFP8 copy" if quant else "") + + (", cluster" if rows <= MC_AR_CLUSTER_MAX_ROWS else ""), + ) + return fused + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + missed( + MC_ALLREDUCE.path, + x.shape[0], + m3_mc_allreduce.decline_reason(x, residual, workspace, x2=x2), + ) + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _prepushed(self, i: int, x, residual, norm_w, quant, prepushed, x2=None, capture=None): + fused = _mc_allreduce_norm( + self.core, x, x2, residual, norm_w, quant=quant, prepushed=prepushed, capture=capture + ) + if fused is None: + raise RuntimeError(f"layer {i}: rows were pushed into an all-reduce that did not run") + if capture is not None: + self._capture_written = True + rows = x.shape[0] + taken( + MC_ALLREDUCE.path, + rows, + f"prepushed {prepushed}" + (", cluster" if rows <= MC_AR_CLUSTER_MAX_ROWS else ""), + ) + return fused + + def _layer(self, i: int, plan: DecodePlan, position_ids, hidden, residual, step, capture=None): + """``capture``: (out, fc_norm weight, eps) for the feed-forward boundary to write the layer's Eagle3 capture + norm (CAPTURE_NORM_IN_BOUNDARY, for a capture layer on this path); a fused boundary that writes it sets + ``_capture_written``.""" + core = self.core + num_tokens = hidden.shape[0] + # Padded rows: the boundary all-reduce polls and re-arms rows t < M = residual.shape[0], CUDA-graph padding + # rows included. Every producer pushes all rows of the same tensor: the o_proj / shared down GEMV t < m = + # x.shape[0] (its row count comes from the host shape, not a live count), the MoE engine every row of the + # num_tokens it was built for (= num_tokens here). The all-reduce checks x.sizes() == residual.sizes(), so + # no padded row is left with a stale flag. + # The GEMV dispatch (and with it every push into a boundary all-reduce) is a decode-graph one. + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + if not decode_graph: + uncovered( + "decode_graph", + "an eager decode step: the decode-graph GEMV dispatch and the boundary pushes are off", + ) + push = plan.attn_mc_ar and decode_graph and ATTN_PUSH.covers(num_tokens) + armed = self._attn_push_target(num_tokens) if push else None + hidden, attn_pushed = self._attention(i, position_ids, hidden, step, armed) + post_norm = core.w[f"l{i}_post_norm"] + if armed is not None and attn_pushed: + taken(ATTN_PUSH.path, num_tokens) + hidden, residual = self._prepushed(i, hidden, residual, post_norm, False, AR_PUSH_MODE) + else: + if armed is not None: + missed( + ATTN_PUSH.path, + num_tokens, + "the o_proj did not push its rows into the armed boundary all-reduce", + ) + hidden, residual = self._boundary(plan.attn_mc_ar, hidden, residual, post_norm) + + norm_w = _next_norm(core, i) + if i not in core.moe_layers: + # A dense layer: its MLP's down projection pushes into the post-MLP boundary's all-reduce, as the o_proj + # does into the attention one (the same slot; the gate_up projection never pushes). + # R6: the next layer's QKV takes its MXFP8 activation from this boundary, as after a MoE layer. + quant = R6_DENSE_PREQUANT and plan.qkv_prequant and i + 1 < core.num_layers + if plan.qkv_prequant and i + 1 < core.num_layers: + _log_once( + logger.info, + "r6_dense_prequant", + "on" if quant else "off", + "[M3 dispatch] r6_dense_prequant " + + ( + "taken: the dense-MLP boundary hands the next QKV its MXFP8 activation" + if quant + else "off (VR default; TRTLLM_M31_ENABLE_R6=1 turns it on), BF16 hand-off" + ) + + " (logged once)", + ) + dense_push = plan.attn_mc_ar and decode_graph and DENSE_PUSH.covers(num_tokens) + armed = self._attn_push_target(num_tokens) if dense_push else None + out, dense_pushed = self._swiglu_mlp(hidden, f"l{i}_mlp", True, armed, dense=True) + if armed is not None and dense_pushed: + taken(DENSE_PUSH.path, num_tokens) + return self._prepushed( + i, out, residual, norm_w, quant, AR_PUSH_MODE, capture=capture + ) + if armed is not None: + missed( + DENSE_PUSH.path, + num_tokens, + "the dense MLP's down projection did not push its rows into the armed boundary all-reduce", + ) + return self._boundary( + plan.attn_mc_ar, out, residual, norm_w, quant=quant, capture=capture + ) + + quant = plan.qkv_prequant and i + 1 < core.num_layers + path = moe_path(num_tokens) + moe_push = None + if path == MOE_ENGINE.path: + # The engine runs inside the MoE; looked up (or found missing) here first so a push is armed only for an + # engine that will take it. + engine = core.engine(i, num_tokens, shared=True) + if ( + engine is not None + and plan.moe_mc_ar + and decode_graph + and MOE_PUSH.covers(num_tokens) + ): + moe_push = self._moe_push_targets(num_tokens) + elif path == MOE16_ENGINE.path and decode_graph: + # Router, routed and shared experts in one kernel; its bf16 sum goes to the boundary all-reduce. + engine = core.moe16_engine(i, num_tokens) + if engine is not None: + _record("m3_moe16_engine", num_tokens) + out = m3_moe16_engine(hidden.contiguous(), engine) + taken(MOE16_ENGINE.path, num_tokens) + return self._boundary( + plan.moe_mc_ar, out, residual, norm_w, quant=quant, capture=capture + ) + if path in (MOE16_ENGINE.path, MC_FINALIZE.path) and plan.moe_mc_ar: + if self._bucket_flags(num_tokens) is None: + self._no_bucket(MC_FINALIZE.path, num_tokens) + else: + (shared, fc2, scales, perm), _, _, _ = self._moe( + i, hidden, return_parts=False, do_finalize=False + ) + _record("m3_mc_finalize_allreduce_norm", num_tokens, quant) + fused = m3_mc_allreduce.m3_mc_finalize_allreduce_norm( + fc2, + perm.view(num_tokens, -1).to(torch.int32), + scales.reshape(num_tokens, -1), + shared, + residual, + norm_w, + core.rms_eps, + m3_mc_allreduce.get_workspace(core.mapping, core.hidden), + use_pdl=get_env_enable_pdl(), + quant=quant, + ) + if fused is None: + raise RuntimeError( + "the MC finalize all-reduce rejected the unfinalized MoE outputs" + ) + taken(MC_FINALIZE.path, num_tokens) + return fused + shared, routed, engine_pushed, shared_pushed = self._moe( + i, hidden, return_parts=True, push=moe_push + ) + if moe_push is not None: + if routed is None: + # The engine folded the shared expert in and pushed routed + shared into set A. + if engine_pushed: + taken(MOE_PUSH.path, num_tokens, "engine, routed + shared") + return self._prepushed(i, shared, residual, norm_w, quant, 2, capture=capture) + else: + if engine_pushed and shared_pushed: + taken(MOE_PUSH.path, num_tokens, "routed, shared") + return self._prepushed( + i, shared, residual, norm_w, quant, 3, x2=routed, capture=capture + ) + if engine_pushed or shared_pushed: + raise RuntimeError("only one MoE part pushed into the boundary all-reduce") + missed( + MOE_PUSH.path, + num_tokens, + "the MoE did not push its output into the armed boundary all-reduce", + ) + return self._boundary( + plan.moe_mc_ar, shared, residual, norm_w, x2=routed, quant=quant, capture=capture + ) + + def _swiglu_mlp( + self, + h: torch.Tensor, + prefix: str, + low_m_gemm: bool, + push: Optional[tuple] = None, + dense: bool = False, + ): + """A gated MXFP8 MLP (the dense MLP with ``dense``, the shared expert): gate_up, SwiGLU-OAI, down, no + reduction. Returns (output, whether the down projection's GEMV pushed into ``push``).""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s", low_m_gemm) + if dense: + act = _dense_swiglu(core, gu) + else: + act = m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + return _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm, push) + + def _routed_chunks(self, i: int, x, router_logits): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: the MoE + scheduler's even split, even chunks on the MoE chunking stream.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + if num_chunks <= 1: + return _routed_experts(core, i, x, None, router_logits, True) + sizes = _split_chunk(rows, num_chunks) + xs, logits = x.split(sizes), router_logits.split(sizes) + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True)) + else: + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True)) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor, do_finalize: bool): + """Router + routed experts: the router logits of a generation step (``_decode_router_logits``) and the experts + quantizing their bf16 input.""" + core = self.core + router_logits = _decode_router_logits(core, i, h) + if not do_finalize: + return _routed_experts(core, i, h, None, router_logits, False) + return self._routed_chunks(i, h, router_logits) + + def _moe( + self, i: int, h: torch.Tensor, *, return_parts: bool, do_finalize: bool = True, push=None + ): + """The MoE block of layer ``i`` without its reduction. + + ``push`` = (the engine's ``arp`` descriptor, the shared down projection's ``ar`` tuple) the decode layer armed + for the MoE boundary's exchange sets, or None. Returns ``(out, routed, engine_pushed, shared_pushed)``: with + ``return_parts`` the shared and routed parts (routed None when the engine folded the shared expert in), else + their sum in ``out``; with ``do_finalize=False`` ``out`` is ``[shared, fc2 out, expert scales, permuted + index]``. + """ + core = self.core + engine_arp, shared_ar = push if push is not None else (None, None) + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + do_finalize + and h.dim() == 2 + and h.dtype == torch.bfloat16 + and 1 <= rows <= MOE_ENGINE_MAX_TOKENS + and rows >= MOE_FOLD_MIN_TOKENS + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + _record("m3_moe_engine", rows, engine_arp is not None) + folded = m3_moe_engine(h.contiguous(), engine, None, engine_arp) + taken(MOE_ENGINE.path, rows) + return folded, None, engine_arp is not None, False + routed_out, (shared_out, shared_pushed) = maybe_execute_in_parallel( + lambda: self._routed(i, h, do_finalize), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False, shared_ar), + core.moe_events[i][0], + core.moe_events[i][1], + # The warmup tuning pass profiles the shared expert's GEMMs: keep it on the main stream, between the same + # collectives on every rank (the capture pass and serving still overlap it on the aux stream). + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if not do_finalize: + return [shared_out, *routed_out], None, False, shared_pushed + if return_parts: + return shared_out, routed_out, False, shared_pushed + # In-place add into the shared output to avoid allocating a temporary. + return shared_out.add_(routed_out), None, False, shared_pushed + + def _generic_boundary(self, x, x2, residual, norm_w): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce.""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _generic_layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + else: + hidden, residual = _native_allreduce_norm( + core, hidden, residual, core.w[f"l{i}_post_norm"] + ) + if i in core.moe_layers: + out, routed, _, _ = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out, _ = self._swiglu_mlp(hidden, f"l{i}_mlp", True, dense=True) + routed = None + normed, updated = self._generic_boundary(out, routed, residual, _next_norm(core, i)) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + capture(spec_metadata, i, updated, None) + return normed, updated + + def _attention(self, i: int, position_ids, hidden, step, push: Optional[tuple] = None): + """Attention of layer ``i`` up to its o_proj (no reduction). ``push`` is the attention boundary's exchange the + o_proj may write into. Returns (o_proj output, pushed).""" + core = self.core + if i not in core.sparse_layers: + # Dense layers 0-2: QKV, the fused norm + RoPE (FP8 q/k/v: an FP8 cache, or the FP8 pool the dense layers + # sit in within a hybrid NVFP4 cache on a decode step), the attention, o_proj. A covered (query rows, + # requests) shape attends on the full-context PrimTS decode (``FULLCTX_DECODE``), the norm/RoPE kernel then + # storing the new K / V rows itself (``DENSE_KV_INSERT_FUSED``); every other step the MSA backend. + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + out_fp8 = core.main_kv_is_fp8 or core.main_kv_is_nvfp4 + plan = self._fullctx_plan(i, qkv.device, step) + kv_dest = None + if ( + plan is not None + and out_fp8 + and DENSE_KV_INSERT_FUSED + and DENSE_QKV_FP8_VIA_BF16 + and qkv.dtype == torch.bfloat16 + and core.head_dim == 128 + ): + kv_dest = ( + step["kv_cache_manager"].get_buffers(i, kv_layout="HND"), + step["msa_out_cache_loc"], + plan[1], + torch.empty( + (qkv.shape[0], core.heads, core.head_dim), + dtype=torch.float8_e4m3fn, + device=qkv.device, + ), + ) + fused = _qk_norm_rope(core, i, qkv, position_ids, out_fp8, kv_dest) + if kv_dest is not None: + taken(DENSE_KV_INSERT_PATH, kv_dest[2]) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + if kv_dest is not None: + q = kv_dest[3].view(q.shape[0], -1) + o = None + if plan is not None: + o = self._fullctx_decode(i, q, k, v, step, plan, kv_written=kv_dest is not None) + if o is None: + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + if not core.fused_projection: + return self._unfused_sparse_attention(i, position_ids, hidden, step, push) + packed, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + ) + if horizontal is None: + raise RuntimeError( + f"MiniMax-M3 sparse layer {i}: no horizontal producer takes this step's caches (an FP8 or NVFP4 HND " + "pool with an FP8 index-K cache)" + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + + def _fullctx_plan(self, i: int, device: torch.device, step): + """(the PrimTS split-KV decode plan, the step's query tokens) where dense layer ``i``'s decode attention runs on + kernels/m3_fullctx_decode: a covered (query rows per request, requests) shape over the layer's FP8 pool. None + otherwise (the MSA backend runs).""" + path = FULLCTX_DECODE.path + if is_torch_compiling(): + uncovered(path, "not a pure generation step on the MSA backend") + return None + span = step["msa_decode_span"] + if span is None or span.row_first != 0: + uncovered(path, "no decode span starting at row 0") + return None + nreq = int(step["num_seqs"]) + qlen = int(span.query_len) + if not FULLCTX_DECODE.covers((qlen, nreq)): + uncovered(path, f"{qlen} query rows x {nreq} requests is not a covered shape") + return None + if int(step["max_seq_len"]) > FULLCTX_DECODE_MAX_KV_LEN: + uncovered(path, "the cache length bound is beyond the plans' K/V capacity") + return None + num_tokens = nreq * qlen + pool, _ = step["kv_cache_manager"].get_kv_subpage_pool(i, "HND") + if pool.dtype != torch.float8_e4m3fn: + missed(path, num_tokens, "the dense layer's KV pool is not FP8") + return None + own_reducer = FULLCTX_DECODE_OWN_REDUCER + if own_reducer and not _fullctx_decode_kernel.has_own_reducer(): + missed(path, num_tokens, "the BF16 split-KV merge op is not built") + return None + # Compiled before warmup (``prebuild_kernels``); the forward never compiles one. + plan = _fullctx_decode_kernel.plan_for(device, qlen, nreq, own_reducer) + if plan is None: + missed(path, num_tokens, "the PrimTS decode plan is not prebuilt") + return None + return plan, num_tokens + + def _fullctx_decode( + self, i: int, q, k, v, step, plan_tokens, *, kv_written: bool + ) -> torch.Tensor: + """Dense layer ``i``'s decode attention on the PrimTS plan of ``_fullctx_plan``: the step's new K / V go into + the paged cache as the MSA backend writes them (``write_msa_phase_kv``; already there when ``kv_written``), + then every query row attends its request's full causal context. The merge also emits the o_proj's MXFP8 + input (``FULLCTX_DECODE_MXFP8``) when the plan supports it.""" + core = self.core + backend = core.attn[i] + plan, num_tokens = plan_tokens + nreq = int(step["num_seqs"]) + if not kv_written: + write_msa_phase_kv( + backend, + k[:num_tokens], + v[:num_tokens], + step["metadata"], + AttentionInputType.mixed, + token_offset=0, + ) + pool, subpages = step["kv_cache_manager"].get_kv_subpage_pool(i, "HND") + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + k_pages, v_pages = _fullctx_decode_kernel.kv_page_views(pool, subpages) + mxfp8 = None + if ( + FULLCTX_DECODE_MXFP8 + and output.shape[0] == num_tokens + and _fullctx_decode_kernel.emits_mxfp8() + and MERGE_MXFP8_MIN_TOKENS <= num_tokens <= MERGE_MXFP8.MAX_ROWS + ): + mxfp8 = MERGE_MXFP8.buffers(output.shape[1], output.device) + _record("m3_fullctx_decode", num_tokens, kv_written, mxfp8 is not None) + emitted = plan.run( + fp8_e4m3_saturate(q[:num_tokens].reshape(num_tokens, core.heads, core.head_dim)), + k_pages, + v_pages, + step["msa_block_table"][:nreq], + step["msa_seq_lens_cuda"][:nreq], + core.sm_scale(i), + output[:num_tokens], + mxfp8=mxfp8, + ) + if emitted: + m3_prequant.SLOT.publish(output, mxfp8[0][:num_tokens], mxfp8[1]) + taken(FULLCTX_DECODE_MXFP8_PATH, num_tokens) + taken(FULLCTX_DECODE.path, num_tokens) + return output + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step, push): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output. ``push`` is the attention + boundary exchange the o_proj GEMV writes its rows into, as in the five-way path.""" + core = self.core + if _producer_covered(core): + # Not a miss: the producer packs the five-way projection's output, which this configuration lacks. + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): a sparse + layer whose caches the producer wrote runs this target's selection and sparse decode when the generation span + starts at row 0; everything else the MSA backend (``m3_msa_attention``: the KV write unless a producer wrote the + caches, the sparse layers' top-k block selection, the FMHA).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + live = int(step["num_tokens"]) + q_live, out_live = q[:live], output[:live] + k_live = k[:live] if k is not None else None + v_live = v[:live] if v is not None else None + idx_q_live = idx_q[:live] if idx_q is not None else None + span = step["msa_decode_span"] + sparse = i in core.sparse_layers + if sparse and k_live is None and span is not None and span.row_first == 0: + self._decode_sparse_core(i, q_live, idx_q_live, step, out_live) + return output + nvfp4 = core.main_kv_is_nvfp4 + _record("m3_msa_attention", sparse, k_live is not None) + m3_msa_attention( + core.attn[i], + q_live, + k_live, + v_live, + idx_q_live, + step["metadata"], + out_live, + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + return output + + def _decode_sparse_core(self, i: int, q, idx_q, step, output) -> None: + """Top-16 block selection and the block-sparse decode of one pure generation step of a sparse layer whose caches + the producer already wrote. + + The selection is the identity when the metadata staged it (every request within 16 blocks). Otherwise the CuTe + DSL scorer writes the per-index-head block maxima, reduced to the KV head, and the long-row selector takes the + top 16 (the built-in selector beyond its shapes). + """ + core = self.core + backend = core.attn[i] + config = backend.m3_config + num_tokens = int(q.shape[0]) + rows = int(step["num_seqs"]) + qlen = int(step["msa_decode_span"].query_len) + block_table = step["msa_block_table"][:rows] + seq_lens = step["msa_seq_lens_cuda"][:rows] + num_kv_heads = int(backend.indexer.config.num_kv_heads) + identity = step["msa_short_ok"] and step["msa_short_table"] is not None + if identity: + topk = step["msa_short_table"][: num_kv_heads * num_tokens * MSA_REQUIRED_TOPK].view( + num_kv_heads, num_tokens, MSA_REQUIRED_TOPK + ) + else: + idx_q_view = idx_q.view(num_tokens, config.num_index_heads, config.sparse_index_dim) + idx_k_cache = step["msa_idx_k_cache"](i) + max_score = step["msa_proxy_max_score_view"]( + config.num_index_heads, step["msa_worst_case_max_k_tiles"], num_tokens + ) + if INDEX_SCORE_T2SC_PATH.covers(rows): + # T2SC: this target's copy of the TP2 / TP8 scorer at the request count's split and prestage. + split_k, early = index_score_t2sc_schedule(rows) + key = ( + idx_q_view.dtype, + int(config.num_index_heads), + int(qlen), + int(idx_q_view.shape[-1]), + int(split_k), + bool(early), + ) + if key not in _index_score_t2sc_kernel.IndexScoreRunner.kernel_cache: + missed( + INDEX_SCORE_T2SC_PATH.path, + num_tokens, + f"the T2SC index scorer {key} was not compiled before warmup", + ) + _record("m3_index_score_t2sc", key) + m3_index_score_t2sc( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens, + max_score.transpose(1, 2), + qlen, + split_k, + early, + ) + taken(INDEX_SCORE_T2SC_PATH.path, rows) + else: + early = rows <= INDEX_SCORE_EARLY_MAX_REQUESTS + key = ( + idx_q_view.dtype, + int(config.num_index_heads), + int(qlen), + int(idx_q_view.shape[-1]), + int(INDEX_SCORE_SPLIT_K), + bool(early), + ) + if key not in _index_score_kernel.IndexScoreRunner.kernel_cache: + # Outside require mode the op below compiles it (the only scorer this step has). + missed( + INDEX_SCORE.path, + num_tokens, + f"the index scorer {key} was not compiled before warmup", + ) + _record("m3_index_score", key) + m3_index_score( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens, + max_score.transpose(1, 2), + qlen, + INDEX_SCORE_SPLIT_K, + early, + ) + taken(INDEX_SCORE.path, rows) + n_valid_blocks = step["msa_n_valid_blocks"][:num_tokens] + group = config.num_index_heads // config.num_kv_heads + max_score_kv = ( + max_score.view( + config.num_kv_heads, group, max_score.shape[1], max_score.shape[2] + ).amax(dim=1) + if group > 1 + else max_score + ) + early_trigger = num_tokens <= SELECT_EARLY_MAX_ROWS + trigger_phase = ( + SELECT_LONG_TRIGGER_PHASE if num_tokens <= SELECT_LONG_TRIGGER_MAX_ROWS else 0 + ) + dsl = False + if LONG_SELECT_DSL.keys and 128 < max_score_kv.shape[1]: + # Compiled before warmup (prebuild_kernels); a missing kernel falls back to the 1024-thread one. + key = _select_long_dsl_kernel.kernel_key( + get_env_enable_pdl(), early_trigger, trigger_phase + ) + dsl = key in _select_long_dsl_kernel.kernel_cache + if not dsl: + missed( + LONG_SELECT_DSL.path, + num_tokens, + f"the 384-thread select {key} was not compiled before warmup", + ) + table = long_select_blocks( + max_score_kv, + n_valid_blocks=n_valid_blocks, + init_blocks=config.init_blocks, + local_blocks=config.local_blocks, + use_pdl=get_env_enable_pdl(), + early_trigger=early_trigger, + trigger_phase=trigger_phase, + fast_tail=SELECT_LONG_SL3, + dsl=dsl, + ) + if table is not None: + taken((LONG_SELECT_DSL if dsl else LONG_SELECT).path, num_tokens) + else: + uncovered( + LONG_SELECT.path, + f"{max_score_kv.shape[1]} KV blocks, outside {LONG_SELECT_MIN_BLOCKS}..{LONG_SELECT_MAX_BLOCKS}: " + "the built-in selector", + ) + table = m3_select_blocks( + max_score_kv, + MSA_REQUIRED_TOPK, + n_valid_blocks, + config.init_blocks, + config.local_blocks, + True, + ) + topk = table.permute(1, 0, 2) + manager = step["kv_cache_manager"] + k_paged, v_paged = msa_paged_kv(manager, i) + nvfp4 = {} + if manager.is_nvfp4_layer(i): + k_scale, v_scale = _aligned_nvfp4_dequant_scales(backend, core.kv_scales[i]) + scales = manager.get_block_scale_buffers(i, "HND") + nvfp4 = dict( + k_block_scale=scales[:, 0], + v_block_scale=scales[:, 1], + k_global_scale=k_scale, + v_global_scale=v_scale, + ) + _record("m3_sparse_decode", num_tokens, qlen, identity) + # The long-context decode (if it runs) prefetches this layer's o_proj weights into L2 and the blocks the next + # sparse layer selected at the previous step; any other route leaves the slots unused, and they are cleared + # after the call either way. + if SPARSE_DECODE_LC_MERGE_ONCE and SPARSE_DECODE_LC_PREFETCH_OPROJ: + m3_attn_lc.PREFETCH.tensors = (core.w[f"l{i}_o_w"], core.w[f"l{i}_o_s"]) + if SPARSE_DECODE_LC_MERGE_ONCE and i in core.n1_ids and num_tokens <= N1_MAX_ROWS: + m3_attn_lc.NEXT_KV.tensors = self._next_kv_prefetch(i, manager) + try: + m3_sparse_decode( + q.view(num_tokens, core.heads, core.head_dim), + k_paged, + v_paged, + topk, + block_table, + seq_lens, + sm_scale=core.sm_scale(i), + output=output.view(num_tokens, core.heads, core.head_dim), + decode_query_len=qlen, + identity_blocks=identity, + **nvfp4, + ) + finally: + m3_attn_lc.PREFETCH.tensors = None + m3_attn_lc.NEXT_KV.tensors = None + + def _next_kv_prefetch(self, i: int, manager): + """(layer ``i``'s selected-block id buffer, the next sparse layer's, that layer's K / V pools and block scales) + for the long-context decode's next-layer prefetch; the last sparse layer's next is the first one (next step). + None when the next layer's cache is not NVFP4.""" + ids = self.core.n1_ids + order = sorted(ids) + nxt = next((j for j in order if j > i), order[0]) + if not manager.is_nvfp4_layer(nxt): + return None + k_next, v_next = msa_paged_kv(manager, nxt) + scales = manager.get_block_scale_buffers(nxt, "HND") + return (ids[i], ids[nxt], k_next, v_next, scales[:, 0], scales[:, 1]) + + +# -------------------------------------------------------------------------------------------------------------------- +# Speculative decoding: one-model Eagle3 at checkpoint precision +# -------------------------------------------------------------------------------------------------------------------- + + +class M3SpecStepFold: + """``SpecStepFold`` of this target. + + Between two decode graphs of a one-model Eagle3 step, the spec sampler stores the step's outputs into its per-slot + store and the engine gathers them back into its flat input buffers for the next step. This object does both in one + launch each (``m3_spec_tail.slot_scatter`` / ``step_gather``): integer copies only, the same values the engine's and + the sampler's own copies write. + + With ``in_forward`` (``STEP_SPEC_FOLD_IN_FORWARD``) the forward of a decode-only step also does both at its tail + (``store_in_forward``, one ``m3_fold_tail`` launch inside the step graph): it stores the step's outputs for the rows + ``begin_step`` armed and stages the next step's inputs for those rows in that order. The arming is a device-side row + count and slot list staged before the replay, so a captured graph whose step was not armed writes nothing. The + sampler then skips its store for exactly the armed rows (``sampler_rows_stored``), and the next step skips its + gathers when its rows are those rows (``begin_step``). + + One instance per target model; the target passes it to ``RuntimeHooks.spec_step_fold``. + """ + + def __init__( + self, in_forward: bool = False, store_copy_wait: bool = False, cpp_tail: bool = False + ) -> None: + self.store: Optional[object] = None + self.step_inputs: Optional[ + tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor] + ] = None + self.in_forward = in_forward + # The forward's tail as this target's C++ op instead of the Triton kernel (m3_fold_tail.fold_tail(cpp=)). + self.cpp_tail = in_forward and cpp_tail + # Device arming read by the forward's tail: ctrl[0] = armed rows (0 = write nothing), slots[:rows]. + self._ctrl: Optional[torch.Tensor] = None + self._slots: Optional[torch.Tensor] = None + # The rows (sequence slots) the next forward stores and stages for, as begin_step armed them. + self._armed: Optional[tuple[int, ...]] = None + # The rows the previous forward staged the next step's inputs for (confirmed by sampler_rows_stored). + self._staged_for: Optional[tuple[int, ...]] = None + # Output buffers the tail ran on: the captured graphs' static outputs (by address) and the last eager step's. + self._graph_outputs: set[int] = set() + self._eager_outputs: Optional[torch.Tensor] = None + self._logged = False + # store_copy_wait: the sampler records this (external) event on its D2H side stream after copying the store + # to the host, and the forward's tail waits on it before it rewrites the store; the sampler can then copy + # on the side stream on the steps this forward stored the rows (see SpecSamplerBase._copy_outputs_to_host). + self._store_copy_wait = in_forward and store_copy_wait + self.store_copies_done: Optional[torch.cuda.Event] = None + + def bind_engine_buffers( + self, + input_ids: torch.Tensor, + draft_tokens: torch.Tensor, + position_id_offsets: torch.Tensor, + kv_lens_offsets: torch.Tensor, + max_batch_size: int, + ) -> None: + self.step_inputs = (input_ids, draft_tokens, position_id_offsets, kv_lens_offsets) + if self._store_copy_wait: + # Recorded once now so a captured wait has a completed record to wait on before the first copy. + self.store_copies_done = torch.cuda.Event(external=True) + self.store_copies_done.record() + logger.info( + "[M3 host] spec store host copies on the D2H side stream, awaited by the forward's tail" + ) + if self.cpp_tail: + logger.info( + "[M3 host] spec fold tail: one C++ launch (no Triton) at the forward's tail" + ) + if self.in_forward: + self._ctrl = torch.zeros(1, dtype=torch.int32, device=input_ids.device) + self._slots = torch.zeros( + max(max_batch_size, 1), dtype=torch.long, device=input_ids.device + ) + + def bind_sampler_store(self, store: object) -> None: + self.store = store + + def begin_step(self, seq_slots: Optional[list[int]], num_rows: int) -> bool: + if not self.in_forward or self._ctrl is None: + return False + rows = tuple(seq_slots) if seq_slots is not None else None + staged = rows is not None and rows == self._staged_for + self._staged_for = None + armed = rows if rows is not None and 0 < len(rows) <= self._slots.numel() else None + # Read only by this step's forward, so both uploads may wait for the step-input pack's flush. + if armed is not None: + slots = torch.tensor(armed, dtype=torch.long, pin_memory=prefer_pinned()) + copy_h2d_if_changed(self._slots[: len(armed)], slots, pack=True) + ctrl = torch.tensor( + [len(armed) if armed is not None else 0], dtype=torch.int32, pin_memory=prefer_pinned() + ) + copy_h2d_if_changed(self._ctrl, ctrl, pack=True) + self._armed = armed + return staged + + # The launches the store wait may precede. store_copies_done is an external event: under graph capture its wait + # is a node of its own, so the kernel launched right after it loses its programmatic (PDL) edge and with it the + # overlap of its pre-wait part. Only a launch that waits for its predecessor's completion anyway may follow it: + # the fold tail (a plain launch, no griddepcontrol). + STORE_WAIT_BEFORE = frozenset({"fold_tail"}) + + def _wait_store_copies(self, before: str) -> None: + """Order the store's rewrite after the sampler's host copies of it; ``before`` names the launch that follows. + Under graph capture a launch outside STORE_WAIT_BEFORE fails the capture.""" + if self.store_copies_done is None: + return + if before not in self.STORE_WAIT_BEFORE and torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + f"M3SpecStepFold: the store wait would precede {before!r}, a launch outside " + f"{sorted(self.STORE_WAIT_BEFORE)}; an external-event wait node drops the next kernel's PDL edge" + ) + torch.cuda.current_stream().wait_event(self.store_copies_done) + + def store_in_forward(self, outputs: dict) -> None: + """At the forward's tail: store the armed rows' outputs and stage the next step's inputs (one launch).""" + if self._ctrl is None or self.store is None or self.step_inputs is None: + return + store = self.store + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets = self.step_inputs + new_tokens = outputs.get("new_tokens") + next_new_tokens = outputs.get("next_new_tokens") + new_tokens_lens = outputs.get("new_tokens_lens") + next_draft_tokens = outputs.get("next_draft_tokens") + if any( + t is None or not isinstance(t, torch.Tensor) + for t in (new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens) + ): + return + rows = new_tokens.shape[0] + # The previous step's host copies of the store run on the sampler's side stream. + self._wait_store_copies(before="fold_tail") + if not m3_fold_tail.fold_tail( + self._ctrl, + self._slots, + min(rows, self._slots.numel()), + new_tokens, + store.new_tokens.squeeze(-1).T, + next_new_tokens, + store.next_new_tokens.squeeze(-1).T, + next_draft_tokens, + store.next_draft_tokens, + new_tokens_lens, + store.new_tokens_lens, + input_ids, + draft_tokens, + position_id_offsets, + kv_lens_offsets, + next_new_tokens.shape[1], + cpp=self.cpp_tail, + ): + return + if torch.cuda.is_current_stream_capturing(): + self._graph_outputs.add(new_tokens.data_ptr()) + else: + self._eager_outputs = new_tokens + + def gather( + self, + new_tokens: torch.Tensor, + next_draft_tokens: torch.Tensor, + new_tokens_lens: torch.Tensor, + previous_slots: torch.Tensor, + previous_pos_indices: torch.Tensor, + previous_batch_len: int, + tokens_per_gen_step: int, + draft_token_buffer_width: int, + num_tokens: int, + num_draft_tokens: int, + first_previous_row: int, + ) -> bool: + if self.step_inputs is None: + return False + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets = self.step_inputs + return m3_spec_tail.step_gather( + new_tokens, + next_draft_tokens, + new_tokens_lens, + previous_slots, + previous_pos_indices, + previous_batch_len, + tokens_per_gen_step, + draft_token_buffer_width, + input_ids, + num_tokens, + draft_tokens, + num_draft_tokens, + position_id_offsets, + kv_lens_offsets, + first_previous_row, + ) + + def sampler_rows_stored(self, outputs: dict, num_skip: int, seq_slots: list[int]) -> bool: + armed, self._armed = self._armed, None + new_tokens = outputs.get("new_tokens") + ran = isinstance(new_tokens, torch.Tensor) and ( + new_tokens is self._eager_outputs or new_tokens.data_ptr() in self._graph_outputs + ) + self._eager_outputs = None + stored = ( + ran + and armed is not None + and num_skip == 0 + and tuple(seq_slots) == armed + and new_tokens.shape[0] == len(armed) + ) + self._staged_for = armed if stored else None + if stored and not self._logged: + self._logged = True + logger.info("[M3 host] spec step stored and next step staged in the forward tail") + return stored + + def scatter( + self, + seq_slots: torch.Tensor, + new_tokens: torch.Tensor, + next_new_tokens: torch.Tensor, + new_tokens_lens: torch.Tensor, + next_draft_tokens: torch.Tensor, + ) -> bool: + store = self.store + if store is None: + return False + return m3_spec_tail.slot_scatter( + seq_slots, + new_tokens, + store.new_tokens.squeeze(-1).T, + next_new_tokens, + store.next_new_tokens.squeeze(-1).T, + new_tokens_lens, + store.new_tokens_lens, + next_draft_tokens, + store.next_draft_tokens, + ) + + +class M3DSparkStepFold(M3SpecStepFold): + """The step fold under the standalone DSpark worker (DSFOLD): the engine's previous-batch gathers in one launch + (``step_gather``). The sampler's slot stores stay on its in-tree path, so ``scatter`` declines every step. + ``taken`` / ``declined`` count the steps the gather ran / fell back to the engine's own copies.""" + + def __init__(self) -> None: + super().__init__() + self.taken = 0 + self.declined = 0 + + def gather(self, *args, **kwargs) -> bool: + if not super().gather(*args, **kwargs): + self.declined += 1 + uncovered( + "DSFOLD", + "step_gather declined the inputs (dtype / layout); the engine's copies run", + ) + return False + if self.taken == 0: + logger.info( + "[M3 dispatch] DSFOLD DSpark step-input gather: engaged (one step_gather launch)" + ) + self.taken += 1 + return True + + def scatter( + self, seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens + ) -> bool: + return False + + +class _DraftForward: + """``draft_model.model`` for the in-tree worker: the drafter layer's forward, called as + ``model(input_ids=, position_ids=, hidden_states=, attn_metadata=, spec_metadata=, all_rank_num_tokens=)`` and + returning (hidden, hidden to carry to the next pass). ``d2t`` None: the draft vocabulary is the target's.""" + + d2t = None + + def __init__(self, draft: "M3Eagle3Draft") -> None: + self._draft = draft + + def __call__(self, **kwargs): + return self._draft.forward_layer(**kwargs) + + +class M3Eagle3Draft(nn.Module): + """The one-layer BF16 Eagle3 drafter (``speculative_model``): its weights (``weights.DRAFT_WEIGHTS``), its + vocab-parallel BF16 head and logits processor, the drafter layer's TRTLLM attention backend (layer index = the + target's layer count, FP8 KV) and its TP all-reduce. + + What the worker reads: ``model`` (the layer forward), ``apply_eagle3_fc`` (the per-layer fc_norms + fc over the + captured target hidden states), ``lm_head`` and ``logits_processor``; the fused draft loop reads the layer's parts + (``draft_input_weights``, ``attention``, ``mlp``, ``ar_norm``). The drafter embeds with the target's table. + """ + + def __init__( + self, + core: MiniMaxM3Tp4ModelingV2, + draft_config: ModelConfig, + start_layer_idx: int, + capture_layers: int, + ) -> None: + super().__init__() + cfg = draft_config.pretrained_config + _ensure_draft_vocab_size(cfg) + eagle = getattr(cfg, "eagle_config", None) or {} + mapping = draft_config.mapping + self.draft_config = draft_config + # The engine's draft weight mapper binds to these two. + self.model_config = draft_config + self.config = cfg + self.capture_layers = int(capture_layers) + self.layer_idx = int(start_layer_idx) + self.dtype = cfg.torch_dtype + self.hidden = int(cfg.hidden_size) + self.hidden_size_in = int(getattr(cfg, "target_hidden_size", cfg.hidden_size)) + self.rms_eps = float(cfg.rms_norm_eps) + self.return_hidden_post_norm = bool( + eagle.get("return_hidden_post_norm", False) or getattr(cfg, "norm_output", False) + ) + if ( + int(cfg.num_hidden_layers) != 1 + or eagle.get("eh_proj_before_attn", False) + or eagle.get("norm_before_fc", False) + or not getattr(cfg, "fc_norm", False) + or getattr(cfg, "kv_lora_rank", None) + or self.capture_layers <= 1 + or self.hidden_size_in != self.hidden + or cfg.draft_vocab_size != cfg.vocab_size + or getattr(cfg, "attention_bias", False) + or getattr(cfg, "mlp_bias", False) + or getattr(cfg, "bias", False) + or getattr(cfg, "tie_word_embeddings", False) + ): + raise ValueError( + "this MiniMax-M3 target drafts with a one-layer Llama-style Eagle3 drafter with per-layer " + "fc_norms over " + "several captured layers, the target's hidden size and vocabulary and no biases" + ) + tp = mapping.tp_size + self.heads = cfg.num_attention_heads // tp + self.kv_heads = (cfg.num_key_value_heads + tp - 1) // tp + self.head_dim = int( + getattr(cfg, "head_dim", None) or cfg.hidden_size // cfg.num_attention_heads + ) + self.q_size = self.heads * self.head_dim + self.kv_size = self.kv_heads * self.head_dim + self.w = _weights.DRAFT_WEIGHTS.declare(_weights.DRAFT_WEIGHTS.dims(self)) + self.lm_head = LMHead( + cfg.draft_vocab_size, + cfg.hidden_size, + dtype=cfg.torch_dtype, + mapping=mapping, + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=draft_config.lm_head_gather_output, + reduce_output=False, + use_custom_cublas_mm=False, + quant_config=None, + ) + self.logits_processor = LogitsProcessor() + self.allreduce = AllReduce( + mapping=mapping, strategy=draft_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + quant_config = draft_config.get_quant_config() + self.attn = create_attention( + draft_config.attn_backend, + self.layer_idx, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=PositionalEmbeddingParams( + type=PositionEmbeddingType.rope_gpt_neox, rope=RopeParams.from_config(cfg) + ), + quant_config=quant_config, + skip_create_weights_in_init=draft_config.skip_create_weights_in_init, + q_scaling=1.0, + attention_chunk_size=None, + sparse_params=None, + ) + if not self.attn.support_fused_rope() or not self.attn.support_fused_qkv(): + raise ValueError( + f"the drafter's attention backend {type(self.attn).__name__} must fuse RoPE and QKV" + ) + self.attn.update_quant_config(quant_config) + # The drafter embeds with the target's table, read off the core on every call (the engine materializes the + # core's weights after construction). + self._core = [core] + self.model = _DraftForward(self) + # The projections (weight keys) on the fixed-tactic low-M kernels (``DRAFT_LOWM``), set by + # ``prebuild_linears``. + self._lowm_keys: frozenset = frozenset() + + def prebuild_linears(self) -> int: + """Put the drafter's BF16 projections ``DRAFT_LOWM`` covers on the fixed-tactic low-M kernels and compile + every covered call. Once the weights are real, before the executor's warmup, on every rank. Returns the + number of low-M calls compiled.""" + keys = ("fc", "qkv", "o", "gu", "dn") + built = _draft_lowm_kernel.prebuild([self.w[k] for k in keys], DRAFT_LOWM.keys) + self._lowm_keys = frozenset(k for k in keys if _draft_lowm_kernel.installed(self.w[k])) + return built + + def _linear(self, x: torch.Tensor, key: str) -> torch.Tensor: + """``x @ w[key]^T``: the low-M kernel at the rows ``DRAFT_LOWM`` covers (``prebuild_linears``), else the plain + GEMM.""" + w = self.w[key] + if key in self._lowm_keys: + out = _draft_lowm_kernel.draft_linear(w, x, None) + if out is not None: + return out + return m3_bf16_linear(x, w) + + @property + def embed_table(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def load_weights_from_target_model(self, target_model: nn.Module) -> None: + """Nothing to alias: the drafter reads the target's embedding table through the core and owns its head.""" + + # ------------------------------------------------------------------------------------------------- the layer + + def fc(self, hs: torch.Tensor) -> torch.Tensor: + return self._linear(hs, "fc") + + def apply_eagle3_fc(self, hidden_states: torch.Tensor) -> torch.Tensor: + """The captured target hidden states [T, layers x H] to the drafter's input width: each layer's fc_norm, + then the fc (the in-tree ``Eagle3ForCausalLM.apply_eagle3_fc``).""" + hidden_states = hidden_states.to(self.dtype) + if hidden_states.shape[-1] != self.hidden: + chunks = hidden_states.chunk(self.capture_layers, dim=-1) + hidden_states = torch.cat( + [ + flashinfer_rmsnorm(c, self.w[f"fc_norm{i}"], self.rms_eps) + for i, c in enumerate(chunks) + ], + dim=-1, + ) + hidden_states = self.fc(hidden_states) + return hidden_states + + def attention(self, x: torch.Tensor, md, reduce: bool) -> torch.Tensor: + """QKV, the backend's fused RoPE + KV append + causal attention, o_proj (all-reduced unless ``reduce`` is + False, where the caller fuses the reduction).""" + qkv = self._linear(x, "qkv") + _record("m3_draft_attention", int(md.num_tokens)) + o = m3_draft_attention(self.attn, qkv, md) + out = self._linear(o, "o") + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def mlp(self, h: torch.Tensor, reduce: bool, silu_cpp: bool = False) -> torch.Tensor: + """gate_up, the plain SwiGLU (with ``silu_cpp``, ``DRAFT_SILU_CPP``: this target's C++ op, bit for bit the + Triton one, in one launch), down.""" + gu = self._linear(h, "gu") + act = None + if silu_cpp and DRAFT_SILU_CPP: + act = m3_draft_silu(gu) + if act is None: + uncovered( + DRAFT_SILU_PATH, "the drafter SwiGLU input layout (the Triton op runs it)" + ) + else: + taken(DRAFT_SILU_PATH, int(gu.shape[0]), "drafter MLP SwiGLU, one C++ launch") + if act is None: + act = m3_silu_and_mul(gu, None, None, None) + out = self._linear(act, "dn") + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def ar_norm(self, x: torch.Tensor, residual: torch.Tensor, norm_key: str): + """One draft-layer boundary in the in-tree structure: the projection's plain all-reduce (BF16 out), then the + fused residual add + RMSNorm (FP32 add, norm of the FP32 sum). Returns (normed, residual sum).""" + return flashinfer_fused_add_rmsnorm( + m3_allreduce(x, self.allreduce, None), residual, self.w[norm_key], self.rms_eps + ) + + def forward_layer( + self, + input_ids: torch.Tensor, + position_ids: torch.Tensor, + hidden_states: torch.Tensor, + attn_metadata, + spec_metadata, + all_rank_num_tokens: Optional[List[int]] = None, + ): + """The drafter layer (the in-tree ``Eagle3DraftModel.forward`` for one layer): the embedded tokens and the + fc'd hidden states normed and concatenated, attention, the post-attention norm with the residual, the MLP and + the final norm. Returns (normed output, the hidden state the next pass carries).""" + previous = attn_metadata.all_rank_num_tokens + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = all_rank_num_tokens + try: + embeds = m3_embedding(input_ids, self.embed_table).to(self.dtype) + residual = hidden_states + h = flashinfer_rmsnorm(hidden_states, self.w["hidden_norm"], self.rms_eps) + e = flashinfer_rmsnorm(embeds, self.w["in_norm"], self.rms_eps) + h = torch.cat([e, h], dim=-1) + h = self.attention(h, attn_metadata, reduce=True) + h, residual = flashinfer_fused_add_rmsnorm( + h, residual, self.w["post_norm"], self.rms_eps + ) + h = self.mlp(h, reduce=True) + spec_metadata.maybe_capture_hidden_states(self.layer_idx, h, residual) + h, to_save = flashinfer_fused_add_rmsnorm( + h, residual, self.w["final_norm"], self.rms_eps + ) + return (h, h) if self.return_hidden_post_norm else (h, to_save) + finally: + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = previous + + +def _eagle3_capture(spec_metadata, layer_id: int, hidden_states, residual, capture_norm) -> None: + """The Eagle3 hidden-state capture of one target layer. + + Mirrors ``Eagle3OneModelSpecMetadata.maybe_capture_hidden_states``. With ``capture_norm`` = (the drafter's stacked + fc_norm weights [layers, H], eps), the capture writes rmsnorm(bf16(h + residual)) * fc_norm[i] in one kernel (the + in-tree capture's bf16 add, then the in-tree fc_norm arithmetic), which the fused first drafter inputs read as is; + otherwise it is the in-tree capture. + """ + if capture_norm is None: + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + return + for i, captured_layer_id in enumerate(spec_metadata.layers_to_capture): + if captured_layer_id == layer_id: + assert spec_metadata.hidden_states is not None + num_tokens = spec_metadata.num_capture_tokens + if num_tokens > spec_metadata.hidden_states.shape[0]: + raise RuntimeError( + "EAGLE3 hidden-state capture token count exceeds buffer capacity: " + f"num_tokens={num_tokens}, capacity={spec_metadata.hidden_states.shape[0]}" + ) + if num_tokens > hidden_states.shape[0]: + raise RuntimeError( + "EAGLE3 hidden-state capture token count exceeds available hidden states: " + f"num_tokens={num_tokens}, hidden_states={hidden_states.shape[0]}" + ) + hidden_size = spec_metadata.hidden_size + m3_spec_tail.capture_norm( + hidden_states[:num_tokens], + residual[:num_tokens] if residual is not None else None, + capture_norm[0][i], + capture_norm[1], + spec_metadata.hidden_states[:num_tokens, i * hidden_size : (i + 1) * hidden_size], + ) + break + + +def _sharded_head(lm_head: nn.Module) -> bool: + """A vocab-sharded TP head without padding (the fused kernels read the local shard).""" + return ( + getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and getattr(lm_head, "padding_size", 0) == 0 + and getattr(lm_head, "tp_size", 1) > 1 + ) + + +class M3Eagle3Worker(Eagle3OneModelWorker): + """``Eagle3OneModelWorker`` with the MiniMax-M3 fused verify and fused draft loop, at the in-tree numerics. + + * Verify: on decode-only batches whose verify rows the fused kernels cover (``tail_eligible``), the target's BF16 + lm_head computes this rank's vocabulary shard of the logits (the in-tree head's own local GEMM; at most + ``TARGET_HEAD_MAX_ROWS`` verify rows stream a lossless 12-bit coding of the same BF16 weight) and one kernel + replaces the logits all-gather, the argmax (or the temperature + top-p draw) and the strict acceptance. + * Draft: the drafter's first pass may take the per-layer fc_norms in one launch (or folded into the hidden-state + capture) and run its fc beside the target head GEMM (forked right after the last captured target layer at most + ``FC_EARLY_MAX_ROWS`` verify rows). On a decode-only step ``DRAFT_STEP`` covers, the draft loop is fused: each + pass's draft input in one launch, the drafter layer's attention and MLP with their all-reduces fused with the + residual add and the next RMSNorm, and one kernel for the BF16 head, the cross-rank argmax and the pass's + bookkeeping. + + Every other batch takes the in-tree paths. Methods whose docstring says "Mirrors" copy the named in-tree method + with the fused branches inserted. + """ + + def __init__(self, spec_config, mapping, use_separate_draft_kv_cache: bool = False) -> None: + super().__init__( + spec_config, mapping=mapping, use_separate_draft_kv_cache=use_separate_draft_kv_cache + ) + # Per-step state, reset at the top of every _forward_impl: the first draft pass's gather rows when this step + # takes the fused first drafter inputs, and the fc output a fork computed beside the target head. + self._m3_first_gather_ids: Optional[torch.Tensor] = None + self._m3_fc_hidden: Optional[torch.Tensor] = None + self._m3_target_lm_head: Optional[nn.Module] = None + # (fc_norm0 weight, stacked fc_norm weights [layers, H]), built eagerly on the first fused first pass. + self._m3_fc_norm_w: Optional[tuple] = None + # Armed per step by arm_capture_norm: (stacked fc_norm weights, eps) or None. + self._m3_capture_norm: Optional[tuple] = None + # Per step: the captured layers written with the capture norm (set by capture_hidden_states). + self._m3_capture_normed: set = set() + self._m3_aux_stream: Optional[torch.cuda.Stream] = None + # Armed per step by arm_capture_norm: the drafter whose first fc is forked at the last captured target layer, + # and the fork that did it, joined by _m3_target_accept. + self._m3_fc_early_drafter: Optional[M3Eagle3Draft] = None + self._m3_fc_early_stream: Optional[torch.cuda.Stream] = None + # The 12-bit coded copy of the target head shard the fused verify streams, built before the warmup. + self._m3_target_head = None + self._m3_force_vec_warm = False + # The streamed copy of the drafter's BF16 head shard the fused draft loop reads, built before the warmup. + self._m3_draft_head = None + # Per step: the next step's input tokens [nreq, D + 1], column 0 written by the fused verify (or the seed) and + # columns 1.. by the fused draft passes; set as _m3_next_new_tokens once the fused draft loop filled it. + self._m3_next_new: Optional[torch.Tensor] = None + self._m3_next_new_tokens: Optional[torch.Tensor] = None + # Per step: the first draft inputs are the fused ones (accepted tokens as they lie, fc output of every row). + self._m3_fused_first_inputs = False + + def capture_target( + self, spec_metadata, layer_id: int, num_tokens: int + ) -> Optional[Tuple[torch.Tensor, torch.Tensor, float]]: + """Where this step's capture norm writes layer ``layer_id``'s capture, for a boundary all-reduce that writes it + itself (``CAPTURE_NORM_IN_BOUNDARY``): (the capture buffer's [num_tokens, H] column block, the layer's + fc_norm weight, eps), or None (no capture norm armed, not a captured layer, or more rows than the buffer). + The boundary writes every row it reduces; rows past num_capture_tokens are unread.""" + capture_norm = self._m3_capture_norm + if capture_norm is None or spec_metadata.hidden_states is None: + return None + if num_tokens > spec_metadata.hidden_states.shape[0]: + return None + hidden_size = spec_metadata.hidden_size + for i, captured_layer_id in enumerate(spec_metadata.layers_to_capture): + if captured_layer_id == layer_id: + out = spec_metadata.hidden_states[ + :num_tokens, i * hidden_size : (i + 1) * hidden_size + ] + return out, capture_norm[0][i], capture_norm[1] + return None + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None, written: bool = False + ) -> None: + """The Eagle3 capture with this step's capture norm (``written``: the layer's boundary all-reduce already + wrote it, see ``capture_target``); on a step armed for the early fc fork (``arm_capture_norm``), the last + captured layer then forks the drafter's first fc onto the aux stream, where it overlaps the remaining target + layers and the head (joined by ``_m3_target_accept``).""" + if not written: + _eagle3_capture(spec_metadata, layer_id, hidden_states, residual, self._m3_capture_norm) + if self._m3_capture_norm is not None: + # Written with this step's capture norm: by the boundary all-reduce (written) or just above. + self._m3_capture_normed.add(layer_id) + drafter = self._m3_fc_early_drafter + layers = spec_metadata.layers_to_capture + if drafter is not None and layers and layer_id == layers[-1]: + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = self._m3_fc_fork( + spec_metadata, spec_metadata.num_capture_tokens, drafter + ) + + def prebuild(self, lm_head: nn.Module, draft: Optional[M3Eagle3Draft]) -> None: + """The fused verify's coded target head and the fused draft loop's head copy, from the loaded weights, + before the executor's warmup, on every rank.""" + start = time.perf_counter() + if self.prebuild_target_head(lm_head): + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] coded target head built before warmup: %.1f s" + % (time.perf_counter() - start) + ) + start = time.perf_counter() + if self.prebuild_draft_head(draft): + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] fused draft head built before warmup: %.1f s" + % (time.perf_counter() - start) + ) + + def _forward_impl( + self, + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + resource_manager=None, + ): + """Mirrors Eagle3OneModelWorker._forward_impl, with the fused verify branch.""" + runtime_draft_len = spec_metadata.runtime_draft_len + # skip the draft forward if the runtime draft length is 0 + if runtime_draft_len == 0: + return self.skip_drafting( + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + ) + + batch_size = attn_metadata.num_seqs + num_contexts = attn_metadata.num_contexts + num_gens = batch_size - num_contexts + + raw_logits = logits + + self._m3_first_gather_ids = None + self._m3_next_new = None + self._m3_next_new_tokens = None + self._m3_fused_first_inputs = False + if logits is None: + # tail_eligible() skipped the gathered target logits: verify from this rank's logits shard. + accepted_tokens, num_accepted_tokens, self._m3_first_gather_ids = ( + self._m3_target_accept( + hidden_states, + spec_metadata, + batch_size, + runtime_draft_len, + draft_model, + kv_lens=getattr(attn_metadata, "kv_lens_cuda", None), + ) + ) + else: + self._execute_guided_decoder_if_present(logits) + + # Sample and accept tokens. ``input_ids`` is required by the relaxed-acceptance path (scans for + # thinking-phase tokens); ignored otherwise. + accepted_tokens, num_accepted_tokens = self.sample_and_accept_draft_tokens( + input_ids, logits, attn_metadata, spec_metadata + ) + if self.draft_only_eligible(attn_metadata, spec_metadata, draft_model): + self._m3_seed_first_pass(num_accepted_tokens, spec_metadata, batch_size) + + # Mamba hybrid models need state updates after token acceptance (a no-op on this target's cache manager). + if self._is_mamba_hybrid_cache is None: + self._is_mamba_hybrid_cache = isinstance( + attn_metadata.kv_cache_manager, MambaHybridCacheManager + ) + if num_gens > 0 and self._is_mamba_hybrid_cache: + attn_metadata.kv_cache_manager.update_mamba_states( + attn_metadata=attn_metadata, + num_accepted_tokens=num_accepted_tokens, + state_indices=attn_metadata.mamba_metadata.state_indices, + ) + + sa_manager = getattr(spec_metadata.spec_resource_manager, "sa_manager", None) + if self.sa_enhancer is not None and sa_manager is not None: + self.sa_enhancer.extend_and_prepare( + sa_manager=sa_manager, + request_ids=spec_metadata.request_ids, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_gens=num_gens, + num_contexts=num_contexts, + max_draft_len=runtime_draft_len, + ) + + if self._uses_external_shared_target_kv: + next_draft_tokens = self._forward_external_shared_target_kv_draft_loop( + position_ids=position_ids, + hidden_states=hidden_states, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_contexts=num_contexts, + batch_size=batch_size, + ) + else: + # Save the old attn_metadata and spec_metadata + self._prepare_attn_metadata_for_spec_dec(attn_metadata) + + # Prepare inputs for the 1st draft model forward + position_ids = position_ids.squeeze(0) + inputs = self.prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + + # Predict draft tokens. ``original_all_rank_num_tokens`` is saved here so the post-loop restore (below) + # can put attn_metadata back into a state the target model expects. + original_all_rank_num_tokens = attn_metadata.all_rank_num_tokens + + # Get the draft KV cache manager if using separate layouts + draft_kv_cache_manager = self.get_draft_kv_cache_manager(resource_manager) + + next_draft_tokens = self._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + # restore attn_metadata to support cuda graph + self._restore_attn_metadata_from_spec_dec(attn_metadata) + # restore all_rank_num_tokens for attention DP + if original_all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = original_all_rank_num_tokens + + # prepare next new tokens to support overlap scheduler + if self._m3_next_new_tokens is not None: + # Filled in place by the fused verify or the seed (column 0) and the fused draft passes (columns 1..). + next_new_tokens = self._m3_next_new_tokens + else: + next_new_tokens = self._prepare_next_new_tokens( + accepted_tokens, + next_draft_tokens, + spec_metadata.batch_indices_cuda, + batch_size, + num_accepted_tokens, + ) + + attn_metadata.use_spec_decoding = True + + # Commit model-owned target state only after draft preparation has succeeded; SpecWorkerBase aborts the + # still-pending snapshots on any exception above this point. + if self._auxiliary_state_handlers: + self.commit_auxiliary_speculative_states( + num_accepted_tokens, + attn_metadata.mamba_metadata.state_indices[:batch_size], + num_contexts, + ) + + return { + "logits": raw_logits, + "new_tokens": accepted_tokens, + "new_tokens_lens": num_accepted_tokens, + "next_draft_tokens": next_draft_tokens, + "next_new_tokens": next_new_tokens, + } + + def prepare_1st_drafter_inputs( + self, + input_ids, + position_ids, + hidden_states, + accepted_tokens, + attn_metadata, + spec_metadata, + draft_model, + ): + """The fused first draft inputs on a step that verified with the fused kernels (or was seeded for them), + else the in-tree ones.""" + fused = self._m3_first_drafter_inputs( + position_ids, + accepted_tokens, + input_ids.shape[0], + attn_metadata.num_contexts, + spec_metadata, + draft_model, + ) + self._m3_fused_first_inputs = fused is not None + if fused is not None: + taken("m3_first_drafter_inputs", attn_metadata.num_seqs) + fused.update(attn_metadata=attn_metadata, spec_metadata=spec_metadata) + return fused + return super().prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + + def _m3_structural_ok(self, attn_metadata, spec_metadata) -> bool: + """The batch conditions shared by the fused verify and the fused first drafter inputs: a decode-only plain + Eagle3 step without attention DP, guided decoding, relaxed acceptance or a suffix-automaton enhancer, of at + most DECODE_MAX_TOKENS verify rows. Host state only, so the TP ranks agree.""" + if self.is_mtp_eagle or self.use_dynamic_tree: + return False + if self._uses_external_shared_target_kv or self.sa_enhancer is not None: + return False + if ( + self.guided_decoder is not None + or self.mapping is None + or self.mapping.enable_attention_dp + ): + return False + if getattr(self.spec_config, "use_relaxed_acceptance_for_thinking", False): + return False + if attn_metadata.num_contexts != 0 or attn_metadata.num_seqs == 0: + return False + draft_len = spec_metadata.runtime_draft_len + if draft_len == 0 or attn_metadata.num_seqs * (draft_len + 1) > DECODE_MAX_TOKENS: + return False + return not spec_metadata.wants_advanced_draft_sampling + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Whether this step verifies with the fused kernels instead of the gathered target logits. + + Decode-only Eagle3 batches (``_m3_structural_ok``) over an unpadded vocab-sharded bias-free target head, all + greedy at a request count ``SHARD_ACCEPT`` covers, or sampled under the temperature + top-p contract (see + _m3_sampled_verify_ok), with nothing else reading the target logits. Every input is host state shared by all + TP ranks, so the ranks agree. + """ + if not self._m3_structural_ok(attn_metadata, spec_metadata): + return False + if getattr(spec_metadata, "enable_penalty", False): + return False + if not _sharded_head(lm_head) or getattr(lm_head, "bias", None) is not None: + return False + draft_len = spec_metadata.runtime_draft_len + if spec_metadata.is_all_greedy_sample: + key = (draft_len, attn_metadata.num_seqs) + if not SHARD_ACCEPT.covers(key): + uncovered( + SHARD_ACCEPT.path, + f"draft length {draft_len} at {attn_metadata.num_seqs} requests", + ) + return False + elif attn_metadata.num_seqs > SAMPLED_VERIFY_MAX_REQUESTS: + uncovered("m3_sampled_accept", f"sampled verify at {attn_metadata.num_seqs} requests") + return False + elif not self._m3_sampled_verify_ok(spec_metadata): + return False + self._m3_target_lm_head = lm_head + return True + + def draft_only_eligible(self, attn_metadata, spec_metadata, draft_model) -> bool: + """Whether a step verified by the in-tree path can still take the fused first drafter inputs: the structural + conditions of tail_eligible, without the target-sampling ones, up to ``FIRST_DRAFTER_INPUTS_MAX_REQUESTS`` + requests.""" + if attn_metadata.num_seqs > FIRST_DRAFTER_INPUTS_MAX_REQUESTS: + uncovered( + "m3_first_drafter_inputs", f"in-tree verify at {attn_metadata.num_seqs} requests" + ) + return False + return self._m3_structural_ok(attn_metadata, spec_metadata) and draft_model is not None + + def _m3_seed_first_pass(self, num_accepted_tokens, spec_metadata, batch_size: int) -> None: + """The first draft pass's gather rows (the last accepted row of each request) from the in-tree acceptance, as + _m3_target_accept leaves them.""" + k1 = spec_metadata.runtime_draft_len + 1 + n = num_accepted_tokens[:batch_size].to(torch.int64) + rows = spec_metadata.batch_indices_cuda[:batch_size].to(torch.int64) + self._m3_first_gather_ids = rows * k1 + n - 1 + + def _m3_first_drafter_inputs( + self, + position_ids, + accepted_tokens, + num_tokens: int, + num_contexts: int, + spec_metadata, + draft_model: Optional[M3Eagle3Draft], + ): + """The in-tree first draft inputs of a decode-only step, with the per-layer fc_norm RMSNorms of + ``apply_eagle3_fc`` in one kernel (unless the capture already applied them) before the fc; the input ids are + the accepted tokens as they lie. None to use the in-tree path.""" + if ( + self._m3_first_gather_ids is None + or self.is_mtp_eagle + or num_contexts != 0 + or draft_model is None + ): + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + ): + return None + cache = self._m3_fc_norm_w + if cache is None or cache[0] is not draft_model.w["fc_norm0"]: + if torch.cuda.is_current_stream_capturing(): + capture_missed("spec_tail", None, "the stacked Eagle3 fc_norm weights") + return None + cache = ( + draft_model.w["fc_norm0"], + torch.stack( + [draft_model.w[f"fc_norm{i}"].detach() for i in range(layers)] + ).contiguous(), + ) + self._m3_fc_norm_w = cache + hidden = self._m3_fc_hidden + self._m3_fc_hidden = None + if hidden is None or hidden.shape[0] != num_tokens: + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + hs = m3_spec_tail.chunk_norm(hs, cache[1], draft_model.rms_eps) + else: + hs = self._m3_renorm_raw_captures(hs, spec_metadata, draft_model) + hidden = draft_model.fc(hs) + input_ids = accepted_tokens[:, : spec_metadata.runtime_draft_len + 1].reshape(-1) + return {"input_ids": input_ids, "position_ids": position_ids, "hidden_states": hidden} + + def _m3_renorm_raw_captures( + self, hs: torch.Tensor, spec_metadata, draft_model: M3Eagle3Draft + ) -> torch.Tensor: + """The captured rows ``hs`` [rows, layers * H] for a reader that skips the fc_norm (the capture norm is armed): + unchanged when every captured layer was written with this step's norm (capture_hidden_states records them), + else -- not expected on any path -- a copy with each layer captured raw normalized by its own fc_norm here, + logged as a miss, so a raw row never reaches the drafter's fc.""" + missing = [ + i + for i, layer_id in enumerate(spec_metadata.layers_to_capture) + if layer_id not in self._m3_capture_normed + ] + if not missing: + return hs + missed( + "m3_capture_norm", + int(hs.shape[0]), + f"captured layer(s) {[spec_metadata.layers_to_capture[i] for i in missing]} were written without the " + "capture norm; their fc_norm is applied before the drafter's fc", + ) + hs = hs.clone() + hidden = spec_metadata.hidden_size + for i in missing: + block = hs[:, i * hidden : (i + 1) * hidden] + block.copy_( + flashinfer_rmsnorm( + block.contiguous(), draft_model.w[f"fc_norm{i}"], draft_model.rms_eps + ) + ) + return hs + + def _m3_fc_fork( + self, + spec_metadata, + num_tokens: int, + draft_model: Optional[M3Eagle3Draft], + after: Optional[torch.cuda.Event] = None, + ) -> Optional[torch.cuda.Stream]: + """Launch the first draft pass's fc_norm + fc on a forked stream beside the target head GEMM (after the event + ``after`` when given, else after everything queued on the current stream); the result is picked up by + _m3_first_drafter_inputs. Returns the stream to join, or None when the fc path does not apply (the first + drafter inputs then compute it).""" + self._m3_fc_hidden = None + if draft_model is None: + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + or self._m3_fc_norm_w is None + ): + return None + aux = self._m3_aux_stream + if aux is None: + aux = self._m3_aux_stream = torch.cuda.Stream() + if after is not None: + aux.wait_event(after) + else: + aux.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(aux): + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + hs = m3_spec_tail.chunk_norm( + hs, self._m3_fc_norm_w[1], draft_model.rms_eps, use_pdl=False + ) + else: + hs = self._m3_renorm_raw_captures(hs, spec_metadata, draft_model) + self._m3_fc_hidden = draft_model.fc(hs) + return aux + + def _m3_fc_join(self, fork: torch.cuda.Stream) -> None: + """Join the fc fork onto the current stream and keep its output's memory alive for that stream. + + _m3_fc_hidden is allocated in fork (aux) stream order and read on the current stream. Without the record, + dropping it returns its block to the aux stream's pool, where later aux work can take it while the current + stream's reads are still queued (eager warm-up / mixed steps). Nothing is recorded under graph capture, as in + maybe_execute_in_parallel: a captured graph relies on the next fork's aux.wait_stream(main) to order the + reuse.""" + main = torch.cuda.current_stream() + main.wait_stream(fork) + hidden = self._m3_fc_hidden + if ( + hidden is not None + and hidden.is_cuda + and not (torch.compiler.is_compiling() or torch.cuda.is_current_stream_capturing()) + ): + hidden.record_stream(main) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model: Optional[M3Eagle3Draft] + ) -> None: + """Before every target forward: whether this step's Eagle3 captures apply the drafter's fc_norm themselves. + + On a fused-verify step whose fc_norm weights are already stacked (by an earlier fused first pass), the + capture writes rmsnorm(h + r) * fc_norm[i] (one kernel per captured layer instead of add + copy) and the fused + first drafter inputs read it without their own chunk_norm. A fused-verify step of at most + ``FC_EARLY_MAX_ROWS`` verify rows is also armed to fork the drafter's first fc at the last captured layer + (``capture_hidden_states``). + """ + self._m3_capture_norm = None + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = None + # The captured layers this step wrote with its capture norm (capture_hidden_states). + self._m3_capture_normed = set() + # A fork result no first draft pass consumed must not reach a later step's (same-size) first inputs. + self._m3_fc_hidden = None + if draft_model is None: + return + if not self.tail_eligible(attn_metadata, spec_metadata, lm_head): + return + # The fork reads every captured column at the last captured layer, so a final-state capture, which is written + # after the model returns, keeps the fc at its stock place. + if ( + attn_metadata.num_seqs * (spec_metadata.runtime_draft_len + 1) <= FC_EARLY_MAX_ROWS + and FINAL_HIDDEN_LAYER_ID not in spec_metadata.layers_to_capture + ): + self._m3_fc_early_drafter = draft_model + fc_norm_w = self._m3_fc_norm_w + if ( + fc_norm_w is not None + and fc_norm_w[0] is draft_model.w["fc_norm0"] + and len(spec_metadata.layers_to_capture) == fc_norm_w[1].shape[0] + ): + self._m3_capture_norm = (fc_norm_w[1], draft_model.rms_eps) + + @staticmethod + def _m3_sampled_verify_ok(spec_metadata) -> bool: + """Whether sampled verify rows can use the fused sampled op: only under advanced_sampling_mode=no_topk, the + deploy-time guarantee that no row uses top-k or min-p (that op applies neither).""" + mode = getattr(spec_metadata, "advanced_sampling_mode", None) + return mode is not None and mode.skips_top_k and not mode.skips_top_p + + def _m3_target_accept( + self, + hidden_states, + spec_metadata, + batch_size: int, + draft_len: int, + draft_model: Optional[M3Eagle3Draft] = None, + kv_lens: Optional[torch.Tensor] = None, + ): + """Fused verification (see tail_eligible); same outputs as the strict acceptance path. ``kv_lens``: the + requests' KV lengths at this verify (the per-request forced-acceptance key, ``FORCE_AL_PER_REQUEST``).""" + force_total = 0 + force_vec = None + if self.force_num_accepted_tokens != 0.0: + is_warmup = spec_metadata.is_cuda_graph and not torch.cuda.is_current_stream_capturing() + if not torch.cuda.is_current_stream_capturing(): + # The draw pool is built by an H2D copy and the draw kernel JIT-compiles per block width, so both + # happen here, before the first captured step (a scratch counter keeps the real draw sequence). + self._ensure_force_accept_rng_state(hidden_states.device) + if not self._m3_force_vec_warm: + scratch = torch.zeros(1, dtype=torch.int64, device=hidden_states.device) + dummy = torch.empty(64, dtype=torch.int32, device=hidden_states.device) + for block in (16, 32, 64): + m3_force_vec( + self._force_accept_rng_pool, + scratch, + dummy, + block, + 1, + 2, + 0.5, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + m3_force_vec_req( + self._force_accept_rng_pool, + dummy, + dummy, + block, + 1, + 2, + 0.5, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + self._m3_force_vec_warm = True + if not is_warmup: + force_total = min(int(self.force_num_accepted_tokens) + 1, draft_len + 1) + force_vec = self._m3_force_vec( + batch_size, force_total, draft_len + 1, hidden_states.device, kv_lens + ) + draft_tokens = spec_metadata.draft_tokens.view(batch_size, draft_len) + if draft_tokens.dtype != torch.int32: + draft_tokens = draft_tokens.to(torch.int32) + # Column 0 receives the last accepted token, the next step's first input (the fused draft loop adds the rest). + self._m3_next_new = torch.empty( + (batch_size, draft_len + 1), dtype=torch.int32, device=hidden_states.device + ) + rows = hidden_states[: batch_size * (draft_len + 1)] + lm_head = self._m3_target_lm_head + batch_idx = spec_metadata.batch_indices_cuda[:batch_size] + head = self._m3_coded_head(lm_head, rows.shape[0]) + # The Eagle3 fc on the aux stream: the fork made at the last captured layer, else one beside the head. A + # sampled verify's coded head holds every SM while it streams: its fc is forked after it instead (overlapping + # the sampled acceptance), where it does not delay the head's start. + fork = self._m3_fc_early_stream + self._m3_fc_early_stream = None + head_first = ( + fork is None + and (head is not None or rows.shape[0] <= FC_AFTER_HEAD_MAX_ROWS) + and not spec_metadata.is_all_greedy_sample + ) + if fork is None and not head_first: + fork = self._m3_fc_fork(spec_metadata, rows.shape[0], draft_model) + # This rank's vocabulary shard of the target logits: the head's own local GEMM (or its coded copy), whose + # all-gather the fused kernels replace (they exchange per-row records across ranks, not logits). + if head is not None: + taken(TARGET_HEAD.path, rows.shape[0]) + logits = m3_target_head_kernel.logits(head, rows) + else: + logits = lm_head.apply_linear(rows, None) + head_done = None + if head_first: + head_done = torch.cuda.Event() + head_done.record() + vocab_offset = self.mapping.tp_rank * lm_head.weight.shape[0] + if spec_metadata.is_all_greedy_sample: + result = m3_shard_accept_kernel.shard_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + force_total, + self._m3_next_new, + force_vec=force_vec, + ) + path = SHARD_ACCEPT.path + else: + n_rows = rows.shape[0] + seeds, offsets = self._rng_state_per_token(spec_metadata, n_rows) + result = m3_sampled_accept_kernel.sampled_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + spec_metadata.temperatures[:n_rows], + spec_metadata.top_ks[:n_rows], + spec_metadata.top_ps[:n_rows], + None, + seeds, + offsets, + force_total, + self._m3_next_new, + force_vec=force_vec, + one_round_k=SAMPLED_ACCEPT_1R_K if SAMPLED_ACCEPT_SM107TP4 else 0, + prewait_params=SAMPLED_ACCEPT_PREWAIT_PARAMS, + early_trigger=SAMPLED_ACCEPT_EARLY_TRIGGER, + ilp_walk=SAMPLED_ACCEPT_ILP_WALK, + ) + path = "spec_tail" + if head_done is not None: + # Enqueued after the sampled acceptance (which starts first) but waits on the head only. + fork = self._m3_fc_fork(spec_metadata, rows.shape[0], draft_model, after=head_done) + if fork is not None: + self._m3_fc_join(fork) + if result is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, None, "the fused verify's exchange workspace") + raise RuntimeError( + "M3 fused verify unavailable after tail_eligible() accepted the step" + ) + taken( + SHARD_ACCEPT.path if spec_metadata.is_all_greedy_sample else "m3_sampled_accept", + batch_size, + f"draft length {draft_len}" + + ( + "" + if spec_metadata.is_all_greedy_sample + else f", SA-1R one_round_k {SAMPLED_ACCEPT_1R_K if SAMPLED_ACCEPT_SM107TP4 else 0}" + ), + ) + return result + + def prebuild_target_head(self, lm_head: nn.Module) -> bool: + """Build the 12-bit coded copy of the target head shard the fused verify streams at <= TARGET_HEAD_MAX_ROWS + rows (rebuilt from the current weight values on every call), on every rank. True when the coded copy decodes + to the BF16 weight bit for bit; otherwise (or for a head the fused verify does not read) nothing is kept and + the head's own GEMM runs.""" + self._m3_target_head = None + if not _sharded_head(lm_head) or not m3_target_head_kernel.head_supported(lm_head): + return False + head = m3_target_head_kernel.prepare_head(lm_head) + self._m3_target_head = head + if not head.coded: + logger.warning( + f"[M3 target head] the coded shard {tuple(lm_head.weight.shape)} does not decode to the BF16 weight " + "bit for bit; the fused verify keeps the head's BF16 GEMM" + ) + return head.coded + + def _m3_coded_head(self, lm_head: nn.Module, rows: int): + """The coded target head for a fused verify of ``rows`` rows, or None for the head's own GEMM: above + ``TARGET_HEAD_MAX_ROWS`` rows (uncovered), on a shard whose coding failed its round trip (uncovered: the GEMM + is the exact path), and when the copy was not built for this weight (a miss).""" + path = TARGET_HEAD.path + if not TARGET_HEAD.covers(rows): + uncovered(path, f"{rows} verify rows (the head's own GEMM runs)") + return None + head = self._m3_target_head + if head is None or head.weight is not lm_head.weight: + missed( + path, + rows, + "the coded target head is not built for the head's current weight (prebuild)", + ) + return None + if not head.coded: + uncovered( + path, "the coded shard failed its bitwise round trip (the head's own GEMM runs)" + ) + return None + return head + + def _m3_force_vec( + self, + batch_size: int, + base_total: int, + max_total: int, + device, + kv_lens: Optional[torch.Tensor] = None, + ) -> Optional[torch.Tensor]: + """Per-request forced accepted counts for a fractional forced acceptance, else None. + + ``FORCE_AL_PER_REQUEST`` (with ``kv_lens``): each request draws from the fixed-seed pool at its own key, its + KV length at this verify (its prompt plus what it has accepted so far), so its draws do not depend on the + server's history or on the other requests; every TP rank holds the same lengths, so the ranks agree. Without + lengths: exactly as _apply_force_accepted_tokens draws for a decode-only batch (one shared counter bump per + step, the (counter, slot) hash), the generic path's sequence. + """ + frac = self.force_num_accepted_tokens - int(self.force_num_accepted_tokens) + if frac <= 0.0 or base_total >= max_total: + return None + self._ensure_force_accept_rng_state(device) + out = torch.empty(batch_size, dtype=torch.int32, device=device) + if FORCE_AL_PER_REQUEST and kv_lens is not None: + keys = kv_lens[:batch_size] + if keys.dtype != torch.int32: + keys = keys.to(torch.int32) + return m3_force_vec_req( + self._force_accept_rng_pool, + keys, + out, + batch_size, + base_total, + max_total, + float(frac), + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + max(16, 1 << (batch_size - 1).bit_length()), + ) + return m3_force_vec( + self._force_accept_rng_pool, + self._force_accept_rng_counter, + out, + batch_size, + base_total, + max_total, + float(frac), + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + max(16, 1 << (batch_size - 1).bit_length()), + ) + + # ---------------------------------------------------------------------------------------------- fused draft loop + + def prebuild_draft_head(self, draft_model: Optional[M3Eagle3Draft]) -> bool: + """Build the streamed copy of the drafter's BF16 head shard that the fused draft loop reads (rebuilt from the + current weight values on every call), on every rank. False when the drafter is not one the loop serves.""" + self._m3_draft_head = None + if draft_model is None or not self._m3_drafter_supported(draft_model): + return False + self._m3_draft_head = m3_draft_step_kernel.prepare_head( + draft_model.lm_head, self.mapping.tp_rank, DRAFT_HEAD_CODED + ) + return True + + def _m3_drafter_supported(self, draft_model: M3Eagle3Draft) -> bool: + """The drafter the fused draft loop runs (``M3Eagle3Draft`` validates its shape at construction) over this + worker's tensor-parallel group with a bias-free vocab-sharded BF16 head without padding.""" + if self.mapping is None or self.mapping.tp_size <= 1 or self.mapping.enable_attention_dp: + return False + if self._d2t is not None: + return False + lm_head = draft_model.lm_head + return _sharded_head(lm_head) and m3_draft_step_kernel.head_supported(lm_head) + + def _m3_draft_ar_norm( + self, + draft_model: M3Eagle3Draft, + x: torch.Tensor, + residual: torch.Tensor, + norm_key: str, + pushed: bool = False, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """One drafter boundary of the fused draft loop: the multicast all-reduce + residual + RMSNorm in its drafter + form (``DRAFT_MC_AR``; bitwise the stock boundary), else the built-in path (``M3Eagle3Draft.ar_norm``). The + choice depends on shapes and a collectively created workspace only, so the TP ranks agree. ``pushed``: the + projection already pushed its rows into the all-reduce (``DRAFT_AR_PUSH``).""" + rows = x.shape[0] + path = DRAFT_MC_AR.path + if DRAFT_MC_AR.covers(rows): + workspace = m3_mc_allreduce.get_workspace(self.mapping, x.shape[-1]) + out = None + if workspace is not None: + out = m3_mc_allreduce.m3_mc_allreduce_norm( + x.contiguous(), + residual.contiguous(), + draft_model.w[norm_key], + draft_model.rms_eps, + workspace, + use_pdl=get_env_enable_pdl(), + drafter_norm=True, + carveout=DRAFT_MC_AR_CARVEOUT, + prepushed=2 if pushed else 0, + drafter_cluster=DRAFT_MC_AR_CLUSTER, + ) + if out is not None: + taken( + path, + rows, + "drafter boundary (cluster kernel)" + if DRAFT_MC_AR_CLUSTER + else "drafter boundary (row kernel)", + ) + if pushed: + taken(DRAFT_AR_PUSH_PATH, rows, "the projection pushed its rows (prepushed 2)") + elif DRAFT_AR_PUSH: + uncovered( + DRAFT_AR_PUSH_PATH, + "the boundary's projection did not push (the AR pushes x)", + ) + return out + if pushed: + raise RuntimeError( + "the drafter's rows were pushed into a boundary all-reduce that did not run" + ) + missed(path, rows, "the multicast all-reduce workspace or operand layout") + return draft_model.ar_norm(x, residual, norm_key) + + def _m3_arm_draft_push(self, weight: torch.Tensor, rows: int) -> None: + """``DRAFT_AR_PUSH``: let the projection ``weight``'s next low-M split-K call push its rows into the boundary + all-reduce that ``_m3_draft_ar_norm`` then runs (the multicast path only; shapes and a collectively created + workspace decide it, so the TP ranks agree).""" + if not (DRAFT_AR_PUSH and DRAFT_MC_AR.covers(rows)): + return + workspace = m3_mc_allreduce.get_workspace(self.mapping, weight.shape[0]) + if workspace is not None: + _draft_lowm_kernel.arm_push(weight, workspace, rows) + + def _m3_fused_loop_ok( + self, attn_metadata, spec_metadata, batch_size: int, draft_model: Optional[M3Eagle3Draft] + ) -> bool: + """Whether this step drafts with the fused loop: a decode-only step the fused verify or the seed prepared + (``_m3_first_gather_ids``) with the fused first draft inputs, a supported drafter, a (draft length, requests) + ``DRAFT_STEP`` covers, the prebuilt head and the exchange workspace. Host state only, so the TP ranks + agree.""" + if ( + self._m3_first_gather_ids is None + or not self._m3_fused_first_inputs + or draft_model is None + ): + return False + path = DRAFT_STEP.path + if not self._m3_drafter_supported(draft_model) or not hasattr( + attn_metadata, "kv_lens_cuda" + ): + uncovered( + path, "the drafter or its attention metadata is not one the fused loop serves" + ) + return False + draft_len = spec_metadata.runtime_draft_len + if not DRAFT_STEP.covers((draft_len, batch_size)): + uncovered(path, f"draft length {draft_len} at {batch_size} requests") + return False + head = self._m3_draft_head + if head is None or head.weight is not draft_model.lm_head.weight: + missed( + path, + batch_size, + "the drafter head's streamed copy is not built for its current weight (prebuild)", + ) + return False + if m3_draft_step_kernel.workspace(self.mapping, head.wf.device) is None: + capture_missed(path, None, "the draft-step exchange workspace") + return False + return True + + def _forward_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ): + """The fused draft loop on a step it covers (``_m3_fused_loop_ok``), else the in-tree loop.""" + if self._m3_fused_loop_ok(attn_metadata, spec_metadata, batch_size, draft_model): + return self._m3_fused_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + batch_size, + num_accepted_tokens, + ) + return super()._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + + def _m3_fused_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model: M3Eagle3Draft, + draft_kv_cache_manager, + batch_size: int, + num_accepted_tokens, + ): + """Mirrors Eagle3OneModelWorker._forward_linear_draft_loop for a decode-only Eagle3 step, with each pass's + glue fused (see ``_m3_fused_loop_ok``): + + * the draft input -- the embedding lookup, the layer's input norm and hidden norm and their concat -- in one + launch (``m3_spec_tail.draft_input``; the second pass gathers the last accepted rows there); + * the drafter layer's attention and MLP, each followed by the all-reduce with the residual add and the next + RMSNorm (the post-attention norm, then the drafter's final norm); + * one kernel for the BF16 head over the head rows (pass 0: the last accepted row of each request), the + cross-rank argmax and the bookkeeping the in-tree loop does between passes: the draft token, the next pass's + input id and position, the draft KV lengths (and pass 0's sequence lengths) (``m3_draft_step``). + + The host metadata transitions are the in-tree loop's. The drafts land in columns 1.. of the step's next-input + tokens (column 0 is the last accepted token), which ``_forward_impl`` returns as they lie. + """ + draft_len = spec_metadata.runtime_draft_len + table = draft_model.embed_table + dev = table.device + head = self._m3_draft_head + gather_ids = self._m3_first_gather_ids + ids = inputs["input_ids"] + hidden = inputs["hidden_states"] + pos = inputs["position_ids"] + if ids.dtype != torch.int32: + ids = ids.to(torch.int32) + if pos.dtype != torch.int32: + pos = pos.to(torch.int32) + num_acc = num_accepted_tokens[:batch_size] + if num_acc.dtype != torch.int32: + num_acc = num_acc.to(torch.int32) + next_new = self._m3_next_new + if next_new is None: + # Seeded from the in-tree acceptance: the last accepted token is the first-pass input at the gather row. + next_new = torch.empty((batch_size, draft_len + 1), dtype=torch.int32, device=dev) + next_new[:, 0] = ids[gather_ids] + draft_tokens = next_new[:, 1:] + w = draft_model.w + eps = draft_model.rms_eps + + with self.draft_kv_cache_context(attn_metadata, draft_kv_cache_manager) as md: + for i in range(draft_len): + x, residual = m3_spec_tail.draft_input( + ids, + table, + hidden, + gather_ids if i == 1 else None, + w["in_norm"], + eps, + w["hidden_norm"], + eps, + ) + taken(DRAFT_INPUT.path, x.shape[0]) + rows = x.shape[0] + self._m3_arm_draft_push(w["o"], rows) + h = draft_model.attention(x, md, reduce=False) + h, residual = self._m3_draft_ar_norm( + draft_model, + h, + residual, + "post_norm", + pushed=_draft_lowm_kernel.take_pushed(h), + ) + self._m3_arm_draft_push(w["dn"], rows) + h = draft_model.mlp(h, reduce=False, silu_cpp=True) + h, to_save = self._m3_draft_ar_norm( + draft_model, + h, + residual, + "final_norm", + pushed=_draft_lowm_kernel.take_pushed(h), + ) + if draft_model.return_hidden_post_norm: + to_save = h + ids_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + pos_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + if not m3_draft_step_kernel.draft_step( + head, + h, + batch_size, + gather_ids if i == 0 else None, + self.mapping, + draft_tokens, + i, + ids_next, + pos, + pos_next, + md.kv_lens_cuda, + num_acc if i == 0 else None, + md._seq_lens_cuda if i == 0 else None, + draft_len, + ): + raise RuntimeError( + "M3 fused draft step unavailable after _m3_fused_loop_ok() accepted the step" + ) + taken(DRAFT_STEP.path, batch_size, f"draft length {draft_len}") + ids, pos, hidden = ids_next, pos_next, to_save + if i == 0: + # The draft step set the device sequence lengths to 1 and rolled the KV lengths back. + md._seq_lens[:batch_size].fill_(1) + md.on_update() + has_kv_cache = md.kv_cache_manager is not None + if has_kv_cache and hasattr(md, "host_request_types"): + md.host_request_types[: md.num_contexts].fill_(1) + if has_kv_cache: + md.num_contexts = 0 + self._prepare_flash_mla_generation_layout(md, 0, batch_size) + md.update_for_spec_dec() + md.use_spec_decoding = False + else: + md.update_for_spec_dec() + self._m3_next_new_tokens = next_new + return draft_tokens + + +def _build_speculation(shell, model_config: ModelConfig): + """The drafter config, the drafter and the worker of one-model Eagle3 (the in-tree one-engine construction for + the llama3 drafter architecture), or of one-model standalone DSpark (``dspark.build_dspark``: the in-tree drafter + and worker).""" + spec = model_config.spec_config + if _dspark.is_standalone_dspark(spec): + shell.model.embed_tokens = _dspark.M3EmbedTokens(shell.model) + return _dspark.build_dspark(shell, model_config) + if ( + not spec.spec_dec_mode.use_one_engine() + or not spec.spec_dec_mode.is_eagle3_one_model() + or spec.eagle3_model_arch != "llama3" + or getattr(spec, "use_dynamic_tree", False) + ): + raise ValueError( + "this MiniMax-M3 target speculates with linear one-model Eagle3 (llama3 drafter) only" + ) + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, + trust_remote_code=True, + attn_backend=model_config.attn_backend, + moe_backend=_get_requested_draft_moe_backend(model_config, spec), + mapping=model_config.mapping, + spec_config=model_config.spec_config, + max_num_tokens=model_config.max_num_tokens, + moe_max_num_tokens=model_config.moe_max_num_tokens, + ) + _set_draft_kv_cache_quant_algo(draft_config, model_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = M3Eagle3Draft( + shell.model, + draft_config, + model_config.pretrained_config.num_hidden_layers, + spec.num_capture_layers, + ) + worker = M3Eagle3Worker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + return draft_config, draft, worker, use_separate + + +# -------------------------------------------------------------------------------------------------------------------- +# Runtime hooks and the shell +# -------------------------------------------------------------------------------------------------------------------- + + +def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: + """The step-loop hooks this target opts into (each behind its constant above). + + * ``pack_step_inputs``: a decode step's small input copies go out as one upload and one scatter at the graph + replay instead of a chain of copies the next graph waits on (~30 us of idle GPU at one request). + * ``skip_unchanged_h2d``: steady decode re-uploads slot indices, query lengths and page tables that rarely change. + * ``loop_on_execution_stream``: the overlap loop's own stream work and the forward on one stream, so step N+1's + inputs wait on graph N in program order rather than through two cross-stream event hops. + * ``merge_tuned_tactics_across_ranks``: tensor-parallel peers capture the same GEMM tactics (rank-local profiling + noise otherwise splits them, and the slowest rank sets every step). + * ``persist_flashinfer_mxfp8_tactics``: with TLLM_AUTOTUNER_CACHE_PATH set, the FlashInfer MXFP8 tactics tuned in + warmup and the decode-graph pass are saved with the cache, so a restart reuses them instead of re-profiling. + * ``disagg_draft_kv``: in disaggregated serving the drafter's prompt context window travels with the target's KV, + so a generation worker's drafter does not start from an empty window. + * ``skip_greedy_rng_upload``: all-greedy speculative batches never read the Philox seeds. + * ``spec_step_fold``: the spec sampler's slot stores and the engine's next-step input gathers in one launch each. + * ``pack_parallel_scatter`` / ``pack_in_graph`` / ``pack_early_copy``: the pack's copies scatter in parallel, issued + by the decode graph's own prologue from a window uploaded ahead of the replay on a side copy stream. + * ``pack_spec_rng_uploads`` / ``spec_small_h2d``: the spec step's seeds, accepted counts and slot ids ride the pack + or stay on the device. + * ``sampler_d2h_side_stream`` / ``iteration_timing_one_event``: the sampler's store copies and the iteration timing + leave the main stream. + * ``skip_accepted_draft_upload`` / ``spec_batch_map_changed_only`` (kill switches ``SKIP_ACCEPTED_UPLOAD`` / + ``BATCH_CHANGED_ONLY``, ``STEP_SKIP_ACCEPTED_UPLOAD`` / ``STEP_BATCH_CHANGED_ONLY``): no per-step upload of the + accepted-draft counts (no reader here), and the drafter's batch_indices / batch-to-slot map uploaded only on + change. + * MSA: identity selections for short sequences (and short/long graph families), the one-launch KV-length update, + unchanged block-table copies skipped, pool views built once, block-id tables copied from the page-index views. + """ + return RuntimeHooks( + fp8_saturating_casts=True, + msa_kv_cache_fixes=True, + draft_pool_fixes=True, + first_content_emit_iters=FIRST_CONTENT_EMIT_ITERS, + mxfp8_pinned_tactics=True, + late_compile_detect=True, + pack_step_inputs=STEP_PACK_INPUTS, + pack_parallel_scatter=STEP_PACK_PARALLEL_SCATTER, + pack_in_graph=STEP_PACK_IN_GRAPH, + pack_early_copy=STEP_PACK_EARLY_COPY, + pack_step_epoch=STEP_PACK_EPOCH, + pack_prologue_op=m3_step_prologue if STEP_PACK_CPP_PROLOGUE else None, + skip_unchanged_h2d=STEP_SKIP_UNCHANGED_H2D, + h2d_changed_span=STEP_H2D_CHANGED_SPAN, + kv_lens_update_once=STEP_KV_LENS_UPDATE_ONCE, + spec_offsets_op=m3_spec_offsets if STEP_SPEC_OFFSETS_ONE_LAUNCH else None, + loop_on_execution_stream=STEP_LOOP_ON_EXECUTION_STREAM, + merge_tuned_tactics_across_ranks=STEP_MERGE_TUNED_TACTICS, + persist_flashinfer_mxfp8_tactics=True, + # Bundled name-keyed tactic pins (m31_pins); kill: TRTLLM_M31_DISABLE_BUNDLED_PINS=1. + install_tactic_pins=lambda rank: m31_pins.install("tp4", rank), + disagg_draft_kv=True, # kill: TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV=1 + skip_greedy_rng_upload=STEP_SKIP_GREEDY_RNG_UPLOAD, + pack_spec_rng_uploads=STEP_PACK_SPEC_RNG_UPLOADS, + spec_small_h2d=STEP_SPEC_SMALL_H2D, + skip_accepted_draft_upload=STEP_SKIP_ACCEPTED_UPLOAD, + spec_batch_map_changed_only=STEP_BATCH_CHANGED_ONLY, + spec_step_fold=spec_fold, + sampler_d2h_side_stream=STEP_SAMPLER_D2H_SIDE_STREAM, + iteration_timing_one_event=STEP_ONE_TIMING_EVENT, + msa=MsaRuntimePolicy( + short_sequence_identity=MSA_SHORT_SEQUENCE_IDENTITY, + kv_update=_msa_kv_update_kernel.msa_kv_update if MSA_KV_UPDATE else None, + skip_unchanged_block_offsets=MSA_SKIP_UNCHANGED_BLOCK_OFFSETS, + reuse_pool_views=MSA_REUSE_POOL_VIEWS, + block_ids_from_views=MSA_BLOCK_IDS_FROM_VIEWS, + forward_staging=True, + ), + ) + + +# TOPP_DET (ported from the sm_103 targets): the spec sampler's top-p renorm (FlashInfer +# top_p_renorm_probs in the no-top-k rejection path, the draft's and the verify's call each step) runs as one +# persistent C++ launch (ops/sampling/m31_topp_det, kernels/minimaxM31/toppRenormDetSm103.cu, built for sm_107 +# in-tree or overlaid as tensorrt_llm/libs/libm31_topp_det.so) through the sampler's top-p renorm override, instead of +# FlashInfer's five launches with a one-CTA-per-row apply (AirTopPRenormApply: 63 us per call, 126 us per step at TP8 +# K7 c1 on VR). +# Bitwise FlashInfer's deterministic renorm on rows with top_p < 1; rows with top_p >= 1 keep every token +# (an intentional difference: FlashInfer drops tail buckets there in ~43% of rows). Taken up to ``_topp_det.MAX_ROWS`` +# rows; larger calls, inputs outside the contract and builds without the op keep the stock call. Kill switch: +# TRTLLM_M31_DISABLE_TOPP_DET=1 (stock FlashInfer top_p_renorm_probs). +TOPP_DET = not m31_disabled("TOPP_DET") +_TOPP_DET_STATE: dict = {} + + +def _install_topp_det() -> None: + """Once per process: install ``_topp_det.top_p_renorm_det`` as the sampler's top-p renorm override when TOPP_DET + is on and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _TOPP_DET_STATE: + return + if not TOPP_DET: + state = "off (TRTLLM_M31_DISABLE_TOPP_DET=1); stock FlashInfer top_p_renorm_probs" + elif _topp_det.topp_det_op() is None: + state = "not taken: the op is not built; stock FlashInfer top_p_renorm_probs" + else: + set_top_p_renorm_override(_topp_det.top_p_renorm_det) + ctas = _topp_det.ctas_per_row(_topp_det.MAX_ROWS, _topp_det.sm_count(torch.device("cuda"))) + state = f"engaged (ctas_per_row={ctas}, rows <= {_topp_det.MAX_ROWS})" + _TOPP_DET_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01 TOPP_DET det top-p renorm: {state}") + + +def _prebuild_topp_det() -> None: + """The override's persistent workspace on this rank's device, allocated before warmup and any graph capture.""" + if _TOPP_DET_STATE.get("installed"): + _topp_det.prebuild() + + +class ModelingV2MiniMaxM31Nvfp4Sm107Tp4( + DecoderModelForCausalLM[MiniMaxM3Tp4ModelingV2, PretrainedConfig] +): + """MiniMax-M3 NVFP4 / sm_107 / tp4, text only, with one-model Eagle3 or no speculation. + + The shell owns the vocab-parallel BF16 ``lm_head`` and the logits processor (``DecoderModelForCausalLM``); with + speculation, the drafter (``M3Eagle3Draft``, its weights from the speculative checkpoint through + ``load_draft_weights``) and the worker (``M3Eagle3Worker``) that verifies and drafts each step. + """ + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + assert torch.cuda.get_device_capability() == _SM, ( + f"this target is built for sm_{_SM[0]}{_SM[1]}, the device is sm_" + f"{''.join(map(str, torch.cuda.get_device_capability()))}" + ) + # Under modeling_v2='require' a covered decode shape that cannot take its fast path raises. + set_require(resolved_mode(type(self)) is ModelingV2Mode.REQUIRE) + model_config = _text_model_config(model_config) + cfg = model_config.pretrained_config + spec_config = getattr(model_config, "spec_config", None) + if ( + model_config.quant_config.kv_cache_quant_algo == QuantAlgo.NVFP4 + and _dspark.draft_kv_override_applies(spec_config) + ): + # The one-model Eagle3 layer has no NVFP4 trtllm-gen decode kernel: keep its modules and the shared + # draft cache on FP8 while the target stays NVFP4. + model_config.extra_attrs["draft_kv_cache_quant_algo_override"] = QuantAlgo.FP8 + super().__init__( + MiniMaxM3Tp4ModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.spec_config = spec_config + # A standalone DSpark drafter takes none of the Eagle3 drafter's hooks (full-context attention, low-M + # linears, boundary capture-norm target). + self._standalone_dspark = _dspark.is_standalone_dspark(spec_config) + self.draft_model: Optional[nn.Module] = None + self.draft_config: Optional[ModelConfig] = None + self.spec_worker: Optional[Union[M3Eagle3Worker, _dspark.M3DSparkWorker]] = None + self.use_separate_draft_kv_cache = False + self.layer_idx = FINAL_HIDDEN_LAYER_ID + spec_fold = None + if spec_config is not None: + ( + self.draft_config, + self.draft_model, + self.spec_worker, + self.use_separate_draft_kv_cache, + ) = _build_speculation(self, model_config) + # The step fold serves the Eagle3 worker's sampler slots; the DSpark worker keeps the in-tree ones. + if STEP_SPEC_FOLD and not _dspark.is_standalone_dspark(spec_config): + spec_fold = M3SpecStepFold( + in_forward=STEP_SPEC_FOLD_IN_FORWARD, + store_copy_wait=STEP_SPEC_STORE_COPY_SIDE, + cpp_tail=STEP_FOLD_TAIL_CPP, + ) + elif DSFOLD and _dspark.is_standalone_dspark(spec_config): + # The DSpark worker keeps the in-tree sampler slot stores; only the engine's input gathers fold. + spec_fold = M3DSparkStepFold() + self._spec_fold = spec_fold + self.runtime_hooks = _runtime_hooks(spec_fold) + _install_topp_det() + + def load_weights(self, weights: Dict, *args, **kwargs) -> None: + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_draft_weights(self, weights: Dict, weight_mapper=None) -> None: + """The drafter's checkpoint (``speculative_model``), loaded by the engine after the target's.""" + if _dspark.is_standalone_dspark(self.spec_config): + _dspark.load_dspark_draft_weights(self, weights, weight_mapper) + return + _weights.DRAFT_WEIGHTS.load(self.draft_model, weights) + + def set_guided_decoder(self, guided_decoder) -> bool: + """Mirrors ``SpecDecOneEngineForCausalLM.set_guided_decoder``: guided decoding with speculation is installed + on the worker.""" + if hasattr(self.spec_worker, "set_guided_decoder"): + return self.spec_worker.set_guided_decoder(guided_decoder) + return False + + def cache_derived_state(self) -> None: + """Build the targets and every decode kernel once the weights are real, before the executor's warmup; with + speculation also the drafter layer's full-context decode attention (``DRAFTER_FULLCTX``), the fused verify's + coded target head, the fused draft loop's head copy and the drafter's low-M GEMMs (``DRAFT_LOWM``).""" + super().cache_derived_state() + self.model.post_load_weights() + _prebuild_topp_det() + draft = self.draft_model + eagle3_draft = draft is not None and not self._standalone_dspark + if eagle3_draft and DRAFTER_FULLCTX.keys: + path = DRAFTER_FULLCTX.path + installed = _drafter_attn_kernel.install( + draft, + {k: FULLCTX_DECODE_RECIPES[k] for k in DRAFTER_FULLCTX.keys}, + FULLCTX_DECODE_MAX_KV_LEN, + FULLCTX_DECODE_OWN_REDUCER and _fullctx_decode_kernel.has_own_reducer(), + get_env_enable_pdl(), + fused_rope_mode=( + DRAFTER_ROPE_MODE + if DRAFTER_ROPE_FUSED + and hasattr(torch.ops.trtllm, "minimax_m3_sm107tp4_drafter_rope_fp8_kv") + else None + ), + on_rope_taken=lambda rows, what: taken(DRAFTER_ROPE_PATH, rows, what), + on_taken=lambda rows: taken(path, rows), + on_uncovered=lambda reason: uncovered(path, reason), + on_missed=lambda rows, reason: missed(path, rows, reason), + ) + logger.info(f"[M3 prebuild] {path} installed on {installed} drafter attention layer(s)") + if self.spec_worker is not None: + self.spec_worker.prebuild(self.lm_head, draft) + if eagle3_draft: + # The drafter linears' low-M kernels, before the warmup. + start = time.perf_counter() + built = draft.prebuild_linears() + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] drafter low-M GEMMs built before warmup: %d calls, %.1f s" + % (built, time.perf_counter() - start) + ) + + def post_load_weights(self) -> None: + super().post_load_weights() + self.model.post_load_weights() + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.Tensor] = None, + position_ids: Optional[torch.Tensor] = None, + inputs_embeds: Optional[torch.Tensor] = None, + return_context_logits: bool = False, + spec_metadata: Optional[SpecMetadata] = None, + resource_manager: Optional[Any] = None, + **kwargs, + ) -> Any: + """One engine step: the target forward, then the logits (or, with speculation, the worker's verify and + draft). Text only.""" + multimodal_params = kwargs.pop("multimodal_params", None) or [] + if any(p is not None and getattr(p, "multimodal_data", None) for p in multimodal_params): + raise ValueError( + "this MiniMax-M3 target is text only; the request carries multimodal input" + ) + worker = self.spec_worker + capture = _plain_capture + if worker is not None: + capture = worker.capture_hidden_states + if spec_metadata is not None: + worker.arm_capture_norm( + attn_metadata, spec_metadata, self.lm_head, self.draft_model + ) + hidden_states = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + capture=capture, + capture_target=( + worker.capture_target + if worker is not None and not self._standalone_dspark + else None + ), + **kwargs, + ) + if getattr(attn_metadata, "csa2_remote_tail_mode", None) == "source": + # A context worker stopped at the encoder/decoder boundary: a disposable token for the context-only + # completion, and no draft worker. + return self.logits_processor.forward(hidden_states, self.lm_head, attn_metadata, False) + if spec_metadata is not None and spec_metadata.is_layer_capture(FINAL_HIDDEN_LAYER_ID): + capture(spec_metadata, FINAL_HIDDEN_LAYER_ID, hidden_states, None) + if attn_metadata.padded_num_tokens is not None: + hidden_states = hidden_states[: attn_metadata.num_tokens] + if worker is None: + return self.logits_processor.forward( + hidden_states, self.lm_head, attn_metadata, return_context_logits + ) + # The fused verify computes this rank's logits shard itself and needs no gathered logits. + if worker.tail_eligible(attn_metadata, spec_metadata, self.lm_head): + logits = None + else: + # A worker may produce the gathered target logits itself (bitwise to the processor); None = the processor. + target_logits = getattr(worker, "target_logits", None) + logits = ( + target_logits(hidden_states, attn_metadata, spec_metadata, self.lm_head) + if target_logits is not None + else None + ) + if logits is None: + logits = self.logits_processor.forward( + hidden_states[spec_metadata.gather_ids], self.lm_head, attn_metadata, True + ) + spec_input_ids = input_ids if input_ids is not None else kwargs.get("orig_input_ids") + spec_position_ids = position_ids + if attn_metadata.padded_num_tokens is not None: + if spec_input_ids is not None: + spec_input_ids = spec_input_ids[: attn_metadata.num_tokens] + if position_ids is not None: + spec_position_ids = _slice_spec_position_ids(position_ids, attn_metadata.num_tokens) + outputs = worker( + input_ids=spec_input_ids, + position_ids=spec_position_ids, + hidden_states=hidden_states, + logits=logits, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=resource_manager, + ) + spec_fold = self._spec_fold + if ( + spec_fold is not None + and spec_fold.in_forward + and attn_metadata.num_contexts == 0 + and isinstance(outputs, dict) + ): + spec_fold.store_in_forward(outputs) + return outputs diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/mxdraft.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/mxdraft.py new file mode 100644 index 000000000000..90b8c6bcd706 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/mxdraft.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MXDRAFT: the DSpark drafter's decode projections on this target's MXFP8 GEMV. + +The drafter checkpoint ships its linears as MXFP8 (E4M3 codes, one UE8M0 scale per 32 columns) and the stock drafter +dequantizes them to BF16 at load, so every decode pass streams twice the bytes it needs. MXDRAFT recovers the MXFP8 +form of each BF16 weight (exactly: the recovery is checked to dequantize back to the same BF16 bits, else the +projection is left alone) and routes the projection's GEMM to ``m3_mx_gemv_bf16`` for decode inputs it serves; any +other call runs the Linear unchanged. The activation is quantized to MXFP8 inside the GEMV, so the output is not the +stock BF16 GEMM's: drafter-only precision, gated on natural AL. Row-parallel projections keep +their all-reduce (only the Linear's GEMM is replaced). +""" + +from __future__ import annotations + +from typing import Iterable, Optional, Tuple + +import torch + +from .kernels.mx_gemv import op as _gemv +from .ops.comm.m3_mc_allreduce_norm import swizzle_sf + +E4M3_MAX = 448.0 +BLOCK = 32 + + +def mxfp8_from_bf16(weight: torch.Tensor) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """The MXFP8 form (E4M3 codes [N, K], UE8M0 scale bytes [N, K / 32]) of a BF16 weight that is exactly an MXFP8 + dequantization, or None. Each block takes the smallest power-of-two scale that fits its amax in E4M3, which is at + most the shipped scale, so the codes are exact whenever the weight came from MXFP8.""" + if weight.dtype != torch.bfloat16 or weight.dim() != 2 or weight.shape[1] % BLOCK != 0: + return None + n, k = weight.shape + blocks = weight.float().view(n, k // BLOCK, BLOCK) + amax = blocks.abs().amax(-1) + exp = torch.ceil(torch.log2(amax.clamp_min(2.0**-126) / E4M3_MAX)).clamp(-127, 127) + scale = torch.exp2(exp).unsqueeze(-1) + codes = (blocks / scale).to(torch.float8_e4m3fn) + back = (codes.float() * scale).view(n, k).to(torch.bfloat16) + if not torch.equal(back, weight): + return None + return codes.view(n, k).contiguous(), (exp + 127).to(torch.uint8).contiguous() + + +def _projections(draft_model) -> Iterable[torch.nn.Module]: + for layer in getattr(getattr(draft_model, "model", None), "layers", []) or []: + attn, mlp = getattr(layer, "self_attn", None), getattr(layer, "mlp", None) + for lin in ( + getattr(attn, "qkv_proj", None), + getattr(attn, "o_proj", None), + getattr(mlp, "gate_up_proj", None), + getattr(mlp, "down_proj", None), + ): + if lin is not None: + yield lin + + +def route_mx_linears(draft_model, rows: Iterable[int]) -> int: + """Route each drafter projection whose BF16 weight is an exact MXFP8 dequantization to the MXFP8 GEMV for 2-D + contiguous bf16 inputs of the given row counts (the decode passes'); compiles those kernels now (outside capture). + Returns how many projections were routed.""" + rows = sorted({int(m) for m in rows if 0 < int(m) <= _gemv.MAX_TOKENS}) + n_routed = 0 + for lin in _projections(draft_model): + weight = getattr(lin, "weight", None) + if ( + weight is None + or getattr(lin, "bias", None) is not None + or getattr(lin, "_m31_mxdraft", None) is not None + ): + continue + rec = mxfp8_from_bf16(weight.data) + if rec is None: + continue + n, k = weight.shape + served = tuple(m for m in rows if _gemv.serves(n, k, m, True)) + if not served: + continue + w8, sf = rec[0], swizzle_sf(rec[1]) + for m in served: + _gemv.precompile(w8, sf, m, True) + original = lin.apply_linear + + def apply_linear( + input, + bias, + lora_params=None, + layer_idx=None, + *, + _o=original, + _w8=w8, + _sf=sf, + _served=served, + ): + if ( + bias is None + and not lora_params + and isinstance(input, torch.Tensor) + and input.dtype == torch.bfloat16 + and input.dim() == 2 + and input.shape[0] in _served + and input.is_contiguous() + ): + out = _gemv.m3_mx_gemv_bf16(input, _w8, _sf) + if out is not None: + return out + return _o(input, bias, lora_params, layer_idx) + + lin._m31_mxdraft = (w8, sf, served) + lin.apply_linear = apply_linear + n_routed += 1 + return n_routed diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/__init__.py new file mode 100644 index 000000000000..1059445198db --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Ops used only by this target. Shared ops stay in the catalog.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/activation/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/activation/__init__.py new file mode 100644 index 000000000000..b8c7426e3933 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/activation/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Activation ops of this target: the gated MLPs' fused SwiGLU.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/activation/m3_silu_and_mul.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/activation/m3_silu_and_mul.py new file mode 100644 index 000000000000..5bcdee3fbb82 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/activation/m3_silu_and_mul.py @@ -0,0 +1,83 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The gated MLPs' fused SwiGLU: ``silu_and_mul`` with MiniMax-M3's SwiGLU-OAI gain, offset and clamp.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SiluAndMul(OpWrapper): + """``out = g * sigmoid(alpha * g) * (u + beta)`` over ``x = [g | u]`` (bf16 [M, 2I] -> [M, I]), with + ``g = min(gate, limit)`` and ``u = clamp(up, -limit, limit)``; ``limit``, ``alpha`` and ``beta`` None give plain + SwiGLU ``silu(gate) * up``. + + The MXFP8 dense MLPs (layers 0-2, intermediate 3072 per rank) and the MoE's shared expert (intermediate 768 per + rank) run it at MiniMax-M3's (limit 7, alpha 1.702, beta 1); the BF16 Eagle3 drafter's MLP (intermediate 4608 per + rank) runs it plain. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert, one decode token", spec=dict(m=1, inter=768, oai=True)), + Cell(why="shared expert, a 48-row decode verify", spec=dict(m=48, inter=768, oai=True)), + Cell(why="dense MLP, a prefill chunk", spec=dict(m=2048, inter=3072, oai=True)), + Cell(why="dense MLP, one decode token", spec=dict(m=1, inter=3072, oai=True)), + Cell( + why="drafter MLP (plain SwiGLU), eight decode rows", + spec=dict(m=8, inter=4608, oai=False), + ), + Cell( + why="drafter MLP (plain SwiGLU), a prefill chunk", + spec=dict(m=1024, inter=4608, oai=False), + ), + ) + + note = """ + The kernel computes in fp32 and rounds once to bf16; `reference` does the same arithmetic in fp32, so the two + differ only by the fast sigmoid's last bits (the test's band). + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + + def reference( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + gate, up = x.float().chunk(2, dim=-1) + if limit is not None: + gate = gate.clamp(max=limit) + up = up.clamp(min=-limit, max=limit) + a = 1.0 if alpha is None else alpha + b = 0.0 if beta is None else beta + return (gate * torch.sigmoid(a * gate) * (up + b)).to(x.dtype) + + def is_valid( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> None: + assert x.stride(-1) == 1, "the gate/up halves are read from contiguous rows" + + +m3_silu_and_mul = M3SiluAndMul() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/__init__.py new file mode 100644 index 000000000000..a7f7fee7a234 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Attention ops of this target: MSA block selection, the sparse decode and the KV-length update.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_dense_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_dense_decode.py new file mode 100644 index 000000000000..ba008f493b94 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_dense_decode.py @@ -0,0 +1,190 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' decode attention (trtllm-gen over the full page table) with the KV split this target sizes per +request count (``modeling.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH``).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.trtllm_gen_dense_decode import ( # noqa: E501 + minimax_m3_trtllm_gen_dense_decode, +) + +__all__ = ["m3_dense_decode"] + +#: The TP4 dense layers' geometry: 16 query heads and one KV head of dim 128 per rank, an FP8 KV pool. +HEADS = 16 +KV_HEADS = 1 +HEAD_DIM = 128 +#: The split this target hands trtllm-gen at one request (modeling.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH), and a model +#: maximum the in-tree step passes otherwise. +CAPPED = 16384 +MODEL_MAX = 1 << 20 + + +def dense_attention_reference( + q: torch.Tensor, + keys: torch.Tensor, + values: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, +) -> torch.Tensor: + """fp32 causal decode attention: query row ``b * qlen + i`` of request b attends its first + ``seq_lens[b] - qlen + i + 1`` keys. ``keys`` / ``values``: [batch, kv_heads, max_len, head_dim] (the request's + tokens in order). Returns fp32 [total_q, heads, head_dim].""" + batch, kv_heads, max_len, _ = keys.shape + heads = q.shape[1] + group = heads // kv_heads + out = torch.zeros(q.shape, dtype=torch.float32, device=q.device) + pos = torch.arange(max_len, device=q.device) + with true_fp32_matmul(): + for b in range(batch): + full = int(seq_lens[b]) + rows = q[b * decode_query_len : (b + 1) * decode_query_len].float() # [qlen, heads, d] + limit = full - decode_query_len + 1 + torch.arange(decode_query_len, device=q.device) + valid = pos[None, :] < limit[:, None] # [qlen, max_len] + for h in range(kv_heads): + qh = rows[:, h * group : (h + 1) * group] # [qlen, group, d] + logits = torch.einsum("qgd,kd->qgk", qh, keys[b, h].float()) * sm_scale + logits = logits.masked_fill(~valid[:, None, :], float("-inf")) + probs = torch.softmax(logits, dim=-1) + out[ + b * decode_query_len : (b + 1) * decode_query_len, h * group : (h + 1) * group + ] = torch.einsum("qgk,kd->qgd", probs, values[b, h].float()) + return out + + +def gather_paged_kv( + kv_cache_manager, layer_idx: int, block_table: torch.Tensor, seq_lens: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor]: + """The K and V of each request in token order, [batch, kv_heads, max_len, head_dim], from the flat sub-page pool + (``get_kv_subpage_pool``: K of slot s at sub-page s * subpages_per_slot, V one sub-page later).""" + pool, subpages = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + page = pool.shape[2] + pages = (int(seq_lens.max()) + page - 1) // page + slots = block_table[:, :pages].long() + k = pool[slots * subpages] # [batch, pages, kv_heads, page, d] + v = pool[slots * subpages + 1] + batch, _, kv_heads, _, d = k.shape + + def order(t): + return t.permute(0, 2, 1, 3, 4).reshape(batch, kv_heads, pages * page, d) + + return order(k), order(v) + + +class M3DenseDecode(OpWrapper): + """``minimax_m3_trtllm_gen_dense_decode`` as ``MsaDecodeFmha`` calls it for a dense layer, with the max_seq_len + ``msa_decode.dense_decode_max_seq_len`` resolves: this target's per-request-count cap, else the step's bound. + + Decode attention of every query row over its request's full causal context in the paged FP8 pool, written into + ``output`` (BF16 [T, heads, head_dim]) in place. ``max_seq_len`` only sizes trtllm-gen's multi-CTA KV split: + every CTA still covers its share of each request's actual length, which may exceed it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over a 20000-token context, the split sized for 16384", + spec=dict(kv_lens=(20000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request, one row, over a 65536-token context (an AgentX-length cache) at the capped split", + spec=dict(kv_lens=(65536,), qlen=1, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request verifying 1 + 3 rows over a 3000-token context: the cap above the length", + spec=dict(kv_lens=(3000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="two requests keep the model maximum (no cap at two requests)", + spec=dict(kv_lens=(20000, 5000), qlen=4, max_seq_len=MODEL_MAX, capped=False), + ), + ) + + note = """ + The split only reorders the merge of the CTAs' partial softmax sums, so the capped split's output is the uncapped + one's to a few BF16 ulps (the test holds it there), and both are held to the fp32 reference within the FP8-KV + band of the in-tree dense decode test. + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + minimax_m3_trtllm_gen_dense_decode( + q, + kv_cache_manager, + layer_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + max_seq_len=max_seq_len, + max_num_requests=max_num_requests, + ) + return output + + def reference( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + """fp32 attention over the pool's K / V (q rounded to the pool's dtype first, as the kernel runs it). + + ``output``, ``max_seq_len`` and ``max_num_requests`` do not change the result.""" + keys, values = gather_paged_kv(kv_cache_manager, layer_idx, block_table, seq_lens) + if keys.dtype == torch.float8_e4m3fn: + q = fp8_e4m3_saturate(q) + return dense_attention_reference(q, keys, values, seq_lens, decode_query_len, sm_scale) + + def is_valid( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> None: + # The multi-CTA KV counters are sized for max_num_requests: more requests overrun them. + assert seq_lens.shape[0] <= max_num_requests, "more requests than the counters cover" + # Each verify row attends a causal prefix of at least one key. + assert bool((seq_lens >= decode_query_len).all()), "a request shorter than its query rows" + + +m3_dense_decode = M3DenseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_dense_qk_norm_rope_fp8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_dense_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..d9e8ca93fff7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_dense_qk_norm_rope_fp8.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' per-head RMSNorm + partial NeoX RoPE of the packed [Q | K | V] into FP8, the bits of the BF16 +``fused_qk_norm_rope`` output cast to E4M3; optionally also storing the live tokens' K / V into the paged FP8 cache.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3DenseQkNormRopeFp8(OpWrapper): + """``qkv`` [T, (q + 2 kv) heads x 128] bf16 -> [T, same] E4M3: Q and K heads normed and roped in fp32, rounded to + BF16 and then to E4M3 (saturating), V cast; ``qkv`` untouched + (``trtllm::minimax_m3_sm107tp4_dense_qk_norm_rope_fp8``). + + With ``kv_buffers`` (the layer's HND FP8 pool [pages, 2, kv heads, 128, 128]), ``out_cache_loc`` (the step's int32 + cache slots) and ``num_live_tokens``, the kernel also stores the live tokens' K and V rows into the pool (the bytes + the MSA backend's KV write stores) and every row's Q into ``q_compact`` [T, q heads, 128] + (``..._dense_qk_norm_rope_fp8_kv``). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, tp4 heads (16 q, 1 kv)", spec=dict(m=1, q=16, kv=1)), + Cell( + why="a 4-request verify (16 rows) storing K / V", + spec=dict(m=16, q=16, kv=1, kv_write=True), + ), + Cell(why="a 48-row decode verify", spec=dict(m=48, q=16, kv=1)), + Cell(why="a prefill chunk on an FP8 cache", spec=dict(m=2048, q=16, kv=1)), + ) + + note = """ + Bit-exact: `reference` runs the catalog's in-place BF16 ``fused_qk_norm_rope`` on a copy and casts every element to + E4M3, which is this kernel's rounding sequence (fp32 -> BF16 -> E4M3) for Q and K and the cast for V. + """ + + def __call__( + self, + qkv: torch.Tensor, + num_heads_q: int, + num_heads_k: int, + num_heads_v: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + is_neox: bool, + position_ids: torch.Tensor, + use_gemma: bool, + kv_buffers: Optional[torch.Tensor] = None, + out_cache_loc: Optional[torch.Tensor] = None, + num_live_tokens: int = 0, + q_compact: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + args = ( + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + use_gemma, + ) + if kv_buffers is not None: + return torch.ops.trtllm.minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv( + *args, kv_buffers, out_cache_loc, num_live_tokens, q_compact + ) + return torch.ops.trtllm.minimax_m3_sm107tp4_dense_qk_norm_rope_fp8(*args) + + def reference( + self, + qkv: torch.Tensor, + num_heads_q: int, + num_heads_k: int, + num_heads_v: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + is_neox: bool, + position_ids: torch.Tensor, + use_gemma: bool, + kv_buffers: Optional[torch.Tensor] = None, + out_cache_loc: Optional[torch.Tensor] = None, + num_live_tokens: int = 0, + q_compact: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """The FP8 copy (the K / V store into ``kv_buffers`` is checked by the test against the reference rows).""" + head_dim = qkv.shape[-1] // (num_heads_q + num_heads_k + num_heads_v) + out = qkv.clone() + fused_qk_norm_rope( + out, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + 1.0, + 0.0, + 0.0, + 1.0, + True, + use_gemma, + False, + 0, + 0, + ) + return out.to(torch.float8_e4m3fn) + + +m3_dense_qk_norm_rope_fp8 = M3DenseQkNormRopeFp8() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_draft_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_draft_attention.py new file mode 100644 index 000000000000..ed4e3048d780 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_draft_attention.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter layer's attention on the TRTLLM backend: RoPE, KV-cache append and causal attention.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + PredefinedAttentionMask, +) +from tensorrt_llm.functional import PositionEmbeddingType, RotaryScalingType + + +class M3DraftAttention(OpWrapper): + """The drafter layer's attention over its fused QKV rows ``qkv`` [T, (q + 2 kv) heads x 128] (bf16): the + backend applies the drafter's RoPE (fused), appends K / V to the drafter layer's paged cache (FP8, the target's + NVFP4 cache keeps its draft layer in FP8) and runs causal attention over the step's live rows (``metadata`` + ``num_tokens``); returns [num_tokens, q heads x 128] bf16. + + ``backend`` (``TrtllmAttention``) and ``metadata`` are the runtime's objects for the drafter layer and step. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="first draft pass of a 4-request verify (4 x 4 rows)", spec=dict(reqs=4, qlen=4)), + Cell(why="a later draft pass, one row per request", spec=dict(reqs=8, qlen=1)), + Cell(why="a context chunk's draft rows", spec=dict(reqs=1, context=512)), + ) + + note = """ + `reference` is exact (float64) causal softmax attention: Q rotated by the backend's NeoX RoPE at each token's + position, over the drafter layer's K / V rows read back from its FP8 cache (dequantized by the backend's + ``kv_scale_quant_orig``). What remains between the two is the FMHA's own arithmetic (Q's rounding to the + kernel's input precision, its P rounding and accumulation order), which the test bounds. + """ + + def __call__(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + num_tokens = metadata.num_tokens + out = backend.forward( + qkv[:num_tokens, :], + None, + None, + metadata, + forward_args=AttentionForwardArgs(attention_mask=PredefinedAttentionMask.CAUSAL), + ) + return out[0] if isinstance(out, tuple) else out + + def reference(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + """The call's [num_tokens, q heads x 128] output, in ``qkv``'s dtype. + + Query token ``j`` of request ``r`` sits at position ``cached_r + j`` (the metadata's cached lengths) and + attends positions up to it. Its Q is rotated here; every K / V row, the step's own included, is read from + the drafter layer's paged cache at the slots the metadata's KV cache manager holds, so this is evaluated + after the call (whose roped, quantized append it then reads back; the test checks that append itself). + Mirrors one KV head per rank and un-scaled NeoX RoPE, the drafter's configuration. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + assert backend.num_kv_heads == 1, "mirrors the one-KV-head-per-rank layout only" + rope = backend.rope_params + assert backend.position_embedding_type == int(PositionEmbeddingType.rope_gpt_neox) + assert rope.scale_type == RotaryScalingType.none and rope.alpha == 1.0 + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + kv_scale = float(backend.kv_scale_quant_orig.reshape(-1)[0]) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = int(metadata.num_tokens) + assert sum(qlens) == total + device = qkv.device + + rot = int(rope.dim) + inv_freq = float(rope.theta) ** ( + -torch.arange(0, rot, 2, dtype=torch.float64, device=device) / rot + ) + pool = manager.get_buffers(layer, "HND") + q = qkv[:total, : heads * dim].to(torch.float64).reshape(total, heads, dim) + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][ + : (length + page - 1) // page + ], + dtype=torch.int64, + device=device, + ) + keys = pool[pages, 0, 0].to(torch.float64).reshape(-1, dim)[:length] * kv_scale + values = pool[pages, 1, 0].to(torch.float64).reshape(-1, dim)[:length] * kv_scale + qpos = cached[row] + torch.arange(n, device=device) + angles = qpos.to(torch.float64)[:, None] * inv_freq[None, :] + cos, sin = angles.cos()[:, None, :], angles.sin()[:, None, :] + rows = q[first : first + n] + x1, x2 = rows[..., : rot // 2], rows[..., rot // 2 : rot] + rows = torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin, rows[..., rot:]], dim=-1) + allowed = torch.arange(length, device=device)[None, :] <= qpos[:, None] + logits = torch.einsum("thd,kd->thk", rows, keys) * sm_scale + logits = logits.masked_fill(~allowed[:, None, :], float("-inf")) + out[first : first + n] = torch.softmax(logits, dim=-1) @ values + first += n + return out.reshape(total, heads * dim).to(qkv.dtype) + + +m3_draft_attention = M3DraftAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_fp8_indexer_qk_norm_rope.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_fp8_indexer_qk_norm_rope.py new file mode 100644 index 000000000000..48543be75675 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_fp8_indexer_qk_norm_rope.py @@ -0,0 +1,114 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The unfused sparse layers' index branch: per-head Gemma RMSNorm + partial NeoX RoPE of [index-Q | index-K], index-Q +out in FP8 E4M3 and index-K inserted into the FP8 index-K cache, in one kernel.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3Fp8IndexerQkNormRope(OpWrapper): + """``qk`` [T, (heads + 1) * 128] BF16 (the index projection's output): returns index-Q [T, heads, 128] E4M3 and + writes index-K (E4M3) at ``out_cache_loc`` of ``index_k_cache`` [pages, 1, 128, 128]. Gemma RMSNorm of each head, + then RoPE on its leading ``rotary_dim`` channels (NeoX). + + The unfused sparse layers (``fuse_qkv_index_projection=False``) run it on the index branch with an FP8 indexer; + the fused layers' producer writes the index cache itself. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, one index head per rank", spec=dict(m=1, heads=1)), + Cell(why="a 48-row decode verify", spec=dict(m=48, heads=1)), + Cell(why="a prefill chunk", spec=dict(m=2048, heads=1)), + ) + + note = """ + `reference` is the in-place fused norm + RoPE in BF16 followed by the E4M3 cast (the kernel rounds through BF16 + before the cast); its index-K is what the kernel writes to the cache. The two kernels may round a handful of + values differently: the test allows fewer than 0.1% of the bytes to differ, as the in-tree test does. + """ + + def __call__( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> torch.Tensor: + return torch.ops.trtllm.minimax_m3_fp8_indexer_qk_norm_rope( + qk, + index_k_cache, + out_cache_loc, + num_heads_q, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + position_ids, + ) + + def reference( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor]: + """(index-Q [T, heads, head_dim] E4M3, index-K [T, head_dim] E4M3): the catalog's in-place kernel on a copy, + then the E4M3 cast.""" + out = fused_qk_norm_rope( + qk.clone(), + num_heads_q, + 1, + 0, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + True, + position_ids, + 1.0, + 0.0, + 0.0, + 1.0, + True, + True, + False, + 0, + 0, + ) + q, k = out.split([num_heads_q * head_dim, head_dim], dim=-1) + return ( + q.reshape(q.shape[0], num_heads_q, head_dim).to(torch.float8_e4m3fn), + k.to(torch.float8_e4m3fn), + ) + + +m3_fp8_indexer_qk_norm_rope = M3Fp8IndexerQkNormRope() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_fullctx_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_fullctx_decode.py new file mode 100644 index 000000000000..ad3abcadaae9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_fullctx_decode.py @@ -0,0 +1,142 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' full-context decode attention at TP4 on Rubin (``kernels/m3_fullctx_decode``): PrimTS split-KV +decode at the recipe ``modeling.FULLCTX_DECODE_RECIPES`` pins per (query rows per request, requests).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) + +from ...kernels import m3_fullctx_decode as kernel +from .m3_dense_decode import dense_attention_reference, gather_paged_kv + +__all__ = ["m3_fullctx_decode"] + +HEADS = kernel.HEADS +KV_HEADS = kernel.KV_HEADS +HEAD_DIM = kernel.HEAD_DIM +#: The plan capacity (the per-request K/V length bound PrimTS sizes its policy for): dispatch's +#: FULLCTX_DECODE_MAX_KV_LEN, which the forward's prebuilt plans use. +MAX_KV_LEN = 1 << 21 + + +class M3FullCtxDecode(OpWrapper): + """Decode attention of every query row over its request's full causal context in the paged FP8 pool, written + into ``output`` (BF16 [T, heads, head_dim]) in place. + + ``recipe`` is ``modeling.FULLCTX_DECODE_RECIPES[(qlen, requests)]``: the PrimTS Q tile and the split-KV fanout. + ``own_reducer`` merges the splits on the BF16 op (``modeling.FULLCTX_DECODE_OWN_REDUCER``) rather than PrimTS's + FP16 reducer and a cast. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over an AgentX-length (165k) context", + spec=dict(kv_lens=(165525,), qlen=4), + ), + Cell( + why="one request, one draft-pass row, over a 165k context", + spec=dict(kv_lens=(165522,), qlen=1), + ), + Cell( + why="two requests verifying 1 + 3 rows at unequal lengths (165k, 120k)", + spec=dict(kv_lens=(165525, 120011), qlen=4), + ), + Cell( + why="two requests, one row each, at unequal lengths (165k, 120k)", + spec=dict(kv_lens=(165522, 120008), qlen=1), + ), + Cell( + why="one request verifying 1 + 3 rows over a short (4.1k) context: most splits are empty", + spec=dict(kv_lens=(4104,), qlen=4), + ), + ) + + note = """ + Same operands and arithmetic as the stock trtllm-gen call (E4M3 q, E4M3 K/V, P quantized to E4M3 per split, + fp32 accumulation): the split differs, so the result differs from stock as a stock split change does (the P + quantization is per split). Deterministic, and the same bits eager and in a CUDA graph. Held to the fp32 + reference within the FP8-KV band of the in-tree dense decode test. + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + recipe, + max_kv_len: int = MAX_KV_LEN, + own_reducer: bool = True, + ) -> torch.Tensor: + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + shape = (decode_query_len, int(seq_lens.shape[0])) + kernel.prebuild(q.device, {shape: recipe}, max_kv_len, own_reducer, get_env_enable_pdl()) + plan = kernel.plan_for(q.device, *shape, own_reducer) + pool, subpages = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + k_pages, v_pages = kernel.kv_page_views(pool, subpages) + plan.run(fp8_e4m3_saturate(q), k_pages, v_pages, block_table, seq_lens, sm_scale, output) + return output + + def reference( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + recipe, + max_kv_len: int = MAX_KV_LEN, + own_reducer: bool = True, + ) -> torch.Tensor: + """fp32 attention over the pool's K / V, q rounded to E4M3 first as the kernel runs it.""" + keys, values = gather_paged_kv(kv_cache_manager, layer_idx, block_table, seq_lens) + return dense_attention_reference( + fp8_e4m3_saturate(q), keys, values, seq_lens, decode_query_len, sm_scale + ) + + def is_valid( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + recipe, + max_kv_len: int = MAX_KV_LEN, + own_reducer: bool = True, + ) -> None: + pool, _ = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + assert pool.dtype == torch.float8_e4m3fn, "an FP8 E4M3 pool" + assert tuple(pool.shape[1:]) == (KV_HEADS, kernel.PAGE, HEAD_DIM), ( + "the TP4 dense-layer pool geometry" + ) + assert q.shape[1:] == (HEADS, HEAD_DIM), "16 query heads of 128 per rank" + assert int(q.shape[0]) == int(seq_lens.shape[0]) * decode_query_len, ( + "q covers exactly the batch" + ) + if torch.cuda.is_current_stream_capturing(): + return # the length checks read the device: eager calls only + assert bool((seq_lens >= decode_query_len).all()), "a request shorter than its query rows" + assert bool((seq_lens <= max_kv_len).all()), "a request longer than the plan capacity" + + +m3_fullctx_decode = M3FullCtxDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_index_score.py new file mode 100644 index 000000000000..f44d566310e7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_index_score.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# PyTorch oracle ported from vLLM (Apache-2.0), _reference_decode_index_score: +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/tests/kernels/attention/test_minimax_m3.py#L188 +"""The MSA index scorer of a sparse decode step: per (index head, query token, KV block) the causal max of Q . K over +the block's 128 index-K positions, with the pre-wait schedule of ``kernels/m3_index_score``.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_index_score"] + +PAGE = 128 +HEAD_DIM = 128 +#: CTAs per request (each scores blocks split, split + SPLIT_K, ...; the scores do not depend on it). +SPLIT_K = 512 + + +def compiled_scorer(key: tuple): + """The compiled kernel of ``key`` (dtype, heads, query length, head dim, split_k, early), compiling it on a miss + (the target compiles every key it can take before warmup: ``layers.prebuild_index_scorer``).""" + from ...kernels.m3_index_score import IndexScoreRunner + + if key not in IndexScoreRunner.kernel_cache: + IndexScoreRunner._compile(*key) + return IndexScoreRunner.kernel_cache[key] + + +class M3IndexScore(OpWrapper): + """``score[h, t, j]`` (fp32, a transposed view of the [heads, blocks, tokens] selector buffer) = the max over the + positions ``p`` of block ``j`` of request ``b = t // qlen`` with ``p <= seq_lens[b] - qlen + t % qlen`` of + ``idx_q[t, h] . index_k[block_table[b, j], p % 128]``, for ``j < ceil(seq_lens[b] / 128)`` (no other entry is + written). FP8 or BF16 q and index-K, fp32 products and sums. + + ``early`` (the target's one-request kernel) issues the Q and first K tiles right after the grid wait from a + pre-wait length hint and prefetches one older block per CTA into L2 before the wait; otherwise every older block of + the CTA is prefetched. Neither changes a score. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over 20000 tokens: the early-issue kernel", + spec=dict(seq_lens=(20000,), qlen=4, early=True), + ), + Cell( + why="one request, one row, 65536 tokens: early issue, 512 blocks on 512 CTAs", + spec=dict(seq_lens=(65536,), qlen=1, early=True), + ), + Cell( + why="one request of 129 tokens: the early tiles of most CTAs lie past the length and are drained", + spec=dict(seq_lens=(129,), qlen=4, early=True), + ), + Cell( + why="one request longer than 512 blocks (the CTAs loop): early issue", + spec=dict(seq_lens=(70000,), qlen=4, early=True), + ), + Cell( + why="two requests: the prefetching kernel", + spec=dict(seq_lens=(20000, 3000), qlen=4, early=False), + ), + Cell( + why="eight requests of mixed lengths: the prefetching kernel", + spec=dict( + seq_lens=(300, 1500, 2049, 4096, 5000, 7777, 8192, 33001), qlen=4, early=False + ), + ), + ) + + note = """ + The products of FP8 (or BF16) values are exact in fp32 and the kernel sums 128 of them in its k-split order, so it + differs from the reference matmul by a few fp32 ulps; it is held to the in-tree scorer bit for bit. + """ + + def __call__( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + _, num_heads, head_dim = idx_q.shape + key = (idx_q.dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + compiled_scorer(key)(idx_q, index_k, block_table, score, seq_lens) + return score + + def reference( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + """The fp32 [heads, total_q, blocks] scores (``-inf`` where nothing is written). ``score``, ``split_k`` and + ``early`` do not change them.""" + total_q, heads, _ = idx_q.shape + qlen = max_decode_query_len + out = torch.full( + (heads, total_q, score.shape[2]), + -float("inf"), + dtype=torch.float32, + device=idx_q.device, + ) + with true_fp32_matmul(): + for b, seq in enumerate(seq_lens.tolist()): + nblocks = (seq + PAGE - 1) // PAGE + if nblocks == 0: + continue + t0 = b * qlen + q = idx_q[t0 : t0 + qlen].float() + k = index_k[block_table[b, :nblocks].long()].reshape(nblocks * PAGE, -1).float() + s = torch.einsum("qhd,kd->hqk", q, k) + q_pos = seq - qlen + torch.arange(qlen, device=idx_q.device) + k_pos = torch.arange(k.shape[0], device=idx_q.device) + s.masked_fill_(k_pos[None, :] > q_pos[:, None], -float("inf")) + out[:, t0 : t0 + qlen, :nblocks] = s.reshape(heads, qlen, nblocks, PAGE).amax(dim=3) + return out + + def is_valid( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> None: + # The block-table row and the score row are read and written up to ceil(seq_len / 128) with no bound. + nblocks = (seq_lens + PAGE - 1) // PAGE + assert bool((nblocks <= block_table.shape[1]).all()), ( + "a request runs past its block-table row" + ) + assert bool((nblocks <= score.shape[2]).all()), "a request runs past the score buffer" + assert idx_q.shape[0] == seq_lens.shape[0] * max_decode_query_len, ( + "idx_q must hold max_decode_query_len rows per request" + ) + + +m3_index_score = M3IndexScore() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_index_score_t2sc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_index_score_t2sc.py new file mode 100644 index 000000000000..45103a9e289c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_index_score_t2sc.py @@ -0,0 +1,168 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# PyTorch oracle ported from vLLM (Apache-2.0), _reference_decode_index_score: +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/tests/kernels/attention/test_minimax_m3.py#L188 +"""The MSA index scorer of a sparse decode step: per (index head, query token, KV block) the causal max of Q . K over +the block's 128 index-K positions, with the pre-wait schedule of ``kernels/m3_index_score_t2sc`` (T2SC: the sm_107 +TP2 / TP8 scorer as this target's copy, at its 1 index head per rank; ``modeling.INDEX_SCORE_T2SC``).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_index_score_t2sc"] + +PAGE = 128 +HEAD_DIM = 128 + + +def compiled_scorer(key: tuple): + """The compiled kernel of ``key`` (dtype, heads, query length, head dim, split_k, early), compiling it on a miss + (the target compiles every key it can take before warmup: ``layers.prebuild_index_scorer_t2sc``).""" + from ...kernels.m3_index_score_t2sc import IndexScoreRunner + + if key not in IndexScoreRunner.kernel_cache: + IndexScoreRunner._compile(*key) + return IndexScoreRunner.kernel_cache[key] + + +class M3IndexScoreT2sc(OpWrapper): + """``score[h, t, j]`` (fp32, a transposed view of the [heads, blocks, tokens] selector buffer) = the max over the + positions ``p`` of block ``j`` of request ``b = t // qlen`` with ``p <= seq_lens[b] - qlen + t % qlen`` of + ``idx_q[t, h] . index_k[block_table[b, j], p % 128]``, for ``j < ceil(seq_lens[b] / 128)`` (no other entry is + written). FP8 or BF16 q and index-K, fp32 products and sums. + + ``early`` (the target's kernel at every request count it covers) issues the Q and first K tiles right after the + grid wait from a pre-wait length hint and prefetches one older block per CTA into L2 before the wait; otherwise + every older block of the CTA is prefetched. Neither, nor ``split_k``, changes a score. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request, 1 + 3 rows over 165521 tokens (AgentX ISL), 1 index head: 1536 CTAs, early issue", + spec=dict(seq_lens=(165521,), qlen=4, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request of 129 tokens: the early tiles of most of the 1536 CTAs lie past the length and drain", + spec=dict(seq_lens=(129,), qlen=4, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request of 400000 tokens: 3125 blocks, so the 1536 CTAs loop", + spec=dict(seq_lens=(400000,), qlen=4, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request decoding one row over 65536 tokens", + spec=dict(seq_lens=(65536,), qlen=1, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request, 1 + 3 rows over 165521 tokens, 1 index head (TP4): 1272 CTAs (one wave), early issue", + spec=dict(seq_lens=(165521,), qlen=4, heads=1, split_k=1272, early=True), + ), + Cell( + why="one request of 129 tokens at 1272 CTAs: the early tiles of most CTAs lie past the length and drain", + spec=dict(seq_lens=(129,), qlen=4, heads=1, split_k=1272, early=True), + ), + Cell( + why="one request of 400000 tokens at 1272 CTAs: 3125 blocks, so the CTAs loop", + spec=dict(seq_lens=(400000,), qlen=4, heads=1, split_k=1272, early=True), + ), + Cell( + why="two requests verifying 4 rows each over 165521 and 20000 tokens: 768 CTAs each, early issue", + spec=dict(seq_lens=(165521, 20000), qlen=4, heads=1, split_k=768, early=True), + ), + Cell( + why="two requests, one of 300 tokens and one of 262144 (2048 blocks: the 768 CTAs loop)", + spec=dict(seq_lens=(300, 262144), qlen=4, heads=1, split_k=768, early=True), + ), + ) + + note = """ + The products of FP8 (or BF16) values are exact in fp32 and the kernel sums 128 of them in its k-split order, so it + differs from the reference matmul by a few fp32 ulps; it is held to the in-tree scorer bit for bit. + """ + + def __call__( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + _, num_heads, head_dim = idx_q.shape + key = (idx_q.dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + compiled_scorer(key)(idx_q, index_k, block_table, score, seq_lens) + return score + + def reference( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + """The fp32 [heads, total_q, blocks] scores (``-inf`` where nothing is written). ``score``, ``split_k`` and + ``early`` do not change them.""" + total_q, heads, _ = idx_q.shape + qlen = max_decode_query_len + out = torch.full( + (heads, total_q, score.shape[2]), + -float("inf"), + dtype=torch.float32, + device=idx_q.device, + ) + with true_fp32_matmul(): + for b, seq in enumerate(seq_lens.tolist()): + nblocks = (seq + PAGE - 1) // PAGE + if nblocks == 0: + continue + t0 = b * qlen + q = idx_q[t0 : t0 + qlen].float() + k = index_k[block_table[b, :nblocks].long()].reshape(nblocks * PAGE, -1).float() + s = torch.einsum("qhd,kd->hqk", q, k) + q_pos = seq - qlen + torch.arange(qlen, device=idx_q.device) + k_pos = torch.arange(k.shape[0], device=idx_q.device) + s.masked_fill_(k_pos[None, :] > q_pos[:, None], -float("inf")) + out[:, t0 : t0 + qlen, :nblocks] = s.reshape(heads, qlen, nblocks, PAGE).amax(dim=3) + return out + + def is_valid( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> None: + # The block-table row and the score row are read and written up to ceil(seq_len / 128) with no bound. + nblocks = (seq_lens + PAGE - 1) // PAGE + assert bool((nblocks <= block_table.shape[1]).all()), ( + "a request runs past its block-table row" + ) + assert bool((nblocks <= score.shape[2]).all()), "a request runs past the score buffer" + assert idx_q.shape[0] == seq_lens.shape[0] * max_decode_query_len, ( + "idx_q must hold max_decode_query_len rows per request" + ) + + +m3_index_score_t2sc = M3IndexScoreT2sc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_msa_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_msa_attention.py new file mode 100644 index 000000000000..0ce3c864d30f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_msa_attention.py @@ -0,0 +1,224 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MiniMax-M3 MSA attention backend's step: KV write, sparse top-k block selection, FMHA.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + SparseBackendForwardArgs, +) + +from .m3_sparse_decode_lc import dequant_nvfp4_pages + + +class M3MsaAttention(OpWrapper): + """One layer's attention core on the MSA backend (``MiniMaxM3MsaSparseAttention``), written into ``output`` + [T, heads * 128] (bf16, the step's live rows). + + * ``k`` / ``v`` given (the dense layers): written into the layer's paged main cache first (NVFP4 with + ``kv_scale_orig_quant``, else FP8 / BF16 as the pool is). None: a producer already wrote the caches. + * ``sparse``: the indexer scores the layer's index-K cache (already written) with ``idx_q`` and publishes the + top-k blocks; the FMHA attends those blocks. Otherwise the FMHA attends the whole page table. + + The FMHA dispatches on the step's metadata (context rows: the MSA prefill FMHA; generation rows: the MSA decode + FMHA, the dense layers' trtllm-gen decode at their capped KV split). ``backend`` and ``metadata`` are the runtime's + objects for this layer and step. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="dense layer, a prefill chunk, FP8 pool", + spec=dict(layer="dense", phase="context", tokens=1024), + ), + Cell( + why="dense layer, a decode step of 4 requests", + spec=dict(layer="dense", phase="generation", reqs=4), + ), + Cell( + why="sparse layer, a prefill chunk beyond 16 blocks, NVFP4 pool", + spec=dict(layer="sparse", phase="context", tokens=4096), + ), + Cell( + why="sparse layer, a mixed step (context + generation rows)", + spec=dict(layer="sparse", phase="mixed", tokens=512, reqs=2), + ), + ) + + note = """ + `reference` is exact (float64) softmax attention over the cache rows the step attends, read back from the paged + cache the call wrote (or a producer wrote before it) and dequantized as the FMHA dequantizes them: FP8 rows as + E4M3, NVFP4 rows as E2M1 x E4M3 block scale x the per-tensor ``kv_scale_quant_orig``. A sparse layer's top-k blocks + are re-selected from ``idx_q`` and the index-K cache with the backend's rule (per-block max over the causal + keys, the query's own block forced in, ties to the lower block). On an FP8 pool the FMHA takes Q as a plain E4M3 + cast, which the reference applies too. What remains between the two is the FMHA's own arithmetic (its P + rounding and accumulation order), which the test bounds. + """ + + def __call__( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + if k is not None: + backend.write_layer_caches( + k, v, None, metadata, kv_scale_orig_quant=kv_scale_orig_quant + ) + if sparse: + blocks = backend.run_indexer(idx_q, None, metadata, idx_k_prewritten=True) + args = AttentionForwardArgs( + output=output, sparse_backend_args=SparseBackendForwardArgs(topk_indices=blocks) + ) + else: + args = AttentionForwardArgs(output=output) + args.kv_scale_orig_quant = kv_scale_orig_quant + args.kv_scale_quant_orig = kv_scale_quant_orig + backend.forward(q, None, None, metadata, forward_args=args) + return output + + def reference( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + """``output`` as the call is supposed to write it, [T, heads * 128] in ``output``'s dtype. + + Every attended row -- the step's own included -- is read from the layer's paged cache, at the slots the + metadata's KV cache manager holds for each request, so this is evaluated after the call (whose write of + ``k`` / ``v`` it then reads back). ``k``, ``v`` and ``kv_scale_orig_quant`` are therefore not read; the + test checks the write itself. Query token ``j`` of request ``r`` sits at position ``cached_r + j`` and + attends positions up to it; on a sparse layer only those in its top-k blocks. Mirrors one KV head per rank, + the target's layout. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + assert backend.num_kv_heads == 1, "mirrors the one-KV-head-per-rank layout only" + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = sum(qlens) + assert q.shape[0] == total and output.shape[0] == total + + pool = manager.get_buffers(layer, "HND") + nvfp4 = manager.is_nvfp4_layer(layer) + if nvfp4: + scale_pool = manager.get_block_scale_buffers(layer, "HND") + k_global = float(kv_scale_quant_orig.reshape(-1)[1]) + v_global = float(kv_scale_quant_orig.reshape(-1)[2]) + q_rows = q.reshape(total, heads, dim) + if q_rows.dtype != torch.float8_e4m3fn and pool.dtype == torch.float8_e4m3fn: + # The FP8 FMHA shares one dtype across Q / K / V: Q is cast to E4M3, unscaled. + q_rows = q_rows.to(torch.float8_e4m3fn) + q64 = q_rows.to(torch.float64) + if sparse: + config = backend.m3_config + idx_cache = metadata.msa_idx_k_cache(layer) + idx64 = idx_q.reshape(total, config.num_index_heads, config.sparse_index_dim).to( + torch.float64 + ) + + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=q.device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + blocks = (length + page - 1) // page + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][:blocks], + dtype=torch.int64, + device=q.device, + ) + if nvfp4: + keys = dequant_nvfp4_pages( + pool[:, 0].view(torch.uint8), scale_pool[:, 0], pages, quad_scales=False + ) + values = dequant_nvfp4_pages( + pool[:, 1].view(torch.uint8), scale_pool[:, 1], pages, quad_scales=True + ) + keys, values = keys * k_global, values * v_global + else: + keys = pool[pages, 0, 0].to(torch.float64) + values = pool[pages, 1, 0].to(torch.float64) + keys = keys.reshape(-1, dim)[:length] + values = values.reshape(-1, dim)[:length] + qpos = cached[row] + torch.arange(n, device=q.device) + allowed = torch.arange(length, device=q.device)[None, :] <= qpos[:, None] + if sparse: + idx_keys = idx_cache[pages, 0].to(torch.float64).reshape(-1, dim)[:length] + chosen = _selected_blocks( + idx64[first : first + n], idx_keys, allowed, qpos, page, config + ) + allowed &= chosen.repeat_interleave(page, dim=1)[:, :length] + for lo in range(0, n, 256): + hi = min(n, lo + 256) + logits = torch.einsum("thd,kd->thk", q64[first + lo : first + hi], keys) * sm_scale + logits = logits.masked_fill(~allowed[lo:hi, None, :], float("-inf")) + out[first + lo : first + hi] = torch.softmax(logits, dim=-1) @ values + first += n + return out.reshape(total, heads * dim).to(output.dtype) + + +def _selected_blocks( + idx_q: torch.Tensor, + idx_keys: torch.Tensor, + allowed: torch.Tensor, + qpos: torch.Tensor, + page: int, + config, +) -> torch.Tensor: + """The blocks each query token of one request attends, [n, blocks] bool. + + A block's score is the maximum, over the KV head's index heads and the block's causally visible keys, of + ``idx_q . idx_k``. Of the token's ``ceil((pos + 1) / page)`` valid blocks the first ``init_blocks`` and the last + ``local_blocks`` are forced in, and the ``topk`` highest-scoring are kept, ties to the lower block (the + backend's ``select_blocks_from_maxscore`` rule). The positive score scale the scorers apply does not change the + ranking, so none is applied. + """ + n, length = allowed.shape + blocks = (length + page - 1) // page + scores = torch.einsum("thd,kd->thk", idx_q, idx_keys).amax(dim=1) + scores = scores.masked_fill(~allowed, float("-inf")) + padded = torch.full((n, blocks * page), float("-inf"), dtype=scores.dtype, device=scores.device) + padded[:, :length] = scores + block_scores = padded.view(n, blocks, page).amax(dim=-1) + ids = torch.arange(blocks, device=scores.device)[None, :] + n_valid = torch.div(qpos + page, page, rounding_mode="floor")[:, None] + block_scores = block_scores.masked_fill(ids >= n_valid, float("-inf")) + forced = (ids < torch.clamp(n_valid, max=config.init_blocks)) | ( + (ids >= n_valid - config.local_blocks) & (ids < n_valid) + ) + block_scores = block_scores.masked_fill(forced, float("inf")) + order = torch.sort(block_scores, dim=1, descending=True, stable=True).indices[:, : config.topk] + keep = torch.arange(order.shape[1], device=scores.device)[None, :] < n_valid + chosen = torch.zeros(n, blocks, dtype=torch.bool, device=scores.device) + rows = torch.arange(n, device=scores.device)[:, None].expand_as(order) + chosen[rows[keep], order[keep]] = True + return chosen + + +m3_msa_attention = M3MsaAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_producer.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_producer.py new file mode 100644 index 000000000000..5a3d1b64ac58 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_producer.py @@ -0,0 +1,223 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' NVFP4 QKV + indexer RMSNorm / RoPE / KV-insert producer, with its constant loads staged +before the PDL grid-dependency wait and an early dependents trigger (``kernels/minimaxM3/producerPw.cu``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +__all__ = ["m3_producer"] + +HEAD_DIM = 128 +ROTARY_DIM = 64 +PAGE = 128 +#: Query heads per rank at TP4 (MiniMax-M3: 64 query heads, 1 KV head and 1 index head per rank). +HEADS_Q = 16 +#: NVFP4 block: 16 values share one E4M3 scale. +FP4_BLOCK = 16 +E2M1_VALUES = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0) + + +def _norm_rope( + x: torch.Tensor, weight: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor, eps: float +): + """Gemma RMSNorm (``x * rsqrt(mean(x^2) + eps) * (1 + w)``) then NeoX RoPE on the first 64 dims, in fp32. + x [T, 128] fp32; cos / sin [T, 32].""" + y = x * torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) * (1.0 + weight.float()) + lo, hi, rest = y[:, :32], y[:, 32:64], y[:, 64:] + return torch.cat([lo * cos - hi * sin, hi * cos + lo * sin, rest], dim=-1) + + +def decode_nvfp4_token( + data: torch.Tensor, scales: torch.Tensor, quant_scale: float +) -> torch.Tensor: + """fp32 [128] of one token's packed E2M1 row (64 bytes, low nibble first) and its 8 E4M3 block scales.""" + table = torch.tensor(E2M1_VALUES, dtype=torch.float32, device=data.device) + nib = torch.stack([data & 0xF, data >> 4], dim=-1).reshape(-1).long() + vals = table[nib & 7] * torch.where(nib >= 8, -1.0, 1.0) + s = scales.view(torch.float8_e4m3fn).float() / quant_scale + return vals * s.repeat_interleave(FP4_BLOCK) + + +def scale_index(within_page: int, plane: int) -> torch.Tensor: + """Positions of a token's 8 block scales in its page's [128 * 8] scale plane: row-major for K, the 4-token + swizzle for V.""" + cols = torch.arange(8) + if plane == 0: + return within_page * 8 + cols + return (within_page // 4) * 32 + cols * 4 + within_page % 4 + + +class M3Producer(OpWrapper): + """The NVFP4 QKV + indexer producer of a sparse layer: ``minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert`` for one + KV head, one index head, head dim 128 and rotary dim 64, with the same arguments and outputs bit for bit. + + ``packed`` [T, (heads_q + 4) * 128] BF16 holds each token's q heads, k, v, index-q and index-k. Every head but v + gets the Gemma RMSNorm with its weight and NeoX RoPE on its first 64 dims (``rotary_cos_sin[position]``). Returns + (q FP8 [T, heads_q, 128], index-q FP8 [T, 1, 128]); writes the token's slot ``out_cache_loc[t]`` (skipped when + negative or past the pool): k and v as NVFP4 (16-value blocks, E4M3 scales against ``kv_quant_scale[1]`` / + ``[2]``, the V scales in the 4-token swizzle) and index-k as FP8. ``early`` triggers the dependent kernel on + entry (its outputs are read only after the dependent's own grid wait). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token (c1 without speculation)", spec=dict(tokens=1, scale=1.0)), + Cell(why="a c1 verify: 4 tokens", spec=dict(tokens=4, scale=3.0)), + Cell(why="a c4 verify: 16 tokens, small activations", spec=dict(tokens=16, scale=1e-3)), + Cell( + why="a c8 verify: 32 tokens, large activations, a skipped and an out-of-pool slot", + spec=dict(tokens=32, scale=40.0, bad_slots=True), + ), + Cell(why="the widest decode step: 48 tokens", spec=dict(tokens=48, scale=1.0)), + Cell( + why="a prefill chunk of 1000 tokens, early trigger off", + spec=dict(tokens=1000, scale=1.0, early=False), + ), + ) + + note = """ + The kernel is the native producer's arithmetic, compiled with the library's --use_fast_math, so it is held to that + kernel bit for bit on every output and cache byte. The torch reference states the math in fp32: the FP8 outputs are + held to it within one E4M3 step, and the NVFP4 K/V, decoded, within one E2M1 step of their block scale. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + prewait_step_inputs: bool = False, + ) -> tuple[torch.Tensor, torch.Tensor]: + if prewait_step_inputs: + # This target's copy (producerPw4): the same outputs bit for bit, with the position ids, slots and RoPE + # rows read before its grid wait (none of them may be written by a PDL predecessor). + return tuple( + torch.ops.trtllm.minimax_m3_sm107_tp4_producer_pw( + packed, + kv_data, + kv_scale, + index_k, + out_cache_loc, + kv_quant_scale, + eps, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + rotary_cos_sin, + position_ids, + pdl, + early, + True, + ) + ) + q, index_q = torch.ops.trtllm.minimax_m3_producer_pw( + packed, + kv_data, + kv_scale, + index_k, + out_cache_loc, + kv_quant_scale, + eps, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + rotary_cos_sin, + position_ids, + heads_q, + pdl, + early, + ) + return q, index_q + + def reference( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + prewait_step_inputs: bool = False, + ) -> dict[str, torch.Tensor]: + """fp32 values before their quantization: "q" [T, heads_q, 128], "index_q" [T, 128], and per token "k", "v", + "index_k" [T, 128] (the values written to the token's slot). The caches, ``pdl`` and ``early`` do not change + them.""" + t = packed.shape[0] + x = packed.float().view(t, heads_q + 4, HEAD_DIM) + rope = rotary_cos_sin[position_ids[:t].long()] + cos, sin = rope[:, 0], rope[:, 1] + q = torch.stack( + [_norm_rope(x[:, h], q_weight, cos, sin, eps) for h in range(heads_q)], dim=1 + ) + return { + "q": q, + "k": _norm_rope(x[:, heads_q], k_weight, cos, sin, eps), + "v": x[:, heads_q + 1], + "index_q": _norm_rope(x[:, heads_q + 2], index_q_weight, cos, sin, eps), + "index_k": _norm_rope(x[:, heads_q + 3], index_k_weight, cos, sin, eps), + } + + def is_valid( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + prewait_step_inputs: bool = False, + ) -> None: + t = packed.shape[0] + # The RoPE row is read at the position with no bound. + pos = position_ids[:t] + assert bool(((pos >= 0) & (pos < rotary_cos_sin.shape[0])).all()), ( + "position_ids must index rotary_cos_sin" + ) + # Two tokens writing one slot race. + slots = out_cache_loc[:t] + live = slots[(slots >= 0) & (slots < kv_data.shape[0] * PAGE)] + assert live.unique().numel() == live.numel(), "out_cache_loc must not repeat a slot" + + +m3_producer = M3Producer() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_qk_norm_rope_fp8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..f92a3e6efaa1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_qk_norm_rope_fp8.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' per-head Gemma RMSNorm + partial NeoX RoPE of the packed [Q | K | V], into a fresh FP8 copy.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3QkNormRopeFp8(OpWrapper): + """``fused_qk_norm_rope`` written out of place in FP8 E4M3: Q and K heads normed and roped in fp32 and rounded + once to E4M3 (saturating at +/-448), V cast, ``qkv`` left untouched. Same arguments as the catalog's in-place + entry. + + The dense layers (0-2) feed the FP8 main KV cache with it: on every step of an FP8 cache, and on the decode steps + of a hybrid NVFP4 cache (whose dense layers sit in its FP8 pool). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, tp4 heads (16 q, 1 kv)", spec=dict(m=1, q=16, kv=1)), + Cell(why="a 48-row decode verify", spec=dict(m=48, q=16, kv=1)), + Cell(why="a prefill chunk on an FP8 cache", spec=dict(m=2048, q=16, kv=1)), + ) + + note = """ + `reference` is the in-place kernel followed by a cast to E4M3. V matches it bit for bit; Q and K round once from + fp32 where the composition rounds through bf16 first, so they are within one E4M3 step of it. + """ + + def __call__( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + return torch.ops.trtllm.fused_qk_norm_rope_to_fp8( + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + + def reference( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + """The catalog's in-place kernel on a copy, then the E4M3 cast (V exact; Q and K within one E4M3 step).""" + out = fused_qk_norm_rope( + qkv.clone(), + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + return out.to(torch.float8_e4m3fn) + + +m3_qk_norm_rope_fp8 = M3QkNormRopeFp8() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_qkv_insert.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_qkv_insert.py new file mode 100644 index 000000000000..ef79579c0c21 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_qkv_insert.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' generic horizontal producer: packed QKV/index rows -> FP8 Q / index-Q, K / V / index-K into +the paged caches.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3QkvInsert(OpWrapper): + """The packed ``[Q | K | V | index-Q | index-K]`` bf16 rows of a sparse layer: per-head Gemma RMSNorm and the + partial NeoX RoPE of Q, K, index-Q and index-K, K / V written to the main paged cache at ``out_cache_loc`` and + index-K to the FP8 index cache; returns (Q, index-Q) in FP8 E4M3 [M, heads, 128]. + + ``kv_scale`` given: the main cache is NVFP4 (packed E2M1 data [pages, 2, kv, 128, 64] uint8 + E4M3 scale pool + [pages, 2, kv, 128, 8], ``kv_quant_scale`` the fp32 (Q, K, V) quantization scales); None: FP8 E4M3 + [pages, 2, kv, 128, 128]. Token-major and step-type agnostic. ``m3_producer`` takes the NVFP4 cache at the tp4 + heads; this entry takes the FP8 cache and NVFP4 pools outside that layout. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="FP8 main cache, a prefill chunk", spec=dict(m=1024, cache="fp8")), + Cell(why="FP8 main cache, one decode token", spec=dict(m=1, cache="fp8")), + ) + + note = """ + The arithmetic is the CUDA producer's (fp32 norm and RoPE with bf16 roundings, then the E4M3 casts); `reference` + returns the fp32 values, and the test holds the FP8 outputs and cache rows within one E4M3 step of them. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + common = ( + num_heads, + num_kv_heads, + num_index_heads, + head_dim, + rotary_dim, + eps, + q_norm, + k_norm, + index_q_norm, + index_k_norm, + rotary_cos_sin, + position_ids, + ) + if kv_scale is not None: + return torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, kv_scale, index_k_cache, out_cache_loc, kv_quant_scale, *common + ) + return torch.ops.trtllm.minimax_m3_fp8_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, index_k_cache, out_cache_loc, *common + ) + + def reference( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + """The fp32 values the producer rounds (a new dict, nothing written): Gemma RMSNorm + NeoX partial RoPE of + Q [M, heads, D], index-Q [M, index heads, D], K [M, kv heads, D] and index-K [M, D]; V [M, kv heads, D] as + packed. The cache rows at ``out_cache_loc`` and the returned FP8 Q / index-Q are these values cast.""" + m = packed.shape[0] + sizes = [ + num_heads * head_dim, + num_kv_heads * head_dim, + num_kv_heads * head_dim, + num_index_heads * head_dim, + head_dim, + ] + q, k, v, iq, ik = packed.float().split(sizes, dim=-1) + cos, sin = rotary_cos_sin[position_ids.long()].unbind(1) + half = rotary_dim // 2 + + def norm_rope(x, w, heads): + x = x.reshape(m, heads, head_dim) + x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * (1.0 + w.float()) + x1, x2, rest = x[..., :half], x[..., half:rotary_dim], x[..., rotary_dim:] + c, s = cos.unsqueeze(1), sin.unsqueeze(1) + return torch.cat([x1 * c - x2 * s, x2 * c + x1 * s, rest], dim=-1) + + return { + "q": norm_rope(q, q_norm, num_heads), + "index_q": norm_rope(iq, index_q_norm, num_index_heads), + "k": norm_rope(k, k_norm, num_kv_heads), + "v": v.reshape(m, num_kv_heads, head_dim), + "index_k": norm_rope(ik, index_k_norm, 1)[:, 0], + } + + +m3_qkv_insert = M3QkvInsert() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_blocks.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_blocks.py new file mode 100644 index 000000000000..76841cca732d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_blocks.py @@ -0,0 +1,82 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MSA top-k block selection from per-KV-head block scores (the built-in selector).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SelectBlocks(OpWrapper): + """Per query row the ``topk`` highest-scoring KV blocks of ``max_score_kv`` [kv_heads, n_blocks, total_q] (fp32), + with the first ``init_blocks`` and the last ``local_blocks`` valid blocks forced in and blocks at or beyond the + row's ``n_valid_blocks`` excluded; [total_q, kv_heads, topk] int32 ascending ids, -1 padded. ``head_major_output`` + backs the result head-major so ``permute(1, 0, 2)`` is contiguous. + + The decode dispatch runs it where the long-row selector does not apply: requests of 17..128 KV blocks. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="eight decode rows of 17..128 blocks", + spec=dict(total_q=8, n_blocks=128, mixed=True), + ), + Cell( + why="one decode row at the 128-block edge", + spec=dict(total_q=1, n_blocks=128, mixed=False), + ), + Cell(why="a 4-row verify at 40 blocks", spec=dict(total_q=4, n_blocks=40, mixed=False)), + ) + + note = """ + Exact: ties between equal scores break toward the lower block id; `reference` sorts with the same rule. + """ + + def __call__( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + nvb = n_valid_blocks.to(device=max_score_kv.device, dtype=torch.int32).contiguous() + return torch.ops.trtllm.minimax_m3_select_blocks( + max_score_kv, nvb, topk, init_blocks, local_blocks, head_major_output + ) + + def reference( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + """Plain torch selection; ``head_major_output`` changes the backing only.""" + heads, n_blocks, total_q = max_score_kv.shape + out = torch.full((total_q, heads, topk), -1, dtype=torch.int32, device=max_score_kv.device) + blocks = torch.arange(n_blocks, device=max_score_kv.device) + for t in range(total_q): + nv = int(n_valid_blocks[t]) + for h in range(heads): + s = max_score_kv[h, :, t].clone().double() + s[blocks >= nv] = float("-inf") + forced = (blocks < min(init_blocks, nv)) | ( + (blocks >= nv - local_blocks) & (blocks < nv) + ) + s[forced] = float("inf") + order = sorted(range(nv), key=lambda b: (-float(s[b]), b))[: min(topk, nv)] + ids = torch.tensor(sorted(order), dtype=torch.int32, device=out.device) + out[t, h, : ids.numel()] = ids + return out + + +m3_select_blocks = M3SelectBlocks() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_long.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_long.py new file mode 100644 index 000000000000..3d076c8181d6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_long.py @@ -0,0 +1,186 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 MSA block selection for decode rows of up to 9216 blocks.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Blocks per decode row the op covers (9 keys per thread of one 1024-thread CTA). +MAX_BLOCKS = 9216 +#: Blocks the selection keeps per (query, KV head). +TOPK = 16 +#: Scores the forcing substitutes: initial blocks rank above local ones, and both above any finite score. +INIT_SCORE = 1.0e30 +LOCAL_SCORE = 1.0e29 + +# M3's checkpoint: no forced initial blocks, the one block holding the query forced in. +_M3 = dict(num_kv_heads=1, init_blocks=0, local_blocks=1) + + +class M3SelectLong(OpWrapper): + """Fill ``out`` [total_q, num_kv_heads, 16] with each query's top-16 blocks. + + ``scores`` is the per-KV-head block maxima [num_kv_heads, num_blocks, + total_q] the CuTe DSL index scorer wrote (amax over the KV head's index + heads). Per (query, KV head) row the op keeps the ``n_valid_blocks[query]`` + leading blocks (clamped to ``[0, num_blocks]``), forces the last + ``local_blocks`` of them (counted from the *unclamped* valid count) to + ``LOCAL_SCORE`` and the first ``init_blocks`` to ``INIT_SCORE`` -- local wins + where both apply -- and selects the 16 largest, ties to the smaller block. + A pick whose forced score is -inf is emitted as -1. The ids are written + ascending with a -1 tail. + + Same contract as ``torch.ops.trtllm.minimax_m3_select_blocks`` (the in-tree + ``select_blocks_from_maxscore``) and bit-identical to it, except on a row + where more than 1024 valid blocks tie at the 16th-largest score: this op + breaks that tie to the smallest blocks as the contract says, the in-tree op + does not. The test checks both against ``reference``. Returns ``out``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="32K-context engine (257 worst-case k-tiles), Eagle3 verify at one request: 4 query rows", + spec=dict(num_blocks=257, total_q=4, valid="near_full", scores="random", **_M3), + ), + Cell( + why="decode without speculation at batch 8 on a 2048-block engine: the 2-key-per-thread path at its edge", + spec=dict(num_blocks=2048, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="1M-context engine at batch 8 x qlen 4: rows on all three key widths, short rows and padding rows", + spec=dict(num_blocks=8193, total_q=32, valid="mixed", scores="random", **_M3), + ), + Cell( + why="duplicated scores: ties broken to the smaller block, and > 1024 keys at the bound (re-read path)", + spec=dict(num_blocks=4500, total_q=4, valid="near_full", scores="ties", **_M3), + ), + Cell( + why="non-finite scores: +inf, NaN ranked above +inf, and rows whose picks run into -inf (emitted -1)", + spec=dict(num_blocks=600, total_q=8, valid="mixed", scores="nonfinite", **_M3), + ), + ) + + note = """ + Certified at one KV head per rank (TP4) with M3's forcing (`init_blocks=0`, + `local_blocks=1`). The kernel takes any head count and forcing; neither is + driven here. + + `scores` may be strided (the op reads it by stride); the target hands over + the scorer's contiguous view, which is what the cells drive. Rows of at + most 128 blocks are within the kernel's range, but the target keeps those on + the in-tree op, so they are not certified here. + + Ordering is by the descending-score key `orderedKey`: a positive NaN ranks + above +inf, +0 above -0, and a negative NaN below -inf -- which, unlike + -inf, is emitted as a block id rather than -1. The cells drive positive + NaN only. + """ + + def __call__( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_sm107_tp4_select_long( + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + fast_tail, + ) + return out + + def reference( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> torch.Tensor: + """The [total_q, num_kv_heads, 16] int32 table the op is supposed to write. + + Exact integer ranking, no floating point beyond the forcing: each + score's bit pattern maps to an unsigned key that is smaller for a larger + score (NaN above +inf), packed with the block id so one sort ranks by + (score descending, block ascending). ``out``, ``use_pdl``, ``early_trigger`` and + ``trigger_phase`` and ``fast_tail`` only say where and how the op writes; they do not enter the result. + """ + num_kv_heads, num_blocks, total_q = scores.shape + device = scores.device + blocks = torch.arange(num_blocks, device=device, dtype=torch.int64) + raw = n_valid_blocks[:total_q].to(torch.int64) + valid = raw.clamp(0, num_blocks) + local_start = (raw - local_blocks).clamp_min(0) + result = torch.full((total_q, num_kv_heads, TOPK), -1, dtype=torch.int32, device=device) + for head in range(num_kv_heads): + forced = scores[head].float().t().clone() # [total_q, num_blocks] + forced = torch.where(blocks[None, :] < init_blocks, INIT_SCORE, forced) + forced = torch.where(blocks[None, :] >= local_start[:, None], LOCAL_SCORE, forced) + bits = forced.view(torch.int32).to(torch.int64) & 0xFFFFFFFF + key = torch.where(bits >= 0x80000000, bits, (~bits) & 0x7FFFFFFF) + in_row = blocks[None, :] < valid[:, None] + packed = torch.where( + in_row, (key << 16) | blocks[None, :], torch.iinfo(torch.int64).max + ) + k = min(TOPK, num_blocks) + picked = torch.topk(packed, k, dim=1, largest=False).values + picked_key = picked >> 16 + keep = (picked != torch.iinfo(torch.int64).max) & (picked_key != 0xFF800000) + ids = torch.where(keep, picked & 0xFFFF, torch.iinfo(torch.int64).max) + ids = ids.sort(dim=1).values + ids = torch.where(ids == torch.iinfo(torch.int64).max, -1, ids) + result[:, head, :k] = ids.to(torch.int32) + return result + + def is_valid( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> None: + """The inputs this op takes and answers wrongly. + + Negative forcing counts are accepted and push the forced ranges past the + row instead of failing; the in-tree op this one stands in for refuses + them, so a caller relying on this one would silently diverge from it. + An ``out`` overlapping ``scores`` is overwritten while it is still read. + """ + assert init_blocks >= 0 and local_blocks >= 0, ( + f"init_blocks ({init_blocks}) and local_blocks ({local_blocks}) must be non-negative; the " + "op accepts negative counts and selects as if the forced ranges were elsewhere" + ) + storage = scores.untyped_storage() + assert not (storage.data_ptr() <= out.data_ptr() < storage.data_ptr() + storage.nbytes()), ( + "out must not alias scores: the op writes it while scores are still read" + ) + + +m3_select_long = M3SelectLong() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_long_dsl.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_long_dsl.py new file mode 100644 index 000000000000..9f79f4b69517 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_select_long_dsl.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 MSA block selection for decode rows of up to 9216 blocks, one row per 384-thread CTA (CuTe DSL).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_select_long import _M3, MAX_BLOCKS, M3SelectLong + + +class M3SelectLongDsl(OpWrapper): + """``m3_select_long``'s contract, cells and reference, run by ``kernels/m3_select_long_dsl``: the same selection + bit for bit as the 1024-thread ``minimax_m3_sm107_tp4_select_long`` (also on rows where more than 2048 blocks tie + at the 16th-largest score), from one 384-thread CTA per row. ``fast_tail`` (the 1024-thread kernel's warp-0 rank) + does not apply and is ignored. Returns ``out``.""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="32K-context engine (257 worst-case k-tiles), Eagle3 verify at one request: 4 query rows", + spec=dict(num_blocks=257, total_q=4, valid="near_full", scores="random", **_M3), + ), + Cell( + why="decode without speculation at batch 8 on a 2048-block engine: the 2-key-per-thread path at its edge", + spec=dict(num_blocks=2048, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="1M-context engine at batch 8 x qlen 4: rows on all three key widths, short rows and padding rows", + spec=dict(num_blocks=8193, total_q=32, valid="mixed", scores="random", **_M3), + ), + Cell( + why="duplicated scores: ties broken to the smaller block, and > 1024 keys at the bound (re-read path)", + spec=dict(num_blocks=4500, total_q=4, valid="near_full", scores="ties", **_M3), + ), + Cell( + why="non-finite scores: +inf, NaN ranked above +inf, and rows whose picks run into -inf (emitted -1)", + spec=dict(num_blocks=600, total_q=8, valid="mixed", scores="nonfinite", **_M3), + ), + ) + + def reference( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> torch.Tensor: + """``m3_select_long``'s reference: the selection is the same bit for bit.""" + return M3SelectLong.reference( + self, + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + fast_tail, + ) + + def is_valid( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> None: + """``m3_select_long``'s domain.""" + M3SelectLong.is_valid( + self, + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + fast_tail, + ) + + note = """ + Certified at one KV head per rank (TP4, Rubin) with M3's forcing + (`init_blocks=0`, `local_blocks=1`), and bit for bit against the + 1024-thread `m3_select_long` on every cell. The kernel takes any head count + and forcing; other forcing is not driven here. + + The kernel is compiled per (PDL launch, early trigger, trigger phase) with + symbolic shapes and strides; the target compiles its keys before warmup + (`kernels/m3_select_long_dsl.prebuild`), and a call outside them compiles + on first use. + """ + + def __call__( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + fast_tail: bool = True, + ) -> torch.Tensor: + from ...kernels.m3_select_long_dsl import select_long_dsl + + if scores.shape[1] > MAX_BLOCKS: + raise RuntimeError( + f"m3_select_long_dsl supports at most {MAX_BLOCKS} blocks, got {scores.shape[1]}" + ) + select_long_dsl( + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + ) + return out + + +m3_select_long_dsl = M3SelectLongDsl() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_sparse_decode.py new file mode 100644 index 000000000000..04d76f241b29 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_sparse_decode.py @@ -0,0 +1,215 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""This target's block-sparse decode attention (Triton, with the one-launch long-context route).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.triton_sparse_decode import minimax_m3_sparse_attn_decode +from .m3_sparse_decode_lc import PAGE, sparse_attention_reference + + +class M3SparseDecode(OpWrapper): + """``minimax_m3_sparse_attn_decode`` as ``M3Attention._decode_sparse_core`` calls it. + + Attention of each query token over the blocks ``topk_idx`` selected for it + (``[num_kv_heads, T, topk]``, ascending ids with a -1 tail), each block + truncated at the token's causal extent, written into ``output`` in place. + The target passes FP8 q, the packed NVFP4 sparse-layer cache with its block + and per-tensor scales, and BF16 ``output`` [T, 16, 128]. Returns ``output``. + + Which kernel runs is decided inside, per step: a selected (non-identity) + table on the TP4 geometry goes to the one-launch ``m3_sparse_decode_lc`` + kernel; an identity table (every request within 16 blocks, staged by the + metadata) runs the Triton producer, or from 17 query rows the multi-query + producer, then the LSE merge. Every route also publishes the MXFP8 o_proj + hand-off to ``m3_prequant.SLOT``; this entry certifies the attention output. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="identity selection, decode without speculation at batch 8: the Triton producer + merge", + spec=dict( + seq_lens=(1, 128, 129, 700, 2048, 1500, 64, 0), decode_query_len=1, identity=True + ), + ), + Cell( + why="identity selection, Eagle3 verify at one request (T=4): one block per producer program", + spec=dict(seq_lens=(1900,), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 4 (T=16): the last row count below the MQ producer", + spec=dict(seq_lens=(300, 2048, 4, 1000), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 8 (T=32): the multi-query producer, padding rows", + spec=dict( + seq_lens=(2048, 130, 5, 0, 1777, 900, 2000, 0), decode_query_len=4, identity=True + ), + ), + Cell( + why="selected blocks, Eagle3 verify at batch 2: routed to the one-launch long-context kernel", + spec=dict(seq_lens=(9000, 40000), decode_query_len=4, identity=False), + ), + Cell( + why="selected blocks, decode without speculation at batch 8: the long-context route at T=8", + spec=dict( + seq_lens=(3000, 2100, 9999, 2048, 2049, 4500, 16000, 2200), + decode_query_len=1, + identity=False, + ), + ), + ) + + note = """ + Certified only as the target calls it: one KV head of 16 query heads, + head_dim 128, FP8 q, packed NVFP4 cache, BF16 output, `num_topk_chunks` + left to the dispatcher. The function also takes BF16 / FP8 caches, other + head geometries and an explicit split-K factor (the in-tree tests drive + those); none is certified here. + + The MXFP8 hand-off each route publishes is certified on the long-context + kernel's own entry (`m3_sparse_decode_lc`); the Triton merge writes it with + the same recipe (`_store_mxfp8_head`). + """ + + def __call__( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + minimax_m3_sparse_attn_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + kv_scale=kv_scale, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + num_topk_chunks=num_topk_chunks, + identity_blocks=identity_blocks, + ) + return output + + def reference( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + """``output`` as the call is supposed to write it: float64 block-sparse attention over the + dequantized NVFP4 blocks (``sparse_attention_reference``), rounded once to ``output``'s dtype. + + Only the packed NVFP4 cache with one KV head is mirrored -- the target's + only configuration. ``kv_scale``, ``num_topk_chunks`` and + ``identity_blocks`` do not change what is computed. + """ + assert k_block_scale is not None and k_paged.shape[1] == 1, ( + "mirrors the one-KV-head NVFP4 cache only" + ) + attn = sparse_attention_reference( + q, + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx[0], + block_table, + seq_lens, + decode_query_len, + sm_scale, + ) + return attn.to(output.dtype) + + def is_valid( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> None: + """The inputs this call takes and answers wrongly. + + Every producer walks a token's first ``min(topk, ceil(kv_len / 128))`` + table entries and indexes the block table with them unchecked, so they + must be real blocks of the request. ``identity_blocks`` is a promise the + multi-query producer relies on: every token of a request reads the same + blocks, all of them. Reads ``topk_idx`` and ``seq_lens`` back to the + host: a test-only cost. + """ + total_q = q.shape[0] + seq = seq_lens.tolist() + max_topk = topk_idx.shape[-1] + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk_idx[:, tok, : min(max_topk, nblocks)] + assert bool(((ids >= 0) & (ids < min(nblocks, block_table.shape[1]))).all()), ( + f"token {tok}: the first min(topk, block count) selected ids must be blocks of the request; " + "the block table is indexed with them unchecked" + ) + if identity_blocks: + assert nblocks <= max_topk and bool( + (ids == torch.arange(nblocks, device=ids.device, dtype=ids.dtype)).all() + ), ( + f"token {tok}: identity_blocks promises the table holds every block of the request in order" + ) + + +m3_sparse_decode = M3SparseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_sparse_decode_lc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_sparse_decode_lc.py new file mode 100644 index 000000000000..76fc68fa5903 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/m3_sparse_decode_lc.py @@ -0,0 +1,436 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-launch long-context sparse decode over the selected NVFP4 blocks (16-CTA clusters). + +Also the plain-torch block-sparse attention and MXFP8 hand-off mirrors the +Triton sparse decode entry (``m3_sparse_decode``) reuses: both compute the same +thing over the same cache. +""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The TP4 geometry the kernel is compiled for. +NUM_HEADS = 16 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 +#: NVFP4 elements per E4M3 block scale. +SF_VEC = 16 +#: MXFP8 elements per UE8M0 scale, and the o_proj width the hand-off is laid out for. +MX_BLOCK = 32 +WIDTH = NUM_HEADS * HEAD_DIM + +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) + + +def dequant_nvfp4_pages( + packed: torch.Tensor, block_scale: torch.Tensor, pages: torch.Tensor, quad_scales: bool +) -> torch.Tensor: + """Dequantize KV head 0 of the given pages of a packed NVFP4 cache to float64 [n, PAGE, HEAD_DIM]. + + ``packed`` is the [pages, kv_heads, PAGE, HEAD_DIM // 2] byte view (low + nibble = even element), ``block_scale`` the [pages, kv_heads, PAGE, + HEAD_DIM // 16] E4M3 bytes ``fused_write_layer_caches_nvfp4`` writes: token + ``t``'s column ``c`` at byte ``t * 8 + c`` of the page's region for K, and at + ``(t // 4) * 32 + c * 4 + t % 4`` for V (``quad_scales``). The per-tensor + global scale is not applied. Every product is exact in float64. + """ + table = torch.tensor(_E2M1, dtype=torch.float64, device=packed.device) + rows = packed[pages, 0].to(torch.int64) # [n, PAGE, HEAD_DIM // 2] + nibbles = torch.stack([rows & 0x0F, rows >> 4], dim=-1).reshape(rows.shape[0], PAGE, HEAD_DIM) + cols = HEAD_DIM // SF_VEC + region = block_scale[pages, 0].reshape(rows.shape[0], PAGE * cols) + t = torch.arange(PAGE, device=packed.device)[:, None] + c = torch.arange(cols, device=packed.device)[None, :] + index = (t // 4) * 32 + c * 4 + t % 4 if quad_scales else t * cols + c + scales = region[:, index.reshape(-1)].reshape(rows.shape[0], PAGE, cols) + scales = scales.view(torch.float8_e4m3fn).to(torch.float64) + return table[nibbles] * scales.repeat_interleave(SF_VEC, dim=-1) + + +def sparse_attention_reference( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, +) -> torch.Tensor: + """Block-sparse decode attention over one KV head of a packed NVFP4 cache, float64 [T, heads, HEAD_DIM]. + + Query token ``tok`` belongs to request ``tok // decode_query_len`` at offset + ``tok % decode_query_len`` and attends the first ``min(16, ceil(kv_len / + PAGE))`` entries of ``topk[tok]`` that are not -1, each block truncated at + ``kv_len = max(seq_len - decode_query_len + offset + 1, 0)``. A token that + attends nothing (a CUDA-graph padding row) is all zeros. ``q`` is E4M3 (or + its byte view); softmax and both products are float64. + """ + q64 = ( + q.view(torch.float8_e4m3fn).to(torch.float64) + if q.element_size() == 1 + else q.to(torch.float64) + ) + total_q, num_heads, _ = q64.shape + kgs = float(k_global_scale.reshape(-1)[0]) + vgs = float(v_global_scale.reshape(-1)[0]) + out = torch.zeros(total_q, num_heads, HEAD_DIM, dtype=torch.float64, device=q.device) + seq = seq_lens.tolist() + positions = torch.arange(PAGE, device=q.device) + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + real_topk = min(TOPK, (kv_len + PAGE - 1) // PAGE) + blocks = topk[tok, :real_topk] + blocks = blocks[blocks >= 0].to(torch.int64) + if blocks.numel() == 0: + continue + pages = block_table[req, blocks].to(torch.int64) + keys = ( + dequant_nvfp4_pages(k, k_block_scale, pages, quad_scales=False).reshape(-1, HEAD_DIM) + * kgs + ) + values = ( + dequant_nvfp4_pages(v, v_block_scale, pages, quad_scales=True).reshape(-1, HEAD_DIM) + * vgs + ) + valid = ((blocks[:, None] * PAGE + positions[None, :]) < kv_len).reshape(-1) + logits = (q64[tok] @ keys.T) * sm_scale + logits = logits.masked_fill(~valid[None, :], float("-inf")) + out[tok] = torch.softmax(logits, dim=-1) @ values + return out + + +def mxfp8_handoff_reference( + out: torch.Tensor, num_rows_alloc: int +) -> Tuple[torch.Tensor, torch.Tensor]: + """The MXFP8 o_proj hand-off the merge writes for BF16 rows ``out`` [T, WIDTH]: ``(q8 [T, WIDTH] uint8, + sf [num_rows_alloc * WIDTH // 32] uint8)``. + + Per 32-element block: ``e`` is the biased exponent of ``amax * rcp(448)`` + rounded up when its mantissa is non-zero (capped at 254), the elements are + scaled by ``2^(127 - e)`` and rounded to E4M3 (saturating), and ``e`` lands + in the 128x4-swizzled scale layout the MXFP8 GEMM reads. Entries of ``sf`` + for rows past ``T`` are zero. Computed from the op's own BF16 rows, so a + comparison against it is exact. + """ + total_q = out.shape[0] + x = out.float().reshape(total_q, WIDTH // MX_BLOCK, MX_BLOCK) + amax = x.abs().amax(dim=-1) + sfv = amax * torch.tensor(1.0 / 448.0, dtype=torch.float32, device=out.device) + bits = sfv.view(torch.int32) + e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + e = e.clamp(max=254) + qs = torch.where(amax != 0, ((254 - e) << 23).view(torch.float32), torch.zeros_like(amax)) + xq = (x * qs[..., None]).clamp(-448.0, 448.0) + q8 = xq.to(torch.float8_e4m3fn).view(torch.uint8).reshape(total_q, WIDTH) + num_sf_k_tiles = WIDTH // 128 + sf = torch.zeros(num_rows_alloc * (WIDTH // MX_BLOCK), dtype=torch.uint8, device=out.device) + tok = torch.arange(total_q, device=out.device)[:, None] + k_sf = torch.arange(WIDTH // MX_BLOCK, device=out.device)[None, :] + offset = ( + (tok // 128) * num_sf_k_tiles * 512 + + (k_sf // 4) * 512 + + (tok % 32) * 16 + + ((tok % 128) // 32) * 4 + + k_sf % 4 + ) + sf[offset.reshape(-1)] = e.to(torch.uint8).reshape(-1) + return q8, sf + + +class M3SparseDecodeLc(OpWrapper): + """Sparse decode attention, split-K merge and MXFP8 hand-off of one TP4 decode step in one launch. + + One cluster of 16 CTAs per query token: CTA ``r`` attends the token's + ``r``-th selected block (``topk[tok, r]``), and the cluster merges the 16 + partials over distributed shared memory into the BF16 row ``out[tok]``. With + ``q8`` / ``sf`` it also writes the row's MXFP8 copy and swizzled UE8M0 + scales for the o_proj GEMM. All operands are the byte views + ``m3_attn_lc.m3_lc_attn`` hands over (E4M3 ``q``, packed E2M1 pages, E4M3 + block scales). Returns ``out``. + + Per block: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16 + (exact), QK and PV on FP16 tensor cores with FP32 accumulation, the softmax + probabilities rounded to FP16 for PV, log2-domain softmax with ``ex2.approx``. + ``linear_softmax`` picks the denominator recurrence (the two agree to + rounding); the target picks it per batch shape. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode without speculation, one request: T=1, log-domain recurrence, MXFP8 hand-off", + spec=dict(seq_lens=(9000,), decode_query_len=1, linear_softmax=False, mxfp8=True), + ), + Cell( + why="Eagle3 verify at one request: T=4, the log-domain recurrence the target keeps for local batch 1", + spec=dict(seq_lens=(20000,), decode_query_len=4, linear_softmax=False, mxfp8=True), + ), + Cell( + why="Eagle3 verify at two requests: T=8 on the linear denominator", + spec=dict(seq_lens=(5000, 12345), decode_query_len=4, linear_softmax=True, mxfp8=True), + ), + Cell( + why="Eagle3 verify at batch 8 (T=32): short requests with -1 tails and zero-length padding rows", + spec=dict( + seq_lens=(16384, 2049, 130, 0, 7777, 4, 30000, 0), + decode_query_len=4, + linear_softmax=True, + mxfp8=True, + ), + ), + Cell( + why="decode without speculation at batch 8: T=8 rows of distinct requests", + spec=dict( + seq_lens=(3000, 2100, 9999, 2048, 2049, 4500, 16000, 2200), + decode_query_len=1, + linear_softmax=False, + mxfp8=True, + ), + ), + Cell( + why="no hand-off buffers (the variant the call site takes past 128 rows), driven at T=8", + spec=dict(seq_lens=(5000, 12345), decode_query_len=4, linear_softmax=True, mxfp8=False), + ), + ) + + note = """ + Certified: the four instantiated variants (`linear_softmax` x hand-off on / + off), all with `early_trigger=True` -- the op refuses `False`. The + hand-off cells are certified bit-exactly against `mxfp8_handoff_reference` + applied to the op's own BF16 rows; the BF16 rows themselves within the band + the test derives. + + `topk` is read for the first `min(16, ceil(kv_len / 128))` entries of each + token; a -1 among them skips that CTA's block, entries past them are never + read. `use_pdl` only changes the launch attribute. The hand-off assumes + `num_sf_k_tiles = 16` (a 2048-wide o_proj input) and buffers with at least + as many rows as the step; `MERGE_MXFP8` sizes them at 128 rows. + """ + + def __call__( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + early_trigger: bool, + merge_once: bool = False, + prefetch: Optional[tuple] = None, + next_kv: Optional[tuple] = None, + identity: bool = False, + identity_step: bool = False, + one_div: bool = False, + bt_prewait: bool = False, + ) -> torch.Tensor: + # merge_once: this target's copy of the op (minimaxM3Sm107Tp4/sparseDecodeLc4.cu), whose split-K merge forms the + # 16 weights once per CTA (same values, same order: bitwise the image op's output). + lc = ( + torch.ops.trtllm.minimax_m3_sm107_tp4_sparse_decode_lc + if merge_once + else torch.ops.trtllm.minimax_m3_sparse_decode_lc + ) + # prefetch: up to two read-only tensors the TP4 op pulls into L2 (a hint; the image op takes none). + # next_kv: (this layer's id buffer, the next sparse layer's, its K / V pools and block scales), TP4 op only. + extra = () + if merge_once and (prefetch or next_kv): + extra = tuple(prefetch or ()) + (None,) * (2 - len(prefetch or ())) + if next_kv: + extra += tuple(next_kv) + lc( + q, + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + out, + q8, + sf, + num_sf_k_tiles, + decode_query_len, + sm_scale, + linear_softmax, + use_pdl, + early_trigger, + *extra, + **({"identity": True} if identity else {}), + **({"identity_step": True} if identity_step else {}), + **({"one_div": True} if one_div else {}), + # bt_prewait: the TP4 op stages the block-table row in shared memory before its grid wait (same output). + **({"bt_prewait": True} if bt_prewait and merge_once else {}), + ) + return out + + def reference( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + early_trigger: bool, + merge_once: bool = False, + prefetch: Optional[tuple] = None, + next_kv: Optional[tuple] = None, + identity: bool = False, + identity_step: bool = False, + one_div: bool = False, + bt_prewait: bool = False, + ) -> torch.Tensor: + """The BF16 ``out`` [T, 2048] the op is supposed to write: ``sparse_attention_reference`` in float64, + rounded once. ``linear_softmax``, ``use_pdl``, ``early_trigger``, ``one_div`` and ``bt_prewait`` do not change + what is computed; the hand-off buffers are checked separately with ``mxfp8_handoff_reference``. + """ + attn = sparse_attention_reference( + q.view(-1, NUM_HEADS, HEAD_DIM), + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + decode_query_len, + sm_scale, + ) + return attn.reshape(out.shape).to(out.dtype) + + def is_valid( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + early_trigger: bool, + merge_once: bool = False, + prefetch: Optional[tuple] = None, + next_kv: Optional[tuple] = None, + identity: bool = False, + identity_step: bool = False, + one_div: bool = False, + bt_prewait: bool = False, + ) -> None: + """The inputs this op takes and answers wrongly (``m3_attn_lc.lc_applies`` is the target's form of the + layout half). + + The kernel addresses a page's token rows at a fixed 64-byte (data) and + 8-byte (scale) pitch and loads them 16 / 8 bytes at a time, so only the + page stride is read from the views. It reads the selected block ids, + the block table and the hand-off buffers unchecked. Reads ``topk`` and + ``seq_lens`` back to the host: a test-only cost. + """ + for name, t in (("k", k), ("v", v)): + assert t.dim() == 4 and t.shape[1] == 1 and t.shape[2:] == (PAGE, HEAD_DIM // 2), ( + f"{name} must be the [pages, 1, 128, 64] byte view of one KV head" + ) + assert t.stride(3) == 1 and t.stride(2) == HEAD_DIM // 2, ( + f"{name} rows must be dense 64-byte rows: the kernel ignores the row stride" + ) + assert t.data_ptr() % 16 == 0 and t.stride(0) % 16 == 0, ( + f"{name} pages must be 16-byte aligned" + ) + for name, t in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + assert t.dim() == 4 and t.stride(3) == 1 and t.stride(2) == HEAD_DIM // SF_VEC, ( + f"{name} rows must be dense 8-byte rows: the kernel ignores the row stride" + ) + assert t.data_ptr() % 8 == 0 and t.stride(0) % 8 == 0, ( + f"{name} pages must be 8-byte aligned" + ) + assert q.data_ptr() % 16 == 0 and q.stride(0) % 16 == 0 and q.stride(1) % 16 == 0, ( + "q rows are loaded 16 bytes at a time and must be 16-byte aligned" + ) + assert k_global_scale.numel() == 1 and v_global_scale.numel() == 1, ( + "global scales are single values" + ) + total_q = q.shape[0] + assert ( + total_q % decode_query_len == 0 and seq_lens.numel() >= total_q // decode_query_len + ), ( + "every query token must belong to a request: seq_lens is read at tok // decode_query_len unchecked" + ) + seq = seq_lens.tolist() + num_cols = block_table.shape[1] + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk[tok, : min(TOPK, nblocks)] + assert bool(((ids >= -1) & (ids < min(nblocks, num_cols))).all()), ( + f"token {tok}: selected block ids must be -1 or below the request's block count; the block " + "table is indexed with them unchecked" + ) + if q8 is not None: + assert num_sf_k_tiles == WIDTH // 128, ( + f"num_sf_k_tiles must be {WIDTH // 128}: the hand-off rows are {WIDTH} wide and a smaller " + "value overlaps them silently" + ) + assert ( + q8.numel() >= total_q * WIDTH + and sf is not None + and sf.numel() >= ((total_q + 127) // 128 * 128 * (WIDTH // MX_BLOCK)) + ), "the hand-off buffers must hold every row of the step: they are written unchecked" + + +m3_sparse_decode_lc = M3SparseDecodeLc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/msa_kv_update.py new file mode 100644 index 000000000000..43166b472eca --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/attention/msa_kv_update.py @@ -0,0 +1,200 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MSA KV-length correction in one launch (this target's ``MsaKvUpdate`` hook).""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The MSA pool's page: one sparse block is one 128-token KV page. +TOKENS_PER_BLOCK = 128 + + +class MsaKvUpdate(OpWrapper): + """Re-derive the MSA decode state after the overlap scheduler moved ``kv_lens``. + + ``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` calls this with its + own buffers at the head of every captured speculative decode step. One + single-CTA launch writes, for ``batch`` requests and ``total_q`` query + tokens (all int32): + + * ``msa_seq_lens[b] = min(kv_lens[b], kv_lens_staged[b])`` and, when given, + ``cu_kv_lens`` as its exclusive prefix sum (``batch + 1`` entries); + * ``out_cache_loc[t]``, the KV write slot of the position token ``t`` + attends up to, through ``block_table``; + * ``n_valid_blocks[t]``, when given, the block count top-k selection is + bounded by. + + It returns None; the cells read the four buffers. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode without speculation at one request: qo = 1, every buffer written", + spec=dict(batch=1, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="decode without speculation at the captured batch 8", + spec=dict(batch=8, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="Eagle3 verify (draft 3): qo = 4 per request, partial acceptance shortens kv_lens below staged", + spec=dict(batch=8, query_len=4, accept="partial", cu_kv=True, n_valid=True), + ), + Cell( + why="the metadata without cu_kv_lens (no extend_ctx promotion): the optional prefix sum is skipped", + spec=dict(batch=4, query_len=4, accept="partial", cu_kv=False, n_valid=True), + ), + Cell( + why="CUDA-graph padding rows (kv_lens 0, pos -1) and no n_valid_blocks buffer", + spec=dict(batch=8, query_len=4, accept="padded", cu_kv=True, n_valid=False), + ), + Cell( + why="positions crossing a page boundary: the slot moves to the previous page of the request", + spec=dict(batch=3, query_len=4, accept="page_edge", cu_kv=True, n_valid=True), + ), + ) + + note = """ + The kernel covers any batch up to the block size's loop (it strides over + 256 threads), and any page size; only `tokens_per_block = 128`, the MSA + pool's page, is certified. + + The op does not bounds-check the request index a token reads + (`q_batch_row`) or the block-table column it derives: both come from the + metadata's own staging, which keeps them in range by construction, and an + out-of-range value reads whatever memory sits there. `is_valid` states both. + """ + + def __call__( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + torch.ops.trtllm.minimax_m3_msa_kv_update( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + + def reference( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> Tuple[torch.Tensor, Optional[torch.Tensor], torch.Tensor, Optional[torch.Tensor]]: + """``(msa_seq_lens[:batch], cu_kv_lens[:batch + 1], out_cache_loc[:total_q], + n_valid_blocks[:total_q])`` as the op is supposed to write them. + + The in-tree metadata's own torch re-derivation + (``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` past its hook + call), statement for statement, returning new tensors rather than + writing the buffers. The buffer arguments only say which outputs exist: + a ``None`` buffer gives a ``None`` result. + """ + page = int(tokens_per_block) + kv_true = torch.minimum(kv_lens[:batch], kv_lens_staged[:batch]) + cu_kv = None + if cu_kv_lens is not None: + cu_kv = torch.zeros(batch + 1, dtype=torch.int32, device=kv_lens.device) + cu_kv[1:] = torch.cumsum(kv_true, 0) + qbr = q_batch_row[:total_q].to(torch.long) + qo_dev = seq_lens[:batch] + pos = kv_true[qbr] - qo_dev[qbr] + q_intra[:total_q] + pos_slot = pos.clamp_min(0) + block_col = torch.div(pos_slot, page, rounding_mode="floor").clamp( + max=int(block_table.shape[1]) - 1 + ) + slots = block_table[qbr, block_col.to(torch.long)] * page + torch.remainder(pos_slot, page) + n_valid = None + if n_valid_blocks is not None: + n_valid = torch.div( + (pos + 1).clamp_min(0) + (page - 1), page, rounding_mode="floor" + ).to(torch.int32) + return kv_true.to(torch.int32), cu_kv, slots.to(torch.int32), n_valid + + def is_valid( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + """The inputs this op takes and answers wrongly. + + Every operand but the block table is read as a dense run from its first + element, so a strided view reads the wrong elements. The request index of + each token and the table row it selects are not bounds-checked. Reads + ``q_batch_row`` back to the host: a test-only cost. + """ + for name, t in ( + ("kv_lens", kv_lens), + ("kv_lens_staged", kv_lens_staged), + ("seq_lens", seq_lens), + ("q_batch_row", q_batch_row), + ("q_intra", q_intra), + ("msa_seq_lens", msa_seq_lens), + ("out_cache_loc", out_cache_loc), + ("cu_kv_lens", cu_kv_lens), + ("n_valid_blocks", n_valid_blocks), + ): + assert t is None or t.is_contiguous(), ( + f"{name} must be contiguous: the kernel indexes it densely from data_ptr and ignores strides" + ) + rows = q_batch_row[:total_q] + assert total_q == 0 or ( + int(rows.min()) >= 0 and int(rows.max()) < min(batch, block_table.shape[0]) + ), ( + "q_batch_row must index a request below batch (and a block-table row): the op reads the " + "request's lengths and table row unchecked" + ) + + +msa_kv_update = MsaKvUpdate() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/__init__.py new file mode 100644 index 000000000000..5da86fe24bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collective-communication entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_allreduce.py new file mode 100644 index 000000000000..d67b040e8bfd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_allreduce.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The built-in TP all-reduce, fused with the residual add and the next RMSNorm.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.distributed import AllReduce, AllReduceParams + + +class M3AllReduce(OpWrapper): + """``comm(x, all_reduce_params=params)``: the runtime's ``AllReduce`` over the TP group. ``params`` None: the + plain sum over the ranks (the drafter's o_proj and down projection). With ``RESIDUAL_RMS_NORM`` params: + ``updated = residual + sum_ranks(x)``, ``normed = rmsnorm(updated) * norm_weight``; + returns (normed, updated). The strategy (``AllReduceStrategy.AUTO``: one-shot / two-shot Lamport, NCCL) is the + communicator's per call. + + The layer boundaries the multicast all-reduce does not take run it: prefill and mixed steps, and generation steps + beyond the multicast kernel's rows. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="tp4, a prefill chunk", spec=dict(world=4, m=2048, hidden=6144)), + Cell(why="tp4, a 64-row generation step", spec=dict(world=4, m=64, hidden=6144)), + Cell( + why="tp4, plain sum of eight drafter rows", + spec=dict(world=4, m=8, hidden=6144, plain=True), + ), + ) + + note = """ + The reduction order across ranks is the strategy's; `reference` sums the ranks' rows in fp32 in rank order, then + adds the residual and normalizes (one bf16 rounding of the sum and one of the norm). A strategy may round the sum + to bf16 before the residual add, so the residual sum agrees within one bf16 ulp of its operands' magnitude. + """ + + def __call__(self, x: torch.Tensor, comm: AllReduce, params: Optional[AllReduceParams]): + return comm(x, all_reduce_params=params) + + def reference(self, xs, comm: AllReduce, params: Optional[AllReduceParams]): + """``xs``: every rank's ``x`` in rank order (a collective's reference takes the group's inputs).""" + total = torch.stack([x.float() for x in xs]).sum(0) + if params is None: + return total.to(xs[0].dtype) + updated = (params.residual.float() + total).to(xs[0].dtype) + h = updated.float() + normed = ( + h + * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + params.eps) + * params.norm_weight.float() + ) + return normed.to(xs[0].dtype), updated + + +m3_allreduce = M3AllReduce() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_mc_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_mc_allreduce_norm.py new file mode 100644 index 000000000000..208cb25aa72e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_mc_allreduce_norm.py @@ -0,0 +1,329 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode layer boundary, 4 ranks.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +_E4M3_MAX = 448.0 + + +def mxfp8_rows(normed: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """MXFP8 of BF16 rows as ``trtllm::mxfp8_quantize(normed, True)``: E4M3 [M, H] and 128x4-swizzled UE8M0 bytes. + + One scale per 32 columns, the smallest power of two at or above ``amax / 448`` (biased exponent, round up), + data = row / scale rounded to nearest E4M3. The scale buffer is padded to 128 rows, padding bytes left as the + op leaves them (the comparison masks them out). + """ + m, h = normed.shape + blocks = normed.float().view(m, h // 32, 32) + amax = blocks.abs().amax(-1) + bits = (amax / _E4M3_MAX).view(torch.int32) + byte = (((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) > 0).to(torch.int32)).clamp(0, 254) + scale = torch.exp2(byte.float() - 127.0).unsqueeze(-1) + data = torch.where(amax.unsqueeze(-1) != 0, blocks / scale, torch.zeros_like(blocks)) + data = data.view(m, h).to(torch.float8_e4m3fn) + return data, swizzle_sf(byte.to(torch.uint8)) + + +def swizzle_sf(byte: torch.Tensor) -> torch.Tensor: + """Scatter [rows, H / 32] scale bytes into the 128x4-swizzled buffer the op writes (zero padding).""" + rows, cols = byte.shape + padded_rows = (rows + 127) // 128 * 128 + r = torch.arange(rows, device=byte.device).view(-1, 1) + c = torch.arange(cols, device=byte.device).view(1, -1) + index = ( + (c % 4) + (c // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + (r // 128) * (128 * cols) + ) + out = torch.zeros(padded_rows * cols, dtype=torch.uint8, device=byte.device) + out[index.reshape(-1)] = byte.reshape(-1) + return out + + +def sf_written(rows: int, cols: int, device) -> torch.Tensor: + """Mask of the swizzled scale bytes the op writes for ``rows`` rows (the padding rows are left unwritten).""" + return swizzle_sf(torch.ones(rows, cols, dtype=torch.uint8, device=device)).bool() + + +def boundary( + partials: Sequence[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + drafter: bool = False, +) -> tuple[torch.Tensor, ...]: + """The boundary arithmetic once each rank's BF16 partial is known (shared with the finalize entry). + + ``drafter``: the Eagle3 drafter boundary (opts bit 16), whose stock is a plain BF16 all-reduce, then FlashInfer + fused_add_rmsnorm: the RMS statistic and the normalization take the unrounded FP32 sum bf16(rank sum) + residual; + only the stored residual is rounded to BF16. + + The stock rounding points of this target (a BF16 all-reduce, then the fused residual add + RMSNorm): ranks are + summed in FP32 in rank order and rounded to BF16, the residual added and rounded to BF16 again; the RMS statistic + and the normalization are done in float64 and rounded to BF16 once. + """ + total = torch.zeros(partials[0].shape, dtype=torch.float32, device=residual.device) + for p in partials: + total = total + p.float() + pre = total.to(torch.bfloat16).float() + residual.float() + updated = pre.to(torch.bfloat16) + u = pre.double() if drafter else updated.double() + inv = torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + eps) + normed = (u * inv * norm_w.double()).to(torch.bfloat16) + if not quant: + return normed, updated + return (normed, updated, *mxfp8_rows(normed)) + + +class M3McAllreduceNorm(OpWrapper): + """One TP group's boundary: ``updated = bf16(residual + sum_r bf16(x_r + x2_r))``, ``normed = RMSNorm(updated)``. + + Every rank pushes its partial through the multicast mapping ``ws_mc_ptr`` into every rank's exchange, then polls + its own view ``ws_uc`` (re-arming the words it read) and rotates the per-CTA call parities in ``flags``; the + workspace is the ``McAllReduceWorkspace`` bucket of ``maxt`` tokens. All ranks must make the same calls in the + same stream order. ``prepushed`` 2 means a producer already pushed ``x`` (and launched this grid after its + pushes); 3 that two producers pushed ``x`` into exchange set A and ``x2`` into set B. Returns + ``[normed, updated]`` or, with ``quant``, also the MXFP8 copy of ``normed`` (E4M3 rows, swizzled UE8M0). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The tp4 decode boundaries (hidden 6144, 4 ranks, eps 1e-6). Buckets 8 / 16 / 48 are separate workspaces; the + # 8-token bucket runs epilogue option 7 at 1, 2, 7, 8 tokens and 15 at 3..6, the others 15. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one decode token at the attention boundary, x only, bucket 8 option 7", + spec=dict(m=1, maxt=8, opts=7, x2=False, quant=False, prepushed=0), + ), + Cell( + why="MoE boundary at 4 tokens: shared + routed summed in-kernel, MXFP8 copy for the next QKV, option 15", + spec=dict(m=4, maxt=8, opts=15, x2=True, quant=True, prepushed=0), + ), + Cell( + why="a full 8-token bucket with the second addend", + spec=dict(m=8, maxt=8, opts=7, x2=True, quant=False, prepushed=0), + ), + Cell( + why="attention boundary whose rows the o_proj GEMV already pushed (prepushed 2)", + spec=dict(m=8, maxt=8, opts=7, x2=False, quant=False, prepushed=2), + ), + Cell( + why="routed engine pushed set A, shared down projection set B (prepushed 3), with the MXFP8 copy", + spec=dict(m=6, maxt=8, opts=15, x2=True, quant=True, prepushed=3), + ), + Cell( + why="12 tokens pushed by the o_proj GEMV into the 16-token bucket", + spec=dict(m=12, maxt=16, opts=15, x2=False, quant=False, prepushed=2), + ), + Cell( + why="the 9..13-token MoE kernel's sum into the 16-token bucket, MXFP8 copy", + spec=dict(m=16, maxt=16, opts=15, x2=False, quant=True, prepushed=0), + ), + Cell( + why="the 48-token bucket at a partial fill", + spec=dict(m=30, maxt=48, opts=15, x2=False, quant=False, prepushed=0), + ), + Cell( + why="the 48-token bucket full, second addend and MXFP8 copy", + spec=dict(m=48, maxt=48, opts=15, x2=True, quant=True, prepushed=0), + ), + # Every other row count of the decode boundaries (modeling.MC_ALLREDUCE): the bucket and epilogue options the + # target picks for it (kernels/m3_mc_allreduce), the second addend and the MXFP8 copy alternating. + *( + Cell( + why=f"{m} tokens: a boundary row count between the ones above", + spec=dict( + m=m, + maxt=8 if m <= 8 else 16 if m <= 16 else 48, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1] if m <= 8 else 15, + x2=m % 2 == 0, + quant=m % 3 == 0, + prepushed=0, + ), + ) + for m in range(1, 49) + if m not in (1, 4, 8, 16, 30, 48) + ), + # Every other row count pushed by the o_proj (1..16) or the 1..8-token MoE engine (with the MXFP8 copy): + # modeling.ATTN_PUSH / modeling.MOE_PUSH. + *( + Cell( + why=f"{m} tokens prepushed: a pushed row count between the ones above", + spec=dict( + m=m, + maxt=8 if m <= 8 else 16, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1] if m <= 8 else 15, + x2=False, + quant=m <= 8, + prepushed=2, + ), + ) + for m in range(1, 17) + if m not in (8, 12) + ), + # The Eagle3 drafter boundaries (modeling.DRAFT_MC_AR): option 16 (the row kernel's drafter form) on 23 / 31 + # at every draft-pass row count 1..8 (pass 0: 4 x requests, later passes: requests), x only. + *( + Cell( + why=f"Eagle3 drafter boundary at {m} rows (opts {23 if m in (1, 2, 7, 8) else 31})", + spec=dict( + m=m, + maxt=8, + opts=23 if m in (1, 2, 7, 8) else 31, + x2=False, + quant=False, + prepushed=0, + drafter=True, + ), + ) + for m in range(1, 9) + ), + # The same boundaries on the 8-CTA cluster kernel (opts bit 128, modeling.DRAFT_MC_AR_CLUSTER, launched without + # the carveout: 23 | 128), alone and fed by the drafter projection's push (prepushed 2, DRAFT_AR_PUSH). + *( + Cell( + why=f"Eagle3 drafter boundary on the cluster kernel at {m} rows, prepushed {pre}", + spec=dict( + m=m, + maxt=8, + opts=23 | 128, + x2=False, + quant=False, + prepushed=pre, + drafter=True, + ), + ) + for pre in (0, 2) + for m in range(1, 9) + ), + ) + + note = """ + The epilogue option bits (1 early residual / weight load, 2 st.async cluster exchange, 4 early parity rotation, + 8 max carveout) are bit-identical by construction; only the four (bucket, option) pairs the target selects are + built. `prepushed` 1 (pushed, grid wait kept) is accepted by the op but no caller passes it. `pdl` changes only + launch overlap and is driven off in the cells. + + `normed` is gated to one BF16 rounding step: the kernel's RMS statistic is a warp / cluster tree of FP32 sums + and its `rsqrtf` is approximate, so a row whose scaled value lands next to a BF16 midpoint can round either + way. `updated` and the MXFP8 copy of the op's own `normed` are bit-exact. + """ + + def __call__( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + cap_out: Optional[torch.Tensor] = None, + cap_w: Optional[torch.Tensor] = None, + cap_eps: float = 0.0, + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_sm107_tp4_mc_allreduce_norm( + x, + x2, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + None, + None, + maxt, + quant, + prepushed, + opts, + cap_out, + cap_w, + cap_eps, + ) + + def reference( + self, + per_rank_x: Sequence[torch.Tensor], + per_rank_x2: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + drafter: bool = False, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute (``drafter``: the opts-16 drafter boundary). + + The documented collective exception: it takes every rank's ``x`` (and ``x2``) in rank order, not the calling + rank's. Each rank's partial is ``bf16(x + x2)`` (``x`` alone without ``x2``), whatever pushed it. + """ + partials = [] + for r, x in enumerate(per_rank_x): + if per_rank_x2 is None: + partials.append(x) + else: + partials.append((x.float() + per_rank_x2[r].float()).to(torch.bfloat16)) + return boundary(partials, residual, norm_w, eps, quant, drafter) + + def is_valid( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + cap_out: Optional[torch.Tensor] = None, + cap_w: Optional[torch.Tensor] = None, + cap_eps: float = 0.0, + ) -> None: + h = residual.shape[-1] + # The op checks norm_w's element count, not its layout: a strided view is read as packed memory. + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + # The workspace is addressed as [set][parity][maxt][world][H / 2] words and never bounds-checked: a smaller + # buffer (another bucket's, or one built for fewer ranks) is polled and re-armed out of range. + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + # Flags are per (token, cluster CTA) of the bucket; the op checks the count, not that they belong to the + # workspace. Flags of another bucket hold another call history, and the parity they select is not the one the + # peers push into: the poll then waits on the other half forever or reads last call's rows. + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + + +m3_mc_allreduce_norm = M3McAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_mc_finalize_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_mc_finalize_allreduce_norm.py new file mode 100644 index 000000000000..0d457022c38f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/comm/m3_mc_finalize_allreduce_norm.py @@ -0,0 +1,197 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MoE finalize + multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode MoE boundary, 4 ranks.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_mc_allreduce_norm import boundary + + +def finalize_partial( + fc2: torch.Tensor, perm: torch.Tensor, scales: torch.Tensor, shared: Optional[torch.Tensor] +) -> torch.Tensor: + """One rank's BF16 partial: ``bf16(bf16(sum_k scale_k * fc2[perm_k]) + shared)``, the routed sum in FP32 in k + order and rounded to BF16, then the shared expert added as stock adds it (one more BF16 rounding). + + The kernel's per-k step is a fused multiply-add; it is reproduced exactly by the float64 product (exact for two + FP32 operands) plus the FP32 accumulator, rounded back to FP32. Dropped entries (perm -1) are skipped. + """ + m, topk = perm.shape + acc = torch.zeros(m, fc2.shape[1], dtype=torch.float32, device=fc2.device) + sc = scales.reshape(m, topk).float() + for k in range(topk): + rows = perm[:, k].long() + keep = (rows >= 0).unsqueeze(-1) + gathered = fc2[rows.clamp_min(0)].double() + step = (gathered * sc[:, k : k + 1].double() + acc.double()).float() + acc = torch.where(keep, step, acc) + if shared is not None: + acc = acc.to(torch.bfloat16).float() + shared.float() + return acc.to(torch.bfloat16) + + +class M3McFinalizeAllreduceNorm(OpWrapper): + """The MoE boundary with the routed experts' finalize folded in. + + Per rank the partial is ``bf16(bf16(sum_k scales[t, k] * fc2[perm[t, k]]) + shared[t])`` (``perm`` -1 = dropped, + ``scales`` fp32 or bf16), then the group's all-reduce, residual add and RMSNorm exactly as + ``m3_mc_allreduce_norm``: same workspace, flags and call-order contract. Returns ``[normed, updated]`` or, with + ``quant``, also the MXFP8 copy of ``normed``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The tp4 target takes this path from 9 tokens (top-4 of 128 experts, fp32 routing weights, the shared expert + # always present): the 16- and 48-token buckets, epilogue option 15. + CELLS: tuple[Cell, ...] = ( + Cell( + why="the smallest finalize call (9 tokens, 16-token bucket) with the next QKV's MXFP8 copy", + spec=dict(m=9, maxt=16, opts=15, topk=4, quant=True, dropped=0), + ), + Cell( + why="a full 16-token bucket with dropped expert slots", + spec=dict(m=16, maxt=16, opts=15, topk=4, quant=False, dropped=5), + ), + Cell( + why="the 48-token bucket at a partial fill", + spec=dict(m=24, maxt=48, opts=15, topk=4, quant=False, dropped=0), + ), + Cell( + why="the 48-token bucket full, MXFP8 copy", + spec=dict(m=48, maxt=48, opts=15, topk=4, quant=True, dropped=3), + ), + # Every other row count of the unfinalized MoE (modeling.MC_FINALIZE): the bucket and options the target picks. + *( + Cell( + why=f"{m} tokens: a finalize row count between the ones above", + spec=dict( + m=m, + maxt=16 if m <= 16 else 48, + opts=15, + topk=4, + quant=m % 2 == 0, + dropped=m % 4, + ), + ) + for m in range(9, 49) + if m not in (9, 16, 24, 48) + ), + ) + + note = """ + BF16 routing weights and the 8-token bucket are accepted by the op (option 7 there) but the target only enters + this path at 9 tokens and up, with the built-in MoE's fp32 weights; neither is a cell. The rest is as in + `m3_mc_allreduce_norm`'s note. + """ + + def __call__( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_sm107_tp4_mc_allreduce_norm( + fc2, + shared, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + perm, + scales, + maxt, + quant, + 0, + opts, + ) + + def reference( + self, + per_rank_fc2: Sequence[torch.Tensor], + per_rank_perm: Sequence[torch.Tensor], + per_rank_scales: Sequence[torch.Tensor], + per_rank_shared: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute, from every rank's MoE outputs in rank order (the + collective exception, as in ``m3_mc_allreduce_norm.reference``).""" + partials = [ + finalize_partial( + fc2, + per_rank_perm[r], + per_rank_scales[r], + None if per_rank_shared is None else per_rank_shared[r], + ) + for r, fc2 in enumerate(per_rank_fc2) + ] + return boundary(partials, residual, norm_w, eps, quant) + + def is_valid( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + ) -> None: + h = residual.shape[-1] + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + # The kernel gathers fc2[perm] without a bound: an index past the FC2 rows reads whatever follows them. + lo, hi = int(perm.min()), int(perm.max()) + assert lo >= -1 and hi < fc2.shape[0], ( + f"perm spans [{lo}, {hi}]; entries must be -1 (dropped) or a row of fc2 (< {fc2.shape[0]})" + ) + + +m3_mc_finalize_allreduce_norm = M3McFinalizeAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/embedding/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/embedding/__init__.py new file mode 100644 index 000000000000..e9f4d8e20b94 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/embedding/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Embedding ops of this target: the generic token-embedding gather.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/embedding/m3_embedding.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/embedding/m3_embedding.py new file mode 100644 index 000000000000..3060c17acdff --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/embedding/m3_embedding.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The generic token-embedding gather over the replicated table (an id outside the table gives a zero row).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modules.embedding import pre_comm_embedding_ops + + +class M3Embedding(OpWrapper): + """``table[ids]`` with a zero row for an id outside ``[0, V)`` (the speculative sampler pads with -1). + + The table is the full (replicated) bf16 embedding [V, 6144]; ids int32 / int64 of any count. The fused decode + embedding (``m3_embedding_firstnorm``) takes decode steps of up to 8 rows; this entry takes every other step. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="a prefill chunk, int32 ids", spec=dict(rows=2048, id_dtype=torch.int32, pad=False) + ), + Cell( + why="a 48-row decode verify, int64 ids", + spec=dict(rows=48, id_dtype=torch.int64, pad=False), + ), + Cell( + why="draft rows padded with -1 ids", spec=dict(rows=32, id_dtype=torch.int32, pad=True) + ), + ) + + note = """ + An exact gather: `reference` must match bit for bit. + """ + + def __call__(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + return pre_comm_embedding_ops(ids, table, 1, 0, None, 0, table.shape[0], False, 0) + + def reference(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] + return rows * valid.unsqueeze(-1).to(table.dtype) + + +m3_embedding = M3Embedding() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/__init__.py new file mode 100644 index 000000000000..9cc374bdf1cc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GEMM entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m31_fc_shard.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m31_fc_shard.py new file mode 100644 index 000000000000..fbc13e9adbfc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m31_fc_shard.py @@ -0,0 +1,141 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""FC_SHARD: the DSpark drafter's fc as a tensor-parallel column shard plus an all-gather, bit for bit the full fc. + +The drafter's fc ([M, layers x H] -> [M, H], bf16, no bias) is replicated on every rank, so each rank reads the whole +weight. Here each rank computes its H / tp output columns and the ranks all-gather them. Each column is computed by +``trtllm::minimax_m31_fc_shard_gemm`` with the cuBLASLt algo the full fc uses (``PINNED_CFG``), so every output column +is the same sum of the same K-split partials as in the full GEMM and the gathered result equals +``torch.nn.functional.linear(x, w)``. ``FcShard.build`` checks that on the device before the first use (real-scale and +2^+-12 block-scaled probes, every row count up to ``MAX_ROWS``) and returns None, keeping the stock fc, when the op is +not built, the shape does not apply, or any probe differs. +""" + +from __future__ import annotations + +from typing import Optional + +import torch +import torch.nn.functional as F + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.distributed import allgather +from tensorrt_llm.logger import logger + +# cuBLASLt's algo for the full fc (K = 30720 -> N = 6144, bf16 in / out, fp32 compute) in the VR image's cuBLASLt: +# ALGO_ID, TILE_ID, SPLITK_NUM, REDUCTION_SCHEME, CTA_SWIZZLING, CUSTOM_OPTION, STAGES_ID, INNER_SHAPE_ID, +# CLUSTER_SHAPE_ID. The build self-check rejects it if this cuBLASLt picks a different one for the full fc. +PINNED_CFG = (66, 393, 22, 2, 0, 1, 35, 0, 3) +MAX_ROWS = 8 # capacity: fixed: the K7 verify rows at batch 1 (c1; c2+ keeps the stock fc) + +_STATE: dict = {} +_OP = "minimax_m31_fc_shard_gemm" + + +def fc_shard_op(): + """``trtllm::minimax_m31_fc_shard_gemm`` if this build carries it, else None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, _OP, None) + if op is not None: + # The op's fake lives in the catalog's M3.1 fake table (registered once, whoever loads it first). + from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import m31_fake_ops + + m31_fake_ops.register() + _STATE["op"] = op + return _STATE["op"] + + +def _probes(rows: int, k: int, device: torch.device): + """A real-scale and a 2^+-12 block-scaled bf16 probe [rows, k] (128-column blocks), seeded.""" + g = torch.Generator(device="cpu").manual_seed(31 * rows + 7) + x = torch.randn(rows, k, generator=g) + yield x.to(torch.bfloat16).to(device) + e = torch.randint(-12, 13, (rows, (k + 127) // 128), generator=g).float() + yield (x * torch.pow(2.0, e).repeat_interleave(128, dim=1)[:, :k]).to(torch.bfloat16).to(device) + + +class FcShard: + """The fc split by output columns over the tensor-parallel group; ``__call__(x)`` = the full fc's output.""" + + def __init__(self, w: torch.Tensor, w_shard: torch.Tensor, mapping, op) -> None: + self.source = w + self.w_shard = w_shard + self.mapping = mapping + self._op = op + self._cfg = list(PINNED_CFG) + self._logged = False + + @classmethod + def build(cls, fc: torch.nn.Module, mapping) -> Optional["FcShard"]: + """This rank's shard of ``fc``, or None (the stock fc stays) with the reason logged. Every rank of the + tensor-parallel group takes the shard only if every rank's check passed, so the ranks agree on whether the fc + all-gathers (the agreement is one tiny all-gather that every rank issues, whatever its own outcome).""" + tp, rank = int(mapping.tp_size), int(mapping.tp_rank) + shard, why = cls._check(fc, tp, rank, mapping) + if tp >= 2 and torch.cuda.is_available(): + ok = torch.tensor([0 if why else 1], dtype=torch.int32, device="cuda") + if not bool(allgather(ok, mapping, dim=0).all()): + why = why or "the check failed on another rank" + if why: + logger.info(f"[M3 dispatch] FC_SHARD not taken on rank {rank}: {why}") + return None + logger.info( + f"[M3 prebuild] FC_SHARD set up on rank {rank}/{tp}: fc {tuple(shard.source.shape)} -> shard " + f"{tuple(shard.w_shard.shape)}, bitwise to the full fc at 1..{MAX_ROWS} rows" + ) + return shard + + @classmethod + def _check(cls, fc: torch.nn.Module, tp: int, rank: int, mapping): + """(shard, None) when this rank's pinned shard GEMM is bitwise to the full fc on every probe, else + (None, reason).""" + op = fc_shard_op() + if op is None: + return None, f"{_OP} is not built" + w = getattr(fc, "weight", None) + if ( + w is None + or getattr(fc, "bias", None) is not None + or w.dtype != torch.bfloat16 + or w.dim() != 2 + or not w.is_cuda + or tp < 2 + or w.shape[0] % tp != 0 + ): + return None, f"fc shape / dtype / tp do not apply (tp {tp})" + cols = w.shape[0] // tp + shard = cls(w, w[rank * cols : (rank + 1) * cols].contiguous(), mapping, op) + try: + for rows in range(1, MAX_ROWS + 1): + for x in _probes(rows, w.shape[1], w.device): + want = F.linear(x, w)[:, rank * cols : (rank + 1) * cols] + if not torch.equal(op(x, shard.w_shard, shard._cfg), want): + return ( + None, + f"the pinned shard GEMM differs from the full fc at {rows} rows", + ) + except RuntimeError as e: + return None, f"the pinned cuBLASLt algo is rejected ({e})" + torch.cuda.synchronize() + return shard, None + + def applies(self, fc: torch.nn.Module, rows: int) -> bool: + """Whether ``fc`` is the fc this shard was built from and ``rows`` is within the checked row counts.""" + return getattr(fc, "weight", None) is self.source and 0 < rows <= MAX_ROWS + + def partial(self, x: torch.Tensor) -> torch.Tensor: + """This rank's output columns [rows, H / tp] (no collective: safe on a side stream).""" + if not self._logged: + self._logged = True + logger.info( + f"[M3 dispatch] FC_SHARD taken: drafter fc as {self.w_shard.shape[0]}-column shards + all-gather on " + f"rank {self.mapping.tp_rank}" + ) + return self._op(x.contiguous(), self.w_shard, self._cfg) + + def gather(self, y: torch.Tensor) -> torch.Tensor: + """The full fc output from every rank's ``partial`` (the tensor-parallel all-gather, in rank order).""" + return allgather(y, self.mapping, dim=-1) + + def __call__(self, x: torch.Tensor) -> torch.Tensor: + return self.gather(self.partial(x)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_bf16_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_bf16_linear.py new file mode 100644 index 000000000000..4fb5af737dc0 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_bf16_linear.py @@ -0,0 +1,41 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The BF16 Eagle3 drafter's projections: ``x @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3Bf16Linear(OpWrapper): + """``F.linear(x, w)`` for bf16 ``x`` [M, K] and a bf16 weight [N, K] (cuBLAS, fp32 accumulation), bf16 out. + + The drafter's fc over the captured target layers (K = 3 x 6144), its fused QKV (K = 2 x 6144, this rank's heads), + o_proj (this rank's input columns), gate_up and down projections run it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="drafter fc, 32 verify rows", spec=dict(m=32, n=6144, k=18432)), + Cell(why="drafter QKV, eight draft rows", spec=dict(m=8, n=2304, k=12288)), + Cell(why="drafter o_proj, a prefill chunk", spec=dict(m=1024, n=6144, k=2048)), + Cell(why="drafter gate_up, one row", spec=dict(m=1, n=9216, k=6144)), + Cell(why="drafter down, eight rows", spec=dict(m=8, n=6144, k=4608)), + ) + + note = """ + cuBLAS's split and accumulation order: `reference` multiplies in fp32 (one bf16 rounding), within the test's band. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x, w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return (x.float() @ w.float().t()).to(x.dtype) + + +m3_bf16_linear = M3Bf16Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_dense_swiglu.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_dense_swiglu.py new file mode 100644 index 000000000000..1fb7a1d759ae --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_dense_swiglu.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense MLP's swigluoai activation as one C++ launch with PDL and an early release (DENSE_SWIGLU_CPP).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +# The dense layers' swigluoai parameters, which the kernel hard-codes (modeling_minimaxm3._build_swiglu_oai_dense_mlp). +LIMIT, ALPHA, BETA = 7.0, 1.702, 1.0 + + +class M3DenseSwiglu(OpWrapper): + """``min(g, 7) * sigmoid(1.702 * min(g, 7)) * (clamp(u, -7, 7) + 1)`` of a bf16 [rows, 2d] gate | up tensor, + rounded once to bf16: trtllm::minimax_m3_sm107_tp4_swiglu_oai (minimaxM3Sm107Tp4/swigluOai.cu), bit for bit the + Triton ``torch.ops.trtllm.silu_and_mul`` at swiglu_limit 7, swiglu_alpha 1.702, swiglu_beta 1. With ``use_pdl`` the + grid waits on its predecessor and releases its dependents before it computes, so the down GEMV stages its weights + during the activation (the GEMV reads the activation only after its own grid wait). Contiguous rows with an even + half width; ``__call__`` returns None otherwise or for other activation parameters (the caller runs the Triton + op).""" + + ARCHS = frozenset({Arch.SM_107}) + + # The TP4 dense MLP shard: gate | up 6144 wide (d = 3072), at the decode rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="the TP4 dense MLP at 1 row", spec=dict(rows=1, d=3072)), + Cell(why="the TP4 dense MLP at 4 rows (1 request x 4 tokens)", spec=dict(rows=4, d=3072)), + Cell(why="the TP4 dense MLP at 8 rows", spec=dict(rows=8, d=3072)), + ) + + note = """ + Elementwise, one rounding: bit for bit the Triton kernel (its instruction sequence: the clamps, (u + 1), the + alpha-scaled ex2.approx sigmoid, div.full), checked over every bf16 gate and every bf16 up pattern. + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float] = LIMIT, + alpha: Optional[float] = ALPHA, + beta: Optional[float] = BETA, + use_pdl: bool = True, + ) -> Optional[torch.Tensor]: + if ( + (limit, alpha, beta) != (LIMIT, ALPHA, BETA) + or x.dtype != torch.bfloat16 + or x.dim() != 2 + or not x.is_cuda + or not x.is_contiguous() + or x.shape[1] % 4 != 0 + or x.data_ptr() % 4 != 0 + ): + return None + out = torch.empty((x.shape[0], x.shape[1] // 2), dtype=x.dtype, device=x.device) + torch.ops.trtllm.minimax_m3_sm107_tp4_swiglu_oai(x, out, bool(use_pdl)) + return out + + def reference(self, x, limit=LIMIT, alpha=ALPHA, beta=BETA, use_pdl=True) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + + def is_valid(self, x, limit=LIMIT, alpha=ALPHA, beta=BETA, use_pdl=True) -> None: + assert x.dtype == torch.bfloat16 and x.dim() == 2, "x: bf16 [rows, 2d]" + + +m3_dense_swiglu = M3DenseSwiglu() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_draft_lowm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_draft_lowm.py new file mode 100644 index 000000000000..6fb8b1572d22 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_draft_lowm.py @@ -0,0 +1,158 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""BF16 x BF16 -> BF16 low-M GEMM of the Eagle3 drafter's linears at fixed FlashInfer tactics (SM10x).""" + +from __future__ import annotations + +from typing import Mapping, Optional, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: (N, K) of the TP4 drafter shards -> ((max rows, kind, tactic), ...), rows ascending. Kind "d" is FlashInfer's direct +#: SIMT kernel with tactic (block, outputs per block, rows per block); kind "s" is its split-K tensor-core kernel with +#: tactic (mma_m, mma_n, split_k, ab_stages). A row count is served by the first entry whose max rows holds it; the +#: rows modeling.DRAFT_LOWM covers are only the ones measured (the fastest tactic at no larger error vs fp64 than +#: cuBLAS). Rubin SM107 TP4 sweep (mclk locked, >= 3x L2 of rotating weights, CUDA-graph timing), us cuBLAS -> +#: low-M at 1 / 4 rows: qkv 8.15 -> 5.90 / 8.32 -> 6.71, o 5.36 -> 3.40 / 5.44 -> 3.66, gate_up 12.93 -> 9.93 / +#: 13.03 -> 12.05, down 7.68 -> 6.16 / 7.64 -> 6.15. The fc is replicated (the TP2 shape): its TP2 tactic and row. +TACTICS: Mapping[Tuple[int, int], tuple] = { + # qkv + (2304, 12288): ((1, "d", (256, 2, 1)), (4, "s", (64, 8, 4, 10))), + # o_proj + (6144, 2048): ((1, "d", (128, 4, 1)), (4, "s", (64, 8, 1, 10))), + # gate_up + (9216, 6144): ((1, "d", (128, 2, 1)), (4, "s", (64, 8, 1, 6))), + # down + (6144, 4608): ((4, "s", (64, 8, 2, 10)),), + # the Eagle3 fc over the three captured hidden states + (6144, 18432): ((1, "d", (384, 2, 1)),), +} +#: Rows each (N, K) is certified at (a subset of what its table would serve). +ROWS: Mapping[Tuple[int, int], Tuple[int, ...]] = { + (2304, 12288): (1, 4), + (6144, 2048): (1, 4), + (9216, 6144): (1, 4), + (6144, 4608): (1, 4), + (6144, 18432): (1,), +} + + +def tactic_for(n: int, k: int, m: int) -> Optional[Tuple[str, tuple]]: + """(kind, tactic) of an [m, k] x [n, k]^T call, or None when the table does not serve it.""" + for rows, kind, tactic in TACTICS.get((n, k), ()): + if m <= rows: + return kind, tactic + return None + + +class M3DraftLowM(OpWrapper): + """``out`` [M, N] BF16 = ``x`` [M, K] BF16 @ ``weight`` [N, K]^T BF16, FP32 accumulation. + + ``kind`` / ``tactic`` pick FlashInfer's kernel: "d" runs ``run_direct_dense`` (SIMT, one dot product per output in + K order), "s" runs ``run_splitk_dense`` (tcgen05 MMA, K split over ``split_k`` CTAs and summed in FP32). Both + round once to BF16. The kernel is compiled on its first call per (M, N, K, tactic, PDL); the target compiles every + covered call before warmup (kernels/m3_draft_lowm.prebuild). + """ + + ARCHS = frozenset({Arch.SM_107}) + CELLS: tuple[Cell, ...] = ( + Cell(why="drafter qkv, 1 rows: d tactic", spec=dict(nk=(2304, 12288), m=1)), + Cell(why="drafter qkv, 4 rows: s tactic", spec=dict(nk=(2304, 12288), m=4)), + Cell(why="drafter o_proj, 1 rows: d tactic", spec=dict(nk=(6144, 2048), m=1)), + Cell(why="drafter o_proj, 4 rows: s tactic", spec=dict(nk=(6144, 2048), m=4)), + Cell(why="drafter gate_up, 1 rows: d tactic", spec=dict(nk=(9216, 6144), m=1)), + Cell(why="drafter gate_up, 4 rows: s tactic", spec=dict(nk=(9216, 6144), m=4)), + Cell(why="drafter down, 1 rows: s tactic", spec=dict(nk=(6144, 4608), m=1)), + Cell(why="drafter down, 4 rows: s tactic", spec=dict(nk=(6144, 4608), m=4)), + Cell(why="drafter fc, 1 rows: d tactic", spec=dict(nk=(6144, 18432), m=1)), + ) + + note = """ + The table is the TP4 drafter's: its shapes are the per-rank shards of the Eagle3 layer at 4-way tensor + parallelism (32 query and 2 KV heads per rank, intermediate 9216 per rank) and the replicated fc, at the b1 + decode rows (1 row in draft passes 1-2, 4 in pass 0). Certified at those shapes and rows only; FlashInfer's + kernels take other shapes, which are not measured here. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + own_splitk: bool = False, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + if kind == "d": + from flashinfer.gemm.kernels.dense_bf16_gemm_direct import ( + DirectTactic, + run_direct_dense, + ) + + run_direct_dense(x, weight.t(), out, use_pdl, DirectTactic(*tactic)) + elif own_splitk: + # This target's copy of the split-K kernel (kernels/m3_lowm_splitk): bit for bit FlashInfer's; with ``ar`` + # it also pushes the rows into a drafter boundary all-reduce (modeling.DRAFT_AR_PUSH). + from ...kernels.m3_lowm_splitk import ( + SplitKTactic, + run_splitk_dense, + run_splitk_dense_push, + ) + + if ar is not None: + run_splitk_dense_push(x, weight.t(), out, use_pdl, SplitKTactic(*tactic), ar) + else: + run_splitk_dense(x, weight.t(), None, out, use_pdl, SplitKTactic(*tactic)) + else: + from flashinfer.gemm.kernels.dense_bf16_gemm_sm100_splitk import ( + SplitKTactic, + run_splitk_dense, + ) + + run_splitk_dense(x, weight.t(), None, out, use_pdl, SplitKTactic(*tactic)) + return out + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + own_splitk: bool = False, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """bf16(x @ weight^T) summed in fp64 and rounded once; ``out``, ``kind``, ``tactic`` and ``use_pdl`` only + choose where and how the op writes.""" + return (x.double() @ weight.double().t()).to(torch.bfloat16) + + def is_valid( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + own_splitk: bool = False, + ar: Optional[tuple] = None, + ) -> None: + """The kernels index ``x`` and ``out`` densely: a strided view is read as if it were packed.""" + assert x.dim() == 2 and x.is_contiguous(), "x must be a contiguous [M, K] matrix" + assert weight.dim() == 2 and weight.is_contiguous(), ( + "weight must be a contiguous [N, K] matrix" + ) + assert out.is_contiguous() and tuple(out.shape) == (x.shape[0], weight.shape[0]), ( + "out must be a contiguous [M, N] matrix" + ) + assert ar is None or (own_splitk and kind == "s" and len(ar) == 5), ( + "ar (the boundary all-reduce push) runs only on this target's split-K copy" + ) + + +m3_draft_lowm = M3DraftLowM() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_fp32_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_fp32_linear.py new file mode 100644 index 000000000000..9702cc0778a4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_fp32_linear.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The router's fp32 projection beyond the decode routers' rows: ``float(x) @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + + +class M3Fp32Linear(OpWrapper): + """fp32 router logits ``F.linear(float(x[M, 6144]), w[128, 6144])`` for prefill and steps above 48 rows (the + fp32 GEMV and the tensor-core router take 1..48). Whatever fp32 matmul precision the process runs with applies + (the engine leaves torch's default).""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="a prefill chunk", spec=dict(m=2048, e=128, k=6144)), + Cell( + why="a generation step above the tensor-core router's 48 rows", + spec=dict(m=64, e=128, k=6144), + ), + ) + + note = """ + Under torch's default fp32 matmul precision the product may be TF32: the test's band covers that rounding against + the true fp32 reference. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x.to(torch.float32), w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + with true_fp32_matmul(): + return x.float() @ w.t() + + +m3_fp32_linear = M3Fp32Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mx_gemv.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mx_gemv.py new file mode 100644 index 000000000000..c6ccde3e5ae4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mx_gemv.py @@ -0,0 +1,166 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 x MXFP8 GEMV (tcgen05, block-scaled, CuTe DSL) on an activation its producer already quantized.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.mx_gemv import op as _gemv +from ..moe.m3_moe_engine import mxfp8_dequant + +__all__ = ["PROJECTIONS", "gemv_is_valid", "m3_mx_gemv", "served_cells"] + +# (N, K) of this target's MXFP8 projections at tp4, the GEMV's tuned shapes. +QKV_SPARSE = (2560, 6144) +QKV = (2304, 6144) +O_PROJ = (6144, 2048) +SHARED_GATE_UP = (1536, 6144) +SHARED_DOWN = (6144, 768) +DENSE_GATE_UP = (6144, 6144) +DENSE_DOWN = (6144, 3072) +PROJECTIONS = { + "sparse-layer QKV": QKV_SPARSE, + "dense-attention QKV": QKV, + "o_proj": O_PROJ, + "shared gate_up": SHARED_GATE_UP, + "shared down": SHARED_DOWN, + "dense gate_up": DENSE_GATE_UP, + "dense down": DENSE_DOWN, +} + + +def served_cells(named: tuple[Cell, ...], x_bf16: bool, max_m: int) -> tuple[Cell, ...]: + """A cell for every (projection, rows) the GEMV serves up to ``max_m`` rows that ``named`` does not list. + + Certification is per projection: a projection's split-K, ring and MMA width differ, so each (N, K, M) the + decode dispatch can send here is its own cell. + """ + listed = {(cell.spec["nk"], cell.spec["m"]) for cell in named} + return tuple( + Cell(why=f"{name}, {m} rows", spec=dict(nk=nk, m=m)) + for name, nk in PROJECTIONS.items() + for m in range(1, max_m + 1) + if (nk, m) not in listed and _gemv.serves(nk[0], nk[1], m, x_bf16) + ) + + +def gemv_reference( + x: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor +) -> torch.Tensor: + """bf16(x @ dequant(weight)^T) for fp32 activation values x [M, K], summed in fp64 and rounded once.""" + return (x.double() @ mxfp8_dequant(weight, weight_scale).double().t()).to(torch.bfloat16) + + +def gemv_is_valid(weight: torch.Tensor, weight_scale: torch.Tensor) -> None: + n, k = weight.shape + # The kernel reads both from data_ptr() as dense buffers at the layout it was compiled for. + assert weight.dtype == torch.float8_e4m3fn and weight.is_contiguous(), ( + "weight must be contiguous e4m3 [N, K]" + ) + assert weight_scale.numel() == n * k // 32, ( + "weight_scale must be the 128x4-swizzled UE8M0 buffer [N * K / 32]" + ) + + +class M3MxGemv(OpWrapper): + """bf16 [M, N] = xq [M, K] (e4m3, 128x4-swizzled UE8M0 scales ``xsf``) x weight [N, K]^T (e4m3, swizzled + ``weight_scale``), M <= 48; None when the shape is not served (the caller falls back). + + ``trigger_early``: release the PDL dependents before the epilogue. ``cfg``: a (split, ring[, contig, nacc, a_tmem, + st_async]) tuning override; default the table's. ``use_pdl``: default the environment's. ``ar`` = (multicast + workspace address, flags address, rank, world, bucket tokens, mode): also push the rows into the next boundary + all-reduce's exchange (then ``op.last_pushed`` is True). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The projections' (N, K) at the token counts each build differs in: the MMA width (8 / 16 / 32 / 48 columns) a + # call uses is the smallest that holds M, and the shared gate_up takes another split at 2 tokens and above 8. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token (split 4)", spec=dict(nk=QKV_SPARSE, m=1)), + Cell(why="sparse-layer QKV, 8 tokens", spec=dict(nk=QKV_SPARSE, m=8)), + Cell( + why="sparse-layer QKV, 48 verify rows: the widest MMA", spec=dict(nk=QKV_SPARSE, m=48) + ), + Cell(why="dense-attention QKV, one token", spec=dict(nk=QKV, m=1)), + Cell(why="dense-attention QKV, 13 rows: the 16-column MMA", spec=dict(nk=QKV, m=13)), + Cell(why="o_proj, one token (split 2)", spec=dict(nk=O_PROJ, m=1)), + Cell(why="o_proj, 24 rows: the 32-column MMA", spec=dict(nk=O_PROJ, m=24)), + Cell( + why="shared gate_up, one token (A from shared memory)", + spec=dict(nk=SHARED_GATE_UP, m=1), + ), + Cell(why="shared gate_up, 2 tokens: its split-4 config", spec=dict(nk=SHARED_GATE_UP, m=2)), + Cell( + why="shared gate_up, 12 rows: the wide split-4 config", + spec=dict(nk=SHARED_GATE_UP, m=12), + ), + Cell(why="shared down, one token (unsplit, K 768)", spec=dict(nk=SHARED_DOWN, m=1)), + Cell(why="shared down, 8 tokens", spec=dict(nk=SHARED_DOWN, m=8)), + Cell(why="dense gate_up, one token", spec=dict(nk=DENSE_GATE_UP, m=1)), + Cell(why="dense down, 8 tokens", spec=dict(nk=DENSE_DOWN, m=8)), + ) + # Every other (projection, rows) the decode dispatch can send here (modeling.MX_GEMV_ROWS). + CELLS = CELLS + served_cells(CELLS, x_bf16=False, max_m=48) + + note = """ + The kernel sums in fp32 in split-K order where `reference` sums in fp64: the bf16 outputs agree within the band the + test derives. Not covered: the all-reduce push (`ar`), which needs a multicast exchange; `cfg` overrides beyond the + table (the target passes none). Shapes outside the table, M > 48, and 17..32 rows at K < 1024 return None. + """ + + def __call__( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv(xq, xsf, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The dequantized product; the launch options are ignored.""" + return gemv_reference(mxfp8_dequant(xq, xsf), weight, weight_scale) + + def is_valid( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + m, k = xq.shape + assert xq.dtype == torch.float8_e4m3fn and xq.is_contiguous(), ( + "xq must be contiguous e4m3 [M, K]" + ) + # A linear-layout [M, K / 32] scale buffer is read as the swizzled one: wrong scales, no error. + assert xsf.numel() >= (m + 127) // 128 * 128 * (k // 32), ( + "xsf must be the 128x4-swizzled scale buffer (rows padded to 128), not the linear one" + ) + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv = M3MxGemv() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mx_gemv_bf16.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mx_gemv_bf16.py new file mode 100644 index 000000000000..f8b2ab00217f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mx_gemv_bf16.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 GEMV (CuTe DSL) on a bf16 activation, quantized to MXFP8 in the kernel's prologue.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.mx_gemv import op as _gemv +from ..moe.m3_moe_engine import mxfp8_quant_dequant +from .m3_mx_gemv import ( + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, + gemv_is_valid, + gemv_reference, + served_cells, +) + +__all__ = ["m3_mx_gemv_bf16"] + + +class M3MxGemvBf16(OpWrapper): + """bf16 [M, N] = mxfp8_quantize(x) x weight [N, K]^T for a contiguous, 16-byte aligned bf16 ``x`` [M, K]; the + quantization is mxfp8_quantize's (UE8M0 scale per 32 = the smallest power of two >= amax / 448). None when the + shape is not served. The other arguments as ``m3_mx_gemv``'s. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The prologue quantizes one 32-element block per thread and round: the projections at the token counts the + # target sends here (no producer-quantized activation), up to the widest each serves. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token", spec=dict(nk=QKV_SPARSE, m=1)), + Cell( + why="sparse-layer QKV, 16 rows: the most its split-4 prologue serves", + spec=dict(nk=QKV_SPARSE, m=16), + ), + Cell(why="dense-attention QKV, 8 tokens", spec=dict(nk=QKV, m=8)), + Cell(why="o_proj, one token", spec=dict(nk=O_PROJ, m=1)), + Cell(why="o_proj, 32 rows: the 32-column MMA", spec=dict(nk=O_PROJ, m=32)), + Cell(why="shared gate_up, one token", spec=dict(nk=SHARED_GATE_UP, m=1)), + Cell(why="shared gate_up, 2 tokens: its split-4 config", spec=dict(nk=SHARED_GATE_UP, m=2)), + Cell(why="shared down, 8 tokens (K 768)", spec=dict(nk=SHARED_DOWN, m=8)), + Cell( + why="dense gate_up, 8 tokens: the most split 2 at K 6144 serves", + spec=dict(nk=DENSE_GATE_UP, m=8), + ), + Cell(why="dense down, one token", spec=dict(nk=DENSE_DOWN, m=1)), + ) + # Every other (projection, rows) the decode dispatch can send here (modeling.MX_GEMV_BF16_MAX_TOKENS). + CELLS = CELLS + served_cells(CELLS, x_bf16=True, max_m=32) + + note = """ + As `m3_mx_gemv`. The in-kernel quantization is mxfp8_quantize's bit for bit except where the kernel's + rcp.approx(448) and an exact 1/448 put a block's amax on different sides of a power of two (not observed on the + cells). Not covered: `ar`, `cfg` overrides. Returns None for rows beyond 8 rounds of the prologue per CTA. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv_bf16(x, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The MXFP8 round trip of x times the dequantized weight; the launch options are ignored.""" + return gemv_reference(mxfp8_quant_dequant(x), weight, weight_scale) + + def is_valid( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv_bf16 = M3MxGemvBf16() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_gemm.py new file mode 100644 index 000000000000..0b422a7167e5 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_gemm.py @@ -0,0 +1,58 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 GEMM on an activation its producer already quantized (rows the GEMV does not serve).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import mxfp8_gemm_autotuned + +from .m3_mxfp8_linear import mxfp8_reference + + +class M3Mxfp8Gemm(OpWrapper): + """``act @ weight^T`` for an MXFP8 activation (e4m3 [M, K] + 128x4-swizzled UE8M0 scales) and an MXFP8 weight + [N, K] (swizzled scales), bf16 out. The GEMM backend per token bucket is the AutoTuner's (the same runner, op key + and tactic cache as the engine-tuned quantize + GEMM); ``tune`` profiles in the engine's tuning pass.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The decode dispatch hands it the boundary / merge MXFP8 activation on rows the GEMV does not serve: the + # unsplit shared down projection at 17..32 rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert down projection, 17 rows", spec=dict(m=17, n=6144, k=768)), + Cell(why="shared expert down projection, 32 rows", spec=dict(m=32, n=6144, k=768)), + ) + + note = """ + Exactly the MXFP8 product (fp32 accumulation, one bf16 rounding): `reference` dequantizes both operands and + multiplies in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + return mxfp8_gemm_autotuned(act, act_scale, weight, weight_scale, output_dtype, tune=tune) + + def reference( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + """The dequantized product; ``tune`` is ignored.""" + return mxfp8_reference(act, act_scale, weight, weight_scale).to(output_dtype) + + +m3_mxfp8_gemm = M3Mxfp8Gemm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_linear.py new file mode 100644 index 000000000000..d7a70d725189 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_linear.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One MXFP8 projection on the engine-tuned path: dynamic MXFP8 quantize of a bf16 activation + block-scaled GEMM.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import mxfp8_quantize_gemm_autotuned + +_BLOCK = 32 + + +def _unswizzle(scale: torch.Tensor, rows: int, cols: int) -> torch.Tensor: + """[rows, cols] UE8M0 bytes from the 128x4-swizzled layout (rows padded to 128, columns to 4).""" + pad_r = (rows + 127) // 128 * 128 + pad_c = (cols + 3) // 4 * 4 + s = scale.view(torch.uint8).reshape(pad_r // 128, pad_c // 4, 32, 4, 4) + return s.permute(0, 3, 2, 1, 4).reshape(pad_r, pad_c)[:rows, :cols] + + +def _dequant(x: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + rows, k = x.shape + s = _unswizzle(scale, rows, k // _BLOCK).to(torch.int32) + factor = torch.pow(2.0, (s - 127).to(torch.float32)) + return ( + x.view(torch.float8_e4m3fn).float().view(rows, k // _BLOCK, _BLOCK) * factor.unsqueeze(-1) + ).view(rows, k) + + +def mxfp8_reference(act, act_scale, weight, weight_scale) -> torch.Tensor: + """fp32 product of two dequantized MXFP8 operands (swizzled UE8M0 scales).""" + with true_fp32_matmul(): + return _dequant(act, act_scale) @ _dequant(weight, weight_scale).t() + + +class M3Mxfp8Linear(OpWrapper): + """``x @ weight^T`` for bf16 ``x`` [M, K] and an MXFP8 weight [N, K] (e4m3 + 128x4-swizzled UE8M0 scales), bf16 + out, on the backend the engine selected: + + * ``tuned_quantize_gemm`` (decode CUDA graphs once the engine tuned per-bucket backends): the AutoTuner's quantize + and GEMM per token bucket (``tune`` profiles them in the engine's tuning pass); + * otherwise the native ``mxfp8_quantize`` (swizzled scales) and then FlashInfer's MXFP8 GEMM (``flashinfer``) or + the CUTLASS block-scaled GEMM, through the autotuner while the engine tunes it (``native_tuning``), scaled by + ``alpha`` (fp32 [1], 1.0 for these checkpoints). + + Every MXFP8 projection of the target (the attention QKV and o_proj, the dense MLPs and the shared expert) runs it + outside the decode-step kernels. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="sparse-layer QKV, a prefill chunk, native GEMM", + spec=dict(m=2048, n=2560, k=6144, path="native"), + ), + Cell( + why="o_proj, a prefill chunk, native GEMM", + spec=dict(m=2048, n=6144, k=2048, path="native"), + ), + Cell( + why="dense down projection, eager decode, native GEMM", + spec=dict(m=4, n=6144, k=3072, path="native"), + ), + Cell( + why="shared gate_up, FlashInfer GEMM", + spec=dict(m=64, n=1536, k=6144, path="flashinfer"), + ), + Cell(why="o_proj, tuned quantize + GEMM", spec=dict(m=64, n=6144, k=2048, path="tuned")), + ) + + note = """ + Exactly the MXFP8 product of the MXFP8 round trip of x (fp32 accumulation, one bf16 rounding): `reference` + quantizes x with the same block rule (UE8M0 scale = the block's power-of-two ceiling over e4m3's 448) and + multiplies dequantized operands in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + if tuned_quantize_gemm: + return mxfp8_quantize_gemm_autotuned(x, weight, weight_scale, output_dtype, tune=tune) + act, act_scale = torch.ops.trtllm.mxfp8_quantize(x, True) + if flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8( + act, act_scale, weight, weight_scale, output_dtype + ) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, weight, weight_scale, alpha, output_dtype) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + """The MXFP8 round trip of ``x`` times the dequantized weight, times ``alpha``; the backend flags are + ignored.""" + m, k = x.shape + blocks = x.float().view(m, k // _BLOCK, _BLOCK) + amax = blocks.abs().amax(-1, keepdim=True) + exp = torch.ceil(torch.log2(torch.where(amax > 0, amax / 448.0, torch.ones_like(amax)))) + scale = torch.pow(2.0, exp) + q = (blocks / scale).to(torch.float8_e4m3fn).float() * scale + with true_fp32_matmul(): + out = q.view(m, k) @ _dequant(weight, weight_scale).t() + return (out * alpha.float()).to(output_dtype) + + +m3_mxfp8_linear = M3Mxfp8Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_pinned_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_pinned_gemm.py new file mode 100644 index 000000000000..ee781b16ff42 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_pinned_gemm.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 GEMM at a fixed FlashInfer tactic (MX-TAC): the same choice on every rank and every start.""" + +from __future__ import annotations + +from typing import Dict, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import FlashInferMXFP8GemmRunner + +from .m3_mxfp8_linear import mxfp8_reference + +# One runner per output dtype (a runner holds no per-call state). +_RUNNERS: Dict[Tuple[type, torch.dtype], object] = {} + + +def _runner(dtype: torch.dtype): + runner = _RUNNERS.get((FlashInferMXFP8GemmRunner, dtype)) + if runner is None: + runner = _RUNNERS[(FlashInferMXFP8GemmRunner, dtype)] = FlashInferMXFP8GemmRunner(dtype) + return runner + + +class M3Mxfp8PinnedGemm(OpWrapper): + """``act @ weight^T`` for an MXFP8 activation (e4m3 [M, K] + swizzled UE8M0 scales) and an MXFP8 weight [N, K] + (swizzled scales) on FlashInfer's MXFP8 GEMM at ``tactic``, no AutoTuner; ``output_dtype`` out.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The decode graph's MXFP8 GEMMs on rows the GEMVs do not serve: the shared down projection on its boundary + # activation (17..32 rows) and the bf16 projections past the GEMV rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert down projection, 17 rows", spec=dict(m=17, n=6144, k=768)), + Cell(why="packed QKV + index projection, 48 rows", spec=dict(m=48, n=4992, k=6144)), + ) + + note = """ + The MXFP8 product (fp32 accumulation, one rounding to the output dtype): `reference` dequantizes both operands and + multiplies in fp32; the test's band covers the accumulation order. The tactic only picks the kernel. + """ + + def __call__( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tactic: int, + ) -> torch.Tensor: + return _runner(output_dtype).forward([act, act_scale, weight, weight_scale], tactic=tactic) + + def reference( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tactic: int, + ) -> torch.Tensor: + """The dequantized product; ``tactic`` is ignored.""" + return mxfp8_reference(act, act_scale, weight, weight_scale).to(output_dtype) + + +m3_mxfp8_pinned_gemm = M3Mxfp8PinnedGemm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_pinned_quantize.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_pinned_quantize.py new file mode 100644 index 000000000000..3f35af65048a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_mxfp8_pinned_quantize.py @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 activation quantize at a fixed tactic (MX-TAC), ahead of ``m3_mxfp8_pinned_gemm``.""" + +from __future__ import annotations + +from typing import Dict, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner + +_RUNNERS: Dict[torch.dtype, object] = {} + + +def _runner(dtype: torch.dtype): + runner = _RUNNERS.get(dtype) + if runner is None: + runner = _RUNNERS[dtype] = MXFP8QuantizeRunner(dtype) + return runner + + +class M3Mxfp8PinnedQuantize(OpWrapper): + """A bf16 activation [M, K] to MXFP8: (e4m3 [M, K], swizzled UE8M0 scales), on the runner's ``tactic``.""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="packed QKV + index projection input, 48 rows", spec=dict(m=48, k=6144)), + Cell(why="shared expert down projection input, 17 rows", spec=dict(m=17, k=768)), + ) + + note = """ + Bit-exact: `reference` is the stock MXFP8 quantize (``torch.ops.trtllm.mxfp8_quantize``, swizzled scales) of the + same input; every tactic computes the same block maxima and roundings. + """ + + def __call__(self, x: torch.Tensor, tactic: int) -> Tuple[torch.Tensor, torch.Tensor]: + return _runner(x.dtype).forward([x], tactic=tactic) + + def reference(self, x: torch.Tensor, tactic: int) -> Tuple[torch.Tensor, torch.Tensor]: + """The stock quantize; ``tactic`` is ignored.""" + return torch.ops.trtllm.mxfp8_quantize(x, True) + + +m3_mxfp8_pinned_quantize = M3Mxfp8PinnedQuantize() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_target_head.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_target_head.py new file mode 100644 index 000000000000..f8deb858c843 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gemm/m3_target_head.py @@ -0,0 +1,346 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The target lm_head shard GEMV of a verify step of at most 8 rows over a lossless 12-bit coding of its BF16 weight, +and that coding (``encode_chunked`` / ``decode_reference`` / ``round_trip_ok``). + +The coding: the BF16 exponents of a trained weight sit almost all in a window of 15 consecutive values +``[base, base + 14]``. Each weight is one byte ``sign << 7 | mantissa`` plus a 4-bit code ``exponent - base``; code 15 +marks an exception, which decodes to +0 (it adds nothing to the MMA sums); its exact product ``x * w`` is added from a +per-tile list. The bytes are laid out in ``mma.m16n8k16`` A-fragment order: +per 16-row tile and unit of 4 k16 steps, each lane reads the sign/mantissa bytes of steps 0-1, of steps 2-3, then 4 +code words (``[tiles][K / 64][3][32 lanes][16 bytes]``). A coded weight is used only after ``round_trip_ok`` proved +it decodes to the BF16 weight bit for bit. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_target_head"] + +#: Verify rows per call (requests x (draft length + 1)). +MAX_ROWS = 8 +#: Warps per CTA (the split-tile scratch holds 2 x 4 x 32 fp32 per warp). +CTA_WARPS = 16 +#: The target lm_head's TP4 shard (200064-id vocabulary) and the hidden size. +TARGET_SHARD = 50016 +HIDDEN = 6144 +#: Codes per window: 0..14 decode, 15 marks an exception. +WINDOW = 15 +#: The largest exponent base whose window top plus one (base + 15) is still a finite BF16 exponent. +MAX_BASE = 254 - WINDOW +#: Rows per block of encode_chunked / round_trip_ok (bounded temporaries). +ROWS_PER_CHUNK = 4096 + + +def _pad_rows(weight: torch.Tensor) -> torch.Tensor: + n = weight.shape[0] + tiles = (n + 15) // 16 + if tiles * 16 == n: + return weight + return torch.cat([weight, weight.new_zeros(tiles * 16 - n, weight.shape[1])]) + + +def _fragment_view(w: torch.Tensor) -> torch.Tensor: + """[tiles * 16, K] -> [tiles, K / 16, 32 lanes, 8 values] in mma A-fragment order (a permutation of the values).""" + n, k = w.shape + # rows: (tile, rh, g); cols: (ks, ch, j, e) -> (tile, ks, g, j, ch, rh, e) + return ( + w.view(n // 16, 2, 8, k // 16, 2, 4, 2) + .permute(0, 3, 2, 5, 4, 1, 6) + .reshape(n // 16, k // 16, 32, 8) + ) + + +def _exponents(weight: torch.Tensor) -> torch.Tensor: + return (weight.contiguous().view(torch.int16).to(torch.int32) >> 7) & 0xFF + + +def _window_base(hist: torch.Tensor) -> int: + """The base of the 15-exponent window holding the most weights (the lowest such base).""" + window = torch.stack([hist[b : b + WINDOW].sum() for b in range(0, MAX_BASE + 1)]) + return int(window.argmax()) + + +def encode(weight: torch.Tensor, base: Optional[int] = None): + """Lossless 12-bit coding of a BF16 [N, K] weight (see the module docstring); ``base`` defaults to this weight's + own window. + + Returns (packed uint8 [tiles * K / 64 * 1536], base, exc_off int32 [tiles + 1], exc_pos int32 [E] (row in tile + << 16 | k), exc_w fp32 [E, 2] ({w, 0})). + """ + w = _pad_rows(weight) + n, k = w.shape + tiles, ks = n // 16, k // 16 + assert ks % 4 == 0, "K must be a multiple of 64" + bits = w.contiguous().view(torch.int16).to(torch.int32) & 0xFFFF + expo = (bits >> 7) & 0xFF + if base is None: + base = _window_base(torch.bincount(expo.flatten(), minlength=256)) + code = expo - base + exc = (code < 0) | (code >= WINDOW) + code = torch.where(exc, torch.full_like(code, WINDOW), code) + sm = ((bits >> 8) & 0x80) | (bits & 0x7F) + sm = torch.where(exc, torch.zeros_like(sm), sm) + # Exceptions: (tile, row in tile, k) in tile-major order; code 15 decodes to +0. + rows, cols = exc.nonzero(as_tuple=True) + order = torch.argsort((rows // 16).long() * (16 * k) + (rows % 16).long() * k + cols.long()) + rows, cols = rows[order], cols[order] + exc_w = torch.stack( + [ + w[rows, cols].float(), + torch.zeros((rows.numel(),), device=w.device), + ], + dim=1, + ) + exc_pos = (((rows % 16) << 16) | cols).to(torch.int32) + exc_off = torch.zeros(tiles + 1, dtype=torch.int32, device=w.device) + exc_off[1:] = torch.cumsum(torch.bincount((rows // 16).long(), minlength=tiles), 0).to( + torch.int32 + ) + # Fragment order, then per (tile, group of 4 k16 steps, lane): 32 sign/mantissa bytes and 4 code words. + smf = _fragment_view(sm.to(torch.uint8).view(n, k)).view(tiles, ks // 4, 4, 32, 8) + cf = ( + _fragment_view(code.to(torch.uint8).view(n, k)) + .view(tiles, ks // 4, 4, 32, 8) + .to(torch.int32) + ) + # Code word of a k16 step: value 2p -> bits 4p..4p+3, value 2p+1 -> bits 16+4p.. + cw = torch.zeros(tiles, ks // 4, 4, 32, dtype=torch.int64, device=w.device) + for p in range(4): + cw |= (cf[..., 2 * p].long() << (4 * p)) | (cf[..., 2 * p + 1].long() << (16 + 4 * p)) + cw = cw.to(torch.int32) # wraps for bit 31; the kernel reads it as uint32 + # Word 0 of a lane = the bytes of steps 0 and 1; word 1 = steps 2 and 3; word 2 = the code words of steps 0..3. + seg01 = ( + smf.permute(0, 1, 3, 2, 4).reshape(tiles, ks // 4, 32, 2, 16).permute(0, 1, 3, 2, 4) + ) # [.., 2, 32, 16] + seg2 = cw.permute(0, 1, 3, 2).contiguous().view(torch.uint8).view(tiles, ks // 4, 1, 32, 16) + packed = torch.cat([seg01.contiguous(), seg2], dim=2) # [tiles, ks / 4, 3, 32, 16] + return packed.contiguous().view(-1), base, exc_off, exc_pos, exc_w.contiguous() + + +def encode_chunked(weight: torch.Tensor, rows_per_chunk: int = ROWS_PER_CHUNK): + """``encode`` over row blocks (bounded temporaries), with the exponent window chosen on the whole weight.""" + n, _ = weight.shape + assert rows_per_chunk % 16 == 0 + hist = torch.zeros(256, dtype=torch.int64, device=weight.device) + for r in range(0, n, rows_per_chunk): + hist += torch.bincount(_exponents(weight[r : r + rows_per_chunk]).flatten(), minlength=256) + base = _window_base(hist) + packed, offs, poss, ws = [], [], [], [] + n_exc = 0 + for r in range(0, n, rows_per_chunk): + p, _, off, pos, ew = encode(weight[r : r + rows_per_chunk], base=base) + packed.append(p) + offs.append(off[:-1] + n_exc) + poss.append(pos) + ws.append(ew) + n_exc += pos.numel() + exc_off = torch.cat(offs + [torch.tensor([n_exc], dtype=torch.int32, device=weight.device)]) + return torch.cat(packed), base, exc_off, torch.cat(poss), torch.cat(ws) + + +def decode_reference(packed: torch.Tensor, base: int, n: int, k: int) -> torch.Tensor: + """BF16 [N, K] from the coded stream with the in-kernel rule (exceptions, code 15, decode to +0).""" + tiles, ks = (n + 15) // 16, k // 16 + p = packed.view(tiles, ks // 4, 3, 32, 16) + sm = ( + p[:, :, :2] + .permute(0, 1, 3, 2, 4) + .reshape(tiles, ks // 4, 32, 4, 8) + .permute(0, 1, 3, 2, 4) + .to(torch.int32) + ) + cw = ( + p[:, :, 2] + .contiguous() + .view(torch.int32) + .view(tiles, ks // 4, 32, 4) + .permute(0, 1, 3, 2) + .long() + & 0xFFFFFFFF + ) + codes = torch.stack([(cw >> (4 * (i // 2) + 16 * (i % 2))) & 0xF for i in range(8)], dim=-1).to( + torch.int32 + ) + bits = ((sm & 0x80) << 8) | ((codes + base) << 7) | (sm & 0x7F) + bits = torch.where(codes == WINDOW, torch.zeros_like(bits), bits) + frag = bits.reshape(tiles, ks, 32, 8) + # Inverse of _fragment_view: (tile, ks, g, j, ch, rh, e) -> (tile, rh, g, ks, ch, j, e). + w = frag.view(tiles, ks, 8, 4, 2, 2, 2).permute(0, 5, 2, 1, 4, 3, 6).reshape(tiles * 16, k) + return w.to(torch.int16).view(torch.bfloat16)[:n] + + +def decode( + packed: torch.Tensor, + base: int, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + n: int, + k: int, + row0: int = 0, + row1: Optional[int] = None, +) -> torch.Tensor: + """BF16 rows [row0, row1) of the coded weight [N, K] with the exceptions applied (what the kernel multiplies by). + ``row0`` must be a multiple of 16.""" + row1 = n if row1 is None else row1 + assert row0 % 16 == 0 + t0, t1 = row0 // 16, (row1 + 15) // 16 + unit_bytes = (k // 64) * 3 * 32 * 16 + dec = decode_reference(packed[t0 * unit_bytes : t1 * unit_bytes], base, row1 - row0, k).clone() + counts = (exc_off[1:] - exc_off[:-1]).long() + tile_of = torch.repeat_interleave(torch.arange(counts.numel(), device=packed.device), counts) + rows = tile_of * 16 + (exc_pos >> 16).long() + cols = (exc_pos & 0xFFFF).long() + sel = (rows >= row0) & (rows < row1) + dec[rows[sel] - row0, cols[sel]] = exc_w.view(-1, 2)[sel, 0].to(torch.bfloat16) + return dec + + +def round_trip_ok( + packed: torch.Tensor, + base: int, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + weight: torch.Tensor, + rows_per_chunk: int = ROWS_PER_CHUNK, +) -> bool: + """Whether the coded weight (code values + exception list) decodes to ``weight`` bit for bit, with its exponent + window below the BF16 infinity exponent.""" + n, k = weight.shape + if not 0 <= base <= MAX_BASE or k % 64: + return False + if ( + packed.numel() != ((n + 15) // 16) * 16 * k * 3 // 2 + or exc_off.numel() != (n + 15) // 16 + 1 + ): + return False + for r0 in range(0, n, rows_per_chunk): + r1 = min(n, r0 + rows_per_chunk) + dec = decode(packed, base, exc_off, exc_pos, exc_w, n, k, r0, r1) + if not torch.equal(dec.view(torch.int16), weight[r0:r1].contiguous().view(torch.int16)): + return False + return True + + +class M3TargetHead(OpWrapper): + """``out[:M, :n]`` (BF16) = x [M, K] (BF16, M <= 8) times the coded BF16 shard W [n, K] transposed: each logit + the fp32 sum of the exact BF16 products, rounded once to BF16 (the in-tree BF16 head GEMM's arithmetic, in + another summation order). + + ``w`` / ``exp_base`` / ``exc_off`` / ``exc_pos`` / ``exc_w`` are ``encode_chunked``'s outputs for W; ``part`` + (fp32 [grid * 16 * 256]) and ``counter`` (int32 [ceil(n / 16)], zero between calls, re-armed by the kernel) are + the split-tile scratch; ``grid`` is the number of 16-warp CTAs (one per SM). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The verify of a fused-verify step at most 8 rows wide: 1 + 3 drafts for one or two requests (4 / 8 rows), and + # every other row count the kernel's one n8 MMA tile serves; one cell with an exception in many tiles, one with + # a shard whose rows are not a multiple of 16. + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"{m} verify row(s) of the TP4 target head shard", + spec=dict(m=m, n=TARGET_SHARD, k=HIDDEN), + ) + for m in range(1, MAX_ROWS + 1) + ), + Cell( + why="4 rows with outliers and exact zeros in many tiles (the exception lists)", + spec=dict(m=4, n=TARGET_SHARD, k=HIDDEN, exceptions=True), + ), + Cell( + why="2 rows over a shard of 4000 rows (250 tiles), K 1024", + spec=dict(m=2, n=4000, k=1024, exceptions=True), + ), + ) + + note = """ + The weight the kernel multiplies by is the checkpoint's BF16 weight bit for bit (round_trip_ok); an exception + adds nothing to the MMA sums and its exact product x * w is added at its tile's flush. A logit therefore differs + from the in-tree head's only by the fp32 summation order before its single BF16 rounding (ulp-level); + integer-valued data, whose sums are exact in any order, is equal bit for bit. Split tiles are summed in warp + order: the result is deterministic. + """ + + def __call__( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_target_head( + w, x, out, part, counter, exc_off, exc_pos, exc_w, exp_base, n, grid, pdl + ) + return out + + def reference( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> torch.Tensor: + """The BF16 logits [M, n] of x against the decoded weight: the fp32 product, rounded once to BF16. + + ``out``, ``part``, ``counter``, ``grid`` and ``pdl`` do not change the result.""" + weight = decode(w, exp_base, exc_off, exc_pos, exc_w, n, x.shape[1]) + with true_fp32_matmul(): + logits = x.float() @ weight.float().t() + return logits.to(torch.bfloat16) + + def is_valid( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> None: + # A split tile's last finisher is the CTA whose arrival makes the count reach its owners: a stale count + # stores a partial sum. + assert not bool(counter[: (n + 15) // 16].any()), "counter must be zero between calls" + # The exceptions are read per tile in list order with no bound against the coded stream. + assert int(exc_off[-1].item()) == exc_pos.numel(), "exc_off must end at the exception count" + assert bool((exc_off[1:] >= exc_off[:-1]).all()), "exc_off must be non-decreasing" + assert out.data_ptr() != x.data_ptr(), "out must not alias x" + + +m3_target_head = M3TargetHead() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gqamix_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gqamix_sparse_decode.py new file mode 100644 index 000000000000..b415e7be538c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/gqamix_sparse_decode.py @@ -0,0 +1,1337 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. +""" + +from __future__ import annotations + +import os +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +def _pdl_enabled() -> bool: + return os.environ.get("TRTLLM_ENABLE_PDL", "1") == "1" + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for _ in tl.range(chunk_start, chunk_end): + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for _ in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on the datacenter Blackwell and Rubin targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3), (10, 7)) + + +# Devices already found to support the NVFP4 decode kernels, by capability. +_NVFP4_DEVICE_CAPABILITIES: dict[torch.device, tuple[int, int]] = {} + + +def _nvfp4_device_capability(device: torch.device) -> tuple[int, int]: + """Return the device's capability, checking NVFP4 support once per device.""" + capability = _NVFP4_DEVICE_CAPABILITIES.get(device) + if capability is None: + capability = torch.cuda.get_device_capability(device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103/SM107; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _NVFP4_DEVICE_CAPABILITIES[device] = capability + return capability + + +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + # Checked once per device; the shape heuristics below consult it per call. + capability = _nvfp4_device_capability(q.device) if kv_nvfp4 else None + if kv_nvfp4: + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else 1 + ) + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = _pdl_enabled() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + if not direct_output: + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + **pdl_launch, + **merge_launch_options, + ) + + +__all__ = [ + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/__init__.py new file mode 100644 index 000000000000..77d7b84d9bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mixture-of-experts entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_moe16_engine.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_moe16_engine.py new file mode 100644 index 000000000000..871f6887b7a5 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_moe16_engine.py @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3's MoE layer for 9..13 decode rows in one CuTe DSL launch (``m3_moe16``): the router, the routed NVFP4 +experts and the shared MXFP8 expert, on caller-owned state (an ``M3Moe16State`` and one ``M3MoeLayer`` per MoE +layer).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.m3_moe.op import M3MoeLayer +from ...kernels.m3_moe.op16 import M3Moe16State +from .m3_moe_engine import moe_reference + +__all__ = ["M3Moe16State", "m3_moe16_engine"] + + +class M3Moe16Engine(OpWrapper): + """One MiniMax-M3 MoE layer for M (9..13 here; the state takes 9..16) decode rows: + ``layer.state.run(layer, hidden, out, arp)``, bf16(shared + routed) [M, 6144]. + + The same arguments and the same math as ``m3_moe_engine`` (the routed part operation for operation, the shared + expert always folded in) on an ``M3Moe16State``: a different schedule for more rows. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # One build per row count (m_max is a trace-time constant); the target routes 9..13 rows here (spec-decode verify + # of 2 x 6 rows lands at 12). + CELLS: tuple[Cell, ...] = ( + Cell(why="9 rows: the smallest build, a partial last slot group", spec=dict(tokens=9)), + Cell(why="10 rows: its own build (m_max is a trace-time constant)", spec=dict(tokens=10)), + Cell(why="11 rows: its own build, an odd row count", spec=dict(tokens=11)), + Cell(why="12 rows: two requests verifying 5-token drafts", spec=dict(tokens=12)), + Cell(why="13 rows: the largest the target routes here", spec=dict(tokens=13)), + ) + + note = """ + Bands as `m3_moe_engine` (same approximations, same quantization boundaries). Not covered: 14..16 rows, which the + state takes and this target never routes here; the all-reduce push (`arp`). No `is_valid`, as `m3_moe_engine`. + """ + + def __call__( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return layer.state.run(layer, hidden, out, arp=arp) + + def reference( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """``m3_moe_engine``'s reference with the shared expert; `out` and `arp` are ignored.""" + return moe_reference(hidden, layer, shared=True) + + +m3_moe16_engine = M3Moe16Engine() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_moe_engine.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_moe_engine.py new file mode 100644 index 000000000000..f7b53b205fd9 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_moe_engine.py @@ -0,0 +1,326 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3's MoE layer for 1..8 decode tokens in one CuTe DSL launch (``m3_moe``): the router, the routed NVFP4 +experts and the shared MXFP8 expert, on caller-owned state (an ``M3MoeState`` and one ``M3MoeLayer`` per MoE layer).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.quantization.utils.fp4_utils import ( + get_reorder_rows_for_gated_act_gemm_row_indices, + get_shuffle_matrix_a_row_indices, +) + +from ...kernels.m3_moe.op import ( + HIDDEN, + INTER, + SHARED_INTER, + TOP_K, + M3MoeLayer, + M3MoeState, + ar_push_desc, + is_supported, +) + +__all__ = [ + "M3MoeLayer", + "M3MoeState", + "ar_push_desc", + "is_supported", + "m3_moe_engine", + "moe_reference", + "moe_route", + "mxfp8_dequant", + "mxfp8_quant_dequant", + "unswizzle_sf", +] + +# The model's constants the kernels are built with: routed_scaling_factor, the shared expert's SwiGLU-OAI +# (swiglu_alpha, swiglu_limit, the (up + 1) offset). +ROUTED_SCALING = 2.0 +SHARED_ALPHA = 1.702 +SHARED_LIMIT = 7.0 + +_LOG2E = 1.4426950216293334961 +_RCP6 = 0.16666667163372039795 # fp32(1 / 6), the kernels' reciprocal of the E2M1 maximum +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) +_F32_MAX = 3.40282346638528859812e38 + + +def _f32(x: float) -> torch.Tensor: + return torch.tensor(x, dtype=torch.float32) + + +def _e2m1_decode(packed: torch.Tensor) -> torch.Tensor: + """fp32 values of packed E2M1 bytes [..., K / 2] (low nibble first) -> [..., K].""" + lut = torch.tensor(_E2M1, dtype=torch.float32, device=packed.device) + lo = lut[(packed & 0xF).long()] + hi = lut[(packed >> 4).long()] + return torch.stack((lo, hi), dim=-1).flatten(-2) + + +def _e2m1_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to the nearest E2M1 value, ties to even, saturating at 6 (cvt.rn.satfinite.e2m1x2).""" + a = x.abs() + q = torch.full_like(a, 6.0) + for hi, value, inclusive in ( + (5.0, 4.0, True), + (3.5, 3.0, False), + (2.5, 2.0, True), + (1.75, 1.5, False), + (1.25, 1.0, True), + (0.75, 0.5, False), + (0.25, 0.0, True), + ): + q = torch.where(a <= hi if inclusive else a < hi, torch.full_like(a, value), q) + return torch.where(x < 0, -q, q) + + +def _e4m3_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to E4M3, nearest even, saturating (cvt.rn.satfinite.e4m3x2), as fp32.""" + return x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn).float() + + +def _ue8m0_up(x: torch.Tensor) -> torch.Tensor: + """UE8M0 code of x >= 0 rounded toward +inf, saturating (cvt.rp.satfinite.ue8m0x2), int32.""" + bits = x.contiguous().view(torch.int32) + code = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + return code.clamp(0, 254) + + +def unswizzle_sf(sf: torch.Tensor, rows: int, blocks: int) -> torch.Tensor: + """The [rows, blocks] scale bytes of a 128x4-swizzled scale buffer (int64 indices, any byte dtype).""" + flat = sf.reshape(-1).view(torch.uint8) + r = torch.arange(rows, device=flat.device).view(-1, 1) + b = torch.arange(blocks, device=flat.device).view(1, -1) + padded_blocks = (blocks + 3) // 4 * 4 + idx = ( + (r // 128) * (128 * padded_blocks) + + (b // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (b % 4) + ) + return flat[idx] + + +def mxfp8_dequant(q: torch.Tensor, sf: torch.Tensor) -> torch.Tensor: + """fp32 values of e4m3 ``q`` [R, K] with its 128x4-swizzled UE8M0 scales (one per 32 elements).""" + rows, k = q.shape + codes = unswizzle_sf(sf, rows, k // 32).float() + return q.view(torch.float8_e4m3fn).float() * torch.exp2(codes - 127.0).repeat_interleave( + 32, dim=1 + ) + + +def mxfp8_quant_dequant(x: torch.Tensor) -> torch.Tensor: + """x [R, K] through MXFP8 (one UE8M0 scale per 32, the smallest power of two >= amax / 448; e4m3 data), back to + fp32 -- mxfp8_quantize's recipe, which every MXFP8 activation here reproduces.""" + rows, k = x.shape + blocks = x.float().view(rows, k // 32, 32) + code = _ue8m0_up(blocks.abs().amax(dim=-1) * _f32(1.0 / 448.0).to(x.device)) + scale = torch.exp2(code.float() - 127.0).unsqueeze(-1) + return (_e4m3_round(blocks / scale) * scale).view(rows, k) + + +def _bf16(x: torch.Tensor) -> torch.Tensor: + return x.to(torch.bfloat16).float() + + +def _gemm(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """a [M, K] @ b [N, K]^T, accumulated in fp64 and rounded once to fp32.""" + return (a.double() @ b.double().t()).float() + + +def _row_perm(rows: int, gated: bool) -> torch.Tensor: + """The TRTLLM-Gen row order: stored row j holds logical row perm[j] (the gated interleave, then the 32-row shuffle + of epilogue tile 128).""" + probe = torch.empty(rows, 1) + perm = get_shuffle_matrix_a_row_indices(probe, epilogue_tile_m=128) + if gated: + perm = get_reorder_rows_for_gated_act_gemm_row_indices(probe)[perm] + return perm + + +def _nvfp4_expert(w: torch.Tensor, sf: torch.Tensor, perm_inv: torch.Tensor) -> torch.Tensor: + """One expert's stored NVFP4 matrix (values [R, K / 2], swizzled e4m3 scales) as raw fp32 [R, K] in logical row + order (the global scale is not applied: it is the per-expert alpha).""" + rows, half = w.shape + codes = unswizzle_sf(sf, rows, 2 * half // 16).view(torch.float8_e4m3fn).float() + vals = _e2m1_decode(w) * codes.repeat_interleave(16, dim=1) + return vals[perm_inv] + + +def moe_route(x: torch.Tensor, router_weight: torch.Tensor, bias: torch.Tensor): + """MiniMax2 routing on the fp32 router GEMV: (top-4 expert ids [M, 4] int64, bf16-rounded weights [M, 4] fp32).""" + logits = _gemm(x, router_weight) + sig = (0.5 * torch.tanh(0.5 * logits).double() + 0.5).float() + keys = sig + bias + ids = torch.sort(keys, dim=-1, descending=True, stable=True).indices[:, :TOP_K] + sk = keys.gather(1, ids) - bias[ids] + total = (sk[:, 0] + sk[:, 2]) + (sk[:, 1] + sk[:, 3]) + w = (sk * _f32(ROUTED_SCALING).to(x.device)) / (total + _f32(1e-20).to(x.device)).unsqueeze(1) + return ids, _bf16(w) + + +def _nvfp4_act(x: torch.Tensor, sc: torch.Tensor) -> torch.Tensor: + """The activation's NVFP4 quantization (fp4_quantize(x, sc, 16)), as raw fp32 values (q x block scale).""" + rows, k = x.shape + blocks = x.view(rows, k // 16, 16) + vmax = blocks.abs().amax(dim=-1, keepdim=True) + sff = _e4m3_round(sc * (vmax * _f32(_RCP6).to(x.device))) + # The kernels' rcp.approx(sf x rcp.approx(sc)) lands where the correctly rounded sc / sf does on every element + # measured (the 1 / (sf x (1 / sc)) spelling differs on ~0.06% of them). + osc = torch.where(vmax != 0, sc / sff, torch.zeros_like(vmax)) + return (_e2m1_round(blocks * osc) * sff).view(rows, k) + + +def _fc1_epilogue(up: torch.Tensor, gate: torch.Tensor, esc: torch.Tensor) -> torch.Tensor: + """TRTLLM-Gen's SwiGLU-OAI FC1 epilogue on the raw FC1 sums (fp32 [I]) and its NVFP4 requantization: the raw FC2 + input values. ``esc`` = (fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha) of the expert, with + act_beta and act_clamp already divided by fc31_alpha (the loader's convention).""" + a, c, alpha, beta, lim = (esc[i] for i in range(5)) + gc = torch.where(gate > lim, lim, gate) + gc = torch.where(gate < -_F32_MAX, torch.full_like(gc, -_F32_MAX), gc) + uc = torch.minimum(torch.maximum(up, -lim), lim) + c1 = a * (alpha * _f32(_LOG2E).to(up.device)) + sgm = 1.0 / (torch.exp2(-(gc * c1)) + 1.0) + h = (uc.double() * a.double() + (a * beta).double()).float() * (gc * sgm) + blocks = h.view(-1, 16) + amax = (blocks.abs() * c).amax(dim=-1, keepdim=True) + sff = _e4m3_round(amax * _f32(_RCP6).to(up.device)) + osc = torch.where(sff != 0, 1.0 / sff, torch.zeros_like(sff)) + return (_e2m1_round((blocks * osc) * c) * sff).view(-1) + + +def _shared_expert(x: torch.Tensor, shared_mxfp8) -> torch.Tensor: + """The shared expert as the target runs it outside the engine: MXFP8 gate_up (bf16 out), SwiGLU-OAI with the + model's alpha / limit / (up + 1) in fp32 (bf16 out), MXFP8 down (bf16 out).""" + gu_w, gu_sf, dn_w, dn_sf = shared_mxfp8 + gu = _bf16(_gemm(mxfp8_quant_dequant(x), mxfp8_dequant(gu_w, gu_sf))) + gate, up = gu[:, :SHARED_INTER], gu[:, SHARED_INTER:] + a = torch.minimum(gate, torch.full_like(gate, SHARED_LIMIT)) + ub = up.clamp(-SHARED_LIMIT, SHARED_LIMIT) + 1.0 + sg = 1.0 / ( + torch.exp2((a * _f32(-SHARED_ALPHA).to(x.device)) * _f32(_LOG2E).to(x.device)) + 1.0 + ) + h = _bf16(ub * (sg * a)) + return _bf16(_gemm(mxfp8_quant_dequant(h), mxfp8_dequant(dn_w, dn_sf))) + + +def moe_reference(hidden: torch.Tensor, layer: M3MoeLayer, shared: bool) -> torch.Tensor: + """The MoE layer of the decode engines (both token ranges compute this), in plain torch: bf16 [M, 6144].""" + x = hidden.float() + m = x.shape[0] + ids, weights = moe_route(x, layer.router_weight, layer.bias) + xr = _nvfp4_act(x, layer.fc31_input_scale.reshape(()).float()) + inv31 = torch.argsort(_row_perm(2 * INTER, gated=True)).to(x.device) + inv2 = torch.argsort(_row_perm(HIDDEN, gated=False)).to(x.device) + y = torch.zeros(m, TOP_K, HIDDEN, dtype=torch.float32, device=x.device) + for e in torch.unique(ids).tolist(): + w31 = _nvfp4_expert(layer.w3_w1_weight[e], layer.w3_w1_weight_scale[e], inv31) + w2 = _nvfp4_expert(layer.w2_weight[e], layer.w2_weight_scale[e], inv2) + esc = layer.escal[:, e] + for t, k in (ids == e).nonzero().tolist(): + acc1 = _gemm(xr[t : t + 1], w31)[0] + # The buffer holds [w3 (up); w1 (gate)]. + h = _fc1_epilogue(acc1[:INTER], acc1[INTER:], esc) + y[t, k] = _bf16(_gemm(h.unsqueeze(0), w2)[0] * esc[5]) + acc = torch.zeros(m, HIDDEN, dtype=torch.float64, device=x.device) + for k in range(TOP_K): + acc = (acc + weights[:, k : k + 1].double() * y[:, k].double()).float().double() + routed = acc.float() + if not shared: + return routed.to(torch.bfloat16) + return (_shared_expert(x, layer.shared_mxfp8) + _bf16(routed)).to(torch.bfloat16) + + +class M3MoeEngine(OpWrapper): + """One MiniMax-M3 MoE layer for M (1..8) decode tokens: ``layer.state.run(layer, hidden, out, arp)``. + + ``hidden`` bf16 [M, 6144] contiguous, M = the state's token count. ``layer`` (from ``M3MoeState.layer``) carries the + TRTLLM-Gen NVFP4 expert buffers read in place, the per-expert scalars, the router weight / bias, the activation's + NVFP4 global scale and, on a ``shared`` state, the shared expert's MXFP8 weights; its state carries the compiled + kernel and the workspace every layer on it shares. Returns bf16 [M, 6144]: bf16(shared + routed) on a ``shared`` + state, else the routed partial. ``out``: write there instead of a new tensor. ``arp`` (``ar_push_desc``): also + push the rows into a multicast all-reduce exchange. + + The routed part is TRTLLM-Gen's NVFP4 MoE on the fp32 router logits: sigmoid (tanh form) + correction bias top-4 + of 128 (ties to the lower id), weights sigmoid / sum * 2.0 rounded to bf16; the activation quantized to NVFP4 with + the layer's global scale; per expert the raw FC1 sums, SwiGLU-OAI (alpha / beta / clamp), the NVFP4 requant with + fc31_scale_c, FC2, bf16(fc2_alpha x sum); the fma chain over the top-4 in order. The shared expert is the target's + MXFP8 SwiGLU-OAI MLP. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # One cell per token count: m_max is a trace-time constant, so each is its own build, and the target builds all + # eight (decode batches of 1..8 requests, spec-decode verify rows up to 8). The whys name what the build differs in. + CELLS: tuple[Cell, ...] = ( + Cell( + why="1 token: pre-tiled shared expert, last phase balanced without trailing FC2", + spec=dict(tokens=1), + ), + Cell( + why="2 tokens: pre-tiled shared expert, the one build without AR_PRE", + spec=dict(tokens=2), + ), + Cell( + why="3 tokens: 4 FC2-B builder warps, the shared expert from swizzled scales", + spec=dict(tokens=3), + ), + Cell( + why="4 tokens: one token per routing warp, logits replicated 128x", spec=dict(tokens=4) + ), + Cell( + why="5 tokens: 5 builder warps, a routing warp takes two tokens, logits 32x", + spec=dict(tokens=5), + ), + Cell(why="6 tokens: one request verifying a 5-token draft", spec=dict(tokens=6)), + Cell(why="7 tokens: the B2IMG global FC1 -> FC2 hand-off", spec=dict(tokens=7)), + Cell( + why="8 tokens: B2IMG with a 6-stage ring, 32 distinct experts at most", + spec=dict(tokens=8), + ), + ) + + note = """ + The kernel's elementwise math uses the PTX approximations (rcp.approx, ex2.approx, tanh) where `reference` uses + torch's correctly rounded ones, and its GEMMs sum in tensor-core order where `reference` sums in fp64: a + quantization boundary (NVFP4 activation / intermediate code, routing near-tie) can fall on the other side, so the + two agree within the band the test derives, not bit for bit. The routed part is bit for bit the stock TRTLLM-Gen + NVFP4 MoE (fp4_quantize + fp4_block_scale_moe_runner) on the same routing; the test checks that too. + + Not covered: the all-reduce push (`arp`), which needs a multicast exchange (the decode forward's boundary all-reduce + exercises it); the non-shared state (`shared=False`), which this target never builds. Every layer on one state + must run in one stream order: the workspace is shared and tagged per call, never reset. + + No `is_valid`: the compiled call checks every buffer's dtype, shape and alignment against its signature and + raises on a mismatch (a misaligned `hidden`, an `out` that is not bf16 [M, 6144], a malformed `arp`). + """ + + def __call__( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + return layer.state.run(layer, hidden, out, arp=arp) + + def reference( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """Plain torch over the layer's buffers (see the class docstring); `out` and `arp` are ignored.""" + return moe_reference(hidden, layer, shared=layer.state.shared) + + +m3_moe_engine = M3MoeEngine() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_router_tc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_router_tc.py new file mode 100644 index 000000000000..6e0a44f38b30 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/moe/m3_router_tc.py @@ -0,0 +1,119 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TF32 tensor-core router of the unfused decode MoE (9..16 generation rows), optionally with the NVFP4 activation.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +_E4M3_MAX = 448.0 +_E2M1_MAX = 6.0 +#: Midpoints between consecutive e2m1 magnitudes; ``_TIE_UP[i]``: round-to-nearest-even takes the upper code. +_E2M1_MIDPOINTS = (0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0) +_TIE_UP = (False, True, False, True, False, True, False) + + +def nvfp4_quantize( + x: torch.Tensor, sf_scale: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """``fp4_quantize(x, sf_scale, 16, False, False)``: (packed e2m1 [M, K / 2], linear E4M3 scales [M * K / 16], + the pre-rounding scaled values [M, K]). + + Per 16 columns: scale = e4m3(sf_scale * amax / 6), data = x * sf_scale / scale rounded to nearest-even e2m1. + """ + gs = sf_scale.float().reshape(()) + m, k = x.shape + blocks = x.float().view(m, k // 16, 16) + vecmax = blocks.abs().amax(-1) + sf = (gs * (vecmax / _E2M1_MAX)).clamp(max=_E4M3_MAX).to(torch.float8_e4m3fn) + out_scale = torch.where(vecmax != 0, gs / sf.float(), torch.zeros_like(vecmax)) + scaled = (blocks * out_scale.unsqueeze(-1)).view(m, k) + mid = torch.tensor(_E2M1_MIDPOINTS, device=x.device, dtype=torch.float32) + tie_up = torch.tensor(_TIE_UP, device=x.device, dtype=torch.bool) + a = scaled.abs().unsqueeze(-1) + code = ((a > mid).sum(-1) + ((a == mid) & tie_up).any(-1)).to(torch.uint8) + negative = (scaled < 0) | ((scaled == 0) & torch.signbit(scaled)) + code = code | (negative.to(torch.uint8) << 3) + packed = code[:, 0::2] | (code[:, 1::2] << 4) + return packed, sf.view(torch.uint8).reshape(-1), scaled + + +def tf32_rna(w: torch.Tensor) -> torch.Tensor: + """fp32 -> TF32 as ``cvt.rna.tf32.f32``: keep 10 mantissa bits, round to nearest with ties away from zero.""" + bits = w.contiguous().view(torch.int32) + rounded = (bits + 0x1000) & ~0x1FFF + finite = (bits & 0x7F800000) != 0x7F800000 + return torch.where(finite, rounded, bits).view(torch.float32) + + +class M3RouterTc(OpWrapper): + """Router logits on TF32 tensor cores for 128 experts and 1..48 tokens: ``x @ tf32(w)^T``, fp32 accumulate. + + ``x`` is BF16 (exact in TF32), ``w`` the fp32 router weight, rounded to TF32 in the kernel. The 16 K slices' + partials are summed in slice order whatever cluster size the device co-schedules (16 or 8 CTAs), so the result + does not depend on it. With ``sf_scale`` also the NVFP4 activation (``fp4_quantize``'s bytes): returns + ``[logits]`` or ``[logits, x_fp4, x_sf]``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The target routes the unfused MoE's pure generation steps of 9..16 rows here (ROUTER_TC_GEN), logits only. + CELLS: tuple[Cell, ...] = ( + Cell( + why="9 rows: a 3-request verify past the MoE engine", spec=dict(m=9, k=6144, fp4=False) + ), + Cell(why="10 rows", spec=dict(m=10, k=6144, fp4=False)), + Cell(why="11 rows", spec=dict(m=11, k=6144, fp4=False)), + Cell(why="12 rows: 3 requests x 4 tokens", spec=dict(m=12, k=6144, fp4=False)), + Cell(why="13 rows: a partial second token tile", spec=dict(m=13, k=6144, fp4=False)), + Cell(why="14 rows", spec=dict(m=14, k=6144, fp4=False)), + Cell(why="15 rows", spec=dict(m=15, k=6144, fp4=False)), + Cell( + why="16 rows: 4 requests x 4 tokens, two full token tiles", + spec=dict(m=16, k=6144, fp4=False), + ), + ) + + note = """ + The kernel uses 16-CTA clusters when the device co-schedules eight of them and 8-CTA clusters otherwise; both + sum the same 16 slice partials in the same order. Only the one the test device picks is driven. + """ + + def __call__( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_router_tc(x, w, sf_scale, pdl) + + def reference( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> tuple[torch.Tensor, ...]: + """fp32 product of ``x`` and the TF32-rounded weight, plus the NVFP4 recipe with ``sf_scale``.""" + with true_fp32_matmul(): + logits = x.float() @ tf32_rna(w).t() + if sf_scale is None: + return (logits,) + packed, sf, _ = nvfp4_quantize(x, sf_scale) + return logits, packed, sf + + def is_valid( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> None: + assert x.data_ptr() % 16 == 0, "x must start 16-byte aligned (cp.async.bulk source)" + if sf_scale is not None: + value = float(sf_scale.reshape(())) + assert value > 0.0 and value != float("inf"), ( + f"sf_scale must be positive and finite, got {value}" + ) + + +m3_router_tc = M3RouterTc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/norm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/norm/__init__.py new file mode 100644 index 000000000000..acec8fdbb26f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/norm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Normalization entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/norm/m3_embedding_firstnorm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/norm/m3_embedding_firstnorm.py new file mode 100644 index 000000000000..7ba04c92eb61 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/norm/m3_embedding_firstnorm.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-step embedding gather and the first layer's input RMSNorm in one CuTe DSL launch, into caller buffers.""" + +from __future__ import annotations + +from typing import Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.embedding_firstnorm.op import DecodeEmbedding, m3_embed_norm + +__all__ = ["DecodeEmbedding", "m3_embedding_firstnorm"] + +VOCAB = 200064 # MiniMax-M3's vocabulary; the decode table is the full (replicated) embedding + + +class M3EmbeddingFirstnorm(OpWrapper): + """Write ``table[ids]`` into ``raw`` (bf16 [M, 6144], a zero row for an id outside ``[0, V)``) and + ``raw * rsqrt(mean(raw^2) + eps) * weight`` into ``out``; returns ``(out, raw)`` -- the first layer's normed input + and its residual. ``ids`` int32 or int64 [M], M = 1..8; ``table`` bf16 [V, 6144] contiguous; ``weight`` bf16 + [6144]. The first call per (M, id dtype, table shape, PDL) compiles (``DecodeEmbedding`` does it at build).""" + + ARCHS = frozenset({Arch.SM_107}) + + # The target builds every (rows 1..8, id dtype) owner; the kernel is one CTA per row, so the rows differ only in + # grid size: the ends of the range and both id dtypes. + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, int32 ids", spec=dict(rows=1, id_dtype=torch.int32)), + Cell(why="eight requests, int32 ids", spec=dict(rows=8, id_dtype=torch.int32)), + Cell(why="one request, int64 ids", spec=dict(rows=1, id_dtype=torch.int64)), + Cell( + why="six rows (one request verifying a draft), int64 ids", + spec=dict(rows=6, id_dtype=torch.int64), + ), + # Every other row count the decode dispatch takes (modeling.EMBED_FIRSTNORM), both id dtypes. + *( + Cell( + why=f"{rows} rows: a decode row count between the ones above", + spec=dict(rows=rows, id_dtype=torch.int32 if rows % 2 else torch.int64), + ) + for rows in (2, 3, 4, 5, 7) + ), + ) + + note = """ + The gathered rows are exact. The norm is flashinfer's RMSNorm arithmetic in its reduction order with a fast + rsqrt, where `reference` sums in fp64 and takes torch's rsqrt: within the band the test derives, not bit for bit. + An id outside [0, V) gives a zero row and a zero norm (torch's embedding would raise). No accepted input is known + to compute wrongly: the host side rejects any other dtype, shape or a strided buffer. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + return m3_embed_norm(ids, table, weight, eps, raw, out, use_pdl) + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """New tensors (``raw`` / ``out`` are not written); `use_pdl` is ignored.""" + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] * valid.unsqueeze(1).to( + table.dtype + ) + x = rows.double() + rstd = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) + return (x * rstd * weight.double()).to(torch.bfloat16), rows + + +m3_embedding_firstnorm = M3EmbeddingFirstnorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/sampling/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/sampling/__init__.py new file mode 100644 index 000000000000..d72c4575bfb6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/sampling/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampling entries this target alone calls: the spec sampler's top-p renorm override (TOPP_DET).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/sampling/m31_topp_det.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/sampling/m31_topp_det.py new file mode 100644 index 000000000000..6f2a848289d7 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/sampling/m31_topp_det.py @@ -0,0 +1,271 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TOPP_DET: the spec sampler's top-p renorm as one persistent C++ launch, exact vs FlashInfer's deterministic +``top_p_renorm_probs`` (``trtllm::minimax_m31_topp_renorm_det``, kernels/minimaxM31/toppRenormDetSm103.cu). + +TOPP_DET v3 (``trtllm::minimax_m31_topp_renorm_det_v3``, kernels/minimaxM31/toppRenormDetV3Sm103.cu) is the same +renorm, bitwise v1.2's output, with the redundant per-CTA selection and a programmatic dependent launch. It is taken +when it is built unless +``TRTLLM_M31_DISABLE_TOPPDET_V3`` is set; v1.2 otherwise. ``TRTLLM_M31_DISABLE_TOPPDET_T3`` keeps v3 on its global +radix passes (no candidate list). + +V4K8 (a library whose op takes ``list_path`` / ``wide_hist``, default on): the one-sync list path (pass 0 lists the +elements >= 2^-12; when the selected bucket lies at or above that edge every CTA finishes the selection, FlashInfer's +chain sums and the write from the list after one barrier) and 8 privatized pass histograms, both bitwise the same +output; ``TRTLLM_M31_DISABLE_TOPPDET_V4`` / ``TRTLLM_M31_DISABLE_TOPPDET_K8`` turn them off. + +The target installs ``top_p_renorm_det`` as the sampler's top-p renorm override +(``sampler/ops/flashinfer.set_top_p_renorm_override``); it returns None, and the sampler keeps the stock FlashInfer +call, whenever the op is not built or the input is outside the kernel's contract. + +Workspace: one zero-initialized byte buffer per device, ``max_rows()`` rows of +``minimax_m31_topp_renorm_det_ws_bytes()``, kept for the life of the process (CUDA graphs captured with it keep its +address). The kernel leaves it ready for the next launch, so it is zeroed once. ``prebuild`` allocates it outside any +CUDA-graph capture (the target calls it from ``cache_derived_state``, before the executor's warmup and captures); a +call that finds no workspace allocates one when the stream is not capturing, and inside a capture returns None +(stock FlashInfer for that call) rather than allocating from the graph's private pool. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled + +# The largest CTA count per row (the persistent grid is rows x ctas_per_row 1024-thread CTAs, all co-resident). +MAX_CTAS_PER_ROW = 16 + +# The rows the workspace holds and the override takes with v3 / V4K8: up to the TP2 WIDE K7 c24 calls (192 rows). The +# VR microbench (rows 9..192 x peaked / randn / near_uniform) measured v3 and V4K8 bitwise FlashInfer's det mode +# on every cell and V4K8 faster on every cell (by >= 67 us per call; e.g. 64 rows 208.5 -> 44.5 us peaked, 192 rows +# 438.8 -> 156.6). The workspace is ~0.2 MB per row (~40 MB per device at 192 rows). +MAX_ROWS = 192 +# v1.2 keeps the TP8 / TP4 K7 c1 calls (the draft's 7 rows and the verify's 8), where GB300 microbenches measured it +# faster than stock FlashInfer (54.3 vs 104.2 us per call at 7 peaked rows, 54.8 vs 105.1 at 8). At +# 56 / 64 rows v1 was slower than FlashInfer's default mode (340 vs 187 / 199 us) and v1.2 is not +# re-measured there, so larger calls keep the stock kernel when v3 is not engaged. +V12_MAX_ROWS = 8 + +# The op when it is built, decided once per process. +_STATE: dict = {} +# Per device index: the persistent workspace and the SM count. +_WORKSPACES: dict = {} +_SMS: dict = {} + + +def _load_op(name: str): + """``torch.ops.trtllm.`` when this build carries it, else None.""" + return getattr(torch.ops.trtllm, name, None) + + +def topp_det_op(): + """The renorm op this process uses, decided once: ``trtllm::minimax_m31_topp_renorm_det_v3`` when it is built + and TOPPDET_V3 is not disabled, else ``trtllm::minimax_m31_topp_renorm_det`` when built, else None.""" + if "op" not in _STATE: + op, ws, v3 = None, 0, False + if not m31_disabled("TOPPDET_V3"): + op = _load_op("minimax_m31_topp_renorm_det_v3") + if op is not None: + v3 = True + ws = int(torch.ops.trtllm.minimax_m31_topp_renorm_det_v3_ws_bytes()) + logger.info("[M3.1] TOPP_DET v3 private op engaged (in-tree)") + if op is None: + op = _load_op("minimax_m31_topp_renorm_det") + ws = ( + int(torch.ops.trtllm.minimax_m31_topp_renorm_det_ws_bytes()) + if op is not None + else 0 + ) + _STATE["op"], _STATE["ws_bytes"], _STATE["v3"] = op, ws, v3 + # v3's candidate-list radix passes (bitwise the same output) unless TOPPDET_T3 is disabled. + _STATE["cand_list"] = v3 and not m31_disabled("TOPPDET_T3") + # V4K8 (a library whose schema carries list_path / wide_hist): the one-sync list path and 8 privatized pass + # histograms, both bitwise the same output, unless TOPPDET_V4 / TOPPDET_K8 is disabled. + v4k8 = v3 and "list_path" in str(op.default._schema) + _STATE["v4k8"] = v4k8 + _STATE["list_path"] = v4k8 and not m31_disabled("TOPPDET_V4") + _STATE["wide_hist"] = v4k8 and not m31_disabled("TOPPDET_K8") + if v3: + logger.info( + "[M3.1] TOPP_DET v3 T3 candidate-list fused passes: " + f"{'on' if _STATE['cand_list'] else 'off'} (cap 4096)" + ) + logger.info( + f"[M3.1] TOPP_DET V4K8: list path {'on' if _STATE['list_path'] else 'off'}, " + f"8 pass histograms {'on' if _STATE['wide_hist'] else 'off'}" + f"{'' if v4k8 else ' (library without V4K8)'}" + ) + if op is not None: + logger.info(f"[M3.1] TOPP_DET row cap: {MAX_ROWS if v3 else V12_MAX_ROWS} rows") + return _STATE["op"] + + +def is_v3() -> bool: + """Whether the engaged op is TOPP_DET v3.""" + topp_det_op() + return _STATE["v3"] + + +def max_rows() -> int: + """The rows the workspace holds and the override takes: MAX_ROWS with v3, V12_MAX_ROWS with v1.2.""" + return MAX_ROWS if is_v3() else V12_MAX_ROWS + + +def ws_bytes_per_row() -> int: + """Workspace bytes per row (0 when the op is not built).""" + topp_det_op() + return _STATE["ws_bytes"] + + +def sm_count(device: torch.device) -> int: + """The device's SM count, cached per device.""" + idx = torch.device(device).index + idx = torch.cuda.current_device() if idx is None else idx + if idx not in _SMS: + _SMS[idx] = torch.cuda.get_device_properties(idx).multi_processor_count + return _SMS[idx] + + +def ctas_per_row(rows: int, sms: int) -> int: + """The largest power of two <= MAX_CTAS_PER_ROW with ctas x rows <= sms (1 past that: one CTA per row needs no + co-residency).""" + g = MAX_CTAS_PER_ROW + while g > 1 and g * rows > sms: + g //= 2 + return g + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's zeroed workspace for max_rows() rows; must run outside a CUDA-graph capture. + Returns it, or None when the op is not built or the stream is capturing.""" + if topp_det_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.zeros( + max_rows() * ws_bytes_per_row(), dtype=torch.uint8, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31ToppDet(OpWrapper): + """FlashInfer's ``top_p_renorm_probs(probs, top_p, is_deterministic=True)`` of fp32 probs [rows, d] with fp32 + per-row top_p [rows], bit for bit on every row with top_p < 1, in ONE launch of rows x ctas_per_row CTAs + (FlashInfer's: five launches, apply on one CTA per row, slow on high-entropy rows). Rows with top_p >= 1 keep every + element (FlashInfer drops tail buckets there in ~43% of rows, in both of its modes); their output is + ``ftz(ftz(p) * norm)``. ``workspace`` is the persistent zeroed buffer of ``prebuild``. ``__call__`` returns None + (the caller keeps the stock call) for a non-fp32, non-contiguous or misaligned input, a width not divisible by 4, + a top_p that is not a contiguous fp32 [rows] tensor on probs' device, more rows than the workspace holds, or when + the op is not built.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The c1 K7 calls a microbench gated bitwise on GB300 (8 row classes x 4 top_p modes x 16 seeds); the + # sm_107 build of the same source is gated by the same MB on VR before the target takes it. + CELLS: tuple[Cell, ...] = ( + Cell(why="the K7 draft's 7 rows at c1 (vocab 200064)", spec=dict(rows=7, d=200064)), + Cell(why="the K7 verify's 8 rows at c1 (vocab 200064)", spec=dict(rows=8, d=200064)), + Cell(why="one row (the smallest grid, 16 CTAs)", spec=dict(rows=1, d=200064)), + Cell(why="TP2 WIDE K7 c8: 64 rows, 2 CTAs per row", spec=dict(rows=64, d=200064)), + Cell(why="TP2 WIDE K7 c24: 192 rows, 1 CTA per row", spec=dict(rows=192, d=200064)), + ) + + note = """ + Exact by construction: the det histograms are integer (any aggregation order is FlashInfer's), the per-pass + selection is FlashInfer's last-block code on them, and the kept sum is FlashInfer's 1024 per-thread chains into + the same cub BlockReduce. Built with --use_fast_math, as FlashInfer's JIT builds it. + """ + + def __call__( + self, + probs: torch.Tensor, + top_p: torch.Tensor, + workspace: Optional[torch.Tensor], + ctas_per_row: int, + ) -> Optional[torch.Tensor]: + if ( + not isinstance(top_p, torch.Tensor) + or workspace is None + or not probs.is_cuda + or probs.dtype != torch.float32 + or probs.dim() != 2 + or not probs.is_contiguous() + or probs.shape[0] < 1 + or probs.shape[1] % 4 != 0 + or probs.data_ptr() % 16 != 0 + or top_p.dtype != torch.float32 + or top_p.dim() != 1 + or top_p.shape[0] != probs.shape[0] + or not top_p.is_contiguous() + or top_p.device != probs.device + or workspace.device != probs.device + or ctas_per_row < 1 + or 1024 % ctas_per_row != 0 + ): + return None + op = topp_det_op() + if ( + op is None + or workspace.numel() * workspace.element_size() < probs.shape[0] * ws_bytes_per_row() + ): + return None + out = torch.empty_like(probs) + if is_v3(): + if _STATE["v4k8"]: + op( + probs, + top_p, + workspace, + out, + int(ctas_per_row), + True, + _STATE["cand_list"], + _STATE["list_path"], + _STATE["wide_hist"], + ) + else: + op(probs, top_p, workspace, out, int(ctas_per_row), True, _STATE["cand_list"]) + else: + op(probs, top_p, workspace, out, int(ctas_per_row)) + return out + + def reference(self, probs, top_p, workspace=None, ctas_per_row=None) -> torch.Tensor: + """FlashInfer's deterministic renorm (rows with top_p < 1; workspace and ctas_per_row are ignored).""" + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + def is_valid(self, probs, top_p, workspace=None, ctas_per_row=None, **kwargs) -> None: + assert probs.dtype == torch.float32 and probs.dim() == 2, "probs: fp32 [rows, d]" + + +m31_topp_det = M31ToppDet() + + +def top_p_renorm_det(probs: torch.Tensor, top_p: torch.Tensor) -> Optional[torch.Tensor]: + """The sampler's top-p renorm override: the det renorm of probs [rows, d] at per-row top_p, or None (the sampler + runs stock FlashInfer) when the op is not built, rows exceed max_rows(), no workspace can be had (first use inside + a capture) or the input is outside the contract.""" + if not isinstance(top_p, torch.Tensor) or not probs.is_cuda or probs.dim() != 2: + return None + rows = probs.shape[0] + if rows < 1 or topp_det_op() is None or rows > max_rows(): + return None + workspace = prebuild(probs.device) + if workspace is None: + return None + return m31_topp_det(probs, top_p, workspace, ctas_per_row(rows, sm_count(probs.device))) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/__init__.py new file mode 100644 index 000000000000..d25c41dc9557 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Speculative-decoding entries this target alone calls: the Eagle3 verify ops, the capture / fc_norm RMSNorms and +the step-input staging.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_capture_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_capture_norm.py new file mode 100644 index 000000000000..be5aa46fe774 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_capture_norm.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Eagle3 hidden-state capture with the drafter's fc_norm folded in (``m3_spec_tail.capture_norm``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_chunk_norm import rmsnorm_rows + + +class M3CaptureNorm(OpWrapper): + """``dst [M, H] = rmsnorm(bf16(h + r)) * w`` (``r`` optional), the rounding chunk_norm applies, into a capture + buffer slice whose row stride may exceed H. Returns ``dst``.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The target's captured layers (hidden 6144) into the 3-layer capture buffer (row stride 3 x 6144), over the + # verify rows of draft_len 3 (4 .. 32 rows); the residual is present for every captured decoder layer. + CELLS: tuple[Cell, ...] = ( + Cell( + why="4 verify rows of one request, layer slot 0, residual added", + spec=dict(m=4, slot=0, residual=True), + ), + Cell( + why="cap 8 (32 rows) into the last layer slot", spec=dict(m=32, slot=2, residual=True) + ), + Cell( + why="no residual (a capture point without one): plain RMSNorm", + spec=dict(m=8, slot=1, residual=False), + ), + ) + + note = """ + fp32 statistic (block sum, rsqrtf), one bf16 rounding after the bf16 sum h + r; the band states the occasional + one-ulp move against torch's statistic. + """ + + def __call__( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_capture_norm(h, r, w, eps, dst) + return dst + + def reference( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + """bf16 sum, then the normalized row; ``dst`` only says where the op writes.""" + x = (h.float() + r.float()).to(torch.bfloat16) if r is not None else h + return rmsnorm_rows(x, w, eps) + + def is_valid( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> None: + # The norm weight is read as H dense bf16 values; a strided view (a column of a stacked weight) is read + # past its elements without an error. + assert w.is_contiguous(), "w must be dense [H]" + + +m3_capture_norm = M3CaptureNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_chunk_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_chunk_norm.py new file mode 100644 index 000000000000..6699a55fc3e6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_chunk_norm.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 fc_norm over the captured layers in one launch: per-chunk RMSNorm of the concatenated hidden states +(``m3_spec_tail.chunk_norm``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3ChunkNorm(OpWrapper): + """``dst[r, c*H:(c+1)*H] = rmsnorm(src[r, c*H:(c+1)*H]) * w[c]`` for the C chunks of ``w [C, H]``; ``src`` may + have a row stride wider than C * H (the capture buffer). Writes ``dst [M, C*H]`` and returns it.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The Eagle3 capture buffer of the target (3 captured layers x hidden 6144, buffer rows wider than the layers + # when the final hidden is also captured), over the verify rows of draft_len 3: 4 / 8 / 32 rows. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request's 4 verify rows, dense [4, 3 x 6144]", + spec=dict(m=4, c=3, h=6144, extra_cols=0), + ), + Cell( + why="cap 8 (32 rows) read from the capture buffer with a fourth (final-hidden) slot in the row stride", + spec=dict(m=32, c=3, h=6144, extra_cols=6144), + ), + ) + + note = """ + fp32 statistic per chunk (block sum, rsqrtf), one bf16 rounding: an output moves by one bf16 ulp now and then + against torch's statistic; the cells' band states it. + """ + + def __call__( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + torch.ops.trtllm.minimax_m3_chunk_norm(src, w, eps, dst, pdl) + return dst + + def reference( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + """Each chunk normalized on its own; ``dst`` only says where the op writes, ``pdl`` shapes the launch.""" + c, h = w.shape + x = src[:, : c * h].reshape(src.shape[0], c, h) + return rmsnorm_rows(x, w, eps).reshape(src.shape[0], c * h) + + def is_valid( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ) -> None: + c, h = w.shape + assert w.is_contiguous(), "w must be dense [C, H] (chunk c read at c * H)" + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + + +m3_chunk_norm = M3ChunkNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_input.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_input.py new file mode 100644 index 000000000000..8f0f69ee812b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_input.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 draft pass's input in one launch: embedding lookup, both RMSNorms and the concat +(``m3_spec_tail.draft_input``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3DraftInput(OpWrapper): + """``out [M, 2H] = [rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden]`` and ``residual [M, H] = + hidden[rows]`` (rows default to 0..M-1) in one launch; ids outside the table embed to zero (the masked lookup of + a vocab-local embedding). Returns ``(out, residual)``.""" + + ARCHS = frozenset({Arch.SM_107}) + + # Every pass of the fused draft loop (modeling.DRAFT_INPUT) at draft_len 3, drafter hidden 6144, eps 1e-6: the + # first pass of 1..8 requests (4..32 rows, the carried hidden read in place) and the later passes of 1..8 requests + # (one row each; the second pass gathers the last accepted rows from the first pass's 4 rows per request). + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"first pass of {n} request(s): {4 * n} rows, hidden read in place", + spec=dict(m=4 * n, h=6144, gather=False, invalid_id=False), + ) + for n in range(1, 9) + ), + *( + Cell( + why=f"second pass of {n} request(s): the last accepted rows gathered from {4 * n} rows", + spec=dict(m=n, h=6144, gather=True, invalid_id=False), + ) + for n in range(1, 9) + ), + Cell( + why="third pass of 3 requests: one row each, hidden read in place", + spec=dict(m=3, h=6144, gather=False, invalid_id=False), + ), + Cell( + why="one row with an out-of-table id: the embedding half is zero", + spec=dict(m=1, h=6144, gather=False, invalid_id=True), + ), + ) + + note = """ + The arithmetic is the in-tree RMSNorm's (fp32 statistic, (x * rsqrt) * w, one bf16 rounding); the statistic's + summation order differs from FlashInfer's, which moves an output by one bf16 ulp now and then (the test's band, + also held against the in-tree modules). The out-of-table id is the masked lookup of a vocab-local embedding; the + drafter's embedding is the whole table, so its ids are always in range. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + torch.ops.trtllm.minimax_m3_draft_input( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, pdl + ) + return out, residual + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + """The two normalized halves (fp32 statistic, one bf16 rounding) and the gathered hidden rows. ``out`` / + ``residual`` only say where the op writes; ``pdl`` only shapes the launch.""" + valid = (ids >= 0) & (ids < table.shape[0]) + emb = rmsnorm_rows(table[torch.where(valid, ids, 0).long()], w_embed, eps_embed) + emb = torch.where(valid[:, None], emb, torch.zeros_like(emb)) + hid = hidden[rows.long()] if rows is not None else hidden[: ids.shape[0]] + ref = torch.cat([emb, rmsnorm_rows(hid, w_hidden, eps_hidden)], dim=-1) + return ref, (hid.clone() if residual is not None else None) + + def is_valid( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> None: + h = table.shape[1] + # Rows are read as hidden + rows[r] * H and table + id * H: both must be dense H-wide rows. + assert table.is_contiguous() and hidden.is_contiguous(), ( + "table / hidden must be dense [*, H]" + ) + if rows is not None: + assert bool(((rows >= 0) & (rows < hidden.shape[0])).all()), "rows must index hidden" + else: + assert hidden.shape[0] >= ids.shape[0], "hidden must hold a row per id" + # Vector loads of 16 bytes: H must be a multiple of 8 within the register tile, and the weights aligned. + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + assert residual is None or ( + residual.is_contiguous() and residual.shape == (ids.shape[0], h) + ), "residual must be contiguous [M, H]" + + +m3_draft_input = M3DraftInput() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_silu.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_silu.py new file mode 100644 index 000000000000..75626e570244 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_silu.py @@ -0,0 +1,58 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter MLP's plain SwiGLU as one C++ launch (``modeling.DRAFT_SILU_CPP``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3DraftSilu(OpWrapper): + """``silu(g) * u`` of a bf16 [rows, 2d] gate | up tensor, rounded once to bf16: + trtllm::minimax_m3_sm107_tp4_silu_and_mul (minimaxM3Sm107Tp4/siluAndMul.cu), bit for bit the Triton + ``torch.ops.trtllm.silu_and_mul`` at its defaults (no limit, alpha 1, beta 0), the activation the drafter's GatedMLP + runs. Contiguous, 16-byte aligned rows and d a multiple of 8; ``__call__`` returns None otherwise (the caller runs + the Triton op).""" + + ARCHS = frozenset({Arch.SM_107}) + + # The TP4 drafter MLP shard: intermediate 18432 / 4 = 4608, gate | up 9216 wide, at the drafter's decode rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="the TP4 drafter MLP at 1 row", spec=dict(rows=1, d=4608)), + Cell(why="the TP4 drafter MLP at 4 rows (1 request x 4 tokens)", spec=dict(rows=4, d=4608)), + Cell(why="the TP4 drafter MLP at 8 rows", spec=dict(rows=8, d=4608)), + ) + + note = """ + Elementwise, one rounding: bit for bit the Triton kernel (the same PTX instruction sequence), checked over every + bf16 gate and every bf16 up pattern. + """ + + def __call__(self, x: torch.Tensor) -> Optional[torch.Tensor]: + if ( + x.dtype != torch.bfloat16 + or x.dim() != 2 + or not x.is_cuda + or not x.is_contiguous() + or x.shape[1] % 16 != 0 + or x.data_ptr() % 16 != 0 + or x.shape[0] > 65535 + ): + return None + out = torch.empty((x.shape[0], x.shape[1] // 2), dtype=x.dtype, device=x.device) + torch.ops.trtllm.minimax_m3_sm107_tp4_silu_and_mul(x, out) + return out + + def reference(self, x: torch.Tensor) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul(x) + + def is_valid(self, x: torch.Tensor) -> None: + assert x.dtype == torch.bfloat16 and x.dim() == 2, "x: bf16 [rows, 2d]" + + +m3_draft_silu = M3DraftSilu() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_step.py new file mode 100644 index 000000000000..5dbb324a7d50 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_draft_step.py @@ -0,0 +1,417 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One pass of the fused greedy Eagle3 draft loop: the drafter's vocab-sharded BF16 head (streamed from its lossless +12-bit coding, ``ops.gemm.m3_target_head``, or as BF16 fragments), the cross-rank argmax and the loop's per-pass +bookkeeping in one kernel (``m3_draft_step.draft_step``).""" + +from __future__ import annotations + +import functools +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +from ..gemm.m3_target_head import decode as decode_coded +from .m3_shard_accept import exchange_tokens, local_argmax, make_keys + +#: Requests (head rows) per pass, and exchange rows per call half of the workspace. +MAX_TOKENS = 8 +#: 16-row weight tiles one CTA can own (the kernel's shared-memory accumulator). +MAX_TILES_PER_CTA = 32 +KERNEL_WARPS = 16 # warps per CTA (draftStep.h kWarps) + + +@functools.lru_cache(maxsize=64) +def max_tile_pieces(n_tiles: int, num_ctas: int, chunks_per_tile: int) -> int: + """The most warps that stream one weight tile under the kernel's whole-tile split: CTA c owns tiles + [c * n / grid, (c + 1) * n / grid), and its (tile, chunk) range is cut into KERNEL_WARPS equal contiguous + segments. The shared-memory atomics add a tile's pieces in arrival order: deterministic for at most 2 pieces (two + addends from 0 commute), not for more (fixed_order then sums them in warp order).""" + worst = 1 + for c in range(num_ctas): + lo, hi = c * n_tiles // num_ctas, (c + 1) * n_tiles // num_ctas + span = (hi - lo) * chunks_per_tile + bounds = [w * span // KERNEL_WARPS for w in range(KERNEL_WARPS + 1)] + for t in range(hi - lo): + a, b = t * chunks_per_tile, (t + 1) * chunks_per_tile + worst = max( + worst, sum(1 for w in range(KERNEL_WARPS) if bounds[w] < b and bounds[w + 1] > a) + ) + return worst + + +#: Draft length of the fused loop, and the drafter's TP4 head shard (200064-id vocabulary) and hidden size. +DRAFT_LEN = 3 +DRAFT_SHARD = 50016 +HIDDEN = 6144 + + +def bf16_fragments(weight: torch.Tensor) -> torch.Tensor: + """BF16 [N, K] -> the same values in A-fragment order [N / 16, K / 16, 256]. + + One lane's 16 bytes of a 16-row tile's k16 step are the m16n8k16 A fragment: element (row h * 8 + g, k ks * 16 + + hk * 8 + jj * 2 + e) goes to [tile][ks][g][jj][hk][h][e] (lane = g * 4 + jj). + """ + n, k = weight.shape + assert n % 16 == 0 and k % 16 == 0 and weight.dtype == torch.bfloat16, (n, k, weight.dtype) + q = weight.view(n // 16, 2, 8, k // 16, 2, 4, 2).permute(0, 3, 2, 5, 4, 1, 6).contiguous() + return q.view(n // 16, k // 16, 256) + + +def from_fragments(wf: torch.Tensor) -> torch.Tensor: + """The inverse of ``bf16_fragments``: [N / 16, K / 16, 256] -> [N, K].""" + tiles, ks, _ = wf.shape + return ( + wf.view(tiles, ks, 8, 4, 2, 2, 2).permute(0, 5, 2, 1, 4, 3, 6).reshape(tiles * 16, ks * 16) + ) + + +def head_weight( + wf: torch.Tensor, + k: int, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The BF16 head shard [N, K] the kernel multiplies by: the fragment-ordered copy, or (``exc_off`` given) the + 12-bit coded stream with its exception lists decoded.""" + if exc_off is None: + return from_fragments(wf) + n = (exc_off.numel() - 1) * 16 + return decode_coded(wf, exp_base, exc_off, exc_pos, exc_w, n, k) + + +def shard_logits( + x: torch.Tensor, + wf: torch.Tensor, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The in-tree head's BF16 logits of the rows x [M, K] against the shard (``head_weight``): the fp32 product, + rounded once to BF16 (the BF16 GEMM's output), as fp32.""" + w = head_weight(wf, x.shape[1], exc_off, exc_pos, exc_w, exp_base) + with true_fp32_matmul(): + logits = x.float() @ w.float().t() + return logits.to(torch.bfloat16).float() + + +class M3DraftStep(OpWrapper): + """One fused greedy draft pass over ``rows <= 8`` requests (one head row each). + + Head input ``x[x_rows[b]]`` (pass 0: the last accepted verify row of each request) or ``x[b]``. The head is this + rank's BF16 lm_head shard in fragment order (``m3_draft_step.bf16_fragments``, global ids ``vocab_offset ..``), + or with ``exc_off`` / ``exc_pos`` / ``exc_w`` / ``exp_base`` its lossless 12-bit coding (``wf`` = the uint8 stream + of ``m3_target_head.encode_chunked``, which must round-trip to the BF16 shard); each logit is the fp32 dot product + rounded to BF16 and the row's token is the global argmax over all ranks' + shards (ties, -0 / +0 included, to the lowest global id), exchanged through the multicast slots + ``[2][8][world]``. The last CTA then writes ``draft_out[b, step]`` and ``ids_next[b]`` (the token), ``pos_next[b] + = pos_src[x_rows[b] or b] + 1``, and advances the draft KV lengths as the in-tree draft loop does between passes + (pass 0 with ``num_acc``: ``kv_lens -= draft_len - num_acc`` and ``seq_lens = 1``; later passes: ``kv_lens += + 1``). ``out_keys`` (optional) receives each row's winning key. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # Every pass of the fused draft loop (modeling.DRAFT_STEP): draft_len 3 at 1..8 requests, the drafter's + # full-vocabulary BF16 head (50016 rows per rank of 4, hidden 6144). Pass 0 reads the verify rows' hidden (4 per + # request) through x_rows and rolls the KV back; the later passes read one row per request. + _PASSES: tuple[Cell, ...] = ( + *( + Cell( + why=f"pass 0 at {n} request(s): head rows gathered from {4 * n} verify rows, KV rollback, seq_lens = 1", + spec=dict( + nreq=n, + draft_len=DRAFT_LEN, + step=0, + rank=n % 4, + world=4, + vs=DRAFT_SHARD, + k=HIDDEN, + ), + ) + for n in range(1, MAX_TOKENS + 1) + ), + Cell( + why="pass 1 at two requests: one row each, kv_lens + 1, rank 3's ids", + spec=dict( + nreq=2, draft_len=DRAFT_LEN, step=1, rank=3, world=4, vs=DRAFT_SHARD, k=HIDDEN + ), + ), + Cell( + why="last pass (step 2) at 8 requests, rank 0", + spec=dict( + nreq=8, draft_len=DRAFT_LEN, step=2, rank=0, world=4, vs=DRAFT_SHARD, k=HIDDEN + ), + ), + Cell( + why="pass 2 at 5 requests with exact cross-rank ties (the lower global id wins)", + spec=dict( + nreq=5, + draft_len=DRAFT_LEN, + step=2, + rank=1, + world=4, + vs=DRAFT_SHARD, + k=HIDDEN, + ties=True, + ), + ), + ) + # The coded head (the fused draft loop's default) at every pass shape, and the BF16 fragments (the coding's + # fail-closed path) at the first and last pass shapes and with ties. + CELLS: tuple[Cell, ...] = ( + *(Cell(why=f"coded: {c.why}", spec=dict(c.spec, coded=True)) for c in _PASSES), + *( + Cell(why=f"bf16: {c.why}", spec=dict(c.spec, coded=False)) + for c in (_PASSES[0], _PASSES[MAX_TOKENS - 1], _PASSES[-1]) + ), + ) + + note = """ + The head is the in-tree BF16 head's arithmetic (BF16 weights and activations, fp32 accumulation, a BF16 logit; + the coded head's weight is the BF16 shard bit for bit; its exceptions decode to +0 in the MMAs and their exact + products are added at the tile's flush) + and the argmax is the in-tree one over BF16 logits, so a token differs from the in-tree path's only where the fp32 + sum, accumulated in another order, rounds to the other side of a BF16 boundary and that decides the row's maximum. + The cells' rows are decidable and the peers' keys sit 0.25 away (or tie exactly); the edge tests (forced BF16 + ties, -0 / +0, top-2 one fp32 step apart that round to one BF16 value, adjacent BF16 values) use integer data + whose fp32 sums are exact in any order and hold the kernel to the in-tree path bit for bit. The cross-rank + exchange is certified on one device with the peers' keys written into the slots (see m3_shard_accept). + """ + + def __call__( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + fold: Optional[tuple] = None, + fixed_order: bool = False, + ) -> None: + # fold: (the 14 fold-tail tensors in buildParams' order, rows, tokens per step) -- the step's forward-tail store + # and next-step staging, run by the pass's last CTA after its tokens (unused by this target, which passes None). + tensors, fold_rows, fold_t = ( + (list(fold[0]), int(fold[1]), int(fold[2])) if fold is not None else (None, 0, 0) + ) + torch.ops.trtllm.minimax_m3_sm107tp4_draft_step( + x, + x_rows, + rows, + wf, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + num_ctas, + pdl, + out_keys, + exc_off, + exc_pos, + exc_w, + exp_base, + tensors, + fold_rows, + fold_t, + fixed_order, + ) + + def reference( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + fold: Optional[tuple] = None, + fixed_order: bool = False, + ) -> dict[str, torch.Tensor]: + """Every buffer the pass writes, computed on copies: "logits" (this rank's BF16 logits as fp32), "local_keys" + (this rank's key per row), "keys" (the winning key), "tokens", "draft_out", "ids_next", "pos_next", "kv_lens" + and "seq_lens". + + Reads ``ws_uc``, ``counters`` and ``kv_lens`` before the call. ``keys``, ``ws_mc_ptr``, ``num_ctas``, ``pdl``, + ``out_keys`` and ``fixed_order`` do not change the result; ``fold`` adds the fold tail's writes (m3_fold_tail), + not these. + """ + h = x[x_rows.long()] if x_rows is not None else x[:rows] + logits = shard_logits(h, wf, exc_off, exc_pos, exc_w, exp_base) + best, idx = local_argmax(logits, vocab_offset) + local_keys = make_keys(best, idx) + tokens = exchange_tokens(local_keys, ws_uc, counters, rank, world, MAX_TOKENS) + half = int(counters[1].item()) & 1 + slots = ws_uc.view(torch.int64)[ + half * MAX_TOKENS * world : (half * MAX_TOKENS + rows) * world + ] + slots = slots.view(rows, world).clone() + slots[:, rank] = local_keys + sign = torch.iinfo(torch.int64).min + win = (slots ^ sign).max(dim=1).values ^ sign + d_out = draft_out.clone() + d_out[:rows, step] = tokens + src = x_rows.long() if x_rows is not None else torch.arange(rows, device=x.device) + lens = kv_lens.clone() + sl = seq_lens.clone() if seq_lens is not None else None + if step == 0 and num_acc is not None: + lens[:rows] -= draft_len - num_acc[:rows] + sl[:rows] = 1 + elif step > 0: + lens[:rows] += 1 + out = { + "logits": logits, + "local_keys": local_keys, + "keys": win, + "tokens": tokens, + "draft_out": d_out, + "ids_next": tokens.clone(), + "pos_next": pos_src[src] + 1, + "kv_lens": lens, + } + if sl is not None: + out["seq_lens"] = sl + return out + + def is_valid( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + fold: Optional[tuple] = None, + fixed_order: bool = False, + ) -> None: + assert not bool(keys[:rows].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + # The kernel reads the head row x + x_rows[b] * K and the position pos_src[x_rows[b]] with no bound. + if x_rows is not None: + assert bool(((x_rows[:rows] >= 0) & (x_rows[:rows] < x.shape[0])).all()), ( + "x_rows must index x" + ) + assert bool((x_rows[:rows] < pos_src.shape[0]).all()), ( + "pos_src must cover the head rows" + ) + else: + assert pos_src.shape[0] >= rows, "pos_src must cover the head rows" + # Each CTA accumulates at most MAX_TILES_PER_CTA tiles in shared memory. + tiles = wf.shape[0] if exc_off is None else exc_off.numel() - 1 + assert -(-tiles // num_ctas) <= MAX_TILES_PER_CTA, "too many weight tiles per CTA" + # The atomics' arrival-order sum is deterministic only while no tile takes more than 2 warp pieces. + assert ( + fixed_order + or max_tile_pieces(tiles, num_ctas, x.shape[1] // (16 if exc_off is None else 64)) <= 2 + ), ( + "a tile takes more than 2 warp pieces on this grid: the atomics' sum is order-dependent (fixed_order)" + ) + if exc_off is not None: + # The exceptions are read per tile in list order with no bound against the coded stream. + assert int(exc_off[-1].item()) == exc_pos.numel(), ( + "exc_off must end at the exception count" + ) + assert bool((exc_off[1:] >= exc_off[:-1]).all()), "exc_off must be non-decreasing" + # A NaN logit maps to a key above every finite one and wins the row silently. + h = x[x_rows.long()] if x_rows is not None else x[:rows] + assert not bool(torch.isnan(h).any()), "NaN head inputs win the argmax" + if fold is not None: + assert len(fold) == 3 and len(fold[0]) == 14, ( + "fold is (the 14 fold-tail tensors, rows, tokens per step)" + ) + if num_acc is not None: + na = num_acc[:rows] + assert bool(((na >= 1) & (na <= draft_len + 1)).all()), ( + "num_acc counts must lie in [1, draft_len + 1]" + ) + + +m3_draft_step = M3DraftStep() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_force_vec.py new file mode 100644 index 000000000000..fcb327e374e3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_force_vec.py @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance, drawn as the in-tree worker draws them.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.m3_force_vec import force_vec_kernel + + +class M3ForceVec(OpWrapper): + """Bump ``counter`` (int64 [1]) by one, then for each of the ``batch_size`` requests ``b`` write + ``min(base_total + (pool[(c * counter_stride + b * slot_stride) & pool_mask] < frac), max_total)`` into ``out`` + (int32), ``c`` the bumped counter: the in-tree ``_apply_force_accepted_tokens`` draw over its fixed-seed fp32 + ``pool``. ``block`` is the launch width (a power of two >= batch_size; 16, 32 or 64 are compiled at warmup). + + Only the forced-acceptance testing knob (``TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`` with a fractional part) + runs it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, fraction 0.5", spec=dict(batch=1, block=16, frac=0.5)), + Cell(why="eight requests, fraction 0.3", spec=dict(batch=8, block=16, frac=0.3)), + ) + + note = """ + Integer arithmetic and one fp32 comparison: `reference` must match exactly. + """ + + def __call__( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + force_vec_kernel[(1,)]( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK=block, + ) + return out + + def reference( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + """The counts (a new tensor; ``counter`` and ``out`` are not written).""" + c = int(counter.item()) + 1 + slots = torch.arange(batch_size, dtype=torch.int64, device=pool.device) + idx = (c * counter_stride + slots * slot_stride) & pool_mask + r = pool[idx] + return torch.clamp(base_total + (r < frac).to(torch.int32), max=max_total).to(torch.int32) + + +m3_force_vec = M3ForceVec() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_force_vec_req.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_force_vec_req.py new file mode 100644 index 000000000000..5a2ffd8b6627 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_force_vec_req.py @@ -0,0 +1,62 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance, each request drawn at its own key.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.m3_force_vec import force_vec_req_kernel + + +class M3ForceVecReq(OpWrapper): + """For each of the ``batch_size`` requests ``b`` write ``min(base_total + (pool[(keys[b] * key_stride) & + pool_mask] < frac), max_total)`` into ``out`` (int32): the forced-acceptance draw from the fixed-seed fp32 ``pool`` + at the request's own key (int32 ``keys``, its KV length at this verify), so a request's draws depend on neither the + server's history nor the other requests. ``block`` is the launch width (a power of two >= batch_size; 16, 32 or + 64 are compiled at warmup). + + Only the forced-acceptance testing knob (``TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`` with a fractional part) + runs it, with ``FORCE_AL_PER_REQUEST``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, fraction 0.5", spec=dict(batch=1, block=16, frac=0.5)), + Cell(why="eight requests, fraction 0.3", spec=dict(batch=8, block=16, frac=0.3)), + ) + + note = """ + Integer arithmetic and one fp32 comparison: `reference` must match exactly. + """ + + def __call__( + self, pool, keys, out, batch_size, base_total, max_total, frac, key_stride, pool_mask, block + ): + force_vec_req_kernel[(1,)]( + pool, + keys, + out, + batch_size, + base_total, + max_total, + frac, + key_stride, + pool_mask, + BLOCK=block, + ) + return out + + def reference( + self, pool, keys, out, batch_size, base_total, max_total, frac, key_stride, pool_mask, block + ): + """The counts (a new tensor; ``out`` is not written).""" + idx = (keys[:batch_size].to(torch.int64) * key_stride) & pool_mask + r = pool[idx] + return torch.clamp(base_total + (r < frac).to(torch.int32), max=max_total).to(torch.int32) + + +m3_force_vec_req = M3ForceVecReq() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_sampled_accept.py new file mode 100644 index 000000000000..7623f5fe582b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_sampled_accept.py @@ -0,0 +1,289 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampled (temperature + top-p) Eagle3 verification over the TP-sharded vocabulary in one kernel +(``m3_sampled_accept.sampled_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_shard_accept import strict_accept + +#: Verify rows per call. +MAX_ROWS = 48 +#: The fused sampling kernel's 1 / T guard. +TEMP_EPSILON = 1e-6 + + +def nucleus_probs( + logits: torch.Tensor, temperatures: torch.Tensor, top_ps: torch.Tensor +) -> torch.Tensor: + """Per-row sampling distribution [rows, Vs] (fp32): w = exp(x / (T + 1e-6) - max), kept iff the mass strictly + above its weight is at most top_p * sum(w) (ties included), renormalized over the kept tokens.""" + inv = 1.0 / (temperatures.float() + TEMP_EPSILON) + x = logits.float() * inv[:, None] + w = torch.exp(x - x.max(dim=1, keepdim=True).values).double() + z = w.sum(dim=1, keepdim=True) + order = torch.sort(w, dim=1, descending=True) + sw = order.values + # mass strictly above each sorted weight: the cumulative sum before the first entry of its tie group + csum = torch.cumsum(sw, dim=1) - sw + first = torch.ones_like(sw, dtype=torch.bool) + first[:, 1:] = sw[:, 1:] != sw[:, :-1] + group = torch.cumsum(first.long(), dim=1) - 1 + starts = torch.zeros_like(csum) + starts.scatter_reduce_(1, group, csum, reduce="amin", include_self=False) + above = torch.gather(starts, 1, group) + keep_sorted = above <= top_ps.double()[:, None] * z + keep = torch.zeros_like(keep_sorted) + keep.scatter_(1, order.indices, keep_sorted) + p = torch.where(keep, w, torch.zeros_like(w)) + return (p / p.sum(dim=1, keepdim=True)).float() + + +class M3SampledAccept(OpWrapper): + """Sampled verification of ``rows = nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard. + + Each row draws one token from its top-p nucleus of softmax(logits / T) over the full (all-rank) vocabulary by + distributed rejection sampling (Philox keyed by the row's seed / offset, the row, the attempt and the CTA), then + the strict acceptance runs against ``draft`` as in ``m3_shard_accept``: ``acc_tokens``, ``num_acc``, + ``gather_ids``, optional ``next_new[:, 0]``. ``tokens`` receives the sampled token per row; ``extra_attempts`` + accumulates exchange rounds beyond the second. The exchange goes through the multicast workspace (two call + halves of ``minimax_m3_sm107tp4_sampled_accept_ws_words(world)`` words) with the (finished rows, call) ``counters``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # draft_len 3 (T = 4 rows per request) at the TP4 caps: 1 / 2 / 8 requests (4 / 8 / 32 rows, 8 CTAs per row) + # and the 48-row maximum (4 CTAs per row); the 50016-column target shard. Single rank (see note). + CELLS: tuple[Cell, ...] = ( + Cell( + why="cap 1 (4 rows), top_p 0.8 over a 12-token head: the draws follow the nucleus distribution", + spec=dict( + nreq=1, vs=50016, vocab_offset=0, check="distribution", top_p=0.8, temperature=0.7 + ), + ), + Cell( + why="cap 8 (32 rows, 256 co-resident CTAs): top_p ~ 0 keeps only the argmax, token exact", + spec=dict( + nreq=8, + vs=50016, + vocab_offset=2 * 50016, + check="argmax", + top_p=1e-7, + temperature=1.0, + ), + ), + Cell( + why="cap 2 (8 rows) with top_p = 1 (no cut: the first draw stands), next_new and forced counts", + spec=dict( + nreq=2, + vs=50016, + vocab_offset=50016, + check="distribution", + top_p=1.0, + temperature=1.0, + force="vec", + ), + ), + Cell( + why="48 rows (4 CTAs per row): the second launch geometry, nucleus distribution at top_p 0.5", + spec=dict( + nreq=12, + vs=50016, + vocab_offset=3 * 50016, + check="distribution", + top_p=0.5, + temperature=1.3, + force="total", + ), + ), + ) + + note = """ + What is sampled is gated statistically, the rest exactly. Every drawn token must lie in the reference nucleus; + the empirical distribution over many calls (distinct Philox offsets) must sit within a stated total-variation + band of the reference's; the acceptance outputs must equal the strict acceptance of the drawn tokens exactly. + The Philox stream itself is not reproduced in torch: the kernel's draw uses the hardware exp / log + approximations (__expf, __logf) and a CTA-order inverse CDF, which torch does not reproduce bit for bit. + + Certified at world 1 only. The decision is replicated from every rank's per-round records, which another rank + computes from its own shard on the fly, so peers cannot be stood in for on one device; the cross-rank path + needs the multi-device job and is not covered here. Rows with top-k or min-p must not reach this op (it applies + neither); is_valid states it. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + prewait_params: bool = True, + early_trigger: bool = False, + one_round_k: int = 0, + decided: Optional[torch.Tensor] = None, + ilp_walk: bool = False, + ) -> None: + torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept( + logits, + vocab_offset, + temperatures, + top_ps, + seeds, + offsets, + ws_uc, + ws_mc_ptr, + counters, + tokens, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + force_vec, + extra_attempts, + use_pdl, + prewait_params, + early_trigger, + one_round_k, + decided, + ilp_walk, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + prewait_params: bool = True, + early_trigger: bool = False, + one_round_k: int = 0, + decided: Optional[torch.Tensor] = None, + ilp_walk: bool = False, + ) -> dict: + """{"probs": the per-row nucleus distribution over this shard [rows, Vs] (global id = vocab_offset + + column), "accept": tokens -> the strict acceptance outputs}. The draw itself is random; ``seeds`` / + ``offsets`` select it and do not change the distribution. World 1: this shard is the whole vocabulary.""" + rows = logits.shape[0] + probs = nucleus_probs(logits, temperatures[:rows], top_ps[:rows]) + + def accept(drawn: torch.Tensor) -> dict[str, torch.Tensor]: + return strict_accept(drawn, draft, batch_idx, force_total, next_new, force_vec) + + return {"probs": probs, "accept": accept} + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + prewait_params: bool = True, + early_trigger: bool = False, + one_round_k: int = 0, + decided: Optional[torch.Tensor] = None, + ilp_walk: bool = False, + ) -> None: + rows = logits.shape[0] + # top-k and min-p are not applied: a row that has them gets a token outside its real sampling set. + if top_ks is not None: + k = top_ks[:rows] + assert bool(((k <= 0) | (k >= logits.shape[1] * world)).all()), ( + "top-k rows must take the generic path" + ) + if min_ps is not None: + assert not bool((min_ps[:rows] > 0).any()), "min-p rows must take the generic path" + # Rows are request-major verify rows: tokens[b * (D + 1) + k]; a mismatched layout pairs drafts with the + # wrong rows. + assert rows == draft.shape[0] * (draft.shape[1] + 1), ( + "logits rows must be nreq * (draft_len + 1)" + ) + # Per-row parameters are read at the row index, unchecked. + for t in (temperatures, top_ps, seeds, offsets): + assert t.numel() >= rows and t.is_contiguous(), ( + "per-row parameters must cover every row" + ) + assert temperatures.dtype == torch.float32 and top_ps.dtype == torch.float32, ( + "fp32 sampling parameters" + ) + assert seeds.dtype == torch.int64 and offsets.dtype == torch.int64, ( + "int64 Philox seeds / offsets" + ) + # The finished-row counter elects the CTA that runs the acceptance. + assert int(counters[0].item()) == 0, ( + "counters[0] (finished rows) must be zero between calls" + ) + + +m3_sampled_accept = M3SampledAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_sampled_accept_ws_words.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_sampled_accept_ws_words.py new file mode 100644 index 000000000000..9193b3dce61d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_sampled_accept_ws_words.py @@ -0,0 +1,59 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Size of the sampled-verify exchange workspace (``m3_sampled_accept.SampledAcceptWorkspace.create``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Exchange rounds per row, verify rows per call, CTAs per row (at most), uint64 words per record. +MAX_ATTEMPTS = 16 +MAX_ROWS = 48 +MAX_SPLITS = 8 +RECORD_WORDS = 4 +#: SA-1R (one_round_k 9 / 82): 9 listed bf16 values ride each record, 3 per extra word. +LIST_WORDS = {0: 0, 9: 3, 82: 3} + + +class M3SampledAcceptWsWords(OpWrapper): + """uint64 words of the multicast workspace ``minimax_m3_sm107tp4_sampled_accept`` exchanges through for + ``world`` ranks: two call halves of ``[attempts][rows][world * splits][record words]``.""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="the tp4 target's workspace", spec=dict(world=4)), + Cell(why="a single rank (local workspace, no multicast)", spec=dict(world=1)), + Cell( + why="the tp4 target's workspace with SA-1R pooled (82)", + spec=dict(world=4, one_round_k=82), + ), + Cell( + why="the tp4 target's workspace with SA-1R top-9 (9)", spec=dict(world=4, one_round_k=9) + ), + ) + + note = """ + A host computation (no device work); it is an op so the workspace size and the kernel's indexing come from one + definition. + """ + + def __call__(self, world: int, one_round_k: int = 0) -> int: + return torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept_ws_words(world, one_round_k) + + def reference(self, world: int, one_round_k: int = 0) -> int: + """Two halves of attempts x rows x (world x splits) records x record words.""" + return ( + 2 + * MAX_ATTEMPTS + * MAX_ROWS + * world + * MAX_SPLITS + * (RECORD_WORDS + LIST_WORDS[one_round_k]) + ) + + +m3_sampled_accept_ws_words = M3SampledAcceptWsWords() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_shard_accept.py new file mode 100644 index 000000000000..8beba4373b02 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_shard_accept.py @@ -0,0 +1,284 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Greedy Eagle3 verification from this rank's logits shard: per-row argmax, the cross-rank (value, index) exchange +and the strict acceptance in one kernel (``m3_shard_accept.shard_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Verify rows per call, and exchange rows per call half of its workspace. +SHARD_ROWS = 48 + +_LOW32 = 0xFFFFFFFF + + +def ordered_bits(v: torch.Tensor) -> torch.Tensor: + """The kernel's order-preserving uint32 image of fp32 values, as int64 (-0.0 maps to +0.0's image: they tie).""" + b = v.float().contiguous().view(torch.int32).to(torch.int64) & _LOW32 + b = torch.where(b == 0x80000000, torch.zeros_like(b), b) + return torch.where(b >= 0x80000000, _LOW32 - b, b | 0x80000000) + + +def make_keys(values: torch.Tensor, index: torch.Tensor) -> torch.Tensor: + """Exchange keys of (fp32 value, global token index) pairs: larger value wins, ties take the lower index.""" + return (ordered_bits(values) << 32) | (_LOW32 - index.to(torch.int64)) + + +def unsigned_max(keys: torch.Tensor, dim: int) -> torch.Tensor: + """Max of int64-stored uint64 keys (the kernel compares them unsigned).""" + sign = torch.iinfo(torch.int64).min + return (keys ^ sign).max(dim=dim).values ^ sign + + +def key_token(key: torch.Tensor) -> torch.Tensor: + return (_LOW32 - (key & _LOW32)).to(torch.int32) + + +def local_argmax(logits: torch.Tensor, vocab_offset: int) -> tuple[torch.Tensor, torch.Tensor]: + """(max value, lowest global index reaching it) per row of a [M, Vs] fp32 shard.""" + best = logits.max(dim=1).values + idx = (logits == best[:, None]).float().argmax(dim=1) # first (lowest) index at the max + return best, idx.to(torch.int64) + vocab_offset + + +def exchange_tokens( + own_keys: torch.Tensor, + ws_uc: torch.Tensor, + counters: torch.Tensor, + rank: int, + world: int, + slot_rows: int, +) -> torch.Tensor: + """Global tokens of the cross-rank pick: this rank's keys against the other ranks' keys already in its slots. + + Slot (half * slot_rows + t) * world + r of ``ws_uc`` holds rank r's key of row t; half = call counter & 1. A + single-device run stands in for the other ranks by writing their keys there before the call. + """ + m = own_keys.shape[0] + half = int(counters[1].item()) & 1 + slots = ( + ws_uc.view(torch.int64)[half * slot_rows * world : (half * slot_rows + m) * world] + .view(m, world) + .clone() + ) + slots[:, rank] = own_keys + return key_token(unsigned_max(slots, dim=1)) + + +def strict_accept( + tokens: torch.Tensor, + draft: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], +) -> dict[str, torch.Tensor]: + """The strict acceptance epilogue from the verify-row tokens [nreq * (D + 1)] (request-major). + + accepted[b] = tokens of request b's rows; num_accepted[b] = 1 + the matching prefix of draft[b] (overridden by + force_vec[b], else by force_total > 0); gather_ids[b] = batch_idx[b] * (D + 1) + num_accepted[b] - 1, and + next_new[b, 0] (when given) = the token at that row. Returns the outputs plus ``next_new`` as written. + """ + nreq, d = draft.shape + k1 = d + 1 + tokens = tokens.to(torch.int32) + accepted = tokens.view(nreq, k1).clone() + match = (draft.to(torch.int32) == accepted[:, :d]).to(torch.int32) + num = 1 + match.cumprod(dim=1).sum(dim=1) + if force_vec is not None: + num = force_vec[:nreq].to(num.dtype) + elif force_total > 0: + num = torch.full_like(num, force_total) + gather = batch_idx.to(torch.int64) * k1 + num.to(torch.int64) - 1 + out = {"accepted": accepted, "num_accepted": num.to(torch.int32), "gather_ids": gather} + if next_new is not None: + nn = next_new.clone() + nn[:, 0] = tokens[gather] + out["next_new"] = nn + return out + + +class M3ShardAccept(OpWrapper): + """Greedy verification of ``nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard ``[rows, Vs]`` + (global ids ``vocab_offset ..``). + + Each row's token is its global argmax over all ranks' shards: keys ``ordered(value) << 32 | ~index`` (ties take + the lowest index, as ``torch.argmax``) are max-reduced over the shard, pushed into every rank's exchange slot + through the multicast slots ``[2][48][world]`` (Lamport: 0 = empty) and max-reduced over the ranks by the last + CTA, which then runs the strict acceptance: ``acc_tokens [nreq, D + 1]``, ``num_acc [nreq]`` (``force_vec`` / + ``force_total`` override the counts), ``gather_ids [nreq]`` (the first draft pass's row per request) and, when + given, ``next_new[:, 0]`` (the last accepted token). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # Every greedy verify step of the target (modeling.SHARD_ACCEPT): draft_len 3 at 1..12 requests (4..48 rows), the + # target lm_head's BF16 shard logits (50016 columns per rank at TP4). The logits come from the head GEMM, so the + # op only reduces. + CELLS: tuple[Cell, ...] = ( + Cell( + why="cap 1: 4 verify rows (1 request x T = 4), rank 0 of 4, next_new staged", + spec=dict(nreq=1, draft_len=3, vs=50016, world=4, rank=0, force="none", strided=False), + ), + Cell( + why="cap 2: 8 verify rows, rank 2, global ids offset by 2 x 50016", + spec=dict(nreq=2, draft_len=3, vs=50016, world=4, rank=2, force="none", strided=False), + ), + Cell( + why="cap 4: 16 verify rows (4 requests x T = 4), rank 1 of 4, next_new staged", + spec=dict(nreq=4, draft_len=3, vs=50016, world=4, rank=1, force="none", strided=False), + ), + Cell( + why="cap 8: 32 verify rows, rank 3, global ids offset by 3 x 50016", + spec=dict(nreq=8, draft_len=3, vs=50016, world=4, rank=3, force="none", strided=False), + ), + Cell( + why="12 rows (3 requests) read from a row-strided view of a wider logits buffer, forced counts", + spec=dict(nreq=3, draft_len=3, vs=50016, world=4, rank=0, force="vec", strided=True), + ), + Cell( + why="the 48-row maximum (12 requests at draft 3), force_total, rows with exact cross-rank ties", + spec=dict( + nreq=12, + draft_len=3, + vs=50016, + world=4, + rank=2, + force="total", + strided=False, + ties=True, + ), + ), + # Every other request count of the verify range, on the 4-rank 50016-column shards. + *( + Cell( + why=f"{n} requests ({4 * n} verify rows): a request count between the ones above", + spec=dict( + nreq=n, draft_len=3, vs=50016, world=4, rank=n % 4, force="none", strided=False + ), + ) + for n in (5, 6, 7, 9, 10, 11) + ), + ) + + note = """ + The op reduces BF16 logits exactly, so the reference is bit-exact, including cross-rank ties (lowest global + index wins). The cross-rank exchange is certified on one device: the other ranks' keys are written into this + rank's slots before the call, which is what their multicast pushes would have put there; the multicast mapping + itself (ws_mc_ptr = this rank's slots, as at world 1) is not exercised across devices here. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + torch.ops.trtllm.minimax_m3_shard_accept( + logits, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + num_ctas, + force_vec, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> dict[str, torch.Tensor]: + """Lowest-index maximum of each row's BF16 logits, the largest key among it and the peers' keys in + ``ws_uc``, then the strict acceptance. Reads ``ws_uc`` / ``counters`` before the call; the scratch and + output buffers and ``num_ctas`` do not change the result.""" + best, idx = local_argmax(logits.float(), vocab_offset) + tokens = exchange_tokens(make_keys(best, idx), ws_uc, counters, rank, world, SHARD_ROWS) + out = strict_accept(tokens, draft, batch_idx, force_total, next_new, force_vec) + out["tokens"] = tokens + return out + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + assert not bool(keys[: logits.shape[0]].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + assert ws_uc.numel() >= 2 * SHARD_ROWS * world, "ws_uc must hold [2][48][world] slots" + # A NaN logit maps to a key above every finite one and wins the row silently. + assert not bool(torch.isnan(logits).any()), "NaN logits win the argmax" + nreq = draft.shape[0] + assert bool(((batch_idx[:nreq] >= 0) & (batch_idx[:nreq] < nreq)).all()), ( + "batch_idx must index the nreq requests" + ) + if force_vec is not None: + fv = force_vec[:nreq] + assert bool(((fv >= 1) & (fv <= draft.shape[1] + 1)).all()), ( + "force_vec counts must lie in [1, D + 1]" + ) + + +m3_shard_accept = M3ShardAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_slot_scatter.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_slot_scatter.py new file mode 100644 index 000000000000..ff07c54de1f0 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_slot_scatter.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The spec sampler's per-slot stores of one step's outputs, in one launch (``M3SpecStepFold.scatter`` -> +``m3_spec_tail.slot_scatter``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +def padded_rows(src: torch.Tensor, width: int) -> torch.Tensor: + """src rows zero-padded (or cut) to ``width`` columns.""" + out = torch.zeros(src.shape[0], width, dtype=src.dtype, device=src.device) + w = min(width, src.shape[1]) + out[:, :w] = src[:, :w] + return out + + +class M3SlotScatter(OpWrapper): + """For i < len(slots), all int32 2-D strided views: ``da[slots[i]] = sa[i]``, ``db[slots[i]] = sb[i]``, + ``dd[slots[i]] = sd[i]`` (each zero-padded / cut to the destination width) and ``dl[slots[i]] = sl[i]``. The + destinations are the store's transposed [slots, width] views.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The sampler store at draft_len 3 (8 slots): new tokens / next new tokens [slots, 4] (transposed views of + # [4, slots, 1]), next draft tokens [slots, 3]; a worker output narrower than the store (dynamic draft length) + # is zero-padded. + CELLS: tuple[Cell, ...] = ( + Cell( + why="8 rows into shuffled slots, widths equal to the store's", + spec=dict(n=8, slots=8, src_w=4, src_d=3, store_w=4, store_d=3), + ), + Cell( + why="3 rows of a narrower step (runtime draft 2): zero padding to the store width", + spec=dict(n=3, slots=8, src_w=3, src_d=2, store_w=4, store_d=3), + ), + ) + + note = """ + Pure index copies: the reference is exact. Slots are not bounds-checked; is_valid states it. + """ + + def __call__(self, slots, sa, da, sb, db, sl, dl, sd, dd): + torch.ops.trtllm.minimax_m3_slot_scatter(slots, sa, da, sb, db, sl, dl, sd, dd) + + def reference(self, slots, sa, da, sb, db, sl, dl, sd, dd): + """The four destinations after the call, on copies.""" + s = slots.long() + out = {} + for name, src, dst in (("da", sa, da), ("db", sb, db), ("dd", sd, dd)): + r = dst.clone() + r[s] = padded_rows(src[: s.numel()], dst.shape[1]) + out[name] = r + lr = dl.clone() + lr[s] = sl[: s.numel()] + out["dl"] = lr + return out + + def is_valid(self, slots, sa, da, sb, db, sl, dl, sd, dd): + assert bool(((slots >= 0) & (slots < da.shape[0]) & (slots < dl.numel())).all()), ( + "slots must index the store" + ) + # Two rows into one slot race (which one lands is unordered). + assert slots.unique().numel() == slots.numel(), "slots must be distinct" + assert sa.shape[0] >= slots.numel() and sl.numel() >= slots.numel(), "a source row per slot" + + +m3_slot_scatter = M3SlotScatter() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_spec_offsets.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_spec_offsets.py new file mode 100644 index 000000000000..36e421412f30 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_spec_offsets.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The overlap scheduler's speculative step correction as one launch (``RuntimeHooks.spec_offsets_op``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.logger import logger + +# The engagement line is logged once per process (module state: __call__ reads no instance state). +_ENGAGED: list = [] + + +class M3SpecOffsets(OpWrapper): + """``positions += position_offsets`` and ``kv_lens += kv_lens_offsets`` (int32, contiguous 1-D views, equal lengths + per pair) in one plain launch: trtllm::minimax_m3_sm107_tp4_spec_offsets (minimaxM3Sm107Tp4/specOffsets.cu), the + same wrapping int32 adds the two ATen ops do. ``__call__`` returns False, having written nothing, for other + layouts (the caller runs the ATen adds).""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="1 request, 4 tokens (draft length 3)", spec=dict(tokens=4, seqs=1)), + Cell(why="8 requests, 32 tokens", spec=dict(tokens=32, seqs=8)), + Cell(why="wrap-around values", spec=dict(tokens=8, seqs=2, wrap=True)), + ) + + note = "Integer adds: the reference (the two in-place ATen adds) is exact." + + def __call__(self, pos, pos_off, kv, kv_off) -> bool: + ts = (pos, pos_off, kv, kv_off) + if ( + any( + t.dtype != torch.int32 or t.dim() != 1 or t.stride(0) != 1 or not t.is_cuda + for t in ts + ) + or pos.numel() != pos_off.numel() + or kv.numel() != kv_off.numel() + ): + return False + torch.ops.trtllm.minimax_m3_sm107_tp4_spec_offsets(pos, pos_off, kv, kv_off) + if not _ENGAGED: + _ENGAGED.append(True) + logger.info("[M3 host] spec step correction: one launch taken (spec_offsets_op)") + return True + + def reference(self, pos, pos_off, kv, kv_off) -> None: + pos += pos_off + kv += kv_off + + def is_valid(self, pos, pos_off, kv, kv_off) -> None: + assert pos.numel() == pos_off.numel() and kv.numel() == kv_off.numel(), ( + "equal lengths per pair" + ) + + +m3_spec_offsets = M3SpecOffsets() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_step_gather.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_step_gather.py new file mode 100644 index 000000000000..16c4000199da --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_step_gather.py @@ -0,0 +1,152 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The overlap scheduler's per-step input gathers from the previous step's device outputs, in one launch +(``M3SpecStepFold.gather`` -> ``m3_spec_tail.step_gather``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3StepGather(OpWrapper): + """For the ``P`` previous requests (sequence slots ``slots``, ``T`` tokens and ``D`` draft tokens each), all + int32:: + + ids[ids0 + p*T + t] = new_tok[t, slots[p]] (new_tok [T', slots, (1)]) + draft[d0 + p*D + d] = next_draft[slots[p], d] + pos_off[i] = lens[ppi[i - b0*T]] for i in [b0*T, (b0+P)*T), else 0 (every i < pos_off.numel()) + kv_off[i] = lens[slots[i - b0]] - T for i in [b0, b0+P), else 0 (every i < kv_off.numel()) + + Every other element of ``ids`` / ``draft`` is left as it was. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # draft_len 3: T = 4, D = 3; the store's [T, slots, 1] new tokens and [slots, D] next drafts (8 slots, the + # max batch); the engine's flat buffers sized for the max batch. + CELLS: tuple[Cell, ...] = ( + Cell( + why="an unchanged batch of 8 previous requests in shuffled slots, nothing before them (b0 = 0)", + spec=dict(p=8, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0), + ), + Cell( + why="2 previous requests after 3 new ones: offsets ids0 / d0 / b0 nonzero, zero-filled offsets outside", + spec=dict(p=2, t=4, d=3, slots=8, b0=3, ids0=12, d0=9, extra_rows=3), + ), + Cell( + why="one request, the store's 2-D view of new tokens (beam dim squeezed)", + spec=dict(p=1, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0, two_d=True), + ), + ) + + note = """ + Pure index copies: the reference is exact. The op does not bound slots, ppi or the destination offsets; is_valid + states them. + """ + + def __call__( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + torch.ops.trtllm.minimax_m3_step_gather( + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ) + + def reference( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + """The four destinations after the call, on copies (``ids`` / ``draft`` keep their other elements).""" + nt = new_tok.reshape(new_tok.shape[0], new_tok.shape[1]) + s = slots[:P].long() + ids_r, draft_r = ids.clone(), draft.clone() + ids_r[ids0 : ids0 + P * T] = nt[:T, s].t().reshape(-1) + draft_r[d0 : d0 + P * D] = next_draft[s, :D].reshape(-1) + pos_r = torch.zeros_like(pos_off) + pos_r[b0 * T : (b0 + P) * T] = lens[ppi[: P * T].long()] + kv_r = torch.zeros_like(kv_off) + kv_r[b0 : b0 + P] = lens[s] - T + return {"ids": ids_r, "draft": draft_r, "pos_off": pos_r, "kv_off": kv_r} + + def is_valid( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + # Reads through slots / ppi and writes at the offsets are unchecked: any of them out of range reads or + # writes another buffer. + s = slots[:P] + assert bool( + ( + (s >= 0) & (s < next_draft.shape[0]) & (s < new_tok.shape[1]) & (s < lens.numel()) + ).all() + ), "slots must index the store" + assert bool(((ppi[: P * T] >= 0) & (ppi[: P * T] < lens.numel())).all()), ( + "ppi must index lens" + ) + assert new_tok.shape[0] >= T and next_draft.shape[1] >= D, "store narrower than T / D" + assert ids.numel() >= ids0 + P * T and draft.numel() >= d0 + P * D, "destinations too short" + assert pos_off.numel() >= (b0 + P) * T and kv_off.numel() >= b0 + P, ( + "offset buffers too short" + ) + + +m3_step_gather = M3StepGather() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_step_prologue.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_step_prologue.py new file mode 100644 index 000000000000..122d43fb175a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/ops/spec/m3_step_prologue.py @@ -0,0 +1,76 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode graph's step-input prologue in one launch (this target's ``RuntimeHooks.pack_prologue_op``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3StepPrologue(OpWrapper): + """``StepInputPack``'s early-copy prologue as one single-CTA C++ launch + (trtllm::minimax_m3_sm107_tp4_step_prologue, minimaxM3Sm107Tp4/stepPrologue.cu). + + ``window`` is the pack's device window (``gdev``): an int64 header ``[host copies, device copies]``, then one + ``(dst, src, bytes)`` descriptor per copy, host copies first (a host copy's ``src`` is a byte offset into the + window, a device copy's a device address). The launch writes up to ``max_desc`` copies of each kind, every host + copy before any device copy, then zeroes the header; with ``epoch`` it increments ``epoch[0]`` once. The default + prologue does the same with two parallel-scatter launches (``step_inputs._graph_scatter_kernel``) and a header + fill.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The TP4 decode step's packs: a few dozen host copies of 4 to a few hundred bytes (positions, KV lengths, slots, + # the spec sampler's uploads), a handful of device copies (the graph runner's static inputs), some reading what a + # host copy of the same window wrote. + CELLS: tuple[Cell, ...] = ( + Cell(why="an empty window (a replay with no new upload)", spec=dict(h2d=0, d2d=0)), + Cell(why="host copies only, odd sizes and alignments", spec=dict(h2d=40, d2d=0)), + Cell( + why="host copies feeding device copies (a device copy reads a host copy's destination)", + spec=dict(h2d=24, d2d=8, chained=True), + ), + Cell( + why="every descriptor slot (128 + 128 of the pack's 256)", spec=dict(h2d=128, d2d=128) + ), + ) + + note = """ + Byte copies: the reference (the default prologue's Triton launches and fill on the same window) is exact. + Destinations are raw addresses; is_valid cannot check them. + """ + + def __call__(self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int) -> None: + torch.ops.trtllm.minimax_m3_sm107_tp4_step_prologue(window, epoch, max_desc) + + def reference(self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int) -> None: + """The default prologue on the same arguments: it writes the same destinations.""" + from tensorrt_llm._torch.pyexecutor.step_inputs import _graph_scatter_kernel + + stage_i64 = window.view(torch.int64) + for device_src in (False, True): + bump = epoch is not None and not device_src + _graph_scatter_kernel[(max_desc,)]( + window, + stage_i64, + epoch if bump else stage_i64, + DEVICE_SRC=device_src, + BLOCK=256, + BUMP_EPOCH=bump, + num_warps=1, + ) + window[:16].zero_() + + def is_valid(self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int) -> None: + assert window.dtype == torch.uint8 and window.is_contiguous(), "window: contiguous uint8" + assert window.numel() >= 16 + 24 * max_desc, "window: header and max_desc descriptors" + counts = window[:16].view(torch.int64).tolist() + assert counts[0] + counts[1] <= max_desc, "at most max_desc copies in all" + + +m3_step_prologue = M3StepPrologue() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/weights.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/weights.py new file mode 100644 index 000000000000..4ae83a2ca8cf --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/weights.py @@ -0,0 +1,910 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight tables and loaders: MiniMax-M3.1 (MXFP8 + packed NVFP4 experts) / sm_107 / tp4. + +Two tables, one per checkpoint the target reads: + +* ``MODEL_WEIGHTS`` (``MiniMaxM3Tp4Weights``): the vision-language checkpoint's language model, + ``language_model.*``. Every rank is handed the whole checkpoint, so every tensor-parallel split is this + table's: the Q heads, the K/V/index-Q heads (one each per rank), the dense MLPs and the shared expert's + intermediate, the o_proj / down-projection input columns, every routed expert's intermediate (all 128 + experts live on every rank), and the vocabulary of the shell's ``lm_head``. The index-K head is replicated. +* ``DRAFT_WEIGHTS`` (``MiniMaxM3Tp4DraftWeights``): the one-layer BF16 Eagle3 drafter's own checkpoint + (``speculative_model``), loaded through the shell's ``load_draft_weights`` when speculation is on. + +Storage is HF ``[out, in]`` row-major. Five families need a transform, each a layout a kernel demands: + +* **MXFP8 block scales** (every attention projection, the dense MLPs and the shared expert): the checkpoint + stores UE8M0 bytes ``[N, K / 32]``; the GEMMs read the 128x4-swizzled order + (``trtllm::block_scale_interleave``). A fused projection concatenates its shards' scales first and swizzles + once. +* **The five-way sparse-layer projection** ``[Q | K | V | index-Q | index-K]`` and the three-way dense one + ``[Q | K | V]``: the per-rank shards concatenated along N. +* **Gated MLPs** (dense and shared expert): ``[gate ; up]`` rows. +* **NVFP4 routed experts** (layers 3..59): ``[up (w3) ; gate (w1)]`` rows, then the trtllm-gen row shuffle of + the weight and of its E4M3 scales (gated-activation reorder + 128-row epilogue tile shuffle, + ``trtllmgen_maybe_get_cached_*_permute_indices``) and the 128x4 swizzle of the scales. +* **Gemma boundary norms**: ``input_layernorm``, ``post_attention_layernorm`` and ``model.norm`` are stored as + Gemma norms, ``(1 + w) * x``; the fused all-reduce + RMSNorm kernels apply ``w * x``, so ``1`` is added + (in BF16) at load. The per-head Q/K/index norms stay Gemma: their kernels apply the offset themselves. + +The routed experts' per-tensor scalars (input scales and second-level weight scales, every expert) load raw; +``modeling`` derives the kernels' global scales, alphas and activation constants from them once the weights +are real. + +Loading contract: ``load(model, weights)`` consumes the engine-provided checkpoint dict, fills every declared +parameter exactly once and asserts coverage in both directions: every declared parameter written, and every +checkpoint key either consumed or in ``expected_unconsumed`` (the vision tower and its projector, which this +text-only target does not build, and the language model's multi-token-prediction head). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn + +from tensorrt_llm._torch.models.checkpoints.hf.minimaxm3_weight_mapper import ( + reciprocal_global_scale, +) +from tensorrt_llm._torch.moe.fused_moe.quantization import ( + trtllmgen_maybe_get_cached_w2_permute_indices, + trtllmgen_maybe_get_cached_w3_w1_permute_indices, +) + +#: ``layers=ALL_LAYERS`` is the common case: a weight every decoder layer carries. +ALL_LAYERS = object() + +_LM = "language_model." +#: Checkpoint key families this target does not build: the vision tower and its two projectors (text only), and the +#: language model's multi-token-prediction head (speculation, when on, is the separate Eagle3 drafter). +_UNBUILT_PREFIXES = ( + "vision_tower.", + "multi_modal_projector.", + "patch_merge_mlp.", + _LM + "model.mtp.", +) + +MX_BLOCK = 32 # MXFP8: one UE8M0 scale per 32 elements along K +NV_BLOCK = 16 # NVFP4: one E4M3 scale per 16 elements along K +EPILOGUE_TILE_M = ( + 128 # the trtllm-gen MoE runner's row tile: fixes its weight and scale row shuffles +) + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + ``shape`` takes the dimension bundle and returns the declared shape. ``src`` is a callable + ``(d, i) -> [(key or key tuple, index into the param | None), ...]`` with keys relative to the checkpoint + root; a key tuple hands the transform those tensors together. ``layers`` is ``ALL_LAYERS``, ``None`` (not + per-layer) or ``d -> layer indices``. ``transform(src, d)`` runs on the destination's device before the copy. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + +# ------------------------------------------------------------------------------------------------ shards + + +def fused_projection(sparse_config) -> bool: + """Whether the sparse layers run the five-way packed QKV + index projection. + + On by default: this target requires the MSA implementation and an FP8 indexer, which the packed projection needs. + An explicit ``fuse_qkv_index_projection`` in the deployment's sparse attention config wins, so setting it to False + is the kill switch (the QKV projection plus a separate MXFP8 index projection). + """ + if sparse_config is None: + return False + # A config that does not record which fields were set (a plain namespace or dataclass) is taken at its word. + fields_set = getattr(sparse_config, "model_fields_set", None) + if fields_set is None or "fuse_qkv_index_projection" in fields_set: + return bool(getattr(sparse_config, "fuse_qkv_index_projection", False)) + return True + + +def _col( + t: torch.Tensor, d: SimpleNamespace, tp: int | None = None, rank: int | None = None +) -> torch.Tensor: + """This rank's rows (column-parallel output split), even split, sliced on the host and moved to the + destination device (only this rank's bytes are read).""" + tp = d.tp if tp is None else tp + rank = d.rank if rank is None else rank + n = t.shape[0] // tp + return _dev(t[rank * n : (rank + 1) * n], d) + + +def _row(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """This rank's input columns (row-parallel input split), even split, on the destination device.""" + n = t.shape[1] // d.tp + return _dev(t[:, d.rank * n : (d.rank + 1) * n], d) + + +def _dev(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """``t`` on the destination device (``d.dev``, set per fill).""" + return t.to(d.dev, non_blocking=True) + + +def _swizzle(scale: torch.Tensor) -> torch.Tensor: + """128x4 swizzle of a row-major ``[N, K / 32]`` UE8M0 scale matrix, flat.""" + return torch.ops.trtllm.block_scale_interleave(scale.contiguous()) + + +# --------------------------------------------------------------------------------------------- transforms + + +def _gemma_fold(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A Gemma boundary norm as the plain norm the fused all-reduce epilogue applies: ``w + 1`` in BF16.""" + return t + 1.0 + + +def _f32(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return t.to(torch.float32) + + +def _divisor(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A stored global scale (multiplier) -> the modelopt divisor: its FP32 reciprocal.""" + return reciprocal_global_scale("expert global scale", t) + + +def _qkv_sparse_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[Q | K | V | index-Q | index-K]``: Q by attention heads, K/V/index-Q by KV heads, index-K replicated.""" + q, k, v, iq, ik = t + return torch.cat( + [_col(q, d), _col(k, d), _col(v, d), _col(iq, d), _dev(ik, d)], dim=0 + ).contiguous() + + +def _qkv_sparse_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v, iq, ik = t + return _swizzle( + torch.cat([_col(q, d), _col(k, d), _col(v, d), _col(iq, d), _dev(ik, d)], dim=0) + ) + + +def _qkv_dense_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + return torch.cat([_col(q, d), _col(k, d), _col(v, d)], dim=0).contiguous() + + +def _qkv_dense_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + return _swizzle(torch.cat([_col(q, d), _col(k, d), _col(v, d)], dim=0)) + + +def _index_qk_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[index-Q | index-K]`` as the checkpoint's MXFP8 (E4M3) values: this rank's index-Q heads and the replicated + index-K, the same rows the five-way projection packs after Q / K / V.""" + iq, ik = t + return torch.cat([_col(iq, d), _dev(ik, d)], dim=0).contiguous() + + +def _index_qk_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """The UE8M0 block scales of ``_index_qk_w``, 128x4-swizzled as every MXFP8 projection's.""" + iq, ik = t + return _swizzle(torch.cat([_col(iq, d), _dev(ik, d)], dim=0)) + + +def _gate_up_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return torch.cat([_col(gate, d), _col(up, d)], dim=0).contiguous() + + +def _gate_up_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return _swizzle(torch.cat([_col(gate, d), _col(up, d)], dim=0)) + + +def _row_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _row(t, d).contiguous() + + +def _row_s(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _swizzle(_row(t, d)) + + +def _col_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _col(t, d).contiguous() + + +#: Permute indices of the trtllm-gen expert relayouts, keyed by (shape, family, scale group): one expert's are +#: every expert's, so each is computed once per load. +_PERMUTE_CACHE: dict = {} + + +def _expert_w31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 operand: ``[up ; gate]`` packed-FP4 rows of this rank, then the gated-activation row + reorder and the epilogue-tile shuffle.""" + up, gate = t + w = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + w, _PERMUTE_CACHE, EPILOGUE_TILE_M, is_gated_act_gemm=True + ) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 E4M3 scales: ``[up ; gate]`` rows of this rank, the same row reorder at scale + granularity, then the 128x4 swizzle.""" + up, gate = t + s = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK, is_gated_act_gemm=True + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled.view(torch.uint8).reshape(shape)) + .view(torch.float8_e4m3fn) + .reshape(shape) + ) + + +def _expert_w2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 operand: this rank's input columns, then the epilogue-tile row shuffle.""" + w = _row(t, d).contiguous() + perm = trtllmgen_maybe_get_cached_w2_permute_indices(w, _PERMUTE_CACHE, EPILOGUE_TILE_M) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 E4M3 scales: this rank's columns, the row shuffle at scale granularity, the swizzle.""" + s = _row(t, d).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w2_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled).view(torch.float8_e4m3fn).reshape(shape) + ) + + +# ------------------------------------------------------------------------------------------------ sources + + +def _one(template: str): + return lambda d, i, t=template: [(_LM + t.format(i=i), None)] + + +def _many(*templates: str): + return lambda d, i, ts=templates: [(tuple(_LM + t.format(i=i) for t in ts), None)] + + +def _per_expert(*suffixes: str): + """One source per routed expert, written at its expert index (all experts are local at tp4).""" + + def build(d, i, sx=suffixes): + out = [] + for e in range(d.num_experts): + keys = tuple(_LM + f"model.layers.{i}.block_sparse_moe.experts.{e}.{s}" for s in sx) + out.append((keys if len(keys) > 1 else keys[0], (e,))) + return out + + return build + + +def _dense_mlp_layers(d): + """Layers with the dense SwiGLU-OAI MLP (``moe_layer_freq`` 0).""" + return d.dense_mlp + + +def _moe_layers(d): + return d.moe + + +def _sparse_attn_layers(d): + """Layers with MSA sparse attention (``sparse_attention_freq`` 1): every layer of MiniMax-M3.1.""" + return d.sparse_attn + + +def _three_way_qkv_layers(d): + """Layers whose QKV is the three-way [Q | K | V]: the dense-attention layers, and every layer without the + five-way packed projection (``fuse_qkv_index_projection=False``).""" + return d.dense_attn if d.fused else range(d.num_layers) + + +def _five_way_qkv_layers(d): + return d.sparse_attn if d.fused else range(0) + + +def _unfused_sparse_layers(d): + return range(0) if d.fused else d.sparse_attn + + +_ATTN = "model.layers.{i}.self_attn" +_MOE = "model.layers.{i}.block_sparse_moe" + + +class MiniMaxM3Tp4Weights: + """MiniMax-M3 NVFP4 / sm_107 / tp4: the language model's table and the passes over it. + + ``declare`` allocates every weight it names, ``manifest`` maps checkpoint keys onto them, ``load`` fills + them; ``_targets`` is the one place that decides which parameters exist. + """ + + #: Free the per-expert relayout scratch after each expert stack. + RELEASE_AFTER = "_e_s2" + + WEIGHTS: tuple[W, ...] = ( + # --- the two boundary norms of every layer (Gemma, folded) --- + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.input_layernorm.weight"), + transform=_gemma_fold, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.post_attention_layernorm.weight"), + transform=_gemma_fold, + ), + # --- attention: per-head Q/K norms (Gemma, unfolded), the packed QKV and the output projection --- + W("q_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".q_norm.weight")), + W("k_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".k_norm.weight")), + W( + "qkv_w", + shape=lambda d: (d.qkv_dense_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in "qkv")), + transform=_qkv_dense_w, + layers=_three_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_dense_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many(*(f"{_ATTN}.{p}_proj.weight_scale_inv" for p in "qkv")), + transform=_qkv_dense_s, + layers=_three_way_qkv_layers, + ), + W( + "qkv_w", + shape=lambda d: (d.qkv_sparse_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in ("q", "k", "v", "index_q", "index_k"))), + transform=_qkv_sparse_w, + layers=_five_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_sparse_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + *( + f"{_ATTN}.{p}_proj.weight_scale_inv" + for p in ("q", "k", "v", "index_q", "index_k") + ) + ), + transform=_qkv_sparse_s, + layers=_five_way_qkv_layers, + ), + # The separate index projection [index-Q | index-K] of the unfused sparse layers: the checkpoint's MXFP8 + # values with their block scales (the rows of the five-way projection's index part), run as an MXFP8 GEMM. + W( + "idx_w", + shape=lambda d: (d.index_heads * d.index_dim + d.index_dim, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(_ATTN + ".index_q_proj.weight", _ATTN + ".index_k_proj.weight"), + transform=_index_qk_w, + layers=_unfused_sparse_layers, + ), + W( + "idx_s", + shape=lambda d: ((d.index_heads * d.index_dim + d.index_dim) * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _ATTN + ".index_q_proj.weight_scale_inv", _ATTN + ".index_k_proj.weight_scale_inv" + ), + transform=_index_qk_s, + layers=_unfused_sparse_layers, + ), + W( + "idx_q_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_q_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "idx_k_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_k_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "o_w", + shape=lambda d: (d.hidden, d.q_width), + dtype=torch.float8_e4m3fn, + src=_one(_ATTN + ".o_proj.weight"), + transform=_row_w, + ), + W( + "o_s", + shape=lambda d: (d.hidden * (d.q_width // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_ATTN + ".o_proj.weight_scale_inv"), + transform=_row_s, + ), + # --- dense SwiGLU-OAI MLP (layers 0..2) --- + W( + "mlp_gu_w", + shape=lambda d: (2 * d.dense_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight", "model.layers.{i}.mlp.up_proj.weight" + ), + transform=_gate_up_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_gu_s", + shape=lambda d: (2 * d.dense_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight_scale_inv", + "model.layers.{i}.mlp.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_w", + shape=lambda d: (d.hidden, d.dense_inter), + dtype=torch.float8_e4m3fn, + src=_one("model.layers.{i}.mlp.down_proj.weight"), + transform=_row_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_s", + shape=lambda d: (d.hidden * (d.dense_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one("model.layers.{i}.mlp.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_dense_mlp_layers, + ), + # --- MoE (layers 3..59): the fp32 router and its bias, the shared expert --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + dtype=torch.float32, + src=_one(_MOE + ".gate.weight"), + transform=_f32, + layers=_moe_layers, + ), + W( + "router_bias", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_one(_MOE + ".e_score_correction_bias"), + transform=_f32, + layers=_moe_layers, + ), + W( + "sh_gu_w", + shape=lambda d: (2 * d.shared_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + _MOE + ".shared_experts.gate_proj.weight", _MOE + ".shared_experts.up_proj.weight" + ), + transform=_gate_up_w, + layers=_moe_layers, + ), + W( + "sh_gu_s", + shape=lambda d: (2 * d.shared_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _MOE + ".shared_experts.gate_proj.weight_scale_inv", + _MOE + ".shared_experts.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_moe_layers, + ), + W( + "sh_dn_w", + shape=lambda d: (d.hidden, d.shared_inter), + dtype=torch.float8_e4m3fn, + src=_one(_MOE + ".shared_experts.down_proj.weight"), + transform=_row_w, + layers=_moe_layers, + ), + W( + "sh_dn_s", + shape=lambda d: (d.hidden * (d.shared_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_MOE + ".shared_experts.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_moe_layers, + ), + # --- the routed experts, NVFP4 (trtllm-gen layouts), all 128 on every rank --- + W( + "e_w31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // 2), + dtype=torch.uint8, + src=_per_expert("w3.weight_packed", "w1.weight_packed"), + transform=_expert_w31, + layers=_moe_layers, + ), + W( + "e_w2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // 2), + dtype=torch.uint8, + src=_per_expert("w2.weight_packed"), + transform=_expert_w2, + layers=_moe_layers, + ), + W( + "e_s31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w3.weight_scale", "w1.weight_scale"), + transform=_expert_s31, + layers=_moe_layers, + ), + W( + "e_s2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w2.weight_scale"), + transform=_expert_s2, + layers=_moe_layers, + ), + # The per-tensor scalars of every expert: the checkpoint stores the multipliers into the quantized range + # (``*_global_scale``); the modelopt divisors modeling derives the kernels' scalars from are their FP32 + # reciprocals, taken by the same helper as the built-in loader. + *( + W( + f"e_{name}", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_per_expert(suffix), + transform=_divisor, + layers=_moe_layers, + ) + for name, suffix in ( + ("isc1", "w1.input_global_scale"), + ("isc3", "w3.input_global_scale"), + ("isc2", "w2.input_global_scale"), + ("ws2_1", "w1.weight_global_scale"), + ("ws2_3", "w3.weight_global_scale"), + ("ws2_2", "w2.weight_global_scale"), + ) + ), + # --- not per-layer --- + W( + "embed", + shape=lambda d: (d.vocab, d.hidden), + src=_one("model.embed_tokens.weight"), + layers=None, + ), + W( + "final_norm", + shape=lambda d: (d.hidden,), + src=_one("model.norm.weight"), + transform=_gemma_fold, + layers=None, + ), + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + mapping = core.model_config.mapping + sparse = cfg.sparse_attention_config + freq = [int(f) for f in sparse["sparse_attention_freq"]] + moe_freq = [ + int(f) for f in getattr(cfg, "moe_layer_freq", None) or [1] * cfg.num_hidden_layers + ] + tp = mapping.tp_size + head_dim = int(cfg.head_dim) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + index_heads = max(int(sparse["sparse_num_index_heads"]) // tp, 1) + index_dim = int(sparse["sparse_index_dim"]) + return SimpleNamespace( + fused=fused_projection(core.model_config.sparse_attention_config), + num_layers=cfg.num_hidden_layers, + sparse_attn=[i for i, f in enumerate(freq) if f], + dense_attn=[i for i, f in enumerate(freq) if not f], + moe=[i for i, f in enumerate(moe_freq) if f], + dense_mlp=[i for i, f in enumerate(moe_freq) if not f], + hidden=cfg.hidden_size, + head_dim=head_dim, + heads=heads, + kv_heads=kv_heads, + index_heads=index_heads, + index_dim=index_dim, + q_width=heads * head_dim, + qkv_dense_rows=(heads + 2 * kv_heads) * head_dim, + qkv_sparse_rows=(heads + 2 * kv_heads) * head_dim + index_heads * index_dim + index_dim, + dense_inter=int(cfg.dense_intermediate_size) // tp, + shared_inter=int(cfg.shared_intermediate_size) * int(cfg.n_shared_experts) // tp, + moe_inter=int(cfg.intermediate_size) // tp, + num_experts=int(cfg.num_local_experts), + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None)] + return [(f"l{i}_{entry.name}", i) for i in layers] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares (meta under the engine's meta init).""" + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, core) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + """param key -> [(checkpoint key or key tuple, index into the param | None, transform | None)].""" + d = self.dims(core) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, core, weights) -> set: + """The vision tower and its projectors, and the multi-token-prediction head: in the checkpoint, never built + by this target.""" + return {k for k in weights.keys() if k.startswith(_UNBUILT_PREFIXES)} + + def load(self, model, weights) -> None: + """Fill every declared weight and the shell's ``lm_head`` shard; assert coverage both ways.""" + core = model.model + d = self.dims(core) + manifest = self.manifest(core) + assert set(manifest) == set(core.w.keys()), ( + "manifest/parameter drift", + set(manifest) ^ set(core.w.keys()), + ) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(core, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(core.w[param_key], ckpt, index, transform, weights, d, consumed) + if param_key.endswith(self.RELEASE_AFTER): + torch.cuda.empty_cache() + # The shell owns the vocab-parallel head: this rank's rows of the full table. + _fill(model.lm_head.weight, _LM + "lm_head.weight", None, _col_w, weights, d, consumed) + shell = { + n for n, _ in model.named_parameters() if not n.startswith(("model.w.", "draft_model.")) + } + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + "checkpoint coverage: leftover keys are not exactly the unbuilt families (vision, MTP head); " + f"unexpected {sorted(leftover - expected)[:8]}, missed {sorted(expected - leftover)[:8]}" + ) + + +def _materialize(entry) -> torch.Tensor: + """``[:]`` on a lazy safetensors slice; a 0-dim tensor (the per-tensor scalars) rejects that index.""" + return entry if getattr(entry, "ndim", 1) == 0 else entry[:] + + +def _fill(param, ckpt, index, transform, weights, d, consumed) -> None: + keys = ckpt if isinstance(ckpt, tuple) else (ckpt,) + for key in keys: + assert key in weights, f"checkpoint key missing: {key}" + dst = param.data if index is None else param.data[index] + d.dev = dst.device + # Host tensors: a transform slices this rank's part on the host and moves only that to the device. + src_all = tuple(_materialize(weights[key]) for key in keys) + if transform is not None: + src = transform(src_all if isinstance(ckpt, tuple) else src_all[0], d) + else: + assert len(src_all) == 1, "a multi-key row needs a transform" + src = src_all[0] + if src.dim() == 0 and dst.dim() == 0: + pass + elif src.numel() == 1 and dst.numel() == 1: + src = src.reshape(dst.shape) + src = src.to(dst.device, non_blocking=True) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(keys) + _release(weights, keys) + + +def _release(weights, keys) -> None: + """Drop consumed checkpoint tensors from the engine's dict as the load goes (bounded host memory).""" + if hasattr(weights, "mark_consumed_keys"): + weights.mark_consumed_keys(keys) + else: + for key in keys: + weights.pop(key, None) + + +MODEL_WEIGHTS = MiniMaxM3Tp4Weights() + + +# -------------------------------------------------------------------------------------------- the drafter + + +def _draft_one(template: str): + return lambda d, i, t=template: [(t, None)] + + +def _draft_many(*templates: str): + return lambda d, i, ts=templates: [(tuple(ts), None)] + + +def _draft_fc_norms(d): + return range(d.capture_layers) + + +def _draft_qkv(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + return torch.cat([_col(q, d), _col(k, d), _col(v, d)], dim=0).contiguous() + + +class MiniMaxM3Tp4DraftWeights: + """The one-layer BF16 Eagle3 drafter (``speculative_model``), tp4. + + The fc over the three captured target layers and its per-layer fc_norms, the layer's three norms, its QKV + (input ``[embeds | hidden]``, so twice the hidden width; Q by heads, K/V by KV heads) and o_proj, its gated + MLP, the final norm and the full-vocabulary head (vocab-parallel). The checkpoint's ``embed_tokens`` is the + target's table (the drafter reads the target's); it is a predicted non-load. + """ + + WEIGHTS: tuple[W, ...] = ( + W( + "fc", + shape=lambda d: (d.hidden, d.capture_layers * d.hidden), + src=_draft_one("fc.weight"), + layers=None, + ), + W( + "fc_norm", + shape=lambda d: (d.hidden,), + src=lambda d, i: [(f"fc_norm.{i}.weight", None)], + layers=_draft_fc_norms, + ), + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.input_layernorm.weight"), + layers=None, + ), + W( + "hidden_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.hidden_norm.weight"), + layers=None, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.post_attention_layernorm.weight"), + layers=None, + ), + W( + "qkv", + shape=lambda d: (d.qkv_rows, 2 * d.hidden), + src=_draft_many(*(f"layers.0.self_attn.{p}_proj.weight" for p in "qkv")), + transform=_draft_qkv, + layers=None, + ), + W( + "o", + shape=lambda d: (d.hidden, d.q_width), + src=_draft_one("layers.0.self_attn.o_proj.weight"), + transform=_row_w, + layers=None, + ), + W( + "gu", + shape=lambda d: (2 * d.inter, d.hidden), + src=_draft_many("layers.0.mlp.gate_proj.weight", "layers.0.mlp.up_proj.weight"), + transform=_gate_up_w, + layers=None, + ), + W( + "dn", + shape=lambda d: (d.hidden, d.inter), + src=_draft_one("layers.0.mlp.down_proj.weight"), + transform=_row_w, + layers=None, + ), + W("final_norm", shape=lambda d: (d.hidden,), src=_draft_one("norm.weight"), layers=None), + ) + + def dims(self, draft) -> SimpleNamespace: + cfg = draft.draft_config.pretrained_config + mapping = draft.draft_config.mapping + tp = mapping.tp_size + head_dim = int(getattr(cfg, "head_dim", cfg.hidden_size // cfg.num_attention_heads)) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + return SimpleNamespace( + hidden=cfg.hidden_size, + capture_layers=draft.capture_layers, + head_dim=head_dim, + q_width=heads * head_dim, + qkv_rows=(heads + 2 * kv_heads) * head_dim, + inter=cfg.intermediate_size // tp, + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + if entry.layers is None: + return [(entry.name, None)] + return [(f"{entry.name}{i}", i) for i in entry.layers(d)] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, draft) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + d = self.dims(draft) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, draft, weights) -> set: + """The drafter checkpoint's own embedding: the drafter reads the target's table.""" + return {"embed_tokens.weight"} & set(weights.keys()) + + def load(self, draft, weights) -> None: + d = self.dims(draft) + manifest = self.manifest(draft) + assert set(manifest) == set(draft.w.keys()), set(manifest) ^ set(draft.w.keys()) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(draft, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(draft.w[param_key], ckpt, index, transform, weights, d, consumed) + _fill(draft.lm_head.weight, "lm_head.weight", None, _col_w, weights, d, consumed) + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + f"drafter checkpoint coverage: unexpected {sorted(leftover - expected)[:8]}, " + f"missed {sorted(expected - leftover)[:8]}" + ) + + +DRAFT_WEIGHTS = MiniMaxM3Tp4DraftWeights() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/__init__.py new file mode 100644 index 000000000000..adf71d21a10a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 NVFP4 / sm_107 / tp8.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/dspark.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/dspark.py new file mode 100644 index 000000000000..7e260121eb11 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/dspark.py @@ -0,0 +1,984 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-model DSpark on this target (phase W0): the in-tree DSpark drafter and worker, unchanged. + +The target keeps its own forward; the shell hands the step to ``M3DSparkWorker``, which is the in-tree +``DSparkWorker`` plus the five calls the shell makes on any worker (``capture_hidden_states``, +``arm_capture_norm``, ``tail_eligible``, ``prebuild``, ``__call__``). In W0 none of them takes a fused path: the +captures are the in-tree DFlash captures, the target logits go through the logits processor and the in-tree +strict accept verifies them, so the speculation is the in-tree one by construction. + +The drafter is the in-tree standalone DSpark drafter for the checkpoint's architecture, built through the draft +model registry. Its context K/V lives in the separate draft KV cache, which must keep the drafter's projection +dtype (BF16): a pool of another dtype is not bound and the worker falls back to a private arena sized by the +served sequence length. So the drafter never takes the target's KV quantization: neither the FP8 draft-KV +override (an Eagle3 need) nor the target's own KV algo (NVFP4 under ``kv_cache_config.dtype="nvfp4"``). +""" + +from __future__ import annotations + +import functools +import os +import time +from typing import Tuple + +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.comm import ( + allgather_cast as _allgather_cast, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + strict_accept as _strict_accept, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_block_tables import ( + enable_parallel_refresh, + refresh_block_tables, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_advance import ( + advance_ctx_and_read_view, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_ctx_store import ( + fold_k_norm_weight, + fused_ctx_store, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_drafter_inputs import ( + drafter_inputs as _drafter_inputs, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_forward_indices import ( + forward_indices, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode.dspark_qk_norm_rope import ( + qk_norm_rope, + qk_norm_rope_append, +) +from tensorrt_llm._torch.distributed import allgather +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + external_drafter_config_kwargs, + get_draft_model, +) +from tensorrt_llm._torch.speculative.dspark import DSparkWorker +from tensorrt_llm._torch.speculative.interface import ( + _RNG_SLOT_SPAN, + should_use_separate_draft_kv_cache, + tail_bookkeep_draft_len, + tail_bookkeep_log_once, +) +from tensorrt_llm.logger import logger + +from .dspark_s4 import MAX_FAST_BATCH, TAILBK_SPEC, S4WorkerMixin +from .kernels import m3_mc_allreduce +from .m31_switches import m31_disabled, m31_enabled +from .mxdraft import route_gu_swiglu, route_mx_linears +from .ops.embedding.m3_embedding import m3_embedding +from .ops.gemm.m31_fc_shard import FcShard + +# BUNDLE_OFF_NOTE: S5 and S2 (prebuild below) are OFF by default (S9b is on at this TP again, see +# S9B_DEFAULT_ON) with the K3 bundle {S9b, S5, S2, FUSED_CAPTURE}: its +# powered K7 natural-acceptance gate (n 1,776) failed (mean d +# -0.032 AL, CI [-0.059, -0.005], median -0.022, sign p 0.037 against) (this applies to the sm_107 target +# too), pending the per-member bisect. Opt in with TRTLLM_M31_ENABLE_=1; TRTLLM_M31_DISABLE_=1 still wins. +# FUSED_CAPTURE does not exist in this target. + +# S9b (fused drafter all-reduce + norm) by default at this TP; TRTLLM_M31_DISABLE_S9B still turns it off. +# measured: TP8 -0.4% / step. +# ON again at this target (S9b only; S5 and S2 stay off per BUNDLE_OFF_NOTE): S9b alone passed its own powered K7 +# natural-acceptance gate here (1,904 pairs, mean d +0.020 AL, CI [-0.003, +0.043], median +0.018). +# TRTLLM_M31_DISABLE_S9B still turns it off. +S9B_DEFAULT_ON = True + +# S9B-MC: S9b's fused drafter all-reduces + norms on the target's multicast all-reduce + norm (its 8-token bucket, the +# drafter-norm form: the unrounded fp32 sum normalized, opts bit 16) instead of the stock fused all-reduce; not +# bitwise, as S9b (a different summation order). The K7 c1 drafter block (8 rows) takes it; other row counts keep S9b's. +# Default OFF: its microbench measured -0.002x and it was flat in-model, so the drafter keeps S9b's fused path. +# TRTLLM_M31_ENABLE_S9B_MC=1 opts in; TRTLLM_M31_DISABLE_S9B_MC=1 still wins. Read at import. +S9B_MC = m31_enabled("S9B_MC") and not m31_disabled("S9B_MC") +S9B_MC_MAX_ROWS = 8 # capacity: fixed: the drafter-norm form exists for the 8-token bucket only + + +# DRAFT_APPEND_FUSE: the drafter's per-layer flashinfer +# append_paged_kv_cache folded into the EX-11 q / k norm + RoPE launch (the same K bits, V copied; same pages / slots). +# Exact. Default ON; kill switch TRTLLM_M31_DISABLE_DRAFT_APPEND_FUSE=1, read at import. Taken only where the drafter +# appends through flashinfer's paged HND cache (append_supported); otherwise the per-layer append runs as before. +DRAFT_APPEND_FUSE = not m31_disabled("DRAFT_APPEND_FUSE") +# FC_SHARD: the drafter's fc (replicated on every rank) as this rank's output-column shard + an all-gather in +# project_target_hidden, bit for bit the full fc (the K + 1 verify rows at batch 1). Kill switch +# TRTLLM_M31_DISABLE_FC_SHARD, read at import. +FC_SHARD = not m31_disabled("FC_SHARD") +_APPEND_FUSE_TAKEN = [0] + + +def _qk_norm_rope_append_logged(*args): + """catalog qk_norm_rope_append, counting taken calls and logging the first (host calls: eager or captured).""" + out = qk_norm_rope_append(*args) + if out is not None: + _APPEND_FUSE_TAKEN[0] += 1 + if _APPEND_FUSE_TAKEN[0] == 1: + logger.info( + "[M3 dispatch] DRAFT_APPEND_FUSE taken: drafter K / V append in the EX-11 launch" + ) + return out + + +def _s9b_mc_op(mapping): + """The fused AR + norm op S9B-MC installs: (normed, residual + reduced) from the multicast all-reduce + norm, or + None (the stock fused all-reduce runs) for operands it does not take. Shapes and the collectively created workspace + decide, so every TP rank agrees.""" + + def op(partial, residual, norm_weight, eps): + if partial.dim() != 2 or not 1 <= partial.shape[0] <= S9B_MC_MAX_ROWS: + return None + workspace = m3_mc_allreduce.get_workspace(mapping, partial.shape[-1]) + if workspace is None: + return None + return m3_mc_allreduce.m3_mc_allreduce_norm( + partial.contiguous(), + residual.contiguous(), + norm_weight, + eps, + workspace, + use_pdl=get_env_enable_pdl(), + drafter_norm=True, + ) + + return op + + +# S5: the largest decode batch the window-capped max_seq_len applies to, by draft length; 0 = off. Per (TP, K) on +# VR: the largest batch at which the window-capped drafter attention is faster than the stock call at every batch up +# to it, in two replicates of the per-call microbench (L = 400K). TP8: K3 shows no win +# at c1 (+0.04 / +0.004 us per call), so off; K7 c1 / c2 win, c4 loses +1.5. +S5_MAX_BATCH_BY_K = {3: 0, 7: 2} + +# S10 (lean drafter glue) applies at every draft length on VR TP8: token-identical to the eager glue at K3 and at K7 +# with the AllReduce autotune pinned (43 / 43 requests over c1 / c2 / c4, seeded T 1.0; the +# earlier K7 divergence ran with the autotune unpinned). TRTLLM_M31_DISABLE_S10=1 turns it off. +S10_MAX_DRAFT_LEN = 63 + + +def s10_applies(num_draft_tokens: int) -> bool: + return num_draft_tokens <= S10_MAX_DRAFT_LEN + + +def s5_max_batch_for(num_draft_tokens: int) -> int: + """K <= 3 takes the K3 entry, K >= 7 the K7 entry, and K in between the smaller of the two.""" + if num_draft_tokens <= 3: + return S5_MAX_BATCH_BY_K[3] + if num_draft_tokens >= 7: + return S5_MAX_BATCH_BY_K[7] + return min(S5_MAX_BATCH_BY_K.values()) + + +def is_standalone_dspark(spec_config) -> bool: + """Whether the run speculates with a standalone DSpark drafter (its own checkpoint, ``decoding_type: DSpark``).""" + return bool( + spec_config is not None + and spec_config.spec_dec_mode.use_one_engine() + and spec_config.spec_dec_mode.is_dspark() + and not getattr(spec_config, "draft_is_embedded_in_target", False) + ) + + +class M3EmbedTokens(nn.Module): + """The target's token embedding as the module the in-tree drafter borrows (``load_weights_from_target_model``). + + The drafter embeds the bonus and mask tokens through ``embed_tokens(ids)``; the target keeps its replicated + table as a raw tensor and gathers it with ``m3_embedding``, so this module calls the same gather on the same + table (an exact gather, so the drafter sees the rows the target embeds). + """ + + def __init__(self, core: nn.Module) -> None: + super().__init__() + self._core = [core] # not a submodule: the table belongs to the target + + @property + def weight(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def forward(self, ids: torch.Tensor) -> torch.Tensor: + return m3_embedding(ids, self.weight) + + +def s10c_c_drafter_inputs( + embed_tokens, + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + mask_token_id, + trained_mask_embedding, + block_size, + tokens_per_request, +): + """S10c-C: the drafter's step inputs in one launch (catalog/spec_decode/dspark_drafter_inputs), for this target's + embedding only: ``M3EmbedTokens`` is an exact gather of the replicated table, so the noise block is table rows (the + trained mask row if the drafter keeps one). None for any other embedding: the stock path runs.""" + if not isinstance(embed_tokens, M3EmbedTokens): + return None + table = embed_tokens.weight + mask_row = ( + trained_mask_embedding if trained_mask_embedding is not None else table[int(mask_token_id)] + ) + return _drafter_inputs( + accepted, + num_accepted, + slots, + ctx_len, + max_ctx, + table, + mask_row, + block_size, + tokens_per_request, + ) + + +# N3 A2 (switch TRTLLM_M31_DISABLE_N3A): the target's strict acceptance and next-step input tokens in one launch each +# (catalog/spec_decode/strict_accept), bitwise to the in-tree torch path. Read once, at import. +N3A_DISABLED = m31_disabled("N3A") +# N3A-KV (switch TRTLLM_M31_DISABLE_N3A_KV, and off with N3A): DFlash's draft-forward KV bookkeeping (the rewind amount +# 1 - num_accepted and kv_lens += 1) in one launch where the in-tree lines run, after the forced-acceptance override, +# bitwise (integer arithmetic). Read once, at import. +N3A_KV_DISABLED = N3A_DISABLED or m31_disabled("N3A_KV") +# Gate-only cross-check (TRTLLM_M31_CHECK_N3A_KV=1; never in a perf arm): every N3A-KV call is compared with the stock +# lines on copies (graph-safe device counters; reported on eager steps). +N3A_KV_CHECK = os.environ.get("TRTLLM_M31_CHECK_N3A_KV", "").strip() not in ("", "0") +# N3 P-gid + P-cat (switch TRTLLM_M31_DISABLE_N3P): the verify-row target logits without the identity gather_ids copy +# and with the vocab concat and the fp32 cast in one strided copy, bitwise to the logits processor. Read at import. +N3P_DISABLED = m31_disabled("N3P") +# N3 A1 (switch TRTLLM_M31_DISABLE_A1): the verify logits' vocab all-gather, concat and fp32 cast in one launch over a +# multicast NVLink workspace (catalog/comm/allgather_cast), bitwise to the NCCL all-gather + strided copy below. The +# verify rows (B x (K + 1), up to 64 rows: K3 batches 1..16, K7 batches 1..8). Read once, at import. Default on +# on VR since the VR op matrix + grid sweep certified it bitwise; the switch turns it off. +A1_DISABLED = m31_disabled("A1") +A1_MAX_ROWS = 64 # capacity: fixed: allgather_cast.MAX_ROWS (K3 at batch 16, K7 at batch 8) +# MCAG: the rejection sampler's gather of the vocab-sharded fp32 draft logits (stock: the NCCL +# all-gather + a concat copy) as one launch on the A1 multicast workspace (allgather_cast.allgather_f32: copied as is, +# bit for bit). VR TP8 K7 c1, [1, 7, 25008] fp32: 32.30 -> 20.15 us (bitwise eager and graph). Taken +# for the B * K draft rows up to the fp32 capacity while A1 is set up; other shapes keep the stock all-gather. OFF by +# default at this target: with it on, two runs of the same requests (every tactic cache pinned on all ranks) sample +# different tokens on a few long requests, and with it off they do not; the stock all-gather returns the same tensor bit +# for bit. TRTLLM_M31_ENABLE_MCAG=1 opts in; TRTLLM_M31_DISABLE_MCAG=1 still wins. +MCAG = m31_enabled("MCAG") and not m31_disabled("MCAG") +# A1 at draft lengths other than 3 (K7 included; switch TRTLLM_M31_DISABLE_A1_K7, K3 keeps A1). +A1_K7_DISABLED = m31_disabled("A1_K7") +_n3a_logged = False + + +def _n3a_log_once() -> None: + global _n3a_logged + if not _n3a_logged: + _n3a_logged = True + logger.info( + "[M3 dispatch] n3_strict_accept taken: fused strict acceptance + next-step input tokens" + ) + + +_n3p_logged = False +_n3a_kv_logged = False + + +def _n3a_kv_log_once() -> None: + global _n3a_kv_logged + if not _n3a_kv_logged: + _n3a_kv_logged = True + logger.info( + "[M3 dispatch] n3a_kv_rewind taken: draft-forward KV rewind + kv_lens bump in one launch" + ) + + +_a1_logged = set() + + +def _a1_log_once(key, message: str) -> None: + if key not in _a1_logged: + _a1_logged.add(key) + logger.info(message) + + +def _n3p_log_once() -> None: + global _n3p_logged + if not _n3p_logged: + _n3p_logged = True + logger.info( + "[M3 dispatch] n3p_target_logits taken: verify-row logits without the gather_ids copy and the concat" + ) + + +class M3DSparkWorker(S4WorkerMixin, DSparkWorker): + """The in-tree ``DSparkWorker`` with the calls this target's shell makes on its worker. W1: the one-launch Markov + chain (``S4WorkerMixin``, switch ``TRTLLM_M31_DRAFT_DISABLE_S4``). N3 A2: the strict acceptance and the next-step + input tokens in one launch each (switch ``TRTLLM_M31_DISABLE_N3A``). N3 P / A1: the verify logits (switches + ``TRTLLM_M31_DISABLE_N3P`` / ``TRTLLM_M31_DISABLE_A1``). TAILBK R1 / Z1: the one-request Philox rows from the + per-token window and no padding-row zeroing in captured batch-1 graphs, bitwise (spec_metadata's tail_bookkeep; + switch ``TRTLLM_M31_DISABLE_TAILBK``). Everything else is the in-tree path.""" + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None + ) -> None: + """The in-tree DFlash capture: layer output (hidden + residual) into the static capture buffer.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model + ) -> None: + """No capture norm: DSpark's fc reads the raw captured layers.""" + return None + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Never: the target logits always go through the logits processor and the in-tree strict accept.""" + return False + + def prebuild(self, lm_head: nn.Module, draft_model) -> None: + """The one-launch Markov chain's shards, buffers, exchange workspace and compiles (outside capture, every + rank); no fused verify, no coded target head, no fused draft loop.""" + rank = getattr(getattr(self, "mapping", None), "tp_rank", None) + # S10: the drafter's strided-view glue (bitwise; kill switch TRTLLM_M31_DISABLE_S10, read at import). + k = getattr(getattr(self, "spec_config", None), "max_draft_len", 0) or 0 + lean = not m31_disabled("S10") and s10_applies(k) + self.enable_lean_glue(draft_model, lean) + # S5: window-capped max_seq_len for the drafter's sliding-window decode (not bitwise: the TRTLLM-Gen tile / KV + # split changes, not the keys read). OFF by default (K3 bundle, see BUNDLE_OFF_NOTE): opt in with + # TRTLLM_M31_ENABLE_S5=1; TRTLLM_M31_DISABLE_S5 still wins. + s5_max_batch = s5_max_batch_for(k) if m31_enabled("S5") and not m31_disabled("S5") else 0 + self.enable_window_capped_decode(draft_model, s5_max_batch) + # The context arena spans every seq slot (CUDA graph metadata narrows max_num_requests to its bucket). + self.enable_full_slot_arena() + # S2: the drafter's replicated fc split row-parallel with one fp32 all-reduce (not bitwise: the partial-sum + # order changes). Collective. OFF by default (K3 bundle): opt in with TRTLLM_M31_ENABLE_S2=1; + # TRTLLM_M31_DISABLE_S2 still wins. + s2 = self.enable_fc_tp_split(draft_model, m31_enabled("S2") and not m31_disabled("S2")) + split = getattr(draft_model, "_dflash_fc_split", None) + ar = split[3] if split is not None else None + ar_path = ( + "MNNVL" + if getattr(ar, "mnnvl_allreduce", None) is not None + else getattr(ar, "strategy", None) + ) + # FC_SHARD (bitwise; collective: every rank builds or every rank declines). S2's split, when on, wins. + if hasattr(draft_model, "_dflash_fc_shard"): + draft_model._dflash_fc_shard = ( + FcShard.build(draft_model.fc, self.mapping) + if FC_SHARD and not s2 and getattr(draft_model, "fc", None) is not None + else None + ) + logger.info(f"M3.1 DSpark S2: fc TP split {'on' if s2 else 'off'} on rank {rank} (all-reduce " + f"{ar_path})") # fmt: skip + logger.info(f"M3.1 DSpark S5: window-capped draft decode for batches <= {s5_max_batch} on rank " + f"{rank}") # fmt: skip + logger.info(f"M3.1 DSpark S10: lean drafter glue {'on' if lean else 'off'} on rank {rank}") + # S9b: each drafter layer's all-reduce fused with the add + RMSNorm after it (not bitwise: the sum is not + # rounded before the add). On by default where S9B_DEFAULT_ON; kill switch TRTLLM_M31_DISABLE_S9B, read at + # import. + s9b = self.enable_fused_ar_norm( + draft_model, (S9B_DEFAULT_ON or m31_enabled("S9B")) and not m31_disabled("S9B") + ) + logger.info( + f"M3.1 DSpark S9b: fused drafter all-reduce + norm {'on' if s9b else 'off'} on rank {rank}" + ) + # S9B-MC: the multicast workspace is the target's (created collectively at its first eager boundary, the + # warmup forward, before any capture); until it exists the op declines on every rank alike. + s9b_mc = self.enable_fused_ar_norm_op( + draft_model, _s9b_mc_op(self.mapping) if s9b and S9B_MC else None + ) + logger.info( + f"M3.1 DSpark S9B-MC: drafter all-reduce + norm on the multicast kernel (<= {S9B_MC_MAX_ROWS} rows) " + f"{'on' if s9b_mc else 'off'} on rank {rank}" + ) + # S10c-A: the drafter's context K / V epilogue + accept mask + paged store in one launch after the K / V GEMM + # (bitwise). Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_A, read at import. + s10c_a = self.enable_fused_ctx_store( + None if m31_disabled("S10C") or m31_disabled("S10C_A") else fused_ctx_store + ) + # TAILBK_SPEC F: the store kernel also runs the k_norm weight multiply (bitwise torch's bf16 multiply; one + # launch fewer per step). Kill switch TRTLLM_M31_DISABLE_TAILBK_SPEC, read at import. + tailbk_f = fold_k_norm_weight(bool(s10c_a) and TAILBK_SPEC) + logger.info( + f"M3.1 DSpark S10c-A: fused drafter ctx store {'on' if s10c_a else 'off'}; TAILBK_SPEC F k_norm weight " + f"in the store kernel {'on' if tailbk_f else 'off'} on rank {rank}" + ) + # S10c-C: the drafter's step inputs (bonus, positions, noise block) in one launch (bitwise: integers and row + # copies). Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_C, read at import. + s10c_c = self.enable_fused_drafter_inputs( + None if m31_disabled("S10C") or m31_disabled("S10C_C") else s10c_c_drafter_inputs + ) + logger.info( + f"M3.1 DSpark S10c-C: fused drafter step inputs {'on' if s10c_c else 'off'} on rank {rank}" + ) + # S10c-D: the drafter's context advance + read view in one launch (bitwise, integer), with ADV_PAR's parallel + # read-view copy. Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_D / + # TRTLLM_M31_DISABLE_ADV_PAR (the one-launch copy), read at import. + s10c_d = self.enable_fused_ctx_advance( + None + if m31_disabled("S10C") or m31_disabled("S10C_D") + else functools.partial(advance_ctx_and_read_view, parallel=not m31_disabled("ADV_PAR")) + ) + logger.info( + f"M3.1 DSpark S10c-D: fused drafter context advance {'on' if s10c_d else 'off'} on rank {rank}" + ) + # S10c-B: the drafter's block-table refresh (decode + placeholder count) in one launch (bitwise, integer). + # Kill switches TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_B, read at import. + s10c_b = self.enable_fused_block_refresh( + None if m31_disabled("S10C") or m31_disabled("S10C_B") else refresh_block_tables + ) + # S10c-B2: that refresh on a (row, column-chunk) grid (bitwise; flat at short tables, removes the serial + # kernel's growth past ~11K blocks per sequence). Kill switch TRTLLM_M31_DISABLE_S10C_B2, read at import. + s10c_b2 = enable_parallel_refresh(s10c_b and not m31_disabled("S10C_B2")) + logger.info( + f"M3.1 DSpark S10c-B: fused drafter block-table refresh {'on' if s10c_b else 'off'} on rank {rank}" + f" (S10c-B2 2-D grid {'on' if s10c_b2 else 'off'})" + ) + # S10c-E: the draft model's paged block-decode index prelude in one launch (bitwise, integer). Kill switches + # TRTLLM_M31_DISABLE_S10C (all of S10c) / TRTLLM_M31_DISABLE_S10C_E, read at import. + s10c_e = self.enable_fused_forward_indices( + draft_model, None if m31_disabled("S10C") or m31_disabled("S10C_E") else forward_indices + ) + logger.info( + f"M3.1 DSpark S10c-E: fused drafter forward indices {'on' if s10c_e else 'off'} on rank {rank}" + ) + # EX-11: the drafter block's q / k RMSNorm + partial RoPE in one launch (bitwise to flashinfer's three). Kill + # switch TRTLLM_M31_DISABLE_EX11, read at import. + ex11 = self.enable_fused_qk_norm_rope( + draft_model, None if m31_disabled("EX11") else qk_norm_rope + ) + logger.info( + f"M3.1 DSpark EX-11: fused drafter q/k norm + RoPE {'on' if ex11 else 'off'} on rank {rank}" + ) + # DRAFT_APPEND_FUSE: on top of EX-11, the same launch also writes the block's K / V into the layer's paged + # cache (bitwise flashinfer's append), and the per-layer append launch is skipped. Kill switch + # TRTLLM_M31_DISABLE_DRAFT_APPEND_FUSE, read at import. + append_fuse = self.enable_fused_qk_norm_rope_append( + draft_model, _qk_norm_rope_append_logged if ex11 and DRAFT_APPEND_FUSE else None + ) + logger.info( + f"M3.1 DSpark DRAFT_APPEND_FUSE: drafter K / V append in the EX-11 launch {'on' if append_fuse else 'off'} " + f"on rank {rank}" + ) + # MXDRAFT: the drafter's qkv / o / gate_up / down GEMMs on this target's MXFP8 GEMV for the decode passes' + # rows (B x K, B <= MAX_FAST_BATCH, up to the GEMV's own row bound), from the exact MXFP8 form of the shipped + # weights (drafter-only precision, gated on natural AL). Kill switch TRTLLM_M31_DISABLE_MXDRAFT, read + # at import. + mx = ( + 0 + if m31_disabled("MXDRAFT") or k <= 0 + else route_mx_linears(draft_model, [b * k for b in range(1, MAX_FAST_BATCH + 1)]) + ) + logger.info(f"M3.1 DSpark MXDRAFT: MXFP8 drafter GEMV on {mx} projections on rank {rank}") + # DRAFT_GU_SWIGLU_EPI (F5, exact): the drafter MLP's SwiGLU in its gate_up GEMV's epilogue (one launch fewer per + # drafter MLP), on MXDRAFT's rows. Kill switch TRTLLM_M31_DISABLE_DRAFT_GU_SWIGLU_EPI. + gus = ( + 0 + if m31_disabled("MXDRAFT") or m31_disabled("DRAFT_GU_SWIGLU_EPI") or k <= 0 + else route_gu_swiglu(draft_model, [b * k for b in range(1, MAX_FAST_BATCH + 1)]) + ) + logger.info( + f"M3.1 DSpark DRAFT_GU_SWIGLU_EPI: gate_up SwiGLU epilogue on {gus} drafter MLPs on rank {rank}" + ) + self.prebuild_s4(lm_head, draft_model) + # N3 A2 compiles (rank-local), then the N3 A1 workspace and compiles (collective), outside capture. + if not N3A_DISABLED and lm_head is not None: + _strict_accept.precompile((self.max_draft_len,), lm_head.weight.device) + self._a1 = self._prebuild_a1(lm_head) if lm_head is not None else None + self._mcag_cols = self._prebuild_mcag(lm_head) if self._a1 is not None and MCAG else 0 + mcag = self.enable_draft_logits_gather_op(self._mcag_gather if self._mcag_cols else None) + logger.info( + f"M3.1 MCAG: draft-logits all-gather on the multicast kernel {'on' if mcag else 'off'} on rank {rank}" + ) + n3a_kv = False + if not N3A_KV_DISABLED and lm_head is not None: + _strict_accept.precompile_kv_rewind(lm_head.weight.device) + self._n3a_kv_buf = torch.empty( + _strict_accept.max_requests(), dtype=torch.int32, device=lm_head.weight.device + ) + n3a_kv = self.enable_fused_kv_rewind(self._n3a_kv_rewind) + logger.info( + f"M3.1 N3A-KV: one-launch draft-forward KV bookkeeping {'on' if n3a_kv else 'off'} on rank " + f"{getattr(getattr(self, 'mapping', None), 'tp_rank', None)}" + ) + + _n3a_kv_buf = None # the persistent rewind amounts of N3A-KV (int32, MAX_REQUESTS) + _n3a_kv_counts = None # gate-only cross-check counters (calls, mismatches) + + def _n3a_kv_rewind(self, num_accepted, kv_lens, nc: int, bs: int): + """N3A-KV for the DFlash hook: ``1 - num_accepted[nc:bs]`` (a view of the persistent buffer) with + ``kv_lens[nc:bs] += 1`` in one launch, or None (the stock lines) for operands or a variant it does not take.""" + n = bs - nc + buf = self._n3a_kv_buf + if buf is None or not _strict_accept.supports_kv_rewind(num_accepted, kv_lens, buf, nc, n): + return None + if not _strict_accept.is_compiled_kv_rewind(num_accepted, kv_lens): + return None + _n3a_kv_log_once() + if not N3A_KV_CHECK: + return _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + want_rewind = 1 - num_accepted[nc:bs] + want_kv = kv_lens[nc:bs] + 1 + out = _strict_accept.spec_kv_rewind(num_accepted, kv_lens, buf, nc, n) + if self._n3a_kv_counts is None: + self._n3a_kv_counts = torch.zeros(2, dtype=torch.int64, device=kv_lens.device) + bad = (out != want_rewind).any() | (kv_lens[nc:bs] != want_kv).any() + self._n3a_kv_counts[0] += 1 + self._n3a_kv_counts[1] += bad.to(torch.int64) + if not torch.cuda.is_current_stream_capturing(): + calls, mism = self._n3a_kv_counts.tolist() + logger.info(f"M3.1 N3A-KV check: {calls} calls, {mism} mismatches") + return out + + _a1 = None # the N3 A1 workspace (allgather_cast's arguments) once set up + + def _a1_all_ok(self, local: bool) -> bool: + """Whether ``local`` holds on every rank of the TP group: A1 is set up on every rank or on none (a rank that + is not eligible, or whose compile failed, turns it off everywhere).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + return _mnnvl_workspace_all_succeeded(_get_mnnvl_workspace_comm(self.mapping), local) + + def _prebuild_a1(self, lm_head: nn.Module): + """The N3 A1 workspace and its K3 verify-row compiles, or None (every rank alike): off by its switch, at + another draft length, without a column-parallel bf16 head over a TP group, or when a compile fails on any + rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + mapping = self.mapping + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return None + cols = int(getattr(lm_head, "out_features", 0)) + sms = torch.cuda.get_device_properties(lm_head.weight.device).multi_processor_count + # Why this rank would not set A1 up (empty: it would). The grid is a function of rows and the SM count: + # requiring every SM up to the workspace's MAX_GRID makes it a function of rows alone, the same on every rank. + off = [ + why + for why, bad in ( + ( + "TRTLLM_M31_DISABLE_A1 set", + A1_DISABLED, + ), + ( + f"draft length {int(self.max_draft_len)} with TRTLLM_M31_DISABLE_A1_K7 set", + int(self.max_draft_len) != 3 and A1_K7_DISABLED, + ), + ( + f"draft length {int(self.max_draft_len)}: more than {A1_MAX_ROWS} verify rows at batch 1", + int(self.max_draft_len) + 1 > A1_MAX_ROWS, + ), + ( + "not a column-parallel, gathered, unpadded, bias-free head", + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or getattr(lm_head, "bias", None) is not None, + ), + (f"head dtype {lm_head.weight.dtype}", lm_head.weight.dtype != torch.bfloat16), + ( + f"{cols} shard columns (whole 16-byte vectors needed)", + cols <= 0 or cols % 8 != 0, + ), + ( + f"{sms} SMs (fewer than {_allgather_cast.MAX_GRID})", + sms < _allgather_cast.MAX_GRID, + ), + ) + if bad + ] + if not self._a1_all_ok(not off): + logger.info( + f"M3.1 N3 A1: off on rank {mapping.tp_rank} ({'; '.join(off) or 'off on another rank'}); " + "the verify logits use the NCCL all-gather" + ) + return None + k1 = int(self.max_draft_len) + 1 + rows = tuple(b * k1 for b in range(1, A1_MAX_ROWS // k1 + 1)) + start = time.perf_counter() + if not _allgather_cast.precompile( + lm_head.mapping, cols, rows, lm_head.weight.device, agree=self._a1_all_ok + ): + logger.warning( + "M3.1 N3 A1: a compile failed on some rank; every rank keeps the NCCL all-gather" + ) + return None + ws = _allgather_cast.workspace_args(lm_head.mapping, cols) + # The engagement line: one per rank, at server start. + logger.info( + f"M3.1 N3 A1: one-launch verify all-gather set up on rank {mapping.tp_rank}/{mapping.tp_size} " + f"(shard {cols} columns, rows {rows}, workspace {ws['ws_uc'].numel() * 4} bytes/rank, " + f"set-up {time.perf_counter() - start:.2f} s)" + ) + return ws + + def _prebuild_mcag(self, lm_head: nn.Module) -> int: + """Compile the fp32 draft-logits gather for B * K rows up to its capacity, on every rank (A1's workspace and + agreement); its shard columns, or 0 (every rank alike) when a compile fails anywhere.""" + cols = int(getattr(lm_head, "out_features", 0)) + k = int(self.max_draft_len) + if k <= 0 or cols % 4 != 0: + return 0 + rows = tuple(b * k for b in range(1, _allgather_cast.MAX_ROWS_FP32 // k + 1)) + if not _allgather_cast.precompile_f32( + lm_head.mapping, cols, rows, lm_head.weight.device, agree=self._a1_all_ok + ): + logger.warning( + "M3.1 MCAG: a compile failed on some rank; every rank keeps the NCCL all-gather" + ) + return 0 + return cols + + def _mcag_gather(self, logits: torch.Tensor): + """The draft logits gathered in one launch on the A1 workspace, or None (the stock all-gather) for a shape + that is not compiled; the decision depends on the shape only, the same on every rank.""" + rows = logits.numel() // max(int(logits.shape[-1]), 1) + if ( + logits.dtype != torch.float32 + or int(logits.shape[-1]) != self._mcag_cols + or not _allgather_cast.is_compiled( + rows, self._mcag_cols, self._a1["world"], device=logits.device, fp32=True + ) + ): + _a1_log_once( + ("mcag", rows, logits.dtype), + f"[M3 dispatch] MCAG draft-logits all-gather not taken: {rows} rows {logits.dtype}; NCCL all-gather", + ) + return None + _a1_log_once( + "mcag", + "[M3 dispatch] MCAG draft-logits all-gather taken: one launch on the A1 workspace", + ) + return _allgather_cast.gather_last_dim_f32(logits.contiguous(), **self._a1) + + def _n3a_eligible( + self, logits, draft_tokens, num_contexts: int, batch_size: int, spec_metadata + ) -> bool: + """Whether this step's strict acceptance takes the fused op: a decode-only, non-ragged step whose logits hold + exactly batch_size x (D + 1) verify rows and whose drafts the op takes. Host state and shapes only, so it is + fixed per captured graph and the same on every rank. Decided before the target tokens are sampled, so the + sampler runs exactly once either way.""" + if N3A_DISABLED or spec_metadata is None or spec_metadata.is_ragged_verify: + return False + if num_contexts != 0 or batch_size <= 0 or draft_tokens.dim() != 2: + return False + n_req, draft_len = draft_tokens.shape + if n_req != batch_size or draft_len <= 0 or n_req > _strict_accept.max_requests(): + return False + rows = logits.shape[0] if logits.dim() > 1 else 1 + return rows == batch_size * (draft_len + 1) and draft_tokens.is_contiguous() + + def _sample_and_accept_draft_tokens_base( + self, logits, draft_tokens, num_contexts, batch_size, spec_metadata + ): + """The in-tree strict acceptance with its epilogue in one launch (N3 A2): the same target tokens (the in-tree + sampler, called once), the same accepted tokens and counts, then the in-tree forced-acceptance override (its + RNG pool and counters advance as on an in-tree step).""" + if not self._n3a_eligible(logits, draft_tokens, num_contexts, batch_size, spec_metadata): + return super()._sample_and_accept_draft_tokens_base( + logits, draft_tokens, num_contexts, batch_size, spec_metadata + ) + if logits.dim() == 1: + logits = logits.unsqueeze(0) + target_tokens = self._sample_tokens_for_batch( + logits, spec_metadata, num_contexts, batch_size + ) + if _strict_accept.supports_strict_accept( + target_tokens, draft_tokens + ) and _strict_accept.is_compiled_strict_accept(target_tokens, draft_tokens): + accepted, num_accepted = _strict_accept.spec_strict_accept(target_tokens, draft_tokens) + _n3a_log_once() + else: + # The sampler already ran: finish with the in-tree epilogue (the op's reference is that code). + accepted, num_accepted = _strict_accept.spec_strict_accept.reference( + target_tokens, draft_tokens + ) + num_accepted = self._apply_force_accepted_tokens( + num_accepted, num_contexts, draft_tokens.shape[-1], spec_metadata=spec_metadata + ) + return accepted, num_accepted + + def target_logits(self, hidden_states, attn_metadata, spec_metadata, lm_head: nn.Module): + """The verify rows' gathered fp32 target logits (N3 P-gid + P-cat), or None for the logits processor. + + On a decode-only, non-ragged step every row is a verify row and ``gather_ids`` is ``arange(num_tokens)`` (each + generation request extends it with the next ``1 + D`` row indices), so the rows are a slice, not a gather. + The processor's head is the column-parallel BF16 GEMM, an all-gather along the vocab (NCCL, then a concat) + and ``.float()``; here the same all-gather runs along dim 0 (no concat) and one strided copy writes the + rank-major shards into the fp32 output: the same bytes, an exact widening. Shapes and host state only, so + the decision is fixed per captured graph and the same on every rank.""" + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + if ( + N3P_DISABLED + or spec_metadata is None + or getattr(spec_metadata, "is_ragged_verify", False) + ): + return None + if attn_metadata.num_contexts != 0 or spec_metadata.gather_ids is None: + return None + rows = hidden_states.shape[0] + if spec_metadata.gather_ids.shape[0] != rows or rows == 0: + return None + if ( + getattr(lm_head, "tp_mode", None) != TensorParallelMode.COLUMN + or not getattr(lm_head, "gather_output", False) + or getattr(lm_head, "padding_size", 0) != 0 + or getattr(lm_head, "bias", None) is not None + ): + return None + shard = lm_head.apply_linear(hidden_states, None) + _n3p_log_once() + tp = lm_head.tp_size + if tp == 1: + return shard.float() + # N3 A1: one launch for the all-gather, the concat and the cast. The decision depends on the rows and the + # set-up only (the same on every rank, fixed per captured graph). + if self._a1 is not None: + if rows <= A1_MAX_ROWS and _allgather_cast.is_compiled( + rows, shard.shape[-1], tp, 0, shard.device + ): + _a1_log_once( + "taken", + "[M3 dispatch] n3_a1 allgather_cast taken: verify logits gathered in one launch", + ) + return _allgather_cast.allgather_cast(shard, **self._a1) + _a1_log_once( + rows, + f"[M3 dispatch] n3_a1 allgather_cast not taken: {rows} verify rows (compiled: verify rows up to " + f"{A1_MAX_ROWS}); NCCL all-gather", + ) + gathered = allgather(shard, lm_head.mapping, dim=0) + vocab_shard = shard.shape[-1] + out = torch.empty((rows, tp * vocab_shard), dtype=torch.float32, device=shard.device) + out.view(rows, tp, vocab_shard).copy_(gathered.view(tp, rows, vocab_shard).transpose(0, 1)) + return out + + def _prepare_next_new_tokens( + self, + accepted_tokens, + next_draft_tokens, + batch_indices_cuda, + batch_size, + num_accepted_tokens, + ): + """The next step's input tokens in one launch (N3 A2), else the in-tree index + concat.""" + batch_idx = batch_indices_cuda[:batch_size] + if ( + not N3A_DISABLED + and next_draft_tokens.dim() == 2 + and next_draft_tokens.shape[0] == batch_size + and _strict_accept.supports_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + and _strict_accept.is_compiled_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + ): + return _strict_accept.spec_next_new_tokens( + accepted_tokens, num_accepted_tokens, batch_idx, next_draft_tokens + ) + return super()._prepare_next_new_tokens( + accepted_tokens, next_draft_tokens, batch_indices_cuda, batch_size, num_accepted_tokens + ) + + # TAILBK (RuntimeHooks.spec_tail_bookkeep, armed by modeling.py's TAILBK; spec_metadata.tail_bookkeep): R1 and Z1, + # this worker's part of the sampler tail bookkeeping (B1 / S1 / A1 are the shared branches in interface.py). + _tailbk_win = None # R1: (spec_metadata, offsets window) the accept call left for this forward's draft block + + def _forward_impl(self, *args, **kwargs): + """The S4 / DFlash step; R1's offsets window lives for one step only (set by its accept call, read by its + draft block), so a step whose accept did not stash one recomputes it.""" + self._tailbk_win = None + try: + return super()._forward_impl(*args, **kwargs) + finally: + self._tailbk_win = None + + def _rng_state_per_request(self, spec_metadata, start=0, end=None, repeat=1, *, slot): + """TAILBK R1: at one decode request, the accept and draft-block Philox (seed, offset) rows read from the + per-token buffers (one int64 add per step instead of eight launches), else the in-tree expansion. + + ``_populate_request_rng_state`` writes row r (0..K) of the request's per-token window as + ``seeds[r] = seed`` and ``offsets[r] = o + r * SPAN`` (o = ``request_offsets[0]``; the host sum fits int64, + or its upload would have raised), uploaded with ``request_seeds`` / ``request_offsets`` on every sampled step. + The in-tree rows are ``(o + slot * SPAN) + k * SPAN``: draft row k at slot K + 1 is ``offsets[k] + (K + 1) * + SPAN`` and the accept row at slot 2K + 1 is ``offsets[K] + (K + 1) * SPAN``. Integer addition is associative + and commutative mod 2^64, so the window ``offsets[0:K + 1] + (K + 1) * SPAN`` holds exactly the in-tree + offsets, even where they wrap, and the seeds are the request's seed by construction. The returned tensors are + contiguous int64 views with one entry per row, as the in-tree ones. Preconditions are host state and shapes + (``tail_bookkeep_draft_len``), so the choice is fixed per captured graph.""" + state = self._tailbk_rng_window(spec_metadata, start, end, repeat, slot) + if state is not None: + return state + return super()._rng_state_per_request(spec_metadata, start, end, repeat, slot=slot) + + def _tailbk_rng_window(self, spec_metadata, start, end, repeat: int, slot: int): + """R1's (seed, offset) rows, or None (the in-tree expansion) outside its preconditions.""" + k = tail_bookkeep_draft_len(spec_metadata, start, end) + if k == 0: + return None + first = spec_metadata.rng_draft_slot(0) + if first != k + 1 or spec_metadata.rng_accept_slot != first + k: + return None + accept = repeat == 1 and slot == spec_metadata.rng_accept_slot + block = repeat == k and slot == first + if not (accept or block): + return None + stash = self._tailbk_win + if block and stash is not None and stash[0] is spec_metadata: + win = stash[1] + else: + win = spec_metadata.offsets[: k + 1] + first * _RNG_SLOT_SPAN + tail_bookkeep_log_once( + "R1", "Philox seeds / offsets from the per-token window (one add per step)" + ) + if accept: + self._tailbk_win = (spec_metadata, win) + return spec_metadata.request_seeds[0:1], win[k : k + 1] + self._tailbk_win = None + return spec_metadata.seeds[:k], win[:k] + + def _zero_padding_rows( + self, + logits, + spec_metadata, + num_contexts: int, + batch_size: int, + rows_per_request: int, + *, + packed_verify: bool = False, + ): + """TAILBK Z1: no padding-row zeroing while a batch-1 decode graph is captured, else the in-tree zeroing. + + Why the in-tree result is a bitwise copy of ``logits`` on every replay of such a graph: + (i) a batch-1 graph replays exactly one real request. ``CUDAGraphRunner._get_padded_batch`` only appends + dummies after the batch's own requests, up to the graph's size, and the executor forwards no empty batch + without attention DP (``_can_queue``: batch_size > 0), so a size-1 graph never holds a dummy. Under + attention DP an empty rank does run a dummy, so attention DP is excluded. + (ii) a real request's slot is its ``py_seq_slot`` in ``[0, slot_capacity)`` and the padding marker is + ``dummy_slot_row == slot_capacity`` (``prepare_rejection_sampling_buffers``), so the padding mask is + all-false and the in-tree ``masked_fill`` copies ``logits`` unchanged. + (iii) every consumer is out-of-place (the non-fused probs: the softmax / top-k mask ops and the m31_softmax_mc + override allocate their outputs, top-p reads probs), so returning ``logits`` itself instead of that copy + exposes no write to the caller's verify logits. The fused modes are not audited and keep the copy. + (iv) the consumers see the layout the copy would have: the skip needs contiguous logits at a 256-byte aligned + address (the copy is a fresh contiguous allocation, which the caching allocator aligns at least that much), + since the m31_softmax_mc override takes contiguous, 16-byte aligned inputs only. + Eager steps (the graph warmup's dummy included) keep the in-tree zeroing; so do padded graphs (2..8), which + can hold dummies, and ragged verification (its layout check). Host state only, fixed per captured graph.""" + if self._tailbk_skips_padding_rows(logits, spec_metadata, num_contexts, batch_size): + tail_bookkeep_log_once("Z1", "padding-row zeroing skipped in captured batch-1 graphs") + return logits + return super()._zero_padding_rows( + logits, + spec_metadata, + num_contexts, + batch_size, + rows_per_request, + packed_verify=packed_verify, + ) + + def _tailbk_skips_padding_rows( + self, logits, spec_metadata, num_contexts: int, batch_size: int + ) -> bool: + """Z1's exact condition: tail bookkeeping on, one generation request and no context rows (num_gens == 1), + no ragged verification, a non-fused sampling mode, no attention DP, contiguous 256-byte aligned logits and a + capturing stream.""" + if not getattr(spec_metadata, "tail_bookkeep", False): + return False + if not logits.is_contiguous() or logits.data_ptr() % 256 != 0: + return False + if num_contexts != 0 or batch_size != 1 or getattr(spec_metadata, "is_ragged_verify", True): + return False + mode = getattr(spec_metadata, "advanced_sampling_mode", None) + if mode is None or mode.is_fused: + return False + mapping = getattr(self, "mapping", None) + if mapping is None or getattr(mapping, "enable_attention_dp", True): + return False + return torch.cuda.is_current_stream_capturing() + + +def build_dspark( + shell, model_config: ModelConfig +) -> Tuple[ModelConfig, nn.Module, M3DSparkWorker, bool]: + """The drafter config, the drafter and the worker of one-model standalone DSpark (mirrors + ``SpecDecOneEngineForCausalLM`` for an external drafter).""" + spec = model_config.spec_config + if not is_standalone_dspark(spec): + raise ValueError("build_dspark serves the standalone DSpark drafter only") + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, **external_drafter_config_kwargs(model_config, spec) + ) + # The drop is safe only because draft_config owns its QuantConfig: from_pretrained reads the drafter checkpoint + # and takes no quant_config from the target. Never pass the target's quant_config in here. + keep_drafter_kv_dtype(draft_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = get_draft_model(model_config, draft_config, shell.lm_head, shell.model) + # A parallel drafter (DSpark) projects its block rows through the target's logits processor, as the one-engine + # construction sets it. + draft.logits_processor = shell.logits_processor + worker = M3DSparkWorker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + logger.info( + f"M3.1 target DSpark: {type(worker).__name__} over {type(draft).__name__}, K={spec.max_draft_len}, " + f"separate draft KV cache={use_separate}" + ) + return draft_config, draft, worker, use_separate + + +def load_dspark_draft_weights(shell, weights, weight_mapper=None) -> None: + """The drafter's own checkpoint, then the target's embedding and head (mirrors + ``SpecDecOneEngineForCausalLM.load_draft_weights`` for DSpark). + + The drafter borrows the target's embedding and head, and ``M3EmbedTokens.weight`` IS the target's table, so a + drafter checkpoint that ships either would load into a module the target owns or be shadowed by it: refused. + """ + borrowed = sorted( + k for k in weights.keys() if "embed_tokens." in k or k.split(".")[-2:-1] == ["lm_head"] + ) + if borrowed: + raise ValueError( + f"the DSpark drafter checkpoint ships tensors this target lends it (embedding / head): {borrowed[:4]}" + ) + shell.draft_model.load_weights(weights=weights, weight_mapper=weight_mapper) + shell.draft_model.load_weights_from_target_model(shell) + if shell.draft_model.lm_head is not shell.lm_head or ( + shell.draft_model.draft_model_full.model.embed_tokens is not shell.model.embed_tokens + ): + raise RuntimeError("the DSpark drafter does not borrow this target's embedding and head") + own = sorted( + n for n, _ in shell.named_parameters() if not n.startswith(("model.w.", "draft_model.")) + ) + logger.info(f"M3.1 target DSpark: shell parameters outside the core and the drafter: {own}") + + +def keep_drafter_kv_dtype(draft_config: ModelConfig) -> None: + """Drop any KV quantization from the drafter's config, so its context pool is allocated in its projection dtype. + + The one-engine construction copies the target's KV algo onto the drafter; on this target that is NVFP4, which + allocates a packed int8 pool the worker refuses to bind (it falls back to its private arena, and a reused + prefix's drafter K/V is lost). The cached quant modes are dropped with it, as the draft KV budget does. + """ + for quant in [draft_config.quant_config, *(draft_config.quant_config_dict or {}).values()]: + quant.kv_cache_quant_algo = None + quant.__dict__.pop("quant_mode", None) + quant.__dict__.pop("layer_quant_mode", None) + + +def draft_kv_override_applies(spec_config) -> bool: + """The target's FP8 draft-KV override is an Eagle3 need; a DSpark drafter keeps its BF16 context pool.""" + return not is_standalone_dspark(spec_config) + + +__all__ = [ + "M3DSparkWorker", + "M3EmbedTokens", + "build_dspark", + "draft_kv_override_applies", + "is_standalone_dspark", + "keep_drafter_kv_dtype", + "load_dspark_draft_weights", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/dspark_s4.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/dspark_s4.py new file mode 100644 index 000000000000..b0aa53afabcb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/dspark_s4.py @@ -0,0 +1,547 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""S4: the catalog's one-launch Markov chain (``catalog/spec_decode/dspark_markov_chain``) in this target's DSpark +worker (each M3.1 target keeps its own copy of this glue: targets share only the catalog). + +``S4WorkerMixin`` goes before ``DSparkWorker`` in the target's worker. For a generation step it serves (switch not +set, 1 <= generation requests <= ``MAX_FAST_BATCH``, greedy drafting, no d2t / guided decoder / DFlash-2 path, a +vocab-sharded borrowed head): + +- the draft head computes this rank's vocab shard of the block logits (the head's local GEMM, no gather) and copies + it into a padded buffer whose tail holds -inf (the step's logits processor, swapped in for the step); +- ``_refine_block_logits`` runs the catalog op over the shards in one launch and keeps the tokens; +- ``sample_draft_tokens`` returns them. + +The tokens are bitwise the in-tree chain's on the gathered logits: the gathered logits are the concatenation of the +same per-rank GEMMs, and the op reproduces the chain's bf16 bias, bf16 add and first-maximum argmax. When the op +does not support a step after the logits were sharded, the in-tree sharded path (``vocab_slice`` + the TP-aware +argmax) runs instead, with the same tokens. + +Setup (``prebuild_s4``, from the worker's ``prebuild``, outside capture, on every rank): the padded ``markov_w2`` +shard, one padded logits buffer per (batch, block), the exchange workspace (collective) and the compiles. + +Every routing decision is Python, so it is fixed into each decode graph when the graph is captured: +``TRTLLM_M31_DRAFT_DISABLE_S4`` is read once, at import, not a switch that can be toggled at run time. The set-up +decision must be the same on every rank (the workspace is collective); it follows from the config and the +environment, and ``prebuild_s4`` checks that the ranks agree before the collective. + +Gate only: with ``TRTLLM_M31_DSPARK_GATE_DUMP=`` set at prebuild (eager runs), each served step also runs the +in-tree chain on the gathered logits and logs both token sets per rank to ``/rank.jsonl``. It synchronizes; +never set it in a perf arm. +""" + +from __future__ import annotations + +import atexit +import json +import os +from typing import Dict, Optional, Tuple + +import torch +import torch.nn.functional as F +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_bias as markov_bias, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_markov_chain as chain, +) +from tensorrt_llm._torch.models.modeling_speculative import dspark_markov_step_bias +from tensorrt_llm.logger import logger + +from .m31_switches import m31_disabled + +MARKOV_RANK = 256 +MAX_FAST_BATCH = ( + 8 # capacity: measured: above 8 requests the chain is within 10% of the reference (16: 0.93x) +) +DISABLE_ENV = "TRTLLM_M31_DRAFT_DISABLE_S4" +S4_COUNT_LOG_EVERY = 1000 # generation steps between the per-rank "served x/y" engagement lines +GATE_DUMP_ENV = "TRTLLM_M31_DSPARK_GATE_DUMP" # gate-only: a directory; never set in a perf arm (it synchronizes) + +# TAILBK_SPEC (DSpark-worker sites): bookkeeping launches +# of the draft tail removed exactly. Here, site C: the chain op's three read-only accepted inputs (ones, arange, +# zeros; 3 launches per step) come from a per-shape cache filled at precompile. Site H: the target's KV-length rewind +# after the draft forward (kv_lens[nc:bs] -= rewind; clamp_(min=0): 2 launches per step) runs in the chain's launch +# (its rewind path, after the grid-dependency wait; nothing between the chain and the stock rewind reads kv_lens). +# Site S4L (S4-logits, the advanced-sampling contract path): the biased block logits' glue around the bias GEMM -- the +# prevs cat / casts, get_masked_input_and_mask, F.embedding, masked_fill, the two widening copies and the add (14 +# launches per step) -- in two launches (catalog dspark_markov_bias; the GEMM unchanged). +# Bitwise the same tokens, kv_lens and biased logits. Default ON; kill switch TRTLLM_M31_DISABLE_TAILBK_SPEC=1, read +# at import. +TAILBK_SPEC = not m31_disabled("TAILBK_SPEC") +# MARKOV_LL (GB300 TP8 K7 b1 exchange 2.25 us + post 0.5 us per position +# against a ~1.25 us single-hop floor): the batch-1 kernel's per-position exchange as one 16-byte {value, tag, index, +# tag} multicast store per CTA (no fence, no flag word), polled and reduced by warp 0 alone (no CTA barrier after the +# poll). Bitwise the S4B8 kernel's tokens (same entries, same total argmax order). Default ON; kill switch +# TRTLLM_M31_DISABLE_MARKOV_LL=1, read at import. +MARKOV_LL = not m31_disabled("MARKOV_LL") + +# Read once, at import: every rank and every step of a process sees the same setting. Any value but "" or "0" turns +# the item off, as the targets' m31_switches reads its switches. +_DISABLED = os.environ.get(DISABLE_ENV, "").strip() not in ("", "0") + + +def disabled() -> bool: + return _DISABLED + + +class _S4State: + def __init__(self, mapping, rows: int, shard: int, offset: int, w1: torch.Tensor, w2s: torch.Tensor, + block: int, ws: dict) -> None: # fmt: skip + self.mapping = mapping + self.ws = ws + self.rows = rows + self.shard = shard + self.offset = offset + self.w1 = w1 + self.w2s = w2s + self.block = block + self.bufs: Dict[Tuple[int, int], torch.Tensor] = {} + + def buffer(self, batch: int) -> torch.Tensor: + return self.bufs[(batch, self.block)] + + +# S4-logits bias form: True = all K positions' bias in one GEMM over B*K rows (S4LC: bitwise the per-step GEMMs); +# False = the in-tree per-step loop over the chain's tokens (exact by construction). +S4_LOGITS_BATCHED = True + + +def _biased_shard_logits( + shard_logits: torch.Tensor, + first_prev: torch.Tensor, + tokens: torch.Tensor, + w1: torch.Tensor, + w2_shard: torch.Tensor, + fp32: bool, +) -> torch.Tensor: + """The in-tree chain's biased block logits on this rank's vocab shard, given the chain's tokens: step k adds + w2 @ w1[prev_k] (dspark_markov_step_bias) with prev_0 = the anchor and prev_k = token k-1, cast to the in-tree + logits dtype: fp32 under the S4 fp32 sum (LogitsProcessor returns logits.float()), else the shard's bf16.""" + if ( + fp32 + and S4_LOGITS_BATCHED + and TAILBK_SPEC + and markov_bias.supports(shard_logits, first_prev, tokens, w1) + ): + # TAILBK_SPEC S4L: the same prevs, embedding rows, GEMM and fp32 sum, in two launches around the GEMM. + B, K, N = shard_logits.shape + bias = F.linear(markov_bias.prev_embeddings(first_prev, tokens, w1), w2_shard) + return markov_bias.widen_add(shard_logits, bias.view(B, K, N)) + B, K, N = shard_logits.shape + base = shard_logits.float() if fp32 else shard_logits + prevs = torch.cat([first_prev.view(B, 1), tokens[:, : K - 1].long()], dim=1) + if S4_LOGITS_BATCHED: + bias = dspark_markov_step_bias(prevs.reshape(-1), w1, w2_shard).view(B, K, N) + return base + bias.to(base.dtype) + out = torch.empty_like(base) + for k in range(K): + out[:, k] = base[:, k] + dspark_markov_step_bias(prevs[:, k], w1, w2_shard).to(base.dtype) + return out + + +class _ShardedDraftLogits(nn.Module): + """The step's draft logits processor: this rank's vocab shard of ``lm_head`` over the gathered block rows, + copied into the padded buffer; any other call goes to the drafter's own processor.""" + + def __init__(self, inner: nn.Module, state: _S4State, batch: int) -> None: + super().__init__() + self.inner = inner + self._state = [state] + self.batch = batch + + def forward(self, hidden_states, lm_head, attn_metadata, return_context_logits: bool = False): + state = self._state[0] + if hidden_states.dim() != 2 or hidden_states.shape[0] != self.batch * state.block: + return self.inner(hidden_states, lm_head, attn_metadata, return_context_logits) + logits = lm_head.apply_linear(hidden_states, None) + buf = state.buffer(self.batch) + buf[:, :, : state.shard].copy_(logits.view(self.batch, state.block, state.shard)) + return logits + + +def _head_is_sharded(lm_head: nn.Module, vocab: int, tp: int) -> bool: + from tensorrt_llm._torch.modules.linear import TensorParallelMode + + return ( + tp > 1 + and getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and getattr(lm_head, "padding_size", 0) == 0 + and getattr(lm_head, "bias", None) is None + and lm_head.out_features * tp == vocab + and lm_head.weight.dtype == torch.bfloat16 + ) + + +class S4WorkerMixin: + """Mixed into the target's ``DSparkWorker`` subclass (before it in the bases).""" + + _s4: Optional[_S4State] = None + _s4_step: Optional[_S4State] = None + _s4_tokens: Optional[torch.Tensor] = None + _s4_step_meta: Optional[Tuple[int, int, object]] = None + _s4_dump_step: int = 0 + _s4_dump_dir: Optional[str] = None + _s4_draft: Optional[nn.Module] = None + _s4_served: int = 0 + _s4_gen_steps: int = 0 + _s4_step_attn = ( + None # the step's attention metadata (the KV-length rewind's kv_lens), while S4 serves it + ) + _tailbk_rewound: bool = ( + False # TAILBK_SPEC H: this step's chain launch applied the KV-length rewind + ) + _tailbk_h_logged: bool = False + _tailbk_s4l_logged: bool = False + _tailbk_c_logged: bool = False + _s4_ll_logged: bool = False + + def prebuild_s4(self, lm_head: nn.Module, draft_model) -> bool: + """Set up the one-launch chain for this worker; False (and nothing set up) when it does not apply.""" + self._s4 = None + self._s4_dump_dir = os.environ.get(GATE_DUMP_ENV) or None + if disabled(): + logger.info(f"M3.1 DSpark S4: off ({DISABLE_ENV} set); the in-tree Markov chain runs") + return False + if not getattr(draft_model, "has_markov_head", False): + return False + w1 = draft_model.markov_w1 + w2 = draft_model.markov_w2 + mapping = self.mapping + tp = mapping.tp_size + vocab = w1.shape[0] + if ( + w1.dtype != torch.bfloat16 + or w2.dtype != torch.bfloat16 + or w1.shape[1] != MARKOV_RANK + or tuple(w2.shape) != tuple(w1.shape) + or mapping.enable_attention_dp + or not _head_is_sharded(lm_head, vocab, tp) + ): + return False + block = int(self.max_draft_len) + shard = vocab // tp + rows = chain.padded_rows(shard, MAX_FAST_BATCH, w1.device) + if not self._s4_ranks_agree(rows != 0 and block <= chain.MAX_BLOCK): + return False + offset = mapping.tp_rank * shard + w2s = chain.pad_w2_shard(w2[offset : offset + shard].contiguous(), rows) + state = _S4State( + mapping, rows, shard, offset, w1, w2s, block, chain.workspace_args(mapping) + ) + for batch in range(1, MAX_FAST_BATCH + 1): + state.bufs[(batch, block)] = chain.pad_logits_buffer(batch, block, rows, w1.device) + # S4 fp32 sum: the chain compares base + bias as the in-tree chain does, the unrounded fp32 sum (its logits are + # fp32 from LogitsProcessor's .float()). Kill switch TRTLLM_M31_DISABLE_S4_FP32SUM (bf16-rounded sums). + self._s4_fp32 = chain.enable_fp32_sum(not m31_disabled("S4_FP32SUM")) + # S4B8: batch-1 calls take the chain's batch-1 kernel (bitwise the same tokens; faster at one request only). + # Kill switch TRTLLM_M31_DISABLE_S4B8, read at import. + s4b8 = chain.enable_batch1_variant(not m31_disabled("S4B8")) + # TAILBK_SPEC site C: the chain's read-only accepted inputs from a per-shape cache (bitwise; see TAILBK_SPEC). + tailbk_c = chain.enable_cached_inputs(TAILBK_SPEC) + # MARKOV_LL: those batch-1 calls exchange through the LL flag-in-payload kernel (bitwise; see MARKOV_LL). + markov_ll = chain.enable_ll_exchange(MARKOV_LL) + # S4-logits: under advanced (rejection) draft sampling the chain still computes the greedy chain's tokens; the + # biased block logits are rebuilt from them (_biased_shard_logits) and the stock sampler gathers and samples + # them as it would the in-tree chain's. Kill switch TRTLLM_M31_DISABLE_S4_LOGITS, read at import. + self._s4_logits_on = not m31_disabled("S4_LOGITS") + chain.precompile(mapping, rows, w1, w2s, offset, blocks=(block,), max_batch=MAX_FAST_BATCH) + self._s4 = state + self._s4_draft = draft_model + atexit.register(self._s4_log_count) + # The engagement line: one per rank, at server start. + logger.info( + f"M3.1 DSpark S4: one-launch Markov chain set up on rank {mapping.tp_rank}/{tp} " + f"(shard {shard} rows padded to {rows}, K={block}, batches 1..{MAX_FAST_BATCH}; fp32 sum " + f"{'on' if self._s4_fp32 else 'off'}; S4-logits {'on' if self._s4_logits_on else 'off'}; " + f"S4B8 batch-1 kernel {'on' if s4b8 else 'off'}; MARKOV_LL {'on' if markov_ll and s4b8 else 'off'}; " + f"TAILBK_SPEC cached inputs {'on' if tailbk_c else 'off'})" + ) + return True + + def _s4_ranks_agree(self, local: bool) -> bool: + """``local`` if every rank of the TP group decided the same; raises when they differ (a rank going on into + the collective workspace alone would hang its peers).""" + from tensorrt_llm._torch.distributed.ops import ( + _get_mnnvl_workspace_comm, + _mnnvl_workspace_all_succeeded, + ) + + comm = _get_mnnvl_workspace_comm(self.mapping) + all_on = _mnnvl_workspace_all_succeeded(comm, local) + all_off = _mnnvl_workspace_all_succeeded(comm, not local) + if not (all_on or all_off): + raise RuntimeError( + "m31 DSpark S4: the TP ranks disagree on setting up the one-launch Markov chain" + ) + return all_on + + def _s4_serves(self, num_gens: int, spec_metadata, draft_model) -> bool: + return self._s4_mode(num_gens, spec_metadata, draft_model) is not None + + def _s4_mode(self, num_gens: int, spec_metadata, draft_model): + """ "tokens" (greedy drafting: the chain's tokens are the draft), "logits" (advanced draft sampling with + S4-logits on: the chain's corrected logits go to the stock sampler), or None (the in-tree chain runs).""" + if not self._s4_common(num_gens, draft_model): + return None + if not getattr(spec_metadata, "wants_advanced_draft_sampling", False): + return "tokens" + return "logits" if getattr(self, "_s4_logits_on", False) else None + + def _s4_common(self, num_gens: int, draft_model) -> bool: + return ( + self._s4 is not None + and 1 <= num_gens <= MAX_FAST_BATCH + and getattr(self, "guided_decoder", None) is None + and getattr(self, "_d2t", None) is None + and getattr(draft_model, "lilicorr", None) is None + and not getattr(draft_model, "has_candidate_selector", False) + ) + + def _host_step_hook(self, spec_metadata) -> None: + """The engagement counter (host only, no CUDA): called by the spec metadata's prepare() on every iteration, + outside any captured graph, it evaluates the same eligibility the captured step was routed by and counts + the generation steps the chain serves. One line per rank every S4_COUNT_LOG_EVERY generation steps and at + exit: "M3.1 DSpark S4: served / generation steps". The counts start at server start, so they include + warmup and padding steps: read engagement as the difference of two lines logged after the server is ready.""" + if self._s4 is None: + return + num_gens = int(getattr(spec_metadata, "num_generations", 0) or 0) + if num_gens <= 0: + return + self._s4_gen_steps += 1 + self._s4_served += int(self._s4_serves(num_gens, spec_metadata, self._s4_draft)) + if self._s4_gen_steps % S4_COUNT_LOG_EVERY == 0: + self._s4_log_count() + + def _s4_log_count(self) -> None: + if self._s4 is not None and self._s4_gen_steps: + logger.info( + f"M3.1 DSpark S4: served {self._s4_served}/{self._s4_gen_steps} generation steps on rank " + f"{self.mapping.tp_rank}" + ) + + def _forward_impl(self, input_ids, position_ids, hidden_states, logits, attn_metadata, spec_metadata, draft_model, + resource_manager=None): # fmt: skip + num_gens = attn_metadata.num_seqs - attn_metadata.num_contexts + mode = self._s4_mode(num_gens, spec_metadata, draft_model) + served = mode is not None + if self._s4_dump_dir and not torch.cuda.is_current_stream_capturing(): + # The gate's independent denominator: every generation step, its eligibility by batch and sampling + # alone, and whether the chain served it. + eligible = 1 <= num_gens <= MAX_FAST_BATCH and not getattr( + spec_metadata, "wants_advanced_draft_sampling", False + ) + self._s4_dump_line({"kind": "step", "num_gens": num_gens, "eligible": bool(eligible), + "served": bool(served)}) # fmt: skip + if not served: + return super()._forward_impl(input_ids, position_ids, hidden_states, logits, attn_metadata, + spec_metadata, draft_model, resource_manager) # fmt: skip + inner = draft_model.logits_processor + draft_model.logits_processor = _ShardedDraftLogits(inner, self._s4, num_gens) + self._s4_step = self._s4 + self._s4_step_mode = mode + self._s4_step_meta = (attn_metadata.num_contexts, attn_metadata.num_seqs, spec_metadata) + self._s4_step_attn = attn_metadata + try: + return super()._forward_impl(input_ids, position_ids, hidden_states, logits, attn_metadata, + spec_metadata, draft_model, resource_manager) # fmt: skip + finally: + draft_model.logits_processor = inner + self._s4_step = None + self._s4_step_mode = None + self._s4_tokens = None + self._s4_step_meta = None + self._s4_step_attn = None + self._tailbk_rewound = False + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + state = self._s4_step + if state is not None and gen_logits.dim() == 3 and gen_logits.shape[-1] == state.shard: + base = state.buffer(gen_logits.shape[0]) + if ( + chain.supports(base, state.w1, state.w2s) + and getattr(self, "_s4_step_mode", None) == "logits" + ): + # S4-logits: the chain's tokens are the in-tree greedy chain's (bitwise); the biased shard logits are + # rebuilt from them with the in-tree bias arithmetic and handed to the stock advanced sampler. + first_prev = inputs["first_prev_tokens"].long() + tokens = self._s4_chain(state, base, first_prev, spec_metadata) + self._s4_log_ll(base.shape[0]) + biased = _biased_shard_logits( + gen_logits, + first_prev, + tokens, + state.w1, + state.w2s[: state.shard], + getattr(self, "_s4_fp32", True), + ) + self._s4l_log(gen_logits, first_prev, tokens, state.w1) + if self._s4_dump_dir and not torch.cuda.is_current_stream_capturing(): + self._s4_logits_gate_dump( + draft_model, gen_logits, inputs, spec_metadata, biased + ) + return biased + if chain.supports(base, state.w1, state.w2s): + tokens = self._s4_chain( + state, base, inputs["first_prev_tokens"].long(), spec_metadata + ) + self._s4_log_ll(base.shape[0]) + self._s4_tokens = tokens + if self._s4_dump_dir and not torch.cuda.is_current_stream_capturing(): + self._s4_gate_dump(draft_model, gen_logits, inputs, spec_metadata, tokens) + return gen_logits + return super()._refine_block_logits(draft_model, gen_logits, inputs, spec_metadata) + + def _s4_chain( + self, state: _S4State, base: torch.Tensor, first_prev: torch.Tensor, spec_metadata + ): + """The chain launch; under TAILBK_SPEC (site H) it also applies the step's deferred KV-length rewind when + the stock _apply_kv_rewind_after_draft would (a real step: not the eager warmup of a graph batch, whose + kv_lens is restored wholesale), and marks it done for that method.""" + rewind = self._tailbk_rewind_args(base.shape[0], spec_metadata) + if rewind is None: + tokens = chain.dspark_markov_chain( + base, first_prev, state.w1, state.w2s, state.offset, **state.ws + ) + self._tailbk_c_log() + return tokens + kv_lens, amount, first = rewind + tokens = chain.dspark_markov_chain_rewind(base, first_prev, state.w1, state.w2s, state.offset, kv_lens, + amount, first, **state.ws) # fmt: skip + self._tailbk_rewound = True + self._tailbk_c_log() + if not self._tailbk_h_logged: + self._tailbk_h_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC H: KV-length rewind in the Markov chain launch " + f"on rank {self.mapping.tp_rank}" + ) + return tokens + + def _tailbk_c_log(self) -> None: + """The TAILBK_SPEC C engagement line: once per rank, after the first chain call that took its accepted inputs + from the precompile cache (host only; a captured step logs it while it is captured).""" + if self._tailbk_c_logged or not getattr(chain, "cached_inputs_taken", lambda: False)(): + return + self._tailbk_c_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC C: chain accepted inputs from precompile cache (taken) " + f"on rank {self.mapping.tp_rank}" + ) + + def _tailbk_rewind_args(self, batch: int, spec_metadata): + """(kv_lens, rewind amounts, first row) for the chain to apply, or None (the stock rewind runs).""" + if not TAILBK_SPEC or not getattr(self, "_kv_rewind_pending", False): + return None + amount = getattr(self, "_kv_rewind_amount", None) + attn = self._s4_step_attn + if amount is None or attn is None or not hasattr(attn, "kv_lens_cuda"): + return None + if spec_metadata.is_cuda_graph and not torch.cuda.is_current_stream_capturing(): + return None # the warmup: the stock method skips the rewind (kv_lens is restored wholesale) + nc, bs = self._kv_rewind_nc, self._kv_rewind_bs + if bs - nc != batch or amount.numel() != batch: + return None + kv_lens = attn.kv_lens_cuda + if not chain.supports_rewind(kv_lens, amount, nc): + return None + return kv_lens, amount, nc + + def _apply_kv_rewind_after_draft(self, attn_metadata, spec_metadata): + if self._tailbk_rewound: + # TAILBK_SPEC H: the chain launch applied it (see _s4_chain). + self._tailbk_rewound = False + self._kv_rewind_pending = False + return + super()._apply_kv_rewind_after_draft(attn_metadata, spec_metadata) + + def _s4l_log(self, shard_logits, first_prev, tokens, w1) -> None: + """The TAILBK_SPEC S4L engagement line: once per rank, on the first step whose biased logits took the + two-launch glue (host only; a captured step logs it while it is captured).""" + if self._tailbk_s4l_logged or not ( + getattr(self, "_s4_fp32", True) and S4_LOGITS_BATCHED and TAILBK_SPEC + ): + return + if markov_bias.supports(shard_logits, first_prev, tokens, w1): + self._tailbk_s4l_logged = True + logger.info( + "[M3 dispatch] TAILBK_SPEC S4L: biased block logits glue in two launches " + f"on rank {self.mapping.tp_rank}" + ) + + def _s4_log_ll(self, batch: int) -> None: + """The MARKOV_LL engagement line: once per rank, on the first served step whose chain call took the LL + exchange kernel (host only; a captured step logs it while it is captured).""" + if not self._s4_ll_logged and chain.ll_exchange_engaged(batch): + self._s4_ll_logged = True + logger.info( + "[M3 dispatch] MARKOV_LL: k3_markov LL flag-in-payload exchange (batch 1) " + f"on rank {self.mapping.tp_rank}" + ) + + def _s4_gate_dump(self, draft_model, shard_logits, inputs, spec_metadata, tokens) -> None: + """Gate-only (eager): the in-tree chain on the gathered logits of this same step, logged beside the + one-launch chain's tokens, one JSON line per step and rank.""" + from tensorrt_llm._torch.distributed import allgather + + full = allgather(shard_logits.contiguous(), self.mapping, dim=-1) + if getattr(self, "_s4_fp32", True): + full = full.float() # the in-tree chain's input: LogitsProcessor returns logits.float() + corrected = super()._refine_block_logits(draft_model, full, inputs, spec_metadata) + stock = torch.argmax(corrected, dim=-1) + num_contexts, num_seqs, meta = self._s4_step_meta or (0, tokens.shape[0], None) + ids = getattr(meta, "request_ids", None) + ids = list(ids[num_contexts:num_seqs]) if ids is not None else None + self._s4_dump_line({"kind": "tokens", "step": self._s4_dump_step, "request_ids": ids, "s4": tokens.tolist(), + "stock": stock.tolist(), "equal": bool(torch.equal(tokens.long(), stock))}) # fmt: skip + self._s4_dump_step += 1 + + def _s4_logits_gate_dump( + self, draft_model, shard_logits, inputs, spec_metadata, biased + ) -> None: + """Gate-only (eager): the in-tree chain on the gathered logits of this same step (full-vocab bias), compared + bitwise with S4-logits' gathered biased logits, one JSON line per step and rank.""" + from tensorrt_llm._torch.distributed import allgather + + full = allgather(shard_logits.contiguous(), self.mapping, dim=-1) + if getattr(self, "_s4_fp32", True): + full = full.float() # the in-tree chain's input: LogitsProcessor returns logits.float() + stock = super()._refine_block_logits(draft_model, full, inputs, spec_metadata) + ours = allgather(biased.contiguous(), self.mapping, dim=-1) + num_contexts, num_seqs, meta = self._s4_step_meta or (0, biased.shape[0], None) + ids = getattr(meta, "request_ids", None) + ids = list(ids[num_contexts:num_seqs]) if ids is not None else None + mism = int((ours != stock).sum()) + self._s4_dump_line({"kind": "logits", "step": self._s4_dump_step, "request_ids": ids, + "shape": list(ours.shape), "dtype": str(ours.dtype), "mismatch": mism, + "equal": mism == 0 and ours.dtype == stock.dtype, + "batched": S4_LOGITS_BATCHED}) # fmt: skip + self._s4_dump_step += 1 + + def _s4_dump_line(self, rec: dict) -> None: + rec["rank"] = self.mapping.tp_rank + path = os.path.join(self._s4_dump_dir, f"rank{self.mapping.tp_rank}.jsonl") + with open(path, "a") as f: + f.write(json.dumps(rec) + "\n") + + def sample_draft_tokens(self, logits, spec_metadata, batch_size, **kwargs): + tokens = self._s4_tokens + if tokens is not None and logits.dim() == 3 and tokens.shape == logits.shape[:2]: + self._s4_tokens = None + return tokens if tokens.dtype == torch.int32 else tokens.to(torch.int32) + return super().sample_draft_tokens(logits, spec_metadata, batch_size, **kwargs) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/__init__.py new file mode 100644 index 000000000000..c5fd79e59d7f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Kernels of the MiniMax-M3 NVFP4 / sm_107 / tp8 target.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/__init__.py new file mode 100644 index 000000000000..4c6fc92cd49c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 decode embedding gather + first-layer RMSNorm as one CuTe DSL kernel (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py new file mode 100644 index 000000000000..78563b4db40d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/m3_embedding_firstnorm_kernel.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fixed M3 H6144 gather/raw residual/RMS, reusing pinned K3 norm kernel. + +128 threads x 8 BF16 x 6 iterations covers the row. Unlike the original +K3 wrapper this target does not promise FlashInfer's bit-identical reduction. +""" + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +from cutlass.experimental import primitives as prims + +NORM_THREADS = 128 +NORM_VEC = 8 +NORM_WARPS = 4 # capacity: fixed: the CTA's warps + + +@cute.kernel +def m3_embed_norm_kernel( + ids: cute.Tensor, # int32 or int64 [N] + table: cute.Tensor, # bf16 [V, H] + weight: cute.Tensor, # bf16 [H] + raw: cute.Tensor, # bf16 [N, H], out: the rows + out: cute.Tensor, # bf16 [N, H], out: the normed rows + vocab: cutlass.Int64, + eps: cutlass.Float32, + tv_layout: cute.Layout, + tiler_mn: cute.Shape, + hidden: cutlass.Constexpr[int], +): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + prims.griddepcontrol(prims.GridDepAction.WAIT) + + smem = cutlass.utils.SmemAllocator() + reduction_buffer = smem.allocate_tensor( + cutlass.Float32, cute.make_layout((1, NORM_WARPS)), byte_alignment=4 + ) + + tok = cutlass.Int64(ids[bidx]) + valid = (tok >= cutlass.Int64(0)) & (tok < vocab) + row = cutlass.Int64(cutlass.select_(valid, tok, cutlass.Int64(0))) + + gX = cute.local_tile(table, tiler_mn, (row, 0)) + gR = cute.local_tile(raw, tiler_mn, (bidx, 0)) + gY = cute.local_tile(out, tiler_mn, (bidx, 0)) + w_layout = cute.prepend(weight.layout, cute.make_layout((tiler_mn[0],), stride=(0,))) + gW = cute.local_tile(cute.make_tensor(weight.iterator, w_layout), tiler_mn, (0, 0)) + + copy_atom_load = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), table.element_type, num_bits_per_copy=128 + ) + copy_atom_store = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), out.element_type, num_bits_per_copy=128 + ) + thr_copy_X = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_W = cute.make_tiled_copy(copy_atom_load, tv_layout, tiler_mn).get_slice(tidx) + thr_copy_O = cute.make_tiled_copy(copy_atom_store, tv_layout, tiler_mn).get_slice(tidx) + + tXgX = thr_copy_X.partition_S(gX) + tXrX = cute.make_fragment_like(tXgX) + tWgW = thr_copy_W.partition_S(gW) + tWrW = cute.make_fragment_like(tWgW) + tXrW = thr_copy_X.retile(tWrW) + tXgR = thr_copy_O.partition_D(gR) + tXgO = thr_copy_O.partition_D(gY) + tXrO = cute.make_fragment_like(tXgO) + + # The row (zero for an id outside [0, V)), straight to the snapshot slot; the weight. + tXrX.store(cute.zeros_like(tXrX, dtype=table.element_type)) + if valid: + cute.copy(copy_atom_load, tXgX, tXrX) + cute.copy(copy_atom_load, tWgW, tWrW) + cute.copy(copy_atom_store, tXrX, tXgR) + + # flashinfer's RMSNormKernel arithmetic, in its order (see the header). + x = tXrX.load().to(cutlass.Float32) + x_sq = x * x + sum_sq = x_sq.reduce(cute.ReductionOp.ADD, init_val=cutlass.Float32(0.0), reduction_profile=0) + for i in cutlass.range_constexpr(5): + sum_sq = sum_sq + cute.arch.shuffle_sync_bfly(sum_sq, offset=1 << i) + lane = cute.arch.lane_idx() + warp = cute.arch.warp_idx() + if lane == 0: + reduction_buffer[0, warp] = sum_sq + cute.arch.barrier() + total = cutlass.Float32(0.0) + if lane < NORM_WARPS: + total = reduction_buffer[0, lane] + for i in cutlass.range_constexpr(5): + total = total + cute.arch.shuffle_sync_bfly(total, offset=1 << i) + mean_sq = total / cutlass.Float32(hidden) + rstd = cute.math.rsqrt(mean_sq + eps, fastmath=True) + w = tXrW.load().to(cutlass.Float32) + y = x * rstd * (w + cutlass.Float32(0.0)) + tXrO.store(y.to(out.element_type)) + cute.copy(copy_atom_store, tXrO, tXgO) + + +@cute.jit +def m3_embed_norm( + ids: cute.Tensor, + table: cute.Tensor, + weight: cute.Tensor, + raw: cute.Tensor, + out: cute.Tensor, + vocab: cutlass.Int64, + eps: cutlass.Float32, + n_tokens: cutlass.Constexpr[int], + hidden: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, +) -> None: + # flashinfer's thread-value layout for one row: thread t, value (v, k) -> column 8 t + v + 1024 k. + blocks = hidden // (NORM_VEC * NORM_THREADS) + tv_layout = cute.make_layout(((NORM_THREADS, 1), (NORM_VEC, blocks)), + stride=((NORM_VEC, 1), (1, NORM_VEC * NORM_THREADS))) # fmt: skip + m3_embed_norm_kernel( + ids, table, weight, raw, out, vocab, eps, tv_layout, (1, hidden), hidden + ).launch( + grid=[n_tokens, 1, 1], + block=[NORM_THREADS, 1, 1], + smem=NORM_WARPS * 4, + stream=stream, + use_pdl=use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/op.py new file mode 100644 index 000000000000..c12bd2500445 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/embedding_firstnorm/op.py @@ -0,0 +1,157 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Host side of the MiniMax-M3 decode embedding + first-layer RMSNorm (m3_embedding_firstnorm_kernel.py): H 6144, +1..8 rows, into caller-owned buffers. + +Compiled on the first call per (rows, id dtype, table shape, PDL), which must happen outside CUDA-graph capture +(:class:`DecodeEmbedding` compiles at construction). +""" + +import threading +from pathlib import Path +from typing import Dict, Tuple + +import cuda.bindings.driver as cuda_driver +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import from_dlpack + +from tensorrt_llm._torch import cute_aot + +from .m3_embedding_firstnorm_kernel import m3_embed_norm as _m3_embed_norm_jit + +HIDDEN = 6144 # capacity: fixed: the checkpoint's hidden size +MAX_ROWS = 8 # capacity: fixed: the kernel's built rows (decode steps) + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + + +def arg(tensor): + return from_dlpack( + tensor.detach(), + assumed_align=tensor.element_size() + if tensor.ndim == 1 and tensor.dtype in (torch.int32, torch.int64) + else 16, + ) + + +def _check(ids, table, weight, raw, out) -> None: + rows = ids.shape[0] if ids.dim() == 1 else 0 + if not ( + 1 <= rows <= MAX_ROWS + and ids.dtype in (torch.int32, torch.int64) + and ids.is_contiguous() + and table.dtype == torch.bfloat16 + and table.dim() == 2 + and table.shape[1] == HIDDEN + and table.is_contiguous() + and weight.dtype == torch.bfloat16 + and tuple(weight.shape) == (HIDDEN,) + and all( + t.dtype == torch.bfloat16 and tuple(t.shape) == (rows, HIDDEN) and t.is_contiguous() + for t in (raw, out) + ) + ): + raise ValueError( + "m3_embed_norm: expects ids int32/int64 [1..8], a contiguous bf16 table [V, 6144], bf16 weight [6144] " + "and contiguous bf16 raw / out [rows, 6144]" + ) + + +def _compile(ids, table, weight, eps, raw, out, use_pdl: bool): + key = (ids.shape[0], ids.dtype, tuple(table.shape), table.device, bool(use_pdl)) + fn = _compiled.get(key) + if fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_embed_norm compiles on its first call per shape: call it once before capture" + ) + with _lock: + fn = _compiled.get(key) + if fn is None: + stream = cuda_driver.CUstream(torch.cuda.current_stream(table.device).cuda_stream) + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_embed_norm {key[0]} {key[1]} {key[2]} {key[4]} eps={float(eps)!r}", + lambda: cute.compile( + _m3_embed_norm_jit, + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + ids.shape[0], + HIDDEN, + bool(use_pdl), + stream, + ), + sources=_aot_sources(), + ) + return fn + + +def m3_embed_norm( + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """Write ``table[ids]`` into ``raw`` (a zero row for an id outside ``[0, V)``) and its RMSNorm with ``weight`` / + ``eps`` into ``out``; returns ``(out, raw)``.""" + _check(ids, table, weight, raw, out) + fn = _compile(ids, table, weight, eps, raw, out, use_pdl) + stream = cuda_driver.CUstream(torch.cuda.current_stream(ids.device).cuda_stream) + fn( + arg(ids), + arg(table), + arg(weight), + arg(raw), + arg(out), + int(table.shape[0]), + float(eps), + stream, + ) + return out, raw + + +class DecodeEmbedding: + """Caller-owned buffers of one (rows, id dtype) decode embedding: ``raw`` (the gathered rows, the first residual) + and ``output`` (their RMSNorm), over ``table`` and the first layer's plain (non-Gemma) input norm ``weight`` / + ``eps``. Built (and compiled) before CUDA-graph capture.""" + + def __init__(self, table, weight, eps, rows, id_dtype, use_pdl=True): + assert not torch.cuda.is_current_stream_capturing() + assert table.dtype == torch.bfloat16 and table.is_contiguous() and table.shape[1] == HIDDEN + assert weight.dtype == torch.bfloat16 and weight.shape == (HIDDEN,) + assert 1 <= rows <= MAX_ROWS and id_dtype in (torch.int32, torch.int64) + self.table, self.weight, self.rows, self.id_dtype, self.use_pdl = ( + table, + weight, + rows, + id_dtype, + use_pdl, + ) + self.eps = float(eps) + self.raw = torch.empty(rows, HIDDEN, device=table.device, dtype=torch.bfloat16) + self.output = torch.empty_like(self.raw) + ids = torch.empty(rows, device=table.device, dtype=id_dtype) + _compile(ids, table, weight, self.eps, self.raw, self.output, use_pdl) + + def __call__(self, ids): + assert ids.shape == (self.rows,) and ids.dtype == self.id_dtype and ids.is_contiguous() + return m3_embed_norm( + ids, self.table, self.weight, self.eps, self.raw, self.output, self.use_pdl + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/long_select.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/long_select.py new file mode 100644 index 000000000000..14e24fbdbcac --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/long_select.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 block select for MiniMax-M3 decode rows of 129 to 9216 blocks. + +Same contract and bit-identical output as ``torch.ops.trtllm.minimax_m3_select_blocks`` on per-KV-head scores +[num_kv_heads, n_blocks, total_q]: init / local forcing, per-query valid-block count, ties to the smaller block index, +forced -inf picks emitted as -1, ascending block ids with -1 tail padding. + +One 1024-thread CTA per (query, KV head) row holds the row's ordered score keys in registers (2, 4 or 9 per +thread, by the row's valid count). The weakest of the 16 maxima of the interleaved groups ``block % 16`` is a lower +bound on the 16th-largest key, so the top 16 is among the keys at or above it; those are compacted to shared memory +and ranked exactly by (key, block). More than 1024 keys tied at the bound (degenerate rows) rank every candidate +against the whole row (re-read). + +The kernel is ``torch.ops.trtllm.minimax_m3_sm107_tp2_select_long`` (this target's copy of ``minimax_m3_select_long`` +with a choice of where it releases the sparse decode), called through this target's ``m3_select_long`` entry, or with +``dsl`` the same selection from one 384-thread CTA per row (``m3_select_long_dsl``, kernels/m3_select_long_dsl). +""" + +from typing import Optional + +import torch + +from ..ops.attention.m3_select_long import MAX_BLOCKS, m3_select_long +from ..ops.attention.m3_select_long_dsl import m3_select_long_dsl + + +def long_select_blocks( + max_score_kv: torch.Tensor, + *, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + dsl: bool = False, +) -> Optional[torch.Tensor]: + """Top-16 select over per-KV-head scores [num_kv_heads, n_blocks, total_q] with 129..9216 blocks. + + Returns the head-major backed [total_q, num_kv_heads, 16] int32 table, or None outside this kernel's shapes + (callers keep the reference op). ``early_trigger`` releases the dependent sparse decode right after the select's + grid wait (modeling's ``SELECT_EARLY_MAX_ROWS``); otherwise ``trigger_phase`` 1 releases it once every score of the + row is loaded and 0 at the select's exit (modeling's ``SELECT_LONG_TRIGGER_PHASE``). ``dsl`` runs the 384-thread + kernel (modeling's ``SELECT_LONG_DSL``; the same table). + """ + if ( + max_score_kv.dtype != torch.float32 + or not max_score_kv.is_cuda + or max_score_kv.dim() != 3 + or not 128 < max_score_kv.shape[1] <= MAX_BLOCKS + or n_valid_blocks.dtype != torch.int32 + or not n_valid_blocks.is_cuda + or n_valid_blocks.shape[0] < max_score_kv.shape[2] + ): + return None + num_kv_heads, _, total_q = max_score_kv.shape + out = torch.empty( + (num_kv_heads, total_q, 16), dtype=torch.int32, device=max_score_kv.device + ).permute(1, 0, 2) + return (m3_select_long_dsl if dsl else m3_select_long)( + max_score_kv, + n_valid_blocks.contiguous(), + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_attn_lc8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_attn_lc8.py new file mode 100644 index 000000000000..8cf83cdf7d75 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_attn_lc8.py @@ -0,0 +1,164 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 long-context sparse decode attention at 8 query heads in one launch. + +Replaces the generic chain's split-K producer (``_gqa_sparse_decode_kernel``, one block per chunk) and the LSE +merge (``_merge_topk_attn_out_kernel``, with its MXFP8 copy for the o_proj) once the top-k block table exists. + +Grid: one cluster of 16 CTAs per query token; CTA ``r`` attends the token's ``r``-th selected block for all 8 heads, +keeps the partial in shared memory, and the cluster merges the 16 partials over DSMEM into the BF16 row and, +optionally, its MXFP8 copy and swizzled UE8M0 scales exactly as ``_store_mxfp8_head`` lays them out. + +Per-block math follows the Triton producer: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16, QK and PV +on FP16 tensor cores with FP32 accumulation, log2-domain softmax with either the linear denominator or the +log-domain recurrence. Accumulation order inside the MMAs and the reductions differs, so results match to rounding, +not bit for bit. + +Schedule: the kernel releases its PDL dependents right after its grid wait, so the o_proj launches and streams its +weights while attention runs. Up to ``PREWAIT_MAX_ROWS`` query rows it also stages the request length, +the query rows and the request's block-table row before that wait, overlapping them with the select ahead when it +releases this grid early (modeling's ``SELECT_EARLY_MAX_ROWS``). Neither changes a value. + +Geometry: 8 query heads per KV head (TP8), one KV head, head_dim 128, page 128, top-k 16. The kernel is +``torch.ops.trtllm.minimax_m3_tp8_sparse_decode_lc8``, called through this target's ``m3_sparse_decode_lc8`` entry. +""" + +import torch + +from ..m31_switches import m31_disabled, m31_enabled +from ..ops.attention.m3_sparse_decode_lc8 import NUM_HEADS, WIDTH, m3_sparse_decode_lc8 + +# Up to this many query rows the kernel stages the request length, the query rows and the block-table row before its +# grid wait. +PREWAIT_MAX_ROWS = 8 # capacity: fixed: measured occupancy crossover +# The split-K merge forms its weights once per CTA (MERGE_ONCE: also enables the identity route and the L2 prefetches) +# and divides once per output (ONE_DIV_MERGE: sum_c w_c acc_c / sum_c w_c l_c in chunk order; deterministic, in the +# float64 reference's band). ONE_DIV_MERGE is default OFF (its VR TP8 natural-AL gate failed the frozen rule): +# TRTLLM_M31_ENABLE_LC8_ONEDIV=1 opts in; TRTLLM_M31_DISABLE_LC8_ONEDIV=1 still wins. +MERGE_ONCE = True +ONE_DIV_MERGE = m31_enabled("LC8_ONEDIV") and not m31_disabled("LC8_ONEDIV") +# Up to this many query rows the merge divides once (VR TP8: ONE_DIV wins at 4 / 8 rows and turns +# to a loss from 16 rows at 128K). TRTLLM_M31_DISABLE_ONEDIV_CAP=1 keeps ONE_DIV at every row count. +ONE_DIV_MAX_ROWS = ( + 1 << 30 if m31_disabled("ONEDIV_CAP") else 8 +) # capacity: fixed: measured crossover + + +def one_div_applies(total_q: int) -> bool: + """Whether the merge divides once at ``total_q`` query rows.""" + return ONE_DIV_MERGE and MERGE_ONCE and total_q <= ONE_DIV_MAX_ROWS + + +class PrefetchSlot: + """The read-only tensors the next long-context sparse decode prefetches into L2 (the attention layer's o_proj weight + and scales), set by the layer around its decode call and taken by the decode; empty means no prefetch.""" + + def __init__(self) -> None: + self.tensors = None + + def take(self): + tensors, self.tensors = self.tensors, None + return tensors + + +PREFETCH = PrefetchSlot() +# (this layer's id buffer, the next sparse layer's id buffer, its K / V pools, its K / V block scales) +NEXT_KV = PrefetchSlot() + + +def lc8_applies(q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output) -> bool: + """Whether the step's operands fit the kernel: TP8 geometry, packed NVFP4 pages, FP8 q, BF16 [T, 8, 128] out.""" + if q.dim() != 3 or tuple(q.shape[1:]) != (NUM_HEADS, 128) or q.dtype != torch.float8_e4m3fn: + return False + if q.stride(2) != 1 or q.stride(1) % 16 or q.stride(0) % 16 or q.data_ptr() % 16: + return False + if int(k_paged.shape[1]) != 1 or int(topk_idx.shape[0]) != 1 or int(topk_idx.shape[-1]) != 16: + return False + for t in (k_paged, v_paged): + if ( + t.stride(3) != 1 + or t.stride(2) != 64 + or int(t.shape[3]) != 64 + or t.data_ptr() % 16 + or t.stride(0) % 16 + ): + return False + for t in (k_block_scale, v_block_scale): + if t is None or t.stride(3) != 1 or t.stride(2) != 8 or t.data_ptr() % 8 or t.stride(0) % 8: + return False + return ( + output.dtype == torch.bfloat16 + and output.dim() == 3 + and output.is_contiguous() + and tuple(output.shape) == tuple(q.shape) + ) + + +def m3_lc8_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + linear_softmax=True, + mxfp8=None, + use_pdl=True, + identity=False, + identity_step=False, + prefetch=None, + next_kv=None, +): + """Drop-in for ``minimax_m3_sparse_attn_decode`` on the TP8 NVFP4 shape; callers check ``lc8_applies`` first. + + ``topk_idx`` is the head-major [1, total_q, 16] view the Triton producer takes. ``mxfp8`` is ``(q8, sf)`` from + ``MERGE_MXFP8.buffers(1024, dev)`` to also emit the o_proj hand-off. With ``MERGE_ONCE``, + ``prefetch`` (up to two contiguous tensors) is pulled into L2 by the op's CTAs before their grid dependency, and + ``next_kv`` drives the next-layer KV prefetch. + """ + merge_once = MERGE_ONCE + total_q = int(q.shape[0]) + q8, sf = ( + (mxfp8[0].view(torch.uint8), mxfp8[1].view(torch.uint8)) + if mxfp8 is not None + else (None, None) + ) + m3_sparse_decode_lc8( + q.view(torch.uint8), + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx[0], + block_table, + seq_lens, + output.view(total_q, WIDTH), + q8, + sf, + WIDTH // 128, + int(decode_query_len), + float(sm_scale), + bool(linear_softmax), + bool(use_pdl), + total_q <= PREWAIT_MAX_ROWS, + merge_once=merge_once, + identity=bool(identity) and merge_once, + identity_step=bool(identity_step) and merge_once, + one_div=merge_once and one_div_applies(total_q), + prefetch=prefetch if merge_once else None, + next_kv=( + tuple(next_kv[:2]) + tuple(t.view(torch.uint8) for t in next_kv[2:]) + if merge_once and next_kv is not None + else None + ), + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_draft_step.py new file mode 100644 index 000000000000..c977362e92b8 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_draft_step.py @@ -0,0 +1,355 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 draft pass for the MiniMax-M3 drafter's vocab-sharded BF16 head (TP decode, <= 8 requests). + +One kernel replaces a draft pass's head and the draft loop's per-pass bookkeeping, at the in-tree precision: + +* The drafter's BF16 lm_head shard is kept a second time, built by ``prepare_head`` before the executor's warmup: in + the lossless 12-bit coding of the target head (``ops.gemm.m3_target_head``; 75% of the bytes, kept only when it + decodes to the BF16 shard bit for bit), else in ``mma.m16n8k16`` A-fragment order (``bf16_fragments``, the same + values). The kernel streams it with BF16 MMAs and FP32 accumulation over the BF16 head input rows; a coded + exception decodes to +0 and its exact product x * w is added at its tile's flush. +* Each logit is rounded to BF16 (the in-tree logits are the BF16 head GEMM's output) and keyed + ``ordered(value) << 32 | ~global_index`` with -0 as +0, so the larger value wins and ties take the lowest index, as + the in-tree ``torch.max`` / ``torch.argmax`` over the gathered logits (``greedy_sample_draft_with_tp_gather``). +* The last CTA exchanges each row's key across the TP ranks through a multicast mapping (Lamport slots, see + ``m3_shard_accept``), then stores the draft token, the next pass's input id and position, and advances the draft KV + lengths as the in-tree draft loop does between passes. + +The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/draftStep.cu) and called through the +catalog entry ``ops.spec.m3_draft_step``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +import torch + +from tensorrt_llm.logger import logger + +from ..ops.gemm.m3_target_head import encode_chunked, round_trip_ok +from ..ops.spec.m3_draft_step import MAX_TILES_PER_CTA, MAX_TOKENS, bf16_fragments, max_tile_pieces +from ..ops.spec.m3_draft_step import m3_draft_step as _draft_step_op +from . import m3_report +from .m3_shard_accept import AcceptWorkspace, create_workspace + +_ENGAGED: set = set() + + +# The fused draft loop streams the 12-bit coded copy of the draft head when it decodes bit for bit; off (the coded +# head is slower on sm_107: the BF16 fragments stream). +HEAD_CODED = False +# The head copy placed per locality domain, each pass as two launches: off at TP8 (one launch over the whole device). +UGPU = False +# The draft step's CTA count (at most one per SM; 0 = one per SM). +GRID = 208 # capacity: fixed: a launch-grid choice, not a size +# The draft head sums a tile shared by more than 2 warp pieces in warp order instead of with atomics (the same sum +# every run; the atomics are deterministic only where a tile takes at most 2 pieces). +FIXED_ORDER = True +FIXED_ORDER_PATH = "m3_draft_step_fixed_order" + + +@dataclass(eq=False) +class DraftHead: + """The streamed copy of one rank's draft lm_head shard and its first global id: ``wf`` is the 12-bit coded stream + with its exception lists (``coded``), or the fragment-ordered BF16 copy (``bf16_fragments``).""" + + weight: torch.Tensor + wf: torch.Tensor + vocab_offset: int + coded: bool = False + exc_off: Optional[torch.Tensor] = None + exc_pos: Optional[torch.Tensor] = None + exc_w: Optional[torch.Tensor] = None + exp_base: int = 0 + # BF16 fragments placed per locality domain (UGPU below): tiles [0, split_tile) live in domain 0 and + # the rest in domain 1; 0 = not placed (one launch over all SMs). + split_tile: int = 0 + + +def head_supported(lm_head) -> bool: + """A bias-free BF16 head whose local shard the kernel can own: whole 16-row tiles, K a multiple of 16, at most + MAX_TILES_PER_CTA tiles per CTA of a one-CTA-per-SM grid.""" + weight = getattr(lm_head, "weight", None) + if weight is None or weight.dim() != 2 or weight.dtype != torch.bfloat16: + return False + if getattr(lm_head, "bias", None) is not None: + return False + n, k = weight.shape + if n % 16 or k % 16: + return False + return (n // 16 + _grid() - 1) // _grid() <= MAX_TILES_PER_CTA + + +def prepare_head(lm_head, tp_rank: int) -> DraftHead: + """The streamed copy of ``lm_head``'s local shard (global ids from tp_rank * rows): its 12-bit coding when + ``HEAD_CODED``, K is a multiple of 64 and the coding decodes to the shard bit for bit, else (fail + closed) the fragment-ordered BF16 copy. + Built once, eagerly, before the executor's warmup (``M3Eagle3Worker.prebuild_draft_head``).""" + weight = lm_head.weight + offset = tp_rank * weight.shape[0] + with torch.no_grad(): + w = weight.detach() + if HEAD_CODED and w.shape[1] % 64 == 0: + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + if round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w): + return DraftHead( + weight=weight, + wf=packed, + vocab_offset=offset, + coded=True, + exc_off=exc_off, + exc_pos=exc_pos, + exc_w=exc_w.view(-1), + exp_base=base, + ) + del packed, exc_off, exc_pos, exc_w + wf = bf16_fragments(w) + wf, split_tile = _place_per_domain(wf) + return DraftHead(weight=weight, wf=wf, vocab_offset=offset, split_tile=split_tile) + + +def _place_per_domain(wf: torch.Tensor): + """(wf, split tile): the BF16 fragments split-mapped over the two locality domains at the granule-aligned tile + nearest half (locality.LocalityMap.place_split; its domain streams built here, before capture), or (wf, 0) when + UGPU is off. Placement only: the streamed values are the same.""" + if not UGPU: + return wf, 0 + from .m3_locality import LocalityMap + + lm = LocalityMap.get(wf.device) + tile_bytes = wf[0].numel() * wf.element_size() + step = 1 + while (step * tile_bytes) % lm.granularity: + step += 1 + n_tiles = wf.shape[0] + split_tile = max(step, round(n_tiles / 2 / step) * step) + per = lm.sms_per_domain + if split_tile >= n_tiles or max(split_tile, n_tiles - split_tile) > per * MAX_TILES_PER_CTA: + return wf, 0 + lm.domain_streams() + return lm.place_split(wf, split_tile), split_tile + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _grid() -> int: + """The draft step's CTA count: GRID when set (at most one per SM), else one per SM.""" + return min(GRID, _num_sms()) if GRID > 0 else _num_sms() + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace ([2][MAX_TOKENS][world] slots), created collectively on the first eager call; a + CUDA-graph capture that finds it missing is a miss.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + m3_report.capture_missed("m3_draft_step", None, "the draft-step exchange workspace") + return None + _WS = create_workspace(mapping, device, MAX_TOKENS) + return _WS + + +def _fixed_order( + rows: int, head: DraftHead, x: torch.Tensor, num_ctas: int, split: bool = False +) -> bool: + """FIXED_ORDER where the launch lets a tile take more than 2 warp pieces (max_tile_pieces; a + split or equal-chunk launch always), with its engagement line (D-FIX); False keeps the atomics, deterministic on + this shape.""" + if not FIXED_ORDER: + return False + eq = getattr(head, "bnd", None) is not None + if not (split or eq): + chunks = x.shape[1] // (16 if head.exc_off is None else 64) + if ( + max_tile_pieces( + head.wf.shape[0] if head.exc_off is None else head.exc_off.numel() - 1, + num_ctas, + chunks, + ) + <= 2 + ): + return False + m3_report.taken(FIXED_ORDER_PATH, rows, "the draft head sums shared tiles in warp order") + return True + + +def draft_step( + head: DraftHead, + x: torch.Tensor, + rows: int, + x_rows: Optional[torch.Tensor], + mapping, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + fold: Optional[tuple] = None, +) -> bool: + """One fused greedy draft pass over ``rows`` requests (see the module docstring); False when the exchange + workspace is missing during a capture. Collective over the TP group. + + Args: + head: the prepared head of this rank (``prepare_head``), coded or BF16 fragments. + x: [*, K] BF16 head input rows (the drafter's final-norm output); row ``x_rows[b]`` (int64) or ``b`` is + request b's. + draft_out: int32 [rows, >= step + 1], unit column stride: column ``step`` receives the tokens. + ids_next / pos_next: int32 [rows]: the tokens, and ``pos_src[x_rows[b] or b] + 1`` (pos_src int32). + kv_lens: int32 draft KV lengths: ``-= draft_len - num_acc`` with ``seq_lens = 1`` on pass 0 when ``num_acc`` + is given, ``+= 1`` on later passes. + fold: on the last pass (modeling's DRAFT_STEP_FOLD_TAIL), (the 14 fold-tail tensors, rows, tokens per step) from + M3SpecStepFold.fused_tail_operands: the step's forward-tail store and next-step staging run by this pass's + last CTA after its tokens, instead of the standalone fold tail. + """ + ws = workspace(mapping, x.device) + if ws is None: + return False + if head.split_tile: + _draft_step_split( + head, + x, + rows, + x_rows, + ws, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + fold, + ) + return True + _draft_step_op( + x, + x_rows, + rows, + head.wf, + head.vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + _grid(), + True, + None, + head.exc_off, + head.exc_pos, + head.exc_w, + head.exp_base, + fold=fold, + fixed_order=_fixed_order(rows, head, x, _grid()), + ) + return True + + +def _draft_step_split( + head, + x, + rows, + x_rows, + ws, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + fold=None, +) -> None: + """One pass as two launches, one per locality domain on that domain's green-context stream, each streaming its own + domain's tiles of the split-mapped fragments; the keys' max and the last-CTA exchange and bookkeeping run once over + both (the last of the 2 x SMs-per-domain CTAs). Forked from and joined back to the current stream; no PDL on the + forks.""" + from .m3_locality import LocalityMap + + lm = LocalityMap.get(x.device) + per = lm.sms_per_domain + main = torch.cuda.current_stream() + fork = torch.cuda.Event() + fork.record(main) + streams = lm.domain_streams() + n_tiles = head.wf.shape[0] + if x.device not in _ENGAGED: + _ENGAGED.add(x.device) + logger.info( + f"[M3 sm_107 tp2] fast path engaged: m3_draft_step_ugpu (split tile {head.split_tile} of {n_tiles}, " + f"{per} CTAs per domain, on {x.device})" + ) + for st, lo, hi in ((streams[0], 0, head.split_tile), (streams[1], head.split_tile, n_tiles)): + st.wait_event(fork) + with torch.cuda.stream(st): + _draft_step_op( + x, + x_rows, + rows, + head.wf, + head.vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + per, + False, + None, + None, + None, + None, + 0, + lo, + hi, + 2 * per, + fold=fold, # run by whichever launch holds the pass's last CTA + fixed_order=_fixed_order(rows, head, x, 2 * per, split=True), + ) + for st in streams: + done = torch.cuda.Event() + done.record(st) + main.wait_event(done) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_drafter_attn.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_drafter_attn.py new file mode 100644 index 000000000000..cae01087ce72 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_drafter_attn.py @@ -0,0 +1,442 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter layer's decode attention on Rubin: ``M3DrafterFullCtxFmha``. + +The drafter layer is a plain ``TrtllmAttention`` over its own FP8 paged cache (the full context, every draft pass). +On SM 10.7 its generation phase runs ``thop.attention``'s in-tree trtllm-gen runner, whose split is capped at 8 CTAs +per (request, KV head): 45-46 us per pass at 165k tokens on 212 SMs, for 85 MB. + +This library keeps the in-tree preprocessing -- ``thop.trtllm_gen_generation_preprocess``: RoPE, the FP8 K/V append +into the paged cache and the FP8 query, as ``FlashInferTrtllmGenFmha`` runs it -- and attends with the full-context +PrimTS split-KV decode of ``kernels/m3_fullctx_decode`` at the same pinned recipes as the dense layers. + +With ``fused_rope_mode`` set, the preprocess and the query cast run as one launch instead +(``minimax_m3_sm107tp8_drafter_rope_fp8_kv``, the same bytes into the query and the cache), the layer's sub-page +pool and block table taken from the KV cache pool pointers and mapping the preprocess reads. + +It serves generation-only steps of the covered (query rows per request, requests) shapes; the target installs it +first in the drafter layer's FMHA list (``install``), so every other step falls through to the stock libraries. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Mapping, Optional, Tuple + +import torch + +from tensorrt_llm._torch.attention.backends.fmha.fallback import FallbackFmha +from tensorrt_llm._torch.attention.backends.fmha.flashinfer_trtllm_gen import ( + FlashInferTrtllmGenFmha, +) +from tensorrt_llm._torch.attention.backends.fmha.interface import FmhaPhase +from tensorrt_llm._torch.attention.backends.fmha.phased import FmhaParams +from tensorrt_llm._torch.attention.backends.fmha.utils import ( + get_attention_chunk_size, + get_bmm1_scale, +) +from tensorrt_llm._torch.attention.backends.interface import AttentionInputType +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) +from tensorrt_llm._torch.flashinfer_utils import IS_FLASHINFER_AVAILABLE +from tensorrt_llm._utils import TensorWrapper, convert_to_torch_tensor, get_sm_version +from tensorrt_llm.bindings.internal import thop +from tensorrt_llm.functional import PositionEmbeddingType +from tensorrt_llm.quantization.mode import QuantMode + +from . import m3_fullctx_decode as fullctx + +# With PDL, the fused RoPE + FP8 K/V launch takes the maximum shared-memory carveout so the drafter decode's large-smem +# CTAs co-reside with it (timing only). +ROPE_CARVEOUT = True + + +def fwd_has_no_mrope(fwd) -> bool: + return getattr(fwd, "mrope_position_deltas", None) is None + + +if TYPE_CHECKING: + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + +class M3DrafterFullCtxFmha(FlashInferTrtllmGenFmha): + """Generation phase of the drafter layer: in-tree preprocess, PrimTS split-KV decode.""" + + def __init__( + self, + attn: "TrtllmAttention", + recipes: Mapping[Tuple[int, int], Mapping], + max_kv_len: int, + own_reducer: bool, + on_taken, + on_uncovered, + on_missed, + fused_rope_mode: Optional[int] = None, + pdl: bool = False, + on_rope_taken=None, + ) -> None: + super().__init__(attn) + self._recipes = recipes + self._max_kv_len = max_kv_len + self._own_reducer = own_reducer + self._fused_rope_mode = fused_rope_mode + self._pdl = pdl + self._on_rope_taken = on_rope_taken + self._on_taken = on_taken + self._on_uncovered = on_uncovered + self._on_missed = on_missed + + def _shape(self, q: torch.Tensor, metadata) -> Optional[Tuple[int, int]]: + nreq = int(metadata.num_generations) + if nreq <= 0 or int(q.shape[0]) % nreq: + return None + return int(q.shape[0]) // nreq, nreq + + def _is_supported(self, q, k, v, metadata, forward_args, *, phase=None) -> bool: + if phase == FmhaPhase.CONTEXT or metadata.num_contexts > 0: + return False + if phase is None and forward_args.attention_input_type == AttentionInputType.context_only: + return False + attn = self.attn + if ( + attn.num_heads != fullctx.HEADS + or attn.num_kv_heads != fullctx.KV_HEADS + or attn.head_dim != fullctx.HEAD_DIM + or not QuantMode(attn.quant_mode).has_fp8_kv_cache() + ): + self._on_uncovered("not the TP8 FP8 drafter geometry") + return False + # The arithmetic matches the stock decode only at unit KV scales (bmm1 and bmm2 carry none here); the layer's + # own scales are checked to be 1.0 at install, so any other per-step scale falls through to the stock path. + if ( + forward_args.kv_scale_orig_quant is not attn.kv_scale_orig_quant + or forward_args.kv_scale_quant_orig is not attn.kv_scale_quant_orig + ): + self._on_uncovered("per-step KV cache scales") + return False + if int(metadata.max_seq_len) > self._max_kv_len: + self._on_uncovered("the cache length bound is beyond the plans' K/V capacity") + return False + shape = self._shape(q, metadata) + if shape is None or shape not in self._recipes: + self._on_uncovered(f"{shape} is not a covered (query rows, requests) shape") + return False + # Compiled before warmup (``install``); the forward never compiles one. + if fullctx.plan_for(q.device, *shape, self._own_reducer) is None: + self._on_missed(shape[0] * shape[1], "the PrimTS decode plan is not prebuilt") + return False + return super()._is_supported(q, k, v, metadata, forward_args, phase=FmhaPhase.GENERATION) + + def _fused_rope_applies(self, params: FmhaParams, attention_input: torch.Tensor) -> bool: + attn = params.attn + width = (attn.num_heads + 2 * attn.num_kv_heads) * attn.head_dim + rotary_dim = int(attn.rope_params.dim) + return ( + self._fused_rope_mode is not None + and params.qkv_input is not None + and attention_input.dim() == 2 + and attention_input.dtype == torch.bfloat16 + and attention_input.is_contiguous() + and int(attention_input.shape[1]) == width + and not params.is_cross + and int(attn.position_embedding_type) == int(PositionEmbeddingType.rope_gpt_neox) + and attn.rotary_cos_sin is not None + and attn.rotary_cos_sin.dtype == torch.float32 + and fwd_has_no_mrope(params.fwd) + and 0 < rotary_dim <= attn.head_dim + and rotary_dim % 16 == 0 + and int(params.num_tokens) == int(params.batch_size) * int(params.input_seq_length) + # A one-request multi-token pass: the in-tree preprocess does not build its token map (buildDecoderInfo + # runs for several requests only) and reads the workspace's, so it keeps that path. + and not (self._multi_token(params) and int(params.batch_size) == 1) + ) + + @staticmethod + def _multi_token(params: FmhaParams) -> bool: + return ( + params.spec_decoding_generation_lengths is not None + and int(params.attn.predicted_tokens_per_seq) > 1 + ) + + def _run_fused_rope(self, params: FmhaParams, attention_input: torch.Tensor) -> bool: + """The pass with the one-launch preprocess; False (nothing run) where it does not apply.""" + if not self._fused_rope_applies(params, attention_input): + return False + attn = params.attn + meta = params.meta + qlen, nreq = int(params.input_seq_length), int(params.batch_size) + plan = fullctx.plan_for(attention_input.device, qlen, nreq, self._own_reducer) + if plan is None: + return False + # The preprocess's view of the layer's cache (buildFlashinferTrtllmGenPagedKvCacheBuffers): the pool's primary + # pointer plus the layer's offset inside its pool, sub-pages of Hkv x tokens_per_block x 128 E4M3 bytes; the + # block table is the pool's [requests, 2, blocks] K / V sub-page indices. + pool_index, layer_in_pool = ( + int(x) for x in meta.host_kv_cache_pool_mapping[attn.local_layer_idx, :2] + ) + tpb = int(params.tokens_per_block) + sub_page_bytes = attn.num_kv_heads * tpb * attn.head_dim + base = ( + int(meta.host_kv_cache_pool_pointers[pool_index, 0]) + + layer_in_pool * int(params.kv_factor) * sub_page_bytes + ) + pages = (int(params.total_num_blocks) - 1) * int(params.kv_factor) + 2 + kv_pages = convert_to_torch_tensor( + TensorWrapper(base, torch.float8_e4m3fn, [pages, attn.num_kv_heads, tpb, attn.head_dim]) + ) + tables = meta.kv_cache_block_offsets[ + pool_index, params.seq_offset : params.seq_offset + nreq + ] + multi = self._multi_token(params) + if not torch.cuda.is_current_stream_capturing(): + # The decode reads V page p as sub-page p + 1 of K's (kv_pages[1:] with the K table), as the preprocess + # path checks; the op itself writes V at the table's V sub-page. + tpb_held = (params.sequence_lengths[:nreq].to(torch.long) + tpb - 1) // tpb + cols = torch.arange(tables.shape[-1], device=tables.device) + in_use = cols[None, :] < tpb_held[:, None] + offsets = (tables[:, 1] - tables[:, 0])[in_use].unique() + if offsets.numel() != 1 or int(offsets) != 1: + raise RuntimeError( + f"the drafter cache does not keep V one sub-page after K: {offsets.tolist()}" + ) + q = torch.empty( + int(params.num_tokens), + attn.num_heads, + attn.head_dim, + dtype=torch.float8_e4m3fn, + device=attention_input.device, + ) + torch.ops.trtllm.minimax_m3_sm107tp8_drafter_rope_fp8_kv( + attention_input, + attn.rotary_cos_sin, + params.sequence_lengths, + params.spec_decoding_position_offsets if multi else None, + kv_pages, + tables, + q, + qlen, + attn.num_heads, + attn.num_kv_heads, + int(attn.rope_params.dim), + tpb, + self._fused_rope_mode, + self._pdl, + ROPE_CARVEOUT, + ) + plan.run( + q, + kv_pages[:-1], + kv_pages[1:], + tables[:, 0], + params.sequence_lengths[:nreq], + get_bmm1_scale(attn), + params.output.view(params.num_tokens, attn.num_heads, attn.head_dim), + ) + if self._on_rope_taken is not None: + self._on_rope_taken(params.num_tokens, "spec" if multi else "plain") + self._on_taken(params.num_tokens) + return True + + def run_generation(self, params: FmhaParams) -> None: + attention_input = params.qkv_input if params.qkv_input is not None else params.query_input + if attention_input is None: + raise RuntimeError("the drafter decode requires QKV or query input.") + attn = params.attn + meta = params.meta + fwd = params.fwd + rope_params = attn.rope_params + if self._run_fused_rope(params, attention_input): + return + # The BF16 post-RoPE query, rounded to E4M3 below. At the unit KV scales this layer runs with, that is the FP8 + # query the stock decode gets (the preprocess's FP8 variant is e4m3(q * kv_scale_orig_quant)). + fp8_context_fmha = False + ( + q_processed, + kv_pool, + block_tables, + _kv_scale_pool, + _bmm1_scale, + _bmm2_scale, + _fmha_workspace, + _cu_seqlens, + _max_q_len, + _max_kv_len, + _window_left, + is_multi_token_gen, + ) = thop.trtllm_gen_generation_preprocess( + attention_input, + params.workspace, + params.sequence_lengths, + params.spec_decoding_generation_lengths, + params.spec_decoding_position_offsets, + meta.kv_cache_block_offsets, + meta.host_kv_cache_pool_pointers, + meta.host_kv_cache_pool_mapping, + fwd.kv_scale_orig_quant, + fwd.kv_scale_quant_orig, + fwd.out_scale, + attn.rotary_inv_freq, + attn.rotary_cos_sin, + fwd.mrope_position_deltas, + attn.local_layer_idx, + params.seq_offset, + attn.num_heads, + attn.num_kv_heads, + attn.head_dim, + params.tokens_per_block, + attn.quant_mode, + params.max_attention_window_size, + params.cyclic_attention_window_size, + params.num_tokens, + params.batch_size, + params.input_seq_length, + params.max_past_kv_length, + rope_params.dim, + rope_params.theta, + int(rope_params.scale_type), + rope_params.scale, + rope_params.max_positions, + attn.position_embedding_type, + get_bmm1_scale(attn), + 1.0, + fp8_context_fmha, + attn.predicted_tokens_per_seq, + get_attention_chunk_size(attn), + self._multi_processor_count, + params.total_num_blocks, + params.kv_factor, + True, + params.is_cross, + ) + qlen, nreq = params.input_seq_length, params.batch_size + plan = fullctx.plan_for(attention_input.device, qlen, nreq, self._own_reducer) + # A first draft pass arrives as qlen rows per request either as a plain multi-row generation step (the + # drafter's own, input_seq_length = qlen) or with spec-decoding lengths (multi-token generation); a 1-row + # pass is never multi-token. + if plan is None or (is_multi_token_gen and qlen == 1): + raise RuntimeError( + f"the drafter decode was selected for {qlen} query rows x {nreq} requests without a uniform plan" + ) + rows = params.num_tokens * attn.num_heads * attn.head_dim + q = fp8_e4m3_saturate(q_processed.flatten()[:rows]) + q = q.view(params.num_tokens, attn.num_heads, attn.head_dim) + # K pages in row 0 of the table, V in row 1; this cache keeps V one sub-page after K, so V page p is page p + 1 + # of the pool (the two views have the same page count, as the kernel requires). + if not torch.cuda.is_current_stream_capturing(): + # The plan runs every request at qlen rows; a multi-token pass must give each request exactly that many. + lengths = params.spec_decoding_generation_lengths + if is_multi_token_gen and not bool((lengths[:nreq] == qlen).all()): + raise RuntimeError( + f"the drafter pass rows per request are not uniform: {lengths[:nreq].tolist()}" + ) + # Only the blocks a request holds are laid out; the table's tail entries are unset. + tpb = int(params.tokens_per_block) + held = (params.sequence_lengths[:nreq].to(torch.long) + tpb - 1) // tpb + cols = torch.arange(block_tables.shape[-1], device=block_tables.device) + in_use = cols[None, :] < held[:, None] + offsets = (block_tables[:nreq, 1] - block_tables[:nreq, 0])[in_use].unique() + if offsets.numel() != 1 or int(offsets) != 1: + raise RuntimeError( + f"the drafter cache does not keep V one sub-page after K: {offsets.tolist()}" + ) + # The preprocess sizes kv_pool by blocks; its table indexes the K and V sub-pages of every block. + # The view stops two sub-pages past the last block's K, as get_kv_subpage_pool's does. + pages = (int(params.total_num_blocks) - 1) * int(params.kv_factor) + 2 + kv_pages = convert_to_torch_tensor( + TensorWrapper(kv_pool.data_ptr(), kv_pool.dtype, [pages, *kv_pool.shape[1:]]) + ) + plan.run( + q, + kv_pages[:-1], + kv_pages[1:], + block_tables[:nreq, 0], + params.sequence_lengths[:nreq], + get_bmm1_scale(attn), + params.output.view(params.num_tokens, attn.num_heads, attn.head_dim), + ) + self._on_taken(params.num_tokens) + + +def install( + draft_model: torch.nn.Module, + recipes, + max_kv_len: int, + own_reducer: bool, + pdl: bool, + on_taken, + on_uncovered, + on_missed, + fused_rope_mode: Optional[int] = None, + on_rope_taken=None, +) -> int: + """Prebuild the plans of ``recipes`` and put ``M3DrafterFullCtxFmha`` first in the FMHA list of the TrtllmAttention + backend of every attention module of ``draft_model``. Call before warmup. Returns the number of attention layers it + was installed on.""" + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + count = 0 + device = next(draft_model.parameters()).device + # The backend is not a module: each attention module of the drafter holds it as ``attn``. + for module in draft_model.modules(): + backend = getattr(module, "attn", None) + if not isinstance(backend, TrtllmAttention) or not hasattr(backend, "_fmha_manager"): + continue + if not bool((backend.kv_cache_scaling_factor == 1.0).all()): + on_uncovered("a scaled FP8 KV cache") + continue + fullctx.prebuild(device, recipes, max_kv_len, own_reducer, pdl) + libs = backend._fmha_manager.fmha_libs + if not any(isinstance(lib, M3DrafterFullCtxFmha) for lib in libs): + libs.insert( + 0, + M3DrafterFullCtxFmha( + backend, + recipes, + max_kv_len, + own_reducer, + on_taken, + on_uncovered, + on_missed, + fused_rope_mode=fused_rope_mode, + pdl=pdl, + on_rope_taken=on_rope_taken, + ), + ) + count += 1 + return count + + +class M3DrafterSlidingFiFmha(FlashInferTrtllmGenFmha): + """FlashInfer's trtllm-gen FMHA for the standalone DSpark drafter's attention on SM 10.7. + + The stock library declines SM 10.7, so the drafter's sliding-window decode falls through to ``thop.attention``'s + in-tree trtllm-gen runner, which picks the SM107a Q64 KeepsAb kernel (9.7 us per call at 7 rows, window 127, grid + 9). FlashInfer's own selector runs its SM100f kernels on SM 10.7 and picks the Q8 SwapsAb one (4.0 us per call on + the same shape). Everything else is the stock library.""" + + @classmethod + def is_environment_available(cls) -> bool: + if not IS_FLASHINFER_AVAILABLE or cls._missing_fused_nanobind_ops(): + return False + return get_sm_version() == 107 + + +def install_sliding_fi(draft_model: torch.nn.Module) -> int: + """Put ``M3DrafterSlidingFiFmha`` ahead of the fallback in the FMHA list of every attention module of + ``draft_model`` it can serve. Call before warmup. Returns the number of attention layers it was installed on.""" + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + count = 0 + for module in draft_model.modules(): + backend = getattr(module, "attn", None) + if not isinstance(backend, TrtllmAttention) or not hasattr(backend, "_fmha_manager"): + continue + libs = backend._fmha_manager.fmha_libs + if any(isinstance(lib, M3DrafterSlidingFiFmha) for lib in libs): + continue + if not M3DrafterSlidingFiFmha.is_available(backend): + continue + at = next((i for i, lib in enumerate(libs) if isinstance(lib, FallbackFmha)), len(libs)) + libs.insert(at, M3DrafterSlidingFiFmha(backend)) + count += 1 + return count diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fc_wait.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fc_wait.py new file mode 100644 index 000000000000..a545b43857ec --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fc_wait.py @@ -0,0 +1,40 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The drafter fc fork's release of the first draft input by a device counter instead of a stream join (modeling's +``DI_FC_FLAG``). + +Triton. ``fc_wait_kernel``: one thread, queued on the main stream after the acceptance. It lets the next kernel (the +first draft input) launch, waits for the acceptance before it, then spins until the fork's counter reaches the number +of forks waited for. Ordering only: no value changes. +""" + +import torch +import triton +import triton.language as tl + +# Spins of fc_wait_kernel before it gives up on the fc fork (~ms): a broken fork / wait pairing then counts an overrun +# in counts[2] instead of hanging the step (the draft input may read a stale fc output: draft tokens only, the verify +# stays exact). +MAX_SPINS = 1 << 22 # capacity: fixed: a spin bound, not a size + + +@triton.jit +def fc_wait_kernel(counts, MAX_SPINS: tl.constexpr): + # counts[0]: fc forks finished (bumped on the aux stream after each fc); counts[1]: forks waited for (here). + # Lets the next kernel (the first draft input) launch now; waits for the acceptance before it, then for the fork. + tl.extra.cuda.gdc_launch_dependents() + tl.extra.cuda.gdc_wait() + target = tl.load(counts + 1) + 1 + done = tl.load(counts, volatile=True) + spins = 0 + while (done < target) & (spins < MAX_SPINS): + done = tl.load(counts, volatile=True) + spins += 1 + tl.store(counts + 1, target) + if done < target: + tl.atomic_add(counts + 2, 1) + + +def fc_wait(counts: torch.Tensor) -> None: + """Queue the wait on the current stream for the fork ``counts`` (int64 [3]) last counted.""" + fc_wait_kernel[(1,)](counts, MAX_SPINS=MAX_SPINS, num_warps=1, launch_pdl=True) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fold_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fold_tail.py new file mode 100644 index 000000000000..292d1df264b1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fold_tail.py @@ -0,0 +1,180 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The speculative step's store and next-step staging at the tail of the forward (one Triton launch). + +For each of the first ``n`` rows (``n`` read from device memory, so a captured graph whose step was not armed writes +nothing), with ``s = slots[i]``: + +* the spec sampler's per-slot store, exactly as ``m3_spec_tail.slot_scatter`` writes it (zero-padded to each store + width): ``new[s][c]``, ``next_new[s][c]``, ``draft[s][c]`` and ``lens[s]``; +* the engine's next-step inputs, exactly as ``m3_spec_tail.step_gather`` writes them from that store for a step whose + rows are these rows in this order (no new tokens, first previous row 0): ``ids[i*T + t] = next_new[s][t]``, + ``draft_ids[i*D + d] = draft[s][d]``, ``pos_off[i*T + t] = lens[s]``, ``kv_off[i] = lens[s] - T``. + +Integer copies only: the values are the ones the sampler's and the engine's own copies write. + +With ``cpp`` the same writes run as this target's C++ op (trtllm::minimax_m3_sm107_tp8_fold_tail, +minimaxM3Sm107Tp8/foldTail.cu: one 32-lane CTA per row) instead of the Triton kernel, for widths up to 32. +""" + +import torch +import triton +import triton.language as tl + + +@triton.jit +def _fold_tail_kernel( + ctrl, + slots, + src_new, + sn_s0, + sn_s1, + SN_W: tl.constexpr, + dst_new, + dn_s0, + dn_s1, + DN_W: tl.constexpr, + src_next, + sx_s0, + sx_s1, + SX_W: tl.constexpr, + dst_next, + dx_s0, + dx_s1, + DX_W: tl.constexpr, + src_draft, + sd_s0, + sd_s1, + SD_W: tl.constexpr, + dst_draft, + dd_s0, + dd_s1, + DD_W: tl.constexpr, + src_lens, + dst_lens, + ids, + draft_ids, + pos_off, + kv_off, + T: tl.constexpr, + D: tl.constexpr, + BLOCK: tl.constexpr, +): + i = tl.program_id(0) + n = tl.load(ctrl) + if i < n: + s = tl.load(slots + i).to(tl.int64) + c = tl.arange(0, BLOCK) + new = tl.load(src_new + i * sn_s0 + c * sn_s1, mask=c < SN_W, other=0) + tl.store(dst_new + s * dn_s0 + c * dn_s1, new, mask=c < DN_W) + nxt = tl.load(src_next + i * sx_s0 + c * sx_s1, mask=c < SX_W, other=0) + tl.store(dst_next + s * dx_s0 + c * dx_s1, nxt, mask=c < DX_W) + drf = tl.load(src_draft + i * sd_s0 + c * sd_s1, mask=c < SD_W, other=0) + tl.store(dst_draft + s * dd_s0 + c * dd_s1, drf, mask=c < DD_W) + ln = tl.load(src_lens + i) + tl.store(dst_lens + s, ln) + # The next step's inputs read the store's padded rows: zero past each source width. + tl.store(ids + i * T + c, tl.where(c < DX_W, nxt, 0), mask=c < T) + tl.store(draft_ids + i * D + c, tl.where(c < DD_W, drf, 0), mask=c < D) + tl.store(pos_off + i * T + c, tl.zeros([BLOCK], dtype=tl.int32) + ln, mask=c < T) + tl.store(kv_off + i, ln - T) + + +def fold_tail( + ctrl: torch.Tensor, + slots: torch.Tensor, + rows: int, + src_new: torch.Tensor, + dst_new: torch.Tensor, + src_next: torch.Tensor, + dst_next: torch.Tensor, + src_draft: torch.Tensor, + dst_draft: torch.Tensor, + src_lens: torch.Tensor, + dst_lens: torch.Tensor, + ids: torch.Tensor, + draft_ids: torch.Tensor, + pos_off: torch.Tensor, + kv_off: torch.Tensor, + tokens_per_step: int, + cpp: bool = False, + one_warp: bool = False, +) -> bool: + """Launch over ``rows`` programs (the first ``ctrl[0]`` write); False, having launched nothing, when the inputs + are outside what the kernel covers. Store views are 2-D [slots, width] (any strides), sources [rows, width]. + ``one_warp`` (with ``cpp``): the in-kernel form (one warp looping over the rows, foldTailWarp in + minimaxM3Sm107Tp8/foldTail.h) that a caller folding the tail into its own kernel runs; the same writes.""" + two = (src_new, dst_new, src_next, dst_next, src_draft, dst_draft) + one = (src_lens, dst_lens, ids, draft_ids, pos_off, kv_off) + if any(t.dim() != 2 for t in two) or any(t.dim() != 1 or t.stride(0) != 1 for t in one): + return False + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in two + one + (ctrl,)) + or slots.dtype != torch.long + ): + return False + t_step = tokens_per_step + d_step = t_step - 1 + if ( + rows <= 0 + or slots.numel() < rows + or any(t.shape[0] < rows for t in (src_new, src_next, src_draft, src_lens)) + or ids.numel() < rows * t_step + or pos_off.numel() < rows * t_step + or draft_ids.numel() < rows * d_step + or kv_off.numel() < rows + ): + return False + widths = (src_new.shape[1], dst_new.shape[1], src_next.shape[1], dst_next.shape[1]) + widths += (src_draft.shape[1], dst_draft.shape[1], t_step) + if cpp and max(widths) <= 32: + op = ( + torch.ops.trtllm.minimax_m3_sm107_tp8_fold_tail_one_warp + if one_warp + else torch.ops.trtllm.minimax_m3_sm107_tp8_fold_tail + ) + op( + ctrl, slots, rows, src_new, dst_new, src_next, dst_next, src_draft, dst_draft, + src_lens, dst_lens, ids, draft_ids, pos_off, kv_off, t_step, + ) # fmt: skip + return True + block = triton.next_power_of_2(max(widths)) + _fold_tail_kernel[(rows,)]( + ctrl, + slots, + src_new, + src_new.stride(0), + src_new.stride(1), + src_new.shape[1], + dst_new, + dst_new.stride(0), + dst_new.stride(1), + dst_new.shape[1], + src_next, + src_next.stride(0), + src_next.stride(1), + src_next.shape[1], + dst_next, + dst_next.stride(0), + dst_next.stride(1), + dst_next.shape[1], + src_draft, + src_draft.stride(0), + src_draft.stride(1), + src_draft.shape[1], + dst_draft, + dst_draft.stride(0), + dst_draft.stride(1), + dst_draft.shape[1], + src_lens, + dst_lens, + ids, + draft_ids, + pos_off, + kv_off, + t_step, + d_step, + block, + num_warps=1, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_force_vec.py new file mode 100644 index 000000000000..cc0ce28d9639 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_force_vec.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance (the in-tree speculative worker's +``force_num_accepted_tokens``), in one launch. Reached through the ``ops/spec`` entries ``m3_force_vec`` and +``m3_force_vec_req``. + +Triton. ``force_vec_kernel``: one CTA bumps the shared step counter and draws each request's count from the fixed-seed +pool with the in-tree (counter, slot) hash, exactly as ``_apply_force_accepted_tokens`` does. ``force_vec_req_kernel``: +one draw per request from its own key (its KV length at this verify), no shared counter. +""" + +import triton +import triton.language as tl + + +# One compiled variant for every batch and accept count: specializing them (== 1, divisible by 16) would compile +# mid-serving the first time a step reaches a new one, past the worker's warm-up. +@triton.jit(do_not_specialize=["batch_size", "base_total", "max_total"]) +def force_vec_kernel( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK: tl.constexpr, +): + c = tl.load(counter) + 1 + slots = tl.arange(0, BLOCK) + tl.store(counter, c) + idx = (c * counter_stride + slots.to(tl.int64) * slot_stride) & pool_mask + live = slots < batch_size + r = tl.load(pool + idx, mask=live, other=1.0) + v = tl.minimum(base_total + (r < frac).to(tl.int32), max_total) + tl.store(out + slots, v, mask=live) + + +@triton.jit(do_not_specialize=["batch_size", "base_total", "max_total"]) +def force_vec_req_kernel( + pool, + keys, + out, + batch_size, + base_total, + max_total, + frac, + key_stride, + pool_mask, + BLOCK: tl.constexpr, +): + # One draw per request from its own key (its KV length at this verify): no shared counter, so a request's forced + # counts depend only on its own prompt and progress, not on what else the server ran before or beside it. + slots = tl.arange(0, BLOCK) + live = slots < batch_size + k = tl.load(keys + slots, mask=live, other=0).to(tl.int64) + idx = (k * key_stride) & pool_mask + r = tl.load(pool + idx, mask=live, other=1.0) + v = tl.minimum(base_total + (r < frac).to(tl.int32), max_total) + tl.store(out + slots, v, mask=live) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fullctx_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fullctx_decode.py new file mode 100644 index 000000000000..71666eb05872 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_fullctx_decode.py @@ -0,0 +1,332 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 full-context FP8 decode attention on Rubin: ``fullctx_decode``. + +The dense layers (0-2) attend every token of their FP8 E4M3 (unscaled) paged KV cache. On SM 10.7 the stock call +(flashinfer trtllm-gen, ``minimax_m3_trtllm_gen_dense_decode``) caps its split at 16 cluster CTAs per (request, KV +head): at TP8 (8 query heads and 1 KV head per rank) one request runs on 16 CTAs of 212. + +This runs the same arithmetic -- FP8 Q (the same E4M3 rounding of q), FP8 K/V, P quantized to E4M3 per split -- on +the vendored PrimTS decode kernel (``attention/backends/prims_ts``, SM 10.7 through the sm_100f family) at a pinned +split-KV recipe, which fans one request out over many more CTAs. The split partials are merged by +``torch.ops.trtllm.minimax_m3_sm107_tp2_fullctx_reduce`` (``cpp/tensorrt_llm/kernels/minimaxM3Sm107Tp2/ +fullCtxReduce.cu``, head-count generic, used as is): unnormalized fp32 accumulation over the active splits, one final +division, one BF16 rounding, the merge form of stock trtllm-gen. Only the split changes against stock +(modeling's ``FULLCTX_DECODE_RECIPES``). + +A plan is compiled per (device, query rows per request, requests) by ``prebuild`` before the executor's warmup; the +forward only reads plans (``plan_for``). The plan is built from PrimTS's own config, compile and workspace helpers at +an explicit recipe; the PrimTS wrapper's planning policy is not used, so no PrimTS state is replaced. With +``own_reducer`` off the PrimTS reducer merges into FP16 and the result is cast to BF16. +""" + +from __future__ import annotations + +import threading +from typing import Dict, Mapping, Optional, Tuple + +import torch + +# The TP8 per-rank geometry: 8 query heads over 1 KV head. +HEADS = 8 +KV_HEADS = 1 # capacity: fixed: the tp8 KV heads per rank +HEAD_DIM = 128 # capacity: fixed: the checkpoint's head dim +PAGE = 128 +# The per-request K/V length PrimTS's config selection sizes its policy for (Q/KV tiles, launch mode); the pinned +# recipes were measured with it. The compiled capacity (``max_kv_len``) may be larger. +POLICY_KV_LEN = ( + 1 << 20 +) # capacity: fixed: the PrimTS policy's K/V length bound (the plans' compile shape) +HEADS_PER_KV = HEADS // KV_HEADS # capacity: fixed: the tp2 query heads per KV head + +_plans: Dict[Tuple[int, int, int, bool], "FullCtxPlan"] = {} +_lock = threading.Lock() + + +class FullCtxPlan: + """One compiled PrimTS split-KV decode at a pinned recipe, its workspace and its FP16 staging output.""" + + def __init__( + self, + device: torch.device, + qlen: int, + nreq: int, + max_kv_len: int, + recipe: Mapping, + own_reducer: bool, + pdl: bool, + ): + from tensorrt_llm._torch.attention.backends.prims_ts import decode as tsdec + from tensorrt_llm._torch.attention.backends.prims_ts.kernels.fmha_decode.fmha_decode_config import ( + get_max_active_clusters_for_cluster_size, + make_decode_config, + ) + + self._tsdec = tsdec + self.qlen = qlen + self.nreq = nreq + self.pdl = pdl + self.recipe = dict(recipe) + splits = int(self.recipe["splits"]) + fp8 = tsdec._cutlass_dtype("float8_e4m3fn") + # PDL contract of the main kernel (prims_ts fmha_decode_kernel.py): every CTA acquires its producer + # (griddepcontrol.wait) before any task reads q, lengths or tables, and triggers its dependents + # (launch_dependents) only after its own tail: an active CTA after its last store and TMEM dealloc, an + # inactive split or zero-work CTA right after its wait. The trigger lets the merge launch before the + # slowest CTA ends, so a dependent may read only step-invariant data (weights) before its own + # griddepcontrol.wait, never this step's partial O/LSE, output, q, lengths or tables; the merge + # (minimax_m3_sm107_tp2_fullctx_reduce) waits as its first instruction. + cfg = make_decode_config( + headdim=HEAD_DIM, + args={"use_pdl": pdl, **self.recipe.get("config", {})}, + seq_len_q=qlen, + seq_len_kv=min(max_kv_len, POLICY_KV_LEN), + batch_size=nreq, + num_heads_q=HEADS, + num_heads_kv=KV_HEADS, + q_dtype=fp8, + k_dtype=fp8, + v_dtype=fp8, + o_dtype=tsdec._cutlass_dtype("float16"), + qkv_layout="pagedKv", + num_tokens_per_page=PAGE, + storage_tokens_per_page=PAGE, + split_kv_mode="gmem_reduction_with_separate_kernel", + splits_kv=splits, + max_splits_kv=splits, + min_loop_iters_per_split=1, + mask_type="causal" if qlen > 1 else "dense", + auto_tuner=True, + split_kv=True, + ) + _check_layout(cfg, qlen, splits) + with torch.cuda.device(device): + max_active = get_max_active_clusters_for_cluster_size(1) + spec = tsdec._decode_launch_spec_from_config( + cfg, + batch_size=nreq, + num_qo_heads=HEADS, + num_kv_heads=KV_HEADS, + head_dim=HEAD_DIM, + seq_len_q=qlen, + max_active_clusters=int(max_active), + ) + compile_spec = tsdec._make_decode_compile_spec( + spec, + device_index=_index(device), + num_qo_heads=HEADS, + num_kv_heads=KV_HEADS, + head_dim=HEAD_DIM, + page_size=PAGE, + max_kv_len=max_kv_len, + seq_len_q=qlen, + q_dtype_key="float8_e4m3fn", + k_dtype_key="float8_e4m3fn", + v_dtype_key="float8_e4m3fn", + output_dtype_key="float16", + use_packed_q=False, + kv_prefix_mode="dynamic", + kv_lengths_mode="dynamic", + ) + self._main, self._primts_reducer = tsdec._get_compiled_decode(compile_spec) + layout = tsdec._make_decode_workspace_layout( + spec.scratch_shapes, + torch.float16, + use_separate_reduction_kernel=True, + use_split_kv=True, + ) + self._workspace_buffer = torch.empty(layout.total_bytes, device=device, dtype=torch.int8) + self.workspace = tsdec._bind_decode_workspace(self._workspace_buffer, layout) + self.workspace.split_kv_counter.zero_() + self.workspace.cu_seqlens_q.zero_() + self.workspace.attention_sinks.zero_() + rows = HEADS_PER_KV * qlen + if tuple(self.workspace.partial_o.shape) != (nreq, KV_HEADS, splits, rows, HEAD_DIM): + raise RuntimeError( + f"PrimTS split scratch {tuple(self.workspace.partial_o.shape)} is not the " + f"[requests, KV heads, splits, {rows} rows, {HEAD_DIM}] layout the BF16 merge reads" + ) + self.own_reducer = own_reducer + self.q_tokens_per_cta = max(int(cfg.q_tokens_per_cta), 1) + self.tile_kv = int(cfg.tile_size_kv) + self.insts_kv = int(cfg.num_insts_kv) + self.causal = qlen > 1 + shape = (nreq, qlen, HEADS, HEAD_DIM) if qlen > 1 else (nreq, HEADS, HEAD_DIM) + self.staging = torch.empty(shape, dtype=torch.float16, device=device) + + def run( + self, + q: torch.Tensor, + k_pages: torch.Tensor, + v_pages: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + mxfp8: Optional[Tuple[torch.Tensor, torch.Tensor]] = None, + ) -> bool: + """Decodes into output; with mxfp8 (E4M3 rows, UE8M0 scales) and the TP8 merge, the merge also writes + the MXFP8 copy of output. Returns whether it did.""" + # The kernel takes a compact query; a column view of the fused QKV projection is copied once. + q = q.reshape(self.staging.shape).contiguous() + runtime = self._tsdec._prepare_decode_runtime_unchecked( + q, + (k_pages, v_pages), + None, + output_dtype=torch.float16, + bmm1_scale=sm_scale, + bmm2_scale=1.0, + out=self.staging, + ) + ws = self.workspace + self._main( + runtime.q, + runtime.k_cache, + runtime.v_cache, + runtime.k_sf_cache, + runtime.v_sf_cache, + runtime.out, + seq_lens, + ws.cu_seqlens_q, + block_table, + block_table, # Dense decode constexpr-elides the membership table. + ws.partial_o, + ws.partial_stats, + ws.split_kv_counter, + ws.attention_sinks, + runtime.num_physical_pages, + runtime.k_page_stride, + runtime.k_head_stride, + runtime.k_token_stride, + runtime.v_page_stride, + runtime.v_head_stride, + runtime.v_token_stride, + runtime.bmm1_scale, + runtime.bmm2_scale, + (), + ) + if self.own_reducer: + reduce = _reduce_op() + args = ( + ws.partial_o, + ws.partial_stats, + seq_lens, + output, + self.qlen, + self.q_tokens_per_cta, + self.tile_kv, + self.insts_kv, + self.causal, + self.pdl, + ) + if mxfp8 is not None and reduce is _tp8_reduce_op(): + reduce(*args, mxfp8[0], mxfp8[1]) + return True + reduce(*args) + return False + self._primts_reducer( + runtime.out, + seq_lens, + ws.cu_seqlens_q, + ws.partial_o, + ws.partial_stats, + ws.attention_sinks, + runtime.bmm1_scale, + runtime.bmm2_scale, + (), + ) + output.view(self.staging.shape).copy_(self.staging) + return False + + +def _check_layout(cfg, qlen: int, splits: int) -> None: + """The scratch layout and split ranges the BF16 merge (``fullCtxReduce.cu`` ``activeSplits``) assumes. + + Rows are [token, local head]; a row's split ranges are those of its grouped Q CTA, whose first token is + ``(token // q_tokens_per_cta) * q_tokens_per_cta``; tiles are grouped ``num_insts_kv`` per loop step; no sliding + window, attention sinks or packed Q. + """ + problems = [] + if not (cfg.use_split_kv and cfg.use_separate_reduction_kernel): + problems.append("not split-KV with the separate GMEM reducer") + if int(cfg.splits_kv) != splits or int(cfg.max_splits_kv) != splits: + problems.append(f"splits {cfg.splits_kv}/{cfg.max_splits_kv}, recipe {splits}") + if cfg.use_cluster_smem_reduction or cfg.use_persistent_scheduler: + problems.append("cluster-SMEM reduction or a persistent scheduler") + if cfg.use_variable_seqlens_q or cfg.use_sliding_window_causal or cfg.use_attention_sinks: + problems.append("packed Q, a sliding window or attention sinks") + if int(cfg.heads_q_per_kv) not in (0, HEADS_PER_KV): + problems.append(f"{cfg.heads_q_per_kv} query heads per KV head") + if qlen > 1: + if not (cfg.groups_tokens_heads_q and cfg.uses_nontrivial_grouped_q_layout): + problems.append("multi-token Q without the grouped [token, head] tile layout") + elif int(cfg.q_tokens_per_cta) * HEADS_PER_KV != int(cfg.tile_size_q): + problems.append( + f"{cfg.q_tokens_per_cta} tokens per CTA in a {cfg.tile_size_q}-row tile" + ) + if problems: + raise RuntimeError("PrimTS recipe outside the BF16 merge's layout: " + "; ".join(problems)) + + +# The split-KV merge runs the TP8 copy of the BF16 reducer when the op library carries it (the same bits). +TP8_REDUCER = True + + +def _reduce_op(): + """The BF16 split-KV merge: the TP8 copy (one row per CTA, its dependent released right after its wait) when the + TP8 op library carries it, else the image's TP2 op; the same per-row arithmetic, so the same bits.""" + ops = torch.ops.trtllm + if TP8_REDUCER and hasattr(ops, "minimax_m3_sm107tp8_fullctx_reduce"): + return ops.minimax_m3_sm107tp8_fullctx_reduce + return ops.minimax_m3_sm107_tp2_fullctx_reduce + + +def _tp8_reduce_op(): + return getattr(torch.ops.trtllm, "minimax_m3_sm107tp8_fullctx_reduce", None) + + +def emits_mxfp8() -> bool: + """Whether the merge run uses can also write the o_proj MXFP8 hand-off (the TP8 copy of the merge).""" + return _tp8_reduce_op() is not None and _reduce_op() is _tp8_reduce_op() + + +def has_own_reducer() -> bool: + """Whether this build carries a BF16 split-KV merge op (the TP8 copy or the image's TP2 op).""" + ops = torch.ops.trtllm + return hasattr(ops, "minimax_m3_sm107tp8_fullctx_reduce") or hasattr( + ops, "minimax_m3_sm107_tp2_fullctx_reduce" + ) + + +def plan_for( + device: torch.device, qlen: int, nreq: int, own_reducer: bool = True +) -> Optional[FullCtxPlan]: + """The prebuilt plan for this shape, or None.""" + return _plans.get((_index(device), qlen, nreq, own_reducer)) + + +def prebuild( + device: torch.device, + recipes: Mapping[Tuple[int, int], Mapping], + max_kv_len: int, + own_reducer: bool, + pdl: bool, +) -> int: + """Compile the plan of every (query rows per request, requests) in ``recipes``. Call before warmup.""" + built = 0 + for (qlen, nreq), recipe in recipes.items(): + key = (_index(device), qlen, nreq, own_reducer) + with _lock: + if key not in _plans: + _plans[key] = FullCtxPlan(device, qlen, nreq, max_kv_len, recipe, own_reducer, pdl) + built += 1 + return built + + +def kv_page_views(pool: torch.Tensor, subpages_per_slot: int) -> Tuple[torch.Tensor, torch.Tensor]: + """K and V page views of a flat sub-page pool (K of slot s at sub-page s * f, V one later), indexed by slot.""" + return pool[0::subpages_per_slot], pool[1::subpages_per_slot] + + +def _index(device: torch.device) -> int: + return device.index if device.index is not None else torch.cuda.current_device() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_index_score.py new file mode 100644 index 000000000000..b158c6aafc2d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_index_score.py @@ -0,0 +1,712 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py +"""MiniMax-M3 index decode block scorer (CuTe DSL, Blackwell) with its pre-wait schedule. + +The kernel of ``tensorrt_llm/_torch/attention/kernels/blackwell/minimax_m3_index_decode_score.py`` (the same scores, bit +for bit) with four schedule options this target compiles in: + +* ``prefetch_old``: before the grid-dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that hold only + tokens older than this step (the producer ahead writes only this step's tokens), up to ``prefetch_depth`` blocks per + CTA (0 = all of them). +* ``early_issue``: the TMA warp reads a ``seq_lens`` hint and its first page id before the wait and issues the Q tile + and the first K tile right after it. The tiles are read after the wait, so their contents are final; the block table + is only written between graph replays, so the early page id is the right one whenever the block is valid. A CTA + whose early tiles lie past the request's true length drains them before it exits. +* ``prestage_old`` (with ``early_issue``, ``INDEX_SCORE_PRESTAGE_OLD``): when the CTA's first block holds only + tokens older than this step (block < (seq_lens hint - decode query length) / 128), its K tile is TMA-loaded into its + stage before the wait; Q (and a first block that is not all old) is still loaded after it. +* ``early_trigger`` (``INDEX_SCORE_SET``): every CTA releases the dependent block selector right after its + setup, so the selector launches and makes its pre-wait reads while this grid scores; the selector still waits for + this whole grid before it reads a score, so the scores and the selection are unchanged. + +Every kernel is compiled before the executor's warmup (``layers.prebuild_index_scorer``) through ``IndexScoreRunner`` +and called through this target's ``m3_index_score`` entry. +""" + +from pathlib import Path + +import cutlass +import torch +from cuda.bindings.driver import CUstream +from cutlass import Float8E4M3FN, Float16, Float32, Int32, Int64, Uint32, cute +from cutlass._mlir.dialects import llvm +from cutlass.cute.nvgpu import cpasync, warp +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + EVICT_FIRST, + fp8x4_to_fp16x4, + mma_sync, + simple_tma_copy, +) +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + TRTLLM_ENABLE_PDL, + griddepcontrol_launch_dependents, + griddepcontrol_wait, +) +from tensorrt_llm.logger import logger + +# With the early issue, a CTA whose first block holds only tokens older than this step also TMA-loads that block's +# index-K tile before the grid wait (bit-identical scores). +INDEX_SCORE_PRESTAGE_OLD = True +# Every CTA releases the dependent block selector right after its setup instead of at its exit (the selector's own +# grid wait still returns only once the whole scoring grid is complete, so the picks are unchanged), at every key. +INDEX_SCORE_SET = True + +__all__ = ["IndexDecodeScoreKernel", "IndexScoreRunner"] + + +@dsl_user_op +def _l2_prefetch_bulk(addr: Int64, size: Int32, *, loc=None, ip=None) -> None: + """Prefetch ``size`` contiguous bytes (multiple of 16) at global ``addr`` into L2.""" + llvm.inline_asm( + res=None, + operands_=[Int64(addr).ir_value(loc=loc, ip=ip), Int32(size).ir_value(loc=loc, ip=ip)], + asm_string="cp.async.bulk.prefetch.L2.global [$0], $1;", + constraints="l,r", + has_side_effects=True, + asm_dialect=llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@cute.jit +def _fp8_to_f16_mma_fragments(src: cute.Tensor): + """Split an FP8 ldmatrix fragment into the two FP16 k-fragments MMA wants.""" + src_elems = cute.size(src) + src_u32 = cute.recast_tensor(src, Uint32) + src_f16 = cute.make_rmem_tensor(src_elems, Float16) + src_f16_u32 = cute.recast_tensor(src_f16, Uint32) + # Packed conversion; faster and fewer SASS instructions than + # src.load().to(Float16). + for i in cutlass.range_constexpr(src_elems // 4): + converted = fp8x4_to_fp16x4(src_u32[i]) + src_f16_u32[i * 2] = converted[0] + src_f16_u32[i * 2 + 1] = converted[1] + lower = cute.make_rmem_tensor(src_elems // 2, Float16) + upper = cute.make_rmem_tensor(src_elems // 2, Float16) + + # FP8 ldmatrix yields four consecutive values along K. Split each group + # into the lower two and upper two for the two FP16 MMA k-fragments. + for i in cutlass.range_constexpr(src_elems // 2): + lower[i] = src_f16[(i // 2) * 4 + i % 2] + upper[i] = src_f16[(i // 2) * 4 + 2 + i % 2] + return lower, upper + + +class IndexDecodeScoreKernel: + """Per-block max index score for one decode step. + + Grid is (batch, split_k): CTA (b, s) walks KV blocks + s, s + split_k, s + 2 * split_k, ... of request b and writes each block's + score directly, so no cross-CTA reduction is needed. CTAs whose split_id + exceeds the request's block count exit immediately. + """ + + BLOCK_K = 128 # capacity: fixed: the MSA block size of the checkpoint + BAR_MMA = 1 + num_stages = 2 + + def __init__( + self, + dtype: type[cutlass.Numeric], + num_heads: int, + max_decode_query_len: int, + split_k: int, + head_dim: int = 128, + prefetch_old: bool = False, + early_issue: bool = False, + prefetch_depth: int = 0, + prestage_old: bool = False, + early_trigger: bool = False, + ): + # prefetch_old: before the grid dependency wait, the TMA warp L2-prefetches this CTA's index-K pages that + # hold only tokens older than this step (the predecessor writes only this step's tokens). + self.prefetch_old = prefetch_old + # early_issue: the TMA warp reads a seq_lens hint and this CTA's first page id before the grid dependency + # wait, then issues the Q tile and the first K tile right after it, overlapping the post-wait seq_lens load. + # The tiles are read after the wait, so their contents are final; the block table is only written between + # graph replays, so the early page id is the right one whenever the block is valid. A CTA whose early tiles + # turn out to lie past the request's true length drains them before exiting. + self.early_issue = early_issue + # prefetch_depth: blocks per CTA that prefetch_old covers (0 = all of them). Prefetching every older page at + # large batches floods L2; the first few per CTA are the ones whose latency the pipeline cannot hide. + self.prefetch_depth = prefetch_depth + # prestage_old (with early_issue): when this CTA's first block holds only tokens older than this step, its K + # tile is TMA-loaded into its stage BEFORE the grid dependency wait (the predecessor writes only this step's + # tokens; the block table and the seq_lens hint are final before the grid starts). Q stays after the wait. + self.prestage_old = prestage_old and early_issue + # early_trigger: release the dependent block selector right after this CTA's setup (see the module docstring). + self.early_trigger = early_trigger + self.dtype = dtype + self.num_heads = num_heads + self.max_decode_query_len = max_decode_query_len + self.split_k = split_k + self.head_dim = head_dim + + def _swizzle_elems(self) -> int: + """Elements spanned by one 128-byte swizzle atom.""" + return 128 * 8 // self.dtype.width + + def _sq_layout(self): + """Composed SMEM layout for the Q tile, shared by the descriptor and the kernel.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_q = self.num_heads * self.max_decode_query_len + layout = cute.make_layout( + (self.max_decode_query_len, self.num_heads, (elems, head_dim // elems)), + stride=(elems, self.max_decode_query_len * elems, (1, block_q * elems)), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + def _sk_layout(self): + """Composed SMEM layout for the pipelined K tiles.""" + elems = self._swizzle_elems() + head_dim = self.head_dim + block_k = self.BLOCK_K + layout = cute.make_layout( + (1, block_k, (elems, head_dim // elems), self.num_stages), + stride=(0, elems, (1, block_k * elems), block_k * head_dim), + ) + return cute.make_composed_layout(cute.make_swizzle(3, 4, 3), 0, layout) + + @cute.jit + def __call__( + self, + gQ: cute.Tensor, # [bs * runtime_decode_query_len, num_heads, head_dim] + gK_cache: cute.Tensor, # [num_pages, page_size, head_dim] + block_table: cute.Tensor, # [bs, max_pages] + score: cute.Tensor, # [num_heads, bs * runtime_decode_query_len, max_pages] + seq_lens: cute.Tensor, # [bs] + stream: CUstream, + ): + num_heads = self.num_heads + head_dim = self.head_dim + MAX_DQL = self.max_decode_query_len + assert num_heads * MAX_DQL <= 32 + + batch = seq_lens.shape[0] + decode_query_len = gQ.shape[0] // batch + grid = (batch, self.split_k, 1) + block = (32 * 5, 1, 1) + + tma_g2s = cpasync.CopyBulkTensorTileG2SOp() + elems = self._swizzle_elems() + + q_tma_atom, q_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gQ, (None, None, elems)), + self._sq_layout(), + (MAX_DQL, num_heads, head_dim), + ) + k_tma_atom, k_tma_tensor = cpasync.make_tiled_tma_atom( + tma_g2s, + cute.logical_divide(gK_cache, (None, None, elems)), + self._sk_layout(), + (1, self.BLOCK_K, head_dim), + ) + + self.kernel( + q_tma_atom, + q_tma_tensor, + k_tma_atom, + k_tma_tensor, + gK_cache, + block_table, + score, + seq_lens, + decode_query_len, + ).launch(grid=grid, block=block, stream=stream, use_pdl=TRTLLM_ENABLE_PDL) + + @cute.kernel + def kernel( + self, + q_tma_atom: cute.CopyAtom, + q_tma_tensor: cute.Tensor, + k_tma_atom: cute.CopyAtom, + k_tma_tensor: cute.Tensor, + gK_cache: cute.Tensor, + block_table: cute.Tensor, + score: cute.Tensor, + seq_lens: cute.Tensor, + decode_query_len, + ): + batch_id, split_id, _ = cute.arch.block_idx() + _, split_k, _ = cute.arch.grid_dim() + warp_id = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane_id = cute.arch.lane_idx() + + NUM_HEADS = self.num_heads + MAX_DQL = self.max_decode_query_len + BLOCK_Q = NUM_HEADS * MAX_DQL + BLOCK_K = self.BLOCK_K + head_dim = self.head_dim + dtype = self.dtype + MMA_N = 8 + num_stages = self.num_stages + Q_TILES = cute.ceil_div(BLOCK_Q, MMA_N) + EPI_Q = Q_TILES * MMA_N + + sq_layout = self._sq_layout() + sk_layout = self._sk_layout() + + smem = SmemAllocator() + sK = smem.allocate_tensor( + dtype, + sk_layout.outer, + byte_alignment=128, + swizzle=sk_layout.inner, + )[0, None, None, None] + # sQ aliases the first K stage: Q is consumed into registers before the + # first K tile is needed, so the two never overlap in time. + sQ_tma = cute.make_tensor(sK[None, None, 0].iterator, layout=sq_layout.outer) + # TMA sees Q as (query, head, dim) while ldmatrix consumes a flattened Q + # column mode. The target profile keeps the rank-2 view even for + # degenerate shapes such as MAX_DQL == 1. + q_tma_elems = self._swizzle_elems() + sQ = cute.coalesce( + cute.group_modes(sQ_tma, 0, 2), + target_profile=(BLOCK_Q, (q_tma_elems, head_dim // q_tma_elems)), + ) + epi_buffer = smem.allocate_tensor(Float32, cute.make_layout((EPI_Q, 4))) + + tma_full_mbar = smem.allocate_array(Int64, num_stages) + tma_empty_mbar = smem.allocate_array(Int64, num_stages) + + # Barrier init and descriptor prefetch do not depend on the predecessor + # grid, so they are the work PDL can overlap. Every CTA runs them, even + # the ones that own no blocks, since ownership is not known until + # seq_lens is read below. + if warp_id == 0: + with cute.arch.elect_one(): + for i in cutlass.range_constexpr(num_stages): + cute.arch.mbarrier_init(tma_full_mbar + i, 1) + cute.arch.mbarrier_init(tma_empty_mbar + i, 128) + cute.arch.mbarrier_init_fence() + elif warp_id == 1: + cpasync.prefetch_descriptor(q_tma_atom) + cpasync.prefetch_descriptor(k_tma_atom) + cute.arch.sync_threads() + if cutlass.const_expr(self.early_trigger): + griddepcontrol_launch_dependents() + + if cutlass.const_expr(self.prefetch_old): + if warp_id == 4: + with cute.arch.elect_one(): + seqlen0 = seq_lens[batch_id] + num_blocks0 = cute.ceil_div(seqlen0, BLOCK_K) + stable = (seqlen0 - decode_query_len) // BLOCK_K + if stable > num_blocks0: + stable = num_blocks0 + if cutlass.const_expr(self.prefetch_depth > 0): + if stable > split_id + self.prefetch_depth * split_k: + stable = split_id + self.prefetch_depth * split_k + for block_id in range(split_id, stable, split_k): + page = block_table[batch_id, block_id] + _l2_prefetch_bulk( + gK_cache[page, None, None].iterator.toint().to(Int64), + Int32(BLOCK_K * head_dim * (dtype.width // 8)), + ) + + # Without early_trigger the dependent block selector is released as the CTAs exit. + early_q = False + early_k = False + early_page = Int32(0) + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + seqlen_hint = seq_lens[batch_id] + hint_blocks = cute.ceil_div(seqlen_hint, BLOCK_K) + if hint_blocks > block_table.shape[1]: + hint_blocks = block_table.shape[1] + if split_id < hint_blocks: + early_q = True + early_page = block_table[batch_id, split_id] + if cutlass.const_expr(self.prestage_old): + # Stage the first K tile now when every token of the block predates this step. + if split_id < (seqlen_hint - decode_query_len) // BLOCK_K: + early_k = True + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, + BLOCK_K * head_dim * (dtype.width // 8), + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + griddepcontrol_wait() + + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + gQ_early = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar, BLOCK_Q * head_dim * (dtype.width // 8) + ) + simple_tma_copy(q_tma_atom, gQ_early, sQ_tma, tma_full_mbar) + if not early_k: + cute.arch.mbarrier_wait(tma_empty_mbar + 1 % num_stages, 1) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + tma_full_mbar + 1 % num_stages, + BLOCK_K * head_dim * (dtype.width // 8), + ) + simple_tma_copy( + k_tma_atom, + k_tma_tensor[early_page, None, None], + sK[None, None, 1 % num_stages], + tma_full_mbar + 1 % num_stages, + cache_policy=EVICT_FIRST, + ) + + # The length that bounds this grid's work is read after the wait (the + # pre-wait reads above are prefetch / early-issue hints only, clamped to + # the table width). seq_lens is a graph-stable buffer that on_update_kv_lens + # patches every step, and num_blocks bounds an unmasked block_table load + # and an unmasked score store, so a stale length is an out-of-bounds write + # rather than just a wrong answer. + seqlen = seq_lens[batch_id] + num_blocks = cute.ceil_div(seqlen, BLOCK_K) + + if split_id < num_blocks: + if warp_id == 4: + # TMA warp + tma_stage = 0 + tma_parity = 1 + first_block = split_id + + if not early_q: + gQ_tile = cute.local_tile( + cute.domain_offset((batch_id * decode_query_len, 0, 0), q_tma_tensor), + tiler=(MAX_DQL, NUM_HEADS, head_dim), + coord=(0, 0, 0), + ) + cute.arch.mbarrier_wait(tma_empty_mbar, tma_parity) + with cute.arch.elect_one(): + Q_size = BLOCK_Q * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(tma_full_mbar, Q_size) + # TMA bounds-checks rows when the runtime decode_query_len is + # below MAX_DQL; padded Q columns are masked before the stores. + simple_tma_copy(q_tma_atom, gQ_tile, sQ_tma, tma_full_mbar) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + if early_q: + # The first K tile was issued right after the wait. + first_block = split_id + split_k + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + for block_id in range(first_block, num_blocks, split_k): + page_id = block_table[batch_id, block_id] + gK_tile = k_tma_tensor[page_id, None, None] + k_mbar = tma_full_mbar + tma_stage + + cute.arch.mbarrier_wait(tma_empty_mbar + tma_stage, tma_parity) + with cute.arch.elect_one(): + K_size = BLOCK_K * head_dim * (dtype.width // 8) + cute.arch.mbarrier_arrive_and_expect_tx(k_mbar, K_size) + simple_tma_copy( + k_tma_atom, + gK_tile, + sK[None, None, tma_stage], + k_mbar, + cache_policy=EVICT_FIRST, + ) + + tma_stage = (tma_stage + 1) % num_stages + if tma_stage == 0: + tma_parity ^= 1 + + else: + # MMA warps; each handles K[32, head_dim] @ Q[BLOCK_Q, head_dim].T + sK_warp = cute.local_tile(sK, (32, head_dim, num_stages), (warp_id, 0, 0)) + q_start = seqlen - decode_query_len + + elems = 128 // dtype.width # 16B + MMA_K = 32 * 8 // dtype.width # 32B + + # Pre-compute ldmatrix addresses. + # sK loads a [16 x 16B] tile: + # ((16, (16B, 2), 1), (32 / 16, head_dim / 32B, num_stages)) + # sQ loads an [8 x 32B] tile: + # ((8, (16B, 4)), (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = cute.zipped_divide(sK_warp, (16, cute.make_layout((elems, 2)), 1)) + sQ_ldsm = cute.zipped_divide(sQ, (MMA_N, cute.make_layout((elems, 4)))) + + # sK: (16B, (32 / 16, head_dim / 32B, num_stages)) + # sQ: (16B, (BLOCK_Q / MMA_N, head_dim / 64B)) + sK_ldsm = sK_ldsm[(lane_id % 16, (None, lane_id // 16), 0), None] + sQ_ldsm = sQ_ldsm[(lane_id % MMA_N, (None, lane_id // 8)), None] + + ldsm_op = warp.LdMatrix8x8x16bOp(num_matrices=4) + ldsm_atom = cute.make_copy_atom(ldsm_op, dtype) + + rQ = cute.make_rmem_tensor( + ((elems // 2, 2), head_dim // (MMA_K * 2), Q_TILES), dtype + ) + rK = cute.make_rmem_tensor((elems, 2, head_dim // MMA_K), dtype) + rC = cute.make_rmem_tensor((4, 2, Q_TILES), Float32) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + for q in cutlass.range_constexpr(Q_TILES): + cute.copy(ldsm_atom, sQ_ldsm[None, (q, None)], rQ[None, None, q]) + cute.arch.mbarrier_arrive(tma_empty_mbar) + + tma_stage = 1 % self.num_stages + tma_parity = 0 + if tma_stage == 0: + tma_parity ^= 1 + + if cutlass.const_expr(dtype is Float8E4M3FN): + rQ_f16 = cute.make_rmem_tensor((4, head_dim // MMA_K, Q_TILES, 2), Float16) + q_lower, q_upper = _fp8_to_f16_mma_fragments(rQ) + rQ_f16[None, None, None, 0].store(q_lower.load()) + rQ_f16[None, None, None, 1].store(q_upper.load()) + + for block_id in range(split_id, num_blocks, split_k): + rC.fill(0.0) + + if warp_id == 0: + cute.arch.mbarrier_wait(tma_full_mbar + tma_stage, tma_parity) + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + for k in cutlass.range_constexpr(head_dim // MMA_K): + cute.copy( + ldsm_atom, + sK_ldsm[None, (None, k, tma_stage)], + rK[None, None, k], + ) + for m in cutlass.range_constexpr(2): + if cutlass.const_expr(dtype is Float8E4M3FN): + rK_lower, rK_upper = _fp8_to_f16_mma_fragments(rK[None, m, k]) + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK_lower, + rQ_f16[None, k, n, 0], + rC[None, m, n], + ) + rC[None, m, n] = mma_sync( + rK_upper, + rQ_f16[None, k, n, 1], + rC[None, m, n], + ) + else: + for n in cutlass.range_constexpr(Q_TILES): + rC[None, m, n] = mma_sync( + rK[None, m, k], + rQ[(None, k % 2), k // 2, n], + rC[None, m, n], + ) + + cute.arch.mbarrier_arrive(tma_empty_mbar + tma_stage) + + k_start = block_id * BLOCK_K + warp_id * 32 + + for q in cutlass.range_constexpr(Q_TILES): + for i in cutlass.range_constexpr(4): + for j in cutlass.range_constexpr(2): + col = q * 8 + (lane_id % 4) * 2 + j + q_local_pos = col % MAX_DQL + q_pos = q_start + q_local_pos + k_pos = k_start + i * 8 + lane_id // 4 + rC[q * 8 + i * 2 + j] = ( + rC[q * 8 + i * 2 + j] if q_pos >= k_pos else float("-inf") + ) + + for q in cutlass.range_constexpr(Q_TILES): + # Thread-local reduction along the BLOCK_K dim. + rScore = cute.make_rmem_tensor(2, Float32) + rScore.fill(float("-inf")) + for i in cutlass.range_constexpr(4): + rScore[0] = cute.arch.fmax(rScore[0], rC[i * 2 + 0 + q * 8]) + rScore[1] = cute.arch.fmax(rScore[1], rC[i * 2 + 1 + q * 8]) + + # Warp reduction among lanes 0, 4, 8, 12, ... + for i in cutlass.range_constexpr(3): + offset = 4 << i + other0 = cute.arch.shuffle_sync_bfly( + rScore[0], offset=offset, mask=-1, mask_and_clamp=31 + ) + other1 = cute.arch.shuffle_sync_bfly( + rScore[1], offset=offset, mask=-1, mask_and_clamp=31 + ) + rScore[0] = cute.arch.fmax(rScore[0], other0) + rScore[1] = cute.arch.fmax(rScore[1], other1) + + if lane_id * 2 < MMA_N: + epi_buffer[q * MMA_N + lane_id * 2 + 0, warp_id] = rScore[0] + epi_buffer[q * MMA_N + lane_id * 2 + 1, warp_id] = rScore[1] + cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) + + head_id = lane_id // MAX_DQL + q_local_pos = lane_id - head_id * MAX_DQL + valid_q = head_id < NUM_HEADS and q_local_pos < decode_query_len + if lane_id < BLOCK_Q and valid_q: + final_score = epi_buffer[lane_id, 0] + for i in cutlass.range_constexpr(1, 4): + final_score = cute.arch.fmax(final_score, epi_buffer[lane_id, i]) + + t = batch_id * decode_query_len + q_local_pos + score[head_id, t, block_id] = final_score + + tma_stage = (tma_stage + 1) % self.num_stages + if tma_stage == 0: + tma_parity ^= 1 + else: + if cutlass.const_expr(self.early_issue): + if warp_id == 4: + if early_q: + # Early tiles past the request's true length: let them land before the CTA exits. + cute.arch.mbarrier_wait(tma_full_mbar, 0) + cute.arch.mbarrier_wait(tma_full_mbar + 1 % num_stages, 0) + + +class IndexScoreRunner: + """Compiled ``IndexDecodeScoreKernel`` variants, keyed by (index dtype, index heads, decode query length, head dim, + split_k, early issue). Batch, query-token count, page count and block-table width stay symbolic, so one compile + serves every decode step of its key. ``prefetch_old`` is always on; ``early`` also caps the prefetch at one block + per CTA; ``early_trigger`` is ``INDEX_SCORE_SET`` for every key of the process.""" + + kernel_cache: dict = {} + SUPPORTED_HEAD_DIM = 128 # capacity: fixed: the checkpoint's index head dim + SUPPORTED_PAGE_SIZE = IndexDecodeScoreKernel.BLOCK_K + # BLOCK_Q must fit one warp's worth of epilogue lanes. + MAX_BLOCK_Q = 32 # capacity: fixed: the scorer's query tile + EARLY_PREFETCH_DEPTH = 1 + + @classmethod + def is_supported( + cls, + *, + q_dtype: torch.dtype, + num_heads: int, + head_dim: int, + page_size: int, + max_decode_query_len: int, + ) -> bool: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + from tensorrt_llm._utils import is_sm_100f + + return ( + is_sm_100f() + and q_dtype in TORCH_TO_CUTE_DTYPE + and head_dim == cls.SUPPORTED_HEAD_DIM + and page_size == cls.SUPPORTED_PAGE_SIZE + and num_heads * max_decode_query_len <= cls.MAX_BLOCK_Q + and max_decode_query_len >= 1 + ) + + @classmethod + def _compile( + cls, + q_dtype: torch.dtype, + num_heads: int, + max_decode_query_len: int, + head_dim: int, + split_k: int, + early: bool, + ) -> None: + from tensorrt_llm._torch.cute_dsl_kernels.blackwell.cute_ptx_utils import ( + TORCH_TO_CUTE_DTYPE, + ) + + key = (q_dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + if key in cls.kernel_cache: + return + cute_dtype = TORCH_TO_CUTE_DTYPE[q_dtype] + page_size = cls.SUPPORTED_PAGE_SIZE + sym_total_tokens = cute.sym_int() + sym_batch = cute.sym_int() + + # 16-element divisibility on every non-innermost stride is what the TMA descriptors for Q and K assume. + def _sym_stride(): + return cute.sym_int64(divisibility=16) + + q_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (sym_total_tokens, num_heads, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + # The index-K pool may be coalesced with the main K/V cache (per-page stride above page_size * head_dim). + k_fake = cute.runtime.make_fake_tensor( + cute_dtype, + (cute.sym_int(), page_size, head_dim), + stride=(_sym_stride(), _sym_stride(), 1), + ) + bt_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch, cute.sym_int()), stride_order=(1, 0) + ) + # The target passes a transposed view of the [heads, blocks, tokens] selector buffer. + score_fake = cute.runtime.make_fake_tensor( + cutlass.Float32, + (num_heads, sym_total_tokens, cute.sym_int()), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + sl_fake = cute.runtime.make_fake_compact_tensor( + cutlass.Int32, (sym_batch,), stride_order=(0,) + ) + fake_stream = cute.runtime.make_fake_stream(use_tvm_ffi_env_stream=True) + kernel = IndexDecodeScoreKernel( + cute_dtype, + num_heads, + max_decode_query_len, + split_k, + head_dim, + prefetch_old=True, + early_issue=early, + prefetch_depth=cls.EARLY_PREFETCH_DEPTH if early else 0, + prestage_old=early and INDEX_SCORE_PRESTAGE_OLD, + early_trigger=INDEX_SCORE_SET, + ) + if INDEX_SCORE_SET and not cls.kernel_cache: + logger.info( + "tp8 index scorer: the block selector is released after the scorer's setup (INDEX_SCORE_SET)" + ) + cls.kernel_cache[key] = cute_aot.compile_or_load( + f"m3_index_score {key}", + lambda: cute.compile( + kernel, + q_fake, + k_fake, + bt_fake, + score_fake, + sl_fake, + fake_stream, + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_layer/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_layer/__init__.py new file mode 100644 index 000000000000..467079831e16 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_layer/__init__.py @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_layer/phase_producer.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_layer/phase_producer.py new file mode 100644 index 000000000000..8d6fe9e43a1f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_layer/phase_producer.py @@ -0,0 +1,489 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode producer as a phase of the TP8 per-layer kernel (m3_layer): q / k / index-q / index-k RMSNorm + NeoX +partial RoPE, NVFP4 K / V and FP8 index-K cache insert, FP8 q / index-q out. + +Bit for bit ``torch.ops.trtllm.minimax_m3_producer_pw`` (cpp/tensorrt_llm/kernels/minimaxM3/producerPw.cu, +m3_producer_kernel<8, *>) at the TP8 geometry: 8 query heads, 1 KV head, 1 index head, head_dim 128, rotary_dim 64, +page 128. One work item = one decode token; the CTA's warp w (< 12) owns packed head w of it, lane l holds elements +[4l, 4l + 4) exactly as the C++ kernel, so every reduction (the RMS sum of squares, the RoPE pair exchange, the +NVFP4 block max) runs the same shuffle tree in the same order. Every floating-point operation is written as its PTX +instruction (mul.rn / add.rn / fma.rn / rsqrt.approx / rcp.approx.ftz / cvt) so the DSL compiler cannot contract +or reassociate differently from the library build (--use_fast_math). The places where that build contracted are +marked CONTRACT and follow the shipped SASS (msa/results/sassdump-*). + +Packed row layout per token: [8 q | k | v | index_q | index_k] x 128 BF16 (the QKV GEMV output rounded to BF16). +""" + +from __future__ import annotations + +import cutlass +import cutlass.cute as cute +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op + +HEADS_Q = 8 # capacity: fixed: the tp8 query heads per rank +PACKED_HEADS = HEADS_Q + 4 # q x 8, k, v, index_q, index_k +HEAD_DIM = 128 +ROTARY_DIM = 64 # capacity: fixed: the checkpoint's rotary dim +PAGE = 128 +ELEMS = HEAD_DIM // 32 # capacity: fixed: elements per lane (per lane) +K_HEAD, V_HEAD, IQ_HEAD, IK_HEAD = HEADS_Q, HEADS_Q + 1, HEADS_Q + 2, HEADS_Q + 3 + + +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +# ---- exact float arithmetic (no contraction) --------------------------------------------------------------------- +@dsl_user_op +def f_mul(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_add(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_fma(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_max(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "max.f32 $0, $1, $2;", + "=f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def f_abs(a, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "abs.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def f_neg(a, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "neg.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def f_rsqrt(a, *, loc=None, ip=None): + # NOT ftz: both producer builds this one is held to (the library producerPw and the target's own) run + # rsqrt.approx.f32 with denormal scaling; .ftz differed on a denormal RMS argument (eps 0). + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rsqrt.approx.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def f_rcp(a, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", loc=loc, ip=ip) + ) + + +@dsl_user_op +def bf16_bits_to_f32(bits16, *, loc=None, ip=None): + """The float value of a BF16 given in the low 16 bits of an Int32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(bits16)], + "{\n\t.reg .b32 t;\n\tshl.b32 t, $1, 16;\n\tmov.b32 $0, t;\n\t}", + "=f,r", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def round_bf16(a, *, loc=None, ip=None): + """__bfloat162float(__float2bfloat16_rn(a)).""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a)], + "{\n\t.reg .b16 h;\n\t.reg .b32 t;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b32 t, {0, h};\n\tmov.b32 $0, t;\n\t}", + "=f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def e4m3x4_word(e0, e1, e2, e3, *, loc=None, ip=None): + """store_fp8x4: bytes [e0, e1, e2, e3] (cvt.rn.satfinite.e4m3x2.f32 puts its first source in the high byte).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(e0), cutlass.Float32(e1), cutlass.Float32(e2), cutlass.Float32(e3)], + "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}", + "=r,f,f,f,f", + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def e4m3_byte_and_value(x, *, loc=None, ip=None): + """(byte, float value) of __nv_fp8_e4m3(x): the encoded scale and static_cast of it.""" + byte = cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n\t.reg .b16 p;\n\t.reg .b32 w;\n\tcvt.rn.satfinite.e4m3x2.f32 p, 0f00000000, $1;\n\t" + "cvt.u32.u16 w, p;\n\tand.b32 $0, w, 0xFF;\n\t}", + "=r,f", + loc=loc, + ip=ip, + ) + ) + value = cutlass.Float32( + _asm( + _T.f32(), + [byte], + "{\n\t.reg .b16 p;\n\t.reg .b32 h2;\n\t.reg .b16 h;\n\tcvt.u16.u32 p, $1;\n\t" + "cvt.rn.f16x2.e4m3x2 h2, p;\n\tmov.b32 {h, _}, h2;\n\tcvt.f32.f16 $0, h;\n\t}", + "=f,r", + loc=loc, + ip=ip, + ) + ) + return byte, value + + +@dsl_user_op +def e2m1x4_halfword(q0, q1, q2, q3, *, loc=None, ip=None): + """The low 16 bits of fp32x8_to_e2m1 for q[0..3] (q[4..7] = 0): byte0 = (q1, q0), byte1 = (q3, q2).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(q0), cutlass.Float32(q1), cutlass.Float32(q2), cutlass.Float32(q3)], + "{\n\t.reg .b8 b0, b1;\n\t.reg .b32 w;\n\tcvt.rn.satfinite.e2m1x2.f32 b0, $2, $1;\n\t" + "cvt.rn.satfinite.e2m1x2.f32 b1, $4, $3;\n\tmov.b32 w, {b0, b1, 0, 0};\n\tmov.b32 $0, w;\n\t}", + "=r,f,f,f,f", + loc=loc, + ip=ip, + ) + ) + + +# ---- global memory ----------------------------------------------------------------------------------------------- +@dsl_user_op +def ld_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm(_T.i64(), [cutlass.Int64(addr)], "ld.global.u64 $0, [$1];", "=l,l", True, loc, ip) + ) + + +@dsl_user_op +def ld_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm(_T.i32(), [cutlass.Int64(addr)], "ld.global.s32 $0, [$1];", "=r,l", True, loc, ip) + ) + + +@dsl_user_op +def ld_f32(addr, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Int64(addr)], "ld.global.f32 $0, [$1];", "=f,l", True, loc, ip) + ) + + +@dsl_user_op +def st_u32(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(v)], + "st.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def st_u16(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(v)], + "{\n\t.reg .b16 h;\n\tcvt.u16.u32 h, $1;\n\tst.global.u16 [$0], h;\n\t}", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def st_u8(addr, v, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(v)], + "{\n\t.reg .b16 h;\n\tcvt.u16.u32 h, $1;\n\tst.global.u8 [$0], h;\n\t}", + "l,r", + True, + loc, + ip, + ) + + +# ---- CONTRACT points (follow the shipped SASS; see the module docstring) ------------------------------------------- +def sumsq_pair(acc, x, y): + """sumSquares += x * x + y * y (C++: one pair of the packed row). CONTRACT: fma(y, y, x * x) then add.""" + return f_add(acc, f_fma(y, y, f_mul(x, x))) + + +def rms_reciprocal(sum_squares, eps): + """rsqrtf(sumSquares / 128 + eps). CONTRACT: the / 128 is an exact * 2^-7; fused with + eps into one fma.""" + return f_rsqrt(f_fma(sum_squares, cutlass.Float32(1.0 / HEAD_DIM), eps)) + + +class ProducerArgs: + """Raw device addresses (int64) + strides, as the C++ Params (producerPw.h) carries them.""" + + def __init__( + self, + packed, + q_out, + index_q_out, + index_k, + kv_data, + kv_scale, + out_cache_loc, + kv_quant_scale, + position_ids, + rotary_cos_sin, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + eps, + num_pages, + index_page_stride, + index_token_stride, + data_page_stride, + data_plane_stride, + data_token_stride, + scale_page_stride, + scale_plane_stride, + ): + self.__dict__.update(locals()) + del self.__dict__["self"] + + +@cute.jit +def phase_producer(tok, a: ProducerArgs): + """Producer work item ``tok``: warp w (< 12) of the calling CTA does packed head w of decode token tok. + The caller guarantees tok < rows and that warps >= 12 skip. Stores only (no CTA barrier inside).""" + tid = cute.arch.thread_idx()[0] + lane = tid % 32 + head = tid // 32 + in_off = (cutlass.Int64(tok) * PACKED_HEADS + cutlass.Int64(head)) * HEAD_DIM + cutlass.Int64( + lane * ELEMS + ) + pk = ld_u64(a.packed + in_off * 2) + x0 = bf16_bits_to_f32(cutlass.Int32(pk & cutlass.Int64(0xFFFF))) + x1 = bf16_bits_to_f32(cutlass.Int32((pk >> cutlass.Int64(16)) & cutlass.Int64(0xFFFF))) + x2 = bf16_bits_to_f32(cutlass.Int32((pk >> cutlass.Int64(32)) & cutlass.Int64(0xFFFF))) + x3 = bf16_bits_to_f32(cutlass.Int32((pk >> cutlass.Int64(48)) & cutlass.Int64(0xFFFF))) + producer_head(tok, head, lane, x0, x1, x2, x3, a) + + +@cute.jit +def producer_head(tok, head, lane, x0, x1, x2, x3, a: ProducerArgs): + """Everything after the packed-row load for packed head ``head`` (warp-uniform; dynamic Int32 or Constexpr) of + decode token ``tok``, on the calling warp: lane ``lane`` holds the head's elements [4 lane, 4 lane + 4) as the + float values x0..x3 of the BF16 packed row (the QKV GEMV output rounded to BF16). Norm weights, KV quant scale, + position and RoPE loads, the RMS tree, RoPE, and the q / index-q / index-K / K-V NVFP4 stores. Shared by + phase_producer and the QKV GEMV owner epilogue (MK-A), so the producer math has one implementation. + + Data flow is branch-free (every load is in bounds for every head: V reads the q norm weight, a non-rotating lane + reads its token's RoPE row, and the results are then discarded with select); only the stores branch, on + warp-uniform conditions, so every shuffle runs on the full warp as in the C++ kernel.""" + l4 = cutlass.Int64(lane * ELEMS) + is_v = head == V_HEAD + zero = cutlass.Float32(0.0) + + # Norm weights (BF16): q / k / index-q / index-k weight by head (V: q's, unused). + w_base = cutlass.select_( + head == K_HEAD, + a.k_weight, + cutlass.select_( + head == IQ_HEAD, + a.index_q_weight, + cutlass.select_(head == IK_HEAD, a.index_k_weight, a.q_weight), + ), + ) + ww = ld_u64(w_base + l4 * 2) + w0 = bf16_bits_to_f32(cutlass.Int32(ww & cutlass.Int64(0xFFFF))) + w1 = bf16_bits_to_f32(cutlass.Int32((ww >> cutlass.Int64(16)) & cutlass.Int64(0xFFFF))) + w2 = bf16_bits_to_f32(cutlass.Int32((ww >> cutlass.Int64(32)) & cutlass.Int64(0xFFFF))) + w3 = bf16_bits_to_f32(cutlass.Int32((ww >> cutlass.Int64(48)) & cutlass.Int64(0xFFFF))) + quant = ld_f32(a.kv_quant_scale + cutlass.select_(is_v, cutlass.Int64(8), cutlass.Int64(4))) + + # Position id, RoPE row. + pos = ld_s32(a.position_ids + cutlass.Int64(tok) * 4) + row = cutlass.Int64(pos) * ROTARY_DIM + coef = cutlass.Int64((lane * ELEMS) % (ROTARY_DIM // 2)) + c0 = ld_f32(a.rotary_cos_sin + (row + coef) * 4) + c1 = ld_f32(a.rotary_cos_sin + (row + coef + 1) * 4) + c2 = ld_f32(a.rotary_cos_sin + (row + coef + 2) * 4) + c3 = ld_f32(a.rotary_cos_sin + (row + coef + 3) * 4) + s0 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef) * 4) + s1 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef + 1) * 4) + s2 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef + 2) * 4) + s3 = ld_f32(a.rotary_cos_sin + (row + ROTARY_DIM // 2 + coef + 3) * 4) + + # RMSNorm (all heads compute it; V keeps x). + ss = sumsq_pair(zero, x0, x1) + ss = sumsq_pair(ss, x2, x3) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=16)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=8)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=4)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=2)) + ss = f_add(ss, cute.arch.shuffle_sync_bfly(ss, offset=1)) + r = rms_reciprocal(ss, a.eps) + one = cutlass.Float32(1.0) + n0 = f_mul(x0, f_mul(r, f_add(one, w0))) + n1 = f_mul(x1, f_mul(r, f_add(one, w1))) + n2 = f_mul(x2, f_mul(r, f_add(one, w2))) + n3 = f_mul(x3, f_mul(r, f_add(one, w3))) + # NeoX partial RoPE: lane l's dim 4l + i pairs with lane l ^ 8 (dim +-32) within the first 64 dims. + pair_off = (ROTARY_DIM // 2) // ELEMS + p0 = cute.arch.shuffle_sync_bfly(n0, offset=pair_off) + p1 = cute.arch.shuffle_sync_bfly(n1, offset=pair_off) + p2 = cute.arch.shuffle_sync_bfly(n2, offset=pair_off) + p3 = cute.arch.shuffle_sync_bfly(n3, offset=pair_off) + rot = lane * ELEMS < ROTARY_DIM + low = lane * ELEMS < ROTARY_DIM // 2 + p0 = cutlass.select_(low, f_neg(p0), p0) + p1 = cutlass.select_(low, f_neg(p1), p1) + p2 = cutlass.select_(low, f_neg(p2), p2) + p3 = cutlass.select_(low, f_neg(p3), p3) + r0 = cutlass.select_(rot, f_fma(p0, s0, f_mul(n0, c0)), n0) + r1 = cutlass.select_(rot, f_fma(p1, s1, f_mul(n1, c1)), n1) + r2 = cutlass.select_(rot, f_fma(p2, s2, f_mul(n2, c2)), n2) + r3 = cutlass.select_(rot, f_fma(p3, s3, f_mul(n3, c3)), n3) + e0 = cutlass.select_(is_v, x0, r0) + e1 = cutlass.select_(is_v, x1, r1) + e2 = cutlass.select_(is_v, x2, r2) + e3 = cutlass.select_(is_v, x3, r3) + + if head < HEADS_Q: + st_u32( + a.q_out + (cutlass.Int64(tok) * HEADS_Q + cutlass.Int64(head)) * HEAD_DIM + l4, + e4m3x4_word(e0, e1, e2, e3), + ) + if head == IQ_HEAD: + st_u32(a.index_q_out + cutlass.Int64(tok) * HEAD_DIM + l4, e4m3x4_word(e0, e1, e2, e3)) + if head >= K_HEAD: + if head != IQ_HEAD: + slot = ld_s32(a.out_cache_loc + cutlass.Int64(tok) * 4) + page = slot // PAGE + if slot >= 0: + if page < a.num_pages: + within = slot & (PAGE - 1) + if head == IK_HEAD: + st_u32( + a.index_k + + cutlass.Int64(page) * a.index_page_stride + + cutlass.Int64(within) * a.index_token_stride + + l4, + e4m3x4_word(e0, e1, e2, e3), + ) + else: + plane = cutlass.select_(is_v, cutlass.Int64(1), cutlass.Int64(0)) + col = lane // 4 + sw = cutlass.select_( + is_v, + cutlass.Int64((within // 4) * 32 + col * 4 + within % 4), + cutlass.Int64(within * 8 + col), + ) + s_off = ( + cutlass.Int64(page) * a.scale_page_stride + + plane * a.scale_plane_stride + + sw + ) + # quantize_fp4x4: BF16-rounded values, 16-element block max over the 4-lane group. + q0 = round_bf16(e0) + q1 = round_bf16(e1) + q2 = round_bf16(e2) + q3 = round_bf16(e3) + lmax = f_max( + f_max(f_max(f_max(zero, f_abs(q0)), f_abs(q1)), f_abs(q2)), f_abs(q3) + ) + lmax = f_max(lmax, cute.arch.shuffle_sync_bfly(lmax, offset=1)) + lmax = f_max(lmax, cute.arch.shuffle_sync_bfly(lmax, offset=2)) + sbyte, sval = e4m3_byte_and_value( + f_mul(quant, f_mul(lmax, f_rcp(cutlass.Float32(6.0)))) + ) + if lane % 4 == 0: + st_u8(a.kv_scale + s_off, sbyte) + outs = cutlass.select_(lmax != zero, f_rcp(f_mul(sval, f_rcp(quant))), zero) + hw = e2m1x4_halfword( + f_mul(q0, outs), f_mul(q1, outs), f_mul(q2, outs), f_mul(q3, outs) + ) + d_off = ( + cutlass.Int64(page) * a.data_page_stride + + plane * a.data_plane_stride + + cutlass.Int64(within) * a.data_token_stride + + cutlass.Int64(lane * 2) + ) + st_u16(a.kv_data + d_off, hw) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_locality.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_locality.py new file mode 100644 index 000000000000..3e88c7fb1ce2 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_locality.py @@ -0,0 +1,448 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Locality domains ("uGPU") of a multi-die GPU (Rubin VR200: 2 domains of 100 SMs + 12 remainder SMs) for the +streaming decode kernels of this target: the SM -> domain map, per-domain weight placement, and the CTA -> SM +assignment a persistent kernel uses to read only its own die's memory. + +A kernel launched with one CTA per SM reads ``%smid`` once and takes its logical block index from a host-built +permutation (:meth:`LocalityMap.logical_map`), so every per-CTA structure (work plan row, epochs, roles) follows its +die. Weights that are split by output rows into per-domain halves (:meth:`LocalityMap.split_rows`) live in one virtual +range whose halves are physically in the two domains. Placement only: the arithmetic is unchanged, so results are +bitwise those of the unplaced weights. + +The SM -> domain map is attested by the driver: the SMs are split by locality domain +(``cuDevSmResourceSplit`` with ``CU_DEV_SM_RESOURCE_GROUP_LOCALITY_DOMAIN_ID``), and a ``%smid`` probe runs on a green +context of each split. Allocations use driver VMM with ``CU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN`` and are checked +with ``CU_POINTER_ATTRIBUTE_LOCALITY_DOMAIN_ORDINAL``; any failure raises (fail closed).""" + +from __future__ import annotations + +import threading +from typing import Dict, List, Sequence, Tuple + +import torch + +_LOCK = threading.Lock() +_MAPS: Dict[int, "LocalityMap"] = {} +_PROBE_SRC = r""" +extern "C" __global__ void smid_probe(int* out) { + if (threadIdx.x == 0) { + unsigned s; + asm volatile("mov.u32 %0, %%smid;" : "=r"(s)); + out[blockIdx.x] = (int)s; + } +} +""" + + +def _drv(): + from cuda.bindings import driver + + return driver + + +def _ck(res, what: str): + """cuda-python returns (CUresult, *values): raise on an error, return the values.""" + err = res[0] if isinstance(res, tuple) else res + if int(err) != 0: + raise RuntimeError(f"locality: {what} failed: {err}") + if isinstance(res, tuple): + return res[1] if len(res) == 2 else res[1:] + return None + + +class _Mapping: + """One virtual range backed by per-domain physical allocations, exposed to torch through + ``__cuda_array_interface__``. Lives as long as a tensor that views it (the tensor holds this object).""" + + def __init__(self, device: int, pieces: Sequence[Tuple[int, int]]): + """``pieces``: (bytes, domain) in address order; each a multiple of the allocation granularity.""" + d = _drv() + self.device = device + self.size = sum(n for n, _ in pieces) + self.va = _ck(d.cuMemAddressReserve(self.size, 0, 0, 0), "cuMemAddressReserve") + self.handles = [] + off = 0 + try: + for nbytes, dom in pieces: + prop = _alloc_prop(device, dom) + h = _ck(d.cuMemCreate(nbytes, prop, 0), f"cuMemCreate(domain {dom})") + self.handles.append((h, nbytes)) + _ck(d.cuMemMap(int(self.va) + off, nbytes, 0, h, 0), "cuMemMap") + off += nbytes + acc = d.CUmemAccessDesc() + acc.location.type = d.CUmemLocationType.CU_MEM_LOCATION_TYPE_DEVICE + acc.location.id = device + acc.flags = d.CUmemAccess_flags.CU_MEM_ACCESS_FLAGS_PROT_READWRITE + _ck(d.cuMemSetAccess(self.va, self.size, [acc], 1), "cuMemSetAccess") + except Exception: + self.close() + raise + self.__cuda_array_interface__ = { + "shape": (self.size,), + "typestr": "|u1", + "data": (int(self.va), False), + "version": 3, + "strides": None, + "stream": None, + } + + def close(self): + d = _drv() + off = 0 + for h, n in self.handles: + d.cuMemUnmap(int(self.va) + off, n) + d.cuMemRelease(h) + off += n + self.handles = [] + if self.va is not None: + d.cuMemAddressFree(self.va, self.size) + self.va = None + + def __del__(self): + try: + self.close() + except Exception: + pass + + +_LIBCUDA = None + + +def _libcuda(): + global _LIBCUDA + if _LIBCUDA is None: + import ctypes + + _LIBCUDA = ctypes.CDLL("libcuda.so.1") + return _LIBCUDA + + +def _alloc_prop(device: int, domain: int): + d = _drv() + prop = d.CUmemAllocationProp() + prop.type = d.CUmemAllocationType.CU_MEM_ALLOCATION_TYPE_PINNED + prop.requestedHandleTypes = d.CUmemAllocationHandleType.CU_MEM_HANDLE_TYPE_NONE + prop.location.type = d.CUmemLocationType.CU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN + prop.location.localized.deviceId = device + prop.location.localized.localityDomainId = domain + return prop + + +class LocalityMap: + """The locality domains of one device. Build with :meth:`get` before CUDA-graph capture.""" + + def __init__(self, device: int): + d = _drv() + _ck(d.cuInit(0), "cuInit") + dev = _ck(d.cuDeviceGet(device), "cuDeviceGet") + attr = d.CUdevice_attribute + self.device = device + self.n_domains = int( + _ck( + d.cuDeviceGetAttribute(attr.CU_DEVICE_ATTRIBUTE_LOCALITY_DOMAIN_COUNT, dev), + "domain count", + ) + ) + self.sms_per_domain = int( + _ck( + d.cuDeviceGetAttribute( + attr.CU_DEVICE_ATTRIBUTE_LOCALITY_DOMAIN_MULTIPROCESSOR_COUNT, dev + ), + "SMs per domain", + ) + ) + self.num_sms = int( + _ck(d.cuDeviceGetAttribute(attr.CU_DEVICE_ATTRIBUTE_MULTIPROCESSOR_COUNT, dev), "SMs") + ) + if self.n_domains != 2: + raise RuntimeError( + f"locality: {self.n_domains} locality domains (this layer handles 2)" + ) + self._domain_sms = self._probe(d, dev) + dom = [-1] * self.num_sms + for k, sms in enumerate(self._domain_sms): + for s in sms: + if dom[s] != -1: + raise RuntimeError(f"locality: SM {s} probed in two domains") + dom[s] = k + self.sm_domain: List[int] = dom + self.remainder: List[int] = [s for s in range(self.num_sms) if dom[s] < 0] + rt = _alloc_prop(device, 0) + self.granularity = int( + _ck( + d.cuMemGetAllocationGranularity( + rt, d.CUmemAllocationGranularity_flags.CU_MEM_ALLOC_GRANULARITY_MINIMUM + ), + "granularity", + ) + ) + + @classmethod + def get(cls, device) -> "LocalityMap": + idx = torch.device(device).index if not isinstance(device, int) else device + idx = torch.cuda.current_device() if idx is None else idx + with _LOCK: + m = _MAPS.get(idx) + if m is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "LocalityMap probes the device: build it before CUDA-graph capture" + ) + m = _MAPS[idx] = cls(idx) + return m + + def sms(self, domain: int) -> List[int]: + return list(self._domain_sms[domain]) + + @property + def sm_table(self) -> torch.Tensor: + """int32 [num_sms] on the device: (domain << 16) | rank within the domain's sorted SMs; -1 = remainder SM. + For kernels that pick work by %smid themselves.""" + t = getattr(self, "_sm_table", None) + if t is None: + v = [-1] * self.num_sms + for k, sms in enumerate(self._domain_sms): + for i, s in enumerate(sms): + v[s] = (k << 16) | i + t = self._sm_table = torch.tensor(v, dtype=torch.int32, device=f"cuda:{self.device}") + return t + + def _probe(self, d, dev) -> List[List[int]]: + """Split the SMs by locality domain (strict, no backfill) and record %smid of a probe kernel launched on a + green context of each part: the driver's own SM -> domain assignment.""" + from cuda.bindings import nvrtc + + full = _ck( + d.cuDeviceGetDevResource(dev, d.CUdevResourceType.CU_DEV_RESOURCE_TYPE_SM), + "dev resource", + ) + params = [] + for k in range(self.n_domains): + p = d.CU_DEV_SM_RESOURCE_GROUP_PARAMS() + p.smCount = self.sms_per_domain + p.flags = d.CUdevSmResourceGroup_flags.CU_DEV_SM_RESOURCE_GROUP_LOCALITY_DOMAIN_ID + p.localityDomainId = k + params.append(p) + res, _rem = _ck( + d.cuDevSmResourceSplit(self.n_domains, full, 0, params), "cuDevSmResourceSplit" + ) + # the probe module, compiled for this device + major = _ck( + d.cuDeviceGetAttribute( + d.CUdevice_attribute.CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR, dev + ), + "cc", + ) + minor = _ck( + d.cuDeviceGetAttribute( + d.CUdevice_attribute.CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR, dev + ), + "cc", + ) + prog = _ck( + nvrtc.nvrtcCreateProgram(_PROBE_SRC.encode(), b"smid_probe.cu", 0, [], []), + "nvrtcCreateProgram", + ) + opts = [f"--gpu-architecture=sm_{major}{minor}".encode()] + err = nvrtc.nvrtcCompileProgram(prog, len(opts), opts)[0] + if int(err) != 0: + raise RuntimeError(f"locality: probe compile failed: {err}") + n = _ck(nvrtc.nvrtcGetCUBINSize(prog), "cubin size") + cubin = b" " * n + _ck(nvrtc.nvrtcGetCUBIN(prog, cubin), "cubin") + out: List[List[int]] = [] + nb = 4 * self.sms_per_domain # more blocks than SMs: every SM of the part shows up + torch.cuda.set_device(self.device) + buf = torch.full((nb,), -1, dtype=torch.int32, device=f"cuda:{self.device}") + for k in range(self.n_domains): + desc = _ck(d.cuDevResourceGenerateDesc([res[k]], 1), "resource desc") + g = _ck( + d.cuGreenCtxCreate(desc, dev, d.CUgreenCtxCreate_flags.CU_GREEN_CTX_DEFAULT_STREAM), + "green ctx", + ) + st = _ck( + d.cuGreenCtxStreamCreate(g, d.CUstream_flags.CU_STREAM_NON_BLOCKING, 0), + "green stream", + ) + ctx = _ck(d.cuCtxFromGreenCtx(g), "ctx from green ctx") + _ck(d.cuCtxPushCurrent(ctx), "push ctx") + try: + mod = _ck(d.cuModuleLoadData(cubin), "probe module") + fn = _ck(d.cuModuleGetFunction(mod, b"smid_probe"), "probe fn") + buf.fill_(-1) + torch.cuda.synchronize(self.device) + import ctypes + + arg = ctypes.c_void_p(buf.data_ptr()) + _ck( + d.cuLaunchKernel( + fn, nb, 1, 1, 32, 1, 1, 0, st, ((arg,), (ctypes.c_void_p,)), 0 + ), + "probe launch", + ) + _ck(d.cuStreamSynchronize(st), "probe sync") + sms = sorted(set(int(x) for x in buf.tolist() if x >= 0)) + d.cuModuleUnload(mod) + finally: + d.cuCtxPopCurrent() + d.cuStreamDestroy(st) + d.cuGreenCtxDestroy(g) + if len(sms) != self.sms_per_domain: + raise RuntimeError( + f"locality: domain {k} probed {len(sms)} SMs, expected {self.sms_per_domain}" + ) + out.append(sms) + return out + + def logical_map(self, per_domain: int, extra: int) -> Tuple[torch.Tensor, List[int]]: + """The SM -> logical block index permutation for a kernel of ``2 * per_domain + extra`` CTAs, one per SM: + logical blocks [d * per_domain, (d + 1) * per_domain) run on domain d's first ``per_domain`` SMs; logical + blocks 2 * per_domain .. run on the rest (the leftover domain SMs, alternating domains, then the remainder + SMs). Returns (int32 [num_sms] on the device, -1 = no block; the domain of each extra block, -1 remainder).""" + if 2 * per_domain + extra != self.num_sms: + raise ValueError( + f"locality: {2 * per_domain} + {extra} CTAs must cover the {self.num_sms} SMs" + ) + if per_domain > self.sms_per_domain: + raise ValueError(f"locality: {per_domain} CTAs per domain > {self.sms_per_domain} SMs") + lmap = [-1] * self.num_sms + for k in range(2): + for i, s in enumerate(self._domain_sms[k][:per_domain]): + lmap[s] = k * per_domain + i + left = [self._domain_sms[k][per_domain:] for k in range(2)] + rest = [x for pair in zip(*left) for x in pair] + self.remainder + ext_dom = [] + for i, s in enumerate(rest): + lmap[s] = 2 * per_domain + i + ext_dom.append(self.sm_domain[s]) + assert sorted(lmap) == list(range(self.num_sms)) + return torch.tensor(lmap, dtype=torch.int32, device=f"cuda:{self.device}"), ext_dom + + def domain_streams(self) -> List[torch.cuda.ExternalStream]: + """One non-blocking stream per domain on a green context holding exactly that domain's SMs (kept alive with + this map): work launched on stream d runs only on domain d's SMs. Fork / join with events from the caller's + stream; graph capture follows the forks. Built once, before capture.""" + st = getattr(self, "_streams", None) + if st is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("locality.domain_streams: build before CUDA-graph capture") + d = _drv() + dev = _ck(d.cuDeviceGet(self.device), "cuDeviceGet") + full = _ck( + d.cuDeviceGetDevResource(dev, d.CUdevResourceType.CU_DEV_RESOURCE_TYPE_SM), + "dev resource", + ) + params = [] + for k in range(self.n_domains): + p = d.CU_DEV_SM_RESOURCE_GROUP_PARAMS() + p.smCount = self.sms_per_domain + p.flags = d.CUdevSmResourceGroup_flags.CU_DEV_SM_RESOURCE_GROUP_LOCALITY_DOMAIN_ID + p.localityDomainId = k + params.append(p) + res, _rem = _ck( + d.cuDevSmResourceSplit(self.n_domains, full, 0, params), "cuDevSmResourceSplit" + ) + self._green = [] + st = [] + for k in range(self.n_domains): + desc = _ck(d.cuDevResourceGenerateDesc([res[k]], 1), "resource desc") + g = _ck( + d.cuGreenCtxCreate( + desc, dev, d.CUgreenCtxCreate_flags.CU_GREEN_CTX_DEFAULT_STREAM + ), + "green", + ) + s_ = _ck( + d.cuGreenCtxStreamCreate(g, d.CUstream_flags.CU_STREAM_NON_BLOCKING, 0), + "green stream", + ) + self._green.append((g, s_)) + st.append(torch.cuda.ExternalStream(int(s_), device=f"cuda:{self.device}")) + self._streams = st + return st + + def domain_of(self, ptr: int) -> int: + """The locality domain ordinal of a device address (-1: not placed in one domain). Through libcuda + directly: cuda-python's attribute helper does not know this attribute.""" + import ctypes + + d = _drv() + lib = _libcuda() + out = ctypes.c_int(-9) + attr = int(d.CUpointer_attribute.CU_POINTER_ATTRIBUTE_LOCALITY_DOMAIN_ORDINAL) + err = lib.cuPointerGetAttribute(ctypes.byref(out), ctypes.c_int(attr), ctypes.c_uint64(ptr)) + if err != 0: + raise RuntimeError( + f"locality: cuPointerGetAttribute(LOCALITY_DOMAIN_ORDINAL) failed: {err}" + ) + return int(out.value) + + def place_split(self, t: torch.Tensor, split_row: int) -> torch.Tensor: + """Split-mapped placement: a copy of ``t`` (contiguous, rows along dim 0) with the SAME shape and layout, in one + virtual range whose rows [0, split_row) are physically in domain 0 and rows [split_row, N) in domain 1. The + split must fall on an allocation granule (rows [0, split_row) a multiple of 2 MiB on VR200); the tail is padded + to a granule in domain 1. Stock consumers read it as a plain tensor; no extra memory once ``t`` is dropped.""" + if not t.is_contiguous(): + raise ValueError("locality.place_split: needs a contiguous tensor") + row_bytes = t[0].numel() * t.element_size() if t.dim() > 1 else t.element_size() + head = split_row * row_bytes + total = t.numel() * t.element_size() + if not 0 < head < total or head % self.granularity: + raise ValueError( + f"locality.place_split: rows [0, {split_row}) = {head} B must be a granule multiple " + f"({self.granularity} B) inside the {total} B tensor" + ) + tail = -(-(total - head) // self.granularity) * self.granularity + m = _Mapping(self.device, [(head, 0), (tail, 1)]) + flat = torch.as_tensor(m, device=f"cuda:{self.device}") + out = flat[:total].view(t.dtype).view(t.shape) + out.copy_(t) + out._locality_mapping = m + for q, k in ( + (out.data_ptr(), 0), + (out.data_ptr() + head - 1, 0), + (out.data_ptr() + head, 1), + (out.data_ptr() + total - 1, 1), + ): + if self.domain_of(q) != k: + raise RuntimeError(f"locality.place_split: byte {q:#x} is not in domain {k}") + return out + + def replace_param_(self, param: torch.nn.Parameter, split_row: int) -> torch.nn.Parameter: + """Re-place a parameter in place with :meth:`place_split` (swaps ``param.data``; the old block returns to the + caching allocator, so call ``torch.cuda.empty_cache()`` between large re-placements).""" + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("locality.replace_param_: not during CUDA-graph capture") + param.data = self.place_split(param.data, split_row) + return param + + def split_rows(self, t: torch.Tensor, dim: int = 1) -> torch.Tensor: + """Place ``t`` (contiguous, 1-byte elements, rows along ``dim`` split into two equal halves) as + [2, *t.shape[:dim], t.shape[dim] // 2, *t.shape[dim + 1:]]: half k (rows [k h, (k + 1) h) of every leading + index) in domain k's memory, one virtual range. Each half's byte size must be a multiple of the allocation + granularity (2 MiB on VR200). The copy is exact; the caller drops ``t`` afterwards.""" + if t.element_size() != 1 or not t.is_contiguous(): + raise ValueError("locality.split_rows: needs a contiguous 1-byte tensor") + n = t.shape[dim] + if n % 2: + raise ValueError(f"locality.split_rows: {n} rows don't split in halves") + half_bytes = t.numel() // 2 + if half_bytes % self.granularity: + raise ValueError( + f"locality.split_rows: a half is {half_bytes} B, not a multiple of the {self.granularity} B granularity" + ) + m = _Mapping(self.device, [(half_bytes, 0), (half_bytes, 1)]) + flat = torch.as_tensor(m, device=f"cuda:{self.device}") + shape = (2,) + tuple(t.shape[:dim]) + (n // 2,) + tuple(t.shape[dim + 1 :]) + out = flat.view(torch.uint8).view(shape) + src = t.view(torch.uint8) + for k in range(2): + out[k].copy_(src.narrow(dim, k * (n // 2), n // 2)) + out._locality_mapping = m # the tensor keeps its mapping alive + for k in range(2): + p = out[k].data_ptr() + for q in (p, p + half_bytes - 1): + if self.domain_of(q) != k: + raise RuntimeError(f"locality.split_rows: byte {q:#x} is not in domain {k}") + return out diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_lowm_splitk.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_lowm_splitk.py new file mode 100644 index 000000000000..0c32d635f131 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_lowm_splitk.py @@ -0,0 +1,1602 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-License-Identifier: Apache-2.0 +"""Blackwell low-M BF16/FP16 GEMM with an in-kernel cluster split-K reduction. + +This target's copy of FlashInfer's ``flashinfer/gemm/kernels/dense_bf16_gemm_sm100_splitk.py`` (the Eagle3 +drafter's low-M linears, modeling's ``DRAFT_LOWM_OWN_SPLITK``): the same kernel and the same split order at every +tactic, bit for bit FlashInfer's output (spec/test_modeling_v2_m3_lowm_splitk.py). The one change: the owner rank's +final cast and store is one method, ``_store_tile``, the seam for the drafter-boundary push epilogue and, on the A side, +the SwiGLU prologue that follow. + +Each cluster rank accumulates an exact K slice in FP32. Peers publish partials +to rank 0 through DSMEM; rank 0 reduces, casts, and stores once. The public +``A[M, K] @ B[K, N]`` problem is swapped internally, so tile dimensions below +use kernel coordinates: kernel-M carries public N and kernel-N carries public M. +""" + +from __future__ import annotations + +import dataclasses +from pathlib import Path + +import cuda.bindings.driver as _cuda +import cutlass +import cutlass.cute as cute +import cutlass.utils as utils +import cutlass.utils.blackwell_helpers as sm100_utils +from cutlass import Int32 +from cutlass._mlir.dialects import llvm +from cutlass.cute import experimental as cute_ext +from cutlass.cute.nvgpu import tcgen05 +from cutlass.cute.runtime import from_dlpack +from cutlass.cutlass_dsl import T, dsl_user_op + +from tensorrt_llm._torch import cute_aot + +#: Per-CTA SMEM capacity reported by CuTeDSL on SM100/SM103. +_SMEM_CAPACITY_BYTES = utils.get_smem_capacity_in_bytes("sm_100") + +#: K extent of one CTA tile. +_CTA_K = 128 # capacity: fixed: a kernel tile constant + +#: Kernel-M tiles; 64 increases CTA count for low-M decode shapes. +_SUPPORTED_MMA_M = (64, 128) + +#: Kernel-N carries public M, which is limited to 32. +_SUPPORTED_MMA_N = (8, 16, 32) + +#: Physical cluster-K sizes; split 1 compiles out the DSMEM path. +_SUPPORTED_SPLIT_K = (1, 2, 3, 4) + +#: Largest public M this low-M policy serves. +_MAX_M = 32 # capacity: fixed: the copied FlashInfer kernel's row bound + +#: Bytes per FP32 partial exchanged through DSMEM. +_FP32_BYTES = 4 + +#: DSMEM mailbox base alignment, in bytes. +_MAILBOX_ALIGN_BYTES = 128 + +#: Size and alignment of one mbarrier, in bytes. +_MBARRIER_BYTES = 8 + +#: Size and alignment of the TMEM base pointer slot. +_TMEM_POINTER_BYTES = 4 + +#: Bytes per BF16/FP16 element. +_AB_ELEMENT_BYTES = 2 + +#: Alignment of the A/B shared-memory buffers. +_AB_BUFFER_ALIGN_BYTES = 1024 # capacity: fixed: an alignment + +#: A/B pipeline stage bounds. One stage is useful only for a single K tile. +_MIN_AB_STAGES = 1 # capacity: fixed: pipeline stages +_DEFAULT_MIN_AB_STAGES = 2 +_MAX_AB_STAGES = 12 # capacity: fixed: pipeline stages + + +@dataclasses.dataclass(frozen=True) +class SplitKTactic: + """One specialization; mma_m carries public N and mma_n carries public M.""" + + mma_m: int + mma_n: int + split_k: int + ab_stages: int + + +def _align_up(value: int, alignment: int) -> int: + return ((value + alignment - 1) // alignment) * alignment + + +def _smem_bytes( + tactic: SplitKTactic, + ab_stages: int, +) -> int: + """Mirror the device allocator's shared-memory layout.""" + cursor = ( + _align_up( + tactic.mma_m * _CTA_K * _AB_ELEMENT_BYTES * ab_stages, + _AB_BUFFER_ALIGN_BYTES, + ) + + tactic.mma_n * _CTA_K * _AB_ELEMENT_BYTES * ab_stages + ) + + cursor = _align_up(cursor, _MBARRIER_BYTES) + cursor += 2 * ab_stages * _MBARRIER_BYTES + cursor += 3 * _MBARRIER_BYTES + cursor = _align_up(cursor, _TMEM_POINTER_BYTES) + cursor += _TMEM_POINTER_BYTES + + if tactic.split_k == 1: + return cursor + + return ( + _align_up( + _align_up(cursor, _MAILBOX_ALIGN_BYTES) + + (tactic.split_k - 1) * tactic.mma_m * tactic.mma_n * _FP32_BYTES, + _MBARRIER_BYTES, + ) + + _MBARRIER_BYTES + ) + + +def _max_ab_stages_for( + tactic: SplitKTactic, + smem_capacity: int, +) -> int: + return next( + ( + stages + for stages in range(_MAX_AB_STAGES, -1, -1) + if _smem_bytes(tactic, stages) <= smem_capacity + ), + 0, + ) + + +def validate_tactic( + tactic: SplitKTactic, + m: int, + n: int, + k: int, + *, + smem_capacity: int = _SMEM_CAPACITY_BYTES, +) -> None: + """Reject a tactic that cannot serve ``(m, n, k)``.""" + if tactic.mma_m not in _SUPPORTED_MMA_M: + raise ValueError(f"unsupported mma_m={tactic.mma_m}") + if tactic.mma_n not in _SUPPORTED_MMA_N: + raise ValueError(f"unsupported mma_n={tactic.mma_n}") + if tactic.split_k not in _SUPPORTED_SPLIT_K: + raise ValueError(f"unsupported split_k={tactic.split_k}") + if not _MIN_AB_STAGES <= tactic.ab_stages <= _MAX_AB_STAGES: + raise ValueError( + f"ab_stages must be in [{_MIN_AB_STAGES}, {_MAX_AB_STAGES}], got {tactic.ab_stages}" + ) + if not 1 <= m <= _MAX_M: + raise ValueError(f"this low-M policy requires 1 <= M <= {_MAX_M}, got {m}") + if n <= 0: + raise ValueError(f"N must be positive, got {n}") + if k <= 0 or k % _CTA_K or (k // _CTA_K) % tactic.split_k: + raise ValueError( + f"K={k} with CTA_K={_CTA_K} does not divide evenly across split_k={tactic.split_k}" + ) + smem_bytes = _smem_bytes(tactic, tactic.ab_stages) + if smem_bytes > smem_capacity: + raise ValueError( + f"tactic {tactic} needs {smem_bytes} B of shared memory but only " + f"{smem_capacity} B are available; max ab_stages is " + f"{_max_ab_stages_for(tactic, smem_capacity)}" + ) + + +def autotune_tactics( + m: int, + n: int, + k: int, + *, + smem_capacity: int = _SMEM_CAPACITY_BYTES, +) -> list[SplitKTactic]: + """Return valid tactics in the shape-specific stage window.""" + tactics: list[SplitKTactic] = [] + for mma_m in _SUPPORTED_MMA_M: + for mma_n in _SUPPORTED_MMA_N: + for split_k in _SUPPORTED_SPLIT_K: + min_stages = _MIN_AB_STAGES if k == _CTA_K else _DEFAULT_MIN_AB_STAGES + base = SplitKTactic(mma_m, mma_n, split_k, min_stages) + try: + validate_tactic(base, m, n, k, smem_capacity=smem_capacity) + except ValueError: + continue + max_stages = _max_ab_stages_for(base, smem_capacity) + # Short K favors shallow pipelines; long K stays near the cap. + tactics.extend( + dataclasses.replace(base, ab_stages=ab_stages) + for ab_stages in ( + range(min_stages, min(max_stages, 6) + 1) + if k <= 4 * _CTA_K + else range( + min(max(5, max_stages - 2), max_stages), + max_stages + 1, + ) + ) + ) + return tactics + + +def default_tactic(m: int, n: int, k: int) -> SplitKTactic: + """Choose the default occupancy-oriented tactic.""" + if n <= 512: + mma_m = 64 + mma_n = 16 if n == 512 and m > 24 else 8 + requested_split = 4 + else: + mma_n = 8 if m <= 8 else 16 if m <= 16 else 32 + if n <= 3072: + mma_m = 128 if m <= 16 else 64 + requested_split = 4 if m <= 16 else 2 + elif n < 8192: + mma_m = 64 + requested_split = 2 + else: + mma_m = 128 if k <= 1024 and m <= 24 else 64 + requested_split = 1 + + if k <= 4 * _CTA_K: + requested_split = 1 + split_k = next( + split_k + for split_k in reversed(_SUPPORTED_SPLIT_K) + if split_k <= requested_split and (k // _CTA_K) % split_k == 0 + ) + tactic = SplitKTactic(mma_m, mma_n, split_k, _MIN_AB_STAGES) + max_stages = _max_ab_stages_for(tactic, _SMEM_CAPACITY_BYTES) + tactic = dataclasses.replace( + tactic, + ab_stages=( + _MIN_AB_STAGES + if k == _CTA_K + else _DEFAULT_MIN_AB_STAGES + if k <= 2 * _CTA_K and m > 8 + else min(max_stages, 6) + if k <= 4 * _CTA_K + else max_stages + ), + ) + validate_tactic(tactic, m, n, k) + return tactic + + +__all__ = [ + "SplitKTactic", + "autotune_tactics", + "default_tactic", + "run_splitk_dense", + "run_splitk_dense_swiglu", + "compile_swiglu_push", +] + + +@dsl_user_op +def _map_shared_rank( + smem_ptr: "cute.Pointer", + peer_cta_rank_in_cluster: "Int32", + *, + loc=None, + ip=None, +) -> "Int32": + """Map an SMEM pointer into a peer CTA's address space.""" + return Int32( + llvm.inline_asm( + T.i32(), + [ + smem_ptr.toint(loc=loc, ip=ip).ir_value(), + peer_cta_rank_in_cluster.ir_value(), + ], + "mapa.shared::cluster.u32 $0, $1, $2;", + "=r,r,r", + has_side_effects=False, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +@dsl_user_op +def _store_shared_remote_v4( + value0, + value1, + value2, + value3, + smem_ptr: "cute.Pointer", + mbar_ptr: "cute.Pointer", + peer_cta_rank_in_cluster: "Int32", + *, + loc=None, + ip=None, +) -> None: + """Publish four FP32 partials into a peer's SMEM, crediting 16 bytes.""" + llvm.inline_asm( + None, + [ + _map_shared_rank(smem_ptr, peer_cta_rank_in_cluster, loc=loc, ip=ip).ir_value(), + value0.bitcast(Int32).ir_value(loc=loc, ip=ip), + value1.bitcast(Int32).ir_value(loc=loc, ip=ip), + value2.bitcast(Int32).ir_value(loc=loc, ip=ip), + value3.bitcast(Int32).ir_value(loc=loc, ip=ip), + _map_shared_rank(mbar_ptr, peer_cta_rank_in_cluster, loc=loc, ip=ip).ir_value(), + ], + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 " + "[$0], {$1, $2, $3, $4}, [$5];", + "r,r,r,r,r,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + + +@dsl_user_op +def _silu_mul_f32(g, u, *, loc=None, ip=None): + """silu(g) * (u + 0) in FP32 for one gate / up pair: the instruction sequence of the Triton silu_and_mul kernel at + its defaults (no limit, alpha 1, beta 0), so the bf16 rounding after it is bit for bit the Triton activation.""" + return cutlass.Float32( + llvm.inline_asm( + T.f32(), + [ + cutlass.Float32(g).ir_value(loc=loc, ip=ip), + cutlass.Float32(u).ir_value(loc=loc, ip=ip), + ], + "{\n\t.reg .f32 t, e, s, p, q, one, zero;\n\t" + "mul.rn.f32 t, $1, 0fBFB8AA3B;\n\tex2.approx.f32 e, t;\n\t" + "mov.f32 one, 0f3F800000;\n\tadd.rn.f32 e, e, one;\n\tdiv.full.f32 s, one, e;\n\t" + "mul.rn.f32 p, $1, s;\n\tmov.f32 zero, 0f00000000;\n\tadd.rn.f32 q, $2, zero;\n\t" + "mul.rn.f32 $0, q, p;\n\t}", + "=f,f,f", + has_side_effects=False, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +@dsl_user_op +def _ld_bf16_f32(addr, *, loc=None, ip=None): + """The bf16 at global byte address ``addr``, widened to FP32 (exact).""" + return cutlass.Float32( + llvm.inline_asm( + T.f32(), + [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b16 h;\n\tld.global.b16 h, [$1];\n\tcvt.f32.bf16 $0, h;\n\t}", + "=f,l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +@dsl_user_op +def _st_f32_bf16(addr, value, *, loc=None, ip=None) -> None: + """``value`` rounded to bf16 (round to nearest even) at global byte address ``addr``.""" + llvm.inline_asm( + None, + [ + cutlass.Int64(addr).ir_value(loc=loc, ip=ip), + cutlass.Float32(value).ir_value(loc=loc, ip=ip), + ], + "{\n\t.reg .b16 h;\n\tcvt.rn.bf16.f32 h, $1;\n\tst.global.b16 [$0], h;\n\t}", + "l,f", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + + +@dsl_user_op +def _fence_proxy_async_global(*, loc=None, ip=None) -> None: + """Order this thread's generic global stores before later async-proxy (TMA) reads of them.""" + llvm.inline_asm( + None, + [], + "fence.proxy.async.global;", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + + +#: Rank that gathers partials and stores the output. +OWNER_RANK = 0 # capacity: fixed: a split index, not a size + + +# The drafter-boundary push (``ar_push``, modeling's DRAFT_AR_PUSH): the exchange words of +# kernels/m3_mc_allreduce, as m3_mx_gemv's ``ar`` push stores them. +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return Int32( + llvm.inline_asm( + T.i32(), + [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], + "ld.volatile.global.s32 $0, [$1];", + "=r,l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +@dsl_user_op +def _st_global_b32(addr, value, *, loc=None, ip=None): + llvm.inline_asm( + None, + [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), Int32(value).ir_value(loc=loc, ip=ip)], + "st.global.b32 [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return Int32( + llvm.inline_asm( + T.i32(), + [ + cutlass.Float32(hi).ir_value(loc=loc, ip=ip), + cutlass.Float32(lo).ir_value(loc=loc, ip=ip), + ], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", + "=r,f,f", + has_side_effects=False, + is_align_stack=False, + asm_dialect=llvm.AsmDialect.AD_ATT, + ) + ) + + +class SplitKDenseGemmKernel: + """Standalone BF16/FP16 GEMM with a cluster-local split-K reduction.""" + + def __init__( + self, + *, + tactic: SplitKTactic, + use_pdl: bool, + has_bias: bool, + b_swiglu: bool = False, + ar_push: bool = False, + ) -> None: + self.ar_push = ar_push + self.acc_dtype = cutlass.Float32 + # The B operand (public A, the activation) is silu(gate) * up of a [M, 2K] gate | up tensor: the B DMA warp + # writes its CTA's K-slice of it to the B tensor (a scratch) before its unchanged TMA loop reads it back. + self.b_swiglu = b_swiglu + self.cta_m = tactic.mma_m + self.cta_n = tactic.mma_n + self.cta_k = _CTA_K + self.num_ab_stage = tactic.ab_stages + self.split_k = tactic.split_k + self.use_pdl = use_pdl + self.has_bias = has_bias + + self.threads_per_cta = 256 + self.epilog_threads = 128 + self.mma_tiler_mn = (tactic.mma_m, tactic.mma_n) + self.cta_group = tcgen05.CtaGroup.ONE + self.tma_op = cute_ext.OperationTypeEnum.SM90_TMA_LOAD + self.cluster_shape = (1, tactic.split_k, 1) + + values_per_thread = (tactic.mma_m * tactic.mma_n) // self.epilog_threads + if values_per_thread % 4: + raise ValueError( + f"CTA tile ({tactic.mma_m}, {tactic.mma_n}) gives " + f"{values_per_thread} " + "values per epilogue thread; remote stores require a multiple of 4" + ) + self.mailbox_elements = (tactic.split_k - 1) * self.epilog_threads * values_per_thread + self.expected_transaction_bytes = self.mailbox_elements * _FP32_BYTES + + @cute.experimental.jit + def __call__( + self, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + bias: cute.Tensor, + g: cute.Tensor, + u: cute.Tensor, + gu_ld: cutlass.Int32, + b_ld: cutlass.Int32, + stream: _cuda.CUstream, + ar_ws: cutlass.Int64 = cutlass.Int64(0), + ar_flags: cutlass.Int64 = cutlass.Int64(0), + ar_rank: Int32 = Int32(0), + ar_world: Int32 = Int32(1), + ar_maxt: Int32 = Int32(8), + ar_tokens: Int32 = Int32(0), + ar_n: Int32 = Int32(0), + ): + # Grid-y packs output-N tile and cluster rank. + self.kernel( + a, + b, + c, + bias, + g, + u, + gu_ld, + b_ld, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ).launch( + grid=( + cute.ceil_div(c.layout.shape[0], self.cta_m), + cute.ceil_div(c.layout.shape[1], self.cta_n) * self.split_k, + c.layout.shape[2], + ), + block=(self.threads_per_cta, 1, 1), + cluster=self.cluster_shape, + smem=cute.Int64(utils.get_smem_capacity_in_bytes("sm_100")), + stream=stream, + use_pdl=self.use_pdl, + ) + + @cute.experimental.kernel + def kernel( + self, + mA: cute.Tensor, # (Gemm_M, Gemm_K, Gemm_L), K-major + mB: cute.Tensor, # (Gemm_N, Gemm_K, Gemm_L), K-major + mC: cute.Tensor, # (Gemm_M, Gemm_N, Gemm_L), M-major + mBias: cute.Tensor, # Broadcast bias; dead when has_bias=False + mG: cute.Tensor, # b_swiglu: the gate half (Gemm_N, Gemm_K, Gemm_L); dead otherwise + mU: cute.Tensor, # b_swiglu: the up half, same layout; dead otherwise + gu_ld: cutlass.Int32, # b_swiglu: elements between gate (and up) rows + b_ld: cutlass.Int32, # b_swiglu: elements between B rows + ar_ws: cutlass.Int64, # ar_push: the boundary all-reduce's multicast exchange words (int32) + ar_flags: cutlass.Int64, # ar_push: its per-(token, cluster CTA) call flags (parity = flag & 1) + ar_rank: Int32, + ar_world: Int32, + ar_maxt: Int32, # the bucket's tokens + ar_tokens: Int32, # rows pushed (public M) + ar_n: Int32, # public N (the hidden size): the exchange row width + ): + """Allocate storage and dispatch the specialized warps.""" + stages = self.num_ab_stage + + ab_dtype = mA.element_type + tiled_mma = sm100_utils.make_trivial_tiled_mma( + ab_dtype, + ab_dtype, + utils.LayoutEnum.from_tensor(mA).mma_major_mode(), + utils.LayoutEnum.from_tensor(mB).mma_major_mode(), + self.acc_dtype, + self.cta_group, + self.mma_tiler_mn, + ) + + mnk_tiler = (self.mma_tiler_mn[0], self.mma_tiler_mn[1], self.cta_k) + block_idx = cute.arch.block_idx() + bidx = block_idx[0] + split_rank = cute.arch.make_warp_uniform(cute.arch.block_idx_in_cluster()) + n_idx = block_idx[1] // self.split_k + l_idx = block_idx[2] + warp_idx = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + + sA = cute_ext.allocate( + ab_dtype, + cute.AddressSpace.smem, + sm100_utils.make_smem_layout_a(tiled_mma, mnk_tiler, ab_dtype, stages), + alignment=_AB_BUFFER_ALIGN_BYTES, + ) + sB = cute_ext.allocate( + ab_dtype, + cute.AddressSpace.smem, + sm100_utils.make_smem_layout_b(tiled_mma, mnk_tiler, ab_dtype, stages), + alignment=_AB_BUFFER_ALIGN_BYTES, + ) + + acc_layout = cute_ext.make_tmem_layout_acc(tiled_mma, self.mma_tiler_mn, acc_stage=1) + c_tiler_mn = (self.cta_m, self.cta_n) + + bar_full = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(stages), + alignment=_MBARRIER_BYTES, + ).iterator + bar_empty = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(stages), + alignment=_MBARRIER_BYTES, + ).iterator + bar_tma_epilog = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + bar_mma_epilog = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + bar_tmem_alloc = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + tmem_base_ptr = cute_ext.allocate( + cutlass.Int32, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_TMEM_POINTER_BYTES, + ).iterator + # ar_push: the owner's BF16 tile (kernel-M = output column, kernel-N = token), M-major. + sD = cute_ext.allocate( + mC.element_type, + cute.AddressSpace.smem, + cute.make_layout((self.cta_m, self.cta_n), stride=(1, self.cta_m)), + alignment=16, + ) + + if cutlass.const_expr(self.split_k > 1): + mailbox = cute_ext.allocate( + cutlass.Float32, + cute.AddressSpace.smem, + cute.make_layout(self.mailbox_elements), + alignment=_MAILBOX_ALIGN_BYTES, + ) + bar_reduce = cute_ext.allocate( + cutlass.Int64, + cute.AddressSpace.smem, + cute.make_layout(1), + alignment=_MBARRIER_BYTES, + ).iterator + else: + # Dummy operands for the compile-time-elided reduction. + mailbox = sA + bar_reduce = bar_mma_epilog + + if warp_idx == 0: + with cute.arch.elect_one(): + for i in range(stages): + cute.arch.mbarrier_init(bar_full + i, 2) + cute.arch.mbarrier_init(bar_empty + i, 1) + cute.arch.mbarrier_init(bar_tma_epilog, 32) + cute.arch.mbarrier_init(bar_mma_epilog, 1) + cute.arch.mbarrier_init(bar_tmem_alloc, 160) + + if cutlass.const_expr(self.split_k > 1): + # Owner arrival plus peer transaction-byte credits. + cute.arch.mbarrier_init(bar_reduce, 1) + + cute.arch.mbarrier_init_fence() + if cutlass.const_expr(self.split_k > 1): + # Publish peer barriers before cross-CTA stores. + cute.arch.cluster_arrive_relaxed() + else: + cute.arch.barrier() + + # Host validation guarantees an equal, tail-free K partition. + k_tile_count = cute.size(mA, mode=[1]) // self.cta_k // self.split_k + k_tile_start = split_rank * k_tile_count + + if cutlass.const_expr(self.split_k > 1): + cute.arch.cluster_wait() + + # Warp 3 is idle; warps 4-7 run the epilogue. + if warp_idx == 0: + self.dma_warp( + bar_full, + bar_empty, + bar_tma_epilog, + cute.local_tile(mA, (self.cta_m, self.cta_k), (bidx, None, l_idx)), + sA, + cute_ext.get_cta_v_map_ab(mA, mnk_tiler, tiled_mma, "A"), + k_tile_start, + k_tile_count, + True, + cutlass.Int64(0), + cutlass.Int64(0), + cutlass.Int64(0), + Int32(0), + Int32(0), + Int32(0), + Int32(0), + ) + elif warp_idx == 1: + self.dma_warp( + bar_full, + bar_empty, + bar_tma_epilog, + cute.local_tile(mB, (self.cta_n, self.cta_k), (n_idx, None, l_idx)), + sB, + cute_ext.get_cta_v_map_ab(mB, mnk_tiler, tiled_mma, "B"), + k_tile_start, + k_tile_count, + False, + mG.iterator.toint(), + mU.iterator.toint(), + mB.iterator.toint(), + gu_ld, + b_ld, + n_idx * Int32(self.cta_n), + Int32(cute.size(mB, mode=[0])) - n_idx * Int32(self.cta_n), + ) + elif warp_idx == 2: + self.mma_warp( + bar_full, + bar_empty, + bar_mma_epilog, + bar_tmem_alloc, + tiled_mma, + sA, + sB, + tmem_base_ptr, + acc_layout, + self.cta_k // cute.size(tiled_mma.shape_mnk, mode=[2]), + k_tile_count, + ) + elif warp_idx >= 4: + self.epilog_warp( + bar_tma_epilog, + bar_mma_epilog, + bar_tmem_alloc, + tmem_base_ptr, + acc_layout, + cute.local_tile(mC, c_tiler_mn, (bidx, n_idx, l_idx)), + cute.local_tile(mBias, c_tiler_mn, (bidx, n_idx, l_idx)), + cute.arch.thread_idx()[0] - 128, + mC.element_type, + utils.LayoutEnum.from_tensor(mC), + mailbox, + bar_reduce, + split_rank, + sD, + bidx, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + @cute.experimental.jit + def _swiglu_to_b( + self, + gate_base: cutlass.Int64, + up_base: cutlass.Int64, + out_base: cutlass.Int64, + gu_ld: cutlass.Int32, + b_ld: cutlass.Int32, + row0: cutlass.Int32, + rows_valid: cutlass.Int32, + k_tile_start: cutlass.Int32, + k_tile_count: cutlass.Int32, + ): + """This CTA's K-slice of the activation, silu(gate) * up rounded to bf16, into the B tensor's valid rows (TMA + fills the rest with zeros), through raw global addresses (the B tensor is in the TMA chain); then the stores + are made visible to the TMA reads that follow.""" + lane = cute.arch.lane_idx() + per_tile = cutlass.const_expr(self.cta_n * self.cta_k) + for k_tile in cutlass.range(k_tile_count, unroll=1): + k0 = (k_tile_start + k_tile) * Int32(self.cta_k) + for j in cutlass.range_constexpr(per_tile // 32): + idx = Int32(j * 32) + lane + row = idx // Int32(self.cta_k) + col = k0 + idx % Int32(self.cta_k) + if row < rows_valid: + src = ( + cutlass.Int64(row0 + row) * cutlass.Int64(gu_ld) + cutlass.Int64(col) + ) * cutlass.Int64(2) + dst = ( + cutlass.Int64(row0 + row) * cutlass.Int64(b_ld) + cutlass.Int64(col) + ) * cutlass.Int64(2) + act = _silu_mul_f32(_ld_bf16_f32(gate_base + src), _ld_bf16_f32(up_base + src)) + _st_f32_bf16(out_base + dst, act) + _fence_proxy_async_global() + cute.arch.sync_warp() + + @cute.experimental.jit + def dma_warp( + self, + bar_full, + bar_empty, + bar_tma_epilog, + g_tile: cute.Tensor, + s_tile: cute.Tensor, + cta_v_map: cute.Layout, + k_tile_start: cutlass.Int32, + k_tile_count: cutlass.Int32, + is_a: cutlass.Constexpr, + gate_base: cutlass.Int64, + up_base: cutlass.Int64, + out_base: cutlass.Int64, + gu_ld: cutlass.Int32, + b_ld: cutlass.Int32, + row0: cutlass.Int32, + rows_valid: cutlass.Int32, + ): + stages = self.num_ab_stage + if cutlass.const_expr(not is_a and self.use_pdl): + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(not is_a and self.b_swiglu): + self._swiglu_to_b( + gate_base, + up_base, + out_base, + gu_ld, + b_ld, + row0, + rows_valid, + k_tile_start, + k_tile_count, + ) + + empty_phase = cutlass.Int32(1) + for k_tile in cutlass.range(k_tile_count, unroll=1): + stage = k_tile % stages + cute.arch.mbarrier_wait(bar_empty + stage, empty_phase) + with cute.arch.elect_one(): + cute.arch.mbarrier_arrive_and_expect_tx( + bar_full + stage, + cute.size_in_bytes( + s_tile.element_type, + cute.slice_(s_tile.layout, (None, None, None, 0)), + ), + ) + cute_ext.tma_load( + g_tile[None, None, k_tile_start + k_tile], + s_tile[None, None, None, stage], + (bar_full + stage).value, + cta_v_map=cta_v_map, + tma_operation_type=self.tma_op, + update_expect_tx=False, + ) + if stage == stages - 1: + empty_phase = empty_phase ^ 1 + + if cutlass.const_expr(is_a and self.use_pdl and not self.ar_push): + # ar_push: the dependent is a prepushed all-reduce, which skips its grid wait: it is released only after + # this grid's own grid wait and its pushes, in the owner's _store_tile (the prepushed >= 2 contract). + cute.arch.griddepcontrol_launch_dependents() + if cutlass.const_expr(not is_a and self.has_bias): + cute.arch.mbarrier_arrive(bar_tma_epilog) + self._drain_producer(bar_empty, empty_phase, k_tile_count) + + @cute.experimental.jit + def _drain_producer( + self, + bar_empty, + empty_phase: cutlass.Int32, + k_tile_count: cutlass.Int32, + ): + stages = self.num_ab_stage + for tail in cutlass.range(stages, unroll=1): + stage = (tail + k_tile_count) % stages + cute.arch.mbarrier_wait(bar_empty + stage, empty_phase) + if stage == stages - 1: + empty_phase = empty_phase ^ 1 + + @cute.experimental.jit + def mma_warp( + self, + bar_full, + bar_empty, + bar_mma_epilog, + bar_tmem_alloc, + tiled_mma: cute.TiledMma, + sA: cute.Tensor, + sB: cute.Tensor, + tmem_base_ptr, + acc_layout: cutlass.Constexpr, + mma_inst_tile_k: cutlass.Constexpr, + k_tile_count: cutlass.Int32, + ): + num_tmem_cols = 256 + cute.arch.alloc_tmem(num_tmem_cols, tmem_base_ptr, is_two_cta=False) + cute.arch.mbarrier_arrive(bar_tmem_alloc) + cute.arch.relinquish_tmem_alloc_permit(is_two_cta=False) + + tmem_ptr = cute.arch.retrieve_tmem_ptr(self.acc_dtype, 16, tmem_base_ptr) + accumulator = cute.make_tensor(tmem_ptr, acc_layout)[None, None, None, 0] + mma_atom = cute.make_mma_atom(tiled_mma.op) + full_phase = cutlass.Int32(0) + for k_tile in cutlass.range(k_tile_count, unroll=1): + stage = k_tile % self.num_ab_stage + cute.arch.mbarrier_wait(bar_full + stage, full_phase) + for k_block in range(mma_inst_tile_k): + if k_block == 0: + mma_atom.set(tcgen05.Field.ACCUMULATE, k_tile != 0) + else: + mma_atom.set(tcgen05.Field.ACCUMULATE, True) + cute_ext.dot( + mma_atom, + cute.append_ones(sA[None, None, k_block, stage], up_to_rank=3), + cute.append_ones(sB[None, None, k_block, stage], up_to_rank=3), + accumulator, + ) + with cute.arch.elect_one(): + tcgen05.commit(bar_empty + stage, None, self.cta_group) + if stage == self.num_ab_stage - 1: + full_phase = full_phase ^ 1 + + with cute.arch.elect_one(): + tcgen05.commit(bar_mma_epilog, None, self.cta_group) + cute.arch.mbarrier_arrive(bar_tmem_alloc) + cute.arch.mbarrier_wait(bar_tmem_alloc, 1) + cute.arch.dealloc_tmem(tmem_ptr, num_tmem_cols, is_two_cta=False) + + @cute.experimental.jit + def epilog_warp( + self, + bar_tma_epilog, + bar_mma_epilog, + bar_tmem_alloc, + tmem_base_ptr, + acc_layout: cutlass.Constexpr, + gD_tile: cute.Tensor, + gBias_tile: cute.Tensor, + epi_tid: cutlass.Int32, + c_dtype: cutlass.Constexpr, + d_layout: cutlass.Constexpr, + mailbox, + bar_reduce, + split_rank: cutlass.Int32, + sD: cute.Tensor, + bidx: cutlass.Int32, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_world: cutlass.Int32, + ar_maxt: cutlass.Int32, + ar_tokens: cutlass.Int32, + ar_n: cutlass.Int32, + ): + # Wait until MMA publishes the TMEM base pointer. + cute.arch.mbarrier_arrive(bar_tmem_alloc) + cute.arch.mbarrier_wait(bar_tmem_alloc, 0) + + acc_view = cute.make_tensor( + cute.arch.retrieve_tmem_ptr(self.acc_dtype, 16, tmem_base_ptr), + acc_layout, + )[((None, None), 0, 0, 0)] + + epi_tile = (self.cta_m, self.cta_n) + tiled_copy_t2r = cute.nvgpu.tcgen05.make_tmem_copy( + sm100_utils.get_tmem_load_op( + (self.cta_m, self.cta_n, self.cta_k), + d_layout, + c_dtype, + self.acc_dtype, + epi_tile, + False, + ), + acc_view, + ) + gD_epi = cute.flat_divide(gD_tile, epi_tile) + + # Match each epilogue thread's TMEM partition in RMEM. + rmem_layout = cute_ext.make_t2r_rmem_layout(tiled_copy_t2r, gD_epi, epi_tid) + rAcc = cute_ext.allocate( + self.acc_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + rD = cute_ext.allocate( + c_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + thr_t2r = tiled_copy_t2r.get_slice(epi_tid) + + if cutlass.const_expr(self.has_bias): + bias_dtype = gBias_tile.element_type + rBias = cute_ext.allocate( + bias_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + rBiasAcc = cute_ext.allocate( + self.acc_dtype, + cute.AddressSpace.rmem, + rmem_layout, + alignment=32, + ) + if split_rank == OWNER_RANK: + cute.arch.mbarrier_wait(bar_tma_epilog, 0) + cute_ext.partition_and_copy( + cute.make_tiled_copy_D( + cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), bias_dtype), + tiled_copy_t2r, + ).get_slice(epi_tid), + cute.flat_divide(gBias_tile, epi_tile)[None, None, 0, 0], + rBias, + ) + rBiasAcc.store(rBias.load().to(self.acc_dtype)) + + cute.arch.mbarrier_wait(bar_mma_epilog, 0) + cute_ext.partition_and_copy(thr_t2r, acc_view, rAcc) + # Make tcgen05.ld visible before TMEM release and RMEM use. + cute.arch.fence_view_async_tmem_load() + cute.arch.mbarrier_arrive(bar_tmem_alloc) + + # Peers publish FP32 partials; only rank 0 reduces and stores. + if cutlass.const_expr(self.split_k > 1): + assert cute.size(rmem_layout) == self.mailbox_elements // ( + (self.split_k - 1) * self.epilog_threads + ) + values_per_thread = cutlass.const_expr(cute.size(rmem_layout)) + values_per_peer = cutlass.const_expr(self.epilog_threads * values_per_thread) + if split_rank != OWNER_RANK: + for value_idx in cutlass.range_constexpr(0, values_per_thread, 4): + _store_shared_remote_v4( + rAcc[value_idx], + rAcc[value_idx + 1], + rAcc[value_idx + 2], + rAcc[value_idx + 3], + mailbox.iterator + + (split_rank - Int32(1)) * values_per_peer + + epi_tid * values_per_thread + + value_idx, + bar_reduce, + Int32(OWNER_RANK), + ) + else: + if epi_tid == 0: + cute.arch.mbarrier_arrive_and_expect_tx( + bar_reduce, self.expected_transaction_bytes + ) + cute.arch.mbarrier_wait(bar_reduce, 0) + for peer in cutlass.range_constexpr(self.split_k - 1): + for value_idx in cutlass.range_constexpr(values_per_thread): + rAcc[value_idx] = ( + rAcc[value_idx] + + mailbox[ + peer * values_per_peer + epi_tid * values_per_thread + value_idx + ] + ) + + if split_rank == OWNER_RANK: + if cutlass.const_expr(self.has_bias): + rAcc.store(rAcc.load() + rBiasAcc.load()) + self._store_tile(thr_t2r, rAcc, rD, gD_epi, c_dtype) + if cutlass.const_expr(self.ar_push): + self._push_tile( + thr_t2r, + rD, + sD, + epi_tid, + bidx, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + # The reduction mbarrier covers remote stores; no cluster barrier needed. + + @cute.experimental.jit + def _store_tile(self, thr_t2r, rAcc, rD, gD_epi, c_dtype: cutlass.Constexpr): + """The owner rank's reduced FP32 tile: one cast to ``c_dtype`` and the store to global memory.""" + rD.store(rAcc.load().to(c_dtype)) + # Preserve TMEM coordinates; the copy predicates output tails. + cute_ext.partition_and_copy(thr_t2r, rD, gD_epi[None, None, 0, 0]) + + @cute.experimental.jit + def _push_tile( + self, + thr_t2r, + rD, + sD, + epi_tid: cutlass.Int32, + bidx: cutlass.Int32, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_world: cutlass.Int32, + ar_maxt: cutlass.Int32, + ar_tokens: cutlass.Int32, + ar_n: cutlass.Int32, + ): + """ar_push: the owner's tile rows into the boundary all-reduce's exchange, as its own push of the output + would store them (kernels/m3_mc_allreduce: word (t, column pair) at the call parity of the (t, column / (N / 8)) + cluster CTA, -0.0 halves as +0.0), then the release of the dependents (the prepushed all-reduce).""" + cute_ext.partition_and_copy(thr_t2r, rD, sD) + cute.arch.barrier(barrier_id=1, number_of_threads=self.epilog_threads) + if cutlass.const_expr(self.use_pdl): + cute.arch.griddepcontrol_wait() + pairs = self.cta_m // 2 + col0 = bidx * self.cta_m + half = cutlass.Int64(ar_maxt) * cutlass.Int64(ar_world) * cutlass.Int64(ar_n // 2) + for w0 in cutlass.range_constexpr(0, pairs * self.cta_n, self.epilog_threads): + word = epi_tid + w0 + t = word // pairs + p = word % pairs + if t < ar_tokens: + n = col0 + 2 * p + par = _ld_volatile_s32( + ar_flags + cutlass.Int64(4) * cutlass.Int64(t * 8 + n // (ar_n // 8)) + ) & Int32(1) + off = ( + cutlass.Int64(par) * half + + cutlass.Int64(t * ar_world + ar_rank) * cutlass.Int64(ar_n // 2) + + cutlass.Int64(n // 2) + ) + _st_global_b32( + ar_ws + off * cutlass.Int64(4), + _bf16x2_word( + sD[2 * p, t].to(cutlass.Float32), sD[2 * p + 1, t].to(cutlass.Float32) + ), + ) + if cutlass.const_expr(self.use_pdl): + cute.arch.griddepcontrol_launch_dependents() + + +import torch as _torch # noqa: E402 (as in FlashInfer) + +_SUPPORTED_TORCH_DTYPES = (_torch.bfloat16, _torch.float16) + + +@cute.experimental.jit +def _bmm_no_bias( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + stream: _cuda.CUstream, +): + c = cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])) + b_sel = cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])) + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + b_sel, + c, + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[0, 1, 2])), + b_sel, + b_sel, + Int32(0), + Int32(0), + stream, + ) + + +@cute.experimental.jit +def _bmm_swiglu( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + g: cute.Tensor, + u: cute.Tensor, + gu_ld: cutlass.Int32, + b_ld: cutlass.Int32, + stream: _cuda.CUstream, +): + """_bmm_no_bias with the B operand built in the kernel from g / u (laid out as b, rows gu_ld apart; B rows b_ld + apart).""" + c = cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])) + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + c, + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[0, 1, 2])), + cute.make_tensor(g.iterator, cute.select(g.layout, mode=[2, 1, 0])), + cute.make_tensor(u.iterator, cute.select(u.layout, mode=[2, 1, 0])), + gu_ld, + b_ld, + stream, + ) + + +@cute.experimental.jit +def _bmm_push( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + stream: _cuda.CUstream, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: Int32, + ar_world: Int32, + ar_maxt: Int32, + ar_tokens: Int32, + ar_n: Int32, +): + c = cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])) + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + c, + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[0, 1, 2])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + Int32(0), + Int32(0), + stream, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + +@cute.experimental.jit +def _bmm_swiglu_push( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + g: cute.Tensor, + u: cute.Tensor, + gu_ld: cutlass.Int32, + b_ld: cutlass.Int32, + stream: _cuda.CUstream, + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: Int32, + ar_world: Int32, + ar_maxt: Int32, + ar_tokens: Int32, + ar_n: Int32, +): + """_bmm_swiglu that also pushes the output rows (_bmm_push's exchange).""" + c = cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])) + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])), + c, + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[0, 1, 2])), + cute.make_tensor(g.iterator, cute.select(g.layout, mode=[2, 1, 0])), + cute.make_tensor(u.iterator, cute.select(u.layout, mode=[2, 1, 0])), + gu_ld, + b_ld, + stream, + ar_ws, + ar_flags, + ar_rank, + ar_world, + ar_maxt, + ar_tokens, + ar_n, + ) + + +@cute.experimental.jit +def _bmm_bias( + gemm_op: cutlass.Constexpr, + a: cute.Tensor, + b: cute.Tensor, + c: cute.Tensor, + bias: cute.Tensor, + stream: _cuda.CUstream, +): + b_sel = cute.make_tensor(b.iterator, cute.select(b.layout, mode=[2, 1, 0])) + gemm_op( + cute.make_tensor(a.iterator, cute.select(a.layout, mode=[1, 2, 0])), + b_sel, + cute.make_tensor(c.iterator, cute.select(c.layout, mode=[1, 2, 0])), + cute.make_tensor(bias.iterator, cute.select(bias.layout, mode=[1, 2, 0])), + b_sel, + b_sel, + Int32(0), + Int32(0), + stream, + ) + + +def _from_dlpack_dynamic(tensor, leading_dim: int, assumed_align: int = 32): + return from_dlpack(tensor, assumed_align=assumed_align).mark_layout_dynamic( + leading_dim=leading_dim + ) + + +def _detect_leading_dim(tensor: _torch.Tensor) -> int: + # Ignore synthetic batch stride, including 1x1. + for dim, stride in enumerate(tensor.stride()[1:], start=1): + if stride == 1: + return dim + raise ValueError("tensor has no stride-1 dimension") + + +def _make_layout_tensor( + shape: tuple[int, ...], dtype: _torch.dtype, leading_dim: int +) -> _torch.Tensor: + permutation = [dim for dim in range(len(shape)) if dim != leading_dim] + [leading_dim] + return _torch.empty( + tuple(shape[dim] for dim in permutation), dtype=dtype, device="cuda" + ).permute([permutation.index(dim) for dim in range(len(shape))]) + + +def _make_compile_repr_tensors( + dtype: _torch.dtype, + has_bias: bool, + a_leading: int, + b_leading: int, + c_leading: int, +): + m, n, k, batch = 64, 8, _CTA_K, 1 + tensors = tuple( + _from_dlpack_dynamic(_make_layout_tensor(shape, dtype, leading_dim), leading_dim) + for shape, leading_dim in ( + ((batch, n, k), a_leading), + ((batch, k, m), b_leading), + ((batch, n, m), c_leading), + ) + ) + if not has_bias: + return (*tensors, None) + return ( + *tensors, + _from_dlpack_dynamic( + _torch.empty((n,), dtype=dtype, device="cuda").as_strided( + size=(batch, n, m), stride=(0, 1, 0) + ), + 1, + 2, + ), + ) + + +def _to_cute_swap(a, b, out, bias): + a_swap = b.unsqueeze(0).transpose(-2, -1) + b_swap = a.unsqueeze(0).transpose(-2, -1) + c_swap = out.unsqueeze(0).transpose(-2, -1) + leading_dims = tuple(_detect_leading_dim(tensor) for tensor in (a_swap, b_swap, c_swap)) + cute_tensors = tuple( + _from_dlpack_dynamic(tensor, leading_dim) + for tensor, leading_dim in zip((a_swap, b_swap, c_swap), leading_dims, strict=True) + ) + if bias is None: + return (*cute_tensors, None, leading_dims) + return ( + *cute_tensors, + _from_dlpack_dynamic( + bias.as_strided(size=(1, c_swap.shape[1], c_swap.shape[2]), stride=(0, 1, 0)), + 1, + 2, + ), + leading_dims, + ) + + +@cute_aot.cached_compile("m3_lowm_splitk._get_compiled_splitk_kernel", lambda: _aot_sources()) +def _get_compiled_splitk_kernel( + dtype, + tactic: SplitKTactic, + use_pdl: bool, + has_bias: bool, + leading_dims: tuple[int, int, int], + b_swiglu: bool = False, +): + if dtype not in _SUPPORTED_TORCH_DTYPES: + raise ValueError(f"split-K dense GEMM supports {_SUPPORTED_TORCH_DTYPES}; got {dtype}") + + kernel = SplitKDenseGemmKernel( + tactic=tactic, + use_pdl=use_pdl, + has_bias=has_bias, + b_swiglu=b_swiglu, + ) + compile_tensors = _make_compile_repr_tensors(dtype, has_bias, *leading_dims) + stream = _cuda.CUstream(_torch.cuda.current_stream().cuda_stream) + if b_swiglu: + # The gate / up halves are laid out as the B operand (public A) is. + gu = _make_compile_repr_tensors(dtype, False, *leading_dims)[0] + compiled = cute_ext.compile( + _bmm_swiglu, kernel, *compile_tensors[:3], gu, gu, Int32(2), Int32(1), stream + ) + elif has_bias: + compiled = cute_ext.compile(_bmm_bias, kernel, *compile_tensors, stream) + else: + compiled = cute_ext.compile(_bmm_no_bias, kernel, *compile_tensors[:3], stream) + return compiled + + +def _validate_runtime_tensors(a, b, bias, out) -> tuple[int, int, int]: + tensors = (a, b, out) + ((bias,) if bias is not None else ()) + if any(not isinstance(tensor, _torch.Tensor) for tensor in tensors): + raise ValueError("a, b, out, and bias must be torch tensors") + if a.ndim != 2 or b.ndim != 2 or out.ndim != 2: + raise ValueError("split-K dense GEMM accepts only 2D tensors") + if a.device.type != "cuda" or any(tensor.device != a.device for tensor in tensors): + raise ValueError("all tensors must be on the same CUDA device") + if a.dtype not in _SUPPORTED_TORCH_DTYPES or any(tensor.dtype != a.dtype for tensor in tensors): + raise ValueError("a, b, out, and bias must share BF16 or FP16 dtype") + if any(not (tensor.is_contiguous() or tensor.t().is_contiguous()) for tensor in (a, b, out)): + raise ValueError("a, b, and out must be dense row-major or column-major matrices") + if any(tensor.data_ptr() % 32 for tensor in (a, b, out)): + raise ValueError("a, b, and out must be 32-byte aligned") + + m, k = a.shape + if b.shape[0] != k: + raise ValueError(f"incompatible shapes: a is {tuple(a.shape)}, b is {tuple(b.shape)}") + n = b.shape[1] + if out.shape != (m, n): + raise ValueError(f"out must have shape {(m, n)}, got {tuple(out.shape)}") + if bias is not None and (bias.ndim != 1 or bias.shape[0] != n or not bias.is_contiguous()): + raise ValueError( + f"bias must be contiguous with shape {(n,)}, " + f"got shape {tuple(bias.shape)} and stride {bias.stride()}" + ) + + return m, n, k + + +def run_splitk_dense( + a, + b, + bias, + out, + pdl: bool, + tactic: SplitKTactic, +): + """Run ``A[M,K] @ B[K,N]`` with the ``mm_bf16`` layouts.""" + validate_tactic(tactic, *_validate_runtime_tensors(a, b, bias, out)) + has_bias = bias is not None + cute_tensors = _to_cute_swap(a, b, out, bias) + compiled = _get_compiled_splitk_kernel( + dtype=a.dtype, + tactic=tactic, + use_pdl=pdl, + has_bias=has_bias, + leading_dims=cute_tensors[4], + ) + stream = _cuda.CUstream(_torch.cuda.current_stream(a.device).cuda_stream) + if has_bias: + compiled(*cute_tensors[:4], stream) + else: + compiled(*cute_tensors[:3], stream) + return out + + +@cute_aot.cached_compile("m3_lowm_splitk._get_compiled_swiglu_push", lambda: _aot_sources()) +def _get_compiled_swiglu_push( + dtype, tactic: SplitKTactic, use_pdl: bool, leading_dims: tuple[int, int, int] +): + """The SwiGLU B load and the ar push in one kernel (the drafter down projection at a pushed boundary).""" + kernel = SplitKDenseGemmKernel( + tactic=tactic, use_pdl=use_pdl, has_bias=False, b_swiglu=True, ar_push=True + ) + compile_tensors = _make_compile_repr_tensors(dtype, False, *leading_dims) + gu = _make_compile_repr_tensors(dtype, False, *leading_dims)[0] + stream = _cuda.CUstream(_torch.cuda.current_stream().cuda_stream) + z64, z32 = cutlass.Int64(0), Int32(0) + return cute_ext.compile( + _bmm_swiglu_push, kernel, *compile_tensors[:3], gu, gu, Int32(2), Int32(1), stream, + z64, z64, z32, Int32(1), Int32(8), z32, Int32(6144), + ) # fmt: skip + + +def compile_swiglu_push(scratch, b, out, pdl: bool, tactic) -> None: + """Compile (only) the SwiGLU + push variant of ``run_splitk_dense_swiglu`` for these operands' layouts.""" + cute_tensors = _to_cute_swap(scratch, b, out, None) + _get_compiled_swiglu_push(scratch.dtype, SplitKTactic(*tactic), pdl, cute_tensors[4]) + + +def run_splitk_dense_swiglu( + gate_up, b, out, pdl: bool, tactic: SplitKTactic, scratch, ar: tuple = None +): + """``out = silu_and_mul(gate_up) @ b`` in one launch: ``gate_up`` [M, 2K] (gate then up, contiguous rows) is + activated by the B DMA warp (the Triton silu_and_mul arithmetic at its defaults, bit for bit) into ``scratch`` + [>= M, K], which the unchanged GEMM then reads; the output is bit for bit ``run_splitk_dense(silu_and_mul(gate_up), + b, None, out, pdl, tactic)``. ``scratch`` must not be read or written by anything else while the kernel runs. With + ``ar`` (``run_splitk_dense_push``'s tuple) the rows are also pushed into the drafter boundary all-reduce's + exchange, from the same kernel.""" + if gate_up.ndim != 2 or gate_up.shape[1] % 2 or not gate_up.is_contiguous(): + raise ValueError("gate_up must be a contiguous [M, 2K] matrix") + m, k = gate_up.shape[0], gate_up.shape[1] // 2 + a = scratch[:m] + if a.shape != (m, k) or not a.is_contiguous(): + raise ValueError(f"scratch must be a contiguous [>= {m}, {k}] matrix") + gate, up = gate_up[:, :k], gate_up[:, k:] + validate_tactic(tactic, *_validate_runtime_tensors(a, b, None, out)) + if any(t.data_ptr() % 32 for t in (gate, up)) or gate.dtype != a.dtype: + raise ValueError("gate / up halves must be 32-byte aligned and of the GEMM dtype") + cute_tensors = _to_cute_swap(a, b, out, None) + g_swap = gate.unsqueeze(0).transpose(-2, -1) + u_swap = up.unsqueeze(0).transpose(-2, -1) + lead = cute_tensors[4][1] + if _detect_leading_dim(g_swap) != lead or _detect_leading_dim(u_swap) != lead: + raise ValueError("gate / up must share the activation's K-major layout") + stream = _cuda.CUstream(_torch.cuda.current_stream(a.device).cuda_stream) + operands = ( + *cute_tensors[:3], + _from_dlpack_dynamic(g_swap, lead), + _from_dlpack_dynamic(u_swap, lead), + Int32(gate_up.stride(0)), + Int32(a.stride(0)), + stream, + ) + if ar is None: + compiled = _get_compiled_splitk_kernel( + dtype=a.dtype, + tactic=tactic, + use_pdl=pdl, + has_bias=False, + leading_dims=cute_tensors[4], + b_swiglu=True, + ) + compiled(*operands) + return out + compiled = _get_compiled_swiglu_push(a.dtype, tactic, pdl, cute_tensors[4]) + ws, flags, rank, world, maxt = ar + compiled( + *operands, + cutlass.Int64(ws), + cutlass.Int64(flags), + Int32(rank), + Int32(world), + Int32(maxt), + Int32(out.shape[0]), + Int32(out.shape[1]), + ) + return out + + +@cute_aot.cached_compile("m3_lowm_splitk._get_compiled_push", lambda: _aot_sources()) +def _get_compiled_push( + dtype, tactic: SplitKTactic, use_pdl: bool, leading_dims: tuple[int, int, int] +): + kernel = SplitKDenseGemmKernel(tactic=tactic, use_pdl=use_pdl, has_bias=False, ar_push=True) + compile_tensors = _make_compile_repr_tensors(dtype, False, *leading_dims) + stream = _cuda.CUstream(_torch.cuda.current_stream().cuda_stream) + z64, z32 = cutlass.Int64(0), Int32(0) + return cute_ext.compile( + _bmm_push, + kernel, + *compile_tensors[:3], + stream, + z64, + z64, + z32, + Int32(1), + Int32(8), + z32, + Int32(6144), + ) + + +def compile_push(a, b, out, pdl: bool, tactic) -> None: + """Compile (only) the push variant of ``run_splitk_dense_push`` for these operands' shapes and layouts.""" + cute_tensors = _to_cute_swap(a, b, out, None) + _get_compiled_push(a.dtype, SplitKTactic(*tactic), pdl, cute_tensors[4]) + + +def run_splitk_dense_push(a, b, out, pdl: bool, tactic: SplitKTactic, ar: tuple): + """``run_splitk_dense`` without bias that also pushes ``out``'s rows into a drafter boundary all-reduce's exchange; + ``ar`` = (multicast words address, flags address, rank, world, bucket tokens) of the 8-token bucket.""" + validate_tactic(tactic, *_validate_runtime_tensors(a, b, None, out)) + cute_tensors = _to_cute_swap(a, b, out, None) + compiled = _get_compiled_push(a.dtype, tactic, pdl, cute_tensors[4]) + stream = _cuda.CUstream(_torch.cuda.current_stream(a.device).cuda_stream) + ws, flags, rank, world, maxt = ar + compiled( + *cute_tensors[:3], + stream, + cutlass.Int64(ws), + cutlass.Int64(flags), + Int32(rank), + Int32(world), + Int32(maxt), + Int32(out.shape[0]), + Int32(out.shape[1]), + ) + return out + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_mc_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_mc_allreduce.py new file mode 100644 index 000000000000..bb87d8ead5bd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_mc_allreduce.py @@ -0,0 +1,439 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Low-latency TP all-reduce + residual + RMSNorm for decode (M <= 8 tokens) over a multicast mapping. + + partial = bf16(x + x2) (optional second input, e.g. shared + routed experts) + updated = bf16(residual + sum_r partial_r) (ranks summed in FP32, rank order) + normed = bf16(updated * rsqrt(mean(updated^2) + eps) * w) + +One cluster of CLUSTER CTAs per token; CTA c owns columns [c * H / CLUSTER, ...), one 16-byte chunk (8 BF16) +per thread. Each CTA pushes its slice of this rank's partial through the multicast mapping (one store lands in +every rank's buffer), releases dependents, polls its own buffer until all ranks' slices arrived, sums them, adds +the residual and reduces the RMS statistic across the cluster through DSMEM. + +Exchange protocol as in the K3 sandwich: int32 words, 0x80000000 = empty (pushed -0.0 halves become +0.0), each +CTA keeps a call counter whose parity selects the buffer half, readers re-arm the words they read. All ranks of +the TP group must issue the same calls in the same stream order. + +The workspace holds two exchange sets. With prepushed 3 two producers pushed this rank's x (set A) and x2 (set B), +e.g. the routed experts' finalize and the shared expert's down projection, and the kernel forms bf16(x + x2) per +rank from the two slots, exactly as the unpushed path does. + +Producers that push (the o_proj / shared-expert GEMV ``ar`` tuple, the routed-expert engine's ``arp`` descriptor) are +pointed at a bucket's exchange by its ``mc`` address, ``flags_for(8)`` counters, ``rank``, ``world_size`` and +``max_tokens``; ``get_workspace`` and ``_bucket`` select it. + +The kernel is ``torch.ops.trtllm.minimax_m3_mc_allreduce_norm``, reached through the ``ops/comm`` entries +``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm``; this module owns the workspaces, the bucket and +epilogue-option choice and the MXFP8 publication. +""" + +import functools +from dataclasses import dataclass +from typing import Any + +import torch + +from ..ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm as _allreduce_norm_op +from ..ops.comm.m3_mc_finalize_allreduce_norm import ( + m3_mc_finalize_allreduce_norm as _finalize_allreduce_norm_op, +) +from . import m3_report + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +BUCKETS = (8, 16, 48) # token capacity of each workspace (grid = bucket * cluster CTAs) +EMPTY_WORD = -2147483648 + +# Attention / MoE boundary epilogue options (each bit-identical): 1 residual + norm weight loaded at entry (before +# the poll); 2 cluster RMS exchange by st.async + mbarrier complete_tx instead of cluster.sync; 4 call parity rotated +# right after the poll (no trailing __syncthreads); 8 max-smem carveout so the next kernel's CTAs co-reside early. +# The 8-token bucket, by token count 1..8 (GB300 TP4 tuning). +AR_OPTS_BY_M = (7, 7, 15, 15, 15, 15, 7, 7) +# The 16- and 48-token buckets. +AR_OPTS_BY_BUCKET = {16: 15, 48: 15} +# The MoE-finalize all-reduce, per bucket. +AR_FIN_OPTS_BY_BUCKET = {8: 7, 16: 15, 48: 15} + + +# Option 2 is also the safe ordering of the cluster RMS exchange: its stores into peer CTAs' shared memory follow a +# cluster barrier arrive / wait pair. Without it they run before any cluster barrier, when a peer need not have +# started, so every call keeps it. +_OPT_CLUSTER_BARRIER = 2 + + +def _ar_opts(m: int, bucket: int, finalize: bool = False) -> int: + """Epilogue options for an m-row call on the given bucket; every option is bit-identical. A bucket missing from + the tables gets 0, and every result keeps _OPT_CLUSTER_BARRIER.""" + if finalize: + opts = AR_FIN_OPTS_BY_BUCKET.get(bucket, 0) + elif bucket == 8: + opts = AR_OPTS_BY_M[m - 1] + else: + opts = AR_OPTS_BY_BUCKET.get(bucket, 0) + return opts | _OPT_CLUSTER_BARRIER + + +@dataclass(eq=False) +class McAllReduceWorkspace: + """Two halves of [max_tokens][world][H] BF16 (int32 words) per rank behind one multicast mapping, plus one call + counter per CTA per cluster size. Create collectively and eagerly on every TP rank.""" + + uc: torch.Tensor + mc: torch.Tensor + flags: dict + rank: int + world_size: int + hidden: int + handle: Any + comm: Any + max_tokens: int = 8 + + @classmethod + def create(cls, mapping, hidden: int, max_tokens: int = 8) -> "McAllReduceWorkspace": + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + words = 4 * max_tokens * mapping.tp_size * (hidden // 2) # two exchange sets of two halves + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "McAllReduceWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 4, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("McAllReduceWorkspace.create failed on another rank") + return cls( + uc=uc, + mc=mc, + flags={}, + rank=mapping.tp_rank, + world_size=mapping.tp_size, + hidden=hidden, + handle=handle, + comm=comm, + max_tokens=max_tokens, + ) + + def flags_for(self, cluster: int): + """This cluster size's per-CTA call counters, or None when they do not exist yet during a capture. + get_workspace creates the cluster-8 counters with the workspace, so for cluster 8 (the only size built) + that None never happens and cannot differ between ranks.""" + flags = self.flags.get(cluster) + if flags is None: + if torch.cuda.is_current_stream_capturing(): + return None + flags = torch.zeros(self.max_tokens * cluster, dtype=torch.int32, device=self.uc.device) + self.flags[cluster] = flags + return flags + + +class McBuckets: + """One workspace per token bucket (8 / 16 / 48): calls of different sizes never share exchange slots.""" + + def __init__(self, workspaces): + self.workspaces = workspaces + self.hidden = workspaces[0].hidden + + def for_tokens(self, m: int): + for ws in self.workspaces: + if m <= ws.max_tokens: + return ws + return None + + +def _bucket(workspace, m): + return workspace.for_tokens(m) if isinstance(workspace, McBuckets) else workspace + + +def decline_reason(x, residual, workspace, cluster=8, x2=None) -> str: + """Why ``m3_mc_allreduce_norm`` returned None for these operands (for the dispatch's miss message).""" + if workspace is None: + return "no multicast workspace: it is created collectively on the first eager decode step" + bucket = _bucket(workspace, x.shape[0]) + if bucket is None: + return f"no workspace bucket holds {x.shape[0]} tokens (buckets {BUCKETS})" + if bucket.flags.get(cluster) is None: + return f"no {cluster}-CTA call counters for the {bucket.max_tokens}-token bucket" + return ( + f"operands outside the kernel: x {tuple(x.shape)} {x.dtype} contiguous={x.is_contiguous()}, " + f"residual contiguous={residual.is_contiguous()}, hidden {bucket.hidden}" + + ("" if x2 is None else f", x2 {tuple(x2.shape)} contiguous={x2.is_contiguous()}") + ) + + +def m3_mc_allreduce_norm( + x, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + x2=None, + quant=False, + prepushed=0, + drafter_norm=False, + carveout=True, + capture=None, +): + """Return (normed, updated_residual) or None when unsupported. ``carveout`` False drops opts bit 8 (the max-smem + carveout) for this call. ``drafter_norm``: the Eagle3 drafter boundary + (opts bit 16: normalize the unrounded FP32 sum, as the drafter's stock FlashInfer fused_add_rmsnorm does); 8-token + bucket only, x only, no MXFP8 copy. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM. ``capture`` = (out [>= M, H] view, fc_norm weight + [H], eps): the boundary also writes the Eagle3 capture of the updated residual, bit for bit + m3_spec_tail.capture_norm (CAPTURE_NORM_IN_BOUNDARY; not with ``drafter_norm``). + + A None makes the caller take another collective, so it must be the same on every TP rank. It is: it depends only + on the shapes, dtypes and layouts of the operands (the same code builds them on every rank) and on whether the + workspace's flags exist, which ``get_workspace`` creates collectively, before any capture, for cluster 8.""" + workspace = _bucket(workspace, x.shape[0]) + if workspace is None: + return None + if ( + x.dim() != 2 + or x.shape[0] > MAX_TOKENS + or x.dtype != torch.bfloat16 + or x.shape[1] != workspace.hidden + or not x.is_contiguous() + or not residual.is_contiguous() + or (x2 is not None and (not x2.is_contiguous() or x2.shape != x.shape)) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + if drafter_norm and ( + workspace.max_tokens != 8 or x2 is not None or quant or prepushed not in (0, 2) + ): + return None + outs = _allreduce_norm_op( + x, + x2, + residual, + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and x.shape[1] % 128 == 0), + int(prepushed), + (_ar_opts(x.shape[0], workspace.max_tokens) & (~0 if carveout else ~8)) + | (16 if drafter_norm else 0), + *_capture_args(capture, drafter_norm), + own_op=own_op(), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +def m3_mc_finalize_allreduce_norm( + fc2, + perm, + scales, + shared, + residual, + norm_weight, + eps, + workspace, + use_pdl=False, + cluster=8, + quant=False, + capture=None, +): + """MoE finalize (sum_k scale * fc2[perm] + shared) + all-reduce + residual + RMSNorm in one kernel. + Returns (normed, updated_residual) or None when unsupported. With ``quant`` the normed row is also published, + MXFP8-quantized, to m3_prequant.SLOT for the next MXFP8 GEMM. ``capture``: as ``m3_mc_allreduce_norm``.""" + m = residual.shape[0] + workspace = _bucket(workspace, m) + if workspace is None: + return None + if ( + residual.dim() != 2 + or m > MAX_TOKENS + or residual.shape[1] != workspace.hidden + or fc2.dtype != torch.bfloat16 + or not fc2.is_contiguous() + or fc2.dim() != 2 + or perm.dtype != torch.int32 + or perm.dim() != 2 + or perm.shape[0] != m + or not perm.is_contiguous() + or scales.dtype not in (torch.float32, torch.bfloat16) + or not scales.is_contiguous() + or (shared is not None and (shared.shape != residual.shape or not shared.is_contiguous())) + ): + return None + flags = workspace.flags_for(cluster) + if flags is None: + return None + outs = _finalize_allreduce_norm_op( + fc2, + perm, + scales, + shared, + residual.contiguous(), + norm_weight, + float(eps), + workspace.uc, + workspace.mc.data_ptr(), + flags, + workspace.rank, + workspace.world_size, + cluster, + use_pdl, + workspace.max_tokens, + bool(quant and residual.shape[1] % 128 == 0), + _ar_opts(m, workspace.max_tokens, finalize=True), + *_capture_args(capture, False), + own_op=own_op(), + ) + if len(outs) == 4: + from .m3_prequant import SLOT + + SLOT.publish(outs[0], outs[2], outs[3]) + return outs[0], outs[1] + + +@functools.lru_cache(maxsize=None) +def _own_op_loaded() -> bool: + return getattr(torch.ops.trtllm, "minimax_m3_sm107_tp8_mc_allreduce_norm", None) is not None + + +# The boundary all-reduces run this target's op (trtllm::minimax_m3_sm107_tp8_mc_allreduce_norm) rather than the +# image's: the same exchange and arithmetic built for the two-tray TP8 group. +OWN_OP = True +OWN_OP_PATH = "m3_mc_ar_own_op" + + +def own_op() -> bool: + """OWN_OP when this target's op is loaded (trtllm::minimax_m3_sm107_tp8_mc_allreduce_norm), with + its engagement line; else the image's op. The same library is loaded on every rank, so the ranks agree.""" + if not OWN_OP: + return False + if not _own_op_loaded(): + m3_report.missed( + OWN_OP_PATH, + None, + "trtllm::minimax_m3_sm107_tp8_mc_allreduce_norm is not loaded (the image's op runs)", + ) + return False + m3_report.taken(OWN_OP_PATH, None, "the boundary all-reduces run this target's op") + return True + + +@functools.lru_cache(maxsize=None) +def capture_supported() -> bool: + """Whether the multicast all-reduce op the boundaries call (own_op) has the capture epilogue (cap_out / cap_w / + cap_eps). The same library is loaded on every rank, so the ranks agree.""" + name = ( + "minimax_m3_sm107_tp8_mc_allreduce_norm" + if own_op() + else "minimax_m3_sm107_tp2_mc_allreduce_norm" + ) + op = getattr(torch.ops.trtllm, name, None) + if op is None: + return False + return any(a.name == "cap_out" for a in op.default._schema.arguments) + + +def _capture_args(capture, drafter_norm: bool) -> tuple: + """(cap_out, cap_w, cap_eps) for the op, from ``capture`` = (out, fc_norm weight, eps) or None.""" + if capture is None: + return (None, None, 0.0) + assert not drafter_norm, "the capture epilogue is not built for the drafter boundary" + out, weight, eps = capture + return (out, weight, float(eps)) + + +_WORKSPACE: dict = {} +_AGREE_COMMS: dict = {} + + +def _tp_comm(mapping): + """A communicator over mapping's TP group, created once (collective over that group only).""" + key = tuple(mapping.tp_group) + comm = _AGREE_COMMS.get(key) + if comm is None: + from tensorrt_llm._utils import mpi_comm, mpi_disabled + + if mpi_disabled(): + comm = mapping.tp_group_pg + else: + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + _AGREE_COMMS[key] = comm + return comm + + +def tp_agree(mapping, ok: bool) -> bool: + """Whether ``ok`` holds on every rank of mapping's TP group (one blocking host collective). + + For a fast path whose availability is decided from rank-local state (a kernel build that can fail on one rank): + every rank must take the same path, so that they issue the same collectives with the same exchange protocol and + the same kernels. Call it once per decision, at build time, at the same point on every rank, outside CUDA-graph + capture; never per step. With one rank, or attention DP (whose ranks run different batches), it returns ``ok``. + """ + if mapping.tp_size <= 1 or mapping.enable_attention_dp: + return ok + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("tp_agree must run outside CUDA-graph capture") + from tensorrt_llm._torch.distributed.ops import _mnnvl_workspace_all_succeeded + + return _mnnvl_workspace_all_succeeded(_tp_comm(mapping), bool(ok)) + + +def tp_agree_built(mapping, built, what: str): + """``built`` when it is not None on every TP rank, else None on every rank (see ``tp_agree``).""" + if tp_agree(mapping, built is not None) or built is None: + return built + from tensorrt_llm.logger import logger + + logger.warning(f"{what}: built here but not on another TP rank; every rank falls back") + return None + + +def get_workspace(mapping, hidden: int): + """Process-wide workspace (created collectively on first eager use); None while capturing before creation. + + The first eager call for a hidden size (the warmup forward, before any CUDA-graph capture) creates one + McAllReduceWorkspace per bucket in BUCKETS on every TP rank (MPI collective over the TP group) plus each + bucket's 8-CTA-cluster call counters; later calls, captured or not, return the same McBuckets. The workspaces + live until process exit. + """ + ws = _WORKSPACE.get(hidden) + if ws is None: + if torch.cuda.is_current_stream_capturing(): + return None + wss = [McAllReduceWorkspace.create(mapping, hidden, b) for b in BUCKETS] + for w in wss: + w.flags_for(8) + ws = McBuckets(wss) + _WORKSPACE[hidden] = ws + return ws diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/__init__.py new file mode 100644 index 000000000000..e361dc614231 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/__init__.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 MoE layer of the TP8 target for low-batch decode as a CuTe DSL kernel: ``m3_moe`` for 1..8 tokens +(op.py) at the TP8 routed and shared intermediate of 384 per rank.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/m3_moe_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/m3_moe_kernel.py new file mode 100644 index 000000000000..a25da7b59491 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/m3_moe_kernel.py @@ -0,0 +1,3971 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MiniMax-M3 routed experts for up to 8 decode tokens on one TP rank: ``m3_moe`` (group-phased weight stream). + +One persistent kernel replaces TRTLLM-Gen's MiniMax2 routing, FC1 (NVFP4 x NVFP4 + SwiGLU-OAI + NVFP4 requantization), +FC2 and finalize, optionally with the router GEMV and the activation's NVFP4 quantization folded in (FUSED_ROUTER). It +writes the rank's routed partial [M, 6144] bf16, bit for bit what the TRTLLM-Gen op returns. + +- Routing (warps 4..7, token t on warp 4 + t % 4): TRTLLM-Gen's MiniMax2 top-4, operation for operation; warp 4 then + makes the distinct experts slots in (token, top-k) order (match.any), slot s in group s / 4. +- Work: the groups' FC1 stages and FC2 tiles stream through one ring per CTA in phases: phase p = FC1 of group p (a + stream-K range of the group's unit-major stages), then FC2 of group p - LAG (this CTA's 32-row output blocks, 4 + slots' slices stacked in one 128-row MMA, B row j M + t). LAG phases of streaming hide the hand-off of group + p - LAG's intermediate between CTAs. +- FC1 stage: a 32-row unit's rows at 4 consecutive 256-K tiles (rows 32 q + r) against resident B rows QS q + t. + FC2 stage: 4 slots' 32 rows at one 256-K tile. Both carry 8 KB of raw scale atoms, spliced into the 4 instruction + atoms by warp 6 with one formula (word [r][q] <- raw atom 4 q + kb, word [r][m1]). +- FC1 epilogue: k-quarters summed in order, SwiGLU-OAI, NVFP4 requant into tagged 16-byte records (3 words of 3 data + bytes + the call's tag); a unit split between two CTAs is summed by its tail's holder (Lamport slot). +- Warps 5 and 8.. (XW warps) build each group's FC2 B (tiles + scale atoms) from the records once they carry the + call's tag; warp 5 copies the scales to TMEM. +- Finalize: bf16(fc2_alpha * acc) per (slot, token) into its top-k position, then per token the fma chain over its + top-4 in order, bf16 out. With an all-reduce descriptor, every output word is also pushed to the exchange + (AR_PRE: call parities and descriptor loaded during the last group's FC2). + +Configuration (trace-time constants) via M3_CONFIG = {"m_max", "num_ctas", "stages", "nb2_cap", "fused_router", "lag", +"xw", ...}; the plan (op.make_phase_plan) must be built for the same lag. +Launched with PDL: barrier setup, TMEM allocation, tensormap prefetch, bias / plan / router weight loads before +griddepcontrol.wait; every producer-output read and every global write after it. +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass._mlir.dialects import llvm as _llvm +from cutlass._mlir.extras import types as _T +from cutlass.cutlass_dsl import dsl_user_op +from cutlass.experimental import primitives as prims + +_CFG = globals().get("M3_CONFIG") or {} + + +def _cfg(key: str, default): + return type(default)(_CFG.get(key, default)) + + +# ============================================================================= +# Problem shape (trace-time constants). +# ============================================================================= +H = 6144 # capacity: fixed: the checkpoint's hidden size +# routed intermediate per rank: 768 at TP4, 384 at TP8 (FC2 K = 1.5 256-K tiles: the last tile's A box is half out +# of bounds, zero-filled by TMA, and only its first KB2_LAST K-64 MMAs / scale atoms are issued; no padding, no extra +# bytes) +I_TP = _cfg("i_tp", 768) +TWO_I = 2 * I_TP +assert I_TP % 128 == 0 and 128 <= I_TP <= 768, I_TP +E_LOCAL = 128 +TOP_K = 4 # capacity: fixed: the checkpoint's top-k +ROUTE_SCALE = 2.0 +SUM_EPS = 1e-20 +M_MAX = _cfg("m_max", 1) +NCTA = _cfg("num_ctas", 152) +FUSED_ROUTER = bool(_cfg("fused_router", 0)) +NROUTE = E_LOCAL if FUSED_ROUTER else 0 # experts whose logits the router GEMV computes +NGEMV = (NROUTE + NCTA - 1) // NCTA # experts per CTA: bx, bx + NCTA, .. +TIMING = bool(_cfg("timing", 0)) +HANG_DBG = bool(_cfg("hang_dbg", 0)) +XVDUMP = bool( + _cfg("xvdump", 0) +) # debug only (needs a plan without split units): FC1 sums into `partials` +YDUMP = bool( + _cfg("ydump", 0) +) # debug only: FC2 outputs per (token, k) into `partials` (breaks its protocol) +LAG = _cfg("lag", 2) # phase p runs FC1 of group p and FC2 of group p - LAG +G_CAP = TOP_K * M_MAX # distinct experts per call +NG_CAP = M_MAX # groups of 4 slots +XW = _cfg("xw", 5 if M_MAX > 4 else (4 if M_MAX > 2 else 1)) # FC2 B builder warps: 5, then 8 .. +THREADS = 256 + 32 * (XW - 1) +GEMV_THREADS = 256 # capacity: fixed: the CTA size +# SHARED: the MoE's shared expert (MXFP8 gate_up [1536, 6144] -> SwiGLU-OAI -> MXFP8 down [6144, 768]) on NSH more +# CTAs (warps 0-3), its output added in finalize: out = bf16(shared + routed) (shared_output.add_(routed_output)). +# The math is mxfp8_low_m_gemm's (native MXFP8 activation quant, mma.sync m16n8k32 e4m3 per 32-K block, block scales +# as one fma, its partial-sum order, bf16) and the Triton silu_and_mul's, operation for operation. Shared CTA s owns +# intermediate positions 32 s .. 32 s + 31 (gate and up rows: MMA tiles = warps 0-3) and down rows 256 s .. + 255; +# 16-row x 768-byte weight chunks (+ their scale atoms) stream through NBW buffers per warp, the first NBW before +# the grid dependency. h crosses the shared CTAs, y_sh reaches the routed CTAs' finalize (both tagged words). +SHARED = bool(_cfg("shared", 0)) +# LATE_TRIGGER: release the dependents at teardown, not after the grid dependency. An early release lets the next +# kernels' CTAs (the all-reduce, then the attention it releases at entry) become resident during this kernel, packed +# onto the SMs it leaves free, where the attention then runs on a fraction of the GPU. +LATE_TRIGGER = bool(_cfg("late_trigger", 1)) +# AR_PRE: the finalize's all-reduce push uses the call parities and descriptor fields loaded while the last group's +# FC2 runs (not two dependent global loads per push on the kernel's tail). +AR_PRE = bool(_cfg("ar_pre", 1)) +# OUT_SKIP (MOE-FINALIZE-DEADSTORE): on a call that pushes into the boundary all-reduce (arp enabled), +# the finalize does not also store its bf16 rows into ``out``: the pushed all-reduce (prepushed >= 2) never reads them +# and returns its own tensors, so the store is dead traffic. Calls without a push store as before. Bitwise: the pushed +# words are unchanged. +OUT_SKIP = bool(_cfg("out_skip", 0)) +# EARLY_AR (scheduling only): when the finalize pushes into the multicast all-reduce, EVERY CTA releases the +# dependents right after the grid dependency, so the all-reduce (which reads only the pushed tagged words, without a +# grid wait) and, through its own early release, the next layer's QKV GEMV become resident on each SM this kernel +# frees, instead of after the last CTA's teardown. The arithmetic is unchanged. +EARLY_AR = bool(_cfg("early_ar", 1)) +# TAIL_TRIGGER (scheduling only): on a pushing call, each routed CTA releases the dependents when its +# weights producer has issued its last stage (the post-stream tail: FC1 drain, hand-off, FC2, finalize), not after the +# grid wait. The op turns it on in place of EARLY_AR (they are alternative release points). +TAIL_TRIGGER = bool(_cfg("tail_trigger", 0)) +assert not (EARLY_AR and TAIL_TRIGGER), "early_ar and tail_trigger are alternative release points" +# With an early release the next kernels' completion no longer implies this kernel's (the pushed all-reduce reads only +# the pushed tagged words, without a grid wait), so the per-CTA call epoch -- read by the NEXT call on this state -- is +# stored before the release point's consumers can finish (once every thread of the CTA has read it), not at teardown +# Every other workspace write of a call (records, partials, logit exchange, y_sh) lands before the +# finalize's push. +EARLY_EPOCH = EARLY_AR or TAIL_TRIGGER +# XSTAGE (fused router): the hidden states land in the (still idle) ring by one bulk copy per token right after the +# grid dependency; the NVFP4 quantization of B1 reads them from shared memory (one HBM / L2 round trip, not ~8). +XSTAGE = bool(_cfg("xstage", 1)) and bool(_cfg("fused_router", 0)) +NSH = 24 if SHARED else 0 +# SH20 (``sh_ctas``): the NSH shared work units (unit u: intermediate positions SH_ROWS u .., down rows 256 u ..) +# on NSHC shared CTAs; CTA s runs units s, s + NSHC, .. (< NSH), the gate_up of every unit before any down (a unit's +# down reads every unit's h). A unit's arithmetic does not depend on the CTA that runs it: bit-identical outputs. +NSHC = _cfg("sh_ctas", NSH) if SHARED else 0 +SH_NU = (NSH + NSHC - 1) // NSHC if SHARED else 1 # units on the busiest shared CTA +assert not SHARED or (1 <= SH_NU <= 2 and NSHC <= NSH), (NSHC, NSH) +# the shared intermediate per rank: 768 (TP4), 384 (TP8). A shared CTA owns SH_ROWS intermediate positions: its +# gate_up MMA tile is SH_SPLIT K slices x (SH_ROWS gate + SH_ROWS up rows) = 128 rows (TP4: 2 x (32 + 32), the order +# of m3_mx_gemv split 2; TP8: 4 x (16 + 16), m3_mx_gemv split 4), and 6144 / NSH = 256 down rows over the full K. +SH_I = _cfg("sh_i", 768) +SH_ROWS = SH_I // 24 +# gate_up K slices (= m3_mx_gemv's cluster ranks, contiguous K ranges) +SH_SPLIT = 128 // (2 * SH_ROWS) +SH_NB = 8 * SH_SPLIT # gate_up B rows (8 q + t: token t's slice-q k-tile) = its MMA N +SH_KB = H // 32 # capacity: fixed: a kernel tile constant (gate_up K blocks) +SH_DKB = SH_I // 32 # down K blocks +SH_DKT = SH_I // 128 # down k-tiles +SH_GT = SH_KB // 4 // SH_SPLIT # gate_up k-tiles (128 K) per slice +SH_STAGES_GU = SH_GT +SH_STAGES = SH_GT + 2 * SH_DKT # + 2 down tiles x SH_DKT k-tiles +SHX_BAR = 6 +# SH_PRETILE: the shared weights come pre-tiled (op._pretile_shared): per shared CTA its SH_STAGES stages, each the +# 16 KB shared-memory image of its 128-row MMA tile (128-byte swizzle) followed by the stage's 512-byte instruction +# scale atom, so a stage is two contiguous bulk copies (not 128 row segments of 128 bytes) and needs no splicing. +SH_PRETILE = bool( + _cfg("sh_pretile", 1 if M_MAX <= 2 or SH_SPLIT != 2 else 0) +) # (in-chain: a win at M <= 2, not 4..8) +if SHARED: + assert SH_I in (384, 768), SH_I + assert SH_PRETILE or SH_SPLIT == 2, ( + "the un-pretiled shared stream is the 32-row (TP4) layout only" + ) +SH_STAGE_BYTES = 16384 + 512 +MMA_M = 128 +KT_BYTES = 128 +INC_K = 2 +NUM_KB = 4 +# FC1 +N1 = 16 if M_MAX <= 4 else 32 # B rows: QS q + t (token t at the stage's k-tile 4 j + q) +QS = N1 // 4 +R1 = 32 +C1 = 4 +U1 = TWO_I // R1 # 48 units per slot +K1_TILES = H // 256 +S1 = K1_TILES // C1 # capacity: fixed: pipeline stages (6 stages per unit) +NUM_BYTES_A = MMA_M * KT_BYTES # capacity: fixed: a stage's operand bytes +NUM_BYTES_BOX1 = R1 * KT_BYTES +SF1_KA = H // 64 +SF1_RB = TWO_I // 128 +NUM_BYTES_SFA_RAW = 16 * 512 +NUM_WORDS_SFA_RAW = NUM_BYTES_SFA_RAW // 4 +NUM_BYTES_SFA = NUM_KB * 512 # capacity: fixed: a stage's scale bytes +NUM_WORDS_SFA = NUM_BYTES_SFA // 4 +NUM_BYTES_B1 = N1 * KT_BYTES +NUM_SFB1_ATOMS = S1 * NUM_KB +# FC2 +R2 = 32 +NB2 = H // R2 # 192 +KT2 = (I_TP + 255) // 256 # 3 (TP4), 2 (TP8) +# K-64 instructions of the last FC2 k-tile: 4 (TP4), 2 (TP8) +KB2_LAST = (I_TP - 256 * (KT2 - 1)) // 64 + + +def _nkb2(t2: int) -> int: + """K-64 MMA instructions (= scale atoms) of FC2 k-tile t2.""" + return NUM_KB if t2 < KT2 - 1 else KB2_LAST + + +SF2_KA = I_TP // 64 +SF2_RB = H // 128 +N2 = max(8, (4 * M_MAX + 7) // 8 * 8) # B rows j M + t +NUM_BYTES_B2 = N2 * KT_BYTES +NB2_CAP = _cfg("nb2_cap", 2) +UNITS_ROW = TWO_I // R1 # 48 records per (slot, token) row +H_ROW = UNITS_ROW * 16 +# HB_REPL: the tagged FC1 -> FC2 records written to HB_REPL copies (HB_STRIDE int32 words apart); CTA bx polls copy +# bx % HB_REPL. Every routed CTA polls every record, so with one copy the hand-off latency depends on where the few +# hot lines land in L2 (TP8 M=1: 1.5 KB of records, +-1 us across allocations); copies spread the polls. +HB_REPL = _cfg("hb_repl", 1) +assert 1 <= HB_REPL <= 32, HB_REPL # (copy i is written by lane i) +HB_STRIDE = (G_CAP * M_MAX * UNITS_ROW * 4 + 255) // 256 * 256 + 32 +PARTIAL_EMPTY = -1 # capacity: fixed: a sentinel, not a size +EPI_THREADS = 128 +EPI_BAR = 1 +ROUTE_BAR = 2 +GEMV_BAR = 3 +B2_BAR = 5 +TD_STRIDE = 64 # tdbg columns per CTA +TOPK_BAR = 4 +B1_AFTER_TOPK = bool( + _cfg("b1_after_topk", 0) +) # (experiment) the B1 build waits for the routing warps' top-k +TOPK2_BAR = 7 +EARLY0 = bool( + _cfg("early0", 1) +) # warp 4 streams group 0 (token 0's experts) right after its own top-k +EARLY0_N = _cfg("early0_n", 99) # stages issued early (capped at STAGES) +# routing warps: 4..7, then the FC2 B builder warps 8.. (idle until the routing is done) up to one warp per token +RW_EXTRA = max(0, min(M_MAX - 4, (THREADS // 32) - 8)) if bool(_cfg("rw_extra", 1)) else 0 +NRW0 = ( + min(4, M_MAX) if bool(_cfg("rw_trim", 1)) else 4 +) # warps 4.. with a token (no idle pollers in the barrier) +NRW = NRW0 + RW_EXTRA +LG_REPL = _cfg("lg_repl", 32 if M_MAX > 4 else 128) +GEMV_SMEM = bool( + _cfg("gemv_smem", 1) +) # (XSTAGE) the router GEMV reads the staged hidden states from smem +LG_STRIDE = M_MAX * E_LOCAL + 32 +POLL_NS = _cfg("poll_ns", 150) +B2_POLL_NS = _cfg("b2_poll_ns", 200) +# B2IMG (needs SFA_CP: warp 6 is free): the FC1 epilogue writes each finished unit's FP4 bytes and scale byte straight +# into a per-group global image laid out as the FC2 B tile + its scale atoms (exactly what the builder warps assembled +# from the tagged records); per group, warp 6 joins the epilogue (fc1_done), fences the async proxy and does one +# release add of this CTA's finished units to the group counter; warp 5 acquires the counter (every group gains +# CNT_PER_CALL per call, CTA 0 pads absent slots), fences, and bulk-copies the image into the B2 buffer. +B2IMG = bool(_cfg("b2img", 0)) and bool(_cfg("sfa_cp", 0)) +IMG_B2 = KT2 * NUM_BYTES_B2 +IMG_SF = KT2 * NUM_KB * 512 +IMG_BYTES = IMG_B2 + IMG_SF +CNT_PER_CALL = 4 * (TWO_I // R1) +B2IMG_NS = _cfg("b2img_ns", 16) # (B2IMG) counter poll back-off +B2_DLY = _cfg( + "b2_dly", 0 +) # (stress) the release warp spins up to this many ns (CTA- and group-dependent) first +W_WARP = 4 # capacity: fixed: a warp role index +X_WARP = 5 # capacity: fixed: a warp role index +S_WARP = 6 # capacity: fixed: a warp role index +MMA_WARP = 7 # capacity: fixed: a warp role index +# E0_M4: at M = 4 the MMA warp routes +# token 3 (NRW0 = 4), so the early group-0 stages go to FC2 B builder warp 8 instead (idle until the routing is done; +# RW_EXTRA = 0 at M = 4). Issue order only: exact. +E0_M4 = ( + bool(_cfg("e0_m4", 1)) and NRW0 == 4 and RW_EXTRA == 0 and THREADS // 32 > 8 +) # M == 4 only: at M > 4 warp 8 routes (RW_EXTRA) +E0_WARP = ( + 8 if E0_M4 else MMA_WARP +) # (M <= 3) the early stages are issued by the idle MMA warp, not by warp 4 (routing's critical path) +E0_BAR = 8 +E1_BAR = 9 +E0_OFFLOAD = EARLY0 and (NRW0 <= 3 or E0_M4) and bool(_cfg("e0_off", 1)) +# B1_AFTER_TOPK holds warps 0..3 (the B1 build, which arrives x_bar[1] when XSTAGE) on TOPK2_BAR until every routing +# warp arrives; with EARLY0 and no E0 offload, warp 4 first waits on x_bar[1] to fill the ring before it arrives, so the +# two wait on each other. Named barriers have no HANG_NS trap, so the kernel would spin forever: refuse the config. +if B1_AFTER_TOPK and XSTAGE and EARLY0 and not E0_OFFLOAD: + raise ValueError( + "m3_moe: b1_after_topk=1 with xstage, early0 and no early-stage offload deadlocks (warp 4 waits on x_bar[1] " + "before TOPK2_BAR, the B1 build waits on TOPK2_BAR before x_bar[1]); set b1_after_topk=0 or early0=0" + ) +EVICT_FIRST = 0x12F0000000000000 +_LOG2E = 1.4426950408889634 +LD_POW2 = ( + 1 if M_MAX == 1 else 2 if M_MAX == 2 else 4 if M_MAX <= 4 else 8 +) # tcgen05.ld .num covering M columns + +_FIXED_SMEM = ( + S1 * NUM_BYTES_B1 + + 2 * KT2 * NUM_BYTES_B2 + + 2 * KT2 * NUM_KB * 512 + + NB2_CAP * M_MAX * TOP_K * R2 * 4 + + 2 * M_MAX * C1 * R1 * 4 + + (M_MAX * (H // 16) if FUSED_ROUTER else 0) + + 6 * 1024 +) +# SFA_CP: the A scale words go by 4-byte cp.async straight into the instruction atom layout (no raw scale box, no +# splicer warp; the MMA warp tcgen05.cp's them), completing on the stage's full barrier: 18 KB stages instead of 26 KB +SFA_CP = bool(_cfg("sfa_cp", 0)) +_RAW_STAGE = 0 if SFA_CP else NUM_BYTES_SFA_RAW +_STAGE_SMEM = NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA +_STAGES_FIT = (227 * 1024 - _FIXED_SMEM - (128 if SHARED else 0)) // _STAGE_SMEM +ACC1_COL = 0 +ACC2_COL = 2 * N1 +SFB1_COL = ACC2_COL + 2 * N2 +SFB2_COL = SFB1_COL + NUM_SFB1_ATOMS * 4 +SFA1_COL = SFB2_COL + 2 * KT2 * NUM_KB * 4 +STAGES = min(_cfg("stages", 12 if SFA_CP else 8), _STAGES_FIT, (512 - SFA1_COL) // 16) +TMEM_COLS = 512 # capacity: fixed: the SM's tensor-memory columns +assert 1 <= M_MAX <= 8, M_MAX +assert STAGES >= 3, (STAGES, _STAGES_FIT, SFA1_COL) +assert M_MAX * H * 2 <= STAGES * (NUM_BYTES_A + _RAW_STAGE + NUM_BYTES_SFA), ( + "x staging must fit the ring" +) +# one shared-memory region for the ring and the FC1 / FC2 B operands (a shared CTA reuses all of it) +_OFF_SFAR = STAGES * NUM_BYTES_A +_OFF_SFA = _OFF_SFAR + STAGES * _RAW_STAGE +_OFF_B1 = _OFF_SFA + STAGES * NUM_BYTES_SFA +_OFF_B2 = _OFF_B1 + S1 * NUM_BYTES_B1 +_OFF_SFB2 = _OFF_B2 + 2 * KT2 * NUM_BYTES_B2 +BIG = _OFF_SFB2 + 2 * KT2 * NUM_KB * 512 +# shared CTA layout in BIG: weight buffers, x quarters (2), x scales [M, 192], gate / up [4, 16, M], h, h scales +# shared CTA layout in BIG: A ring (16 KB stages), raw scale atoms (4 per stage), spliced atoms, B (x: [stage][16 rows] +# [128 B]; h: [6][8][128 B]), x / h scale codes, gate_up partial exchange +_SB_A = 0 +_SB_RAW = None +# slices 1.. gate / up partials [slice - 1][u][p][8 tokens] f32 +_SH_XCH_BYTES = (SH_SPLIT - 1) * 2 * SH_ROWS * 8 * 4 +_SH_FIX = ( + SH_GT * SH_NB * 128 + + SH_DKT * 8 * 128 + + M_MAX * SH_KB + + M_MAX * SH_DKB + + _SH_XCH_BYTES + + 2 * SH_ROWS * 8 * 4 + + 1024 +) +SH_NST = min(8, (BIG - _SH_FIX) // (NUM_BYTES_A + 4 * 512 + 512)) if SHARED else 1 +_SB_RAW = SH_NST * NUM_BYTES_A +_SB_SPL = _SB_RAW + SH_NST * 2048 +_SB_BX = (_SB_SPL + SH_NST * 512 + 1023) // 1024 * 1024 +_SB_BH = _SB_BX + SH_GT * SH_NB * 128 +_SB_XC = _SB_BH + SH_DKT * 8 * 128 +_SB_HC = _SB_XC + M_MAX * SH_KB +_SB_XCH = (_SB_HC + M_MAX * SH_DKB + 15) // 16 * 16 +_SB_GU = _SB_XCH + _SH_XCH_BYTES +_SB_END = _SB_GU + 2 * SH_ROWS * 8 * 4 +# shared CTA TMEM columns: gate_up accumulator (N SH_NB), down accumulators (2 x N 8), A scales per stage, B scales +SH_ACC_GU = 0 +SH_ACC_DN = SH_NU * SH_NB # one gate_up accumulator (N SH_NB) per unit of the CTA +SH_SFA = SH_ACC_DN + 16 * SH_NU # one pair of down accumulators (2 x N 8) per unit +SH_SFB_GU = SH_SFA + 4 * 8 +SH_SFB_DN = SH_SFB_GU + 4 * SH_GT +SH_TMEM = 256 # capacity: fixed: the shared CTAs' TMEM columns +if SHARED: + assert FUSED_ROUTER, "shared expert: fused router only" + assert SH_STAGES - SH_NST <= 64, SH_STAGES + assert SH_NST >= 3 and _SB_END <= BIG and SH_SFB_DN + 24 <= SH_TMEM, (SH_NST, _SB_END, BIG) + + +# ============================================================================= +# Inline-PTX helpers. +# ============================================================================= +def _asm(ret, args, ptx, cons, side=False, loc=None, ip=None): + return _llvm.inline_asm( + ret, + [a.ir_value(loc=loc, ip=ip) for a in args], + ptx, + cons, + has_side_effects=side, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +# ---- all-reduce push (the MoE boundary's multicast exchange, set A): +# arp = int64 [enable, mc, flags, rank, world, maxt] +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32_if(addr, value, pred, *, loc=None, ip=None): + """st.global.b32 when pred != 0.""" + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip), + cutlass.Int32(pred).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tsetp.ne.s32 p, $2, 0;\n\t@p st.global.b32 [$0], $1;\n\t}", "l,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _ar_push(arp, v, low, t, cl): + """Push bf16 channel pair (cl, cl + 1) of token t (v = channel cl's bf16 value on this lane, its neighbour's on + lane ^ 8) into this rank's set-A slot of the exchange; the slot half is the all-reduce CTA (t, cl // 768)'s call + parity. Only lanes with ``low`` store; every lane of the warp must call it (shuffle).""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + world = arp.load(idx=4) + parity = cutlass.Int64( + _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cs // cutlass.Int32(H // 8)) + ) + & cutlass.Int32(1) + ) + off = ( + parity * arp.load(idx=5) * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + arp.load(idx=3)) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(arp.load(idx=1) + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +def _ar_parity(arp, t, cl): + """The call parity _ar_push reads for token t's channel cl (fixed between this kernel's grid dependency and the + next all-reduce, which this kernel releases at teardown): loaded ahead of the finalize (AR_PRE).""" + return _ld_volatile_s32( + arp.load(idx=2) + + cutlass.Int64(4) * cutlass.Int64(t * cutlass.Int32(8) + cl // cutlass.Int32(H // 8)) + ) + + +def _ar_push_pre(ard, parity32, v, low, t, cl): + """_ar_push with the parity (_ar_parity) and the descriptor fields ard = (mc, rank, world, maxt) preloaded.""" + hi = cute.arch.shuffle_sync_bfly(v, 8) + pred = cutlass.select_(low, cutlass.Int32(1), cutlass.Int32(0)) + cs = cutlass.select_(low, cl, cutlass.Int32(0)) + mc, rk, world, maxt = ard + parity = cutlass.Int64(parity32 & cutlass.Int32(1)) + off = ( + parity * maxt * world * cutlass.Int64(H // 2) + + (cutlass.Int64(t) * world + rk) * (cutlass.Int64(H // 2)) + + cutlass.Int64(cs // cutlass.Int32(2)) + ) + _st_global_b32_if(mc + off * cutlass.Int64(4), _bf16x2_word(v, hi), pred) + + +@dsl_user_op +def _bar_arrive(bar_id: int, threads: int, *, loc=None, ip=None): + """bar.arrive: signal a named barrier without waiting (its memory ordering is bar.sync's).""" + _llvm.inline_asm( + None, + [], + f"bar.arrive {bar_id}, {threads};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _st_relaxed_u32(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "st.relaxed.gpu.global.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _st_relaxed_u64(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int64(val)], + "st.relaxed.gpu.global.u64 [$0], $1;", + "l,l", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_relaxed_u64(addr, *, loc=None, ip=None): + return cutlass.Int64( + _asm( + _T.i64(), + [cutlass.Int64(addr)], + "ld.relaxed.gpu.global.u64 $0, [$1];", + "=l,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _nanosleep(ns: int, *, loc=None, ip=None): + _llvm.inline_asm( + None, + [], + f"nanosleep.u32 {ns};", + "", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +@dsl_user_op +def _gtime(*, loc=None, ip=None): + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), + [], + "mov.u64 $0, %globaltimer;", + "=l", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@dsl_user_op +def _smid(*, loc=None, ip=None): + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [], + "mov.u32 $0, %smid;", + "=r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + ) + + +@cute.jit +def _stamp(tdbg: cutlass.Array, bx, i: cutlass.Constexpr): + if cutlass.const_expr(TIMING): + tdbg.store(_gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(i)) + + +@dsl_user_op +def _rcp(a, *, loc=None, ip=None): + """rcp.approx.ftz.f32 (the reciprocal TRT-LLM's FP4 quantization uses).""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(a)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _div_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fma_rn(a, b, c, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b), cutlass.Float32(c)], + "fma.rn.f32 $0, $1, $2, $3;", + "=f,f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _add_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "add.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _sub_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "sub.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _mul_rn(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "mul.rn.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e4m3_bits(x, *, loc=None, ip=None): + """e4m3 (round to nearest even, saturating) of x as an int 0..255.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, 0f00000000, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3_to_f32(b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Int32(b)], + "{\n.reg .b16 t, l, u;\n.reg .b32 h;\ncvt.u16.u32 t, $1;\ncvt.rn.f16x2.e4m3x2 h, t;\nmov.b32 {l, u}, h;\n" + "cvt.f32.f16 $0, l;\n}", + "=f,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _e2m1x2(hi, lo, *, loc=None, ip=None): + """One byte of two E2M1 values (lo in the low nibble), round to nearest even, saturating.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b8 t;\ncvt.rn.satfinite.e2m1x2.f32 t, $1, $2;\ncvt.u32.u8 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2(x, *, loc=None, ip=None): + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.ftz.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _bf16_round(x, *, loc=None, ip=None): + """x rounded to bf16 (nearest even), as f32.""" + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\nmov.b32 $0, {0, t};\n}", + "=f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _redux_max_u32(v, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(v)], + "redux.sync.max.u32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16 (nearest even) bits of x in the low 16 bits.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rn.bf16.f32 t, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ue8m0_rp(x, *, loc=None, ip=None): + """UE8M0 code of x rounded toward +inf, saturating (__nv_cvt_float_to_e8m0(x, SATFINITE, RoundPosInf)).""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(x)], + "{\n.reg .b16 t;\ncvt.rp.satfinite.ue8m0x2.f32 t, $1, $1;\ncvt.u32.u16 $0, t;\n}", + "=r,f", + False, + loc, + ip, + ) + ) & cutlass.Int32(0xFF) + + +@dsl_user_op +def _e4m3x2(hi, lo, *, loc=None, ip=None): + """Two e4m3 (nearest even, saturating) in 16 bits, lo in the low byte.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Float32(hi), cutlass.Float32(lo)], + "{\n.reg .b16 t;\ncvt.rn.satfinite.e4m3x2.f32 t, $1, $2;\ncvt.u32.u16 $0, t;\n}", + "=r,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _min_xorsign_abs(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "min.xorsign.abs.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _div_full(a, b, *, loc=None, ip=None): + return cutlass.Float32( + _asm( + _T.f32(), + [cutlass.Float32(a), cutlass.Float32(b)], + "div.full.f32 $0, $1, $2;", + "=f,f,f", + False, + loc, + ip, + ) + ) + + +@dsl_user_op +def _ex2_full(x, *, loc=None, ip=None): + """ex2.approx.f32 (no flush to zero), Triton's tl.exp.""" + return cutlass.Float32( + _asm(_T.f32(), [cutlass.Float32(x)], "ex2.approx.f32 $0, $1;", "=f,f", False, loc, ip) + ) + + +@dsl_user_op +def _l2_prefetch(addr, nbytes, *, loc=None, ip=None): + """cp.async.bulk.prefetch.L2 of nbytes (a multiple of 16) at global address addr (16-byte aligned).""" + _llvm.inline_asm( + None, + [ + cutlass.Int64(addr).ir_value(loc=loc, ip=ip), + cutlass.Int32(nbytes).ir_value(loc=loc, ip=ip), + ], + "cp.async.bulk.prefetch.L2.global [$0], $1;", + "l,r", + has_side_effects=True, + is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, + loc=loc, + ip=ip, + ) + + +def _mx_osc(amax): + """mxfp8 quantization of a block with absolute max amax: (UE8M0 code, the element multiplier).""" + code = _ue8m0_rp(_mul_rn(amax, _rcp(cutlass.Float32(448.0)))) + cf = cutlass.Int32( + cutlass.select_( + code == cutlass.Int32(255), + cutlass.Int32(0x7FFFFFFF), + cutlass.select_( + code == cutlass.Int32(0), cutlass.Int32(0x00400000), code << cutlass.Int32(23) + ), + ) + ).bitcast(cutlass.Float32) + return code, cutlass.select_(amax != cutlass.Float32(0.0), _rcp(cf), cutlass.Float32(0.0)) + + +def _mx_words(v, osc): + """e4m3 words of len(v) (a multiple of 4) values times osc, in element order.""" + words = [] + for t in range(len(v) // 4): + lo = _e4m3x2(_mul_rn(v[4 * t + 1], osc), _mul_rn(v[4 * t], osc)) + hi = _e4m3x2(_mul_rn(v[4 * t + 3], osc), _mul_rn(v[4 * t + 2], osc)) + words.append(lo | (hi << cutlass.Int32(16))) + return words + + +def _twiddle(score): + """Float bits mapped so that unsigned order is float order (cub TwiddleIn).""" + bits = ( + score.bitcast(cutlass.Int32) + if hasattr(score, "bitcast") + else cutlass.Float32(score).bitcast(cutlass.Int32) + ) + return cutlass.select_(bits < cutlass.Int32(0), ~bits, bits | cutlass.Int32(-2147483648)) + + +@cute.jit +def _wait(bar: cutlass.Array, parity): + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +def _tptr(base_row, col, dtype=cutlass.Int32): + return cutlass.inttoptr((base_row << 16) | col, 6, dtype) + + +@dsl_user_op +def _match_any(value, *, loc=None, ip=None): + """The mask of the warp's lanes holding the same 32-bit value.""" + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int32(value)], + "match.any.sync.b32 $0, $1, 0xffffffff;", + "=r,r", + False, + loc, + ip, + ) + ) + + +@cute.jit +def _waitd(bar: cutlass.Array, parity, tdbg: cutlass.Array, site: cutlass.Constexpr): + """_wait, or (HANG_DBG) give up after ~2^22 polls and record the site in tdbg[bx, 24 + warp].""" + if cutlass.const_expr(HANG_DBG): + n = cutlass.Int32(0) + while (not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity)) & ( + n < cutlass.Int32(1 << 22) + ): + n = n + cutlass.Int32(1) + if n >= cutlass.Int32(1 << 22): + bx, _, _ = cute.arch.block_idx() + w = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + tdbg.store( + cutlass.Int64(site), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + w + ) + else: + while not cute.arch.mbarrier_try_wait(bar.data_ptr(), parity): + pass + cute.arch.sync_warp() + + +@cute.jit +def _adv(stage, phase): + """The next ring position's (stage, phase).""" + stage = stage + cutlass.Int32(1) + wrap = stage == cutlass.Int32(STAGES) + return cutlass.select_(wrap, cutlass.Int32(0), stage), cutlass.select_( + wrap, phase ^ cutlass.Int32(1), phase + ) + + +@cute.jit +def _seg(i0, i1, k): + """Segment k of the FC1 item range [i0, i1) (items = unit-major stages): (unit, first stage, end stage). Order: + the last unit (its head, the part a split unit's next CTA does not cover), then the first unit, then the rest.""" + uf = i0 // cutlass.Int32(S1) + ul = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) + unit = uf + k - cutlass.Int32(1) + unit = cutlass.select_(k == cutlass.Int32(0), ul, unit) + unit = cutlass.select_(ul == uf, uf, unit) + j0 = i0 - unit * cutlass.Int32(S1) + j0 = cutlass.select_(j0 < cutlass.Int32(0), cutlass.Int32(0), j0) + j1 = i1 - unit * cutlass.Int32(S1) + j1 = cutlass.select_(j1 > cutlass.Int32(S1), cutlass.Int32(S1), j1) + return unit, j0, j1 + + +@cute.jit +def _nsegs(i0, i1): + n = (i1 - cutlass.Int32(1)) // cutlass.Int32(S1) - i0 // cutlass.Int32(S1) + cutlass.Int32(1) + return cutlass.select_(i1 > i0, n, cutlass.Int32(0)) + + +@cute.jit +def _sfb1_to_tmem(warp, lane, base_row, base_col, src: cutlass.Array, b1_ready: cutlass.Array): + """B1 scales to TMEM: instruction (j, kb)'s column holds, at lane n = QS q + t, token t's 4 scales of k-tile + 4 j + q, K 64 kb .. (src: int32 words [M, 96] of the token scales); each warp writes its sub-partition.""" + q = lane // cutlass.Int32(QS) + tl = lane - q * cutlass.Int32(QS) + sf_ok = (lane < cutlass.Int32(N1)) & (tl < cutlass.Int32(M_MAX)) + sfv = [] + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + j = a // NUM_KB + kb = a % NUM_KB + wsf = src.load( + idx=cutlass.select_( + sf_ok, + tl * cutlass.Int32(H // 64) + + (cutlass.Int32(4 * j) + q) * cutlass.Int32(4) + + cutlass.Int32(kb), + cutlass.Int32(0), + ) + ) + sfv.append(cutlass.select_(sf_ok, cutlass.Int32(wsf), cutlass.Int32(0))) + for a in cutlass.range_constexpr(NUM_SFB1_ATOMS): + prims.tcgen05_st( + "32x32b", + _tptr(base_row + warp * cutlass.Int32(32), base_col + SFB1_COL + a * 4), + sfv[a], + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(b1_ready) + + +@cute.jit +def _b1_addr(t, kb16): + """Word index in the resident B tiles of the 8 FP4 bytes of token t's 16-value block kb16.""" + kt = kb16 >> cutlass.Int32(4) + j = kt >> cutlass.Int32(2) + n = (kt & cutlass.Int32(3)) * cutlass.Int32(QS) + t + cc = (kb16 >> cutlass.Int32(1)) & cutlass.Int32(7) + return ( + j * cutlass.Int32(NUM_BYTES_B1) + + n * cutlass.Int32(KT_BYTES) + + (cc ^ (n & cutlass.Int32(7))) * cutlass.Int32(16) + + (kb16 & cutlass.Int32(1)) * cutlass.Int32(8) + ) // cutlass.Int32(4) + + +@cute.jit +def _build_b1( + tidx, + warp, + lane, + base_row, + base_col, + x32: cutlass.Array, + xsf32: cutlass.Array, + sB1: cutlass.Array, + b1_ready: cutlass.Array, +): + """B (the M tokens' whole K) into the resident tiles from the NVFP4 activation, its scales to TMEM.""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + for r in cutlass.range_constexpr((M_MAX * (H // 16) + EPI_THREADS - 1) // EPI_THREADS): + b = tidx + cutlass.Int32(r * EPI_THREADS) + if b < cutlass.Int32(M_MAX * (H // 16)): + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + v2 = x32.load(idx=b * cutlass.Int32(2), vector_size=2, alignment=8) + sB1_32.store( + (cutlass.Int32(v2[0]), cutlass.Int32(v2[1])), idx=_b1_addr(t, kb), alignment=8 + ) + _sfb1_to_tmem(warp, lane, base_row, base_col, xsf32, b1_ready) + + +@cute.jit +def _build_b1_fused( + tidx, + warp, + lane, + base_row, + base_col, + xb32: cutlass.Array, + sfsc: cutlass.Array, + sB1: cutlass.Array, + s_xsf: cutlass.Array, + b1_ready: cutlass.Array, +): + """_build_b1 from the bf16 hidden states, quantized exactly as fp4_quantize(x, sf_scale, 16).""" + sB1_32 = cutlass.Array( + sB1.data_ptr(0), shape=(S1 * NUM_BYTES_B1 // 4,), dtype=cutlass.Int32, alignment=16 + ) + s_xsf8 = cutlass.Array( + s_xsf.data_ptr(0), shape=(M_MAX * (H // 16),), dtype=cutlass.Int8, alignment=16 + ) + sc = cutlass.Float32(sfsc.load(idx=0)) + rsc = _rcp(sc) + rsix = _rcp(cutlass.Float32(6.0)) + NBLK = M_MAX * (H // 16) // EPI_THREADS + QB = ( + 3 if NBLK % 3 == 0 else (2 if NBLK % 2 == 0 else 1) + ) # blocks whose loads are in flight together + # a rolled loop over groups of QB blocks (a fully unrolled one is ~50 KB of SASS at M = 8: instruction fetch bound) + for rg in cutlass.range(NBLK // QB, unroll=1): + wall = [] + for rr in cutlass.range_constexpr(QB): + bq = tidx + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(EPI_THREADS) + for h in cutlass.range_constexpr(2): + wall.append( + xb32.load( + idx=bq * cutlass.Int32(8) + cutlass.Int32(4 * h), + vector_size=4, + alignment=16, + ) + ) + for rr in cutlass.range_constexpr(QB): + b = tidx + (rg * cutlass.Int32(QB) + cutlass.Int32(rr)) * cutlass.Int32(EPI_THREADS) + t = b // cutlass.Int32(H // 16) + kb = b - t * cutlass.Int32(H // 16) + w = [] + for h in cutlass.range_constexpr(2): + v4 = wall[rr * 2 + h] + for z in cutlass.range_constexpr(4): + w.append(cutlass.Int32(v4[z])) + xs = [] + for z in cutlass.range_constexpr(8): + xs.append((w[z] << cutlass.Int32(16)).bitcast(cutlass.Float32)) + xs.append((w[z] & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + vmax = cute.math.abs(xs[0]) + for z in cutlass.range_constexpr(1, 16): + vmax = cute.arch.fmax(vmax, cute.math.abs(xs[z])) + sfb = _e4m3_bits(_mul_rn(sc, _mul_rn(vmax, rsix))) + osc = cutlass.select_( + vmax != cutlass.Float32(0.0), + _rcp(_mul_rn(_e4m3_to_f32(sfb), rsc)), + cutlass.Float32(0.0), + ) + by = [ + _e2m1x2(_mul_rn(xs[2 * z + 1], osc), _mul_rn(xs[2 * z], osc)) & cutlass.Int32(0xFF) + for z in range(8) + ] + lo = ( + by[0] + | (by[1] << cutlass.Int32(8)) + | (by[2] << cutlass.Int32(16)) + | (by[3] << cutlass.Int32(24)) + ) + hi = ( + by[4] + | (by[5] << cutlass.Int32(8)) + | (by[6] << cutlass.Int32(16)) + | (by[7] << cutlass.Int32(24)) + ) + sB1_32.store((lo, hi), idx=_b1_addr(t, kb), alignment=8) + s_xsf8.store(cutlass.Int8(sfb), idx=b) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + _sfb1_to_tmem(warp, lane, base_row, base_col, s_xsf, b1_ready) + + +@cute.jit +def _swiglu_quant(xv, gate_raw, is_up, a, c, alpha, beta, lim): + """TRTLLM-Gen's FC1 epilogue (the swiGlu Bmm E2m1 output), operation for operation: lane value xv is the up row, + gate_raw its gate row (fp32 sums). Returns (q, the scale's e4m3 bits) of the 16-row NVFP4 block (lanes r, r ^ 16 + and r ^ 8 cover it; amax over the warp's up lanes of a half). a = fc31_alpha, c = fc31_scale_c.""" + neg_max = cutlass.Float32(-3.40282346638528859812e38) + gc = cutlass.select_(gate_raw > lim, lim, gate_raw) + gc = cutlass.select_(gate_raw < neg_max, neg_max, gc) + uc = cutlass.select_(xv > lim, lim, xv) + uc = cutlass.select_(xv < -lim, -lim, uc) + c1 = _mul_rn(a, _mul_rn(alpha, cutlass.Float32(1.4426950216293334961))) + sgm = _rcp(_add_rn(_ex2(-_mul_rn(gc, c1)), cutlass.Float32(1.0))) + gs = _mul_rn(gc, sgm) + h = _mul_rn(_fma_rn(uc, a, _mul_rn(a, beta)), gs) + h = cutlass.select_(is_up, h, cutlass.Float32(0.0)) + amax = cute.arch.fmax(_mul_rn(cute.math.abs(h), c), cutlass.Float32(0.0)) + for off in (16, 8, 4, 2, 1): + amax = cute.arch.fmax(amax, cute.arch.shuffle_sync_bfly(amax, off)) + sfb = _e4m3_bits(_mul_rn(amax, cutlass.Float32(0.16666667163372039795))) + sff = _e4m3_to_f32(sfb) + osc = cutlass.select_(sff != cutlass.Float32(0.0), _rcp(sff), cutlass.Float32(0.0)) + return _mul_rn(_mul_rn(h, osc), c), sfb + + +@cute.jit +def _plan_range(s_plan: cutlass.Array, p): + w = s_plan.load(idx=p) + return w & cutlass.Int32(0xFFFF), w >> cutlass.Int32(16) + + +@cute.jit +def _sh_issue( + lane, + sid, + g, + gs, + s_big: cutlass.Array, + sh_full: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, +): + """Stage g of the shared CTA's stream into ring slot g % SH_NST (warp 0). g < SH_GT: gate_up k-tile g of both + slices: quadrant Q = 2 q + u (q slice, u 0 gate / 1 up) = rows 32 sid .. (+ 768 for up) at k-tile SH_GT q + g, its + raw scale atom at raw slot Q. Else down tile j = (g - SH_GT) / 6 (rows 256 sid + 128 j ..), k-tile + (g - SH_GT) % 6. ``sid`` is the unit, ``gs`` the CTA's stage index (its ring slot is gs % SH_NST).""" + st = gs % cutlass.Int32(SH_NST) + bar = sh_full.subview(st) + if cutlass.const_expr(SH_PRETILE): + if lane == 0: + src = (sid * cutlass.Int32(SH_STAGES) + g) * cutlass.Int32(SH_STAGE_BYTES) + prims.mbarrier_arrive_expect_tx(bar, SH_STAGE_BYTES) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), gusf.subview(src), bar, 16384 + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + gusf.subview(src + cutlass.Int32(16384)), + bar, + 512, + ) + cute.arch.sync_warp() + else: + if g < cutlass.Int32(SH_GT): + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 4 * 512) + cute.arch.sync_warp() + if lane < 4: + q = lane >> cutlass.Int32(1) + u = lane & cutlass.Int32(1) + row0 = sid * cutlass.Int32(32) + u * cutlass.Int32(SH_I) + kt = q * cutlass.Int32(SH_GT) + g + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A) + lane * cutlass.Int32(4096)), + tma_gu, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview( + cutlass.Int32(_SB_RAW) + + st * cutlass.Int32(2048) + + lane * cutlass.Int32(512) + ), + gusf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_KB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(SH_DKT) + kt = gd - jt * cutlass.Int32(SH_DKT) + row0 = sid * cutlass.Int32(256) + jt * cutlass.Int32(128) + if lane == 0: + prims.mbarrier_arrive_expect_tx(bar, NUM_BYTES_A + 512) + prims.cp_async_bulk_tensor_shared_cta_global( + s_big.subview(st * cutlass.Int32(NUM_BYTES_A)), + tma_dn, + (kt * cutlass.Int32(128), row0), + bar, + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(cutlass.Int32(_SB_RAW) + st * cutlass.Int32(2048)), + dnsf.subview( + ((row0 >> cutlass.Int32(7)) * cutlass.Int32(SH_DKB // 4) + kt) + * cutlass.Int32(512) + ), + bar, + 512, + ) + cute.arch.sync_warp() + + +@dsl_user_op +def _red_release_add(addr, val, *, loc=None, ip=None): + _asm( + None, + [cutlass.Int64(addr), cutlass.Int32(val)], + "red.release.gpu.global.add.u32 [$0], $1;", + "l,r", + True, + loc, + ip, + ) + + +@dsl_user_op +def _ld_acquire_u32(addr, *, loc=None, ip=None): + return cutlass.Int32( + _asm( + _T.i32(), + [cutlass.Int64(addr)], + "ld.acquire.gpu.global.u32 $0, [$1];", + "=r,l", + True, + loc, + ip, + ) + ) + + +@dsl_user_op +def _fence_proxy_async_global(*, loc=None, ip=None): + _asm(None, [], "fence.proxy.async.global;", "", True, loc, ip) + + +@cute.jit +def _sfa_to_tmem(sSFA: cutlass.Array, stage, base_row, base_col): + """(SFA_CP, elected MMA thread) the stage's 4 scale atoms smem -> TMEM (what the splicer warp does otherwise).""" + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SFA1_COL + stage * cutlass.Int32(16) + kb * 4), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + + +@cute.jit +def _sf1_cp( + lane, stage, sf_line, c, sSFA: cutlass.Array, s1w: cutlass.Array, ab_full: cutlass.Array +): + """(SFA_CP) FC1 stage scale atoms: word [r][q] of atom kb <- raw atom 4 q + kb (raw line sf_line = 4 x atom) + word [r][c % 4] (what the splicer copies out of the raw box), 4-byte cp.async by the warp's 32 lanes; their + arrivals (noinc) complete the stage's full barrier together with the A boxes' bytes.""" + src0 = ( + (sf_line >> cutlass.Int32(2)) * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (c & cutlass.Int32(3)) + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for q in cutlass.range_constexpr(C1): + for kb in cutlass.range_constexpr(NUM_KB): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + q)), + s1w.subview(src0 + cutlass.Int32((4 * q + kb) * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _sf2_cp( + lane, + stage, + g, + blk, + t2: cutlass.Constexpr, + n_here, + s_el: cutlass.Array, + sSFA: cutlass.Array, + s2w: cutlass.Array, + ab_full: cutlass.Array, +): + """(SFA_CP) FC2 k-tile t2 scale atoms: word [r][jj] of atom kb <- raw atom (rows / 128, K block 4 t2 + kb) of slot + jj's expert, word [r][blk % 4].""" + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + src0 = ( + ( + (e * cutlass.Int32(SF2_RB) + (blk >> cutlass.Int32(2))) * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(128) + + lane * cutlass.Int32(4) + + (blk & cutlass.Int32(3)) + ) + # (a partial last k-tile has no atoms past I_TP) + for kb in cutlass.range_constexpr(_nkb2(t2)): + prims.cp_async_shared_global( + sSFA.subview(dst0 + cutlass.Int32(kb * 128 + jj)), + s2w.subview(src0 + cutlass.Int32(kb * 128)), + 4, + "ca", + ) + prims.cp_async_mbarrier_arrive(ab_full.subview(stage), noinc=True) + cute.arch.sync_warp() + + +@cute.jit +def _sh_unit_stage(gs, nu): + """CTA stage gs -> (unit slot ui, the unit's stage g): the gate_up stages of the CTA's nu units unit by unit, then + their down stages unit by unit. One unit: (0, gs).""" + if cutlass.const_expr(SH_NU == 1): + return cutlass.Int32(0), gs + ngu = nu * cutlass.Int32(SH_GT) + is_gu = gs < ngu + gd = gs - ngu + nd = cutlass.Int32(SH_STAGES - SH_GT) + ui = cutlass.select_(is_gu, gs // cutlass.Int32(SH_GT), gd // nd) + g = cutlass.select_(is_gu, gs - ui * cutlass.Int32(SH_GT), cutlass.Int32(SH_GT) + gd - ui * nd) + return ui, g + + +@cute.jit +def _shared_cta( + tidx, + warp, + lane, + sid, + s_big: cutlass.Array, + sh_bar: cutlass.Array, + epochs: cutlass.Array, + tdbg: cutlass.Array, + bx, + xb32: cutlass.Array, + tma_gu, + tma_dn, + gusf: cutlass.Array, + dnsf: cutlass.Array, + hx: cutlass.Array, + dnx: cutlass.Array, + arp: cutlass.Array, + tmem_ptr_i32: cutlass.Array, + gu8: cutlass.Array, + dn8: cutlass.Array, +): + """Shared CTA sid (see SHARED; SH20: its units sid, sid + NSHC, ..): tcgen05 MXF8 (E4M3 x E4M3, UE8M0 per 32 K), + m3_mx_gemv's tile order. gate_up: one + 128-row MMA tile = 2 K slices x (32 gate + 32 up rows), B rows 8 q + t = token t's slice-q k-tile; down: 2 tiles + of 128 rows, full K. Warps: 0 weights (TMA), 1 TMEM + scale splice + MMA, 4-7 epilogue; 2.. activation quant.""" + big32 = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Int32, alignment=16) + bigf = cutlass.Array(s_big.data_ptr(0), shape=(BIG // 4,), dtype=cutlass.Float32, alignment=16) + big8 = cutlass.Array(s_big.data_ptr(0), shape=(BIG,), dtype=cutlass.Int8, alignment=16) + # SH20: this CTA's unit count (units sid + NSHC ui) and stage count + nu = cutlass.Int32(1) + if cutlass.const_expr(SH_NU > 1): + nu = cutlass.select_(sid < cutlass.Int32(NSH - NSHC), cutlass.Int32(2), cutlass.Int32(1)) + n_st = nu * cutlass.Int32(SH_STAGES) + sh_full = sh_bar # [0, SH_NST): stage full + sh_empty = cutlass.Array(sh_bar.data_ptr(8), shape=(8,), dtype=cutlass.Int64, alignment=8) + b_gu_rdy = sh_bar.subview(16) + b_dn_rdy = sh_bar.subview(17) + acc_gu = sh_bar.subview(18) + acc_dn = sh_bar.subview(19) + if warp == 0: + prims.prefetch_tensormap(tma_gu) + prims.prefetch_tensormap(tma_dn) + if lane < SH_NST: + prims.mbarrier_init(sh_full.subview(lane), 1) + prims.mbarrier_init( + sh_empty.subview(lane), 33 + ) # the MMAs' commit + each splicer lane's reads done + if lane == 0: + prims.mbarrier_init(b_gu_rdy, 1) + prims.mbarrier_init(b_dn_rdy, 1) + prims.mbarrier_init(acc_gu, 1) + prims.mbarrier_init(acc_dn, 1) + if warp == 1: + prims.tcgen05_alloc(tmem_ptr_i32, SH_TMEM) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + if warp == 0: + for gs0 in cutlass.range_constexpr(SH_NST): # layer constants: before the grid dependency + ui0, g0 = _sh_unit_stage(cutlass.Int32(gs0), nu) + _sh_issue( + lane, + sid + ui0 * cutlass.Int32(NSHC), + g0, + cutlass.Int32(gs0), + s_big, + sh_full, + tma_gu, + tma_dn, + gusf, + dnsf, + ) + # the rest of this CTA's weights into L2 (64 gate / up rows of 6 KB, 256 down rows of 768 B) + if cutlass.const_expr(SH_PRETILE): + # this CTA's stages beyond the ring, 4 per lane + for r in cutlass.range_constexpr(2): + pg = cutlass.Int32(SH_NST) + lane * cutlass.Int32(2) + cutlass.Int32(r) + if pg < cutlass.Int32(SH_STAGES): + _l2_prefetch( + gusf.subview( + (sid * cutlass.Int32(SH_STAGES) + pg) * cutlass.Int32(SH_STAGE_BYTES) + ) + .data_ptr() + .toint(), + SH_STAGE_BYTES, + ) + if cutlass.const_expr(SH_NU > 1): + # SH20: the second unit's stages (rows sid + NSHC), into L2 as well + if nu > cutlass.Int32(1): + for r in cutlass.range_constexpr((SH_STAGES + 31) // 32): + pg = lane + cutlass.Int32(32 * r) + if pg < cutlass.Int32(SH_STAGES): + _l2_prefetch( + gusf.subview( + ((sid + cutlass.Int32(NSHC)) * cutlass.Int32(SH_STAGES) + pg) + * cutlass.Int32(SH_STAGE_BYTES) + ) + .data_ptr() + .toint(), + SH_STAGE_BYTES, + ) + else: + for r in cutlass.range_constexpr(2): + prw = sid * cutlass.Int32(32) + cutlass.Int32(r * SH_I) + lane + _l2_prefetch(gu8.subview(prw * cutlass.Int32(H)).data_ptr().toint(), H) + if lane == 0: + _l2_prefetch( + dn8.subview(sid * cutlass.Int32(256 * SH_I)).data_ptr().toint(), 256 * SH_I + ) + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(not LATE_TRIGGER): + if arp.load(idx=0) == cutlass.Int64(0): + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + ep = epochs.load(idx=bx) + if cutlass.const_expr(EARLY_EPOCH): + prims.barrier_cta_sync(0) # every thread holds this call's epoch + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + NQ = THREADS - 64 # activation quant threads (warps 2 ..) + if warp == 0: + for gs in cutlass.range(SH_NST, n_st, unroll=1): + _wait( + sh_empty.subview(gs % cutlass.Int32(SH_NST)), + ((gs // cutlass.Int32(SH_NST)) - cutlass.Int32(1)) & cutlass.Int32(1), + ) + ui, g = _sh_unit_stage(gs, nu) + _sh_issue( + lane, + sid + ui * cutlass.Int32(NSHC), + g, + gs, + s_big, + sh_full, + tma_gu, + tma_dn, + gusf, + dnsf, + ) + elif warp == 1: + # ---- MMA: gate_up (scale atoms spliced to the tile's row order), then down + idesc_gu = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=SH_NB, + m_dim=128, + ) + idesc_dn = prims.Tcgen05MxInstrDesc.build( + a_dtype=cutlass.Float8E4M3FN, + b_dtype=cutlass.Float8E4M3FN, + scale_format=1, + n_dim=8, + m_dim=128, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + desc_a0 = prims.Tcgen05SmemDesc.build( + s_big.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bx = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BX), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_bh = prims.Tcgen05SmemDesc.build( + s_big.subview(_SB_BH), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + _wait(b_gu_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for gs in cutlass.range(n_st, unroll=1): + # SH20: CTA stage gs = unit slot ui's stage g; its accumulators are the unit's own TMEM columns + ui, g = _sh_unit_stage(gs, nu) + sq = (sid + ui * cutlass.Int32(NSHC)) & cutlass.Int32(3) + st = gs % cutlass.Int32(SH_NST) + if gs == nu * cutlass.Int32(SH_GT): + _wait(b_dn_rdy, 0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + _wait(sh_full.subview(st), (gs // cutlass.Int32(SH_NST)) & cutlass.Int32(1)) + raw = cutlass.Int32(_SB_RAW // 4) + st * cutlass.Int32(512) + if (g < cutlass.Int32(SH_GT)) & cutlass.Boolean(not SH_PRETILE): + # MMA row 32 Q + i = quadrant Q's row i: word 4 i + Q <- raw atom Q's word 4 i + sid % 4 + w4 = [ + big32.load(idx=raw + cutlass.Int32(128 * qq) + lane * cutlass.Int32(4) + sq) + for qq in range(4) + ] + big32.store( + ( + cutlass.Int32(w4[0]), + cutlass.Int32(w4[1]), + cutlass.Int32(w4[2]), + cutlass.Int32(w4[3]), + ), + idx=cutlass.Int32(_SB_SPL // 4) + + st * cutlass.Int32(128) + + lane * cutlass.Int32(4), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + prims.mbarrier_arrive( + sh_empty.subview(st) + ) # this lane's generic reads of the slot are done + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + if g < cutlass.Int32(SH_GT): + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview(cutlass.Int32(_SB_SPL) + st * cutlass.Int32(512)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bx + cutlass.Int64(g * cutlass.Int32(SH_NB * 128 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr( + tcol + SH_ACC_GU + ui * cutlass.Int32(SH_NB), 6, cutlass.Float32 + ), + da + 2 * kb, + db + 2 * kb, + idesc_gu.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (g > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_GU + g * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_GT - 1): + prims.tcgen05_commit(acc_gu) + else: + gd = g - cutlass.Int32(SH_GT) + jt = gd // cutlass.Int32(SH_DKT) + kt = gd - jt * cutlass.Int32(SH_DKT) + dsf = prims.Tcgen05SmemDesc.build( + s_big.subview( + cutlass.Int32(_SB_SPL if SH_PRETILE else _SB_RAW) + + st * cutlass.Int32(512 if SH_PRETILE else 2048) + ), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + prims.tcgen05_cp( + s2t_shape, + _tptr(base_row, base_col + SH_SFA + st * cutlass.Int32(4)), + dsf, + multicast=s2t_mc, + ) + da = desc_a0 + cutlass.Int64(st * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_bh + cutlass.Int64(kt * cutlass.Int32(1024 >> 4)) + for kb in cutlass.range_constexpr(4): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, + prims.CTAGroup.CTA_1, + cutlass.inttoptr( + tcol + SH_ACC_DN + ui * cutlass.Int32(16) + jt * cutlass.Int32(8), + 6, + cutlass.Float32, + ), + da + 2 * kb, + db + 2 * kb, + idesc_dn.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + (kt > cutlass.Int32(0)) | cutlass.Boolean(kb > 0), + cutlass.inttoptr( + tcol + SH_SFA + st * cutlass.Int32(4), 6, cutlass.Int32 + ), + cutlass.inttoptr( + tcol + SH_SFB_DN + kt * cutlass.Int32(4), 6, cutlass.Int32 + ), + ) + prims.tcgen05_commit(sh_empty.subview(st)) + if g == cutlass.Int32(SH_STAGES - 1): + prims.tcgen05_commit(acc_dn) + cute.arch.sync_warp() + if warp >= 2: + # ---- the activation, MXFP8 per 32-K block (native mxfp8_quantize), into B (row 8 q + t of stage tile i = + # token t's k-tile SH_GT q + i, 128-byte swizzle) and its codes + qt = tidx - cutlass.Int32(64) + for r in cutlass.range_constexpr((M_MAX * SH_KB + NQ - 1) // NQ): + q_ = qt + cutlass.Int32(r * NQ) + if q_ < cutlass.Int32(M_MAX * SH_KB): + t = q_ // cutlass.Int32(SH_KB) + b = q_ - t * cutlass.Int32(SH_KB) + wv = [ + xb32.load( + idx=t * cutlass.Int32(H // 2) + + b * cutlass.Int32(16) + + cutlass.Int32(4 * z), + vector_size=4, + alignment=16, + ) + for z in range(4) + ] + v = [] + for z in cutlass.range_constexpr(4): + for zz in cutlass.range_constexpr(4): + w = cutlass.Int32(wv[z][zz]) + v.append((w << cutlass.Int32(16)).bitcast(cutlass.Float32)) + v.append((w & cutlass.Int32(-65536)).bitcast(cutlass.Float32)) + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(v[z])) + code, osc = _mx_osc(amax) + words = _mx_words(v, osc) + kt = b >> cutlass.Int32(2) + sl = kt // cutlass.Int32(SH_GT) + it = kt - sl * cutlass.Int32(SH_GT) + n = sl * cutlass.Int32(8) + t + c0 = (b & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BX) + it * cutlass.Int32(SH_NB * 128) + n * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + + (((c0 + cutlass.Int32(1)) ^ (n & cutlass.Int32(7))) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_XC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR, number_of_threads=NQ) + if warp >= 4: + if warp < 8: + qd = warp - cutlass.Int32(4) + # B scales of gate_up to TMEM: column of stage i, lane n = 8 q + t: token t's codes of k-tile SH_GT q + i + n = lane + t = n & cutlass.Int32(7) + ok = (n < cutlass.Int32(SH_NB)) & (t < cutlass.Int32(M_MAX)) + for i in cutlass.range_constexpr(SH_GT): + kt = (n >> cutlass.Int32(3)) * cutlass.Int32(SH_GT) + cutlass.Int32(i) + wsf = big32.load( + idx=( + cutlass.Int32(_SB_XC) + + cutlass.select_(ok, t, cutlass.Int32(0)) * cutlass.Int32(SH_KB) + + kt * cutlass.Int32(4) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_GU + i * 4), + cutlass.select_(ok, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + et = tidx - cutlass.Int32(128) + if et == 0: + prims.mbarrier_arrive(b_gu_rdy) + _stamp(tdbg, bx, 54) + for ui_c in cutlass.range_constexpr( + SH_NU + ): # SH20: this CTA's units, one after the other + if cutlass.Int32(ui_c) < nu: + unit = sid + cutlass.Int32(ui_c * NSHC) + # ---- gate_up out: quadrant qd lane l = MMA row 32 qd + l; slice q = qd / 2 holds token t at + # column 8 q + t + _wait(acc_gu, cutlass.Int32(ui_c)) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if cutlass.const_expr(SH_SPLIT == 2): + av = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_GU + ui_c * SH_NB, + cutlass.Float32, + ), + num=16, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(2): + for tt in cutlass.range_constexpr(8): + bigf.store( + cutlass.Float32(av[8 + tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(2)) * cutlass.Int32(32) + lane) + * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd < cutlass.Int32(2): + # m3_mx_gemv's owner sum: (0 + rank 0) + rank 1, then bf16 + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + (qd * cutlass.Int32(32) + lane) * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + else: + # 16-row octants: warp qd = slice qd, lane l = (u = l / 16 gate / up, position l % 16); its + # tokens are + # accumulator columns 8 qd + t + av = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_GU + ui_c * SH_NB + qd * cutlass.Int32(8), + cutlass.Float32, + ), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + if qd >= cutlass.Int32(1): + for tt in cutlass.range_constexpr(8): + bigf.store( + cutlass.Float32(av[tt]), + idx=cutlass.Int32(_SB_XCH // 4) + + ((qd - cutlass.Int32(1)) * cutlass.Int32(32) + lane) + * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if qd == cutlass.Int32(0): + # m3_mx_gemv's owner sum: (((0 + rank 0) + rank 1) + rank 2) + rank 3, then bf16 + for tt in cutlass.range_constexpr(8): + tot = _add_rn(cutlass.Float32(0.0), cutlass.Float32(av[tt])) + for s_ in cutlass.range_constexpr(1, SH_SPLIT): + tot = _add_rn( + tot, + bigf.load( + idx=cutlass.Int32(_SB_XCH // 4) + + (cutlass.Int32((s_ - 1) * 32) + lane) + * cutlass.Int32(8) + + cutlass.Int32(tt) + ), + ) + bigf.store( + _bf16_round(tot), + idx=cutlass.Int32(_SB_GU // 4) + + lane * cutlass.Int32(8) + + cutlass.Int32(tt), + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + # h = silu_and_mul(gate, up) of the unit's positions SH_ROWS unit + p, under the call's tag + for r in cutlass.range_constexpr((SH_ROWS * M_MAX + 127) // 128): + pq = et + cutlass.Int32(r * 128) + if pq < cutlass.Int32(SH_ROWS * M_MAX): + p = pq // cutlass.Int32(M_MAX) + tt = pq - p * cutlass.Int32(M_MAX) + ga = bigf.load( + idx=cutlass.Int32(_SB_GU // 4) + p * cutlass.Int32(8) + tt + ) + up = bigf.load( + idx=cutlass.Int32(_SB_GU // 4) + + (cutlass.Int32(SH_ROWS) + p) * cutlass.Int32(8) + + tt + ) + q_a = cute.arch.fmin(cutlass.Float32(ga), cutlass.Float32(7.0)) + q_bu = _min_xorsign_abs(cutlass.Float32(up), cutlass.Float32(7.0)) + q_ub = _add_rn(q_bu, cutlass.Float32(1.0)) + q_e2 = _ex2_full( + _mul_rn( + _mul_rn(q_a, cutlass.Float32(-1.70200002193450927734)), + cutlass.Float32(1.4426950216293334961), + ) + ) + q_sg = _div_full( + cutlass.Float32(1.0), _add_rn(q_e2, cutlass.Float32(1.0)) + ) + q_h = _mul_rn(q_ub, _mul_rn(q_sg, q_a)) + _st_relaxed_u32( + hx.subview( + tt * cutlass.Int32(SH_I) + unit * cutlass.Int32(SH_ROWS) + p + ) + .data_ptr() + .toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(q_h), + ) + if cutlass.const_expr(SH_NU > 1): # the next unit reuses the gate / up staging + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if et == 0: + _stamp(tdbg, bx, 50) + # every shared CTA's h: (token, K block) per thread, 32 tagged words, MXFP8 into B (h) and its codes + for rr in cutlass.range_constexpr((M_MAX * SH_DKB + 127) // 128): + q_ = et + cutlass.Int32(rr * 128) + if q_ < cutlass.Int32(M_MAX * SH_DKB): + t = q_ // cutlass.Int32(SH_DKB) + kb = q_ - t * cutlass.Int32(SH_DKB) + base = t * cutlass.Int32(SH_I) + kb * cutlass.Int32(32) + hv = [cutlass.Int32(0) for _ in range(32)] + pend = cutlass.Int32(1) + while pend != cutlass.Int32(0): + for q4 in cutlass.range_constexpr(8): + v4 = prims.load_ext( + hx.subview(base + cutlass.Int32(4 * q4)), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for z in cutlass.range_constexpr(4): + hv[4 * q4 + z] = cutlass.Int32(v4[z]) + okw = cutlass.Boolean(True) + for z in cutlass.range_constexpr(32): + okw = okw & ( + ((hv[z] >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) == tag16 + ) + pend = cutlass.select_(okw, cutlass.Int32(0), cutlass.Int32(1)) + vals = [ + (hv[z] << cutlass.Int32(16)).bitcast(cutlass.Float32) for z in range(32) + ] + amax = cutlass.Float32(0.0) + for z in cutlass.range_constexpr(32): + amax = cute.arch.fmax(amax, cute.math.abs(vals[z])) + code, osc = _mx_osc(amax) + words = _mx_words(vals, osc) + kt = kb >> cutlass.Int32(2) + c0 = (kb & cutlass.Int32(3)) * cutlass.Int32(2) + rw = ( + cutlass.Int32(_SB_BH) + kt * cutlass.Int32(1024) + t * cutlass.Int32(128) + ) >> cutlass.Int32(2) + big32.store( + (words[0], words[1], words[2], words[3]), + idx=rw + ((c0 ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big32.store( + (words[4], words[5], words[6], words[7]), + idx=rw + (((c0 + cutlass.Int32(1)) ^ t) << cutlass.Int32(2)), + alignment=16, + ) + big8.store(cutlass.Int8(code), idx=cutlass.Int32(_SB_HC) + q_) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + okh = lane < cutlass.Int32(M_MAX) + for kt in cutlass.range_constexpr(SH_DKT): + wsf = big32.load( + idx=( + cutlass.Int32(_SB_HC) + + cutlass.select_(okh, lane, cutlass.Int32(0)) * cutlass.Int32(SH_DKB) + + cutlass.Int32(4 * kt) + ) + >> cutlass.Int32(2) + ) + prims.tcgen05_st( + "32x32b", + _tptr(base_row + qd * cutlass.Int32(32), base_col + SH_SFB_DN + kt * 4), + cutlass.select_(okh, cutlass.Int32(wsf), cutlass.Int32(0)), + ) + prims.tcgen05_wait(prims.Tcgen05Wait.STORE) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=SHX_BAR + 1, number_of_threads=128) + if et == 0: + prims.mbarrier_arrive(b_dn_rdy) + _stamp(tdbg, bx, 51) + for ui_c in cutlass.range_constexpr( + SH_NU + ): # SH20: this CTA's units, one after the other + if cutlass.Int32(ui_c) < nu: + unit = sid + cutlass.Int32(ui_c * NSHC) + # ---- down out: tile j row 32 qd + l -> y_sh row 256 unit + 128 j + 32 qd + l + _wait(acc_dn, cutlass.Int32(ui_c)) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for jt in cutlass.range_constexpr(2): + dv = prims.tcgen05_ld( + "32x32b", + _tptr( + base_row + qd * cutlass.Int32(32), + base_col + SH_ACC_DN + ui_c * 16 + jt * 8, + cutlass.Float32, + ), + num=8, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + rowo = ( + unit * cutlass.Int32(256) + + cutlass.Int32(128 * jt) + + qd * cutlass.Int32(32) + + lane + ) + for tt in cutlass.range_constexpr(M_MAX): + _st_relaxed_u32( + dnx.subview(cutlass.Int32(tt * H) + rowo).data_ptr().toint(), + (tag16 << cutlass.Int32(16)) | _bf16_bits(cutlass.Float32(dv[tt])), + ) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if et == 0: + _stamp(tdbg, bx, 52) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if warp == 1: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc(cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), SH_TMEM) + if cutlass.const_expr(not EARLY_EPOCH): + if tidx == 0: + epochs.store(ep + cutlass.Int32(1), idx=bx) + + +@cute.kernel +def m3_moe_kernel( + tma_a1_desc: cutlass.GridConstant[cuda.TensorMap], + tma_a2_desc: cutlass.GridConstant[cuda.TensorMap], + tma_s1_desc: cutlass.GridConstant[cuda.TensorMap], # w3_w1_weight_scale as 128-byte lines + tma_s2_desc: cutlass.GridConstant[cuda.TensorMap], # w2_weight_scale as 128-byte lines + tma_gu_desc: cutlass.GridConstant[ + cuda.TensorMap + ], # SHARED: gate_up e4m3 [1536, 6144], box 128 B x 32 rows + tma_dn_desc: cutlass.GridConstant[ + cuda.TensorMap + ], # SHARED: down e4m3 [6144, 768], box 128 B x 128 rows + logits: cutlass.Array, # f32 [M, 128] (not FUSED_ROUTER) + bias: cutlass.Array, # f32 [128] + x32: cutlass.Array, # int32 words of x_fp4 [M, 3072 B] (not FUSED_ROUTER) + xsf32: cutlass.Array, # int32 words of x_sf [M, 384 B] (not FUSED_ROUTER) + escal: cutlass.Array, # f32 [6, 128]: fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha + plan: cutlass.Array, # int32 [4 M + 1, NCTA, NG_CAP + 1] by G: per group i0 | i1 << 16, then b0 | nb << 16 + partials: cutlass.Array, # int32 bits of f32 [G_CAP * 48, M, 32]: split units' head sums + hbuf32: cutlass.Array, # int32 [G_CAP * M, 48, 4]: tagged intermediate records + epochs: cutlass.Array, # int32 [NCTA] + out: cutlass.Array, # bf16 [M, 6144] + route_dbg: cutlass.Array, # int32 [M, 8]: ids, weight bits (CTA 0) + tdbg: cutlass.Array, # int64 [NCTA, 32] timing stamps (TIMING builds) + xb32: cutlass.Array, # FUSED_ROUTER: int32 words (bf16 pairs) of the hidden states [M, 6144] + rw: cutlass.Array, # FUSED_ROUTER: f32 router weight [128, 6144] + sfsc: cutlass.Array, # FUSED_ROUTER: f32 [1] fc31_input_scale + lgbuf: cutlass.Array, # FUSED_ROUTER: int64 [LG_REPL, LG_STRIDE] logits: tag << 32 | f32 bits + guw: cutlass.Array, # SHARED: uint8 gate_up e4m3 [1536, 6144] + gusf: cutlass.Array, # SHARED: uint8 its swizzled UE8M0 scales + dnw: cutlass.Array, # SHARED: uint8 down e4m3 [6144, 768] + dnsf: cutlass.Array, # SHARED: uint8 its swizzled UE8M0 scales + hx: cutlass.Array, # SHARED: int32 [M, 768]: h as (call tag << 16 | bf16), tag 0 = never written + dnx: cutlass.Array, # SHARED: int32 [M, 6144]: y_sh as (call tag << 16 | bf16), tag 0 = never written + arp: cutlass.Array, # int64 [8]: all-reduce push (enable, mc, flags, rank, world, maxt); enable 0 = off + s1w: cutlass.Array, # SFA_CP: int32 words of w3_w1_weight_scale + s2w: cutlass.Array, # SFA_CP: int32 words of w2_weight_scale + img32: cutlass.Array, # B2IMG: int32 [NG_CAP, IMG_BYTES / 4] per-group FC2 B image (tile + scale atoms) + b2cnt: cutlass.Array, # B2IMG: int32 [NG_CAP * 8 + 8] per-group unit counters (index 8 g) +): + tidx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp = cute.arch.make_warp_uniform(cute.arch.warp_idx()) + lane = tidx % 32 + if tidx == 0: + _stamp(tdbg, bx, 0) + if cutlass.const_expr(TIMING): + if tidx == 0: + tdbg.store( + cutlass.Int64(_smid()), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(63) + ) + + s_big = cutlass.Array(cutlass.Int8, BIG, space=cutlass.AddressSpace.smem, alignment=1024) + sA = cutlass.Array( + s_big.data_ptr(0), shape=(NUM_BYTES_A * STAGES,), dtype=cutlass.Int8, alignment=1024 + ) + sSFAraw = cutlass.Array( + s_big.data_ptr(_OFF_SFAR), + shape=(max(1, _RAW_STAGE // 4) * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sSFA = cutlass.Array( + s_big.data_ptr(_OFF_SFA), + shape=(NUM_WORDS_SFA * STAGES,), + dtype=cutlass.Int32, + alignment=1024, + ) + sB1 = cutlass.Array( + s_big.data_ptr(_OFF_B1), shape=(NUM_BYTES_B1 * S1,), dtype=cutlass.Int8, alignment=1024 + ) + sB2 = cutlass.Array( + s_big.data_ptr(_OFF_B2), shape=(2 * KT2 * NUM_BYTES_B2,), dtype=cutlass.Int8, alignment=1024 + ) + sSFB2 = cutlass.Array( + s_big.data_ptr(_OFF_SFB2), + shape=(2 * KT2 * NUM_KB * 128,), + dtype=cutlass.Int32, + alignment=1024, + ) + ys = cutlass.Array( + cutlass.Float32, NB2_CAP * M_MAX * TOP_K * R2, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_part = cutlass.Array( + cutlass.Float32, 2 * M_MAX * C1 * R1, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_bias = cutlass.Array(cutlass.Float32, E_LOCAL, space=cutlass.AddressSpace.smem, alignment=16) + s_red = cutlass.Array(cutlass.Float32, 9 * M_MAX, space=cutlass.AddressSpace.smem, alignment=16) + s_xsf = cutlass.Array( + cutlass.Int32, + M_MAX * (H // 64) if FUSED_ROUTER else 4, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + ab_full = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + ab_empty = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + sfa_ready = cutlass.Array(cutlass.Int64, STAGES, space=cutlass.AddressSpace.smem, alignment=8) + acc_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_full = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + acc2_empty = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_ready = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + b2_free = cutlass.Array(cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8) + # this CTA's FC1 share of group g is out: one barrier per group parity (a single barrier could complete twice + # while warp 5 still works on an earlier group, and its parity wait would then miss them) + fc1_done = cutlass.Array(cutlass.Int64, 4, space=cutlass.AddressSpace.smem, alignment=8) + b1_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + x_bar = cutlass.Array( + cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8 + ) # x landed, x consumed + img_bar = cutlass.Array( + cutlass.Int64, 2, space=cutlass.AddressSpace.smem, alignment=8 + ) # (B2IMG) image landed + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + s_el = cutlass.Array( + cutlass.Int32, G_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16 + ) # slots, then G + s_kpos = cutlass.Array( + cutlass.Int32, G_CAP * M_MAX, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_w = cutlass.Array( + cutlass.Float32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_ids = cutlass.Array( + cutlass.Int32, M_MAX * TOP_K, space=cutlass.AddressSpace.smem, alignment=16 + ) + s_plan = cutlass.Array(cutlass.Int32, NG_CAP + 1, space=cutlass.AddressSpace.smem, alignment=16) + # this CTA's plan rows for every G (static data: loaded before the grid dependency) + s_misc = cutlass.Array(cutlass.Int32, 4, space=cutlass.AddressSpace.smem, alignment=16) + s_plan_all = cutlass.Array( + cutlass.Int32, (G_CAP + 1) * (NG_CAP + 1), space=cutlass.AddressSpace.smem, alignment=16 + ) + s_esc = cutlass.Array( + cutlass.Float32, 8 * G_CAP, space=cutlass.AddressSpace.smem, alignment=32 + ) # slot s at 8 s + + is_routed = cutlass.Boolean(True) + if cutlass.const_expr(SHARED): + sh_full = cutlass.Array(cutlass.Int64, 32, space=cutlass.AddressSpace.smem, alignment=8) + is_routed = bx < cutlass.Int32(NCTA) + if bx >= cutlass.Int32(NCTA): + _shared_cta( + tidx, + warp, + lane, + bx - cutlass.Int32(NCTA), + s_big, + sh_full, + epochs, + tdbg, + bx, + xb32, + tma_gu_desc.get_ptr(), + tma_dn_desc.get_ptr(), + gusf, + dnsf, + hx, + dnx, + arp, + tmem_ptr_i32, + guw, + dnw, + ) + if is_routed: + # ---- before the grid dependency + if warp == 0: + if tidx < STAGES: + prims.mbarrier_init(ab_full.subview(tidx), 33 if SFA_CP else 1) + prims.mbarrier_init(ab_empty.subview(tidx), 1) + prims.mbarrier_init(sfa_ready.subview(tidx), 1) + if tidx < 2: + prims.mbarrier_init(acc_full.subview(tidx), 1) + prims.mbarrier_init(acc_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(acc2_full.subview(tidx), 1) + prims.mbarrier_init(acc2_empty.subview(tidx), EPI_THREADS) + prims.mbarrier_init(b2_ready.subview(tidx), 1) + prims.mbarrier_init(b2_free.subview(tidx), EPI_THREADS) + if cutlass.const_expr(B2IMG): + prims.mbarrier_init(img_bar.subview(tidx), 1) + if tidx < 4: + prims.mbarrier_init(fc1_done.subview(tidx), EPI_THREADS) + if tidx == 0: + prims.mbarrier_init(b1_ready.subview(0), EPI_THREADS) + if cutlass.const_expr(XSTAGE): + prims.mbarrier_init(x_bar.subview(0), 1) + prims.mbarrier_init(x_bar.subview(1), EPI_THREADS) + if warp == MMA_WARP: + prims.tcgen05_alloc(tmem_ptr_i32, TMEM_COLS) + prims.tcgen05_relinquish_alloc_permit() + if warp == W_WARP: + prims.prefetch_tensormap(tma_a1_desc.get_ptr()) + prims.prefetch_tensormap(tma_a2_desc.get_ptr()) + prims.prefetch_tensormap(tma_s1_desc.get_ptr()) + prims.prefetch_tensormap(tma_s2_desc.get_ptr()) + for i in cutlass.range_constexpr(E_LOCAL // 32): + s_bias.store(cutlass.Float32(bias.load(idx=lane + i * 32)), idx=lane + i * 32) + for i in cutlass.range_constexpr(((G_CAP + 1) * (NG_CAP + 1) + 31) // 32): + q = lane + cutlass.Int32(32 * i) + if q < cutlass.Int32((G_CAP + 1) * (NG_CAP + 1)): + gq = q // cutlass.Int32(NG_CAP + 1) + s_plan_all.store( + plan.load( + idx=(gq * cutlass.Int32(NCTA) + bx) * cutlass.Int32(NG_CAP + 1) + + q + - gq * cutlass.Int32(NG_CAP + 1) + ), + idx=q, + ) + rwv = [] + if cutlass.const_expr(FUSED_ROUTER): + tg = cutlass.select_(tidx < cutlass.Int32(GEMV_THREADS), tidx, cutlass.Int32(0)) + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + rbx = cutlass.select_(eg < cutlass.Int32(NROUTE), eg, cutlass.Int32(0)) + rwv.append( + [ + rw.load( + idx=rbx * cutlass.Int32(H) + + tg * cutlass.Int32(4) + + cutlass.Int32(1024 * j), + vector_size=4, + alignment=16, + ) + for j in range(H // 1024) + ] + ) + if cutlass.const_expr(TIMING): + if tidx == 0: + s_el.store(cutlass.Int32(-7), idx=G_CAP) + prims.fence_mbarrier_init() + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + + cute.arch.griddepcontrol_wait() + if cutlass.const_expr(XSTAGE): + if warp == 0: + if lane == 0: + prims.mbarrier_arrive_expect_tx(x_bar.subview(0), M_MAX * H * 2) + cute.arch.sync_warp() + if lane < M_MAX: + prims.cp_async_bulk_shared_cluster_global( + s_big.subview(lane * cutlass.Int32(H * 2)), + xb32.subview(lane * cutlass.Int32(H // 2)), + x_bar.subview(0), + H * 2, + ) + ar_on = arp.load(idx=0) != cutlass.Int64(0) + ar_off = arp.load(idx=0) == cutlass.Int64(0) + if cutlass.const_expr(not LATE_TRIGGER): + if ar_off: + cute.arch.griddepcontrol_launch_dependents() + if cutlass.const_expr(EARLY_AR): + if ar_on: + cute.arch.griddepcontrol_launch_dependents() + if tidx == 0: + _stamp(tdbg, bx, 1) + + ep = epochs.load(idx=bx) + tagv = ((ep & cutlass.Int32(0x7F)) | cutlass.Int32(0x80)) << cutlass.Int32(24) + tag16 = (ep & cutlass.Int32(0x7FFF)) | cutlass.Int32(0x8000) + ltag = cutlass.Int64(ep + cutlass.Int32(1)) << cutlass.Int64(32) + if cutlass.const_expr(FUSED_ROUTER): + # ---- router GEMV (experts bx, bx + NCTA, .., threads 0..255), the minimax_m3_router_gemv op's order + for rg in cutlass.range_constexpr(NGEMV): + eg = bx + cutlass.Int32(rg * NCTA) + if (eg < cutlass.Int32(NROUTE)) & (tidx < cutlass.Int32(GEMV_THREADS)): + accs = [cutlass.Float32(0.0) for _ in range(M_MAX)] + xsrc = xb32 + if cutlass.const_expr(GEMV_SMEM and XSTAGE): + _wait(x_bar.subview(0), 0) + xsrc = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + for j in cutlass.range_constexpr(H // 1024): + for t in cutlass.range_constexpr(M_MAX): + xw = xsrc.load( + idx=cutlass.Int32(t * (H // 2) + 512 * j) + tidx * cutlass.Int32(2), + vector_size=2, + alignment=8, + ) + x0 = cutlass.Int32(xw[0]) + x1 = cutlass.Int32(xw[1]) + a = accs[t] + a = _fma_rn( + (x0 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][0]), + a, + ) + a = _fma_rn( + (x0 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][1]), + a, + ) + a = _fma_rn( + (x1 << cutlass.Int32(16)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][2]), + a, + ) + a = _fma_rn( + (x1 & cutlass.Int32(-65536)).bitcast(cutlass.Float32), + cutlass.Float32(rwv[rg][j][3]), + a, + ) + accs[t] = a + for t in cutlass.range_constexpr(M_MAX): + v = accs[t] + for off in (16, 8, 4, 2, 1): + v = _add_rn(v, cute.arch.shuffle_sync_bfly(v, off)) + if lane == 0: + s_red.store(v, idx=warp * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < M_MAX: + sacc = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(8): + sacc = _add_rn(sacc, s_red.load(idx=cutlass.Int32(w * M_MAX) + tidx)) + s_red.store(sacc, idx=cutlass.Int32(8 * M_MAX) + tidx) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS) + if tidx < LG_REPL: + for t in cutlass.range_constexpr(M_MAX): + lw = ltag | ( + cutlass.Int64( + cutlass.Float32(s_red.load(idx=8 * M_MAX + t)).bitcast( + cutlass.Int32 + ) + ) + & cutlass.Int64(0xFFFFFFFF) + ) + _st_relaxed_u64( + lgbuf.subview( + tidx * cutlass.Int32(LG_STRIDE) + + cutlass.Int32(t * E_LOCAL) + + eg + ) + .data_ptr() + .toint(), + lw, + ) + if cutlass.const_expr(NGEMV > 1): + cute.arch.sync_warp() + cute.arch.barrier( + barrier_id=GEMV_BAR, number_of_threads=GEMV_THREADS + ) # s_red reused next + if tidx == 0: + _stamp(tdbg, bx, 21) + + # ---- routing (warps 4..7, token t on warp 4 + t % 4); the epilogue warps build B meanwhile + if ((warp >= W_WARP) & (warp < cutlass.Int32(W_WARP + NRW0))) | ( + (warp >= cutlass.Int32(8)) & (warp < cutlass.Int32(8 + RW_EXTRA)) + ): + rw_i = cutlass.select_( + warp < cutlass.Int32(8), warp - cutlass.Int32(W_WARP), warp - cutlass.Int32(4) + ) + # ILP: poll every token's logits first, then run the independent top-k chains back to back + lgv_all = [] + t_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = rw_i + cutlass.Int32(NRW * tr) + if cutlass.const_expr(M_MAX % NRW != 0): + t = cutlass.select_(t < cutlass.Int32(M_MAX), t, cutlass.Int32(M_MAX - 1)) + t_all.append(t) + lgv = [] + if cutlass.const_expr(FUSED_ROUTER): + want = ltag >> cutlass.Int64(32) + rbase = ( + (bx % cutlass.Int32(LG_REPL)) * cutlass.Int32(LG_STRIDE) + + t * cutlass.Int32(E_LOCAL) + + lane + ) + done = cutlass.Int32(0) + res = [cutlass.Int64(0) for _ in range(4)] + while done == cutlass.Int32(0): + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(4): + res[z] = _ld_relaxed_u64( + lgbuf.subview(cutlass.Int32(z * 32) + rbase).data_ptr().toint() + ) + ok = ok & ((res[z] >> cutlass.Int64(32)) == want) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32( + -1 + ): + done = cutlass.Int32(1) + else: + _nanosleep(POLL_NS) + for z in cutlass.range_constexpr(4): + lgv.append(cutlass.Int32(res[z]).bitcast(cutlass.Float32)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 22) + else: + for i in cutlass.range_constexpr(4): + lgv.append( + cutlass.Float32( + logits.load( + idx=t * cutlass.Int32(E_LOCAL) + cutlass.Int32(i * 32) + lane + ) + ) + ) + lgv_all.append(lgv) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 61) + res_all = [] + for tr in cutlass.range_constexpr((M_MAX + NRW - 1) // NRW): + t = t_all[tr] + lgv = lgv_all[tr] + own = (rw_i + cutlass.Int32(NRW * tr)) < cutlass.Int32(M_MAX) + # top-4 (keys sigmoid + bias, ties to the lower id) + keys = [] + for i in cutlass.range_constexpr(4): + sg = _fma_rn( + cutlass.Float32(0.5), + cute.math.tanh(_mul_rn(cutlass.Float32(0.5), lgv[i])), + cutlass.Float32(0.5), + ) + keys.append(_add_rn(sg, s_bias.load(idx=i * 32 + lane))) + hi = [_twiddle(keys[i]) for i in range(4)] + taken = [cutlass.Boolean(False) for _ in range(4)] + ids_t = [] + kt_t = [] + for k in cutlass.range_constexpr(TOP_K): + bh = cutlass.Int32(0) + bl = cutlass.Int32(0) + for i in cutlass.range_constexpr(4): + lo_i = cutlass.Int32(65535 - i * 32) - lane + hv = cutlass.select_(taken[i], cutlass.Int32(0), hi[i]) + lv = cutlass.select_(taken[i], cutlass.Int32(0), lo_i) + better = (cutlass.Uint32(hv) > cutlass.Uint32(bh)) | ( + (hv == bh) & (cutlass.Uint32(lv) > cutlass.Uint32(bl)) + ) + bh = cutlass.select_(better, hv, bh) + bl = cutlass.select_(better, lv, bl) + mh = _redux_max_u32(bh) + ml = _redux_max_u32(cutlass.select_(bh == mh, bl, cutlass.Int32(0))) + e_k = cutlass.Int32(65535) - ml + for i in cutlass.range_constexpr(4): + taken[i] = taken[i] | (e_k == cutlass.Int32(i * 32) + lane) + ids_t.append(e_k) + kv = cutlass.Float32(0.0) + for i in cutlass.range_constexpr(4): + kv = cutlass.select_(e_k == cutlass.Int32(i * 32) + lane, keys[i], kv) + kt_t.append(cute.arch.shuffle_sync(kv, e_k % cutlass.Int32(32))) + sig = cutlass.Float32(0.0) + e_l = cutlass.Int32(0) + for k in cutlass.range_constexpr(TOP_K): + sk = _sub_rn(kt_t[k], s_bias.load(idx=ids_t[k])) + sig = cutlass.select_(lane == cutlass.Int32(k), sk, sig) + e_l = cutlass.select_(lane == cutlass.Int32(k), ids_t[k], e_l) + total = sig + for off in (16, 8, 4, 2, 1): + total = _add_rn(total, cute.arch.shuffle_sync_bfly(total, off)) + wv = _bf16_round( + _div_rn( + _mul_rn(sig, cutlass.Float32(ROUTE_SCALE)), + _add_rn(total, cutlass.Float32(SUM_EPS)), + ) + ) + res_all.append((t, own, wv, e_l)) + if (lane == 0) & (warp == cutlass.Int32(W_WARP)): + _stamp(tdbg, bx, 62) + for tr in cutlass.range_constexpr( + (M_MAX + NRW - 1) // NRW + ): # stores after every chain (one branch-free block) + t, own, wv, e_l = res_all[tr] + if (lane < TOP_K) & own: + s_w.store(wv, idx=t * cutlass.Int32(TOP_K) + lane) + s_ids.store(e_l, idx=t * cutlass.Int32(TOP_K) + lane) + if bx == 0: + route_dbg.store( + wv.bitcast(cutlass.Int32), + idx=t * cutlass.Int32(8) + cutlass.Int32(4) + lane, + ) + cute.arch.sync_warp() + if cutlass.const_expr(EARLY0 and not E0_OFFLOAD): + if warp == cutlass.Int32(W_WARP): + # slots 0..3 are always token 0's four (distinct) experts and group 0's stage ranges are the same + # for every G (the plan guarantees it): fill the ring with this CTA's first group-0 stages now + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(W_WARP): + _bar_arrive( + E0_BAR, 64 + ) # token 0's experts are in s_ids: warp 7 issues the early stages + if cutlass.const_expr(B1_AFTER_TOPK): + _bar_arrive(TOPK2_BAR, 128 + 32 * NRW) + cute.arch.barrier(barrier_id=TOPK_BAR, number_of_threads=32 * NRW) + if cutlass.const_expr(E0_OFFLOAD): + if warp == cutlass.Int32(E0_WARP): + cute.arch.barrier(barrier_id=E0_BAR, number_of_threads=64) + # slots 0..3 are always token 0's four (distinct) experts and group 0's stage ranges are the same + # for every G (the plan guarantees it): fill the ring with this CTA's first group-0 stages now + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) + if lane == 0: + _stamp(tdbg, bx, 56) + n0 = cutlass.Int32(0) + w0 = s_plan_all.load(idx=G_CAP * (NG_CAP + 1)) + ei0 = w0 & cutlass.Int32(0xFFFF) + ei1 = w0 >> cutlass.Int32(16) + for k in range(_nsegs(ei0, ei1)): + ug, j0, j1 = _seg(ei0, ei1, k) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_ids.load(idx=ug // cutlass.Int32(U1)) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if n0 < cutlass.Int32(min(STAGES, EARLY0_N)): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(n0), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + n0 * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(n0), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview(n0 * cutlass.Int32(NUM_WORDS_SFA_RAW)), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(n0), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, n0, sf_line, c, sSFA, s1w, ab_full) + n0 = n0 + cutlass.Int32(1) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if lane == 0: + s_misc.store(n0, idx=0) + cute.arch.sync_warp() + _bar_arrive(E1_BAR, 64) + if warp == W_WARP: + if lane == 0: + _stamp(tdbg, bx, 20) + # slots: lane l < 4 M is (token l / 4, top-k l % 4); the distinct experts in lane order (match.any) + valid = lane < cutlass.Int32(TOP_K * M_MAX) + my_e = cutlass.select_( + valid, + s_ids.load(idx=cutlass.select_(valid, lane, cutlass.Int32(0))), + cutlass.Int32(-1) - lane, + ) + same = _match_any(my_e) + leader = cute.arch.popc((same & (cutlass.Int32(0) - same)) - cutlass.Int32(1)) + first = valid & (lane == leader) + bal = prims.vote_sync(0xFFFFFFFF, first, prims.VoteSync.BALLOT) + rank = cute.arch.popc(bal & cutlass.Int32(cute.arch.lanemask_lt())) + slot_l = cute.arch.shuffle_sync(rank, leader) + for z in cutlass.range_constexpr((G_CAP * M_MAX + 31) // 32): + if lane + cutlass.Int32(32 * z) < cutlass.Int32(G_CAP * M_MAX): + s_kpos.store(cutlass.Int32(-1), idx=lane + cutlass.Int32(32 * z)) + cute.arch.sync_warp() + if first: + s_el.store(my_e, idx=rank) + if valid: + s_kpos.store( + lane % cutlass.Int32(TOP_K), + idx=slot_l * cutlass.Int32(M_MAX) + lane // cutlass.Int32(TOP_K), + ) + if bx == 0: + route_dbg.store( + my_e, + idx=(lane // cutlass.Int32(TOP_K)) * cutlass.Int32(8) + + lane % cutlass.Int32(TOP_K), + ) + gcnt = cute.arch.popc(bal) + if lane == 0: + s_el.store(gcnt, idx=G_CAP) + if lane < cutlass.Int32(NG_CAP + 1): # the plan row of this G (preloaded) + s_plan.store(s_plan_all.load(idx=gcnt * cutlass.Int32(NG_CAP + 1) + lane), idx=lane) + cute.arch.sync_warp() + if cutlass.const_expr(B2IMG): + # CTA 0 pads every group counter to exactly CNT_PER_CALL for this call (absent slots, unused groups) + if (bx == 0) & (lane < cutlass.Int32(NG_CAP)): + nh = gcnt - lane * cutlass.Int32(4) + nh = cutlass.select_(nh > cutlass.Int32(4), cutlass.Int32(4), nh) + nh = cutlass.select_(nh < cutlass.Int32(0), cutlass.Int32(0), nh) + pad = cutlass.Int32(CNT_PER_CALL) - nh * cutlass.Int32(UNITS_ROW) + if pad > cutlass.Int32(0): + _red_release_add( + b2cnt.subview(lane * cutlass.Int32(8)).data_ptr().toint(), pad + ) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 24) + cute.arch.sync_warp() + _bar_arrive(ROUTE_BAR, THREADS) + if warp < 4: + if cutlass.const_expr(B1_AFTER_TOPK): + cute.arch.barrier(barrier_id=TOPK2_BAR, number_of_threads=128 + 32 * NRW) + if cutlass.const_expr(FUSED_ROUTER): + if tidx == 0: + _stamp(tdbg, bx, 57) + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(0), 0) + if tidx == 0: + _stamp(tdbg, bx, 58) + xs32 = cutlass.Array( + s_big.data_ptr(0), + shape=(M_MAX * H // 2,), + dtype=cutlass.Int32, + alignment=16, + ) + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xs32, sfsc, sB1, s_xsf, b1_ready + ) + prims.mbarrier_arrive(x_bar.subview(1)) # the ring is free for the weights + else: + _build_b1_fused( + tidx, warp, lane, base_row, base_col, xb32, sfsc, sB1, s_xsf, b1_ready + ) + if tidx == 0: + _stamp(tdbg, bx, 23) + else: + _build_b1(tidx, warp, lane, base_row, base_col, x32, xsf32, sB1, b1_ready) + if warp != W_WARP: # one sync site for every thread but warp 4's (which arrived) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=ROUTE_BAR, number_of_threads=THREADS) + if cutlass.const_expr(EARLY_EPOCH): + if tidx == 0: + epochs.store( + ep + cutlass.Int32(1), idx=bx + ) # every thread read it before the route barrier + if tidx == 0: + _stamp(tdbg, bx, 2) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(s_el.load(idx=G_CAP)), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(28), + ) + + g_n = s_el.load(idx=G_CAP) + ng = (g_n + cutlass.Int32(3)) >> cutlass.Int32(2) + pr2 = s_plan.load(idx=NG_CAP) + blk0 = pr2 & cutlass.Int32(0xFFFF) + nb = pr2 >> cutlass.Int32(16) + + # ---- weights producer (warp 4): phase p = FC1 stages of group p, then FC2 tiles of group p - LAG + if warp == W_WARP: + if cutlass.const_expr(XSTAGE): + _wait(x_bar.subview(1), 0) # the staged hidden states are quantized + stage = cutlass.Int32(0) + phase = cutlass.Int32(1) + skip = cutlass.Int32(0) + if cutlass.const_expr(E0_OFFLOAD): + cute.arch.barrier(barrier_id=E1_BAR, number_of_threads=64) + if cutlass.const_expr(EARLY0): + skip = s_misc.load(idx=0) # group-0 stages already in the ring + stage = cutlass.select_(skip == cutlass.Int32(STAGES), cutlass.Int32(0), skip) + phase = cutlass.select_( + skip == cutlass.Int32(STAGES), cutlass.Int32(0), cutlass.Int32(1) + ) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + c = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + e = s_el.load(idx=slot) + sf_line = ( + (e * cutlass.Int32(SF1_RB) + (c >> cutlass.Int32(2))) + * cutlass.Int32(SF1_KA) + + j0 * cutlass.Int32(C1 * NUM_KB) + ) * cutlass.Int32(4) + kx = j0 * cutlass.Int32(C1 * KT_BYTES) + row = c * cutlass.Int32(R1) + for j in cutlass.range(j0, j1, unroll=1): + if skip > cutlass.Int32(0): + skip = skip - cutlass.Int32(1) + else: + _waitd(ab_empty.subview(stage), phase, tdbg, 1) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + NUM_BYTES_A if SFA_CP else NUM_BYTES_A + NUM_BYTES_SFA_RAW, + ) + for q in cutlass.range_constexpr(C1): + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(q * NUM_BYTES_BOX1) + ), + tma_a1_desc.get_ptr(), + (kx + cutlass.Int32(q * KT_BYTES), row, e), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + ), + tma_s1_desc.get_ptr(), + (cutlass.Int32(0), sf_line), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf1_cp(lane, stage, sf_line, c, sSFA, s1w, ab_full) + stage, phase = _adv(stage, phase) + kx = kx + cutlass.Int32(C1 * KT_BYTES) + sf_line = sf_line + cutlass.Int32(NUM_BYTES_SFA_RAW // 128) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + for bb in range(nb): + blk = blk0 + bb + for t2 in cutlass.range_constexpr(KT2): + _waitd(ab_empty.subview(stage), phase, tdbg, 2) + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx( + ab_full.subview(stage), + n_here + * cutlass.Int32( + NUM_BYTES_BOX1 + if SFA_CP + else NUM_BYTES_BOX1 + NUM_BYTES_SFA_RAW // 4 + ), + ) + for jj in cutlass.range_constexpr(4): + if cutlass.Int32(jj) < n_here: + e = s_el.load(idx=g * cutlass.Int32(4) + cutlass.Int32(jj)) + prims.cp_async_bulk_tensor_shared_cta_global( + sA.subview( + stage * cutlass.Int32(NUM_BYTES_A) + + cutlass.Int32(jj * NUM_BYTES_BOX1) + ), + tma_a2_desc.get_ptr(), + ( + cutlass.Int32(t2 * KT_BYTES), + blk * cutlass.Int32(R2), + e, + ), + ab_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + if cutlass.const_expr(not SFA_CP): + prims.cp_async_bulk_tensor_shared_cta_global( + sSFAraw.subview( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + cutlass.Int32(jj * NUM_WORDS_SFA_RAW // 4) + ), + tma_s2_desc.get_ptr(), + ( + cutlass.Int32(0), + ( + ( + e * cutlass.Int32(SF2_RB) + + (blk >> cutlass.Int32(2)) + ) + * cutlass.Int32(SF2_KA) + + cutlass.Int32(4 * t2) + ) + * cutlass.Int32(4), + ), + ab_full.subview(stage), + ) + if cutlass.const_expr(SFA_CP): + _sf2_cp(lane, stage, g, blk, t2, n_here, s_el, sSFA, s2w, ab_full) + stage, phase = _adv(stage, phase) + if lane == 0: + _stamp(tdbg, bx, 3) + if cutlass.const_expr(TAIL_TRIGGER): + if ar_on: + cute.arch.griddepcontrol_launch_dependents() + + # ---- scale splicer (warp 6): every stage, atom kb word [r][q] = raw atom 4 q + kb word [r][m1] + if (warp == S_WARP) & cutlass.Boolean(not SFA_CP): # (SFA_CP: no splicer) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + for p in range(ng + cutlass.Int32(LAG)): + i0 = cutlass.Int32(0) + i1 = cutlass.Int32(0) + for part in cutlass.range_constexpr(2): + cnt = cutlass.Int32(0) + m1 = cutlass.Int32(0) + nk = cutlass.Int32(0) + if cutlass.const_expr(part == 0): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + nk = _nsegs(i0, i1) + else: + if p >= cutlass.Int32(LAG): + nk = nb + for k in range(nk): + if cutlass.const_expr(part == 0): + ug, j0, j1 = _seg(i0, i1, k) + m1 = ( + ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + ) & cutlass.Int32(3) + cnt = j1 - j0 + else: + m1 = (blk0 + k) & cutlass.Int32(3) + cnt = cutlass.Int32(KT2) + for _j in cutlass.range(cnt, unroll=1): + _waitd(ab_full.subview(stage), phase, tdbg, 3) + raw0 = ( + stage * cutlass.Int32(NUM_WORDS_SFA_RAW) + + lane * cutlass.Int32(4) + + m1 + ) + dst0 = stage * cutlass.Int32(NUM_WORDS_SFA) + lane * cutlass.Int32(4) + for kb in cutlass.range_constexpr(NUM_KB): + w0 = sSFAraw.load(idx=raw0 + cutlass.Int32((0 * 4 + kb) * 128)) + w1 = sSFAraw.load(idx=raw0 + cutlass.Int32((1 * 4 + kb) * 128)) + w2 = sSFAraw.load(idx=raw0 + cutlass.Int32((2 * 4 + kb) * 128)) + w3 = sSFAraw.load(idx=raw0 + cutlass.Int32((3 * 4 + kb) * 128)) + sSFA.store( + ( + cutlass.Int32(w0), + cutlass.Int32(w1), + cutlass.Int32(w2), + cutlass.Int32(w3), + ), + idx=dst0 + cutlass.Int32(kb * 128), + alignment=16, + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + cute.arch.sync_warp() + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFA.subview(stage * cutlass.Int32(NUM_WORDS_SFA)), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFA1_COL + + stage * cutlass.Int32(16) + + kb * 4, + ), + sd0 + (kb * 512 >> 4), + multicast=s2t_mc, + ) + prims.tcgen05_commit(sfa_ready.subview(stage)) + stage, phase = _adv(stage, phase) + + if cutlass.const_expr(B2IMG): + # ---- (B2IMG) group releases (warp 6): this CTA's finished units of group p, one release add per group + if warp == cutlass.Int32(S_WARP): + for p in range(ng): + i0, i1 = _plan_range(s_plan, p) + n_done = cutlass.Int32(0) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + n_done = n_done + cutlass.select_( + (j0 != cutlass.Int32(0)) | (j1 == cutlass.Int32(S1)), + cutlass.Int32(1), + cutlass.Int32(0), + ) + _wait( + fc1_done.subview(p & cutlass.Int32(3)), + (p >> cutlass.Int32(2)) & cutlass.Int32(1), + ) + if cutlass.const_expr(B2_DLY > 0): + t0d = _gtime() + dl = cutlass.Int64( + ( + (bx * cutlass.Int32(37) + p * cutlass.Int32(11) + ep) + & cutlass.Int32(63) + ) + * cutlass.Int32(B2_DLY // 64) + ) + while _gtime() - t0d < dl: + pass + if (lane == 0) & (n_done > cutlass.Int32(0)): + _fence_proxy_async_global() + _red_release_add( + b2cnt.subview(p * cutlass.Int32(8)).data_ptr().toint(), n_done + ) + cute.arch.sync_warp() + # ---- (B2IMG) FC2 B loader (warp 5): group g's image once its counter is complete + if warp == cutlass.Int32(X_WARP): + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + cnt_target = (ep + cutlass.Int32(1)) * cutlass.Int32(CNT_PER_CALL) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + ((g >> cutlass.Int32(1)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + if lane == 0: + _stamp(tdbg, bx, 29) + caddr = b2cnt.subview(g * cutlass.Int32(8)).data_ptr().toint() + if lane == 0: + cv = _ld_acquire_u32(caddr) + while (cv - cnt_target) < cutlass.Int32(0): + _nanosleep(B2IMG_NS) + cv = _ld_acquire_u32(caddr) + _fence_proxy_async_global() + prims.mbarrier_arrive_expect_tx(img_bar.subview(buf), IMG_BYTES) + prims.cp_async_bulk_shared_cluster_global( + sB2.subview(buf * cutlass.Int32(IMG_B2)), + img32.subview(g * cutlass.Int32(IMG_BYTES // 4)), + img_bar.subview(buf), + IMG_B2, + ) + prims.cp_async_bulk_shared_cluster_global( + sSFB2.subview(buf * cutlass.Int32(IMG_SF // 4)), + img32.subview( + g * cutlass.Int32(IMG_BYTES // 4) + cutlass.Int32(IMG_B2 // 4) + ), + img_bar.subview(buf), + IMG_SF, + ) + cute.arch.sync_warp() + _waitd( + img_bar.subview(buf), (g >> cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 108 + ) + if prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + cute.arch.sync_warp() + if lane == 0: + _stamp(tdbg, bx, 25) + # ---- FC2 B builder (warps 5, 8 ..): group g's records (all 48 per (slot, token), the call's tag on + # every word) into B2 buffer g % 2 (row j M + t, k-tile t2 = record / 16) and its scale atoms, then + # tcgen05.cp to TMEM. + if ((warp == X_WARP) | (warp >= cutlass.Int32(8))) & cutlass.Boolean(not B2IMG): + xi = cutlass.select_( + warp == cutlass.Int32(X_WARP), cutlass.Int32(0), warp - cutlass.Int32(7) + ) + if cutlass.const_expr(G_CAP > TOP_K): + if (bx == 0) & (xi == cutlass.Int32(0)): + # rows of slots this call does not use: tag 0, never taken for a later call's + for i in range( + g_n * cutlass.Int32(M_MAX * UNITS_ROW) + lane, + cutlass.Int32(G_CAP * M_MAX * UNITS_ROW), + 32, + ): + for rr in cutlass.range_constexpr(HB_REPL): + hbuf32.store( + ( + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + cutlass.Int32(0), + ), + idx=i * cutlass.Int32(4) + + (cutlass.Int32(rr * HB_STRIDE) if rr else 0), + alignment=16, + ) + s2t_shape, s2t_mc = prims.S2TCopyMode.S2T_32x128b_WARPX4 + sSFB2_8 = cutlass.Array( + sSFB2.data_ptr(0), shape=(2 * KT2 * NUM_KB * 512,), dtype=cutlass.Int8, alignment=16 + ) + sB2_32 = cutlass.Array( + sB2.data_ptr(0), + shape=(2 * KT2 * NUM_BYTES_B2 // 4,), + dtype=cutlass.Int32, + alignment=16, + ) + nrec_cap = 4 * M_MAX * UNITS_ROW # records of one group + hb_off = cutlass.Int32(0) + if cutlass.const_expr(HB_REPL > 1): + # this CTA's copy of the records + hb_off = (bx % cutlass.Int32(HB_REPL)) * cutlass.Int32(HB_STRIDE) + NZ = (nrec_cap + 32 * XW - 1) // (32 * XW) # records per lane: lane + 32 (XW z + xi) + for g in range(ng): + buf = g & cutlass.Int32(1) + if g >= cutlass.Int32(2): + _waitd( + b2_free.subview(buf), + ((g >> cutlass.Int32(1)) - cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 101, + ) + _waitd( + fc1_done.subview(g & cutlass.Int32(3)), + (g >> cutlass.Int32(2)) & cutlass.Int32(1), + tdbg, + 4, + ) # this CTA's own share of group g is out + if cutlass.const_expr(TIMING): + if (lane == 0) & (xi == cutlass.Int32(0)) & (g < cutlass.Int32(8)): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(32) + g + ) + n_here = g_n - g * cutlass.Int32(4) + n_here = cutlass.select_(n_here > cutlass.Int32(4), cutlass.Int32(4), n_here) + nrec = n_here * cutlass.Int32(M_MAX * UNITS_ROW) + rec0 = g * cutlass.Int32( + 4 * M_MAX * UNITS_ROW + ) # rows (4 g + j) M + t are contiguous + done = cutlass.Int32(0) + rspin = cutlass.Int32(0) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 29) + npoll = cutlass.Int32(0) + got = [[cutlass.Int32(0) for _ in range(3)] for _ in range(NZ)] + while done == cutlass.Int32(0): + ok = cutlass.Boolean(True) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + v4 = prims.load_ext( + hbuf32.subview( + (rec0 + qi) * cutlass.Int32(4) + (hb_off if HB_REPL > 1 else 0) + ), + dtype=cutlass.Int32, + count=4, + order="relaxed", + scope="gpu", + ) + for wq in cutlass.range_constexpr(3): + got[z][wq] = cutlass.Int32(v4[wq]) + ok = ok & ( + ( + ((got[z][0] ^ tagv) | (got[z][1] ^ tagv) | (got[z][2] ^ tagv)) + & cutlass.Int32(-16777216) + ) + == cutlass.Int32(0) + ) + npoll = npoll + cutlass.Int32(1) + if prims.vote_sync(0xFFFFFFFF, ok, prims.VoteSync.BALLOT) == cutlass.Int32(-1): + done = cutlass.Int32(1) + else: + _nanosleep(B2_POLL_NS) + if cutlass.const_expr(HANG_DBG): + rspin = rspin + cutlass.Int32(1) + if rspin > cutlass.Int32(1 << 14): + done = cutlass.Int32(1) + tdbg.store( + cutlass.Int64(200), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(24) + warp, + ) + for z in cutlass.range_constexpr(NZ): + qi = lane + cutlass.Int32(32 * XW * z) + cutlass.Int32(32) * xi + if qi < nrec: + w0, w1, w2 = got[z][0], got[z][1], got[z][2] + rr = qi // cutlass.Int32( + UNITS_ROW + ) # row within the group: j M + t = B row n + c = qi - rr * cutlass.Int32(UNITS_ROW) + t2 = c >> cutlass.Int32(4) + cc = (c >> cutlass.Int32(1)) & cutlass.Int32(7) + lo = (w0 & cutlass.Int32(0xFFFFFF)) | (w1 << cutlass.Int32(24)) + hi = ((w1 >> cutlass.Int32(8)) & cutlass.Int32(0xFFFF)) | ( + (w2 & cutlass.Int32(0xFFFF)) << cutlass.Int32(16) + ) + sB2_32.store( + (lo, hi), + idx=( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_BYTES_B2) + + rr * cutlass.Int32(KT_BYTES) + + (cc ^ (rr & cutlass.Int32(7))) * cutlass.Int32(16) + + (c & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + # scale of block c: atom (t2, kb = (c % 16) / 4), row rr, byte c % 4 + sSFB2_8.store( + cutlass.Int8((w2 >> cutlass.Int32(16)) & cutlass.Int32(0xFF)), + idx=( + ( + (buf * cutlass.Int32(KT2) + t2) * cutlass.Int32(NUM_KB) + + ((c & cutlass.Int32(15)) >> cutlass.Int32(2)) + ) + * cutlass.Int32(128) + + rr * cutlass.Int32(4) + ) + * cutlass.Int32(4) + + (c & cutlass.Int32(3)), + ) + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + if cutlass.const_expr(XW > 1): + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=B2_BAR, number_of_threads=32 * XW) + cute.arch.sync_warp() + if (xi == cutlass.Int32(0)) & prims.elect_sync(): + sd0 = prims.Tcgen05SmemDesc.build( + sSFB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=128, + base_offset=0, + layout=0, + ) + for a in cutlass.range_constexpr(KT2 * NUM_KB): + prims.tcgen05_cp( + s2t_shape, + _tptr( + base_row, + base_col + SFB2_COL + buf * cutlass.Int32(KT2 * NUM_KB * 4) + a * 4, + ), + sd0 + + cutlass.Int64( + (buf * cutlass.Int32(KT2 * NUM_KB) + cutlass.Int32(a)) + * cutlass.Int32(512 >> 4) + ), + multicast=s2t_mc, + ) + prims.tcgen05_commit(b2_ready.subview(buf)) + if (lane == 0) & (xi == cutlass.Int32(0)): + _stamp(tdbg, bx, 25) + if cutlass.const_expr(TIMING): + if g < cutlass.Int32(8): + tdbg.store( + _gtime(), idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(11) + g + ) + if cutlass.const_expr(TIMING): + tdbg.store( + cutlass.Int64(npoll), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(30), + ) + + # ---- MMA (warp 7) + if warp == MMA_WARP: + idesc1 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N1, + m_dim=MMA_M, + sparsity_version=0, + ) + idesc2 = prims.Tcgen05MxOmmaInstrDesc.build( + a_dtype=cutlass.Float4E2M1FN, + b_dtype=cutlass.Float4E2M1FN, + scale_format=0, + n_dim=N2, + m_dim=MMA_M, + sparsity_version=0, + ) + desc_a0 = prims.Tcgen05SmemDesc.build( + sA.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b10 = prims.Tcgen05SmemDesc.build( + sB1.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + desc_b20 = prims.Tcgen05SmemDesc.build( + sB2.subview(0), + leading_byte_offset=16, + stride_byte_offset=1024, + base_offset=0, + layout=2, + ) + tcol = (base_row << 16) | base_col + _waitd(b1_ready, 0, tdbg, 5) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + stage = cutlass.Int32(0) + phase = cutlass.Int32(0) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + acc_b2 = cutlass.Int64(0) + acc_sf = cutlass.Int64(0) + for p in range(ng + cutlass.Int32(LAG)): + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + _waitd( + acc_empty.subview(ab), + ((k1 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + tdbg, + 102, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc_ptr = cutlass.inttoptr( + tcol + ACC1_COL + ab * cutlass.Int32(N1), 6, cutlass.Float32 + ) + for j in cutlass.range(j0, j1, unroll=1): + tw0 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 6) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 6) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw0) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b10 + cutlass.Int64(j * cutlass.Int32(NUM_BYTES_B1 >> 4)) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = tcol + SFB1_COL + j * cutlass.Int32(NUM_KB * 4) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(NUM_KB): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc1, + (j > j0) | cutlass.Boolean(kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc_full.subview(ab)) + k1 = k1 + cutlass.Int32(1) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + buf = g & cutlass.Int32(1) + tw2 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + _waitd( + b2_ready.subview(buf), (g >> cutlass.Int32(1)) & cutlass.Int32(1), tdbg, 103 + ) + if cutlass.const_expr(TIMING): + acc_b2 = acc_b2 + (_gtime() - tw2) + if (lane == 0) & (g == ng - cutlass.Int32(1)): + _stamp(tdbg, bx, 4) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_empty.subview(a2), + ((k2 >> cutlass.Int32(1)) & cutlass.Int32(1)) ^ cutlass.Int32(1), + tdbg, + 104, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc2_ptr = cutlass.inttoptr( + tcol + ACC2_COL + a2 * cutlass.Int32(N2), 6, cutlass.Float32 + ) + for t2 in cutlass.range_constexpr(KT2): + tw1 = _gtime() if cutlass.const_expr(TIMING) else cutlass.Int64(0) + if cutlass.const_expr(SFA_CP): + _waitd(ab_full.subview(stage), phase, tdbg, 7) + prims.fence_proxy( + "async_shared", space=prims.SharedSpace.shared_cta + ) + else: + _waitd(sfa_ready.subview(stage), phase, tdbg, 7) + if cutlass.const_expr(TIMING): + acc_sf = acc_sf + (_gtime() - tw1) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + da = desc_a0 + cutlass.Int64(stage * cutlass.Int32(NUM_BYTES_A >> 4)) + db = desc_b20 + cutlass.Int64( + (buf * cutlass.Int32(KT2) + cutlass.Int32(t2)) + * cutlass.Int32(NUM_BYTES_B2 >> 4) + ) + sfa_c = tcol + SFA1_COL + stage * cutlass.Int32(16) + sfb_c = ( + tcol + + SFB2_COL + + buf * cutlass.Int32(KT2 * NUM_KB * 4) + + cutlass.Int32(t2 * NUM_KB * 4) + ) + if prims.elect_sync(): + if cutlass.const_expr(SFA_CP): + _sfa_to_tmem(sSFA, stage, base_row, base_col) + for kb in cutlass.range_constexpr(_nkb2(t2)): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF4NVF4, + prims.CTAGroup.CTA_1, + acc2_ptr, + da + INC_K * kb, + db + INC_K * kb, + idesc2, + cutlass.Boolean(t2 > 0 or kb > 0), + cutlass.inttoptr(sfa_c + kb * 4, 6, cutlass.Int32), + cutlass.inttoptr(sfb_c + kb * 4, 6, cutlass.Int32), + scale_vec_size=prims.Tcgen05MMABlockScale.BLOCK16, + ) + prims.tcgen05_commit(ab_empty.subview(stage)) + stage, phase = _adv(stage, phase) + if prims.elect_sync(): + prims.tcgen05_commit(acc2_full.subview(a2)) + k2 = k2 + cutlass.Int32(1) + if cutlass.const_expr(TIMING): + if lane == 0: + tdbg.store(acc_b2, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(26)) + tdbg.store(acc_sf, idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(27)) + + # ---- epilogue warps (0-3) + if warp < 4: + row_w = base_row + warp * 32 + for r in cutlass.range_constexpr((6 * G_CAP + EPI_THREADS - 1) // EPI_THREADS): + qi = tidx + cutlass.Int32(r * EPI_THREADS) + sl_e = qi // cutlass.Int32(6) + if (qi < cutlass.Int32(6 * G_CAP)) & (sl_e < g_n): + fld = qi - sl_e * cutlass.Int32(6) + s_esc.store( + cutlass.Float32( + escal.load(idx=fld * cutlass.Int32(E_LOCAL) + s_el.load(idx=sl_e)) + ), + idx=sl_e * cutlass.Int32(8) + fld, + ) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + k1 = cutlass.Int32(0) + k2 = cutlass.Int32(0) + SH_ADD = SHARED + wys = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + pars = [cutlass.Int32(0) for _ in range(NB2_CAP * M_MAX // 4 + 1)] + ard = (cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0), cutlass.Int64(0)) + for p in range(ng + cutlass.Int32(LAG)): + if cutlass.const_expr(AR_PRE): + if (p == ng + cutlass.Int32(LAG - 1)) & ar_on: + # one 8-byte load each (the vectorizer merges adjacent arp.load()s into 16-byte reads + # at 8-byte offsets: misaligned) + ard = tuple( + cutlass.Int64(_ld_relaxed_u64(arp.subview(i_).data_ptr().toint())) + for i_ in (1, 3, 4, 5) + ) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_cl = cutlass.select_( + f_bb < nb, (blk0 + f_bb) * cutlass.Int32(R2), cutlass.Int32(0) + ) + pars[bt] = _ar_parity(arp, f_t, f_cl) + if cutlass.const_expr(SH_ADD): + if p == ng + cutlass.Int32(LAG - 1): + # this CTA's y_sh words, loaded while the last group's FC2 runs (re-polled in finalize) + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + f_pair = warp + cutlass.Int32(bt * 4) + f_bb = f_pair // cutlass.Int32(M_MAX) + f_t = f_pair - f_bb * cutlass.Int32(M_MAX) + f_ch = cutlass.select_( + f_bb < nb, + f_t * cutlass.Int32(H) + + (blk0 + f_bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + cutlass.Int32(0), + ) + wys[bt] = _ld_relaxed_u32(dnx.subview(f_ch).data_ptr().toint()) + if p < ng: + i0, i1 = _plan_range(s_plan, p) + for k in range(_nsegs(i0, i1)): + ug, j0, j1 = _seg(i0, i1, k) + ab = k1 & cutlass.Int32(1) + slot = p * cutlass.Int32(4) + ug // cutlass.Int32(U1) + cblk = ug - (ug // cutlass.Int32(U1)) * cutlass.Int32(U1) + unit = slot * cutlass.Int32(U1) + cblk + _waitd( + acc_full.subview(ab), + (k1 >> cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 105, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + acc = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, base_col + ACC1_COL + ab * cutlass.Int32(N1), cutlass.Float32 + ), + num=N1, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc_empty.subview(ab)) + pb = ab * cutlass.Int32(M_MAX * C1 * R1) + for t in cutlass.range_constexpr(M_MAX): + part = cutlass.Float32(0.0) + for w in cutlass.range_constexpr(4): + part = cutlass.select_( + warp == cutlass.Int32(w), cutlass.Float32(acc[QS * w + t]), part + ) + s_part.store(part, idx=pb + (t * C1 + warp) * R1 + lane) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + for tr in cutlass.range_constexpr((M_MAX + 3) // 4): + t = warp + cutlass.Int32(4 * tr) + if t < cutlass.Int32(M_MAX): + xv = _add_rn( + _add_rn( + _add_rn( + s_part.load(idx=pb + (t * C1 + 0) * R1 + lane), + s_part.load(idx=pb + (t * C1 + 1) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 2) * R1 + lane), + ), + s_part.load(idx=pb + (t * C1 + 3) * R1 + lane), + ) + finish = cutlass.Int32(1) + if (j0 != cutlass.Int32(0)) | (j1 != cutlass.Int32(S1)): + paddr = ( + partials.subview( + (unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane + ) + .data_ptr() + .toint() + ) + if j0 == cutlass.Int32(0): + _st_relaxed_u32(paddr, xv.bitcast(cutlass.Int32)) + finish = cutlass.Int32(0) + else: + pbits = _ld_relaxed_u32(paddr) + spins = cutlass.Int32(0) + while pbits == cutlass.Int32(PARTIAL_EMPTY): + pbits = _ld_relaxed_u32(paddr) + spins = spins + cutlass.Int32(1) + _st_relaxed_u32(paddr, cutlass.Int32(PARTIAL_EMPTY)) + xv = _add_rn(pbits.bitcast(cutlass.Float32), xv) + cute.arch.sync_warp() + if cutlass.const_expr( + XVDUMP + ): # debug: FC1 sums into the (unused) partial slots + partials.store( + xv.bitcast(cutlass.Int32), + idx=(unit * cutlass.Int32(M_MAX) + t) * cutlass.Int32(R1) + + lane, + ) + if finish == cutlass.Int32(1): + gate_raw = cute.arch.shuffle_sync_bfly(xv, 8) + is_up = ( + (lane >> cutlass.Int32(3)) & cutlass.Int32(1) + ) == cutlass.Int32(0) + es = slot * cutlass.Int32(8) + qv, sfb = _swiglu_quant( + xv, + gate_raw, + is_up, + s_esc.load(idx=es), + s_esc.load(idx=es + cutlass.Int32(1)), + s_esc.load(idx=es + cutlass.Int32(2)), + s_esc.load(idx=es + cutlass.Int32(3)), + s_esc.load(idx=es + cutlass.Int32(4)), + ) + byte = _e2m1x2(cute.arch.shuffle_sync_down(qv, 16), qv) + bt = [ + cute.arch.shuffle_sync(byte, i) & cutlass.Int32(0xFF) + for i in range(8) + ] + rw0 = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | tagv + ) + rw1 = ( + bt[3] + | (bt[4] << cutlass.Int32(8)) + | (bt[5] << cutlass.Int32(16)) + | tagv + ) + rw2 = ( + bt[6] + | (bt[7] << cutlass.Int32(8)) + | ((sfb & cutlass.Int32(0xFF)) << cutlass.Int32(16)) + | tagv + ) + if cutlass.const_expr(B2IMG): + if lane == 0: + # B2 row n = j M + t, k-tile c / 16, 16-byte chunk (c / 2) % 8 + # (swizzled), half c % 2; scale: atom (t2, kb = (c % 16) / 4), + # row n at byte 16 n, byte c % 4 + n = (ug // cutlass.Int32(U1)) * cutlass.Int32(M_MAX) + t + t2i = cblk >> cutlass.Int32(4) + cci = (cblk >> cutlass.Int32(1)) & cutlass.Int32(7) + gbase = p * cutlass.Int32(IMG_BYTES) + lo = ( + bt[0] + | (bt[1] << cutlass.Int32(8)) + | (bt[2] << cutlass.Int32(16)) + | (bt[3] << cutlass.Int32(24)) + ) + hi = ( + bt[4] + | (bt[5] << cutlass.Int32(8)) + | (bt[6] << cutlass.Int32(16)) + | (bt[7] << cutlass.Int32(24)) + ) + img32.store( + (lo, hi), + idx=( + gbase + + t2i * cutlass.Int32(NUM_BYTES_B2) + + n * cutlass.Int32(KT_BYTES) + + (cci ^ (n & cutlass.Int32(7))) + * cutlass.Int32(16) + + (cblk & cutlass.Int32(1)) * cutlass.Int32(8) + ) + // cutlass.Int32(4), + alignment=8, + ) + img8 = cutlass.Array( + img32.data_ptr(0), + shape=(NG_CAP * IMG_BYTES,), + dtype=cutlass.Int8, + alignment=16, + ) + img8.store( + cutlass.Int8(sfb & cutlass.Int32(0xFF)), + idx=gbase + + cutlass.Int32(IMG_B2) + + ( + ( + t2i * cutlass.Int32(NUM_KB) + + ( + (cblk & cutlass.Int32(15)) + >> cutlass.Int32(2) + ) + ) + * cutlass.Int32(512) + ) + + n * cutlass.Int32(16) + + (cblk & cutlass.Int32(3)), + ) + else: + if cutlass.const_expr(HB_REPL == 1): + if lane == 0: + hbuf32.store( + (rw0, rw1, rw2, tagv), + idx=( + (slot * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(UNITS_ROW) + + cblk + ) + * cutlass.Int32(4), + alignment=16, + ) + else: + if lane < HB_REPL: # copy `lane` + hbuf32.store( + (rw0, rw1, rw2, tagv), + idx=( + (slot * cutlass.Int32(M_MAX) + t) + * cutlass.Int32(UNITS_ROW) + + cblk + ) + * cutlass.Int32(4) + + lane * cutlass.Int32(HB_STRIDE), + alignment=16, + ) + k1 = k1 + cutlass.Int32(1) + prims.mbarrier_arrive( + fc1_done.subview(p & cutlass.Int32(3)) + ) # warp 5 may poll group p + if tidx == 0: + _stamp(tdbg, bx, 10) + if cutlass.const_expr(TIMING): + if p < cutlass.Int32(8): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(40) + p, + ) + if p >= cutlass.Int32(LAG): + g = p - cutlass.Int32(LAG) + for bb in range(nb): + a2 = k2 & cutlass.Int32(1) + _waitd( + acc2_full.subview(a2), + (k2 >> cutlass.Int32(1)) & cutlass.Int32(1), + tdbg, + 106, + ) + if cutlass.const_expr(TIMING): + if (tidx == 0) & (g == ng - cutlass.Int32(1)): + tdbg.store( + _gtime(), + idx=bx * cutlass.Int32(TD_STRIDE) + cutlass.Int32(8) + bb, + ) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + yv = prims.tcgen05_ld( + "32x32b", + _tptr( + row_w, + base_col + + ACC2_COL + + a2 * cutlass.Int32(N2) + + warp * cutlass.Int32(M_MAX), + cutlass.Float32, + ), + num=LD_POW2, + ) + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + prims.mbarrier_arrive(acc2_empty.subview(a2)) + sl = g * cutlass.Int32(4) + warp + if sl < g_n: + a2s = s_esc.load(idx=sl * cutlass.Int32(8) + cutlass.Int32(5)) + for t in cutlass.range_constexpr(M_MAX): + kp = s_kpos.load(idx=sl * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + if kp >= cutlass.Int32(0): + ys.store( + _bf16_round(_mul_rn(cutlass.Float32(yv[t]), a2s)), + idx=( + (bb * cutlass.Int32(M_MAX) + cutlass.Int32(t)) + * cutlass.Int32(TOP_K) + + kp + ) + * cutlass.Int32(R2) + + lane, + ) + k2 = k2 + cutlass.Int32(1) + prims.mbarrier_arrive(b2_free.subview(g & cutlass.Int32(1))) + if tidx == 0: + _stamp(tdbg, bx, 6) + cute.arch.sync_warp() + cute.arch.barrier(barrier_id=EPI_BAR, number_of_threads=EPI_THREADS) + if tidx == 0: + _stamp(tdbg, bx, 19) + # finalize: (block bb, token t) per warp round: the fma chain over t's top-4 in order + for bt in cutlass.range_constexpr((NB2_CAP * M_MAX + 3) // 4): + pair = warp + cutlass.Int32(bt * 4) + bb = pair // cutlass.Int32(M_MAX) + t = pair - bb * cutlass.Int32(M_MAX) + if bb < nb: + acc_o = cutlass.Float32(0.0) + for k in cutlass.range_constexpr(TOP_K): + acc_o = _fma_rn( + s_w.load(idx=t * cutlass.Int32(TOP_K) + cutlass.Int32(k)), + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ), + acc_o, + ) + if cutlass.const_expr( + YDUMP + ): # debug: y of (token, k) into the partials workspace + partials.store( + cutlass.Float32( + ys.load( + idx=( + (bb * cutlass.Int32(M_MAX) + t) * cutlass.Int32(TOP_K) + + cutlass.Int32(k) + ) + * cutlass.Int32(R2) + + lane + ) + ).bitcast(cutlass.Int32), + idx=(t * cutlass.Int32(TOP_K) + cutlass.Int32(k)) * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)), + ) + ch = ( + t * cutlass.Int32(H) + + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(SH_ADD): + # + y_sh (this row's shared output, from its shared CTA), as shared_output.add_(routed_output) + wy = wys[bt] + while ((wy >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF)) != tag16: + wy = _ld_relaxed_u32(dnx.subview(ch).data_ptr().toint()) + acc_o = _add_rn( + (wy << cutlass.Int32(16)).bitcast(cutlass.Float32), _bf16_round(acc_o) + ) + if (ar_on == cutlass.Boolean(False)) | cutlass.Boolean(not OUT_SKIP): + out.store(cutlass.BFloat16(acc_o), idx=ch) + if ar_on: + cl = ( + (blk0 + bb) * cutlass.Int32(R2) + + cutlass.Int32(4) * (lane & cutlass.Int32(7)) + + (lane >> cutlass.Int32(3)) + ) + if cutlass.const_expr(AR_PRE): + _ar_push_pre( + ard, + pars[bt], + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + else: + _ar_push( + arp, + _bf16_round(acc_o), + ((lane >> cutlass.Int32(3)) & cutlass.Int32(1)) == cutlass.Int32(0), + t, + cl, + ) + + # ---- teardown + if tidx == 0: + _stamp(tdbg, bx, 7) + if cutlass.const_expr(not EARLY_EPOCH): + epochs.store(ep + cutlass.Int32(1), idx=bx) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + cute.arch.sync_warp() + prims.barrier_cta_sync(0) + if ar_on | cutlass.Boolean(LATE_TRIGGER): + # the all-reduce polls the pushed slots without a grid wait: release it after every push of this CTA + cute.arch.griddepcontrol_launch_dependents() + if warp == MMA_WARP: + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Float32), TMEM_COLS + ) + + +@cute.jit +def m3_moe( + a1_tensor: cute.Tensor, + sfa1_tensor: cute.Tensor, + a2_tensor: cute.Tensor, + sfa2_tensor: cute.Tensor, + logits: cute.Tensor, + bias: cute.Tensor, + x32: cute.Tensor, + xsf32: cute.Tensor, + escal: cute.Tensor, + plan: cute.Tensor, + partials: cute.Tensor, + hbuf32: cute.Tensor, + epochs: cute.Tensor, + out: cute.Tensor, + route_dbg: cute.Tensor, + tdbg: cute.Tensor, + xb32: cute.Tensor, + rw: cute.Tensor, + sfsc: cute.Tensor, + lgbuf: cute.Tensor, + guw: cute.Tensor, + gusf: cute.Tensor, + dnw: cute.Tensor, + dnsf: cute.Tensor, + hx: cute.Tensor, + dnx: cute.Tensor, + arp: cute.Tensor, + s1w: cute.Tensor, + s2w: cute.Tensor, + img32: cute.Tensor, + b2cnt: cute.Tensor, + stream: cuda_driver.CUstream, +): + tma_a1_desc = cuda.create_tensor_map_tiled( + global_address=a1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H // 2, TWO_I, E_LOCAL], + global_strides=[H // 2 // 16, TWO_I * (H // 2) // 16], + box_dims=(KT_BYTES, R1, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_a2_desc = cuda.create_tensor_map_tiled( + global_address=a2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[I_TP // 2, H, E_LOCAL], + global_strides=[I_TP // 2 // 16, H * (I_TP // 2) // 16], + box_dims=(KT_BYTES, R2, 1), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_s1_desc = cuda.create_tensor_map_tiled( + global_address=sfa1_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * TWO_I * (H // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_s2_desc = cuda.create_tensor_map_tiled( + global_address=sfa2_tensor.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[128, E_LOCAL * H * (I_TP // 16) // 128], + global_strides=[128 // 16], + box_dims=(128, NUM_BYTES_SFA_RAW // 4 // 128), + swizzle=cuda.TensorMapSwizzle.none, + ) + tma_gu_desc = tma_a1_desc + tma_dn_desc = tma_a2_desc + if cutlass.const_expr(SHARED and not SH_PRETILE): + tma_gu_desc = cuda.create_tensor_map_tiled( + global_address=guw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[H, 2 * SH_I], + global_strides=[H // 16], + box_dims=(KT_BYTES, 32), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + tma_dn_desc = cuda.create_tensor_map_tiled( + global_address=dnw.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[SH_I, H], + global_strides=[SH_I // 16], + box_dims=(KT_BYTES, 128), + swizzle=cuda.TensorMapSwizzle.s128b, + ) + m3_moe_kernel( + tma_a1_desc, + tma_a2_desc, + tma_s1_desc, + tma_s2_desc, + tma_gu_desc, + tma_dn_desc, + logits, + bias, + x32, + xsf32, + escal, + plan, + partials, + hbuf32, + epochs, + out, + route_dbg, + tdbg, + xb32, + rw, + sfsc, + lgbuf, + guw, + gusf, + dnw, + dnsf, + hx, + dnx, + arp, + s1w, + s2w, + img32, + b2cnt, + ).launch( + grid=(NCTA + NSHC, 1, 1), + block=(THREADS, 1, 1), + cluster=(1, 1, 1), + stream=stream, + use_pdl=True, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/op.py new file mode 100644 index 000000000000..76665ed9706e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_moe/op.py @@ -0,0 +1,828 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Host side of the MiniMax-M3 routed-expert kernel on one TP8 rank (routed and shared intermediate 384 per rank), +over the TRTLLM-Gen NVFP4 buffers read in place. + +``m3_moe`` (m3_moe_kernel.py, :class:`M3MoeState`): 1..8 decode tokens (decode batches, spec-decode verify). The call +takes the bf16 hidden states and folds in the router GEMV (bit-exact with trtllm::minimax_m3_router_gemv_fp4) and the +activation's NVFP4 quantization. With ``shared=True`` it also runs the shared expert on dedicated CTAs and returns +bf16(shared + routed); otherwise it returns the rank's routed partial, bit for bit what the stock fused MoE returns. + +Usage (build before CUDA-graph capture; every call is capture-safe afterwards):: + + state = M3MoeState( + device, num_tokens=6, shared=True + ) # workspace + compiled kernel, shared by all layers + layer = state.layer( + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=router_weight, + fc31_input_scale=fc31_input_scale, + shared_weights=(gate_up_w, gate_up_sf, down_w, down_sf), + ) + out = layer(hidden) # bf16 [M, 6144] +""" + +from __future__ import annotations + +import importlib.util +import os +import sys +from pathlib import Path +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._torch import cute_aot + +from ...m31_switches import m31_disabled + +HIDDEN = 6144 # capacity: fixed: the checkpoint's hidden size +INTER = 384 # capacity: fixed: the routed experts' width per tp8 rank +NUM_EXPERTS = 128 +SHARED_INTER = 384 # capacity: fixed: the shared expert's width per tp8 rank (the routed one) +SHARED_CTAS = 24 # capacity: fixed: an SM split, not a size (m3_moe's shared-expert CTAs (beside num_ctas routed CTAs)) +TOP_K = 4 # capacity: fixed: the checkpoint's top-k +U1 = 2 * INTER // 32 # FC1 units per expert +# The router GEMV takes at most 2 experts per CTA (router weights held in registers). +MIN_CTAS = 64 # capacity: fixed: an SM split, not a size + +_UNIT_BYTES = 32 * (HIDDEN // 2 + HIDDEN // 16) # one FC1 unit's weight + scale bytes +_ROW2_BYTES = INTER // 2 + INTER // 16 # one FC2 weight row's bytes (values + scales) +_S1 = 6 # capacity: fixed: pipeline stages (FC1 stages (items) per unit) +_ITEM_BYTES = _UNIT_BYTES // _S1 # capacity: fixed: a ring item's bytes +_modules: Dict[tuple, object] = {} +_compiled: Dict[tuple, object] = {} +# m3_moe trace-time options by token count 1..8 (index num_tokens - 1). +# ``sfa_cp``: 18 KB ring stages, the A scales copied by cp.async into the instruction layout (every token count). +SFA_CP_BY_M: Tuple[int, ...] = (1, 1, 1, 1, 1, 1, 1, 1) +# ``b2img``: FC1 -> FC2 hand-off through per-group global B images + release counters (7 and 8 tokens). +B2IMG_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 1, 1) +# ``ar_pre``: the finalize's all-reduce push with its call parities / descriptor preloaded (all but 2 tokens). +AR_PRE_BY_M: Tuple[int, ...] = (1, 0, 1, 1, 1, 1, 1, 1) +# token counts whose OUT_SKIP engagement line was logged (once per process each) +_OUT_SKIP_LOGGED: set = set() +# Ring ``stages`` (0 = the kernel's default, the most that fit). 6 at 8 tokens measured 0.4-0.5 us faster than the +# default 7 (same MMAs, bit-identical outputs). +STAGES_BY_M: Tuple[int, ...] = (0, 0, 0, 0, 0, 0, 0, 6) +# ``hb_repl``: copies of the tagged FC1 -> FC2 records (applied when > 1); CTA bx polls copy bx % hb_repl, which +# spreads the polls of the few hot record lines over L2. 7 and 8 tokens hand off through the B images instead. +HB_REPL_BY_M: Tuple[int, ...] = (32, 32, 8, 8, 1, 1, 1, 1) +# Per-token-count overrides of the tables above: at 8 tokens the FC1 -> FC2 hand-off through tagged records instead of +# the B image (ported from the GB300 target: 0.15 us/call faster there, paired 4/4 with bit-identical outputs, +# MOE_CLK64_MR6C_READ; it also leaves warp 6 idle at 8 tokens). Kill switch ``TRTLLM_M31_DISABLE_MOE_M8CFG`` restores +# the tables' values. +M8_CFG_BY_M: Dict[int, Dict[str, int]] = {8: {"b2img": 0}} +# SH20 (VR SHFC2-4, CTX design SHFC2_VR_UB_AND_DESIGN_v1): at SH20_TOKENS the shared expert's 24 work units run on +# SH20_CTAS shared CTAs (CTAs 0 .. 3 run two units, both gate_ups before either down), so the routed grid takes the +# 4 SMs this frees: num_ctas = SMs - SH20_CTAS = 192 gives every routed CTA exactly one of the 192 FC2 32-row blocks +# (188 left 4 CTAs with two: the call's stragglers). Bit-identical outputs (only the CTA -> unit mapping and the +# routed CTA count change). Upper bound (VR TP8, 8 tokens): -0.94 us/call. Kill switch TRTLLM_M31_DISABLE_MOE_SH20, +# read at import. +SH20 = not m31_disabled("MOE_SH20") +SH20_CTAS = 20 # capacity: fixed: an SM split, not a size +SH20_TOKENS = frozenset( + {8} +) # capacity: fixed: the K7 c1 rows (the two-unit CTAs finish off the critical path there) + + +# OUT_SKIP work counter (pushing engine calls that skipped the out store, this process) +OUT_SKIP_TAKEN = [0] + + +def shared_ctas_for(num_tokens: int, shared: bool) -> int: + """The shared-expert CTAs beside the routed ones of an engine built for ``num_tokens`` (0 without the shared + expert): SH20_CTAS at SH20_TOKENS while SH20 is on, else SHARED_CTAS.""" + if not shared: + return 0 + return SH20_CTAS if SH20 and num_tokens in SH20_TOKENS else SHARED_CTAS + + +# ``lg_repl``: replicas of the router logit exchange at up to LG_REPL_MAX_TOKENS tokens (fewer stores before the +# exchange; bit-identical outputs). Above that the kernel's default applies. +LG_REPL = 32 +LG_REPL_MAX_TOKENS = 4 # capacity: fixed: measured crossover +# VRPOLL: 16 replicas instead of the kernel's 32 at 7 and 8 tokens (the K7 c1 rows): the exchange's tail CTAs poll +# fewer lines. Bit-identical outputs. Microbench (VR TP8, CUDA graph, 57 calls): -0.25 us/call at 8 tokens, -0.24 at 7. +# Kill switch TRTLLM_M31_DISABLE_MOE_VRPOLL, read at import. +VRPOLL = not m31_disabled("MOE_VRPOLL") +VRPOLL_LG_REPL = 16 # capacity: fixed: measured optimum (8 / 16 / 64 swept) +VRPOLL_TOKENS = frozenset({7, 8}) # capacity: fixed: the measured rows +# Byte-balanced FC1 ranges below one unit per CTA (:func:`_fine_ranges`) for groups too small to give every CTA a +# whole unit; False gives such groups whole units on the CTAs with the least FC2 work. +FINE_PLAN = True +# HALFPLAN: when every CTA's byte-balanced share of a group is the same and at most half a unit (the partial last FC1 +# group with SH20's 192 routed CTAs at 8 tokens), the fine ranges' boundaries sit on half units (S1 // 2 items), so a +# split unit is halved (3 + 3) instead of extended to its end (1 + 5) and no CTA gets more than half a unit; other +# groups keep their ranges. Bit-identical outputs (microbench: -0.18 us per call at m8 random). Kill switch +# TRTLLM_M31_DISABLE_MOE_HALFPLAN, read at import. +HALFPLAN = not m31_disabled("MOE_HALFPLAN") +# m3_moe phase lag: phase p streams FC1 of group p and FC2 of group p - M3_MOE_LAG (the kernel's default lag). +M3_MOE_LAG = 2 + + +_ENGINE_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "m3_moe_kernel.py") + + +def _kernel_module(config: dict, path: str): + key = (os.path.basename(path),) + tuple(sorted(config.items())) + mod = _modules.get(key) + if mod is None: + name = ( + f"{__name__}_" + + os.path.splitext(os.path.basename(path))[0] + + "_" + + "_".join(f"{k}{v}" for k, v in key[1:]) + ) + spec = importlib.util.spec_from_file_location(name, path) + mod = importlib.util.module_from_spec(spec) + mod.M3_CONFIG = dict(config) + sys.modules[name] = mod + spec.loader.exec_module(mod) + _modules[key] = mod + return mod + + +def _view(t: torch.Tensor, align: int): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align) + + +def is_supported(device: Optional[torch.device] = None) -> tuple: + if not torch.cuda.is_available(): + return False, "no CUDA" + major, _ = torch.cuda.get_device_capability(device) + if major != 10: + return False, "needs the sm_100 family (tcgen05 block-scaled NVFP4 MMA)" + try: + import cutlass # noqa: F401 + except ImportError: + return False, "CuTe DSL (nvidia-cutlass-dsl) is not installed" + return True, "" + + +def ar_push_desc( + mc_ptr: int, flags_ptr: int, rank: int, world: int, max_tokens: int, device +) -> torch.Tensor: + """The ``arp`` descriptor that makes a call also push its bf16 output into a multicast all-reduce exchange (set A + of m3_mc_allreduce's workspace at ``mc_ptr``, slot half = the all-reduce CTA's call parity in ``flags_ptr``). + With it the kernel releases its PDL dependents only after every push; the default (enable 0) keeps the old + behaviour.""" + return torch.tensor( + [1, mc_ptr, flags_ptr, rank, world, max_tokens, 0, 0], dtype=torch.int64, device=device + ) + + +def _shared_views(shared_weights, who): + """(gate_up, its scales, down, its scales) of the shared expert as flat uint8 views, checked.""" + if shared_weights is None: + raise ValueError(f"{who} (shared): shared_weights are required") + gu_w, gu_sf, dn_w, dn_sf = shared_weights + want = {"gate_up": (gu_w, (2 * SHARED_INTER, HIDDEN)), "down": (dn_w, (HIDDEN, SHARED_INTER))} + for name, (t, shape) in want.items(): + if t.dtype != torch.float8_e4m3fn or tuple(t.shape) != shape or not t.is_contiguous(): + raise ValueError(f"{who} (shared): {name} weight must be contiguous e4m3 {shape}") + if ( + gu_sf.numel() != 2 * SHARED_INTER * HIDDEN // 32 + or dn_sf.numel() != HIDDEN * SHARED_INTER // 32 + ): + raise ValueError( + f"{who} (shared): the UE8M0 scales must be the swizzled [N, K / 32] buffers" + ) + return ( + gu_w.view(torch.uint8).view(-1), + gu_sf.view(torch.uint8).reshape(-1), + dn_w.view(torch.uint8).view(-1), + dn_sf.view(torch.uint8).reshape(-1), + ) + + +def _pretile_shared(gu8, gusf8, dn8, dnsf8, n_cta: int = 24) -> torch.Tensor: + """The shared expert's weights as m3_moe's shared CTAs stream them (SH_PRETILE): per CTA s, per stage g, the 16 KB + shared-memory image (128-byte swizzle: 16-byte chunk j of row r at j ^ (r % 8)) of the stage's 128-row MMA tile and + its 512-byte instruction scale atom (row R's 4 codes at word 4 (R % 32) + R / 32). CTA s owns P = SHARED_INTER / 24 + = 16 positions in SPLIT = 128 / (2 P) = 4 K slices of GT = 48 / SPLIT = 12 k-tiles. Stages 0..GT-1: MMA row + R = P O + i (O = 2 q + u: slice q, u 0 gate / 1 up) = weight row u SHARED_INTER + P s + i at k-tile GT q + g; then + down tile j (rows 256 s + 128 j ..) at k-tile kt, j-major (SHARED_INTER / 128 k-tiles each). Built once per + layer.""" + dev = gu8.device + H_, I_ = HIDDEN, SHARED_INTER + P = I_ // n_cta + split = 128 // (2 * P) + gt = H_ // 128 // split + dkt = I_ // 128 + gu = gu8.view(2 * I_, H_) + dn = dn8.view(H_, I_) + gkb, dkb = H_ // 32, I_ // 32 + + def sf_bytes(sf, n, b, kbn): # code of (row n, K block b), broadcast + return sf[ + ((n >> 7) * (kbn // 4) + (b >> 2)) * 512 + (n & 31) * 16 + ((n >> 5) & 3) * 4 + (b & 3) + ] + + r = torch.arange(128, device=dev) + q = r >> 5 + ri = r & 31 + octant = r // P + chunk = torch.arange(8, device=dev) + swz = chunk[None, :] ^ (r[:, None] & 7) # source chunk of image chunk j: j ^ (r % 8) + out = torch.empty( + n_cta, gt + 2 * dkt, 16384 + 512, dtype=torch.uint8, device=dev + ) # capacity: fixed: a workspace of the kernel's fixed geometry + for s in range(n_cta): + # gate_up: MMA row R -> weight row, slice + nrow = (octant & 1) * I_ + P * s + r % P + sl = octant >> 1 + for g in range(gt): + kt = sl * gt + g # per row + tile = gu[ + nrow[:, None], (kt[:, None] * 128 + torch.arange(128, device=dev)[None, :]) + ] # [128, 128] + img = tile.view(128, 8, 16)[r[:, None], swz] # image chunk j = tile chunk j ^ (r % 8) + out[s, g, :16384] = img.reshape(-1) + kb = torch.arange(4, device=dev) + codes = sf_bytes(gusf8, nrow[:, None], kt[:, None] * 4 + kb[None, :], gkb) # [128, 4] + atom = torch.empty(512, dtype=torch.uint8, device=dev) + atom[(ri[:, None] * 16 + q[:, None] * 4 + kb[None, :]).reshape(-1)] = codes.reshape(-1) + out[s, g, 16384:] = atom + for gd in range(2 * dkt): + jt, kt = gd // dkt, gd % dkt + row0 = 256 * s + 128 * jt + tile = dn[row0 : row0 + 128, kt * 128 : (kt + 1) * 128] + img = tile.reshape(128, 8, 16)[r[:, None], swz] + out[s, gt + gd, :16384] = img.reshape(-1) + out[s, gt + gd, 16384:] = dnsf8[ + ((row0 >> 7) * (dkb // 4) + kt) * 512 : ((row0 >> 7) * (dkb // 4) + kt) * 512 + 512 + ] + return out.view(-1) + + +# --------------------------------------------------------------------------------------------------------------------- +# m3_moe (m3_moe_kernel.py): 1..8 tokens per call, group-phased weight stream. +# --------------------------------------------------------------------------------------------------------------------- +# Pre-tiled shared weights per layer, keyed by the source buffers (which the engines read in place): every token-count +# state of a layer streams the same copy. An entry holds its sources, so their addresses are not reused while cached. +_PRETILED: Dict[tuple, tuple] = {} + + +def _pretiled(gu8, gusf8, dn8, dnsf8) -> torch.Tensor: + key = (gu8.device, gu8.data_ptr(), gusf8.data_ptr(), dn8.data_ptr(), dnsf8.data_ptr()) + hit = _PRETILED.get(key) + if hit is None: + hit = _PRETILED[key] = ((gu8, gusf8, dn8, dnsf8), _pretile_shared(gu8, gusf8, dn8, dnsf8)) + return hit[1] + + +def _fine_ranges(weights, total: int): + """Contiguous FC1 item ranges [s_c, e_c) over the CTAs, sized as close to ``weights`` (ideal real-valued item + counts summing to ``total``) as the split-unit protocol allows: every unit (S1 items) has at most one CTA boundary + strictly inside it (its head and tail parts, summed by the tail's holder), so a range that starts inside a unit + runs at least to that unit's end. Returns the item counts.""" + n = len(weights) + items = [0] * n + s = 0 + acc = 0.0 + half = _S1 // 2 + if HALFPLAN and n and max(weights) == min(weights) and total % half == 0 and total <= half * n: + # equal shares of at most half a unit: boundaries on half units, so every CTA gets 0 or S1 // 2 items, every + # split unit is halved and no range needs extending + for c in range(n): + acc += weights[c] + e = total if c == n - 1 else min(total, max(s, half * int(acc / half + 0.5))) + items[c] = e - s + s = e + assert s == total + return items + for c in range(n): + acc += weights[c] + e = total if c == n - 1 else min(total, max(s, int(acc + 0.5))) + if s % _S1 and e < total: # started inside a unit: run to (at least) its end + e = max(e, min(total, (s // _S1 + 1) * _S1)) + items[c] = e - s + s = e + assert s == total + return items + + +def make_phase_plan( + num_ctas: int, + m_max: int, + lag: int = 2, + tail_w: float = 1.0, + early0: bool = False, + fine: bool = FINE_PLAN, +): + """The work split of ``m3_moe`` for each distinct-expert count G (4 .. 4 m_max): FC2 32-row output blocks in + contiguous per-CTA ranges (the same for every group), and per group p (slots 4 p ..) a range of its unit-major FC1 + stages per CTA, balancing the bytes each CTA streams in phase p (FC1 of group p plus FC2 of group p - lag, and the + trailing FC2s with the last group). Ranges are empty or >= 6 stages (a unit is split over at most two CTAs); a + group too small for every CTA gets whole units on the CTAs with the least FC2 work, or (``fine``) byte-balanced + ranges of any length with at most one CTA boundary inside a unit (:func:`_fine_ranges`). Returns int32 + [4 m_max + 1, num_ctas, m_max + 1] (per group i0 | i1 << 16, then b0 | nb << 16) and the largest block count.""" + gcap = TOP_K * m_max + nb2 = HIDDEN // 32 + plan = torch.zeros(gcap + 1, num_ctas, m_max + 1, dtype=torch.int32) + nbs = [nb2 // num_ctas + (1 if c < nb2 % num_ctas else 0) for c in range(num_ctas)] + b0s = [sum(nbs[:c]) for c in range(num_ctas)] + blk_bytes = 32 * _ROW2_BYTES # one block, one slot + for g in range(TOP_K, gcap + 1): + ng = (g + 3) // 4 + for c in range(num_ctas): + plan[g, c, m_max] = b0s[c] | (nbs[c] << 16) + for p in range(ng): + n_p = min(4, g - 4 * p) + total = n_p * U1 * _S1 + fc2 = [0] * num_ctas + for c in range(num_ctas): + if p >= lag: + fc2[c] += nbs[c] * min(4, g - 4 * (p - lag)) * blk_bytes + if p == ng - 1 and not ( + early0 and p == 0 and m_max >= 2 + ): # early0: group 0's ranges the same for every G + for q in range(max(0, ng - lag), ng): + fc2[c] += int(tail_w * nbs[c] * min(4, g - 4 * q) * blk_bytes) + items = [0] * num_ctas + if total >= _S1 * num_ctas: + + def counts_for(limit): + return [max(_S1, (limit - f) // _ITEM_BYTES) for f in fc2] + + lo, hi = 0, max(fc2) + total * _ITEM_BYTES + while lo < hi: + mid = (lo + hi) // 2 + if sum(counts_for(mid)) >= total: + hi = mid + else: + lo = mid + 1 + items = counts_for(lo) + extra = sum(items) - total + while extra > 0: + c = min( + (c for c in range(num_ctas) if items[c] > _S1), + key=lambda c: (-(fc2[c] + items[c] * _ITEM_BYTES), c), + ) + items[c] -= 1 + extra -= 1 + elif fine: + # water-fill: CTA c streams fc2[c] + w_c item bytes, equal where w_c > 0 + lo_f, hi_f = 0.0, float(max(fc2) + total * _ITEM_BYTES) + for _ in range(100): + mid = 0.5 * (lo_f + hi_f) + if sum(max(0.0, (mid - f) / _ITEM_BYTES) for f in fc2) >= total: + hi_f = mid + else: + lo_f = mid + wts = [max(0.0, (hi_f - f) / _ITEM_BYTES) for f in fc2] + sw = sum(wts) + items = _fine_ranges([w * total / sw for w in wts], total) + else: + chosen = sorted(sorted(range(num_ctas), key=lambda c: (fc2[c], c))[: total // _S1]) + for c in chosen: + items[c] = _S1 + i0 = 0 + for c in range(num_ctas): + if items[c]: + plan[g, c, p] = i0 | ((i0 + items[c]) << 16) + i0 += items[c] + assert i0 == total + return plan, max(nbs) + + +# TAIL_TRIGGER: token counts whose engine state logged the taken line; pushing calls that release at the producer tail +# (counted at launch, so in eager mode only; a graph replays the captured call). +_TAIL_TRIGGER_LOGGED = set() +TAIL_TRIGGER_TAKEN = [0] + + +class M3MoeState: + """Workspace + compiled kernel of ``m3_moe`` (1..8 tokens per call) on one device, shared by the layers that run + on it in one stream order. The call takes the bf16 hidden states and computes the router logits (bit-exact with the + trtllm::minimax_m3_router_gemv_fp4 op) and the NVFP4 activation itself. ``shared``: the shared expert runs on + SHARED_CTAS dedicated CTAs beside the ``num_ctas`` routed ones and the call returns bf16(shared + routed). Build + before CUDA-graph capture; the first call compiles.""" + + fused_router = True + + def __init__( + self, + device: torch.device, + num_tokens: int, + num_ctas: Optional[int] = None, + shared: bool = False, + inter: int = INTER, + early_release: bool = True, + ): + """``inter``: the routed intermediate per rank; this engine is built for INTER (384) only. + On a call that pushes into the boundary all-reduce (``arp``), every routed CTA releases the PDL dependents when + its weights producer is done (the kernel's TAIL_TRIGGER, default on whatever ``early_release`` is). With + TRTLLM_M31_DISABLE_MOE_TAIL_TRIGGER, ``early_release`` picks the release point: True right after the grid + dependency wait (EARLY_AR), False after the push (teardown).""" + if inter != INTER: + raise ValueError( + f"m3_moe (TP8): routed intermediate {inter} per rank is not supported (only {INTER})" + ) + self.inter = inter + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "M3MoeState allocates and compiles: build it before CUDA-graph capture" + ) + sms_all = torch.cuda.get_device_properties(device).multi_processor_count + sh_ctas = shared_ctas_for(num_tokens, shared) + if shared and (num_ctas or sms_all) + sh_ctas > sms_all: + raise ValueError( + f"m3_moe (shared): num_ctas + {sh_ctas} shared-expert CTAs must fit the {sms_all} SMs " + f"(all resident at once)" + ) + if not 1 <= num_tokens <= 8: + raise ValueError(f"m3_moe takes 1..8 tokens per call, not {num_tokens}") + self.device = torch.device(device) + ok, why = is_supported(self.device) + if not ok: + raise RuntimeError(f"m3_moe: {why}") + sms = torch.cuda.get_device_properties(self.device).multi_processor_count + self.num_ctas = num_ctas or sms + if not MIN_CTAS <= self.num_ctas <= sms: + raise ValueError( + f"m3_moe runs {MIN_CTAS} .. {sms} CTAs (all resident at once), not {self.num_ctas}" + ) + self.m = num_tokens + # One token: the trailing FC2s weigh nothing in the last phase's balance. Group 0's FC1 ranges are the same + # for every distinct-expert count (early0). + plan, nb_cap = make_phase_plan( + self.num_ctas, num_tokens, M3_MOE_LAG, 0.0 if num_tokens == 1 else 1.0, True + ) + self.config = { + "m_max": num_tokens, + "num_ctas": self.num_ctas, + "nb2_cap": nb_cap, + "fused_router": 1, + "timing": 0, + "i_tp": INTER, + } + if HB_REPL_BY_M[num_tokens - 1] > 1: + self.config["hb_repl"] = HB_REPL_BY_M[num_tokens - 1] + if num_tokens <= LG_REPL_MAX_TOKENS: + self.config["lg_repl"] = LG_REPL + elif VRPOLL and num_tokens in VRPOLL_TOKENS: + self.config["lg_repl"] = VRPOLL_LG_REPL + from tensorrt_llm.logger import logger as _vrpoll_log + + _vrpoll_log.info( + f"MiniMax-M3 MoE engine VRPOLL (TP8): {VRPOLL_LG_REPL} router logit replicas, tokens={num_tokens}" + ) + if shared: + self.config["sh_i"] = SHARED_INTER + self.config["shared"] = 1 + if sh_ctas != SHARED_CTAS: + self.config["sh_ctas"] = sh_ctas + from tensorrt_llm.logger import logger as _sh20log + + _sh20log.info( + f"MiniMax-M3 MoE engine SH20 (TP8): the shared expert's 24 units on {sh_ctas} CTAs, " + f"{self.num_ctas} routed CTAs, tokens={num_tokens}" + ) + self.shared = shared + # 18 KB ring stages (A scales by cp.async into the instruction layout) per token count, unless configured + self.config["sfa_cp"] = SFA_CP_BY_M[num_tokens - 1] + # FC1 -> FC2 hand-off through per-group global B images + release counters, per token count + self.config["b2img"] = B2IMG_BY_M[num_tokens - 1] + # the finalize's all-reduce push with its call parities / descriptor preloaded, per token count + self.config["ar_pre"] = AR_PRE_BY_M[num_tokens - 1] + # MOE-FINALIZE-DEADSTORE (OUT_SKIP): a pushing call skips the bf16 ``out`` store the pushed + # all-reduce never reads (exact: the pushed words are unchanged). Default ON; kill switch + # TRTLLM_M31_DISABLE_MOE_OUT_SKIP=1. + if not m31_disabled("MOE_OUT_SKIP"): + self.config["out_skip"] = 1 + if num_tokens not in _OUT_SKIP_LOGGED: + _OUT_SKIP_LOGGED.add(num_tokens) + from tensorrt_llm.logger import logger as _os_log + + _os_log.info( + f"MiniMax-M3.1 MoE engine OUT_SKIP: a pushing call skips the dead bf16 out store " + f"(tokens={num_tokens})" + ) + if not early_release: + self.config["early_ar"] = 0 + if not m31_disabled("MOE_TAIL_TRIGGER"): + # TAIL_TRIGGER (-0.035x): release when each CTA's weights producer is done + self.config["early_ar"] = 0 + self.config["tail_trigger"] = 1 + if num_tokens not in _TAIL_TRIGGER_LOGGED: + _TAIL_TRIGGER_LOGGED.add(num_tokens) + from tensorrt_llm.logger import logger as _ttlog + + _ttlog.info( + f"[M3 dispatch] MOE_TAIL_TRIGGER taken (TP8 MoE engine, tokens={num_tokens}, epoch stored early)" + ) + if STAGES_BY_M[num_tokens - 1] > 0: + self.config["stages"] = STAGES_BY_M[num_tokens - 1] + if num_tokens in M8_CFG_BY_M and not m31_disabled("MOE_M8CFG"): + self.config.update(M8_CFG_BY_M[num_tokens]) + # E0@m4: at 4 tokens the early group-0 FC1 stages are issued by FC2 B builder warp 8 instead of the routing + # warp 4 (issue order only, bitwise; TP4 -0.19 / TP8 -0.21 us/call). + if num_tokens == 4: + from tensorrt_llm.logger import logger as _e0log + + _e0log.info( + "MiniMax-M3 MoE engine E0M4 (%s): early group-0 stages on warp 8, tokens=4" % "TP8" + ) + try: + self.mod = mod = _kernel_module(self.config, _ENGINE_PATH) + except AssertionError as err: + raise ValueError( + f"m3_moe: num_tokens={num_tokens}, num_ctas={self.num_ctas} does not fit the shared " + f"memory / TMEM budget (fewer CTAs means more FC2 blocks per CTA): {err}" + ) from err + kw = dict(device=self.device) + self.plan = plan.to(self.device) + self.partials = torch.full( + (mod.G_CAP * mod.U1 * num_tokens * 32,), -1, dtype=torch.int32, **kw + ) + # the FC1 -> FC2 records; with HB_REPL > 1 copies, HB_STRIDE int32 words apart + hb_words = mod.G_CAP * num_tokens * mod.U1 * 4 + if mod.HB_REPL > 1: + hb_words = mod.HB_REPL * mod.HB_STRIDE + self.hbuf32 = torch.zeros(hb_words, dtype=torch.int32, **kw) + self.epochs = torch.zeros(self.num_ctas + mod.NSHC, dtype=torch.int32, **kw) + self.route_dbg = torch.zeros(num_tokens, 8, dtype=torch.int32, **kw) + self.tdbg = torch.zeros(self.num_ctas + mod.NSHC, 64, dtype=torch.int64, **kw) + self.lgbuf = torch.zeros(mod.LG_REPL * mod.LG_STRIDE, dtype=torch.int64, **kw) + self._dummy = torch.zeros(4, dtype=torch.int32, **kw) + self._ar_off = torch.zeros(8, dtype=torch.int64, **kw) + # B2IMG: per-group FC2 B images and their unit counters (int32; every group gains CNT_PER_CALL per call) + self.img32 = torch.zeros(max(1, mod.NG_CAP * mod.IMG_BYTES // 4), dtype=torch.int32, **kw) + self.cnt = torch.zeros(mod.NG_CAP * 8 + 8, dtype=torch.int32, **kw) + # shared expert: h and y_sh as (call tag << 16 | bf16), tag 0 = never written + self.hx = torch.zeros(num_tokens * SHARED_INTER, dtype=torch.int32, **kw) + self.dnx = torch.zeros(num_tokens * HIDDEN, dtype=torch.int32, **kw) + self._fn = None + + def layer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight=None, + fc31_input_scale=None, + shared_weights=None, + ) -> "M3MoeLayer": + """One MoE layer's operands (the TRTLLM-Gen NVFP4 expert buffers, read in place, never copied). + ``shared_weights`` (a ``shared`` state): the shared expert's (gate_up e4m3 [768, 6144], its swizzled UE8M0 + scales, down e4m3 [6144, 384], its swizzled UE8M0 scales).""" + layer = M3MoeLayer( + self, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + e_score_correction_bias, + router_weight, + fc31_input_scale, + ) + if self.shared: + layer.shared_weights = _shared_views(shared_weights, "m3_moe") + layer.shared_mxfp8 = tuple(shared_weights) + if self.mod.SH_PRETILE: + # (gate_up, pre-tiled stages, down, dummy): the kernel streams the pre-tiled copy + gu8, _, dn8, _ = layer.shared_weights + layer.shared_weights = ( + gu8, + _pretiled(*layer.shared_weights), + dn8, + self._dummy.view(torch.uint8), + ) + return layer + + def run(self, layer, hidden, out=None, arp=None) -> torch.Tensor: + m = self.m + if arp is not None and self.config.get("out_skip"): + # OUT_SKIP work counter: pushing calls issued with the out store skipped (host calls: eager or captured). + OUT_SKIP_TAKEN[0] += 1 + if OUT_SKIP_TAKEN[0] == 1: + from tensorrt_llm.logger import logger as _ost_log + + _ost_log.info( + f"[M3 dispatch] OUT_SKIP taken: a pushing MoE engine call skips its bf16 out store " + f"(tokens={m}; first of the counted calls)" + ) + if arp is not None and self.config.get("tail_trigger"): + TAIL_TRIGGER_TAKEN[0] += 1 + if ( + hidden is None + or hidden.dtype != torch.bfloat16 + or tuple(hidden.shape) != (m, HIDDEN) + or not hidden.is_contiguous() + ): + raise ValueError( + f"m3_moe (fused router): expects contiguous bf16 hidden states [{m}, {HIDDEN}]" + ) + fz = ( + hidden.view(-1).view(torch.int32), + layer.router_weight.view(-1), + layer.fc31_input_scale.view(-1), + self.lgbuf, + ) + lgv, xq, xs = self._dummy.view(torch.float32), self._dummy, self._dummy + if out is None: + out = torch.empty(m, HIDDEN, dtype=torch.bfloat16, device=self.device) + sh = layer.shared_weights if self.shared else (self._dummy.view(torch.uint8),) * 4 + args = ( + layer.w3_w1_weight, + layer.w3_w1_weight_scale.view(-1), + layer.w2_weight, + layer.w2_weight_scale.view(-1), + lgv, + layer.bias, + xq, + xs, + layer.escal, + self.plan.view(-1), + self.partials, + self.hbuf32, + self.epochs, + out.view(-1), + self.route_dbg.view(-1), + self.tdbg.view(-1), + *fz, + *sh, + self.hx, + self.dnx, + self._ar_off if arp is None else arp, + layer.w3_w1_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + layer.w2_weight_scale.view(torch.uint8).reshape(-1).view(torch.int32), + self.img32, + self.cnt, + ) + if self._fn is None: + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_moe compiles on its first call: call it once before capture") + import cutlass.cute as cute + + aligns = [ + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 4, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + 16, + ] + sig = [_view(t, a) for t, a in zip(args, aligns)] + key = (_ENGINE_PATH,) + tuple(sorted(self.config.items())) + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_moe {os.path.basename(key[0])} {key[1:]}", + lambda: cute.compile( + self.mod.m3_moe, + *sig, + cute.runtime.make_fake_stream(), + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + self._fn = fn + self._fn(*args, torch.cuda.current_stream(self.device).cuda_stream) + return out + + +class M3MoeLayer: + """One MoE layer on a :class:`M3MoeState`: ``layer(hidden)``. The expert buffers are read in + place, never copied; the per-expert scalars are packed once (layer constants: rebuild the layer after reloading + weights).""" + + shared_weights = None # the kernel's view of the shared expert (a shared state) + shared_mxfp8 = None # the shared expert's (gate_up, its scales, down, its scales) as given + + def __init__( + self, + state, + w3_w1_weight, + w3_w1_weight_scale, + w2_weight, + w2_weight_scale, + fc31_alpha, + fc31_scale_c, + act_alpha, + act_beta, + act_clamp, + fc2_alpha, + bias, + router_weight=None, + fc31_input_scale=None, + ): + e = NUM_EXPERTS + expected = { + "w3_w1_weight": (w3_w1_weight, (e, 2 * INTER, HIDDEN // 2)), + "w3_w1_weight_scale": (w3_w1_weight_scale, (e, 2 * INTER, HIDDEN // 16)), + "w2_weight": (w2_weight, (e, HIDDEN, INTER // 2)), + "w2_weight_scale": (w2_weight_scale, (e, HIDDEN, INTER // 16)), + } + for name, (t, shape) in expected.items(): + if t.element_size() != 1 or tuple(t.shape) != shape or not t.is_contiguous(): + raise ValueError( + f"m3_moe: {name} is {t.dtype} {tuple(t.shape)}, expected contiguous 1-byte {shape}" + ) + self.state = state + self.w3_w1_weight = w3_w1_weight.view(torch.uint8) + self.w3_w1_weight_scale = w3_w1_weight_scale.view(torch.uint8) + self.w2_weight = w2_weight.view(torch.uint8) + self.w2_weight_scale = w2_weight_scale.view(torch.uint8) + self.escal = torch.stack( + [fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha] + ).float() + self.escal = self.escal.reshape(6, e).contiguous() + self.bias = bias.float().contiguous() + if ( + router_weight is None + or fc31_input_scale is None + or router_weight.dtype != torch.float32 + or tuple(router_weight.shape) != (e, HIDDEN) + or not router_weight.is_contiguous() + or fc31_input_scale.dtype != torch.float32 + or fc31_input_scale.numel() != 1 + ): + raise ValueError( + "m3_moe (fused router): needs router_weight f32 [128, 6144] and fc31_input_scale f32 [1]" + ) + self.router_weight = router_weight + self.fc31_input_scale = fc31_input_scale + + def __call__( + self, + hidden: torch.Tensor, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """``hidden``: contiguous bf16 [M, 6144]. ``arp`` (int64 [8], see :func:`ar_push_desc`): also push the output + into an all-reduce exchange.""" + return self.state.run(self, hidden, out, arp=arp) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_prequant.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_prequant.py new file mode 100644 index 000000000000..48a7827f59b3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_prequant.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Hand-off of an MXFP8-quantized activation from the kernel that produced it to the next MXFP8 GEMM. + +A producer that already holds a BF16 row in registers (the multicast all-reduce + RMSNorm, the split-K attention +merge) also writes its MXFP8 form (E4M3 rows + UE8M0 scales in the 128x4-swizzled layout of +``trtllm::mxfp8_quantize(x, True)``) and publishes it here keyed by the BF16 tensor. The MXFP8 linear whose input +is that tensor takes it and skips its own quantize kernel. Every take clears the slot, so a hand-off is only ever +consumed by the first MXFP8 linear that runs after the producer. + +Module state shared across files (one instance each per process): + +* ``SLOT`` (PrequantSlot): published by ``m3_mc_allreduce`` (``quant=True``), the sparse-decode merge and the fused + decode attention (MXFP8 output); cleared by the sparse decode before it runs; taken by the MXFP8 linear dispatch + and by the one-launch QKV + attention + o_proj path (which also peeks ``pending``). +* ``AR_PUSH`` (ArPushSlot): armed by the fast forward around the attention call; the o_proj GEMV takes the target + and marks ``pushed``; the fast forward checks ``take_pushed`` before the attention boundary all-reduce. +* ``MOE_PUSH`` (MoePushSlot): armed by the fast forward before the MoE; the routed-expert engine takes ``engine`` + and the shared expert's down projection takes ``shared``, each marks its pushed output; the fast forward + checks ``take_pushed`` before the MoE boundary all-reduce. +""" + + +class PrequantSlot: + def __init__(self): + self.pending = None # (data_ptr, rows, width, e4m3 [rows, width], scales) + + def publish(self, x, xq, xsf): + self.pending = (x.data_ptr(), x.shape[0], x.shape[-1], xq, xsf) + + def clear(self): + self.pending = None + + def take(self, x): + """(e4m3 [x.shape[0], width], scales) if x is the published tensor, else None.""" + pending, self.pending = self.pending, None + if ( + pending is None + or x.dim() != 2 + or x.data_ptr() != pending[0] + or x.shape[0] != pending[1] + or x.shape[1] != pending[2] + ): + return None + return pending[3], pending[4] + + +SLOT = PrequantSlot() + + +class ArPushSlot: + """The attention boundary's all-reduce exchange for the o_proj GEMV to push into. + + The fast forward arms ``target`` (m3_mx_gemv ``ar`` tuple) around the attention call; the o_proj marks the output + it pushed in ``pushed``; the boundary then runs the all-reduce with ``prepushed`` = the ``ar`` tuple's mode.""" + + def __init__(self): + self.target = None + self.pushed = None # data_ptr of the pushed output + + def arm(self, target): + self.target = target + self.pushed = None + + def take_target(self): + target, self.target = self.target, None + return target + + def take_pushed(self, x): + pushed, self.pushed = self.pushed, None + return pushed is not None and x.data_ptr() == pushed + + +AR_PUSH = ArPushSlot() + + +class MoePushSlot: + """The MoE boundary's all-reduce exchange for its two producers to push into: the routed-expert engine (set A, + ``engine`` = its ``arp`` descriptor) and the shared expert's down projection (set B, ``shared`` = m3_mx_gemv + ``ar`` tuple). Each marks the output it pushed; the boundary then runs the all-reduce with ``prepushed`` = 3.""" + + def __init__(self): + self.engine = None + self.shared = None + self.engine_pushed = None # data_ptr of the pushed routed output + self.shared_pushed = None # data_ptr of the pushed shared output + + def arm(self, engine, shared): + self.engine, self.shared = engine, shared + self.engine_pushed = self.shared_pushed = None + + def disarm(self): + self.engine = self.shared = None + + def take_engine(self): + engine, self.engine = self.engine, None + return engine + + def take_shared(self): + shared, self.shared = self.shared, None + return shared + + def take_pushed(self, shared, routed): + """(routed pushed, shared pushed) for these outputs; clears the marks.""" + e = routed is not None and self.engine_pushed == routed.data_ptr() + s = shared is not None and self.shared_pushed == shared.data_ptr() + self.engine_pushed = self.shared_pushed = None + return e, s + + +MOE_PUSH = MoePushSlot() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_report.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_report.py new file mode 100644 index 000000000000..7f864bd240c6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_report.py @@ -0,0 +1,34 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The fast-path reports a kernel module makes from inside a call: ``taken`` (an engagement line), ``missed`` (a covered +shape that cannot run: raises under ``modeling_v2='require'``), ``uncovered`` and ``capture_missed``. + +The policy (logging once, raising under require) is the target's: ``modeling.py`` installs its functions here at import, +so a kernel reports without importing the target module. Until then every report is a no-op. +""" + +from __future__ import annotations + +from typing import Callable, Optional + + +def _noop(*args, **kwargs) -> None: + return None + + +taken: Callable[..., None] = _noop +missed: Callable[..., None] = _noop +uncovered: Callable[..., None] = _noop +capture_missed: Callable[..., None] = _noop + + +def install( + *, + on_taken: Callable[[str, Optional[int], str], None], + on_missed: Callable[[str, Optional[int], str], None], + on_uncovered: Callable[[str, str], None], + on_capture_missed: Callable[[str, Optional[int], str], None], +) -> None: + """Route this module's reports to the target's policy.""" + global taken, missed, uncovered, capture_missed + taken, missed, uncovered, capture_missed = on_taken, on_missed, on_uncovered, on_capture_missed diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_sampled_accept.py new file mode 100644 index 000000000000..c3c2edd5d078 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_sampled_accept.py @@ -0,0 +1,230 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused sampled verification for the MiniMax-M3 Eagle3 target head (TP-sharded vocabulary, decode rows <= 48). + +Replaces, per decode step, the full-vocabulary all-gather of the target logits, the fused sampling kernel and the +strict-acceptance ops for rows sampled with temperature and top-p (the AgentX contract). Rows with top-k or min-p +active must stay on the generic path; the caller decides. + +Sampling is top-p rejection sampling, distributed over the TP ranks. With w = exp(logit / T - max) over the FULL +vocabulary and Z = sum(w), a token is in the nucleus iff the mass strictly above its weight is at most top_p * Z +(the fused sampling kernel's kept set, ties included). A candidate is drawn proportionally to w and accepted iff it +is in the nucleus; a rejected candidate lies below the nucleus, so the next candidate is drawn over the tokens heavier +than it. The accepted token is distributed as w restricted to the nucleus. The first candidate lands in the nucleus +with probability >= top_p, so most rows decide on their second exchange. + +Per verify row, 8 CTAs (rows <= 32) or 4 CTAs (more rows) on every rank each own a slice of the rank's logits shard, +staged in shared memory once. A candidate is drawn in two stages without per-token noise: each CTA draws a local +candidate by inverse CDF over its slice, and the CTA whose log-mass plus Gumbel noise is largest supplies the +candidate (CTA j wins with probability T_j / sum T, its token with w / T_j). Uniforms come from a counter-based Philox +keyed by the request's seed and offset, the row, the attempt and the CTA. Each round every CTA multicasts one 4-word +record (key, candidate logit, and max and mass in round 0 or the mass above the current candidate later) and reads +the records of all CTAs of the row on all ranks. Every CTA takes the same decision from the same records in the same +order, so all ranks agree without any intra-GPU handshake. Words carry the call's tag: a slot needs no clearing and +two call halves keep a rank one call ahead from overwriting a slot still being read. +""" + +from typing import Optional + +import torch + +from . import m3_report + +# The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/sampledAccept.cu) and called as +# torch.ops.trtllm.minimax_m3_sampled_accept. +MAX_ROWS = 48 # capacity: fixed: the kernel's built rows (decode steps) + + +def _op_args() -> frozenset: + """The optional arguments the loaded trtllm::minimax_m3_sm107tp8_sampled_accept takes (the tp2 op is used as it + is, read-only: an older image's op has neither prewait_params nor early_trigger).""" + global _OP_ARGS + if _OP_ARGS is None: + schema = torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept.default._schema + _OP_ARGS = frozenset(a.name for a in schema.arguments) + return _OP_ARGS + + +_OP_ARGS = None + + +def one_round_k() -> int: + """The SA-1R variant (ONE_ROUND_K) when the loaded op takes it, else 0 (two rounds).""" + return int(ONE_ROUND_K) if "one_round_k" in _op_args() else 0 + + +# The vocab shard limit of trtllm::minimax_m3_sm107tp8_sampled_accept (kMaxVs, kept from the TP2 kernel it copies): +# 200064 / 2 = 100032 columns. +MAX_VS = 100352 # capacity: fixed: the vocabulary shard bound + + +class SampledAcceptWorkspace: + """Exchange slots behind one multicast mapping (two call halves), the (finished rows, call) counters, the per-row + token scratch, an extra-attempt counter and the SA-1R round-0 decided counter. Created collectively, outside + CUDA-graph capture.""" + + def __init__(self, uc, mc_ptr, rank, world, handle=None, comm=None): + self.uc, self.mc_ptr, self.rank, self.world = uc, mc_ptr, rank, world + self.handle, self.comm = handle, comm + dev = uc.device + self.counters = torch.zeros(2, dtype=torch.int32, device=dev) + self.tokens = torch.zeros(MAX_ROWS, dtype=torch.int32, device=dev) + self.extra_attempts = torch.zeros(1, dtype=torch.int32, device=dev) + self.decided = torch.zeros(1, dtype=torch.int32, device=dev) + + @classmethod + def create(cls, mapping, device) -> "SampledAcceptWorkspace": + # SA-1R's records carry the round-0 list words: the size depends on the variant + words = int( + torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept_ws_words( + max(mapping.tp_size, 1), one_round_k() + ) + if one_round_k() + else torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept_ws_words( + max(mapping.tp_size, 1) + ) + ) + if mapping.tp_size == 1: + uc = torch.zeros(words, dtype=torch.int64, device=device) + return cls(uc, uc.data_ptr(), 0, 1) + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + ok = not torch.cuda.is_current_stream_capturing() + if not _mnnvl_workspace_all_succeeded(comm, ok): + comm.Free() + raise RuntimeError( + "SampledAcceptWorkspace.create must run outside CUDA-graph capture on every rank" + ) + handle = _make_mnnvl_mcast_buffer(comm, words * 8, mapping, mapping.is_multi_node()) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("SampledAcceptWorkspace.create failed on another rank") + return cls(uc, mc.data_ptr(), mapping.tp_rank, mapping.tp_size, handle, comm) + + +_WS: Optional[SampledAcceptWorkspace] = None + + +def workspace(mapping, device) -> Optional[SampledAcceptWorkspace]: + global _WS + if _WS is None: + if torch.cuda.is_current_stream_capturing(): + m3_report.capture_missed("spec_tail", None, "the sampled-accept workspace") + return None + _WS = SampledAcceptWorkspace.create(mapping, device) + return _WS + + +def sampled_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + temperatures: torch.Tensor, + top_ks: torch.Tensor, + top_ps: torch.Tensor, + min_ps: Optional[torch.Tensor], + seeds: torch.Tensor, + offsets: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, + ws: Optional[SampledAcceptWorkspace] = None, + use_pdl: bool = True, +): + """Sampled verification (temperature + top-p) from this rank's bf16 logits shard [nreq * (draft_len + 1), V / tp]: + outputs as shard_accept (accepted [nreq, D+1], num_accepted [nreq], gather_ids [nreq]); None + when unavailable. Per-row sampling parameters and Philox (seed, offset) are the spec metadata's, one per row. + top_ks / min_ps are not applied: rows with top-k or min-p active must be routed to the generic path by the + caller.""" + del top_ks, min_ps + nreq, draft_len = draft_tokens.shape + rows = nreq * (draft_len + 1) + if logits.dim() != 2 or logits.shape[0] != rows or rows > MAX_ROWS or logits.shape[1] > MAX_VS: + return None + if logits.dtype != torch.bfloat16 or logits.stride(1) != 1: + return None + ws = ws if ws is not None else workspace(mapping, logits.device) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept( + logits, + int(vocab_offset), + temperatures[:rows].float().contiguous(), + top_ps[:rows].float().contiguous(), + seeds[:rows].long().contiguous(), + offsets[:rows].long().contiguous(), + ws.uc, + ws.mc_ptr, + ws.counters, + ws.tokens, + ws.rank, + ws.world, + draft_tokens.int().contiguous(), + batch_idx.int().contiguous(), + accepted, + num_accepted, + gather_ids, + int(force_total), + next_new, + force_vec, + ws.extra_attempts, + bool(use_pdl), + **{ + name: bool(on) + for name, on in ( + ("prewait_params", PREWAIT_PARAMS), + ("early_trigger", EARLY_TRIGGER), + ) + if name in _op_args() + }, + **({"one_round_k": one_round_k(), "decided": ws.decided} if one_round_k() else {}), + **({"ilp_walk": True} if _ilp_walk(rows) else {}), + ) + if one_round_k(): + m3_report.taken("m3_sampled_accept_1r", rows, f"one_round_k {one_round_k()}") + return accepted, num_accepted, gather_ids + + +# Off at TP8: the sampled verify loads each row's sampling parameters after its PDL dependency wait, and releases its +# dependents at the acceptance epilogue (both are timing only; the same values and tokens either way). +PREWAIT_PARAMS = False +EARLY_TRIGGER = False +# SA-1R: each CTA's round-0 record also carries its largest logits (82: each thread's top 2 pooled, the CTA's top 8 +# plus the bound of everything it does not list), so a row whose first candidate is provably in the nucleus decides in +# round 0. The same tokens, accepted counts and Philox draws as the two-round kernel, bit for bit; 0 = two rounds. +ONE_ROUND_K = 82 # capacity: fixed: a record format, not a size +# SA-PW: the owner walk takes its tokens 8 at a time (the same draw, tokens and Philox draws as the one-token walk, +# bit for bit). Off. +ILP_WALK = False +ILP_WALK_PATH = "m3_sampled_accept_ilp_walk" + + +def _ilp_walk(rows: int) -> bool: + """ILP_WALK when the loaded op takes ilp_walk, logged once per row count as taken.""" + if not ILP_WALK or "ilp_walk" not in _op_args(): + return False + m3_report.taken( + ILP_WALK_PATH, + rows, + "the sampled verify's owner walk takes 8 tokens at a time", + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_select_long_dsl.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_select_long_dsl.py new file mode 100644 index 000000000000..8a83506694bf --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_select_long_dsl.py @@ -0,0 +1,382 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 block select for MiniMax-M3 decode rows of up to 9216 blocks, one row per 384-thread CTA (CuTe DSL). + +Bit for bit the selection of this target's ``m3_select_long`` (``torch.ops.trtllm.minimax_m3_sm107_tp2_select_long``, +one 1024-thread CTA per row), from a CTA of 384 threads: + +* Every block's score is forced (the first ``init_blocks`` and the last ``local_blocks`` of the row's valid blocks) and + mapped to an order key held in registers: thread t owns blocks t, t + 384, ... (4, 11 or 24 of them by the row's + valid count). Larger scores get smaller keys; NaN sorts above +inf. The key is the 1024-thread kernel's unsigned + key with its top bit flipped, so signed Int32 order here is that kernel's order. Every thread loads at a clamped + block and masks the result, so a thread's loads issue back to back. +* 384 is a multiple of 16, so the 16 interleaved groups ``block % 16`` are the lanes' ``lane % 16``. The weakest of + the 16 group maxima bounds the 16th-largest score, so the top 16 is among the keys at or above it. +* The candidates are compacted by a CTA prefix count, each as one int64 (key << 32 | block), and ranked exactly by + that value (one signed compare per pair); more than 1024 of them (keys tied at the bound) rank every candidate + against the whole row instead. +* Warp 0 emits the 16 picked block ids sorted ascending, -1 for an empty or -inf pick. + +The kernel waits on its grid dependency first; it releases the dependent sparse decode right after that wait +(``early_trigger``), once every score of its row is loaded (trigger phase 1) or at its exit (phase 0). Kernels are +compiled per (PDL launch, early trigger, trigger phase) with symbolic shapes and strides; ``prebuild`` compiles and +loads them before the executor's warmup, so a decode step never compiles one. +""" + +from pathlib import Path +from typing import Dict, Tuple + +import cutlass +import cutlass.cute as cute +import torch +from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream, make_fake_tensor +from cutlass.utils.smem_allocator import SmemAllocator + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.cute_dsl_kernels.blackwell.utils import ( + griddepcontrol_launch_dependents, + griddepcontrol_wait, +) + +# The 384-thread select reads its row's valid-block count before its grid wait instead of after it, which takes one +# dependent global load off the post-wait path; the selection is the same either way. Exact: the count +# (MsaMetadata.msa_n_valid_blocks) is written only before the step's first layer, behind a plain launch. +SELECT_N_VALID_PREWAIT = True + +SELECT_THREADS = 384 +SELECT_WARPS = SELECT_THREADS // 32 +# 1M tokens + draft tokens at 128 tokens per block. +MAX_BLOCKS = 9216 # capacity: fixed: the kernel's built blocks per row +MAX_CAND = 1024 # capacity: fixed: the kernel's candidate staging +TOPK = 16 # capacity: fixed: MSA's top-k +NO_KEY = 0x7FFFFFFF # capacity: fixed: a sentinel, not a size (an unused key slot (the reference's 0xffffffff)) +# The key of -inf (the reference's 0xff800000): never emitted. +NEG_INF_KEY = 0x7F800000 # capacity: fixed: a key encoding, not a size +INIT_SCORE = 1.0e30 +LOCAL_SCORE = 1.0e29 +# (keys per thread, rows up to this many valid blocks) +ITEM_CLASSES = ((4, 4 * SELECT_THREADS), (11, 11 * SELECT_THREADS), (24, MAX_BLOCKS)) + +# Shared memory of the phase, int32 words: candidates [MAX_CAND] int64 (key << 32 | block, so one signed compare +# orders them by (key, block)), picked keys / blocks [TOPK] each, per-warp group minima [SELECT_WARPS][16], group +# minima [16], per-warp candidate counts [SELECT_WARPS]. +_CAND = 0 +_PICK_KEY = _CAND + 2 * MAX_CAND +_PICK_BLOCK = _PICK_KEY + TOPK +_WARP_MIN = _PICK_BLOCK + TOPK +_GROUP_MIN = _WARP_MIN + SELECT_WARPS * 16 +_WARP_CNT = _GROUP_MIN + 16 +SMEM_WORDS = _WARP_CNT + SELECT_WARPS +SMEM_BYTES = 4 * SMEM_WORDS + + +def _key(score): + """Order key of a float score: smaller key = larger score, NaN first; signed order = the reference's unsigned.""" + bits = cutlass.Float32(score).bitcast(cutlass.Int32) + return cutlass.select_(bits < cutlass.Int32(0), bits & cutlass.Int32(0x7FFFFFFF), ~bits) + + +def _imin(a, b): + return cutlass.select_(b < a, b, a) + + +def _imax(a, b): + return cutlass.select_(b > a, b, a) + + +def _forced_key(scores, head, block, query, init_blocks, local_start): + raw = scores[head, block, query] + s = cutlass.select_( + block >= local_start, + cutlass.Float32(LOCAL_SCORE), + cutlass.select_(block < init_blocks, cutlass.Float32(INIT_SCORE), cutlass.Float32(raw)), + ) + return _key(s) + + +@cute.jit +def _select_row( + row, + scores: cute.Tensor, # float32 [num_kv_heads, num_blocks, total_q], any strides + n_valid: cute.Tensor, # int32 [>= total_q] + out: cute.Tensor, # int32 [total_q, num_kv_heads, 16] + init_blocks, + local_blocks, + num_kv_heads, + smem: cute.Tensor, # int32 [SMEM_WORDS], shared memory + bar_id: cutlass.Constexpr, + loaded_trigger: cutlass.Constexpr, + raw_prewait, + n_valid_prewait: cutlass.Constexpr, +): + """Select row ``row`` (= query * num_kv_heads + head) with this CTA's SELECT_THREADS threads. Uses named barrier + ``bar_id`` over SELECT_THREADS threads. ``loaded_trigger`` releases the dependent grid once every score of the + row is loaded (trigger phase 1). With ``n_valid_prewait`` the row's valid-block count is ``raw_prewait``, read by + the caller before its grid wait; otherwise it is read here.""" + tid, _, _ = cute.arch.thread_idx() + lane = tid % cutlass.Int32(32) + warp = tid // cutlass.Int32(32) + query = row // num_kv_heads + head = row % num_kv_heads + if cutlass.const_expr(n_valid_prewait): + raw = raw_prewait + else: + raw = n_valid[query] + valid = _imax(_imin(raw, cutlass.Int32(scores.shape[1])), cutlass.Int32(0)) + local_start = _imax(raw - local_blocks, cutlass.Int32(0)) + cand = cute.make_tensor( + cute.recast_ptr(smem.iterator + _CAND, dtype=cutlass.Int64), cute.make_layout(MAX_CAND) + ) + + if tid < TOPK: + smem[_PICK_KEY + tid] = cutlass.Int32(NO_KEY) + smem[_PICK_BLOCK + tid] = cutlass.Int32(0) + + if valid <= TOPK: + # Every valid block is picked (thread t holds block t). + if tid < valid: + smem[_PICK_KEY + tid] = _forced_key(scores, head, tid, query, init_blocks, local_start) + smem[_PICK_BLOCK + tid] = tid + if cutlass.const_expr(loaded_trigger): + griddepcontrol_launch_dependents() + for ci in cutlass.range_constexpr(len(ITEM_CLASSES)): + items = ITEM_CLASSES[ci][0] + lo = TOPK if ci == 0 else ITEM_CLASSES[ci - 1][1] + hi = ITEM_CLASSES[ci][1] + if (valid > lo) & (valid <= hi): + key = cute.make_rmem_tensor(items, cutlass.Int32) + local_min = cutlass.Int32(NO_KEY) + for s in cutlass.range_constexpr(items): + # Every thread loads (a clamped block past the row) so the loads issue back to back, unbranched. + block = tid + cutlass.Int32(s * SELECT_THREADS) + k = _forced_key( + scores, head, _imin(block, valid - 1), query, init_blocks, local_start + ) + key[s] = cutlass.select_(block < valid, k, cutlass.Int32(NO_KEY)) + local_min = _imin(local_min, key[s]) + local_min = _imin(local_min, cute.arch.shuffle_sync_bfly(local_min, 16)) + if lane < 16: + smem[_WARP_MIN + warp * cutlass.Int32(16) + lane] = local_min + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + if cutlass.const_expr(loaded_trigger): + griddepcontrol_launch_dependents() + if tid < 16: + g = cutlass.Int32(NO_KEY) + for w in cutlass.range_constexpr(SELECT_WARPS): + g = _imin(g, smem[_WARP_MIN + w * 16 + tid]) + smem[_GROUP_MIN + tid] = g + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + bound = cutlass.Int32( + -2147483648 + ) # the smallest key: signed keys, unlike the reference's unsigned ones + for g in cutlass.range_constexpr(16): + bound = _imax(bound, smem[_GROUP_MIN + g]) + + # Candidates (key <= bound; unused slots hold NO_KEY > any bound): a CTA prefix count gives the slots. + cnt = cutlass.Int32(0) + for s in cutlass.range_constexpr(items): + cnt += cutlass.select_(key[s] <= bound, cutlass.Int32(1), cutlass.Int32(0)) + incl = cnt + for d in cutlass.range_constexpr(5): + up = cute.arch.shuffle_sync( + incl, _imax(lane - cutlass.Int32(1 << d), cutlass.Int32(0)) + ) + incl += cutlass.select_(lane >= cutlass.Int32(1 << d), up, cutlass.Int32(0)) + if lane == 31: + smem[_WARP_CNT + warp] = incl + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + before = cutlass.Int32(0) + total = cutlass.Int32(0) + for w in cutlass.range_constexpr(SELECT_WARPS): + c = smem[_WARP_CNT + w] + before += cutlass.select_(cutlass.Int32(w) < warp, c, cutlass.Int32(0)) + total += c + slot = before + incl - cnt + for s in cutlass.range_constexpr(items): + if key[s] <= bound: + if slot < MAX_CAND: + cand[slot] = (cutlass.Int64(key[s]) << cutlass.Int64(32)) | cutlass.Int64( + tid + cutlass.Int32(s * SELECT_THREADS) + ) + slot += 1 + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + + if total <= MAX_CAND: + for i in cutlass.range(tid, total, SELECT_THREADS, unroll=1): + mine = cand[i] + rank = cutlass.Int32(0) + for j in cutlass.range(total, unroll=8): + rank += cutlass.select_(cand[j] < mine, cutlass.Int32(1), cutlass.Int32(0)) + if rank < TOPK: + smem[_PICK_KEY + rank] = cutlass.Int32(mine >> cutlass.Int64(32)) + smem[_PICK_BLOCK + rank] = cutlass.Int32(mine & cutlass.Int64(0xFFFFFFFF)) + else: + # More than MAX_CAND keys tied at the bound: rank each candidate against the whole row. + for s in cutlass.range_constexpr(items): + if key[s] <= bound: + block = tid + cutlass.Int32(s * SELECT_THREADS) + rank = cutlass.Int32(0) + for j in cutlass.range(valid, unroll=1): + kj = _forced_key(scores, head, j, query, init_blocks, local_start) + less = (kj < key[s]) | ((kj == key[s]) & (j < block)) + rank += cutlass.select_(less, cutlass.Int32(1), cutlass.Int32(0)) + if rank < TOPK: + smem[_PICK_KEY + rank] = key[s] + smem[_PICK_BLOCK + rank] = block + cute.arch.barrier(barrier_id=bar_id, number_of_threads=SELECT_THREADS) + + if warp == 0: + # The 16 picked block ids ascending (NO_KEY = empty or -inf, emitted as -1), bitonic over the warp. + v = cutlass.Int32(NO_KEY) + if lane < TOPK: + pk = smem[_PICK_KEY + lane] + if (pk != cutlass.Int32(NO_KEY)) & (pk != cutlass.Int32(NEG_INF_KEY)): + v = smem[_PICK_BLOCK + lane] + for size_log in cutlass.range_constexpr(1, 6): + for stride_log in cutlass.range_constexpr(size_log - 1, -1, -1): + other = cute.arch.shuffle_sync_bfly(v, 1 << stride_log) + ascending = (lane & cutlass.Int32(1 << size_log)) == cutlass.Int32(0) + lower = (lane & cutlass.Int32(1 << stride_log)) == cutlass.Int32(0) + v = cutlass.select_(lower == ascending, _imin(v, other), _imax(v, other)) + if lane < TOPK: + out[query, head, lane] = cutlass.select_( + v == cutlass.Int32(NO_KEY), cutlass.Int32(-1), v + ) + + +class _SelectLongDslKernel: + """One row per CTA: the grid wait, then ``_select_row``. ``n_valid_prewait`` reads the row's valid-block count + before the grid wait, so that dependent load leaves the post-wait critical path. That is exact because + ``n_valid`` is written only by ``on_update_kv_lens`` and the host copy in ``prepare()``, both before the step's + first layer and behind a plain (non-PDL) launch, so no grid this select can overlap writes it + (``attention/test_modeling_v2_m3_n_valid_contract.py``).""" + + def __init__( + self, use_pdl: bool, early_trigger: bool, trigger_phase: int, n_valid_prewait: bool + ): + self.use_pdl = use_pdl + self.early_trigger = early_trigger + self.loaded_trigger = (not early_trigger) and trigger_phase == 1 + self.n_valid_prewait = n_valid_prewait + + @cute.kernel + def kernel(self, scores, n_valid, out, init_blocks, local_blocks, num_kv_heads): + row, _, _ = cute.arch.block_idx() + smem = SmemAllocator() + words = smem.allocate_tensor(cutlass.Int32, cute.make_layout(SMEM_WORDS), byte_alignment=16) + raw_prewait = cutlass.Int32(0) + if cutlass.const_expr(self.n_valid_prewait): + raw_prewait = n_valid[row // num_kv_heads] + griddepcontrol_wait() + if cutlass.const_expr(self.early_trigger): + griddepcontrol_launch_dependents() + _select_row( + row, + scores, + n_valid, + out, + init_blocks, + local_blocks, + num_kv_heads, + words, + bar_id=1, + loaded_trigger=self.loaded_trigger, + raw_prewait=raw_prewait, + n_valid_prewait=self.n_valid_prewait, + ) + + @cute.jit + def __call__(self, scores, n_valid, out, init_blocks, local_blocks, stream): + num_kv_heads = scores.shape[0] + self.kernel(scores, n_valid, out, init_blocks, local_blocks, num_kv_heads).launch( + grid=(scores.shape[2] * num_kv_heads, 1, 1), + block=(SELECT_THREADS, 1, 1), + stream=stream, + use_pdl=self.use_pdl, + ) + + +#: Compiled kernels by (PDL launch, early trigger, trigger phase, valid count read before the wait). +kernel_cache: Dict[Tuple[bool, bool, int, bool], object] = {} + + +def kernel_key( + use_pdl: bool, early_trigger: bool, trigger_phase: int +) -> Tuple[bool, bool, int, bool]: + """The compile key of a launch (the trigger phase does not matter with the early trigger), with this target's + ``SELECT_N_VALID_PREWAIT``.""" + return ( + bool(use_pdl), + bool(early_trigger), + 0 if early_trigger else int(trigger_phase), + bool(SELECT_N_VALID_PREWAIT), + ) + + +def _compile(key: Tuple[bool, bool, int, bool]): + if key not in kernel_cache: + heads, blocks, rows = cute.sym_int(), cute.sym_int(), cute.sym_int() + scores = make_fake_tensor( + cutlass.Float32, + (heads, blocks, rows), + stride=(cute.sym_int64(), cute.sym_int64(), cute.sym_int64()), + ) + n_valid = make_fake_compact_tensor(cutlass.Int32, (cute.sym_int(),), stride_order=(0,)) + out = make_fake_tensor( + cutlass.Int32, (rows, heads, TOPK), stride=(cute.sym_int64(), cute.sym_int64(), 1) + ) + kernel_cache[key] = cute_aot.compile_or_load( + f"m3_select_long_dsl {key}", + lambda: cute.compile( + _SelectLongDslKernel(*key), + scores, + n_valid, + out, + cutlass.Int32(0), + cutlass.Int32(0), + make_fake_stream(use_tvm_ffi_env_stream=True), + options="--enable-tvm-ffi", + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + return kernel_cache[key] + + +def select_long_dsl( + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool, + early_trigger: bool, + trigger_phase: int, +) -> None: + """Launch the select on the current stream (compiling its key first if ``prebuild`` did not).""" + _compile(kernel_key(use_pdl, early_trigger, trigger_phase))( + scores, n_valid_blocks, out, init_blocks, local_blocks + ) + + +def prebuild(keys, device) -> list: + """Compile every key and launch each once on a small dummy row (loading its module). Returns the keys.""" + scores = torch.zeros( + 1, 2048, 4, dtype=torch.float32, device=device + ) # capacity: fixed: a dummy prebuild row, not a size + n_valid = torch.full((4,), 2048, dtype=torch.int32, device=device) + out = torch.empty(1, 4, TOPK, dtype=torch.int32, device=device).permute(1, 0, 2) + built = [] + for key in keys: + key = kernel_key(*key) + _compile(key)(scores, n_valid, out, 0, 1) + built.append(key) + return built + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[1] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_shard_accept.py new file mode 100644 index 000000000000..8caaee389d0a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_shard_accept.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Fused greedy Eagle3 verification for the MiniMax-M3 target head (vocab-sharded TP head, verify rows <= 48). + +Replaces the all-gather of the target logits, the argmax and the strict-acceptance ops of a greedy verify step with +one kernel over this rank's BF16 logits shard (the target lm_head's own local GEMM output): + +* Every logit becomes a 64-bit key ``ordered(value) << 32 | ~global_index`` (ties pick the lowest index, as + ``torch.argmax``), max-reduced per row: warp shuffles, then ``atomicMax``. +* The last CTA to finish pushes the rank's per-row key into every rank's exchange slot through a multicast mapping + (Lamport: 0 = empty, the key is never 0), polls its own slots for all ranks, takes the global argmax, re-arms the + slots and runs the strict acceptance. Call n uses slot half n % 2. + +The argmax is taken over the BF16 logits without arithmetic, so the tokens are those of the in-tree path's argmax +over the gathered logits. The kernel is compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/ +shardAccept.cu) and called as ``torch.ops.trtllm.minimax_m3_shard_accept``. +""" + +from dataclasses import dataclass +from typing import Any, Optional + +import torch + +from . import m3_report + +#: Verify rows per call, and exchange rows per call half of the workspace. +SHARD_ROWS = 48 # capacity: fixed: the kernel's built verify rows + + +@dataclass(eq=False) +class AcceptWorkspace: + """Per-rank exchange slots ([2 halves][SHARD_ROWS rows][world] uint64) behind one multicast mapping, the local + key accumulators and the (finished CTAs, call) counters.""" + + uc: torch.Tensor + mc_ptr: int + keys: torch.Tensor + counters: torch.Tensor + rank: int + world_size: int + handle: Any = None + comm: Any = None + + +_WS: Optional[AcceptWorkspace] = None +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def create_workspace(mapping, device, slot_rows: int) -> AcceptWorkspace: + """Exchange slots ([2 halves][slot_rows rows][world] uint64) behind one multicast mapping over the TP group, the + local key accumulators [slot_rows] and the (finished CTAs, call) counters. Collective over the TP group; call + outside CUDA-graph capture, in the same order on every rank.""" + words = 2 * slot_rows * max(mapping.tp_size, 1) + if mapping.tp_size > 1: + from tensorrt_llm._torch.distributed.ops import ( + _make_mnnvl_mcast_buffer, + _mnnvl_workspace_all_succeeded, + ) + from tensorrt_llm._utils import mpi_comm + + session = mpi_comm() + session_group = session.Get_group() + group = session_group.Incl(mapping.tp_group) + session_group.Free() + comm = session.Create_group(group) + group.Free() + if not _mnnvl_workspace_all_succeeded(comm, True): + comm.Free() + raise RuntimeError("exchange workspace creation failed on another rank") + handle = _make_mnnvl_mcast_buffer( + comm, max(words * 8, 4096), mapping, mapping.is_multi_node() + ) + uc = handle.get_uc_buffer(mapping.tp_rank, (words,), torch.int64, 0) + mc = handle.get_mc_buffer((words,), torch.int64, 0) + uc.zero_() + torch.cuda.synchronize() + if not _mnnvl_workspace_all_succeeded(comm, True): + raise RuntimeError("exchange workspace creation failed on another rank") + mc_ptr, rank = mc.data_ptr(), mapping.tp_rank + else: + uc = torch.zeros(words, dtype=torch.int64, device=device) + mc_ptr, rank, handle, comm = uc.data_ptr(), 0, None, None + return AcceptWorkspace( + uc=uc, + mc_ptr=mc_ptr, + keys=torch.zeros(slot_rows, dtype=torch.int64, device=uc.device), + counters=torch.zeros(2, dtype=torch.int32, device=uc.device), + rank=rank, + world_size=max(mapping.tp_size, 1), + handle=handle, + comm=comm, + ) + + +def _workspace(mapping, device) -> Optional[AcceptWorkspace]: + """The exchange workspace, created collectively outside CUDA-graph capture on first use.""" + global _WS + if _WS is not None: + return _WS + if torch.cuda.is_current_stream_capturing(): + m3_report.capture_missed("m3_shard_accept", None, "the shard-accept workspace") + return None + _WS = create_workspace(mapping, device, SHARD_ROWS) + return _WS + + +def shard_accept( + logits: torch.Tensor, + vocab_offset: int, + mapping, + draft_tokens: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor] = None, + force_vec: Optional[torch.Tensor] = None, +): + """Greedy verification from this rank's BF16 logits shard. + + Args: + logits: [nreq * (draft_len + 1) <= SHARD_ROWS, V / tp] BF16, unit column stride: this rank's columns + (global ids ``vocab_offset ..``) of the verify rows, request-major. + draft_tokens: [nreq, draft_len] int32, the drafts being verified. + batch_idx: [nreq] int32, each request's row in the verify layout. + force_total / force_vec: forced acceptance (see ``m3_shard_accept``). + next_new: optional int32 [nreq, >= 1]; column 0 receives the last accepted token. + + Returns: + (accepted_tokens [nreq, draft_len + 1] int32, num_accepted [nreq] int32, first-pass gather ids [nreq] + int64), or None when the shape is not served or the workspace is missing during a capture. Collective over + the TP group. + """ + nreq, draft_len = draft_tokens.shape + if ( + logits.dim() != 2 + or logits.shape[0] != nreq * (draft_len + 1) + or logits.shape[0] > SHARD_ROWS + or logits.dtype != torch.bfloat16 + or logits.stride(1) != 1 + ): + return None + ws = _workspace(mapping, logits.device) + if ws is None: + return None + dev = logits.device + accepted = torch.empty((nreq, draft_len + 1), dtype=torch.int32, device=dev) + num_accepted = torch.empty(nreq, dtype=torch.int32, device=dev) + gather_ids = torch.empty(nreq, dtype=torch.int64, device=dev) + torch.ops.trtllm.minimax_m3_shard_accept( + logits, + vocab_offset, + ws.keys, + ws.counters, + ws.uc, + ws.mc_ptr, + ws.rank, + ws.world_size, + draft_tokens.contiguous(), + batch_idx, + accepted, + num_accepted, + gather_ids, + force_total, + next_new, + _num_sms(), + force_vec, + ) + return accepted, num_accepted, gather_ids diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_spec_tail.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_spec_tail.py new file mode 100644 index 000000000000..996371370fde --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_spec_tail.py @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Small fused kernels of the MiniMax-M3 Eagle3 step (GB300, decode). + +``chunk_norm`` / ``capture_norm`` apply the drafter's per-layer fc_norm RMSNorms to the captured target hidden states +(the in-tree ``apply_eagle3_fc`` norms, same fp32 arithmetic); ``draft_input`` is a fused draft pass's input (the +embedding lookup, the drafter layer's input_layernorm / hidden_norm and their concat, same arithmetic); +``step_gather`` / ``slot_scatter`` move the step's integer outputs between the spec sampler's store and the engine's +input buffers. +""" + +from typing import Optional, Tuple + +import torch + +# The kernels are compiled into TensorRT-LLM (cpp/tensorrt_llm/kernels/minimaxM3/specTail.cu) and called as +# torch.ops.trtllm.minimax_m3_{draft_input, chunk_norm, capture_norm, step_gather, slot_scatter}. + + +def draft_input( + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + eps_embed: float, + w_hidden: torch.Tensor, + eps_hidden: float, + use_pdl: bool = True, +) -> Tuple[torch.Tensor, torch.Tensor]: + """(cat([rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden], -1) [M, 2H], hidden[rows] [M, H]) for + int32 ids [M] and int64 rows [M] (rows None: hidden[:M]); ids outside the table embed to zero.""" + m, h = ids.shape[0], table.shape[1] + out = torch.empty((m, 2 * h), dtype=table.dtype, device=table.device) + residual = torch.empty((m, h), dtype=table.dtype, device=table.device) + torch.ops.trtllm.minimax_m3_draft_input( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, use_pdl + ) + return out, residual + + +def chunk_norm( + src: torch.Tensor, weights: torch.Tensor, eps: float, use_pdl: bool = True +) -> torch.Tensor: + """Per-chunk RMSNorm of src [M, >= C*H] (row stride may exceed C*H) with weights [C, H] -> [M, C*H].""" + c, h = weights.shape + out = torch.empty((src.shape[0], c * h), dtype=src.dtype, device=src.device) + torch.ops.trtllm.minimax_m3_chunk_norm(src, weights, eps, out, use_pdl) + return out + + +# The overlap scheduler's per-step gathers and the spec sampler's slot stores as one kernel each (step_gather / +# slot_scatter) instead of ~10 and 4 small torch ops; called by the target's SpecStepFold. + + +def step_gather( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 +) -> bool: + """See step_gather_kernel; False when the inputs are outside what the kernel covers (caller falls back).""" + ts = (new_tok, next_draft, lens, slots, ppi, ids, draft, pos_off, kv_off) + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or next_draft.dim() != 2 + or new_tok.dim() < 2 + ): + return False + if (new_tok.dim() == 3 and new_tok.shape[2] != 1) or not ( + lens.is_contiguous() + and ids.is_contiguous() + and draft.is_contiguous() + and pos_off.is_contiguous() + and kv_off.is_contiguous() + ): + return False + torch.ops.trtllm.minimax_m3_step_gather( + new_tok, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 + ) + return True + + +def slot_scatter( + slots, src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft +) -> bool: + """See slot_scatter_kernel (2-D int32 views, zero padding); False to fall back.""" + ts = (src_new, dst_new, src_next_new, dst_next_new, src_lens, dst_lens, src_draft, dst_draft) + if any(t.dtype != torch.int32 or not t.is_cuda for t in ts) or slots.dtype != torch.long: + return False + two = (src_new, dst_new, src_next_new, dst_next_new, src_draft, dst_draft) + if any(t.dim() != 2 for t in two) or src_lens.dim() != 1 or dst_lens.dim() != 1: + return False + torch.ops.trtllm.minimax_m3_slot_scatter( + slots, + src_new, + dst_new, + src_next_new, + dst_next_new, + src_lens, + dst_lens, + src_draft, + dst_draft, + ) + return True + + +def capture_norm( + h: torch.Tensor, + residual: Optional[torch.Tensor], + weight: torch.Tensor, + eps: float, + dst: torch.Tensor, +) -> None: + """dst [M, H] (row stride may exceed H) = rmsnorm(h + residual) * weight with chunk_norm's rounding.""" + torch.ops.trtllm.minimax_m3_capture_norm(h, residual, weight, eps, dst) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_target_head.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_target_head.py new file mode 100644 index 000000000000..f5a1b229c777 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_target_head.py @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The target lm_head shard of a verify step of at most 8 rows, streamed from a lossless 12-bit coding of its BF16 +weight (75% of the bytes; ``ops.gemm.m3_target_head`` has the format). + +``prepare_head`` codes this rank's shard once, before the executor's warmup, and keeps it only when it decodes to the +BF16 weight bit for bit (``round_trip_ok``); a shard that does not keeps the in-tree BF16 GEMM. ``logits`` runs the +kernel: BF16 x BF16 MMAs with fp32 accumulation over the full K and one BF16 rounding per logit, the in-tree head's +arithmetic in another summation order. The kernel is compiled into TensorRT-LLM +(cpp/tensorrt_llm/kernels/minimaxM3/targetHead.cu) and called through the catalog entry ``ops.gemm.m3_target_head``. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +import torch + +from ..ops.gemm.m3_target_head import CTA_WARPS, MAX_ROWS, encode_chunked, round_trip_ok +from ..ops.gemm.m3_target_head import m3_target_head as _target_head_op + + +@dataclass(eq=False) +class TargetHead: + """One rank's target lm_head shard in the 12-bit coding, and the kernel's split-tile scratch. + + ``coded`` is False when the coding did not reproduce the weight bit for bit (the in-tree GEMM then runs).""" + + weight: torch.Tensor + coded: bool + w: Optional[torch.Tensor] = None + base: int = 0 + exc_off: Optional[torch.Tensor] = None + exc_pos: Optional[torch.Tensor] = None + exc_w: Optional[torch.Tensor] = None + part: Optional[torch.Tensor] = None + counter: Optional[torch.Tensor] = None + grid: int = 0 + + +_NUM_SMS: Optional[int] = None + + +def _num_sms() -> int: + global _NUM_SMS + if _NUM_SMS is None: + _NUM_SMS = torch.cuda.get_device_properties( + torch.cuda.current_device() + ).multi_processor_count + return _NUM_SMS + + +def head_supported(lm_head) -> bool: + """A bias-free 2-D BF16 head of whole 16-row tiles whose K the coding covers (a multiple of 64).""" + weight = getattr(lm_head, "weight", None) + if weight is None or weight.dim() != 2 or weight.dtype != torch.bfloat16: + return False + return ( + getattr(lm_head, "bias", None) is None + and weight.shape[0] % 16 == 0 + and weight.shape[1] % 64 == 0 + ) + + +def prepare_head(lm_head, grid: Optional[int] = None) -> TargetHead: + """Code ``lm_head``'s local shard and check it decodes to the BF16 weight bit for bit (fails closed: a shard that + does not is returned with ``coded=False`` and nothing kept). Built once, eagerly, before the executor's warmup + (``M3Eagle3Worker.prebuild_target_head``).""" + weight = lm_head.weight + with torch.no_grad(): + w = weight.detach() + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + if not round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w): + return TargetHead(weight=weight, coded=False) + if grid is None: + grid = _num_sms() + n = weight.shape[0] + return TargetHead( + weight=weight, + coded=True, + w=packed, + base=base, + exc_off=exc_off, + exc_pos=exc_pos, + exc_w=exc_w.view(-1), + part=torch.zeros(grid * CTA_WARPS * 2 * 4 * 32, dtype=torch.float32, device=weight.device), + counter=torch.zeros((n + 15) // 16, dtype=torch.int32, device=weight.device), + grid=grid, + ) + + +def logits(head: TargetHead, x: torch.Tensor, use_pdl: bool = True) -> torch.Tensor: + """This rank's BF16 logits shard [rows, n] of the rows x [rows <= MAX_ROWS, K] (BF16) from the coded shard.""" + assert head.coded and 1 <= x.shape[0] <= MAX_ROWS, (head.coded, x.shape) + if x.stride(1) != 1 or x.stride(0) % 8 or x.data_ptr() % 16: + x = x.contiguous() + n = head.weight.shape[0] + out = torch.empty((x.shape[0], n), dtype=torch.bfloat16, device=x.device) + return _target_head_op( + head.w, + x, + out, + head.part, + head.counter, + head.exc_off, + head.exc_pos, + head.exc_w, + head.base, + n, + head.grid, + use_pdl, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_tokmap_guard.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_tokmap_guard.py new file mode 100644 index 000000000000..db0077117fe3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/m3_tokmap_guard.py @@ -0,0 +1,185 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The token map of ``thop.trtllm_gen_generation_preprocess`` for one request with several tokens. + +For multi-token generation the op's QKV preprocessing places every row by the workspace token map (``tokensInfo``, +(request, token) per row), but builds that map only when ``isBuildDecoderInfoKernelNeeded()`` holds, which at one +request with the default RoPE it does not. The op then applies whatever map the last multi-request call left in the +workspace: a row can land on another request's length and pages, and its RoPE position and K/V slot are wrong. This +module wraps the op so that, for one request and several tokens, the map the build would write, (0, t) for each row t, +is written into the op's own map slice of the workspace (the offset from the op's layout function) just before the +call. Every other call passes through unchanged. The rows' positions still come from the request's generation length, +exactly as with a freshly built map. + +The write is a device copy from a map prebuilt at install (no allocation, no host sync), so it is captured into the +decode graphs with the call it precedes. +""" + +from __future__ import annotations + +from typing import Callable, Dict, Optional, Tuple + +import torch + +from tensorrt_llm.bindings.internal import thop +from tensorrt_llm.logger import logger + +#: Argument names of ``thop.trtllm_gen_generation_preprocess`` in order (the nanobind binding). +_ARG_NAMES = ( + "qkv_input", + "workspace", + "sequence_lengths", + "spec_decoding_generation_lengths", + "spec_decoding_position_offsets", + "kv_cache_block_offsets", + "host_kv_cache_pool_pointers", + "host_kv_cache_pool_mapping", + "kv_scale_orig_quant", + "kv_scale_quant_orig", + "attention_output_orig_quant", + "rotary_inv_freq", + "rotary_cos_sin", + "mrope_position_deltas", + "layer_idx", + "seq_offset", + "num_heads", + "num_kv_heads", + "head_size", + "tokens_per_block", + "kv_cache_quant_mode", + "max_attention_window_size", + "cyclic_attention_window_size", + "num_tokens", + "batch_beam", + "input_seq_length", + "max_past_kv_length", + "rotary_embedding_dim", + "rotary_embedding_base", + "rotary_embedding_scale_type", + "rotary_embedding_scale", + "rotary_embedding_max_positions", + "position_embedding_type", + "bmm1_scale", + "bmm2_scale", + "fp8_context_fmha", + "predicted_tokens_per_seq", + "attention_chunk_size", + "multi_processor_count", + "total_num_blocks", + "kv_factor", + "need_build_kv_cache_metadata", + "cross_attention", + "skip_fmha_workspace", +) +_INT2_BYTES = 8 # capacity: fixed: an element size + +_original: Optional[Callable] = None +_max_rows = 0 +_maps: Dict[torch.device, torch.Tensor] = {} +_offsets: Dict[Tuple, int] = {} + + +def _no_taken(rows: int) -> None: + pass + + +def _no_missed(rows: int, reason: str) -> None: + pass + + +_on_taken: Callable[[int], None] = _no_taken +_on_missed: Callable[[int, str], None] = _no_missed + + +def _indexed(device: torch.device) -> torch.device: + """``device`` with its index (an unindexed ``cuda`` means the current device), the key ``_maps`` is built under.""" + if device.type == "cuda" and device.index is None: + return torch.device("cuda", torch.cuda.current_device()) + return device + + +def _one_request_map(device: torch.device, rows: int) -> torch.Tensor: + """The prebuilt (0, t) map's first ``rows`` rows on ``device``.""" + device = _indexed(device) + m = _maps.get(device) + if m is None or m.shape[0] < rows: + reason = f"no {rows}-row token map was prebuilt on {device} (install() builds it before the warmup)" + _on_missed(rows, reason) + raise RuntimeError(f"m3_tokmap_guard: {reason}") + return m[:rows] + + +def _map_offset(args: dict, rows: int) -> int: + x = args["qkv_input"] + skip = bool(args.get("skip_fmha_workspace", False)) + key = ( + x.dtype, + rows, + int(args["num_heads"]), + int(args["head_size"]), + int(args["rotary_embedding_dim"]), + int(args["num_kv_heads"]), + skip, + ) + off = _offsets.get(key) + if off is None: + layout = thop.get_trtllm_gen_generation_workspace_layout( + x.dtype, 1, rows, key[2], key[3], key[4], key[5], 0, False, skip + ) + off, size = int(layout["tokens_info_offset"]), int(layout["tokens_info_size"]) + if off < 0 or off % 4 != 0 or size < _INT2_BYTES * rows: + raise RuntimeError( + f"m3_tokmap_guard: the op's generation layout has no {rows}-row token map " + f"(offset {off}, {size} bytes): the layout changed under this guard" + ) + _offsets[key] = off + return off + + +def _guarded(*args, **kwargs): + a = dict(zip(_ARG_NAMES, args)) + a.update(kwargs) + rows = int(a["num_tokens"]) + if ( + a.get("spec_decoding_generation_lengths") is not None + and int(a["predicted_tokens_per_seq"]) > 1 + and int(a["batch_beam"]) == 1 + and rows > 1 + and not bool(a.get("cross_attention", False)) + ): + ws = a["workspace"] + src = _one_request_map(ws.device, rows) + off = _map_offset(a, rows) + # view (never reshape: a copy would leave the op's own workspace unwritten); raises on a strided workspace. + dst = ws.view(-1).view(torch.uint8)[off : off + _INT2_BYTES * rows] + dst.view(torch.int32).view(rows, 2).copy_(src) + _on_taken(rows) + return _original(*args, **kwargs) + + +def install( + device: torch.device, + max_rows: int, + on_taken: Callable[[int], None], + on_missed: Callable[[int, str], None] = _no_missed, +) -> bool: + """Wrap ``thop.trtllm_gen_generation_preprocess`` (once per process) and prebuild the (0, t) map for up to + ``max_rows`` rows on ``device`` (a one-request call past it is ``on_missed`` and raises). Returns whether this + call installed the wrapper.""" + global _original, _max_rows, _on_taken, _on_missed + device = _indexed(device) + if device not in _maps or _maps[device].shape[0] < max_rows: + t = torch.arange(max_rows, dtype=torch.int32, device=device) + _maps[device] = torch.stack([torch.zeros_like(t), t], dim=1).contiguous() + _max_rows = max(_max_rows, max_rows) + _on_taken = on_taken + _on_missed = on_missed + if _original is not None: + return False + _original = thop.trtllm_gen_generation_preprocess + thop.trtllm_gen_generation_preprocess = _guarded + logger.info( + "[M3 prebuild] m3_tokmap_guard installed: one-request multi-token generations get their (0, t) token map " + f"(up to {max_rows} rows)" + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/msa_kv_update.py new file mode 100644 index 000000000000..8c2a67356d3f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/msa_kv_update.py @@ -0,0 +1,89 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One kernel for MSA's device-side KV-length correction (MsaAttentionMetadata.on_update_kv_lens). + +The torch formulation is ~23 one-CTA kernels (min, cumsum, gathers, div / clamp / remainder, the block-table lookup, +the valid-block counts) and runs at the head of every captured decode step, twice. This kernel computes the same +int32 results in one launch: + + kv_true[b] = min(kv_lens[b], staged[b]) -> seq_lens_out[b], cu_kv[b + 1] = prefix sum (cu_kv[0] = 0) + pos[t] = kv_true[r] - qo[r] + intra[t] (r = q_batch_row[t]) + out_loc[t] = table[r, min(max(pos, 0) // page, ncols - 1)] * page + max(pos, 0) % page + n_valid[t] = (max(pos + 1, 0) + page - 1) // page + +``msa_kv_update`` is the target's ``MsaKvUpdate`` (``tensorrt_llm._torch.pyexecutor.runtime_hooks``): the MSA +metadata calls it with its own buffers and falls back to the torch formulation when it returns False. The kernel is +``torch.ops.trtllm.minimax_m3_msa_kv_update``, called through this target's ``msa_kv_update`` entry. +""" + +from typing import Optional + +import torch + +from ..ops.attention.msa_kv_update import msa_kv_update as _kv_update_op + + +def msa_kv_update( + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], +) -> bool: + """The fused correction (see module doc); False, with nothing written, when an operand is outside what the + kernel covers. Arguments in ``MsaKvUpdate`` order; ``seq_lens`` are the per-request query lengths (``qo``).""" + ts = [ + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + msa_seq_lens, + out_cache_loc, + ] + ts += [t for t in (cu_kv_lens, n_valid_blocks) if t is not None] + if ( + any(t.dtype != torch.int32 or not t.is_cuda for t in ts) + or block_table.dim() != 2 + or block_table.stride(1) != 1 + ): + return False + if any(not t.is_contiguous() for t in ts if t is not block_table): + return False + if ( + kv_lens.numel() < batch + or kv_lens_staged.numel() < batch + or seq_lens.numel() < batch + or msa_seq_lens.numel() < batch + or q_batch_row.numel() < total_q + or q_intra.numel() < total_q + or out_cache_loc.numel() < total_q + or (cu_kv_lens is not None and cu_kv_lens.numel() < batch + 1) + or (n_valid_blocks is not None and n_valid_blocks.numel() < total_q) + ): + return False + _kv_update_op( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + return True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/__init__.py new file mode 100644 index 000000000000..e6c3d806a4a3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3 MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) as a CuTe DSL kernel, configured for the TP2 +projection shapes on SM107 (see op.py).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/m3_mx_gemv_kernel.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/m3_mx_gemv_kernel.py new file mode 100644 index 000000000000..ddf9a8b102ae --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/m3_mx_gemv_kernel.py @@ -0,0 +1,1511 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 decode GEMV on tcgen05: ``y[M, N] = x[M, K] @ W[N, K]^T`` for M <= 8, block-scaled E4M3 x E4M3. + +The weight is the MMA's A operand (128 output features per CTA), the activation is B (8 token columns), +fp32 accumulation in TMEM. Both operands come in their MXFP8 storage: E4M3 values (K contiguous) and UE8M0 +scales in the 128x4-swizzled layout, whose 512-byte atom per (128 rows, 128 K) is exactly the source of a +``tcgen05.cp`` 32x128b.warpx4 scale copy, so both scale tensors are bulk-copied unchanged. For an activation of at +most 32 rows the atom of k-tile ``k`` sits at byte ``512 k``. + +A cluster of SPLIT CTAs shares one 128-row weight tile; rank r streams the interleaved k-tiles r, r + SPLIT, ... +through a RING-stage ring. Before the grid dependency each CTA issues its first RING weight tiles (and their scale +atoms) and prefetches the rest into L2; after it, the activation slice of the CTA (values and scale atoms) lands in +one buffer. Rank r owns rows [r * 128 / SPLIT, (r + 1) * 128 / SPLIT) of the tile: the other ranks store their fp32 +partials of those rows into its shared memory and arrive on its barrier (release, cluster scope); the owner adds the +SPLIT partials in rank order and stores bf16. + +``prod`` (#117, the sparse QKV + index projection at split 8, at most 8 tokens): instead of storing the packed row, the +owners stage their bf16 rows, the CTAs exchange them by st.async so that rank t holds token t's 128-dim row of this +tile's packed head, and rank t's warp 4 runs that head's producer (``m3_layer.phase_producer.producer_head``: RMSNorm, +RoPE, FP8 q / index-q / index-K and NVFP4 K / V stores) on it, bit for bit what the separate producer kernel does with +the stored row. + +Warps: 0 weight TMA, 1 activation TMA, 2 TMEM allocation + scale copies + MMA, 3 idle, 4-7 epilogue (warp 4 + i +reads TMEM lanes 32 i .. 32 i + 31). +""" + +from __future__ import annotations + +import cuda.bindings.driver as cuda_driver +import cutlass +import cutlass.cute as cute +import cutlass.experimental.cuda as cuda +from cutlass import dsl_user_op +from cutlass.experimental import primitives as prims + +from ..m3_layer.phase_producer import ProducerArgs, bf16_bits_to_f32, producer_head + +CTA_M = 128 # capacity: fixed: a kernel tile constant (weight rows per CTA = the MMA's M) +MMA_N = 8 # token columns +CTA_K = 128 # capacity: fixed: a kernel tile constant (one k-tile: 128 E4M3 bytes per row = one 128-byte swizzle row) +MMA_INST_K = 32 +# 4 MMAs per k-tile, one per scale byte of the atom. +K_BLOCKS = CTA_K // MMA_INST_K # capacity: fixed: a kernel tile constant +THREADS = 256 +SF_ATOM_BYTES = 512 +SF_ATOM_U16 = SF_ATOM_BYTES // 2 +SF_COLS = 4 # TMEM columns of one scale atom +W_TILE_BYTES = CTA_M * CTA_K +X_TILE_BYTES = MMA_N * CTA_K +EVICT_FIRST = 0x12F0000000000000 +A_COLS_PER_MMA = MMA_INST_K // 4 # TMEM columns of one 128 x K32 E4M3 granule of A + +a_dtype = cutlass.Float8E4M3FN +b_dtype = cutlass.Float8E4M3FN + + +def num_k_tiles(k_in: int) -> int: + return k_in // CTA_K + + +def acc_cols(nacc: int, mma_n: int = MMA_N) -> int: + # nacc independent accumulators of MMA_N columns: MMA j of the CTA accumulates into j % nacc, so the M128 x N8 + # MMAs do not form one serial dependency chain; the epilogue adds them. + return max(16, nacc * mma_n) + + +def a_col0(nacc: int, my_tiles: int, mma_n: int = MMA_N) -> int: + # a_tmem: [acc | SFA per k-tile | one SFB slot (reused in issue order) | A: 32 columns (128 x 128 E4M3) per k-tile] + return (acc_cols(nacc, mma_n) + (my_tiles + 1) * SF_COLS + 31) // 32 * 32 + + +def tmem_cols( + ring: int, nacc: int, a_tmem: bool = False, my_tiles: int = 0, mma_n: int = MMA_N +) -> int: + need = acc_cols(nacc, mma_n) + 2 * ring * SF_COLS + if a_tmem: + need = a_col0(nacc, my_tiles, mma_n) + my_tiles * K_BLOCKS * A_COLS_PER_MMA + cols = 32 + while cols < need: + cols *= 2 + return cols + + +@dsl_user_op +def read_globaltimer(*, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int64( + _llvm.inline_asm( + _T.i64(), [], "mov.u64 $0, %globaltimer;", "=l", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _mapa_u32(smem_ptr, peer, *, loc=None, ip=None): + """The shared::cluster address of this CTA's shared-memory location in cluster CTA ``peer``.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [smem_ptr.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(peer).ir_value(loc=loc, ip=ip)], + "mapa.shared::cluster.u32 $0, $1, $2;", "=r,r,r", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_async_v4(dst, w0, w1, w2, w3, mbar, *, loc=None, ip=None): + """st.async of four int32 words to a shared::cluster address, completing ``mbar`` there by 16 bytes.""" + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, + [cutlass.Int32(dst).ir_value(loc=loc, ip=ip), cutlass.Int32(w0).ir_value(loc=loc, ip=ip), + cutlass.Int32(w1).ir_value(loc=loc, ip=ip), cutlass.Int32(w2).ir_value(loc=loc, ip=ip), + cutlass.Int32(w3).ir_value(loc=loc, ip=ip), cutlass.Int32(mbar).ir_value(loc=loc, ip=ip)], + "st.async.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [$0], {$1, $2, $3, $4}, [$5];", "r,r,r,r,r,r", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _ld_shared_v4_f32(smem_ptr, *, loc=None, ip=None): + """ld.shared.v4.f32 of four fp32 at a 16-byte-aligned shared-memory pointer.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + out = _llvm.inline_asm( + _llvm.StructType.get_literal([_T.f32()] * 4), [smem_ptr.toint(loc=loc, ip=ip).ir_value()], + "ld.shared.v4.f32 {$0, $1, $2, $3}, [$4];", "=f,=f,=f,=f,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + return [ + cutlass.Float32(_llvm.extractvalue(_T.f32(), out, [i], loc=loc, ip=ip)) for i in range(4) + ] + + +@dsl_user_op +def _ld_shared_v4_b32(smem_ptr, *, loc=None, ip=None): + """ld.shared.v4.b32 of four int32 words at a 16-byte-aligned shared-memory pointer.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + out = _llvm.inline_asm( + _llvm.StructType.get_literal([_T.i32()] * 4), [smem_ptr.toint(loc=loc, ip=ip).ir_value()], + "ld.shared.v4.b32 {$0, $1, $2, $3}, [$4];", "=r,=r,=r,=r,r", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + return [cutlass.Int32(_llvm.extractvalue(_T.i32(), out, [i], loc=loc, ip=ip)) for i in range(4)] + + +@dsl_user_op +def _test_wait_cluster(mbar, parity, *, loc=None, ip=None): + """mbarrier.test_wait.parity.acquire.cluster on a barrier of this CTA completed by other CTAs' st.async.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + done = cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [mbar.toint(loc=loc, ip=ip).ir_value(), cutlass.Int32(parity).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .pred p;\n\tmbarrier.test_wait.parity.acquire.cluster.shared::cta.b64 p, [$1], $2;\n\t" + "selp.u32 $0, 1, 0, p;\n\t}", "=r,r,r,~{memory}", has_side_effects=True, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + return done != cutlass.Int32(0) + + +@dsl_user_op +def _absmax(acc, v, *, loc=None, ip=None): + """max(acc, |v|).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(acc).ir_value(loc=loc, ip=ip), cutlass.Float32(v).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .f32 a;\n\tabs.f32 a, $2;\n\tmax.f32 $0, $1, a;\n\t}", "=f,f,f", has_side_effects=False, + is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _rcp_approx_ftz(x, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Float32( + _llvm.inline_asm( + _T.f32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], "rcp.approx.ftz.f32 $0, $1;", "=f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _e4m3x4(v0, v1, v2, v3, *, loc=None, ip=None): + """Four f32 to E4M3 (RN, satfinite), packed little-endian into one int32 word (v0 in the low byte).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (v0, v1, v2, v3)], + "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t" + "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _ld_volatile_s32(addr, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Int64(addr).ir_value(loc=loc, ip=ip)], "ld.volatile.global.s32 $0, [$1];", "=r,l", + has_side_effects=True, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _st_global_b32(addr, value, *, loc=None, ip=None): + from cutlass._mlir.dialects import llvm as _llvm + + _llvm.inline_asm( + None, [cutlass.Int64(addr).ir_value(loc=loc, ip=ip), cutlass.Int32(value).ir_value(loc=loc, ip=ip)], + "st.global.b32 [$0], $1;", "l,r", has_side_effects=True, is_align_stack=False, + asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) # fmt: skip + + +@dsl_user_op +def _bf16x2_word(lo, hi, *, loc=None, ip=None): + """bf16(lo) | bf16(hi) << 16 (RN), with -0.0 halves turned into +0.0 (0x80000000 marks an empty exchange word).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(hi).ir_value(loc=loc, ip=ip), cutlass.Float32(lo).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b32 w, l, h;\n\t.reg .pred pl, ph;\n\tcvt.rn.bf16x2.f32 w, $1, $2;\n\t" + "and.b32 l, w, 0xFFFF;\n\tsetp.eq.u32 pl, l, 0x8000;\n\t@pl and.b32 w, w, 0xFFFF0000;\n\t" + "shr.u32 h, w, 16;\n\tsetp.eq.u32 ph, h, 0x8000;\n\t@ph and.b32 w, w, 0x0000FFFF;\n\t" + "mov.b32 $0, w;\n\t}", "=r,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _bf16_bits(x, *, loc=None, ip=None): + """bf16(x) (RN) as the fp32 bit pattern of the rounded value.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(x).ir_value(loc=loc, ip=ip)], + "{\n\t.reg .b16 h, z;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _bf16_round(x): + return _bf16_bits(x).bitcast(cutlass.Float32) + + +# The swigluoai activation of MiniMax-M3's dense MLP, gate * sigmoid(1.702 * gate) * (up + 1) with gate clamped to <= 7 +# and up to [-7, 7], in exactly the instruction sequence the Triton silu_and_mul kernel compiles to (modules/swiglu.py +# on SM107): the gate clamp on the packed bf16 pair, the up clamp as min.xorsign.abs, two rounded multiplies into +# ex2.approx, div.full for the reciprocal, two rounded multiplies, round to bf16. No multiply feeds an add, so the +# explicit .rn forms compile to what the Triton kernel's contractable forms do. +SWIGLU_OAI_ALPHA = 1.702 +SWIGLU_OAI_LIMIT = 7.0 + + +@dsl_user_op +def _swiglu_oai_bf16x2(gate_word, up_word, *, loc=None, ip=None): + """The activation of one packed bf16 pair of gate and up values: its two bf16 results as one int32 word (the low + half from the low halves).""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), + [cutlass.Int32(w).ir_value(loc=loc, ip=ip) for w in (gate_word, up_word)], + "{\n\t.reg .b32 g, glim, gl, gh, ul, uh, one, lim, t0, t1, e0, e1, s0, s1, r0, r1;\n\t.reg .b16 hl, hh;\n\t" + "mov.b32 glim, 0x40E040E0;\n\tmin.bf16x2 g, $1, glim;\n\t" + "mov.b32 {hl, hh}, g;\n\tcvt.f32.bf16 gl, hl;\n\tcvt.f32.bf16 gh, hh;\n\t" + "mov.b32 {hl, hh}, $2;\n\tcvt.f32.bf16 ul, hl;\n\tcvt.f32.bf16 uh, hh;\n\t" + "mov.b32 lim, 0f40E00000;\n\tmin.xorsign.abs.f32 ul, ul, lim;\n\tmin.xorsign.abs.f32 uh, uh, lim;\n\t" + "mov.b32 one, 0f3F800000;\n\tadd.rn.f32 ul, ul, one;\n\tadd.rn.f32 uh, uh, one;\n\t" + "mul.rn.f32 t0, gl, 0fBFD9DB23;\n\tmul.rn.f32 t1, gh, 0fBFD9DB23;\n\t" + "mul.rn.f32 t0, t0, 0f3FB8AA3B;\n\tmul.rn.f32 t1, t1, 0f3FB8AA3B;\n\t" + "ex2.approx.f32 e0, t0;\n\tex2.approx.f32 e1, t1;\n\t" + "add.rn.f32 e0, e0, one;\n\tadd.rn.f32 e1, e1, one;\n\t" + "div.full.f32 s0, one, e0;\n\tdiv.full.f32 s1, one, e1;\n\t" + "mul.rn.f32 r0, s0, gl;\n\tmul.rn.f32 r1, s1, gh;\n\t" + "mul.rn.f32 r0, ul, r0;\n\tmul.rn.f32 r1, uh, r1;\n\t" + "cvt.rn.bf16x2.f32 $0, r1, r0;\n\t}", "=r,r,r", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +@dsl_user_op +def _rope_bits(c, a, s, b, *, loc=None, ip=None): + """bf16(c * a + s * b) with separately rounded products (no FMA), as fp32 bits: the stock RoPE arithmetic.""" + from cutlass._mlir.dialects import llvm as _llvm + from cutlass._mlir.extras import types as _T + + return cutlass.Int32( + _llvm.inline_asm( + _T.i32(), [cutlass.Float32(v).ir_value(loc=loc, ip=ip) for v in (c, a, s, b)], + "{\n\t.reg .f32 p, q, r;\n\t.reg .b16 h, z;\n\tmul.rn.f32 p, $1, $2;\n\tmul.rn.f32 q, $3, $4;\n\t" + "add.rn.f32 r, p, q;\n\tcvt.rn.bf16.f32 h, r;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}", "=r,f,f,f,f", + has_side_effects=False, is_align_stack=False, asm_dialect=_llvm.AsmDialect.AD_ATT, loc=loc, ip=ip, + ) + ) # fmt: skip + + +def _din_load4(table, hidden, hidden_half, word_idx): + """Four int32 words at word_idx of the hidden rows (hidden_half == 1) or the embedding table: one CTA-uniform + base-address select instead of a branch.""" + addr = cutlass.select_( + hidden_half == cutlass.Int32(1), hidden.data_ptr(0).toint(), table.data_ptr(0).toint() + ) + ptr = cutlass.inttoptr( + addr + cutlass.Int64(word_idx) * cutlass.Int64(4), mem_space=1, dtype=cutlass.Int32 + ) + return ptr.load(count=4, alignment=16) + + +def _stamp(timers, timed: bool, bx, slot: int): + # Diagnostic: %globaltimer of milestone `slot` of CTA bx (timers [grid, 8] int64). + if timed: + timers.store(read_globaltimer(), idx=bx * cutlass.Int32(8) + cutlass.Int32(slot)) + + +def _k_tile(rank, i: int, split: int, my_tiles: int, contig: bool): + # contig: rank r streams k-tiles [r * my_tiles, (r + 1) * my_tiles); else the interleaved r, r + split, ... + if contig: + return rank * cutlass.Int32(my_tiles) + cutlass.Int32(i) + return rank + cutlass.Int32(i * split) + + +@cute.kernel +def m3_mx_gemv_kernel( + tma_desc_w: cutlass.GridConstant[cuda.TensorMap], # W [N, K] E4M3 as bytes, box 128 x 128 + tma_desc_wsf: cutlass.GridConstant[ + cuda.TensorMap + ], # W scale atoms [N / 128 * K / 128, 256] u16 + tma_desc_x: cutlass.GridConstant[cuda.TensorMap], # x [M, K] E4M3 as bytes, box 128 x 8 + tma_desc_xsf: cutlass.GridConstant[cuda.TensorMap], # x scale atoms [K / 128, 256] u16 + y: cutlass.Array, # [M * N] bf16, token-major + x_words: cutlass.Array, # x_bf16: the bf16 activation [M, K] as int32 words + timers: cutlass.Array, # [grid, 8] int64 milestones when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, # ar_push: the all-reduce workspace's multicast mapping (int32 words) + ar_flags: cutlass.Int64, # ar_push: its call flags (parity of the next call = flags[0] & 1) + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[ + int + ], # 1: push the output rows into the next all-reduce; 2: and trigger after it + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], # token columns of the MMA (8, 16 or 48): M <= mma_n + din_i: cutlass.Array, # din: int32 token ids [M] + din_table: cutlass.Array, # din: embedding [V, K / 2] bf16 as int32 words + din_hidden: cutlass.Array, # din: hidden rows [M, K / 2] bf16 as int32 words + din_w: cutlass.Array, # din: [embedding norm | hidden norm] weights [K] bf16 as int32 words + din_f: cutlass.Array, # din: fp32 [eps_e, eps_h] + din_vocab: cutlass.Int32, + din_g: cutlass.Array, # din, flag 1: int64 hidden row of each token, as int32 words + din_res: cutlass.Array, # din, flag 2: the gathered hidden rows out [M, K / 2] bf16 as int32 words + din_flags: cutlass.Int32, # din: 1 = gather the hidden rows, 2 = write them to din_res + rope_cs: cutlass.Array, # rope: fp32 (cos, sin) [positions][64][2] + rope_i: cutlass.Array, # rope: int32 cache lengths [M / T] + rope_bt: cutlass.Array, # rope: int32 K / V page tables [M / T][2][max_blocks] + rope_q: cutlass.Array, # rope: E4M3 Q out [M][hq][128] as uint8 + rope_pool: cutlass.Array, # rope: paged E4M3 cache [pages][hkv][tokens per page][128] as uint8 + rope_scale: cutlass.Array, # rope: fp32 [1] E4M3 quantization scale + rope_dims: cutlass.Array, # rope: int32 [max_blocks, tokens-per-page log2, hq, hkv, T] + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], + x_swiglu: cutlass.Constexpr[ + bool + ], # x_bf16: x_words is [M, 2K] gate | up; the activation is swigluoai(gate, up) + # gu_swiglu: the weight is a gate_up projection with its rows interleaved per 128-row tile (rows 0..63 = gate + # features 64 i .. 64 i + 63, rows 64..127 = their up rows; gu_interleave). The owners pair each gate row with its + # up + # row (the RoPE ownership) and store swigluoai(bf16(gate), bf16(up)) into y [M, N / 2] instead of the rows. + gu_swiglu: cutlass.Constexpr[bool], + # prod: the producer's addresses and strides (ProducerArgs fields; `packed` is unused), then num_pages and eps. + pr_q: cutlass.Int64, + pr_iq: cutlass.Int64, + pr_ik: cutlass.Int64, + pr_kvd: cutlass.Int64, + pr_kvs: cutlass.Int64, + pr_loc: cutlass.Int64, + pr_quant: cutlass.Int64, + pr_pos: cutlass.Int64, + pr_rope: cutlass.Int64, + pr_qw: cutlass.Int64, + pr_kw: cutlass.Int64, + pr_iqw: cutlass.Int64, + pr_ikw: cutlass.Int64, + pr_ips: cutlass.Int64, + pr_its: cutlass.Int64, + pr_dps: cutlass.Int64, + pr_dls: cutlass.Int64, + pr_dts: cutlass.Int64, + pr_sps: cutlass.Int64, + pr_sls: cutlass.Int64, + pr_pages: cutlass.Int32, + pr_eps: cutlass.Float32, + prod: cutlass.Constexpr[bool], +): + k_tiles = num_k_tiles(k_in) + x_tile_bytes = mma_n * CTA_K + my_tiles = k_tiles // split + rows_per_rank = CTA_M // split + tx, _, _ = cute.arch.thread_idx() + bx, _, _ = cute.arch.block_idx() + warp_id = cute.arch.warp_idx() + rank = cutlass.Int32(0) + if cutlass.const_expr(split > 1): + rank = cute.arch.block_idx_in_cluster() + tma_ptr_w = tma_desc_w.get_ptr() + tma_ptr_wsf = tma_desc_wsf.get_ptr() + tma_ptr_x = tma_desc_x.get_ptr() + tma_ptr_xsf = tma_desc_xsf.get_ptr() + n_tile = bx // cutlass.Int32(split) + m_offset = n_tile * cutlass.Int32(CTA_M) + + smem_a = cutlass.Array( + cutlass.Uint8, ring * W_TILE_BYTES, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_b = cutlass.Array( + cutlass.Uint8, my_tiles * x_tile_bytes, space=cutlass.AddressSpace.smem, alignment=1024 + ) + smem_sfa = cutlass.Array( + cutlass.Uint8, ring * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + smem_sfb = cutlass.Array( + cutlass.Uint8, my_tiles * SF_ATOM_BYTES, space=cutlass.AddressSpace.smem, alignment=128 + ) + w_full = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + mma_done = cutlass.Array(cutlass.Int64, ring, space=cutlass.AddressSpace.smem, alignment=8) + act_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + acc_done = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + reduce_ready = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + tmem_ptr_i32 = cutlass.Array(cutlass.Int32, 1, space=cutlass.AddressSpace.smem) + # Partials of the rows this rank owns: [source rank][row][token] fp32 (its own slot stays unused). Rows are padded + # to mma_n + 4 words: the owner's 16-byte row reads (lane = row) then hit all 32 banks once per 8 lanes; at a + # stride of mma_n they were 8- to 16-way conflicted (4.3 us of the 48-token QKV tail). + mb_stride = mma_n + 4 + mailbox = cutlass.Array( + cutlass.Float32, + split * rows_per_rank * mb_stride, + space=cutlass.AddressSpace.smem, + alignment=16, + ) + # prod: this CTA's bf16 owner rows by token ([8][rows_per_rank]), and the packed-head row of token `rank` gathered + # from the cluster (128 bf16 as 64 words, completed on pr_full by 8 x rows_per_rank * 2 bytes of st.async). + pr_stage = None + pr_row = None + pr_full = None + if cutlass.const_expr(prod): + pr_stage = cutlass.Array( + cutlass.Uint16, 8 * rows_per_rank, space=cutlass.AddressSpace.smem, alignment=16 + ) + pr_row = cutlass.Array(cutlass.Int32, 64, space=cutlass.AddressSpace.smem, alignment=16) + pr_full = cutlass.Array(cutlass.Int64, 1, space=cutlass.AddressSpace.smem, alignment=8) + # din: per-token RMS factor of the half of x this CTA's k-tiles lie in. + din_rs = None + if cutlass.const_expr(din): + din_rs = cutlass.Array(cutlass.Float32, 8, space=cutlass.AddressSpace.smem, alignment=16) + + if warp_id == 0: + prims.prefetch_tensormap(tma_ptr_w) + prims.prefetch_tensormap(tma_ptr_wsf) + prims.prefetch_tensormap(tma_ptr_x) + prims.prefetch_tensormap(tma_ptr_xsf) + if prims.elect_sync(): + for s in cutlass.range_constexpr(ring): + prims.mbarrier_init(w_full.subview(s), 1) + prims.mbarrier_init(mma_done.subview(s), 1) + prims.mbarrier_init(act_full, 1) + prims.mbarrier_init(acc_done, 1) + if cutlass.const_expr(split > 1 and st_async): + # Completed by the peers' st.async bytes; this CTA's own arrival carries the expected count. + prims.mbarrier_init(reduce_ready, 1) + prims.mbarrier_arrive_expect_tx( + reduce_ready, (split - 1) * rows_per_rank * mma_n * 4 + ) + elif cutlass.const_expr(split > 1): + prims.mbarrier_init(reduce_ready, (split - 1) * rows_per_rank) + if cutlass.const_expr(prod): + # Completed by the 8 ranks' st.async (this one's included) only on a rank that receives a token row. + prims.mbarrier_init(pr_full, 1) + if rank < num_tokens: + prims.mbarrier_arrive_expect_tx(pr_full, split * rows_per_rank * 2) + if warp_id == 2: + prims.tcgen05_alloc(tmem_ptr_i32, tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n)) + prims.tcgen05_relinquish_alloc_permit() + prims.fence_mbarrier_init() + if cutlass.const_expr(split > 1): + prims.barrier_cluster_arrive_relaxed() + prims.barrier_cluster_wait() + prims.barrier_cta_sync(0) + if tx == 0: + _stamp(timers, timed, bx, 0) + + if warp_id == 0: + if prims.elect_sync(): + for i in cutlass.range_constexpr(ring, my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [k_global * cutlass.Int32(CTA_K), m_offset], [] + ) + if cutlass.const_expr((n_out // CTA_M) * split >= 80): + # Wide grids launch in part only after the predecessor (e.g. the fused attention) frees its SMs, which + # loses their pre-wait weight prefetch: the CTAs of the first half also stage their partner's + # (bx + grid / 2) weights into L2. + if bx < cutlass.Int32((n_out // CTA_M) * split // 2): + pb = bx + cutlass.Int32((n_out // CTA_M) * split // 2) + p_rank = pb % cutlass.Int32(split) + p_m_offset = (pb // cutlass.Int32(split)) * cutlass.Int32(CTA_M) + for i in cutlass.range_constexpr(my_tiles): + p_k = _k_tile(p_rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_prefetch( + tma_ptr_w, [p_k * cutlass.Int32(CTA_K), p_m_offset], [] + ) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + if cutlass.const_expr(i >= ring): + while not cute.arch.mbarrier_try_wait( + mma_done.subview(stage).data_ptr(), (i // ring - 1) % 2 + ): + pass + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.mbarrier_arrive_expect_tx(w_full.subview(stage), W_TILE_BYTES + SF_ATOM_BYTES) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_a.subview(stage * W_TILE_BYTES), + tma_ptr_w, + (k_global * cutlass.Int32(CTA_K), m_offset), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfa.subview(stage * SF_ATOM_BYTES), + tma_ptr_wsf, + (cutlass.Int32(0), n_tile * cutlass.Int32(k_tiles) + k_global), + w_full.subview(stage), + l2_cache_hint=EVICT_FIRST, + ) + elif warp_id == 1: + prims.griddepcontrol(prims.GridDepAction.WAIT) + if cutlass.const_expr(trigger_early): + # Dependents launch once this grid's own dependency is satisfied (not at entry): consumers that read + # step metadata or caches before their own grid wait rely on every grid ahead of them having passed its + # wait, so the chain of releases never runs ahead of a producer still in flight. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if tx == 32: + _stamp(timers, timed, bx, 1) + if cutlass.const_expr(not x_bf16): + if prims.elect_sync(): + prims.mbarrier_arrive_expect_tx(act_full, my_tiles * (x_tile_bytes + SF_ATOM_BYTES)) + for i in cutlass.range_constexpr(my_tiles): + k_global = _k_tile(rank, i, split, my_tiles, contig) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_b.subview(i * x_tile_bytes), + tma_ptr_x, + (k_global * cutlass.Int32(CTA_K), cutlass.Int32(0)), + act_full, + ) + prims.cp_async_bulk_tensor_shared_cta_global( + smem_sfb.subview(i * SF_ATOM_BYTES), + tma_ptr_xsf, + (cutlass.Int32(0), k_global), + act_full, + ) + elif warp_id == 2: + tmem_raw = tmem_ptr_i32.load() + base_col = tmem_raw & 0xFFFF + base_row = tmem_raw >> 16 + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + ring * SF_COLS + idesc = prims.Tcgen05MxInstrDesc.build( + a_dtype=a_dtype, b_dtype=b_dtype, scale_format=1, n_dim=mma_n, m_dim=CTA_M + ) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + if cutlass.const_expr(a_tmem): + # A staged in TMEM: before the grid wait every weight k-tile and its scale atom go on into TMEM as they + # land (the copies of a stage refilled later are committed to its mma_done); once the activation lands, + # its scale atoms and the MMAs, A read from TMEM. + base_row = tmem_raw >> 16 + base_col = tmem_raw & 0xFFFF + sfa_col0 = base_col + acc_cols(nacc, mma_n) + sfb_col0 = sfa_col0 + my_tiles * SF_COLS + a_base = tmem_raw + cutlass.Int32(a_col0(nacc, my_tiles, mma_n)) + s2t_shape, s2t_multicast = prims.S2TCopyMode.S2T_32x128b_WARPX4 + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + if prims.elect_sync(): + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_cp( + prims.Tcgen05CpShape.SHAPE_128X256B, + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), + 6, + cutlass.Int32, + ), + desc_a + ((MMA_INST_K >> 4) * kb), + group=prims.CTAGroup.CTA_1, + ) + prims.tcgen05_cp( + s2t_shape, + cutlass.inttoptr((base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32), + desc_sfa, multicast=s2t_multicast, + ) # fmt: skip + if cutlass.const_expr(i + ring < my_tiles): + prims.tcgen05_commit(mma_done.subview(stage)) + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + if prims.elect_sync(): + for i in cutlass.range_constexpr(my_tiles): + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + sfa_tmem = cutlass.inttoptr( + (base_row << 16) | (sfa_col0 + i * SF_COLS), 6, cutlass.Int32 + ) + sfb_tmem = cutlass.inttoptr((base_row << 16) | sfb_col0, 6, cutlass.Int32) + prims.tcgen05_cp(s2t_shape, sfb_tmem, desc_sfb, multicast=s2t_multicast) + for kb in cutlass.range_constexpr(K_BLOCKS): + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw, 6, cutlass.Float32), + cutlass.inttoptr( + a_base + cutlass.Int32((i * K_BLOCKS + kb) * A_COLS_PER_MMA), 6, cutlass.Int32 + ), + desc_b + ((MMA_INST_K >> 4) * kb), idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), + i > 0 or kb > 0, + sfa_tmem, sfb_tmem, + ) # fmt: skip + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + else: + if cutlass.const_expr(x_bf16): + # Completed by a thread's arrive: a suspended try_wait wakes late on those. + while not cute.arch.mbarrier_test_wait(act_full.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(act_full.data_ptr(), 0): + pass + if tx == 64: + _stamp(timers, timed, bx, 2) + for i in cutlass.range_constexpr(my_tiles): + stage = i % ring + while not cute.arch.mbarrier_try_wait( + w_full.subview(stage).data_ptr(), (i // ring) % 2 + ): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + sfa_tmem = (base_row << 16) | (sfa_col0 + stage * SF_COLS) + sfb_tmem = (base_row << 16) | (sfb_col0 + stage * SF_COLS) + desc_sfa = prims.Tcgen05SmemDesc.build( + smem_sfa.subview(stage * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_sfb = prims.Tcgen05SmemDesc.build( + smem_sfb.subview(i * SF_ATOM_BYTES), leading_byte_offset=16, stride_byte_offset=128, + base_offset=0, layout=0, + ) # fmt: skip + desc_a = prims.Tcgen05SmemDesc.build( + smem_a.subview(stage * W_TILE_BYTES), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + desc_b = prims.Tcgen05SmemDesc.build( + smem_b.subview(i * x_tile_bytes), leading_byte_offset=16, stride_byte_offset=1024, + base_offset=0, layout=2, + ) # fmt: skip + if prims.elect_sync(): + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), desc_sfa, + multicast=s2t_multicast) # fmt: skip + prims.tcgen05_cp(s2t_shape, cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), desc_sfb, + multicast=s2t_multicast) # fmt: skip + for kb in cutlass.range_constexpr(K_BLOCKS): + inc = ((MMA_INST_K * 8 // 8) >> 4) * kb + j = i * K_BLOCKS + kb + prims.tcgen05_mma_block_scale( + prims.MMABlockScaleKind.MXF8F6F4, prims.CTAGroup.CTA_1, + cutlass.inttoptr(tmem_raw + cutlass.Int32((j % nacc) * mma_n), 6, cutlass.Float32), + desc_a + inc, desc_b + inc, idesc.set_sf_ids(a_sf_id=kb, b_sf_id=kb), j >= nacc, + cutlass.inttoptr(sfa_tmem, 6, cutlass.Int32), cutlass.inttoptr(sfb_tmem, 6, cutlass.Int32), + ) # fmt: skip + prims.tcgen05_commit(mma_done.subview(stage)) + if tx == 64: + _stamp(timers, timed, bx, 3) + if prims.elect_sync(): + prims.tcgen05_commit(acc_done) + elif warp_id >= 4: + lane = tx % 32 + row = (warp_id % 4) * 32 + lane + if cutlass.const_expr(x_bf16): + # The bf16 activation of this CTA's k-tiles to MXFP8 in shared memory, as + # torch.ops.trtllm.mxfp8_quantize(x, True) does (UE8M0 = round-up(qz_amax * rcp(448)), + # E4M3 RN-satfinite): one 32-element block (token qz_t, k-tile qz_i, block qz_kb) per thread and round, + # the E4M3 bytes into the 128-byte-swizzled B tile, the qz_scale byte into the tile's qz_scale atom. + qz_tid = tx - cutlass.Int32(128) + prims.griddepcontrol(prims.GridDepAction.WAIT) + din_half_k = k_in // 2 + din_hidden_half = cutlass.Int32(0) + if cutlass.const_expr(din): + # Draft input x = [rmsnorm(table[ids]) * w_e, rmsnorm(hidden) * w_h] (an out-of-range id embeds to + # zero): this CTA's k-tiles lie in one half, so it needs that half's statistic of every token. 16 + # threads per token, 16-byte loads, all of a thread's loads in flight at once. + din_hidden_half = cutlass.select_( + rank * cutlass.Int32(my_tiles * CTA_K) >= cutlass.Int32(din_half_k), + cutlass.Int32(1), + cutlass.Int32(0), + ) + din_tok = qz_tid // cutlass.Int32(16) + din_part = qz_tid % cutlass.Int32(16) + din_tok_ld = cutlass.select_(din_tok < num_tokens, din_tok, cutlass.Int32(0)) + din_id = din_i.load(idx=din_tok_ld) + din_valid = (din_id >= cutlass.Int32(0)) & (din_id < din_vocab) + din_hrow = din_tok_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + din_hrow = din_g.load(idx=din_tok_ld * cutlass.Int32(2)) + din_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + din_hrow, + cutlass.select_(din_valid, din_id, cutlass.Int32(0)), + ) + # The first hidden-half CTA of tile 0 reads whole hidden rows here: it also writes them out. + din_copy = ( + ((din_flags & cutlass.Int32(2)) != cutlass.Int32(0)) + & (bx == cutlass.Int32(split // 2)) + & (din_hidden_half == cutlass.Int32(1)) + & (din_tok < num_tokens) + ) + din_base = din_row * cutlass.Int32(din_half_k // 2) + din_ss = cutlass.Float32(0.0) + din_vecs_per_part = din_half_k // 8 // 16 + for din_round in cutlass.range_constexpr(0, din_vecs_per_part, 16): + din_n = min(16, din_vecs_per_part - din_round) + din_loaded = [] + for din_j in cutlass.range_constexpr(din_n): + din_widx = din_base + ( + din_part + cutlass.Int32((din_round + din_j) * 16) + ) * cutlass.Int32(4) + din_loaded.append( + _din_load4(din_table, din_hidden, din_hidden_half, din_widx) + ) + if din_copy: + for din_j in cutlass.range_constexpr(din_n): + din_v = din_loaded[din_j] + din_res.store( + (din_v[0], din_v[1], din_v[2], din_v[3]), + idx=din_tok * cutlass.Int32(din_half_k // 2) + + (din_part + cutlass.Int32((din_round + din_j) * 16)) + * cutlass.Int32(4), + alignment=16, + ) + for din_j in cutlass.range_constexpr(din_n): + for din_e in cutlass.range_constexpr(4): + din_word = cutlass.Int32(din_loaded[din_j][din_e]) + din_lo = (din_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + din_hi = (din_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + din_ss = din_ss + din_lo * din_lo + din_hi * din_hi + for din_o in cutlass.range_constexpr(4): + din_ss = din_ss + cute.arch.shuffle_sync_bfly(din_ss, offset=1 << din_o) + if din_part == cutlass.Int32(0): + din_eps = din_f.load(idx=din_hidden_half) + din_r = cute.math.rsqrt( + din_ss / cutlass.Float32(din_half_k) + din_eps, fastmath=True + ) + din_keep = din_valid | (din_hidden_half == cutlass.Int32(1)) + din_rs.store( + cutlass.select_(din_keep, din_r, cutlass.Float32(0.0)), idx=din_tok + ) + prims.barrier_cta_sync(3, thread_count=128) + qz_b_words = cutlass.Array(smem_b.data_ptr(0), shape=(my_tiles * x_tile_bytes // 4,), dtype=cutlass.Int32, + alignment=16) # fmt: skip + qz_items = my_tiles * K_BLOCKS * mma_n + qz_rounds = (qz_items + 127) // 128 + # A group's loads first (clamped to a valid row), then its math: one L2 round trip per group of + # QZ_GROUP rounds instead of one per round, without holding every round's data in registers. + QZ_GROUP = qz_rounds if qz_rounds <= 8 else 4 + qz_loaded = [] + for qz_r in cutlass.range_constexpr(qz_rounds): + qz_item = qz_tid + cutlass.Int32(qz_r * 128) + qz_t = qz_item % cutlass.Int32(mma_n) + qz_kb = (qz_item // cutlass.Int32(mma_n)) % cutlass.Int32(K_BLOCKS) + qz_i = qz_item // cutlass.Int32(mma_n * K_BLOCKS) + if cutlass.const_expr(qz_r * 128 + 127 >= qz_items): + qz_i = cutlass.select_( + qz_item < cutlass.Int32(qz_items), qz_i, cutlass.Int32(0) + ) + qz_k_global = rank + qz_i * cutlass.Int32(split) + if cutlass.const_expr(contig): + qz_k_global = rank * cutlass.Int32(my_tiles) + qz_i + qz_t_ld = cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + qz_x_base = ( + qz_t_ld * cutlass.Int32(k_in if x_swiglu else k_in // 2) + + qz_k_global * cutlass.Int32(CTA_K // 2) + + qz_kb * cutlass.Int32(MMA_INST_K // 2) + ) + qz_wvecs = None + if cutlass.const_expr(din): + # The draft input's raw values (table row of the id / hidden row) and its norm weights. + qz_col = qz_k_global * cutlass.Int32(CTA_K) + qz_kb * cutlass.Int32(MMA_INST_K) + qz_hcol = qz_col - din_hidden_half * cutlass.Int32(din_half_k) + qz_id = din_i.load(idx=qz_t_ld) + qz_hrow = qz_t_ld + if (din_flags & cutlass.Int32(1)) != cutlass.Int32(0): + qz_hrow = din_g.load(idx=qz_t_ld * cutlass.Int32(2)) + qz_row = cutlass.select_( + din_hidden_half == cutlass.Int32(1), + qz_hrow, + cutlass.select_( + (qz_id >= cutlass.Int32(0)) & (qz_id < din_vocab), + qz_id, + cutlass.Int32(0), + ), + ) + qz_src_w = qz_row * cutlass.Int32(din_half_k // 2) + qz_hcol // cutlass.Int32(2) + qz_vecs = [ + _din_load4( + din_table, + din_hidden, + din_hidden_half, + qz_src_w + cutlass.Int32(qz_v * 4), + ) + for qz_v in range(4) + ] + qz_wvecs = [ + din_w.load( + idx=qz_col // cutlass.Int32(2) + cutlass.Int32(qz_v * 4), + vector_size=4, + alignment=16, + ) + for qz_v in range(4) + ] + else: + qz_vecs = [ + x_words.load( + idx=qz_x_base + cutlass.Int32(qz_v * 4), vector_size=4, alignment=16 + ) + for qz_v in range(4) + ] + if cutlass.const_expr(x_swiglu): + # The up half of the same columns, K values after the gate in the row. + qz_wvecs = [ + x_words.load( + idx=qz_x_base + cutlass.Int32(k_in // 2 + qz_v * 4), + vector_size=4, + alignment=16, + ) + for qz_v in range(4) + ] + qz_loaded.append((qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs)) + if cutlass.const_expr((qz_r + 1) % QZ_GROUP == 0 or qz_r + 1 == qz_rounds): + for qz_g in cutlass.range_constexpr(len(qz_loaded)): + qz_item, qz_t, qz_kb, qz_i, qz_vecs, qz_wvecs = qz_loaded[qz_g] + qz_vals = [] + qz_amax = cutlass.Float32(0.0) + if cutlass.const_expr(din): + qz_rs = din_rs.load( + idx=cutlass.select_(qz_t < num_tokens, qz_t, cutlass.Int32(0)) + ) + for qz_v in cutlass.range_constexpr(4): + for qz_e4 in cutlass.range_constexpr(4): + qz_word = cutlass.Int32(qz_vecs[qz_v][qz_e4]) + if cutlass.const_expr(x_swiglu): + qz_word = _swiglu_oai_bf16x2( + qz_word, cutlass.Int32(qz_wvecs[qz_v][qz_e4]) + ) + qz_lo = (qz_word << cutlass.Int32(16)).bitcast(cutlass.Float32) + qz_hi = (qz_word & cutlass.Int32(-65536)).bitcast(cutlass.Float32) + if cutlass.const_expr(din): + # bf16(x * rs * w), the draft-input kernel's rounding. + qz_wword = cutlass.Int32(qz_wvecs[qz_v][qz_e4]) + qz_lo = _bf16_round( + qz_lo + * qz_rs + * (qz_wword << cutlass.Int32(16)).bitcast(cutlass.Float32) + ) + qz_hi = _bf16_round( + qz_hi + * qz_rs + * (qz_wword & cutlass.Int32(-65536)).bitcast( + cutlass.Float32 + ) + ) + qz_vals.append(qz_lo) + qz_vals.append(qz_hi) + qz_amax = _absmax(_absmax(qz_amax, qz_lo), qz_hi) + qz_bits = (qz_amax * _rcp_approx_ftz(cutlass.Float32(448.0))).bitcast( + cutlass.Int32 + ) + qz_e = ( + (qz_bits >> cutlass.Int32(23)) & cutlass.Int32(0xFF) + ) + cutlass.select_( + (qz_bits & cutlass.Int32(0x7FFFFF)) != cutlass.Int32(0), + cutlass.Int32(1), + cutlass.Int32(0), + ) + qz_e = cutlass.select_(qz_e > cutlass.Int32(254), cutlass.Int32(254), qz_e) + qz_scale = ((cutlass.Int32(254) - qz_e) << cutlass.Int32(23)).bitcast( + cutlass.Float32 + ) + qz_scale = cutlass.select_( + qz_amax != cutlass.Float32(0.0), qz_scale, cutlass.Float32(0.0) + ) + qz_q = [_e4m3x4(qz_vals[4 * qz_j] * qz_scale, qz_vals[4 * qz_j + 1] * qz_scale, + qz_vals[4 * qz_j + 2] * qz_scale, qz_vals[4 * qz_j + 3] * qz_scale) + for qz_j in range(8)] # fmt: skip + if qz_item < cutlass.Int32(qz_items): + if qz_t < num_tokens: + qz_row_w = qz_i * cutlass.Int32( + x_tile_bytes // 4 + ) + qz_t * cutlass.Int32(CTA_K // 4) + qz_c0 = qz_kb * cutlass.Int32(2) + qz_b_words.store( + (qz_q[0], qz_q[1], qz_q[2], qz_q[3]), + idx=qz_row_w + + (qz_c0 ^ (qz_t & cutlass.Int32(7))) * cutlass.Int32(4), + alignment=16, + ) + qz_b_words.store( + (qz_q[4], qz_q[5], qz_q[6], qz_q[7]), + idx=qz_row_w + + ((qz_c0 + cutlass.Int32(1)) ^ (qz_t & cutlass.Int32(7))) + * cutlass.Int32(4), + alignment=16, + ) + smem_sfb.store( + cutlass.Uint8(qz_e), + idx=qz_i * cutlass.Int32(SF_ATOM_BYTES) + + (qz_t % cutlass.Int32(32)) * cutlass.Int32(16) + + (qz_t // cutlass.Int32(32)) * cutlass.Int32(4) + + qz_kb, + ) + qz_loaded = [] + # Generic-proxy stores made visible to the MMA's async-proxy reads, then one arrival for all. + prims.fence_proxy("async_shared", space=prims.SharedSpace.shared_cta) + prims.barrier_cta_sync(2, thread_count=128) + if qz_tid == cutlass.Int32(0): + prims.mbarrier_arrive(act_full) + if cutlass.const_expr(st_async): + while not cute.arch.mbarrier_test_wait(acc_done.data_ptr(), 0): + pass + else: + while not cute.arch.mbarrier_try_wait(acc_done.data_ptr(), 0): + pass + prims.tcgen05_fence(prims.Tcgen05Fence.AFTER_THREAD_SYNC) + # tcgen05.ld widths are powers of two: the accumulator columns in descending power-of-two pieces. + acc_all = [] + ld_col = 0 + for ld_w in (128, 64, 32, 16, 8): + if cutlass.const_expr((mma_n * nacc - ld_col) >= ld_w): + piece = prims.tcgen05_ld( + "32x32b", + cutlass.inttoptr( + tmem_ptr_i32.load() + cutlass.Int32(ld_col), 6, cutlass.Float32 + ), + num=ld_w, + ) + acc_all = acc_all + [piece[c] for c in range(ld_w)] + ld_col += ld_w + prims.tcgen05_wait(prims.Tcgen05Wait.LOAD) + acc = [] + for t in cutlass.range_constexpr(mma_n): + v = cutlass.Float32(acc_all[t]) + for a in cutlass.range_constexpr(1, nacc): + v = v + cutlass.Float32(acc_all[a * mma_n + t]) + acc.append(v) + prims.tcgen05_fence(prims.Tcgen05Fence.BEFORE_THREAD_SYNC) + if tx == 128: + _stamp(timers, timed, bx, 4) + n = m_offset + row + if cutlass.const_expr(split == 1): + for t in cutlass.range_constexpr(mma_n): + if cutlass.Int32(t) < num_tokens: + y.store( + cutlass.BFloat16(cutlass.Float32(acc[t])), idx=cutlass.Int32(t * n_out) + n + ) + if cutlass.const_expr(ar_push > 0): + prims.griddepcontrol(prims.GridDepAction.WAIT) + # The flags are rotated by the previous all-reduce after its dependent release; this load is ordered by + # the wait above, which waits for ALL in-flight upstream grids, not only the immediate predecessor (PTX + # griddepcontrol; MoE test_pdl_transitive_wait.py). Keep the wait before this load. + # The parity of each (token, column slice) slot: the all-reduce's CTA t * 8 + slice flag. All loads are + # issued before the first store so they overlap (one L2 round trip, not one per token). + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_v = cutlass.Float32(acc[t]) + ar_hi = cute.arch.shuffle_sync_bfly(ar_v, offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), _bf16x2_word(ar_v, ar_hi) + ) + else: + owner = row // cutlass.Int32(rows_per_rank) + local = row % cutlass.Int32(rows_per_rank) + if cutlass.const_expr(rope or gu_swiglu): + # Rank r owns head dims [r * P, (r + 1) * P) and their rotation partners 64 + ..., P = rows / 2 + # (gu_swiglu: gate rows and their up rows 64 + ...). + pairs = rows_per_rank // 2 + owner = (row % cutlass.Int32(64)) // cutlass.Int32(pairs) + local = (row // cutlass.Int32(64)) * cutlass.Int32(pairs) + ( + row % cutlass.Int32(64) + ) % cutlass.Int32(pairs) + if owner != rank: + base = (rank * cutlass.Int32(rows_per_rank) + local) * cutlass.Int32(mb_stride) + if cutlass.const_expr(st_async): + dst = _mapa_u32(mailbox.subview(base).data_ptr(), owner) + mb = _mapa_u32(reduce_ready.data_ptr(), owner) + words = [cutlass.Float32(acc[t]).bitcast(cutlass.Int32) for t in range(mma_n)] + for g in cutlass.range_constexpr(mma_n // 4): + _st_async_v4( + dst + cutlass.Int32(16 * g), + words[4 * g], + words[4 * g + 1], + words[4 * g + 2], + words[4 * g + 3], + mb, + ) + else: + for t in cutlass.range_constexpr(mma_n): + prims.mapa(mailbox.subview(base + cutlass.Int32(t)), owner).store( + cutlass.Float32(acc[t]) + ) + prims.mbarrier_arrive( + prims.mapa(reduce_ready, owner), scope=prims.MemScope.CLUSTER + ) + else: + if cutlass.const_expr(st_async): + while not _test_wait_cluster(reduce_ready.data_ptr(), 0): + pass + else: + while not prims.mbarrier_try_wait_parity( + reduce_ready, 0, scope=prims.MBarrierScope.CLUSTER + ): + pass + if lane == 0: + if row // cutlass.Int32(rows_per_rank) == rank: + if (row % cutlass.Int32(rows_per_rank)) == 0: + _stamp(timers, timed, bx, 5) + totals = [] + for g in cutlass.range_constexpr(mma_n // 4): + sums = [cutlass.Float32(0.0)] * 4 + for q in cutlass.range_constexpr(split): + parts = _ld_shared_v4_f32( + mailbox.subview( + (cutlass.Int32(q * rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g) + ).data_ptr() + ) + sums = [sums[j] + cutlass.select_(rank == cutlass.Int32(q), cutlass.Float32(acc[4 * g + j]), + parts[j]) for j in range(4)] # fmt: skip + for j in cutlass.range_constexpr(4): + totals.append(sums[j]) + if cutlass.const_expr(rope or gu_swiglu): + # The own-rank mailbox slot is unused: it carries the totals to the RoPE / SwiGLU stage. + mailbox.store( + sums[j], + idx=(rank * cutlass.Int32(rows_per_rank) + local) + * cutlass.Int32(mb_stride) + + cutlass.Int32(4 * g + j), + ) + elif cutlass.const_expr(prod): + # The bf16 the packed row would hold (RN, as the y store below), staged by token. + if cutlass.Int32(4 * g + j) < num_tokens: + pr_stage.store( + cutlass.Uint16( + (_bf16_bits(sums[j]) >> cutlass.Int32(16)) + & cutlass.Int32(0xFFFF) + ), + idx=cutlass.Int32((4 * g + j) * rows_per_rank) + local, + ) + else: + if cutlass.Int32(4 * g + j) < num_tokens: + y.store( + cutlass.BFloat16(sums[j]), + idx=cutlass.Int32((4 * g + j) * n_out) + n, + ) + if cutlass.const_expr(ar_push > 0): + # Every lane of an owner warp is an owner (rows_per_rank >= 32), so lane pairs hold row pairs. + prims.griddepcontrol(prims.GridDepAction.WAIT) + # The flags are rotated by the previous all-reduce after its dependent release; this load is ordered + # by the wait above, which waits for ALL in-flight upstream grids, not only the immediate + # predecessor (PTX griddepcontrol; MoE test_pdl_transitive_wait.py). Keep the wait before this load. + ar_par = [ + _ld_volatile_s32( + ar_flags + + cutlass.Int64(4) + * (cutlass.Int64(t * 8) + cutlass.Int64(n // cutlass.Int32(n_out // 8))) + ) + & cutlass.Int32(1) + for t in range(mma_n) + ] + for t in cutlass.range_constexpr(mma_n): + ar_hi = cute.arch.shuffle_sync_bfly(totals[t], offset=1) + if (lane % cutlass.Int32(2)) == cutlass.Int32(0): + if cutlass.Int32(t) < num_tokens: + ar_off = ( + cutlass.Int64(ar_par[t]) + * cutlass.Int64(ar_maxt * ar_world * (n_out // 2)) + + (cutlass.Int64(t * ar_world) + cutlass.Int64(ar_rank)) + * cutlass.Int64(n_out // 2) + + (cutlass.Int64(n) // cutlass.Int64(2)) + ) + _st_global_b32( + ar_ws + ar_off * cutlass.Int64(4), + _bf16x2_word(totals[t], ar_hi), + ) + prims.barrier_cta_sync(1, thread_count=128) + if cutlass.const_expr(prod): + # Transpose: thread e < 2 T sends half e % 2 (16 bytes = 8 dims) of token e / 2's staged rows to cluster + # rank e / 2, at dims [rank * rows_per_rank + 8 (e % 2), ...) of that rank's packed-head row. + pr_e = tx - cutlass.Int32(128) + if pr_e < num_tokens * cutlass.Int32(2): + pr_t = pr_e // cutlass.Int32(2) + pr_h = pr_e % cutlass.Int32(2) + pr_w = _ld_shared_v4_b32( + pr_stage.subview( + pr_t * cutlass.Int32(rows_per_rank) + pr_h * cutlass.Int32(8) + ).data_ptr() + ) + _st_async_v4( + _mapa_u32( + pr_row.subview( + rank * cutlass.Int32(rows_per_rank // 2) + pr_h * cutlass.Int32(4) + ).data_ptr(), + pr_t, + ), + pr_w[0], + pr_w[1], + pr_w[2], + pr_w[3], + _mapa_u32(pr_full.data_ptr(), pr_t), + ) + if warp_id == 4: + if rank < num_tokens: + # The producer reads step inputs (positions, slots) and writes the caches: ordered after the + # upstream grids by this thread's own grid wait (already satisfied: the activation landed). + prims.griddepcontrol(prims.GridDepAction.WAIT) + while not _test_wait_cluster(pr_full.data_ptr(), 0): + pass + pr_lo = pr_row.load(idx=lane * cutlass.Int32(2)) + pr_hi = pr_row.load(idx=lane * cutlass.Int32(2) + cutlass.Int32(1)) + mask = cutlass.Int32(0xFFFF) + producer_head( + rank, + n_tile, + lane, + bf16_bits_to_f32(pr_lo & mask), + bf16_bits_to_f32((pr_lo >> cutlass.Int32(16)) & mask), + bf16_bits_to_f32(pr_hi & mask), + bf16_bits_to_f32((pr_hi >> cutlass.Int32(16)) & mask), + ProducerArgs( + packed=cutlass.Int64(0), + q_out=pr_q, + index_q_out=pr_iq, + index_k=pr_ik, + kv_data=pr_kvd, + kv_scale=pr_kvs, + out_cache_loc=pr_loc, + kv_quant_scale=pr_quant, + position_ids=pr_pos, + rotary_cos_sin=pr_rope, + q_weight=pr_qw, + k_weight=pr_kw, + index_q_weight=pr_iqw, + index_k_weight=pr_ikw, + eps=pr_eps, + num_pages=pr_pages, + index_page_stride=pr_ips, + index_token_stride=pr_its, + data_page_stride=pr_dps, + data_plane_stride=pr_dls, + data_token_stride=pr_dts, + scale_page_stride=pr_sps, + scale_plane_stride=pr_sls, + ), + ) + if cutlass.const_expr(rope): + # Token t of request t / T sits at cache position kv_len - T + t % T. Q / K heads: GPT-NeoX RoPE over + # the 128-wide head, bf16(cos * x -/+ sin * pair) on the bf16 projection; then E4M3 of x * scale, Q to + # rope_q, K / V into the page of that position. + pairs = rows_per_rank // 2 + rp_e = tx - cutlass.Int32(128) + rp_p = rp_e % cutlass.Int32(pairs) + rp_t = rp_e // cutlass.Int32(pairs) + if rp_t < num_tokens: + rp_max_blocks = rope_dims.load(idx=0) + rp_tpb_log2 = rope_dims.load(idx=1) + rp_hq = rope_dims.load(idx=2) + rp_hkv = rope_dims.load(idx=3) + rp_T = rope_dims.load(idx=4) + rp_d = rank * cutlass.Int32(pairs) + rp_p + rp_a = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + rp_p) * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_b = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + cutlass.Int32(pairs) + rp_p) + * cutlass.Int32(mb_stride) + + rp_t + ) + ) + rp_req = rp_t // rp_T + rp_tok = rope_i.load(idx=rp_req) - rp_T + (rp_t - rp_req * rp_T) + rp_head = n_tile + if rp_head < rp_hq + rp_hkv: + rp_c = rope_cs.load(idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2)) + rp_s = rope_cs.load( + idx=(rp_tok * cutlass.Int32(64) + rp_d) * cutlass.Int32(2) + + cutlass.Int32(1) + ) + rp_ra = _rope_bits(rp_c, rp_a, -rp_s, rp_b).bitcast(cutlass.Float32) + rp_rb = _rope_bits(rp_c, rp_b, rp_s, rp_a).bitcast(cutlass.Float32) + rp_a = rp_ra + rp_b = rp_rb + rp_qs = rope_scale.load(idx=0) + rp_w = _e4m3x4( + rp_a * rp_qs, rp_b * rp_qs, cutlass.Float32(0.0), cutlass.Float32(0.0) + ) + rp_lo = cutlass.Uint8(rp_w & cutlass.Int32(0xFF)) + rp_hi = cutlass.Uint8((rp_w >> cutlass.Int32(8)) & cutlass.Int32(0xFF)) + if rp_head < rp_hq: + rp_q = (rp_t * rp_hq + rp_head) * cutlass.Int32(128) + rp_d + rope_q.store(rp_lo, idx=rp_q) + rope_q.store(rp_hi, idx=rp_q + cutlass.Int32(64)) + else: + rp_kv = cutlass.select_( + rp_head < rp_hq + rp_hkv, cutlass.Int32(0), cutlass.Int32(1) + ) + rp_kh = rp_head - rp_hq - rp_kv * rp_hkv + rp_page = rope_bt.load( + idx=(rp_req * cutlass.Int32(2) + rp_kv) * rp_max_blocks + + (rp_tok >> rp_tpb_log2) + ) + # 64-bit byte offset: the pool view can exceed 2^31 bytes. + rp_dst = ( + (cutlass.Int64(rp_page) * cutlass.Int64(rp_hkv) + cutlass.Int64(rp_kh)) + << cutlass.Int64(rp_tpb_log2 + cutlass.Int32(7)) + ) + cutlass.Int64( + (rp_tok & ((cutlass.Int32(1) << rp_tpb_log2) - cutlass.Int32(1))) + * cutlass.Int32(128) + + rp_d + ) + rp_addr = rope_pool.data_ptr(0).toint() + rp_dst + cutlass.inttoptr(rp_addr, mem_space=1, dtype=cutlass.Uint8).store(rp_lo) + cutlass.inttoptr( + rp_addr + cutlass.Int64(64), mem_space=1, dtype=cutlass.Uint8 + ).store(rp_hi) + if cutlass.const_expr(gu_swiglu): + # Thread 128 + e: pair p = e % P of token e / P. The gate and up totals round to bf16 as the y store does + # (RN), then the swigluoai of the Triton silu_and_mul kernel (_swiglu_oai_bf16x2, bit for bit). + gs_pairs = rows_per_rank // 2 + gs_e = tx - cutlass.Int32(128) + gs_p = gs_e % cutlass.Int32(gs_pairs) + gs_t = gs_e // cutlass.Int32(gs_pairs) + if gs_t < num_tokens: + gs_g = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + gs_p) * cutlass.Int32(mb_stride) + + gs_t + ) + ) + gs_u = _bf16_round( + mailbox.load( + idx=(rank * cutlass.Int32(rows_per_rank) + cutlass.Int32(gs_pairs) + gs_p) + * cutlass.Int32(mb_stride) + + gs_t + ) + ) + gs_w = _swiglu_oai_bf16x2( + (gs_g.bitcast(cutlass.Int32) >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF), + (gs_u.bitcast(cutlass.Int32) >> cutlass.Int32(16)) & cutlass.Int32(0xFFFF), + ) + y.store( + cutlass.BFloat16((gs_w << cutlass.Int32(16)).bitcast(cutlass.Float32)), + idx=gs_t * cutlass.Int32(n_out // 2) + + n_tile * cutlass.Int32(CTA_M // 2) + + rank * cutlass.Int32(gs_pairs) + + gs_p, + ) + if tx == 128: + _stamp(timers, timed, bx, 6) + if cutlass.const_expr(ar_push == 2): + # The all-reduce polls the pushed words; it launches after this CTA's rows are out, not at grid end. + prims.griddepcontrol(prims.GridDepAction.LAUNCH_DEPENDENTS) + if warp_id == 4: + prims.tcgen05_dealloc( + cutlass.inttoptr(tmem_ptr_i32.load(), 6, cutlass.Int32), + tmem_cols(ring, nacc, a_tmem, my_tiles, mma_n), + ) + + +def _bytes_map(t, cols, rows, box_rows, swizzle): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint8, + global_dims=[cols, rows], + global_strides=[cols // 16], + box_dims=[CTA_K, box_rows], + swizzle=swizzle, + ) + + +def _atom_map(t, atoms): + return cuda.create_tensor_map_tiled( + global_address=t.iterator.toint(), + dtype=cutlass.Uint16, + global_dims=[SF_ATOM_U16, atoms], + global_strides=[SF_ATOM_BYTES // 16], + box_dims=[SF_ATOM_U16, 1], + swizzle=cuda.TensorMapSwizzle.none, + ) + + +@cute.jit +def m3_mx_gemv( + w: cute.Tensor, # [N, K] E4M3 as uint8, K contiguous + wsf: cute.Tensor, # weight scales, 128x4-swizzled, as uint8 + x: cute.Tensor, # [M, K] E4M3 as uint8, M <= 8 + xsf: cute.Tensor, # activation scales, 128x4-swizzled (atom k at byte 512 k), as uint8 + y: cute.Tensor, # [M * N] bf16 + x_words: cute.Tensor, # x_bf16: bf16 x [M, K] as int32 words (x / xsf are then unused placeholders) + timers: cute.Tensor, # [grid * 8] int64, written when `timed` + num_tokens: cutlass.Int32, + n_out: cutlass.Constexpr[int], + k_in: cutlass.Constexpr[int], + split: cutlass.Constexpr[int], + ring: cutlass.Constexpr[int], + trigger_early: cutlass.Constexpr[bool], + contig: cutlass.Constexpr[bool], + timed: cutlass.Constexpr[bool], + nacc: cutlass.Constexpr[int], + a_tmem: cutlass.Constexpr[bool], + st_async: cutlass.Constexpr[bool], + x_bf16: cutlass.Constexpr[bool], + ar_ws: cutlass.Int64, + ar_flags: cutlass.Int64, + ar_rank: cutlass.Int32, + ar_push: cutlass.Constexpr[int], + ar_world: cutlass.Constexpr[int], + ar_maxt: cutlass.Constexpr[int], + mma_n: cutlass.Constexpr[int], + use_pdl: cutlass.Constexpr[bool], + stream: cuda_driver.CUstream, + din_i: cute.Tensor, # din / rope inputs: see m3_mx_gemv_kernel (placeholders when off) + din_table: cute.Tensor, + din_hidden: cute.Tensor, + din_w: cute.Tensor, + din_f: cute.Tensor, + din_vocab: cutlass.Int32, + din_g: cute.Tensor, + din_res: cute.Tensor, + din_flags: cutlass.Int32, + rope_cs: cute.Tensor, + rope_i: cute.Tensor, + rope_bt: cute.Tensor, + rope_q: cute.Tensor, + rope_pool: cute.Tensor, + rope_scale: cute.Tensor, + rope_dims: cute.Tensor, + din: cutlass.Constexpr[bool], + rope: cutlass.Constexpr[bool], + x_swiglu: cutlass.Constexpr[bool], + gu_swiglu: cutlass.Constexpr[bool], + # prod: see m3_mx_gemv_kernel (zeros when off). + pr_q: cutlass.Int64, + pr_iq: cutlass.Int64, + pr_ik: cutlass.Int64, + pr_kvd: cutlass.Int64, + pr_kvs: cutlass.Int64, + pr_loc: cutlass.Int64, + pr_quant: cutlass.Int64, + pr_pos: cutlass.Int64, + pr_rope: cutlass.Int64, + pr_qw: cutlass.Int64, + pr_kw: cutlass.Int64, + pr_iqw: cutlass.Int64, + pr_ikw: cutlass.Int64, + pr_ips: cutlass.Int64, + pr_its: cutlass.Int64, + pr_dps: cutlass.Int64, + pr_dls: cutlass.Int64, + pr_dts: cutlass.Int64, + pr_sps: cutlass.Int64, + pr_sls: cutlass.Int64, + pr_pages: cutlass.Int32, + pr_eps: cutlass.Float32, + prod: cutlass.Constexpr[bool], +) -> None: + # x_swiglu's up half rides the slot din uses for its norm weights, and it activates the bf16 x. + assert not (din and x_swiglu) and (x_bf16 or not x_swiglu), ( + "x_swiglu needs x_bf16 and excludes din" + ) + k_tiles = num_k_tiles(k_in) + tma_desc_w = _bytes_map(w, k_in, n_out, CTA_M, cuda.TensorMapSwizzle.s128b) + tma_desc_wsf = _atom_map(wsf, (n_out // CTA_M) * k_tiles) + tma_desc_x = _bytes_map(x, k_in, num_tokens, mma_n, cuda.TensorMapSwizzle.s128b) + tma_desc_xsf = _atom_map(xsf, k_tiles) + m3_mx_gemv_kernel( + tma_desc_w, tma_desc_wsf, tma_desc_x, tma_desc_xsf, y, x_words, timers, num_tokens, n_out, k_in, split, ring, + trigger_early, contig, timed, nacc, a_tmem, st_async, x_bf16, ar_ws, ar_flags, ar_rank, ar_push, ar_world, + ar_maxt, mma_n, + din_i, din_table, din_hidden, din_w, din_f, din_vocab, din_g, din_res, din_flags, rope_cs, + rope_i, rope_bt, rope_q, rope_pool, rope_scale, rope_dims, din, rope, x_swiglu, gu_swiglu, + pr_q, pr_iq, pr_ik, pr_kvd, pr_kvs, pr_loc, pr_quant, pr_pos, pr_rope, pr_qw, pr_kw, pr_iqw, pr_ikw, pr_ips, + pr_its, pr_dps, pr_dls, pr_dts, pr_sps, pr_sls, pr_pages, pr_eps, prod, + ).launch( + grid=((n_out // CTA_M) * split, 1, 1), + block=(THREADS, 1, 1), + cluster=(split, 1, 1), + stream=stream, + use_pdl=use_pdl, + ) # fmt: skip diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/op.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/op.py new file mode 100644 index 000000000000..8a89dd14b0b6 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/mx_gemv/op.py @@ -0,0 +1,713 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""MXFP8 x MXFP8 decode GEMV (tcgen05, block-scaled) at the TP2 per-rank projection shapes (SM107). + +Compiled on the first call per (N, K, split, ring, early trigger, PDL, push, MMA width), which must happen outside +CUDA-graph capture, or ahead of any call by ``precompile``; once ``freeze`` closes compiles, a call whose kernel is not +compiled declines instead. +""" + +from __future__ import annotations + +import threading +from pathlib import Path +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + +from ...m31_switches import m31_disabled + +MAX_TOKENS = 48 # capacity: fixed: the decode kernels' row bound (decode steps) +MMA_WIDTHS = ( + 8, + 16, + 32, + 48, +) # token columns of the compiled MMA; a call uses the smallest that holds M +_SMEM_LIMIT = 220 * 1024 # capacity: fixed: the device's shared-memory limit + + +def _mma_n(m: int) -> int: + return next(w for w in MMA_WIDTHS if m <= w) + + +def _smem_bytes(ring: int, tiles: int, split: int, mma_n: int) -> int: + return ( + ring * (128 * 128 + 512) + + tiles * (mma_n * 128 + 512) + + split * (128 // split) * (mma_n + 4) * 4 + + 1024 + ) + + +_lock = threading.Lock() +_compiled: Dict[tuple, object] = {} + +# (N, K) -> (split, ring[, contig, nacc, a_tmem, st_async]) of the TP2 per-rank projections. The optional fields default +# to _CONTIG, _NACC, _A_TMEM, _ST_ASYNC. SM107 graph us at M 1 / 4 / 8 over weights rotated beyond L2, against the +# stock MXFP8 quantize + FlashInfer CuTe DSL GEMM: QKV 4.11 / 4.12 / 4.10 vs 8.6-8.8, o_proj 3.90 / 3.96 / 3.70 vs +# 6.9-7.2, shared gate_up 3.11 / 3.09 / 3.10 vs 8.1-8.4, shared down 2.22 vs 4.7-4.9, dense gate_up 8.15 / 8.56 / 8.55 +# vs 10.2-10.5, dense down 5.89 / 5.72 / 5.46 vs 8.7-9.0. +_CONFIGS: Dict[Tuple[int, int], tuple] = { + # The sm_107 tp8 target's per-rank projections (8 ranks over two trays), (split, ring) from an SM107 sweep + # (locked, MXFP8 in, back-to-back PDL, rows 1 / 4 / 8; the GB300 tp8 table has none of these shapes on SM107). + # sparse-layer QKV + index projection: 2.37 / 2.43 / 2.42 us + (1536, 6144): (8, 8), + # dense-layer QKV: 2.34 / 2.35 / 2.36 us + (1280, 6144): (8, 8), + # o_proj: 2.05 / 2.07 / 2.07 us + (6144, 1024): (2, 12), + # shared-expert gate_up: 2.16 / 2.17 / 2.17 us + (768, 6144): (8, 8), + # shared-expert down: 1.49 / 1.50 / 1.52 us + (6144, 384): (1, 8), + # dense-layer gate_up: 3.09 / 3.12 / 3.08 us + (3072, 6144): (4, 12), + # dense-layer down: 2.29 / 2.30 / 2.29 us + (6144, 1536): (2, 8), +} +# Per-token-count configs (N, K, M); none at TP2. +_CONFIGS_BY_M: Dict[Tuple[int, int, int], tuple] = {} +# Above 8 tokens (wider than the TP2 decode step); none at TP2. +_CONFIGS_WIDE: Dict[Tuple[int, int], tuple] = {} +# VRCFG-SWI: the swiglu-prologue calls (bf16 gate | up in) at up to SWIGLU_CFG_MAX_ROWS rows take their own ring: the +# prologue's per-k-tile quantize shifts the best ring depth. Same split and k-tile order as the table, so bit for bit +# its output. SM107 sweep (8 rows, PDL back-to-back, weights rotated past L2): dense down 4.66 vs 5.86 us. Kill +# switch TRTLLM_M31_DISABLE_MXGEMV_VRCFG, read at import. +MXGEMV_VRCFG = not m31_disabled("MXGEMV_VRCFG") +SWIGLU_CFG_MAX_ROWS = 8 # capacity: fixed: the swept rows +_CONFIGS_SWIGLU: Dict[Tuple[int, int], tuple] = {(6144, 1536): (2, 3, 0)} if MXGEMV_VRCFG else {} + +# Defaults of the optional config fields: contiguous k-tile ranges per split rank, one TMEM accumulator, the A +# operand staged in TMEM (when its columns fit), st.async split-K partial exchange. +_CONTIG = True +_NACC = 1 +_A_TMEM = True +_ST_ASYNC = True +_ensured = set() + + +_DUMMY: Dict[torch.device, torch.Tensor] = {} + + +def _dummy_timers(device) -> torch.Tensor: + """Placeholder for the kernel's timer buffer, which it only writes when compiled with timers on.""" + t = _DUMMY.get(device) + if t is None: + t = _DUMMY[device] = torch.zeros(8, dtype=torch.int64, device=device) + return t + + +def _arg(t: torch.Tensor, align: int = 16): + from cutlass.cute.runtime import from_dlpack + + return from_dlpack(t.detach(), assumed_align=align).mark_layout_dynamic(leading_dim=t.dim() - 1) + + +# (rule 1b, deterministic) The decode projections at up to STRIDED_MAX_ROWS rows split K over strided k-tiles per +# split rank (contig 0: rank r owns k-tiles r, r + split, ...) instead of contiguous ranges: another fixed fp32 order. +# SM107 TP8 4 rows, us/call contiguous -> strided: sparse QKV 2.405 -> 2.371, dense QKV 2.323 -> 2.291, +# o_proj 2.061 -> 2.019, dense gate_up 3.047 -> 3.006, dense down 2.313 -> 2.287; deterministic (3 runs bitwise) and +# max error vs the fp64 product equal to the contiguous order's. Kill: False. +STRIDED_K_TILES = True +STRIDED_MAX_ROWS = 8 # capacity: fixed: the strided-input kernels' built rows +_STRIDED = frozenset({(1536, 6144), (1280, 6144), (6144, 1024), (3072, 6144), (6144, 1536)}) + +# #117: the producer epilogue serves the sparse-layer QKV + index projection, 12 packed heads of 128 (8 q, k, v, +# index-q, index-k), one head per split-8 cluster. +PROD_N = 12 * 128 # capacity: fixed: the producer's packed heads +# The kernel's producer arguments on a call without the producer epilogue (20 addresses / strides, pages, eps). +PROD_OFF = (0,) * 21 + (0.0,) + + +def get_config(n: int, k: int, m: Optional[int] = None) -> Optional[tuple]: + cfg = _CONFIGS_BY_M.get((n, k, m)) if m is not None else None + if cfg is None and m is not None and m > 8: + cfg = _CONFIGS_WIDE.get((n, k)) + cfg = cfg or _CONFIGS.get((n, k)) + # The split-K epilogue gives each of the split CTAs 128 // split rows of a 128-row output tile. + if cfg is None or n % 128 != 0 or k % 128 != 0 or (k // 128) % cfg[0] != 0 or 128 % cfg[0] != 0: + return None + if ( + STRIDED_K_TILES + and m is not None + and m <= STRIDED_MAX_ROWS + and (n, k) in _STRIDED + and len(cfg) == 2 + ): + cfg = cfg + (0,) + return cfg + + +def _declines(n: int, k: int, m: int, x_bf16: bool, cfg: tuple) -> bool: + """Whether a (split, ring, ...) config declines an [M, K] x [N, K]^T call on shape grounds.""" + split = cfg[0] + if not 0 < m <= MAX_TOKENS or 128 % split != 0 or (k // 128) % split != 0: + return True + tiles = k // 128 // split + mma_n = _mma_n(m) + # 17..32 rows at K < 1024 decline (an unsplit K below 1024 is slower than the FlashInfer GEMM there); no TP2 + # projection has K < 1024. + # The in-kernel bf16 -> MXFP8 quantization (x_bf16) costs one 32-element block per thread and round on 128 threads; + # beyond 8 rounds per CTA, mxfp8_quantize + this GEMV (or the fallback GEMM) is faster. + if (16 < m <= 32 and k < 1024) or (x_bf16 and tiles * 4 * mma_n > 8 * 128): + return True + return _smem_bytes(min(cfg[1], tiles, 2), tiles, split, mma_n) > _SMEM_LIMIT + + +def serves(n: int, k: int, m: int, x_bf16: bool) -> bool: + """Whether the GEMV with its default config takes an [M, K] activation (bf16 or MXFP8) against an [N, K] weight.""" + cfg = get_config(n, k, m) + return cfg is not None and not _declines(n, k, m, x_bf16, cfg) + + +def _resolve( + m: int, + n: int, + k: int, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple], + x_swiglu: bool = False, + prod: bool = False, + gu_swiglu: bool = False, +) -> Optional[tuple]: + """The compile key of an [M, K] x [N, K]^T call and its runtime push words, or None when the call is declined. + + ``ar`` is the push tuple of ``_run``; only its (world, bucket tokens, mode) enter the key, the addresses and the + rank are runtime arguments.""" + if cfg is None and x_swiglu and m <= SWIGLU_CFG_MAX_ROWS: + cfg = _CONFIGS_SWIGLU.get((n, k)) + cfg = cfg or get_config(n, k, m) + if cfg is None or _declines(n, k, m, x_bf16, cfg): + return None + split, ring = cfg[0], cfg[1] + contig = bool(cfg[2]) if len(cfg) > 2 else _CONTIG + nacc = int(cfg[3]) if len(cfg) > 3 else _NACC + a_tmem = bool(cfg[4]) if len(cfg) > 4 else _A_TMEM + st_async = bool(cfg[5]) if len(cfg) > 5 else _ST_ASYNC + if prod and not ( + n == PROD_N + and split == 8 + and m <= split + and st_async + and not x_bf16 + and not x_swiglu + and ar is None + ): + # The producer epilogue: the sparse QKV + index projection (one 128-row packed head per cluster of 8), at most + # one token per cluster rank, the st.async exchange, MXFP8 input, no all-reduce push. + return None + if gu_swiglu and not ( + (128 // split // 2) * _mma_n(m) <= 128 + and n % 128 == 0 + and not x_swiglu + and not prod + and ar is None + ): + # The SwiGLU epilogue: one owner thread per (gate / up pair, token) among the 128 epilogue threads, bf16 out. + return None + if nacc != 1: + # More than one TMEM accumulator is broken on SM107 (nondeterministic or garbage + # outputs). No shipped table uses it; decline (the caller's fallback runs) until the kernel path is fixed. + return None + ring = min(ring, k // 128 // split) + tiles = k // 128 // split + mma_n = _mma_n(m) + while ring > 2 and _smem_bytes(ring, tiles, split, mma_n) > _SMEM_LIMIT: + ring -= 1 + if a_tmem: + from .m3_mx_gemv_kernel import tmem_cols + + if tmem_cols(ring, nacc, True, tiles, mma_n) > 512: + a_tmem = False + # ar = (multicast workspace address, flags address, rank, world, bucket tokens, mode): push the output rows into + # the next m3_mc_allreduce call's exchange slots (mode 2 also defers the dependents' launch until after the push). + ar_ws, ar_flags, ar_rank, ar_mode, ar_world, ar_maxt = 0, 0, 0, 0, 1, 8 + if ar is not None and split <= 4 and n % 2 == 0: + ar_ws, ar_flags, ar_rank, ar_world, ar_maxt, ar_mode = ar + if ar_mode == 2: + trigger_early = False + if use_pdl is None: + use_pdl = get_env_enable_pdl() + # The kernel's ``timed`` flag (per-CTA milestone timers) is always off. + key = ( + n, + k, + split, + ring, + trigger_early, + contig, + False, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + x_swiglu, + prod, + gu_swiglu, + ) + return key, (ar_ws, ar_flags, ar_rank) + + +def _compile( + key: tuple, args: tuple, m: int, runtime_ar: tuple, stream, extra: tuple, prod_args: tuple +): + """cute.compile the kernel of ``key`` (see ``_resolve``) on these argument views; cached.""" + import cutlass.cute as cute + + from . import m3_mx_gemv_kernel as kernel + + ( + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + x_swiglu, + prod, + gu_swiglu, + ) = key + ar_ws, ar_flags, ar_rank = runtime_ar + with _lock: + fn = _compiled.get(key) + if fn is None: + fn = _compiled[key] = cute_aot.compile_or_load( + f"m3_mx_gemv {key}", + lambda: cute.compile( + kernel.m3_mx_gemv, + *args, + m, + n, + k, + split, + ring, + trigger_early, + contig, + timed, + nacc, + a_tmem, + st_async, + x_bf16, + ar_ws, + ar_flags, + ar_rank, + ar_mode, + ar_world, + ar_maxt, + mma_n, + use_pdl, + stream, + *extra, + False, + False, + x_swiglu, + gu_swiglu, + *prod_args, + prod, + ), + sources=_aot_sources(), + ) + return fn + + +def _views( + x: torch.Tensor, + xsf: Optional[torch.Tensor], + weight, + weight_scale, + y, + x_bf16: bool, + x_swiglu: bool = False, +) -> tuple: + # x_swiglu: x is [M, 2K] (gate | up); the GEMV's K is half its width. + k = x.shape[1] // 2 if x_swiglu else x.shape[1] + if x_bf16: + x_bytes = x.view(torch.uint8) + xsf = _placeholder_scales(k, x.device) + x_words = x.view(torch.int32) + else: + x_bytes = x.view(torch.uint8) + x_words = _placeholder_words(x.device) + return ( + _arg(weight.view(torch.uint8)), + _arg(weight_scale.view(torch.uint8).reshape(-1)), + _arg(x_bytes), + _arg(xsf.view(torch.uint8).reshape(-1)), + _arg(y.view(-1) if y.numel() else _placeholder_words(x.device).view(torch.bfloat16)), + _arg(x_words.reshape(-1)), + _arg(_dummy_timers(x.device)), + ) + + +def _run( + x: torch.Tensor, + xsf: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + x_bf16: bool, + trigger_early: bool, + cfg: Optional[tuple], + use_pdl: Optional[bool], + ar: Optional[tuple] = None, + x_swiglu: bool = False, + prod_args: Optional[tuple] = None, + gu_swiglu: bool = False, +) -> Optional[torch.Tensor]: + global last_pushed, last_uncompiled + last_pushed = False + last_uncompiled = None + import cuda.bindings.driver as cuda_driver + + m, k = x.shape + if x_swiglu: + k //= 2 + n = weight.shape[0] + if weight.shape[1] != k: + return None + prod = prod_args is not None + resolved = _resolve(m, n, k, x_bf16, trigger_early, cfg, use_pdl, ar, x_swiglu, prod, gu_swiglu) + if resolved is None: + return None + key, runtime_ar = resolved + if not prod: + prod_args = PROD_OFF + fn = _compiled.get(key) + if fn is None: + if _frozen[0]: + # Compiles are closed (the model built every kernel it dispatches to before warmup): decline, the caller + # reports the miss. + last_uncompiled = key + return None + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "m3_mx_gemv must run once per shape outside CUDA-graph capture first" + ) + # prod: no packed row is stored (the producer epilogue consumes it); y is the placeholder. + y = torch.empty( + 0 if prod else m, n // 2 if gu_swiglu else n, dtype=torch.bfloat16, device=x.device + ) + args = _views(x, xsf, weight, weight_scale, y, x_bf16, x_swiglu) + extra = _extra_args(x.device) + stream = cuda_driver.CUstream(torch.cuda.current_stream(x.device).cuda_stream) + if fn is None: + fn = _compile(key, args, m, runtime_ar, stream, extra, prod_args) + ar_ws, ar_flags, ar_rank = runtime_ar + fn(*args, m, ar_ws, ar_flags, ar_rank, stream, *extra, *prod_args) + last_pushed = key[11] > 0 + return y + + +def precompile( + weight: torch.Tensor, + weight_scale: torch.Tensor, + m: int, + x_bf16: bool, + use_pdl: Optional[bool] = None, + ar: Optional[Tuple[int, int, int]] = None, + x_swiglu: bool = False, + prod: bool = False, + gu_swiglu: bool = False, +) -> Optional[tuple]: + """Compile, without launching, the kernel an M-row call against ``weight`` takes; returns its key, or None when + such a call is declined. ``ar`` = (world, bucket tokens, mode) of a pushing call (its addresses and rank are + runtime arguments, so none is needed here). Runs outside capture, even while compiles are frozen.""" + import cuda.bindings.driver as cuda_driver + + n, k = weight.shape + push = None if ar is None else (0, 0, 0, *ar) + resolved = _resolve(m, n, k, x_bf16, True, None, use_pdl, push, x_swiglu, prod, gu_swiglu) + if resolved is None: + return None + key, runtime_ar = resolved + if key in _compiled: + return key + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError("m3_mx_gemv.precompile allocates and compiles: call it outside capture") + dev = weight.device + x = torch.zeros( + m, + 2 * k if x_swiglu else k, + dtype=torch.bfloat16 if x_bf16 else torch.float8_e4m3fn, + device=dev, + ) + xsf = None if x_bf16 else torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=dev) + y = torch.empty(m, n // 2 if gu_swiglu else n, dtype=torch.bfloat16, device=dev) + args = _views(x, xsf, weight, weight_scale, y, x_bf16, x_swiglu) + stream = cuda_driver.CUstream(torch.cuda.current_stream(dev).cuda_stream) + _compile(key, args, m, runtime_ar, stream, _extra_args(dev), PROD_OFF) + return key + + +def freeze(frozen: bool = True) -> None: + """Close (or reopen) compiles: while frozen, a call whose kernel is not compiled yet declines (returns None and + sets ``last_uncompiled``) instead of compiling. ``precompile`` and ``ensure_compiled`` still compile.""" + _frozen[0] = bool(frozen) + + +def frozen() -> bool: + return _frozen[0] + + +def is_compiled(key: tuple) -> bool: + return key in _compiled + + +def _extra_args(device) -> tuple: + """Placeholder arguments of the kernel's draft-input prologue and RoPE / KV epilogue (compiled off).""" + i32 = _placeholder_words(device) + f32 = i32.view(torch.float32) + u8 = i32.view(torch.uint8) + d = (_arg(i32), _arg(i32), _arg(i32), _arg(i32), _arg(f32), 0, _arg(i32), _arg(i32), 0) + r = (_arg(f32), _arg(i32), _arg(i32), _arg(u8), _arg(u8), _arg(f32), _arg(i32)) + return d + r + + +def m3_mx_gemv( + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """``xq`` E4M3 [M <= 8, K] with ``xsf`` its 128x4-swizzled UE8M0 scales; ``weight`` E4M3 [N, K] with + ``weight_scale`` swizzled. Returns bf16 [M, N], or None when the shape is not served.""" + return _run(xq, xsf, weight, weight_scale, False, trigger_early, cfg, use_pdl, ar) + + +def m3_mx_gemv_bf16( + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """The same GEMV for a bf16 ``x`` [M <= 8, K] (contiguous), quantized to MXFP8 inside the kernel exactly as + torch.ops.trtllm.mxfp8_quantize(x, True) does.""" + if x.dtype != torch.bfloat16 or x.dim() != 2 or not x.is_contiguous() or x.data_ptr() % 16 != 0: + return None + return _run(x, None, weight, weight_scale, True, trigger_early, cfg, use_pdl, ar) + + +def m3_mx_gemv_swiglu( + gate_up: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, +) -> Optional[torch.Tensor]: + """``m3_mx_gemv_bf16(swiglu(gate_up), ...)`` in one launch: ``gate_up`` bf16 [M <= 8, 2K] (contiguous, gate then + up) is activated in the kernel's quantize prologue with the swigluoai arithmetic of the Triton silu_and_mul kernel + (alpha 1.702, limit 7, the (up + 1) offset; bit for bit), then quantized as the bf16 path does.""" + if ( + gate_up.dtype != torch.bfloat16 + or gate_up.dim() != 2 + or not gate_up.is_contiguous() + or gate_up.data_ptr() % 16 != 0 + or gate_up.shape[1] % 2 != 0 + ): + return None + return _run(gate_up, None, weight, weight_scale, True, trigger_early, cfg, use_pdl, ar, True) + + +def gu_interleave(weight: torch.Tensor) -> torch.Tensor: + """A gate_up weight [2F, K] (gate rows then up rows, F a multiple of 64) reordered per 128-row tile for the SwiGLU + epilogue: tile i holds gate rows 64 i .. 64 i + 63 then up rows 64 i .. 64 i + 63. Rows only move, so each output + feature's dot product (bytes and K order) is unchanged.""" + f = weight.shape[0] // 2 + gate, up = weight[:f].reshape(f // 64, 64, -1), weight[f:].reshape(f // 64, 64, -1) + return torch.cat([gate, up], dim=1).reshape(2 * f, -1).contiguous() + + +def m3_mx_gemv_gu_swiglu( + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + use_pdl: Optional[bool] = None, +) -> Optional[torch.Tensor]: + """``swiglu(m3_mx_gemv_bf16(x, gate_up))`` in one launch: ``weight`` / ``weight_scale`` are the MXFP8 form of a + ``gu_interleave``d gate_up [2F, K]; returns bf16 h [M, F], bit for bit the bf16 GEMV's gate_up rows followed by the + swigluoai of the Triton silu_and_mul kernel (alpha 1.702, limit 7, (up + 1)). None when not served.""" + if x.dtype != torch.bfloat16 or x.dim() != 2 or not x.is_contiguous() or x.data_ptr() % 16 != 0: + return None + return _run( + x, None, weight, weight_scale, True, trigger_early, None, use_pdl, None, False, None, True + ) + + +def m3_mx_gemv_producer( + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + q: torch.Tensor, + index_q: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + trigger_early: bool = True, + use_pdl: Optional[bool] = None, +) -> bool: + """#117: ``m3_producer(m3_mx_gemv(xq, xsf, weight, weight_scale), ...)`` in one launch, bit for bit: the QKV + + index GEMV of the sparse layer (``weight`` [12 x 128, K]) whose owner epilogue hands each token's packed head to + the producer phase (``m3_layer.phase_producer.producer_head``). Writes ``q`` FP8 [T, 8, 128] and ``index_q`` FP8 + [T, 1, 128] and the token's cache slot (NVFP4 K / V + scales, FP8 index-K) exactly as the producer does; the + caches and tensors are those ``minimax_m3_producer_pw`` takes (kv_data [pages, 2, 1, 128, 64] u8, kv_scale [pages, + 2, 1, 128, 8] u8, index_k [pages, 1, 128, 128] e4m3). Returns False (nothing launched) when the call is declined. + """ + m = xq.shape[0] + if ( + m > 8 + or weight.shape[0] != PROD_N + or q.shape != (m, 8, 128) + or index_q.shape != (m, 1, 128) + or not (q.is_contiguous() and index_q.is_contiguous()) + or kv_data.dim() != 5 + or kv_scale.dim() != 5 + or index_k.dim() != 4 + ): + return False + tensors = ( + q, + index_q, + index_k, + kv_data, + kv_scale, + out_cache_loc, + kv_quant_scale, + position_ids, + rotary_cos_sin, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + ) + prod_args = ( + *(t.data_ptr() for t in tensors), + index_k.stride(0), + index_k.stride(2), + kv_data.stride(0), + kv_data.stride(1), + kv_data.stride(3), + kv_scale.stride(0), + kv_scale.stride(1), + kv_data.shape[0], + float(eps), + ) + out = _run( + xq, xsf, weight, weight_scale, False, trigger_early, None, use_pdl, None, False, prod_args + ) + return out is not None + + +_PLACEHOLDERS: Dict[tuple, torch.Tensor] = {} +# Whether the last call pushed its output into an all-reduce (see _run's ``ar``). +last_pushed = False +# The key of the last call declined because compiles are frozen and its kernel was not compiled (see ``freeze``). +last_uncompiled: Optional[tuple] = None +_frozen = [False] + + +def _placeholder_scales(k: int, device) -> torch.Tensor: + key = ("sf", k, device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(k // 128 * 512, dtype=torch.uint8, device=device) + return t + + +def _placeholder_words(device) -> torch.Tensor: + key = ("w", device) + t = _PLACEHOLDERS.get(key) + if t is None: + t = _PLACEHOLDERS[key] = torch.zeros(16, dtype=torch.int32, device=device) + return t + + +def ensure_compiled(weight: torch.Tensor, weight_scale: torch.Tensor, use_pdl: bool) -> None: + """Compiles the kernel of this weight's shape outside capture (a one-token dummy call).""" + if ( + weight.dim() != 2 + or weight.dtype != torch.float8_e4m3fn + or get_config(*weight.shape) is None + ): + return + n, k = weight.shape + if (n, k, use_pdl) in _ensured: + return + _ensured.add((n, k, use_pdl)) + xq = torch.zeros(1, k, dtype=torch.float8_e4m3fn, device=weight.device) + xsf = torch.full((128 * (k // 32),), 127, dtype=torch.uint8, device=weight.device) + m3_mx_gemv(xq, xsf, weight, weight_scale, use_pdl=use_pdl) + for width in MMA_WIDTHS[1:]: + m3_mx_gemv( + torch.zeros(width, k, dtype=torch.float8_e4m3fn, device=weight.device), + xsf, + weight, + weight_scale, + use_pdl=use_pdl, + ) + m3_mx_gemv_bf16( + torch.zeros(1, k, dtype=torch.bfloat16, device=weight.device), + weight, + weight_scale, + use_pdl=use_pdl, + ) + + +def _aot_sources(): + """The Python files this module's CuTe DSL kernels are traced from (the target package and the shared Blackwell + kernel helpers): the ``sources`` of their shipped AOT objects.""" + target = Path(__file__).resolve().parents[2] + shared = Path(cute_aot.__file__).resolve().parent / "cute_dsl_kernels" / "blackwell" + return cute_aot.py_files(str(target), str(shared)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/triton_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/triton_sparse_decode.py new file mode 100644 index 000000000000..2d1c5a628e97 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/triton_sparse_decode.py @@ -0,0 +1,1754 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. + +This is the MiniMax-M3 NVFP4 / sm_107 / tp8 target's copy. On top of the +in-tree decode it routes long-context TP8 steps (8 query heads, one KV head) to the one-launch +``m3_attn_lc8`` kernel, short-sequence verify steps of at least +``MQ_MIN_TOKENS`` rows to the multi-query producer, releases PDL dependents +early, and has the merge emit the MXFP8 o_proj hand-off (``m3_prequant.SLOT``). +""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers + +from . import m3_attn_lc8, m3_prequant, m3_report + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 # capacity: fixed: the MSA block size of the checkpoint + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 # capacity: fixed: a launch-grid target, not a size + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +# Fewest query rows for which the merge writes the MXFP8 o_proj hand-off: every decode step, so the MXFP8 o_proj +# never runs its own quantize kernel after attention. +MERGE_MXFP8_MIN_TOKENS = 1 # capacity: fixed: a row lower bound +# The sparse decode (LC8, the Triton merge) also writes the MXFP8 o_proj hand-off (bit for bit +# torch.ops.trtllm.mxfp8_quantize of its BF16 rows) and publishes it to m3_prequant.SLOT. +MERGE_MXFP8_HANDOFF = True +# The one-launch long-context decode (m3_attn_lc8) at the TP8 heads: its shapes (modeling's SPARSE_DECODE_LC8 keys: +# "prewait" = up to m3_attn_lc8.PREWAIT_MAX_ROWS query rows, "plain" above) and its report path. Short-sequence +# identity steps run it too, taking the identity selection from the device lengths (LC8_IDENTITY, with its merge-once +# form). +LC8_KEYS = frozenset({"plain", "prewait"}) +LC8_PATH = "m3_sparse_decode_lc8" +LC8_IDENTITY = True +# The identity-selection producer's shapes (modeling's SPARSE_DECODE_PRODUCER keys) and its report path. +PRODUCER_KEYS = frozenset({"multi_query", "per_row"}) +PRODUCER_PATH = "sparse_decode_producer" + + +class MergeMxfp8: + """MXFP8 copy of the split-K merge output for the o_proj GEMM. + + From ``MERGE_MXFP8_MIN_TOKENS`` to ``MAX_ROWS`` rows the merge kernel writes the E4M3 rows and swizzled UE8M0 + scales next to the BF16 output and publishes them to ``m3_prequant.SLOT``, so the MXFP8 o_proj skips its own + quantization. One 128-row buffer per width serves every layer: each o_proj consumes it before the next layer's + merge rewrites it, in stream order. + """ + + MAX_ROWS = 128 # capacity: fixed: the merge hand-off's rows (decode steps) + + def __init__(self): + self._buffers = {} + + def buffers(self, width: int, device: torch.device): + key = (width, device) + if key not in self._buffers: + self._buffers[key] = ( + torch.empty((self.MAX_ROWS, width), dtype=torch.float8_e4m3fn, device=device), + torch.zeros((self.MAX_ROWS * (width // 32),), dtype=torch.uint8, device=device), + ) + return self._buffers[key] + + +MERGE_MXFP8 = MergeMxfp8() + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +# Mixed ctx+gen steps launch on [num_ctx:] slices (unaligned seq_lens / block_table) with arbitrary +# total_q; specializing on those compiled a new variant mid-serving (~650 ms TTFT stall at c8). +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + out_ptr, # final [total_q, num_heads, head_dim] for FUSED_MERGE, else unused + counter_ptr, # int32 [total_q, num_kv_heads] arrival counters for FUSED_MERGE, zero at rest + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + num_bt_cols, # block-table width: a top-k id at or past it is not a block of the request + num_pages, # KV pool pages: a page id outside [0, num_pages) is not read + stride_out_n, + stride_out_h, + stride_out_d, + num_kv_heads, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge + FUSED_MERGE: tl.constexpr, # the last chunk to finish merges the partials in place + EARLY_TRIGGER: tl.constexpr, # release PDL dependents (weight prefetch) at entry + IDENTITY: tl.constexpr = False, # identity selection: slot i is block i (t_ptr is not read) +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + # Dependents still wait for this grid to complete; triggering here only + # lets them start their input-independent prologue (e.g. weight loads). + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for it in tl.range(chunk_start, chunk_end): + if IDENTITY: + # Slot `it` of the identity selection, bounded by this row's own kv extent above. + blk = it + else: + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + # Fail closed: a top-k id outside the request's block-table row, or the page it names outside the pool, + # contributes nothing instead of a read through a wrong page (valid inputs are unaffected). + blk_ok = (blk >= 0) & (blk < num_bt_cols) + page = tl.load(bt_row + blk, mask=blk_ok, other=0).to(tl.int64) + blk_ok = blk_ok & (page >= 0) & (page < num_pages) + page = tl.where(blk_ok, page, 0) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + # A slot that failed closed is skipped whole: no score, no rescale (an all-masked first block + # would make exp2(-inf - -inf) = NaN). + if blk_ok: + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + if FUSED_MERGE: + # Split-K serial reduction: every chunk publishes its partial, and the + # chunk that arrives last for this (token, KV head) merges all of them + # exactly as _merge_topk_attn_out_kernel does, then re-arms the counter. + tl.debug_barrier() + counter = counter_ptr + pid_b * num_kv_heads + pid_kh + arrived = tl.atomic_add(counter, 1, sem="acq_rel", scope="gpu") + if arrived == NUM_TOPK_CHUNKS - 1: + tl.debug_barrier() + o_rows = ( + o_ptr + + pid_b * stride_o_b + + (pid_h + off_h[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d + ) + l_rows = lse_ptr + pid_b * stride_l_b + (pid_h + off_h) * stride_l_h + lse_max = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + lse_max = tl.maximum(lse_max, lse_c) + # All chunks empty (a CUDA-graph padding row): zero, never NaN. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + denom = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + denom += tl.exp2(lse_c - lse_max) + denom = tl.where(denom > 0, denom, 1.0) + merged = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + for c in tl.static_range(NUM_TOPK_CHUNKS): + lse_c = tl.load( + l_rows + c * stride_l_c, mask=h_mask, other=float("-inf"), cache_modifier=".cg" + ) + w_c = tl.exp2(lse_c - lse_max) / denom + o_c = tl.load( + o_rows + c * stride_o_c, mask=hd_mask, other=0.0, cache_modifier=".cg" + ).to(tl.float32) + merged += o_c * w_c[:, None] + tl.store( + out_ptr + + pid_b * stride_out_n + + (pid_h + off_h[:, None]) * stride_out_h + + off_d[None, :] * stride_out_d, + merged.to(out_ptr.dtype.element_ty), + mask=hd_mask, + ) + tl.atomic_xchg(counter, 0, sem="relaxed", scope="gpu") + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + num_bt_cols, # block-table width: a top-k id at or past it is not a block of the request + num_pages, # KV pool pages: a page id outside [0, num_pages) is not read + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge + IDENTITY: tl.constexpr = False, # identity selection: slot i is block i (t_ptr is not read) +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + if IDENTITY: + blocks_0 = tl.where(chunk_offsets < count_0, chunk_start + chunk_offsets, sentinel) + blocks_1 = tl.where(chunk_offsets < count_1, chunk_start + chunk_offsets, sentinel) + else: + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for cand in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + if IDENTITY: + blk = chunk_start + cand + else: + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + if IDENTITY: + blk_0 = tl.where(valid_0, idx_0, sentinel) + blk_1 = tl.where(valid_1, idx_1, sentinel) + else: + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + if IDENTITY: + blk = chunk_start + cand + else: + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + # Fail closed as in _gqa_sparse_decode_kernel: an id outside the row, or a page outside the + # pool, reads nothing. + blk_ok = (blk >= 0) & (blk < num_bt_cols) + page = tl.load(bt_row + blk, mask=blk_ok, other=0).to(tl.int64) + blk_ok = blk_ok & (page >= 0) & (page < num_pages) + page = tl.where(blk_ok, page, 0) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + # A slot that failed closed is skipped whole: no score, no rescale (an all-masked first block + # would make exp2(-inf - -inf) = NaN). + if blk_ok: + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.jit +def _rcp_approx_ftz(x): + return tl.inline_asm_elementwise( + "rcp.approx.ftz.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _store_mxfp8_head(x, q8_ptr, sf_ptr, row, head, num_sf_k_tiles, HEAD_DIM: tl.constexpr): + """Quantize one row's [head * HEAD_DIM, (head + 1) * HEAD_DIM) slice to MXFP8 as + torch.ops.trtllm.mxfp8_quantize(x, True) does: UE8M0 scale = round-up(amax * rcp(448)), E4M3 RN-satfinite, + scales in the 128x4-swizzled layout.""" + NUM_SF: tl.constexpr = HEAD_DIM // 32 + xr = tl.reshape(x.to(tl.float32), (NUM_SF, 32)) + amax = tl.max(tl.abs(xr), axis=1) + sfv = amax * _rcp_approx_ftz(tl.full((NUM_SF,), 448.0, tl.float32)) + bits = sfv.to(tl.int32, bitcast=True) + e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(tl.int32) + e = tl.minimum(e, 254) + scale = ((254 - e) << 23).to(tl.float32, bitcast=True) + scale = tl.where(amax != 0, scale, 0.0) + q = (xr * scale[:, None]).to(tl.float8e4nv) + off_q = tl.reshape(tl.arange(0, HEAD_DIM), (NUM_SF, 32)) + tl.store(q8_ptr + row * (num_sf_k_tiles * 128) + head * HEAD_DIM + off_q, q) + k_sf = head * NUM_SF + tl.arange(0, NUM_SF) + off_sf = ( + (row // 128) * num_sf_k_tiles * 512 + + (k_sf // 4) * 512 + + (row % 32) * 16 + + ((row % 128) // 32) * 4 + + k_sf % 4 + ) + tl.store(sf_ptr + off_sf, e.to(tl.uint8)) + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr = False, + q8_ptr=None, # MXFP8 copy of out: [total_q, num_heads * head_dim] E4M3 + sf_ptr=None, # its UE8M0 block scales, 128x4-swizzled + num_sf_k_tiles=0, # ceil(num_heads * head_dim / 128) + MXFP8_OUT: tl.constexpr = False, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + if EARLY_TRIGGER: + # Release the dependent (the o_proj GEMM) before waiting on the attention producer, so its weight + # prefetch overlaps the attention kernel as well as this merge; its griddepcontrol.wait still waits + # for this grid (and therefore the producer) to complete. Because this trigger precedes this grid's own + # wait, it says nothing about what upstream has finished: a dependent may read only step-invariant data + # (weights) before its griddepcontrol.wait, never this step's attention output, q, top-k table or lengths. + tl.extra.cuda.gdc_launch_dependents() + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + if MXFP8_OUT: + _store_mxfp8_head( + o_merged.to(out_ptr.dtype.element_ty), + q8_ptr, + sf_ptr, + pid_b, + pid_h, + num_sf_k_tiles, + BLOCK_SIZE_D, + ) + + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on the datacenter Blackwell and Rubin targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3), (10, 7)) +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +# Multi-query producer (triton_sparse_decode_mq) for identity-selection verify steps with at least this many query +# tokens; below it the per-token producer is as fast. +MQ_MIN_TOKENS = 17 # capacity: fixed: the multi-query producer's crossover +# Warps per multi-query producer program. +MQ_NUM_WARPS = 8 + + +def _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + *, + sm_scale, + output, + decode_query_len, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + identity=False, + identity_step=False, +) -> None: + """minimax_m3_sparse_attn_decode through m3_attn_lc8: same table, output and MXFP8 hand-off contract. ``identity`` + takes the short-sequence identity selection from the device lengths (the table is not read); ``identity_step`` + marks a step with no select (the kernel reads q after its grid wait), also when the staged table is read.""" + use_pdl = get_env_enable_pdl() + total_q, num_heads, head_dim = q.shape + mxfp8 = None + m3_prequant.SLOT.clear() + if MERGE_MXFP8_HANDOFF and MERGE_MXFP8_MIN_TOKENS <= total_q <= MergeMxfp8.MAX_ROWS: + mxfp8 = MERGE_MXFP8.buffers(num_heads * head_dim, output.device) + # The early trigger lets the o_proj launch and stream its weights while attention runs. + m3_attn_lc8.m3_lc8_attn( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + linear_softmax=_sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=1, + gqa_group_size=num_heads, + max_topk=int(topk_idx.shape[-1]), + decode_query_len=decode_query_len, + ), + mxfp8=mxfp8, + use_pdl=use_pdl, + identity=identity, + identity_step=identity_step, + prefetch=m3_attn_lc8.PREFETCH.take(), + next_kv=m3_attn_lc8.NEXT_KV.take(), + ) + if m3_attn_lc8.one_div_applies(total_q): + m3_report.taken(ONE_DIV_MERGE_PATH, total_q) + if mxfp8 is not None: + m3_prequant.SLOT.publish( + output.view(total_q, num_heads * head_dim), mxfp8[0][:total_q], mxfp8[1] + ) + + +# Engagement path of the identity-selection producers reading their blocks from the kv lengths (IDENTITY). +DEVICE_IDENTITY_PATH = "sparse_decode_device_identity" +# Engagement path of the LC's one-division split-K merge (MSA_ONE_DIV_MERGE). +ONE_DIV_MERGE_PATH = "sparse_decode_lc_one_div_merge" + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + if kv_nvfp4: + capability = torch.cuda.get_device_capability(q.device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103/SM107; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + # Long-context TP8 steps: attention, split-K merge and MXFP8 hand-off in one launch. A selected table on an NVFP4 + # cache with FP8 q at the TP8 heads (8 query heads, one KV head) is the kernel's (LC8_KEYS: "prewait" up to + # m3_attn_lc8.PREWAIT_MAX_ROWS query rows); operands it declines are a miss. + lc_key = "prewait" if total_q <= m3_attn_lc8.PREWAIT_MAX_ROWS else "plain" + lc_covered = ( + kv_nvfp4 + and widen_q + and (not identity_blocks or (LC8_IDENTITY and m3_attn_lc8.MERGE_ONCE)) + and num_topk_chunks is None + and q.dim() == 3 + and int(q.shape[1]) == m3_attn_lc8.NUM_HEADS + and num_kv_heads == 1 + and lc_key in LC8_KEYS + ) + if lc_covered and not m3_attn_lc8.lc8_applies( + q, k_paged, v_paged, topk_idx, k_block_scale, v_block_scale, output + ): + m3_report.missed( + LC8_PATH, + total_q, + "the step's q / KV pages / scales / top-k table / output are not the layout the kernel is built for", + ) + lc_covered = False + if lc_covered: + _longattn_lc_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + # The identity selection from the device lengths (the staged table is not read). + identity=identity_blocks, + # Every identity step (no select ran): q after the grid wait. + identity_step=identity_blocks, + ) + m3_report.taken(LC8_PATH, total_q) + return + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else 1 + ) + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + # Every query token of a request reads the same blocks under identity selection (short-sequence mode): one + # program per (request, block) covers all of them and dequantizes each block once. + use_mq = ( + identity_blocks + and total_q >= MQ_MIN_TOKENS + and kv_nvfp4 + and num_kv_heads == 1 + and decode_query_len > 1 + and head_dim == 128 + and gqa_group_size * decode_query_len <= 128 + ) + if identity_blocks and kv_nvfp4 and widen_q: + # The identity-selection route is modeling's SPARSE_DECODE_PRODUCER ("per_row"; "multi_query" needs one KV + # head, so it never runs at the TP2 heads); a variant the range does not certify is a miss (raises under + # require). + producer_key = "multi_query" if use_mq else "per_row" + if producer_key not in PRODUCER_KEYS: + m3_report.missed( + PRODUCER_PATH, + total_q, + f"the identity-selection producer variant {producer_key!r} is not certified at {num_kv_heads} KV heads", + ) + if use_mq: + num_topk_chunks = max_topk + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = get_env_enable_pdl() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + device_identity = identity_blocks + if device_identity and not use_mq: + m3_report.taken(DEVICE_IDENTITY_PATH, total_q) + single_kernel_args = ( + { + # No split-K arrival counters: the separate merge kernel combines the chunks. + "merge": (output, None), + "tail": (output.stride(0), output.stride(1), output.stride(2), num_kv_heads), + # An early release pays off when the dependent is the consumer of the + # final output (it can prefetch), or a merge grid that itself releases + # its dependent at entry: the o_proj then launches during this kernel and + # streams its weights while attention runs. Above 8 query rows the + # concurrent weight stream slows the larger attention grid more than the + # prefetch saves (GB300 TP4: -1.1% at 48 rows). + "options": {"FUSED_MERGE": False, "EARLY_TRIGGER": direct_output or total_q <= 8}, + } + if query_group_size == 1 + else {"merge": (), "tail": (), "options": {}} + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + if use_mq: + from .triton_sparse_decode_mq import launch_mq_producer + + launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + sm_scale=sm_scale, + decode_query_len=decode_query_len, + use_pdl=use_pdl, + early_trigger=False, + num_warps=MQ_NUM_WARPS, + ) + else: + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + *single_kernel_args["merge"], + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + block_table.shape[1], + k_paged.shape[0], + *single_kernel_args["tail"], + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + # Identity selection from the lengths this kernel reads, not the staged table. + IDENTITY=device_identity, + **single_kernel_args["options"], + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + m3_prequant.SLOT.clear() + if not direct_output: + width = num_heads * head_dim + mxfp8 = ( + MERGE_MXFP8.buffers(width, output.device) + if MERGE_MXFP8_HANDOFF + and MERGE_MXFP8_MIN_TOKENS <= total_q <= MergeMxfp8.MAX_ROWS + and head_dim == 128 + and output.dim() == 3 + and output.is_contiguous() + and output.dtype == torch.bfloat16 + else None + ) + mxfp8_args = ( + { + "q8_ptr": mxfp8[0], + "sf_ptr": mxfp8[1], + "num_sf_k_tiles": width // 128, + "MXFP8_OUT": True, + } + if mxfp8 is not None + else {} + ) + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + # Release the o_proj at the merge's entry so it streams its weights during the merge. + EARLY_TRIGGER=True, + **mxfp8_args, + **pdl_launch, + **merge_launch_options, + ) + if mxfp8 is not None: + m3_prequant.SLOT.publish(output.view(total_q, width), mxfp8[0][:total_q], mxfp8[1]) + + +__all__ = [ + "MERGE_MXFP8", + "MERGE_MXFP8_MIN_TOKENS", + "MergeMxfp8", + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/triton_sparse_decode_mq.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/triton_sparse_decode_mq.py new file mode 100644 index 000000000000..048d31b82d68 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/kernels/triton_sparse_decode_mq.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multi-query block-sparse decode producer for MiniMax-M3 short-sequence verify steps. + +In short-sequence mode every request attends at most ``topk`` KV blocks and block selection is the identity, so +all ``decode_query_len`` query tokens of a request read the same blocks. The per-token producer +(``_gqa_sparse_decode_kernel``) then dequantizes every block once per query token. This producer runs one program +per (request, block) for all of the request's query tokens: rows are (token, head) pairs, each block is +dequantized once, and the causal limit is a per-row mask. + +It writes the same per-block partials (output scaled by 1/l and the V global scale, log2 LSE) as the per-token +producer with one block per chunk, so ``_merge_topk_attn_out_kernel`` merges them unchanged. +""" + +import triton +import triton.language as tl + +from .triton_sparse_decode import _dequant_nvfp4_rows + + +@triton.jit( + do_not_specialize=["num_reqs"], do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"] +) +def _gqa_sparse_decode_mq_kernel( + q_ptr, # [num_reqs * QLEN, num_heads, head_dim], FP8 or FP16/BF16 + k_ptr, # packed NVFP4 K: [num_pages, 1, page_size, head_dim / 2] uint8 + v_ptr, # packed NVFP4 V: same layout + k_sc_ptr, # K block scales [num_pages, 1, page_size, head_dim / 16], token-linear + v_sc_ptr, # V block scales, 4x4 token-quad order + k_gs_ptr, + v_gs_ptr, + t_ptr, # topk table [1, total_q, topk]; the request's last token's row is read + o_ptr, # partials [topk, total_q, num_heads, head_dim] fp32 + lse_ptr, # partials [topk, total_q, num_heads] fp32 (log2) + block_table_ptr, + seq_lens, + num_reqs, + sm_scale, + stride_qn, + stride_qh, + stride_k_blk, + stride_k_pos, + stride_v_blk, + stride_v_pos, + stride_ksc_blk, + stride_vsc_blk, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + QLEN: tl.constexpr, + GROUP: tl.constexpr, # query heads per KV head (16) + BLOCK_R: tl.constexpr, # next power of two of QLEN * GROUP + HEAD_DIM: tl.constexpr, + BLOCK_SIZE_K: tl.constexpr, + SCALE_COLS: tl.constexpr, + USE_PDL: tl.constexpr, + EARLY_TRIGGER: tl.constexpr, +): + pid_r = tl.program_id(0) + pid_c = tl.program_id(1) # top-k position == block index under identity selection + sm_scale_log2e = sm_scale * 1.4426950409 * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + + off_r = tl.arange(0, BLOCK_R) + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, HEAD_DIM) + off_dp = tl.arange(0, HEAD_DIM // 2) + off_s = tl.arange(0, SCALE_COLS) + row_t = off_r // GROUP + row_h = off_r % GROUP + r_mask = off_r < QLEN * GROUP + tok = pid_r * QLEN + row_t + + if USE_PDL: + tl.extra.cuda.gdc_wait() + if EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + seq_len = tl.load(seq_lens + pid_r) + # Token t of the request sits at seq_len - QLEN + t and attends positions [0, kv_len_t). + kv_len = tl.maximum(seq_len - QLEN + row_t + 1, 0) + blk = tl.load(t_ptr + (pid_r * QLEN + QLEN - 1) * stride_tn + pid_c * stride_tk) + blk_ok = blk >= 0 + blk = tl.where(blk_ok, blk, 0) + page = tl.load(block_table_ptr + pid_r * stride_bt_b + blk, mask=blk_ok, other=0).to(tl.int64) + # Rows past the last token's position are never read: their cache bytes may be stale (0 * NaN in the PV dot). + n_valid = tl.where( + blk_ok, tl.minimum(tl.maximum(seq_len - blk * BLOCK_SIZE_K, 0), BLOCK_SIZE_K), 0 + ) + n_mask = off_n < n_valid + + q = tl.load( + q_ptr + tok[:, None] * stride_qn + row_h[:, None] * stride_qh + off_d[None, :], + mask=r_mask[:, None], + other=0.0, + ).to(tl.float16) + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + page * stride_k_blk + off_n[:, None] * stride_k_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + page * stride_ksc_blk + off_n[:, None] * SCALE_COLS + off_s[None, :], + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ) + valid = ( + (blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None]) & n_mask[None, :] & r_mask[:, None] + ) + qk = tl.dot(q, tl.trans(k_deq).to(tl.float16)) * sm_scale_log2e + qk = tl.where(valid, qk, float("-inf")) + m = tl.max(qk, axis=1) + m_safe = tl.where(m == float("-inf"), 0.0, m) + p = tl.exp2(qk - m_safe[:, None]) + p_sum = tl.sum(p, axis=1) + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + page * stride_v_blk + off_n[:, None] * stride_v_pos + off_dp[None, :], + mask=n_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=n_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=HEAD_DIM, + SCALE_COLS=SCALE_COLS, + SF_VEC=HEAD_DIM // SCALE_COLS, + ).to(tl.float16) + acc = tl.dot(p.to(tl.float16), v) + # An empty partial stores zero with LSE -inf, so the merge gives it weight 0. + scale = tl.where(p_sum > 0.0, 1.0 / p_sum, 0.0) * v_global_scale + lse = tl.where(p_sum > 0.0, m + tl.log2(p_sum), float("-inf")) + acc = acc * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + tok[:, None] * stride_o_b + + row_h[:, None] * stride_o_h + + off_d[None, :], + acc, + mask=r_mask[:, None], + ) + tl.store(lse_ptr + pid_c * stride_l_c + tok * stride_l_b + row_h * stride_l_h, lse, mask=r_mask) + if USE_PDL and not EARLY_TRIGGER: + tl.extra.cuda.gdc_launch_dependents() + + +def launch_mq_producer( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + block_table, + seq_lens, + o_arg, + lse_arg, + *, + sm_scale, + decode_query_len, + use_pdl, + early_trigger, + num_warps=8, + num_stages=1, +): + """Launch the multi-query producer; arguments as minimax_m3_sparse_attn_decode after its NVFP4 views.""" + total_q, num_heads, head_dim = q.shape + num_reqs = total_q // decode_query_len + max_topk = int(topk_idx.shape[-1]) + rows = decode_query_len * num_heads + pdl_launch = {"launch_pdl": True} if use_pdl else {} + _gqa_sparse_decode_mq_kernel[(num_reqs, max_topk)]( + q, + k_paged, + v_paged, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + num_reqs, + sm_scale, + q.stride(0), + q.stride(1), + k_paged.stride(0), + k_paged.stride(2), + v_paged.stride(0), + v_paged.stride(2), + k_block_scale.stride(0), + v_block_scale.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + o_arg.stride(0), + o_arg.stride(1), + o_arg.stride(2), + lse_arg.stride(0), + lse_arg.stride(1), + lse_arg.stride(2), + block_table.stride(0), + QLEN=decode_query_len, + GROUP=num_heads, + BLOCK_R=triton.next_power_of_2(rows), + HEAD_DIM=head_dim, + BLOCK_SIZE_K=int(k_paged.shape[2]), + SCALE_COLS=head_dim // 16, + USE_PDL=use_pdl, + EARLY_TRIGGER=early_trigger, + num_warps=num_warps, + num_stages=num_stages, + **pdl_launch, + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/m31_switches.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/m31_switches.py new file mode 100644 index 000000000000..51f966c595fd --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/m31_switches.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Kill switches of this MiniMax-M3.1 target's default-on items (each target keeps its own copy: targets share +only the catalog). + +Every item is on by default; ``TRTLLM_M31_DISABLE_=1`` turns item ```` off and restores the path it replaced. +The environment is read once, at import, so every rank and every step of a process sees the same setting (a step must +not change its graph mid-run). Each item lists its id and what the switch restores where it reads it. + +Values (case-insensitive, surrounding blanks ignored): ``1`` / ``true`` / ``on`` / ``yes`` set a switch; empty / ``0`` / +``false`` / ``off`` / ``no`` leave it unset; anything else raises at import, naming the variable, so a typo fails loudly +instead of silently turning an item on or off. +""" + +import os +from typing import FrozenSet + +_PREFIX = "TRTLLM_M31_DISABLE_" +_SET = frozenset(("1", "true", "on", "yes")) +_UNSET = frozenset(("", "0", "false", "off", "no")) + + +def _is_set(name: str, value: str) -> bool: + """Whether the switch variable ``name`` is set to ``value``; raises on a value of neither kind.""" + word = value.strip().lower() + if word in _SET: + return True + if word in _UNSET: + return False + raise ValueError( + f"{name}={value!r}: expected 1 / true / on / yes (set) or 0 / false / off / no / empty (unset)" + ) + + +# The item ids switched off in this process, read once at import. +_DISABLED: FrozenSet[str] = frozenset( + name[len(_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_PREFIX) and _is_set(name, value) +) + + +def m31_disabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_DISABLE_`` turned the item off for this process.""" + return item_id in _DISABLED + + +# Items that are OFF by default (rule 1b without a passing gate c, or a failed gate c): ``TRTLLM_M31_ENABLE_=1`` +# opts in; the item's ``TRTLLM_M31_DISABLE_`` still wins. +_ENABLE_PREFIX = "TRTLLM_M31_ENABLE_" + +_ENABLED: FrozenSet[str] = frozenset( + name[len(_ENABLE_PREFIX) :] + for name, value in os.environ.items() + if name.startswith(_ENABLE_PREFIX) and _is_set(name, value) +) + + +def m31_enabled(item_id: str) -> bool: + """Whether ``TRTLLM_M31_ENABLE_`` opted a default-off item in for this process.""" + return item_id in _ENABLED diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py new file mode 100644 index 000000000000..184b2cc201f1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py @@ -0,0 +1,6919 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""ModelingV2 target: MiniMax-M3.1 (MXFP8 + packed NVFP4 experts) / sm_107 / tp8 -- self-contained modeling code. + +MiniMax-M3.1's language model has MiniMax-M3's shape (60 layers, hidden 6144, 64 query heads and 4 KV heads of dim +128, 128 routed experts top 4 plus one shared expert) with a 1e7 RoPE base and MSA block-sparse attention on EVERY +layer (an index branch scores 128-token blocks and the top 16 are attended); layers 0-2 keep the dense SwiGLU-OAI MLP, +layers 3-59 the MoE. The checkpoint is an MXFP8 export whose routed experts are packed NVFP4 (compressed-tensors +names, global scales stored as multipliers; ``weights`` takes their FP32 reciprocals); the KV cache is NVFP4 with an +FP8 index-K on every layer. The layer kinds are read from the config (``sparse_attention_freq``, ``moe_layer_freq``), +so this code is the MiniMax-M3 target's with no dense-attention layer in use. + +``tp8`` is eight Rubin GPUs over two trays (multi-node NVLink): head-split attention (8 query heads, one KV head and +one index head per rank, each KV head shared by two ranks) and tensor-parallel experts (every expert on every rank, +intermediate 384 per rank), no attention DP. +Speculative decoding, when configured, is one-model Eagle3 (draft length 3): the BF16 drafter (``M3Eagle3Draft``) and +the BF16 target head, both at checkpoint precision; decode steps of up to 2 requests verify and draft through fused +kernels at that precision (``M3Eagle3Worker``). + +**The core** (``MiniMaxM3Tp8ModelingV2``) holds the weights (``weights.MODEL_WEIGHTS``) and the runtime resources the +steps use -- one MSA attention backend per layer, the TP all-reduce communicator, the RoPE table, the MXFP8 GEMM +dispatch state the engine tunes -- and dispatches each step, on the host, from its context count alone +(``_select_target``): + +* ``DecodeTarget`` takes every pure generation step. Up to ``DECODE_MAX_TOKENS`` tokens (up to 2 requests, each one + token or a draft verify of 1 + 3 rows) with 1-D input ids it runs the decode-step kernels and picks each layer + boundary's collective by the step's token bucket (``DECODE_PLANS``); it runs its own generic layer for the first + layer when the fused embedding does not take the step, and for every layer of a wider step. +* ``PrefillTarget`` takes every step with context rows: prefill chunks and mixed steps, every layer in its generic + form over the MSA backend. + +The two targets share no body: each owns its layer, attention and MoE compositions and its ``step_args``, the one +projection of the step metadata its forward reads. What both call are the module-level op adapters, each picking the +kernel for one call from shapes and layouts, which read no step metadata themselves. + +Every fast path the decode step can take, the rows each covers (``FAST_PATHS``, each paired with the entry whose +``CELLS`` certify it) and what a step that cannot take one does are below: **uncovered** (outside every covered +range: log once, run the generic path) or **missed** (inside a covered range but a precondition is missing: raise +``FastPathUnavailable`` under ``modeling_v2='require'``, else log once and fall back). A row range is empty unless a +landed cell has certified it at the TP8 geometry on Rubin. Every fast path a GEN rank takes prints a rank-tagged +engagement line once per (path, rows) (``taken``). Kernel modules report through ``kernels/m3_report``, which this +module wires to the same policy at import. + +**Hand-offs between kernels.** A producer that already holds an activation in registers writes its MXFP8 form for the +next MXFP8 projection: the decode boundary all-reduce for the next layer's QKV, the sparse-decode merge and the +long-context decode for the o_proj. It publishes it in ``kernels/m3_prequant.SLOT`` keyed by the BF16 tensor; the next +MXFP8 projection on a decode graph takes it, and every take clears the slot. The decode boundaries' push targets (the +o_proj / dense-down / shared-expert GEMVs and the MoE engine writing straight into the boundary all-reduce's exchange) +are passed explicitly from the layer to the producer and back. + +Compiled kernels (the MoE engine, the decode GEMVs with their push variants, the index scorer, the long-row select, +the full-context decode plans) are built before the executor's warmup; a forward never compiles one, and one it finds +missing is a miss. Other state a fast path needs (multicast workspaces, push descriptors) is built on the eager decode +steps; a CUDA-graph capture only reads it, so a capture that finds it missing is a miss (``capture_missed``). + +**What this target asserts rather than adapts**: SM 10.7, the topology and checkpoint (``routing.py``); the +MiniMax-M3 MSA sparse attention runtime with the fused five-way QKV/index projection, an FP8 indexer and an FP8 or +NVFP4 KV cache, no torch.compile and no piecewise prefill graphs (``_validate_runtime``); text-only input (the vision +tower is not built and a step carrying image input raises). +""" + +from __future__ import annotations + +import atexit +import copy +import dataclasses +import os +import threading +import time +from dataclasses import dataclass +from types import MappingProxyType +from typing import ( + Any, + Callable, + Dict, + FrozenSet, + Hashable, + Iterable, + List, + Mapping, + Optional, + Tuple, + Union, +) + +import torch +from torch import nn +from transformers import PretrainedConfig + +from tensorrt_llm._torch._experimental.modeling_v2._router_index import ( + ModelingV2Mode, + resolved_mode, + step_contract_enabled, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import ( + m31_fake_ops, # noqa: F401 (registers the fake impls of the M3.1 torch ops) +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.moe.fp4_block_scale_moe_runner import ( + fp4_block_scale_moe_runner, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( + flashinfer_fused_add_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_rmsnorm import ( + flashinfer_rmsnorm, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.quantization.fp4_quantize import ( + fp4_quantize, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 import m31_pins +from tensorrt_llm._torch.attention.backends import AttentionMetadata +from tensorrt_llm._torch.attention.backends.fmha.msa_prefill import _aligned_nvfp4_dequant_scales +from tensorrt_llm._torch.attention.backends.interface import PositionalEmbeddingParams, RopeParams +from tensorrt_llm._torch.attention.backends.prims_ts import _worktile_scope +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import MiniMaxM3MsaSparseAttention +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + MSA_REQUIRED_TOPK, + msa_paged_kv, + require_msa_module, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) +from tensorrt_llm._torch.attention.backends.utils import create_attention +from tensorrt_llm._torch.custom_ops.torch_custom_ops import ( + FlashInferMXFP8GemmRunner, + MXFP8QuantizeRunner, +) +from tensorrt_llm._torch.distributed import AllReduce, AllReduceFusionOp, AllReduceParams +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl +from tensorrt_llm._torch.model_config import ModelConfig +from tensorrt_llm._torch.models.modeling_speculative import ( + _ensure_draft_vocab_size, + _get_requested_draft_moe_backend, + _set_draft_kv_cache_quant_algo, + _slice_spec_position_ids, +) +from tensorrt_llm._torch.models.modeling_utils import DecoderModel, DecoderModelForCausalLM +from tensorrt_llm._torch.modules.embedding import LMHead +from tensorrt_llm._torch.modules.linear import ( + MXFP8LinearMethod, + TensorParallelMode, + flashinfer_mxfp8_autotune_active, + flashinfer_mxfp8_decode_graph_capture_active, +) +from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor +from tensorrt_llm._torch.modules.multi_stream_utils import maybe_execute_in_parallel +from tensorrt_llm._torch.moe.fused_moe.routing import RoutingMethodType +from tensorrt_llm._torch.pyexecutor.breakable_cuda_graph import is_in_breakable_cuda_graph +from tensorrt_llm._torch.pyexecutor.h2d_cache import copy_h2d_if_changed +from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import MambaHybridCacheManager +from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy, RuntimeHooks +from tensorrt_llm._torch.pyexecutor.sampler.ops.flashinfer import ( + set_softmax_override, + set_top_p_renorm_override, +) +from tensorrt_llm._torch.speculative import SpecMetadata +from tensorrt_llm._torch.speculative.eagle3 import Eagle3OneModelWorker +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + set_chain_speculative_sampling_override, + set_chain_speculative_sampling_slot_override, + should_use_separate_draft_kv_cache, +) +from tensorrt_llm._torch.utils import ActType_TrtllmGen +from tensorrt_llm._utils import mpi_rank, prefer_pinned +from tensorrt_llm.functional import PositionEmbeddingType +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.mode import QuantAlgo + +from . import dspark as _dspark +from . import weights as _weights +from .kernels import m3_draft_step as m3_draft_step_kernel +from .kernels import ( + m3_drafter_attn, + m3_fc_wait, + m3_fold_tail, + m3_mc_allreduce, + m3_prequant, + m3_report, + m3_spec_tail, +) +from .kernels import m3_fullctx_decode as m3_fullctx_decode_kernel +from .kernels import m3_index_score as _index_score_kernel +from .kernels import m3_sampled_accept as m3_sampled_accept_kernel +from .kernels import m3_shard_accept as m3_shard_accept_kernel +from .kernels import msa_kv_update as _msa_kv_update_kernel +from .kernels import triton_sparse_decode as _sparse_decode_kernel +from .kernels.long_select import long_select_blocks +from .kernels.mx_gemv import op as mx_gemv +from .m31_switches import m31_disabled, m31_enabled +from .ops import gqamix_sparse_decode as _gqamix_kernel +from .ops.activation.m3_silu_and_mul import m3_silu_and_mul +from .ops.activation.m3_swq import m3_swq +from .ops.attention.m3_draft_attention import m3_draft_attention +from .ops.attention.m3_fp8_indexer_qk_norm_rope import m3_fp8_indexer_qk_norm_rope +from .ops.attention.m3_fullctx_decode import m3_fullctx_decode +from .ops.attention.m3_index_score import m3_index_score +from .ops.attention.m3_msa_attention import m3_msa_attention +from .ops.attention.m3_producer import m3_producer +from .ops.attention.m3_qk_norm_rope_fp8 import m3_qk_norm_rope_fp8 +from .ops.attention.m3_qkv_insert import m3_qkv_insert +from .ops.attention.m3_select_blocks import m3_select_blocks +from .ops.attention.m3_select_long import m3_select_long +from .ops.attention.m3_select_long_dsl import m3_select_long_dsl +from .ops.attention.m3_sparse_decode import m3_sparse_decode +from .ops.attention.m3_sparse_decode_lc8 import m3_sparse_decode_lc8 +from .ops.comm.m3_allreduce import m3_allreduce +from .ops.comm.m3_mc_allreduce_norm import m3_mc_allreduce_norm +from .ops.embedding.m3_embedding import m3_embedding +from .ops.gemm import m3_mxfp8_cublaslt as mxfp8_cublaslt +from .ops.gemm.m3_bf16_linear import m3_bf16_linear +from .ops.gemm.m3_dense_swiglu import ALPHA as DENSE_ALPHA +from .ops.gemm.m3_dense_swiglu import BETA as DENSE_BETA +from .ops.gemm.m3_dense_swiglu import LIMIT as DENSE_LIMIT +from .ops.gemm.m3_dense_swiglu import m3_dense_swiglu +from .ops.gemm.m3_draft_lowm import ROWS as DRAFT_LOWM_TACTIC_ROWS +from .ops.gemm.m3_draft_lowm import m3_draft_lowm, tactic_for +from .ops.gemm.m3_fp32_linear import m3_fp32_linear +from .ops.gemm.m3_mx_gemv import ( + DENSE_DOWN, + DENSE_GATE_UP, + GEMV48_MIN_TOKENS, + GEMV48_SHAPES, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, + m3_mx_gemv, +) +from .ops.gemm.m3_mx_gemv_bf16 import m3_mx_gemv_bf16 +from .ops.gemm.m3_mxfp8_cublaslt import m3_mxfp8_cublaslt +from .ops.gemm.m3_mxfp8_gemm import m3_mxfp8_gemm +from .ops.gemm.m3_mxfp8_linear import m3_mxfp8_linear +from .ops.moe.m3_moe_engine import M3MoeState, ar_push_desc, m3_moe_engine, shared_ctas_for +from .ops.moe.m3_router_tc import m3_router_tc +from .ops.norm.m3_embedding_firstnorm import DecodeEmbedding, m3_embedding_firstnorm +from .ops.sampling import m31_chain_spec_mc as _chain_spec_mc +from .ops.sampling import m31_softmax_mc as _softmax_mc +from .ops.sampling import m31_topp_det as _topp_det +from .ops.spec.m3_capture_norm import m3_capture_norm +from .ops.spec.m3_chunk_norm import m3_chunk_norm +from .ops.spec.m3_draft_input import m3_draft_input +from .ops.spec.m3_draft_silu import m3_draft_silu +from .ops.spec.m3_draft_step import m3_draft_step +from .ops.spec.m3_force_vec import m3_force_vec, m3_force_vec_req +from .ops.spec.m3_sampled_accept import m3_sampled_accept +from .ops.spec.m3_shard_accept import m3_shard_accept +from .ops.spec.m3_spec_offsets import m3_spec_offsets +from .ops.spec.m3_step_prologue import m3_step_prologue + +_SM = (10, 7) +# PROXYPB: compile the MSA indexer proxy's fmha_sm100 variants before warmup (``_prebuild_msa_proxy``). Compile and +# load only, no numerics change. TRTLLM_M31_DISABLE_PROXYPB=1: each variant compiles on its first call instead (which +# can be after the server is ready). +MSA_PROXY_PREBUILD = os.environ.get("TRTLLM_M31_DISABLE_PROXYPB", "0").strip() in ("", "0") +# R6: the dense-MLP layer boundary also publishes the next layer's MXFP8 QKV activation (as the MoE boundary does): +# M3.1's layers 1-3 follow a dense MLP but have the sparse five-way QKV. TRTLLM_M31_DISABLE_R6=1 restores the BF16 +# hand-off (the next QKV quantizes its own input). +# VR: OFF by default. On sm_107 the hand-off's QKV output is not bitwise against the BF16 path and +# R6 has no passing gate c (with R6 off, gate c passes). +# TRTLLM_M31_ENABLE_R6=1 turns it on (for its gate c); TRTLLM_M31_DISABLE_R6=1 still forces it off. +_R6_ENABLED = os.environ.get("TRTLLM_M31_ENABLE_R6", "0").strip() not in ("", "0") +_R6_DISABLED = os.environ.get("TRTLLM_M31_DISABLE_R6", "0").strip() not in ("", "0") +R6_DENSE_PREQUANT = _R6_ENABLED and not _R6_DISABLED +# CTXNORMQ: a prefill layer boundary of at least ``CTXNORMQ_MIN_ROWS`` rows runs the plain all-reduce, then +# one kernel for the residual add + RMSNorm that also writes the next MXFP8 projection's activation (and, at a MoE +# boundary, the routed experts' NVFP4 input), instead of the fused all-reduce + norm followed by the consumers' own +# quantize passes. The quantized outputs equal mxfp8_quantize / fp4_quantize of the norm bit for bit; the norm itself +# is rule 1b against the fused all-reduce's (reduction order). Kill switch: TRTLLM_M31_DISABLE_CTXNORMQ=1. + +# PrimTS (the full-context decode) and FA4 trace in one process: scope task scheduling's WorkTileInfo rebind to the +# PrimTS traces (attention/backends/prims_ts/_worktile_scope.py). +_worktile_scope.arm() + +CTXNORMQ = os.environ.get("TRTLLM_M31_DISABLE_CTXNORMQ", "0").strip() in ("", "0") +# GQAMIX: a mixed ctx+gen step's sparse generation rows (MsaDecodeFmha) run the Triton block-sparse decode +# through ops/gqamix_sparse_decode, a verbatim copy of the in-tree kernel module whose two decode kernels do not +# specialize on the step's row count or on the [num_ctx:] slices' alignment, registered on MsaDecodeFmha's +# sparse_decode_hook. Those specializations compiled new variants after the server was ready. Kill switch: +# TRTLLM_M31_DISABLE_GQAMIX=1. +GQAMIX = not m31_disabled("GQAMIX") +# DSFOLD: under the standalone DSpark worker, the engine's previous-batch input gathers (input ids, draft tokens, +# position / KV-length offsets) run as one m3_spec_tail.step_gather launch instead of ~a dozen eager index and copy +# launches at the step boundary. Integer copies of the same values into the same buffers; the sampler keeps its +# in-tree slot stores. Kill switch: TRTLLM_M31_DISABLE_DSFOLD=1. +DSFOLD = not m31_disabled("DSFOLD") +CTXNORMQ_MIN_ROWS = ( + 1024 # capacity: NORMQ microbench (GB300): the fused pass wins at 1160 and 16384 rows +) +# -------------------------------------------------------------------------------------------------------------------- +# Row ranges +# -------------------------------------------------------------------------------------------------------------------- + +# The rows every TP2-lineage fast path below was certified for: 2 requests verifying 1 + 3 drafts (the GEN worker's +# batch). The embedding + first norm, the boundary pushes, the generic layer's multicast all-reduce, the bf16 GEMVs, the +# MoE engine and the Eagle3 tail keep this bound. +TP2_ROWS = 8 # capacity: fixed: the TP2-lineage paths' certified rows +# R1VR8: the decode step bound is 32 rows (8 requests verifying 1 + 3 drafts, the K3 c8 step), so a c4 / c8 step +# stays on the decode layer instead of running every layer's generic form. Each fast path keeps its own row range: a +# step past it runs that op's generic form (the MoE above the engine's 8 rows: the router + routed experts; the +# boundary pushes above 8 rows: the plain multicast all-reduce). The multicast all-reduce + norm takes the 16- and +# 48-token buckets (bitwise against the 8-row path at 1..32 rows, 8 ranks on 2 trays) and publishes the +# next MXFP8 projection's activation. Steps of up to 8 rows are unchanged. +# TRTLLM_M31_DISABLE_R1VR8=1 restores the 8-row bound (every wider step runs the generic layers). +R1VR8 = os.environ.get("TRTLLM_M31_DISABLE_R1VR8", "0").strip() in ("", "0") +DECODE_MAX_TOKENS = 32 if R1VR8 else TP2_ROWS # capacity: fixed: the decode kernels' step bound +# VR-BANDS (MIDCONC): the decode step bound is 64 rows (8 requests verifying 1 + 7 drafts, the K7 c8 step) while every +# op keeps its own certified row range above (DECODE_MAX_TOKENS and the per-op tables): an op past its range runs its +# generic form and the rest of the step stays on the decode layer, instead of the whole step running every layer's +# generic form past 32 rows. Steps of 33..64 rows take the "wide" plan (the built-in all-reduce at both boundaries; the +# multicast kernel's buckets stop at 48 and its certified rows at 32). TRTLLM_M31_DISABLE_VR_BANDS=1 restores the +# 32-row step bound. Read once, at import. +VR_BANDS = R1VR8 and not m31_disabled("VR_BANDS") +DECODE_STEP_MAX = ( + 64 if VR_BANDS else DECODE_MAX_TOKENS +) # capacity: fixed: the step bound; ops keep their own rows +# SWQ: past 8 rows the shared expert runs unfused (the 1..8-token MoE engine folds it in below) and its down +# projection (6144, 384) has no GEMV row table, so it would take silu_and_mul + mxfp8_quantize + the MXFP8 GEMM. +# The SwiGLU instead writes the down projection's MXFP8 activation directly (ops/activation/m3_swq.py: bitwise to +# silu_and_mul + mxfp8_quantize on every block that quantize emits finite; a scale-code-0 block, which it emits as +# inf / NaN, is kept finite), taken by ``_linear``'s prequant branch. Rows 9..32: R1VR8's decode step bound. +# TRTLLM_M31_DISABLE_SWQ=1 restores the bf16 SwiGLU + quantize. +SWQ_SHARED_DOWN = (6144, 384) # the shared expert's down projection at the TP8 width 384 +SWQ_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + {} + if os.environ.get("TRTLLM_M31_DISABLE_SWQ", "0").strip() not in ("", "0") + else {SWQ_SHARED_DOWN: frozenset(range(TP2_ROWS + 1, DECODE_MAX_TOKENS + 1))} +) # capacity: fixed: rows past the engine's fold up to the step bound +_SWQ_LOGGED: set = set() + + +@dataclass(frozen=True) +class DecodePlan: + """The layer-boundary collectives of one decode token bucket (GB300 tp4 measurements in the comments).""" + + name: str + max_tokens: int + # The attention boundary's multicast all-reduce + residual + RMSNorm (5.05 vs 5.27 us native at 1 token). + attn_mc_ar: bool + # The same kernel at the MoE boundary with the shared + routed add folded in: wins up to 48 tokens. + moe_mc_ar: bool + # The MoE-boundary all-reduce also emits the next QKV activation in MXFP8 for the decode GEMV / GEMM. + qkv_prequant: bool + + +# tiny: every step up to 8 tokens (c1..c8 without speculation, the c1 verify, every draft pass); small: the c2 +# verify; verify: the c4 / c8 verifies. Each is one multicast all-reduce bucket (kernels/m3_mc_allreduce.BUCKETS). +# The multicast all-reduce + residual + RMSNorm at both boundaries of every TP2 decode step (one 8-token bucket), the +# MoE boundary's also publishing the next QKV activation in MXFP8. SM107 TP2 hop: 3.5..4.4 us at 1..8 rows against +# 8.7..9.5 us for the stock AUTO (NCCL) all-reduce + fused residual RMSNorm, and 13.3..14.0 us at the MoE boundary +# (stock add + all-reduce + mxfp8_quantize); bitwise equal to stock (ops/comm/m3_mc_allreduce_norm). +# False puts both boundaries back on the built-in all-reduce. +# On by default: the intermittent illegal memory access first seen with this stack on was the +# built-in Triton sparse decode's (since fixed); with the stack on, GSM8K 4-in-flight ran crash-free. +MC_AR_TP2 = True +DECODE_PLANS = ( + DecodePlan("tiny", TP2_ROWS, attn_mc_ar=MC_AR_TP2, moe_mc_ar=MC_AR_TP2, qkv_prequant=MC_AR_TP2), +) + ( + # R1VR8: the 16- and 48-token multicast buckets (9..16: 5.29-6.26 us vs 5.75-7.16 generic; 17..32: the hop pays + # +1.06 us at 32 rows, repaid by the MXFP8 hand-off to the GEMVs). + ( + DecodePlan("small", 16, attn_mc_ar=MC_AR_TP2, moe_mc_ar=MC_AR_TP2, qkv_prequant=MC_AR_TP2), + DecodePlan( + "verify", + DECODE_MAX_TOKENS, + attn_mc_ar=MC_AR_TP2, + moe_mc_ar=MC_AR_TP2, + qkv_prequant=MC_AR_TP2, + ), + ) + if R1VR8 + else () +) +if DECODE_STEP_MAX > DECODE_MAX_TOKENS: + # VR-BANDS: 33..64-row steps (K7 c5..c8) with the built-in all-reduce at both boundaries. + DECODE_PLANS += ( + DecodePlan("wide", DECODE_STEP_MAX, attn_mc_ar=False, moe_mc_ar=False, qkv_prequant=False), + ) + +# The dense layers' decode attention (trtllm-gen, msa_decode.dense_decode_max_seq_len): the max_seq_len it is handed +# per request count. It only sizes the multi-CTA KV split (every CTA still covers its share of the actual length; the +# result changes only by the split-merge order), and the model maximum over-splits AgentX-length caches at one +# request: GB300 TP4, 16 heads, FP8 KV, 64K context: 15.1 us at 16K vs 28.6 us at 1M. Two requests are neutral (the +# grid is unchanged) and keep the step's bound. + + +# TP2 skeleton: no cap (the built-in split) until a TP2 cap is measured and certified. +DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH: Mapping[int, int] = MappingProxyType({}) + +# The dense layers' decode attention on PrimTS split-KV decode (kernels/m3_fullctx_decode) instead of the trtllm-gen +# call, per (query rows per request, requests): the Q tile (``config``) and the split-KV fanout (``splits``) with the +# BF16 merge. At TP8 (8 query heads, 1 KV head per rank) the stock split is capped at 16 cluster CTAs per (request, KV +# head), i.e. 16 of 212 CTAs at one request. Rubin TP8 geometry, FP8 KV, chained us per call at 165k / 323k, every +# recipe passing the fp32 oracle (microbenches, locked clock, cold L2): (4, 1) 12.65 / +# 15.88 at tile 16 / 96 splits / 12 K/V stages; (4, 2) 15.85 / 21.32 at tile 32 / 96 splits. At one request, 108 or +# more splits at tile 16 fall off a cliff at 165k, and fewer K/V stages are slower at both lengths. The drafter's +# one-row passes (the 8-row Q tile has no separate-reduction profile with FP16 partials, so tile 16): (1, 1) 10.75 us +# at 108 splits / 12 K/V stages; (1, 2) 13.35 at 64 splits / 12 K/V stages. +FULLCTX_DECODE_RECIPES: Mapping[Tuple[int, int], Mapping[str, Any]] = MappingProxyType( + { + (4, 1): MappingProxyType( + { + "config": {"tile_size_q": 16, "groups_tokens_heads_q": True, "kv_stages": 12}, + "splits": 96, + } + ), + (4, 2): MappingProxyType( + {"config": {"tile_size_q": 32, "groups_tokens_heads_q": True}, "splits": 96} + ), + (1, 1): MappingProxyType({"config": {"tile_size_q": 16, "kv_stages": 12}, "splits": 108}), + (1, 2): MappingProxyType({"config": {"tile_size_q": 16, "kv_stages": 12}, "splits": 64}), + } +) +# False puts the dense layers back on the trtllm-gen call at every shape. +FULLCTX_DECODE_ENABLED = True +# The dense layers' FP8 q/k/v (decode steps of the hybrid cache): the TP2 op rounds Q/K to BF16 and then to E4M3 in +# its store, the bits of the BF16 fused_qk_norm_rope output cast to FP8. False runs that BF16 kernel and a separate cast +# (the same bits, one more kernel). +DENSE_QKV_FP8_VIA_BF16 = True +# On a full-context decode step that op also stores the new K/V rows into the layer's paged pool (the bytes +# write_msa_phase_kv's indexed copies store, one launch instead of about ten). False writes them with +# write_msa_phase_kv. +DENSE_KV_INSERT_FUSED = True +# The engagement line of that fused store (dispatch.taken, one per row count and rank). +DENSE_KV_INSERT_PATH = "m3_dense_kv_insert" +# The drafter MLP's C++ SwiGLU (DRAFT_SILU_CPP, spec_worker._m3_draft_mlp): its own taken / not-taken lines. +DRAFT_SILU_PATH = "m3_draft_silu" +# The drafter boundary all-reduce fed by its projection's push (DRAFT_AR_PUSH): its own taken / not-taken lines. +DRAFT_AR_PUSH_PATH = "m3_draft_ar_push" +# The split-KV merge on the BF16 op (minimax_m3_sm107_tp2_fullctx_reduce) instead of PrimTS's FP16 reducer and a cast: +# Rubin, 165k, graph-us per call (1, 1) 14.5 vs 18.5, (4, 1) 17.3 vs 21.3, (1, 2) 18.5 vs 23.1, (4, 2) 23.3 vs 27.5. +FULLCTX_DECODE_OWN_REDUCER = True +# The TP8 merge also writes the MXFP8 copy of its BF16 rows (E4M3 + 128x4-swizzled UE8M0, bit for bit +# trtllm::mxfp8_quantize) into the MERGE_MXFP8 buffers and publishes it to m3_prequant.SLOT, so the dense layer's +# MXFP8 o_proj skips its own quantization. Kill: False leaves the o_proj to quantize its BF16 input. +FULLCTX_DECODE_MXFP8 = True +FULLCTX_DECODE_MXFP8_PATH = "m3_fullctx_merge_mxfp8" +# The K/V length bound the full-context plans are compiled for; a step whose cache length bound is longer is a miss. It +# covers the TP2 AgentX config's max_seq_len (1 << 20) plus the speculative rows the cache reserves past it. +FULLCTX_DECODE_MAX_KV_LEN = 1 << 21 # capacity: fixed: the full-context plans' compiled K/V bound +# The Eagle3 drafter layer's generation phase on the same full-context decode (kernels/m3_drafter_attn: the in-tree +# RoPE + FP8 K/V append, then the pinned PrimTS recipe) instead of thop.attention's in-tree runner, which caps its +# split at 8 CTAs per (request, KV head) on SM 10.7: 45-46 us per draft pass at 165k. +DRAFTER_FULLCTX_ENABLED = True +# The standalone DSpark drafter's attention on FlashInfer's trtllm-gen library (kernels/m3_drafter_attn +# M3DrafterSlidingFiFmha): its SM100f Q8 SwapsAb decode kernel instead of thop.attention's in-tree SM107a Q64 KeepsAb +# one (4.0 vs 9.7 us per call, 5 calls per step at c1). A different kernel, so not bitwise the in-tree output. +# Kill switch: TRTLLM_M31_DISABLE_VR_DRAFT_FMHA_SM100F=1 (the in-tree runner), read at import. +VR_DRAFT_FMHA_SM100F = not m31_disabled("VR_DRAFT_FMHA_SM100F") +# On those passes the drafter's RoPE, FP8 query and FP8 K/V append run as one launch (the op +# minimax_m3_sm107tp8_drafter_rope_fp8_kv: the in-tree preprocess's GPT-NeoX rotation and unit-scale E4M3 store, and +# the torch E4M3 cast of its BF16 query) instead of thop.trtllm_gen_generation_preprocess and the cast. Kill constant. +DRAFTER_ROPE_FUSED = True +# The engagement line of that launch (dispatch.taken, one per row count and rank). +DRAFTER_ROPE_PATH = "m3_drafter_rope" +# thop.trtllm_gen_generation_preprocess builds its workspace token map only for several requests, yet a one-request +# multi-token generation (the drafter's first pass at one request, any stock trtllm-gen fallback) places its rows by +# that map: without a rebuild it applies the last multi-request call's map (rows on another request's length and pages, +# token-0 drafts). kernels/m3_tokmap_guard writes the one-request map (0, t) into the op's map slice right before such +# calls: what the build writes (padded rows included), so bitwise the op with a fresh map. False:. +PREPROCESS_TOKMAP_PATH = "m3_preprocess_tokmap" +# How the rotation's products are summed, as the in-tree preprocess build sums them (0: two products then the add, +# 1: fma(cos, x, sin * pair), 2: fma(sin, pair, cos * x)); the op's bitwise test pins it. +DRAFTER_ROPE_MODE = 1 + +# The sparse layers' NVFP4 QKV + indexer producer (ops/attention/m3_producer, every step its geometry fits) triggers its +# dependent kernel on entry, so the index scorer launches and streams its pre-wait index-K prefetch while the producer +# runs (its outputs are read only after the scorer's own grid wait). Rubin TP2, 57-layer PDL chain behind a 21 MB +# pre-wait L2 prefetch (the 165K scorer's shape): 3.59 (native) -> 1.93 us per layer at 4 rows, 3.64 -> 2.00 at 8. +PRODUCER_EARLY_TRIGGER = True +# The producer also reads its per-step inputs (position ids, cache slots, RoPE rows) before its grid wait, leaving only +# the packed QKV row after it. Their writers are host-to-device copies before the replay and plain-launched kernels +# earlier in the step (the input preprocessing, msa_kv_update or its torch fallback), so they are final before the +# producer can start; a PDL-launched writer of any of them would have to turn this off. The call site's +# position_ids.reshape(-1).contiguous().to(torch.int32) and packed.contiguous() launch a plain torch copy right before +# the producer when the dtype or layout differs: also a plain writer, so the contract holds. Bit-identical outputs. +# TP8: True runs this target's copy of the producer (trtllm::minimax_m3_sm107_tp8_producer_pw, +# minimaxM3Sm107Tp8/producerPw8.cu, the TP2 producerPw2 at 8 query heads) with the pre-wait reads; False (the kill +# value) runs the shared minimax_m3_producer_pw at 8 query heads, which reads them after its wait. +# Step-input writers added since: the step-input prologue's C++ op (STEP_PACK_CPP_PROLOGUE) and the C++ spec fold tail +# (STEP_FOLD_TAIL_CPP) are plain launches like the Triton kernels they replace (their tests assert it on the captured +# graph); STEP_KV_LENS_UPDATE_ONCE only drops an earlier writer whose outputs the surviving msa_kv_update rewrites. +# The spec step correction (STEP_SPEC_OFFSETS_ONE_LAUNCH: position_ids, kv_lens) is a plain launch too (its test +# asserts it). +PRODUCER_PREWAIT_STEP_INPUTS = True +# #117: on a decode step of at most 8 rows the sparse layer's QKV + index GEMV runs the producer in its owner epilogue +# (kernels/mx_gemv m3_mx_gemv_producer, m3_layer.phase_producer.producer_head): one launch and one PDL edge fewer per +# sparse layer, bit for bit the GEMV + m3_producer chain (q, index-q and every cache byte). The QKV linear defers its +# GEMV (``_linear``'s ``defer``: a ``_DeferredGemv``) and the producer call site runs the fused kernel, or the plain +# GEMV when it declines. +# Microbench (TP8 c1, 57 layers, cold L2, both orders): -0.61 .. -0.98 us/layer at 165K / 323K. Kill: False. +QKV_PRODUCER_FOLD = True +QKV_PRODUCER_FOLD_PATH = "m3_qkv_producer_fold" + +# The fused embedding + first RMSNorm kernel: one owner per row count and index dtype. SM107: 1.55..1.65 us against +# 3.4..6.0 us for the stock embedding + RMSNorm at 1..8 rows (the gather bitwise, the norm one ulp off on ~5e-6 of the +# elements: summation order); it also hands the first layer its residual, +# so layer 0 takes the decode layer instead of the generic one. False runs layer 0 on the generic layer. +# On by default: the intermittent illegal memory access first seen with this stack on was the +# built-in Triton sparse decode's (since fixed); with the stack on, GSM8K 4-in-flight ran crash-free. +EMBED_FIRSTNORM_TP2 = True +EMBED_FIRSTNORM_MAX_TOKENS = TP2_ROWS if EMBED_FIRSTNORM_TP2 else 0 +# The o_proj pushes its rows into the attention boundary's multicast all-reduce (prepushed mode 2) up to this many +# tokens; wider steps run the GEMV and a plain multicast all-reduce. SM107 TP2: -0.5..-1.2 us per layer, bitwise equal +# to the unpushed boundary. False keeps the GEMV and the boundary separate. +# On by default: the intermittent illegal memory access first seen with this stack on was the +# built-in Triton sparse decode's (since fixed); with the stack on, GSM8K 4-in-flight ran crash-free. +ATTN_PUSH_TP2 = True +ATTN_PUSH_MAX_TOKENS = TP2_ROWS if (MC_AR_TP2 and ATTN_PUSH_TP2) else 0 +# The dense layers' MLP down projection pushes its rows into the post-MLP boundary's multicast all-reduce (prepushed +# mode 2), as the o_proj does into the attention boundary: the all-reduce no longer waits for the whole down GEMV grid +# before loading and pushing the rows itself. SM107 TP2 (nsys): the two non-capture dense layers' +# post-MLP hops end 5.5-5.8 us after the down GEMV vs 2.4-2.5 us for a pushed hop. Bitwise equal to the unpushed +# boundary (the same bf16 words). False keeps the down GEMV and the boundary separate. +DENSE_PUSH_TP2 = True +DENSE_PUSH_MAX_TOKENS = TP2_ROWS if (MC_AR_TP2 and DENSE_PUSH_TP2) else 0 +# The generic layer's boundaries (prefill, mixed steps, Eagle3 capture layers): the most rows that take the multicast +# all-reduce + residual + RMSNorm, alone and with the shared + routed add folded in: the decode boundaries' certified +# rows, under the same constant (MC_AR_TP2; off = the built-in all-reduce). +GENERIC_MC_AR_MAX_TOKENS = TP2_ROWS if MC_AR_TP2 else 0 +GENERIC_MC_AR_FOLDED_MAX_TOKENS = TP2_ROWS if MC_AR_TP2 else 0 +AR_PUSH_MODE = 2 +# The 1..8-token MoE engine (router, routed experts and the shared expert in one launch, ops/moe/m3_moe_engine at +# the TP2 width 1536). Rubin TP2 ablation chain (57 layers behind a 58.4 MB PDL producer, row = (P + MoE) - P, both +# orders): 31.3 / 50.7 us/layer at 4 / 8 tokens against the built-in MoE's 42.4 / 55.4 (FlashInfer MXFP8 shared +# expert); bit for bit the built-in block's shared.add_(routed). KILL_MOE_ENGINE: the built-in MoE at every row count. +KILL_MOE_ENGINE = False +MOE_ENGINE_MAX_TOKENS = 0 if KILL_MOE_ENGINE else 8 +# The engine's push of its sum into the MoE boundary's multicast all-reduce (prepushed mode 2) on the rows the +# engine serves, with the multicast boundary (DECODE_PLANS' moe_mc_ar): the boundary reads the pushed words instead +# of loading and pushing the rows itself. The forward arms the push only on rows MOE_PUSH covers. +MOE_PUSH_MAX_TOKENS = MOE_ENGINE_MAX_TOKENS if MC_AR_TP2 else 0 +# When the engine pushes its sum into the boundary all-reduce, every CTA releases the PDL dependents right after its +# grid dependency wait instead of after the push, so the all-reduce (which polls the pushed tagged words, not the grid) +# becomes resident on the SMs the engine frees. Scheduling only: the same bits (world-8 push matrix). Off at TP8: Rubin +# TP8 greedy early column, same-job A,B + B,A, flat (period +20..+26 at 155K..323K); True turns it on. +MOE_EARLY_AR_RELEASE = False +# A greedy Eagle3 verify step of up to this many requests verifies with the fused shard argmax + acceptance kernel +# over the target lm_head's BF16 shard logits (kernels/m3_shard_accept), at the configured draft length. +SHARD_ACCEPT_DRAFT_LEN = 3 # capacity: fixed: the built Eagle3 draft length +# 0: (the gathered logits + argmax run). Rubin TP2 greedy verify (microbench): stock 120.4 us at b1; fused +# with the coded head 117.0 us; the BF16-head + shard-accept arm is not measured yet. +SHARD_ACCEPT_MAX_REQUESTS = 2 # capacity: fixed: the TP2 GEN batch +# A sampled (temperature + top-p, advanced_sampling_mode no_topk) Eagle3 verify step of up to this many requests +# verifies with the fused sampled acceptance (kernels/m3_sampled_accept, the TP2-shard kernel) over the target +# lm_head's BF16 shard logits, at the configured draft length (the AgentX contract). 0: (the gathered logits +# and the in-tree sampler run). Rubin TP2 b1/b2 (p5 microbench): 180.1 / 181.8 -> 115.1 / 113.9 us per verify. +SAMPLED_VERIFY_MAX_REQUESTS = 2 # capacity: fixed: the TP8 GEN batch +# A step verified by the in-tree path takes the fused first drafter inputs (the capture norms and the first pass's +# gather) up to this many requests. +FIRST_DRAFTER_INPUTS_MAX_REQUESTS = 0 # capacity: fixed: off at TP2 (0) +# A step verified by the fused kernels (dispatch.SHARD_ACCEPT / SAMPLED_ACCEPT) of up to this many requests takes the +# fused first drafter inputs (spec_worker._m3_first_drafter_inputs: the accepted tokens as they lie, the first pass's +# gather rows, the fc_norm chunk norm or the armed capture norm, then the in-tree fc). 0: (the in-tree first +# drafter inputs run, the captures stay in-tree and the fused draft loop steps aside). +FUSED_FIRST_INPUTS_MAX_REQUESTS = 2 # capacity: fixed: the TP2 GEN batch +# On a fused-verify step of up to this many verify rows the Eagle3 captures write rmsnorm(bf16(h + residual)) * +# fc_norm[i] in one kernel (ops/spec/m3_capture_norm) once the fc_norm weights are stacked; the fused first drafter +# inputs then skip their chunk norm. 0: (the in-tree capture, then the chunk norm). +CAPTURE_NORM_MAX_ROWS = 8 # capacity: fixed: the TP2 verify rows (2 requests x 4) +# The first draft pass's per-layer fc_norm over the captured columns in one kernel (ops/spec/m3_chunk_norm) at up +# to this many verify rows; wider first passes take the in-tree first drafter inputs. 0:. +CHUNK_NORM_MAX_ROWS = 8 # capacity: fixed: the TP2 verify rows (2 requests x 4) +# The fused draft loop's two all-reduce boundaries per pass (o_proj -> post-attention norm, down -> final norm) of up +# to this many rows run the multicast all-reduce + residual + RMSNorm (kernels/m3_mc_allreduce, opts bit 16: the +# drafter's stock rounding, normalizing the unrounded FP32 sum as FlashInfer fused_add_rmsnorm does) instead of the +# built-in all-reduce + RMSNorm. Kill: 0. Rubin TP2 (p16 microbench): 10.0..11.0 -> 4.6..5.3 us per boundary. +DRAFT_MC_AR_MAX_ROWS = 8 # capacity: fixed: the TP8 verify rows +# The drafter's down -> final-norm boundary (the all-reduce right before each pass's draft step) runs without the +# max-smem carveout (opts bit 8, which AR_OPTS_BY_M sets for 3..6 rows): the carveout made the 212-CTA draft step wait +# for the reduce's CTAs (TP8 profile: pass 0 at 4 rows ran its reduce at 233,472 B smem, passes 1-2 at 16,384, and +# pass 0's step +10 us; TP4's draft-step penalty, same mechanism). Bit-identical. Kill: True. +DRAFT_MC_AR_CARVEOUT = False +# The drafter's o_proj and down projection, on the copied split-K kernel (DRAFT_LOWM_OWN_SPLITK), also push their rows +# into the drafter boundary all-reduce's exchange (the canonical words its own push would store, at the call parity), +# which then runs prepushed 2 instead of pushing x itself; the copy releases its dependents only after its own grid +# wait and the pushes (the prepushed >= 2 contract). Bitwise. Rubin TP8 world 8, residuals chained boundary to +# boundary: 8.56 -> 8.08 us per boundary (1 row), 9.21 -> 8.10 (4 rows), ~ -4 us/step. Kill: False. +DRAFT_AR_PUSH = True +# On a fused-verify step of up to this many verify rows the drafter's first fc is forked onto the aux stream right +# after the last captured target layer, overlapping the remaining target layers and the head (spec_worker). +FC_EARLY_MAX_ROWS = 0 # capacity: fixed: measured fork crossover +# A sampled fused verify of up to this many verify rows on the in-tree head GEMM forks the drafter's first fc after the +# head instead of beside it (spec_worker): the fc's split-K grid otherwise holds the SMs first and the head GEMM starts +# late. Kill: 0. Rubin TP2 c1 (locked nsys): the head started 17.4 us after its input was ready. +FC_AFTER_HEAD_MAX_ROWS = 8 # capacity: fixed: measured fork crossover +# The forked fc's completion reaches the first draft input through a counter instead of a stream join: the fork bumps a +# device counter after the fc, a one-thread wait kernel queued after the acceptance spins on it, and the fork is joined +# after the first draft input. That input then has one producer on its own stream (a programmatic, PDL edge) instead of +# the acceptance on another graph stream: in-model the acceptance -> draft input edge cost 2.0-2.4 us (TP2) and +# 3.2-3.8 us (TP8 tp8-full) per step. Ordering only: the same values. Kill: False (join before the draft input). +DI_FC_FLAG = True +DI_FC_FLAG_PATH = "m3_di_fc_flag" +# A decode-only Eagle3 step of up to this many requests drafts with the fused draft loop (spec_worker): per pass the +# draft input in one launch (kernels/m3_spec_tail.draft_input), the drafter layer's in-tree attention, MLP and +# all-reduces, and one kernel for the draft head (its 12-bit coded copy, or the BF16 fragments when the coding does +# not decode bit for bit), the cross-rank argmax and the pass's bookkeeping (kernels/m3_draft_step), at the +# configured draft length. The head kernel takes one row per request, at most 8. +# The fused draft loop streams the 12-bit coded copy of the draft head (when it decodes bit for bit); False keeps the +# BF16 fragments. +DRAFT_STEP_DRAFT_LEN = SHARD_ACCEPT_DRAFT_LEN # capacity: fixed: the built draft length +# 0: (the in-tree draft loop runs). Rubin TP2 per draft pass (head + argmax + exchange): 122.6 / 126.0 -> +# 102.8 / 102.8 us at b1 / b2 (BF16 fragments; the coded head is slower on sm_107: DRAFT_HEAD_CODED False). +DRAFT_STEP_MAX_REQUESTS = 2 # capacity: fixed: the TP8 GEN batch +# H-R2-3: on the fused draft loop's last pass, the step's forward-tail store and next-step staging (the C++ fold tail, +# STEP_FOLD_TAIL_CPP) run in that draft_step's last CTA after its tokens (foldTail.h foldTailWarp, the standalone tail's +# code bit for bit), removing the draft_step -> fold_tail launch edge; M3SpecStepFold.fused_tail_operands then makes +# store_in_forward skip its launch. Off until its in-model pair. Kill: False. +DRAFT_STEP_FOLD_TAIL = False +DRAFT_STEP_FOLD_TAIL_PATH = "m3_draft_step_fold_tail" +# The Eagle3 drafter's BF16 linears (qkv, o_proj, gate_up, down, fc) run FlashInfer's fixed-tactic low-M kernels +# (ops/gemm/m3_draft_lowm) at the rows its TP8 table is certified for, up to this many rows; other calls keep the +# in-tree cuBLAS GEMM. Kill: 0. Rubin TP8 shards (mclk locked) at 1 row: qkv 6.15 -> 3.94, +# o 4.97 -> 2.42, gate_up 8.09 -> 5.78, down 5.62 -> 4.00 us; the replicated fc 21.50 -> 17.77 (TP2). +DRAFT_LOWM_MAX_ROWS = 4 # capacity: fixed: the certified low-M rows +# The split-K ('s') tactics run this target's copy of FlashInfer's kernel (kernels/m3_lowm_splitk): bit for bit the +# same output at every certified cell (spec/test_modeling_v2_m3_lowm_splitk.py); the seam for the drafter-boundary push +# epilogue and the down projection's SwiGLU prologue. Kill: False (FlashInfer's kernel). +DRAFT_LOWM_OWN_SPLITK = True +# On top of DRAFT_SILU_CPP: where the drafter MLP's down projection runs on a split-K tactic of the target's copy +# (DRAFT_LOWM_OWN_SPLITK), its activation is built in the GEMM's B load from gate | up instead of a separate launch +# (kernels/m3_draft_lowm.draft_down_swiglu): bit for bit the activation + GEMM +# (spec/test_modeling_v2_m3_lowm_splitk.py), one launch fewer per draft pass. Kill: False. OFF: bitwise but +# slower (world 8, chained: 10.14 / 10.81 -> 17.38 / 66.85 us per boundary at 1 / 4 rows, +# ~ +70 us/step); +# the down projection keeps its boundary push through draft_linear and the C++ SwiGLU (DRAFT_SILU_CPP) runs. +DRAFT_SILU_FOLD = False +DRAFT_LOWM_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = { + nk: frozenset(m for m in rows if m <= DRAFT_LOWM_MAX_ROWS) + for nk, rows in DRAFT_LOWM_TACTIC_ROWS.items() +} +# The identity-selection sparse decode moves from the per-row producer to the multi-query producer at this many rows +# (kernels/triton_sparse_decode.MQ_MIN_TOKENS); the multi-query producer needs one KV head, which the TP8 heads have +# (8 query heads on 1 KV head per rank), so identity steps of at least 17 rows take it. +SPARSE_DECODE_MQ_MIN_TOKENS = 17 # capacity: fixed: the multi-query crossover +SPARSE_DECODE_KV_HEADS = 1 # capacity: fixed: the tp8 KV heads per rank +# The long-row top-16 selector takes engines of 129..LONG_SELECT_MAX_BLOCKS KV blocks of 128 tokens; it holds 2, 4 or +# 9 keys per thread by the row's valid-block count. At most 128 blocks, or past the maximum, the built-in selector +# runs; 0 turns the path off. Rubin TP2 (2 KV heads), 165,521 tokens, behind the scorer, ablation per layer: +# -0.9 us at 1 request and -1.2 us at 2 requests against the built-in selector. +LONG_SELECT_MIN_BLOCKS = 129 # capacity: fixed: past the built-in selector's 128 +LONG_SELECT_MAX_BLOCKS = 9216 # capacity: fixed: the select kernels' built blocks per row +LONG_SELECT_KEY_WIDTHS = (2, 4, 9) +# The long-row select releases the sparse decode after it right after its own grid wait (the decode stages q and the +# request length before its wait) up to this many query rows; past it the trigger stays at the select's exit. On Rubin +# TP2 the early-launched long-context sparse decode (SPARSE_DECODE_LC) crowds the select's CTAs off the SMs already at +# one request (4 rows): releasing at the exit is 336 us per decode step faster at 1 request and 350 at 2 (in-model, +# graph time, 165K-context mixed draws, locked memory clock), so the early release is off at every row count. +SELECT_EARLY_MAX_ROWS = 0 # capacity: fixed: off at every row count +# With the long-context decode's single merge (m3_attn_lc8.MERGE_ONCE), that op's CTAs also pull the layer's o_proj +# weight and scales into L2 before their grid dependency (cp.async.bulk.prefetch.L2, a hint: no value changes), so the +# o_proj GEMV after it finds them L2-warm (the SM107 TP4 LC's form, sparseDecodeLc4). Kill: False. +SPARSE_DECODE_LC8_PREFETCH_OPROJ = True +# With the long-context decode's single merge (m3_attn_lc8.MERGE_ONCE), that op also records each step's selected +# block ids per layer and pulls into L2 the K/V pages + scales of the blocks the NEXT sparse layer selected at the +# previous step (65-68% of a layer's distinct selected blocks recur one step later at TP4), so that +# layer's sparse decode reads them L2-warm. Hints only; up to N1_MAX_ROWS query rows. Kill: False. +SPARSE_DECODE_LC8_PREFETCH_NEXT_KV = True +N1_MAX_ROWS = 128 # capacity: fixed: the next-KV prefetch's row bound +# Without the early release, the long-row select releases the sparse decode once every score of its row is loaded +# (1, before the exact rank of the candidates) instead of at its exit (0). The release still follows +# the select's own grid wait, which the sparse decode's pre-wait q / length / block-table reads rely on. The decode's +# 16-CTA cluster launch and prologue then overlap the rank: Rubin TP2 in-model, locked memory clock, AgentX draws, +# 2 instances vs the baseline: replay period -162 us fixed, -94 at a 323K-token request. +SELECT_LONG_TRIGGER_PHASE = 1 +# The long-row select runs as kernels/m3_select_long_dsl (ops/attention/m3_select_long_dsl): one 384-thread CTA per row +# instead of one 1024-thread CTA, the same selection bit for bit, with 4, 11 or 24 keys per thread +# (LONG_SELECT_DSL_KEY_WIDTHS) and the same trigger choices. On the Rubin TP8 decode chain (57 layers behind the +# producer stream, 1 / 2 requests) the select is -0.15 / -0.12 us per layer at 165K tokens, 0.00 / -0.01 at 323K and +# -0.04 / -0.04 at 500K against the 1024-thread kernel. Kill: False (the 1024-thread kernel). +SELECT_LONG_DSL = True +LONG_SELECT_DSL_KEY_WIDTHS = (4, 11, 24) +# The MSA index scorer (ops/attention/m3_index_score) of a sparse decode step, up to INDEX_SCORE_MAX_REQUESTS requests +# (0 turns the path off; wider steps run the in-tree CuTe DSL scorer): its CTAs per request by request count, and up +# to INDEX_SCORE_EARLY_MAX_REQUESTS requests the early issue (Q and the first K tile right after the grid wait, one +# older block per CTA prefetched into L2 before it). The split is captured with the graph, so it is one value per +# request count over every context. Rubin TP2 (2 index heads), 165,521 tokens, behind a 58 MB PDL producer, ablation +# per layer: 1 request 3.27 us at 1536 CTAs (512: 3.76, in-tree 256-CTA scorer 6.07); 2 requests 4.97 us at 768 +# (in-tree 6.71). Within 0.2 us of each context's best split from 16K to 400K tokens except 2 requests at 400K (+0.6). +# The early issue reads the request's first page id from the MSA page table before the grid wait: exact because the +# page table is written only on the host side before the replay (attention/test_modeling_v2_m3_block_table_contract.py); +# the request length read there is a hint, every load and store is bounded by the one read after the wait. +INDEX_SCORE_MAX_REQUESTS = 2 # capacity: fixed: the TP8 GEN batch +# At 1 request the split is one wave of CTAs (1272 = 212 SMs x 6 resident at this CTA's shared memory): at the +# 1-index-head geometry (57-layer chain behind the producer, scores and the selection bitwise) -0.03 / -0.27 / +0.06 us +# per layer at 165K / 323K / 500K tokens against 1536. Kill: 1536. +INDEX_SCORE_SPLIT_K: Mapping[int, int] = MappingProxyType({1: 1272, 2: 768}) +INDEX_SCORE_EARLY_MAX_REQUESTS = 2 # capacity: fixed: measured early-issue crossover + +# Synth only (TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS, fractional): the fused verify's forced accepted counts are +# drawn per request, keyed by the request's KV length at the verify (its prompt plus what it has accepted), instead of +# from the server-wide counter, so a request's forced counts do not depend on what the server ran before (A/B draws +# become history-independent). Production sampling never forces counts. False: (the shared counter, the in-tree +# sequence). +FORCE_AL_PER_REQUEST = True + +# The decode MXFP8 GEMVs (kernels/mx_gemv, SM107 TP2 table), per projection (N, K): the rows each serves, on an +# activation its producer quantized (the MoE boundary for the QKV, the attention merge for the o_proj) and on a bf16 +# one (quantized in the GEMV's prologue, bit-identical to mxfp8_quantize). SM107 graph us per projection at 1..8 rows: +# 2.2..8.6 against 4.7..10.5 for the stock MXFP8 quantize + FlashInfer CuTe DSL GEMM (kernels/mx_gemv/op.py). +# False puts every projection back on the stock decode-graph GEMM. +# On by default: the intermittent illegal memory access first seen with this stack on was the +# built-in Triton sparse decode's (since fixed); with the stack on, GSM8K 4-in-flight ran crash-free. +MX_GEMV_TP2 = True +# MX-TAC: the decode-graph MXFP8 projections the GEMVs do not serve take one fixed per-bucket decision instead of the +# AutoTuner's per-start choice (whose L2-hot profiling left c2 bimodal at +-6%): FlashInfer's CuTe DSL quantize and +# CuTe DSL GEMM. Stock's candidates per bucket are quantize {TRT-LLM, CuTe DSL} x GEMM {CUTLASS, CuTe DSL}; on SM107 +# every TP2 projection at 1..16 rows runs fastest on this pair, cold and L2-hot (graph us, cold: 4.7..10.3 against +# 5.0..11.5 with the TRT-LLM quantize and 9.7..58 with CUTLASS), and all four candidates give the same bits (quantize +# bytes and scales, GEMM outputs). A constant is rank-agreed and start-independent by construction. +# False returns these projections to the AutoTuner. +MXFP8_DECODE_QUANT_TACTIC = 0 # MXFP8QuantizeRunner.CUTE_DSL +MXFP8_DECODE_GEMM_TACTIC = 0 # FlashInferMXFP8GemmRunner.CUTE_DSL +_ALL_ROWS = frozenset(range(1, TP2_ROWS + 1)) +_GEMV_PROJECTIONS = ( + QKV_SPARSE, + QKV, + O_PROJ, + SHARED_GATE_UP, + SHARED_DOWN, + DENSE_GATE_UP, + DENSE_DOWN, +) +MX_GEMV_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + {nk: _ALL_ROWS for nk in _GEMV_PROJECTIONS} if MX_GEMV_TP2 else {} +) +# R1VR8: on an MXFP8 activation the GEMV serves the QKV and both gate_up projections up to +# the step bound (32 rows: 5.36 / 4.90 us vs 8.13 / 7.95 for the quantize + FlashInfer GEMM) and the o_proj up to 16 +# rows; at 17..32 rows the o_proj takes the CuTe DSL GEMM on the merge's MXFP8 (2.97 vs 4.06 us) and the down +# projections, whose input is bf16, the pinned quantize + CuTe DSL GEMM. +if MX_GEMV_TP2 and R1VR8: + for _nk in (QKV_SPARSE, QKV, SHARED_GATE_UP, DENSE_GATE_UP): + MX_GEMV_ROWS[_nk] = frozenset(range(1, DECODE_MAX_TOKENS + 1)) + MX_GEMV_ROWS[O_PROJ] = frozenset(range(1, min(16, DECODE_MAX_TOKENS) + 1)) +MX_GEMV_BF16_MAX_TOKENS: Dict[Tuple[int, int], int] = ( + {nk: TP2_ROWS for nk in _GEMV_PROJECTIONS} if MX_GEMV_TP2 else {} +) +# Generation steps of 33..64 rows (K7 c5..c8) run past the decode bound, so their MXFP8 projections took the quantize + +# FlashInfer CuTe DSL GEMM, which steps from ~8 to ~11.8 us/call above 32 rows on the K = 6144 projections +# (SM107). Two existing kernels take those rows instead, on the same quantized activation (rule 1b): +# * VR_GEMV48 (switch TRTLLM_M31_DISABLE_VR_GEMV48): the GEMV (it serves up to 48 rows) on GEMV48_SHAPES at 33..48; +# * LINEAR_CUBLASLT (switch TRTLLM_M31_DISABLE_LINEAR_CUBLASLT, shared with the sm_103 targets): cuBLASLt's MXFP8 GEMM +# (ops/gemm/m3_mxfp8_cublaslt) on its SHAPES at every other row up to 64. +# A bf16 activation is quantized first by the CuTe DSL quantize (the bytes of mxfp8_quantize). SM107 us/step over the +# 240 calls of a K7 step: -583..-592 at 33..48 rows, -494..-497 at 56..64. Off: back to FlashInfer's GEMM. +VR_GEMV48 = not m31_disabled("VR_GEMV48") +LINEAR_CUBLASLT = not m31_disabled("LINEAR_CUBLASLT") +GEMV48_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + { + nk: frozenset(range(max(GEMV48_MIN_TOKENS, DECODE_MAX_TOKENS + 1), mx_gemv.MAX_TOKENS + 1)) + for nk in GEMV48_SHAPES + } + if MX_GEMV_TP2 and VR_GEMV48 + else {} +) +LINEAR_CUBLASLT_ROWS: Dict[Tuple[int, int], FrozenSet[int]] = ( + { + nk: frozenset( + m + for m in range( + max(mxfp8_cublaslt.MIN_TOKENS, DECODE_MAX_TOKENS + 1), mxfp8_cublaslt.MAX_TOKENS + 1 + ) + if m not in GEMV48_ROWS.get(nk, ()) + ) + for nk in mxfp8_cublaslt.SHAPES + } + if LINEAR_CUBLASLT + else {} +) + +Key = Hashable + + +@dataclass(frozen=True) +class RowRange: + """The shapes one fast path takes, and the catalog entry whose cells certify each of them. + + ``keys`` are the covered shapes: row counts for most paths, (query rows per request, requests) for the attention + kernels, a variant name for thresholds. ``certifies`` maps a cell's spec of ``entry`` to the keys that cell + drives (empty for a cell of another variant). + """ + + path: str + keys: FrozenSet[Key] + entry: OpWrapper + certifies: Callable[[Mapping[str, Any]], Iterable[Key]] + + def covers(self, key: Key) -> bool: + return key in self.keys + + def certified(self) -> FrozenSet[Key]: + return frozenset(k for cell in self.entry.CELLS for k in self.certifies(cell.spec)) + + +def _rows(lo: int, hi: int) -> FrozenSet[int]: + return frozenset(range(lo, hi + 1)) + + +def _spec_rows(field: str, **match: Any) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + """A cell's ``spec[field]`` when the cell's other fields match ``match``.""" + + def certifies(spec: Mapping[str, Any]) -> Iterable[Key]: + if any(spec.get(name) != value for name, value in match.items()): + return () + return (spec[field],) + + return certifies + + +def _const(*keys: Key) -> Callable[[Mapping[str, Any]], Iterable[Key]]: + return lambda spec: keys + + +def _sparse_decode_producer(spec: Mapping[str, Any]) -> Iterable[Key]: + if not spec["identity"]: + return () + rows = spec["decode_query_len"] * len(spec["seq_lens"]) + mq = rows >= SPARSE_DECODE_MQ_MIN_TOKENS and SPARSE_DECODE_KV_HEADS == 1 + return ("multi_query" if mq else "per_row",) + + +def index_score_schedule(requests: int) -> Tuple[int, bool]: + """The index scorer's (CTAs per request, early issue) at this many requests (a covered request count).""" + return INDEX_SCORE_SPLIT_K[requests], requests <= INDEX_SCORE_EARLY_MAX_REQUESTS + + +def _index_score_requests(spec: Mapping[str, Any]) -> Iterable[Key]: + """A cell certifies its request count when it drives the TP8 index head (1 per rank) with that count's schedule.""" + requests = len(spec["seq_lens"]) + if spec.get("heads") != 1 or requests not in INDEX_SCORE_SPLIT_K: + return () + if (spec["split_k"], spec["early"]) != index_score_schedule(requests): + return () + return (requests,) + + +def _long_select_widths(spec: Mapping[str, Any]) -> Iterable[Key]: + """The key widths a cell's rows reach: every width up to its blocks' when rows of mixed lengths are drawn.""" + blocks = spec["num_blocks"] + widest = 2 if blocks <= 2048 else 4 if blocks <= 4096 else 9 + if spec["valid"] == "mixed": + return tuple(w for w in LONG_SELECT_KEY_WIDTHS if w <= widest) + return (widest,) + + +def _long_select_dsl_widths(spec: Mapping[str, Any]) -> Iterable[Key]: + """The 384-thread select's key widths a cell's rows reach (every width up to its blocks' for mixed rows).""" + blocks = spec["num_blocks"] + widest = 4 if blocks <= 4 * 384 else 11 if blocks <= 11 * 384 else 24 + if spec["valid"] == "mixed": + return tuple(w for w in LONG_SELECT_DSL_KEY_WIDTHS if w <= widest) + return (widest,) + + +EMBED_FIRSTNORM = RowRange( + "embed_firstnorm", + _rows(1, EMBED_FIRSTNORM_MAX_TOKENS), + m3_embedding_firstnorm, + _spec_rows("rows"), +) +# The identity-selection Triton producer + merge (every request within 16 blocks): the per-row producer, and from +# SPARSE_DECODE_MQ_MIN_TOKENS rows the multi-query producer (one KV head), certified at the TP8 heads on SM107. +SPARSE_DECODE_PRODUCER = RowRange( + _sparse_decode_kernel.PRODUCER_PATH, + _sparse_decode_kernel.PRODUCER_KEYS, + m3_sparse_decode, + _sparse_decode_producer, +) +# The TP8 long-context sparse decode at 8 query heads over one KV head (kernels/m3_attn_lc8): attention, split-K merge +# and the MXFP8 o_proj hand-off in one launch, for every decode step with a selected table; "prewait" up to +# m3_attn_lc8.PREWAIT_MAX_ROWS query rows, "plain" above. Exact-class (in the float64 reference's band; run-to-run +# deterministic). An empty range routes every step back to the Triton producer + merge. +SPARSE_DECODE_LC8 = RowRange( + _sparse_decode_kernel.LC8_PATH, + _sparse_decode_kernel.LC8_KEYS, + m3_sparse_decode_lc8, + lambda spec: ("prewait" if spec["prewait"] else "plain",), +) +# Every token count of a sparse layer on an NVFP4 KV cache at the TP8 heads (prefill and decode): the producer takes the +# geometry, not a row range. Kill switch: frozenset() routes every sparse layer back to the native producer. +PRODUCER = RowRange("m3_producer", frozenset({"nvfp4_tp8"}), m3_producer, _const("nvfp4_tp8")) +INDEX_SCORE = RowRange( + "m3_index_score", + _rows(1, INDEX_SCORE_MAX_REQUESTS), + m3_index_score, + _index_score_requests, +) +LONG_SELECT = RowRange( + "long_select", + frozenset(LONG_SELECT_KEY_WIDTHS) if LONG_SELECT_MAX_BLOCKS else frozenset(), + m3_select_long, + _long_select_widths, +) +LONG_SELECT_DSL = RowRange( + "long_select_dsl", + frozenset(LONG_SELECT_DSL_KEY_WIDTHS) + if LONG_SELECT_MAX_BLOCKS and SELECT_LONG_DSL + else frozenset(), + m3_select_long_dsl, + _long_select_dsl_widths, +) +MC_ALLREDUCE = RowRange( + "mc_allreduce_norm", + _rows(1, DECODE_MAX_TOKENS) if MC_AR_TP2 else frozenset(), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=0), +) +ATTN_PUSH = RowRange( + "attn_push", + _rows(1, ATTN_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +DENSE_PUSH = RowRange( + "dense_push", + _rows(1, DENSE_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +# The router of a pure generation step of 9..16 rows on the generic layer (above the MoE engine's 8; the TP2 GEN batch 4 +# with 3 drafts): the TF32 tensor-core router (minimax_m3_router_tc, 16-CTA clusters, one reduction order), the +# arithmetic the built-in gate takes on SM107 at these rows (2.9-3.5 us/call against ~9.8 us for the cuBLAS TF32 +# split-K router; closer to the fp32 product: max |dlogit| 1.7e-6 vs 6.7e-5). Devices that cannot co-schedule 16-CTA +# clusters keep the fp32 router. Context steps never take it. +# ROUTER48 (switch TRTLLM_M31_DISABLE_ROUTER48): up to the op's certified 48 rows (the K7 c3 / c4 steps of 24 / 32 rows, +# and 33..48 rows where a step runs at them). SM107 microbench (56 rotated router weights): 4.27 / 4.53 / +# 5.00 / 5.38 us/call at 24 / 32 / 40 / 48 rows against 6.34 / 6.37 / 6.46 / 6.48 for the TF32 split-K cuBLAS router the +# fp32 path runs under TORCH_ALLOW_TF32_CUBLAS_OVERRIDE, i.e. -116 / -103 / -82 / -61 us per step over 56 MoE layers. +ROUTER48 = not m31_disabled("ROUTER48") +ROUTER_TC_GEN_MIN_ROWS = 9 # capacity: fixed: past the MoE engine's 8 rows +ROUTER_TC_GEN_MAX_ROWS = 48 if ROUTER48 else 16 # capacity: fixed: the op's certified rows +ROUTER_TC_GEN = RowRange( + "m3_router_tc", + _rows(ROUTER_TC_GEN_MIN_ROWS, ROUTER_TC_GEN_MAX_ROWS), + m3_router_tc, + _spec_rows("m"), # every cell checks the logits, with or without the NVFP4 activation +) +MOE_PUSH = RowRange( + "moe_push", + _rows(1, MOE_PUSH_MAX_TOKENS), + m3_mc_allreduce_norm, + _spec_rows("m", prepushed=AR_PUSH_MODE), +) +MOE_ENGINE = RowRange( + "m3_moe_engine", _rows(1, MOE_ENGINE_MAX_TOKENS), m3_moe_engine, _spec_rows("tokens") +) +# The GEMM paths are keyed (projection (N, K), rows): each projection runs its own split-K / ring / MMA +# configuration, so a row count is certified per projection. +MX_GEMV = RowRange( + "m3_mx_gemv", + frozenset((nk, m) for nk, rows in MX_GEMV_ROWS.items() for m in rows), + m3_mx_gemv, + lambda spec: ((spec["nk"], spec["m"]),), +) +MX_GEMV_BF16 = RowRange( + "m3_mx_gemv_bf16", + frozenset((nk, m) for nk, hi in MX_GEMV_BF16_MAX_TOKENS.items() for m in _rows(1, hi)), + m3_mx_gemv_bf16, + lambda spec: ((spec["nk"], spec["m"]),), +) +MX_GEMV48 = RowRange( + "m3_mx_gemv48", + frozenset((nk, m) for nk, rows in GEMV48_ROWS.items() for m in rows), + m3_mx_gemv, + lambda spec: ((spec["nk"], spec["m"]),), +) +LINEAR_CUBLASLT_GEMM = RowRange( + "m3_mxfp8_cublaslt", + frozenset((nk, m) for nk, rows in LINEAR_CUBLASLT_ROWS.items() for m in rows), + m3_mxfp8_cublaslt, + lambda spec: (((spec["n"], spec["k"]), spec["m"]),), +) +# (draft length, requests) of a greedy verify step. +SHARD_ACCEPT = RowRange( + "m3_shard_accept", + frozenset((SHARD_ACCEPT_DRAFT_LEN, n) for n in range(1, SHARD_ACCEPT_MAX_REQUESTS + 1)), + m3_shard_accept, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) + +# Rows of a fused-draft-loop all-reduce boundary (drafter hidden 6144) on the multicast all-reduce. +DRAFT_MC_AR = RowRange( + "m3_draft_mc_allreduce_norm", + _rows(1, DRAFT_MC_AR_MAX_ROWS), + m3_mc_allreduce_norm, + lambda spec: (spec["m"],) if spec.get("drafter") else (), +) + +# (draft length, requests) of a sampled verify step. +SAMPLED_ACCEPT = RowRange( + "m3_sampled_accept", + frozenset((SHARD_ACCEPT_DRAFT_LEN, n) for n in range(1, SAMPLED_VERIFY_MAX_REQUESTS + 1)), + m3_sampled_accept, + lambda spec: ((SHARD_ACCEPT_DRAFT_LEN, spec["nreq"]),), +) + + +# (query rows per request, requests) of a decode step whose dense-layer attention runs on PrimTS split-KV decode. +FULLCTX_DECODE = RowRange( + "m3_fullctx_decode", + frozenset(FULLCTX_DECODE_RECIPES) if FULLCTX_DECODE_ENABLED else frozenset(), + m3_fullctx_decode, + lambda spec: ((spec["qlen"], len(spec["kv_lens"])),), +) + +# (query rows per request, requests) of a generation-only drafter pass that runs on the full-context decode. +DRAFTER_FULLCTX = RowRange( + "m3_drafter_fullctx", + frozenset(FULLCTX_DECODE_RECIPES) if DRAFTER_FULLCTX_ENABLED else frozenset(), + m3_fullctx_decode, + lambda spec: ((spec["qlen"], len(spec["kv_lens"])),), +) + + +def _verify_rows(max_rows: int) -> FrozenSet[int]: + """The verify rows of 1..8 requests at the shard-accept draft length, up to ``max_rows``.""" + return frozenset( + n * (SHARD_ACCEPT_DRAFT_LEN + 1) + for n in range(1, 9) + if n * (SHARD_ACCEPT_DRAFT_LEN + 1) <= max_rows + ) + + +# Requests of a fused-verify step whose first drafter inputs are the fused ones; certified by the chunk-norm cells +# at those requests' verify rows (the norm is the only kernel of its own; the gather rows and the fc are in-tree). +FUSED_FIRST_INPUTS = RowRange( + "m3_fused_first_inputs", + _rows(1, FUSED_FIRST_INPUTS_MAX_REQUESTS), + m3_chunk_norm, + lambda spec: ( + (spec["m"] // (SHARD_ACCEPT_DRAFT_LEN + 1),) + if spec["m"] % (SHARD_ACCEPT_DRAFT_LEN + 1) == 0 + else () + ), +) +# Verify rows of a fused-verify step whose Eagle3 captures apply the fc_norm. +CAPTURE_NORM = RowRange( + "m3_capture_norm", _verify_rows(CAPTURE_NORM_MAX_ROWS), m3_capture_norm, _spec_rows("m") +) +# Verify rows of a first draft pass whose fc_norm runs as the chunk norm. +CHUNK_NORM = RowRange( + "m3_chunk_norm", _verify_rows(CHUNK_NORM_MAX_ROWS), m3_chunk_norm, _spec_rows("m") +) + +# ((N, K), rows) of a drafter BF16 linear on its low-M kernel. +DRAFT_LOWM = RowRange( + "m3_draft_lowm", + frozenset((nk, m) for nk, rows in DRAFT_LOWM_ROWS.items() for m in rows), + m3_draft_lowm, + lambda spec: ((spec["nk"], spec["m"]),), +) +# Verify rows of a step whose target logits come from the die-placed head (in-tree GEMM halves per die). + +# (draft length, requests) of a fused draft loop, and the rows of its passes' draft input: the first pass's +# requests x (draft length + 1), one per request on the later passes. +DRAFT_STEP = RowRange( + "m3_draft_step", + frozenset((DRAFT_STEP_DRAFT_LEN, n) for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1)), + m3_draft_step, + lambda spec: ((spec["draft_len"], spec["nreq"]),), +) +DRAFT_INPUT = RowRange( + "m3_draft_input", + frozenset( + rows + for n in range(1, DRAFT_STEP_MAX_REQUESTS + 1) + for rows in (n, n * (DRAFT_STEP_DRAFT_LEN + 1)) + ), + m3_draft_input, + _spec_rows("m"), +) + +# The dense layers' down projection takes the gate_up GEMV's output and applies the swigluoai activation in its +# quantize prologue (kernels/mx_gemv x_swiglu, bit for bit the Triton silu_and_mul kernel), instead of a separate +# silu_and_mul launch between the two GEMVs; rows the bf16 GEMV serves. Kill constant: False runs the activation +# kernel as before. +# OFF: bitwise but SLOWER (TP2 same-job pair: +60 / +65 us/step fixed, both orders; every GEMV CTA +# re-activates gate | up in its quantize prologue, redundantly across the N tiles). Same lesson as the drafter fold. +DENSE_SWIGLU_FOLD = False +DENSE_SWIGLU_CPP_PATH = "m3_dense_swiglu" + +FAST_PATHS: Tuple[RowRange, ...] = ( + EMBED_FIRSTNORM, + FULLCTX_DECODE, + DRAFTER_FULLCTX, + SPARSE_DECODE_PRODUCER, + SPARSE_DECODE_LC8, + PRODUCER, + INDEX_SCORE, + LONG_SELECT, + LONG_SELECT_DSL, + MC_ALLREDUCE, + ATTN_PUSH, + DENSE_PUSH, + MOE_PUSH, + MOE_ENGINE, + ROUTER_TC_GEN, + MX_GEMV, + MX_GEMV_BF16, + MX_GEMV48, + LINEAR_CUBLASLT_GEMM, + SHARD_ACCEPT, + SAMPLED_ACCEPT, + DRAFT_MC_AR, + DRAFT_STEP, + DRAFT_INPUT, + DRAFT_LOWM, + FUSED_FIRST_INPUTS, + CAPTURE_NORM, + CHUNK_NORM, +) + +# -------------------------------------------------------------------------------------------------------------------- +# Selection: pure functions of the step shape +# -------------------------------------------------------------------------------------------------------------------- + + +def plan_for(num_tokens: int) -> DecodePlan: + for plan in DECODE_PLANS: + if num_tokens <= plan.max_tokens: + return plan + raise AssertionError(f"no decode plan covers {num_tokens} tokens") + + +def is_decode_step(num_tokens: int) -> bool: + """Whether a pure generation step of ``num_tokens`` rows goes to the decode target.""" + return num_tokens <= DECODE_STEP_MAX + + +def moe_path(num_tokens: int) -> Optional[str]: + """The MoE kernel of a decode step: the engine, or None (generic).""" + if MOE_ENGINE.covers(num_tokens): + return MOE_ENGINE.path + return None + + +MX_GEMV_PATH = MX_GEMV.path +MX_GEMV_BF16_PATH = MX_GEMV_BF16.path +# The dense layers' swigluoai activation as this target's C++ op (ops/gemm/m3_dense_swiglu, +# minimaxM3Sm107Tp8/swigluOai.cu: bit for bit the Triton silu_and_mul at limit 7, alpha 1.702, +# beta 1), PDL-launched with an early release so the down GEMV stages its weights during it; the down GEMV reads the +# activation only after its own grid wait. TP2 chain microbench: -0.69 us per dense layer vs the Triton +# kernel; its oracle passed 23/23. False runs the Triton kernel. +DENSE_SWIGLU_CPP_PATH = "m3_dense_swiglu" +# FlashInfer's CuTe DSL MXFP8 GEMM on the producer's MXFP8 activation (rows the GEMV does not serve). +PREQUANT_GEMM_PATH = "flashinfer_mxfp8_gemm" +# The MXFP8 quantize (tuned backend) and FlashInfer's CuTe DSL GEMM: every row the decode kernels do not serve. +GENERIC_GEMM_PATH = "mxfp8_gemm" +# Past the decode bound: the GEMV (VR_GEMV48) or cuBLASLt's MXFP8 GEMM (LINEAR_CUBLASLT), on either activation. +GEMV48_PATH = MX_GEMV48.path +CUBLASLT_GEMM_PATH = LINEAR_CUBLASLT_GEMM.path + + +def linear_path(n: int, k: int, num_tokens: int, prequant: bool, low_m_gemm: bool) -> str: + """The decode-graph MXFP8 GEMM of an [N, K] projection on ``num_tokens`` rows. + + ``prequant``: the producer published the activation in MXFP8. ``low_m_gemm`` is accepted for the call sites' + symmetry: no low-M GEMM configuration is certified at TP2, so it selects nothing. + """ + if num_tokens in GEMV48_ROWS.get((n, k), ()): + return GEMV48_PATH + if num_tokens in LINEAR_CUBLASLT_ROWS.get((n, k), ()): + return CUBLASLT_GEMM_PATH + if prequant: + if num_tokens in MX_GEMV_ROWS.get((n, k), ()): + return MX_GEMV_PATH + return PREQUANT_GEMM_PATH + if num_tokens <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0): + return MX_GEMV_BF16_PATH + return GENERIC_GEMM_PATH + + +# -------------------------------------------------------------------------------------------------------------------- +# Fallback policy +# -------------------------------------------------------------------------------------------------------------------- + + +class FastPathUnavailable(RuntimeError): + """A covered decode shape could not take its fast path under ``modeling_v2='require'``.""" + + +_lock = threading.Lock() +_require = [False] +_logged: set = set() + + +def set_require(require: bool) -> None: + """Whether a miss raises (``modeling_v2='require'``); set by the target at construction.""" + _require[0] = bool(require) + + +def require_mode() -> bool: + return _require[0] + + +def _log_once(level: Callable[[str], None], path: str, reason: str, message: str) -> None: + key = (path, reason) + with _lock: + if key in _logged: + return + _logged.add(key) + level(message) + + +def missed(path: str, rows: Optional[int], reason: str) -> None: + """``path`` covers this step but cannot run: raise under require, else log once; the caller falls back.""" + at = "" if rows is None else f" at {rows} rows" + if _require[0]: + raise FastPathUnavailable( + f"modeling_v2='require': the MiniMax-M3 decode path {path!r} covers this step{at} but cannot run: " + f"{reason}" + ) + _log_once( + logger.warning, + path, + reason, + f"[M3 dispatch] {path}{at}: {reason}; falling back (logged once per reason)", + ) + + +def capture_missed(path: str, rows: Optional[int], what: str) -> None: + """``what`` is built on eager steps but a CUDA-graph capture found it missing (see ``missed``).""" + missed(path, rows, f"{what} is not built and a CUDA graph is being captured") + + +def taken(path: str, rows: Optional[int], what: str = "") -> None: + """``path`` ran on this rank at ``rows``: one rank-tagged info line per (path, rows, what), the engagement line + every GEN rank prints for each fast path it takes (``what``: an optional detail, e.g. the projection).""" + at = "" if rows is None else f" at {rows} rows" + detail = f" ({what})" if what else "" + _log_once( + logger.info, + path, + f"taken: {rows}" + (f": {what}" if what else ""), + f"[M3 dispatch] rank {mpi_rank()}: {path} taken{at}{detail}", + ) + + +def long_select_covers(num_blocks: int) -> bool: + """Whether ``LONG_SELECT`` takes rows of ``num_blocks`` KV blocks (the engine's per-row block width).""" + return LONG_SELECT_MIN_BLOCKS <= num_blocks <= LONG_SELECT_MAX_BLOCKS + + +def long_select_dsl_keys() -> Tuple[Tuple[bool, bool, int], ...]: + """The (PDL launch, early trigger, trigger phase) keys of the 384-thread select a decode step can launch, compiled + before warmup; none when ``LONG_SELECT_DSL`` covers nothing.""" + if not LONG_SELECT_DSL.keys: + return () + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + pdl = get_env_enable_pdl() + keys = [(pdl, False, SELECT_LONG_TRIGGER_PHASE)] + if SELECT_EARLY_MAX_ROWS: + keys.append((pdl, True, 0)) + return tuple(keys) + + +def long_select_fallback(num_blocks: int, rows: int) -> None: + """The in-tree selector is about to run instead of the long-row select: a miss when ``LONG_SELECT`` covers + ``num_blocks`` (the kernel declined operands its cells certify; raises under require), uncovered otherwise (rows of + at most 128 blocks, or wider than ``LONG_SELECT_MAX_BLOCKS``: the in-tree selector serves them in every mode).""" + if long_select_covers(num_blocks): + missed( + LONG_SELECT.path, + rows, + f"the long-row select declined {num_blocks}-block scores it covers", + ) + else: + span = f"{LONG_SELECT_MIN_BLOCKS}..{LONG_SELECT_MAX_BLOCKS}" + uncovered( + LONG_SELECT.path, f"{num_blocks} KV blocks, outside {span}: the built-in selector" + ) + + +def uncovered(path: str, reason: str) -> None: + """The step is outside ``path``'s covered shapes: log once per (path, reason); the generic path runs.""" + _log_once(logger.info, path, reason, f"[M3 dispatch] {path} not taken: {reason} (logged once)") + + +def reset_logged() -> None: + """Forget which (path, reason) pairs were logged.""" + with _lock: + _logged.clear() + + +# Kernel modules report from inside their calls through kernels/m3_report: the same policy. +m3_report.install( + on_taken=taken, on_missed=missed, on_uncovered=uncovered, on_capture_missed=capture_missed +) + + +# -------------------------------------------------------------------------------------------------------------------- +# Target constants +# -------------------------------------------------------------------------------------------------------------------- + +# The generic boundary takes the low-latency multicast all-reduce + residual + RMSNorm up to this many tokens; with +# the shared + routed add folded in, up to its own maximum: the decode boundaries' certified rows. +MC_AR_MAX_TOKENS = GENERIC_MC_AR_MAX_TOKENS +MC_AR_FOLDED_MAX_TOKENS = GENERIC_MC_AR_FOLDED_MAX_TOKENS +# The shared expert folds into the routed engine from one token (dedicated shared-expert CTAs). +MOE_FOLD_MIN_TOKENS = 1 # capacity: fixed: a row lower bound +# SMs left to the concurrent kernels while a persistent routed engine runs. +MOE_ENGINE_RESERVED_SMS = 24 # capacity: fixed: an SM split, not a size +# Id of the target's final hidden state in Eagle3's ``layers_to_capture``. +FINAL_HIDDEN_LAYER_ID = -1 # capacity: fixed: an Eagle3 layer id, not a size +# The trtllm-gen MoE runner's routing kind for MiniMax-M3 (sigmoid + correction bias, renormalized, scaled) and its +# gated activation (SwiGLU with the per-expert alpha / beta / clamp operands). +ROUTING_MINIMAX2 = int(RoutingMethodType.MiniMax2) +ACT_SWIGLU = int(ActType_TrtllmGen.SwiGlu) +INDEX_SCORE_PATH = INDEX_SCORE.path + +# -------------------------------------------------------------------------------------------------------------------- +# Streaming: also emit a response on decoding iterations 2..FIRST_CONTENT_EMIT_ITERS (RuntimeHooks), so a request's +# first content leaves one step after its first token. 1: the stock schedule. +FIRST_CONTENT_EMIT_ITERS = 2 + +# The runtime-value dump (TRTLLM_MODELING_V2_VALUE_DUMP=1): the metadata of every fast-path and entry decision a step +# takes, recorded on eager steps and captures (replay runs no Python), printed once at exit. +# -------------------------------------------------------------------------------------------------------------------- + +_VALUE_DUMP = os.environ.get("TRTLLM_MODELING_V2_VALUE_DUMP") == "1" +_VALUES: set = set() + + +def _record(*meta: Hashable) -> None: + if _VALUE_DUMP: + _VALUES.add(meta) + + +def _print_values() -> None: + if _VALUES: + rank = os.environ.get("OMPI_COMM_WORLD_RANK", os.environ.get("RANK", "0")) + for meta in sorted(_VALUES, key=repr): + print(f"[modeling_v2 value dump] rank {rank}: {meta}", flush=True) + + +if _VALUE_DUMP: + atexit.register(_print_values) + + +def _is_vl_config(pretrained_config: PretrainedConfig) -> bool: + model_type = getattr(pretrained_config, "model_type", None) + if model_type == "minimax_m3_vl": + return True + if model_type == "minimax_m3": + return False + return any( + "ForConditionalGeneration" in a + for a in (getattr(pretrained_config, "architectures", None) or []) + ) + + +def _text_model_config(model_config: ModelConfig) -> ModelConfig: + """The model config with the language model's ``text_config`` as its pretrained config. + + The checkpoint declares the vision-language wrapper; its ``text_config`` arrives as a plain dict (no typed + sub-config is registered), so it is materialized into a ``PretrainedConfig`` with the wrapper's dtype and + embedding tie carried over. A text config is returned unchanged. + """ + if not _is_vl_config(model_config.pretrained_config): + return model_config + cfg = copy.deepcopy(model_config) + text = getattr(cfg.pretrained_config, "text_config", None) + if text is None: + raise ValueError("MiniMax-M3 VL config has no `text_config`") + if isinstance(text, dict): + out = PretrainedConfig() + for k, v in text.items(): + setattr(out, k, v) + text = out + if ( + getattr(text, "torch_dtype", None) is None + and getattr(cfg.pretrained_config, "torch_dtype", None) is not None + ): + text.torch_dtype = cfg.pretrained_config.torch_dtype + if ( + getattr(text, "tie_word_embeddings", None) is None + and getattr(cfg.pretrained_config, "tie_word_embeddings", None) is not None + ): + text.tie_word_embeddings = cfg.pretrained_config.tie_word_embeddings + return dataclasses.replace(cfg, pretrained_config=text) + + +def _validate_runtime(model_config: ModelConfig) -> None: + """The runtime this target is written for (it is not in the checkpoint): routing declines every other one + (``routing._runtime_mismatch``), so this is the backstop for a construction that did not go through routing. + + The MiniMax-M3 MSA backend (it owns the step metadata and the index-K cache), the fused five-way QKV/index + projection or the separate QKV and index projections, an FP8 indexer and an FP8 or NVFP4 main KV cache. + """ + sparse = model_config.sparse_attention_config + if ( + sparse is None + or getattr(sparse, "algorithm", None) != "minimax_m3" + or sparse.implementation != "msa" + ): + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.algorithm='minimax_m3' with implementation='msa'" + ) + if getattr(sparse, "indexer_kv_dtype", None) != "fp8": + raise ValueError( + "this MiniMax-M3 target requires sparse_attention_config.indexer_kv_dtype='fp8'" + ) + mode = model_config.quant_config.quant_mode + if mode is None or not (mode.has_fp8_kv_cache() or mode.has_fp4_kv_cache()): + raise ValueError("this MiniMax-M3 target requires an FP8 or NVFP4 KV cache") + + +def _sparse_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = cfg.sparse_attention_config["sparse_attention_freq"] + return [i for i, f in enumerate(freq) if int(f) != 0] + + +def _moe_layer_ids(cfg: PretrainedConfig) -> List[int]: + freq = getattr(cfg, "moe_layer_freq", None) + if freq is None: + return list(range(cfg.num_hidden_layers)) + return [i for i, f in enumerate(freq) if int(f) != 0] + + +class _Mxfp8GemmState(nn.Module): + """The MXFP8 GEMM backend state the engine's warmup drives. + + The engine finds MXFP8 projections by a module's ``quant_method`` and, for every one, opts it into native GEMM + tuning and (on decode CUDA graphs) the auto-dispatched FlashInfer GEMM, then marks each tuned. Every projection of + this target dispatches identically, so one method object carries that state for all of them; the forward reads it + at each call (``_mx_mode``) and passes the resolved backend to the GEMM entries. + """ + + def __init__(self) -> None: + super().__init__() + self.quant_method = MXFP8LinearMethod() + + +@dataclass(frozen=True) +class MxMode: + """The MXFP8 GEMM dispatch of one call, resolved from ``_Mxfp8GemmState`` and the engine's contexts.""" + + decode_graph: bool # inside a decode CUDA-graph warmup or capture + autotune: bool # inside the engine's FlashInfer MXFP8 tuning pass + tuned_quantize_gemm: ( + bool # per-bucket tuned quantize + GEMM (decode graphs, tune_decode_graph_backends) + ) + flashinfer: bool # FlashInfer's MXFP8 GEMM on the quantized activation + native_tuning: ( + bool # the native GEMM through the autotuner (warmup, before its tactics are registered) + ) + + +def _mx_mode(state: _Mxfp8GemmState) -> MxMode: + method = state.quant_method + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + autotune = flashinfer_mxfp8_autotune_active() + return MxMode( + decode_graph=decode_graph, + autotune=autotune, + tuned_quantize_gemm=method.tune_decode_graph_backends and decode_graph, + flashinfer=method.backend == "flashinfer" + or ( + method.backend == "auto" + and (autotune or (method._flashinfer_autotuned and decode_graph)) + ), + native_tuning=method.needs_native_autotune, + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# Projections: the MXFP8 GEMM dispatch (``linear_path``) and its hand-offs +# -------------------------------------------------------------------------------------------------------------------- + + +def _mxfp8_linear( + core, x: torch.Tensor, w: torch.Tensor, s: torch.Tensor, mode: MxMode +) -> torch.Tensor: + """One MXFP8 projection on the engine-tuned path: MXFP8 quantize + GEMM, backend per ``mode``.""" + _record( + "m3_mxfp8_linear", + tuple(w.shape), + mode.tuned_quantize_gemm, + mode.flashinfer, + mode.native_tuning, + ) + return m3_mxfp8_linear( + x.contiguous(), + w, + s, + core.mx_alpha, + torch.bfloat16, + mode.tuned_quantize_gemm, + mode.autotune, + mode.flashinfer, + mode.native_tuning, + ) + + +_PINNED_RUNNERS: dict = {} + + +def _pinned_runner(cls, dtype): + runner = _PINNED_RUNNERS.get((cls, dtype)) + if runner is None: + runner = _PINNED_RUNNERS[(cls, dtype)] = cls(dtype) + return runner + + +def _pinned_mxfp8_gemm(act, act_scale, weight, weight_scale, out_dtype): + """The decode-graph MXFP8 GEMM at the target's fixed tactic (``MXFP8_DECODE_GEMM_TACTIC``), no AutoTuner: the + same choice on every rank and every start.""" + _record("pinned_mxfp8_gemm", tuple(weight.shape), act.shape[0], MXFP8_DECODE_GEMM_TACTIC) + runner = _pinned_runner(FlashInferMXFP8GemmRunner, out_dtype) + return runner.forward([act, act_scale, weight, weight_scale], tactic=MXFP8_DECODE_GEMM_TACTIC) + + +def _pinned_mxfp8_quantize(x): + """The decode-graph MXFP8 activation quantize at the target's fixed tactic (``MXFP8_DECODE_QUANT_TACTIC``).""" + _record("pinned_mxfp8_quantize", x.shape[0], MXFP8_DECODE_QUANT_TACTIC) + return _pinned_runner(MXFP8QuantizeRunner, x.dtype).forward( + [x], tactic=MXFP8_DECODE_QUANT_TACTIC + ) + + +_MX_TAC_ENGAGED: set = set() + + +def _note_mx_tac(kind: str, n: int, k: int, rows: int) -> None: + """MX-TAC's engagement line, once per (activation kind, projection, rows).""" + key = (kind, n, k, rows) + if key not in _MX_TAC_ENGAGED: + _MX_TAC_ENGAGED.add(key) + logger.info( + f"[M3 sm107 tp8] MX-TAC engaged: pinned MXFP8 decision (quantize tactic " + f"{MXFP8_DECODE_QUANT_TACTIC}, GEMM tactic {MXFP8_DECODE_GEMM_TACTIC}) " + f"for the [{n}, {k}] projection at {rows} rows, {kind} activation" + ) + + +def _gemv_variants(n: int, k: int, push: Optional[str], world: int): + """Every (rows, bf16 activation, push (world, bucket tokens, mode)) call the decode dispatch routes to an [N, K] + projection's GEMV: an MXFP8 activation on the rows ``MX_GEMV_ROWS`` gives it, a bf16 one up to + ``MX_GEMV_BF16_MAX_TOKENS``; each pushing when the projection's boundary push covers the rows. ``push``: "attn" + (the o_proj into the attention boundary), "dense" (the dense MLP's down projection into the post-MLP boundary) + or "moe" (the shared expert's down projection into the MoE boundary).""" + push_rows = {"attn": ATTN_PUSH.keys, "dense": DENSE_PUSH.keys, "moe": MOE_PUSH.keys}.get( + push, frozenset() + ) + variants = [] + for rows in range(1, DECODE_MAX_TOKENS + 1): + for x_bf16, served in ( + (False, rows in MX_GEMV_ROWS.get((n, k), ())), + (True, rows <= MX_GEMV_BF16_MAX_TOKENS.get((n, k), 0)), + ): + if not served: + continue + variants.append((rows, x_bf16, None)) + if rows in push_rows: + bucket = next(b for b in m3_mc_allreduce.BUCKETS if rows <= b) + variants.append((rows, x_bf16, (world, bucket, AR_PUSH_MODE))) + # VR_GEMV48: the generation steps past the decode bound, on the (quantized) MXFP8 activation, never pushing. + variants.extend((rows, False, None) for rows in sorted(GEMV48_ROWS.get((n, k), ()))) + return variants + + +def _uncompiled_note() -> str: + key = mx_gemv.last_uncompiled + return "" if key is None else f" (its kernel {key} was not compiled before warmup)" + + +class _DeferredGemv: + """#117 (``QKV_PRODUCER_FOLD``): a sparse layer's QKV + index MXFP8 GEMV, recorded instead of launched. + + ``_linear(..., defer=True)`` returns one on the decode GEMV path of the 12-head packed projection at <= 8 rows with + an MXFP8 hand-off. The producer call site either runs the GEMV with the producer in its epilogue + (``_qkv_producer_fold``) or calls ``materialize`` (the plain GEMV) before anything reads the packed row; + ``placeholder`` (unwritten, the packed shape) stands for that row until then and is never read.""" + + def __init__(self, act, act_scale, weight, weight_scale, use_pdl): + self.placeholder = torch.empty( + act.shape[0], weight.shape[0], dtype=torch.bfloat16, device=act.device + ) + self.act, self.act_scale = act, act_scale + self.weight, self.weight_scale = weight, weight_scale + self.use_pdl = use_pdl + # Set once the record ran, fused (the producer GEMV) or plain (``materialize``). + self.done = False + + def materialize(self) -> torch.Tensor: + """Launch the plain GEMV this record stands for; returns the packed row the projection would have returned.""" + self.done = True + rows, (n, k) = self.act.shape[0], self.weight.shape + output = m3_mx_gemv( + self.act, self.act_scale, self.weight, self.weight_scale, True, None, self.use_pdl, None + ) + if output is None: + raise RuntimeError( + f"the deferred [{n}, {k}] GEMV declined at {rows} rows" + _uncompiled_note() + ) + taken(MX_GEMV_PATH, rows, f"[{n}, {k}] mxfp8 in") + return output + + +def _linear( + core, + x: torch.Tensor, + wk: str, + sk: str, + low_m_gemm: bool, + push: Optional[tuple] = None, + defer: bool = False, +) -> Tuple[Any, bool]: + """One MXFP8 projection ``x @ w[wk]^T``; returns (output, whether its GEMV pushed into ``push``). + + Outside decode CUDA graphs: the engine-tuned MXFP8 quantize + GEMM. Inside them the projection's (N, K) and the + step's rows pick one path (``linear_path``): + + 1. an activation its producer already quantized to MXFP8 (``m3_prequant.SLOT``, taken here): the tcgen05 decode + GEMV on the rows it serves, else the decode-graph MXFP8 GEMM on that activation; + 2. a bf16 activation on the rows the GEMV serves quantizing in its prologue; + 3. otherwise the decode-graph MXFP8 quantize + GEMM. + + MX-TAC: on a decode graph whose MXFP8 projections the engine tunes (``tune_decode_graph_backends``), the GEMM + (and, on a bf16 activation, the quantize) the GEMVs do not serve take one fixed decision + (``MXFP8_DECODE_QUANT_TACTIC`` / ``MXFP8_DECODE_GEMM_TACTIC``: FlashInfer's CuTe DSL quantize and CuTe DSL GEMM) + instead of the AutoTuner's per-start choice (whose L2-hot profiling left c2 bimodal at +-6%). On SM107 every TP2 + projection at 1..16 rows runs fastest on this pair, cold and L2-hot (graph us, cold: 4.7..10.3 against 5.0..11.5 + with the TRT-LLM quantize and 9.7..58 with CUTLASS), and all four candidates give the same bits (quantize bytes + and scales, GEMM outputs). A constant is rank-agreed and start-independent by construction. + + A kernel declining rows its path covers is a miss; the chain then continues down the list. ``push`` (an + ``m3_mx_gemv`` ``ar`` tuple) is the boundary all-reduce exchange the GEMV writes its rows into. ``low_m_gemm`` + marks the projections the fused low-M MXFP8 GEMM may take (``LOW_M_GEMM_MAX_TOKENS``: no TP2 configuration is + built, so ``linear_path`` never selects it). ``defer`` (``QKV_PRODUCER_FOLD``, the sparse layers' packed QKV + projection): on path 1's GEMV at <= 8 rows of the 12-head projection, a ``_DeferredGemv`` is returned in place of + the output and nothing is launched. + """ + w, s = core.w[wk], core.w[sk] + x = x.contiguous() + mode = _mx_mode(core.mxfp8) + if not mode.decode_graph: + prequant = _CTXNORMQ_MX.take(x) if CTXNORMQ and not mode.tuned_quantize_gemm else None + if prequant is not None: + return _ctxnormq_mxfp8_gemm(core, prequant, w, s, mode), False + return _mxfp8_linear(core, x, w, s, mode), False + use_pdl = get_env_enable_pdl() + rows = x.shape[0] + n, k = w.shape + prequant = m3_prequant.SLOT.take(x) + if prequant is None and x.dtype != torch.bfloat16: + path = GENERIC_GEMM_PATH + else: + path = linear_path(n, k, rows, prequant is not None, low_m_gemm) + _record("linear", (n, k), rows, path, push is not None) + if path in (GEMV48_PATH, CUBLASLT_GEMM_PATH): + return _linear_wide(path, x, prequant, w, s, rows, use_pdl), False + if prequant is not None: + act, act_scale = prequant + if defer and path == MX_GEMV_PATH and n == mx_gemv.PROD_N and rows <= 8: + return _DeferredGemv(act, act_scale, w, s, use_pdl), False + if path == MX_GEMV_PATH: + output = m3_mx_gemv(act, act_scale, w, s, True, None, use_pdl, push) + if output is not None: + taken(path, rows, f"[{n}, {k}] mxfp8 in") + return output, push is not None and mx_gemv.last_pushed + missed( + path, + rows, + f"the GEMV declined the [{n}, {k}] projection's MXFP8 activation" + + _uncompiled_note(), + ) + if path == PREQUANT_GEMM_PATH: + uncovered(MX_GEMV_PATH, f"the [{n}, {k}] projection at rows the GEMV does not serve") + if mode.tuned_quantize_gemm: + output = _pinned_mxfp8_gemm(act, act_scale, w, s, torch.bfloat16) + _note_mx_tac("mxfp8", n, k, rows) + return output, False + return m3_mxfp8_gemm(act, act_scale, w, s, torch.bfloat16, mode.autotune), False + if path == MX_GEMV_BF16_PATH: + output = m3_mx_gemv_bf16(x, w, s, True, None, use_pdl, push) + if output is not None: + taken(path, rows, f"[{n}, {k}] bf16 in") + return output, push is not None and mx_gemv.last_pushed + missed( + path, rows, f"the GEMV declined the [{n}, {k}] projection's input" + _uncompiled_note() + ) + if path == GENERIC_GEMM_PATH: + uncovered( + MX_GEMV_BF16_PATH, + f"the [{n}, {k}] projection's bf16 input beyond the decode GEMV's rows", + ) + if mode.tuned_quantize_gemm and x.dtype == torch.bfloat16: + act, act_scale = _pinned_mxfp8_quantize(x) + output = _pinned_mxfp8_gemm(act, act_scale, w, s, torch.bfloat16) + _note_mx_tac("bf16", n, k, rows) + return output, False + return _mxfp8_linear(core, x, w, s, mode), False + + +def _linear_wide(path: str, x, prequant, w, s, rows: int, use_pdl: bool) -> torch.Tensor: + """A projection of a generation step past the decode bound (``GEMV48_PATH`` / ``CUBLASLT_GEMM_PATH``): the + producer's MXFP8 activation, or the CuTe DSL quantize of a bf16 one, into the GEMV or cuBLASLt's MXFP8 GEMM. A GEMV + decline is a miss; the pinned decode-graph GEMM then takes the rows on the same activation.""" + n, k = w.shape + activation = "mxfp8 in" if prequant is not None else "bf16 in, CuTe DSL quantize" + act, act_scale = prequant if prequant is not None else mxfp8_cublaslt.mxfp8_quantize(x) + if path == GEMV48_PATH: + output = m3_mx_gemv(act, act_scale, w, s, True, None, use_pdl, None) + if output is not None: + taken(path, rows, f"[{n}, {k}] {activation}") + return output + missed( + path, + rows, + f"the GEMV declined the [{n}, {k}] projection's MXFP8 activation" + _uncompiled_note(), + ) + return _pinned_mxfp8_gemm(act, act_scale, w, s, torch.bfloat16) + taken(path, rows, f"[{n}, {k}] {activation}") + return m3_mxfp8_cublaslt(act, act_scale, w, s) + + +# -------------------------------------------------------------------------------------------------------------------- +# Layer boundaries: all-reduce + residual + RMSNorm +# -------------------------------------------------------------------------------------------------------------------- + + +def _can_mc(x, folded: bool = False) -> bool: + """Whether the generic boundary takes the multicast all-reduce + residual + RMSNorm: the rows + ``MC_AR_MAX_TOKENS`` / ``MC_AR_FOLDED_MAX_TOKENS`` (with the shared + routed add folded in) give it.""" + return ( + isinstance(x, torch.Tensor) + and x.dim() == 2 + and x.shape[0] <= (MC_AR_FOLDED_MAX_TOKENS if folded else MC_AR_MAX_TOKENS) + and x.dtype == torch.bfloat16 + and x.is_contiguous() + ) + + +def _mc_allreduce_norm(core, x, x2, residual, norm_w, *, quant=False, prepushed=0, capture=None): + """The multicast all-reduce + optional second addend + residual + RMSNorm (``m3_mc_allreduce_norm``), or None + without a workspace or for operands the kernel does not take (the same on every rank). With ``quant`` the normed + rows are also published, MXFP8-quantized, to ``m3_prequant.SLOT`` for the next MXFP8 projection; ``capture`` = + (out, fc_norm weight, eps) also writes the layer's Eagle3 capture norm of the updated residual.""" + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + if workspace is None: + return None + _record( + "m3_mc_allreduce_norm", x.shape[0], x2 is not None, quant, prepushed, capture is not None + ) + return m3_mc_allreduce.m3_mc_allreduce_norm( + x, + residual.contiguous(), + norm_w, + core.rms_eps, + workspace, + use_pdl=get_env_enable_pdl(), + x2=x2, + quant=quant, + prepushed=prepushed, + capture=capture, + ) + + +# The CTXNORMQ hand-offs: the boundary's MXFP8 activation for the first prefill MXFP8 projection reading the normed +# rows, and its NVFP4 form for the routed experts. Separate from ``m3_prequant.SLOT`` so only a CTXNORMQ boundary +# feeds the prefill path. +_CTXNORMQ_MX = m3_prequant.PrequantSlot() +_CTXNORMQ_FP4 = m3_prequant.PrequantSlot() +_CTXNORMQ_STATE = {} + + +def _ctxnormq_op(): + """The op for the CTXNORMQ prefill boundary, or None: ``trtllm::minimax_m31_ctx_residual_rmsnorm_mxfp8`` is + built for SM103 only and refuses this device, so the boundary keeps the fused all-reduce. Decided once per + process.""" + if "op" not in _CTXNORMQ_STATE: + _CTXNORMQ_STATE["op"] = None + logger.info( + "[M3 dispatch] CTXNORMQ prefill boundary norm + quantize: not taken: the op is not built for SM107 " + f"(rows >= {CTXNORMQ_MIN_ROWS})" + ) + return _CTXNORMQ_STATE["op"] + + +def _ctxnormq_allreduce_norm(core, x, residual, norm_w, fp4_scale): + """A prefill boundary as the plain all-reduce + the fused residual add, RMSNorm and quantize, or None (switched + off, fewer than ``CTXNORMQ_MIN_ROWS`` rows, op not built). Returns (normed, updated) like the fused all-reduce; + the MXFP8 activation (and with ``fp4_scale`` the NVFP4 one) wait in the CTXNORMQ slots for their consumers.""" + if not CTXNORMQ or x.dim() != 2 or x.shape[0] < CTXNORMQ_MIN_ROWS: + return None + op = _ctxnormq_op() + if op is None: + return None + reduced = m3_allreduce(x, core.allreduce, None) + _record("m31_ctxnormq", x.shape[0], fp4_scale is not None) + normed, xq, xsf, xfp4, xfp4_sf = op( + reduced, residual.contiguous(), norm_w, core.rms_eps, True, fp4_scale, get_env_enable_pdl() + ) + _CTXNORMQ_MX.publish(normed, xq, xsf) + if fp4_scale is not None: + _CTXNORMQ_FP4.publish(normed, xfp4, xfp4_sf) + else: + _CTXNORMQ_FP4.clear() + return normed, reduced + + +def _ctxnormq_mxfp8_gemm(core, prequant, w, s, mode): + """The GEMM ``m3_mxfp8_linear`` runs after its own quantize, on the CTXNORMQ activation (bit-identical operands).""" + act, act_scale = prequant + _record("m31_ctxnormq_gemm", tuple(w.shape), act.shape[0], mode.flashinfer, mode.native_tuning) + if mode.flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8(act, act_scale, w, s, torch.bfloat16) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if mode.native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, w, s, core.mx_alpha, torch.bfloat16) + + +_GQAMIX_STATE: dict = {} + + +def _gqamix_decode(*args, **kwargs) -> None: + """``minimax_m3_sparse_attn_decode`` for MsaDecodeFmha's sparse generation rows, through the target-local copy.""" + if not _GQAMIX_STATE.get("logged"): + _GQAMIX_STATE["logged"] = True + logger.info( + "[M3 dispatch] GQAMIX: mixed-step sparse decode through ops/gqamix_sparse_decode" + ) + _gqamix_kernel.minimax_m3_sparse_attn_decode(*args, **kwargs) + + +def _gqamix_register(backend) -> None: + """Register ``_gqamix_decode`` on the layer backend's MsaDecodeFmha (a library without the hook is left alone).""" + manager = getattr(backend, "_fmha_manager", None) + for fmha in getattr(manager, "fmha_libs", ()): + if getattr(fmha, "sparse_decode_hook", False) is None: + fmha.sparse_decode_hook = _gqamix_decode + + +def _native_allreduce_norm(core, x, residual, norm_w): + """The built-in fused all-reduce + residual + RMSNorm.""" + _record("m3_allreduce", x.shape[0]) + return m3_allreduce( + x, + core.allreduce, + AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual, + norm_weight=norm_w, + eps=core.rms_eps, + trigger_completion_at_end=False, + ), + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# The MoE: router, routed experts (trtllm-gen NVFP4), shared expert +# -------------------------------------------------------------------------------------------------------------------- + + +@dataclass +class _MoeScalars: + """The routed experts' kernel scalars of one layer, derived from the checkpoint's per-tensor scales once the + weights are real (``MiniMaxM3Tp8ModelingV2._derive_moe``).""" + + fc31_input_scale: torch.Tensor + fc2_input_scale: torch.Tensor + fc31_alpha: torch.Tensor + fc2_alpha: torch.Tensor + fc31_scale_c: torch.Tensor + act_alpha: torch.Tensor + act_beta: torch.Tensor + act_clamp: torch.Tensor + + +def _router_logits(core, i: int, h: torch.Tensor) -> torch.Tensor: + """The fp32 router logits of MoE layer ``i`` when the MoE runs unfused (the decode steps' engines route + in-kernel): ``F.linear(float(h), w)``, the built-in gate's arithmetic. No router kernel serves TP2 rows + (``ROUTER_GEMV_MAX_TOKENS`` = ``ROUTER_TC_MAX_TOKENS`` = 0), so the experts quantize their bf16 input + themselves.""" + _record("m3_fp32_linear", h.shape[0] if h.dim() == 2 else 0) + return m3_fp32_linear(h, core.w[f"l{i}_router"]) + + +def _decode_router_logits(core, i: int, h: torch.Tensor) -> torch.Tensor: + """The router logits of a pure generation step on the generic layer: ``ROUTER_TC_GEN``'s rows take the TF32 + tensor-core router when the device co-schedules 16-CTA clusters (decided at prebuild); every other step the fp32 + router (``_router_logits``).""" + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + ROUTER_TC_GEN.covers(rows) + and core.router_tc_cluster16 + and h.dtype == torch.bfloat16 + and h.stride(1) == 1 + and (h.stride(0) * 2) % 16 == 0 + ): + taken(ROUTER_TC_GEN.path, rows) + _record("m3_router_tc", rows) + return m3_router_tc(h, core.w[f"l{i}_router"], None, get_env_enable_pdl())[0] + return _router_logits(core, i, h) + + +def _routed_experts(core, i: int, x, x_sf, router_logits, do_finalize: bool): + """The trtllm-gen NVFP4 MoE runner over this rank's tensor-parallel slice of every expert, routing in-kernel. + + ``x_sf`` None means ``x`` is the bf16 activation, quantized here with the experts' global input scale. Returns + the finalized rows, or the unfinalized ``[gemm2 out, expert scales, expanded->permuted index]``. + """ + moe = core.moe[i] + if x_sf is None: + x, x_sf = fp4_quantize(x, moe.fc31_input_scale, 16, False, False) + rows = x.shape[0] + x_sf = x_sf.view(rows, -1).flatten() + _record("fp4_block_scale_moe_runner", rows, do_finalize) + outs = fp4_block_scale_moe_runner( + router_logits, + core.w[f"l{i}_router_bias"], + x, + x_sf.view(torch.float8_e4m3fn), + core.w[f"l{i}_e_w31"], + core.w[f"l{i}_e_s31"].view(torch.float8_e4m3fn), + None, + moe.act_alpha, + moe.act_beta, + moe.act_clamp, + core.w[f"l{i}_e_w2"], + core.w[f"l{i}_e_s2"].view(torch.float8_e4m3fn), + None, + moe.fc31_scale_c, + moe.fc31_alpha, + moe.fc2_alpha, + core.num_experts, + core.top_k, + None, + None, + core.moe_inter, + 0, + core.num_experts, + core.routed_scaling_factor, + ROUTING_MINIMAX2, + do_finalize, + act_type=ACT_SWIGLU, + topk_weights=None, + topk_ids=None, + output=None, + tune_max_num_tokens=core.max_num_tokens, + use_dp=False, + ) + return outs[0] if do_finalize else outs + + +def _split_chunk(rows: int, num_chunks: int) -> List[int]: + """The MoE scheduler's even split: the first ``rows % num_chunks`` chunks take one more row.""" + base, extra = divmod(rows, num_chunks) + return [base + 1] * extra + [base] * (num_chunks - extra) + + +# -------------------------------------------------------------------------------------------------------------------- +# Attention +# -------------------------------------------------------------------------------------------------------------------- + + +def _nvfp4_layer(manager, i: int) -> bool: + return bool(getattr(manager, "is_nvfp4_layer", lambda _layer_idx: False)(i)) + + +def _producer_covered(core) -> bool: + """Whether ``PRODUCER`` covers this target's sparse layers: an NVFP4 KV cache at the TP8 heads (8 query, one KV + and one index head). A covered layer that cannot take ``m3_producer`` is a miss.""" + return core.main_kv_is_nvfp4 and (core.heads, core.kv_heads, core.index_heads) == (8, 1, 1) + + +def _producer_layout(kv_data, kv_scale) -> bool: + """Whether a sparse layer's NVFP4 pools are the layout ``m3_producer`` is built for: [pages, 2, 1, 128, 64] with + its scale pool [pages, 2, 1, 128, 8].""" + return ( + kv_data is not None + and kv_scale is not None + and kv_data.dim() == 5 + and tuple(kv_data.shape[1:]) == (2, 1, 128, 64) + and kv_data.stride()[2:] == (128 * 64, 64, 1) + and kv_scale.dim() == 5 + and tuple(kv_scale.shape[1:]) == (2, 1, 128, 8) + and kv_scale.stride()[2:] == (128 * 8, 8, 1) + and kv_scale.shape[0] == kv_data.shape[0] + ) + + +def _nvfp4_fused_write_layout(data: torch.Tensor, scales: torch.Tensor, num_heads: int) -> bool: + """Reject invalid HND pools; select the fused producer only for its packed layout.""" + if ( + data.dim() != 5 + or scales.dim() != 5 + or data.shape[0] <= 0 + or data.shape[0] != scales.shape[0] + or tuple(data.shape[1:]) != (2, num_heads, 128, 64) + or tuple(scales.shape[1:]) != (2, num_heads, 128, 8) + or data.dtype not in (torch.int8, torch.uint8) + or scales.dtype != torch.uint8 + ): + raise ValueError("MiniMax-M3 NVFP4 data and scale caches must have matching HND geometry") + for cache in (data, scales): + if ( + cache.stride(4) != 1 + or cache.stride(3) != cache.shape[4] + or cache.stride(2) < 128 * cache.stride(3) + or cache.stride(1) < num_heads * cache.stride(2) + or cache.stride(0) < 2 * cache.stride(1) + ): + raise ValueError( + "MiniMax-M3 NVFP4 cache rows must be packed and heads/planes/pages must not overlap" + ) + if scales.stride(2) != 128 * 8: + raise ValueError("MiniMax-M3 NVFP4 MSA scale cache must have packed head blocks") + return data.stride(2) == 128 * 64 and data.stride(0) % 2 == 0 and data.stride(1) % 2 == 0 + + +#: The step phases ("decode" / "prefill") whose producer engagement has been logged, once per process. +_PRODUCER_ENGAGED: set = set() + + +def _producer_miss(covered: bool, rows: Optional[int], reason: str) -> None: + if covered: + missed(PRODUCER.path, rows, reason) + return None + + +def _sparse_qkv_producer( + core, + i: int, + packed, + position_ids, + kv_cache_manager, + msa_idx_k_cache, + msa_out_cache_loc, + deferred: Optional[_DeferredGemv] = None, +): + """The horizontal producer of a sparse layer: Q / index-Q (FP8) out, K / V / index-K written into the paged + caches (NVFP4 or FP8 main cache), for every step type (token-major). An NVFP4 cache at the TP8 heads takes + ``m3_producer`` (``PRODUCER``); a covered layer that cannot is a miss. Returns (q, index_q), or None for caches + neither kernel takes. + + ``kv_cache_manager``, ``msa_idx_k_cache`` (the per-layer index-K cache getter) and ``msa_out_cache_loc`` are the + calling target's ``step_args`` entries. ``deferred``: the QKV GEMV ``_linear`` recorded instead of launching + (``QKV_PRODUCER_FOLD``; ``packed`` is then its unwritten placeholder). The ``m3_producer`` branch runs it with the + producer in its epilogue (``_qkv_producer_fold``); any other branch launches it first (``materialize``). A None + return leaves it unlaunched.""" + covered = _producer_covered(core) + rows = int(packed.shape[0]) + if packed.dtype != torch.bfloat16 or position_ids is None: + return _producer_miss( + covered, + rows, + "the packed QKV dtype, position ids, head dims, norm or RoPE configuration", + ) + manager = kv_cache_manager + if manager is None: + return _producer_miss(covered, rows, "the step has no KV cache manager") + buffers = manager.get_buffers(i, kv_layout="HND") + index_k_cache = msa_idx_k_cache(i) + out_cache_loc = msa_out_cache_loc + layer_uses_nvfp4 = _nvfp4_layer(manager, i) + scale_buffers = ( + manager.get_block_scale_buffers(i, kv_layout="HND") if layer_uses_nvfp4 else None + ) + kv_quant_scale = core.inv_kv_scales[i] if layer_uses_nvfp4 else None + if layer_uses_nvfp4: + supported_main = ( + buffers is not None + and scale_buffers is not None + and _nvfp4_fused_write_layout(buffers, scale_buffers, core.kv_heads) + and buffers.is_cuda + and scale_buffers.is_cuda + and scale_buffers.device == buffers.device + and kv_quant_scale.is_cuda + and kv_quant_scale.dtype == torch.float32 + and kv_quant_scale.numel() >= 3 + and kv_quant_scale.is_contiguous() + ) + else: + supported_main = ( + buffers is not None + and buffers.is_cuda + and buffers.dtype == torch.float8_e4m3fn + and buffers.dim() == 5 + and tuple(buffers.shape[1:]) == (2, core.kv_heads, 128, 128) + and buffers.stride(4) == 1 + and buffers.stride(3) == 128 + and buffers.stride(2) == 128 * 128 + and buffers.stride(1) >= core.kv_heads * buffers.stride(2) + and buffers.stride(0) >= 2 * buffers.stride(1) + and buffers.stride(0) % 4 == 0 + and buffers.stride(1) % 4 == 0 + ) + supported_index = ( + index_k_cache.is_cuda + and index_k_cache.dtype == torch.float8_e4m3fn + and index_k_cache.dim() == 4 + and tuple(index_k_cache.shape[1:]) == (1, 128, 128) + and index_k_cache.stride(3) == 1 + and index_k_cache.stride(2) == 128 + and index_k_cache.stride(1) >= 128 * 128 + and index_k_cache.stride(0) >= index_k_cache.stride(1) + and index_k_cache.stride(0) % 4 == 0 + and index_k_cache.stride(1) % 4 == 0 + and buffers is not None + and index_k_cache.shape[0] == buffers.shape[0] + ) + cos_sin = core.rotary_cos_sin + supported_rope = ( + cos_sin.is_cuda + and cos_sin.dtype == torch.float32 + and cos_sin.is_contiguous() + and cos_sin.dim() == 3 + and tuple(cos_sin.shape[1:]) == (2, 32) + ) + if ( + not supported_main + or not supported_index + or not supported_rope + or out_cache_loc is None + or not out_cache_loc.is_cuda + or out_cache_loc.dtype != torch.int32 + or not out_cache_loc.is_contiguous() + or out_cache_loc.numel() < rows + ): + return _producer_miss( + covered and layer_uses_nvfp4, + rows, + "the layer's NVFP4 / index caches, RoPE table, KV quant scale or cache slots are not the layout the " + "producer is built for", + ) + w = core.w + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + if ( + layer_uses_nvfp4 + and (core.heads, core.kv_heads, core.index_heads) == (8, 1, 1) + and _producer_layout(buffers, scale_buffers) + ): + if deferred is not None: + folded = _qkv_producer_fold( + core, + i, + deferred, + buffers, + scale_buffers, + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + cos_sin, + pos, + ) + if folded is not None: + return folded + packed = deferred.materialize() + _record("m3_producer", rows) + q, index_q = m3_producer( + packed.contiguous(), + buffers.view(torch.uint8), + scale_buffers.view(torch.uint8), + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + core.heads, + get_env_enable_pdl(), + PRODUCER_EARLY_TRIGGER, + PRODUCER_PREWAIT_STEP_INPUTS, + ) + phase = "decode" if rows <= DECODE_STEP_MAX else "prefill" + if phase not in _PRODUCER_ENGAGED: + _PRODUCER_ENGAGED.add(phase) + logger.info( + f"[M3 sm107 tp8] fast path engaged: m3_producer ({phase}, T={rows}, " + f"early_trigger={PRODUCER_EARLY_TRIGGER}, prewait_step_inputs={PRODUCER_PREWAIT_STEP_INPUTS})" + ) + else: + if layer_uses_nvfp4 and covered: + missed( + PRODUCER.path, + rows, + f"the NVFP4 pool {tuple(buffers.shape)} / scale pool {tuple(scale_buffers.shape)} is not the " + "producer's layout", + ) + if deferred is not None: + packed = deferred.materialize() + _record("m3_qkv_insert", rows, layer_uses_nvfp4) + q, index_q = m3_qkv_insert( + packed.contiguous(), + buffers.view(torch.uint8) if layer_uses_nvfp4 else buffers, + scale_buffers, + index_k_cache, + out_cache_loc[:rows], + kv_quant_scale, + core.heads, + core.kv_heads, + core.index_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + ) + return q.flatten(1), index_q.flatten(1) + + +def _qkv_producer_fold( + core, + i: int, + deferred: _DeferredGemv, + buffers, + scale_buffers, + index_k_cache, + out_cache_loc, + kv_quant_scale, + cos_sin, + pos, +) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """#117: the deferred QKV GEMV with layer ``i``'s producer in its epilogue (``mx_gemv.m3_mx_gemv_producer``), the + same outputs and cache bytes as the GEMV + ``m3_producer``; None (nothing launched) when the kernel declines.""" + rows = int(deferred.act.shape[0]) + device = deferred.act.device + q = torch.empty(rows, core.heads, core.head_dim, dtype=torch.float8_e4m3fn, device=device) + index_q = torch.empty(rows, 1, core.head_dim, dtype=torch.float8_e4m3fn, device=device) + w = core.w + _record("m3_qkv_producer_fold", rows) + ok = mx_gemv.m3_mx_gemv_producer( + deferred.act, + deferred.act_scale, + deferred.weight, + deferred.weight_scale, + q, + index_q, + buffers.view(torch.uint8), + scale_buffers.view(torch.uint8), + index_k_cache, + out_cache_loc, + kv_quant_scale, + core.rms_eps, + w[f"l{i}_q_norm"], + w[f"l{i}_k_norm"], + w[f"l{i}_idx_q_norm"], + w[f"l{i}_idx_k_norm"], + cos_sin, + pos, + use_pdl=deferred.use_pdl, + ) + if not ok: + missed( + QKV_PRODUCER_FOLD_PATH, + rows, + "the producer GEMV declined the step" + _uncompiled_note(), + ) + return None + deferred.done = True + taken(QKV_PRODUCER_FOLD_PATH, rows, "[1536, 6144] mxfp8 in + producer") + return q.flatten(1), index_q.flatten(1) + + +def _qk_norm_rope(core, i: int, qkv: torch.Tensor, position_ids, out_fp8: bool) -> torch.Tensor: + """The per-head Gemma RMSNorm of Q and K and the partial NeoX RoPE of a dense layer's packed [Q | K | V], in one + kernel: in place, or an FP8 copy (Q / K normed and roped, V cast) when ``out_fp8``.""" + qkv = qkv.contiguous() + pos = position_ids.reshape(-1).contiguous().to(torch.int32) + args = ( + qkv, + core.heads, + core.kv_heads, + core.kv_heads, + core.head_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_q_norm"], + core.w[f"l{i}_k_norm"], + core.rope_theta, + True, + pos, + 1.0, # factor: no YARN + 0.0, # low + 0.0, # high + 1.0, # attention_factor + True, # is_qk_norm + True, # use_gemma + False, # use_mrope + 0, + 0, + ) + _record("qk_norm_rope", qkv.shape[0], out_fp8) + if out_fp8: + return m3_qk_norm_rope_fp8(*args) + return fused_qk_norm_rope(*args) + + +def _dense_fp8_op_applies(qkv: torch.Tensor, position_ids) -> bool: + """Whether a dense layer's FP8 q/k/v come from this target's op (``minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8``, + else the BF16 fused_qk_norm_rope output cast to FP8).""" + return ( + DENSE_QKV_FP8_VIA_BF16 + and hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8") + and position_ids is not None + and qkv.dtype == torch.bfloat16 + ) + + +def _dense_qk_norm_rope_fp8( + core, i: int, qkv: torch.Tensor, position_ids, kv_dest: Optional[tuple] = None +) -> torch.Tensor: + """A dense layer's FP8 q/k/v: QK norm + RoPE with Q/K rounded to BF16 and then to E4M3 and V cast, the bits of the + BF16 fused_qk_norm_rope output cast to FP8 (K is stored in the FP8 cache), in one launch of this target's op + (``DENSE_QKV_FP8_VIA_BF16``); without the op, the BF16 kernel and the cast. + + With ``kv_dest`` = (the layer's HND pool view, the step's int32 cache slots, live tokens, a [tokens, q heads, 128] + FP8 buffer) the op also stores the live tokens' K and V rows into the pool (the bytes ``write_msa_phase_kv`` + stores, one launch instead of about ten) and every row's Q into the buffer; ``kv_dest`` is passed only where the + op is known to run (``_dense_fp8_op_applies``).""" + if not _dense_fp8_op_applies(qkv, position_ids): + return fp8_e4m3_saturate(_qk_norm_rope(core, i, qkv, position_ids, False)) + _record("m3_dense_qk_norm_rope_fp8", qkv.shape[0], kv_dest is not None) + args = ( + qkv.contiguous(), + core.heads, + core.kv_heads, + core.kv_heads, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_q_norm"], + core.w[f"l{i}_k_norm"], + core.rope_theta, + True, # NeoX RoPE + position_ids.reshape(-1).contiguous().to(torch.int32), + True, # Gemma norms + ) + if kv_dest is not None: + return torch.ops.trtllm.minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv(*args, *kv_dest) + return torch.ops.trtllm.minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8(*args) + + +def _dense_swiglu(core, gu: torch.Tensor) -> torch.Tensor: + """The dense MLP's SwiGLU-OAI activation: this target's C++ op (``m3_dense_swiglu``, minimaxM3Sm107Tp8/ + swigluOai.cu: bit for bit the Triton silu_and_mul at limit 7, alpha 1.702, beta 1, PDL-launched with an early + release so the down GEMV stages its weights during it) at the rows its cells certify, in any graph or eager step; + otherwise the Triton kernel (``m3_silu_and_mul``). TP2 chain microbench: -0.69 us per dense layer vs the + Triton kernel. Every decline is logged once.""" + path = DENSE_SWIGLU_CPP_PATH + rows = gu.shape[0] + max_rows = max(cell.spec["rows"] for cell in m3_dense_swiglu.CELLS) + params = (core.swiglu_alpha, core.swiglu_limit, core.swiglu_beta) + act = None + if rows > max_rows: + uncovered(path, f"{rows} rows, past its certified {max_rows}") + elif params != (DENSE_ALPHA, DENSE_LIMIT, DENSE_BETA): + uncovered(path, f"activation parameters {params}") + else: + act = m3_dense_swiglu( + gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta, get_env_enable_pdl() + ) + if act is None: + uncovered(path, "the gate_up layout") + else: + _record("m3_dense_swiglu", rows) + taken(path, rows, "dense swigluoai") + if act is None: + act = m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + return act + + +def _fp8_index_qk_norm_rope( + core, i: int, idx_qk: torch.Tensor, position_ids, msa_idx_k_cache, msa_out_cache_loc +) -> torch.Tensor: + """The index branch of an unfused sparse layer: index-Q in FP8 out, index-K into the FP8 index-K cache, normed and + roped in one kernel (the built-in ``_fused_fp8_index_qk_norm_rope``). ``msa_idx_k_cache`` / ``msa_out_cache_loc`` + are the calling target's ``step_args`` entries.""" + if position_ids is None or idx_qk.dtype != torch.bfloat16: + raise NotImplementedError( + "MiniMax-M3 fused FP8 indexer requires BF16 activations and position_ids." + ) + index_k_cache = msa_idx_k_cache(i) + if index_k_cache.dtype != torch.float8_e4m3fn: + raise ValueError( + "MiniMax-M3 fused FP8 indexer requires an E4M3 index-K cache, " + f"got {index_k_cache.dtype}." + ) + rows = int(idx_qk.shape[0]) + _record("m3_fp8_indexer_qk_norm_rope", rows) + return m3_fp8_indexer_qk_norm_rope( + idx_qk.contiguous(), + index_k_cache, + msa_out_cache_loc[:rows], + core.index_heads, + core.index_dim, + core.rotary_dim, + core.rms_eps, + core.w[f"l{i}_idx_q_norm"], + core.w[f"l{i}_idx_k_norm"], + core.rope_theta, + position_ids.reshape(-1).contiguous().to(torch.int32), + ).flatten(1) + + +# -------------------------------------------------------------------------------------------------------------------- +# Layer helpers both targets' bodies call: the Eagle3 capture hook and the norm after a layer +# -------------------------------------------------------------------------------------------------------------------- + + +def _plain_capture(spec_metadata, layer_id: int, hidden_states, residual) -> None: + """The in-tree Eagle3 hidden-state capture.""" + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + + +def _next_norm(core, i: int) -> torch.Tensor: + """The norm the boundary after layer ``i`` applies: the next layer's input norm, the final norm after the last.""" + return core.w[f"l{i + 1}_in_norm"] if i + 1 < core.num_layers else core.w["final_norm"] + + +# -------------------------------------------------------------------------------------------------------------------- +# Kernel prebuilds the core runs before the executor's warmup +# -------------------------------------------------------------------------------------------------------------------- + + +def _in_tree_scorer(): + """The in-tree CuTe DSL index scorer's runner (importing it registers ``cute_dsl_minimax_m3_index_decode_score``); + imported on first use, as the CuTe DSL ops are.""" + from tensorrt_llm._torch.custom_ops.cute_dsl_custom_ops import ( + CuteDSLMiniMaxM3IndexDecodeScoreRunner, + ) + + return CuteDSLMiniMaxM3IndexDecodeScoreRunner + + +def _dummy_scorer_operands(dtype, heads, qlen, dim, page, device): + """One request of two blocks: real-shaped operands that load a compiled scorer module.""" + rows, blocks = 1, 2 + tokens = rows * qlen + idx_q = torch.zeros(tokens, heads, dim, dtype=torch.bfloat16, device=device).to(dtype) + idx_k = torch.zeros(rows * blocks, 1, page, dim, dtype=torch.bfloat16, device=device).to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device=device).view(rows, blocks) + seq_lens = torch.full((rows,), blocks * page, dtype=torch.int32, device=device) + score = torch.empty(heads, blocks, tokens, dtype=torch.float32, device=device) + return idx_q, idx_k.squeeze(1), block_table, score.transpose(1, 2), seq_lens + + +#: Mixed (context + generation) steps run their generation rows on the built-in sparse decode +#: (``MsaDecodeFmha._run_sparse``). Its Triton producer and merge specialize on the split-K factor (16 / 8 / 4 / 2 / 1 +#: by the row count on SM107), on the row count's divisibility and on the 16-byte alignment of the request rows (a +#: mixed step's generation requests follow its context requests, so its block-table and length rows start mid-buffer), +#: so ``_prebuild_builtin_sparse_decode`` launches them per decode row count up to this many rows, both alignments. +BUILTIN_SPARSE_DECODE_PREBUILD_MAX_ROWS = ( # capacity: fixed: the prebuilt built-in decode rows + 64 # capacity: fixed: the mixed steps' generation rows prebuilt +) + + +def _prebuild_builtin_sparse_decode(query_lens, num_heads, num_kv_heads, device) -> list: + """Compile (and load) the built-in sparse decode's Triton producer and merge for every generation row count a mixed + step can have, so no step compiles one after the server is ready. Each launch mirrors the operands + ``MsaDecodeFmha._run_sparse`` passes for an NVFP4 layer (FP8 q, the HND pool's K / V planes, the scale pool's + planes, the head-major top-k view, BF16 output) on dummy values, on the current stream. Returns the row counts.""" + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.triton_sparse_decode import ( + minimax_m3_sparse_attn_decode, + ) + + page, head_dim, topk = 128, 128, MSA_REQUIRED_TOPK + pages = topk + pool = torch.zeros( + pages, 2, num_kv_heads, page, head_dim // 2, dtype=torch.uint8, device=device + ) + scales = torch.zeros( + pages, 2, num_kv_heads, page, head_dim // 16, dtype=torch.uint8, device=device + ) + global_scales = torch.ones(2, 4, dtype=torch.float32, device=device) + done = [] + for qlen in sorted(set(query_lens)): + for rows in range(qlen, BUILTIN_SPARSE_DECODE_PREBUILD_MAX_ROWS + 1, qlen): + requests = rows // qlen + q = torch.zeros(rows, num_heads, head_dim, dtype=torch.bfloat16, device=device).to( + torch.float8_e4m3fn + ) + indexes = ( + torch.arange(topk, dtype=torch.int32, device=device) + .repeat(rows, num_kv_heads, 1) + .contiguous() + ) + # One leading context request: its rows are skipped, so the generation rows start mid-buffer. + block_rows = torch.arange(topk, dtype=torch.int32, device=device).repeat( + requests + 1, 1 + ) + lengths = torch.full((requests + 1,), topk * page, dtype=torch.int32, device=device) + output = torch.empty(rows, num_heads, head_dim, dtype=torch.bfloat16, device=device) + for first in (0, 1): + minimax_m3_sparse_attn_decode( + q, + pool[:, 0], + pool[:, 1], + indexes.permute(1, 0, 2), + block_rows[first : first + requests], + lengths[first : first + requests], + sm_scale=head_dim**-0.5, + output=output, + decode_query_len=qlen, + k_block_scale=scales[:, 0], + v_block_scale=scales[:, 1], + k_global_scale=global_scales[0, :1], + v_global_scale=global_scales[1, :1], + ) + done.append(rows) + return done + + +# -------------------------------------------------------------------------------------------------------------------- +# The core +# -------------------------------------------------------------------------------------------------------------------- + +# Engine states (compiled kernel + workspace) per (device, token count[, shared fold]), shared by every layer: a step +# runs the layers one after another on one stream. +_ENGINE_STATES: Dict[tuple, object] = {} + + +# AR-DET: AllReduce AUTO's per-process timing autotuner made VR runs differ across server starts +# (0 / 888 same-node and 0 / 296 across node pairs with it off, bitwise to the +# modal autotuned output). The target turns it off for the whole process at init (AUTO then takes the C++ size / world +# heuristic, a pure function of the shape), unless TRTLLM_M31_ENABLE_AR_AUTOTUNE=1. An explicit +# TLLM_DISABLE_ALLREDUCE_AUTOTUNE is left as given. Read once, at import. +AR_AUTOTUNE_ENABLED = m31_enabled("AR_AUTOTUNE") # the m31_switches value contract (1/true/on/yes) + + +def _pin_allreduce_heuristic(rank: int) -> bool: + """Default the AllReduce autotune off for this process (see AR-DET); returns whether it is off. Logs one line.""" + if not AR_AUTOTUNE_ENABLED: + os.environ.setdefault("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "1") + off = os.environ.get("TLLM_DISABLE_ALLREDUCE_AUTOTUNE", "0") == "1" + if off: + logger.info(f"[M3.1] rank {rank}: AR_DET engaged (AllReduce autotune off for this process)") + logger.info( + f"M3.1 AR-DET: AllReduce autotune {'off (C++ size / world heuristic)' if off else 'on'} on rank {rank}" + ) + return off + + +class MiniMaxM3Tp8ModelingV2(DecoderModel): + """The MiniMax-M3 text model of this target: its weights, its runtime resources, and the per-step dispatch.""" + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + _validate_runtime(model_config) + super().__init__(model_config) + cfg = model_config.pretrained_config + mapping = model_config.mapping + _pin_allreduce_heuristic(mapping.tp_rank) + self.mapping = mapping + # Whether the 9..16-row generation router takes the TF32 tensor-core kernel (set by prebuild_kernels). + self.router_tc_cluster16 = False + self.vocab_size = cfg.vocab_size + d = _weights.MODEL_WEIGHTS.dims(self) + self.num_layers = d.num_layers + self.hidden = d.hidden + self.heads = d.heads + self.kv_heads = d.kv_heads + self.head_dim = d.head_dim + self.index_heads = d.index_heads + self.index_dim = d.index_dim + self.q_size = d.heads * d.head_dim + self.kv_size = d.kv_heads * d.head_dim + self.moe_inter = d.moe_inter + self.num_experts = d.num_experts + self.top_k = int(cfg.num_experts_per_tok) + self.routed_scaling_factor = float(getattr(cfg, "routed_scaling_factor", 1.0)) + self.rms_eps = float(cfg.rms_norm_eps) + self.swiglu_alpha = float(getattr(cfg, "swiglu_alpha", 1.702)) + self.swiglu_beta = 1.0 # the (up + 1) offset of SwiGLU-OAI + self.swiglu_limit = float(getattr(cfg, "swiglu_limit", 7.0)) + self.max_num_tokens = model_config.max_num_tokens + self.moe_max_num_tokens = model_config.moe_max_num_tokens + self.sparse_layers = frozenset(_sparse_layer_ids(cfg)) + self.moe_layers = frozenset(_moe_layer_ids(cfg)) + mode = model_config.quant_config.quant_mode + self.main_kv_is_fp8 = bool(mode is not None and mode.has_fp8_kv_cache()) + self.main_kv_is_nvfp4 = bool(mode is not None and mode.has_fp4_kv_cache()) + # The sparse layers' QKV and index projections: one five-way packed projection, or the QKV projection and a + # separate MXFP8 index projection (fuse_qkv_index_projection=False) run on two streams. + self.fused_projection = _weights.fused_projection(model_config.sparse_attention_config) + + self.w = _weights.MODEL_WEIGHTS.declare(d) + + # The engine reads these two off the module tree: the MXFP8 GEMM state it tunes, and the opt-in to the + # automatic FlashInfer MXFP8 path on decode graphs. + self.mxfp8 = _Mxfp8GemmState() + self._use_flashinfer_mxfp8_decode_graph_default = True + + # The built-in fused all-reduce + residual + RMSNorm of the layer boundaries the multicast kernel does not take. + self.allreduce = AllReduce( + mapping=mapping, strategy=model_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + + # Streams: the shared / routed expert overlap and, when the MoE chunks its input, the chunking overlap. + self.aux_stream_moe = torch.cuda.Stream() + chunked = self.moe_max_num_tokens < model_config.max_num_tokens * mapping.dp_size + self.aux_stream_moe_chunk = torch.cuda.Stream() if chunked else None + self.moe_events = {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers} + self.moe_chunk_events = { + i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.moe_layers + } + # The unfused sparse layers' index branch overlaps the QKV branch on its own stream. + self.aux_stream_attn = None if self.fused_projection else torch.cuda.Stream() + self.attn_events = ( + {} + if self.fused_projection + else {i: (torch.cuda.Event(), torch.cuda.Event()) for i in self.sparse_layers} + ) + + # Partial NeoX RoPE (rotary_dim of head_dim): the cos/sin table every attention kernel reads. + rope = RopeParams.from_config(cfg) + self.rope_theta = rope.theta + self.rotary_dim = int(rope.dim) + self.rotary_cos_sin = rope.create_rope_const_params(interleave=False)[1].reshape( + rope.max_positions, 2, -1 + ) + + # One MSA attention backend per layer (dense and sparse; the step metadata is the backend's). + quant_config = model_config.get_quant_config() + self.attn: List[MiniMaxM3MsaSparseAttention] = [] + for i in range(self.num_layers): + backend = create_attention( + model_config.attn_backend, + i, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=None, + quant_config=quant_config, + skip_create_weights_in_init=model_config.skip_create_weights_in_init, + q_scaling=1.0, + sparse_params=model_config.sparse_attention_config.to_sparse_params( + pretrained_config=cfg, layer_idx=i + ), + ) + backend.update_quant_config(quant_config) + if not isinstance(backend, MiniMaxM3MsaSparseAttention): + raise ValueError( + f"layer {i}: expected the MiniMax-M3 MSA backend, got {type(backend).__name__}" + ) + if i not in self.sparse_layers: + # The KV split trtllm-gen dense decode runs at, per request count. + backend.dense_decode_max_seq_len_by_batch = DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH + self.attn.append(backend) + + # Built once the weights are real (``post_load_weights``): the KV (Q, K, V) scales, the MoE scalars, the + # decode engines and workspaces, the two targets. + self.kv_scales: Dict[int, torch.Tensor] = {} + self.inv_kv_scales: Dict[int, torch.Tensor] = {} + self.moe: Dict[int, _MoeScalars] = {} + self.mx_alpha: Optional[torch.Tensor] = None + self.moe_engines: Dict[int, Dict[Tuple[int, bool, bool], Optional[object]]] = {} + self.embed_owners: Dict[tuple, DecodeEmbedding] = {} + # Each sparse layer's selected block ids per query row at its last long-context decode + # (``SPARSE_DECODE_LC8_PREFETCH_NEXT_KV``, up to ``N1_MAX_ROWS`` query rows), read by the previous sparse + # layer's decode to prefetch this layer's KV. + self.n1_ids: Dict[int, torch.Tensor] = ( + { + i: torch.full( + (N1_MAX_ROWS, MSA_REQUIRED_TOPK), + -1, + dtype=torch.int32, + device=torch.device("cuda", torch.cuda.current_device()), + ) + for i in sorted(self.sparse_layers) + } + if SPARSE_DECODE_LC8_PREFETCH_NEXT_KV + else {} + ) + self._built_for: Optional[tuple] = None + self._prefill: Optional[PrefillTarget] = None + self._decode: Optional[DecodeTarget] = None + + def sm_scale(self, i: int) -> float: + return (self.head_dim**-0.5) / float(self.attn[i].q_scaling) + + # ---------------------------------------------------------------------------------------------- after the load + + def post_load_weights(self) -> None: + """Derive the kernels' state from the loaded weights, then build both targets and every decode kernel. + + Idempotent per set of weight storages: the loader's post-load walk reaches this module directly and through + the shell, and a second call on the same storages does nothing. + """ + identity = ( + self.w["embed"].data_ptr(), + self.w["l0_in_norm"].data_ptr(), + self.w["final_norm"].data_ptr(), + ) + if self._built_for == identity: + return + device = self.w["embed"].device + derived = nn.Module() + derived.register_buffer( + "mx_alpha", torch.ones(1, dtype=torch.float32, device=device), persistent=False + ) + for i in range(self.num_layers): + # No KV calibration in the checkpoint: unit (Q, K, V) scales and their reciprocals. + derived.register_buffer( + f"l{i}_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + derived.register_buffer( + f"l{i}_inv_kv_scales", + torch.ones(3, dtype=torch.float32, device=device), + persistent=False, + ) + for i in sorted(self.moe_layers): + for name, value in dataclasses.asdict(self._derive_moe(i)).items(): + derived.register_buffer(f"l{i}_{name}", value, persistent=False) + self.derived = derived + self.mx_alpha = derived.mx_alpha + self.kv_scales = {i: getattr(derived, f"l{i}_kv_scales") for i in range(self.num_layers)} + self.inv_kv_scales = { + i: getattr(derived, f"l{i}_inv_kv_scales") for i in range(self.num_layers) + } + self.moe = { + i: _MoeScalars( + **{ + f.name: getattr(derived, f"l{i}_{f.name}") + for f in dataclasses.fields(_MoeScalars) + } + ) + for i in sorted(self.moe_layers) + } + self._build_embed_owners() + self.prebuild_kernels() + self._prefill = PrefillTarget(self) + self._decode = DecodeTarget(self) + self._built_for = identity + + def cache_derived_state(self) -> None: + """The staged loaders' entry to the same build (``post_load_weights``).""" + self.post_load_weights() + + def _derive_moe(self, i: int) -> _MoeScalars: + """The routed experts' kernel scalars from the checkpoint's per-tensor scales (trtllm-gen NVFP4 layout). + + Global input scales: the reciprocal of the largest per-expert input scale (gate and up must agree). Per expert: + ``alpha = 1 / (input_scale * (1 / weight_scale_2))`` for the gate (fc31_alpha), the up half (its GEMM2-input + rescale ``fc31_scale_c = fc2_input_scale * up_alpha``) and the down projection; SwiGLU-OAI's activation + constants per expert, with beta and the clamp in the pre-dequant GEMM output domain (divided by fc31_alpha). + """ + w = self.w + isc1, isc3, isc2 = w[f"l{i}_e_isc1"].data, w[f"l{i}_e_isc3"].data, w[f"l{i}_e_isc2"].data + if not torch.allclose(isc1, isc3): + raise ValueError(f"layer {i}: the routed experts' w1 and w3 input scales differ") + fc31_input_scale = isc1.max().reciprocal() + fc2_input_scale = isc2.max().reciprocal() + ws2_1, ws2_3, ws2_2 = ( + w[f"l{i}_e_ws2_1"].data, + w[f"l{i}_e_ws2_3"].data, + w[f"l{i}_e_ws2_2"].data, + ) + fc31_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_1)) + up_alpha = 1.0 / (fc31_input_scale * (1.0 / ws2_3)) + fc2_alpha = 1.0 / (fc2_input_scale * (1.0 / ws2_2)) + fc31_scale_c = fc2_input_scale * up_alpha + n = self.num_experts + device = fc31_alpha.device + act_alpha = torch.full((n,), self.swiglu_alpha, dtype=torch.float32, device=device) + act_beta = torch.full((n,), self.swiglu_beta, dtype=torch.float32, device=device) + act_clamp = torch.full((n,), self.swiglu_limit, dtype=torch.float32, device=device) + act_beta.div_(fc31_alpha) + act_clamp.div_(fc31_alpha) + return _MoeScalars( + fc31_input_scale=fc31_input_scale, + fc2_input_scale=fc2_input_scale, + fc31_alpha=fc31_alpha, + fc2_alpha=fc2_alpha, + fc31_scale_c=fc31_scale_c, + act_alpha=act_alpha, + act_beta=act_beta, + act_clamp=act_clamp, + ) + + def _build_embed_owners(self) -> None: + """The fused embedding + first RMSNorm's caller-owned buffers, one per (rows, id dtype) of a decode step.""" + table, weight = self.w["embed"], self.w["l0_in_norm"] + self.embed_owners = { + (rows, dtype): DecodeEmbedding(table, weight, self.rms_eps, rows, dtype) + for rows in range(1, EMBED_FIRSTNORM_MAX_TOKENS + 1) + for dtype in (torch.int32, torch.int64) + } + + def projections(self) -> List[Tuple[str, str, Optional[str]]]: + """Every MXFP8 projection in layer order: (weight key, scale key, the boundary its GEMV pushes into).""" + out = [] + for i in range(self.num_layers): + out.append((f"l{i}_qkv_w", f"l{i}_qkv_s", None)) + out.append((f"l{i}_o_w", f"l{i}_o_s", "attn")) + if i in self.moe_layers: + out.append((f"l{i}_sh_gu_w", f"l{i}_sh_gu_s", None)) + out.append((f"l{i}_sh_dn_w", f"l{i}_sh_dn_s", "moe")) + else: + out.append((f"l{i}_mlp_gu_w", f"l{i}_mlp_gu_s", None)) + out.append((f"l{i}_mlp_dn_w", f"l{i}_mlp_dn_s", "dense")) + return out + + def decode_query_lens(self) -> List[int]: + """The query rows per request a decode step can have: 1, and 1 + each draft length speculation runs.""" + lens = {1} + spec = getattr(self.model_config, "spec_config", None) + if spec is not None: + # The verify width the config itself declares, whatever it names its draft count. + per_step = getattr(spec, "tokens_per_gen_step", None) + if isinstance(per_step, int) and per_step > 1: + lens.add(per_step) + for drafts in ( + getattr(spec, "max_draft_len", None), + getattr(spec, "max_total_draft_tokens", None), + *(getattr(spec, "draft_len_schedule", None) or {}).values(), + ): + if drafts: + lens.add(1 + int(drafts)) + return sorted(lens) + + def prebuild_kernels(self) -> Dict[str, object]: + """Compile (and load) every kernel a decode forward dispatches to that would otherwise compile on its first + call: each projection's decode GEMVs (every row count, activation and boundary push), the MoE engines of + ``MOE_ENGINE``'s token counts, the index scorers at the decode query lengths (this target's and the in-tree + one wider steps take), the 384-thread long-row select, the dense layers' full-context decode plans, and the + built-in sparse decode a mixed step's generation rows run. Runs on the current (main) stream before the + executor's warmup, in the same order on every rank, then synchronizes; after it, GEMV compiles are frozen and + the forward only looks kernels up (a missing one is a miss). Returns the seconds each part took and the keys. + """ + times: Dict[str, object] = {} + # The 9..16-row generation router: decided once, from the device, never inside a CUDA-graph capture. + # The built-in gate's conditions, mirrored exactly: 128 experts, a hidden size in 1024s, 16-CTA clusters. + cfg = self.model_config.pretrained_config + self.router_tc_cluster16 = ( + int(cfg.num_local_experts) == 128 + and int(cfg.hidden_size) % 1024 == 0 + and bool(torch.ops.trtllm.minimax_m3_router_tc_cluster16(int(cfg.hidden_size))) + ) + logger.info( + f"[M3 prebuild] rank {mpi_rank()}: TF32 tensor-core router for {ROUTER_TC_GEN_MIN_ROWS}.." + f"{ROUTER_TC_GEN_MAX_ROWS}-row generation steps: " + + ( + "engaged (16-CTA clusters)" + if self.router_tc_cluster16 + else "not used (no 16-CTA clusters)" + ) + ) + if not self.mxfp8.quant_method.use_cutlass: + raise ValueError("the MiniMax-M3 decode dispatch needs the CUTLASS MXFP8 GEMM") + from .kernels import m3_fullctx_decode as _fullctx_decode_kernel + from .kernels import m3_select_long_dsl as _select_long_dsl_kernel + from .kernels import m3_tokmap_guard + + device = self.w["embed"].device + # The one-request token-map guard of the stock generation preprocess (``PREPROCESS_TOKMAP_FIX``): every path + # that can run it (the drafter's full-context decode or its fallback, any trtllm-gen fallback), so it is + # installed whatever else is enabled. + m3_tokmap_guard.install( + torch.device("cuda", torch.cuda.current_device()), + TP2_ROWS, + lambda rows: taken(PREPROCESS_TOKMAP_PATH, rows, "one-request token map"), + lambda rows, reason: missed(PREPROCESS_TOKMAP_PATH, rows, reason), + ) + mx_gemv.freeze(False) + use_pdl = get_env_enable_pdl() + world = int(self.mapping.tp_size) + start = time.perf_counter() + for wk, sk, push in self.projections(): + weight, scale = self.w[wk], self.w[sk] + mx_gemv.ensure_compiled(weight, scale, use_pdl) + n, k = weight.shape + for rows, x_bf16, ar in _gemv_variants(n, k, push, world): + mx_gemv.precompile(weight, scale, rows, x_bf16, use_pdl=use_pdl, ar=ar) + # #117: the 12-head packed projection also takes its MXFP8 rows <= 8 with the producer epilogue. + if ( + QKV_PRODUCER_FOLD + and n == mx_gemv.PROD_N + and not x_bf16 + and ar is None + and rows <= 8 + ): + mx_gemv.precompile(weight, scale, rows, False, use_pdl=use_pdl, prod=True) + # SWQ (Triton) compiles on first use, which must not happen inside a graph capture: one call per down + # projection shape it serves covers every row count. + swq_shapes = {tuple(self.w[wk].shape) for wk, _, _ in self.projections()} & set(SWQ_ROWS) + for n, k in sorted(swq_shapes): + gu = torch.zeros( + max(SWQ_ROWS[(n, k)]), 2 * k, dtype=torch.bfloat16, device=self.w["embed"].device + ) + m3_swq(gu, self.swiglu_limit, self.swiglu_alpha, self.swiglu_beta) + logger.info(f"[M3 prebuild] SWQ: compiled for intermediate {k}") + # LINEAR_CUBLASLT / VR_GEMV48: cuBLAS creates its handle on the first call, which fails inside a graph capture, + # and the CuTe DSL quantize compiles on first use: one eager call per projection shape and row count, both + # activations (the GEMV48 rows' GEMVs compile above, through _gemv_variants). + wide_shapes = set() + for wk, sk, _ in self.projections(): + weight, scale = self.w[wk], self.w[sk] + nk = tuple(weight.shape) + if nk in wide_shapes or (nk not in LINEAR_CUBLASLT_ROWS and nk not in GEMV48_ROWS): + continue + wide_shapes.add(nk) + for m in sorted(GEMV48_ROWS.get(nk, frozenset())): + mxfp8_cublaslt.mxfp8_quantize( + torch.zeros(m, nk[1], dtype=torch.bfloat16, device=weight.device) + ) + for m in sorted(LINEAR_CUBLASLT_ROWS.get(nk, frozenset())): + x = torch.zeros(m, nk[1], dtype=torch.bfloat16, device=weight.device) + act, act_scale = mxfp8_cublaslt.mxfp8_quantize(x) + m3_mxfp8_cublaslt(x, None, weight, scale) + m3_mxfp8_cublaslt(act, act_scale, weight, scale) + if wide_shapes: + logger.info( + f"[M3 prebuild] LINEAR_CUBLASLT / VR_GEMV48: cuBLAS and the quantize warmed for {sorted(wide_shapes)}" + ) + torch.cuda.synchronize() + times["gemv_s"] = time.perf_counter() - start + start = time.perf_counter() + for i in sorted(self.moe_layers): + self._prebuild_engines(i) + torch.cuda.synchronize() + times["moe_s"] = time.perf_counter() - start + start = time.perf_counter() + times["scorer_keys"] = self._prebuild_index_scorer(self.decode_query_lens(), device) + torch.cuda.synchronize() + times["scorer_s"] = time.perf_counter() - start + start = time.perf_counter() + times["select_keys"] = [] + if long_select_dsl_keys(): + # The 384-thread long-row select (``SELECT_LONG_DSL``): one kernel per launch key. + times["select_keys"] = _select_long_dsl_kernel.prebuild(long_select_dsl_keys(), device) + torch.cuda.synchronize() + times["select_s"] = time.perf_counter() - start + start = time.perf_counter() + times["fullctx_plans"] = 0 + if FULLCTX_DECODE.keys: + # The dense layers' full-context decode: one compiled PrimTS plan per covered (rows, requests). + times["fullctx_plans"] = _fullctx_decode_kernel.prebuild( + device, + {key: FULLCTX_DECODE_RECIPES[key] for key in FULLCTX_DECODE.keys}, + FULLCTX_DECODE_MAX_KV_LEN, + FULLCTX_DECODE_OWN_REDUCER and _fullctx_decode_kernel.has_own_reducer(), + use_pdl, + ) + torch.cuda.synchronize() + times["fullctx_s"] = time.perf_counter() - start + start = time.perf_counter() + times["builtin_decode_rows"] = _prebuild_builtin_sparse_decode( + self.decode_query_lens(), self.heads, self.kv_heads, device + ) + torch.cuda.synchronize() + times["builtin_decode_s"] = time.perf_counter() - start + start = time.perf_counter() + times["proxy_variants"] = self._prebuild_msa_proxy() if MSA_PROXY_PREBUILD else [] + times["proxy_s"] = time.perf_counter() - start + mx_gemv.freeze(True) + logger.info( + "[M3 prebuild] decode kernels built before warmup: GEMVs %.1f s, MoE engines %.1f s, index scorer " + "%.1f s (%d variants), long-row select %.1f s (%d variants), full-context decode %.1f s (%d plans), " + "built-in sparse decode %.1f s (%d row counts), MSA indexer proxy %.1f s (%d variants)" + % ( + times["gemv_s"], + times["moe_s"], + times["scorer_s"], + len(times["scorer_keys"]), + times["select_s"], + len(times["select_keys"]), + times["fullctx_s"], + times["fullctx_plans"], + times["builtin_decode_s"], + len(times["builtin_decode_rows"]), + times["proxy_s"], + len(times["proxy_variants"]), + ) + ) + return times + + # ------------------------------------------------------------------------------------------ the MoE engines + + def _expert_operands(self, i: int): + """The routed experts' operands the engines read (trtllm-gen NVFP4 SwiGLU-OAI layout), and the input scale.""" + w, m = self.w, self.moe[i] + tensors = [ + w[f"l{i}_e_w31"].data, + w[f"l{i}_e_s31"].data, + w[f"l{i}_e_w2"].data, + w[f"l{i}_e_s2"].data, + m.fc31_alpha, + m.fc31_scale_c, + m.act_alpha, + m.act_beta, + m.act_clamp, + m.fc2_alpha, + ] + return tensors, m.fc31_input_scale + + def _shared_weights(self, i: int): + w = self.w + return ( + w[f"l{i}_sh_gu_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_gu_s"].data, + w[f"l{i}_sh_dn_w"].data.view(torch.float8_e4m3fn), + w[f"l{i}_sh_dn_s"].data, + ) + + def engine(self, i: int, num_tokens: int, shared: bool) -> Optional[object]: + """Layer ``i``'s prebuilt routed-expert engine for ``num_tokens`` (1..8), router folded in (and the shared + expert with ``shared``). None after a miss: the build failed or was never made (a forward never builds one).""" + return self._lookup( + i, self.moe_engines[i], (num_tokens, True, shared), MOE_ENGINE.path, num_tokens + ) + + def _lookup(self, i: int, engines: dict, key, path: str, num_tokens: int) -> Optional[object]: + if key not in engines: + what = f"layer {i}'s engine" + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, what) + else: + missed(path, num_tokens, f"{what} was not built before warmup") + return None + engine = engines[key] + if engine is None: + missed(path, num_tokens, "its build failed (logged)") + return engine + + def _prebuild_engines(self, i: int) -> None: + """Build (compile, on first use of each shared state, and warm) every engine a decode step of layer ``i`` can + take: ``MOE_ENGINE``'s token counts with the shared expert folded in. Whether an engine is kept is agreed + across the TP group (one host collective per engine): a build that fails on one rank drops that engine on + every rank, so the ranks never mix an engine (and its boundary push) with the generic MoE.""" + engines = self.moe_engines.setdefault(i, {}) + for num_tokens in sorted(MOE_ENGINE.keys): + key = (num_tokens, True, True) + if key not in engines: + engines[key] = m3_mc_allreduce.tp_agree_built( + self.mapping, + self._try_build( + i, MOE_ENGINE.path, num_tokens, self._build_engine, shared=True + ), + f"[M3 dispatch] {MOE_ENGINE.path} at {num_tokens} rows: layer {i}'s engine", + ) + + def _try_build(self, i: int, path: str, num_tokens: int, build, **kwargs) -> Optional[object]: + try: + return build(i, num_tokens, **kwargs) + except (ImportError, RuntimeError, ValueError) as err: + logger.warning( + f"[M3 dispatch] {path} at {num_tokens} rows: layer {i}'s engine failed to build: {err}" + ) + return None + + def _build_engine(self, i: int, num_tokens: int, shared: bool) -> object: + """The 1..8-token engine (router, routed experts and the shared expert in one launch, at the TP8 width 384). + Rubin TP2 ablation chain (57 layers behind a 58.4 MB PDL producer, row = (P + MoE) - P, both orders): 31.3 / + 50.7 us/layer at 4 / 8 tokens against the built-in MoE's 42.4 / 55.4 (FlashInfer MXFP8 shared expert); bit for + bit the built-in block's shared.add_(routed). When the engine pushes its sum into the boundary all-reduce, + every CTA releases the PDL dependents right after its grid dependency wait instead of after the push + (``MOE_EARLY_AR_RELEASE``), so the all-reduce (which polls the pushed tagged words, not the grid) becomes + resident on the SMs the engine frees: scheduling only, the same bits (Rubin TP2 greedy early column, same-job + A,B + B,A: c1 -15 us).""" + tensors, scale = self._expert_operands(i) + device = tensors[0].device + sms = torch.cuda.get_device_properties(device).multi_processor_count + kwargs = {} + if shared: + kwargs["shared_weights"] = self._shared_weights(i) + state = _ENGINE_STATES.get((str(device), num_tokens, shared)) + if state is None: + state = _ENGINE_STATES[(str(device), num_tokens, shared)] = M3MoeState( + device, + num_tokens=num_tokens, + num_ctas=sms + - (shared_ctas_for(num_tokens, shared) if shared else MOE_ENGINE_RESERVED_SMS), + shared=shared, + early_release=MOE_EARLY_AR_RELEASE, + ) + engine = state.layer( + *tensors, + self.w[f"l{i}_router_bias"].data, + router_weight=self.w[f"l{i}_router"].data, + fc31_input_scale=scale.reshape(1).float(), + **kwargs, + ) + # Compile (first use of the state) and warm. + m3_moe_engine( + torch.zeros(num_tokens, tensors[0].shape[-1] * 2, dtype=torch.bfloat16, device=device), + engine, + ) + return engine + + # ------------------------------------------------------------------------------------------ the index scorers + + def _index_geometries(self) -> list: + """Every sparse layer's (index dtype, index heads per rank, index dim), deduplicated.""" + geoms = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + config = backend.m3_config + dtype = torch.float8_e4m3fn if backend.indexer_kv_dtype == "fp8" else torch.bfloat16 + geom = (dtype, int(config.num_index_heads), int(config.sparse_index_dim)) + if geom not in geoms: + geoms.append(geom) + return geoms + + def _scorer_variants(self, query_lens) -> list: + """The scorer keys a decode step can take: every sparse layer's index geometry at each decode query length, + with each covered request count's schedule (``index_score_schedule``), where the kernel serves it.""" + runner = _index_score_kernel.IndexScoreRunner + keys = [] + for dtype, heads, dim in self._index_geometries(): + for qlen in sorted(set(query_lens)): + if not runner.is_supported( + q_dtype=dtype, + num_heads=heads, + head_dim=dim, + page_size=runner.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ): + continue + for requests in sorted(INDEX_SCORE.keys): + split_k, early = index_score_schedule(requests) + key = (dtype, heads, int(qlen), dim, int(split_k), bool(early)) + if key not in keys: + keys.append(key) + return keys + + def _in_tree_scorer_variants(self, query_lens) -> list: + """The in-tree CuTe DSL scorer's keys (``CuteDSLMiniMaxM3IndexDecodeScoreRunner``: index dtype, index heads, + decode query length, index dim, its 256-CTA split) a decode step outside ``INDEX_SCORE`` can take: every query + length at which a step of more than ``INDEX_SCORE_MAX_REQUESTS`` requests fits in ``DECODE_STEP_MAX`` (e.g. + 3..8 requests decoding one row each), where that kernel serves the geometry.""" + in_tree = _in_tree_scorer() + widest = DECODE_STEP_MAX // (INDEX_SCORE_MAX_REQUESTS + 1) + keys = [] + for dtype, heads, dim in self._index_geometries(): + for qlen in sorted(set(query_lens)): + if qlen > widest or not in_tree.is_supported( + q_dtype=dtype, + num_heads=heads, + head_dim=dim, + page_size=in_tree.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ): + continue + key = (dtype, heads, qlen, dim, in_tree.SPLIT_K) + if key not in keys: + keys.append(key) + return keys + + def _prebuild_msa_proxy(self) -> list: + """Compile and load every fmha_sm100 variant the MSA indexer's proxy pass can pick (OnlyScore over the paged + index-K: one KV head, page 128, no split-KV), so no context chunk compiles one after the server is ready (a + 65..128-token chunk's variant did, stalling the steps behind it by 0.8-1.6 ms). The variant follows only the + longest query row: fmha_sm100 packs a row of up to 32 tokens by its pack factor, then takes the 128- or 256-row + query tile and the single-warpgroup kernel from the packed length, so query lengths 1..129 reach every variant. + Compile and load only, nothing runs: no numerics change. fmha_sm100 keys its cache by its installed sources, so + a source overlay builds its own variants. Returns the variant names.""" + require_msa_module() + from fmha_sm100 import api as fmha_api + from fmha_sm100 import jit as fmha_jit + + only_score, page = 2, 128 + names = [] + for i in sorted(self.sparse_layers): + backend = self.attn[i] + dtype = torch.float8_e4m3fn if backend.indexer_kv_dtype == "fp8" else torch.bfloat16 + heads = int(backend.m3_config.num_index_heads) + for qlen in range(1, 130): + pack = fmha_api._compute_pack_factor(qlen, heads, 1) + packed = qlen * pack + key = ( + fmha_jit._dlpack_dtype_code(dtype), + 128 if packed <= 128 else 256, + packed <= 64, + only_score, + page, + False, + pack, + ) + name, _ = fmha_jit._variant_key_from_runtime(*key) + if name not in names: + fmha_jit.get_fmha_variant(*key) + names.append(name) + return names + + def _prebuild_index_scorer(self, query_lens, device) -> list: + """Compile every key of ``_scorer_variants`` and ``_in_tree_scorer_variants`` and launch each once on + real-shaped dummy operands (loading its module), on the current stream, so no decode step compiles a scorer. + The kernels' batch, token, page and block counts are symbolic: their compiles need no step metadata. Returns + the keys.""" + runner = _index_score_kernel.IndexScoreRunner + keys = self._scorer_variants(query_lens) + for key in keys: + dtype, heads, qlen, dim, split_k, early = key + runner._compile(dtype, heads, qlen, dim, split_k, early) + q, k, bt, score, seq = _dummy_scorer_operands( + dtype, heads, qlen, dim, runner.SUPPORTED_PAGE_SIZE, device + ) + runner.kernel_cache[key](q, k, bt, score, seq) + in_tree = _in_tree_scorer() + in_tree_keys = self._in_tree_scorer_variants(query_lens) + for key in in_tree_keys: + dtype, heads, qlen, dim, split_k = key + in_tree._compile(dtype, heads, qlen, dim, split_k) + q, k, bt, score, seq = _dummy_scorer_operands( + dtype, heads, qlen, dim, in_tree.SUPPORTED_PAGE_SIZE, device + ) + in_tree.kernel_cache[key](q, k, bt, score, seq) + return keys + in_tree_keys + + # --------------------------------------------------------------------------------------------- the dispatch + + def _select_target(self, attn_metadata: AttentionMetadata): + """Which target runs this step: one host-side integer, fixed per CUDA-graph capture. A step with context rows + (a mixed step included) is prefill's; every pure generation step is decode's, whatever its width.""" + return self._decode if attn_metadata.num_contexts == 0 else self._prefill + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.IntTensor] = None, + position_ids: Optional[torch.IntTensor] = None, + inputs_embeds: Optional[torch.FloatTensor] = None, + spec_metadata: Optional[SpecMetadata] = None, + capture: Callable[..., None] = _plain_capture, + capture_target: Optional[Callable[..., Optional[tuple]]] = None, + **kwargs, + ) -> torch.Tensor: + """``capture``: the Eagle3 hidden-state capture of a layer (the worker's, with its capture norm), and + ``capture_target``: where a decode boundary all-reduce writes a capture layer's capture norm itself + (``CAPTURE_NORM_IN_BOUNDARY``; the worker's, None without one).""" + if (input_ids is None) ^ (inputs_embeds is not None): + raise ValueError("You must specify exactly one of input_ids or inputs_embeds") + if kwargs.get("lora_params") is not None: + raise ValueError("this MiniMax-M3 target serves no LoRA adapters") + return self._select_target(attn_metadata).forward( + attn_metadata, + input_ids, + position_ids, + inputs_embeds, + spec_metadata, + capture, + capture_target, + ) + + +# -------------------------------------------------------------------------------------------------------------------- +# The targets +# -------------------------------------------------------------------------------------------------------------------- + + +class PrefillTarget: + """A step with context rows: prefill chunks, and the mixed steps that carry generation rows beside them. Every + layer runs in its generic form; the attention core is the MSA backend's for every row.""" + + def __init__(self, core: MiniMaxM3Tp8ModelingV2) -> None: + self.core = core + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it; read once here rather than per forward. + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + Counts and per-step host scalars are read here; device tensors are the metadata's persistent buffers, held + by reference; ``msa_idx_k_cache`` is the metadata's per-layer index-K cache getter. ``metadata`` is the + prepared object itself: the MSA backend's step (``m3_msa_attention``) takes it as its argument. + """ + return dict( + num_contexts=md.num_contexts, + num_ctx_tokens=md.num_ctx_tokens, + num_tokens=md.num_tokens, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + metadata=md, + ) + + def forward( + self, + attn_metadata, + input_ids, + position_ids, + inputs_embeds, + spec_metadata, + capture, + capture_target, + ): + """``capture_target`` is the decode boundaries' (unused: no prefill boundary writes a capture norm).""" + core = self.core + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) prefill CUDA graphs" + ) + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + _record("prefill", int(step["num_tokens"]), int(step["num_contexts"])) + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + hidden, residual = self._layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + return hidden + + def _swiglu_mlp( + self, h: torch.Tensor, prefix: str, low_m_gemm: bool, dense: bool = False + ) -> torch.Tensor: + """A gated MXFP8 MLP (the dense MLP, the shared expert): gate_up, SwiGLU-OAI (the dense MLP's on this target's + C++ op, ``_dense_swiglu``), down, no reduction.""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s", low_m_gemm) + if dense: + act = _dense_swiglu(core, gu) + else: + act = m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + out, _ = _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm) + return out + + def _routed_chunks(self, i: int, x, router_logits): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: + the MoE scheduler's even split, even chunks on the MoE chunking stream.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + # CTXNORMQ: the boundary already wrote this activation's NVFP4 form. + fp4 = _CTXNORMQ_FP4.take(x) if CTXNORMQ else None + if num_chunks <= 1: + if fp4 is not None: + return _routed_experts(core, i, fp4[0], fp4[1], router_logits, True) + return _routed_experts(core, i, x, None, router_logits, True) + sizes = _split_chunk(rows, num_chunks) + xs, logits = x.split(sizes), router_logits.split(sizes) + sfs = [None] * num_chunks + if fp4 is not None: + xs = fp4[0].split(sizes) + sfs = [t.flatten() for t in fp4[1].view(rows, -1).split(sizes)] + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + else: + outs.append(_routed_experts(core, i, xs[c], sfs[c], logits[c], True)) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor): + """Router + routed experts, finalized: the router logits and the experts quantizing their bf16 input + (chunked).""" + return self._routed_chunks(i, h, _router_logits(self.core, i, h)) + + def _moe(self, i: int, h: torch.Tensor, *, return_parts: bool): + """The MoE block of layer ``i`` without its reduction. Returns ``(out, routed)``: with ``return_parts`` the + shared and routed parts (routed None when the engine folded the shared expert in), else their sum in + ``out``.""" + core = self.core + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + h.dim() == 2 + and h.dtype == torch.bfloat16 + and 1 <= rows <= MOE_ENGINE_MAX_TOKENS + and rows >= MOE_FOLD_MIN_TOKENS + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + _record("m3_moe_engine", rows, False) + taken(MOE_ENGINE.path, rows) + folded = m3_moe_engine(h.contiguous(), engine, None, None) + return folded, None + routed_out, shared_out = maybe_execute_in_parallel( + lambda: self._routed(i, h), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False), + core.moe_events[i][0], + core.moe_events[i][1], + # The warmup tuning pass profiles the shared expert's GEMMs: keep it on the main stream, between the + # same collectives on every rank (the capture pass and serving still overlap it on the aux stream). + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if return_parts: + return shared_out, routed_out + # In-place add into the shared output to avoid allocating a temporary. + return shared_out.add_(routed_out), None + + def _generic_boundary(self, x, x2, residual, norm_w): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce.""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + if norm_w is not core.w["final_norm"]: + fused = _ctxnormq_allreduce_norm(core, x, residual, norm_w, None) + if fused is not None: + return fused + return _native_allreduce_norm(core, x, residual, norm_w) + + def _layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + else: + fused = _ctxnormq_allreduce_norm( + core, + hidden, + residual, + core.w[f"l{i}_post_norm"], + core.moe[i].fc31_input_scale if i in core.moe_layers else None, + ) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + if i in core.moe_layers: + out, routed = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out = self._swiglu_mlp(hidden, f"l{i}_mlp", True, dense=True) + routed = None + normed, updated = self._generic_boundary(out, routed, residual, _next_norm(core, i)) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + capture(spec_metadata, i, updated, None) + return normed, updated + + def _attention(self, i: int, position_ids, hidden, step): + """Attention of layer ``i`` up to its o_proj (no reduction). Returns (o_proj output, pushed): a prefill step + arms no boundary push, so pushed is always False.""" + core = self.core + if i not in core.sparse_layers: + # Dense layers 0-2: QKV, the fused norm + RoPE (FP8 q/k/v for an FP8 cache, on this target's op; else in + # place in bf16), the MSA backend (it writes the layer's cache), o_proj. + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = ( + _dense_qk_norm_rope_fp8(core, i, qkv, position_ids) + if core.main_kv_is_fp8 + else _qk_norm_rope(core, i, qkv, position_ids, False) + ) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + if not core.fused_projection: + return self._unfused_sparse_attention(i, position_ids, hidden, step) + packed, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + ) + if horizontal is None: + raise RuntimeError( + f"MiniMax-M3 sparse layer {i}: no horizontal producer takes this step's caches (an FP8 or NVFP4 HND " + "pool with an FP8 index-K cache)" + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output.""" + core = self.core + if _producer_covered(core): + # Not a miss: the producer packs the five-way projection's output, which this configuration lacks. + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True) + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): the MSA + backend (``m3_msa_attention``: the KV write unless a producer wrote the caches, the sparse layers' top-k block + selection, the FMHA over the context rows and any generation rows beside them).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + live = int(step["num_tokens"]) + q_live, out_live = q[:live], output[:live] + k_live = k[:live] if k is not None else None + v_live = v[:live] if v is not None else None + idx_q_live = idx_q[:live] if idx_q is not None else None + sparse = i in core.sparse_layers + nvfp4 = core.main_kv_is_nvfp4 + if GQAMIX and sparse: + _gqamix_register(core.attn[i]) + _record("m3_msa_attention", sparse, k_live is not None) + m3_msa_attention( + core.attn[i], + q_live, + k_live, + v_live, + idx_q_live, + step["metadata"], + out_live, + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + return output + + +class DecodeTarget: + """A pure generation step (no context rows), of any width. + + Up to ``DECODE_MAX_TOKENS`` tokens with 1-D input ids (up to 2 requests verifying 1 + 3 rows, the GEN worker's + batch), per layer: attention (o_proj all-reduce deferred, its rows pushed into the boundary all-reduce) -> + attention boundary (all-reduce + residual + post-attention RMSNorm) -> MoE (the 1..8-row engine pushing its sum + into the boundary all-reduce) or dense MLP (its down projection pushing into the post-MLP boundary) -> MoE / dense + boundary (all-reduce + shared/routed add + residual + the next layer's RMSNorm, emitting the next QKV's MXFP8 + activation). An Eagle3 capture layer runs the same decode layer and captures its feed-forward boundary's reduced + residual sum (the boundary writing the capture norm itself when the worker gives it a target). The first layer + runs the generic layer when the fused embedding does not take the step. A wider step (or one carrying embeddings) + runs every layer in its generic form, with this target's attention (the dense layers' full-context decode, the + decode selection and sparse decode where the producer wrote the caches). + """ + + def __init__(self, core: MiniMaxM3Tp8ModelingV2) -> None: + self.core = core + self._moe_push_descs: Dict[tuple, Any] = {} + # Set by a feed-forward boundary that wrote the layer's Eagle3 capture (see _layer's ``capture``). + self._capture_written = False + # Off unless TRTLLM_MODELING_V2_VALIDATE asks for it; read once here rather than per forward. + self._contract_pending = step_contract_enabled() + + def _check_step_contract(self, md) -> None: + """Opt-in first-forward fail-fast (TRTLLM_MODELING_V2_VALIDATE): the metadata fields this target consumes + must exist (they are private trtllm surface). Calling ``step_args`` is the check; once per target instance.""" + if not self._contract_pending: + return + try: + self.step_args(md) + except AttributeError as exc: + raise AssertionError(f"attention metadata surface drifted: {exc}") from exc + self._contract_pending = False + + def step_args(self, md) -> dict: + """The one projection of the MSA metadata this target's forward reads, computed once per step. + + States the two context counts instead of reading them back: this target runs only when ``num_contexts`` is + zero, so reading them would ask the metadata to confirm what the routing already decided. + + Graph classes: counts, the generation span, the identity-selection flag, the worst-case score extent and the + cache length bound are host values fixed per capture (prepare() stages them before the forward); the block + table, lengths, cache slots, identity table and valid-block counts are the metadata's persistent buffers, held + by reference and refreshed in place every step; ``msa_idx_k_cache`` / ``msa_proxy_max_score_view`` are the + metadata's getters of the per-layer index-K cache and of the score scratch view. ``metadata`` is the prepared + object itself: the MSA backend's step (``m3_msa_attention``) and the full-context decode's K/V write + (``write_msa_phase_kv``) take it as their argument. + """ + return dict( + num_contexts=0, + num_ctx_tokens=0, + num_tokens=md.num_tokens, + num_seqs=md.num_seqs, + max_seq_len=md.max_seq_len, + kv_cache_manager=md.kv_cache_manager, + msa_idx_k_cache=md.msa_idx_k_cache, + msa_out_cache_loc=md.msa_out_cache_loc, + msa_decode_span=md.msa_decode_span, + msa_block_table=md.msa_block_table, + msa_seq_lens_cuda=md.msa_seq_lens_cuda, + msa_short_ok=md._msa_short_ok, + msa_short_table=md.msa_short_table, + msa_proxy_max_score_view=md.msa_proxy_max_score_view, + msa_worst_case_max_k_tiles=md.msa_worst_case_max_k_tiles, + msa_n_valid_blocks=md.msa_n_valid_blocks, + metadata=md, + ) + + def forward( + self, + attn_metadata, + input_ids, + position_ids, + inputs_embeds, + spec_metadata, + capture, + capture_target, + ): + core = self.core + self._check_step_contract(attn_metadata) + step = self.step_args(attn_metadata) + # The decode kernels' rows (a per-capture constant): up to DECODE_STEP_MAX tokens of 1-D input ids. + kernels = ( + inputs_embeds is None and input_ids.ndim == 1 and is_decode_step(input_ids.numel()) + ) + if kernels: + num_tokens = int(input_ids.numel()) + plan = plan_for(num_tokens) + _record("decode", num_tokens, plan.name) + if plan.name == "wide": + taken("VR-BANDS wide decode step", num_tokens) + hidden, residual = self._embed(input_ids, num_tokens) + else: + if is_in_breakable_cuda_graph(): + raise RuntimeError( + "this MiniMax-M3 target does not run under piecewise (breakable) CUDA graphs" + ) + plan = None + _record("decode", int(step["num_tokens"]), "generic") + hidden = ( + m3_embedding(input_ids, core.w["embed"]) if inputs_embeds is None else inputs_embeds + ) + residual = None + for i in range(core.num_layers): + captured = spec_metadata is not None and spec_metadata.is_layer_capture(i) + if kernels and residual is not None and captured: + # The capture layer on the decode layer (its boundaries push); the capture reads the MoE / dense + # boundary's reduced residual sum, as the generic layer's fused boundary capture does. Given a target + # (``CAPTURE_NORM_IN_BOUNDARY``), the boundary all-reduce writes the capture norm itself. + target = ( + capture_target(spec_metadata, i, num_tokens) + if capture_target is not None + else None + ) + self._capture_written = False + hidden, residual = self._layer( + i, plan, position_ids, hidden, residual, step, capture=target + ) + taken("capture_layer_fast", num_tokens, f"layer {i}") + if target is not None and self._capture_written: + taken("capture_norm_in_boundary", num_tokens, f"layer {i}") + capture(spec_metadata, i, residual, None, written=True) + else: + capture(spec_metadata, i, residual, None) + elif not kernels or residual is None or captured: + hidden, residual = self._generic_layer( + i, position_ids, hidden, residual, step, spec_metadata, capture + ) + else: + hidden, residual = self._layer(i, plan, position_ids, hidden, residual, step) + return hidden + + def _embed(self, input_ids: torch.Tensor, num_tokens: int): + """(hidden, residual) of the fused embedding + first RMSNorm; (embedding, None) when it does not run, so the + first layer runs its own input norm. SM107: 1.55..1.65 us against 3.4..6.0 us for the stock embedding + + RMSNorm at 1..8 rows (the gather bitwise, the norm one ulp off on ~5e-6 of the elements: summation order).""" + core = self.core + if not EMBED_FIRSTNORM.covers(num_tokens): + uncovered(EMBED_FIRSTNORM.path, f"more than {EMBED_FIRSTNORM_MAX_TOKENS} tokens") + return m3_embedding(input_ids, core.w["embed"]), None + owner = core.embed_owners.get((num_tokens, input_ids.dtype)) + if owner is None or not input_ids.is_contiguous(): + missed( + EMBED_FIRSTNORM.path, + num_tokens, + f"no embedding owner for {input_ids.dtype} ids (built for int32 / int64 after the load)" + if owner is None + else "the input ids are not contiguous", + ) + return m3_embedding(input_ids, core.w["embed"]), None + _record("m3_embedding_firstnorm", num_tokens, input_ids.dtype) + taken(EMBED_FIRSTNORM.path, num_tokens, str(input_ids.dtype)) + return m3_embedding_firstnorm( + input_ids, owner.table, owner.weight, owner.eps, owner.raw, owner.output, owner.use_pdl + ) + + def _bucket_flags(self, num_tokens: int): + """(bucket workspace, its 8-CTA call counters) of the boundary all-reduce for ``num_tokens``, or None.""" + workspace = m3_mc_allreduce.get_workspace(self.core.mapping, self.core.hidden) + workspace = None if workspace is None else m3_mc_allreduce._bucket(workspace, num_tokens) + flags = None if workspace is None else workspace.flags_for(8) + return None if flags is None else (workspace, flags) + + @staticmethod + def _no_bucket(path: str, num_tokens: int) -> None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, num_tokens, "the multicast all-reduce workspace") + else: + missed(path, num_tokens, "no multicast all-reduce bucket holds the step") + + def _attn_push_target(self, num_tokens: int): + """The push tuple into the attention boundary's all-reduce (the o_proj's; also the dense down projection's into + the post-MLP boundary, the same exchange), or None (after a miss). + + None comes only from the token count or from workspace state created collectively in the same eager call on + every rank, so all ranks agree. (A lone non-pushing rank would still be safe: its all-reduce pushes the same + bf16 rows into the same set A.)""" + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(ATTN_PUSH.path, num_tokens) + return None + workspace, flags = found + return ( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + AR_PUSH_MODE, + ) + + def _moe_push_targets(self, num_tokens: int): + """(engine push descriptor, shared down-projection push tuple) for the MoE boundary's exchange sets, or None + (after a miss). + + This decision must match across ranks: prepushed 3 reads set B of every rank, which a non-pushing rank never + fills. None comes only from the token count or from state created in the same eager call on every rank + (workspace, flags, the descriptor cache), and the engine it is armed for is agreed at build time + (``_prebuild_engines``), so it does.""" + hidden = self.core.hidden + found = self._bucket_flags(num_tokens) + if found is None: + self._no_bucket(MOE_PUSH.path, num_tokens) + return None + workspace, flags = found + key = (workspace.mc.data_ptr(), flags.data_ptr()) + desc = self._moe_push_descs.get(key) + if desc is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(MOE_PUSH.path, num_tokens, "the MoE engine's push descriptor") + return None + desc = self._moe_push_descs[key] = ar_push_desc( + workspace.mc.data_ptr(), + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + flags.device, + ) + set_b = workspace.mc.data_ptr() + 2 * 4 * workspace.max_tokens * workspace.world_size * ( + hidden // 2 + ) + return desc, ( + set_b, + flags.data_ptr(), + workspace.rank, + workspace.world_size, + workspace.max_tokens, + 2, + ) + + def _boundary(self, use_mc: bool, x, residual, norm_w, x2=None, quant=False, capture=None): + """All-reduce + (second addend) + residual + RMSNorm: the multicast kernel, else the built-in all-reduce. + ``capture``: the Eagle3 capture the multicast kernel also writes (``_capture_written`` records it).""" + core = self.core + if use_mc: + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w, quant=quant, capture=capture) + if fused is not None: + if capture is not None: + self._capture_written = True + taken( + MC_ALLREDUCE.path, + x.shape[0], + ("x + x2" if x2 is not None else "x") + (", MXFP8 copy" if quant else ""), + ) + return fused + workspace = m3_mc_allreduce.get_workspace(core.mapping, x.shape[-1]) + missed( + MC_ALLREDUCE.path, + x.shape[0], + m3_mc_allreduce.decline_reason(x, residual, workspace, x2=x2), + ) + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _prepushed( + self, i: int, x, residual, norm_w, quant, prepushed, source, x2=None, capture=None + ): + fused = _mc_allreduce_norm( + self.core, x, x2, residual, norm_w, quant=quant, prepushed=prepushed, capture=capture + ) + if fused is None: + raise RuntimeError(f"layer {i}: rows were pushed into an all-reduce that did not run") + if capture is not None: + self._capture_written = True + taken(MC_ALLREDUCE.path, x.shape[0], f"prepushed {prepushed}, {source}") + return fused + + def _layer(self, i: int, plan: DecodePlan, position_ids, hidden, residual, step, capture=None): + """Layer ``i`` on the decode kernels. ``capture``: (out, fc_norm weight, eps) for the feed-forward boundary to + write the layer's Eagle3 capture norm (``CAPTURE_NORM_IN_BOUNDARY``); a boundary that writes it sets + ``_capture_written``.""" + core = self.core + num_tokens = hidden.shape[0] + # Padded rows: the boundary all-reduce polls and re-arms rows t < M = residual.shape[0], CUDA-graph padding + # rows included. Every producer pushes all rows of the same tensor: the o_proj / shared down / dense down GEMV + # t < m = x.shape[0] (its row count comes from the host shape, not a live count), the MoE engine every row of + # the num_tokens it was built for (= num_tokens here). The all-reduce checks x.sizes() == residual.sizes(), so + # no padded row is left with a stale flag. + # The GEMV dispatch (and with it every push into a boundary all-reduce) is a decode-graph one. + decode_graph = flashinfer_mxfp8_decode_graph_capture_active() + if not decode_graph: + uncovered( + "decode_graph", + "an eager decode step: the decode-graph GEMV dispatch and the boundary pushes are off", + ) + push = plan.attn_mc_ar and decode_graph and ATTN_PUSH.covers(num_tokens) + armed = self._attn_push_target(num_tokens) if push else None + hidden, attn_pushed = self._attention(i, position_ids, hidden, step, armed) + post_norm = core.w[f"l{i}_post_norm"] + if armed is not None and attn_pushed: + hidden, residual = self._prepushed( + i, hidden, residual, post_norm, False, AR_PUSH_MODE, "o_proj push" + ) + else: + if armed is not None: + missed( + ATTN_PUSH.path, + num_tokens, + "the o_proj did not push its rows into the armed boundary all-reduce", + ) + # R1VR8: past the TP2 rows the attention boundary also publishes the gate_up projections' MXFP8 + # activation (the shared / dense gate_up GEMVs at 9..32 rows); up to them the step is unchanged. + hidden, residual = self._boundary( + plan.attn_mc_ar, hidden, residual, post_norm, quant=num_tokens > TP2_ROWS + ) + + norm_w = _next_norm(core, i) + if i not in core.moe_layers: + # A dense layer: its MLP's down projection pushes into the post-MLP boundary's all-reduce, as the o_proj + # does into the attention one (the same exchange; the gate_up projection never pushes). SM107 TP2: the + # two non-capture dense layers' post-MLP hops end 5.5-5.8 us after the down GEMV unpushed vs 2.4-2.5 us + # pushed; bitwise equal to the unpushed boundary (the same bf16 words). + # R6: the next layer's QKV takes its MXFP8 activation from this boundary, as after a MoE layer. + quant = R6_DENSE_PREQUANT and plan.qkv_prequant and i + 1 < core.num_layers + if plan.qkv_prequant and i + 1 < core.num_layers: + _log_once( + logger.info, + "r6_dense_prequant", + "on" if quant else "off", + "[M3 dispatch] r6_dense_prequant " + + ( + "taken: the dense-MLP boundary hands the next QKV its MXFP8 activation" + if quant + else "off (VR default; TRTLLM_M31_ENABLE_R6=1 turns it on), BF16 hand-off" + ) + + " (logged once)", + ) + dense_push = plan.attn_mc_ar and decode_graph and DENSE_PUSH.covers(num_tokens) + armed = self._attn_push_target(num_tokens) if dense_push else None + out, pushed = self._swiglu_mlp(hidden, f"l{i}_mlp", True, armed, dense=True) + if armed is not None and pushed: + return self._prepushed( + i, + out, + residual, + norm_w, + quant, + AR_PUSH_MODE, + "dense down push", + capture=capture, + ) + if armed is not None: + missed( + DENSE_PUSH.path, + num_tokens, + "the dense MLP's down projection did not push its rows into the armed boundary all-reduce", + ) + return self._boundary( + plan.attn_mc_ar, out, residual, norm_w, quant=quant, capture=capture + ) + + quant = plan.qkv_prequant and i + 1 < core.num_layers + moe_push = None + if moe_path(num_tokens) == MOE_ENGINE.path: + # The engine runs inside the MoE; looked up (or found missing) here first so a push is armed only for an + # engine that will take it. + engine = core.engine(i, num_tokens, shared=True) + if ( + engine is not None + and plan.moe_mc_ar + and decode_graph + and MOE_PUSH.covers(num_tokens) + ): + moe_push = self._moe_push_targets(num_tokens) + shared, routed, engine_pushed, shared_pushed = self._moe( + i, hidden, return_parts=True, push=moe_push + ) + if moe_push is not None: + if routed is None: + # The engine folded the shared expert in and pushed routed + shared into set A. + if engine_pushed: + return self._prepushed( + i, shared, residual, norm_w, quant, 2, "MoE engine push", capture=capture + ) + else: + if engine_pushed and shared_pushed: + return self._prepushed( + i, + shared, + residual, + norm_w, + quant, + 3, + "MoE engine + shared push", + x2=routed, + capture=capture, + ) + if engine_pushed or shared_pushed: + raise RuntimeError("only one MoE part pushed into the boundary all-reduce") + missed( + MOE_PUSH.path, + num_tokens, + "the MoE did not push its output into the armed boundary all-reduce", + ) + return self._boundary( + plan.moe_mc_ar, shared, residual, norm_w, x2=routed, quant=quant, capture=capture + ) + + def _swiglu_mlp( + self, + h: torch.Tensor, + prefix: str, + low_m_gemm: bool, + push: Optional[tuple] = None, + dense: bool = False, + ): + """A gated MXFP8 MLP (the dense MLP, the shared expert): gate_up, SwiGLU-OAI (the dense MLP's on this target's + C++ op, ``_dense_swiglu``), down, no reduction. Returns (output, whether the down projection's GEMV pushed into + ``push``).""" + core = self.core + gu, _ = _linear(core, h, f"{prefix}_gu_w", f"{prefix}_gu_s", low_m_gemm) + n, k = core.w[f"{prefix}_dn_w"].shape + rows = gu.shape[0] + if not dense and rows in SWQ_ROWS.get((n, k), ()) and _mx_mode(core.mxfp8).decode_graph: + # The MXFP8 activation is published keyed by ``gu`` and taken by the down projection's prequant branch. + act, act_scale = m3_swq(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + m3_prequant.SLOT.publish(gu, act, act_scale) + if rows not in _SWQ_LOGGED: + _SWQ_LOGGED.add(rows) + logger.info(f"[M3 dispatch] SWQ: SwiGLU + MXFP8 producer rows={rows} [{n}, {k}]") + return _linear(core, gu, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm, push) + if dense: + act = _dense_swiglu(core, gu) + else: + act = m3_silu_and_mul(gu, core.swiglu_limit, core.swiglu_alpha, core.swiglu_beta) + return _linear(core, act, f"{prefix}_dn_w", f"{prefix}_dn_s", low_m_gemm, push) + + def _routed_chunks(self, i: int, x, router_logits): + """The routed experts from the bf16 activation over at most ``moe_max_num_tokens`` rows per runner call: the MoE + scheduler's even split, even chunks on the MoE chunking stream.""" + core = self.core + rows = x.shape[0] + num_chunks = (rows + core.moe_max_num_tokens - 1) // core.moe_max_num_tokens + if num_chunks <= 1: + return _routed_experts(core, i, x, None, router_logits, True) + sizes = _split_chunk(rows, num_chunks) + xs, logits = x.split(sizes), router_logits.split(sizes) + aux = core.aux_stream_moe_chunk + ev_main, ev_overlap = core.moe_chunk_events[i] + if aux is not None: + ev_main.record() + with torch.cuda.stream(aux): + ev_main.wait() + outs: List[torch.Tensor] = [] + for c in range(num_chunks): + if aux is not None and c % 2 == 0: + with torch.cuda.stream(aux): + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True)) + else: + outs.append(_routed_experts(core, i, xs[c], None, logits[c], True)) + if aux is not None: + with torch.cuda.stream(aux): + ev_overlap.record() + ev_overlap.wait() + return torch.cat(outs) + + def _routed(self, i: int, h: torch.Tensor): + """Router + routed experts, finalized: the router logits (the TF32 tensor-core router on ``ROUTER_TC_GEN``'s + rows, else the fp32 router) and the experts quantizing their bf16 input (chunked).""" + return self._routed_chunks(i, h, _decode_router_logits(self.core, i, h)) + + def _moe(self, i: int, h: torch.Tensor, *, return_parts: bool, push=None): + """The MoE block of layer ``i`` without its reduction. + + ``push`` = (the engine's ``arp`` descriptor, the shared down projection's ``ar`` tuple) the decode layer armed + for the MoE boundary's exchange sets, or None. Returns ``(out, routed, engine_pushed, shared_pushed)``: with + ``return_parts`` the shared and routed parts (routed None when the engine folded the shared expert in), else + their sum in ``out``. + """ + core = self.core + engine_arp, shared_ar = push if push is not None else (None, None) + rows = h.shape[0] if h.dim() == 2 else 0 + if ( + h.dim() == 2 + and h.dtype == torch.bfloat16 + and 1 <= rows <= MOE_ENGINE_MAX_TOKENS + and rows >= MOE_FOLD_MIN_TOKENS + ): + engine = core.engine(i, rows, shared=True) + if engine is not None: + _record("m3_moe_engine", rows, engine_arp is not None) + taken(MOE_ENGINE.path, rows) + folded = m3_moe_engine(h.contiguous(), engine, None, engine_arp) + return folded, None, engine_arp is not None, False + routed_out, (shared_out, shared_pushed) = maybe_execute_in_parallel( + lambda: self._routed(i, h), + lambda: self._swiglu_mlp(h, f"l{i}_sh", False, shared_ar), + core.moe_events[i][0], + core.moe_events[i][1], + # The warmup tuning pass profiles the shared expert's GEMMs: keep it on the main stream, between the same + # collectives on every rank (the capture pass and serving still overlap it on the aux stream). + None if flashinfer_mxfp8_autotune_active() else core.aux_stream_moe, + disable_on_compile=True, + ) + if return_parts: + return shared_out, routed_out, False, shared_pushed + # In-place add into the shared output to avoid allocating a temporary. + return shared_out.add_(routed_out), None, False, shared_pushed + + def _generic_boundary(self, x, x2, residual, norm_w): + """A generic layer boundary: the multicast all-reduce + residual + RMSNorm on the rows it takes (with the + shared + routed add folded in), else the built-in fused all-reduce.""" + core = self.core + if _can_mc(x, folded=x2 is not None): + fused = _mc_allreduce_norm(core, x, x2, residual, norm_w) + if fused is not None: + return fused + if x2 is not None: + x = x.add_(x2) + return _native_allreduce_norm(core, x, residual, norm_w) + + def _generic_layer(self, i: int, position_ids, hidden, residual, step, spec_metadata, capture): + """Layer ``i`` in its generic form: input norm (first layer without the fused embedding), attention, the + attention boundary, the MoE (or the dense MLP) and the boundary into the next layer's input norm, every + collective fused with the residual add and the norm. An Eagle3 capture layer captures the boundary's reduced + residual sum.""" + core = self.core + if residual is None: + residual = hidden + hidden = flashinfer_rmsnorm(hidden, core.w[f"l{i}_in_norm"], core.rms_eps) + hidden, _ = self._attention(i, position_ids, hidden, step) + if _can_mc(hidden): + fused = _mc_allreduce_norm(core, hidden, None, residual, core.w[f"l{i}_post_norm"]) + hidden, residual = ( + fused + if fused is not None + else _native_allreduce_norm(core, hidden, residual, core.w[f"l{i}_post_norm"]) + ) + else: + hidden, residual = _native_allreduce_norm( + core, hidden, residual, core.w[f"l{i}_post_norm"] + ) + if i in core.moe_layers: + out, routed, _, _ = self._moe(i, hidden, return_parts=_can_mc(hidden, folded=True)) + else: + out, _ = self._swiglu_mlp(hidden, f"l{i}_mlp", True, dense=True) + routed = None + normed, updated = self._generic_boundary(out, routed, residual, _next_norm(core, i)) + if spec_metadata is not None and spec_metadata.is_layer_capture(i): + capture(spec_metadata, i, updated, None) + return normed, updated + + def _attention(self, i: int, position_ids, hidden, step, push: Optional[tuple] = None): + """Attention of layer ``i`` up to its o_proj (no reduction). ``push`` is the attention boundary's exchange the + o_proj may write into. Returns (o_proj output, pushed).""" + core = self.core + if i not in core.sparse_layers: + return self._dense_attention(i, position_ids, hidden, step, push) + if not core.fused_projection: + return self._unfused_sparse_attention(i, position_ids, hidden, step, push) + # #117: the QKV GEMV may be deferred to the producer call site (``QKV_PRODUCER_FOLD``). + packed, _ = _linear( + core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True, defer=QKV_PRODUCER_FOLD + ) + deferred = None + if isinstance(packed, _DeferredGemv): + deferred, packed = packed, packed.placeholder + horizontal = _sparse_qkv_producer( + core, + i, + packed, + position_ids, + step["kv_cache_manager"], + step["msa_idx_k_cache"], + step["msa_out_cache_loc"], + deferred, + ) + if horizontal is None: + raise RuntimeError( + f"MiniMax-M3 sparse layer {i}: no horizontal producer takes this step's caches (an FP8 or NVFP4 HND " + "pool with an FP8 index-K cache)" + ) + q, idx_q = horizontal + o = self._attention_core(i, q, None, None, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + + def _dense_attention(self, i: int, position_ids, hidden, step, push): + """Dense layers 0-2: QKV, the fused norm + RoPE (FP8 q/k/v: an FP8 cache, or the FP8 pool the dense layers sit + in within a hybrid NVFP4 cache on a decode step), the attention core, o_proj. + + On a covered pure generation step (``_fullctx_plan``) the attention is this target's PrimTS split-KV decode + over the layer's full causal context, and the norm/RoPE op also stores the step's new K/V rows into the + layer's paged pool and the query compact, as the decode reads it (``DENSE_KV_INSERT_FUSED``; the bytes + ``write_msa_phase_kv`` stores). Otherwise the MSA backend.""" + core = self.core + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fp8 = core.main_kv_is_fp8 or core.main_kv_is_nvfp4 + plan = self._fullctx_plan(i, qkv.device, step) + kv_dest = None + if ( + plan is not None + and fp8 + and DENSE_KV_INSERT_FUSED + and _dense_fp8_op_applies(qkv, position_ids) + and hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv") + ): + kv_dest = ( + step["kv_cache_manager"].get_buffers(i, kv_layout="HND"), + step["msa_out_cache_loc"], + plan[1], + torch.empty( + (qkv.shape[0], core.heads, core.head_dim), + dtype=torch.float8_e4m3fn, + device=qkv.device, + ), + ) + if fp8: + fused = _dense_qk_norm_rope_fp8(core, i, qkv, position_ids, kv_dest) + if kv_dest is not None: + taken(DENSE_KV_INSERT_PATH, kv_dest[2]) + else: + fused = _qk_norm_rope(core, i, qkv, position_ids, False) + q, k, v = fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + if kv_dest is not None: + q = kv_dest[3].view(q.shape[0], -1) + o = None + if plan is not None: + o = self._fullctx_decode(i, q, k, v, step, plan, kv_written=kv_dest is not None) + if o is None: + o = self._attention_core(i, q, k, v, None, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + + def _fullctx_plan(self, i: int, device: torch.device, step): + """(the PrimTS split-KV decode plan, the step's query tokens) where dense layer ``i``'s decode attention runs + on ``kernels/m3_fullctx_decode`` (``FULLCTX_DECODE``): a covered (query rows per request, requests) shape over + the layer's FP8 pool. None otherwise (the MSA backend runs). + + Rubin TP2, FP8 KV, 165k context, graph-us per call (stock trtllm-gen in brackets): (4, 1) 18.8 (29.3), (1, 1) + 15.7 (28.0), (4, 2) 24.5 (29.3), (1, 2) 20.2 (28.1); the stock split is capped at 16 cluster CTAs per (request, + KV head), i.e. 32 of 212 CTAs at one request. With the BF16 split-KV merge (``FULLCTX_DECODE_OWN_REDUCER``): + (1, 1) 14.5 vs 18.5, (4, 1) 17.3 vs 21.3, (1, 2) 18.5 vs 23.1, (4, 2) 23.3 vs 27.5.""" + path = FULLCTX_DECODE.path + span = step["msa_decode_span"] + if span is None or span.row_first != 0: + uncovered(path, "no decode span starting at row 0") + return None + nreq = int(step["num_seqs"]) + qlen = int(span.query_len) + if not FULLCTX_DECODE.covers((qlen, nreq)): + uncovered(path, f"{qlen} query rows x {nreq} requests is not a covered shape") + return None + if int(step["max_seq_len"]) > FULLCTX_DECODE_MAX_KV_LEN: + uncovered(path, "the cache length bound is beyond the plans' K/V capacity") + return None + num_tokens = nreq * qlen + pool, _ = step["kv_cache_manager"].get_kv_subpage_pool(i, "HND") + if pool.dtype != torch.float8_e4m3fn: + missed(path, num_tokens, "the dense layer's KV pool is not FP8") + return None + from .kernels import m3_fullctx_decode as _fullctx_decode_kernel + + own_reducer = FULLCTX_DECODE_OWN_REDUCER + if own_reducer and not _fullctx_decode_kernel.has_own_reducer(): + missed(path, num_tokens, "the BF16 split-KV merge op is not built") + return None + # Compiled before warmup (``prebuild_kernels``); the forward never compiles one. + plan = _fullctx_decode_kernel.plan_for(device, qlen, nreq, own_reducer) + if plan is None: + missed(path, num_tokens, "the PrimTS decode plan is not prebuilt") + return None + return plan, num_tokens + + def _fullctx_decode(self, i: int, q, k, v, step, plan_tokens, *, kv_written: bool): + """Dense layer ``i``'s decode attention on the PrimTS plan of ``_fullctx_plan``: the step's new K/V go into the + paged cache as the MSA backend writes them (``write_msa_phase_kv``; already there when ``kv_written``), then + every query row attends its request's full causal context.""" + from tensorrt_llm._torch.attention.backends.interface import AttentionInputType + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + write_msa_phase_kv, + ) + + from .kernels import m3_fullctx_decode as _fullctx_decode_kernel + + core = self.core + plan, num_tokens = plan_tokens + nreq = int(step["num_seqs"]) + if not kv_written: + write_msa_phase_kv( + core.attn[i], + k[:num_tokens], + v[:num_tokens], + step["metadata"], + AttentionInputType.mixed, + token_offset=0, + ) + pool, subpages = step["kv_cache_manager"].get_kv_subpage_pool(i, "HND") + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + k_pages, v_pages = _fullctx_decode_kernel.kv_page_views(pool, subpages) + # The merge also writes the o_proj's MXFP8 hand-off (``FULLCTX_DECODE_MXFP8``) where its TP8 copy runs. + mxfp8 = None + if ( + FULLCTX_DECODE_MXFP8 + and output.shape[0] == num_tokens + and _fullctx_decode_kernel.emits_mxfp8() + ): + merge_mxfp8 = _sparse_decode_kernel.MERGE_MXFP8 + if _sparse_decode_kernel.MERGE_MXFP8_MIN_TOKENS <= num_tokens <= merge_mxfp8.MAX_ROWS: + mxfp8 = merge_mxfp8.buffers(output.shape[1], output.device) + _record("m3_fullctx_decode", num_tokens, nreq, kv_written, mxfp8 is not None) + emitted = plan.run( + fp8_e4m3_saturate(q[:num_tokens].reshape(num_tokens, core.heads, core.head_dim)), + k_pages, + v_pages, + step["msa_block_table"][:nreq], + step["msa_seq_lens_cuda"][:nreq], + core.sm_scale(i), + output[:num_tokens], + mxfp8=mxfp8, + ) + if emitted: + m3_prequant.SLOT.publish(output, mxfp8[0][:num_tokens], mxfp8[1]) + taken(FULLCTX_DECODE_MXFP8_PATH, num_tokens) + taken(FULLCTX_DECODE.path, num_tokens) + return output + + def _unfused_sparse_attention(self, i: int, position_ids, hidden, step, push): + """A sparse layer without the five-way projection (``fuse_qkv_index_projection=False``): the QKV projection + and its fused norm + RoPE (FP8 q/k/v into an FP8 cache) on the current stream; the MXFP8 index projection + (the checkpoint's block scales applied) and the FP8 indexer kernel on the attention aux stream; then the MSA + backend (it writes the main KV cache) and the o_proj. The NVFP4 producer does not apply: it packs the + five-way projection's output. ``push`` is the attention + boundary exchange the o_proj GEMV writes its rows into, as in the five-way path.""" + core = self.core + if _producer_covered(core): + # Not a miss: the producer packs the five-way projection's output, which this configuration lacks. + uncovered( + PRODUCER.path, + "fuse_qkv_index_projection is disabled: QKV + separate MXFP8 index projection", + ) + + def _main(): + qkv, _ = _linear(core, hidden, f"l{i}_qkv_w", f"l{i}_qkv_s", True) + fused = _qk_norm_rope(core, i, qkv, position_ids, core.main_kv_is_fp8) + return fused.split([core.q_size, core.kv_size, core.kv_size], dim=-1) + + def _index(): + idx_qk, _ = _linear(core, hidden, f"l{i}_idx_w", f"l{i}_idx_s", True) + return _fp8_index_qk_norm_rope( + core, i, idx_qk, position_ids, step["msa_idx_k_cache"], step["msa_out_cache_loc"] + ) + + (q, k, v), idx_q = maybe_execute_in_parallel( + _main, + _index, + core.attn_events[i][0], + core.attn_events[i][1], + core.aux_stream_attn, + disable_on_compile=True, + ) + o = self._attention_core(i, q, k, v, idx_q, step) + return _linear(core, o, f"l{i}_o_w", f"l{i}_o_s", True, push) + + def _attention_core(self, i: int, q, k, v, idx_q, step) -> torch.Tensor: + """The attention core over the step's live tokens (a padded step's pad rows are left as allocated): a sparse + layer whose caches the producer wrote runs this target's selection and sparse decode when the generation span + starts at row 0; everything else the MSA backend (``m3_msa_attention``: the KV write unless a producer wrote the + caches, the sparse layers' top-k block selection, the FMHA).""" + core = self.core + output = q.new_empty((q.shape[0], core.heads * core.head_dim), dtype=torch.bfloat16) + live = int(step["num_tokens"]) + q_live, out_live = q[:live], output[:live] + k_live = k[:live] if k is not None else None + v_live = v[:live] if v is not None else None + idx_q_live = idx_q[:live] if idx_q is not None else None + span = step["msa_decode_span"] + sparse = i in core.sparse_layers + if sparse and k_live is None and span is not None and span.row_first == 0: + self._decode_sparse_core(i, q_live, idx_q_live, step, out_live) + return output + nvfp4 = core.main_kv_is_nvfp4 + _record("m3_msa_attention", sparse, k_live is not None) + m3_msa_attention( + core.attn[i], + q_live, + k_live, + v_live, + idx_q_live, + step["metadata"], + out_live, + sparse, + core.inv_kv_scales[i] if nvfp4 else None, + core.kv_scales[i] if nvfp4 else None, + ) + return output + + def _index_scores(self, i: int, idx_q, step, num_tokens: int, qlen: int): + """The per-index-head block maxima of a non-identity decode step: this target's CuTe DSL scorer at the request + counts ``INDEX_SCORE`` covers (each count's CTAs per request and early issue, ``index_score_schedule``), the + in-tree CuTe DSL scorer beyond them. Rubin TP2 (2 index heads), 165,521 tokens, behind a 58 MB PDL producer, + per layer: 1 request 3.27 us at 1536 CTAs (in-tree 256-CTA scorer 6.07); 2 requests 4.97 us at 768 (in-tree + 6.71).""" + config = self.core.attn[i].m3_config + rows = int(step["num_seqs"]) + block_table = step["msa_block_table"][:rows] + seq_lens = step["msa_seq_lens_cuda"][:rows] + idx_q_view = idx_q.view(num_tokens, config.num_index_heads, config.sparse_index_dim) + idx_k_cache = step["msa_idx_k_cache"](i) + max_score = step["msa_proxy_max_score_view"]( + config.num_index_heads, step["msa_worst_case_max_k_tiles"], num_tokens + ) + if INDEX_SCORE.covers(rows): + split_k, early = index_score_schedule(rows) + key = ( + idx_q_view.dtype, + int(config.num_index_heads), + int(qlen), + int(idx_q_view.shape[-1]), + int(split_k), + bool(early), + ) + if key not in _index_score_kernel.IndexScoreRunner.kernel_cache: + # Outside require mode the op below compiles it (the only scorer this step has). + missed( + INDEX_SCORE_PATH, + num_tokens, + f"the index scorer {key} was not compiled before warmup", + ) + _record("m3_index_score", key) + m3_index_score( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens, + max_score.transpose(1, 2), + qlen, + split_k, + early, + ) + taken(INDEX_SCORE_PATH, rows) + else: + uncovered( + INDEX_SCORE_PATH, + f"{rows} requests, past {INDEX_SCORE_MAX_REQUESTS}: the in-tree CuTe DSL scorer", + ) + _in_tree_scorer() # registers the op + _record("cute_dsl_minimax_m3_index_decode_score", rows, qlen) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens, + max_score.transpose(1, 2), + qlen, + ) + return max_score + + def _select(self, i: int, max_score, step, num_tokens: int): + """The top-16 table of a non-identity decode step from the block maxima reduced to the KV head: the long-row + select (the 384-thread kernel, ``SELECT_LONG_DSL``, when compiled) on rows of ``LONG_SELECT_MIN_BLOCKS.. + LONG_SELECT_MAX_BLOCKS`` blocks, else the built-in selector (``m3_select_blocks``).""" + config = self.core.attn[i].m3_config + n_valid_blocks = step["msa_n_valid_blocks"][:num_tokens] + group = config.num_index_heads // config.num_kv_heads + max_score_kv = ( + max_score.view(config.num_kv_heads, group, max_score.shape[1], max_score.shape[2]).amax( + dim=1 + ) + if group > 1 + else max_score + ) + from .kernels import m3_select_long_dsl as _select_long_dsl_kernel + + num_blocks = int(max_score_kv.shape[1]) + table = None + dsl = False + if long_select_covers(num_blocks): + early_trigger = num_tokens <= SELECT_EARLY_MAX_ROWS + if LONG_SELECT_DSL.keys: + # Compiled before warmup (prebuild_kernels); a missing kernel falls back to the 1024-thread one. + key = _select_long_dsl_kernel.kernel_key( + get_env_enable_pdl(), early_trigger, SELECT_LONG_TRIGGER_PHASE + ) + dsl = key in _select_long_dsl_kernel.kernel_cache + if not dsl: + missed( + LONG_SELECT_DSL.path, + num_tokens, + f"the 384-thread select {key} was not compiled before warmup", + ) + _record("long_select", num_blocks, num_tokens, dsl) + table = long_select_blocks( + max_score_kv, + n_valid_blocks=n_valid_blocks, + init_blocks=config.init_blocks, + local_blocks=config.local_blocks, + use_pdl=get_env_enable_pdl(), + early_trigger=early_trigger, + trigger_phase=SELECT_LONG_TRIGGER_PHASE, + dsl=dsl, + ) + if table is not None: + taken((LONG_SELECT_DSL if dsl else LONG_SELECT).path, num_tokens) + else: + long_select_fallback(num_blocks, num_tokens) + _record("m3_select_blocks", num_blocks, num_tokens) + table = m3_select_blocks( + max_score_kv, + MSA_REQUIRED_TOPK, + n_valid_blocks, + config.init_blocks, + config.local_blocks, + True, + ) + return table + + def _decode_sparse_core(self, i: int, q, idx_q, step, output) -> None: + """Top-16 block selection and the block-sparse decode of one pure generation step of a sparse layer whose caches + the producer already wrote. + + The selection is the identity when the metadata staged it (every request within 16 blocks). Otherwise the + index scorer writes the per-index-head block maxima, reduced to the KV head, and the long-row selector takes the + top 16 (the built-in selector beyond its shapes). The long-context decode (``m3_sparse_decode`` on + ``kernels/m3_attn_lc8``) prefetches into L2 this layer's o_proj weights (``SPARSE_DECODE_LC8_PREFETCH_OPROJ``) + and the next sparse layer's K/V pages of the blocks that layer selected one step back + (``SPARSE_DECODE_LC8_PREFETCH_NEXT_KV``, ``_next_kv_prefetch``); hints only. + """ + from .kernels import m3_attn_lc8 as _attn_lc8_kernel + + core = self.core + backend = core.attn[i] + num_tokens = int(q.shape[0]) + rows = int(step["num_seqs"]) + qlen = int(step["msa_decode_span"].query_len) + block_table = step["msa_block_table"][:rows] + seq_lens = step["msa_seq_lens_cuda"][:rows] + num_kv_heads = int(backend.indexer.config.num_kv_heads) + # The index scorers are compiled and loaded by prebuild_kernels, so no step compiles one and the identity can + # be taken from the first decode step. + identity = step["msa_short_ok"] and step["msa_short_table"] is not None + if identity: + topk = step["msa_short_table"][: num_kv_heads * num_tokens * MSA_REQUIRED_TOPK].view( + num_kv_heads, num_tokens, MSA_REQUIRED_TOPK + ) + else: + max_score = self._index_scores(i, idx_q, step, num_tokens, qlen) + topk = self._select(i, max_score, step, num_tokens).permute(1, 0, 2) + manager = step["kv_cache_manager"] + k_paged, v_paged = msa_paged_kv(manager, i) + nvfp4 = {} + if manager.is_nvfp4_layer(i): + k_scale, v_scale = _aligned_nvfp4_dequant_scales(backend, core.kv_scales[i]) + scales = manager.get_block_scale_buffers(i, "HND") + nvfp4 = dict( + k_block_scale=scales[:, 0], + v_block_scale=scales[:, 1], + k_global_scale=k_scale, + v_global_scale=v_scale, + ) + # The long-context decode (if it runs) prefetches this layer's o_proj weights into L2; any other route leaves + # the slot unused, and it is cleared after the call either way. + if _attn_lc8_kernel.MERGE_ONCE and SPARSE_DECODE_LC8_PREFETCH_OPROJ: + _attn_lc8_kernel.PREFETCH.tensors = (core.w[f"l{i}_o_w"], core.w[f"l{i}_o_s"]) + if _attn_lc8_kernel.MERGE_ONCE and i in core.n1_ids and num_tokens <= N1_MAX_ROWS: + _attn_lc8_kernel.NEXT_KV.tensors = self._next_kv_prefetch(i, manager) + _record("m3_sparse_decode", num_tokens, qlen, identity) + try: + m3_sparse_decode( + q.view(num_tokens, core.heads, core.head_dim), + k_paged, + v_paged, + topk, + block_table, + seq_lens, + sm_scale=core.sm_scale(i), + output=output.view(num_tokens, core.heads, core.head_dim), + decode_query_len=qlen, + identity_blocks=identity, + **nvfp4, + ) + finally: + _attn_lc8_kernel.PREFETCH.tensors = None + _attn_lc8_kernel.NEXT_KV.tensors = None + + def _next_kv_prefetch(self, i: int, manager): + """(layer ``i``'s id buffer, the next sparse layer's, that layer's K / V pools and block scales) for the + long-context decode's next-layer prefetch; the last sparse layer's next is the first one (next step). None when + the next layer's cache is not NVFP4.""" + n1_ids = self.core.n1_ids + order = sorted(n1_ids) + nxt = next((j for j in order if j > i), order[0]) + if not manager.is_nvfp4_layer(nxt): + return None + k_next, v_next = msa_paged_kv(manager, nxt) + scales = manager.get_block_scale_buffers(nxt, "HND") + return (n1_ids[i], n1_ids[nxt], k_next, v_next, scales[:, 0], scales[:, 1]) + + +_ENGAGED: set = set() + + +def _log_engaged(path: str, mapping) -> None: + """One info line per fast path and process (every GEN rank logs its own): the run's engagement evidence.""" + if path not in _ENGAGED: + _ENGAGED.add(path) + rank = getattr(mapping, "tp_rank", "?") if mapping is not None else "?" + logger.info(f"[M3 sm_107 tp8] fast path engaged: {path} (tp rank {rank})") + + +class M3SpecStepFold: + """The target's speculative step-input staging (the in-tree ``SpecStepFold`` protocol). + + Between two decode graphs of a one-model Eagle3 step, the spec sampler stores the step's outputs into its per-slot + store and the engine gathers them back into its flat input buffers for the next step. This object does both in + one launch each (``m3_spec_tail.slot_scatter`` / ``step_gather``): integer copies only, the same values the + engine's and the sampler's own copies write. + + The forward of a decode-only step also does both at its tail (``store_in_forward``, one ``m3_fold_tail`` launch + inside the step graph, this target's C++ op): it stores the step's outputs for the rows ``begin_step`` armed and + stages the next step's inputs for those rows in that order. The arming is a device-side row count and slot list + staged before the replay, so a captured graph whose step was not armed writes nothing. The sampler then skips its + store for exactly the armed rows (``sampler_rows_stored``), and the next step skips its gathers when its rows are + those rows (``begin_step``). The sampler copies the store to the host on its D2H side stream and records + ``store_copies_done`` after the copies; the forward's tail waits on it before it rewrites the store. + + * ``store`` -- the spec sampler's per-slot store (``bind_sampler_store``). + * ``step_inputs`` -- the engine's (input_ids, draft_tokens, position_id_offsets, kv_lens_offsets) device buffers + (``bind_engine_buffers``). + """ + + # Read by the shell's forward (the tail runs only when the fold stores in the forward). + in_forward = True + + def __init__(self) -> None: + self.store: Optional[object] = None + self.step_inputs: Optional[tuple] = None + # Device arming read by the forward's tail: ctrl[0] = armed rows (0 = write nothing), slots[:rows]. + self._ctrl: Optional[torch.Tensor] = None + self._slots: Optional[torch.Tensor] = None + # The rows (sequence slots) the next forward stores and stages for, as begin_step armed them. + self._armed: Optional[tuple] = None + # The rows the previous forward staged the next step's inputs for (confirmed by sampler_rows_stored). + self._staged_for: Optional[tuple] = None + # Output buffers the tail ran on: the captured graphs' static outputs (by address) and the last eager step's. + self._graph_outputs: set = set() + self._eager_outputs: Optional[torch.Tensor] = None + self._logged = False + # The (external) event the sampler records on its D2H side stream after copying the store to the host, waited + # on by the forward's tail before it rewrites the store (see SpecSamplerBase._copy_outputs_to_host). + self.store_copies_done: Optional[torch.cuda.Event] = None + + def bind_engine_buffers( + self, input_ids, draft_tokens, position_id_offsets, kv_lens_offsets, max_batch_size + ): + self.step_inputs = (input_ids, draft_tokens, position_id_offsets, kv_lens_offsets) + # Recorded once now so a captured wait has a completed record to wait on before the first copy. + self.store_copies_done = torch.cuda.Event(external=True) + self.store_copies_done.record() + logger.info( + "[M3 host] spec store host copies on the D2H side stream, awaited by the forward's tail" + ) + logger.info("[M3 host] spec fold tail: one C++ launch (no Triton) at the forward's tail") + self._ctrl = torch.zeros(1, dtype=torch.int32, device=input_ids.device) + self._slots = torch.zeros(max(max_batch_size, 1), dtype=torch.long, device=input_ids.device) + + def bind_sampler_store(self, store: object) -> None: + self.store = store + + def begin_step(self, seq_slots: Optional[list], num_rows: int) -> bool: + if self._ctrl is None: + return False + rows = tuple(seq_slots) if seq_slots is not None else None + staged = rows is not None and rows == self._staged_for + self._staged_for = None + armed = rows if rows is not None and 0 < len(rows) <= self._slots.numel() else None + # Read only by this step's forward, so both uploads may wait for the step-input pack's flush. + if armed is not None: + slots = torch.tensor(armed, dtype=torch.long, pin_memory=prefer_pinned()) + copy_h2d_if_changed(self._slots[: len(armed)], slots, pack=True) + ctrl = torch.tensor( + [len(armed) if armed is not None else 0], dtype=torch.int32, pin_memory=prefer_pinned() + ) + copy_h2d_if_changed(self._ctrl, ctrl, pack=True) + self._armed = armed + return staged + + # The launches the store wait may precede. store_copies_done is an external event: under graph capture its wait + # is a node of its own, so the kernel launched right after it loses its programmatic (PDL) edge and with it the + # overlap of its pre-wait part. Only a launch that waits for its predecessor's completion anyway may follow it: + # the fold tail (a plain launch, no griddepcontrol). Waiting before the last draft_step instead cost +13 / +17 + # us/step. + STORE_WAIT_BEFORE = frozenset({"fold_tail"}) + + def _wait_store_copies(self, before: str) -> None: + """Order the store's rewrite after the sampler's host copies of it; ``before`` names the launch that follows. + Under graph capture a launch outside STORE_WAIT_BEFORE fails the capture.""" + if self.store_copies_done is None: + return + if before not in self.STORE_WAIT_BEFORE and torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + f"M3SpecStepFold: the store wait would precede {before!r}, a launch outside " + f"{sorted(self.STORE_WAIT_BEFORE)}; an external-event wait node drops the next kernel's PDL edge" + ) + torch.cuda.current_stream().wait_event(self.store_copies_done) + + def store_in_forward(self, outputs: dict) -> None: + """At the forward's tail: store the armed rows' outputs and stage the next step's inputs (one launch).""" + if self._ctrl is None or self.store is None or self.step_inputs is None: + return + store = self.store + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets = self.step_inputs + new_tokens = outputs.get("new_tokens") + next_new_tokens = outputs.get("next_new_tokens") + new_tokens_lens = outputs.get("new_tokens_lens") + next_draft_tokens = outputs.get("next_draft_tokens") + if any( + t is None or not isinstance(t, torch.Tensor) + for t in (new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens) + ): + return + rows = new_tokens.shape[0] + # The previous step's host copies of the store run on the sampler's side stream. + self._wait_store_copies(before="fold_tail") + _record("m3_fold_tail", rows) + if not m3_fold_tail.fold_tail( + self._ctrl, + self._slots, + min(rows, self._slots.numel()), + new_tokens, + store.new_tokens.squeeze(-1).T, + next_new_tokens, + store.next_new_tokens.squeeze(-1).T, + next_draft_tokens, + store.next_draft_tokens, + new_tokens_lens, + store.new_tokens_lens, + input_ids, + draft_tokens, + position_id_offsets, + kv_lens_offsets, + next_new_tokens.shape[1], + cpp=True, + ): + return + if torch.cuda.is_current_stream_capturing(): + self._graph_outputs.add(new_tokens.data_ptr()) + else: + self._eager_outputs = new_tokens + + def gather( + self, + new_tokens, + next_draft_tokens, + new_tokens_lens, + previous_slots, + previous_pos_indices, + previous_batch_len: int, + tokens_per_gen_step: int, + draft_token_buffer_width: int, + num_tokens: int, + num_draft_tokens: int, + first_previous_row: int, + ) -> bool: + if self.step_inputs is None: + return False + input_ids, draft_tokens, position_id_offsets, kv_lens_offsets = self.step_inputs + return m3_spec_tail.step_gather( + new_tokens, + next_draft_tokens, + new_tokens_lens, + previous_slots, + previous_pos_indices, + previous_batch_len, + tokens_per_gen_step, + draft_token_buffer_width, + input_ids, + num_tokens, + draft_tokens, + num_draft_tokens, + position_id_offsets, + kv_lens_offsets, + first_previous_row, + ) + + def sampler_rows_stored(self, outputs: dict, num_skip: int, seq_slots: list) -> bool: + armed, self._armed = self._armed, None + new_tokens = outputs.get("new_tokens") + ran = isinstance(new_tokens, torch.Tensor) and ( + new_tokens is self._eager_outputs or new_tokens.data_ptr() in self._graph_outputs + ) + self._eager_outputs = None + stored = ( + ran + and armed is not None + and num_skip == 0 + and tuple(seq_slots) == armed + and new_tokens.shape[0] == len(armed) + ) + self._staged_for = armed if stored else None + if stored and not self._logged: + self._logged = True + logger.info("[M3 host] spec step stored and next step staged in the forward tail") + return stored + + def scatter( + self, seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens + ) -> bool: + store = self.store + if store is None: + return False + return m3_spec_tail.slot_scatter( + seq_slots, + new_tokens, + store.new_tokens.squeeze(-1).T, + next_new_tokens, + store.next_new_tokens.squeeze(-1).T, + new_tokens_lens, + store.new_tokens_lens, + next_draft_tokens, + store.next_draft_tokens, + ) + + +class M3DSparkStepFold(M3SpecStepFold): + """The step fold under the standalone DSpark worker (DSFOLD): the engine's previous-batch gathers in one launch + (``step_gather``). The sampler's slot stores stay on its in-tree path, so ``scatter`` declines every step. + ``taken`` / ``declined`` count the steps the gather ran / fell back to the engine's own copies.""" + + def __init__(self) -> None: + super().__init__() + self.taken = 0 + self.declined = 0 + + def gather(self, *args, **kwargs) -> bool: + if not super().gather(*args, **kwargs): + self.declined += 1 + uncovered( + "DSFOLD", + "step_gather declined the inputs (dtype / layout); the engine's copies run", + ) + return False + if self.taken == 0: + logger.info( + "[M3 dispatch] DSFOLD DSpark step-input gather: engaged (one step_gather launch)" + ) + self.taken += 1 + return True + + def scatter( + self, seq_slots, new_tokens, next_new_tokens, new_tokens_lens, next_draft_tokens + ) -> bool: + return False + + +class _DraftForward: + """``draft_model.model`` for the in-tree worker: the drafter layer's forward, called as + ``model(input_ids=, position_ids=, hidden_states=, attn_metadata=, spec_metadata=, all_rank_num_tokens=)`` and + returning (hidden, hidden to carry to the next pass). ``d2t`` None: the draft vocabulary is the target's.""" + + d2t = None + + def __init__(self, draft: "M3Eagle3Draft") -> None: + self._draft = draft + + def __call__(self, **kwargs): + return self._draft.forward_layer(**kwargs) + + +# The drafter's projections by weight-table name, for the low-M linears' engagement and prebuild. +_DRAFT_LINEARS = ("fc", "qkv", "o", "gu", "dn") + + +class M3Eagle3Draft(nn.Module): + """The one-layer BF16 Eagle3 drafter (``speculative_model``): its weights (``weights.DRAFT_WEIGHTS``), its + vocab-parallel BF16 head and logits processor, the drafter layer's TRTLLM attention backend (layer index = the + target's layer count, FP8 KV) and its TP all-reduce. + + What the worker reads: ``model`` (the layer forward), ``apply_eagle3_fc`` (the per-layer fc_norms + fc over the + captured target hidden states), ``lm_head`` and ``logits_processor``; the fused draft loop reads the layer's parts + (``draft_input_weights``, ``attention``, ``mlp``, ``ar_norm``). The drafter embeds with the target's table. + + The BF16 projections (fc, qkv, o_proj, gate_up, down; not the head) run FlashInfer's fixed-tactic low-M kernels + (``ops/gemm/m3_draft_lowm``, the TP8 port) at the rows ``DRAFT_LOWM`` covers, compiled before the warmup + (``prebuild_linears``); every other call is the in-tree cuBLAS GEMM. Rubin TP2 (mclk locked, 4 rotating sets) at + 1 row: qkv 13.18 -> 9.71, o 7.17 -> 5.33, gate_up 22.27 -> 17.70, down 12.52 -> 9.76, fc 21.50 -> 17.77 us. The + split-K ('s') tactics run this target's copy of FlashInfer's kernel (``kernels/m3_lowm_splitk``, bit for bit the + same output); with ``DRAFT_AR_PUSH`` the o_proj and down projection of the fused draft loop also push their rows + into the drafter boundary all-reduce's exchange (``arm_push`` / ``take_pushed``); the direct ('d') tactics never + push. + """ + + def __init__( + self, + core: MiniMaxM3Tp8ModelingV2, + draft_config: ModelConfig, + start_layer_idx: int, + capture_layers: int, + ) -> None: + super().__init__() + cfg = draft_config.pretrained_config + _ensure_draft_vocab_size(cfg) + eagle = getattr(cfg, "eagle_config", None) or {} + mapping = draft_config.mapping + self.draft_config = draft_config + # The engine's draft weight mapper binds to these two. + self.model_config = draft_config + self.config = cfg + self.capture_layers = int(capture_layers) + self.layer_idx = int(start_layer_idx) + self.dtype = cfg.torch_dtype + self.hidden = int(cfg.hidden_size) + self.hidden_size_in = int(getattr(cfg, "target_hidden_size", cfg.hidden_size)) + self.rms_eps = float(cfg.rms_norm_eps) + self.return_hidden_post_norm = bool( + eagle.get("return_hidden_post_norm", False) or getattr(cfg, "norm_output", False) + ) + if ( + int(cfg.num_hidden_layers) != 1 + or eagle.get("eh_proj_before_attn", False) + or eagle.get("norm_before_fc", False) + or not getattr(cfg, "fc_norm", False) + or getattr(cfg, "kv_lora_rank", None) + or self.capture_layers <= 1 + or self.hidden_size_in != self.hidden + or cfg.draft_vocab_size != cfg.vocab_size + or getattr(cfg, "attention_bias", False) + or getattr(cfg, "mlp_bias", False) + or getattr(cfg, "bias", False) + or getattr(cfg, "tie_word_embeddings", False) + ): + raise ValueError( + "this MiniMax-M3 target drafts with a one-layer Llama-style Eagle3 drafter with per-layer " + "fc_norms over " + "several captured layers, the target's hidden size and vocabulary and no biases" + ) + tp = mapping.tp_size + self.heads = cfg.num_attention_heads // tp + self.kv_heads = (cfg.num_key_value_heads + tp - 1) // tp + self.head_dim = int( + getattr(cfg, "head_dim", None) or cfg.hidden_size // cfg.num_attention_heads + ) + self.q_size = self.heads * self.head_dim + self.kv_size = self.kv_heads * self.head_dim + self.w = _weights.DRAFT_WEIGHTS.declare(_weights.DRAFT_WEIGHTS.dims(self)) + self.lm_head = LMHead( + cfg.draft_vocab_size, + cfg.hidden_size, + dtype=cfg.torch_dtype, + mapping=mapping, + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=draft_config.lm_head_gather_output, + reduce_output=False, + use_custom_cublas_mm=False, + quant_config=None, + ) + self.logits_processor = LogitsProcessor() + self.allreduce = AllReduce( + mapping=mapping, strategy=draft_config.allreduce_strategy, dtype=cfg.torch_dtype + ) + quant_config = draft_config.get_quant_config() + self.attn = create_attention( + draft_config.attn_backend, + self.layer_idx, + self.heads, + self.head_dim, + self.kv_heads, + pos_embd_params=PositionalEmbeddingParams( + type=PositionEmbeddingType.rope_gpt_neox, rope=RopeParams.from_config(cfg) + ), + quant_config=quant_config, + skip_create_weights_in_init=draft_config.skip_create_weights_in_init, + q_scaling=1.0, + attention_chunk_size=None, + sparse_params=None, + ) + if not self.attn.support_fused_rope() or not self.attn.support_fused_qkv(): + raise ValueError( + f"the drafter's attention backend {type(self.attn).__name__} must fuse RoPE and QKV" + ) + self.attn.update_quant_config(quant_config) + # The drafter embeds with the target's table, read off the core on every call (the engine materializes the + # core's weights after construction). + self._core = [core] + self.model = _DraftForward(self) + # The low-M linears: (N, K, M, PDL) of every call prebuild_linears compiled, and (N, K, M) of every call that + # logged its engagement line. + self._lowm_built: set = set() + self._lowm_engaged: set = set() + # DRAFT_AR_PUSH: the weight whose next low-M call also pushes its rows into a boundary all-reduce (data + # pointer, the bucket's multicast words, its flags, rank, world, max tokens), set by arm_push; and the data + # pointer of the output the last armed call pushed, 0 if none (read once by take_pushed). Host state set and + # read in one forward's issue order. + self._push_arm: Optional[tuple] = None + self._pushed = 0 + + @property + def embed_table(self) -> torch.Tensor: + return self._core[0].w["embed"] + + def load_weights_from_target_model(self, target_model: nn.Module) -> None: + """Nothing to alias: the drafter reads the target's embedding table through the core and owns its head.""" + + # ------------------------------------------------------------------------------------------- the projections + + def linear(self, x: torch.Tensor, key: str) -> torch.Tensor: + """``x @ w[key]^T``: the low-M kernel where ``DRAFT_LOWM`` covers the call, else the in-tree cuBLAS GEMM.""" + w = self.w[key] + out = self._lowm_linear(w, x) + return out if out is not None else m3_bf16_linear(x, w) + + def _lowm_linear(self, weight: torch.Tensor, x: torch.Tensor) -> Optional[torch.Tensor]: + """x [M, K] @ weight^T through the low-M kernel when ``DRAFT_LOWM`` covers it; None for the in-tree GEMM. + A projection none of whose row counts is covered never reaches the low-M checks.""" + n, k = weight.shape + if x.dim() != 2 or not any(nk == (n, k) for nk, _ in DRAFT_LOWM.keys): + return None + m = int(x.shape[0]) + if not DRAFT_LOWM.covers(((n, k), m)): + uncovered(DRAFT_LOWM.path, f"the drafter's {n}x{k} linear past its tactic table's rows") + return None + if not x.is_contiguous(): + # The first pass's fc may read a column window of the captured hidden states; the in-tree GEMM takes + # strides. + uncovered(DRAFT_LOWM.path, "a strided activation (the in-tree GEMM reads it in place)") + return None + use_pdl = get_env_enable_pdl() + if x.dtype != torch.bfloat16 or weight.dtype != torch.bfloat16: + missed(DRAFT_LOWM.path, m, "the activation or weight is not BF16") + return None + if (n, k, m, use_pdl) not in self._lowm_built: + missed( + DRAFT_LOWM.path, m, f"the {n}x{k} kernel at {m} rows was not compiled before warmup" + ) + return None + kind, tactic = tactic_for(n, k, m) + if (n, k, m) not in self._lowm_engaged: + self._lowm_engaged.add((n, k, m)) + logger.info( + f"[M3 sm_107 tp8] fast path engaged: m3_draft_lowm {n}x{k} at {m} rows (device {x.device.index})" + ) + out = torch.empty((m, n), dtype=x.dtype, device=x.device) + ar = None + if ( + self._push_arm is not None + and self._push_arm[0] == weight.data_ptr() + and kind == "s" + and DRAFT_LOWM_OWN_SPLITK + ): + ar = self._push_arm[1:] + self._pushed = out.data_ptr() + _record("m3_draft_lowm", n, k, m, kind, ar is not None) + return m3_draft_lowm(x, weight, out, kind, tactic, use_pdl, DRAFT_LOWM_OWN_SPLITK, ar) + + def prebuild_linears(self) -> int: + """Compile (and run once) every covered row count of every drafter projection, outside capture, on the + current stream, before the executor's warmup. Returns the number of calls compiled.""" + from .kernels.m3_lowm_splitk import compile_push + + use_pdl = get_env_enable_pdl() + built = 0 + shapes = {tuple(self.w[key].shape): self.w[key] for key in _DRAFT_LINEARS} + for (n, k), weight in sorted(shapes.items()): + if weight.dtype != torch.bfloat16: + continue + for _, m in sorted(key for key in DRAFT_LOWM.keys if key[0] == (n, k)): + if (n, k, m, use_pdl) in self._lowm_built: + continue + kind, tactic = tactic_for(n, k, m) + x = torch.zeros(m, k, dtype=torch.bfloat16, device=weight.device) + out = torch.empty(m, n, dtype=torch.bfloat16, device=weight.device) + m3_draft_lowm(x, weight, out, kind, tactic, use_pdl, DRAFT_LOWM_OWN_SPLITK) + if DRAFT_AR_PUSH and DRAFT_LOWM_OWN_SPLITK and kind == "s": + # The push variant a boundary all-reduce may arm (o_proj / down): compiled only, never run here + # (its pushes need the all-reduce's exchange and a matching call). + compile_push(x, weight.t(), out, use_pdl, tactic) + self._lowm_built.add((n, k, m, use_pdl)) + built += 1 + return built + + def arm_push(self, key: str, workspace, rows: int) -> None: + """Arm ``w[key]``'s next covered low-M call to push into ``workspace``'s bucket for ``rows``.""" + bucket = m3_mc_allreduce._bucket(workspace, rows) + flags = None if bucket is None else bucket.flags_for(8) + self._pushed = 0 + self._push_arm = None + if bucket is not None and flags is not None and bucket.max_tokens == 8: + self._push_arm = ( + self.w[key].data_ptr(), + bucket.mc.data_ptr(), + flags.data_ptr(), + bucket.rank, + bucket.world_size, + bucket.max_tokens, + ) + + def take_pushed(self, x: torch.Tensor) -> bool: + """Whether the armed call pushed its rows; disarms. ``x`` is what the boundary all-reduce is about to reduce: + it must be the pushing call's own output (the prepushed all-reduce reads only the pushed words, never ``x``), + so an operation between that call and the all-reduce raises instead of silently reducing the wrong rows.""" + pushed, self._pushed, self._push_arm = self._pushed, 0, None + if pushed and x.data_ptr() != pushed: + raise RuntimeError( + "the drafter's pushed rows are not the boundary all-reduce's input (an operation ran in between)" + ) + return bool(pushed) + + # ------------------------------------------------------------------------------------------------- the layer + + def fc(self, hs: torch.Tensor) -> torch.Tensor: + return self.linear(hs, "fc") + + def apply_eagle3_fc(self, hidden_states: torch.Tensor) -> torch.Tensor: + """The captured target hidden states [T, layers x H] to the drafter's input width: each layer's fc_norm, + then the fc (the in-tree ``Eagle3ForCausalLM.apply_eagle3_fc``).""" + hidden_states = hidden_states.to(self.dtype) + if hidden_states.shape[-1] != self.hidden: + chunks = hidden_states.chunk(self.capture_layers, dim=-1) + hidden_states = torch.cat( + [ + flashinfer_rmsnorm(c, self.w[f"fc_norm{i}"], self.rms_eps) + for i, c in enumerate(chunks) + ], + dim=-1, + ) + hidden_states = self.fc(hidden_states) + return hidden_states + + def attention(self, x: torch.Tensor, md, reduce: bool) -> torch.Tensor: + """QKV, the backend's fused RoPE + KV append + causal attention, o_proj (all-reduced unless ``reduce`` is + False, where the caller fuses the reduction). The backend's generation phase runs this target's full-context + decode with the fused RoPE + FP8 K/V append when the shell installed it (``kernels/m3_drafter_attn``).""" + qkv = self.linear(x, "qkv") + _record("m3_draft_attention", int(md.num_tokens)) + o = m3_draft_attention(self.attn, qkv, md) + out = self.linear(o, "o") + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def mlp(self, h: torch.Tensor, reduce: bool, cpp_silu: bool = False) -> torch.Tensor: + """gate_up, the plain SwiGLU, down (all-reduced unless ``reduce`` is False). ``cpp_silu`` (the fused draft + loop): the SwiGLU on this target's C++ op (``ops/spec/m3_draft_silu``, minimaxM3Sm107Tp8/siluAndMul.cu), bit + for bit the in-tree Triton silu_and_mul in one launch either way (no Triton on the GEN decode path).""" + gu = self.linear(h, "gu") + act = None + if cpp_silu: + act = m3_draft_silu(gu) + if act is None: + uncovered( + DRAFT_SILU_PATH, "the drafter SwiGLU input layout (the Triton op runs it)" + ) + else: + taken(DRAFT_SILU_PATH, int(gu.shape[0]), "drafter MLP SwiGLU, one C++ launch") + if act is None: + act = m3_silu_and_mul(gu, None, None, None) + out = self.linear(act, "dn") + return m3_allreduce(out, self.allreduce, None) if reduce else out + + def ar_norm(self, x: torch.Tensor, residual: torch.Tensor, norm_key: str): + """One draft-layer boundary in the in-tree structure: the projection's plain all-reduce (BF16 out), then the + fused residual add + RMSNorm (FP32 add, norm of the FP32 sum). Returns (normed, residual sum).""" + return flashinfer_fused_add_rmsnorm( + m3_allreduce(x, self.allreduce, None), residual, self.w[norm_key], self.rms_eps + ) + + def forward_layer( + self, + input_ids: torch.Tensor, + position_ids: torch.Tensor, + hidden_states: torch.Tensor, + attn_metadata, + spec_metadata, + all_rank_num_tokens: Optional[List[int]] = None, + ): + """The drafter layer (the in-tree ``Eagle3DraftModel.forward`` for one layer): the embedded tokens and the + fc'd hidden states normed and concatenated, attention, the post-attention norm with the residual, the MLP and + the final norm. Returns (normed output, the hidden state the next pass carries).""" + previous = attn_metadata.all_rank_num_tokens + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = all_rank_num_tokens + try: + embeds = m3_embedding(input_ids, self.embed_table).to(self.dtype) + residual = hidden_states + h = flashinfer_rmsnorm(hidden_states, self.w["hidden_norm"], self.rms_eps) + e = flashinfer_rmsnorm(embeds, self.w["in_norm"], self.rms_eps) + h = torch.cat([e, h], dim=-1) + h = self.attention(h, attn_metadata, reduce=True) + h, residual = flashinfer_fused_add_rmsnorm( + h, residual, self.w["post_norm"], self.rms_eps + ) + h = self.mlp(h, reduce=True) + spec_metadata.maybe_capture_hidden_states(self.layer_idx, h, residual) + h, to_save = flashinfer_fused_add_rmsnorm( + h, residual, self.w["final_norm"], self.rms_eps + ) + return (h, h) if self.return_hidden_post_norm else (h, to_save) + finally: + if all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = previous + + +def _eagle3_capture(spec_metadata, layer_id: int, hidden_states, residual, capture_norm) -> None: + """The Eagle3 hidden-state capture of one target layer. + + Mirrors ``Eagle3OneModelSpecMetadata.maybe_capture_hidden_states``. With ``capture_norm`` = (the drafter's stacked + fc_norm weights [layers, H], eps), the capture writes rmsnorm(bf16(h + residual)) * fc_norm[i] in one kernel (the + in-tree capture's bf16 add, then the in-tree fc_norm arithmetic), which the fused first drafter inputs read as is; + otherwise it is the in-tree capture. + """ + if capture_norm is None: + spec_metadata.maybe_capture_hidden_states(layer_id, hidden_states, residual) + return + for i, captured_layer_id in enumerate(spec_metadata.layers_to_capture): + if captured_layer_id == layer_id: + assert spec_metadata.hidden_states is not None + num_tokens = spec_metadata.num_capture_tokens + if num_tokens > spec_metadata.hidden_states.shape[0]: + raise RuntimeError( + "EAGLE3 hidden-state capture token count exceeds buffer capacity: " + f"num_tokens={num_tokens}, capacity={spec_metadata.hidden_states.shape[0]}" + ) + if num_tokens > hidden_states.shape[0]: + raise RuntimeError( + "EAGLE3 hidden-state capture token count exceeds available hidden states: " + f"num_tokens={num_tokens}, hidden_states={hidden_states.shape[0]}" + ) + hidden_size = spec_metadata.hidden_size + m3_spec_tail.capture_norm( + hidden_states[:num_tokens], + residual[:num_tokens] if residual is not None else None, + capture_norm[0][i], + capture_norm[1], + spec_metadata.hidden_states[:num_tokens, i * hidden_size : (i + 1) * hidden_size], + ) + break + + +def _sharded_head(lm_head: nn.Module) -> bool: + """A vocab-sharded TP head without padding (the fused kernels read the local shard).""" + return ( + getattr(lm_head, "tp_mode", None) == TensorParallelMode.COLUMN + and getattr(lm_head, "padding_size", 0) == 0 + and getattr(lm_head, "tp_size", 1) > 1 + ) + + +class M3Eagle3Worker(Eagle3OneModelWorker): + """``Eagle3OneModelWorker`` with the MiniMax-M3 fused verify and fused draft loop, at the in-tree numerics. + + * Verify: on decode-only batches whose verify rows the fused kernels cover (``tail_eligible``), the target's BF16 + lm_head computes this rank's vocabulary shard of the logits (the in-tree head's own local GEMM) and one kernel + replaces the logits all-gather, the argmax (or the temperature + top-p draw) and the strict acceptance. + * Draft: the drafter's first pass may take the per-layer fc_norms in one launch (or folded into the hidden-state + capture) and run its fc beside the target head GEMM (forked right after the last captured target layer at most + ``FC_EARLY_MAX_ROWS`` verify rows; after a sampled verify's head at most ``FC_AFTER_HEAD_MAX_ROWS``). On a + decode-only step ``DRAFT_STEP`` covers, the draft loop is fused: each pass's draft input in one launch, the + drafter layer's attention and MLP with their all-reduces fused with the residual add and the next RMSNorm (the + multicast all-reduce at ``DRAFT_MC_AR``'s rows, fed by the projection's push), and one kernel for the BF16 head, + the cross-rank argmax and the pass's bookkeeping. + + Every other batch takes the in-tree paths. Methods whose docstring says "Mirrors" copy the named in-tree method + with the fused branches inserted. + """ + + def __init__(self, spec_config, mapping, use_separate_draft_kv_cache: bool = False) -> None: + super().__init__( + spec_config, mapping=mapping, use_separate_draft_kv_cache=use_separate_draft_kv_cache + ) + # Per-step state, reset at the top of every _forward_impl: the first draft pass's gather rows when this step + # takes the fused first drafter inputs, and the fc output a fork computed beside the target head. + self._m3_first_gather_ids: Optional[torch.Tensor] = None + self._m3_fc_hidden: Optional[torch.Tensor] = None + self._m3_target_lm_head: Optional[nn.Module] = None + # (fc_norm0 weight, stacked fc_norm weights [layers, H]), built eagerly on the first fused first pass. + self._m3_fc_norm_w: Optional[tuple] = None + # Armed per step by arm_capture_norm: (stacked fc_norm weights, eps) or None. + self._m3_capture_norm: Optional[tuple] = None + # Per step: the captured layers written with the capture norm (set by capture_hidden_states). + self._m3_capture_normed: set = set() + self._m3_aux_stream: Optional[torch.cuda.Stream] = None + # Armed per step by arm_capture_norm: the drafter whose first fc is forked at the last captured target layer, + # and the fork that did it, joined by _m3_target_accept. + self._m3_fc_early_drafter: Optional[M3Eagle3Draft] = None + self._m3_fc_early_stream: Optional[torch.cuda.Stream] = None + # DI_FC_FLAG: int64 [3] device counters (forks bumped, forks waited, wait overruns), allocated by ``prebuild``; + # whether the latest fork bumped them; the fork whose stream join is deferred past the first draft input. + self._m3_fc_counts: Optional[torch.Tensor] = None + self._m3_fc_flagged = False + self._m3_fc_pending: Optional[torch.cuda.Stream] = None + # The 12-bit coded copy of the target head shard the fused verify streams, built before the warmup. + self._m3_force_vec_warm = False + # The streamed copy of the drafter's BF16 head shard the fused draft loop reads, built before the warmup, the + # head weight version it was built from, and the replaced copies a captured graph may still read. + self._m3_draft_head = None + self._m3_draft_head_version: Optional[int] = None + self._m3_draft_head_graveyard: list = [] + # Per step: the next step's input tokens [nreq, D + 1], column 0 written by the fused verify (or the seed) and + # columns 1.. by the fused draft passes; set as _m3_next_new_tokens once the fused draft loop filled it. + self._m3_next_new: Optional[torch.Tensor] = None + self._m3_next_new_tokens: Optional[torch.Tensor] = None + # Per step: the first draft inputs are the fused ones (accepted tokens as they lie, fc output of every row). + self._m3_fused_first_inputs = False + + def capture_target(self, spec_metadata, layer_id: int, num_tokens: int) -> Optional[tuple]: + """Where this step's capture norm writes layer ``layer_id``'s capture, for a boundary all-reduce that writes it + itself (CAPTURE_NORM_IN_BOUNDARY): (the capture buffer's [num_tokens, H] column block, the layer's fc_norm + weight, eps), or None (no capture norm armed, not a captured layer, or more rows than the buffer). The + boundary writes every row it reduces; rows past num_capture_tokens are unread.""" + capture_norm = self._m3_capture_norm + if capture_norm is None or spec_metadata.hidden_states is None: + return None + if not m3_mc_allreduce.capture_supported(): + return None + if num_tokens > spec_metadata.hidden_states.shape[0]: + return None + hidden_size = spec_metadata.hidden_size + for i, captured_layer_id in enumerate(spec_metadata.layers_to_capture): + if captured_layer_id == layer_id: + out = spec_metadata.hidden_states[ + :num_tokens, i * hidden_size : (i + 1) * hidden_size + ] + return out, capture_norm[0][i], capture_norm[1] + return None + + def capture_hidden_states( + self, spec_metadata, layer_id: int, hidden_states, residual=None, written: bool = False + ) -> None: + """The Eagle3 capture with this step's capture norm (``written``: the layer's boundary all-reduce already wrote + it, see ``capture_target``); on a step armed for the early fc fork (``arm_capture_norm``), the last captured + layer then forks the drafter's first fc onto the aux stream, where it overlaps the remaining target layers + and the head (joined by ``_m3_target_accept``).""" + if not written: + _eagle3_capture(spec_metadata, layer_id, hidden_states, residual, self._m3_capture_norm) + if self._m3_capture_norm is not None: + # Written with this step's capture norm: by the boundary all-reduce (written) or just above. + self._m3_capture_normed.add(layer_id) + drafter = self._m3_fc_early_drafter + layers = spec_metadata.layers_to_capture + if drafter is not None and layers and layer_id == layers[-1]: + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = self._m3_fc_fork( + spec_metadata, spec_metadata.num_capture_tokens, drafter + ) + + def prebuild(self, lm_head: nn.Module, draft: Optional[M3Eagle3Draft]) -> None: + """From the loaded weights, before the executor's warmup, on every rank: the fused draft loop's head copy and + the drafter's low-M linears.""" + start = time.perf_counter() + if self.prebuild_draft_head(draft): + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] fused draft head built before warmup: %.1f s" + % (time.perf_counter() - start) + ) + if DI_FC_FLAG and draft is not None and self._m3_fc_counts is None: + # The fc fork counters (DI_FC_FLAG), zeroed here, before warmup, in order before any fork. + self._m3_fc_counts = torch.zeros( + 3, dtype=torch.int64, device=draft.w["hidden_norm"].device + ) + start = time.perf_counter() + built = draft.prebuild_linears() if draft is not None else 0 + torch.cuda.synchronize() + logger.info( + "[M3 prebuild] drafter low-M GEMMs built before warmup: %d calls, %.1f s" + % (built, time.perf_counter() - start) + ) + + def _forward_impl( + self, + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + resource_manager=None, + ): + """Mirrors Eagle3OneModelWorker._forward_impl, with the fused verify branch.""" + runtime_draft_len = spec_metadata.runtime_draft_len + # skip the draft forward if the runtime draft length is 0 + if runtime_draft_len == 0: + return self.skip_drafting( + input_ids, + position_ids, + hidden_states, + logits, + attn_metadata, + spec_metadata, + draft_model, + ) + + batch_size = attn_metadata.num_seqs + num_contexts = attn_metadata.num_contexts + num_gens = batch_size - num_contexts + + raw_logits = logits + + self._m3_first_gather_ids = None + self._m3_next_new = None + self._m3_next_new_tokens = None + self._m3_fused_first_inputs = False + if logits is None: + # tail_eligible() skipped the gathered target logits: verify from this rank's logits shard. + accepted_tokens, num_accepted_tokens, self._m3_first_gather_ids = ( + self._m3_target_accept( + hidden_states, + spec_metadata, + batch_size, + runtime_draft_len, + draft_model, + kv_lens=getattr(attn_metadata, "kv_lens_cuda", None), + ) + ) + else: + self._execute_guided_decoder_if_present(logits) + + # Sample and accept tokens. ``input_ids`` is required by the relaxed-acceptance path (scans for + # thinking-phase tokens); ignored otherwise. + accepted_tokens, num_accepted_tokens = self.sample_and_accept_draft_tokens( + input_ids, logits, attn_metadata, spec_metadata + ) + if self.draft_only_eligible(attn_metadata, spec_metadata, draft_model): + self._m3_seed_first_pass(num_accepted_tokens, spec_metadata, batch_size) + + # Mamba hybrid models need state updates after token acceptance (a no-op on this target's cache manager). + if self._is_mamba_hybrid_cache is None: + self._is_mamba_hybrid_cache = isinstance( + attn_metadata.kv_cache_manager, MambaHybridCacheManager + ) + if num_gens > 0 and self._is_mamba_hybrid_cache: + attn_metadata.kv_cache_manager.update_mamba_states( + attn_metadata=attn_metadata, + num_accepted_tokens=num_accepted_tokens, + state_indices=attn_metadata.mamba_metadata.state_indices, + ) + + sa_manager = getattr(spec_metadata.spec_resource_manager, "sa_manager", None) + if self.sa_enhancer is not None and sa_manager is not None: + self.sa_enhancer.extend_and_prepare( + sa_manager=sa_manager, + request_ids=spec_metadata.request_ids, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_gens=num_gens, + num_contexts=num_contexts, + max_draft_len=runtime_draft_len, + ) + + if self._uses_external_shared_target_kv: + next_draft_tokens = self._forward_external_shared_target_kv_draft_loop( + position_ids=position_ids, + hidden_states=hidden_states, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + accepted_tokens=accepted_tokens, + num_accepted_tokens=num_accepted_tokens, + num_contexts=num_contexts, + batch_size=batch_size, + ) + else: + # Save the old attn_metadata and spec_metadata + self._prepare_attn_metadata_for_spec_dec(attn_metadata) + + # Prepare inputs for the 1st draft model forward + position_ids = position_ids.squeeze(0) + inputs = self.prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + + # Predict draft tokens. ``original_all_rank_num_tokens`` is saved here so the post-loop restore (below) + # can put attn_metadata back into a state the target model expects. + original_all_rank_num_tokens = attn_metadata.all_rank_num_tokens + + # Get the draft KV cache manager if using separate layouts + draft_kv_cache_manager = self.get_draft_kv_cache_manager(resource_manager) + + next_draft_tokens = self._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + # restore attn_metadata to support cuda graph + self._restore_attn_metadata_from_spec_dec(attn_metadata) + # restore all_rank_num_tokens for attention DP + if original_all_rank_num_tokens is not None: + attn_metadata.all_rank_num_tokens = original_all_rank_num_tokens + + self._m3_join_pending() + # prepare next new tokens to support overlap scheduler + if self._m3_next_new_tokens is not None: + # Filled in place by the fused verify or the seed (column 0) and the fused draft passes (columns 1..). + next_new_tokens = self._m3_next_new_tokens + else: + next_new_tokens = self._prepare_next_new_tokens( + accepted_tokens, + next_draft_tokens, + spec_metadata.batch_indices_cuda, + batch_size, + num_accepted_tokens, + ) + + attn_metadata.use_spec_decoding = True + + # Commit model-owned target state only after draft preparation has succeeded; SpecWorkerBase aborts the + # still-pending snapshots on any exception above this point. + if self._auxiliary_state_handlers: + self.commit_auxiliary_speculative_states( + num_accepted_tokens, + attn_metadata.mamba_metadata.state_indices[:batch_size], + num_contexts, + ) + + return { + "logits": raw_logits, + "new_tokens": accepted_tokens, + "new_tokens_lens": num_accepted_tokens, + "next_draft_tokens": next_draft_tokens, + "next_new_tokens": next_new_tokens, + } + + def prepare_1st_drafter_inputs( + self, + input_ids, + position_ids, + hidden_states, + accepted_tokens, + attn_metadata, + spec_metadata, + draft_model, + ): + """The fused first draft inputs on a step that verified with the fused kernels (or was seeded for them), + else the in-tree ones.""" + fused = self._m3_first_drafter_inputs( + position_ids, + accepted_tokens, + input_ids.shape[0], + attn_metadata.num_contexts, + spec_metadata, + draft_model, + ) + self._m3_fused_first_inputs = fused is not None + if fused is not None: + fused.update(attn_metadata=attn_metadata, spec_metadata=spec_metadata) + return fused + return super().prepare_1st_drafter_inputs( + input_ids=input_ids, + position_ids=position_ids, + hidden_states=hidden_states, + accepted_tokens=accepted_tokens, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=draft_model, + ) + + def _m3_structural_ok(self, attn_metadata, spec_metadata) -> bool: + """The batch conditions shared by the fused verify and the fused first drafter inputs: a decode-only plain + Eagle3 step without attention DP, guided decoding, relaxed acceptance or a suffix-automaton enhancer, of at + most TP2_ROWS verify rows (the fused tail's certified rows; R1VR8 does not widen it). Host state only, so + the TP ranks agree.""" + if self.is_mtp_eagle or self.use_dynamic_tree: + return False + if self._uses_external_shared_target_kv or self.sa_enhancer is not None: + return False + if ( + self.guided_decoder is not None + or self.mapping is None + or self.mapping.enable_attention_dp + ): + return False + if getattr(self.spec_config, "use_relaxed_acceptance_for_thinking", False): + return False + if attn_metadata.num_contexts != 0 or attn_metadata.num_seqs == 0: + return False + draft_len = spec_metadata.runtime_draft_len + if draft_len == 0 or attn_metadata.num_seqs * (draft_len + 1) > TP2_ROWS: + return False + return not spec_metadata.wants_advanced_draft_sampling + + def tail_eligible(self, attn_metadata, spec_metadata, lm_head: nn.Module) -> bool: + """Whether this step verifies with the fused kernels instead of the gathered target logits. + + Decode-only Eagle3 batches (``_m3_structural_ok``) over an unpadded vocab-sharded bias-free target head, all + greedy at a request count ``SHARD_ACCEPT`` covers, or sampled under the temperature + top-p contract (see + _m3_sampled_verify_ok) at a request count ``SAMPLED_ACCEPT`` covers, with nothing else reading the target + logits. Every input is host state shared by all TP ranks, so the ranks agree. + """ + if not self._m3_structural_ok(attn_metadata, spec_metadata): + return False + if getattr(spec_metadata, "enable_penalty", False): + return False + if not _sharded_head(lm_head) or getattr(lm_head, "bias", None) is not None: + return False + draft_len = spec_metadata.runtime_draft_len + if spec_metadata.is_all_greedy_sample: + key = (draft_len, attn_metadata.num_seqs) + if not SHARD_ACCEPT.covers(key): + uncovered( + SHARD_ACCEPT.path, + f"draft length {draft_len} at {attn_metadata.num_seqs} requests", + ) + return False + else: + if not self._m3_sampled_verify_ok(spec_metadata): + return False + key = (draft_len, attn_metadata.num_seqs) + if not SAMPLED_ACCEPT.covers(key): + uncovered( + SAMPLED_ACCEPT.path, + f"draft length {draft_len} at {attn_metadata.num_seqs} requests", + ) + return False + self._m3_target_lm_head = lm_head + _log_engaged( + SHARD_ACCEPT.path if spec_metadata.is_all_greedy_sample else SAMPLED_ACCEPT.path, + self.mapping, + ) + return True + + def draft_only_eligible(self, attn_metadata, spec_metadata, draft_model) -> bool: + """Whether a step verified by the in-tree path can still take the fused first drafter inputs: up to + FIRST_DRAFTER_INPUTS_MAX_REQUESTS requests, the structural conditions of tail_eligible, without the + target-sampling ones.""" + if attn_metadata.num_seqs > FIRST_DRAFTER_INPUTS_MAX_REQUESTS: + uncovered( + "m3_first_drafter_inputs", f"in-tree verify at {attn_metadata.num_seqs} requests" + ) + return False + return self._m3_structural_ok(attn_metadata, spec_metadata) and draft_model is not None + + def _m3_seed_first_pass(self, num_accepted_tokens, spec_metadata, batch_size: int) -> None: + """The first draft pass's gather rows (the last accepted row of each request) from the in-tree acceptance, as + _m3_target_accept leaves them.""" + k1 = spec_metadata.runtime_draft_len + 1 + n = num_accepted_tokens[:batch_size].to(torch.int64) + rows = spec_metadata.batch_indices_cuda[:batch_size].to(torch.int64) + self._m3_first_gather_ids = rows * k1 + n - 1 + + def _m3_first_drafter_inputs( + self, + position_ids, + accepted_tokens, + num_tokens: int, + num_contexts: int, + spec_metadata, + draft_model: Optional[M3Eagle3Draft], + ): + """The in-tree first draft inputs of a decode-only step, with the per-layer fc_norm RMSNorms of + ``apply_eagle3_fc`` in one kernel (unless the capture already applied them) before the fc; the input ids are + the accepted tokens as they lie. None to use the in-tree path.""" + if ( + self._m3_first_gather_ids is None + or self.is_mtp_eagle + or num_contexts != 0 + or draft_model is None + ): + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + ): + return None + if not self._m3_first_inputs_covered(num_tokens, spec_metadata): + return None + cache = self._m3_fc_norm_w + if cache is None or cache[0] is not draft_model.w["fc_norm0"]: + if torch.cuda.is_current_stream_capturing(): + capture_missed("spec_tail", None, "the stacked Eagle3 fc_norm weights") + return None + cache = ( + draft_model.w["fc_norm0"], + torch.stack( + [draft_model.w[f"fc_norm{i}"].detach() for i in range(layers)] + ).contiguous(), + ) + self._m3_fc_norm_w = cache + hidden = self._m3_fc_hidden + self._m3_fc_hidden = None + if hidden is None or hidden.shape[0] != num_tokens: + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + _record("m3_chunk_norm", num_tokens) + hs = m3_spec_tail.chunk_norm(hs, cache[1], draft_model.rms_eps) + _log_engaged(CHUNK_NORM.path, self.mapping) + else: + hs = self._m3_renorm_raw_captures(hs, spec_metadata, draft_model) + hidden = draft_model.fc(hs) + input_ids = accepted_tokens[:, : spec_metadata.runtime_draft_len + 1].reshape(-1) + _log_engaged(FUSED_FIRST_INPUTS.path, self.mapping) + return {"input_ids": input_ids, "position_ids": position_ids, "hidden_states": hidden} + + def _m3_first_inputs_covered(self, num_tokens: int, spec_metadata) -> bool: + """Whether the fused first drafter inputs of ``num_tokens`` verify rows are covered: their request count + (``FUSED_FIRST_INPUTS``), and the chunk norm's rows (``CHUNK_NORM``) unless the captures already applied the + fc_norm.""" + k1 = spec_metadata.runtime_draft_len + 1 + if num_tokens % k1 or not FUSED_FIRST_INPUTS.covers(num_tokens // k1): + uncovered( + FUSED_FIRST_INPUTS.path, + f"{num_tokens} verify rows at draft length {k1 - 1} (the in-tree first drafter inputs run)", + ) + return False + if self._m3_capture_norm is None and not CHUNK_NORM.covers(num_tokens): + uncovered( + CHUNK_NORM.path, f"{num_tokens} verify rows (the in-tree first drafter inputs run)" + ) + return False + return True + + def _m3_fc_fork( + self, + spec_metadata, + num_tokens: int, + draft_model: Optional[M3Eagle3Draft], + after: Optional[torch.cuda.Event] = None, + ) -> Optional[torch.cuda.Stream]: + """Launch the first draft pass's fc_norm + fc on a forked stream beside the target head GEMM (after the event + ``after`` when given, else after everything queued on the current stream); the result is picked up by + _m3_first_drafter_inputs. Returns the stream to join, or None when the fc path does not apply (the first + drafter inputs then compute it).""" + self._m3_fc_hidden = None + if draft_model is None: + return None + layers = len(spec_metadata.layers_to_capture) + if ( + draft_model.hidden_size_in != spec_metadata.hidden_size + or layers != draft_model.capture_layers + or self._m3_fc_norm_w is None + or not self._m3_first_inputs_covered(num_tokens, spec_metadata) + ): + return None + aux = self._m3_aux_stream + if aux is None: + aux = self._m3_aux_stream = torch.cuda.Stream() + if after is not None: + aux.wait_event(after) + else: + aux.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(aux): + hs = spec_metadata.hidden_states[:num_tokens, : spec_metadata.hidden_size * layers] + if self._m3_capture_norm is None: + _record("m3_chunk_norm", num_tokens) + hs = m3_spec_tail.chunk_norm( + hs, self._m3_fc_norm_w[1], draft_model.rms_eps, use_pdl=False + ) + _log_engaged(CHUNK_NORM.path, self.mapping) + else: + hs = self._m3_renorm_raw_captures(hs, spec_metadata, draft_model) + self._m3_fc_hidden = draft_model.fc(hs) + self._m3_fc_flagged = self._m3_fc_bump(hs.device) + return aux + + def _m3_renorm_raw_captures( + self, hs: torch.Tensor, spec_metadata, draft_model: M3Eagle3Draft + ) -> torch.Tensor: + """The captured rows ``hs`` [rows, layers * H] for a reader that skips the fc_norm (the capture norm is armed): + unchanged when every captured layer was written with this step's norm (capture_hidden_states records them), + else -- not expected on any path -- a copy with each layer captured raw normalized by its own fc_norm here, + logged as a miss, so a raw row never reaches the drafter's fc.""" + missing = [ + i + for i, layer_id in enumerate(spec_metadata.layers_to_capture) + if layer_id not in self._m3_capture_normed + ] + if not missing: + return hs + missed( + CAPTURE_NORM.path, + int(hs.shape[0]), + f"captured layer(s) {[spec_metadata.layers_to_capture[i] for i in missing]} were written without the " + "capture norm; their fc_norm is applied before the drafter's fc", + ) + hs = hs.clone() + hidden = spec_metadata.hidden_size + for i in missing: + block = hs[:, i * hidden : (i + 1) * hidden] + block.copy_( + flashinfer_rmsnorm(block, draft_model.w[f"fc_norm{i}"], draft_model.rms_eps) + ) + return hs + + def _m3_fc_bump(self, device: torch.device) -> bool: + """DI_FC_FLAG: count the fc just queued on the aux stream (the wait kernel's release). The counters are zeroed + by ``prebuild``; without them the fork keeps the stream join.""" + if not DI_FC_FLAG or self._m3_fc_counts is None or self._m3_fc_counts.device != device: + return False + self._m3_fc_counts[:1].add_(1) + return True + + def _m3_fc_join(self, fork: torch.cuda.Stream) -> None: + """Join the fc fork onto the current stream and keep its output's memory alive for that stream. + + _m3_fc_hidden is allocated in fork (aux) stream order and read on the current stream. Without the record, + dropping it returns its block to the aux stream's pool, where later aux work can take it while the current + stream's reads are still queued (eager warm-up / mixed steps). Nothing is recorded under graph capture, as in + maybe_execute_in_parallel: a captured graph relies on the next fork's aux.wait_stream(main) to order the + reuse.""" + main = torch.cuda.current_stream() + if self._m3_fc_flagged: + # DI_FC_FLAG: the first draft input waits on the fork's counter; the stream join follows that input. + self._m3_fc_flagged = False + self._m3_join_pending() + m3_fc_wait.fc_wait(self._m3_fc_counts) + self._m3_fc_pending = fork + _log_engaged(DI_FC_FLAG_PATH, self.mapping) + else: + main.wait_stream(fork) + hidden = self._m3_fc_hidden + if ( + hidden is not None + and hidden.is_cuda + and not (torch.compiler.is_compiling() or torch.cuda.is_current_stream_capturing()) + ): + hidden.record_stream(main) + + def _m3_join_pending(self) -> None: + """Join the fc fork whose join DI_FC_FLAG deferred (its output's readiness already went through the wait + kernel): after the first draft input, at the end of the step, and before any newer fork is waited for.""" + fork = self._m3_fc_pending + if fork is not None: + self._m3_fc_pending = None + torch.cuda.current_stream().wait_stream(fork) + + def arm_capture_norm( + self, attn_metadata, spec_metadata, lm_head: nn.Module, draft_model: Optional[M3Eagle3Draft] + ) -> None: + """Before every target forward: whether this step's Eagle3 captures apply the drafter's fc_norm themselves. + + On a fused-verify step of ``CAPTURE_NORM``'s rows whose first drafter inputs are the fused ones and whose + fc_norm weights are already stacked (by an earlier fused first pass), the capture writes + rmsnorm(h + r) * fc_norm[i] (one kernel per captured layer instead of add + copy, or the layer's boundary + all-reduce itself, ``capture_target``) and the fused first drafter inputs read it without their own + chunk_norm. A fused-verify step of at most ``FC_EARLY_MAX_ROWS`` verify rows is also armed to fork the + drafter's first fc at the last captured layer (``capture_hidden_states``). + """ + if self._m3_fc_early_stream is not None: + # A fork the last step made but did not reach the verify: its wait keeps the counters paired. + self._m3_fc_join(self._m3_fc_early_stream) + self._m3_join_pending() + self._m3_capture_norm = None + self._m3_fc_early_drafter = None + self._m3_fc_early_stream = None + # The captured layers this step wrote with its capture norm (capture_hidden_states). + self._m3_capture_normed = set() + # A fork result no first draft pass consumed must not reach a later step's (same-size) first inputs. + self._m3_fc_hidden = None + if draft_model is None: + return + if not self.tail_eligible(attn_metadata, spec_metadata, lm_head): + return + rows = attn_metadata.num_seqs * (spec_metadata.runtime_draft_len + 1) + # The fork reads every captured column at the last captured layer, so a final-state capture, which is written + # after the model returns, keeps the fc at its stock place. + if ( + rows <= FC_EARLY_MAX_ROWS + and FINAL_HIDDEN_LAYER_ID not in spec_metadata.layers_to_capture + ): + self._m3_fc_early_drafter = draft_model + if not CAPTURE_NORM.covers(rows): + uncovered(CAPTURE_NORM.path, f"{rows} verify rows (the in-tree capture runs)") + return + # The normed captures are read only by the fused first drafter inputs; the in-tree ones norm again. + if not FUSED_FIRST_INPUTS.covers(attn_metadata.num_seqs): + return + fc_norm_w = self._m3_fc_norm_w + if ( + fc_norm_w is not None + and fc_norm_w[0] is draft_model.w["fc_norm0"] + and len(spec_metadata.layers_to_capture) == fc_norm_w[1].shape[0] + ): + self._m3_capture_norm = (fc_norm_w[1], draft_model.rms_eps) + _log_engaged(CAPTURE_NORM.path, self.mapping) + + @staticmethod + def _m3_sampled_verify_ok(spec_metadata) -> bool: + """Whether sampled verify rows can use the fused sampled op: only under advanced_sampling_mode=no_topk, the + deploy-time guarantee that no row uses top-k or min-p (that op applies neither).""" + mode = getattr(spec_metadata, "advanced_sampling_mode", None) + return mode is not None and mode.skips_top_k and not mode.skips_top_p + + def _m3_target_accept( + self, + hidden_states, + spec_metadata, + batch_size: int, + draft_len: int, + draft_model: Optional[M3Eagle3Draft] = None, + kv_lens: Optional[torch.Tensor] = None, + ): + """Fused verification (see tail_eligible); same outputs as the strict acceptance path. ``kv_lens``: the + requests' KV lengths at this verify (the per-request forced-acceptance key, FORCE_AL_PER_REQUEST).""" + force_total = 0 + force_vec = None + if self.force_num_accepted_tokens != 0.0: + is_warmup = spec_metadata.is_cuda_graph and not torch.cuda.is_current_stream_capturing() + if not torch.cuda.is_current_stream_capturing(): + # The draw pool is built by an H2D copy and the draw kernels JIT-compile per block width, so both + # happen here, before the first captured step (a scratch counter keeps the real draw sequence). + self._ensure_force_accept_rng_state(hidden_states.device) + if not self._m3_force_vec_warm: + scratch = torch.zeros(1, dtype=torch.int64, device=hidden_states.device) + dummy = torch.empty(64, dtype=torch.int32, device=hidden_states.device) + for block in (16, 32, 64): + m3_force_vec( + self._force_accept_rng_pool, + scratch, + dummy, + block, + 1, + 2, + 0.5, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + m3_force_vec_req( + self._force_accept_rng_pool, + dummy, + dummy, + block, + 1, + 2, + 0.5, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + self._m3_force_vec_warm = True + if not is_warmup: + force_total = min(int(self.force_num_accepted_tokens) + 1, draft_len + 1) + force_vec = self._m3_force_vec( + batch_size, force_total, draft_len + 1, hidden_states.device, kv_lens + ) + draft_tokens = spec_metadata.draft_tokens.view(batch_size, draft_len) + if draft_tokens.dtype != torch.int32: + draft_tokens = draft_tokens.to(torch.int32) + # Column 0 receives the last accepted token, the next step's first input (the fused draft loop adds the rest). + self._m3_next_new = torch.empty( + (batch_size, draft_len + 1), dtype=torch.int32, device=hidden_states.device + ) + rows = hidden_states[: batch_size * (draft_len + 1)] + lm_head = self._m3_target_lm_head + batch_idx = spec_metadata.batch_indices_cuda[:batch_size] + # The Eagle3 fc on the aux stream: the fork made at the last captured layer, else one beside the head. A + # sampled verify's head (the in-tree GEMM up to FC_AFTER_HEAD_MAX_ROWS verify rows) holds + # every SM while it streams: its fc is forked after it instead (overlapping the sampled acceptance), where its + # many split-K CTAs do not delay the head's start. Rubin TP2 c1: the head started 17.4 us after its input was + # ready with the fc beside it. + fork = self._m3_fc_early_stream + self._m3_fc_early_stream = None + head_first = ( + fork is None + and rows.shape[0] <= FC_AFTER_HEAD_MAX_ROWS + and not spec_metadata.is_all_greedy_sample + ) + if fork is None and not head_first: + fork = self._m3_fc_fork(spec_metadata, rows.shape[0], draft_model) + # This rank's vocabulary shard of the target logits: the head's own local GEMM, whose all-gather the fused + # kernels replace (they exchange per-row records across ranks, not logits). + logits = lm_head.apply_linear(rows, None) + head_done = None + if head_first: + head_done = torch.cuda.Event() + head_done.record() + vocab_offset = self.mapping.tp_rank * lm_head.weight.shape[0] + if spec_metadata.is_all_greedy_sample: + result = m3_shard_accept_kernel.shard_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + force_total, + self._m3_next_new, + force_vec=force_vec, + ) + path = SHARD_ACCEPT.path + else: + n_rows = rows.shape[0] + seeds, offsets = self._rng_state_per_token(spec_metadata, n_rows) + result = m3_sampled_accept_kernel.sampled_accept( + logits, + vocab_offset, + self.mapping, + draft_tokens, + batch_idx, + spec_metadata.temperatures[:n_rows], + spec_metadata.top_ks[:n_rows], + spec_metadata.top_ps[:n_rows], + None, + seeds, + offsets, + force_total, + self._m3_next_new, + force_vec=force_vec, + ) + path = "spec_tail" + if head_done is not None: + # Enqueued after the sampled acceptance (which starts first) but waits on the head only. + fork = self._m3_fc_fork(spec_metadata, rows.shape[0], draft_model, after=head_done) + if fork is not None: + self._m3_fc_join(fork) + if result is None: + if torch.cuda.is_current_stream_capturing(): + capture_missed(path, None, "the fused verify's exchange workspace") + raise RuntimeError( + "M3 fused verify unavailable after tail_eligible() accepted the step" + ) + return result + + def _m3_force_vec( + self, + batch_size: int, + base_total: int, + max_total: int, + device, + kv_lens: Optional[torch.Tensor] = None, + ) -> Optional[torch.Tensor]: + """Per-request forced accepted counts for a fractional forced acceptance, else None. + + With ``kv_lens`` (FORCE_AL_PER_REQUEST): each request draws from the fixed-seed pool at its own key, its KV + length at this verify (its prompt plus what it has accepted so far), so its draws do not depend on the + server's history or on the other requests; every TP rank holds the same lengths, so the ranks agree. Without + lengths: exactly as _apply_force_accepted_tokens draws for a decode-only batch (one shared counter bump per + step, the (counter, slot) hash), the generic path's sequence. + """ + frac = self.force_num_accepted_tokens - int(self.force_num_accepted_tokens) + if frac <= 0.0 or base_total >= max_total: + return None + self._ensure_force_accept_rng_state(device) + out = torch.empty(batch_size, dtype=torch.int32, device=device) + block = max(16, 1 << (batch_size - 1).bit_length()) + if FORCE_AL_PER_REQUEST and kv_lens is not None: + keys = kv_lens[:batch_size] + if keys.dtype != torch.int32: + keys = keys.to(torch.int32) + return m3_force_vec_req( + self._force_accept_rng_pool, + keys, + out, + batch_size, + base_total, + max_total, + float(frac), + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + # One launch: bump the shared counter and draw, with _apply_force_accepted_tokens' exact arithmetic. + return m3_force_vec( + self._force_accept_rng_pool, + self._force_accept_rng_counter, + out, + batch_size, + base_total, + max_total, + float(frac), + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + block, + ) + + # ---------------------------------------------------------------------------------------------- fused draft loop + + def prebuild_draft_head(self, draft_model: Optional[M3Eagle3Draft]) -> bool: + """Build the copy of the drafter's BF16 head shard that the fused draft loop streams (rebuilt only when the + head weight tensor or its version changed; a replaced copy is kept alive), on every rank. False when the + drafter is not one the loop serves.""" + old = self._m3_draft_head + if draft_model is None or not self._m3_drafter_supported(draft_model): + self._m3_draft_head = None + return False + weight = draft_model.lm_head.weight + # Idempotent: the same weight tensor at the same version keeps the built copy, so a repeated + # cache_derived_state never re-places it under a captured CUDA graph. + if ( + old is not None + and old.weight is weight + and self._m3_draft_head_version == weight._version + ): + return True + if old is not None: + # A rebuilt copy (new weight values): keep the old one alive, since a graph captured earlier may still + # read its (placed) memory until the graphs are recaptured. + self._m3_draft_head_graveyard.append(old) + self._m3_draft_head = m3_draft_step_kernel.prepare_head( + draft_model.lm_head, self.mapping.tp_rank + ) + self._m3_draft_head_version = weight._version + return True + + def _m3_drafter_supported(self, draft_model: M3Eagle3Draft) -> bool: + """The drafter the fused draft loop runs (``M3Eagle3Draft`` validates its shape at construction) over this + worker's tensor-parallel group with a bias-free vocab-sharded BF16 head without padding.""" + if self.mapping is None or self.mapping.tp_size <= 1 or self.mapping.enable_attention_dp: + return False + if self._d2t is not None: + return False + lm_head = draft_model.lm_head + return _sharded_head(lm_head) and m3_draft_step_kernel.head_supported(lm_head) + + def _m3_draft_ar_norm( + self, + draft_model: M3Eagle3Draft, + x: torch.Tensor, + residual: torch.Tensor, + norm_key: str, + carveout: bool = True, + pushed: bool = False, + ): + """One drafter boundary of the fused draft loop: the multicast all-reduce + residual + RMSNorm with the + drafter's stock rounding (``DRAFT_MC_AR``: opts bit 16, normalizing the unrounded FP32 sum as FlashInfer + fused_add_rmsnorm does; Rubin TP2 10.0..11.0 -> 4.6..5.3 us per boundary), else the in-tree structure + (``M3Eagle3Draft.ar_norm``). ``carveout`` False drops the reduce's max-smem carveout (``DRAFT_MC_AR_CARVEOUT``). + ``pushed``: the boundary's projection already pushed its rows into the exchange (prepushed 2). The choice + depends on shapes and a collectively created workspace only, so the TP ranks agree. Returns (normed, residual + sum).""" + rows = x.shape[0] + path = DRAFT_MC_AR.path + if DRAFT_MC_AR.covers(rows): + workspace = m3_mc_allreduce.get_workspace(self.mapping, x.shape[-1]) + out = None + if workspace is not None: + _record("m3_draft_mc_ar", rows, pushed) + out = m3_mc_allreduce.m3_mc_allreduce_norm( + x.contiguous(), + residual.contiguous(), + draft_model.w[norm_key], + draft_model.rms_eps, + workspace, + use_pdl=get_env_enable_pdl(), + drafter_norm=True, + carveout=carveout, + prepushed=2 if pushed else 0, + ) + if out is not None: + _log_engaged(path, self.mapping) + if pushed: + taken(DRAFT_AR_PUSH_PATH, rows, "the projection pushed its rows (prepushed 2)") + elif DRAFT_AR_PUSH: + uncovered( + DRAFT_AR_PUSH_PATH, + "the boundary's projection did not push (the AR pushes x)", + ) + return out + if pushed: + raise RuntimeError( + "the drafter's rows were pushed into a boundary all-reduce that did not run" + ) + missed(path, rows, "the multicast all-reduce workspace or operand layout") + return draft_model.ar_norm(x, residual, norm_key) + + def _m3_arm_draft_push(self, draft_model: M3Eagle3Draft, key: str, rows: int) -> None: + """DRAFT_AR_PUSH: let projection ``key``'s next low-M call push its rows into the boundary all-reduce that + ``_m3_draft_ar_norm`` then runs (the multicast path only; shapes and a collectively created workspace decide + it, so the TP ranks agree).""" + if not (DRAFT_AR_PUSH and DRAFT_MC_AR.covers(rows)): + return + workspace = m3_mc_allreduce.get_workspace(self.mapping, draft_model.w[key].shape[0]) + if workspace is not None: + draft_model.arm_push(key, workspace, rows) + + def _m3_fused_loop_ok( + self, attn_metadata, spec_metadata, batch_size: int, draft_model: Optional[M3Eagle3Draft] + ) -> bool: + """Whether this step drafts with the fused loop: a decode-only step the fused verify or the seed prepared + (``_m3_first_gather_ids``) with the fused first draft inputs, a supported drafter, a (draft length, requests) + ``DRAFT_STEP`` covers, the prebuilt head and the exchange workspace. Host state only, so the TP ranks + agree.""" + if ( + self._m3_first_gather_ids is None + or not self._m3_fused_first_inputs + or draft_model is None + ): + return False + path = DRAFT_STEP.path + if not self._m3_drafter_supported(draft_model) or not hasattr( + attn_metadata, "kv_lens_cuda" + ): + uncovered( + path, "the drafter or its attention metadata is not one the fused loop serves" + ) + return False + draft_len = spec_metadata.runtime_draft_len + if not DRAFT_STEP.covers((draft_len, batch_size)): + uncovered(path, f"draft length {draft_len} at {batch_size} requests") + return False + head = self._m3_draft_head + if head is None or head.weight is not draft_model.lm_head.weight: + missed( + path, + batch_size, + "the drafter head's streamed copy is not built for its current weight (prebuild)", + ) + return False + if m3_draft_step_kernel.workspace(self.mapping, head.wf.device) is None: + capture_missed(path, None, "the draft-step exchange workspace") + return False + _log_engaged(path, self.mapping) + return True + + def _forward_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ): + """The fused draft loop on a step it covers (``_m3_fused_loop_ok``), else the in-tree loop.""" + if self._m3_fused_loop_ok(attn_metadata, spec_metadata, batch_size, draft_model): + return self._m3_fused_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + batch_size, + num_accepted_tokens, + ) + return super()._forward_draft_loop( + inputs, + attn_metadata, + spec_metadata, + draft_model, + draft_kv_cache_manager, + num_contexts, + num_gens, + batch_size, + num_accepted_tokens, + original_all_rank_num_tokens, + resource_manager, + ) + + def _m3_fused_draft_loop( + self, + inputs, + attn_metadata, + spec_metadata, + draft_model: M3Eagle3Draft, + draft_kv_cache_manager, + batch_size: int, + num_accepted_tokens, + ): + """Mirrors Eagle3OneModelWorker._forward_linear_draft_loop for a decode-only Eagle3 step, with each pass's + glue fused (see ``_m3_fused_loop_ok``): + + * the draft input -- the embedding lookup, the layer's input norm and hidden norm and their concat -- in one + launch (``m3_spec_tail.draft_input``; the second pass gathers the last accepted rows there); + * the drafter layer's attention and MLP (its SwiGLU on this target's C++ op), each followed by the all-reduce + with the residual add and the next RMSNorm (the post-attention norm, then the drafter's final norm): the + multicast boundary fed by the o_proj / down projection's push where covered (``_m3_draft_ar_norm``); + * one kernel for the BF16 head over the head rows (pass 0: the last accepted row of each request), the + cross-rank argmax and the bookkeeping the in-tree loop does between passes: the draft token, the next pass's + input id and position, the draft KV lengths (and pass 0's sequence lengths) (``m3_draft_step``). + + The host metadata transitions are the in-tree loop's. The drafts land in columns 1.. of the step's next-input + tokens (column 0 is the last accepted token), which ``_forward_impl`` returns as they lie. + """ + draft_len = spec_metadata.runtime_draft_len + table = draft_model.embed_table + dev = table.device + head = self._m3_draft_head + gather_ids = self._m3_first_gather_ids + ids = inputs["input_ids"] + hidden = inputs["hidden_states"] + pos = inputs["position_ids"] + if ids.dtype != torch.int32: + ids = ids.to(torch.int32) + if pos.dtype != torch.int32: + pos = pos.to(torch.int32) + num_acc = num_accepted_tokens[:batch_size] + if num_acc.dtype != torch.int32: + num_acc = num_acc.to(torch.int32) + next_new = self._m3_next_new + if next_new is None: + # Seeded from the in-tree acceptance: the last accepted token is the first-pass input at the gather row. + next_new = torch.empty((batch_size, draft_len + 1), dtype=torch.int32, device=dev) + next_new[:, 0] = ids[gather_ids] + draft_tokens = next_new[:, 1:] + w = draft_model.w + eps = draft_model.rms_eps + + with self.draft_kv_cache_context(attn_metadata, draft_kv_cache_manager) as md: + for i in range(draft_len): + x, residual = m3_spec_tail.draft_input( + ids, + table, + hidden, + gather_ids if i == 1 else None, + w["in_norm"], + eps, + w["hidden_norm"], + eps, + ) + if i == 0: + self._m3_join_pending() + rows = x.shape[0] + self._m3_arm_draft_push(draft_model, "o", rows) + h = draft_model.attention(x, md, reduce=False) + h, residual = self._m3_draft_ar_norm( + draft_model, h, residual, "post_norm", pushed=draft_model.take_pushed(h) + ) + self._m3_arm_draft_push(draft_model, "dn", rows) + h = draft_model.mlp(h, reduce=False, cpp_silu=True) + h, to_save = self._m3_draft_ar_norm( + draft_model, + h, + residual, + "final_norm", + carveout=DRAFT_MC_AR_CARVEOUT, + pushed=draft_model.take_pushed(h), + ) + if draft_model.return_hidden_post_norm: + to_save = h + ids_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + pos_next = torch.empty(batch_size, dtype=torch.int32, device=dev) + if not m3_draft_step_kernel.draft_step( + head, + h, + batch_size, + gather_ids if i == 0 else None, + self.mapping, + draft_tokens, + i, + ids_next, + pos, + pos_next, + md.kv_lens_cuda, + num_acc if i == 0 else None, + md._seq_lens_cuda if i == 0 else None, + draft_len, + ): + raise RuntimeError( + "M3 fused draft step unavailable after _m3_fused_loop_ok() accepted the step" + ) + ids, pos, hidden = ids_next, pos_next, to_save + if i == 0: + # The draft step set the device sequence lengths to 1 and rolled the KV lengths back. + md._seq_lens[:batch_size].fill_(1) + md.on_update() + has_kv_cache = md.kv_cache_manager is not None + if has_kv_cache and hasattr(md, "host_request_types"): + md.host_request_types[: md.num_contexts].fill_(1) + if has_kv_cache: + md.num_contexts = 0 + self._prepare_flash_mla_generation_layout(md, 0, batch_size) + md.update_for_spec_dec() + md.use_spec_decoding = False + else: + md.update_for_spec_dec() + self._m3_next_new_tokens = next_new + return draft_tokens + + +def _build_speculation(shell, model_config: ModelConfig): + """The drafter config, the drafter and the worker of one-model Eagle3 (the in-tree one-engine construction for + the llama3 drafter architecture), or of one-model standalone DSpark (``dspark.build_dspark``: the in-tree drafter + and worker).""" + spec = model_config.spec_config + if _dspark.is_standalone_dspark(spec): + shell.model.embed_tokens = _dspark.M3EmbedTokens(shell.model) + return _dspark.build_dspark(shell, model_config) + if ( + not spec.spec_dec_mode.use_one_engine() + or not spec.spec_dec_mode.is_eagle3_one_model() + or spec.eagle3_model_arch != "llama3" + or getattr(spec, "use_dynamic_tree", False) + ): + raise ValueError( + "this MiniMax-M3 target speculates with linear one-model Eagle3 (llama3 drafter) only" + ) + draft_config = ModelConfig.from_pretrained( + spec.speculative_model, + trust_remote_code=True, + attn_backend=model_config.attn_backend, + moe_backend=_get_requested_draft_moe_backend(model_config, spec), + mapping=model_config.mapping, + spec_config=model_config.spec_config, + max_num_tokens=model_config.max_num_tokens, + moe_max_num_tokens=model_config.moe_max_num_tokens, + ) + _set_draft_kv_cache_quant_algo(draft_config, model_config) + draft_config.extra_attrs = model_config.extra_attrs + use_separate = should_use_separate_draft_kv_cache(spec) + draft = M3Eagle3Draft( + shell.model, + draft_config, + model_config.pretrained_config.num_hidden_layers, + spec.num_capture_layers, + ) + worker = M3Eagle3Worker(spec, model_config.mapping, use_separate_draft_kv_cache=use_separate) + worker.set_draft_model(draft) + shell.epilogue.append(draft) + shell.epilogue.append(worker) + return draft_config, draft, worker, use_separate + + +# -------------------------------------------------------------------------------------------------------------------- +# Runtime hooks and the shell +# -------------------------------------------------------------------------------------------------------------------- + + +# TRTLLM_M31_DISABLE_HOSTTAIL=1 turns off the six items of the sm_103 targets' HOSTTAIL bundle together (HOST_D2H_SIDE, +# SMALL_H2D, BATCH_CHANGED_ONLY, SKIP_ACCEPTED_UPLOAD, PHILOX_PACK, KV_LENS_ONCE below), as there. +_HOSTTAIL_OFF = m31_disabled("HOSTTAIL") +# SKIP_ACCEPTED_UPLOAD (RuntimeHooks.skip_accepted_draft_upload): the step's per-request accepted-draft +# counts are no longer uploaded and spec_metadata.num_accepted_draft_tokens is None. Nothing on this target reads that +# tensor (modeling.py, dspark.py, dspark_s4.py, mxdraft.py, kernels/, ops/; the in-tree DFlash / DSpark workers and +# the Eagle3 path: tests/unittest/_torch/executor/test_skip_accepted_draft_upload.py scans for readers). Kill switch: +# TRTLLM_M31_DISABLE_SKIP_ACCEPTED_UPLOAD=1 (the stock per-step upload, packed under spec_small_h2d). +STEP_SKIP_ACCEPTED_UPLOAD = not (_HOSTTAIL_OFF or m31_disabled("SKIP_ACCEPTED_UPLOAD")) +# BATCH_CHANGED_ONLY (RuntimeHooks.spec_batch_map_changed_only): the standalone DSpark (DFlash) +# drafter's batch_indices and batch-to-slot map are uploaded only when they change (both constant in steady decode); +# the buffers hold the same values as with the stock every-step copies. Kill switch: +# TRTLLM_M31_DISABLE_BATCH_CHANGED_ONLY=1 (both copied every step). +STEP_BATCH_CHANGED_ONLY = not (_HOSTTAIL_OFF or m31_disabled("BATCH_CHANGED_ONLY")) +# The rest of the step loop's host-side items, each default on with its own kill switch; every one is exact (the same +# values reach the device, only the copies' stream, size, grouping or timing change): +# * HOST_D2H_SIDE (RuntimeHooks.sampler_d2h_side_stream): the spec sampler's store copies to the host on a side +# stream. TRTLLM_M31_DISABLE_HOST_D2H_SIDE=1: the stock copies on the main stream (the spec fold's store_copies_done +# event is then never re-recorded, so its tail wait is satisfied, and stream order keeps the copies ahead of it). +# * SMALL_H2D (spec_small_h2d): the sampler's slot ids stay in a device buffer re-uploaded only on change and the +# accepted counts ride in the step-input pack. TRTLLM_M31_DISABLE_SMALL_H2D=1: the stock per-step uploads. +# * PHILOX_PACK (pack_spec_rng_uploads): sampled batches stage their Philox seed/offset uploads into the step-input +# pack. TRTLLM_M31_DISABLE_PHILOX_PACK=1: the stock separate uploads. +# * KV_LENS_ONCE (kv_lens_update_once): a speculative overlap step re-derives the MSA KV-length metadata once, after the +# KV-length correction. TRTLLM_M31_DISABLE_KV_LENS_ONCE=1: both passes. +# * PACK_PSCATTER (pack_parallel_scatter): the pack's non-overlapping copies scatter with a program per copy. +# TRTLLM_M31_DISABLE_PACK_PSCATTER=1: the serial one-CTA scatter (and, as below, no in-graph pack). +# * PACK_IN_GRAPH (pack_in_graph; needs PACK_PSCATTER): the decode graph's prologue issues the pack (and, with it, the +# step epoch). TRTLLM_M31_DISABLE_PACK_IN_GRAPH=1: the eager upload and scatter between the graphs. +# * PACK_EARLY_COPY (pack_early_copy; needs PACK_IN_GRAPH): the prologue's window is uploaded on a side copy stream +# while the previous graph runs, and the prologue is this target's one launch (m3_step_prologue). +# TRTLLM_M31_DISABLE_PACK_EARLY_COPY=1: the prologue copies the pinned window itself (scatter launches + fill). +# * H2D_CHANGED_SPAN (h2d_changed_span): a changed packed 2-D upload stages only each row's changed column span. +# TRTLLM_M31_DISABLE_H2D_CHANGED_SPAN=1: the whole view. +STEP_HOST_D2H_SIDE = not (_HOSTTAIL_OFF or m31_disabled("HOST_D2H_SIDE")) +STEP_SMALL_H2D = not (_HOSTTAIL_OFF or m31_disabled("SMALL_H2D")) +STEP_PHILOX_PACK = not (_HOSTTAIL_OFF or m31_disabled("PHILOX_PACK")) +STEP_KV_LENS_ONCE = not (_HOSTTAIL_OFF or m31_disabled("KV_LENS_ONCE")) +STEP_PACK_PSCATTER = not m31_disabled("PACK_PSCATTER") +STEP_PACK_IN_GRAPH = STEP_PACK_PSCATTER and not m31_disabled("PACK_IN_GRAPH") +STEP_PACK_EARLY_COPY = STEP_PACK_IN_GRAPH and not m31_disabled("PACK_EARLY_COPY") +STEP_H2D_CHANGED_SPAN = not m31_disabled("H2D_CHANGED_SPAN") +# BLOCKOFF_GUARD (RuntimeHooks.skip_unchanged_draft_block_offsets): the standalone DSpark drafter's separate KV cache +# manager skips its per-step block-table copy (one launch between the step graphs, full max_seq_len width) when the +# copied rows' page indices are unchanged since its last copy into the same buffer; a skipped step leaves the bytes +# the copy would write. TRTLLM_M31_DISABLE_BLOCKOFF_GUARD=1 copies every step. +STEP_BLOCKOFF_GUARD = not m31_disabled("BLOCKOFF_GUARD") +# BLOCKOFF_PATCH (RuntimeHooks.patch_unchanged_draft_block_offsets; needs BLOCKOFF_GUARD): the drafter's sliding window +# changes one page index per row on about a third of the decode steps (a tail page allocated or re-allocated, a front +# page released); such a step rewrites the copied rows with the copy kernel's int32 transform done on the host and one +# host-to-device copy, instead of the full-width copy kernel. TRTLLM_M31_DISABLE_BLOCKOFF_PATCH=1 copies those steps. +STEP_BLOCKOFF_PATCH = STEP_BLOCKOFF_GUARD and not m31_disabled("BLOCKOFF_PATCH") +# SLOT_IDS_CHANGED (spec_slot_ids_changed_only; ported from the sm_103 targets): the spec +# metadata's per-request slot-id table (batch_slot_ids; rejection sampling) is re-uploaded only when the slot ids +# change, not copied every step. With BLOCKOFF_GUARD it empties the eager step-boundary stream the next decode graph +# waits on. OFF by default on this target: its token-identity gate c failed here (259 / 272 requests identical, the +# rest diverging mid-sequence), so the skip misses a slot-id change on this step loop. Opt in with +# TRTLLM_M31_ENABLE_SLOT_IDS_CHANGED=1; TRTLLM_M31_DISABLE_SLOT_IDS_CHANGED=1 still wins. Read at import. +SLOT_IDS_CHANGED = m31_enabled("SLOT_IDS_CHANGED") and not m31_disabled("SLOT_IDS_CHANGED") +# TAILBK (the generic sites; RuntimeHooks.spec_tail_bookkeep -> spec_metadata.tail_bookkeep): +# the one-model sampler tail's bookkeeping launches removed, bitwise the same tokens, probs and counts (c1 K7 advanced +# graph on GB: ~19 attributed launches per step; VR not yet counted). Five items, all on this switch: +# * ``R1`` (dspark.M3DSparkWorker): at one decode request the accept and draft-block Philox rows come from the +# per-token seed / offset window with one int64 add, instead of the per-request expansion (8 launches). +# * ``B1`` (interface.advanced_sample_draft_block): at one decode request the draft block's temperature / top-k / top-p +# / min-p are views of the per-token buffers instead of four repeat copies. +# * ``Z1`` (dspark.M3DSparkWorker): no padding-row zeroing while a batch-1 decode graph is captured (it replays one real +# request, never a dummy, so the zeroing is a copy). +# * ``S1`` (interface.sample_draft_tokens): a draft block with no d2t map skips the int32 -> int64 -> int32 round trip. +# * ``A1`` (interface rejection accept): a decode-only accept returns the acceptance kernel's outputs instead of a ones +# fill and two copies into fresh buffers. +# Default ON; TRTLLM_M31_DISABLE_TAILBK=1 restores every in-tree line. Read once, at import. +TAILBK = not m31_disabled("TAILBK") +_TAILBK_LOGGED: list = [] + + +def _log_tailbk(worker) -> None: + """The TAILBK engagement line, once per process when the model arms it: the items its worker can take (the DSpark + worker all five; another worker's accept only A1). Nothing when TAILBK is off or without speculation.""" + if not TAILBK or worker is None or _TAILBK_LOGGED: + return + _TAILBK_LOGGED.append(True) + items = "R1 B1 Z1 S1 A1" if isinstance(worker, _dspark.M3DSparkWorker) else "A1" + logger.info(f"[M3 host] tail bookkeeping: {items}") + + +def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: + """The step-loop hooks this target opts into. + + * ``pack_step_inputs``: a decode step's small input copies go out as one upload and one scatter at the graph + replay instead of a chain of copies the next graph waits on; the scatter is parallel (a program per copy), issued + by the decode graph's own prologue (this target's one-launch C++ op, ``m3_step_prologue``) from a window copied + on a side stream while the previous graph runs, so no operation is left between the step graphs and the replay + does not start with a host copy (its copy-engine -> compute turnaround was 10.3 us at c1). Sampled speculative + batches stage their Philox seed/offset uploads, and the accepted-draft counts, into the same pack; the sampler's + slot ids stay in a device buffer re-uploaded only when they change. + * ``pack_step_epoch``: the step epoch (step_inputs.step_epoch), one int64 device counter the decode graph's + prologue increments once per replay (program 0 of its first scatter launch, no extra kernel) and never resets. + The per-layer megakernel pieces tag their per-step device flags with it (contract v1: tag = epoch << 8 | layer). + * ``skip_unchanged_h2d``: steady decode re-uploads slot indices, query lengths and page tables that rarely change. + * ``h2d_changed_span``: a changed packed 2-D metadata upload stages only each row's changed column span. On a step + where a request crosses into a new page the MSA block table changed by one entry, but the whole padded table + (32 KB a request) was staged, overflowing the 16 KB in-graph window into an eager upload and scatter between the + step graphs (TP4 57-69 us at the ~1 in 19 block-crossing steps). + * ``kv_lens_update_once``: a speculative overlap step re-derives the MSA KV-length metadata once, after the + forward's KV-length correction (the first pass is rewritten in full by the second). + * ``spec_offsets_op``: the overlap scheduler's speculative step correction (position_ids += offsets, kv_lens += + offsets) as this target's one plain launch (``m3_spec_offsets``) instead of two ATen adds at the head of the + decode graph: the same int32 adds. + * ``loop_on_execution_stream``: the overlap loop's own stream work and the forward on one stream, so step N+1's + inputs wait on graph N in program order rather than through two cross-stream event hops. + * ``merge_tuned_tactics_across_ranks``: tensor-parallel peers capture the same GEMM tactics (rank-local profiling + noise otherwise splits them, and the slowest rank sets every step). + * ``persist_flashinfer_mxfp8_tactics``: with TLLM_AUTOTUNER_CACHE_PATH set, the FlashInfer MXFP8 tactics tuned in + warmup and the decode-graph pass are saved with the cache, so a restart reuses them instead of re-profiling. + * ``disagg_draft_kv``: in disaggregated serving the drafter's prompt context window travels with the target's KV, + so a generation worker's drafter does not start from an empty window. + * ``skip_greedy_rng_upload``: all-greedy speculative batches never read the Philox seeds. + * ``spec_step_fold``: the spec sampler's slot stores and the engine's next-step input gathers; the decode forward + stores its outputs and stages the next step's inputs at its own tail (``M3SpecStepFold``). + * ``sampler_d2h_side_stream``: the spec sampler copies its per-slot store to the host on a side stream, overlapping + the next step's input staging. + * ``iteration_timing_one_event``: the iteration log times the device step with one event per boundary on a side + stream (the reported step times are the same). + * ``spec_tail_bookkeep`` (``TAILBK``): the sampler tail's no-op bookkeeping launches dropped, bitwise. + * Kill switches (the ``STEP_*`` constants above; ``HOSTTAIL`` turns off the first six together): + ``sampler_d2h_side_stream`` HOST_D2H_SIDE, ``spec_small_h2d`` SMALL_H2D, ``pack_spec_rng_uploads`` PHILOX_PACK, + ``kv_lens_update_once`` KV_LENS_ONCE, ``skip_accepted_draft_upload`` SKIP_ACCEPTED_UPLOAD, + ``spec_batch_map_changed_only`` BATCH_CHANGED_ONLY, ``pack_parallel_scatter`` PACK_PSCATTER, ``pack_in_graph`` + PACK_IN_GRAPH (with ``pack_step_epoch``), ``pack_early_copy`` PACK_EARLY_COPY (with ``pack_prologue_op``), + ``h2d_changed_span`` H2D_CHANGED_SPAN. + * ``skip_accepted_draft_upload`` / ``spec_batch_map_changed_only`` (kill switches ``SKIP_ACCEPTED_UPLOAD`` / + ``BATCH_CHANGED_ONLY``, ``STEP_SKIP_ACCEPTED_UPLOAD`` / ``STEP_BATCH_CHANGED_ONLY``): no per-step upload of the + accepted-draft counts (no reader here), and the drafter's batch_indices / batch-to-slot map uploaded only on + change. + * MSA: identity selections for short sequences (and short/long graph families), the one-launch KV-length update, + unchanged block-table copies skipped, pool views built once, block-id tables copied from the page-index views + (the stock builder makes a Python list of the whole padded row: on the Vera CPU 282 us per call at one request + and 561 us at two, against 8 and 13 us). + """ + logger.info(f"[M3.1] SLOT_IDS_CHANGED (BP-3) {'on' if SLOT_IDS_CHANGED else 'off'}") + return RuntimeHooks( + fp8_saturating_casts=True, + msa_kv_cache_fixes=True, + draft_pool_fixes=True, + first_content_emit_iters=FIRST_CONTENT_EMIT_ITERS, + mxfp8_pinned_tactics=True, + late_compile_detect=True, + pack_step_inputs=True, + pack_parallel_scatter=STEP_PACK_PSCATTER, + pack_in_graph=STEP_PACK_IN_GRAPH, + pack_early_copy=STEP_PACK_EARLY_COPY, + pack_step_epoch=True, + pack_prologue_op=m3_step_prologue, + skip_unchanged_h2d=True, + kv_lens_update_once=STEP_KV_LENS_ONCE, + spec_offsets_op=m3_spec_offsets, + h2d_changed_span=STEP_H2D_CHANGED_SPAN, + loop_on_execution_stream=True, + merge_tuned_tactics_across_ranks=True, + persist_flashinfer_mxfp8_tactics=True, + # Bundled name-keyed tactic pins (m31_pins); kill: TRTLLM_M31_DISABLE_BUNDLED_PINS=1. + install_tactic_pins=lambda rank: m31_pins.install("tp8", rank), + disagg_draft_kv=True, # kill: TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV=1 + skip_greedy_rng_upload=True, + pack_spec_rng_uploads=STEP_PHILOX_PACK, + spec_small_h2d=STEP_SMALL_H2D, + skip_accepted_draft_upload=STEP_SKIP_ACCEPTED_UPLOAD, + spec_batch_map_changed_only=STEP_BATCH_CHANGED_ONLY, + spec_slot_ids_changed_only=SLOT_IDS_CHANGED, + spec_step_fold=spec_fold, + sampler_d2h_side_stream=STEP_HOST_D2H_SIDE, + skip_unchanged_draft_block_offsets=STEP_BLOCKOFF_GUARD, + patch_unchanged_draft_block_offsets=STEP_BLOCKOFF_PATCH, + spec_tail_bookkeep=TAILBK, + iteration_timing_one_event=True, + msa=MsaRuntimePolicy( + short_sequence_identity=True, + kv_update=_msa_kv_update_kernel.msa_kv_update, + skip_unchanged_block_offsets=True, + reuse_pool_views=True, + block_ids_from_views=True, + forward_staging=True, + ), + ) + + +# TOPP_DET (ported from the sm_103 targets): the spec sampler's top-p renorm (FlashInfer +# top_p_renorm_probs in the no-top-k rejection path, the draft's and the verify's call each step) runs as one +# persistent C++ launch (ops/sampling/m31_topp_det, kernels/minimaxM31/toppRenormDetSm103.cu, built for sm_107 +# in-tree or overlaid as tensorrt_llm/libs/libm31_topp_det.so) through the sampler's top-p renorm override, instead of +# FlashInfer's five launches with a one-CTA-per-row apply (AirTopPRenormApply: 63 us per call, 126 us per step at TP8 +# K7 c1 on VR). +# Bitwise FlashInfer's deterministic renorm on rows with top_p < 1; rows with top_p >= 1 keep every token +# (an approved exception: FlashInfer drops tail buckets there in ~43% of rows). Taken up to ``_topp_det.MAX_ROWS`` +# rows; larger calls, inputs outside the contract and builds without the op keep the stock call. Kill switch: +# TRTLLM_M31_DISABLE_TOPP_DET=1 (stock FlashInfer top_p_renorm_probs). +TOPP_DET = not m31_disabled("TOPP_DET") +_TOPP_DET_STATE: dict = {} + + +def _install_topp_det() -> None: + """Once per process: install ``_topp_det.top_p_renorm_det`` as the sampler's top-p renorm override when TOPP_DET + is on and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _TOPP_DET_STATE: + return + if not TOPP_DET: + state = "off (TRTLLM_M31_DISABLE_TOPP_DET=1); stock FlashInfer top_p_renorm_probs" + elif _topp_det.topp_det_op() is None: + state = "not taken: the op is not built; stock FlashInfer top_p_renorm_probs" + else: + set_top_p_renorm_override(_topp_det.top_p_renorm_det) + ctas = _topp_det.ctas_per_row(_topp_det.MAX_ROWS, _topp_det.sm_count(torch.device("cuda"))) + state = f"engaged (ctas_per_row={ctas}, rows <= {_topp_det.MAX_ROWS})" + _TOPP_DET_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01 TOPP_DET det top-p renorm: {state}") + + +def _prebuild_topp_det() -> None: + """The override's persistent workspace on this rank's device, allocated before warmup and any graph capture.""" + if _TOPP_DET_STATE.get("installed"): + _topp_det.prebuild() + + +# SOFTMAX_MC (ported from the sm_103 targets): the spec sampler's softmax (FlashInfer +# softmax(logits, temperature) in softmax_op and compute_probs_from_logits_op, the draft's and the verify's call each +# step) keeps FlashInfer's Map kernel and runs its Reduce kernel's write pass on ``_softmax_mc.WRITE_CTAS`` CTAs per row +# (ops/sampling/m31_softmax_mc, kernels/minimaxM31/softmaxMcSm103.cu, built for sm_107 in-tree or overlaid as +# tensorrt_llm/libs/libm31_softmax_mc.so) through the sampler's softmax override, instead of one 1024-thread CTA per +# row. Bitwise FlashInfer's softmax on its Path A shapes (rows <= 128, vocab >= 24576; GB300 microbench at +# write CTAs 4 / 8 / 16). GB300 per call at 7-8 rows: 9.2 vs 21.8 us. Taken up to ``_softmax_mc.MAX_ROWS`` rows; larger +# calls, inputs outside the contract and builds without the op keep the stock call. Kill switch: +# TRTLLM_M31_DISABLE_SOFTMAX_MC=1 (stock FlashInfer softmax). +SOFTMAX_MC = not m31_disabled("SOFTMAX_MC") +_SOFTMAX_MC_STATE: dict = {} + + +def _install_softmax_mc() -> None: + """Once per process: install ``_softmax_mc.softmax_mc`` as the sampler's softmax override when SOFTMAX_MC is on + and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _SOFTMAX_MC_STATE: + return + if not SOFTMAX_MC: + state = "off (TRTLLM_M31_DISABLE_SOFTMAX_MC=1); stock FlashInfer softmax" + elif _softmax_mc.softmax_mc_op() is None: + state = "not taken: the op is not built; stock FlashInfer softmax" + else: + set_softmax_override(_softmax_mc.softmax_mc) + state = f"engaged (write_ctas={_softmax_mc.WRITE_CTAS}, rows <= {_softmax_mc.MAX_ROWS})" + _SOFTMAX_MC_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01-SM softmax multi-CTA write: {state}") + + +def _prebuild_softmax_mc() -> None: + """The override's persistent partials workspace on this rank's device, allocated before warmup and any graph + capture.""" + if _SOFTMAX_MC_STATE.get("installed"): + _softmax_mc.prebuild() + + +# REJ_MC (ported from the sm_103 targets): the spec sampler's rejection step +# (trtllm::chain_speculative_sampling in speculative/interface.rejection_sampling_one_model, once per verify step) runs +# on ctas_per_row CTAs per request (ops/sampling/m31_chain_spec_mc, kernels/minimaxM31/chainSpecSamplingMcSm103.cu, +# built for sm_107 in-tree or overlaid as tensorrt_llm/libs/libm31_chain_spec_mc.so) through the rejection sampler's +# override, instead of one 1024-thread CTA per request. ctas_per_row is sized from this device's SM count (the CTAs +# of a request meet at global-memory barriers, so batch x ctas_per_row stays co-resident). Bitwise the stock op's +# tokens and counts (GB300 microbench: K7, batch 1 / 2 / 4 / 8 x 5 cases x 64 seeds at ctas_per_row +# 4 / 8 / 16, eager and graph). GB300 per call at batch 1: 31.2 vs 41.2 us. Taken up to ``_chain_spec_mc.MAX_BATCH`` +# requests; larger calls, inputs outside the contract and builds without the op keep the stock op. Kill switch: +# TRTLLM_M31_DISABLE_REJ_MC=1 (stock trtllm::chain_speculative_sampling). +REJ_MC = not m31_disabled("REJ_MC") +# REJ_SLOTS (VR only): on top of REJ_MC, the rejection step reads each request's draft +# distributions in place from the spec metadata's slot-indexed buffer (draft_probs[slot, k, :]) instead of the +# sampler first gathering them into a contiguous copy (one ~5.6 MB index_elementwise per step at c1). The same floats, +# so bitwise the gathered call. OFF by default on this target: it is exact but measured flat at c1 (the in-graph gather +# is cheap here). Opt in with TRTLLM_M31_ENABLE_REJ_SLOTS=1; TRTLLM_M31_DISABLE_REJ_SLOTS=1 still wins (the +# gather + REJ_MC). +REJ_SLOTS = m31_enabled("REJ_SLOTS") and not m31_disabled("REJ_SLOTS") +_REJ_MC_STATE: dict = {} + + +def _install_rej_mc() -> None: + """Once per process: install ``_chain_spec_mc.chain_spec_mc`` as the rejection sampler's override when REJ_MC is + on and the op is built, and log the engagement (or why it is not taken) once per rank.""" + if _REJ_MC_STATE: + return + if not REJ_MC: + state = "off (TRTLLM_M31_DISABLE_REJ_MC=1); stock trtllm::chain_speculative_sampling" + elif _chain_spec_mc.chain_spec_mc_op() is None: + state = "not taken: the op is not built; stock trtllm::chain_speculative_sampling" + else: + set_chain_speculative_sampling_override(_chain_spec_mc.chain_spec_mc) + ctas = _chain_spec_mc.ctas_per_row( + _chain_spec_mc.MAX_BATCH, _chain_spec_mc.sm_count(torch.device("cuda")) + ) + state = f"engaged (ctas_per_row={ctas}, batch <= {_chain_spec_mc.MAX_BATCH})" + slots = REJ_SLOTS and _chain_spec_mc.chain_spec_mc_slots_op() is not None + if slots: + set_chain_speculative_sampling_slot_override(_chain_spec_mc.chain_spec_mc_slots) + state += f"; REJ_SLOTS slot-indexed draft rows {'on' if slots else 'off'}" + _REJ_MC_STATE["installed"] = state.startswith("engaged") + logger.info(f"[M3 dispatch] ST-01-REJ chain speculative sampling multi-CTA: {state}") + + +def _prebuild_rej_mc() -> None: + """The override's persistent workspace on this rank's device, allocated before warmup and any graph capture.""" + if _REJ_MC_STATE.get("installed"): + _chain_spec_mc.prebuild() + + +class ModelingV2MiniMaxM31Nvfp4Sm107Tp8( + DecoderModelForCausalLM[MiniMaxM3Tp8ModelingV2, PretrainedConfig] +): + """MiniMax-M3 NVFP4 / sm_107 / tp8, text only, with one-model Eagle3 or no speculation. + + The shell owns the vocab-parallel BF16 ``lm_head`` and the logits processor (``DecoderModelForCausalLM``); with + speculation, the drafter (``M3Eagle3Draft``, its weights from the speculative checkpoint through + ``load_draft_weights``) and the worker (``M3Eagle3Worker``) that verifies and drafts each step. + """ + + def __init__(self, model_config: ModelConfig[PretrainedConfig]) -> None: + assert torch.cuda.get_device_capability() == _SM, ( + f"this target is built for sm_{_SM[0]}{_SM[1]}, the device is sm_" + f"{''.join(map(str, torch.cuda.get_device_capability()))}" + ) + # Under modeling_v2='require' a covered decode shape that cannot take its fast path raises. + set_require(resolved_mode(type(self)) is ModelingV2Mode.REQUIRE) + model_config = _text_model_config(model_config) + cfg = model_config.pretrained_config + spec_config = getattr(model_config, "spec_config", None) + if ( + model_config.quant_config.kv_cache_quant_algo == QuantAlgo.NVFP4 + and _dspark.draft_kv_override_applies(spec_config) + ): + # The one-model Eagle3 layer has no NVFP4 trtllm-gen decode kernel: keep its modules and the shared + # draft cache on FP8 while the target stays NVFP4. + model_config.extra_attrs["draft_kv_cache_quant_algo_override"] = QuantAlgo.FP8 + super().__init__( + MiniMaxM3Tp8ModelingV2(model_config), + config=model_config, + hidden_size=cfg.hidden_size, + vocab_size=cfg.vocab_size, + ) + self.spec_config = spec_config + # A standalone DSpark drafter takes none of the Eagle3 drafter's hooks (full-context attention, low-M + # linears, boundary capture-norm target). + self._standalone_dspark = _dspark.is_standalone_dspark(spec_config) + self.draft_model: Optional[nn.Module] = None + self.draft_config: Optional[ModelConfig] = None + self.spec_worker: Optional[Union[M3Eagle3Worker, _dspark.M3DSparkWorker]] = None + self.use_separate_draft_kv_cache = False + self.layer_idx = FINAL_HIDDEN_LAYER_ID + self._spec_fold: Optional[M3SpecStepFold] = None + if spec_config is not None: + ( + self.draft_config, + self.draft_model, + self.spec_worker, + self.use_separate_draft_kv_cache, + ) = _build_speculation(self, model_config) + # The DSpark worker keeps the in-tree sampler slot stores; only the engine's input gathers fold (DSFOLD). + if not _dspark.is_standalone_dspark(spec_config): + self._spec_fold = M3SpecStepFold() + elif DSFOLD: + self._spec_fold = M3DSparkStepFold() + self.runtime_hooks = _runtime_hooks(self._spec_fold) + _log_tailbk(self.spec_worker) + _install_topp_det() + _install_softmax_mc() + _install_rej_mc() + + def load_weights(self, weights: Dict, *args, **kwargs) -> None: + _weights.MODEL_WEIGHTS.load(self, weights) + + def load_draft_weights(self, weights: Dict, weight_mapper=None) -> None: + """The drafter's checkpoint (``speculative_model``), loaded by the engine after the target's.""" + if _dspark.is_standalone_dspark(self.spec_config): + _dspark.load_dspark_draft_weights(self, weights, weight_mapper) + return + _weights.DRAFT_WEIGHTS.load(self.draft_model, weights) + + def set_guided_decoder(self, guided_decoder) -> bool: + """Mirrors ``SpecDecOneEngineForCausalLM.set_guided_decoder``: guided decoding with speculation is installed + on the worker.""" + if hasattr(self.spec_worker, "set_guided_decoder"): + return self.spec_worker.set_guided_decoder(guided_decoder) + return False + + def cache_derived_state(self) -> None: + """Build the targets and every decode kernel once the weights are real, before the executor's warmup; with + speculation also the drafter's full-context decode, the fused verify's coded target head, the die-placed + target head copy, the fused draft loop's head copy and the drafter linears' low-M kernels.""" + super().cache_derived_state() + self.model.post_load_weights() + _prebuild_topp_det() + _prebuild_softmax_mc() + _prebuild_rej_mc() + if self.draft_model is not None and not self._standalone_dspark and DRAFTER_FULLCTX.keys: + path = DRAFTER_FULLCTX.path + recipes = {k: FULLCTX_DECODE_RECIPES[k] for k in DRAFTER_FULLCTX.keys} + installed = m3_drafter_attn.install( + self.draft_model, + recipes, + FULLCTX_DECODE_MAX_KV_LEN, + FULLCTX_DECODE_OWN_REDUCER and m3_fullctx_decode_kernel.has_own_reducer(), + get_env_enable_pdl(), + fused_rope_mode=( + DRAFTER_ROPE_MODE + if DRAFTER_ROPE_FUSED + and hasattr(torch.ops.trtllm, "minimax_m3_sm107tp8_drafter_rope_fp8_kv") + else None + ), + on_rope_taken=lambda rows, what: taken(DRAFTER_ROPE_PATH, rows, what), + on_taken=lambda rows: taken(path, rows), + on_uncovered=lambda reason: uncovered(path, reason), + on_missed=lambda rows, reason: missed(path, rows, reason), + ) + logger.info(f"[M3 prebuild] {path} installed on {installed} drafter attention layer(s)") + if self.draft_model is not None and self._standalone_dspark and VR_DRAFT_FMHA_SM100F: + installed = m3_drafter_attn.install_sliding_fi(self.draft_model) + logger.info( + f"[M3 prebuild] VR_DRAFT_FMHA_SM100F: FlashInfer trtllm-gen FMHA installed on {installed} drafter " + "attention layer(s)" + ) + if self.spec_worker is not None: + self.spec_worker.prebuild(self.lm_head, self.draft_model) + + def post_load_weights(self) -> None: + super().post_load_weights() + self.model.post_load_weights() + + def forward( + self, + attn_metadata: AttentionMetadata, + input_ids: Optional[torch.Tensor] = None, + position_ids: Optional[torch.Tensor] = None, + inputs_embeds: Optional[torch.Tensor] = None, + return_context_logits: bool = False, + spec_metadata: Optional[SpecMetadata] = None, + resource_manager: Optional[Any] = None, + **kwargs, + ) -> Any: + """One engine step: the target forward, then the logits (or, with speculation, the worker's verify and + draft, and on a decode step the spec fold's store of its outputs). Text only.""" + multimodal_params = kwargs.pop("multimodal_params", None) or [] + if any(p is not None and getattr(p, "multimodal_data", None) for p in multimodal_params): + raise ValueError( + "this MiniMax-M3 target is text only; the request carries multimodal input" + ) + worker = self.spec_worker + capture = _plain_capture + capture_target = None + if worker is not None: + capture = worker.capture_hidden_states + if not self._standalone_dspark: + capture_target = worker.capture_target + if spec_metadata is not None: + worker.arm_capture_norm( + attn_metadata, spec_metadata, self.lm_head, self.draft_model + ) + hidden_states = self.model( + attn_metadata=attn_metadata, + input_ids=input_ids, + position_ids=position_ids, + inputs_embeds=inputs_embeds, + spec_metadata=spec_metadata, + capture=capture, + capture_target=capture_target, + **kwargs, + ) + if getattr(attn_metadata, "csa2_remote_tail_mode", None) == "source": + # A context worker stopped at the encoder/decoder boundary: a disposable token for the context-only + # completion, and no draft worker. + return self.logits_processor.forward(hidden_states, self.lm_head, attn_metadata, False) + if spec_metadata is not None and spec_metadata.is_layer_capture(FINAL_HIDDEN_LAYER_ID): + capture(spec_metadata, FINAL_HIDDEN_LAYER_ID, hidden_states, None) + if attn_metadata.padded_num_tokens is not None: + hidden_states = hidden_states[: attn_metadata.num_tokens] + if worker is None: + return self.logits_processor.forward( + hidden_states, self.lm_head, attn_metadata, return_context_logits + ) + # The fused verify computes this rank's logits shard itself and needs no gathered logits. + if worker.tail_eligible(attn_metadata, spec_metadata, self.lm_head): + logits = None + else: + # A worker may produce the gathered target logits itself (bitwise to the processor); None = the processor. + target_logits = getattr(worker, "target_logits", None) + logits = ( + target_logits(hidden_states, attn_metadata, spec_metadata, self.lm_head) + if target_logits is not None + else None + ) + if logits is None: + logits = self.logits_processor.forward( + hidden_states[spec_metadata.gather_ids], self.lm_head, attn_metadata, True + ) + spec_input_ids = input_ids if input_ids is not None else kwargs.get("orig_input_ids") + spec_position_ids = position_ids + if attn_metadata.padded_num_tokens is not None: + if spec_input_ids is not None: + spec_input_ids = spec_input_ids[: attn_metadata.num_tokens] + if position_ids is not None: + spec_position_ids = _slice_spec_position_ids(position_ids, attn_metadata.num_tokens) + outputs = worker( + input_ids=spec_input_ids, + position_ids=spec_position_ids, + hidden_states=hidden_states, + logits=logits, + attn_metadata=attn_metadata, + spec_metadata=spec_metadata, + draft_model=self.draft_model, + resource_manager=resource_manager, + ) + spec_fold = self._spec_fold + if ( + spec_fold is not None + and spec_fold.in_forward + and attn_metadata.num_contexts == 0 + and isinstance(outputs, dict) + ): + # The decode forward stores its outputs into the spec sampler's store and stages the next step's inputs + # at its tail (one launch inside the step graph). + spec_fold.store_in_forward(outputs) + return outputs diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/mxdraft.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/mxdraft.py new file mode 100644 index 000000000000..9a77447f5092 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/mxdraft.py @@ -0,0 +1,203 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MXDRAFT: the DSpark drafter's decode projections on this target's MXFP8 GEMV. + +The drafter checkpoint ships its linears as MXFP8 (E4M3 codes, one UE8M0 scale per 32 columns) and the stock drafter +dequantizes them to BF16 at load, so every decode pass streams twice the bytes it needs. MXDRAFT recovers the MXFP8 +form of each BF16 weight (exactly: the recovery is checked to dequantize back to the same BF16 bits, else the +projection is left alone) and routes the projection's GEMM to ``m3_mx_gemv_bf16`` for decode inputs it serves; any +other call runs the Linear unchanged. The activation is quantized to MXFP8 inside the GEMV, so the output is not the +stock BF16 GEMM's: drafter-only precision, gated on natural AL. Row-parallel projections keep +their all-reduce (only the Linear's GEMM is replaced). +""" + +from __future__ import annotations + +from typing import Iterable, Optional, Tuple + +import torch + +from tensorrt_llm.logger import logger + +from .kernels.mx_gemv import op as _gemv +from .ops.comm.m3_mc_allreduce_norm import swizzle_sf + +E4M3_MAX = 448.0 +BLOCK = 32 + + +def mxfp8_from_bf16(weight: torch.Tensor) -> Optional[Tuple[torch.Tensor, torch.Tensor]]: + """The MXFP8 form (E4M3 codes [N, K], UE8M0 scale bytes [N, K / 32]) of a BF16 weight that is exactly an MXFP8 + dequantization, or None. Each block takes the smallest power-of-two scale that fits its amax in E4M3, which is at + most the shipped scale, so the codes are exact whenever the weight came from MXFP8.""" + if weight.dtype != torch.bfloat16 or weight.dim() != 2 or weight.shape[1] % BLOCK != 0: + return None + n, k = weight.shape + blocks = weight.float().view(n, k // BLOCK, BLOCK) + amax = blocks.abs().amax(-1) + exp = torch.ceil(torch.log2(amax.clamp_min(2.0**-126) / E4M3_MAX)).clamp(-127, 127) + scale = torch.exp2(exp).unsqueeze(-1) + codes = (blocks / scale).to(torch.float8_e4m3fn) + back = (codes.float() * scale).view(n, k).to(torch.bfloat16) + if not torch.equal(back, weight): + return None + return codes.view(n, k).contiguous(), (exp + 127).to(torch.uint8).contiguous() + + +def _projections(draft_model) -> Iterable[torch.nn.Module]: + for layer in getattr(getattr(draft_model, "model", None), "layers", []) or []: + attn, mlp = getattr(layer, "self_attn", None), getattr(layer, "mlp", None) + for lin in ( + getattr(attn, "qkv_proj", None), + getattr(attn, "o_proj", None), + getattr(mlp, "gate_up_proj", None), + getattr(mlp, "down_proj", None), + ): + if lin is not None: + yield lin + + +def route_mx_linears(draft_model, rows: Iterable[int]) -> int: + """Route each drafter projection whose BF16 weight is an exact MXFP8 dequantization to the MXFP8 GEMV for 2-D + contiguous bf16 inputs of the given row counts (the decode passes'); compiles those kernels now (outside capture). + Returns how many projections were routed.""" + rows = sorted({int(m) for m in rows if 0 < int(m) <= _gemv.MAX_TOKENS}) + n_routed = 0 + for lin in _projections(draft_model): + weight = getattr(lin, "weight", None) + if ( + weight is None + or getattr(lin, "bias", None) is not None + or getattr(lin, "_m31_mxdraft", None) is not None + ): + continue + rec = mxfp8_from_bf16(weight.data) + if rec is None: + continue + n, k = weight.shape + served = tuple(m for m in rows if _gemv.serves(n, k, m, True)) + if not served: + continue + w8, sf = rec[0], swizzle_sf(rec[1]) + for m in served: + _gemv.precompile(w8, sf, m, True) + original = lin.apply_linear + + def apply_linear( + input, + bias, + lora_params=None, + layer_idx=None, + *, + _o=original, + _w8=w8, + _sf=sf, + _served=served, + ): + if ( + bias is None + and not lora_params + and isinstance(input, torch.Tensor) + and input.dtype == torch.bfloat16 + and input.dim() == 2 + and input.shape[0] in _served + and input.is_contiguous() + ): + out = _gemv.m3_mx_gemv_bf16(input, _w8, _sf) + if out is not None: + return out + return _o(input, bias, lora_params, layer_idx) + + lin._m31_mxdraft = (w8, sf, served) + lin.apply_linear = apply_linear + n_routed += 1 + return n_routed + + +# DRAFT_GU_SWIGLU_EPI (F5, exact): the drafter MLP's gate_up GEMV writes swigluoai(gate, up) from its epilogue (the +# interleaved gate_up weight, m3_mx_gemv_gu_swiglu) instead of the bf16 gate | up rows plus the separate Triton +# silu_and_mul launch; bit for bit the same h (tests: test_mxdraft_gu_swiglu.py). Kill switch +# TRTLLM_M31_DISABLE_DRAFT_GU_SWIGLU_EPI=1. Engagement: "[M3.1 mxdraft] DRAFT_GU_SWIGLU_EPI taken rows=" once per +# row count and process, from the code doing the work. +DRAFT_GU_SWIGLU_EPI_PATH = "DRAFT_GU_SWIGLU_EPI" +_GU_SWIGLU = (7.0, 1.702, 1.0) # (limit, alpha, beta) of the kernel's swigluoai +_gu_logged: set = set() + + +def route_gu_swiglu(draft_model, rows: Iterable[int]) -> int: + """Fold each drafter MLP's SwiGLU into its gate_up GEMV epilogue where the MLP is the swigluoai GatedMLP the kernel + implements; compiles those kernels now (outside capture). Returns how many MLPs were routed.""" + import os + + if os.environ.get("TRTLLM_M31_DISABLE_DRAFT_GU_SWIGLU_EPI", "0") == "1": + return 0 + import torch.nn.functional as F + + rows = sorted({int(m) for m in rows if 0 < int(m) <= _gemv.MAX_TOKENS}) + n_routed = 0 + for layer in getattr(getattr(draft_model, "model", None), "layers", []) or []: + mlp = getattr(layer, "mlp", None) + gu = getattr(mlp, "gate_up_proj", None) + if ( + gu is None + or getattr(mlp, "split_gate_up", False) + or getattr(mlp, "activation", None) is not F.silu + or (mlp.swiglu_limit, mlp.swiglu_alpha, mlp.swiglu_beta) != _GU_SWIGLU + or getattr(gu, "bias", None) is not None + or getattr(gu, "_m31_gu_swiglu", None) is not None + or gu.weight.shape[0] % 128 != 0 + ): + continue + rec = mxfp8_from_bf16(_gemv.gu_interleave(gu.weight.data)) + if rec is None: + continue + w8, sf = rec[0], swizzle_sf(rec[1]) + served = tuple( + m for m in rows if _gemv.precompile(w8, sf, m, True, gu_swiglu=True) is not None + ) + if not served: + continue + original = gu.apply_linear + original_act = mlp._apply_activation + + def apply_linear( + input, + bias, + lora_params=None, + layer_idx=None, + *, + _o=original, + _w8=w8, + _sf=sf, + _served=served, + ): + if ( + bias is None + and not lora_params + and isinstance(input, torch.Tensor) + and input.dtype == torch.bfloat16 + and input.dim() == 2 + and input.shape[0] in _served + and input.is_contiguous() + ): + out = _gemv.m3_mx_gemv_gu_swiglu(input, _w8, _sf) + if out is not None: + out._m31_gu_swiglu = True + m = int(input.shape[0]) + if m not in _gu_logged: + _gu_logged.add(m) + logger.info(f"[M3.1 mxdraft] {DRAFT_GU_SWIGLU_EPI_PATH} taken rows={m}") + return out + return _o(input, bias, lora_params, layer_idx) + + def _apply_activation(x, *args, _oa=original_act, **kwargs): + # The GEMV already applied the SwiGLU (its output is h, half the width): pass it through. + if getattr(x, "_m31_gu_swiglu", False): + return x + return _oa(x, *args, **kwargs) + + gu._m31_gu_swiglu = (w8, sf, served) + gu.apply_linear = apply_linear + mlp._apply_activation = _apply_activation + n_routed += 1 + return n_routed diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/__init__.py new file mode 100644 index 000000000000..1059445198db --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Ops used only by this target. Shared ops stay in the catalog.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/__init__.py new file mode 100644 index 000000000000..b8c7426e3933 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Activation ops of this target: the gated MLPs' fused SwiGLU.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/m3_silu_and_mul.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/m3_silu_and_mul.py new file mode 100644 index 000000000000..e7ceefc5bdab --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/m3_silu_and_mul.py @@ -0,0 +1,83 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The gated MLPs' fused SwiGLU: ``silu_and_mul`` with MiniMax-M3's SwiGLU-OAI gain, offset and clamp.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SiluAndMul(OpWrapper): + """``out = g * sigmoid(alpha * g) * (u + beta)`` over ``x = [g | u]`` (bf16 [M, 2I] -> [M, I]), with + ``g = min(gate, limit)`` and ``u = clamp(up, -limit, limit)``; ``limit``, ``alpha`` and ``beta`` None give plain + SwiGLU ``silu(gate) * up``. + + The MXFP8 dense MLPs (layers 0-2, intermediate 6144 per rank) and the MoE's shared expert (intermediate 1536 per + rank) run it at MiniMax-M3's (limit 7, alpha 1.702, beta 1); the BF16 Eagle3 drafter's MLP (intermediate 9216 per + rank) runs it plain. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert, one decode token", spec=dict(m=1, inter=384, oai=True)), + Cell(why="shared expert, a 48-row decode verify", spec=dict(m=48, inter=384, oai=True)), + Cell(why="dense MLP, a prefill chunk", spec=dict(m=2048, inter=1536, oai=True)), + Cell(why="dense MLP, one decode token", spec=dict(m=1, inter=1536, oai=True)), + Cell( + why="drafter MLP (plain SwiGLU), eight decode rows", + spec=dict(m=8, inter=2304, oai=False), + ), + Cell( + why="drafter MLP (plain SwiGLU), a prefill chunk", + spec=dict(m=1024, inter=2304, oai=False), + ), + ) + + note = """ + The kernel computes in fp32 and rounds once to bf16; `reference` does the same arithmetic in fp32, so the two + differ only by the fast sigmoid's last bits (the test's band). + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + + def reference( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> torch.Tensor: + gate, up = x.float().chunk(2, dim=-1) + if limit is not None: + gate = gate.clamp(max=limit) + up = up.clamp(min=-limit, max=limit) + a = 1.0 if alpha is None else alpha + b = 0.0 if beta is None else beta + return (gate * torch.sigmoid(a * gate) * (up + b)).to(x.dtype) + + def is_valid( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> None: + assert x.stride(-1) == 1, "the gate/up halves are read from contiguous rows" + + +m3_silu_and_mul = M3SiluAndMul() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/m3_swq.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/m3_swq.py new file mode 100644 index 000000000000..aeb534e7462b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/activation/m3_swq.py @@ -0,0 +1,162 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SWQ: the shared expert's SwiGLU-OAI written directly as the down projection's MXFP8 activation.""" + +from __future__ import annotations + +from typing import Optional + +import torch +import triton +import triton.language as tl + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +@triton.jit +def _rcp_approx_ftz(x): + return tl.inline_asm_elementwise( + "rcp.approx.ftz.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1 + ) + + +@triton.jit +def _swq_kernel( + x_ptr, + x_stride, + q_ptr, + sf_ptr, + M, + INTER: tl.constexpr, + NB: tl.constexpr, + NB_PAD: tl.constexpr, + P2: tl.constexpr, + swiglu_limit: tl.constexpr, + swiglu_alpha: tl.constexpr, + swiglu_beta: tl.constexpr, + HAS_LIMIT: tl.constexpr, +): + # One program per row of the 128-row-padded scale layout; rows past M only zero their scale bytes. + r = tl.program_id(0) + blk = tl.arange(0, P2) + blk_ok = blk < NB_PAD + # The 128x4 swizzle of trtllm::mxfp8_quantize(x, True) (quantization.cuh get_sf_out_offset_128x4). + sf_off = ( + (blk % 4) + + (blk // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * NB_PAD) + ) + if r >= M: + tl.store(sf_ptr + sf_off, tl.zeros([P2], dtype=tl.uint8), mask=blk_ok) + return + cols = tl.arange(0, P2 * 32) + valid = cols < INTER + row = x_ptr + r.to(tl.int64) * x_stride + # silu_and_mul_kernel's arithmetic (modules/swiglu.py), then its bf16 store. + a = tl.load(row + cols, mask=valid, other=0.0).to(tl.float32) + b = tl.load(row + cols + INTER, mask=valid, other=0.0).to(tl.float32) + if HAS_LIMIT: + a = tl.minimum(a, swiglu_limit) + b = tl.clamp(b, -swiglu_limit, swiglu_limit) + v = (a * tl.sigmoid(swiglu_alpha * a) * (b + swiglu_beta)).to(tl.bfloat16).to(tl.float32) + # cvt_warp_fp16_to_mxfp8: SF = amax * rcp.approx(448), UE8M0 toward +inf, scale = rcp.approx(SF), E4M3 RN. + v2 = tl.reshape(v, (P2, 32)) + amax = tl.max(tl.abs(v2), axis=1) + sf = amax * _rcp_approx_ftz(tl.full([P2], 448.0, tl.float32)) + bits = sf.to(tl.int32, bitcast=True) + exp = (bits >> 23) & 0xFF + mant = bits & 0x7FFFFF + e = tl.where(exp > 0, exp + (mant != 0).to(tl.int32), tl.where(bits > 0x400000, 1, 0)) + e = tl.minimum(e, 254) + # At code 0 the scale 2^-127 is a float subnormal, so rcp.approx.ftz returns inf (native publishes + # inf/NaN there). Use its exact reciprocal 2^127 instead; every other code keeps + # native's rcp.approx (exact on a normal power of two), so each block native emits finite stays bitwise. + pinv = tl.where( + e == 0, 1.7014118346046923e38, _rcp_approx_ftz((e << 23).to(tl.float32, bitcast=True)) + ) + inv = tl.where(amax != 0.0, pinv, 0.0) + q = tl.reshape((v2 * inv[:, None]).to(tl.float8e4nv), (P2 * 32,)) + tl.store(q_ptr + r.to(tl.int64) * INTER + cols, q, mask=valid) + tl.store(sf_ptr + sf_off, tl.where(blk < NB, e, 0).to(tl.uint8), mask=blk_ok) + + +class M3Swq(OpWrapper): + """``silu_and_mul`` (SwiGLU-OAI) over ``x = [g | u]`` (bf16 [M, 2I]) rounded to bf16, then MXFP8-quantized as + ``trtllm::mxfp8_quantize(h, True)``: (E4M3 [M, I], UE8M0 scales in the 128x4-swizzled layout), bit for bit, in one + launch. The TP8 shared expert's down projection (intermediate 384, no GEMV row table) reads it at ``SWQ_ROWS``.""" + + ARCHS = frozenset({Arch.SM_107}) + CELLS: tuple[Cell, ...] = ( + Cell(why="shared expert, the smallest row count it serves", spec=dict(m=9, inter=384)), + Cell( + why="shared expert, a 32-row decode verify (c8; the widest R1-VR-TP8 step)", + spec=dict(m=32, inter=384), + ), + ) + + note = """ + Bitwise to the two-kernel path (silu_and_mul's fp32 arithmetic and bf16 rounding, then mxfp8_quantize's + rcp.approx scale rule): q bytes and every scale byte, padding included. + One deliberate exception: a block whose scale code is 0 (amax / 448 <= 2^-127), where native's + rcp.approx.ftz of the subnormal scale is inf and its e4m3 bytes encode inf/NaN, is quantized with the exact + reciprocal 2^127 instead, so it stays finite; its scale byte is native's. + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> tuple[torch.Tensor, torch.Tensor]: + m, i = x.shape[0], x.shape[1] // 2 + nb = i // 32 + nb_pad = (nb + 3) // 4 * 4 + q = torch.empty(m, i, dtype=torch.float8_e4m3fn, device=x.device) + sf = torch.empty((m + 127) // 128 * 128 * nb_pad, dtype=torch.uint8, device=x.device) + _swq_kernel[((m + 127) // 128 * 128,)]( + x, + x.stride(0), + q, + sf, + m, + INTER=i, + NB=nb, + NB_PAD=nb_pad, + P2=triton.next_power_of_2(nb_pad), + swiglu_limit=limit or 0.0, + swiglu_alpha=alpha if alpha is not None else 1.0, + swiglu_beta=beta if beta is not None else 0.0, + HAS_LIMIT=limit is not None and limit > 0.0, + ) + return q, sf + + def reference( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> tuple[torch.Tensor, torch.Tensor]: + """The two kernels it replaces (bitwise is the contract).""" + h = torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + q, sf = torch.ops.trtllm.mxfp8_quantize(h, True) + return q.view(h.shape), sf + + def is_valid( + self, + x: torch.Tensor, + limit: Optional[float], + alpha: Optional[float], + beta: Optional[float], + ) -> None: + assert x.dtype == torch.bfloat16 and x.stride(-1) == 1, "bf16 [M, 2I] with contiguous rows" + assert (x.shape[1] // 2) % 32 == 0, "I must be a multiple of the 32-element MXFP8 block" + + +m3_swq = M3Swq() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/__init__.py new file mode 100644 index 000000000000..a7f7fee7a234 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Attention ops of this target: MSA block selection, the sparse decode and the KV-length update.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_dense_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_dense_decode.py new file mode 100644 index 000000000000..789ae79db88e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_dense_decode.py @@ -0,0 +1,190 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' decode attention (trtllm-gen over the full page table) with the KV split this target sizes per +request count (modeling's ``DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH``).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.trtllm_gen_dense_decode import ( # noqa: E501 + minimax_m3_trtllm_gen_dense_decode, +) + +__all__ = ["m3_dense_decode"] + +#: The TP4 dense layers' geometry: 16 query heads and one KV head of dim 128 per rank, an FP8 KV pool. +HEADS = 16 +KV_HEADS = 1 +HEAD_DIM = 128 +#: The split this target hands trtllm-gen at one request (modeling's DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH), and a model +#: maximum the in-tree step passes otherwise. +CAPPED = 16384 +MODEL_MAX = 1 << 20 + + +def dense_attention_reference( + q: torch.Tensor, + keys: torch.Tensor, + values: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, +) -> torch.Tensor: + """fp32 causal decode attention: query row ``b * qlen + i`` of request b attends its first + ``seq_lens[b] - qlen + i + 1`` keys. ``keys`` / ``values``: [batch, kv_heads, max_len, head_dim] (the request's + tokens in order). Returns fp32 [total_q, heads, head_dim].""" + batch, kv_heads, max_len, _ = keys.shape + heads = q.shape[1] + group = heads // kv_heads + out = torch.zeros(q.shape, dtype=torch.float32, device=q.device) + pos = torch.arange(max_len, device=q.device) + with true_fp32_matmul(): + for b in range(batch): + full = int(seq_lens[b]) + rows = q[b * decode_query_len : (b + 1) * decode_query_len].float() # [qlen, heads, d] + limit = full - decode_query_len + 1 + torch.arange(decode_query_len, device=q.device) + valid = pos[None, :] < limit[:, None] # [qlen, max_len] + for h in range(kv_heads): + qh = rows[:, h * group : (h + 1) * group] # [qlen, group, d] + logits = torch.einsum("qgd,kd->qgk", qh, keys[b, h].float()) * sm_scale + logits = logits.masked_fill(~valid[:, None, :], float("-inf")) + probs = torch.softmax(logits, dim=-1) + out[ + b * decode_query_len : (b + 1) * decode_query_len, h * group : (h + 1) * group + ] = torch.einsum("qgk,kd->qgd", probs, values[b, h].float()) + return out + + +def gather_paged_kv( + kv_cache_manager, layer_idx: int, block_table: torch.Tensor, seq_lens: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor]: + """The K and V of each request in token order, [batch, kv_heads, max_len, head_dim], from the flat sub-page pool + (``get_kv_subpage_pool``: K of slot s at sub-page s * subpages_per_slot, V one sub-page later).""" + pool, subpages = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + page = pool.shape[2] + pages = (int(seq_lens.max()) + page - 1) // page + slots = block_table[:, :pages].long() + k = pool[slots * subpages] # [batch, pages, kv_heads, page, d] + v = pool[slots * subpages + 1] + batch, _, kv_heads, _, d = k.shape + + def order(t): + return t.permute(0, 2, 1, 3, 4).reshape(batch, kv_heads, pages * page, d) + + return order(k), order(v) + + +class M3DenseDecode(OpWrapper): + """``minimax_m3_trtllm_gen_dense_decode`` as ``MsaDecodeFmha`` calls it for a dense layer, with the max_seq_len + ``msa_decode.dense_decode_max_seq_len`` resolves: this target's per-request-count cap, else the step's bound. + + Decode attention of every query row over its request's full causal context in the paged FP8 pool, written into + ``output`` (BF16 [T, heads, head_dim]) in place. ``max_seq_len`` only sizes trtllm-gen's multi-CTA KV split: + every CTA still covers its share of each request's actual length, which may exceed it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over a 20000-token context, the split sized for 16384", + spec=dict(kv_lens=(20000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request, one row, over a 65536-token context (an AgentX-length cache) at the capped split", + spec=dict(kv_lens=(65536,), qlen=1, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="one request verifying 1 + 3 rows over a 3000-token context: the cap above the length", + spec=dict(kv_lens=(3000,), qlen=4, max_seq_len=CAPPED, capped=True), + ), + Cell( + why="two requests keep the model maximum (no cap at two requests)", + spec=dict(kv_lens=(20000, 5000), qlen=4, max_seq_len=MODEL_MAX, capped=False), + ), + ) + + note = """ + The split only reorders the merge of the CTAs' partial softmax sums, so the capped split's output is the uncapped + one's to a few BF16 ulps (the test holds it there), and both are held to the fp32 reference within the FP8-KV + band of the in-tree dense decode test. + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + minimax_m3_trtllm_gen_dense_decode( + q, + kv_cache_manager, + layer_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + max_seq_len=max_seq_len, + max_num_requests=max_num_requests, + ) + return output + + def reference( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> torch.Tensor: + """fp32 attention over the pool's K / V (q rounded to the pool's dtype first, as the kernel runs it). + + ``output``, ``max_seq_len`` and ``max_num_requests`` do not change the result.""" + keys, values = gather_paged_kv(kv_cache_manager, layer_idx, block_table, seq_lens) + if keys.dtype == torch.float8_e4m3fn: + q = fp8_e4m3_saturate(q) + return dense_attention_reference(q, keys, values, seq_lens, decode_query_len, sm_scale) + + def is_valid( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + max_seq_len: int, + max_num_requests: int, + ) -> None: + # The multi-CTA KV counters are sized for max_num_requests: more requests overrun them. + assert seq_lens.shape[0] <= max_num_requests, "more requests than the counters cover" + # Each verify row attends a causal prefix of at least one key. + assert bool((seq_lens >= decode_query_len).all()), "a request shorter than its query rows" + + +m3_dense_decode = M3DenseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_draft_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_draft_attention.py new file mode 100644 index 000000000000..c2b23aa18750 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_draft_attention.py @@ -0,0 +1,131 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter layer's attention on the TRTLLM backend: RoPE, KV-cache append and causal attention.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + PredefinedAttentionMask, +) +from tensorrt_llm.functional import PositionEmbeddingType, RotaryScalingType + + +class M3DraftAttention(OpWrapper): + """The drafter layer's attention over its fused QKV rows ``qkv`` [T, (q + 2 kv) heads x 128] (bf16): the + backend applies the drafter's RoPE (fused), appends K / V to the drafter layer's paged cache (FP8, the target's + NVFP4 cache keeps its draft layer in FP8) and runs causal attention over the step's live rows (``metadata`` + ``num_tokens``); returns [num_tokens, q heads x 128] bf16. + + ``backend`` (``TrtllmAttention``) and ``metadata`` are the runtime's objects for the drafter layer and step. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="first draft pass of a 2-request verify (2 x 4 rows)", spec=dict(reqs=2, qlen=4)), + Cell(why="a later draft pass, one row per request", spec=dict(reqs=2, qlen=1)), + Cell(why="a context chunk's draft rows", spec=dict(reqs=1, context=512)), + ) + + note = """ + `reference` is exact (float64) causal softmax attention: Q rotated by the backend's NeoX RoPE at each token's + position, over the drafter layer's K / V rows read back from its FP8 cache (dequantized by the backend's + ``kv_scale_quant_orig``). What remains between the two is the FMHA's own arithmetic (Q's rounding to the + kernel's input precision, its P rounding and accumulation order), which the test bounds. + """ + + def __call__(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + num_tokens = metadata.num_tokens + out = backend.forward( + qkv[:num_tokens, :], + None, + None, + metadata, + forward_args=AttentionForwardArgs(attention_mask=PredefinedAttentionMask.CAUSAL), + ) + return out[0] if isinstance(out, tuple) else out + + def reference(self, backend, qkv: torch.Tensor, metadata) -> torch.Tensor: + """The call's [num_tokens, q heads x 128] output, in ``qkv``'s dtype. + + Query token ``j`` of request ``r`` sits at position ``cached_r + j`` (the metadata's cached lengths) and + attends positions up to it. Its Q is rotated here; every K / V row, the step's own included, is read from + the drafter layer's paged cache at the slots the metadata's KV cache manager holds, so this is evaluated + after the call (whose roped, quantized append it then reads back; the test checks that append itself). + Mirrors grouped KV heads (query head h reads KV head h // (heads / kv heads); two per rank at tp2) and + un-scaled NeoX RoPE, the drafter's configuration. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + kv_heads = int(backend.num_kv_heads) + assert heads % kv_heads == 0 + group = heads // kv_heads + rope = backend.rope_params + assert backend.position_embedding_type == int(PositionEmbeddingType.rope_gpt_neox) + assert rope.scale_type == RotaryScalingType.none and rope.alpha == 1.0 + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + kv_scale = float(backend.kv_scale_quant_orig.reshape(-1)[0]) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = int(metadata.num_tokens) + assert sum(qlens) == total + device = qkv.device + + rot = int(rope.dim) + inv_freq = float(rope.theta) ** ( + -torch.arange(0, rot, 2, dtype=torch.float64, device=device) / rot + ) + pool = manager.get_buffers(layer, "HND") + q = qkv[:total, : heads * dim].to(torch.float64).reshape(total, heads, dim) + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][ + : (length + page - 1) // page + ], + dtype=torch.int64, + device=device, + ) + # [kv heads, tokens, dim]: each KV head's rows in token order across the pages. + keys = ( + pool[pages, 0] + .to(torch.float64) + .transpose(0, 1) + .reshape(kv_heads, -1, dim)[:, :length] + * kv_scale + ) + values = ( + pool[pages, 1] + .to(torch.float64) + .transpose(0, 1) + .reshape(kv_heads, -1, dim)[:, :length] + * kv_scale + ) + keys = keys.repeat_interleave(group, dim=0) # [heads, tokens, dim] + values = values.repeat_interleave(group, dim=0) + qpos = cached[row] + torch.arange(n, device=device) + angles = qpos.to(torch.float64)[:, None] * inv_freq[None, :] + cos, sin = angles.cos()[:, None, :], angles.sin()[:, None, :] + rows = q[first : first + n] + x1, x2 = rows[..., : rot // 2], rows[..., rot // 2 : rot] + rows = torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin, rows[..., rot:]], dim=-1) + allowed = torch.arange(length, device=device)[None, :] <= qpos[:, None] + logits = torch.einsum("thd,hkd->thk", rows, keys) * sm_scale + logits = logits.masked_fill(~allowed[:, None, :], float("-inf")) + out[first : first + n] = torch.einsum( + "thk,hkd->thd", torch.softmax(logits, dim=-1), values + ) + first += n + return out.reshape(total, heads * dim).to(qkv.dtype) + + +m3_draft_attention = M3DraftAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_fp8_indexer_qk_norm_rope.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_fp8_indexer_qk_norm_rope.py new file mode 100644 index 000000000000..2fecad0e9afa --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_fp8_indexer_qk_norm_rope.py @@ -0,0 +1,114 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The unfused sparse layers' index branch: per-head Gemma RMSNorm + partial NeoX RoPE of [index-Q | index-K], index-Q +out in FP8 E4M3 and index-K inserted into the FP8 index-K cache, in one kernel.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3Fp8IndexerQkNormRope(OpWrapper): + """``qk`` [T, (heads + 1) * 128] BF16 (the index projection's output): returns index-Q [T, heads, 128] E4M3 and + writes index-K (E4M3) at ``out_cache_loc`` of ``index_k_cache`` [pages, 1, 128, 128]. Gemma RMSNorm of each head, + then RoPE on its leading ``rotary_dim`` channels (NeoX). + + The unfused sparse layers (``fuse_qkv_index_projection=False``) run it on the index branch with an FP8 indexer; + the fused layers' producer writes the index cache itself. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, two index heads per rank", spec=dict(m=1, heads=2)), + Cell(why="an 8-row decode verify", spec=dict(m=8, heads=2)), + Cell(why="a prefill chunk", spec=dict(m=2048, heads=2)), + ) + + note = """ + `reference` is the in-place fused norm + RoPE in BF16 followed by the E4M3 cast (the kernel rounds through BF16 + before the cast); its index-K is what the kernel writes to the cache. The two kernels may round a handful of + values differently: the test allows fewer than 0.1% of the bytes to differ, as the in-tree test does. + """ + + def __call__( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> torch.Tensor: + return torch.ops.trtllm.minimax_m3_fp8_indexer_qk_norm_rope( + qk, + index_k_cache, + out_cache_loc, + num_heads_q, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + position_ids, + ) + + def reference( + self, + qk: torch.Tensor, + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + num_heads_q: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + base: float, + position_ids: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor]: + """(index-Q [T, heads, head_dim] E4M3, index-K [T, head_dim] E4M3): the catalog's in-place kernel on a copy, + then the E4M3 cast.""" + out = fused_qk_norm_rope( + qk.clone(), + num_heads_q, + 1, + 0, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + True, + position_ids, + 1.0, + 0.0, + 0.0, + 1.0, + True, + True, + False, + 0, + 0, + ) + q, k = out.split([num_heads_q * head_dim, head_dim], dim=-1) + return ( + q.reshape(q.shape[0], num_heads_q, head_dim).to(torch.float8_e4m3fn), + k.to(torch.float8_e4m3fn), + ) + + +m3_fp8_indexer_qk_norm_rope = M3Fp8IndexerQkNormRope() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_fullctx_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_fullctx_decode.py new file mode 100644 index 000000000000..a5439efc660b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_fullctx_decode.py @@ -0,0 +1,142 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' full-context decode attention at TP8 on Rubin (``kernels/m3_fullctx_decode``): PrimTS split-KV +decode at the recipe modeling's ``FULLCTX_DECODE_RECIPES`` pins per (query rows per request, requests).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + fp8_e4m3_saturate, +) + +from ...kernels import m3_fullctx_decode as kernel +from .m3_dense_decode import dense_attention_reference, gather_paged_kv + +__all__ = ["m3_fullctx_decode"] + +HEADS = kernel.HEADS +KV_HEADS = kernel.KV_HEADS +HEAD_DIM = kernel.HEAD_DIM +#: The plan capacity (the per-request K/V length bound PrimTS sizes its policy for): dispatch's +#: FULLCTX_DECODE_MAX_KV_LEN, which the forward's prebuilt plans use. +MAX_KV_LEN = 1 << 21 + + +class M3FullCtxDecode(OpWrapper): + """Decode attention of every query row over its request's full causal context in the paged FP8 pool, written + into ``output`` (BF16 [T, heads, head_dim]) in place. + + ``recipe`` is ``modeling's FULLCTX_DECODE_RECIPES[(qlen, requests)]``: the PrimTS Q tile and the split-KV fanout. + ``own_reducer`` merges the splits on the BF16 op (modeling's ``FULLCTX_DECODE_OWN_REDUCER``) rather than PrimTS's + FP16 reducer and a cast. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 1 + 3 rows over an AgentX-length (165k) context", + spec=dict(kv_lens=(165525,), qlen=4), + ), + Cell( + why="one request, one draft-pass row, over a 165k context", + spec=dict(kv_lens=(165522,), qlen=1), + ), + Cell( + why="two requests verifying 1 + 3 rows at unequal lengths (165k, 120k)", + spec=dict(kv_lens=(165525, 120011), qlen=4), + ), + Cell( + why="two requests, one row each, at unequal lengths (165k, 120k)", + spec=dict(kv_lens=(165522, 120008), qlen=1), + ), + Cell( + why="one request verifying 1 + 3 rows over a short (4.1k) context: most splits are empty", + spec=dict(kv_lens=(4104,), qlen=4), + ), + ) + + note = """ + Same operands and arithmetic as the stock trtllm-gen call (E4M3 q, E4M3 K/V, P quantized to E4M3 per split, + fp32 accumulation): the split differs, so the result differs from stock as a stock split change does (the P + quantization is per split). Deterministic, and the same bits eager and in a CUDA graph. Held to the fp32 + reference within the FP8-KV band of the in-tree dense decode test. + """ + + def __call__( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + recipe, + max_kv_len: int = MAX_KV_LEN, + own_reducer: bool = True, + ) -> torch.Tensor: + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + shape = (decode_query_len, int(seq_lens.shape[0])) + kernel.prebuild(q.device, {shape: recipe}, max_kv_len, own_reducer, get_env_enable_pdl()) + plan = kernel.plan_for(q.device, *shape, own_reducer) + pool, subpages = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + k_pages, v_pages = kernel.kv_page_views(pool, subpages) + plan.run(fp8_e4m3_saturate(q), k_pages, v_pages, block_table, seq_lens, sm_scale, output) + return output + + def reference( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + recipe, + max_kv_len: int = MAX_KV_LEN, + own_reducer: bool = True, + ) -> torch.Tensor: + """fp32 attention over the pool's K / V, q rounded to E4M3 first as the kernel runs it.""" + keys, values = gather_paged_kv(kv_cache_manager, layer_idx, block_table, seq_lens) + return dense_attention_reference( + fp8_e4m3_saturate(q), keys, values, seq_lens, decode_query_len, sm_scale + ) + + def is_valid( + self, + q: torch.Tensor, + kv_cache_manager, + layer_idx: int, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + recipe, + max_kv_len: int = MAX_KV_LEN, + own_reducer: bool = True, + ) -> None: + pool, _ = kv_cache_manager.get_kv_subpage_pool(layer_idx, "HND") + assert pool.dtype == torch.float8_e4m3fn, "an FP8 E4M3 pool" + assert tuple(pool.shape[1:]) == (KV_HEADS, kernel.PAGE, HEAD_DIM), ( + "the TP8 dense-layer pool geometry" + ) + assert q.shape[1:] == (HEADS, HEAD_DIM), "8 query heads of 128 per rank" + assert int(q.shape[0]) == int(seq_lens.shape[0]) * decode_query_len, ( + "q covers exactly the batch" + ) + if torch.cuda.is_current_stream_capturing(): + return # the length checks read the device: eager calls only + assert bool((seq_lens >= decode_query_len).all()), "a request shorter than its query rows" + assert bool((seq_lens <= max_kv_len).all()), "a request longer than the plan capacity" + + +m3_fullctx_decode = M3FullCtxDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_index_score.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_index_score.py new file mode 100644 index 000000000000..674b16e7744f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_index_score.py @@ -0,0 +1,167 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# PyTorch oracle ported from vLLM (Apache-2.0), _reference_decode_index_score: +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/tests/kernels/attention/test_minimax_m3.py#L188 +"""The MSA index scorer of a sparse decode step: per (index head, query token, KV block) the causal max of Q . K over +the block's 128 index-K positions, with the pre-wait schedule of ``kernels/m3_index_score``.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_index_score"] + +PAGE = 128 +HEAD_DIM = 128 + + +def compiled_scorer(key: tuple): + """The compiled kernel of ``key`` (dtype, heads, query length, head dim, split_k, early), compiling it on a miss + (the target compiles every key it can take before warmup: ``layers.prebuild_index_scorer``).""" + from ...kernels.m3_index_score import IndexScoreRunner + + if key not in IndexScoreRunner.kernel_cache: + IndexScoreRunner._compile(*key) + return IndexScoreRunner.kernel_cache[key] + + +class M3IndexScore(OpWrapper): + """``score[h, t, j]`` (fp32, a transposed view of the [heads, blocks, tokens] selector buffer) = the max over the + positions ``p`` of block ``j`` of request ``b = t // qlen`` with ``p <= seq_lens[b] - qlen + t % qlen`` of + ``idx_q[t, h] . index_k[block_table[b, j], p % 128]``, for ``j < ceil(seq_lens[b] / 128)`` (no other entry is + written). FP8 or BF16 q and index-K, fp32 products and sums. + + ``early`` (the target's kernel at every request count it covers) issues the Q and first K tiles right after the + grid wait from a pre-wait length hint and prefetches one older block per CTA into L2 before the wait; otherwise + every older block of the CTA is prefetched. Neither, nor ``split_k``, changes a score. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request, 1 + 3 rows over 165521 tokens (AgentX ISL), 1 index head (TP8): 1536 CTAs, early issue", + spec=dict(seq_lens=(165521,), qlen=4, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request of 129 tokens: the early tiles of most of the 1536 CTAs lie past the length and drain", + spec=dict(seq_lens=(129,), qlen=4, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request of 400000 tokens: 3125 blocks, so the 1536 CTAs loop", + spec=dict(seq_lens=(400000,), qlen=4, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request decoding one row over 65536 tokens", + spec=dict(seq_lens=(65536,), qlen=1, heads=1, split_k=1536, early=True), + ), + Cell( + why="one request, 1 + 3 rows over 165521 tokens, 1 index head (TP8): 1272 CTAs (one wave), early issue", + spec=dict(seq_lens=(165521,), qlen=4, heads=1, split_k=1272, early=True), + ), + Cell( + why="one request of 129 tokens at 1272 CTAs: the early tiles of most CTAs lie past the length and drain", + spec=dict(seq_lens=(129,), qlen=4, heads=1, split_k=1272, early=True), + ), + Cell( + why="one request of 400000 tokens at 1272 CTAs: 3125 blocks, so the CTAs loop", + spec=dict(seq_lens=(400000,), qlen=4, heads=1, split_k=1272, early=True), + ), + Cell( + why="two requests verifying 4 rows each over 165521 and 20000 tokens: 768 CTAs each, early issue", + spec=dict(seq_lens=(165521, 20000), qlen=4, heads=1, split_k=768, early=True), + ), + Cell( + why="two requests, one of 300 tokens and one of 262144 (2048 blocks: the 768 CTAs loop)", + spec=dict(seq_lens=(300, 262144), qlen=4, heads=1, split_k=768, early=True), + ), + ) + + note = """ + The products of FP8 (or BF16) values are exact in fp32 and the kernel sums 128 of them in its k-split order, so it + differs from the reference matmul by a few fp32 ulps; it is held to the in-tree scorer bit for bit. + """ + + def __call__( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + _, num_heads, head_dim = idx_q.shape + key = (idx_q.dtype, num_heads, max_decode_query_len, head_dim, split_k, early) + compiled_scorer(key)(idx_q, index_k, block_table, score, seq_lens) + return score + + def reference( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> torch.Tensor: + """The fp32 [heads, total_q, blocks] scores (``-inf`` where nothing is written). ``score``, ``split_k`` and + ``early`` do not change them.""" + total_q, heads, _ = idx_q.shape + qlen = max_decode_query_len + out = torch.full( + (heads, total_q, score.shape[2]), + -float("inf"), + dtype=torch.float32, + device=idx_q.device, + ) + with true_fp32_matmul(): + for b, seq in enumerate(seq_lens.tolist()): + nblocks = (seq + PAGE - 1) // PAGE + if nblocks == 0: + continue + t0 = b * qlen + q = idx_q[t0 : t0 + qlen].float() + k = index_k[block_table[b, :nblocks].long()].reshape(nblocks * PAGE, -1).float() + s = torch.einsum("qhd,kd->hqk", q, k) + q_pos = seq - qlen + torch.arange(qlen, device=idx_q.device) + k_pos = torch.arange(k.shape[0], device=idx_q.device) + s.masked_fill_(k_pos[None, :] > q_pos[:, None], -float("inf")) + out[:, t0 : t0 + qlen, :nblocks] = s.reshape(heads, qlen, nblocks, PAGE).amax(dim=3) + return out + + def is_valid( + self, + idx_q: torch.Tensor, + index_k: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + score: torch.Tensor, + max_decode_query_len: int, + split_k: int, + early: bool, + ) -> None: + # The block-table row and the score row are read and written up to ceil(seq_len / 128) with no bound. + nblocks = (seq_lens + PAGE - 1) // PAGE + assert bool((nblocks <= block_table.shape[1]).all()), ( + "a request runs past its block-table row" + ) + assert bool((nblocks <= score.shape[2]).all()), "a request runs past the score buffer" + assert idx_q.shape[0] == seq_lens.shape[0] * max_decode_query_len, ( + "idx_q must hold max_decode_query_len rows per request" + ) + + +m3_index_score = M3IndexScore() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_msa_attention.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_msa_attention.py new file mode 100644 index 000000000000..f1a7f210a590 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_msa_attention.py @@ -0,0 +1,249 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MiniMax-M3 MSA attention backend's step: KV write, sparse top-k block selection, FMHA.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.attention.backends.interface import ( + AttentionForwardArgs, + SparseBackendForwardArgs, +) + +from .m3_sparse_decode_lc8 import dequant_nvfp4_pages + + +class M3MsaAttention(OpWrapper): + """One layer's attention core on the MSA backend (``MiniMaxM3MsaSparseAttention``), written into ``output`` + [T, heads * 128] (bf16, the step's live rows). + + * ``k`` / ``v`` given (the dense layers): written into the layer's paged main cache first (NVFP4 with + ``kv_scale_orig_quant``, else FP8 / BF16 as the pool is). None: a producer already wrote the caches. + * ``sparse``: the indexer scores the layer's index-K cache (already written) with ``idx_q`` and publishes the + top-k blocks; the FMHA attends those blocks. Otherwise the FMHA attends the whole page table. + + The FMHA dispatches on the step's metadata (context rows: the MSA prefill FMHA; generation rows: the MSA decode + FMHA, the dense layers' trtllm-gen decode at their capped KV split). ``backend`` and ``metadata`` are the runtime's + objects for this layer and step. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="dense layer, a prefill chunk, FP8 pool", + spec=dict(layer="dense", phase="context", tokens=1024), + ), + Cell( + why="dense layer, a decode step of 2 requests", + spec=dict(layer="dense", phase="generation", reqs=2), + ), + Cell( + why="sparse layer, a prefill chunk beyond 16 blocks, NVFP4 pool", + spec=dict(layer="sparse", phase="context", tokens=4096), + ), + Cell( + why="sparse layer, a mixed step (context + generation rows)", + spec=dict(layer="sparse", phase="mixed", tokens=512, reqs=2), + ), + ) + + note = """ + `reference` is exact (float64) softmax attention over the cache rows the step attends, read back from the paged + cache the call wrote (or a producer wrote before it) and dequantized as the FMHA dequantizes them: FP8 rows as + E4M3, NVFP4 rows as E2M1 x E4M3 block scale x the per-tensor ``kv_scale_quant_orig``. A sparse layer's top-k blocks + are re-selected from ``idx_q`` and the index-K cache with the backend's rule (per-block max over the causal + keys, the query's own block forced in, ties to the lower block). On an FP8 pool the FMHA takes Q as a plain E4M3 + cast, which the reference applies too. What remains between the two is the FMHA's own arithmetic (its P + rounding and accumulation order), which the test bounds. + """ + + def __call__( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + if k is not None: + backend.write_layer_caches( + k, v, None, metadata, kv_scale_orig_quant=kv_scale_orig_quant + ) + if sparse: + blocks = backend.run_indexer(idx_q, None, metadata, idx_k_prewritten=True) + args = AttentionForwardArgs( + output=output, sparse_backend_args=SparseBackendForwardArgs(topk_indices=blocks) + ) + else: + args = AttentionForwardArgs(output=output) + args.kv_scale_orig_quant = kv_scale_orig_quant + args.kv_scale_quant_orig = kv_scale_quant_orig + backend.forward(q, None, None, metadata, forward_args=args) + return output + + def reference( + self, + backend, + q: torch.Tensor, + k: Optional[torch.Tensor], + v: Optional[torch.Tensor], + idx_q: Optional[torch.Tensor], + metadata, + output: torch.Tensor, + sparse: bool, + kv_scale_orig_quant: Optional[torch.Tensor], + kv_scale_quant_orig: Optional[torch.Tensor], + ) -> torch.Tensor: + """``output`` as the call is supposed to write it, [T, heads * 128] in ``output``'s dtype. + + Every attended row -- the step's own included -- is read from the layer's paged cache, at the slots the + metadata's KV cache manager holds for each request, so this is evaluated after the call (whose write of + ``k`` / ``v`` it then reads back). ``k``, ``v`` and ``kv_scale_orig_quant`` are therefore not read; the + test checks the write itself. Query token ``j`` of request ``r`` sits at position ``cached_r + j`` and + attends positions up to it; on a sparse layer only those in its top-k blocks. Mirrors grouped KV heads (query + head h reads KV head h // (heads / kv heads); two per rank at tp2), each KV head selecting its own top-k + blocks from its own index heads against the rank's single index-K head. + """ + manager = metadata.kv_cache_manager + layer = backend.layer_idx + heads, dim = backend.num_heads, backend.head_dim + kv_heads = int(backend.num_kv_heads) + assert heads % kv_heads == 0 + group = heads // kv_heads + page = int(manager.tokens_per_block) + sm_scale = dim**-0.5 / float(backend.q_scaling) + num_seqs = int(metadata.num_seqs) + qlens = [int(n) for n in metadata.seq_lens[:num_seqs].tolist()] + cached = [int(c) for c in metadata.kv_cache_params.num_cached_tokens_per_seq[:num_seqs]] + total = sum(qlens) + assert q.shape[0] == total and output.shape[0] == total + + pool = manager.get_buffers(layer, "HND") + nvfp4 = manager.is_nvfp4_layer(layer) + if nvfp4: + scale_pool = manager.get_block_scale_buffers(layer, "HND") + k_global = float(kv_scale_quant_orig.reshape(-1)[1]) + v_global = float(kv_scale_quant_orig.reshape(-1)[2]) + q_rows = q.reshape(total, heads, dim) + if q_rows.dtype != torch.float8_e4m3fn and pool.dtype == torch.float8_e4m3fn: + # The FP8 FMHA shares one dtype across Q / K / V: Q is cast to E4M3, unscaled. + q_rows = q_rows.to(torch.float8_e4m3fn) + q64 = q_rows.to(torch.float64) + if sparse: + config = backend.m3_config + idx_cache = metadata.msa_idx_k_cache(layer) + idx64 = idx_q.reshape(total, config.num_index_heads, config.sparse_index_dim).to( + torch.float64 + ) + assert config.num_index_heads % kv_heads == 0 + idx_group = config.num_index_heads // kv_heads + + out = torch.zeros(total, heads, dim, dtype=torch.float64, device=q.device) + first = 0 + for row, request_id in enumerate(list(metadata.request_ids)[:num_seqs]): + n, length = qlens[row], cached[row] + qlens[row] + blocks = (length + page - 1) // page + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0][:blocks], + dtype=torch.int64, + device=q.device, + ) + qpos = cached[row] + torch.arange(n, device=q.device) + causal = torch.arange(length, device=q.device)[None, :] <= qpos[:, None] + for g in range(kv_heads): + if nvfp4: + keys = dequant_nvfp4_pages( + pool[:, 0].view(torch.uint8), + scale_pool[:, 0], + pages, + quad_scales=False, + kv_head=g, + ) + values = dequant_nvfp4_pages( + pool[:, 1].view(torch.uint8), + scale_pool[:, 1], + pages, + quad_scales=True, + kv_head=g, + ) + keys, values = keys * k_global, values * v_global + else: + keys = pool[pages, 0, g].to(torch.float64) + values = pool[pages, 1, g].to(torch.float64) + keys = keys.reshape(-1, dim)[:length] + values = values.reshape(-1, dim)[:length] + allowed = causal + if sparse: + # One index-K head per rank, shared by every index head (the cache is [pages, 1, page, dim]). + idx_keys = idx_cache[pages, 0].to(torch.float64).reshape(-1, dim)[:length] + chosen = _selected_blocks( + idx64[first : first + n, g * idx_group : (g + 1) * idx_group], + idx_keys, + causal, + qpos, + page, + config, + ) + allowed = causal & chosen.repeat_interleave(page, dim=1)[:, :length] + h0, h1 = g * group, (g + 1) * group + for lo in range(0, n, 256): + hi = min(n, lo + 256) + logits = ( + torch.einsum("thd,kd->thk", q64[first + lo : first + hi, h0:h1], keys) + * sm_scale + ) + logits = logits.masked_fill(~allowed[lo:hi, None, :], float("-inf")) + out[first + lo : first + hi, h0:h1] = torch.softmax(logits, dim=-1) @ values + first += n + return out.reshape(total, heads * dim).to(output.dtype) + + +def _selected_blocks( + idx_q: torch.Tensor, + idx_keys: torch.Tensor, + allowed: torch.Tensor, + qpos: torch.Tensor, + page: int, + config, +) -> torch.Tensor: + """The blocks each query token of one request attends, [n, blocks] bool. + + A block's score is the maximum, over the KV head's index heads and the block's causally visible keys, of + ``idx_q . idx_k``. Of the token's ``ceil((pos + 1) / page)`` valid blocks the first ``init_blocks`` and the last + ``local_blocks`` are forced in, and the ``topk`` highest-scoring are kept, ties to the lower block (the + backend's ``select_blocks_from_maxscore`` rule). The positive score scale the scorers apply does not change the + ranking, so none is applied. + """ + n, length = allowed.shape + blocks = (length + page - 1) // page + scores = torch.einsum("thd,kd->thk", idx_q, idx_keys).amax(dim=1) + scores = scores.masked_fill(~allowed, float("-inf")) + padded = torch.full((n, blocks * page), float("-inf"), dtype=scores.dtype, device=scores.device) + padded[:, :length] = scores + block_scores = padded.view(n, blocks, page).amax(dim=-1) + ids = torch.arange(blocks, device=scores.device)[None, :] + n_valid = torch.div(qpos + page, page, rounding_mode="floor")[:, None] + block_scores = block_scores.masked_fill(ids >= n_valid, float("-inf")) + forced = (ids < torch.clamp(n_valid, max=config.init_blocks)) | ( + (ids >= n_valid - config.local_blocks) & (ids < n_valid) + ) + block_scores = block_scores.masked_fill(forced, float("inf")) + order = torch.sort(block_scores, dim=1, descending=True, stable=True).indices[:, : config.topk] + keep = torch.arange(order.shape[1], device=scores.device)[None, :] < n_valid + chosen = torch.zeros(n, blocks, dtype=torch.bool, device=scores.device) + rows = torch.arange(n, device=scores.device)[:, None].expand_as(order) + chosen[rows[keep], order[keep]] = True + return chosen + + +m3_msa_attention = M3MsaAttention() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_producer.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_producer.py new file mode 100644 index 000000000000..ff34b8eac631 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_producer.py @@ -0,0 +1,223 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' NVFP4 QKV + indexer RMSNorm / RoPE / KV-insert producer, with its constant loads staged +before the PDL grid-dependency wait and an early dependents trigger (``kernels/minimaxM3/producerPw.cu``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +__all__ = ["m3_producer"] + +HEAD_DIM = 128 +ROTARY_DIM = 64 +PAGE = 128 +#: Query heads per rank at TP8 (MiniMax-M3: 64 query heads; 1 KV head and 1 index head per rank). +HEADS_Q = 8 +#: NVFP4 block: 16 values share one E4M3 scale. +FP4_BLOCK = 16 +E2M1_VALUES = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0) + + +def _norm_rope( + x: torch.Tensor, weight: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor, eps: float +): + """Gemma RMSNorm (``x * rsqrt(mean(x^2) + eps) * (1 + w)``) then NeoX RoPE on the first 64 dims, in fp32. + x [T, 128] fp32; cos / sin [T, 32].""" + y = x * torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) * (1.0 + weight.float()) + lo, hi, rest = y[:, :32], y[:, 32:64], y[:, 64:] + return torch.cat([lo * cos - hi * sin, hi * cos + lo * sin, rest], dim=-1) + + +def decode_nvfp4_token( + data: torch.Tensor, scales: torch.Tensor, quant_scale: float +) -> torch.Tensor: + """fp32 [128] of one token's packed E2M1 row (64 bytes, low nibble first) and its 8 E4M3 block scales.""" + table = torch.tensor(E2M1_VALUES, dtype=torch.float32, device=data.device) + nib = torch.stack([data & 0xF, data >> 4], dim=-1).reshape(-1).long() + vals = table[nib & 7] * torch.where(nib >= 8, -1.0, 1.0) + s = scales.view(torch.float8_e4m3fn).float() / quant_scale + return vals * s.repeat_interleave(FP4_BLOCK) + + +def scale_index(within_page: int, plane: int) -> torch.Tensor: + """Positions of a token's 8 block scales in its page's [128 * 8] scale plane: row-major for K, the 4-token + swizzle for V.""" + cols = torch.arange(8) + if plane == 0: + return within_page * 8 + cols + return (within_page // 4) * 32 + cols * 4 + within_page % 4 + + +class M3Producer(OpWrapper): + """The NVFP4 QKV + indexer producer of a sparse layer: ``minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert`` for one + KV head, one index head, head dim 128 and rotary dim 64, with the same arguments and outputs bit for bit. + + ``packed`` [T, (heads_q + 4) * 128] BF16 holds each token's q heads, k, v, index-q and index-k. Every head but v + gets the Gemma RMSNorm with its weight and NeoX RoPE on its first 64 dims (``rotary_cos_sin[position]``). Returns + (q FP8 [T, heads_q, 128], index-q FP8 [T, 1, 128]); writes the token's slot ``out_cache_loc[t]`` (skipped when + negative or past the pool): k and v as NVFP4 (16-value blocks, E4M3 scales against ``kv_quant_scale[1]`` / + ``[2]``, the V scales in the 4-token swizzle) and index-k as FP8. ``early`` triggers the dependent kernel on + entry (its outputs are read only after the dependent's own grid wait). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token (c1 without speculation)", spec=dict(tokens=1, scale=1.0)), + Cell(why="a c1 verify: 4 tokens", spec=dict(tokens=4, scale=3.0)), + Cell(why="a c4 verify: 16 tokens, small activations", spec=dict(tokens=16, scale=1e-3)), + Cell( + why="a c8 verify: 32 tokens, large activations, a skipped and an out-of-pool slot", + spec=dict(tokens=32, scale=40.0, bad_slots=True), + ), + Cell(why="the widest decode step: 48 tokens", spec=dict(tokens=48, scale=1.0)), + Cell( + why="a prefill chunk of 1000 tokens, early trigger off", + spec=dict(tokens=1000, scale=1.0, early=False), + ), + ) + + note = """ + The kernel is the native producer's arithmetic, compiled with the library's --use_fast_math, so it is held to that + kernel bit for bit on every output and cache byte. The torch reference states the math in fp32: the FP8 outputs are + held to it within one E4M3 step, and the NVFP4 K/V, decoded, within one E2M1 step of their block scale. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + prewait_step_inputs: bool = False, + ) -> tuple[torch.Tensor, torch.Tensor]: + if prewait_step_inputs: + # This target's copy (producerPw8): the same outputs bit for bit, with the position ids, slots and RoPE + # rows read before its grid wait (none of them may be written by a PDL predecessor). + return tuple( + torch.ops.trtllm.minimax_m3_sm107_tp8_producer_pw( + packed, + kv_data, + kv_scale, + index_k, + out_cache_loc, + kv_quant_scale, + eps, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + rotary_cos_sin, + position_ids, + pdl, + early, + True, + ) + ) + q, index_q = torch.ops.trtllm.minimax_m3_producer_pw( + packed, + kv_data, + kv_scale, + index_k, + out_cache_loc, + kv_quant_scale, + eps, + q_weight, + k_weight, + index_q_weight, + index_k_weight, + rotary_cos_sin, + position_ids, + heads_q, + pdl, + early, + ) + return q, index_q + + def reference( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + prewait_step_inputs: bool = False, + ) -> dict[str, torch.Tensor]: + """fp32 values before their quantization: "q" [T, heads_q, 128], "index_q" [T, 128], and per token "k", "v", + "index_k" [T, 128] (the values written to the token's slot). The caches, ``pdl`` and ``early`` do not change + them.""" + t = packed.shape[0] + x = packed.float().view(t, heads_q + 4, HEAD_DIM) + rope = rotary_cos_sin[position_ids[:t].long()] + cos, sin = rope[:, 0], rope[:, 1] + q = torch.stack( + [_norm_rope(x[:, h], q_weight, cos, sin, eps) for h in range(heads_q)], dim=1 + ) + return { + "q": q, + "k": _norm_rope(x[:, heads_q], k_weight, cos, sin, eps), + "v": x[:, heads_q + 1], + "index_q": _norm_rope(x[:, heads_q + 2], index_q_weight, cos, sin, eps), + "index_k": _norm_rope(x[:, heads_q + 3], index_k_weight, cos, sin, eps), + } + + def is_valid( + self, + packed: torch.Tensor, + kv_data: torch.Tensor, + kv_scale: torch.Tensor, + index_k: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: torch.Tensor, + eps: float, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + index_q_weight: torch.Tensor, + index_k_weight: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + heads_q: int, + pdl: bool = True, + early: bool = True, + prewait_step_inputs: bool = False, + ) -> None: + t = packed.shape[0] + # The RoPE row is read at the position with no bound. + pos = position_ids[:t] + assert bool(((pos >= 0) & (pos < rotary_cos_sin.shape[0])).all()), ( + "position_ids must index rotary_cos_sin" + ) + # Two tokens writing one slot race. + slots = out_cache_loc[:t] + live = slots[(slots >= 0) & (slots < kv_data.shape[0] * PAGE)] + assert live.unique().numel() == live.numel(), "out_cache_loc must not repeat a slot" + + +m3_producer = M3Producer() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_qk_norm_rope_fp8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..88180c2f1a1f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_qk_norm_rope_fp8.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense layers' per-head Gemma RMSNorm + partial NeoX RoPE of the packed [Q | K | V], into a fresh FP8 copy.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch._experimental.modeling_v2.catalog.attention.fused_qk_norm_rope import ( + fused_qk_norm_rope, +) + + +class M3QkNormRopeFp8(OpWrapper): + """``fused_qk_norm_rope`` written out of place in FP8 E4M3: Q and K heads normed and roped in fp32 and rounded + once to E4M3 (saturating at +/-448), V cast, ``qkv`` left untouched. Same arguments as the catalog's in-place + entry. + + The dense layers (0-2) feed the FP8 main KV cache with it: on every step of an FP8 cache, and on the decode steps + of a hybrid NVFP4 cache (whose dense layers sit in its FP8 pool). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one decode token, tp8 heads (8 q, 1 kv)", spec=dict(m=1, q=8, kv=1)), + Cell(why="a 48-row decode verify", spec=dict(m=48, q=8, kv=1)), + Cell(why="a prefill chunk on an FP8 cache", spec=dict(m=2048, q=8, kv=1)), + ) + + note = """ + `reference` is the in-place kernel followed by a cast to E4M3. V matches it bit for bit; Q and K round once from + fp32 where the composition rounds through bf16 first, so they are within one E4M3 step of it. + """ + + def __call__( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + return torch.ops.trtllm.fused_qk_norm_rope_to_fp8( + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + + def reference( + self, + qkv, + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) -> torch.Tensor: + """The catalog's in-place kernel on a copy, then the E4M3 cast (V exact; Q and K within one E4M3 step).""" + out = fused_qk_norm_rope( + qkv.clone(), + num_heads_q, + num_heads_k, + num_heads_v, + head_dim, + rotary_dim, + eps, + q_weight, + k_weight, + base, + is_neox, + position_ids, + factor, + low, + high, + attention_factor, + is_qk_norm, + use_gemma, + use_mrope, + mrope_section1, + mrope_section2, + ) + return out.to(torch.float8_e4m3fn) + + +m3_qk_norm_rope_fp8 = M3QkNormRopeFp8() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_qkv_insert.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_qkv_insert.py new file mode 100644 index 000000000000..c6f4be426ba3 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_qkv_insert.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sparse layers' generic horizontal producer: packed QKV/index rows -> FP8 Q / index-Q, K / V / index-K into +the paged caches.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3QkvInsert(OpWrapper): + """The packed ``[Q | K | V | index-Q | index-K]`` bf16 rows of a sparse layer: per-head Gemma RMSNorm and the + partial NeoX RoPE of Q, K, index-Q and index-K, K / V written to the main paged cache at ``out_cache_loc`` and + index-K to the FP8 index cache; returns (Q, index-Q) in FP8 E4M3 [M, heads, 128]. + + ``kv_scale`` given: the main cache is NVFP4 (packed E2M1 data [pages, 2, kv, 128, 64] uint8 + E4M3 scale pool + [pages, 2, kv, 128, 8], ``kv_quant_scale`` the fp32 (Q, K, V) quantization scales); None: FP8 E4M3 + [pages, 2, kv, 128, 128]. Token-major and step-type agnostic. ``m3_producer`` takes the NVFP4 cache at the tp8 + heads; this entry takes the FP8 cache and NVFP4 pools outside that layout. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="FP8 main cache, a prefill chunk", spec=dict(m=1024, cache="fp8")), + Cell(why="FP8 main cache, one decode token", spec=dict(m=1, cache="fp8")), + ) + + note = """ + The arithmetic is the CUDA producer's (fp32 norm and RoPE with bf16 roundings, then the E4M3 casts); `reference` + returns the fp32 values, and the test holds the FP8 outputs and cache rows within one E4M3 step of them. + """ + + def __call__( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + common = ( + num_heads, + num_kv_heads, + num_index_heads, + head_dim, + rotary_dim, + eps, + q_norm, + k_norm, + index_q_norm, + index_k_norm, + rotary_cos_sin, + position_ids, + ) + if kv_scale is not None: + return torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, kv_scale, index_k_cache, out_cache_loc, kv_quant_scale, *common + ) + return torch.ops.trtllm.minimax_m3_fp8_qkv_indexer_norm_rope_kv_insert( + packed, kv_cache, index_k_cache, out_cache_loc, *common + ) + + def reference( + self, + packed: torch.Tensor, + kv_cache: torch.Tensor, + kv_scale: Optional[torch.Tensor], + index_k_cache: torch.Tensor, + out_cache_loc: torch.Tensor, + kv_quant_scale: Optional[torch.Tensor], + num_heads: int, + num_kv_heads: int, + num_index_heads: int, + head_dim: int, + rotary_dim: int, + eps: float, + q_norm: torch.Tensor, + k_norm: torch.Tensor, + index_q_norm: torch.Tensor, + index_k_norm: torch.Tensor, + rotary_cos_sin: torch.Tensor, + position_ids: torch.Tensor, + ): + """The fp32 values the producer rounds (a new dict, nothing written): Gemma RMSNorm + NeoX partial RoPE of + Q [M, heads, D], index-Q [M, index heads, D], K [M, kv heads, D] and index-K [M, D]; V [M, kv heads, D] as + packed. The cache rows at ``out_cache_loc`` and the returned FP8 Q / index-Q are these values cast.""" + m = packed.shape[0] + sizes = [ + num_heads * head_dim, + num_kv_heads * head_dim, + num_kv_heads * head_dim, + num_index_heads * head_dim, + head_dim, + ] + q, k, v, iq, ik = packed.float().split(sizes, dim=-1) + cos, sin = rotary_cos_sin[position_ids.long()].unbind(1) + half = rotary_dim // 2 + + def norm_rope(x, w, heads): + x = x.reshape(m, heads, head_dim) + x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * (1.0 + w.float()) + x1, x2, rest = x[..., :half], x[..., half:rotary_dim], x[..., rotary_dim:] + c, s = cos.unsqueeze(1), sin.unsqueeze(1) + return torch.cat([x1 * c - x2 * s, x2 * c + x1 * s, rest], dim=-1) + + return { + "q": norm_rope(q, q_norm, num_heads), + "index_q": norm_rope(iq, index_q_norm, num_index_heads), + "k": norm_rope(k, k_norm, num_kv_heads), + "v": v.reshape(m, num_kv_heads, head_dim), + "index_k": norm_rope(ik, index_k_norm, 1)[:, 0], + } + + +m3_qkv_insert = M3QkvInsert() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_blocks.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_blocks.py new file mode 100644 index 000000000000..76841cca732d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_blocks.py @@ -0,0 +1,82 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MSA top-k block selection from per-KV-head block scores (the built-in selector).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3SelectBlocks(OpWrapper): + """Per query row the ``topk`` highest-scoring KV blocks of ``max_score_kv`` [kv_heads, n_blocks, total_q] (fp32), + with the first ``init_blocks`` and the last ``local_blocks`` valid blocks forced in and blocks at or beyond the + row's ``n_valid_blocks`` excluded; [total_q, kv_heads, topk] int32 ascending ids, -1 padded. ``head_major_output`` + backs the result head-major so ``permute(1, 0, 2)`` is contiguous. + + The decode dispatch runs it where the long-row selector does not apply: requests of 17..128 KV blocks. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="eight decode rows of 17..128 blocks", + spec=dict(total_q=8, n_blocks=128, mixed=True), + ), + Cell( + why="one decode row at the 128-block edge", + spec=dict(total_q=1, n_blocks=128, mixed=False), + ), + Cell(why="a 4-row verify at 40 blocks", spec=dict(total_q=4, n_blocks=40, mixed=False)), + ) + + note = """ + Exact: ties between equal scores break toward the lower block id; `reference` sorts with the same rule. + """ + + def __call__( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + nvb = n_valid_blocks.to(device=max_score_kv.device, dtype=torch.int32).contiguous() + return torch.ops.trtllm.minimax_m3_select_blocks( + max_score_kv, nvb, topk, init_blocks, local_blocks, head_major_output + ) + + def reference( + self, + max_score_kv: torch.Tensor, + topk: int, + n_valid_blocks: torch.Tensor, + init_blocks: int, + local_blocks: int, + head_major_output: bool, + ) -> torch.Tensor: + """Plain torch selection; ``head_major_output`` changes the backing only.""" + heads, n_blocks, total_q = max_score_kv.shape + out = torch.full((total_q, heads, topk), -1, dtype=torch.int32, device=max_score_kv.device) + blocks = torch.arange(n_blocks, device=max_score_kv.device) + for t in range(total_q): + nv = int(n_valid_blocks[t]) + for h in range(heads): + s = max_score_kv[h, :, t].clone().double() + s[blocks >= nv] = float("-inf") + forced = (blocks < min(init_blocks, nv)) | ( + (blocks >= nv - local_blocks) & (blocks < nv) + ) + s[forced] = float("inf") + order = sorted(range(nv), key=lambda b: (-float(s[b]), b))[: min(topk, nv)] + ids = torch.tensor(sorted(order), dtype=torch.int32, device=out.device) + out[t, h, : ids.numel()] = ids + return out + + +m3_select_blocks = M3SelectBlocks() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_long.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_long.py new file mode 100644 index 000000000000..9a92050c6abb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_long.py @@ -0,0 +1,193 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 MSA block selection for decode rows of up to 9216 blocks.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Blocks per decode row the op covers (9 keys per thread of one 1024-thread CTA). +MAX_BLOCKS = 9216 +#: Blocks the selection keeps per (query, KV head). +TOPK = 16 +#: Scores the forcing substitutes: initial blocks rank above local ones, and both above any finite score. +INIT_SCORE = 1.0e30 +LOCAL_SCORE = 1.0e29 + +# M3's checkpoint at TP8 (one KV head per rank: 4 KV heads over 8 ranks, each group on a rank pair): no forced initial +# blocks, the one block holding the query forced in. +_M3 = dict(num_kv_heads=1, init_blocks=0, local_blocks=1) + + +class M3SelectLong(OpWrapper): + """Fill ``out`` [total_q, num_kv_heads, 16] with each query's top-16 blocks. + + ``scores`` is the per-KV-head block maxima [num_kv_heads, num_blocks, + total_q] the CuTe DSL index scorer wrote (amax over the KV head's index + heads). Per (query, KV head) row the op keeps the ``n_valid_blocks[query]`` + leading blocks (clamped to ``[0, num_blocks]``), forces the last + ``local_blocks`` of them (counted from the *unclamped* valid count) to + ``LOCAL_SCORE`` and the first ``init_blocks`` to ``INIT_SCORE`` -- local wins + where both apply -- and selects the 16 largest, ties to the smaller block. + A pick whose forced score is -inf is emitted as -1. The ids are written + ascending with a -1 tail. + + Same contract as ``torch.ops.trtllm.minimax_m3_select_blocks`` (the in-tree + ``select_blocks_from_maxscore``) and bit-identical to it, except on a row + where more than 2048 valid blocks tie at the 16th-largest score: there the + in-tree op fills the tied slots in atomic order (its staging holds 2048 + candidates), while this op breaks the tie to the smallest blocks as the + contract says. Up to 2048 tied blocks both take the smallest. The test checks + both against ``reference``. Returns ``out``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 4 rows over 165521 tokens (1294 blocks, the AgentX ISL): the 2-key path", + spec=dict(num_blocks=1294, total_q=4, valid="near_full", scores="random", **_M3), + ), + Cell( + why="the TP2 engine's 5888-block rows at two requests x 4 rows: the 9-key path", + spec=dict(num_blocks=5888, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="the longest covered rows (9216 blocks, 1.18M tokens), 2 requests x 4 rows: the 9-key path at its end", + spec=dict(num_blocks=9216, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="~1500 blocks tied at the 16th score: the in-tree op also takes the smaller blocks (bit for bit)", + spec=dict(num_blocks=2000, total_q=8, valid="near_full", scores="ties1500", **_M3), + ), + Cell( + why="two requests x 4 rows of mixed lengths, 4500-block engine: every key width, short and padding rows", + spec=dict(num_blocks=4500, total_q=8, valid="mixed", scores="random", **_M3), + ), + Cell( + why="duplicated scores: ties broken to the smaller block, and > 2048 keys at the bound (re-read path)", + spec=dict(num_blocks=4500, total_q=4, valid="near_full", scores="ties", **_M3), + ), + Cell( + why="non-finite scores: +inf, NaN ranked above +inf, and rows whose picks run into -inf (emitted -1)", + spec=dict(num_blocks=600, total_q=8, valid="mixed", scores="nonfinite", **_M3), + ), + ) + + note = """ + Certified at one KV head per rank (TP8, Rubin) with M3's forcing + (`init_blocks=0`, `local_blocks=1`). The kernel takes any head count and + forcing; other forcing is not driven here. + + `scores` may be strided (the op reads it by stride); the target hands over + the scorer's contiguous view, which is what the cells drive. Rows of at + most 128 blocks are within the kernel's range, but the target keeps those on + the in-tree op, so they are not certified here. + + Ordering is by the descending-score key `orderedKey`: a positive NaN ranks + above +inf, +0 above -0, and a negative NaN below -inf -- which, unlike + -inf, is emitted as a block id rather than -1. The cells drive positive + NaN only. + """ + + def __call__( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_sm107_tp2_select_long( + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + ) + return out + + def reference( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + ) -> torch.Tensor: + """The [total_q, num_kv_heads, 16] int32 table the op is supposed to write. + + Exact integer ranking, no floating point beyond the forcing: each + score's bit pattern maps to an unsigned key that is smaller for a larger + score (NaN above +inf), packed with the block id so one sort ranks by + (score descending, block ascending). ``out``, ``use_pdl``, ``early_trigger`` and + ``trigger_phase`` only say where and how the op writes; they do not enter the result. + """ + num_kv_heads, num_blocks, total_q = scores.shape + device = scores.device + blocks = torch.arange(num_blocks, device=device, dtype=torch.int64) + raw = n_valid_blocks[:total_q].to(torch.int64) + valid = raw.clamp(0, num_blocks) + local_start = (raw - local_blocks).clamp_min(0) + result = torch.full((total_q, num_kv_heads, TOPK), -1, dtype=torch.int32, device=device) + for head in range(num_kv_heads): + forced = scores[head].float().t().clone() # [total_q, num_blocks] + forced = torch.where(blocks[None, :] < init_blocks, INIT_SCORE, forced) + forced = torch.where(blocks[None, :] >= local_start[:, None], LOCAL_SCORE, forced) + bits = forced.view(torch.int32).to(torch.int64) & 0xFFFFFFFF + key = torch.where(bits >= 0x80000000, bits, (~bits) & 0x7FFFFFFF) + in_row = blocks[None, :] < valid[:, None] + packed = torch.where( + in_row, (key << 16) | blocks[None, :], torch.iinfo(torch.int64).max + ) + k = min(TOPK, num_blocks) + picked = torch.topk(packed, k, dim=1, largest=False).values + picked_key = picked >> 16 + keep = (picked != torch.iinfo(torch.int64).max) & (picked_key != 0xFF800000) + ids = torch.where(keep, picked & 0xFFFF, torch.iinfo(torch.int64).max) + ids = ids.sort(dim=1).values + ids = torch.where(ids == torch.iinfo(torch.int64).max, -1, ids) + result[:, head, :k] = ids.to(torch.int32) + return result + + def is_valid( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + ) -> None: + """The inputs this op takes and answers wrongly. + + Negative forcing counts are accepted and push the forced ranges past the + row instead of failing; the in-tree op this one stands in for refuses + them, so a caller relying on this one would silently diverge from it. + An ``out`` overlapping ``scores`` is overwritten while it is still read. + """ + assert init_blocks >= 0 and local_blocks >= 0, ( + f"init_blocks ({init_blocks}) and local_blocks ({local_blocks}) must be non-negative; the " + "op accepts negative counts and selects as if the forced ranges were elsewhere" + ) + storage = scores.untyped_storage() + assert not (storage.data_ptr() <= out.data_ptr() < storage.data_ptr() + storage.nbytes()), ( + "out must not alias scores: the op writes it while scores are still read" + ) + + +m3_select_long = M3SelectLong() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_long_dsl.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_long_dsl.py new file mode 100644 index 000000000000..cd76a4e6d5d2 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_select_long_dsl.py @@ -0,0 +1,142 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Top-16 MSA block selection for decode rows of up to 9216 blocks, one row per 384-thread CTA (CuTe DSL).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_select_long import _M3, MAX_BLOCKS, M3SelectLong + + +class M3SelectLongDsl(OpWrapper): + """``m3_select_long``'s contract, cells and reference, run by ``kernels/m3_select_long_dsl``: the same selection + bit for bit as the 1024-thread ``minimax_m3_sm107_tp2_select_long`` (also on rows where more than 2048 blocks tie + at the 16th-largest score), from one 384-thread CTA per row. Returns ``out``.""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request verifying 4 rows over 165521 tokens (1294 blocks, the AgentX ISL): the 2-key path", + spec=dict(num_blocks=1294, total_q=4, valid="near_full", scores="random", **_M3), + ), + Cell( + why="the TP2 engine's 5888-block rows at two requests x 4 rows: the 9-key path", + spec=dict(num_blocks=5888, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="the longest covered rows (9216 blocks, 1.18M tokens), 2 requests x 4 rows: the 9-key path at its end", + spec=dict(num_blocks=9216, total_q=8, valid="near_full", scores="random", **_M3), + ), + Cell( + why="~1500 blocks tied at the 16th score: the in-tree op also takes the smaller blocks (bit for bit)", + spec=dict(num_blocks=2000, total_q=8, valid="near_full", scores="ties1500", **_M3), + ), + Cell( + why="two requests x 4 rows of mixed lengths, 4500-block engine: every key width, short and padding rows", + spec=dict(num_blocks=4500, total_q=8, valid="mixed", scores="random", **_M3), + ), + Cell( + why="duplicated scores: ties broken to the smaller block, and > 2048 keys at the bound (re-read path)", + spec=dict(num_blocks=4500, total_q=4, valid="near_full", scores="ties", **_M3), + ), + Cell( + why="non-finite scores: +inf, NaN ranked above +inf, and rows whose picks run into -inf (emitted -1)", + spec=dict(num_blocks=600, total_q=8, valid="mixed", scores="nonfinite", **_M3), + ), + ) + + def reference( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + ) -> torch.Tensor: + """``m3_select_long``'s reference: the selection is the same bit for bit.""" + return M3SelectLong.reference( + self, + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + ) + + def is_valid( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + ) -> None: + """``m3_select_long``'s domain.""" + M3SelectLong.is_valid( + self, + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + ) + + note = """ + Certified at one KV head per rank (TP8, Rubin) with M3's forcing + (`init_blocks=0`, `local_blocks=1`), and bit for bit against the + 1024-thread `m3_select_long` on every cell. The kernel takes any head count + and forcing; other forcing is not driven here. + + The kernel is compiled per (PDL launch, early trigger, trigger phase) with + symbolic shapes and strides; the target compiles its keys before warmup + (`kernels/m3_select_long_dsl.prebuild`), and a call outside them compiles + on first use. + """ + + def __call__( + self, + scores: torch.Tensor, + n_valid_blocks: torch.Tensor, + out: torch.Tensor, + init_blocks: int, + local_blocks: int, + use_pdl: bool = False, + early_trigger: bool = False, + trigger_phase: int = 0, + ) -> torch.Tensor: + from ...kernels.m3_select_long_dsl import select_long_dsl + + if scores.shape[1] > MAX_BLOCKS: + raise RuntimeError( + f"m3_select_long_dsl supports at most {MAX_BLOCKS} blocks, got {scores.shape[1]}" + ) + select_long_dsl( + scores, + n_valid_blocks, + out, + init_blocks, + local_blocks, + use_pdl, + early_trigger, + trigger_phase, + ) + return out + + +m3_select_long_dsl = M3SelectLongDsl() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_sparse_decode.py new file mode 100644 index 000000000000..6525a3341b15 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_sparse_decode.py @@ -0,0 +1,215 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""This target's block-sparse decode attention (Triton, with the one-launch long-context route).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.triton_sparse_decode import minimax_m3_sparse_attn_decode +from .m3_sparse_decode_lc8 import PAGE, sparse_attention_reference + + +class M3SparseDecode(OpWrapper): + """``minimax_m3_sparse_attn_decode`` as ``M3Attention._decode_sparse_core`` calls it. + + Attention of each query token over the blocks ``topk_idx`` selected for it + (``[num_kv_heads, T, topk]``, ascending ids with a -1 tail), each block + truncated at the token's causal extent, written into ``output`` in place. + The target passes FP8 q, the packed NVFP4 sparse-layer cache (one KV head) with its block and per-tensor + scales, and BF16 ``output`` [T, 8, 128]. Returns ``output``. + + Which kernel runs is decided inside, per step: a selected (non-identity) table on the TP8 geometry goes to the + one-launch ``m3_sparse_decode_lc8`` kernel while modeling's ``SPARSE_DECODE_LC8`` covers it; an identity table + (every request within 16 blocks, staged by the metadata), or a selected table with that range killed, runs the + Triton per-row producer (the multi-query producer for identity steps of at least 17 rows), then the LSE merge. + Every route also publishes the MXFP8 o_proj hand-off to ``m3_prequant.SLOT``, bit for bit + ``torch.ops.trtllm.mxfp8_quantize(output, True)``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="identity selection, decode without speculation at batch 8: the Triton producer + merge", + spec=dict( + seq_lens=(1, 128, 129, 700, 2048, 1500, 64, 0), decode_query_len=1, identity=True + ), + ), + Cell( + why="identity selection, Eagle3 verify at one request (T=4): the Triton producer + merge", + spec=dict(seq_lens=(1900,), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 2 (T=8), a padding request", + spec=dict(seq_lens=(2048, 0), decode_query_len=4, identity=True), + ), + Cell( + why="identity selection, Eagle3 verify at batch 8 (T=32): the multi-query producer + merge", + spec=dict( + seq_lens=(1900, 2048, 700, 1500, 1200, 2000, 300, 1024), + decode_query_len=4, + identity=True, + ), + ), + Cell( + why="selected blocks, Eagle3 verify at one request (T=4): the one-launch long-context kernel", + spec=dict(seq_lens=(40000,), decode_query_len=4, identity=False), + ), + Cell( + why="selected blocks, Eagle3 verify at batch 2 (T=8): the one-launch long-context kernel", + spec=dict(seq_lens=(9000, 40000), decode_query_len=4, identity=False), + ), + ) + + note = """ + Certified only as the target calls it: one KV head of 8 query heads, head_dim 128, FP8 q, packed NVFP4 + cache, BF16 output, `num_topk_chunks` left to the dispatcher. The function also takes BF16 / FP8 caches, other + head geometries and an explicit split-K factor (the in-tree tests drive those); none is certified here. + + The MXFP8 hand-off of the long-context route is certified on its own entry (`m3_sparse_decode_lc8`); the Triton + merge's (`_store_mxfp8_head`) is held to `torch.ops.trtllm.mxfp8_quantize` by this entry's test. + """ + + def __call__( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + minimax_m3_sparse_attn_decode( + q, + k_paged, + v_paged, + topk_idx, + block_table, + seq_lens, + sm_scale=sm_scale, + output=output, + decode_query_len=decode_query_len, + kv_scale=kv_scale, + k_block_scale=k_block_scale, + v_block_scale=v_block_scale, + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + num_topk_chunks=num_topk_chunks, + identity_blocks=identity_blocks, + ) + return output + + def reference( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> torch.Tensor: + """``output`` as the call is supposed to write it: float64 block-sparse attention over the + dequantized NVFP4 blocks (``sparse_attention_reference``), rounded once to ``output``'s dtype. + + Only the packed NVFP4 cache is mirrored -- the target's only configuration -- one KV head at a time: KV head + ``h`` serves query heads ``[h * group, (h + 1) * group)`` with its own top-k row. ``kv_scale``, + ``num_topk_chunks`` and ``identity_blocks`` do not change what is computed. + """ + assert k_block_scale is not None, "mirrors the NVFP4 cache only" + kv_heads = k_paged.shape[1] + assert kv_heads == 1, "mirrors the TP8 cache (one KV head per rank) only" + group = q.shape[1] // kv_heads + heads = [] + for h in range(kv_heads): + heads.append( + sparse_attention_reference( + q[:, h * group : (h + 1) * group], + k_paged.view(torch.uint8), + v_paged.view(torch.uint8), + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + k_global_scale, + v_global_scale, + topk_idx[h], + block_table, + seq_lens, + decode_query_len, + sm_scale, + ) + ) + return torch.cat(heads, dim=1).to(output.dtype) + + def is_valid( + self, + q: torch.Tensor, + k_paged: torch.Tensor, + v_paged: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + sm_scale: float, + output: torch.Tensor, + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, + identity_blocks: bool = False, + ) -> None: + """The inputs this call takes and answers wrongly. + + Every producer walks a token's first ``min(topk, ceil(kv_len / 128))`` + table entries and indexes the block table with them unchecked, so they + must be real blocks of the request. ``identity_blocks`` is a promise the + multi-query producer relies on: every token of a request reads the same + blocks, all of them. Reads ``topk_idx`` and ``seq_lens`` back to the + host: a test-only cost. + """ + total_q = q.shape[0] + seq = seq_lens.tolist() + max_topk = topk_idx.shape[-1] + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk_idx[:, tok, : min(max_topk, nblocks)] + assert bool(((ids >= 0) & (ids < min(nblocks, block_table.shape[1]))).all()), ( + f"token {tok}: the first min(topk, block count) selected ids must be blocks of the request; " + "the block table is indexed with them unchecked" + ) + if identity_blocks: + assert nblocks <= max_topk and bool( + (ids == torch.arange(nblocks, device=ids.device, dtype=ids.dtype)).all() + ), ( + f"token {tok}: identity_blocks promises the table holds every block of the request in order" + ) + + +m3_sparse_decode = M3SparseDecode() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_sparse_decode_lc8.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_sparse_decode_lc8.py new file mode 100644 index 000000000000..a845ac68f257 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/m3_sparse_decode_lc8.py @@ -0,0 +1,487 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-launch long-context sparse decode over the selected NVFP4 blocks at 8 query heads (16-CTA clusters). + +Also the plain-torch block-sparse attention and MXFP8 hand-off mirrors the +Triton sparse decode entry (``m3_sparse_decode``) reuses: both compute the same +thing over the same cache. +""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The TP8 geometry the kernel is compiled for: 8 query heads on one KV head. +NUM_HEADS = 8 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 +#: NVFP4 elements per E4M3 block scale. +SF_VEC = 16 +#: MXFP8 elements per UE8M0 scale, and the o_proj width the hand-off is laid out for. +MX_BLOCK = 32 +WIDTH = NUM_HEADS * HEAD_DIM + +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) + + +def dequant_nvfp4_pages( + packed: torch.Tensor, + block_scale: torch.Tensor, + pages: torch.Tensor, + quad_scales: bool, + kv_head: int = 0, +) -> torch.Tensor: + """Dequantize KV head ``kv_head`` of the given pages of a packed NVFP4 cache to float64 [n, PAGE, HEAD_DIM]. + + ``packed`` is the [pages, kv_heads, PAGE, HEAD_DIM // 2] byte view (low + nibble = even element), ``block_scale`` the [pages, kv_heads, PAGE, + HEAD_DIM // 16] E4M3 bytes ``fused_write_layer_caches_nvfp4`` writes: token + ``t``'s column ``c`` at byte ``t * 8 + c`` of the page's region for K, and at + ``(t // 4) * 32 + c * 4 + t % 4`` for V (``quad_scales``). The per-tensor + global scale is not applied. Every product is exact in float64. + """ + table = torch.tensor(_E2M1, dtype=torch.float64, device=packed.device) + rows = packed[pages, kv_head].to(torch.int64) # [n, PAGE, HEAD_DIM // 2] + nibbles = torch.stack([rows & 0x0F, rows >> 4], dim=-1).reshape(rows.shape[0], PAGE, HEAD_DIM) + cols = HEAD_DIM // SF_VEC + region = block_scale[pages, kv_head].reshape(rows.shape[0], PAGE * cols) + t = torch.arange(PAGE, device=packed.device)[:, None] + c = torch.arange(cols, device=packed.device)[None, :] + index = (t // 4) * 32 + c * 4 + t % 4 if quad_scales else t * cols + c + scales = region[:, index.reshape(-1)].reshape(rows.shape[0], PAGE, cols) + scales = scales.view(torch.float8_e4m3fn).to(torch.float64) + return table[nibbles] * scales.repeat_interleave(SF_VEC, dim=-1) + + +def sparse_attention_reference( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + decode_query_len: int, + sm_scale: float, +) -> torch.Tensor: + """Block-sparse decode attention over one KV head of a packed NVFP4 cache, float64 [T, heads, HEAD_DIM]. + + Query token ``tok`` belongs to request ``tok // decode_query_len`` at offset + ``tok % decode_query_len`` and attends the first ``min(16, ceil(kv_len / + PAGE))`` entries of ``topk[tok]`` that are not -1, each block truncated at + ``kv_len = max(seq_len - decode_query_len + offset + 1, 0)``. A token that + attends nothing (a CUDA-graph padding row) is all zeros. ``q`` is E4M3 (or + its byte view); softmax and both products are float64. + """ + q64 = ( + q.view(torch.float8_e4m3fn).to(torch.float64) + if q.element_size() == 1 + else q.to(torch.float64) + ) + total_q, num_heads, _ = q64.shape + kgs = float(k_global_scale.reshape(-1)[0]) + vgs = float(v_global_scale.reshape(-1)[0]) + out = torch.zeros(total_q, num_heads, HEAD_DIM, dtype=torch.float64, device=q.device) + seq = seq_lens.tolist() + positions = torch.arange(PAGE, device=q.device) + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + real_topk = min(TOPK, (kv_len + PAGE - 1) // PAGE) + blocks = topk[tok, :real_topk] + blocks = blocks[blocks >= 0].to(torch.int64) + if blocks.numel() == 0: + continue + pages = block_table[req, blocks].to(torch.int64) + keys = ( + dequant_nvfp4_pages(k, k_block_scale, pages, quad_scales=False).reshape(-1, HEAD_DIM) + * kgs + ) + values = ( + dequant_nvfp4_pages(v, v_block_scale, pages, quad_scales=True).reshape(-1, HEAD_DIM) + * vgs + ) + valid = ((blocks[:, None] * PAGE + positions[None, :]) < kv_len).reshape(-1) + logits = (q64[tok] @ keys.T) * sm_scale + logits = logits.masked_fill(~valid[None, :], float("-inf")) + out[tok] = torch.softmax(logits, dim=-1) @ values + return out + + +def mxfp8_handoff_reference( + out: torch.Tensor, num_rows_alloc: int +) -> Tuple[torch.Tensor, torch.Tensor]: + """The MXFP8 o_proj hand-off the merge writes for BF16 rows ``out`` [T, WIDTH]: ``(q8 [T, WIDTH] uint8, + sf [num_rows_alloc * WIDTH // 32] uint8)``. + + Per 32-element block: ``e`` is the biased exponent of ``amax * rcp(448)`` + rounded up when its mantissa is non-zero (capped at 254), the elements are + scaled by ``2^(127 - e)`` and rounded to E4M3 (saturating), and ``e`` lands + in the 128x4-swizzled scale layout the MXFP8 GEMM reads. Entries of ``sf`` + for rows past ``T`` are zero. Computed from the op's own BF16 rows, so a + comparison against it is exact. + """ + total_q = out.shape[0] + x = out.float().reshape(total_q, WIDTH // MX_BLOCK, MX_BLOCK) + amax = x.abs().amax(dim=-1) + sfv = amax * torch.tensor(1.0 / 448.0, dtype=torch.float32, device=out.device) + bits = sfv.view(torch.int32) + e = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + e = e.clamp(max=254) + qs = torch.where(amax != 0, ((254 - e) << 23).view(torch.float32), torch.zeros_like(amax)) + xq = (x * qs[..., None]).clamp(-448.0, 448.0) + q8 = xq.to(torch.float8_e4m3fn).view(torch.uint8).reshape(total_q, WIDTH) + num_sf_k_tiles = WIDTH // 128 + sf = torch.zeros(num_rows_alloc * (WIDTH // MX_BLOCK), dtype=torch.uint8, device=out.device) + tok = torch.arange(total_q, device=out.device)[:, None] + k_sf = torch.arange(WIDTH // MX_BLOCK, device=out.device)[None, :] + offset = ( + (tok // 128) * num_sf_k_tiles * 512 + + (k_sf // 4) * 512 + + (tok % 32) * 16 + + ((tok % 128) // 32) * 4 + + k_sf % 4 + ) + sf[offset.reshape(-1)] = e.to(torch.uint8).reshape(-1) + return q8, sf + + +class M3SparseDecodeLc8(OpWrapper): + """Sparse decode attention, split-K merge and MXFP8 hand-off of one TP8 decode step in one launch. + + One cluster of 16 CTAs per query token: CTA ``r`` attends the token's + ``r``-th selected block (``topk[tok, r]``) for all 8 heads (transposed MMAs, + the heads on the n dimension), and the cluster merges the 16 partials over + distributed shared memory into the BF16 row ``out[tok]`` [1024]. With ``q8`` / + ``sf`` it also writes the row's MXFP8 copy and swizzled UE8M0 scales for the + o_proj GEMM. All operands are byte views (E4M3 ``q``, packed E2M1 pages, E4M3 + block scales). The kernel releases its PDL dependents right after its grid + wait. Returns ``out``. + + ``prewait`` stages the request length, the query rows (widened to FP16) and + the request's block-table row (up to 8192 entries, longer rows are read from + global memory) before the grid wait; the arithmetic is the same, so both + variants write the same bits. + + Per block: NVFP4 K/V dequantized to FP16 (exact), FP8 q widened to FP16 + (exact), QK and PV on FP16 tensor cores with FP32 accumulation, the softmax + probabilities rounded to FP16 for PV, log2-domain softmax with ``ex2.approx``. + ``linear_softmax`` picks the denominator recurrence (the two agree to + rounding); the target picks it per batch shape. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode without speculation, one request: T=1, log-domain recurrence, MXFP8 hand-off, prewait", + spec=dict( + seq_lens=(9000,), decode_query_len=1, linear_softmax=False, mxfp8=True, prewait=True + ), + ), + Cell( + why="Eagle3 verify at one request (the TP8 operating point): T=4, log-domain recurrence, prewait", + spec=dict( + seq_lens=(20000,), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=True, + ), + ), + Cell( + why="Eagle3 verify at two requests: T=8 on the linear denominator, prewait at its row limit", + spec=dict( + seq_lens=(5000, 12345), + decode_query_len=4, + linear_softmax=True, + mxfp8=True, + prewait=True, + ), + ), + Cell( + why="a 1M-token request: a block-table row longer than the staged 8192 entries, read from global", + spec=dict( + seq_lens=(1048576,), + decode_query_len=4, + linear_softmax=False, + mxfp8=True, + prewait=True, + ), + ), + Cell( + why="Eagle3 verify at batch 8 (T=32): short requests with -1 tails and zero-length padding rows", + spec=dict( + seq_lens=(16384, 2049, 130, 0, 7777, 4, 30000, 0), + decode_query_len=4, + linear_softmax=True, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="decode without speculation at batch 8: T=8 rows of distinct requests, staged after the wait", + spec=dict( + seq_lens=(3000, 2100, 9999, 2048, 2049, 4500, 16000, 2200), + decode_query_len=1, + linear_softmax=False, + mxfp8=True, + prewait=False, + ), + ), + Cell( + why="no hand-off buffers (the variant the call site takes past 128 rows), driven at T=8", + spec=dict( + seq_lens=(5000, 12345), + decode_query_len=4, + linear_softmax=True, + mxfp8=False, + prewait=False, + ), + ), + ) + + note = """ + Certified: `linear_softmax` x hand-off on / off x `prewait`, the variants + the cells drive. The hand-off cells are certified bit-exactly against + `mxfp8_handoff_reference` applied to the op's own BF16 rows; the BF16 rows + themselves within the band the test derives, and the two `prewait` variants + against each other bit for bit. + + `topk` is read for the first `min(16, ceil(kv_len / 128))` entries of each + token; a -1 among them skips that CTA's block, entries past them are never + read. `use_pdl` only changes the launch attribute. The hand-off assumes + `num_sf_k_tiles = 8` (a 1024-wide o_proj input) and buffers with at least + as many rows as the step; `MERGE_MXFP8` sizes them at 128 rows. + """ + + def __call__( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + prewait: bool, + merge_once: bool = False, + prefetch: Optional[tuple] = None, + next_kv: Optional[tuple] = None, + identity: bool = False, + identity_step: bool = False, + one_div: bool = False, + ) -> torch.Tensor: + # merge_once: this target's copy of the op (minimaxM3Sm107Tp8/sparseDecodeLc8m.cu), whose split-K merge forms + # the 16 weights once per CTA (same values, same order: bitwise the image op's output). + lc8 = ( + torch.ops.trtllm.minimax_m3_sm107_tp8_sparse_decode_lc8 + if merge_once + else torch.ops.trtllm.minimax_m3_tp8_sparse_decode_lc8 + ) + # prefetch: up to two read-only tensors this target's op pulls into L2 (a hint; the image op takes none). + # next_kv: (this layer's id buffer, the next sparse layer's, its K / V pools and block scales), target op only. + extra = () + if merge_once and (prefetch or next_kv): + extra = tuple(prefetch or ()) + (None,) * (2 - len(prefetch or ())) + if next_kv: + extra += tuple(next_kv) + lc8( + q, + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + out, + q8, + sf, + num_sf_k_tiles, + decode_query_len, + sm_scale, + linear_softmax, + use_pdl, + prewait, + *extra, + **({"identity": True} if identity else {}), + **({"identity_step": True} if identity_step else {}), + **({"one_div": True} if one_div else {}), + ) + return out + + def reference( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + prewait: bool, + merge_once: bool = False, + prefetch: Optional[tuple] = None, + next_kv: Optional[tuple] = None, + identity: bool = False, + identity_step: bool = False, + one_div: bool = False, + ) -> torch.Tensor: + """The BF16 ``out`` [T, 1024] the op is supposed to write: ``sparse_attention_reference`` in float64, + rounded once. ``linear_softmax``, ``use_pdl``, ``prewait`` and ``one_div`` do not change what is computed; + the hand-off buffers are checked separately with ``mxfp8_handoff_reference``. ``identity`` attends each token's + first min(16, blocks) blocks (its own causal length), whatever ``topk`` holds. + """ + if identity: + topk = _identity_table(seq_lens, decode_query_len, int(q.shape[0]), topk.device) + attn = sparse_attention_reference( + q.view(-1, NUM_HEADS, HEAD_DIM), + k, + v, + k_block_scale, + v_block_scale, + k_global_scale, + v_global_scale, + topk, + block_table, + seq_lens, + decode_query_len, + sm_scale, + ) + return attn.reshape(out.shape).to(out.dtype) + + def is_valid( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + k_block_scale: torch.Tensor, + v_block_scale: torch.Tensor, + k_global_scale: torch.Tensor, + v_global_scale: torch.Tensor, + topk: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + out: torch.Tensor, + q8: Optional[torch.Tensor], + sf: Optional[torch.Tensor], + num_sf_k_tiles: int, + decode_query_len: int, + sm_scale: float, + linear_softmax: bool, + use_pdl: bool, + prewait: bool, + merge_once: bool = False, + prefetch: Optional[tuple] = None, + next_kv: Optional[tuple] = None, + identity: bool = False, + identity_step: bool = False, + one_div: bool = False, + ) -> None: + """The inputs this op takes and answers wrongly (``m3_attn_lc8.lc8_applies`` is the target's form of the + layout half). + + The kernel addresses a page's token rows at a fixed 64-byte (data) and + 8-byte (scale) pitch and loads them 16 / 8 bytes at a time, so only the + page stride is read from the views. It reads the selected block ids, + the block table and the hand-off buffers unchecked. Reads ``topk`` and + ``seq_lens`` back to the host: a test-only cost. + """ + for name, t in (("k", k), ("v", v)): + assert t.dim() == 4 and t.shape[1] == 1 and t.shape[2:] == (PAGE, HEAD_DIM // 2), ( + f"{name} must be the [pages, 1, 128, 64] byte view of one KV head" + ) + assert t.stride(3) == 1 and t.stride(2) == HEAD_DIM // 2, ( + f"{name} rows must be dense 64-byte rows: the kernel ignores the row stride" + ) + assert t.data_ptr() % 16 == 0 and t.stride(0) % 16 == 0, ( + f"{name} pages must be 16-byte aligned" + ) + for name, t in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + assert t.dim() == 4 and t.stride(3) == 1 and t.stride(2) == HEAD_DIM // SF_VEC, ( + f"{name} rows must be dense 8-byte rows: the kernel ignores the row stride" + ) + assert t.data_ptr() % 8 == 0 and t.stride(0) % 8 == 0, ( + f"{name} pages must be 8-byte aligned" + ) + assert q.data_ptr() % 16 == 0 and q.stride(0) % 16 == 0 and q.stride(1) % 16 == 0, ( + "q rows are loaded 16 bytes at a time and must be 16-byte aligned" + ) + assert k_global_scale.numel() == 1 and v_global_scale.numel() == 1, ( + "global scales are single values" + ) + total_q = q.shape[0] + assert ( + total_q % decode_query_len == 0 and seq_lens.numel() >= total_q // decode_query_len + ), ( + "every query token must belong to a request: seq_lens is read at tok // decode_query_len unchecked" + ) + seq = seq_lens.tolist() + num_cols = block_table.shape[1] + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + kv_len = max(seq[req] - decode_query_len + offset + 1, 0) + nblocks = (kv_len + PAGE - 1) // PAGE + ids = topk[tok, : min(TOPK, nblocks)] + assert bool(((ids >= -1) & (ids < min(nblocks, num_cols))).all()), ( + f"token {tok}: selected block ids must be -1 or below the request's block count; the block " + "table is indexed with them unchecked" + ) + if q8 is not None: + assert num_sf_k_tiles == WIDTH // 128, ( + f"num_sf_k_tiles must be {WIDTH // 128}: the hand-off rows are {WIDTH} wide and a smaller " + "value overlaps them silently" + ) + assert ( + q8.numel() >= total_q * WIDTH + and sf is not None + and sf.numel() >= ((total_q + 127) // 128 * 128 * (WIDTH // MX_BLOCK)) + ), "the hand-off buffers must hold every row of the step: they are written unchecked" + + +def _identity_table(seq_lens, decode_query_len, total_q, device): + """[T, 16] ids (i < min(16, blocks of the token's causal length) ? i : -1), the selection identity mode reads.""" + ids = torch.full((total_q, 16), -1, dtype=torch.int32, device=device) + for tok in range(total_q): + req, qoff = divmod(tok, decode_query_len) + kv_len = max(int(seq_lens[req]) - decode_query_len + qoff + 1, 0) + n = min(16, (kv_len + PAGE - 1) // PAGE) + ids[tok, :n] = torch.arange(n, dtype=torch.int32, device=device) + return ids + + +m3_sparse_decode_lc8 = M3SparseDecodeLc8() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/msa_kv_update.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/msa_kv_update.py new file mode 100644 index 000000000000..43166b472eca --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/attention/msa_kv_update.py @@ -0,0 +1,200 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MSA KV-length correction in one launch (this target's ``MsaKvUpdate`` hook).""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: The MSA pool's page: one sparse block is one 128-token KV page. +TOKENS_PER_BLOCK = 128 + + +class MsaKvUpdate(OpWrapper): + """Re-derive the MSA decode state after the overlap scheduler moved ``kv_lens``. + + ``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` calls this with its + own buffers at the head of every captured speculative decode step. One + single-CTA launch writes, for ``batch`` requests and ``total_q`` query + tokens (all int32): + + * ``msa_seq_lens[b] = min(kv_lens[b], kv_lens_staged[b])`` and, when given, + ``cu_kv_lens`` as its exclusive prefix sum (``batch + 1`` entries); + * ``out_cache_loc[t]``, the KV write slot of the position token ``t`` + attends up to, through ``block_table``; + * ``n_valid_blocks[t]``, when given, the block count top-k selection is + bounded by. + + It returns None; the cells read the four buffers. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="decode without speculation at one request: qo = 1, every buffer written", + spec=dict(batch=1, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="decode without speculation at the captured batch 8", + spec=dict(batch=8, query_len=1, accept="full", cu_kv=True, n_valid=True), + ), + Cell( + why="Eagle3 verify (draft 3): qo = 4 per request, partial acceptance shortens kv_lens below staged", + spec=dict(batch=8, query_len=4, accept="partial", cu_kv=True, n_valid=True), + ), + Cell( + why="the metadata without cu_kv_lens (no extend_ctx promotion): the optional prefix sum is skipped", + spec=dict(batch=4, query_len=4, accept="partial", cu_kv=False, n_valid=True), + ), + Cell( + why="CUDA-graph padding rows (kv_lens 0, pos -1) and no n_valid_blocks buffer", + spec=dict(batch=8, query_len=4, accept="padded", cu_kv=True, n_valid=False), + ), + Cell( + why="positions crossing a page boundary: the slot moves to the previous page of the request", + spec=dict(batch=3, query_len=4, accept="page_edge", cu_kv=True, n_valid=True), + ), + ) + + note = """ + The kernel covers any batch up to the block size's loop (it strides over + 256 threads), and any page size; only `tokens_per_block = 128`, the MSA + pool's page, is certified. + + The op does not bounds-check the request index a token reads + (`q_batch_row`) or the block-table column it derives: both come from the + metadata's own staging, which keeps them in range by construction, and an + out-of-range value reads whatever memory sits there. `is_valid` states both. + """ + + def __call__( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + torch.ops.trtllm.minimax_m3_msa_kv_update( + kv_lens, + kv_lens_staged, + seq_lens, + q_batch_row, + q_intra, + block_table, + tokens_per_block, + batch, + total_q, + msa_seq_lens, + cu_kv_lens, + out_cache_loc, + n_valid_blocks, + ) + + def reference( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> Tuple[torch.Tensor, Optional[torch.Tensor], torch.Tensor, Optional[torch.Tensor]]: + """``(msa_seq_lens[:batch], cu_kv_lens[:batch + 1], out_cache_loc[:total_q], + n_valid_blocks[:total_q])`` as the op is supposed to write them. + + The in-tree metadata's own torch re-derivation + (``MiniMaxM3MsaSparseAttentionMetadata.on_update_kv_lens`` past its hook + call), statement for statement, returning new tensors rather than + writing the buffers. The buffer arguments only say which outputs exist: + a ``None`` buffer gives a ``None`` result. + """ + page = int(tokens_per_block) + kv_true = torch.minimum(kv_lens[:batch], kv_lens_staged[:batch]) + cu_kv = None + if cu_kv_lens is not None: + cu_kv = torch.zeros(batch + 1, dtype=torch.int32, device=kv_lens.device) + cu_kv[1:] = torch.cumsum(kv_true, 0) + qbr = q_batch_row[:total_q].to(torch.long) + qo_dev = seq_lens[:batch] + pos = kv_true[qbr] - qo_dev[qbr] + q_intra[:total_q] + pos_slot = pos.clamp_min(0) + block_col = torch.div(pos_slot, page, rounding_mode="floor").clamp( + max=int(block_table.shape[1]) - 1 + ) + slots = block_table[qbr, block_col.to(torch.long)] * page + torch.remainder(pos_slot, page) + n_valid = None + if n_valid_blocks is not None: + n_valid = torch.div( + (pos + 1).clamp_min(0) + (page - 1), page, rounding_mode="floor" + ).to(torch.int32) + return kv_true.to(torch.int32), cu_kv, slots.to(torch.int32), n_valid + + def is_valid( + self, + kv_lens: torch.Tensor, + kv_lens_staged: torch.Tensor, + seq_lens: torch.Tensor, + q_batch_row: torch.Tensor, + q_intra: torch.Tensor, + block_table: torch.Tensor, + tokens_per_block: int, + batch: int, + total_q: int, + msa_seq_lens: torch.Tensor, + cu_kv_lens: Optional[torch.Tensor], + out_cache_loc: torch.Tensor, + n_valid_blocks: Optional[torch.Tensor], + ) -> None: + """The inputs this op takes and answers wrongly. + + Every operand but the block table is read as a dense run from its first + element, so a strided view reads the wrong elements. The request index of + each token and the table row it selects are not bounds-checked. Reads + ``q_batch_row`` back to the host: a test-only cost. + """ + for name, t in ( + ("kv_lens", kv_lens), + ("kv_lens_staged", kv_lens_staged), + ("seq_lens", seq_lens), + ("q_batch_row", q_batch_row), + ("q_intra", q_intra), + ("msa_seq_lens", msa_seq_lens), + ("out_cache_loc", out_cache_loc), + ("cu_kv_lens", cu_kv_lens), + ("n_valid_blocks", n_valid_blocks), + ): + assert t is None or t.is_contiguous(), ( + f"{name} must be contiguous: the kernel indexes it densely from data_ptr and ignores strides" + ) + rows = q_batch_row[:total_q] + assert total_q == 0 or ( + int(rows.min()) >= 0 and int(rows.max()) < min(batch, block_table.shape[0]) + ), ( + "q_batch_row must index a request below batch (and a block-table row): the op reads the " + "request's lengths and table row unchecked" + ) + + +msa_kv_update = MsaKvUpdate() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/__init__.py new file mode 100644 index 000000000000..5da86fe24bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collective-communication entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_allreduce.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_allreduce.py new file mode 100644 index 000000000000..8daa2d6b9540 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_allreduce.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The built-in TP all-reduce, fused with the residual add and the next RMSNorm.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.distributed import AllReduce, AllReduceParams + + +class M3AllReduce(OpWrapper): + """``comm(x, all_reduce_params=params)``: the runtime's ``AllReduce`` over the TP group. ``params`` None: the + plain sum over the ranks (the drafter's o_proj and down projection). With ``RESIDUAL_RMS_NORM`` params: + ``updated = residual + sum_ranks(x)``, ``normed = rmsnorm(updated) * norm_weight``; + returns (normed, updated). The strategy (``AllReduceStrategy.AUTO``: one-shot / two-shot Lamport, NCCL) is the + communicator's per call. + + The layer boundaries the multicast all-reduce does not take run it: prefill and mixed steps, and generation steps + beyond the multicast kernel's rows. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # No cell names a world size: the result does not depend on it, and the certifying matrix runs at every world + # size one node can form (the target's two-tray group of 8 cannot be formed on one node). + CELLS: tuple[Cell, ...] = ( + Cell(why="a prefill chunk", spec=dict(m=2048, hidden=6144)), + Cell(why="a 64-row generation step", spec=dict(m=64, hidden=6144)), + Cell(why="plain sum of eight drafter rows", spec=dict(m=8, hidden=6144, plain=True)), + ) + + note = """ + The reduction order across ranks is the strategy's; `reference` sums the ranks' rows in fp32 in rank order, then + adds the residual and normalizes (one bf16 rounding of the sum and one of the norm). A strategy may round the sum + to bf16 before the residual add, so the residual sum agrees within one bf16 ulp of its operands' magnitude. + """ + + def __call__(self, x: torch.Tensor, comm: AllReduce, params: Optional[AllReduceParams]): + return comm(x, all_reduce_params=params) + + def reference(self, xs, comm: AllReduce, params: Optional[AllReduceParams]): + """``xs``: every rank's ``x`` in rank order (a collective's reference takes the group's inputs).""" + total = torch.stack([x.float() for x in xs]).sum(0) + if params is None: + return total.to(xs[0].dtype) + updated = (params.residual.float() + total).to(xs[0].dtype) + h = updated.float() + normed = ( + h + * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + params.eps) + * params.norm_weight.float() + ) + return normed.to(xs[0].dtype), updated + + +m3_allreduce = M3AllReduce() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_mc_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_mc_allreduce_norm.py new file mode 100644 index 000000000000..41bf2298a885 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_mc_allreduce_norm.py @@ -0,0 +1,317 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode layer boundary, 2 ranks on SM107.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +_E4M3_MAX = 448.0 + + +def mxfp8_rows(normed: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """MXFP8 of BF16 rows as ``trtllm::mxfp8_quantize(normed, True)``: E4M3 [M, H] and 128x4-swizzled UE8M0 bytes. + + One scale per 32 columns, the smallest power of two at or above ``amax / 448`` (biased exponent, round up), + data = row / scale rounded to nearest E4M3. The scale buffer is padded to 128 rows, padding bytes left as the + op leaves them (the comparison masks them out). + """ + m, h = normed.shape + blocks = normed.float().view(m, h // 32, 32) + amax = blocks.abs().amax(-1) + bits = (amax / _E4M3_MAX).view(torch.int32) + byte = (((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) > 0).to(torch.int32)).clamp(0, 254) + scale = torch.exp2(byte.float() - 127.0).unsqueeze(-1) + data = torch.where(amax.unsqueeze(-1) != 0, blocks / scale, torch.zeros_like(blocks)) + data = data.view(m, h).to(torch.float8_e4m3fn) + return data, swizzle_sf(byte.to(torch.uint8)) + + +def swizzle_sf(byte: torch.Tensor) -> torch.Tensor: + """Scatter [rows, H / 32] scale bytes into the 128x4-swizzled buffer the op writes (zero padding).""" + rows, cols = byte.shape + padded_rows = (rows + 127) // 128 * 128 + r = torch.arange(rows, device=byte.device).view(-1, 1) + c = torch.arange(cols, device=byte.device).view(1, -1) + index = ( + (c % 4) + (c // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + (r // 128) * (128 * cols) + ) + out = torch.zeros(padded_rows * cols, dtype=torch.uint8, device=byte.device) + out[index.reshape(-1)] = byte.reshape(-1) + return out + + +def sf_written(rows: int, cols: int, device) -> torch.Tensor: + """Mask of the swizzled scale bytes the op writes for ``rows`` rows (the padding rows are left unwritten).""" + return swizzle_sf(torch.ones(rows, cols, dtype=torch.uint8, device=device)).bool() + + +def boundary( + partials: Sequence[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + drafter: bool = False, +) -> tuple[torch.Tensor, ...]: + """The boundary arithmetic once each rank's BF16 partial is known (shared with the finalize entry). + + ``drafter``: the Eagle3 drafter boundary (opts bit 16), whose stock is a plain BF16 all-reduce, then FlashInfer + fused_add_rmsnorm: the RMS statistic and the normalization take the unrounded FP32 sum bf16(rank sum) + residual; + only the stored residual is rounded to BF16. + + The stock rounding points of this target (a BF16 all-reduce, then the fused residual add + RMSNorm): ranks are + summed in FP32 in rank order and rounded to BF16, the residual added and rounded to BF16 again; the RMS statistic + and the normalization are done in float64 and rounded to BF16 once. + """ + total = torch.zeros(partials[0].shape, dtype=torch.float32, device=residual.device) + for p in partials: + total = total + p.float() + pre = total.to(torch.bfloat16).float() + residual.float() + updated = pre.to(torch.bfloat16) + u = pre.double() if drafter else updated.double() + inv = torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + eps) + normed = (u * inv * norm_w.double()).to(torch.bfloat16) + if not quant: + return normed, updated + return (normed, updated, *mxfp8_rows(normed)) + + +class M3McAllreduceNorm(OpWrapper): + """One TP group's boundary: ``updated = bf16(residual + bf16(sum_r bf16(x_r + x2_r)))``, ``normed = + RMSNorm(updated)``: bitwise the stock all-reduce + fused residual RMSNorm (+ mxfp8_quantize) of SM107 TP2. + + Every rank pushes its partial through the multicast mapping ``ws_mc_ptr`` into every rank's exchange, then polls + its own view ``ws_uc`` (re-arming the words it read) and rotates the per-CTA call parities in ``flags``; the + workspace is the ``McAllReduceWorkspace`` bucket of ``maxt`` tokens. All ranks must make the same calls in the + same stream order. ``prepushed`` 2 means a producer already pushed ``x`` (and launched this grid after its + pushes); 3 that two producers pushed ``x`` into exchange set A and ``x2`` into set B. Returns + ``[normed, updated]`` or, with ``quant``, also the MXFP8 copy of ``normed`` (E4M3 rows, swizzled UE8M0). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The tp2 decode boundaries (hidden 6144, 2 ranks, eps 1e-6): every decode step fits the 8-token bucket, epilogue + # option 7 at 1, 2, 7, 8 tokens and 15 at 3..6 (kernels/m3_mc_allreduce.AR_OPTS_BY_M; every option is + # bit-identical). + CELLS: tuple[Cell, ...] = ( + Cell( + why="one decode token at the attention boundary, x only, option 7", + spec=dict(m=1, maxt=8, opts=7, x2=False, quant=False, prepushed=0), + ), + Cell( + why="MoE boundary at 4 rows (one request verifying 1 + 3): shared + routed summed in-kernel, MXFP8 copy", + spec=dict(m=4, maxt=8, opts=15, x2=True, quant=True, prepushed=0), + ), + Cell( + why="MoE boundary at 8 rows (two requests), MXFP8 copy for the next QKV", + spec=dict(m=8, maxt=8, opts=7, x2=True, quant=True, prepushed=0), + ), + Cell( + why="attention boundary whose 8 rows the o_proj GEMV already pushed (prepushed 2)", + spec=dict(m=8, maxt=8, opts=7, x2=False, quant=False, prepushed=2), + ), + # Every other decode row count (modeling's MC_ALLREDUCE), the second addend and the MXFP8 copy alternating. + *( + Cell( + why=f"{m} rows: a boundary row count between the ones above", + spec=dict( + m=m, + maxt=8, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1], + x2=m % 2 == 0, + quant=m % 3 == 0, + prepushed=0, + ), + ) + for m in range(1, 9) + if m not in (1, 4, 8) + ), + # R1VR8 (modeling's MC_ALLREDUCE past 8 rows): 9..16 rows on the 16-token bucket, 17..32 on the 48-token + # bucket, option 15 (kernels/m3_mc_allreduce AR_OPTS_BY_BUCKET), the second addend and the MXFP8 copy + # alternating (bitwise against the 8-row path at 1..32 rows). + *( + Cell( + why=f"{m} rows: an R1-VR-TP8 decode step on the {16 if m <= 16 else 48}-token bucket", + spec=dict( + m=m, + maxt=16 if m <= 16 else 48, + opts=15, + x2=m % 2 == 0, + quant=m % 3 == 0, + prepushed=0, + ), + ) + for m in range(9, 33) + ), + # The Eagle3 drafter boundaries (modeling's DRAFT_MC_AR): option 16 (the drafter's stock rounding) on + # 23 / 31 at every draft-pass row count 1..8 (pass 0: 4 x requests, later passes: requests), x only. + *( + Cell( + why=f"Eagle3 drafter boundary at {m} rows (opts {23 if m in (1, 2, 7, 8) else 31})", + spec=dict( + m=m, + maxt=8, + opts=23 if m in (1, 2, 7, 8) else 31, + x2=False, + quant=False, + prepushed=0, + drafter=True, + ), + ) + for m in range(1, 9) + ), + # Every other row count the o_proj pushes (modeling's ATTN_PUSH). + *( + Cell( + why=f"{m} rows prepushed by the o_proj GEMV", + spec=dict( + m=m, + maxt=8, + opts=(7, 7, 15, 15, 15, 15, 7, 7)[m - 1], + x2=False, + quant=False, + prepushed=2, + ), + ) + for m in range(1, 8) + ), + ) + + note = """ + The epilogue option bits (1 early residual / weight load, 2 st.async cluster exchange, 4 early parity rotation, + 8 max carveout) are bit-identical by construction; only the four (bucket, option) pairs of the sm_103 kernel are + built. SM107 TP2 against the stock AUTO all-reduce + fused residual RMSNorm + mxfp8_quantize, rows 1..24, x alone + and x + x2: normed, updated and the MXFP8 copy (data and the scale bytes of valid rows) bitwise equal. `prepushed` + 1 (pushed, grid wait kept) is accepted by the op but no caller passes it. `pdl` changes only launch overlap and is + driven off in the cells. + + `normed` is gated to one BF16 rounding step: the kernel's RMS statistic is a warp / cluster tree of FP32 sums + and its `rsqrtf` is approximate, so a row whose scaled value lands next to a BF16 midpoint can round either + way. `updated` and the MXFP8 copy of the op's own `normed` are bit-exact. + """ + + def __call__( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + cap_out: Optional[torch.Tensor] = None, + cap_w: Optional[torch.Tensor] = None, + cap_eps: float = 0.0, + own_op: bool = False, + ) -> list[torch.Tensor]: + # The capture operands only with a capture: an op without the capture epilogue (an older image) takes the + # rest unchanged. own_op: this target's copy (trtllm::minimax_m3_sm107_tp8_mc_allreduce_norm, the same + # kernels with the capture epilogue; modeling's MC_AR_OWN_OP). + op = ( + torch.ops.trtllm.minimax_m3_sm107_tp8_mc_allreduce_norm + if own_op + else torch.ops.trtllm.minimax_m3_sm107_tp2_mc_allreduce_norm + ) + return op( + x, + x2, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + None, + None, + maxt, + quant, + prepushed, + opts, + *((cap_out, cap_w, cap_eps) if cap_out is not None else ()), + ) + + def reference( + self, + per_rank_x: Sequence[torch.Tensor], + per_rank_x2: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + drafter: bool = False, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute (``drafter``: the opts-16 drafter boundary). + + The documented collective exception: it takes every rank's ``x`` (and ``x2``) in rank order, not the calling + rank's. Each rank's partial is ``bf16(x + x2)`` (``x`` alone without ``x2``), whatever pushed it. + """ + partials = [] + for r, x in enumerate(per_rank_x): + if per_rank_x2 is None: + partials.append(x) + else: + partials.append((x.float() + per_rank_x2[r].float()).to(torch.bfloat16)) + return boundary(partials, residual, norm_w, eps, quant, drafter) + + def is_valid( + self, + x: torch.Tensor, + x2: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + prepushed: int, + opts: int, + cap_out: Optional[torch.Tensor] = None, + cap_w: Optional[torch.Tensor] = None, + cap_eps: float = 0.0, + own_op: bool = False, + ) -> None: + h = residual.shape[-1] + # The op checks norm_w's element count, not its layout: a strided view is read as packed memory. + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + # The workspace is addressed as [set][parity][maxt][world][H / 2] words and never bounds-checked: a smaller + # buffer (another bucket's, or one built for fewer ranks) is polled and re-armed out of range. + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + # Flags are per (token, cluster CTA) of the bucket; the op checks the count, not that they belong to the + # workspace. Flags of another bucket hold another call history, and the parity they select is not the one the + # peers push into: the poll then waits on the other half forever or reads last call's rows. + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + + +m3_mc_allreduce_norm = M3McAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_mc_finalize_allreduce_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_mc_finalize_allreduce_norm.py new file mode 100644 index 000000000000..8b74c9974397 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/comm/m3_mc_finalize_allreduce_norm.py @@ -0,0 +1,215 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MoE finalize + multicast all-reduce + residual + RMSNorm (+ MXFP8 copy) of the decode MoE boundary, 4 ranks.""" + +from __future__ import annotations + +from typing import Optional, Sequence + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_mc_allreduce_norm import boundary + + +def finalize_partial( + fc2: torch.Tensor, perm: torch.Tensor, scales: torch.Tensor, shared: Optional[torch.Tensor] +) -> torch.Tensor: + """One rank's BF16 partial: ``bf16(bf16(sum_k scale_k * fc2[perm_k]) + shared)``, the routed sum in FP32 in k + order and rounded to BF16, then the shared expert added as stock adds it (one more BF16 rounding). + + The kernel's per-k step is a fused multiply-add; it is reproduced exactly by the float64 product (exact for two + FP32 operands) plus the FP32 accumulator, rounded back to FP32. Dropped entries (perm -1) are skipped. + """ + m, topk = perm.shape + acc = torch.zeros(m, fc2.shape[1], dtype=torch.float32, device=fc2.device) + sc = scales.reshape(m, topk).float() + for k in range(topk): + rows = perm[:, k].long() + keep = (rows >= 0).unsqueeze(-1) + gathered = fc2[rows.clamp_min(0)].double() + step = (gathered * sc[:, k : k + 1].double() + acc.double()).float() + acc = torch.where(keep, step, acc) + if shared is not None: + acc = acc.to(torch.bfloat16).float() + shared.float() + return acc.to(torch.bfloat16) + + +class M3McFinalizeAllreduceNorm(OpWrapper): + """The MoE boundary with the routed experts' finalize folded in. + + Per rank the partial is ``bf16(bf16(sum_k scales[t, k] * fc2[perm[t, k]]) + shared[t])`` (``perm`` -1 = dropped, + ``scales`` fp32 or bf16), then the group's all-reduce, residual add and RMSNorm exactly as + ``m3_mc_allreduce_norm``: same workspace, flags and call-order contract. Returns ``[normed, updated]`` or, with + ``quant``, also the MXFP8 copy of ``normed``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # Not taken at tp2 (modeling's MC_FINALIZE is empty: the decode step is at most 8 rows); the cells are the tp4 + # target's, on the SM107 kernel. The tp4 target takes this path from 9 tokens (top-4 of 128 experts, fp32 routing + # weights, the shared expert always present): the 16- and 48-token buckets, epilogue option 15. + CELLS: tuple[Cell, ...] = ( + Cell( + why="the smallest finalize call (9 tokens, 16-token bucket) with the next QKV's MXFP8 copy", + spec=dict(m=9, maxt=16, opts=15, topk=4, quant=True, dropped=0), + ), + Cell( + why="a full 16-token bucket with dropped expert slots", + spec=dict(m=16, maxt=16, opts=15, topk=4, quant=False, dropped=5), + ), + Cell( + why="the 48-token bucket at a partial fill", + spec=dict(m=24, maxt=48, opts=15, topk=4, quant=False, dropped=0), + ), + Cell( + why="the 48-token bucket full, MXFP8 copy", + spec=dict(m=48, maxt=48, opts=15, topk=4, quant=True, dropped=3), + ), + # Every other row count of the unfinalized MoE (MC_FINALIZE): the bucket and options the target picks. + *( + Cell( + why=f"{m} tokens: a finalize row count between the ones above", + spec=dict( + m=m, + maxt=16 if m <= 16 else 48, + opts=15, + topk=4, + quant=m % 2 == 0, + dropped=m % 4, + ), + ) + for m in range(9, 49) + if m not in (9, 16, 24, 48) + ), + ) + + note = """ + BF16 routing weights and the 8-token bucket are accepted by the op (option 7 there) but the target only enters + this path at 9 tokens and up, with the built-in MoE's fp32 weights; neither is a cell. The rest is as in + `m3_mc_allreduce_norm`'s note. + """ + + def __call__( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + cap_out: Optional[torch.Tensor] = None, + cap_w: Optional[torch.Tensor] = None, + cap_eps: float = 0.0, + own_op: bool = False, + ) -> list[torch.Tensor]: + # The capture operands only with a capture: an op without the capture epilogue (an older image) takes the + # rest unchanged. own_op: this target's copy (trtllm::minimax_m3_sm107_tp8_mc_allreduce_norm, the same + # kernels with the capture epilogue; modeling's MC_AR_OWN_OP). + op = ( + torch.ops.trtllm.minimax_m3_sm107_tp8_mc_allreduce_norm + if own_op + else torch.ops.trtllm.minimax_m3_sm107_tp2_mc_allreduce_norm + ) + return op( + fc2, + shared, + residual, + norm_w, + eps, + ws_uc, + ws_mc_ptr, + flags, + rank, + world, + cluster, + pdl, + perm, + scales, + maxt, + quant, + 0, + opts, + *((cap_out, cap_w, cap_eps) if cap_out is not None else ()), + ) + + def reference( + self, + per_rank_fc2: Sequence[torch.Tensor], + per_rank_perm: Sequence[torch.Tensor], + per_rank_scales: Sequence[torch.Tensor], + per_rank_shared: Optional[Sequence[torch.Tensor]], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + quant: bool, + ) -> tuple[torch.Tensor, ...]: + """The boundary the group is supposed to compute, from every rank's MoE outputs in rank order (the + collective exception, as in ``m3_mc_allreduce_norm.reference``).""" + partials = [ + finalize_partial( + fc2, + per_rank_perm[r], + per_rank_scales[r], + None if per_rank_shared is None else per_rank_shared[r], + ) + for r, fc2 in enumerate(per_rank_fc2) + ] + return boundary(partials, residual, norm_w, eps, quant) + + def is_valid( + self, + fc2: torch.Tensor, + perm: torch.Tensor, + scales: torch.Tensor, + shared: Optional[torch.Tensor], + residual: torch.Tensor, + norm_w: torch.Tensor, + eps: float, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + flags: torch.Tensor, + rank: int, + world: int, + cluster: int, + pdl: bool, + maxt: int, + quant: bool, + opts: int, + cap_out: Optional[torch.Tensor] = None, + cap_w: Optional[torch.Tensor] = None, + cap_eps: float = 0.0, + own_op: bool = False, + ) -> None: + h = residual.shape[-1] + assert norm_w.is_contiguous(), ( + "norm_w must be contiguous; the kernel reads it as packed memory" + ) + need = 4 * maxt * world * (h // 2) + assert ws_uc.is_contiguous() and ws_uc.numel() >= need, ( + f"ws_uc holds {ws_uc.numel()} words, the {maxt}-token bucket of {world} ranks needs {need}" + ) + assert flags.numel() == maxt * cluster, ( + f"flags has {flags.numel()} counters; the {maxt}-token bucket's are exactly {maxt * cluster}" + ) + # The kernel gathers fc2[perm] without a bound: an index past the FC2 rows reads whatever follows them. + lo, hi = int(perm.min()), int(perm.max()) + assert lo >= -1 and hi < fc2.shape[0], ( + f"perm spans [{lo}, {hi}]; entries must be -1 (dropped) or a row of fc2 (< {fc2.shape[0]})" + ) + + +m3_mc_finalize_allreduce_norm = M3McFinalizeAllreduceNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/embedding/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/embedding/__init__.py new file mode 100644 index 000000000000..e9f4d8e20b94 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/embedding/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Embedding ops of this target: the generic token-embedding gather.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/embedding/m3_embedding.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/embedding/m3_embedding.py new file mode 100644 index 000000000000..71fce6b253cb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/embedding/m3_embedding.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The generic token-embedding gather over the replicated table (an id outside the table gives a zero row).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.modules.embedding import pre_comm_embedding_ops + + +class M3Embedding(OpWrapper): + """``table[ids]`` with a zero row for an id outside ``[0, V)`` (the speculative sampler pads with -1). + + The table is the full (replicated) bf16 embedding [V, 6144]; ids int32 / int64 of any count. The fused decode + embedding (``m3_embedding_firstnorm``) takes decode steps of up to 8 rows; this entry takes every other step. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="a prefill chunk, int32 ids", spec=dict(rows=2048, id_dtype=torch.int32, pad=False) + ), + Cell( + why="an 8-row decode verify, int64 ids", + spec=dict(rows=8, id_dtype=torch.int64, pad=False), + ), + Cell( + why="draft rows padded with -1 ids", spec=dict(rows=32, id_dtype=torch.int32, pad=True) + ), + ) + + note = """ + An exact gather: `reference` must match bit for bit. + """ + + def __call__(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + return pre_comm_embedding_ops(ids, table, 1, 0, None, 0, table.shape[0], False, 0) + + def reference(self, ids: torch.Tensor, table: torch.Tensor) -> torch.Tensor: + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] + return rows * valid.unsqueeze(-1).to(table.dtype) + + +m3_embedding = M3Embedding() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/__init__.py new file mode 100644 index 000000000000..9cc374bdf1cc --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GEMM entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m31_fc_shard.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m31_fc_shard.py new file mode 100644 index 000000000000..ac0b0aa0cbe5 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m31_fc_shard.py @@ -0,0 +1,141 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""FC_SHARD: the DSpark drafter's fc as a tensor-parallel column shard plus an all-gather, bit for bit the full fc. + +The drafter's fc ([M, layers x H] -> [M, H], bf16, no bias) is replicated on every rank, so each rank reads the whole +weight. Here each rank computes its H / tp output columns and the ranks all-gather them. Each column is computed by +``trtllm::minimax_m31_fc_shard_gemm`` with the cuBLASLt algo the full fc uses (``PINNED_CFG``), so every output column +is the same sum of the same K-split partials as in the full GEMM and the gathered result equals +``torch.nn.functional.linear(x, w)``. ``FcShard.build`` checks that on the device before the first use (real-scale and +2^+-12 block-scaled probes, every row count up to ``MAX_ROWS``) and returns None, keeping the stock fc, when the op is +not built, the shape does not apply, or any probe differs. +""" + +from __future__ import annotations + +from typing import Optional + +import torch +import torch.nn.functional as F + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.distributed import allgather +from tensorrt_llm.logger import logger + +# cuBLASLt's algo for the full fc (K = 30720 -> N = 6144, bf16 in / out, fp32 compute) in the VR image's cuBLASLt: +# ALGO_ID, TILE_ID, SPLITK_NUM, REDUCTION_SCHEME, CTA_SWIZZLING, CUSTOM_OPTION, STAGES_ID, INNER_SHAPE_ID, +# CLUSTER_SHAPE_ID. The build self-check rejects it if this cuBLASLt picks a different one for the full fc. +PINNED_CFG = (66, 393, 22, 2, 0, 1, 35, 0, 3) +MAX_ROWS = 8 # capacity: fixed: the K7 verify rows at batch 1 (TP8 serves c1 only) + +_STATE: dict = {} +_OP = "minimax_m31_fc_shard_gemm" + + +def fc_shard_op(): + """``trtllm::minimax_m31_fc_shard_gemm`` if this build carries it, else None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, _OP, None) + if op is not None: + # The op's fake lives in the catalog's M3.1 fake table (registered once, whoever loads it first). + from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import m31_fake_ops + + m31_fake_ops.register() + _STATE["op"] = op + return _STATE["op"] + + +def _probes(rows: int, k: int, device: torch.device): + """A real-scale and a 2^+-12 block-scaled bf16 probe [rows, k] (128-column blocks), seeded.""" + g = torch.Generator(device="cpu").manual_seed(31 * rows + 7) + x = torch.randn(rows, k, generator=g) + yield x.to(torch.bfloat16).to(device) + e = torch.randint(-12, 13, (rows, (k + 127) // 128), generator=g).float() + yield (x * torch.pow(2.0, e).repeat_interleave(128, dim=1)[:, :k]).to(torch.bfloat16).to(device) + + +class FcShard: + """The fc split by output columns over the tensor-parallel group; ``__call__(x)`` = the full fc's output.""" + + def __init__(self, w: torch.Tensor, w_shard: torch.Tensor, mapping, op) -> None: + self.source = w + self.w_shard = w_shard + self.mapping = mapping + self._op = op + self._cfg = list(PINNED_CFG) + self._logged = False + + @classmethod + def build(cls, fc: torch.nn.Module, mapping) -> Optional["FcShard"]: + """This rank's shard of ``fc``, or None (the stock fc stays) with the reason logged. Every rank of the + tensor-parallel group takes the shard only if every rank's check passed, so the ranks agree on whether the fc + all-gathers (the agreement is one tiny all-gather that every rank issues, whatever its own outcome).""" + tp, rank = int(mapping.tp_size), int(mapping.tp_rank) + shard, why = cls._check(fc, tp, rank, mapping) + if tp >= 2 and torch.cuda.is_available(): + ok = torch.tensor([0 if why else 1], dtype=torch.int32, device="cuda") + if not bool(allgather(ok, mapping, dim=0).all()): + why = why or "the check failed on another rank" + if why: + logger.info(f"[M3 dispatch] FC_SHARD not taken on rank {rank}: {why}") + return None + logger.info( + f"[M3 prebuild] FC_SHARD set up on rank {rank}/{tp}: fc {tuple(shard.source.shape)} -> shard " + f"{tuple(shard.w_shard.shape)}, bitwise to the full fc at 1..{MAX_ROWS} rows" + ) + return shard + + @classmethod + def _check(cls, fc: torch.nn.Module, tp: int, rank: int, mapping): + """(shard, None) when this rank's pinned shard GEMM is bitwise to the full fc on every probe, else + (None, reason).""" + op = fc_shard_op() + if op is None: + return None, f"{_OP} is not built" + w = getattr(fc, "weight", None) + if ( + w is None + or getattr(fc, "bias", None) is not None + or w.dtype != torch.bfloat16 + or w.dim() != 2 + or not w.is_cuda + or tp < 2 + or w.shape[0] % tp != 0 + ): + return None, f"fc shape / dtype / tp do not apply (tp {tp})" + cols = w.shape[0] // tp + shard = cls(w, w[rank * cols : (rank + 1) * cols].contiguous(), mapping, op) + try: + for rows in range(1, MAX_ROWS + 1): + for x in _probes(rows, w.shape[1], w.device): + want = F.linear(x, w)[:, rank * cols : (rank + 1) * cols] + if not torch.equal(op(x, shard.w_shard, shard._cfg), want): + return ( + None, + f"the pinned shard GEMM differs from the full fc at {rows} rows", + ) + except RuntimeError as e: + return None, f"the pinned cuBLASLt algo is rejected ({e})" + torch.cuda.synchronize() + return shard, None + + def applies(self, fc: torch.nn.Module, rows: int) -> bool: + """Whether ``fc`` is the fc this shard was built from and ``rows`` is within the checked row counts.""" + return getattr(fc, "weight", None) is self.source and 0 < rows <= MAX_ROWS + + def partial(self, x: torch.Tensor) -> torch.Tensor: + """This rank's output columns [rows, H / tp] (no collective: safe on a side stream).""" + if not self._logged: + self._logged = True + logger.info( + f"[M3 dispatch] FC_SHARD taken: drafter fc as {self.w_shard.shape[0]}-column shards + all-gather on " + f"rank {self.mapping.tp_rank}" + ) + return self._op(x.contiguous(), self.w_shard, self._cfg) + + def gather(self, y: torch.Tensor) -> torch.Tensor: + """The full fc output from every rank's ``partial`` (the tensor-parallel all-gather, in rank order).""" + return allgather(y, self.mapping, dim=-1) + + def __call__(self, x: torch.Tensor) -> torch.Tensor: + return self.gather(self.partial(x)) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_bf16_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_bf16_linear.py new file mode 100644 index 000000000000..b973b175d8b4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_bf16_linear.py @@ -0,0 +1,41 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The BF16 Eagle3 drafter's projections: ``x @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3Bf16Linear(OpWrapper): + """``F.linear(x, w)`` for bf16 ``x`` [M, K] and a bf16 weight [N, K] (cuBLAS, fp32 accumulation), bf16 out. + + The drafter's fc over the captured target layers (K = 3 x 6144), its fused QKV (K = 2 x 6144, this rank's heads), + o_proj (this rank's input columns), gate_up and down projections run it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="drafter fc, 32 verify rows", spec=dict(m=32, n=6144, k=18432)), + Cell(why="drafter QKV, nine draft rows", spec=dict(m=9, n=1280, k=12288)), + Cell(why="drafter o_proj, a prefill chunk", spec=dict(m=1024, n=6144, k=1024)), + Cell(why="drafter gate_up, 16 rows", spec=dict(m=16, n=4608, k=6144)), + Cell(why="drafter down, eight rows", spec=dict(m=8, n=6144, k=2304)), + ) + + note = """ + cuBLAS's split and accumulation order: `reference` multiplies in fp32 (one bf16 rounding), within the test's band. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x, w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return (x.float() @ w.float().t()).to(x.dtype) + + +m3_bf16_linear = M3Bf16Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_dense_swiglu.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_dense_swiglu.py new file mode 100644 index 000000000000..a0013e3ded3b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_dense_swiglu.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The dense MLP's swigluoai activation as one C++ launch with PDL and an early release (DENSE_SWIGLU_CPP).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +# The dense layers' swigluoai parameters, which the kernel hard-codes (modeling_minimaxm3._build_swiglu_oai_dense_mlp). +LIMIT, ALPHA, BETA = 7.0, 1.702, 1.0 + + +class M3DenseSwiglu(OpWrapper): + """``min(g, 7) * sigmoid(1.702 * min(g, 7)) * (clamp(u, -7, 7) + 1)`` of a bf16 [rows, 2d] gate | up tensor, + rounded once to bf16: trtllm::minimax_m3_sm107_tp8_swiglu_oai (minimaxM3Sm107Tp8/swigluOai.cu), bit for bit the + Triton ``torch.ops.trtllm.silu_and_mul`` at swiglu_limit 7, swiglu_alpha 1.702, swiglu_beta 1. With ``use_pdl`` the + grid waits on its predecessor and releases its dependents before it computes, so the down GEMV stages its weights + during the activation (the GEMV reads the activation only after its own grid wait). Contiguous rows with an even + half width; ``__call__`` returns None otherwise or for other activation parameters (the caller runs the Triton + op).""" + + ARCHS = frozenset({Arch.SM_107}) + + # The TP8 dense MLP shard: gate | up 3072 wide (d = 1536), at the decode rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="the TP8 dense MLP at 1 row", spec=dict(rows=1, d=1536)), + Cell(why="the TP8 dense MLP at 4 rows (1 request x 4 tokens)", spec=dict(rows=4, d=1536)), + Cell(why="the TP8 dense MLP at 8 rows", spec=dict(rows=8, d=1536)), + ) + + note = """ + Elementwise, one rounding: bit for bit the Triton kernel (its instruction sequence: the clamps, (u + 1), the + alpha-scaled ex2.approx sigmoid, div.full), checked over every bf16 gate and every bf16 up pattern. + """ + + def __call__( + self, + x: torch.Tensor, + limit: Optional[float] = LIMIT, + alpha: Optional[float] = ALPHA, + beta: Optional[float] = BETA, + use_pdl: bool = True, + ) -> Optional[torch.Tensor]: + if ( + (limit, alpha, beta) != (LIMIT, ALPHA, BETA) + or x.dtype != torch.bfloat16 + or x.dim() != 2 + or not x.is_cuda + or not x.is_contiguous() + or x.shape[1] % 4 != 0 + or x.data_ptr() % 4 != 0 + ): + return None + out = torch.empty((x.shape[0], x.shape[1] // 2), dtype=x.dtype, device=x.device) + torch.ops.trtllm.minimax_m3_sm107_tp8_swiglu_oai(x, out, bool(use_pdl)) + return out + + def reference(self, x, limit=LIMIT, alpha=ALPHA, beta=BETA, use_pdl=True) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=limit, swiglu_alpha=alpha, swiglu_beta=beta + ) + + def is_valid(self, x, limit=LIMIT, alpha=ALPHA, beta=BETA, use_pdl=True) -> None: + assert x.dtype == torch.bfloat16 and x.dim() == 2, "x: bf16 [rows, 2d]" + + +m3_dense_swiglu = M3DenseSwiglu() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_draft_lowm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_draft_lowm.py new file mode 100644 index 000000000000..10694aa0ce6d --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_draft_lowm.py @@ -0,0 +1,160 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""BF16 x BF16 -> BF16 low-M GEMM of the Eagle3 drafter's linears at fixed FlashInfer tactics (SM10x).""" + +from __future__ import annotations + +from typing import Mapping, Optional, Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: (N, K) of the TP8 drafter shards -> ((max rows, kind, tactic), ...), rows ascending. Kind "d" is FlashInfer's direct +#: SIMT kernel with tactic (block, outputs per block, rows per block); kind "s" is its split-K tensor-core kernel with +#: tactic (mma_m, mma_n, split_k, ab_stages). A row count is served by the first entry whose max rows holds it; the +#: rows modeling's DRAFT_LOWM covers are only the ones measured (the fastest tactic at no larger error vs fp64 than +#: cuBLAS). TP8 shards (64 query / 4 KV heads of 128, KV heads replicated across rank pairs; intermediate 18432): +#: qkv 1280 x 12288, o_proj 6144 x 1024, gate_up 4608 x 6144, down 6144 x 2304. Rubin SM107 TP8 sweep (mclk +#: locked, rotating weights >= 3x L2, CUDA-graph timing), us cuBLAS -> low-M at 1 / 4 rows: qkv 6.15 -> 3.94 / 6.26 -> +#: 5.69, o 4.97 -> 2.42 / 4.48 -> 2.68, gate_up 8.09 -> 5.78 / 8.11 -> 6.99, down 5.62 -> 4.00 / 5.74 -> 3.94. The fc is +#: replicated (the TP2 shape): its TP2 tactic and row. +TACTICS: Mapping[Tuple[int, int], tuple] = { + # qkv + (1280, 12288): ((1, "d", (256, 2, 1)), (4, "d", (128, 4, 4))), + # o_proj + (6144, 1024): ((1, "s", (64, 8, 1, 10)), (4, "s", (64, 8, 2, 6))), + # gate_up + (4608, 6144): ((1, "d", (128, 2, 1)), (4, "d", (64, 4, 4))), + # down + (6144, 2304): ((1, "s", (64, 8, 2, 6)), (4, "s", (64, 8, 1, 10))), + # the Eagle3 fc over the three captured hidden states + (6144, 18432): ((1, "d", (384, 2, 1)),), +} +#: Rows each (N, K) is certified at (a subset of what its table would serve). +ROWS: Mapping[Tuple[int, int], Tuple[int, ...]] = { + (1280, 12288): (1, 4), + (6144, 1024): (1, 4), + (4608, 6144): (1, 4), + (6144, 2304): (1, 4), + (6144, 18432): (1,), +} + + +def tactic_for(n: int, k: int, m: int) -> Optional[Tuple[str, tuple]]: + """(kind, tactic) of an [m, k] x [n, k]^T call, or None when the table does not serve it.""" + for rows, kind, tactic in TACTICS.get((n, k), ()): + if m <= rows: + return kind, tactic + return None + + +class M3DraftLowM(OpWrapper): + """``out`` [M, N] BF16 = ``x`` [M, K] BF16 @ ``weight`` [N, K]^T BF16, FP32 accumulation. + + ``kind`` / ``tactic`` pick FlashInfer's kernel: "d" runs ``run_direct_dense`` (SIMT, one dot product per output in + K order), "s" runs ``run_splitk_dense`` (tcgen05 MMA, K split over ``split_k`` CTAs and summed in FP32). Both + round once to BF16. The kernel is compiled on its first call per (M, N, K, tactic, PDL); the target compiles every + covered call before warmup (``M3Eagle3Draft.prebuild_linears``). + """ + + ARCHS = frozenset({Arch.SM_107}) + CELLS: tuple[Cell, ...] = ( + Cell(why="drafter qkv, 1 rows: d tactic", spec=dict(nk=(1280, 12288), m=1)), + Cell(why="drafter qkv, 4 rows: d tactic", spec=dict(nk=(1280, 12288), m=4)), + Cell(why="drafter o_proj, 1 rows: s tactic", spec=dict(nk=(6144, 1024), m=1)), + Cell(why="drafter o_proj, 4 rows: s tactic", spec=dict(nk=(6144, 1024), m=4)), + Cell(why="drafter gate_up, 1 rows: d tactic", spec=dict(nk=(4608, 6144), m=1)), + Cell(why="drafter gate_up, 4 rows: d tactic", spec=dict(nk=(4608, 6144), m=4)), + Cell(why="drafter down, 1 rows: s tactic", spec=dict(nk=(6144, 2304), m=1)), + Cell(why="drafter down, 4 rows: s tactic", spec=dict(nk=(6144, 2304), m=4)), + Cell(why="drafter fc, 1 rows: d tactic", spec=dict(nk=(6144, 18432), m=1)), + ) + + note = """ + The table is the TP8 drafter's: its shapes are the per-rank shards of the Eagle3 layer at 8-way tensor + parallelism (64 query and 4 KV heads, the KV heads replicated across rank pairs; intermediate 18432) and the + replicated fc, at the b1 decode rows (1 row in draft passes 1-2, 4 in pass 0). Certified at those shapes and + rows only; FlashInfer's kernels take other shapes, which are not measured here. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + own_splitk: bool = False, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + if kind == "d": + from flashinfer.gemm.kernels.dense_bf16_gemm_direct import ( + DirectTactic, + run_direct_dense, + ) + + run_direct_dense(x, weight.t(), out, use_pdl, DirectTactic(*tactic)) + elif own_splitk: + # This target's copy of the split-K kernel (kernels/m3_lowm_splitk): bit for bit FlashInfer's; with ``ar`` + # it also pushes the rows into a drafter boundary all-reduce (modeling's DRAFT_AR_PUSH). + from ...kernels.m3_lowm_splitk import ( + SplitKTactic, + run_splitk_dense, + run_splitk_dense_push, + ) + + if ar is not None: + run_splitk_dense_push(x, weight.t(), out, use_pdl, SplitKTactic(*tactic), ar) + else: + run_splitk_dense(x, weight.t(), None, out, use_pdl, SplitKTactic(*tactic)) + else: + from flashinfer.gemm.kernels.dense_bf16_gemm_sm100_splitk import ( + SplitKTactic, + run_splitk_dense, + ) + + run_splitk_dense(x, weight.t(), None, out, use_pdl, SplitKTactic(*tactic)) + return out + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + own_splitk: bool = False, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """bf16(x @ weight^T) summed in fp64 and rounded once; ``out``, ``kind``, ``tactic`` and ``use_pdl`` only + choose where and how the op writes.""" + return (x.double() @ weight.double().t()).to(torch.bfloat16) + + def is_valid( + self, + x: torch.Tensor, + weight: torch.Tensor, + out: torch.Tensor, + kind: str, + tactic: tuple, + use_pdl: bool, + own_splitk: bool = False, + ar: Optional[tuple] = None, + ) -> None: + """The kernels index ``x`` and ``out`` densely: a strided view is read as if it were packed.""" + assert x.dim() == 2 and x.is_contiguous(), "x must be a contiguous [M, K] matrix" + assert weight.dim() == 2 and weight.is_contiguous(), ( + "weight must be a contiguous [N, K] matrix" + ) + assert out.is_contiguous() and tuple(out.shape) == (x.shape[0], weight.shape[0]), ( + "out must be a contiguous [M, N] matrix" + ) + assert ar is None or (own_splitk and kind == "s" and len(ar) == 5), ( + "ar (the boundary all-reduce push) runs only on this target's split-K copy" + ) + + +m3_draft_lowm = M3DraftLowM() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_fp32_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_fp32_linear.py new file mode 100644 index 000000000000..9702cc0778a4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_fp32_linear.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The router's fp32 projection beyond the decode routers' rows: ``float(x) @ w^T`` through torch's GEMM.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + + +class M3Fp32Linear(OpWrapper): + """fp32 router logits ``F.linear(float(x[M, 6144]), w[128, 6144])`` for prefill and steps above 48 rows (the + fp32 GEMV and the tensor-core router take 1..48). Whatever fp32 matmul precision the process runs with applies + (the engine leaves torch's default).""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="a prefill chunk", spec=dict(m=2048, e=128, k=6144)), + Cell( + why="a generation step above the tensor-core router's 48 rows", + spec=dict(m=64, e=128, k=6144), + ), + ) + + note = """ + Under torch's default fp32 matmul precision the product may be TF32: the test's band covers that rounding against + the true fp32 reference. + """ + + def __call__(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + return F.linear(x.to(torch.float32), w) + + def reference(self, x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + with true_fp32_matmul(): + return x.float() @ w.t() + + +m3_fp32_linear = M3Fp32Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mx_gemv.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mx_gemv.py new file mode 100644 index 000000000000..1f5d61ccd1ee --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mx_gemv.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 x MXFP8 GEMV (tcgen05, block-scaled, CuTe DSL) on an activation its producer already quantized.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.mx_gemv import op as _gemv +from ..moe.m3_moe_engine import mxfp8_dequant + +__all__ = ["PROJECTIONS", "gemv_is_valid", "m3_mx_gemv", "served_cells"] + +# (N, K) of this target's MXFP8 projections at tp2, the GEMV's tuned shapes. +QKV_SPARSE = (1536, 6144) # the fused QKV + index projection (index values disabled) +QKV = (1280, 6144) +O_PROJ = (6144, 1024) +SHARED_GATE_UP = (768, 6144) +SHARED_DOWN = (6144, 384) +DENSE_GATE_UP = (3072, 6144) +DENSE_DOWN = (6144, 1536) +PROJECTIONS = { + "sparse-layer QKV": QKV_SPARSE, + "dense-attention QKV": QKV, + "o_proj": O_PROJ, + "shared gate_up": SHARED_GATE_UP, + "shared down": SHARED_DOWN, + "dense gate_up": DENSE_GATE_UP, + "dense down": DENSE_DOWN, +} +# VR_GEMV48: the projections the GEMV also serves on 33..48-row generation steps, where it beats cuBLASLt's MXFP8 +# GEMM (SM107 us/call at 33..48 rows on the same MXFP8 activation: QKV_SPARSE 6.34..6.35 vs 6.94..6.95, +# SHARED_GATE_UP 5.87..5.88 vs 6.81..6.82; the dense-attention QKV is the sparse one's shape class). +GEMV48_SHAPES = (QKV_SPARSE, QKV, SHARED_GATE_UP) +GEMV48_MIN_TOKENS = 33 # capacity: fixed: past the R1VR8 decode bound (32 rows) + + +def served_cells(named: tuple[Cell, ...], x_bf16: bool, max_m: int) -> tuple[Cell, ...]: + """A cell for every (projection, rows) the GEMV serves up to ``max_m`` rows that ``named`` does not list. + + Certification is per projection: a projection's split-K, ring and MMA width differ, so each (N, K, M) the + decode dispatch can send here is its own cell. + """ + listed = {(cell.spec["nk"], cell.spec["m"]) for cell in named} + return tuple( + Cell(why=f"{name}, {m} rows", spec=dict(nk=nk, m=m)) + for name, nk in PROJECTIONS.items() + for m in range(1, max_m + 1) + if (nk, m) not in listed and _gemv.serves(nk[0], nk[1], m, x_bf16) + ) + + +def gemv_reference( + x: torch.Tensor, weight: torch.Tensor, weight_scale: torch.Tensor +) -> torch.Tensor: + """bf16(x @ dequant(weight)^T) for fp32 activation values x [M, K], summed in fp64 and rounded once.""" + return (x.double() @ mxfp8_dequant(weight, weight_scale).double().t()).to(torch.bfloat16) + + +def gemv_is_valid(weight: torch.Tensor, weight_scale: torch.Tensor) -> None: + n, k = weight.shape + # The kernel reads both from data_ptr() as dense buffers at the layout it was compiled for. + assert weight.dtype == torch.float8_e4m3fn and weight.is_contiguous(), ( + "weight must be contiguous e4m3 [N, K]" + ) + assert weight_scale.numel() == n * k // 32, ( + "weight_scale must be the 128x4-swizzled UE8M0 buffer [N * K / 32]" + ) + + +class M3MxGemv(OpWrapper): + """bf16 [M, N] = xq [M, K] (e4m3, 128x4-swizzled UE8M0 scales ``xsf``) x weight [N, K]^T (e4m3, swizzled + ``weight_scale``), M <= 48; None when the shape is not served (the caller falls back). + + ``trigger_early``: release the PDL dependents before the epilogue. ``cfg``: a (split, ring[, contig, nacc, a_tmem, + st_async]) tuning override; default the table's. ``use_pdl``: default the environment's. ``ar`` = (multicast + workspace address, flags address, rank, world, bucket tokens, mode): also push the rows into the next boundary + all-reduce's exchange (then ``op.last_pushed`` is True). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The projections' (N, K) at the token counts each build differs in: the MMA width (8 columns up to the TP2 decode + # step's 8 rows) is one, so one cell per projection pins the shape, and the rest are every served row count. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token (split 8)", spec=dict(nk=QKV_SPARSE, m=1)), + Cell( + why="sparse-layer QKV, 8 rows: two requests verifying 1 + 3", + spec=dict(nk=QKV_SPARSE, m=8), + ), + Cell( + why="dense-attention QKV, 4 rows: one request verifying 1 + 3", spec=dict(nk=QKV, m=4) + ), + Cell(why="o_proj, one token (split 2, K 1024)", spec=dict(nk=O_PROJ, m=1)), + Cell(why="o_proj, 8 rows", spec=dict(nk=O_PROJ, m=8)), + Cell(why="shared gate_up, 4 rows (split 8)", spec=dict(nk=SHARED_GATE_UP, m=4)), + Cell(why="shared down, one token (split 1, K 384)", spec=dict(nk=SHARED_DOWN, m=1)), + Cell(why="dense gate_up, 8 rows (N 3072)", spec=dict(nk=DENSE_GATE_UP, m=8)), + Cell(why="dense down, 4 rows (K 1536)", spec=dict(nk=DENSE_DOWN, m=4)), + ) + # Every other (projection, rows) the decode dispatch can send here (modeling's MX_GEMV_ROWS): up to the R1VR8 + # step bound of 32 rows, each served (projection, rows) its own cell. + CELLS = CELLS + served_cells(CELLS, x_bf16=False, max_m=32) + # VR_GEMV48: the generation steps of 33..48 rows (past the decode bound) on the QKV and shared gate_up projections, + # on an MXFP8 activation (modeling quantizes a bf16 one first). + CELLS = CELLS + tuple( + Cell(why=f"{name}, {m} rows (VR_GEMV48)", spec=dict(nk=nk, m=m)) + for name, nk in PROJECTIONS.items() + if nk in GEMV48_SHAPES + for m in range(GEMV48_MIN_TOKENS, _gemv.MAX_TOKENS + 1) + if _gemv.serves(nk[0], nk[1], m, False) + ) + + note = """ + The kernel sums in fp32 in split-K order where `reference` sums in fp64: the bf16 outputs agree within the band the + test derives (SM107: at most as many misrounded elements as the stock FlashInfer CuTe DSL GEMM on the same + operands, rand / wide-scale / cancellation / tie / zero inputs). Not covered: the all-reduce push (`ar`), which + needs a multicast exchange; `cfg` overrides beyond the table (the target passes none). Shapes outside the table + and M > 48 return None; the decode steps send at most 32 rows (R1VR8; the o_proj up to 16), and the 33..48-row + generation steps send the GEMV48_SHAPES only (VR_GEMV48). + """ + + def __call__( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv(xq, xsf, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The dequantized product; the launch options are ignored.""" + return gemv_reference(mxfp8_dequant(xq, xsf), weight, weight_scale) + + def is_valid( + self, + xq: torch.Tensor, + xsf: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + m, k = xq.shape + assert xq.dtype == torch.float8_e4m3fn and xq.is_contiguous(), ( + "xq must be contiguous e4m3 [M, K]" + ) + # A linear-layout [M, K / 32] scale buffer is read as the swizzled one: wrong scales, no error. + assert xsf.numel() >= (m + 127) // 128 * 128 * (k // 32), ( + "xsf must be the 128x4-swizzled scale buffer (rows padded to 128), not the linear one" + ) + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv = M3MxGemv() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mx_gemv_bf16.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mx_gemv_bf16.py new file mode 100644 index 000000000000..3b5d21235f29 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mx_gemv_bf16.py @@ -0,0 +1,100 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode MXFP8 GEMV (CuTe DSL) on a bf16 activation, quantized to MXFP8 in the kernel's prologue.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.mx_gemv import op as _gemv +from ..moe.m3_moe_engine import mxfp8_quant_dequant +from .m3_mx_gemv import ( + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, + gemv_is_valid, + gemv_reference, + served_cells, +) + +__all__ = ["m3_mx_gemv_bf16"] + + +class M3MxGemvBf16(OpWrapper): + """bf16 [M, N] = mxfp8_quantize(x) x weight [N, K]^T for a contiguous, 16-byte aligned bf16 ``x`` [M, K]; the + quantization is mxfp8_quantize's (UE8M0 scale per 32 = the smallest power of two >= amax / 448). None when the + shape is not served. The other arguments as ``m3_mx_gemv``'s. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The prologue quantizes one 32-element block per thread and round: the projections at the token counts the + # target sends here (no producer-quantized activation), every TP2 decode row count. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, one token", spec=dict(nk=QKV_SPARSE, m=1)), + Cell(why="dense-attention QKV, 8 rows", spec=dict(nk=QKV, m=8)), + Cell(why="o_proj, 4 rows (the attention output in bf16)", spec=dict(nk=O_PROJ, m=4)), + Cell(why="shared gate_up, 8 rows", spec=dict(nk=SHARED_GATE_UP, m=8)), + Cell(why="shared down, 4 rows (K 1536)", spec=dict(nk=SHARED_DOWN, m=4)), + Cell( + why="dense gate_up, 8 rows: 24 K tiles per split-2 CTA", + spec=dict(nk=DENSE_GATE_UP, m=8), + ), + Cell(why="dense down, one token", spec=dict(nk=DENSE_DOWN, m=1)), + ) + # Every other (projection, rows) the decode dispatch can send here (modeling's MX_GEMV_BF16_MAX_TOKENS). + CELLS = CELLS + served_cells(CELLS, x_bf16=True, max_m=8) + + note = """ + As `m3_mx_gemv`. The in-kernel quantization is mxfp8_quantize's bit for bit except where the kernel's + rcp.approx(448) and an exact 1/448 put a block's amax on different sides of a power of two (not observed on the + cells). Not covered: `ar`, `cfg` overrides. Returns None for rows beyond 8 rounds of the prologue per CTA. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> Optional[torch.Tensor]: + return _gemv.m3_mx_gemv_bf16(x, weight, weight_scale, trigger_early, cfg, use_pdl, ar) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> torch.Tensor: + """The MXFP8 round trip of x times the dequantized weight; the launch options are ignored.""" + return gemv_reference(mxfp8_quant_dequant(x), weight, weight_scale) + + def is_valid( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + trigger_early: bool = True, + cfg: Optional[tuple] = None, + use_pdl: Optional[bool] = None, + ar: Optional[tuple] = None, + ) -> None: + gemv_is_valid(weight, weight_scale) + + +m3_mx_gemv_bf16 = M3MxGemvBf16() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_cublaslt.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_cublaslt.py new file mode 100644 index 000000000000..ab681314c87b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_cublaslt.py @@ -0,0 +1,127 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The MXFP8 GEMM through cuBLASLt for generation steps wider than the decode kernels' rows. + +cuBLASLt's block-scaled MXFP8 matmul (``torch._scaled_mm`` with UE8M0 block scales), on an MXFP8 activation or on a +bf16 one it quantizes first. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner + +from .m3_mx_gemv import DENSE_DOWN, DENSE_GATE_UP, O_PROJ, QKV_SPARSE, SHARED_DOWN, SHARED_GATE_UP +from .m3_mxfp8_linear import mxfp8_reference + +# The projections the cuBLASLt GEMM takes, and its rows: past the R1VR8 decode bound (32) up to the K7 verify rows +# at batch 8 (64). SM107 microbench (us/call on the same MXFP8 activation, 33..64 rows): QKV_SPARSE +# 6.94..6.98 vs FlashInfer's 10.14..10.18, O_PROJ 3.06..3.10 vs 3.81..3.86, SHARED_GATE_UP 6.81..6.82 vs 10.04..10.08, +# SHARED_DOWN 2.49..2.51 vs 2.86..2.87, DENSE_GATE_UP 7.21..7.24 vs 10.46..10.47, DENSE_DOWN 3.62..3.77 vs 4.48..4.64. +SHAPES = (QKV_SPARSE, O_PROJ, SHARED_GATE_UP, SHARED_DOWN, DENSE_GATE_UP, DENSE_DOWN) +MIN_TOKENS = 33 # capacity: fixed: past the R1VR8 decode bound (32 rows) +MAX_TOKENS = 64 # capacity: fixed: the K7 verify rows at batch 8 + +_QUANTIZE = {} + + +def mxfp8_quantize(x: torch.Tensor): + """The bf16 activation's MXFP8 form by FlashInfer's CuTe DSL quantize. + + E4M3 [M, K] + 128x4-swizzled UE8M0 scales: the bytes of ``torch.ops.trtllm.mxfp8_quantize(x, True)``. + """ + runner = _QUANTIZE.get(x.dtype) + if runner is None: + runner = _QUANTIZE[x.dtype] = MXFP8QuantizeRunner(x.dtype) + act, act_scale = runner([x], tactic=MXFP8QuantizeRunner.CUTE_DSL) + return act.view(x.shape[0], x.shape[1]), act_scale + + +def _e4m3(t: torch.Tensor) -> torch.Tensor: + return t if t.dtype == torch.float8_e4m3fn else t.view(torch.float8_e4m3fn) + + +def _ue8m0(scale: torch.Tensor) -> torch.Tensor: + """The swizzled scale bytes as one flat UE8M0 vector (whatever element type the buffer was allocated with).""" + return scale.reshape(-1).view(torch.uint8).view(torch.float8_e8m0fnu) + + +class M3Mxfp8Cublaslt(OpWrapper): + """``act @ weight^T`` in bf16 for an MXFP8 weight [N, K] (swizzled scales), via cuBLASLt. + + The activation is MXFP8 (``act`` e4m3 [M, K] + ``act_scale`` swizzled) or, with ``act_scale`` None, bf16 and + quantized first. The GEMM is cuBLASLt's block-scaled MXFP8 matmul (VEC32 UE8M0 scales), reached through + ``torch._scaled_mm``; the algorithm is cuBLASLt's heuristic for the shape and the process's workspace (no + timing-based selection). + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = tuple( + Cell(why=f"[{n}, {k}], {m} rows, {src}", spec=dict(m=m, n=n, k=k, prequant=src == "MXFP8")) + for (n, k) in SHAPES + for m in range(MIN_TOKENS, MAX_TOKENS + 1) + for src in ("MXFP8", "bf16") + ) + + note = """ + Exactly the MXFP8 product (fp32 accumulation, one bf16 rounding): `reference` dequantizes both operands and + multiplies in fp32; the test's band covers the accumulation order. Against FlashInfer's MXFP8 GEMM on the same + operands (SM107) the output was within 0.61 ULP[bf16] at every shape and row, bit-identical on most cells. + Not covered: shapes outside SHAPES, rows outside 33..64. + """ + + def __call__( + self, + act: torch.Tensor, + act_scale: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + ) -> torch.Tensor: + if act_scale is None: + act, act_scale = mxfp8_quantize(act) + return torch._scaled_mm( + _e4m3(act), + _e4m3(weight).t(), + scale_a=_ue8m0(act_scale), + scale_b=_ue8m0(weight_scale), + out_dtype=torch.bfloat16, + ) + + def reference( + self, + act: torch.Tensor, + act_scale: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + ) -> torch.Tensor: + """The dequantized product of the MXFP8 operands (a bf16 activation quantized first).""" + if act_scale is None: + act, act_scale = mxfp8_quantize(act) + return mxfp8_reference(act, act_scale, weight, weight_scale).to(torch.bfloat16) + + def is_valid( + self, + act: torch.Tensor, + act_scale: Optional[torch.Tensor], + weight: torch.Tensor, + weight_scale: torch.Tensor, + ) -> None: + m, k = act.shape + n = weight.shape[0] + assert (n, k) in SHAPES and MIN_TOKENS <= m <= MAX_TOKENS, ( + f"m3_mxfp8_cublaslt is certified for {SHAPES} at {MIN_TOKENS}..{MAX_TOKENS} rows, got [{n}, {k}] x {m}" + ) + assert act.is_contiguous() and weight.is_contiguous(), ( + "m3_mxfp8_cublaslt takes contiguous operands" + ) + assert act_scale is not None or act.dtype == torch.bfloat16, ( + "a bf16 activation, or an MXFP8 one + scales" + ) + + +m3_mxfp8_cublaslt = M3Mxfp8Cublaslt() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_gemm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_gemm.py new file mode 100644 index 000000000000..2bfc6ee495fb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_gemm.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-graph MXFP8 GEMM on an activation its producer already quantized (rows the GEMV does not serve).""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm._torch.custom_ops.torch_custom_ops import mxfp8_gemm_autotuned + +from .m3_mxfp8_linear import mxfp8_reference + + +class M3Mxfp8Gemm(OpWrapper): + """``act @ weight^T`` for an MXFP8 activation (e4m3 [M, K] + 128x4-swizzled UE8M0 scales) and an MXFP8 weight + [N, K] (swizzled scales), bf16 out. The GEMM backend per token bucket is the AutoTuner's (the same runner, op key + and tactic cache as the engine-tuned quantize + GEMM); ``tune`` profiles in the engine's tuning pass.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The decode dispatch hands it the boundary / merge MXFP8 activation when the GEMV declines rows it serves: the + # sparse-layer QKV and the o_proj at the tp8 shards. + CELLS: tuple[Cell, ...] = ( + Cell(why="sparse-layer QKV, 4 rows", spec=dict(m=4, n=1536, k=6144)), + Cell(why="o_proj, 32 rows", spec=dict(m=32, n=6144, k=1024)), + Cell(why="shared expert down projection, 17 rows", spec=dict(m=17, n=6144, k=384)), + Cell(why="shared expert down projection, 32 rows", spec=dict(m=32, n=6144, k=384)), + ) + + note = """ + Exactly the MXFP8 product (fp32 accumulation, one bf16 rounding): `reference` dequantizes both operands and + multiplies in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + return mxfp8_gemm_autotuned(act, act_scale, weight, weight_scale, output_dtype, tune=tune) + + def reference( + self, + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + output_dtype: torch.dtype, + tune: bool, + ) -> torch.Tensor: + """The dequantized product; ``tune`` is ignored.""" + return mxfp8_reference(act, act_scale, weight, weight_scale).to(output_dtype) + + +m3_mxfp8_gemm = M3Mxfp8Gemm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_linear.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_linear.py new file mode 100644 index 000000000000..3f07fe2fb256 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_mxfp8_linear.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One MXFP8 projection on the engine-tuned path: dynamic MXFP8 quantize of a bf16 activation + block-scaled GEMM.""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import mxfp8_quantize_gemm_autotuned + +_BLOCK = 32 + + +def _unswizzle(scale: torch.Tensor, rows: int, cols: int) -> torch.Tensor: + """[rows, cols] UE8M0 bytes from the 128x4-swizzled layout (rows padded to 128, columns to 4).""" + pad_r = (rows + 127) // 128 * 128 + pad_c = (cols + 3) // 4 * 4 + s = scale.view(torch.uint8).reshape(pad_r // 128, pad_c // 4, 32, 4, 4) + return s.permute(0, 3, 2, 1, 4).reshape(pad_r, pad_c)[:rows, :cols] + + +def _dequant(x: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + rows, k = x.shape + s = _unswizzle(scale, rows, k // _BLOCK).to(torch.int32) + factor = torch.pow(2.0, (s - 127).to(torch.float32)) + return ( + x.view(torch.float8_e4m3fn).float().view(rows, k // _BLOCK, _BLOCK) * factor.unsqueeze(-1) + ).view(rows, k) + + +def mxfp8_reference(act, act_scale, weight, weight_scale) -> torch.Tensor: + """fp32 product of two dequantized MXFP8 operands (swizzled UE8M0 scales).""" + with true_fp32_matmul(): + return _dequant(act, act_scale) @ _dequant(weight, weight_scale).t() + + +class M3Mxfp8Linear(OpWrapper): + """``x @ weight^T`` for bf16 ``x`` [M, K] and an MXFP8 weight [N, K] (e4m3 + 128x4-swizzled UE8M0 scales), bf16 + out, on the backend the engine selected: + + * ``tuned_quantize_gemm`` (decode CUDA graphs once the engine tuned per-bucket backends): the AutoTuner's quantize + and GEMM per token bucket (``tune`` profiles them in the engine's tuning pass); + * otherwise the native ``mxfp8_quantize`` (swizzled scales) and then FlashInfer's MXFP8 GEMM (``flashinfer``) or + the CUTLASS block-scaled GEMM, through the autotuner while the engine tunes it (``native_tuning``), scaled by + ``alpha`` (fp32 [1], 1.0 for these checkpoints). + + Every MXFP8 projection of the target (the attention QKV and o_proj, the dense MLPs and the shared expert) runs it + outside the decode-step kernels. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell( + why="sparse-layer QKV, a prefill chunk, native GEMM", + spec=dict(m=2048, n=1536, k=6144, path="native"), + ), + Cell( + why="o_proj, a prefill chunk, native GEMM", + spec=dict(m=2048, n=6144, k=1024, path="native"), + ), + Cell( + why="dense down projection, eager decode, native GEMM", + spec=dict(m=4, n=6144, k=1536, path="native"), + ), + Cell( + why="shared gate_up, FlashInfer GEMM", + spec=dict(m=64, n=768, k=6144, path="flashinfer"), + ), + Cell(why="o_proj, tuned quantize + GEMM", spec=dict(m=64, n=6144, k=1024, path="tuned")), + ) + + note = """ + Exactly the MXFP8 product of the MXFP8 round trip of x (fp32 accumulation, one bf16 rounding): `reference` + quantizes x with the same block rule (UE8M0 scale = the block's power-of-two ceiling over e4m3's 448) and + multiplies dequantized operands in fp32; the test's band covers the accumulation order. + """ + + def __call__( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + if tuned_quantize_gemm: + return mxfp8_quantize_gemm_autotuned(x, weight, weight_scale, output_dtype, tune=tune) + act, act_scale = torch.ops.trtllm.mxfp8_quantize(x, True) + if flashinfer: + return torch.ops.trtllm.flashinfer_mm_mxfp8( + act, act_scale, weight, weight_scale, output_dtype + ) + gemm = ( + torch.ops.trtllm.mxfp8_mxfp8_gemm_autotuned + if native_tuning + else torch.ops.trtllm.mxfp8_mxfp8_gemm + ) + return gemm(act, act_scale, weight, weight_scale, alpha, output_dtype) + + def reference( + self, + x: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + alpha: torch.Tensor, + output_dtype: torch.dtype, + tuned_quantize_gemm: bool, + tune: bool, + flashinfer: bool, + native_tuning: bool, + ) -> torch.Tensor: + """The MXFP8 round trip of ``x`` times the dequantized weight, times ``alpha``; the backend flags are + ignored.""" + m, k = x.shape + blocks = x.float().view(m, k // _BLOCK, _BLOCK) + amax = blocks.abs().amax(-1, keepdim=True) + exp = torch.ceil(torch.log2(torch.where(amax > 0, amax / 448.0, torch.ones_like(amax)))) + scale = torch.pow(2.0, exp) + q = (blocks / scale).to(torch.float8_e4m3fn).float() * scale + with true_fp32_matmul(): + out = q.view(m, k) @ _dequant(weight, weight_scale).t() + return (out * alpha.float()).to(output_dtype) + + +m3_mxfp8_linear = M3Mxfp8Linear() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_target_head.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_target_head.py new file mode 100644 index 000000000000..f8deb858c843 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gemm/m3_target_head.py @@ -0,0 +1,346 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The target lm_head shard GEMV of a verify step of at most 8 rows over a lossless 12-bit coding of its BF16 weight, +and that coding (``encode_chunked`` / ``decode_reference`` / ``round_trip_ok``). + +The coding: the BF16 exponents of a trained weight sit almost all in a window of 15 consecutive values +``[base, base + 14]``. Each weight is one byte ``sign << 7 | mantissa`` plus a 4-bit code ``exponent - base``; code 15 +marks an exception, which decodes to +0 (it adds nothing to the MMA sums); its exact product ``x * w`` is added from a +per-tile list. The bytes are laid out in ``mma.m16n8k16`` A-fragment order: +per 16-row tile and unit of 4 k16 steps, each lane reads the sign/mantissa bytes of steps 0-1, of steps 2-3, then 4 +code words (``[tiles][K / 64][3][32 lanes][16 bytes]``). A coded weight is used only after ``round_trip_ok`` proved +it decodes to the BF16 weight bit for bit. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +__all__ = ["m3_target_head"] + +#: Verify rows per call (requests x (draft length + 1)). +MAX_ROWS = 8 +#: Warps per CTA (the split-tile scratch holds 2 x 4 x 32 fp32 per warp). +CTA_WARPS = 16 +#: The target lm_head's TP4 shard (200064-id vocabulary) and the hidden size. +TARGET_SHARD = 50016 +HIDDEN = 6144 +#: Codes per window: 0..14 decode, 15 marks an exception. +WINDOW = 15 +#: The largest exponent base whose window top plus one (base + 15) is still a finite BF16 exponent. +MAX_BASE = 254 - WINDOW +#: Rows per block of encode_chunked / round_trip_ok (bounded temporaries). +ROWS_PER_CHUNK = 4096 + + +def _pad_rows(weight: torch.Tensor) -> torch.Tensor: + n = weight.shape[0] + tiles = (n + 15) // 16 + if tiles * 16 == n: + return weight + return torch.cat([weight, weight.new_zeros(tiles * 16 - n, weight.shape[1])]) + + +def _fragment_view(w: torch.Tensor) -> torch.Tensor: + """[tiles * 16, K] -> [tiles, K / 16, 32 lanes, 8 values] in mma A-fragment order (a permutation of the values).""" + n, k = w.shape + # rows: (tile, rh, g); cols: (ks, ch, j, e) -> (tile, ks, g, j, ch, rh, e) + return ( + w.view(n // 16, 2, 8, k // 16, 2, 4, 2) + .permute(0, 3, 2, 5, 4, 1, 6) + .reshape(n // 16, k // 16, 32, 8) + ) + + +def _exponents(weight: torch.Tensor) -> torch.Tensor: + return (weight.contiguous().view(torch.int16).to(torch.int32) >> 7) & 0xFF + + +def _window_base(hist: torch.Tensor) -> int: + """The base of the 15-exponent window holding the most weights (the lowest such base).""" + window = torch.stack([hist[b : b + WINDOW].sum() for b in range(0, MAX_BASE + 1)]) + return int(window.argmax()) + + +def encode(weight: torch.Tensor, base: Optional[int] = None): + """Lossless 12-bit coding of a BF16 [N, K] weight (see the module docstring); ``base`` defaults to this weight's + own window. + + Returns (packed uint8 [tiles * K / 64 * 1536], base, exc_off int32 [tiles + 1], exc_pos int32 [E] (row in tile + << 16 | k), exc_w fp32 [E, 2] ({w, 0})). + """ + w = _pad_rows(weight) + n, k = w.shape + tiles, ks = n // 16, k // 16 + assert ks % 4 == 0, "K must be a multiple of 64" + bits = w.contiguous().view(torch.int16).to(torch.int32) & 0xFFFF + expo = (bits >> 7) & 0xFF + if base is None: + base = _window_base(torch.bincount(expo.flatten(), minlength=256)) + code = expo - base + exc = (code < 0) | (code >= WINDOW) + code = torch.where(exc, torch.full_like(code, WINDOW), code) + sm = ((bits >> 8) & 0x80) | (bits & 0x7F) + sm = torch.where(exc, torch.zeros_like(sm), sm) + # Exceptions: (tile, row in tile, k) in tile-major order; code 15 decodes to +0. + rows, cols = exc.nonzero(as_tuple=True) + order = torch.argsort((rows // 16).long() * (16 * k) + (rows % 16).long() * k + cols.long()) + rows, cols = rows[order], cols[order] + exc_w = torch.stack( + [ + w[rows, cols].float(), + torch.zeros((rows.numel(),), device=w.device), + ], + dim=1, + ) + exc_pos = (((rows % 16) << 16) | cols).to(torch.int32) + exc_off = torch.zeros(tiles + 1, dtype=torch.int32, device=w.device) + exc_off[1:] = torch.cumsum(torch.bincount((rows // 16).long(), minlength=tiles), 0).to( + torch.int32 + ) + # Fragment order, then per (tile, group of 4 k16 steps, lane): 32 sign/mantissa bytes and 4 code words. + smf = _fragment_view(sm.to(torch.uint8).view(n, k)).view(tiles, ks // 4, 4, 32, 8) + cf = ( + _fragment_view(code.to(torch.uint8).view(n, k)) + .view(tiles, ks // 4, 4, 32, 8) + .to(torch.int32) + ) + # Code word of a k16 step: value 2p -> bits 4p..4p+3, value 2p+1 -> bits 16+4p.. + cw = torch.zeros(tiles, ks // 4, 4, 32, dtype=torch.int64, device=w.device) + for p in range(4): + cw |= (cf[..., 2 * p].long() << (4 * p)) | (cf[..., 2 * p + 1].long() << (16 + 4 * p)) + cw = cw.to(torch.int32) # wraps for bit 31; the kernel reads it as uint32 + # Word 0 of a lane = the bytes of steps 0 and 1; word 1 = steps 2 and 3; word 2 = the code words of steps 0..3. + seg01 = ( + smf.permute(0, 1, 3, 2, 4).reshape(tiles, ks // 4, 32, 2, 16).permute(0, 1, 3, 2, 4) + ) # [.., 2, 32, 16] + seg2 = cw.permute(0, 1, 3, 2).contiguous().view(torch.uint8).view(tiles, ks // 4, 1, 32, 16) + packed = torch.cat([seg01.contiguous(), seg2], dim=2) # [tiles, ks / 4, 3, 32, 16] + return packed.contiguous().view(-1), base, exc_off, exc_pos, exc_w.contiguous() + + +def encode_chunked(weight: torch.Tensor, rows_per_chunk: int = ROWS_PER_CHUNK): + """``encode`` over row blocks (bounded temporaries), with the exponent window chosen on the whole weight.""" + n, _ = weight.shape + assert rows_per_chunk % 16 == 0 + hist = torch.zeros(256, dtype=torch.int64, device=weight.device) + for r in range(0, n, rows_per_chunk): + hist += torch.bincount(_exponents(weight[r : r + rows_per_chunk]).flatten(), minlength=256) + base = _window_base(hist) + packed, offs, poss, ws = [], [], [], [] + n_exc = 0 + for r in range(0, n, rows_per_chunk): + p, _, off, pos, ew = encode(weight[r : r + rows_per_chunk], base=base) + packed.append(p) + offs.append(off[:-1] + n_exc) + poss.append(pos) + ws.append(ew) + n_exc += pos.numel() + exc_off = torch.cat(offs + [torch.tensor([n_exc], dtype=torch.int32, device=weight.device)]) + return torch.cat(packed), base, exc_off, torch.cat(poss), torch.cat(ws) + + +def decode_reference(packed: torch.Tensor, base: int, n: int, k: int) -> torch.Tensor: + """BF16 [N, K] from the coded stream with the in-kernel rule (exceptions, code 15, decode to +0).""" + tiles, ks = (n + 15) // 16, k // 16 + p = packed.view(tiles, ks // 4, 3, 32, 16) + sm = ( + p[:, :, :2] + .permute(0, 1, 3, 2, 4) + .reshape(tiles, ks // 4, 32, 4, 8) + .permute(0, 1, 3, 2, 4) + .to(torch.int32) + ) + cw = ( + p[:, :, 2] + .contiguous() + .view(torch.int32) + .view(tiles, ks // 4, 32, 4) + .permute(0, 1, 3, 2) + .long() + & 0xFFFFFFFF + ) + codes = torch.stack([(cw >> (4 * (i // 2) + 16 * (i % 2))) & 0xF for i in range(8)], dim=-1).to( + torch.int32 + ) + bits = ((sm & 0x80) << 8) | ((codes + base) << 7) | (sm & 0x7F) + bits = torch.where(codes == WINDOW, torch.zeros_like(bits), bits) + frag = bits.reshape(tiles, ks, 32, 8) + # Inverse of _fragment_view: (tile, ks, g, j, ch, rh, e) -> (tile, rh, g, ks, ch, j, e). + w = frag.view(tiles, ks, 8, 4, 2, 2, 2).permute(0, 5, 2, 1, 4, 3, 6).reshape(tiles * 16, k) + return w.to(torch.int16).view(torch.bfloat16)[:n] + + +def decode( + packed: torch.Tensor, + base: int, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + n: int, + k: int, + row0: int = 0, + row1: Optional[int] = None, +) -> torch.Tensor: + """BF16 rows [row0, row1) of the coded weight [N, K] with the exceptions applied (what the kernel multiplies by). + ``row0`` must be a multiple of 16.""" + row1 = n if row1 is None else row1 + assert row0 % 16 == 0 + t0, t1 = row0 // 16, (row1 + 15) // 16 + unit_bytes = (k // 64) * 3 * 32 * 16 + dec = decode_reference(packed[t0 * unit_bytes : t1 * unit_bytes], base, row1 - row0, k).clone() + counts = (exc_off[1:] - exc_off[:-1]).long() + tile_of = torch.repeat_interleave(torch.arange(counts.numel(), device=packed.device), counts) + rows = tile_of * 16 + (exc_pos >> 16).long() + cols = (exc_pos & 0xFFFF).long() + sel = (rows >= row0) & (rows < row1) + dec[rows[sel] - row0, cols[sel]] = exc_w.view(-1, 2)[sel, 0].to(torch.bfloat16) + return dec + + +def round_trip_ok( + packed: torch.Tensor, + base: int, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + weight: torch.Tensor, + rows_per_chunk: int = ROWS_PER_CHUNK, +) -> bool: + """Whether the coded weight (code values + exception list) decodes to ``weight`` bit for bit, with its exponent + window below the BF16 infinity exponent.""" + n, k = weight.shape + if not 0 <= base <= MAX_BASE or k % 64: + return False + if ( + packed.numel() != ((n + 15) // 16) * 16 * k * 3 // 2 + or exc_off.numel() != (n + 15) // 16 + 1 + ): + return False + for r0 in range(0, n, rows_per_chunk): + r1 = min(n, r0 + rows_per_chunk) + dec = decode(packed, base, exc_off, exc_pos, exc_w, n, k, r0, r1) + if not torch.equal(dec.view(torch.int16), weight[r0:r1].contiguous().view(torch.int16)): + return False + return True + + +class M3TargetHead(OpWrapper): + """``out[:M, :n]`` (BF16) = x [M, K] (BF16, M <= 8) times the coded BF16 shard W [n, K] transposed: each logit + the fp32 sum of the exact BF16 products, rounded once to BF16 (the in-tree BF16 head GEMM's arithmetic, in + another summation order). + + ``w`` / ``exp_base`` / ``exc_off`` / ``exc_pos`` / ``exc_w`` are ``encode_chunked``'s outputs for W; ``part`` + (fp32 [grid * 16 * 256]) and ``counter`` (int32 [ceil(n / 16)], zero between calls, re-armed by the kernel) are + the split-tile scratch; ``grid`` is the number of 16-warp CTAs (one per SM). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The verify of a fused-verify step at most 8 rows wide: 1 + 3 drafts for one or two requests (4 / 8 rows), and + # every other row count the kernel's one n8 MMA tile serves; one cell with an exception in many tiles, one with + # a shard whose rows are not a multiple of 16. + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"{m} verify row(s) of the TP4 target head shard", + spec=dict(m=m, n=TARGET_SHARD, k=HIDDEN), + ) + for m in range(1, MAX_ROWS + 1) + ), + Cell( + why="4 rows with outliers and exact zeros in many tiles (the exception lists)", + spec=dict(m=4, n=TARGET_SHARD, k=HIDDEN, exceptions=True), + ), + Cell( + why="2 rows over a shard of 4000 rows (250 tiles), K 1024", + spec=dict(m=2, n=4000, k=1024, exceptions=True), + ), + ) + + note = """ + The weight the kernel multiplies by is the checkpoint's BF16 weight bit for bit (round_trip_ok); an exception + adds nothing to the MMA sums and its exact product x * w is added at its tile's flush. A logit therefore differs + from the in-tree head's only by the fp32 summation order before its single BF16 rounding (ulp-level); + integer-valued data, whose sums are exact in any order, is equal bit for bit. Split tiles are summed in warp + order: the result is deterministic. + """ + + def __call__( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_target_head( + w, x, out, part, counter, exc_off, exc_pos, exc_w, exp_base, n, grid, pdl + ) + return out + + def reference( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> torch.Tensor: + """The BF16 logits [M, n] of x against the decoded weight: the fp32 product, rounded once to BF16. + + ``out``, ``part``, ``counter``, ``grid`` and ``pdl`` do not change the result.""" + weight = decode(w, exp_base, exc_off, exc_pos, exc_w, n, x.shape[1]) + with true_fp32_matmul(): + logits = x.float() @ weight.float().t() + return logits.to(torch.bfloat16) + + def is_valid( + self, + w: torch.Tensor, + x: torch.Tensor, + out: torch.Tensor, + part: torch.Tensor, + counter: torch.Tensor, + exc_off: torch.Tensor, + exc_pos: torch.Tensor, + exc_w: torch.Tensor, + exp_base: int, + n: int, + grid: int, + pdl: bool = True, + ) -> None: + # A split tile's last finisher is the CTA whose arrival makes the count reach its owners: a stale count + # stores a partial sum. + assert not bool(counter[: (n + 15) // 16].any()), "counter must be zero between calls" + # The exceptions are read per tile in list order with no bound against the coded stream. + assert int(exc_off[-1].item()) == exc_pos.numel(), "exc_off must end at the exception count" + assert bool((exc_off[1:] >= exc_off[:-1]).all()), "exc_off must be non-decreasing" + assert out.data_ptr() != x.data_ptr(), "out must not alias x" + + +m3_target_head = M3TargetHead() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gqamix_sparse_decode.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gqamix_sparse_decode.py new file mode 100644 index 000000000000..b415e7be538c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/gqamix_sparse_decode.py @@ -0,0 +1,1337 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# SPDX-License-Identifier: Apache-2.0 +# Vendored from vLLM (Apache-2.0): +# https://github.com/vllm-project/vllm/blob/6f91edf96d3f3272945809c04702380053bff4de/vllm/models/minimax_m3/common/ops/sparse_attn.py +"""Triton block-sparse GQA decode attention for MiniMax-M3. + +Flash-decoding over the blocks the indexer selected: one CTA per (query token, +top-k chunk, KV head) accumulates a partial output plus its log-sum-exp, and a +second kernel merges the chunks by LSE weight. That beats running the +context-schedule FMHA kernel at decode, where a single query token leaves most +of a 128-row Q tile idle. + +GB300/SM103 tuning targets GQA group 16, top-k 16 and four-token Eagle3 +verification. Split-K, query pairing and launch tables require four KV heads; +the linear-softmax policy also covers one and two KV heads. Other hardware +and shapes retain the default policies, including supported SM100 NVFP4. + +Vendored from the vLLM source linked in the file header (v0.26.1rc0-77-g6f91edf96). +Differences from upstream: + +* K and V are separate HND paged views ([num_pages, num_kv_heads, page_size, + head_dim]) with independent strides rather than one cache fused along the + last dim, because that is how the M3 KV pool is laid out here. +* Only the scalar KV-scale mode is kept; M3 stores unscaled E4M3 K/V. +* A packed NVFP4 cache is read in place, dequantized in-register from its E2M1 + nibbles and E4M3 block scales. +* Partial-output and LSE scratch come from the persistent buffer arena so their + addresses survive CUDA graph replay. +* A single top-k chunk writes the final output directly, avoiding scratch and + the otherwise redundant LSE merge. +* The merge kernel zeroes rows whose chunks are all empty instead of letting + them produce NaN, since CUDA-graph padding rows flow on into the rest of the + network here. +""" + +from __future__ import annotations + +import os +from typing import Optional + +import torch +import triton +import triton.language as tl + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + check_decode_span_shape, +) +from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers + +# One sparse block is exactly one KV page. +SPARSE_BLOCK_SIZE = 128 + +_FP8_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2) + +_TL_DTYPES = { + torch.bfloat16: tl.bfloat16, + torch.float16: tl.float16, + torch.float32: tl.float32, +} + +# Total CTAs the split-K partitioning aims for before it stops splitting. +_TARGET_GRID = 256 + +# NVFP4 quantizes in groups of 16 cache elements along head_dim, one E4M3 scale +# byte each; see the fp4_quantize call in msa_scatter.fused_write_layer_caches_nvfp4. +NVFP4_SF_VEC_SIZE = 16 + + +def _pdl_enabled() -> bool: + return os.environ.get("TRTLLM_ENABLE_PDL", "1") == "1" + + +@triton.jit +def _e2m1x2_to_f16x2(packed): + """Convert one packed E2M1 byte into its low/high FP16 values.""" + return tl.inline_asm_elementwise( + asm="""{ + .reg .b8 src; + .reg .b32 pair; + mov.b32 {src, _, _, _}, $2; + cvt.rn.f16x2.e2m1x2 pair, src; + mov.b32 {$0, $1}, pair; + }""", + constraints="=h,=h,r", + args=(packed,), + dtype=(tl.float16, tl.float16), + is_pure=True, + pack=1, + ) + + +@triton.jit +def _dequant_nvfp4_rows( + packed, # [BLOCK_N, D // 2] uint8, two E2M1 nibbles per byte + scale_bytes, # [BLOCK_N, SCALE_COLS] uint8, one E4M3 scale per 16 elements + BLOCK_N: tl.constexpr, + D: tl.constexpr, + SCALE_COLS: tl.constexpr, + SF_VEC: tl.constexpr, +): + """Dequantize one packed NVFP4 [BLOCK_N, D] tile to fp16. + + The caller gathers the block scales in whichever layout its role stores + them, so this handles only the value math and the scale broadcast. The + per-tensor scale is applied by the caller. + """ + # Blackwell converts both E2M1 nibbles with one instruction. Joining the + # low/high results along a new innermost axis and flattening it rebuilds + # the original element order without scalar exponent/sign reconstruction. + value_lo, value_hi = _e2m1x2_to_f16x2(packed) + values = tl.reshape(tl.join(value_lo, value_hi), (BLOCK_N, D)) + + scales = scale_bytes.to(tl.float8e4nv, bitcast=True).to(tl.float16) + scales = tl.reshape( + tl.broadcast_to(scales[:, :, None], (BLOCK_N, SCALE_COLS, SF_VEC)), (BLOCK_N, D) + ) + return values * scales + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max(16, triton.next_power_of_2(args["gqa_group_size"])), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query token, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + pid_b = pid_bc % total_q + pid_c = pid_bc // total_q + req_id = pid_b // decode_query_len + q_offset = pid_b - req_id * decode_query_len + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + query_pos = seq_len - decode_query_len + q_offset + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + + # Bound the walk by the token's own valid block count rather than a + # sentinel: the selector emits ascending ids and only pads with -1 past + # min(topk, cdiv(kv_len, blk)) entries, so those are never dereferenced. + num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk = tl.minimum(max_topk, num_blocks) + chunk_end = tl.minimum(chunk_start + chunk_size_topk, real_topk) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so the result + # matches a caller that widened q itself before the call. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + cur_idx_ptr = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + for _ in tl.range(chunk_start, chunk_end): + blk = tl.load(cur_idx_ptr) + cur_idx_ptr = cur_idx_ptr + stride_tk + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len + if KV_NVFP4: + # K block scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + qk = tl.where(pos_mask[None, :], 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) + p = tl.exp2(qk - m_ij[:, None]) + l_ij = tl.sum(p, axis=1) + if USE_LINEAR_SOFTMAX: + alpha = tl.exp2(m_i - m_ij) + acc_o = acc_o * alpha[:, None] + else: + acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + if KV_NVFP4: + # V block scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = l_i * alpha + l_ij + else: + lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + o_base = o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h + tl.store( + o_base + off_h[:, None] * stride_o_h + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + lse_base = lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h + tl.store(lse_base + off_h * stride_l_h, lse_out, mask=h_mask) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics( + { + "BLOCK_SIZE_H": lambda args: max( + 16, + triton.next_power_of_2(args["gqa_group_size"] * args["QUERY_GROUP_SIZE"]), + ), + "BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"]), + } +) +@triton.jit( + do_not_specialize=["decode_query_len", "total_q"], + do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"], +) +def _gqa_sparse_decode_pair_kernel( + q_ptr, # [total_q, num_heads, head_dim] + k_ptr, # paged K: [num_pages, num_kv_heads, page_size, head_dim (or /2 packed)] + v_ptr, # paged V: same layout as K + kv_scale_ptr, # scalar dequant scale, or a dummy when USE_SCALE is False + k_sc_ptr, # paged K block scales: [num_pages, num_kv_heads, page_size, D/16] + v_sc_ptr, # paged V block scales: same shape, 4x4 token-quad order + k_gs_ptr, # NVFP4 per-tensor K dequant scale, one fp32 + v_gs_ptr, # NVFP4 per-tensor V dequant scale, one fp32 + t_ptr, # topk_idx: [num_kv_heads, total_q, topk] + o_ptr, # partial out, or final [total_q, num_heads, head_dim] when DIRECT_OUTPUT + lse_ptr, # partial lse (log2), unused when DIRECT_OUTPUT + block_table_ptr, # [num_reqs, max_blocks] + seq_lens, # [num_reqs] + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + stride_qn, + stride_qh, + stride_qd, + stride_k_blk, + stride_k_h, + stride_k_pos, + stride_k_d, + stride_v_blk, + stride_v_h, + stride_v_pos, + stride_v_d, + stride_ksc_blk, + stride_ksc_h, + stride_vsc_blk, + stride_vsc_h, + stride_th, + stride_tn, + stride_tk, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_bt_b, + BLOCK_SIZE_K: tl.constexpr, # == SPARSE_BLOCK_SIZE (128) + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_H: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_SCALE: tl.constexpr, # apply the scalar dequant scale to an fp8 cache + WIDEN_Q: tl.constexpr, # q arrives FP8 and is widened in-register + COMPUTE_DTYPE: tl.constexpr, # dtype q is widened to; drives the QK/PV math + USE_PDL: tl.constexpr, + KV_NVFP4: tl.constexpr, # cache holds packed E2M1 plus E4M3 block scales + USE_LINEAR_SOFTMAX: tl.constexpr, # measured NVFP4 online-softmax recurrence + QUERY_GROUP_SIZE: tl.constexpr, # consecutive Eagle queries sharing one block set + MAX_TOPK: tl.constexpr, + SCALE_COLS: tl.constexpr, # head_dim // NVFP4_SF_VEC_SIZE, NVFP4 only + DIRECT_OUTPUT: tl.constexpr, # one chunk needs neither FP32 scratch nor a merge +): + sm_scale_log2e = sm_scale * 1.4426950409 + if KV_NVFP4: + # Both per-tensor dequant scales factor out of the inner loop: QK is + # linear in K, and V's scale commutes with the softmax weights. Reading + # them from device memory keeps the launch free of a host sync, so + # CUDA-graph capture stays unconstrained. + sm_scale_log2e = sm_scale_log2e * tl.load(k_gs_ptr) + v_global_scale = tl.load(v_gs_ptr) + # Split-K over the topk dimension: pid(0) folds (query group, chunk). + pid_bc, pid_kh = tl.program_id(0), tl.program_id(1) + num_query_groups = total_q // QUERY_GROUP_SIZE + pid_g = pid_bc % num_query_groups + pid_c = pid_bc // num_query_groups + groups_per_req = decode_query_len // QUERY_GROUP_SIZE + req_id = pid_g // groups_per_req + q_group_offset = (pid_g - req_id * groups_per_req) * QUERY_GROUP_SIZE + pid_b = req_id * decode_query_len + q_group_offset + pid_h = pid_kh * gqa_group_size + chunk_size_topk = (max_topk + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_start = pid_c * chunk_size_topk + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + seq_len = tl.load(seq_lens + req_id) + # Query rows are flattened as [query-in-group, GQA head]. + off_h = tl.arange(0, BLOCK_SIZE_H) + q_lane = off_h // gqa_group_size + h_lane = off_h - q_lane * gqa_group_size + h_mask = off_h < gqa_group_size * QUERY_GROUP_SIZE + q_token = pid_b + q_lane + query_pos = seq_len - decode_query_len + q_group_offset + q_lane + # Full-CG padding uses zero-length request rows. Clamp to an empty + # attention range instead of letting padded rows produce negative lengths. + kv_len = tl.maximum(query_pos + 1, 0) + kv_len_max = tl.max(tl.where(h_mask, kv_len, 0), axis=0) + + # Each query can have a different valid-block count at a page boundary. + # The selector puts those valid entries first and pads only the tail. + kv_len_0 = tl.maximum(seq_len - decode_query_len + q_group_offset + 1, 0) + num_blocks_0 = (kv_len_0 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_0 = tl.minimum(max_topk, num_blocks_0) + chunk_end_0 = tl.minimum(chunk_start + chunk_size_topk, real_topk_0) + if QUERY_GROUP_SIZE == 2: + kv_len_1 = tl.maximum(seq_len - decode_query_len + q_group_offset + 2, 0) + num_blocks_1 = (kv_len_1 + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K + real_topk_1 = tl.minimum(max_topk, num_blocks_1) + chunk_end_1 = tl.minimum(chunk_start + chunk_size_topk, real_topk_1) + + off_n = tl.arange(0, BLOCK_SIZE_K) + off_d = tl.arange(0, BLOCK_SIZE_D) + if KV_NVFP4: + # Packed byte column and block-scale column. The NVFP4 path runs only at + # head_dim == BLOCK_SIZE_D (128), so neither needs a tail mask. + off_dp = tl.arange(0, BLOCK_SIZE_D // 2) + off_s = tl.arange(0, SCALE_COLS) + d_mask = off_d < head_dim + hd_mask = h_mask[:, None] & d_mask[None, :] + bt_row = block_table_ptr + req_id * stride_bt_b + + m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + if USE_LINEAR_SOFTMAX: + # Keep the linear softmax denominator and reuse the producer's alpha. + # This removes one exp2 plus one log2 from every visited KV page; only + # the final LSE written for split-K merging still needs a logarithm. + l_i = tl.zeros((BLOCK_SIZE_H,), dtype=tl.float32) + else: + lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) + acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) + q = tl.load( + q_ptr + + q_token[:, None] * stride_qn + + (pid_h + h_lane[:, None]) * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, + ) + # Widen before anything reads q.dtype: K, V and p are all cast to it below, + # so leaving q narrow would silently run the whole attention in FP8 and + # quantize the softmax probabilities. E4M3 -> BF16 is exact, so this + # reproduces the values the caller's standalone widening kernel produced. + if WIDEN_Q: + q = q.to(COMPUTE_DTYPE) + kv_scale = tl.load(kv_scale_ptr) if USE_SCALE else 1.0 + + idx_ptr_0 = t_ptr + pid_kh * stride_th + pid_b * stride_tn + chunk_start * stride_tk + if QUERY_GROUP_SIZE == 2: + idx_ptr_1 = idx_ptr_0 + stride_tn + idx_0 = chunk_start + idx_1 = chunk_start + count_0 = chunk_end_0 - chunk_start + count_1 = chunk_end_1 - chunk_start + chunk_capacity: tl.constexpr = (MAX_TOPK + NUM_TOPK_CHUNKS - 1) // NUM_TOPK_CHUNKS + chunk_offsets = tl.arange(0, chunk_capacity) + sentinel = 2147483647 + blocks_0 = tl.load( + idx_ptr_0 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_0, + other=sentinel, + ) + blocks_1 = tl.load( + idx_ptr_1 + chunk_offsets * stride_tk, + mask=chunk_offsets < count_1, + other=sentinel, + ) + chunk_equal = tl.sum(blocks_0 != blocks_1, axis=0) == 0 + # Equal chunks take the measured shared-page loop. A divergent chunk + # walks the sorted union, bounded by the sum of both row lengths. + num_candidates = tl.where(chunk_equal, count_0, count_0 + count_1) + else: + num_candidates = chunk_end_0 - chunk_start + + for _ in tl.range(0, num_candidates): + if QUERY_GROUP_SIZE == 2: + if chunk_equal: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + else: + valid_0 = idx_0 < chunk_end_0 + valid_1 = idx_1 < chunk_end_1 + blk_0 = tl.load(idx_ptr_0, mask=valid_0, other=sentinel) + blk_1 = tl.load(idx_ptr_1, mask=valid_1, other=sentinel) + use_0 = valid_0 & (blk_0 <= blk_1) + use_1 = valid_1 & (blk_1 <= blk_0) + has_block = use_0 | use_1 + blk = tl.minimum(blk_0, blk_1) + idx_0 += use_0.to(tl.int32) + idx_1 += use_1.to(tl.int32) + idx_ptr_0 += use_0.to(tl.int64) * stride_tk + idx_ptr_1 += use_1.to(tl.int64) * stride_tk + query_active = h_mask & (((q_lane == 0) & use_0) | ((q_lane == 1) & use_1)) + else: + has_block = True + blk = tl.load(idx_ptr_0) + idx_ptr_0 += stride_tk + query_active = h_mask + + if has_block: + # int64 page offsets: a large cache overflows int32 well before the + # per-page block offsets do. + page = tl.load(bt_row + blk).to(tl.int64) + pos_mask = blk * BLOCK_SIZE_K + off_n < kv_len_max + q_pos_mask = blk * BLOCK_SIZE_K + off_n[None, :] < kv_len[:, None] + if KV_NVFP4: + # K scales are token-major linear within the page-head region; + # see _fused_nvfp4_paged_scatter_kernel in msa_scatter. + k_deq = _dequant_nvfp4_rows( + tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[:, None] * stride_k_pos + + off_dp[None, :] * stride_k_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + k_sc_ptr + + page * stride_ksc_blk + + pid_kh * stride_ksc_h + + off_n[:, None] * SCALE_COLS + + off_s[None, :], + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ) + k = tl.trans(k_deq).to(q.dtype) + else: + k = tl.load( + k_ptr + + page * stride_k_blk + + pid_kh * stride_k_h + + off_n[None, :] * stride_k_pos + + off_d[:, None] * stride_k_d, + mask=d_mask[:, None] & pos_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + k = (k * kv_scale).to(q.dtype) + active_pos = query_active[:, None] & q_pos_mask + qk = tl.where(active_pos, 0.0, float("-inf")) + qk += tl.dot(q, k) * sm_scale_log2e + m_ij = tl.where(query_active, tl.maximum(m_i, tl.max(qk, axis=1)), m_i) + p = tl.where(active_pos, tl.exp2(qk - m_ij[:, None]), 0.0) + l_ij = tl.sum(p, axis=1) + alpha = tl.where(query_active, tl.exp2(m_i - m_ij), 1.0) + acc_o = acc_o * alpha[:, None] + if KV_NVFP4: + # V scales use vLLM's 4x4 token-quad order rather than K's + # linear one: [token // 4, scale_col, token % 4]. + v = _dequant_nvfp4_rows( + tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_dp[None, :] * stride_v_d, + mask=pos_mask[:, None], + other=0, + ), + tl.load( + v_sc_ptr + + page * stride_vsc_blk + + pid_kh * stride_vsc_h + + (off_n[:, None] // 4) * (4 * SCALE_COLS) + + 4 * off_s[None, :] + + (off_n[:, None] % 4), + mask=pos_mask[:, None], + other=0, + ), + BLOCK_N=BLOCK_SIZE_K, + D=BLOCK_SIZE_D, + SCALE_COLS=SCALE_COLS, + SF_VEC=BLOCK_SIZE_D // SCALE_COLS, + ).to(q.dtype) + else: + v = tl.load( + v_ptr + + page * stride_v_blk + + pid_kh * stride_v_h + + off_n[:, None] * stride_v_pos + + off_d[None, :] * stride_v_d, + mask=pos_mask[:, None] & d_mask[None, :], + other=0.0, + ).to(q.dtype) + if USE_SCALE: + v = (v * kv_scale).to(q.dtype) + acc_o += tl.dot(p.to(v.dtype), v) + if USE_LINEAR_SOFTMAX: + l_i = tl.where(query_active, l_i * alpha + l_ij, l_i) + else: + lse_ij = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_i = tl.where(query_active, lse_ij, lse_i) + m_i = m_ij + + # An empty chunk of an active row must store zero, or the merge hits 0 * NaN. + if USE_LINEAR_SOFTMAX: + scale = tl.where(l_i > 0.0, 1.0 / l_i, 0.0) + lse_out = tl.where(l_i > 0.0, m_i + tl.log2(l_i), float("-inf")) + else: + scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), 0.0) + lse_out = lse_i + if KV_NVFP4: + # Applying V's scale per partial is exact: the merge is a weighted + # average whose weights sum to one, so a constant survives it. + scale = scale * v_global_scale + acc_o = acc_o * scale[:, None] + tl.store( + o_ptr + + pid_c * stride_o_c + + q_token[:, None] * stride_o_b + + (pid_h + h_lane[:, None]) * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, + ) + if not DIRECT_OUTPUT: + tl.store( + lse_ptr + pid_c * stride_l_c + q_token * stride_l_b + (pid_h + h_lane) * stride_l_h, + lse_out, + mask=h_mask, + ) + + # After the stores, never before: this releases either the merge grid or, + # for direct output, the next PDL-enabled consumer of the final tensor. + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +@triton.heuristics({"BLOCK_SIZE_D": lambda args: triton.next_power_of_2(args["head_dim"])}) +@triton.jit +def _merge_topk_attn_out_kernel( + o_ptr, # partials: [NUM_TOPK_CHUNKS, total_q, num_heads, head_dim] + lse_ptr, # partials (log2): [NUM_TOPK_CHUNKS, total_q, num_heads] + out_ptr, # merged out: [total_q, num_heads, head_dim] + head_dim, + stride_o_c, + stride_o_b, + stride_o_h, + stride_o_d, + stride_l_c, + stride_l_b, + stride_l_h, + stride_out_n, + stride_out_h, + stride_out_d, + NUM_TOPK_CHUNKS: tl.constexpr, + BLOCK_SIZE_D: tl.constexpr, + USE_PDL: tl.constexpr, +): + pid_b, pid_h = tl.program_id(0), tl.program_id(1) + + if USE_PDL: + tl.extra.cuda.gdc_wait() + + off_c = tl.arange(0, NUM_TOPK_CHUNKS) + off_d = tl.arange(0, BLOCK_SIZE_D) + o = tl.load( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d, + mask=off_d[None, :] < head_dim, + other=0.0, + ).to(tl.float32) + # Empty chunks contribute -inf, hence weight 0. + lse = tl.load(lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c) + lse_max = tl.max(lse, axis=0) + # A row whose every chunk is empty (a CUDA-graph padding row) would give + # -inf - -inf here. Zero it instead: the padded output is discarded, but a + # NaN would survive into the residual stream and the all-reduce. + lse_max = tl.where(lse_max == float("-inf"), 0.0, lse_max) + weights = tl.exp2(lse - lse_max) + denom = tl.sum(weights, axis=0) + weights = weights / tl.where(denom > 0, denom, 1.0) + o_merged = tl.sum(o * weights[:, None], axis=0) + out_ptrs = out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + + if USE_PDL: + tl.extra.cuda.gdc_launch_dependents() + + +def resolve_num_topk_chunks(total_q: int, num_kv_heads: int, max_topk: int) -> int: + """Split-K factor over the top-k blocks, as a power of two. + + Depends only on shapes that are fixed for a captured batch size, so the + launch geometry is frozen inside a CUDA graph. + """ + target = max(1, min(max_topk, _TARGET_GRID // max(1, total_q * num_kv_heads))) + return 1 << (target.bit_length() - 1) + + +# NVFP4 instructions are supported on the datacenter Blackwell and Rubin targets. +_SM100F_CAPABILITIES = ((10, 0), (10, 3), (10, 7)) + + +# Devices already found to support the NVFP4 decode kernels, by capability. +_NVFP4_DEVICE_CAPABILITIES: dict[torch.device, tuple[int, int]] = {} + + +def _nvfp4_device_capability(device: torch.device) -> tuple[int, int]: + """Return the device's capability, checking NVFP4 support once per device.""" + capability = _NVFP4_DEVICE_CAPABILITIES.get(device) + if capability is None: + capability = torch.cuda.get_device_capability(device) + if capability not in _SM100F_CAPABILITIES: + raise NotImplementedError( + "MiniMax-M3 NVFP4 sparse decode requires SM100/SM103/SM107; " + f"got SM{capability[0] * 10 + capability[1]}." + ) + _NVFP4_DEVICE_CAPABILITIES[device] = capability + return capability + + +# Shape-specific tuning was measured on GB300 only. SM100 uses the defaults. +_NVFP4_TUNED_CAPABILITIES = ((10, 3),) + + +def _sm100f_nvfp4_num_topk_chunks( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> Optional[int]: + """Select GB300-measured split-K factors for M3's SM100-family decode shape.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return None + local_batch = total_q // decode_query_len + return { + 2: 8, + 3: 8, + 4: 8, + 5: 16, + 6: 4, + 7: 4, + 8: 2, + 9: 2, + 10: 8, + 11: 4, + 12: 2, + 13: 2, + 14: 2, + }.get(local_batch) + + +def _sm100f_nvfp4_use_linear_softmax( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> bool: + """Use the GB300-measured lower-register recurrence on validated M3 shapes.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads not in (1, 2, 4) + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return False + local_batch = total_q // decode_query_len + # The TP4/DEP32 single-request graph is 8.9% faster with the original + # log-domain recurrence; every other measured DEP8/16/32 bucket is neutral + # or faster with the linear denominator. + return not (num_kv_heads == 1 and local_batch == 1) + + +def _sm100f_nvfp4_query_group_size( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> int: + """Group Eagle queries using the GB300-measured shared-page policy.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + or total_q % decode_query_len != 0 + ): + return 1 + # The sorted-union kernel is 6.5%, 1.6%, and 24.2% faster at local batches + # 11, 12, and 14 with split-K 8. Other measured buckets are neutral or + # slower and retain the single-query producer. + return 2 if total_q // decode_query_len in (11, 12, 14) else 1 + + +def _sm100f_nvfp4_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select native-NVFP4 launch geometry for the SM100 family. + + The FP16 QK/PV path is register-sensitive. GB300 dense replay shows four + warps/two stages wins through local batch 28, while two warps/one stage + avoids the next residency cliff at local batches 29--32. Leave every + unmeasured architecture, model geometry, and larger batch on Triton's + established defaults. + """ + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability not in _NVFP4_TUNED_CAPABILITIES + or num_kv_heads != 4 + or gqa_group_size != 16 + or max_topk != 16 + or decode_query_len != 4 + ): + return {} + local_batch = total_q // decode_query_len + if local_batch == 1 or local_batch in (2, 3, 5, 6, 7, 10, 11, 12): + return {"num_warps": 4, "num_stages": 1} + if local_batch == 4: + return {"num_warps": 2, "num_stages": 1} + if 8 <= local_batch <= 28: + return {"num_warps": 4, "num_stages": 2} + if 29 <= local_batch <= 32: + return {"num_warps": 2, "num_stages": 1} + return {} + + +def _sm100f_nvfp4_merge_launch_options( + *, + total_q: int, + num_kv_heads: int, + gqa_group_size: int, + max_topk: int, + decode_query_len: int, + capability: Optional[tuple[int, int]] = None, +) -> dict[str, int]: + """Select GB300-measured one-warp merges for M3's SM100-family graphs.""" + if capability is None: + capability = torch.cuda.get_device_capability() + if ( + capability in _NVFP4_TUNED_CAPABILITIES + and num_kv_heads == 4 + and gqa_group_size == 16 + and max_topk == 16 + and decode_query_len == 4 + and total_q % decode_query_len == 0 + and total_q // decode_query_len in (2, 4, 5, 6, 7, 10, 11) + ): + return {"num_warps": 1} + return {} + + +def _check_nvfp4_inputs( + k_paged: torch.Tensor, + v_paged: torch.Tensor, + nvfp4_args: tuple[Optional[torch.Tensor], ...], + *, + head_dim: int, + scale_cols: int, +) -> None: + """Validate the packed-NVFP4 cache against what the kernel assumes. + + The kernel indexes block scales by a flat within-page offset and reshapes a + [., SCALE_COLS] scale tile up to head_dim, so it needs contiguous scale rows + and head_dim exactly BLOCK_SIZE_D. Both hold for MiniMax-M3; check them + rather than let a mismatch read neighboring pages. + """ + k_block_scale, v_block_scale, k_global_scale, v_global_scale = nvfp4_args + if any(arg is None for arg in nvfp4_args): + raise ValueError( + "MiniMax-M3 NVFP4 sparse decode needs all four of k_block_scale, " + "v_block_scale, k_global_scale and v_global_scale." + ) + if head_dim != triton.next_power_of_2(head_dim): + raise ValueError(f"NVFP4 sparse decode requires a power-of-two head_dim; got {head_dim}.") + for name, data in (("k_paged", k_paged), ("v_paged", v_paged)): + if int(data.shape[3]) * 2 != head_dim: + raise ValueError( + f"NVFP4 {name} must pack two elements per byte: expected " + f"head_dim/2 ({head_dim // 2}) columns, got {int(data.shape[3])}." + ) + for name, scale in (("k_block_scale", k_block_scale), ("v_block_scale", v_block_scale)): + if tuple(scale.shape) != tuple(k_paged.shape[:3]) + (scale_cols,): + raise ValueError( + f"NVFP4 {name} must be [num_pages, num_kv_heads, page_size, " + f"{scale_cols}] to match the cache; got {tuple(scale.shape)}." + ) + if scale.stride(3) != 1 or scale.stride(2) != scale_cols: + raise ValueError( + f"NVFP4 {name} needs contiguous per-page scale bytes; got strides " + f"{tuple(scale.stride())}." + ) + for name, gs in (("k_global_scale", k_global_scale), ("v_global_scale", v_global_scale)): + if gs.dtype != torch.float32 or gs.numel() != 1: + raise ValueError( + f"NVFP4 {name} must be a single-element FP32 tensor; got " + f"{gs.numel()} element(s) of {gs.dtype}." + ) + + +@torch.no_grad() +def minimax_m3_sparse_attn_decode( + q: torch.Tensor, # [total_q, num_heads, head_dim] + k_paged: torch.Tensor, # [num_pages, num_kv_heads, page_size, head_dim] + v_paged: torch.Tensor, # same layout as k_paged + topk_idx: torch.Tensor, # [num_kv_heads, total_q, topk] + block_table: torch.Tensor, # [num_reqs, max_blocks] + seq_lens: torch.Tensor, # [num_reqs] int32 + *, + sm_scale: float, + output: torch.Tensor, # [total_q, num_heads, head_dim] + decode_query_len: int, + kv_scale: Optional[torch.Tensor] = None, + k_block_scale: Optional[torch.Tensor] = None, + v_block_scale: Optional[torch.Tensor] = None, + k_global_scale: Optional[torch.Tensor] = None, + v_global_scale: Optional[torch.Tensor] = None, + num_topk_chunks: Optional[int] = None, +) -> None: + """Block-sparse GQA decode attention, written into output in place. + + q may be FP8, as a fused QK-norm/RoPE producer emits it. The kernel widens + it to output.dtype in-register rather than making the caller materialize a + widened copy, which would cost a standalone kernel on every sparse layer. + The attention math is unchanged either way, since E4M3 widens exactly. + + kv_scale is an optional scalar dequantization factor for an FP8 cache; + MiniMax-M3 stores unscaled E4M3, so it is normally None. + + Passing the four NVFP4 scale tensors selects the packed-NVFP4 cache instead: + k_paged/v_paged then hold two E2M1 elements per byte, the block scales are + paged E4M3 bytes in the layouts msa_scatter writes, and the two per-tensor + dequant scales are single-element FP32 device tensors. + + num_topk_chunks overrides the split-K factor and exists for tests: the + merged result must not depend on it. + """ + total_q, num_heads, head_dim = q.shape + num_kv_heads = int(k_paged.shape[1]) + check_decode_span_shape( + "MiniMax-M3 Triton sparse decode", total_q, int(seq_lens.shape[0]), decode_query_len + ) + if int(k_paged.shape[2]) != SPARSE_BLOCK_SIZE: + raise ValueError( + f"MiniMax-M3 sparse decode requires page_size={SPARSE_BLOCK_SIZE}; " + f"got {int(k_paged.shape[2])}." + ) + if num_heads % num_kv_heads: + # The kernel covers heads pid_kh * gqa_group_size + [0, group), so a + # head count that is not a whole number of groups leaves the tail heads + # unwritten and the merge kernel reads uninitialized partials. + raise ValueError( + f"MiniMax-M3 sparse decode requires num_heads ({num_heads}) to be a " + f"multiple of num_kv_heads ({num_kv_heads})." + ) + nvfp4_args = (k_block_scale, v_block_scale, k_global_scale, v_global_scale) + kv_nvfp4 = any(arg is not None for arg in nvfp4_args) + scale_cols = head_dim // NVFP4_SF_VEC_SIZE + # Checked once per device; the shape heuristics below consult it per call. + capability = _nvfp4_device_capability(q.device) if kv_nvfp4 else None + if kv_nvfp4: + _check_nvfp4_inputs(k_paged, v_paged, nvfp4_args, head_dim=head_dim, scale_cols=scale_cols) + # Two E2M1 elements per byte; Triton must see them unsigned so that the + # high-nibble shift does not sign-extend. + k_paged, v_paged = k_paged.view(torch.uint8), v_paged.view(torch.uint8) + k_block_scale, v_block_scale = ( + k_block_scale.view(torch.uint8), + v_block_scale.view(torch.uint8), + ) + max_topk = int(topk_idx.shape[-1]) + gqa_group_size = num_heads // num_kv_heads + use_scale = k_paged.dtype in _FP8_DTYPES and kv_scale is not None + # Triton needs a real pointer even for the unused arguments. + scale_arg = kv_scale if use_scale else output + k_sc_arg, v_sc_arg = (k_block_scale, v_block_scale) if kv_nvfp4 else (output, output) + k_gs_arg, v_gs_arg = (k_global_scale, v_global_scale) if kv_nvfp4 else (output, output) + + widen_q = q.dtype in _FP8_DTYPES + if widen_q and output.dtype not in _TL_DTYPES: + raise ValueError( + f"MiniMax-M3 sparse decode cannot widen FP8 q into {output.dtype}; " + f"supported compute dtypes are {sorted(d.__str__() for d in _TL_DTYPES)}." + ) + # Packed NVFP4 dequantizes to FP16. The production producer supplies FP8 + # q, which widens exactly to FP16, so keep both dot operands there instead + # of converting every K/V tile to BF16. Accumulation remains FP32. + if widen_q: + compute_dtype = tl.float16 if kv_nvfp4 else _TL_DTYPES[output.dtype] + else: + compute_dtype = tl.float32 + + query_group_size = ( + _sm100f_nvfp4_query_group_size( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else 1 + ) + if num_topk_chunks is None: + if query_group_size == 2: + measured_chunks = 8 + else: + measured_chunks = ( + _sm100f_nvfp4_num_topk_chunks( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else None + ) + num_topk_chunks = ( + measured_chunks + if measured_chunks is not None + else resolve_num_topk_chunks(total_q, num_kv_heads, max_topk) + ) + elif num_topk_chunks & (num_topk_chunks - 1): + raise ValueError(f"num_topk_chunks must be a power of two; got {num_topk_chunks}.") + + direct_output = num_topk_chunks == 1 + if direct_output: + # With one chunk, acc_o is already normalized and final. Point the + # producer at the caller's output and compile away the LSE store. + # Keeping a real dummy pointer preserves Triton's pointer contract. + o_arg = output + lse_arg = output + o_strides = (0, output.stride(0), output.stride(1), output.stride(2)) + lse_strides = (0, 0, 0) + else: + # Persistent arena rather than torch.empty, so the partials keep one + # address across CUDA graph replays. FP32 keeps the split-K factor from + # perturbing the merged result. + reserve = torch.cuda.is_current_stream_capturing() + o_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads, head_dim], + torch.float32, + buffer_name="m3_sparse_decode_o_partial", + reserve_buffer=reserve, + ) + lse_arg = get_memory_buffers().get_buffer( + [num_topk_chunks, total_q, num_heads], + torch.float32, + buffer_name="m3_sparse_decode_lse_partial", + reserve_buffer=reserve, + ) + o_strides = tuple(o_arg.stride()) + lse_strides = tuple(lse_arg.stride()) + + use_pdl = _pdl_enabled() + pdl_launch = {"launch_pdl": True} if use_pdl else {} + use_linear_softmax = kv_nvfp4 and _sm100f_nvfp4_use_linear_softmax( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if query_group_size == 2: + launch_options = {"num_warps": 4, "num_stages": 1} + else: + launch_options = ( + _sm100f_nvfp4_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + + # The pair path merges the two sorted selector rows on device. Equal blocks + # share one K/V dequantization; divergent blocks update only their owning + # query, so the optimization does not rely on runtime selection equality. + if total_q % query_group_size or decode_query_len % query_group_size: + raise ValueError( + f"query group {query_group_size} must divide total_q={total_q} and " + f"decode_query_len={decode_query_len}." + ) + + producer_kernel = ( + _gqa_sparse_decode_pair_kernel if query_group_size == 2 else _gqa_sparse_decode_kernel + ) + pair_kernel_options = ( + {"QUERY_GROUP_SIZE": query_group_size, "MAX_TOPK": max_topk} + if query_group_size == 2 + else {} + ) + producer_kernel[ + (total_q // query_group_size) * num_topk_chunks, + num_kv_heads, + ]( + q, + k_paged, + v_paged, + scale_arg, + k_sc_arg, + v_sc_arg, + k_gs_arg, + v_gs_arg, + topk_idx, + o_arg, + lse_arg, + block_table, + seq_lens, + total_q, + gqa_group_size, + head_dim, + max_topk, + sm_scale, + decode_query_len, + q.stride(0), + q.stride(1), + q.stride(2), + k_paged.stride(0), + k_paged.stride(1), + k_paged.stride(2), + k_paged.stride(3), + v_paged.stride(0), + v_paged.stride(1), + v_paged.stride(2), + v_paged.stride(3), + k_sc_arg.stride(0), + k_sc_arg.stride(1), + v_sc_arg.stride(0), + v_sc_arg.stride(1), + topk_idx.stride(0), + topk_idx.stride(1), + topk_idx.stride(2), + *o_strides, + *lse_strides, + block_table.stride(0), + BLOCK_SIZE_K=SPARSE_BLOCK_SIZE, + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_SCALE=use_scale, + WIDEN_Q=widen_q, + COMPUTE_DTYPE=compute_dtype, + USE_PDL=use_pdl, + KV_NVFP4=kv_nvfp4, + USE_LINEAR_SOFTMAX=use_linear_softmax, + SCALE_COLS=scale_cols, + DIRECT_OUTPUT=direct_output, + **pair_kernel_options, + **pdl_launch, + **launch_options, + ) + if not direct_output: + merge_launch_options = ( + _sm100f_nvfp4_merge_launch_options( + total_q=total_q, + num_kv_heads=num_kv_heads, + gqa_group_size=gqa_group_size, + max_topk=max_topk, + decode_query_len=decode_query_len, + capability=capability, + ) + if kv_nvfp4 + else {} + ) + _merge_topk_attn_out_kernel[(total_q, num_heads)]( + o_arg, + lse_arg, + output, + head_dim, + *o_strides, + *lse_strides, + output.stride(0), + output.stride(1), + output.stride(2), + NUM_TOPK_CHUNKS=num_topk_chunks, + USE_PDL=use_pdl, + **pdl_launch, + **merge_launch_options, + ) + + +__all__ = [ + "NVFP4_SF_VEC_SIZE", + "SPARSE_BLOCK_SIZE", + "_sm100f_nvfp4_merge_launch_options", + "_sm100f_nvfp4_num_topk_chunks", + "_sm100f_nvfp4_query_group_size", + "_sm100f_nvfp4_use_linear_softmax", + "minimax_m3_sparse_attn_decode", + "resolve_num_topk_chunks", +] diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/__init__.py new file mode 100644 index 000000000000..77d7b84d9bfe --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Mixture-of-experts entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/m3_moe_engine.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/m3_moe_engine.py new file mode 100644 index 000000000000..32eb8514c2e2 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/m3_moe_engine.py @@ -0,0 +1,343 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3's MoE layer for 1..8 decode tokens in one CuTe DSL launch (``m3_moe``, the sm_107 TP2 build: routed +and shared expert width 1536 per rank): the router, the routed NVFP4 experts and the shared MXFP8 expert, on +caller-owned state (an ``M3MoeState`` and one ``M3MoeLayer`` per MoE layer).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.logger import logger +from tensorrt_llm.quantization.utils.fp4_utils import ( + get_reorder_rows_for_gated_act_gemm_row_indices, + get_shuffle_matrix_a_row_indices, +) + +from ...kernels.m3_moe.op import ( + HIDDEN, + INTER, + SHARED_INTER, + TOP_K, + M3MoeLayer, + M3MoeState, + ar_push_desc, + is_supported, + shared_ctas_for, +) + +__all__ = [ + "M3MoeLayer", + "M3MoeState", + "ar_push_desc", + "is_supported", + "m3_moe_engine", + "moe_reference", + "moe_route", + "mxfp8_dequant", + "mxfp8_quant_dequant", + "shared_ctas_for", + "unswizzle_sf", +] + +# The model's constants the kernels are built with: routed_scaling_factor, the shared expert's SwiGLU-OAI +# (swiglu_alpha, swiglu_limit, the (up + 1) offset). +ROUTED_SCALING = 2.0 +SHARED_ALPHA = 1.702 +SHARED_LIMIT = 7.0 + +_LOG2E = 1.4426950216293334961 +_RCP6 = 0.16666667163372039795 # fp32(1 / 6), the kernels' reciprocal of the E2M1 maximum +_E2M1 = (0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0) +_F32_MAX = 3.40282346638528859812e38 + + +def _f32(x: float) -> torch.Tensor: + return torch.tensor(x, dtype=torch.float32) + + +def _e2m1_decode(packed: torch.Tensor) -> torch.Tensor: + """fp32 values of packed E2M1 bytes [..., K / 2] (low nibble first) -> [..., K].""" + lut = torch.tensor(_E2M1, dtype=torch.float32, device=packed.device) + lo = lut[(packed & 0xF).long()] + hi = lut[(packed >> 4).long()] + return torch.stack((lo, hi), dim=-1).flatten(-2) + + +def _e2m1_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to the nearest E2M1 value, ties to even, saturating at 6 (cvt.rn.satfinite.e2m1x2).""" + a = x.abs() + q = torch.full_like(a, 6.0) + for hi, value, inclusive in ( + (5.0, 4.0, True), + (3.5, 3.0, False), + (2.5, 2.0, True), + (1.75, 1.5, False), + (1.25, 1.0, True), + (0.75, 0.5, False), + (0.25, 0.0, True), + ): + q = torch.where(a <= hi if inclusive else a < hi, torch.full_like(a, value), q) + return torch.where(x < 0, -q, q) + + +def _e4m3_round(x: torch.Tensor) -> torch.Tensor: + """x (fp32) to E4M3, nearest even, saturating (cvt.rn.satfinite.e4m3x2), as fp32.""" + return x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn).float() + + +def _ue8m0_up(x: torch.Tensor) -> torch.Tensor: + """UE8M0 code of x >= 0 rounded toward +inf, saturating (cvt.rp.satfinite.ue8m0x2), int32.""" + bits = x.contiguous().view(torch.int32) + code = ((bits >> 23) & 0xFF) + ((bits & 0x7FFFFF) != 0).to(torch.int32) + return code.clamp(0, 254) + + +def unswizzle_sf(sf: torch.Tensor, rows: int, blocks: int) -> torch.Tensor: + """The [rows, blocks] scale bytes of a 128x4-swizzled scale buffer (int64 indices, any byte dtype).""" + flat = sf.reshape(-1).view(torch.uint8) + r = torch.arange(rows, device=flat.device).view(-1, 1) + b = torch.arange(blocks, device=flat.device).view(1, -1) + padded_blocks = (blocks + 3) // 4 * 4 + idx = ( + (r // 128) * (128 * padded_blocks) + + (b // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (b % 4) + ) + return flat[idx] + + +def mxfp8_dequant(q: torch.Tensor, sf: torch.Tensor) -> torch.Tensor: + """fp32 values of e4m3 ``q`` [R, K] with its 128x4-swizzled UE8M0 scales (one per 32 elements).""" + rows, k = q.shape + codes = unswizzle_sf(sf, rows, k // 32).float() + return q.view(torch.float8_e4m3fn).float() * torch.exp2(codes - 127.0).repeat_interleave( + 32, dim=1 + ) + + +def mxfp8_quant_dequant(x: torch.Tensor) -> torch.Tensor: + """x [R, K] through MXFP8 (one UE8M0 scale per 32, the smallest power of two >= amax / 448; e4m3 data), back to + fp32 -- mxfp8_quantize's recipe, which every MXFP8 activation here reproduces.""" + rows, k = x.shape + blocks = x.float().view(rows, k // 32, 32) + code = _ue8m0_up(blocks.abs().amax(dim=-1) * _f32(1.0 / 448.0).to(x.device)) + scale = torch.exp2(code.float() - 127.0).unsqueeze(-1) + return (_e4m3_round(blocks / scale) * scale).view(rows, k) + + +def _bf16(x: torch.Tensor) -> torch.Tensor: + return x.to(torch.bfloat16).float() + + +def _gemm(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """a [M, K] @ b [N, K]^T, accumulated in fp64 and rounded once to fp32.""" + return (a.double() @ b.double().t()).float() + + +def _row_perm(rows: int, gated: bool) -> torch.Tensor: + """The TRTLLM-Gen row order: stored row j holds logical row perm[j] (the gated interleave, then the 32-row shuffle + of epilogue tile 128).""" + probe = torch.empty(rows, 1) + perm = get_shuffle_matrix_a_row_indices(probe, epilogue_tile_m=128) + if gated: + perm = get_reorder_rows_for_gated_act_gemm_row_indices(probe)[perm] + return perm + + +def _nvfp4_expert(w: torch.Tensor, sf: torch.Tensor, perm_inv: torch.Tensor) -> torch.Tensor: + """One expert's stored NVFP4 matrix (values [R, K / 2], swizzled e4m3 scales) as raw fp32 [R, K] in logical row + order (the global scale is not applied: it is the per-expert alpha).""" + rows, half = w.shape + codes = unswizzle_sf(sf, rows, 2 * half // 16).view(torch.float8_e4m3fn).float() + vals = _e2m1_decode(w) * codes.repeat_interleave(16, dim=1) + return vals[perm_inv] + + +def moe_route(x: torch.Tensor, router_weight: torch.Tensor, bias: torch.Tensor): + """MiniMax2 routing on the fp32 router GEMV: (top-4 expert ids [M, 4] int64, bf16-rounded weights [M, 4] fp32).""" + logits = _gemm(x, router_weight) + sig = (0.5 * torch.tanh(0.5 * logits).double() + 0.5).float() + keys = sig + bias + ids = torch.sort(keys, dim=-1, descending=True, stable=True).indices[:, :TOP_K] + sk = keys.gather(1, ids) - bias[ids] + total = (sk[:, 0] + sk[:, 2]) + (sk[:, 1] + sk[:, 3]) + w = (sk * _f32(ROUTED_SCALING).to(x.device)) / (total + _f32(1e-20).to(x.device)).unsqueeze(1) + return ids, _bf16(w) + + +def _nvfp4_act(x: torch.Tensor, sc: torch.Tensor) -> torch.Tensor: + """The activation's NVFP4 quantization (fp4_quantize(x, sc, 16)), as raw fp32 values (q x block scale).""" + rows, k = x.shape + blocks = x.view(rows, k // 16, 16) + vmax = blocks.abs().amax(dim=-1, keepdim=True) + sff = _e4m3_round(sc * (vmax * _f32(_RCP6).to(x.device))) + # The kernels' rcp.approx(sf x rcp.approx(sc)) lands where the correctly rounded sc / sf does on every element + # measured (the 1 / (sf x (1 / sc)) spelling differs on ~0.06% of them). + osc = torch.where(vmax != 0, sc / sff, torch.zeros_like(vmax)) + return (_e2m1_round(blocks * osc) * sff).view(rows, k) + + +def _fc1_epilogue(up: torch.Tensor, gate: torch.Tensor, esc: torch.Tensor) -> torch.Tensor: + """TRTLLM-Gen's SwiGLU-OAI FC1 epilogue on the raw FC1 sums (fp32 [I]) and its NVFP4 requantization: the raw FC2 + input values. ``esc`` = (fc31_alpha, fc31_scale_c, act_alpha, act_beta, act_clamp, fc2_alpha) of the expert, with + act_beta and act_clamp already divided by fc31_alpha (the loader's convention).""" + a, c, alpha, beta, lim = (esc[i] for i in range(5)) + gc = torch.where(gate > lim, lim, gate) + gc = torch.where(gate < -_F32_MAX, torch.full_like(gc, -_F32_MAX), gc) + uc = torch.minimum(torch.maximum(up, -lim), lim) + c1 = a * (alpha * _f32(_LOG2E).to(up.device)) + sgm = 1.0 / (torch.exp2(-(gc * c1)) + 1.0) + h = (uc.double() * a.double() + (a * beta).double()).float() * (gc * sgm) + blocks = h.view(-1, 16) + amax = (blocks.abs() * c).amax(dim=-1, keepdim=True) + sff = _e4m3_round(amax * _f32(_RCP6).to(up.device)) + osc = torch.where(sff != 0, 1.0 / sff, torch.zeros_like(sff)) + return (_e2m1_round((blocks * osc) * c) * sff).view(-1) + + +def _shared_expert(x: torch.Tensor, shared_mxfp8) -> torch.Tensor: + """The shared expert as the target runs it outside the engine: MXFP8 gate_up (bf16 out), SwiGLU-OAI with the + model's alpha / limit / (up + 1) in fp32 (bf16 out), MXFP8 down (bf16 out).""" + gu_w, gu_sf, dn_w, dn_sf = shared_mxfp8 + gu = _bf16(_gemm(mxfp8_quant_dequant(x), mxfp8_dequant(gu_w, gu_sf))) + gate, up = gu[:, :SHARED_INTER], gu[:, SHARED_INTER:] + a = torch.minimum(gate, torch.full_like(gate, SHARED_LIMIT)) + ub = up.clamp(-SHARED_LIMIT, SHARED_LIMIT) + 1.0 + sg = 1.0 / ( + torch.exp2((a * _f32(-SHARED_ALPHA).to(x.device)) * _f32(_LOG2E).to(x.device)) + 1.0 + ) + h = _bf16(ub * (sg * a)) + return _bf16(_gemm(mxfp8_quant_dequant(h), mxfp8_dequant(dn_w, dn_sf))) + + +def moe_reference(hidden: torch.Tensor, layer: M3MoeLayer, shared: bool) -> torch.Tensor: + """The MoE layer of the decode engines (both token ranges compute this), in plain torch: bf16 [M, 6144].""" + x = hidden.float() + m = x.shape[0] + ids, weights = moe_route(x, layer.router_weight, layer.bias) + xr = _nvfp4_act(x, layer.fc31_input_scale.reshape(()).float()) + inv31 = torch.argsort(_row_perm(2 * INTER, gated=True)).to(x.device) + inv2 = torch.argsort(_row_perm(HIDDEN, gated=False)).to(x.device) + y = torch.zeros(m, TOP_K, HIDDEN, dtype=torch.float32, device=x.device) + for e in torch.unique(ids).tolist(): + w31 = _nvfp4_expert(layer.w3_w1_weight[e], layer.w3_w1_weight_scale[e], inv31) + w2 = _nvfp4_expert(layer.w2_weight[e], layer.w2_weight_scale[e], inv2) + esc = layer.escal[:, e] + for t, k in (ids == e).nonzero().tolist(): + acc1 = _gemm(xr[t : t + 1], w31)[0] + # The buffer holds [w3 (up); w1 (gate)]. + h = _fc1_epilogue(acc1[:INTER], acc1[INTER:], esc) + y[t, k] = _bf16(_gemm(h.unsqueeze(0), w2)[0] * esc[5]) + acc = torch.zeros(m, HIDDEN, dtype=torch.float64, device=x.device) + for k in range(TOP_K): + acc = (acc + weights[:, k : k + 1].double() * y[:, k].double()).float().double() + routed = acc.float() + if not shared: + return routed.to(torch.bfloat16) + return (_shared_expert(x, layer.shared_mxfp8) + _bf16(routed)).to(torch.bfloat16) + + +_ENGAGED: set = set() + + +class M3MoeEngine(OpWrapper): + """One MiniMax-M3 MoE layer for M (1..8) decode tokens: ``layer.state.run(layer, hidden, out, arp)``. + + ``hidden`` bf16 [M, 6144] contiguous, M = the state's token count. ``layer`` (from ``M3MoeState.layer``) carries the + TRTLLM-Gen NVFP4 expert buffers read in place, the per-expert scalars, the router weight / bias, the activation's + NVFP4 global scale and, on a ``shared`` state, the shared expert's MXFP8 weights; its state carries the compiled + kernel and the workspace every layer on it shares. Returns bf16 [M, 6144]: bf16(shared + routed) on a ``shared`` + state, else the routed partial. ``out``: write there instead of a new tensor. ``arp`` (``ar_push_desc``): also + push the rows into a multicast all-reduce exchange. + + The routed part is TRTLLM-Gen's NVFP4 MoE on the fp32 router logits: sigmoid (tanh form) + correction bias top-4 + of 128 (ties to the lower id), weights sigmoid / sum * 2.0 rounded to bf16; the activation quantized to NVFP4 with + the layer's global scale; per expert the raw FC1 sums, SwiGLU-OAI (alpha / beta / clamp), the NVFP4 requant with + fc31_scale_c, FC2, bf16(fc2_alpha x sum); the fma chain over the top-4 in order. The shared expert is the target's + MXFP8 SwiGLU-OAI MLP. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # One cell per token count: m_max is a trace-time constant, so each is its own build, and the target builds all + # eight (decode batches of 1..8 requests, spec-decode verify rows up to 8). The whys name what the build differs in. + CELLS: tuple[Cell, ...] = ( + Cell( + why="1 token: FC1 -> FC2 records replicated 32x, logits 32x, last phase without trailing FC2", + spec=dict(tokens=1), + ), + Cell( + why="2 tokens: records 32x, logits 32x, the one build without AR_PRE", + spec=dict(tokens=2), + ), + Cell(why="3 tokens: records 8x, logits 32x, 4 FC2-B builder warps", spec=dict(tokens=3)), + Cell( + why="4 tokens: one request verifying 3 drafts (the TP2 b1 point): 6-stage ring, B2IMG hand-off, " + "the 4 two-block FC2 CTAs stream no FC1", + spec=dict(tokens=4), + ), + Cell( + why="5 tokens: single records, a routing warp takes two tokens, logits 32x", + spec=dict(tokens=5), + ), + Cell( + why="6 tokens: single records, routing warps 8.. take tokens 4..5", spec=dict(tokens=6) + ), + Cell(why="7 tokens: the B2IMG global FC1 -> FC2 hand-off", spec=dict(tokens=7)), + Cell( + why="8 tokens: two requests verifying 3 drafts (the TP2 b2 point): 5-stage ring, x read from global " + "(no XSTAGE), no early group 0, the 4 two-block FC2 CTAs stream no FC1", + spec=dict(tokens=8), + ), + ) + + note = """ + The kernel's elementwise math uses the PTX approximations (rcp.approx, ex2.approx, tanh) where `reference` uses + torch's correctly rounded ones, and its GEMMs sum in tensor-core order where `reference` sums in fp64: a + quantization boundary (NVFP4 activation / intermediate code, routing near-tie) can fall on the other side, so the + two agree within the band the test derives, not bit for bit. The routed part is bit for bit the stock TRTLLM-Gen + NVFP4 MoE (fp4_quantize + fp4_block_scale_moe_runner) on the same routing; the test checks that too. + + Not covered: the all-reduce push (`arp`), which needs a multicast exchange (the decode forward's boundary all-reduce + exercises it); the non-shared state (`shared=False`), which this target never builds. Every layer on one state + must run in one stream order: the workspace is shared and tagged per call, never reset. + + No `is_valid`: the compiled call checks every buffer's dtype, shape and alignment against its signature and + raises on a mismatch (a misaligned `hidden`, an `out` that is not bf16 [M, 6144], a malformed `arp`). + """ + + def __call__( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + state = layer.state + if (state.device, state.m) not in _ENGAGED: + # The engagement line (once per device and token count, on every rank: the first call, eager or captured). + _ENGAGED.add((state.device, state.m)) + logger.info( + f"[M3 sm_107 tp2] MoE engine m3_moe engaged: {state.m} tokens, {state.num_ctas} routed + " + f"{state.mod.NSH} shared CTAs, {state.mod.STAGES}-stage ring, on {state.device}" + ) + return state.run(layer, hidden, out, arp=arp) + + def reference( + self, + hidden: torch.Tensor, + layer: M3MoeLayer, + out: Optional[torch.Tensor] = None, + arp: Optional[torch.Tensor] = None, + ) -> torch.Tensor: + """Plain torch over the layer's buffers (see the class docstring); `out` and `arp` are ignored.""" + return moe_reference(hidden, layer, shared=layer.state.shared) + + +m3_moe_engine = M3MoeEngine() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/m3_router_tc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/m3_router_tc.py new file mode 100644 index 000000000000..13295b82cb90 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/moe/m3_router_tc.py @@ -0,0 +1,151 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TF32 tensor-core router of the decode MoE for 2..48 tokens, optionally with the NVFP4 activation.""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +_E4M3_MAX = 448.0 +_E2M1_MAX = 6.0 +#: Midpoints between consecutive e2m1 magnitudes; ``_TIE_UP[i]``: round-to-nearest-even takes the upper code. +_E2M1_MIDPOINTS = (0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0) +_TIE_UP = (False, True, False, True, False, True, False) +#: The kernel builds each block's output scale through two ``rcp.approx.ftz.f32`` reciprocals (~2^-23 relative): +#: a value this close to an e2m1 midpoint may round to either neighbour, everything further out must match. +TIE_WINDOW = 2.0**-20 + + +def nvfp4_quantize( + x: torch.Tensor, sf_scale: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """``fp4_quantize(x, sf_scale, 16, False, False)``: (packed e2m1 [M, K / 2], linear E4M3 scales [M * K / 16], + the pre-rounding scaled values [M, K] that ``near_tie`` reads). + + Per 16 columns: scale = e4m3(sf_scale * amax / 6), data = x * sf_scale / scale rounded to nearest-even e2m1. + """ + gs = sf_scale.float().reshape(()) + m, k = x.shape + blocks = x.float().view(m, k // 16, 16) + vecmax = blocks.abs().amax(-1) + sf = (gs * (vecmax / _E2M1_MAX)).clamp(max=_E4M3_MAX).to(torch.float8_e4m3fn) + out_scale = torch.where(vecmax != 0, gs / sf.float(), torch.zeros_like(vecmax)) + scaled = (blocks * out_scale.unsqueeze(-1)).view(m, k) + mid = torch.tensor(_E2M1_MIDPOINTS, device=x.device, dtype=torch.float32) + tie_up = torch.tensor(_TIE_UP, device=x.device, dtype=torch.bool) + a = scaled.abs().unsqueeze(-1) + code = ((a > mid).sum(-1) + ((a == mid) & tie_up).any(-1)).to(torch.uint8) + negative = (scaled < 0) | ((scaled == 0) & torch.signbit(scaled)) + code = code | (negative.to(torch.uint8) << 3) + packed = code[:, 0::2] | (code[:, 1::2] << 4) + return packed, sf.view(torch.uint8).reshape(-1), scaled + + +def near_tie(scaled: torch.Tensor) -> torch.Tensor: + """Elements whose e2m1 rounding the kernel's approximate reciprocals decide (see ``TIE_WINDOW``).""" + mid = torch.tensor(_E2M1_MIDPOINTS, device=scaled.device, dtype=torch.float32) + a = scaled.abs().unsqueeze(-1) + return ((a - mid).abs() <= TIE_WINDOW * a).any(-1) + + +def fp32_logits(x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + """``float(x) @ w^T`` as a true fp32 product (TF32 off).""" + with true_fp32_matmul(): + return x.float() @ w.t() + + +def tf32_rna(w: torch.Tensor) -> torch.Tensor: + """fp32 -> TF32 as ``cvt.rna.tf32.f32``: keep 10 mantissa bits, round to nearest with ties away from zero.""" + bits = w.contiguous().view(torch.int32) + rounded = (bits + 0x1000) & ~0x1FFF + finite = (bits & 0x7F800000) != 0x7F800000 + return torch.where(finite, rounded, bits).view(torch.float32) + + +class M3RouterTc(OpWrapper): + """Router logits on TF32 tensor cores for 128 experts and 1..48 tokens: ``x @ tf32(w)^T``, fp32 accumulate. + + ``x`` is BF16 (exact in TF32), ``w`` the fp32 router weight, rounded to TF32 in the kernel. The 16 K slices' + partials are summed in slice order whatever cluster size the device co-schedules (16 or 8 CTAs), so the result + does not depend on it. With ``sf_scale`` also the NVFP4 activation as ``nvfp4_quantize``: returns + ``[logits]`` or ``[logits, x_fp4, x_sf]``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # The target routes 2..48 tokens here: plain logits on the unfused path, logits + NVFP4 on the fused-router path. + CELLS: tuple[Cell, ...] = ( + Cell( + why="two tokens, logits only (the smallest call the target makes here)", + spec=dict(m=2, k=6144, fp4=False), + ), + Cell( + why="8 tokens with the NVFP4 activation: one 8-token MMA tile, every expert tile quantizes one row", + spec=dict(m=8, k=6144, fp4=True), + ), + Cell( + why="13 tokens: a partial second token tile", + spec=dict(m=13, k=6144, fp4=True), + ), + Cell( + why="the 48-token maximum, six token tiles, logits only", + spec=dict(m=48, k=6144, fp4=False), + ), + Cell( + why="the 48-token maximum with the NVFP4 activation", + spec=dict(m=48, k=6144, fp4=True), + ), + # Every other row count the router takes (modeling's ROUTER_TC), with and without the NVFP4 activation. + *( + Cell( + why=f"{m} rows: a router row count between the ones above", + spec=dict(m=m, k=6144, fp4=m % 2 == 1), + ) + for m in range(2, 49) + if m not in (2, 8, 13, 48) + ), + ) + + note = """ + The kernel uses 16-CTA clusters when the device co-schedules eight of them and 8-CTA clusters otherwise; both + sum the same 16 slice partials in the same order. Only the one the test device picks is driven. + """ + + def __call__( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> list[torch.Tensor]: + return torch.ops.trtllm.minimax_m3_router_tc(x, w, sf_scale, pdl) + + def reference( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> tuple[torch.Tensor, ...]: + """fp32 product of ``x`` and the TF32-rounded weight, plus the NVFP4 recipe with ``sf_scale``.""" + with true_fp32_matmul(): + logits = x.float() @ tf32_rna(w).t() + if sf_scale is None: + return (logits,) + packed, sf, _ = nvfp4_quantize(x, sf_scale) + return logits, packed, sf + + def is_valid( + self, x: torch.Tensor, w: torch.Tensor, sf_scale: Optional[torch.Tensor], pdl: bool + ) -> None: + assert x.data_ptr() % 16 == 0, "x must start 16-byte aligned (cp.async.bulk source)" + if sf_scale is not None: + value = float(sf_scale.reshape(())) + assert value > 0.0 and value != float("inf"), ( + f"sf_scale must be positive and finite, got {value}" + ) + + +m3_router_tc = M3RouterTc() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/norm/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/norm/__init__.py new file mode 100644 index 000000000000..acec8fdbb26f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/norm/__init__.py @@ -0,0 +1,3 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Normalization entries this target alone calls.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/norm/m3_embedding_firstnorm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/norm/m3_embedding_firstnorm.py new file mode 100644 index 000000000000..c2e14b5d8c4b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/norm/m3_embedding_firstnorm.py @@ -0,0 +1,90 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode-step embedding gather and the first layer's input RMSNorm in one CuTe DSL launch, into caller buffers.""" + +from __future__ import annotations + +from typing import Tuple + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.embedding_firstnorm.op import DecodeEmbedding, m3_embed_norm + +__all__ = ["DecodeEmbedding", "m3_embedding_firstnorm"] + +VOCAB = 200064 # MiniMax-M3's vocabulary; the decode table is the full (replicated) embedding + + +class M3EmbeddingFirstnorm(OpWrapper): + """Write ``table[ids]`` into ``raw`` (bf16 [M, 6144], a zero row for an id outside ``[0, V)``) and + ``raw * rsqrt(mean(raw^2) + eps) * weight`` into ``out``; returns ``(out, raw)`` -- the first layer's normed input + and its residual. ``ids`` int32 or int64 [M], M = 1..8; ``table`` bf16 [V, 6144] contiguous; ``weight`` bf16 + [6144]. The first call per (M, id dtype, table shape, PDL) compiles (``DecodeEmbedding`` does it at build).""" + + ARCHS = frozenset({Arch.SM_107}) + + # The target builds every (rows 1..8, id dtype) owner; the kernel is one CTA per row, so the rows differ only in + # grid size: the ends of the range and both id dtypes. + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, int32 ids", spec=dict(rows=1, id_dtype=torch.int32)), + Cell(why="eight requests, int32 ids", spec=dict(rows=8, id_dtype=torch.int32)), + Cell(why="one request, int64 ids", spec=dict(rows=1, id_dtype=torch.int64)), + Cell( + why="six rows (one request verifying a draft), int64 ids", + spec=dict(rows=6, id_dtype=torch.int64), + ), + # Every other row count the decode dispatch takes (modeling's EMBED_FIRSTNORM), both id dtypes. + *( + Cell( + why=f"{rows} rows: a decode row count between the ones above", + spec=dict(rows=rows, id_dtype=torch.int32 if rows % 2 else torch.int64), + ) + for rows in (2, 3, 4, 5, 7) + ), + ) + + note = """ + The gathered rows are exact. The norm is flashinfer's RMSNorm arithmetic in its reduction order with a fast + rsqrt, where `reference` sums in fp64 and takes torch's rsqrt: within the band the test derives, not bit for bit. + SM107 against the stock embedding + RMSNorm module (flashinfer's kernel, 768 threads): the gathered rows are + bitwise equal; the normed rows differ in one bf16 ulp on ~5e-6 of the elements (~1.6% of rows, 1-2 elements each: + the fp32 sum-of-squares tree differs, so the rsqrt input can differ in its last bits). + An id outside [0, V) gives a zero row and a zero norm (torch's embedding would raise). No accepted input is known + to compute wrongly: the host side rejects any other dtype, shape or a strided buffer. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + return m3_embed_norm(ids, table, weight, eps, raw, out, use_pdl) + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + weight: torch.Tensor, + eps: float, + raw: torch.Tensor, + out: torch.Tensor, + use_pdl: bool = True, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """New tensors (``raw`` / ``out`` are not written); `use_pdl` is ignored.""" + valid = (ids >= 0) & (ids < table.shape[0]) + rows = table[torch.where(valid, ids, torch.zeros_like(ids)).long()] * valid.unsqueeze(1).to( + table.dtype + ) + x = rows.double() + rstd = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps) + return (x * rstd * weight.double()).to(torch.bfloat16), rows + + +m3_embedding_firstnorm = M3EmbeddingFirstnorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/__init__.py new file mode 100644 index 000000000000..b3ba19cee80e --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampling entries this target alone calls: the spec sampler's top-p renorm override (TOPP_DET), its softmax +override (SOFTMAX_MC) and its chain speculative sampling override (REJ_MC).""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_chain_spec_mc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_chain_spec_mc.py new file mode 100644 index 000000000000..613e1f7add7b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_chain_spec_mc.py @@ -0,0 +1,382 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""REJ_MC: the spec sampler's chain speculative sampling (rejection) on many CTAs per request, exact vs +``torch.ops.trtllm.chain_speculative_sampling`` (``trtllm::minimax_m31_chain_speculative_sampling_mc``, +kernels/minimaxM31/chainSpecSamplingMcSm103.cu). + +The target installs ``chain_spec_mc`` as the rejection sampler's override +(``speculative/interface.set_chain_speculative_sampling_override``); it returns None, and the sampler keeps the stock +op, whenever the op is not built or the input is outside the kernel's contract. + +Workspace: one zero-initialized byte buffer per device, ``MAX_BATCH`` requests of +``minimax_m31_chain_speculative_sampling_mc_ws_bytes()``, kept for the life of the process (CUDA graphs captured with +it keep its address). The kernel leaves it ready for the next launch, so it is zeroed once. ``prebuild`` allocates it +outside any CUDA-graph capture (the target calls it from ``cache_derived_state``, before the executor's warmup and +captures); a call that finds no workspace allocates one when the stream is not capturing, and inside a capture returns +None (the stock op for that call) rather than allocating from the graph's private pool. Calls that share it must be +stream-ordered (the sampler's are). +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +# The largest CTA count per request (the grid is batch x ctas_per_row 1024-thread CTAs, all co-resident; ctas_per_row +# is sized from the device's own SM count, so the same entry holds on GB300 and VR). REJ v2 microbench (GB300): one +# request 32.3 us at 16 (v1) -> 24.8 us at 64 (v2), 25.8 at 32, 25.5 at 128; bitwise at every count. +MAX_CTAS_PER_ROW = 64 + +# The requests the workspace holds and the override takes: the TP8 / TP4 K7 calls up to c8. A GB300 microbench +# gated it bitwise at batch 1 / 2 / 4 / 8 and measured 31.2 vs 41.2 us per call at batch 1; larger calls +# keep the stock kernel. +MAX_BATCH = 8 + +# The widest vocab the kernel takes (64 tiles of 4096; kChainSpecMcMaxVocab). +MAX_VOCAB = 64 * 4096 +MAX_DRAFT = ( + 32 # capacity: fixed: the kernel's shared acceptance probabilities (kChainSpecMcMaxDraft) +) + +# The op when it is built, decided once per process. +_STATE: dict = {} +# REJ_V2 engagement: the grid sizes (ctas_per_row) a served call has used, each logged once per process (host only; a +# captured call logs while it is captured). Distinct from the install line, which reports the batch-MAX_BATCH grid +# (16 on VR for v1 and v2). +_V2_TAKEN: set = set() + + +def _rej_v2_taken(batch: int, ctas: int) -> None: + """The REJ_V2 taken line: once per distinct ctas_per_row a served call launches with.""" + if ctas in _V2_TAKEN: + return + _V2_TAKEN.add(ctas) + from tensorrt_llm.logger import logger + + if len(_V2_TAKEN) == 1: + # REJ v3 engagement (the kernel the lib launches; TRTLLM_M31_DISABLE_REJ_V3=1 keeps v2), once per process. + if kernel_version() == 3: + logger.info(f"[M3 dispatch] REJ_V3 taken (ws_bytes_per_row {ws_bytes_per_row()})") + else: + logger.info( + f"[M3 dispatch] REJ_V2_KERNEL kept (REJ_V3 off: kernel {kernel_version()}, " + f"ws_bytes_per_row {ws_bytes_per_row()})" + ) + logger.info( + f"[M3 dispatch] REJ_V2 taken (max_ctas_per_row {MAX_CTAS_PER_ROW}, ctas_per_row {ctas} at batch {batch})" + ) + + +# Per device index: the persistent workspace and the SM count. +_WORKSPACES: dict = {} +_SMS: dict = {} + + +def chain_spec_mc_op(): + """``trtllm::minimax_m31_chain_speculative_sampling_mc`` if this build carries it, else + None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_chain_speculative_sampling_mc", None) + _STATE["op"] = op + _STATE["ws_bytes"] = ( + int(torch.ops.trtllm.minimax_m31_chain_speculative_sampling_mc_ws_bytes()) + if op is not None + else 0 + ) + return _STATE["op"] + + +def ws_bytes_per_row() -> int: + """Workspace bytes per request (0 when the op is not built).""" + chain_spec_mc_op() + return _STATE["ws_bytes"] + + +def kernel_version() -> int: + """The kernel the op launches: 3 (REJ v3) or 2 (v2, or a lib without the version op); 0 when not built.""" + if "kernel" not in _STATE: + if chain_spec_mc_op() is None: + _STATE["kernel"] = 0 + else: + version = getattr( + torch.ops.trtllm, "minimax_m31_chain_speculative_sampling_mc_version", None + ) + _STATE["kernel"] = int(version()) if version is not None else 2 + return _STATE["kernel"] + + +def sm_count(device: torch.device) -> int: + """The device's SM count, cached per device.""" + idx = torch.device(device).index + idx = torch.cuda.current_device() if idx is None else idx + if idx not in _SMS: + _SMS[idx] = torch.cuda.get_device_properties(idx).multi_processor_count + return _SMS[idx] + + +def ctas_per_row(batch: int, sms: int) -> int: + """The largest power of two <= MAX_CTAS_PER_ROW with ctas x batch <= sms (1 past that: one CTA per request needs + no co-residency).""" + g = MAX_CTAS_PER_ROW + while g > 1 and g * batch > sms: + g //= 2 + return g + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's zeroed workspace for MAX_BATCH requests; must run outside a CUDA-graph capture. + Returns it, or None when the op is not built or the stream is capturing.""" + if chain_spec_mc_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.zeros( + MAX_BATCH * ws_bytes_per_row(), dtype=torch.uint8, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31ChainSpecMc(OpWrapper): + """``trtllm::chain_speculative_sampling(draft_probs, draft_token_ids, target_probs, seed, offset)`` bit for bit + (accepted tokens [batch, D + 1] and token counts [batch]) in ONE launch of batch x ctas_per_row CTAs (stock: one + CTA per request). ``workspace`` is the persistent zeroed buffer of ``prebuild``. ``__call__`` returns None (the + caller keeps the stock op) for a non-fp32, non-contiguous or misaligned probs, a vocab not divisible by 4 or above + MAX_VOCAB, mismatched shapes, non-int32 draft tokens, non-int64 or mis-sized seed / offset, inputs on different + devices, more requests than the workspace holds, a batch x ctas_per_row above the SM count, or when the op is not + built.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The K7 calls the microbench gated bitwise on GB300 (5 cases x 64 seeds at ctas_per_row 4 / 8 / 16); + # the sm_107 build of the same source is gated by the same MB on VR before the target takes it. + CELLS: tuple[Cell, ...] = ( + Cell(why="K7 at c1 (vocab 200064, 16 CTAs)", spec=dict(batch=1, k=7, v=200064)), + Cell(why="K7 at c2 (vocab 200064)", spec=dict(batch=2, k=7, v=200064)), + Cell(why="K7 at c4 (vocab 200064)", spec=dict(batch=4, k=7, v=200064)), + Cell(why="K7 at c8 (vocab 200064, 128 CTAs)", spec=dict(batch=8, k=7, v=200064)), + ) + + note = """ + Exact by construction: every CTA of a request runs the stock Philox acceptance loop, the (target, residual) mass is + the stock 1024 per-thread chains into the same cub BlockReduce, each tile sum is the stock per-tile + cub reduce, and one CTA runs the stock tile-order prefix and in-tile cub BlockScan search. Default flags, as the + stock kernel. + """ + + def __call__( + self, + draft_probs: torch.Tensor, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, + workspace: Optional[torch.Tensor], + ctas_per_row: int, + ) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + if ( + workspace is None + or not all( + isinstance(t, torch.Tensor) + for t in (draft_probs, draft_token_ids, target_probs, seed, offset) + ) + or not draft_probs.is_cuda + or draft_probs.dtype != torch.float32 + or target_probs.dtype != torch.float32 + or draft_token_ids.dtype != torch.int32 + or seed.dtype != torch.int64 + or offset.dtype != torch.int64 + or draft_probs.dim() != 3 + or target_probs.dim() != 3 + or draft_token_ids.dim() != 2 + ): + return None + batch, k, v = draft_probs.shape + if ( + batch < 1 + or tuple(draft_token_ids.shape) != (batch, k) + or tuple(target_probs.shape) != (batch, k + 1, v) + or seed.numel() != batch + or offset.numel() != batch + or v % 4 != 0 + or v > MAX_VOCAB + or k > MAX_DRAFT + or any( + not t.is_contiguous() or t.device != draft_probs.device + for t in (draft_token_ids, target_probs, seed, offset, workspace) + ) + or not draft_probs.is_contiguous() + or draft_probs.data_ptr() % 16 != 0 + or target_probs.data_ptr() % 16 != 0 + or ctas_per_row < 1 + or 1024 % ctas_per_row != 0 + or (ctas_per_row > 1 and batch * ctas_per_row > sm_count(draft_probs.device)) + ): + return None + op = chain_spec_mc_op() + if op is None or workspace.numel() * workspace.element_size() < batch * ws_bytes_per_row(): + return None + tokens, num_tokens = op( + draft_probs, draft_token_ids, target_probs, seed, offset, workspace, int(ctas_per_row) + ) + return tokens, num_tokens + + def reference( + self, + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace=None, + ctas_per_row=None, + ) -> tuple[torch.Tensor, torch.Tensor]: + """The stock op (workspace and ctas_per_row are ignored).""" + return torch.ops.trtllm.chain_speculative_sampling( + draft_probs, draft_token_ids, target_probs, seed, offset + ) + + def is_valid( + self, + draft_probs, + draft_token_ids, + target_probs, + seed, + offset, + workspace=None, + ctas_per_row=None, + **kwargs, + ) -> None: + assert draft_probs.dtype == torch.float32 and draft_probs.dim() == 3, ( + "draft_probs: fp32 [batch, D, vocab]" + ) + + +m31_chain_spec_mc = M31ChainSpecMc() + + +def chain_spec_mc( + draft_probs: torch.Tensor, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, +) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + """The rejection sampler's override: the stock op's (accepted tokens, token counts), or None (the sampler runs the + stock op) when the op is not built, the batch exceeds MAX_BATCH, no workspace can be had (first use inside a + capture) or the input is outside the contract.""" + if ( + not isinstance(draft_probs, torch.Tensor) + or not draft_probs.is_cuda + or draft_probs.dim() != 3 + ): + return None + batch = draft_probs.shape[0] + if batch < 1 or batch > MAX_BATCH or chain_spec_mc_op() is None: + return None + workspace = prebuild(draft_probs.device) + if workspace is None: + return None + g = ctas_per_row(batch, sm_count(draft_probs.device)) + out = m31_chain_spec_mc(draft_probs, draft_token_ids, target_probs, seed, offset, workspace, g) + if out is not None: + _rej_v2_taken(batch, g) + return out + + +def chain_spec_mc_slots_op(): + """``trtllm::minimax_m31_chain_speculative_sampling_mc_slots`` (REJ_SLOTS) if this build carries it, else None.""" + if chain_spec_mc_op() is None: + return None + return getattr(torch.ops.trtllm, "minimax_m31_chain_speculative_sampling_mc_slots", None) + + +def chain_spec_mc_slots( + draft_buf: torch.Tensor, + slot_ids: torch.Tensor, + num_draft: int, + draft_token_ids: torch.Tensor, + target_probs: torch.Tensor, + seed: torch.Tensor, + offset: torch.Tensor, +) -> Optional[tuple[torch.Tensor, torch.Tensor]]: + """The rejection sampler's slot-indexed override (REJ_SLOTS): ``chain_spec_mc`` of the gathered + ``draft_buf[slot_ids, :num_draft, :vocab]`` (vocab = target_probs' last dim), reading those rows in place instead of + from the copy -- the same floats, so bitwise the gathered call. None (the sampler gathers and runs the plain path) + when the op is not built or the input is outside the contract.""" + op = chain_spec_mc_slots_op() + if ( + op is None + or not all( + isinstance(t, torch.Tensor) + for t in (draft_buf, slot_ids, draft_token_ids, target_probs, seed, offset) + ) + or not draft_buf.is_cuda + or draft_buf.dtype != torch.float32 + or draft_buf.dim() != 3 + or not draft_buf.is_contiguous() + or draft_buf.data_ptr() % 16 != 0 + or draft_buf.shape[2] % 4 != 0 + or slot_ids.dtype != torch.int64 + or slot_ids.dim() != 1 + or not slot_ids.is_contiguous() + or target_probs.dtype != torch.float32 + or target_probs.dim() != 3 + or draft_token_ids.dtype != torch.int32 + or seed.dtype != torch.int64 + or offset.dtype != torch.int64 + ): + return None + batch = slot_ids.shape[0] + v = target_probs.shape[2] + if ( + batch < 1 + or batch > MAX_BATCH + or num_draft < 0 + or num_draft > draft_buf.shape[1] + or v > draft_buf.shape[2] + or v % 4 != 0 + or v > MAX_VOCAB + or tuple(draft_token_ids.shape) != (batch, num_draft) + or tuple(target_probs.shape) != (batch, num_draft + 1, v) + or seed.numel() != batch + or offset.numel() != batch + or target_probs.data_ptr() % 16 != 0 + or any( + not t.is_contiguous() or t.device != draft_buf.device + for t in (slot_ids, draft_token_ids, target_probs, seed, offset) + ) + ): + return None + workspace = prebuild(draft_buf.device) + if ( + workspace is None + or workspace.numel() * workspace.element_size() < batch * ws_bytes_per_row() + ): + return None + g = ctas_per_row(batch, sm_count(draft_buf.device)) + if g > 1 and batch * g > sm_count(draft_buf.device): + return None + _rej_v2_taken(batch, g) + return op( + draft_buf, + slot_ids, + int(num_draft), + draft_token_ids, + target_probs, + seed, + offset, + workspace, + int(g), + ) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_softmax_mc.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_softmax_mc.py new file mode 100644 index 000000000000..c6ac17835482 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_softmax_mc.py @@ -0,0 +1,186 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SOFTMAX_MC: the spec sampler's softmax with its write pass on many CTAs per row, exact vs +``flashinfer.sampling.softmax(logits, temperature)`` (``trtllm::minimax_m31_softmax_mc``, +kernels/minimaxM31/softmaxMcSm103.cu). + +The target installs ``softmax_mc`` as the sampler's softmax override (``sampler/ops/flashinfer.set_softmax_override``); +it returns None, and the sampler keeps the stock FlashInfer call, whenever the op is not built or the input is outside +the kernel's contract (FlashInfer's Path A shapes: rows <= 128 and vocab >= 24576; here also rows <= ``MAX_ROWS``). + +Workspace: one partials buffer per device (``MAX_ROWS`` x ceil(``MAX_VOCAB`` / 8192) float2, 8 KB), kept for the life +of the process (CUDA graphs captured with it keep its address). Every launch overwrites what it reads, so it needs no +initialization. ``prebuild`` allocates it outside any CUDA-graph capture (the target calls it from +``cache_derived_state``, before the executor's warmup and captures); a call that finds no workspace allocates one when +the stream is not capturing, and inside a capture returns None (stock FlashInfer for that call) rather than allocating +from the graph's private pool. Calls that share it must be stream-ordered (the sampler's are). +""" + +from __future__ import annotations + +from typing import Optional, Union + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +# The write pass's CTAs per row (GB300 microbench: 9.2 vs FlashInfer 21.8 us per call at 7-8 rows; +# bitwise at 4 / 8 / 16). The grid (rows x WRITE_CTAS) needs no co-residency, so the SM count does not bound it. +WRITE_CTAS = 16 + +# The rows the override takes: the TP8 / TP4 K7 c1 calls (the draft's 7 rows and the verify's 8), where the MB measured +# it. Larger calls keep the stock kernel. +MAX_ROWS = 8 + +# FlashInfer's Path A bounds (OnlineSoftmax: batch <= 128 and d >= 24576); outside them FlashInfer takes its +# single-block Path B, which this kernel does not reproduce. +PATH_A_MAX_ROWS = 128 +PATH_A_MIN_VOCAB = 24576 +# FlashInfer's Path A slice (DEFAULT_SLICE_SIZE) and the widest vocab the workspace holds. +SLICE = 8192 +MAX_VOCAB = 1 << 20 + +# The op when it is built, decided once per process. +_STATE: dict = {} +# Per device index: the persistent partials workspace. +_WORKSPACES: dict = {} + + +def softmax_mc_op(): + """``trtllm::minimax_m31_softmax_mc`` if this build carries it, else None.""" + if "op" not in _STATE: + op = getattr(torch.ops.trtllm, "minimax_m31_softmax_mc", None) + _STATE["op"] = op + return _STATE["op"] + + +def partials_numel(rows: int, d: int) -> int: + """fp32 elements of the partials workspace for rows x d (one float2 per row and 8192-wide slice).""" + return 2 * rows * ((d + SLICE - 1) // SLICE) + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's partials workspace for MAX_ROWS rows of up to MAX_VOCAB; must run outside a + CUDA-graph capture. Returns it, or None when the op is not built or the stream is capturing.""" + if softmax_mc_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.empty( + partials_numel(MAX_ROWS, MAX_VOCAB), dtype=torch.float32, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31SoftmaxMc(OpWrapper): + """FlashInfer's ``softmax(logits, temperature)`` of fp32 logits [rows, d], bit for bit, on FlashInfer's Path A + shapes (rows <= 128, d >= 24576): FlashInfer's Map kernel, then its Reduce kernel's prologue in each of + ``write_ctas`` CTAs per row, each writing its share of the row (FlashInfer: one CTA writes the whole row). + ``temperature`` is None (1.0), a Python number (one temperature) or an fp32 [rows] tensor (one per row), as + FlashInfer takes it. ``partials`` is the persistent workspace of ``prebuild``. ``__call__`` returns None (the caller + keeps the stock call) for a non-fp32, non-contiguous or misaligned input, a width not divisible by 4, a shape + outside Path A or the workspace, a temperature tensor that is not a contiguous fp32 [rows] tensor on logits' + device, or when the op is not built.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The c1 K7 calls a microbench gated bitwise on GB300; the sm_107 build of the same source is gated + # by the same MB on VR before the target takes it. + CELLS: tuple[Cell, ...] = ( + Cell(why="the K7 draft's 7 rows at c1 (vocab 200064)", spec=dict(rows=7, d=200064)), + Cell(why="the K7 verify's 8 rows at c1 (vocab 200064)", spec=dict(rows=8, d=200064)), + Cell(why="one row (the smallest grid)", spec=dict(rows=1, d=200064)), + ) + + note = """ + Exact by construction: the Map kernel is FlashInfer's, every write CTA reduces the same partials with the same cub + BlockReduce and Float2SoftmaxReduceOp as FlashInfer's single Reduce CTA, and the per-element write + keeps FlashInfer's source form. Built with --use_fast_math, as FlashInfer's JIT builds it. + """ + + def __call__( + self, + logits: torch.Tensor, + temperature: Union[torch.Tensor, float, None], + partials: Optional[torch.Tensor], + write_ctas: int, + ) -> Optional[torch.Tensor]: + if ( + partials is None + or not isinstance(logits, torch.Tensor) + or not logits.is_cuda + or logits.dtype != torch.float32 + or logits.dim() != 2 + or not logits.is_contiguous() + or logits.shape[0] < 1 + or logits.shape[0] > PATH_A_MAX_ROWS + or logits.shape[1] < PATH_A_MIN_VOCAB + or logits.shape[1] % 4 != 0 + or logits.data_ptr() % 16 != 0 + or partials.device != logits.device + or partials.numel() * partials.element_size() + < 4 * partials_numel(logits.shape[0], logits.shape[1]) + or write_ctas < 1 + or write_ctas > 65535 + ): + return None + if temperature is None: + temp_arr, temp_val = None, 1.0 + elif isinstance(temperature, torch.Tensor): + if ( + temperature.dtype != torch.float32 + or temperature.dim() != 1 + or temperature.shape[0] != logits.shape[0] + or not temperature.is_contiguous() + or temperature.device != logits.device + ): + return None + temp_arr, temp_val = temperature, 0.0 + elif isinstance(temperature, (int, float)) and not isinstance(temperature, bool): + temp_arr, temp_val = None, float(temperature) + else: + return None + op = softmax_mc_op() + if op is None: + return None + out = torch.empty_like(logits) + op(logits, temp_arr, temp_val, partials, out, int(write_ctas)) + return out + + def reference(self, logits, temperature, partials=None, write_ctas=None) -> torch.Tensor: + """FlashInfer's softmax (partials and write_ctas are ignored).""" + import flashinfer.sampling + + return flashinfer.sampling.softmax(logits, temperature) + + def is_valid(self, logits, temperature, partials=None, write_ctas=None, **kwargs) -> None: + assert logits.dtype == torch.float32 and logits.dim() == 2, "logits: fp32 [rows, d]" + + +m31_softmax_mc = M31SoftmaxMc() + + +def softmax_mc( + logits: torch.Tensor, temperature: Union[torch.Tensor, float, None] +) -> Optional[torch.Tensor]: + """The sampler's softmax override: FlashInfer's softmax of logits [rows, d] at temperature, or None (the sampler + runs stock FlashInfer) when the op is not built, rows exceed MAX_ROWS, d exceeds MAX_VOCAB, no workspace can be had + (first use inside a capture) or the input is outside the contract.""" + if not isinstance(logits, torch.Tensor) or not logits.is_cuda or logits.dim() != 2: + return None + rows = logits.shape[0] + if rows < 1 or rows > MAX_ROWS or logits.shape[1] > MAX_VOCAB or softmax_mc_op() is None: + return None + partials = prebuild(logits.device) + if partials is None: + return None + return m31_softmax_mc(logits, temperature, partials, WRITE_CTAS) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_topp_det.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_topp_det.py new file mode 100644 index 000000000000..21d3fa62178c --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/sampling/m31_topp_det.py @@ -0,0 +1,257 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TOPP_DET: the spec sampler's top-p renorm as one persistent C++ launch, exact vs FlashInfer's deterministic +``top_p_renorm_probs`` (``trtllm::minimax_m31_topp_renorm_det``, kernels/minimaxM31/toppRenormDetSm103.cu). + +TOPP_DET v3 (``trtllm::minimax_m31_topp_renorm_det_v3``, kernels/minimaxM31/toppRenormDetV3Sm103.cu) is the same +renorm, bitwise v1.2's output, with the redundant per-CTA selection and a programmatic dependent launch. It is taken +when it is built unless +``TRTLLM_M31_DISABLE_TOPPDET_V3`` is set; v1.2 otherwise. ``TRTLLM_M31_DISABLE_TOPPDET_T3`` keeps v3 on its global +radix passes (no candidate list). + +V4K8 (a library whose op takes ``list_path`` / ``wide_hist``, default on): the one-sync list path (pass 0 lists the +elements >= 2^-12; when the selected bucket lies at or above that edge every CTA finishes the selection, FlashInfer's +chain sums and the write from the list after one barrier) and 8 privatized pass histograms, both bitwise the same +output; ``TRTLLM_M31_DISABLE_TOPPDET_V4`` / ``TRTLLM_M31_DISABLE_TOPPDET_K8`` turn them off. + +The target installs ``top_p_renorm_det`` as the sampler's top-p renorm override +(``sampler/ops/flashinfer.set_top_p_renorm_override``); it returns None, and the sampler keeps the stock FlashInfer +call, whenever the op is not built or the input is outside the kernel's contract. + +Workspace: one zero-initialized byte buffer per device, ``MAX_ROWS`` rows of +``minimax_m31_topp_renorm_det_ws_bytes()``, kept for the life of the process (CUDA graphs captured with it keep its +address). The kernel leaves it ready for the next launch, so it is zeroed once. ``prebuild`` allocates it outside any +CUDA-graph capture (the target calls it from ``cache_derived_state``, before the executor's warmup and captures); a +call that finds no workspace allocates one when the stream is not capturing, and inside a capture returns None +(stock FlashInfer for that call) rather than allocating from the graph's private pool. +""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.logger import logger + +from ...m31_switches import m31_disabled + +# The largest CTA count per row (the persistent grid is rows x ctas_per_row 1024-thread CTAs, all co-resident). +MAX_CTAS_PER_ROW = 16 + +# The rows the workspace holds and the override takes: the TP8 / TP4 K7 c1 calls (the draft's 7 rows and the +# verify's 8), where GB300 microbenches measured it faster than stock FlashInfer (v1.2: 54.3 vs +# 104.2 us per call at 7 peaked rows, 54.8 vs 105.1 at 8). At 56 / 64 rows v1 was slower than FlashInfer's default +# mode (340 vs 187 / 199 us) and larger shapes are not re-measured, so larger calls keep the stock kernel. +MAX_ROWS = 8 + +# The op when it is built, decided once per process. +_STATE: dict = {} +# Per device index: the persistent workspace and the SM count. +_WORKSPACES: dict = {} +_SMS: dict = {} + + +def _load_op(name: str): + """``torch.ops.trtllm.`` when this build carries it, else None.""" + return getattr(torch.ops.trtllm, name, None) + + +def topp_det_op(): + """The renorm op this process uses, decided once: ``trtllm::minimax_m31_topp_renorm_det_v3`` when it is built + and TOPPDET_V3 is not disabled, else ``trtllm::minimax_m31_topp_renorm_det`` when built, else None.""" + if "op" not in _STATE: + op, ws, v3 = None, 0, False + if not m31_disabled("TOPPDET_V3"): + op = _load_op("minimax_m31_topp_renorm_det_v3") + if op is not None: + v3 = True + ws = int(torch.ops.trtllm.minimax_m31_topp_renorm_det_v3_ws_bytes()) + logger.info("[M3.1] TOPP_DET v3 private op engaged (in-tree)") + if op is None: + op = _load_op("minimax_m31_topp_renorm_det") + ws = ( + int(torch.ops.trtllm.minimax_m31_topp_renorm_det_ws_bytes()) + if op is not None + else 0 + ) + _STATE["op"], _STATE["ws_bytes"], _STATE["v3"] = op, ws, v3 + # v3's candidate-list radix passes (bitwise the same output) unless TOPPDET_T3 is disabled. + _STATE["cand_list"] = v3 and not m31_disabled("TOPPDET_T3") + # V4K8 (a library whose schema carries list_path / wide_hist): the one-sync list path and 8 privatized pass + # histograms, both bitwise the same output, unless TOPPDET_V4 / TOPPDET_K8 is disabled. + v4k8 = v3 and "list_path" in str(op.default._schema) + _STATE["v4k8"] = v4k8 + _STATE["list_path"] = v4k8 and not m31_disabled("TOPPDET_V4") + _STATE["wide_hist"] = v4k8 and not m31_disabled("TOPPDET_K8") + if v3: + logger.info( + "[M3.1] TOPP_DET v3 T3 candidate-list fused passes: " + f"{'on' if _STATE['cand_list'] else 'off'} (cap 4096)" + ) + logger.info( + f"[M3.1] TOPP_DET V4K8: list path {'on' if _STATE['list_path'] else 'off'}, " + f"8 pass histograms {'on' if _STATE['wide_hist'] else 'off'}" + f"{'' if v4k8 else ' (library without V4K8)'}" + ) + return _STATE["op"] + + +def is_v3() -> bool: + """Whether the engaged op is TOPP_DET v3.""" + topp_det_op() + return _STATE["v3"] + + +def ws_bytes_per_row() -> int: + """Workspace bytes per row (0 when the op is not built).""" + topp_det_op() + return _STATE["ws_bytes"] + + +def sm_count(device: torch.device) -> int: + """The device's SM count, cached per device.""" + idx = torch.device(device).index + idx = torch.cuda.current_device() if idx is None else idx + if idx not in _SMS: + _SMS[idx] = torch.cuda.get_device_properties(idx).multi_processor_count + return _SMS[idx] + + +def ctas_per_row(rows: int, sms: int) -> int: + """The largest power of two <= MAX_CTAS_PER_ROW with ctas x rows <= sms (1 past that: one CTA per row needs no + co-residency).""" + g = MAX_CTAS_PER_ROW + while g > 1 and g * rows > sms: + g //= 2 + return g + + +def prebuild(device: Optional[torch.device] = None) -> Optional[torch.Tensor]: + """Allocate (once) the device's zeroed workspace for MAX_ROWS rows; must run outside a CUDA-graph capture. + Returns it, or None when the op is not built or the stream is capturing.""" + if topp_det_op() is None: + return None + dev = ( + torch.device("cuda", torch.cuda.current_device()) + if device is None + else torch.device(device) + ) + if dev.index is None: + dev = torch.device("cuda", torch.cuda.current_device()) + if dev.index not in _WORKSPACES: + if torch.cuda.is_current_stream_capturing(): + return None + _WORKSPACES[dev.index] = torch.zeros( + MAX_ROWS * ws_bytes_per_row(), dtype=torch.uint8, device=dev + ) + return _WORKSPACES[dev.index] + + +class M31ToppDet(OpWrapper): + """FlashInfer's ``top_p_renorm_probs(probs, top_p, is_deterministic=True)`` of fp32 probs [rows, d] with fp32 + per-row top_p [rows], bit for bit on every row with top_p < 1, in ONE launch of rows x ctas_per_row CTAs + (FlashInfer's: five launches, apply on one CTA per row, slow on high-entropy rows). Rows with top_p >= 1 keep every + element (FlashInfer drops tail buckets there in ~43% of rows, in both of its modes); their output is + ``ftz(ftz(p) * norm)``. ``workspace`` is the persistent zeroed buffer of ``prebuild``. ``__call__`` returns None + (the caller keeps the stock call) for a non-fp32, non-contiguous or misaligned input, a width not divisible by 4, + a top_p that is not a contiguous fp32 [rows] tensor on probs' device, more rows than the workspace holds, or when + the op is not built.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The c1 K7 calls a microbench gated bitwise on GB300 (8 row classes x 4 top_p modes x 16 seeds); the + # sm_107 build of the same source is gated by the same MB on VR before the target takes it. + CELLS: tuple[Cell, ...] = ( + Cell(why="the K7 draft's 7 rows at c1 (vocab 200064)", spec=dict(rows=7, d=200064)), + Cell(why="the K7 verify's 8 rows at c1 (vocab 200064)", spec=dict(rows=8, d=200064)), + Cell(why="one row (the smallest grid, 16 CTAs)", spec=dict(rows=1, d=200064)), + ) + + note = """ + Exact by construction: the det histograms are integer (any aggregation order is FlashInfer's), the per-pass + selection is FlashInfer's last-block code on them, and the kept sum is FlashInfer's 1024 per-thread chains into + the same cub BlockReduce. Built with --use_fast_math, as FlashInfer's JIT builds it. + """ + + def __call__( + self, + probs: torch.Tensor, + top_p: torch.Tensor, + workspace: Optional[torch.Tensor], + ctas_per_row: int, + ) -> Optional[torch.Tensor]: + if ( + not isinstance(top_p, torch.Tensor) + or workspace is None + or not probs.is_cuda + or probs.dtype != torch.float32 + or probs.dim() != 2 + or not probs.is_contiguous() + or probs.shape[0] < 1 + or probs.shape[1] % 4 != 0 + or probs.data_ptr() % 16 != 0 + or top_p.dtype != torch.float32 + or top_p.dim() != 1 + or top_p.shape[0] != probs.shape[0] + or not top_p.is_contiguous() + or top_p.device != probs.device + or workspace.device != probs.device + or ctas_per_row < 1 + or 1024 % ctas_per_row != 0 + ): + return None + op = topp_det_op() + if ( + op is None + or workspace.numel() * workspace.element_size() < probs.shape[0] * ws_bytes_per_row() + ): + return None + out = torch.empty_like(probs) + if is_v3(): + if _STATE["v4k8"]: + op( + probs, + top_p, + workspace, + out, + int(ctas_per_row), + True, + _STATE["cand_list"], + _STATE["list_path"], + _STATE["wide_hist"], + ) + else: + op(probs, top_p, workspace, out, int(ctas_per_row), True, _STATE["cand_list"]) + else: + op(probs, top_p, workspace, out, int(ctas_per_row)) + return out + + def reference(self, probs, top_p, workspace=None, ctas_per_row=None) -> torch.Tensor: + """FlashInfer's deterministic renorm (rows with top_p < 1; workspace and ctas_per_row are ignored).""" + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + def is_valid(self, probs, top_p, workspace=None, ctas_per_row=None, **kwargs) -> None: + assert probs.dtype == torch.float32 and probs.dim() == 2, "probs: fp32 [rows, d]" + + +m31_topp_det = M31ToppDet() + + +def top_p_renorm_det(probs: torch.Tensor, top_p: torch.Tensor) -> Optional[torch.Tensor]: + """The sampler's top-p renorm override: the det renorm of probs [rows, d] at per-row top_p, or None (the sampler + runs stock FlashInfer) when the op is not built, rows exceed MAX_ROWS, no workspace can be had (first use inside + a capture) or the input is outside the contract.""" + if not isinstance(top_p, torch.Tensor) or not probs.is_cuda or probs.dim() != 2: + return None + rows = probs.shape[0] + if rows < 1 or rows > MAX_ROWS or topp_det_op() is None: + return None + workspace = prebuild(probs.device) + if workspace is None: + return None + return m31_topp_det(probs, top_p, workspace, ctas_per_row(rows, sm_count(probs.device))) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/__init__.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/__init__.py new file mode 100644 index 000000000000..d25c41dc9557 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Speculative-decoding entries this target alone calls: the Eagle3 verify ops, the capture / fc_norm RMSNorms and +the step-input staging.""" diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_capture_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_capture_norm.py new file mode 100644 index 000000000000..be5aa46fe774 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_capture_norm.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Eagle3 hidden-state capture with the drafter's fc_norm folded in (``m3_spec_tail.capture_norm``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_chunk_norm import rmsnorm_rows + + +class M3CaptureNorm(OpWrapper): + """``dst [M, H] = rmsnorm(bf16(h + r)) * w`` (``r`` optional), the rounding chunk_norm applies, into a capture + buffer slice whose row stride may exceed H. Returns ``dst``.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The target's captured layers (hidden 6144) into the 3-layer capture buffer (row stride 3 x 6144), over the + # verify rows of draft_len 3 (4 .. 32 rows); the residual is present for every captured decoder layer. + CELLS: tuple[Cell, ...] = ( + Cell( + why="4 verify rows of one request, layer slot 0, residual added", + spec=dict(m=4, slot=0, residual=True), + ), + Cell( + why="cap 8 (32 rows) into the last layer slot", spec=dict(m=32, slot=2, residual=True) + ), + Cell( + why="no residual (a capture point without one): plain RMSNorm", + spec=dict(m=8, slot=1, residual=False), + ), + ) + + note = """ + fp32 statistic (block sum, rsqrtf), one bf16 rounding after the bf16 sum h + r; the band states the occasional + one-ulp move against torch's statistic. + """ + + def __call__( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + torch.ops.trtllm.minimax_m3_capture_norm(h, r, w, eps, dst) + return dst + + def reference( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> torch.Tensor: + """bf16 sum, then the normalized row; ``dst`` only says where the op writes.""" + x = (h.float() + r.float()).to(torch.bfloat16) if r is not None else h + return rmsnorm_rows(x, w, eps) + + def is_valid( + self, + h: torch.Tensor, + r: Optional[torch.Tensor], + w: torch.Tensor, + eps: float, + dst: torch.Tensor, + ) -> None: + # The norm weight is read as H dense bf16 values; a strided view (a column of a stacked weight) is read + # past its elements without an error. + assert w.is_contiguous(), "w must be dense [H]" + + +m3_capture_norm = M3CaptureNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_chunk_norm.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_chunk_norm.py new file mode 100644 index 000000000000..2f71f1b381b1 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_chunk_norm.py @@ -0,0 +1,74 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 fc_norm over the captured layers in one launch: per-chunk RMSNorm of the concatenated hidden states +(``m3_spec_tail.chunk_norm``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3ChunkNorm(OpWrapper): + """``dst[r, c*H:(c+1)*H] = rmsnorm(src[r, c*H:(c+1)*H]) * w[c]`` for the C chunks of ``w [C, H]``; ``src`` may + have a row stride wider than C * H (the capture buffer). Writes ``dst [M, C*H]`` and returns it.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The Eagle3 capture buffer of the target (3 captured layers x hidden 6144, buffer rows wider than the layers + # when the final hidden is also captured), over the verify rows of draft_len 3: 4 / 8 / 32 rows. + CELLS: tuple[Cell, ...] = ( + Cell( + why="one request's 4 verify rows, dense [4, 3 x 6144]", + spec=dict(m=4, c=3, h=6144, extra_cols=0), + ), + Cell( + why="two requests' 8 verify rows (b2) from the capture buffer, dense [8, 3 x 6144]", + spec=dict(m=8, c=3, h=6144, extra_cols=0), + ), + Cell( + why="cap 8 (32 rows) read from the capture buffer with a fourth (final-hidden) slot in the row stride", + spec=dict(m=32, c=3, h=6144, extra_cols=6144), + ), + ) + + note = """ + fp32 statistic per chunk (block sum, rsqrtf), one bf16 rounding: an output moves by one bf16 ulp now and then + against torch's statistic; the cells' band states it. + """ + + def __call__( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + torch.ops.trtllm.minimax_m3_chunk_norm(src, w, eps, dst, pdl) + return dst + + def reference( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ): + """Each chunk normalized on its own; ``dst`` only says where the op writes, ``pdl`` shapes the launch.""" + c, h = w.shape + x = src[:, : c * h].reshape(src.shape[0], c, h) + return rmsnorm_rows(x, w, eps).reshape(src.shape[0], c * h) + + def is_valid( + self, src: torch.Tensor, w: torch.Tensor, eps: float, dst: torch.Tensor, pdl: bool = True + ) -> None: + c, h = w.shape + assert w.is_contiguous(), "w must be dense [C, H] (chunk c read at c * H)" + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + + +m3_chunk_norm = M3ChunkNorm() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_input.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_input.py new file mode 100644 index 000000000000..4fe14d538a5f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_input.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 draft pass's input in one launch: embedding lookup, both RMSNorms and the concat +(``m3_spec_tail.draft_input``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Largest hidden size the kernel's register tile holds (4 x 256 threads x 8 bf16). +MAX_HIDDEN = 8192 + + +def rmsnorm_rows(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor: + """bf16(x * rsqrt(mean(x^2) + eps) * w) per row, fp32 in between ((x * r) * w, as the kernels multiply).""" + xf = x.float() + r = torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + return (xf * r * weight.float()).to(torch.bfloat16) + + +class M3DraftInput(OpWrapper): + """``out [M, 2H] = [rmsnorm(table[ids]) * w_embed, rmsnorm(hidden[rows]) * w_hidden]`` and ``residual [M, H] = + hidden[rows]`` (rows default to 0..M-1) in one launch; ids outside the table embed to zero (the masked lookup of + a vocab-local embedding). Returns ``(out, residual)``.""" + + ARCHS = frozenset({Arch.SM_107}) + + # Every pass of the fused draft loop (modeling's DRAFT_INPUT) at draft_len 3, drafter hidden 6144, eps 1e-6: the + # first pass of 1..8 requests (4..32 rows, the carried hidden read in place) and the later passes of 1..8 requests + # (one row each; the second pass gathers the last accepted rows from the first pass's 4 rows per request). + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"first pass of {n} request(s): {4 * n} rows, hidden read in place", + spec=dict(m=4 * n, h=6144, gather=False, invalid_id=False), + ) + for n in range(1, 9) + ), + *( + Cell( + why=f"second pass of {n} request(s): the last accepted rows gathered from {4 * n} rows", + spec=dict(m=n, h=6144, gather=True, invalid_id=False), + ) + for n in range(1, 9) + ), + Cell( + why="third pass of 3 requests: one row each, hidden read in place", + spec=dict(m=3, h=6144, gather=False, invalid_id=False), + ), + Cell( + why="one row with an out-of-table id: the embedding half is zero", + spec=dict(m=1, h=6144, gather=False, invalid_id=True), + ), + ) + + note = """ + The arithmetic is the in-tree RMSNorm's (fp32 statistic, (x * rsqrt) * w, one bf16 rounding); the statistic's + summation order differs from FlashInfer's, which moves an output by one bf16 ulp now and then (the test's band, + also held against the in-tree modules). The out-of-table id is the masked lookup of a vocab-local embedding; the + drafter's embedding is the whole table, so its ids are always in range. + """ + + def __call__( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + torch.ops.trtllm.minimax_m3_draft_input( + ids, table, hidden, rows, w_embed, w_hidden, eps_embed, eps_hidden, out, residual, pdl + ) + return out, residual + + def reference( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + """The two normalized halves (fp32 statistic, one bf16 rounding) and the gathered hidden rows. ``out`` / + ``residual`` only say where the op writes; ``pdl`` only shapes the launch.""" + valid = (ids >= 0) & (ids < table.shape[0]) + emb = rmsnorm_rows(table[torch.where(valid, ids, 0).long()], w_embed, eps_embed) + emb = torch.where(valid[:, None], emb, torch.zeros_like(emb)) + hid = hidden[rows.long()] if rows is not None else hidden[: ids.shape[0]] + ref = torch.cat([emb, rmsnorm_rows(hid, w_hidden, eps_hidden)], dim=-1) + return ref, (hid.clone() if residual is not None else None) + + def is_valid( + self, + ids: torch.Tensor, + table: torch.Tensor, + hidden: torch.Tensor, + rows: Optional[torch.Tensor], + w_embed: torch.Tensor, + w_hidden: torch.Tensor, + eps_embed: float, + eps_hidden: float, + out: torch.Tensor, + residual: Optional[torch.Tensor], + pdl: bool = True, + ) -> None: + h = table.shape[1] + # Rows are read as hidden + rows[r] * H and table + id * H: both must be dense H-wide rows. + assert table.is_contiguous() and hidden.is_contiguous(), ( + "table / hidden must be dense [*, H]" + ) + if rows is not None: + assert bool(((rows >= 0) & (rows < hidden.shape[0])).all()), "rows must index hidden" + else: + assert hidden.shape[0] >= ids.shape[0], "hidden must hold a row per id" + # Vector loads of 16 bytes: H must be a multiple of 8 within the register tile, and the weights aligned. + assert h % 8 == 0 and h <= MAX_HIDDEN, "H must be a multiple of 8, at most 8192" + assert residual is None or ( + residual.is_contiguous() and residual.shape == (ids.shape[0], h) + ), "residual must be contiguous [M, H]" + + +m3_draft_input = M3DraftInput() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_silu.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_silu.py new file mode 100644 index 000000000000..b2487a213990 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_silu.py @@ -0,0 +1,58 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The Eagle3 drafter MLP's plain SwiGLU as one C++ launch (modeling's ``DRAFT_SILU_CPP``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3DraftSilu(OpWrapper): + """``silu(g) * u`` of a bf16 [rows, 2d] gate | up tensor, rounded once to bf16: + trtllm::minimax_m3_sm107_tp8_silu_and_mul (minimaxM3Sm107Tp8/siluAndMul.cu), bit for bit the Triton + ``torch.ops.trtllm.silu_and_mul`` at its defaults (no limit, alpha 1, beta 0), the activation the drafter's GatedMLP + runs. Contiguous, 16-byte aligned rows and d a multiple of 8; ``__call__`` returns None otherwise (the caller runs + the Triton op).""" + + ARCHS = frozenset({Arch.SM_107}) + + # The TP8 drafter MLP shard: intermediate 18432 / 8 = 2304, gate | up 4608 wide, at the drafter's decode rows. + CELLS: tuple[Cell, ...] = ( + Cell(why="the TP8 drafter MLP at 1 row", spec=dict(rows=1, d=2304)), + Cell(why="the TP8 drafter MLP at 4 rows (1 request x 4 tokens)", spec=dict(rows=4, d=2304)), + Cell(why="the TP8 drafter MLP at 8 rows", spec=dict(rows=8, d=2304)), + ) + + note = """ + Elementwise, one rounding: bit for bit the Triton kernel (the same PTX instruction sequence), checked over every + bf16 gate and every bf16 up pattern. + """ + + def __call__(self, x: torch.Tensor) -> Optional[torch.Tensor]: + if ( + x.dtype != torch.bfloat16 + or x.dim() != 2 + or not x.is_cuda + or not x.is_contiguous() + or x.shape[1] % 16 != 0 + or x.data_ptr() % 16 != 0 + or x.shape[0] > 65535 + ): + return None + out = torch.empty((x.shape[0], x.shape[1] // 2), dtype=x.dtype, device=x.device) + torch.ops.trtllm.minimax_m3_sm107_tp8_silu_and_mul(x, out) + return out + + def reference(self, x: torch.Tensor) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul(x) + + def is_valid(self, x: torch.Tensor) -> None: + assert x.dtype == torch.bfloat16 and x.dim() == 2, "x: bf16 [rows, 2d]" + + +m3_draft_silu = M3DraftSilu() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_step.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_step.py new file mode 100644 index 000000000000..e63389ffc6eb --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_draft_step.py @@ -0,0 +1,440 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One pass of the fused greedy Eagle3 draft loop: the drafter's vocab-sharded BF16 head (streamed from its lossless +12-bit coding, ``ops.gemm.m3_target_head``, or as BF16 fragments), the cross-rank argmax and the loop's per-pass +bookkeeping in one kernel (``m3_draft_step.draft_step``).""" + +from __future__ import annotations + +import functools +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import ( + Arch, + Cell, + OpWrapper, + true_fp32_matmul, +) + +from ..gemm.m3_target_head import decode as decode_coded +from .m3_shard_accept import exchange_tokens, local_argmax, make_keys + +#: Requests (head rows) per pass, and exchange rows per call half of the workspace. +MAX_TOKENS = 8 +#: 16-row weight tiles one CTA can own (the kernel's shared-memory accumulator). +MAX_TILES_PER_CTA = 32 +KERNEL_WARPS = 16 # warps per CTA (draftStep.h kWarps) + + +@functools.lru_cache(maxsize=64) +def max_tile_pieces(n_tiles: int, num_ctas: int, chunks_per_tile: int) -> int: + """The most warps that stream one weight tile under the kernel's whole-tile split: CTA c owns tiles + [c * n / grid, (c + 1) * n / grid), and its (tile, chunk) range is cut into KERNEL_WARPS equal contiguous + segments. The shared-memory atomics add a tile's pieces in arrival order: deterministic for at most 2 pieces (two + addends from 0 commute), not for more (fixed_order then sums them in warp order).""" + worst = 1 + for c in range(num_ctas): + lo, hi = c * n_tiles // num_ctas, (c + 1) * n_tiles // num_ctas + span = (hi - lo) * chunks_per_tile + bounds = [w * span // KERNEL_WARPS for w in range(KERNEL_WARPS + 1)] + for t in range(hi - lo): + a, b = t * chunks_per_tile, (t + 1) * chunks_per_tile + worst = max( + worst, sum(1 for w in range(KERNEL_WARPS) if bounds[w] < b and bounds[w + 1] > a) + ) + return worst + + +#: Draft length of the fused loop, and the drafter's TP8 head shard (200064-id vocabulary) and hidden size. +DRAFT_LEN = 3 +DRAFT_SHARD = 25008 +HIDDEN = 6144 + + +def bf16_fragments(weight: torch.Tensor) -> torch.Tensor: + """BF16 [N, K] -> the same values in A-fragment order [N / 16, K / 16, 256]. + + One lane's 16 bytes of a 16-row tile's k16 step are the m16n8k16 A fragment: element (row h * 8 + g, k ks * 16 + + hk * 8 + jj * 2 + e) goes to [tile][ks][g][jj][hk][h][e] (lane = g * 4 + jj). + """ + n, k = weight.shape + assert n % 16 == 0 and k % 16 == 0 and weight.dtype == torch.bfloat16, (n, k, weight.dtype) + q = weight.view(n // 16, 2, 8, k // 16, 2, 4, 2).permute(0, 3, 2, 5, 4, 1, 6).contiguous() + return q.view(n // 16, k // 16, 256) + + +def from_fragments(wf: torch.Tensor) -> torch.Tensor: + """The inverse of ``bf16_fragments``: [N / 16, K / 16, 256] -> [N, K].""" + tiles, ks, _ = wf.shape + return ( + wf.view(tiles, ks, 8, 4, 2, 2, 2).permute(0, 5, 2, 1, 4, 3, 6).reshape(tiles * 16, ks * 16) + ) + + +def head_weight( + wf: torch.Tensor, + k: int, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The BF16 head shard [N, K] the kernel multiplies by: the fragment-ordered copy, or (``exc_off`` given) the + 12-bit coded stream with its exception lists decoded.""" + if exc_off is None: + return from_fragments(wf) + n = (exc_off.numel() - 1) * 16 + return decode_coded(wf, exp_base, exc_off, exc_pos, exc_w, n, k) + + +def shard_logits( + x: torch.Tensor, + wf: torch.Tensor, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, +) -> torch.Tensor: + """The in-tree head's BF16 logits of the rows x [M, K] against the shard (``head_weight``): the fp32 product, + rounded once to BF16 (the BF16 GEMM's output), as fp32.""" + w = head_weight(wf, x.shape[1], exc_off, exc_pos, exc_w, exp_base) + with true_fp32_matmul(): + logits = x.float() @ w.float().t() + return logits.to(torch.bfloat16).float() + + +class M3DraftStep(OpWrapper): + """One fused greedy draft pass over ``rows <= 8`` requests (one head row each). + + Head input ``x[x_rows[b]]`` (pass 0: the last accepted verify row of each request) or ``x[b]``. The head is this + rank's BF16 lm_head shard in fragment order (``m3_draft_step.bf16_fragments``, global ids ``vocab_offset ..``), + or with ``exc_off`` / ``exc_pos`` / ``exc_w`` / ``exp_base`` its lossless 12-bit coding (``wf`` = the uint8 stream + of ``m3_target_head.encode_chunked``, which must round-trip to the BF16 shard); each logit is the fp32 dot product + rounded to BF16 and the row's token is the global argmax over all ranks' + shards (ties, -0 / +0 included, to the lowest global id), exchanged through the multicast slots + ``[2][8][world]``. The last CTA then writes ``draft_out[b, step]`` and ``ids_next[b]`` (the token), ``pos_next[b] + = pos_src[x_rows[b] or b] + 1``, and advances the draft KV lengths as the in-tree draft loop does between passes + (pass 0 with ``num_acc``: ``kv_lens -= draft_len - num_acc`` and ``seq_lens = 1``; later passes: ``kv_lens += + 1``). ``out_keys`` (optional) receives each row's winning key. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # Every pass of the fused draft loop (modeling's DRAFT_STEP): draft_len 3 at the TP8 GEN worker's 1..2 requests, + # the drafter's full-vocabulary BF16 head (25008 rows per rank of 8, hidden 6144; 1563 tiles, 8 per CTA on 212 + # SMs). Pass 0 reads the verify rows' hidden (4 per request) through x_rows and rolls the KV back; the later passes + # read one row per request. BF16 fragments only: on sm_107 the 12-bit coded head is slower than the BF16 stream + # (modeling's DRAFT_HEAD_CODED is False). + CELLS: tuple[Cell, ...] = ( + *( + Cell( + why=f"pass 0 at {n} request(s): head rows gathered from {4 * n} verify rows, KV rollback, seq_lens = 1", + spec=dict( + nreq=n, + draft_len=DRAFT_LEN, + step=0, + rank=7 * (n - 1), + world=8, + vs=DRAFT_SHARD, + k=HIDDEN, + coded=False, + ), + ) + for n in (1, 2) + ), + Cell( + why="pass 1 at two requests: one row each, kv_lens + 1, rank 6's ids", + spec=dict( + nreq=2, + draft_len=DRAFT_LEN, + step=1, + rank=6, + world=8, + vs=DRAFT_SHARD, + k=HIDDEN, + coded=False, + ), + ), + Cell( + why="last pass (step 2) at one request, rank 0", + spec=dict( + nreq=1, + draft_len=DRAFT_LEN, + step=2, + rank=0, + world=8, + vs=DRAFT_SHARD, + k=HIDDEN, + coded=False, + ), + ), + Cell( + why="pass 2 at 2 requests with exact cross-rank ties (the lower global id wins)", + spec=dict( + nreq=2, + draft_len=DRAFT_LEN, + step=2, + rank=3, + world=8, + vs=DRAFT_SHARD, + k=HIDDEN, + ties=True, + coded=False, + ), + ), + ) + + note = """ + The head is the in-tree BF16 head's arithmetic (BF16 weights and activations, fp32 accumulation, a BF16 logit; + the coded head's weight is the BF16 shard bit for bit; its exceptions decode to +0 in the MMAs and their exact + products are added at the tile's flush) + and the argmax is the in-tree one over BF16 logits, so a token differs from the in-tree path's only where the fp32 + sum, accumulated in another order, rounds to the other side of a BF16 boundary and that decides the row's maximum. + The cells' rows are decidable and the peers' keys sit 0.25 away (or tie exactly); the edge tests (forced BF16 + ties, -0 / +0, top-2 one fp32 step apart that round to one BF16 value, adjacent BF16 values) use integer data + whose fp32 sums are exact in any order and hold the kernel to the in-tree path bit for bit. The cross-rank + exchange is certified on one device with the peers' keys written into the slots (see m3_shard_accept). + """ + + def __call__( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + tile_lo: int = 0, + tile_hi: int = 0, + total_ctas: int = 0, + fold: Optional[tuple] = None, + fixed_order: bool = False, + ) -> None: + # fold: (the 14 fold-tail tensors in buildParams' order, rows, tokens per step) -- the step's forward-tail store + # and next-step staging, run by the pass's last CTA after its tokens (modeling's DRAFT_STEP_FOLD_TAIL). + tensors, fold_rows, fold_t = ( + (list(fold[0]), int(fold[1]), int(fold[2])) if fold is not None else (None, 0, 0) + ) + torch.ops.trtllm.minimax_m3_sm107tp8_draft_step( + x, + x_rows, + rows, + wf, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft_out, + step, + ids_next, + pos_src, + pos_next, + kv_lens, + num_acc, + seq_lens, + draft_len, + num_ctas, + pdl, + out_keys, + exc_off, + exc_pos, + exc_w, + exp_base, + tile_lo, + tile_hi, + total_ctas, + None, + None, + tensors, + fold_rows, + fold_t, + fixed_order, + ) + + def reference( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + tile_lo: int = 0, + tile_hi: int = 0, + total_ctas: int = 0, + fold: Optional[tuple] = None, + fixed_order: bool = False, + ) -> dict[str, torch.Tensor]: + """Every buffer the pass writes, computed on copies: "logits" (this rank's BF16 logits as fp32), "local_keys" + (this rank's key per row), "keys" (the winning key), "tokens", "draft_out", "ids_next", "pos_next", "kv_lens" + and "seq_lens". + + Reads ``ws_uc``, ``counters`` and ``kv_lens`` before the call. ``keys``, ``ws_mc_ptr``, ``num_ctas``, ``pdl``, + ``out_keys`` and ``fixed_order`` do not change the result; ``fold`` adds the fold tail's writes (m3_fold_tail), + not these. + """ + h = x[x_rows.long()] if x_rows is not None else x[:rows] + logits = shard_logits(h, wf, exc_off, exc_pos, exc_w, exp_base) + best, idx = local_argmax(logits, vocab_offset) + local_keys = make_keys(best, idx) + tokens = exchange_tokens(local_keys, ws_uc, counters, rank, world, MAX_TOKENS) + half = int(counters[1].item()) & 1 + slots = ws_uc.view(torch.int64)[ + half * MAX_TOKENS * world : (half * MAX_TOKENS + rows) * world + ] + slots = slots.view(rows, world).clone() + slots[:, rank] = local_keys + sign = torch.iinfo(torch.int64).min + win = (slots ^ sign).max(dim=1).values ^ sign + d_out = draft_out.clone() + d_out[:rows, step] = tokens + src = x_rows.long() if x_rows is not None else torch.arange(rows, device=x.device) + lens = kv_lens.clone() + sl = seq_lens.clone() if seq_lens is not None else None + if step == 0 and num_acc is not None: + lens[:rows] -= draft_len - num_acc[:rows] + sl[:rows] = 1 + elif step > 0: + lens[:rows] += 1 + out = { + "logits": logits, + "local_keys": local_keys, + "keys": win, + "tokens": tokens, + "draft_out": d_out, + "ids_next": tokens.clone(), + "pos_next": pos_src[src] + 1, + "kv_lens": lens, + } + if sl is not None: + out["seq_lens"] = sl + return out + + def is_valid( + self, + x: torch.Tensor, + x_rows: Optional[torch.Tensor], + rows: int, + wf: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft_out: torch.Tensor, + step: int, + ids_next: torch.Tensor, + pos_src: torch.Tensor, + pos_next: torch.Tensor, + kv_lens: torch.Tensor, + num_acc: Optional[torch.Tensor], + seq_lens: Optional[torch.Tensor], + draft_len: int, + num_ctas: int, + pdl: bool = True, + out_keys: Optional[torch.Tensor] = None, + exc_off: Optional[torch.Tensor] = None, + exc_pos: Optional[torch.Tensor] = None, + exc_w: Optional[torch.Tensor] = None, + exp_base: int = 0, + tile_lo: int = 0, + tile_hi: int = 0, + total_ctas: int = 0, + fold: Optional[tuple] = None, + fixed_order: bool = False, + ) -> None: + assert not bool(keys[:rows].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + # The kernel reads the head row x + x_rows[b] * K and the position pos_src[x_rows[b]] with no bound. + if x_rows is not None: + assert bool(((x_rows[:rows] >= 0) & (x_rows[:rows] < x.shape[0])).all()), ( + "x_rows must index x" + ) + assert bool((x_rows[:rows] < pos_src.shape[0]).all()), ( + "pos_src must cover the head rows" + ) + else: + assert pos_src.shape[0] >= rows, "pos_src must cover the head rows" + # Each CTA accumulates at most MAX_TILES_PER_CTA tiles in shared memory. + tiles = wf.shape[0] if exc_off is None else exc_off.numel() - 1 + assert -(-tiles // num_ctas) <= MAX_TILES_PER_CTA, "too many weight tiles per CTA" + # D-FIX: the atomics' arrival-order sum is deterministic only while no tile takes more than 2 warp pieces. + assert ( + fixed_order + or max_tile_pieces(tiles, num_ctas, x.shape[1] // (16 if exc_off is None else 64)) <= 2 + ), ( + "a tile takes more than 2 warp pieces on this grid: the atomics' sum is order-dependent (fixed_order)" + ) + if exc_off is not None: + # The exceptions are read per tile in list order with no bound against the coded stream. + assert int(exc_off[-1].item()) == exc_pos.numel(), ( + "exc_off must end at the exception count" + ) + assert bool((exc_off[1:] >= exc_off[:-1]).all()), "exc_off must be non-decreasing" + # A NaN logit maps to a key above every finite one and wins the row silently. + h = x[x_rows.long()] if x_rows is not None else x[:rows] + assert not bool(torch.isnan(h).any()), "NaN head inputs win the argmax" + if fold is not None: + assert len(fold) == 3 and len(fold[0]) == 14, ( + "fold is (the 14 fold-tail tensors, rows, tokens per step)" + ) + if num_acc is not None: + na = num_acc[:rows] + assert bool(((na >= 1) & (na <= draft_len + 1)).all()), ( + "num_acc counts must lie in [1, draft_len + 1]" + ) + + +m3_draft_step = M3DraftStep() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_force_vec.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_force_vec.py new file mode 100644 index 000000000000..2001bf183b54 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_force_vec.py @@ -0,0 +1,136 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-request forced accepted counts of a fractional forced acceptance, drawn as the in-tree worker draws them.""" + +from __future__ import annotations + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from ...kernels.m3_force_vec import force_vec_kernel, force_vec_req_kernel + + +class M3ForceVec(OpWrapper): + """Bump ``counter`` (int64 [1]) by one, then for each of the ``batch_size`` requests ``b`` write + ``min(base_total + (pool[(c * counter_stride + b * slot_stride) & pool_mask] < frac), max_total)`` into ``out`` + (int32), ``c`` the bumped counter: the in-tree ``_apply_force_accepted_tokens`` draw over its fixed-seed fp32 + ``pool``. ``block`` is the launch width (a power of two >= batch_size; 16, 32 or 64 are compiled at warmup). + + Only the forced-acceptance testing knob (``TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`` with a fractional part) + runs it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, fraction 0.5", spec=dict(batch=1, block=16, frac=0.5)), + Cell(why="eight requests, fraction 0.3", spec=dict(batch=8, block=16, frac=0.3)), + ) + + note = """ + Integer arithmetic and one fp32 comparison: `reference` must match exactly. + """ + + def __call__( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + force_vec_kernel[(1,)]( + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + BLOCK=block, + ) + return out + + def reference( + self, + pool, + counter, + out, + batch_size, + base_total, + max_total, + frac, + counter_stride, + slot_stride, + pool_mask, + block, + ): + """The counts (a new tensor; ``counter`` and ``out`` are not written).""" + c = int(counter.item()) + 1 + slots = torch.arange(batch_size, dtype=torch.int64, device=pool.device) + idx = (c * counter_stride + slots * slot_stride) & pool_mask + r = pool[idx] + return torch.clamp(base_total + (r < frac).to(torch.int32), max=max_total).to(torch.int32) + + +m3_force_vec = M3ForceVec() + + +class M3ForceVecReq(OpWrapper): + """For each of the ``batch_size`` requests ``b`` write + ``min(base_total + (pool[(keys[b] * key_stride) & pool_mask] < frac), max_total)`` into ``out`` (int32): + one draw per request at its own key (its KV length at the verify), no shared counter. ``block`` is the launch + width (a power of two >= batch_size; 16, 32 or 64 are compiled at warmup). + + Only the forced-acceptance testing knob (``TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`` with a fractional part) + runs it. + """ + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="one request, fraction 0.5", spec=dict(batch=1, block=16, frac=0.5)), + Cell(why="two requests, fraction 0.3", spec=dict(batch=2, block=16, frac=0.3)), + ) + + note = """ + Integer arithmetic and one fp32 comparison: `reference` must match exactly. + """ + + def __call__( + self, pool, keys, out, batch_size, base_total, max_total, frac, key_stride, pool_mask, block + ): + force_vec_req_kernel[(1,)]( + pool, + keys, + out, + batch_size, + base_total, + max_total, + frac, + key_stride, + pool_mask, + BLOCK=block, + ) + return out + + def reference( + self, pool, keys, out, batch_size, base_total, max_total, frac, key_stride, pool_mask, block + ): + """The counts (a new tensor; ``out`` is not written).""" + k = keys[:batch_size].to(torch.int64) + r = pool[(k * key_stride) & pool_mask] + return torch.clamp(base_total + (r < frac).to(torch.int32), max=max_total).to(torch.int32) + + +m3_force_vec_req = M3ForceVecReq() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_sampled_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_sampled_accept.py new file mode 100644 index 000000000000..d86ee7e23625 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_sampled_accept.py @@ -0,0 +1,282 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Sampled (temperature + top-p) Eagle3 verification over the TP-sharded vocabulary in one kernel +(``m3_sampled_accept.sampled_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +from .m3_shard_accept import strict_accept + +#: Verify rows per call. +MAX_ROWS = 48 +#: The fused sampling kernel's 1 / T guard. +TEMP_EPSILON = 1e-6 + + +def nucleus_probs( + logits: torch.Tensor, temperatures: torch.Tensor, top_ps: torch.Tensor +) -> torch.Tensor: + """Per-row sampling distribution [rows, Vs] (fp32): w = exp(x / (T + 1e-6) - max), kept iff the mass strictly + above its weight is at most top_p * sum(w) (ties included), renormalized over the kept tokens.""" + inv = 1.0 / (temperatures.float() + TEMP_EPSILON) + x = logits.float() * inv[:, None] + w = torch.exp(x - x.max(dim=1, keepdim=True).values).double() + z = w.sum(dim=1, keepdim=True) + order = torch.sort(w, dim=1, descending=True) + sw = order.values + # mass strictly above each sorted weight: the cumulative sum before the first entry of its tie group + csum = torch.cumsum(sw, dim=1) - sw + first = torch.ones_like(sw, dtype=torch.bool) + first[:, 1:] = sw[:, 1:] != sw[:, :-1] + group = torch.cumsum(first.long(), dim=1) - 1 + starts = torch.zeros_like(csum) + starts.scatter_reduce_(1, group, csum, reduce="amin", include_self=False) + above = torch.gather(starts, 1, group) + keep_sorted = above <= top_ps.double()[:, None] * z + keep = torch.zeros_like(keep_sorted) + keep.scatter_(1, order.indices, keep_sorted) + p = torch.where(keep, w, torch.zeros_like(w)) + return (p / p.sum(dim=1, keepdim=True)).float() + + +class M3SampledAccept(OpWrapper): + """Sampled verification of ``rows = nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard. + + Each row draws one token from its top-p nucleus of softmax(logits / T) over the full (all-rank) vocabulary by + distributed rejection sampling (Philox keyed by the row's seed / offset, the row, the attempt and the CTA), then + the strict acceptance runs against ``draft`` as in ``m3_shard_accept``: ``acc_tokens``, ``num_acc``, + ``gather_ids``, optional ``next_new[:, 0]``. ``tokens`` receives the sampled token per row; ``extra_attempts`` + accumulates exchange rounds beyond the second. The exchange goes through the multicast workspace (two call + halves of ``minimax_m3_sampled_accept_ws_words(world)`` words) with the (finished rows, call) ``counters``. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # draft_len 3 (T = 4 rows per request) at the TP8 GEN worker's batch: 1 / 2 requests (4 / 8 rows, 8 CTAs per + # row), and the 48-row maximum (4 CTAs per row, 12.5 KB of staged logits per CTA); the 25008-column target shard + # (vocab 200064 / 8). Single rank (see note). + CELLS: tuple[Cell, ...] = ( + Cell( + why="b1 (4 rows), top_p 0.8 over a 12-token head: the draws follow the nucleus distribution", + spec=dict( + nreq=1, vs=25008, vocab_offset=0, check="distribution", top_p=0.8, temperature=0.7 + ), + ), + Cell( + why="b2 (8 rows): top_p ~ 0 keeps only the argmax, token exact", + spec=dict( + nreq=2, + vs=25008, + vocab_offset=7 * 25008, + check="argmax", + top_p=1e-7, + temperature=1.0, + ), + ), + Cell( + why="b2 (8 rows) with top_p = 1 (no cut: the first draw stands), next_new and forced counts", + spec=dict( + nreq=2, + vs=25008, + vocab_offset=7 * 25008, + check="distribution", + top_p=1.0, + temperature=1.0, + force="vec", + ), + ), + Cell( + why="AgentX sampling (T 1.0, top_p 0.95) at b1 with forced counts: the draws follow the nucleus", + spec=dict( + nreq=1, + vs=25008, + vocab_offset=0, + check="distribution", + top_p=0.95, + temperature=1.0, + force="total", + ), + ), + Cell( + why="48 rows (4 CTAs per row): the second launch geometry, nucleus distribution at top_p 0.5", + spec=dict( + nreq=12, + vs=25008, + vocab_offset=0, + check="distribution", + top_p=0.5, + temperature=1.3, + force="total", + ), + ), + ) + + note = """ + What is sampled is gated statistically, the rest exactly. Every drawn token must lie in the reference nucleus; + the empirical distribution over many calls (distinct Philox offsets) must sit within a stated total-variation + band of the reference's; the acceptance outputs must equal the strict acceptance of the drawn tokens exactly. + The Philox stream itself is not reproduced in torch: the kernel's draw uses the hardware exp / log + approximations (__expf, __logf) and a CTA-order inverse CDF, which torch does not reproduce bit for bit. + + Certified at world 1 only. The decision is replicated from every rank's per-round records, which another rank + computes from its own shard on the fly, so peers cannot be stood in for on one device; the cross-rank path + needs the multi-device job and is not covered here. Rows with top-k or min-p must not reach this op (it applies + neither); is_valid states it. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + ) -> None: + torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept( + logits, + vocab_offset, + temperatures, + top_ps, + seeds, + offsets, + ws_uc, + ws_mc_ptr, + counters, + tokens, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + force_vec, + extra_attempts, + use_pdl, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + ) -> dict: + """{"probs": the per-row nucleus distribution over this shard [rows, Vs] (global id = vocab_offset + + column), "accept": tokens -> the strict acceptance outputs}. The draw itself is random; ``seeds`` / + ``offsets`` select it and do not change the distribution. World 1: this shard is the whole vocabulary.""" + rows = logits.shape[0] + probs = nucleus_probs(logits, temperatures[:rows], top_ps[:rows]) + + def accept(drawn: torch.Tensor) -> dict[str, torch.Tensor]: + return strict_accept(drawn, draft, batch_idx, force_total, next_new, force_vec) + + return {"probs": probs, "accept": accept} + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + temperatures: torch.Tensor, + top_ps: torch.Tensor, + seeds: torch.Tensor, + offsets: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + counters: torch.Tensor, + tokens: torch.Tensor, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], + extra_attempts: Optional[torch.Tensor], + use_pdl: bool = True, + top_ks: Optional[torch.Tensor] = None, + min_ps: Optional[torch.Tensor] = None, + ) -> None: + rows = logits.shape[0] + # top-k and min-p are not applied: a row that has them gets a token outside its real sampling set. + if top_ks is not None: + k = top_ks[:rows] + assert bool(((k <= 0) | (k >= logits.shape[1] * world)).all()), ( + "top-k rows must take the generic path" + ) + if min_ps is not None: + assert not bool((min_ps[:rows] > 0).any()), "min-p rows must take the generic path" + # Rows are request-major verify rows: tokens[b * (D + 1) + k]; a mismatched layout pairs drafts with the + # wrong rows. + assert rows == draft.shape[0] * (draft.shape[1] + 1), ( + "logits rows must be nreq * (draft_len + 1)" + ) + # Per-row parameters are read at the row index, unchecked. + for t in (temperatures, top_ps, seeds, offsets): + assert t.numel() >= rows and t.is_contiguous(), ( + "per-row parameters must cover every row" + ) + assert temperatures.dtype == torch.float32 and top_ps.dtype == torch.float32, ( + "fp32 sampling parameters" + ) + assert seeds.dtype == torch.int64 and offsets.dtype == torch.int64, ( + "int64 Philox seeds / offsets" + ) + # The finished-row counter elects the CTA that runs the acceptance. + assert int(counters[0].item()) == 0, ( + "counters[0] (finished rows) must be zero between calls" + ) + + +m3_sampled_accept = M3SampledAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_sampled_accept_ws_words.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_sampled_accept_ws_words.py new file mode 100644 index 000000000000..3c3974cca0d8 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_sampled_accept_ws_words.py @@ -0,0 +1,59 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Size of the sampled-verify exchange workspace (``m3_sampled_accept.SampledAcceptWorkspace.create``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Exchange rounds per row, verify rows per call, CTAs per row (at most), uint64 words per record. +MAX_ATTEMPTS = 16 +MAX_ROWS = 48 +MAX_SPLITS = 8 +RECORD_WORDS = 4 +#: SA-1R (one_round_k 9 / 82): 9 listed bf16 values ride each record, 3 per extra word. +LIST_WORDS = {0: 0, 9: 3, 82: 3} + + +class M3SampledAcceptWsWords(OpWrapper): + """uint64 words of the multicast workspace ``minimax_m3_sm107tp8_sampled_accept`` exchanges through for + ``world`` ranks: two call halves of ``[attempts][rows][world * splits][record words]``.""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="the tp8 target's workspace", spec=dict(world=8)), + Cell(why="a single rank (local workspace, no multicast)", spec=dict(world=1)), + Cell( + why="the tp8 target's workspace with SA-1R pooled (82)", + spec=dict(world=8, one_round_k=82), + ), + Cell( + why="the tp8 target's workspace with SA-1R top-9 (9)", spec=dict(world=8, one_round_k=9) + ), + ) + + note = """ + A host computation (no device work); it is an op so the workspace size and the kernel's indexing come from one + definition. + """ + + def __call__(self, world: int, one_round_k: int = 0) -> int: + return torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept_ws_words(world, one_round_k) + + def reference(self, world: int, one_round_k: int = 0) -> int: + """Two halves of attempts x rows x (world x splits) records x record words.""" + return ( + 2 + * MAX_ATTEMPTS + * MAX_ROWS + * world + * MAX_SPLITS + * (RECORD_WORDS + LIST_WORDS[one_round_k]) + ) + + +m3_sampled_accept_ws_words = M3SampledAcceptWsWords() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_shard_accept.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_shard_accept.py new file mode 100644 index 000000000000..6df70b3f467b --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_shard_accept.py @@ -0,0 +1,266 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Greedy Eagle3 verification from this rank's logits shard: per-row argmax, the cross-rank (value, index) exchange +and the strict acceptance in one kernel (``m3_shard_accept.shard_accept``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + +#: Verify rows per call, and exchange rows per call half of its workspace. +SHARD_ROWS = 48 + +_LOW32 = 0xFFFFFFFF + + +def ordered_bits(v: torch.Tensor) -> torch.Tensor: + """The kernel's order-preserving uint32 image of fp32 values, as int64 (-0.0 maps to +0.0's image: they tie).""" + b = v.float().contiguous().view(torch.int32).to(torch.int64) & _LOW32 + b = torch.where(b == 0x80000000, torch.zeros_like(b), b) + return torch.where(b >= 0x80000000, _LOW32 - b, b | 0x80000000) + + +def make_keys(values: torch.Tensor, index: torch.Tensor) -> torch.Tensor: + """Exchange keys of (fp32 value, global token index) pairs: larger value wins, ties take the lower index.""" + return (ordered_bits(values) << 32) | (_LOW32 - index.to(torch.int64)) + + +def unsigned_max(keys: torch.Tensor, dim: int) -> torch.Tensor: + """Max of int64-stored uint64 keys (the kernel compares them unsigned).""" + sign = torch.iinfo(torch.int64).min + return (keys ^ sign).max(dim=dim).values ^ sign + + +def key_token(key: torch.Tensor) -> torch.Tensor: + return (_LOW32 - (key & _LOW32)).to(torch.int32) + + +def local_argmax(logits: torch.Tensor, vocab_offset: int) -> tuple[torch.Tensor, torch.Tensor]: + """(max value, lowest global index reaching it) per row of a [M, Vs] fp32 shard.""" + best = logits.max(dim=1).values + idx = (logits == best[:, None]).float().argmax(dim=1) # first (lowest) index at the max + return best, idx.to(torch.int64) + vocab_offset + + +def exchange_tokens( + own_keys: torch.Tensor, + ws_uc: torch.Tensor, + counters: torch.Tensor, + rank: int, + world: int, + slot_rows: int, +) -> torch.Tensor: + """Global tokens of the cross-rank pick: this rank's keys against the other ranks' keys already in its slots. + + Slot (half * slot_rows + t) * world + r of ``ws_uc`` holds rank r's key of row t; half = call counter & 1. A + single-device run stands in for the other ranks by writing their keys there before the call. + """ + m = own_keys.shape[0] + half = int(counters[1].item()) & 1 + slots = ( + ws_uc.view(torch.int64)[half * slot_rows * world : (half * slot_rows + m) * world] + .view(m, world) + .clone() + ) + slots[:, rank] = own_keys + return key_token(unsigned_max(slots, dim=1)) + + +def strict_accept( + tokens: torch.Tensor, + draft: torch.Tensor, + batch_idx: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + force_vec: Optional[torch.Tensor], +) -> dict[str, torch.Tensor]: + """The strict acceptance epilogue from the verify-row tokens [nreq * (D + 1)] (request-major). + + accepted[b] = tokens of request b's rows; num_accepted[b] = 1 + the matching prefix of draft[b] (overridden by + force_vec[b], else by force_total > 0); gather_ids[b] = batch_idx[b] * (D + 1) + num_accepted[b] - 1, and + next_new[b, 0] (when given) = the token at that row. Returns the outputs plus ``next_new`` as written. + """ + nreq, d = draft.shape + k1 = d + 1 + tokens = tokens.to(torch.int32) + accepted = tokens.view(nreq, k1).clone() + match = (draft.to(torch.int32) == accepted[:, :d]).to(torch.int32) + num = 1 + match.cumprod(dim=1).sum(dim=1) + if force_vec is not None: + num = force_vec[:nreq].to(num.dtype) + elif force_total > 0: + num = torch.full_like(num, force_total) + gather = batch_idx.to(torch.int64) * k1 + num.to(torch.int64) - 1 + out = {"accepted": accepted, "num_accepted": num.to(torch.int32), "gather_ids": gather} + if next_new is not None: + nn = next_new.clone() + nn[:, 0] = tokens[gather] + out["next_new"] = nn + return out + + +class M3ShardAccept(OpWrapper): + """Greedy verification of ``nreq * (D + 1) <= 48`` verify rows from this rank's BF16 logits shard ``[rows, Vs]`` + (global ids ``vocab_offset ..``). + + Each row's token is its global argmax over all ranks' shards: keys ``ordered(value) << 32 | ~index`` (ties take + the lowest index, as ``torch.argmax``) are max-reduced over the shard, pushed into every rank's exchange slot + through the multicast slots ``[2][48][world]`` (Lamport: 0 = empty) and max-reduced over the ranks by the last + CTA, which then runs the strict acceptance: ``acc_tokens [nreq, D + 1]``, ``num_acc [nreq]`` (``force_vec`` / + ``force_total`` override the counts), ``gather_ids [nreq]`` (the first draft pass's row per request) and, when + given, ``next_new[:, 0]`` (the last accepted token). + """ + + ARCHS = frozenset({Arch.SM_107}) + + # Every greedy verify step of the target (modeling's SHARD_ACCEPT): draft_len 3 at the TP8 GEN worker's 1..2 + # requests (4..8 rows), plus the 48-row maximum; the target lm_head's BF16 shard logits (25008 columns per rank + # at TP8, 8 exchange slots per row). The logits come from the head GEMM, so the op only reduces. + CELLS: tuple[Cell, ...] = ( + Cell( + why="b1: 4 verify rows (1 request x T = 4), rank 0 of 8, next_new staged", + spec=dict(nreq=1, draft_len=3, vs=25008, world=8, rank=0, force="none", strided=False), + ), + Cell( + why="b2: 8 verify rows, rank 7, global ids offset by 7 x 25008", + spec=dict(nreq=2, draft_len=3, vs=25008, world=8, rank=7, force="none", strided=False), + ), + Cell( + why="8 rows (2 requests) read from a row-strided view of a wider logits buffer, forced counts", + spec=dict(nreq=2, draft_len=3, vs=25008, world=8, rank=3, force="vec", strided=True), + ), + Cell( + why="the 48-row maximum (12 requests at draft 3), force_total, rows with exact cross-rank ties", + spec=dict( + nreq=12, + draft_len=3, + vs=25008, + world=8, + rank=5, + force="total", + strided=False, + ties=True, + ), + ), + ) + + note = """ + The op reduces BF16 logits exactly, so the reference is bit-exact, including cross-rank ties (lowest global + index wins). The cross-rank exchange is certified on one device: the other ranks' keys are written into this + rank's slots before the call, which is what their multicast pushes would have put there; the multicast mapping + itself (ws_mc_ptr = this rank's slots, as at world 1) is not exercised across devices here. + """ + + def __call__( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + torch.ops.trtllm.minimax_m3_shard_accept( + logits, + vocab_offset, + keys, + counters, + ws_uc, + ws_mc_ptr, + rank, + world, + draft, + batch_idx, + acc_tokens, + num_acc, + gather_ids, + force_total, + next_new, + num_ctas, + force_vec, + ) + + def reference( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> dict[str, torch.Tensor]: + """Lowest-index maximum of each row's BF16 logits, the largest key among it and the peers' keys in + ``ws_uc``, then the strict acceptance. Reads ``ws_uc`` / ``counters`` before the call; the scratch and + output buffers and ``num_ctas`` do not change the result.""" + best, idx = local_argmax(logits.float(), vocab_offset) + tokens = exchange_tokens(make_keys(best, idx), ws_uc, counters, rank, world, SHARD_ROWS) + out = strict_accept(tokens, draft, batch_idx, force_total, next_new, force_vec) + out["tokens"] = tokens + return out + + def is_valid( + self, + logits: torch.Tensor, + vocab_offset: int, + keys: torch.Tensor, + counters: torch.Tensor, + ws_uc: torch.Tensor, + ws_mc_ptr: int, + rank: int, + world: int, + draft: torch.Tensor, + batch_idx: torch.Tensor, + acc_tokens: torch.Tensor, + num_acc: torch.Tensor, + gather_ids: torch.Tensor, + force_total: int, + next_new: Optional[torch.Tensor], + num_ctas: int, + force_vec: Optional[torch.Tensor] = None, + ) -> None: + assert not bool(keys[: logits.shape[0]].any()), "keys must start zeroed" + assert int(counters[0].item()) == 0, ( + "counters[0] (finished CTAs) must be zero between calls" + ) + assert ws_uc.numel() >= 2 * SHARD_ROWS * world, "ws_uc must hold [2][48][world] slots" + # A NaN logit maps to a key above every finite one and wins the row silently. + assert not bool(torch.isnan(logits).any()), "NaN logits win the argmax" + nreq = draft.shape[0] + assert bool(((batch_idx[:nreq] >= 0) & (batch_idx[:nreq] < nreq)).all()), ( + "batch_idx must index the nreq requests" + ) + if force_vec is not None: + fv = force_vec[:nreq] + assert bool(((fv >= 1) & (fv <= draft.shape[1] + 1)).all()), ( + "force_vec counts must lie in [1, D + 1]" + ) + + +m3_shard_accept = M3ShardAccept() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_slot_scatter.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_slot_scatter.py new file mode 100644 index 000000000000..ff07c54de1f0 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_slot_scatter.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The spec sampler's per-slot stores of one step's outputs, in one launch (``M3SpecStepFold.scatter`` -> +``m3_spec_tail.slot_scatter``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +def padded_rows(src: torch.Tensor, width: int) -> torch.Tensor: + """src rows zero-padded (or cut) to ``width`` columns.""" + out = torch.zeros(src.shape[0], width, dtype=src.dtype, device=src.device) + w = min(width, src.shape[1]) + out[:, :w] = src[:, :w] + return out + + +class M3SlotScatter(OpWrapper): + """For i < len(slots), all int32 2-D strided views: ``da[slots[i]] = sa[i]``, ``db[slots[i]] = sb[i]``, + ``dd[slots[i]] = sd[i]`` (each zero-padded / cut to the destination width) and ``dl[slots[i]] = sl[i]``. The + destinations are the store's transposed [slots, width] views.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The sampler store at draft_len 3 (8 slots): new tokens / next new tokens [slots, 4] (transposed views of + # [4, slots, 1]), next draft tokens [slots, 3]; a worker output narrower than the store (dynamic draft length) + # is zero-padded. + CELLS: tuple[Cell, ...] = ( + Cell( + why="8 rows into shuffled slots, widths equal to the store's", + spec=dict(n=8, slots=8, src_w=4, src_d=3, store_w=4, store_d=3), + ), + Cell( + why="3 rows of a narrower step (runtime draft 2): zero padding to the store width", + spec=dict(n=3, slots=8, src_w=3, src_d=2, store_w=4, store_d=3), + ), + ) + + note = """ + Pure index copies: the reference is exact. Slots are not bounds-checked; is_valid states it. + """ + + def __call__(self, slots, sa, da, sb, db, sl, dl, sd, dd): + torch.ops.trtllm.minimax_m3_slot_scatter(slots, sa, da, sb, db, sl, dl, sd, dd) + + def reference(self, slots, sa, da, sb, db, sl, dl, sd, dd): + """The four destinations after the call, on copies.""" + s = slots.long() + out = {} + for name, src, dst in (("da", sa, da), ("db", sb, db), ("dd", sd, dd)): + r = dst.clone() + r[s] = padded_rows(src[: s.numel()], dst.shape[1]) + out[name] = r + lr = dl.clone() + lr[s] = sl[: s.numel()] + out["dl"] = lr + return out + + def is_valid(self, slots, sa, da, sb, db, sl, dl, sd, dd): + assert bool(((slots >= 0) & (slots < da.shape[0]) & (slots < dl.numel())).all()), ( + "slots must index the store" + ) + # Two rows into one slot race (which one lands is unordered). + assert slots.unique().numel() == slots.numel(), "slots must be distinct" + assert sa.shape[0] >= slots.numel() and sl.numel() >= slots.numel(), "a source row per slot" + + +m3_slot_scatter = M3SlotScatter() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_spec_offsets.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_spec_offsets.py new file mode 100644 index 000000000000..56c5f54b1161 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_spec_offsets.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The overlap scheduler's speculative step correction as one launch (``RuntimeHooks.spec_offsets_op``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper +from tensorrt_llm.logger import logger + +# The engagement line is logged once per process (module state: __call__ reads no instance state). +_ENGAGED: list = [] + + +class M3SpecOffsets(OpWrapper): + """``positions += position_offsets`` and ``kv_lens += kv_lens_offsets`` (int32, contiguous 1-D views, equal lengths + per pair) in one plain launch: trtllm::minimax_m3_sm107_tp8_spec_offsets (minimaxM3Sm107Tp8/specOffsets.cu), the + same wrapping int32 adds the two ATen ops do. ``__call__`` returns False, having written nothing, for other + layouts (the caller runs the ATen adds).""" + + ARCHS = frozenset({Arch.SM_107}) + + CELLS: tuple[Cell, ...] = ( + Cell(why="1 request, 4 tokens (draft length 3)", spec=dict(tokens=4, seqs=1)), + Cell(why="8 requests, 32 tokens", spec=dict(tokens=32, seqs=8)), + Cell(why="wrap-around values", spec=dict(tokens=8, seqs=2, wrap=True)), + ) + + note = "Integer adds: the reference (the two in-place ATen adds) is exact." + + def __call__(self, pos, pos_off, kv, kv_off) -> bool: + ts = (pos, pos_off, kv, kv_off) + if ( + any( + t.dtype != torch.int32 or t.dim() != 1 or t.stride(0) != 1 or not t.is_cuda + for t in ts + ) + or pos.numel() != pos_off.numel() + or kv.numel() != kv_off.numel() + ): + return False + torch.ops.trtllm.minimax_m3_sm107_tp8_spec_offsets(pos, pos_off, kv, kv_off) + if not _ENGAGED: + _ENGAGED.append(True) + logger.info("[M3 host] spec step correction: one launch taken (spec_offsets_op)") + return True + + def reference(self, pos, pos_off, kv, kv_off) -> None: + pos += pos_off + kv += kv_off + + def is_valid(self, pos, pos_off, kv, kv_off) -> None: + assert pos.numel() == pos_off.numel() and kv.numel() == kv_off.numel(), ( + "equal lengths per pair" + ) + + +m3_spec_offsets = M3SpecOffsets() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_step_gather.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_step_gather.py new file mode 100644 index 000000000000..a0da8b337574 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_step_gather.py @@ -0,0 +1,160 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The overlap scheduler's per-step input gathers from the previous step's device outputs, in one launch +(``M3SpecStepFold.gather`` -> ``m3_spec_tail.step_gather``).""" + +from __future__ import annotations + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3StepGather(OpWrapper): + """For the ``P`` previous requests (sequence slots ``slots``, ``T`` tokens and ``D`` draft tokens each), all + int32:: + + ids[ids0 + p*T + t] = new_tok[t, slots[p]] (new_tok [T', slots, (1)]) + draft[d0 + p*D + d] = next_draft[slots[p], d] + pos_off[i] = lens[ppi[i - b0*T]] for i in [b0*T, (b0+P)*T), else 0 (every i < pos_off.numel()) + kv_off[i] = lens[slots[i - b0]] - T for i in [b0, b0+P), else 0 (every i < kv_off.numel()) + + Every other element of ``ids`` / ``draft`` is left as it was. + """ + + ARCHS = frozenset({Arch.SM_107}) + + # draft_len 3: T = 4, D = 3; the store's [T, slots, 1] new tokens and [slots, D] next drafts (8 slots, the + # max batch); the engine's flat buffers sized for the max batch. + CELLS: tuple[Cell, ...] = ( + Cell( + why="an unchanged batch of 8 previous requests in shuffled slots, nothing before them (b0 = 0)", + spec=dict(p=8, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0), + ), + Cell( + why="2 previous requests after 3 new ones: offsets ids0 / d0 / b0 nonzero, zero-filled offsets outside", + spec=dict(p=2, t=4, d=3, slots=8, b0=3, ids0=12, d0=9, extra_rows=3), + ), + Cell( + why="one request, the store's 2-D view of new tokens (beam dim squeezed)", + spec=dict(p=1, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0, two_d=True), + ), + Cell( + why="an unchanged batch of 3 previous requests (c3) in shuffled slots", + spec=dict(p=3, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0), + ), + Cell( + why="an unchanged batch of 4 previous requests (c4) in shuffled slots", + spec=dict(p=4, t=4, d=3, slots=8, b0=0, ids0=0, d0=0, extra_rows=0), + ), + ) + + note = """ + Pure index copies: the reference is exact. The op does not bound slots, ppi or the destination offsets; is_valid + states them. + """ + + def __call__( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + torch.ops.trtllm.minimax_m3_step_gather( + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ) + + def reference( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + """The four destinations after the call, on copies (``ids`` / ``draft`` keep their other elements).""" + nt = new_tok.reshape(new_tok.shape[0], new_tok.shape[1]) + s = slots[:P].long() + ids_r, draft_r = ids.clone(), draft.clone() + ids_r[ids0 : ids0 + P * T] = nt[:T, s].t().reshape(-1) + draft_r[d0 : d0 + P * D] = next_draft[s, :D].reshape(-1) + pos_r = torch.zeros_like(pos_off) + pos_r[b0 * T : (b0 + P) * T] = lens[ppi[: P * T].long()] + kv_r = torch.zeros_like(kv_off) + kv_r[b0 : b0 + P] = lens[s] - T + return {"ids": ids_r, "draft": draft_r, "pos_off": pos_r, "kv_off": kv_r} + + def is_valid( + self, + new_tok, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + ids, + ids0, + draft, + d0, + pos_off, + kv_off, + b0, + ): + # Reads through slots / ppi and writes at the offsets are unchecked: any of them out of range reads or + # writes another buffer. + s = slots[:P] + assert bool( + ( + (s >= 0) & (s < next_draft.shape[0]) & (s < new_tok.shape[1]) & (s < lens.numel()) + ).all() + ), "slots must index the store" + assert bool(((ppi[: P * T] >= 0) & (ppi[: P * T] < lens.numel())).all()), ( + "ppi must index lens" + ) + assert new_tok.shape[0] >= T and next_draft.shape[1] >= D, "store narrower than T / D" + assert ids.numel() >= ids0 + P * T and draft.numel() >= d0 + P * D, "destinations too short" + assert pos_off.numel() >= (b0 + P) * T and kv_off.numel() >= b0 + P, ( + "offset buffers too short" + ) + + +m3_step_gather = M3StepGather() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_step_prologue.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_step_prologue.py new file mode 100644 index 000000000000..a6c626ff2d2f --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/ops/spec/m3_step_prologue.py @@ -0,0 +1,76 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode graph's step-input prologue in one launch (this target's ``RuntimeHooks.pack_prologue_op``).""" + +from __future__ import annotations + +from typing import Optional + +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import Arch, Cell, OpWrapper + + +class M3StepPrologue(OpWrapper): + """``StepInputPack``'s early-copy prologue as one single-CTA C++ launch + (trtllm::minimax_m3_sm107_tp8_step_prologue, minimaxM3Sm107Tp8/stepPrologue.cu). + + ``window`` is the pack's device window (``gdev``): an int64 header ``[host copies, device copies]``, then one + ``(dst, src, bytes)`` descriptor per copy, host copies first (a host copy's ``src`` is a byte offset into the + window, a device copy's a device address). The launch writes up to ``max_desc`` copies of each kind, every host + copy before any device copy, then zeroes the header; with ``epoch`` it increments ``epoch[0]`` once. The default + prologue does the same with two parallel-scatter launches (``step_inputs._graph_scatter_kernel``) and a header + fill.""" + + ARCHS = frozenset({Arch.SM_107}) + + # The TP8 decode step's packs: a few dozen host copies of 4 to a few hundred bytes (positions, KV lengths, slots, + # the spec sampler's uploads), a handful of device copies (the graph runner's static inputs), some reading what a + # host copy of the same window wrote. + CELLS: tuple[Cell, ...] = ( + Cell(why="an empty window (a replay with no new upload)", spec=dict(h2d=0, d2d=0)), + Cell(why="host copies only, odd sizes and alignments", spec=dict(h2d=40, d2d=0)), + Cell( + why="host copies feeding device copies (a device copy reads a host copy's destination)", + spec=dict(h2d=24, d2d=8, chained=True), + ), + Cell( + why="every descriptor slot (128 + 128 of the pack's 256)", spec=dict(h2d=128, d2d=128) + ), + ) + + note = """ + Byte copies: the reference (the default prologue's Triton launches and fill on the same window) is exact. + Destinations are raw addresses; is_valid cannot check them. + """ + + def __call__(self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int) -> None: + torch.ops.trtllm.minimax_m3_sm107_tp8_step_prologue(window, epoch, max_desc) + + def reference(self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int) -> None: + """The default prologue on the same arguments: it writes the same destinations.""" + from tensorrt_llm._torch.pyexecutor.step_inputs import _graph_scatter_kernel + + stage_i64 = window.view(torch.int64) + for device_src in (False, True): + bump = epoch is not None and not device_src + _graph_scatter_kernel[(max_desc,)]( + window, + stage_i64, + epoch if bump else stage_i64, + DEVICE_SRC=device_src, + BLOCK=256, + BUMP_EPOCH=bump, + num_warps=1, + ) + window[:16].zero_() + + def is_valid(self, window: torch.Tensor, epoch: Optional[torch.Tensor], max_desc: int) -> None: + assert window.dtype == torch.uint8 and window.is_contiguous(), "window: contiguous uint8" + assert window.numel() >= 16 + 24 * max_desc, "window: header and max_desc descriptors" + counts = window[:16].view(torch.int64).tolist() + assert counts[0] + counts[1] <= max_desc, "at most max_desc copies in all" + + +m3_step_prologue = M3StepPrologue() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/weights.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/weights.py new file mode 100644 index 000000000000..e5dac780e99a --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/weights.py @@ -0,0 +1,929 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Weight tables and loaders: MiniMax-M3.1 (MXFP8 + packed NVFP4 experts) / sm_107 / tp8. + +Two tables, one per checkpoint the target reads: + +* ``MODEL_WEIGHTS`` (``MiniMaxM3Tp8Weights``): the vision-language checkpoint's language model, + ``language_model.*``. Every rank is handed the whole checkpoint, so every tensor-parallel split is this + table's: the Q heads, the K/V/index-Q heads (4 each over 8 ranks: rank r holds head r // 2, as the built-in + loader duplicates them), the dense MLPs and the shared expert's + intermediate, the o_proj / down-projection input columns, every routed expert's intermediate (all 128 + experts live on every rank), and the vocabulary of the shell's ``lm_head``. The index-K head is replicated. +* ``DRAFT_WEIGHTS`` (``MiniMaxM3Tp8DraftWeights``): the one-layer BF16 Eagle3 drafter's own checkpoint + (``speculative_model``), loaded through the shell's ``load_draft_weights`` when speculation is on. + +Storage is HF ``[out, in]`` row-major. Five families need a transform, each a layout a kernel demands: + +* **MXFP8 block scales** (every attention projection, the dense MLPs and the shared expert): the checkpoint + stores UE8M0 bytes ``[N, K / 32]``; the GEMMs read the 128x4-swizzled order + (``trtllm::block_scale_interleave``). A fused projection concatenates its shards' scales first and swizzles + once. +* **The five-way sparse-layer projection** ``[Q | K | V | index-Q | index-K]`` and the three-way dense one + ``[Q | K | V]``: the per-rank shards concatenated along N. +* **Gated MLPs** (dense and shared expert): ``[gate ; up]`` rows. +* **NVFP4 routed experts** (layers 3..59): ``[up (w3) ; gate (w1)]`` rows, then the trtllm-gen row shuffle of + the weight and of its E4M3 scales (gated-activation reorder + 128-row epilogue tile shuffle, + ``trtllmgen_maybe_get_cached_*_permute_indices``) and the 128x4 swizzle of the scales. +* **Gemma boundary norms**: ``input_layernorm``, ``post_attention_layernorm`` and ``model.norm`` are stored as + Gemma norms, ``(1 + w) * x``; the fused all-reduce + RMSNorm kernels apply ``w * x``, so ``1`` is added + (in BF16) at load. The per-head Q/K/index norms stay Gemma: their kernels apply the offset themselves. + +The routed experts' per-tensor scalars (input scales and second-level weight scales, every expert) load raw; +``modeling`` derives the kernels' global scales, alphas and activation constants from them once the weights +are real. + +Loading contract: ``load(model, weights)`` consumes the engine-provided checkpoint dict, fills every declared +parameter exactly once and asserts coverage in both directions: every declared parameter written, and every +checkpoint key either consumed or in ``expected_unconsumed`` (the vision tower and its projector, which this +text-only target does not build, and the language model's multi-token-prediction head). +""" + +from __future__ import annotations + +from dataclasses import dataclass +from types import SimpleNamespace +from typing import Any, Callable + +import torch +import torch.nn as nn + +from tensorrt_llm._torch.models.checkpoints.hf.minimaxm3_weight_mapper import ( + reciprocal_global_scale, +) +from tensorrt_llm._torch.moe.fused_moe.quantization import ( + trtllmgen_maybe_get_cached_w2_permute_indices, + trtllmgen_maybe_get_cached_w3_w1_permute_indices, +) + +#: ``layers=ALL_LAYERS`` is the common case: a weight every decoder layer carries. +ALL_LAYERS = object() + +_LM = "language_model." +#: Checkpoint key families this target does not build: the vision tower and its two projectors (text only), and the +#: language model's multi-token-prediction head (speculation, when on, is the separate Eagle3 drafter). +_UNBUILT_PREFIXES = ( + "vision_tower.", + "multi_modal_projector.", + "patch_merge_mlp.", + _LM + "model.mtp.", +) + +MX_BLOCK = 32 # MXFP8: one UE8M0 scale per 32 elements along K +NV_BLOCK = 16 # NVFP4: one E4M3 scale per 16 elements along K +EPILOGUE_TILE_M = ( + 128 # the trtllm-gen MoE runner's row tile: fixes its weight and scale row shuffles +) + + +@dataclass(frozen=True) +class W: + """One weight role: what it is, where its bytes come from, what happens on the way. + + ``shape`` takes the dimension bundle and returns the declared shape. ``src`` is a callable + ``(d, i) -> [(key or key tuple, index into the param | None), ...]`` with keys relative to the checkpoint + root; a key tuple hands the transform those tensors together. ``layers`` is ``ALL_LAYERS``, ``None`` (not + per-layer) or ``d -> layer indices``. ``transform(src, d)`` runs on the destination's device before the copy. + """ + + name: str + shape: Callable[[Any], tuple[int, ...]] + src: Callable + dtype: torch.dtype | None = None + transform: Callable | None = None + layers: object | None = ALL_LAYERS + + def layer_indices(self, d: SimpleNamespace) -> list[int] | None: + if self.layers is None: + return None + if self.layers is ALL_LAYERS: + return list(range(d.num_layers)) + return list(self.layers(d)) + + +# ------------------------------------------------------------------------------------------------ shards + + +def fused_projection(sparse_config) -> bool: + """Whether the sparse layers run the five-way packed QKV + index projection. + + On by default: this target requires the MSA implementation and an FP8 indexer, which the packed projection needs. + An explicit ``fuse_qkv_index_projection`` in the deployment's sparse attention config wins, so setting it to False + is the kill switch (the QKV projection plus a separate MXFP8 index projection). + """ + if sparse_config is None: + return False + # A config that does not record which fields were set (a plain namespace or dataclass) is taken at its word. + fields_set = getattr(sparse_config, "model_fields_set", None) + if fields_set is None or "fuse_qkv_index_projection" in fields_set: + return bool(getattr(sparse_config, "fuse_qkv_index_projection", False)) + return True + + +def _col( + t: torch.Tensor, d: SimpleNamespace, tp: int | None = None, rank: int | None = None +) -> torch.Tensor: + """This rank's rows (column-parallel output split), even split, sliced on the host and moved to the + destination device (only this rank's bytes are read).""" + tp = d.tp if tp is None else tp + rank = d.rank if rank is None else rank + n = t.shape[0] // tp + return _dev(t[rank * n : (rank + 1) * n], d) + + +def _heads(t: torch.Tensor, d: SimpleNamespace, head_dim: int) -> torch.Tensor: + """This rank's rows of a projection split by heads of ``head_dim`` rows: an even split when there are at least as + many heads as ranks, else the head of rank // (ranks per head), duplicated on those ranks.""" + n = t.shape[0] // head_dim + if n >= d.tp: + return _col(t, d) + h = d.rank // (d.tp // n) + return _dev(t[h * head_dim : (h + 1) * head_dim], d) + + +def _row(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """This rank's input columns (row-parallel input split), even split, on the destination device.""" + n = t.shape[1] // d.tp + return _dev(t[:, d.rank * n : (d.rank + 1) * n], d) + + +def _dev(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """``t`` on the destination device (``d.dev``, set per fill).""" + return t.to(d.dev, non_blocking=True) + + +def _swizzle(scale: torch.Tensor) -> torch.Tensor: + """128x4 swizzle of a row-major ``[N, K / 32]`` UE8M0 scale matrix, flat.""" + return torch.ops.trtllm.block_scale_interleave(scale.contiguous()) + + +# --------------------------------------------------------------------------------------------- transforms + + +def _gemma_fold(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A Gemma boundary norm as the plain norm the fused all-reduce epilogue applies: ``w + 1`` in BF16.""" + return t + 1.0 + + +def _f32(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return t.to(torch.float32) + + +def _divisor(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """A stored global scale (multiplier) -> the modelopt divisor: its FP32 reciprocal.""" + return reciprocal_global_scale("expert global scale", t) + + +def _qkv_sparse_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[Q | K | V | index-Q | index-K]``: Q by attention heads, K/V/index-Q by their heads, index-K replicated.""" + q, k, v, iq, ik = t + hd, xd = d.head_dim, d.index_dim + return torch.cat( + [_col(q, d), _heads(k, d, hd), _heads(v, d, hd), _heads(iq, d, xd), _dev(ik, d)], dim=0 + ).contiguous() + + +def _qkv_sparse_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v, iq, ik = t + hd, xd = d.head_dim, d.index_dim + return _swizzle( + torch.cat( + [_col(q, d), _heads(k, d, hd), _heads(v, d, hd), _heads(iq, d, xd), _dev(ik, d)], dim=0 + ) + ) + + +def _qkv_dense_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + hd = d.head_dim + return torch.cat([_col(q, d), _heads(k, d, hd), _heads(v, d, hd)], dim=0).contiguous() + + +def _qkv_dense_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + hd = d.head_dim + return _swizzle(torch.cat([_col(q, d), _heads(k, d, hd), _heads(v, d, hd)], dim=0)) + + +def _index_qk_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """``[index-Q | index-K]`` as the checkpoint's MXFP8 (E4M3) values: this rank's index-Q head (rank r holds head + r // 2, as the five-way projection's ``_heads`` split) and the replicated index-K, the same rows it packs.""" + iq, ik = t + return torch.cat([_heads(iq, d, d.index_dim), _dev(ik, d)], dim=0).contiguous() + + +def _index_qk_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """The UE8M0 block scales of ``_index_qk_w``, 128x4-swizzled as every MXFP8 projection's.""" + iq, ik = t + return _swizzle(torch.cat([_heads(iq, d, d.index_dim), _dev(ik, d)], dim=0)) + + +def _gate_up_w(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return torch.cat([_col(gate, d), _col(up, d)], dim=0).contiguous() + + +def _gate_up_s(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + gate, up = t + return _swizzle(torch.cat([_col(gate, d), _col(up, d)], dim=0)) + + +def _row_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _row(t, d).contiguous() + + +def _row_s(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _swizzle(_row(t, d)) + + +def _col_w(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + return _col(t, d).contiguous() + + +#: Permute indices of the trtllm-gen expert relayouts, keyed by (shape, family, scale group): one expert's are +#: every expert's, so each is computed once per load. +_PERMUTE_CACHE: dict = {} + + +def _expert_w31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 operand: ``[up ; gate]`` packed-FP4 rows of this rank, then the gated-activation row + reorder and the epilogue-tile shuffle.""" + up, gate = t + w = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + w, _PERMUTE_CACHE, EPILOGUE_TILE_M, is_gated_act_gemm=True + ) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s31(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + """One expert's FC1 E4M3 scales: ``[up ; gate]`` rows of this rank, the same row reorder at scale + granularity, then the 128x4 swizzle.""" + up, gate = t + s = torch.cat([_col(up, d), _col(gate, d)], dim=0).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w3_w1_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK, is_gated_act_gemm=True + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled.view(torch.uint8).reshape(shape)) + .view(torch.float8_e4m3fn) + .reshape(shape) + ) + + +def _expert_w2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 operand: this rank's input columns, then the epilogue-tile row shuffle.""" + w = _row(t, d).contiguous() + perm = trtllmgen_maybe_get_cached_w2_permute_indices(w, _PERMUTE_CACHE, EPILOGUE_TILE_M) + return torch.ops.trtllm.shuffle_matrix(w, perm.to(w.device)).view(w.dtype) + + +def _expert_s2(t: torch.Tensor, d: SimpleNamespace) -> torch.Tensor: + """One expert's FC2 E4M3 scales: this rank's columns, the row shuffle at scale granularity, the swizzle.""" + s = _row(t, d).contiguous() + shape = s.shape + sf = s.view(torch.uint8) + perm = trtllmgen_maybe_get_cached_w2_permute_indices( + sf, _PERMUTE_CACHE, EPILOGUE_TILE_M, num_elts_per_sf=NV_BLOCK + ) + shuffled = torch.ops.trtllm.shuffle_matrix(sf, perm) + return ( + torch.ops.trtllm.block_scale_interleave(shuffled).view(torch.float8_e4m3fn).reshape(shape) + ) + + +# ------------------------------------------------------------------------------------------------ sources + + +def _one(template: str): + return lambda d, i, t=template: [(_LM + t.format(i=i), None)] + + +def _many(*templates: str): + return lambda d, i, ts=templates: [(tuple(_LM + t.format(i=i) for t in ts), None)] + + +def _per_expert(*suffixes: str): + """One source per routed expert, written at its expert index (all experts are local at tp8).""" + + def build(d, i, sx=suffixes): + out = [] + for e in range(d.num_experts): + keys = tuple(_LM + f"model.layers.{i}.block_sparse_moe.experts.{e}.{s}" for s in sx) + out.append((keys if len(keys) > 1 else keys[0], (e,))) + return out + + return build + + +def _dense_mlp_layers(d): + """Layers with the dense SwiGLU-OAI MLP (``moe_layer_freq`` 0).""" + return d.dense_mlp + + +def _moe_layers(d): + return d.moe + + +def _sparse_attn_layers(d): + """Layers with MSA sparse attention (``sparse_attention_freq`` 1): every layer of MiniMax-M3.1.""" + return d.sparse_attn + + +def _three_way_qkv_layers(d): + """Layers whose QKV is the three-way [Q | K | V]: the dense-attention layers, and every layer without the + five-way packed projection (``fuse_qkv_index_projection=False``).""" + return d.dense_attn if d.fused else range(d.num_layers) + + +def _five_way_qkv_layers(d): + return d.sparse_attn if d.fused else range(0) + + +def _unfused_sparse_layers(d): + return range(0) if d.fused else d.sparse_attn + + +_ATTN = "model.layers.{i}.self_attn" +_MOE = "model.layers.{i}.block_sparse_moe" + + +class MiniMaxM3Tp8Weights: + """MiniMax-M3 NVFP4 / sm_107 / tp8: the language model's table and the passes over it. + + ``declare`` allocates every weight it names, ``manifest`` maps checkpoint keys onto them, ``load`` fills + them; ``_targets`` is the one place that decides which parameters exist. + """ + + #: Free the per-expert relayout scratch after each expert stack. + RELEASE_AFTER = "_e_s2" + + WEIGHTS: tuple[W, ...] = ( + # --- the two boundary norms of every layer (Gemma, folded) --- + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.input_layernorm.weight"), + transform=_gemma_fold, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_one("model.layers.{i}.post_attention_layernorm.weight"), + transform=_gemma_fold, + ), + # --- attention: per-head Q/K norms (Gemma, unfolded), the packed QKV and the output projection --- + W("q_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".q_norm.weight")), + W("k_norm", shape=lambda d: (d.head_dim,), src=_one(_ATTN + ".k_norm.weight")), + W( + "qkv_w", + shape=lambda d: (d.qkv_dense_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in "qkv")), + transform=_qkv_dense_w, + layers=_three_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_dense_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many(*(f"{_ATTN}.{p}_proj.weight_scale_inv" for p in "qkv")), + transform=_qkv_dense_s, + layers=_three_way_qkv_layers, + ), + W( + "qkv_w", + shape=lambda d: (d.qkv_sparse_rows, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(*(f"{_ATTN}.{p}_proj.weight" for p in ("q", "k", "v", "index_q", "index_k"))), + transform=_qkv_sparse_w, + layers=_five_way_qkv_layers, + ), + W( + "qkv_s", + shape=lambda d: (d.qkv_sparse_rows * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + *( + f"{_ATTN}.{p}_proj.weight_scale_inv" + for p in ("q", "k", "v", "index_q", "index_k") + ) + ), + transform=_qkv_sparse_s, + layers=_five_way_qkv_layers, + ), + # The separate index projection [index-Q | index-K] of the unfused sparse layers: the checkpoint's MXFP8 + # values with their block scales (the rows of the five-way projection's index part), run as an MXFP8 GEMM. + W( + "idx_w", + shape=lambda d: (d.index_heads * d.index_dim + d.index_dim, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many(_ATTN + ".index_q_proj.weight", _ATTN + ".index_k_proj.weight"), + transform=_index_qk_w, + layers=_unfused_sparse_layers, + ), + W( + "idx_s", + shape=lambda d: ((d.index_heads * d.index_dim + d.index_dim) * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _ATTN + ".index_q_proj.weight_scale_inv", _ATTN + ".index_k_proj.weight_scale_inv" + ), + transform=_index_qk_s, + layers=_unfused_sparse_layers, + ), + W( + "idx_q_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_q_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "idx_k_norm", + shape=lambda d: (d.index_dim,), + src=_one(_ATTN + ".index_k_norm.weight"), + layers=_sparse_attn_layers, + ), + W( + "o_w", + shape=lambda d: (d.hidden, d.q_width), + dtype=torch.float8_e4m3fn, + src=_one(_ATTN + ".o_proj.weight"), + transform=_row_w, + ), + W( + "o_s", + shape=lambda d: (d.hidden * (d.q_width // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_ATTN + ".o_proj.weight_scale_inv"), + transform=_row_s, + ), + # --- dense SwiGLU-OAI MLP (layers 0..2) --- + W( + "mlp_gu_w", + shape=lambda d: (2 * d.dense_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight", "model.layers.{i}.mlp.up_proj.weight" + ), + transform=_gate_up_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_gu_s", + shape=lambda d: (2 * d.dense_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + "model.layers.{i}.mlp.gate_proj.weight_scale_inv", + "model.layers.{i}.mlp.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_w", + shape=lambda d: (d.hidden, d.dense_inter), + dtype=torch.float8_e4m3fn, + src=_one("model.layers.{i}.mlp.down_proj.weight"), + transform=_row_w, + layers=_dense_mlp_layers, + ), + W( + "mlp_dn_s", + shape=lambda d: (d.hidden * (d.dense_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one("model.layers.{i}.mlp.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_dense_mlp_layers, + ), + # --- MoE (layers 3..59): the fp32 router and its bias, the shared expert --- + W( + "router", + shape=lambda d: (d.num_experts, d.hidden), + dtype=torch.float32, + src=_one(_MOE + ".gate.weight"), + transform=_f32, + layers=_moe_layers, + ), + W( + "router_bias", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_one(_MOE + ".e_score_correction_bias"), + transform=_f32, + layers=_moe_layers, + ), + W( + "sh_gu_w", + shape=lambda d: (2 * d.shared_inter, d.hidden), + dtype=torch.float8_e4m3fn, + src=_many( + _MOE + ".shared_experts.gate_proj.weight", _MOE + ".shared_experts.up_proj.weight" + ), + transform=_gate_up_w, + layers=_moe_layers, + ), + W( + "sh_gu_s", + shape=lambda d: (2 * d.shared_inter * (d.hidden // MX_BLOCK),), + dtype=torch.uint8, + src=_many( + _MOE + ".shared_experts.gate_proj.weight_scale_inv", + _MOE + ".shared_experts.up_proj.weight_scale_inv", + ), + transform=_gate_up_s, + layers=_moe_layers, + ), + W( + "sh_dn_w", + shape=lambda d: (d.hidden, d.shared_inter), + dtype=torch.float8_e4m3fn, + src=_one(_MOE + ".shared_experts.down_proj.weight"), + transform=_row_w, + layers=_moe_layers, + ), + W( + "sh_dn_s", + shape=lambda d: (d.hidden * (d.shared_inter // MX_BLOCK),), + dtype=torch.uint8, + src=_one(_MOE + ".shared_experts.down_proj.weight_scale_inv"), + transform=_row_s, + layers=_moe_layers, + ), + # --- the routed experts, NVFP4 (trtllm-gen layouts), all 128 on every rank --- + W( + "e_w31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // 2), + dtype=torch.uint8, + src=_per_expert("w3.weight_packed", "w1.weight_packed"), + transform=_expert_w31, + layers=_moe_layers, + ), + W( + "e_w2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // 2), + dtype=torch.uint8, + src=_per_expert("w2.weight_packed"), + transform=_expert_w2, + layers=_moe_layers, + ), + W( + "e_s31", + shape=lambda d: (d.num_experts, 2 * d.moe_inter, d.hidden // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w3.weight_scale", "w1.weight_scale"), + transform=_expert_s31, + layers=_moe_layers, + ), + W( + "e_s2", + shape=lambda d: (d.num_experts, d.hidden, d.moe_inter // NV_BLOCK), + dtype=torch.float8_e4m3fn, + src=_per_expert("w2.weight_scale"), + transform=_expert_s2, + layers=_moe_layers, + ), + # The per-tensor scalars of every expert: the checkpoint stores the multipliers into the quantized range + # (``*_global_scale``); the modelopt divisors modeling derives the kernels' scalars from are their FP32 + # reciprocals, taken by the same helper as the built-in loader. + *( + W( + f"e_{name}", + shape=lambda d: (d.num_experts,), + dtype=torch.float32, + src=_per_expert(suffix), + transform=_divisor, + layers=_moe_layers, + ) + for name, suffix in ( + ("isc1", "w1.input_global_scale"), + ("isc3", "w3.input_global_scale"), + ("isc2", "w2.input_global_scale"), + ("ws2_1", "w1.weight_global_scale"), + ("ws2_3", "w3.weight_global_scale"), + ("ws2_2", "w2.weight_global_scale"), + ) + ), + # --- not per-layer --- + W( + "embed", + shape=lambda d: (d.vocab, d.hidden), + src=_one("model.embed_tokens.weight"), + layers=None, + ), + W( + "final_norm", + shape=lambda d: (d.hidden,), + src=_one("model.norm.weight"), + transform=_gemma_fold, + layers=None, + ), + ) + + def dims(self, core) -> SimpleNamespace: + cfg = core.model_config.pretrained_config + mapping = core.model_config.mapping + sparse = cfg.sparse_attention_config + freq = [int(f) for f in sparse["sparse_attention_freq"]] + moe_freq = [ + int(f) for f in getattr(cfg, "moe_layer_freq", None) or [1] * cfg.num_hidden_layers + ] + tp = mapping.tp_size + head_dim = int(cfg.head_dim) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + index_heads = max(int(sparse["sparse_num_index_heads"]) // tp, 1) + index_dim = int(sparse["sparse_index_dim"]) + return SimpleNamespace( + fused=fused_projection(core.model_config.sparse_attention_config), + num_layers=cfg.num_hidden_layers, + sparse_attn=[i for i, f in enumerate(freq) if f], + dense_attn=[i for i, f in enumerate(freq) if not f], + moe=[i for i, f in enumerate(moe_freq) if f], + dense_mlp=[i for i, f in enumerate(moe_freq) if not f], + hidden=cfg.hidden_size, + head_dim=head_dim, + heads=heads, + kv_heads=kv_heads, + index_heads=index_heads, + index_dim=index_dim, + q_width=heads * head_dim, + qkv_dense_rows=(heads + 2 * kv_heads) * head_dim, + qkv_sparse_rows=(heads + 2 * kv_heads) * head_dim + index_heads * index_dim + index_dim, + dense_inter=int(cfg.dense_intermediate_size) // tp, + shared_inter=int(cfg.shared_intermediate_size) * int(cfg.n_shared_experts) // tp, + moe_inter=int(cfg.intermediate_size) // tp, + num_experts=int(cfg.num_local_experts), + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + layers = entry.layer_indices(d) + if layers is None: + return [(entry.name, None)] + return [(f"l{i}_{entry.name}", i) for i in layers] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + """Allocate every weight the table declares (meta under the engine's meta init).""" + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, core) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + """param key -> [(checkpoint key or key tuple, index into the param | None, transform | None)].""" + d = self.dims(core) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, core, weights) -> set: + """The vision tower and its projectors, and the multi-token-prediction head: in the checkpoint, never built + by this target.""" + return {k for k in weights.keys() if k.startswith(_UNBUILT_PREFIXES)} + + def load(self, model, weights) -> None: + """Fill every declared weight and the shell's ``lm_head`` shard; assert coverage both ways.""" + core = model.model + d = self.dims(core) + manifest = self.manifest(core) + assert set(manifest) == set(core.w.keys()), ( + "manifest/parameter drift", + set(manifest) ^ set(core.w.keys()), + ) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(core, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(core.w[param_key], ckpt, index, transform, weights, d, consumed) + if param_key.endswith(self.RELEASE_AFTER): + torch.cuda.empty_cache() + # The shell owns the vocab-parallel head: this rank's rows of the full table. + _fill(model.lm_head.weight, _LM + "lm_head.weight", None, _col_w, weights, d, consumed) + shell = { + n for n, _ in model.named_parameters() if not n.startswith(("model.w.", "draft_model.")) + } + assert shell == {"lm_head.weight"}, f"unfed shell parameters: {sorted(shell)}" + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + "checkpoint coverage: leftover keys are not exactly the unbuilt families (vision, MTP head); " + f"unexpected {sorted(leftover - expected)[:8]}, missed {sorted(expected - leftover)[:8]}" + ) + + +def _materialize(entry) -> torch.Tensor: + """``[:]`` on a lazy safetensors slice; a 0-dim tensor (the per-tensor scalars) rejects that index.""" + return entry if getattr(entry, "ndim", 1) == 0 else entry[:] + + +def _fill(param, ckpt, index, transform, weights, d, consumed) -> None: + keys = ckpt if isinstance(ckpt, tuple) else (ckpt,) + for key in keys: + assert key in weights, f"checkpoint key missing: {key}" + dst = param.data if index is None else param.data[index] + d.dev = dst.device + # Host tensors: a transform slices this rank's part on the host and moves only that to the device. + src_all = tuple(_materialize(weights[key]) for key in keys) + if transform is not None: + src = transform(src_all if isinstance(ckpt, tuple) else src_all[0], d) + else: + assert len(src_all) == 1, "a multi-key row needs a transform" + src = src_all[0] + if src.dim() == 0 and dst.dim() == 0: + pass + elif src.numel() == 1 and dst.numel() == 1: + src = src.reshape(dst.shape) + src = src.to(dst.device, non_blocking=True) + assert tuple(dst.shape) == tuple(src.shape), (ckpt, tuple(dst.shape), tuple(src.shape)) + assert src.dtype == dst.dtype, (ckpt, src.dtype, dst.dtype) + dst.copy_(src, non_blocking=True) + consumed.update(keys) + _release(weights, keys) + + +def _release(weights, keys) -> None: + """Drop consumed checkpoint tensors from the engine's dict as the load goes (bounded host memory).""" + if hasattr(weights, "mark_consumed_keys"): + weights.mark_consumed_keys(keys) + else: + for key in keys: + weights.pop(key, None) + + +MODEL_WEIGHTS = MiniMaxM3Tp8Weights() + + +# -------------------------------------------------------------------------------------------- the drafter + + +def _draft_one(template: str): + return lambda d, i, t=template: [(t, None)] + + +def _draft_many(*templates: str): + return lambda d, i, ts=templates: [(tuple(ts), None)] + + +def _draft_fc_norms(d): + return range(d.capture_layers) + + +def _draft_qkv(t: tuple[torch.Tensor, ...], d: SimpleNamespace) -> torch.Tensor: + q, k, v = t + hd = d.head_dim + return torch.cat([_col(q, d), _heads(k, d, hd), _heads(v, d, hd)], dim=0).contiguous() + + +class MiniMaxM3Tp8DraftWeights: + """The one-layer BF16 Eagle3 drafter (``speculative_model``), tp8. + + The fc over the three captured target layers and its per-layer fc_norms, the layer's three norms, its QKV + (input ``[embeds | hidden]``, so twice the hidden width; Q by heads, K/V by KV heads, a KV head on several + ranks when there are fewer than ranks) and o_proj, its gated MLP, the final norm and the full-vocabulary head + (vocab-parallel). The checkpoint's ``embed_tokens`` is the target's table (the drafter reads the target's); it is + a predicted non-load. + """ + + WEIGHTS: tuple[W, ...] = ( + W( + "fc", + shape=lambda d: (d.hidden, d.capture_layers * d.hidden), + src=_draft_one("fc.weight"), + layers=None, + ), + W( + "fc_norm", + shape=lambda d: (d.hidden,), + src=lambda d, i: [(f"fc_norm.{i}.weight", None)], + layers=_draft_fc_norms, + ), + W( + "in_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.input_layernorm.weight"), + layers=None, + ), + W( + "hidden_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.hidden_norm.weight"), + layers=None, + ), + W( + "post_norm", + shape=lambda d: (d.hidden,), + src=_draft_one("layers.0.post_attention_layernorm.weight"), + layers=None, + ), + W( + "qkv", + shape=lambda d: (d.qkv_rows, 2 * d.hidden), + src=_draft_many(*(f"layers.0.self_attn.{p}_proj.weight" for p in "qkv")), + transform=_draft_qkv, + layers=None, + ), + W( + "o", + shape=lambda d: (d.hidden, d.q_width), + src=_draft_one("layers.0.self_attn.o_proj.weight"), + transform=_row_w, + layers=None, + ), + W( + "gu", + shape=lambda d: (2 * d.inter, d.hidden), + src=_draft_many("layers.0.mlp.gate_proj.weight", "layers.0.mlp.up_proj.weight"), + transform=_gate_up_w, + layers=None, + ), + W( + "dn", + shape=lambda d: (d.hidden, d.inter), + src=_draft_one("layers.0.mlp.down_proj.weight"), + transform=_row_w, + layers=None, + ), + W("final_norm", shape=lambda d: (d.hidden,), src=_draft_one("norm.weight"), layers=None), + ) + + def dims(self, draft) -> SimpleNamespace: + cfg = draft.draft_config.pretrained_config + mapping = draft.draft_config.mapping + tp = mapping.tp_size + head_dim = int(getattr(cfg, "head_dim", cfg.hidden_size // cfg.num_attention_heads)) + heads = cfg.num_attention_heads // tp + kv_heads = max(cfg.num_key_value_heads // tp, 1) + return SimpleNamespace( + hidden=cfg.hidden_size, + capture_layers=draft.capture_layers, + head_dim=head_dim, + q_width=heads * head_dim, + qkv_rows=(heads + 2 * kv_heads) * head_dim, + inter=cfg.intermediate_size // tp, + vocab=cfg.vocab_size, + tp=tp, + rank=mapping.tp_rank, + dtype=cfg.torch_dtype, + ) + + def _targets(self, entry: W, d: SimpleNamespace) -> list[tuple[str, int | None]]: + if entry.layers is None: + return [(entry.name, None)] + return [(f"{entry.name}{i}", i) for i in entry.layers(d)] + + def declare(self, d: SimpleNamespace) -> nn.ParameterDict: + w = nn.ParameterDict() + for entry in self.WEIGHTS: + dtype = entry.dtype or d.dtype + for key, _ in self._targets(entry, d): + w[key] = nn.Parameter( + torch.empty(*entry.shape(d), dtype=dtype), requires_grad=False + ) + return w + + def manifest(self, draft) -> dict[str, list[tuple[Any, Any, Callable | None]]]: + d = self.dims(draft) + rows: dict[str, list[tuple[Any, Any, Callable | None]]] = {} + for entry in self.WEIGHTS: + for key, i in self._targets(entry, d): + rows[key] = [(ckpt, index, entry.transform) for ckpt, index in entry.src(d, i)] + return rows + + def expected_unconsumed(self, draft, weights) -> set: + """The drafter checkpoint's own embedding: the drafter reads the target's table.""" + return {"embed_tokens.weight"} & set(weights.keys()) + + def load(self, draft, weights) -> None: + d = self.dims(draft) + manifest = self.manifest(draft) + assert set(manifest) == set(draft.w.keys()), set(manifest) ^ set(draft.w.keys()) + all_keys = set(weights.keys()) + expected = self.expected_unconsumed(draft, weights) + consumed: set = set() + for param_key, sources in manifest.items(): + for ckpt, index, transform in sources: + _fill(draft.w[param_key], ckpt, index, transform, weights, d, consumed) + _fill(draft.lm_head.weight, "lm_head.weight", None, _col_w, weights, d, consumed) + torch.cuda.synchronize() + leftover = all_keys - consumed + assert leftover == expected, ( + f"drafter checkpoint coverage: unexpected {sorted(leftover - expected)[:8]}, " + f"missed {sorted(expected - leftover)[:8]}" + ) + + +DRAFT_WEIGHTS = MiniMaxM3Tp8DraftWeights() diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py index 03977da8e8af..20bb8f2682cc 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Route the MiniMax-M3.1 sm_103 TP4/TP8 targets; other shapes decline.""" +"""Route the MiniMax-M3.1 sm_103 TP4/TP8 and sm_107 TP2/TP4/TP8 targets; other shapes decline.""" from __future__ import annotations @@ -14,6 +14,7 @@ TEXT_ONLY = True _SM = (10, 3) +_SM_RUBIN = (10, 7) _SHAPE = (60, 6144, 128, 3072, 64, 4) _M31_ROPE_THETA = 10_000_000 _M31_SPARSE_ATTENTION_FREQ = (1,) * 60 @@ -91,10 +92,10 @@ def _parallel(m) -> Optional[str]: Attention is head-split and the routed experts are tensor-parallel with every expert on every rank: the expert split decides which expert weights - each rank loads and into which shapes (per-rank expert widths 768 at TP4 and 384 at TP8), + each rank loads and into which shapes (per-rank expert widths 1536 at TP2, 768 at TP4 and 384 at TP8), so it selects a target rather than a runtime branch. """ - for tp in (4, 8): + for tp in (2, 4, 8): if ( m.world_size == tp and m.tp_size == tp @@ -108,6 +109,14 @@ def _parallel(m) -> Optional[str]: return None +def _m31_tp2_routed() -> bool: + """Whether the MiniMax-M3.1 sm_107 TP2 decode target takes its topology. On by default; + ``TRTLLM_M31_DISABLE_TP2=1`` keeps the built-in model.""" + from .minimax_m31_nvfp4__sm_107__tp2.m31_switches import m31_disabled + + return not m31_disabled("TP2") + + def _m31_speculation_mismatch(spec_config: Optional[Any]) -> Optional[str]: """Why the speculation is not one the M3.1 targets serve, or None when it is (none, or a standalone MiniMax DSpark drafter). Read from the drafter checkpoint's ``config.json``.""" @@ -198,8 +207,34 @@ def route( return ModelingV2MiniMaxM31Nvfp4Sm103Tp8 + # The same export on 2, 4 or 8 Rubin GPUs (sm_107), on the same terms (TP2: moe_tp 2, every expert on both ranks). + if ( + compute_capability == _SM_RUBIN + and shape == _SHAPE + and m31_miss is None + and parallel in ("tp2", "tp4", "tp8") + and (parallel != "tp2" or _m31_tp2_routed()) + and m31_spec_miss is None + and kv_cache + ): + from .fake_ops import register + + register() + if parallel == "tp2": + from .minimax_m31_nvfp4__sm_107__tp2.modeling import ModelingV2MiniMaxM31Nvfp4Sm107Tp2 + + return ModelingV2MiniMaxM31Nvfp4Sm107Tp2 + if parallel == "tp4": + from .minimax_m31_nvfp4__sm_107__tp4.modeling import ModelingV2MiniMaxM31Nvfp4Sm107Tp4 + + return ModelingV2MiniMaxM31Nvfp4Sm107Tp4 + from .minimax_m31_nvfp4__sm_107__tp8.modeling import ModelingV2MiniMaxM31Nvfp4Sm107Tp8 + + return ModelingV2MiniMaxM31Nvfp4Sm107Tp8 + raise NoTarget( - f"MiniMax-M3.1 requires sm_103 TP4 or TP8/EP1, shape {_SHAPE}, MXFP8/NVFP4 experts, " + f"MiniMax-M3.1 requires sm_103 TP4 or TP8/EP1 or sm_107 TP2, TP4 or TP8/EP1, shape {_SHAPE}, " + "MXFP8/NVFP4 experts, " f"FP8 or NVFP4 KV, and standalone MiniMax DSpark or no speculation; " f"got SM={compute_capability}, shape={shape}, topology={parallel}, " f"KV={kv_cache_quant_algo}, checkpoint={m31_miss}, speculation={m31_spec_miss}" diff --git a/tensorrt_llm/_torch/attention/backends/fmha/flashinfer_trtllm_gen.py b/tensorrt_llm/_torch/attention/backends/fmha/flashinfer_trtllm_gen.py index 1f43ae3bb385..b7e0a50c60a3 100644 --- a/tensorrt_llm/_torch/attention/backends/fmha/flashinfer_trtllm_gen.py +++ b/tensorrt_llm/_torch/attention/backends/fmha/flashinfer_trtllm_gen.py @@ -62,6 +62,7 @@ from tensorrt_llm.quantization.mode import QuantMode from .interface import FmhaPhase +from .msa_decode import dense_decode_max_seq_len from .phased import FmhaParams, PhasedFmha from .utils import ( get_attention_chunk_size, @@ -1163,6 +1164,8 @@ def run_generation( bmm1_scale if fp8_context_fmha and bmm1_scale is not None else bmm1_scale_static ) gen_bmm2_scale = bmm2_scale if fp8_context_fmha and bmm2_scale is not None else 1.0 + # A layer may cap the KV split's max_seq_len per request count (a no-op unless it sets the cap). + max_kv_len = dense_decode_max_seq_len(attn, max_kv_len, batch_beam) flashinfer.decode.trtllm_batch_decode_with_kv_cache( query=q_processed, diff --git a/tensorrt_llm/_torch/attention/backends/fmha/msa_decode.py b/tensorrt_llm/_torch/attention/backends/fmha/msa_decode.py index c6c6af4d08cd..ba2f8b00432a 100644 --- a/tensorrt_llm/_torch/attention/backends/fmha/msa_decode.py +++ b/tensorrt_llm/_torch/attention/backends/fmha/msa_decode.py @@ -22,7 +22,7 @@ from __future__ import annotations import math -from typing import TYPE_CHECKING, Optional +from typing import TYPE_CHECKING, Callable, Optional import torch @@ -39,6 +39,21 @@ from ..sparse.minimax_m3.kernels.trtllm_gen_dense_decode import DenseDecodeWorkspaceLayout +def dense_decode_max_seq_len(attn, max_seq_len: int, num_requests: int) -> int: + """The max_seq_len trtllm-gen dense decode is handed for a step of ``num_requests`` requests. + + A model may cap it per request count through its attention module's + ``dense_decode_max_seq_len_by_batch`` ({requests: max_seq_len}). The value + only sizes the kernel's multi-CTA KV split: every CTA still covers its share + of each request's actual length, so a cap changes the result only by the + split-merge order. The request count is fixed per CUDA graph, so the choice + is graph-safe. + """ + caps = getattr(attn, "dense_decode_max_seq_len_by_batch", None) + cap = caps.get(num_requests) if caps else None + return max_seq_len if cap is None else min(max_seq_len, int(cap)) + + class MsaDecodeFmha(PhasedFmha): """MiniMax-M3 generation attention on the Triton and trtllm-gen kernels. @@ -52,6 +67,10 @@ def __init__(self, attn: "TrtllmAttention"): # Where this layer's trtllm-gen scratch sits inside the shared # attention workspace, settled by prepare_workspace each step. self._dense_layout: Optional[DenseDecodeWorkspaceLayout] = None + # Optional sparse decode a model target may register for this layer's + # generation rows: called with minimax_m3_sparse_attn_decode's + # arguments in its place. Unset, nothing changes. + self.sparse_decode_hook: Optional[Callable[..., None]] = None @classmethod def _is_available(cls, attn: "TrtllmAttention") -> bool: @@ -201,7 +220,8 @@ def _run_sparse( ) # q may still be FP8 from a fused producer; the kernel widens it # in-register, so it is passed through as it arrives. - minimax_m3_sparse_attn_decode( + decode = self.sparse_decode_hook or minimax_m3_sparse_attn_decode + decode( params.query_input.view(num_tokens, attn.num_heads, head_dim), k_paged, v_paged, @@ -261,7 +281,9 @@ def _run_dense( sm_scale=(head_dim**-0.5) / float(attn.q_scaling), output=params.output.view(num_tokens, attn.num_heads, head_dim), decode_query_len=params.input_seq_length, - max_seq_len=int(metadata.msa_max_kv_len), + max_seq_len=dense_decode_max_seq_len( + attn, int(metadata.msa_max_kv_len), int(seq_lens.shape[0]) + ), max_num_requests=int(metadata.max_num_requests), staged_subpage_table=staged_table, staged_subpages_per_slot=staged_factor, @@ -270,4 +292,4 @@ def _run_dense( ) -__all__ = ["MsaDecodeFmha"] +__all__ = ["MsaDecodeFmha", "dense_decode_max_seq_len"] diff --git a/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py b/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py index a560980ad860..a116c5c43632 100644 --- a/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py +++ b/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py @@ -22,10 +22,13 @@ from __future__ import annotations import functools -from typing import TYPE_CHECKING, Optional +import inspect +from typing import TYPE_CHECKING, Callable, Optional import torch +from tensorrt_llm.logger import logger + from .interface import FmhaPhase from .phased import FmhaParams, PhasedFmha @@ -178,6 +181,307 @@ def _msa_sparse_api(): return sparse +_TP2CTX_Q8KV4_STATE: dict = {} + + +def _tp2ctx_q8kv4_run(): + """fmha_sm100.prefill_q8kv4.run_prefill (overlay v3, forward_only) when TP2CTX_Q8KV4 may run, else None. + + Option value, opt-in only: TRTLLM_M31_ENABLE_TP2CTX_Q8KV4=1 turns it on, and + TRTLLM_M31_DISABLE_TP2CTX_Q8KV4=1 (read per call) keeps it off regardless. The package probe is cached. + """ + import inspect + import os + + if os.environ.get("TRTLLM_M31_ENABLE_TP2CTX_Q8KV4", "0") != "1": + return None + if os.environ.get("TRTLLM_M31_DISABLE_TP2CTX_Q8KV4", "0") == "1": + return None + if "run" not in _TP2CTX_Q8KV4_STATE: + run = None + if torch.cuda.get_device_capability() in ((10, 3), (10, 7)): + try: + from fmha_sm100.prefill_q8kv4 import run_prefill as run + + if "forward_only" not in inspect.signature(run).parameters: + run = None # an overlay older than v3 would combine without C1 / C2 + except ImportError: + run = None + _TP2CTX_Q8KV4_STATE["run"] = run + return _TP2CTX_Q8KV4_STATE["run"] + + +def _run_tp2ctx_q8kv4( + run, + q, + k_paged, + v_paged, + scale_buffers, + metadata, + batch, + cu_q, + cu_kv, + k2q_row_ptr, + schedule, + sm_scale, + topk, + k_global_scale, + v_global_scale, + out, + in_place, + mxfp8_sink, +) -> None: + """TP2CTX_Q8KV4: the Q8KV4 forward into split partials, then the installed combine as the DSL path calls it.""" + from src.sm100.fwd.combine import combine + + if not _TP2CTX_Q8KV4_STATE.get("logged"): + _TP2CTX_Q8KV4_STATE["logged"] = True + logger.info( + "[M3 dispatch] TP2CTX_Q8KV4: Q8KV4 C++ sparse prefill forward (E4M3 Q) + installed combine" + ) + q8 = q if q.dtype == torch.float8_e4m3fn else q.to(torch.float8_e4m3fn) + o_partial, lse_partial = run( + q8, + (k_paged.view(torch.uint8), v_paged.view(torch.uint8)), + (scale_buffers[:, 0].view(torch.uint8), scale_buffers[:, 1].view(torch.uint8)), + kv_indices=metadata.msa_block_table[:batch], + kv_indptr=None, + cu_seqlens_q=cu_q, + cu_seqlens_k=cu_kv, + k2q_row_ptr=k2q_row_ptr, + schedule=schedule, + sm_scale=sm_scale, + topk=topk, + seqused_k=metadata.msa_seq_lens_cuda[:batch], + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + forward_only=True, + ) + write_in_place = ( + in_place + and out.dtype == torch.bfloat16 + and out.dim() == 3 + and out.is_contiguous() + and not any(_overlaps(out, other) for other in (q, k_paged, v_paged)) + ) + o_out = ( + out + if write_in_place + else torch.empty( + (int(q.shape[0]), int(q.shape[1]), int(q.shape[2])), + dtype=torch.bfloat16, + device=q.device, + ) + ) + kwargs = dict( + cu_seqlens=cu_q, split_counts=schedule.split_counts, use_pdl=True + ) # no output_scale: vg is in K1 + mxfp8 = None + if ( + mxfp8_sink is not None + and int(out.shape[0]) >= getattr(mxfp8_sink, "min_rows", 0) + and write_in_place + and out.shape[-1] == 128 + and _combine_takes(combine, "o_fp8") + ): + rows, width = int(out.shape[0]), int(out.shape[1] * out.shape[2]) + rows_padded = (rows + 127) // 128 * 128 + # The combine writes the live rows' scales; the padding rows' scale bytes must be zero (MSA contract). + # A mixed-step caller owns C+G buffers and supplies only their context slices here. + # Returning None declines the sink (for example inside a breakable graph boundary). + allocate = getattr(mxfp8_sink, "allocate", None) + if allocate is not None: + mxfp8 = allocate(out.view(rows, width)) + else: + mxfp8 = ( + torch.empty((rows, width), dtype=torch.uint8, device=out.device), + torch.zeros((rows_padded * width // 32,), dtype=torch.uint8, device=out.device), + ) + if mxfp8 is not None: + kwargs.update(o_fp8=mxfp8[0], sf_out=mxfp8[1]) + combine(o_partial, lse_partial, o_out, None, **kwargs) + if not write_in_place: + out.copy_(o_out) + elif mxfp8 is not None: + mxfp8_sink(out.view(out.shape[0], -1), mxfp8[0].view(torch.float8_e4m3fn), mxfp8[1]) + + +@functools.lru_cache(maxsize=None) +def _combine_takes(fn, name: str) -> bool: + """Whether the installed combine accepts keyword ``name`` (C1's o_fp8 / sf_out).""" + import inspect + + return name in inspect.signature(fn).parameters + + +_TP2CTX_BF16X_STATE: dict = {"taken": 0, "skip": {}, "q8kv4_taken": 0, "e4m3x_taken": 0} + + +def _tp2ctx_bf16x_run(): + """(run, reason): fmha_sm100.prefill_bf16x.run_prefill (BF16X fork m3b) when TP2CTX_BF16X may run here, else None. + + Default ON; TRTLLM_M31_DISABLE_TP2CTX_BF16X=1 (read per call) keeps the CuTe-DSL forward. The probe is cached. + """ + import os + + if os.environ.get("TRTLLM_M31_DISABLE_TP2CTX_BF16X", "0") == "1": + return None, "disabled_by_env" + if "run" not in _TP2CTX_BF16X_STATE: + run, reason = None, None + if torch.cuda.get_device_capability() != (10, 7): # certified on VR + reason = "not_sm107" + else: + try: + from fmha_sm100.prefill_bf16x import run_prefill as run + except ImportError: + run, reason = None, "no_prefill_bf16x_package" + _TP2CTX_BF16X_STATE["run"] = (run, reason) + return _TP2CTX_BF16X_STATE["run"] + + +def _tp2ctx_counters_atexit() -> None: + """Register the exit-time final-count lines once per process.""" + if not _TP2CTX_BF16X_STATE.get("atexit"): + import atexit + + _TP2CTX_BF16X_STATE["atexit"] = True + atexit.register(_tp2ctx_bf16x_final) + + +def _tp2ctx_e4m3x_ok(): + """None when TP2CTX_E4M3X may take an fp8-q call, else the skip reason. + + TRTLLM_M31_DISABLE_TP2CTX_E4M3X=1 (read per call) keeps the CuTe-DSL fp8 forward. The loaded fork must be an E4M3X + build (its E4M3 path the exact DSL transcription), else the upstream Q8KV4 kernel would run: probed once. + """ + import os + + if os.environ.get("TRTLLM_M31_DISABLE_TP2CTX_E4M3X", "0") == "1": + return "e4m3x_disabled_by_env" + if "e4m3x_build" not in _TP2CTX_BF16X_STATE: + try: + from fmha_sm100.prefill_bf16x import jit as _bf16x_jit + + _TP2CTX_BF16X_STATE["e4m3x_build"] = bool(getattr(_bf16x_jit, "_E4M3X", False)) + except ImportError: + _TP2CTX_BF16X_STATE["e4m3x_build"] = False + return None if _TP2CTX_BF16X_STATE["e4m3x_build"] else "fork_not_e4m3x" + + +def _tp2ctx_e4m3x_count() -> None: + """TP2CTX_E4M3X calls that returned, logged at counts 1, 2, 4, 8, ...""" + _tp2ctx_counters_atexit() + _TP2CTX_BF16X_STATE["e4m3x_taken"] += 1 + n = _TP2CTX_BF16X_STATE["e4m3x_taken"] + if n & (n - 1) == 0: + logger.info(f"[M3 dispatch] TP2CTX_E4M3X taken n={n}") + + +def _tp2ctx_q8kv4_count() -> None: + """TP2CTX_Q8KV4 calls that returned, logged at counts 1, 2, 4, 8, ...""" + _tp2ctx_counters_atexit() + _TP2CTX_BF16X_STATE["q8kv4_taken"] += 1 + n = _TP2CTX_BF16X_STATE["q8kv4_taken"] + if n & (n - 1) == 0: + logger.info(f"[M3 dispatch] TP2CTX_Q8KV4 taken n={n}") + + +def _tp2ctx_bf16x_count(reason=None) -> None: + """Taken (after the call returned) / skip counters, logged at counts 1, 2, 4, 8, ... + + The last power-of-two line is a lower bound of the final count; the exact one is the exit line, which a killed + process may not log. + """ + _tp2ctx_counters_atexit() + if reason is None: + _TP2CTX_BF16X_STATE["taken"] += 1 + n = _TP2CTX_BF16X_STATE["taken"] + if n & (n - 1) == 0: + logger.info(f"[M3 dispatch] TP2CTX_BF16X taken n={n}") + return + skips = _TP2CTX_BF16X_STATE["skip"] + skips[reason] = n = skips.get(reason, 0) + 1 + if n & (n - 1) == 0: + logger.info(f"[M3 skip] TP2CTX_BF16X reason={reason} n={n}") + + +def _tp2ctx_bf16x_final() -> None: + """At process exit: the exact final taken / skip counts (the power-of-two lines only bound them).""" + logger.info( + f"[M3.1 prefill-fwd] TP2CTX_BF16X final taken={_TP2CTX_BF16X_STATE['taken']} " + f"skip={dict(sorted(_TP2CTX_BF16X_STATE['skip'].items()))}" + ) + logger.info(f"[M3.1 prefill-fwd] TP2CTX_Q8KV4 final taken={_TP2CTX_BF16X_STATE['q8kv4_taken']}") + logger.info(f"[M3.1 prefill-fwd] TP2CTX_E4M3X final taken={_TP2CTX_BF16X_STATE['e4m3x_taken']}") + + +def _tp2ctx_fwd_path_log(q8kv4_run, bf16x_run, bf16x_reason) -> None: + """Once per process: which sparse prefill forward this process runs by default.""" + if _TP2CTX_BF16X_STATE.get("path_logged"): + return + _TP2CTX_BF16X_STATE["path_logged"] = True + if q8kv4_run is not None: + path = "Q8KV4 (opt-in TRTLLM_M31_ENABLE_TP2CTX_Q8KV4=1)" + elif bf16x_run is not None: + path = ( + "BF16X for bf16 q / E4M3X for fp8 q (default; TRTLLM_M31_DISABLE_TP2CTX_BF16X=1 or " + "TRTLLM_M31_DISABLE_TP2CTX_E4M3X=1 keeps the DSL)" + ) + else: + path = f"DSL (BF16X off: {bf16x_reason})" + logger.info(f"[M3.1 prefill-fwd] active path: {path}") + + +def _run_tp2ctx_bf16x( + run, + q, + k_paged, + v_paged, + scale_buffers, + metadata, + batch, + cu_q, + cu_kv, + k2q_row_ptr, + schedule, + sm_scale, + topk, + k_global_scale, + v_global_scale, + out, + write_in_place, + mxfp8=None, + mxfp8_sink=None, +) -> None: + """TP2CTX_BF16X: the BF16X forward into split partials + the installed combine (output_scale = v_global_scale). + + ``mxfp8`` (TP2CTX_COMBINE_MXFP8 on the E4M3X / BF16X fork, in-place output only): the fork's run_prefill forwards + ``o_fp8`` / ``sf_out`` to the combine, which also writes the MXFP8 copy of the final output (bitwise + ``mxfp8_quantize(out.view(rows, -1), True)``); the pair is then handed to ``mxfp8_sink`` for o_proj.""" + result, _ = run( + q, + (k_paged.view(torch.uint8), v_paged.view(torch.uint8)), + (scale_buffers[:, 0].view(torch.uint8), scale_buffers[:, 1].view(torch.uint8)), + kv_indices=metadata.msa_block_table[:batch], + kv_indptr=None, + cu_seqlens_q=cu_q, + cu_seqlens_k=cu_kv, + k2q_row_ptr=k2q_row_ptr, + schedule=schedule, + sm_scale=sm_scale, + topk=topk, + seqused_k=metadata.msa_seq_lens_cuda[:batch], + k_global_scale=k_global_scale, + v_global_scale=v_global_scale, + out=out if write_in_place else None, + **({"o_fp8": mxfp8[0], "sf_out": mxfp8[1]} if mxfp8 is not None else {}), + ) + if not write_in_place: + out.copy_(result) + elif mxfp8 is not None: + mxfp8_sink(out.view(out.shape[0], -1), mxfp8[0].view(torch.float8_e4m3fn), mxfp8[1]) + + def run_msa_nvfp4_sparse_gqa( q: torch.Tensor, k_paged: torch.Tensor, @@ -191,6 +495,9 @@ def run_msa_nvfp4_sparse_gqa( v_global_scale: torch.Tensor, num_rows: Optional[int] = None, out: torch.Tensor, + in_place: bool = False, + schedule_order: Optional[str] = None, + mxfp8_sink: Optional[Callable[[torch.Tensor, torch.Tensor, torch.Tensor], None]] = None, ) -> None: """Run sparse attention over M3's packed NVFP4 cache with the MSA CSR kernel. @@ -198,6 +505,19 @@ def run_msa_nvfp4_sparse_gqa( tiles 128 query rows at a time. num_rows narrows it to the leading batch rows, so a mixed step can hand its generation suffix to the Triton NVFP4 decode and size this worklist from the context prefix alone. + + in_place (an attention layer opts in with ``msa_sparse_out_in_place``): the + MSA combine writes the result into ``out`` itself when the loaded build + accepts a caller-owned output, instead of returning a tensor this copies. + + schedule_order (an attention layer opts in with ``msa_k2q_schedule_order``): the order of the k2q CSR schedule's + work items, passed to the MSA build_k2q_csr when it accepts it (e.g. ``"lpt"``: heaviest item first). The order + only changes which CTA runs which item, never what an item computes or where it writes, so the output is the same. + + mxfp8_sink (an attention layer opts in with ``msa_sparse_out_mxfp8_sink``; requires the in-place output): the MSA + combine also writes the output as MXFP8 (E4M3 [rows, heads * dim] + swizzled 128x4 K32 UE8M0 scales, bitwise + ``mxfp8_quantize(out.view(rows, -1), True)``) when the loaded build accepts ``o_fp8`` / ``sf_out``, and the pair is + handed to ``mxfp8_sink(out_2d, o_fp8, sf_out)`` so the MXFP8 o_proj can skip its own quantize (COMBINE_MXFP8). """ from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( MSA_REQUIRED_TOPK, @@ -238,24 +558,161 @@ def run_msa_nvfp4_sparse_gqa( ) cu_q = metadata.msa_cu_q_lens[: batch + 1] cu_kv = metadata.msa_cu_kv_lens[: batch + 1] - q2k = kv_block_indexes.permute(1, 0, 2).contiguous() - topk = int(q2k.shape[-1]) + topk = int(kv_block_indexes.shape[-1]) page_size = int(k_paged.shape[2]) if topk != MSA_REQUIRED_TOPK: raise ValueError(f"MiniMax-M3 MSA NVFP4 requires topK={MSA_REQUIRED_TOPK}, got {topk}") - k2q_row_ptr, k2q_q_indices, schedule = sparse.build_k2q_csr( - q2k, - cu_q, - cu_kv, - page_size, - total_k=total_k, - max_seqlen_k=max_kv_len, - max_seqlen_q=max_q_len, - total_rows=total_k_rows, - qhead_per_kv=int(q.shape[1]) // int(k_paged.shape[1]), - return_schedule=True, - ) + # A short pure-prefill step hands every layer the same identity selection (staged once per step by the + # backend), so its k2q CSR worklist is layer-invariant: build it for the first layer and reuse it. + identity = metadata._msa_prefill_identity + csr_key = (batch, int(kv_block_indexes.shape[0])) + cached = metadata._msa_identity_csr + if ( + identity is not None + and cached is not None + and cached[0] is identity + and cached[1] == csr_key + ): + k2q_row_ptr, k2q_q_indices, schedule = cached[2] + else: + q2k = kv_block_indexes.permute(1, 0, 2).contiguous() + k2q_row_ptr, k2q_q_indices, schedule = sparse.build_k2q_csr( + q2k, + cu_q, + cu_kv, + page_size, + total_k=total_k, + max_seqlen_k=max_kv_len, + max_seqlen_q=max_q_len, + total_rows=total_k_rows, + qhead_per_kv=int(q.shape[1]) // int(k_paged.shape[1]), + return_schedule=True, + **( + {"schedule_order": schedule_order} + if schedule_order is not None and _csr_builder_takes_order(sparse.build_k2q_csr) + else {} + ), + ) + if identity is not None: + metadata._msa_identity_csr = (identity, csr_key, (k2q_row_ptr, k2q_q_indices, schedule)) + q8kv4_run = _tp2ctx_q8kv4_run() + bf16x_run, bf16x_reason = _tp2ctx_bf16x_run() + _tp2ctx_fwd_path_log(q8kv4_run, bf16x_run, bf16x_reason) + if ( + q8kv4_run is not None + and page_size == 128 + and int(q.shape[1]) == 16 * int(k_paged.shape[1]) + and int(q.shape[2]) == 128 + ): + _run_tp2ctx_q8kv4( + q8kv4_run, + q, + k_paged, + v_paged, + scale_buffers, + metadata, + batch, + cu_q, + cu_kv, + k2q_row_ptr, + schedule, + sm_scale, + topk, + k_global_scale, + v_global_scale, + out, + in_place, + mxfp8_sink, + ) + _tp2ctx_q8kv4_count() + return + write_in_place = in_place and _sparse_func_takes_out(sparse.sparse_atten_nvfp4_kv_func) + if in_place and not _SPARSE_OUT_LOGGED: + _SPARSE_OUT_LOGGED.append(True) + logger.info( + "[M3.1 tp2ctx] SPARSE_OUT in-place: " + f"{'on' if write_in_place else 'off (the loaded MSA build takes no out=)'}" + ) + mxfp8 = None + # TP2CTX_COMBINE_MXFP8: the old NVFP4 sparse func and the E4M3X / BF16X fork (run_prefill o_fp8 / sf_out, + # forwarded to the combine) each take the MXFP8 sink only when their loaded build accepts it. With a fork loaded the + # sink follows the fork: a fork without o_fp8 keeps the call on the fork (o_proj then quantizes as stock) instead of + # falling off to the slower DSL path for the sink. + sink_old = _sparse_func_takes(sparse.sparse_atten_nvfp4_kv_func, "o_fp8") + sink_fork = bf16x_run is not None and _combine_takes(bf16x_run, "o_fp8") + sink_ok = sink_fork if bf16x_run is not None else sink_old + # A sink may set ``min_rows``: shorter calls skip the MXFP8 copy (the extra combine epilogue costs more than + # o_proj's quantize saves there) and o_proj quantizes as stock. + if ( + mxfp8_sink is not None + and int(out.shape[0]) >= getattr(mxfp8_sink, "min_rows", 0) + and write_in_place + and out.dtype == torch.bfloat16 + and out.dim() == 3 + and out.shape[-1] == 128 + and out.is_contiguous() + and sink_ok + ): + rows, width = int(out.shape[0]), int(out.shape[1] * out.shape[2]) + rows_padded = (rows + 127) // 128 * 128 + # The combine writes the live rows' scales; the padding rows' scale bytes must be zero (MSA contract). + allocate = getattr(mxfp8_sink, "allocate", None) + if allocate is not None: + mxfp8 = allocate(out.view(rows, width)) + else: + mxfp8 = ( + torch.empty((rows, width), dtype=torch.uint8, device=out.device), + torch.zeros((rows_padded * width // 32,), dtype=torch.uint8, device=out.device), + ) + if write_in_place: + # The combine writes out after the forward has read q and the paged K/V; an out that shares their memory + # (a view of a fused QKV buffer) would depend on that kernel order, so it is refused. + for name, other in (("q", q), ("K", k_paged), ("V", v_paged)): + if _overlaps(out, other): + raise ValueError(f"MiniMax-M3 NVFP4 in-place sparse output aliases {name}") + if bf16x_run is not None: + if mxfp8 is not None and not sink_fork: + bf16x_reason = "c1_mxfp8_sink" + elif q.dtype == torch.float8_e4m3fn and _tp2ctx_e4m3x_ok() is not None: + bf16x_reason = _tp2ctx_e4m3x_ok() + elif q.dtype not in (torch.bfloat16, torch.float8_e4m3fn): + bf16x_reason = "q_dtype" + elif page_size != 128 or int(q.shape[2]) != 128: + bf16x_reason = "page_or_head_dim" + elif int(q.shape[1]) != 16 * int(k_paged.shape[1]): + bf16x_reason = "not_16_q_heads_per_kv" + else: + bf16x_reason = None + _run_tp2ctx_bf16x( + bf16x_run, + q, + k_paged, + v_paged, + scale_buffers, + metadata, + batch, + cu_q, + cu_kv, + k2q_row_ptr, + schedule, + sm_scale, + topk, + k_global_scale, + v_global_scale, + out, + write_in_place, + mxfp8=mxfp8, + mxfp8_sink=mxfp8_sink, + ) + if q.dtype == torch.float8_e4m3fn: + _tp2ctx_e4m3x_count() + else: + _tp2ctx_bf16x_count() + return + _tp2ctx_bf16x_count(bf16x_reason) + if mxfp8 is not None and not sink_old: + mxfp8 = None result = sparse.sparse_atten_nvfp4_kv_func( q, k_paged.view(torch.uint8), @@ -279,8 +736,77 @@ def run_msa_nvfp4_sparse_gqa( page_table=metadata.msa_block_table[:batch], seqused_k=metadata.msa_seq_lens_cuda[:batch], schedule=schedule, + **({"out": out} if write_in_place else {}), + **({"o_fp8": mxfp8[0], "sf_out": mxfp8[1]} if mxfp8 is not None else {}), ) - out.copy_(result) + if not write_in_place: + out.copy_(result) + elif mxfp8 is not None: + mxfp8_sink(out.view(out.shape[0], -1), mxfp8[0].view(torch.float8_e4m3fn), mxfp8[1]) + + +def _overlaps(a: torch.Tensor, b: torch.Tensor) -> bool: + """Whether the byte ranges a and b span intersect (host-side pointer arithmetic).""" + + def span(t): + start = t.data_ptr() + extent = ( + 1 + sum((size - 1) * stride for size, stride in zip(t.shape, t.stride())) + if t.numel() + else 0 + ) + return start, start + extent * t.element_size() + + a0, a1 = span(a) + b0, b1 = span(b) + return a0 < b1 and b0 < a1 + + +_SPARSE_OUT_LOGGED: list = [] # one engagement line per process (an attention layer opted in) +_SPARSE_FUNC_TAKES_OUT: dict = {} + + +_SPARSE_FUNC_TAKES: dict = {} + + +def _sparse_func_takes(func, name: str) -> bool: + """Whether the loaded MSA sparse_atten_nvfp4_kv_func accepts the keyword ``name`` (decided once per function).""" + key = (func, name) + takes = _SPARSE_FUNC_TAKES.get(key) + if takes is None: + try: + takes = name in inspect.signature(func).parameters + except (TypeError, ValueError): + takes = False + _SPARSE_FUNC_TAKES[key] = takes + return takes + + +def _sparse_func_takes_out(func) -> bool: + """Whether the loaded MSA sparse_atten_nvfp4_kv_func accepts a caller-owned ``out`` (decided once per function).""" + takes = _SPARSE_FUNC_TAKES_OUT.get(func) + if takes is None: + try: + takes = "out" in inspect.signature(func).parameters + except (TypeError, ValueError): + takes = False + _SPARSE_FUNC_TAKES_OUT[func] = takes + return takes + + +_CSR_BUILDER_TAKES_ORDER: dict = {} + + +def _csr_builder_takes_order(func) -> bool: + """Whether the loaded MSA build_k2q_csr accepts ``schedule_order`` (decided once per function).""" + takes = _CSR_BUILDER_TAKES_ORDER.get(func) + if takes is None: + try: + takes = "schedule_order" in inspect.signature(func).parameters + except (TypeError, ValueError): + takes = False + _CSR_BUILDER_TAKES_ORDER[func] = takes + return takes def run_msa_prefill_gqa( @@ -337,6 +863,9 @@ def run_msa_prefill_gqa( v_global_scale=v_scale, num_rows=num_rows, out=out_view, + in_place=getattr(attn, "msa_sparse_out_in_place", False), + schedule_order=getattr(attn, "msa_k2q_schedule_order", None), + mxfp8_sink=getattr(attn, "msa_sparse_out_mxfp8_sink", None), ) return diff --git a/tensorrt_llm/_torch/attention/backends/prims_ts/__init__.py b/tensorrt_llm/_torch/attention/backends/prims_ts/__init__.py index e40d8ce0d41c..c49ad2fc908d 100644 --- a/tensorrt_llm/_torch/attention/backends/prims_ts/__init__.py +++ b/tensorrt_llm/_torch/attention/backends/prims_ts/__init__.py @@ -14,6 +14,8 @@ """Experimental task-scheduled attention entry points.""" +from . import _worktile_scope # noqa: F401 (first: scopes task scheduling's WorkTileInfo rebind) + from .block_sparse import ( BlockSparsePagedTSWrapper, BlockSparseTSWrapper, diff --git a/tensorrt_llm/_torch/attention/backends/prims_ts/_block_sparse/compiler.py b/tensorrt_llm/_torch/attention/backends/prims_ts/_block_sparse/compiler.py index 1a9d42857f94..d557504e2651 100644 --- a/tensorrt_llm/_torch/attention/backends/prims_ts/_block_sparse/compiler.py +++ b/tensorrt_llm/_torch/attention/backends/prims_ts/_block_sparse/compiler.py @@ -14,6 +14,7 @@ """Compilation and caching for PrimTS block-sparse attention adapters.""" +from .._worktile_scope import with_scalar_work_tile_info from collections.abc import Callable import functools @@ -29,6 +30,7 @@ _COMPILE_OPTIONS = "--enable-tvm-ffi --opt-level 3" +@with_scalar_work_tile_info def _compile_block_sparse(key: _BlockSparseCompileKey) -> Callable[..., object]: """Compile one contiguous or paged attention adapter for ``key``. diff --git a/tensorrt_llm/_torch/attention/backends/prims_ts/_worktile_scope.py b/tensorrt_llm/_torch/attention/backends/prims_ts/_worktile_scope.py new file mode 100644 index 000000000000..45fd51f97d97 --- /dev/null +++ b/tensorrt_llm/_torch/attention/backends/prims_ts/_worktile_scope.py @@ -0,0 +1,105 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Keep CuTe DSL task scheduling's WorkTileInfo rebind inside PrimTS. + +Importing ``cutlass.experimental.task_scheduling`` rebinds six attributes of CUTLASS' +``WorkTileInfo`` (``__init__``, the two MLIR value hooks, ``tile_idx``, ``is_valid_tile`` and +``update_from``) to a scalar three-axis form, for every kernel in the process. Kernels that +subclass ``WorkTileInfo`` with a different tile shape (FA4's four-axis tile) then fail to trace. + +This module is imported first by the package. It snapshots the original attributes, imports +the task scheduling resources (which apply the rebind) and snapshots the scalar set. Until a model +calls ``arm()`` the rebind stays global, as before. Once armed, the originals are put back and +``scalar_work_tile_info`` installs the scalar set only while a PrimTS kernel traces and compiles; +compiled kernels do not call these Python methods. Nothing about the generated code changes. +Kill switch: TRTLLM_PRIMS_TS_SCOPED_WORKTILE=0 makes ``arm()`` keep the global rebind. +""" + +import contextlib +import functools +import os +import sys +import threading + +from cutlass.utils.static_persistent_tile_scheduler import WorkTileInfo as _WorkTileInfo + +_ATTRS = ( + "__init__", + "__extract_mlir_values__", + "__new_from_mlir_values__", + "tile_idx", + "is_valid_tile", + "update_from", +) +_MISSING = object() +_RESOURCES = "cutlass.experimental.task_scheduling.resources" + + +def _snapshot() -> dict: + return {name: _WorkTileInfo.__dict__.get(name, _MISSING) for name in _ATTRS} + + +def _install(attrs: dict) -> None: + for name, value in attrs.items(): + if value is _MISSING: + if name in _WorkTileInfo.__dict__: + delattr(_WorkTileInfo, name) + else: + setattr(_WorkTileInfo, name, value) + + +# Only scope the rebind when this module saw the originals, i.e. task scheduling was not imported +# before PrimTS. Otherwise keep whatever is installed (the previous global behavior). +_ORIGINAL = None if _RESOURCES in sys.modules else _snapshot() +import cutlass.experimental.task_scheduling.resources # noqa: E402,F401 (applies the scalar rebind) + +_SCALAR = _snapshot() +SCOPED = False +_LOCK = threading.RLock() +_DEPTH = 0 + + +def arm() -> bool: + """Scope the rebind to PrimTS traces from now on (a model opt-in); returns whether it is scoped. Keeps the + global rebind when task scheduling was imported before PrimTS or TRTLLM_PRIMS_TS_SCOPED_WORKTILE=0.""" + global SCOPED + with _LOCK: + if ( + not SCOPED + and _ORIGINAL is not None + and _DEPTH == 0 + and os.environ.get("TRTLLM_PRIMS_TS_SCOPED_WORKTILE", "1") != "0" + ): + SCOPED = True + _install(_ORIGINAL) + return SCOPED + + +@contextlib.contextmanager +def scalar_work_tile_info(): + """Install the task scheduling WorkTileInfo form while a PrimTS kernel traces (re-entrant).""" + global _DEPTH + if not SCOPED: + yield + return + with _LOCK: + if _DEPTH == 0: + _install(_SCALAR) + _DEPTH += 1 + try: + yield + finally: + _DEPTH -= 1 + if _DEPTH == 0: + _install(_ORIGINAL) + + +def with_scalar_work_tile_info(fn): + """Decorator form of ``scalar_work_tile_info`` for the PrimTS compile functions.""" + + @functools.wraps(fn) + def wrapper(*args, **kwargs): + with scalar_work_tile_info(): + return fn(*args, **kwargs) + + return wrapper diff --git a/tensorrt_llm/_torch/attention/backends/prims_ts/context.py b/tensorrt_llm/_torch/attention/backends/prims_ts/context.py index 1163324ec632..171c02224d12 100644 --- a/tensorrt_llm/_torch/attention/backends/prims_ts/context.py +++ b/tensorrt_llm/_torch/attention/backends/prims_ts/context.py @@ -35,6 +35,7 @@ register trace templates. """ +from ._worktile_scope import with_scalar_work_tile_info from dataclasses import dataclass import functools import importlib.metadata @@ -1753,6 +1754,7 @@ def _paged_context_compile_spec( @functools.cache +@with_scalar_work_tile_info def _get_compiled_context( compile_spec: _ContextCompileSpec, ): @@ -1973,6 +1975,7 @@ def fake_compact(dtype, shape, assumed_align): @functools.cache +@with_scalar_work_tile_info def _get_compiled_paged_context( compile_spec: _PagedContextCompileSpec, ): diff --git a/tensorrt_llm/_torch/attention/backends/prims_ts/decode.py b/tensorrt_llm/_torch/attention/backends/prims_ts/decode.py index 3fcec5efdc2a..d720bb765993 100644 --- a/tensorrt_llm/_torch/attention/backends/prims_ts/decode.py +++ b/tensorrt_llm/_torch/attention/backends/prims_ts/decode.py @@ -15,18 +15,23 @@ """Task-scheduled paged decode with a FlashInfer-style plan/run lifecycle.""" +from ._worktile_scope import with_scalar_work_tile_info from collections.abc import Callable, Sequence +import dataclasses from dataclasses import dataclass import functools import math import numbers import struct +from pathlib import Path from typing import TYPE_CHECKING, Literal, NamedTuple, Optional, Union, cast import torch from flashinfer.api_logging import flashinfer_api as flashinfer_experimental_api +from tensorrt_llm._torch import cute_aot + from . import _q_token_kv_block_sparse_policy as _sparse_policy from ._block_sparse.common import _num_sparse_pattern_heads @@ -1510,6 +1515,7 @@ def _make_decode_compile_spec( @functools.cache +@with_scalar_work_tile_info def _get_compiled_decode( compile_spec: _DecodeCompileSpec, ): @@ -1903,67 +1909,79 @@ def fake_compact(dtype, shape, assumed_align): ), ) + # The device index does not change the compiled code: one shipped object serves every device. + aot_spec = dataclasses.replace(compile_spec, device_index=0) with torch.cuda.device(device_index): - compiled_main = cute.compile( - main_tensor_adapter, - q_fake, - k_fake, - v_fake, - k_sf_fake, - v_sf_fake, - out_fake, - seq_lens_fake, - cu_seqlens_q_fake, - block_table_fake, - q_token_kv_block_sparse_page_memberships_fake, - partial_o_fake, - partial_stats_fake, - counter_fake, - attention_sinks_fake, - Int64(1), - Int64(1), - Int64(1), - Int64(1), - Int64(1), - Int64(1), - Int64(1), - Float32(1.0), - Float32(1.0), - direct_q1_fake, - stream_fake, - cfg, - seq_len_q, - num_qo_heads, - num_kv_heads, - head_dim, - max_kv_len, - max_active_clusters, - static_full_split_prefix, - static_native_uniform_kv, - options=_COMPILE_OPTIONS, - ) - compiled_reducer = None - if cfg.use_separate_reduction_kernel: - assert reduction_tensor_adapter is not None - compiled_reducer = cute.compile( - reduction_tensor_adapter, + compiled_main = cute_aot.compile_or_load( + f"prims_ts_decode.main {aot_spec!r}", + lambda: cute.compile( + main_tensor_adapter, + q_fake, + k_fake, + v_fake, + k_sf_fake, + v_sf_fake, out_fake, seq_lens_fake, cu_seqlens_q_fake, + block_table_fake, + q_token_kv_block_sparse_page_memberships_fake, partial_o_fake, partial_stats_fake, + counter_fake, attention_sinks_fake, + Int64(1), + Int64(1), + Int64(1), + Int64(1), + Int64(1), + Int64(1), + Int64(1), Float32(1.0), Float32(1.0), direct_q1_fake, stream_fake, cfg, + seq_len_q, num_qo_heads, num_kv_heads, head_dim, max_kv_len, + max_active_clusters, static_full_split_prefix, + static_native_uniform_kv, options=_COMPILE_OPTIONS, + ), + sources=_aot_sources(), + tvm_ffi=True, + ) + compiled_reducer = None + if cfg.use_separate_reduction_kernel: + assert reduction_tensor_adapter is not None + compiled_reducer = cute_aot.compile_or_load( + f"prims_ts_decode.reducer {aot_spec!r}", + lambda: cute.compile( + reduction_tensor_adapter, + out_fake, + seq_lens_fake, + cu_seqlens_q_fake, + partial_o_fake, + partial_stats_fake, + attention_sinks_fake, + Float32(1.0), + Float32(1.0), + direct_q1_fake, + stream_fake, + cfg, + num_qo_heads, + num_kv_heads, + head_dim, + max_kv_len, + static_full_split_prefix, + options=_COMPILE_OPTIONS, + ), + sources=_aot_sources(), + tvm_ffi=True, ) return compiled_main, compiled_reducer @@ -4519,3 +4537,9 @@ def batch_decode_with_paged_kv_cache( "validate_q_token_kv_block_sparse_group_size", "prepare_prims_ts_batch_decode_with_kv_cache", ] + + +def _aot_sources(): + """The Python files the PrimTS decode kernels are traced from: the ``sources`` of their shipped AOT objects.""" + return cute_aot.py_files(str(Path(__file__).resolve().parent)) + diff --git a/tensorrt_llm/_torch/attention/backends/prims_ts/kernels/fmha_context/fmha_resources.py b/tensorrt_llm/_torch/attention/backends/prims_ts/kernels/fmha_context/fmha_resources.py index c4d1b5512f92..c21b2c9f5ab8 100644 --- a/tensorrt_llm/_torch/attention/backends/prims_ts/kernels/fmha_context/fmha_resources.py +++ b/tensorrt_llm/_torch/attention/backends/prims_ts/kernels/fmha_context/fmha_resources.py @@ -1200,8 +1200,6 @@ def compute_coords( if cutlass.const_expr(self.cfg.has_uniform_varlen): seqlen_k = Int32(self.cfg.uniform_seq_len_k) else: - from .helpers_paged import _load_runtime_seq_len_kv - seqlen_k = _load_runtime_seq_len_kv( self.seqlens_kv, self.max_seq_len_kv, batch_coord ) @@ -1224,8 +1222,6 @@ def compute_coords( kv_request_begin = batch_coord * Int32(self.block_table_row_stride) kv_page_idx_ub = Int32(self.cfg.max_num_pages_per_seq_kv - 1) else: - from .helpers_paged import _load_block_table_row_bounds - kv_request_begin, kv_page_idx_ub = _load_block_table_row_bounds( Int32(self.block_table_row_stride), self.cfg, @@ -1301,13 +1297,9 @@ def compute_page_coords(self, stage_info: StageInfo) -> tuple[Int32, Int32, Int3 kv_request_begin = batch_coord * Int32(self.block_table_row_stride) kv_page_idx_ub = Int32(self.cfg.max_num_pages_per_seq_kv - 1) else: - from .helpers_paged import _load_runtime_seq_len_kv - cached_seqlen_kv = _load_runtime_seq_len_kv( self.seqlens_kv, self.max_seq_len_kv, batch_coord ) - from .helpers_paged import _load_block_table_row_bounds - kv_request_begin, kv_page_idx_ub = _load_block_table_row_bounds( Int32(self.block_table_row_stride), self.cfg, @@ -1744,8 +1736,6 @@ def _producer_load_page_offsets( kv_request_begin: Int32, kv_page_idx_ub: Int32, ) -> None: - from .helpers_paged import _resolve_kv_tile_idx_context - cfg = self.cfg # Context's K/V tile index is kv_tile_start + loop_offset; for V the # producer runs one tile ahead in TAIL, so reuse the same expression @@ -1836,8 +1826,6 @@ def cache_tile_page_ids( tile_offset: cutlass.Constexpr[int] = 0, ) -> cutlass.Array: """Retain one tile's page IDs after its SMEM window is released.""" - from .helpers_paged import _resolve_kv_tile_idx_context - tile_idx = _resolve_kv_tile_idx_context( stage_info, kv_tile_start, tile_offset=tile_offset ) @@ -5798,3 +5786,14 @@ def tma_store( prims.cp_async_bulk_commit_group() if cutlass.const_expr(self.cfg.gmem_o_store_wait_after_write): prims.cp_async_bulk_wait_group(0, read=True) + + +# helpers_paged imports FmhaConfig from this module, so it is imported here, +# after FmhaConfig is defined. The methods above call these helpers while being +# traced inside the kernel's runtime loops, where the CuTe DSL rejects a first +# import of a module; importing at module scope keeps those calls import-free. +from .helpers_paged import ( + _load_block_table_row_bounds, + _load_runtime_seq_len_kv, + _resolve_kv_tile_idx_context, +) diff --git a/tensorrt_llm/_torch/attention/backends/prims_ts/kernels/fmha_decode/block_sparse_inspect.py b/tensorrt_llm/_torch/attention/backends/prims_ts/kernels/fmha_decode/block_sparse_inspect.py index 5728a7c6d373..e5f8de86571f 100644 --- a/tensorrt_llm/_torch/attention/backends/prims_ts/kernels/fmha_decode/block_sparse_inspect.py +++ b/tensorrt_llm/_torch/attention/backends/prims_ts/kernels/fmha_decode/block_sparse_inspect.py @@ -26,6 +26,7 @@ summary; no route payload is constructed. """ +from ..._worktile_scope import with_scalar_work_tile_info import functools from collections.abc import Callable @@ -450,6 +451,7 @@ def _fake_compact( @functools.cache +@with_scalar_work_tile_info def compile_block_sparse_inspection( *, device_index: int, @@ -495,6 +497,7 @@ def compile_block_sparse_inspection( @functools.cache +@with_scalar_work_tile_info def compile_paged_block_sparse_metadata_inspection( *, device_index: int, diff --git a/tensorrt_llm/_torch/attention/backends/prims_ts/mla_decode.py b/tensorrt_llm/_torch/attention/backends/prims_ts/mla_decode.py index 3d5fb159a329..0e82e5330728 100644 --- a/tensorrt_llm/_torch/attention/backends/prims_ts/mla_decode.py +++ b/tensorrt_llm/_torch/attention/backends/prims_ts/mla_decode.py @@ -15,6 +15,7 @@ """Task-scheduled paged MLA decode with a plan/run lifecycle.""" +from ._worktile_scope import with_scalar_work_tile_info from collections.abc import Callable from dataclasses import dataclass, field import functools @@ -1044,6 +1045,7 @@ def _make_mla_decode_compile_spec( @functools.cache +@with_scalar_work_tile_info def _get_compiled_mla_decode( compile_spec: _MLADecodeCompileSpec, ): diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py index 93af5a432f8c..6e66595e7ee1 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/cache_manager.py @@ -28,6 +28,7 @@ from __future__ import annotations +import os from typing import TYPE_CHECKING, List, Optional, Sequence, Tuple import numpy as np @@ -234,6 +235,25 @@ def resolve_sparse_layer_schedule( return list(sparse_layer_ids), list(disable_index_value_layer_ids) +# Model opt-ins (RuntimeHooks.msa_kv_cache_fixes), armed process-wide by the KV-cache creator before any sizing or +# allocation (arm_msa_kv_cache_fixes); both off otherwise. +# KV pool zero-init: zero every KV pool view (packed K/V, NVFP4 block scales, paged index keys) once at allocation, so +# readers that consume a whole block or page (the full-context decode on a last page's tail) never turn uninitialized +# bytes into 0 * NaN = NaN. Kill switch: TRTLLM_M3_DISABLE_KV_POOL_ZERO_INIT=1. +_KV_POOL_ZERO_INIT = False +# Exact KV size estimate: price M3's real per-token footprint in the pre-allocation estimate (index-K on every sparse +# layer, FP8 dense and shared Eagle layers) instead of the generic all-NVFP4 K/V sum. Kill switch: +# TRTLLM_M3_DISABLE_KV_SIZE_ESTIMATE=1. +_EXACT_KV_SIZE_ESTIMATE = False + + +def arm_msa_kv_cache_fixes() -> None: + """Arm the KV pool zero-init and the exact KV size estimate in this process (RuntimeHooks.msa_kv_cache_fixes).""" + global _KV_POOL_ZERO_INIT, _EXACT_KV_SIZE_ESTIMATE + _KV_POOL_ZERO_INIT = os.environ.get("TRTLLM_M3_DISABLE_KV_POOL_ZERO_INIT", "0") != "1" + _EXACT_KV_SIZE_ESTIMATE = os.environ.get("TRTLLM_M3_DISABLE_KV_SIZE_ESTIMATE", "0") != "1" + + def derive_shared_draft_layout( num_layers: Optional[int], num_kv_heads, @@ -302,6 +322,13 @@ def extend_attention_op_pools_for_shared_draft_layers( ) +# NVFP4 index-K (indexer_kv_dtype="nvfp4", the agg opt-in): one 128-token page is the Q8KV4 decode indexer's packed +# layout, 8192 bytes of E2M1 codes (64 per token, low nibble = even element) then 1024 bytes of E4M3 16-element +# scales (8 per token), no global scale: 72 bytes per token, viewed as [pages, 1, 128, 72] uint8. +NVFP4_INDEX_K_DIM = 128 +NVFP4_INDEX_K_BYTES_PER_TOKEN = NVFP4_INDEX_K_DIM // 2 + NVFP4_INDEX_K_DIM // 16 + + class MiniMaxM3KVCacheManagerV2(KVCacheManagerV2): """KVCacheManagerV2 subclass with a V2-managed paged index-K cache per sparse layer. @@ -338,6 +365,7 @@ class MiniMaxM3KVCacheManagerV2(KVCacheManagerV2): # Decode-step shortcuts a model opts into (RuntimeHooks.msa); see configure_decode_shortcuts. skip_unchanged_block_offsets = False reuse_pool_views = False + block_ids_from_views = False _main_kv_mapper_kind = MapperKind.NHD # Virtual attention-op pools for shared draft layers: @@ -399,11 +427,15 @@ def __init__( self.disable_index_value_layer_ids = set(int(i) for i in disable_index_value_layer_ids) self.sparse_index_dim = sparse_index_dim self.indexer_kv_dtype = str(getattr(sparse_attention_config, "indexer_kv_dtype", "bf16")) - if self.indexer_kv_dtype not in ("bf16", "fp8"): + if self.indexer_kv_dtype not in ("bf16", "fp8", "nvfp4"): raise ValueError( - "MiniMax M3 indexer_kv_dtype must be 'bf16' or 'fp8', got " + "MiniMax M3 indexer_kv_dtype must be 'bf16', 'fp8' or 'nvfp4', got " f"{self.indexer_kv_dtype!r}." ) + if self.indexer_kv_dtype == "nvfp4" and sparse_index_dim != NVFP4_INDEX_K_DIM: + raise ValueError( + f"MiniMax M3 NVFP4 index-K requires sparse_index_dim {NVFP4_INDEX_K_DIM}, got {sparse_index_dim}." + ) super().__init__(*args, **kwargs) self._unchanged_copies: dict[int, _UnchangedCopy] = {} @@ -448,7 +480,7 @@ def __init__( self._index_k_buffers[layer_idx] = super().get_index_k_buffer( layer_idx, num_heads=1, - head_dim=self.sparse_index_dim, + head_dim=self._index_cache_row_elems(), dtype=torch_dtype, kv_layout=kv_layout, ) @@ -458,6 +490,47 @@ def __init__( dtype=torch_dtype, device=device, ) + # NVFP4 index-K: the fused producer writes E4M3 index-K at each token's main-pool slot into an FP8 tensor of + # the main pool's page count, so one such sink is shared by every sparse layer, and each layer's NVFP4 pages + # are packed from it after its producer (kernels/nvfp4_index_k.write_from_sink). The index-K pool must + # address the same pages as the main pool: the producer's slot is the index-K slot. + self._nvfp4_index_k_sink: Optional[torch.Tensor] = None + if self.indexer_kv_dtype == "nvfp4" and self._index_k_buffers: + main_pages = { + int(self.get_buffers(layer_idx, kv_layout).shape[0]) + for layer_idx in self._index_k_buffers + } + index_pages = { + int(view.shape[0]) for view in self._index_k_buffers.values() if view is not None + } + if len(main_pages) != 1 or index_pages != main_pages: + raise ValueError( + "MiniMax M3 NVFP4 index-K needs the index-K pool to address the main pool's pages: main pages " + f"{sorted(main_pages)}, index-K pages {sorted(index_pages)}" + ) + self._nvfp4_index_k_sink = torch.zeros( + (main_pages.pop(), 1, self.tokens_per_block, NVFP4_INDEX_K_DIM), + dtype=torch.float8_e4m3fn, + device=device, + ) + logger.info( + "[M3.1] AGG_NVFP4_INDEXK on: index-K pool NVFP4 (72 B per token, " + f"{len(self._index_k_buffers)} layer(s)), FP8 sink {tuple(self._nvfp4_index_k_sink.shape)}" + ) + if _KV_POOL_ZERO_INIT: + self._zero_kv_pools(kv_layout) + + def _zero_kv_pools(self, kv_layout: str) -> None: + """Zero every layer's K/V bytes, the NVFP4 block scales and the paged index keys once, before any request + runs. The pools are written only at live slots; every byte a kernel writes is unchanged.""" + for layer_idx in self.layer_offsets: + views = [self.get_buffers(layer_idx, kv_layout)] + if self.is_nvfp4_layer(layer_idx): + views.append(self.get_block_scale_buffers(layer_idx, kv_layout)) + views.append(self._index_k_buffers.get(layer_idx)) + for view in views: + if view is not None: + view.zero_() def _validate_speculative_config(self, spec_config: DecodingBaseConfig | None) -> None: # Dynamic-tree relocation moves packed K/V bytes without their NVFP4 @@ -503,7 +576,7 @@ def get_layer_bytes_per_token(self, local_layer_idx: int, data_role: Role): return 0 index_dtype = self._torch_dtype_for_index_cache() index_elem_bytes = torch.empty(0, dtype=index_dtype, device="cpu").element_size() - return self.sparse_index_dim * index_elem_bytes + return self._index_cache_row_elems() * index_elem_bytes if self.dtype != DataType.NVFP4 or has_index_k: cache_bytes = super().get_layer_bytes_per_token(local_layer_idx, data_role) @@ -527,6 +600,77 @@ def get_layer_bytes_per_token(self, local_layer_idx: int, data_role: Role): cache_bytes += self.get_layer_bytes_per_token(local_layer_idx, Role.INDEX_KEY) return cache_bytes + @classmethod + def get_cache_size_per_token(cls, model_config, mapping, num_layers=None, **kwargs): + """Pre-allocation bytes per token, including what the generic estimate leaves out. + + The base classmethod sums K/V for the target layers only, all at the cache dtype. This manager also holds + one index-K head per sparse layer (replicated on every TP rank), keeps the dense target layers and the + appended one-model Eagle layers in FP8 when the cache is NVFP4 (see :meth:`_build_cache_config`), and + allocates those shared draft layers itself. When armed, the difference is added to the per-token term. + """ + base = super().get_cache_size_per_token(model_config, mapping, num_layers, **kwargs) + if not _EXACT_KV_SIZE_ESTIMATE or kwargs.get("is_draft", False): + return base + extra = cls._extra_bytes_per_token(model_config, mapping, num_layers, kwargs) + if isinstance(base, tuple): + return (base[0] + extra,) + tuple(base[1:]) + return base + extra + + @classmethod + def _extra_bytes_per_token(cls, model_config, mapping, num_layers, kwargs) -> int: + config = model_config.pretrained_config + config = getattr(config, "text_config", None) or config + num_target_layers = int(num_layers if num_layers is not None else config.num_hidden_layers) + local_layers = ( + list(mapping.pp_layers(num_target_layers)) + if mapping.pp_size > 1 + else list(range(num_target_layers)) + ) + # The same layer split the manager allocates with (resolve_sparse_layer_schedule). + sparse_ids = set(resolve_sparse_layer_schedule(None, None, config, num_target_layers)[0]) + sparse_layers = [layer for layer in local_layers if layer in sparse_ids] + dense_layers = [layer for layer in local_layers if layer not in sparse_ids] + tp = 1 if mapping.enable_attention_dp else mapping.tp_size + kv_heads = max(1, int(config.num_key_value_heads) // tp) + head_dim = int( + getattr(config, "head_dim", config.hidden_size // config.num_attention_heads) + ) + + sparse_config = getattr(model_config, "sparse_attention_config", None) + index_dim = int(getattr(sparse_config, "sparse_index_dim", None) or 128) + index_kv_dtype = str(getattr(sparse_config, "indexer_kv_dtype", "bf16")) + if index_kv_dtype == "nvfp4": + index_bytes = NVFP4_INDEX_K_BYTES_PER_TOKEN + else: + index_bytes = index_dim * (1 if index_kv_dtype == "fp8" else 2) + extra = len(sparse_layers) * index_bytes + if index_kv_dtype == "nvfp4" and sparse_layers: + # The fused producer's FP8 index-K sink (one FP8 row per token, shared by every layer). + extra += NVFP4_INDEX_K_DIM + + kv_elems = 2 * kv_heads * head_dim # K and V + quant_config = model_config.quant_config + nvfp4 = quant_config is not None and quant_config.quant_mode.has_fp4_kv_cache() + if nvfp4: + # The base prices every layer as NVFP4 data + block scales; dense layers are FP8. + extra += len(dense_layers) * (kv_elems - (kv_elems // 2 + kv_elems // 16)) + draft_layer_bytes = kv_elems + else: + draft_layer_bytes = None + spec_config = kwargs.get("spec_config") + if spec_config is not None and not kwargs.get("use_separate_draft_kv_cache", False): + num_draft = shared_draft_layer_count(spec_config, None) + if draft_layer_bytes is None: + elem = ( + 1 + if quant_config is not None and quant_config.quant_mode.has_fp8_kv_cache() + else 2 + ) + draft_layer_bytes = kv_elems * elem + extra += num_draft * draft_layer_bytes + return extra + def is_nvfp4_layer(self, layer_idx: int) -> bool: """Whether ``layer_idx`` stores packed NVFP4 data and block scales.""" return self.dtype == DataType.NVFP4 and int(layer_idx) in self.sparse_layer_ids @@ -737,8 +881,16 @@ def _torch_dtype_for_index_cache(self) -> torch.dtype: """Return the independently configured index-cache storage dtype.""" if self.indexer_kv_dtype == "fp8": return torch.float8_e4m3fn + if self.indexer_kv_dtype == "nvfp4": + return torch.uint8 return torch.bfloat16 + def _index_cache_row_elems(self) -> int: + """Elements of the index-cache view's last dim per token: the head dim, or NVFP4's 72 packed bytes.""" + if self.indexer_kv_dtype == "nvfp4": + return NVFP4_INDEX_K_BYTES_PER_TOKEN + return self.sparse_index_dim + def get_index_k_buffer(self, layer_idx: int) -> Optional[torch.Tensor]: """Return the V2-managed index-K view in the selected backend's layout. @@ -751,6 +903,10 @@ def get_index_v_buffer(self, layer_idx: int) -> Optional[torch.Tensor]: """Plain-tensor index-V cache for non-disabled sparse layers.""" return self._index_v_buffers.get(layer_idx) + def get_nvfp4_index_k_sink(self) -> Optional[torch.Tensor]: + """The fused producer's FP8 index-K sink [main pages, 1, 128, 128] (NVFP4 index-K only, else None).""" + return getattr(self, "_nvfp4_index_k_sink", None) + def has_index_value(self, layer_idx: int) -> bool: return layer_idx in self._index_v_buffers @@ -878,7 +1034,10 @@ def _build_kv_buffers( return full_view[:, :2] def configure_decode_shortcuts( - self, skip_unchanged_block_offsets: bool, reuse_pool_views: bool + self, + skip_unchanged_block_offsets: bool, + reuse_pool_views: bool, + block_ids_from_views: bool = False, ) -> None: """Turn on the decode-step shortcuts a model opted into. @@ -890,6 +1049,8 @@ def configure_decode_shortcuts( self.skip_unchanged_block_offsets = skip_unchanged_block_offsets # The feature base already caches every pool view in _cached_pool_view. self.reuse_pool_views = reuse_pool_views + # ``block_ids_from_views``: get_block_ids_per_seq already copies each row from the page-index buffer. + self.block_ids_from_views = block_ids_from_views def _kv_scale_slot_geometry( self, layer_idx: int, kv_layout: Optional[str] diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py index 20e4fcff808a..76b05b194e92 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/common.py @@ -111,7 +111,7 @@ class MiniMaxM3SparseParams(SparseParams): score_type: str = "max" disable_index_value: bool = True implementation: Literal["triton", "msa"] = "triton" - indexer_kv_dtype: Literal["bf16", "fp8"] = "bf16" + indexer_kv_dtype: Literal["bf16", "fp8", "nvfp4"] = "bf16" fuse_qkv_index_projection: bool = False @property @@ -242,8 +242,8 @@ def from_sparse_params( class PagedKvSlotMapping(NamedTuple): """One step's paged-cache slot mapping (see build_paged_kv_slot_mapping).""" - req_to_token: torch.Tensor - slot_ids: torch.Tensor + req_to_token: Optional[torch.Tensor] + slot_ids: Optional[torch.Tensor] out_cache_loc: torch.Tensor block_ids_cpu: torch.Tensor @@ -255,6 +255,7 @@ def build_paged_kv_slot_mapping( qo_lens_cpu: torch.Tensor, qo_offset_cpu: torch.Tensor, device: torch.device, + device_tables: bool = True, ) -> PagedKvSlotMapping: """Build the backend-neutral paged-cache slot mapping. @@ -277,6 +278,9 @@ def build_paged_kv_slot_mapping( staged with non-blocking copies, so this neither syncs the host nor reads device memory back. It still allocates per step, so call it from prepare(), never from the forward path. + + With device_tables=False, req_to_token and slot_ids are not built (None) and out_cache_loc stays a pinned host + tensor, for callers that only stage out_cache_loc into their own persistent buffer. """ tokens_per_block = int(kv_cache_manager.tokens_per_block) # block_ids_per_seq is a [batch, max_blocks_per_seq] tensor; row b holds the @@ -288,13 +292,15 @@ def build_paged_kv_slot_mapping( max_blocks = int(block_ids.shape[1]) max_kv_len = max_blocks * tokens_per_block - # Expand block ids -> per-token slot ids. - block_ids_dev = block_ids.to(device, non_blocking=True).to(torch.int64) - within_block = torch.arange(tokens_per_block, device=device, dtype=torch.int64) - # Outer product per batch entry: [batch, max_blocks, tokens_per_block] - slot_grid = block_ids_dev.unsqueeze(-1) * tokens_per_block + within_block - req_to_token = slot_grid.reshape(batch, max_kv_len).to(torch.int32) - slot_ids = torch.arange(batch, device=device, dtype=torch.int32) + req_to_token = slot_ids = None + if device_tables: + # Expand block ids -> per-token slot ids. + block_ids_dev = block_ids.to(device, non_blocking=True).to(torch.int64) + within_block = torch.arange(tokens_per_block, device=device, dtype=torch.int64) + # Outer product per batch entry: [batch, max_blocks, tokens_per_block] + slot_grid = block_ids_dev.unsqueeze(-1) * tokens_per_block + within_block + req_to_token = slot_grid.reshape(batch, max_kv_len).to(torch.int32) + slot_ids = torch.arange(batch, device=device, dtype=torch.int32) # out_cache_loc: per-new-token slot ids, in flattened query-token order. # Expanding the per-request lengths on the host reproduces the same slot @@ -325,7 +331,10 @@ def build_paged_kv_slot_mapping( + (pos - block_col * tokens_per_block) ).astype(np.int32) ) - out_cache_loc = async_tensor_h2d(out_cache_loc_cpu, torch.int32, device) + if device_tables: + out_cache_loc = async_tensor_h2d(out_cache_loc_cpu, torch.int32, device) + else: + out_cache_loc = maybe_pin_memory(out_cache_loc_cpu) return PagedKvSlotMapping(req_to_token, slot_ids, out_cache_loc, block_ids) diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/nvfp4_index_k.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/nvfp4_index_k.py new file mode 100644 index 000000000000..e849bf9b9c6c --- /dev/null +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/nvfp4_index_k.py @@ -0,0 +1,215 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""NVFP4 index-K cache (``indexer_kv_dtype="nvfp4"``): the writer, the prefill gather and the page remap. + +A page is the Q8KV4 decode indexer's packed layout: 8192 bytes of E2M1 codes (token ``t`` at ``t * 64``, low nibble += even element) then 1024 bytes of E4M3 16-element scales (``8192 + t * 8 + g``), no global scale; 72 bytes per +token instead of FP8's 128. The kernels live in ``libm31_idxk_nvfp4.so`` (C++ / CUDA, ``torch.ops.m31idxk``): + +* ``write``: the unit-scale NVFP4 quantization of vLLM's fused index-K writer, from E4M3 or bf16 rows, into the + token's slot; slots < 0 are skipped. +* ``gather_fp8``: each page's elements as ``E4M3_RN_satfinite(e2m1 * scale)`` into contiguous FP8 pages, so the + FP8 index scorers (the fmha_sm100 proxy pass and the CuTe DSL scorer) run unchanged on the gathered copy, as + vLLM's NVFP4 prefill indexer does. +""" + +from __future__ import annotations + +import os +from typing import List, Optional, Tuple + +import torch + +from tensorrt_llm._utils import maybe_pin_memory +from tensorrt_llm.logger import logger + +PAGE = 128 +DIM = 128 +BYTES_PER_TOKEN = DIM // 2 + DIM // 16 # 72 +PAGE_BYTES = PAGE * BYTES_PER_TOKEN # 9216 +LIB_NAME = "libm31_idxk_nvfp4.so" + +_STATE = {"loaded": False, "counts": {}} +# Default ON; the kill restores the original pageable offset copy for item pricing. +_COMPACT_NOSYNC = os.environ.get("TRTLLM_M31_DISABLE_NVFP4_COMPACT_NOSYNC", "0") != "1" + + +def _load() -> None: + if _STATE["loaded"]: + return + import tensorrt_llm + + path = os.path.join(os.path.dirname(tensorrt_llm.__file__), "libs", LIB_NAME) + if not os.path.exists(path): + raise RuntimeError( + f"MiniMax-M3 indexer_kv_dtype='nvfp4' needs {LIB_NAME} in tensorrt_llm/libs ({path} is missing); " + "set TRTLLM_M31_DISABLE_AGG_NVFP4_INDEXK=1 to fall back to FP8 index-K" + ) + torch.ops.load_library(path) + _STATE["loaded"] = True + logger.info(f"[M3.1] AGG_NVFP4_INDEXK kernels loaded: {path}") + + +def _count(kind: str) -> None: + counts = _STATE["counts"] + counts[kind] = n = counts.get(kind, 0) + 1 + if n & (n - 1) == 0: + logger.info(f"[M3.1] AGG_NVFP4_INDEXK {kind} taken n={n}") + + +def is_nvfp4_index_cache(cache: Optional[torch.Tensor]) -> bool: + """Whether ``cache`` is an NVFP4 index-K view: [pages, 1, 128, 72] uint8 (or its NHD transpose).""" + return ( + cache is not None + and cache.dtype == torch.uint8 + and cache.dim() == 4 + and int(cache.shape[-1]) == BYTES_PER_TOKEN + ) + + +def page_view(cache: torch.Tensor) -> torch.Tensor: + """The cache as [pages, 9216] bytes (each page contiguous, the page stride kept).""" + return cache.as_strided((int(cache.shape[0]), PAGE_BYTES), (int(cache.stride(0)), 1)) + + +def write(rows: torch.Tensor, out_cache_loc: torch.Tensor, cache: torch.Tensor) -> None: + """Quantize index-K ``rows`` [T, 128] (E4M3 or bf16) into ``cache`` at ``out_cache_loc`` (slots < 0 skipped).""" + _load() + torch.ops.m31idxk.nvfp4_idxk_write(rows, out_cache_loc, page_view(cache)) + if not torch.cuda.is_current_stream_capturing(): + _count("write") + + +def write_from_sink( + sink: torch.Tensor, out_cache_loc: torch.Tensor, num_tokens: int, cache: torch.Tensor +) -> None: + """Pack the first ``num_tokens`` slots of ``out_cache_loc`` into ``cache`` from the producer's FP8 ``sink``. + + ``sink`` [pages, 1, 128, 128] holds each token's E4M3 index-K at its slot (written by the fused producer); slots + < 0 are skipped. Capture-safe (no host sync). + """ + _load() + torch.ops.m31idxk.nvfp4_idxk_write_from_sink( + sink, out_cache_loc, int(num_tokens), page_view(cache) + ) + if not torch.cuda.is_current_stream_capturing(): + _count("write") + + +def gather_fp8( + cache: torch.Tensor, page_ids: torch.Tensor, out: Optional[torch.Tensor] = None +) -> torch.Tensor: + """Pages ``page_ids`` (int32, < 0 gives zeros) as contiguous FP8 [N, 1, 128, 128].""" + _load() + n = int(page_ids.shape[0]) + if out is None: + out = torch.empty((n, 1, PAGE, DIM), dtype=torch.float8_e4m3fn, device=cache.device) + torch.ops.m31idxk.nvfp4_idxk_gather_fp8(page_view(cache), page_ids, out) + if not torch.cuda.is_current_stream_capturing(): + _count("gather") + return out + + +def compact_step_pages( + block_table: torch.Tensor, kv_lens_cpu: torch.Tensor, rows: int +) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """The step's pages in the flattened page-table order and the tables that address the compact copy. + + Row ``r`` holds ``ceil(kv_lens_cpu[r] / 128)`` pages, ``block_table[r, :n_r]``; the flattened order is the one + ``msa_kv_indices`` uses. Returns ``(page_ids [P] int32, kv_indices [P] int32 = 0..P-1, block_table [rows, cols] + int32`` with row ``r`` = ``offset_r + 0..n_r-1``). + """ + counts: List[int] = [-(-int(n) // PAGE) for n in kv_lens_cpu[:rows].tolist()] + device = block_table.device + parts = [block_table[r, :c] for r, c in enumerate(counts) if c > 0] + page_ids = ( + torch.cat(parts).to(torch.int32) + if parts + else torch.empty((0,), dtype=torch.int32, device=device) + ) + total = int(page_ids.shape[0]) + if _COMPACT_NOSYNC: + # The row offsets go through pinned memory with a non-blocking copy: a pageable host-to-device copy synchronizes + # the stream first, which in a breakable-graph replay made the host wait for all queued work at the first sparse + # layer of every step with context rows (same values). + starts: List[int] = [0] * rows + for r in range(1, rows): + starts[r] = starts[r - 1] + counts[r - 1] + offsets = torch.tensor(starts, dtype=torch.int32) + if device.type == "cuda": + offsets = maybe_pin_memory(offsets).to(device, non_blocking=True) + offsets = offsets.unsqueeze(1) + else: + offsets = torch.tensor( + [sum(counts[:r]) for r in range(rows)], dtype=torch.int32, device=device + ).unsqueeze(1) + cols = max(1, max(counts) if counts else 1, int(block_table.shape[1])) + table = offsets + torch.arange(cols, dtype=torch.int32, device=device).unsqueeze(0) + table = torch.minimum(table, torch.full_like(table, max(total - 1, 0))) + return page_ids, torch.arange(total, dtype=torch.int32, device=device), table + + +def decode_history_workspace(seq_lens: torch.Tensor, cols: int) -> torch.Tensor: + """The Q8KV4 reader's scheduler workspace: each request's history-page count (pages before its local one), + prefix-summed (uint8 [(B + 1) * 4], int32 view). Depends on the step only.""" + batch = int(seq_lens.shape[0]) + workspace = torch.empty(((batch + 1) * 4,), dtype=torch.uint8, device=seq_lens.device) + scheduler = workspace.view(torch.int32) + history = torch.div(seq_lens - 1, PAGE, rounding_mode="floor").clamp_(min=0, max=cols) + scheduler[:1].zero_() + torch.cumsum(history, dim=0, out=scheduler[1:]) + return workspace + + +_READERS = {} + + +def score_decode_rows( + idx_q: torch.Tensor, + cache: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + query_len: int, + num_heads: int, + workspace: Optional[torch.Tensor] = None, +) -> torch.Tensor: + """Index scores of decode rows straight from NVFP4 index-K pages, [heads, cols, rows * query_len] fp32 in the + selector's layout: the Q8KV4 decode reader (fmha_sm100_q8kv8 run_scores_hpr; exact FP16 dequant, FP32 + accumulation), the reader every pure-decode step of the agg target uses. History pages are written; a row's local + and later pages are left as allocated, which the selector never ranks (local blocks are forced).""" + from fmha_sm100_q8kv8 import q8_indexer_interface as qi + + batch, cols = int(block_table.shape[0]), int(block_table.shape[1]) + device = idx_q.device + _, view, kernel_out = qi.q8kv4_hpr_buffer(batch, cols, num_heads, int(query_len), device) + if workspace is None: + workspace = decode_history_workspace(seq_lens, cols) + key = (device, int(num_heads)) + w = _READERS.get(key) + if w is None: + w = _READERS[key] = qi.BatchDecodeIndexerQ8KV4Wrapper(num_heads=int(num_heads)) + w._block_table = block_table + w._seq_lens = seq_lens + w._workspace = workspace + w._query_len = int(query_len) + w._scores = torch.empty((batch, 0, 0), dtype=torch.float32, device=device) + w.run_scores_hpr( + idx_q, cache.reshape(int(cache.shape[0]), PAGE, int(cache.shape[-1])), kernel_out + ) + if not torch.cuda.is_current_stream_capturing(): + _count("decode_rows") + return view + + +__all__ = [ + "BYTES_PER_TOKEN", + "PAGE_BYTES", + "compact_step_pages", + "decode_history_workspace", + "gather_fp8", + "is_nvfp4_index_cache", + "page_view", + "score_decode_rows", + "write", + "write_from_sink", +] diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/paged_cache.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/paged_cache.py index b3cbb2716240..0c04481999fd 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/paged_cache.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/paged_cache.py @@ -8,6 +8,54 @@ import torch +# fp8_e4m3_saturate: E4M3 casts saturated to +-448, the bits the CUDA FP8 writers produce (__NV_SATFINITE / +# cvt.rn.satfinite); NaN stays NaN. Where torch's own E4M3 cast already saturates (c10's float8_e4m3fn conversion, +# shared by its CPU and CUDA kernels) the cast alone is used: an explicit clamp would add a kernel per cast for the same +# bytes. A torch whose cast does not saturate gets the clamp. +# Kill: FP8_SATURATE = False (the plain cast, whatever it does). +# SHARED_FP8_SATURATE: whether the E4M3 cache writes below and the dense-decode query cast use it. Off unless a model +# opts in (RuntimeHooks.fp8_saturating_casts, armed by the executor through arm_fp8_saturation): the plain cast. +FP8_SATURATE = True +SHARED_FP8_SATURATE = False +_E4M3_MAX = 448.0 + + +def _cast_saturates() -> bool: + """Whether this torch's E4M3 cast maps every out-of-range value, infinities included, to +-448.""" + probe = torch.tensor([449.0, 463.0, 464.0, 1e4, float("inf"), -464.0, -float("inf")]) + want = torch.tensor([448.0, 448.0, 448.0, 448.0, 448.0, -448.0, -448.0]) + return torch.equal(probe.to(torch.float8_e4m3fn).float(), want) + + +# Probed once at import on the CPU, so no CUDA work runs here (nor later inside a graph capture). +_CAST_SATURATES = _cast_saturates() + + +def fp8_e4m3_saturate(values: torch.Tensor) -> torch.Tensor: + """``values`` as float8_e4m3fn, saturated to +-448 (bitwise the plain cast for every finite in-range value). + Values already in float8_e4m3fn are returned as they are (they are finite by construction or already NaN).""" + if values.dtype == torch.float8_e4m3fn: + return values + if FP8_SATURATE and not _CAST_SATURATES: + values = values.clamp(-_E4M3_MAX, _E4M3_MAX) + return values.to(torch.float8_e4m3fn) + + +def arm_fp8_saturation() -> None: + """Saturate the E4M3 cache writes and the dense-decode query cast in this process + (RuntimeHooks.fp8_saturating_casts).""" + global SHARED_FP8_SATURATE + SHARED_FP8_SATURATE = True + + +def to_e4m3(values: torch.Tensor) -> torch.Tensor: + """``values`` as float8_e4m3fn: saturated when armed (SHARED_FP8_SATURATE), else the plain cast.""" + return fp8_e4m3_saturate(values) if SHARED_FP8_SATURATE else values.to(torch.float8_e4m3fn) + + +def _to_cache_dtype(values: torch.Tensor, dtype: torch.dtype) -> torch.Tensor: + return to_e4m3(values) if dtype == torch.float8_e4m3fn else values.to(dtype) + def write_kv_slots( cache: torch.Tensor, @@ -60,11 +108,11 @@ def write_kv_slots( if layout == "HND": # Advanced indices on dims 0 and 2 broadcast to [num_tokens] and # move front, giving a [num_tokens, num_heads, channel] target. - cache[page, :, within, :] = values.to(cache.dtype) + cache[page, :, within, :] = _to_cache_dtype(values, cache.dtype) else: - cache[page, within] = values.to(cache.dtype) + cache[page, within] = _to_cache_dtype(values, cache.dtype) else: - cache.index_copy_(0, out_cache_loc.to(torch.long), values.to(cache.dtype)) + cache.index_copy_(0, out_cache_loc.to(torch.long), _to_cache_dtype(values, cache.dtype)) __all__ = ["write_kv_slots"] diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/trtllm_gen_dense_decode.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/trtllm_gen_dense_decode.py index 075ebe04da12..2f47b6cde0fa 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/trtllm_gen_dense_decode.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/kernels/trtllm_gen_dense_decode.py @@ -25,6 +25,7 @@ import torch +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import to_e4m3 from tensorrt_llm._torch.memory_buffer_utils import get_memory_buffers from .msa_utils import check_decode_span_shape @@ -278,7 +279,7 @@ def minimax_m3_trtllm_gen_dense_decode( # q/k/v, so an FP8 pool needs FP8 Q. M3 stores unscaled E4M3, so this is a # plain cast and a no-op when the fused producer already emitted FP8. if kv_pool.dtype == torch.float8_e4m3fn and q.dtype != torch.float8_e4m3fn: - q = q.to(torch.float8_e4m3fn) + q = to_e4m3(q) reserve = torch.cuda.is_current_stream_capturing() workspace_size = _workspace(q.dtype, num_heads, int(q.shape[2]), int(kv_pool.shape[1])) diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py index fc9ec623cc0e..cd892526a8c8 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_backend.py @@ -31,6 +31,8 @@ from __future__ import annotations +import os +import sys from dataclasses import dataclass from typing import NamedTuple, Optional, Tuple @@ -42,8 +44,9 @@ from tensorrt_llm._torch.pyexecutor import h2d_cache, step_inputs from tensorrt_llm._torch.pyexecutor.h2d_cache import copy_h2d_if_changed from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy -from tensorrt_llm._utils import maybe_pin_memory +from tensorrt_llm._utils import maybe_pin_memory, mpi_rank from tensorrt_llm.bindings import DataType +from tensorrt_llm.logger import logger from tensorrt_llm.models.modeling_utils import QuantConfig from .common import ( @@ -64,7 +67,32 @@ uniform_subpages_per_slot, write_subpage_block_table, ) -from .msa_indexer import MsaIndexer, cutedsl_score_runner +from .msa_indexer import MsaIndexer, cutedsl_score_runner, proxy_plan_kwargs + +# NVFP4_STEP_CACHE (AGG_NVFP4_INDEXK): run_indexer builds the NVFP4 compact page tables once per step, not per +# layer. +_NVFP4_STEP_CACHE_OFF = os.environ.get("TRTLLM_M31_DISABLE_NVFP4_STEP_CACHE", "0") == "1" +# MIX_NVFP4_READER (not bitwise): on a mixed step with an NVFP4 index-K cache the generation rows +# are scored from the NVFP4 pages by the Q8KV4 decode reader (the one every pure-decode step uses: exact FP16 +# dequant) instead of the FP8 scorer on the step's gathered E4M3 copy, so only the context rows' pages are gathered +# (microbench: -5.8..-7.3 ms per c56 mixed step). Kill switch: TRTLLM_M31_DISABLE_MIX_NVFP4_READER=1. +_MIX_NVFP4_READER = os.environ.get("TRTLLM_M31_DISABLE_MIX_NVFP4_READER", "0") != "1" + + +_MIX_NVFP4_READER_LOGGED = False + + +def _log_mix_reader_taken(ctx_rows: int, gen_rows: int, query_len: int) -> None: + """Report the first executed mixed-step reader call in each rank process.""" + global _MIX_NVFP4_READER_LOGGED + if not _MIX_NVFP4_READER_LOGGED: + _MIX_NVFP4_READER_LOGGED = True + print( + f"[M31 MIXR] taken rank={mpi_rank()} ctx_reqs={ctx_rows} " + f"gen_rows={gen_rows} query_len={query_len}", + file=sys.stderr, + flush=True, + ) def _cache_device(meta) -> torch.device: @@ -112,6 +140,15 @@ def _host_prefix_sum(lens_cpu: torch.Tensor, dtype: torch.dtype) -> torch.Tensor return maybe_pin_memory(torch.cat([sums.new_zeros(1), sums]).to(dtype)) +# SUBPAGE_IN_GRAPH: the sub-page block table is expanded from msa_block_table on the device inside the forward, once, +# by the first dense layer that reads it (msa_subpage_rows), so the block table can join the step-input pack and the +# step boundary issues no expansion; otherwise prepare() expands it after an unpacked block-table upload. +# TAIL_FILL_IN_FORWARD: prepare() records where msa_out_cache_loc's live slots end and the forward invalidates the rest +# (on_update_kv_lens, which opens every forward, so a captured forward fills its own tail at each replay) instead of an +# eager fill between the decode graphs; otherwise prepare() fills it. +# Both are taken only for a model that opts in (MsaRuntimePolicy.forward_staging); otherwise prepare() keeps both. + + def _fill_unused_slots(buf: torch.Tensor, start: int) -> None: """buf[start:] = -1 after a write of the live slots buf[:start] (see h2d_cache.fill_tail_if_changed).""" h2d_cache.fill_tail_if_changed(buf, start, -1) @@ -195,6 +232,11 @@ class MiniMaxM3MsaSparseAttentionMetadata(TrtllmAttentionMetadata): # factor, or 0 where the pool has no single one; see msa_subpage_rows. msa_subpage_block_table: Optional[torch.Tensor] = None _msa_subpages_per_slot: int = 0 + # SUBPAGE_IN_GRAPH: the rows prepare() staged for the forward to expand (0: none), and whether this forward did. + _msa_subpage_rows: int = 0 + _msa_subpage_expanded: bool = False + # TAIL_FILL_IN_FORWARD: the first msa_out_cache_loc row past this step's live slots (None: prepare() filled it). + _msa_tail_start: Optional[int] = None # Inputs for on_update_kv_lens: each query token's request row and offset # within the request, and the kv_lens prepare() staged (the upper bound the # correction is clamped to). The write slots are re-derived from @@ -214,6 +256,9 @@ class MiniMaxM3MsaSparseAttentionMetadata(TrtllmAttentionMetadata): # never captured. Built once per step in prepare() and reused by every # layer. _msa_prefill_proxy_plan: Optional[tuple] = None + # The plain proxy plan (num_kv_splits 1, the planner's tile) over the same rows, built only for + # MsaRuntimePolicy.prefill_proxy_oracle. + _msa_prefill_proxy_plan_ref: Optional[tuple] = None _msa_prefill_gqa_plan: Optional[tuple] = None _msa_prefill_dense_plan: Optional[tuple] = None # Per-token valid-block count for the prefill-side indexer proxy. It is @@ -302,6 +347,11 @@ def as_pinned_int32(lens: torch.Tensor) -> torch.Tensor: # The attention plans cover the context rows; the proxy plan covers # whatever rows the CuTe DSL scorer did not take. See _msa_attn_plan_rows # and _msa_proxy_plan_rows. + @property + def msa_prefill_proxy_plan_ref(self) -> Optional[tuple]: + """The oracle's plain proxy plan for this step (MsaRuntimePolicy.prefill_proxy_oracle), else None.""" + return self._msa_prefill_proxy_plan_ref + @property def msa_prefill_proxy_plan(self) -> Optional[tuple]: """Prebuilt indexer proxy plan for this step's fmha_sm100 rows.""" @@ -369,6 +419,15 @@ def msa_subpage_rows(self, row_first: int, row_last: int) -> Tuple[Optional[torc table = self.msa_subpage_block_table if table is None: return None, 0 + rows = self._msa_subpage_rows + if rows > 0 and not self._msa_subpage_expanded: + # Once per forward (on_update_kv_lens, which opens every forward, resets it), from the block table the + # pack landed, over the rows prepare() staged; a captured forward records it once, so every replay + # expands its own rows. + write_subpage_block_table( + self.msa_block_table[:rows], self._msa_subpages_per_slot, table[:rows] + ) + self._msa_subpage_expanded = True return table[row_first:row_last], self._msa_subpages_per_slot def _msa_main_kv_is_fp8(self) -> bool: @@ -808,6 +867,19 @@ def _msa_runs_no_fmha(self) -> bool: span = self._msa_decode_span return span is not None and span.row_first == 0 + def _msa_proxy_plan_kwargs( + self, + fmha_sm100, + qo_lens_cpu: torch.Tensor, + kv_lens_cpu: torch.Tensor, + rows: Optional[Tuple[int, int]], + page_size: int, + ) -> dict: + """Prefill proxy arguments shared with the inline planner through proxy_plan_kwargs.""" + return proxy_plan_kwargs( + self.msa_policy, fmha_sm100, qo_lens_cpu, kv_lens_cpu, rows, page_size + ) + def _msa_proxy_plan_rows(self) -> Optional[Tuple[int, int]]: """Batch rows this step's fmha_sm100 indexer proxy plan must cover. @@ -840,7 +912,8 @@ def _msa_index_kv_dtype(self) -> torch.dtype: casts Q to the cache dtype, so the cache decides what the scorer sees. """ indexer_kv_dtype = str(getattr(self.kv_cache_manager, "indexer_kv_dtype", "bf16")) - return torch.float8_e4m3fn if indexer_kv_dtype == "fp8" else torch.bfloat16 + # NVFP4 index-K is scored as E4M3: the scorers read its gathered FP8 copy (run_indexer). + return torch.float8_e4m3fn if indexer_kv_dtype in ("fp8", "nvfp4") else torch.bfloat16 def _cutedsl_indexer_supported( self, *, num_index_heads: int, page_size: int, decode_query_len: int @@ -876,6 +949,18 @@ def _msa_kv_lens_may_change(self) -> bool: or (runtime_features is not None and runtime_features.has_speculative_draft_tokens) ) + def _fill_tail_in_forward(self) -> None: + """TAIL_FILL_IN_FORWARD: -1 over msa_out_cache_loc past the live slots prepare() staged (see the prepare() + comment on padded rows), before anything in the forward reads it.""" + start = self._msa_tail_start + buf = self.msa_out_cache_loc + if start is not None and buf is not None and start < buf.shape[0]: + logger.info_once( + "[M3 host] MSA out_cache_loc tail filled inside the forward", + key="msa_tail_fill_in_forward", + ) + buf[start:].fill_(-1) + def on_update_kv_lens(self) -> None: """Re-derive lengths, slots and valid-block counts from the corrected kv_lens_cuda. @@ -895,6 +980,9 @@ def on_update_kv_lens(self) -> None: extend_ctx promotes speculative generation rows into the context prefix. Staged host bounds remain valid upper bounds as lengths shrink. """ + # A new forward: the first dense layer that reads the sub-page table expands it again (msa_subpage_rows). + self._msa_subpage_expanded = False + self._fill_tail_in_forward() super().on_update_kv_lens() if not self._msa_fields_ready or not self._msa_kv_lens_dynamic: return @@ -982,6 +1070,7 @@ def _build_step_plans(self) -> None: per-step tensors. """ self._msa_prefill_proxy_plan = None + self._msa_prefill_proxy_plan_ref = None self._msa_prefill_gqa_plan = None self._msa_prefill_dense_plan = None self._msa_prefill_n_valid_blocks = None @@ -1030,7 +1119,9 @@ def _build_step_plans(self) -> None: # flag. use_fp8 = self._msa_main_kv_is_fp8() - def plan_for(rows: Optional[Tuple[int, int]], **plan_kwargs) -> Optional[tuple]: + def plan_for( + rows: Optional[Tuple[int, int]], num_kv_splits: int = 1, **plan_kwargs + ) -> Optional[tuple]: """Plan one site over the given rows, or None where it has none. Slicing keeps the length tensors' pinned backing, so a single-phase @@ -1045,7 +1136,7 @@ def plan_for(rows: Optional[Tuple[int, int]], **plan_kwargs) -> Optional[tuple]: kv_lens_cpu if whole else kv_lens_cpu[first:last], qo_offset=qo_offset_cpu if whole else qo_offset_cpu[first:last], page_size=page_size, - num_kv_splits=1, + num_kv_splits=num_kv_splits, causal=True, **plan_kwargs, ) @@ -1063,16 +1154,24 @@ def plan_for(rows: Optional[Tuple[int, int]], **plan_kwargs) -> Optional[tuple]: # Proxy plan: MQA (num_kv_heads=1) max-score pass over the index # branch; output_maxscore feeds the indexer's top-k block selection. # A short pure-prefill step selects every valid block, so it needs none. + proxy_rows = self._msa_proxy_plan_rows() self._msa_prefill_proxy_plan = ( None if short_prefill else plan_for( - self._msa_proxy_plan_rows(), + proxy_rows, num_qo_heads=num_index_heads, num_kv_heads=1, output_maxscore=True, + **self._msa_proxy_plan_kwargs( + fmha_sm100, qo_lens_cpu, kv_lens_cpu, proxy_rows, page_size + ), ) ) + if self.msa_policy.prefill_proxy_oracle and self._msa_prefill_proxy_plan is not None: + self._msa_prefill_proxy_plan_ref = plan_for( + proxy_rows, num_qo_heads=num_index_heads, num_kv_heads=1, output_maxscore=True + ) attn_rows = self._msa_attn_plan_rows() # Sparse layers: kv_block_num=topk limits attention to top-k blocks. self._msa_prefill_gqa_plan = plan_for( @@ -1110,6 +1209,8 @@ def _build_msa_fields(self) -> None: decoding the inputs for on_update_kv_lens are staged as well. """ self._msa_fields_ready = False + # Set again below once this step's live slots are staged; until then the forward fills nothing. + self._msa_tail_start = None if not self._msa_buffers_ready: return request_ids = self.request_ids @@ -1187,7 +1288,11 @@ def _build_msa_fields(self) -> None: # would make its shape dynamic, and a negative slot is what makes those # rows cache-write no-ops instead. Invalidate the unwritten tail so they # cannot reuse the previous step's live slots, which address real pages. - _fill_unused_slots(self.msa_out_cache_loc, total_new_tokens) + if self.msa_policy.forward_staging: + self._msa_tail_start = total_new_tokens + else: + self._msa_tail_start = None + _fill_unused_slots(self.msa_out_cache_loc, total_new_tokens) if kv_indices is not None: step_inputs.copy_(self.msa_kv_indices[: int(kv_indices.shape[0])], kv_indices) @@ -1198,8 +1303,9 @@ def _build_msa_fields(self) -> None: block_table_changed = copy_h2d_if_changed( self.msa_block_table[:batch_size, :block_table_cols], maybe_pin_memory(block_ids_cpu.to(torch.int32)), - # write_subpage_block_table below reads it on the device before the replay. - pack=self.msa_subpage_block_table is None, + # A prepare-time write_subpage_block_table below reads it on the device before the replay; with + # SUBPAGE_IN_GRAPH nothing does, so it can join the pack. + pack=self.msa_subpage_block_table is None or self.msa_policy.forward_staging, ) # Packed only when the device scan below (h2d_cache off) does not read it before the replay. if h2d_cache.enabled(): @@ -1271,7 +1377,14 @@ def _build_msa_fields(self) -> None: # Sub-page expansion for the trtllm-gen dense layers, staged once here # instead of once per layer, outside capture into a graph-stable # buffer as with the slot table above. - if self.msa_subpage_block_table is not None and block_table_changed: + # With SUBPAGE_IN_GRAPH the forward expands this step's rows instead (msa_subpage_rows). + self._msa_subpage_expanded = False + self._msa_subpage_rows = batch_size if self.msa_policy.forward_staging else 0 + if ( + not self.msa_policy.forward_staging + and self.msa_subpage_block_table is not None + and block_table_changed + ): write_subpage_block_table( self.msa_block_table[:batch_size], self._msa_subpages_per_slot, @@ -1562,8 +1675,17 @@ def run_indexer( # scorer and cache writer below both honor their source strides. idx_q_view = idx_q.view(num_tokens, config.num_index_heads, config.sparse_index_dim) idx_k_cache = metadata.msa_idx_k_cache(self.layer_idx) - configured_for_fp8 = self.indexer_kv_dtype == "fp8" - expected_cache_dtype = torch.float8_e4m3fn if configured_for_fp8 else torch.bfloat16 + configured_for_nvfp4 = self.indexer_kv_dtype == "nvfp4" + # NVFP4 index-K takes the FP8 indexer's contract (E4M3 index-Q, a cache the fused producer populated) and + # is scored through its gathered FP8 copy below. + configured_for_fp8 = self.indexer_kv_dtype == "fp8" or configured_for_nvfp4 + expected_cache_dtype = ( + torch.uint8 + if configured_for_nvfp4 + else torch.float8_e4m3fn + if configured_for_fp8 + else torch.bfloat16 + ) if idx_k_cache.dtype != expected_cache_dtype: raise ValueError( "MiniMax-M3 index-K cache dtype does not match indexer_kv_dtype=" @@ -1606,11 +1728,58 @@ def run_indexer( # Short pure-prefill step: every token selects all its valid blocks (staged once per step). num_kv_heads = int(self.indexer.config.num_kv_heads) return identity.unsqueeze(1).expand(num_tokens, num_kv_heads, int(identity.shape[1])) + # NVFP4 replaces both page tables with ones addressing its gathered copy; the stock path reads them as staged. + step_block_table = None + kv_indices = metadata.msa_kv_indices + if configured_for_nvfp4: + # AGG_NVFP4_INDEXK: gather the step's NVFP4 pages into contiguous E4M3 pages (the keys vLLM's NVFP4 + # prefill indexer scores) and point both page tables at that copy; the scorers below run unchanged. + # Eager only: a captured pure-decode step scores NVFP4 pages directly in the model's decode target. + from .kernels import nvfp4_index_k + + if torch.cuda.is_current_stream_capturing(): + raise RuntimeError( + "MiniMax-M3 NVFP4 index-K: run_indexer gathers pages eagerly and cannot run under CUDA graph " + "capture; the model's decode target must score NVFP4 pages itself" + ) + rows = int(metadata.num_seqs) + # MIX_NVFP4_READER: a mixed step gathers only its context rows' pages (they are the prefix of the + # flattened page order the proxy plan addresses); the generation rows read the NVFP4 pages below. + mix_reader = _MIX_NVFP4_READER and span is not None and span.row_first > 0 + nvfp4_pages = idx_k_cache + if mix_reader: + rows = int(span.row_first) + # NVFP4_STEP_CACHE: the compact page list, its tables and the gather buffer depend on the step only + # (block table, lengths, rows), not the layer: build them for the first sparse layer of the step and + # reuse them; each layer still gathers its own cache into the buffer. Keyed by the host lengths object + # itself (prepare() stages a new one every step, and the cache holds it, so its identity is not reused) + # plus rows and the block table buffer, whose only writer is prepare(); inference tensors carry no + # version counter. Kill switch: TRTLLM_M31_DISABLE_NVFP4_STEP_CACHE=1 (per-layer build, as before). + bt, kvl = metadata.msa_block_table, metadata.msa_kv_lens_cpu + key = (rows, bt.data_ptr()) + cached = None if _NVFP4_STEP_CACHE_OFF else getattr(metadata, "_nvfp4_step_pages", None) + if cached is None or cached[0] is not kvl or cached[1] != key: + page_ids, kv_indices, step_block_table = nvfp4_index_k.compact_step_pages( + bt, kvl, rows + ) + gather_out = torch.empty( + (int(page_ids.shape[0]), 1, 128, 128), + dtype=torch.float8_e4m3fn, + device=bt.device, + ) + cached = (kvl, key, page_ids, kv_indices, step_block_table, gather_out) + if not _NVFP4_STEP_CACHE_OFF: + metadata._nvfp4_step_pages = cached + _, _, page_ids, kv_indices, step_block_table, gather_out = cached + idx_k_cache = nvfp4_index_k.gather_fp8(idx_k_cache, page_ids, gather_out) + else: + mix_reader = False block_table = None seq_lens_cuda = None decode_query_len = None gen_first = 0 ctx_rows = 0 + gen_scores = None if span is not None: ctx_rows = span.row_first decode_query_len = span.query_len @@ -1619,7 +1788,11 @@ def run_indexer( # PhasedFmha derives the attention phase's token offset, so the # scorer and the decode kernels agree on the boundary. gen_first = num_tokens - decode_query_len * (row_last - ctx_rows) - block_table = metadata.msa_block_table[ctx_rows:row_last] + block_table = ( + step_block_table + if step_block_table is not None and not mix_reader + else metadata.msa_block_table + )[ctx_rows:row_last] seq_lens_cuda = metadata.msa_seq_lens_cuda[ctx_rows:row_last] # One selection path, over the scratch prepare() staged for whichever # scorer owns the rows: a span starting at row 0 leaves the CuTe DSL @@ -1654,15 +1827,36 @@ def run_indexer( if span is not None else None ) + if mix_reader: + # The generation span's scores in the selector's layout, at the store's tile width; the step workspace + # is built once per step (keyed like the page cache). + tiles = int(max_score.shape[1]) + ws_key = (metadata.msa_kv_lens_cpu, ctx_rows, tiles) + ws = getattr(metadata, "_mix_nvfp4_ws", None) + if ws is None or ws[0] is not ws_key[0] or ws[1:3] != ws_key[1:]: + ws = (*ws_key, nvfp4_index_k.decode_history_workspace(seq_lens_cuda, tiles)) + metadata._mix_nvfp4_ws = ws + gen_scores = nvfp4_index_k.score_decode_rows( + idx_q_view[gen_first:], + nvfp4_pages, + block_table[:, :tiles], + seq_lens_cuda, + decode_query_len, + int(config.num_index_heads), + ws[3], + ) + _log_mix_reader_taken(ctx_rows, num_tokens - gen_first, decode_query_len) return self.indexer.select_blocks( idx_q_view, idx_k_cache, idx_sm_scale=idx_sm_scale, - kv_indices=metadata.msa_kv_indices, + kv_indices=kv_indices, qo_lens_cpu=metadata.msa_qo_lens_cpu, kv_lens_cpu=metadata.msa_kv_lens_cpu, qo_offset_cpu=metadata.msa_qo_offset_cpu, proxy_plan=proxy_plan, + proxy_plan_ref=metadata.msa_prefill_proxy_plan_ref, + proxy_policy=metadata.msa_policy, max_score=max_score, n_valid_blocks=n_valid_blocks, require_cutedsl=span is not None, @@ -1671,6 +1865,7 @@ def run_indexer( decode_query_len=decode_query_len, gen_token_first=gen_first, ctx_rows=ctx_rows, + gen_scores=gen_scores, ) def sparse_attn_predict( diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_indexer.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_indexer.py index ea08907e34af..379723657668 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_indexer.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_indexer.py @@ -13,11 +13,19 @@ from __future__ import annotations +import atexit import functools -from typing import TYPE_CHECKING, Optional +import inspect +import os +import sys +import time +from typing import TYPE_CHECKING, Optional, Tuple import torch +from tensorrt_llm._utils import mpi_rank +from tensorrt_llm.logger import logger + from .kernels.msa_utils import ( MSA_REQUIRED_TOPK, per_token_valid_blocks, @@ -99,6 +107,84 @@ def _cutedsl_score( return True +_PLAN_TAKES_MAXSCORE_TILE: dict = {} + + +def _plan_takes_maxscore_tile(plan) -> bool: + """Whether the loaded MSA fmha_sm100_plan accepts ``maxscore_qo_tile`` (decided once per function). + + The public planner forwards ``**kwargs`` to its module's ``_fmha_sm100_plan_impl``, which is where the argument + is declared, so that signature is read too. + """ + takes = _PLAN_TAKES_MAXSCORE_TILE.get(plan) + if takes is None: + takes = False + for fn in (plan, getattr(inspect.getmodule(plan), "_fmha_sm100_plan_impl", None)): + try: + if fn is not None and "maxscore_qo_tile" in inspect.signature(fn).parameters: + takes = True + break + except (TypeError, ValueError): + continue + _PLAN_TAKES_MAXSCORE_TILE[plan] = takes + return takes + + +def proxy_plan_kwargs( + policy, + fmha_sm100, + qo_lens_cpu: torch.Tensor, + kv_lens_cpu: torch.Tensor, + rows: Optional[Tuple[int, int]], + page_size: int, +) -> dict: + """Extra fmha_sm100_plan arguments for a prefill proxy plan over batch rows ``rows`` under + MsaRuntimePolicy.prefill_proxy_* (or {} for the plain plan). Shared by prepare()'s plan and the inline plan. + + A span longer than 128 rows takes maxscore_qo_tile (and its splits) only where the loaded planner accepts it: + without the tile, splits alone leave the 160..15K-row proxy as slow as one split. A span of + at most 128 rows keeps the planner's own tile and takes only the split rule. Every choice gives the same score + table bits as the plain plan (0 differing scores in every config). + """ + if policy is None or rows is None: + return {} + first, last = rows + if last <= first: + return {} + max_rows = int(qo_lens_cpu[first:last].max()) + tile = policy.prefill_proxy_qo_tile + kwargs: dict = {} + if max_rows > 128: + if tile is None or not _plan_takes_maxscore_tile(fmha_sm100.fmha_sm100_plan): + return {} + kwargs["maxscore_qo_tile"] = tile + splits = policy.prefill_proxy_kv_splits or 1 + else: + splits = 1 + table = policy.prefill_proxy_kv_splits_by_rows + bucket = next((entry for entry in table if max_rows <= entry[0]), None) if table else None + if bucket is not None: + # The step's longest KV sets the floor; multi-request steps land within 3% of their best split this way. + _, top_split, min_pages = bucket + kv_pages = -(-int(kv_lens_cpu[first:last].max()) // page_size) + splits = top_split + while splits > 1 and kv_pages < min_pages * splits: + splits //= 2 + logger.info_once( + f"AGG_PROXY_SPLITS taken: prefill proxy num_kv_splits {splits} (rows {max_rows}, kv pages {kv_pages})", + key=f"msa_agg_proxy_splits_{splits}", + ) + if not kwargs and splits == 1: + return {} + kwargs["num_kv_splits"] = splits + if "maxscore_qo_tile" in kwargs: + logger.info_once( + f"AGG_PROXY_Q128 taken: prefill proxy plan maxscore_qo_tile {tile}, num_kv_splits {splits}", + key="msa_agg_proxy_qo_tile", + ) + return kwargs + + def _proxy_max_score( idx_q: torch.Tensor, idx_k_paged: torch.Tensor, @@ -109,9 +195,12 @@ def _proxy_max_score( kv_indices: torch.Tensor, sm_scale: float, causal: bool, + policy=None, ) -> torch.Tensor: """Run the fmha_sm100 MQA proxy pass and return the per-block max score. + ``policy`` (MsaRuntimePolicy) plans it under the same prefill proxy rule as prepare()'s plan (proxy_plan_kwargs). + Follows MSA's two-call pattern: fmha_sm100_plan builds the plan with output_maxscore and num_kv_heads 1, then fmha_sm100 runs with output_o disabled so only the per-block max score is produced. Returns @@ -140,7 +229,17 @@ def _proxy_max_score( page_size=page_size, output_maxscore=True, causal=causal, - num_kv_splits=1, + **{ + "num_kv_splits": 1, + **proxy_plan_kwargs( + policy, + fmha_sm100, + qo_lens_cpu, + kv_lens_cpu, + (0, int(qo_lens_cpu.shape[0])), + page_size, + ), + }, ) _, max_score = fmha_sm100.fmha_sm100( idx_q, @@ -155,6 +254,77 @@ def _proxy_max_score( return max_score +# Proxy oracle (MsaRuntimePolicy.prefill_proxy_oracle): every planned proxy span is re-scored under the plain plan +# (num_kv_splits 1, the planner's tile) and must match it exactly; per-rank coverage goes to stderr every 30 s and at +# exit (the final, partial interval included). +_PROXY_ORACLE = { + "passes": 0, + "tokens": 0, + "entries": 0, + "pad_diff": 0, + "t_last": 0.0, + "atexit": False, +} + + +def _proxy_oracle_flush(final: bool = False) -> None: + st = _PROXY_ORACLE + print( + f"AGG_PROXY_ORACLE{'_FINAL' if final else ''} rank {mpi_rank()} passes {st['passes']} tokens {st['tokens']} " + f"score_entries {st['entries']} score_diff 0 table_diff 0 out_of_domain_diff {st['pad_diff']}", + file=sys.stderr, + flush=True, + ) + + +def _proxy_oracle_record(scores, ref_scores, table, ref_table, n_valid_blocks) -> None: + """Compare one planned proxy span with the plain plan's; raise on any difference. + + The score domain is each token's valid blocks (block < n_valid_blocks[token], the only entries the selector reads); + entries past it (padding / future blocks) are counted as out_of_domain_diff, not compared. The selected table is + compared whole (every token, KV head and top-k slot, -1 padding included). + """ + st = _PROXY_ORACLE + if not st["atexit"]: + st["atexit"] = True + atexit.register(_proxy_oracle_flush, True) + rank = mpi_rank() + if scores.shape != ref_scores.shape or table.shape != ref_table.shape: + print( + f"AGG_PROXY_ORACLE REJECT rank {rank}: shape scores {tuple(scores.shape)} vs {tuple(ref_scores.shape)}, " + f"table {tuple(table.shape)} vs {tuple(ref_table.shape)}", + file=sys.stderr, + flush=True, + ) + raise RuntimeError("AGG_PROXY_ORACLE: planned proxy differs from the plain plan (shape)") + heads, blocks, tokens = scores.shape + nvb = n_valid_blocks.to(device=scores.device, dtype=torch.int64)[:tokens].clamp(0, blocks) + if int(nvb.shape[0]) != tokens: + raise RuntimeError( + f"AGG_PROXY_ORACLE: n_valid_blocks covers {int(nvb.shape[0])} of {tokens} tokens" + ) + in_domain = torch.arange(blocks, device=scores.device)[None, :, None] < nvb[None, None, :] + diff = scores.contiguous().view(torch.int32) != ref_scores.contiguous().view(torch.int32) + score_diff = int((diff & in_domain).sum()) + table_diff = int((table != ref_table).any(dim=-1).sum()) + st["passes"] += 1 + st["tokens"] += tokens + st["entries"] += int(nvb.sum()) * heads + st["pad_diff"] += int((diff & ~in_domain).sum()) + if score_diff or table_diff: + print( + f"AGG_PROXY_ORACLE REJECT rank {rank}: score_diff {score_diff} table_rows_diff {table_diff} " + f"(heads {heads}, blocks {blocks}, tokens {tokens}, pass {st['passes']})", + file=sys.stderr, + flush=True, + ) + raise RuntimeError("AGG_PROXY_ORACLE: planned proxy differs from the plain plan") + now = time.monotonic() + if now - st["t_last"] >= 30.0: + st["t_last"] = now + _proxy_oracle_flush() + + def _combined_topk_table( ctx_table: torch.Tensor, gen_table: torch.Tensor, @@ -205,6 +375,12 @@ def _group_max_reduce( return max_score +# The proxy writes every valid block and selection masks the rest. Reusing the +# step buffer avoids allocating and filling it at each sparse layer. +_PROXY_MS_REUSE = os.environ.get("TRTLLM_M31_DISABLE_PROXY_MS_REUSE", "0") != "1" +_proxy_ms_step: Optional[tuple] = None # (proxy_plan, tokens, max_score) + + class MsaIndexer: """Predictor submodule: proxy MQA scoring and top-k block selection. @@ -215,6 +391,9 @@ class MsaIndexer: def __init__(self, config: "MiniMaxM3SparseConfig"): self.config = config + # A model's selector: (max_score_kv, n_valid_blocks, init_blocks, local_blocks) -> the head-major + # [total_q, num_kv_heads, topk] table of select_blocks_from_maxscore, or None to keep the built-in select. + self.selector_hook = None def select_blocks( self, @@ -227,6 +406,8 @@ def select_blocks( kv_lens_cpu: Optional[torch.Tensor] = None, qo_offset_cpu: Optional[torch.Tensor] = None, proxy_plan: Optional[tuple] = None, + proxy_plan_ref: Optional[tuple] = None, + proxy_policy=None, max_score: Optional[torch.Tensor] = None, n_valid_blocks: Optional[torch.Tensor] = None, block_table: Optional[torch.Tensor] = None, @@ -235,6 +416,7 @@ def select_blocks( require_cutedsl: bool = False, gen_token_first: int = 0, ctx_rows: int = 0, + gen_scores: Optional[torch.Tensor] = None, ) -> torch.Tensor: """Return [total_q, num_kv_heads, topk] selected block indices. @@ -260,7 +442,11 @@ def select_blocks( gen_first = int(gen_token_first) scored = False - if ( + if gen_scores is not None: + # MIX_NVFP4_READER: the backend already scored the generation span from the NVFP4 pages. + max_score = gen_scores + scored = True + elif ( max_score is not None and block_table is not None and seq_lens_cuda is not None @@ -301,6 +487,7 @@ def select_blocks( qo_offset_cpu=qo_offset_cpu, kv_indices=kv_indices, idx_sm_scale=idx_sm_scale, + policy=proxy_policy, ) if n_valid_blocks is None: @@ -314,24 +501,54 @@ def select_blocks( gen_table = self._select(max_score, n_valid_blocks[gen_first:]) if gen_first == 0: + if not scored and proxy_plan_ref is not None and proxy_plan is not None: + ref_scores = self._proxy_scores( + idx_q, idx_k_paged, proxy_plan=proxy_plan_ref, max_score=None, + qo_lens_cpu=qo_lens_cpu, kv_lens_cpu=kv_lens_cpu, qo_offset_cpu=qo_offset_cpu, + kv_indices=kv_indices, idx_sm_scale=idx_sm_scale, + ) # fmt: skip + _proxy_oracle_record( + max_score, + ref_scores, + gen_table, + self._select(ref_scores, n_valid_blocks), + n_valid_blocks, + ) return gen_table # The proxy scores the context prefix into its own buffer, under the # plan prepare() built over rows [0, ctx_rows). Context pages are the # prefix of the flattened page table, so kv_indices needs no slice. - ctx_table = self._select( - self._proxy_scores( - idx_q[:gen_first], - idx_k_paged, - proxy_plan=proxy_plan, - max_score=None, - qo_lens_cpu=None if qo_lens_cpu is None else qo_lens_cpu[:ctx_rows], - kv_lens_cpu=None if kv_lens_cpu is None else kv_lens_cpu[:ctx_rows], - qo_offset_cpu=None if qo_offset_cpu is None else qo_offset_cpu[:ctx_rows], - kv_indices=kv_indices, - idx_sm_scale=idx_sm_scale, - ), - n_valid_blocks[:gen_first], + ctx_lens = ( + None if qo_lens_cpu is None else qo_lens_cpu[:ctx_rows], + None if kv_lens_cpu is None else kv_lens_cpu[:ctx_rows], + None if qo_offset_cpu is None else qo_offset_cpu[:ctx_rows], ) + ctx_scores = self._proxy_scores( + idx_q[:gen_first], + idx_k_paged, + proxy_plan=proxy_plan, + max_score=None, + qo_lens_cpu=ctx_lens[0], + kv_lens_cpu=ctx_lens[1], + qo_offset_cpu=ctx_lens[2], + kv_indices=kv_indices, + idx_sm_scale=idx_sm_scale, + policy=proxy_policy, + ) + ctx_table = self._select(ctx_scores, n_valid_blocks[:gen_first]) + if proxy_plan_ref is not None and proxy_plan is not None: + ref_scores = self._proxy_scores( + idx_q[:gen_first], idx_k_paged, proxy_plan=proxy_plan_ref, max_score=None, + qo_lens_cpu=ctx_lens[0], kv_lens_cpu=ctx_lens[1], qo_offset_cpu=ctx_lens[2], + kv_indices=kv_indices, idx_sm_scale=idx_sm_scale, + ) # fmt: skip + _proxy_oracle_record( + ctx_scores, + ref_scores, + ctx_table, + self._select(ref_scores, n_valid_blocks[:gen_first]), + n_valid_blocks[:gen_first], + ) return _combined_topk_table(ctx_table, gen_table, head_major=True) def _proxy_scores( @@ -346,6 +563,7 @@ def _proxy_scores( qo_offset_cpu: Optional[torch.Tensor], kv_indices: torch.Tensor, idx_sm_scale: float, + policy=None, ) -> torch.Tensor: """Run the fmha_sm100 proxy pass over `idx_q` and return its max score. @@ -362,8 +580,15 @@ def _proxy_scores( kv_indices=kv_indices, sm_scale=idx_sm_scale, causal=True, + policy=policy, ) + global _proxy_ms_step fmha_sm100 = require_msa_module() + tokens = int(idx_q.shape[0]) + reuse = _PROXY_MS_REUSE and max_score is None + cached = _proxy_ms_step + if reuse and cached is not None and cached[0] is proxy_plan and cached[1] == tokens: + max_score = cached[2] _, scores = fmha_sm100.fmha_sm100( idx_q, idx_k_paged, @@ -375,6 +600,8 @@ def _proxy_scores( max_score=max_score, sm_scale=idx_sm_scale, ) + if reuse and scores is not None and scores.is_contiguous(): + _proxy_ms_step = (proxy_plan, tokens, scores) return scores def _select( @@ -388,8 +615,16 @@ def _select( decode kernel reads the table head-major and takes every generation row, while fmha_sm100 reads whatever strides it is handed. """ + max_score_kv = _group_max_reduce(max_score, self.config) + hook = self.selector_hook + if hook is not None: + table = hook( + max_score_kv, n_valid_blocks, self.config.init_blocks, self.config.local_blocks + ) + if table is not None: + return table return select_blocks_from_maxscore( - _group_max_reduce(max_score, self.config), + max_score_kv, topk=MSA_REQUIRED_TOPK, n_valid_blocks=n_valid_blocks, init_blocks=self.config.init_blocks, diff --git a/tensorrt_llm/_torch/autotuner.py b/tensorrt_llm/_torch/autotuner.py index 6fac5ce27ff1..1ac393a8e2a3 100644 --- a/tensorrt_llm/_torch/autotuner.py +++ b/tensorrt_llm/_torch/autotuner.py @@ -271,6 +271,14 @@ def unique_id(self): """ return tuple(self.__dict__.values()) + def tactic_name(self, tactic: Any) -> Optional[str]: + """A name of ``tactic`` that stays the same across builds (e.g. its kernel or config), or None. + + Tactic caches shipped with a model key entries by this name rather than by the tactic value, which may be + an index into a list that a rebuild reorders. None means the runner has no such name. + """ + return None + @contextlib.contextmanager def autotune(tune_mode: bool = True, @@ -471,6 +479,11 @@ def __init__(self): # from choose_one when an excluded op is first seen. self.excluded_op: Set[str] = set() + # Entries loaded from a name-keyed tactic bundle that have not been + # looked up yet: cache key -> the tactic's name (TunableRunner.tactic_name). + # The first lookup checks the stored tactic against the name. + self.bundled_names: Dict[Tuple, str] = dict() + # Cache metadata for local storage and validation self.lib_version = tensorrt_llm.__version__ self.creation_timestamp = time.time() @@ -491,6 +504,7 @@ def clear(self) -> None: self.cache.clear() self.independent_op.clear() self.excluded_op.clear() + self.bundled_names.clear() def fallback_entry(self) -> Tuple: # runner_id = 0, tactic = -1 @@ -938,6 +952,9 @@ def __init__(self, warmup=2, repeat=10, stream_delay_micro_secs=1000): self.warmup = warmup self.stream_delay_micro_secs = stream_delay_micro_secs self.profiling_cache = AutoTunerProfilingCache() + # Told about bundled entries no tactic matches and about shapes a bundle + # did not cover (``unresolved(cache_key, name)`` / ``retune(custom_op, shapes)``). + self.bundle_reporter: Optional[Any] = None self._primed_cached_tactics: set[tuple[str, str, str, str]] = set() self.is_tuning_mode = False self.skip_dynamic_tuning_buckets = False @@ -1162,6 +1179,20 @@ def choose_one( input_shapes, tuning_config, apply_map_to_tuning_buckets=True)) + if is_cache_hit and self.profiling_cache.bundled_names: + cache_key = self.profiling_cache.get_cache_key( + custom_op, + runners[best_runner_id], + input_shapes, + tuning_config, + apply_map_to_tuning_buckets=True) + if self._resolve_bundled(cache_key, runners[best_runner_id], + inputs, **kwargs): + best_tactic = self.profiling_cache.cache[cache_key][1] + else: + is_cache_hit = False + best_runner_id, best_tactic, min_time = ( + self.profiling_cache.fallback_entry()) # Early return if it's not tuning, use cache found one or fallback one if not self.is_tuning_mode: @@ -1214,13 +1245,24 @@ def choose_one( tuning_config.distributed_tuning_strategy): for p in profiles: tensors = self._prepare_input_tensors(p, inputs) - is_cache_hit, *_ = self.profiling_cache.search_cache( + is_cache_hit, hit_runner_id, *_ = self.profiling_cache.search_cache( custom_op, runners, p.get_opt_shapes(), tuning_config, apply_map_to_tuning_buckets=False, ) + if is_cache_hit and self.profiling_cache.bundled_names: + cache_key = self.profiling_cache.get_cache_key( + custom_op, + runners[hit_runner_id], + p.get_opt_shapes(), + tuning_config, + apply_map_to_tuning_buckets=False) + is_cache_hit = self._resolve_bundled( + cache_key, runners[hit_runner_id], inputs, **kwargs) + if not is_cache_hit and self.bundle_reporter is not None: + self.bundle_reporter.retune(custom_op, p.get_opt_shapes()) if not is_cache_hit: # Initialize runner and tactic as None in case of no valid tactic or runners are found with nvtx_range(f"{custom_op}, shape {p.get_opt_shapes()}"): @@ -1318,6 +1360,37 @@ def _prime_cached_tactics( f"runner={type(runner).__name__}, tactic={tactic}, " f"shapes={profile.get_opt_shapes()}. Error: {e}") + def _resolve_bundled(self, cache_key: Tuple, runner: TunableRunner, + inputs: List[torch.Tensor], **kwargs) -> bool: + """Check a cache entry loaded from a name-keyed bundle on its first lookup. + + True when the stored tactic still carries the bundled name, or when another valid tactic of ``runner`` + does (that tactic replaces the stored one). Otherwise the entry is dropped, the reporter is told, and + False sends the caller to the miss path (tuning, or the fallback outside tuning). + """ + name = self.profiling_cache.bundled_names.pop(cache_key, None) + if name is None or cache_key not in self.profiling_cache.cache: + return cache_key in self.profiling_cache.cache + + def name_of(tactic: Any) -> Optional[str]: + try: + return runner.tactic_name(tactic) + except (RuntimeError, IndexError, ValueError, TypeError): + return None + + runner_id, tactic, min_time = self.profiling_cache.cache[cache_key] + if name_of(tactic) == name: + return True + for candidate in runner.get_valid_tactics(inputs, None, **kwargs): + if name_of(candidate) == name: + self.profiling_cache.cache[cache_key] = (runner_id, candidate, + min_time) + return True + del self.profiling_cache.cache[cache_key] + if self.bundle_reporter is not None: + self.bundle_reporter.unresolved(cache_key, name) + return False + def _profile_runners( self, custom_op: str, diff --git a/tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py b/tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py index f500034c4df7..cb48da541dd3 100644 --- a/tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py +++ b/tensorrt_llm/_torch/custom_ops/cpp_custom_ops.py @@ -1849,3 +1849,10 @@ def _(like: torch.Tensor, out_shape = shape if shape is not None else list(like.shape) dtype = out_dtype if out_dtype is not None else like.dtype return like.new_empty(out_shape, dtype=dtype), output_buffer_kind, 0 + + # The MiniMax-M3 / M3.1 ops' fakes live with the targets that call them; register them with the rest. + from .._experimental.modeling_v2.catalog.fakes import m31_fake_ops + from .._experimental.modeling_v2.models.minimax_m3 import \ + fake_ops as minimax_m3_fake_ops + m31_fake_ops.register() + minimax_m3_fake_ops.register() diff --git a/tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py b/tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py index f01da1dc55ed..69cc41e928d4 100644 --- a/tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py +++ b/tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py @@ -9235,8 +9235,8 @@ def is_supported( @classmethod def _compile(cls, q_dtype: torch.dtype, num_heads: int, - max_decode_query_len: int, head_dim: int): - key = (q_dtype, num_heads, max_decode_query_len, head_dim) + max_decode_query_len: int, head_dim: int, split_k: int): + key = (q_dtype, num_heads, max_decode_query_len, head_dim, split_k) if key in cls.kernel_cache: return @@ -9283,7 +9283,7 @@ def _sym_stride(): cute_dtype, num_heads, max_decode_query_len, - cls.SPLIT_K, + split_k, head_dim, ) cls.kernel_cache[key] = cute.compile( @@ -9308,19 +9308,24 @@ def forward( seq_lens: torch.Tensor, score: torch.Tensor, max_decode_query_len: int, + split_k: int = SPLIT_K, ) -> None: """Score one decode step, compiling the kernel on first use. + ``split_k`` is the number of CTAs per request; each block is scored + by exactly one of them, so the scores do not depend on it. + The compile allocates, so it must not land inside a CUDA graph capture. It does not: CUDAGraphRunner.capture runs eager warmup forwards first, and those cover every geometry the graphs it then captures will replay. """ _, num_heads, head_dim = idx_q.shape - key = (idx_q.dtype, num_heads, max_decode_query_len, head_dim) + key = (idx_q.dtype, num_heads, max_decode_query_len, head_dim, + split_k) if key not in cls.kernel_cache: cls._compile(idx_q.dtype, num_heads, max_decode_query_len, - head_dim) + head_dim, split_k) cls.kernel_cache[key](idx_q, index_k_cache, block_table, score, seq_lens) @@ -9334,6 +9339,7 @@ def cute_dsl_minimax_m3_index_decode_score( seq_lens: torch.Tensor, score: torch.Tensor, max_decode_query_len: int, + split_k: int = 256, ) -> None: """Write per-block max index scores for one decode step, in place. @@ -9352,6 +9358,8 @@ def cute_dsl_minimax_m3_index_decode_score( passed without a copy. max_decode_query_len: compile-time bound on decode_query_len; num_index_heads * max_decode_query_len must not exceed 32. + split_k: CTAs per request (compile-time); each walks blocks + split, split + split_k, ... The scores do not depend on it. """ if not is_sm_100f(): raise ValueError( @@ -9376,6 +9384,7 @@ def cute_dsl_minimax_m3_index_decode_score( seq_lens, score, max_decode_query_len, + split_k, ) @torch.library.register_fake( @@ -9387,6 +9396,7 @@ def _( seq_lens: torch.Tensor, score: torch.Tensor, max_decode_query_len: int, + split_k: int = 256, ) -> None: return None diff --git a/tensorrt_llm/_torch/custom_ops/mxfp8_cute_pinned_tactics.py b/tensorrt_llm/_torch/custom_ops/mxfp8_cute_pinned_tactics.py new file mode 100644 index 000000000000..1511adaed71a --- /dev/null +++ b/tensorrt_llm/_torch/custom_ops/mxfp8_cute_pinned_tactics.py @@ -0,0 +1,160 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Pinned FlashInfer CuTe DSL MXFP8 GEMM tactics for SM107 (MiniMax-M3 TP2 projections), above 32 rows. + +TRT-LLM runs FlashInfer's CuTe DSL MXFP8 GEMM with FlashInfer's config sweep skipped, so above 32 rows (the +split-K low-M kernel's limit) every call takes the default tile (128, 128), cluster (1, 1): ceil(N / 128) CTAs, a +quarter of the SMs streaming the weight. This table holds FlashInfer's own tuned tactic per (N, K, M bucket) on +SM107. Every entry is non-split-K (split_k 1, each output keeps its full-K accumulation order), so its output equals +the default tactic's bit for bit (measured on all entries), at 18..40% less time from 33 to 16384 rows. A fixed +table rather than live tuning keeps the choice identical across runs and nodes. Used where TRT-LLM calls the +FlashInfer runner (decode-graph capture, autotune); eager prefill takes other GEMMs. Off unless armed: the executor +calls arm() for a model whose RuntimeHooks set mxfp8_pinned_tactics. Kill: TRTLLM_MXFP8_CUTE_PIN=0. +""" + +import os +from typing import Dict, Optional, Tuple + +import torch + +from tensorrt_llm._utils import get_sm_version +from tensorrt_llm.logger import logger + +ENABLED = os.environ.get("TRTLLM_MXFP8_CUTE_PIN", "1") == "1" +MIN_ROWS = 33 # 1..32 rows keep FlashInfer's split-K low-M path (its tuned tactics there are split-K: not bitwise) +MAX_BUCKET = 16384 + +Tactic = Tuple[Tuple[int, int], Tuple[int, int], bool, bool, int] +# (N, K, M bucket = next power of two >= M, 64..16384) +# -> (mma_tiler_mn, cluster_shape_mn, swap_ab, use_prefetch, split_k) +TACTICS: Dict[Tuple[int, int, int], Tactic] = { + (3072, 6144, 64): ((256, 64), (2, 1), True, False, 1), + (3072, 6144, 128): ((256, 64), (2, 1), True, False, 1), + (3072, 6144, 256): ((256, 64), (2, 1), True, False, 1), + (3072, 6144, 512): ((256, 64), (2, 1), False, False, 1), + (3072, 6144, 1024): ((256, 128), (2, 1), False, False, 1), + (3072, 6144, 2048): ((256, 256), (2, 1), False, False, 1), + (3072, 6144, 4096): ((256, 192), (4, 2), False, False, 1), + (3072, 6144, 8192): ((256, 192), (2, 1), False, False, 1), + (3072, 6144, 16384): ((256, 192), (2, 1), True, False, 1), + (4608, 6144, 64): ((256, 64), (2, 1), True, False, 1), + (4608, 6144, 128): ((256, 64), (2, 1), True, False, 1), + (4608, 6144, 256): ((256, 64), (2, 4), False, True, 1), + (4608, 6144, 512): ((256, 128), (2, 2), False, False, 1), + (4608, 6144, 1024): ((256, 192), (2, 1), False, False, 1), + (4608, 6144, 2048): ((256, 192), (2, 1), False, False, 1), + (4608, 6144, 4096): ((256, 256), (2, 1), False, False, 1), + (4608, 6144, 8192): ((256, 192), (2, 4), True, False, 1), + (4608, 6144, 16384): ((256, 256), (2, 1), True, False, 1), + (4992, 6144, 64): ((256, 64), (2, 1), True, False, 1), + (4992, 6144, 128): ((256, 64), (2, 1), True, False, 1), + (4992, 6144, 256): ((256, 64), (2, 4), False, True, 1), + (4992, 6144, 512): ((256, 128), (4, 2), False, True, 1), + (4992, 6144, 1024): ((256, 192), (2, 1), False, False, 1), + (4992, 6144, 2048): ((256, 192), (2, 1), False, False, 1), + (4992, 6144, 4096): ((256, 192), (2, 1), False, False, 1), + (4992, 6144, 8192): ((256, 192), (2, 1), False, False, 1), + (4992, 6144, 16384): ((256, 192), (2, 1), True, False, 1), + (6144, 1536, 64): ((256, 64), (2, 1), True, False, 1), + (6144, 1536, 128): ((256, 64), (2, 1), True, False, 1), + (6144, 1536, 256): ((256, 64), (2, 1), True, False, 1), + (6144, 1536, 512): ((256, 128), (2, 1), False, False, 1), + (6144, 1536, 1024): ((256, 256), (2, 1), False, False, 1), + (6144, 1536, 2048): ((256, 192), (2, 4), False, False, 1), + (6144, 1536, 4096): ((256, 192), (2, 1), False, False, 1), + (6144, 1536, 8192): ((256, 192), (2, 1), False, False, 1), + (6144, 1536, 16384): ((256, 192), (2, 1), True, False, 1), + (6144, 4096, 64): ((256, 64), (2, 1), True, False, 1), + (6144, 4096, 128): ((256, 64), (4, 1), True, False, 1), + (6144, 4096, 256): ((256, 64), (2, 1), True, False, 1), + (6144, 4096, 512): ((256, 128), (2, 1), False, False, 1), + (6144, 4096, 1024): ((256, 256), (2, 1), False, False, 1), + (6144, 4096, 2048): ((256, 192), (2, 4), False, False, 1), + (6144, 4096, 4096): ((256, 192), (2, 1), False, False, 1), + (6144, 4096, 8192): ((256, 192), (2, 1), True, False, 1), + (6144, 4096, 16384): ((256, 192), (2, 1), True, False, 1), + (6144, 6144, 64): ((256, 64), (2, 1), True, False, 1), + (6144, 6144, 128): ((256, 64), (2, 1), True, False, 1), + (6144, 6144, 256): ((256, 64), (2, 1), True, False, 1), + (6144, 6144, 512): ((256, 128), (2, 1), True, False, 1), + (6144, 6144, 1024): ((256, 256), (2, 1), False, False, 1), + (6144, 6144, 2048): ((256, 192), (2, 4), False, False, 1), + (6144, 6144, 4096): ((256, 192), (2, 1), False, False, 1), + (6144, 6144, 8192): ((256, 192), (2, 1), False, False, 1), + (6144, 6144, 16384): ((256, 256), (2, 1), True, False, 1), + (12288, 6144, 64): ((256, 64), (2, 1), True, False, 1), + (12288, 6144, 128): ((128, 64), (1, 2), True, False, 1), + (12288, 6144, 256): ((256, 128), (2, 1), False, False, 1), + (12288, 6144, 512): ((256, 256), (2, 1), False, False, 1), + (12288, 6144, 1024): ((256, 192), (2, 2), False, False, 1), + (12288, 6144, 2048): ((256, 192), (2, 1), False, False, 1), + (12288, 6144, 4096): ((256, 192), (2, 1), False, False, 1), + (12288, 6144, 8192): ((256, 256), (2, 1), False, False, 1), + (12288, 6144, 16384): ((256, 256), (2, 1), True, False, 1), +} + +_STATE: dict = {} +# Process-wide once armed: another model later or concurrently in the same process, at a table shape on SM107, +# also takes the pinned tactic. Its outputs are unchanged (each entry is bitwise the default tactic), only faster. +# The LLM API runs one executor per worker process. +_ARMED = [False] + + +def arm() -> None: + """Engage the table for this process (a model opted in through RuntimeHooks.mxfp8_pinned_tactics).""" + _ARMED[0] = True + + +def _bucket(m: int) -> int: + return max(64, 1 << (m - 1).bit_length()) + + +def tactic_for(m: int, n: int, k: int) -> Optional[Tactic]: + """The pinned tactic of an [m, k] x [n, k]^T call, or None (FlashInfer's own path).""" + if not ENABLED or not _ARMED[0] or m < MIN_ROWS or m > MAX_BUCKET or get_sm_version() != 107: + return None + return TACTICS.get((n, k, _bucket(m))) + + +def run( + act: torch.Tensor, + act_scale: torch.Tensor, + weight: torch.Tensor, + weight_scale: torch.Tensor, + out_dtype: torch.dtype, +) -> Optional[torch.Tensor]: + """The MXFP8 GEMM act @ weight^T on FlashInfer's CuTe DSL runner at the pinned tactic, or None when no entry covers + the call (the caller runs FlashInfer's default path). Bitwise equal to that default path where an entry exists.""" + m, k = act.shape + n = weight.shape[0] + tactic = tactic_for(m, n, k) + if tactic is None: + return None + try: + from flashinfer.gemm import gemm_base as _gb + + key = (out_dtype, act.device) + st = _STATE.get(key) + if st is None: + major, minor = _gb.get_compute_capability(act.device) + st = _STATE[key] = ( + _gb._cute_dsl_gemm_mxfp8_runner(major, minor, True, out_dtype), + _gb._get_cache_buf("mm_mxfp8_workspace", _gb.DEFAULT_WORKSPACE_SIZE, act.device), + ) + logger.info( + "[MXFP8 GEMM] pinned CuTe DSL tactics engaged above 32 rows (TRTLLM_MXFP8_CUTE_PIN)" + ) + runner, workspace = st + out = torch.empty((m, n), dtype=out_dtype, device=act.device) + runner( + inputs=[act, weight.t(), act_scale, weight_scale, out_dtype, out, workspace], + tactic=tactic, + ) + return out + except Exception as e: # noqa: BLE001 - a FlashInfer API change falls back to the default path, once loudly + if "failed" not in _STATE: + _STATE["failed"] = True + logger.warning( + f"[MXFP8 GEMM] pinned CuTe DSL tactic unavailable ({type(e).__name__}: {e}); default path" + ) + return None diff --git a/tensorrt_llm/_torch/custom_ops/torch_custom_ops.py b/tensorrt_llm/_torch/custom_ops/torch_custom_ops.py index 7ac21c784c47..797a473b9467 100644 --- a/tensorrt_llm/_torch/custom_ops/torch_custom_ops.py +++ b/tensorrt_llm/_torch/custom_ops/torch_custom_ops.py @@ -39,10 +39,11 @@ from ..cublaslt_utils import IS_CUBLASLT_AVAILABLE from ..cute_dsl_utils import IS_CUTLASS_DSL_AVAILABLE from ..flashinfer_utils import IS_FLASHINFER_AVAILABLE, get_env_enable_pdl +from . import mxfp8_cute_pinned_tactics as _mxfp8_cute_pinned from .fast_custom_op import fast_custom_op if IS_FLASHINFER_AVAILABLE: - from flashinfer import autotune as _flashinfer_autotune + from ..flashinfer_autotune_cache import autotune as _flashinfer_autotune from flashinfer import mm_mxfp8 as _flashinfer_mm_mxfp8 from flashinfer import mxfp8_quantize as _flashinfer_mxfp8_quantize from flashinfer.fp4_quantization import nvfp4_quantize as _flashinfer_nvfp4_quantize @@ -648,6 +649,12 @@ def get_valid_tactics(self, inputs: List[torch.Tensor], """Return the generic fallback followed by every compiled tactic.""" return [-1, *range(self.mxfp8_gemm_runner.get_num_configs())] + def tactic_name(self, tactic: int) -> Optional[str]: + """The CUTLASS config of a native tactic; "fallback" for -1.""" + if tactic == -1: + return "fallback" + return self.mxfp8_gemm_runner.get_config_name(int(tactic)) + @classmethod def sync_all_tactic_caches(cls, tuner: AutoTuner) -> None: """Register every profiled MXFP8 tactic in the native serving cache.""" @@ -802,6 +809,13 @@ def forward( backend = "cutlass" context = nullcontext() else: + # Above 32 rows on SM107, for a model that opts in: the pinned tuned tactic (bitwise the + # heuristic's output, faster); see mxfp8_cute_pinned_tactics. Otherwise, or with no entry: + # FlashInfer's heuristic config. + pinned = _mxfp8_cute_pinned.run(act, act_scale, weight, + weight_scale, self.output_dtype) + if pinned is not None: + return pinned # Skip FlashInfer's CuTeDSL config sweep; use its heuristic config. backend = "cute-dsl" context = _flashinfer_autotune(tune_mode=False, diff --git a/tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py b/tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py index 0063abfa4f1c..a2410c0e6569 100644 --- a/tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py +++ b/tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py @@ -542,6 +542,13 @@ def get_valid_tactics(self, inputs: List[torch.Tensor], return tactics + def tactic_name(self, tactic: List[int]) -> Optional[str]: + """The gemm1 / gemm2 kernel names of a ``[tileN, config]`` tactic; "fallback" for the default.""" + tile_n, config = (int(v) for v in tactic) + if tile_n == -1 or config == -1: + return "fallback" + return self.get_runner().get_config_name(tile_n, config) + @classmethod def get_dynamic_tensor_specs( cls, diff --git a/tensorrt_llm/_torch/cute_aot.py b/tensorrt_llm/_torch/cute_aot.py new file mode 100644 index 000000000000..7d4bd02d42ed --- /dev/null +++ b/tensorrt_llm/_torch/cute_aot.py @@ -0,0 +1,224 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Ahead-of-time CuTe DSL kernels shipped as wheel data. + +``compile_or_load(key, compile_fn)`` returns the compiled function of a CuTe DSL kernel without compiling it when this +package ships its exported object: ``cute_aot_data//.o``. The toolchain names the CuTe DSL and +TVM-FFI versions and the GPU architecture; the digest is that of ``key`` and of the contents of ``sources``. ``key`` +must name everything the compile specializes on (kernel, constexpr arguments, argument dtypes / static shapes / +alignments, compile options), since an object is served for any compile under the same key; ``sources`` are the files +whose code the kernel is built from (its module and the modules it traces), so an edited kernel no longer matches its +old object. The key also carries the process's model switches and CuTe DSL settings (``TRTLLM_M31_DISABLE_*``, +``TRTLLM_M31_ENABLE_*``, ``CUTE_DSL_*`` other than directories and logging), which kernel modules may read when they +are traced: objects are harvested under the serving environment, and a process with other settings compiles. +Without a shipped object, ``compile_fn()`` runs and one ``[M31_CUTE_AOT] compile `` line is logged, so a run can +count the kernels that compiled in-process. + +``TRTLLM_M31_CUTE_AOT_DIR`` replaces the shipped directory. ``TRTLLM_M31_CUTE_AOT_EXPORT_DIR`` exports every kernel this +process compiles there (``/.o`` + ``.h``, and ``.key`` holding the key and the source +digests), the layout to copy into ``cute_aot_data``. ``TRTLLM_M31_CUTE_AOT_DISABLE=1`` always compiles. + +The served object is the exported compile of the same kernel source under the same key and toolchain, so its device +code is that of the in-process compile. A kernel compiled with TVM-FFI is exported and loaded as a TVM-FFI function +(``tvm_ffi=True``); its calling convention is then TVM-FFI's. +""" + +import functools +import hashlib +import inspect +import os +import tempfile +import threading +from pathlib import Path +from typing import Any, Callable, Dict, Optional, Sequence, Tuple + +from tensorrt_llm.logger import logger + +DIR_ENV = "TRTLLM_M31_CUTE_AOT_DIR" +EXPORT_DIR_ENV = "TRTLLM_M31_CUTE_AOT_EXPORT_DIR" +DISABLE_ENV = "TRTLLM_M31_CUTE_AOT_DISABLE" +DATA_ROOT = Path(__file__).resolve().parent / "cute_aot_data" + +_lock = threading.RLock() +_loaded: Dict[Tuple[str, str], Any] = {} +_counts = {"loaded": 0, "compiled": 0} + + +def _disabled() -> bool: + return os.environ.get(DISABLE_ENV, "0").strip().lower() in ("1", "true", "on", "yes") + + +def toolchain() -> str: + """The toolchain an exported object is valid for: the CuTe DSL and TVM-FFI versions and this GPU's architecture.""" + import importlib.metadata as md + + import torch + + def version(dist: str) -> str: + try: + return md.version(dist) + except md.PackageNotFoundError: + return "none" + + major, minor = torch.cuda.get_device_capability() + return f"cutedsl-{version('nvidia-cutlass-dsl')}-tvmffi-{version('apache-tvm-ffi')}-sm{major}{minor}" + + +def digest(key: str) -> str: + """The file stem of ``key``'s object (also its symbol prefix, which must be a C identifier).""" + return "k" + hashlib.sha256(key.encode()).hexdigest()[:32] + + +_ENV_PREFIXES = ("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_", "CUTE_DSL_") +_ENV_IGNORED_SUFFIXES = ( + "_DIR", + "_PATH", + "_LOG_LEVEL", + "_LOG_TO_CONSOLE", + "_LOG_TO_FILE", + "_DISABLE_FILE_CACHING", +) + + +def env_settings() -> Tuple[Tuple[str, str], ...]: + """The environment settings a traced kernel module may read (see the module docstring), sorted.""" + return tuple( + sorted( + (k, v) + for k, v in os.environ.items() + if k.startswith(_ENV_PREFIXES) and not k.endswith(_ENV_IGNORED_SUFFIXES) + ) + ) + + +def full_key(key: str, sources: Sequence[str] = ()) -> str: + """``key`` with a digest of the source files' contents and the environment settings appended (the string + ``digest`` is taken of). Only the file contents count, not where the files are installed.""" + if sources: + combined = hashlib.sha256(" ".join(sorted(_file_sha(str(src)) for src in sources)).encode()) + key = f"{key} | src {len(sources)} files {combined.hexdigest()[:16]}" + env = env_settings() + return f"{key} | env {' '.join(f'{k}={v}' for k, v in env)}" if env else key + + +_file_shas: Dict[str, str] = {} +_tree_files: Dict[Tuple[str, ...], Tuple[str, ...]] = {} + + +def py_files(*dirs: str) -> Tuple[str, ...]: + """Every ``.py`` file under ``dirs`` (sorted): ``sources`` for kernels that trace code from these trees.""" + found = _tree_files.get(dirs) + if found is None: + found = _tree_files[dirs] = tuple( + sorted(str(f) for d in dirs for f in Path(d).rglob("*.py")) + ) + return found + + +def _file_sha(path: str) -> str: + sha = _file_shas.get(path) + if sha is None: + sha = _file_shas[path] = hashlib.sha256(Path(path).read_bytes()).hexdigest()[:16] + return sha + + +def object_path(key: str, tool: Optional[str] = None) -> Path: + """The shipped object of a full key (``full_key``).""" + root = Path(os.environ[DIR_ENV]) if os.environ.get(DIR_ENV) else DATA_ROOT + return root / (tool or toolchain()) / f"{digest(key)}.o" + + +def _load(path: Path, symbol: str, tvm_ffi: bool) -> Any: + from cutlass.cute.runtime import load_module + + return getattr(load_module(str(path), enable_tvm_ffi=tvm_ffi), symbol) + + +def _export(compiled: Any, key: str, tool: str, symbol: str, tvm_ffi: bool) -> None: + """Export into a private directory, then rename each file into place: every rank of a role may export the same + (identical) object concurrently, and a reader never sees a partial file.""" + out = Path(os.environ[EXPORT_DIR_ENV]) / tool + out.mkdir(parents=True, exist_ok=True) + with tempfile.TemporaryDirectory(dir=out, prefix=f".{symbol}.") as tmp: + if tvm_ffi: + compiled.export_to_c( + os.path.join(tmp, f"{symbol}.o"), + function_name=symbol, + export_only_tvm_ffi_symbols=True, + ) + else: + compiled.export_to_c(tmp, symbol, function_prefix=symbol) + Path(tmp, f"{symbol}.key").write_text(key + "\n") + for name in sorted(os.listdir(tmp)): + os.replace(os.path.join(tmp, name), out / name) + + +def compile_or_load( + key: str, compile_fn: Callable[[], Any], *, sources: Sequence[str] = (), tvm_ffi: bool = False +) -> Any: + """The compiled function of ``key`` built from ``sources``: the shipped object when there is one, else + ``compile_fn()``.""" + tool = toolchain() + key = full_key(key, sources) + with _lock: + cached = _loaded.get((tool, key)) + if cached is not None: + return cached + symbol = digest(key) + path = object_path(key, tool) + compiled = None + if not _disabled() and path.is_file(): + try: + compiled = _load(path, symbol, tvm_ffi) + _counts["loaded"] += 1 + except Exception as e: # noqa: BLE001 (a stale or foreign object falls back to compiling) + logger.warning(f"[M31_CUTE_AOT] stale {path.name} for {key}: {e!r}") + if compiled is None: + logger.info(f"[M31_CUTE_AOT] compile {key}") + compiled = compile_fn() + _counts["compiled"] += 1 + if os.environ.get(EXPORT_DIR_ENV): + _export(compiled, key, tool, symbol, tvm_ffi) + _loaded[(tool, key)] = compiled + return compiled + + +def cached_compile( + name: str, sources: Callable[[], Sequence[str]], *, tvm_ffi: bool = False +) -> Callable[[Callable[..., Any]], Callable[..., Any]]: + """Decorate a function that builds and compiles one kernel from hashable arguments (in place of + ``functools.cache``): each argument set is compiled or loaded once per process through ``compile_or_load``, keyed + by ``name`` and the arguments' reprs (bound to the signature, defaults applied).""" + + def decorate(fn: Callable[..., Any]) -> Callable[..., Any]: + signature = inspect.signature(fn) + + @functools.cache + def compiled(*args: Any, **kwargs: Any) -> Any: + bound = signature.bind(*args, **kwargs) + bound.apply_defaults() + key = f"{name} " + " ".join(f"{k}={v!r}" for k, v in bound.arguments.items()) + return compile_or_load( + key, lambda: fn(*args, **kwargs), sources=sources(), tvm_ffi=tvm_ffi + ) + + return functools.wraps(fn)(compiled) + + return decorate + + +def counts() -> Dict[str, int]: + """How many kernels this process loaded from objects and compiled.""" + return dict(_counts) diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k00eed757e39938fb731637e794868820.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k00eed757e39938fb731637e794868820.o new file mode 100644 index 0000000000000000000000000000000000000000..4550dbb88e757dac99b1462bf106d395394d9075 GIT binary patch literal 103384 zcmeFa3t&~%nKr!6B_}5*x06E%354Vj0wf`X5E6{Y<%9&noybK{fh1gm+|onPji&W@ zaI9KKLpop8nfYe;Y^Ou0Sf|skC7qc{tAFdLwbp;8`p?YYI%sX6R{2t^M8)$z?^^q^ z)=Ef1rJeul=hm>#v)^@n_j=y&Yg=_E^)hD%sO1`+sr}MRmNWJa;`@BzNqX<^v&uk?phU6U9)cGgpP@^$>Z;VCTp1zcsefx?2$8j!Q_2d3q-u_Ylbe#9$yzHT0_b-3qT>lE3SK=I)a!t=azx{=mZ+Y_z zFQ?6z)3fK)#UJ&b|J4iqJrDf4|K#JpM!K4wo(CU%`Ib`;ynNq`$NQJR zc&dNTffxHvKJcUdb6@*)|M|UdAx(A9eGlIM^2r$wyxfDj&p-co{~pwNe$|Wp%TU+J z8I@5}gMe_7^>{pY^$c7NHAsIIv^rI455Z_nz- z(f+UXm+pJ9|CYyJ?C*K&SN-=tNNKL^IroJJUOtI5_f>qozii1@`%|8Hp+C_0LjNAL z`TX8Dk*2nXbhZ46KYjVWlE?eY9zb0cFZ3@jdA|Rj`+nVj|H;1ol+t-U=T_YFa@nar zefiu9)K&4-{&VR+?qBZvA=>j+|M}@}BF+4sWxmg&ocmrr_l1-FWob|J_q^~UwChFm z@7MihtKLDH1wG3i`qP(t9yrNmbUnpYcyZYi_pL2`>ub@Jy9!UI_+o2QzE^0_eT{fn zjUV^S<&Lf0hA}qQ6&ZNVGxj9OTHLJ09uJU-@h}$SO^k^a7pr*|W5VSk+1Qh3-ToDn z?>J*M-@~;co=24R6voZFYwCO6+qekhFfwrIj%QBD<9Vp@`L4%RW{puAkJea>U@U$c zWA)Rq_{TLCW5bNa<0Fj47XxFMIvslz-eC8Ti(9 zk%3Y#%N)m!u6@_zJNa9W7u;bdZ!7Q(w1ke%XI^jSo*dt$wy%16Py9SG&@vvMUvv$e z`1`uvwg++TmB>Kb_kq6*d@JyefxiX(Mc}Q#KLkDrd=z*A@KeBl4txlBD)6^~9|YbG zoB{lG;5hIK;J4kqC%z0E1-=gWr@(&!ybE|T@V9^;0=@xw4DeThKM%Yb*aQ3};5&hr z0RJY^+jbA|ZNLkFUk5$_{571n;=B&m%W(ZKfR_WW04@Ul0&oRzJ#Yy4jzzs~AA5Sy zo`JRxfUm*%J)CFb+>i52oZrTII?m^Do`&)6h- zmB)5{zw+4b$10C4!u7-i*889E;fXP^7+|aYg59pwc{hPwR@gOx)fg3=Y6&?7P|dg zNdqlEi_d3EPk2wj?}uMIK80<0>K85FNV-IAY;jLNw&VLVj_rJG#<5-T8OL^Sn{jLs z+Gn-#I%+Gm5$&UPzHa&r-p+5K4d0+Pq@8O0EhhmzeGs4Tr_af!TEB$P-SpXas`an%`OoQdl4ws}to!!&yaRjQ z_M$C)rjA~l((*azY|B%B($eoW^)v--nuaz_Lz||dP16)Ty^c6a){}iNZ$N!(sXp(i z)~)z_6MgoaYTbp;+vv0VROHxjjCkxOuxiTzj%oB4X2i+S?TM|^oz{&b!K{YtgMdxq#5nsTd}Bjd5{fEN4&H* zR5vgkG1NKOd0W05@@!LHG=zLBVZBs=;;dy$AA2Q@&sb`IjKCHswo(kbf2OT2sDy2>G`lpJ&Q% z7(yO}JOa6?1H0Wn#q|a=->xC@y$$(NQyv{cek z9~?ryfbwUF{C_@#d@JQQ2E!PgH^`XR30PU*)EPX9SdKg9L>DSgi1^k1U%LtKB1(vKUQ{;QOJi0j{= z^tpr6f0xn^as4?;pEo%DPbvKn*Z+;uj~|@=XOw=3>t`wbgu&_Gr1V2vKTqlN2d96V z(hqUHpVChpoc=vZKg9J9D1E`;^dHyv2JYh56}u3?_@KF7i@W$(iL3A*U6|{UFM0~T z|J^l95YsM3Y}?TD!R~s*yo-7+-BH(bZt@TJ(L612)Pq>qkLxLr2aqlm>C+Gcr}vyg zyw``A^p|(C-Z!sf1HVDs`EF`m?{5)@zIQY-@QZ_p4ZfJv^FHF!iyuaM2N19RXKJAI z1H`W%y|cRK(wFOc`=+Iop8IiRpbs(I$NwG~_|MlP181MF>wO(D*Bh_b_5K2}*e~8^ z`0TU!V&jjQ+U^RC`sB5u1d0re-?!2jA_*W3RE%FhPQBfk69m_@xG|2rFa z3$fp?Cok%~^krA?+lUk2sbAFl(K{JE|CzeH^z|<;?)l({4DF1d{gHuzqbT#vNbd)` zm-k%!koA6aa2a&5w5RWHJw1K17xkX=Eb4v#DK>Dic}35AN7=w{54n2Z+qkml-Befa zZ@$e2-d(e*=bd^t>T#h?ckf%7F0{kldp_AU(EkN$qpP>?8u-pHMh4&mdoN4?9>)gy z5I^=!!}YI251GLK0Q?V-WdeU4cpCIPozen-5%?P5+3wzRU#{;tH_kQC_hUA2Zt}9y zb6-T7*AWLD_0ZZ5UkCCt%the8Q{cDhetx{>sUEt9xy#8B*OFXfoV59HM*ff2d{3FX z)B2KJ9vf}?;z$qGPw6MQB0R=z3k+Syr8~ZZvFF0r!(0r0^J23HFM&l4{$n^afY^K|~{6HiY#y$tEw==$Hz?P)`vf5zX5`thglrE5p?F5b&= z-suw@=UzO)aqj77IUaYh5B)kZt)kRl+|U!iwGz5MeNK)Y+_^%pPaZR^`E z-n zJvsPXgFk=qnx1T&r}58CJwBYL@Xy!xcyXT0Kd`BF$^w>6a6ZLbT6o0QhKcVOB=g0TF@lamRn-Ayqy!+SV zdX696dhxO3&eE@DezxZ;p3N8kroQ>~-#)$h^v4t2y}$ZeVb8DszM$uwf0)?wzn;$T zdGD2+p7+7)i{R#Ahj9Hi;C}~R2)vhJEH|C* zd!AvO;`}tuN1HD8HRF2y^{4y(p78Y-`!*6@d%Ew-gx6l|TSIu=Y1kFubr<^@39moh zca-q@i+u}$JsU3e!M>k(dc$ed(|f{m!^OVYz(;YON%D=S`_|!lJ%cc%>`SF`Zob%;jPuhspYHRBxo_Lc!Y+X8 zSVvq8?l-^()x#bvf?cSCeg1)BlRmWDp4lNW2&42>z1sm&Jl213t^g-ks3{JNF%8O}JQ00yiQt zcN)%aH}j->TyBqxc~cRdn};WZVeN(8DV`hFSpqlvu4to~?7~J-UHkQBJNGhQgw?Vn zv}in~tT@zms3Y3Zx(%PK^O5}m^K<`-y?r(4HU=cJqFtdH2r!w>F!S^*aw7Iv8Em+J5M^ zJ-6S!Z>Npd34CZ)RqeLs+08pTjUM+tzw$XY-Cj`ykkJVE2YS2Rdq-JNCCW z?>ZRW-`Zi9)6m+{x`ba^bmRKP%~j3Y4jRon_UxyEo9WL^skNZF68#kY`XlJZp|;lO z&K-X|9f|Hdv@?1udW5?a-MR0NVzi+jt$!39*|Brm;oZ%^Mk_=V<%E1BW_V4|G)i5e(n$t?m1FMz{Y_+-BT1-WUB|@{6YF zF(^X$)_u*D%|P1^?`Um4w0~Bm(cH0p_s#>AHO;#Zw?=nFTleg1?trgr-hOaj z^R}IcUYmF9g#SCZedms5gxbx~ogIfUCD_rt?MO4f&g8;pjjr&+Xb6gI?LaxG=NY}b=PwSx}5?J(WZ;f{B>DY7d0FO%SGN^RKWq(w$=5P+T6haSH0rcZqe%+^E zW?T66^g-AE&Q0ZH4!ZtBTz9i{cKj6~sx(s+d&KpZtn21#NE({r;VT+yQSt0W%D>w6y=vB{j{&-j6)$Wi zXY3^`maV>FS;MlraQ(q(JI0bYXt>Zi+8q4tqD$=^2KZhNKO3f1r#<^xiQM6ig9rB= zDi6bY?!+2Vc-HKS>hkanbgHPimMuEGXWx$Sp(BSncJ2@FKYXYoykqCCJqLD%*RNT# za$WNci`TAOwq|wn;!W!puWne}z-Fuo&koOMjc#uXAKKqMtMZ!GFsi1u(z+X-L1=B= z%kIH-?@T|#o+)-Gea`YT3>|*ZXN!6CnbOeV4LP4jv;61X<;7S;zH7XqzH!td#;Vx6 z9(Ey%c}+ZvT~Kf(wfMp$TRxM$rr-+ps)A>*mlRyio>lO4c1FQv?6iVQ*^>$`VJ8$k zjU89;RQ8C1r?4&s7qfd6JekE5T*M3oPh#x~4zo4|7qS)w7qCqVp2(UMoX?gjcmj(k zcs#2FuCw)<$HEHEWkCgxBa%4zbgUJd2%BaF88W z(r2*r1!Bfu%o{u7E~~7 zv4TA;=3*CIZfm^5Rw&rTVoa8El2H0JUd~BZ9c};?-d)$@JR!l{D?QB9>~VQ#1=)pf zGIsnq#$Ll;@Of~pG-BI#vW>sX;Vp>}V^|PI-~#^02L=BpW08MCJ>LaBgX`a8tmSFg z!Dn#&2aKKhA!C*Q%-ECodkyVyFOMBRCO%{0N)x|i;+TnFGjXkn zFPQjV6N5RtKM@mySsZtnIBepjCI&OP{1FpJOnltLO(s5J;uaG>Y2vONk-rHz;PSDT zKwi@SCJW~Z++yK8f!i!RUf^~MPY{^&LFHY5Q>OB$|Ad1D0#knpR~8CP{Uh8I7MS`& zm|ps#{8syp7YS^&@Azbat@a&9&_e00_C=-&O#1M-{H)8wjF<0cEoDOP=QzMxDg+L4 zoWWYE1kU9+$Xej2sr)d~lEV9~x3c6^h-qJ1SSxLIJ)z7iH|7Fe{egk9JsuxMWiYuP2RXkQ6?X^+67 zeI+ciUtrNbw5MHQ(Y_E1b_gum=V!6o1QzY{vkSKiEZXO1Vb}%g&vEqE1%5|Nd;-{q z`yKhVkRu7l!50K>;`+h(F!3f6cbT}w#4nk+&BVbUiTv$WeiIv3eiO&6{3gEF%KxIs z-(}@D@gr7#6I<ZNMCE><0g)nnEl^EzSP7`CKm0(cr&qRAI6i3Mf)&b z{zc>$?JHqjCKl~0VXv83v=8I?UqyP+K8#lri}qnWnpm{2goS@9w-4z}EZP@hFPT`h z5993>kzTaV&)Q8a+UIAU+HuUAA7(KhHi}Mx`tjHFIfUW(>%w0P{L=;Ulfg9pu$?V+fAv_1{`$$*gRh{&mWoRYLSe=9q{n60H##`yF>&GvX-jZAA_#LhU`#BIz1kPr%ydzW+j!F2d$Y4(& zfdQlAUi@v#U~vO25nX2N#Z1VvxR^nY?6C}XEZVSy+L}1?U?zKZChIc14NVwZiPsXD zRDPF{)Nn$=c}6IK5Q4dLppdhL>`@~#L8{5hf$@HJ4NHr%5Gk^~so#VA-bam;OqxF> zd}uMsOEXxATHM}*Im1sgS!3RlKy<`B?0hDB{5D^L!N#!w(idg0#YbR^D1#evoL} zxl}8!@xD$WHrpFF0(`vji)UxDc&Cq<_a5~ZLjUMJ-77lYAbkcpBDbXASvdIHKV$4KLO3BN{%g;S(Bu(!|AH_S6yF3Bq(2aX)i0 z9_<6;ug=0WKF{CD8cX4D`60%$K3=IP z8YIAOBMG+(2VKPYIu=b{LImJ^2H#&lOvJa>!&z~3dlKBuAxm1 zSEBwH>gUIsq(7gdzn3%Fy(y>ds|7>$R?(BF7}3`4?6pXa3K<1Lpj?m6;tnmorJ1LGX|SWTYI7YZ#o)z2LyuipcI zHbzSy^2b01D?P?{)=V_FJ%aI#_PF8yrc(R4eIcAI_7~b1ZcCA;u&{)wer^%X`rVxY zRv^jU7(aY`+U2ETUm*?~r!yLl4$fdX3eIFOFOocnuM%-E)z8~+)=#*Yo%H!>o%H!?jSd_=x%dserx0hoN?8&>1{f0g1HWFDF z&anS6&c|a$Ru(NBCn~|uA7rx68^Hu$NT%gv+^=rPV}+Dn$h(YCFh==#{73Q380;bC z2j^>-ksWmS1AE#zR+HxhB|f?0o#QllZqVV^L&%?}$;SsBzO8|Lf+o)os{EU%$qRxi z{|X(s$*&-FhwG2*bvVcreob=HbAA;CRsKy@VB~tP`<6r16X3ItX5%rOxmN!Trw)`^sfPs?xGe{q{{4)ALz-{I4GJ-*PViRW$ z$zMkwg8xqc1pl4;E%@s6$D`g)#49vjoc4U;v&02>$44Z?BuA>1E4 zE<$)J`a|+Dl6)re^Wj6OO@AAr;pAW=A`tzH@g6to@ZkdQBH^zXOO|l^Vfy!YXQSc| z9yS(vW5{9E{}m%Mvl2MjPM=`~Bxe2TY%%0G1U9Bi?M3~aWlH-He?vW1{7vI6tHE#e z7y9`o^w|(WXN`J}UGg@@u>DQpWxec&Ywtkg( zf!nA22KI*CAEYnLbfd9OeG^$8Fb-4y1xkNNf58Z(Tz_OwCN<>R`l9gw<6w_B(m&!M zyM2Vi4eX3upBj&K8O06kO}Ty3GHF6!wU6|dZP)Li`BoXD8)mn63SZiA@KmM!T}GLo z(S*a+?{sB+b{R7oN~HdD8A}^jpVJ=df396W^=CyU+=1P`c;|Ahz6yU#s;|o^$9Qu| z>1QhSt9&GXsq&BdpKsUiqW+gKx&ik1DQyTV?JsGd6)vm4UB;}2N=07Vz|K4Ui+3(# z7_oGid>bd-PqGg=+W5OJlO||Z`SH$c*$kfD8b5OteUSdb5hUQ_hx|pApD&`B_oK_G z#Q1gE8}FR0=%>r5ZeVXY?d>wI$t3&a)L+A9DfQ3M>t9rC*Pkio>!dKVzwXW~ynTYh zjt|{pIA>zL3igBK)rwp!r6GNah-ckad}?Bk6@PL}{V$220Neh|Z=mG|tNrk=4fJ?b zV6s1jnH1k!a(Aa!@YAj@$%=Cg-i%l3Cw+%W zYTu76OP@SF$>sQ+`d1r4dec6ke>sc>yjecgUuny^KUh?4V{T7;tky~YsF9pNxNEPk zbs5(-MC|)T_h+u3F9zG=7kom%YUwN9d7Yvkn%`ZE@^J9>cNsMeRGmmq_iK(nMvKaJ zc^L0!WIkM@l{bmiSmnJW{F~Ds(LaZ8++O4`>VTvVM_(enV_&#FLW~Benf|%xBvBh% z4gX)r{pq|2E}7%`xoDW&2@aBr@*I1K`VYf=G@-mK^#8F8_FOcd$`v}(jaNDsDdYKJW4t$J<$sRbAH+_MerbF*P?JfYSU&o+`}5bXgh?!^ck#8#&&X zmEP*V6u;2@&!r|h{m;p_u+{%Ix&Pn1qW(V@Eh2Y`BZKoZZpivri2iB)Cq60q(c`sg zndwMj*(+LaGcAP4A2t3Ovj0;2Lj4b0{T25=r`W<)|J&vM-+hJsFQer_9KUP-)p${l zmna@Y$&xv`l+ID1&5>`Dc&5+kCF7F#;xbs|4xdJ$A3eSqNk8|>`gy~D ziuZSP_Vc;u9Mhdh`my6p_}fADV8DH9 z&*N99pSk1?aY*{HF{&>4&;^MdK%;>Zf)j{X8P;r$f;R;)3u;%K8TRTRpy$#}nd>LH0`*;GaC-AbUg}ZdCfhc*J}l z*`5#Jc`3I~OI7{Nv-F4k1Vh@VCRu;giv9-oSB=-6i!LE|i^I~NtdG(7Z#AB!@nhZ) z(_fb=@hjCk< z=d4%i&#UzHJc@6}B8Ss|b$w5X=dF0oulAp|8cFfo_-qSX{;Eap|4aTJBp+=&`&@K9 zxlC)aQSP@Mzh>h5t>yluOY!SM?DvuS$H!Be zcs@z~M8qdPd;RSVG5-!z6Zrg%<6xSF(f?KU^LspXI$y~5H?2sH9yjq7PC!UPUU5?yYW1~;WYF~`wfZ;V6Re?^^51CX823``WcN6$-bgL z@lN|a5I@6y2Fy48d~;_IXY2=QJbb~~uYvX}`O&^^qc)j###zq?hb)im1I_<=dDYVW zE*!L9OW)6c_4ryjKI$^Y*3x>NH9j9T(h{_Qkd}@0IyLbTTq5ThCMrs3l*ID1;{tgcyt@^vMe;{$E*?$@@ zMgdlXOb^G~lg`_dO8eh9_Hpbpaoi|mkqC?MN+Ityf@x>u^6)$x>ZOC{?>5rYWqCLD zr>Qsybz%MJBhlZ7qXmHiWVYH%&xiaGDh>y)F9ZYOq$mF>%i~6oNNc8#8cGWbqXtNgezQ7P}2 zM;IQ6(-Fjat&0JV6&_sR^Pgqm9PHCe_R#Zq7XvQHE#?^eB{&X@pK4;ve<07uFLC7O z4lyr{5By<#xYXx^(4WpHbwAo*J}digupdD2C&E66^&6Yd>-~Sj`$zMM(eODSKVK94 zia7TN&$Dag_yhj9b{_T5oIhg!Bj-=R=6kB(m(BMaxG#s_-NrbWdj}KW8O;ue?|sDg zZvxN({5J7Dy@9>w@SF4fZU=W88G+X%equbN%KUrO5PVO+4BwF-hS(WDaRUq85#9ik zkLl@Z`6|D4{_5iu_Cw<{J>8M{T7<{{a{ZH)@!D;KG`@~e6^ z8n57c4sJLOoB!ivdjLKcg-UXW{}$hibITMwb$q#r5lzzmnc=Jt3;t*D=dU*Z_xb+^Pd-}ygI~(?=nUogHJ)eJ((@fle|SD$8L^JIF!YGa#UHvfa1Z$X>?HAehwACsQfQ3R{5{TFH<%CtMLb(=Qq&uhBbb{|M4>4fhX|&m!>^{Kd&q( z887p{DqNx9>Y{QJ3;q{REpz1H|D3DKf9ZKI_y5ve|gsC|6lqay(0ccl>Jp* z#zN2={x9Tsq0Rq%sMq0pw=r>|j7P=)j~u>p z{&zVT{Ef(bM?APt##e*?a(RRO|0Iq7sy{|NqVaz^v?u%jDl7-c{MY$O&x>Wb&Ywz@ zXC6HMpQXkB)tGO~`E`C4Yy4OF6*mUk7uEmI!F*dT{~C?|Q#Jmp{2a=@sQfR00hG&E z?O(T1qVc~}sh{=>wX6K+kMkw}KQYYJXV3quipuh3{#O@QDtOM+3KI+d7nGDca`6A! zKREwO&_SF3-}L`C-k;I>|9a*5bCSuVRenuV_&l=zpQ)se8*{n# z`9I)ew^5<-|2l|d{ujf+%Hvz*U$;@I@n4M()cqqg{~x3AzocB{zse89?;8JA`x7_9 zN`Kz&iY&RNExsM~z0#_ePulOE_L)^FJJx`QB|znk3^<@gEC>PW`?H_K3{?qDGnT z@CQp|JTm`_u%IsUZ?eK?;=f$pVE;c|Ngp@naqaVeIRDEu{?pTMng1cESFV4IHoh_Z z6}dY8b$b&xuG9EmT%z({w=WeM|JC(}xG_iRPu#dh{Q6jw$Dfm_=^L1qY7i;tHDGHxQ_Wv`K^l^h` zN9MuXQ?BuU8H5V|G5m((e}%?>ou8_`i5v4Z{!cAa`9GMSs$A!%8qbh@QRDxTGL_$h z`KiiPehy_{RR3RArt*7me4)zK_=5OR%4_woLXmj4T+_g(0D8`j(K#N6R~aV^a-9XzF$_B&YkQDXt; z`vRN)=6ajWf8v+J_ikef7Qh`mGXKXkFt5h{<4*e!|1XgF-ffgC{`yg43~%3<%dY>A z6#rL@QOox!&-deo9{(@r+UNg>_#fe=%NRYSJ^)3{<9hck>#p?iyNxFi5n|4{+Cy({#~6v zbQ^Q#s&aMytj9BCUsV2ARI2`6jZd&2ZLTW6R+-=E^CgO3RQ^}O0LuMW3;enW)YkZ^WadN`}wF*qVRF> z^SN?uKVJo+6H`N^ANb!$gozrszGi6q^GZr&zG6Ho{u}&P{O`lj@^bPAmc7ILRhggT zc~Q3Th+_Q2jaf?nX#b7e9<2Y>%KeQSs(pwXt2Dk=;meO5{g8c;%Nxvh**|0d81yaM z7qSO3UmrHAu^uS*Z%(bs&yo4B^0T&9=HJ7{T&xE=<;M+`@6g{^@x;Q8x63N#Is50p zzw)Y;3a*~B+{EYkew(?qOC34*=DjT6FdnXGzYX7iD}BF%#V5bar#Amx=%3ghKJxxL z#lLYrN&DM4pURc}YuNAKzydO#z^}(;KB@Cf#JkG)ebkuF`83`3cPhW&&y@Xfaic~V zzgVxGuKK%q8h^?(f2NQ3YJ>~U_{IIsK?8A!`wf4i@SoO`)bTwMKPswJeyIEhzvroP zb^NK%n}{zn&^E^(59XK3AJ~H`l`n(*(L7bI`lF%zsmiCSDw$6@|GSNO^HlkKMStY) zHmUw@9K~;z{=u)ZIoAF)v7fb~cBQj_4f4vl%S|lyw^q$t>c~H_e;c&leF(nM`wRB- ztmFRA^7fD3e_^>;@?V4artBBvd`r{#_T*LNo3bBmD88xw=@auU_fx*H2p=*$J{UYd zzS?}#_Lt=i=9{xW4EzG$6b!zZ*yfuf|AX)?SLWNReuRKHt{UHz{aKuE85-Zdc~$wQ z?Ee~yZ&%sBjsKKye^|b?ss3%kV7@u~u>{`~48EDz=9?q`gYYdZ^DTI_`KIiL;(QBg zd^>$r`KIi*8j5dM*}vs~%C|o(-?aTx69@Cn+5aT?reN^R#5Ug?`5%ODB{JWZ`VU`4 z|AzgL4J;({3+o^M=nQ`avmC#wET=bOGBsE$vpA835i_XFf> z&kJ?FVR=AlpT0i0h-;hw!{*k- z9@=r%`|y4a!by2N>3kdM`TJl#1eN#Y!`l8^_5GTW_umfYgUYv0`urV_C#-|pi)*## zZ=dw}d&!{vHyYn2YtP^FD;GKYX~4H|)jS24Rjd1N%IDNN@;?aQA~N4D`1f%AT`j&% zZeY0@-=3BER;KNDsnqtnOy=?AoIJdL)8k2X{9`<8&j-}!QQbz7=Ks|9Sw@a0 z^R3ZpzVAL!^KT5WXq~0pyvDZ|WWE(>`%Ch* z{Usy4Upi7eInw*3`uk;K^}`yUgWoSxpJ#zjAdKwa)OZej)Z$5%k8wkc7OOsb9;N!X z+I%&hRO78~W2E=XM6~6UKTG4=9PRrhRKC*tUuwLm?w=Xz{WW#{KYxVpmnhTtrpC+o z{^>#cOEmvBEzCP-Kkvr-qt1R2@T<1CO2KnVDwX{u`DGQ39DJL5)#h(x-{jv5y4?Gv zO!jx1{X2PoM(^*2+fTyxo8l{|t^F4K{jQf}{)M#tANk7ukCFCw5B|Q8smlHp>Obid zhxvU@!@utVK80jHVgKn=89y8?4v~kl`V%+w_sK%qesJ}D^P$G89Ixu@EA;(Ga{VLo zwLtTK>id*&V|YG3Y?KtcrT)SGX#7*-TlIN6t)I-${GIy!4G$Y-#p?bM)gICQ8z>xy zJzvwmzo|OE!r31Jeiau~DYztDX=1S+Syoi;$iXk~W%&iZT={;IA->N|*-vuZ-^%+l zdOi*Je4F#9K<1OTf&Gijry^y)3GB17A4K1O^{C`Sv@lM-IU^Wk1QQ{w=)ySCMaF zjc>2Yd@E4)n+*PbB6YogFyBUsAAQ<>MxAe|%KD}L{eIo}JzkYB`u7tRg;lv~kHJ@s zZ|eIC`uCyG_Xnwbn;KU6G?>q-d}Kar>m%Ct6X|@;(fBq_vln7@f)5!!AJo5}Y^3ie z%Gda&+6&@ai@F|}tHqCb0baQsKNf}5?Bl|b{zCzhPk5l+V@Ba?(-%^zH^!=mu!TpEma$@!dmk>`FNeI@hzOI@=3M--A0<`-_o`6r`p{S-#4g!zbE2} zT-AT+?;D3SzKv`zRR5NntNO1zrG5B*S&eV1z34JB27Mn8eZR8g-zNAa3_j%t3Y`5R zz~Qta1s8{=npp5HH(UDtAn+~YllV8~_mPMAJztccWS4)L?>EMPVeQ+*_Z#cK2V7%# z+%f#z5u@L8#_zwU8QA$x>06MV#?Kx2{m~sR`h9o)`}AKiyxQ*v!<_7ejDG(ezbBMq zFi-4W%zto+exFGB{$$KAA!otcOgZ*j=NT-)zY&>!pB^{_Iq%OIvgC~X;?;-f z_omabqT=g96MY<`ydr%Td)Ls1JL;pK=cM1m;^pO_Ji9+gpBClK#rH|0;Yd&Y zOF~i~)el^Q`bgWRzNtPB-5-v5dy<*m9+Vfq16A)sUTVbD47A6e9L7hpJ>D84fbV-Y z?;q0FbfbL8t@cxUC~JFDJ8w_O@OUY;)&G!@!q&oKxI4hcz1cSvAFIoEb zD*8tLq>rrfsQx(GgY>+9uFoXuf6VGH+CyImF2*CqM@^@P3>pr+zs<)-Qj)Dte4l+y z)W_?$^1Gu+jHf64L!ZTl*Au}GJxFi$U>-W=?ndC(-^e03C5f1S_ zI_{U7&zCvB3cVpC#ZBXf>l^)pK2j((=O_4w_NAD=---5QvgBw=iqI#z&HHNz|G>*z zgYs?z#sHx4%;kskxc}htOvpC_%kqx#lAPTd9b-`WxB2j^vOIruf-Daihmc;*zct&D zKScS*U_VV{v)9VsGS-n3-?pGUIsZ1{Uq$jNRz3S^qPcyuKYszmw8~Zy{q3 z`fJL8sk~i~@3!T8(O*^G2KgRaz90Qn%fAKky|(-_=&zc8ALRRO`K{=$Dn9`EXKeW$ z=&xG-LC9~l<@mvIOK!Hm9r8O!j`3Jyc-=9KPm}+!cS#h!NZ99&DLC29h;a7!x_u1x zGe;-AL8F15jHY!pRz1;9kOuylGT}(B+HBOeUXr`bg~y` zGlN;38Cd^v@`LYwRi2%!%G1(hdB{j)Ch&#z`2BB_e;FCFye2v(Ma;K(eXtMN*^(Uo zB|U)wM&EBM{7W>``~?G%z9u>&1p^2NF!~!Z#-Zb+g!+BKEtjL_WG_P>qEAE z{GbD_MY8FqafH)Po5a~#RA%j0-nhMK>&T+LryuI8^S zSM!(TtLcmL)%4+fHGOWHnm#*CO&?5C)0dY!?IV5V=c_m^&BoifKau=vCZqSyj+4UZ zeNOOcDaPl={C4z-I5&kCCY!{ z#FQgG+nGw?QwNO~*q?B;txZbrFE&yGG@w#R4*L?0?%5;D!_ixAQSsirDn4*P#h>|% ziVq%C@vXP2xV>G)cif@kBS$2R{-mac(O!Okcz9^kj zf6R{9FxjsN z4G`0Q+4Iw;er~_=@cS|}zjKrSdt+u$onO%W%fshiT>dKAw|5!olbQAn_h*w7Z}||f z0z2cs*f?;bBu6}b5LlLPx=E5F{@NTMd%4Aqr+3~g$&vp?q?hyGv|f@^{>>CW9W>*g zVq;sAEDssGP@bHB$68rV`O$&MjduRFm9ji!+>H9<{OdPJaxdlIiTu0l{I{%@3 z$IDv>xmw;1$es9m5A-jWw{n$K9`6q-jI6W!y9#o(JlGQ{Uf+xRA(yv$jZ_}*FDi_z zvfH;0a=E-UTkZHB{mnC45=2hHAJXrdHK-u6kMfg0H@2>^<6&ULt6N39dN=QXp0Sa~ zpIhzqn~fXo_!8-BjIA4O|MEtrx8|{}T<-XrjV(UZZ}}U<8*4p&OzY*_j*r4M52UL9Epo%}UPQ?eCRJ`sM z6*p~D@hvm}r24n5mGOe;>J8wN?N7qdwYw!b`n%pu{d4?TARx)npDkNdynCgJ+pm`} z{Kv`z`<42x-yz9S-p-vW-nC1`H{77&)>ako*sJ2`Di!ag0gofX??=d3zrG#F8eie) zrcJUO?cEZ9e5L6R!_gaWl;wH&eIwhy;QoiBH{B%5!_k{>R`KS|DsFC8@%HU1ZfjHV zo;?zVK3DC6J+|*(v9ZC!K-l$mJh7$~_D#q!e#w7tvoQFAc%%KGf_F!EEBF>35Ab*e z=|hNDsJ~oaJpZa2;ZN8q(ho0x_06#FY_%B=EQqdYrucY`jjvw|1IxDAxQPt3h$o=G zdylYfE8aHVJ%Hoei2`6*m?!= z1^<2mirZuYh`-^&(1-@f=xZ`;!L9&GfjR zw?>;*qe9`oa6gKT)r+P4=+8P}x%|aTBst}8LIWZVc6*jUE|XhpEqBY-wJz# z21M%Z_Roi0DSs2iFFc-uz7U^c{K(}kSg4nW3M2FF@)km_mIuCy^-_$7JY%co|8Lra z3L*>5{^nu-PZGv2_WzjAPnx&N{yJ>je3Km?ApS?Z){K<&dwgbml(<=q7dG*D;byzM zmR3$j^4Dlj$k?=r_`>ZEj~{N{MCI{#fyWOwbGa2iAYQnM(zBPWc;TjIyF9dKsj=zi zk4etNdPlKwv)2#A=UdNW{C8|6^CtX1>WBR17%p&si2Q!UkGySd3e9+6`EnIsf4z#EnpE7_sN%?46}Qx>cUt$ zcmU+IcjZb|zG{_<0|6B;Tc+aGt5v*ior+hiP;o$kpM#p~y( zc;kE(-*AJ9Z@f{(n>VX?%N7;4E>Q8-g%XB8n}0p*tK^RtHo`s$%=y;n#1p_szYO-8 z%{ToY{BNV`f9I`%{kQWkh%Q(M`zn=RUl02x$(vi66}*0<9slwAwr*`z~eE6L$Myz|I@ z%(wl2J)SK%{yK#3FI4j{u9oGCbi!ShZnid&c9}sBPV^#JlryhOD^i)0C~ABr}rII`9{cR*yYbe|5f>okXMi#{HQUSx7hv`>u-oRw%Gm@ z^Wj^g)!v(tex_Ytv*K^VMxfe`2f$yf7X(!Q@8$kKV6PW=#eVBE@c-Q32fTLu+}{Vd z-17HUJd#BD(O&rfYVM!O|8svI2z*R*w*7sz=I@ip+)Dnwx;lmnj=x`E?w_#yebSa7 z+#S0gzIC8Cu=gm{`{P`-buUGM+>r`A}HiYqHsyl$O}XI-!2swx$4+^FK! zGgQ1`gNj#{t9bT}DsHM!@y$1@c)?5w!+(`WV4sEm<^HpV`%k~^4`pclhrlVAI~6eF*XWj8aEV z`*BKvEjh*iGs_$~*;iqIl5GDyYq~5C@$Zk8^Up4p<;Wj^!H$%g_O;lknIg+W#&nb? z=f7sEET{Z1*pV_j|FzR(dB_mouWzky6&u%uWjW=?{fkVu^Ut5;$k)I=i|^xi@-HlM zlEqe+L2x&?3_)KgMIo{QV(bAN)0yj{qtXCOL4(p!ZwM@=Q4<)PE+kYcH*1uNSa?F?E z&&ySRUc&u(x$VzO#P`eC{=B@zu8;fkaxS<0Is7+&US`eT@jPy>JwNCEJW~EKZ?83f z4+LVs++XwgyDtJL`SW=z$UpP>3tzuqu*$~hk3Z_;-_MOC=KB4N8MK}O6`B4!P^w}t z_7lr;u^-Y&KWmmMFPX05nKM;Pzh@$sS5u?n@+m5wJzK@}evh30+G|x@F-^tST%+Qt zQ&l{FzKSa+srb6t(+iJM z{>DZX*U}H>O7dmPRO~C1F#Hv_&yv5KSVI2J@xKM-uwN1`tbjd{a8(ufE#b;a*b52I znM3}Smk0Z^0OJpQl;rt*e_6ovr^QB+7C%fZgZ*a{ZU0+PO#ZaMPLKJc8~dB>_@Qtr z>?JF-(@!dbePUIFdHvxq>>aB#^TYqn%eVaT*%Y*|s2cW&&9U=Ou7!PJKElF(F9=)d z`%=h%&yCprJLPKl?_9~glm7-b?H%{u6Z0H-VN%HaJ$c7}6pnY~WZ!*!{^0oUNfTsw z$Pm9z?c|@FEz6PL2Zt5OGwo-wF=ebQ51GGb?c|@9Bg-j240dF^oxgOPEDvEnpVL2z zKc>T6IqL(Ie-iRfu=CHzlI7UXhxStVp(-s_q@&WOgY70S+;x(`mf3Gthrk4*L;Fx698&|5Z8o8p$y8=NVJE|4z33x6eoMOD4&=|DJ!H<4@OMyd+Px z{U!PvHgbKo|9*zXLvnHq{I%w{@ZY(@e+zxl^G)19mTUX(0QbjyJoEW@ZosJz&+p^8 z!e5*F#o)h%KQ;FUV*T+w;DUI51tG2;V0{z*$y<$8RRks-<9&-{MaZ^{1#0|BPfiH82;YEmcK&;%wtCeyT?eFAcp%l6TerJoS^FLyd_eEik(mGkj65#fx3#rtf=dk_@hy!N--wN5=N4!j2z_xJIkN3Gb zpRNAu-yfmiXA7CTGfm{jRE6$W$Vf|%p|@#NKi}V!p1u&ZS@A~WeOSKd2cB=>e&kvA z=U!Yy{q9aTYalt5hxWQVGa2WHS$>X@nXw)h;FL`JiT6G6k-y)7_7tQ4mB4PYx7;73 zr=OwQMfM*!$H?IM;S_m(cQjp;MFm2iHAY$n*<)+I_fMZ?8Kj>?8g`sw`~ma*@Eo6Y zvBo2vxYcx^Kj$$%4vXgU`;Y!SbOg6j9B%55J|5hi8A^X2FxV3EU;KW7U$}n>asS@s z_YeC?8TF=p+eLalMDIV6`vHEBKQFTSpF#ac{%J-=+Sg8=EK3kZ{rLuYUp$Pu@PUWq zcOS`3XQUuR5j0-&j7&cMGiW@4zjq(WhrCkA!I!LPW(KJ$@tkFgy_{#6>k zS((&8vp&54G70aCnD6^Rzu51ZhJc9rNA-K}J`zNJv;L{cTuusW54B@_;(o-DAK1Dd z)L*V2dOsBX@kevL2)OA;qW3rb(Y&NQ8wxmHAtV42PTVy_b<j#0tu=wL4#J*XT}c#`=derdWV@F`Lj~!ewo%_ko$QH!Y%q=Xb5VZcxA{IJ z^yy8Q?Kkxk49fTKo@L0&TFm?y&&W^y&*I~xXja11mzh5)N!%|aLVxeMmid$MjY6V+ zyy}Vk)E~Iybjod1yw%Yex4ru&oOe6NzVH~?T7xdS(n*<7v#AilINmK zvL`u4-WbAMXOzB_`E%GAkskZssQ*58!Nf_7^qI@TF0K!(uitYc^XIW9fib_=-^ae| zNHTx_g56k{#@dqceo-jk52J35Q-H?={1FS2JUifTv2Ygf*nmG~VM?D9@E^DEWZ-cD z{}~HYeqX@L6px-(f}7}gY{xY)(-FNxc8Zm0Y2 z*lT*R+{E0T=XBVZkH;J+k=}2*aVc`sA^gjNXeKJK_)+MJ-;tVNQPe<#zUXb(Z-0X7 zOEcfEdHgomt8e21z3;v)1MPRi9#i{;f5mV1y-nZ`>~H8*b^eF` zCsX-}Gxw7}abrBClJ5}Dj|x%0n-3lSejmz9?!f!@XOMyH1@xno4_t$ImrN-${dHDn zz)$^81Ruy`&(6gAj;Vg)cS?3wc|q4IclAF+!i> zeE{$w-R>XsL+cA97yNlh8oy|!qEE4ZFV9Hj{0R_$(4Nog7p`;nPK{=?x9>ZkF7`t0_b_Xh#nmE4~R_Wjw`z~8@A_;9VH@AL2v(Ftzb z|C~Lj@!_vVV}B;t?dAHuoc*!-Hv!ctd>~E3{^0%z{SIk=CQ$sU@?r4(L;aW6|2>-i zA4=hTI5EL#zp3vDV@ms3!ckW`DLlIJ~~ka(nOacU1ZwWPkYk z(nITerfu(sus^?pz8{k8y`}G=>`%(3Dqg~WZY=<&G&EpUhGMd(|%LmN%%qY(b^O8pML1CJqPn;*u(qo z#PXdLe?Hcvj(7S!*A!jC(|oWD1LO}NuT(>jNsj{=WAeJlD#zElCt4XF@K`=*zFhk_ocAoqt?Gz zA6$eI(F@_9`S&C7^^F%Z*^Ec1G3J5wu2ty0I7B=jH@qxjT|)l5v0h}a2X4z`vpSN* z{F;>XMl(AW^)2D?@tFtt_kSqsaW7GRPr^r&IdeUce?OHKe?FV(w!Ytqkv?htFv{si z>z%;RN7N(U=coAdIp`PP$3*k*_O)WY#cg=_`U_t)gTC_2?~mg7(ci|D%}7m$=pS!S zE+DlB>q}HVNjYE5@(J_uF+LhoA_Q@W{8+D~>Tw_nhkouc=L8@{X8dx>iKx zi}kj|Y%#z2Cw%{s$bqlf!1#FP5V6HP_GPec@%>p0=`-02wBBg0hhpi3^z+Oi{&grK z!&frc>qin8c>KWnqzmb3=@tj2$NCZ0>$pPO!?fN?>!WVIKEUO#u3-;gw7GGAd@L5A z^+duR=!avzp2%n7C{Xm@ZMfN!&-3ybgMJ&1d?e~)XRzM*kgM^C)gLGQYr~{h#tZbh z3jR|ImP-mU;i93KYc+f<%j;%_0K!8e)=CH=cg~6 zylnn7wsNHU+lI?OZ{+;#!{y&Ma(-jD{C_!e{@8H&|7GO-_YRjojWvy=zpml(H;kPB zk>T=pj-1~be?yL+e;hggu;cH-$oYpIe_^(MB>fFL{+ceIAND8SImGt^GgF_ir;Ud% zD9@J~kDR`IdEk4zGgW+n8(Cz;Gx5#1aPIxg3K&etJ+AAVTAKEsSFEl=U6 z&i}hv=arVH@KG-hZ@*t@c?$pZ^0L|8S6ZIJH@&?7xbpH8e(B|Xh249l?Nj)qm$#2S zaHZub{L#y+W?ffWp28QsypOKDJcS>6dC#yvyIlLYe;9>*RrsR&p9k0@mn;8r{&HBp ztYm+ArR5FFmuz;NPc4@%$O>_e1^cLiOi3!{}x`vZuO(MaOZ1@@6msEe&)8BKTiDZuU<^)>;KI= zU;EW}d>sP=@Bi!JEuQau^wEdszw?A^%jZ7+?BOl1#3~jhzWbdg_C_B5>pPpbyi#-g z`5!;^^(THc{-t01^5+|0Ih<}~|7QP(pa1nC_l&2SH#ff$dH$Voue?(8QvZkl*KZG} zfAgcyede=UJTpW&&x`E+ywY!qx=hi-iHCmnVoIB6$4#R8FY#K#TYmf0nt^qR5BWUm z&Wm@>9-vR3+we*v@wMW2_|-(>H;KfDm)?0Q{H)mCpZIHjJ-bV+{w3c3CE=&Mk@xuZ z#OFSyv#au=mU!ypxnFVahdQx`OH5U`ix*gaenik0TfNyj@ z>ULd>yTmV{;K`Ipg}RIaDUHV?rx6bcQI3-4s7p>0qd6-MN<)7?ak(bUK`Ykb+EI7R zeI{|e%ROPM?*mpx4a*9!xb(0OSFlVsoV;lSIV}qemw(d;a#{u$E`Q4ia+>xJm%nWU zd3XeQ`v~%q5#+`Q^2!n9u@U68BgpR^K^_@F-Zg@J=?L;iMvxyLL4INc`IB7!Sk(tC z@)a7`-%EfI0d~fu1taVzPsuo+tEU6Jl{*4 z!A=Pm%A*uwhxd?jvut`uU)OkRfS6y`z~)I7!$r@)UCYu?XM~F4hpUl+tAA%frnd<# zZzP*jXLHuocvFtBQ78@hbWAlncF4o+kSoLA@PTP^+-|o^#&He1$7u2t6}vSY_iJ(u z$CEX=hT}d>uHkr+Cf9J>tI0JyNUrDCuv@QR!*0F(8g}db*Rj?qmbi9~X38#Mb^)F`#4SxOPvF`3VOaU=gwYiUh>p$>XPffcGJ;r}SBoEEdl~ zp|oRD9E#i%@vyij9+c%u`7tX0j8(p0FW-x*X;8@d<@%E%Ni3c;T=@YipYOLv^MZQ$ zzKD;}O}bq9$&q9hPadXxln(t^<#Ycgx6dE(=qqHL*{Ip0Ki>F<_{PajVi>D7)zM?nsyLA)J2b4C7GA>_-xOc1u>5V4Sr3)N%W`63 z-^lhtZS!UAMclgc*tRJ5Cm6v`4+rBuSw7+NZQVrU#%W*jDB9xqkWnn)YtA?Q8#p z?IU}s+&{}cUaoznT^w!uj8E9U%i7h;wNJD^KHB!hK4JSVYnQK5`|kaO?IXLc++WM4 zU#0eSeZuxd^!8ak;VQN7kx{gd`#m-0Qf%N-y^WSnxk_zp8buq`m@}rdkLm5RV$Q47 zzD;WTM&sj@_WAYp*)glR@8B{%#OIde!`SE+rF|anRr4Wo{j!Y~KE&;n<-@eE?aJGy z#2dM~{;Zf;TKZBrF6I<&uOxSEW0?}{-p?mKeN*ihX-o~JfA?5Kj z-dld}hs^Jf`>nC#$Q5m*j;sx)tRxWYe?Mcx?eqpw`rxsFX(jIs-P@t$?sVQ;Mq^`0 zc}hHmv_WIbk;~eMrH`x)b)FRdy=epYmiLzKeew|6aHqSeHw+W*A?284^WOe{)m>|B z8&`I|q!=$P%L=1-6DQ7t_BsjkQuy$LDv~To$s(m#GA+p>MWoB+8b(yI4lM z8+eliB6PQ{w@m^8n+05;1(c#d_eY9w|1{fTff_{$WQ+dLkN#LJ+CoJO)IS2G=iGbl z%$?y-q@#F|TtOUu=XK9LbMKuqcZM?%Eo=VQ?7gTD(Ng$oG(!D%8V%1~iw4XKeYo%X zoBivS(db?0d`Ht^$%mztgV_=B=J@$bAdZvgFH=f_=+UpGFGyEuFu&XvFSXH0(6) z?+@L-u6wIQgIWX8vgE^3X+MrM>|2)lee0Z$eM{%7(P&s^Un0*{*T*2^P}ZZ{=CPEw zYB?-sSVYrv)1HHHu5qA?9JXJTQ*&57LhHR^bYXw6%-9a_nU_I?t@#JV%!>WdN`Jjy zi?G61BgiZxXv|-Y(QBK(s#(7hIT=C2R?p%wZ|&u+;JJlmE2y7fKeOUPZiD@#f&I*i z54p{~vY+XEH44`KXtyY|&0D{SDA@H3O6e zd?gB&d?gABe{WG}o2PyeQLtNOWOC-jdBKXWM8T4;L_y*2EedV(Q#A^^eQ!CRh3nKj zuf%b0*-GX3V#SwH!sCk-pQ6Nb7oD$0f%SII(v7oY+q`tQma9CAD2QVK<`jB{q|6JH zLAjbPOFm`aZSz@*k}03vTNK*nqiRcc`)&~hk$FKY*R7&pcZw*r%R?#iYMY13D8W*+ zVt;Q@Xq$)b)^e5C5`}fA7!e)vM+S?Pc0FTK`ieM)G``lf=F1TQXL5@E-lEVp|5RJL z+xM0z*p;;it5(_}dB%L8ui2;MjY<>_inDV#ySL=SIv1Y3H*hUPx35M4x>4s2_@g`g*m3kEH z`+1ajAJqAVDDeAQcV9U+uD^GcbKY4LXvMhQ8mST`yJa>8!IS&iif?!7eNE@@Eec)b zoOcuj81HsyMFVCs$J@ zh7g>qMfxK9?8o}|-DXp8@ukNRf-a&K7eM9SN&65?1TEKr`a%x06Nas@6#)}AX13)8 zTT%{X#s7%5MV|Vv%Sc(hp2)y|JlXai?d--UUh8!h>0LM4yUF~%E#F5sGk7MHywwo!CaibakEmBU8-^>ZH-X=B z1v5Xj&z7Huy5&=}U_9`vocbM4`Y$PL41YNh9y-*^BTd^#(6JBy@wW=RcJ3W`Yp5MP zw!7?y#1$Bu$95Z+5#{w5*?w9tEY2J+Scsk)XCVXCPiFXkB4-NF+W=}u`J#g&C8-zWgvZmOC z`{EAt3GR;PF~JJB{^CbJD!m34;i&a2 z-t&X|z;+wkY{Gk1*iy^7kolajf7)QWUkp(|@79#(KkJuFkm*)jxn7mzEs;5z^Q(?yGof z6+Aha?8*A$gmH5>>f0v!t|`&p5Mi-#yx?T`i0kjdzvDjb*`@ruJO@C!d0rJ zVAxd0X$N~9w<$M!f$UZUP=h-a-evbXhfS<)%85MiauwNMP z(`?ItJ?y3dKgBi;_(^udfS+I+27H`dH{fsJ^P!dck1=?#n}UzBngP4nq5(h7G6wt@ zOB(Q_EN;M$u$TcKVIc#4n0Wz*wfH{7+y;D@xeOTIab>^{upPUq|8=%)z=zl^1HPaA z3b4Mv2iPwR{QKBV19rhP-BtN}*@l7N5AT>%>p#e92HeLo27G|U4Y-%V*^%PUe!RU} z*&j{@XOaqrcYqn>_c1qMJ%0BvmjT0Z(|~($ePPFDm;E18Kow)&sA4`yQ2v~Q;aQC> z3_Ad(3LE@zwdcrZ%zn0q^`v`j4xfu{Jp^llFM$R4--3U~(;;nkl(W{IZXKQw|AFHM z+&+E_7F$9v0o;Om;!i?dPup1YSsU9Nv@y32wgK3l0$LDw@b=&x_}_jR&ilt3%rR8k z*MP?IhaSQl&TQ~HPQr1*O~QDLK>jx2Bw^@zTpy1S$cLWFafWb+Fdie2pCPUv%*9c=;j{bWIy9qZ4W0HjYAmI$*>x3JGLxeX8ZxC)0juGA_e1kA9 z)}Z}O!fwJh35N)85sn`f@^QeuHt0Ry5I9M=c3j{L;q4~`UL+iUO5hsdCM=&|{TL6l zmw85DY(KaEoWR&##GB6xjO{}lf_WeHu{~VgC$RLd0cSDDm;UVp1eX3~;03tIm;P;? z78v8P&*o&_pumjVce0(A1a@-V%iJRZyEyJ=@o|9i-U-5d|FtyzIRgty@p z3CQ;ncI5>Q5)PFF4iT7XAT$Lhyxuz$YyH1OBAo z3;%#mSojD0bAm7Y^RR8g!oLBQfrK36CH#Z_Mp*dgWSR4VFZ^?|p5TMb@rWCKE(8lZ zr(j&ee6jNZ#QX0U3&DQ_k5O=Za=uCKUzAp|WYM3g+_dTOEWppkIR)Jp#& z(z~Gl+>q;-_3>T3uxnb1+a*Ygq<}K_!oPG!MlJ~fHQPkZ>$EFiYBUrRWZ3V>MS~l+ zXxIOxK^x`-6UP3D`XQ4W-^PYPaq~~{j!g93tUo6BH4@V{Z6tY7NM{eKyhchg@Z1E`O3W__GTFa$8X45jfn!QY52>f^lL5K#O4EhwYS zVU0pxKBX;IN`=qQ3Pv){SE!Hn4TbbG2BN8x0q;GqhyF)Qa*XuXl<{Y^|7tLAJs|cU zjW}Vb(SueQ`>&S;eh4<$euEI;Z$jXNq>;4*ITaRt?TkYyD*$}Xfd2O^>Tg1hg+=y% ztRIUhe5^?NA(LyLlD_P}sE<>ES^osY=M9VcKZcwHi<^Ja`>^4=)R+Be9Q4;gKV+)^ zHtBCD`dGizll}Jy5UljCL(YRm_FwcLrzErfNg%+{2o{Ts@t1*jgN|B+Rh=>o`aiH} zzngZP+<(wb`TjT8kK+bovXd}d7Inz{L2ji!N`GEG=(0>3pQ0N5^EdRBwrYLos~i`) zg7I61cCf0)Npd_J#hDymax3)~!qcS-Yq`oa=uMps z&m}Hgs7*)9qpw_;Po1w`3@^^7qO+B`FcXiScS= z?8-5?xXyepSd3dopl30jmBa!Nvz-%cSo}P#V7Uv88p{uFbxeorM zyxD}WoL}lBS6eM4eDj6q&@#5$(!a4QNsLo2I(jXC^<-%-IayBnr>^CO)ws=$=a$Z2 zo51}TBVQAvSCfH)waw_lm5X7n*3CEDtppm@gSlH>Exe^#^5XRKm2 zu|$3>O?vZ{*~Oe?Y=|aqoju)%8YyCyk z->Kh?W4n`q)a5zI17+()^rzu3XQ0*?>Gz1{wo<3iNE`#zU-$MTh-E2-I` zi}_CNUK+ogkIr`7Kaqftn_HUh(4N%Q9L#6QK(vEgndBmubN;IxWNRcld2ueWR&En_ zEl$*Sh^wztOwZ(FFgDusw{voQCL=J`%4==vmE&v_ZDkXy)@>PCr1>v(C0`y|F2Fo( zIZvA9+RaNwxs{y1&HSawEn>ZEg~aG8&i__pL2K_4v^NmB+^+9vasiaXev#b53uk3in!O+lX@^-f%)`~&8t#gxb?E=KTp zH|s4V$5)g7(bZf9jw?wxX87S)F*1H_u5{uhnXaWP<J(3TU)tOx^}Hx)bO;xD?WcPmky*0^{cgFo(F;glFNceg_SZeN=pkD zN=x-%x?ahq=c~(=Y+bW6lC5V)IVmzXa|XQ2RoBvmQUwjC@n4fPd64!V`vdp4zVjVu zi*L}!TO`jrZP%vYev8L!kMupxSNY_56bc>ZlHvm{K{54Cj~jRbgJq`g(a?|WKXLr= zzCUsPY2Um3?+D8%U*H1b<3}Cj@w^l+vuMR7Zyv1mJ(|(%_ulCHY5$+O{(Ru=L+?KD z&f$Av58pHP2-iX1KGw8h{aBtpz1qKRYo`zLsNE-J;rocRG1Bk)vHe{K@}E=qT1*r^ zEZ1wrBfk+J{VJ<`2MJ_$X%~o~-g}=zkF2+BnqbSp{+q9@o0)LunwdoVzLkzH*_3tV zjXB}uhtMvYvg#wx+t6?iKQlE}ah;o%!7%cy)~$s+o~gDLxC+)n6!Pup$|4SF3VB?S zF_t$04k+=F?FtfIjw9JG{sr14X{x1HfsS@UM3m#Fzm0qe!fIvb!8UZbEx}TLx()pk zprb$AP(KzqJ|Au4&%eTg2+HI778W@^Uv48`Kp8G+{6Lx&eFyBI{RT1R_=U9;5jR|{ zfg*MN@EzsjL?7W?#%9URCeh`7`DbdcT!)t9^kZrlqxD&jLm2uU+3~`SIann9O`zW= zSGa-xL+UacU?1rlgn!4i82b>mNN>=(@81*sKZzcrd5)^p+IK(LlYI?U4w2s1O>{5O z;gYbXi=Ek%1xqUz2qvV$n|=xSEbKD9I&8$=^J9~=UB{k7K#3t z$xcnNqh1BtQ0TCps+F4xU6uc!4c!U*7Yq8Sw)X_l&3*=nZf@^3?I3e|e@k?;o&P1e zxxJq%c2xf#p>c)w)%FGzx_WGmx1q~UgLWPkP3NCM>P$#KqR?+DbX9&^p{w#i>fdJj zHHEIq|5~A|_8ruj%>Fouj`37yZ#U85wkQKVMiYWL{xd{3$3IDQtQU{%vIJ?6;5fpy zVOd@!I@+g@a8`!s=JD7dI@+hIxcrtvr%P#^{_hH%rW#KFlsW<0QT=(2f@`MBwm3i@ zhk~pdk5gDAJ;V{*>p@#d-=^_%!$g?;*OyDV)p`-B zLLL+i)IxToc)64>8t5Xj#l`gea<)?J!o_*0uMR$@mr(xf+1ZH^5!#7rVYLj^cX>w1 z_e!~%Eu`z!bb&i28U*eMKLr$RQCkS-PP%}0u5@N8Uo8}8O66kV&V69V3tcolyS%WH zUV)Q7Hn>t>u4i+wy;|YhewLv6rEV?y|d6lJmKrwr?38*cK1EtE4Yj>DE9PV zex>d^rOjS6$}JW~h@*8=({w*ALDCII3ioYo){vS}DBbjtKkdWaz z1Jcx70huBEfRa>`5y0}e$eHl*6#QdK7J`4ZcwQGZ1u0*{G#w9-G6#^k*!#doNji*c zCN`;FbA1xaE#60k`e<7%Tcj_~uO&~`FL49DAk7@Z(5mp?EN{mDMD@4RKP%T^s6y#C zLH0aUBWjWqvHelQirOlRVMa-F`!`4*&r{W6_Ww2DTkXHh8$xl1l(89@ib{RC|2~9@ T&2^*7;=0gpQeY7ETj>8k3zWkd literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0218894bfe0a33f003526be955b1a5a2.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0218894bfe0a33f003526be955b1a5a2.o new file mode 100644 index 0000000000000000000000000000000000000000..e027ee7debdcc3ec54ba7570723b457e3f80b1c2 GIT binary patch literal 28240 zcmeHv3vg8DmFC~KTcY(q5*UbHNG&03fhEy;f@}<}H|U8agg}#QyCt=dtmm{Exn-QV z4KprdV;gO}JKoxHmYH}{ZVWEZR%&Lw-ZVyb4>v$a+BU=Pc~IJn0fZvXwP5!54fMf^}q29)n!VS?pOA{9{DVV zeg2z1E%IrXJ#xj)=KqhK9XUM1<{|rBcnPciVZP_J-!!my-qYB8L29`Befx?dr|rwg z)^p(uR{cLT&ueF)r`hc9c#-#kR}P*!fAr~77doCfb@JH3Q-7v;K8NffT=Q>Dyb?Wf z^jI|InOC9*FTNZ-*YV5f`HOEv+sb|sJ^0GWXwowuN7tUY5dHb%XQC%3K8_}@ULSe; z;?t*6OKT!yXP!FM`pU8B%g{eR_KRrS(UZ|5?H@;<`shM5WleSD$TQEJ8bf|hmL83^ z{qk7!!O34nUq1MY==ql~MlXzg1evPH+h?9W^)kx#_MJb7-hJ>`^vTEHh~9niq{y@M z3}o(&JlXlosS8g%bE@^^(df%3k44wM@kVs)W7s)*G1`{)F=XzEtbP1()bH_APrmVk zXiDM_qHAF%1#(Z`c_#YQ8)T<8($;zK)REGIr+!}YgXoi`$D%(!b257Vjh@5L+^N5lA&JMEq zvv(y%$i~@9jG2hFv-dI@1D`|L=CADii5?SQxAHof=kqmak0cv=jpF*tt{+E;_m7w} zZaDX#9jFaP5^;zAmt96Xs5!}OkKL?%ktag=lRVn`HJ(TE!g*Xt&%69(myP>~6MAVI z8zRXib&=Gn`Uv^TCHTcB@Efv6K0^NBgxxi`j@vvx`WW}WLH%2CAH#JJ*D+kSWOjvo zDTiEo8vFd?57<0p=P%r? zMJ`s_B4_VPh>#DFKYaq<{CKkiyu=9k68RkY5+BPcUoCteuWzBKFZtJ(U3HNwi|Q{^ z|NKjB0@Dpj+TnMrZM|syG^q*=14(z%4APt4m>_8)`fKn(AZz|Iw>({ zyLft{N!+zDmgr#{SOQ9vO_F6ZBQwE3&^Lf5b)TCMG{1XiMka?stZZ^B=r5a`Eeno~ zP7Y5Q^xz&z&z9*AgHsdJz91_L`a{7oQ8n-2p5VA|X2v^F>Ge+x_=micQ-KMu1cUy` znW?~ZFfdwEwR^%pF}1$jtMdki%WH_~8z0?0Sza?$;dZ-gs%pxs_;npY61fh%{)Q&J zZq!rM>lUcS%!n`GAG&oa67bLX1GA_RdMV%^za_j4_3+&i6&dmm?Ah%d4)`YgqIawe zUs)UnVK(R+AKxADO^@)Pn+{9`r-F0SP$z6^awh1T47zUt_Z{?2Pxu3aw?sFiYj|C( zeaV?}=;R$}X50dOsHzk1vz&X&gs|#cUZHDsu$)xPW3kek5$g!G6uIvG_A=x;OO@Np zkl)K?;nl8RPY1IafTrh;L=u>pGUSU2+6>qib(&Oc&w1;Pv$9&h}%_dPi&mbu;+073jNW!V{7Bq zI#&Y@Fwj>b(IF3YRSm9TVoe7_2#W^H2QE4t86EcpN2exT!Ktb7nGzQU8NYAF?<%h> zt15Brpj%mWEvw%%IzHr@nVSjvCtMSIW`eFE|M2Lf-?hD~tG&m&qp7=RYgebYY3KH) z&c>!jR@&vNbd~x7gCnk)32(W3z0ZZLsjPH9gzrZzWgTOWgYQ`9niRiFrRWV1hlfe`V+woi@`C~ZZo4~aj z01}mp?qACL6?y*x_Y(vlk$+L2s90zuCX(U{im}7;!OB~J+p|Ne3m^nr)hjh$;nG)gXNkj(2goOQ-8OU9Zy>oUYL6N={embQ7nWi00*I zoF-D_zlqbEbb2$VH|umCr~7o;!)cFBH*>m~Xl}oS(=9sQ$Y~xIy#5WGZqV_2IDL;! zw{jX`EX&u%={BOd{dP{b>v#vJ`7uAYzlGCVbbKqPx9ao;PH)iZT29kCK-k~LX$)5~ z|NA+8zfN~^x?87vINhVu+c~{mr+YcwOEfQk9jEJb{9aDqOEmm5HNd#PPY--zNB<-g zg_8J?8`H?&(;X{9i?}?EIV8V2;KTf-biQFfrJEn zzlj8UZ>O_GvknIzODjMd@x9SQxQNeZ(SG-N2sh%h zv_v0YczaYd5G~^k@|6uF&*P2vZ&y9xM!eS45nV(1e@OkazK-OL`0iUoxQwra6(P?e z!aYJB<=vD>IFCQYq*ya>Z4t$qfhRRkteWUVc#c{A9*T8CKPiFw+mP2Lx@;T;rfCVUIUstMmpv1-CMP^_EqT8ecOzKvqVgx^oGV#2#A)=YR0#hMA zS}(<_39qY*r|+d$=Q!rq1CJ(YEZ>#N7&a|MX{;cSr->eh(A)Wh=dpZ`3D3{d@zeQO zk5jbBRGbERcRtIf-4N`J(0u8He0|v)>knEFOmDknXUK1LD7T+-%;WL74))63<)C>w{VaQYj^$<|9uik#vvt-6dsXGI z2V?jfNW*xUS(pdDjrajbu+zFo& zP=`Ec!T(nUbF#@El;r%Ht&u!$MA&bdv+;khW*XMRXJ9{+iw)}W|Hc~$_-f3b^qi^B zkJ&l2gNG0=U&{f6K2NlsM*i6Ux-k9m@?-vQoC|VSgH1a-pRQ?vY$~9RXU=hD@qzs6 zP#&JBJm={=0NVqsCt$=$=Lc=GnaRBD%w>bWXhor9EH7+g{{jDOo6W@m040uN{&%_J z@w}vXyr4kGPmsPdJszj&)NH>jEgq-o)r`AS<8fL(NSxcRS{#ql^1;wA3ej@GpbNgE zW%Jml7QlGeHk+Liub+aU&5&=KExAL-kzY}WmKPF7`#NVd(f-TOL}<^V5KV`Meo=^~ zLo-d&jhUwD#!S<6Eoq(~Ef35%Ef35zP0wbUmIDUe8c5bCRQZU9@wzBP)2YN^zcrAi zxpbWBN7Hefey-4m(1(QQu8PM~^AKzzaeQT=j>CT2Y~fP9{WzY6<)&f3ZPv9U9#6;A zVc@NSr5eSDC_m*-%L$|WZL@h9@%pqJFzaVzFBPSep05a z8mQ1{{b%G~kL6OFerA0A8!)t+@uix`U)iVSiNtyPZ?;&EEO7_^_h76x%HK9y zvqrZE`J&JoEZ>asx6O7I$K%CyI?l^qb$2}8gsI%H-x_Gu5b#KT+FV>0;9OZXlIbhgt4K!+vaX76c%=+~j92yDP;=E~7pd1TC zS|8y9<5z27n}&cvQn-6{5hw%i*5Hr=$N52Ppj|@%3Yz!N77YW2pgF!l<9l<_9*}R= zFd&)mdo`{G<@_nP0+LP?d-sKz_RM(jxXhA-^wMtX<{$xaENAUD(HYZVL7vjD;U% z{}t!gkgvyfmnX^}>&cd+EflFpg0`-&mxf~5$o|o+H^V%?`U4jI1MWC|$Y+s!?D)0h zbK~Uo@}0sjMRP%LYCo*cuU>!0*Oz}kr}h)juMXwpJf`{wH1+Q`jPSg_;h%>>{5a{i zvG-%?Vtqo}jc_bWp1);;avy^T*67T^V_n$GT@XrpI8;l zhdYNii~S3o&t>K2BRleOabB3KW#N(6kDqVlasTJF5Ofyp=M4FkD~UF6EsJaukM_%1 zp~{f>jp`!6@&#U*(B zIp9C})OOR|QSwI<{V9F_gZgJ99v&6(%G-mckFfZD2KIFOVHi1`o{Rnw`FS{v_FBp9 zGnmq+&J z(tt(*q}R7)jz&{TV!T-zGPOT%AELvL^6#%u{+2m0+9>&=5RV7MDe}8s{K@@q%bbhn z#`Q6NAwMc^IBSb6KhNKs-!elU<6)dUFAvQoltlTRh+pi@V~o_WAB-z6@+$~Zeds&^ zt+rr!$rh3dT(G*SNg6Z%B5b@FfJU*G%Q@4o#WCefDO zZ^in0k!-lw;Uh1Fe-r!8d%wG~-j?D>zRYO-QOA!TY$=J!zZJBt1hcw< z+ZNe4od%pXZ~JYvP&4qyR^Y#Um{ypUc$xnvz<%ZtE9kh&X@$C{!B6buedOxS@GH_s=@j%*Twu(y3Y?t>j{*RT=5Um!A z)unnIby*pX`o-){Q+_Gd{AguIGGEur)L)Fc$5huv*1Be*uG)sQip%VQC3s}oBAhDh zv+?iJiKc%sxGqjQ*V9LbUl&JXUl&IKuZz3T+b#Vognv`dLtgh!d!onUpAcvF+iXc1 zyot+#CJpk~dOX>#@;*1Xb*aP2H@wo*lIzTu(jgu_n=$olm9fexT-~jDN?zR+{?B|j z_t3r-77w!Yu)?YJd9rwz!fCuxX(9V+|AO;@I1f?!bUMb9#e)=1V+l_#M0;v|qAep4 zKHii^KaKB4+Z&( zzaO~jw*mbw_T&qNFaE^~%O=NbalML*zMP@o4@{#2ul4%@)su8z=(hpHFX0DTvOtWTu(trOX*-AcM>Z?;U zs$fXEDtRPl5F?d2>7%M-rr^Gi{?DY|f1>j@Q?lyc0Srrst5TTGjgeeZfn;bzR>e^* zv=a?swMlx1X{17WSKRbZt1S5o{~shvF+#G-j_R7?!P@S%Qr}}KMv-(8mr&y?ODeln zA746CQUjf!@YyK!-IAl)*s9+R{P6|!gHpdpwML61uk=*^&qDcn`dg#{(kD5q{h1Kz90YD_uCqD_}GAH~iAh+3IX43Np$)HSSqW0s3Pz>Dv0**~(BKvVk>U-jt z7AFJtBJ5N7>n(CB{~wwRFVuh27;i9DQ2%9^O&q^hP^aZGhR&*~{&7N}H>2Swf226* z&+7#9eR0!2rL0bGhyMNr^8X`QA-9wiXXRHlAM;Cnk0t*fDt(zD=~GCXx`SJ;LZzNu zApZ|#hq|~FW7hnYAGJb%AM`zz{GE+L(OiDYU+F3T%>iQ}{S8uoiz$B!Nvl4+G!wJV zG`cyi{lijUs_Qpv`}aWqj~A%F(sxP)4Hl^Xt@%^jP@Agvec>@KWlDW@SNg>Mf7K(G z)Z;TQHxLDqA@An=sH`e3q|{Y64;2c@tjAd@o&~Ibs`yfOrKbdT*Y)hEyuYbVq=t_A zit#;zmG$GhDtme#tQ+6%>l@m$YiD=);6z{VSoQr=zPg5vh8^`IgA>ic{+-?9gVR0R z$G1Mz=@I&hC+26QOa~ZylLN{#z<~ z+gf&a`s!+@np$>mAL?tJs^9uh%TUFR>HdllD#!KY8d2}c8_5iyj5m;J9-63qa93Yj zudi;RWxTnqcetP0wtT9wV)y+I)(=tLW_IoDe7Jq0b!4z&$8i6a&Y8Hn_KxovYTV@> zZ|&-pb!hD0SxXsI0;YV4RZ>9h=1cF*#Ey}He)KWge5kdjS=0w_KNzo_gO9*>s)u*= zRqq+7>>e3v8J``P?w)Gf`cQLEU$wjA;WqSj_rtqJtL`J8^iGVA2E1eLit-H`YO4l@ z{cc}n<*?gbSzS>x;IFN&9w_%!`zpLbZg6ziH|W2{EA+PWcju{BXqYJLd-6<>cr%`f zVR{$d!dzj@Tkdz)c-MPL-@84q$M0?E#cKooz6p0o4Y54r_7;Av-+WPck8AI@zm1xd zmzDGSs6pNKa5whSdA8iteEL-nzqkNCiqr${CAiGH_z@boDQ!<(>G`hqoa5gl{#)mt zq<$~$1!0-whq!?F_~s4dDX-^D^7{ICOWL87?`bdau@X4Ry?zsN*oyL>X(`c0{UI&m zyW02oD;FGc4k3u_^3-VHMJBx1f|Gos3BRQ9_O!z(l)(#P#3s4%n~+oL*87Xlb+Aw5 zzZAry_62r0q8_TT4S6bpjqer&@1+iL_c@5qXE4q%&oc|Xk7GMG4W$LrLi=MOFBf|j z9lI%C1qpehJn?*5Da0ospA^poHHB-bseM$vXu2i7@_*4@jQttvrFdpz#o&{?M>3V) zN70}Q$#-0X|2ya}Bv1RKh4|~Q!G9fmD$lFD9~e_{Mc;ocWamF%0mthryFkSi$u1=S zwyf`Qi~K0$$+WD2_t9JfNS$?%SR&n+3WI5fki!TW??1Z%A zk{zh{ec&%qMj-n%Xi7oYC;lNx|E*xN@8eEod_2ryLOnmc}^d-rseD#xH@KnE1zGi-rtS8x5Wg`>x z{TL;yoeh$oXR+fk*)g9sJz(NvKc}aViEoxacn$u$Cce48^xxH#tmSk{zO}w#*+JI& z(s{u`cAk@bYkmLJWXD|o*G+tLecv_l)gFuTyL1iyP8mN`FBt;7d=E(e-4_0zn&i#> z`=aDq%YRhzsU7Gas#J2y%v8S07xANho>aby-)@pO=S%x%O4fYU`-!ai9yHl8$IBtf zx5mq>l23M2St!5vO?=wdsPYl6;k~Tju+ml#Y5cA=0`ZKt-8itiyIq!Dq` z<<-3LyoH}3{pq-cujYZX7QPzq)jnS3tH%3cl#i0)V~FKxi-oVo`)LbbjrU9}yJx%u zqm#bC9OG{wZ1?S%92{xmuNZF%;WrxmorB^%!#BMX;jc0rogBq$y$5DyzM9(Po(WN) zQU46P@uo*^9r)|SZ!A6Q56s|ITC%F60|C^##UB*a#XINdz2z;Py`^}6@Zgw`YDRnX z_=9ypyl{MAPtZ?XArHkk?vSt1KRY_;kK@xD8+~Q|5&3%K8y9gH`O)jXyp!}gSlPLKL|#b1I+`J30+q`6Jz z>mkCFRQ~GzU8zsD&FPTTSNmm!sr(f^ga_hT+i*jlkg&>I>3=Wtx9WSO{zh4Um7glN zqL-lZ);=NZkv~up@pr52gHqpG|FG0QWKpn|-w%AD{@dUe`CgLQQ~yzuDt)z|e zD?+qfurW3bp#2N+gUanfkh@*Ld#t z@5jtJM*|YJ6Wc7dJ#)_fAAA3=z4yQO5of+$wraVqX-pN3y~5%(1!ZhtfBfFVZ#^s( zly25p*=wu$r);Tu|vHxph-6vbwG0-0G%wUB-$oBMnSi;_XM`y=oq;^(bvQ6slw`EdW z$2-$m;Xmu1o&oUbn(fUR^gVd~>Asa$j`Z!i{&e4}3s3ibTlWls_E~(6%{}{Kr2EY2 z$m7oMMNS_%6B%4^HqyP|eB`M$7b4f5x)2$B>U!kK1@A{5X}S_Qeg6H(*jbN;2X`Im zd-Cb0`%)Jy3D0f(UgYZh(~-m1{~_|E^NmQunQM{mRo6jR7!Dsj+PCw}k-ozVPDfT= zI1_1Tz7Xj?@HwD0PIgMEkBpq;0_7n$3H{?ES=skm}2vhzCi z_xs`S0`#}$DB3v_nb-7UWDxk3SI$S~p1lIush!^o4?cCY@5u#E_l3`&iCo2};rbhq zd1uc?DxB9NR~K9X&9d;^rh|Rmutmj{(`frlBz)$4WTo>$B)s5S@30J_D^O8XWyNykLF%o|NjmXaBXCrg3ycfCp z_?5`GS&xMWA34}pk^EQC^=#y)fTzs;gYe+vfE|1>l5z#KkUd4qzM1mtrz+=Oc`2B( zKl{~`Lr(&-cvG)pM^M34oxnr;M4zYJ~gxEW2i9L!*6@R^!_*3zErx?7o_s8LA$)q<2 z9@39p{m0@Vx1FWE_fGN9;7esazuLfhQZYU*&1ORbKh}rNA+Gzg+0dnuiO2eDwPTn5 zTE8{$6zidJ#I)YWRsbJ;j`c+Uh4ox}MIU;%g!NpW%!dAT5$m~<#)htrVMAjZqbtcry!8}8A(SM4@dT8GGc1jmMa4if3r- zd`~!aaZ%X$mEth@hI~SPqW0c}T`$9aF07xy6i?4CwEt~f*W`9a0> zZ#J-=ND1pFV||ve2gT{fYm36S z#uVQq|J}ET*&dw;9>g*K(AVA8*S%l$zcl7zuhyaw6rE`rE$|*$C)f&n9B6zO!SNY` zPdal^x5LUVf3Vr#*vi%clS5b?BTrAoRo9qpyiL<>jM>Lw%9F+BWOm$sUF5Yz3Zb*V z5jsh-IX-c7>W{=#-^G$WY$;2EMAL{ex3#{tE!gJu;Z9vUnnaynYg+4@>q0EIxuwmY z+q^rsZC69{_Lg|Pwq4eD=f-YoTbcsiHkRAw54GhA)oN`sw!C{Fd~3adCV#N@ zo^UfX>7eL+_a(tbEHAx}AgNqGFtj6<@2!BV;CW=ga*x z@Ov~>RQo%A8;GaFRUes zk|Tk1PX%)Cqib5~{?5)h%n{w+Imf|o86)_7Ya_C*WVY7>Ab9P)bO=IKe2(LD5%o2j zDoQJg+$Grm;~Yg4IwT>|CHQP7P@pY@SS)F;^SbGDp`p>+*3i=IZfj|2Y@P4M$o6|% z{qFolxr^t!H_;`xa2YFZZ)mJ@w|2C)`J3EL?X7L@I{)^DX1{ym+O?}T)NCrN+)%N0 zO-)(##I3AuEi zF79#5>>2U<(H!U4>QkA0yB&T3@1u6n_GIdvAp&tSbR(VFW!+AmzEKpb+gP5dt~0l( zPG?m{dTNvEHZ*H{Aj<@J`leuQf& zAy6nThL1MJy7p>O_70Hbp#C&AwL{p850aTqc6v{8lpny21%=Paa_Nvx70{;!St@lN zSQoLeUuLteZ|YHY0{{}b&dK)w1=D%ITe4Hx>0I`A9YP0o2za#L6~aK0AnL!YLuZ{7 zxO%5e$TJxglv((oy<M5;4}I_VRd z7^UVWX2QDXsGi2_X;cTkFzB2pG$wwok24MQ0pAVY|DnHN+_rWH^zUEG9^4BgFW|qH z#{Q&Ze3Xk%_#7vj9m*WPsj8|<$dk>|gPEC`ppQD9;qB*yGBe3P<4In~Lv?oXC7shl zp7N9v(j3qq?H7i0Cl3m#PuJe0DEj~nlD}ZTj3A4W{Lw{1-?9#k-5_Uc!hU+t#+;}! zQVQ*_2QFaOZ6mNN1sLzSAYqIXUuFVR~n@#=ACjmMJoM`398Y*HtVfB4~8{WU&5xIY5x z#P|^Q2#EgPvgs7RH19;C2e|*Iz`wDA_$yL%@;BlG^T#AHf1tfzrm^C~njok61-|_7 z&n3I44{y&*k9eI#{^aqB{wTh-5s4fR8KImUnjLsN{u8yo!qk4(-mG{$^&iE2kz}KI zi&D^BScv$-d>D%_il3|`>Mx7O&okuDtXO>YJGH{#^rTq4Vt$zwnm(QAljHF=eG1ii zyd40)3;ZhH){sA^yG-%s0NyS9c`A$J9jFU_ibp%}GTsp1uy0PxzOV=PHwi%gev9l) z_bAfd)Sq3&XMYLo((&Btp7w|9oLAL2hFW^Z3M<_nj_-Y!?%-fPKTuK4)o;{*KC-e^A~1ZzQoXfIYF zWiRePX|EKszSLf9KIkvP{5{8{-$?fA+G~&NKl2Zuzs(SDq`$)wZ+08$V<-L3(|l&P z$MvWAQt)&C@_2J_dPh9o9NZohZ3{3`NB+G?8q4{+Ff@KV z`JeZX^`d-g4zH>8pldItezkrDNI&EYYCO$!Vm)N+1UM#hdDHYl)LFm=mqA~ykIvUe zo$&qH>>EnHmqCa1@+DQLUVBk*ryE&-Kia``+7%=hO&AaT*>L?OZG3(*y9#s$@ zr}w~rPXIvczt4&JMq?+*A;kXy^iTQA_<9a|LLbw5o)Ody>vuMvuhx+Zq&@ZEr1<)c z^>9VdMk_gDRIT6oM4HcZxqc&l@hDZsFXq$o6eTd^LI37>%Z0sd#`V~KdQXx`pV=XY z^PO8CW{P}<_{T=sWpqgEt&irjJH%tZlj1p=4N}J>2nBnQKD>|?h>!H3&B5p6|Dp1u z6x&;vzEpphznnDW)O=DHO0m)E!s7w?3gP+umJwjIeiU{A&pQzDlD^ND!t+H3<%VyK$6MwvKA+iLa{U-Fo)rPK6fOZ856IsO zLz5jR|(s(kA2g>(N@eaIcJOFQsci;`<0rtrW znd6`G5%Mq3Zy;|*$S@wDFX3bHe@&zO3GszO`SJ_+qrA|xY0Ql}+4u57)R7)ggQU+@ z+MhX+4ED9B2UC&^_T~PhsY>k!Aup_<#^+j$?IvM+^&9rK+s*5@9RPIC<2{=tVgBHy z61`x)*x$iKK11gP^=a&x4otP8=!j>%Fv!HA9_zu=e@qZru z?_v<8os}#YyySnA8G7Jh}j9k^Nl`QpH7ds zK0Rw%pA6^ah@a7{PqFXg3GiXVSi)V zKX0e?D4FlS`TBJh<87av=M%{H)7UxWr~8tUX}unkndYE;hv!rDIE&T;%y;7a=j3LV zX#K)^@!Q^Co7OAjOL~w$Z4l2Jpbz0=>lfrt$NEM4o#+(E|N19czfAiH^LmB;t?Qd< z{o;7@{=&4M0N%8o4eQ?6e(O!30cyHx(-BY899{>OTnj`N}3qtZU`ANDJm^n8WS z$Cv*Q`)_P7FY|a~>|u%@ou7yDeEM}dFUvISw||D`2lh<9-;S?m@$;O1TF<7(&!aHk z58rP;F#XQ!*)$&S4fazYPGv#e+F^BwE{b#(jbX~y-e zFf>7DcpxSAe>6U(OrR4fvA@nvO%G0;z}H`~zozwc`UAil_Sa6@aK!i5PUi^w>uJB+ z^(`HCz4Mg#XiZi#<*I4RgLq`{~c|JWcGUE~pmA3r}NCB^)zPe#Jd z`3KT@`*gk}+SB6a`FW$$*Sw!jV_$_lV!w(8aK2OC@liYqL(BW+z6I`E z;JyX!Tj0J0?pxq9wg5dWK%24gX#l zyyKi>{}Wq%#tvx9kkn{C2Kso>XF8KpSS8^h0~id&gL`dUGWaPQ(zN-!J=Nb&lI*A# zrxqs{rF!gI`PNB74p4%c;MDd_R*YJaW^H>NwfG3h!rQ(H9vihoAHm0E8f)bP8;j#@ z!|;^y=}VmjH`z8kGsFE@B*|S`vQe`{_?SUmiTPPlovt zDl_gkqus{{yT|mVo)X|=db<+iV|u$~OeJU|rWXVuU6p1eH(HHLZvRX)vk#9b9_>cS zBuz{*Xm=;cY{P77mfWUWB~NH`T}T$|6^$5uFKk^@-aPisqg6wY#_UdlP*57}I1EqH zkPX)B`s#-<_S~Gmi#DiTn#V>>$YPX7fQo$jSbe4=`9Jwgp$)SH^-XOXXN&!qE#iK? zo%K|kiD#cU{{h{quS@nhEUFXu-W1gqzf(KIH8%7-%#vU*SlX7HNHRUO(s`3_{tqN; zH@g_a%T0Kp?ohC1;2&{lIKZJ@I&D))+?bxDY1_#7}j6iNA6{9b%a1 z-?G$~YG%C0QeS7OtM>WyfcC4!2|jhn<-KE_qTz0NJ}rGwYN`Eh+epMS8rd)M*>t4d zM$^(rZCLS%+Wh>au}U{m-82&A`op7GgcZg09pfqyzZxVov_|ANkIUrPxO58vto(_r zZKe65*G)D{u>M%(PeiYB0BPd@vzg|qkfnYIb>$W@SJ^Cc7ys*L!`x-2f6KzZ6q&NI zJ&&cn&Qhnfha8%4?lR8yBKAgF+wQa)P~gZYc@=-IosB9tmNqUG*&EO5*$i~EUejl~ zuYTOkMA99I`vYNy3tRz3(B6?wJBIHq z-GIht_QZR~_zn%J#Z9cnYKJCQTI%!jS(B(|gY-#>R!oERN%!}ca5u``j=Q_1Pc*aN zZ5y9_CHO3Qw@RC+%}*|&we{b1-87FT4?h94%9DuRvd_8mUNp9sv|$Y5-^Z&N#LdI; zw;qpHeLm~D$vz2A)vWy`qF0-?dtu)QeZS~uc3<>;xoJEl+K1EAsM|m3im!SHm~qup zqxMZ*UsH-bHsj~n8a=V118(fHRWYoq%(||6KqO+=!=FnfIvY=b?-hUdCpPS#p#5IH zu}9zZ?;8@d52z?KhSuv^g$w5jbT7)fB>KC@99g4#DmJFw@m)lV6}#nna=8HDF$@&i zhV)CmkmyE9jB+&Z@k#iOo~>^7mP3#Br{{26yZU1fd{26&a&HDRi(o!oL z$1X&dUx4@cTtv7$iwDYqnefL-$b|R!Tr6d*vkdPr!RL1G@gW0|?dh869X$kpo4!!s zAG{VlUeG)*j{(jPt>UPJcXI)~`}b|}ur{?P-nW%uf>GrVhLnv>90nKf+lsfN;>Cv` zo?)N{J1^pWTT=`eaT|Nzmc%IfVT5-HppkA%z~hmy+So&40f^5GqE1VABL2UZd|$IT z$I6d?#A!L-678;_e|(4HkPNi1p}v$j%0lE1Yn#NkP7e5bsu+rumf}J_=cX=VdWNLq}ZV-+vNem)ymdY!2 zDhvMyx`sM|zmKRRnW%kJk^CML&=O?8(fmU47hCA?Dt=ymZfH1>{{je1{9QWvPe#1p zyY&0w`q6Z8or(+TZz4kd4*{h9QB0_}s|Ao6HWo=618oHVfdu`p+9F7DBqjAv#bQ3@ zm;4?}{{hLb;+Oa-q^Ymi`Y({wW52>R<6G z_yYhW;y;;ypN52$pMLZSt4;uJHrfAG$uGN)U99%s0RBHspugh3AvqDI-2c}8DJICK z>h~ZddF(PpcBrn3pTM7#kLFS}K4Trp8#P|l)z}}&s>X$tPt`p#K#Y7GB*&AYQ&%;< z)K&2*{Engxn_7#?ibN?{U0l%EUc0Ecam%6&RXd9sH+nbMwQs4e%&%?QT(zrkeT%oK zWOd1=;`-XA{(u0P!HIKD#DjGE?d8)dS$(D^QOG5)hkzdHy7?| zT)DPt!=Ccxjmsaest;6dXk1ZH^?1deit2*O#;r}u^L;A>Z*k+YMqhPto_{NEcU5)4 z@^)X-vOLHT@D(hn+F872XI*t=>z3*@PpoQMlV4X})n2!vaS!@vh8*Ra$~U#w7A)(a z{^~ZbY$>kTQ|eo^DX+17$I4j0jqFcVT zDOA68(OTG}sho7)zLjj3-%?tzW4*U{P5zdq9c4RZG^3!@Rbzi`KyZqz9v8XvuUNTdHoZs$pvZZ8; zx2h^oxva6IzNK{Yj&rc%??*buDQ zl~?d+ZSCUPCHb{W3JZL7OMLlDA6>G&F2B%QQ0wzQy0k{n)i!MR*7|SvlPMedZ>7+$ zq~O;}K1n6-4(}RYw0QC2MR|Gn@sq-Wn&sZcR=@l%slxBW_`OYep*(1|cjlM+b;vV+ z&%O6+UxOa`x%u2rYEssosKgF4OPijW&1mjd^FI;ga7ytt9~a|Z^AJtER5qqQ=y^^* zZvU^zf9Ck})NeVT7cCQg9VZZ9`$nLrzK$E{UsLxhoF~V9OMl)j>qPe>g~Ro{y0_3# zdT?3ca4l8$rB2s#`nT+a=l_sd!l_+e4EP+wqBO^XC;CzYp1(7XXjVC29!DKKFJ@e# zYa9g~u_h`nfY;7`qmCv_B(?-Q6rqPQcAT6<(8Qk}aNxOGf!kmwIA4J{z_?B%dVljq z-ZYpJltlYjLC;slVH`F?zX~bn$48;6DPK60Ud)BwTG)OJ(~rWjpJn%aomeDCO^z za5b*JA?18o!qs?sO}5i39jM@&0Z&(*qR;iHQ)xr{1m7m>7X+An7gvJkp&u$|#f80z zE5UC{xLv}PeiP81iXB2_O8i6%JWs->0j_un^i>Ie0C3~hai0M$w{#qGO2VmMJo$)~ z)AEk`HNs1!p47gQjYwBXxV4>X3D2^$qwGcP7*C5j4eiS}0dh%B7~n?w)3<@&G{BAe z@|U}!MXQ{1q(7|s_DTm?^*txy)^>g_;Z}X$Hnd}u|04t3sP7axeyCsL8ku(+_;Cp* zd5p8!2?>A10#}E}R{39(^j7)BOSc#!d8ToXh_A)6Gu5w(#d-->{VMoz1HCb>PDr@5 z-?t^)+Ha~Hx74q3ytpKM*m#j}YDdXJ8f-JbXb$GG?aj6IkMp09EW-S);;pM`YpJQ@f(eBno{&>i zg)T-7m$w8J!QJ-7a_UCmGjDK5YfWpwUpxGFLW3=J-Zn4G<=-M8g_l-AlTwZnB)PS% zZTU+^ZH+I-UU>ArP5Aq#pJe}T;`!8@c`+JW)i1GrNq?tROs}8%`v4XZnQEEZIwup< zsWSgiNI6ZbDx-Zq!DLzxgL~xNMY)bCgj%mueKu|=ep&U7wTu)V8s#ll*F*tZ=G-Rj3g+{?nnn)hC3T9#j`#PV`0edGDP)StwlHO? zCIAU+?=+=$9QT&&c^jV--jbQW!1Wz`eu;ZPi%LS?|G}S+?|Y%|*EwqBgRfY9=T>?9 zKK7~|=e(Yo|LRqHc6O;9({dAQj(dx!+!F&zRlZ31tCseC@RcPf7asZezC*kFy^(+T z>yq$+lgP6?-k(KKf8X{;j_o<}$gwj&+&b3>=j{&eV}XI~xq;mX&B&ffak(95^IGj#fqvqM)u`|eO{?z=;QD;D&<-1g|P zJu4qQmV3?ozEih8H}uN0dxus$`r6P#@BVb?)Wh!%J@njJ;Hvui9)0xKJ`+VlYeRdszdCek!@EPLHxQ<>Z^o;S9J~3oM~-dVyBB%S4rRap)1iBwdllHT zL-)LM2AJ#mPQCN+v4`+E1^rHMcy1{BwwH$Xpsto@&koJZdIv&ADcP*!DHLr`{B?-?>!5f4{M!IO?YqUp}@Pq+|ak8eA}^Y zWq&c0{mxH^w!LzCDBDvqyluwzH8bA-VKjS7$@AHni8a{=OXAetI|q|hezcj#N~~F@ z5^FB%x_qK_Z{`Znqhpq4dkB9PX{&CBH z3zZ#sH(7NM{L6K@beXYYi({3p$)%p3C4b|Et>KZAh|{+abI0>l--(MoeaF52neUWX zeW#`mp}g?nFk#9yy=_^AI}s_w0i*}Uw6@PWk>Y@S5E2EH6#m~%~d^Wqum-f;HvZ>-I(`G%(oc@x4TSHYLb{;SR$1U)k&JQB%PzuGci z)y>Ebk1Y0Bzj}HBa0TI!_k8MC*8sa`Liq98$>9$@)v9h5@ZXyd9-6k$;4{>(YOYmv zQ-PZ}A$*|lqVPyr(4l*-Tla$<&>cMJabU4$;)Z&ZWjnT9hkIj3{UY3tVaGwrgB@3+ z&ifBuea3Txu>-mv#68>b)?zJtWF2&z4eV)SM@?AOT?aev&kqmP*QD*(vP9Kg0$e89 z5OUa1sBkp0>;tX}~;hS)e zc3QjzZA$mlUf_G^9`;p2{`daA^2|Zd(@Ed{>1W(?V6Jl4AkdcT+nAD|rTU9$rBCa&sl#=X(|;4Qc(y(z|4;@;?8L-$7S zw+I8hXF@lMscE-zyJVfIxlPs0M|}PB-_Ji&zsk1Z;`(skWbn$u4xLNgI(ybZXYg3( znb0|s5gw|64A%MS)wm~};k)`ZxM!Vj$G!1&*_Ux|bUs*zd!utb-9zUY(0Q?^=*-i= zeJ@k}YA5*>e)fY;zV+Op>IUWo`ob7DPT!r;H{#9P^Okqw%;898->|oE&#%4tdp`I| zR^JES%ss!%@%Np`&+dD7c1|CS1zwC}G^gQS&p|v-TA-I;PHIqketKdp<~XGig@1y6 zKH&B4pt3vbRMp8JV}4tu4wRt3B%s#->ZQE?+f`KnGBB1M%P#6&^RK`3j{NIiU$W!m z135c>u{1n#!aHTpS?`pe-kYrbdDzo`y%ic93Jkn|?9|)dk(Xh|$ur)OGF*r9?t1*x z+tz_o`H5pEU(7mi@{EP~bn?tOd21hk+w0#k$12=2L-h}xyx23c+&k&v50`~U-d^Py zdH4PAJ^pI)#i3guu2bW#c_v=3@?EBu9@Q;S_JtOCGJO_@<`QD%25Bl8C)q#ia_3a^g zgC~E__Ir!>3_O;&K78vb-|4q~J5G5scDy~isP9Z9JhC16r@!hMIlVYMa_TFdk&{1I z(RVh>GjirS&w(@3JqJ#AsFBmxVP44h95~s4vJa?{Q@4PQ0Ndc*aq@1@$nM;{~<>zL@UaacTpu6LIZ8Jzs~sAKZw!Ccp3GPt<|am~&3wtwyN5 zYY>yv{wG(V-ykLi5EK96RWk=L2Wh^(pE$GmW>w|)sV|jGP;*+)m+d+3M$9+0eZ!HZ zn1hz|ebiCY_wju%>2*}ntLH8qqa2$QyA6oZ8niLib9;C_S8b^x)HNy=^HlFvlp%Sy zc*ci&VVt;5^bJ}U_&tjH$EfvRHVzEdizBS(HRHfzUHggkI%O7rN__68#Am?v@59HL zxW0tzx5nu|F1Hl;3$-`<<{zQgcyKGnDUa)Cy}yrgU$m#%A^kFaob7pVobAK%|NE!J zb30u(&h~u;^2h3z98(leuL1W-?e6SP7mWj-Jr0ahCi{eKL_Zo=tT&Dm$MTy$C7$*9 zr*W2R8z+wSVfke5r)h(K>oINJXS55G!noQ|`XdWt=`S&s9>iRSeb;A_aqnj=hdK5$ z)%RtT`=mDfUHb+586W?VR|mBxjW zaqu+k#Cg>o-$%)B(R%zZVJEGvun(9s@K?z?@;CPWGyNVqlko3>51t_0Zs2&YPgnL0 zt~4epg+&$C{%R*abY1=Km;c72f~vV01d^pZ9yrRDlXo z@$=G#rf7RpYnNJ$oKniQvOUVe%EGcN<(=U5SYD6vgI>Jk}JCmD{F88aBpS>$|!lhKx0}cXdRgu}Euc zTO`&IZD@`(#XFm#Ep1KhvHI4wj>e|emiD_No961+mViS_cS%cBK5II zWNAm|=9*YXTcoZzx}tVYMWpVImGzq=6|mE`>vLd6S95)|sqyn;NVKV|DY^+3p_QUd zt)Gi%gB|ss3qu;4)@{5yvOZeh)?~MiWYm|tIOt{#^{uUUN9#MAb*Jl$cEmbjn>$gQ z@;chPV)gB@iqAp#Hq>{vHANdf7tM^O(YDC8)J!vU>IQT&J_l{c=D1^4b3Zp-nA=tx zG*$zfNyRx9bGa^i9l=YH3tqoihJ3LM!2&q)A8A=QO6Y&F4EcAqdUp+5)H;$e{$FrDrLKeRJH#W!sAl)*EQ@jn9Ji}MSal1Qb7N%xeXcQ;-0bGIZCWy+u2k|^`* zTcwCUOA0YG>+Gur!!GMPYfQz-1|7Vpqc{|~?T!_-D{4YZJEEO*%%T$=lF)g{e0=l~ zMrSOJSX{cXu|7nn3oWhnv6hbZP^_b)wQE)gJ-eyCt0^>RUU}uL&>eIsuUe>TwtmvX2z%L#Us0rSlb>KMP$qsBePro9$N=2{+Jvfqjo#~LAeD! zTkgeFGu8dPM-3JzpP{qVZqPL;I#VS;Kb@j8R5R%M6zx;XL2pUXUKIwNmS?Gc99(%U zmhVvs(CK{Af%3gLobpadDr+TcHXr)4%R362o%x-x z-I>33vMrDFr}~0Z((*l^`!99o2h_kcXMRBSU**h){$a2B0)rs)J|s-N?{)js8$ zTC9R_(mo7B0|5lq9quWCMt)D6PPk3d_5@Ud>6h|VZ`=<} zgDUw}GG86eSKp3j`_a`@GL!Tr+KRS&6{QV%41F-i*i8`i8-}`cQ;)u0EGT`aTu6e0be78Z_|qrESmC>z(xI4QAPSC zlzKFa?42CXNR}YiDncEwZ*n|K(~9U5`KlnAkwM)nS^Zd+P5X2QWV$e(m64zcs2}ox zFN|mVQZ)RLi~6Vl+Sg0@plPb0{^2Em<;Imo#XFaDYW*h1tt8=nR6q13yzWnhLndG8 z{0!DFo{<5hO1@(Bi~D>ePt)lCRXrJ|eQf=z9`{*x4+Q;6zFHpjW#|_6{MJkKD^Z)& z{<8aXhSq<{2RSsJOhx~sR6Pj&k>4}LmsB@E1Byb#gH^b|t`BoKwMWmC%%r;33;#o2PaLgphqU%*+MaW|eI^&%?XUfN{4sS!p_M@vTNE#c zSE+0W-@9xiw(LWvi2ko)<>lw+!cOl-7|JKR=Q*z<{vnx;Zk5%se z0Q*J!`)Ix(`$mmV=IEw8%so;^Utj<7;S>p0CQ|T6z1QqcncTauCf(_I%kB&!UmpPVo1)<5^iWUg`M` z^XrQnvD~2fr~va~V;;(@+-Vx!i}`jo+CMj`egKAP|Gi%?OwTXSw+;8DlRf?* zsi5r<{TKGS`e$SLh37NWw`Fssl^V}>U6$&Pm3qFi`(q{5PxI5b`eRjkzJNc_pDL-p z>iNm;kCili==n(Z$0}pLJzfD{MH6c064U>-RjR3=x&JNO_@HI42<`Z_qS>mPvS~gz zp6^+JyO`Dg9%n8T0S#}%(v=25(7j2`cn9?x9yh4tyOjq&VMynI0MA56#l<(QwK zkDmW*eeC#Q1$2D;4)KoutK)+@jEN7#LoglR1@7@2@lX}@&bMcPlz+WdDBQW$_V2Ac zJ?QbRL}hkm+Wzbf>G>Ax8L#%QUeDo~Zu>N3oPkNG>JD?{l4Z*ZDY zjhObLK8L?cU~dlXClVTu@w_yelSBIRe0ml37t~{YBoFa_RWu`$DomRB682(2h~%;T zRT|CG=4YAt74~Tc`>)E%B06EO_n@!eUN71DYJVBLy`HHmcI3mp!e=s)TEBUAeN|D* ztZxmC^_%T4n=ZuoPYZ%%Kh`TKZ*|-IV!fZz>)pMiJ<>SlEAnp?=VJsXdqMYfXYuKw zpXQ&Q`vU3p)jFEbGSllR^at|44k3zntT#)$v-JE1f9ds=6*cQa%7?tFsL!X{+xW8} zn(3qAEXnJyXqJ!KN6&wlf3V+5_fOQfBU+TD*K5VL{4JaPnR&qL^*D}^tZp9$ek9so zV!ZmnMwRqo{wgW}P4m~bUCQ1ek$+}8_KQ6_o~tllru+?mxb~B@{)*<>6R^GBw|e6C z&`(ur|HW~r(Quj9?o=%3~G0#EzjcSSt2oZ>r^;_;;t^-bQ- zdV2KwFHPI^QGc`hBly z|EXgBohs+wEWCfIlim-#n4K(2&9^B3t>R#?AKCVNRigG^gz<@7q~|LS)>98EI@3rd zAJF@8ieD{{+Uqs+7sNaHgZtO5+d$|&7Z_h{M zZ?s<_=3iRhaeI`;CzM;Du^yxN4;ID|z9fHrsa!G!+%xL0@rw3*qx{SqxD67z0qJMoq}us zYTBQvjuQvG+Ghx)6PjsBXM z>aR!51C7-F9q#@JeaId-G8@i4y66n>QQ^M zFg~{+&zxuU`U@^`_2*o%-pqBZH&NfW{DD9}$*1_C^<{<%1W6^czC?TErrLwn%iFd( z+GCZ&ANKf3?P2<}*4O1P-CpkX5yn%qptk*i_?lqswb%dgA}C`L&I^0u7|$I3k?ouI zcaN^$6_2PtoewdfUF}W!T_|1=4|^|m>~Hk=Iyv3`D39^9*yaBVji~VNlS|Ba>gq4LeJaOjAM_ux5A8!!@&B9tGTYr>u)g}F{^5$}XYw5J z1bxxp-2J6wJpCmv)n8CPXy(`Q0jwvXkE{LZJk;5LFn|1R{Ri`%tN-ZnEZu)FAGqUv z75dYF`BoG5zB&6m`(64U&-eAX`a{Wi{ek8)nxH1oe1`dk<}af2Q*?o*3y8+}mG%dw zCB%oTzrIfE2}itNndiVmDDYT+*!u}OFZs0bkNLjG@9saYc>jlz^ZEzY6OQ=Lfd7`O zVaIynRz82$>tCF|rfBGk{#uIj2%3QP`hd;@v0g}k*8af!P#CwYeOQ3#^H!QKx-ES^ zi1oZ)A1&LMoBqB5`%8>h`aIWlUP1!^&0nZ**~Wxl&rkY%p~RerFF}o16~bSW<6eAU zjeU&%{sZgh_{8}r&FdSCM?oG>^>|p&U7`0&$)LTS+57dZ^!W?ow=nLraMG*1alL-= z_1ssKR5zh68f6iGRj@CNTi&ne^MYL6N4e(C)(>Nz5Yq*1AIaEoH_P-!PdVjK6|1)A=>M$UDzM!^|9VR zl&I@)-VAYcK&tnnx!v~pUuPk{&wzea-Ia8{kl^jK(QEYg zdFT%ojo*{wb9H>rqxgnDZ-{z1-{Ad8Y9NCT+z<^gZSu2|w4N~e{u{8rH?+z3Az~i%b$(%d?mUX`xzyiaZ$bBU zL5h!pqQ|uV==%UzZYQh{x2L|pwBvV2G+=+-E+c;|r~)jDNL0R(F}bzt`Wt>hZgBHni(p_kq3tjm|Hp^EZyi9Z|o%Up4#1 z$I(A^{G0u2X}khCmO#)D-cDsIkGCdUhO`v7j^`tuuj2Vb_m>-@A=byl-!#_8(3i75y1f=fOG_Q{ zQ2%t+&&2Cg*3Xo`l=U<8Wvri}OIRQM{o$f$C`98+QA*z;9I&%}tF*speoNasDM{P? zG<|VW&nJ5PAp1+FlDtw|9@=NBZl4*{KIlI;M1w3}>yLPuLG3j))m{ZjQ=iedkn7X_ zK>VcR<{4!0;SUR2f2P@PXX6&+OLr7H~AB}eodo4Kt2-n56r(f z9}ej6Ys`H1IL^0e{L$wJ82=YVgSvj!x2HS9-k+z&dq+HDKYzh^*6k(Z*AZ{pe;0^1 z-5)}w)c@1|aKxXMUy1`J_P^b~XueFx*Of`yu4-D>ca?roG@W0><&FO?N}BS9p2X!% zd<3;V^c)ziFQ~m~M-F*jrQ3Tdc62Fy(SJk9GOqsw^@nN687Z2|U!I)lqNgXXcF|>A z-sm%f%dbr%g%MfUV5Gf<6|++*JixI_&#~| zAlCP0KEn7uWj3w%4UP5A?D7QgvuXZ8d}BXSN&UBT9nRaaUm}|NYv(%q`wG;LS;o}= zLB7h47a_b7kY}H#XUB^X9%xt8q+fT;g4U8a4R_j!oDO#5=E=uXI>z_P{ z?9=7(ea?e_q~_qj#NN-qKlbxQOOTII4m8?(eibO){+R!_W(O-!mPR)`5KDAHg$g<8 z!U`2OH2kqOdt!xJYiQ`R$qG^fsY3F{EAWTq@CWqyX12PP{F65s4*;I<;QaI2YpX%q z>u-Et`0a!`$NN+47v1|)=kqSuKjZsQ_j&ht_Lm`Z08QUtmta0sJbx>US01a|Yk$8+ z>piT$a9-r4@0T%&>iy~0w!u4p@+1`Q)#vL!_ydzq+YIbq<@bZjHlIB2{9eyLBp?1C zwLa{xE?6JsoBEb*bIOPRHlI*$vj4Cirtb?rvA!qAs&BE<_EG+r^?mf|>-(MTYgGd- z`!ekFyL5YH_*i~po^rR>Y_iABCwq{;Y};8!`z(h4YSjsse1-Suk<#)No$u=Q8C|}e zKc@U*byCR30Qi4Ve&c!a$J8J3>h@0w`6sPE^3V6r{`2IIssE>zPx0$lXI=jBium=( z{^|23#)@BUkDWhe{5FdEf{MOR)$OhOXMsKnai}B!(m7O27^Ot>F z_RqD7=4a>kd7s!nFKnNe_P1mA&qJ>BzGa&~O2>P@cpmlNAx%y3Mf$=6UwGgP4}9T) zFFf#RJwRu@*y;5D7aXD$zv%sDGs{JLJU&gcye&EO za7j(03jECDnKBD7e0OY+FXbbbe(C(+X!>hA z^L(SU#wUA{JUzN6W6+E}X?t9^$ax-M4-IP%j^QCY3gGY2qhLv}GSjn8A(GOmX3FV+ zHRbqR^po!s*Aq&35R&~+ODEkAbfl2}9%`d^)bDJOg>*UDO5#lgbBqVFaB{ zy5CK|5$A$7t%-ft5h_be&njpt-tQ4=w~15e&!vupIJ_Q zd{A|(&$4>b5%fvzn{GdD=h6Lip0E9GUCGyJUwTLJr5itnFD<7pxpywfU6RT7bS0WT zo~u;=?>@uvp5Yo#A}5tM)5`p$cqD8Z^MEB|gL-XE?rI;|z2u4|Mb?y?=MJly85CO{ zMGc;T!)c=IH*;xZ*Lvb=&B#TcF?xCX@o27w>3?+j89^WAlh$ERzen4)(hAhzLp}`X z(a!dvUH3C{t#nM{IU5zGZz3saKO#l~c8vHi7W?E_OfiD5Mw~Hg`Fcn{J30JtUf$^uM%k6Pa*hzx^%0?OSS-nkm(~T`YSR zO3i{RqXilq*qIL`YM6G!{hY9^ZF{yXO6y= z5q5SQwB?h?r!ne6bHYxy3`@$OUw6>s9Q3!Q9L6{o#Fu^tlN7$rl%9J>@8_^o_v4zv zC}JDFpzCM+&=`5%{Dc>2$d^2HBoY{{g)ga3QCUxFJcf%gB9nbfzG}HYX3n{gj@Iv~ zD~HtpQ*QziQUBikC0@$z%Ds zD_R41(2#$#>EH8jMD^lSU00Aej}zBqG(k?G{rd4`wUHekLaU@rhcUJv5I9p z{YKM3u1!-u(P6Z&Ge3a`E7Ek4>c{Mrlaj)V*;NL zbRX+CqDTws&vYH@_n%f$CKCF*{l2YDs#;M7}ls4dN%FT|309ed|dMG^pTuG z7mo=0_W|uEc$}HLfG^Zf(%4)cgVv$J5LT%Km`?WZ1Hy?SrG6jK@EI>)pPEG1f<5P_ zuwgaW53!{Hgm)&|P>xpqC#Kwg7L6mz(=$ZW7iC(mU)I8o-xMGW)y3+thp0WIOp>*l z#(1P=`;z)sVks{4_x3`K=S}&&sgm}M^gBXamk$dy8cpbg>|(lqlJ2tn3Om2kz*3j3 zrhKX4u#aVV+FyMW!)&Ucx!EnsbG|G;&2~s}1=b2)DRO(=Y!@{FVMgp=jJ1|GRUgsGxAgA)HXCY2{l z0v~5OiNYjvsgp_fwT3~Jnxx-XaL~m55dxdalYCb)<-;zf2r}f+_(J+G5jd`YQTotn z)b{H{KBjzXzo1h@*aYUViG#_=B>P<;(2K#O`jK2&)nB!lB-oh@HpQw#`3FX+f43=L zYLHYvCE-5RWXgv{{YOlBj$bNIrpWU264gVa)L#?<7%8Ox#m_95E0^aux*6qrP(Cc` zuQug5{%+R_##w05kD^fH=FWC?bJp3|8|uBgHi0~@`I)% z!c@1vte;|ne8PXf#=ujTVpAbsxjeByYdH#O?w_fO%nkQfzB=n8UAbS#a!iL!13SxU zRxKmkSMFbY<#KiSgBUkPqa9H*;T9#G0aAPDtqM%}wng`fCOF zqdM1y)V-;A}+R?Bfg1;ox>MR5K?eieMr9C!pF2w4;&y$u(+0dfBk~OeF zP4`Axp=27@J~t&}##$J9YfD#KeGKX~hUjmGgvj=e_E3HF?v3;Zg)}|Ag1B~?6{5d3 z)DQ#SwjS&|>RUImb0H7Eetma)DnHa%AFB`9h4HVWKD&H6OJf}J+Z$WjuAi4GY2P^V z8=E$@G&Ds>?du!x`^K_-qt05-+{qCmFSFf2nr*c7M^}P9&)d&K%chpD7Q|9$-R967>;&EEJp3G`rsHEf z7pCG5K!qGd+;|;2A?RYm@rSGcR~+$a&!z>W^U~T+hs~TE@^#?oV4_I8z7ht+q`6Le zb-Xx4X^~EOIv#ZY;o6jPbhgpdb>|zEr#pi3Jo!4_=4mx_7Zp++c*Il6e>w&ce`+&7&t;yexh1rv)E~x` z@(r%D@Nt^3izX^cb-9;@|Hyj6_s5xk;{R6eck{kuOP-3XMh&q4(ZNl(%kx%$*JsPx z>d42dCU$Dfy>x9#Uf{{xZ|D7K@XrdLD*EoE?~r~Z_XPu`ORnSoLbp*&g!mKV$#XKr@}C~s;tl+Lp8JPW7!VJwj7r-M21osGxA zq)0ErE{;WK7XH@qsQeLtM#Gnkfv*~aUq42@_m3eT-#z0r!crmFM~UP09o&zm&tHte z-#UgqYsS!L=@|TJW61yU82mTKs26aZCL*8q{hT<`E)x8Z0bBxpA%k(f)7(!1f1-o` zl({GQcq1#7cA7hq@8mCsy_85!k$tQGU2XVM&RvFonUK@ykYl!(TFzDnAN#RX+Uek5 zoZ>3={1|+{i38G;#wAWg=8%KP72|V-l_iccXKgr%*M!x=$M9^$6^ZjPX zX8eAG=U>%gJoAeMp7qo(H9O=pKP2$1XQ|<{eCC%4JnK12;F(_`@T_O0z%xHA@T_O8 zz%zfjz_Xq!1)lk<1)lY+6L{v|De$c4T>{ViW`W;#51fwFD)7wj6nM5PCh*MXpC{&c zNErFu0?+(}kl$` z34v$+Nr7iQ&k8*A2L=A@J$8GY6L{ti3;c4U&qo5!eEnmv5XtfUl##FhCqwhKJ-@s_?$9G|;4*z^+u&-^C^p6z-{;F-Tm;y2jx zcT2qC?-zKs>v@4^{tE)n^Jc%mGyjmlvz~_qp7}>4euJ(5QHeMFHwB*c91wWspAdM~ z^Q6Et|E$2Xo`V9<{Bsh&!Pb9R;tgN_@d7|N9#~JmIXIEykNGvb6rS~@ze`1l@ystW ze3s98@~;jvp7~QH-sl+;@^=}2slf9(xlG`hKTF`(n(@3s;F(`7@cRsYk-#%QEb#2l zT7hT&a)D16`6~sU`Ktx~gz-b2z%&0Yf#-F?T7hSNqrkJC%>vK-R)Obnrc>aV9}{@i zGcNGVPYArv*t=EWnZI4&*{)uJXZ{lc&vrd2@XUWo;MuO70?+(i0?&5s7I^0G6L?-H z>=$_EKQHjC=L-VQ{C)Gk-wf*{%};&-{}D z&vu;^c;*iZJll0n;F&)x@NCyd0?&M(ewZLVesUa|A4N=!hs@_cQOe_CuZia(BZtRB z<`)b3Y}ZtQXMRZFISxw&p7~`0&vwlcc;;6Me5ujDTHu+#NZ@+}i=YbD;uUoP>6 zzgpng4|M|1{5u7Hz^upa5_smX6?oROQQ(>1Ebv9Gc7JIVc;5*QFI)UeT_D+Fk z{#t=&yBYjA7yJ7;*{J6lgUHCI3uJ&U7R*5(B!h;fT_}c}Z z_3RaR=073uJgz<|@XUWo;P;vK+9~kN-!1UGPS`8(%-<*Q9OwH5p83xUJjeM90?+)z z0?&RpBJj*VD)2mSzAo_0e?#C|&o>30`2zxf$oTVwz%zeP;91Xe0?+(mf#-JoNZ^@| z-z{^+GtZko!{_mY`F??CyK)7d`9XnayXem`Qeyec4+%WmRVwhzFB5oW#-CXN&-@C3 zXFV$gp83@R&w4Hrc;<%%p6AV4foDEHkIv&si1*6^&-^=ueD=d#0?+)l0?&SE6nN%0 z3q1RwRp6Q5De&xvn7}hXA@Ho{R)J^!g96WbZWnmw_X<4g`GmkT|4D&oJ)aVI=I<1E zZpU2$&-~p2f7JB%y#mkt{Q@5}*npQ=HDss?1#Gqp80D9p6BsKfoFcRz_XsM0?+(Tfe)E@ zjtM;T;{wm)2L4QqP8>hXPY8U-^vA6N&-`A2XFoh4@XUWw;JIHsCGgDODe$~b*d_4H z-!1U$x4i<-{Cxt?e%mkb%zs|s*>5ihJoEbnzTd?2A%SQ9>jKYydqd!v|E9q6JT)Nj z%s(OU?1z&A&-}9j&wdybc;=rIc=p4vz%&0Nfv+&}tY3O@tlyZQOE1G9@%YAm;8ztn zo|%tdKX>8T55 zAC?O|^H&Ny`(d@fGrvyY*$;OLJoE1oc=p3ufoDFyf5`32erOhW=C=y@oo2l16nN&x z1%9oGw{C%FenQ~cZ~UGk#}D%#6nGxTwhQ^p?-lr3qt6op&-|wZp3l8^3Ow_72|Vk$ zTi}_$SKwLCeFD$?{Q}=>^nYI9ncpw)yiPkL@XS9f@Eli11fKav1)k&Tb%AI88v@U9 z^`^ix|D3?{bBx0R&-{-BzSo?e^TWOzKg{=;hk!Z$*)IJzc`=ePp85E5p|0_5zmXp_ ze8w|BB=D?fslYS8OyF71Spv`e3V~-mD+QkU)e_I^J&8B`ZbMc7c^>-z_-Ly z-5s~ybkhp`g6r)~u{HHAT}_QQMLXJ--X4ptu8+pLRQbBDt_#A{Qq*31`^q(S5nEbH z&{Ak_zdLKHY2UQ0DF(?+^=-zkmUU5B4X$mQDsS(IHI*;B?aSA6#p)Y2*r=OeJcQQ7 z@T%E5R7PAIkAg04V|}fDxzokBUE0(fSx+zOs+BA6xV5&Wz7;Q{-rCW)v9(E&ha-(P zOS>h~YU>s0Xm3DmpW45XNMo!cYP*@`ZfwWa6rve>jW&P#|k11w|_)Hi2{R#8RkgNRi|7w%?i8fclFy~6wgCd@MLoaMl zQ=KnefoHaZyx*SLV2p+4F7f^{JYlFE9+yL zuLmKN%9Cto>NMq_=6$ruW?Iioj$6TKI zPaB@BKV<5c`f_<&I&_Ee`2W1boAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi7z(=Yt^fb4>VH-Je^s|`f4OYsa$VDyDjIu*#cB%5SY~hR-otM_ zEESY))>AWJtNVGj+xCCxg|^SkVC?XaExYd>yw2IO=l%<>=kfYi+@tA4MN8LfyM7S* z?KpPz!wq`qgDP9-nvEU)e>T>4w1XW6?U9}gR`|VaPydG{?5$tv?C^}#o|?0^$$dk% ziPYDTo^)3De|1m)Fxu&w?X5bDJ#y;l!OHUo26w#o^x(?VPY-@m_gn?-Gk6`IbMo8a zz7xm8k2=2-K7Qatcx3*`aNqn>;r**mhcE3v9Uj^LUihi`?}i_0Js&=P>fLb4%!flG zI}Qv!_4Lz&sq>eF=Cph#d|}@4@V@tc8h*<8X1MvprEuTM_dr(|3LQK+xc$U|!F}_O zhbvE?2sgK#4)+~+Gd%LprSOFvmq4>PRN*{0cwzn%gZoyapZ(tn&uPW@r``-#oWB&_ z{vM6@$Dz=CjJNtA`Z*Dv+xqSB2=JBXPle~4JP+QfpFawX>_0g8)cmIhL#IxJFW}Yu z-kagMCr^eeobQD%%s&sBWuZB(PYm`!7ZvA^qwf>p(1}ywO6TctX#S<}Q>%&RrJ>M? z1A}uq&ll!@JG^iHo8Sw2fb4yECC~Xnr~>%0EV`GI5*htfRYU<%rlQhrOv$jcngxjLC!l?a$6ZW43{}Mj1MPr4p z!v{2taFhCmbjPFA?kHo0uL73C+lan?2tWCtswDK`hGO`g=jydxuN^b`Z#;kez^2Vr zK1u$kd~X20_X>RTp6}h+zSnaD-#a>n?;W-H9{Hs5zX6N?DSy3@4_@2#!%(DT@>|0X z=!Y--Q}I=|ouys;Rq@r4m&*Eoy_WT-Vt$;R#jXzjP`~;*{CX&xT|HYe>F`j4cKGby z>eq(%vwoUKOdEJ)1@Mt)S%2i8S^uS1^sDcbu>N!Rv8(^Gi1nXOV^=REv8ySdEy7$I z-kZ=r{7iDXNj#IF8?<|^bf z%Egk|H4o&u@V90CBap8`$~6L>N5Jz4cqSQ1zKh^<9QyY{CgE7#e{*&_(}+3GDDC=w#%jq!813;>zVW3kM{RS~> z%-9E>NLNjXxT5{XRipJ~?2p|neXb`&;}iW1&C}1vNQpnjwY2G1KCaTZkGV2ES5xMB zLaB?3Le4J~hsZW$6S5Qa_YU-W4*GK;eg={~{X5YAH*sBqS0(Oya6N$Q3B3L{?oZ?T zZ!|va_0F>iAsg(f{LeO?U3~}ocg@s8F4*TH{FdzWYuJEnegQT=yNLAp`0^px)nLb_=|Ugmryo}pg{~zP zUm^S5(}!6f&%t@%$NWWaw_2}$xAq@2=3rm$P}T{y0xt(T-+`+KuOz(E znTv)URd)CSZN8QcRt-!pVRg(rJr!47W47@&O}8;-ABQDR7MqjVar zNs?`;7dL1Bv$*OzSfYn5WeMPD3Q;cTXzJ(;bb9M?r>@gx-xq@c>hcQ<>hc#auFKEM%Ue`X zhkpf&^B&G$vb3%)4{{2*?gBA7n!EvDfY;x|C!Igg-r3&S?MHLMwYPP2dfPhl?gIC1@cLVQ zfrh)n%rFg?#cE3qluIW!pb>Bv*ieaM_E`?SYo@TWEf?r|29`m^xE8B+9U_mAnaGVV z_tKDiX~>mMe-LkbTTMd1?6zAow2(2D-MfZK3> zVJ#7qTnVImD&TuJUDHzc^z_VTj>w*#*$#fo7{TXRJCSw8v;FG;h_?1KbO=IKypG^S z_0{VuN-K)oCD{Mt97Pm5Bq7ozcx@$+zcUD5Ea__Wy6JSGxy9Sr+}`HyY;SMrnCHgK z_IW#e?)*gy7SD69r^|xEWvsZXxuwzF(cRJMYjwAFb#%HLeOsH`eC~BsRV&xltuL!t zTT!*Tu59DFvel(!rEGqcdy#v-H_*`J?r6n%z{6fQh9+L=dSa{vRo&emw(80(P z^5{NY+~bznGh+8QbDU#qN@ez~cGv}NAGV9W@1xNfJP;>Cm(!VD*6n2J%SF+;jYc=r zb>=qJ>8#3#r#`7}L$|i65vH%iF;ksQv(HK8HYK}KvCqc%Fp4V~w#vY5C3u^b%KYfw z1xKvL%kBaV42J*wWuGp*%$Bu|*;5>hyt~JS`y~>-14mn}cx?j$g~p-l$MSkjUO&RM zln^N77o%qzW4*hy2>TU~WT2rmmenov#V1LQlO5lg7~uzSDWIr#vITTVrwZs(11yyW z_g7!Gv0r4f-u?Cny9@v^-RNX{_9fc+xSO(5*>N}fQMZtRJqbMe>kVQeNf3=+)~&NG z#Eafz6a3tV3d&5p&N64TL_e6sUO&WU1ruz)nexUC zULVS~7Y213;Ta=5THIOAky;m`ME(9!g^8h@Nr!B%%jC zH)x+oXPu&dqUY00(4&9o!$#uxt3TlU=R!UMJ@`qJ_5wfXkLbDnGCDQl2X&p~^H+y$ z#BWk?60FGWo1pN5KFkR33OFYbzo};~M1oSkjGVNo|k1HU2-S%tf1O4j(8*>t#ltA_k&g8>K2EhMN znmv{L84IO<3-kkgMlg3Kt()*K-aa=7a}3vNlS#h8Y*zmK5n!oc{0l_?;LIzLh!&%t z<0Sn>qmO9rd{p zc1hOj-_vFW=gpz<<`I9e&z^3_JU-vEd z0=*sdSlfvHn3Ln}s0(}v@NVGUCcGPUfiDAo8t{1nKP=i$LtWs%3VbH;9)X{o#{17i zUEq_oJ>4$gv53<6O#<&iUEmXRAau^$X!&4?alpasMa(3fgSk zmz2K+=gc8-V)$9JqV?DKeBt)s^AGJwUv~7*`FY!>muEf4XkCp&_Hz4Y!oJZ0`XW^) zd&564pL6+qrTP4eG*Vd}qk*Yub_4IRY#mS7`)FN#AsjBK1w<*_D3|l|Vn80x#$DX#GpHY!CgaTZKMoi!JZ5Dmg{5I&D&({pD?+mFw_J% zz*#h*^(*I3AivGvZzR9N;%{~yKkX#{DO%6$B7dOup`O-D(O$*3nL!7qcf|b7!SzA@ zc23hr^C!HOug92wD_}oS#Qd|yL$dD~B9QBMVQ~C-vOgam@uECAm)BH0=-q{-U&Sv! z$*0HU%W)!JiqA^#gVHRGmg0K)s7@ zWC8wQH`8g?lUOuiJmhDi<(IUv^-1Oj$lox3<`X}vpniNO?DseTln=mvDE_`o4oU4l zAo-8R;yd&Sc`Wf=H^lesIPn}gFquT^R}W;y;`Pu%ZAHLNk=qd8b)D?X<2&*n_%l8_ zCV%cpqV=^rSqUKQY4$(Tzs(q*?Z~zT1|}^J53kkL|fZNAm1{jAT2>pU@ui^YQq6;dK`|pU78UuY!DH zJxLMt8rnnMn#yMqGlI#egP{TPPwFoP!$tA-jNmxbjd+^AMtlP5MtmaDEh9b&^%&l= zp7Hg~?vn9jO#iN8Sg*N%__xsh1pcv)CMOl;ADEwKQ6>Ar{>b+;$sI*JB>PW^*&qII zo}UXjJ+~*#3FxyhI8CSJmXZF>`~|IF(-PU`2+wbZro+FIGo$`Bv=IA&oJ@$10{UUv zZ#cOBBuDorSvgVq^R(Y^By)R_|M2~Zo6~0zJ@C1~WJlDWkl&^U+*xk)9wB=2r`-;| zK4QEyqK7|a8vQLdnB_LMFWl{R(|qIf+#hWYSV#CH#;5jCfBZ0wJrm|g7C9S9BmN3| zK`+`TQNDwG+BWb6c;)L`nhW`vJK7)QPY+Djr+_}$H1|OvVJl_!uuI z3w6HVM1QbvCb^5B<^$v{3}$9V2i=`~=ivc*x8BmFolYQr!g(gnn$8&$(md<9u?7_;R7j z9M4`v{Bq7xCnas8KPj7sIIXcrcypm--y4BA%1J6R{mJ z#`DBP>R-k4grJyBtOlbXf7sK_&;W(bN3j1o)QzQ96rJ|x*dL2SOxhp5!Tp`I6Y&?z z^(=X@Bp3Xp2lP}rF^Tk${SRr&qxpis|1`**f${O-U%lW*JO02&yc1qeV4NQUypH*@ z+kvA36dTryQy71@u!q0?{c-G<6mMVS=X**&w=56#>wNtnf8*=(8IwJ&`I#vnf&C5y z7!nj|KkIqMW$@Q^+(GdW{Ed_QF;l)n`HY_gru>8Q_vB}ou#nK#?c_6gpK3ni=iDp( z@_Z)hf3bXqpHG}iv)?d(L43Pz{_>~rKg74sI)53RuQ-xl_9Fch`3UBl%R&Cg&sX4| z=6q!#@W%C-)+=kiVpx9>FS4*+V>uD=9{Mw_=UAUj>o@9ISg)h>Xpi+Mw%;1k%$mt zN*2#o-0|}j$dl!!_#Vwyf`)u!biRUk>duPhE5p#=C(lQ+Xnp1J68%~75yVSVKH|W5 zGu+_!6XYXVV!@~QJSsoBbMq1MpZ}2g$UIa08omFf>;gsZzYheNogh5_!1;Z7*UMwZ ztD~lPWjJ4l|J+WzitfL$zM1#i3-S5FSo`hWTpYDN@%T(fYV>(g4#kuE_!)ddy_MoqBHy3$c!vBaeYc(GPuLHnvDcBm?oLRgc%C#d%|ZE4VbD(DI)UfA zSpUR%(a{YoUOYp*NdI@JOlsfh-b7Pie!rX?$+a(X}@8P zUl`xC-+;bN@r>im`w!E819($>8|L}GK%AKA*$qjk_ks_U8}s{W(8B<@xp-3qe2Gupd8!&l~KM`F@=H$A6*yc(saOxCGe2bm`SogP>zc;bp zo??t|g~16r!v|Pme@OEqbHYSeNbJ9}vH#ARK%dVU_TSTJzi5bePTG*f_TNtD82fJ= zs8G3`cvp2B`|otDmS9lqSJB>udUQXEd~Jg9yb1UW<9QVMe5f_D|O>`ZVoF#?D{=fb;!Qf;oRhep|lp*5VuTBg6Lt z(%33GY~sa`&(e}0_tW{EEJE>+zhV8s=f(WISDa@U&cly_=qAp?kKsHVls$>><5;CB z&9c8v)3M)YUjqU6XY4<3$R9orhd+q(^(^}Qlz$$n(+B2>R0qGSEcTPXqTfa8CpGH1MfwfDX?QY6tKw2iEh8Yrp^SN0G0-)K~r6&42d~Kd&73 zSB)QC_?yU~LN>easa{vDyKipJvA#aW`t7yPj=cZRb&;=eRLYu!zwFED?dwZnXMX(3 z?|&D0+s?{r*UUJkuc=K~Q;XdQLi$f&{?vG{CfHFgPAyKPS5d0Ru9a_|EO-GmS|~V;afKBFQZ#CeYU=qR;)#!O!0WPMB#~#7r7SjP)xk`QU15Ybgyg>k3K4opRqNXmaIe5Q5c7iJ*kME z4LVKoSp0!gaa)p2_Kvb8L}uJ>2EFS^Lhn(zX{0##sNAmj_^8}&A$Pn!qH=)|l2u8D zAyEvH**_J@?4v!3EMekI(nL9fbT{H`^c*yEZqu!t$JMzmIE#*oPK>q}x-KejONlpQ zg`SGhMq}2|ssT0VaTK1oAswvI^;HjI?zy=K_0s07SM%6_RmI4UfECs2DSD10@ppWs z&_;O|jjdJ`>xvQM>3SMe{l;H`V(s_pR+(LfapT!G>Y1R6hgVjhenv}e79G#p@jS=I547E}F6)D@FNk3n^+!)PM@ zpiZ}Z=>r~YnCRcO)R$^zyvI_nw$y1n>YpzW=zpU)(IyZd|Auw)h+A#*N$HDHOYOJX zN4z#S!}}S&qK-A%&Ga-@A69(4K0iBdtlW)NH_pUf8H2c&3AZ9YzoB2nYp*5=O|3E8 zo93mS6rGoD!GN`WJZ)QPKJRs-&h$7j$jW~_dX)!A9s7}%n%6FK{Yl`JS;X39v#eeG z+roym%S`{arTtQ5&Bp#bmU^|Nj%@Y%YnO4g7rr-Ee7jLJAjgqj@_lC4r=!XPDzthA z^=w0(@Koo@)W)SE!=oL5O`8EW)@XW;3$fX=hjgqVFk_|1TWnWAIh^d(t$IP7dYyqf zbw#?SQFBy1w$HHXG)htGI(w1pv6Pz|jdGE(bQ3REAx>{47jT1JgN}8CGC*R5G(tKa zmBq%%^-SJt$@8cmqxHoIhLM?dWH(3*_RTh!i~A#n{aO*`iGgT@mBX+o&UEV4L*(0t=z`z^Y%Hk_Wp;i zn`Ua_=+i=gGuQqqI;)4E; z-d}S5&1ZR^x3j&^yM9gc$+%+^u|5{{_Kw=IdEh~@n7x-7qn?_zZ|VB#QtY)EKl|3M z!29Sh9J_5*#vFlvR^kz#VbPJS0S%@J&2habh zhSYBSQC(gx>`@7x@cf^bmov7d9DiGm*Y%$NLkc2W&^6w(eF*+bdPA|{Bq;jTI@FbR z9|4>n%yX*izs=!&@S`CU(J~Gn-Gg{e7k6tDy?9Pnh6zTMCo!dLOg!nuh!oH1iuI%X zMLnlW(V{so;yGQJ3LL#gpVK8Fiattkn>RGm`7xcPy+_s?q8}&=Kx}0Yb=m~PYyV@} z?n@TcTHE7~AuVTIqTdztN17-O*?`tN8cVUGG(`5W_DSuk<%DmdilJC($uHz*;$lga zwBMKQj~m+3tyS;jKh)l1VWUx++zlKR@;u_*+P;F2_@Ct7=c++il0AxSXGs;iB_Z&0FJm#H*SY)^@7ykq%;P$3b#F zDLQpk^GjV-yKO~l*LM_^6^T-^s<@z~t6@=b%cezZYquA*tn+SY?Ao-kCcmL|L+y^j zHSOM_l2s+^i<=r+mv?U7SkuzrU%RfQV(0RPf+oP$))Kz7W!aid8!MaYH>}Uwys>hn zcSGThmddKywL8m~w=93Ow#i?!wq->@?V}YtD>fF?v}|r&o?pL0v@LE~)>6N*IM274 z_q%dq!Sb&9)@6C%!Czmnq;`Ap>g|miYdSVbdU*46oq<+WXnD_VA9j5hF5zP@~Y zS3|+FZW^y~LuGq$#m>_DMeFlg%C}WU$1UHs+FP`&y{vrOI!RN$scn7h#zpJ5-xOUz z{#bO&H@60xHZQ7zK3dC3=B=Aaclqt51>4qmi&y7wYTZ`0y{M*p(}u!8+%YS5ZrlXE z>&urv-q2mx40&6itH=&z z{pOnXl1<*)T7S*5mXfCS(hb|Hy~QP~mY1*2-&|hSwW+nGYjZhXi&k}QUQy|9Y+c_C z{Z*{VU;lXHiprKv8wz0Wn*2>|YipY}3wv!ZuH1^g^BOl+c3JJ>E%86Tsd@1u%HPzr zuC=8(P`4v*N&eD@>l+s}EGj7U;oj#hD5&>(^Oh`K>RXz(by=ODYiQo;ZSY<1Csx++ z-&UbtRl%>F+)gF$2G1tXU$l7f;zfCROF7MQZ%c=-PP~`Y;s<4X-qyTe9%H+5?&!xS z=lqd-*O$HoM=O$8%vG#Zk({@kSrln@%K`!|HNM%4=a}D?NaBtjWK1mBMm3682 zd!E&g*#A87zd62^`VHp`qGzISr002{@n)=qF9c{o7D2dcz- z-t#)%H5$b!@t!j+=y^mO)xu^NS0M#`j2|;x?t~N&S3z&wc?)DaE*XZMo05ydB*B&c z3w<&6O~|ESq6hI#Un(bhj|3{e4@05x=vQ3_{(H2KM^F2kc<_g>1OEZw#LpRS2gcOA z(yqh*^GLjizFL~#H7N&;8E@P-q`Yov0yV#yK~Mc$mT=X!K*H4~wN&<>Bm1eACR2L; zd&xhZF2#$QSIk4O*ju;~{ECFzC0xlj0sSf8A=E^PpJajON%$1NRa*jmQNpJJZrn2NHo%Rk;xP%Q zadDa$Eyv{@jcbILN;#>2#T${XlyGZ58zp?Yr5~j)>c@CW)MMzMo^ztga|XDP{`hs^ zR}65Yytz{ERz7D-xK-W(X&|e-uS>YKpPxy%Ro-73`Z4nVkpXU$H&f0Z8rK*j^R5Fw zBH_f3aW#8R!XL802MqK^`@SaOR{rTJI#h_CDdLvDFP3mMuJXku30LDP_z?rWaop!5 z+&b!50~qR0UrJ19$*&B!IZpiPR-aC z3~-}dO%ks3py*p9+$!&j5^j~ZRE8WiZk`->r2%dnHz?t1Tt&ZI!mZikpnuh#q7;E#%e_elG# zu)x)N@3+9!de21K-BDNH+~y5*Gwcx>y(_(4Z4FJ2@}Hb6!vmS|>yG?4p6KTq+0E`e zsH3{(wr2dyV|_=*tu<@wY84DM`#RXzn;f}y;6E=pHuYn^KnH#rLW+3A)#P3hmcXj#*D(KPB1a0(|`W|a;@R{I3d%h<5 zb9ZC2xE15k&sWs7(a+5#*-l9Y0k zAX(7R(V4&GrnPa0?1gXMxN(0l^>#M!rZ%5^H!tR6hx+Z-JM@=b#RB^Y-}fSj$W+VJ z(-oPZPL=tGLds=YRT=H`2`1BmDBL6O&d4~X5Gr1&`YhZ~%H>f2dG+rUH_yp@j6PeV zcqhYgGza^80BL+7_rVwe=X^B#va6)(qXLCV7)M@qr|HDo_DRa z_g-tAb88Dz0N+jZMbfu*W+RK81Ec3ir@d+T=w z-pXL3Z|?FAys^zU@UD;D|6e|K>O>E_A8|j+rm+>@$PW*`xsLtp-@WYqf?#C(IbZIn z=Y2Vp*Js&Ow&I_>;lW{~^Ll(gYed-#gZDmr>YlrwU4;89pT7Iq|LqNrB5pta?(cZ{ z(UDV6pB_2z+M^?n;J$FnPe*Qi?I$CPaPP!@(|xawY<~3o$hEj%hkN=JR}TE>wNF2L z|Ki= z8QJ{I>5+~@&y1Y9=O-iQ|LWC|mk+#xILim_y7%s9Pc6RZ*#VUO@{b=G=|Gt;Z+T{9 z6UsWZnBrVDaQ>0IpS`>D-e(v7m9=_cA>^g_ z>)84T^#7}og`Llg-1x{dBLlDeX5{XBDbA{a^Pj%w*;9yfSNYdRN;iIWB<<0sN78Gb z9_fJ2FCTajajFK$R+}IFvuE!rd1R#Y9+Xx7^vLFtACLTK_|=iSPYsWxEnG8j{@Od9 zEj|5b&z`>)WtD$*V~~TV6+;wF8^(`?F^U?m5L}6rSeEzqskqyS6WU<*&NaZqGlHmJ->X_T7Ao z+BNKBD*|XUk2kV?FMMpeCp>!2H~kdJn!Id<;}4Jtf0zz`6F$-8VJn`5Pk1~en||tf zZ{S)=cY?7M-$hsUbSiTq;#}`i|@x>E{FOrYi{y1d#qwT-r`Q-Weg9p#ATl%wiZ}kkkbJ#oZ z?mBk=`CSYBZ}0aFTx|5ik0+rYO&Pf6*S8%1Eqwdr_p1j3;m|zh_h-#w$F`sO+sNQ| z4unU)c~y9Hp`T^F;Y&I7Up_y$!%p2YFJ-i8R?55KZEyQKa#P-I|ABAtE8O?tz60Uq2>)~7?ZDRp7XyD9cqQ;U;9THa zYoNDp5c-U^zYTmP?r-A04EGV-m*D;y?u&4L8TSRazli&M-2WN(eB6J5`)u5w#eD|u z&)_}{_or~r!uQ|BO{^l3IcXiYA z-@Cfmn|-?ZHGF=VK4+b7eiNTZ=yT@j=J)XVZTdXcxXE9nURNY7u%af9pmP3Z79q(ks@^FQG8x9M}n>E<8c^Y`d; z`swCp@cAkFoW}JWo*bEa%brQ2*2gD}c5I#WF6qV2p19(cozQc;ty2kfDuqs^(5VzU zl{z*fUC36X3)v4m=i?aH=MIz=rLqF2oA1TvyXkZ4>E_Sj^Ktr|a=Q78`22bLJW1#? zGjjasoBq*`*Zk0BSlZ~^6-|Evn|Js%2Hq}J04u>E2 zH*NI3+nC1wuV%)sKaY818s>~0(e;DZ$gqV-@oQ!LjWYg*IPuq?-W0Usuaog>Wc+FwpWRxLaw*>!NB&jFtE7Bu z9QijPUnAu^%J;>Qe-85N;^g~c&0xAG`yy}4 zLvhl*2l?$%eoGvA?mDEOBGTUxM_vy3?NWX`j(jVnpDNOSF^>EYrI+%*i6j33rJp9! z|1ysJrs~o3+aWjoCW>%RPHgzS6hBV*=PCa5*!W+j_;JEdQT*K4_>WTjIN{%>_%mYT z{{zL36aGDlpBEed2NXX}_)`>rW^DXtD1MyqXDR-y*!aJo_;JGjnc~lmjsGIWj}!hf z#h()!|22vqCwzqB&y9`$CdH2v{x-#*7aRY*b%W`*^Kh+SNQZyWTCd4lbbpB_|2rP6 z^=K@59Ap0*+csiOy8(0C^#gD3Uxztw&A_|2Ru7z?|HDpNr;Q!+VJ;j%cpBvCh?jx* zLCk?C51hw*Zy0mZmu_Q&FJ8q)e~WqN8yVGu|Al$zo5#YVzdnq)!Dl87yoLGc#ovVo zM=@XhkBsz%Z)5)Y&g)wT-u+7T;P8U9h39`79v#MdU$2H*XPXYe)56JK9fGx*NynFIfkv3cRIKC@xq?H@Ae8HWDh z(a~eb^S1Ed+xs^UT>KpyeCO~c*rIM=_-np_;bk?0=Y2JUZ#~XNFE(B~@a8c#`d>Ys z!8dnZH}FP=XYjY*Vxw&d-EL zF$NA^m<>FGjSge}IJ^MiUxyvCfd3Bo?;y(p{yOji*m)7f1^x{1mB7oqgXh1pZs7b3 z&*<<^+35NCn--q`4C4F>^Ppos+S}p#Kmmqz2*&R;jN8$l;19{3*CVi-3sHu~Uy ze)!*{fiYj^&jQFRy{vj5xV&Z{bIm%Q@1Hys9?eGHzx0I%A3HYhqNlogVIVDx_0rsn zC20stU9<4yy8JUGh;!06=VBed&%L;*pnBoJ_vf5B`Tf~vHX(jHg@0}JKs(a>2!AKn z%{p@@g&muD@lK9so;k^J-o=v~=bd?y;~5u+QLmE=$`=NT)(@m3tc1cBtsDpxZ5RkZ zwutcer=JNFZRGL*PoKkeI`#GC11FD7gMp&WT({vCAs=2MWxGgrt((49reA>c@O7U4N{Vxx6K4(bzl!o*K;c^kUWT8} zLfA}%RpEXm?t!AM1G)IT0)K&`Z38*DFW{f+2U2jqf`9(VfFJky{PXq!FYfRe#NRPM z{!@v+_TuXYrr=(|Ki@Erg?l;wymMeO?q&S*u7M2P$&T&C4OGw3h4?%7fg>Ac>V9@4*ci$=M230Pq_ncf!7zo)6pHC!MDH0Mt`@?Gx*Mzz*F#kcmev24BVFy zf2&7_A0YnL3=Yo+Mt?MXA8;-3G+?Z2hyN6K7qAZ)>)zqpfLnln%g{!+{|0xgkB55@ zelzgD0AB-qfWepR&kX;V!B27jKJLfrFAg^%eBDRR4F4_Rk6awyMR@y};ja+hesOpk z;T>ntuK@43I9yBk`ZL4F2w#73_!?l}4Ht*ezn}d64QEi!;7Q-ki^I!+kKw+A&&e?7*ab?6Uj&@WV@fBu2iCw_ zl+nWTMR7xOq@D*@N|;r#NziB(#Vqe>@9FLCZQhGd_WkHVf%(0$r~Oc?#mWyI?rkeS zbW?fnfsRA_4!gMJh{QLQJ0DsOAM9%GW#zqXR&TkeT4T$R-p=Npp2maA8V_2H``ZrQ z)F>s_w;k#^+`Xl_tLNs9TW;xWbMOv*>_rc~~C!h7rz0DhWP|fb^H#Al@?mcWZwsst(j2r3CiD_ui zSb=(qdVK(@(bL}C-PZcSsYrKQPh0m*s1aJJyRGvB;cci#^9Q0Lt!;ac>~GxH-F&c3 zw2m3jT&{(HW-ZN~o%_3+yV^PFy1Ebd9wx?f)N=SxPjB;~-ii-^`?fT99c=4v`9L%? znvK`RY)c*}yH4JKYQP7e4ejdaKFdQtFch}imRBfI1Is~0?ThVnJz^eV&sfI2e>e^K za2oPIZCV)X`bCY6n1f;}-`v?)(FoLXq_w%R=it%`tFgCbf7_vo6^;9kGpg-cL2KjQqm4YAshKaGSmsBd2(oPM zMLsC#$iYKBW?>&ro&MXcIJB;oCbnk4#HUXG^VeSXz_R0D+rjd~`}Xy;^;WRP#zQxO zA!7Q!W6OpujY0y_HFtJ2_r!@{)32+!ySJmamYLvux|mP3t#Rht?hL?t(9gi`<3w(N^Mb9|d*wTHyOSjI$xyb?WGBCUS>* z4o=@ti?@ZAg%&q=x3q_P4mK{WxUxBfqDfcUcf&IX zL)QcB4ut!c1Q^atF*+dcpAukj9e&Ygn|bt^;?U)v>VBTc_Md;BAHIlmSNcVHGe{%G zD%l%8c3}$hOT3g_&~OE5e4)URFJb32T+W`;@M3mW!)5GA4KHFt8ZKpLG`x^Krr{EH zQo{?_2@Mys2Q_>J>(_7*yHmsSSwzEy%+hcH>(X$DwQD$^HEDPrYtZmqRGEer6}=p*#)nPr?Yb!&S7UYJdHgG?6&t*c1Dv=VJ9`5 z%}!|Xvsk|-&twq|Pi9>j4zeZSD!Q%26p`|z^XKy$||(@DJ%r++HVre z)-d{F4f|Nc!!CHdw!fpV(6EO^m@4Naq4?YUoRgkvGypcuPmpTMah=4l2FC%3!xGm^TrY99#7z=6NZciHp2QJ}n9c1GO!eg$11050^&?y{S753S;re+3 zlm3MJ^98o`IS~@r*5?E!D3qVA&xt~TZGBG67ueS41SS>~-_|F*OyGXIJc*ga;VXqa zz%fqoTqSTe$C)gAjlg*}UMFyf<)+M)6*I1aJ!Hi4^byhGry#Pz!b z7Wx#jvwH*<`V_MIj|(jHDP%*<0tEc8iX7w!~T=##>R?h*Jw z)ISC7&+Zc#Dg(p*;6CYJ$L)jsB(9fuNa6;G*`JH}O%jJCZnx7*+-0Yi*s|0Ag-9Q< z(@T7(onB&FpCO5Deb^U7dRw2c#I`;WiEVv`B)0X*{-Q|V&&vz2CW)EEk4YSmIQvT? zem2LMtV`lNiO)(L;&>{nI3ePfa6FIoOI*Qmh+UAlO5*zeE#ikIJ|VHt2mT_l(5H}v zzpUz0$oeG~`V_Kr5(|CcuU}F1!MEBZ7W%;dB^LS=ve3h-J}InAVxbTGNn)W7{P8bU zeNtGH#6q7G_L#&%ANb!_Ree&JuWCB;&naSE5cOYyv8WhhQYHFF_-z;d`swchtdHP_ z{_G%s$nAl@(w~2xAOB)poO#jbPY&xN6%&u0%=)5gf;zybE;HWut= z#neDu^&>vy=YPmbn?~!Yu@uw*`3EglOg8PR#~R`nS*&*ENg%o+KK61Jd*tSnG1dW? zj`)R{Y{OAVD1jGpq+eq3<`?mSAF$F>B0yaK7xLNQQF|HWM|mE|vs1{?xcuoXlFP+D z$$gNgq(p4FNG})PkiUk*rbd3XclDw)x`e(_E1ml>4|;wHi}t1PbsI49&qsTp_A*EI z0@*hmiU?RRMZ!$1-1ggBtFrftlSt zo}6RkUxY(TA(FFuQ_27DIXX7x&B45du?#ls;n;`yvyJaOil`@Rd_k(u3m#Ty+h6j{ zwD;0pVJcxS;+t;I>rj3K<@4)Bvfm`ve$QsHJJV3!9B#7!d(ngbXVO^YAYxPb=ufrw zM1A@#8o=EC@*ay#qW0+uUC3b1E@8Dt0s-D$WN-9mKXmj*en?+?)V99x2Q&ZLBSAg= zCxq%;Bc#8N>?1C3y!__gYR88bXm1nO&OA%i#U;wu?D@}GY|CACdLe()bR^Evr{5|IL^NC)puyhOx8EuW(D>rw;sBc0!O^}ejPi;Cu#6&~Z^#P+ zL~uL5XkURVeP$mawiV4N$TZOCWw zjX_&Zn+_&@aZ_Dy1CR6pMT z5KiIylQyREd5B(6oL6-6OULfAntuihX*kH{X?U{mcbT5b=lVJ2 zXX2b9E+_viM*i;NY?iBFN>BT(qWvcE_T>8W{8BmQ`g5Gdagh^0gJZ5gmwPx4IdUJz zTz@VX`E$N1`Ez}h{JFjyi~PC19DC3oz2Wx%=#P$DW79zgj2{us=liYc)2YH^72x*| zve=(mIb(7H6QuS$4teAniZA5-R&Gv&()0Nu`dj#ioMCdl_gi^)spCI8apvvL$Z zdZK-^4f&iLm!J0`|G9>IUXIJxZII_1@=%V;-zAV281lj#lfRCBlAjo!WcyNoUzEc% zeqLeZSDd5scY!7+e^J|mu-jff1XB8;Hjo} z{fE-e$#MBC(ubV-QTll~E}!#b{1g3CPQ>B6NbmYHrN<-`9hzFc>;J-j5l1f4C$$F# z6g9sXdpPn0_CN=xrjN0QBTrxtOvvr@{Z>v6MskU>V(dfh<=R8=)vcfC58U<^{efE_ z!B?j}UoiPE`Ulq^C+YJM_^ZC&0Jh-GHPpbkp5KP{nH*%8-irE=e7+(l{&M%B*mArs zBzJPLwO%0l7yUuh+JFxirt(1Ghpo&^>M!j1!Nbk38REc7QuntqtS!#wuZ&6-TaER^!-+8AVLcOXTP8Xt?yCd>+`pMYiW>{gLZtBzcJe>Ka191g^U_Z`XBm} z08{dh_L=jmvLFk%`Sn{%ruj8}7qf-HxTHUqYx-0DbDZ)q|DpA{!xyrDWq@xkIrXc6 zzo_+FrulQfwLHkuU45b!R&P#z#IH=Py{Z0nPWin4MT{KIsee(BrMUU^TUSoQ@Zi`l z+P8u&rP?|5TdCEr-&z%{Q0z_XLEgU*4Z&Bcf1Xo5wg2isgf`Qh`c}dJ)cRg!)c5Kj zEzj-v-aaqeNZMp!|A+b`Ylc#P^7q`B`p*MY7+4D7CKbo(Dadh$a@3+EHV<8so!Oi2&@bNPc8sXygkp~xxzas?wk9+0{9eaK23%O`#1{vxHXQsPs3 z*WWzkZ@G*dSSfFf5#Qwp_fI^~a^pYM&1;L~TQMHyb3TQQ^e(?q-VwNe9nw#MKVtv& zsqUH7U5oTjECJ3VT?Bq13%JhjVB}Bx37_Ht{Tb|Ew}DUU{-Vh@!hCiivf$(OYW=)`^f!r z%7XT1`_i@kiTB@{z0CPR?KH|?@MQ_ouNVGpm)Ez!m=BotnB|Yy>4iP8)2-Vhhg5X! zF>AezZF@AR_Bb)VJ;ts7HLwTmHmd$%*aQ93b%=pWwzoMS$)CpeubuvF{Q1WNR<1vS z%((rq9}hmJtK%Q}L!Mn*Z_e+Q zOr!a_J$^*{#-D%1t-msUQTu;&`KNk>N#0mOCga1!;ua@1ts6X~^mH7ht#~A;o z!sSK$i0Lo%`C%0OpE|#w{!txYXguN1H=O>$>>ri>0nf*H|DcTbWS=VA9(aBLoEy_W zMpXNpyhQt~rQr^jVjpKd_xspKpI=7%RvGiZD#JckO{2$!u0O3d#?v_VG3PUeeH?q) z{&S~lpDxWl{Z>kljt@HJ$M{c;9A1TeX#M2j_QF5ZpY~NL<0*}g=6r>elNWzJYK}kp zd^g4)VLx>~+HYMoZ3?n;^z9qRUeFKo#Z+g$i05bAzrwJG!oRR>Kdg_*zhdlHr`oSV zvtMj|_4&tB-I+98;v)MIpD-P=FK$0${H67jHa_d~GqSgQye!A(RoZ-v>_htziR}|- zKEiZ+1W83@K2l}l=i|-K-U%G#^_ggYluN@WF54cef5hXP=`Z^H%-oNm^~^Nn?$%%5 zpTYS6f5evC^O1mFe`h`>`s3_6$3JBKwfWf7mstNCd$Oqy0m-{`!7U-1=wh_0O~GE9*c0eC#hSvHm#Z!LQ%9{`!0@cKyF#8tw19^*85Z zaoT_U`B?V&^RdoLtbZvvG_K#b{`!2(oPW{&7}3?OzrG(9xBbVTk7bWPA6t{8{`!8~ zQ{9!a`?BjVhZiyl=L2<@>iWc-e@&Z4J)~Q|Da!uX5}fbJ^hX?dKE|VLwj3B2&MWc$ zKgHR9!1|ldzeok%-*KFoZDaWV7SXvO0-hqhoX^iQ(fHR@e}SJ*;`_H@cmD>@UxI9^ z98c-IQ`&m@c#YC$v2qKE?F;qQp2q_(R^>QPVm_aL+0KY&CjO|A2)r0k-kvV2hSUWtU)cmcow!}mGoB_ z=@%H~V?Q2SN_Khu)}nl>oQ>%`g>LUst$gx-_jwih3-;TaRDH^4VS7@^Z?Tcz5C3nLssDeu8+H4oKoTBNBV<5ciQFOhxChL%ID*UWA7z+4Y^%D z*&BSpmr&jMmb>L+e0j(!U1FB6`dhyRzVzGWZ$tUZV#-(j1J8%@X?POlk-hOes#q<5 zrCUC)f8~{C`BhH+rT;u=mrwTB`(tYFHBNl8|BQTh|BvjC=c^@Z`AUDw>pyjdS-$JP zWPkAUgk3(_|Eie!hn(_BU-0iSJN^gh--TgVz5B^yy^}~6Whpg2ZhCY7R zA4wnh<4HUJ`*{AY|55o1^4IbNj7b zWayjX`XlKJ|2yOA8|~BkV`@Lw-$?&u`Gjr0asR7S%h&r~vajoZq(A)aNxMAqSG~Wb z@?8HS{ifx+`#tlfUZy(?eG7GQ(be*MCU= ztMc9b1M&~}-*QzSrN8I)UwM_0pW<&+zl?l$y-D?hzg4;Ukv&}gkpA=Y-Ss*75BzPV z8$a5o%n!IdYUZ2eyZ%b`f&V>c+k@IepC3GA7039y>(8YBHTmv*n)C<1!m2)N-Tnmg zNoo(>pBCklf4Kexe_rjTr~2#u1AiTM^`-n=evm%!k8`%1`UAZ`ddQk;^1<)&f$TdY z-(CNaec>-_Tz%+y7(XAxFy9dV7Slhu{!8{n|9rvLhwK~MKZj9&F`i~2{+0Re`itxd zzN}TtSNdn}U$OmjNXrlYdC@MP{9W%K(Vtyo_=hq+i}rH)!?DsI2)XM|Lhky9kh|?C z>Y zQ{>0-{Tn^~RCI?ppYe{UPt=;m0-`j&AMikTae6Tdx9xMB&Ub0?XK3-G7WZbZuGmjl zh|izo6d-)unuqDXTi$VN7N%1!j#~Kky;9$(m2coukvILz?Xw8`v7Z!$^87<7WIwk) z1x9+#E|CXMznF#CC+m0sPhXhgSJLDAPW<~*cKzu*pRQlI$cz5v`7L1;?33+oenlzt zg(zDe?2i3|QI;dlvt{G_i-!SEr~V8c$??mdV6*0x;0UXa&J%kWaB--V zW1I))IK8k;Vmz;bJhQ0Wm7ninlgU5u2mkZv=NqBF$rt_mIJBRm@E7|z(LU!r7y%~u zkDfvD6&rW8N1hI}as7$^6X9pNc3u$cp;YSsYt;o5TkzDE56&T}gJA((Gy`ztno15SDK(p7#vWKHJ$p6u{F zgX0W|iH|Bj>3N^ZzvI^A$!dPbt&9vEXKH*h`HTJn(`DEHlJgUtx)xve_v2O`mLF<< zGd2H>TFLlXG*9QJ?*HfyGIY7wA0*>%@jR8kvHj#z4u$E{F? zF4y~?errmw9`vyN7yQnp^@z=H;2A3a!QaBXl3e1u&F`XFr5Y}tQzkLa^C14bP`N7y z-=`$xJLX@P#CPi7)bqty&uQl^1^+Yo`4SqhIseZEntA;u%75@nJ8ujAMO41$rRE_& z+kV(zO{Mvbjq&|X!7pdLmHcz~nTO;q|BqWUu|4NvZM^6G!8f?}Zhw$v>^I{1ZM{7| zr134&;AcqV??YA=*DuTI55)MN;`9eN->vzVJ|2V5S!#b_j^}3oVe%2_jq!dF>Zgv+ zMIpWX7=9Z4O>%zf<8yIH=eIe(nPZIK$@!^|-z6cP-wQQ<)BHo1>->#ce0D9`oR8;A zjqyAb8js({twL?QH|Gn)e|mgr`z!d)ANMKzpK0(Pe9u$)558--WL~+%g8!ky3RnLB z3;*ePp2Po(fsb)(PL%)ny1aHC8+;pb`Hl1YsdJE@Z9km%O${sfAuE&fE7Re>KV zx!U}}9Pf`?bFlrQj$h{dyrM{#FOA`kE;soTJAUf?UtXm1e}$3%<<8$%;#=lw{jN0f z*XQ>pKh6F!YArJOUsR;?-|QcY4gTx>8@}%lY|{BZOXWNIgW2TYc7K5JykcI-ES3Mu zLwF>l%m-E$mPsu5UsPP`%EAB2_jmlC=C1eo_&?P>eB|3U>JA@P8`j+f;}Db6{SV@5imVb5)$2|2TfCJzoR=n^gYK zGx!;rs`8cI-$CoYX$kp1&zOJbYkWSA?|bR}ONzFhCx2Gg4^gWUrO6B98OA4Um#X|& z2BA9sSA=xA$$z-NmR|S&s70$YdGYacm9aiAH^%=Z20x39@n7dp)LLohF4x;v8teBm zWBgxiWp|N8tOYSC_xytw`=4E`q@|4R-2FVf1#_kV)zI{)X0{ZMDUKevz<@2vML ziYn)*d@nDq(r{VHN{Qw8U%K3tga51FU;ZoKm*@PSXYzj@Ki}c-za+4m*Jq;RKlr75 zzpLNE^V%@{)#3lVR18oG&QGPy8(Z%1%i({R^DXS~pZKNnKR;i^$@yQPJnyCb0r0HV#oh04F2o>jP%C%z5>=$`Cl2*%U`DX z?{TZx;J?Yw)yPj?oIh6?{I4+hZ}u-G2LDZd#`Z7z{J-4bzuCW(8vNJ!8LxlQ_w$nR zzs%sj-v7|^oi3gK^HjcL{Ld%uFz;^p&H{#TT&)^K@gmBf<&Wh-4d_`l};<-hWN z4bFe%`?X^H5ApAdIs6X?Zs7Hq82^cX*f|&bad+5lxU*PgWDaZ4|RN8E|``g}cVtAw@ zj@`f1`Cp6irkK9~-oE{)be)gZ~9;e}MV_@=8p})cJl@xrSG-DU(?6 zzqq8-m4p9l|LFX$LIoZEHv~Sy8T>x-KjiM8@aON!_eJ=4Y#IC?a_0~42mi?fRsQSz zEh{{%0c&mZtSp3ff&js8RV{uS>(B1ZqgTzt9xN3!{XIiHW6A6(je;KT9%zu)+O z=l>-BV?W6w8N|;wsL%f~{|6uDV*%~F4}#9mVFF2)eI5YQ8MQz7WMC5)PkcV$JfG+5 zfipa{yB@&&UOE5E`Fp3zS24d2%K1C(@6!0gFRV|^`Mv!--+mrP`@i8Z3)}IJTNTFn zo1!B9{LK}{`I|yxes7Nd73l8N`DC%NK3}S>=O5@UEutB!-9KVI0Qqj-OlzW*1y zUe@RPC1@@+{rL0Oc8O{`dyo^KVYpX^(ys^TKan4+~1`2h(2Fm zt2^Eh$%r=7<+6WG9=`F-S{a{it3uV3ZgBEEhONdAF8#`ze1eQomZ zxV31JI-jQYRs9?5<3d#)JAPK;0k`VkR~qS88vI$yt;7G}`&ZZCqq;vsr^=TXECo5Q!6;t7Jwx0P$%^CjS0c~y~y zD^?e4=TR!xl(_Og3g5KzCq?f+-|X)*VSM0xJE8H-IR8;;od39dzKu8DeaN@}{(Ljf ze;9n59m_ZOJcr<$hQT-O{0I2v%Ks>Q3#sGVM+4%0i|^z3W`DoW=G#e)Z^n6xO5?o6 z<@0UmHmtYkecUMDoL_#(xBvcpGtXNXe487~H~0L6;G2fQH|@Ly_~y$0D10kX`S#hs zs`q<7seB)nk8h7@d^65BR2t_SE}w57j&E{`&)1*k`t83z-^}w32H)~y`R1Ni5PZ`x z_@&Y5P7Th>r*5QM;U!$!Lu^z?&C3QWi@2{BqIp%sbYF%UO z*Ho@n*N<4w8vCF6dMx(*MC|%h->+G@T3tWVdQ`Lc+iX;~MrK{HzyYfE@-?j4#F9eoy4|^Z^ zJ_P@B&ND7j-d|4Z$%S4g(Mx~5F$dnuY*<$ERgtn$s=zg@IMk6&T%UGE>FRx-Yq8GJ9t*TdBE_5B@l ze>G~w@=<@jv3!lr_mu|U&HY}JpUL)TtJbLej@^$n`EBx(PN&I>_lM^G?3y(?-`9@2 zKScdqSb6?M`?C=YhqnLV`ADg;Kf6ffKaJ;QHKi)wm#@SVEaf~?)#_r21>e`KEpg@V zE8oc;c;5gH)X`->pG@KJn|IDDy%T`L;+i<$!`lAr6Fjy1d@?nt{=NV|f1teYgr8qB z&mTOg@?G#b)!}n0=ev7;QJiN?bbgVZ59<6(P1W)I`8qB(zE7Z^=fnCJue)&hWS&oq zJ#Ua|jMt#MHa=H^5NdwofBz`T!75zd1YtNf+sy(&M=?>p0ZWAnTLI#^uJ{%!#}Ob3In z6_xt=!sRQ=-1CHxSFN_s7Ye=>6_>bj@b!H@Zye|SBHDR^qD$uM_~!}0*RwAFaQ;Yp zKal-=(mhX5#ArO=7yN^MAHxse@>M_YZ?+fO(>PC1X7Dp5ML%DVYMd`9C^)C>PZb+{ z3ZZ#jevyBu&to3wF5>S~vio!JyFia`@;CPTH#(mSK}fYc{rm6KKj{9de?N)%t&g`Q zW$JiK&l~mhvF97~`EzNR%Et$+a(w^Lo&Utn_r>beZcpkT^!aibCKzu09FdS zJwf01snf^H95fIvhkw%z{)Gzl^8-aC%6S1APfP9d27-STW%l_3yw3ytd!N7W6Ni5R zbv@h=pl^^~%6K_G|M31P<^5g#7Wnv_%g2ZCdx>Ok_dG$0ah^cFzsDFaZ9dZbdsM!V zKdF2;Ze<&MEH?Ovm#--8Prv`Br_agO<+0hn{K-^Dt=On&O)Kb;rQ`L%M~^(f{a#&~FsSFz)(&bJg}ewwQBKXyFS z&o9yY<;?R0VutMSZw49^m$N>t4B79~5bx7jURa@>Cs*tKt1#sVYC=pT>MNc73DsNuQt6?U^T}R_gd?j_36LMe{rX8XT9yx0%Lx zSCONf55V|VnQOn#Lh>(Uzwbiut+-Hm9|rh_7jplB`KFyGcp`ua60Y}=Z_4`-Ip2m= zzJ-kQ1ck-)|F!*4u2lEWRfiCvN$Cm{_>i%jnzQvwTRp)Pbe_fugUxvYl zSbsI?dKH+^^c{<-_YW5+% zndb>YyuBR0%{KT}ly09F5d131QQj8;{*~q0@0SpKE6l6V&J+CM`KFyGcqf1tG~jw4 z`KG)Nl=JO^%C|h@Ji#pEJV7$P(fg&;`P$4FzUkvzuE949Kla7>M!Qv)!nY9KZ=;s4 z&lkbBTwQMREtc;(-->4GeA4H0c>h(dF3-~ZHQxJGbiS3$()kp_=UiQG@;P?=)cH1R zmd>Zyntt@YKwYl$nfTVOujl5V!Erf!n``hblwRnb4*-1vW# z@vTH%&noIeD%QpB1y4V-z8@+A+ zQuvl<@Ja8V@%~j~J(#BXTht2KrOW4+1;+Yqj=`t7T6*v~Tj!g}XEFQYh4A`aVXWT@ z4L<4XKkzwQ=bOpr*#1K2+xUE<^|sEpDO&yL{hQ`_0(HGc?@KWFmKT`io(}-u=A?%- zT$ov;ohQi6w$B%UkKkM4_oGf={igk%%s$uc@-ZJwjJ&eJ1C;m{L zw{JQN5|+P@jQaS(guSFcuOF)ioK6_!Q~UtQDL(4A!t$}lU`~|J<+;{$@AarKUi!>P zo@?cB%uA*8-9919g8cccfVX>wfoHlH_F7?iX~PH?&)>ho%FPLBI8T26NR?M5_%zxlb!FiNF$Its%?a`2^pM2kUv@ea}feu~v^Dth0?mw{KeR$uu z5M!Gg^{YK{LGvfmzR_EY##j@4AswlZ{vL~E0*UrTebL_d0d2RvAGsNHIDi2BDlglc z34imt?Sc4nETK-|!U+Gpr|7NFrxQ6+@WYvSA9={j8;BZtnD89noj%^^CG@`X94{L} z1CqR*FqQ*DQjYSju!?+Mh>Cdo^WT?3MTviJqP%VXApDKmwrlT2DxYIspS^E5n5x>f z_iYDfaQqHo7U(XRQ~-?2&z`>zIOGp;NSNe>DRRaq@%+^JntXv@ybmrya!;V!KS|7o zIc5dj1!FRQK7ItcC#8t@N!jwkF=s}J@(Q{`W3qf6zo6S+U^8CmlM@jAE7#9k&^^ii z0bdzE6rvR%zn;&^hW@F~4tw=a-K#q;u)Kpse&>EJAO$0}B| zfHg7ybnGi5KWa}ZU$igTWBQoPPu9`@X@$cBgiVE0}mY(`M=?_RBm@YgNfruaI z&ZP-9UDRICZ$<|FzJrU?@B>$L33~;)(=){H`v|`nT*@#l|8|KajG7P6uc5tkw(U)r_wTa540L<5#P~w?T*AnH zJ{FecdANU2*X-IakS+P-XHVS6?%2)ne62~Aho7D3PMaeAkNg2ReX8_li8J``qa#a> z@91Dflh}|y77g1n&Hn@{77i+ylin}zwa2e{45MP zzYzbE#_y3V?VU76?K?(;^b1Sa{oN^8=meTvi_zx`6@0nmeK=Vua`yzaP zIMZU>|M~nD^SznfUT#i4=>?wJ?c?(YJ|BH*2@7^Jj=Q!)f9U7M{E<|_e3H@pc_!op zdHQECpGx~EL0qDKTpuDd>4W(=P&4-`pN#&5Tr}^2hj*baA)#&+D#HXK*!A0@WAJF`t%IgZz{68OZub7XL{JCxH zp0u$sF9?{zBI#^6m17_5!!h!!J!;P$ME$*(Kc0Jt=g%1IyB=~r|3@D{zyDc#)UJ;k zKl}5E<7@r`d+NWRb05x=DZhtdufOPlzjK~UUw_pe9cGqTj|qQ{m%d)6hy7}gsOfKm z{&QGgvh?%-hRTndzCL+++E7lIK9W2=ZIC5Q|5)<$wBS#e{#^3()_Cc&StObLBIBj6 zOP>DD@zO_qlZu<=-Pd~2zypcTpxc*bbKAFsZaa zwDmvrr+;y|`RA$noB3ZzlK-W?kKD}vId)=P{V(->$7cRR?BQ|qzts1sn)#n(Coebu zs`2dK&mOtl{KMnrZ?VS`<+VzZKgB`b~cHAG5L+_&!1&KztsH3_2(w`+@u0NNsf4$WF#`Wh5 zm!99a{(Od=Q}p{N_$(GY6^zywOM88Mznrf>(SOw+ZTB|t_}}OAeKS6-5BWvwLmvO5 zTKw9h@3435@`jA(BOgF4jPc>WA6no;3w&sS4=wPa1wOREhZgwbS%98pVZVOpIy}S~ zzRiP2b)m=K`q7J}ua1qq_uhM%*gCxR)T!$yUl{v01ibTJ-!Gqj>8GzVcJa0gIN@gh zcWf-{z1z->eb@f)w*UOu*iYXb``zz;_b>nUDz4MN`Q@S4fBDOQ`}VhPW;Y%F;@FRm zzU2Q#-+PU}`SzopkNwGepE&Zde~Oe}GxiVPe)K^2fiK{$|$M zUytQ~;iaRQ!y{w&J@eZ&slBVlzVy=3FZ%Y3j{WHGds4pf&f~v0@-g2Yk;msm(qI4k zu75f*xqbAlf8?UEA07STOOelw-PZlZmsXwl_1I4Z{L;T(wLkphu>(|?k%B8EG7Ej| z-PZej-~2VEq_@3w+t|OoKKANOV`H=5d2g)qrEmS6xZe8ZTc7y%w?6Uu-@o@SRT6n5i9DP{erFPSe-inFN#rNE{L7VZ zv+%>@nBN_PN2jwP>DG+oAMtq7ykMpsCYa@^?UZaqGF;Uy{&f2DZvABCMb_FxYu4|M z4UOU8Mmmd7494g=TwM7ONg1Pun$@*8r4w_j*RwS)rrvkz}{)P+?%1} zXu6KQ29Bl~asx*LhTOo>R6}myXo?{>aCDL(H*k#HOmASXS-ye2rv3)@n)Nrap`X_% zKc>D$`7!n~%8#-4AG&uFas@{lkTg@XSoWiIfl`^U`iu5?C;Zzn)9j_d!N7$|hjayWX{il@|Ks0K2r94fZ z26dxpvU>K*y1g^QnJk)_xQwLj5z*?v$5FKoSz$bQ%u1ZU@QLy2ptS*w zv(@_yzse40vuJkWGUEE~^B60u|B}e$!O%wSV2_J+vJ5JRm*vhqQU&;jZ^HN>yfiz)15VgHNzJ4jHd>s86bp0k> zs(#duX!f!D5Vd^O-=x1zQsv|5*L0csQU9aqXZJa3`Ko@h&zY!x?U$)v$kfm7vl8hi z`>cuT*L9itCG7JO=_mWViRx!vrhe3CYVB+Hmx=U~{GF(Nk;~LCVSk%QKcRngqWaxA z0sZ)xqR%z7J|;|gs2s5S>-R~=I$g(0?gN>rpVLR%BeT&5N*ks`75O-QjNV7n-bAvw zU_`TDz|_y4t0l6Z81KA(RX&b>dLMo1`u(1L^n~?mqWU{y+@5SFiyL1Ff{V3re%30{drq*bKYjz^$Z(o(bO=y2s9SHU1N{2QjV{?VJA8Oc_e)( zf!rGv>5KhMKjuDRy3^_ z7wgub+6KHS(i)hZ$6CXi79}}ZBd#3Q;?{&U;>%NoHDL`!u35vM+#346BF&%wFxF6N z>rIJc4W+hm<=*rtS%bHIy!76H@Es6YNq>3PpxOE#$QnUj8)6D}*W&obWVdbO$+6XH zGbMidv?yl^`IehrwMHa30pIY`{+!KsS{wYKtRZ+E9140<<5&Y5*HjyZPByL_PsGy6 zHzp5FjxP^HNzO2dQRJF6GACe-RI-NM&(ccd4`YpJP}COMM&nt-8x*z89LkIEcn(eJX-snEsx=~)Yz^88OXwSR&Hs!oxWnng!o@B+yFEz^eL%h6-=K8QwpH~-7CLc*C^vfA77I97<-R#0Y8t`(xL72?XhLB|Sl z<#>E-TS1Xuo)r?#(Ck_!yN_pus5^6tYlXOS#R_rdiWL<3;{dc_yq^p|IagflX`mdWnpSwWd8#rFlb7Oq^eLj3fu6%_g9St036Y@#D> zj5WMsUKzJk#kdq#4b_T>wzaT3qIykgHHLhDv{UZB=qtWEUU zlPgzNdhyc-xsMvNGu4M86Iu6a*82m@zOb+(rXhc67?Bm@r%&Tn9(a88^`zRT!4N;ScnlGVgRoxdu7N{#YI-xA^)4bYPg zbS>I*4+`@ZW~N+Z+wsYV_qic!^#WACwXh1qm^a&-BlfC20nbHh5_{)bBoz780CR%KS^u@FD-;7!9<-MpS5`;<0o67rBP4?DIQ67p1~JhbM8lp&xVfJ26T9W{d}&c+7WeTu;9p zjR`MWL??A<{^Yqx`FIuu@XZ^EJPa^M1BDa#rBT3|UCXxTovk$yXc^2FsQ$Jl!D+j&(b$K90 z9{Y4vPp`{qK4r*bp1>LMm}gSD9NnYwlx946RZRJM`f!XK-J`Lut>>@H zr-=8D^iEoFVfNBTpR$6tMB?10X#91!mo42#S6Cr*)YheB+bmAfde+b`x zlo#c@M)^#b%8NuJC1!DfD2mS|3I8=up6WP-u|lq^{f_M}nV(&Ly~K3;LzH(;<{x@T zdyZZ};5C8QS zat^X@l>YXu2H#&UC@ZM=(YzntmlfR9nZ5*ZW{5Y>Rx-T%{K6CmFJ*W)u!1Y-ha4^x zIPxXzoQBKUa~fXE&T6=fJ*nYE^z}3)zfyKa!wcDC8ZKccHN1eG&~Pz(P{UWSehn9~ zJ2gC?-bJjGU&!#1D+L#@E)9oRyN2`ezHc@DJl3G$xvXBpb6A~*@rzCxp2aGFs~!8! zWFZabv1|?F{rVctW%z+GCH-`EPQy9$gVBn78p98`EA2a#ozdh|*hvj%d1g@93V!pr)5}zm%xJlydVu7jtls>XRV5%?S3nc`gEahrueo#=PCrY^{K~Cl9SxlXK0zgwmuar1h(}V+9Ghj zUH(>qnZ!eQVF=|H;5fi8>=rni<4hK65;#xd`h5b2IG)NPodTC|JdZurEpP?LA$HCZ zxJu&e+XN0vT>mM7g+7HWa<{-jpF%ctpTI(&LU!)60tV0Nq?bFAq#y;V4+VT z-Yy}q(5H|!of25+gYq5|Sm=Y^r-(ISJnCPG ze&|Y+jXs3IZ$tR2$KTl(5C*^W<7bO7deZ;=&p!{HvAk%!qmS>&JN?cZVR~O^5NYWO zAQg(?;~nAeVBDn34nHUHD=3dH`n8G-@sa=g9gLlHnc=UX&yN>wuOBG>WD$k`yM`)& z>uRZwJwJb11UupOI$^yLE+NY^^0C8DCWvp#FZI3<34Aor8P_Tvo??VD_AwsLH&iv* zUt{jzyT0{Oj33A0;kJFkGCT-Xd~B+o3D6oIK0PM90pU}#U45G5 zdI&#m$_kX&wl^(7c<2P}!|7K&!WS#yAt~-nkdL1F(h)T;Ob~O0Wk^dv7fE+;0?Bvd zrT?K!cZ2k2JAs*g8U$#Xm@AYmg4$&Qij##uTOHeS4KS#gC#AeoHbtEwG$13tlPYRz zoRv-g2xX>RcKg{u@<+EqBDv!uc6z%V z??c8E7MAhs+hfj;c8L(Xd^?@p{x>1BC0(buxWFk0sJm`F1+ne`g~gUi#-`dfR`geCm?S^eYg6(WndlquIZHwywWa z!)_}ZvK>hOsW|oTlIiXCgPZd9H_Ip9kWIt$jfR#b*z*VbZl|Z%zi&FYEVKD3i*ar? zU+ueE9_ecH!bxC!Xp#=1rsF|R8WAP5`DNejbo;A!?5y42QZ24^TWTshkF+eS>D;|+ z#|<}DcV6GTtM$n4hV4sR4(__)z>1F?ZmwRpW!=u2_LhSid-pVK?`-MXaee2en`>HE zY(HGR>E>+>OBLCH-mbj|cOc#N>#BF}y5WYd9krbsuf1X4o{t>9vATWVp4y!UcQ@?p zY2D1z);3h`>}{#rzM^yUHk94FZ}+YhNA@n;-richT8qtx9-|}xMtJM>)YzK9!wIa zrftvm!|Qf8qmJ8Gb*^haynfgI`sSLN!yA_E>}amuv885N`@Yu7RY&*KZA+rho{HvO zOFLT*C5zW}q;=QQeXX^tx~blA_1koF1KDYB?Z#VLj;`p~-LSQ@p>q2zd)KLU+qrq~ zp^x0Mh1#)B>bh&=(S~KM@$;Na<^9^82p zKDp^5OLyMVTDQ4#_pVCCCw3la-nC+1bKE@L_FA`Vd)MC1RTaB;ZSL8VEbc)b_i*#B z?FaVkTA8GMMft>+cy)LDMU1At8Ek>sk)3x_QseRX4UItM{SJH?`F6Oxiv*%S620hjt{X&z_@u(VsOe zt4$KGQ^u>gX>aAxByqcB+^sk6-eq+ru)AX?S$EiVX)>E`XsH7?68PITo1YCe;994Z zpj?}08!4~mT}R^AWqrHs|MuM6(z&X;75#MFe$tGWs9)0K_3W)o(0@7c+NoTtb=Srt z)c?ohf>Yl^sPD3xn-cks6K@&CBfr>{q%J!}*$I5n$&cH)cUclU*K#{IY@ArUT@%f( zCXuh$?JLXOKB0Jf>L!-oo?2^S`PH^hEMDEj^uwHELix2+PE5a+%B1>j;#{lUcxX>U zB46FCa-_2*(b&Bn^QMYD@cE4uTW*o_5A;jzd&HdL#u^&$c2=}DY(COZx$(%}gBVvD zFlJO@tXRMKmfao8ueN7xjR!kBx*HEvT)ATJ^41m0_O-R_+gGu?vZboIxn*B#%c@nF zMO<0Y)+pk&bnI(xX=`jb(%aeG)6;lxS>r*gaev#vn;NfgJJfTyd;h_cJjm@1sZH-GC#hr(Xj}?vLzom^; zvck6(9%$=6)YciV4pJhjvZcAJ=jM)EZs}}u@D71{mR45nZCuva+IzICt%Vyx8f0%X z)M)MLgrMWl{u??D^;R|Z9&B#hcewjtbFY)n`sUu|jXbDk_w^f4m%WFr#@3F5lrbt< zC5;!WL1RVX>d>v9_yp>*w0tSAk-fO_+_D`fB<6YM1oQbWpS_NvNJTuR_@@jv^=!t2 zw1A=O`rzE~XS|>F|KBN}3p^0~V&)e_&J@3uM-czqlO#T^OYzvoxw7p|+^*oqycq)l^p8q)3OPHOBE+smhqW}fpTN{{%II%vnm=k<0T z>oc=Ig*Jb&mm=&&n>D!Xw3YZw@#|&ydM;&br_ASs47c0mSy^9uAKK>W&t<)s z+@CG+i;P~D`BdOA4PAElrx8BI-r+|0UD9Rxkw1m^ON>XV{CXV!Qg}q}`~EtY9LX!nU#)MI60VNT^$Eglr=fgi ziK_E|tE4kg_-rNoq!O;iKc|GN@vEeNoB4Ms;cEP+m2fqGzjP*3pMVS}d#b&6NQN&m z!t12doA$q6hMV?pkl|D=jqUbTB}qa&viGJ*_E8y5`Ad`VBuz5hBu5#J{3Nh93ukC~Ws_cZS9IMm#Ilwn@p z+I(H}kwY!**Yb(_28$J0th~Fev$?#t&FW2Slumu+@&X-s;}o?vA}jdfO;e#79Ccw6%GC+f5xU zZCbdnY+HNdzV7COZ4*#&AIj^6j*W*X{&m;w+_GMncFW<`Bb_LIf-6eiDxHU$TN`^1 zH@0%cM1`Q9&{IHB7qy01JGl{@Rn>+=Er(m%uJ7n`bPx_IKuUhf$nC&-2ioL9F=#;;q~LFbwsT;T(CPrjWdMyc zfW(m73Lr@b5CH@c$c+&oA674dC4q!G5QCaz4nT|yDgq%kSwP!spk_L#iiEWRU~RFD zKzWc#Xf+NuVgb}%bo*d@7!6uS1=0^P8yReX>WB5OVRA70Vf+Rl10;qXh7%kh?nhUT z&OZ;eAD4a~*fC5I)eq{YA_H{yHv;9c r#ovZdh{g|4%^+`pnxZiMu=v{#v>4rPkd__35d8Rut^fb4>VH-Je^od4etGGNWxA#@RW$Z0i`5jAvCN*>y@%g= zSSl#pth>73R{Qg8x9xw^2W_97#@OLOTXyfe_?)w4&;D0j&*SqO+@tAuMN8M~yM7S* z-57T5!}WUTgGyWIx{V$Fzc$u;w4EIW?UC*bR`|VaPv3_n?CoFa?C`YI?&`C)iM@li z@zmFm?sQi8e{@gZ5ZdXQ?d@8OJ$&k!fr|472DV>*W?;qXX9m8hd#-`@S$q!9I{EEz z?}_8#$DQ8^A3ty+JUsVgxOeWU@cvb&!fA-4S=R|mR%eTYBz*n3<6`pnSJb0&m{v!y1?-SwBiBsVU=jm{0?xpb4tBB{tq0or~ z1G7&)IS>NR7v_FDyl?JX;0tQ55CT0MO6tRF;H3Twe^o^IW^W>-A$s|BdI5AK0|n z$|uSHl<)P!_g;l>-uJz`+xNO};(JF&@x7xK-y@$?{?~8uKjp7C^TF%8ei(|BOniIj zA^q@$e=fe}wzIU0zbd{q{PNPiU$0?(shA&UXRvESKh&?i0lyy1X4lS^OgKDPuN^-7 zcl!0A{j87X5!3n~TMm5WIo22X7uI*_RsGt#C9Ln<1MJ$rE@XY@)7Z5ON$gq*Xp1n{ zhV~}(4gE9g8+u4PeDQzi*Dk)x4iEipLTKn)?C`~Utf}Kz-|&wn0+tvWdWaqV%|oF5 zZ%LuE>-RiHa}7KnzP{@<(_9kkyO8H1_$OJ0z$=aU9?1k5FF{5pWFpy;C$MWjp}7kA zjB>GLcHIMcF8tlnzG28$F69~q&%@w(7(A1VB;Q5wIR^cEA(Qj(wa|NzWBApC&@kk9 z4{{jgCmDucO$-e~9_XU)!Z>#L5_B^Ba#DzC-Erk|FweDF7(ZEKeY9?TT=~Ng!G4RF zHEQewPo%5HL|oDSqR{oE z;;Ur8`}#2J<9Rp_{Fr~}&+XQ$-mCdXjk(yBdUS%KGfkraACz^1Eyu@!&bQ<0!6yly zbmpRAN0jZpK&!90omByoOIRH%`63|0u5dc9!U2i&+9VnnaZI+8f(D0v+Bu+^K6UK!ZF^&DLy*mDZR_yO zYkgu~$M&YyEp4%SeW$EHF)w;k-`3*ycCdLJzF@~ZAzE#HXGgQQy}j1k+`QG>(dY}* z)_dDKY9+SA*V^6|@OK277B1M@;%o6XH*IarU(}Xg`|pdv0JZsr3u+4r7T4zI<>f70 zP>cT-6y!aczi4r7Z64$la@_-Bv^RPKzJ_~8kbtk<7kB~^!Ab#N^F85hki&aV2-4uI z>)cwqCE#uG3F}w^-gzbnrdd5L4FRvekxx2*psk~=qsx!xgllVU@9?&EP? zHojvwqn_e$oD^EhOMqv%V*PaI2!cAWG7YyLNe?n!b^-1_nihpiF#EcsS3@iM?*eYa z`GvJaP;w=Z?x}$9-E>V$-P7GYlQ|-Lx@S80En@_qW9>xN70>ps10dSk&(I+VRq;83 z57k$#D=#fCa+hHLk8>1J=#Yd+7vZyoK>m&(e6ggn!Rw~eg{Ee2M^js?yQ8hGxqXfs zGu!8F_qp>I&MTPXUPqUCg-cj*XH#>7yS=Nu!`I?&>1^+CH~6+Rwffv^D=Sy5sa?0U zdQExds@kO+)-GLDy0nzdt#mJR&-DiC8{O?KI1hN#>&DQ;E1j?4`wt7R+u4(V+vhtN zc|soDr;B^sGJ8hs{#K52Y>lbRzQqo^pzWh}(f0#1I)ew|Wavsdv&*`jEPbUYTDQ^Y zhPuw&raGNf8S&I7)otk3HaWud6*y+9wQ2TQsobVyS1R_I7#~J)CBs%3xUB$hQ&O29 z-Miq3RruIlpn<{gKmXXL3NN!|t!4HU2P5z9w&8w}gm1^uRtr8`fk2^g==!m|UYFO8 za4jVS3i-vz*~VDUE-k`-1tb}0FpXt(34QTNlH+8@cO*vm0bB|w>YQvI9nz@+`qThR zrNRAGS8VJT*{o;3J;JU4Kuk9{*`9rgc0TUL>{NE#&3@D+WMEGLkN$dsm`D;t<1g*f z*=FKJ@3skk9zX?UIzH!|>~O$tM++9|KbXb_=d;|P?E$v4hDG#)N$ib7Y(_A__S;Es zZs+yEYEnyRGdXv7cdI?3m+ z3fqX^q~HWtk=r*x;RSt|5#AMWP9T0$&qk8iKpHFj3-pgVm-kKHpEC_Le;xWm|4;2> z5weHf0ecQsvIln&EfvUjDUE%lYixv@gYdad=tm#BuBN7j%ll?FC5Bkpy zPMk>oHkRvO)Y+Mrb)}!OtglP+!k*|K_Hc41CwqXO!T=jTfsZfhJ-ZSj>;nME2jG7h z0b7JTGO|$Mmv(9F3NfGw|5zT-)2Pa{k2ycCfbezOub~h0uLo?*NqABM**7?y4W`o5AYem-08G#!oPU?+#t*`RHIEK`3ACC*$YR2rGoJ<5dFi`uSOzT zjDD7r^cRuxE$zZGK@(g0vxy)0S7G4chq=8pFzfp?qmZqx;SDezN(&lC6|(S8c*0{>OuGlBOA{LD1o ze5C3}jVUmygKlxYC zX5+r3{4F?Z7KszX&zKRdzs~0iw+EkpXixgGqkqoNJ2t&6>v=}&Y9z9k+dmWbjTX=s zsXEyk{(az11P zr%$K&$Nll=)c`kmr3`I`fHx3K52sT}V> zU9=~Ev;!~w4gL-NPLJwW*_#+3d%r{arh62rZyL|8{By7b`somOIiE-BU!rAu=zlt? zTloj+pMN7oc#K!}!eUc@J-aeu`mSrW=$q!ZLEn77W^jFHNd3Y8Got#g%f@=)vgq68 zis`$O^yP}pZ$3ZJUg{hDX9S@w2oUM-5$i9_4{kuIuPn2?)L(Qx7%XCCSu;)Yji#@j zUC0N{q6w{EIe!NEZ3cfM`5hL2v-9|AC;3m&dS(~-1Fa8rv|ft#D!xq*Iyk)}=5G$J z5AwHjnm&?0;jMf<#{63j`-vjvpEVwmea{eqT)zv0W5<&H`S^$zWy!g`rs6@*E-d{j ze)&m0Jtkj{6Y)}fSNZ_wH%%|3_~W<1WVC+M{Ly*5)CrFtH~fX-?-kG?e!i^gG-^NU zopd7$@CUn?PP?AOqH$v(KN~5(q>ZglGCx56hWRs>_)!J*<2zu#od8fi0RN%*`wBTE zwf}(RKN^eg&?n@v#CP2g-?QVybL7Be5~*K3kQs~DgY&iJ0Xs!*LwwhDvM-PC$baC^ z`0AMaxjTv0*Ro_KfUu|8|49EfV|=zB-+_EX$yZ1Dj>Gw@>mQC6`3~7HCBn{OK<>X} zpPTz9{F~>C4w+w)JiMsUex5~@>6!DPkKPhH^ z_`i96&gb;po-`+*&%)poot9fh`aAPqY5kg#$gV_qels`~{*{~=^{>JC*bn4nLVOg^ z57Bi>~Xmh|i!XGg{wU7GahiUBDFh{b;*+?4k zSJ(r3(LRat9puxt{wKjJU*FPP$j{u-{vdyPV5&X|^br^4^H61KdLUb$O!bMFKo}q6 zOyoJHc%xFBb@X zWB4@SlTbI}T{!hn?FpZbx)Gl-&Ws<2dJGSFxxaFKKwhf5z=6<@4f8p7lzf~|E)icY zRGQ=2OUNI0vsChL_|q8nW+nMkDz)&J`t2f~d=WJ&hV^0yRUQwflKoPjM^(gg(sv@Z zBgS~1m`MGrc%Bdxvx!w>6yy(kx)>Uu(D?}VUx&J|)QY0h{v7*bafnI#!#BCVlXfEh zV!56r7f5o!UwS}Kr4y4#H`)J?wk(=22>efj+!+`jAO6(?ezfBceZ)KA^#sQGA;24$ zFS{K$DnPMey*P#OcME&?>)s#3eo686b$-65^mE(tV871S5Arv@KA$n!)0&@|@)6kY zK!71Zk@mBmXIue)oyQ#%55eCUxgRs-JCx7(Nnpx9D1T3WmI(_9eceeullQ6SGk(s! z(l5_vqW%}lXZZQVxitGt^B2Uo8|E*64*x@Z`>gYqk@<=v`4unHPmzyczPTLakNkWE z{%Ouv<^ykBpJ~0a<|~Hv7x5ws>ot}W5$~Zt(|V5e*|dJ6o`v-~N{{wfk7E0+LCw5g zW4;nTx?Yc(ukiV8$yaDIPDS`5-%lv}!=5;Q%$x{p#D)2O`cuqT$RFb7E7L~FSKQNX zp08x_e8n9 zj5o~=iq9kRqq{dBA^-XRG9Q^^ieDr5-;`aT$o=<$AhQ#M=N~w~FYA0| z)OdB&6t4{D>+qjDiC5A6H`X`vetSN?Ul?t_y_<`p)+ZjH=}3*fFUp~K@&KRj6URf! z!8DgCK4JYd?XR~`d`jf|a~{u-AEocM^ZW_>fi(68^4Hx7i4@P1#-}+bA1VymDO@M; zd>8AVI4?T7p2dr2h!^Sq&iiju{6apZ2l%%S(eED!AB|_=KOOOm_Ct|Ok>A~ZJTvV# z%<&83oAw*fw<(@+ym|j&+HU}Fif=tnm!rC&la!e$#QD)PGRw2lm5$ zW+Htb!`J6?_`Y%1_}KpZVZJ}-=chd1esezPCmQzShwy!aeInnFbN~2Hv>%V1_h3Bi zpJL}%SpP@v$0tv@d3>A1{e5!$_=fl~X)@n`sr~rK_+~o)$qi1PWQlLF^B?Q}7W4NO z_S=(;@vSg8PG|T6OY9G6eq@dt4-1L?cQ*FlS>x#YIm7;Y3hfsS@ymPBxUrI3NugGu9_T655Lw;oV zeLxypNrz3m81h+K66Ah5zk@|69`Zj}fAD=VKkpUi8HV%lqaeD4^YCLh4+mv;;`=yO zX-u>1uhVqw_u1D#!2KEf&ztgx@5A8_;(R@ezCY#PN9y#2c_P)p?@G&loA1}6@;MpF z$Ill@NU^@!$w7Ggp>*n>zHX8IX|ePFyjzpke14F|z6gHAeij|z{HLtzqgZ}|?+;A* z$!Ac9RQJn$4cynjeGS~#zKdTKbA;M{{K|oK|Kj=|KKxPSt1tIf{ch9W|Kraq z#{5mgM;HD!a;T8a?0ve&RpaiRopY?Wm$5#3&2z)=|4VJ;YaErbI^nN-b9#DvQ`ngw zzxs#YN8Yisr8TRk9n)9WB&@E%ZUiCyr!y(8=hF{IHBJ6#k=Id;JvayWs{~MjYGdpv zS9ydk;(iOCZJH=}0X14EIE`_Y6$4TL(D&-IDI6CW>fE#RS;cq@!kskpos?$^S97p2s z`AVUU@Gcr#tti$NBgWH>G^qNGzXHYD@71j`yA0#Tvu)HfK^G6NtU&#Yl-e-z2yLQ) zQQL{}IMY+Bomcth*OsJR?O_ZLJK=?fLqWX2KkCqMfbnHj$1hq;^`}r*OcFf?)u|4n ziTHy$-SVXmc(7rjf5%c^teNp1OTEfcr}3zNzC@t^4dO(bKz#h0*2yDox6LP|FG?-7 z-)M_OuLyUg{cfLCS_YnRQk zcJW^e8`dr}{X3TSi;*=O`}0`pRhBxk)f=x}#?@Z<-e~ddX3>BgM|#QknO&cXDi5g8 z>KW9t4Ryj(ohwrtlZp(Fb^tbQ8rWE^={YXMX3rkdv4+5ml^%a$y8_DLWUp@33+mMC zG}Ngp(lw2mqw2AJhE1nYic;6wi(F5n+}dcAi;Sk5c)1F3dONv*8|)f%tRs{G5-X$; z((#BaHcqZ*@?J}xNBtPBFGeto%&a53L1Mt)kM+<8e9W{)e->-xr!nSd7d7K?|JiY4 zwdZJc;|#kW!^Ijhn$A8c-l+5YG2Fb0i&&xJ!^pTpd(8MJX_nPpx32yaFAO_5O}mOL zM(yNe#8u;(Ec8c-OEhvUHrWzx&i&qm`}@#8&RmVRy5H{nzw>DDX^d{=HeR21&Y`vU zKX%q z^mq0ClJjpq%lo{Y?S0<$Yno5S9h->tv8cCq#E#7a4~oU?z04T()U175*H@Kdug&<` zw{{iYM~C6qZL2cs2>iPekN6IYj$|$8=)%WM_kU272pZa|9NE@W1GwHZ_Dwy(er;uL1YWM#(TC8!GB4AP;58}ivF|? zb*0_M0Otquoa)AZX7N7w(U6H~35SpDK|H66yS0g4Jf|ze1f$ARm{K+-p7dfwisy92 z`ceL(p3|jh(VQ3YoUTj-j$WhB>5>pdA0@cM8=C3-n9kDPBkT3iFO&r!wlau1Z35!8 z|FLZMC5vjU?eUKxEoWS!-{tg=G*KL~0j+m5mSRU~i0onQliF3u3ExN+L$T75U&!yo z#gZy%zc1S#H?*Z&tKP|fsJ+L+MuRlD8#pTDdBnT5eK{fV^GtDx{r!q;pDRI1$JX|= zpL;S+`(u{&s%LBaX+j}>k0OxQQ?mUBvQFdEZ%wT2$?ws{2>$!!6mc8c)2(W+?05(r z6U;+MLpf`iVrqQVPTBtnG`?ATMMv$4&f1>lIjO+nri_teqIBnoE43$@Zc&Z?#XlL~ zeU?Na@@BET(x9^Nf9Ptc6ZpIEI^v1?Hx=35V**;73^-a}i2q^>9UkG&%g+oQ$FrXw zj){L?NA{B*FWQ~`Q*kYLP+YTcA^A;2sQ(85X?zqDYWzk4q>hb6(ndjR(SC27@xLP5 z=SoT%pNd6&%qQD>EaM-O?Unyhdva-O`vOefr{j$O^-Ul~ar3{7S;tp?RDt$AXz#I% z-!I!M|5M|ub_zZjfOzfSiPN5@gta}rEDBM_4>Ozee?_*J>_;zF{jWj${~Skt)&7cX z2{+~Tw~kLfK|0-BCkZW=OgW&ss`doF({?^;>P-xWgVL~R5#cA*Q{+W-?6NIK_g&mY6xH2ykzyp4Hb=b>(}LN z+EB5=yS{LHb46v%njK}!nwLFZ)99~W)4Y5^&Ew@e$~P>iZr;?gEWd8KXj|O8q`7WG zah`7z?{~$91XMtT2|#Zl+|=LEN|X{Fnqxd%XgI4EnJt^T(-3$I&Rt4RolhmVEMWxZ_%2S#jUII z>RZ+|)@`b8E7|C+sqt4YX)bAOD_y^}%3EBra#`7`{7q#`J2$p8cWx@fXW`1uP0K6% z4K3@spuh6f`RjHzEU#$ZxPAfbU7f$Nbxlp*@=a)a4iCFDhtQ(onc1zo2ko!J-9tc?qYhZ`$Im z_uc3fD{J|?Rp_lMc-zUHRPt`}Z1Vht1qB5Q^YZZe6ud@dnYX#!C;yhz;Ds_iZ%bY< zkFi}jclG0wbNq}pPaq{Qoa}%k>Sle06wB1v+DXH0bkPF@)sSJo~uHk+T?u~oQ zr)a^YvNrWW&vW__`+rINOUKtzzu|mQ^i1>(oIw29JqkUIb;Lmby1HNPJUZqZ`ipj1 zC%PXf9ImI-y@hTH;O7($*HU#~>U2G)f5T3A{ttg8ociU(fX_7qqnQ>w(U%(V(YGS4 zaK18z26$1dv_#i@3v{Y`>-8nHwX^rtk%S2?9~@_b2|1LqV<=)l6F)r=zy@)tu@js} z9u83PfhzHy_q>*OjYe@wyyr{{dL9u+w6Gb*RY*Y} zBm1e6CR2L;2gyI4F2#$QSI!^h*OO_k4O*xR@g{Hlc8C0xlj4*e2#T${XkZ@~18zg+Hr5~j)>c@CW)NSaWo^ztg z^9Hz){`d{xR}FBZytz{ERz7D+xK-YMX&|e-Z%DYcpPxy%Ro-73`Z4nVkpXU$H&f0Z z8rK*j^KJk?BH_f3aW#8h!XL4~`wjF)`@SyWR{rTJI#h_CN#d6OEs$_EuJXl330LDP z_z?rWaop!6+&bg#m6HHz?t1Tt&ZI!mZikpnuh#pS z;E#%ecT4*%x4_kU@3+9!de21K-CkSQ)ang%G3*f zCwg5YyVacsbyVBb+Jx6U*0s0aUbEKD7Qs-Hubqv)$&p(J{(8yLsh{u#+VN@#DQZ() z0CJc4I)u1*aS6T0vTRk&T)a=Ne!HMr27RpYbrf~r{gZW_9X^5zdNeda8@#2yCz|Sg zCb-a^uTj44Zgdv6U_5$#MQtm+9&g2pbt_Aoyv@G)j+Jc4Qy?%ZTI`?N4{4y(AMDX@UnUQuW(3V zr4^8*l%oX6y!!Tz{6)8{jk{zoeCx)I`@+;a*}$9HeDdAAn2+u1-K}@&n_a~M`w9Q< zMG%pxmZ_(!GC`dz^AClT%e1O8+UFBYrUg;BN8X*0aZDjpyi)ZUxS^EGqX6>i-yv?E zm-!fdw?^?!hT~`s_74Ek_{Q9tTDoNL^-#r7)cET9oNP~h8%vc6-VH`*YoAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi7z(=Yt^fb4>VH-Je^s|`f4OYsa$VDyDjIu*#cB%5SY~hR-otM_ zEESY))>AWJtNVGj+xCCxg|^SkVC?XaExYd>yw2IO=l%<>=kfYi+@tA4MN8LfyM7S* z?KpPz!wq`qgDP9-nvEU)e>T>4w1XW6?U9}gR`|VaPydG{?5$tv?C^}#o|?0^$$dk% ziPYDTo^)3De|1m)Fxu&w?X5bDJ#y;l!OHUo26w#o^x(?VPY-@m_gn?-Gk6`IbMo8a zz7xm8k2=2-K7Qatcx3*`aNqn>;r**mhcE3v9Uj^LUihi`?}i_0Js&=P>fLb4%!flG zI}Qv!_4Lz&sq>eF=Cph#d|}@4@V@tc8h*<8X1MvprEuTM_dr(|3LQK+xc$U|!F}_O zhbvE?2sgK#4)+~+Gd%LprSOFvmq4>PRN*{0cwzn%gZoyapZ(tn&uPW@r``-#oWB&_ z{vM6@$Dz=CjJNtA`Z*Dv+xqSB2=JBXPle~4JP+QfpFawX>_0g8)cmIhL#IxJFW}Yu z-kagMCr^eeobQD%%s&sBWuZB(PYm`!7ZvA^qwf>p(1}ywO6TctX#S<}Q>%&RrJ>M? z1A}uq&ll!@JG^iHo8Sw2fb4yECC~Xnr~86J5MFfIGaPZGacGjj2zK=PjK z?SlIFR3);6h+a9ea%7_~oe!v2%sU&05rXsqyc z_<*JnZc^Wn?s$~i9c8TWRlst18`0Mf;U^zdm4rUrPz=BGT)npIwPQyAjpvUa*tEII zC&~Ym?+w8BUV(4k^SwLU_j+#Ndq>Ccy`vW2BcD|MH(>EU<*zsL!E3vI7>blkerxyv z{qTiQ`TfUk_!ot7l6l9Uf}X4xjy7 z{o3$;)=%?@X#yP|1>%a7he)XLa)_?9kcJ*HtvHtUE?COOib~OdGMVM>D zdlUMH|B3YvKcF4H_&@cl7vEurhyNxaH2f`g_+kUr)CsJA=2A&UJ+x4nxE{XMB$a4|=lPtsFmBxIRWP*&BAfpp9k!;D6*wvrVT!nl_ zxmYs0=7BsH{8dFaSG51QYP7zL{jr;+&-H|8e4?MBdHVSnDe=d+mNxy$$5k5lF;}MNYRWuM zD0OjB$oYlh5ZQ)oLUyA5-hp1vL4Pj9&p@)Le+T;iCa!Dns>FQ{t_N^Ef!F`W{b^kP zjmC$)-g!15WP@Fm|JlZ~tM5Snu9eER*h ztUp}B`k6lP2-#HG|MItBEBNxe@Zjm4sJ1 zbJ4J)$_`(k&DYYws)5NRtd5zdr{bz>%r@Sp={Cmfu3-7I|I#&3bwWSTD>jJ+uHJ%wCC6T`(iLaU4CIfUH($M^78T)71ZHh z!Q#A!^Or2GtILC&Law_&jE*L6z}I;92omsh_yUhXB3LQlYq=}D4RUzz3PBou^HoN(=J9i85`&b+(8eH*;~R$rju zt}ru9!)3ABk^|+^$qi@(+yypN;+TDwL+_d?tZd5#x}Je$P%*B>s$GZ3BV;CWoAg@;S@iP0JQp#|pUlG~~|T$EZbmEjCyS+P7})@O9>~y1KT<>Y|uc zWvl80rY;X#wPtSzkKR!{wM71MSLI|kicE8Wv|+bwEz3AO(F$gN%HiWj!8c;FZsR+4 zGwMkW$4Q~3ya;%PE7ng>jv%N9D>HEGk@O(rWf$P?qiIpN1hcP8dNs77|1RJ*oL^W= z1SMAj>7EMs-c8rE)IB{tvza5Zr)Rc<-!ewYZ!>9nz@+`qThRrNRBx zmu>79*{pZJJ;E*nKuk9}*`9rgc0TT=>{NE#&3@D^WMEGMkN$dtm`D;t(Ld4iX(s5=KlEWEas1UEaQ<^4pMf6yq)B^$AM{7`Tz?sz8u5d=PV)Jy!#3hK zDL4sM|d<$Z(q=S)M*UyuIK|C9SzgzRB= zz@9@@?EYOuO9k>>N@HK?9v|W6AbhS9`q9U)udS`+^1hJ`c_-`SOA!b3gZ^`alP8nE zjpzCob#~?@UFoMh>+90IuqXP5J)GRh$sXXRFu*2G;^T{Y@2-Rh`v3s)0r+1=z!o8o zj4TrPvTluCCI&R&A1eZS8daJ0G3Un>5Wa5vHS~f0^?;2z2~SEO`vzz7;Ufd!e<;nK zO8$(6(!T}z0X`#`JCoK;_!n=V8-zKAYqiNF-(WT?fBp!tR51PpqJMDal}JR3(a&*` z{vuMovTiIBG_j>WoA`l$6$T!BklRZW;}r&G&EfJ;`_pNdpYtN@EZWn2%A1b*+z7iQ z>-FzxGlTQy(0KERKiFqaw__fkZ}(46Wq;{pGlO$y5q>VmFJw=2v$>?&#QU-63>Sgk z4tlI@M1Rc5@pjY&z65wT@NN^{jk>^>0Y44+Jb@n;?WdtG@LvT!6L^on&rak0XQD3f z$=aT77w}j_Y5XREccCuu3A&JHi^OM3{A}ilYa$mHttKx z--2`IkT@~?tXa|eYka&`_uHgzx<`@vrt$2`KZi=7pH6|7^Le!XC0e$J{%4Z9m4A@_ z`6p6@$9Uz>FE#boyDKB6@A@{2zG;3N^v&mM2G@6n)F1pmBdYKEY^)b9i@sg1n7*q> zU#{5v=JNyXrM}UBMiAP90Fi-SvHsHh;0BcX$}-DK{YBS!oO~;@ixigVQ@={^sENAb&fj z>7)4*-pbcw%)b?|pD1GfS>qwu_Y4up^}8@QemvQqkB@j!o}9~TDjxLi!qTtem!IU* zWAf!V5iiAOrT1}u)AT}$KYklbM(a1tADzcbo$&Z_!=EYsUIrcF=S!+iqYj|nMK`hl zf3TbBwChPMnlK*nv(fTP+SvLe^8@5>m_PH0A5~C4z7zI)901A(;6D_9UnYm7_8*Y^ zM`Q6F`h+}|_^un`dv=_7jvSawBK4~WGGp<2XrZ@MA$hD$o-e>b7TL6 zfAf6NA@fU;hZmK91Z)nzKK~DuKPB7V#&V{{!+Pe#Rzc*`d_7CHQMBRy0eK0}*Sicq zBY6vZf#(AV|4H9%OXm5pgXhQg+@K?Q_CH3lo#anw5Bd3ce7^9yi=0p7E3a2UKCzyp z2zm|ep>9p(Gl?0&WYodX0Qo2NmxAG~e(XH$&6mU&)zK{~B6|{XkA8#76=BFzq)S z+<%gz`;)AkDE)cbZ#a^07cvxpw}++eaJ>QBgT(*y1-H+qi{J^9ma2VWmC zUK-KEpE8a9mK)4+8`~G|cDrf5aeD5LHV3RD{1M|*`=~#Dn8uz7b0mwLjieEOg}tB` z?UN|qK|XC8cmllg^)1bX{LCHg5Avr6rt4EcA8}zm4_Bq82eS34RG*9qgz+(6P8RBX zzlr`}-%N5BKg|coTNupDjK(u7Kg548#=Aa0w0+^Ca& z&n-k9=>l~~@?4<(ni`FrRbB$jABQ67l6il{uch zi2QLkOC|q?KaFE=RFOZWQVV~n-!9_G7g3{PSTBZAPG7pE}(Zeb69{rltCFDc%>#?SYZer{PF?AQ7FLH@?q=QAdITJtkgJ_7q42rwil z(tg(SjLYD!>$rpBA^007_hY7fhw>Re2~7D1CqnRQEb08q?y-i%vZuk z*XuF!6+XW$`3h~usR)1M`w3-#*c0cEnUjHyxG>*Oe~S4E`9s`%WyTo!ihIV5^OY=~ zuejsqE08D4P4PXNuLKSG#^`(n@zk9a%~yt@zfYczWYPM{<0bmD&!4a#NMo-ff8CvsNbx*rVw!{Up~9e@!gT`Acd`D7 z^P;00SiE?Kc#;0^y#F@EFXU5tfPeZB{rrLO(Rc>_(-F^TKNQIn`Q5F@Gt+*<9KSHW zX}1fpZ-^gLrtIB~+~CwHmiQJs|FQ0GF@JAjzdgkm z-wJ~hbcPSG#Qu=xN9Kfyu#nh)XJh}JHGw{#Gwi>o(SFeo@0_$DiS56g&N24iI8dQ- zJMpgSHum4?SS`Vz*sr3!3-#!J6#3c&<9QSC8OHM{^8ZZAH^u&w&xg$D{*&*&$)8Z7 z@ow^D=0=_J)$Siu|^G->tTAQl}5h6R8e zx>xRL;GPEVY2cm)?rGpt*8m-!Bh(JyTMn$}7uSCO;g2F;eW|bdx10a&AAVjr?ynj@ zy6`uVLxpU1-&4J=T6f>voMU}`jP=`VpB;JspX(xDH2I}PUPm?d;2hwu5I_m4jkBj*;Ssus z2kKZ|!{01}_i@gY-(h>p*j{ZJ0=ZVtKp!i5jx$kXH2~N&k4U_uNDuC`afxV88GxqE z;*O6; z@-GS)h}S>Q^@+k0A1`t(;Gvj!?W6o}iRoV1ARc{G9zSDiG%Z<&rlT+pA$w90JsWhI z-RIghJzU2qm16`dGuFLYg0-j)(?#tJ*HiQyN8<1JN}-MN zE*e{{DApAt#?$pQsQQh+0>#?z)vYqS4CBVLZPYVC7Z0zjK>duC+A#7cZK8ou+llcw z(^G4lSNP`FmZV+jWeg8H;f01nLA<~pb!a%i__C_w8!e{#lc*~ui5`RMREN<-{6U>= z`O*hG*f7z*ZK*HS%y^HbUTvw2sp=9AJFrIy-nwU2mh zZie?Wd_^5=w43Q^tUj#xczu3$+*r9At8Sc$y)p)IFB5J>ettv0iq~FE5}I0Lwl~d7 zJt;ab-GTvY`*_;6(tO_QMxE(#Vvv>pc=Rd{kUI7wEj6!Q=K7PsE3=5T%Vt@-__u`( zYnPe+ZA<&5$eNA)c`Ws6OC8zj_17-rYA<|mtoU}LXh4o5z2y7Mu1`mm2UKYF4C>j2 zI^n6#m8p$OMTSQ^0Gl=gY^>4r92a7&R}981VOEJ@f$|Gp*5|#Txl(jQQC`&3N2@cHCI)Iab{` z!|ug!v4)JLvrmdQ>ik{|H?QI%R;c(eI_}UOGyX}MWp&rBt3SmH!%j}qt{{t1J2@F~ z)wm`L{XyaqjU0tlCKN@@*qg%O+*XQkXXzl$ET{q3t z#L=gPR(|5qTlPbyd}vhr=$@0$!G!1vLdy?|mHtkvoAeWB<+I8Yk6vx=XvGEn9lgKg z{F~46K5u7xpLhM5=96*9CSrXo>g^r1WAng+VljI!Ge$i%Yv0oK)uq^LGk*51U4i$} zVK{c%s*E`T|E$C#KEt9TS<5-P@NpA^QKA^9&>HhR-lsteRV{Ju3tNAK5^lvpDgP4>GDQjZ)ss z^MB$#+h?B7J%5wAnLRUh}ZtdvfY;~ zs@yS04wIsJ)`2_C#lGPxG8qU~yB%$T3lRa>SL|6HSk(M&I!#1H8|YC`8^Y zl~)>67XF8>hB|@23$G)dsDD$D?L8)-#mRu9^@aE^w$R}b{=EFm&~ZHb`Qe!O`#Q3p z^mx(k>>rEk;`_xl3m1~#M1=Z(0+7Z>F`*`I5|Xy~i^CG1*@EFSRF^wzgl4$@^5C@xQ(q#3*k5&X{$4iA)1lm0Ku_LBYR#j5|cX#Zd1$gkR8mM!6?-2T?_ z$tOssTk0jD<&r4}R9DrWz_;6uM{zknEyG3UtD3jQ@rYM7FRblU-6I{u*p7qbd{T7k zs^*uvs&?Co)~@d;Dk~DDWL0rNOIO39;+9Q|*4Az>YFX#q(Ac$UV@-ZT>xSALg=^Zq zMJ20B))zN5v@Y-5ys@UG!M}E0OU2IR4Fyert*s?|Y0I)Tn>JQ9)o)mzw|QgbO7Dik z9W9krwQF~lFK=1?Xl;|fW^K!gg4#zbc2;aGsA<{Ux;(#rg=kycvaF?kV{x8uGw*lh z#)9Qt^{vbDz=OZOU`g%v;?>(5H`a7)+PM1hm94Au8_R3E8dtRJ#29VhqkMh&`mTn8 zW!*Gh&11%0%ZlgwK;lkW1{OAEHG@fNSn-_*LTY@8Zms<>@+UPJ5pruxk_ z?IoMMwYC15Wi2I5?WG&GReOs|RxK}Ioxi!ftZP$iOV{RdycVtM+PtFD-`Kjo8~UqQ zlfVA)#ub$G}{-ZlA~+Sb-KZ5H<0UR=2qedjf9tn9Md#arTkd{gt{N0h&*Yh7zg zbD(ZV-qNM{OO_ODEnHGi@7vnYkl(PVpngeVedEL4Wy|s&&Z`r24b5A<4ZiFB#L7DU z+bZ;{D)_aN+o|N;;MwH)ixw|lyeKbkDW_TPZRzmUiT9FP{Gg1_+nN{5V{BK>9sT&^ zoIi5!`qGzRocsm(+(c?I)*i26+V1JvwA5@o$OXS2sSJo~uHk+j?u~oQCuzZ@vM%+0 z&$Idw`=2NNH^9v$}${RO+M6WtFK4%bua z-aYpJ>~b-JF_zhNgl|HEGir+#@c;ByVZXto7U^rZ%T^jDEqI$s_~1H2$s zTB2*Y2|Cri_4*>(+S&W+NWz4c4~{dzgd9rQF%+?&iLV|AV57Ly+6m4h4+p6DK$UpU zdtS%8Mx!_--gBk}J&%Z^TG$NZDx{!~@neR|osa_JD(H z+I9GU9*GyxS4$JTCgq?pE>RHL>%Mz~U(`&My0ck)5-vD^J8WcUPL7fVo;>+P%Wc?onn0*UZqR+!P zR8EQudka^BUy*RTge&07p<+!|~agFd&DJS)>cq7u45^n8hql8bl^rQ4e{TNS)dJO&3b52xw&Hy*kAHNR# ziUDqvH&^Q2%I9nex5_&p4P=$~bqTlj^D_yz%KJ-0KSusPGQf@UX3F_P;~HaR-gV$d zB%Js$u4d0k_(K->fPvm<-`6DE%0E3thYImCMcnfD#S*T@Rle9H;c8q3KVqOaj{BU1 zTgUySgj>f=mGhRyHO?28gpZmp5>EXnUPyo~2Dm)p;c^`@z@vZM1I%JMn3C7csTuo% z0dAD5Ny3#L6n%??TjhOG!maX_%8;YR&6DG5k$W6$SV38e>iJs^BVaJZFKc_%L9Bt2i)hfvcrgoqwwS)p|c0{83Tx9%;W7 z7PwmP{T8@d@0m!uJL>A2+q{8phCM=~ccr(ht)b~r{*#kscpx)=-I4#s6a8ExyV;!w zbyU~f){LKdtncW!wPtNyt%9LuUk4j|lOwkd{O2Xdrhd#9=)g}yNKu>X1CYDi*D1us z50}vISeCD@osZv>YuF*EmO~$FeVs*}`2ETHu1+681w9&?ppD*A-($@UJ`-GM&(|b> z?rv-rw_-f{`HH$W`gy#SE7z|oZT7bK8ah|CH+HpP_?z7QaT~O>dmHOI+v^%RV?rRX zC-@XqA&XJMN_n;P1jAg_Tx9l2VQm zBnuikI`fy@v^MULz3|N&H|`In-p&Tz)aH}#=EZ#MP`}-JhyJpwSYSWl`(6YQnQEDO zx*`+QsWSgiNV!a_Dx-Zq!DLzxg?r@P85zeELd7dppM@JrxjYIWul}9l<~f;<(PwKE z?_@ZR=3svhAdPR#t*NC;244?V3`LEvuFuN$)VHxztKi*egvQ2KP(l+=RvlKi2@1up zqPN!nyBz;|?LV?06yK`9bO>*?3E@@-6ovU}rQI&ukCI=uf5y_l%D)fzc=mHh|G6xO jC;O2rsP-y{-j9K;V^fo}9rTVL9+&bH8}uYEYy1BXnJ^dN literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k09cc9c2f6949a2cd3d99c8317996d977.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k09cc9c2f6949a2cd3d99c8317996d977.o new file mode 100644 index 0000000000000000000000000000000000000000..f0ba4a72dce54a389ca14746d89d04d02e3ef40b GIT binary patch literal 63464 zcmeIb34Bz?l{Z?wsJpcw2_Yl_0$L!9*n}h&gX5(o0TP(a0)t?Zv|0;DM%!yixUrRS zyKx-6#0Z=(wkMekFY#p5#5iR65|qtK?4RvP9Eb4b%lKu!mk`@=1o6Vi9!GMF`~6Q< z-P=`HEkt6+FXM69&Aq42sZ*!UR(0zxx^3;ob#|ML8O6rF&Ek+z%h=Ex@pB(P^|3Ti zdRe$;(Bc2(Y_H>g(+_R?3K%=`ykqw97w~)0F?-?faC`~B-{Txjvkh80UcB{t(LbiL z(KmP6qi<|+L@zqnk^gkCF^>PTX9@}_?6)ekDeR89>M_sm^O9vR;7?1|x) z_GgEW-}}?yb6JG0=YCx-9-^rORfr9U;iFyq5*rLqc8tq`0o3NW_9%3J@=kD4w}1)zdF2d-Is@xAAM#xrRtgC7RY>l-%Frb z6(wD5c=Xd}?kaj@c;UV1tN5AW4Mjg0{@0;bhVMQ;G@QKP>gc)a?>w{c#HY`kyB>WN ze|h-aq@N9Ma6JWiULHQb=q1ox6J7866!_eA=G;BUhZm+jIvjoGr;zJejPI4hd+HLdhc>P{Q)@P4^!c9!YAr%Y{k>?37d^@Q;$Dy_gqiyjxo04hp3yw z+X!Bd!EfH!QW<@7X9fJwH+u2bC;C-?PUN3&IBW|IA0>Y@e9;GA{5E{`Ltp%0`C|AI zzIbc`Up!{bAj`%y^iRJ z-w8iXLLPNRulwCCM_-3;_di}f;PH89F{d*l`i=5|Q{M;=d~cs`bb*s)m$C=y6LWqEjg}>%}+W8`hVjaZE1H5ME}h<+Wc$V=+NQt(Uz~< zMx$RUA7Ir(&i%|0e=bb9>5;~{3PJd0sc7Pxq!a~_%PrdfTsZd zGT=`E-VE3Q_$c7p0Ivi5x^JNQPQW(x1qim^`8S=1NeHtR{*{T z@JhgyfO7!fS^>Eo1CVF5`7OX#;`kIJ$BC0getFzk_28ay*LTpK$y#j(^1Q3pl=x z<3Hf|8jknj_y-&#IQ|~T{WxxeK0b?aEW&X=jz7V;cHr0m{2bu@IL^Yc1ILxN(bz%u z^IMT4>+Hox8ttV=nw~E?vgb!7NA`ZP8&mZ!wq0rm%sSgAM#Cl?_mSMY|(` zmwt3j-Sa=XCScDz5qK5n=jlA-MBq)F57T-2iNHHJe~Zp1ouGD}A0zL3!!^@<;hL65 zlP-EU4LP5l9nL!Nt)$VqUq`NC)&0(X%>UlA$L6t*J@&i0uP0q38SCtejx_#g$&scn zE;+I%vgF9#`XxsSAfF-QRU|9P2>D3P7v%iG<@^?8_&Ujub|UZtoPUSTQ%?k*#QBft zJmo~-S)Biv&Xc*ELzBW;H}6gw4ShUmv}JSBMUso1J~r>>9gwrekf{hVErd)9A=5(0 zw9wQU$wIm!SxA3~cRrtSdG17C5$elxB5)th@1}G2iNI%Zew5B#Cjwu<`9IKkl8|Rw z_~?N*oue(UIw1>Uk?RO?_xI4*#~%A+-LO;YX&z*n51HmerumR*zNV)au%1%&WZH5E z`m3S-oF@Vw$N6qLcbo|9!FfHM+fM}Aaefn>+k`DKXD2^+XU+A{{g3f$x!ZZM_o#D} z*#@s^VaI~a&u1RlSC%|*(^bxaTPpB7>Ktgf&pCP*Xq!Kau|%8$brq3o!oKf2>(<#X z`jgrJ6<}=pIjkR(v36{UtQojY)-6CAC)H6u;JF^hl;m(tQNi)grP|oQ>|`e9;@e$( zU3ku6|M>#s#p`VFQ~TBv>lqvMCb1I%`<4^lBpZ&Kam1MHn{afT(CIgxSno9Gub1>2 zB>nmX^wlTUdkp$2Nxx3guT4O|=EV9mgT7MIS4jGDNzaZfEN4IWL;ig$D+V@Xve89Y zH|@(QA6#>w@dud1o$1)K1;O!Vgh_SwU_v>C%`{R?Wc(LzfOSPUp|_0 zJMdP&iJ;z=J+}TnqEAr&4@5t8Ed3XVK0*DLi9Tm6{i8&mp#D2VpF5WR2SlHs{zpWg zH$TWH2iC?O;FCd@0{T?6OT(IYQuG|weM4BI z{{A*L@X|-v=<8U2zL8o!@Sj+ZzIo6$`nwLS5$;cl{t@fckw5tcMzL=FZEDJbx3Hdl z`?bx{i(e`q7@D8F;M~uAqeED`z4PzB(ciw{8-4yK zj*Y(bdilUhuiFNm|A}qj_gKGOn1TLdZ1g`G$_Iw;q4sR_Jl1)Cm{KwD&cCzKm$3$X zWp2g5#V^?gUd4LxwaSWtw_i(-{x)^Pf*0;z8-4332049@-#0pX5Ik@54ZO8?Lv-X% zY~byV_0UOmbm(6k(V=A(1LqtS1AlysjgI)QkG^@3js9nsZQ#wF8>4Te+6G?#HXD6o z%ckgSm3H)FL!b76mosdT!#;4{Z5tiFhh(%33|)yi^nTwc=E8vsGXUqZ(IKoOhvuXH ztI$IR;O_$dE^rxuzY2Ih^t^~@0pAb!O2EtP1LwX}89kSaYlxq*(Q|XxFF1EUXkNg2 z=%9o4dicJOhvC`;^LR4mZ?fwa6aLcd!V&G?+44i}TAucsRDSGo z=?i^P>YwNfY(Bo$ZB9wt$HiN}1K+d3_i$~7`E$hYfd4t+e@W4pBmL(d@Jd-+9!*i^}cXfxXUD}Mcz(@!0ux`We34sm$eseTUUjr4Ok z@6^*A&K()TxccW8FYpwuiKd{gi0T)ujCu;!Mm@kSBKYyCr#yx0INrnCXLFg3e|34Z z|KJoZYk%dGQ_0?{=t9sh08O+q>s0bHzR}|cvqro&-}jSIPi>TRVpWhI5;y9O|*F^0&!e>CgElU1VhF|j)+oO|lEaB%jMl*0M=I1-2lW<(j z&v!;saU?x9Ur|fr99@9l*`LgaKL4rd(H9R-i@x+gUi6KB%8ed7`0}PVITOzS7t|F`Npj1Yu}w2{g21k-81=r-HH zTmQmF|5Rxkc>A9ar-=KZ`N$ioI4-03Egv0vfa14eU}!F2?9q@I4_c8eL)>A`2VenHNAII_F){!AU z>ML(JHS`UFZx|WcNpQ`np)V0!GcvS=;BBXnR{(Dt8LA?9`>COW1aBW1x(=}8#*rc9 z_x{IkJcWJ+`W-t)hL!<7h~rYi?>sfM4fT~cUQg|IjttF3eP!(_v;$l_GBktWU8jZ? z5xi?;XbQo*PoW<0?vbHX;`4VSLv9=&|GQH|4si|Kd`9F2#5(pB*CO`UU=FH8KBz!m zC`W#NQp-sn=8F&W#fSOg!+bGOzQFp3m(Sy0tb_QE{@MD-w{!ekV?_(|>Tg8$f(R89L?rQGp4t59Xab_G3 zv8g?gO1o z4V;N&Rv+kU@;7$10@2dG_r{j??p6Nowt#<6N3bo>ZSq+Y=nkyoRTaCoul1Mt>pMdJ z#+Ei>?5CeeX_3HRf^mv*eE^2h)f@;mHGXgg5^U;f3hu{j%QyFpj_nVjzu8 z^*ww2dxC+sCSe_GMW9%#L6|iJT3h!91D(x0=sJTP-5nJ19BSxj@9GY;cb9ws+_xdn z+13aTCY>eb)ag!!+dp{dc2rM5rV)ZK$if<}M+0Y9&2D)Z9IW8MQn zz%tMcKIo{Yt-Z_I*@sJ~zqu<8E!EP()>?7-OQ-+(*IvfNvaPMDt+->)p01|u66W`} z??;4)<^Q%#Yd84?20<5SZ3%QGATYwOGZ5@<>2B$0=c^Kv2XV)`>;tvdywr@l07wUR z0jy;;uXkCi*~fVOq%rkhvlE|;G4)TO-p(d5U!6deNQy#-sPB~Z#%@R&L~-yP4M`{u zSM&N5y%JC@>a#NA@`M9|BnP4w;M6BTljBmsZc-RNVv$_jVFR`a|n|Ot&4J$>ew9KHec2?%cIe+VJoU8{_k~v&&mF*K0wvj z@xsI9jGe}2+2$SV*Q_u1R(1qC;Y*^><{9myt;BB+RdsfU5bu?kXT7xR)Y2ND;P!TR zbhLIY_9A;WVUNgLx~zElV($(*6t7s#DtcO48@*izy1JX%ylp*Q-QLEgJuU4`-tAkq zY~1GGv9@O0`YoINYiqZ!-MnV)8n$GMcbRudAlT6C?P~LvmRuR|qHB_s_T6v~!XoQF zb|>ncOFazFPO;^5KH0Amn34WjJ@Gt z7bY{OgiF~44K5*xFBF*grR=N*7qjOycnLeL!He0`8oY=NYVbmKN`n`$lNwya`Zahy zJEp-`um?4G9_!QKLUu@l=d!Q{&tV}AE?}J+>}AawJe$>N@GMrV!86%b4bErP8a#vf zG$rUlPmI zVB}&AcCfIGU9i~=e@Cv+U>gfF70-i&=(jj|Oxns}0BpS5ZoqLyjM+D0%iZs=IZM;n z>0OLv?q+tKWW$Y||nSTeNH-wpUb*BTPVKEXS z4a7DNjh4e(B@AtF*dt+|gli?dRl=DPu9I+`ggYgiC*iP!n;jn}+NO+Zm;oW>Zha?Q|=de%0hz<_-Nf^bR+w@Ub! zgzF^SFX2uJpOkRsWWjGM;1rvSm1GIHR>FhR1g+LL}#Z183*1PCU55L{9$U>ZNcotTgbFXTgZ zC>5}fFNcMf3HTVN&tey<1>6tVh4Fgv77lL@7A{*l^T8JhHuMpeu%VCB5*GcXFz-hN zz3316k+7kUvl2G-fyE;6_e%MXNw`SD?BfD&=%Y@;hCT)*?346fzo4&{aHoWYd^v1T z!a}|rmRTq0k8%1G=2(@>ocV>U5Am2e1rNI6w-q+;gr55FJ4r{xs2AgQW?~M-B7@c$ z^mERFeNY_JMjXx*4@<{+*n#tvvVI?K)3xEZm#EP%3LW2($BXj#wmdEuWDFs~=l#w) zqGs#~aWLzBb)sU8rCx&OT||SAQGaY4dISGnA0xNgV(d5I>cQ{%3^wh+RMABN4#ny0 z@dI-0eh6@VI*WurC{)MTvl+n4)emBZeKDOK33{Mq=yCAA4EFR=))$)UAwP|sjb#x3 zzR+Y(zXqp!PAc%UkRwLvcWMq4{rqh9P$)OXb(Did`}7u;7Gw@GQ0LZR2lzQ33MJ*z zdNSt12*5ur#2iH5xfN@uUuLkXX(s_u2|L*N4ED&)t{7_pm;(Aa>1^!*MzblBJMd_~ zG{ns>=m9?vN^^w)ars}I%?1weDF-!9^k)M;%SDdH@y}!sUM%7`-T}PJ6*lmqJ+c=S zw0{*3+gbQk-PJ45Qr<|2-WHYR`K2tJn=K&C%a@@I;>1*U+;-|(kL)(q@Qd7lL0yG6fgy<3RB`y;Y} z1XBdUo_X*`v%vq#4Ve7%{B5jiLAOK;f42Io*8Vy8|6Ln3dh(Y{qXY1dgq+N$z@^>p z821nQ2Y7oZ2{{qxv_$~_IudlHQ26usM*P3f%k!VljI*U*4s%AsdICh#xfP@Sy`BJ(cG0Q2#I5Shb;l8IM+d+rgiYkMzUi zQP<~2;KM-i@)GIK8Q0&L40b3P{pE9=dDu%fLwKhqJbj_OOcEeI-f0&6^vtk^pJBo0XVM}i-hQS9 zKPxk=;b+J3&>!i8$1m3h$?MHz7;v0lL7X1)_l4$U!jcMpt_5G1Nd}9zpQqueKR#jd z{*rkA2v=zjH`f<~!9@El9FL`;#3uotml-zkhXH41hL!$^Kf=%CM=&lIB%I9oc}+N# z!*dK6`9YKKMF)rD@(Fnh4f^NZ93R(T@|gJ6=Xc7Nuzm$GKzTmH))z|lFiaPwzCu_G zoA{;hmrP@U!u=)Eo2A1cK01RR_`~_w1+gZ9y&evwq^SNG=}pz*>48vwS{N-v|A#@J zn_B|dE#ouNn`0yh+5c2ye@wukC5pV{--|W*Fdsra#(YTrGuh)Y#`{DD`zGw;p&|5% zc!WLinu8$^(HD3!4f^LZ?65!jHS*;a;AiAgel+#htH)2IcRI@f9!1KF>BQ78?>|4= z>_3C%?<{_yDd`cv&hVcwJ<0Vh#V?hUW4`a;Vwuc`8{V7Po%=&#J>`O?(aN=<(BpPy&;&-J~Sk%3M7 ztXAY9{jAdbsV`LKWVB!~uRlt#e6}h2TBeP+FSHnWKQ1qgA7245{VmeFgyHg?3ipq` z(9&F_7lmK3CVwRK5sQ3RY4Sz1_{DnO60bD=LbHFHSTB;nWd9^zk%^~#2mhjrS3cgQ z(!b*4rT8dO>_hyQo9!LM9~%a7{(T|%-+4nH@D!Q+Dc<0J zn*QN`zPLQp|6;R!62?DI^AE_giV;Io-|(lG4SC4_R>b20^RKQC2l~Ga%%pW#Qlft1OFO~k3Z78 z)Z%~84>sW9_FI(8ucyuUSg83I$!oEni*HOxc^?Y7W0c|f`h(BMu;;4=fAY6^3>le< z^ym5FdRnn>+&*OA0*_A{FZ@l>AJ1o4U~u`dp0esQ(z_sTFUZ6FjToB!!@k3Ye8RqZ zyj__~I}VDy;_*4;;Qgub!1FU1T=FOXF#Yw#Bu*dKUlMz)2jRID^pl~F7t^uc3wze1 zF%m#3JAWAZQm?;}Z?t?!u$W4D6O50?w=%z4^JS#BK$#zxg1;9X^CIS--Z_l&3yOsG z`61F<$Z$C$@xq>@9~9G``ubRz9}q8G|4i49mxe&|E3S{Y{R|lEU#_28RX^1e=trMF zBO#JT7Oo%KorteLDF5;GHkbc8=)*dH=H~MK3^N|4YU{;FZ?>@=;$iSVj1LCW$CpVW z(D-mYM&rviVEDr(^LmM{Kd|3~{Ulm1bZ))C_lNj;0}Eus|FGZUVQ)y7_M4IirM<-U zOEIqh%!2V+uOFAFcwDbruYbn+8?vv_2K;D!bPU5Yn!Nh;P^5RhMc$8OsPi#j-@T>sTa-b{ zHuORHSC?m@MIOvg$Kvg2y`$4F)%t%ZRJ@EfT#Wez@#KtO-;g~pU%svKE4A<|v-FSr z->=fIu+U#=>3`)%Ed9Iq25i{Kf2%y~9hKi|3%{!@{a0q_>xHX5?4;WM8cX}@G4Ix7@<7LX!vlTcP{Sg1fW_wz{AwHfq_;2C->3UPhPyH{-h_43@1IGFZJ1~4a zoc|Szto)ao?MXiP`)Py!QqF&siKqUTWyI&tEr8+A9+m%C|G(O7p8k05`&&Pbcg6DJWhYKv9uO13zbNRAKK0Y7Mm`?dGo^SQ>JrK-t(sIhc z9}T5v4=Vjb9t;=P2XBl0&ItD3)&3s{57)^Tc#&&J`bW*yGy)Zx64gt0${_Q_(t!@eilpX*{%Nq_i( zA74-6PMoyg!?fOh&T|cKPV4P0W7bQWd|cnS-RO&tAN#rBYs4E}-zB^2s6V-0r2Bm* z$-MG@-$S7kZod?1zlTEU98M<~_w%amss%9>_%HUeVV@M$UVWjY=@fqk|D&O-^s|aR zBcWW)en&$&nthLk@|@&vaeGg*(C5P;;`pN>+D=yOInzS#6PD)b!5%zWJ zu#4_CDDkh`8~zA?iu(inF-i5uqoMpH-T$0PwBWrg|H68o`_D11vbg{B>u~-_2}^%+ zp3?A||7iY!`9OoAKiyyGdiXE!mps*9&U&!Jh(h`3J@S{qxIb`zQT9j4Uub>ze)@}= zuaIB<|MwT@2PN(=PwR00X$c$t@|=c$|NX_Q`b+D3?=KW@a{l7}GGxqew7zsr_hR@4 z{=a2@Ijh;2Q9DgbjaLrQzRye_5saOXhp;FM}3;G4Ce{f1&IB|JM1%r`g|~SFOYO zTP^-ltKr{&fAOjQ@_CQA-}WBPN7zv@{DH4u%=??dUrOF*fBBosFLj#z-FeMAoZo5j zmym{k|NVvTr<(Io(R=SNK8wGY_k)GM(Eg+0Pw&Ti_it|f64vbR&O4;T`F$3Dc~HZ@ zJAc9R!kimyAodTH>ubK>p6i@V_MvLNznBw$-bL)k;PEQee@=SD^9Lf|yxaZQqoHhV zKeaD3Y5FSFUo86pWA|e(z5h!5)%}tOLV1?`*nG==?4(K1hO+-MRoic)=jG^r6AIUt zwx40_zaG=<=g#UkUOVZYk4zd2Ree>{x+(QIvh)4Kmc`=9Fmk~My5KQzAIv<3W9Ec>1M zei-eSUwS_(UE2?jgsl7DQ{wxF{QV03c}dM4@Mqf3${sZIEv^r8p4Q>K(>m;Y&aelk zPciqyv0j8dviSXQ%BPLAKbfuAW9en>Pxg_4X+N9qPv*(zLB#zQJb%ph6Vv2=Vqa*Q zhaEEVE4^PM?yqE;csy>T+<&M2Aa%ZcC^U`pn`Y8aneNm4W6~s*9`=NLW4$QvXVCjR za1fR&+k^i!oqh@)2TnjSA9p*^q=`3Lr7><*6sWB*gT-+}#0#9zGsqoKlFHJ@Ae zQ*l4g-3(@izehqMdJO)N&{VuXaR*%z@&5ZlIhOtZS(9}Co~^_O&0kC0ooE{8Puh@$ z@+I!~c-bA*yaw>mkato%9tdAx;aBVqfjG_&x5JFW`K7WFb_WfK%X>65XObQ-b2Wa@ z54H#5`ig|+Y5K?Wj_xp68T}!ikFGLq#rC ze@88;Pb2yde8{D6DA5!Ft^e)$)m!sze&F4*!U(tSv z7kA{qkGCg(jq8W_FBYs_Gyp_2hRQQwrU@#QAze4vvR$0coIzC?R4|@JY|4Xk(&ynM&=M_Xgm=ups zE&f4YQk1=1(kGGpr3;-a75(Y)kLRJ?#~_{X7t-e(cfSD({g>sXtdi|*@b`Qzeki`k zKScYb&`+-BkJjg5==m1YzR%PAafRle@K?{e}Dw{-8Y1GuD1{-6zr5h@VLBLX96ie;t>f_!YWODfDD-3l_RGpkzrwUX;5qr(MW%g^hID%ZKX;aMp~Tbk6FJKL3bOA!_tS>F!oG#x zl*Oif>3LUGKJt%2gC6rQ;$N{B*-y8ZZV$?jy1$SwmFAPtOZj?VrHD$Bg-tzu&{>vv_>YaX+W{~IyorF^KzhczCp_DO7yczlpO zEb%i}jTh*%P>m0(J%FDVkB4N)KgaFC^3d={vJd5R0n_-h+?b9fd^!3rRN`l>{^@yd zCI0X}5#ocz@As4b#e5?28^uee*+1!Xp@z5GPq&ZEhsyJFR{fKI<|+KG_8|MI@+7uT zq4eK`@uK82KHnl<;_+j(59y!s2iN}-JU-^e?M2VOXSuzmykqS_@l#~tGc572C2>3` z&+CoNXV&@VQu#^7gZVsPqWm<^U82ZmjW6=YRSG_lJ=FQeYL5i*VVz$J)p)V$pW-8K zAO8HHyIPTl>=WN#Hu9H||Ac*f2K^IcpTELxSJJu$Q~s|yx{$3%76NNGSA(q(Pz2qOn7Yk=>AXf zavAf9@w_MWZ_P*6`9#l8GG3k@U;ks{W3D@7%Fp)$F1a523*tk`S4kL;KA(?`k3#n$ zMIN%pa@`(d@>kfP|7*pE@qDa|k3OTnzcfC~=W`R~vpMbu75;yP_&~n;bK+yJyI+xq z?6FF>hh@Dr&wWy($Mup4zx(*G#si=4jpvKGJ*@MKy55!XVLrbs=6^N+^Xn5+-uHjK zHBo$sd}PcIV!idWA}`q|zP~BfTZ1OvqJLb^Of;Y9>#M_{w_cwVs_TEOzi>Tb_9yfk zzkZ?q^!c9FOV4Ta>U=}=s{O3_iGM-_^Ktz80{TI|k?W_II6l5U60nj#9QyNi(1&$@ zRDb^NI(`2#!S@~TJn&@mc~Id%kQ=I>uQ5kH;vc>gLv`>YU~%I`M};`<($f3)`< z@jjKd|19o5xL7~#KV)Kk%b%xsBmO-RJl|&>&m9Ns%Jb^!_?}3%!%Oi@-xI<6Wm&i% zP)9KJkLMi;CV8fg2>goP#o6?J4rAi^>mLWP{ml#ZG4Va1tdQNo-_N1;;Fs3R?6iZ2 z_svr2`Qumdy$%{b{;CZBUXfKk4gT@$G2eSJzvpxaEbzS}JND%0L8uSo;rN_Tu6?@;^uz=BoKOyj`B$WJ4 z@$aK#crSQ?q8JuZ!L~7T9A=Y&!?lk7wvOI z`y%3ptFJ+?fa^1q{_y=R-d_>$3kANJ+UMcpud+S(;r&TFAAkMZcz<%XlVCo+`nUM^ zy9nm;)Q|A@D?Pz{@%{=6Co$~bd4e;OWxEXb2??{!5`BP@k6enUjw3a4l5YLtYO&j*7Ie(<%Ft`f;c5 z@4=Y(dGn-xIUf2c6#4)M*}sraz@DHhc^Yw-?LEO6skp&p!q?W!i;?}felQ)GN$wppX6{M2d2 zOSVV&AH~OP=qD#d><<%6`kI<7{F%c}(svQVOLkJ<)IaIl&~HI7JEr?{PE7Cv5!cVu zm>M5EKb)2E&i#3sNBSS^yA<}!mh+#{J}*b+S81P|WEqdr-=|I`eelBNpXu>(fc#|( z{0-i4R>n^~`CE3jw2$PMo66@RGha+k7V#te%@a&ck?|}0%g)yG!JHHk?}EM{IBlvv zeosvBhft93jsuA9voe0Zm=1fVaQzwfE(j*)(ff(K@bSz}74gb=|2!X@mGSdDjmP7W z{w4LDoGkN?9N*kj5uZXnvS$kWyk<|D5#sioHck4!4i98o^%BkpEs=pC6ufhP60gGBcDyS8S#boE>+*2U_lx@ zh6?v*%6Gc|%}f{hP51}&l_ujG^oKE?v`Guh_GGW5By~O_`RRQq!~T3ez98ch{^<#3 zPm=jS`oGhu&M%(eB-{<8BK(W^(^7&LK7N-=m6zn#$Ct+6M@D;Y4{ERG6F%QuknzI( z-Q~g!Hk1Fg72=uKPxEPpoUh29PMTl5%&)blr8l2n z_;_fV`E&W;ADQv_b_?{uCM)x!4f9QwI^TMN=~LAC zlKhkG*;$YED%X$4quRe9XnYO@L>!-zqRux3!PHc-AQyPpD>FXdP<%|vQu903kCWB; z#S^qoQS&d2PtE6)|AhVH{afT?s{d2}v_6sXV#j)9vdk|$zDORKk0k7#BIZ9MzU-;A zp5jI7L+_s!9x6U1MVH5JSLf4f=ZJVQ3-5gz%_&ejVU?}v-;K~`WuHf~?| zgM+W{==&;keQypvjLisKpIE+6s=Ys6Uu^k4>1_S`q|m2!eSDhcYe&pQ35mado{sBp z%vCY``1SizHZ8*H@+8j`;UeaXmi` zc(NW(QqSUfyX21&h$6C6! zkFRM5Oz+~?msvq+|7Ey-hd+Ejm?ZrR?@a@KqDy2$&Z9M*9hK|(*fQ*2U_Zi+_iMSo zBNdWSxC=wtNdcNsso26 z;HP~5oENGujha6E8qNC?6H4xjXd_%>tet8r{B!p`)|*F;{WuV zjeRQiAO8`1>$Z1d@4WMc?>_p6t5`>DQCeT@{^>)*v5I3qiKX6N99!!A&fAY=j;UzA zIK|cb&*@Lc_W%8B|1c)$c=O9|((3u0+iu0buRRv~H9`L`ePG=6SnRrgAARN5fBL~|`2W_| zE;?+f&Xf_p8ZYOIas^JZVH?Pn>PmTw<8QaqP24zKZl|YGY^mwM)9%bc+f`W9mp9^B zgtV0OxA?;W4u@?dVl!Tr3`+x*r_WMo4mwpDQY;kOv+bDM3(N=UH{?XVN( z7>wOuv(Nar>n&DJg5ll_e_Vjrih@g(OW|p2=2CdNbh;Ftwq`Dc&zt~XHvyiuCN9Ol zc>+9bJzT1N=LGno3Gkr_@U(Ssss6(g;Av~%Qtb~-fcH&+@0$Q$Jpul~3Gl}z!1qsp zKgsc5EPIRj9;ThzAI1<7DQr-N1^?4vH`r{+c0{&OCnD2U)heT11VVX(GlhQk&3BZI zu*N#NuKrN%c=91hAuwsg;$w~&~Lf!$_zsBpxg z!$@R0-U8cQ7Q6+vyDfMNZ1-63R(OK`?8$n2l2^&GZrcd*o4g^&2TWt0q520Bcnb4D zEuxVB%$;4OC;t$1@^!yCY)W71HrZKKzl5B)eP1Yy88+f6(CXvGRAK>j;Ea z+pGP%d@dGoU4lPG2l?d?`KLKG`8_@li$JO4@lW<8vqv zINnyzFmuEUpJD3lO0E}cPF)XB6=K2A6Q672>rrhT2})k#x%8lh4;%Qo`8-!nCTf|> z4E|JQSMd&?gGC%z^u~*Zm*X54FLL4d?cF{%i@5pou|^LXe_`kG+e$U4NMVxh}rYiLC5$JZkj4a>`{KU>3M ztnog{pAGjV4aV%3kdg8cUng4mQh&w|{hH?!^f$<_GOcTDT>?E5WsWC-+N&W69TP|} zc2$3=9%&tuaLj+T9%*Is{`N>~o%gp#TK&AgJ@WMsUE^4N_pfG0S{=Q=W2E)cd*5S% z%#}o|DB2d{g?g2lOV3_oGT772_FNx{^rp!cc8a#7qhzyfT=uYs>yUJW{mQXA9P|hsl5C0aESc+& z^n+))jrOYV7Ut2Rv-#}mXtVw;X#tC#aNoN~?L9E^z-%30dWPq@$1j~UxK z>zAjkaaJ_cV5PAK*GR^*%9ua-`p|sy2>0g9dwWTA@2(=P!d1RHJB5C9ZvIv|T@}iY z?w)apqO?Y$iJjI=@|ionY_iWuca5+LoZrrT46Kz>0#MnFx2?wF$ewA>7WvVZX&a$b zX=F>$P{A_;A0HRm^Zsf;d@$A8A9b&^_ZaOJk(hl|dRYnc@(mn&D&Ebp+py&tFN80$ z;LVo`=7St~ua1wua%#wBIrJcFqvlH9;G@qhs!G8mt&ohpV2=q|I9-C-|PO)O717+cztGX*ot53 z;aQk;gi9d~=#-Tye};jjH;b{qTV!q@*; z>-gvSiwjErb=FgdGt$<#rYr?bu9!2*7~Vs?FxiAl8Q$Yk;1Y(H*e(>9_@(Tu1{bsE zGpCSQf$LUu}n7qF8WT*Ue{cs@I(!B?;cHFzHD)8ImONQ39n3P9<9 z4#NZZ3S7WCHQ39VHF!3w)8JXGR)c4PGws)IE7UMj{9FSc3@O{c-SfpcC!+V-o?CtdFeQB|0SA<=IUe^=914g(-j*zmA{V@8bGH`5w!m2|kG3MPPv65>-M#o7hR_?r%(=SLaSQ_T7fO(InT`Lc zX%5pYO7K~Ur>spdyo2LwCG3?j&3c6QNw`kJbrPmokMLm$H%s`KguN0TlyIkn&q}yR z!nCj;c|#KRN|UMXO&ga@w@aFK*dt`%^Jgpa}Bh(GBk#fCrSx>mrX4}!zh0;cg3%r**G$d|)9 zv7(R5hqwOWw*(*K^jR$PW&x8vTo`Zo7TF#&E?YY5gD(=jp$|M}!(l@oVF`==aGh{O z;0=BBN!ZYb_o%=d`Zy_Juaw_+zrY*%I4xmAA9bG*ctal-B#m@(uckahuK*etmKZY3XgkGHZWDl|j z{m*~SSle!fgtgvS(@?$n9z8f+O@lzWM#^KXQJxXiX1%diT4Sk~Aj`AxG3xurp*Qe< z?hz(|E)cwyvYOW$tEV{fW4wNHyh5IR4fzva^&3qhj9L3P{AdH(N6_A9>3>AF?^D`S|3*8*e`lZ~QTs#R5Dg9g zrT()Lv|oYtn2m6etLvYAQ?&PrgZZz;9Bdofe=@=Ny|TSwKRY;({jKs-+>lN$d>8*D zD8~B1I2!Fqj=!`WT$UN}Y3x9zCf z+fXh_<)(_V)}DrC6|K9LZM*TN^49HvosB)aYHLax+IHT!Z^aEAf%3{tl{+e$8`{=& z@2;(BZRp&#y> zI&La&-m|-EN87I29bJtZc-yMlvK`$G)io(WW?O4@*^Sp%?cE$GU)`~`YVY>Oof|qT z*5ACQsd{tU1T+;*yK6cscLgxUn$@k9%^hoY?%f)wsOVU`Y)4C=eA}jqWzBmU%T^!Q zUA<)jd3Kisc9ym_v`<9W+0(eQbWdZ|>L86fp?vFat|dLySFO9H;lPTPUA3EAYs+eG zsjpP^wqrwm`wh2jB0E+~S$D2GP`j)#G0&>KH{Vpw+mvnI5N}gig*IioyLRmyx7~87 z&+1*Z8}`6|dIIp7L^|8lF6C(7T+-0Cqq#nzZYwvftJ++;yJ~IEuC~^m-BtL(FLqaN z=xl7;aR5HK{)W;Ww=`C7Xx+85O!0{w`vN;x>`*)SCYf0z>ReKLey|3f6Wla;v z*U0q)y^m*qBQ8YWm$tdlcXdPC>YI1(SbbB&MB{GXu)m>d$AtE&SSILpwQrkXJi8Cn zBcIhStD1nWRnk@LuP-|=0d1$G-F(xoouSro^ls`&jvaa}ok*u^8>$f-@ z5NoZWar!mlY#s3m?CeQAmNm^X|LwlHp>=hz5qUacp0v`9mzQ+9uKKcZ@|Q{1O#Ox$ zcdqN9{GTWm%<;Bkyvr)~kLNok-7=yhzt}m!SayiM$MHpzAJ=pDvI+ED#r0gf?(*n% zUM|0i@qERwuk5$xvgmeKUmm~RRiVq{SJixZbk&z9AJ!a~#jm04^5knMn^3;>Jl3k# zwePMS&sR67G1A&F-rT(g>!y<3@cDHmn{JWo59Foh-C|8~Qw7a;J4zaBH}uq&t?Q|8 z!@N?9Iin17#hMMb>}pwljj_7(x3#te{rgH*H#DqnDBH7g_43t$vWCWGjjLBTtXfuj zrC-oBwCo8qH2E8Px?2NXUH-OZ{bavg`a?35PO(wiez+I(ftLpvB{EgiQI-44}BBVm<0}!LJ zs}+cr_PsZ@w0E!acee%ndpd$`fo_w}nm~789j~g`wSBF>%wOLT@;A1$5o167OiGIc z{*pOYd2jv1Coq=M;!-{$HjD=6MZ8m5dYje0XBe=EbU1y`tuD&DRRoQx=V};B-HNVzo4DbkM%eRUQYHOe6QV# z!}Ai>Drx(%^(2x_*;nrSzG(Uw#${7>eT?fi!aC5iwH)HC$JPsax+2w!pdVXLRJb#v z69W?T*3+ygL6(T_2%RkGfQqe9v2e`8!moyzH>s3$%?S!u*i`Z)A0 zP;??bx#QGdGfw@saq6E#J;`$x{ZldGbLKem{0bW)L{Ix$RE+q%b{zUf&?J(d>Jrso z03YJtCy_?{eiHOlzm+y?P>i-^IH&qLS-*y38QUTGoRsy3UCtQemGwrP{z8r`Q|`|i z@p)eI@#PA}M*TggpKR=K19g{lnLhBRdb%%01rPG^@)-W5`Y>+&QF&4v*ef_veYMPU zPFZh^ZwmMszDB5QiJxkzFOl`rQE#-Rs*lS08K{>-<~6-ay&N*H>6i7?uN0Nn82k0q zuUfwydZ0r54cQ1LKc`~l(=6*}S@?7+eAK%@eM&v{Q_a$^)T{L08>ij_`%)n})$uNu z^;S7o$$IN}&q@PX$2%nJt$f~)_15vet?*Ih&zEsU{MGTUQtH*YdFwd!9_cj12e<0Y z;#2C?TLCAPdX;`isaNTJ(!Z_zLrT3${|lvF<)0>@js%4N+92tAlglU!a#9yj}H|Ui0)_Clb^~7I|$3dlD-b&+q z-caggs^RsVXyJVnK*_i>1^W_dK~&&v3D&{A*Y`{yk6 zMxM?@e^d-UMm{dF)EoKO*#9=@&&u(4TIh{@+;6Ej@-Y*O?k<0QOM4)AfMH$U7}yx- zX>Vx0o-fqbhUjmV6$hJI1I67

B4$S1iQH-_qXF&G=bkOQ5x>q1%t^mZo60zpJOA zp{c8j71!gRKZ|91Pn!_7rKyWeeBmT7Dq7n2bX*4Y{-$78OGmp@LrZ-SI;m>v7OG&y z?H%1s#Z{YcT+-DYXxJyH)n!3xoyMrzDJ>5-IE9lWst8EOdY1-e?(4^H1%{Ddr z_XGoNO_w3!9`x4@8U5`2Z|H+3}jw4(dVyrSe*Y3&F!`nx;)ja)D>Ac!aA z6rdQ3Izn7K#R<%7rM2x19gR)fTUwhM|9lx(VdG^CJrLa6H35E7u5%2eoDK|-{Q9EpGC<3~#}AAV|CbzVTLshbM)=ow-oo0S0UJkp zj*8k7Z?z0cxsB^<184Maz+s$FomGch{~tkNrMJRAll_m^extDuBiN16-x3H<@>|R2 zU$oW;*h-5RiV}|hm~2niscNyxe>?DGBA-fx^UhaA#RXYU!=R<8(cZA%V;H|x_tYfx R68+1}SK&yEbo{LC|38Hk;$8p% literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0a41d63ce5fcc4e48ef95d3f5f0df2d0.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0a41d63ce5fcc4e48ef95d3f5f0df2d0.o new file mode 100644 index 0000000000000000000000000000000000000000..777d73529ffb178a31d845a546477f4411014525 GIT binary patch literal 34072 zcmeHw3v^W1dG0=EM$$-#(MUWj0Y-x?p5`F|vZ0OB(F-9NkPwL1O|KY90}|*lYb2SG zAuB?pjxiWa&Aq{^aY`CDmyscLZqqvI&1&SvmpBflSk2A#O0!moOo;^8#S>o#663kw zzaKN_91RGt8ym`onRE94|9kKMwfFz;J)<*UDqXQm*EFVz#$INznu0Qx*%Q0>@LLZ{ z1*MyHSNGd$f0pgG{U3Uv?K9IDJ2Ys^?tL4tbGGa`|Ay;%ync;)G##&K>3VJF_d~xO z!>)d?UJt!rX$xJmu|xmQ#(IyovqPXg+?~M+zMJjo`=FS;`AeN0nwHvKebzRychEMT z`a0a5&IekI}Z$8nD@lM-c{&l-?ziFTQL5qH^SxTFNL?g zOXK}vr%kR%P!8DM#FOFS8Rx^vTK1J6Cw`@R_~MI!fGfaUNuqOTvoPu{OA4t=n`2!7|edTr;c$Bh0P&mZ5nX>*iM zlK&~+>xb{X4Bx!xdv~_)b>G1Ej*jAcM=icbKB@e#-{ODDUvK1t*LMCO6e*tg=FkKB zp$q>~bk%KVX%~N4banW}(!O7t3yA~uf7Jq9?WJ}&lXQOG+3`4I{UZ! zwV{2ikLD56`X5;ieB@cy7x`D#cj;yQ>f6Pv@7#Ut>c1^wedp8I)eA}NY6@rzG1rFn zB=ima3+o$tKs$8tf9Y2*zReB|{Y^q>=$q`&#d@r%<5=JD4<`ba7#ezj9s2bHp#2|7 zp|k6EKSFa2JRiEY^A*!v66?E==OXwgS%$zXjrk7A1Q{3i*t3 zv1E4519>j|ZE4>yh9M7h(RX1SJ9G&;8GbP-#I)|Xaygji>THaktg${?H$JNTL5N_#LChL8 z_JJqTRbwKqX#Y{=NPQXmV>e5m>j}~LL_baQ^zkuL;*W7HZR(YeDmCt7u1wF>l)0Wz z>f*wX^K(TZvJKgU>_q*&4ZWU&{#=Nkfn-nLcJ%)ZTvy{&f%|S;_v3m3uYbb*X3YZo`&@+IlAV498<5Q}z~*NcvA#jrkL;j)`n@%* zFI>#}nBM;g*;Lv8@;6~C`0_jO<9A`Re~0~I{=g1hJ_x%S?ASO}=!5+9qsqe2wWOje zWWRg*Fze$8oCki)U-WjX^{RJj{y}3dcDWv%py*7~D8L70onXuHa-j3=xO(tP!YiG* zXxI^DyD!k{Yi?&%z~mBE$IR1Han&_u8*9^a8)Nn{Sn_1CIhh@|UlMt3p+e~FAB9Yk zY;&EsIs0G4Ro~7MJ!}a}07sLEa(;Vbdq<$dTZcP!?P?Kqeywe9Y;6d#`K@gozWJ?> z&F|RW)VifDRom@h=Dt?%q;_O`dzdYhZKdOI3@f!ca+ zdq=IrcKBM`+XDWMK+~dyTU&fB-sYyQt@#hN<=6hlVlY5$e!;@pMGKeK=I7<*Em~NM ze+w7qJ)Hm0lG@rl$SLHy3&dz|^agwlcaI0+E!rLH+_pT76!B^M0 zwRTIu+u{?}u>!pFO%P18dRiI+UVkH>bpAkFM_WgiAI%BZ*4p0TZSBar3*5Kf>u>P| z>hB6O!!%qLt1UTDE}h(fM!;QQLnV&cXF2q)nZnApT%hY2SOyj2TCCc&i#$SRA~(L= zOGECZA$M9@7{~SRr6HfO?A^3%k#($qn@>aT{C$jCq}O1BHNS1kmUdr99;>ZweXKT$ zSy{TWR$yxLuvKgFw)5y6#Zyb^b_9F_HtRON zV>hFoE=P#PSc&>XLUFH`oWksD$%?y~Ew$+tSqPbFZzeT(PEhU1{~2 z^2$}Ur5o0kt|}=lVe=~8i`?_Pf%-;wdkfA39`?F1H1SI3EBO4w!s~YS1mN}s4o04k zNB8OC9=FV%5xc*c;~ZOKDzk5~!!Bt1uwC?hAC1o7fjAkuoX+gBZYN7$E{xV~G`gX# zGq& zf7N9h`*}9&*=LWi%K#A54NkUuZ=#)#yD>YJ9e1-Ib_p5Slfa|Do**WY1kw1VT{_!L zyy)FF!OwlDpiIZ>oRb|2*zIV+BK-%_*x&+|8?@bbvCdqGgQ5uP!^L%!hnNH%H-6QOO4H#3-|N9j+dvGlz`-PTj_4VuPa=BY zbA$Hrbk-^QCwe~31U>qPK5Qh8zv_L?e=g)R(1V{eX)o}D{)nFIFQY>veo)s*K7Uo% zM*JoPC%}r_z6lC1=);Wgu7Gm_@tb-!lEenmSizs8f7H3Wuk-$#X{h<@&>#ALaxaUJ zJ?swHbFh-#zmsUGK)y?9?8{waBitN>&vimS`q*_fH8ot`*Rvt-M4fyo;(&h8e{OK% zMDn+>T>qlZ&b+8A{gh>WO_~?>ME|gdlRG)t1N;;O*!T&2d{OV&nGj*`13*3i|H}y2 zBIJ>gMFL;irLoJzfF}H7c|cF2D$_pV{I~+b*KNOoKG44&urVj$NeN`%;B-EGq#yhb zrrA@;pRrK-H$y+bX9RPn)4B=&;_Y*TFvn1hHj(5T$Yy2F9R`*P#=k)HGp1jOM6?+F zY$xe2BIPUX!ZJY{Eb$uf#J^wQ>lVPjX3QY@xc`%X1#LF& zOUmDZvuBeyG5pM#(fX@=zHoc+`G@wTFFX3@{Jdq;%d(CzT2~{HJ>348uy3@0zEIW4 z-tZ61=UhHtX+HlvjTJqo33BpZ;LD!-scbIlBl|Pc!(VgBp4?wC9{Ja1B9ZeUBRG9J z#Xs(kKcoJaoBHqBIW^`_g9os_CD_Q{_@Z!n0sIT=ebm3mf2JnTcvHFm93*>Ajr!N1 zQ!5BeNr?I@)`RK6DN~3(G3IYmGO5n}Z4cVJ&|dl5D$?&1m&xB8z`KP#k4@!x2kN3d z`J)|p>2L6F=y!Tlzslak0NML3(l_0sNPW|IcIBUg#n4ZOz{~kOQvVVy+e81;N!`jn zNdNp3DZ*pCvgekV`s>-55z}{Ft3}^5zYY54^EHF(J45OZ{+|)mcU?Bt3ztRTE>}$7 zm836MY<~0kf%a10=szO}Z9#xYe~(yyX?}16N_}OS<)!|j>%m|lE6bW?l5aG9_3T7G za28Ey{mS_~H^@&Wh{#ow36A*ucQ zB>&M^e1|?Ek0rkAhWMTxC!Qk*CX-10>VeEyydGSjEf3f!avS2iu9JOvd`JERf5u10 ztnw7!-lD*=Q(&HhLFw;AKJ{rC>#8%n-9%6A;jUtarQyvTRRekl=l4g+%kCHvgi zKjGgzUv$X)lH}n<h)+biWyB|;9>ZJK zGrqprT{51G>feoxZe|7O~sz(4lVj;0u_|!h?j~}G5XTluGB4;CM#9u)V z=tcV^%6E`Y+xnjXuY7$=b0I%-NBe{P>4B;GB+y4(n9oC%sp)}ieKOT2Vgg}&jF*#z zI^S=iKiD^u+{I7x0rD0EGc%*{%*qe(--Ge4&kwC%2i?HC%BSF8C=?%Gz?Tq_e=#@e zq~CK3P)E8z9g;j3X#eI&i0PO7BRRpM-%MJn{BGW8{9!l#d#{3-ko@$J*jUqnJ_iV?B!Pw+1!ydX4!? z_~?2)YQDnfw<@e5{4sMPun`yL`{_?GUm<^po3Bh8C0}t*yK%mf z#q$++{CoxSWVtE6NAs1SA>SC8uOObfv!eOR5cK!)^N}oCUwOPlf0ld%@zRu!I56Hc zH~9S+`AC*n@F_lz$dB&ae1!bxKV?2L*A%}-?!PI!K#}|J{Xu3Y2+u!oeqYx4(x~z3 zs3~3<&e!2Tw-c|T`){mo=Kb~pe7-Q+etQ=eN3BmhKGTsJeO{D9@#H=}-zScTl!Iw5 zQ+&euY1&_Jq4<=@_vbvGAwNprW#{=5_5*3`HRP|m5)vt%Cyh^YP(D-;v{SfF;Q21r zKXG1kbUlj~&k!%t|AY76ruc5I!2uz<)a88SRH6nIgZt^>}95Zet{3WF=;a2f2sZW$oOVD|H%zbo@9w{vGX76{ucB1CidHt zjPb1?I8JBy088u-X?|pm8xISK{dYF@-&y16^Et!*dkXCr4e`!N8 zB^vK0PGoM>DPKLm0Qz+427SW*>AFRqrv1q1`RgBWzF$f(=dZ|b%l6({d_#U@_h+QTd#VYd$|nV_yJ2Vn2%xaQ;))^PyXS)K4#y*WL-y(#R>k6!-$ z?;>y6S!vDcX~*={H3_S0up2>0|H&+h>-qG>QB9LyTI6+9V-L;&{t5w+=ea&nc;e$lt_3_46R&-g|IIPoD;vb4kILg`Y_+B(>(F!*#vx=+DxznD zPLn(qf8bQymSmGXBWwwg8Mm83?|PEZdsJ>3DGoj=w<|tADz{t69j}k5TwsJ`Rgz&y z6oX{;Ped~NNRJ{*m^hO(QO+RUjW`=Q2hE(@bSvj^b*>A}qNAb{qwR&R3(Hzl;>}p0 zry{h`n02%&Kn;2vfhTTA2di~`7`2@k zk25{B+IfX6R~jz=I7F{acp$63vYFSn5@lI*mvD^CbfPZxAQi1mff0uudLvt8G3mePL>e{Z{*k z*XCw;Kf_nl(MG$Oo<{4#ijUXlr^k(zyV2^#nb;#^5ce|SR^;b5^s9L7)g+;*HEMg) zywsDT^U^ICu(pq8T*9w!D_`Hx4h@&Ku0Khje3+GVak3A{3kSi5YNwTpjS z*syk)>EE)nUxKXJ*q_Hzud>vUtzLiaGOqT*_eP6vH;M-2IMPeL$L#u4RCz#!R?ncG zZKxBT>Rg%Hm{eqVv;(kd)4;}RP0w*5HhXrHjx_{ktn_$`?FuM|lfAlCFQ`+m(@>|b zNY^xKj;hD@88)3pDNJ2!FLXVYa&x0mE;5>K;^ivD>8<1fZm?_6v5rs%NUV@XNXH|x z*f_bK$$Kq%9`$3iz8JwUGP92C28jWGFV;gJ@G;XG{b{U`pTwA-Uet`o{inx`)t;l( zjWg_C3>Rz2Xgd42c%#nm#c=Z~E@Fj>4VB*9|MsK7Co#H}+jxE6K8M!c|Il^Q zOidhlT4?1b9=&BhWXgv|w2$sN`5a7$t{}Ahpjheew7N+@aaKO7Jn`t&=8jfe(BIMf zOU}RfEbp^+w)a`ruW3FRcWffo$D-bz5j!>yJSY~k_b_ABQ?vF>U0+p#y*A@#-`W*; zA038cx2?*kBk<2kJmNDfI+C@VqYEE5;rqq+kAWn>f!f(r)++qs6MZ<=VYm0Pv9w1Uico5BVUYs*)(> z%{>1n?z4S1x!F_C=e+Qp=f}6O@6P$nZ#UTfdgk0|c|V@|ql0Ow<;^JzFyv?8`9D>V z+KoS|%gcp5D!>z-|MSu^#x|GXZ_Dwz-t&J*L1YWM#(TC8!GA$-C^nn~MZa2yy3+0= zfb)ZSPIdja*}M;aG-M)L%Hbn>5YOr2Zf&9$&*{oA!Km^irj(6|C%qVv;yGQhew4qc z=X5DrH0MP;rz=x|qu1zjx+Fx=M+t88hGsfHrn9v7$a;PB17!h-tqh`0n}B%jeDRz{G$R5@{sa=(v@QqY46e}(Hh5SrhEUA+A zd$Rp;LtDDF>Ye-sm z%2~@4Q{$_4%Knd`@r~LmI%-dJ*7h{dNd*=+WsDpXr8`GlsXftji)!>8e=@-PERI6t z%@TQ~L1p28=xV4F_&e}A;)(h<71`cn0$Q95I9gwb{~`+=9^uc+PYoT%v!5T1iNCKS z`$>-%?auzOxGuh5T(fW?`AtNq{|SIJK8gu7exm?V$HpRQqoB2DzbDT4UzY82B_)kd z#iBmulkGj0@sG*&%73XnxwN(YVoctr;*9^bO&~^b^LNIq<10U^K>HrF_gKd7m+h7R zsqs}i1)mH+y!LO!X-`wa+Ma$a3Q@-oGn@2(S+Z;mpEnKs%y|A=Ul;V{|3!6LZ7Zo*cT(qWUTVeBB@A`($jT@@->s!{>Y%f^d z<}EB zFWBB(QCYKQN7=IGWslZ0`m5J8FJD;mX!(xv4GXKAH?=IwuUjtK7Bw$zuG>(Q=i9{l zU9n-|vd+4erFr1NU$^j~nr%g^wl!?1Zr`|J)#EE#R^>O8)pRy2Z{C41TER!zy0Ueh z^$VAF(RdB(E82?6ca+pET9?;cwzVQUZrRpV-omAArDa>!N}95bt?ODgELykirsx*t zk4Cp_Q%kUM)1pe~qos^w-m-~wm)};haO-Ms(W?B7En7>s6;^j`Twf4~J7)Qg4I9CC zUD>k7>$?h?Aa65t^>|%zL7s0zk-x6R2m7q6sB2yQ_)4mF}dUeUa9{X*EgI)7v9nwrK2SyoL=GomRVei~WypY+C$?@;7y@ zZE0=_)Nar7F3xXwXi>edU`u`dV&CE=zAZ}&8WwFS*pk<x&(G&3Qj4+ncs0{@P1UBPX5&FF`29#_KwNVT_j7S?++#jT3oezl zsrP%H)eqbMTjGCrd^Pp!&gVtXMBl&(#M`b>=xMCO2Krak{c`8gF<;l8x63-weP7{l zJ*Dm~bW;F7r*OEIsQVJ9>skHlcEa;N{FQL(mlp#**AR?mS@1+(V!%g#6={X@r7<+X z^J1kXy5^gpQ{7vyFQBcRy{C>OOlbMwI1^0Bp_Cm%5eu65>VW_@h)a!~;5_nhfQk=P ziTAwcwY+OIic{h}XIjwnh&ZB!%`mP)3i=p7X1Lr5DIl(b-njD?$TnOu3_CX^7llcJ zEB_bzV(c4`OTk1B;+?)!PV^oLRDK_ZLgUe|ybk>LXdjQB_BZk14_^oVeZYyIGu#f0 zsd=SchyUl0coBV-G{LJ<4jMDwxUWlj-O>bVel>xf`nfFOs_jAvSDVxl*?*4gr%IYk z>G|&^|9H9-FKS*rEBSm$!qt3wRrb>_4XEJj0Z&(hqNmlUQ^8YwIefFM|FZzIZ{kYy zc^HSvNpWFs;!5x<5^k4pCEqyor+kM{<0XEA1)eA2lK@w33G@XCpA5Kh%ecz`H>Qfm zB%H>@X=1b-mv=O-5ndwYr2Z9eM7l!4t^I6}@Tr!5l)k7R<0(%gxV;6{0KrQWT4&XRDey#3NZR(W5OaBDw5m2j)PzcBP;i67%?c0|G-w7~lf^hW!>D&bcC=_xu?h@VN~mcK8Sa5b*-#YPEN<0|-J1HEzF zBNA>M_ZJdw9XD0ZTN>9mUtAJCV!lW?^`m$p0X7@p@`#7ab=Uxp{&5d5i{)TSUN@&^ z>~jXVQLaV_S9(zN%@S^v_XP>J%3C5sjv6;lj=RDDH;x;Wa5b)?-zDMJalb3!)^RCs zp+dO}O(~^2ifdF9+{0^(HOi}ktGIE*0$1^&-vU>0V8{YjORqZrRQ;>Szga+>lZ)aj1Crj}_X8gJ%|BWa5xkh%g zI}hrpwyCuVKl51E-hOM%T02_=LruPRHu@$IUCq2IAATU9d;zb99}T~IB9KGyg;3On%olXaaPK7tB*G&DgQyd}QJn(BQfxX_-j zQU2WB=qzr*c=YoXwXO8?cq>+{TUpZNZT8i7tZZxOY{u|6x%=ZbXm0a1)ONJhHgLv- zKwwYsDXKyiqlC-a0*c^P$6~n+H(@hxU~7AAyWdwo@^?i8Z4KTIFPqQbg+mG}t$-vY z93@ER*SB}%KXlXDxI^~BH*egyKbU$u8+cQjkH4E2^RZq1cIzGb%dTR9{h0525JY6E zW$NjQOi(Aw{6it-GOenN_W1;pX+ae3k#}cg98(AtuT*^|ZYbsQD1f~BcZi!KG9RPQ z)+pY|a2(CS{vJRY-+7Us=ssyZ?y^GRs>Ofvsaxle6vgjvpSE@)H~MBrj|G{|~LJBaZ+8 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0c5fc19711de959b6135918a6874bc83.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0c5fc19711de959b6135918a6874bc83.o new file mode 100644 index 0000000000000000000000000000000000000000..a372f29a862dedcf133c917c567dd8bebe3acbc3 GIT binary patch literal 317664 zcmeFa3w%_^bw57)erP4FRx%(EX!QVu07+oW#xGd$umC^61Uru=mW06un}@s-aAhm; zN}&lj#0ZkscAI|;j+3y&q$Eunlcq5w)Fs4qF>TW{&94SRTudAn>*qR_V`qQgb7tl~ zRd;fwPuJd|4LSG*7ZDB$BsSvyW$!`KF4y8mP zC_y_=CB)9tzRG<+$M1}Jw^B>^Rlmfa4zDdhTgjG$lgP~wjY1H zGIj7Qa9)q^xi+Lf<;EQ+?t1yNC#K@Q7WX~g)DUg=|nPoJnSe(=PL zpZWS={om~$%uau1@Z1x}24DQqOMr>ys_c}tNb89QPh?;E8shIC+(!ApZ`)^I9=z)& zO6$sO)UjF6a0}cNHnpxRx4ckvFuKUc1AD-Pm z_`trsgNwfYcZ27ye|m7r_Lp&gX)rS7n(T{(J5DSPej4&TA^Gh8yTQdT{Ah4Z!wZA; zFZ>cT*JiivLt5(}Jn_SIUmrZz0R7zf?BL=he?OR=^3vb~&`)H_>}jKm^*#vo!;p? zsr+-}#KBW&GjgTk;A;iqV9qBFQh6fY&ebax46V6i!O-fUXK3|ZxPPTGyJkJ^X$nus zuGxh9y%ZjwUGoXt`zc(JUGpj2AE5BK?3xF0-$miE*)@NR`=3y_JiGdfxE~YLUVJ{h zyMI}vJ+myj`_W|^KN1+~ejs{q^;Lnl&V_fUlQ*9AQ=Yr;jUH^@*MDvD#&iDmuk>FF z9sFVY+`wCuzjp8Boh$q6cdnXSzw-`n{mwhTKgG& zcLf&=-Fq70Nn^6B2j>s1d3pZOrr*vVy3e~{sCNS7`~>p<8u0hx{s!)waF617{pjrK zBIu}kRCY}Q@F@HyU6Nfr7Wc2Hyq$k^*6K zWOMC5_N#od+|N3PTpKs}hgPrmzZLczAn<@e{PNk?UB#>TomCcXoCn|5o{SU{;e&${)OAJo!|Rd zc9BO6rJw$2wxju~q&x69PwU9dmWr|IV_`gw|e?l5}F^E>4?;wgFE z&VK{)xgPl=neUpA;(Ru*qx=Q4k-n1G8?W?cH$X3IzcV*GeM%_XIK!81K)zq}iJkQ= z;<+}I#og5{Ls3uS^{MFh9M9|Vyl;f(Gl{-*IQ>;bKf?1U(U%RUFCzL8o+pU@lHv5D zh<=3UV~Kv$aQX>EKf?1#L_c~seFM>t@ceS3A2XbOHqnppd@j+K52wG5=tp?InCQn2 zr@x8lM|i%B=*JDGzm@1mc>WQhuNY2$8_|#O{0^cYKb-z9q95V;dZM2&oIXwT!=FcC zXEcVodqvAoyEln(c*Q;s#!+N{)V4l_{_52w*J0eU0OObW**ET-hjGo^>|6J>WY30L z-=wj$ef*P~=Xoyo8(;Rtsi~o}7?+=dZJ#a{*)yX(+0(vo{R@@e?2F@MfAZU!)DY-* z+Ii%D5W?809BGV2TH}yr1=1dmG1G)>TSnaD`7^P+PNWW2mV55$Lwi`~PYs1T#7=7e zD#wd^4!`W-XA$HfdP!<%nkV|1M+;MfhrceCM@mu$=M;*2_JBtuf-u_8z5@u~oVqf4 zOX|vk>8YMM0dY_2k%jlRM_TSZMYzJ$P(66HdAzUe`G#0tAE0p4D;?hy%cn$ALvs-B z`*Y^4=vKZ`^(~98--Q;@$8aUsjW zIi5@I?)WyRcaUW5dlF%lmyUx7zfE2a6ApQaCig6xg8aQ-VWsDmzeZTew&!~YLpG9m>r)6r-*c+O zJyjG|dOu7wHhuR2NBS-hpIp}FDfng8A+7V+D=mkm>?F$^&$3re0gixnjwkkt=Lf){ zY$#7vN06r`Tb?SMdGee>p72cPX)5w`Kl0Rpcsfrle}gdP39@%Qjj+m7%ikib@>F*e zVU?%D^jzhsgTkcuf!(IwJ)pJgeGlNelugE3r;Jtq3o_!Fma!2s<{;xKN>kay;U6Nb zWIS~YVI||?A0bRK!q)dZgD}b1C+=yXu#)i<(Lly2q{|ZV3enl+Yxz6Kx6~%zG^czW z{{s2&Ov_gd`PyQzS;W)wbv%o(lCSUY5!N<)9APD2%X0`T`A*SuC0`$fwS1_DL}!=p z6yUmcn|zl!<=gX5kPpwad^1VDqSRp33Fgan%JUBhEBUJa5n&~t=bsQ(Wqg*xO1?e{ zEBUG@tmT9LP64*d*Y{({*J+dQ8mD}xo`ZaNrsZpbe3g*zFyd+b?fGYfm3)VPg0PZr z&%Yq7qQ*?Ud_ z29E3YX~;cbllumz+6rhE4_A{L0HM%@(YBOUh7^!Sjl~u zo-4UKC=9tDxVPV`+n}}Ub`RjX-8LDQJ7qllH;@s}w2U>7@qWm7iqh0|``-~(GM+k% zu-5I12rC))oopTcDv{bdzs?J^z)TzAMO z<4UKDRo{b*c&25X2pOO9rv^RenK#$(QxsM*dIk|zGM=KalJV&OMp((%L187Mhr(Ki zH~U^W4A?GT$E%R+zJEBW>ijh3%4^~x#W?DF*uLB5kV`MRC*?fFZ{hi6*8$&fDx`3@tV zmT%7+2rKyx{{~?t-<~%SR{Goe7Q#w@RTNh89VQwrUyFF94>-GgRlkLN=WX)!I^{d{ zO~{95TD~hG-<)F9W5m<)wd4?1@^uK0DPM~RVI^Ok7h#gmDRH;Ay( zVGD(ojHif(WVF?jKH%*#R)ru#+$Q68r;Jr!10K(`jI$tPCFv3Iw2UobgtZ*BG zm+NsyA;L<&z9NK`d>zFIEBRXD2rKzc(Q_qVAHreCr^ma8L08vgljjRgd8!@<9?!Hq z*Fv7Bpf67ec&aizMPVh6rxal&&nXJ4GCf*`u(G=j3M+X$6s9swS^Y=VC3!jnT$i%R z__$NXmfgVPnU--jWPF}v9L2n4+ecw7;~0dsj1<-~mLp6uVocsaVJ#ztNyb`h{JICv z?B&`r7BVcg$@qj*#*Qxlk7ruOt0Ck4kg*T(v>rRgA*{=_0%2XQ;}KReE}Vd{lJOKh z*X2rKl2HsS+jqt8^c2;PctCHLa}VIUcAK2v%9Hb9;PFh$c{%A40VI^k^g_WGAhz4>>`BuKN2ROTYEj38L(JRDPs`U)i?G(?M1-{-Cn2o$I2mEBM|!UHNMS7>+Udifv&+{}hx9Wx`Hnc{J2e12 zo@x0mgM5!dzCDPiDe#Y15w-yWjT^3fa|IJF z6xQ;gANGLGF5e!&b-Qiy{i9R9j%~o>nU=2}^0m=CkJ8ijHxprPf0rVx?e8*#wf!|C zO!A?hYoV~VKcdm{Rbg&?xl_Kwfa?yl96a{}F=YBze&&?1?+-zPXTxMfTvNs?5Y{rz zLRiaqCBj<9s}NQ?Y@x80k!T>}xu?WXbfMMebb!XLOV8Divni~7yx}X{cglB3 z{cnC#h&6?xeQ*8mBJ2|=-uI@jVBgTA@$8TgU{*N!~}9+64h2!G?rOM^SfN19sfX@2w9vFw{WS`I$yyX0>8 zJeR-trCYN<|J)7PU;OFv?5}Sx$e#3-L#DAPxE^z!(7OK_3ZkSz6zOsi>LXNue2U(**8}fiW%Erx34a_4Zhru!KeFC_;&vQ zI`P!Mx#1)5C4V>zzc2fNv1V)TyU@C)azT0TY(d##f5<+)Kcu-plp(KNUiq}}2M=sP z9O9RWxTE`9=+6zmM_lq-6B1?QTMuJh=k54TK$;h#!9JBCouBD0=wY`P#dFL>(#I~6 zJ^&qj8D$Z~{b}6$p}%)3XPwuCOFGSel*@7%=aMhl8CP=tNASY>rZwA(DQBH0t+QsA zviwchoy?m54=<8l%X3TAT{_RYOZufQ>6W{st94P% zziVB&%cU%DpnZ1J4r!hJ)+N2qW%?n|yM*9lxv2#=Dexze^oxoqfY4 z{a?DI`;JSx=zE~E<1S@scS)ypw$dd%jW68T?>f-Av9o(XccDJ&ceS%mx|F%pC7ssU zXqWUAF6ojk>1sjuZtLuFm$E#Kw$+V2Xq_FqNP6A(-|Dih_2$fQbVE4U+ z%;hd+c{*^B_1`7^-@Byy2bXmJ47zvQ&R#?sZu-0vo{Q7j$6e;J-zD9rT+%%Nx_4V= zUvMeQG?#R`ZZ^53Z+1y{rAs=UzV5g7q8`25I=tScZ2yY!>BaQb&%2~QQqG%QjCX(6`PyigvS|C!ZS9RN+d&=gt1fl( zgiE>uF6ngoT5jwsyr3Q0=Meevf8;XVKX6H>`M0^GU+I!=wM#mkzShq_zX$d7UYD{( zLFcAD)_Ev$Nnh%cZnR6fe}50^>2#N}{I%Cz8>Dr1*d_hbF6oZBr2CWiKxZdi%F^kQ zPV21OC4G-ex{tf0qc!Pwd))Ckm$KBjq|-W^?2>+(OS&ePbUJQ zq|-XP#wC4=OS%Ov>2&(Ku6_&k=-t-g2VKfWef-7D8Gh-K{=7@NS6$NKrOtOiN#Y*oXGef04(h|BLZ@{)_SMM!XA^!|4B_9+$h|{Z4abtd)s*SMsc>5{I_B^~W6bkj#-GUSRb>XI(Mf7yd}5NF=)Yy6MgLwCNK8%^hG z=ynfxBrn82;)lKI@{BzMmj3FN?|4L9oZp6B(zv|BL&5ZHA%0=RxEuErerR_^ ziHH-MVb|TO((6~P?Gh+kQA=2F#3Q`eDdF{cg|E=(@%lW%Uw|$}UA`iLy=vlqc@AKU zMtH^dq>PHJeXEr3WS^4rZV^n0St0<5#u8;i*XpkBba%%Jg!H{Mt-G{s z-G)_d>gn=T>$^6jZ|vyo+Pvo8d)KZq;SWf-YsSo3E7~q?TiM;yx$1Uipct*_=vvjb zvTH36Yu4X+%bNAwv)a1Xb+p~FA-%4n+vGF9qr2leel+*P%NMlGY+JFxXj{2v9WidZ zbY|O(Hj{L?2yIQsr_9&;ARAq)JJPFGzJD^3Ue&cKy$Lx&DWzAfeP37`^3n0W$jHi7 zD>mNQc1OBn-6~l+`42i898XZqZtqyT_Re%i=W4EWo#_qT8>rSZbo++&UELk)yPMtz z=6icb=ekwt+us+(jAG-wbL;hzN^8ebfaJcE$wzXj#ilKbR+P0=PK(}vP+0oXuZbp;Q)_wb( ztJXI)x81q1BfT=+v1V;sH~PA^+c&IjTd@iw*tV6c(En|?ebve~47JLBO)38+?$^#&zqv@)L8gooQoxDGif1wWZfXoIASjZMy@5<*v0G zHe+Cq4*Gw_TXNd+f6X+-(LPh@tW7n z&hUX77u?t;F{r5>Yu9vijXLbT@(-gKfWkT7p0Tw&(!576W)SkM-e72ok@tbi=K&U=2M~g ztX6lOt}a7*O#FY}>HS2dmv0*vo!$c@azku=FlenYGk<;i-t_!OS|8^eNX z%2^7RgjsF`ISjMqi=OM&fWo?F`4d*yjVsfChadF-ysz-fv`3S4sAJ-|Pn@q3eg#*G zqYik2IOKrGizgj$g}{zfTl{fiuLB+{c01s5@vs9PBL*DsXaR?+jo&Ddaln@d!vU9x zP6u2nRy*K?Xm`LR;zJHNE|xmrVzJNx7m1VuE)=sIa7;8g-~v(SfTJSmfFojz0}hM0 z0}hF>1Lo$-rjLNgc*S{-SC<#sIS1?$8IK*0df|XQA|vd0HUi?e#4pI^JS}K1;UxE9 zf!#MQC%iXAg|E%$@y|ah&R>P7IUx?zdqfUh%W-u-sqW|1T}+kuxVk6R{UjDLDV~@v z?@4t}se8M+XViVSx*t;alj<%SWc-x6cdGktbw8l)ht&PJx}Q{c|BP883B_P9ok-!g z6u))~j}juY5(~2U9mnqg8rp#_zQuz6&Ec8+v`#;zJXdqt$XzK$P^Yhm& z^v3g}vkWr7A&(pO}ar z28f4R(Vi~^{bKz70CDgEs8l;>S0IdE=k4IX3hCViS}dA%Zh*bx*VKvlu=A#SL62W2 zei{4@;CBeW^Z1GTfX6R|-vEB%0dlKsNf%7ZvemD_#MJ8`6$|z$AHJi9kCBEekbvZ?+5F zA5icC1wX0alVc=3@r!z5Vkri`nqSo87YAVb1QWlgr%og*B~0?*8zZ7!!4%&E_$2Cu zj_*PI^9rW)iJxe<_z^y?V3LRM$vb8KmVOL^BEL-veptaN1)o%Ke6^&nyG_z36+BD9 zO$uJ9;FN+tq~N6rUajDE1se+9y#{52{BKq8A;7E;hVi)~1s_oGNd-Tt;k2YDJmv2s z>L0}?Jj0JlnDQI)6p1+M9pO#-A4qsF=vki%ruYn_e%kq+KPF+)SJV>}WS4}O{Db27 zaXUUP$W|?UTpU-hj!*V$;e#ThV2LjhWYZSDNHo1@=N}Yg=N2CFC|JiQ+qdvVVnD$< zeX0)@J}Ba7PY9;+!8cY!Q;meFd?>%W(Y_F#^u_jswuazY3|}VBE12{V^@PPzv_XVl z&3M3QiwOP@!%4K=5|;D{u@vnT;Wd1GlM_y&{j%dzXv-Wh+BG|TV5dh+1;08auRvQz zFv&~h+xh>Yy#Y-5Blyr4Jt6{lp`t&8c8y??m+%97B)y^F4B9tJPuK4Q3f`^i+j#}c z`i=VcbxCj5Zw1Tx3;ils)>qh*#w&TmpUL=7D!56(+Me6<@B<3g<$cHjV^4|$4vORf z$=|de2Mqgkz(wMK1BU%6nDj&T6aR+fNBSlENhx@*lD|{IvlzZi>{hV0pW_N%&G@ho z-<145#BfqHDOlS}dmcQH2Or9V&*#BO^i_8IX$Ne#p8@n$cKh(nDi!{Un26tj{+jwd zl(#>Q0T22u{PE8}fqEFUbLBtr`J-X>WMzB^U#^~`4tUq$cPGU_eZ@uh->Umt>i(v> zKQ77GZZW(veG8Z6J=Z=CG+}&^ofEzGwak}~Ej|j)O>iKrA%%Px`AIrf;f$oIlbJ+CR!)3PHY* z{tX6lu0Zhu=`U~%>5JPtv?p89{CY8e!LzViir#SU@t%DVe1p35s%Pbl0A#LprIWc2K=q-&HwZ#S|C0@(5zV)}`dJ^}=@L;g zO=vwVeo`VT#$*`Xi4XITJysYRM*IF_=s)t3koNsKuCMv>qkWB%f$9E<k#oo;TjH ze19i-^2_hPrF^%j{Oj^<`u&zK*g4BLqtbh?%J;|x%JifEUx964b(%*0Sf}OK` zGw*TvK68Qc_5b&kFZp}OVbz(E{-FH&oKnE2<}E@coyq_1Ro>+o{#5m-ZBjO z0XO%0y)=IlG`~B(1oIu5s0optFv#bACMF2c;yKA5M>V+=8TvxR)pnj3-dYV z|J^R>UgP=mc@Oe`#xZ|thP;$tq|ftzM}9D0aa)Jghae}&y2h)xva?b^M0<@-4Fw+yeq>hj&2SH4fY%jNsY zcP`)l`!L@}eP*xTiXzbfe~Wn`zkNMkZv#CTHz6j@G?2d)5y)TricdUE{^1~2JL!Uz z{POlv4t?-Cl#zdGV+*Q)Q z4LA@We;UJ}_x6r*)VJ1+uR8q04*v%7m+z11H{$%Bvn8+7|MJAm+S z0r~6ijpq6P3t$N9f;~On9S7-dTP*cK{uB5gklzC0 zmy5~xJJT_Iq5`n;A3OZt(C;HgVSxNU@CSzl{9{jT#5T!xbQmRo;cso-m<<0B3pfD( z-*ErfFn>d2rOFJohtGTS+rNQyIF6E({Sor#Xb&jAc7G3G@)!BwFT!dR^6wFk!{1Hy z3GFZI2maA0Rjpj7gyCoo&LpvVaarF|)UetWvz5FuJ0>RgKTDfCyLPKQFuLl>s@ za=;rbAXtv)<$4bFZ%hySYu&gzOly(m89mu6pBBlMCXb-IR z2>4y_|K;vSepGuy_0h~P@FbrdK9-)W+?$d=>YJZ8=1_e@dqeza#WnY)v}bQ`K-Pb? z-qzonbnxr%jX7cCDk(nwNBRRsZ3ymmQ{S{+K>kR%KDGq-kY5~C<1eNU`bF4+o6&xf ze^Zb@pYbzzM4-#>SbCBgt;qb4Kbancls>h7g9;unn!>LGuj3CG)5EiD@Ghe|bjYSp z(uW=HH?9VCzZ{g1ehu1Lt}ceJzq7Q z?@T9(sC>CU!TQQ0Myb_*$n+At5zS2R$%{(wsh%7*jG+M+?OY$F{hcTMJ*@g)+1_}2eN^+!^<%0pcKBGj)KUMj zJ{_a_&E>)6RUpDCVt@;9t%3De2*m4QfOn=#V$*@j#by1b^)ecY^Z1_PV|^n@an1HF z{tp>t)PR}k^&1%Ppg!}3^$2Qzi9SH>0pN$y z;qWBDn%^!XiqiCm=u40X_Vtiaf!4^vKV*#M{SKP`A!8J0h+**i_a0Hm`$nXA9*;5C(1sIKI>{0 z_FHP%ugA3CLbhLq$B?}ilD(c`cvJ=K)drW1g}pM2@v_5S3I3dzW!a}^*!p39`|q3| z-#?5#zkS4bQT7+Q{ES$P`wtUNCR0=%1fzVK(f)DwsKys$|I|5idy4Xo8N#o|lhmI3 zLuo%YRH+N?`>`~4M|c2xg#MaiJ}{;W0I(NB=M|2xA3Sr%&TGH6UaAXx&&V zhRB|2zXRwoU$yo#!Cyi3huWjg6|#P!e+=??llsS_ay(1z8_N&4b>ou}bAA71&W}hD zzMa?a{r&efG^as{56Mw=kXTiyUo>&s{NP#9^e}faTX+8Yxo*bp>+S^FEK7{ z6t99H2|UtouD%owD8&!<)@)1@84UwJ(U^HD$=k&KhDOY%YiBkPemv&Wn7>XoMo+u- zmRqpt4g8*_`F3(<10_iKYUcD!%&A%GTs%uJkLq~m)bzt5DP zkIg) zjBixuA>(E)4X|W?&Saxus-0f@ z=jv(y9qc`k{=m{(DE(^?A8<+9S5F3-JGX=3a$`D=FB^y+=>tEF@llB{NjEg`_(9Q+ zGH$*({s`p<7w50c_grp{D5LTKT#~MDB>m-7{6u=$Qaa?4)0gyR=J*8i)Ef2m6n`qk zM|lt&2K~)!AK(X==LfSSf0TbMkAJ!T?(ip!>42dUrJq`(7Q+Y3XBb}$d?R2xej>Vm z6E8NU=T4)%Ob=#Q9|@xYcsu>1$$9j1kzlIP4|?b;Vaxzt(>Fr?$w?c1AYn8C)_BTa zvdV^se3t^Y^gW@{h9`Z_g#uGENFFZlh1lL_#)o`Yg5Dm#64ON!4}Hu6 zY{!o&cjB)GY{!?6b>goAY{!osXT!66D6mwMS$@|7Z>JxJ=D{w{HGu8-IGR@rPx(aw zr!F=1KL>a_eF@w@COy($gtsb%xJKC<#fJf;W}5QM2E9FgqBJl5A_$b4qtc`JFpyNU znSKlC?eR-v`SBrOYPKyt3?_A%nSKiN_V}^F{P++s)nbbegGyatrauq#_V|TG`SBrO zDrJiggG*g*ravF__V`8UULE>d1OZd?Z1G``sVmL&7l7U#ALFpR_z*BP-xePRnVMy$ ze;w%U@yR{pOdkTK7TDs$U{hC{>9>O39zVk4J3Svm`7eTisq1XASC64bZFfbH`7 z0~VgrpE`9RA{Qw6W~Y7EUl04uwIBA&_6A>ubHon>ZSm1Q-86L(;x9Jky#aW8dcly5 z9_ih569h_KZ_@jG%Ac;<+Z#b|&u{1^2#~r#<+s*Y;=|m>*8WfB@ks8FY#%qlKL`O* zH=6vHECK!fy8poVZz|PK%!hoGzoaoWWY=H9xak%T@JlDi{wQoveCU=g=(`{-=2Cod z9OEtY53`}asQ_7D+#b&T0P3%(SN$dW_t7?f{#s-7=nT=z{tNBp)Tw6vIez_3_V|=O zv~fS>S+=)JmlEEJe+%b_^N;=KwZ?LYW9kRvgR(O2pWoE|>*!_n_~EW40vY5j+vG2V zer_@K19=k0^5u5@H5+AR)L-*>68(9Y@hRe<^&|bW$K`&_M0)gSTJM7Yk@~MN+`jGjDJ{v_c-P~-)C)3r{PWZCR zoN!Hz6TadKCtOkCgfG9`36CG|gs;5P2~U{dglEli!j+Xy`0A^j@R%`9_^PX%aCx~C zzV=!tJa()NhJLQOhV8?&hjHVa@SHhLI39Pxvu8Wuk`gD}(&B^@2`8LNIpNY$Cp>SS z6OP54@cj8sxUkR(FIeD&i;A4^b=NuJ;$kP<+UkVEVJ94oI^jsf2^SPN;e`vG@Z!Zz zc+ny!eEs!K*zb43H{9Ta0|6&|$9T#HFIa%_kqr(-F&>dH^Iuqi@q`V3-F3wdxV07I z9UFepVvHYbaP9Se2YkZ~=>Ki}r9E(YBY&fe_%y1}ji90YjxtIn5=><*;lyNulO|kR zMKG12#K&p~rZOY@8D$iXCz#5J`dhqDGMDt}r)}o&=Z-RpDhW?zB;yyC6HH|y;qW+u zF$Pg^WH!Mq1WSJPq+g3)G)4G%gy-}N<`cYtU`a20hki}(#HjQGA;Oc6WqSVW2yV6L zN0WXn{=r3rr#d0&LpKn7Bf-Zwe`Ta^rpNm<$gh~1dEHEJ3!GO(iqu)2i-mWAA72^# z$NMdg_oE&!%Jcs%5%PVQPh!Y~KhU>sT*+sqVGP3Sw+X{9FhHdBQH*~#U_I5BJ47(Z zNncp6u>5PV2l(?G>#yYh^MVxbJJS0d{@QN1hhA6vV-UaG@Z$}811-$0JD;({^q(8j`@k{W+le&=KP`b3(4wL?__YP9N*vMN4gPC|$<;6YvP-w}bgndmCkb5$G%8D4$(<`refe#Ms`= z@(mSz2=R4#&`;FLPb2adH+P`W`@6`0#Ka$|l=@DA0n3lQMCnq!pMvbmqw^n8`X$)P zztjimAG-*kj}Yhoqc(ZS{@QIY#V-Ra%Zu_GcH}4DKJ)WW{8nmys9&V7e0yK3@m#)f z%b$<&GxE>tF(5(yNFNDa|6%xyEDtJgZBIJCy8PK5sX$3z@L!X?bs}i#zszn=tUuC^ zw3i6$XSI!fBzq%!V0~l##f?6?HNUsfM@HkNK6pJDk03wm!>S)oFJXJ6{LB3M^6Wi8 z`m)y_581Z?dYyl+5AAqhs}H?4JnYM6|CP+&s=t)qi?au9Ptrcg{sKi?bb4>2j{%LB z`mogx)`wL;Z2EXRd!YPpb?~#*zcZ?Sk%6fCm7hPf2e$eq?a#6oS)aDs_)~sO`*?xs zkFz}=slI$($7g*EZ;!vA@fWHuF7>h7#*ga9<2Kk)-=5yVd%>C57!qmNVbn+)GvGg zVAT)VzT!+M*!KY&KWz`vK0_B>e~90=Z2X-4MP7Y4vV`@S-`*aw#V2{4aa(<*@`~B)qu7(*A5r?$Uuu1jzK=Wj7h3Hv*S{ir`wITW_Wp$0U#tH@d2oFp zgCPAp?ls#}sc)HoYA)j%g2rW{?W!Sg#9<>cmw-Y zL+OaszH)w}w($z3H@tm1Y2!!oQC77+38wj|%&&*@PxDckUur)?ML)5{r~LlR0f&pu z*x*pQz}7#JJdukUpV0Ws-kyPMYb zYyZ2Gf3cdsejm*6hxzMQKZ)%_yJ`PnocZzo+yj7VKVvoR&ke+Ac3G(QFJXUi7{`O* zqWzu2-^b5S@8uqKUI*T+5!EkyNWWA+w0^N)wt(Aj+K-R?G(&z&XOW^@56I^?!uu*B zL;T{MTpkId%&Yb%)B2Ow+q(f}fs6JNLw~@RQ3Uq)<9(mzexyfum0okNkI%ya%=@9L z(th4w#UyAis?q^@z7RctyjAHSA1x#C1c&r~3Wbm7sQy^^;vCf<4bN8mb8$Rjei0l6 zqb|U+ReBaaoTK{07vp968eeO~3h4|gy}mPB$;0>~*w4^g#QRNhb#neb3jPH0w~&oX z|4V>jDia9@8MgdIA%@uo&yait%-TvnUP$?ksr9AV=_v0{()g`iqR4-8$jgl;=Bu;Q z5!Q#6Z+1G&`eGRTOM2yepZB`~*7C9r&>xS&{z{Ucb;fupFYB0LDKD1+`I9L9^D2GJ zXS@dGBr}^EMU-#8y-9zMRbSu_hzpXR7}k+LAPCq)yVV4(Petf`2XQh5o@hTR0n>Ij zz}_uB0slY}=NZ-f>%Tn3YjzlK0FGY0@5wm~Q~7rjeFiWt;0p}O+MD0UA%Q+Rk1Dy9 z)@OyJ2R_^tk$I=|(VisxBN1s|Da41pb`z-hA^+e<@F&DvKJUq|VAvk`6F-AqXCV3v zzYmuj!|`Wp`I1{1e?7yfPsy!}moVgq|A~(ajJkbb8$kzRApJYEl_E(nS8Lj7i3g7x=r>FZ5oKKa2zNxiie<9?h{ixJG<9!0o ze+l+8Q~q=N&>sVj_(3y$@FV_Bnm_!7*zXoJ+fUF_drR@Ty#+m$PljM>FJVv3{h$ZT z?H}`-I*s<**?eY&5)-x~Rr_$XNE^1%K* zxnJ%r?9YS04*T`wemR2Suj}inko)BbuA}{QW8{80f>*2kb}6tkVeGeCYKO7kuH6n} zza3P^^ed>m8o8;O$otop0G`fpEy1vV6dJBa^?@vANnzuLl&7{A8CkC=Xi#8ZDhV*K$Jh(E!?j~Ktw!jG8#7>W13UHau0 zh(FfCkC^^A3qNA|aSK0U{1OX4V*G@KA2EKZg~%)*ZtztF;u7{AEEj~Kt$!jBj~ zY~e?YAF=Qw#>akjZr|Rn{0c1mi1G3M+B?SQ^$4^l;6Gw~yr1=s@pb#_h)?aO-QLlD zwnh$#*Qvp!_wUXX3v>Ut>Mv@I+FJYmqH-gdjH5l^K_lvWtub+;6F+{u6F+H^6F+vW z6F+&f9lt5vP)qIWMB3lS{Hv<$_;RC^UP!Q|uiocU<6p$54XCNfejUH2#uD5dgG+0{c__> z`Xaak6rSaw@uHI+0;Wn#dKzEZ`OEh;^!$$Ghk&V4lOFHCIOy>{qlRG*o6^M%5I9w6 z($jd;F3*gq0L}R$=!+q6YPv~3Rd~sM$X?LD5gr*xMF}P`@L!W%J>wX;DY|J zzpjzu<6wXG4>V)^L+_L7`C}s8IE}}9YQKBfnAw2F;eL{V;$uJEG^kfhrTu$IuiR+j z_0uM`A3xmHAb31PMvnPKtud_;`jPJ|93lIiI@N?BUrFz@X$0%}QAw}F>-U4gun(3O zG~!q##@`bsBEz^+BoFqd)z&)U$&;P%q)AS=s>%r`lTNs%#tBzfJK^!;op42k6Rxau z!V@Mq;qr1PJZ6j&9yiVjj~(lTOG=z@Jnn=`OPz2c;e-ndop3DXgr`q;!bL?+I1+Kf z;jj}fC~(5js1ptZoUq^TghL@G91PlE)SudkwJ3dSKhosMXzy+NTar~t2V7Hw_SlB6 z7?1YV23J|fnQAWWS zf@u#E)mQi*c!M7Kuq7Ob5q>(ssNe7}Oe6WEe}nep1PcjId64v>5`rnuyuYPOaD8U_ zvjN`UQY?n#{+2T#ak2YZ^6B4ke@ivZr{S3-S!g~D|Elvm+G3Bx|AG9W{p_UH&fK4Z z^MRTG>-Tw&rDN897U=I-x`4L#;(`XCzff;nzp#t>HIJv1zZLB<{BiJy$6g1u_8-GP z9DCCNhhp46DgQFg(~Qvzb($XQ0|i{(YCi(jAEkfMgaZX3;%Dtwh~YRnGd|Y8;v%E` zLytX3{`La$Uobt|TlW79D16;)TJMO9-3o^P$F}~0{WnVho9ijOA19PH*HaX1uBR|e z{$jF!xt<{RGhjfW{K?YaPPQcd$-I8gFC6OplfL4%{E$BE`L(ToAiYZRr@y1VZ0jRR zUUPlKl$ZKr=^y9yz!-I9S{~c_g-K8KL+20c53yPsJ>{SDZTF{B{nq|;te?cj*yyo7 z<0xOUp9&kE^pUi|RGv!3ZudV!D9*XsrS_u)LGV^9bgE!qQbFDVKax39(cJ_42>e*u`z zw|=Y${!6rhgij{Xe-M2{x(}Hi{!#cl$zE~s`bdAT^LzyKXPv24a!r@y^LG{U`XlqJ zE`oev`GPu~AB6U66voTc|CABOqdVAt%I&{Ozoa)R^Bd1#{bLFIdkJ1INRYo2>wzV` z1$p;rmjz6E>90+ci4`W^QJ%0b$9~HZ?9Fk$Io^NXZV2_g1-u`Bg6#Wo z>TYn!{hdGPq1uRRD}E@i{PgI2qZHxk;`BpnC4g2Na6irqm;CS)_Ji}X z`=JkpvEQ(@hbrT^Ot|93m;C*%|2*(G9r&r-3R4d4*`MU}o*{rwahJ$)nZ?dbFE z4gM(1ZIyqS{P|mbUTI&bug4m_pV~$r1V#NT$9l#Y9O-BO9trFZ?dwLMalt-+yo2BO z|0_UzehmEY`n<9*oo^5RH&W-!7v=AMkE4D8zg?d%LFajJ{dv`6uRj<+vVQgXle9m& z3jFl`Xu=acpFfE84CnU`u2*;FSw%z(ccq+KB@m^e;w01zYnr?8|()kGSc7gAlQo!F-aKv)d|Lj*(4mW z`fKnMXNrX5`x~bp#QF8}34=@94}Ho57ugT?laoF&CQS2xTYjIe#QJ$&ez88}kRR(u z#83Ml`TRrtdo-rnJXr zH+TkMA7l^a_g^gg;r{p$%YGOh*h<}}ojwv)`d9Q}nr@iwamjZ16?4m8Y5Tvnze-WQ zHQ;xY-&T2k2$vt)Z`ha5zg6q6&uL#P@%zZq8l(SK5VjP)*6|9hqVs{ZpG&$r_5$t*=|x}^P^=TlODi}6K2*6V3}#~0== zRGujXQh7SRH-q`D{rfW9-v;RgVe%Iii)S18`!d{r@b^pJ|KFE!j9<|I<(G%Ey={bk zZQ~i(+YP8s8JoTx=?Kppi z(v#o+!g--Me}-Oe(&u41&a1)r;M3NAuNbW#H}@tt)A)+V=NJzWKR@wP=V496dJMr{ zN+0I~w{CRWE834v@?YupBmaEVCpKgGd<_WH_iFm@!}$I&mA}(;o~rGehiQS8C@Qi2(X%ITr}9r9s3)YlV|{!Kcsb~UZX2f_^*Fd(>&*5CcWg8{*NZvyNC zUcsw@-{zJ6Pr^f=CA~q;5Bs;N{lodpA^N5|Nqx48;Xp>i5f9=0;=>9*kMX{Yh6CQM zIRDpspiSb>d>ZLjNPqTtq_-mPyeQ0nuzvt}v_OPk?E@SKoV39)z)dzd3^-+j{eaQN zG5>#X&OcG!`R%V`e)7_A{1*H7mr$Sc&nG^Xp6-QaaMApO?7Q0ABB|#nje!r;^O|5=Et(W6TF1mFCG98 z1=EV5}PAX2WAJ5`@3uE-;3;Dr#7w3y7i8b06 zjOUgB4pmt&;qm?$UxdeaI-Fss^ArZ|1REO0_!{j4zUO3pzu^qd(c^?Rec}@hlp>u^{rF8(-XNg%kItutyy-9mp$qnb^S7o`KdUeBNBh+! zLEs^;qdjNXHh+Np-rjqmI9%p=)xGy}`c!{W-qily4>atB&*Q5#?(PLB&)Yh^1pA>j z_1gAtVn5Tz@%31`$p1j=XPY*O4D3ao*8uqLyMfFw40&5Oz9i>sG~Pgc$v^+Lq?eC> zlIMBC-z@jL!obK)1AVn_d^LO}CpdujIhPEpk6U``nS}T-D2jiY&$Hn15r2OX`Pl5nkkd<#7EzojP$d|cvJ zHvskmPD!|ag4o#~kn17pJZ$VQssptg{~uWb{ngb0lIJObAI=AxWa3Zn5H;!g3R>G> zt3O>FMEofxp6MGWn=tZQldhGcYpyW_gyJ{?7PSrC>-JAUxtz{rZrMKZ)a)QhZ*| zL;P}M;sjHEu3xnO#a_QkD@;7suVjqmqcmjtw4cUKUsGw(Q+lN_T6?t8t8(lYs!2~Q zr1jV&ev)6T(&Webs3@em!Rwct-|;Ya%U+?6BD4HpKd8?|B%7{JmEJ8{`hV8V|#WbEn(>R zA+HH@{D_~rQ@%KU$Q#ly$H($SWd`ck@i9HnFvkz`{sAj~koN~L%<%(ZQJJ#E_XSOu z;|IcOdsK(&_sAdr$@#$%MEA(^D9Yz6VdQvwOHX7o@dLLo?T^O#^=5xq!Dk5$ZT80%%;SyV zX8#xkucrPlu-Tuq!~V_wI!!+Tc;9A!lZJ_(ce8(%f>$&^Kl1$!nZG4~ZTdgD18_*~ z_Xow%Vh`S@v6V-$7w_Bf`iP8A@7MS({n7gkX&)J|yp9syrcbPAz(2X!oBD|Pz1@8M zImgeK4;j+-N1V@~e&*|&#&;>Gp2v6RXugxLe|#So>7&2FdgFSG&*3ll9wMmm#EAZc z=fRKYPv`j$kUq{&7fD3t`4Zlrpzk$M=GlJ^e@Sz16cc+~wBCpM+T81xumFtuS<)Nk z@5ScEWPsm+^*xCX()jw!65u0PZ^Ju#74rOOZ@O33b!rf3eI$*uNQghf81JWj;^P_y zf2`N}djU862I~!29|4Th*!&X4ewGXA-!Xp1{0i@%1?YVy_Sd0&F@NClxBQE<1dV4f z1>o~n^7Yw&A6{S8`mUz+Y1{e6(2v7E1Grp$FXQ99A8!Zp=eP3r4E9T4J(#l2@_>E- z{u%Q;705^PM{XZz{s=sM?}~~V`oenY2AWU$sQfyWKLzt&Zw6@Xf0<|$u|KDX$^++J z{7A2V(fbqNkMq3w1tFdfOMQg-{Ro;*5u6tCeb#x}|GH94M^vq!zMin$kMmf#fmpy(V1zeg^EVb=xe7SN>=Ee)u2r6WN~T+wYIm_n(D& zzXkf|_i5NZWqE|Sg5E642mVRDfXVxxK8^hgs{e{n`H}yEx_Vs7pP}A&A{cl&&mH)T z`|{5?zGvRrb5hhvd*J=**k4P{#T>L^Tz*NMU+#p--0XPU`)6p6l7_GN>V30P!keP_ z{v`JE*kIrTK_?u-`)(#Z_M;|vKd(yPAMTQ`)*-jNK0BkEUU$#T(d&bFU#<909x0zA zR`C8?dVj6>Thrn#Vukbpo=+au`xz+y zV@3S_o#g)-((%MNCI+K=J0{Glk1a94q>Pc#Uu#(b^2 zF!A+kZvAVv`qRa+s&3j3uG3!ve7wYlNB=z@{FVO{`lI+Q9}dr^F(8$%<9))@J2*b! zDLKN2yGrHQl^#HUTYAR%g!acmAH?rh*?r$r`~Zi-_Vh^}!Yh7=Uk-VczqXpjpM}^z z0)5EzI6kG%I_LD&eiUMW_G(K{8Pd1K$NLAs9HKUb^pEkZ9A7gG{*Z^#SNzD|Wu4cB z_K3=t$DhQX^sB}%raVRw4c#UGV@t4qoc8;ZzB9bv3-%VY+iMMdZ%EUhE>4htj2`g! z8PGl^IKIdfKgs<;yr#dXnDS4}AM{Q38I%0ve)MFzkSnZZKgFHAKc4oZgTBU~_X(7I zM=Bv7>4WWo<*VuT7TU{$gj}5BKL#b!Pqb$9O)(Z8XwH`E))H z=6t^V5gLD%(*#sr=)ci_jimBj_>h;%oAHPb`RIUg zbNv7?`*XBEbYU9j3EJpGX>SqRK0TB1<5X99ypH<$4A%dcT;Z|bQq3@F8_ROGe=X zC6XT9Ak+Kxc`ghi{UB8+&Y#p@0r^ijeZb*#nC%t({c8M$2{`a6bThCo;Ky-Y5SgIT ze}R-A`2pVKKOebf{RKHa%7K(8#QZE62EqX3Mfo8T>jYqtnE?Zz52W$Z6EBP)90C+`lj<$AfL~e;7AX6sV^#TyM5LU^84Xt ze6xIP<-ziWwEb&&^2#ru>jO4pu|2W>%4Tl?)81wINxYd}h+&&ODEmcy;q(I>-@<#9 z{h9Si+B4ALPf;KB4?-alQo2PeU}{JVWPm26?{9 zF!o2_{EDD>^MfHCZ@)n8nHoQ-_6+*g^KF(F?GH9^@TJ>B?yC982=P__ICN;^WGY7ycu!T^^jbrtA;dWc@ndm!SA;|Fw86McPXxCD*wO<9qPO z`Bd6oReb0B7ZhLGKfZ5_^1%68+I|$h^ZgB?=k~JpJhf+3pJe*aF>LXRF};Sv>B927{zA21vcG`6%HaS-K>35djOPaQ3WO}r z39>xVKf=F7`*ZF6E%XV8vyvC|bUtn+VD=vX-qKSNb=s4&JWB9=EgBH!>Tte>Oi$XQ zYM&sVoIZkver9<6n)4SMCmEXkr?fvB z4}jj=8@BQXdA4oz7y2nfdFkelXuK z^(E^w$tTAjYJ7_Gam@6<5B^>%U+~9;@wraFnEgv+FQz_q{VJCIqZ}`jJTzX^@*q7~ zf0xSne>h#Lm%$fD_8Rg z?vI+&<>YWN@$f&P;UW8#{r9$wsE@k5F~1}{_eb)5Wk11~5X<^0+aI-mN6s&{Z7eN~ z>iDa%{<5*GtX#vGkB80g$zlEm`v_G>)cJCN(O!hS#rpffod2rF{DtIiLXyxw{O!B^ zrI1I#jiRdGUxx9wgpV!3`VG2Az5Y{6>pf9he~I^x>HE7pIZg5SY1{qy{GVR$Vfxaj z(kI4`mEecvKD~a!_;9IgPr!xoCDwWk<4eM3gTi=BH}rZ8<8_FN*IY$SCIdLK8Kf?Q%Hq|_%y$yxgfV$Xg_Px{t$(86o=s; z`y)e4&v>RMC6K+e{!r%z{>7IX;SY+6fj)*}G&AvM%KXnUge9Q68pE#}Jmw^Ga=dFJ#T{&}<|0(yv0pwKOU`r0rtlgjPRBDAGuxyeiR?_IQBzfzZUO*dQ$Cws-^u?hr*YDRK3qPY=0ugXE}eo zU9KnA{uc6#RR)M&;HPwcVL6-DSb9GW>FqMcj-~#J*Go{|cptNzx=;3B1OK=&dhFNs z0k7Uq`?3)XsP75#`#_Hvqw#@cAx6{pO%dPmJ{-yu-$#|@7v90|`#p~L9jHDsKky&G z_j=9r5Fhqu=?n3pzjDfMX9CCOWB=PgKjwr-)+LRCS_VIkF ztSS$BUzElRxVStYHzJHLCp_dsdO02+Ch)!w>h~jhe4xI64*f=`JUSCnzE7tMnI82Q z?<4X1Z&>O^eomwO!0*%WFA_a2*7xH^IM5D2mnYIIWdFqq@RQ{k;`+Q)e%}}G69-Iv zDm?2a(H-IPeGGWwr``|6{&o4iTJ?SE){V=p^1}OoWbc7aZhx?TTLk@t!@H3H=1(jy zujf7s3!+Bnu_Es-yw6DevtA#h`bHH5=OwZJ9x)2!_(+|1iTCAg^`{a112M8Myic#} zS^76{o*l#BC*NP7_-H>I>y6OgP9soARXw**$~S=i3Hp-jMbHPWKeE1PK8W^>zK=lm z!27A6ZKL|Y`%gRP>hCcZa#yGIMfx}G4ffQ!@l9(#@ynH{k45r(z-qrCzDF$KP9+b0 z@0too*Dr%=x4PKgsrNrG9|Fey-y*zTW!if`#>dK@S)T(2+<$1FHNOF)B#^Q3r}-?~ z6U}GA5BuTyeb%7dzYhNDeJCXlzQm#?$K=5Y3ugW}r_~W3?>Qz_`Y7L$UbWvd zDfLbIB63`im)<9i^7+paCjCNxNxS|qA81nWvw8VF<3ar4{CGab{Fxu?i^~K1YaRMh z@vZWc@|g0WKXTARzxn4`TnPU9{oI_?HYEO}fWEiRus7+1{V69LY`4QPY}ZtjIszU;NJ zk>irsQ0PPRCc$-SUP$}C)w7sz)#!=W!cA;SW8sn*8rnE6X&i?T9-6i!O@l%m>No@l zAqhzcD6Y%n(lk(-2J*$e|NohpduQ&-mIb7R-^&Gz&z|`>bLPxBXJ*dKc!!+&n!E>f zeekDHe#U!fV_3f9pFn=@9)kIS{I|?f#m*~m{XQJuu#XZiwdYC|?{hd?N3ZgS zRx1DM82%LfW3i`@$ARmfaQw|2{AJOf`hA{!B~cqB{jy*4Fye6mE$|G0PxR%M|laYzj1zHy;sWk(S6H@Ry5@CemJYI<D8^)E8N%2$#; zVOd{6d{g_KJ~9GkdnLU(?~0#f|8@IcV7)k;AEcx{p{K0&*NV4hqVr3L2W!7Q6WM#T zuRWLLg=wmM=toDsQQvlummY_I%udJn*>N5A7kXGZ=--5QtrkLA%4gCFlTp8N}Z zvH$X3(~#{3G|5?pW8gqaq;nq^zg4v3+EF8r%yxg zsrWGXGyd?9i4PKc?+?SFVTaKE1Ai0dOR|2l3jU*v?*pFN|5fuzXMTk8<-E}sG(QJV zch8GpJPce{&15;&c^~>SG5JA$+((P^Uif(arFSvs1{G354>H+9}PYy`-_mTZd7E(57#TFcV*yECJrk91o10yKr|lo zWv<1V3}A5*e$Q(_d||4}_K1J zT5F#lrI7vb@~I?B`e!M}WP9`Z*p^Fw2*_v;0JDyGXJ zzJd5Nfj@)g3;uK`9-sKLfiH#lrRbb6zyF>}rV*w}y{C=#Ex>xCEl+rV9?OGadeHYK zc=Hk5Ug>S*Vqk3y`vFq_xtNed_uUE zTF~;>i2BaA?aPiWs_v~TT6=L5)v z?OPEW&)LjU;M+p_T4MTQ?OF=YG>y-g`|`n6u4lXl@86sSe0!LGWlSG1JTsDirhgOC z=b0Y;1Nzop0JufN`QXB6{EL2Rvh@evV_&-#1(sKZ>ErLmuU)$u_!XL-Ql;Fj{7vX% zl$Wh!&IJR)@x}5kLi&|DeFpt2^3F8iU%s-Y?2gxRxe8R59V0qY+u=CjbY|0nd?aw8Zen+37KDoa;(w{K*s9?d1 z${(4@`+yyO&+q9+`C7i9Zx=-Ujr#Hd*opj$bbge-Afy-lm$`PnI{T^h!E(G`zs{l8 zZO_c$wA1($0n1SBJL8-)OnQ`8-=5RZy*2MN)(<@9H;Dq%YkM~{SbRoU9`L0~wp;vl zjmP+j+9&P7aUou~&het!w}_=J4$EImaf|w=Qp~#JKbx&0ff@gqYx~t%WVRpvQwH)N zZ39UCf$_aoU_HKhpF5Kp`hCpj8kiPJjr0_7-!l9&d}Te}&pp?M*}q_4C{Gw)b8UQ~ zGPlEpJ7(K(8}u9X3pSvZvR`{p8b)c)31i$3UkIUj4$H#_Izi_!SU zcqmlPKi|gB4CcUODM}I7Alx z7hGWB*}u!({%xOSwIBS&^~ex#krAQ4GjHM1|0i?D&t9<02<@A4{a^Zy*9`)xg zfCFvOmtSe&(LNqOuX$}@`&O^E@yjdir^7+C^3Pr6;>~;ZrhhNG$i|a?Hf~l}{)!bA z9`-Pov)d1U7yfUgkCs>Ft`6k``eJ!y#YHO&J)+MT{!sIBy8L2r63LD98Ty~RHTrL6 zP+SqpgOr!Q5Du?uAIq!c)O!!+z6`|U$oSUq@=8l99M+H@#h?}bnp+>+Th#PmJeG49 z?@2kI$oo6e{$g?W;`cAUlRhfkRqh7xmD;VjQKn2Yiu(0 z{Sfg0&<6`P{9jrh;C__r9`XJS<^b<+NIXCu>n918r~N5sJrDAz)|Xb1M33tcSf7;j z&geetL}DZJYyS}Ia{)knXcqX1{kowLMtNM{=TL1mmqdw)a2}r9Z5n zcsE7Q0j>`dy;#4LJS!Z95B z!gyNH$NR^do%%~S|2o&<5B8Hxe#jr`DI0n)9&rD9(#E^;z8?Gt<9%iP;=FnyOb>Z; z#slt;!TX3Uw!HGbZ7j%m>OOAXx89h;d_3ewW5pRizPZ1>S?*VBmiIxaP}Tk4&1i2% z^`GWfod3AbTH^1ZA0)mYF8|wxaGuQYL+H2VeKF|E<}Nezfj^{&`c>Vx&i7X{xwQt5 z{&&a6PDxMy(!PHk3l>;ISNFxSzX%)p$d8My^4LEc95|D^!h-R>r%(N*$5)E-!}*pe zFU}9_ulrcvFzM4cU*mJWVaBI*zrM)3xgTBepYM^!`5DIl>ha;euN&8=>@T~$@$nhg zhuHu3&ouNMSPc2J^yen(x2#___`^G)4>JDB5T<^z^rz@IOFtf_ygKp*IR1XIEf4Hp zTYvDqV^=@K^)vaG20cn%8|w5W@m!Bl^3g2%n$}eJ2gZ-3FNOZNezx_|dKph|NAAd8 zx%!&*aXc#dOM#!3zV21_aHEwU_xn5Y)IRL$YnDfQYTh4k^tY6k$zA1?r|%05$D^fB zqc2sYKE^lM`aEuL5dSIhxskoH?8zo2UvvOX zdnsSkk49fP_5i6t?wNDc(7IMIrW{P zKPI0IUJ^zSK|(ZrDnG8v8HreZ#?(`5xw5ok3ONkA0pG zLHV7*4HAEGgFY|9c%*YAKlb}kpR?az7Z1(-{aZ5sBI(usKIVHjaJD(9?w_cOX9(kq z*dD}>4(xc4w>RReO8h+TpQ|Mip9F#8`|AFk*JHWy8lX9!M*lC}td0xHCO_ye;(R&z zXVx2W{^tgij8EoIcuxe!qfCDE??`~v&axyBeK|6GE~oGOBpzZUIK!(7h(9piMgpu} zr~)s1{X+jW#pev&@#RV{co*1xMjDSfn<^Wipjv3g&xCmF%mA3^*{qkhjG{E+-%4d>e}x;9v`C)YE6DR%?QC;t&oG?2Xv4dX}tGJfBW=BJQP(XS@{ z1o0Y&Ds!UpBwoEB@-iokSD$0{Gv&S{=(7U*+vwXng3dW^ym}^${pEfWpUn7o&R_BV ztd3XDnD{2SzlHI}qOYQOb*0~QJn?8SJ08C*=MhZ255`l0-4n&DBRycD5Bwdi%$4&- zR(!gw7lB{qejwQ&j^fkx``kJ{eKaUK@%bae9g+M<{CTR_Vd{fC+=O^@)-Uuacy_12 zh#$uM1NYs`R{M7_W1tUkr-G|G{@jiCXS_G#&*um~7>_EgN#2X z9C$OR&n4a*>CO8`@*WECAJFg1LtfPVJHZS|k1QF+@A5o>*bl~|BR|@g;RPVVKfpX+ zuo1BIH`)Wd*c+VRlKNr4>dqSof5P~4;F+K6H;_-ttJJ6C+v5Jnz;!q2csr%9M4nAN zS})_#8yok4UlK2l_BrwB&?h6q)AaaItl;FEHgRIH{HT!HfsQTenhwL~vVcmHGAhloQ{M`F?5GmkT?fzgl^o;STrv(r{o+FBP*XKLJcy~LVTj?7a4_Y6TGB|&W&*;YioQ|jdRQiGO>1j2;*YW9)kLf+Y zO8b=lPkVU3L+%5X_^v|cpC!N2{}{h2|DfXE$Z(g;myks3S4Ow$kNpwwG=>w;&gVix z{xBY!TSxd*e#BR+@u6VIQ;OkBhM&%QCgRy(zq{;sb;M71XGnsMPv`jn`v0gz%4d96 z0ieW3p+9*ZpbA*x+aceGkB0*+{UPg(@VAx!%y@F(OMw-SF7fU0c=w_T5a;@-;ITaf z`X!!B;6el+V0kYiKE4o^C-LYUKlsqTSUfrSFZ?S+{FC_PB^rK6;>{y|9V0%S3-I_d zj2CqHtJ6pQrRrDfhl<~*uZ8j{{YQNPe@NL!rH`HR5D)%=N{i@osSor|=I_PvcWsgD z{&c)K38DalSR*oe^2(h@$bn391LAQY?f8a5{PnD%O<-d*MJvRDRW_;s3Mw<=8zDyJQqtACi z-?rO$=;M<@81g)D-Mvw~JoZnJp8;QXygi>!4CRsbJ(OR_=fHKzC_m%valaJpgY1vc z-`bJm@eq8eRIoa~xXx~phne$^~uWqii#q@i7q~D*H^Nwwb1n-XD`c;{qA}@H_+6A3o8);FlZ)*7rdrZ>}tJ>G*Z{3x_J_QGD^iDxf`l zKPUVIrZ%~ zizjT4T;M3}slShR;I;|cb5OTuZ@fK=WBflxdkVIGcqQ#wKS_J)&o>Y3SR~AH`sYF2 zp2Kl@BK@{J{r&{~vus&-z6|{_8d&m$@p*jt+BqTpk$lykUx>?_d*3hi*~9TD@q049 z)cBM23xU=6i^e1UBXge{-}kirt5Mw77wT8EuNdTIe^}=4PmBF;3GKh&vA>TAp}emN z{I+bfGhRoUdu!e!D3ObJ7!Uc(Y`JgR>~A~v3FUQW)Oiyf58sCQmbs5f^iOHH<0R5A z{x{N>{Mo$loUp_biUwMeMzT0#~&wsy(h_qiiW<KYO&tD`s#X1PsD#+MA>BXEWV$ z-*_+KXBJQ0pK9%3IC$hw5w`s)!nQv}IP|A*ez6nR%Uk+vuCeknG z4~{Mdtmw!2YDIrXm(Y*-FX&%Cb^3ccy1cSQKjBq{s6B*NmtFXxjT%mP+ZI9|E&Vle zUROjv?)NVCDgGyg-t~YL{U;gv`{oP%-GpK9^UK;J`9S}~{nyFhyA~rn&iI#Y;N3;|{(|PWh7Xio_<@ZI#{9t1XK1fEui18j=QVNPLo~m=N}XT& zCfcLUFDEc9}=Z_}Gu68S6eH*nuy=x<5-CWyQKCiYjX_8h~$ zlLTqc1opko|3Z7rct0F(&wB9t1lm(D{>m$D&&2jV6Sk+`-Vb~(?th^@W_s_S2FG8(c=3nlsrj5<@8Nn!=#N1EwO5Ms zxISIGg8DeN9sxZ3gJR?#^x!^@#%TVF^zbL5^$F-7?6>j$C*BV$>xsDkA+q>|F-;9~gU$`D!UMZx-jXV($UZ%k+u9RrdR3*o#)4|MRpyh5je(+V8%Y z{q8%a{f54*x7W%(h`lEL9rNZn^mizGq3rj|puY$9!L{G>WA=M~%zn@Jj%B~^0R46L zTG21|n)LU@=r2tucGu7q{2ZeJB{eliTZ4 zr)aMiM)$jjy*~9+*IqBO>>Kp&1nUcZud2On6Yk0%Xzxfs@6xf?IM3PhY3@+NS^YXs>PhUyt_MrvEi+uTS(or26^wSmw*L1+SXD zj?b?PaeF<*`Ssi>+Uq&d{8a4q+_|p3E>3E%3$LoZo^(E1h}r9ier>O11B+esa_x0E zAN?-hY-BW|y!IKQ4P+>iF}wYeq9V$Zm{ zXwO$Wqxq@W>)8^I8SUT8bzsN?C3%eXaNJ1xDz1l5I)Cek+3SdYGryMgK{j*3{d>#* zSU=%X&oS(E*q&FzUY|gFmb_Z+IflIs+w)qq*Z&RkYmw;J zvAwqGe?8i3oBr3My|(FpjoRy%Kfm@epMxoQ_3U+ge%%(g*HfHdpEN~#-5$+<#9p6t zk~_biF1>j?dyN~yUqyR8>HMuNX0Idq&HVb6+Us!s_G;Mc6KGGtv2U-SJ;$)uVS8Q; zdwl}!aqaahXwNb1b=aO)!(K12_OEeX1n)tp?G&|Njrh7~KO5`S?YK`bwx1ov*U5hN zyalqn$nz$U#}5S4WBb|RdF~A3hY-&;Z90aJSsyQEd@H_;`_;33cfWe}Y>CZd{G6PS_YsACb^!Ig3yrbH`ei{4ki|2KFRWG8KaJYYs_7D#D zuQeR*Uo*Z8{5SC@q<>jT_#efWwd2xq50Bm|`6(?!e7{ZqG8O-);>}(L{VV2md1Ds+ zgzvZbPxyevf5H!J4CTq$kH+{g@n$DFzaGHrpO5fBR_z8k{|focF#<~DQ|^n-<-+le z^I!wnZQcPZy>s7>e(#9pP0r(CJp02Vk-j)ofBqQpy(6@rQXkJR-1daW`%&b>cy-E0 zy!xR^hR5siL8%?-0i3-qgmM1oCY;~V<>~W&snkzNFVDLjsmwSDY{2`=;Ln<&25+1UZczE#5D&!tz06VE9|-!|aUP85+T-&6A>*A> zK($H)yy?!=}8PhD~`b3O?Kj`90|I zyo-@{XZ)Z&Zu}Pd(|M1f)mybg8msyX{2kbF3mcD5@w>h}H$D1K&g<#>%p_j(Ic8`7 zqkg_$$?#3#|F>BFB4Y8G{>1TA)1RB<{eleLH>?-Q`V#0(*6>$7ex-k^fTe#hepl@f zAf$gD-gl#X%6mJ9T7AB!#{8;3guj4I{}|ZxkAZdnApUh=#|B-&vH0QE4}K)cAw#d6 z-(eE+gZgGiQD-=T_8;lNeQdQ<0Xb@8RT*ED&$5&Nd1U+5VW5}Sezs3uOJ;#5*1nPV zf_XA_+v5eRyaj>(l%2mb=#uwZMFq+bdnEF4Twj)8zdrz7Np@*w^nvfZoUL^qoOo z@Cp5%LVM*DU+%|ce)JFS%YV{M-xARW{o7v2C&b?l(reQ58u?ev{NyWmaxO2IeBH%9zHe?2x{jbOAV^$k0J zd!?gM?k~{%YOl;r%6+C`dm6v%rf-+~3Z*@0e`k=B`h|aoAIAF?$#2@}OTnY#V^;m< zy;|vCD7DC zYJ4&Mq4BXOJ^G{jAv=A0WmXn9o>=sx6EuAhJ&CWm>62=Gi0u>kXcGD&dG9@Br|%3} zBYsQzX#y8Q{`Jh-$ok~G4CO5)_->6a1rLr_4gI3ezA`=@!5E)SxG&YFztyL|E%c+l z_R6#Z@Arz@Q}~KaUwb7b_wAeXe80%%H~96SoxT(C;l%f9`rsdo6E^m?z2eLLs;0hT z(9^Mz_3QSvR~ma|e_P8(G3Y(T#&-tIQTq>&9|?gCzgS<5_2vIS^h%KBA#e<1H? z0LHC0q#y64W4r@?VvfM-JuJZAV+E#vO2MpC?D|W=SceTKfTVf`%A&+ z^A0`t1D3>L|9{)%cYd~&p8fM3MGr3OQ1m{o;GZde?nQd5JkZx^*N^4NMv-bTcN#y^gqEE?hX&3NTeZYT%{)HJ2C_ibJUkcjVt@b1nu;0}8 z@(I8G@qsFUYt(peD+PD8Sn1jRZz=jVtM(p7d6HT2ukg~W z zU+~rOJ(hha22IoK@{s=_MZXzu?aeW~mM_(w1wrwvw*D^#drykd^Dym?e$U=3Jqo-< z59G(U%jbKkR(|+zIk|tz(r>!IRDYK4M|!Kg(qW{xV6^8!)gC1ufWJ=sV18Km-5F%X zeiTD{+urEXbGzK%Wc*R&|C*8~)&9~wz`ON5qRRi7(Z3HVSn2yja8hhMbS3Qi+AG;2 zr;L$({L)wzuq9s)tMZh7B|QpO`mz*!xYgnx*Hf}~`K4e~?h`Zo#C$brr$>2*6u)%) z9z}Vm-=qirtMq@&_%Gdq@+^GmFv_!Fr2mRN{^-AmI~wx)iyc*@H|^tnq!vBN(q6!B zd0!}S5x^#JWR*6V3+GsWQVK3@fF$|pT4{d3YkrQqGsdzYdQw>4Vz zgMVFJ;;(A^ny++^x#i#45-VTidr<3>VsKBJjpuuNgP_Np*N6V@w#zT=LjAxSe&_LC zw^bkL`OIBaU`=~UBcPwjHGQSZKX(121>-%^ypTX>ZJ$$eGiH|5iR zap_GYYo4}&_)_JIjbF6N zD^>1I+-t#TzuYgT~=dn`u{E$ALB=*3oo9A@`w6d ze&fF1X|8;^{g3t47(ZNo*7N&=Nr%3={yH$r#mD*i*PpfVrOKU&J3W5dUR3UByvKn* z)BPC-{$$rDEg1Y(@0o|=@8;1e@S!|ay7yZA0e}B8=i(djAb7|>;OqE%yM+gTUH+AV z8=GDJy8NGa?4-8Vb% zA9eka1@pbwDqyqTny+MY7C+J7_jQn-?;Cn6g%7*(6&v zc=Tt=wLci&vY^8EXvM#MrTuyJyD3j{e+tUO`^qV|zdx5}`@Uz|SJ$6^^I77*8{*^R z`LEiD|Bl9&Du0>%ONak=-$nf6A%1sZw}bz53-R9$@!M0|9sK@%#D6Qq|6%tZI`~iI zi2r7YzozRN2Y<(2;vdua&fuV|hac4U9T$U-H4*=f5dUSte>uee`R&9%8shI4{QV*R zBN^hq9^&s4{Cy$*)(;c^NQnP};J*;!Ka?i^;Sm2h!GA8q?;0ikp%DK!g8!QkzcWev z*FyYX3I4A_{7p9!|6qu}Tkv;>_#Hm+Uk&k}7W}6}{QI{Ne<;N77ySMZ|L%7a|CJE` z3Bi9N#6R$WC4X(5!5xCXBgFs1EDMkItB(o(VjSC5*@zojZ6#UK*f7e|W z9{Fz){7oVLQ!N%A_3se;ju5|ZpM^*M_Y400A^zhz7yoX-zdOY5-5cVuK3EJs+7!Y^ zTJfIt?YFz|Z4&-1`aigzy3cO}c&#_WKi7w4y%+5-21k4k9U}f3_795KueR>1 zB>qT}`L%zH`^h4Hp!|6nFY!nZ_K-hmZ@bN(L5Dw1|A8M%H@{2VI}fKHb6ll}nxaaNnoI3m|?)=8sD^&v(+Nf(^0qx?}i@9X$JU48O?w8OHC@&1!kp z@Y}DfbLwBZSuLL#d@5M$l$Q!tJFs6_<>XHVWe1-M&UawHqVMC^{g(=sIrvm?mIM2h zGh*d0aqy{Nkpugc{#g1x2cHUx4(wNYW9bVHJ{52^2cMQlzcMS9o})R!rvhrb2*!GV zY)|R?OFh5R63d@+(x(DyJA7e%=~()tgOA5AeAV1Pc`3#Z;xTqtGI)T@tD5*M=-Z5U zOX>Q-vbulyCd5mM09^ehL17(O+oaQwlHZ z?-{^PDp=No34dR~vK~qJ2?fh~CgJZX80*on5BbbtM1i3B(f-kQ6dvny5|4HG@GnjM zh{s}j;;r1Se=?iZ9$KvVl zi={8S>Aw+Aza^Hw>ZX4*p1#HorTx(S#P*#{|JUo%vjgn(NMN=1k-GHk06RT8z)Jsc zUHTd-u+w{pe~I}2PAzZ+{;62{f|LGG zU3v1=x^_r-8sJJjCd_Io*1u%@w4@l8Q@VO z@kd*{t)I&QMu!mY1O6**fT1x6Uk&&>9e_~*;eAN|FI|AK!(sC8C;)~IBi!5SC9prO zHI&EqBv`-4_K$6o`|uOkPiK4NgMTAEf3FKV9KBi9LCfi$4qanKu4B7rzX6*jU!D@~?35=L0{>#$V{- z%fQ2iNBPxzFx;<@_N@jUJHQdX$IZVM_=1gh_D>UtX8~4yMxGh}Fm|Kp)2;t^}|a_yhE(?6;Wxb-=P;Gk9sZ-z)36gO`T; zjk13>cq!~V=*#bFO#Lnzn|vSa%aPVh9_zV;GX~z(DEBe_?oR~Y2D}gL;%V5+-;ZBX)~`~;-m)Yq23 z-*0DqZHhkVv!19t&3~wesJweQd$P*A--`v4i{f{ORkX}>2 zmp|;E!t`Ax{o}Fn5Bol(HA=rLF^B03hCdZg%P;n`L2vG;TOZp~Mtb?2_CTKkramob zw9(&6UqGIZj31!~`oB4{ zCptgt7khGm{PpLt{YAF_;CjM7MuS!Vg1Yw7+=>39JUs65NA!P{cxkVfzdx~%<@d4t zBRj#Lxpn+Ow5ds-MEV3qn^pg*QF`I8qpw|i!}i_dpT_zYu|Bqs{6Gf?J*4+wj9>To zKJ9%tUiK!IGXD~xx7AMsXFBz>yejZvdH1^QPXv}d()E`my^@do-bCUIz+rv6{1WRs zi}g{y_r>Vj<@;xw^h!R?VftlE&-UDE)ldH(@^h}0zD_CAQ zxAVuwPh_8iRFT^Vip}?Q2;7n_1pk ztNwL%{`&gg!t@tM^)CqRm9GD-cK-VMwf&R%my)kCR-Kba`LD3+gMtY4b&P(DFaK>O zKjrft{~gT#cIF5D_2VPqztf~A{d@i2V)~6H{k<`H-J9??+39!rzr*xbne-FL_q$Ab z@WYX3_!sE_gO^_etPJr1&VS%D@rRJd$lu_2sCZ7i4`6NI;`TLef5Z4jZJ(k4F&`M% zq1RVvc~1!I6U;w_^@#(Se^h_eJez_?+E-Oa|@t=cWDmSo^;iYyaO**#1Uoe?Dw~{e1zGw*NOL zZa>~9g?_`yyZZiT{P*$NFa7_8sQo;?VYeUdyxsm;(*9Yo_Wxb1{hyz({aI=Mtg!v{ z_X|wg{%a;~|0$~d4HJxi#+M(j{nG!Ri`sumto_|?`@5z6=KKcYAw>V(8*BgHPS}2> z|GLBWPjp@nwe!>ZkN%39pJDw7_RG-+JnxvDm>+XOe$?rM!X*3qj!q@pa){kYn~`S=7s#Ivk%2d_#yJ~R}nwv#rQGb<;Q&C z$Nbp%xF^Ps|9u>O%%37Z)O>Yn^06@FN1c51O~Mb6kGms&ER693(?wff^b0@wWBm9` zj30L$hade@o15c&9Y#E;Wr{J?b9=Ev#6kJDrPxHHC&PaTIJ zr%#a|Z7+i#OGAFt$;XmO_#yJKKjO#I7(X!Gw)t^}@Z*daKRy}b$G+q6_Zj2v)5#z_+QIseqi~>=Erj3$MP6I_Qd${(c|!A`4stq>ZhzvSBCtklaKNw{1Ey0^N1fSWBgd< z@?(|oV^xeFe-`7%?Z@HAs;Tk=4#-sP%bJiMb@H)#5`Kt$d?ez>nixMWa`|zQ@Z+Kw zKmIhvkK2yJkBg?t4_SYnn*Z_UkRNsOv33%EhBc^W$y8kGIA6@kcR!+;|**yltxdC`_Fn?+E!(Cm(O0gdZXw zJ0gC(BgPMGH`x4mr|{#QF@9{1@#6!>;m13t$`4Fnrm8=FE96I=d~BSAA0i*`kNELh zF@9`v`LRj(u_?xnKaBBX+j01@X{!9d1b3?Z_?=_(m@TvV9(64L6GCS7yJ|3i-JVX#o*Y}%_ z25*^lHNwv&-WB^PyMuhXfZQhjeRt57E<128z0rYN(p3k}r1v=R%=DN8ccjN1xSYoD z59_-lNHuK)9`{|!eO9}J<>`Wj-yNKvE<5nK>5UdVit`7fR{5jBDXCi=xF>b10~b=a zTkx&`_p^Gn+G#JU`gaAVV7)eiM}zLv9xMH5(3QHwf_DWyJ#P9rseKNm6>!d510pXGQqY;KJ11R(_;E1se+Zn7#q`p{+@M*20el7p3lX z;DOW^9Qe(tF$+d{JzU>WzMvwqSzikHwW)&^el+;C)Poj`^1303_?W&C z^0q8pb^32z>Jc}+esF^6vq*n-dXJNSb?PxI{qEqb^q7NRm3rL4pOGGS@QYGUIQS)L zMpzj6hdj)&`0eis=3stg^M7G#+{zDm>5cF`=&#-jBbei>KV7iO9}W6aN38t2gZb&Q zg@-&1MEMtcl!uu}Z~AvvP{91m(6b)=pPM>rl}CB#2-7D(PdDZ>CjCy9*IN$pDZ+EG zAb?NxC*_f`5qA5x-g&S?O6`;e4w+>X)jOe^=0g^<6_xE7SL2 zfd-$dpZaHylRn!QOW(cBNuNuPIq5TVW9hrFAZC@HPLDh3d0lHn5Bh&?1hYR+ULC=> zuQiyt%7r_BErNMJUkB!kx;(@)pnbE?GW9<#{O-hpWk?V39ZOvJWGr~w_?h!vxV7lQ zEz$Uw^5uNB;U~(^EAzn<|N70*srp85DtU64;A)nMu|RK zpvMJZ*gJ`D;Q28qV0Hcs^dWxa29_rGQ9dH)(URUi`@B^lLkIm>=h5Qr9oTWFr}{_P zSGjNUUd+e!c{-lQOR_lqo;J>-IrbOl!6ZHp_v=Y~2kB*c#8(KsbhEmyMxQ70ONb}% z%48AvhxY8k`SaQqw2$ZA1a|L#T)J7pPvkfZ@qTcUS1x;HbN=g4<%|T$Gw0)$ZcfyQ zH|Li|gQcf!u=Hd6yq2_gd(__1U`g6ekNc0E@<)R+Pb0q!e-JN%#zf~0ZbQ6>-G09U zUxei8^G>6|d8ZYu{J3u!`S~e-W@K2Oj}m$1eO12PKWg|jg8gbKPvb{$zN(D+hi75n;A=&HBEDrb$ecC`P}(o& zDO)t2
7N!#~#?h4WnZ{ZIPGqOAhcM0XmAI^Jp2SvUZe~0^dLw-K=Ebli?%lJ~~ zk8pmf(bT8M*AfMbe^_g1A8wNSk!x*`C)BrXXj+pudSyZq;k+jAuWUng(|CUNNDu0p zhWklG{v`ee_w7gbVWGXnpn2M-_5-i|Z``Mu4ex&fp7--nKjD0*+~1Lc0~S_bza zqJG?GE&53E%YB6%(S1Z{FWQ$Se?@;#p9G%V2R@h|^cbIB8bNyCjlL;WnwrJ_HH7*{ z4(y3OV1C>$&VZD`vvF%S@(*N>s`0D+=cSv~^=4*#puN%kArG;CQl8i=J>F5CWYGGJ z<##l*{0)dNlm0&CTuPse-y^>-?PYt+dC^Yj1LC0%5O~J(tc@a0BJbcI^aASjotJKuC}zKdCRIe625lZ&n3XlZu{nTIk84|0%DPv()|F)MtQi3rgu8 z6lUK4!v1^aXV9n9)O-!+Uo!&dbbqg{q-1^~_lYC_THr-q_4!PkcRkBnruA>1oR5(E zN~J%*pMmQRX1Jgx`uOqs^TBocIQw>|zYgJkTgk5UL-OXzDW`D5v{q31@=#?_gjf7@ z^wrI{Kb88Z)`9f#_su}xZ9z8T$Ma%OGTw%FSm&GXi0FUj%J_V)?*8Pd&L89Hk2{~l z{pzE^shL|rp3%QISGrTQmxdoB!8Awz-CXJEI@7}6g!=;F(0PO9=k@*deci$zZSO{d z`Q0L4A-u51#V_cw@rTs)(kStRG9Micia1ySdcyqkdTbc>A=N~@KL73yw?^mDpW*oH ziu57+=Vpu#`ct(lRDYL()>g{5u?KPaL4V8qqPDCm_Ovsj?nCYpeW%8EsBd-oV$Uc~ zke`y@)g}Ix8V?w6{YW19j~Ks^ydnQ+p!Kb8uaDc1Xn&2pJLI=ue#cMQH`rVC9)M1N z$Zuu*P>A0Z%msU*`bLB1w9P->4{hTQ`B_a5_z(IqKY@CR@{a_~jyxFqjQeTDUNJt^ zwx`s8La@<)f!CS^bw>Zeo;mhycQC7k2Hf=b?qFt%EiafKw2zqyj_V*R!ATZP^d+Hd!NW2IHj+aQU^-|nD*@}m5^gAR2cy5{%p zV7hv5K*R8#`^jJBFZa-&X9sAQ`q|Y#z+3)x1Ll8iq7Mzdun$WAY5Rcmy@jwmgy3H1 zwPeojuAbsm(H?!Tyu`&qN=l=y~B&_GR>WD0@IcFUlL}yxV(`4mIEBzr24n{!e+{LB=E2o1FfEy#jsWz$*G0z|V7?@iUMuMD;%*_CHN|GVd4om5j&v zfs7~4Cw6?6r^E1>^)Mdq=7;vl{ro9q{}{gv`ruDg$vKr` z{lxLY{D@zc`{m>Im*atW+@FEPKib<&WN&ewyJKH4zk$8XcwC?h@6YJSa0Bkd=PR3I z?a$1T>{`G0n>)%FPMI=)q&}-;u#v`3@dN%m*2DObU*dn@FB-;s5FZbEXOLkTPwB%u zf*CV50*&_@L_h8h(!Eu{M&5S^nclJkH}#HL@Mv)Ij13OlIiu{rvuB9@HTrY-BCY?y z?*m+4N%xjPkEzceUJKgrX@8vcqrDb>wI|~@#5eWY`U&fYGYVGy{%|>}5By+#w!ZR* zS4H&+J(*rxzxl(<>*`x&l|LHv%xJS%6M?-TR6kE&!fS>3|k(t zK0RZtRo-Z@V8+D`eBq4CEZ86Jjp~#BM*lhWt(96~ z)1CRt&6UnQ;A0@aQl2KaoE? z8Z4jg=#R{=d15T}`cyvCX5L>B|6w%9!SRgB_wgEtaE$38FZoOg>OEZlO704#q4=o2 zLw+;XkNFvTbFFs$hkR@PuglZ^u$-@ge;?(?{L~*_=gPC4e}ewQeTa=1PF$bRegTTK{ytP` zQQHx^KW@YNR1@)>k7EDUA8yn7(&!7h-d2|9tr73d zP@d#Ne$F?6o(Fw~ z@|rTsa$$M!_r&2>>G?iXIv=iw;5~Sx&$KLCuF_+_)SZ7oo}gbsdM1#kn=7eSd%Xer zSB(d*AECWYf3v?Tsc5}vLVI#jbYGlWUyAAD^D`@jpi%61^!&H4FJV2Y)m~4!xzgO~ z>f^S!KE`?jKlZQaV`n|7ULQ9heWV{ogXyvLrn8*&qkLwaqmO;emNI2jr}Z`MgKQ|# zK0v;qkD*VrJ{EbOp@yrB2lSUO%hQTp;{B|eekpvC`Ca|n7~#=A-~Y8p|BJr3I2!+y z2fKX0j(ui+=nD~WtuLTIF9tD+p18i4pnR+!HHYH|>7kFCB7I4HoQ}$a{_~R;NBS4- zg?)nj+x$<-`mq^*CI8}eQULWc%Ex+5U3v~^Ne}&n`Yrv-{b*jl9+i*%FwvKheq9&i zm(_mJcTRsmpZ=Qc|E;4w#rT8$SQqL~&)u@9D*)Cl~Pp`p32};6Lm^reNro z{<7=~U`L-m)0%YjANYszTSEQ+bWER!yye!0^sqgZjNX6(eS*)vDa8MX`rp+z(BE{& z@iF~k({9LO20<@g+6lpRnf1Of2#EBK*YZZ^{dt=&=-)G3G{2^zXCrc z{fhN*N8gD4wEQ>F=lGvW|6UZ*tMo737koyaW$E51Sp4rg{Y&>x;g$a7_L+u_{_P6& zOI-g>UEf*$cc`zmK863x`AamOX-{&e+w{QxEp}l`AEJFqKVKZ$iz(?>*e9h=`@{Ce z{posrx+kLy#%o2N&N{X}h5U>L>t;p#JrRAmb{72$<1ZoJZ`K?7Qu71+bp6r)U+T*> z;dq?bAAPOq%he&hN?-OzF#W6SsUbYEzFZXImA*VJf{nhM73z;)jlNuG>9;z4*=N%O zeYr4%mA{Dfg|$}x5sb~CG+o@$JU47cM0*AYW-X7r;G-xX4?9Z@0Zx? zU!y^JrmGK&Gi`l%-1#)>bL=JOWA=U!;$58iJ^R<*F9Gb#-yzS=e$`RV&yO=7XS^=< zD;N(T;}`4k+<$s4&&Q9uzKQlvww`(1^-J_ee7zIvw~;@m=GW1D7yRdbQphjNubug} z8jouEpZerz{ruY5FG6}}el7A~?-xn`NA-hVXTJS(YtmW2JQAB{MDf7u=vZYqTEkBmOl`$t$$gg%@T;z5tIzIjZ2xXRLJ zb^4Iu9QdfeNzb$>{xZ~uWh;MNA2vq&O`r$+PldH69xn6;i&`Hxa{o#03&kE9;$Z}~ z;^DH0Pk#gtsO$JU$OrhX;@>vNd%SoLS;d!0{Fa=jc@Yn+M(JUnqVqJMXU$O38MbuUQZzJ&EU+_^=*1pZXlxqwfy|J;|-i&+`X|M4mh@ zI1H{Hl=%Ds9^aBr#pAUPr+9$Dlk*4AA0zKYbN4}Ga?u>|%XqIa$#`?oClBy^f}irn z!9WVxRjpoo#ZSn7Y#;Vd<^JOK z;Y1>=AM|d&7FUrLHmdTEH2CrT827vr&Rf{$i@?u#`cvM^%IN~2O~5bR+#%bevMGxC z)(j=QyBT1m;}ZvV6nXo92xC9M>91mNy~Mkl^8&)JLz-WkS|Fc|&UvHwdpsCVffTRh z8TXZ5PXU+xex4@?W^E;ZO#Gi;nH}w4LH@zNLE+y~oYy!Y`wKkW~9O4nf1 zO5(ZNgITjcomYDv>Cyh1ah`zzHMPtK_4%cAO23aI@q&D>z5x6n`w!_6U)D(2te@Oe z@g#m0`bFZ=O}tHNI^uaCAHTs zoG$v8@rrlshk^i4U{;!(+KFBpgzBe{a=ZF0p$X?CYN_2VX|KZeZd%SEL>cyd7eO~A) zo)?-U=Y@JDett2|3-$K8=Y^7UsEtIvAWy}h(CeNT@{{g)p@N(jDum~Sd>$Z!{Fw7X z37`EDofn!Ly$=rlgFiCf#U3erG>7sSofkrSz(No3rAkj>F2U%$P!I8;KC5_h=paYu zg{C|F!Fjzx*go`+Unxf8=b;|($CshsCHEKfp#7(+{S+N<8}DzPmy_|scpQ|+{#OCi z=wHD3;M7xP{ONcdl<)U|KJ1^0zCrtoLcg*PXkRi>1WMX_V6pd!>%3Ea1ZXGQamEe$ z7U|~}fED?}ehlpUTrE*@o~W_XUr^(tDlpD>^@sfh{&faP*?%?Xm)a{y6_0k<^j9jm z5#>kyHIMg4gG4o6P=CLSH`2@g0xb0Nd>7g~@X2K!=@a>W{L1M3X&B$_+Be)c-B0_` ztMy4~1ocUN#ZTz(PLYRI`o8eM8=-z1+#vUbCxIWKybW%U`@#j5^g3R@81%__n5WCn z2MfFG{wfCZ=IufK{aPP{_cv($H4@-_qO>2CKtBx(wWTOuXrG*CjoTC0Z%4m7c-Zgu z;p9T{PoM8eu8rxB{BTn4W5N4JqTh$QQL-i9OM$+KEfE6RSowp_vRoTRqPK*IJh2g$^--g-U-+@ z9ftS#7 z^03a&^A|E+d5Y_S9xtvRQ}9RS)A#SAKa~C7@b6*|8{d=Bhj$@=Xb<$CoagZpXbjHHoF^H`-tPEc4*l5wM|zju9glmz-~i@%7_>L^4-NgGk606bV~ zR)1*U`-eQ)zY~7`5d0kK6aIsr6Y7IVzR+J7uPY&+>Qnq5*x{a!jMoo-4T}6K{RVqd zlkp+%B_Td~V5sQSkMqKPGG6@jt)Vy(yhsDWe^p}k9GRNC`uYmJw=k33Dh3Aza&plqIl9BViXg~Eo)B92S zRU&0H(z6{^Cd`7PdD)K7Zo)3W?Io12ln3i|Oq2>Ov;%4-8$Y(e>%sC={!^ou_% z^kRHvFuq0J>2?AxRHpI#SlQsSm1#9`&%COP#~oX6e7T3r$G<2)TgmDGO5%Gw;Im=> z2);3^{W*hg^m?lJ$>dKq#_&n6r)=Tb*Sp zKR)}jxUa8oBhrh01Aa+G-n%MzEAb=B%UtWLzGMHu|ABlc{yzfwl>W$S{}6ca6Y`N( z_NXlRb^ibf80mLx@!22Hu8Nr$%-SoSphwo_$y*v+6^U_RDd8}NM z4)N$e*u&h_NPqe%u!l!d^bdN9fX(^HQcx@^4Xx-Y;XbDj#(BDd?2oW6 z=G!#-&==$%rkCM(fWPI}wrhXu=|<>()mrEvu}831v-yiTA4&hkOX3IOxjwYxMFRN3 z^i|R$^BvxYu;Yke4c-~A;P2ASYPiUxM*PpzY#ERGJ_)QxrFdRc^FN9CqnZ!Uet>`Q z|5UsRmxnw^{$tON(p!;U z+6#Sw`B2u!0$s8lcY0ua4%~J%r}Oww-g7;5_C0P-LC+~6J&)3UUY+869{DG<-|fS` zPupSQ`Q!d;Tz}?~o}ysMt;TR`bedj7H8*ZQ0K80p({SyYJjQl0?M%;0?-90-#}V!UlncyKt@YJXYQ2b11( zO%M7H^eF$F>m%PKJ5j9Hzm}4CR`6he()<%C0^u-{JFyub*jflTt9MrZV+JwzeFEhZ>_g-J~Dl} z;wbC^K1~03n)u_i>Gbb;{||?Dhg#3`=fWRm`^2Y(e+2$EzX!0$ zJK8fB{x|iV^wGh^fR(-!AMBx>?t5*|;Ql7=ADH!U=sRcri1ILhmGyozfBn_ZC-wbJ zT0XEI8_kFB;Cg8);>V%&Y=0_jKkaY!F=xET=a(bz#q!EM7y;-d>i5w9C6lFpJ0J~_ zKZNq%$G#Zk{22W`6ZetX`V{lg4w=g!4dp#v-n<|w`z^9w0)H6u)g;CH^ci~ozhj&^ z@lBu~ME;einD+|tKAgy(lGh&4GcBSA@;@;2NJ`FcX#1(>J8Hgpm52Vue#(Qo{ov=; zp>)UTZ2N4kH@p?=?U+9(dXjAK)}egI>3tf1q{YMfWwhQ5{RjR?dKurKck9s1j?)V| z{d}%(`qBEP;M+P*r-Q)y-o^3j<2^cz-$QD@W9v{;$LSPa;;-g-0~!Nf)<2a#NmBj? zH&kRj4Cnu%^=s(A(k{Tji#tQ@q!;sb*cbAP z@-nyq_h*40o#y;A^JD!Wf9o{#6%-czFQn)CFuO7;lf_PZJ5u4f5@*d6jJTMeq5UC3#sWi z)`IrEztfZX=INcBe>A|}Q`__7c&zb$I8K+)Ut;fZ-<`Go4F0}-jPvnpLJ8^rj>^tdE{Bi6h#`|fa=!wUN_1c>tZ%V$f|AzIt$bLiqpda7T zsgfFf^AAI`7WlT~zfJZ2F56oso*(!NdsRz5BKyf1ju&6}0UX$g{_){|$oeYe7whGV z-S$Dgw++ER;D&7m|DZn3N22wL^`bA>-#+D=@(=wk`hjEuW_$$xrsgk)7K?vDc<@ry zKfIq&;&r%uoG@`_I&;^ex+yqCO%$SAibbOI9G`Pn9pQlrQ$Btg?sxi;)*d zHRU;+Pa+xHhxHjL?@W%bR`0;Ic#qS3ceQxCD(XJR+L`1R=pCFzi|AyZ|(GM%GPQL z^beF4p2f@x%n7u>C~sg$?Y0~F@j>r6*T>oa;P2|8L2-pVZw%uH>*+&=KE-dQZ~OGI z)5pidTi_3>Px3ds9kAlZ5;?E~PB`lPoZ-F-raV7yS`e(z|d9Rtq#2tM=uKDs zE$ctRFCKUg?~k(SVfx3s4mW+9cii-4ui&Onhv_GjpUqyMo4)1v=`Zv60GwI>a_FD1 zd|W5#rk|wz>A386D&_y!;{!?-J+nN_r;p9gwb)Q`(x*MF&mBAcws`v3{ls>EQGOot z?r_VG?$fN3ANKDe?@l*;?7m97e5T(}pZ@CO(f@>Zw_ASfzMA^-?{(8h=W8dVf1`Ke z=~saMd%Z6ltNc6S<7Erd=ez?`N`IMiKQ-htqJP5jN1gb)$(bupyw71xgX7R?pt-V&lw~&JNt+g)RZMJK#{P_HHVCeBzP+!j^ z_4Q0p-cKa*S)Y>8KyYQnK+co$!jP;CMxPQ6V zclibX9qS`G&+KO&ZS!3I)%+dT=5T?sm!nVaUA4q>`B$rJ57#HS)MDy)_CG;iY=0B$ z?dpASX&?M=cmEXoqt5;+^IQ9&pg*>sYW731eWm7euj&0##54G~pET?*w8z;$jmtOu z(<$wz!rycI6Z>gSe_|T%jE5EOekt)k#?HB9vS+q?*>pO-%nw0|2h8y?2Ipgk83X`?Voyj+MNA_ zN%-x%@?3u(xZD2v{kegmC!G4?^Q)zsRXCcF#|=XlCs&KV{SV?l$J29tKt7HKjE@aN zWhXt?b9w)q%=d-A4c;Ipu-xwf+!jQw(0_%8te+n3iS-71p40LDmw9TwhVq|bd$2!^0G!5y)kdz@4D49# z;87n;CF*-r>ZARFIV2p?qw)W#_N#9@OXtV`RQZi8zsC0;(H~vG^Fq({DML>|r+4Y8 zPp{&q2C#c%(~tEoS^qQZ!Cdc@^|MW|mss!eeV4!f_N+I?{-Jz;e{;7tlKv~1N8no- z9;0FQ@2%NvnK-Wh!Tz)d%@DBaSnXcGB0u0Q7hJUbln@`W)$0FJC8r)xVZ1Wr1Me?j zfudS#)cZeeNWTTyQ%7r3Ugp|q%HL?ikG(C7KdX@-uL1H%dMa5Jk41V&|K=ptZ>uXr zM`M3%ZY4R5{E__{@EiL(>V7HwM*brnO4c9IXamdoaFkx{-$eMd>>r4XaeYgVAKafT z`7s`3fU5N?(wBJmEL`Y=JPho(+DVW7E7iU+{Cv2zi}V&Gh442y7ycP}+gQaPz+-<) z!DWGEKS<*^e4(!@syViiu`s~vNh71)%_EVe~cf!WfXdoqll$zlYV^+}OGiKi3#(VzKBTK4}8{fLi7YJO%sTl@#Tsz0h*qV!3XT&KtW zhboWb3-reAX=txF-kzcUOJczZpB~QxLj|WiuJ_`HYU5gzC-t-Y?E2mHMZ~j3>CqpY zf2vQ_myz}Qv6=*-Vtfub>-UToO*X3>;4<_@J^kc9K2879%@SS}`db@@ZfQC{v~Qd* z{vw(`j>XD*9m1;dZ~XZN2j1Yo8yt9p18;EP4Gz4)fme?MoX5b!8hih#_{(4YvUUM( z@qW{pJFdFn+5eF}tb^{YPuH zXPTJZ4V7q&d-oxk-P zXz_s`y>QLGpKj^-+5XMtXP*4#eQJ|AENDfo_SNg&H(dKb?d+bN`qwSB+EX8S`o7Nq zeCj{($9-4gkJ>Mr)ppOr=9O> ztrj+X{L+u_-+$TV{BGI5|Hl1$_p7hJfB*je{d@cIPt(5rO?z(y41E6bz5TuW`}_Cq z-`l%?r2ocCKfcSmwO0G$OCNmi^)F3(a@oPtYR`ZA)3-l;-_K`!Xw8fz-#POm_klRH z<$blwZ+N7qXY6~;UUmPowadZSHvE$;U(!(a_U-q|g!k{~*$Ki;?4gD|>i^yz^xVky z!ajd{Q(?b{|50Wd+W5zwx-9)D$V;NLS*GXh-&^gi-stuF`!WR|r6$Y#qZ-!WBsU~@ z-o6`sm1$3JH6x^=KA>2K;oO~`bk*9!tr8UHwT1EoB$q;HSP)IX)h-FV~P z%$~4u@2$Ob-~V0x(f@Pw?AvOM|FibgPrrBGOaE5GX}YKW{9k`@&1X*gXzlqAec`2# z-~N`59(`f$fA(DV%f=1gtTpccS#7@&+AZI$eg0p6@&~($OB*Ok$NPd`)P7$3W$mRc zwfi7WlmBz&PrQG>^JwXl+i{K+ks1HvrI%_iTwnY7)4%-rlheUkbV}PLGcIqKzTxuZ zWivc1YBeO2)qkvg^ud2B@A<+1%=uqO;KI7g$xu-kGcI3q`I;HY)!X|ao_h-ZNV&oD zJ+FWNzWw0I$e#Y*{{85Qf|TrC1_{4p`pUM;)~sH0$tBaPm}3@joCgx!cVi*5zjyE6 zeSJ;km88O3bNS^n*0d*AUcUOW>6fprjxhAcD@fAb-o5>M_U`TXr%(6BUV8S*FaG3z z{q24K{KD6M`OnY3aNoJV`03BTy?kWLPsz34e*rq-+1)>X_R1?;YPFqz`-$Iws#d#j zh&F7$4D}U=EBwyKFY~p`Wx`cl&(yhGMZ;Jot=bBo6 zCV{*+_zf?j&t;JeMeAxKe8Rz3HU8@j4YM~kWxZb~H4S4}hje}clhrS0H_g>)stsec zl`Y*({Li1g2`AfDvMKl#dzd33(>w<3)Z>v-e+)Rfq`{y4&h%_uLX({xsTg8J{7J{a zGlYov=40R)@*#xOMh`43Qpa*HI zQL4r?EREJ_?zMl@d1SH5%0pQ4OaX7#dCdI&TeqF@qIcCs2$}b#8dij=4Nag@KCUTE zh_5O<0{~YJyr+rnU%A>lzv0a}@Sv=a__N`+uL7Ko;24h@2HrMPjS|&3m0s3pB~LUe zZb-yO1y1C6D|5MIG!j-eoCXPPXiK9`eqh&&-i!Y10@~+&jAK#S=b?R7;-MeVo0nkF z<}kKa0!GMVL%Ct4*Cs9U=l0G2nOAP`Q4l1!0XH<1@z?NUP3T(ugnu*Mi@IjQfI=AQ z@e?Tt#~sJjNHp%ssz`FNu1?>dQk`f`)<^qMqyFM@IVrzFDsO0Ts9dEAJRU#$PMDvb zU-mI{r_{E(o;?<&BZ|^EKVMFrze0X)o1C9ZUK{*8@VemV1C#S}*=vKJ@-nBg8uoa9 zjmc+5>5$`%_ws9lpKBGRukrD|ZgPIEeQogb#@7WuZXRjmLi__xh_QRXnOq$RyA=@(^wb%Pi8B=6?@Y7?%0P|cS=SGHA} zR&K)FN#;~$uBJRp%PURB%c56`)`TAA4VdmJk8;(#Rnu3_eg^ZO!)L(T{J7?H0_^1k zPd?6X>io4>f7a{cMCh;6pOt#x7ssVdJXqhKlE=le;(hFTvDWOdtlIEu)H{JZ#AXf? zaI%4%R3lBz16TFs(yMZ-(o8FV2VBQpnW0284$Q7N8@m{7kONZ3KCm>98P8%baX&eBy`^^W3${^*kPsk z2dj+Q<<`<$s0xy>Dp(cc+4tOge%<#z-T2_g%#c#wVzY66=Q-!T=j%P^es-MKys`!V zT%6bS4f5NCtJOoba+k|hXXPp%acy6&%T}&3zkRtbTe-^oX5~6|s^b95mOl02M)U7> zX*Qj{l(MP0u(!B3tvy!O{TAtWU)t@`zgT?H((9OfbNvA0jhnW+wnV|%brU+e#diJ`ngNH zT>cL|mnGJb&ey)+=epq8@U>6--1bbZ6)MlR+bDh7YdG11adm)R=5fRBD#GLD!SqKP zy0$$ET6yNqYo$xq+3FhX!bbcwPcUGfExaDnkk7=wQ=fEq(`D=m zlkVJW_eSP_-(9{-+Y6P0OD_~km-0+I`a*7xMymO*MY}Th^CnyB@QB`BnfnEWQ!luC zKizG;b*Wcv;D=Yxzu)?LLvJRX9kcrg_F0_Wjh@o&#ul7@-Mxf5h>z^@=>N9;q)Bnl zR3qN)*hyITD!b|Sjz@-<8VHa~< z!?r5%`BMCV^(okz;GP8+Nx&>gcKZJ?<{-Am|%#se=j}n_V2aVMOMa__uyU+`Db>wKXWJ(t0Cc@M%bvZ!Yd^!Ha4-M)m}R2 z_K}P)@1wJ7e5pSL_9Gr!rthyBUrz5{dKi^*GYi~nAS>7C?v5ze0tffh#h$W=D|XvP zTF+)gZBl3Ve#P6QXd!F&@OF==#FOE^wt50$J}giC35oC|X{o~8OGdX!zi9Dhp zh~KFmKI2k;qT-iqck!niTsW-K*b@~c&1Nq56o&hST)Xe3^c02*hiM)nKEBUQ(^q|B zqG)@wO=GpYuQP`QL#4b=aIkxH2d|E0l820F~3!rJ1JQ(zy6k zbZ=Zme5;Eu-)HZhTPR&u@g@i1lRa*3C~@KQ4JFRIp@?%&0F}7Q?g@huSMYL^zc$8s+cPG~?|l6LZP_{yZ0so-&C&wGyN&pRzDZ@PtU&_RAlm?e5;l z_`Q(fYOwMy#>85qTKze1<&J(T>b1OTb?@I#Ra&=QikS5&T0ic3Z|TXp%T?pu{juT7 zCR_{mMr&cWroZ$;`eRnX+M-p1kB?RrmlWsD`Bz#Vv#hR~W#wn}E=Mb7yDoYx+p+6d zHgC^lp5JpU`wNZwl-atpTU{T^9<(^)VrkN!%-Aeurnn>Nx}Z6J_-$@4h{wN;_`k6D z)TiA&us(ezzdIg&J5$^h4Ta||AE{8fvoMn&+I>%%Dc`G z3%kzk`d(by3FFem(-bWCihQidaVBtmK6Z{GI^xQ^yQoVe$JLV6=r${)t@>WC>zmQ> zSGWSZWCw}X=bXXeW|vZ{^eK*p?JPL@g_l1ZJJzu~L-Xd_N>AO{GamV1yL)NfjLrQ< z{q$gCtY1Fo=9;iy+BlE({fbeyXeD>Pz2`A$(C!S@k5}&xXUKyXf_FfsEyw+YB z@9r@uUq7jgzns}RzHXcN%gOTReEGMlzsHRKayq@Qy~F0OtXo9dL5UsF*o)SzABV>Z z)?i!2r-Nf##%F@AE#r3upRX2Qr~Pkas_p+$$-eJos_p;MZl!;ask;AC{9k3N?f;TJ z-t5a?Uv2;CQhd6;>i+G;bW!?Dz1x5384UFBQVX;GOX+vjJNruMF#?pn6z^7Tu^y^; zw(q5K{WerBUn%}O4OPoms{aQKRqMAD|5puW|FL~XD$bRK9vtWVLr)i)r($MFY&(Kw@8AsjhkbF{P+cj1? zo3VRv+?J}>@p}A--HvmfsjE6>Z-4w*hFJMZ=OfDJMJw+y@_cTK`bgx@{kh(=6U~@m zv$JFh(>y}>ar1lJ^sJXBogedm(&1;#-X~rD_cg!r8T_Nq7wYr7--rJHzLwuzv3P_s zKbFl$Up`-Pzx;b`Lv4S_`RZw_4|g2gSJZKhwV#WA)g}w=14; zq8*PL-pM|8!aS>XRP%q;%KKil_q$Pt)yl{HR4Xr;hYddopTT{WeMsv0eEQkNxoiBI$ES;=ly}V2NOxU!OeLN|K<$KcMo!GhEK6jGGPMGcX z^SJR(!iJsXu@mOqD9gLC-$%*wQPNm_Z0SoLp}w~C6FbOnJA0V_Gj)G9&mYFF-`G{< zLUd=TyC=5V$F8yHzOL~8HOK7WPE)Z;Tgp3jKU_(R=js0r(^p}ohiRQ1-saMNs@SIc z>}0oCeM^sjr*M~N?s+Lc!OnPhg>1Vkak2Q)*;#k4R-0>O|L|BS zxmKfPgyF;E2uh?0><*Rs?cZYY3l<&}Z-tN$NKJd3} z7QC^#`sc3pZFi;5U0b&iyl8je{Lhv=yu29;yBkK`YDDqj3dM*2NW3e%Y3_d0k`@aq z55a=tW{)?2oPAhfc_A&inm| zH1GXf_?dj?mGit-hQx80o4Kgr`lfrQPhJO$-ZMW_>oh-;{AsKm-HhbI&tyBgyA#=F z?9+^A*q=Xo*4puzzD~|Is5TzSY!^Hh&D+|ZfgZp&bUf{&yf&h~i$~OUBNihrZMWJ@*spBY zMx1jv;k{N?gLykQQgT5Seq5hh8*xvzKA);NtTy7T2A;OJ!`4RJ_t1#LoK{;!y5lDK z$l9D{RZ)32z17guzD`#Qh9|GKn`NH1cz69Kh@SqfcnZNiiuI)J_1A-vX9st1XQH#S zKD4v$k?c(2?Cf^ss&>|7CwD_tJJY$7blcg(_~@EO;Get-cG)$J@{>zJV22A#;hx0W zgb%NAJOm$J^LPk8yaqy^Vb{Xhs3-2Wg}qznm$oO-6QdfdkJnhbbQL3T&x%!8WHq)4 zpLWlVJp`X|&yqa^zso&aMxJ3;V@=VRcvotS`-^HY8?V7jrRz}PbtI;!*7IFI+$4|i zS`+iwf)B4fJp>G|l|P`Ew{dwudZy`Hvr)6K3i0h!@LVy) z_-bbWs$EfFI)?py_mO-%zy37zdNQ5H(UQc?Z>>|4QE9z)$aMX?p8jF^JJe2d8W>a z)J9zLxbR_*z{ziN=D#-LoPV#4IID!(h^vjbihI;+Xa1k;m6XK!g4&sX&Qc6N-kJYz zr}B30pLy2K{MXL>vro9Y=Qp_$QM>2Ae9pamC8BoEf4Np-&xgEEZ~IByt6V$tUpw>f z4rBknyp4_O;=wGroPz_!|mtBttYnSXnx!VQM<99%o|pS=Iz zJw0-Jyfgo`d;Tl$;dke_YiIs>R;YI7zkZW9+Hdmqq1u`Mr|Hap;$G$2ng80Ef4`ou zo%xTQv)aL#|HL?|-SZ!gqx=(o^Td0)e&jsQFwXpE@|j>FQ#qRio7Rh zMQV2-yWUnl>ySL=uD$K2_O_qM9ohe!!?p`!j!V~mS;J~ftB~4=i+WV;>UQi%VY?Bv zU3_iC`4N_UJ5g;_S@PlEd-weB4v?^V$rS0DF4}RNs{Da^29sl6pr~CawD`LMW zUBAii*y^l2_-)_u4}RNs{DXg=&iaQ|#C|n;yEpxSPjC0@zTmh0bzkuB=CAuc?N4j- z$-u_)lu`k7gle%*PvQmZC6k6 z_K4nAt)BXBe7j$DSHA9Vg}3{5X}01Ozg6Chi|;e_$sn^4WP;R2YIk9OJ|(uSzr8zX z&!=3PvSz=;v@6n}2Zc;NRpuZ}}2)fav+9=81i@v!aZ)p*$U^N~&V^UD9b2fshR%bZKm27I`U|3Y2B z->)x^`?{*$%KZu#@=RQ#n{;`;&|n_$Ion6w_rLR!y#uctZbJC`5dJX2A40eh;SVDG zGQuA~IE(O02=7JsMT8p=egWY<2tSW-J;J*Y-i2_+g>7xW7-Z7>?BLxUzsKvd4tyHn zR2crkh_y%L4_$i8Z^iO!Dd)F$&v(N1x}18|W|N~04Vh!vj$OyH`Nq@>imOYzxMSIa zjZTj&K%nwCp+7+7KT3#n)4% zzbp6kqTAPtKBvwd?90z}Z<~FgzlnvPciH%RhTMGUm@sQ&p7yIH{lOMjGpgMOwGveO zz0>5%D@eQA_1!6WM(>VtD-oMqO0BJM*xG)j;^LplmVT5s4}ZC0cD7o0=>Eer)%qBj z#|@`W)SKUEu~C{zo0oa$>*5c)|7vA&zKHqIpE15#K4ciKmgZEAX4^Qpg4!fCi zyM0t+%I$YF?IsB3Q7sLbCmoK>-fk6Fr&XxD9`_gH-Kwy}>*!|--mPLwypEB{yZLV? znmaxs$8fd2N~T<%Iy+lixeDKcxB9ZhfnT+6Ute`rUuAx?`f@7}?z5V<6DC$RtNZf4 zH>>*{<$3B8pZHD?hUa@LQ(O*u%b&ed*28$5(m9J;isJU-`Vzw}WC3>}I@$pY|&jiMeg1wAzf9ypSrs zQ+ehox&9g6!RqqZ3m@O>&cc;@Ubr@|*C;8tZev?HK0Nj;#plD=c>HB|Y+8yh`gpfe zF6WWlhnC(+Cb=Isd%7*jB=@Be4aOa7l*nK$|UodsLCX5`s1ohGM^>S#B52_&*!V=k=#$ZYFi}tqs4u)Zj$Z(YdhH2u^sHI zt16S+*Pq1pO~=U2kLTKxO{+m8v4Qf;MH#Y-mA ztW?Ys+ojkpeg{W$_3mgcy#3MYvERIz<@rqc?$F+3Pk5?(2HUJ!PvKcnrrGY;V*7S4 zWUSqb^1{#PQT#J?{^i{{DzDais%b|*AM9Z-df4#$aU}MB?34Af-Do5Ku;FTD+|kiZ zl_PGq*5v-HogpBzll|E$xyo@EbBe8c5?4@iesw_pal<>YiTkK_3|y_2s*NP(^=^mD z^;wrV^RJaS_pf6`w|KggvQ@cVgWa3oYX^R(iI$O_$PkJnrkDZj(AF+Yqj>OE&08W;+|=`%9Q{tLiLRWu3uv)oj|y{+Q3hhO61jJgV77raS|u;HE9@X&cynG?3lBR*4IN~KFP)$UDZD&|+Nr-xgOxbeM{)j`$SymZgcqpc8F z28I)L5?d{7wu0GiMa29!nR82}dbBl1wb{RF?ZvJ&SmD&-=@&O?@$J?o$-4n+YXxiV z+FBv8nqmKE_`R4HxX*VxTw4b?4{vu$RXIcOUbOdj^E%*R=bP>NdbOEjyEKV8vb=6c z?{wY3{I{JGAL?9(>v^%&#ir*v%BzwJ_}LF{*d2{-p<*kQV5=PGEyviKm%=vMRmSKI%{;@u0f?1{RZdm6yi_Ry&J7Jlg^Iq-3azs z%9K5H0QP1)6n*Cutzr6N|qc5M|bL{2YjjsG32;T%* zSC@}ENGUJ(Cdjaj!!jq{0=erWCCbiRyIyCXWZM*sa1|77?!95^H|cnvaW7nr^_lJ+ zJmvXleZ}v`+vA?U+PtT4Rg!UE64tb{s$%?>-^FQf>vK~>)b~^?Zj6RCTz(&Ciml+r zx$jl~+FCknY~QO~Pjgy)T>o!ke6MriF#fT6UCF-6x5jn5USd5s>Y2IdU7=M*#<5SO zV;-_@7t6U|`N&(OKbf)jlR7_`s<%<@OAytXs{mJLV%^{BrS(BR+`oSZX5O+aZO*=w z;y)UW-2BH%v9>388ds)m+Bj^Cqi>}5Z1>L9)C;@p4Q#GFZ>QQ^f5#hmxS!Mu?j55c zAC^7!>A;8WosX$vkot7J&o8xMugOUrJypLa{gV?1f}n4}mD%d-9eW3mJ$Duru+hG% z#uuXaCi|id$-B>jIy-7E)#t#;I`hQwqiU-Pi+63~wmFNp{`X8hHD>nq$KT?!Rr%&s zzPiM7vExB9JZWK7iK`Fy?p_xTy`m4h>I$bj3+KUqul<_bGqvKg_IjrJcd|bh5@*43 zwf^kJr%g$T@%laWD^TbkiMPHkRQUTsB7U(VJ~e?oA`%MpR}*^3$4%4 z)9a`&H(v8@Mc}qld6M%@ZT?A)-_BGroJY79;oAI@tgpxQe_O1lHW{C_{v2N~K5qTj z`ty6&pWX3MQOBD#{~VsP_?})*J>$m*iS=ErKiB&6Gv1$T{@I;>dUx;osI52NyY)uR zKiB&6R>xNrzXX&#pO8HMa#!b)@yYWu^Of-xZ~lL_=Lfq+X6IL2mvUR}Eq%514#q>G ze+=7rc*gsC?QNvB`6DrZJlp*_c|EJ-k+t?U$|&b9b}!b;jfgrzf0SWh@bd~<+0Dw*lqcB zcEfxjT}=DToNKc@o)IEhU+&a+xxTvJ%lg`;Rm|__>}k30Q!}}BcFML`_?`(r8w=ad z)FX1YBQ;KO!HQI)WDQl-=LFM1|M&&4G2Gv38p-7MNOLqFlF z$km4r_aOWw3x`jyS-g94?I&aL3WxEPPf+Gag^LC-NKKwIcIy}#>secvSqm}8kn9QivLX<&f z?qBNWt#BVls_d_DsnE*Z)lb;&KkQ<%rS!kywBX}mKFvO#QU9&|Px-L>a^-l^`&;$- z=l;jYtbz}_uZP;-x2lxSX((TB*q`jlAHToV*rSV9(LdrYHw3{w`(!>)U;6EF_qU$y zbbp7P$}{0ZGVbq}qqO#Cka;s>4dh{@2z|-?q_w$#k0p*8f{e`;RT|zqjx|v2^xXvCqOETHL>|^#95}e`I~$KBGnZ zEZN_GZnoHG)jq+$v9$IXu+N-*iuMU^Sw8lev(Li6weLHY?k_E0`}D4v4fa|1clP(X zrTdR&+kH!CpQ-<3Y3&j3oPFl(v+n->|BSOw!9Mx_Z2Vst`$8&MX-EaVds9L1l6|tN zpxr*bFQlXeSsUZ8Cru~+M|7j{HUN`N}&C}ho zwErO$KLv_LE=j(zMa~h*xsSDcwD{~tC_W4=++dc#KnX*sOKKbMJ zZJ$;91V3LFPCOh{YLrk0jneF zIjblAz54IVf6sj&UD5}v&ctL6=XbT+q+|aA_k3iKdMD`hVfXyA3y)gZrit)*=OC2{ zrhM2vAMC;lKJ1>KbKw;qcF)U&;ars7<+mr*5Ul=SIln!Q|8x{~`Rz%y2BWr~ar<-j z>`A>G^xBwkVV6E-;lkVH^eIa}=fiIQVSYAF9PjdT@#{Y9>~Zl^|12u6J740;x8TF> z{%;pv@!?e;9=#s%?)cfoPx-KW6Gxcdhu!gFnBIro+ZSAT-G_&LIBWBeRZ-yHvEbqZ z8^88fFzLfN3y1cHVf%4)A9l|Ny80aOVfQ?t3%4&tyo-0`%h}j->0P`FyFTQ?uDrWb zdxHTRcP=cx*dK-Gjd%66>ccL5Sie6T#Vfys!6@wNYfoxd;QD1b|6Retxp;gwaDBBL zpAFh=Y`btNz4hO6{H|c#hokttf$P)d_`Sh`4=cUv=jHfa!RUX8=f5{_eZL&PH)uEC z;lfJq#zQ&Y_CFGZ$HV&jNKiZ*gYoxNdu517BPcvn6bZZ|*nyoI|%`)s@nMqy{Ki=X;h#E<)M?=MCB)A(NS z;WCecKNpy7H;NPWkYnw{IQsy_X|i z@e4kz_AX9D@liM%to+S*cvrCct$28Eu>QB=VOzofop^X}ke!Z)t^fUY6n6EqC-sqF z#fM$}hW+PvqWF1l|CA4RhT)E2ZYGLX{|Wwn6dn%a?FeHw3ipTMT#)+*ao+mNKTL-6 ze~=9K&L_hI|3fl7`cIPK)rDmL32d&4_m8^H4+fd$hG6ucnved5&1*K_WwLfWYV(%; z3qP5UI*%7^ZZ30w{gwaz&;I*?|Gr`A-BxcY-SM%TKixJ0eaeStZJ%zl-Wfzk!C^4C z@tx?ql#dVJu(gug$oKrq>iQL1W4TTFxHXsCjzAyq;aOXkxsCbXv2|fJeb%24Ab%~w z!+iGozunh0SE;uBn6Dq5jW_~bXGLyCc!}59nJ(x)7)&A`=}nvH^PA+ObPxOeNlQ={ z_)3W-J{;;c{;>UgsDA)z5d5JzwYs3GLjO~@6bwEPs2-*~(PG`m>8iJa*Uf!}y6RbH zeU6%mb-~LO`Tvbjw}n2ozh(N%*|MF5KZ*M7Pgs#&jjtRLx5*X)s{$>ez-JY`c_eDmF;B)OAh)zbf|Pxmc9E|h@jKWGtde&W_|$5d`B z_ymq|lFU)f^;?B9PnYhtZ;C}kA{e;-=uJ$RG35-iP2)5QMQc0#?Y>>Vb?Ngk<4B{4 zw&lYAqm$9jcD28kgD*yZj4G_}<34@27gWbg{}qe>H9NFZI(=XD>02cDZ>E1Ws*uq< zxMiRIj(1LO|1a8y=_gIFEGqZEFUl|%eURJOy6Pk^Q|iTANl2I|2a$JHtt{j=*KPnB}<>@{*~SrSo>Ex_1{-a zsFr@=SEBvThkK9q?`9*Wf87L|joc~GME&;{lJ@&-)K&@xU$ykVTBZEsKK-z_-@0ko z{@lMCH?B?xCZqk8GWLIHU7>HKciBE_I_t85#wYiu@v86m{+z8E7fh${fbSsjbfKj2 zEYN>zeCb>1`o1|j^ttor&z<+&=f+2ePK{m2y>=isdh-0qGecuT=P$(a%KhT-(AnIn zlOrRi&wlMka|35yJ8|aRQ0vM4C&q@(UO0FDM{|AQck9pXKhW8hd#(M@f$0CAZ_B-P z`s`TeC);ut-mHed-hTLS$Kj5S7#jf|hTnKAv2#~BvFM@EbEk%5$q($0 zCqDPqSb5j`4;<)p1%4x}kNpR&KF%93bYk!gYwA+F#?2c`M!oiW*d}J$!46A3{zkl3 zVA%OzEa!Qs^I*qo&aQGxTzK=ssWPxX5jXNRD|0!r<6yfpTYLT5Td|z?cXl4?crDRW zzV_CISmFbR_a8nOm0SID@N6uD!yT_3+TR)Pm4j#F>G!+pP|D%)vQ@6@M^=ShWbkY( zPBpDIJb8Y!oG@ON3;ScHymshNN9Uo};`M*wKrB91R%Lwp%-7;gaQHxIu43OfaO!NV z{SUlXHJg1C8uNgog6v+wX>!8+~CQvles9d{U1voMuf22(~s>x zn0w=!V?%N0WPHrNXn*Cb!aQU7ogF-V=Eq(O)$)I__`#uXoIW*l!WHw!PQ5jFlIaJ- zn6T*+`wdFo?K2mKPmT@+KiB^AIqe~q|Jb?Q8$-?;xVWF&A2Aj2t{b`d15tb|@3Hfz z2U+}heMR~E{=519gJ9qOAczWUV`y+_JSW(`llFv7pDWuCrTR>mDoPaN!}_(X6I;yp z``SrFV}GuGG%rM4BE7GHeO5sw=Z{=t_C)pJyijbH74bH!#G>Q*?ySB$%v-#E*fm+O z&pKCZT7qTKUMlzRrz~$oBE8BRD?luL)E~VEb;+ZtF1gznAJvEDB~At0_j(YxiBKt{ zzZGlcI_QnFuULN3{zKnl*A-qnpybitvi*?7x!Kak`oBw=?@QEvBLA>Ixrz;^RG&V! zf8V~P_;Q|yZQ*E>Y{{Uq`EmQB^ltvO_=qoMVPm8{&|DnPr(*x%^jzLSIZZsT{XsNq zmvd5B)V}>;7mC{l|JPA+f6)|Q&a1rlMEynOv-#h}D@}QSktd9mCu&bE1}u%O1uAwD zk;sm-%K^L_g}g3>|)IaH>PbuQayES*fXwy+NBw^;g8|2t@X+AU15t1v~3_x8Fq zi}=zmY##>$w|r3^(ckg-L&5o9RIIG2Qs2?iCS;X*pWQ3smN!bhZ~x&ST+&D=XBTh( z$X>VDv3Sob%V6Ju_P`cC6**Lv*Dr`HkC68{>^rdE!WZ6T1{LYGUJBDYzGQ;ga3HX? zQz?t^?^6D+SzI~)=iTze#k6VISoacQGZmwt#!sQt9s7QaH zLtHFvd6tdw!za%Ujtu3_op&o^*`seAgr|P#D|Ub9M^nuW9}CY?Hwm!sme}8I_IGUB z38vkFw%hs|ejxvw>FLaG?fUKd-)Z>0y}ujf?AXB&i2l2;1J2Qg9~da{duAH`?%uzb z{rfM^HvZm+f4Awm?&fC0XJhGHUk|tLy}iF@2e>ZZ8(wl>4WIRa<9pZZHp|x#m+yU# zy03=V-&_6pmX$LT{QK})5xM9JWKwnJ*j?x)QFR!3tgfx_dji%YVh2r1tsW{eD;633J<2|NqE1$LD@3 z(&P14?K(-cUv~zia;uBkuh@tW&wNMx%k66>bZ0+JMb-GRC;V(BS*B={G zZNGnWlXCs^CidGcOI7pvi%s(Xi%s%(HoMJk2MN#HN1^ih&ksS*dEg_ae^9V}_xrzJ zF}*RS%a^$P?Y7R+Hs}9x`NwtFzHW2*d?@-e{6Ft?=HqVYciTsiPhTvb4@XG&|5!}_ zzL-80)8jsPeiL2i|D2sV=j*b&>4&=0?e@OXmiI%E{CrH0^YwlxlF!HVI6oKD2dy!*Ij+aeb{}ke;WFP_eZS%X|J>XXT9$1f2C9lcFSN)cXNWa$z1f8<3B3VzOQ)w zkm(FMSTY?7RwI`aF2qc@4buMewro26*X3@Un9Wy!2)8 zvhyZ*=_}x6=PmHkSHX|E3)i;Y1uy*`c$I4ny!3VO-9G>O;H7VX@ALcv@X|B;LUT*& zfho_|dtLrSdINabnFTMs5xned0x!K8yzI<@m);6~(%at#UU~<3m8%oH^dsO^t~_|@ zUEo!&Zt&83z^h!n;HCG2U-9KV4qo~Ic-c7!UivV2**OAU`Y8AXZ_gNb=@-EddcFW& z`Xu-%&tCyAeG0tFH4R?+40x4m7QFO1@G93;@Y1h=pY--GfS0}qe%kb*A@yDtFTDZ0?976f-UvQ3RTu8C3B2?c@UpWNy!1BkYR7i)(mTM{dwV*;OFsf$ z<;sJX-UVLe>IN^p7re^V2VQzVc+LCA!Al^u%$`T%&f;~;qH!{FuTN5D%T1+Q|AftNlGUgf$7Uiu_>mFo(4 z=~Lk4XQshRp8+pBXTeLK11~$Tf|ouIUgt@!ftS7je%-HUu7j6e1h0Bn0xx|Tyz1d5 zc7Eg^n2iC=NfqF>)>VQeelvZz{}1D;H3wjNUrx}XU6ODhteCs zkNWj(7QFOE@G4gmc^uTqdLF#&>;f;n z8@%l70WZA|{G_k{e(=%3C177-F@LJd11226Iyvnr>Uiy9TD%S>h=?}oGT)_`Tep3EW zdcD`RKAiXEZ2&Jl3toP>5xn#!@bbgW;H9^MS3R_Wm);Iu>$(o`(mTP+&LiNZ=fNNM z{kaRg^j`3?vk$!Ve(-9?c-eUkyz~X|vhzB4>5JfH=MC`Ei{NGF5_st=;AQ76 z@X~LCmz{UOOJ4;qJMV&*eh<9tTmvutKKOzkUmM`1KLD?C1+7W{COzYI`8SQDdhpU4 zzz=wPvf!mRftR0e1}{AaUUs&Cm);6qcD8|+-VT1w+us3R`VsJ&$MfK&cY#;CcY~MS z177Xk3tswh@T!La@X`mt%g+ykmp%esc8-FVJ_i1}ug`Jt(kH;n&PnjnuYgxOPJx#` z4PJhJ2E6oH@G93FcwbK-ftTJ6UghclFTE4I#?cY*((~Z&cze3QOYZ?MKi>;p zdLMY%*$-a&aqzNp0KD`;@R{F;j+ckQOCJRGR+leSKa7FMScb?7RV9dJ(+ZaS6QiW$^OzH^EC^0k3l1 z0x$hGc$Mo8cb)+1D7^u^%9RB# zy&1g9l>;xm1-$%BD|qQ`;ALk!c0RJuXE%81J>X?$FL>#F;ALk&c{oj(m;`V4rLYZkopIq(`s zSHVl42j8#t9(d{3!OPDtf|q^+yzDH3m%apEb}oaLeiQtlw|@n^^xNchy@R~hSHY{@ z?}C?p54_ra4ZQRX@T!Lg;H3xdK}*}T-jkotcwPQYdOdjA*#KU87W}BM&qnalbKqrX z3wY_R;MI<8;H9^Nm!IzdFTE4I%5?<1^gMW#s|&pJUhpbcA9(5g;N@qIgO@%4UUm+G zmp%+$c8-9TJ`P@XUIZ__0A6-ZfR{cAUUpsqFMSHU?3@NKeGa_ryb50WJb2l84ZQRP z@Urtdc(wD%?&o6_QeiOXPwE|xHE$|vgx4}!l1Aa{FJ@C@+ftR0O1226Y zyzIOWUit=j+4%sx^x%i#554^vub2Fx*BiiVou36Sy%D_Hy$QVZX7FnF9C+z%;8hRp z;H7tfm!IzhFZ~F3*_j6~y$k%5ueWaS()+;6&VKOHkAqh`4uF?F2wr}E7`*fm@G6)4 z?FzT4KS&<~uX2rpmp%br<(dR9{R(*bnJMtnr@_n48Sv6)!OPA$@Y1h>*L{8S;H6&! zKj-_y0(j{+z^fjL;H58tS3N9)mwpqx>R|=E^jqLn54XWfzYAV=-UBav4ZQ4J2QU3T zc-grDUit&@vNQNhvOi0&_qx`HMfpAO(zD=Iu14_Eo4{+FHiMU*1ApDy(*j<48~9Pb z4sQo9y#u`L>;y0U2zc3<2QR$~yzJ}-FTEGM?Cb+Cy&t^nJPuy^0C?Fs2wwUy_*=gI zN5D%T1Fv~-9K7@b_;uf26X2y!f>*h&fR{c6UgeqwFMS5Q$~6mK`c?3g-u`*;(yxKn zI9&iQ{W^G!(?#&om%ytYmcdKE310L53V7+az{}3t;HBRIf7kb)Rq)c+z{}2c@Y3&t zS37Qim;L~}=Ka8akH>AA-=$}~E`O+U)q|Jb0AA(Ff|uS5UggSxm)-(i^Fk|l>22U; zXFGW59pGhWCwS>y;ALkwc3!g3XFqu9$HB|a0r1jCz{^he`%LlvAbkwH z>>LL#{UUhTSpYA60=(>;1TTFGeDIsm_3LTy(r3V{T(jV%&wz{}2M@X~LBZ}RoI0$%!U@S4}|fS0}sUhRGty!3nE)$VKH zrEh>&Jv;y}T`vHWf0Lij_!qy(A4;$Hx_H^y0A6|)e53%vAR@G4gyc5JfH z=MC`Em%!)!_*w=p{U&&oYX!XYTi`X0ZiAP82Yj2iXBE8kd*J2g*T73(2QNGCgO|Pm zUUohJFFo+Dx|Dw#@b+iCE`KP!0lfUY`<>JH`ayalc(r>Ic=;AQ6wcDS16I~T}%{W^Hr zxd>kR4e+wF2wwUUc-grOUivNa-p|6ye{VsUfc@MnwHSn@?9lZ1n@OqB* z0eI;_^xE&z^_2zc2! z3SRmc_!(cH%3U0WWGl2xTD0Dioef^6Kg@!cosHn7H-VR(&ETcyz{}1S@Y37Buln)T4qkc(c$K#ky!0dB zHIDM&rFVf}^7eFtm);9re!dU9^nUQN^Ei0v1K?%nAb9D+;5WSeBjBZvftQ~j2QU31 zc(r>0yz~k1YWGR-(x<_z9%jHxp9L>JKL=j=Rq(QN9=!Bx;Op&oq}{dvUiuC2va<+Y z`Vx4x<1%>ZH^Ix#uYi|+3%ts88@%*8;8m_w@Y2`7t6b~grQZiHKeGW|`UCK?Q$MyK z|0X@-egMHX`9s-R?{)Fg8^O!YCh*dm!OPAZc22U;XFGW5N5IR@Jb39{ z;ALkwc3!g3XFqu9$HD8qivjS`2f@4Fb$8n^cg^(U z=>_l_rxW0%Pl9jr_FMrkeHy&xks0vPXTi(PIq=f2f|s51;H6&!-!Fd%Uiu<<%_BF! zOD}>~zghw>eHpy^)lKlyZ-ZAo+yO6r6};w=yWpkY11~$*z)N2TKj`c8K6vSY|7j=r zH`$r-y8N5;dhlw;2Jq6e;5Cmlf|uR|Ugc^AFFgldBjBay z!OPAr@Y1`%%g!F~()+>7&g0;v4}h1QgW#nPgO{Bn;H8g(mz`tar5C`<&I$0+C&A0k zE8wM1ftQ`r;HA%imz}fVrC$YK@Z)P9y!30}Rjvi_(yxQpI9dcR{Ra49Z%+}t^kwk! z^EbguUjZ*WZ-JM78@%ki177+n_&IO?UGUP^z{}6CgO`3EyxM&Oyz~d))$W1+nLPPJ z=?z|||IC6{{WOA?pKk&$y&1gh%z>BQ0{*(M&sOl#JHX4%PVmx?fLA-_!AtJ~FF)T6 zUV0CBm8%!L^gi$^S3h{^gWy%JVeryNz{}5!f|ou9UUrUymwpkv>@0wneg(YjoB}U> z8ocbB0WW9@hJ z`|))Ly!2J@D%V}`((i%SI9dZQeI5L5Z_j=3(jS1ApAYPJV%;YHCOzYI`nP)UvcCbm z?976f-Uz<_x1!^(Ch*d8;N|CAz)NohuXb+(FTEYS+Pwq3^gMXgLl=1I-QeZtd%#QY z1ur}Mz)SB3-{kA_IC$y9;AQ6sc9@hl z&O6|x-vci@*T73(2QNGCgO|PmUUohJFI_*lDgP!rGyaD-DR!^&o6+NejU8ZwFqAN4e%;g5xn%9;8m^_@X~LAm!G)} zUiuyIvU3%@^t<3?=RNS!?}L|}8{nls053a(!{KtTwBD1R@w)tj?5qbby#c)J%z~HR z3|@BTz)NodFFRYoOK$@&JKMoa?*K15JHboOgYWg@s|&pJZtyBs4|wUl;5Cl=z)SB3 zf5h8!9K7^F@bdG+;H8g%mz|^FrH_G^o#Wu8Uj#qw?Jt0rJ_%la{t9^MQ{dI^)8M7g zfLFWEf|ouIUiEMdyz~X|^7GfhOJ4*pJ8yuOUIahp>vIXb^kwk6E_M^V^cC=P{(gX4 z;H9sES3TSXFZ~{P^^-O5($~Rj9=Q)*`UZH_+XL{@gCC9jr2L`kE#q~q52a_x`}@io z$$PyCy!==*c_l_cN5^HPl7M{_sw1bFMS%k{QL}f z>9gQ9PUpZ&zY1RCbRN9)Yv9Mdzghqt1gIBxU1TXzIdEc+@ zkoWp3c=`Fe;HBRKFFV)3OJ4^+;p_80cePfS2A0UVh*Rc7(Ff=NNeD1@N+S0=)D|@Urs?cXbKvEN zuY#9855DNn&tC&C{W|zLKYka%OTPhL^;QHgeF?nA=`wifH^EPOdse_pzYSjV$Q|&~ zSHa89yWpkY11~$*z)N2TzbJnQUit&@nn!{o$>U||8Lw-7sD4!sUU~y~^{Xs+>CNC( z4>|DCTfl1`X$3F64ZQ4Z2QR$?{IajlPVmz6;ALkQco|DngWy%JVeryN!K++j;H8g)mmj+bUU~t%?3@5EeGVrA75+W zrLTimx$c9Pz5!n2=mB`?!H*@^hc~@F8Lw-7D7^u^{CpO?^hWTqvkAQPX7I8z2VQy$ z_y^wpR`Al>!OPEgfS2A0UhRGay!1SHwR;zM={?|8553@}_kow6?*}jaIC$AP0ABha z`1;@W_e+D9J`P@XUIZ__0AB4l0bcqfc=`D&;H6K2SGlIaOP>L+a?OI5J`Y~yx&~hQ z0(kkE>)@p?f|s2)z)LTJmz_)CrLTaOowvYCzYSh?-T^Ot6};@c3tsv?@Un9ayz~w5 zvhx9W>A`=Q^l!2=<8}Er>Gj}cX9IZYS@5#65xn$f@Hs!ea^R)6fLFO%!AoxguW{54 zUU~=k25(O%cR}SR^ef=y=cm9+p9U{GXTVFJ1>fWAa}K=pYv5()0(j}y!K)n? z!ArjZUVgp^UiuPvm1`Ni^qb&St`+do?|@giR>4cZ3toQa9(d_%;AQ7Jc1226Myz1cwc+=?P>8s#n=Uwp9?}1l4u7Q`n4qkr#K6vRH;8m^%;H3xo zq(4--GG3QIl%55zay5dN-UMELrWw5S9C+E;0$zG6c-h$oUV0~Z*?9!K^gMXk*#%yD zH+b3E173PBc-h$pUitud**OSa`Y?FeIRak#D0ta9244C&c-eUoyz~NiT}PMzFMX1{ zpD(VE_xd#Wg1;Uy177+pc-7k+c z`Z9Rgc@wm8a$>cLBI053bU;H5W$pY-+F1YUXzc-h$sUV0mNwPQPY=^fxTk92~UegwSAl?N}q z3%ttJ4PJU5c$KRky!7MX<;MoVOCJO;JBPtb9|12rN5MceGR<){5p8)_rc4~4e-(*fR~-Yv7|qgp7FZ;;f%Mx z9=!A{c=`E8@Y0*WtKFNyOV5E%&;ee0CwTe!BjBay!OPAr@Y1`%U-k9b z173PRc-eUzyz~L^YR5tF(ucvz&yRqYJ_=st8UrtV9K6bP5xn$C@G938@Y1Ki%g;=M zmp%htcFuyAJ_lZQUIj0G0le(I4qo~qc-eUay!0Y?*|`K>`Z9Rgc@w9gSF z=jXsnzY1REng=ia8hDj!0lf4Z;8m_7cvC4%f!BGYcJR_Wz!&{|(FtC99{h?Qzg^&^cY{~G^?;Y&3tr>254`k#@JrsF*n= zftTJ6UUqhXm);3pb{+vQJr7=Xc7d1P3to2iftTJ7UUnV_FMR;K>>LCyeHgs#904zV z41DlAbzz4Z2QU31c$KRFUit)hjiX8M(yxHO@5lQTczO?OP>QTJFkM5J`Y}Y zUIQvIFV^vvh0`@?##(;qf~S373GOK$`(Ki>pidNX*HD+gYB3wV{Q6}>LFzeGI(p z90xCb0=(>;1TXyxc-c7xUivh6**ODV`Yd?aIR{?)Ja|2)a1Ffl1@J1@b@0*`!D}4d z0581=zRi#KCGgU3f|s9P0WbX)c-eUyy!1QZW#=k*>36|*di(Ezm%a{Oe*Qjq=^NnH z?hn9A55AD}hidnX*X0kTXThr;8o^6%0xv(`3|@K;yzFcNFTIt#ug^B}Uhf1iJCA^u zo(HdX>;f;n8@&8{4|wUl;8m_Z@Y4Iq`*Iy8@AYBuD%S{j>7(G~XU4!w9|tcxFM^j| zAn)y*An)}l@Un9ny!09HvU3)^^f~ad^D21h^W?pq*T{Q)5xnfY0bY6$yzE>8FMS!j z?7RtH`U-h(=PmMHzXN{2>#N|U-vzI7-2*Ru4ZOzDI(X^#$$NV?$a_8LPWm_b`Ha`) z-=x=rmz@pZrDws*&PMRko4}8IJDb5vZvihq-wIxO8+f&QJ9z0G%8hz)PP1 zuYNKKUiuaAnn$La>dYb_+eHOgxZ4SKjtKd~{^Wddl1D~6U>T?0S^dflG+Y)%` z%i!fFZ-SS;0$%lS3%vB(;8hQIz)N2RuX?x(Uiv-oy}o|dz)Rl%uX=a@UV8Awq(4+W zWV|kaD7_xM>Y)L=^elMQLnC-VZ+i8`1bW4qo~o_&M)Chrvr90k3)+1uuOJy!_ZW zc@M@QJ z@Y3&t*SOmNFZ}^{wM+2hkzbYnl-}g^L#ZHWj*#&GX0LZb&w2e7)Az;rRi=@-Gv&H{Mp6X0d%BzWmpz{}1l@Y1KjFM0cCz)QahUUtrdmwpYr>|6jZ{W^Hr zxd>kR4e-bPdk%}>rQZZEJ6FI~94xy$!tVYzHsB1AK?K zzZ1OlF7UFm8@%)$@UpWPy!1Zsva=t&^yA=1z5N5=rH_D@oulBTkAat+z{}2M z@X~LBmz^u%rQZVI>fhUZ8@%+p;AQ7M@Y2`7%g%N1((i+pog3h#KL9`G?GJjA{!My= z*X7@2XBNElM)0z;3B2@X@Uk-pUV007{XTUocC4%ftQ`_;H7tfmz|y9r5^#W_oL>)OYZ?MJA1)P?*lJ8`@u^;4qkQ+ zfR{c9UhnrC1}}XKyzCqYFa08T*;xQDeFD7foCGiZ3ix{6M+aW|EO^;D2VVMB@Un9r zy!30}W#DR&cc>5Q@OJ4#nJD0&rzX@J;u7H<*3%u;S4PN>k@WbBzRq)c+z{}2c z@Y3&tmz^8nr9S{KJA;*5q54`Tr?*}h^5WMUh1}}XCyzCqWFMSNW z>>LL#{UUhXUs?b!{R(*5IR#$&GBFa0if*?AAV^fmCZa~-_&``~qcegnMp2jF#n zKKQAmf0Lf^y8PRKKhIVVUV1LZzhqg~N7)7~F+Q%hdi^7Ip1;Mj@voHWX%m#Mt{r5& zF4q4XTp+#OM-2OK=_@WX+vdXmgA%{;LL?NSXB#461G?-9zCLvR?9j-C6K|Y8d-D7@ zgU_5F8aeryH&35Eed6T#^C!Ou;XgfB4Y;{ewe?UqAfD!Tqnjet3W9$%CDTI^H60Tvr^Zg4 zyl`RY{Md;LZ=E_dbm2nqnK$get+EI{YnF9=wdX5+CnBu0lh0bV;cg$v{IL8=tN%cVI z!TlW_hYogjSgO|#AF#g?z@gVW4j<}3UKM#Hk|bhPO=rLHg`qL4+@X_aeAAwO54xa29`o`%~ zLkT_VRYSuk-aLQu%uvwN^D|%SikkFG=LX*z8A`UY>lr6TBD+qUJA2CZ_RwAI#EHSN zbLXR;rrz_`*^zT62Ol@*utSC!jh#C&7?w4vBdedN=uyxrARdfrD_)16KX<+iM4jkM zCr3Yb_SCt-p}x~2LxVxfVDOm>W9P?CzG2_rJQIGOKR0-C>}0fyo~Vv`tW;g0_z2ZB z_Kh4UM-dT!Y`iwrC%BuDuurqg)i8UdgH`v)|%fO9Xb_e;4^yTDXWI``n2GC*C+WZvNzq%lO1=2To|nSk+gW za6)d_d(TJznzP#(qB|Ord5o`QLgCftlDbU~iOszx+iANJY7pdoxXsT)iq7f}|Lx`% zx2L=hm%i2_t5*K6NO9AJn_@KYHF@w=OYHXlvJdNP)bE%3Q&`^%wi}n;PhUzN6ny%> z^y?EvD4oLpll|drVjYhDKaoOS_AC8|?2nhWNKc+&mw&bTTlqht_ESB%`rBvS0VnmhH++muuA&>ADv*CE&u=k literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0f8ddce1d7ec250fbf885785c6685dcf.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k0f8ddce1d7ec250fbf885785c6685dcf.o new file mode 100644 index 0000000000000000000000000000000000000000..c39b85b4b05f0906dcdbb86c226e58a15f615d37 GIT binary patch literal 74024 zcmeIb3w%`9bvL@_q0wk0B=Hgg1R6kqBtQu9wlN+FAqmVw2F$BT^ajaDNOvUgU@P@_ zNL;^a3(_0MO`7s0X^V()NSZG}X%&AUx0%iY%O=c^?vt_53vGw{Mh{R(AP^gj6LLl@61dE{a*`o8*uCkI>5=hdw*4Q@hTXO>Wz zn|rT3`Ow9OS|7bw@WZo%5B>2|gAb-ZH&~eQ(%_Yc-xw_XA@y}rZvpUy__S<$67+v{ zu%Pv&!8@OPX|VTKzZ`t%Q7W^x_sYYMTs(s^4;FoGu(0AU2UDJUaWGK%;$RDCzIy02 zlv&eDyxRQKA76Yh|H;9^N6=T%i-VizgREzIW4Oe|)j`kuw}e`6;g4p-oRcxTD}#Uk#_+mwP_N7u}Ka_qh?8*N}&;^kdFk z?&yws$kKVe-y4gxg9)J__FctD9WTMu^Rz44zaJdLK^~`0r|2C?3im{b{ zkG#3Oj%e#?$jw{ZH}t-}rwno!8h-Dd@1NG?Ii7sJ<#81%86|mCvKWFaejBp-ku3gx z$zpU2Sv)n4ES_@6BFU(d$9{)AD)}8tCa=uvYrC>x*^l45$JP68ySw+j4eZ2~Jq6x( z4tjcrYP^u+B-l}3@2x+-`{b`7+ozu??em9%vzgbMF@wc+^!{^b`21f-`@VZ9G+f|i z8E<)fXa2+Eh3>F3chB|>f9vMZaP17=do5YM_nM#e^qu}$Xt?DYuHoJr!0bz;zL>S8-j4>ub2q!}VWq z&BgU1t}}6c3D;@3zJTjwT%W}?1K022nu_b+;hK!=cX0LK`Yl{XK*v+K{ub9Wxc&y$ zFXH-ZT>li;H*tLw*I(iq!}S-q9>sMh`0-hcV-c=Las2_twHwz4;O7EggX?Tu+i_jx z8Xh^$etZw?W39XBM5BAziKffNCk}kC_{7046`v?W{>V($_iymyBa>O*h==w4HkI{} zZAHBkb+3KzmfFkTyQR*ZdA9Bi++U^pjI(ub<9?9t)6dqukNbD%e!^L*=RXCTAB^sp z9*XX0c`E6>;MM`}^SRMU$G(*`T>I15Ev)La_cVNe@Y1RI?Bh@Wy!IPO?-7l)?nNgW zzqjN>)0dWH{K*KkPhP1PFe~0_;(0%IJ zx@U3!J-QE^t$PXgFVKAor*j}RI_d8HNyCv(Bn`K0OL~v!Vi!)$zk4_6+@Wa72Tg^b zsSq?3f~G=?XG9C}ifAGJLD#u2y(T4@3;-Te*vGC zhrI7~pY#qhSN|<7>{PhKNN~1+ila;>MO(dEc!vuK;ztlt=xb%}uxlQldNZ=bZU`s*CmI zrZC}$@AL7P@QTO%;{w>kTV0S-clFs#j132q*x5Sw_Ornx7p~iI#hBb%arK-v%WpZm z$*aoWCd+S@I!VVh2H`E6p3FD2Y&$gY>6)#jbG!+Jx}dz&SAq=o3l%Y zHwRJ2+1Fg;=f%n2jXv^uK5xrvCye~XvhP)+`#y9(@Wm3pJ`VmQ@GB&~A`bp9fnOu> z+v4EA1^haR-yH{k6!;MEk_Q&u&mzBE*84=9djAA`mBb&2ga0h>TjI3)T3KJfl{v)c z@^GAb?*o6I#NQnUpS1z?CyM&_$H5l?f1kvkjDz1s^(Tq?Uy6f&it0=JH{#%*rTUXa z{h!9cA1xgY+y}fge)cr1Ej{&rtcS(dD0_@^SLNMdhcB zF8>`WA1D9ssC@S5^53KKaq^#~^3z9`e}T%!$$yE;&lp|)B9)Jm|1VU2=IHXTQTaIe zSE+o?=<;t+`8fH5RDRaz@^4f5IQj2T`Prk(zrUd`a39Y%>_PzYLF>I*cixG7SMGOR zSnrW9dK!N1t?d0ikUP0_PfEe``_p-j%Zf3*3MhyB^YH8npASQkLcxd?N?T8aTlhpeg#HvHT z4fPEpe*L%9K*2kRXWxBuTkm^cF6|pwm{M@%N1@>X#BcBaYiRi2UI`6f{y}NqD~P{d zeWkST=ZMRG{u_pS7xLKfYrih-d+pb*zRN#w_5A`d+_jnLe}oPHdqZj8;KNj(4PQm9 z_shv;eeeG(8~zpIz}M%M^}Y9HSKk|m8QAnA!y1C$$&ur{{=Q##CL!du2 zJbWB&-W%$B=iuhvq2IE;ciT6CFIBw*f92^NSYFn5#Z%Vzo2S|EP|a<&xD5I z5Bsjo1U!um4_{uZQmo`5D$F@Z~A+;WW4JZ-2U%^01aUGw$A!OUO*K z4_DNGfBWAXYk86ny*+llc2MkN58R4rI>-*~8ilK6I$Y1Nrkp z{*rn}Jn297qpiT=(%!TcWxeU^Ht=@;@P*KDCffd~C)9WD`0OE9X=#B!C4}|UtfBlA z;^MdY{QTfBKo3=Qp8zGv)u)O})*i^AGr(-Z10* z1C)1s`p^R$o__u`hqH%Hb2$6_^BkTwG=OoPURYG%&nxc@ATOWt7p>~`=WXou1Gk9a zXQrO_=T&gLpVyzlX*%<@6}_jAPv*3q-Z1%mO0crG5akO{rgy`n^C>TehR+(8h4o;v=Cp)VzO7yM<$Cwsr*`Pk53Y^XW^SI>Ox{QEQAeZTx_Ztv@VJ-heK zznRth@6Y7)zWvWxy}yB84?$0dce?uC`71X3+YPS1cmEuE3cVj#2)mJr>vGcH(&2%} zNq@`w2Ic{VJsNlna3$c$fU(9M_#?o30DAyq4Loo!;0D0IW|$*fe~ByB$^)IqzYFlc z0=^aSAqH8lK0ojS206v`8C;K74-M2Hf5YwP2mYGi+lL1B5WM63z?TW$F*LB9;GO4T zR{-xE8mJ_A*ZG0t1n(LexD~MHj-dhA_tVeZaUT8jo%ZY=8dwhaIIc?xzvukGPULUE z^){-vXJ}v^@;B@~k9vUj4h_sCc;ESfMFj5~8kkJ*{`1HOynkpQmD>4(p@C#vpZSCH z10JysZoVk&0(2ewiW{N(M|k}%~~;p0c)L%4?UNycX)%cN#Tr9(~O!%eN7 ztQwd+!nzTOxzliUyP0Q#$L038m^T%nx=fyQhJ6@zuQ*36RR}lxJ3*r?R{V2ubMB|& z>ORDLA-0Akfub2yvZ%AUvn$+HSC2b&J=P{*ey!KVTWbEt-6$}G?Qy-{H`w|qc}CRjQZ-4?5Qog$8qk;vGWkHV0T!jS)NVc}@e&#S3H z42n>`uC=DP22jJ1#=4r$wq?bUny!X}O^1tD)*L)i7j6vKwY1iB!PnI^w71sOHz9hh zX>5Z3Yj0?3tU;(<6K?7{f+ay?P5rSNp3n5gmt9}uM?eUgtm{HM=;%n>;ZA2~AB9f; z@}EP#n@wPupA@KKSt8FP@u3#A8<@4>#0GmRkeeH$R?o)X5> zP1lUH`=Zf&<{!B)n(ybgqRe7vBNA3T^R7dNsOx390!i&rHYK{JMEyi#(R^n&{usK= zB1|T@F49TLV{>`k9GTZFuc}`)SCmcdf6)ECEcrr9EQv&$XS9#D3ZDa%)zKA!zHfk^4brYtOKTk|cd)Cyy|r_35Z1E^dqlxy%ZpYl z4(_H)(aN=~>_|&%W3cmBXIE2OuZ9RP20ECY}~tR zW8?6d(dWTy;x0sEW*&u2XboW~w8 z;CU=+z;ju|fakCd0}irg1I}f&20WYXHQ-sS+JJLdl>yIWAp@SliUF5e{GHB%2As_@ z4R{)v6s>#~yXMy6sqB&gPhl4fcrtq)usz?C*m(m#k)1Z+Om@mBpTT+zd^(F7@C4Rj zz-g@3fKyqu0S8zWV4I#476NR`o1d*Q;AB>8l=rb9V4HtQEYpBtiw)SrqAqsL(o`8vdD$nu+%oKj1fc+9awMf9367E?l;A{y;mkT&3 z;f~b;&X;iQ%>ph~_*(_6`nyynVAWrBg@9Fm{dlOB#-sWx-YQ_#-v!7v;jR9530U=a zVUK`Sf5m?wVAWq{jesMd&*$>7%z9ZKZTei3SPx{I+Fv8-zaZg|gfou{e3gW&C0s4x z9trPN^(9=Z>fa^mH>>&*R{ixzSoL>F!X3Q)6jqFbIigREFDhX*z6%mo{0trw<+Ejf z9TE;oxL?Bg63)C^lvn($m9Xe9$j(Vv^~dfJDIk9to@Q zU6Qcs&$9-$I46%?gFU)59kyr&&MzV3J)dCg96lHDxdc1afqC<0B8Gr{qBx5_-r10U zB+v9Ak2m0F>9~)2aKB3CAHp$Z8$JiAH0VO2>-+Nhp1i&*uPa0`1{1*J9u)(ApDH1# zd=-n9i)<@jg4QXjDzEZSB`B}(AM`bLk86be46RZicsYYjKQ>=Fh+fJc-6U(*I}%dwwbFiOee}IUl(+l0ohFMCO)9 z40uv`RD-8SJR{WO{46l~axQx!GGT;Ro}T6Mv*)+7v@r9~1a(vodeENtiAd@Mil;|> z7y;T(i!cxIw4)j^9$CeyrQe{Il@gR_aJ+-#9W-?juEW zVf^($0n-F=m7L(vD`aF{-Y9PuMz*eHvcYWum@7mUk>M)pS|XR z{(DEFZKy!~!(JKliS~LTY2~cnwpWitn3v|KBX}*9UEB_S`~7^rh~Ka$&sp@-d_W$K zDeen-aJFB0B+aZZ_mMb7sQ(!8Mp(q(Z9jZf5~85A5`@P zzn%8*iAY6~ZhvCkDVRu}`1oSon;h^o1MZ0gD$ufJ?|LHs3KB?V|9T=-6;T7us-VpT zg^zVlb+k97B5L3#JMfb#NZ9Q96CL=>il~9#>cDTQAO*AQ-{!z?u83;*^G6sNY!W)~ zVg}is9ek4^KwYQ@{yfuQ2|TkqmD%SHc{hnI1*^Bk!X8dm=Y=$l~&JGBPQh zIu!WF;9n;@@Ke&G7JfVMQyus$7)4bd_Mv;41D~B9weT6hPj}#Fq%)FNK_Bol9r&Dd zF3%QzmW}^@2In7-fH*&C{ImIkZFYOPc6rX9U^x)(&=2^-I)z3=j`l8iGxNXVb$w+oO*B4nI z^PkT3gTsO!Ft=8FG@jWOEaWq2!SUtO9L;Rk_rgL&6 zzq)5@^pL;L%^+qd{(ckkS{_11k0{?0nNv>TtiogcQo-$!MNbg%f?Yq>J~3&(C+zgXGTwn;ys~`Y%h<`p0~E8BLHiUkfu}t~7drg8(sQ#h9vXjk2w2{K62`x-oV}=z=N7h>YFqW!JH~Ts2Eq-C9(T7-n1>9kU&;PN zu=!WU>%-pULOv%M@=o)g69S(1@1pS+@{KWT{Dl?V-^uaCx)&Mr$0F0T_DLVJobCI9 zR{I_O=5+JSo8VfJzTW7(b?+Avo2IVJW{_+rSA6e{gLS-#o370Z&q1O#9sv*^t5aANYLyxjWV!;^%6pf%5I(f2nD&yUPrD zc|0=17e!$%zb|AE|KKZ3ev}i*BoD8-h<`Ie3cd{eAbx}-KfaOf>*| z!SNSTUr7Ji@jL9V!@rQff>K?gX1ruaN>6u zzZnlW{e4elbq1D;+WedTKkoP`pwZ^PSdCYXe}#h8_?z|d`)_dkH_32W&3_W237L4j z?$G}Tb*}Kxk5Tr*j3;8Uu z{H10ug};QmG3BLPjbHM^vY(0{hQIv6IQ&3#TE-bB?W6Rbl*R2OS$EwZPGXxyag3kr zr_(=XWbpl8%N|TH)=SR$k9DW14S$?}@cB2^17sgEi6rt5c&P&5tTY8fU$$ENli0<; z{wkf1kbHMkU*r4P*q_7tMa9RoKEwL&B^mG1`ANza881-%3|55wRx4rr#{MGUB|0Aa z(6o2pX=x!>sve!qRRI~76eRs=R4{zbo7t)+du04FPf~cZ)4pBj{1d; z{;}T0g2?W_C}6ISmN@D!wfh(AZEe4m<{RtXf707qKA9YZYCqP!%+X$ngC4AZGxhqb z0{Z%z{G&eq(APA3zmMj3wM`$Ff2?0$*4w*zvc6u9b+0wri*?`Rpa<*Mpk9A{Kwlq| z{jm4fT_e|E;!?Yfi9+c}3wHE|`S(NmN z_8Xu-#j5^xz%v4{JR1Mac$@mqv+C1&9s55&QS~1KoIBZGpAmm&kLo|)s!#MoA3~}= z(LXmp2F9L$Gv1~C7hCm-e(1wbRehp=;bbz$cKrpT`Y*QX6aCPKYE_@;FAPMD{>^xk z`d?wyC;FiemsEYCf9YhBH@p9$QT?y6>dXC}T2-IuFA13EGiJO;{jam?6aA3?%c?%n zzk0G6|F0U=zb$`6Kjghb)hGII3Yh0VW_(5c+ww^CL*9E-eWHJTKtKOM`7~QziC)O- ztExQFJ8iNauT%R$YkWi>WLcoOr9U;2-g}0xm#2etIG@)!t9T_>KLU zS2NhBaQ<8V0OA2UpPpL;d=kUsfRy|U=h=L}hvEg=|7xWDo_GDXb2ROLEj8QY?Q8V` zzYyl!LJiF7gK; z4+?%FvH}~7c=mh>_CwMBMv70#;k#kw0pt7%?qOtP$`C&<1UwlV4id)xNacf1qA(?t z$N4wyPf~k+dOisCpNM!lTrS(k^E4dJl=xUA$mn4Zo_K$=llt$q#~X{xVWAKU+4!^- zI{swD>mAX{V}A5_@?<1A`I3e|iTI80Ps{VMSY!h8i`M!2;qh?L8$@S{KR8c=yFxr= zmB+!bRz4Qt(uo>Kpg$IwkDCX`#c&wsb2x6Z>*G8R?6G0mAJ)egi%fUG3wTqiy;x)+ z&Zi&XDj?=(#%$uBO&^{f*Z38Sh@y7=0v2QsR4MpmBxkl)tN(b!8*~Gy#vhB!a2IG;v1tLyJX90%I4q3>9)RzCPmv`z`*e}m{rcC4T9IH9&4gZ`~F^($+Y zuAfTZlP!B7>_d`-NqDeY8-%T^@!+83f z#UGed`s}nG$lk1qK{U z&zCUvlem2-wDHjQ842|r>r1Ep{ipOjQ`h%({;iyT^7q%{k5?P#;XJ;M+WL(B4IY0g zej)z#m7{$HKM`5Y^?S9Y?`t@`#?tpJ$g{2ACnM8vpk~9@rSHc61N8k85Zu%K@oa}aJM}Su(=1(|occamA5DK8bm+5FAJb-x zt?#4t(e&4O4t<_)=riJ#8Dr}^kCwT_ig;y-!+(#h&nF`@W|;bJ#v8Q0sxtMR0v;sG zzfW`M`|QAiEM4D&Y54}6mtH7gq3_u<7Tb8}d(MZh?{xmf&$r3H@A7Zr+p5yyRqK~ zeb@F6q3>%Q`gqd^t?zn!h)2iK_aNs!|HIb{PX9g!C|%$49r|40(3b$VAI8#mgzqZx z_&(PWe=m0E^Abnoevf$JBRo zJ%IhN@@iAxVV@{j`aa#E@A-jXwyy7s(_oS{e_xcoK*B=bXXNDBc<6iKhpz8IUEf>% z8~J#ySKnvw{TrpfIB)Q6L46y(*?1m|^t0MnZ_j|d>iRqr4|qtJuebX-HEO-h&x?%x zap*6z<1HS)lYC46t=8i=TlU1y?TNi!H{)}sKc{%z)K|aXv=0jm{iXG{S>74%oAEpR zH^fEPM{_+8i;VVd^Pu$+|wDj06|xa4cfRE6)9qSY$l^o|mla^U2791v*Uo7v}n4p#jr=AU)op z=PwR-)B1XJ&;y7L)Y*09GGNH-)9GlBrNnjZ^2?44}CBG(DglE*Y{8POZj-N zSKqO}tvz3f_=oTREB^FEX8Eeoo`RkF{zPOQ*S~d^KhJ?b()EAVEDOf@dgX)P<)^D!w#`%=jIRCX$t@B>mzk_|i zj;^8y{`(eFUw!L!c(neT@c{XMr@r_@oPlz`&hOZ74;uD?zMr9&ckZ8!_W#cH`(mHj zzcbzr1Wmlt-{(nUapC$l-x1Fj`Aq$F#@lJ&rfq-7p6U7$FFrBj`C^}`&&!PZuy@!m z*V}i-6EIgQarz4!>vx~e)bC_R`@x`zH}@w<->8^yioGeEbuUIdh;u;(2*IuSNU!b3}Rj{7#*>@%{TJ z{cAb?`s{(BkN7^Xv44)|v$*~h+xmy^mut^MXy;?rc^Ew|BS(b!9qpfe++O%Bf9yE_ zikR|#GE!pb-^oaxaehYpAbX3%&+8Dc`*eE%eKhnP^`Stze9kxY`SGyVOa8#NH>2Zk zQ~yW%D>I(=m6-iG{j(|G@$}F1&uJxN>+fj&GW9pTMAy&9BSl`fHr~#lKi&xuL&^w{=t7a&Z~SM@OhQf|K%C_H`@O}yj6m} zkM@7dfz|i3#`S*={WR@?)8CFhPafC*<^68@Gp_&3)Aeh#J{2l%N&oFUe;&{O<>l%6 z=3MX6^FB9tUbQ$-VxK>eemTyoiXG=wzC821D)>RiyQB2YIIqh2fb}gyd%lhIZG!f^ z9LaOK){EwP?qsCM;jhei zG!_xj81I6&pTbOkwZO4{G}n74BgKY4i$%l?jelRxjs*Khs2Hs4?6>o=^|$o^aBCt^S3X6c{! z`OB-e{`Ew#-)cQCP3t%DyraEd^BMZ0)_3;#i+Enr-fuPcQ=RLZ(fhCI#(pd8fuT=$ z{?IrN!SkiY{;M-S#q&MMdc24ApW}RDfpMNd@7pr{?IQ3~k6--`f9`z#(Yd~Z`&EhS zbAiKu2GVus+?4@Z1D%gYC6>!WXheVzgT znC;ijHvrEFsPhqd-kFB?Jgfbh|1JK~IR6;%&%Z%^WQIO!&kHMkw7xGR_0RhL4_^;D z&L6TI=MUz2!|3xzQ~#X$G}<3J^^aawArqHJdVhxQpT^cd_(MnhX+EzHe`@F>`9t%3 zAgFqm`xkQ@{w&84f11w^jMhJ=zM1x9^z%e{j`-7*|IzyAjIYPjx3S~T(fVh`*YrG* z^Zdb~Z@CVA%l6H%&l8|uIsTvl=UL|yLf^8|%=3p2|9K+gd?9$l`gV;rnCbfnJU%@q zC!FZdbRZ=N9kD*m+npLxFEJP(I|G0qP#pGG`O=Y!*XAIE3< zM|xh__E*p^Os@^o^X7Uy`FJ=TFVeO5$I+h=&pPARSY-6`d#1iS_0`np8`oc_e}p?$ ziQAj`4*zHRJG_4g@vJ?*(f)3ic)bdBVs1bNG0!KfWB-pIP63l>Y3Tt#61Y z9p?#?9Onriwm-w;zbbM0!sEX;$f)-dXE3xs*J?k`_iO2WCN!Rodc4mh zPJP-B$pxOjKmBFL_hb3_y!QSO@%>sKi{OETOq@^i^Z9G``>^qS8|U|VjPL8B{0m{h zUGYAtT$Ts=g492~ANz$2HYvhbEy4YO(f{;_o4-n%(fjW4{a?Zpz4M2FhkU<*_eqNu z7xVYkJNr-TW^Ot`!24YI`-e?@E?a56PlB%xz104#s{ena=Lfn0Q4;=rx_bZr1IWVr z@!j1ESvjHQ`-w50`5}VcL_Z%NUOmIuLV`grm4||h@)+MrJm1*^b}|}2@L7?C?p@eW zpj9P}KP$3;!+cFZ^~3W8j=uDLIm7P`7dYTWHjMX?uZ-m3+gg-(``(q2`~^V+E?gj< zcoTScc!^yf?XQe1UJ%mC|LrH)0^;{b8eUU`{YQ6rsiS<+f@-b2JG{)mqkbNb7i9gF zk>UllM)~D>dGM!XL5GcJsGpC3Ui1(D1{kMz2E3TQ!OiIt_#(2%8jPpi47enyV2r2o zSTRJKl9+FLKM?{NO6vI(K7CyNG5@39M}EZA2RGhls>?_09`^`qg)9>jP#^EdP68z4 z6Y>pt<-hNsvtoqRqQwB}d?g*< zm&bkr2^cYLfZ$@llRO;Gr|;9}F?LGgn+eVbEb-_M_QOrm*-`&CzJH$UC770*9rZ)> zeRK|U`s?v~5)y{^0{@N$6aPBu-zAv&!%LFHelXM)?+^Bei+n}EBLOD-VlOCGu-6}6 zn!MD8fnV#z2#}&w-sQ*p?8SKwhk;L`2UHZE%?U3Vk^OOhH78s&BKwo@;t|;&hyCG_ zWIExO?W4YT3CTN8pl?n%sfg@}4KH3So=-v!$1fd`^m2H9qH4xHKHnVX^-Got`bB%|=L-5b9`l_QS?Q(U+u_e>aClV`!K+Z5L z-$LGLehN6f_Wb1Ig}XALeXc+KlHbsm^*ahjsDEpI>z9vE|Em7|cs!8iLiCr+`9nc)^LE zZqYjDpP}!V&#cHIuCIKKIlS0M{c{_{$Cp1M{f(6Gr5VDW2!0X&7qN2|ebCp5w!SVU zeVwH1E5^s|g&1Ehw$o4qD*Y8L%^%eG^kVUQW!hYpx@{>v9C;3wHi+IgX^U3Exw7*dB1BC!* zMG8^~&)Wd}sbg>imn1&`Q+p)8x;@bS3E^pnRQglzr;fosT$1y7I-T3&1vv0j^RvE0 zj9_-7!h@{FBelo`k`(fx84TT_1urZ&`Xg7U#_JEK;|D8vlJ;l*6k&fz{+2?X(h$nY1b)H)PbGh& z<{S2mej?a{=Y%JWnEIcVX8M=(5!s*Aw<*&|o{@O}@c$E}{ge1<(}aE!Jv<&r7yY9K z;PoX_Iem)XbHYX(uj zFU>#t_Ne|yUTFVE%0K2ib2`nxDi43m^^Io{y|7z_V z5~saL9g+R<_Ol|{Q)oVDjUnV=N-5stZTT?$Km5BRJ}~`1{Bf@Ir&2z=UR|G@@qr;9u#f9^IOFqV z!#|V0lRPSUB7K~~f?h@c`Z6uvhy7e%?}#73uc@rsDi3+$@qv&hnopK1?2()w@W+d} z(&vZPvvPj8{^ARYiodAu4_f*U`67A9X6Iym5AZ2-H2nu0zzZFD;{5W5Q|IaWMe&i3 zU9#%aeA14-oFCpl?a%1zzaWRj`fpBHEvJ#m^)m^(XOu8L4!h^*_5<@bSJz+2+dNx8 zbKwt6`{J}8bHb*6&k39Q3;CljJc@+==Y-uUv|i(h!^!l82@Uq9>i%H;Tx?w-@%blt zDbeJG=DW;2Lhb4DfxX2fF@GM&myjop=klS+3;HL0N@5{V9={hy`r&3BHjMd9W~Xcz z^W$X~1dQL06Tg2}1`4oBM|+qru^&k9H>UkSIr-o}KfiMR9^a?*_wCgenZM8XDt_-u z|2;mmZ|tvKAbaH*;V)3-@9RtF`*-sFdDtIYiv2yY-@*6yu-|3;o*v*yVGsYkO1>ZX z0^$q&z9sRqV+ZI*e{TE~IPE`R|Bcc9;dI~$ z|9%>8k;=n9Tl=5%`+V>ZZv5UFmuJGiyq!IgqJEDs8ejv-9QJ@eJ_h)5GWNq%^diRZ zj<{Lz5fDK927k+s4axrdx%^a8Jj#<@o_-rUQF%jNz)$mi+>e>>dsFXsT!Z+<`hAnP zwEgE1Pvx=iv(Jg&XB6_RfB#I?56SxAU*!?K{=FFQ(D?P~0cq+#ZvA}rh4K2Y9;^P| z@$1tAX$kv(VEp>EW7R)Detmi%AYuOl0`Xc0E z3V**iji-_1LEpb!3VFE3PA8%V@*4O3gG}-#&)FV+kM!7EH{PCco~qMF?LE()yixix z$D;2XJ9ne)WslX~=h!!Iw7uY1?M2!78*MLttoEAOcW<=4;<4JRX8noUg&S>e zOnF(s{_#fJ8&h7q?B#LVlj~i3e#f*gFWz{2W7?M<_7jc1j~%0hHl<=l&pR#a;y*Vo zeuNnEU487FJIc#H!~L(bJjH7~QT)Kmf5j+YdF)+Aza@{~SN5=eM?Cre0W669=jN zQkC-ZXDP7y*~oi8ePiTb-+XiAH&4I*{>abXe1GKLzxnOcufH)e;(L1J)_)xam2bT9 zx4-%6J#X?q?@`3&O7#ZBPERQh05v$pg)J{vsxR;kj|@v)iw*cb5AB$_QqzH_ov-7r zn-D;jrq2$z(*o)5@CTJVNv@%oOZ|{i6s0KH_~Qm%Z20MTkK3rvHv}2`vCB1c6?)x? zwvM}^?*0+#Ref%Exo3XD_YNy1q9z6y`qvYdcw^yd3u7!ih16r=L*w9S$u(B}s&VkN z#2Tx9^*DH1QjJxA?>Kl`LXB0wb{sq{nZ~N$JPy8N9DHOPe3awARPqiBeT8;(|DLv4 z11zLAUkP8@$O5j)R;d?4cuE_*0s6SNeX3-LHP({u{QXG(2p%B_u&5|QJF0Qt*R?4PeTGD1`0J`|UKXDJPYCs*bEEVXY- zV(*0c$Q&B$O81w)!^9Y3?wCu5-A;ItiH{|ju-gI0ybin*o^0Y{Xum8!&6nt-(-!mC zW%YcwqYt8i+B3?4PG=sC#hK^!jn0e7yisM79b<98Zoi3-N2ilU9UjGN&|%<-AM?wm zXx25;u%1#z+%pqPKO^m!H)Fv&c#`aPDnBtak;Nv)TyE=Z+r)c99)^j?)bVEfF48V# zg1Pn7+xJ58XoB?mIzB0s#4zwN+7D3s>ZG4rSG~S3!aL$)RKxOCF<- zDD}~=`rsB=@53MRvlu2<>%(Y2C6vNqDPyz`^Pur4NET@3Rg=a*2#zd}xR2CODvPC# z(ML3i#vsqM_z%?^G^T~pSS&4Z`x8PFSZu;r?UNPlS5}l;d7Ve;p>!5YPuzY+2wUzM zj`lCZN|*ixi6P|2;L#eAQ!C0i?i*juqXvBm(Xj$SW)dw)mzNXL!&Q~I5VKB?c=OJD8pNMF8_K4l*g(I@T0 z_0rdT9rSSP}Z=bgg)+u ztSRrX%xZ>3kK0GvY7kcVGi75X9*Z%oCJ7%+=}&-nxs$0rggI_~7)ZiL1N}PQ)vy?Q z0iWg;zTe+Jf*pnwH#lPJvU?0Jl1CYji%#yvg-veIXp>`Z$qjcrZp4zIPJ$b8@!%HW zi5qe8lH0@$4X<+}>bp^HWPWJez+#oS!RJ0Hu555_@VSqRPvP9)a~~J)Yy0%SJ%R2FwyvMp3J2N~(~Jg47?RL>1#TV5dTHKJ=K|*v#SU@tF~OyT-WuQFvZDC-{^iCsBB8fU zG-_x5C@x^GdCcMK_r+`~H7+CzE+Afwk5A#U!hJxz`dF(dE@<_2jkxi6^mp{IbpB%P z*(LC>bOvIT8Nbn|pEbQqvNyo9&gN*1pcY{|X*9hO=n0de{n!>p`(5 z;p<^+g~i9q^)juA;&RC=WQeS?jn}ylN{Sc7m+}>yy+;s*)e+I#C3>RbFKElk1(&Dq z??w>Mj@nmg*x_?E9Kgu0hwhEf8n1xW zmGE>Q=R?m3af*8-#a^OVNhRgE=cfCHSQ+l_V<8;Q=?P!dG@g4}i;C_{_Y|@AbopIF zwA4^5QBhO0w*!D@g0nqp-n}NhB*=nd13L|8%`sQhdcQ%fz-OCy9LdU5Eby5oo>_Pl zqxSPnd{mui<25rJ?`S`2y^oRLiJ^Z7eRluV0`F+Q-=goF zz&q$y8*L1)Es0CNRX-ZH{iyX7*$$k>jpk3ts((u09sFTdeH=K%86UIiXU4_%TlFu* zZ9in;nTNOU;7=3=LL~gKLLB;S`f3H0TEHlOvfE#ZAoHrSsXqE+l;jPqS1WjAGt4@@8`Fo%nDiFc?o`3 zJS&E`mdHSU8vj#-QOkIeuo@yNW$XcUFzq4j(YV@^jkiS?v;O`d*w^LleGH-UU*j2 zTkU1{({+E4J@jXDzw-4zo6^RW0bQUz=Y?f)jcsGy%XR&K;z5Y~o$$+Qe-CH_-GJ77d zJ>Qeqc>_O@oi^Z1hM-X^k2jhd_;eOE;0g3&R@!*dSgir4vT6ekuqwbdJ@|DBz_z^k z*%|{*rq|18<$Wv&*ydjn-rc6dh$IZyL#w)LF1M0@3PE(3Vip~SIth5Ym+Pdf6cYeb z;lg9Zo|z-ey~WKuH6E9DSsJ@`7T-F4im_UJyahRyWh86jOkVtk3&J^e4#D3w9Msp6 zgL#s%Q~30J4McB_GVj_BFSsdQ&;q~1T;;@l4wD}z_?(2vuM&Jg!XXJ?lQ8)m!b6&P z{VEAV`Z!E}mGIRPu9k3zgvqZGzDL4)C45f8YyM=kW4Df@ypN2l28gf@wSi_uyqv z1QY!Pv!H;9UV=O32$<;O_;~_W{YCQxtop;tWqErjbR&3my=#>MwLiz^cDmxILmr^%upvZwOZX zv3mrp`m2o!SoIhEw18EA(T4=A`V005I0E{7E+4!0C$c>H@VO?je#kcUr}$ZXPQW2K zo>LO8Qseu&z*kE+D&f5nz9!*X3Ds1qrMELVs=c*Dv7?UVaJ-eN*7o`1&QR z#uxgZ0d&(Vwm2t`oq|2OG#~b80c_Mt*rdmB#pe<}#bl>2 zk6ye|8P=2j^N)8n^hqYNcXa!{yuK%|^zH=8Z$*7de$=FxhHM_C97ldg!duWMC3>5d zdZ?1%cVRy%IrA6Gpvjr9qB_dg_m$SsAdqg6^r$H9MUidgt7x&@kuO1(?Py2kpH5I- z;cxU>fC_y3+);5Y&zDTma3AOS6SW+1`*oOQk9a$zfqM@5^b#dC-fD@>gFUBw#h;MO zPXoU_;+F%AcGmIysYV_FdyzjW)28Q~jEd>ysB^T^N}`q`KX{5(^Jc0>{t_)eDA8RB z+A(Y2I7HJ67K}{$5~7qCv(5@e)J|VT`2aPi2yPQvHyF4E&4%PeF0v zYaXKAS$~tjcgSMajKjs1$m`TW#aZ(gDIQr71gKq zjU?;W8kr6u1MVwmkLWLRlw-0!etyqN@#YU(q%2F!U#3+MH_xUfue@fO@@=Nu} zBslA@0Dlnjk^CC;Px%RQkb?X#W6u7S9BoGZ80v={_1DPyJz9P0U)59cI};i4>OU{* zEBU4V$s{@JuS5pC5iM&B`a}PZ=r|}Y)?deTu$`#?={V!BmG#y9K}`AlJNqZSA)Xfh z-0H%(QSpPis`^C7f3_Y(mRb5~^aeoctGep_5v@uuoYfRuD+NTa$FshmN|;xrU+Svr z9W33syYgT|sYn~PmX)*~X;@y?x^MZ;JMJuP-Bq`z@yNctJC-%H?YZO7%G=xPN;hoX zu)D0ep{=58|K1&~4IMjowQjnrta0Uz_R>vvZQr|0!yW4CsBhbedONn1?%Q+69UVI> zTPtq6|4IOc>m6o#Z7z5&_@%>@4sXJo|SjkFTaiYTeh$5 z;Kn;kw=Hidsk@`Dw9V?bu5?*@dC9@s>&mvtG8HRYD-UjN?%1)jwW{Qf+bR!kt1DgG zzOnM)uEsr^+siiHRo+y!t!bkPB_KnMTx73yH z+*-E0`9NdI+GG2xwvR*4{^Ghl%UT-_k5{haNaLPm2O2BahH2b!>DzSIUgA@IWyReM z$5yuN+qo@3p+r7E|@a=bRrFpE9wC<@mws(1B{5C5O-gRdwuT!#Zvt4IH zCF+#y@7%X1VZ9ZSpH=(zZax6{IZ_9iiN~|8hb0|{w-qPA^IMB&Q;%44Q*@h+P{14oekrS`|#$Y4VAmc zou9JhqTIg2JI5K%{$us9XM2}dj#I8xmMc42Uvg}mvK_MQwmbLjiL@r*yTvCtcJOuC zcs$+MPzBvcAaC21e(o)UuC+!I^sDr&g4(Lvb0q#)%A2MA+kaO>>)LQ5>~vf^=`5GX zE}7*z>q`>YFRNTL^&4s2Q*ngse>`2V#(Nm!U0!xHk?dIImQy*Bi#_9vWw+=%fh=0> zaXxn~ABWGCoX>kJuCLsl>us+rk*uismHqCxu5$aUuCKlQm67Xfud@02%2iz-eTX@( ztG$Mj>!YusWL)|-ab2sdIJ|#vB3a$6>qu)uBHvw(xT$zQWWJ(!>)kT`fL&_dFJg*2 z%gEpDE^ger`N-aqiX-)H@GE=aGfLnq$~WJ=uVuw8Dq5>)Yi$YF94bDrrm?Z1X<6gy zriPN0#RuvStXZ>i^_rCpt5&U9+1PNPMwDx4IZ)ToRMT*ztF^APv!-o%OCekSB0G5~3tXj5WMa_!hwX14MYF5|m3Lj|_e=6!) zJDX~j)rc$47q`VV;@`5GH8pa;_sl)i6h7S48gC2|BZjiIuA}p=mb>q6ZL;8<0`6Q^ zvZlUfc}-*2v5uw&&Irkn`a00j*x3q1%i)7}v>fhQQ`6N}S972}+*a3RwNqZ#Rae2Y z%J%Krh{4piM`{{d+Nj|g`dB3$6x0;Yy(xIlCqId?EGt^ZN2C@vuDf^OoP4$`Ck?-S z3XEDuQK39uQv6wr%L)zt94%ld?Mj;!`i%Qey#J@~v;N1^zL@@JqGc-I$P0)+kBn1( z9g5*7z)DlpCT>UCtXix6z~|DQNdME!&rkZolrLueS$0y@jHFFDybjU!k5+BQzN;2y zA4|XFN>m@^sdkzwi~DlbMtOSXXWU=(5`M9Uw|JuAw`ll$M}4A;elAm`e6PH#9b;T5 zLiN7#pQxw$abEk;mY4lI-&=R$@Kl>KOV0i1d?JZIvMXy`cum|qjd8iOT_3e>Bdmw7 zOl1Xmd4M;qm|^1L?OTiTv{D_NJX*gBE6Vc~s>(>+6l=_r9bHj%C~^g@!FY|Ape#G`8yNjzl?mM=Mws-r1W!E z0($72n99@s7A2*h>l2i3M45Q>Q(nCMA7aCW+V7D_rN5s>dCIS*%^Dc;Un}#= zIhL{AlFoB7U(L%!HC~yo^z^DMpDFiem3}@a=?TdnTT=NCqx?j^LoGUdP>M_s+Nb;; z315};JdG>mN99AZ&&vGQWqy^kb6%OR#y1)5DOn@b6p5ee$S;=pGmx+9Qr5?0{!HY{ zA@iJWEng0q=bV=L)UPC!=cxU9>Q~R-1wK%s_7!aelblm>w$m*0XFJ;I(Av>YfqJxj z?5A4kw3e@z|89bOKjx7V(W#Gjh0J%-xklzY$9qXmkaN5PGT+(GTQc7{-gmWjboz6o zUQzq{c-Lt8x^J#dknfd3L+#+v2P>`9^6?0Sm7dh{_42Q3`Fi0`3qXU z-o9T76Sbq4r>_K1B7W-DJ5S~>a^!E3Lht1NUYYOY|HouL^-I28r7Ec;q(^FRnsBW$ zpW2r!;Z-_hzEdB2WIna8>tnx`Py5y?y`|+#Q^Qed5}p{IE3eGUp2IsOhud1a4J zJMxu1W+LeBtf_A~To*pZ5SKUBZK*qQxS{zr9;k1O&~J?wg_~OIin^L2UF>>SEYwg< z%i)$T#_t+i>ROu`x@xd)X$p7MbRKDFXzJ`_MfLb|Lroq&(k6&)Y3gL-FP!9}qUG>` z_UkBpv?<)#(tcR7p`|_yo>Vq<30AP8!|h#7MU~s`Skl>5*KkObssOKcHg%PDb%k5% zk90Lrt|*U+Ms8zWdDGFBh9)ClaJH$r=0Lcvt?4>QJb?bXKx55eD!*mR?ycp5wOiX8 zkF=uu>s(RtS!r#rYpm&NuW96ji2;E;L8k!4So9HM?PM3Q)=C=>H?%i4?P_UlYP@k8 zSYG3G481OVu(PHUi#}G=*%j`ptH(Z)IN;tP|+Y#w{$xVD+|gz4m-U_Fza;C^I#o|j`tJii-u#`T lg)-7LS(%1Gp{S~_=I?2A?ChJWoX-&T`z2!tN{>_j{{^aFp<)04 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k107e6a9efe22fee8544bed815eb03368.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k107e6a9efe22fee8544bed815eb03368.o new file mode 100644 index 0000000000000000000000000000000000000000..6275b20f4b3d9d5a63e91561fa8f4e3a72fc690b GIT binary patch literal 34048 zcmeHw3v^V+nP%PFEzuG}-4YK=7^6lO=3yR^KsL<8v#l3GG9VC02sX2#B{fLUlKQkH z(=y~lh*^&x7|hr^A!p(&nRqg888XgJW*p7tY~&puLLAIuO*ZS3$vGjIArfE@>G3)! zW8C}w_2|C$s*!~4#5POW-S^i2|5f$Bs{X&~irZf0VhzQ0uuzgKMw->|Wx|JTO)k9V@8pna}4lNEh8$20JLDSPu5Iy*Wqt+(d9ZC3vU z+YIXKx!w#`^nY~Ez%bhBn(fU7j6HJZ>7lAihlcjN`}EMdvri9wL-&k=_8EMRE;{wC zNdL(bk%!a29XWC6WMpLNsYw6QGm)n@oQ+(4>TG1>sdpn!E`29*f7_+Vi8Jp+Qs+My z9@%qf=*g#_9!gvKKzNb=+mXvlPDGw~_s5YZ)8B}+p1d0AU-vHPio)T;hlh5bJT&yg z(i4%YvnM01?PnwXhu(;c+pB@UIIT^W( zPwTsHL>8Yq6{$>rH*$IDCD5!6FKT;is2{qhymSJ6pNxc0o{3bYpN)iHMPCS4LYJw@pn={}cYZ4peuwz^ab(e@ zE4W^XOr8Hwc;x=ahANZ)Ix;f#hmiLY+Wk?uat-862LJb+iu@?~SCJGg=Q`+WS}(m6 zOgWJAy_DqMnv_>_LS#pejTJdyBTetEX@pPB)I6i-Z8Q4`w?k(|G5Z52>^~F!C469q z#)?kE2Q-avGy5;-j)$q;amI>X1uU1h5q}m zOutBaLLI=FHd2ksh};! zTpK=^G%);6Y+(3a?dX;Nu8&@Mn;jkgo22mYH`&oEO;}T>vw@L+oCR2Nc=%p+^jG(S z_OGXe&u=~O5Y094eDub?S50$CtnWgeE8w4G83wO3<~t-4WV{L)(;*YdmNJu#{)px( zj`(W}tO$V*egOzTZ3mxFmm7h(JqjSbMc@nQ83!UX#_ z#HSPngCh`gxjXfRB-yc#Ip)xz|6e*0_(kvOJ@y zOFZGUmBr!oFO-DIHe?gB6ZQ8t^m-Bcb0K~PQ#=EE(Em4Z-GomS?t5`PgzHIs{wMCw z;`*;NKJ4}Ovq@na?5h0FHjRzG4gI_3>tPq{a|M1&cKRi3KsLV&o1b671}?yUWC!Kb zzuwFSBBgAA>4OiEO_lwxeG|5VFTVpneit_TSJ*G^5A5i*Be1K%jxBSAKFCi$tS%1U zm{M|`>~}{WW_|R+dEm$VLtnRA-}p}LKWNOw&NrbG6rE`r1^5yIpp-86Ny1-xBsS(h)=wM>ZC(A3@K_jYzRc>Vre-mVs3 zu%XG@+0`JiUB33tj$oiG*t(){SDUZR>u=rFUhqIiLBsDBg8>=}3Rg9(YADFh&tFm4 zfd2|t=08~Q03a(G@*$^?>oyRhv&9?qHQzph1bv;p;3JR-Rtoz3w}rPs4)1LtNVBi8 zdsoBGptsE@tYZaumzf}#W=*s-1igV4KIsC%j;@Zbo&cH?uA{xP%iG?Se;c@OlQ+=j z3pU*rW`=3FELK}`pjlAr}%tXe& z+(|?3q#>WTv@n6|-$_G0W7)fD*&^#$0h3QdKL7g|wMcKq25VWz&YhjUu6)+e(Edn6 z470v`eS^R>h0vwJBFv0$X{-&oNT0CyjE#^4gnz9#9_(2D*$fZK3> zVJ%UVJPD+GD&TuRU7cwMdV3c#NAy7NLI=NPjNnVGoyfZ4+5R;EL|gkgIs~CAKF{Gp z^$m5EWtGM5Qtbb6jv@*jk`U?Hg{WhXP3L#x3jg~=iX9Xy>4?uU3tys z%IXab<@H<2HgOMlX z(|v}x$1SsG#_uO{oMUTAWA>eP*adALw2Qv)qR|;V5GO;|GMHV~?PTd|#j(1LMmN-T z<~G&otjdU|KB;a)x3;^ZOkam%rUsj4UzEmeN_M4UUx@Kx6justm5JLb@HQun1<<_< zj<^vYy9+ch82;xU`&{8=w(Kp;p6X!a-Mu#4KOo_IaJ1Ei&n_TPXdJqJD6co<^#feX z2!TR=F?P1$aqT`W%6& z4{W?Zlggu#$yDcY;omj^Hu}8?j9tR%%_k}Q#B#6c@@6p)~ z;zjSZ34ZQE1!X=y=|;VJC(<*X=VOQJessFOS) z7R^B|X$D}Ysjl<7jp`SjEF(CRuTIfbN2Z9nlTHems%M7unZ3{>#^d6{+&S7d)pniuv&|3x7^ojbX-Pv1U>{~o#|o5LS6gSIG_cZINDc@Ov^0X5+d89_F4 z2C7^i=zncccar{@_95rT6%@X1`z6c(ydJc%bi$JoV6TkOd_H`15d2?o+SACNVX(js z=m+@BP~Lp9AGbH!=Y{knHe9RCBKd}LSjAUq)ujS{E))HI^RGvvTAY4SI_WnmpYrFTzBtOR%6j9w+WgRx zMKs}FChG4IX@ojVT(!C$@gH<87>CB9rRe+i2lWN zMtD2w0$&Qe8+f+~??zqV%YmN*e7?XBi}rI+7x+H~J_~q{z%O+2{c3-*l__)D>>)A|Mf!2HYO^O@%V z&z!8}tC}Due+Itdt3Q#=MSX04W_tK*9@&%oE5;-L+Ce08eq@H`&!_3a{qd*N|5{W3 zef#Fd{prGCtcOW9^0z2C&DrJfFRb4&|04gHn?&Qy<^FSo>^V2)Ul-D~qTn39{;Bn5 zerV1dqEC+d+ng+_bALOC_Aazn{`lfpnsc#z3uKe>tDfH7N@Nzzn)xSi`_R#-)Qn&IC(m#J9MR<%? z@zqtP{`&T1#`WFUZqYZ*Z-c)1e9h$g&XoFt|7XVZ-I#;*!e!C7%N5smHR;P0pWl3b zpuN;L`p*nOTM!^R*vHpLt}njeNPT6S<)!{&>%oO$tmg|&@=c_#zJ2z%{JDPw`E3S& zBl#T`f3w?29y`f@hSoE?JuW}3m!f^#-yEFY5%)I-*9ZCAMNJ>e9~V}v$C!U>VLwsC z{IkYGvhO(}koibaXxcQgKOZ0Qq9P@a*Hk>{+lQt9J%Gpv0g?~lSk0&0bi@gVHR2@||hyyk6kiTL6EG2$aLH)#D*zZvQC?A0TP&|H_3_|VSBl(ZV<2&>Tc}(%$ z9M5yadPOFX`pgLG@pz7SxHf2~NG;;|h2@CfS@HON+NrsMx=!}x@g4aF{27mq$)9_t z(E3`Dq6853H2WXv-)4-@_7i)Ne<=AHDc^CV|Ki5`Geo{a_DhYjix`mNZ!XzqeE)=h z^Ze2=3rsq)Ah*U%p7Mt&{gXL4pJ1$8hqNd8IvrDC`k-kuqnin7qa5WqXTSodj_g`uKnv=|~MR|U6VJ`eDB`fA%uuoAaHw)sUfPR?v zCl2mEDbjxygtBvE^p|MA;z;54BLBgB$qui>~Xmh|i!XGg{wU7DZ`%d;ugdeAG$& z74?B$w4Xxx4)SRm-=FdI&FMn?amV_D{29Ty`fSigU6{|q)oB^Q9Q|&p&%y-4_!uuY z8+E?lM1Qbv7P(7+<^$v{3T0)9LlEfG$`A41hw*OC53OGa=tLm=+wAX0Xg(R|1J)mt zzXNZY55SxJ9eBfhfPV5qX8(u3&n5fvdUitexVIDKdUjnlTe4kWAi^b zBaU|hKLvFo-i1>S)t>Mfs2lN_)6MwlsK@b;m-{Q%2jr!?3mgdj*f5{-CdkM6vh$rn+hVpuPRQRVSqF4-^b7^)(klfH4> zVT$L;$<)7!=Sd+kn^+A-LH=-{hsDoFu>U&JgQZp!o%ZKNK_(6{X@B@S_jl4xG*BYf zvy_#RT=0i@nnouk(O$Cu5p7K@Ul8~ohqyB^K0f@b5Bz8+?)`vw!s|(l`x609W4`Qn z;HUt_hV|kM#@{dO5orANRQ7X$}>z@Na$-a`Hbgxn$H9{_e#G! zpNaWjJfGp`6BnKKaq}0%x0~iKe~j}v#J5j7e;J#vI8t8rBK;Kk2kaeb!s%9^hj)?dVnY^>K`zb*L+Z62ryf8_fKWq;Tc=V4iNA{TXGzMuUB^A+-kg!#(63Gx;9yz%pu zY@V;U6Xz?CC)-W&J(jP84Ee^`dIe>o~KN_-`>x~QR@?r&vc|l&x>*?p4`Rf z`z-N1#AS+4SU*ks>zx#zlKK9e$1~(d8T;)#f5Lvi$xb7G-Jg_9@qEe*r-MK52-zuI zC-Hn2>z_C;I=+=9if4!yc%Je*-G7_n7xF1R$e%vMo<9&i7S9kLG7!)B^Q$b8-`#pV zGwnCb@eAXd_8ZW*DV}k>dH-SBZvbzKZ^JzQ{+RLXn#o_R@eI$8;`Rr>88}ZGJdFJV z=?C`1er6UukKya{MLciZHzU43zmM;KsO_}z2Hk8iWNzu%oWz9D|hzMJpA)P8(yd^4T@%_Vk%lkMF7EArclCvGji zAwM$weSnj#r^6;*4EZcA335MO(8Zz@5BV3?A3QJS=e^=Q!*Cvc97K~i4}THo;h^kI z{xyzOTAY^swNuA_pM4nw+@G=k9G5>l4~IX9^Yv_ce#)Om>h!=ond;zoy=A}6_iHiv z(izFe&lgEZvA){LL3sP&4C z;ipwo|El@J%YPF+Qp6VaKiTK1b@wmMeX+lvu>pJSvm>Mb+z|Z=N2P8``pf>@zW)AH zcJ7r|{{6Snx9qIEcGJ8U^-Z-&n`*HeK}i4cLW=7J^oOIGCjV%W*HMkXI0yLa1Wr}lkBLMq?IHWr+Mu3(JHpj61;#KEeJ>bUuPw#6>HZ1H&U#yP|f$B@NBfe}?t?dA0as0tlE1psveN}qlb+!Y~lJGaGsYnj{v}CXwD~a$irMeQg zr#zk)cVavee^J0dqW*cFPZXZSc#&rT55**EALD)k<32Uqqw&lIg>Oo&LG`*oY}_A4>RXB-O71Fo$G?L*r@2lXnUdS z;)?dv@n@?BCynWy7$K)L>TwL7xFH>E()IQCVeYwk`###3^=Td~pRr=(M*xc(^;A9A zk^Eb}QfOl|L1RR+E06Z+dN0wCjD0;Uy=$&~PY-6Zpp*8V)XKcgpMdOBPf8anxgO0(uOp zQym5qaR+s}<;x!MV8cZJmZiQ*Gvhs$`bJBg#-slE@__#9#Yr}S`1s@2$s2C9&BvuL zPAjwDY9EQ(OosP!d^MeDw8``|Q6E-(qCP)8Zmir*R5#AVJ{feEC2FrG z2~DjD+neU4J|#9U-GTvY`$XEd(tOtIMxE&i;*XX8MD!{LkU9=9n`x~IS?Z6YuFOI< zL7He;yZB!T8`dr}{acpytB@@l`}0`p8!dGjj|`e{?J}JVO`4wTLR|J7ARTK6%2?^~72gd|q?5h6 zRWGPhuk%o+u1ME3YObos_ZK#uMk!9)VlQ?*k~+E3C=Z!PH;M9;Tge05VAY^s9bpWR zcp+_&et-XQ-i?tBoZX4@&v_5|t>*FUe=BF1i6LJ6PabvaTM0I0qxD&rIArj1q ziFA0Yc%#nm#BcL%O~eYCQ)6Qe?JfBU)tLRN8&)%{?0TY@W7xrI+I3_vY6mAHEsgo< z#3ULy&b=*ZeC`h>-PwP_+$gs??{1Yo(aL_SeSGvt@JWnrwQuVBhBEB289(pVuEYE2035q) zRpN2=VB!(@69kWVZbe72ma}u=<0kxIRCZ2W#QPt6BJLr)p?_WD{r3XQ3;ms00=&eE zf7Z51*D75&SD^VM$~imzJz$=!(LEI#)5d*g(Mr|aa(#KN0N^nV&N0s6q(i>A=$Wz@ z6O1a4+o^y0OGk4N@8yd3qx?m^mrGHjIWOY9Tv-Yn zyT;zjB_WDFM(`1DXr}YyIZJzwtT)9TA`3u#We|1R1SD$zL)q?27UNjk;~zg-&bLIr zYv~_lqBvv&+SkxniXEjPvWK-#YPV5N_!g=dij|i9LOur zcoc!Wo{{a}lXV)O{9DI%8p0SF~K~9G?ep}DW=9(?Uel= zLF4h-D>`aVbk_Ft{ykEG#Z4I_$3*GP6<2CcG`*r4`^G;J;C+_HAo6CFywaeu@PF89 zs1x`*@H*m&`ZpEX-eUq8xNCL+}T1AsI>iV0=kCVy>XniAIb^y5&7Isus3r2q4>y<|UjvFd*_+W+SS@~igOWJ|ayx4(6K@(I$Z`rQag z9=l}80o7HtC-6sY#~T;Vmv*Us_kv z($u!5YkPf-zbUYJi@$R3nx?`Qz&6(szRbUR)3*Ammd34h`P=KO)_J!U?eSMt*KXch zvBtmV;o6o!&1V1F!rF%`_g2;y*7&!#ttn_+E83R$SNj|5OY(i&dB5xG3)gfvwyn+w z4}r$Q2WoehY}nmgU(>m*e#4{d+BOt4SJZYlul4W6813MrqOPK@yQy$>4~^HnwW_0} za&KAVin@G%#jdK@xD~rLc#BtelvnK9B55kNwb!-Puc+HSDZ0XfiRjjBZws|-Ur`Nx zv{jJIJGYbW3OdRPcWv^PY$({)wyS)1aZS&*twq6vV^;31-v+)LE7m;P)Kk<7dHvAU zqm89S`M&y+Kx3N^_Nl9CY~S?gda}XVx>j%T=Jh4*8}ggl>RKAN*L0L_^VZe|YF7J8 zTRO_N?%L=rDP6y&Vnf09it_GlZT{}<75J=J-@ScpRiL@8t_S+7+*DBaX!F`C|F*4# zuy;+tw)V}nE!%~?c9&G`MBn+%^;O+gyLd|jk8W#SNx!*dDh+M^)?mXPymHj{fOoZT zr?0SZr_Z;lXywXAU-PPhB41 z)1MbT6MZu$5MTQzpr^5(Gtj@L?$@RtpZazEdAqC=-S-s^*E8zgLN^ESiwcKpnYu4a zcRj0r-A;J^4}B$^`sKxd&ocz0g%&)~ml^P}UqM=z{_<2B;CZpq5}kh%bgFyn^#!!G zvtO$t2@~3uU}qxaP|A*z6APO7lLHRC)+BJ7>;&hLhXYi6ph}|mJ8$7#qfvsAX#Xnc zc|;u3!e$s(Aq9P$A2VF;gcJ~0L2ul73uFf_nFjqRxhPB$T=~Dy7h~UmTnZ+75a0Ai zRzYjR^bB^1AF*UEWoACb?Bwj?nQJUa2 zDF=<2Xx!JOyl!a%HNRRxPyJkzaMiX@!qq0VO!l8E``IW>ru6*xl7GBSiWfDnj!HgX zmT)znUX%R{N&_nRR=_jVpy+86>QwO7UJlu(4!`zEeLpO0~Iahez_C*&QCYlN3c zIjMie8~75HS{muG%Of7W`G;%Puv84-2gYrnd<#VBg zTjd>;2C~X~TEeaU{6xa7^8Vb=kCFcm3~-~oS#ti+xN2`in#jKi{5c6HevGTxF$url z0v|Nc8}0j=gj@Nix8_hGerAhX{=HJd)ws$RTO?eKtKiQW=#AqZlW^;}KbLUpxM_0U z(zwR?;*#(&^F_j`AH@p^u)_eCM?74v=M3=JKjr~ur5sGj>*myqeZc@X%GDy_N)L+O zFX2{sUyyLCyk#=vsB!blv27=+dl>O z5D`j?xWwpH+&E@|tN1WzfvY$$Y=Nt#SDk;Van*Xi5d2Y5@Lp-ZwHCNq?*kUNTJM=i zyE_{iTid+avv-}hyS=I9Vg3`7<#+)ze$A2prW5@S$=@j0u6jp5Rkdg)Bx1*K`CG z!L5$P@|q`MGjDKLXG3Se*EIHbM1vj8-Yzd&#=nI_3M;LHBxM{WNR~Bqb`?A@X>EKi z_QI1lZNi^N{V3aaQ=5;wmlt!fQ~gfs=k!-x#p3!gz7HaJ$P~-e%dkvL?~-|kLdxY? zRrz@U!DKoRgL~xNIT^ncLd7XnUw|7*wLA(SuRHdNn`1H$qi1Rq=VTa;{uxUHEB`*=6WQJPx3O#s@*8=(If?9tx4y1~pkPU}rG9gmHvef(PJv<+?YOPnN4t#_v5?cCc^lUlTSIK(DkW4QmdzrDZn z?T-U!n)u6PvXkTf{(G&x_S$Q&{rn!>xn{#!tHr`(u&^JpFp{Z^9sg7K+QTnB%mq$2 z3sm&myuZ$L+y0Gy$lH^}*n!iw%pU)$gqAK$a^|F zvE~16_4ExQpVeafl^12_x*j}vxUslfuczf_W&~Walp(8IH9oqB8 zQ$tT8&fWOSpqr-ts0e|~5|!plQv?|)-x!7r$+Pla+pUw~i3rYE8Qw}x^XUmm*k z$(M&hul;H0z6XhCRp{*f51c##o_q7YJ+xr$H-}=LdTA)O^rfK&=zMO^8StzOk*(H0 z^`(>d<~%vH-~p7C_tMb%oEL}wW$^W(`;H6_#pGTWI(ySSCl?(3(#f+op{%@b4xOFy z%c1p-=b_JQL+2Kr0nhcJb&fAW&b=qk-hX6hf$OQE&`ZC7UN575uMaKQ_$GL62(5eQ zOD97Q9N{!dkFjJ9uY2m=irm+})fRJCW^armP!aR9Oh2`&$HtaB(PkEFprRT+Hr?VG z8L&-1LbNI?Tdw;9XqrDvhrek)QDtGvkHaS{7NSi*a@y*=iSiv{Z28ZSHk;?sWIYYP zxv;q;^!D~*_@QUy(jCtpHu`fU|GZ$cERa4*{wRIX17G|heDxDw{CN3dU;I=B zGWjC;sPxBvlRrxToyaH8&gpA9TT<|=OLtg8@3vS&mrB@yv)gm+7kAr2!(Kc5I11ya zBXr{*Za?@IeEaY-MSV_>dls|X6GGoB>g)Yppzp_fJR`YwmN0|utGHlu9Qm`&j=sZ= z+&0TG@`F!%Myh5yE;Xb$F4aG0>pT2A&qzbFtuOTNo{{?BT1EyR3LI?su4N?jjiNqQ zHfTTmOVGF3`syFIj2wQrsINW`d7j63`^v!Fs-%6_*B?asKBT80{b|U!2e=*hJ!{|L z9|GSFTmyUt_}jqEz&8Q^8u%N)yMR9p{2cJ#1Ah*94)70v9|GP6oDBR;;4cDi0=5Ak z1ilk^E$~~OzWRHBZv(yo_!Zzb;BO(`g?J0n^N{}cz!ktZ0nY=zA9w|D32+MV9mUYw z)(3q?>MsIci}-ECOA!wtUV``y#0wFhLp&ex8N_oC{|0d;;$I_9M|=|T48$)ZPDcC! z;snIcA&x`*bHveze~Q?K_(zCGp~q8*|AP3Nh~Gi{HN_a5${F(BI>mbaSiCRLH8n_g}4Rr3d_joKK83SFh*8c^A6Nn3l7ws&Ofm0 z=lKVAKc0V}80n+wtnc@j%SV%0->8lC{Uwg|(fEpd2YhFKetp&HpI`5@CLZ;@f$MX0 zop99mHm-;0I{v8dJzQU;>nTSmpYsKb`|dzRnkP`v@Kn?#_r^i{@ytL{*AJpbs(u^1 zo|PT8AIALe9ym0YefH@;RDCz<66siFU3j4O=Zg>2J-+zBuHfPWyQ>!;$bx=S$4`;2 zq$BhrJzo*?2iNll(BZqJgX^g8r?~zRUB?~uJ%{U`({=1o-^;jufv#h?o`dm$q}z5z zjrczoHPWyt>JsV2P8^zh+cxM~A$7`uP79#Z0_d~=IxWy`M!Jx#NEfmn`a7S`xIXuw ztRR);JnDN8*Z0wN^ikheaD9-j9Y=j%!}Z_Mb(E%0THs*U+xC%$H|)>_eUaM;{qApJ zv(G;Lg{mRDu+v=VG#@(6hfedM(|pBFuV6i8w3FVJ+fZHwm1jTd`y8%!(zWfVZx^ns z>DqeK*Np31>Dr>TCFbmy-`rDi6Kwwiehv58FLfWZk1$LB^$qM$Tm9+81A7W%`fmM{ zz3=v7{0`dt8XmNd+za0Nub?hLdtX&?@cMw~C-$nf)=S?dU5XZ&@Dsr-p{C%8-^<~B|Z7#mc!PkXnZPs7qVqCn@0zb8uA6>`T zh&zfM^;tI`bw^ncZ$gYZSvMlK9aZ@^99?IZ{5J{y^@4w01pcz4>ztCmRPe7A{A(ic zuRglYCHYGPf3e^%68!AIf+F@SFZAEDqPTB;A{$wVb<>`dqLGDIOP$4duhT0)pC;(@ zBG7k%o+{{hW9cg`nY|=$eFhuRWzH%ZS?@+3b6K;Io*f~58_LMx>0FjBrw!?gMA<9G zmc8d5(DMcTh6wb7pf3~jwGrsw1bwBTZ;C+w0qEBW`nCx4y`XzQ7dFuKeh%rYMZV8P z$oFf|%LIK_1o~G%-w;9Wnc}`!OX4tZ%jO9A-UIzELBBl$J*5Qsr)v4{i9pW-{VqX2 z7=gZx@+WEeACEw9ru>5b-3avOD1Wk+|F;q7dy7V5?*iTIH$kLZrj1R1koY5{{~hs9 zAItwW;*XI2P2x`(%l{PdM@at>@y{5`|5M_Rkp6SxPaVtu9Pvj;e}VYZ#`3>R{1MVm z693Gx{J$pt2BK~W}^1n^|5z;Rb|E#h6@0Ik$ z-o?|EaUmA|L3_R`YxaR0OXiO)*z?g`^fczR3!B$sZMz0*+|{9ryGyY4Ee>6}qbPKC z&hw4N{o3e08`j27&||@PhdgIK%0 z_aB~--@oD+IsIZ$-z!*qz4}T~-yg6h`@=g7*A}F)kuz@<^__Xk(s%krOWz-{emkFz z@<-Xof7TTB4c$-q*~mGp^Zt}v-1pvpu#wlW27G-^ao?qHSo+?;dhyMY;=Xs^j1T=j zZhh`653dPbe4as15A^qpjO>HVJ3W0Dcdri(|AqCv+p-QeDGLq$lPxs3w7BoAt+?-< zr`gD`_omR>``E~z+bw->Z{HBQ5NGLo>xXRQ!sd;kH%qK2$AU7geXk{0pog{ZT(o6m z=zh}C(l>Z5=Fo>dBbW>O&Zh&2V&%ZZ}ZQTy*!;a8RfCd<~y<8JF((5q!@A z-^1Pv^XIVF2LH3e|Dr;pw)kH;AuD!KQOLEdI23@ioKivWjv;&t&u-ekQ$l9r)`h{hvM+ zsz;uG!S8U%%-;Q!wl8gXKgVgkhdE9iKFo1y?{SW23=g7Shv(v4Blp?^pSHn!gsE< zjO2iCaJRM3nZ2ItHdv$42bT!icB0)B&R;6>&qsdvI?sPC@odm}u7mtflic}~zA`kE*TrbDZnY}qQ4e@+_T^@2Ep3ARq4%rdU;nx)*D`NNz__u_} ze+u!dpSLwM6>&blz9p1^IFDa%3r#`1h+l6H#UUm;*3YY?dXD7cH}GP5==2w7hF*Or zEp+CQ)X;^m&j=mb_qpN6qq}pzneh406SmI`|6_@__n)5mOz(T?*1kV|D>L-^_hyCO z{O4;z|M^Tt=h*@zk_!I*8so8&_;;=gc$qe!FHtI2K*nu zHv;cr@a6K}!510)6!9~N_mvM1dXZjobMN5y2;V$BxSeoC@8CBGR}2qsCcLE=;|lPW z;lWbETYCrh5#BmHcq6dwmf=B+?}wkcr5EM&9ky*79$X5%5AhPBZ|@!4g7gx^H&MRr z!-I2>UQ*eMe882%gXx5K^bRg0ykmGUnefhDqyz699*iS7e=|H7jrf_r=^eCbd*J$$ z+PHwej&sE|==-ZN2bEwvD8{%@gz@<~WlZ{Hy!d3i_+-5JWW1PcyukX1hp*yatb_QE z{waU`mLFMIBCGtqMz=DHg>Ik>d{4vlZyA10Fz-R^!7m!WsVtGij0t<{+M4Sc+gUj% z*+jKsC1!OYwpy8Oip^rRS(rTz%XATJ@eF4$>`v`IVv!_R*-tec1>5oOY8PcsYO!?> zb9mTF76pxF5@%j}eS1e+hp!q}GVW^9Fps_M_06?@me<_UQJ2@eH?Lz)L-VedFs|to zcyFG5QPa}Y>g!;69d-VWJgsWpn$C_!UwgZ^X{oo#@7-P3wAU*rTkD$JTiQ1ITH9}H zxc&CVIvsD(aC<@F%4+XYZ*50cYh4W&B7{}@+UvZv?TsKbH1EEpp}AwFx1-7D-PO|8 z>eAHtufowbt`a z*V@+7(Lz0*qneiH_6}cjNB+maeQSKJO?7QGAB$#2v+=r^+ma`Wp_4bDG2vs-hO#=U z&+^odO&6AJ%PTZq1B*$;I2Ox%?b%lX!LequJhKkG~hhQvh>V#nT<;WcFE}K4nb$cdR5QVNCk-NVl>n z?EH3(lrK24ZCbj!QcD;R=`N^Z<0l#-K}6T_^jI|!s1oT(i3WXkXe=pEJqwo}4eAn? zAl-c^K~EQCR?hju>25*j5OSmdI%}bDFVbx|_c4c21b%J<0$o!;eYz=5aNQLMMCzxfo{<#ikG?5mwj2K}ut@ zdEN|>Rxe^%u52qxCix$9d8b5qW*?wroOt2ka>h>Jux!(|b*tAExl3BwTH#Atpu;md zM_Ym4E=p?c@T0$%V4ii;sZ&Fvk2<%zqot*>eUTfZXC2On+yzVXmMwB`qbP6rDpuUt z&{*qk?`rR;YjQVrws*K|>vlCX*SWWD-n?OpciWnZE$cRK@~)}ex@ObrHLKa;&F-b{ z#lE(hdUt!1w;=yopBqJ!u5|8(dk`jF_pp1AZeQYLcy@}-qwA?o2G`+%u4T`oYvQ3` zPYPdO&hejpmmR){eAn8w@@9}mj1{sAHgdG(MlD)0eOT1?RC-3SP`k zD0mS&uHc2NU%?AluYz;gF$L$a!wR0y4k>sZdsM-5S&xFV*?tAjVF3ltW_|@{u~r4U zS-pZYS(Soku}TGB!^#z$!O9ez&O8d9$?}1Vbo-_;w}MkyqJn48kYeCZVdt$zJe>_F zcp5vQ;AD0jINaVztXH8=Wrr1<$POv|39Lt<$FqQfr?6H9yI7Ti<5;0D_!p>W)(%&&wDA>XR%t+^*g!nhxd7rct zp#gC4Zn+t8`Y5w*z>&MxX0aEzSa}5=jKQxTKYMP5KIYQ6Xb$$A4C-mjgU;ffm;PWCU8LDUV(cA?w5F~R$e7=tR7FwcVsz2fU0u5{W#KEC^=G$e{IdS#0?Yaz5?I#%yuh;lo^4uwS^pk^_4*4e>z|J`CFv{c-zu=I zf4{)8{9yC&Y+)t7DgKc zTiUSA9_wWBxDMEGy+Wk#!EL%G{B{#J^g^KcJrQ3L@w*~krm-<}5kBwBHPEw^2|=dI zwb*JcSx*;Gzl$jOW%{9U_$B>=j)QktM%nKml@>y$6Ifc8Sd(V~=f$&Uy2RRjKX7$C z3;MyRnU1lS6F`px0bwqFkH@nEZBAGjcIL!rZqX{Ix$3Ml{tKcMO zzX7NDXN}U@YDx+WeL9mp=ASW2Hi}EZKzn>MbG5Nq)IhD}LpI2>Kjx2`5l!VePy@(! z`PnSuZ!O0f>emUZG_4ni0!EQ@3GB(+9HXoOFc$o?4&jNa)gB*?1UrHc4xtO5aKzBF-lCI?!n-AoF0}tDo{YU_1NtSh(ci<~#}i<`Xa}{Bw9jLp zLmeGI*RF}?|E`E<-|ZZAI;bOB$Uf6FOby7fUBfhx7;%*e)4*b+*PC#v3AdWC+l2il zOaqZoe!ztDO?bZvuQcHv6ZV+!qb6Kt!iP+_+=LIC@G%qaHQ|1Nv+eBZuF-f#n74N( zONGDc0sdJeG5PPgJ6UONhahYIZuVy-|0(p3dp9WjL4S03cMSPKznytxfxrbFQKQ^{ z=pSG!mg;=Oq$33Q-+?wqEDaC5|6_c3rPI#$d+;VF%Z_K(ZZUi$kpE_~(#~mUN(zBE z_L^3n=pbDFXHmaejc#LaYx4N;Gp@YXoHS;Sj_%{~(%P(KJMwqZ&zaU{<6Ae5p?@|W z`fwASJs3^;KhQNgYMq986+;div~X-o4deY?U_Cy|=A`mpwXia2f6-sf_O?PkuOHcm z`mf2J8$b_$#KRe~pFM29lL>5p49desA9Z%JGZu_r_R&BSxT$=M$4YymK0W?vPTGFR z@%8~fvs3x4?(=c%D29w@$h+Ff5=1r(k^aY{XZN{Z(7a)j21poa_0^4}6%&*av@f`Me!XFb$!{MkE^AGBS`p0bq zZuOx4&yjuXF?>9v`G}8?cDp`*z}~%`%)u=Xp!&>#y}lgwPuQ!+9~ZZYae++1&-ICS1*AOW4|XTu)Qtyj&mKP(G%%HH z&tUgd6FtclQ0U1fda8>SAL0CICi+ZQK%u9H>F5t+4?ewcdr_|@m;@l4-VFblF(V(BF~XmF{4q|3=^$*sB)Q<={u1<4K-UKhoc==coP+|9D&4i`p-Z&voYL`eg#ovol%{3Hy?M;r1YZfd6^I z^$T{-SL(z4g*L-HKiPx)CCpFl8*U$}Z?1EtUOw+1vHakK^u_un*uB`)UTDvYQh%~f z8sAEW>z6)*&!-0c!|g}@aE&uxFOTYr_A}I%w;v`*uKx=OjPwoLQ?x$`(Cvx#dROYR zBw~BfQZlkVss6AhmYf=WGw}QwF_ItqL)BiqJ@XkEOq6f#-!y)Dl={uI2lV!!_J;j1 z!ApH2jhA76r1mC%mikfs#*U9*FvwHKN78?h!N1{;@%*4m=>M2MVHC?7eSg{G&vKUO z`jb6o+xa1jXiwfhJVyQ1@zy*ZqI|=6M(v;LEZ5~z`!BEu;I7gh++VT4k(l&1`akr? zad+7MJ%02jyU{*oe_b+zPDBjy4C9lHw{O@#qqXsc6cqL${q^=Z7sL6(_KagscVhUg z1pid@mnY(}{tP&akQozR96Lw-!?^!X7&HF`{nR)joPOG7p5JNwrttXK4OyEkDoCTd+REcoJ?O ztlw~>P1v8#Z({m|zLY`vA>;czcALG7;VZ#34A>}#d%EkFW4LF zbFsk@_OmPV-4p(JW&QMoKUvw21iO<=`OWKj?BAX>+9M@C0Cl9l1+l&|?hk_9GvwWS zvA)50;FSJL<3oCKc)dgVtLuNXr=3v+g+Dyvw};QOaK5CRUvYn!iBE^|Aj-pda@?rz ztawb927T520nL}R6-E&GmnIwMKRzDB8ug!(tge65{Q}LGsy-NhUoz@5FCN3cL7%xM z`^`7$^XX*ce8a~Nmrd*qtrEimbW@dP(q!u*R({7aPbAM@ue)%T;YH~N!3 zynmqf%*P>iSf5~bfk|GWsr+S5_OHhBmz(&nHI={O)28wrd_)Pz_FCzr+n?e3tTM^_ zl&SoZWaE6<fp0?Hk_j(f)EqGMBHfchDcF8Re&pk>@t($LHh!E9*z&b-GfY$NV#u^J=PJww|Bn zGxVncng1cu&yl0kNxwPC;rV$9u-nN}gnl?rc+5X*j?q3Of03SFtWRE*@;7t&`8u7- zUzn_~S57|!jP+isQU1JzX8FtX{G=cJeNf6@!sV~j=~VvGjDOq1{*b`+4f`9%_OO3wbhn{Awf!`DxV^Oe zVf$E0~-Hr1$=W&eRbRIZ+)|mAH&ZCs|*kk?} z4EPv)r{R7I=4)j?hx0G2KhQj4{ekne7uu-zQoyOqZ!+0I|5OK^_sa9G($0G4Do)4x zQ$WDs^Sd4z*tWq zf3!B=CC1SGr?c&BI;}_WgT1i7ruK7+_T%fvY2o(4`q8{zQuO2Yj&@X$JU;$q^7~`Q zsQKvr!Ue3y?j8tcf6Q;^ z*e>uGfAOa&`49S|qSW?|w+|@okMk_0y>R2fL;542JhU&Sdt>{iM3KKre>vz+jj}7) z9z_czNjLk`L4Q0tilF0snc6!kz)M5@M0@X7acYmiTKhWek1F(7ejduS_Ey>p{a3-L zk9t4oMs+-ZJ`wwSqVN~opDFG9s&hFnp7tA=lllwYKh^yM_h%ydMKGUF07$ z{$5T0Q2pg2;2(#$!NdM>SjDNw1eX5MtI+=n|Df|T!+1KLf6)0D?MJzP>nSw z@#B!v-j0;RDo#CS@{eAH{@3{jp2y|VsL3&&E9W8neoBHpliG*gFVN1%aU5s#m%9~z z8GHVngxN!sK=MuJpTTbR{QIDPnsR=P=l!PA{SSToFrL>y-Xm>Eb~+E1^vTY8=yKOOHE1Zm49&fjOvHlDW!{h20tiyW=`m<-K_0KZNU+nNBb67rZ7s$Zn$FY1Cq$&x0 zXU|s8|K}+3P#%sO!sQ43a}|5{`12eAh|ge zT7Nj`&xtYG@1TD{jIsWBB_ui{J_(Q11o-pt!0EH5aU4hG0WWf61OZ**#j|52ru(%^ z9Q#pPSl>eB`9$0wi3!_-p1;EL$%f}Qg567%`UU;DN_}wu*U_WMU#7?pb}v`*_xSOW zz5cw(5{#z{6?wA%#_0XE!11U-KG_cqgh2MA@qUG=yla*6sK0m&_X}aKym|2{!d@1P z4@tbgC58K&(%-?qBxhQo;ODRCe9LF(Pf8JgT6nVk!g$DgeUbdsx zz)$T>_lISFq35ILajfY>&!Zac5ve_m?F0L#r^e4Sw&(1OX*q`WP}>vyb7t8W2s%9< zkz%-?LH%W}<2dx;?X9(MwmWu_;J3hjOBDUcKl&v0iyIrvR{~=o(C|ue}wimJpVMdeP=sP8RU;`uRKF} zbbnawZ(4gu``e(8I^TjmHv@m(P#fQ(sQ=8Hg(qS(91VW8z3_aEW6+=vwU=ARnUsIF zJ2p>mAM&^JI-UG4-7%nJY9C{NFx#Kn%h=x@0#@6Pw};_*E3>`L{fpYu=--$R4gHP$ z4gJmVJQ(@ku&ytShid;Yx0ktnBDY7le^7gv`p+C=e}O%-js3&i9@Ku}{tyHGXFHr& z9!h_t_M!1y!&JX?8eRmx9Q$V*`p;PV)APrM@xj8|LwVjTNBiEh7~iPBBOTfk#Eq?+`q^lR~qP%+QZl%&FvAPf0*Z&Y-4{h+n@SJxPAEZfsQhRKGZ(p^J_VN z$?;EXACKgJmfGh-m`~*MiI~5sf0WDe_U0U$!tfxK_6<5eA!>{J}`}c>U=WS(W>yLJF0X%w*RR9PyOXG<`elmC+u$?kIeIl zIzEa1qCY=6w!e(+A9EajU4Q;O)P(D?_t8HL;}xxk)%kpE|HyXiH|Rs{u}p1`G2>T2 z@_$(ULq7j1`bUo}@BQ`<{rTIm?L*_$qXzjOLjS;cbtU~{j^nUFA8L=4YI~U0TXP-9 z6n^ZNbo|%%4|9Lu^Syk2n74;{eu=QY(w|S(=6~b(&-W*~zE^*}HCg}A#v?gDXzQ)x z27Rf0!sn6NdaGZjo9vJM%w+S4y1se{{O0{hwsHNB^%wRddU=|C!}}2`Po3{+y>v?9 zH_kW2Z)`vF_{8_?n2*Ez3)lzajaWaO;q>tONW+Hl!#3tTR5>4oKbfD`RG!a`@Vx^( z?>j|*-WTVi>hs~+`&D-IpVn%;Ulk#LlHWFkpJ(^udlx8Qd0+38>3KkT-rbAy)tptD9NrgbnY1@$;q^+DKapIETc1n2qoE zxPkchIKZFeXRL}a<;U}hgh~G?!@xM7e*@pE&?Ml!34G7TtdEK_+462C-fz{O-!8qE zMt=%8|Hbzx(&YPU`;i2B)@~Onj=;ZHg72MlPto50APoId{8Ox39ngn(V9#uP55w08ZV?Tq_+30d*@*I6+iaRFI)+7km;0@*M5596NkIR}=b4M@ zJLsetB?L0f!og9AuL~Bp+_&D@O{lG(6`BEbGxZSFMvgdW)0 zHZ_XA2!UXSd<(bdK-k{Swj_tJSu?XVeo=m+UGrB-&rB8XyNmK?r_=MiJn-_S zL}~9c2B^2cZCcSv}7QM2j&KpwdMF75q1q2JwIZW|kr{uBj$ z?a|u%j+*{A6i*o?`J<#1=EtlyS0=eK50H05MLb@p7Ia>o9q%8A0RO$A(1orZrqA_f z;Qk_G8N5Gn|8a`;6#h6pM)(sjwQowS@E?I^#0h`l_?}e^A9z9IPsev>PA5aZX`@E} z$ZDHDD&%qdINNMa;eYVA&5+MxguinCLwm&v{}b2|C;U(NXUeG1N7#GDsL+Swd)Bdx zDQv#b2d_D`CGyveNk#Hc_-l;8UqMg8-3uPLf84W!WyG^hhWyEKM*pGyP4;cA#`oex z|EBhB<^5ml?-<^J!~LJ$e{0qHKOYZh=xMFi#;>e48wMZ|c>6KkT$;Z|r5ODutIaWk zJiEnGg{ zr_;VSRSbVd)8Fv?M1X}-~eXYo@t*5Zw zYr|3lxU~ZF8_Fx_u!Xg! zr!>Yx5JAh2^&1tBKjIES|rQ_-wnj z9^f(>hXtleV0hoOw@^K5&xK0MO*5?<*VWg-FhGoryyuSOyeKjvpz zfAaoK-$TQClh!xXpE>>1X7)gg{Qg@Ymf??-64t)|1`K(nU2-(k>TmU1S=APvpMMW? zbyv11zn_1vptMWYC(KXZ66XGb_Av7+{sMcd>-&dro@{tuN3PG6??o8b=cBgLu2%MA z`W^=USTubdNIMPNZKs zJAS45DepHK^~3iGv7WwSdG3kyTghI!QvH`(69LcYZc{ z;PpGdU>O?yg6GB2xVyeGI>WknbcucEdtT@F-+els91M!)|0yKn#YjyHeu&Ts$uodjlyv&Rne)qD|O)EltPf@2p; zoFn!kr{85=q~M=f>D0y&7Y{m}hU~L^3Y&tWTHJea#l~OchvWNfmf@fUU%}@gphc`G zewKm9W;gPXAt|Ul`z%JDfPshjer2(wuYeYGB(l#Eu=J1KY_X<)&T)|ykz%-42px<8 zTb+sMbQCfXosJ?VqE}5qr=y37^4Cv7r=y06^0!Vxr=x|5^7|*D(^0}i`2&;C^CzM2 zpM*|F1rwFuGYQ=@3H{MY=w*}84^2WZpM-vR68f=8=)IHB`#Jsb!i&uF1f8t@Y!v5; zu?&ZeS|Iw(7E6p3BSX03P|voMHj2?g<1DJN$I{Qb>F&Z|R$E09em2@a$^!V4Qvr>K z&N?H~`!zZp92b@DjV1X-tJ!to5yxVob6Lwe7s~W#%=p(*%Fx%pW5B9T(7cq!tRku! z50=uMDcTr^+(+|f^8}eny}SwqMNYQVWOL`HQGcs<4MZ1 zI@SDE@Q24_Tf{Nhny&bRo%{jD*8=&=oSE}3vL1FbrC2>y7PLaB&Y{wyJW(tdWwjaU zihP`-$1;yR)pnZYJ3I~+gi(z7jpavsqFFF{0{N;xCz|CuQFIVS8c%+VCx!)MCX%n% zC)F%J)`KZ0c6|9-{U(r)858yq({^QHn>NkP>fR)-;c90yi{#yOVmp`fg z0mVM~X8C`${&;c&_L29z`FvpXkIC#G2h#@QCbEw*9@2bKlzuK&X@kGsN0m0o%5mw^&&JCy#=>#)OFtSfzZmPr%`g3Ay!@gsjho+vZW?rrU;indDJ(c; zeEY?F@PJnQ`1un&xB;0ke*Q!cUKLIpKff5u#?@bIpYikmH?%(=3)7XcKpD%ZuNS#o z_Tn6uBixs44(#EgXzuf>Lt?JyZBSdT+XNkt7xp`X zL}!>_Bhp!%)k$=V#}bk5i3<`vVC}aW=v)uz6D}tZ7mSg5L`Bjg&KfKAa7CaqSCG@8 zhjmH>y2oX8NIk-Iu16%BDZ2GLw7g`amlQTVCVk4R<9ND7v|Ilqc4J}PBI=I1 zMY7wK>SldEZECPvMBPxg@pLoWjP$)CyTM;%pB$&n^)`*D8~inrZjtnwNVkY(8`?B- z+2J;g=x>I;@PYdq>it*QZJaj!KZyBcPM$TA#%VwlIB|WSw^3RUPcSMT8d>8ZZH&V2G^lv^iHOOZ& zf0#q&uRxM5+MJ)nSy#pvrZVkSA8RxoQO2o(yyro3i0)SDINT9ozDCbc=>eU-NuwvK zbl6RVPK|CVzh9E^IzpuKbuwO0;B-^@9$o%9jh?TTAJFMjEynvE7PKd@dgmnU8FxV# z+I%6Mw^55*hDnZPp%eF21ctbx8YjQ=!`Rj;vcq_He#X%I!HkfKpIHafI#pRaYiEoT zpF5cHE9Vll=bwm9JM@X@w4a`cPP^ZU=(Jazh)z4piRiQsoQO`lwu$Jp$C`*vJ0cTZ zu|MryOmxM*w2wFuMGyXF^Y+CP!6IO^>ZZ#C*Ij`?4s(v)707Y&OO0ol;5SG4U(!wxwuIH4xM9=Km;CArrLMAsCkyg)FJV=m4aMDZy^!5f?+9HcZho9l|#fOu#Xu`D0k z89k8v54yYoAw3}K>(NW23;Pl$|9jZ;=z1#tw56cZH9g=!fv)9iM|OOkNx$A@=VP4x zT06Uk_;$M4eg81~iSPWwv?lib1#i7mY5T|di?Z_nW!Cc#CAii##x4QR4DHTJA;bG@ z=cnqpfZ_ch1I}l7dFp(YPG7d9f-_i| zg43Bt!82JtaFK4`H0D-tD*X{f10C-?DmaCmw;Jiw>5l>$_@~hyl{DaF`Y58IeUn(P zLZ8YGD>#uIQuy&@9)%u{&ACzj6xOO>7pqcm94l8a{-`c+*#Baf2RQ6+PPS6P(X>i1 z@H?0rIBdTtmZ)INWD2(7u?2P>cZG%j(-dgLG-nzy^*fF?+j*a~6rlkyR9JA6+?GDd ztQ)M%=CxVu1uoWeH)AL8V_!i3%FWP6MjB1b!Jd=B-00-zaop`jNAAJzP!MaNefXgh zTh|AeeO0UN5ESGuUSQm17XBx2Ii`7%@PMGxyhj+`!Rb{3!<#s!d5`EGf$IgX5}4*a zq6Y+S75I?AH183;Utqt$0|L{$$IFMoN#B6LZh`X!_6WRR;3|P>-s9y9+#~QIfjt8E z3;d|S0|J)`OoJtre@I}rz~usa1U@WqmB7aY4hY;U@F9Ww1#ZPl$t1rLIMx!+s&X~V zIF4oci!|&M_!!1o;!hMfajAw=1#VrTVYk31R%tj#;QZ?~oG);XN5d-xK3}3?kHF=n z8YX+h;sKpw>orXEC+x-=mh{o|jb}aZWx`Z{q959;VXD6a^=rj@K%5^u4oecNf^QO* z^&b#e*1zhD8eP_ZKww$_s{I;W)_*`?S^uhgHM*?-fWWf;RrhIhS^oinW&NwLXdwM$ z{Rae=^)G)wqs#iA5LnjVwla;`GqTwU>W^3|RN|_gPtUZ7w!m+Glw>zngDJW$Zf`Kk@Y)C5^!INVnqUR#{)UI>Z+zc)GNY zN91%t6`TBvzaovI>v($j)qY8;M0!$USf75eDolauS&|7chBqFTAl-dvte)jaUu;Ns z3;vFA-l>d2=?-tC5#$e8$3?iClOmHV^N%bNF z@zAS>QXOfzN*L5ElXC7CO;Khdnjj;;6DlfdoDd&Nc#?z}f^0vTME*(nJtj5q`5hkI zTEsn@bT{W;r_oyln;v~t8eh!$_agx!#+3hn_}GVA$dx`K^UHR82n0&=2tFAd6Zs{U<;#4s{r4jCMEU(f z0Ocn>bAB2}$OPzKN(f-C;9-XrQGOcN0~*x-9Yki5S)!+i3t2D~Dgqy$4VaQ?yrTRh zUkM_Am6B+3GSHrYJkr0|#K%N_zoGv{YQI*DTPa%mi5{m33f!s1@mX4&hKTB~Fah5P z!%_JN3hBW^JdKqLqK0%{2^oDuYhQ%&2mY7FBK@Drr(iN4s}uP>rt&i$=j=@SFXg8p z!JK~??1Lo-`LCk?lOZj?TZ@>veCbE)kw1w19#j4kB7ct|Kb0@@rHb@)Bt*(zbwp#3 zZbaqNkYvuk90{0>u*p^RKPB?JH8J|XrsiNOU;<%l5NE0=2)neCju3)8pU8 zKM{glKgd|-Cq3S8KDaE?`=`0g9U?)*#_~v4*)Po5B(4$;V$8>rWJ5?s5zGE1W0_B~ z?JnA~t#o%ykrqle78f>l)+{Y<+_7}aEw>gmZuM=i?c7mWQBc#g{gyq;Z*K7wm2528 zR$O1xw6z1vJ>uxKqU0%^rwC=Xel?4Xco{rY)rY*=Ev+ov)=^VdvAl8pW|ZBrYsdEGoz+V#>T63I z_g1%V*|?@~Q$cO1zrJSa#?Bo}x8?8LvOK@8vKVF5fq&;MJGU>ty?W_QR9?Z3rrm39 zE!woSrqFkbuc%2c*H=`~vbu2h&A#GIf@kfr#?syE>su?fG?o?Ka#QK`aOa$j+A%bKOz8hk}tHWn|f z-&I??e>CQwWX`tsNNCvTX$O}*{Qm8?d>&P%Nur7 zZfdM7thl|p#Avr|>#Lh@zI`LLW0}x(``WI`rL~b|mhQgo)*_y#aMSv5o|01JDcsq< zWBa)ImI-^7?WkP83;xsTgU>{=*~Vs}NAsrqnx<{_)e&u5vT<$crh=WNYdUu{HFoYS z#Seb5vuu59ZPT_c_~g2q3%1=}TeiM&$M!;lPi))c+rE64FQUwFdzEajXsvEsmA_;A z`u3fZ@iuYZ7T@-YJv+Crn52ER@~OW>s=MwlTKSXt-Fm8b<cviWzbP~Qs!B@Puy0B{!-d4f8>DC?F{f*<;UAL2{JM3C8nN8Q!l%a2o z<8PZ~|Ew%VUu*P_Q?Bf1Ye|-GduQajtgaX1-_F}=8dtT|Vw{dRPMZ0~8<$kR_UgiM z#xI?(p33#tZeQC;Spx3t<^<7%LcRb(G`IZtN`Nj50>atBMdmLZX<#9WAES<#8 zrQFVyYcG#)`{l|j9?w@~`-*ZaE{ktx+2zUGS?a$$d8PH2$5(cF`eDs+S@LQMFHgUk z!b$a8$NO68+UA{=t*T3=nE zZ@I5by0b9gV*Bc5l;CZSj938^5*1DpDDW;@wW`p&qF~uF?=o)zsBN8f+COW3jqP>bu_^gp?Y{!= zO7BWh-#cdSscUPlYm8I}LD8zR(bw93Tf^R2tVVU>PKlK zLt(4y8qdSlFWdj8<15ZbTwja-2Td~Z*K!8!p9d!4r~T=d4g9Hb0<^lWsnX@gKI(cb z{_hh1KIyB|zLxS2sZnJ!qt0E7Yv~k&uj&oH#95$pI`F}i4x)be4f%G)icbP~x z>$y^-o9jCu8pvGVL6L5jb3vq=>-(-jj!}PnbXnKaSl^X~bmQDyK2Ew*I1TBkf43fz zJ%)7SErDZ(bR++uA>GLD5&mtK?>D3y`Ty0BZj|p5&O~yI{D~r+>}edmb42<=Q+l~@ zdb9mMBht5cO|0@yqe}uqj=R$4pyvw|lD_ntg3u4D0e*-v(c2 zb4~qCe4)O^Pk##~udS}pm)B9}?_ig^Ve@QI?VU9> zb?xmeuNwbck;vxGCQaFfx^_1C#z|}{8k%>tTn6{vy0-R)mS$mwhUzxhq_nO>GX=|Q zZt1AYE8TR<;`R<-%^r-IL()G6tjW$Ws_ zyV`tBb(f*yE|k{+9lg!OzhT3+jjJ`&ZfvRTY((*wxufK*(%9mw^>(y)Yq?@tg`l3M zrv|mU7;A{VQDE3Ij+X{oK-+R#{6d*wQ??Aptyy02|_ySE*iK9<+s(bnOsM%>lJ zbAk-Td z7r=5}dGIra)Zc4prC6`gGM%PKxdxDHvG0IR<%?-d%Y=XBa~J0FB$hEf zPeu7jwlVYzz2*K|(q#D(2XH}YW*g?5&^XNeX8cQ0zBzxs$iG3J!)W5h*l#hEC;iQ# z`#;Pn8aDIZBl4T;9}xLzpK1(d{qF)jQu~$vLCbYsWTtA+QdH)b?e{baHkVCVD*qq- Q%k&S!ksA059b@wUe{uDuo&W#< literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k16be2b692ddbe79e186bc00a13703cbe.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k16be2b692ddbe79e186bc00a13703cbe.o new file mode 100644 index 0000000000000000000000000000000000000000..5d6acfa82b7bd041515cf81e73af2d2c379e3a27 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+mFh-3m%)>k+Kp1A?+13MuWI#e7UXvNoEj37>N1v8t zT85kmk!5Uy!Nlx_oQ<<&;_=clWSpJMIGW9w$P*vpaWI25*{n|{=Y+@%kpO#e$Lm01 z-246Y=)U)=L4ZB6VJYaoxBmaHs{d8>|5e?({pF=Amg$O1Skd=#J$)aRvbTPzvqMwUyKBzcCiV{6 z##3L1yE9qQ|Is~tLujXKwzujq_VB4^1}e|*AK3ohGXpD5KQr(x-E$SR&*F7x=E)Z$ zy(f-G9&>&ta(w@Z$ncz#k={9{BKuaIj$GPzIx@WPy~xvZ-i>cQZE-KC+N8cwR;S;AKmCnWo8Sw2fb6|^rOy09xB|MIl?)o_EiL)QNci11BiojpjLbZL zF>>Lt^N}&r9t{sa4E!?idhui=74X!Ve-a*kZ0|tD-WO2^EqG7Wa<8QRdSA`V^DhNc zcjtaTHMzSc_0`-E+0kQTMGn|V)4OYY@Tm!!=jvJ8gkHjJ)>%=^{=f{LXXr+RoRG8T~hrKYn1- zW+|T}|5Lu#58rzQzIo60?rh)dzJc!@9mV&KT6~XuQu$xM#s8GQ-pB{9?fhXlS~~Hq zp$GLt7yhN>s@u*oF8;FQ>hMcT`+l{C^`&EeoSn|D4*gKS`a1l2Fqd6DTRP#;V7+$e z?BD3uhW4>Onnz6Qe{?zU(dSrS^j}%urC0Q;@07B>bN8{U|F(ejozGxbFQl-mX`n5} zTpQYx)Hn1mtZ(Q+?a;;lrC+`H4m&jT*Gb`_Z?i)e>#?SeV|~Lvnh02Oc<4cP=+_T| z_J5>=&#vG7D9ttSeCXQFS50$CtnWgei{PJR83L~~=DQ>lWV{3!osfxSOP#>3{u9kr z$Y+#`rLt=t$aCRumi7%pz6vSVFnAsY&%@xEWF+}6g3mGN-wT+cA9x~NH74VV_8(V`)R(b8cC+;Po-mD1^iwrYA0Hzv@fg=Krd;{BO5;A}%Jy7M zo9zjwFDwo_zfck;+mKDjPSoE!(Cazq&xQCIO!f3_NB`f#bv0gl=jq$PUV< z-(SP}BBiX4>HUwAO_lvGe;c-fFTV>veh)VLci1oP5A4w8gRrZ?j*U}?C%sf3ES6ySau{KS&F=iiwB~KQcli6|mWs%nwD}>Je zLC7S@Hv7cQ**_ImeLG9`u*ED198Dt1x$TYZ9l;K-4|nR?)gtQrTG!s#+7M!MTiZJP zb6X#u+p)c=bxT{kUf(I}kI#+W)VH++yd7+Ahd*xxgIpNw`+dI6i9R+uR`__8{ zE&gErU14UJhRb5LB?ro-lN-uUwr$zc?(ZmIb#<+e z*Tpa^m#(Z6n7RUN)tbERJbK6Q)Drp2U6qsFC^F3i(uUo#wJhQAWGk5c8HbM@0pEbl zx{dGH&8R0i94CdA@&e#lu6RG)d4j+PR;J?CBk4iL%TB=EM>Aq@31;7r^lE5D|6Ras zIKQx#C`!Hr(mfUMy^F3H>ASnTXD~-}clQhjzh#Wzv#p)Ty5iaX4FE)2`x!a}p(wbroe5#qLt<|8b5Y3LTOV=_0(g5Gc?Qf-jbKHhA51y3o|@?PzLib$7J2HMh@p zV`lrk?S6ORg1HN4yVucWZqX7}(%ID9;BN0~@9?*{TRPi2+ztLMO|5?S+N!D*YwFf5 ztyxo1wW@CEhP6vql`Sn}bE@16+;hCa`bKwq3l0Px@wzcI@k-|_`254d>vpymaQi$5 zBTp!x`%H0TH^QW;(Yi*_DcY2F8a`T&b{C7H%uS+vIc> zK=&>vIolZP*{Ma@FM%Wj4Q8;ME}<_zN%EZR_>SZ#KY&XEh0n?6(jlEHpid97 zbQ(NRec8r-p38dn*`w?-0K{~IlkI*g+0Mt^n48XyyV;MrgbeIS;L%@C2op(yX#AyJ zI@?UV=-oEK&wZ$%OvCG(lN}1$?P$TG{RcAG;5?QevfbzU)3Ge7A4p-ZA7s-*Nw(ii zdSg4U59Zp7Lb{Fcj1eC4g~ms7QA?T#ZDYLYp%gtve=>t*2FE9jr~U^Hrm%BF&p3Sw z(F30!vX7^;PSHQn^JymN(LeNIBXI)NA8`KjA)kRB{A5Ubfgkio^jv>g9UAe2x=!*1 zsv|byHzhOyR^;|gQg}fhVT5-DofC-P^s~_vHju%JzKZ@)=kmV6`*UWX7Vx1z^#9~j zEK2sUJ7CYjD)zumqNM`)E@iNhFDd`r8qOwh!Z{%qn0{#6uw=pk+|O^jC*oIaDwL+wvzV1CYyva@JU^QmA8 z>a(KklC1mQ)24-H&!q8Y6MwMJ?k>k{KHnaglFt6p$)<&7O(*;;j-SW&cClHc+T;hY z=nNHu-VS=KZA5>}$?S|az$fWKp3M@UEBW6n@fz^Ne?Z`U^Wa|(Jw)aWphy<*`Jvn{+dtrXd(hs6_R8N@k$xw;O#bEo-Yx8TYzoIa zP#5jVAMLYK*1EB_oUg?>5&Ue4!{`j=?g9{Qg~ z>Q??i`sbfW5gy}}e|@p3zn-00aeez*E&8VUZO}KLuUTB*SyF%S|E!q4eYsdKTo!%1 zTycF@k-l8<`OW7C+Dm<-|Ev(S1p%V{J!1W(`N0h+^_63mm->sX2ZP0|JZFYUzR~p6 zvlIEiSu~;bEALMrzs=xpB)`MrZ+0F(?IiyxTF>kvf1vfjN9(0%uj1RZkb~1Z;{N8~ z`XGNhr|Bd46W+?#W6Zzhu%9Sm{#oN8+4l?)$o0D@GG{@e=ma$@$)5Br&0S+ z@1z@9fIra1bjo&;i^q+H{A{HBk~Y3R$@~EM8|Kd(;zt$KkMDr}o&bRI0r(Ha-XCBEy1_@0{}o+Ae)lSuvQ!R&av9-OBw585em8{)gJlYM!7NB#qU z#z)8G&)q4szLuvd0farx{zv+^8RN74_zvV7NL>rC{?ag93~$d0jX~Xrr|E0NC!ub{CnMc5;!{wM z<1OnMU*GI58Ba#_?<$7%n)^p!Gwn~{A5YQbq@w%-^YbjKWM9}H`F=LJqlkxO|4DKC z!~f0ma~`MX_M|xheHMi#>$Kc5(%+f?PV3j?WOg~q^P9mb@UPVDn12nmxvHeL-UX1=c?Kd2$++O5Ae1GER^f^Qie10g^5%VYHx0ykAjvKv4iJtsv zmxHg57%zk9;ZNB{f6EW$xQ*?LcDda&-#9(@N1FrI5&nqrseQ~JKg?jyMmUm1&qgzd zzoH({i}op$?;xMH_3s6*e0|GsAwP4+`h)zL!72JA&_`XE&qGz|nZaEBeyUHz1j6_j zFE0mmzTZTDux~cGOMvDBEC5pF;NK`40Ha3K`}D_mT*Dv`;YLZ32*|byzd?FrzUuZ+k&nnFCB-Eks*!)k< zjN>zaPeI*?cj44SwI_Tg>PCFlI5U16>Tx{e<^Ia`0ePwJ0tZ4rHq7VzQSxyY;ZI}O8&%{_>C_@n7O;zW@!zYca`sTD=1{W;t-Sehi`CyC+$Q7 zC2~DWT`0)~f0;o&olZ=m-DLlR+Ok-_An-p4ac5wBeE3%n_|c9(_z~}f*OM6MhXAi* zzU*?~r~t)=_2Lx9-zDr3@V!5V{et4{Yy5mq>F1W^!G4{uALMU*eLiEdr!_w_dRiFpCTW@d~-R-ANlzT z{L`GT%md!IKGS+-%~uTTFXBZG)@v*$BHlxPru7`_vuXWCJqPP`j2`W=9>w=tgPM80 z#(X7wY`q>eU*YrHlCRKaoQm*AzMoL`hdpurm^~5Ls0;J`^rx7wkUu2MSEi1Vuehh) zIA6)(`HDMnz5;o2+!Wtq`AW!;Z;Z@W5KrAXv3z9+`upVhNDi&9JYJ$dOFn{lY05_& z7;mZ@{C(?T{q{V3zA)N;dlwf+txr5Y(~%l|UX(}i_>|1|=RBSvKg!%?=lK)%0~zdfU6yJt;{{0E#*=3WzSmPN!Pm0?g{AS`jssDh~5A28i z%tZP;hOf`(@Ok6T@$vonLwtYE&rf;2{l+}dPc-bu590F%`$WDU=l=2Mv>%V3_h3Bi zpW^3NSpP@v$M2tf5bvC{A&Kw5oz7AA z-#AdAay#*^>NfV@nOH5spxCdXy$ki&eiZrIIOBN}@L9(5DDwYo$~VRSlh23j*#496 zzsa9aV)1U`MCL}F^40V6pih@>&?oGlsay1E+K-H$zy1N|`=umv{)+sz{Ha@uZ^(}f z-w()OE9tO_7ehWvOM=`_7k02H#Y6sv^#`99^YdPDo?$o-KMJCoI1fLD^KejhC%=zl zmBtLq{yIa)exH2}1l*so|GXi8_&gl`AkNow=<`$ld8AGsm?u*m{I0a@xA}f8CZCg$ zeEfWogcR$mog9R>AIhZu>EjmJpB6v=FSt2*&F2Rh?2F(>>}Syd&VS0gK8oij`24_> zpL_;&NOiB=)4)9q+|$544cybfr>+4yJV&VQ$G03<_s_5W{=*+d|Kg?I>fdhq+rR%= z<(R)}_~^o4M-LXU8NE;UxN6l*%M8N82iru`1vTgLWiOAyF49|L{7=y}d$jnx2P(>x;ajv+m`)5avD zJ!JrzHk->reA7h13#idT!D);utOSta5o1(S&kqw%e2gov zf)VPAUZNf6ZH$#SG-Gl6o-`|-(j9%VbN&^!8J{QNd#Y(j5PP&_upBFi@G-@@61S&Z zo)>pwJd%G=z(At@d9F_sp2T>OYXJ|%Bx)bye{)>-$_9z(WAX$TTdirSIy4=FaR}Ly zis;#((-e=zA2=1aCD~-p2wOsA#_eX%yPhKS9+R6!N`Q~a?MjS~$?X<$C+Z_67Z@Q~ zm1Gza#UPpeQ<2O*(xb={Ce9>Hj5A1gBhE(7K{Mwz-O71Fo$G?L*r@2lXnUdS;_}wC zL^D?CsRV5_W*w~>P=g*v;E5a3!D?M!`4Hxwn|n|XZO(c$j}2H=jQj{#u}@Fa^Bl>) z<12+W!n@tj-$hI-h1YIJ$vI6xpQfkA>QADsm?U}(s#6_C z6Y&Riy5&nB@Lw9#&+r_uVb;uH1x*>Pj#ZnU~_Cich}#Jx0)-Ic6 z?c(1SHmqG{`nN6Z7b9yn_UEzGt1We8tJhz}(lw2mr|R*2hE1nYiqqHHi(QYW-P~xDi;Sk5M7auadMmkr8|)f%tRs{G5-+3? z((#BaHbJgu@?J}xNBtPBFGeto%&a53L1Mt)i}lb4e9W{)e->-xr!nSd7c~=c|JiY4 zwdZJc;|#kO!^Ijhn$A8c-l+3?G2Fb0i&&xJ!^pTpd(6ZqX_nPpx32yaFAO_5O}m0D zM(yNe#8u;(Ec6G7OEhvkHrbMH%>ACEd;8Eo%v?>hy5H*jzx`q z^mp|BlJjpq%lo{Y?S0<$Yno5S9h->t@u;_F#E#7a4~oa^Ji1WkN6CWj$|$8=)%WM_QIdXrXFuxpprX0DNQvfzIN@Lq5o; zx-3R{Gtd7?_-vm|Zua!^c`tnT`SC66d$WG?+YPqAnm&7K!9Pv;@xhGrisrO=81i%Q z{GVz_?ZzL~739MnmEZ}_|9N>CW1GwIx8-zOx_1S8Zr?r;qZ|?i05>1w>Hs>=X7P6U{rY$Q_9A~lU|HS@tm%B zKgwU!bGj5Qn)4!_)0M5jv1{x(T@s?`V+6N(Lo=Np(^=YkWW7H2fwBO^R|Zk1O+cdd zKbGyjWKpfPJ^mQda>gb4T~2?biQO3Bi^m;D+r0NXNpVw{VTG4z62>9Tierq zZf}D2$1Lqt&(`)+g+c-zMIf)IWcv?foyMo{npoSD-=m8W{P)W#;x@FWTh(6K@gO=T zn1_&ta@I1%)cC5Mvj5{~e53Y?j@lEQwLQ&qQh~)y86(F;>CO{ZYELxXq8fY0pA7Im zOJfjuvshkfP+9mNx*F;P{w}=%Gz z;_vIoe$wMbyR&~Rt_vR!*Bo3(eiISu{|P`EAH{?kzfl0GV`GuDG0<-gRPT-w@xAtvwB3C91%CJ>{z`8#9Q@s%G{qJ0nAdo1Jk z%l69u)cC5Mg5M87qV{hmXirnZ+Ma$a3Q;EjGn@2(S+Z;mpEnc&(y?AM{D5Wb)<~MiNFDPl=xL{4~w&Lcs-t`Tg8#mMx*0-#$ z-Cnf1&0Ac$vUFWZV|~lAj!heCn(G5=);3q{SXMv35wJD2gfDAevU=l&%0}P%bp@L? zRIc!@FWTN*Syj7cNBOemWslW125QzcFP~rgSjCQt4fAW7H?=G)^eq={OPZH7`!S8SNStkc)BqyRhweDfF8ZYx=}tzkn=`^F8co>r$ zm9OippTDGw#%ow#*;Z1qqs+HpT|smC*2>tp$;%7iq(bdo@iKJ*}QT6eAv6DaAWJ5+Qv=7UfW74x1jHWh7FaSR=aph15a#hTKK5) zH+8LRX>JPEZ7*Ep^UwD!S~9<(!RLQuiNA31B42$$fwyqMBLxfUef~N@SKqY7TkpT# zPpquvzpX;Qs)AoTxt&Vk4W3QDVBx}r3knMGV=4F*m1W-McE5Ztt;O%j_`NL!p#liC zGw+Umd~)6&xp#i)OOT^*ZXq|3T8yns0(mb#J}CfVOt_zB-aHq2+_)OfVsbQg$3gENJ4Z2Ljk2E^F)r=aGj4 zRD7UHqUSxYY3MP6G@ARc|qW4Ik^7{xBnuvbob>P28`$Y7#zexmt+-wuIAHgvY&owKm}h9c%~W@J*`Ha3ZCN2;hSasv;ebj z<4W`e7>CM9aba)aO7JTZZkKQ+-#GNAe1}luC4PbhULfI<09S1Z^hF83A8_NAahCyZ zOcjqwIE{J8y<7R5A>mee`=x=b^1d$N)_#5};Z}KnVd%%m|3?P6QQmAhe`s7| zj4Zeg{IG-*KgQMUh=f0Ef%hBejrM&_!ma$%Q*@{hKa<2Qe_tr!YFy=ujS{ZLRq(?G zdgHi9B-}dgFC^SLZn~VeG_G;JxFmeUe35YKNAW@eY&O8<5f7K^umK+X;~roZ%E6Sp zZcfeE7YuNtT#XX0^q}aQCEP0S3leUXw@ii{HEw|%cZC6N95*E4YFtIXOTw+=eow-! z<5J#2g>n~~Qc8Ce*QhAChu0Wulvf2;apQ;uuHr+#1+L=2kOi)mUUmMd`d91y4Dd%q z!MmmXmRsOzy$@L6YQ1M7?QXC0HMM$!T?~7K2JZ@QXKQ`qWBey4m*RoU_;pAA8&C9e zjqGN39@J4?Q)?4`=F!*QerwHIJ6i-pP5yQ^`X)zi9r({nj!yl!KiH0+hLEB*`GSzU z+}|O@#SfRz?^u?vs-1)1ldIn@sFpz=Yy2I>9r*o8UuTD(pn@I^P0$8!ng8*odcO%S zwC8V>KX*4ei(4=r{d`4TEB!p)iWTctmNj{s{q-Fy+ZsBXG5k&L{?Hz@SZdx06#$Nd5ZJY4-Qg3JfZff(1H}hgPwyR%my+eQ3RV=Tc_*1ePcWGl#NZxzcSgoBg;4QI)u-czQZA1I$ZKGSxH%&8 zG5Tzc;++h~u^jC00HpDaxiz(P$>8guilM0S)%7{qp87VHY8AX2jL`V_3QB0=$*RNZ zRzacoRrJ>Sf0N^1ul+|BgyLKEmkHsmHX+=KprSBet+d-@`w{ZX_Rm@xSo!w@pU8d= o=|7j{@MJ%71=U{V(EBj3b!=*Kww>Pb!{btZVuPOKWo`fe0BLa>_y7O^ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1a497f11e71eecd11561cfae8dbf9368.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1a497f11e71eecd11561cfae8dbf9368.o new file mode 100644 index 0000000000000000000000000000000000000000..b1f5455eec402813846cbe127513a6565f007168 GIT binary patch literal 73272 zcmeIb3w%}8nK!)7B_}8MlMq5e0y%*I;T94|ARxj?2qd6D49e}b%^ea+Am2&Q6HUkC z!CJLeBJDToOs6kj>vRYsmg)4@lHN?KZ|m6BQhqZXXXgD?iq;9XRVH;b@piudvo3qD zy>pVFwKH$~x-{(btY@ufJ?puxXRW=@K6h@uev8xLV5&ISH<$^jTE;4J&DsF34zOgj z^s{jNu&d$c`F__w(H~k36fpMCC0G9G7xDL!D}TwaaD5qn|Blb-nyK2-_1!zZJ@{r4 z8-HVubMW<@uEBR*?4kd3vC}7e*h6UhIW~ipe=R>S^u{LkvlpD~p@QUa{Y6*S=}WFm zqV+kJ!OH*H85kNtKc~a>vj&X4IQhW~ryscg!cts+^XdC9{D02CINA>2@1c%Yo)|s- z<1?d&UVUQpaav2t7ux9WdUj6iiTVDS3 zh2-U{2RqI@K6*OkE2CSV`^o4lzy9&)-~+!MJ^lD^(XM=O@WBT!+;ZlD3->L5e01A$ zXGS{?KR0^%fuD?C{@dSX!3&Fja%S}Y&pa`DU+ND>m!v&6dim3@jxPBLjkRuYG4M<9*RkVq z@c*}?i#wkiz2))eMhAcM>(Tokq&8~@FMs-h3#ZZMzT&TpF4^+s(S#>{JepYZl@j@ubZb$wrku4%_Te+3!>~d3pX&*X2#6KYRBM$KX3hoP+OfVh>&3 zv)KLCLD%3|gBx}nhdAmP-1w{8Py7zHed?*oA#cDxm$}_(gMVK+bnfrNL*G6W7+>sW zX|vd|`qy2a)Bovm!*|%}+va-4zkW?%ym7YY-Ht5JyX`-44V`)+Fy3+4H8l9of${cV zIL1eghfj2T)iFN!rOF{zJK{d|Q{Z>IhT6aA7(eyd%AxjR^!Xv;?TZ%|G-e#Tw*3U^ zkD)#b^-qGvJ%D=vzvmn}^$ozc18xEQGT^TOJ`DIez&{84CBO#&Ujz6Dfd3Nk#{tg= z{B^*`0q+Jp1Mrste+uvpz%IZi0N)9C3*g@chT87|d>h~mfL{b01pK$S9>8@M>WfkT zmw@X5Uk7+0;7&p1}2O zT)&L#o49@f*Wcm#7r4HL>w~!d8rKM}|BmaexbA{H{v7jIitDYo{s{BhjcW_=^MG%_ zbuO+)a9!;fj~-(`y8|(@(OLXZtF!c>wo4@s9r%99LkGWD@=z7(qq%J8msrcAGuTkn z#fIKaVnY;P(eI(Am%o2)0K3toD@q6xM(L#ghxz$DaIE<5%O} zB_11{OCM_e{_=;~zPS9M1Ciwq9c*6yPyzT;Jgy^NiAV59e7-2x56 z3Ht0g)AR*={tNmXC-}(;pXhtTJ>K!E8@#|5xs2d*I0@+n^#$S7@u#U&kmtWtl0?z zpRB(QvVRtT9rwH6?LXljXO7`(JJ`u!`z7B)hgK#G-LlR-bbA&4PPm6U9(0f2hqmp1 zj=4nKLyc9DYr}!>xEr@P-)%@>|4$QRH(o~mn1I}|BT_xIQPwR+A2-#}IH0)|*TjTy zeNn;bzf5wlq5K3U*5bQ7JTJWLa{g>F;^IaJ?9^FzW-DXk{y27~$+`24KhA;c4qP!O z=XP9OXRPh7KeN@X+FvKzZZX_LZ_d zduT}|`&k3{KeW1PXq%6XFGb#TD64XODRQaHi1!A*9{5a&Ul;>_0QhW)FP@06aO9sO zdfW2YxIuGn<@h!~`q;;shx)=8^}8`f5wGX83_3y7FOy@ho;dcQdw?&I_zf}eCx9=L z_$@K;Uk1KH;&;Tre;xSs62ChJ{#M`vz)KkzeE$IT)w18mWAytu@U;?uAO`-=fxkWm z-IuF|5*@xVuFJzQ`n?DIT@rtL41Crm^iLQ4?}>pg2L3LIKM@1JoBC&n{$GrNKTQ24 z{;M(YKcN0IME_sJz~5Rqo_H7VcDsq7-jO-6{y}OVqy8_b{mhB&zd-F{)PI@UXH9JX z1htP*|4nK?YhwHFQu`S7->3H36WjlQ+Q+DWmfGh`Z2ug!k5PYt+RvWY{^!&_M*Y81 z``n4`U#9jk>R+Muc@x{eO6_CRk5c^ zV=wRtXqSlgN$8i1oH%9hGV;C=o1cM7r%vk_MO*u48HrN%At`(35zfPG%!Ab-0i)83ylBr#lZNb zA5{*$h}`R?7b}N;g`DhHZ!&y#ppK2d{JYAbmw)FNy7VK*(7z*pyON9Xqip;?S}KP| zKTZAF_$$csem$dV=)HeqfkSvwk>|~vzrIs z`XK|K0q`FfA3p|~cLs*uI=F3c>}@vm&XKK_oAmnL&L;<{4ttr? z)1ECkguS$Ta7F+3c7D&Ym#6b4ogTZ|_Jx5#8lT!1I08J^ZBL9n#=CcX6Sn7o?O|_* z^>eJj1^aWu{^ACsuGF7-K`U`tWZj->FPmm z;pRava7zh(YUVj_;TDef^8T5er_*048$5Mv2IuwErWxlF{56A1(0(!63~tIem+<4j z_~~OAV}3{AI|-<#zJtfo$4+fZ2fbQO@ANU>SR(2%F7_#Y@UhhxXXG{~)sMfj5w^3& zF4pswYlDCvmW%XA-apGe*54n zu+!P7%RyZQu4{1h7Va3#!sl}Qc?)+AX5zYtf36$!;JSc+zG2Xf>wNyXe$a_4YzFOj z4U+w=#9#Zu8wb;IE#aSU8cf5rn19|qn1bsv{&~+}60Ri2_Jw>IfE}hoIUvZ|C}{=^4Q16z8K%X_{(XZ7<|li^VnZ+YB={dPu+a(yRXJ`G66RMveon0XzdR_O~OS0=x&X3o!P-BX3{ExAb zcwC?QALm9~Vh`MYLBs|4I?fe0!}qJP25mw-s6t$*M11~%B_@3oFFuMFAH|E0;>C3F z0{IaypU1z*gZLl)Q}^4OzUg2-_RNa{?_>@K-9V|6FxA%L&x^)~a1G!u9)Ib~M`R|H zLv6vsZJj-=4wyp1I+2MvlW}!AnJdNRaJn4KorE-97FR0684SBq+()cb2q*il;88ZK zxFkND`!{iQ9%7yVt6*{9Xg0Mh?rHDo4fZxQ$v^)&Nc(yCE%XYl@-kms~TE+`?}j&I1x##xv8hEp|z(Ih>pVt zZ|XSQThY+l)zom{NU*D^*Pv6~)Z4U$S5@u1adX4UhUO!ohSrWQBHTcKMoSw94JDYT znAe}cGbf^DsTIui-D^|S?V#f+e(U|Z*(gtcKFO@9&-X>Dshda&U@u&Jv}XvbdB zRBWk%o3%7`b{-5ib+>b;>kc02JwhJOp_U_udwQD=_m=z#m~Tr{cUN1m?8wXUJ% zNM}QH8?x7i);6qvM_Sri8<1)@1lxL#VoT84(A?L+>zPix^y-Ko1tTEY)C)Qo=xEpB z9{XS)rB46H?Kmu2oz1$#SsJB}RYS&2b|s3R}?lgc%xT5%Tx>B1?1y{zT+9(y(W z7_U#6Q2%#MqLVhE{)eb{vJ`glPJt?sEec(t-WL`XS7d!MSaIv?^mwGz-? z)MxlKe(n-YvcP%)J_ZD6P+Wm}|H(9?UZPnYZ*SK7C8Af-Q3U9$h01-Xcj4T}UZN=c z+y)5xOa=4lr)ygBUEy#M^G5Fq7kT+d(Po*Qh=i5Q+!d%0eck8s1(N8km+j~?6&fcT z2^ZOi@kTIg7GZplx+o{8j?Lpd=E=Htc~#>oT~RgB|FFk^u(3 zcI@6-y|vQ6=}52}wj>H2p3ynlYWy9bs_xzp{CyMFSwEdRb#yk7bNhRb9O>*?=125w z!x@pkbX9TLGXHM66qm1MRYyBITm3zKJ-uyR{;s1vz5dp=109Fk{5S60dHt@2-J9!o zZQZ$}Ve{S_H}9z4T+Np6^sn+SZwj`w`+K??N=w!>`7t!{O6P942Vvv&5W5HU?iF5! zXQvoF1no}uGMEl8^jUcxeWo^4+!^NQt2zF2A8^AK(Ql1gj5muoVr(UQ-Nmk?Gq;3G z*%b?1LL6TyFz_qbMGIWap0~iu*?9}Rj6Gw4m$G3Cyo8;zz>C>g3tYrbS>Q$Nqy=8c z9QZYYk_lFzyi-^ zC4egp`Q|Xc1ydFpH4K9bf6f^B( z>?Hoqw zhb6pM!UGa+l<-Ljw@dh}gu7M$RKX9;FVO+(Lb^cnr}+@vjpdYJ;-BEgbO95;1lMH< znD`?&Fhjt^55dE^0%j8Ko+DtdgwJB_B>FxHUn~$X$%p#;uy7NeKEKpr66cOIXR%f2U|K=x4BQ z2@Coe?5u^7^j4z9X+?q8WpW@O9r8rq&1r z;$qaRT&&twFG1rjqN1bfPfpTa;Xmwaj zm*!L!2?ZczQN^B110K#!WdZ)an93ds2C4%17#)5vjXkr14TMqxWarU~(KMnz5Skrm z*WjFxGfH8$JPU%ol+PXw&4`jr(z6_1_RLO}9Ar*XNO#?+3;o@XhBnWj{4weQ$Do%S z!W~wQN6z%~G**)XLH|ey!K_w;`w^c6$mtLD zsoA;#r?ZDciS97)YCaE#GTlVbggx%C1`mYL5BrViFERgTXnt|-BCKF6F&;Z zcyhBuP# zIADXxk#&5n4JOCd@pU%1$OiAV!Q}XQ|8^VPE#X$ip6rWa0Rl`Gd}#%%S==iHBJ{r^ z#N8QnfT8bf0?|+=+BZ@N?y*XIBMFAJkw;is@d5rH% z)%X^9{%acBexK?u_h0pKs#f|0F-Y;Ba{mzsR^SVUwQ%vpv`Eyi8c4hM#Z47iLm0G5asD;Ay-pnmr#+ z=nvsW!f{+)3_1K=p%ecdP%(hTmu&4`3+du$2EC4W-^W5TW?*~6(que|^k=Am z#>aa&w52u-yun|8I;#YXBKQd{()opbFSPK7^%~+)>owWe%mAe$iXQx41oDkFFdSZSH>KxlP!n9?JIpEB5&&d(YPKQ#YrWBhoU|3ZEcVa$JFb(l7b zhI|)U`VWLE0_?QT?=>^{dSSF*%jAwr^1IH$FU>#C7(W5)O|$iTEpG zq!5F@q5%7{Iew&nG2g0jJl0d9U#R-O>L7Sak%BLAeQ)3gE0P|yUuNLrc>59q=JMIb z4@R{1Wd=Thx34f@@Ou>IRsw$0p)cgel%G+02ZP7@i~QRjFZ!&Xzt3XNkq}O6wRjlmuVU4rwUoa#9!C1FwZyl9(8d{*Pni1NVCw6~X(WG~ zhMDpwc?F0i^52()eAg?O&h5`qp@AkB<~TdH!eEs}?V@zv1(x0IumflUS1RitH)dfaQDx`g|XsX1*yl z{X)E|@b!((AN2VqTlL+KS9pl_a=zC1Y>&SbpD8~y^{d5WG2dj}|4F}j#F92&zXo&t z*2?*|l77S5d@250&l@5kLjmZe=6tpFLCu#IP|co~5lei>v!0pYL8d-!d00`O@m3&v&&t5R&t=u1^EV*Z6so;h*W+dP4Ejhi?n;;$_cre@&ebpNR#UANJqm zuRaAM|J`ovk7>VogrB!jKGj`!g`cMkgp!f}nCs_2$Qxjf%k_rN;}V9Y|B3x-G4}gL z!F(O0Mn&qn&K_VidN)si1S z7D}@0uOt15w*CotN`e=zAAbJuh%S$e)G*NI`01ARk^UL#=?mFEGr*oO`5g$&!YeuS z{xdE85r5r`CMfOo;gH*d^h%DGkMl~82kOzr&$8GP_D=zJR_Awis@Xpc^3Ab~NBd1W z%2DGDgjOVxxv2RKgpd#829gQqClPZBrv0ccuL$BZ`6G&)zdyj9(fOH|YURJc)_=Z@ zpEXH(el-v(46v{2{47ec=EDnY^cLFqDdIQyO#T?`HIcw??C>#ZlliMKxln{Zg(sCly6z*x4|M0(DRu4YaWm3 zc#<(rKOXa&k;Liqcu4zcUx0m6r=Kx_UY^n4f%fpfBGsPOt67%$JQ~Wf+<&0?`HlXx zKY>35RR7}?Pd$YOp7_h>Ujh^JCH~+q--Tl~6YSe<+5=y|Jf^*IShEj-H}x&>er!JG-;=Gd1uYKMoj>rD$jTOMV6QXyT z_(&*&(ajfLcs??N1p+K!z&I}kIVwC}KN|9Jetg6aUvKB-k~|E54ka;ffSRKK|8OwF zoq>-EejwtLGsY(!930WIIo=tY}7L* z>DlrH%lSV1@8A2F#~lViK0gPCGw!v*nFCfh>k$haPyT^>ehG9QeYuCFki75*`N;hb z?U!i3%lCWV_I`}Fr~U5E3HgwPKQ7-yt_?9?@{c6_ybEx=CrsnXd~h@JH!SaXcm??v z`ac>< zuM>2C)S{0pH`$k|-&wZy*)RYTeA=%CFIlhI&&2z0cKVbLv#_bZfl#`3KAFZW_JaEXkA_mXeWV!n zlFs4u$?YY@ZZ8!7m3*O(bRAFUF;;uYve`?v&0g&K&$HP}s%9_9Uqf?j^nYJ_$qWQ+ z_L4P>32N~o`xyyKd&xU*h276v;8dHvWPWgav7MJGd&$=AW!Srq4=>Kw?E9~Y_JVw9 z*wi1L*X!qr6YV9F+efBhFEcqjb8>sh)a`-d3*`fvz09=67rXwmZT6C9vlqMm=T2iU z`5&^qWCiH>N6m-FkBeLZMtsQ|k+8IvxmT>Pd&~mc?Ir6&w->)|FV*kUUa|r_-pcg{ z`&Y|({X~1o;`Whc*vl*q&zjs`vh4PP0S);gzRc3`6a7EWW-oJX_F~t+e;Ru!_>k=- zJ3z;0y1nFK{X#M9Wv+MvfuARnJ^6hSFYU*jZGr9flKr9EOOb9b&vEUH zV%LA+H1@LKL$;T^z+RiZ%=KIC#b0Ex2Tx&%6?TjBGG$M8d&&E~>;?H2AB{```KKkH z;rrbgGxA9v@_9JA-;A9L6)zL(7nVoez5*8enz(<-v+ZB3@t>Y|q4{w7;O9ihXW4%Y zgc5RVfmHTk;TP>64(7RO|ETcO?N?5O=K6H|c{t?Hb87t`4i!>B(&!~%`>yg`%3ty= z`Km=e;K#CFBcG&r;D$e;81W*nR>3Zg_tz;nj>Cm}C5-1A&_980*NFTT_w$b8_LkbO z6w?0jb?*%vPx(Wd!@579{YYHy^}w6*NwVz^bbZkN(;AQM@`n60e>=|QVa*ry_`DPJ z5^VXO-9Hm?yrBE@)cIhKd({ASL_$&wkuf5zgxmD3;DZ43eVvLbN?vi zZ{+(V|5JAPYxy>v$He8&In<=Z5`bX)wP(|%JP6XmbxQ^+qT?bTjj6iSJTTM`Iyb#t>=$)o@Aa6#j#dCUzKcq}Lf#o8RygyDlCQwK$1JcWZ-+SFX_Wa6e_o(pllO{g&j%@A?#_e) z===!Ji`U%elV2d9^CUXY5$DHV`hEp}-VpmU#6PuOtMi^@j!(mSg8jaBKN9ghLC>cj z4axnQ(LO!5(W3tpZGEEnlZE`oTyGu?$@2l(AM`bUrS|D~z(dc6W?A#0kjxe_CeJTJ zba(OrFM$LjzRuJ8p9p2+_L4dNiBO>ypDDgt&lm7Ki059AQu3kmaif4fohSO(14X-stnTJCuSO-+79 zHlg~{dRoFBXjd@yGbwb#*gQYA=Szi!y1XKI9@QJ*y*WR4zRzSny`pSP341X*2oYUx&{Gc!tlAjCv zt<|>i)|lgQ`?9W27{7R7YNnijoO%9h`lrSJ(SAiy#!5q8fJ>IR%M2Lj_ZI)h{aDX1 zxD)cB^U4JpKPzbdrCEuqWq*FZ)ANkcUiepq0q=x-W?Jkq(!WTfNA{L&@jv*x#Xe!L z7JKCS)t(o?{>0*Mum>%E5dRCs`#$-8i^iL4@h5H%Sbvp0Ow?b2=e#i=zJExk@U8A2 zMEZ*?^y&F6-Jd8vJ+Jkr=X-_zY5#}#(7)8uUd8tmO`rPt2juU|N-dP~cky^;p2y~s ze*D=Pc*czThXeDq>KFKeJlwGs_&AKe+%i7!^XDcmk$Ae_?DJeS_+J5fx_o)O(DX(5 zatwWG&r?j)x30gPHvJsO__!Z}^?P#t6naJs{z#rhY$0G2E?@FDU0-+}+H(a+MUV8M zx5xc^&zJ?j%z}@lAAP+Q`cd-7l71&E%+4_7P3upVLNZ;?jNE`QQbwWc5P7gK)@(&v0rKXF{YR(&k+lvwC5 zoL(Q=^EIRozooz3-*kPD{B?ay;y>E+K)=5}3Op4We^;ZAsr-ZDW35InmOg5&`apbB z{>1g6`3u)afmI)co;nNttI&t`JQC?6VCnxM=)-t^$;|gvzVQD2!|&Ov@pm=)nA$%I zJnb62So$!}5BYjC-=+^7n5y{2^7&l-ANF`<_YbV^`ub7m8L-g53Vo>O)ucYcmi`}tK8)w* zJ`DdT@I0dNcQyK$+CS!dPHFUF>ElMLKJ4+z?jN>zwZL=MLjNlCF>Kl|`Nv61|KCp^ z_ISkeG2{8eSo)cmPo2~FvFBq~qmQZlgVtj+{={KDoGPE9e73;zjK&YG_j_&nGW~<{ zuR_mx)gJj5_Q%t#=h#1~_BbNbRlKa{3>-t4LY&@SiG2gZ4Pxg3bm%lxq z<@*(lf4hF@d}%8Gh#`M19*KO(TpvgunYre9B+b|F8PUe)`MQEd{;S0=+TTpCkBh2* z41HMboAjaQS0rEEo=HD?zD4U%foDvkZ_ z0_CHoeYz&xU$oqB#C+|~t6A>%$9P`>o`;-m+)u{y3ijt?>3uG4_)B*)zF!ifzvX$@ z=T|^}+WD+{-l-kur#|e@3hDbeubc0~!1oF5{9B*DO=TIOGAI9jz%bs2f$z&C_p`Dp z;-~HkKM#5~NZe5o&)2?+_eF@BWqd#QYG0vzAiR*jZ^k~J6~^)J{`e<~N zd+~?ww`7D^Iln(n{6OB${&E%|JiR~U8UFqbm!DuK@OVE#VF>R!16WQl+LwpQ%ex^C zM(_7P`*OUmB&=ZIvqI%%H+s;XXaJuTD&sKcg2oH35jZ+ly_Dv42Icov4fq-pkN1m| zhgO&Qh!%~9_w|&AD$4v8cwJeE20Me-n*Bk)JhUDUd{Ytqzw-%JM)Hj&<7fgJn1!r9v>AIJnxg+5)eMUt@vS$F1csL;K^b zAn1w;dZ^_6ujQ|q1;ptU1k0j=zQUJ#)*1cNsK1NO^LXL?*T-KYAlZL)RPcjJ;0uCl zqJkd z;imT!6$C3Pq&y^lYhC($!OuDyKkIe=BX_u>tP>VSX7I&SsqYLI*EjTu^#?!H1bsG- z!9I4n3Fh|E{1({@hq=C+-{JP-4PGPX!@%!x@b8xMA(-Y9C+7ng_C@?lc`N?csue}Z z+p`W2u!w^8HUB`~Nk!bhi`?OkvPN)7w{^gOA=WEDKJxcBVLZG~ES>tR=czUXE26Mi zM&C-O@iP$b8wsZM1oCwDmw_=9uu3rSe7&h1qxB-6!(qJMpp0OwhYLdPa*~Ie{OQ@9 z>>j)yjp%!3;>CL}EnqeOG8Ml&u7cmSQPS@plHav#k>qy~tGPTCzt3F-zaIW-tv@8c z9=1dByI91Z0U4hzJ`nO#)CzsUe@*+x_nW7*r+I0P!jNYgdlK=$oNv6R zk@RKQlgDOH@$ry=i|r6QE26bkLU5=2Ac+#yY=xb_K5ErS>hY?T^RCC zV~^#0Jv8kx!P|(@OndY)YkW&cFzu1zT|xrcqt~=Y+;3L#EuY7?a`t&?j|Ef5H@uH- zdVb@hq@O<|zwzv;tBY^*z5ssXr;)EczQxB+EnnT=Nxs70HTi1(F68TBuW0cN)_GFK zH^>v~Ym!(mg+J4JO!j5QJ8x1AU=;EX_ubf!v6CuZdFYJOXRon#&vfr#nP^EfDebq>PWv6rWb7 zYWb+!8%#)*@rdRJd6p%M_#yPcdrM{cJd_ALh zq4Z1fOphPnCz<=RLEml3cPV}v`a}F7d!bZGkJoqr*ofB@f35VrDI#8TdvSV$NvR_K zav1YVNUlbI#yAZ51d;=+EtOqWx+4;#JvSLX_n3;n|;?+n>l6g}*1T3d8=U%olk)O_;`Cyit<3 z$!|hJBf*pUOYHb>`ipx){P(g3$?wvs^Tl~{ozusEdp!N{*DDXZ$B^&ceUQ#s84sJ& zVb5#Q^!+9I2W@qgzhVE9Ci4dwPmyh%Rq+AareIB)h_}N2?fXN-+cX(3Wjv){w~&R$ zPq$mfYZ*^BxsCV=f0*8Wr^`<_MTwqi5AP>GHP@@y`KhqqP3(*&U(D{Tihs0Tt>>@H zV>~}y0r=W6wgp3oW$S% zqu_#Ix$%216kiI0tE0w#ox^LQGJoUog3b%{^$h*Xt?_I1YCS$e-)rRfs(+at-|p^% zH4UqH?!x@nWypM7#=C1~$n~7VWPkL&50yW2dmUEiD{l0!@X35%#>X{l^!yk6ms!{I zwdry{z}rXeSf{POB#-iPK@WpM9`^k*@k39Ps`Uc$Tdkir5T1@cHGiQ8_^Dt#I)C+9 z;uZ4$47q-CesCVZ_vfNLtq1Y^eIg1b{tSNse;J{4Un7t_o(cV}k@TeeSFe`qExsm2 z?LFy6K7jU?{cX(r+RE>vc;1%indb-g{95COpC93Kxy5Bz)#KkuRR;l5`dTeq6ME%b}=sXaYGE&U7fD9Wx~!TAIIWSMMF`kggP+9${JczE7Ce*i!7c|mj$^YID(6`YYV zlamuLx6kv+K572e^Un=c=~0rGslQC$EUo{B`iv;akKrpwrQU2|U*un; z?}Qv-Ps-mCXAAu*ct*C=R~&mOANx0v?}4P0pU78DI5BFqkMwLA51^0Zke}D9%NzEU zEA>zQgnT|NSNgluZ(NS1ZwLI@E9_H@?+vEY3kayl_U>HapNfBPj`TN4KWVny4@o#X zN5&`FKLsyfqY~zDL2y>GK3+j^W`gdY1;H%(LZc|qACI3GO?##DEqy%!|1(J+ym0$V zp$7;InETH~(|BUHcNKt_DTG6 z`{w>2kB{-wQm^KAYz$L2qCt^VW5*X$AYU6v>P3HHVP zf$yKVg!p(FbLBT&Xgug^M!xhH31{L3^HjKf!XM(ix_-&twDrM>{VG2nSM*8#$!x!z zzk}=}PS8UOjEDH>w%Yp)kJVpO=90cm|0s)-^O61IEdEdNDF>~UeQ^7}V%oPim^DZG zr}VdsI5~e#&l~jRlYOZ1kYCIZ@tNjN>jS*&itcZj8M;5w__=(1eqN0E&72|DJ5-AK zrd#uaS#wv@a5A3o_2Y`Uer%|+=L=c#`!eXk^_873{gcm!$Ac^S`jMF_=7aXqpUUui ze!?Gwea{9;$qW2i;>CvgOfw$9-ZwPQjFLRK{lng~Liw3wzm$X0`hw>RW)jS0jPW*9 zWkpGUDqfI1a?!Ppang8YZt^P5)!0I2l z^K}2<@rL~`aeuOz*7xlBn!f5cm+_OvhkwnUJx6O_5X_0{@g4l) z)Q1Y$tGItj?UjDL!MXEfeB|>Zdb<5``)1>UKTyJYS0Me9=)wPUa;*NBoy&DaLvnk} z^_%N2t*?H+w0~}&@UQvWdW-cveYWoZ-2NF}VNXOZk$iIVbon7Z&6n|k(?fokqx;u} zdV7A9o9idusBrycXItgxv*cr0p*cKX;%%NL`_1F}Q}*i(=FXGxg5RGed!EA%N&Ae4 zyfWzvM<|pJP`)#Bfs7yAe#qVnfR*{Mo9ro%Rm=XEHt>Fqr~A4&vLS80DT`!&#`}{#=YzEJzYX+qbJ$JB{1CsE%J?DWlTsw(74*3R@j7F%j8|O0 zd_C~%>*dTM84slXy-{nt@R`5z&F7byuIE=gzWdGf&l_C4So*)D=krkKTG}+ zA+xH-X|8 z=aIv+dH%(1ocsmxob;pSOZJ`1o|XJ1@b!iGQSrzd%wNHCH;NCqpGEdEn|;YLpC!D# zl6Ty48NXpaE8q{FVi}*MJ-L_3cu%lftOx&Lp|_ONQ}HQhnaqdLp8P9$xr|SeUJ8Dd zk{1qRz2N&VUc=>S-EY|Uzp(dZBERD62kl?t^!%Iduj%oS_LF)(NBoHR&-=qaQi?^s z!(kD>OQe7B^(tqX%tv4^$0?pK*YYRWm)mW%uk>OgpTK;0JTH;{fq4D__~q{-XYzSN z>aWLdzP^^oc=Sc0@A2sQ*M=(b{!e8;_VdgQ^{M9m_i3^>pZPo#?2pHb66sI#C?85& zA=eA8Kjc>_WWOrDXNA&J3FbUt{$wvccEF?dCm283I1d1x{Hw%bKQmV7`JjmB*+zeA zU(DO9_?et6>>ZWhpX@!IZM3vs!P~3#FF9F{Z^)-gMEs-pPU}rVsfZ^k{wJ*x{;S}` zl_EX~nD|-F>AxYLCrl|3@t1)f`4cOZ@eeS!pH(tmaQPuVSoZITXZrpd`b>)I@dNT- zDdH#57xG)i#~07T76k3*XRZ>lo|1j<1U;|U>L2`Pa6thD^e|qMZeQMDPO0!8;UAFC zO0oVbICYh!FhjvzYEWw=7gAt z@AUh?IB&=aI=T3HkfHla(2rBl$MdKw==Xxf^Sbro`RmeNm-&0GIM2%g9?j|g7Ng%U zu1UC=Kol`Q&JP(p@q_P=5PiaOexA+e^a<`J`jAgeLV!RN(I4LrA$CyU1*9MPJs&EL z2f2Ue^hq?;%_I6!|Et9PsPEwW2ckzRzn}JWkIb&9eP=2=AvwnZ*PVk@2 zJ~v(ez*POKr|(}Y`?uQ1--z+Mr|(}k)%c&EzCXQCl;ro4ef%iKe`fmrO!hCd_s8>b zeHYp1r<0$0-(*brm$8%6_4iFR{@14OZ@lj>7XKyevFXP5Pc{DI)Au*tw-#&sB6e!B z@!|iL_fy7vf2#HSH#L2azkhm4KZWpT%lCPrj+(x)_fOBzS1fwf?D6+IUqgSf=ncGo zdWJq@(fi-;pPr%LSoDh6*~#XszTem=?JH({=(Y5}xsMLMs31#5{H67fAigNVUh4YZ zarX1^p5pP)-kyHnoEM7cy!{nR`8;3ElqB!Q14 z@R0;QlE6n2_`gL0be4&erQx@5NI3E&a^08S-0&|q|Mk0fUW~r5)Ai1sZ$_Wo_~gco z$rq!)j9$5OY~wrm@BOeQ8Xb#9-+ud!SHJnhui3ZX+v6RGes=b$cXr-#a&+V1UC%}z z^DIF{`@1td{R`sz|Joz}uXl|;`ByL8apvMXuf9C@@_VQ6ee>^E{_opb&%O3c_X`jG z^1Jh&`b5n=uYK+A*M4=%bK##hrPV~1INs=ZI~sj0`rdzj<<1wLM4J8No$vnoh3K2F z;p3ZMx%1t(?~LNN=iiLJ_Chpz+nYeY1w{1lYv>Yv;f43!1@JAP-+nUs9jbc!2z9*k z8`S8@@4XiN>8*eEEneRI#T~D`=Wrys6UX>&p^|SJG&se9V=G6JC-E(gzstAG0#~}- z98LwEPOn@JJT%8DQ|Bf+lM_?l;&<1NIUQq82a|`;d z9FE-87$ti+G+5#t(~fW-86XG#!l6L?=*PY#!l70eHwf>$A59 ziZjJThJ!{?2w&OC5*;<2(ym2)Ws5tJ{+v7RUOC2E8>#*GqQg=A4iS2bHgu{P6E6*u zj)*I3ZcQZmmDOy$=^G9QoxnP_CS%NiAclV(H4IkzcN%Qv=>~_RhP*kywqRybmVolYCx20Pehr%mdFC~{2q4)VXgCLFMJC>m3{*W z*cb4zhz~@K`&CxFJK$y!cO+fMTj?hdeRYF{M-!cX9EOe%)05Em1UxL_nSy?p=npIU zJlg2=;{)+50-WLH{(RBJLffN==nVf!VAeBW@ zZS*f8B31r1aqOVGLZGiAQptpfh2<3TgX@YinTZVnwpp zxMloFZ7^ayLu4Ia*`SV}V*J3w`ICGCN<-Z_g7b~)^f2(~?ZvT^vZ^Sqg?`URN`OLys z$Yy7v!&s}1@$ArPjfECHG?JW*%+ZC+4--_6G034zJY|L^US)>P`FT86^CwAMFtLcs zX>v(&)bLKcqBr8yq-ev7c~0J6#OlfW3qP5>zlgOQvJxaa8kUq!%6Y~7s{WnGhjES1(~K9!c*wL9oZ!;5m8m$O73ss|fW^IE4ropKFgXanZw_cx`Y<_Qp7$#Q zT9-al4&q~R5W{0AI~rD2u6vBlYrNP^#82)ut^^&g=v}SH*zD876M2YX{n85+d^O%U z6>YM8MT=iGPy2molU4trXp_DEp=gsOeu%U!yMsp1p47@o_L}@-@<7o~Mnz{!X69SW zSBIlbhtq?c2%*pAbTgHiNjx4+9`-up3GW_u$HcRw2;rF{;Lz|ij!7esWk~XgnE3b2TgbcxLkE zOfq<%)Yc}GTTI>{7jAoGTd{Z>HsuzZw**6OA3ATEPGj;mo!kO{K)FFKQ_2lGS3VhA zZht`Drqk(ka?^a_zg(x&$?d;Wr_;&pztc~rliPo%PN$PwY@JTr1yr`;PJy;J2_s7t z8R12`!w~a4uN^D)#7wB(Q`J(Bz0d*dGSt(~Shx5d#`vxP?xv}`ZQo;bH?1nU%0qvZ zYcmt+Pr=5u!p-qjbk+Z4zX|QAUT>#;ru$-iR)cg-V?U+rrdb)>IWok(KXs?!n1fze z?esY_MK0m+ImT$&P?-f$wBV5&Pl;zS@$*vgJ9$;0^gR|}v{y7s2|y>u*Q3(qu;Zz@ zvmzDucKrPC$BElI3}+L@_fpX71N6sHuPlo`P=88ZMA`9y`VSl5yE*!S`r{~Dmd}5n z{=)|UPko^N=KNo=^mhhq`kkOZtk2M&aTfruFa1FJGpwlM1NAq#Z1_O^&H3N+f%*?4 zSr_Gu-v3x?O!>Wk|1j<2A3-1f}MWMXMr!V;+cWpD)2>CeAvLhE$G|v zYF~`kPsJLa8F>6YRIKq0|NL&u{$b;c{#P;iV@7OFcEzM`*#BJuZk`ZwfvB4+=v z;eSH{Z9kz4f(~##hRbVf3d*Z=;L;&EH??fjep#3m8Cx>{b8g3voYz14g8m4 z(l_nvKVstz`#ykg>`nAPbAH(ZZ=avhf1|+rto~=(-)96q+ln7n)5q5dV~ua1@%2KE zw~cScHynk$Xn`}?c?&#) zJpoD9&z&qXCCmoeg z078WWkJ7nvqs)1|lerpP4tHrX^M8}EM*I!n&%HR$h>R3X?8RM_$GX8>0LfQ363C1A zD>=v5F#UZCOm7P__u6h3q#?d&kGRVm0g@(%X&omxEMZzV2tFxcTE__1* z5)R{~Aq3O>5*?}RB3}AIFwKWxdZ`DQ7dycC0A;+Np_vjt525$s2}Cp_^(uy2Wg znQD)hh!EZ@;qDRv`y_m2m4Hb;)W3U;fJq(%Us)$$nm@rOHwakJ&tN{-i%CC&4Qvvy zpbz{O0So%T*9utB2ObLo^%wLrSa`dD8|8R)JpyhA?4kMLd$ZKOTf$dhO9Yq5_Kn{b zu#)F_2?r!T@Ew7#mGD^!*Gbs_U4d8f9FVY>FLts|3cOdg@0PHbFZ3y4CC|EH(O%3q zl?_W+%oo3r`aOXc^fTD7ga!Q!7Wlrv3;MuISjjW+lujRb2@Coekc+@KN`74xY4{<^ zLgvQ-Xd_tR+EBORuPkLFji>c>vWhE}_dA;!_ zI0La4^%*{sAOHUnD0(S-fzb*!CO@x0z5nDyKI>4wT&wrX_PvwP(MW%wkEC9RU_uGV zniliM0k4tfV6>-)A!GIbq3m~)ahM{y*!#~wHB>X9sF$<0%MR3rp1xM>D9P=GKu9Q&nW2)IVTjqn{>*;^+U0cQq03_WoN1zFRgk zE}xhEP5%}sd;fY=-(zlmjQ-Ed{)I+O?EVEpAl(65yLV*&0Vy2PF~t}A`1hg$Bqs3x zK&~L@7Z<}uRDY$%;~-3R0ohJn&dUA@R^zLFO8>W_^Hlvqk^uFmcJ}@hM@R&4FG~sM zN$@UlrT!Gx!vZw^9YJTJxyh)JA5O|E4Jr$d&l*hC6tAd1(YKV70&28W+Kddi$3Tzx zud=mcCq+d_^S@a7>qgu{N=Ewg=_JB`BcMMk5?7>UeEyb3sKXH-A0HnrsP}#<`qdG0 zTCstQe)IcSjPdJ!CORqmr}3%S#K+oX|A1}$2V{R`zto>Xg1tZeh|B#k#?Si~G)D3B zf6aHbss752wxNFn{R6h~1G4{s)}O{#{gnOYq9RuR^RmCPU*exalD$9uND@{fY;vvg zul-jsuwPt^f9>+$h5nz6G5>biU+E8K%Jpv_pZtboQgO)`!tz1o59+G=6CWS6A54}R z{%IKvz+JLJUiI;aSLGM7)r3oDlvgPKQdiYawLVz6Yj@4TmP%1JZLeC{ zd9-C!Rp-7{yKcIrvh&8KJ*`Lg?X53u>DqJCq4FD!G*xcezG-(=drQ}r-u-*)J6pPU z-PpPHwyM_h`XiNFZ`--IRKp$W?QZVcg?{zdSMJ+$(@ovGYC5-EchiCWHypX8vi-pR zn%!Od_U`U!-NyUY>|MFLx23keymQ-5jNN-+-=6ZL&8zC$TWdORZSLN+ee=p4rL8rg z_Lf!KkM3KwyJY{a@{+c_RT!fU?f2ibe^2@C&8x1X@k;k~9o&3N<&IS?E1Pa=s_Zhx zZK^CiQoZuv4NX-$WScEzoizuywRhL=>a1OP({(imcQjS5J+isx;Ek<&wjHV3dRui{ z?T)T#+ElgeuRpSBUlZn7zqWHz`;qEB2kV-us*Y@4wY#IKa@Y2%RqY2_SFY{bU%PV} ze)g9%?J4bSIXqpv?xU@HN)NQwtPRqD#-iH8#zfgSXvM$@{F_vCZtWsRn&k?(f;RXVQLUQl7Q@ z_HH`>`#IVKn~5c}?S~~Fhj)~;bnR|$jw#zs+qcy0DBWMP`RKl`&ZGNl@CUotU%Rcl zwQF}DY;x-jrMqu$t=-nSZ_i52CUzfc+Ead@DJD%*Uz_&ScQ<#gE!nqcThIRK+II1_ zN1FE3AKJfX^)&hwF-@@CjY@3F^ zRxTgNeKP$kzYt?jt#dVYZA;hM+xG8XdrQl7^FF-o)|Q&x)9R;cm1wu`@UCg*v%jwy z@oev^nrYf~%63(^Hm~fPrfs)uyW^I9dqSO)$lZ{WoIB)NI-N{6x75NnCb74j%0Ks3 z!Ph!NlZ>nUYzxt9+H*AaT&mk;{M&z9OXu2PE8=v_IB9P;SzNNV>uFv&N&GU}wbQtv z);(K}Qv8qQ3&wm8W4^1ZZk^0_jCQN29ofa6Y38z9j6I1h8uYlFdsj^(=Nc~Oy<4uX z-JYwZS2dZfDE-QD>#wTa{@Safx4$NIb@Xc5udZF~)$xa%Bv^=@}b>)vfg_paP> zw7CoG%3iD)E3sBoZ@Yb8N7=P1A8Y99>-=|o;uDxlX>lo^k=kTBZm-Av;<=7IyfvNnQffa%jS6{7@eghXFeMAB zd4x7HRBlY36Zow2bMF7w^XJ}&lfRJqSAsINZ{-cdKMzdPp7y7o)7odNZwGfL&uKL1 zCq9z=XzE}1{xak9nP15ItL(Vi*>PLN0R!>&k6Qn4s}0)CsTUoS^+$W^ovhm8bG4#T zo$C9n^9ydmFVpabOf>xU8otQZpZMz3@ow3maf*4N3H5vI1N4*BCtQX>%gz3SpRwDK zczVoUC3Qcso>+2d=gZH(D7rq0c{#MxAASGk;yYC95chJ@N*GpLtn+Kpo_4Ggi%T0< zVMTksQ&m;`D(bbv9Elb6bTVZvp9Ywr$w$qrLe?ukQg-oAn3t;4Dcy{Eq7#sn%70QP zY2N`s$D%W9lKN{Wsoyn8{qv|NelB8sD#}0SOv29y4o0XwopVu9{<&e2_N{0Wi+`$% zRsSC75d8s(RQ~%(w5R$yI zD{>1$W0Fol{y2lG|1|2;)k!y+-X~3F0Q9MTK*Fy`exAgY>ce>CmdX#rh5Z&+s;`xC zP7Z6#ZwBZoTca+SvfoTweTl4}jd~>;s=8U$=b~Otnb-7d_3~(%*PN2|G_GWo*Qj%Q z8dtBs5ptkH^c8Oelbuts(`lFWb8U3+R0JTg{-&F_o7sgeZC{I-cILrS#O{3I~pCG|2*keL|>n8g;uYx&2^L1d!^A3 z9o+IaNh<;`TD{&rAnn^uKcv;`?f+4$*XbuqVsq}GH5`>J0ivVxvtA0zUa#ijMtgDvH8!nNR8)O{LyWb{tE#^!{pS%|y^8nG z+v-)E_F+6KijIoMCANAMkJUN2YJXAAzuVSc#p6@9dKHhENVYGEXFvN<3ZJkZUy=|dhcC|YeBGk}vxTBZxPputIooy|>4cND|1$!HM zj<&S4_4KgfX8iNMNFF}gC7A7K>tWMxoaCmW`wqQ@kk;7699nC?=q^7M` zh=LU#KGNG(T(je*|f!DGW=YJdIpySG;h(QZG|dbAV6U*(RHYo+r@Q)@%-k%m^z zn3xdQ6MPC#%tfCe_D*I4W3RONaLbX_wi`P-+gjg053I2DDyH5PJlNCFgH0bR?&%Ho zHZ|kg*Tt{FBdyrgv10zOFk+#->oG&x@WF(6lj4@1-qO`q&EX&azWeI!;g9f8L3nph zgSh^Y9x8~)|NH&>4Xisd<+2buOA9N?CuN?aka9UtAfs~zs?&1?LI&s`ly*J20Ib$U zj>JEU@JA_G0DpS9S1xQ7GGC)Kot8+I1E^f=6;#srav3v7sOK*1<0)8O>3J&ZPqg(i zEcsUZYlTzeD>#e~RA-mrBXXy2Z*PZxD#y3?KPmfPug+oQfQoMkvgKf&__vqeB*v

z)$c*b5W_&i<>Z562Ki7;i6Op$0(@tk-n>cjW`}R3! zSKYfw0PXkY&A2>Sm3z-V-`QvHefHVs+*9}7dhq6Dw|G4s#mS@oTBUPLRchbM>GQBY z4XZIg0^2i5sU1grrF-7T=cuoA>d$ff0H0sr9MIy@knex-=eG~M)c2cQHT>~cynUxv z`1*e9Q#(%kJX3$|Q+s+l)s9JdiB(5@#Z>R90i~*5rt%ew`ab^3LevWn|Mr1bcK7=t zKl|&_@cv^c^Ll(giJ<+y?T;Pa^TuO`PkwLv;qA{qcKBIuco;l-@Yylt&F6<6sn|P| zTk!p%Q;&Xs==iB04?Xhyn?v7Q_SVp;UVnb?(9*}>8hYfz9}gXW^uwV?zJCh1>b}0mA3uE0^V<(^ zyKe8$t#3U))ZF&g(4OsY4jtd{;n0Zc_!fAHwxNANig`A%&3{!sSquMX`&Tg}g(8k(B<0Wdf8^|d~Hc+Yi@ z9iH;R_l91-?)jnD-b7p1y)pD!-bX_@za&1__q~SlJs&=HxDWC?vUjg;>yg&ChWc7R z0G4!{3w<6xJoPnTPJK^fwxO+@Ck|?P_OzTD%APW>@AzZe58u|% zzkh1z#K{kVxv6jIvd0deIQb|rdx!2h_WaPckA6IqQ}I8CrfmOcXzJFFfSKQS{Q1YA z)1IO1-Fw+f#aGO6J3>j-UKqG<$RDi`kipRoMqhFO$_DFH%4G7cYDSWdpu(dyb&&ed{azA0?{~dc*rY9x5~RBi_Jm zRCnORWc2~iuhjL@bw=|o_EkD27kPe?{5N0U@oU4w$Ka>$!RL+?sJ^3@disv|0#iRI z_4XZ~pvc#r`rYL4Fz`FbKbgmGANZTT;}4bo{}tEU0~?J?n@N!MF^hWb^_ z4Qlm7;Kq&#?=QL}JX{`h$e!nx{Xjcp2Mu!UU*H+Lp$>IfkIgsY-0D#`59g!kae&I8 z$F*ql!2{Qx^vt(uxms@0bP zmq|K=96A)cb$IU{=m1(ausOhnJUNe+`olw1Kc&|q-L~$N&9ykUdcAiu&Y{?O6SyHpnK>X`c;AdxBsr{Y78pp}OmGMBk!_S8dW z&{*cFkU5eO9;yKkmigJ0I47B5ySi04XPNK7xwUoqmvL@oKCl|+R^~c7hs;wT^8!!t z$!CH4cBcAOFWD7#4uDR!_1vje56sHx3nOlvxI3e7*q6WOJ>S@=XCj$>XM9C_e&Z|H z^YK@*`abq$?)gP-pzmlwcHf87bNeV3_z=fvPQ$sLgLt0wBEJxGQiIa-(^G3P$0?O4 z`ZJ940iSON)$Luas*n9W=C>7Ue<{XG0&)$YT`KFpLsjR12V&Xb?BbqPzy5`9_}71Z z*^XlmN6-B7*1ls;sNsBn@2VC4?W>+pz0oN@>3Z_y zKW6Ped982%;N!JF9q?T?^|;S7j9B@@WwS2*i__oB+TT0$#JuqGBmU#QaeXAmk@vH9 z4E*I4JF+2j5#(h{}kzCL4k%pWdKmRw+{^Ts* z{#9Pz9v|A)%894W5EM+wU#eGw?*>rttFP{uA%}cO3U+?0A2Aao@>Ecz8R?Pkhxgd}2X(`1n^m z!^gh6wC_}wXZYlep8Y2$d-k7bSHmZ6#Jo`8*?+77bstj0$8SSE0&IhC$FaLT!@Kj| zzHA_pxasHDPMmUL4){C@j1Slb*kZD8$MFp~j>WMZ?c5K3-@O@gO+nwWAF2H(Fz1}O zTMbix*T5&K|BtP}xPea$z$gCIr=|{I4$^deK6dK*TUB+yuf9||M$Kr(ST=Lq&6sa$ z`_4obVGdf@cecHz@3#-Us@G9TpPsw4kFsx)?>4|kYtYA7&+Xy$++43JMq49dF;Deu zK^@|Eo9E(iFI^&0P~^SX<`V_kc}cAY#+J|#ZyQ{pqA z`*&dDOdMau@f#OuKd!e3`1AEQ+vXo3*TvwLT_itlpXL4z>V46lYKOGTRFUN<8<|jThO!uY><+W;Il6RCw)InAN+gAwEEAe7bb;^>PKmhEX2}ZAeJ7$T!-tf&&1>2 z&sYz0>}P84%cysuKKy;h1>5C4*yTpFSL#BWro@{81s=Ix7_YY<>A z3Wv4QMd73$JPSRsU$w`7Hu()&kN*&Q(%K5w0W${vTXHqo8`u6*10LFw2<(9ko+8|C z;P_gfj%*toDJH5j2!L4ot6qHQxbnj<|BXil)gcUEN?zsh(B{V)n?+Jj$O7@onZTD5gHZ-@bYftAJHro7s6{(Yk_SO#UbX3He;;{;&TBKoPtfj8AGh(x`rnb)Z zXfzgSX=#nb+M^BYBTey+rf73(Q(LUArM11WsinE??#O*J^=W5ob6az3T|Ck{D}6R2 z(stjvj;ctcvo$iKa!w=?>zHW`**YuI+}79>k90P)N1GzCnVk(wDPP;v0M2(1@iuMp zNPGP~O%1U~T`Uq=)ZVeFCf42>S-n2Gw01^iBvJ`Ijb5JvH9FVVMVlHwKZ-<~I-8>R zK_m21w5jED;cd{P?sK6?V^jUcyCdtOb*)XNcO;>%!bQO_Yp82!xjS0dv0e|lj%a(V zJ+`R>)v2t#tut2F7OVUmxNk#UM{85G;d9Z==o+ny>`P6wp;LFDlkhp{LsrM_vzq$3 z8N%GRTA|T8*iI_WSj_c0%{qdYBIiARu?+cQ8G;3HxQL3RN|xWeJYq^Bm65uZ=DJQjd#B2&Ce|;X zt2NpACZ;JU^xPz6m;;ZGyr@b zt|s9$Z1I5U^Z@9g-hvd}lGRR&=cXC|1JHfgzmO@3RAz~EP6={%(Xk+Jb0Sf$0?EyZ z@_;^7is;iM6V6+i{c}Mub^SfLrVu+DwDF=2VN>MxJD1ijtqCn^k9J@m#U$D!q5YEC z_^cyRM=TCsT(q&VE=0Qv%`J7Y=JvKwti8RZb6N-^yQ!|TDKukNMb)&>oph+EzFsZd z*xb?>>fF>BYibR(ZtRSO8k^QNw>5=UuUN6{j>w%iuexLDirXVMuUUQb?X@@8s_RyS zW`(Y+!)8RtY$?=*&@{vJ=9F(jQhAqQODf{?_-B->LxoBW;8TfB{vLc3b`?YTti^_N zXguhb;T#|TloIVLQg=k^(!&th#%Fbd=P#~c`grQ=()(}~`mrlWzRLAi zjx&D3`9avgS>6L%P;YX1VuImE`e=DBPxJRco`F(lIfTWa{rx72Vn|oxQ=hMHMtLPM^(K6T?~a4J5mBi(3V=sornD8GeN^8T zUWNQ%zUqFUBB}lf=hWVn1zw%9 z95+5E$BBQnlN-ZvYQFm6#$;w7kszlCsJ%rxpF>Vd`76DqoCb*_pF@Md=1aK$6cXfH zya|C!C+H7-x5w2OYQJOUV6OUEf!h1Wfuw2%AqV`+^3^xC<^=kYO%6ccqor#9HqdqQ z2h>L%%9Cp%A%Dc9To@|E?vI^u<8K|u27T&v@`smv#v|bmr-VO% zKl$@Y3Opp=JLDe+i9M3B*WUYKpYI?`?Zf}OAIM0m+aWuN@m#+8PCVeJ76S_Y`Nnkk zvsV|X0#87_RiHL+@_ON^N{zoFPd#@9{5Ol11W2Hw{pzUlLcWAjk7ubPE8zbb$x@Vh zi_r%BZG1dS=N09T7O28#Mg|SPDDR%&PpbKlfV>p`;w_qE+ABaF{n0%kGpTN{ zZ$?6^kmq%8fZWC8(Oy|mRtELAB0TI}U8H>4fruaSuV1N44IcJ~J7;{m z#y|MG$NK_|U&r`Cf9$K()r^bgK573{+1F56^ zANmXb_m7YJNB2+2Oa7c156~a}ujf|n|Ii=)3WuU(?6agR!!+pmU9Nh4I@O;|0(8`) zKY?GI-AiC36F<5i5dIi-_#@gUd#LXMPWEV__~P)-=irZB!XME-xl76)qfUR+_R#*A zLH-DRHgEE${SW-AyY#v!0e#@l(;e~D_@gVnA|8y7`_l0T_AQM1w0}A5?MuZMvY)hP zDqe6rG4aBeiWfd#Dn7hT_V%UYgW)%u8r1fMy8gH?GN^q|vytJ5g%G=!s4En#k36c{7 zJ^l*RH?ZF5M@I8IT&iB!r~*L}DTw(-j!%sL)O;F3phSBW&iVEb&5vDq#$INEj>|sXUAZEkx<=LYcSZC3d1N1o*Ke&L`vvtPo$LpDz+Od&NY-9gLH>ETQGt@t z-{l{Tq+h_{AJd^v(4h~;>pA{mTtTzMeBK=&i&Lk61cZNN3;(!~zIo<`4$@~t{|E;A z(W3E>>7;KkKq6`X$a2ST*uxc{8$UY#dPc|NDn~rNA{CFTXgnV@@wkfGr}fE*@mQ`G zs?+NQ_ygicm59ex6d%&@xZ3u=iO0ZK(}JpFp^d-Ws?r9#FP4brU@-{rhK z(DAlZWp-v7d-jC%dJF4WpSCZ}m!>~)!H4=Qv7SNxkFmbB>IL+A6Z1ogKgyRxvnY%7 zJMBMNSbwrPsD1$L(fUho{gq##g3&B;mxT8irg@ZC$NibOCWbZ9KGts;of%3`7=x3P zYQ(Z9>UY??6zlI?y53Bf`F*-7i{|E%{JfsJ2G^ey;r+x9{(nt0Ba<3T+VueR#RQJz zG4?8pW@+`a?D`7+sR?YqYHt?h6S_T;H(=JwMqcgz7H`(G)g_K{;ESHiNNV|Jnf9uq zUb=ojvU#lEjlE312w|V*&tyN=tElfa^FR52z+CU_)wLm#zCOIrdIkFG@Tv6$zwWLq z-d)!5sQZDO^!gg(;i|68^m-cOf$Xnci2NPvi?XgPy`F)+^!nNxwbut!j`&s`_4{>y zTYDBpGyOE(C3*cF&GJ+K==BcfcU+IB$0yp`5iQQr>-7@DfAgk5Wx z1QOW;@#?!9Rnm|7t+)_*ivQdCmAQf@`%HK2Ct#sM{fGH9WpCtN`wg`Ij^=6h_Il6T z9e2(Tw0>~U4^PI&WD&pQJR|RuSl?=Yvw4iq3bVjR{~jsuUlq@+Apg#!{(rSpJgn0xCgZ2kd z-aS#wR}8Mas5X70b$bF*vqxPM*J;{o9lHhhyBfbMD|Ai6=A(b>z)bq^A}PWwZ^3|Gp(0{c0y4=`Se;(3@z zEWe`#{@0_IQFBOKzl4@ICJ*~nM+;r?wx}y#+egcb_Ijeh%plOZzjQv3srgg?VSM#O zFU!RCwxhfxll)!du^-}%n)qSo%X8+dvaYdu{Oa}AtJ5*w9s41W&oy77{laJt7C_)n z;}`o&h$mxH^9{ zN3@?V2OH#F{Y~i+heRujf`n%-E`oKLNO3xV&G@sD~HHPLh%r`WDQNAFRFVy)$$|HWI?SW|t{^1(0 z*uQc4`_=glJeUHH^@q8hp#8E>>;IVVy94g=0cTKGgeZuKg$q05pH0y~P`M2K4+y*NfPXGT)ahM2lDz!d|o=N9!H0{(cPW z=lIy!sLkse#G@d`QymWryDIhdQZi`PGkfpPO7G9We~aRNFLvgXFRs@w{_Y2glj;_< zMNt<1R}KBbIOXdVyhWSDZTzl$pd_0*NhovyGzzmVYmdB%R5)>ne`Rbuc!5ZceY+H>HgIF zNtyafWVKe~X}_4D9YIJ9eapJc^|Rieh5Th*ImLSYNb((>?gu};fr_?-FC9M0SF>?ExxYzVrt4dTW&U zLhC7OUoDTftK3}Q)Abg{58`*#bjaIL|FOCLjn1y1{Wtc<9npZfUbWYYPhxy(|F_q# zW$|*nUPh@|G~Ov*#)IV{s&4l$#>WeF{|Bl6G2ax$r_jh!g(f~>e1~R|Kf_xBsyrY0 ziO7RO`Sp|!PSMv3l*jjPH^xhlx8-G7Xm3h+nChGH;ID=}YdH^j#>eOA_RCZ4&nc(& z=h*(5A1zLXkT?EO3je$$IVqJVe;t>+(v`n7IoXvjOVa+NuK&t(;IB&3{$SeQu1?}v zM3aZT=S7Q)r%|3R%8nGkpReKhLywpF(GbgH{cjS>WAj(CJi5Q;Ma#+@{Lubnme2a@ zM3&FizntZ>`72mHn=fT~^!LZ}qM;DQm*SMX#n^yn{Z?pu(fpRycU+QoMAG?7lX^bU z@q_d)n@If13_tYGMBP7AsDChi=0}6fU&{}FnL_s$W_YI%Yw|G1R- z`IPSOiMXOm$&2wDN|tl`M`=7vN=`}T$)B%GPIcucC$Dwo%elUlXA0M6d5WpMv^)-f zedP)8clfKe*LnO^mzM~C8=v+!U4B{G-)y|1`2)AU>GkUT=!CTYX?ss){jB`gvVL0r z>iFfj8=o35bJw8A0fU^m`>|` zo5y-*dPM^G=`{bqziB?B@!L_4{dQb0QJ%(YN4@#J0_|g#vF(3cpt9q|aIXaTnf>(a zcnRDioe#vv!M!MtX&Lqi#4p7Ljm<;PLPwPp&pBdKj0DlR>r!Ker zf(U2Md=bK@=8yFj@gIxuY4hlx`=XZ=lfPd=^6GrBgm#SZrKr|FJ6cp^){~|@I~ug> zYg4}M!NMX{nm`x83b4es2T{y;LF}ZL(j^Jk zA2&9Y*Y%4FX}qTEj~h$!rt4o)3_L1x{Y%HC^16OWaY}yO{`hgEpRSMlGq(PTnt=@y zb3Ft5$m=Qe_v~t*b^l}j-;y1yL|uw*xM7;|g_SDg%okOvu+76BTe8Pis6=$X8{@?`upre!m<9FXE&hD^(EbJS&G`gE5s|0x65a~ zU!(ON)?e7q^U?hWm_+sU>DSO+;?JH2xCiZ(s_*`hEvL&2T))ci2N!QTcFz92o_~lx z`eVfQp1wePRBqc_yv@lU_S>{wz03B4J?Q&_3%1uYT6+tOepEhcd;Oojy;1dxnD)-A z-@Bi_y;1eE{dZpdR;mG)Ju}RHVcI`3{LH^GU%CBbI_YD|$$ltbyzP|39~Pj$%GFU9 ze}(rtAf@>$+E3R0H!^=yJ}UpS=kQ0s{lCb6i8^+%@?(6s?W66VmLKKk+UMzW_>U_8 zr{+)N<&@y>6XV4%ZzV-gF!XYSYf z+A|{l{HZeKL3=;fGcq33Dq0UXzdyWSJUG98KDwSBJs#|K?Jq9gbT;kp{o?-U{{VS1 z{TJyA3w&XLFD&qd1-`Jr7Z&)>WdYjD$9`S^&#*~a^0M#YCCmRUdHtqeU;D}Z!)G4& zWYN_Ro+$nGFMsrhfAjJ``M)>k(aFC)bNHDDzjpWj$N%Bp={NlEpTAR3`nRW#eDu-D z1%5UCa@zP`Nm~O*!QWuX^t)-l{^tkozyCM?JVAMK{W)ipA0;*Vd6je?zrBFAJ-L}V zpXl-f9(q72ov-msbLBlAzs@ti&AD^~QB9-D`H9CfVK%tjfmVE8W zo$y~WKT$RKQXzroNbz<#N5aBPxy!p<<+zX#poRA z_1+xf{)zqu<{{tp#69RQ{zSt)0nZtA25PyHnlUv2Swy^dkKFj$r8saF_j=~N^2D0jE3j>?+Gx>emU(k(@F5%K=$XRU(6s&BOmAvzp%6Yv z-^NqYcEs4qF=J~~AHmmyW^9eZb$@gylupFd!eLeoEP! z`U>McZQDd9%-Bzx=#^XOE25p#ItTMo|@+5 z_9yJVk0d9_N_vqDG*i3$-hiXy9IZtr z97LQ8;)`#Z@#*{ObV|>?Bd>>`sUF8QMN#-RY(dA*_@Nj%f-Pxy_H*cz-przp3vRc67Cog@+QmFRl$&JwIFk^(1d8 z59eKVwV(RX@W}nECx>uOndSKO&dr~xR9N4b z(2ZFP_j&Q~ zZdET#{ny{D@qOo-X_G2nyW&Sr6yz;!$-xb|{O-H|$nV-4z>QXe`0T?Cu3_AC5}Je~ zKB38Yrx?27&{eR~)zIYv@7fFBpwwD?dhzMUN8Jb>_;_zcUA$|r1L2#ltEr#qU3;|h zpgegw<@ebl4O@!pIudhB9BYARG#0pcc>Wq*SL9a5I)okoe$yD0o_J>*B(hhBc6_< zy+>~u6(S)jnCI3_R@K_$O`vDm^Yb0(9?wtz9#(T_os;hx7{-UY%VaX|VKsNcIm>nd zU!?CyGgOXJ_2{rk4j^DUnfI{5fkjHaht<*p>0bOPRu!&|q!`O%)T~ndqW) zr20Ry_5Q2q99f_4#q&BD+Vds)p-0>XXe$tg+G26oLew9UC$U(mzeQ1MJ!Kq8V#zP` z{U}0=7j6B$j=FRz?S8$j9~NRXTGt7ooDz*sl3mtcN`)9E=8*os#nvygB(|}vPuFM< zk5GTNsLz~b{Yl0k9bpF8;~TdAdfPA^=({dipZp#y&XK>$4uz2AC*|jUd;$f;Gi-5u zSR;lKx6k#s|L;TP^VMe@)hC>+|25Fa1j0>Kg4?e|ArjRmOadQgI)=)`bCHus=SwVs z8nsD3_v6SD`3G=ps!#k~$<`0Mh{DNGM)8H@Unp?g{ycI#n?~xt4wU15U+TXg}-GfDg3f%<i(DQ)0iNe z@b83Kc#B+abZ#C9Ks|NlCz#BEirWYpT`#; zx!!90@X*cCXnWLFxEa4q))4CqRfjt2=r@gGP0>y#uh9MLo7zJ3YcTlHvKvBcBM~}} zM9S~}(hZ^2`nS%?zuHzgEi`jlXvWOBUz-+M*V0}Wn;U7WYMr&NZOXLJ-M67i!}=x% ze1^u3R-wM7y79)kW{#NI%%7^OL#nP&O?@zvk8u1Kwy4+MRVR z8`-GfhhG=CyDe27YOIUZX*NK*+B5ajMY`Oxn*Qe~Z*01+xdA@`2eEHz z!29E6{YG8l>JFt=Y>b84*M;iaH?}o~P~+||;TLM^@MBa+U#_!#3tRs6PI$Wh6S z*RB(SEC!Ar3ItrS$EPiuW{@sROFtPpb8_(4j-#E4Jn^O~C=iqCI_1~?;$Wp&I{9gT z(Bp?&Q_3;eMiSRuZbhE#aLRM|Yk!-i#n3}kaCP9(KPmfZAB6v@$yI6D=*)rF`k`x7 zrybRTwXW#Er}&?vBQ<1Gnpe6%xQvcGd3nCR3j04-cqZnR>i2%}JV}@``e)NL9Gxoc zq6v!9>b%Rtf8u@0|EHOM7WhWqxAMPfI8Q`TqXw9N+PUd;N&d=jA;1%_Ev88S$D>*% zHRfHuHpMUJ>AYw1|2+5?MLUbXHSU`v8}WU~Lg|p_I6vQM6ceCd{Yqc^N}O+mD#D(2MLj9oE?M380U4(0AH% z;*V{`RO+>7#NSC@0(~hFA8c=>(n?F0eD1RJD+HfL2Oqw)BxYM2bo1P$X`Zawg)~;-F(YIF(|SPWrM=RJha9 zuNCyhY*Ur(!;IEjy6pEBOD8!?8I0o=2i+;pD-QnT{~WguS~~kb)87#IK^x}}2|Ux^ z7WiPB5$YX*XZoPPSG61bX@O_@8G&DG@n;2|>H51muq1nuzJ80>FBwIi^<{d%hHS^gJ*jV zSbyjec%~-=|3Qo2BJfP#F7O>4y8Bd*z%%_RfoD0N7I>!b6!H7q}(ei&m;FyW@R{cV9~z1|UcrVj{wzm@-} zz%%`rz_XmE1fJ=G0>8}iKP~V~KO^wFE&i;)GhM$-*%2SkTL08PdIUPhAEpQGB8mN< z^(wY>#xuP{;8`!cHdrUdGd(2mtXG-9Gre5kS+8jV&-5yRXFr@P@JtU2{GhdGt-v#V ziNLd7%LJb3D+QkYaJ9fQeT~4gUUvyR(;EeTiNQ z!+)5b5O}u37QvtCTLqr&uwCGpj<;Os#Qqt$*W{lPc&0xs@T}KPfoJ+IiMRZBOT4AO zAn>f$ivrK|mjs^Y&3=Jr`YQs@avl_TroSQaR?b5bZ|Uy}Jj*#C@Jv4{@GR#sfoJ+D zfoC}f1)k}rCEm(;M&d19|2PO-fc=5x4A_km8Gr2CAX9jj6K`+PiSbM?wshvta`Icd z8PD{I5^v=U3I2XdFB5oPCzlI6)29i1za5X20?+ih0w1*TVV=M{T~?G|{Z?-O`lC%ho=On*_} zSJcy>2C`>uZ!Lhc%}~sJnMB-;F*3*;90Lz0?+h8 zfoHu=3p~@$2t4a`R^XZLr!Vo5IKHtT+Lwx^)`v{zpMYlkpdF9JmJi26rk4o*tk*<= zXL?BB*$>MEp6TTR&w5Q0c&1kg{1z+!T!Cl$Jb~vp9~O9~*Gjzgw`E?XX(l znZ8Eg!#3XDCGbpNEAT95qrfwLy}-w;oGk*+^e%yCITHfU^eqC<{kT=&nZ8}%d0p2d z@JxS7;90My1)k|U1)lZl6?mra6?oQbpTINy1%Y2{`|Cx4XZlM5&vNz)Jkwthc$V{^ zz%%^~f#-F_A%SQ5+XBDO`olW{&-9}L&vrN_@Jv4?@N9=cfoJ+@foD6M5qPGb6?nD- zZ_0DLVtT-C)^mJgIrA)?@k|d2Jj+=u@J!b)w{*l?mUE)OGd(2uvz&M}m`*GY(<=o& zXyaRzz%zZWz_VWS1fJ<(fgiB`Tr2QQUn1~>)_;}>JkwVTJkPUh1fJ<@1)lY46nLhu z7kJhSKh^27C)4qhvO2LpuwF5NXL?-VSugyIkBdLkw@AG0udNbq>DvXKGhwtu4C$iGbBnZ8EwXFJ>_@JwGT@N9=hfoJ-9foD6k z2t3m}1fJ~>6L_X41fJ#GBJfP#D)21lc7bPlkHE8>PYFEJpB8wQbEm*Fy;tD5A9o2n z({~Ger5*2k1)k|I2z-Z)^DhcK(_a#Jwp+izGyN5T=Qwas;F5OMO|Joblc^y(B@Jyd5__JOifoFP|zz?(=`N{>J>C*(B^{Nzjrq31lEggpcJb`C= zSm1fw)e1b*mk2zMyJZ5;^fdy{cDPI6nZ8!w!`42H0?+jI0?%@`2t3m}1U_N?IVSK- zj|)7<4gAcLPV7HSPY8U%uAjFEJkxswp6&3Iz%%`6f#-3tQ{b82EAYHd*d_2x-!1TL zx4i<-^nC))c6&kKnf{`{v)x`2c&7IYe8}4K6@h2^+XByadq?1z{;t5Uwf;6B@Jv4{ z@N9=;0?+hQ0?&3B6nLhe7I?P98G&c|S%L4d{;VH%frqkxGCi-*_$|jbwgW$+$atpX z*8_B7Jlmnf(izY6i2~1d2njsX%LJb7P%iLHpC<5Zhf0BGdX>QU+Wws@@Jz22c(%h5 zfoJ+MfoD6c6nLhu7I?P98i8l}T>{T`SS#>M=jRT&zt|4z1)k|Gg8!j=O@DU?Jk#-W zd@g_JxBk{8@Jvq#Jll<*Z)E>r`c{GGIJRBzXL^so@3QhdCGbq&De!|GMxS1RXZkLI zXE}EZJk$3IJj=OH;Fw7~OyjAsO%>1PFgu+8)r-|Wl&!*qX<+do;afTc5@>G-)_op`(y+xQR^ zc&3L0p5-hPc&3*NJj*#v;F(@2@GNJQz%zZW#NTW5nJ4jdi6uLwNTk6Ak1m+rieebCbBK5!@fjHT24)gjZR`aeH^Lj6u#YkY5av3(eT`rSz{ zv2=QlV3LCmpL#&2`_h>&-*fJ_bjHoKCrq!j`SpT6U~h`wD(L(?N57zRyJ|yIw5_S7 zGg9B&Ru|o*Dxys-brtKH+nOVF(P-T!6`@BIm(^`-Ygm7qhTa^lY6*78gGfVw=TLP7F}5vjdiMu`p(Yt!qg(PUVF!~RjVV0TZ+*l zNN%2)wav8Mx40<=&P{c#Ri*Nt(b)B)gh7AUF3zP@bniw86 zt4Cu*HF#8XQ5)-O^~0Smx-n_f`p7zZP**KmcIWci=DHR#8k+vy=n!)lqa#WF}=tBIe%9I^XaBkR^RM6#HsT=I@F-H=urLMCxVzV**pmj7BX=!PV zpi3IoN9Zx+Xfr(-TG!Ir-q=LXXGiXvsZTpwF(_K=;*r)_>9ZM;w)@s~R6+LE$c)N4 z5Vd2bZMAh4+yzgrM)VV#k=V@6hNYCRZE67LI~v-fO}A}kgNXgzcpFB-p!Q|=3MC}3n`8e>2v6)GN{8KQc z`o!CrI&6JUr;~cF{PYZ00Y1`(@rO-4mY2E6JjpSV{w3D_tS9MDHs<j{`nT%Yp=P@eRciR(+c#L`=qi6-sG_1_T;Wcv-MJd*yER{yxIPx{lX@?4+we+&)F WwyDaRKc@e9e#{z%WF#Mv_5Tm3Ki z_xtP7eeYF+0DEG?QqX;G{r_K8|Euc%tGadj%ga_S*ENl)qOn(5tfruhW%k7GJ^a?g zQbFlv-PQfJ+Mj2;ZU38IX#4Cm#tskKvU}gf>zpln&cEV%9wXxo#?d&jUk923S!tZ5!`aUdSZ~sbXho_}>SD&>_>>ad? zr@oGKr?bNUqkH;>&`#HEZ`We%;Zx5HRGdFBu;cPG11nEIGw@B_a}Bi5;&piT$!~{y zPaF?F=KN0h_<z_`;4$pji|ucOD$LFz?BMeXG&W{_limw_yBJZ-vXxUkY!( zOym7&C^Qe_tv-l;PK4*Qd^@S4Mq00*og9ds_N&I#=^xj+Hl&vSjv(H}y zUl+q;rau}QCj4^n`Y_-#euH*@hIac0$^qM#crrXZ<9s+-%f9-P#IIHlUwk=`yeIqn z$%)<7$**Mx$&MZyD|Em{n%-Sq2cMdtd9IzcP3R@u7M&GF?GK!={{;A#@PREFD|`b! zplO7g&^xF*9;0?g87q7ZupHh-^z}pd$p@7sp$|6{!|y!TuJ3yNn9+aZ`QryRZI1Fu z@;~K!{qVh4;hXn;@9y@!?wk1D(NTQwsKxilCzb#8Tl`P?>&<-d`mP^_A|(^w9(qVW zeBqyquet3k?c%SBuMNMvtnb%rSzjvV$Jv?e+RzX6Yj41>2ea9=vn3M_57ujk&;Ff$ zeP}=Hqj|)%{zq2;A9;@TMgE2LU3yi&_HGI5JNE#)_OAu3?|d4&b|HyfO95>W=GxHS zgubDFW_?2sX@@WV5B=K3ciG{gzfA}YeTyBwSdTSz9P1nY(L}%!LqiX-!@qe5wErzB zbaum@M`^Bs=fl@`y=IzAVtp6#Tm=6l%Mf^_G2bJZAmb&-=!8rpTk-^U?I$!>A)iq$ zmdvhuAkT%rTh=!W`O2kS!{B)sJP(6sl9A-Q2tLQ4e=lTm{=F7@4{{8@nh+X>9PdF6 zqx>Yp@T-ZTVaNkr^j#Ro4qt*!hF?w!F|9kUTn^^BHXGw7YpjpfjgKpT7$Vqj5wk{( zec*|7)tHDY+J9U*QeVdY*sap%dO|cl(NEJneSD0R_+wm8n|k%*N{#!NE7Nl=Wv(Zb zx~M4R{6cYvY(q97J5hh{La*nbKNsR>AlcKm1O0y!*EM)m;JzEz1Gt{R>wn??G_LK9>-(!LG{xY-8EAccFjRbUoyPeJ;Xp$xgq94anvfVDqyDtZxwZBReRcet#|N z3zx7yruRQeHdXe&@-5g3zWg5i_%dwvZ?Ip?AK2k5hhSHO9UG?#eUP7iTv-&lo>Y95 z>~~)uW_>&l=Yb#d7rotXz3RQ1f7F4v#@aO9#+ZE!mONQ(PG-mLmqlJ%q!2p$pF$={ zwxv$ooc)jDs_$Tl9=3!ffTKx7IlsNJy(7@!t;3zVcD0B)zt*-lwl)OW{MNP(-~86c z=XdOAYTeovtJinR`s4GXH}!2Tes2ex-{A{(%on27)^~O^d)wPh|%8Y4fq=F9YF%Vc3?zXfrj_cn~Lq2EOyJ^`X>sSG|o`&50`xv!Iuf+yye%scq?Y@pYR$JTpcx@E3 zYT2q|xB3+EvRs#7ug7C$X&IYfWP8XV*y&X+$t?rJtw&wP^Zp>_- zx83K?FPOh*u6sRQ<`*ty#hp#f4es`?_6}c*yQQv zx=$DPxMlW?*!`^>=hzxknSHArc0t=m?4s`nXmkb-#L3W=bY_=zJ6ZZlQM7KO(G7K- zxlMICt1{xLPpaF{t!;9I=__%}RBO}hvs1ZE$*xrFvoJo4;!1|CGH_b~-ln87Ke~6p z5v%aByFdej;Xi-brwT8#WvyfO6bB>k?zZ86v4roy(N+sy+kil!ap?N7yk3{rk8mv| z1Pb}Z$l1nN&n_*(egz~MXfTasbqRg(Ns{Aa$9E=1_yJrBDC(SSJ{{7j0{YYdOQpg6 zRab267ul?5zdgdP06TWFE~DujatG)XdC0r3?}JO`jcraJup6DJoP_tD2bgTddBIKh#vUd zpnW`@b&CFpo=-DDkN%+#8;Rqu`hfGF3;7K6;3rMm3;duzqUZX{=+KBC)OC{2Ulq0y zze&Lfup+l_g2D^>FeAJx;G97Grk;%?v4J#J_!sCObuRClygz3eYW_O(hyI`1$0B48 zy94$dtYi=FB3det?@}83O4ry3HwWQ!ozRayc707v4VU-LY{)xNCtr#; zd1RzO;Foo2>oev-B2mga< z_Ehp`ER_B&&=2q#!QAPzZo4uiS=kFmfTe=*FA)8V=~p8WEk-}v zN&1UO`IdEInV^X+{n^A1{Hri9V+OaECdMla%$&{Tq4uZKFhA!;*jcov`II*m^*IrC zN!IHwYtw^sXVZ9di9gt9PnTmZpKlLNO=W-WWYdFlW)gl5$1h+{cCk66+QbL3=nNHs z-VS=KZA5>}$?6t0`Edy;1hHq&lZW#mi%v#cnx^s-!JfW3*cWfW{`Z`|H;3CHXHXP z=5JFnsm}dvFWS4%UisT<((e?P$=@8nyM;ZEP33q8>Y_dQ zqaAqZZ}4yEcY0L6%HG5P+4~*RH{GL1ebab$<)4Ek&`*cJ%lSM~{}L_RL;urB-O4{m z|NIjv!ehL$7nYd%>)DkN(|28~Mc*{P4f^KuHG}IrL+TIypApq}T{hMWmqp($S4`iP zq%T)&e)IW(_EO*IKO+cjL4Zhqk63?cesBXyePx;DrT(Jp!C(<9%bI19Z!~@N>_R?p z7ENgV%K0SMhCn(81{)F@JM#eUQJM z)AW)232)`=G3MV2*iRHO|E%$l?0bd?r&0S+@1z@9 zfIry9blUYK7L6MV`PoSMC2eedlKBDhH_V@T#E&YdAKwZ4Jplma1MnY;zps!(Qu_}` z{-d$@4t+u%OMKT2@jW|EJVy>pCXxEp1DUaSJ-9$y5wKI_HpF*bC;RgFj{FDyjE|1V zpSzQ2eJx8?0tkDW{g3o-Gsb88@tw#wlzerR?>L;ly8hvKk?)ZGQX=de2IT%r_PM!# z!oPXG=#cp($-|4vKLR!fU!VWG%Ab;L?_fDo<6%8>VyhtXX}+E%+bG&_|A4%N=j&aD zpOL(UJ;3t;g#V=Pwk7lY*unE-dv4H?JnNq#*-r8&w1@nBJU(A|-9^qP@|8C#A)i=J zQUtw*_E5K`@|nbpU^41psGt0k`b)uZQM^4PI0khio~Ex6pMbg%pNMqJh)+U2hPSL| ze0{UKWIP$wzbhHmYwjQZEwn#@f9#{lNk#bw=I2>d$-b~Z^8HM5M-dOn{*z+%hyR=B z=K@a8?MZV2`Ya4i(P_D5q`x!&mDaB*iR?;*=Qo2>;a|y_QU4lTfc-#DCd5Yp{SfUp z9Nd4Bqx+MroGATy+HW|LxxL7L`2NJr>9dF)_}pN!BkE7cZ_@+rEH`?O5Iywglw^7Spvh5XDN?GN&&2d3(iKp$~oJ`Yuci;{40s6@en*AUCK9%gt^BwS;5j4yP$V>RB|6kH5pMrm(P<(s|UqVFw#oVZq ze$OpH9q9sfNb+2u{hK2preE@pS)t>O_s2lMaJ8384FP7_B z@*+tt_)8DysdQoz=_dOh(w0Z_1%dx*kUIn8jwPp^tbcyq>@~KLmIK^JTXK zM+GQ0tQV&+{%&Cpf8G0I*e@yGzRu70lzwho9_-io`a%B2*XJ`Pds_1|Q$7Ow9SATa zDAIn`^NcItuk*Nr;vx7OBllybe24NGKM73v2j%a{&oW^lp|3m1XYxMPe8$hYSNi4o zOw|8k`3ygwIG1L>Y5s!vcEkMT&*6WFZ=ZGkGBRIrB){TC`YG}e%r}>V{E?rpz(39T z$^zhx>ocua)_ldV{vuvvVZFw3BH}&tXIjs(KAYBW)U&W&N9oZX>rrgKHK>`_Ys^=| zN7w67^A$e7E%^#<#;FK@xG+zlC@{N)C3gW3dE1ItiL4Th-AIYNimB&l;XURtpFHQM~1LI9| zgWpe(k7S7jpW^d~{OIn@N63HvzsyJGn&Q{U{WoP7D02UOAjs?l;rR#7@5?%088u!V zHN`8#`8xdPPU2N`|Bdy{yx(4c&lg79Z|~;fsP&1*XF5`&&x>*>o;<+k`^52(axl$h ziceTSP5bMu6rU3L{+!1%h)fL-g|p!bjs7_)kYXqy11MQ{;EIAJ0tt4Rid$ z_@?~^^lge~9BZ8Q)CjKe@rllPvKqcK&1C-(vpW!hU;_ zF}@WB$LS0oV2S-9&5z7+<6$AO|IWt#J8K+$K4;i}Poe#yA>KJ@LlWD6JDsEKzj2^K zzEgfBhrQ_e%-p{1y3a*}mJ0Z^(}f-w#M* ztLU(a7ehWvOM=`_=XbCO#Y6sv^#`99^YdPDo?$o-KMJB-I1fLD^KejhC%%tkmBuv7 z{yI&^exH2}1l*so|GX)G_&gl`AkNpb=<`$ld8AGsm?u&l{I0U>xA}f8DxZ^)eEfWo zgcR$mog9R>A4;eG>EjmJpB6j+&$~5w&F2Sc?2F(>>}Syd&VS0fK8och`24_>pL_;& zNOiy5*T8)Z+}FT;4cynjr>+4yJV&VQ$G03<_b;yh;lm$AzWQ=+)$ca{{XhP^V$9z( ze01S&BZmsvtlp=4Ts7|AIXTCAdl~Dq*E~1;{=d{lzQ$20YZCstH>an!H-(+~@vDFM zedHZGTUN7X+A)1iO~RTQ>_!mMe>#iedOm$|RMX^_7I_`j*n@L`ze)fls5Zu)a+OEu zA|9w?bq#;B4Bp2$Q+|)_En|DNr3mC&9Rq!==sC_rja36+(>x;ajv_s{)5avCJ!Jrz zHkbFO#(OQnj(TxwaU#8nQayI9Z1Y6H3#idT!D)=EtQe4@5o1(Q&kqw%e2lBFff4G9 zUZNf6ZH$#SG-EOR-V`gI(j9$?bKzCC1)nG3d#Wi&5PP&lupBM%@KMFO;p~ zJd%G=z(BnId9F_sp7?l?YXJ|%#A_ere@jgF$_DZ1qw@F}Tcc^oIy4=HaR}Lyis;#( z(Hlru@tiS&$dy|1YJD5vI6xpQfkA>QAArm?U}(s#6_C6Y&Ri zy5&nB@Lf%y10t&>OGZkta^UzA#Ezui9K zwYe4E&+rv>w9#&*r_uVb;^X!C*>Pj#ZnU~_Cich}#Jx0k6y=)-Ic6?c(1S zHmqG{`gbhtmmq64_UEzGt1NY7t2bV|jH|uyz0u;^&7uK0j`Wi6GrK+&RUS~G)ibDP z8|s9oI#;GPCKVYT?Eq}rG_bKo({o&i&7M7^V-0~BD?Q#~y8_DLWUp@33+mMCG}Ngp z(lw2mqw2AJhE1nYic;6vi(HSV+}dcAi;Sk5c)1F3dONv*8|)f%tRs{G5-X$;((#Ba zHcqZ*@?J}xNBtPBFGeto%&a53L1Mt)kM+<8e9W{)e->-xr!nSd7d7K?|JiY4wdZJc z;|#kW!^Ijhn$A8c-l+5YG2Fb0i&&xJ!^pTpd(8MJX_nPpx32yaFAO_5O}mOLM(yNe z#8u;(Ec8c-OEhvUHrWzx&i&qm`}@#8&RmVRy5H{nzw>DDX^d{=HeR21&Y`vUKX%q^mq0C zlJjpq%lo{Y?S0<$Yno5S9h->tv8cCq#E#7a4~oU?z04T()U175*H@Qfug&<`w{{iY zM~C6qZL2cs2>i1WkN6CWj$|$8=)%WM_B>~#=r#JBE(uZeQGz?Xp_$H)=`8I%vR)tkKv@7{D}$)hCLmt>AIo-M zvZ&VD9)ApJIpY%juAo2CL~+OlwBFHJiXEjPvWK-#YF8yEd?Qs1#Y#(lAwLrrORA*( zzHEQo(3WnkdME#(_8tox4btRp;HZ%25%1Ra<%Go7GsPwL{uS9iSAvv|t?g+)_hg** z$1Lqt&(`+SghKotMIf)IWcv?foyMo{npoSD-=m8W{P)W#;x@FWTh(6K@en#Dn1_&t za@I1%)cC5Mvj5{~e6#k7j@lEQwLQ&qQh~)y86(F;>CO>XYELxXq8fe2pA7ImOQH~Y zvqWBLP+9mNx*F;P{w}DNBHydGegJm?B|DL;_vIo ze$wMbyR&~Pu8STN*DPE}eiISu{{cW6AH{?kzexb8V`GuDQP4WH-y3KAugLbfl9I-! zVo@LS$@U)0_{U^><-gRPT-w@x5hm}`amN4pW)P#e`8#9Q@s%G{pnVV8do1Jk%l69u z)cC5Mf=>n@Ui)|Aw5KUyZBIWIg{b3)nN9k?BHK&$qZh0G*P{J@jw8Qne?_*0n{xYG z$0wg4oo=aZ;mpD_Xn0y=YmHC?%_k7dCg+7Zf*dDp*^yy{LJecSA$xrj6D4^(`A}b`-8@ z^A?q?Dp_CLSl_a|WAn!9=6e6ybD^_|p z6z*uQsH|DLvut_u^2cf#{ncxmS1hb~tbAwr#)Z|*n_HIW*R2q3i<_4=*KI7$^KIt+ zuH3kAd1qbA(me3suUojdW_$7K?F}2N+c#}o{lv+_n+wpB#OE!(!*TeP%oS=qLAlBR4^>-v_B1?#up65Yc5 z(dd?MZV5JSE~tb)TFOY~t(!@A`E8{Ox2^FOug>4pvTfP+qUx?q8wvw)$1LBuaTEBi zD_j0VeOF-<*~Dvmi3Kw zo2%POHhF7m{MAdFOB&lsH*Bl&7MHAAUbZ@abJ?=aO)brxo6GPjSk<|CMTNhiWqlX) zSH32H{Syrb}(JdQSg_o$&k*e$?^Qe4>ExDx!Tgxe)t$u|!DDc>Q~c!{52f#*s1B*0Z$0)0`!Cj)NWGVV6Ojj7@> z38!&!niws|Ak*<_*Yd;$$e5$1%r7!BocuLf5=%1c*qRR6IxRL(& z4d7P|aHG7rQtwtiXGyqK-hOEytGsVWxV4|3Nw`(sUmE%`^8b+mZj?7u&L0}r7$fs; z06!w(#E)?`dtSmHw!r%h^hW!>F5y=G=_xu?h@VN~mcK8Oa5b*-#YPEN<0|+O1HEzF z=Ox@a?k^?WI&P|*w=}MCzPKcO#C(x(>PPWH0&FqBxcm!{o@{B7RkYsylzg- z*cS|Nqg;&=uJoYjnaSO zLWOb{no>%46xXOIxQEvmYm`?7S8?Nc3tYvAehXa1fguZAExqdeQ}wUb`&rn9R)q3x@z}0%sMB3e6Ti4X;4RkT=5gNQJy`8P~jgRr4oLq(nGUL}B`ENYY&o#1J z-FZ+)wN0%}_?gGL_V(Lr*4o)37;5shv(Yy>a_hi`+3K2k_&vG$9fE2(^s(00QPhFopRDWb@DWtdqoE1f;4Sq%-c;{1!G-pG zjq>O2MrUy=#-pFFsBNX6$6L8_{i@O?Z?mtyV^v#2XETPs#oZsbL35k8p|+!~wt+Jy z1Oj`4Pf-=J7$scZ7ElDYI~L1rxCNVe1KZkb+x@=!k-sY%Xlw9xc-egZE*w%=X(c2n zi-zjdMm-!fd zwnp(zhT~`s_74Ek_{Q9tTDoNL^-#r7)cET9oNP~h8%vc6-VH`*Yu(0Z99KmrQHfDkxsCzA^#lH17zCz@`@ z!CJgj1MR80FWnsLZZlLYZGGCLyWOVMN7=1a%Ck?mPkVMty){rT>}YLb)j99CF8}=N z&m=*sKGx&XVg2v=*80}BzVBPt|B{<280R4QYq{usUX@=GsELQW40 zRt-6PzsmMF{tx|7wl9aVBhNark3Was^N#GLFCcyqzdzs_MQ5qJ6u*7rkHUXWWMgma zwufKe?g+o_U`PJsV8@SjvLnd*6*iNtc{tlU_(mD~Z~LE__K~##Oo_8 zjjj1lyLWIHChh-7k-9dFb58Rfw-foUrK9@V~!u$C>M1yyHyLinZaU z6Az9YpZ@KU%}@Pe90}j~^2qTAUq-$);qYB|ow@GBoo8-a@!-gor%sGCwLUd+ z{LWvDocq?xBj*piggmRmx7~I7nd2+&JQGIU=YRg-NE7NjzxAn+&8X}63d-~8@VN(X zKXZHYU1#!tabo25FFiDJTk;blOH-a2Id{h^BTIilb$u$F5BgI4nzlU%`M))i-~80b zbq_u@5`O8ABe&m0dDe%|-ErrcMD40WJ})9NB({I<&oQu50Av>Ulu-h)vafip7_$4b626Rf^UqR zoBs637S|Jy=cSSJ%U(pD%fp*pUj(1q&YZjB_{h?vhepCr{sMA6h4#HXvUKaK$a6(_ z^W9%M6Tb5}r%`&mEqiqHL$_7szx1t+_?xo_;$6Y2_@874$gW-oTjPe!Z1!MPEqrXY z%{zA1G5a{tYV2%{;SZn*f0zw_6FyO6V{1;qCu}yN%|8CD-F+36JI2_WpCD}kFC%z8 z0>62Eds+C6-KFqD@7UWnKGtvc=UD#vy2G|q`6&6L@yi{zuqABQadsQh;_pFFp4u;pA?(J$Y=(H4HI%^rTcj2$_*JKy={en)uJ=Y$`} zVH|aZuYBQ#qp!iY`yVYCbbCGXnbVmP{$9!8!1scKKRVzY%XhMrIqXo?>kilPKRKM} zJM8%N^Ic=#|Fn0kX0Ge)rgYcajgLD9`+w&hYie~2hX2z$*7#f7*zip^A8q=sZ7lrt zl0jB6?CgIU^j(g@#`|q!{a-E_Y%D;TCotYVcXm-t+M&xEk0SjL($kUt2zcBI+zI@S zeX#!rz&8Nb0lx_RZQxemtAKw6{B_`cz@G+w9QYrAKL@-J`1`QfXLKFUhiew@Ep}+I_ zjLUN?>IzX^?i2pIaD6*nPdnlNDz1;xwd;icYqbxW<43U+=cq8s6OWj|L1VMm#!Tr z{QGcSOV{=j{#IOHN7puCOU&8vzrMBVD(L=a_%+?`e7pCkbBx)BE^lJTIvSr%J#wHp ze(<_aIR|ej#qX$du<0)6*loz$_*Jwe{1b{TPq6 zV^e6u;FU5hA7z}BM)iQ_X2c2c!K%ERW63u`z?^{F1RP9+a`xwE*e4G17OTjy#^uc^P~< zkiJ~jy>@)v2W|zuP|~l6K|c!mYDwP|gZ>TB*Gc-e81(Ohewn22ia|dJx)*e*14Hh| zk-kBe`&^82zXH8N()Y!ne--qrWAJ^kbTGk|I?8R?8l&7hpx-R%H^iW)m!bR&QU2B# z^a9Xtmh_`B=)0(VnkavN40N=vKc8A>B4>eEMCK zKSuiBQ~ufG^M8%<$4LJM*^r2mxiXN=GPIOUI#{xiy- zIX?eWls`uL8OlF*eEwfi{ut@Mru_59=YNs%$4Ea<`Lo97e}(eLNFSm6myFN<2IY^D z{wC$0KR*9EWrGPf^K@-oNPvIPp0CE9dnC`6{UaOpd^8t5f_d%r?VGT+-H0{rhVYyF z%dqw>4ZnS3N%-8tCz{Rswa6g{*2Zqot&O9$Wnx^3_k ztQTJ`D;<36)#UK+6Sw3)_vMY@H=kgT(+l~%V`GQF^Cs`$oBOwfNB_bG-)h?oom7N} zzv&1MuPPlp=O`Wg^CN6*)OS_*jYDkgPo1{GH+Ekgem&7P_}UNH*z4Q3hF>kSqaGXT zv=6?NVuKv^!SmB>V=c?#syY9M(gJ9CX&h&xPC! zdlSs#@tD8KuJ3GrBur`8%N(EbY{@44rO|~L<=@%<6KyX~=S^mQ>|*ohdc#yd<a-}poLo(;Z-y&2}uQJ(|;=Y;>og(HsSU%J67VR=b7X?1Bh`LZ(3_wRn@ z9ZLo8-#Wa5ClAdZwUw0QyW_psH(fHC7mu`Qm*w}DWe?;bProB;w1UT%jBd^;$qzr8 zHPHX)yn)Tg-$?1-{8YFRW&Rz%{<66Phbiq)=ICLLGY9%P&KT|IIAh=x$8$!9(XRd_ z1^MpW4dDc&wwPAPe#;_Z-WrQD{J>bsW#OZEceioPM__tSw`wz|Jvi6tF9EkUn zhnFIMKJtXi(gxz6^o|`rls4+Id4C*_bSfJ@G-I^CYzFvMaDK-RrH&>b9ra?L;(;8S zje3T!w^RDq`77Z&m)geikZ*XuebAk|h08WvC+Nc~C2cp+u8Ni~m*tnBJbay(zm)P^ zZREKO{69^6mr(lF@Ok*@T%=_pZ5`rE5xaA@h0}4p20wT1_V6skOZatV*oAlzzrH5y zM7)q+SB33};WNm;BTW8Nj9=s8o#7dX3;Ff6;S|IL{CZb-I^yO0dUrSxG3l{!aW%Db zEFZtKKc5$V_KS1F&)=OHe(|1+@az9PCw%PC=SJ_J)|>y0l+TABaC~O;U&?#~-+c5l z1MkeU5B~96+2NPJH$VL9zg`ml??S_ ze~b=KL;UFf7#MblJ#gb0F)pC5<6Lng`u+yYL1h>ZN--{!V0?aD8Qy z*}n-HWwuk_6&Do`ir9XDxx8#0i-SaSDQ7`vV`o=Km%kQQD(-0!n8&`(#@6}(D`;)& zYA9$uSkQH#sdZml6xVf2e6YZ{sB3F!_jj>^u7*HYfoPhquDh$*-`VMFS>Zx;B(qw7XM?Z4vl)b@ z*8SHuwRWxZb+!0?``S8M{9Oj04gN0wCZ1HfXXi#=v9Go*;Hz(HA;v!X896Ny_zKZZ z(XNj{Gddgn9S!v#pN4cabT)JxM2ldhj)vxsg}0#{{*OgN>Kkgi_xtvB_*)u;b*u^g z0xbpItj^!uyuZWW-pD&$dq-PW8})dO>e^a6yZo(Pg&za=t@F3HG<4K`EX)kE@wQlP z$rI(!$qi^u_!!twHAnYZp8B!r!pgS1K@%BRPAcZHSe5G(>j=3LnLPSr8S=?8%v^;t7~ia z)iz-D>Z@%$J3dadC|o&?}i{?>F)v`RMg$l+G(xq zlcm!~ZpWbwwY0FcCS3f|>2H7Ur6!ggEe$ONZTt3hHgpv-pRe^GdWcy5@7TI=t4~nS z>HN)2{>~T~RQGH5cXTy%HMOrYxw;&bD9JvmWo?0yuF39vGh~nTU8Ztpfm+|xjJrSrH>1nAZ zdG?4b=@2~!mtFx4jw_MwIhJCiOERnE`J?F`N$8S%6azYIp>P}09XR)~h6n;bw*i5! zDWE;Q6sIKJ91P|$cjV?^o||8aJj<<2Bvv|eUWNow*7@*kK_q^ZOo6T`P(8s=Fwa_z zJA`V}iINJfi*S_E*aBWQOQto-SkbYSc(|Og(>N^Kwrlf- z%_W|)wvKlAk_dEoM(1d2@!Lm9?Og%%_cF}09y)buYW7p-_H?ziHFqxeVDxOj8Ih-G zRl(}zo?R3btXa=WyPKNpJ)J$BT@5XsmhR3jPkqC_rq%|}&h6W;-r?J|v1-TW?c01C zt9NePwqfH2wqm~#mbFoQWIE@BrnTu2gM z$T8?E*;x%2uxB*9f}Pgza&}6?%h-^Hm$Cs3=d+U<&SU)=Uc!!Pcrm+I!;4s-hI83r z4KHLt4KH8;4d<|S4SQIlhO=3XhUc?t4PU}4HJrsNG(3-aH9VIU0+$&2&14=8XRuTa z&!Hj3ls}zau$%F0c2>i)*l7*VWT$|m_D*918hr-q*KjI3rsYpzeHuNP1vNaKwQD$u z)o3`8RcbhaRRBljiDzEmsK2?{It@=_g<5_W^8iQn7spaHjImh54i>br3pTs*cZ?Mp zwy_{H(|ISM{M((pPufagfJzj)2JyTIvtNxPcb~)NEJ|YDDu$;b7(0odGe651b7@?( z24`Lt^)%*1=k}np2k|=#zLnq`+ySAt1etSvy92FZ5fWlPGTYqLT8^tFhBi2MOYD`n zM&e3|QzfpExKZMEi8CY)O584SpTr)Ck4YSm_@u;n5)Vlnl=!s7g%Y2Y_^`wmBwi;m zyqmYDPhxmK$6kriJ2<{qV)P!4Di7O?pk+@&tc8O0)9F%xK;y#Io6n%!M zuNpYPmdtQtgxAkFPGCdx1a?b&;SzyUB~JASoFQ@H0)ageS7Kp6_Rf>I9TNiKLW%qG z1YRfcNxWf2bg#sxu}~mP`be-PunPqOQ~L>fiUbz&CbMJ30#o~mo?0O=wcmyIoy7yV z#E*i@mc~xPHwhc<-zn*`e5J&y{l_F$?Y|(gYQOhdQC_vbPh!>nvl6TJ7w!`IRr`Yy z8|{}^wcmp^CCRJWUn8+<|4E5e`yJ~tnKLVwokV}EOvBm&zrIZT%F(a!JB=ShAMJx4 zoT->YvB;ox2K}7#G1?$FGDjWG1UG|sV79^aTA6+Tx9M8&+fTV67Xroa$oOp;za`_< zA{#>&;q$&)13gBC5LCKai)|3eM!JN?T||{%r5~FhzoNh2ap*={g#8X&X(9A%3d`(~ zYw{f6f@Jn+k6gPS2ChwJp#U-p)iL%|3g}|Z!WMyke=$DzAY*r}DQFEHCp zei=C%Ng@7yf$8o64X3$>OgJ+zKSFD(>FH4Pv)Syvz?=x_C@CES?Wyf7se{cY1GQI< zIKa<&Um$MIG^)>q7Jz?JfX%1;?Uh(V{VIi(XAS^Sz$kJ)g*|w^E5e!p6OeyFGTYe0 z*bo)4gO2hm1Kj)~Kkz+)WLFS~%l~{f8|-25LJCgQX9GReMUKYlPo@x^Tujm(pu1c_ zMHl7e<^$zl!NYbIe&x5_B8p3SLjlfSq(L8{-ehhnB%uD4um|>T(gf^z(yt5hrLxHH zq3=^E(BCvyA~Gxe+y^?uG4SJ^f;InlZ8H0Acf{?Yj%XwO%o3Ok$gxvk8c58z#)4^J zG1D6@IKzV5E!bnh0Sl&q$jm=z!G#ul*n-ztaGwQxE%;swuCU-^7F=n;{T6)Ef(I;k zNa9>4d!#3l%m{OPXR{3Ws}bOzB?^=Op1+Be=XXi6@OP^}YvrFo|G4dHEq^F5ExJ1f z|4_imysARrqOQ0I_aFKP*p8(-A2I0&0seQS!<9h81MmMBAD-)W^8FsX$<1<;nY~vI zA1UO&*{r;K7EDPYn8;od^~nyx`F|Ges~2IITJ>!Qf8IXQ5A|P*KCcEn2oeu3lKz}g{hdi+hvQKnHu`9@o4shm_~ncQTacUT z$9SyS6Yc2>%yQHALyfn02AGrTZ}(hCWM@`F-)^^|Z;Xde81jQ3%9r6EPj{VD5<0s*xQQBGcSMlC*2HdS8GwecMKcpwm; zK*M3gj`;`eLHmMo_&Uii^(tO0nN2k-6KcMe{Zsy_|2vU0%LSKDR|AfAd z1`?CVp}2p8|B*ma5;;pG5A}in1A*j3j%EEv11U*C*q-O-^(Q3mVeY~pszU4 zQygarC6327Q&(S!)M2_bxI{LRJ-}4TRqw)!PJu3gR zf`3$h@#FfZK7Xg!LV=l#l@bGlXSc(D=FBO?WxVvCzCgU2Vfc^gFHJ2lxW9w~Ii~V= zV}0PU(2dcQkEfvkHcRXRS0VL}2LcHRW`7O!CaSY7&i|f3k}HT3qJFG5dec=07xnkf zR_9+F_XRTDZj-!yfw@lh7YxVJe=wh+ZZ)5ge`Vk`1;roZfA2i4{)b^dH#MP;?ClWi z9YbH-KizD^81L+%-b+{}vLi@&vge??Ncm~~G@qq&vZ60=22JvXdKX&jPnu=apF;Do zhfxC!{yFYct-T&MeABR3Um)L|q0w{Q>~%w4$g_x_cSrdxnPUe~`U>?f*5nNZR+#Kf z`dY5poAi}o$ZLasmpU13UZno}0(ovWWaht2Yi}rk(b3Q^**ihw9|~ZzYS5|vLXD34 zld_EZ9b)}L3NrMyQmogF@uWa&AIa}A%2WS_f4rgeMfS^N`Cu01H*XFxSM3Y+&b9JO z$~E}eNS-uC1~KHB?+hCHS&90xF#XsJetm&U+#ZwtLcQ6VzDV9YgTDjiVgFGj?{>(W z!*JX{LHgsuIW)gSlkg{yg9T$Gs_>fANj*2?n0wJYA@_(YA?4RCP-fXXDN*2jp|d{pBNbW zguUKU@~n($FIqa9<%#N(+7EqVi7DvYftSpQSNX9Y)b+*fS;$CXvVLp-rt#COwQsI7 zXxM}74gFz)SMtOfFQfiQ_9lN;@=^Q7kB_w8it^LPN0NWJ$-m)`$^2kT%70%VC6Z0$ z<^H=bkmIf}y=M9K@zy#XqJGnOM)uElR~r1u{!5)fxT}(%`zsbW z3X}X%dvW{Ycsr{9z5x1@)2tt>zpk7^Cm<$%rt!(a?Hl#aX<~dK0i}LOe#0K;<9Yt5 zJ`>p^-55Trk$(pI%LB<+ZwB2ZD2xd&k)5ahVczejjGO;L0WyvZ&pzU?&hIpS(|Cix z#~F$$;e~~^?EbASoHs4{rudSaR2+Y*>AEDkfOZgeNtImU*53pZ*%&d=e%l=@tmOs=x zM_WJh@xZP8m&S*AGo$Mrl0Q=`4|_TpHBkD)JppI*{0iqu+IbcChq?Gvm|0$oC#THv z&bRDWF45|v`I5H22(rDaW}4?eK3^u7CBxNZa>dm+CTWXO9;|XrCMDwq(*=wDfZg)o8v);n* zQ7H&m1_vyrRr3UthrMW@%=5N-X?JHF;>f zithj7xPGJiJ=$N+naTMZ>mBsRS!Vv}B#toJg^{1-2?@?UL~C;8y-!;1e(&VQXjr}|gTjLx6if#J_{&HTsv z|7Avb@_+dAONzhHk5{9AdS-O~BK^Q$v&{U*`L{Fb4=G&UsK0USjQWS5drbA|_7n7| zy+rw_{sld%Z$XdhPtXm0;e0`#KXG1{e+%RMkpVmu0L>o_OlM^y%cbY@#}1ycg=9o`K~(eD(`M| zujh2EPu^Gwx-&jTYgF-@$=sqf;ylH&Ki1AmA4_3>!2KO(4Fu-%pN-@6MjdCg>o_YQ zG1gNkKTXWPsqu8b>0Bq94ReqVp)c&O$$oBWKfZpP6}1o6kJj~)CLh=LG*=Dr#@E-9M_QiB>wr_eI`K$7mqk)V# zr-q$zv_MjHt3MsZd85G(=gDO6v>>ky?UVLCtmBM6iG_V#&U-a_0zW@w!rq#_(0?_I z_UQM2uGGf^$P=@_r%Hdp{h0FZe{`?m)zf|>drE(y`=y3|;C@U@{|H9y&;7%6-bMaF zvxm_hUJhv^?5x$y(~L5TXtDII5=mRR}6GaCJa_YXRcH2mXXH%^2R-pBES z##`gO2=j|^o+|vqgW44R!Bb!`g!gg$ z2wMEZINulkk%!toO#k@EjUOJ(-mdgK9cL6utj3RZ8vTR!54t~M%rDjNzkdu_{6o8+ zmN0|%+aIQXeB{OtuV!yodWDWNDlPs|tu_K%O; z_;F0Lw=2D0#~CLr{xP7@|2qG`^S7KEnH=M}b{@j-r=&Qu$v*TvgE$|@ah%y-ZqfW@ z{P}kpW)E2d@wc3RhI;k$@1ucP+W9q}_nSfYKaBCid|m^7_jIH=={#7`r#kN@{>J$v zl@Fw+>F3`WY5MthJZ=Y>&%0-9=cn3vBKT?Nk?OpANVAtK?UatwPwP128HvSuGoH?m z@eTt&@8;**bibO$^Lje3Ty4_-%8NR$>?4MBp3ToIGnn8n&a>71kyL*EOz)e_~es<^2tMKZfi#$0#44Y33io`MLT0I}{M7%P1nB2c^67A-BRf zAJ^`891UbGFxM9fT%w)F<9P#DBMK{hgaSglieD%&8}Ac@Xv@XxI~th3z-UF+XX6c{)w!R zg{VnV-USQv^Z$h!Kh%fghG_kvz#>iGeSrd35Uh;)@cagz_@#jTILW_Qvu7x0NgH9`LTJMg^uvp7zq`hb^vFoJ-t@QMWq z3e)}Cm9E36Eh=xZ_Ix7lkHkmyLC;^|`DD}c8=>A+TKhtQe62mW|Lf}0_^;OZhkDm& z<@*A7sor?rWF_psOyj5eZ@l#XzCe-dUK4-P9}I**`lIoFt);$8wfd;PnD19XUj>Vk z)1|&_7$4Gje@lz@H?6-T|H`~s#WFvCC8uzybG4ywy1!`lPwGF%Af3`To|kj=D~$0J z{E9LX*2?m6@Rux%-%)?-3#6Owr{H`a_gl68j^{hg{zlKQL^0jJU2Gk1G=JpwH9Zf5 z{)_n{T0i{3biZ^v^qa@Omw+Zn|MsLs{e{L;*wgg9&UpJRbe&Y~=k{8v@eB1XGs#c< za$N(a{A6#sKdkx-Js-W8V<8Vcj~Z=1-G3Z!PqTfX|9Kh7i_P|2kTomMWDnh*$iHyD zbE%}$^AYK$`x(?<7P(GA9&T@8-&{|^a+%)-{jJpGBmWpu`SE-f`oGCu+kw+Hd+GL| z@kRHSQ14Qf2Yv{!&)tv*{nzw-5uP7*od!|)Bk3=ipPmP>+CPSUP0v4#x9auQi|6Bs7=foDziux)U*j4!$wT(?7&x2C zFYqK381^B5yI|1C|K_>Q8kp>3?hjV|lfBIS?QUS*e%v0W=dG;zw)QWwr`f+TADa3b z`5XG1>3J~nzfnV88V~jUVYQdlKGyys>=ErBWDiUKS!nJr$e(NOA69#Sz9`xs;vxS6 zmmAAN<&R_^8qWo$_RXW=MdFLmf3B(jjMqOse{32bY}_8&^JaPC`_6KFqyCa=)KB_c zs?n|X)9oY2L(}tdR{fKIW|;U}?Lqbv^P3poV%sNI`ftqsVj9o*e1rZH?LSugkp5{r z5aSo`9}A=QqUX)$xjcrv6%sSspHa^MzU_7r!+|{e{*miCY?6oUv0AssxbZ8f z@_$hMLp}d0`$wOu@4fa9q|hKtC97 z=uZLXzxe({rg~rPFp|K}-kXGmBk=E&;QJ={J_CRLnlR)`4@|f3bU_}<1AXS=`zIb; zI_xAbwa4M0`96s-@~;6t%{3x$dLW6HCmm6Fdxt~tqGMRJcTK>alu9)E9tr3_{ya;f z_C}KMm7if7VLQpkF`h3j?|Dq>L!4)adQIPh5cZUhk3)X6Z`}7ex-V$#VkcilPBN=pT4o>F^`m@PKdG`hM2l&e^ zJ=jd2UMlE$gmIX4x=`}t-;1*I_SL=tdA2)~0?f|ksU0QxIOgrEeUr=U?#PVGMAZm! z^!+7wM}~Nxgkv~K$6OcxzJ!72O~dEsWPYMwBHnMJ&!FP`#oaM0ZnmIv{p56HMkIgk zAMTDEmpIR&{3*_yj*Li#$S>(x)A0RC3bOp%h^(K}b2{eX&yLFg{pWOeGK~{@V0Xui zIQk+4f)o60T%Tv7`gV7uxr768I?CtF%@O%!{i#mjuZo_XA>VhG^)Hx5&-3!Y>zf`Y z-hT{IeRgi&v(kR|3P;T<^I|LOg=cG3U&ctAr>d#xD1aylFsfMnqIV}`nfzs9AT{U@ixHHY}~ z!0qF4dpHpN0sTE*^e^TA3lbJY>1fZw#Dyk|`WYICoWF!A1;^Fht$8r z;nDJo?{D)YSzgqS6pmdkQNO@fED-PSDF42q*n#g^OMk_9enqKx-HX#v{?>e!<-`lu z%05?A#qsa`aXRI%b@5Mg8TsL#4pt-b4?7t72V?lC75OV+*c)##MsX9%ao=+%f!1%ZC-lLC?Jh1)W{)IO;sW9??Y|21KkWPCjySm5 zGCSHK{b?1}Yxw?vy;rXHFqDyg9_yTD43G4^gy(u9OeWxd#x&%oS0fQ9KjvpzfAaoK z-#^28lh!xXpE>=R?d;BY^}V-X0>d9AB`m)81`K}XJ!&)*?Y9T)ta1l0&%b}Up(j_? zKg7RRP~M~36U|TG4(9#>dsy>p{sMjK>-)QLo@{zwN3GAb??sr`=MhJFPdob&eGdbF z9GbqLpp}RIwC6VjKg^7n?~Ui>^z~qQ&n@h+cgs(EAJNPY-zV&O=H2s4olL$;cIw^A zr@h}~mJi<}#CrN&^Yct5-#Yf>yOmFSf5Ix?gYTYS;bii8*y(pGpZ0vVRlcvhdw$-@ z84Bnu=$a1PJepEr4R4^RB!8^{NFYGe(;4`Z}{dH{_xd#FMTWPzq~j6>5FT>^X>Ck zn*T0xQ{3n2cUC1)Yu}CM(}K1)Yu_CM(}Q1)Yu>CMzG9f=)*Zla&unK`)$wes~Hx z9UV+of8P{z?-cZVr=V9%K|eMHy>beA{}l9-Q_u&dpbv5S{l#xG?*nww`jZIG8Sy8s z6hG@s3o^2~{mZOE}P9=5D1p3*x-BLWt>T8I@Pa;DR7Q~;N3W_{* z_8F5tBFU6P^VX9#=o|5hQ9tC6V_#d-pHr>;-KaVQB~8RX-W$(C@ssh_ z^pj!bpWwxmlQ1!V(Y{IeW5$Gj=uB2*hnoJ>F3cu4a_$-MIknhpMXAJuHY=M-}r z-e(`>bB>w+d(zRfm@`WLg@ynd-ODr08P#NCt&i(;X7z#hFoxJ_8D@fLD4`jn)}CNp zgX(fXRQQUT$N1c7tWO=_YTW3M~y3$yrG$nTLmy!eA64*vX)l1!sJy-r3G zfFcrqb9p&dPDodNHc@#w7EV}R`O!q>(5rV9BZ~O z@MrJ5aYGUNBr?*PgT{|>d{Gy#lj=49V$?T&jXYV6v4xT#@MK>lgwkIK0Y3b!}#Z%WPGIY)1VvU z<3)~*lZ{NYCYn$aYCKf2c}!H-G&ag>d)=&AZ?e=%9EaNM(}J{29CAQxwQiGiJZ?DT z3K5-Qf{jUMiFP;9ZC+bUx;HUI^q_snZlZHJAWyWOU}7j<$q^SzjzoKck|QYwoh5}h z9dg*G$Dn(Y>@Fool+NXdr87;oA(tpiI*O0MJLu**Bpu-tGlmY`LWiWISakdeK&2yl zqEVjaQ)MqLo=CQsb$jCt*<#imj5lO6)orqi+mJ0}l1=%PRmO>Ai>bGvDfGsovc;4g zZHuM1cPpFyz1Y;Gx0te_Z4=36)fvhAuJi_fQGId(n;SNbDI5GXmTa-)noPEsb(?G& zyY8q>WBQw^FTC&mhIao|dYize?~C3{HhoWeGuiZg*PF?vF>UUR+BBx@CYzdU{;tQ~ zn01?N8nYiyAzN&lj^8ts@O74H=i(TOh*iW{I-!qw+A@F{d)&^Yt_r)Peb^U8K$h6+Kq)IX%itR`mt1{tp_aQ>G1y$1h2K`+$n4;u7-*f4q@#0Gl;>vv9~o^dmn zA^$QuZzGG_Mv0GYnH%?2B(^bINu8VD`C)8l4e4Q`J3r&-L&%H(m5!6avQE|3&f<)5 z@^c4EdF@<+_WYC4X@@=;o%Yj{(P{TP8J+g3lhJ8sIT@Yyfs@f`*ESiQ_E?kAX-8zC zYx<|Xi-oS~m-Z2r_)AZbNG`W8o)DG+qg5|mrX<}Q4CXO+x+ z!e0+`*1!n`182=0Mpoi`nM8C=f$9r}f_c{J+#yt-P87V+DZ^1pV+**f>Mg@|NmKP# zs?3Oq|NC9vS;@afw%04Mh@30k{O@Bg26WEApSqM(x~2ylDA2Wf4atemH5u3YoP3OP zUg~7GQog+&cKbgqc;Gw#G^>R@y!5p{S3CY?$?}}Sf1m%v-6=_%n-f+d&m3`QrI_LU zwhJ>1T*UCckO>#k%TpI}4EjoTR>KAC84a&sr!~Bsozn0!x&>|Gx0DTNIG>%=a31T| z@Dg@R!;9Iy8eYWuG@Q#0Yj`2u(=gS)fZ@S)6V74n8uqY84QI0&4bNxQ8oq>8YB-Bk zXm}p;YIrUy1THc3o5?&H&Y(Z4XrkjCNDZg63wASoHvN%6Q~p`>MliKb^H}IEmG0IFVIq7=L6JIO>1#^hbY9_HeUx8lFb0 z1XF$&^8iQn7spaHjG0Ws4m`HNF5s@P^naQH&6ws)Gp2sW@pdQgleQ8V07HchH_09I zBFuiZojH6Co3kj1wcmojd4=Cu{G9n&#>hyci8VO$vX}>*`~r@}d7~a9@H4?*{IHq}z=w68%C9aW} z<~^bZC2p7an8Y;i5q(JFfW&7drg@Lo?-ca~CH6>MD6v=K!xGm>O!FSEU*bNAk4fy6 zcu3-VB|a;0g~T*ilD>{f?2))qVz0#g64yw4QsSV*0}>yTcu3+(yrfM0tAP`2_!HOp z0yB;iSn6_t-4gdR|J z#H#(?FABP9|B%G0{occZuG&8&v1-5fHbGbIACg$L-+Q~DtM(5`tlICzqJiX7?H`g@ zwZHIAL09cRDY0t5V_hb5X63Te)E}`_@Zuu{$MVppyTKH?Da7w2`l$!)cH-4yj2`qq z|2g>^!vF}>-zla)$xbo->3NFZlJQGlAVja%+W0(9eV)QUTGEFkeiC(3z^X?bA9IYq zs>*{kz1bBzU9D;;KRwf`!e!J5gv+HoYL)V&NH)^dYH5QdT|$;&;iJ;~C&;hp?>0ua zgD%iHEdkvJ>9X-A+Glw>zngDLW$Zf`Kk+plEsekvNVnrZ+!c)HS$SC&kI zC=T@(fMps*m+|!ItNw~qjr6qCs65BysxTd*=cr7GF}(4x66u~}nhmiml-Wdgnuilh1w zG}2wicp9seM1hRF3Nrep)}a{npZ(t=i}HV}pMphwtU;FdTI$cbpJ!*vf2lkT3D)wf zp&u+c$bU8Y9}J7~9uYBX{mPHFpnM4By_WK)W%)i+d8%KP%aG~wkPxeU&2f=Ixe?V* zLz1=p8YEyg!X{Uj{~1}{BiI=KTFk+Ap!^qNw0~5VSN5}m1KHnNKlK~Z>8bDHpBO={ zA5^T$lN|5099)*KDu3J^wyl2&pYp*M5-s#_6-@T{0s;I7I_q7MsT+`++Dcf4MtF*DM zWmDJQ>Z<0t_8mK$H(y^`zox3KWb^ggtBXvu16}R4Ejv)I>gtj`yRW^reMfoormL>q zxA&U1>q;8;?JeKcvZs1iXZ;plw!FG{S65v{)tcrl+fjGdzCF9wbl0w`YOF7BK3Lnn zW9!D^ZAJCvfyTO3Tf6tH+EuuB$C|>1>QdCvfc$%}-Mf3u4YjMTqWX&VwCvw_UCFjp zb;bT`{Ut3%z5bG-whhJmukn{|lX*6+ZZ6-yrLnzgM{`B-wO5tz-{vn_-?p)Q|IYf| zTiQxDU%#QDVq41;c}g4hR<)Jw@uQ7Z>zm6O+cxapU+FI`ZQHnNSChYF$JWwSjr;10 z*Z1tL*gl0kdkg)$i<;|Nr^?sfUBA0%Uw!%d4r+Hy`8Ho)O?s*=-*iJ=&zhz^)!Uk@ zi>q#^Ei>zF*OuDWYi`&|cC3)H?%vc>y{bMo&+`4(UsuA*6mQ!SEmKyGGR1p4_w1gq z+-j-Miapg^_Q8L;{qUJsI@{VRMOQ1@7Y~!@`+ss{JYoe^T*^FwO85hs`lFE^@V$OZ|U4S zRo)h!x6Qx1>cHOJYp1ZUsGs^vthO8eBI=*Y@3v68t5@wkxTkniQ%oNy-`^wkzKQ28 zZkR&8dafVneIom-ej)0f+~%t8in^Bd*YDl6{<^xU+TFV4U|spHDeY6bO61$qx?_s= z?Cq(=cviite2RR{GGFPz+Txxm^0v#o+pgQQJJ39V-VHs;wnMK)Q|WYLT?P8a1pc;N z_0Q^3^tI-|1of(ZwuyN8cX!8b%Z5fd{_VZKu6cb&J;v#nUVGIrtv>kUohI+iuSH5Jvfo?82MIFKJts*Q?zB5sCxomH285ncdeR2&*fat z)tfFZ-|ma$S2~feDErEKt1c?v-inLkx3@fSas0{~FD_rj#mR>?$3^k0E511S>WZh7 zZ!_;}<(pdfR!`)sTg-i=xo)Dldjr-@g?r)in+mtyAlDxlmm2qqHN|zMG~ewitgqhE zU0uAXyS4@MN;T$;V$2mAw%o9%X*Et=MewyWH+A?96xCh2s<^OleQm?~;#KSFYK!X{ z*7}RqHSDXcFS_*7ed~N8UtQBae_eyGuDh$*-`VMFS>s#fk@9Jr9sN;%|3aRx& zj{44K5Sm)|U)$8$wa(Yo;`i-q>uB+J8GJVQyZoDYQt6(Z8-2yT+O~kNzNv*6`{-xn zv`FA9T<|H+ji3KK+EP?d#9O2`X|@}xaKCuIEh`D%rUr$6D&(pQ#LMwB->JTrHZl}; zCSBtFvi&R0|K<9s`<|q)CI6G)O!@112Jz3GQ{<=p=~qnoGZH0ePr9VW;Gb}B(tXK) zpZX7J|2XSw>Hm}wS1~tkvp8TN+5W>+{zqzq_L=0fwu#CkKb1~Wd2zi#@z{`@`epmq zoJ3!4q8mCf(XTeq^DN~_F8bWE3VBX>RWnJukcG-U@E*#kdaUsfcsbdB^Rsp<4No>$ zlbC4d_;eDQx-(td%l!TCDSHk)L*~<3p@-ZIPev zRJ8&k-E^2EsUn?r=6b+w5}X%A^`P2C;{v6tekAn8*#ALRm1gGj8KhHwuS`_^CwYSW zO;B_!K655WzkGu99TTKKgLIPTEb6DA`sXDR$n$F)j8J|$=c1tc=M@v=uScF(@>5!@ z^b6oa{QD$P^l5)j&oHb^qbPn`F}J)x*PVTKysSfyIQ7O zZQ3|g$mh8s2`~_YN9pEbmA{n!i%)abZdX?lj+3Y+#iQb>GGBu z=kvNL-T00>C`l3!AG17{No85nRa>0MPhCOPP4g55mG0$;u|^rI^s}=6+-pf!q+tljs=wVow*ZZ&b zcemCxUd0#c8w2#WQ3^U5n*9Y`4S_Cpu{#!GSFw=zRBO*P}k+dzNMj~%h%am zSJ%+l$qH)m&%0vT+T9|AZEEObQ*WH)rlP5JU)x3GKG@LF+0@o5)zDPi0iBdLbO}|k zg4VXKhJx~K*RJU7^4A>@xi&#pI~ux5y1F`=YP-7{C{^S~K`pi3zoFq^Q(c3WE;QTF z=-b!fZ)vy)5%;0KF39L>rTkZ4y=&_Rq1vr&_1(>={vvmj+$zm&{(4_mo3EY=CK?3s zgq#A3wwPOpy;C%Uu~*vITGv+Ju(PSTq5j>=z;f#^qUrvQ{hhu}Z2DM1XIDp;zZP*% z3y(Y6>anY11^i#(#KL-4qlL8LyJ(sI_IKJB?*V^<#|Q<&(gk&$T}5k+(M$I77zZPg zVz{YgOW*zxI8+cL|6lj-4>0e@C6~s?`=Wqfkn0>pl+%HN{9J<4%xeWAdg&j8M)TqV zSj~%^h<}#gM@up90JRqT4(L?BoW?{U{Hvb3u-2!ris^YO zDo?!4VMxlY_ScH0>Q^|33re%X;Va;#FPs{qPPSe{}4 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1dffa8015f154621562b41428d264bb1.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1dffa8015f154621562b41428d264bb1.o new file mode 100644 index 0000000000000000000000000000000000000000..f254bf6238d5267f5b2ffc8031fdc5f68b2bb506 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Ko}rc@V1?h$_Vj;P!ruC|&JNE=?WsO%o7^{K zn@D{f=}BjW|3~-q52Kx~+1{$d*dwQ&9;`TjU~tEKPYx%zQXB zvg5$uQ%^rVm^yz+Xim#_!x!cq5AS>LXW^%uZ-$#sTnhKCdJlAkq0qsDgWFFW7~D7i zc(~&9iEwk<>2TkHH^UqcgBRvMF}QCH`q}^8@SIkRf9lO}`T0xX z?eEceUkHWfW4tv7(a(wS+}7`eM}V(5e=0oZ(#d;zcK z_udT8J$W)*?tCwNVg7m0EDOzPePXZ=x+p(?9DSb%hfbUdS2$0HL-Q|%pISpaFAarG z92lH?@`=F^c)l?IJK=rv-vnRK17z>RD|yZrL*>xrg@-@`y`?06CmeeB&2Y-rli@k% zFM_X&;c+t`35^hb1$f;7_=@vr_oryLf3O^|eTgT-BM$KoD>k5ap%j1|5LSPpL^`uY+4^*LJ;j%;>-I{P6>uHdpy1 z`JeK=0r=i4@XdR^cW3)v&kcO<=or3t)Z%;Olgj@FEdHnb^+rB;ZP$-Nk&?-84L_hC zzVJ`QSKW4&cJbH6S4Uo2-v66*tUnd=!9@z#}Vxk37rzBmd0$FTJ8)eW!%=pSzD;{g*|o|9l#|dLfBjO#y8Y=GySy zg#O`wV*SGpXooNU5B=)Jci7?KzexxUf14e?*nl;40_z|7@npafL&FcS!~gmKX#ZPM z=mv(FvJIw&Y3d>Q8B|LO!Eh zESX*NK%NVKyS#q{@|8=uM!@q3cpd@IBqPap5qyqA|6a)C{5vi5F60<_B_T8dIo^dF zM)^sGkyjE!BajEW=)W+59livejJ%W-Vp>mJxg5-Mbq>Z))>uES8y{ExC`7Q|B4&*l z`@j?FswojywEwtrw7!h}v74pO^@M1AqMxC8`uP|s@yEE9HvP)Sl^XXkSElD`$~;df zb#YP1`NiT8*@kRFcB1~?fnLu+e=fw&K(eQQ2m1dOu50nCz_t`&@+IlAV478<5Q}z~*NcvHl_0kL;j)`u%mR zKU~84nLh9c*;Lv8^0#3t`0~5(s9a8{Da0^>~aG-LD89}QGgH1I>A=r11I>#Hwzc|Ny)Dh#+VYpQ=hyxFVlY5meqlk~qWpro{JgxpMFn;E zSFkwm;rt~_>+13#r;zI|5Tm2X8}K#WJ%R*$9lpS0kO)=^_*(7?Z-X4(yF!pgUwzlM zx~&0kt4~6zttCL zxGT&I({Nd=w&Xy$baDe40e687l{jXf<L@(7uU-1u@Y z4Y`+w+-YfH9M`{>hJ4Plchj;(*0BO^J`K6^_c3abUWX0Vg7&RjJA9patgf!@vAQT` z_43tq0#lcVty;6UgGcWuo?0S*xvO%r8%3r$K-#cdww7fao@fQLKjZN6qu{mJtlRjG z-HdvY!*Nn*DK7$^;fnRslOqV~!O9HWdL%u_c-aNG`)FDeF2U>@l3oq1=)VWJ4d)lu z5<$t8K)R;_zIW3#Ep<;%&ur$1?CF{9;J1tse4e!vSyw#UzX5<~Yd=GWAXLTc2wqfQ zv!T4SyvSXG{Xfo8M4>|xB3**lRs#7ugYd_- zx5MYoU$kKHJog5=EGS&Yio2Ry8r>b;9i6^bcWYNir@PU&wYkmbUSCg3muJyzXF60B&FCVB`sT zbe}Hnam(x(vHP1j&apM6GW%9L?1Hur+eP2^(dY~wh?Akq>C7(ccCz&4qG;VlqZ{fv zbDQdPR%OIfpH#P@TietK(^uh`sm`X^=cIC*l3l6TXJdRA#gzK6LqlO)H2#eE10B5=})Gy^uWY~iPZn#p(J*W=ozO^B6{F+ zgZ7DZ)+zcYdOpnrJ^F_}Y$T4q>I2SyF61-NgP$~MFYtr@h@R^&qf;Y(P}fO5e^uB< z{3Zn_!HV3z2?{Uh!;J8*fO8V@n|d~q#0Jw?;h&>_)VaKG@cx`>sQK&BANqfCAB&JZ z><-v-sFK~ki)g7pzDsHBtKH)x+#H0@bwWS-_zg8RHC)~|vLWweoqQ?cfPT<_ZgBEs z^0)C^|Dw*$yre7rlx2NGniuv&|FDOXJ2}|{{1gV*#7TU7QSaTA5MdtxKt2Hf%Lv#a z8XlW0#2mP58&kfSyKGrhUx$aRr30+kOLmpnpAJV@|@863D*6nSA)j0QetD zv!{|jW1;kKfqsC`2C;NaSagPq zKyL><);6L)=Hz%g>H=Q^yc>A83GYT-;FkkG4fs5P9~SMWp)T-W13nXYkHF7P21B|4dJy@uqYCIYjoH9`&yw zr&bu4mJszfMEW z;4GTZ`jzvikl$wTH1=kw4J-P*3ZnXs_bi%%FqQJ7WIk;QAnc zJE!TR`4isC*JI4Tm9U>EV*Xj4WaemYELW)0r8%##)H_acN$4i~?_;SOaEB;;v9pdLps!pR0px#9{ zvH*Xuo9VRcNi3Q$9`du%@=My-`XuuMXwlZL+$Zd%4x=!}x@g4aO{23n| zlRx()(fV4JtOOADH2WXv-)4-@_TxK|Zz%cdDc^B8e|_!4i6Y-2`=vzKISk1Cm+W(6 z|Ac?@e9tR4egK>{99;$0{_@Ylaq?_56sWAsFHnQf8_g_<|Ap z&(DRNp4*e=1oT-LoTk%q%SeA`{wu9t(-PU`2+wbZro+FIGo$`Bv=IA&oJ@$10{UUv zZ#cOBBuDorSvgVq^R(Y^By)R_|M2~Zo6~0zJ@C1~WJlDWkl&^U+*xk)9wB=2r`-;| zK4QEyqK7|a8vQLdnB_LMFWl{R(|qIf+#hWYSV#CH#;5jCfBZ0wJrm|g7C9S9BmN3| zK`+`TQNDwG+BWb6c;)L`nhW`vJK7)QPY+Djr+_}r<&d850QOW4xR! z)cJlB{lUJODLO>@1;ooL|KScA%I3KY7nEV}h z(|iEl6<^%MT8#Mbr{Czsvm*+d+HzR1650IDeQUAZBQ9cF#LZSHhBEE!({EN9! zC;gsVh&s{*>X78QK>IgGLQKEpAIS+8{btfq<*yg<0Q^E5YJOH?ekY&~g-7RqVtNdp z27D6gM!XBB9;!Xz(@{6#GbWhv6Ht%gAusn=t`Epdbr(1g`mtd?=Z=w&^T{RR%Y{mF zJbMxO<8GEp{tbT`$KI$Ue@dkm{!+hP#FHT{@ycux9G#CF6O z&l3}=e-+OYf?_tYYK(&XVNW+h0~9(R!T#$|Ha<$^@1Pm_yZsDPIx_maefH!I_Ar6 z2aXC*Y*;T&Vf@{~9{&3G$FW~gynT(I?P2LPB4+lh5RRs`-qcbFcKv z^O>ms#qt?`K5;J1e#86)@$I_#%b&sj5Z^xQ{AF~$;z)kki}X|EBbaY42l*pEUx9y` z^Oc3b8`o!AudMlsVf{tC$ijM!7F{y^hkOJ=UYxerrfGuh*Eb zgpaP*W9BP-ep~Vt+Kf{X{>b+e%Koq?&L1-;0~>K+zMuXS^A+-kxcSPAG4d7nj2q`G zSv+5H$In+FPnMhFdo*7O8uE?N`3mBxJ1d&63`2jPJRixT^_9m<^k>OO5HC&nhy&xz zaD(4ZkdI`E1)t*csQl>8%}2<8{=dve=9%Kx=>0ck7btT7eIUr}1mXDy&hN{*ULG@E z9W})(!}&V==XT;%bpMU@&Ai`Uh|d?s+HddX;;8kB$7ecHqtAQu}{rDk#-e8~1_v73@{uAxTW9K~>5BsOs z`4!gx(fjeK({3E!rf`3s8b7`veoUFl_g`v1K03ac&VO=)Q>R$sTkQPDy1&Kzy@~zy z6k~iV3{KD)KEM+DLz*9%6DGn!V*j0u{dd*``h3o?|DHzsMMJ!E(uO3q|8_dZ*ni_d zh05*3yUN?xf2U)$1cPF~iuNwlqx(_hYZHv;O~7Xu&!foyGb!H``%gX}GNb!XzW*kF zLW#z^$&;BIb;?)IFN8i_xMP7T=H`8NMHo z##Yl|6EB8*mX-v$pU&@O5sHWW4eJj+FXrdH;ylA}9)1)=H*p?*4CmpX>`8nd$0|)} zmi={_j{QFSItaKwWB++W{_uG?{6UCgxT2wwKBl-CG zA_*zhS35ZfZ$F$){nN)SvOg_${-1Yq@|w>N(%6^4kJ!(m1DyYqb$=AgPw@GHDL?rP z>X7PQxu=198n~x{dm6Z>flplnba;+XJAiLFu%2ID`@@Gnj(qK#qs<8*>0Dpx5N>FW_J>?3I z&_z5@$LbpXW*NMXbEfX6{ULYTG{5wf)`Mug@V%3qMjcjp7eYklZxos zpwlFe#UD5owwfj3T%GHJv*@Vk#Ath=>!Px@lz1~% z=&2ZOG-e&G3Q&U{N8yPZ(!p9?U;QBFo|}76FKy0xHIEHgRgC-ySW&&6qUSggf6rG6 zZIpM>*lIyo(Z~mcx46ZXSCFYkwU7JOKH$NIiT-U%eW_-~do1-TOP$7}{`nGt{%ggFHi7u~H>{IK+-jRoN?(*(YQNP! z;Nkd9NFFrpJjvR{rDBt2{vJ*pIZ-ymp!EPXe#ZBGxXOW$ohM z7B;M1X8N}+?Uy2JHumSS)T=CYWUJR-yNs*7@V&9(+l`_DIga#_A27Q<9aSDsq17{} zXB+B-r#e@rHZBzz9_;{Z+6=IBV!Hy$;bgCF)eGv>>kQPX zE7CQMnxpEmeTGe^QHoO6+lyR}rQFG-j@X7sJIGGM3IhDc-2_dokR+ii=pG;=|~;Lwn5lCux?|UAM0O6fX=rIZeBQEJp3* zWW-hDnk@83iAywcEH>E^Zp{7OgnRqYKh9i@x4Pfz{J;HZ@M(;0#vPl8^|7e8chru}0}qPD?7hqw_0+6=OV`(wVz15k*|&BD z-baVw*lnva<_P?=5|8){i;iS1=jg)6P51%v{W!7xkM%$LWSmQ2=wH`(|2>K~`retb ze~DLU{%3A$b*W0 z1aN*Z&#A8eHi!4YkA_S{%Q$>=58^pp+^tRY;yGOzCKy$o#FVlz@uU|cQaqv6Vs8X%i5y{ZC}O zFI!Y=ZI3^Ow48B?epk{TX`(n}16uEBEX9t}5ZS}pC$+1R6TXQmhGL~9zmT7aizQXk z{y?@rZfHxlR=tz|PzU#bd;h9zpDRI1$JX|=pL-%s z`(u{&s%LBa8A2g`k0OxQQ?mUBvQFdEcTKGA$?ws{DE-LMMv$4&f1>lIjO+nri_teqV(j5E43$@9#M_H<4*>7pCwU< zyjd!*G^i~64_ys)0)G!)M?6vgrXt&WOhAj10Y~c#@n3AA!z288`MIIvc=q$dG4c0x zWIyTgqTSgqi0k6}#Wf2TlHWvx`hNhB#z!%sCTeyH$Z4|T~?f1qR|EscnuB4># zsaVvg!_8!al1G2sH zKQ+E;r{Gfoh}Zt@IPGalSliQ&MIq|=VP=#5FU$6l{piK2|8;2ppX11{+FzC};ilaF z*73j2GSy@o?X!*|a+Jfqq&8;i)>sN}l#VyNP>T8She4BZ{ zt7;2Ybk(;m%L5Po`hq1j+l$w1Z>+8E*i^gb@l~yB@*B%)x*Auu?8F#t;G=9q*@mu$ zf@R$_UgO4!_TuuLrS*$8h4V&3j=Y-EZh_XN-kKVJ^|F?druNc}+p4_9C97AIt;ye9w!CXoYfIPWGQ1Y8?%KSv!r$1sp&R-u zUz@+-@y3-EEt@tLz~0sQo7&dZG;J34+Fo3-6@BM5)>d>`?cy!*KfbAX@gvIL)U~d) zr8!WyBfoL$R`1fh{KBpIg^QOI?JGq@o-VL5jp1)}E;>C;d@|JR%72cK(U!8a_slgA*_`I!o!92!x<=oMa zPtN%h_pUF08OF(9kk3t|7Gv%4YNqX;u1!nL#)Dk&`;p3kxaJz}=i%PC$9$3&Tq^5R z@Ao{bAF=4X!+nc6HLgVlpRA63!3=qfdDp&OO2i2Jo0dWiVswY z_q^xzylXUyQ{p{mTF~=|II4xsFs?!h`WQcExZDXTAg+Smxbqgsc3d(HJ2xd4g-L=d z{}=jV>|2ma!9)+@oxW5~^d1RRejkQH}r>jBH(^}N2;3>WwzD3smQGnUEaV7dZ zj6>z5xUjcyCHNHyw@bK^Zvy&LzC);q5Cr0+--mxQ^jKv zPUGSHLmi-CJ9&LD)Z;< zB-}dguO!?$ZmOKOG_G;JxFmeke35YKNAW@eY%##)5f7K^hyfn`;~roZ%fXbqZcfeE z7Y%TuTuln~~Qc8Ce*QhAChu0Wul2-*+apO4)T*Ze03tYv4VGCR>z3Ti^^{>|Z+2D_gg7--K zt+c?^dhfTu)q2lF+TBrC-`wU6bTjM`8ojH$U2P3bkMf_KT#g4ab@HL{!C zc~D1n&27#2naBE$j$3Qi*3~K)YW8)ou{Sw#>%f0ra%}3ye1Q)9G=vnjxjq27%Y2Nke{f=eXnwt6eJ-LP*f@%fyvCh|7)QR7ptncdd5meBlp$XdPE%iOt+~708h4y?+ z^5^cxW^pUVqo1#+YonjXTeWJ#>e6O!i?5+`b$er13x>bR-5<9>OS`wRuCu+akuxR) z0(*i_Q5CWnC0x-SPz1L+7RznC37dHX+dAqx{Jw_KzbhJOZ}fJ0*#iD998y?m6(lL; zC_%EIp`$Z@$xUnH4%rLeym90HVCwB`;7x5l`EFj!#}4(|t#{}zyNU(&6Ta_75Rs{t zsi!M4L7gh|4~3M=w5l@N=Mzk(1yQ(1-kp(gOd(XfQuSH5p_I#`0P^bJDQ=#V`51k+ zM)6LD<7f`{_W;uP#@w1(x@7S6P{mNx`0DzsY)^d~OO*=VjYeo}d<7*m@nqFuRhytt z{3?2D{V(PC*K7Zg1)=y>{iQ>At4#>EDxfILS1avy*?yG#vi&ob23G!kz{j(nL;BBU kIXu~qTtT&0IrM%EY#p1Lob8}@{P4JxpV*)$d0E^4KZFAkeELz5CUyxo0ilz?Bt<=nTDv8O)>)3>3W2&<2+1lD+Fv9>gHtz9caAMs1 zo&Ry~?TdJLW@{>D`AfI&J?A^;{I7HVm%8thP3_H^&Bj!+u|H!5rMQf_#*AkVfAp|a zNV-_K`-I*5KMGv-U(p5KLxqgZpS2epc@Nh)d%^0z!Tmh0zr{0Dmnm7gU)%G8$Umg8 zPk-H~MLy}WN3Pk~{NLNzk;5}=9TXrW6wNt*1n8v zy%^45HUCrdymkS4n$7->7kTf0<-qAn$DTj^e#gPnr;Z;u{l}W;Q^+2|HUIX+E72pz zjz?1tz7jof`DpZF$1kFnj=vdgEB|@)z$>SsNe4fSu0H>MbgcP&^wh+M(d3nDBPTCE ze>%0SHZpepxznw$9FHD_{-v>>N865_iXLhIF#6mF??+Qs)kKaQJa~Ex`8`{9EZX*q zy7knt=+RThqpRP1GdlJm>>RrsZA<$QG7m;pKm9c7_w?y!-~4Vg zCGoq_)v%KSxo7V?AARmkva>$Y)_LIck+K7)e^&b4=(AO|lhSy?E;RZs-J(o+|kBq{MJ{(vJ&5)K4Bet8tu!jn%w`KDOBiyZ6Xh&DlnJhZ(E+ zF?dB>N7#BDe(*_GL*&@14K+o^8I+!O~a5KmXcp zkF??b!FSmF<$Y`(emtMJQi~)Z9xm-;uU+~Hn}07wi=4IF=93n)Pm?HClGwFE+lhzY zhTi)#39r5X6ZTr{XY92P)+R*$<{n$*Ogj7F#Xan`i|?`diw`&=7aQ0-V&vfqgKYl7 z{fQB>aiI!hCSvWvLyX427g4tPYkPjI$HbSdyiVr%bQRho$;Muzxc#P9G5MbT_fN4 z2>#Fo{3H0wNAQ<)_`qMohtlC^7gyLL7rp~u*~jM7nzcv;-({sUU% za+NJ|;r@gO`4IWj$MDS$A8~+}7$ILGpCe!5V>#ukg&*eiEfV!5|N6YEE^=*A{Z;Cp z|KIAHfX09y@+*w{7p1pc?)?iJ%V5(TP)5?2%|;XG)pkzuk4^Y=A_rQMd2lVlwS;9* zw%4UGe_+x-KEt*EDJE1yK+rdUCv~5j5H!DgXGSK6LacmpD(EktoGlNI zjZO|v8T8<8Nzaz+4}((^)4m`p5BfvFa#1z!;O^kKZ)V0jQRVed4ETq@zKQUEP;njHq!xgo}^o@`1nyjdus&u>Ewbiv1)%?1NAc@=ruDPWN zuN(Cg^|}M9F*D)|_=oPCiUj;K{=h71gkB2x$L|PlLp^+VL`8=D1G{&5hXcL|zvvw+ z!&e^1L6{Bt#>aOBeA6R5=%xcx!KvWfG}H;3nw$yxCWG!fzFZOyzCxckZ`^LR)FQ~!YLq6}!L{-Q;G&<`IOk&YH9DKq%Tvh3v8J~I_ zi)2i8ch@ezV@v+Hv5tDj%DDZt%a?!iqek`~?7+&WhKFbTK{xYyC*jU={oT{i)ZrBf zydJwW-wdDfb(}P@=((d7a<`bzRspy3Q-yfJ^_-s=&jj^*oWJ-w{!cYJTUgA_WDCee zhX_c|WAkt4Cjf$0pFiaMWJT)tzXX42hRJ@RfPNnbAN1&vk`UwRRq$Ph({;XNvv;Jt z%38^v2j7nK5NnDd6-y>PQ$o4FM)&m8r^4Y9=8QcRE^+cl#)w~PH6m%%297*1gsx+| zY)aDGus}ZHz2OpTHqLJ$+e|_-aHt~Eza<_k;`x(#N+aTSUE32ICo}9tT)#wrbZ*<+ zxVg^NfCCKll}L2RLtRyiYnWKm!4Sft0rP>2PDe(^eZkSGNmp=cYJ8^Dg+a#eoAJ9U zs>-WNUEAnZUbCLn?;af=a?Q-m1pO1PiQO|n*N}gBbkgtI+SS$GLSM=WI>V^4$cSmWen{g?E-#HqTL zKTCIGj-^5dBDZxk;$eexEKy>UAVMX)G^P&eu?;WWTdh%AnJR%9Qh(T?Ag`Kk?dqJ#lm(x*Gm2vj`${U z?E-*A<)ZuN@_tR;Kg0bE0+7hRs83WZG!hd@@dbsl$_Fc7;s3Ahd-vF4?5|*r9PCOu z`^g*z0!q)|DoSJ7b26F=K@X&{y&=?HRG+c4>Fn6!i7|danhXp7k>vYHo3WL+&ZV*W zfFps#VeN`VGQ6Io+w_D>%JEr;9jU zq|+NX&BsD+|6xwk;lHq7%V`>yg#ERgUaR9>obJ-;dQR8tbS0-Nb-IevRXW|o=_aCi z`R$zM1MRsk)Q54JU{RipoZhI@k8t`Co$ljwpH6!??ID`mZ{~Efj<;~SMW-7%-Kf(I zoNmzR2RZ$qPPcLzVJ+>qak@>%+d18?(;b}V{|4agy@}JCbbK?XH|z8|POl@Hmw!E{ zXEr=;}hhI){s0OpSb-U z<%IL`?Ht+bEGOI;-%3k~=Hn-~S6@%GG5$ClM05Y*_=D+WkH_yD=~`weA)RpU4;*hu zBV5Ki@=Hu3T=`pOsBtmja(n>3Xfff2e>JBP-b@wdd*9jb{@-{)o8czITa%9m@P%W3|`c!Kg2kuMtdv<(!SoQ6K)zg!<5hI#!S zrdU<^JvEmBT#gqU*J^9y^p^pb9H3!9tA)jQYST=APUCp?*k;Vc;UVDVb0+W0njU^`9Db@}Bqy*}3Ltbm8STW&k6e}jY zonp;|cTlXE@J$q}CVVr+stI35v2McGQ>>ftEfgyz{85S(6W&d+X2N?Y)=c-cwJpQ{Sd`E$1%U|e=&FLb#p7iuMgIH^i32t?Vxky_ zqi)}&+ph@1e~@T9MfR5*&@^-VE#E^ZB?w@@xpz-ij9map{#KOF{`LNRawk!L`0M^B z3*hfE9)`-%{smyjbaD;wa`?B5hoMg3j zeo{PMSg7M~kiIiL9;fNlY`;7$9;fNmjJs3gaauk|oZGKn9FNoT!O$-b(Q?6{3%{Y| z@Ytsoz#urXwTvhO^1elafqfv zGfmTtnWpK+Ow)8NX`UY~56n0%56m=8&t{sI0|wn1NY*G+`G|+{x;RABsl;KwHISyc zbe!r(({Y@Bp3sNTIl}W+#N(;?2sV*8zPw1sVZUv*XsOsk_zr(^0c z@YcXmjp9R;pYo^Wgi-#s+5C)neOeBf^)qsoiqgsQFU9iDjHhX${O}51zI*C*9OZ8f zRBE*TGxD#;aw$$fE57~>7~0MFQcdKq?9=i@;=FzGz7k)4TCN!S@XyS6oR%*J-Wu4T z(R$h_A1$xU_OtGd*Qe!`q2D%}xhx*1<&}Y>{mbLqkCvB){nkLOM(c5ve`~<$6zv5T z+OutT+5KkwZL{U~#nbn#jHfG$b(*(-v0Kzn*ek|3P*vz;aJcuAd(^YT|e5RW%uDmU!623j=)Jd*0)hcK0!?XO-PuU`j2 zv;I~M1A@|b6o)ooxo^fxv=ReH`5jme81`EOjap+IPAdtse!T{VMnbk{(;XWF6<8qB z`UoEwzgh!ZGz1Ki!rd#2K^b_r28R?l_Pectb`1e2Xx=}YGz=Jm=J+~|@6APfK)zYS zfMmiS)M!5;+Y7iy^MEqwjT%22A$@K?&c7k=;Oi49Pw`nP+8-v+U#)?XYSCVbUcFkh zucC8uM0VYEkSsqT*)U!6;Hw}BTGiSmQV@m1N_ zotki$3pEpo(JM0Vttk!%8q>n_B45((ggepg*2_-F3|c?W3Spk+dw1#rJ>wx8ej1L8h#$5 z1+tCxAkHUhLV3A#Ud(8{iS;h_6M5GlKizm`5&K{1Oxv3y)~<4W+_GQwF6`qxHwF6- z#==js|BCZy$k$`L%M;~~^<+!ZCW=%fL0i|?OGB|7WdCILTVbAG{eFx7es`Qcl8ExvtL zJd;iB;pJmG^3$HlVJE14sr|4%zkdB4Utj+JIJKXEeoZJh_bJssps9biV1(!W4gWk8 z;>St9iJgh1i}eX@H^Q-OdH$9W%FX!%`nhLfix~W`@Eglx=#Mad9^oX~`E}V|8E{IR zPpk+Qz@5X7i2Vzl&t>NoAUpDLabB3GW#f_8kDqVlbN}bH5Og-}=M4Gf%ZWB{Et_l; zkNhn=lxO%e^p}S6c>8jH1zr;<$P@c5;h#03oB~t_+3WtA^EpgS?-c#fvOi3gk%Yg& zp9?r%FY%;t0h@3N7uBCFNx|#z3&@Y-+* zE$8-`jpxU*19^P^7)u0>{!R(79O^&zOT;VeEehl;=Y2_w0?@jDcOx#4e|8`zM)OxL zjW;M?Ug!nVX48*MbDH{(+AoK2*k6S4iD)DT_3zS9b~e#@S5Y3S56a8uLmH2G|7Pd$ z^2q*N8qg?!^!m2U(P&CZj5kX|ruOIULv;8_{{JhKzhzF0HcGxY#Nz>Riu`UCe{%oZ zGUwvCas5kFegrs(>`k)#Jb!b3%M5vphjH?}JT#k766JRyez7-?F;c^RFs{7FuP{jU zq4NZ^+JfaJTS&6J6+u3{i}M+@$CkOzvi>&(NbDyVe+yv$Ge+N6h{g8V*$K<|a|hC* zaqfQJZGpQjaJL2Sw!qyM_+l-v7~+_DPkfG#O;>(H6X)x{rB5XGpJ}Jh{?89Ty7r?f zd@1ldM_!2?J%SWIa%45z(%bv1Js z|3=WZQq1xOZd+vIbQ*Bl%(VN(vYkut$h1W`LD*~K|2HR^zD{scoOEuckIuX) zj>f(zjso5kccGtK`ZI;^;+e{$`<^|~WAQzRvqx>VBn_UzWkHh$d1F1EY}a{~8{E3o z;pAIh>1oMz=1b`ikDkq#dbY|~&Ry`%J?h5~BzMFe!Z>GhAEIq7nYJHw89;R@$ zHzJvZ?5n*B&H@(1=_HINiw7zDG*0m3LbRvWC)zR+;S(;LSD<9DNnS!@A|+84lB!Gf zIOwjPr50p(eVl}w{Yz+?Iou$=`%j~Eu^{wsm)n``roV2emh#ry^aFz9prOmL6Cd&? z4p;arz(LAy1AoGfRVKyV$cY6}|i?FD;v3 z-(UTA|FGTupYAU$bibYd(-+cHH;*T;K^|YS-v)RxaUOx|Ra`C{jvmH!28Tu&IE3=x zn#Oe)7yToKejBh8J$s|y22ev&N78+v@BQcTgDY}o8>8=RcqF}n6p+ySaUFI_u<`vB zA+LJ1(ZZJ```!)s3b)=@f$_zE*Tko#I8)+xO-q<6pQ?oTT@#(~$0zZ-rhDU=p#BoU zxirJDKV2h4)bE;1sc1&RUr?uVJ1L1yeDMqn$5tr1YnSjb})yF@VDXBqDQ21<=`fkZl zZEV%=2LAK{`a!8*tXmWhR{cWJAk!X;+_zNzvcV{S+6!3qp8*3R*OmV*>R2Qf|1^9= z>8pNx0fvd^k#g#GLFy}7<*)Qq|Ib4CX8N0?0n#TqtN!nSM^&`ADGRG(%tC-jq)+`E z#$`_U${)AcU}nK`Wr`Vll7<&P8xeWOk=&xo7;DP?tfEA;m* zkpE6uA-9wiXX4j2AM;Cnk0t*@N?&G3`V`Wp?%Vy?cCcx@YV7=Eoa-ecextx9qBWtgdFb zx21V*M_5x6th$ z?(eI3d}z}I^6Kems8~NTFi|r*)Y9zs^{wBHe75eW+U6c;X?|jGuI8~Jy?H&ynwwe0Hj)vcduYT31QsIPUZ ze)HokLzUa6`zuGN95<6|M7^tSB{P6B-a@8%Xkz_iJNnvseRUHpY4eu(NevtxVb6YUeNBZHOOhWj^l&cxNVcYOCy;|}+DYgezVLu3E;dg_CP%0Km3 z-B8Ow%leU_#*R6YPRn3b_gsHpb8tbu&5!q@y~lR+)QtN2YQ}I)qTR;^C&nxLx3|GR zMjE$m?{p7NY#SNqM<1ijhgy4@MSbu#gYorq@Dcb<&G3%Cn%x6c-6KOS-ih(ifOo9I@2ebi57kx;4h&YhEAUxWHR!8aSG{(f ze{i6x+AHJ+M~8ib{u{g!Z!3Qno_ZyQiL$=;&IE}!-I*AsciSz@6~??3es`_6#!LF% zt%2QsZ$mF$1L*fnxI=1a<*~S@=u7=3i=qczd%pg4)TE-kg4ah4=C&ugv476D<)s$T zuXgyw1@KX<9&j(kW!|}u(11;8YwB{(H?bT$v2wtD++HY*Cjkf$Qp z__?y+g*(LE=O8|xxi~}3FGMW#evPf%G?W%d3+;=Ayjl6(r=1^2GCLg%F>F zd{R6U)D*6zruI?wqG^`+%Kt@sG4{u(m*Sa?m4Hw39?4XGA4P*MB;RoZ{%@hbkUZ^= z7UHkH0sjZ!Q+ZzH{lJ)tEBb%NLU#TL3pZY0*##=DNOmFlld`@iEb^m}Cp*)!Ym{!K zDH4{|E5{(~&1TXyjkZUtXCm3CaR0~Nmy{3Xf=WS<93 zDG2+-KP2g22{!u2sETuQajlF|A@z+V0mN&g^?twk!JEWqvZTDLEZfP$EeAPC@ z`l{sTf-kqcobxpk-@Fz5p5#-$`sputs^2JIGrw5YlkBUqkqP>L7bUBmb&{WNvEwn> zF`qK+H1V;I(^JUAH_IQm0smbS-&|k%_B17HIh~Skt#4R%khQ*aPOy-j7bV|X-#;|j zF_-@bCce48@0$2(Z$j326(3;|xgos$26h5v^pd2|21Ecw>*ACr7)2l~R4 zN=})X%2)X!ezeb%%2)B*P4eb^Y2QrAn(tR7-3>P7sLz12X+Vj#1-;T zjN=aZ8vV1QgZ?-^y@AnJ?jMn_E53CRhmjw>zRNpFuQzUQ-`3GM>KpeD20NyPc8??b z+uRNEJ{X_!4S9o8-XUHvQ6Ushlv9wRF6J6GPX&~~7tM4XD{61MHNI4Pqb-=X>N+hGmPU<}D?7 zKOSf)PFE8)W@eeyZGxUV_S7`-HIef1xDe?^fBz zq`tNOTcrLWi-NWMe&7rBpHuqJ6`4KtA2q4cS8M6#kg+vyQaK|xP={p0P%-HDWLEwE E3x~>ZLjV8( literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1fc6e741d499c4f215a1ac023c31ddf4.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k1fc6e741d499c4f215a1ac023c31ddf4.o new file mode 100644 index 0000000000000000000000000000000000000000..893b3e9fc638aff9cbcf77193728d1d59ede059e GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+mFh-3m%)>k+A=xk!&$M0$$$*4Fyf!nUB{fK(N1v8t zT85kmk!5Uy!Nlx_oQ<<&;_=clWSpJMIGW9w$P*vpaWI25*{n|{=Y+@%kudh)j@N<2 zxcB?((S7e#g8+MC!&1A<8{uKJ@4OeJ&)I~agU~x6)jz_?)*XM zw-eZv57+CV52|gUt2TD%|Jqpp(N1;gJp&(>vNwOJvqLjeduz|yruGlp zCR1OBd(&Cb|Is}IBWR~glJ4QWrcFn%nxF@P+xu!%x2Vlkii{H^MC^E{6M8ya&3XQ0Tycp=~Gj4?Vfy zc)05HiEvB%>2Ux4H^QS2UJPH@ei1ZqkD%c_r8ESXu*53mVG(-*ZXSco_{fr zygU2*$%(zS$**Jw$&MZyD{{a_n%-O60H2zod9IwbP3b4xW}Ov9?GK!={}lL_@PW-5 zD|!t+plO7g(m$*_9;J3i87q1PupHh-^z}pd$p_V?p%2%W!0$X)uI_yGn9+aZ`QryR zZJzQ;@;~K!gYdnV;hT4T@Ame+-s||@(Q$n5sKxilCzbyVTKrG>>-Bu_>dqgABBfK` z9JyaVbm3o0uDIk^=PRbUB-VE!&pY6sWElakH0HY`6J)#y8J&=cWJ{jHuKW|t zRmf+QizTzG9>{ayZ^{QoAz!7GYZN?>g6C22Ofr&u?|{z<=-&&OoPVo@-h~{aFDHaX zA;-Ir!ze$=F#2+0XcY257Xue2u|pT3lhGHGLQLz8E0=?LuFS>w$r>A=b>rjeABG6_ z8^o+}V;^`TT{S1-iuNB@kJXp4KX#+^xtPxLc2&j24ICH@#!(`H=$xLV^r=F0S3 zNty2nr4|;4oL?vjk!{E(WGCwHZRqtJ^yfnS3?zF7wxj=V;kp{HD%|(tx*yjQc>N>p zPviRUG(PP0_Ol5g8|*vO!m9053@dwz8Lo3sB(T=nfN(ZiOo1aLHsC>M4%cXkE3ybZWh*Pb>}=hynq=Juu_TiD*w ztGBD! z7pQOac6QZEY?rUSvm@Z|3bZUP*xKf6^R~8ZZO?zGBftJX7J~um^NR}Vm*myw=jG)s zE~v-9g2KFq^B-DLpI@H`IfYzzfEb<4-hi*^&JiTw>+}U4gG8`Wz}I?5cpK#K-VuT{ z`5L;n)^7=T+kC=0R)BY*34&?XNJ~S&>u=_h&L8OL>gek6qdDO^+B>_v?Ol0yfcrLj z{cXNL;~imUn1;(@wIv72rIQ=b2)F}msKhb*EQj7PQ&`!S3v?|5%b;Rhi&eW$kw?f( z}J#x9FCJhOL-pf3|Fk5-W);D09Iz=)+6ab#>-B?-AB`+a0zDLko0P3MgLvE zZ8*QMmIz9&1kybf@V$$!X{oz=d*?7mWOwfz2ft;E;Pb7W$hzX${tW;`Tl*P01feQk zhw-BNs&$oRmBsE-?Ei6&A_^Un5a~mBZ6T1qD+pgK?QZhA>2#r`)!Wt5(eCc*=xFVn z@5apbc{_dX{KX3k=eyU@Wns}$R?^+l+T`x+>Fn~gx!bxsyWCB_EiLUn_uA^}6>I9( zmDjGRtX@@LzF}?osg3muJyl!WE0kvx=$DPxMlW?*!_(h=h&K4nSF~Lc0t>R?V|5{Xmkb-#L3X5bY_=zJ6ZZtakOru z(G7K-xlMICt1{xLPpaF{t?k|j(^ue_sotj9=caO-l3l6T=U{vo#gz*=#3g}Y< zER_cL*Icr(pJ%haef9{u1OPGJ_OtkZHH)f}@<8Jn&9w7sJ0(kV-7sN!8AR52C zM`xRf7roae__+ralv#M4bFxDLyB#f9Wbi;58(ze6gSLBmOVe0HKaj*;JIH1S6Kubk z_WE{SAI`QH1$7(Y86!O83r>z?qn0og+QxXZgGqXn{$v_U4@^#&O#KfXOk(GVo^kpl zq6a=VXrD}HouYrD=hIBkqkrhbM&kHuKH&W4LOugM_yMonKHvxa5k1#mMwdqXpstgA z{+h6j_)Q8FV!Wcj?73VXYJWNn^K*WLoke?^PkA#? zpBG^lWxe4&ZB}spTpDja@dx|t?s3fL^Xx2>$iJ10)~!fAX)O z&BlF6`CD-AToNaSpFKNTf0fS{ZVx{H(4O>VNB^9kw`_Vv))7YQY9z9U+dmWbjTX=s zt2)^m{(K_{)P2E>R;qPGZJXL8Qgykl09cc z{cG5%6$PdzMEw=(!K~o)=|rCx^S9}lROkM-2kl*Gul#Kl>36!z)V+T(|1F=Mc*{P4f^KuHG}IrL+TIypApq}LpIh6mqp($ zS4`j4q%T)&e)IW(_EO*IKO+cjL4e3$pICosesBXyePx;DrT(Jp!EiCF$eLr4Z#;eV z?LZ!`EC$?vfEo1Mo`JIQ~F)-$`vA837Op!HI;SMhCD(81{)F@JM# zeUQJM)AX_Y32)`=G3MWL*iRHO|E%$l?0bd?XCBEy1_?{gno+Ae)lSuvQfy`LE9$us^57;Sk8{)gJlYM!7NB#qU z#z)8G&%H^szE&hF0farx{zv+^8RN74_zvV7O1=ilcO1@NUj1;g$alzoDG_!K19JZ* z`&{2Y;om%8bjbXYZ4_;|e?VTs z^Yt#n&q&^)KH&KP!hh0t*^+sF?BMyaJvZn`p7YO+CzRm9-l9~?jq+C`O0h6 zkWZ{9DS}=@d#GDe`AlL)Fd20)G)VqQ{iR^IDBhkCoPfF!Pt(_kPe9#>Pei(9#3!L1 z!&}xfzP{OAGM?Kc1w?Nk#bw=I2>d$-b~Z^8HM5M-dOn{?lUi zhyR=B=ORwe?MZV2`YZ}g*J-(Bq`x!&oz}1EiR@B@=QqPM;9tp^QU4lVg#AEHCd5Yp z{Rr(h9Nd4Bqx+MroGATy+HW|LxxL7L`2NJr>9dF)_}pN!BkE7cZ_@+rEH`?O5Iyci;{40s6@en*AUCK7;Jb^BwS;5j4yP$V>RB|6kN7pMrm(P<(tIUqVFw z#oVZqe$OpJ9q9sfNb+2u{hK2preE@pS)t>O_s2lMalg#)@sK@Y-m-{Q%2jr!?3mgdj*f5`S$H~X}}#q)%qm`$t}qac6S-NVoTh0aH?|2o)%rB)Q3_UG6ii$hG>AHL50owO72 zm&o-jxlob|{?Y?_DxH`_dddC=wPn$KLEwK9%}RIzf0J|-|+qf_6v%)uk!OfrJtLY2m5utevrTM_4$m+p4R-#l#jrE z`vVLKinO2gJmV7h>pt$FcnJO`$o-fp-=TcQPXbf^LHT>~GfY@W=<8PUnY>RmpYe0< zm410X6ZOAXKEux^&ZXI}o4+8wT{D0AGx#6k+h?7>jLlaZ$uD`4eu{hq^UdWTf8^&Y z@K1BTvIuzN`b_JUHD58TzlaxESg*01hH+dm-)ziQ~Vmc|EBB$Mee`%2brB9JpaJ?eMR?6 z|LKTlv>%FOiu~^8>@SBeFq`?DHKd>M6 zGgImF7`{HA!{?1VC&%{Z5AgjtKR@O9_Uns4Kh>}wKZws8>{I!EocqV0(|$a5-h=V5 ze~O)7Vf`PwAHR3{_2b(#?(g@;k8g+{)9&T_FSQ>Z8{bUlKe@qsr&;1#?EJ^Nzs3B$ zf&KP0V|*(LPSP1Zz!LjInje{yCc{Ev|DBEfch)5Oe9o}{o=*EkL%eg+h9tKCb~?w| zf8#)f%B{q^>RZ@l+Mc5)Ejek7gxr;l4?e_HJPKkvrmHJ=}(u`hxjv7bc;IRB~W`6!m3;PV4h ze)1XAA=TY-R|9u7a90C&HE>r0pSlL<@EoCb5Z`iOy+6PD`wxE<`HL6(Yks@wZ~y*h zRTKWI>7xsO9XVLU=JY?+=c;q}&&xU1-_O{9z3$o3_y4s%@->c1S)K5g{W*R8{VD9s zk6-@%?;>y6S$W;+naA|ibqTBMup2>0|H&MR>-qG>QB9LyTI6+9V-L;&{xSiSpxOj` z%4Hs*i+P}q)iwOhGI*chO!*zQw~X!4mLiaA4Gi?LqUSghHC79NP4kGvJBsw+PMeU3 z_LKo=+I-%h8t;_^JL)B=C5iMZPW9NeicM1mFQ7&X1*b7Cvl2jx$Ba=!JwHf1@i8vH z0!FAWdWm+Nw+U9>(2T|Kds3`;N_X@n&VtKqGd@qk_f%7mAogjAU^!ai;iHOm#cxl! zJTLC}cqIR#fPr}Z^IV@OJn``&*8(1jiPt{L|K^zPl?~$2N9FM|wp!DYb!a*Y;}EhZ z716Unr%4`*KX58;OR~wHF}8%rjN8qicRfkyJt{Yi6bB!b+Z7)lmD?@kj@L(2E-*r} zD#|I@bkf(NWQf(e^^u#TD%- z@n)>hQwiE=%sN^PpawmT!4o&6gVnme@&U{}H}{}E+MM-i9viT#82J&f;s!lM&v7LF zj;|Ej81JI7)rw+WF=9MjOM|N4_$yGX{T|&av&%4UJljS+6Lj(L$_muaSg8#okI^O? z7`2@kk25{B)_Iw4er-wG6R~jz=I7F{acp$63vYFSn4&FI*mvD^CbfPZxAQi1mfdgw@w~$vu!>peQ|1; z{bu`!*XBleKf_nl@kYCmp2q9LijUXlXUC0|yYcGAnb;>|5ce|SR^;c`^{aU8)g+;* zHEw&;ywsDT^U^ICu(pq^y>8T*9w!D_`Hx4h@&Ku0Khje3+GVak0lYGcSi5YN zwTpjS*syk)>EE)nUxKXJ*q_Hzud&pTtzLWWGOqT*_r{BF*NX<^IMPeL&+PgPRCz#! zR?ncGZKxBT>Rg%Hgj8gBv;(kdGr`7cP0w*5HhXrHjx_{ktn_$`?FuM|lfAlCFQ`+m zGf}6mNY^xKj;hD@88)3pDNbE$FLphaa$}=WE;62O;^ivD>CNNRz2csl!}c%#nm#&Gj0E@Fj>4`bsF?J?t@q*+#X-MacyyfEzKH0?67 z7`2m=5m$|Cvd|wSF44%b*knt%KKFYP?(ReXFmpBD>VC8H|JI|yr!l&f+jxE6I)~QY z|Il^QOidhnT4?1b9=&BhWXgxew2$sN`5a7$t{}Ahpjhc|x4KC`aaKO7Jn`t&=8jfe z(BIbkOU}RfEbsGnw)c70uW3FRcWffo$D-bzF*`O7JSY~k_b_ABQ?vF>U0+p(y*A@# z-`Zt(A038cx2?*!Bk<2kJmNDfI+C@VqYEE5;RnU{kAWn>f!f(ysdqqe9i(a_wC%0QkrT0-eQ)hkTGx zO<9!kMxOr@_t`$1-0Z36a-RS0bCcWH_vZcPw;ODKHGBTdynmYUDRz{G$R5@{sa=hn@Xb^)6e}(Hh5Srh zEUA+A`?CFULtDDF>Ye+0PHh z#NXGE{iMf>c4z-sTnq0L*DPE}eiISue+nRtk77bi-Y9_7v9UWjpGroYF=2|sk%oxh_M|9 z$@!${)K$$dbye-Q7Oz>?SzKN$O6kgyg4XWF#U-s97q6+?R@}PQyS}M=_WP*B^tscl()!*bEKq;+X)!-kSP z-zMJgiVX$Jx*OV-=79%)L%~CJ+e%h#YuZrTxpBj)$5*ti%5SQu>uy@!x&vdhgO7@J z73;bi3zqiKcuni8I!Y>clr=0~m)BacwJJJp#nx5c;-wws6>)JLfUbpRr z=nC@3qg%GAE!eziaW(YORzWgv*+ja_?zW%j)pnF_^w!n+YnQf`Hg}Y*-&*4>DP6g&Vpaa8it_G_ZLQs#D)3sovU}6=Dt}Yk zx*q7Sa&`W?$D5W{wQgKr0DITwZ){&v*StyCYgo)|Np1_WUi45BVN0%x@}Oy0o!yOF@2-H{aWsSFpHoaehcPh%Z6(7&qgmphM6_@@4xUDk>2 z2MUMlDRpn5n-2Ing~PQ>-IqCC&+6Z_6Q2L!uY^;-ycqDghF~fU;N9&PRHeRU*ZLdyronP5T=rR*4rSkS~*4+OADT-MkL&La;8 zsQ5sYc+Y!Y%ezLSI3?b5rUgBZh+|sV4C5-KppWrmhRdCh0^%y@jXQ6F?7$_%uya## zQJ5sS@_(T(#=Zr)6ioCW-swx_MDLM6<@aGIG#>rRYrubx_VMUxe-jV>@HOB+0G#+a z!|lMBnpfI2_&&QFuIAILvY$a|Km}h9c)A)CJ*`Ha3ZCN2;hSasv;ebj z<4W{-7>CM9aba)bO7P1PZkKQ+-z4;>e1}kzC4Pzpo+shc09S1Z^hF837jWa2ahCyZ zOcjqwIE{J7y<7R5BjHwg2c?0m^1deF)_#5};Z}KnVd%%m|3?P6QQk~Be`s7| zjLf?R{IG-*KgQMUh=f0Afe#w!jrM(2!ma$%Q*@{hKhwl5e=n49HLmi-W(im0D)?ao zy>Z+l5^f#$7ZPqAH&xDC8rL{qToOKJzDPLrqj(_!HXGpbh=+alXr!;yS=gbQT~&Y<#-@7e%+D(#uNQq zBfHU^2X$26(%yofd2HzHyt!uW-ED%Q7GEbDf0HA(4*cgO$ESYG7wE)KLr76u8Um2J z!q+9l#SfRz?^sr>s#}2HlWW{AsFpz=YkXbBUHJXUhVCvOK?OY;nxIYIGT&n@jXo1x zXwTOyf9`I47PnwL`uU3bcKUg|6)V=QENk($`Wm}db~JUjV)z@}{c#(#c6giWyE^Kd zIAcN}uqXHwRUwN}!et!+MR2oYvD~H`u$ecowX?p{?`s_UyP|=PCU2LQE#&XQA%&Gz zK$0?!5+n;7JG=59x?yeH9(&;%w{6_tOTCr-yQ$45-pq^H*r|TG^)~%oSFyZ);`coW zA~Mx7^>tY$s97@qP)NB?KEY&K5QTf>-5D9j6hg%-RiBL;O1V4=Ag}%%;^v6V z$LO;)igz*`M{}^h1CYiy=GN5GC4;YrDu$xQSJ!7{d+OU*YE1N^lZ@Bjb+ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k221e3d1cbd479c8444e94c698e544677.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k221e3d1cbd479c8444e94c698e544677.o new file mode 100644 index 0000000000000000000000000000000000000000..6e3e0779624239ff123baab0cd79ad7b74778d2b GIT binary patch literal 354304 zcmeFa3w%}8l|R1kx$m2IfRI2SHz7cH7{VifFE~LIe3Bw+Yg?1>5|l^033{Wn-b8D$ zbtE|bkNTT_YOpUZN`KVp%t)tGp=BJ0wicOA$LaJp1xg*H)>_-wGGfZ-v;uI_v;8hip** zO3?OJ3$f#Puxj_uai0iQP5lL)FX8@Agh4v8DjED2eZP6vx3a&<69cc`U}axh7Rz2oxjd-|{VTDJd6Jg>qtyW#o% zsrk?MKk>k!{^@v5x$LF>TTZ>&|HSuy)xY7@SNjta=VdQ{{?Crw^3rFI3{jNhx&KF^1c2n=x*8b3g{ZM*#{pya?73vj@*CVll@mbzo&owrsw;2Kk$S8 zm!J4m|EXJvW=?iJ?kG^xseUz)Qk`P8jn>7N4psn34DfBmb!?Eg&tuR$|Edw=}{N3N**?2&&u zxu^d#yLa_p{LjU-8oS`q$UL z)Ias5UxMc1?ESlu)}{xK{L{v-^`EMTey)DL|B7Wl?9WblrT;U~PkhRPY&=kPHooaI z%U^!=iB!D1>gjkSvpjyFDoy2^JTurBLm36E%<`2YvwT#*9yl5twVQDE>A}X=FA*>N z%z{k-cQjbG`#5kvSGHmQda&_1%ib3V5Kg{7ZpE&oxUH`aHXgwH8jk12c`IAewn@*E zMHO$nd`bNaZ{8KizOmWLzIln*@$wB*LchH^m_6GPf*poow~_3^U)=f7f5HAv=4DFb znJ+BwK6Z9uBGbM6>FxGFT}Tv-D$XpQgY<8{ATS`7?h5Vtv_0^0NndL1tiZsUseysj zHGzS)|LD26>FSq5zX=9+|2Amt9tZ}IKe2nucfteWaNoprsldRld+dSE8}s(9dp!JR z=M{Kv!n?cefwrIYT->@l{N}n_0N-T~tXqrc9eCb^X9u1);<*{m>+!r5&*gaDjOR6Y zuEO&QJX`R*49}0^xd_i2@LYiBwRq0Oa~YoXcwUL;Bs`bkIUdhT@Ek)=@q+mHGr!oi z5O5jb3-K()a~__Bc+SQ%f#+xOTp1V`yhprnJ9N@&P1~{hJLm0K({tXAwTsT%akF*a z4l4h_v0~pbv>AgV#J<<_#J<6x*hl4wcstguymVmw+Dix4MFIosZo%^_RoV5M@Jvy7 ze0Kfqc-~3jaoP2s#IuLOmD%<8;rST~kIk-s5YLAwJSMyTFYx?d6t2jw`vRVa1hp5R zi}m#^kGE!)C;A>+zU9X7K<8%?`_^3$e)CkUFO|IdWQg+Id1qo@>+YV5lUJV#wSJ}N zV(8#gt&75MQvO=4$val}Ox>|&(bOGlt*JY1{>~I9Z{B=SK7Sq@!?sTOTmRDHfpw=A z58M*Dbl}e82u~`{uIpbsu>RG>1GoQf@xWcyr2|{XL(Wek|E~joC!W8>^L9KFct3S? zc3mNKR8y8+UkbPk_oPwTbz|`SiptyR$Cm%~JymDd9SOZzXZ5A}t~nVx8O@ZoZbBYl zqhxcfKMtvU)S`?g?fCKp@xt22C+)boXVQ*Es*ETfgQ|RFJ}R7iti0?y7q>3@&c!RO z_@0&5AbbUdV|!L!kMME|NB69}3E>+l9NEJsQd+KR z$TeZdn#U*XSlcsU$IUAzyp3F=UAfATW+|l^-m`K%!eb~L+Ou*J!Zj2Q?paxn@Kg$0 zQZBSj@i(?DzY_UAgnL~g^k%v=G#~=|FJ3SDQtOWA@3`fH;J(g9!F{*RitStXY~aoH zg`t7_iMA{>(D_hs-};{5zE-Q};+D9$txbp(FQZ?KqfcDlbIHDi>fIF3hv*%i>%r%8 zJQMNE^1ABXf0-8$`>NuC`?`2j=!Jy=v`0bfg#^M=&;}8_GSW1V{*RXIrk8HYwtxHM z*~JC0Bi}Cv#g3^> z;)NEJ#jQ0>1BpQC*=gwajQ3OVzI&MWvxt7gQ2GmqewgeY{9yPmB&^j|XE@U#haQFOQY|$?qoE1EAmG z3Z z#);dyUk&iPIP#DfWe-dbBtH9Cf!*JYc<~Z@-@*d4qyHP>IKpT@oAx67A^ZHqb@usr z=h+TAMdpM=Ry8wAb%6$LY93C1EX%O z{##COAIaMEG{Pz`)%y^Bi@bCb4tYr=cP^iT{45M4ep&rZu_9K2^Z{2vz5{^kk{;Pc znzB{D1ljOT%T^EBeg@eBf5*I8wqq1ldJg102eR@^tLG z$WwzSPnBk#4je_E@J{Dx8uD~E@>G40W#K#p{sCdi6J)P`7GagAz&|3a@^q5IDo@=M zR(YzXFzLOw&(Zq5|4*UpVRmQvi1z{y$6NQz0$B0JD z2mS2?&MRNjk0D>XN4|?p`F6el`S4E5*8usdAYV7)Y5Cgz6=5Y`_lpQC`P%*sVI|+v zpCGK{J4Wx7eBBiG$kzlquYAV<*JV8NEjHyl_5;X=cUr!gkZ&R6tNwT9E9=Nl5hnRy zXVw1)VQqi^fv~o}pCPR6kHXsihz9a;J6i$!18xe}W$R4!Q7J2>y%1%8fMDRbZXbi( zy&k!*GUaajC&-O=TJ9w26?R#DoYRwX{~TeZ*Xk1pE4c%|Kv?PZB!!jS-4s@GS5p{r zf98_Ebn7&TO(`DcNp9m`%cb-I8>-J@Y zm5l37A*^*vVXa%D(PbaFRcr zu#&IoR|qTls$WG|$rt-I!b-lK^j^u=L}4von|&q#I*FYs;1hj&`O ziIDG^5b8Fir|b4g*Zo_Bwf+4M!rK1c zKv>(~n+R+Bqp-F=qS5jN0%zKQ^U4?a9ppRhk#DOh-+`wfAKq#C&WC&pi(r3G2r1#qXC=B@$efK$a_!!=KW$Z@WI??3%ocEeC9{U>b zc&BB&05Y~f#sie5vca7}gq4g3LI^7z?hGTWWZV`(Sn05u!b-*iM5A?BVV~&+&MRMa z6zLax*I4 zR`ML9uqw~zMk1_iu8qP&z1AazwR}zD%mL7Ok9S(W zd5~`q@--ozman=VVI^PF41|?@)eQ(M`C>B>R`Ttn_e#Dd3Tye$2M0jsm2W5Dx;~G5 zKQiTOyB~PG)ACJ)d@UH?9i#NL{mnvH+uv-2wf)UOSleGC!XzL1xF!l~`y(1HU$r<> zJ=c`)0N}d4P5VxLR}47wDnB)4?EVyJ@NS5Vi0jBW4`D6ie1x@(=Oe6TyZ~XP!zK!A z8Holmo_avx(r+hIre*Wc_K&7Zs74QSI#EH({ff{gs?8_ixJjky#Qfd z)(a8VWlisOSyPzg{5#0$X&VpVy;sgQ#IHN-k@J-tIX?kB-sQ@?)*x8ajmUu?lm2 zRq%s}=Ik$vvh&}-x88pok+t!y} z=`zeO9-ST8L35&+D*}yg{3bv9#`dOtj|E5F8VHFMr)J!cJ^5XG;6J~5UH0dHepU82 ztMan1FN@s}&IM}rlRZw$UakKKK8Mm&4+Z(%=^6WP}$#Ii4c@#^f!$q_uS!845K zGCc83;ET<8VjlI2AHoxJre9o+X8_M@@f6v$X9pj~-0IXxBp32da_zq~(D+`wQ{UKpBj%7l zlEr*4&;R0Et+U=jmjRUz%6&%@${6cHcI$N^jd@}u^2_CxON)8n-g^;;_+{e$=>8Ep z^uzB_pZwNG#Ymb{kKudHTk#!_G~bB^>r)1Fe$Mkj5BohQp5^CApMQ?@Sn<>QWt2q( z&u8)Mf&SjEoONDHebQ&QFno4(baOF8R2X`MCtl;s~{{$$qt z|LGj*wLI4){H61}PyS!2$eY>65Ih zw`iaJv_o2Fzw=2S^qGDXbZ^x!=r)k#yCL+>ZO0#~vvEFU-s4jTT4!JPN&h!K>AvZc zF7Y1d?66N+T7A-KovrpsPyU4;``rjSKX!H-=-#Q1`a|vP9-lIg@JXk2HrgkBrBAw~ zPr6#rz1upQ>r<9z(YE@r2d%S1=SZ*n{_B0V6+iUJzbg37IxO|!KaKK1`lL(xqL)CXD=fSKYiYjz`5z{6F&3Upo>}@=2$4cBxPLD}2&j7v6Q9t@bI) zYw$zWCxhf-kxc`S5rD(DB-6pR#EC(QWP3KHEVZ@2fs_^Q2F@y*}x5 z`dV(RCwxac^o}8N9o!!`=p=lldi!holal3Z;zoKz1zOqBA>FI>)7A&cc?yElOo&epuZD)V)Qx@vu&qZf6M)X5}y-&IuebU_ox_4V=?LKA6Z8yH+(>`E-!opBqzn0^`>oIN@Y^@M((`? zO=~uE2$Zp?C9D+>2n#DFEXxwXf?&W321FR4i+a0a&8Cjc zsjJ)CJMLJ2=banYIPkR+?wC1i{>qlwEvq}b+SjaN28z+jwvII|t2;ITv3}Fd*R9{w zIlrZIV_VDG&8dxToerPHZJlkG@vB82S#fF0td^CV(=DslZzRSovuCx;Y;j13iqO)4 ze9C;i53VD5cbz4etwULq6Kx7a3W-X62TfTh^x9Hm;GS zll!8r-gtv*wyJHzhMQAu?d!PGwWl_BZl+q#(5lUwIy&1nbvC>Y%y(5=`^Gh?Rqu;p zMzL{Ta?6rms-}|*(EH+jP=-1?rp@x(_e~YnWy=|Qs{&S?inlG+@j9d*p)1dZ6h%Yc|0CX8_=SSz*$CjW`9J1FuqmD1_vXC6OT9m=-tVPJ7FpRM#P%J0r(>O^d?J5v|qpb+$adM*AASq(yR8)G@7bQ;8!GRmTmopHfooWotrmr=$M{F!@maeV#%4a>*q{Q zUQdtu#`DFZE$cU|PIh#4bgtQ$+_ZCXv6sN02NWoR&IRhRq z_8agx@w5R~3an)H#2+hm8SogCE)q)(xKP*zTp;EfaK30T;5<=hzzLBw;J7F^;Fu^j;HZchFgJG| z{f0%x5~l-}E-$o;1{@TbfESN?VZZ^A5nenS0r6WF5@d6MCbX}ZR1e_8d~ob8VOQ^*pMcqD;O&T_K;CSeNwp&_VQl zN%`D+t$glR&%M`4II~hd8&)YiK7diY!|K`lmlEEqo`==*sCtT>ivEl8DR#?e!(Yi~ zyLx86A>m&2O#YjMi+>`YePM2r^KB=CzdbX-(Mm_t~bH92XRnO4O2GNR2fi<&K zlSD7>J_(BQ*R=+lAz&Yz zod)zHHty4Ko({v|*$Y>tw-9v}ZhtHKmIkc0vC((*Rtk|EhbQhn-0c(4*IbKtEAb9@ zW(}SXgBI<6AMU+<;D@^v?SC)Y{v;f)Ocvp%fX99Ghe#8*I4ndl?j-I8+&1oJ+l(PTZ}pg3qr3 z4mq0?^`aSOgl}NAbfv%{SwD3*M(e(+eE#y^rAe1@u6Vf#;WTKY@3+ z8@>)6e-jXWe+T;gc=r#u(WkZk6tr06ojC^D*C!QDsq%XoJR^8OO^xQkx_J?>`Q$zS1#yA^jk?hNi;+c9x(2vm<1NuzWr>C)!qt|rYb?==`zY>)36^NCMI7A46A=gZ ze%!^A5XQY9cQG~@WNgj$Vd9V30h2zLHBMydMv^2%?o?{0x_Z*D3g@g4-2*TES@r7o*-& zeBze~Cgu#GlC#}6QWvw|so;@65c zL-QkiyMjp`!rNcw?1|(&2#WkRD|o4bTNT`{;D)bA`bQLOD|nZVui&S3d<8$J;C2PS zsNjr(Pipw9lK)u+_W>sP%Q0jTlpoY*4O4!4M@pFFO#}+rE(lNgi2}}`&J!&0#e!_b z#TSc=f^~ed9~WOJ8m5`?$);RxHZX!t)8CV8nmkNz5! z8W-h{;0Cl?D0%U;rtg<9$xHaw-$?qC3U05kWcqC1s9y@E`bzpas$f~aLZWz#q}TPU zS;4aYK)(u>^(7+qD|ji(Qz%XwaIr{^mHafm*??hh23#oi8Zhk1fML&-k{{)d%G*{j z>4)HU1?%$fQ}8avqx==D%U_I>{9a^yOf)F?Ifj#>Rl&M^dvoCZIq>NmIEntlTmH>C za4%qQ`Df79d&@UCzeI#8^F{JYD0K8mdui-aoKG_Xf=Zh1(0H_yxPlcg21AJ$&nRbt z2+vjTZ^bgYjkr;F=%PM?p1)JiH`VhE^*kua1bP~t%hGdri*w%Vxy6IT440;FR`bw{ zryImB(H4*p9LhItPs_>A1@S*=|G^5^ZsB|9Ip~RTjN}2z{UiAK&&Snfs!c#F2GOTQ9sBF8i`*m zrKe_Zw-LTjd^w$n5qU^FSR{_m5D%n9>|21_TTkbSLq)>cYO!8L5gMEmMPf%PbP-Yz zgZqhoyQoNu!7$+3GAQKPW#ZEvB49P+MWPt|gQ7G&y|zlyM=AcHR1^?haHxMm{6m7@ z6iN9hJ=O@N_x%}?UjxV$za?AxVmmp0CQqEI0>20qB`)woetXliLsb6FD;-h535`To^M62EDBt|F!0|EQ zZ}$^k%slR1_W3h|_7m3D4LLf)p1F%o%qvDZpD39Wkl1>zbDuCzw z5Dfl}k64K9r4L#ry=8gnd!LISX0aGNhVp%`NId#nG@|I){vQQi(KEcSOPVh%9B^0w zRuxXd+a3i59mpW*|72JYY{5a(@UtC|Q(ve*PyO{^C_)`l0Q5B@#ZzAfOdS%#A@S7z z2bc^{!=D072B+ZiF3k*2wfsX5~DVMaPS_kKNYDU*H>jvv5FMgkFqc1uNI5C zlLQOcAJyM3d>?$j52JMyYRLyklOQuum-emQL)s~Kk09o$kn$6{+vJ3&+*4t zzczw=A^od>&nQs5MEVO~MEc?O679=2YOX6;XEE^-X5*doK!oh(9z-!1ivlIGW1bQko zv|p8JS^e8jbNh?-K1$VJ@`Js674|{{NHk(-F9PUa@%=%j!0ie3w<-m|)cyie1XN#o zwnh+F0UTe&o?_nf||ZCb#pJ4h|2N|quY<2ruGr?X&CK4^syxVijemI z=Ukt2NE#G@p{&o4D{^OP}*qP;ajXW0JNBhQ~``+P-C`NsaZ#qN950kBgyX#4u;8}$M})vci4Yc^F8d( zX#W}heQnyBP7Eyz<2QmkF}`~M^Tq_zd_p?N;{glb4hU}CYFRWM5XAp5`S;}b3y~U+ zkBcO)rRRsBgkkYoevDBte^ME9#s|#*TOESjW0E^J_irhm8gBf)@84wSp zCn5r-A9%jA2yi%z5uzN=vputInDc)&;yd$yXN~#4-1?R~UhPR2^FDMEf&BX??$`A+En(R4wW#`BoJi3!YyJhKJMR59O%jrK9I zx)Ae6_&G4PxL(Zn_!vLNSF6lYd-j-Rj`uM>O2vxZ`2wncPW|TbxG{c*{CfPKug3ph z!u%$WFDbvAzes|neg@kmjQN9n@BG1|={UDfF=xIa7NgmZxA8|jXZ6Hq3TPX zQ9pXN7P|EZ{FdN*3Vn;h{G{w(w|qYA<;V5i2777`b`=h3uO3Z@*$C1ZQGcEMg8m`Q_YB^R{Hpefi8w&=jPz&iKOVEB2)Bpa`26qp6pqt@sFmb82ZHbCT>mS_n&3#e{hGc^auM2 ziEnBDj^`&8f3zCUZbyGI$rIl#p9qx??2YujU+Wv~TW`8HS`H4{-e`VNz~|BML+PnV z-qmMsdM0}ujOX?-%Hlbh@Ca~ofjhwYJR=x>10_B{7||k>d3?L)zo;?Te4+w z49kdcp?!V5YRLRtZvE;>PZMN+6obn=Qmg%lM_@0-WelCpurij}IW#2nerG-v^^AeTM2)BQc>6P+=ASb=#Ii>eZ z*I+64PokCUWA68WT=~e~&X9|t{q6f#sQgQRoAgBI=S4=2QJ<{d4`v6ET|zZ~@+tD#4FxI zQ+vQ6sZZEzZ5k-l_z?N?fQ|1b$nQhx0#vub!4YSrK6_Ke#S;;Q`9nTx&q@9;)nCZ> zP&xr?2#CZhpocyFXO}3zl2Tm7ar(Y+>JuUsxu96!Ol{Z~!^(0FJtL6E3!=E0`-eA8=5Tm`hU)gUY z|1fobs=RyPe>(QT_Gj7;#&?FjL0V5Inbt(>Lr5`Ek|*EP{TK{xJ5N zV~>?N_FR@@pCb+Xd^kPEwRfaAyXfXE~J1wEB-%vK^vdmiE1bJ(%x0=8#{%gLS#*q#|KtAss!;E`is&l)!E zIV@iOb1}=c*YG>`59avs?jiI>eMf#^U$Q^Z_L0ud=l;cklS#@h+c)&ti1ioTeX917 z>Ki#H-2TFT^3%ei_6jijgT4JQUT=K3OEi9f|5U{7i$$FUUDO^GiD*h#M9&wuKit{_ z#Pe0^Pm4_db4#Tdp!z}I$1%TCk-pzukMkV2U-wI9h>!UP$cs;pkZ&dI_Xzbj--uEF zfcnq=Z(}-MyPri?{@q<82L@{qm`vHogp(6<-2v%HNxdVGucz}KeZ2^;a_ z#1HY&UMGmZ2j=w-+@AqIF+Gy|-;pfu42hS49EtO*Ja8T2d+`m3?}g`p-V3h)zdld=X2kcxR{?+8gI@}M4&LhS zvO({~F9v@H4|y*Fy~5X!y#{Ng!D{_mwKkWKhD9w}eq0PA{(SIL@oS4v-etk-ue%P5 z*0jEZWu$+ek3)T!oUXSVJmzD_A0YNYnxBI`Pfp(}96XnoRZsDGeh=|ce)kfr=cgv8 zLqW&B5WgaAF+ZN~13j%bWATYU!x6SWjF(h;G0TY$c}h}|5Y@-QQ?N&tCr>zd<}b%z zF4lho9^sK*Nh%zp8q53`4&|}GqWHs}gEW5R`VV|U0_ z|EZ+*u zN07eaN?(nS0}oi2cVjw&`7DPX_7V#@Fw%oQqAv_Cl=H8Ur|}`sd*V+{hu!u7`fTh$ ze2wS$+iEF&y&k7FomZAI;84`-e`ZWg5G(IMLtTdIe92n`=4 zi0;WtUj$guLto%8X!}#4Lq8en(Dd6OPcdMV|3uIiIrJZeL!;?Q{u00@{bbM=JM^^U z%1d7g*p%m^pf7RgTj1b(=_dg;`L}|;)Sr)U4*klpCjT1&D|)u4yOSO~m&Yo^ALGPt z!)VdT5A6GP(0k*5vf9Ld1oU3{EwK{(Zglvq#jX`EzdI4%D}V5AvftYsd;3IsK2aDZ*DN-Zv?F3tM=eF#BX=<(_m3=;03ar_;dOe(#xp!P_{ptk={lpe;a0b`In~G1NO#`2dVvM{}BAB{n~{1 zw>a@{o$ZN_@(ebRJcIjDzQ`Z?2dE=+H|K}TdmZRa{cQlh^$x$AF+u6tC&mM^e{YuS zxns=FrVOj{n*(~2->u-c&fzz!(Zdhr*9chAEB)UL`ZL2Y{GJTv}0z-D|X#GXs>;ose!suU6%1o=y>U&JpV{m^{ueA(aSrG)e+4r6?P{v{MkP5!2~a@ z_mlFLrl;lD|BO&E@a!+6yppLX`fIvuDxX+wx<2mUAx|6582`ggi)1&2m6=)n2tj_I-$K97>eWh*#=?w2lG!SNCX9k{5}gvX*k zqsvzG5x5g{*$R$*ltBkBY&GGMNhVy}V!|Uvm~iP$CR~pGnl4+VHyYh3UABV9pnIgt zR`B>68Fb)rD^0liBPN_gcSV=2;!mt#(19nQyP?Zgc)cK5!*%E`=&}`lY88VHJQ>{q zT{huS-vrv{jMLsNNKFb_zz()&LaIHQM)t_<6WcEU-9Djy(EJ18F+a!Shgh2MHvMD% z3sMyYWMIq>^^NdOdqHrAGEcIKl|n2?RnfkENssgvq{{MWzK7#89OL@UFv}nRFxBsS zRee~Hs^RZ*E`4|e8ITKy%E>?(K1lqdMP$G(K2k!WXt*q0T2{P*COdI)`aw28jYs_* z4~N5{Uyw>vCpFCc#Y8v&5|8htXkRKxUxL~f$X|x>8u6z*OaA#&>JW7K=S`&wqT#Z1 z@g$6N7u34-~8JlLNwgYqAu zWp9`drt!^i3ynN!9M-0Ia?O6fmm*E+(x0O78;rZ?p zMa8x0HXgq)EYq(iUCZ=?XrCscd8f-(_8h*QK?e@qZNkAjP1u4vN0+VQN4pqw;0W9? zx@?8d?_kh@3-2)DqD~VoYd7KYjV3%A?gL%6;y>mV1{GYJo&jy?{;~@F&3rQ;XhX2KP4FX*xr|H}0YI`CMy3v}5EKYks94m=LsKV7!MC+9Hez*RS!a1Hzc zx@<*XjqaQ-TfuQ|0VF&lmQ2Oqp3-G2eBKrY9XJ8^lP+7~>)=G|HIdM@FC}ga0bPUj83n?7=gCG%)rWiqG;d2HxcVG2pLs_^-am6CeDq0qo`9a;XQ;{As+s z%8~z4;7$H;$n3=q|790@;)DNHfW7>0y3B)T{;LuH8b|)ifH(QK0Dq~&|LO&v_~5?? zu$O=95)YpF!+`Cp9Ql_3Z}Nx3Xzht2|{({t0!#^rb+rWF{3xNu4FH-W;_`enK z5{F;Yd=Ed!yAZI74|}HZDhk-P6@Bjbzc#(#d^7&}fX(z_;Pyf%y^AmK(4#%_jQ80- zE&$$)4+YxiJMk~V+=Ej;QQu~{;~mIbn!XTtGd>E~zQBoZ&2s!(_^Y?4CJ7jreIemF z{h<8bN8@jXJ>zfcf74=;Gu{K<7+(X<9bYei{^;WQ4EVRD=_!y7u$oVW|7(nw7obAgLu3S zr6PX|QY9rt#Gm|q!gt`@9AZJ~V?EY_)QB8^y1@0nk^cp$!u&!VpW&i1imLt91*u}! z-)4MpOohfDr1*)-u^MK4ILDukC#wv6q^8<{V|9}aI9fkZ!^odIo|p3>W$FB>4Fgi7Z8l}%hH9V1XG?QTvS3Z=}5w%VuFJX9A^JH?7)#Y;Ufey|7eWhD8Y~i z{xHv%Q<>3zPRKWj%2$sE;D6UP?04SZgknoAF7iF2pml`vZuo|CCRlxu8!sT#hyzprFJ6^bKjtP%wHsMmZBXrqHz7qH=blD0n zhWkL5t>8TL&ve-ePJDzx2acnErOQ_M7`i{YYz0qT%%B5LK=V(Rt?)JIALz0bT;0N; z11HhG(`74s6`EhVYz2=;`%IUu;9x5NTsFb5-`VbX5pXh9w4Bk5C;zt)?hIYFieCVK zhb~*e`EW<*vK1Ufe@~YUcp2hHKFXkj52L@P%U1LubfzZHv2 zxE$RlUAEHiSOn;@6+Gq=1|9yzZuvuh3sTeE_7eKX{M*eQdMZ!2Uv$|@{yFaa1j|3- z8b&+#5~KW*snXRZe*U;hym0e3ZlMPs!+eh0elq_`bkB6zD*c38{*13WpV1C}raM2v z`0?nT>9SS)26ujk@kN^$?cn1ceULpAsvc6uZya5Q2(J7E8$IzM|75qlXa4zc@9DBt z`Z3oZV*Kc<813Mr#`q_hDqH5kqdaQhf6!$UKfu*+ALz0bJOTa%UABUga3AQhRr(>< zzPS8z=Z7M0`JN#Au-x)uSX7M%H_ks`%r|j=5~m~>U+C)pAjL0m_09D0khectkP5r{ z!fzOPwm(Kg_>HD8Ac&d*-h=zG%ydAEfq#DBtk?yeBl z=hy*WFSM=->t$kCkE7taq5!WiN-9{dFKSTm@eJk#HcNFoFD)k z#!rHuH-2Nni>LT#pzIna{v0^W-uUB@o;Ut%XkO!&@%Uyw;wPQ>v*7w^Jo)S6Adffx zOqiOBr}SxlV!RW-0j{Bo$M<%~tK-vrCrzhn{B};C=1;~s@#|r_8c*q0AU`@j>3>?O z7fIs(yxL(Onzuf?6C@8f&C~{m$Bc0+k?_{ zH9(Vo2I#AlyjY(Xv4Pg#TT9cEN*#KYf3!t*wZlJVtWPRUPlo#6@tOtwA^UQG06fOaQD?jeJjQ4Afko;Ecw_tvJibRFb>L}y2z+U} z(jDIdruAr0s0b>3l!2ck58AWZG&N@#{rv{@H_YD`^!FN?Jtp}EcVj);K~4{)tHy_j zkM<`_dE@UlfNA_lFv~;Z(X<8k%f=gAz+=23_dBWY7vRrb)nEerN&ZwwuAidspP)Cs z4+Bs7pzq@`$6kWGUs11zigkpgK4B$dHGL13?I+IX;q{m$fSvCd7+>r4dV<&gRgwS8 z`5|!y_8Zowr_cdma=jLRZ>Zz%4|+Wy{IeRSr95)~AMyG>@CU+46Cavy!a?f3aoH+9 zO<&SwD>y!iK?g1xW5R{p)w=X06(+tI)t4??@r!X+>B0r$813NmbH-OAMw|FjRCl^; z#jlLqFI={QN1=MtWh*#uB!do|m|(&~zK5c^(`BppV`+Gg%U19hRCl^;1=D&Lx@-lH zNA;%5R&aGSgASZTb*9T!_(_u(bl{0p-Ei3oKbc((7oNbbh=NfcD_s8+^+f^?SkC-O)E)0I zKH>V0%s*b8H2CFDB7<@H=S`*p)G)pm8vY~v&m4a!}0R*0Anj17=>+0|7Act0mOth~SbLF$id;ULNO z8S_VIzo*0Dq4P8Odvb32z3DueKJ70+`q(d>$M&#G?yqP}+qD^Vf4aXvl!DxR;d#6$ zFq6(RdlGYo$evseFUt25UbWv=DqmOxQz!M@b`iz;d!on1>$zzCG&&G5o%%0sujP7e zUIlxczW>Vg`p)?aXy2)U(CfYL?XswBsXSw1QX$y`mp|JR?BU)n#!7p@et~=Gd{gQ> z*uTe_%Glpa4YuAdbSTBu5pR$`*z3JrAWSOy8rq)`DZI!-|4=HHA0}AsM1yKk(>XAQsI2IXQf}#H`%k+cNFX2 zo%|e!eWfhD-x~7se(u5#>-3cTl;4EPFXe~IFV1ij@k4Q!ALl>ez|R&1bpB<2gPh-w zY5veJ;YlB1M<3zBRu3NXsrg-~@kEuhM;ioGKG=Us@!j$WLqD;c{G0ZZt1sw-_g{h= z@=y8*>Gc)5JaJ$mUx14OWUs3Ma(OBJkUpe6$ntf|JL=?5mp7H4Y5$SJb($aaLHsuu zaICQ114AE0jyxxp$@)eHL;B$Q@b>x5wFgSC)5BluQ9Ku>N?e%q6~qA- ze9`(QE^o@8*WO7VGXGKJFItEu$+7q0^noVH!IOL$53KD)>N|eU;xBifRzU9G_KK6THqrPze`u6Q7^yl_Rsy{gB;m7_GDV+3K8+Y`n>xbE% zvVJ`6DeK46p61qO44Q zQl#1;2``Of9@_s+qn?BE)&-=mnt7j`@B8KDw0-t-Hh#b#;2ZJ^IVLbcQd$2v9 z&gT)uOwZ@_(0TK8zDW)Ah4T_<^jM(w`!CsYI`-FyD$jcsc;fMEAb%u}V0~UuE`4LVsWt<=8oydrng*ho1w0-_{(A0$JNO&~!mK>$ z%fk6VG(OYgwZ^RrrZY_Y{b6-V{{Do7s{jWfn!3=Q zRHv*#rC-o*2OJrs`otIH!S^}lHwbl*{;N~bL7qS*0>;bjCoaBVkmp7me2~Y}j6cZp zNj#tJ(&yu+1k?q77%wlN`Z!pJ{GFirn%E%Cx3G>Gjx$U&5}#n0Y*fN|43oLh{4vST z<6Dv+@fScp+&-!N@b|9FUs=+hp!Eg8LF!sqKBl(@@xjG`vA?DEEQ&|Ze;e{0tb)Af zAfK8aiuJ1W(H@m-jn{Ji={y>Y$1uMgA$e=h>iPCa`B`MPNx>zkA1*!VFCx#=p!mo? z=2z4N3C|w{N>YWkP5CLu{5Q@|$9XJa)(@RG2mR;TYi|9*{6<`4@N4H}By}`@fubS( z5zO-$*iNoteBZmbD-Qa75`U8CJ3#+;3XYLK2YLk`kBNAi&Z9aC5i^({ssc>bpsP69LFhYR{hJ|x#Ga{i$&nZL{@#iHL* z{z3`{?dzQ1YC1pE8^4D+;hzou%T#_TKIfO>6FuU~`Mc^)qF3odKgkcXJPSBJ^!*NQ-L8{=L2rFc*6C&>$bv5pW4 zDCg8_v3_h90U1fp<)KG z5B8(+etMby3}MJe`NMg8EI-MI`0;cQ&F4{DSocC7+o^sRp2d8oUT>L9SvWA2E~XDm zPRDKUc?|J1=CyQu=qtDUsXwLI=uab*9@bMTd7&1-$y6fY;m_Zrsz4v7`A*=gJ88Pt zV^66NpVy$|G4%=hl2nk-C(-8#;QLcpw&xH7_;}hvb9z+em(rKrndTS5ekvr0In5-m zAH{hP!3Kc(`z)2mAYhz-#P$k(s_w;B^3w5GuDtl(R!Hj9@)+mo9xM`}j(pjdPfc6;nPxHrA-nN>L4s}Fac^;?rv*J^Wx6^}mBB)%Qzk ze=>mSd^2pA*v|{(ivnpnU@#2U;lnNro+| zk&-?D{?;Ta|H&jj+BbZ!w~~PM{`VzQ?gc~_`Um)X#`!JakM+mp)ISGh|5#js-$$YH z)8C^@a2_({SAXAu{YJ|PE~oTpej4qI?!VApV|@>QfA!4Y2c7{Z$mm#4NLD;47P0QfyN z^aoY^dpk*X9#KAAKP>i-*dCGoWtfkV7x1TWb`Ir;PpRbg(_7lZKqIs ze{dP#D8qFG!~T$8R)8LlY5J+^dl>LsJnbvM6#CKer@43;-@(iDl8oO@{D+NSJ$(F_ zPQQlms4o@ii7YSoS0{>S{5UMX3B&Vq@?YuVhmAkZ#SfdG@h*PY_*E``*!blVZ@rcL z74Hy#jEf&O{jn~7*z`xb_+jIZa`D5)FLUw3#vkqChmBwC;)ji2;^K#mU+UtAjX%Q0 z4;w$<#Sa_5z{L+6ztF`G8^6fK4;!D)(?a|4R^^ZL1^N3k@WaOE^R{ICiKyLz>m%Aj zgP(-m_`F_oKl>ki9y8h_=R82QezP`xah-R4UPb!CWHI>jAR6@@>no?2_}Sx3{6$ku ze8U(MKYyARkNsX3lYer)vp#cPwHIHJ9y8jEe?g5GkNBaB(7DW~^=&L);{-1r?M0Cp ze{Q80Pw}x}+H)Sqobg_KMS3b90O96;W|bFD@v$J=IUl4pJqzxHyFRiaT}1~>dh|D= z!i%T)w7zn-BmX$8KXT*KdAWz#LT=5PzlrYSR^1FXi$h zdMuc-XE^j@A)dmceXUJbV*R2?-vD}d{pa{n2ha4?@b69fBF^(o;f9xfp?jVh?1A!&0+s78QC|p#fywo_;E(-; z3#o!=f1B_yAbTS5SM;b5_7n$>(19WLRDw~zlIc*ySw9IF^Gm#b&)rW~j`QLafAGV4 z$8xfFUhl}`fe`E^BmG;d&qLNXvVSPg*8%?+oi8qIaKMH9VSQtS`04#vrK!lIg#h{d z1N7&ybQtM-&$|gthPq^Wgzr$lcR)r8|ErYv(*ht`-wAn2QlUvmP@Xq~^!fZZoUzQ~ z3BVPYpCgFRS73dHCxN|K$mhk;`o@r6|A78`2JEYz{U=^e$?F@Z@%TW&G(Me2>2V%; zePjGW6OJ{SaP$Hbj?6aUu=?d>oAibCk;OAje9;^eE*oXSh3A`a!Nn$=Ki`D&E;8ZL z875pZ%Y;XcGvN^pCOj%>!ehpma7Bd)S5}(v*s&&DRb|5C)h|wK{nXT$`08pCo;cBj zCrmKmDZGBpEx$Tm59h*Dr<(MWM|)t@pK%EuZ>sZL7NkbUsDst({}!a;5pw5UI97}a z4i7x3kmqYDKJ0NpDw$8sr%N9#MFS_{gM^Qa9HC*R59h3J47vM18K0N4|I@Rck?{rY z{uSn*KN0PTC;g(F^HK_P&Pyrf{g^boBl-3DDM<$|8As!7UL*_tx#y=~ykwlug8Afn z(yt!xmf`nEC_c~8F#ib82cQw>_5}0ceBP4&zK;2TBBJL>WTaQJH8#!Rhx!kCKChYS zPX>5>V>#xBL4)6m#n{n19(#7-dobRz#pkI3QvQq_Z`nEHEd@K{Ed|%<@s@(u>G75q zrty{+rty{+rty}7Yp8wvV7x_L%Ln5v7$x=ZZ!uo|V7%q%#~rW0ztQ#Q{X5>eFlW4V ze$IGnUe0)HP9@bRcRX@I?s%)*#LvwgZ(T_K;yh=()mZKL&#dq2d^7(18ZVywV;b+w zb^PzyIpeK5`V#GtZzh)BoBsRxh~MbMpOrJ-8aL96Kcgx){(QuD$4g0fJca(LHa!XZ zRm}9v@zEG#d{vPiO9%3Ld`TyNp79jR=NUgK z{*ypI#-Xo*JK`PBc*akP9t!o0uRP-^md7}snB}MO)o6#mXFSF7Om@dlioZF&nhN*H zm1lA~R*Un>!+JiR^z*jkt5kF$`7-)EHb3L5$&mN=9be&ljog1it;mTY+F$05x1b*$ z-{}2hG~bys-okm*G5WqytG=JS<9I6_iIKY@$2(*X;V{%C$5%*?;ydFnqz8Gz&iIPQ zU+VX&$Oti?Pw8h6|K!G3lmEEmtK9L_+l-gczTiBE zobi}v{Dt~4bo}*RkFU)0ERFFNScEG5$KS(EJ_- zc`DMA_^&Ht{fK_97MJ|qCdX%)obj20o$;B1o$;B1o$;9$rtz5ZB)@M#UW~_hy_!DX1HUiC>&=>+ z@fgwDr~$a3U-!I3xgHhcJJ0tg(l6GRqG{ssD)eof?}7Aio=MOgk9p3M06RxN*;c??kxU$lOlSvb z;fWJHFzU}(_xl#=_kz@@obOxF5ZjaUJu95^y({E?&w_np|3c39EXzG#jrmW=`5rdj z{ocm$BRS)>c+U5)Sk8HHdG2|A%s=6N|6+axlexcgGzK@lfzaOwQ$ln(|-@9_Yr$yZH8tNm)U;O)Ja{Lv=_%7;> z*N`68i}LqOz5WyQyxvfL?~>=Em1op=4Es0k=?Vpe4cVyIshffQM8SYqtl;Ah^6wM# zcpOauwO?rOjPo$yf7{eP==Sei&jZP&f7|mxuznTmZKpRNJznpO_ARQ`$Kq8!$+xr% zA--t5HbV^E50Y<*;-}>PnUl1Bw%8I$1!Mi@p;W${exNUi*w2UaFJS%(kM*^OQh8+F zxPU)G`&A?PX92nU?ZWwQ7;rS7+jD0>UOXQfcj&RdGLIZ|y&n+!FfRF#{S9GL-uD`=e$4i&#|3vUC#cv{3}=Wcs(qR7JUS9{urWz;@|x%`~^ALjWpU0)8Rlsm=z$UY)*vpop& z?@Q|Pr~M(s54&Vy;`t)+D$ds-dfv~8_Q}XE%FEdASu>vY&&60CwSVr(JFwmM7l@t! zKeVqnk8irY2JI`cJVE>Ies3Q1jcA{l z-lF}1w0^!)o&}yv3UQj?v6OSc`)L?16d1rrcyA%_1p)&W31j~g&JQo7{*flvo;&r#`zfiL3uRE`gO!Tc%zKI~v| z2AdyeApHt=e+|wD!g=a^eizfDJ#X9^Ss)D%`usl0J6h^r z2Rq>+JnRGe5%_ycDUVOKga3$1M_#ON!}(Tcmw_JzaVGQ+r1IasFwHzDGWa{f;Ct z)+ccL^=K**b71(FrCX`*SMp%|o3G&Z<|cX^-QSJ}sDES*xM0ta?*Vnbs$1WFYScH3ziP?h;Pb4p9(!hIXs{YEpC3s11)lP! z_Me6jb`MICgpNh~f4fFV1SWla9=s6R%UNqs* zNf$QsgZ9VJPwn_2{-LvDdq!N30d`tPmBt-AG46%^E^ zxIjt9yv)T&LU2JK5JmzivLd-}f*SrdlQXcR4tTkT1!-Fek~-#(B3oWDR+Npg3T%)by~ z-+dnY?6c3_=dsT|I=;2YGv?PPnt2r;gZ2jVzyI@~jz^>I5%sgwDD(Fbte^T4`ro8I z&N=m$gHJ*}AC$^L@7rZ5Px4>2yf>r2fddilYqe9JhVUL8v+MC64Pigo!l1tf{a+=X z_tQcDbBAVTGB6PkqV@pu8^R&%hk-F(W^j8d11AIV*Y@{&MuPPGBi$dgH)Nd}|Es0F z4Z{2&PkVrG34p&qpEn?yBWmP& zWK`*A5}@PB;(oo>KWzW^KaKR&nc({-J>%_>H}c7T4eU`%fn0wcr#*Vg!rC4&0m_HH zE(b5h5C`xN{uQFnk$;&@4}JFx*0Z|&4%)A}kq;fOhxRSY@;LuPdCTI#fbbyc{b9Ln z=fUqPdD8Pitbbd!e57B2{lZ26%Oi-Hj`XlsIKNm~MFM^QKRa|tR%h~P0oQN155KbE z;J@`4wyXYE{6PBv{=y$^<+y`KyorocezHH=fIUO|)*kqjMXJ_qq~ ze+&FMh5bNjkKUhI_Iy|_0>6^-dSNf)L3CIc>3d!nmie%(*Izyk{+5J4@*XtN60t)S%&=zn;!GW zWq1$6!ie|W1AeIbzasLA{e?c4o@MdCfK3m6VZWe{^z`fcus}srczT)cULYS>fN* zVT9v5tLww~@0l!n{Db~nEcp$8Cp+bQs@PX9@%#n%e- z`knKpp!dw?j7aV4OC_FsrMgAKZ6VI1f}S%5F18WQRJUn7=KFx7GYyko;CywfhtHld zFy_lh-%&O7?GU)TYVto5Jo{@9Iay`iGtZJPf$1Vam%Iz-u&I1iT(_ zL&F)sn*d`DDfOl4nYG{U@8@8C=hM&guV5yN`4sv~;sLJ0{KtOK4^JHm)q@ZVK`;9I zV)Mfq-G%uF*TWHniyvXmM~X7vGwa!8zgq1dGyY##MRKmM;SY-SuEcwk`33YDU={E2 zF3dLwWB$PcC$>+>czpU0PQTE6$^S-|zuw09_4!ZO;XlUjho41W^PxXLzt*poeyS6E zfUzF;>E8=}1Q@^gIG&*YxZdUjL)J$~Z`V&4B8L7W#B2I7zX)u+q|19T{}Xn#nfXyY zewx|?AICH8IR{Zaxv?!fR@WdLD| z9{_yLfe{}jcuT=!TkyWt0>ITaNiX^l@nwR24!j&Nx`^coehlzN2VMpEAqU`z%M!QrvU%hfyV)34B7Jb1K#Ms4+4J3 zfjg0b2@ID9r0^pVdKMeTW4*Vs+mmGKkFiwbV{Y}6P2R;CJ!hxRze4ms5alp?y za18jn4m=6?bq9VFFivP$AMEE)+}G6|U_8-^pFhs=g@KOc>N&{!%X@>R&8{qWC}q)b zxf~*W;?DdijMPEQ93_#hqUKk)_R2mZ~NxKTz)%lmF7bz5;z?WECj+ z%W|Jv`bY33ng6p4mA)fSdQ5$lID-43OHU`KL$*HDpJ&-7KiUuduG60{zT$Wi}T~5xIKwCBlL7s z-TsAq6j-)H|Da#KT2E>IqQ6RN|B*yuy}h*A?)5L$hwx{0>5HWQ(5ceip}06?>lbsh=2lBYwrkDOI=1H$95A_ZC`Bi<~6C{5nURbb8jZfI4rFdV0yh`M~odI9; zc$M+H#KNF&C>|KIu+-mCpa7WuK!3{qqox=9i1I93=#&1-cSC@bd;@;!5bag0^+g-U z%W_NqdG>D>{=S+Yq7QO%JS=$QN*!CbE9Oes4 zEMC=D;RL|cx3rm&`Z=7`_~;Ucf`tEkD%9=2u$#Ov@{RL|jY)mvXLOSf7p}Lk@VBp* z{7vdt?UzZQ;IrMNMf}}F{z3${PR2Ll5nu1qQ;_wR%z}^~_47eT+uzu4*yyJ(^|E{! zgo@uc`S^T=^%?pwz8bbXp)c3NxdMmJkn$_u+G%VvB- zUK)ORox(5EKcHuupI^0K(+l}j^c#L)zw1W7JfA+u&&?JO`2_#A`sFM7%y*x3R z3HiIr=2!gr1q(}gioa$)0RGK8};9`eg8dHsj}{eqw0lRwy#{08#Nk-yLS<@xlXzyFEFOMgb6_sduG zCF4K%f=w^t5X)err*p0xJhKW%?S{u2G4|EIQoDX*|0rT@NY>jQm|zc2aeJ3h+t z^?2oYn(*Qr}+E04;KzvSm^Cc%ik?1-s%g; z-zL(^JX-!XaXcsbA%D=;FXhdWzooW5&HRC|JCKkxE!$ZNr)ANw8fpwjZF z>VLv7KP&Q?=*P^XHofpS;OUMtf1@{)E8pE)C~rm|Vf=dikMWZ9H~2w*ifv<|C?Ede z$$spszdClLJ@nxy%d!>GT|y7vLo4)%tO&ds za8d3H5mfJ|<$6We5`OtO>gy?>Jp6=G(4HmDlI;Hx&HIhVj`U_NezXhty!h)1UeZSy z>XQW4x{N;nf9yylXY&g`;4Wk8Q~W4dSol%STUhu3KXa2_U`UgJM@erc+pB6veQbYN z5ZkWnU(#&P$$cS#$&W%&WR>vwy`Zlb{B7YUv|rknEm{6xyrR8%%TMgbgJ0+$oqj+0 z5k(3ANsr?n3Hd*E1oyv8dTB3)qJgEoooKHvPuf>$_hCE->EXM;PeU*3D`oulXK-Ie zfABv5KeDuM@?G8wSg8U99|v5J^AZJ@+rY2(gkLN_f&-NPq`&j#0XE&gIL{wxDE^I6 zo^gNHzz-3Ib~muJAKRe@miG6c|4n}Ks}t>Y;V$sYg()8zmi{mD0V?%L|L6EXlZM%T z9gmK`Nc-F6{7Di>|CUM#zu5jvl=MH_9~S)n4H9_}`R+k`s!sbWXpe@a{n*|#u(TiN z8wQs4W4qtL(*GUcrzubP0e-qLs4+0wkNW}3RAccw=)rx4l|1W#zl8GNhyMZfZcwg~`3@Qr?k zbK$5MT+;A<@>?Bk3W5-G1RZ-{=l{PPx+W&gLM9##c5K4PxS}h zQ=ZAP{ti%p`>a!k;m`Yr=sQk8!JoAMvG@=F8F2c3EAIP4eoTJ)Gx`3xh%Zp9f?*Pm z4)$Q}0SWJ-@7Ekhd~o>alN`R+1b^ue_tC_D@GrUZeA3^Uln?%~JryLEA0eMCk2w^5%%4^Fftmm1y@HQ$ z+GFZd^bLA&77qJ@f3+}c{HO1~c={IpT+?wW$LA$XyaD*5m6X1Pe;WMBb{=d5L!{T< zhs1Rs)84Z`!F}+2OVE1$HFX%z6&rbl{CfUrY(G{ny8oi!M~IIA|9v)9?n6@EaQ{x$ zN^-xh4FBygq`a;ae^7Owf)z@9j0^PVqTsVk$QSG<);BXFfF&O5<-Oo%WXcoy{S$LP zknf2h|0$f8LqD{f_DAww81K6QZ~gUdHT*al+=8rzAIcvV;9nVkv2NvWN*MML{$+qM z;0eP%!haPoX5@r1p0WNEdV3l4LFmmV{SSWOyj<`-f-P@!AMfjH{V)ENyw5N4MHdOx|I9y+{%?8re|Dlho0L3=J*~pu z8?DmQ;~VhWXKEO zTa}N4Vn4=7d|fd0R%?^U1Mq7P@cH-F7IR+&?-5qMetKw2uwUj|S?_&Zq(6G73;rP5 zUl=gc|K7^(t-*dZ9|PR4^8ZZcXVI6P_Yft2&%(g?8>{?%D!sfP%J*ygRC((C+e$pN zvI^48{h6bO%BWxaC!fXpigD1Jw4eSX3syFg^tk^CcT)BM?W@$^j{0?etdID9t$gIq zd3e881jxwGQr!PX%5&JX!1!YMdN#`ZxbMI8 zKsNKco9_Leun8dY5=nZN*OL2G;BT+(FN}Yt-+F%}Z+a{LTPeR)`7YY8@&C_nfAbGT ziqL-S_htBR2#x5=rBj6pVXiHW3W)rw=~Yv+$jECF88cG0Q-dT zCi0K=R+~Ye_)-YHtC3!oUphVbjrUYl`@m1(PcE?Z0cQL|ctgR3EEAe$J$C+N=Us`%L?4u}ae-PmN zR{jxMDDmB;z936_kVMM|?<-1wS^03rKkCEu2K*t>ir!q7^b#bD{bjPc)ywvvz1WW; z`K^A{zV4;p%k*=;CNjYK5$~LB75Nc*IkqN4#XX?BKUuaJiJ5B!n(S*WV7f8{rv z_AF_3dF%J3Qvo)3aGyn|U()PQ4n!J;z3OD$6Ix$Ynvr@x{v)P8WID2i$yp!LOMIk> z{YoDJ#sr-C)qBTy&ufMX9PtLEKd~NF^H0eSeS~-hlxV>Ve|lH`nV}!@g7Vr`c}o5( z%@Wp+PI(of?;MwFw8u-pG`0CRZDs2nLg~3z6x5B{`)rG2C{&D z;C1{xz?knbzK+N~hK|e^nj-#cy^r?bKB$Sum?hpTPwp!$^77Z?!9~;$xIZWHFwx$@ zl>l|Twk%=JUnlmd_tpt75csd<{v_zjGdo0B`XBYH`GvecPI|IBo|M7obbP3myx$Fe z$^8x#ApAr5REV}cfMvg7qA?-uvH0y1GGF5TR~c)_!~Q`0D@jjK!C72LlTYo3fPU)w ziG83S{e$v^{W9ND{1yd=BnB_zOfPwH3Q$10?LHC_oSIMeUeA2oA2eppNNA?g?M zP5n``+#&rNwB)^W;5EO6|3WXvt3!{BlJLD&Xxyny}xV0MV2*T}!7Pt6ykzXYE#@+)}qN9d{j)W}alFZu^~>2F6q5_+UR zXVtIe9rCF7mB@>#U&}A>s(hh58pU?!H5%~X(Z97g8n)@Ne#d@dWYfQS4f~7yWKFA7MVG=6lvYX#J|}1J-+Z?^L%B`SJez zAQXB_?hC5?auf~{q<+dX88~z9Igw{?eu4DI4pxG21PQo5^-_rVH>HdnPms@AT*z2{ zN%{Js1#lL7>hv^@&A6Sog^kV;MCDWUDy@>CP18MoBKcld{ zvqQnMUaO-$09&U0%+IO!#6E4`0keN(JVRcN9n6GTL+}19`JvCNAP?xDV+XMuF!-M! zzR75Da0dBNMNaf*Sd#bzfQ3IepJTk}mL$2j*NI1t^x*fBvtbV7Md5iqcv9x?8eSbN znF@KHq4OWd`6JY)IbTP7B%XKZ_CueHFkU>thWvs29Xo>7ne%9!Udtoqrz0BQ0?ROd zu8MD517E!|p65I8mWuwbO=^#Y!>9WzjaTQVls5c<{Y3o552LrZUnu8W9=|Ui0lx5I+84p2 zKhyCi*|5`7m8nfR7BBGG797-B!3=G*i~ zB`(!OUb$Zi`XLW`e4_s$4>qiTFq z!c2w&Khpsi@(Aw{H8$~EfcR?UC;w^k=T-azh)jd%;WD$yLz*q`osE++$4;+LTO+Ver7 zMzr8l@iIWqvrl8cVxR60q(^*AadkEF$M}xy@RjmD8PXpLG9~mM4v7%o5b$pYGld@- zxGgw(8i`H(67Xm0SFl|CJ&YwBp$}- zYVdb$jr2-9i}U5sJ3oK*IQ40*Vepqf4SoCzqPj#MW4u)2wQD869zT`%;}90ze;SXI ztRVOq`;YeU9$vUrO8Vh|=b@Pv$W`xQemYmj{wqLyNdKDXb2*>W@#eCOH}yN;{N^_W ze(gBsSJ~iRq(*!vjNjF`kCC4tJ%|@8`*%Iqzxzev`(WVSVEd!2w4d=<5HA|%msz>+ zOZY9qeZWIk5njXegTQNeg77f#RfQM03cQ9NY~%Y$`Cnjt+~2=Eg86tp-vYdr@(Fpu z`b@d@At@Oot7g z^Z!*y@8Um#^cIi&yk6zv*JJ%*@yNdcu#2yt_YD3x@XG-w_(z+;D(qD^*4X!?mSMl& z;^F_<1K8we``4i(EFSL#1;GmBuV>MJkdIot8j8T;k-rPDn|~*!>lWX}{2xL74L1K~ z935CZ_KxiuOR<2n|~{oPZp2* zH=#Z^|8|`JC3xn?e7UjO=D!)sQ5T=vd=u~u%ilYo9$ma>&c-^M|IS4&z7pSx_PF)m zg}JGlKbPBt@^{$$cS9Q|_)2^m`0eK3wZz3giv8VNk$u)i|SN0a?>qX_y1QjXv zFD_+U_^0SQ@E7kf-?0aU-)ZGn#YffglQ{l!8-c&Y%1=~aQ3*cB3&M@Nv^=1^08_cf zRtpEYy@2nwa5lRI@OBFajDFMDt>H?H8`R^O4oZN*;qHz>jErcXM$8 z3N^@U{rqUNGY5fa+)g~&TWnspjrLypA29ZZRD3M{_8R6t&B3KS5H@D-R3 zehucw_{*vG>+w`=_S6M!^B44cS|)$jY6p+;8p(O6r9Tb^o&M$ex0HPp?PDg?x2D;_ z^{pD8Qh$$rZ_AXQP4olsSg&9K1Nq~8O7vqv6o9R-8QsI_)~xohKaqZOvr8Ce_)mWY zOc>OBY(H#SJe0Hjhx7w+u4w6%`tn1ze{}t%C#UJz41JSFaiWhTf4=7)3@B-@DzD4J z(*EKM^p9bszg4dDDJrO!xTbAgIyP5%PG zp8iAn2=7yf{_92&uV=5I{~|~KaXuF$`Vao1*l*3YkX7^@%E$bUD_u~i>f?Sy!@$t@ zkdI`(C+*8|JUi{f=rj6>^y%|@q=CGw=6i%{|BdsIg#s4%i^Tp;uaY!)dThJdc z@BA7vKA}NFQEb{SX)R@b@dS{j%RH-G!e#3r?_*=++fXrVCFz@^tT0R&rgI`0# zIhb#L4Gm}S6wt!Ost-rI9hmJ^?|oQ#Dedy{Y58kk;@}~Fz59H)a<>onVgAOiq3P?n zT|fn+{#;7lIx*eg*U;%Z?h??#xNXY}9mx`5)FJ8V~xRA89TU{fBt?JdZK@3+I23PZwXv%KSm=^9tsJq7N*b!v%~a zKgQ4EycutA;=T#S)1otaJaIqxP%w-K%hY&$6ZchsAF8g@;|=jRi$4n9#B>qy@%8>R zVok0@1=RBUg?>4oN$gWtBf+XaKaKdZyZ-)z=&r&u3_wV9z8S|WWi6tzeKn%$X?Pg{Kauzi~KlplKUUBzj_}E#|M63ziBAG zFTKB?;IY5Y`%N;RcuDO~<9)W?=a5_Y^D_3YG5^;6E&JCealerB8OV=@i}L_A1^uZ$!5iC&ve}0`z;UO?eS|m?(F@xhwn=9 zuOr33Qi^}Y95#$Bzwkb0j`SM-;XNdt_c?H00?-*e+s_GnMfe9_Qou~#?ZAV8@vrfl0t_hxtkPox06IXyBCq2{-WKydNbes2pNyyW-XD5!NAHgunDt}BK=+?2zYnnL zzo(`D&ZW{*AF2N1`ZdiY|h|Dsg?6+HRJ`-VC8kLfSGZ>jb#_4pybsIPmR{vds21-B8;@u|j( z;wSaB=`Z3bKWe-vJPd~Bcds2^hQCbD@n!f)nB&X5Z))F%3GpNL_-qiW!r+K~v!uWx!&~G^ZlJfvLkAVL7&R1bC5*YqQIFB48 zGtCG5A_}WKiDmS4b0&-P3NR4o7u3Iq7amBwb=<$};(JN@#Fy)(<876AS+c)`^Jkpz zk#qP!AIcw!ACaTnfc_4s|8!iJ^AdSq@`BJ`r*)jzukRO`_S`ekq3&CKoBK8R%z5c@ z=RFUe?=DdGlWf8IW;)(br#Up^mG@gXeZdEQ^1YjWwcO(`geN;a{I?!koRq(x@AF{$ zu))OmB>hcZ`l)zb&=RD^$2W-Slb@u%QM_)b69^1#qdBDLsnD}leZZfJmoIMt68e8U zBIn24KoI`O`SErc4~ie3miKBX?{c06Jo_uB&V%GVx{FOcJr#Kw0sogse=Ozuo~Z9+ zymYBvub1hsf%k@&us;m{UW^wm{he>3eBNgS{F4vm1G82203+i48`c{sLp2pOUBPc-cZ*Kl^9$x}<;c ze$zDe&tyKS=Tj(O_Sek$AK)s75d3S}*?!!=s;cr~Vn83@g}Oe0!M~i#zZc_y++Tu# zC7QpypQVSW^v{cNQSeJO{&XCPe^Y_{2EC}C`yb$ss{aJm$HMQ+mfwYO;Dz7hC-zH; zH}u2a;Qd5=Y@g!4@HgQ%@XT-8kN6?%zln-Ee}4x1$BNz--qT}$7v=oYoUcBE_eZqG z68QjL9?&&-&{Lp&OXPdf>7TtK&qa0qD*6ZV6P*FMc;80mTk`(v4(2Q1#8f6-h0G-NDjA1^Y3)rE&+-K5BjI$zHW|J)86`Id4cLbRo-;m+f4;x z@Y8WmcZ=m~{@^`vK>VnmMu-kk_SA4Y513jeo*-*6#Y|^!=#L# z$j?+v>hwqXkB0!frS^aUV?EJ@=lX2>rY1|`AA>|HKgPS3cY&wlvINX9{P64>)+?}Y z!vPCqe(?ME4hD-OlF&_g#(m^T9s5Sw&-G2%RRy}N&kLZR3e*Rt{5sC}8i39I9p)?D z)UO6#i!suHmHV=&&%Ada@8Lq;@g86x`m+lD0AI)Xur!eM@jiEq_u#uxfQ510!-L_T z>cNPYVc_#hA96m55ADPIZs4EZkWTnjP3Xn@O=9@`$shsUp1+eF`{%hwzH zx_n$WGyH&m%$4#d{io>Td=;P0pX+q_jq%ZE%cH)4{|)mSeeIQx{(J`WP1pX;cmDLd z^xwmEu2q$<<9)btAaGdq;cUZ)^W!d@4|bXS8RpLhyG?%FT{{BJvB%^u5RQU7b$;xZ zJp+Bq_T!WOv-*nuM`-`aW`Dov+f8bG`SP>cm!AuJS$-Y@sptcL;D4LQ%ge_VeN&n~ ztWWBAuPmsdcFG^(_aJ{qaMaMN>HmtR7yevA{{>TiPRX;$|LTaM$H3>|&qM!514I6W zzKgnkRUYm=d-a9E4|IKaZ^Rql;K#T5ULVGH*8(L!9v=E64eva7KKL(UfYkee;BSrg z*4z)pc(~5WAM6+Hvz1rCe9zG4e@o<19I)g)Jee@2e1zq}-pKo~JC8-O zp8~w{XT*F_?3L)Bz3^wz=cCL7`Z1r8`SuL;1=>sh4Ejqq*+0;4=zp<)qK^RY((~D% z?Qh{M$o&hpj-Q&ppN_A)juu1NC+hp|l)mpt^!-mo-*+eaelgo0wUIW__aj02{lhfB z>2GHCHPQ{M{zeEeyTrYKD$P*N3j05*t}VHKlq3K;Cs^c6PVWC zwO==OTlXF#S_>CVmn9i}3(`wtOS2(Do_K&z-*wWi6jW z?|Mxy)(;o4pQP}HUi{yDLNB$qOK-H13|IOe^u)!4ex`^0&<8A00C~s!MJhh|9qNZrP-=c2RHBTTpUZixZ(p%rfYSxipQHZ>%E$Ub?knRbp%3pv z%K8ENO6K?VaBumolLw+k`a9~Ah**{15m|x5L9Af#K1A4^KM{84PlWCKDHj|a%FN`9K7+q) zZ7`D|{o*f6`T>*v>kR#aC2=5@`8D^i#xn1X{t?jAaOfwzt(vrl@b-oe-!`t{LU8o@ z>`cz+FVa7j&60k3PnGloCjA43{;?8%fORH(`FJq2G5g-=Uk&=qe2?>2!e)MFU^}1F zcsrkizwnLGOwQ;l*pte}QX;=^;Qj1%Jqy^uTK5sPHcL{&b^oNGeoBq)7%ep@>-}Uqv^o5x(cF%FX=&g6*KfUwd zrRYDg`7-}m?aY6kmHXjdeHfpd|G*Nc`On+14}Z+vdw$A3Oih+$KN>%XzB$ocqx1vU z&q$B;D}t`*^()HTd2CE_oBWJdX7wNX>ta*Fy(5jW|E_(3zL;fS)+Xz1nXi0A3P|j` zgtu1ZUB$l30|0Zj@1WO=Z|KL_^ObJhKKzgRE#~Ugo3E@%<|~+Qr1uLDkE&%#xbl2O zGX(s3`zrRGBuIO%VBcrWztA2t-Y=)yvlsNwp*>YIUwN@ag$MdVYcVgj _(`F?LF3C$$S9#)3LMu zLVIeF(yy`}r+x)&?e`m~pD-sdoG$@}Jr{i|_8joM^)2o3ySCr3A4AUghQ6!~D*BcE z2K|7o{U-f`@DH@~j|9sGOC{P1vFGoGe%j}EZNEYP_0D)E{bP!LWxqi`U~9ig|5%Ft zF-5=F^LIo4`)R*P|2x@d(hvLW@qeSDU)$^X%g@irUe9%Z2lhJIuMm43ME{Y)<9)N& zJt=!VmbTa3^Rw3fz*S0-> zAMCZWeiwV~oR7`bUemyRknOdl{{w2TUHU(W_S&WY@2I_g_v`Cw^yeHN@0-0&udl0V zdp*zf_0oCT>m{l6^-?JyvDb2)8SRCVyo$Z9zOVM0izoew{hhVGuBPmDLcg}x*1zc8 z+H1Rh`>(RTt_D@lUcZO-T*Y47_PihVdJg@w=KX5VRqVBG&))}o{XVR(<$(4ByuP;d ze?aZEOaBMaUc2=F9kti*etq4D{tq!H^#6K4_QTWb>%p|Wp6B{nyF<=-KYS=z|H%4! zp%jqVYkBS)Exx+F9(-Tz^)=UTgDHER&~Mh)vcCqB->tp2>$msAUeBRDzP)}A?YWA* zw(WU8?DZVlvzAt z9*_Rr`)aS#>uWvVe%I&Mz4KgO_ay5%vDdv)Kw_`ud4_ki*AkHVecKPemVY}=W3j$^ z$MezmYOk$-`}<+9=g=PCUMKxw_6z3Ro~zhv+n)EsUeBRDzP)}A?YWA*w(WU0_B!YJ zd(`!LweiQn{n+Hb0?wy#UY+>Q0fv9P=ii3iryQEA{Jr7-6;9@y^Khic`&jyTM($6@ zZk6#LgTHDI?pM0!<2>Kyc{<|BsrwDorQ$FIdyMq;N%Ot}@Yp|JhT~LjXR7n+k@=rD zmgRa;;*Ukx^EU(R`I`YY{$}tG>cD-VM^HcR6Ucj!M~85q&G@It`E@A%+R6E~@}Cp` zGoGIg!e8FB5B^chaK6px|H!oL?*d zIq^S}^J_dm;?j@vZPNel&#ysG%kkG*4cPjh!9R#J>#g zipIZ8&acrQmVU_7#!&nj-~IVD?vtK#=-&d^^EU(R`I`ZrgTL8a*Vj9XgTW8g{RFum zfcpvg4N5*G{w(e%i2tqFj|=dRO1-DH^WbyA|7`9zV7=$X$Gxx@{yf^ea{a3A^P)q< zKVHfYCnwZ>2l~TYOx@>&K8Jtr1WS)TZ8-ZG*oQ@^?f3jxhP_ z2wMmGd*Zy(zdx4v{Pc$|Tml;RLnJ;q;wi~{LE;bmBl`0_04zW456Jf$e-|2;GCkUb z_u{g=9-!X`RsMwCRLJ^11>)!Ae>O8ymhyf8e<8$=C;ckE$4ndUlOfOmKj>eyAMvO3 zdArib=6;&Qe|;74iSd4d?cbMG{ALocY7yS=d|kcAIRl3g1*`Ws5p74q1&Q~JXjmF9 zY8dZ#O8zztmo#k3!+V~BH|2FH_;La9OfCex?_%Payt987`m-6xdniUew;j{gi~R-p z+#9tuQ29R30}{UiJaZ2HH* zx_@B*b{^ZOE4Ugz{QAL|hE<+C9r==SqE1V7z>MqZuv3;&$<1J>Kc~iT zEnX^9FcZILC|;h#H-`McK4gPuL%qJBKB!cxC|~p+=3f=uzo$Mh<(H(6;3%T#-NnsWU!n7K@Sx;j`9cOG{0)4oyv_UQW?C*wPE=Z;AG z)P0Tk(Mp43P3ylEaqkf~e@_$lUGYiz=as!=d7Zkv6>-PoZhG@RNH6g+|Lv6Un2J8e z>nlFyrmx3wB}m#Q_L${&==>|k$;Zog?;1XkK7kY+S9K2U5gj= zy7(Rw&tB(W5od02>*xAS;*)87J^p&7;nY7AU+0Wp@~i)}pTAw+cS**N8t;e~q5FT} z>u&mbe0{~v*U=w*Z-zHiB|rLOC?1shGP-|E{r$Q;*vpEJe+!-ar~WMRA5!g8{Hb3C9_5?*>+yn4Cq3JT7nMmb-d|Mw=okm=&~r%De?aQ5 z$BW8t{`xB@59v+$EKj9>T*|A*i}C(o!k_vS@;mA4-vQo%YjL?S4%n&xahIR9xF_%8 z>+wjhSAIO$?!utwAaCfJ{sR2CqEGR&{ypRe-q2r*`- zjmCjB=~s75;5SZ`>pCWOV*){VZUoyjt9u z9S7{tw_4>tpy}_(xcGWJP;}|7#qFVsNByf5J!U+X0vBJoue)3H5#u|Gd=~m$eEoYU zKj5eE^|&u`>qGfB&|o|Le?8T);dlKzNYCs#Kj(*k%L2}s`ijHjz&h=Hf#%PQZ_=;Y zuf}s_tXr0oh?l169bMEydc^-B+)a8^dL_T)VnnV)tbez(PE>#qPt`APb6 z-oXd;v&iq_Poh2-o-*;d6hA7>u(}+0{ho7Fi_i4502}^Pn!$*V|BZGZpIPkSVQ;cw zH_J2q{~gc=*w9;p*t>Y}`!uIprvI>?k#X^eM)%6u;7xwM7wD(YEq2rMy*D5Kjk1HMe)u}4GiJQjVSi=1Xxa;Vmj~X|hxgyq^jPEB zUebejPsW~oUBOKmuY8{=Lj}Z3ZCKyoy`Djbzv!PMBaGK{M9+^9Z*|DUGrx3zT#)47i~h-Z{5c--!MSvTpr&k1;k_*#5JJM-V5^CP}WZi!Pr#$R^WgMTaJeBaPlZ@#GBZr9@v`fecMUpZp^ z5-?ql7x1^w!FOOj(bqz0R^K$UJq|u#ey*pI@zOuwrXPwQ6@EWz`8|A{gU5Q{px_T$ z{PiWa_j=o32Lyk>;>S9Pf5GBc3x2i5uP74#b&Fpm_*KNqdhG^*Zy=2I2=q(Yw&&H6 zXehp3@YienP&_8^n1xpeyn?XEha-P@A2hAsFHpu%s`y>T-epS(JzO94cDell{qM6A%fF(<3qOL$!orVJ zo<6KM|AoZQHTi4t*F1d}HhZBP@o`Zh`M>JTFDmh=Q@*@}#TJs<@f?5cYd$`ReEe6x z>fR3K5D$Jr{=!Ti@2_1j z>BB0=vx@)0{e$H){+4t6;ryZ&|9Z8X#oO`zYl08$_&hmc@p#|47N1z`!_$j=`1nE} z{?!E*Mtf`VKOeHNl=s}A5C6(_KK#oAK76dtho9}Su$2Ey*}_u(SGs)opLO`~(RLqx znlF_qen|eQP76!G-PoEV@_iU;mKhoXVN2Ho17;|yf&m?dthCHm;O-a*)m=v|4j)V>2onB z!@5Ag$VZ5JeMtS_wGX4%YKZ}B2uRk`?Ks-#~pEK{3-3nOZ zFUG;Ykpk3t8rH||c@)m4I=%CT^!`ab?nusSFkZkPdRMFXfq2gh`wyw}s*-!Ygz}5- zc>&<4az^M8dj+_VI#0{H=QqIT-19HMS@*n+^hAmt*)Ih>k#`;gyezkMyhPv&-uX-= z^7ng?p4(ZreSoC{CiY{0592L;{)l*!$$1pxy&=A{%Kw*KpQFB7gw(J+R-b|F2-*WjzFc38rU(PI@`5kob-|{eMoSC&QfdavJ2O z{|~A3w0BN=Ih}FS|5+-1)laYc$M}16cmGi;eZx<$`zuL5!vo)QkQd!v9zZ+v=>Boi za|6&#kD=qF*Zq^EKa!%S<hZv>Dd8pdT_)^e|lDWcCecs z9qgpn@{`a%^Jl5_Rh=I56Yeik;@;Jldg2l-%sp8D z#n$)l9t3_&;rHwORj0l`e+c+>7N5cUz?NQJUkimveb9fI#6MH5SJ{5{Kj{&?jVGMo zDNhz(oK=3qE59@gU-j_qv+y#0Z229t@EjmX{hhP$GQMs8u331|7ZzWhg(ttHeE6q5 zb?jmA6E>Uu0e(Zj$a&7h-H<22?*sle7cb{e34Rao`&_)7A0_x*z<{`lGy`%zpes0PlGc z`tZVDz`5+bfIsv$VB*gP_Y;2~@h=~zKOgm7+eF`6GC!@H&9Zlr^xp|S$@C9M`f^*x zOJbkeCb}n7`tM|ePr2#87kq~4$5r~UxoQUmh~x$wBuFD^HJ5_SAx%yo(D+}+MkI_NqOg$K4}0i^$EU5)&uxSmG??E z_#FOe`G@+|c$(7nM@f0)FYxRSDUan}4nEKNA7cHN(cT~X?WMl*`tNc!_(G~aHNH)I z+h?l4%ctvW0hl@W%c}g>o${fNGk8y-Dv9xzPA~S6>9syg;|WW7On)}`BI$pa^dmjx z1EWvzgZ-D5hm*lwOux^hKbZ~gHtA6w^?Mbte3~Am-!%QtJM@EJmG1?2u)IB{Jjl1_Y&#qtV8(aS^%)iOxNBx`K{HgH+`n5i^_20<+8%=)Hf0LVkcKupk z+WPCvzux3W{Ttl;v+LLTT-Be6yX3uTWJ%-?0zbp^PWm&x60BwUYgoRP2hP}M&YAI> zrFB>QK9Bz3_?P(##vkN6J)Y2COITjjp?B2ezk<9}z7s5B`eBp)J6X;~lK%Z}N*>P! zLrgzt(x1%+3*7Xx$3s8U_nGu35zE$1|2zQdU+PoX*E~#@d~IC;{mG%91I*rLobzpM z4`uy8`>gG&;x}MzkClH4U~RvxKaKcfOx%t0cKi&kk@nYy~=6(~~|7+ZD zh5Xmzdp-`Z?H&t%9OfJEuGsz9d&OSdtLx+W2sfxR#Gl{G{Wc-s*9l{M@V(1>f(z>X zOZ6URa4#HQF@$hG=}P{$$j?vwZ50paPY6FgaSeW)bogAPsr+j{VO8D`qYw+WQ!;c3n zKXQ~8%;yY0X5WXrCO>Y!0zW7KavqwDm(K`&N3Ot+@$tFvL;MGXALE}PKgKOTKC|EF z$A00*{wwn1xE(Lk4nM{%Kjb}G%vTLRX5Xj0CO_`K0zWvRBR@u!KPuVtS)mV}__OrK zgAdMyAL1_|{CM!Q5gLB{ zs>6?mEI-^H!^(Ehx|_EX>TK1gv5A$-an8~gIbtMcRFhv&i% z@n;c!Jp4rz-uAHN#}_f3clq%p;m4O!{P-=VKTrFF+z{A*?(pN`6hFQ+Pkz9H!5+?! zA9r1qAN%&rg&*R-BmCHR7x}Tz^5ZTnS6qJFE&RAU#g7+L{CLjc$G#Ll?w%(<)Ov4z z{J7(){MfT+F8mOGCE>@OJIIecmLGRux#jZXPT|L$DSrH`6hD5&;m4j7Kkl3-Kd2w) zD<8LCl^?ry&4nN0pC$a*bvya7%ktxPEFWEd>=u6PPVwVxiXXr1@MBksAG_zt4_JXX8e7SahULfBZ9YG?2|u=__;Du1k7pfzG*bN7Hcx&u-UUByxhg+4 zZJG-|#2--jvFR4_W0U2_E!eJb$IE8n$L172zLDa`GY&sCrTDRVp8QbzUGvkYH(r$= z8#m5{AL73#{MdLS`LWUR<3?=1xcsbl)rT8&04}Q$|{hW0A zlUNVS*TUa#;tAI;D^UJ^uo2qZ_V*vyUtz!n68wW@zdD@Edi%X=52))>a>$PQ z;V-FMp!Soe;#$yP-TZ*J?^_RQ|H*h+5H$oVc=+#S132`CfzlTjn@cqkg6hEfpYVLIhKOJ9}>u}1Ojt6qw zFfrv%$0NBnz4X1gVGm!)RUP;g?w8cO^ow)cZ%N84=T-G=d; zz|--<++Ta;ZyIs&C*!T~H%`)@j2jgfJ{1q-QK8fR1v!eN;m65%Gwh3-|7Li?yYP+Z zxCA~GSMm+7yrEpyD{mX@SCal@ynWb(PsLsNaWDU1E^_jpiih(p58t1oE4rZ<@15rc zJ-8=V^Wg5>m`l_1Q}O!zIVb(ecsJ~mi(i+& z^W%R0%@8mr|E+y){!{U2 zzU8Oi1_83^PsP>z86Qspaqx@t=Y0Gwj%U?A(z^xMtrL1r#S8M6{PZ7FJI>@c)8E3` zoE^`n;=#PE4;fDj;}QJWjS09d|5V(UuR>#)_5*+W8V}!{Z}|9~H+cAt{J6!Bu>9M` zJbWqNviQ}+?^y2P^Z7FtznA!1M;$!ityLdxU;yJ&?FD`d1lEObfxx-&&FC-}z7Y&^ z;g2TrT44Pf5_uH3n8;fp`13C-{|Ov%Lch&EFY)Xx`0bw;9A~^OTwLV`dw2N=;&JuL zd9A=WFJ`* zE&a#)8S1*Wo)3lfc-J0c6MWI3zl`&*P&r6zdZfJ%CGDM#*LeEkVpEM))1TAv+7G+> zg!2_=K7;!ndEtk-?-4e8JpcQ!xx<6u4dHljtwZ$EM@x!wn>r`o2&6c8yo{5r4_U z-|pd0B7UCRU%0P>*|N;ugK z^)TkQ^*Bu8pUD0|NB8f16Xogk3gT~tHhvB8LvhCp?UBTbLVm`};WrZ@z7yJ8i!;(b z$uIE`oOlUnFWMJ!0Ej+8didXTu{S3w6XKs!cjFxZUvE}AWCq;^eZ};+A3}K${wP@P zn@wy$6h+Zbud4B;&-b9;I;bzqcnF)NME<~Utk-(5+{SOzhf4pE9uvO{^(lWZ#$yE@ za3}S(#1jEOF#k{FS74>Bl|18pYUod;&sF)AIMc)OWq$(vDCqkR_$kZsFE$q?`r*Pa z!JZZhLZ8+TNUzKVq?+j=Zz}#K@1LRm>E?zkmun)=fKSIC&5sb)_usfa{Y&tpTdiji z52GNk#8*Xs?rLiN&-jwak9d;>X;ItXO8cR2Hv})~`%7b8@QUA>;mY2eXhw29T;8)!zh8y? zr{iM6kDrSDh~#}tdp|Eu=zrtW>GiO`o<(`-_4Rz=cU+H;c0oUVDB9)B4^K1~=V%{i zs)`>|h({7NG)q9wiRQ@gM;!cf&0v@un`r!0$2T2YEc!y*x9NEKVv#QkUq9mFq0e)a zUn4)$@rn^YeGQK6z}x)GMqHS;1)<>Vecv$31r69!bN`q2$m&C(KcAz;!TZJ{&lf`p zcZpwqRo2f#QAWkPqCP|Vv^}T2O_xXeZuoUEDbU#Wuq)9A(w;~jXqjoWl>CO}WPAgEGF}SyCiTG{OL&EZpY3^< zpC_8dVJuLBneAn+ds$Dg zgYa>_PyH;^0Ty}4B7KhSQFw{pt@NRek5-8N`#nfMTD9ph&FOVyyl1>=v%+s!ifJM)#>;@#8<8+X38~co8snYT zlKS2dd%uGOnfSHL&#dy7#A}@nYJ|1@#(aB+faLe9BU$Xn?XdO}^q{{VA#CiASDx4h zZr}1l{vYgx{x7KbxY{04zL_8K5hK!{wzr;r1%A32CH58banGKp``%H&1--Q=Z$$^C z{<*G4_NCe%-6q+!{s|B4Za|>b5Jvmz&C(;nKRtiJ`eLU|FYV9MzXBiHkNDKT6_55) zK}LbnuP?^^{o}yqLj|9T5iSNDWX9vEI2z>uO5oBc_e%|o{lxx#$e+MN{S6OZ*f085 z+lO$nq4hKPeV*eH^-*J4{5F}o24yurm-?Yl68vU8A2#^XDCNoEv0vCMLFrldab zgY|I$ne^f0rldZhCmMD2S2($Gc6|;%uwUBW?UlFQs}JL;-z{%CF7&(hGo0Lz)Gzd* zey99!a$Qot(1-Rq{S!{!pzA|1YW((dzaeQ~x9B5`FTsbCYi8Hyl{c2uC-k5`r@zC= zF0Ve}r$g^_d}F_BufoaYNqvUDUU}8o^?Cih%HyZh=g>PH-_g(gmZZIllllxlz48_$ z^%;J8@;5rWK8N1v_@n)_07-p=v+MK9>r3hbJ=5{W`fFbP;iNvH2lDCAJ00KNKjx*c zc=cg^*uT=jPscm^xgTffo5p_QMh6Zj2a@`PKd8^4cRIelf0L7bI=;1ky9aOW-{rw8 z`uBM7mi~JjIGpTA>O=dc<6HXgbMVvgvi=7Q4E=wi>CFG$EX#S27(6q-Li{5n#|M1| z`;~rA-q-KPp`RW7{@OM4JMh!Zs-xezURaV|FW|V_)$d26M}!`w-(jzWTs=QPdn(C# z1miP3U-e-1UQoKeioZTNkgBhn-Gh(zan||Wtor7=K1rv))_Ubcv(Hbe^U8h+tCst@9C<##9wWj{&c+k$_v&**=5zPz!|(>GXOh;gKpAblkY zLp}eL_5F0*fr)ccehB3z^FO8s|0;N`Mb9@S{mGci{iME&VLL_(Kkys-8!XSQ|Duj2 zX!?WY>G`Uxr<3&u@?-tU^G#2_-TYUO7tDXkP`CE{`eIljyYOAPJlLPI%g>WB$<^&a zdodp^s2LpRw>?3!-jeo#-<+-|?GLlrWd8BGT0aQ^Tt8uchxUVinjTet7x}H`#~I+A z^U}9RW{qFa)2Rj!>1o6KwfIrVF8p2%`_1`3DAMzVi_I>oJA8^>#`|G=WW9>>XRa@$ zeycweH6}gkD<|#8TL_cRdJgn^>rLo~^m@WuKY|}8n%%s4knC4@{21}ukMgnqxq;Lh zdN7}r^`u$9;5?V=Tj3F}?P9b2#sp^ma^1q<|0%4O*a3z=SRdp^*Y~x$gO;p6v43RN zZ$a>rGS?3srS${ExAVRd&Z|j2KJs%!`0dKi#pZPdg}+nA7vx_aG?V(EzffK&THj&I z!~9JYqM`@=jd&e)KLhW;R@q7txx9* zUt9k!F1!0fCz|c$wEmS9k=4KbXgELiH|=wuw?8yn|CYS{gz0!sYX4}1w;z<&zufrect%Nc=Gfk`PJ;amAxbK^xA6H-;uPx zZPtDp^m$Z8YH1(JPumaAUZB4`{rFm$`!8Kqf4`dHetM$cK%cKau-_B=`!VpR*+y!G zUYw6WA9J{9nE8w9IFjRo^5*Wpp!~GHfxf2tQTeeyK>gc+{THD9KXnijFUi3NbrIknMbF_DrtzY%mY9E&8^6=B0ctgtQ^F4FZ=b*>ax5~am z9qTvP_NVnb&Jz-S`eIkG^Wcx7&k8}*r+fd~?!V){?TO~_)%7X*Z#v#RobZ?9Aw6GB z>&sngKiKY<-!N>)EBN>R>dTQ2xW3%_4?$mUwd3!Dp)VVjUZpRWC$Q+t6)rpneK}^+ zTYWif^#$g;*IWG0O<(SI^xG_bIcn=y`f{m-VgJ>9cbAj@9p<}N)sHvW_Fh{*J|BHX z2>83A9~WFhKi;-re)@6y0(ZVM9gi*e|Mg=sKl103{~+|^7CYWP82WLurB~@kDbxNyXyLRo1?F0={x1luj$8rha6}4`mW*R ze>eJWxoyw2_1((oerfOA^j&(tV!r#2>GbcoA1UkSfvf93@V_3Hv!6-(AiIXXFF)w@&Ig75pXdJMZ2kF;`;lnxwe}ycwciN+;+*&J zeCQqaAHjdG|E|3Mcr)24k@m1Np=HSQMb>KgyeX ze?s66y0_t@{*(R5N&+)|5%_6)B9ku_c9ipf|Q~QVM^ElwKzQy^_ zW=EgR(uW*=sy?Aty+7>h!%a^9v_32(=Rv3s=R?)+8vir9e^}G{Q0~7-{6q0C!~GXu zKNjI{_Ga*>#=i~n0eo$k02j6{eQswEyKYNo&5Bl=lGQS zEwDE`4rPN&;!%b39c7=uNPhSq{Li1meHOTj(4S7%^Zu4r+fycsqVYc|9)#YX6r0{m>s%zX9|35>9}*={@$pw;eqVZMhHkU#Jvc z41?P9L7x8IHR_iOs{v<&7EoD%_Xf+J$B6@AiFf$@k>IJ*8N@rrH-uxGvjOyRq1Ga- z{mED!ixz!#f$#0+3J(3`Z!S$g7i<&TCq%w_UQq9o%l!oSgZDfaD1Qjuo@ajz_W>?# zQ}n;SH8^_AY0ud(nEDl`y`zhR+Ot89?JJn}J)3GDLdkjUI~F?a!+R;nZ`;H17zg@3 z5#n_MUVGpch!)n{U@_sad7t(Vzz@_v%}=CY^S*|`cO>}pOy8Hll7Aot4|?$016#fN z!7uTT7yf~sTKqtb?KAfkaDwRR0#NQV0Kelmq&l15#(>TaF@U(4a5Js zDE_tjz6S0iM4Z5y^5EZ;r$4-bPb0oK^+i$Vf36vHQ{O;eiN8L|i`??)5ADJy;-b6{ zJM)I3m-#6`xKAVP2P_j9y!R~SVg4uQWu|`M+u45HXA%5q=sVIA8G7QN`>!$HCMpNS zUx)VIuRoIaVr~6MPy9sXr`q3eLjXxq^BatZ>8YZ`(>44@{UueuYOmo(-txCY=)t9A zDenaQ$x-}_sc-shB=LAndG*O)S&{Y8zZK_Q(8t|>A@nGIAb!Ck#80gInc@fh<-PJw z<~{mqaYs-^dNbZwzUU7rANm935uRB0w&q`FH~DAIQ^8N=->&_Q3(c&4Ul?vbkiS~o z<&{4@l@ol@o^Y9RP!Rnu^%0&}r}}e69AxeNER9||VAu=X z&&s>)hd)O<=vVE*c&bk!J<2!sn(GOnNB0NXqwgcA`)l;Sll~`OwO911+`p6d1VXQj zAIy*H@g49+(D&zl!1@3?^r`mSE?c+e}C_o6-Juu8m| z|DZoZiJy(1a9Kg-7v6nC=*#XNrhf_ignb**`dH)#^!MPtm*{iK8|v$6=2nV+eci}k zd(rYw_z8CmBToWjx_})D#Oq*x@_p$H^PBgAYq&3&!iVfn{r-41oGi)I%dFqfU$~U* zm#5@uQsx)+$xKlhR3#s%Pxd<|D(e05T3jkQ`5|9nM0%7y1OK7FY0pf5*PGqyeLX#2 zt2YasYQsd!XEUq)+cX@1+W9A3@`wJT z{JetlJbb9)mFw}kEWTUjv%7`gt6_hm&IZz#X`j)*%i?`9KNLLrcUgRAhV4z@-Qxcu zFy!&+Lt&dJP`)B~_DPQ48}CJW(?5jw5ZR!laJJmHl^a|$NjBg|TTl;@Y zl&9McDJ+JZ+z)3woGJyJ@o;{E`O}cz?h^d*aOyCc|9vE2A-I40)FI42X#f{ue*LY* zm_On90uO&*w2A`!SHM4n`S_tvM60-CCNSQMIdogJs_MeqqE%xW-cSByf06u=`TPaU z@8kT(Eev^n>d@`kRlo=L<60i>c|Ub%SKh+=Gt7UFO@CnoaCnD>(VtHp+TrBS0OtEK zy1WPH7SDTv>2>^|o%hRZ06!>xK6U7pyQRGU!uFHC&HH?K%bh;Fb+?5dymma`9f!i8 zAv<)a?{%*4E=YKJQ-9B7=0>(x-Jf2K`_q|>y+8dR?@w=(`_mb@KfN0B)tfMWke>5Z z$Sd|sH(LED`pda5O?}A45zDvtqm{pJazDDB=vT-)^#SQI{-3z7dy~D7O?(_Myo$On zjs6*m*UNou@sEc8e0>V#u|ln1>&=@bAYhQ(pT3!R=l=9YZZIVGrw@4iS$klO6lc~$ z^?1{I+uk3HpnkPfa^mN$6MZcDXv?he1OH`ZAILw@Z{G*d{*sk=x!lJT{$l(M#o1s4 z^kIG=_80sSf((Cfzk1UeprpO$%fY`s7%b1?0$a8x8wI2PAA4^DU)6Qp3+{97mAD8Y z1i=mt#9V{y$PTdu2rLp3Ur`cB>L#)a9hy*E7>LQQ#8w4v#0v=*K~n6Ji6)QtN?%E$ zy0oRP->cH`oARclzHXUT)5o+*$1~-h&J>-vcCadcmY zZ67z`doI*poB0Qd?_K)5_Iug>=tcP2zsXP3k1i9x`91~alZ?4X`VINr)N_~iukXtQ zP52Q0F+QOFiN2rOrR$eHzX*CI^(S3x%=a&F-@`I;=kZ=s&(c&s*#{&K^1rF)PTijL z_Yx>Spii}ZX~#3YJ-E+-@but&a_jaStuZ4wGe08dJ0ZU}e)QZ?-*LZ}-qFeD!`BOa zJ!da$H9hAzp_ z_}7yM;raAk>UZ}PZ!6-wd|F}HiT%u(IrZFMYWp=aXQu66AqD(ra6Hyz4%4%2|H*Wt z{{9W5v-sKidonRS$6Oc^xxY<+&m?kRwm_ek0ImD$%(8dhU~ee1^vyS-rqE;Qg>Sb8 zoTano`@xa>EjN*U7ULJaV~$5Jo^K~8wSU<2cXq;ksja_Zmf@77R%r|k*d|6#mG?l-G?JD-I6<3TS}4VK3D20&*OhG?Xu z&f~s)(6^{Lmd5ugKrd2rEsgIrfL^R_G&JT<{qmiqO6F5OoH6#GZNCTi(DzI^eZ>In z=`P(K*4XdI^r&}Haxq@k?eBwo0+pi%Y4zd3J()Q;8f@Y>`s`g)Kf2bK@69Z*-;3!Q zfqjGedvH&P_^@Y}(zm05`ZM^q^pg$hmTs00?&$A9^MpI-yEJVLB5kj)TeSl~dh&X{ z=!?H+<^%8iS4^8AKNg&zh#=1h)CS6D-S2JIw3Z+3bN?P?><#MQ5kKksVU0k43_S8f z>jSMX|NF<#KQD>>e~;$RLernoD$$=WEa^{b&_C$sez;S3Y)^OmbBgXmnfA*eUARY2 zn%e)0?zGuIokDwuJK|64@3*c29^ao?H$9`;G@R`dM}F*2Es^)}vx??N_&y}5zeDsV z;Y-$!arq)K~hR(VlIdJkY;)QVD4N3w@9JhYv2SvF&Fe zXpFvTJ`yjA|M7jOnG5NPK4=-($1`;K?<=Zdvd|`0${;WOmv5MrQ0bv~1KjQ-FEgmj!p&2Ly%A()yX=H~KRN|FhI$&^AFE zbb8SKfVa-T>-1-=J(t(x<7{_oCgOvmai89u((_{wS=%4T-xMi4&R@9O^pX2Q%71c? z@;})tC+{Wy>BgYI3}2_K-Lk00L$ zUB0kj=o3U^{JAh|=;mplXM)Zd`g|>D^kob4*1ZNa`m%-nhJGoH`0I3kuj7aQczj<7 z_hI9{2;X_9`C(An9p}g9^kBZ0>A(7zw%;i|Y+v^NK8FAQ2|9%ApNHuC^644W|LXag zbF*;&naLfU9<(pmzxA>E*FmHHGal|F9~D4)6y>S$Y1BR3=b^_3hhuUUgX~0T=#BD?RS~5X>t$EdKJkFeS!Lj<_CeSytke(=W+CXN@%|;p1gyC#{)d- z1NzIb8PuKcM1P6%DN;Y1{@U6jFWp1^%LMHM*cW;0f6-qB(AV#RbYs{DDcA?I9s7Xn z3rlPJ0PTt&Z6BENjAj}8K&wVgV?Hn0J_t?!HE#O=QBE2A;F4Y*gTB@CLB_tZ^1p8O zfz98yx_yB5OZN*xp_olEr2i*uFQg}GFIWR0H%@!OcE8uoUPzk|5r3k+P)phgPh>C5 z#`qUt{11Kf=Cc>*enDvD*j|`P>1X;n*b8)B2A&+-3$r8oitGh@p;=@v;C#GiFW|Z& z&t4#VB8&LRz94&n><7p~w6+&8HumfVvLE`5yS5jc{y))PzThX&h}YX-FHDd6f7lDK59(q%$v!asdt@KD{js(mu$}?|V*7xayK5g{egN*; zJ}~wLmP@QV_1|QlFs;L*p@~(|nX`ACN*RXCI(He;>_n^8H*`Z=>te@xZ=-JS%#Z z>G?QKL;r={W;RrhH!)weZu${*o(7jW0W?0#JX5@&3BUhm=u}5k9`33**yukDv0~{riJyJ;t#w zY5&t+wMyG_H2;S7wF2wqw0_FzU%HzL6Y^80AU(79&NTPW(S4DBbq~@TYNzFT{q`=Z z56e-&@I?7nI-fxI*-}ut{v&^BzM>PfPM@wHsg(9l`UdtQ!Vee1)tR4ez!1UAuftx% zd~_J2EgUh`c&Uur7=^b5)(p8v!#R*bJ?_&&Xm=08(K+?QkKKh1q` zZvDXg)J!#O_pdeIO!Egb)kQmhfbj?Fzg}lOcfp=!#YIU@kQ{*L_5V!M|SFo*qF zDBnyfl|}B>)BGXw{{vxK-yfqFae$v)pVH%Pz1{=*#!R}{xIqW0`^Q5m1^4sj(vb5s zp8@&KS@}?YrShZ{#i!@HwLCTU!o8f{FUIm{d9^%dKEJ#?HCi6s3Q*rk9^3-x%X5Ar z^3>W3S<1XH>JW2U;!I$U4MC6%fFU&13kM2*%UK)3K{lJ%JWFqqL{AYQ2bbFtW zJR`n57bham^yzVZM*n5&b6kJYMZMo;Lh}5`muGY$^3>T6ij>bEnktxB{kiDNb7>;- z%$VWlPpI2_oIj}|3;;(SGd_>odrG%2Z9|*(g$a{?G4f!1?&~jke?Ij;WKGb+{qJ(E zpSk~qeo(&zS2LeY{qJV){!73A{TR(B=U* zH`GS%m``cyo)J2@nzzTf4(GR6jVp72P+`elR?yyl6kJ zw!a9E@K3afXU+Dd%$WQFwH1}K4{#4NP{SmTHU;QKh|gUR3GXq?hAqa==wMAW}}=^ScN4!2HNt8uPmc_fU8Hk5<1CjqWaQ>21@sJxZ$Ed=GS1u(&&= z%`I)jV!jReY4?{QKQX_yxI0YK`YM$fop;0juo+qU+uAp9uf>F(sk4J$2L5f+-WJib z!gY@uYm>{P_p=zz@&0YtZ{GSE(vS62eW4uggVpN=a{tq{UvFzN_dQ4RY44BM3o$>g z_oH+Dd5QMp7}c)x?=L?_`K51j*^|%9jaJy<28Kn1%HK-4ChegUS-G8C} zVg48AEA99Z<$ojEGuel_ilTnqg81~`@I&K~Xnw)=$7tWC`qOy)9F1q3`vQ@_y`RPM z{~to5k`i0ipbp317nniq8MIlizt8N4#(pf_|LXG_Sg*kPppO2|8!(?t_bcPRebUb; zU;Vw5((yx`-Cy5p=ih5}{jv2Q=`;PI4g&jm*6le_v&;z(>mQguu-EGrCrEny z3UEh#(%-j3g7x?9Kx2NUhSp+vztRe%-&9x=-fYOXiR5X-dL8iCPaFO`$uk!TM1B9_ zO%NT=oj0NWoiAO_@W%64&!GNnnESKu(s+`rGkVN;lJ<{l4yJC+_&(IIk z(4nnQm>;mskJ=Z7r+d37JUYKY^+TU;fc!W=8lT4i-s|5$qrK_=RQG?VFO=U}*V_7b zxJS(}=O4`Zf!RF>o1X8P{a?KPo}v90)Iau`@hZ!Y`e*FHsDJD-85s4CGra!M>%UO` zSnu-sL$5!BJIzmIb&llq{*TK28Ri48Lw{z!H%e;MJ^`Cx?KZ0hgg?pf2QDQfq@ zsnGv*diyQ0|5fyVGvoT<^nW_P|I+k-lz&&#|51K9{U6rm-~9cbu1~06rTt%h&7H^( z?oZ#0{ohX9Pdf9Fux~K`KIV9CMfa`od?)ZUzH8+19q8G6)8*$YVIR!y zUb2MD3q9Tf9{aCxVTn6`N&BhCn!hCb!=1mxatqI2HU%{P!T=KClg_aA4b6|VQ-6r@ z9ricV22V5o(({#A_eOs3`V8D_X#A$1b9(%CDw#u` z^D*onnC$bJ=e+U#{ypuYf2)t{hclkj`Dgn#YEM@)o}&S=GoEXoI{jO!;(R8~58awY z`gDCl{r1LhdVjMs--+?fx;?`+Hv_N7b3dT{+}J+n&1ahL)xllw*Z%VG`oTPrSqJquUJ3Bd?RR@Z_{+DCYsMg`v|*h z@$mfRoTGGI4j5K`Z~TS%H=HN8>-D7H_54Sol83S?P3>Rnd%OMJ%17%9`k_V* z|44t0Nk7kzn)N82AH{snv}k@3>B0PEEzx@Z68u=h*I`FPyFZ!?7EFKK-M`~CF;T7RFf4hoxs$NCfQ zXQK5jnm^Ou!$W=k;^XGLqn7uj{Uz(w$ltC=Un0NO?O7k!f9%ha>)UdD(yT{Pgve!l zpXEJ0^tXKw-p}xKtF_<%Y>ytyDs>2$hf4PYQTlU^|ATu}F`d)v{TVtB7UKLC)kZyk zj`GI(rruwP`iT28?(*fu`6KKPD%Q{h1|4I=eNeQ19<{&0k458cJzr1tk=rwg33?9J z11NuWeMkFTy4&mzwyT@6SJ`PVb`HWuF$TU*Yzl5aoeJT}R>3{JKWy`Fz~J19$5G*?rv}J^O?J zO#X!^zpi=6A<_?jHA}7Cb5fg6N)`0@aNQo3KgRca_){KU*6*Kz{bU~U|7Eoqw8?O^ zSByU=JB{yS)T`_|`#sT{(Lau6ULyrwJb>}q*F&7gQ zl0 zU#q$}KD4jL-1v&YI+WLXPu>LoJ=XuW691`nds?-&|!b%!m?rhi_7DWnhW{j@Nqv6`isQ!qWY0md;Rd^@6$scm)8GH zxIBrTNdAofeMMOx_Lkz$jwOFu9rDA^`rnI`;diOSK7P*F^5iFgr}}nQ^_2_%xXhn@ zz-QIb3E?-ZV-vz3Rs9pf531u1UhYSU@B85W)fb)pAEd9E)hWk6ji)=bzC(Y2{?ynX z=%Z}^vwjzU?@RYTq3loQd;Q6*P@eZ*x2NMO;#*iDzJ&?#?NtRQeY5x0$LFz;KC*A9 z2i5is`gclypPT+}a}-dgU*`uMcg17v^@Dq+*GK2MAA&8T?KPWUu&*({P#>M|#(o_? z|MrEC@4{9FY{%B4rlx7i&%a_y<{zEc)g~sL=f-^`{`qh!FGoJakMq{nBF$_2Dd~K8 z6dvLAeJuOT5C-X`^VvE(R8IS+^W?zi&3SYazOg_MKI~KAt-TJsjUVYFb2+W)37#LP z_$jLu#gF}8W1bHudrag%{Ojnoe_P&-2#xL84DNd{iH**Kl|MfY`^zssjF;W=+lFhb zy!w9+ohP&Tf$^BEZ`U9^x<5hNYsu~NisYjV)A@6aCi$%&QKM+FKmn^s5Tb zyMMrz59NIve*IO2PwA)eY#biu6hN16k7Jh?g-7GpI6VD9=qnD-yKe#MU$>_}whu0t z`TgNrS`$q z^}nU|!PVOvr10YY5p%wzi>}x{a_$>Qv=6S%{wlQ(uHN48?E|`q>T1&O+^?YRgX5lk zkTBmN&Xddejvd~7!1bT+=)8*f7W(TuNWVY6EIl8;ZcmdMbmlv}{sjIhy`5AZw7vk^ zn=jBbtv8jO@2NOmlk*kp_Oz)plS-cFz327n~y4JU_|2p<82+!#Co z{r@%KyK-1x#r^`&(=<)r3&Q$n*lX=kE}zF?B)}eJR9MSQuo8Dm!c!ZDjKutunEOzEy~X;+%Y;w&LGLRT5I)LtLC=r0JfEpyc@|i_FHb4n z-1oU|IGf;vOOKDzdVsIx*L00S z0!*DS?zm6Gt^rGTq_+|B?1TIC^Oy#JycloAc&g9XUt$JmkRR@JzAtJ&2oL*#bpD(5 zf5MylRdjx#z9T>4`@n7ezl`|weLA|nhS2_XebH|F zV!Tdmgva*-bam713-ZeKsK_26eGUCW>v!hy;|teY`K`W;r8zxacFm;V0Q;^*&!7zVI8 z&?inb*Qpav!nVeKmaOXd;Kq$>@$Xt#ra;1_qo(i=-!B$%`gh-b#h+lzV4j!%^=16= zFGT$Hm$GMGq(AVOO{m3Zi?6=?|Gr)CDBV~5$)0E4<9kJC7P-K`GX(6N^h9h3!jU*^w+*eizF&J?fb|G{be34!S{>wT8PP)T|Zv< z{u}o++%!A9=Uda3{^#9?merp4rQ$t5{>iVtx^F~nZu=avd*}Cm__v>U;MMzo^5U~E zejWav-FH6t_D=fn@YOs&EdIE-@4q~7dH$WJF8^8aOJDvA6WEKz;^i0r_+s$^kk9@T z{@M4p^iT0SuNFU}Q?&0-im#3Q@vHdzwHyC+R{q3CzS8uRcV#q4@#K?>7Cm{# zQ~h^5^@Qd<@g$y(>2O{2)RRx#aiaIBr@qppUfFlKX3c+8BhNQ0^(&wI+`d% zK2Y<@-`~9VWo&8hzx)Su@F2uUUNzr=O0)3>sySh9<3>~;qBjP4s!1F4Tj71f#>PNx zR2zd0H60tUV>65YFnS~i{_(ujAmF^}Fk!0P#SxKOIpF9-Ku` zYU6!5<7q=0F+)_uM-nE&Sz<`~!GL1gKvAY479ykV`{ffz;bGt44LEh9J$-OOi9-ej z8#r;p&%So~#pSt|SML6eGsP|6dGRN^Ui{qUKS9C&$+5Qo`oKSY;MJnq@R@~g``<7B z@CQGfIavJMJ-2;l*2M>@wf*jEFaQ3!uN+_27e4>{AQPW`t3O|SH2Uv{Xo2Pa`|>{{ zfUkZq^R0JL(O|6n$3NQlTJh!Ae)t=&zWUjpP3LF^CbxFu#+uo=8}19*+Z!9x_d%5$ zyS#7Nms9({_*46H$A3IiePko47igJAwPC{zQ~?@qd8)bpsoqR)e_rq0+OT2c#{1^B zH*UD0Q(u(9U8pgDs& zp!!j-pVZLi6Hh!58m4;ksi&TJvU$;>r@}k#cxq9uJx~{lf3k01;kWO6t@znjfBbng zwQsK8TYR+e$~XV1_}b@RKK0<`&lUgR&P&DOJ^Sui_}sJjqgZ_L^5tiX#UH=?3Ag^L})B-vj^j za&aZKBuYH~Kl^EgjH5{|{Ez&jEHt`0WosI7ZN(}DizQn)Bn`--|B z?x`S0Qaeu$lP0nMV0r-OHT7$1gBcA7gJ$Rs%vgn>gE-7tOMq#g(Nq_)V&~MIa1+!AxHbYVV(Si^>q!e=+jvT!eA5?TTf(A z;3mA-J>|LQt^2ovV18%aboH}%GYpK)-BeU13?~`_J*75=oij^sCK$) zr>p`Fsyd7cM$v(c5_cz~=s?MRK{)?`+W97g0QneuM~HNUgxud&#+{Ccko$}>?sTMs z+;Kw~Nmvp-9oax?--CGG@}+WbqcnEeo~{Y=2!jkB!(@S_KpjjC;(sMzYcB4)_)NOS=N9-3oIbkCD34tn9& zzV4UnDE;kg)#~8G^~i%9Iq4_Zx(#$KlS<}MuPBg!utCg zMzx)hqP|T0oV z9#Ymi0)fZiX*@y`c9CJE-N-_V$27@fmRuADjf;}4IBB89yOpErYf8+aw~QOhM=z*}nA9_@cz#sb~-4OHuYAdr-#%yHaWsZ2^WM z6?ybfnT%xtQdqLAeJjuzz2!`>*CO81dLeTqF8BQ<{u$#K-x?^i<7%FMUFL)KQ(~=1+4Ax_SPoRIQ8W%pC(? zF|6sWrSPw=ZzNqk!}gu| zsiCVUd*3+O`#NtNS-M&P4L8|y*h@jB*$=6)CrZ;Q#zg%LVRzEn36HB1VwqBPiE=7j z)PP-GdjDk&CPd66ll`|7+Dfkp)#uXmc7N>-kN&1#a69eP!@+{l=agEMuhIB97jBEf zxFf7}Faa<#){r~>;1>POya~LWa1pn<#S6;~+x>e2;d|k|5&kIuDPAuRu@pMGYW*|$93Je?j}4m%j*(Axdlq^%uk%TK?S zX^Y!+5Xg3&qr^o0%yT9E2=xi2Js0hecX}(WfmkE*+tq~3qc`*A-$|GVOZ#Qe8&uMi zm9zwD$?0XZq&Y|7?WFRwB%PCpbE-WtCuRIGzBK14XQc*>@uej}8aQ?@?eR%VavmmX zVrb`hpIIu*2|M&eiZt;CqcyEIWvv=(@Y4Nh+MmF)zeo$iJwc0_W(B$FAwLNin&>u9 zC(`d~a;u3W_d?k)auFtrvx%i)m(8a9S=&3Y3yiQ;?>9kOwsZ|QSxLCHBc!!hUF=(HMIf$q`dX|^mrw^HKX8^=JO z^Z6D|vxyKKM3+X;X}eK#k`7tMr|k^KoWG+mMqMeKFpgR&JohlFZ|X#nyW{_4lDp%7 zFv;EVPamBzsVpY}eTJuW4v9`JQVFcAOF5?q-7+at0#4kcI^?m=^U)`I5au|}^Tn;C zUAGeFeBVKose9IrXq~~%`M!t;JBZE%egPQwd>_%;&jj4vILrD^Jik{!Y} zYv<^EUz~QMA=cDqE6~sMdDkI$o;7&batZkl4 zr9<+=r*jVk(oZw|I459J`UEVEZN$%6fuIwvDc*fxldbewRq{h;Rvo{wKYcPb@@Gm8 z^3wU%F=nqwzS#Xh$?R5!+`Ut2j%>EcQXJR%oMj+(^~byRlTRyWQG#n4SI%A2n8UBt zj)e)E3S~W@Qv;PWZTmwwrHm8O)aEHXquuGWG|lHwhy{u=dRF5vF$Rv@=#P2i9+yLy z6LpWeA)4=zy^3R{L$Ru$R4Y~T;l!GfvG(ZFFJF7fokmio#%-`8@eVx_59yFW?~GnR zqQ}@7#@*UEus-aQrW~S?yyqV4ZA>~Oxc8!gnWvuIz4PjB)T#}r1plb6_Mx313Gqka zL-i!M$MT~H6Wn9@VYrl*j-!tPSr4_uQJpvt=0u&K+>dpKoh!i3CS4y<&uQLZP2&To z@8%a{Q{gR07Pkjl@nt{Z?P=6u6b0;y{b^{KDEXxPkwV8ksPWyO74EPxJpW&Zd+=^L zjbp+O!d%LwF;n5u&jo0lPJGV^d0rIZ?GFoKu&fc`Z-D&N4iTTAshvumq^}i*XZLw# zpWbc_pj<$XRL#?agY1Q z{Q-TBNawehdnBzCm=%>U=T<_5(mX&B}-=ND-bU`o>W^GDKG zQrE7;&$WnmvV>74{WOk>HWf)($3c^SGbMrXv&1LGnZ%R+raC>IG>!*zwfS&8;&{En zoXYWBZMluZf2DRiX1hfA6ZJo(c2}FZM4HPDZj|%mG}l-<&lla#83ZbC?lyAv$MOyu zcXtx+LgKfJDfc&H$9k9Z>+;9)&u(&8ROx;#xxd1_`#|m%rS;>AbI$RcbS7z^0p+?q zHi$ByQuz3d8MDg<>mB)kEG=45h*umAOsADihn_~XvvN2cXP&2Hy+A*C^D;Y}X?0Dn z++p>Oc0A+f%*>YTI6aA(Sz3=m%Q%Tu>u(?^Ju&Ridhdfn>t2(>Exm0kPq%Fq5Tt%+ z{{#Hdj>kWA!eyGnWjYF1*3xkPlep0f?#saXFHE^IM7g0lJ$tz8S8>UEke0mU}AEe}y!J zSLuFV7%pyU#f^B4W{Is1x7DiYv<@&z$h(EZ!F-^CC2czvZIb56{m-rRdh?4w@4K*9)>10p(uBqUP0E89tF5P$y<>ZqlTJBOZ$k8xEJxaRCCgFvhSFWqv~CXG zR4d0Ry*4b%AgMh{OeNa4dh#`7S!t6Am7A=dY8k7Xy0*7zp;ILr)0pO!%IA8!521&5 z>{?tApbj?lPB#*glvAMw$U9Redn>PM%{Ot2s?=gdiwn(2a=No8a|-C(YK!x4H@?=l zA5V4-HmUlsYKub-OtyPtt9{&CsE-qUTGfgqtC@AA-e(Isca6spQ>ulzd+hbF!i*~> z_g#iuf_n0*QZF;v+-W^VmAU_j7Ei9Vymo!ZpB*%_L>^DZOLAtkoTca7$0KLbEQp+i zmvP$JD^IROUeX=(khjsun?jQ|mD&EABrm0Jo+*cM;iaq-rOg_QvTHE8kk(+LIeEPX zBSWM){xW#ICNlxNUZa@+Ua#3m{I#Y{j%3Q@2(1dq7%4}};6qv^ngBjct40&R*U~DI z#9wRLw~)`=Xu);hOpXS{{xfvYVJIq*h*`L){Osx40JQ=eY# z<7kGqYE>c=sw*+Fmy&eSN`}Npy3&e6(hNY--f0=Gq{nk!`mMJDF#cRAGpn!OjH>|S z&y_MW`}%~5tbtQ)B|yqQ{#+?Dldq~X|N217R6N&Fo%yex*|^p-9CD??qm%l5f9Bsm zA61?Cf6LAM*X8Qe!Mf_qe|6@c#;E#%%`(~7px+EL|J5@WRNJbvBFKbli}TNQ%Td!r z=`w3=)fQK6ag+misx7YC;#5cApQDrG^T}=nO{ulE>db$oUaUIv@0=Bw5_|v0Kbu>f z`S<&qH~$#`+1kfnSE@7reqE{V`G3QA^?0+&)jj`u{P0HawDQWNMB60Xmgdj=`)6mW zGyiY7ng2{KqYh@OGym0@e={XgofWBgf>CPON_3fd$m)Jbs&Uns|7we?wm8ax>O5ph z-((SM1b<$(dcRp(U*Swz^P0DqN_|)*_i7)99HRXQR7Cg&f-WK6pAgsRYkMlBI`eCgQ8dzGiiw;bf20xx~A z947l715e)S%>SEV=HG8e)tUdd+{}M%u2vnat&*a3om#!^C#k0~xl-Np?^}4)J^$tFN_Ee_UswE@ ze}AuXb>?5&w%ku$uYSV^Z>=rt=%h$zzl=hgU#jHp= zmsSVU)tUe5%s-hT`l+@k-|~D$RxCDBXPagsuZTSTL z_13x7SrI?~t2>aV;11+!i-X>{UZaC4H4mvZZuM)gw*N_78Ie}@m~^#|(|vMvt(X2m7(W|Db-Wh5Xy>!Jp{(XE{@$0@mzVg?7ef;(Oy6w2zuEx`#RO zMfbLpQdFDh7q&Gl8&#dNDS%!It9M4BNQ8aHbjJ+BnC_UX9Mc^`sWII#${5of;*RNF zU%|a0>B~W6W2Ea$j?)wg!ie%R739j~r7K4mjGd1q?jj#c+(kZ?xFa83sZG&MRO%}! z{jp-Dx@q#y$fFe1J2#(_VstG$J=Go6RIagC`=+m`QyBbq|0bT~GdMt59s5R9vPnX zR~jbrv&8>?l>^=jPx#m2%>Q1USo033vT#P7zdNOZbvvQ$Ok1N=SJE57jC$Z}fK1gC zXCaScx|V4A0+_>_CNLR6*AqRKjH4uD9LG%Izd_Km1l=g;*@B)U=(h=auAt`&`W=G4 zNzn5I{dPg$DCiqBEy^UA(NOn_&hrDoEl{fM_4VmBjjgq78gp~GonGlX4m%Z6*lQYB z&ZThOIE=+Xx(0PoSRABlT&+!Yh-wkpD9JuZyG+an>6wZ7Anh?RAJhUS<^vuQe3XXq z7UHCW57$9EZVE>{+Jz)NRmc^t+waXIB3oRbC*)1Qbr;N^Y6!T9-c zpw^9VxGXN^;S{tWJ(O>bTz(x(#`twB8ROThM9hKEl{=q|30zFSgVE(pWj@9)*t%rx zE9wh+$3iYB6iaoK+n-RM?)TA(?1-GrP&E&xeMu|dx(%9On#oGaO*`UjpV3sernz2^ zK8eTnPV{$4T=|(~v&^(1ok*}*0udMHPEM+H_({m8b%H}rr(ootU-!Sc}Pc96WmQ7|)|#iLXS*;b{vaHU9Xur5fSK z8^04DSs0Oj-ispTjqxeqA7DhP4(S*4deT8K;M@32Qj~>1FWo zJkb6ipP);;@X}$L<1ZVYV;vUi3PmX&=*hmq;^||mgYIEfe$j?L?G`COYQi-;$= zy!?DlE(<@O6Xg|gC5Pk3l~_0YI$E13vxv(H$IoXGmLt=cI43-z zM|ysSN_vG#J}SAB_L3Mew#(4iPH)ykOB;kX1Aa!ll*DWQkSfs(vX)h%$1@JA6vjl|--tM+jFnoS#7~q? zN@LP^MuwZ{54NYoCCwX1%y{A(((~War=%TAiL8#2H5iG%T6#Q^%Q%uoph>xu)RJUa z*PD*>Gv%!luw^de=aC&5ro9119azc7NEAaxvx7cyl3l-3P2M=5Y8g;gSE{?OxBJ!h`bm9Er8JDEFQeHd8BUTV zATf#k+c;Jal4KfBd@}y4rIX}HN>i-+X(k8FSnL709&%i5n3eorN&g_jbo;d4Amly` zGt}d=3uJhbF4MO}*1*K97t#oh^-T)s^okQr z!*v~t_s%XO_uXap=~esgYTs>j^%d+0s`lNbI=+(b8&4lloin%F6j3!_ zk2&+CwU6C)5Oqsr%h&jma|Jx^sd0}E`s%CAn@ax6^yF8t zt8IMi*togYIT7ZpWcSW`gZt^q@n2a#Ela>lf4pLi>pYWfCwJsrNa&+ociV?gsdEct z6Vpvf)6E@xD#ypb$mqb6#hN~2f-x}dGqQE%e(|kH2h6QU^46gEo|OQ0j(aChO5WnL z1Tw`pKOLAOGpSe}dy+wkt`c&Q+cO_<#<~5BE&2`dUQceR%ckG@DWxkMm>D?Kn z94)s^1E-^Xep{WEasEee2V?u~G-?YP^SHBsE)XOXjNhr?m!Id#w6h}GC# z*M3tY>PkmCY~0;2sut_`!@zC`BXN$b_zbAm5rf7jA2x(_Ob_ z&RJb|X6yEwO3S)I_Z{l;)#>-{*sMhXN6&=v3$}Wh{Q4T_jgwzrXWn(oFOo;%u z6~;;l-O~-crR^39P8&(elvQ|*fW7B+^WwkY&(nvg4-v5I!R$ijD z`&G4o_%^##QX1kLdFh7Os1j09s4RZ5RxZ6kc0JO^ckUU@lyz_8r68hb%^jgW$4**K z=iBXR+_FczAA`(uMHH1zsevQiI&U=&8YrW3I$V2uY!oK_p<5JsW#{H$+)qlTo6+M8 zHf0sv>=(+kPOv<`1sO?9oNd<{p30MWZh8I2XdB-kKHTlZ( zJ6u*yc@obp&leeE%aiD8d7^z@hw?1ZMH6hF|HNhGlqd1r^1Q$pTb@K;!}1JLZ*==4 zUF-J0VXSk}MstZem-IN7CtJM8m*|Q0OY#lw;i63KD`r#}=1g`*eskzAtbhDf+TXMR zsqbBY41v`V_NHwJ8$Z(Ouh5@sMcw{Th`W7jaThK1Is1bpC}}wgk>Lb0=r(o2Kc{O( z@a~NDj~lQ96|`^7)CKQW788u>k?KbeBL6mYB7vj_i@0epve@Xp*)Yi+e0r#kOBwwA z?$I12>7g4<~zhOsUzb zsaC)J`s}aZ4S%S5%L&R8-3EqW_5poX&~1V5o)dJ3pr=%Q7WKos^S-M@-{(E|o>F(} zFACgWtjBnKYPT13#^~RuN2VuAPgVcFW!uZtuFv`8_F1jZ)%xsy3wsiuG4cDWs(yw( z!XQsiBj{x+&~IsbrmCM&AFxpY|LVzgT|ZKye!R86tv;pf^Qm2*tNK~Dzp8$QIx^4H zuFqBbxmut7FN*v2s6RhK`w_-+r=1tRyFZ^!-wN{GOTX_kwc8UGY%sr~trMt+>g~IF z|HfO|e)sFUUq8-H{rSY|E#uVKtM-t!hID(M?CW#hdZOPx^VK_1tM%EdA5+^Nsn+MJ zeec`%)%uLCsyd(Ot`AM=_5NyouGVLr>Y^|Eudg{{FSA z-QQNN&whT=d`7iC+Zhem7ovVRCw4P@Le`j5M=_uGtsq>t7bmkC$^R&QR9ZgcLj3eQ z5XNG7#Ws0WP%4jo@HQ}t3t>zV1fQ5?i5J;4;Z;DYY?>&S=u9o6vY-RCpHi;kl+!-U zw;ZB!ZJHuAflID@t*+LUENb|b*856loss{Q>r!RJWT|ZxB!%A2svvkfGE_gHL3;b* z&x7%C#~s?*bALnq>vOrdHNxBwkcF7g``8g@X{f>k8l4XNqw~7_b9r8Pz!L^}a}wuo z_fQs5epve72>P?2-Sgmk9d|k}PPBF}J?(Bo{Ji6XV%+D@6hbf4ZV2Gv>BtoEH0i&U zqQB`$sbDX?XVve|=bKfgxIf>V(SI6DtHsM#-(j-{KHUMAbSM7N0d%GZe$_aDL*cH* zn;!a`9*QsDn{SrUW%7vrMsjJSPD(#tA)Z>K{n8yP^mLh*;`sY5twoql9(u`>eki_c z9sh`pujNZ7nf;c&>iB;(A^pUY>DTp3sUaKxkiDWK-G@J>lZQC{D44rwc@Z~{;Fq4A zQPWsj{H-q{uzX$V?B9g%q`&7FzxwY_rWF-S`tG#)BE3JJRZo0%(eZD5b?&FtGk5&> zOZ`*?k-^7UJs;FSv8bWBez8KLN(~oH`DPD$Nin^q(?7M zxF^POpRaJMfkVGT^?US9>ZnK0Q+*!&c6G?3Z&Z1YzCrbP^jy{D(Qi{bJbI4m^yt}Y zlSenI4v(It)_ZgV9jA2CGgGyBbiHcv=ozZnqwDBCB**`BHP53ns?npTsf|6P54_X3cim!fxSvSNzskHv?7RPchA!B5 zN8Zo_`^~%WxOwM(*W&Cu`*((JDTF%QmSOYmv+u_L&(i5vz!zEIe`J<=*`+k>s z=bFtsf0unPH*eKy-hHdgdkF8;gQ+0B_JqnGLwrTlfy<#P&QfZ?zDMl4qiFbj_B~|Z zUHAkT;rs1-$i7v|@LTM=)4u!cyI|iV_MJ@|e20DK?R&t!hwVGPw1qU5KJ+$(_b`4V z`1LOg)DV8d^lJ)KDxO~)LT`bY7R`*96&!&GpT>gov8sBI$4YX3iu7- z*98-0qycZ-?lbZ>yl=!MYUo?kLcAB@_b#;CcSBFYp6SGI*xmt(08DZDEw!o zT4n{RZ*HJkp!;*s`A!}zIPz~h(0 zuOGi5qIZMFuM@vKe*O3r@XJ019)A7!74RFzPyI6Z_~q~$#!r0)G=4e!I`PZn*NV15r_J%zCF%l-!j4evv^`{B12 zIQ)k2YdnDX4D&f2fbvYCXi1li+#E)K4}2j0yX%4Y@0QQUe~)}M{(In?@!#oH`I-SlM-Saopn{?r zvNYX8Li7k|m<~M|&rl?&%=9#nN<+o#+lr)46q`3pX1)g;y8UtH!1{ygIn6 zJL2%_f@anK9z&Bnb-^5!hmN55$bSHO0D6Mu3E+Ro(iA_1H!wGdKf56hTZ=LA; zttR|YOZS7O^wGX_oqykrX-d!Vh@nZ|x}ZVV3!Oipr-4?NT=zyrbtQIhRC(wFq9gyP zKe0PTPS6vvd!wQ{6}vYo^`PtCfco#cH=r%J?uZ|C&-I`Gn(GevQ6FRX1~vT8hNk?f z3u;xzPYg}@Lv#-Hg7B0-RGx*88v2BGU#W&W3{B-*7i3fibQa+$|A`*?MMDp2I;(Pz zxcE7$W3NvSKkn0wzvk1q<364JhEI3?x0JdK@gLRkp{@U&p-Em{p0A*s@lg7RZh_t+ z&Pj{!yKHEZSG!L$4fMRFJMT%G{Il(=U}_APwGdXly0n7d8h@_4D?IiY|0#e zYW+{|%jgH_{hEDWvhSVnPfrFuY2Hcqad& zlD>zx6n_|YSyDI`vVJ&Z!_jk0@C%l{1Y3_DDbHrumPz~tcG6Aa<E42em4C&Y0lU z`;lJCAD;EQSMYf@lEYy<&yudSAu^vMI3dDezRk8MUgn=D6Arsi^nE)#q|B7J=XI6l z+stxYenInLtc&Lxm?ui<+sk~1zW_f}vrA@+n^38V&-vDGy?xg5IiGoUwHD z2lY#ue?tlXOW@CGjOFbAbyIJD8cDjvK^v}8+rZEE)9q{i^9AtVwz%DVsSX< zHX%c0nB?!QEsFEPQILx2qZQE;qdEb;+S^ckBKW<+ z*24#Ty0&lKy(_zXS+;9S_m)p@-@UzimlIg_Gds3FlHIyx=gv<(^6-1Jou6L*;HMwm z-n?b$gS)psvg^_A_hvWfck?HfE^AwoUEXr{GV|{*EXh9hsYiCV-M%Ed>yt_F6)me) zwXSMyb%0xzC5O6lX>03}?Ayzc2w(mk_#lBW9$;wc>c5Znj2CP`owtOXG z<)BdtyFayMrz6$fZ7W-sQ<506^Z?=AvEchQ8Z#TDp9NE*8YSvK7929&$5gS?h|s-Nbc& zCJr=8R$LmpKDlda3|I=eVz-?SBh`-M%C=QDWF7Crk2ygvZEL%`b-7nv9)4_><9pev zrK?shYjf+;>bM@ZQJd1-v7li7yUHhv|g($hJ>?`u)o_H~zrU;`*yh17J7}=-!n_ZEvl~y>s zobXKiT9GJ)u9Ll+zghOX;YPiub+vy-fiL!Q-wR*YC(2me#ajPP{I+}}_c+W|FeK6G zMgqm+yYd_Vx<6)o6a?Bxi{h@?4W}f2)2&4@#D3hcmMYVSyCE4bCE@AL9qUHA&%YDu z*);maOLY}<|4OIwBTmO3i;A*B1-6glKdPS;1X(d6D(%C;8+o;X!}ur)#IaJ5 zts}!I@vluF)On3{{ok#+KNI_=zu9&1(l+RgU0d08@zPaFn?KCp#Nk~M-%>J+(5A^; zii1>(m$fL^NF^bZ`nR?d!q#{z)Z%4JLGMD4x=V{BDDlsxmG)0~P8IaVvSkWpRuqK( zI|_d}+{O_;^4;Lva9h=mo+#KzHu|GPzNKWLp*;c5AuL|D0%67h=)WVmRw~$OQQ#%v zAuR14;qO*3-^%0JoP%r{7x_b(FN^)@zazI*N}F_1sBscY-^TD6%@L88o__G{-zx|uScllJl^iViA zm_PNuIm?&f+-v#R#^;5z!3hWRy4Nlu<)rby1%j3fR# zqF9y6*`_G*CmgJ#|kK{esAwJUn|X3d{>gTFB1V!qh&SsqTqLCfcQ#r#7K-WI72 z?XZJ)`F#Q(^#cOW{G*oNCgeP3`IIjEr=n-T62!knbId>MEkXQ64*yIUe!T@zK2ZDR z{)_g+(!=eS`Ev!nVEemy0?+&f0zYi=S%GJMo4~ie(=>?H0?+)Mz~?M}t-v#Xy}+j} z{$7D+euu!%wfGGJ&-_k-XF0bCJo9%5e4h<}r@%A6OW;|~-2%`29)V{$_X<4o4+%WW zd060?-zV@a=MjNt{!xKvIgbfE^ZNy!Be8z@l5auEuZxd%b5{)=GO~6%Sq?I=wW@u{J8?pa?TTY=4SD?gk%-C(K;W7GjKH&;Ck3AQ z1%dBanAYv;w7@feP~bUTX9S-4Ljuq3=y`!>{#k)foJ{#fj?yRNmk&Q-z@O`7QaN`ncpJtEN830 zGrvvXSD?^w%=XC89 zc;h|F%s(ygoUTEEXa11DuebH{d4XsC3j)t} z-LSwj|D3>cy3Pwc^DhWIr)xyunSW8>IbEXy&wTYRUmxzU=}lWc>qF*e1fKP9y}&b{ z?)jjH^)2h+xdPAptiW?VGzmQOn+2ZjIzAW0@-x3h;91UA5kB+V1b%^SzpDkF`RfIq z<-Awmna}61IDfbtHwZlQH;M2EZFy}Lc;EPoWL{xg23l&`x+5==3f+e zPS>cwGyjsnb35X5O{^c7pKkW`Ro2Rrv3$lezfs^>&(9Tj=Fby&mUDr?Ge0ZvEN7Fz zGrw8jJFWan1fKb=0?++;o4_-FwZL<^=LDYlYXzRmeZ9ale}lktK5Pe?Z__&SwOk z`6mUQ%kh-JGru74gI52a7I@|l3OuLljKDL0NZ>hL&kH>BhXtO~bxz=!e_r5O&s-3A z=8p(G%Xv}YnLjGQQc zodVDNZ355jXotWvf2Y8owe7u2;F;ee@T}+e3Ow`k0?%?D6nN$z5_p#Lu)s6FPvFm4 z`Hu)Z^N$HU>-m0xXZ~@4=W;(G@XUW&;JMrf1fKb)1fKJuAn?pTE%2=82L+z_X9S+* z91?iuKQHhXZGN5=c;=rIc$V|Lz%&1Xz;ii{2t4yI3jCmLFQWp_{7VAQ=~C}DdXM!R z^V62k`jFF=5qRd$6?jh9Jb`Ea0)b~elNEU8Hwiq;*(~tPUn1}DK z{#t=&IoAt3^Y0aSma{|PnZH5cSv8PYFEh`GUYR|Fpn!x&{TF`DX;4+tHA~ zGyi#kZ?*0HtiUsWSm0UDpA&fIpBH$R^Mb%Le?;I}&Wi%i{853=S@|ysJoD4<@%16w z`5DV+eaQTJf#-5>6nN&(6?iW9c>>S;CV}UCXclOk){_hlc=5G^tPS*~BXZ}us=X7-mJoEPo zJf|x!@XS9b@T_MJ2|V);3p~r&C-BTaBJeEdQGsXvF@fiOefo?|S1fJ7XFYwH76nJi@a|NFH^8|jkm1lv# zGrvjTTWou87I@|_5qOrfMc|p=D)204o4_-FwZOBSIe};XdVyy-?-h9FcL+Soxk2EW zze(U(&dmbP{7!*CV)K8Sz%zfRz;i#?CGgDe5%>XHUV8YzrZv9xWIEeJt6SSe_G(Voel^*^G^vp=R-l@nSWZ~x!)fY zc;=rGc$RZW;F2W@$c3Ow^K2|TAu z;rlxDus&pd+VXk4%<0MqJoD!YJf~}(z%ze=z;nNl6?o=12|UZ$Ebz={Idej<^F=e zGk;j%x!lhQJo85cp7Y_Nz%zeT;91XK5_snGg<-7USkAP)C`RhTjO8=_yv@&gfoJ|a zfoC}v2t4z%0?*~xB=F2{7WhG{|Cb0n^IHU-)7vWW%x@ETPSCWfoFc&UX986jrDxS@>w4;zh2nG<;C-z)L~w7|2R0|L+dX9S+*JSp(ZKPB)iXF=eZe@5c1oI?_C`OgbH%XwDdng4>ovz)^M z&-`-&&vKp@c;=4?e1~mc7X_aAqXN%*;F7>IUzuyhqxD~IM`_Dv{l@%^z_(d>>II(p za|NFD{5*kY{sMt#IkN)K{3d~CIhzHZ`AY{Viyufog9u#=y z9};*j$HM~8{62vnwEF*uz%&1-z;n8e2|V-r1)kG&T;Q2MAn=^7X9S-4Ck3AM%qf9q zenH?_&eH#27&-#t|8G&ax>jj?qjRMbd&J}p(&l7l-bAiA!ze(T=wtY1VJoA?bJg2Kg z;F;el@Z65t1fKb;1^&2|CnxaCUoY^i=kFDG=647@%eg_|nZHTkSZH z@XS9h@aJuQo)CEEKO^ug=ShKQ{waayax4fu^G^%>pw<6_0?+(20?+9h5_sl6FYuhM zvjWfja{|xlIxq0dzaa3eXGR2``4}ClJd6+Jj=gYgwJy31fKb81)k+xFYwI2 zSKwLB4uNO>27%{&7n=m0`I`lP#J0yyfoJ{>fzK{B^GiDgp7~t@&-u1n;F;ee@Z3)K z3Ow`k0$*?CIVkYVKP>RvkMs#V^N$ET%Xw7bnSV^+Sv8PYOKOt5X8c{DQ!9y*e%M%pVeX&WGm(p8015p8Js(1fKcB0?%@u6L{vI7x=X{ zKQ9P8^G5}q<-8>D%vbhPQmh}i9MhK1`i=P+fgiN}OTEA|zfs`-|I+RlrqML9!?-UY z2}zL7x)2E^R^lWip&%pAtarzgIEnApyXWM*7j)}85eda+u=hag84!5)ycY@=E+{Bm zP*6~~pm0G!K|w(z3JMn#6ciK`NTHyhprCL;5gWhHWO(j!Q7)tG&CWAFnE&^E_+@r> zGsAU_%hM<1xvrQzeM+9|nwF<0u6t|ejuLC-U^JJUdV2>1XonJeQ|m$T!0NOL_XWJm>t4JpERl=YA(o zznAB^_vGm#;m>~LJmfx%hR%6NACu>tkIK`><=HtQPmjrOh5H$or%%hXGa*k;%JUpk z^7OQPH=O?&d3si!>&nT~^YUERj68i#p6e>f)64RlGmG-{sysVa4);{Jd&p$%d_)Do_;2O7G7WH z^7IRNuIo~sekITA=vtnBBYzn7+{)AM@JT?oV2to{{IA&&tzt^6bpZ(`V#+;eHn6 z=_Pq~mgVV-@;t|?JbgvJ8_xflJiRW@b*;D%(0GduG1wmds` z<>?)HcJ9g359Qf;Bu_t9{xcdn=OKMep6iOr)5qm`9Zks7WAYE-^&Xd}Pswx6Ps`I2 z^6X5?(+7W+@Zbyghn;D8dPaUK+|R5$JulBWKO;{s$n)H1<>_Z-^7M6ib~fbcP5D%~pBwV@EqQis%hPw{d5&#)`mTI8oc|qp`kp-3wJ%RUkmtG% z<>@E#Tvu0~ek#v7b0$wemuKgNJpEFhomcYoTX}Zg$uz11-jHvE-(RjNPv4a1oNvj~x8%8R+w$}sd0wY&dHSyWY&fqv z^7MUq&iMm*`k_3}EYEW}k*A-@b05y-=@;^x^Oy4UD|vQa%hPY<%i(_B%F}!D z?0k@?NB-{V`Nne`37zweJ}Tc0=ku66Ju1(2jmy&~zbCQC*(N?lJfMF zJUi3!^o%?^v-0#Ad3F}$>9g|eoRg=QM`p5BmWXH%ZOA=oRC*KIK-+g)dfjsx^P@aAy&+GJ9o_->~81{7K z>1XnM9yyn%U&yobQl5S#&(3Rk`i=ZH=b=3PUY^e*J$d?rJRet)^#AbB%c1l0Lq4uX z<>}+{+=mHydQ6_rBXN29q&)kle)92*5u97@`QJ(9n%F|cmImc@9^twDd*X8LAd3HAC=`DG7ZpqWP<=MF-PjAb! zb61|;k!R>^zmHpUJcHT%LX*!vd-jm-8dmiNJBY*$&=ZBp0qoMQjL;9FJJEQXSad~!5$kSu;mtlWgo<1edIX^8= zPssD!lk)VGJkLEXPtVG8A9C{aygcXpj6A&{&(2wS`keetxSu6?`ieX|Yx4BEJkN1m zp5Bn}hV#EEPv4N|x;EwMEqShMOP=1A=el;~=^c5_nLT;>zC1e*^zdEcjei6 zDo;O?XXm**{X(9dm-6&0d3IjQ)9>Wjc`r}z$+Po8o*wy!r{^0xM?&X(qmRn7b4;E- zE+6^J;lE#=kf+Dwxvsc8eNvv+(Ud%WTK+b?-V^fllsxBrTArSfXJ=NPo|9*1UY2N={ z<>|Zf?Ci+X_vCqw`||Vy`EEG>59R4c@?6)kJpDwT>*~tW&*izU3wipbJm<`nJpEdp zoj3CITX}Zg$+9Bu8p5Bt@oZphCZ_D%CcjW19d7k^OJbhoD`*0vnKa}U3Ka!^( z%d_)Dp5B$uh5LCbPrs07=cPRTN}lI@Q(?5xSt>+<}2gmrm(L;f!OJ~mBx`lfs-ynb8q^euVr z+qOJ?N1oSdTb{lvpAUOF^7MUqK93y8(+}m@c_dFimS^XQJiRMl<2;n7pUd-k_Pc&2T@X^7NQI zJLB^7NqL^*lstV}z8gNjB;@Hyd9EuZPfyEpT^V_LUY_fkk*62rImc$@>2vbzEXmW$ z^6Xrcr`P1!S(m4;%d@i~PjAYzb3>lKDbLQ9Jbhc9ojdaMwmds`<>?)HcJ9g3_vP7n zAWuJ*Z->{{i9Ee4&vl*3)6e929i7Y5FXT7Eo=bW9wLItijXeEUo}G8{^m}=B_T=dg z@`qu6r{uX0)AIC$Jm-8;o}Q9tXIh?~ zkv|FdGb>M@k!NQ?o<1wjbDWc>m*l(Q{4dMX7v;IGsyux~p6jZ~(;M~)V<=J^8Pd}Dt=ZQT1OrD+R^7IRNc3#TU zujJWzElxz8*^n9a_gwFZK>u6M-J|^D{ulJ}teL|jd zJ|<6(%d>M*o<1ed&S`miLjFALPs-EN@|^P-d3si!=bn?N=jD0sGxGE~dG13=o?e#c zoL`iuSLNBcB2TZ$UxoWwm!~)7*|{N4-<0P$w&dws^4)O$Z_Cqn*~nU z59GP7LwWj#(OFPoI`w1 zLgzfBkIJ)iOr9QD%(` z+>xiZ<=MF_Pw&X5!~Q*a`hh&>{GmMkNS^0@EKfg?=ec*~>F4s?hYNZ7r99{Sl|21g zo}D-H^jrB{xSx0O^apu%Mm`;$dz^3dkA zT%JBH&vhl_=}CFcnUp*|Ezi!3JUuJV&YV2GAkWTOdHS3@J4^EPvOGH%<>^&0 zPk(;MeHaVf=Vw%&bADW&J|WM}m^?i$Uk>+kQl6fWXMa+jo|5M|rse4w`EEG>v-0$u zJlB<%r_ac9T?Ki1NuKK}%hMO-IcKW!^c8t_*5v7Rd3LVL(>LVVxhYR?$+L4yp1v*5 z&K-GrTb`Y}^7MUqb{@#n59Qf;Bu_tMvo?ejeg!?%wPoI2R)64Ru@ZaMt z%F}D|+=se6eO;c9lZHIKDc=pBUpD0FoATVZmOOn+p8K{fPv4Q}zP07)yY^v!$3FB! zdG6bhJpEXnbMi!<-j(M*oXXSB5+eWdLD8gMndO2q>svTAI9YAQF-pexIBGA{w%y6WAgMVdG5osJUt=LeMrjF zQ}W!0v^+f{&wa?s({u9NhrB#}Mm`e$eCmQceNO&3oS!9mdRdnwI z^h0@mo^vEmKbGe?p2*X?@;t{=dHR|BdDwq0PrsDs=Q&sM^lN#Z%Z)t!R-V`0ojm>Uq&dSs0@tfcGl(T>+AUi~VSh)S zejv}zLwWjDTh?ypgBh%CqxMo_;T%{PW?@ z!+Y}dk^Jx+^^Qq(V^l5o^CgkZ!d3L7c>1la( zX5{HvdH(#6`NG zY{}ENiHJUuVZ&KY@nL7ttn^7J|R4!>`nJiRK<&J}rj zO`e@~dHT9MI~(%!raZqt)rLHMOP-zE^7I{fcDCi|yYlSp$kX@a-~Ik7^7JElb{@;q zPvqIzm8YM|v-3=zelDL2{~h~TrXo^SNA z&^h1O8I`Ax%d>Mro*t8DXI!2>DSs07Ps!7h^6X5>)6??o%*fNT^6bpX)ARED_xUsO z^f`HUmgMPWd3G+!)2s6AT#={O+6`NGY{}ENi6I1 z2i`CL?qDQzZ205dBJ@P~j>Ygb{cP~g`d`|Q{~LMd&whUxj@6@M!_ZS_PvpzW>f1_n zt@wKB?fmLDk!P!w>in}WmfkKE=T}$fzljtVzADe>=f8Tpu=s;V`bj-9Sr4vOsn;UK zKB-lTOK)ra@1L(0zkKoHdF6xh^M%*tCmC0`|zPwR`+N6(YV zmmj5){kQt?)rFw9wuhjZCw=(}GJldt#tNjPx zufxY1dG>a>R(baQAO84n)@t($Uk=r${pa7mci*kmR+nD)@9Cfpeg6%g)bf0`^7Ya} zU#!l*sYLSmpL~`bKGM&Y%U@M1PoMJOBcoUy?kX<7UFiR4|K|^}VzFFX zULAg<@zL|u+v@Us`TzT#A3q=;-%)M3Sbn_L;XUf#pW&qszuCV5?|*Z6x8A>ppDwTd zHyC~peKx=H$=ikHa;30Tt&}5^<;b(O+G=h7b^q&c-aLN2x?G;G%@2Qw{O}&-`&X5H zjK9Whs(t;Y_{A4X#o}6}R;&zOi2aAO{HC}vc=Z)i{acUtUtib#L*a4UvxT+V^N-%0 zEMGnz8-r=@ziL8#cin40Tl)FWtCe@({(SiDwf;-$b@7A#)BI+ovhes0!aI6Bzg8)h z*Q)*RSbF=@ANP-+RI&DEzWBxR>YMr6yZ1A=G5tH;Pv863!Y9QS#n;RA{(16d@Q#Zg zyeRUL>EHTaoKWN#4yWg@|5f`pBa#09{lb^+;%_~+KO27a?)&?n9-e#|{I=JF_S;Wt zB$5f=o(`Xf`0glw6TTjNei{7AY$kl0uj2nc{Po(i{~VUVr;EX-7(VY!1o&B>4SxLZ zgm3fJO89a8>vF7 z`(JFo3~%rI$JhL68ifP?+N*Z+&RXMd-E>v#l3GG9V!kug$DzNevR{(WfPu zmLVrXWF2ELm@%`l&&C-t@nmQjGR{tB9L?rzVH-Je^s|`f4OYMGF{V{DjIv0#cB%5SY~hR-otM_ zEESY))>AWJtNVGj+xCCxg|^SmVC>M4ExYesyw2LP=l%<>=kWSh+@tA4MN8LfJAV-R z?KpPz!}WUTgDP9-nvEU$e>T>4q=Ov-?ctsbR`|VaPydG{?CoFa?9hzVo|-eZ$$dk% ziPYEOo^)3De|1m)Fxu&w?d>{@J#zAy!OC;{2e)5(W^l!+X9mBid#-}^S-cL-Iq~gq z-|=JN$DH2@AKQOCJTm`8xNrW+@V-^2!WZ|Q3Xkl&6n=XCd*O##&xMbjd@r0bb4h4q z`~JbFpLu35b^gPlIW6A_pPzRuy!X;i!%sWk3O66W817qf33P>_(18Pk+m7!a+&lkR zxboETaC6(KaNqv7!Xpn|44>bA5i|v%3g>~r^Yfn^+`9_>?E6l5PAkSg`Bu2%+{N&= zOElgeheGo)-l_xW=XiK->$k%rz*n9-8J=_E9C)XG{wOrE@4(>G^Pd?Eoje{sk5}`h zx59HzoCsGqFNM#~KL?tnp*gKj4)#G873Yqj@8jXn@sr_7=c#aL{>AXqtBB`ELZRdP z2j`x6axes*&(HsMc<=nTz!&rY+57NHp7VuJ1$24-A<#f?DT&_>hu(WDoU-Lac+R;C z;Ojzo+{{NqBZOZDULQIU-h1vG+WiUI?HjBBY;WR;@W_LJY1vnPlK9n{kqa*el6Pl+ zKRL0dCi%7OAlcDlV}%abNYi_2>fuw9G|$yDwn=@2+pM#~sQrNx_MZg*50hYtth`xRZKlz}lB=q6>V)&is>b0G(A2s@KJb(Parp;A8 zN&csNZvejcDtz;v@7>wH*K-5kJ2Hmv9kKWx`K0o{0gL}Bf4z|pUfcP@P^4t?+rtm& zhtB^~@m05-rCs<{@zs%+%ld!4hV`dnew>-bt`7fDzxoFJdMKM+JySC2&`^VR=*-{h z*M|48ews&28+dd%@R8?Of8?K8|HW7JtM8Vu{)SI;N0t0|x@!dx5P zlh8l>Ppp6V0qxL*|EXWS@Gd(v{5J`q;cu}+7aFjpPGJ2bKbj0!VrcjQcIaOp0PSBV zh0d(s{V2^f@O zsRczL=NF1YWE-*x*@^ml7kWJl{kaf71IeEL?dboTxUR;l68Am0?#J~wUjG~Sr*Qo@ z8Xxw0_ql|S4R%%jXB*G1z6<@kX6hjq>~jHrOLqD-Y(O?Y51XG^%=(95KeB`J>G#*L z{%{HFXZpaSWK(7T%in^n;LGp9k1xSy{|5WT{DB?1d=Pdu*s*E4&ZH(E+Vab!l=45uL}zb#w+gz4f?L*X~wP=hwQ9rnbf)ThP|t>08kD z_=3*u&23xSWA%nES$}*%^roS`)$i?O3p#zl&ILlWx`wXK7H>yKowucBtGBbs7pQCS zc68QBY^SfSqdnm73^XrZw6)dO>TPM>+Lr%tdw$)&F9rkD5?;b${z7AjDaYzI!1$-@cg||Tt?_D8Cqp!Ye zYu%QBx78=CV+D8@m>`&D4YV`_y#6LW>HLBA&i2l3KbjM+y{)6u+t!(P7r1YO*Wc<3 zG~5+thH1DgR$FqQTspY{jextrhDsc>&vNKpGli9Hxj@%5una23wOF<55P5{mL~eY! zmxkO+L+-S+FplfrOG7?q*}G}kBI{TIH=l;w`TH2PNUy;LYeD;#EginjJXTlN_IO&yU+$`$>_(Ak4v;qNmaSzehbLOW>`ys-{3!SaY}Rdj z$8JVF#o;(9w3L?s&v3>1>B$iU^(I zuUc18T2bUK!TulTD5B6I36VaG*A@c#JA?4WlCDOtn@$&+TfCjk?QQPP_V$*Jd2Y;X zpSQ#3&R@KsV4iy&T^1BBWyM|1EsgGu?v74htGl(Uqto5!+tS?TbFZzcTCt{XU0KbV zimFw0WgFI(ttu@mW%H}ti{10Rfrch`M=Q<)mU!J5ns}x26@30-;dMKE5^(!M2P03& zqx*Dmk6UKXh~3}JagMDimD#u0VHdPrVi$eiN24=%AWnuZr!%{(+sV?Gi=uTKjc%yx z%x$XES(On_eNx?qZf#Q|OkaUxraGHupOeaMN_M4UpN;Wh6jw5Am4Vwz@HQ=#`O&=# zj#!PC-31yL4FCDdK3#a3Eo&{ar#KjScaIJC4@>xV9BsAYwG{{y8i%eQ%j-3H{Rr1m zLZFaejGk?b_3qRn>{mdNfrippR=3a>pCma>c5Fvtgdf1AfTG^X7SJJ`Dxgmduv8k{ zUwzreev!?3_t_)tG62MMqm%94n`r0bZp=<)$K32k-9iTT6!7S;H;9QOK{S3@x6U>b zFM5wn@N*w3C^PXo>tu%lb~{?I$iRU#HnfoC25t8_Qm3(qejtgxagfalCfI&6<<0H9 zK9p@Q4C*$*Ge&sG7n~T$MlE47w2kp*1(WnB{fRV|9+;Rgk@_Dzn8eN!J>&FAL=Sv! z&_0pQIz|6P&!?H7NB_`=jl}U+f57?Ag?t8j@RKI(1%A*U(R2M}bZW#8>N?5iuMXRY z-=yFqSdrT|LE!~`m=WF;a84qAQ_n<_*kBqf{4?~AI+yoN-k&oKHGe(&L;p|hWf8K6 z-2rb*M?BJ2YI$Oqtm839{_ zJTkIa;LEx-c9|H^gnuj#=xJ1C+Q*z9S3vl>?bpx;`qu+C<|I5Rf$STc$%l^&fd8R1 zdn)-e7E1qS=m+?WVD3y>H{oBreQpru7_QYOlYE2Oto((;z*52Z=ZXHonO7naEk-}b zN&1UO`O3PnOwh!Z{%qn0{#6)w@Ih`bO^jCT@IP zqO8|n(q;zd&7tw;5r44H?rz6CKHu)2p345x$z}%U&LaF=j$g>0>}GRGwTbs*(HSlR zy&d#e+lc0aF9UuW@Oc71EZR>)UEse8d?xT7fuEhm`_DvO z;FGo8-7es5?0`Edy;1hHq&t{3wmi%v)cnx^s-!Jg>3*lc6K1lL$|0n+n+HBmH zl)nY%%pq}N_*t`}_1F1);r8J35A8``cJ$BrdB>)gXFbnoU5!NcaQkP%zR?2uB2_1Q z!#^;ebNPIw`TUDCR{Vk{$jN_!FMr`@vbm^_?$1mQf6XO(a(~5mc4mA^q4;l9l-jQU?YFyi^8de@Gq?QQU4R&@n ztuQbxA?mMK4`v3ZO(Ximn7>WSq&oMvJ!tPjd*yGdNWaruCVz7P?-uqvI-TPksEhXG zk9Odtzrnwu-P0n7-@VEc&MTZO}KLuNhq58B%}n|BR@<>$9<5xGeg1xnlaR zB7M1H^PA5Pw3qru{~1AO3j#z2dd2!n^Me~u>MP4EFZCB)4~B|ZdDd)`d}HaWcPH|J zGiXBVSI(b6ew)GHNPdUK-|Reo+DZPCw4T{T{y^(PJ*}6by^3!$gAPvbi20j?>x2C5 ztfr6VPk1X|k1_w2!+xTO`DcxXWZ%<7AlL80;P~-me?C6qMR{^Auc>&@yAw;lieG+` zPmjr$<3zj^pOxOn`AyRcDgO9vFd41iG=Fp+FLlD>%ME{~_|Qv>!F3(@_?Npw;{gkI@y=UcjQ0tXMA)_ z{@jy9>uY(k5V_`1<@mRQ{A~dk4#z8V~E46I%t5PxJLG*+$WZ`v>GDJYVlJ z{EXx+>;;|=Ap9qNmo1s+#}1w!+jE1Cn?IWk*~Z_1^L8! zk|O9ew1>JimCqz*1d~w*Lj&ZW)L#mQi{kAW!EvY?@icvn_yp9A_(Y^zMtlIeT{9`XoPAbYjFh9?rO7?~Qk?&`cJBoNn_MZ~7Km6Z3 zKNoU(ZcmyM&}U(Anoi3tBmJHJ3tGRXC9=yAp5F{jhkqq!M*VAOA@&0~nGhcZ^ux5@ zaB%-gj_yyga-#I-Xush|=Jq20;rkOer_UmK;B$k?j;KE&zfBLgv)t%CLiFTMyBvId z#CT~$4}Z!u`de-=%WZ65xXbOP`NrwFKiV9yj_^l}Pwk`r_+c7*Hq4PMawd{S{1x_s zUbIi5d%Bkmi$dK4ASZ`8)8Y z`2f7h-+?#G2k0j^X!d{j`*gA|&v(FYM$j-HATQyg{(n)Udd14mu=TQ~$ob;WD?T9g+ zCni$=DxN0<#cX0V7zO#m?rw$#D0Du8{nx>6EVZKOv_HrGSR7)~{_sui@1&iGzgVtk z$pw;J@RuIYQ|ZJc(nIz?s4a`;3j+VsAa@4F$A^FQf*ALSMI&&*Xio`HY`)uk_3F znW+E8@)>?UaW>6dYe+M%*O;$_ zkFM8a<|}-DTk;j!j8hT*$oCV<{;(&`A2TNd8*yR2pZXN@74nC;`O1tj@)h@t8|N!o zJYR9g&sQK%mYd>xG+zlC@{Q5?3gW3dE1ItiLw}z#ICJMk*I|Hk@e-fu6&=L=))w|8-I)cVBZGaaeX=S4XbPwwOMeey&|Ih5uy z#V4$vrv3F6icg7rf6n6>@}u-!cAh_BKaj@WK>oTbA(7&F(!?|erH;!*pxW7-0AKwr^rcCAgFSQ>Z9p6mnKe@rFQ!Mc!A>KJ@LlWD6JDp?fzj2^K z<#ys-)otv*)3I8DL9t&&dl%}_{V4La3C8m#;4_TpQRM%bly8dtC!Y_Q(fudif0I9< zMC0A$$;^#9<*VlwLZ2?(pikI8UAO4dv>zEefBgf__lpVU{1y3a`QBTLZ^(}f-w#M* zE9tO_7ehWvOM=`_=XbIQ#Y6sv^#`99^YdPDo?$o-KLVngI1fLH^Kel1B)*Shm8LYy z{yI&^exH2}1l*so|GXi8_&gl`AkNpb=<`$ld8AGsm?u&l{I0a@xA}f8DxZ^)eEfWo zgcR$mog9R>A5N$K>EjmJpB6j+&$~H!&F2Sc?2F(>>}Syd&VS0gKZ@li`24_>pL_;& zNOiB=)4)9q+|$544cybfr>+4yJV&SP!FNK}{@vFc8 zUF01*E2~{STzzFq4 zFVT+kHqOc$nz0yuPl^>!>5l%0bI}#H8J{QNd#Wi&5PP*mupBM%@KMFO;p~ zJd%G=z(BnId9F_sp7?l?YXJ|%#A_ere{)Rt$_DZ1qw@F}TdiryIy4=HaR}Lyis;#( z(HlruP=g*v;fWj4!D?M!`5@+=n|n|%ZO(c%j}2H=jQj{#QN5m`=Qt97$5#q% zly}kCYDKZG7%`r%r$N^y>8T*9w!D_`Hx4h@&Ku0Khje3+GVak1-vqgSi5YNwTpjS z*syk)>EE%me*{^xu|JQcUTvu(TfP3;WnArr?~N7TZWIm3aio`gpV{^4sPcdct)50b z+fXMw)wwdYajD4gXa``^W`K>=nx5lAZ1(IX9cu{8Sn2T=+Z9j_Cwq0PUQnlAXP{19 zk*;af9956)Gi*AIQk1&ZUgUZ_<>p4CTx2ZW#LHEP(_6^}++f$BV;!LkkXRv&kd8-X zv2k)ellNNkJnF}2eKCSzWM&=N4H5(XUaW^c;A5sW`mEUEZ13}~U(FVK7v<+L3KHgTR|q}}itMg^+5<=V4c0Pv9w1UiEg5BVUY>e49X z%{>1n?z4S1x!Kb%=DhUX7bmu|@6G+qZ#UTfYSz3Nc|V!{kG-M)L%HgAX5YOr2Zf&9$&*{oA!Km^Srj(6|C%qVv;yGQhew4qc z=X5DrH0MP;rz=x|qu1zjx+Fx=M+t88hGsfHrn9v7$a+Kc17!h-tqh`0n}B%je=OU5 z$)Z|od;Bq^<%~=8yPW<=6U8AL(0WH>DRz{G$R5@{sa>_4@J&=P6e}(Hh5SrhEUA+A z`?CEpLtDDF>Ye>$I zAGNesJzLw)5DM{o6oI^+l-sm z${EWPQ{$_4%Knd|@r~LmI%-dJ*7h{dNd*=+WsDpXr6)&RsXfv3h-&m5e=@-PEQvzo z%_H(kgUZ7H(A7{U@OR;L#1r*zDzd%D1hhCAaJ0S<|HT$MJi?!spBXxiXFoq26MtVv z_LCkj+MW4haV@xCT(fW?`AtNq|0e)xd=wLE;zj|aj*UgqMnP-Qeovh7zarb`N=h1^ zibZ|QC);~0;~$mnmH$$Ea%pS(0!-egr&P^L?S{nRo*0xmaSk|zp39vP_gfDGbx_aY=%BK4D>+&{js9fP) zU%0)cvZ{8?j`C$K%O0z3^4F|sS-z-{2 z8Wt_>rtuorSGE^d>?o~Yye_Y$d~0QN-14ofyhTgf%gVQ|l{DoW+t#&iSiElAP0=mN zAB%3;rq*E7ro~mzM{7CByk!&VF2B8W(bm=8;#K(@Tep^NE2`<L*5qX>WTW2!aU!GVt;+B5B6DCS>Lw$iIrr7xHx}FK|%c%?~=lT#Y?u-7v?W6Y~0e=`0%0vRO;&lT|@I0Z-ei8Ke4iw z|F#PKstSJX*)#ShB(ysde`JjQnB+|iFu z&iNzv&M$ol#>roh&rPHjW9^9=rtO-pO-s$jgIw_Yk;;I$<{Iwj;oi8%e2NxaDr-~k z_dKT`w*PtJe{*~-^&8F?MbAXv$O**Tt}*CotiuNS*VX-U=aF&W&|kF6I???=;cz{v z?k#lF06(j6xR$E>Qm5-V{Tp_|^FRERaO#&A13uRfjAmQ#L|9YYZdn)vF005*zCt)1XJ@^FBP4^)Zw zyyvyNYcz^e;yq_t(DR5ms)fxku0jg>7(ZsX+zBZlu7cjU^A^Z19tl)_ABIBX(XYG?{P$=dkDm57@!(6Y1OEZw#LsDN2gcOA z(yqh*3rM_(zFL~#btwmp8E@P-rMzxw0yV#yK~Mc$mT=W}k%X&FYN_l$NA^=KO{Vnx z_mY1+U5Xbqubz{9z9QjjKD{pc8IT54@b!SFt3lDzYSgLVDZU)OS=RqifZ4ZjCHg#! zL*<0Hu(xp~_!SAaOSqD60{T!%A>r13Hc0q%OFv3q)Q|C$sK?MhJ?BJ~=M8Wp{jux7 zuNdG)d2^-St$fawaI3rn(m+;u-;i)?KR=UjtGvH7^kd}zBLmzhZ>F3-G_Em5=3NJV zSi*@P<7)Q2gg<0~4;bi;_I+K#t^Ctdbf^$NQ^YNQFOYCGuJXku30LDP_+bORaop!6 z+&bjp8}Gc! z)KOh?TQh#iU!&ny`5gRfWHfe6joXRNlH0N zkSu8E=*)lkrnPa0?1gXMxN(0l^>#M!rZ%5^H!tR6hx+Z-JM@=b#RB^Y-}fMh$W+VJ z(-oPZPL=tGLds=YRT=H`2`1BmDBL6OPRlr^5Gr1&`YhZ~%H>f2dG+rQH_yv_j6PeV zcqhYgGza^80BL+7aM~@VJzp*q|qQS=;}AK7$#z literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k257864cebcfb331fdedf891003cb1c7a.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k257864cebcfb331fdedf891003cb1c7a.o new file mode 100644 index 0000000000000000000000000000000000000000..ca3862b7861edb8171ebf87dbca88a1619610580 GIT binary patch literal 28192 zcmeHv3vg8DmFC~KTcY(q5*QS{kXl060!yOx1lbt3-k>Lz5CUyxo0ilz?Bt<=mgmBeJ@b!>-ZW2&<2+1lD+u!q6;g?l_1oEZ0h z=YQOL`yw8m*_w)3{?hGx&-upXn+%*jJ%|5)>U3fUvL7G9rxF?#I8 z$!N;q7o&$hI39hs^B2)e&%7FKFaLS;(2HlHNryj-uDkGFbiCz4^vvXk(d0E7B5!=~ z%-Phk+Q|5Yr_Z*%crtn%`j^Ik9&JByCVH&n!|2oRzZXqeTN62U`0&|r&T zPDUS|`bG5kp`S-D9seNu-uU~FsgAsH;hD3?QMNbk{%-W%Lnot8J@IPv-Ve_3JbzYp z0W$YTp6WV$_PwVMpKUvHB6|GH$>_RQUyY7`2sz;T5^?Ty%Q?GtE znv(e4=sMU*f!tGfUx+^aD%q)vw09jkd#vox*`JkuH~LiB$>`56oQYmK^YiHOlkdak zg=kXQrpUX`JaINRN0ZT32hlcdU1y@NqfOo@ zh3?wTk=HLAIvY88A{r?>6Fpe=K{Ux$aOK@I&-6eiko0)LpC=`Tdy;-!5Tbta*jbGe zePGkVJ%jL>l{U|(=jfB!pd;e!Kg0e-xYxJHX4As#LrV5ctqge|*@RQ?{e+#0{fwP@e?vm#Z|<^1&ZV;-zPpc|diQO%@b0~i$h(bf0WtEx#UZwE z@t(v8*|=DRF%z+N@qR{Q;BzS3!qt60)??z!R$eFbe7Y9xkz`}1D6T*6emO$Czr&ny z)42!jKy5IZh&%K@?>5>&%}Hi^>{jKAJrT;EF8(cGu!MZS(x_L)`y6>feU@IIcsuPU5m9v#aDA zAHg5Gfqw*l`3U}!4j=ey_)t3h?A_J&$i?r#R}Qd+v=%K=0lD-v_UVUjvjxa5ymzk_ z`Jl=cxp+@PgnWqn>0|ijhYvcyON@{&k<7IwKtPATEps^qEIw`Sp zGy@~;ZWecKoF#hLMwWmQ<&b3g?C5MT5cCb=N!{ls16AX-1RqmPePfk}?cy-?3NJT9%eG_AQrYdTuE8T8)ZFOx$HNS2kNFq0YZ*Ict zMm%Lz232@AeQpJ39s7=YH07U&pSI=74i;`&3OY;SoDqrAN7t@ReEP9rXRr~ z8I#?ewaf3=l0R;&qu#MHZhh_Y<=_0Mk-Zl?u=44Vky(Gx&Ai?zxU*b;_jWdSdIbWn z$1cq`%cp!DCrvDRZmWgdP3E(;!0p&&TT>n9CtBI;_c(v)HT<7ybhfaRoy!)Giw+Tx zp22~Iou2>*T7CYI^OF^+-~ST)<79?ChwGQ?2cPfyI2x-iK2eY1X7MOArq zscQ$_%4_Oa!``upVb|>ZY|uaHn%p}ZbPfAQ#-{wP?cLoSz1|(oJ-u7IyS&Xiw>Nh+ zH8-)cZdaA7%oi9MbDf5cMOarOlGj`dDn)_+OQ%bcof`LlF4 z=2$LdAaYAbQ;rb_`~mhhI7A*_p;9oG$lQ!wUd9}fPGBXV3w7GTTyeCWWyR4N`It%G zMm00hJQL7%9HiNEV@&I?F}v4pbF3}leM^3Fv*ai#VMkIJdkNP$TrL!S2G`Lv#!llp zhl}CB)`d%JLml%R?3aj7M@A|O1ERjC%aJdl!=7Eu9LY`wQ!H%9bFJi$982>zy>V5O5@r zcwYa zh8_o<87k#?De*X-Lr*5$#c>z$-XZxyPE+#;dKag65&a6`U7YUH@ztDOtF{6JujMq2OTzvJPH)ihZccaWbOWawbh?t$l{#I;=_;LW=5#aBy!>`f z^MUrAZq$cynqX0$&79t>(+_g`L7ndBbiYn}IPD>t+i&4?i;lN)x>cu}INhYvjht@O z>H9c+pH8=N8euK%w{yB($2&OPq0^n5=KluZ?Y)K5TXcLYr?=|#Mow=enwP(h)3h!S z?Xiv17``O^5T_r~=^jq^=yWfqdv$s{r?(T$?e}rIPsi&yU9Z#kbNYTsXNMAE1$cI> zfIlN1QUlCE@h~&^u^r{JtqCPMXh@D}h=O4{_j2<`NNb zGQE+`mIo3NSAmRu27TUMiCmt?9Oz|V%Shny=G>poUMm2-ECK%?WlK_@bwWNPGqi-u z(-4@jj3~qZB;`w`Npgo|G1JM_Wc7Vp5LUakqr-DWW zi5?}|3EE|%vp|;!dXVDNRYv0hB+%aE--TtBpbLrS_1okjT*TwEj?B;l9>R@yEGyN= z6E0ubNVJSE$X7LzJddweD8KFo!j1T>ttYxx>;W&9*$hCE9M_vq!@ zoJcs2KW_iQM8b{uvnLQP;x`@kSpwla@$xo12DB9rmg5uTi`J7oAD_7WUFC%H z@$DV5*HuoqF}{_S63xd?Zm*$%Xk+|wIEd!{#qs;n$sUj2SJJi2P(nK4+#fjJm`1pa zcjT9tM!52~%uv%(!sYk?e92P64gYFMCA@_y%*)@JO1Lq;Y)m1X$1|@_T?*kwJa0)R zT#g5@zcra~V?5|cBAkyW+g;l_B<<|LetN8En9lW=1^s?R07p32AZ`*R6b@thgj zo7Hw;?qBHu9dlU9>&aUA&-hXw|C zxYBq8Iy+PoqrT6}wDIz+36-zXK$p|}jqwEKDI#Aq>}i`QHaQJ_#DBRyK8*1CJwUOl z@_T$f1GpS7IIh*!#_6vBF2@gnZ=hH-^oxPJ#P}iXcT+4IcnNShz6iX5eBV^wN*WtX zbQO&aT%WgZQZvP(L2C|*6+=GBPO)s@G`gB`Z8MDphP?J5jRhw8ei}_ff1H`bi1Y--f)_MzLbT+bLE|cn8Ir z3Gbv>GvQk(R!#U;id7T7kz(D1*HNsS@NE<;Cj23a6%*b=v1Y=1Db`H*c8XO4*ZL?{ zO?Z8MJbgdKI>#};9(*iGV+FKh!6v0RjTHv+HPOQmdZU2wd{*Ew;RX3Re!c+fZ;BR~ zO41#`omujK2`vK zm+>%Mj`lAAL#8*@122bv%Xk>>0$yHF9*3_l*ZqH(;%R;P`Z(OZUiWW~yVvXS!0qRf z|Lgv$g;ufS$JW)+`2p3hBAA2vo^Iy#MSH9c=3sdoelSP&UtvIV=c@c(ThFw4ggWFo z3;w@4n43fPpd{zl9F638Bf@_3yp4aMnrT=MUx58k9yX+>{~K>4;A=2{(sPzRKj!4p zjvYe0e60r&`aIEk8u??tn}O+%mml+g(|nM-8f@CxrF2aT1JZqjSix1>ikMi(D z<+()re@BdAJpm(5IxlFS%Sz^DXD%E3MJo;^V|if{`)~MX`&=G2{FFG3`QPP=$Mcio z@xnqKe}(j&>G3#Cr)K-*Y4JEsuV&nx8jsWRLE_wg_0o8pmJf!0afp@+23`0KEr-WG zwE)J$_PLytc>NR%ZH9dNTY(#wm`B*p~^=*jMv2>nocDS`)z?V&86d1 zKbnr?^z(#1gx(=MZ*@GLnvY-;iQ}t^bR72E=Zco=?Z@#nEH@4N?Q^bW@pw9>4g+rs zEY~PLMENOyT22_{Z=cK0h}Wm(fLT8yXSpbyEdO#W|IB!rCdv=5;N`okLB~=4wm_vu z>pvs^1}vB2^t0mY--w~zj4#(j{>nZrPbALUC+{oq<)`I}p%4GejK^vDV&H9oO&YDI zjq=g*%4|PtWxPHuuMGY6xy%*uI4!RX9PM8o-+r{bH0-wpYBgGqtNhynPN!%uu+X0E zb1UvK+i#yMzdN43drdrDS*+8%{fphAe!^Zc_VarFu-_JNSBv}=?!j1Xl)rthcCBs? z^2MRGSiTwMZ=dTbiN{Oob)1*K`rdfF8B@7ozb(+FA>ffz|K5+O+-!f{x_JG12%7b` zYZwrezN0v_3Cn#mUZRy4ILhz9a=@_P7HHC%;&56?nDrYpI5ZNnJ)iE_9H_tok=94} z!1&b`*rp+1kQDA-Qw++$do(ztz_H(L3v_4*Ktc2V*`i^<5H!a(YJ6`l+5_?}8U`d2 zexFAB3E5u2J(>rUL2uUh*$C-#`*Hpac?VygNO_9SO40r>f&OX>lvIoMQuMlYqJ0&e zlOx&#H0r-Oumu09FxNLLs|hqu98bJ=9SEa6N=tQrF`%08ygboKJLRTE#k}29-;aPnBWt4B@3ei6b zUr|NzYT)JNqCXX0c^AdMf!}?%=zoRZwTxWJz-w!TzbHHt3pz92)g}B*;VbXC+k|6( zFn|@0T%S_^By6Dg=Jw$an*)w+Ob~)be{T+yv{HRJ>d&>b{X5*>kA<3~#SxOH&oj z`Q5uJPtnjPdsJV(f8qANQb+B{^wwT*)+c3|26zPMhj#c z>p`4P)P(YK>AaZHdK2qi>?iWBLVl*{@)GvH(wVkDN331t`ndI=>Rs5!d2R~!AB=?` zWB(QB(U5Pzc9$p0AM451q%9PwNP@Pmua}2nImrI8?AO9PzlMVr{e$i}eaL5%eC+h~ z_41v?FGF~Hi2S%d>Yo>i^3pw$aG5P${?9nB%8&hW9<}GpAog3-K3aVHtbQ_^ z+QZAobmXT!nZr&~`%?R1eSYowJHEdB|8Z(R0sWd#Ztml%e?U|JZo>%A`y2jwB*c%C zeiJ(vOBd@C+HQnn+4B4?Bb1x-3G{Q%#g;JmU*R`a#n2yN{5--*wDaq-y)xjGIGCLZ}) zb|}yAXXq~v?6~Tf#qULOBJf4zkz%HRp4fn%OP-qxE2zEF%ejgFhE= zyg}kg69P8j6fUYiTa$v<;1`h}$GQJ#RFyC<-&^$mjnO#y)qxyQztdpz{>v+HaS0xO z4){+2wcSill>E^|e?tGyLH%b4 zxjdjz0O|E@ou|>1k{EB6hfM9y+lT1zWBmVDD1Ym`7;Th%afrtQ;uQJaDE{RBw{_md zbL0A#sQd_U5ZPN~`FZ~4{8kw97!Tv*d3k6yp(M)hMEqiJ9%H12{a{>qkzZku>O_5{^pZ%X7esuLm)A&;0 zcaFUnJAMo)eC*gdwym%4SNlHxZS1#yAN$+KUjFFn=ef2NNAeX$>ydhX9AQgIO#Y3a zZKasy4cxZG#_2TRw3$0#tAm<>Kc49Tt`(&vUg6&iIKVt&MNQ?=lUC$R{0+zLniZ!# zYyF^>Y#U@KlaWj|ZC0HUh}lw7Ax~wpCt`>zakSY8%rkudvHNs+LA&$nr zA&vsx5O<-UTL&_Q@8X%tqx+sc(PQyFh_i=mwj>Rn!ev2|26Gc_2x_I5RabCn0mI#SmhM1?p8e|ukH%}XTF}xZm^(+9F88vbq5~`*9t1MzHby6(O&B zwaLPlB>UcV_zJh)*MjlIf7isPr8raKcTLNfDxa!^_+1m7@W&_dyQY=#Oi+J`;9QnP2b@k#$(lVmITc&RT=(TIZK=$hn_oFR--=A@6Pl9_`0 zV){RodjEya*-Xi*e-|(;AFfGZIv+-INd=Oj(O4Bnwa`v9gw-bL9i>qU=`C^7KdrLl zFXXpKam5J9E;}JG{nOg+bwIe^TZ&OEUBo3dX(phuTlMkJWlCz06BIt1q`q5nR2y6M zdw@T&h<;G&7wZ&FFT=#FGlOo9YyX{6U#jajYy0;?|KBW9 zf2Hq~3K}d>|6B8?xS=*x@A1N8T*{LA>aO&O{g2fnm(=4kE;kT`k|FQr{HUxdE~L~o zHxCsG$*jj&DxL+bf2#OWccr(dzIR8}L(TOfHFh>sPV613YM9tn)!X-Q{ls=(|M1>j zJ9{dICj0xwYaW{R)i-uF?r0btnrsOU?ChBsn(5s>vGtKAUw_Y|6RmsdAFi(%>1%D7 z-_>6+G2FUiz8Cfe+91t-%*Vi}GQ#a8#*4fxnH$KomFgn~Hatqyok%9h- zM~1gdBCp=g#)`Vp!O5Dr;no(nufJ|D^4Y$tYKMEUwdK*F`I?7^k$=ZjPt8y(@_>B9 zM9;+FRM#9Y)0Q1$zIyO>_KfbD3{Aj(+vp7P-%{Dv-nysDS6?^X+`4D`aDUr$!`4Sy zhbwo?3{;L%Ic_A^gnC!qOlA;eyopT9@MPV?yZYPveD#y96D{q1BLmd771K?XdmehY zVVLSRyK86HqaBlNqeGQDMh3QY&BoQWZ({Fo(=PW!TX&zVL({;{2I_;w%0K;Z{c!7G zYu)H@Q|G)%r*){RXMUi+CAg^G)<^o#-s8J^YsP&2HRHIZ(C*_ylM|H#JKNzOqfI+@ zcDaWpcZ?1WppVh!!)?7SqCR+=!9?9Wd<4EzGqS6{X76BC&**UL#N6Oa&vg6NM_PLO zYuufWwxh3m9^Exo{QxIse!&0J6`a@dAH??o`S135-+(8i8<*CWQi%ocm z1t<9?6Mk9Y9cf2XD1+z4NKJARw;-p~t@jt8>tG+tuMEVa_V_IY2Sq(pV;k~R1RFnB z7QAq$xceN$=Q9^)sQHD6#on*6otuWzB5ARGv5=RGJd2Lql&^w>yiuNbKCKYqlaNn} zXM&o>wcOM`s$MkB5?}ehXfMY881+&-v#}EJN!}xw%I~9S(8c6CufzW>^cRz-{n29l z4cFoS0DLOXOS~T#Q*lNA&sfaPA7SCf>npoJ#TCgeCjW-4?`ezt805*$jO-evTPgWr z$!`KpsYu%SrR1x2{!>|gzSx7DimU%1%juR~e3@IpS5BoJm+U~r?+1UGG6LCWKvN3B zKJkx8`d5O@zJojQX(3DLRdHi);!gaH(xv51Z=1VdPx%h1sBPOy3*Rm6<$$l+hFD*f z{9N$mmX~wBX5yQ-qTiEz%2z-A1yA)Gp~|L>w?wX;$3^DTBfCOhU+ zrrjn!_HlX&nfPY;L)YQIW#XIbOW&TRWG$yt@~!m^%MP;Em(B?mv-6zfTkHFWCOhWx z|G>mI*Y_x%L zd^O&y{kzInjrS!eA0@@d5X;jR3tx@*GZwxY@0nP3&w2;PrhI{U#@{H|;oCbkG}_Kz z8QvViZ!q{f1jT!UZ+R!eUrjhRHHO!856;ehF}10^lcGRl{#kbOO^@6<@YjRiTzbwQ zn8mBHWL3uo1E_baKPak;cgWFu$XmPm%J4qmp>ZMAg7)b32kV1)f%xFwpr5!x9*S|? zVPBJfZfwXO$EP@ov z)_oAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi7z(=Yt^fb4>VH-Je^s|`f4OYsa$VDyDjIu*#cB%5SY~hR-otM_ zEESY))>AWJtNVGj+xCCxg|^SkVC?XaExYd>yw2IO=l%<>=kfYi+@tA4MN8LfyM7S* z?KpPz!wq`qgDP9-nvEU)e>T>4w1XW6?U9}gR`|VaPydG{?5$tv?C^}#o|?0^$$dk% ziPYDTo^)3De|1m)Fxu&w?X5bDJ#y;l!OHUo26w#o^x(?VPY-@m_gn?-Gk6`IbMo8a zz7xm8k2=2-K7Qatcx3*`aNqn>;r**mhcE3v9Uj^LUihi`?}i_0Js&=P>fLb4%!flG zI}Qv!_4Lz&sq>eF=Cph#d|}@4@V@tc8h*<8X1MvprEuTM_dr(|3LQK+xc$U|!F}_O zhbvE?2sgK#4)+~+Gd%LprSOFvmq4>PRN*{0cwzn%gZoyapZ(tn&uPW@r``-#oWB&_ z{vM6@$Dz=CjJNtA`Z*Dv+xqSB2=JBXPle~4JP+QfpFawX>_0g8)cmIhL#IxJFW}Yu z-kagMCr^eeobQD%%s&sBWuZB(PYm`!7ZvA^qwf>p(1}ywO6TctX#S<}Q>%&RrJ>M? z1A}uq&ll!@JG^iHo8Sw2fb4yECC~Xnr~?=P>{A$g}#g_ufd$PZu zoY+&7{AzZP?C7zvLI-T5={+^|@Tp0f=jvJ8q&~uJ(OF^C{=f~g z|5Ls<0N;BBzIo60?rh)dxqc~rF{l8wz`cpAK&dy?2hkvMFeI0&1l+CW5Etzz9s6jh?_HXrT z!~0o3%_F7_JhB4#$g`|J^3SaQ(kuGacS>0Qx%=4Fe<@)7=hN8L3rXy13TTTk*M|2d z^bh|N>mPnVJACnf>Q^tm!wwJsO+sk+TkP<~2CS(QSpUe6CIglj8h(Hs{?`XU``1aK zvm5q2LURo~AHKHhRnuG&>${NWBKRj+hQTY1`7X%>881ObCuAbok|(jNKcTq_`HXV0 zWOmI1c`p2IS^o&+tB`VyfaekLJOZ9cMw0I$_#B7+y^zWIcUtIO$T9LtLTChXybC#u z@{AP;oWe_;YUdcQzEWt|8do5eHr^>H%p)E3DNjOKST5M^D$E5k8v$+`jwBXH11=rOwZMnd7e<} z;-ZlA3&kO_4cUb3ME$)3y`F>qT!^26WKaJN^#4s<*Wgu&`yN~m;CceD|Bd_8xc(cB z4|~1yY(mHeyDI;)jb~Tif&N`H^^gnpxd^`{JN+6qAe&!+&CeFF{vp_p?4W%5{k5z= zT*CU9KJW*F~%5B!+F=k98Dq01szQtoqKeQq zoplo1>1*p~5BNI+%>|3Lwfb7UEzR57@|U#d*ZuoqFhE^?;i9_5`2}_Pd3kvSi|X)i z(c-*^^Or2GtILC&Law_&jE*L6z}I;92omsh_yUhXB3LQlYq=}D4RUzz3PBou^HoN(=J9i85`&b+(8eH*;~R$rju zt}ru9!)3ABk^|+^$qi@(+yypN;+TDwL+_d?tZd5#x}Je$P%*B>s$GZ3BV;CWoAg@;S@iP0JQp#|pUlG~~|T$EZbmEjCyS+P7})@O9>~y1KT<>Y|uc zWvl80rY;X#wPtSzkKR!{wM71MSLI|kicE8Wv|+bwEz3AO(F$gN%HiWj!8c;FZsR+4 zGwMkW$4Q~3ya;%PE7ng>jv%N9D>HEGk@O(rWf$P?qiIpN1hcP8dNs77|1RJ*oL^W= z1SMAj>7EMs-c8rE)IB{tvza5Zr)Rc<-!ewbDPh-uBvL~+Pd{+HES!X zR@aqnTvxWbw5*iPuW}c-=X(PUP413XoCiGYbz^AamCjf2`G2B9wx(2O-)e_l(Dq@w==(kzoxuZfGITke*=5~MmcCpRt=njHLtSTX zQ=QJLjCksk>Na$1n;K#IN*puQ*);o{RBltUD;4`}j1Qx@l3}Y1+*X3OX{pSQ?p<)i zYP{?&(7<5$&tLZG!pm$~>zF;o!N|LNY`9+{;X81&)r!|PAW&!=x_&IL*W~phTuTXo zLVhuNwlUVbON+2y0Z9fLN@H2wLSKB6fn_ zWgGiNHtXGQkFd)C5YvrLwr5|WosYXIJCz-GvmbQ}8Q7D+qrcuDCXxiv_+{NX+d{nP zJvPD5eW;+!#Os`s9S+#-Xu%=_2h-TlLY5n}-4{Hynnm=3N$mAQY*sMA_M0hh?BMmG zY+IAhdyj1j=%Z?&VMfCGth&dG-)sJgZ_w~>o22IBYsfVNj`sd*hc&& z1t-CZ+`b74FX+RJ@UDP!67idQHj=~!(^%o3p?}o5yl?RSoN1`}>(L+je{vs-kUi`U z*mJ0g-M@=ysX)FB(0^`l@?`S2 z@m&9+&d$7~EB%ybeO;Ot_C)`%hm$)w*#rC(2H3<&e0)*w-IWkw9{@l;0RPJf*dpYS zkph7)>(R+~)n4Q8|Q=Z^qO1>;{J`UhuTiA1y*{TwIh zFCyhD>&7xc6I=STi68h^Vc@|BxxF+oUSVL?94-&FKb?m8IWNM_qCL&0yy>XVjj&6y zUjLppGdOP!jW>_@gMIdNJLd8EcK`HL_LojJGdOn^;pcMvLiR*An@g%qydR6sa1rS3 zpvT%q^v9eWZ%19=OMrI+?>6Dxs0(}<@Y8_L6Zm1#ej4fm|5e~Kf%gdf>@?ngCh7v8 ztnKM`0gpwL#%~gM7wQ6^pbL4nNPM>Be~ZLxz!U#|fv;Z(|9bF2l8^g8`B%_pf6mX_HoZLSIY#ShB(j&=KNI$i7SI=|I@ufk zf%%-v=PS+UU!<|(=QTl2{tJBh^FNc#MSXOCW_tK*F4>d&E5;-L+Cn69K4b)E&ZPLq z{qg72{|Zz8y}PEz{AuVQ*0%&3`5RvpPA`OiVZD#~7x~Zh1R8HT_n$*#&*@SB8ggod zfoTa*f5mz*GdOJ;(I>|IZCWPPxxej2dl%X(e_KuZo#rz6n*(^au;;Pq9PdC~v?qVG z126pz{tf-kjOtg}n;0N_zfJn4dlac}8qco$bEpLR=@fW5pGWIoqGfyNeu5YvGo94Gc-+aDiaD8V;{lWh;qWZ4S#(Lqh=-cIr>AQ;b z<%-R3K0nZ2>KpxM1feYm5Eo3g@Za}H8EVI1SUvxbfDq`hXvrY1irLW#y$Oq1% z39Vl_e**b!27e>@9TtDH^Z02e`A^Y$W*7Mbtq=9IUW)cAzRe6eIK3n0Zw{^x^0#xE zKAJz_t$aPk{96J0i6Z8oH6D_E&k%uJzYBxo$CLf}_=p$f$+^6y;z92&Ed45e`AI%K zCSQ&d@lt$NdLQREO)sSQHz9pbR!G! z2fLY0yPm|N3F9F@8!f-2jjc~IKS2J5`7@vRQ3dtmJ7K@a0ib*U{zLKiWpYSr{{hK= zG#1~XPsn45@46wrXUB=>$brcuQonj2GZwFh7HTU3c8c7F_^#_@Umo9)|G=N|(J}dR zPZF)K<;h9_VNbLFk^XJQ_-sGE6ZwXcub%Q9hx1q0KAb4>9kO3agq_2H+<(bFH}+5X zH_sOxGQT8wcv1OBz~8cz$fp4LXu%|6?TEN&bZPke`po=L@g9$oWLR@_H5I6YEKe zpx4kI>ef^~lb8`qMjZ?dkbhEtDHtw_w`T;$p>D*}^flrWP&eWek!~6BNvOy0mi3IU zZ+4fACu90|6~lVX{lmY7_9yU{hF4@E=PENGc+Ckm7E#%uc3w559DM*d=$_R(|*Ij z{UbrAqxT5WlRxct@bwYn zr4c>+Dbwh0xxp;Av3=ogx0~i0r|14?bHF;nA2B|)kNV?>Y3!LWN3zJ-NE-21*b92m zK8f-jL;Uw*yzBEr>lXq#5eWY_`}-l9PsaIx^~dDzz?MC4!0jXLS~ z+(OimE>MRg&js4QITB*}CI3iHu;@3FmMVX}hzH;o+EDYe3iCSwbtpVK{}a<=_%z^? zP&eXTIQ3BN37?L-5uY)^jGusd3=es^zjA#*UaGsmfzXc)^Er2ne4I}%5nnD;nd8}u z$RBsJRPt~5(>V4<75P&tweXkv?INCh5j84?^QcKW55zD4+3@z?6Sb{+|2{6BZKsx}AI`?^DfZ{G5BGU!KoI z{V$f!@bigtY4#iDFNkl~&0qc${)hPXS?4dK^A$((%U-0PA|Jtgb2-Q#`S}X`)10p? z1m3tl(|TpiR}AYf;zbtLYb+-s-a~(;^&IQ7Y5hh$3+r{19__In#r9i6nt8p(d?kEz zy&f}P;q%*)uh3?kittCipHTLPJ#qe+IT_f93-kT-rSJC}9);IHhdm%nw7;C@1n~S5?Cmx^aNR2)(%At62AD{1&Cql}hG?ytp zVf{4iueVZsO62=<9?y^;rSG=${0aMkH1<03*WC$;6wi|;ra34dDh%2wTqp2+7wexm zFFLw`#fxW%7wP}b`)^bHLO!Jj_@@ui&mRaMjc4FL9r29zLy=68-`#pVGwnCb@eAXd z_8ZW*DV}k>dH-SBZvbzKZ^JzQ{)F-DvdLep@eH3Q#q1A$({Y|Oa8T+8_QQT=GJPJy z*XMKiym8mW*#7)MzCY*Zr##<&V=3Mk*`fKo;LxXVLXo_|Ieg+Q|v$ae8`OMKl%Qf{0Su* z?yCEV@;9tM_`H~(_lolj!+H2o5Z%Oi_%WP^gR&>_eH^ParCIjZ zX*%}%>}w$4{*3+S4f(_8;qV7>zMe&&pYqQmb^5?Ok?P=gm1V!p_iIu4oQ&k-=Zhqy zSYPerAiVu>I`vNMmo^uK0Ye9TgV-+kEmQ=gk)8cVMr8% zWcE)*GW%$cB1@P!lQdDzAl;2P8$AcjoZECO=W%te3(lgWq7$R-g|3Ur+fw4qSfQt4 zw9%M#v}!;NdK`r(Zb%1fbbZx>n0s#SLA|s&>(x9qU{x{lBVa}KdWxRoNcaVEVcdAOjd~{N;^CDQsGreN8%7?bO*AlSJ24(- zdTNdH3g7(NlC&$mjNxG?ywGqch!^;y4h;twUsiQ|qs3Hz5_QES(PL1Z>M)v!Kd93! zU;2Ot8z%a5nJmOZ{d{X+N)KdGc_7Shm z&G3GPuc%{XjWx}n<&u{2g@!G3NLQ`wZ_NIBM zCq?I_TQFd4A5Ys>n$LUPs53oI46^bck6z^gQpbLzrRKHETz?XHWfrk^*(_@p|F*DU z?K0E9ZE3$0S+lV}kELF1sUusx{@P_+?S=1+72j?Y4ajk%mwcbu_35bcfC{aiK|R}0 zCp^`;GPQB3$na>(X%2+UaN@fO<^P!1=1b*o-br(S2EPF<0% zY1AB5kL@#TI*n44y3StYdMxGUMx$J0EZxM*RfyAD$pzeC*Pvq^p$w2%A&roZM`f{b zay^sxTJk*V$7p>qf?;H49oY>M1O8sDhd$tArZxJrSR+4;F+aPg8ISwVjvK2z$Eq7= z*u5Ao){wDu_DS(Zo!^V$=2cw83Kbtl#~s>Z#y?53tnRvX^{04Y*vV!z8S zIQq2E%1=Cc%YMj|4~=Rc-E;Cem=Ik-X!$|0(%)%ylYZi?d{%kl(W}iJt+=4SqxYAb zfAd-1=k09o^R8dhd@}CXM68cRy}hG$Y#w+}EN1Uz#;B)e?OVFOx)ghD#?QXBEAT!# z499L;l`%))pOtvTXIOM3YdJ?3K5oJfi0{XV?SHKQ(I?|v0z?10#{2J4ywUg0jQvZz zLi0a!Tcc|gE}SpWd=ll19S1gXo?)cj@EJynRddU=XN3UZBO3^G7AGF^K}OZ3QOcWn z{!iRz`)qczr(Vc;@w+cfY-Qh@`<>icRDaYTI<8{5~|B!;n7IcmGY#)OElHO2kI0=e=wGMTq-A4fD z2lJfj`fqc1AN**@M6`^%0uWmnM4dJP@!J1bw)>Jr zwbu6dV@S&xm*{r|{gEb$LpGrGj>b~#C=HQ4tbJ0uYB}MXsA4EqTJj6|nYdU|CGGcR z`{RbTbZgZ+`46@CSlDQkCU*lzg*=aVx3;e!B)*<0F0uEo$o9Dsq;za;Py4wi;E*I*w;QKO7T(Uq|+n z9xvLR{bO-me80G6;X?A8h*1Ae0MhsT$)}j60IOBgsw$GK6G(HuJ z`j}6)_gKb1Cfh6jrS{~~*7l1rd7p|i{?|8y7{$%s8MBVB{HPM`d(qxw8Gk^wSN^BQ zSM3yhDgg1?za6JNO$lpz`mrcP9Y4%$(*I@IUa}v(SoOaa?f+{W`BnSNvL)P<+uu4q z`2^{7OT8qtTr%Z=>Z;lk_;%azC@$xxWw_{kRrA(39`UN?g|(fkd!&OH+i{SbPl`@m z)%;Rd)oxqS+VvepWksTttSVmA($!E<+_I@)ZSD4=mUZ3@ja{2I*5o&|Zm8W+xTf7( zRI;jMeQ{Gm>+;Uc8*5q`{A<^>RP0>du&4>JwY7vVZCSQv)5glC`VH&zHgBw4>D^Gc zqouN{cJ0pcA%UVjB+DkWVtM(R`tXf{aI)8I{S=Xl4mafg^conSb+PtFD-`Kjo8~UqQ zlfVA)#ub$H+hy=(F}wXLmf+AQp~y|{8K`p#?ISlMN@i?_u8_@?H?k0^gr*Sgl0 z=0M$!Maz~H6mDJY+q$fN>ypL3CHeV{^@aJ1@|P4mywzLKP*5l68k)Cy8+_OMiIsKy zw^is@Rq$&kw^PZx!L!Np3l=Y4T#%Qyl+!HtwsiRF#Cu6Ceo)5eZOsejF}5q`j(&V{ z&L6pVed$XuPX2;?ZX&f9Yme73ZTEC-T52{P{(@cBiS7pqhwCYI zZ=st8_&J5cwN%}gI$h7|->?&&|KYEMQ@^|z@VSOyG~0qF`ceZv`m0DQoiC500bUR* zEzz~y1fA;MdVLXX?d*MZBw<3!2gjLULJpZcX#8(dluu)uU?F8qMhXYi6ph~>w zJ+I?kqfwj^?>WiF@E-t9{G8!-U`)*` z?K=EFkHm}UtECBElXB3Q@y2~a%IlUUQ1h!9^wiH~30G|wNx0gimdgHfWIxr?WJ=F} zFZsvQrFc>E>RHL>%Mz~U(`&My0ck)5-vD^J8WcUPL7fVo;>+P%Wc?onn0*UZqR+!P zR8EQudka^BUy*RTge&07p<+!|~agFd&DJS)>cq7u45^n8hql8bl^rQ4e{TNS)dJO&3b52xw&Hy*kAHNR# ziUDqvH&^Q2%I9nex5_&p4P=$~bqTlj^D_yz%KJ-0KSusPGQf@UX3F_P;~HaR-gV$d zB%Js$u4d0k_(K->fPvm<-`6DE%0E3thYImCMcnfD#S*T@Rle9H;c8q3KVqOaj{BU1 zTgUySgj>f=mGhRyHO?28gpZmp5>EXnUPyo~2Dm)p;c^`@z@vZM1I%JMn3C7csTuo% z0dAD5Ny3#L6n%??TjhOG!maX_%8;YR&6DG5k$W6$SV38e>iJs^BVaJZFKc_%L9Bt2i)hfvcrgoqwwS)p|c0{83Tx9%;W7 z7PwmP{T8@d@0m!uJL>A2+q{8phCM=~ccr(ht)b~r{*#kscpx)=-I4#s6a8ExyV;!w zbyU~f){LKdtncW!wPtNyt%9LuUk4j|lOwkd{O2Xdrhd#9=)g}yNKu>X1CYDi*D1us z50}vISeCD@osZv>YuF*EmO~$FeVs*}`2ETHu1+681w9&?ppD*A-($@UJ`-GM&(|b> z?rv-rw_-f{`HH$W`gy#SE7z|oZT7bK8ah|CH+HpP_?z7QaT~O>dmHOI+v^%RV?rRX zC-@XqA&XJMN_n;P1jAg_Tx9l2VQm zBnuikI`fy@v^MULz3|N&H|`In-p&Tz)aH}#=EZ#MP`}-JhyJpwSYSWl`(6YQnQEDO zx*`+QsWSgiNV!a_Dx-Zq!DLzxg?r@P85zeELd7dppM@JrxjYIWul}9l<~f;<(PwKE z?_@ZR=3svhAdPR#t*NC;244?V3`LEvuFuN$)VHxztKi*egvQ2KP(l+=RvlKi2@1up zqPN!nyBz;|?LV?06yK`9bO>*?3E@@-6ovU}rQI&ukCI=uf5y_l%D)fzc=mHh|G6xO jC;O2rsP-y{-j9K;V^fo}9rTVL9+&bH8}uYEYy1BX%fA^; literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k29ad84cf333079e0a1d3b932c4ded45e.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k29ad84cf333079e0a1d3b932c4ded45e.o new file mode 100644 index 0000000000000000000000000000000000000000..a26755b517583be80f75fc224faacbef4c106fd9 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yRsARA`l+13MuWI#e7UYl9bk{Tq?qfbjR zEkjO($U4SgFk@z8pN%tQ;>pl5WSpJMIGWAb$QvKxIGDkjY}O}}b3$Z>kpMHe<8>f0 z?*0CHbl-c`Ai$p3Fcfs(TmS!8)&Hvc|Eg}?{_@fl%XCd+s%Y$07ON>JW0}3Ndk??$ zuvAdGSx?P?t?uX9ZrlH%7ur5MgRw(Hw(P!l@j7eEp8GGjp2O>3agU}G6)jz_?fgOL zx8vB=57+CV52|dTYc_W1|JhjIkq&kUw1;~#Skd>gJ^de+vbTSwvqLjdduqud~E;m@W}iV;lBAN!~0g93SZoJDm=39Quyil?}Z;~Jr_Q9^1X1%%tt~a z+xHJX{me6isq-HW&1v~g`24(M;k}oB8h+aOR=D~2#cOQ0(Xg$^7T+;)8b;NJPi z!j-3vhnw3@h5Pou6&`u$V)*>_i=bH)s&F0{JU{=*!M&@{&%W=3=d@z{lW&D9&Rq;| zyF}ytaVRt&&J9=!a$N?K^dQVL~d}@;Bxq8MnsgH1*bygI$KXAhSli**%2R3W0=neRQ zrV(yZ-;nNjjM^Pxtmrksa(Elj*AL+*A5@iwK3rb{zw=zZw)6F)M*ofHk002yxymQW z|CH|y!1rE-Z{G90JKOhqZs2=I#_+u(7T+VERQ@+$@jvCSH}b)2JAW98lumwo_yPUU z`F|?8>bA4A3%@G4I`Z<;{$HMOYr}gI z`iK9C^$$Ow9lG#8^{W@&Wrv3UCLuKZEq3Ta1J=|DtbgQ3lL1Q%4L`sR{p$mu{p+OA znf1FLrMU*44_({&nrSYH^#*Y5xf1tB`VyfaekLJOZ9cMw0IW_#B7+y^zWIcUtH@$T9M2LTChXyazdq z@{w4S(fIhg0_9E_i=v3^=NKCb#cQzEWt|8do5eHr^>H%p)E3DNjOKST5M^D$E5k8v$+`jwBXH11=rOwZMnd7e<} zqT-PA3nd}44cUb3ME$)By`F{sT!^26WKaKg^#4s(OR(9$!G1A+V23Urgk24GY??0gL4NviRdMKAQppvv z-#vYp_3=EM2Y$?7^meQD>i25@L1QjFVN;|>0s5s z=6Z2+=0A(8zMUm{*kYCdj;0Xhf{vz+&OoQP9(U^6-74z*TG!Fk))-_9+S)sP3)&uE z(7C<2ZA*Ks-q0oMk1vScG_<$+y`5}9r!UyKK!{e?(AC-E?dYiUwzO>Zb~gC}bq(H* z&N_+h^tE-g2mGCZ=E8!lt-e-oOY_#Y{D<4~>;8Q)7@#h{sGx39L0(;cUS3{dK^^`T zEXsQ%|KY`Tb$O6e$aNQp(b41$_!{pXK?1%GU*K^_1Sf%eY!&Tc=N6Ry3jqtn~gnRgesZ-dw0>I*d7 z6=sHMxGYv%a-du~xdDxUyTFD@9J9}I=v^~~m2J5|*E6sTD#o=~wd)Xhgv>;4e7To~ z+)G35w6rje>)%U5K4;mxY1ty{SOGVmhTQr47_~^R!3JwV`<5*ozRo;WSJ(DB>5RsmsGwt=ZebqjwZfEs?+6RXN#>BGVioZP+ba%MuPxw1U~6a`^aB@D13k+xU*% zjCzX0aZ+e0F9Dw6iuKczBM9oj$_(6kBt6J@*$KG&NLmyw!R+gjUJb42zYDkx=NHxz zLCKXsx~BrZchNO1b$3tCZ03mU?wRf2w~P^dp0yKMS3KLl4uEKDKTU@qRK@EsUQ}PT zuA;1>*jPC|bfwx|&-W-5uQ>oxWCgYgb37yV19$xy|QZTUE7UP2IYsHESxW zR@E)tuy*OHvZZBgewDk>J>MH>XmWS7;ymCHuNy-XuXMhG&p#}@Zf8#dZeQqND;INdhNMtH~RGdYQzufI?3m+4%>*| zq~Ii2k=r*x;RSt|5#AMWP9lC&&qR{gU>Ym>GxU!-m-kKHpEC_Le?9s`|4;2@5weHf z0ecQrvHN!tEfvUjF^zqtdwhhOgYdad=tm#FuC}(8%ll?FC5BkpyPM%Eu zHlFKW)Y<8mb)}#3tglP+!k*|K_Hc41CwqXOq5zvXiH|Spy*m>k>;nME2jG7h0b7JT zGEyk;OS?68nHbQ7e=HB^X;fv}$DAKmK=``t*U$(0*8?`@Bs?jB>>HfPhmQ<^|DiN{ zD)}=OO8;i)2l$L&?o3)Y;a|LcZV=`euGJ=!e1qAn{Ds57Qo;D=iT=TvS0WKDMnA_% z`in^UmUd&ApouO0*~Ab0t0?f`gWO)47_TTWYYvx(+Mi0p{G1nIXV9MJQ{Hsc=SJ8? zS+BpO%?!?)L*vaO{$QWo-Hv&DzTH1PmHnlY%?!?+MfkZKzmPrI&E}G76Ys~OGh7UM zJLs{t5&cmo$JYt=L!6{gf&VJ-nZSDles&t~KNEF< zPu6yKyMV|1r16^s-i5lrC+I?+%@Ut2`QI$@8t}xwU*PK(!oMDTkmTe3PyQ9O*|;w$ ze+$l;L*m5nvt~u>uk-oB?ZM|C+LONQ=%4fRj!iGmdY;j`8j0-T_RoZUqXqQEs!sNX ze_%f6^7%^h`4?%db5S4NpP3&1noIWN{)+L)zcv$zoDUhnnKLQ= zaew?d^}pQIfA7xeF@G95fb}iGM*hYZg;NXRUs&&>{zd*XJ%PrX&i&^g*>if-zlNMz zQD9m^)L*e4%nVMOM)Zj>f18#`b?$F_(B6gi%HLLzey6!i{^kJQE$n%8I>$Ru7wyR( z?Z8WagMUN6Go$)d_9h0%-tUmU=^jPuo5r&%{~RiXemVtS&gaqkmuT4@`kzVaR{lZy z=buOs9^;k2u-MdJ@6L>vzU$j8`lk79&^Mp28C>5PQh)IOjHtfrv$0;dEc$l2V*0Kk zeYs-uo6irlm-^y$jN&b_xp4mnIKfg`J035gZ%BR zrjO=Ncq?CzG5?msexiu^XN`ws-_t}O*YBd>`0-?aK0e|_d2%kVsd&)46HC8}Uw)EL zkI9$gM7$KAmEOnsP1B1g{`hS$8Li(me{>!%b;9Gz4S%Nidj)iepD(LAjXHpO7v0DL z{DE$!)2=76c*1zd&qm8HX=Ce?%ny*iVgAe~epEsI*bdn52>>V`fd5eZeT5v7+J8Xu zABn|x=o9i-;=68$@7ZzUIdWh!iPWzi$c)A7p@rJ=fSn?@A-?N6*_X$6=D>Yk9H~K-kmlf24n#F+SUm?LfYvP z{>}46hs-ZY9$r-b5wJP<`usms{*-Kc2g{in59^r|TLqC%^Ytv*M$v}*2jnF@U+*&f zjN~oq1)dKe{3m^vEt%)X4xS&|bAyiL+5Z^Hc9K7#J>=)(@%h5*E^zmyr~_GQhhQe5XQ%NIa#Rl z{U-W@eKW~j{4^gRZ&5HaGaApV{1E@W81MT0(E5ddP6Wcg&HjFn=96(gVEr-qJMgCY z0KCcHfj7(t=qER5_J8>Mbh0nccffB(&@dk$FX5y9e^H}+3jT#c@$n^m2@&}hbE8iB zJ-ZNfqzlv`$#b6eZ;phRe#t+Q6D<18q@~JVFX93Cg*Mdutit?GKphH?&i};p7(Na7 zB-D*~7fwA?d%~xqZp3FyFykkn9>YUk?yp=QkeBK%a3J(!!+g#iBOm9JOT?G+Rpxm1 z67t7gES3Bl{xpugSw;SoN-g|le!GY#Uqp?HVZ9hemB)kWWWUtsQ5Er=^qq+9h%ufg zCQ|<@o+kvwY+^MS1^L77ZiWUZbUuRp*THTqwW8>>Kga%99AeV`@J;UTq@9SrM6PGa zizK<=FFl~A(uql=hwOh)TNce11pcQ%?hK5N5C7@~KiaVeKH{D5dIID85a12Wmt77V z6`e>^9m;3?BrxS4l)ooG%Y=o5zHTR<$@^6E89(P<>6hm- zQU8nOGyHtwY?}Rs`3vIPb@P`$h5sSGeb)KQ=zPVI{E8Rpr^rVz-&_vzM}EEn|1{?- z3xPMT&$M1y^A*GTi+GWR^%~2Gi1*N+X+6jKY+Aoj&%$~grAK?LN3s3ZkY--5F<%KE zU9ZQ?SNQz490T#8Y=xG+!Bp{yup=l11w)kC*7rl8+!>n(`3`#+%^= zzn>r<$r1}b#phA^(Vd%*kpKLL%tz*#;@9Z?H)R(na{s+Q$m|5+`3KJL%e!6~GhQ7r z#Vf=4I{fE$;#GA2jrGmE-(HB%7slFe@8aU9^@+!4I#Q$0i*hKQ+{fqp7CBE|EhiD?eXhl+xB3fBod-^KbT z&Wnz$XYt}0;zjzu^ZwfuzmQMq0siSj^z#S8N8=gzPe(kX{ZJ%R23!68l4%ADI&-!a`#IosIo>)&%-|&anTUM*BrWymQiqB)0!{I>*?5<3NSV z?ZmsP+t`1nW3>c>V!w*^F4Uv@QRHhAjOR_jXBf|;$p14b-xT{#J|8lp`%k|ACVxVS z#=FUrnHzP=SI;ekK3%#&pRj+rZqcV{KQeay`Ujlv7Zc3+EArd&y|)(MkRKVoACSgY z(qR)XhJ2Qm1i7Ei?_?2*hx`re4?ZvE=e^=Q!*Cvc1VlG+9)1+(;h^kEd>_XuO=*_> zb()U-KKmL7xIbh6c|-p2c{uz*oUdom=coMhNS!_~Poz5dU1`~G^Zi;>J|`pj`1v9U zDb`mzIS6k*oKF4I$1SoyEq4B&cXRTZ&kxer7r~F%&!PjI|CD!s6w6QW`GF}v`3&li z>R!30fqNRbr-6GKxTk?nT?2G@j!-**Z#l4@UtIhBhd+vZ_2s_m-){Q5fB1RjxW8)r z==|SA4i>T5eNXqgYTbQvbB^})G1hOdeQxCaf3AysjiXXlC;Vk!PH$gd3OoJdSAYMz z$UAnnw08B3qx$OFgw?g!jUc4|bT-BHeEQ<3rpYfY@;a)q2j>8Pg#b!WZJa&j3Xjmm zJW$8#8vbS(ypMCH{0`e&#`b7S5XiN92KrdhbDW79s{z2Kc|_tJMS5_jjY~v($^bNN z9`8?$_gaD-^^(+*M0ypcdhA;HrpbaAP@{!{(->D+2_VIz#;B&AA0nRk7*}2cBh(kY zL_5ygI4f^x#$xzADONnCJNja0!4Tj2Jz^l^7t8Bt!c?RG#!O;2-%a0=-Hst zB#*@(I2E@g*<{ZsTS8>U?Pk!so+R`hm77M2gOAGXijR-V?G|#!>mw=`7$I4eWEc{~ zAesGBk<32YqsS5_&LmBgGe~zM&PLBcGv_wl%6VL!>w>fBsOZFKd!g&%^0t(CGgj!S z1Z^~C9jzKrgC0lWi5t?vYF%IXAm*N%dr&WJ&U!VE4Omr-{0LZay`G}yI1+!yR|;*E zchT5tMX|0JF`llcLDg^k6)4tzk8YLOWf(V}ZKIwEx_EeH1?p$C)P|8qX%h{M+D?qe znVwqXyuvrXwj}LJFJpMv2`@Ap3gQL+s6)d6#+Owc-)J$_pF&+RN%R<0r#g%#;t%R{ z%a=aj!G?+c9ZP+&X2yFg^=eC<#-slE5`q3Vh!brB@$qk1Cy%(*HlLKfIJL}vt9`_4 zb2Gf3<}2!0quoqTWA$Oh$LsU6PgXg=@txF+sD(kmFDwaH|k7}6N9Y$$D>zyfYh-cX{mYbGS{C1UYSL#T{g?w#lJ0V zSi8*h?^xO|M%HZX&ts`qTk6PGufKK~S9{@mW5u@{MFVmi=_TK1c6~akJfK3Wr%}%~ z)Co^@u1sxQDl$CU0ob$|U}LqW=eQ7?J-bQA8Uiy`dc4JU1(d_dUfrq})T!4Qs8d&@ zYZ^62)noe%n@*z?r>?aZyB<%uxzQ*W8A~_uauwqAR&oJ1*fr=_M<@d%R!AeH<55{` zoLtZ3y_P(W`Y~Exj9?g!A<$m}!mvEY`?RW6aMkYR2RKv*X5U&#~&p z8Fnv*i#22{oqbZgQRnw!xOo*9u|ma%(Q$|NnDI~2EUUY2UHvIu7DU=(=g9 zCXPNWwDJ>=-m)JuM=LJq@96y{ z=ihvm_jx^@fj8!$y(0Qg^!!?1LFH}V*4NKfAq;Xm%z}!uJQhR6mRssGh_b} zuh9I@+*a#ag$w5kG@nE{ZO4I4oM#wmH++WCBGue-?O84W_{atVoxzERe2`IfS(Nf- zp8pf~**=@x?CBSCUi$8f6I3_M;D{`ACObTZD>!os=cz~L3B(o4c2@aN@chK}Re&kx7M-`A1- zq{oYPXZ~1R7u_$eS-6n=CL+}T6M!^6iU~DwqX1IJ#v*B>ptWefC(ig^k?nIOC5=zT zqCV!6?LC(9kIMGSf2lpWw6*;rOx~yCjQ{mbAVzWXcgC#aD?h44`(Cv7SjHca?Unzj z@l`top9(;{_V2`LPgBC$o_;I}QO6H6oAiHKwwLTjFIN4pLHqw2M}F1*vTO-A<@UFZ zPd-69-CQpTEtgC=pt`E|1isyNJc`TtX&Ej$U)8)djz_$zd0}m*>K^GJ#&#Se=aZsS zS2e%XRkhn%yk=cT@zP>ZN>`Q?v~)ETmb7dvTvNNPxMi((ePh?g4K?`*-lh$eE4=HA zwzpJP)vno5zN}^0W3^5Gnl&xU3u+&$*io^epr&P0>$3d%<)Up#%aWG*4JCQLO}yU~ z8w!?n)weFm0}uZCf`@Cjm8{yYo(CGAVgx2}~m%lbv17wV@LgZN z?1_f%qGrh30$n{(Us{ys+fd@KZ}q`G>niKpRzIMwU@2mTJ0?(rCl3aTe>!t<5jq_Yt!;de`D*qZs@OK zb^f|18kbkLY+PRed)MS|Y+F;?v`N@&TS?^>^qtqZp|Z>u zfS;(4e+xHhijR-FLSz{)4yRSJpaRA38#K}G2n9z!DzMxPxNI5eDqh5Rybc7M+3Yl zR$8KKxd}Sez4iJM+S=Ls>PW(bmJg0I!Gs)2*)bHcpoy;@2w|_hBeB9{tMez<-bS@#txP6A%8#b>Ke$ocKA-?ZB9t zSK4*>e*uXX(N{|oye{RSG2@NQnRT?V)@RXi%; zG%ijPqve>qqj8P!GASqZuXrQU6%uakXM==KxAdd*Mg160iFyqE({oN#dENjw(jU7H z{E7i?ls8xE-OA@|3Af5SAPr=d_YDcR_VY6dx61oVLqA6TKQh3L@@C5UL*p7_WZre) zhb5f&F|KCMOZY<;_<(`lXy4Z*+{!;aMTZLUGez9;_eBz}##O%9B;jgY1wU+{H;((f zgj>h`rG#6@O_lSO#x>3tmxPa+FA`4uC|*c_%?7wU;^A@~Ho&8Q+yl%aIhc~y&8Zpt zf&p%nt4YF@9u$3xgj?l(Ny4r2mdTK##?6!Ct}wuj;|3*MjjQN)Nw{^~?@73IT*_Oh zQ0_ugO6iW`8Wjci@ET)H@~Yq}Zai;+tN1WrfvY$$Y=Nt#SDk;V{?&Rv8~jmG@E&Qu z^N8&Atvc_9jPe9r({nj!pfzFVKOXhLEB**9RbXxvx`* ziytnb-?1!TRXZQQC)coDP%VQ#*7!P$JMsII^j%} zU{CNVszMf{gv;6kir`kqV!4esVKZ-FYe!v&-`6nucSQs3jowZ#TfpCiLkcUcfFxxc zB}f)DbaduFeAC*vL-xWqZ``;)n0h-KcvG8CzMB{Gu|xfK>mB;bu3~}xgztM0L}aRE z>gkG1P^ZfLLm}ldt*VUn`2>?`K@{$hcc*0>QwSBWRDBk1DCP1ffV}#5h@0nSK1QFd zQM{AkIGTg~J%BX6F}J3cE*X41R527azPdgq+f(1hQmuk_qY)Y#UqJ~?JXv*E-6kj$ zzlz>k|L=1A>$U&Lf>3;`{?Z}5)h2{n5l|H7tCe<}Y(GkV+5TBe11tYN;N#iPA^qpF k9G>h)uAthh9C{xHwvJ6r&UDZ_et2BUPi)YWysYj2KLGg}_W%F@ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k29d76b4da51071cbe677d0d83ec77d99.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k29d76b4da51071cbe677d0d83ec77d99.o new file mode 100644 index 0000000000000000000000000000000000000000..1b766a144e2030b7df638bb9bfcafe0a5fdbbd93 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAPh6{Z0i9+G9V!k2sX2#B{fK(N1v8t zT85kmk#&s0V8+bGJ{xDq#FL?A$T&NhaWtE=kvBfXaWI25*{n|{=Y+@%BLVi{j@Lnn zaqsunqx;^g1_AcOhM}PQ-unN)s{U8i|5tVE_Lorp1UeP4%Z2cX)9;2;W7}&oS{T%pCcuot(KmBI7?84>n zj`wK1KMsZFW4yJ8(9g;6+?H>Lhk-A@a5_Ba)CKTP{rpjA_`soor{+IB5ITJ_d=amv z_udT8J#{Kv=6o-FasCC+EDz0Td19a!x+uGF0)3whhfbajmpjjdL-Q|(pIS>iFAIfE z9vql^>WP66c)mFQ+u{B5-vnRK17z>TD|yZrLS@k9#fLxxy`?06I~;oV&2Y-LQ{g!m zE`hI0;c+t`2@Ml|C3t=4R5-Bro2W(UcR9vdrgz($(hT~!C4nxuI~&eR>teXsilzISX4-#cdUJ@QHAfBhE!Q~r7*AH25vhoMOEx=v|>%07lKJrd6>pOoR8~K;TtnWe^8@ZUoMp8gqh`Bbj zFQISfpIG0}1KN>G|5G2i^bR{R^fw8ip>MGxm+G;mPGEh*Kbj0!Vrb|AcI00l0PSBV zh0bl-`v}c7@O5)ZhTzv!w|uKgP1jD z>;q4vtHwlJ(f;F#(fTs>$8MHB*At@giGGIW>EmOh#2@2Y+Vra*S7_YFT$!GclzE;| z>XO2c^9w~GvJKgU>_q*&1HGPy{#=Nkfn-nLPW1mxT-V`Mj{9z058`?fum6qvGr0a6 zjSqXh^K3%M2D>W%vyEpX??C^qnR>_t`&@$GlAV4H8<5Q}!sh1|v%W#tkL;j)`u+8+ zFI>#}nBM;g*;Lv8%C}%E`0~5(y__T|AWR{>`Fa4LD89}QGgH1I>A=q?Hz#*ZyoN`wW~$c`L(vav9%${7PPi?_!hK2 zwxDBYQ|q?2SiQbe)*o9Cy{T_&@q0Vif(~D>V}TH@w!X8Y+1uV;>uqk{?(Jyw1#0WP z?H#oe+u>_%ZwvT40!@n-ZEx|lc$=HHx8^Tx%dh?S#bAKi{DMWb`OB8n=I7<*EnZZM ze~XsnJ)FOES#50|

t<1!A-}dIP?OyGM|KuiY1T3=+Xg0blc7;cbw^dshh3;H&H0 zUb`*eZSe`~SOMMzCJ3fkJuM9ZufLH`I)9+8qphRMkLHAHYi;lFwsz#*1@2q#^|$x} z^>>AtVHz%r)s`G6mria#Bj7Hup%TaJvmAQYOkrhPF3|N1EQ5-1EmrN?MIIqDksDv` zr6KpykUK3cjN|(E(vZ(t_HJ6X$U0WQ&8H!E{ys)6((AFoTF|y_Tf46#kJZ+;K2{sW ztXZ+9R$yxLuvKgFw)5y6#Zyb^eiXb0n{^xC zv71p(ayU*3E#*bPGhDHLx^o0U9ax!xTaTm%885p5cOOfO!X=n}UDB(e75#Ssx8eN4 zS|TX95=i${!1o@yrls!f?w-vYk-gos9sHItg3q&dBI}B0`_};wZS8025QM6D9mR|4 zYd4maloh&*vH!<8iYRnQLZnOa+D0IMM-aYP+}YrD)9FG}v$vzEt<~Mp*4ErU&yAVw z^S1lk`HL4Undja}mjwmOSy5+GbA!9RtG&b5;%@0|?{GKxwl%f-+#4z?R-f>XYg=bZeU$VftzuGu7HO`@Lv2VEE5p_UXdQY*`zaJ;lMuySr_;Un=1{akSNf*LEOKXdJqJEU(w(^&?zM z2!TR=F?zN!*0WoSuwMa51{zFbSzSV3e3Ili*@<0=5q& zf8`Y$`$abEIbe^lD*zDF4NkUqf1;g_yE!|Rop7@sbqN{Rlfa|Do**WY1kv~_x^%Xc zc+tCUf}i_PL79oyc_%v(u-nmsMfwk=vB8BbH)y-hGxY%$(GMlD*AKHL{FPxF z@tYK!1S@j;CMdk14>Q8M0?tXqZ|b>75*tWk1%HPAQRni$!TWQjq2{kcf9U_o{VYQE zusdMS!3uW&Zla|E`7WoiuXK%%aB~no*9raT<2P1US95va$cDU=b@HW%1NuS#xxvYk z$=}9v{fjy~`;xBoQ=0X4XMja?xIG~pks0(u%%nf5W~#}yF1Zu>R#f&TS?jX4QVN+A0NXY%1A{osEv z&7MmBjD^y_75V`_BbYmr)=l^qZ=V~4IfkmW$t2%EHYkSSDymO9rd{p zc3IZz-qU6V=gpz<<`I9e&)zP_JU-vEZ z1ic;fSlfvHxRc}Us0(~C@NVGUCcGPUfnNdqG~n|Den_;RhPuFi75GfxJpw;FjrX64 zy1*xEd%Ik~V-cnC8wK8ly1*ysLY}P>pDp>{D)Acd#J^wQ>lVVl9(<7GJG@yNfn5{aA-8Nrz| zDgJSP{5kc%%G7_)?&&dq8a#ycEx|_q#utS%3*lc_@1y=j{xdy+#+%Ol=P=oGdepxL zomxR)T0+!cu^!9}PMb#bi7|hhmPvK)Z~M^Rh4#wd){=gwxlI1%0NyR^d3-v@J5U$x z$sg^&OMioZL%%bl`c?KO2FTuTlfLO5Me3Wzvn&4`EQWqM1YXYP(fXHY*&h0zN$OVq zLHg&PND&_6l|H}B)L+l;jF`UbS}ppf`EAfQpRXBQ-x*SW@c)dczU#8FUbrm!cDZ8u zt{{E6V)L8N544y1M*kT>XbS>F`g_FsOY?&pQ0gnoEHCvJT@MBeS!vd6lYC?8t7kXz zfpcg=>sQX7Kz^IS-$;Ik#oz2ae%eX?)3l!1MgBnRLmjP`qP>c5GlLFJ?}+)EgX@F* z?YyRs=1+JlUym{WR>6Lvi1}xYhh*QgL?GAig5db^WPd(B;zen4F0ZM0(6bv$zlvXe zl24Dxm*Ygd6rYve$N5du3n>2hZ7>yyk6kiTL6%qMt@#Ziw&MapF00U^0o+uO7&Z#p}U^+Nyw^BDW#F>pIz&$9LpE@MnB< zO#a-RMC)s5vJybp)9inwf15Er+fVF5zME$9KB4U zdXggOHMEDiHI>gKW(1Q_2SfejpVVIphKu6u8NqR=8}T%KjrauAjrc^QTSj~m>M^`! zJ>%<}-6iA6nEqYCuwHZj@NcF43H)O}O-?GxKQKSfp-T3J{gLlyk~@lcNcNu+vp@Xb zJU*ZrX{i~5uV=+PKSRbXGZ;Na3S^sIhhb21@uF- z-*9mMNsjJMvT~yI7ihoXNapq;|Ka-+H>b}cdf;<|$&RQ$A-_!zxU<~oJwo*4PkS7E zeZ+WaL=S(;H2Pa^Fw1RhU$DpRruoL{xj)(*u#WIYj8E;O{`g@UdnU}0EOIWAM*J1@ zfL^puqI?JWw5|UM@XFV>G#BzSceFpqpB|X5PXT?zh50;Gk(wUJ)~8Z^GA0nl$9Oqe zsPp|M`h$Hl$zA+3A0Tf*Ff%h6&#e3q|2-J*`ux!Pg@8^3!oSV_ewgNyaXw)EG5I_2 zruhK8$=`uD%m?TvH)!^M`1^FSFVAMn2~^kc((&K)Bk=aWmsmx~qV zc=jUl$2}~S{2Tr>j=fPq{*+2B{3U+7h$mk}jf!Et7($iDgXv_y)aOtY@tpLXi0z0m zo+lx!^B7pr_J_Nu-H$C6i3dL7o$z`B5YZ-&6XzWqGh)=j#Xg8(*K#n(S%K&rJCU>~}E0 zkf2EWS=U(ZT z=QC0Ni{&%?eBykX{f7Aq;@fremp_I7A-;Xq`OD~h#gY877wMB%&{E_b`l>K2(oIhqx1~%ftd_VIk<}2h6ar2cKW8^FD88^;X zvUtAYj-RhUo-8-T_h`NnG~^qj^A*HXcUClC8G`;kc|MXw>no3!=+BamAYPjC5eLSb z;Re5-ARoyR3qHl?QTfrGn~#wH{D;g(=9%Kx=>0ck7btT7eK5%E1mXDy&hJY*Umi1F z9W%u%!}&V==XT;%bpMU@&Ai`Uh|d?s+HddS;;8kB$7ecHqtAQu}{rF*g-e8~1_v73@{tNBLW9K~>5BsOs z`4!gx(fjeK({3E!rf`3s8b7`veoUFl_g`v1K03ac&VO=)Q>R$sTkQPDy1&Kzy@~zy z6k~iV2u{!$KEM+DLz*9%6DGn!V*j0u{dd*``h3o?|DHzsMMJ!E(uO3q|8_dZ*ni_d zh05*3yNcV`f2U)$1cPF~iuNwlqx(_hYZHv;O~7Xu&!foyGb!H``%gX}GNb!XzW*kF zLW#z^$&;BIb;?&SEQCH?x`r_i$106! zmi={_j{QFS8VI;QWB++W{_uG?{6UCgxT2wwKBl-CG zA_*zhS35ZfZ$Fey{nN)SvOg_${-1Yq@|w>N(%2WlkJ!(m1DyYqc6}7fPw@GHDL?rP z>X7PQxu=198n~x{dm6Z>flplnba;+X+mCNKuoMp_x3W@XRm&C`2BybjeL!xQr0E>Wp7STZ*K}a`{P%B z|GUWBcDAB=-HhY#qs<8*>0DqMLN>FW_J>@Eo z(1kou$LbpXW*NMXbEfu(q@Eulp7Jf%DOGUuYJY%4xb!uM2DkRbMGiC{Te;^Cu;b;WN_xjZlK z_;@7$qJV*T{qtO(C_M4;BG&>Qiiy`g%Kz4w?v)MV(MRR+Gqz6Cl67c03gZy6Cl%4N zL8nO`i$8EGZcDPszEQS>$c)>~pm#k<=shYojT8qTmD?2`AC=oJY`==tAeY8iBB}|-2nkZ+G?na!Ao`Yu2ZMv27xH{JbXVFp7iP82#*M+66De-2k z&{GlGXv{iVC7=d9j=~c+q=R+3zUD#9JvaBD9@?DsXdWA|su=kZu);b$MbB|0{*JE{ z+9>a$vDJ!VT`^)jT~C9m-}oy~to=USDznQlZamvYJri{C@X89*&uFO)BahN18W^>m z7>_eOwaR&wZ+>k_+SMM$@URnJXgCza3;a=sh69W*t2(~XVyZuhx?+;(F{n;;7)`_< z)ajNleZYea6aCwk`ZCRo_gLzcmO70`{qrRP{nv;SZ36M}Z&)XfxYahFl)f;v#D1%N z#A|aiyr1PO>R6-QOiyF=Va3Pm^Rwf|%H3FX<4o+4F^GGaa4Yij8~Rne_G*&Q)EcwB zX3|QO8)3%l7^IkXVOpg9Myou=ow5Su-FNyi!jGgf-M#dZah!^vLVsu$F$*BPi& zSEOqiHAmHB`wW{-qZFoYuot=>OS!qxC>I$^H}P^6;`CN>0XNt+=vYT610+^RBc$U| zS!|qK&*Z(9JdgS@T3?J{7@1i|c7w!#zZdJF5BQjAjs7gw$WLR;&n{}lc$y%FNTXXWGtP1QoK>;_hPtt6&JBW#fQ;xhxVB9Ptq)_yKY_mDP9`F6i&* z{Uzt$e3tilJKOub>(?}&j5{_F>tj)G->4m%2OboQ+54C=>Zw`#maeZY!Css3vv2Jx zypImUvD;Q<%n|r!B_8n^79Gi2&e4UBoACYO`*C9XAM1bg$vBt5(7&$n{<{@#^u04< z{}Qjz{LkFh=~|f!=L?!y1_GVKiHCfUQDsS# z@@Ag@6ZhFZTiooa7jjYf% zq;}(v>hf}7k8<#Y=l{IClCiC&_}g;4uJ`;OQV`jKuJNAjL-1eH8;T7lLD8?)p{}(1 z2;lr+o>N`_Z4U2)9}Ss^mUH;%9>jCHxLcd(#dEqcOfafEi791c;z=(?qN#DC7R`AP&*{ok;OI5_oGuAb^ihJ_yrG%SkLfJ!J+fXO{Xkg&Vk?8F(_Wl*wK39U2j;-x!KlenO z_Qx&lRnOM;GlW9?9z`Ipr)B#OWSz#R@0wWKli#C@QT+GIDdIM?r(4xt+3_$sCYXni zhH}m_#nkw!owEO9XndpgijLY7owYs9b5eoDO&KG{MCr~ES87i*-J%+O$Da)FK8vFe zd9zGjX;4}CAG#Xq1pY3(j(DQ}O+~i%n1B{11CG`g;=jm3he!DH@-sun@$Bb^W8&}Y z$bQn}MZ0ri-Eq8Xv`knz&g2sbgc2v{BFowBHwJ{IAINxssB` zr(#hb^U3xe%lOA-d*#2>o?P16ehDV;Q*p-s`W6tQxcNI{*721em7{$R+IuYH_sjOm z|J3-Zoq|sVAYS{o`iWW6@)-Nt<-n@8y^^U^k4c<)+ottZ_^6OhRRqrfV z*XAuOUQ@iWsIk6fWyhA9s^)tC`VGxxyH?gOY6NV3HQ`H|m#^DgQ{Gs&X=C1&n)21& zO$9rf%PXqa?B|{W9i1u z`bEpTXuO6^Z`(q;%Wo@Lw0)hoXl?%HmhCHc6jpU@-c%5XJ7(Fgn$6(5 zu5{(&^<4!`khdASdc3Zsr@6zJ_eDYGaePaQ&L1*0p){EgKu_ zwp6thZ}wJK`>U2W7dN(*Y}#JwEh=8KvUF|!meLiSn_HSYx0K?wcunV)RptJMmW^G| zU)j3+jgL31DsSGrX%XyQmA|=leRbm&VXqxU<=fDAUPDcJr`0arV*lfto0dGH{7qeJ zTbi2!wL2FrZ+LiV-I4}x0Uk!0Uti~2`tZXIc@4`J`|5GCe0i;)t8d!ot@mB;CssD_ z-&UbtRl%>F+)gF$2G1tXU%X_=lEry>%Q($SZ*#k^R=gKi;|FDY-j=*z9%H+6?&!xS z=lqd-_m{o|M=O$8%vG#Zs)Amf)rln@%K`!|HNM%4=a}D?NaBtjWK1mBMl?|!) zd!E&g+W$Q9zd62^`VHp`qGzIS-~{4r&lvPH)=>lfYwCWL^Vqm==r7o1o#=j`aJZgU z_ZGTofS*@5Tuan_iPQD0{tY|f`5*pDIQ7em0iSCKMzbw=qAxMvqrZx@+WGQ08sG)7 z(h^r002{=m+D>pDc{o7D2dcz- z-tz|DH5$b!@t!j+=y^mO)xu^NS0M#`j2|;x?t~N&S3z&wc?)D4E*XZMo05ydB*B&c z3w<&6O~|ESq6hI#Un(bhj|3{e4@05x=+|5a{(H2KM^F2kc<_g>1OEZw#Lro72gcOA z(yqh*^GLjizEYatH7N&;8E@P-q`Yov0yV#yKu`T#k#N;^k%X&FYKiPWNA^=GO{Vnx z_mY1+U5Xbqub!2BzAWKtKD{RU>6Zpn@J)cHt3lDzI@GD)DZU)ORo4GefZ4ZjCHg#! zL*!%E#cOFY9xHRr5~j)>c@CW)NSaWo^ztga|XDP{={|Q zR}FBZytz{ERz7D-xK-YMX&|e-uS>YKpPxy%Ro-73`Z4nVkpXU$H&f0Z8rK*j^R5Fw zD&fSBaW#8R!XL80`wjF)`@SaOR{rTJI#h_CDdLvDFOhIHuJXl330LDP_)!DBaop!5 z+&bwE=D%Hz?t1Tt&Y}!mZikpnuh#q7;E#%ecT4-N zvcT1P@3+9!de21K-CkSQ)ang%G3*fiPIRx%!=gY9;is-q%ssf#093>+J9mRM4ZL3EJQ-@jcd5?=!)L_I!=< z=kCU4aT~^?pRcHGrJu)Jy?W!Ck|uAnufAhVTSI3vhQG<(AGbkso4291qph}qGbRKA zdxB3<6|xv5T-g>-1h+aC%Wb#`n|TA<+iTnXzWUL>D;j8P@OF6F0{$)>Qdns5&0*$dyiapV4A>g{acO>I8;ZeGmCcJcjzy>iUsx)zVAa2k*Suc zr>inSohtJWg_O&*sxsQ=6HKNBQMgCmot1G+Aym9l^;x)~l*^+4^6K9uZl06*7=5-z z@lJ;0Xb$%G0Mhuz+?ratWbpM+#Zc7v>iVo~PkkFpr3&5+Mrdq&1tm1`WYuA1tDsQ) zDtc@EzsvEj*Zw05Lh-HoONa1Qn-FewKv9^lR@xo1{V4fm`)4c-to-|ck7qxJ^q(to jc(Nb4f@-gF=mQwoIyN;q*G}*F;c+QHu|ZGrvbO*Kmlzpv literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2a14e4b7e315b91abc4b37e7c8e5912a.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2a14e4b7e315b91abc4b37e7c8e5912a.o new file mode 100644 index 0000000000000000000000000000000000000000..fcd4b0ff7b5f64805a3ff0834b389afc730c5d8b GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY5sR!clAVT>AC*noLR2-z?b&$b>QBm)ux3BhJobW05q=+UPo znU*0ZLS!93Fqkp3vCqaCGVx?+88XgJW*p7tY~+m(aU9HGO*ZS3$vGi1!$^QVxZ`yo zG4B2TdUW4=)gZv0*f11y-&_CxSJnTj`v0nK-Tw0ORV#E&W2$KE6&9~4C}Y|E@p})y z^{{kMx>;ZCkgf6Oxo+G4p%>adGmEhUBevXucknuG%boWxxSql5UvZD7Qxq*-FYo+8 z__veTr4Kjg;SZ{9;mbC5;Q!g!z@aX70JI1DvRLu=ay^3|ma(^frLzOG()(&p*`^JQ z*rrfl2m3Nv@&DC5gQIAtYqqx>b|z-qXXYPCPyQP2F<|w9nvmVD7PR zM+S}_i9GE5PUOhmqmi)%$07p@jz^wab0Tv7sS}a0r{0S^x!~Q%{q1KWM~=T6Nt^v( zcx?OL;U}MddN_T-1L3)C--(=^e3lQNdh~o`VAXq|D-MVE?Hk^9bnoyJ z3ywspP8^N2cASU|?0qvbcK`Xv+3n{+vm{*U+&6r7!DGWutU*6deJ3)v9pfK=Gg5ix zd}P~uG~OSF!wWFpntkZ!Xk=ddwy|TU_a|ug)NmzWPb43UjNJ>EmV5Ch$zQD<^r<|5W&w@PW-5 zD}Eh5plO7gIxwO;9;SAO7%P4iusq&I^z}pd$p_VC;SV>I!tXqnF7JHpu+e`L`QryR zZJzQ;@;~K!L-4&<;G4I7@8(zYl^3ETIqh-_H z8oft9aQ2@{FS+e3Wdr(mQ2r@bsPR(!Uh3!7~}`(%BStDGjtG zm}{eZk_Jcri4Bh4qa8T+KlMxJ-eCtu|0XFs`Ym?gTocyR$!u`!N7DdH4v*f$4*csq zp#AHV@Tm>EAELPio)28!`KoCyiS=E`a}NBIETiC+#(bA#f{f=OqZ2ZbY^hV(rJvAT zg?vW2SSq{hfjnpbcKP5KX?X za=Z&UjPjEVW3MEK#~=@MF?e<|J8&L48G9)u%(T9Qaygji(p-$6s_q*&1HGPx{#=Nk!Bo%ScJ%*ET-V}Nh5J5S_u_gKum6qv z6S)2xjSqXh^K4Ss2D>W%v)#ciy#xKbX6s=W>~ju&OLqD-Y(O?Y3!9%RVuK^FAK5|q z^!w}BV5E!S)D@a1>m$M3;r{|5WT{ed00upf3c*s*1%&Vb;fUa31(Ef6?3Z)@$Ca`v;A=*o7u^f}%4`qW~Y0b%L$L%Yn|f|sk;5;&Splnc9By1IkiULWq%wYOc=`L(gDrK35-7It)Y z`xkaRvaox5YsZ$(c)h7d)*o3IyJ_ld4|u!T!ft=4d!Z1mv8kuK&D+)0=xuA;>g{gv z2OFEbUEPfm+wJe@>I??DgRMo2x3>G+y=|>qI|?7@ENuMu#bAKO!s5k^g+)b;g#`r# zMT;BpZ}F0X2MZrq+SphCIfY!efEZmZ-k`tv))6G=@A3y9fkd!U(BF1TcpK#K-V%Z| z`+YrI8@B|#?S5e$E5N(Z1i>_GqNO3|4Ycq{7YKHCcXsy%(426c9bMhtj_!h6z6S1vOv7cd+L8n1(#Z{I1l$5PRN|O@mP2ouDXeVE1-g=fWl%A$#j0JG$RlJX za`nsYG~{*~av_B7GF3Mu*SxY zM;c?8)yr2m3QS`GwrZ{3E*`yOcxs9K<(A6Ht`?c*0%^l;*;}J&C9FCJhOL-CSELXgrzC1zT11qy|>yh*z<7Fq{?n4=7 zHk@BrOB5wv0_mO#_})d=jP%`oeRG&2y1Q?VgWobn@cGtGWL@!W|2hDot^Fh&f>0H& zgLqMWO?_o~Wr@2C`+uCHh(d=XMEU?;TL={B4#5}8dYZj%I$daO^LDp(cDTDcJKMVE zyD_u<-Y&npuxR0u`R;nUEG%BeN_$${n%!N!UETh6cY9A)x4YTDrM1KFUSD0kYF%Uf z^4fKk)oU7;H>_X2rhIuhTTtyTaxd@(n_Apm?Kls3(Cfy~#4DYz;PVd)uiM#UfZG>2 z7+%kJs{Qg>wb8Ic?%)Z4AyP)lZcG35pG&+L^;$-MTCbP@Boh*H!Bv!Z4 z=!Uw^+@?C6RT=TrC)I7})^=Bv>8o(e)M(S}bJMv^$*xrFb1*)P;!1_BvT$1k-e#n; z0J?X<5o_?WyFdej;Xi-bX9_Q~<*aA+GzTN^?z7?k0SVuZqpfzlwgQ1d;q!iFi8)VaKG@cx_`s0DoJ5B)#> z1dEbA><-v-q?+BmlW3_xzVjLEE4_C_xj6`*?}UEzJL>D|>bSgbOCqMZJG#Qj~oF0Qms? zFDqz^l1E011b%t1#x4*8n(&X6K|O=2O#7Jg;|dC2xBVLWK>vEs#+-yFC6IkXv-$AR zA@DzvVNWN2#zGm`4E+F~70RDY>n8k*x6cp39HVvGG?H&PmsLD}5LhZ0|18npJNsfZ zs>SK&I!S*~Dc|y5EE6=br9YeafqxYT@4c7XOB3T22k)E9<)QW`GB7{qN7*T~r}14A*^X?=3JdR()9_wZENVUm# zW6>Ec0lgjcSlfvHu#@BMs0(}<@NVGUCcGPUfnN^%4B!g{epIxdfx5td75Hr6Jpw-` zgZH0}y1=JuyL(;0V-cnCTLj*Py1*ytLY~bMpDX#_Eb$uf#D75GeT(2<_ufnLasMa( z3fXMjmz2MS=FTN?;`saSi`8G_^M%`k&p)&$ec91J=jUykUXk-0qjfbJ-NWsl4g1Cl z=u1?c><$0Ge9q_dmFDv=GFa*Jnjk0t1-|0>pULK;KE6LQJ^VGF?8*HV_Ct9X8?slWc6S#f>)IxPC8`EAfQpRZY5-&sEV*XjgKXN6Kx2SY>TpVVI(hKu3tS)oa&8}T%Kjrb(gjre4wTSj~e z>T$efJ>%<}-6iA6g#KO4uwHZj2yCYP3H;*;nw(UWe_(!|LY3?b`y=1aCU+F^knBG_ zZh!c{d44Y9^xU2_C!o*b&$Q!zId10P4kV@bAPlsU>)I)7@yk5{PDvK_DqB$S@cvi zgZL}%2fb*YLirBzY1`0a;FYg$87|~!?pS}2KQlN}pAPz{3-fujIz2O(tKUWSX_!D5 zALHfapw9Q3=nwYICU*(Ye1N>gq3rBfJhSpc{P$zLEAvC^7XmsF2>&+w`+k~F#`%Et z$K>z8o8|-XCVvOsFdv|w{E*rI;qNoazC7OnzgZ!}e1N=!kNN+3jq)k@7YfD47x5)T zCd4m;yLL% z8QT$KJWo!h{#86r3W?doYB37(huysl4N&NO1pBZ3y;y2R(P@8<{joU2r2XL=+}}w% z(LkwO&r+92a=~9_P*0~5lV~5=f4{aOmM;kWPea@p7#|=0)enBOBlmp7JK^;t#`z(@ z>zFUQ95^aKv0=S9j`4R1djx#%Ph!8Mc>5YZ-&6XzZh5d@=j#Xg8(*JKn(S%K&rJCU z?6)__kf2EWSrfWMw24vL51Z<5@PnerXVX96TJB%&{E_b`l>K2(oIhqy12*cyd_VCi<}2h63Gno3!=+BamAYPjC z5eLSbARoyQ3qHl?arx2Bn~#wH{D;g(=9}Wz`29C!7btT7y*I?{1mXDy&hIOF zUY;;s9Wuo$!}&V==SJdHZ2yh*&Ai`UgwGcy+HddT;;8kB$7ecHqtA=-D4yKO=lirN zka8r$Wr|N&KTZ4VEfk-U`Tm^8Gvr5^yX-uF!hRrwy^j2KS5h*?^OPwW4$6m$Lv{++ zNj%@h`X|nd4sBqG;u+#a=D+j)+Z4Z$Pw7Ga=|k-E2g1kV8TijcJfr6zQxaftovKc z-)q=!PdCQ5;?QKB;R7tOKcxAQJ$VW&B=+CA*nj6trqAaL`|laFUo^x!Cv8aL`){Xn zg8erRRH)oYysN%}{dXo-OE4(*t7z{+J+>c3zBbu--UNJ>@jQzBKb!JRvH#@rAv?DJ z(CzM#cn>LNPQKx+M%p&O1r5p4K`)BGFeVXlzqwX z<5;C7!?M55(6QfVUjqU6XY4<(${#)thd+q(^&I;Alz$$n(+B3sR0qGSE&FZ0UyI4- zWF#LyUnC*L`f4W!;q6B=sek&oMfRt~&;JXqOa9abnHSnoxfDX?QYKQPG2iEtC%fJ8dN71joG*I)~O@H?f zKd+kfSIr-t{hR3iVm4>s$$nRzdthGP;ei3h2JLmvj=lfSjnS`hRNC64zZ}Tx9~eku zCx86P?|&D4+s>BPt(|pPUt5>7whp@yg!G@zp}1a1UmVpm`K3i(M>YQ79N;e!Knbc% zvZr0-5xRs2>Udql-z){?vG{CfQLhO)pKRS4q0Zu2pQBCU^lgS|~V;agmh*QZjCg8tVCe;)#!O z@l`NFebGy_@`h$Cj^C4J#Z$VYFLf@y$Ts8iBz#Xb4GChumJF6-B@sTRSXbiq zl*{wtPK-zLFA5k))IZPliNcc@FLEv5p_oMNWBhN9>t5L)5q(Ua0Ap)4Emeo6V=xXO zdr}cS8+4lDvG@b0;=|cEh|IX%40_j7gx+Iv(?|*MF}Yod@iDpGLheL;#N+}a zB&(7PL!uZYvwtd**~fbnS;EAbq=|6`>8{4v_&I3i+@@POPpETUa26XCofvH|bX`)> zk(Ox23O$vgjmE5_)c|VH<2XEVLpoTi>#Og@+;eje>Zi?Fzvi(4tBR2y0W0z8X?mU` z`FDJ!(8hTejjdJ`>xvQQ=}H<@{U%<4;_dh7R+(LfaTD1#=9!>NgjZIee#T2}79x*_eDiBd(JuBghKHT-Lc^gTUf_>9G#p@jS=I547E}Fk)D@FNk3n^+ z!)PM@piZ}Z=>r~YnCRcO)R$^zyvI_nvD9fi>YpzW=)XanXcLH!f7Lp9#PzoMr1T}} z<@W3CBT<`c;r%3EQ70PhT6&tO4=X-VpPwB!R_-RM8)ss_j6vMXgj|YxHNaMt&M&es)na5%-@R zH&%O2R5#AB+c8|MArtBBlj4m!za7KPtGI|2Dn5*lJG944e3E8a-F55gPw~RAlhd?| z$YRt^PDWfcuE|1wkhnx6$77Q%>FV6?NxHoc{lm=FM63Jt&i@;a2A{_0R&EpZdE*>f zd;de%O*1ul{Ar<;pG5SQ{g5di8rMFy=j3xRDYk;p@`GZfzuD?0{UliVtnwtHSDQOp zaY27m?=Lz3=Ci!d+u7ddUB9OJWZbcdSRapid&ce9Jn*1+%-+L{QBTdSp_ur>@ zqwk#=`d4JD;V2cfxj)s>q^i6Aq9~w=$h!+J_P?Iy`k7}5)}Jt z9qLNE4*|{(<~h}s-{$f@_|cGwXc>o(??F7Li@UXnUOcBO%LJp!CY(VQBjiuO88X|jG`=oX?a>BPz#ZauYg@ZC^=9d_7ZK;_qLP?eir_>Dbzy z_H&OVXn)w!UiEBkKT9Yi;86tfdR(^uK-Ot|`mTw!J^4Ml7{~vRoFZ;Rd%9Kal^yq^ zV}f}IX(*>GQ%sGo+9~@#g2q>Cujr^f(OKKmJSP=c+>|kLOq9Mnai#V|(xd`n-&ACKj|pgTGT>-^A^uA(ba;e6FF!MMoXCCw zI41tSj_fBrUbH*)$Ktx=ZgI`Qh2%F8q5hu$r14Qqs3{u-kUBONNgD&LNBcbq#{Y_J zpD!tCd@2_8F~4l@v5bFMwpad3?a8IB?U!KkKAB+ruWte|ikrVPW*uMoQ5D+vqrJy6 z{*Y|1{7;Rq+9~*503>Svc7paaC9Lh~$D$B*0x+{l{}*I?$$so&)&DxQ|E~$;SM4vz zmT*&Uf9v?<6QtA4K1pb~WXl27RkbJZjke=aT+UC+aIyKS=B;r&;#JKHYdclt z;~+Vo6rH-N`K7L^-PV$I^<5>)OGGJKUAnlfr>Us4ZDY~8x@{$G>%AMAdp0)I7B;nS zsM}t=w$ocuwz{mow56$iMfawL+P0>^y7g_9J61F;ZUJmv9pTH{maW~`P}SnwP+zdA zp=yus_MFRJ1SPRt$4VuB~ZJrZRO&+hbwnfHY~1f+tj|I(6>^wEp1!Y=4&V| z@NeS%u4-7kqQ}?1tN=U&e2X8b+g7?}TXREg*T#l5kFIK8Q`lTl*VDYRZ3o8a03Q|g z743V63)(8SR>j7x*t*7BvaEA?#n$zbreb49eS1Su{kChO zTUzzR~M*V)>hWiS-xRwjkmOH^@@r$g_|mt_iSu$>)BL+SJCR8O)IMc&F%HQ z&|l@+!um&>S5~!c+^`t-t}Wcyv97LVld#ve(yA@!yP&zDs>f;2y@gBsOMDOdiwcW<%L=`|rX{|j2mKE=E%g^KD_rbt6m(6kTf9yFEB(aE zdj8ug^s6fPwUZmE6kg@oT8yewg!r@x3?#rF7XZ3H`3D5uVSHh`ZUJUqrLok|S!4rMC0U!HSq*czBC(!^e zh?SP;+OC04b#J}Ch_-h2zB-aHq2+_)OfVsbQg$3gENJ4Z2LjkEE_HT-^T@*iDn3vp z(es|y^RCe-K}qzSX+h5;;%?3h>{feIk0=-z0)Rcm?cPZBol+|9P^X8fh}6 z=f9WyqvP#D%?uE5R>HxLv}Pe3Q|i@*P4=k@%?=c!7jZ2VAu!&=)2AF2Idj#$5)uF;zS) z;WREz6JzCwyrXfA@Ny|9^{;p%(p3^}?WaM)XIlDE`l5b}r$l{*{^>aJj)Ys~9g+sJ%KN&6Tl@K$gj?nPrJ)}q{~sCPMtQU4{GoA;F|yza z@PiUg{1{iW=Op}o3w+2xZ?x}g5^m+6o}xpA_?a$l`TG(HSK}&QY>{v^u7V#l&>P2n zPQtC@{!+rN*myq zeZc@X%GDy_N)L*@O~S46z9`{VdCO(UQR5cKaaS4O#&JUuuEtgLyCmE??)N0zIxgic zR48|$DW!BragBFeqC6I9Tnp$XdTE%!gt+T=IEh4%a{ z^5^a*W^oI~qo1#6?4X~=TeYfwb$P3|&EM3$y0f{b4Z~mK?vLA`t<&4w*xlLK%o!5` zfjz;es0vw(60YbBDuU}Bi{&?8gU!6btzC^>0e{o@-xUpZHha6hY$1Oa4k@g(3X+s_ zlptBy)YV=1z%^^*CfN&LyKxi#VCs!*;7x5l`EFj!$1e5TtvBf}yNU(&6Ta_35Rs{t zsi%uFLA^`n9||d#X;o#k&nK8n3u17OygMo5m_n#{rRw+LhEgt%0?2D%hq!r8=415P z8pS&qj$=94-vdbF8*^)F>5{?MLlr|&q_5Uu%zf${;EC|K7>Ms+*TWvzPRY65zzFKLw$@b&qm+haiG_dmT2R@Pg9MXR- l$l=L;ClMq5e0y%*I;T<4>fQTm{kbnX)Ag^niHzblk{wG0BG&3F# z)~aJHC)ifGsiTRC=l}h!$KGr2 zoFr)N%x$kr!#=-So}we~*y+_T}9jZTMysp4Q?XC|a-87s>*YXiJGz>?9@ z&%(8XuDYM(`(6K&{?MwgfU)D3UHPY9#@{Qh`~|62aTINJUxo5D)JnjaW=Yd!n%OHOvYAURxn$(42bvMZBl z{VB^}r9X8B28Phj>2UqH4r9+xe&pimhabAQ5Z7OQ=An!K-5D4~+dljqZ+q>j;nP1j zGkoCnr-q-zb^ewg4d3zl4~G}x+K%g{W4|8W{M41yy=A>4!`!x9}M?D{OjS1qq;qaBe`}Oc^2Y!V%CH)UR^3cW8iyyw&kFj6-{*%LP81uC)FAQ(OSf>|L zn;ZJCJo(VYhuR;xIRA%dh9CNqr-mO){qFFBv=@f2eCGAx1wW***7VN@egXd4wmu2| z|897G`wPQ&Jo&I{LmxRW_ACS&pdqbG}=74 z#Va>stR-I@zLN5z;mw}!f}dXvzqasIw7Ic=ljqZ*^WeoRpE*6eAo;1`{vZ4he7%7A z{d#!8mN(F5UH_(Ie{!+^;nN&P^$CvrkxfrMSUdk$e-})+H~(CMCtREGo%|5#HQ-{U zUg*r>4A(Zo#-=&~qnBJ$PZO@*$x01-08ZG$RM?xaiFyYsJrA33I0!fO^kt{_X6kp6 zvC{9LZWiw&Xgv+Pd2`$P{(BM$FJ;~ z?|x^$tAC`<4LgoQ9QE|y^ozSr{06pt>e=!EZ@@p3x!q~~|4=?~?jOPf-#ic)o$qF8 z)7a74H(j37|KW1Oci8D8Gd-hUyCE=IKizY!Ez5JQ^?BF8sh0wyZ3kTg{XY$iw*Jg9 zI&{yyC)&Q^814T;`2edPa-aGU@H<@tt&clKPkpw0pmhoQd>8Tdz+HgfcMhETI^eqiHvxVX@RtD}1bj2#p8);>;C+B^0Q@}QzXbeAz_S5= z4e&9*I{{At{6)Z@2D}xp3-AfR_W<4q_&0%p*82e;0lW_I%YcJ`{|?uExNb-N64d`C z;99^p1D*@`Gk{kDUJp16@ZA;Q+cf}wMqA$jyb9O1a9x4xFs@5+eI3_@xW0z#JX~MJ zbvCa5ifcZuKfyH@*NeDL!}SGRr{MZMT+?uU9@ivXpTRXA*KgzM!u1=tM#0BZxW0?) z7jb%X_Bb5qy4#KF6PF_;Y+dL7zQm z8a|KDe?gz)1V1_96FqOaN84U^gBSQBml6E#k0G;9JpHNqVYigi9Pl&`Jk0}7^T5+Q zi=19YKBdda(B)2yS4-o$&oq1zpZC&d*O`WW_}oaJoo5;j;`1H!*&%d^H9Mj2Q?)lk z_TR%_+e7Yay(iqG%rSUl8#@_nz3e-FV0ps89c$bJcU9o;gnOXv5%=hWXxsYdm`lVx zP+t+bF&y}oyMCkdT3rJB{~8$EaRvEf0&>T;NaesyvTi>5xT%iD0nJUgCMJYyiwaKv zWs-vpt(7T@dPdEph8^T+cM7jJUFPMtMpHZeBpk7H*VoZHU$;~cnd#T9dMZo$=c z#@hatGn?G1{mrudX4!sIjP})MHhES1D%pOcY`-B!`^qz$l2!ZlvVDbYUoP9T;|t2! zkL$qyft3{ln|*9_A@ZgJS>>Y(kxN}cyf^T*z-LPQ+!*+Mz-LSRlCk(QNB%jYw>giE z8Z>8?k8bv(kA19Js4t9BzY}8=@p?|npc6#>B02WTv11>&ANXR4Ul#*^0{9Y%-xve` zMc~UMerpW;*MMIu@jGMS?*u*oyp(~#_w%T)l>I&#qu)<}ua@|IG4Ov5{4Fu)zFIMm z=K*;O~|A6EW~Rsegv(|9A}iLFzB@Ux|T# zp88J_{eKn%e`on<;=RDz?IwbHN9Nf2N2q;_`oEy|Q^&UdJhhKe|3zw_HMadz)ILW2 zH>mxzvF*Q2?PJtGL+!K2wtt@5$Eg1vwa*#b{sn3uqy8ebpFXzzPpEy2`hTVNxntYE zO6_CRzeerz#iTCn)OI%2VebAn--dT9O z$dUg|2ljll7Cnu1?agf)k=t%Sj$7IP&i?hteJlE}-Cf>)W%hU5_5E7(s0+EV7x)CU zOGNu5^h-ugoYH>U}#>#{3|~Sj1D1pd;i}8qd$K+Fnam>ZH%fM#< z{0BxykAmhsfq{4SZ|)y?mkqpkXcJ^o-9PjtSO3t8ih(Pxih;MEW}_o@H}}7Fl#TvJ zmt)|qUAOeVndBJw&DYuJo7=YZzp>tlaU2-aIq<7A2l#LfycX{m9sUgQ=olDUg*EiE zfl;i516OkaPh+D)$Rmg5q5jK|LmJ?J1pJS{r2+mj;CYbqLTU^6vw&9tUf~?L@`d&N zSEgYP@gp{RW%j1|S3Zk2FC!m1>Y}qAelFx?*qdM-Pr&+3dVPP})BRM3z0B!J&z2m* zURpi4qW}BbzGK3yk_e`)97ez}cfSM;ztl^SucH?3-qc z6eXZ8e(n5I>+{bQq0K2*-bgjS&KTKLP(Hu^*}QY7p3ObC3GG{{{!456ThZsA@ONtc z^m7kT-O-$p2RNK_?i7c!M^14#``mdBPa7G+yiUzqGT&QR*`J8IBC21wvfo>{q2CMK zLV};2dd^$8k>kC*eod+J{2(xT`e??8 z-x2s$0_v%6|Izf3Q|r?~ubR_4ebhIShL$$k{PS)7X}B)opLh1B;JS!^-qoLkE6K5S?rxgr=zRQL`hITzW={a zH^4UmKEPnhHRp!D&tRvxK8x$onvtP8)UUtw+|WM|eCx>2E`n>%4Sj*&+L57c1aCiw zxB__l$WRr*JI)OqC3wfk&`p3{w~Y)TzMp#bwsRO~;FN3U$j}PFM{!+B_+96QwxfPM zt~XP^T_Z!YQNMooIrIa(dt@k=;63Mt781N?WM~S(d(WXB@ZOQ3B%<@*Muy^XefGbd z8*+&~aO*`87vSqSSKI*Kuf!U(9`T?8aiJXX`FTrB`Z!*E94|hO7azxq$>Ig_BVOw9 zFY+M%NB`9P`nGR4n2()*nWLGLIUIBYrAES3Ta7<28Xv$lfWLVBr86Iq8B-3l1P`{f zcd;5^3JL2(Cgx1W)#+re6qm#4axixi(sWr|sSIZ@>>hC+v0NdX?AwAz*{tlc_;Bvu z#MOC#c>=7A#et*g)N)BzYgc!$yP*-E)U~HW!2DX*)q1cw#FiX9)ZMb=;GIjl540WJ zcgTdB4ompXCB}!QLmizB-E2vBOQ?H^m|9)a;qLZ^uCBU{6?GkYBTII$N4J5lO7Ep{u2?xvL$BwuAd`YdhFo zR@dFpP`B?;u%n^dpi|k<-LR2YRqWZZp>BCyknWW zU9AnlmgYa4i3D4^T7q|CM$l5QrTq`W+AxoXKZuDmw=^E!U$-yV(9t5aW3OmfVyS_f zH8r%i?+-R~wsNQI3?Aw}L>|wfrb7q2x*HC57ykj6Z&O2OM@z8j4?@jQ8=s3^m%LJj zPOd<`;txQFYC2Y%<+Xn>F06FRCp4ad<)Wg;V%4upx%0DH63nlsO#!jRvfD9Zra~+u(-5t|KWyUbFiVUy{;Q;U0u_m_PWLv zWUqD2Em;2!HMKO?A=R!6wsarHmY}(=v8RsLGoARd>mz;`jDTcAH|SuX!yN~^?1O!r zI{ja_F*PlB5{_nli!m_=irDMsVefzpvx{Fy|-N8HIAtL?XzGcIfI)Q=H zHMF-ibj4_(+^@4C*xlCMcIY6_N(>rA9eLRwRIWMEiaQ@j7fu1}Wi_w&*sIwmczw#4 z`oDJ)owPCa-$lKXrLYV42vo6bQRou&EG#SJX-G0yaq$xkSuZbZd3~a_63}kcXZSRJ z?iEe4zX>I9xi9>0uIZz?%Y(l zsocN*P_PrWBnp`eoujS9-#)79><+=-*JGXa)2UNidjmPQzx&Xk_O3;KM9&tS5&4&` zSW>dczmqOYN>{Ur!)@)&{;r;`?v@UJ$KkGSe{;*ewu3GH9ox3uvb}EShT83$wr#E3 zuzSabt(6-p+2U>f75>Ey!KPM!S4Z8l;#CcP3{AY!xf||5*mym_?nk|Qsh8o|DMk-L zyVJc4ro#(;R-Q+nsSOo(hWYt=j{n^I+^|LTTjdtxO(TvNTh88ev8(CKE#YPCss%14 zj;|IN_@(TU1zy5lw7`qm1q-~0owvXX*`Nhpz|L9V`RuF(E@G!F@H}?X0?%civ%qs$ zp9L;t4_M&YENp>iv5*BWV4W7&&sr^TKC8FDGudtnJcHF(;5=4sfpb~F0#9effXfZ} z<}kkn&SpLfJdHw%);^0}b?Wd`cF6)~vI`b?3Of(j)OQ9uXThhlQx@3APFmWhu|5kv zm4z*E3hT7M$*kT2C$Sm}oXDyHoBSlO0ASPJysXRu$FpKfdk^yiHsu$`d=?n7*aEv) z*ukzkoXXx2D=e^s0*!{}PD1cDH}^?LITV0{cgL-`=0=(G796?Pxg73g$*iUa560kc z5P$CZc|0CcOpI0BMR}w-765~IzLACTcl8Kk^|&$6cNdu69A@s-ovsTAVo_q^IpQ?H zE=jmp!dE5il`v+@+m}fg(%`U9!jKk+0}_TbIh-xwfP||hTq9w>gzF_-BjHX77fCoQ z;oTDMlW@I+PfEB|!e=GiDdEmk!4J(Z(UHnJkuK2uX+8wkV>u<5_$Rm~UBJXI!T5d$ z#}j`97f%r|@k8+0Tmdr)*Uu2JSHdT;b`pJ`gfA2bnB+tK>4pp8Ngka3TmjSk3GOQr zu%Ms8eE0?wwHNd=Sl>bc3;Mt>7OrT&6`1`96}aJ?LFaHD`*0edih zV6%WbB|HdQqW;CQJ-b`LN}d4;2P8f$;c5vFO1MV#?-l)(JOdII^F=g}u$XTu8F7XIkVgfHT+CKrE@b0_}#@aKb%o`hW8KCGeeKm5=C+%w_K zD6X6lmpjqRFk7rP_`FiqAHZ$84*cz>9^eawuJ6n1HFeC58%?6 z$|9ivge$(B92hq*VHbNXjXin99c68(NJRTt#6R89l?DAHAq2B(4eo`%3Xszu=uxwE z15Rg;h7#Rj;MIH{4Q0BCpb2~2VGZsJp&#}e(Vt`f=V^X%?j-6j`F||rb`w7e#&~kG zMC0LMydDN?r((b!9e%z`K>S|>>aQG*2HfPB4$QaFFJMvthnaxAHkbmBjwi>};eZV$ zN7nJxHkce+$Jf~4A{)Hh29x9K{abBtr-Yjsd%7oz1qm=&@a3hfYJRsAh|vGi5O-(P z0fxS_kp~#%McB)Ow@6U5kAytt?hy7E3AvR$aQ|4=9T(;LqJKcoiBy$G4BA4!k_P*a zt04E__7DGf`LM`Qz@3*BrZOkYleiN0A5omK=*_^|Eeg1RzU^>T&Pl#6RpVRa z`LAhg%Y&-F;O~i0S_SPIqfW$I@b~1AYBS!=MeY--Oe57Rc|>|s*aks@^Bd`nv%&Ee z__0uBr5yjtxIOQ78N2&&)LB2Yp0(^G!Vbo#e$A z7(RdE*PqESVUwQ%vpv`6tV~#*hM#T27iLoMF#FH3;Ay-pnmr#+=nLUR!f{+)3_1K=p$T6dP%(hOam6<`3+du!z=|Oo>};NMc9YQUjoM$ zs`i%!eN%ppG4Uv3%O@fsXupC8punE`LgfKkt_^vG=4kjdXQVfi=d5Tg{k6~vht_gZ z{2$V9=tg|JiX9gU;3q;A6}o*!de>Y0^@-4wDcH`iG-6R&v zg;RLEGUCkw3;jq4rf zQpq<_hP)O9C<9RKBfU#)@+u8bI;ik8|4d_iiU-93I{s4lzR-#(aDM}i^e$uAu2Yfz zvD`A>zR=3bFr`BVKP9j)ou5?}erW#L#`y6x|GE6&!I=Nt$}nvX4f)Qq^zRFm1=wkw z-y5bNJZb!{W^#um`CVh-m*$^mjGuu0LRvJR`pf+S@#i=2NyJ|fBZV0J6$RKA&G93> z^ZC|{w6tPNRjlY{UQS&$J-YhFqh9}eh{LyFEQ{5ynUGg zgWtn2w_@;{4t*gXrhJUjIT$?FTjbmJc-gyF<-?HI8?HzQ`%T?`Lwq&i6e!+BdVLmq zj)ZVJtHr}eZw0Fqt)={}@i5YRqb0ueg>ITc`GcwNb*8?4kVf*y>6a;gl2?FOBL96w z$ak%RNxn4yGCBVzX#OvHX?rT>zisUN$IO?nX1oWVZ-}hJjQ3;a-ooXA(;?UXeXz8?c;jK%eiElgu~8reBCx6~4aFd4oRR zWUIb=@yZU-Ue4DVpY8FN;xpxgrhc_}Easc6`#(^ke-)cGEX3}q1n=i$G z>-j+gXP(W98Swc+9+vsKZS$Q=%R3cm&(?Tk@qf_k3oV?IfX3#0we>;F*R97B znr|Vo#OLc0@j`?7d~0<5?$G8N%bxA?mE6+|_Ez;(DEQWru8L;SAS|?3`&zDvQeZDKz z0g#-Zb$#kXzQ)gw4F62m))R`KK72cX7ccuB_t(@h@tIhl`C-3J{_0aO^4~4Sewg-` zhxqv#mb7 zeSiDp5s$f_rSrR3{!>OAqJSQqH`YTvroPiG`D~=uYEMsuQZ4!M6QLx_emc^dXzQPV zry_Xa`r+pdpVQ@$ks1ct96#OCKGHiyJ#8WTX9n0)Cck~5X?O*P-hZm4KjN>O(FCQv zJ{od+kY360@^L=t@jyM=_*oWv!u~11&g%S5Pc{3eLB2Vb@o2vZ3#KCZU7AGZqUP5Z zLOzTeMkbt}M9e9e_M^JIB8boAk0_F!KfuoG{LD(V@?T)I)SH*jIFZ z<|SG4;W;*Xb8Y+-@f&(3|MP9_7h1-DEVN*$em(+!a`OltR`Wyt`&E;lNbh1By(PBs zk?)R}@}PKTmEQ{6_{&$>#`l=#8{`iq0roY0{8Ae|YrH|elSJW8&7bG9KE3~HTmKs@ z{LuXQip}TmK>oVsCY`^&(Aogq-d6M}-?Gkcokbp?=P~!!JRZ|=Bx9U@JmxngiPPut zkoMEQ0Q-hcKVuBNJfpt@?csk#sy(e&(=79OERLbudI}9Z z@t4oP^d;s?{J~$oDcj?G=&{iBd|f_7zsTrM{J}qVEBf0w{Y3_z#-EpDuII-9!~Vam z)1NiZPQTdbPyE5&I~DzWPQS#!)A&o1$X^Bj+W^C!2X*>m?VqndhW*2yA5ip#d}uo$ z^hfr&D#={WNItOFXLR~w?AvYH17E*9roC}kvk!qc^)2vzZG5Y~1m2Xt=x@qf;7$1n zydf{-`_}an=gsq6;zTVEtC3zVB97iyd zelR`6yZ0X9=hu`E>*M48T82~W|7a*HgXE#ngL#_$PlV!U0UPy!E&At|NHkI z=5dEXkk8M-;fx2YaAuzs&ib4Mjwk=XJ-!4wf4(rWInh7`5TsZJgkEJ3;iDpC31ZwN_{;R zO5$)5!Pvi6J-C~51BZ0c{RrqIQP)pjC@z=mThTue%1FAT>5tAcb$y+n`=J(nWVy+{ zO#M!?wa`Yl)aHtFj%+MjFtL)`!3 z{!Ry;#(BwF&3?w-Z?n^?we!g|X0aFC4|pt;!tEo)u$OcWr;l$hDRz6I z_^;#(eWdGnI*+m1OP0-EvTgQa*MFYPUQ#uCLH-(=VWa=s+Dm32V6&I3K}=AK7un|} zEbS%lf)#eZXn|90_LBMG?ZtLprtBqKx0gZh9zMJ{U$gJO#@Y+=p+QrBbY8EYCyuq3 zOl}{UhP_PX@YM0`B~!NtiZ7H8X!bJI8ei=C&$ii1p3PqD`ky(8z2twy_L3E#;~zC2 z9zQN|1sL%qZ%D$@US?jk!tN0ZY`2%JkKA7Ty1i6>Kzqpw@OUfNAM9T(=k;UlC5zif zmSHc`I6Q59d&#of3kEdgi}*54$B*^@Je$4DwAqVY|NcqrrQjpBm+Sx?pXv6JhxH4^ zu$P(Qi35I~O!nmWNxZZlceVw#+e`LGZZAc;y`1;nM%>Et`^n=A?U%4V8}^c8xnDBY zUb4A;WE=L9!{MCq?Il~c2Z}G1;7*MPd&#lJ7rXxHfk88WI1%#O?8UDC!b$99&PQx7 zd4b(FdztCC+Ka!)Vh^6eVk_(x=Vi*C?DmrPTiFZpEj}8V1oBTyKEwCBQ>NsTKIHu@ zy5EeQ3l%RD>lc}N$ye+bX0|F=%D+>>uL}9QLkiE~1ato=z)(`D^($ zoyWxG)>!19sO8%vzjRytq0@d-9%JRN=Tpcp$K=y`K1KSp%ikVPCzdzP11$UZ4-(Qr^4I+>4deYM$p560w~&A4DFqAp=bg34+v7fGfyc_<$VW;3KD+$2d{D@L zw^jaHKKR?pe`yT)XW7|AuF7DRmoT2-6IxQlea^h@6^kDhd(dStI2!mr00W_FL!1_0d#(Z=fkTW^vN#_ z(0LM_=ZN!TFMXeaKW~Wr8RDN>uhn@^GRLQ3J;8ooyB~@8o}lN`kA>uZ&1j#VTW`^S zincyc{K-OoW3D%kh2;5w><{{yzf${jJm8_{L({DJP)KHr7?bChA-X&Hu$MrB5npHN z{ZE9laeK)e|3s)zi_a8at>+7P9>ntiNGbWy`M6O)pUx9~?BOC_2N?H{Z0DnSDc18& zYdpsN6HlLoK4pWlaQbn~&mN|fRL*B+ifR8ezIA==3oZ714oyvdMK+=O(|TIW9&S}I z_A@DT!`M7OwdYHPg}S^VcplXo;JrCNc)rhb5}+Cn{t4^T^wm2RN0q$5-bf#FE&7k( zc~L!|L;jSW#hZ)s^An+>1YLe7LJJb~`M-7#i_cE=F^?1scvzq<4kMAM$7PUo;bUEJ2U=Cwn#JL-}oj<^FagG~dEMKVQZArsRR=6+LGy z`zqVJD}f%7W&G66AXV_<~gU)C;35PC?r1@_FF4$yk7;4U#>oZnmgANONDgWyicht4bKX#6at z`7g^#Tq*nW`<sNbT0Q(b* zzrh}~_(A;774P@t`z;!8rp2GQJz)J+_Apj|1)dAWeE9w$ox-=ee-P;{ve2jJw{(A^ z`1GRIKhnFv)_$R-y^8NCnm+aO56It_m0Bp}@8a>yJde#M{rIyp@QfMv4+rLJ)i3Y` zdAMUO@NtmOV$1lz&z_mMK;r3sv(IzM;D0IT>GI|ALem%J%Q5t&Jx?)K-@5*`+4OS^ z4WqYLm%4nC8S?0FNQqE>R0zq(vPk$;$QbCsW0SDO1^e~noK`q^-<`_*7%F`POgu! z{$lEb^joCSv&*0KQ>p2P{KeFtgY-Gu)K47OuT>v&JjE9Jb0^n__IwTL!*A(t_cvW1 zB!68W3Ot|F_`4o`Oza=C zJ*PB!vGlRSstzDXZ?ens-t?V0qW z=UcQM6?jH8`c`?_^JDw|k?#+%pOg71zRv-BHsghW&G?P?DNsIY+NW#G{YA_DM$Fg# zyqe{He~kAN;CaaD#{FbGuV8;Zmfqh&Ct#h8_tn+{;QTh@YCP{5^ zydOf;EaLmY*Lw=(1L1}I{WA9PtT3Jje(*^_PCgHu(?g@9Jb*uZza=BYO8Nb9;s^3} z_Li~$;pu%L=lT0OTz-O`z~g-~g(19a5nw67XkQvCE$xIj7`@K}?Mv}~lCXk-&kB{6 z?C_vH(EvUxRKj7-1&tS6C2(}CdO6MO49f4T8t@G!9`6$=4XrHk5iJ@I@8>BEm6iA{ z@S2ih4R!`^H2Z^oX=p7T_@*NIf9sD}3CTB_jIS-?c}!>UCcQo8v#z8@wTJ(r{YtND zkN1@TE=xAxID*#@f1DnEbPDiV1YAi!9`xfh`uX6;T~ZIc;>Q_`*YL!zr=-(@Ptfp7 zf%le#E%+o2PyG{1`ZPTFS(go1%$N8{(b{jre3MH~YVAv*ud^mh^AXP{ki5kFJbvJ{ z`FabjaAJ`aPF`e%xxN%XRXv@I@cj8;+)s${+w)#8tK`^Ni1DiH1r ze{=S_OnZkt!XKj*9RC*g$DAJ8y`dD~599IghaJFYdcPCypD2HDNq@-0`<|$Huj2hs zphx48cX0p4^C4U?#KD!Gl@@qa{3;8)Hf}YC8QLFX1wmI-&_gBfe*8R@=)>tO!6pI@{q704>!G!s32HYCgmadTkX>4 z3x3wv_*tv-AGzBVW$myqGK0rcrM@#@kj~d8$JB#8+nV0Mh{=nlQ;z@#t2P&SfNfhx; z!1wh8&|k(M>EGlZQeN(Ypqu*#QH1}w3W9Om{v?d?5zYE!e3-@GFX%$Lg#KncpzqUl z8u1_=FRbDP{NdTXpeJF20h4}86Dv3zMm#tM{kYxKpBMNq#*ZWV3ibx$6E_M-#%E7d z#&aG&u%0B8b3Dhb9D68w2({HFE}Z_t|{G=>!E3n3Ep~)X4<2d zS>szmf@zNw?-CNo9=)bL;(oJ=Z}~jFm9lzij|CIQH@tsta(?5Zq@Ujyy@VG`D~F`wdZj{q+WKO1g>TVJ6&uy*C*4i~UeR z(CZcb(TKx0d1ZU)f2AqHo{2x~=k)cO+q2hPuem)ZP9pD{qD0@ckEFzUf{pl^XtU>} zB&$6qC6PTR8uskv-@o_r^_t5Y{GT%Hna9_YGCnp^d|H{R<)dzIFdX${G;`1Eq`4eH<*|~@eD=6N&Njk3N8qi8o%#C@ueWR zGHUGCIlL+=^EVzZ=)6E*&(Ocr8oySq)Z-)cy-JR+`j_bO?YEHQM@1 z@+d78^e`yoVc#zkKlDVYS}!2KmHK%D;pym8^A~!6p9;pK^H-lGULpU_kn1Ps2j>BN ze=ge7dJxawC!%2D&+r%Umk~<$)dR`nnb6-VNl(guFH>e@vT3`d0ntgx2u$k2T^v5&2MQrs=Q8*t!aBe++#T1e0-eDhu?1 z^?$1F&+8h+c^!{eT)!z%!7uvo_6b>5e@~ny{f+y7K`<3BJU8hV1k>%|=Nkz=26$DLc;cF1lBX|Ck3ZhvlxZ}c z=ui3t_ZL+BNk#i~%X|xh87ap8m(wdt)z1gL!AyE#1r>~!4|-^QLG35VKBi8U{>J&i z`2+n#nQTw`oih>06+41L39!G@d^GFoRKk=lM^tv&kM>vY5vyp&vg~) zQIeObzf9jWt^c~(j3~*E<6)0kAz!-hPCQ*oZD9ZF8nyF5@I(5_z!w7e`UaTvMf`Am zAs;1s^0BZ5pULq!FC{$er3`Nz(fem}ygD!SW()fw{~~=S}X3-ZKMS;F}{Jdn^E1hrY>k0UuO8Vf1+g}PjKw!Y!e=eE+v#uhQUf8VJ zmmBAEX8U!u*-^4LWq;gXE=haEc^md)^izPMU_7jMxw?N~eVnfQ&wV`@?UMADS!C~S zw-L{z{iWgsak_nFfx23c$DXHo@I1CTKG}CR9{AP!6aU=4xjzVb@%iBVlb>nh^-@0IZOSnskz@p0r|e2zq~)|&5=|HR{kZfbnw1N`-&APIZONB?9G z_2)Lr<&{1|>R;N6H_zrjGi?4d)9OE-e9azV-z9m{pI~3yANc->ONft`F;jlSg~o%v zrsPY1k#HtnFi(ZsC;TDKtLvBSOQaYdixj~A#)c{<2G;siakz<7v{ZmYdd z@mT#eWhUv{^pBD_IUm_S&f@gIP1Ae@cJLh?Dc@^t?e|KG}yF z5BbFm5ua)Pv_8PQuIm1lnW6g=ji1ZM=jX+k-_$8$y+fs#Z@M)&LnZd%ln*zb}IxTwmGw(m(lpcs#hOuOFG2Vm@du{iy^mtkdm#I#5bp;MWo_*41X3 z@c{O|u5oIVr}ue@rj1`bX|8-9LD|VgEzipKPS{J$tt1 zFM#v>GJbOZ^af`XNPFl0K>m#17u4)ICrb9l3)k=TD9O`=bE9Mr3g-66UJ~O!PVq7? zhvJLEBY!P0*N0=^r)0K_pEN%FYx?vVTKj@vPE?QY;1{PpRLEY%{Yz@E^y>}IoF(HU zpC8fF?U&m(8x{P464tu{>7PUo{-2X$^}p;~t}7an+heZZTz_eO_4}p$bNhsU&DPdi ztnca5b^qt~&+rO+B66AJlbff@5AkWXj1Qb1^1~e6zt+{-^P}8cKk-I|>nA(gDnFkk zAIl2O;Q117^Bmc49@n3;UvDsXmW&tt{xsS147|`%*k?TCl}TSXLZN(s@|~%3Wc=Xv zL-t+(tjveqWKVhc!5YyY(+1wp@pM0$;-m0CR2ls1@_@f`e^hwL2M;3gg7LOteD53? zpZWNRe{;zn1%Kqv+1!5k{4n2w;FL@~{vv;xs^=fb-{#7Ei_=3s!u^|*1i$33+4y}+ z<*ztjDir=i_DJ`~rp^)m#O)K~#nBf;MM3}jdSJ~y=^xzwJadJ;s`i=u{r_sbbro*T zx~ZQT{B_m9W4vP`Kk`X`$MXnT`28-5FMJMMzR6ypAM^kWe&@;fFXxw;soRS;IA@OZ z7jA!~zeF~q%{OJ9%+Gj#(&ucDR{poHHqVTwun!&&e9GQw{8apE4=>Wb?fIPbej$%% zKIyNG;(3Nyd_0wZ$Mg4pG9$lqyLEpj`$hb5DSZ*Yg}lF7zvANb{U-9|c``p!{Mqhj zCMVAk{PXh(OTNVAt>t%EFV;1R{W8}dUq5E>e2UAQ@U>cghW=z<$t>I87x`3?jDL_f zzyCU4#$zsjM+| zW@pItg5ycQIrzO>H9y3!g))9f`J@!dcm;hfMZC_KFXI*0FJBM*`g%FFNX7%He{a+p zFMQ^&dh_{ZrtA3?kMDkS{qqLr&zJr$>G^zCe@s{);}zlgdBa>T4`H7;A6+2h5$})v z_(Cmzb5p)T`cv@%_fN8|@djAG@;_Q{yk0%N;(pLV>CckC1o{1a5S8{g6~8~M6@#T1&z9s+S{*jwN@r(1w;psg8;x%l1ty@i~ficdLb-18ZJ-k ze#5^1g}pBl`4wM3X#WzY=ihXHO^=7PpVad?;zz`P-XH#vvP9%N92W7rSo#NFuW}a2 zd<6D#jN@Xt*a34|5Wy4KhIoOn`-WVpCfzona@MP{&>77mi{!0@}ab)a=qaCLw=P)_N(H1 zRwzA{V9o>PPxj)&@0|<#O9Fq%$;NpA@Z?{`7Wny^5d7$->@I z3I566)A4)hYCLMcl($#wUvjb@-;hrgi}*+Joz|O#Wg?!a_@A^w_^*N!my7r$V4}a6 z(|=1oPnc3H;x7X|@+Y=T#y`N^epbkM!R3eeVA;PTp6UB<$SWzT#}CMVxrm=cU&wC} zA74BVTM)FLpSg;~dP?@a4fMQTtA7|jcQ4nUq$J(GyuqAh!heK+Kt9XG`m5m76~g`n zjPVn7{o;95uM7O~0)CN?70dk?#b2Di$1RiVACJd~Kg;F%Ct>djx&CpuAjp{R&yZK5 z9{(XP3;*=ItQ=3yj}B@~e&8>7KZFi@`TN~@4ftI*OUq}-KbE_qG#~Cy@Q-yB;&pV& zzwVz;`Ha`C?9Xz3MC}{3``w^_|7y0z#r&fEP!i8y@Jrk#414yl8WV;;d0DG~UrCa` zH%U(*;F(fk590e5^!pz4{R=}l^7&Kz9%+D24*#J2eU|r^r#hW@UpKvPoRJb=Ifmcw z^i+`iaQ}+GpTqL~2fV-hz8+W9Lk3UpKSzAS^WeNczu$%DPjf;{#CQ5VV4OGP1f5*` zJjl@fCFsW~=;L|RrS$v2;(6U#@%;6&ZkPG{tT@lh0Upii{uZO(E3Qhooj?>ZKh6&s zJn@6?j}U#ra(=#-&*>A~N%SF~s)PW6D55{UA42S)zzaw}^m{&39FK7S&gqe8s+&dh zrT$ll`%&M*_YXvmRDM70xh|PqQTz5(c0!)#MprrThjCNoFcVHe8KbdT%i2-P!YeJy-smz$Jz(oBkC+}Y^ z`#0OizX{`aPTs#}qVfM~^8WNfQIg*)_VJ?_|NP|rne1O^?~mu>dM>fQnoNG`eUmZe zU&2mK*55bL_+Op8zwy4mSo{~WCng);KhgM4PTt>m-&(Bki`c30#)tn~-cK3x{i)XP z-&FNH`N8Qa{S?BVE#K#fI;wg`J~%x?U$N*_vL`?2d=34@qSyDq=^6TrMel!oaC(M* zW6>*OXUChb`hH`*w6B=)q1)2`_8vO;qJk_L@t4*^g7~5cd#UMRPI?|98hw_>LwkGr zJ#$_tp7Zu^I(xPDRXy*q_c)ArI%s#_PT$${co?`b1C}9mmKdu{ciN7zj^6z@;z6hSMM3Q z>An2-)1ohbA$ku|;A_`zJo1U39%i46E_9!dZc5ql{=Cwf(ZBiUz%TBpb^RGCzWRQG z>)R(>->#Mauj_Y=zVyNy-`d&v$49!3b>DN_bMNlSdHsgv|LaKexi`*tzI6QO-=6*K zk-g8n9o*r%^4fp@^z8CKWAA?TS~U9hJ<)f5`t%zwA?<$Y>G%KRjp(~?_AeFwyAK|Av2}FI{>& zy8ni!|2HppzwxajQRYZ;CywylLpk3>XmE-H$5@UePvScqf3I(m1ul2HIh+bSoo2Zl zc!-Xbr_M}tCMTx8!|%2qbvj0z4kiz$!_vS^yem$}lTxUd_@fRTAC~wZI~=(yG3s`- z#TWUVgHak){oU$t=6=%i4l5_7e2EO>>jm4YiSTr+G!dS*#S`J_SZN|WZG$Jm2PVPO zws)fb)sx_9n>$heno01qt(~am>M2j(>dlJ1p=7ow9x>iZjMUhJ!~@ z2w&dJ5*=0T(ym2)d6PSl{+wIyTRy^?>#6;BqJvTVE)jZ*Hgw7v6E6*uj*QEz?o1^5 z<&|u$=^G9Qoya;iC1cEhAclV(RSZ`8_Zn>F=~joMioC0wCc=xOYHybCMjoL+!3y)4 zB^e%*kywqRybmVolYCx20Pehr%mdFC~JRf%pVXd>L&wU5qUcQwC>2c_L0v;CeOh7+O^amAv9&L2`@qu_2 zfzUPj<+-mVTIT2tcv-|d!5A0;bC6G9FdmWgF%mGn2o#}>VWFRhu93tE=!erlALmM< z@Q9_;PYNWlNYeQ9lLN^tk~|T8`g!O<^@IwKXgd9rKnjbbj88u`kjf&dHu{$lk;?y; zICjuoBG6Y6sd!Ap!g31v!F5F$&&DE#6SHW)FU zDYA~QY*5EfFn(a{{7F6mC74yj~(AZPb#U=IBD^hY7038stzWo-#uduQEgD>^vT;`I99sm{`Q+G`S=>YIrAJ z*&A_cQnca4Jjd@ZV)gj_g`bSyU&P|^`wKfCzrTpH}>r0B2JFq zU&P7r`^z{su72cNI4)k=;kfw!2j$P#@(E%AgO8ObC%Y?(l0Bv`(Jn0FR=%c^Q4@dE zf>%+OXjCllcpOd5@O^WwsAwu#ZvS&MEkcg^DJRhmxCW!n33#OSi08R>x-_2p!??!h zNyZCfJY?DlPH<`3%0wK{iu6%(z~VkA2ec-AlpKUVFbA|MeUuz9&j*zOtxF#%2l25u zh~Y7m9Stfg*FDDOHD2r{;>Y(ISAvdL^sd)qZ1!p4i9E!xe(8k@z8-I!h&I{2qQ$S8 zr~S6H$*O->w8>t7SG36zKSJ7;-9bHQk89;5drkf^d7$VgqoT7ZGxHthtHIHx!|6dz zgwW@5x|zz%Bp#0{4|<*Pgm(|RW8zs-gz(G}aA7R&h$S}*Z_F+Fl8h2!&}LpCL&zu+ z3m*_Nv#O#BDtE#}xP!Qjnl(l+HDNZ|S<-n6kzJTrN7CK0=KQaIsQ!aU)-A zM^ghPX9EA>{KK>lKj_1M{L+6SsY3pS{nQA2u@%n@e5=40S@B^5Ultp$_Qmx3)G@|q z1|E_mynTGbKhMVOA2!bD&&M2}8L|0NZ2E@%+YmX6S%-5$wg&5)yVWM|qQ%KhMF~ z4rot4&k@GL9|nM;p63{YALMBKLIh~qc(9#~SQf zUt^3W`2DEIyCmrk%J~Mg@i6hFUPe#6+>AeWx|gY#OP^JIr_buC71Kt zZotd%4g(!7Wy&UkULry;%`efB$_DWg5Q1qw1pAQi5KR0NT#uJz5KR0Me0I8ki9doF!ad=MAA+yW z7cf)p@lp}OdnH^`EMT96FRTzS$%p#atP(KEgWwBm1WfZMIJ{24f_?_O8W6CcpTRoU z3s}$xeuIDoec*9MLi`B&z;70?pr66&w+Ogij#u0z;8wsMnqNr3of5tPTOxYJvVGv2 z0#@=oE8&2|`@bdd)e`QLaE*l7w*_9wvs1!izNzezgvERzuct+OF<soeB~SmeI(^_JEa+!2-*Wc&sWA$rjN{LeiT#w-hYJAI_DGSHR2 z;6PXU4l-SP(VvPJJ!wrRpQp0OR$n9G+c72;WIgJ3J%@W`qF}H8rewulud*7dH_kOc zX)R3x_&ZtPvhr*2o$}fy})P%8_Bbk>1)M~lH6_x)J~I#Js?$4 zZA02XMs_Dzw6JkOs{TnLrWK_>RYmqm{R1{OdTC-Pe*Uj`cN6h$@4rdlJ7qKD@>jCI z>E8ln?_Z1R`_0Xd(f>u+ztE_O-M>Hxq%&Y^_nz$ECxt^grubqX{{d8h#2Ee`&J_gx z;$ql{>aX;841}pJAls?SS=nE~YJAmC>Hki2o~VCF5}^Ln&fcHm2#EmhWhvo23EnHN z)Su#dSb)aAL+DI2*BdqR!%2CiL1p3bS%ayX;uZBL`j&E1K=qbNn~?$c1n3d}6}EQl zq^JmK{ufJsorqgV$w+@bokaLc0{Y`3aYb6j=Wl6*x|<+48Xv`idjChFvPL2WWVBTv zqu&5O8e{yLAB!f+{%L$FHu13***{<#|6$o**)R2{kYMjmKkD*OjPdjS1&vYs{9p6k zZK}Vrqs{0aLH~el{DAD=r}d}tRX=6Fxu}TM|AOqV?3ehbkYw*qKbnNq2%B81{Hy;} z4D1&d<6oOP*mm^)RE+tz%Kl1!FjKC7`}pKHB$Kks#t@bdDt}N{)t~tIu>D}N%ZOA9e!S}Bn5Mi!`Iow?eya8U^6fjT_BWM_vVKd&^7g|` zD=OOetk{0r9p&vi8g?}w-m|-QSyRWZ+YXf8dZ?j%{g(ARD_Wa6Hg@mbUEAK&xqV0b zrXv;2rL~93Hyzowdzpqi(B0YCu^s(tZz$clEw^y}qy!p0$dv85-M|tbMy;VCq z_Uzu-)x4Sat=heOXLnO|ZE5@FZ5X?I-=1BihZ|Sawl-I_-`Uu?eanXBTbDIgg<6|d zY&pDV#m?fr+e?dEc2{7G7PQ}c+umKJcQvlKnZ{eTr(^$yJIc4NXjXWo zvO|^2_utx3u~oL&Skhj#e{*YR?e_NS<+t5jwSQ|v`RYR(s`l?_-nIEq#ik>bE!A5) zCTUaAvbXlo`aKPpW9{np^{t00ckQofsHix!Va3k2hVt!ODps`aYhJ#(XK(ekN%+}Y z+^}m|d(*+m+I1do-nDFBbJgk~%{wN4n~v-zIW<;oysN3Fv~ADst?j#)*WT5*UYFa> z&5Z|dy=x2Uv0C!FYh%yu70t0}R_#A>M>+4aeCuYj&-yC#S-!Vx&#rO%l}LG3@7cY1 zAMEFF18gRi%(fhqd>q_b+|;qNwK1k_*KgTawRPFvstt$tbhIDdTZKQ^#op@8oy{FP zdtj5BZe6zXuIB2^?R$1D*KA_vfreeB`x;`>H1)N9S8Zow`|9F7yEb?2ovdvKZ+oa= zSM7nlyH-x3Uok%UORTvY_9Dig%GM~LYjfiKv zS5!^Xu3fgPxU+G2&m?U-W!tTH?AaA+A4l$noaEdg*JYE*bVE}$d}ADY+ot?;cLjW{ zJv7d^%Fi|ut%hBPW6!0sRmQ))N1EDK2b&S6W5!8)yYb?ZwOv=^@^Rvq(XN%o4K?rD zc$ngUEMG9@dl2(oQE}&ZwqvwgLG8#cc1<#uonq{9Y|)^{<=nku5;<3KIq%+heeHH# zFTINKY(?o;j$3MR#>0&r zSXXvq%~+1LqH^eSuY zyJsC}2_9@|k2MF05mVXH(AjmQ?XJ7pTMT%+fV-A0FKet@QPXz3v9tzbpw{;NVI{GtO z+Bm2yp0&n*_aFTcl(%fjGOj|k$#mRRi~Geh9eH?*I_;&@eu^3u@|NNs+zwz$7F6>& z+Q?AZkvt>tS?8a+|IG8}-ba%^pZeE=GPQ5!4a7eWPtu%`hQbx&~8t?(iK*Lp%M^_irw~L$wa^04J@4Va3HdzZUIj$2zvSv~d+y zwC6ijRmHEOUOUW@SW!18=D^`l`$ zZb4{7(h0~PYf$x{L4CS9=|XIq@O|{h*%lhf4SF)k1+hu(&>gAMqO|MojkEVIeDOpeBN>+J|I=82B z_4*x<0~MmLcq5qXoQj=JtE``Cql2d+cp*Ca-Jm|LUjE(%Z+A+o*V}({oO&SWjfUvp zmcLN~TD^Wt;H*}!w;$5#_4Wa2-*);TtzK{c&sx1sKUo?R(b3!ErEa{CJoV^ZBT9IY+vR_|thdX*Ue?pNw6?2KEuDn?h|Y*rX_xgxUy6jII%U1xANyoI(bxTPP^+i& zYgOLV>Sd_msALHc9i5-GQdstSH5WJ9lPjpPX`P~?>H{2NtW{oB{Uzx?pR?7gc>kiU zUd3r2#-pO>sCZm#t5@+@orA0Pm*o6AZS7S&K4q&{@tBFEyQ{9T?O;Q&haoR-Zn&l4 z@WH0mn|Y$XA;bzpY)P=Cyt_5@b6Z1uOH+3p_AM>J?z*nS zO-(IbU2I7s{`o*84<7Ci%(k_3vB@`1a#PWEaNnWpXnki(u&eFRK`Dl|#vo);)zU3Q z!Im66)ZMbAYU^!_ySf{i4v1D8A*<~z-R0fg!M4W3-7Qor+M}bTwz;9Q<<7RI7E8U5 zY)fn1zF*`!#|Sa+xwrC{clm{Fmgb}w*=W@Fi-s3%dZk+ z)sAg@zwB?H|5e$a_NjWY^M5b!vGg~XDLP)2ooO1BimLufe@|mz``FawygU>dl!75@ K^nFBo|NkHA`7&bw literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2af3de0526072a130bf76c531284a306.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2af3de0526072a130bf76c531284a306.o new file mode 100644 index 0000000000000000000000000000000000000000..63e65f13c82bda09b16b9350ffb6cd21865c4c77 GIT binary patch literal 92968 zcmeFa4SZGAnJ>J~N0O5dl0$$HNK6hPgd|`H;lrSaCn1o40uhw2X`6&EgYuCZKsTD! z zz0kh`b)C49(p=Mh{)q=(d|=l@FP8nw$^Hkv^ko12nLp_-&w8Q%{1@NqFaH;+>+0?@ z(97|+bM+J8|6BcKyI$zO^@$hyyWf1H|AB`n&7$t}UwrVz6G(G^#Z&#|%fH#5`sDNd zY4y+d?*z}U-}V~PRCklCRzCTq7w<29qQCq>)K&3(|H{&z_WyI=Z~GrO(bu0^wz&KJ z_4mD4e)3B%p1&S-ReZDm{J5X>uT1_4_<6Jc^;xeW&9&VtlD~j*?tk(87f8 z{rtaxuNTn1-}aZUdK+n$bgy{$OD}dmc!JaDda9>zV8xU7uPuA?TkWa$6rN5^j;u}n zL7_$c8uqaT0rZ*28(F&rIyT7@9z5%tbb@HjUbeu|2hfB*OoF}%ooM#31{@YN3bW!4L0x=D4cQ@$~aupQqf`QPM|S z7sJrS??P8U(ZxS4U5t#Ni(LtHvCGg!(otI-wlJU_i>_xYuBfBxZJp6-i# zz1<%!Wk=3$DD!`Ci?4g2$qzkFf*(!pzV6p|9(@P8ef&o?J%Mnjpy%%+J?xc%8Q;Gx zJoue!!h>agmNl6jTKldq`NZ#he%KB>aYsS&U~~4NYnk7lwKFIA!`5&5dXE1pJlH%H zzhCeS9{>BAp4NwO?WOQw>komy0em~~&w#%J{AJ+J0RIH|1n?o?CBRPu{{`?q;1b~P z0zU-26*v?4Dc~sZ^}ug=dyantxE=T!;9mfL6?g~mbl~p*KMcGccmnV@fxiH}8rTQ? z72vypmjk~O?rFUb_zvJDz^?%B0sa=wpTT(@u9xHbSAkaoUk_Xa{6*jj;HALXz<1U5 zw0`940Y8JS9{^v4^LseY!?_>lIXJ(C^DLZS$9X2sui-o$=W{q0;`}nsc{u+Y&XaNe z1oc{^ube#VV=MD3A60C&uuh39M(x$9e|RSr7SD-0P8WXpT_-A}(WPquu7-yhKLaVIHXfZek;H56GpH5^&H^U0(SL#z7y z&lX0q@BB{EVDrn-Ygxl_|8b1_p|f2x*r%WVb@SgPeMmeuduJWl_QNZWY=7*^BRis3 z9=T=9l}Dz5KbyyEh*#nf{1Kn8$Z>=7`5o}^cf>=+$(Db@@9)v?^ph<=#_u1}@3fOG zFW~oc^gEUF*_Ro~zGHLJ;K9!%4enf>^da%Z&UDSVV?Fp>Yx7hJp31>fIe01uPvwrx zh!>I-@j~*0o%3;w^K&2Sic(#HlPwS7_XG4hM0SbIHW{~Z2yKH&eb(&0AyK3=w)Zer7A^+h0NZa}qv?c2AX|9c48xBA1Z(i>G zuql=Ow-&~3I*+(96*1$^XkE{B@>&`4_~{zegECj(oR%6{TRQE;SJOSLr!bX?G5DTj z9uJ=Pd4FC8e|Vh-dg^UFxq`95P!c=Y;$3qxl;pvAHO^?0cNNaQlY06aPOk9V>93dR zSIYD&;-qglxgubvub1hU%k<0Qq^~=1$+qcBH(9{k#eM-&RrEvogpA zXCYp?ExTrL7GkLL@beCRCFm0+y(kX-Goa^6dU-5;k*Dx9mA5jV4LW5`tr=VyLLRfO z0$eYSbA2`HDCO6ASxz~7aDBF{yDGNs+wKFsQqq^ip??MR`I5do4*i>;S4;ZpIP~v; zzF5-N$Dy}_9tK^?z~TEBxLzmo?TC}_Ezlbzy*&>7cF=E#Q*P$co-|KzfcIrIPQDV* z?~(L};?S2+{_!IJU&Nt*hVo1Li8%DzDSx)e|D8DWhbX_K|1b{yo0NZo$p1nd`e$ke z)9wM?)SE51?wJ^SeGjFNbNxkm-+oY?eVq4aUCKSt>%$EN=#rH^y{ z?fn`n=fmuTlCq*I%df`LXHW zqV#dD_fz^SV$;7z>Em4gfYKMlrvGSZPue~Fy5<+spdU2XYxWi&DfJY7--EdxjYUsm z?0TUD5Zl&ue{joE#JsiLAKq2deSZ2+cF{a-=#USwZ~)g+K~F=vbfnKf3_Py; zJmS4R#H7EuoAtbQ4I6w1ap$|~H9fyW9Qxj&@ZhiaA~yJPQuq6aPX~SZ z@J+;iznxy&^WirFs$l%L6{VJ+G&D2K&EAJbHTiuEN;)*c?EIMAs?;n@O7X7!(0U8cPhqh>d%kXJl###Fn2kTa4pFr^rSTzXXO8A%@4G> zJFPFN<+0JGFAjH8{gi&1C(L8q*0lI_e0bOQpnD$Z9_C^gHwT)0&_6%)FR6RTm-+Jm z%1WDE)15KDwmWn2QeN)oo(m5KQTEHeaL=hj1p}U%nzBG@81tno21-+LEoE`p@uh{Q zOOfWdFMptcpRX8LF|DSo`$zewkN+s|^a`YJrR#rlb$2WB{4@TJFP(b&FkL$|W#BN! zQ%)b}ICtPU$GNAU<#_TyAKG<%W<^<`xUM@5*GlR7tcBfy;$__d&}I?-(WKLX;^mwk z;Q1$Vo=!Y9zx(*137psCODCL84b^v-BYhdtbT7?5o%(!u@Wi3)fsiNskEys$dAkpd zA2_~rJj!d}<()Vb97w};)QkB^2z;z?>gl_~OVH2Q*_5_Moa^Cv3 z3VPohN!viQ>)rhIGXG5EhpzMdS5cZ9oHUD3{xwwYOuD|R`*rB)R9u^aYt=Ykg>#^I zb$1SaFTh`*cun_2oM-aijoryO&)~mr?Dpe4o&R3j?Zp{7gY@gVNq?&F*IIH@_jsHu z`R|*%vv989zt?wW|-{xq-q+!v;Hzxwc$?$;j4?SA*K zCUf8gP!92Bb*O44)itQ`qCRu_x(NL8wdI}5MFz_?;C{I4)m=dyzVso3h=stzIwtp zo$fnC_@;rr>wtYX5A?ymAOF$Kr%_MOao_rZzInihaGpc-4X69o;rddXucv$)2KuJs z`qGW3kq>y|Kwlo=O{e>25#BV=H-Yfx)3^@2d7v+y%K5W_z7(8)^k=90d}8j~`l9d) zuyw2>E`#mYVGLRde^3j*Py_$`W6dXh;x9h&7oYfxPy9t9e}VXjAI$3r(v8YQM{4fqQn@ohMV@t1AX@SbJf2i~EK(Ho?pON$@C>RnHOCAzYpAYU9n4g>WweHzwv5Gx=_iwM*b9=@9+jj2RvDd|0 z4@i7_h4W+U-ra32`&q^Q?biMZ(X^(m2lnr3*|)E0_q?XvR?{upci-M5DK~B3vu|(v zs+P8WckI0L&RyFbyiVYKbE~SiG|g+;w*O$;_N}}SS=g4AecPM1?b`*y&ONu>ymQa~ z>Zbj>Tbg$4ZQtFp-zlfAWq-?ZeyMiTP0O09nzrn2Bl zTg<-X7p3du9jIRT6X-*`Il9jBtA8>kZ1*j1&~P0r2Nl&9+xhm1I6_7uBcDDALp}*Z z{;P$Bv8rF()Pxump?u4(rphLutp~QXH0|3xx6*3bzx9^wdny++-EyF%eOr6W&RtFW zG1fJ0-Mg!4%XUPsP209({M);A`?e;8+D+}-_aDHNU|Z9cgH8N8QwyIvy21~D5fs_7 zALXE;1H1R^Gb{Tfbo&3?j6-v^6xf;=W06Pl*c`v*ags9 z3my04x)19<<}rkWuiJn?zv)1GI_R90aZe;t$^t|8L`nnvr${r~EJR{kX8y&vAoBY6 z&k{r`k1o?ezv-ZQBGE{xSxq2{YIBGZgw(}xl&-NNo;P1!Yn5laUc0a8GL?V1>-(jw z&(s0Bj1@0zCTHvn7Ry$zUs1QBCbV>KdmD5~9JGo>>u3w{w}URV?YCg>OEJ!dXw_-w zt`;)x(Eh!9ckP=Ug7@5xHKNemc@^_#ht|`nV!Uzq05=diOHu3*2^@RjV0hG(;9H9U*;YPg)8 z)^Hg+rQuR`T*EV2mxfE&qZ*#UIyGF(4r_Qii)grrSsI?k+B6(utr{+5%^EIX8#R0d zYt(Q)YtV2W3u|~Ps|2ob(bI^u}+Pi$s!sa$J#WU!J0Lk&KfnG#u|WKeo|Q&*wwcHtJZJ|tJKmb zvk|qh6(z%gP`Za!Tlb#xMKqE5Uh;!Z$^WK0Z z_a>jmKR1JQ{tx&f{ISPiXJv~WpG$smtcSmJFnePy1vd>unTAWok9KHl@aD&)Rqisuptz|XfXBEk+e8&lo-k4i+a%7FI3jVY#GMj{B<_;9P2y7$mrC3#u_f^ti7O>OD{(~P3ldjL4C~@(hvTrsA&EOBhV^lJgT!HpACr3IwM15)Ky%Ozk7w91@t?L%3_2z&3xU ziUhX#J2hQko4->CREU3@zwkVPNgh;Qr^Jk7>;k$<&;vHUM&KaFnJj#rz_~VFDsYJ7 zAPd(ET*`3)3tumACC4EaUL$Zd$3-l>PT(-d#Vou*;0BIMSmU1wEch#By`K?S@K?$z zw+JlwD`j2V1s433vfxgE1%IIL5?Jt8%0g`d3;s%3=RSc2f2EAwA+X@Dl(pR{u;8zh zUHGiPg1;bZzDHoeUyz--S75;(BPoEV0dBr^H>nyZ}2ZF_XCRZ$$op#GMicInHDkB+iw%@wiAI;yB1oNnFZt0Sk5u zdL_pp)+TW^$3^Un#9@w$StXttAp2?HxP(O{7W|d6GZG8_pikct=>>n#7l{Rb&|irK zf6&h-M0&wrDQlKk@K?%CNi6t-K4RxT@gw*vWsMRG{*Yc`!C#OCF(;$+f9;pmWK32nQR%_+eI0?pd*(#X&}B%g@_QwMVF7nlrB#TyMwhN#k;4NQiq z(pwE2GH{!LOATxpxYEE816LdPuz|w{?lf?Nfgd$+mw}HP_>{!Oe)jZ1^aNq*Mf7K` zZGY4$(7zfRlYYK_H>)q(FS|?Vv#GCI{$E1>@4rDyAGHFG`BM2&3*)a$L3r+d9~1}r zK}rJbwUW?PIOrnu>qvXbaxwsJ-?0Bz4v^vD2;jfsOy=$I3IoX!?WsR7QRHWlboQpG z&vqYF{-@EtZIqw%{XJ10H?U#Ldow^jb!g}iFKrE5+8caIt^Kk4({0l4M`0znCq_fL`CNKN&_-)jAd_H(#=3F8U)f8q|m>%Vto zLLPw>a=5lUq8;h>dKBk3+Of*OGc|n73Ix>gBib=TqenZY8+e+A;co*GrT$Lq2HG;` z@Y89P1R@$P4$xq3^V?|^1tJ;_Az*Ro(T+l+zJdTGUa70KSrbB*H0}-f%n|_L$p6gpKU+i<~lMQ-KAc6tgrB5>G6ZyiSO{e<#0Lkn3 zVLY2)q|XjSU<`J8==bKld+$Q^bBl2V>aM=&3BEV5*;cL|MPL zBaLCYs+5;5^wTLX1N#hB`Zz}V=+>XfFuhiA7K3|Ha1gH+aWU1;`Iq$*E+#t1NxXkJ z{~V`q9CGNX9CLm--NP}rC#5{juTmc8S1FJ4%dser^UJX)kMqm15B}s`*MGyG9JPkV zL!IIOBitU3SmVdjLh?{0?DNN2>^Z%HO)=W?W-uL+uwAJPWs{Yhv*--zkDb^JIEaSE9Cm% z_OA!@i9y#s=Wx2K50pPANP~~)50T&1AIhH_bnQ{(SM?44BS?d%nqSq=PAfkcap)p{ z$dM1#Ul4>R((=3dFXR_-=puhYd8F#)#mK{<4Ak zgWE?4Pepr(K0%?+L4K}2lv<9r*`!V`w!#9@ztH!nRf8WFcoPXfZY^KV{f8XiqaF3y zc<_i->yIFZUH{`&R#qi&ijzLm4oGDEg%*-L*G0~ z`Z-$tx;@f(soNj5Ki{d}L+vkR)B%qEl+}ea{!8m<<;!kwr!}{(QlnSbvDe-9MLSk7 zXe=Gl-f(ll=}+dDFt5nNm6u5G`WG&bY<^=wrhl$|ny8Jf z#`s^z$J51OT$1|yTzkkX5)fUK=lV<3e*o^I5e*)X_CJ=%o@>vizQKX?&&&ayLhe*x ztgk-To=fgUV3b#XfCaw9ac4UFk2TOgb^VmaA2psK%%dxwL&PiaXG*+c`qya3axlgZ zsgHX6(rGQtLb#~(Z&*%`Z2hg#(ObNe~((Y zIjqS`k4GpTLWxoyvGY4K9(u$o@JHJZ zMeA+SLrDA4jH=>MEz8{6$~Q`^6IjO{O{OX#4ee(TtZU9z@BC zJoI=lZu^z^gW6wew^z2m!Nzv`Ew%j_V{CtgbZ?^lSZ_7rKePX-a(SGCe#Gd55x;hn z8S!5!44EHXA3CjBS^R#49N%dEM)JbJ>6(0Szl-|^B|ahf%(mr$^?u+SM_#skBC34; zHSl>}{4oB}@DIYY3*~U4$j6B{F`mTuk3{k@( z5b~k*0*|;L#v^TggT`AkzEkxH@kWe4(&r;F`Uv||=NsgYXoMS;yr7Sm52QHr0o*U; z{;5Hi-(p*S*iR7GKQ*fIdoyqkZ~tih({dVaaoF-xuqlki%`izP_i$^L9KJ(A!U2kd%0C>O>pc<5jcT{=;Kz|5O@IaZvkVFIxLyA2I&e zY`-4Q()?6i|D*K>ntr(L*Vh9dv4Z}HO}E=0)Z3qHw^zh7!F(Ir?Qd1vzcE4k_4P%> zvou`dun?Hi}Wuh(I}kJ3J_PZ{F=B#kE`K1p`g-(D5- z?+~%T=WiSbGi;3ZuX66+R7K zvHP1UEEHxR*e_z3+v6DYI)_^SPoOlD>KTQW$=Ub@r!EyIeGf}`juLK)Sv9$ z|D^U-X#DZ@^HclX`-w@Y|5_veby|Mnm&enb-xSz4?Cm4F zJX(Lu%G1~LGd2FHKK*_%?CnuI{~F}a$%eQbdC~oRRi0FT$dOkP(qsPVwdMOTaC#n1 z&~16s_@&w()nDx7C;nkSU3UF*(Eh@h`bwSjbiZMG9<^VTPyFJ3l;5tO^ry(J59|HM ztb*yBAGsfu>YweDNBqNn9=Gd%82M+$)L-eOCw^z;x$EI1Z`ez+T7S7)KgN?|R>>^0 zepTQ2{iNe|{cBMFoS6Dm{phsj<GaKlCR}tzYrCy#1Mz&H7#aCHccXPuca8{I7~>zpJk#f7l-$ zM-%nWq4+#Gdjm7n6Dx%{r4Y4Dru>LbYy`uBa8-)M*KkE#D$ zeIx$oMf_#ux$8xwukcsdYJG}- z;`&;y?H{N5U40?`p>Mr*{iLtDe>!I6%rWHUclC$(zbemNe^>(y{mW7LQT#oZf7Mk+ zdC4KjPmCATzVtluk3yc*KIq$2w>*-EYd^$)VIEDVT=~QQ<+|y$_<+k}NugQ4>p!VI z(7)$xc~F1o@xd{xC`R91eJ1{|$#dgp;ve=^r1EpM>rW6*Qh(_BG&7I%!)*`r8MlN4 z=IQ;q{y<;<)#aDUbM1rpfqtB^>EsV|e{{^sH0{Cf+5^dVa-KWin*$7eDRBAG;z6Nr zG5*QbUy?84nP1xckbGnP^J27L=tCCL&&zY?TO?1|OGvF>@y}epV*RtLuT&oN^FQqR zN#Aw<2!D2sp&!clEc(l}AJHFf{|UOQPlE31hoHOtC+Kc}3A!tPL3iaR=#Kus>NDm~ z*dJXspKT=h$^G1Tenh)ptIcm9uY0lIx$ZRLcex)siv6Z?!8gFLf1BT5qxhcgPxAX~ z&jkJ(FF*eJ%yaD|Jv!i(e@yaqo~yUZgZN~5m~S1mCZy04MSTAx?4kZZuUr0mb7G!X zG52#rA1Ao}m-=_o1d@+!FUPEmAwO`&MC?~U`Sd&t#bZo94|CL-i0&8~N*NIQua8=j zpzac5{$79o6G%)4qS7~Myi4Qd)yeX(znbG*dVUAxAG4A<4oV!gN)YbTj0y+nQEMg( zhgsOhN3EQU-n-rQAI0-4Lkgz--s=4DDE4#f>64RnoC9;g{qc*UJyB~i3y9iy{g1Sl zq?Mp@TRul=zm%3fPfH)Qxb|{(1^;FE9Yr1-XwOlr0O7w|-%%?c;gpM`))iWNqgJ7T z%Xv|Dc~R_V3$m!Fl$Re$Ci%JTnP%kY)+5T`^1YIUShT?|@2FLj>{s&Rd8ZOgcbxV| zt?5R26{0NqmzOt(RkCO+zsUJ1PNoN`G$F7WH0W%&iA6IqIn_8VgfgL9l#G+W|6jx&oZT>ANa%uo7(Kj@!N z-~R}HO?%Oww;}sf)|0V*6W3m8{f_K^8{J=jGq9VthwOhe_KETPob7MPKGU#Y#+4W5 zM-#{&+VaDAGlBfSjcxl1IQAy&FHK_D55%k5?_>x|oO^E1ZxKJ~+WH7z zuUFPvj#`<4vx+@J-_ulk!}Dj{-qQ{HPnH<@;Rs!QiCWmWgZA@->r;BVUcPS6v|m+C zZ`$KgD~RcXTE1zYMFl!t*Y~4Vu4cbcYl^17Q7e&s7UOwDwSHay;a?2<#oIU({};6q z*>6dKYQM4mGhegsSbvkqo(tf>RQ`4UbkquI_8zr#|I=xWtE1(5TVG+{Ip}a4j=fK& z@yW)p@1oq&9I|)YzKf@pYq%tTw#2V-doKu8xOCY2xMA%b@zF#$C<9{aKPwLn|8>9WhzFt-Bf1I`-7wfqbav>dCe%Nn3A*^7>z8w1>$L(#LWB<9x z?%MlNYYLX@U7XndvyAmd+^oIFQ=Z`;8@GpKjOM`it=& z4~)^_jQ@Fj{Xt@k|CI&hQ&sz~3YBVjeo=+Q!v2d(X1nyuZ~xP-9F>ZFLe7a%~tLI zs8wFB;;5y^8?-*5&R_7nLtUG0|M{43;c)E#ijaa~|FG|T)&3V2S82Gqq*7vG|E1-# zT{`Ul>dQO+E6?Ba_`kr6{|i{DYX1#^7T%uG+W%5*KP{ezoqz#JvH!veH)uE;>jThF zY5&KprQF_@I^%r-#H;#)LfD6k$sgeMkMIY4zJHo?>(2M5Dff42{*U#3<$k#B53-&5 zW-nF8=VR71&d)T*AA~p#4eJkPV*qmP?Wi?vnp)mbt16`WgQ#_-X8)vbs=Y%$r>W&d zExtM|+Q{vbo_{aNrTlQ>Lp+Go_d+Adx4e~l6U>*G86qe{1b)1T<^ z4L$#5>GoeBA`Zv?3*m5a2n_oVmCi3v?Z2pEp@xeqt0flpUs83oONae0zP$FYyzhhC zf5^1|5Px3Gng2HjKF!-RTKivaJfDc?ts|~|!~P~vglXG5o}Zo2sNiGPa&BMCo$+`X z$IBf1424wtC;L)y;`o0C;te-`yJ#)&5JXuGVn*{A!7X{m))l>Y~t-3t^L>5G39xPPOC`g9eK%Kp8E{VV&?VE?se??#FAeiv?Uvujj)y&U$hm!E8` z|C#o`giD|Q6W9K)1L<}1a5GoF(b|7ld48Yb z?dve!s`mqmC(sJ5YyUGQY*g?uE6nXZ?AZTx9AD?ye=(*Ls{PNHq2k2$KU3T9$jARK zm%pP{X;`&)jJMaR_?R`5^D}eU@xK(|jXFM;Y2!J@--o*`nO1A$f%>A>3d8=(EA;U{c6`?9=6XY{f6?uKc7;Cv z$BxfBT_2wl$Nv=-`uMNUPmWsE#`v$VH_-S`kH^{ii}k-^if5Jae+KEdjS>HsRIMyl z?Z13}gNA1>Tr9D$|BC9XUHau8|7RNOeSudr)h zcz@son!(!g!+y>QpHlEKYYDfnC64`1;5gZ_e~PbF`=2>e#fj~|OxthF`Fq^u5A*FM z`uJQqLABRoRvG7~Y*_m*H`d$C`FvHGUcNd0&ok`*1}=U654Zm+V?MtMB-Q>)4gX-q z=kpEwH|h>LLpE}*N&shJW+kZu+Zr`!?snd1)H0Mhczv$zCB_=?s z|J2tHj#^h6_J55wKj^gT>mJqZzeJty!yn8PE2vKVUpoJW64m}^FI=hNis}Z5h5c7v zz1XG0{_8Jk{KtBx$JQY7=j#3+jQ`N*vO>`9_a|%b*An*w;NH++#~*w@KpKaKq(-$L zNV7-04_DidgZ2BP)?#fx9N&+v?8oQ!>Fn<(ek-{B)j8{Li!q?6{duisKgX<6p1yQg zd#$L@_TS_{=mWa{~=x|RsDlGp2MGL;DiBK)x>j z(4UE7gcSNi^YcKMs>8wUAJbPQy}3S3>xue$nC?HK)+)oE6V2Dv@p=y80e!u}j8Dj4 z==QA7*B`N}5kI)?kDb5A`Ul;9_4)ZD*42m?)bezH19?s|#^dtYi{1Tn7@sSuZqRV` z{FM^F&iCV7y|BTh!yf&^+9UdBO#5;8{yXJ;cDB7~`|sWjtf0QXZ1y%=+uz3RO?m!= zzu(Gvo`vE+eSI41VcPmqtbHx!_O{ru&nj*I9QC)Z&!wgMc({16ZcpXLco!3In)Z`J z>kD@Lh4mc6Kj``qwG!EfxxO7U9xc|#J99h|+;Eoo^T}7@r#oIJwjXo-TlYs8uNLd$ zp*~(ktwi=@u7B(E0r=;|`gnP*=3mG^8~foVi#rs~_*cHr-rpwnw^mf&;O=jO{Z(GQ zQev^+wQ6yLONYJT0|&70c>7()y8mrBd!y&Io$+n<<+e9%KN+_-`*|w(3vO@D^IMn2 z-n9K{@$5~vuS>SK+)wO{g@t`n|9$-SX6!$k5^Hbn{xKTgs&CLR>`h|F-dy@0#oo02 zW!nPFxbQAlJULt2zs2p%e!kGQH|P1p%VKZZ{;+uVcA3Ywsh`-}AJ^WD{a$&o_U7)_ z685HH*qg+Ty}9&1ioI$3xxN;t8Kb>9`>pu=?fX18-(P|G+cB%0$9wkkPk6ptdmfjM zZ_e|^aqX>K9gmJ#vv_&4ocWrVzq$J@&G=G}M~_;wW~t@H&d>Dmj`l0*@#sQqsCMUL zn7?W9CGI~O@uVJ)V!n~D+Gp(i<_0N0oZydQ=VKM6YCMVcNF%<~8~{DY0XZF8z;UkJ|pK=L0mO9?SgB+3&>dQQ7YVd*u5mmFL6x`pNg*_041Wo(0X9 zvz_@_wXxqR(f;PyLEXNpal=ium+=YqH!srcEp~mQ5|5X=<;9yHsq-i7*M@?->2bft zSRWa$=}+u_q}X^Zs6CIEukCk;-QQq7-(c#mx_(0Teyy=SqTla~-QS@5mso!idp}iQ zFVXiq#O`k}*NgP|F~v{jekXB4zr{>SbwqSYsNy~4CNdR|fW&x!XJ#oB{zZv~pYna>Z$+Mj7}vG$;^N9y)wuAj!* zpLu^oXg`D^$L~eP{RQ3L%=Oq<`!nq=J7#}TKCjHNH*7yPmW@n$h8|zcfexXG6rt3fKEn6>NUtfp6(8f3OdDq1D7Hbc>y@d>W zGoOb`bblb$9xkuF1-Z8He|Wq+{xaKJY0Q2ebT|&j-pu_x`IY+qolup%e@CouVn3g; zpC_p8=lS@@x3Fq&Z2{ULK9=}0;rOP!Z-?94v#PxXjr}wE#{QW^XoM~?b3C1@p ze(D2pe^R@jm`LnBB`X6O{Yf$a&cz`tOvf7*S zJ}ho;&ujK(>{rP*_N!dFy^TD+nf6xBThITAYj5WB>UbDRvo}4SOKfktebanWx3~F* zz3K5>VtW(2Iqmk){86{Jg@(PE{z8AA#oUjd_<6NN_Eu`xo4((N?Cnvgr@h|_^R*Jg z-ewy1Ruw9D_n*N2<`-3JcwupcwqGT`bhb;stoBCZJ-%<0Dl{rU-{aK2&u2fcs_jqt zj{x0a9gF>4@FF?AABxZCPVwCK`X-MrmG`l6`*Yr}M(dwqJ;mOS61DQR{UgV$k>kr) z|5>cl^z0Z9E@qjFh_!#+KK1uSJYtm=QFq(z zGv}94E7AC;zaQoitGr0xf1<|&u(yfm036QvHc7QV*jGum{k|s=KNjUH?|TBhw4hS% zKN0Jd<)NyNX+O$n?M>UCk~?O58#%tj`xun>4R%_7<$Z?~zlF5@EAZdi{t|QkcFgkg z^-jOD-Wdw%`fNUbot&(XcYeRRo)Y_hrV`$}{2xBv1!MGGk1vah^!oMZ8DTF_C)M6e z`_RW%TEEc8JJN48zdjz|{dazyJ}II7>Gm`m`mUC*$9I^o`gOXwJ{oJUmD>B(itvFZ zwR}DPMf~p9>H7OC=>3-)_3*Vt#ZT|}FPupLT*xt@9Zu)a|W2q}!7| zU&H(G411fL(4KXBOY}aZe8b*!d#3mG8v9Xl)p!!)pZPwdqU>UKzX^?h6H7H*o@2is zN!VK`-+n)mSg*tf94_a6usHUXtIp>h43I{RC7w(;z7-kv_8;neZlbn-1^uh-FG*}~ z^nO#-e@@ckJKCQ-^7xjjrKfn-Ik>$AjPdOXkk$2*T)dxAt$&K9k9gmyF}|g1^r#hb zsuTCery2HEkfV=Zy8l0FWfse^C6Z+gnkNZg1xJ6Ey5izyCq+e{~M-FJ>C+ zCpkI#_@&z`jsH4bAOGoooUtEeDmom8GrsZn``Z}ym7A9D?l%E0$Ovh;C|E49u(zD- z5|@tg?UL3j=U8&KLhtAKx7hEop!_7qo?nf9zj21e@IFX+sY4p71z&bF;u`PmVP%<0dp<99={G#b{p`KYdj8}OLO*l|Dt^lj0AFb(`4wAN@b+g=`;mSo-9vhnSOK1w?D+{3*5dk80t36sXPZF!G> z&neJe=+EWp@tEPcH7GBXG{wPGer_@b3>>^X@CSJ}qr@~^g1o1-PaG2d4Vd#YIfdT0 zZ>JBmPf4E2F_oW%^pl1Jf4IogC#TRCd~ry6&P0+wKY03yLy|w9ep-9VkSHH1M0t}! zBDGzAfci(&|1j}4QMSj-T}QWfCswac5-h7l%qJOD>lGz1G zPk}zk zy%Fw@>G{J>xw8W+W52+7vZw)t z(ILj8CGC?@pxrd@r?Xypd_nq_nXb;*Or1V3*_fdPjGpyiv8f*CY zgm`6&mBUqOGQ}f5Uc*xMW4WRi8CWEVHoNi_zXPN4}O&Pu=@_OAh*{ku=iI}V&WV9ee~2H8Izp;C>rn4`V00!_Rjr@ z(m&k(b1bKSh`+X+I~~mZNlpi0d>=#UFMJO$de>yPf6hI`mb6a-19txvdbIvR|3SXl zLcaIXc6^GRSid@f#HK6#i)n?hjCYuhXC7d4J;llcGP5i|7RSiKZR=l;QIqye%TH`7~iu|pTj@mH|noR z{z zr1_7Kch<=A&f=`4aDSxQk0b9ap|7Knx8{#hdHvwWly?e^U!#_HN}3~Y>5nvduQ2sH z1sQDp&VszNT>ny(<<1Qb9txn&*QrC&PWsEtu60yX*8at zPs|!l-sq3BV~}?s$&vS#V66Y%Y05hZ8EpNg_$Ecs?@X>=Bo98mVZ4de??94ck7(}{ zd_RTeU-+I2;*l&b^xKsO*Y6}$r}>jqUEUA}W<0uHQ+HKf&^FFDjt<6zbFb z8JBk;jr7NrcN#5dBEGTZogN^0r#kXZa`_kIWm*yodq?X}P`{Y(&p{Eqw| zGvrVE4^mC}ryzqZ|22Ai@)*8fd{>g%ACRxtjW^IR4*rQBq2FTuHpTMs{^0$G z_$d2uj6Y8n^5^k1;!A%L$=~b9KiTjvUMYWO%RkwvkIUaH7StU7;`a*qN8w*+{7mwa z{CyK24sd@CejNP~^7p6UgQugF{}p$kFjk8{v_ix~#E$d5X~;vYFE(PlLHn;f45ED? z{VHd&DB}I7dg2`PC*$)+M*2{{rdTrwHN@?c@B==QxQ)t(z0&u%#Ub+R>!pY`UR0XFxpp%E2K)rxs!T!P`NfzLV)O9%H?a3$!gn>#c>Ld&T-9mG{dv z?7`HbAuj@0C~+FCPZIV)J{=7OUa>#+~}>JQrO zanqj}A-$$Aq>q@-ibLd&yWZW1^+I}|0s7Be@2)?1fwj>B;m{$zJ{&hc-9V@Oke{Ca zZmh=^v;B$k(+$C4^WT&>Ki%LMHvhee^S6zZ|J#Z4TO;LvHF5sPNcr=bl}LVvN6LRw z;{2T><$o}7{zpg3|4iciw*JM{A3y6z)cz6muR3x55%sS%asCnY@6qA&!~ewno?e$K zPxXF86YQy!Jvf{^hAWTB^0twEs^wjW@@9{{yue81Vf}14{@{VFWM}> zjXgeM`D1+_ty%s?c6`M0$NIiYv-~i7Vz~0Tea3lz*NFC6$xe;6yb}uRed)7 zM=$&KSozcJ&n)jSJ3ZF&GRaYk8$3wKv3i$68+HNaejb_VTJnD(~CunX&R0 z9;v(z_Uu^8YZ$3Kd_H}+^7!~L3jeCvi)nuu?98Q>ui1}T{+Z#+A8z~|(S9Cf|2iDM z!<9Fp{WP;*D&>6&@xPehR5IG%F?T=i=i}g$ew5#MaIv?Mr~eU;@6GfyALIwkPk8!D zEq(pLi|nFZU$3$MwV9@BJBW%J$vu z!J`O~i!aao>bEeb`Tg&I?>jp*^z{?vH~TLB>ZQLL`f%{W!C(E=50CvS<)ak#QQ<}V zzq>CC{a`3%sPN)Y@6aLs-?{%e^s^7CfPeqrzr}I#jhFYl{qoEI`+MKLgWbOO>q9?1 z_#6MXFJAop>)(6Q^Xboh^w|TSekoFM-OxXM@5$T3kNjrnz}J7XsOzV{dHSlKy_0|D z*T4DIhL;YE>+65#?QgyDz2yCagYW;Z1E2PNPi8-P@uQ`0@AF>ybknDrUJC#8?a42_ zwBStt@BiQ59T@lRi*NsY@L=Y*FTVfU!LR!^i)y}dchjd|TF~`tUTCXb*3hdje=SlW z82IVaPd)j@)HD6Oy2rouSGy+M_4{u_U)mc-Tt|M5Xbv3zyH}^;s3kd{>S%UzUx=-zkD5JobFE>;E`wz4@h0? zVID8hlhZ!n^n1qR1tSWs@zV_;PkJWkw9gU zUTGGL=r?*isa{yNeN9-Vr+$~Tc3}-QTm5PD=Usho)d1VpOmoT~z+0j$lqb^AeX+Q7 zSvK8dtf{{}jVvGEh+C|93=iGI^Q_1~oncWJ{`J%|*y`UZSmz0x*OSkwactI8zd1+v zDCFiP=$LAD{g99QAx~zY?gI=ORE^i`^{6;%VDAKjo~mQ7fujL~Zs2H&K{s$T*`OOZ znq<%o9Q7M?1IN(K{08=#^&8l0@^4_T*?ton{CSP~W7=!fA0scL{up_G-1Vb9$`}{) z40vP4IB%Y6pBO;XSXiw0Kq1L~r%x^U0Q<<C&1oitqeDd7|rO&Otl zln(jW<@51PBVU^%i*!)pC<1#uo6wv>WiB$ zJDkm;*(24f^-HNKxAbs2i>7P!XnbUbu`wufr1Fy>w_dq3gMaQ<%byX>V9|`>%O4jW z$D-p#DqoWujk`5@ucv8sh%$z+L-gH9b!ctiW3$@#LI%T^pHR1S+n}+#W(Q(awT`SX zZh&TKb;P6*I&QS%9LC_9zb2dTV6PTf67~|CT z1jBf=Js3?^>3PHRx{>tK`~(nsL0S?4clto#k@ zucEfcG~_3BDoLe}z+dYn@;9u%Q|njzSNglr^4E5W{E@%Z`p@={!|^Bm;%ND^E|I@s z{pxW13I3y_tEZaUnc%KFOk2n$)7zYTqgb=9R+`U z+|y$&%?CD^JlbQ*W#X}M6g=uNXGG(Vnf%!?=Vjt=qt4%G#yE|?fXSZ|v&#FN7)CT5 zm19UUe=1szF(dfsW{p3e|FVrCYW=E@7GnrDNh|!1z+daw`P1T!TvL8_%&aVZY4Q_u z3a?+G*U&tA3&U*PJ(PL1m)7^(IKhWCIj`4V+d}}v50_4Bb}k*!k)V%YLtVTL`Fvi) zpR2WXKx7s9rz>VX$^&&qXQD;p5oWZ5YSs z2_$jUmM16kztW84nv4v8EzahjGIr5@t5kbzv&*LRN*cOv<)pD^lC@jsSJ9t$YtF(McL{>5vweCZrLc zo+6|PX()768ev~zY2cY0tZ&$nTkvC(hSFMZavW(Wt&L0frg7ckeIGBsHz0He1C^3r zl}5xrR%uYA`SIw6H$$|B`aWK3u$0Ly0hSb(?j6TlOVWr>PmPkLkR>Vfu}Xu{nl2X9 z#S!}Wq~W#(Pg2HON_3pC`;6X*u1O2a=x+)Epl~@{h)(*O1KN(2t zJ|B~%$krf0!CTg1bpw5Cw1bPDsU6n>;6w-*0GIyhZrJbO3V&?Uz>P58_u0MK-qbj{ zVfQ^2Y2(sm-;=cB(s9EoQ(B5bSEVs3-5_c7+R|wF*ref=nm!>sAsSB_wx&Z`ap|(P z4AP2A$CyRZ;JW3OuS%nLd_3KVjdLLn=!bm|bl5mn#;r~aOA(e5%ewKYFB9=;dV&~M zOJk2|O8iO@Tbh?VE7ym35nD8!>l=fFz47SYOzvwUq>u6F(M;}ZNWuzTmEfr2S511u z#;=N`c%-gk^S2wtb14WehmAXhRge^vxYLdDrJj%!;?fb@WRa9f&s-^~QizNuir1tt zY}~50)X*PQm-z4Dq~MJkp}A6sOIM^2m##=bp`NFgp=k%B_M zG${-lr>at*-y`XRp$puu8O=-MMr4XE#G@-xh@aoJ2!(!WQW!QqHKi~dCh2HWzydmt zJISu&wlYO{N3!d9bbBSLm89jA9?Vy|{GjM&uSp^IkW1+iAa*RfnK6)A|>J%f~@@#yyK zp4Kwr(IEwU_O8&!ECp>XW7ILHA_XPhOhddGNDyyoY1DXA_o|q~_$eBqvBiysTka#x>E3B z!I&&*xV2I*24-n#aq~w5e7%k9;4o4c#agMp{ywUhb4*eIms}Tcv%?!V+Dhe8V6?s- zmyU>bJU|y^1cmbWIod6D4C=KOa@hIVLH1lcQd01#UShF`=sq*aPh&I&O)^rw6>Y z6ywve8H#j;bZrjvvqB8MVpwxwhJ$On>wQ5a*zq-vFVvdMPXV{F?LR zlFzdUuE-noP82I?o{(pN7EbJSHL=o5wRnp%lLuHWF8qLnq4?@iN!(cLTQ)$3>_v~DD> zeh_0vf0$m4%vDs{y2DlE4I8k1s!H5$+z?<8)UH ze>$CJdV2rrbehZPbUl6wCZJRNtLKL@gFn(}Izln@NX&g?BY(`DRh=G*x!0)ESWMDBzmm;^!%~%)9G>Kr_~=TKRthGjC^%^WdeG20(v+By&;ya^?w}w z)97*Jr|EYb`Dt{LUt==8{;F6g9Nxt&4$M6GYfG~-?ukT7Szzd%NNIro#Jjg}%$MoS zQ)hH<+zpF@-&Idpy;rE9E+my<8AwaK$x zf1|{7B7QG-eP?C)XJvcC64S{)Cjc{IY&!iO?|g5Q|F+{&x^Caq^=B);cla4K)PI$q z-ACumA@;yu7CrvAe>rhC`*!&|?{D<|^~~ARD*w6QCl6<3tk{({2WcjYH!fE(e1qk} zcn8mAcxSbOD;YkhePNnIpTo{-xPtvs!&kC18lKIb)$lCZ`lXat&Q5E%jGfYODLby= znXF60CG1fR&tRPzE@p={Je_8%O8rF)FMCt)G}fl!5Np+NA#2uf0o$nID_En3^I3z2 z^H^BJQ&}Z&jU(SFETrLF7Su4l8>HbJcEPLWpTy2;cp^Ka;R)9M}T3Tso)e=sijY5Az)X2 zN%$&|iZLo`7%%qsunQiqt^X7ts+gim6~mkaUgPI8X-^G00FJ_ghs%9=L(F@Fm-(7} z9{=17*7-g@SA;)W-0+tz=01e{$RBd}OBb`VF#J>4{&fMIof4mwxItnDnN#_XN*t27OJZ6iAo_8MnP}G|QJs@%AW`Tno zXR_vIfpaD9+#zs?;~?wZC2%Rn1?)n*z?B?_SP1X-B>t*7E@F*$3moRSm_a0ACB ztoMro3;s&knNEQPf2Hige-~KrSIUB461%IWi`DuX#e?ivvjKG4wAnW{Rfdzj7cIpLz1%Ck+{#SvIqrJ)K zkII(>hKPV6Z}K%8n6G*=^XC_{E{s=a7vjD6S1}e@ zg=L@r35>rW^!3c|Fit@a{q&VSMEDfs)1RMzWd=WR(3nZ*kK~!YM@^XCyPJW$bOex# z;$a%^==i?8sbF6}EAgACj}FXF>|3ZDU&Po-hk1Q=a{TM|e1Ou^JM`>hF)@$hTFH+c z-#;%dJJ;=bMxAk8LY8ZkV_!c$OnRF>)^iy$_-LRrjz#>sY`jAIG{4Rl@Rh5K{Vie# zpS|L*P=12zUWPBQ%6X1GuYw8i>$ZHtGEW9r@!=DK=CPPxpJZNU8*x25=<*X9Cj^I& z#8tC*=gJ9;LCIr$JHqu>``OjbpoDS+^@-Jro!?=JVV(Nb$Z}6le#Q$}5 z#LjQ`4o9Klu?70nAG~B83v35@*U! zemx?f^KTzAQ<+PhEAqz~d8SI`!Tqy>=`#5%%1`BMhs@usT{M&ow8v2%@n37CV=}*` z*k8Q4jePhGbej$Lv{L#9_w|_Sw!)ssGS?P+8{KCsm>z zrzDVPwLaq2whJ|j`ny>wh?^KA=m+}l#Kbk)-JwXIusFW`mu-`cQt!LF5SQ1|{Fn>H*suw~xb)@}8>Zr{?jZq>4?)pNJi zTdiB?tvax2-ulYT>lRdQ-&l(}wj=%Kn>TM*aOakJ*He9SH|@S<*{wCJ=WVTOxw)lg zw^MIR&D_0pRkz&OQoCBFSw4SP{Vgk7+t#kz)lhZw_4T)`ZmC(ccUk={H*MRna&PU5 zJL)yHzw=}lY*6v+4Z~e}ensuvc=e6$GR<-Eh z=7u#1_}N_9vSIG7t$PxsYdf%Q!`vO)>KC}OI5k8U%6$^jd!l1er%AuZdiVBdLLV*SBtoE8C^3me;SIySaYZfla%29oSrt zKj_8ghLvsGcCSAOom_F_-1T>EYgoB!(}pTVC)VHAvSGoFmbhiQ{k3$%+O{pb7FBNA zuyWt#L}_>Pw0m1Nti5gXhJ^|GSJY4T60hx!zKHr0>D@|d_r`gfZ{Jk4d}mx8sK4c) zl>2g?wrYC<{l!zik(y?veZ4uDfCD?nQTOUcczpt%=&bXXWi%>(?jjpW1mM z-KIV360~RY!7cD-8|T$0NViL-tG#_o)xiX5+hp3+w{F^C?HWezj+|uMA=kNyWV&o? z18id$eOqJO=f+yt+AeFDdTl#fPGz-hI1s-rb*<9>ZN6jcu0`$J;HTsINi*GWeo0Ta zZ%frM{>w?%O7&XXHZ1?Y)mvR`6jc#Q4w$9}I~JCO-1u zIrpAB+`ZkQgTBaZc4qH)&pG#;GxzVDowjiO?=}}=dwsCCFtU&|c4E2$(&4x`y@4%L zvhRek7|RoWjtU$2c~tm$x}1*gbXs{MNn=GlU+VWnD!Q4mbn<3K+v((u&ZnarOQ#>4 zIa0}U^6B(*@*C=RRLr%}vOhDOG**x4bEM`Z?cF6fH)UsF%$Kthi*){gb!mP^o+;iQ z!ToM3>rNkQP3Oz4N*(r&GGsIo8*4j2t+MBNK+>4 zX84Y`xLD(H{iLji;FJn0FgH5i6Xv>uJ>P)l9ApO1oHCn2J{%8 zUeV|W)N6&!zMW@d`A=Q#`@H{)fiIu=YU}mquf4E&Z2RV;I|LW%2I=Rx+)`oGZ2wwM z5ENKZ%Iig^$%R$biEhb_bXZOJ@$##KCZRzGx$-jn%>?^1ZN*&7my&|*Z+c# zq_4Z(23ft50AtKbyE-net${C$) z-SXFx9wXNE1%4e{2TJ-RTubHj=UD7Pz zNIy)Vnm_#s^d7jio1AS4_~8WnWCDH%aMWiF`p2T?=gtIueucyc(qo>^50f_(y<0tuo!v^dSwIWsr~fGS%;rKC5s?e2MbE-xK~P;bXMU zQMa*u&w@NP){tc@@jYkYS;Dsiu6%>gEa8~GBUcFIT@5Fb1+qdo_DiY?MCI+VUmc$W zKVU)mN;j*W~E>@1S{w@^yQQ8m{ln#}jZhXi&~}X}Y);X)q!FqK2<% zxK6*O;W~Yh#WV(PPnaP+6XdhaIu6$2lmL2vqhl5o@i(}ZKc zxVNiPq)CGFNad)pZI*D9Pc9KGX9+jw;}YR0U!RXRHJpA@6LNmna9U~vexC*b%F*@N zPtG-QWedEb0ITAIzSa3v;ln~YykkLkg|E^4xoY5Qy}x7NOJv*t^oK>sQR^`s3u0WY z$HxtNwH`MOTus&$16S*@B@f*pt9ZU0oVVay?%Lya%XjAA5C`=mZEJ7a$^^V-XCmH? zth5me3Syq`MV9#HdUlOF5repe2N4Tfj>E&y%2eR;NFx1KT`KGG(Aqd~qM*X_tBn-Y z3p@zDhEHzrDgpRpltp5cJTv;{sRLnTJLe=-8GLn;N3Taw;8j`?M^w@SV+ZQm zCBEP}yaSh><@2l>*ma&l#VYg{fsV{a`tk9piIQ~fM8j>>p!<{&rFbf}hV8Pb!Cax3 zGzip_ddjM_MYj-Qrw0UataQY88ZMvoYTSKv9jv`>3e)XiE@UADeO4xng2=AG{d`^A zgN6%Hos|(+If&)c8wU$9;7hI3;~#0K9|4bH8X+T}bjAsz+)%uF(Jb$h7@?MUm>MB{ z^M8<3kSqUxyM7D%4js8V1l_2}h5vmz=cq>74ph~`JOg6-Spge|@d9O9Ka4W=SIAwSC2%T3}} z@wIAG{j2&Cd_m0g;U!8+nDl1-N9y0?Unc%>m4~4YmEKgf4uJB~Qwovy-%fW!*VBGL z{HFbD#E)^RUQGSpgZ6IE@1`vo?-4Vafk#orub$s3bZqvGEZ?1x{I|(5SnKb?{|^tC Bjpl5WSpJMIGWAb$QvKxIGDkjY}O}}b3$Z>kpMHe<8>f0 z?*0CHbl-c`Ai$p3Fcfs(TmS!8)&Hvc|Eg}?{_@fl%XCd+s%Y$07ON>JW0}3Ndk??$ zuvAdGSx?P?t?uX9ZrlH%7ur5MgRw(Hw(P!l@j7eEp8GGjp2O>3agU}G6)jz_?fgOL zx8vB=57+CV52|dTYc_W1|JhjIkq&kUw1;~#Skd>gJ^de+vbTSwvqLjdduqud~E;m@W}iV;lBAN!~0g93SZoJDm=39Quyil?}Z;~Jr_Q9^1X1%%!flG z+xHJX{me6isq+_y=Cph#e16`s@ZL*54L|LCE8KkiVz_U`CD0XxLI(~EZacnzaPRzM z;mT9T!_94{!hQSS3XeQ=F?@deMbIn?RX7g}o}d5Z;NDf}XWw_ib6PR}$+yB4=Prh~ zU83>+I24+X@m3u`KgYv!TfZG10lxCw$?%*L=fFGl^GBhPeFp}gp8w2X=;ZP6dAyo0 zy%nB&;zYQ@c`1B;{yESr3C(GJa(n*Jg8ni=a{#L&> zypQ$MJYw3wqsxJhJjePY|IGR?zN%k+x0Lmty^me}mqONmE{$D1pTw@FfVLQOZFo;Y z|L{Mt{^19-Ll^$1e)Yn;?9lMvB!q^)#SUF)z?wRN^^g2$GGK|J;Ro2Ee|-S7f1MOM zvwruZG}plMp=&!|GtDKjz6*ISfPa!@7`)P$?~zQ9@gih&LMD&K1NFXF|MUezw&XF#(m6{>A9LR&l5^r zR2*`Cp(I4MA)An$sK0li*R#-{3-L3M?CIZ*{=bRqYP>3O--GLZT#w`Rzj1#G*MFn& zVXt?eO9oQh-fp#C{a)=qXw1bfH=q*~ooN~c_@Jy4Y&l*IbiN%|4_--lr85@| zJF0B=1=@Tq9jqFdT*B&@d3q|Yy2fndZJKUl%sviFo-8&ev*Y&5BCjo02%Y^eA(JHA zTrY0U{AY31x3feKdxRx`qbWqWprfgyGtlX+$DO)%w~9Ky)^#+sH3r#&w)Rfng0{yO zbZ&2M+tMDZH+0GR;|roU4ehOdZzo&O=?iu)5TeyJbal3PJ38vTEiGHUolU+#U4yry zvrb|=eQh1>0e@$pxv*estFP7D(!8}Te{p+$-M=pe1Jvag71R|jDX7cO%gZY)sKdX4 zMR^bBFMg!1E)Q}Fx$XinI-0xzU*p{)NWjxoZSeYAeSwC% z!ptxYm&IyJ4wOqLH=q%47uZmVWA<4Ny=$hhvMm?rdIpw3#kdx$b{!&*keSGhFZa@r zduhm>mKMfw{d;N1=PY|SEn8$AE8ynSkUM`LqZa8k*kCPa-?F8{*O|xa>e?Q!i(*zT zU0EkEb$QsTHG4aF^p4`GCGwZMDkr;9WSRq{4ZCG)S;FCoRxtZh4j(@Xz5$zc8{e^; zQBQF=P6{pMCBQRWv3`1T1VKGmnSoo6qz4%4J+mGBmNA0Qvvwluif8-R0T6BNr|A%cs(2m7i|VV^ zRg_f}yGybE$2p29bVx#^i}Bh*Ab)2NzF6AT=ylWSLUW6^v$?&^-PzvW(lO7CneFp- z_}ux03l`0DucOO?q9v@PtGT7o-O=6A>1%bjc6D^R8+}`v+kEb|RaGn2)U8`uv!Ea%@%$^auznSA4TT?2tZ?VHJX#22T^nD+V&ftMK8M>U#?6PhrOJ6RI)@?Mpp{_Hx zsZM8AMm+UNbsM_1O^q;p1&*2OY?^&eDz_=wm5O~f#)nZ{$*@%hZY#mtv{dFt_bxbM zHC}cXXkak>=P&zo;bpe0walL4VC3CBHry|k@a;I-YQ<|S5GXVbT|buBYx4RLu4RNk zA-@HH*fJ*^Iy^}4VLpoJJpBiAPG`PR| zvW@*BoAvIqN7!Wmi0MWr+r2l@&d1%Doyv~6*^jz~4D2c3(O+*66G?(-{H5JG+f2OZ zJvPD5eW;+!#Oth+9SYd(Xu%=_2h!NkLY5n}-8b-NC5z|>lGqyu*{ooK?Ke~2+|KJm z+4iEKZX-Nngok{=iIHs75++027;jcENsrQ>NMq@Ni3t;_|G|Sv>@3kUPM<{dz~=_- z6X~o|^iTABnhARJ4}I839Dnr(oc~`nohP?1}zi4<~nWvIqDn3b2Wj`1qpUyE7reJ^+Ax0REQ|utmrt zBZUILv|D4Bi2+Uc$MS%lMpdSL%=vKzgsQ zl0Rdi^lyfKfX@i#&ZKn{{>9tp24RljT5U4PH<-=JUpNdb6^wtL=pUSUB@)qM^mCk~ zzlfA?X*ZS$n%L5xP5i*WiUJQl$nB+x@rnYo=5Tqa{i!s}&v_Ac2JLA+m%L|C=RV1D^Qz3w-@T_}7CEl6>6%$-ja&8}}vU zZ^1cpNSqjc)~sm#bv|FXJ^1`Xd(xL3{d0cavFYVm&of$ABauDa{+Y0Ew1B=?)ydxQ z56tIWK3{1*|00c*yr2nk@?YS~U-+49F6yKEGt=XeL|qCNSe z9eC+)@NejMW>mk*-oyae`yJ9Z-J?i-(|C5}pF^e4Pp81k`8- zU#{5v=JNyXrM}UBMiAP90Fi-SvHsHh;0BcX$}-DK{YBS!oO~;@ixigVQ@={^sENAb&fn z>7)4*-pbcw%)jNZpD1GfS>qwu_cRg6^}8rIemvQqkB@j!o}9~TDjxLi#L}2qYj|nMK`hl zf1sP`wChPMo-iKrv(fTP+SvLe^8@5>m_PH0A5~C4wgdKi0szVf;6D_9Um=I2_8*Y^ zM`H0E`h+}|_^un`dv=_7jvSawBK4~WGGp<2XrZ<|V5i7!i0`^i_T}*%`49XVA03lF z_axE!TAr)~5cV|tAL-v_jL-ICJCJWE`RXa(aX5c@MA%sj$o-e>b7TL6 zfAf6NA@fU;hZmK91Z)nzKK~DuKPB7V!E&a?!+Pe#Rzc*`d_7CHQMBRy0eK0}*Sicq zBYBH@f#(AV|4H9vOXm5pgXhQg+@K?Q_CH3lo#anw5Bd3ce7^9yi=0p7D{oXmKCzyp z2zm|ep>9p(Gl?0&WYodX0Qo2NmxAGit-Q4&oii!ePMs(`~e(XH$&6mU&)zK{~B6|{XkA8#76=BFzq)S z+<%gz`;)AkDE&FwZ#a^07cvxpw}++eaJ>QBgT(*y1-H+qi{J^9lv2VWmC zUK-KEpE8a9mK)4+8`~G{a=U51aeD5LHV3RD{1M|*`=~#Dn8uzBb0mwLiKG#KMZKUG z?UN|qK|XC8coMww^)1bX{LCHg5Avr6rt4EcA8}zm4_Bq82eS34RG*9qgz+(6P8RBX zzlr`}-%N5BKg|coTNKR9jK(u7Kg548#=Aa0w0+^Ca& z&n`qA=>l~~@|>srni`FrRbB$jABQ67l7Hl{uch zg#2+AOC|q?KaFE=R*^rYQVV~X-!9_G7g3{PSTBZAVk9a4%p1?Ri1b74UWtRg- z1t>PG7bh|PE@2OU{rltCFDc%>&d>Lher{PF?AQ7FLH@?q=hG&8TJtkgJ_7sg4=^Ms z(tg(SjLYD!>zIS$A^007_hY7fhw>Re2~7D1CqnRQEb08q?y-i%vZuk z*XuF!6+XW$`3h~usR)1M`w3-#*c0cEnUjHyxG>*OeTw-C`9s`%WyTo!ihIV5^OY=~ zuejsqE08D4P4PXNuLKSG#^`(n@zk9a%~yt@zfYczWYPM{<0bmD_(-F^TKNQIn`Q5F@Gt+*<9KSHW zX}1fpZ-^gLrtIB~+~CwHmiQJs|FQ0GF@JAjzdgkm z--?10bcPSG#Qu=xN9Kfyu#nh)XJh}JHGw{#Gwi>o(SFeo@0_$DiS56g&N24iI8dQ- zJMpgSHum4?SS`Vz*sr3!3-#!J6#3c&<9QSC8OHM{^8ZZAH^u&w&xg$D{*&*&$)8Z7 z@ow^D=0=_J)pHA>PnT}cC+wfDTl8t#kBpta{sHIv#RPNyiu|^G@2$l*TAQl}5h6R8e zx>xRL;GPEVY2cm)?rGpt*8m-!Bh(JyTMn$}7uSCO;g2F;eYvmtx10X%AAVjr?ynj@ zI{!D3gGFq1-_yOWT6f>voTGhxjP=`VpBs7qpX(xD;}X%HG5}4R z$NN*`y_R4{y(G0HkzU2A9=le)X|mu2)M%mLG{zNH0!Zqzj+S`%sA665+fy#ji#t9Z z$-gLIAYT7G*Cz^3e7wlDfQMq@wU6?@Ii`DMgLw2&dHjs6*0f|DnvTLagzQO0^lZ>+ zlE>l?oQm6$Y_eyREg>@Fb~ET*PZD~M%1tB1!AIqG#m7hGb_==V^%0c|jF7BKG7O1g zkj(z6NM;}HQDg}dXObq$8Kk=rXQSt!nRAPgXg=@txF+sD(kmFDwaH|k7}6N9Y$$D>zyfYh-cX{mYbGS{C1UYSL#T{g?w#lJ0V zSi8*h?^xPDf~?utpT|qP^Yd) z*EDL5s>k*jHl0Q(PF-s+c0HbQbE8o%GL~-Qgl3uxrqti&(a5pbWJ|a)_j?lV?L+@Cb2Z-Teyj8U_M^e4F}ju8czxbJht}Tz&~?*H zO&on%Xyqp!y=6aS%7;d^kM247988F=Ahi6TSn2Pyx=BBARz9md@#xj&j#gaI-_iR^ z&cFFA@AG!H_j%W^X+9ZuY$DdjqTZfSJ2nqIC>FE#Fk{qHv-WLWUsZ;^HsfdC+7);o z9fo7Kt;(1q@Xty-;xjBdlC_+p3m-S(2gLW|#P&bd|LBu(E`gzcUE}@tDBkFMXU6^| zUZMG)xvkc<3Kz~7Xg-N@+KvO8IL|QBZuktNMXI^w+Ou2$@R1D!I)f7r`5>d}vMA-v zJpU)|vwb$X+0!rPy!71{C$_Th&Hc@9H`xAa*1Q>cKbijHgK4Q1Eh!5zx z^MiR#b^W(FybpdfWFlI^;iG#H&*|cBZK4;?>B=y{sPYu1l#Piey%>?=IbE@Sl)u!2 z6*{a^1_Wl*wK39U2j;-x!Klfyu z_D3!4RnOM;GlW9?9z`IpCuRE&WSz#R@0wWKli#C@QTz|cDdIM?r(4xt+3_GcCYXni zhH}O-#nkw!owEPqXndpgijLY7owYs9b5eoDO&KG{MCr*9S87i*J)#) zk^Q8{i*{%JSX>v~FRod*ko+bh)c+HJG(L(6HF2W=Qpd(3X``UEXul`U_+OFjb0sB> zPsO4>=9BF`mhq3u_R4>$J-M{C{US`>r{j$O^-Ul~ar1Y^tm7*`szm!p z|Ecj+I|ZK#K)m+v#A#1c!rGpGEDBM_4>Ozee_6Jd>_;zF{jWj${~AYr)&8<<2{+~T zw~kLfK|0-BF9|J|OgW&ss`doF-FCE=s`+UdE;?V;yfuzTysCL&ZKvuU=^(~-93YquAz zZub_Kt}Ix;IxR94ll*-^f%W!Yo3P5zoSEz1jPAFJ3=v7w-*WmD_2{QBjhZAr_Lmii4PdA?1& z-xV7QmUY#)F3AHA{`!K&wcAQoZEM_6)3I^GswY;ouF7vLukC7F-m(K@w1JQEb>-{2 z8VZ(l(|C>RE89ydc9hi@uFGpF-&z?Rw|wg=Z}F1$rR7`KN}BSGZR=V$6t3HLQ*;IS zW6>?!)EaEsR9FRlw3d_1TQ-sI^4rS_wyyS;tjgcmx^?Nc;+pP_>x%+$$E?_~VI%mi zFJJaVLw8X#;1&a zTK?NA^s6fPwUgVa`0B)aX)S(G#^-I#3+6GlGv|(e zd~(hoxp#i)OE6CUf_!cwwHRwp)G%$=bZuH{HXh`H-;Y!V#5LD&KM(iDJ?2xi;8Ize zdcWs6{jmMd6aSmzYpLIGz9@Po`bJJ5-gb>aPh%Z6(7&$kmphM)`-c9aUDk>22MUMl zNp)|bn+Et-IqCC&*|T=6Q2L!uY^;-ycqDghF~fU;N32p7{eRU*ZLdyronP5T=rR*4rSkS~*4+OALTx#tE=aGj4RD7UH zyyrcyP28`*`%UzljHb_&V?(08adz=5}CA z%`5FX{J((2i|DJR30{|S(3tVYeN)QomL^d1s~Pmv&t(Z$Z3`q^ZBol*|2eXsYH2d1 z=f9Wyrqy1yAwi@XfORj{?lTg)7nL zVH_$a#D%?$E5WZwxLv}Pd=t>0@*P4=l=w*&c%Fn$0bI2u&=)0qD&WQ~<1PbSZsj=S zsD#tFI8BU}WAcv1HNwlJoYcSKjYwBWxV4`R5~75G4xN*IZ@?#1Kdb| z>^kr(2Dnk)T&Z^}pR*<0D(`?akX7C{B;4B1&m`O`?=KDg82SIm05{5;Dd!K3YmAY3 z*MT3FaN@_fnmsS!4_V*?2704?Uzczz|MV0cD#Xteam(KqNw^wU`C^lVt8o?luz}t< z?(-6E9ru?KZXGvO&RZJSIA2^6K5D*5IQ64=AptfU;PQxv%XQcQkN$BFFpK11N?tdo zX6y?FxKXYq30Hbh^eqx@mG>nHx5`^4Lyj6ZPma6705^^slyEh!qTeOq)^Wcl;nr~} zZ=piD3r#7dJBn*m6x_pWj5W!tf~&akyale}!+-^@;=r&4u9jYP{;B#`>-}u-M@7MV zr2Up#;A*}1Ti|NFXCm$HsH<;o^9H&Z_6UvM72dA4hNj2(Pfjky1DWybj{G;C=;s>Q z&F(y?qq^p{X8g=!eMiTwHEZi?6$~}|I@s8o9JzJiKQB2p_2a%k2YwntirQQsfZXN2 zP9ZLSxP*SkvV2wTeEgnV!*)Tn4Ek8(>n!fX?@!iub@~V@=+V#wZS@1rSJbu9&*QCFv2JBqv$w_9(7Cd`v8x5c-{kI(+n}Z0+gR7xUf0MO z69R!f!KbJSS&R}cYY!-bTOEt#Hr|BIyn(G9bsc_R!|2}?4YW6UJH2cHe-{oZth54> zlyQ_GST-taepxNb~f;)HlKVqFXm&1`t8;`^p{=50{aQy_aKPK zRLj)U6`7z;mHCH4%4J$r8SV24Cewl_+#~N!%Q&VGDqgAjEZk7afC~ACleNMKgzKx|?1@A^9G&a715}J6j>ae;^ zP$+&Cy|w<|<@ncY|B(fu_*VU;LwKuA2)81jD9l$Y?Katdl>D;&vz7){{(Zp5v!6ry m&t*A0*^gX7wO2XxJ`8Lfo0^>Ipm+T6xRjsRpeK1*+y8$itRJ-i literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2c5121f2f0610b0471c2b91d16d809e0.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2c5121f2f0610b0471c2b91d16d809e0.o new file mode 100644 index 0000000000000000000000000000000000000000..289cf30a147307425ce75d45107a1ebaf7226949 GIT binary patch literal 51264 zcmeHw4S1E+mF_-2Nlt!9LO4KxNDdk`V1)1!MMXFXK~PZ;Q^z~Ca|lV0Nd7!I(8HMf zc!*2~6g7CA(azX5T012a`_NwRQ<>pT!_=|Vst={!xjuKCX+*{eh??e7OC0swckR8` zIp4REAliB6PJP^Pa=x|pyVqW8?X`ct{r#MKSFFCy>+vW~9`z%YrkIQGzfT{B^u@W@B^JioW!7x{-D zm4*)vBG2pb{UU<;`*uHgY~M={9(((HyN~UD=D}l6dcz}-(Sy&P*)KmceE;12!?^|D zA3puS_lHlO{`v6z&%8YRz16P_pT6mp;h)|7*6^tZPY+-A%say^dG8G8TzqBU&srZk zwr}-A$MP<}qVMEQ-yeSQnf=4pKlIA*{qOvI_~Zlc4B!9#)4)~r^*!{^vAdqxeQej| z`-g9M<(c8;wpWJt?S6Uqrtjoi4;;H6pOet<)aLIGXW#t7@IKVl{LJa$Ihk((b9G-|>(`F$yZpgp zv)}sO@Qar}GyJocQP<@!4gW0f-Qk?~NX}J#KSTb$cOE>}2Yv3}zhBpNf9os5eXVZ+ zOSUbBJr5nbfA8*NeHDGfcb$fu2VVmArQyE3_lEDf>-2E;>?H>C&I8B#TKa}h-L!wW z55BtL8Po-R?%Mq>FxU2-ylnTeIrDZN+x71EhVOs(8Q6S8>wI#;yTkYAyaUYAzU$}i zI<~9)uZFYV`uXs#7f%gmdrHsln!S77?DxJG&E8)6Ty|z+UH0M9IJI~0P_in3HuHEB z>o%ywy2+mK$bfJ1KFaHVFIn~ea`lT}c@ZPX8}LopcN}?d+F0p-H(7PW8$RgqP@dr* z_XcjHvIFlVs}6&Ip)Qv$GhS?QtkQRKx#t(jfAhiC@W>$I^bN$^@dDL%VydU_xGym0 ztx|8_$r*}b?a_}f3XcH4hvJiEe00xW_no}2^cNqtd-^`O)2rkCY{Z9n_9Ol?`j-96 zA3qWas=K!@RCVPle6Vzy=kDza$Nun0pkD3qQ~$X@-90O2c-a%f2bWDS`0Q*PXREvG z7pc1JTf$f6d?|cY?&aZa%Vw*)!`Z9;czt&DA0MW?3E`0|;LBhA-4$P#M<_x9mytsT&P`_i->PcGB4M~*|s%YePh+TjVSx~s|Vg7EOH>a-ozHLA`J zTn5fQzJa{*N{8MhZoOx1L_T<|_odJ~3B6~n()H=~@LZ2$8&|V#z_HbP`;9mzy(z|4 zzVTaCT zZk-R;Luc?<=Q+?hk`W&EKnDA|el3nkXZUW`Ivlgkx8m6Pdaw@1R_E<>Y;~SR$Iy8; zbYA8OzFq$%Rrf8#eJ}YHehz?7zC8^7z`~rqFvg8jcV_gB_;UBX;hQw)Y$UVqtS@8V zZ+-rKzbVb?dn-54_vSU(eeW#G?W1wPhcS%iGaT#rhv!Hy=o-vN4NA{VkFLl3rc|Qn z&(OyQe7-$YHc_Xl27iLNZH+ouioTM7P6Mcu@_KJY-hH`y-dqXal=ZC3fxLh5P2YFg zH~r_gBx`^9&U@c_tQs9{AwB8~O%IJtKqO;s<`aCOq=?KV}_VTtr z`xjl0v{X*}OXxbs7aDrx?aO=z2Om}=3kvRdWPbkoM^5+xd**wK_MP;3Motzaj%A02 zhEJ^Vj8r~c`^()IWIS-g_8I$Xfj#rRtb_0Wo9Ez$%~SufclhBYp;d4C-}t!pm!~Fq zMsiTc;H}}2H{ZbDX=6~xi?nV9y-;}ll#2>s*jf^~2)K}59 z{=aYb6zn^b>p6Jlm+IiDdBMK7BjJ(z!8?@&9R}Vz*8A&+mxOOR=|A4m>UW_2L~H~eL#(z^!fJ;M&QQ=-=32V zz+dFsGkB+GhN{mV*bpStwAzLVd}?>lwrioU_0s)Hx* zh0HtE2(@oDVv*W@a1Htk;$Z;s@PB=3&H&~f&DY11=4`q_RR#R&meL7oelz;8ncG%i zo~i9S8(EIIr>5`2_UgWm?s-A4pOQX3XX*IlxTLskKzvpsMzD_C$LqMoUKK=LV`4Es z_3S_yl6Rx${BVcP6W4)$LF<8lM^WDxv;LXqfx$X)g!R1qJn&f0eqy~&nZ=(HpZ6*8 z8L<63@Np)-U%>Zc=jlH#w;cF$wKx0bAEDRz;8vffJg%Sh{tn80(Vl9L^vgx(*`9~b zvwc|p-+xLxx6?J}*}gAA{&@Y8V~XPG72rOp-JShu@_FF1&jaI>$v$Bl(T~n6)*H_g z$MQFQN<8cH&*xdL^*nK`56dTeKTR9_M~`V6KBHZj6wa$1r9ZMTmi{Zo(!-eRaK81K zWZeB3%VCcFO!a*U`qlh5vQ*vakCIc*r>~cvr)@NEpU+%_ z0p`4LSSy_uPR7BLuoL@K`!YXFK1OTtAHq&rTVee_f8c*4>&V|Y=bsbs&|XAfAAImA z;r0SY=lJXseA8H{6c$rh@2g&X=zHxuU;1m0DpZx7AdtMu@u#$t7^ z+2^a~46piItsldB5zLfAqy&<8v&sys#VP|DO(*2s&P|=MXsmt%j`(|9tD*IGq;peS zV_eN`YmYU}ZQD9Gwz;`&V|$uz*kb9eb5jQm?X4a4F*P^V6pzg{rbQaI#9Hb*J0q5i zHMMoNN29SwOG|4c)*fxx6lscgG)0?Ro7!UaEv@a1O)brBcSg1@(1)F^&27!C_3=pS z!t~MnNZZzp9g8B7&eq8M$}1z0SjPhE$kv6C=C;PBc%-wTJ=%0V9o9BAKAJE zY2R>HQ$s9LAB#kmw|8u-jNZ8Mubp2Rsk?1;{kBLY>@;?L4$SD>R3B|>{QMXa zZR%`_ZiPi?rD#*j=OWr*NB!r*kjADBTkec(jMld{nbwhv`nfI+x>-YgOUs?n`i@Py z({)7KW9_kR9VkwD?QNa0`nFi*=OBC=>N{GSq79#mW=7L!TVz{mrX4zU13DR>gEnMy z+%c=UpPMesZL19$uYv8P;v9>)T&Gz_n1#nD{QY7X^2IU)3*gA_)3R`k(Enl?@}F$= z?i#kJbtL2bmm$CZ=P^7;-`d*LI=6k}#?Gc#rHbGrDw4{%X~j(ulM|_o)VDO(ck05PRMn>IGLHHGTdtXX|)t^*e@cr`wiXwsbd;Am4F%Ff#D1CfB z+d2M?ezXb7UFkP+ry6Gk)N>v+G*S62ou&4Iu1?XJDgpY*6rG_qfv!){ezgkp_7v?? zVbE!LUe&MEkjKmNJt_e@oe!0Cz7LyIz8Oj7U5zcNh|lAnU#<=mDAkWo;UqK>J_EE{ zhK>I46ntZ6u@D>1VSN0vD|M_;yqs_HmshGVI^bG-*5lKG&ry8T9LU8dgwI-hI`HYi zrxzdZ4XEp?qn;iFP!dqJ9Uo7o>a}#Cr4Lv-VCjBK7hAd#4Nm0=pM&x$+t8FxbST%I zKa}sJgQ~a4#0{6PgkPvPGkp|3qSj@)1HN+RADCw3k^WR)$BeXm59k4Gw6c9Uo*dPG zu`@qMRbKAQM|sc`TV!vJC#eQ3ZR$^|wR26ru`j9mEzS9%c}kChOea87UuAj#G+dOT z!RS2gYz_2xdm>Q};eXye!A3rjB z0r%LCf2JAfeA5nj{5b)Y1AfAX<0~wGGqzV-q2ry%K%$tW?~m>GNA~+e`&~B|UXibMN7ZC9ykqTft~yqrR&J@aca0`;)u{qCFRtAJ1oZts z$lq0mV@jZr-yQdBpKF%3C!i9Vz@HZiR8QOwg9=shjbwp3QlP#b&+wz`sAMMTPqg1p zZN_v@JOIs{`0RiiAHZz`Y5WVFAUNHXuYR~CndzsX@j!mX#hMOKkP^MYZ}O=V5PhMc z0~GX@F6QLkv@p9@tb-_KRG%1Ys_K-dlF&%5q zopE3@C4K= z1#0^?uNRT2)U=s->ZzHC*NiZcN`Z^|tD=6=FQL>!S?c&2#Ct}v6uI6Y>VSRI;#r#Z z6OH(u7|qC_?v-5paF(Gnv#5Tii{e=s3G`|751n5W$DOJUdRp9qTXpTYXZGcr(< z(&dlB-tG*q>2XHCsz)+?Df-0%wF>s>@yGMAPl0|h>Nm|t{zCsmf7XnSn)h>QJSmBr za9cbCeUaZi1L`bAjWi%49=t`>CLS<{Q+ssJ$V{rMZGH#Zb4DPkuCO%P^WqY(pGKrG zwdaviRaWB5All~z4S!86@_Y68Q<%t7N9KXf(EjrUQ2~uVGhp5VD{pVU+CDAApKAZ( z`3`;IZ^-M8!}}(rwg1xgobi#oX~m}fwLgzPtS&C{W{|~RikBm6R9R7G7STyB(eRfy zLr*I@zEJ)}ub;vN$;4Y(QC0@EZzkd4->RZaAGN=ZKl1m5#{C)(e;!(Bv;$^ze}?|9 z{tepQpO>M3I{HtU>@TJ6{(|}mpZF%!3E&^_uKx3MzUUuo-2DOei}?3Xi~GlqN9gN{ zf7I`aZ`hA`)$z^#S=E)1p_|L|ajtr?l*)%GKsz>|J%L}DT~5wKvh&A1A|9h69`A9) z;|9cEmxw>q=ZZg!7q0k=n)u70_=Eo2xB1iYhIoF$_K%Js6K^j@gEra_5C6PIrN-CL zRy|+MjT`OtU-Z|NTVlD0W+XFTQoI@(EcpBD@vJNwuk`$e`Sr(JuzV=ojtjw<=Nj`- z{*JthskrLFd^->Ao10YM1xf7-d-s2}Fg?FO-|jfPX}YxT58YjUUXN(~dN7{R^rz<^ z>OWmM>G59Ed4f)zqv-NaM{_6W`Ih?gEA!Nkw z)zqZMv!`dGKl=3iR72za5$caVUk%ky^ONq689vASgZ^md2VZJ_Sc(h(z|;H!{n4Ll zbbq4x$n?jW)O@4+q~cHMall>QN!k-2YZ?+37VaOcTG}DES|W<^$BX za!c68GxASUd%4$pu6Vs9pW@v=cD#FaJiFox>(!N8Ud^7RD@6RiPw|iOfaB#t%umoq z&wo}QJs&vY|96Ua^j{qx9N(~);vpU16W!xC;-M<)3mXra`2qg*ErtFa>y3YJvhFkf zEmfJFR@t7Ao^Pw7S$6(GdDEWSKCIvLeD|jqkEy+p^!g0)vr_VbUlsK(HS^KOI)1X^ znT*%%6+nHuMw}z5@|mhInnm+(B6ASKJjz$a{Q;cgz?(E5ma2@-3=_XYXoW`1n^C{R z-=(lOm(DK|W_+Kg%A&b6Jf!B+OQV4dVqkrjMeB`AqZt9xFG)1wuPW-rf)L4L=(1>* zK8MP(^DEg$+3de6Zx+!BT_5QiFzX|uua0MnH|v?IVn;skMNgUPk_);1KqmDUqd(d2 z)#ndP7sYeD)F8=ztXEJzy*`B-^?G-|E)A8;`OdlKE7CXW%}Ln{y1UEzm)aBl=)NZ} zy}sH&^I2wkJ%#>2{?{Qy@s9O*Syz^x-{3F3zVb#h((R-D>(}Rd%KCF6&i`m%A<65n zXqJ|*=ReFpuKo%A_C$kvL6a;t^0#lx&PwJ>dU(r0F=)Z?dK?_+$} zKh-h+mSO%4(B>Afl*oTY0pq_8qfbw9cF;BdmWcVcB-gYb#RJ)g_|O4O^eY&TvXj^; z+D_7S{$s_3h5aDRd{wHRos9XHT%_kK57tvV741$L#+gZtM_ zXfKl}p0R$9WBnV3eDcrn0)IEHw;bmOW$`==-06Jy-=Fe7+OMdq$V>iZ|Chzlhd|?e zfaDbx;e21qU!rN$H&M5b%}4teQgI}-zmA81>8?aSPyGe+0iTbczZAtM6G`|tK`Ica~jc+~zZ4ByQtV)rw80u!gZ z`g5LGZ{|7Ho2c)xKu%6SiktpAPd%HVatcW$yS_wwOz?7h(0X~-fHFHw)F0P4{9(pV zY7Yu6ZV#8gbbGniM;K4(%pS@31M!uWiZAUC+Mu@y`-k0e+8<1>w`KdL{oSqo<%&nh zOXoujXjglae&>o;#KZonMQMNQ@pW3d{ZSs{X|c=y=NeDx{3<&Mcc$ZI`!;8P@Ttd& zi;DW8s$)DY3Xn_ecq8=Aj}o|vL1X?haT7{Ai~z$}RP zaP`+$X+7bHcbqS!@em4p>U@IsOFnJEite%#Wby{F%xJ^z{w1eh-7z>vzn5MR8xC1C%e0^6=NnEqUqd z7dT(VdRL!+yY^pbe4zOd=B(_7AAPD%wAQf2PGJx%{X1lb%LC&2N82`b^UP zo5b~Dy`QA=cKvxxd=Z&T8>%>e&BFP}L@!M+yq>uxZug&cKH9%)qN4Lb^bf3e3e+|6 zJbk{X%Wt6dgI8}oBR;VHc^K#Wm~VSgfb=;|*SGZLV;WEO`uCc6P%lSJ`FwM}j`>rm z^*SHxeV-nG_4=OV7iQD?4g0_PdK2`?ruCE72maIT!S+9uua-uaBo`qc@!>^#W-pCi z&9v@+7k5n$le|K$?;8Ko=pxz>v-r}$(&z%FHNLEChTb2Tq1&?n^1Tpk@y`?OXIhWP zOQV@deZFSu&siD`Fm3a5lC++)`k}r&rcHhE{BVNu&H5ekuT54lUe|XG(^g)UzFwG? zw~%R@Z?1o&^E34RFld{fmDKB5(pSsRX4>ZG(&mCg-w90X{E5+mW!j!0y`P2gG=CX| z4(mU$t9H7MufhbbpFIqF7ALP@d1$|~c(n?F*5fhgqIhw>_AiYmTA!yiOnYQ@_Ng5xtKeYUbOxygTB+Z~{e^oMV^C#&CtkU`Onb!G* z(VAk6*ohjNPa%I|*NW-XVP*siOnk?ytM&Lu?fZ(pzGJV~;`*cZM^KNq%V_)ry(gNJ zKA+d)Z7|q_0gA`ls;*#={DtvTpMPMybnWkbJ)WCBA2IvYXb$wZ)Oh)L!1TAV$4iX2 zlWja}{inqz>+vUO#=}By*YfoF%L&pqkfFy6u(!_vaNA_=+4|a4W%0Coec*ZZ~pE8aQ*h}$sF~^7X z&m|lm)*qL0JXreE##gY6{G;`e@umG!UPk^YxB6AZ%a(^h>;8rHauxdDdLn5&!g_W| zw4#Lk#qzlSm8JS$Sy`(8>3AtGA%AN5h(GkdvQ+=WdW`x@dORk7IO0Xe>p9}(d5Zr} zA1^fj>-HWeUTD5@#0&Iw$IAjPZ{lBXNHCuCagQI9)9tJ6nIhVE>hIXT6+yD6A{Aeb z@x!!lknG|1offCZYn57_YOm$Xsl9?U{$l-&_F3+1A8K#1SIgJ-mX{Pd|3{b>DguJ)ttQf?3Qci3aw59@JU|H;wUV`)8kMvrIu0PxU)?nsAg8vZjIG?Mb9jJ~CAJX{bCz{q19UIOP9RM9Pw7&jB`9XvKfbqd< zjelQ{Z-MxQ7h<_ULyV@gwRauQ#^3_s`v%=*=y5ZsQqYtpzn7u?OS7L zT(3g?f%xPKqmOC7Ks0S3u4XV-vrUZ;i zSdZ3%Gt6 z4+bfo_>(jrT{hg7f8QQX)1pM z<Tt&+)p;>QAAGy*&fBR_fnEo3kFne$1ypPBjt&bM^^-Y%cHzlrvPFn(fx-beTI zB<=a=S5RN#&mRZ4M_cjn1W7oG#k$qmuAGf~VPha2fWM83r-DO{fd0|Sry)x*20c}rX zzH+zMJhI2+lRd~^x$Cr}eU`z0gF?O+?=3+}%lFdt&2z{%`Qyqzqy}92XDi*`)AF-Z z?LW5uCVyP{xNXiQ{~Y=wUfupFCI6)LNB(I4+%DvwL;tbmr}ZBtpW=7FkZ+#L(*D=+ zn<4uruHTOrzuF#?KW_Y%s6m&%ChPs=bbV9w{;{L|v0yl7eI|e0`c|vcuKJ{XV~juN zt`Ex}=reA8&%3U#tlaitI^O%m{ldQoGl-@mB=Oco>}?Pzy07_@3vog_|(hCS6%zJv9}`s z`@eku+go3M;FT>`z5kP+9=hbWr+>BE@9prdzwDn#5)yW@)+;RWRGYai_eaU<P!q0@F;0Cnf1 zu5`i)m&}I~(}^CqIahi8!0q*U@I*LO$dj9&W8Spk^IQ!}ym>kKKE2lWc|B*LpSh1j z_1hBF`7?9qbBo85?S-VY0-lwBIK}uF{sG!s=nsCNREQ|gS#=i8Lw{0Q1u zI_UFEE12X3#;GrE>ofWHjjC^xE8pjp`rPiyzYjil+p~)D)B2@+28BpUr|KD} za=bH+FF-%}4sku9ga;wn50!M%{Xj3-NNJxh_0jxL{H$MuY-ATNx)N0dlHnH+i@ zhfUkj5*JOswk(?eoxQYYBW+RZ;m(flj4!E4va4>dL8xJ>b$jnvb*NunxJ>a4GYey z>e*l;C`ic+T!<$Mcb{zs79+dX6W`w1dGIBMDn#;OPZ+PV&@lZ+m!Dner+m*@+zg3) zZQE*ZPBlK{!yG-@nRqmHKefP{j!E2eYo6NG0#P@O#W^NM{D=|19E&MNa%9YQlYS

b~o;2S$&wC*Y?M-E(%kKLMk?v;7!-js0jB7hZ*-uO6Mp9xL={=`2qV3_0qf z3kl>e&+Js4z682FwUcgBS399BO`l;SfZHkMZ)z*VXxg`lOt`V1Hn(-6cBOWiu~@f@ z*UUm*p8w3flJ{Z$>-LW2b1(bc^viBPu&-0>7PdL~R}N>Avg?IImuvpYWSOO01) z{T1mScIuywjr$NAXml}8j}aT)`kfAA>UnN|exfh6J&RLu{vYm7@*{Le_c7x`^6Mx1 z$rUGK_<3|6bNMm7l01yd7~BVH;N|K}Irh-JRGn9=$GK7CE&`fj6cCdTqn`AN3Ij?) z(Kuwr{SClmd8Paw6i#DE2aiu;NC(H<;g|WnuKfE()d4NB#yo;?unRzW@_K(t0`_b(K?MLURs-O1Y;m52*MLqC^$5oFjKa+fw z=khbh6@5GPm&YzSxfGwu<*QNl=oohUOUQsfJuVrpeC_W%m%m9zIs2jKF3!hVJ*;np z_tG-G+t;67c_}r2&N9~OH;I+m&ZaL)GxG}I7pY^z*Z3v&ExKVQmpQE?|X&{7ly%4Wp#OE`=yA%%!tj8yT zPcJ@|*WwtTUVQrTQMgg75})<>B=GUxfV%MR(hh``-lITl(Ys4&HBB_dJkbX%O}&-q zqn4(gr}J?J#&~*n=m7SliKcv_!-xoHegZfArD@zAhZ}5AMAt|7p%VAINh?d;`H5NX ze0ogEBFX!{SOA|C- z6r=qwLVQ(2AJ}yOpYh&ZN+PJ|)AxAqC0$LKNEju|dt(eqF3{;(@H6f4xxR1;1sU2Z z7wjs{A1&c4ns1&2a~8PWPjZT!EPZU=b8YS{b!M9PT&LymsI>vA%yIxBRwwhGYdBG) z)O)TipYa0rsYwq?=(ZFk!+e~J{a z1qefRu{!J_Y7Z%s%B`hY0cn%@O8u2sii<3h>r&%6TYi74q&bj&r}&}rVWCE&ZSoNE zC{g5*?y~&#l!@De%~$&G4qLv=ve?J6JnaE~ZH)4JMR}Gi%g-_n=?F8xe!pbP$8E** zMc4Oad5U|87^Q!$?Fu0)PwLO@_%Jd^X4vBRYm?|oTtAoN_TP%a=PJ)QDo;3Bp87et zK!mBHIYXy~=8MV`CSl0b-$4{6nafigd%y!1Rce#Ig6g1&{R2cDl_&YGWXp$LOc7+r zqw$6GuMs$9%P(;Zd1JL-2lBB*p!Ty7Zv=&JG2aiFhb0u<(@*xhK%nVnb*dl9mGw%K zVUl2Hve*=>4(0C~tNx7!Qf86nnS#;r*JR6wMg5JoJjXAUr;rxzSBubj?;oT7*PBhn zA@il}r$a8!adZR9ccXk*)UWO`1v&n>elExS8DNZ6zIcrCH0#LnRbU`gI}krE|5w`b zMvL@c>Hk|%{!hlRpUV%~q6kyn{<40!U;BiAhcR7~IV!dl@;8?!Hu>_il%tSN^-ohx z`mgP;%6>cRBVD;)qzkx@!nT2(kL(e zI_l{+<6=$GPN%5QS2s1ag&Ob+arjZlt3&G}5ju`U%J018>QJ5jEy?mPw^dez7F2}h zFIfDQiqOWE_WIc3NYkR$g&W&uSA_1o5k(p{H96q(HFmrL8(P{MHb?MFm@UpSkl(ft z;+xxI3l~7F{(Z@`Ov;89ZHrg~18TZE(gG#ZxV8l;8MD{J(3_e&TkB&`uQ5cwc^4wv z+uK6*(L1-$4`OQiqDtbLR1u$IVw)&PW>{Q6ZueaRUmdXz`*2n5YrZE0G z>NDljSsLSz-`3dNdM$?y3vlE&Hf?QgXo`^9*EZnyIb``po#Sc_rPgeTh1xfUHneYP zYox00yam4iR*xV1L;7;%SBT|Gj{0Klp$$#&ziEO{w7%`mrqGu!Uod~sm5UZHykgN} z1OXpkN%PnBZCl|BU5(Rj_2v2LdPd1#06i9*BOgIV3X(8kOQ8+hLi2IP<4&XS^Od>?ALCq@iXSiyIf}UPI&?zN#lZ1Htbi+y__Sx!0@8VD z?Jt7OoE-9X;OJnYNW69t42Vf{o%ZT@afs3)o$_=%=>EgCDdp&FW2x)Tw<=%c2+C3N zb-XRqYUnO1q&o0upOpV}3?ly2MPggUf!Fq-X;i1})k3tiHzj|RjZ~3MX<6y^;5_=~ z$;#!d*(^HaHs-IKgd6FWRabsHw%D$-5oHU;SG{1Tr=euFse?LM7 z$>E`v{q{qMApU?AmGTedjP<)txL%J0$BFc9E8o0i(d?rzWUTXVBZp>mcQSlEW4kjz z;Q_f(JcA$24`X3OKYhW8j|@JC*{Sbuz%Gu(6cYzG!mhFSBgh+zUpfxHY8-z3IQ8B; zj{G3(DnfgO@J)&1^(`EarO#iD!{0HEKI_KOXZbk%S>wq6$vFJi$Eg=^ohBjQ-e{p| zjx_pl5+^@o;A%SfMfRA!ZnwuXz@OycKVgqaKAu2MrCxhP@}2xuu$L0av8O_sUTgVM z&K;INQ^;v_$l+5<612m?$9Y&P^*Z?G`Erx>+&Fyw5?nw?Pa2mv723@W;+yB9O-|7A zuLR#b?`-%bmd|!^e+gMW`-l1Fl5 z*z(CP8iP5lcJMD?CcZa0_%j{+F3TtV*(S=~Yx$Q6{vFw-F#Cr^t+)J9?QZ#`XDNg6 zjn{_iM0}?{haB=L&UxJKw|tI&=07j+Ywb9HMBthKlE8OZ{852t{-D4=8Z&yI7I@|l z34FiBpAmTGpB4B6a{MQ6NwAIAh>jKaG0fA>dPY68o2L*n=_0MU6Xa11D2dqA4 z1fKaH3j7^bJ}-KB{9(TS@i>^s;~U#$-%6Squb5wG7f+05yMmU_@|j;O@N5@e^RE-* znI95(wyR9wnO`CB9EX(x&-}##zt;M5iNG^IEbwest-v#XmB4cxt`>OauN8Q9v67#KOyjJSFgY`f3LvvI$^)SGyj0Vv!2fiJoBFucwQ$Q5_snK3w*zgx90_( z`7a4PuZxZfJo8@@c(&_xfoJ}Jz_VQ^1fKbW0?&4x7I@|l2|U|%M&OzMp}=z->YFnh z;~Vn>`lf^Q_{Qs^Jj>_tkokoI&vpd`p85Q1L_9ul9F_K`OgYG>-n6(Gyjmlv!49|&-~{Fp4Sye1fKaX34GyKF(@D%6?o>q zF7WJ!0fA@!34v!n3<^B+PYXQzVMySae@5Wh4`&6Q`Mg=r;}7fUx10DpzA-;w`HW{h z^8}vxg#yod>X%HzuPmSW#RAWImI(RG$16K^VtqR7_*NnC%&!!9wri2VGk>wb^SW+{ zz%xHA@M~@S)CxTFR|`DPvug#O`L_!^+jWP)Gk?9nvt5k>&-_gS&vvy4JoE7r;4c5O zT`_@YewW1C@irmxmcK*bSqZ zbDZxNc;+7vc#iXD1)ll+0?&SUUf`L3MBsbveEO2WGykZ-v!1UCJo8@@_#M`tuM0f$ z2L+z>JT36d9};+O$1?)Y{Idej^X7*F&wRW$(iP8am*4W`coh(MwkuEInO`jMY*&fE zGe0EoJg$}rJoC#1p7pN~c;;6MJnOkg;F-Tz;CbF$BJj))3;ckMw_1T`{#t=&Khz04 z^KTb;_QM?l&;0cQ&wgkWc;;^sc=kh!z%w5|2j_|(*0W3CnV%4N)^mrzGk>SRv!1&J zp7}ij&w4&8@XUW);JF>25P0VI3Ve_4?@tRn^Y;tO`OgV_!us=& zz%&0PfoH!R6?o>qD)2l{y(aL?e_i0&4+8?v{1XDtei#&Z=ARaL_QR0CGyjah57~G= zEAY(s(?c+h@rV6jfAlUj9x^{q;Mos_me1oM^MeAK7I?moaY*2qe@5Wf+V#L$foJ}Q0?&5w&A}XR z%=g=yhB-cCR(`=c^QsV9WxJcqH zKP>R9XRW|9f0e+qo~s3(`D+E9^{f+k=HD*xyf1Wzz%zf7z_Xq$0?+&ofoDBq0?+)o zz_XrR0?+(}!1KCyhrlzxN8nk{M+Khwj|)8O`GmkTzgOT{&!+{R`FjPP`}KZ-XZ`_! z=YIXHz%&0jflt`}bx7cuKVbQEU%K-?_Cd?1`@o(2A;Yfj$8K?a&$*xZ7Pr`bGr!W(n*@Kr-W0!6@cDU;e!)*Y z#IU(3+Sb(48QIX>Rv+D_=0=-Z>gR53ZflOzN2B%IRD>Q;TwTAVtzpxR8oDA*KU!~a zv8H%T;nC79v8D(fP(yxwOJwtchN}4s=5JiEvGR)fl^ZG-T{*vD!G^2mH_pGJadG8U zO_f?C@Xaw*ciT*{rN5hE=nXgM@D&&=9t+P1E2ia~NyeXF&rc|#OdgKKP4bKBZuO>?(L9-=g&RvTZehW53Z`5D|1+u9o0 zxUo4B>1>Kcn&>fVG)8-C1kX;#IwOk_&AP(54V^JOY8r_!{u4u4ZMNOVe3NP|^_`s& zOUCqr*3np`rKL53CTZ9dp~sM;&GckweM@V5V-r1}9of1-A9l8)Q?%B{BdrV5NAn|X zTQ_zrg6^%6`IT2fxsC<4*4Bjx7d*Kd(NAn%PlvTl4G?;3LwmI8My*<;eZyS{wMcy| z5?S8fv8_4=yX!VZudkh7i4L>6ep`gc6gxzGrmsK2_+uX(bCo~aZ^%r3;L7G=m~*9T zK@m?rp%=EOJUdQWQrXi{CKZj7R7E(&F$b?bY`%M}k#xHY4V658^VpvW<*9x<{F_4f zyVlk#>tmX~{|rK@Jjr&Z^|pL{+{qpNOx#_Bewnzue_Z}GA%OK~b^{8JWq+dI kAlUt6Kiv|~<=OrRQL(I>O77iFfAQWi>lo6JVnmkze;>ZcG5`Po literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2d44f4edabc0f33b82391a9d8dc1a6ce.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k2d44f4edabc0f33b82391a9d8dc1a6ce.o new file mode 100644 index 0000000000000000000000000000000000000000..25c44e41bef30f7e96903b7c639be8287ec6a3c8 GIT binary patch literal 88552 zcmeFa4SZF_nLj@Fh2-XiB5Q}Uc@-_TxVaWF{^g*iLIgCC0qOb7eFY$TFS6KQhTwliL*Z7XC)9kc#efO?! z4!o7YM&I1-9e87tZ{S@Yd-%V7>|}p8dl+dy!=|$3Un>j`zPXzH(E; z|M8CU>% zDL(BR9|Qki8ClTr?8t48Jv%b+%5O$K^&q8LIdJa&2cAEPH20N#b)(^v z?K30o;Q9RCmyxD&fMm7)@jre3zLLjAN*_R7WzUSPFZtofKMntOkBmK^k=La4*$!Qdy<|(|m?(zFJFL>oEU1|3eo=HoIY)<=5 zp+)@~_Oayw^qI#S*}Mxnc7-QAde(QvNuo7+*>Xo8Kok0K1@ukmM5Bi-e+oL`@eu8b zlP`J$H&VWS#+H8vaYa0jDC-I6%^RCm54^d(8hRKWefO>>PpJAFPe0%Ac}i^^C4IDY zF$`V&2k7dDy7-5si;;13u|J6}_8YoLI%@0VkfD#Z{*I@U=jIOXJGXktPu{)DGw{v< z@4&mO*~90yFYv#;$2V}X!4Ey2gmE-w;D%q_+4nkh`^5LE1_R+x0rUH_XR#xj&-{I4 z@SA(XqhG%^Ji5TovfuEfocwQ}AGX6z-dT_`+Bhra-SDQj{q4Ca@3wv4H+bUb;nB9A zc}7qCebr!N&XMcd_9O16i2FOleYt9|Z6olHfxiI!Y2Y&89{~3O9|6tq@F#&c z0{ejbfbRxg3;cR`u$phKpvk(I~L-4JFY)OySCxl40;jh4Y(HIdH~lIp3$)*>?e0&Y;5$F zJ>245@^I^m2y2cdu)F z@w?YGd4s2#Ud8wG^ga7j)0_A{Lf^AaHNA)LZ`1e5rzl?_6y2N_ifo=4j%;p!eA2t2 z4a5GY3L`m(zCLNR@n_NNSltQ#3C#PUv;9}Hk3R9M#=o2NF7eptUHEXzcdvT5^-EVh zygPc;!+UmJ_3#|A zQ>h~};)P^Iypa50=X@UH{M?JWqEuJlRMUg_{weyNdaCKO_})j~Q%*H~0pI_ezE2YT z%#8FMdecAJ{;D6m3`-fEy|VF-A+wJ@@h6QVekrFb!P9*3G#@<82T$`gIsFoTlqx5u zFSnt-%~YTNRMW@seJ6eQood>R@4M)`_f*qv zhx`1a%rkUdJL~UidolR%-ioxr+ph5s-cgNDpMS9ZLI3D|NZa;Vv?b~vY^;u67Y;w+ zZ(Qqrw;_%FpC-n(o`c_*2A{D#S~GZqj9Y*_eu|@dQ06*Z)6*iGOXi&XT!x1Y7N#*V z2j7#z{lRlS?@tzBJiNgJJ@wX~TF2OEXc9Zs1DC>N>AY#DsO#08+FPos2W`# zLLRfOBE%QRiQk4gN_adk%PFS|@rz{LD`M;3doSqal74+0dLQV^Bz;6v!LG;r`(sT2h%;li@Y!Q z$I15|==VtaopI>7tC4?-$bWAfdKu{VNP1r!`bNs1Bl3SC4t+o6m-N4jL;pVIpDObI zEDrtls?qd&KsWU!ig?eo*!Tx2eVq6|r}S6ErvDP9j}w2A(&xsef1J|CiT?(rpB|h3 z+mt>|{C6q+jM((wr}S~+pQiLPW79uN>EpydPw8jHrvFz;A1D6bD1BaR`j;tvocQyU zK0h}7tCT)Y{0OC=9h?45N*^cwZAxDdoBqAkgX#D1cx_xrhknpnuhColaEYhzn;xw7 zXfApJbN?Hg*21S<1K+l0;O#xD;qz7xyn9#Gz`40U=%97l*byIm;Q->(Ku<@y45ZJ5 z4?KC`9Q?gu_@uwSn+?8vEgO9ue&-t*RfGQxKlIHb;n80mfN${WNds@eKfU<7@Zc!? z)n8pQP)9C-K3RfEIx(-xfjad>nXKHGc$9v=P0FT`eoJN zui%UQ>Me%v9>lTHmtU_MeED_H;EO->4E`E^+l4&TKgLG?tGQ}$6m@TY;V243nNJon|*1LvlDMu&gQ zM$gS%x8U5Tk>;21gO2!UZ-?sK*Ln7`97Z&QE1x9N!iio@FFWYWDPkI<906kL)2 zy-nZI*6y^wq?X4fn!Y$ZK=o7lIi4{0aof`4*YWOM-+=CUpnF)0Vcxvh;Di48p?{MG z#(Y^n381X>MO6ct%c=*mR;}jc{_)e{(ICqHnJ+we`bfb=PgT`|Kw22=9= z9hrIYD91C;oZxuI#SnzJrf>1c_imz$P@ln z8saJMz>z5zPpqDT^6GebCyxX#rXwEpVto<M}TtU~$MQn~Xfe#5|d=;1j}JWR`^d$=THSEwZ@%}DGwUG!hzrZeZjzM z|2TW#zrL40@a9i*2i}5RUxb~GZt)Dh{Wom%cdI>v?|cz<3cDYkk8vXd*QI2?Rind? zko{H<4$lR~cr^SFa4qmuV61D0KMA}Y*awVt@9^Eg&A_iS^bxMV!4>P{;cmp=0sQa4 zHvsQt(B=9w!#`xuQ(V7?>yi44!wraEee;>&zbAb2#o_IQH=h~)GU3e^hc^-4at7lH z@Rp0iwS>2x89qXI>&4+4fPJ@I9LD&5;(NE8K|O;feA_M#F9kk=>tdpBKQp`q@vCvY zk@9W7I6N2et9P71KHwb}hw}*EdS-Yb;ae{bPbIwb4B~-zUL4M#a{lPza4N3f`=c|% zKC$*~dtQtSuyyPsu7T~>U=CW1@t_*xLKVj6?`vbyhvUVEhK95@m^fR_@v@9g#|?hd{@KX)~@}n z9o?)Rk;O#t!V~jm;_CG>-(;W1>+>*w2HbNQd|3>8FYIn{->}Lic-glFk22ZC*`hjY zy5w^&O9`_|HVGWfqLgLbZQVUxJx#mt&AuMmCosP@bhqtqu~^yu13j%}`)@Dn+1tK< z_W>6-AC&m^GUrS4fqk7#J*=#!)#@n|O>1aA*wfL}-QBQnX~RCNVNdJ6+Z!ZhYwP~* z16>=MI=k;^zw^$HRtIkpxO+)O<*tUM4J|!~I$N80A+oStP2H^xE!`a;wC~?@OZ)zw z%7&hOO%1ybbnR>EamuM_>S`H_AhZqbX!?U_NK5OkgL@iwcQx&66}@9dG?i&F zFtg^Sj*dNDO`UDr=sLR&^c*0I=cxI>{_dWp{XOM>0P5S^)VZ&IGqvy~6D#~67(tOuJtzki9o)CS z+pO$E*XjRvD-La_CCAo`n7HfozklszPb^#ZweBlBuzPoRYfm|AXxM){EJV2fTQ;oO z&>$!gs9lX+N-^dnHa8R66{!KPX>wya{&!h(7EBm|-Q4Pcfs} zM|u3@nE1c-QaRZ%@jpPkmrZ8%eQAKec1Og!x#eJ zw*i5^Q$TwT(=|Kuo=BvG1;*})lmz%!k!F!uh{U$c{HqWl^7?}%f=K02GzI!jf$E7w zBPC`vfhelYB}x!d7om^hSP{>gFXP(e)vnj>D~hJ_FL!+}$ofnjplIxP;V?O4&tbD{ z#9Pl4|H`xmqfsh72ii&f$MIH>g=&#@2fG-hG^HRy`zbYJJfUFKu7nY5Jt~d z>=A{QEG=8MD71|(Wy@Ex>VxeaEurp1-94@QLi-MO_k>zncen3v4Q<`D>833W+tzH} zvToDHhBZ63uGv_#riNX$DYP_nRZ~}UTc~?q!;YGj+F;>Ce@UaV1m|x;0 z?1F~NiQ@}%9QtB*R>NiN1r1-tp40Fm_LPPfvLOwZvNIZ9z)owpgq_gveAchwdF+^m zuVlR%E@nqHJeNf@T*NF5&taV!4zV^37qUhT7qA@~p3UktoX_esoX5f%p2f<6s~q{x zWFZaDU_lK}ry)g2pUWs!>O!XOP|6*z^?ozv7m-A7HinYA|7_Z zA7tP==$tO!;IMc|e;6{A<`*2U-(&42dHWw@KV9aY*8Ri901eEpds&LlRpOKPPdy#AhXr zNPI!!N{OM}ygf%HhW2wDmKfH-aj(R%9**lI4oiGY;(CeuC2o}Xgv6Z^pO&~kSCm%| zobE|sXJK$8{~b1-A#kIOXA0bA<5>cC+Bi>Ok_VN?W(!R1CyZ16ynJde;c%hA)IP$E zA%Uqqg!|#J5I;75r;7x(`8z#VV4J_wa5N~r&0qaefk_^`Jc${{0akytpa*Pxt-wK! zvsnEN0?)AVYJo!>2U&fsz$F|PuzEbjL-m()9AfpG1g_+`h}CZqILvV|tKTkg9mn%n z;~xnu_$y)0eN14%UkMBE5?Jt8!cMmeEch#7p>}}N@;xfB;4i>J4+wk$?M*@d zvIhl*$bhq0C&mJ;za`$m<&(lLNZcrK`DaD?HionPW! zJHN!o?EDhf+xd@){5F4`65ITpme}Tx9T(~Qd49-8VkU9F!~u!fpNsTDo>{C>;u)Nt z#fBsfaU5i!&x!OU92c;N#N`}^*jb4yIWA)1&x`b7j*D5p#C06cV;3YA{6YWzLZlb` zm9Sok1%D;%ti*yp=-U@Wdchy`Nn*ht^jl)VAN2Q&BE8_Rgmp?R_$y&U5)1y2zF(vl z`~_J=V!>Y!k3a~z;1BxumjVm^0?b!Ao%!>NSuf2;S7Lsehv~3_vD5d%&&KBhK0}zN z>Y;c3Am%&||4$d@FTiYz;G22T=T8r?EPRjn@O_1h-;3MS`|#OAsZlQiUEh<}cjfgR zd0i%wF-#HM@3q$ePf{iXJKkQ4)QD&&UP9+Cx}DyR?@y54reE%B^e)dB`#DObg}{s1 zY~~@kCY=LZmc_nzXsxIcV+7lk#iABkBt*y9v)Q0e5p?tm`%)HrxT|I@wRvpl!EE-_ zV%BT@gDglw41MJ++Bt zcCmcwpw9XcAIkF|x6-odT*6oiT7dF1EtXF*?X1Td;$O2_?ab3a6e2!$KASyuN6Hw( z_{-9fz9@^WIRu@e3|`QYf3d~8U!(_q#7a+z0CE0bDrAF)$TOngM|~d9GgC;>IQ^My zqRYiT(S4w&q(p4G$PWdT`Cr9hQ=`1v`wk1T120g^4B^>TEW75y_ceC0BJ(4W++0<7p{|nIn`)<A0}S~37`-?0B*9wfs<2;g6F7V|=V zxq)Ph_S7DnCi1gL275)+XAd7#{zuWi7Rpch{-!98=TBJPD*+ml$HtEE@@95`eE)I}0VDsB+6$K(14#DAZ>CwZ5MtubVNXE$@Jv`f> z=W_$M%a0z;bLmfJvoi;2;zEd!KFo#uW(6Wp2{-*rH$B(48GN(PrB65Lxq%2Q&ZS>r z(5LZ@Lz_Yg#H|G; z1Em-AUMnXUq5Q%h^iuyz50cw!uQe^G*pDZA_zHub8+7gIA?VL^gFYkZ+Sew~XBzZb zLD$|EgPv#5^MkJaVLXR@>HHN0b^9xH>C(P}PJhvO9SSndzUH{;xqTG{b^DvE(Mdnt z{t*4^_LmRk=LDH!e}%li-Tw7}J}v0l=VC)2D1UB{CT!6kBEPFYlz&FhwMUWP)i=sN zD@Zpq)cme~QvUp4#G#A)AxA!xzaWT_NXzf)zmQ+Vp^N-U<$)U#DnBvuaOer-fdNX* zA0rQkopmdaao?K|4LxXRx;# z_M+pTd3vqs*?hc|>tAy`iyqG9dlD|c3p9RFe|4r(KlsgI#Gq^+>^~6s9o%83e0@HR z9xl<^OY=RW%?zHt5d6-l!QA7tNAnl>_>^D6UUJ%l^o7}8RPNR{o6Q79kn%6k+C%aS zhC$`>qw!!)%?w9gqz@Pz9DO7C!yj_^BOIz>Lu!4xKh|p%*RYpW{^nV_=_H1Z`4;7h$!{-T8p4>9wq%^t$y7eX};3!kJ_K_)bFA8moVx8M}HR7gf#w3 zYH0V&Zf~!(q^4Y>SJtreZu_E#*D+`;1!-^7mHu1|esT@{yEYrsl~aE7@Je*{PwcS`-s*%Gb( z6=wa_#ZLX%V!ciRlkN2$o`R=Q5FG!|E7Um~`&Ae}h`vmti}M{wpC_z zKTXP)^rud#pN~JJ@OFP6F4E-HYh53Rz+CM5d#zQ_U%Px7e{V4Q2jkC(OYgPvvN1h7 z{8RfEDfN>-cOB>ma=e+P)lc#c5!LCBDYiU$dZNqrb4=~43?sc9pU}Qseq%_MPxY5O zbUq(cFLN*-Pf}Q=E05z=>R2I)v-j6}t(7%lr@yFwt_ko>W=DTvPjFamc|{LjtI3D_ z&y^?-f%DsIEw7>KM0)D46@dtCRy*ZE--lFtxY{Uh4qI-Q_nesD-1dm}x%S4#i(Ezx zP~_ptOQd(l7cP$+MhYs^KixG))W$aQ{aljgsxYFYK0n3XJ{Lr@Lm*@FFnEt3CMnfY`$5&0znr3Hqn*pVIuJ z`ZI*NcO{BI{-}lVOz~ID@ilsQEg0iL>Z9(z^jfR4;Vvru8TYsx`f8}tsrq7R9 zv-}Y||I>W@f$!wXm*iPPER#I3e*`=$Y-8*{bNTI%@_RH3@@RAAaXyYbFnmHzVm=yYv-k6L*kxc*)2`hI#gN|f@T`AT2knf}ltR)Ifar^o&}>=Qfp zdi#UK615-ub!dOVG#lIPZ&ce~IezxHa( zPZegv-Bt8qeE;bYE7u>f)7$M={1@u~8N{O7{@i>U+wE^t+aI{3_CMWKMAIcg7Prr+ zrOM+5w9jZi*^?$8(_fpPO?Q~=@rw4_Pzr=2@Y=51N?e<%0 z`%heA`^)6;CfbkvR>S`@`=2V8;YH|2j6N9t>){24|5pM-=7H-&ueC6n-;a><8?E0+ zUI?77$p`nl`1qjsC!`;XYMmo1-&Dxc0vlure@D+r2w6n`_;K9b2t_g|xs zkM5tw%7@nXu02gaKIG6V^LcqP`5aZ{Q+bKjB&^<>OPGF27Z_{BWKi?)X%%%I~Nqzu5Nb z{@T-BYvq9sTYjoMCNh8P{w(Q_><~F$FVOr~k`L`qPe?w(pT+o~%cI7Yhwx`($Lk%c zJhonbDp066!_7`7b`)AQ~ia_m$y=d)+eZ-8%X8U!2me!~0{vYi>(DK7=zrG*%h!yll zY`We4px*u&c6)_C6U?`<-TpSU{ci;R56XkU68Lla{v!NYnl2G+dyvx;MN$9L_@ni= z>Az-Y(~8h--(H2!tgU*PAJ_N+#kl7C4&pgj?l}g9? z5%YfhV|N7H`|rDO9xP_Oh5UMD1t_2P)BDj~Hh+&=ImUj!VQ;t}I3}|5@#Rq~PrLsg z#rjI6$KKz=`G`-b^2o}Gpgg<&=;6uQ{k7=fZ2U9?57OVn{{5S7{zoj_E{DKH{>5~j z%ebFEJ&*dsEkAZWD_6|Cd_Mk@K%{@c{pq7_eK=2$f11H>hQTlP|1PTe3-f40 z#pOr8e;GYoAns>7^<#hi0X2V7PQ+<1)vw>Lr1KEEys-cNwwix_9yQLb|4M`Zd0PGC zpL2tZ2!FoU!v1rwT3%@mw#St678vC%H2A@O`8#U<#d-RE`&C+g)Strl17-cu!(|$O ze1G{fYIzkoFgJytC0cp3|E$Xg`*FCTY|8`t&w1+pGqsUr9V88J5 zDnHlc==Y;n8u_m^_`!bcQ*M50zk5G<66(Ls$bW;DpZMkeHRm@K_6>V`&n}PlBeV1L z{rxPBf2vQve++v&X6N68{JA+0mm@E_zpu)Z>JK^cnuPRNk9uwSJ_MYRM+JNb!!*iXM*KjuT6w=wH0anjTMhq-yweo;R03;Xfg^^?66x%FZH|F~5!m-8df zr&9fkobrf&*w3SO{SP7k{FwU7o%F=-!aR3>oa7CANm1)Bb?e9ca@?A?(5zq8H-3NV zgkAq8)W0~UepNqut)+Q1J&E>{ym9{}pw?gE*3a8tw$!Yj-w#ps=P|o}lD|G4Q-4=F z?I-z9&!ZXIl|S?+U9CUYt)I6)Yr0v#tG^_F*ym}xev<#yG3|HtmE;fm!{ce9{>9|4 zr-U^AGV(~iE`QL^OvxY4#~im-WElKt=YP09e$6ibAzr?#e^me6Ja_+$`XBn2rIsI~ zZ+=(bNPhG4$N_fwgZ>3o`6=Twm)|w>4SrKxeI)ro|Gw$+8$GO#$JBqWz7hY6^Ju)c z<;V4Jnp(d;{*ruM{UiRNZ%^9wk-qBVE!F4h7x9;!=k6bozGA$}QR`F2C$6uh+Ie!S z-_;l5ANn?A*H8MYk59*~+{K2x{I32G|5xX^`w^Rfp?|q5KgxK|<2@)Pp~ zwJ#%&#w#IDY9I7%mRlal!?hpczc7!MSFZdq{>^aHYyJV3$MuC~{qFcl?ScM1W6Oj3 zL-!AkTSYPY?&>q~e{G)YPZR&JuOgM7Yg~JVKS}+e>(l%^(hs*i&}ZBt5}2p=>-qzI z{b!e7D$lhK;s^TioK2_kKp&5eTUn+(_+5J-`A*Ms*MEzFp)Un4KXiY?nU93N#f(p` z{*rv*&%9vsL-LIspI4#%LLah`ercY&{vvt8UP5a9%J|InD|UQ#^_9wle*TADKk2(Z zK4Lt(*3b`Seir@Z+K=cDxBmp))h9uB^+V9z{u6Y!zXaWtzo5JF6Ld%aU-B92DV&#H zu#D{>`N{Lhc%DSNf2*x;Ag@p0{BzA2`0sQ+yQl#2NOkY8;ygFM-$wpD-LGt+`)wZy zwDR(4eM<7v_otrjBI!}UsQi6ee>>0F2bd@i;*;eSW#K&K)Kq#xiJzy0J=7lD;gP2xL zyU)4;>Mk+X^R@RqhQt)$mA*prT{7y3cUe2!IpH9a_kp}_o>9Y#p{=4<{S^03MTpYD#Ywe9% zg$6F=McL&=alS3cqM}k>ekg_H=eB2#k)K z{qyMOA;GU{FZ%O0WS@%tVm~LYz0!Uj**~6#sXh3KKof5d+5bfB6Z7|r_IOM7nU3=_ zuDq~5no8rLEkDdRQ)&FSv29-g$KHhfrArL^fp}H>osQ|(#hSejlRbZpbIp%0T)OwaSu_@6yt``7Im z_M5Gi*N5k8RcwwIradA*)JxUhh2Tb=pNm5}J=Q*TdUE^J=jVAL-TqDgCf~4s(;tYn zPksI_3F-E~KpQ{E|Iq2W{YEYBO+zLKr0>=eV?GasbohNd4~P# z;~UvO-M+Q;7xOS|9O1>L1N73^onz{}sg* z8m^pIF0ruxlF~&k9rl0C<(>bP=kdA!Uts$G1*}B1|FeOcd3z>m|Esn0wRk>uDkdbw z{tKtxq~RRw4?sVq{U5hhb9-Cu%=ZNluR0zS!aiI~;{k5}i1C20_s?){-Sz$)<^C?M z|FPe%+z+?MgB+*6MXS~M`M5QQ^E1a84?-M=5{?IpFaf#t)@RL`qn6iaRfN>>AZlHu z*+1!eO7seZvXmxP5!-VpHb@?yL8F_wTAz% z&+q7u3f=zA@kIA;=y@DY3Bsc@@{V zblCr@%WMD2`#-q-hfMnq@#n{!_5a1d2Hu{D+W%VPc||;b9dYd&_BWM0Oxxb^Jnhta z1s}K8a{F5A%*SguUgOwjD5Tmy*_Vow`~O$M-*EjmzW?W(-=O{Nd23bs?6VfFQSou> zO3u%f&Uhg9|6Tuo-aOsj<{S20cBNWg?D~JPmOg5&;?n1TU@x#YH~^|W)9GH-{)@Ho z$DDs*Puln}5&O~C|0Robd)Mbn*poIsL@nKZlG}gjBHiBg`M1wnri~A={!Ec3FY<5n z`G3(O-QKU##*aR0h2j6}{tTWEt%>ONPp|VsaO{7M@X#FlFD_pdQtiK_;u;N?E~}JS z*#DvxMJ^rofBogPf8{(VxBnv3{)_nYppN|qFVX(1YnbwUM6Xq)JDvF? zsaq9%+^XjGRqfb+6~|SM{THEw-TA-Inmbp;6S9BjycPKaSpO^M(P018XYWRd^u8Bv zZ;Pr_d%Ya?ua}=BHjL1X}EORH4@wQU+L0e|KZDP|H}KpxcwKK_Fv4Omv-#G zB(RQa-$dvCu<|@V+1Cx2Z`Jz&#ZzgA*0ujDr|wYjaVyO2J?z;34IJO#*ncsW6RQ1R zd8LYz+y8v+yd$^&ewV*Kt0b)2JLcOPRD9f;&-t03aQ-iWd!x?J3$*zh^Y47Myx94F ziIzTUt>@C`f4KdZLwsuaH-eZ2kYh3!}pa16@`+xIH`=7_(f8f}EIIxDd zXQK9hz4kmDx392kUwEJ3R9eB>^22$~sUK1BaqD_+U)MYKKb7MY$NtH`R_%ZOd=)3R z{{`B4W6s~BE`L~WU$4*4GnTQUGHN&m@jrv9sj>%*-i6Q z`(LzTy@tyw>m(NTUw+Lhmk#@{eP8oG_A@=U2GMw~p8vu84}D%x2)g~g7wvsp;(h>z zH#FE854HxXczF{W4>Wtk`*F4NIM~1MvsP*6;rMxMI4?4d}{ei&G|G{4) zmEq@--SH3p0OafP5B-@YW=Nqwv_21nsX7F1|5&~%>COFV+E3K?!}ReZYHcv=IoWz$ zov(3z4ECgsH>Q6=7rHcc{zdB2@)Z`%302LdE< z@qB3V^LNSZZIO1qjoX{@{0V>GmGe9c`G5NUH1@-^{iRs@TE*>cm1Cb3+W9%^Z(W~D zO7!_~)hgYdN{#t0#@{sUCztjYZ2t@UImY;)>qpc|W*_GMcFcUVN}un{`ABfXS>o@z zzY1U7`8v7%nET)Qc!c?Cl|CQp^HtPJW>4n+x4s_0_`FJ=FR#qvyAs`R&QT zO5RzQYkpg#ohRe=WRVb z+00mbbI*^_{8o9BhGB0KJND+%e=qi?oiBSQKzA@N`h|F-dy_c#on~@T+1%m-kkGReEs%K zZms-$1=erJty1pq+0Q@W`EKobTt2@!&l|_Jw^DUJI&LlGGK_(SJM5_6*y4suE(%`)BH=^e>D6_-5|5EoqW6wjaH_8{>@W{CSr@3A$FH!x;SpQP@Kcm(q`i#FK zH`Vnl`J2Z1tJ&hjitBGKQ_fEbdvwoJ34c>!#~xk!@5LUq^HnW@CA@Pc?r%Ehowz+J z=bd1W{5+-dd^q1f`KG(Sc^toQLF?rlXMI*_oOeogzIjnlx35ava8vDNN|N)ZwI z-QOt3k zpV0Vsov}Zn-|vh)-=L2#vExbX{ZxIwL_hBkd%nTkFVfe?2{1WUfzoJgVrOQ@n=baX<(9b)SeBkT7a&^6TEI=#r%jJJ^`%~Tr!R>FzwLhG1 zPIA5|nf;mP|IOzM%BJY{Y2IHj&ll+WAA7!t?*6*_6{fw>^NQ;Docw%ItUc)VR-oCN z`TTIK{h9U_YY+NQ{(^3A=6-Cf{h9Wb6LY>OpI7GC zn|Zz{SYGFz7lOSNROsi8aK6AeUu2&*!hQ_qyUX`{QKJ2oO4Z(;3oPL6nfQE%<9gzJ zZ=CxF^gM?8{t<3(PpbA-kmNj%IR9_YhxU13_dFk+KX&IA(nsB2%*i=>H}r=UtQATdY0k_7*bi&3qm%+5Lf7d$_#z7UbH(|8RSoa+&R| zB<4I1Ivl~VH}gDCez|^rCsbjd-x2$pIL~LC=Lu@(c|Q30Ev(vG;4<4Ap4U;{x5Mr2 zDb?PB#`&3iKj><@!o^Cp-rFgXc#LdnWH&o!?eqxK-_4 zAD`*@aGkE(OT6cEbbG7JSJ!vu`i-7H)#>JWda?Upw}<>e-QKRrSJ!Xm`V8w?UbDRvp3zJOKxwvebahUx3^`6z3Kj3a(fe}Iqmk)`cb#H6^6Z;$Q1?z0Ei5ts+$Fo1d>OXYoR z-2R;RtI__a*iW&~qeQKI?fl4bYkdDQcKj^X&co3A9<=i*kO!@g5#;%ksmAlCy8nsi zGjU$lozHN-9o_F@(noc^d<5^8ruBxcFQ^}{=Tp<0_O9+Piv3-EKE>mfT7LaJD!tEM zT|XwDKh5_`#M-}ZpZfbE9}y_*{k|vRKNjUH?|TBhq@Y}$KN0(trJ;%s=se0q?M*wMayl@Zw{PP1HvW1J z&toX>8|=0G%KHw+i}a!_dEU0erG78>!bPnbxMjp-}(LOeoE~7 zndb4<^FMsP3&!ZXZV$ypdj0zIjCh_L>ZID6X&?IhO8Xc3d`J4N=GW%~y#LOx)2~Qs zf4V&_g1)Qe>;4_qtA3qs?vKXWYq|EmwIcjrlUly+|H6Ow>vaA774-hg9r}Eiiw;L{ z?9Y5ZQ*q8b_q++&+cf?COpEgE_c4j{nMDQVZhFjjnU{CJGS2%Gweu;j1ZMK~Tt<7t z4L*B3G2;$lsdEe@K4iD3ft53=7 z4Y&X7K;t9M)9Ut??0rbd>@9Zw)a|V_q}!9eUc>wF411fN)Sh*FOZGmbe8b*!d#3mG z8s|}FsQx78Kl6P^MLEUpc@vudrj=;8G}nGVlCZZ>zWshAv0sTFaJZc3!Q$B440SzM zeEIE7c|S0>xBpPrbJMi*E9hVCd`WV9qxYMt^Z^j#XwG3b+0aKp>NbGoh= zllAfa`Z-;>_yKDfP~M!bX=7r1LmV%krHbD-q4E}k?!UseP@>Q2nv^?9&p(aqj|ZMV zGeDw8ApX38uE}X!cP0PVSBUlE7%3?)Z~B;QAIH~MPa320U3w-|R|cfNzDV%v(kG9} z@_GJTE5#qYoiRQ}@bZ~cJk^(LrA{I`*Ky!`7ceii-+u*uRSh`Mm6guz(UEs{hWtS^ zE`QPol1HcP|9e+5ZwhOa`8`~Jh<~9!RNkZjGz`J+AAg!yUkH5fI_6DfLo&b6KjK~L z8|0leCi#)_${v&aNIWT1?+^cEwZ8&gDH-&O%e?+43n4#Yzg&K>r|TDsL@UydmKI~)n-b(TbdwKbU+ezO20we$IyNI!Bd|;L6KlBgwiy=Sy{WF@+}qv8GaxA6Nh|b{0B(* z`~I`|d?eq$kMVvn<}1eana@`kpJ#TV`BWYW6*#BM=cDm~&o58o_uIM{$DNzOKlt-v zJ|I>wpD>yaXM#?U=YJOCZ`wx)B8c{Jen{bnAIz^*K2g>33G?zHpW3uAK?IQ>^D(i5 zK=U#B!;;$Nd64rn^N=J{5{&0LT~eQxit+hdU6Yu|k%iye&Sp<`i}9M0vkv?|L6_8@ zrI=swdjQ_Ua=wPCkUXC3PIc6m#`9ku8e=j7^Jyy5(@!fPPcc?oZVo+R7rrzY_YaK5uI0rwtm)5Bb#| zRP*1B_8v{1pB9`6^PfqcpB(6f`G=F|?;J1x4EBj+?YG9uUza?8WW4;5fbZT^N*{4?F=w5Mf3M-57o2& z_gnr9m4CDRO7`fu}%tef2r?NG0Q)}&Lk?I+h?5T!^gGHW9*xkTHd(!*~o@2wY;FJ z&+2$Y<3|a5@>0vw?9VLk!ljp|*_&D38TQnr@>eomdylebF15V!@ygrEp1aiYD#t4? zpZ)WtmKPqcyq7M$yt?tqdyKu1h(A6*OkjM~?8UUd9`+yaw|vch%&#Qx!u9v|ONeFbaw-~H})KDgdrI9Ym&@139j^owKfj=nqk^Dln)_|H?{ zOJ(mBzGMHp`@-0F#!|-$-x(VkJHr3H_~o%L{%-88x8C}PfBbiZKlgu-N@c+&I`mf00e)YWrzj$kG&$UndrwDhw z_PcM>|8KqZv%6mV)@ygYR_Mv_r(fjWe-&Q@DA?!0Z52;OO8VQJe$NyyprqC875xp`S=XKh5c1s(71)A0@^9&KN8roefE|U_`&!<4N^1o}u-9zAi4Fd|M*T7EHR_L%mr;L=yg%sr(H>=v3wkbkW9B$-o@$?%K+{dHpAt@C)JciTPYtKCXzDoSqjbo}E}zeDDu01+0LP6Jm!B3+ zW6`v6%Gbw7n*XXYiqossV#JakP2TBXC7Q0)7dKr_IEO`Z#;aHBml9KM8Q}~T&Cu%6 z_{a)pv1rzKtoFT-LE`e0>XvRBGJ|91fwY`Jufk@JE(k@ztjov z=cTsV6LiAyChDBh)5kM8-lVGZarkT0`I~g9{3RT(qV|Ys$WQ9jB$Yl6e{JuRzl7tR zTED8Va=e=;f1U4>KN>GJd$GsIMEuEdaiaWL?~}iT<7y)Q1pm>A@)voZ{3RThFB5-9 z-zR@Gu50~kkLj0*zux!BU)bc&o)az;f5#@kAD{PhpGzA9>r5W)Ips3(SU&+Cb)Pe$ z@yAU5oH-=H7`vP0)@B6FO^U%=$g@mb})Wz4afj>*}gqUUhBl zr9D2^SMXt9&g-@J_uvHa;L>T|&ZWac67+HGsGqkZpRbMhGbFYih^!+2T!k+EGTZN> z1YVUc+nJCq^(Z0T8}?%R5DPpvXyr>^AtBxF2RFOUpPb7R=C|kCMES)U0-LOE(D<|0 z04`atU#=|^<`?ow+?3?U;R&}zxF(yNdE8F{*(1zGW~hK@=rGYyuaij)R+C}O4IO^Xtpq!E`6X>n;n8u96=LYk0C?Gr1+elH$_6lX+`N8u96AQL+@WB!zyd(qMiXZ)|%HeQ?rnTZ1P~V=W~*S=fE% zP-a}ZH%qjZ`aWKM=uWznhCfoKMe8|I1r)j}jmQL~5g};|+0v-|prrwpvZLcjgF#wx=~5aI>if9) zg*3SCxO7z-EI0w(NF&{_AO8-2VA8eIGQTA1nP<$A={kOYvo0|J0WS|1=|s538lI=QPECC66uLOCu}Shj<=a zG=u9KgM_{D=-w#{WRaVg-IBma^1tK!k4Sz=v^m0nyrP9$=Rqjj&FUzNf{ z_B3=|Na%l>txbeur6Bey_&%1)r6L8fx@VA5G#=ev-P2x1JUXOcuih2lxVNii-}oUTHO560N-!p zI+#ET6WA-&caA6Ib6%1Zz$MoO-0bki^|sQu6d3KV$E9N<+190abg!aIE?v_F{#41O z?4M7_=e#5-cvGTYY6WgNDL$dA3pfYuBRX!0#-|6owiM&jaUhCxg>)@}6ynY;Vt>oM z*AT`kMXYVl(lZKi&qtoXYCq=wO!MXxPxEfvv$6M!aNC04!*S0MR%O=s-Di}(!|0i$ z>dfkts+H5y=|kVEEB*XKPXTcb2KWsSdYhcGqfc?(qO6pQtQz0$Nex4Z)ld$oK2@+1 zLzOq^ohEkLJOR%|+E}so-9%1N-e!O!%q)+J_f}?A1X+k5Y>0vx=Jhc545y&a(CHC$ zs0fXM9@6R5;UWyhFW)BPg^0XD~fi+T3L?b&%v04I;LolwwzbiDb|n(~>`DwB zvZ62pAM^PW-D?K>&>6c8_0Zb!=sS|o8^3&;Y|DmS@*r@6;yvz1;PkmE~WM?F}pS`42D}DW~9lpPszi3YRKNbAoq3q0c9qEgaX1aL$ zYX!smzAsF1@Dhgi^D4NU;pM~^<~a1l?5u{%*b5rIian>{MeHdJFQnrTN_nO1jD{Dm z(;6;eCp0{t^=o(@JEq|)S+9nR*-;J8B@a%izlh<*QwpBLIyD?(Z5l3QjT$asJ2X6- z)oVDP)oD17g*7~jl>=8f@}0>-8lJ&|8pcb%HJr=vD|AZ!E7(~LPh@=$H2M^FLc>A4lUhxm&3ZL@7K>F4@)xOL)McI;{=gLBdWp|UI*sZ?hj#GvJ0ylSaZIB+(Zdoq zO57+hjp{^?NZclIzr-{v^7IJ|S_V#HS_hoFnqr1E+hSU2_HAA#rc9z>N~K zc>=dd+&N$1PKnQ!2u$*z@*<@IQ~P=Oiv*_j5+1rrU}_)XMvT>VdwBXK0^9uc<7Wqn zZu1w!Bum)lFS1-jfk_@zUPNNXaexK43VJ}|@J@k)9A~l4MuBHY+`n7k5XV9G zT!+9V9OJbr-2#_$9Af2oPayGE$#D^Dyj$Qf$HlDoPXw;xcpiK1et`vlCG2dkz=A&r z_p<^E{;;n6oWO#=5*F?kSnyZE`u|E`!5_*yDX`!V<((E-@Q3pLAAtpbDDN8r3;s%2 z=Mw@8{(>y>q`-ndjGq4_u;4GihMpBz@E2h9|19tcv^NF)5&o&b5D_rsjj@2_RnO&- z!p=&(L*n`u1-((?A&J{0F8`&VcS_tZu_bZvB|(o!9Fh2_#1|y)mALU&BKzBBnmlt56VL@jS_e&g*xcr=;2RY7SLlV!Bxce+jza4|ItIf1txfCjAinm9Wzi3;u#E{JNkE{(@{s zV!>a4h2e`*e!*XW`6{O~e||CR$9!~l1!MhJ;}@wn;HPB%U(8eZgrKiyX`X@}`tj2u zm_6w~|M>YYhyWnaoJZI9sxldk#5Pa%MusEW^dgr7M}f8F>CXE|i=piT<# z`R!W+9KVD4lY$w)C?$Tp?bB2G$s!5=yNVitaGm7G_R*ga(N4U*R;V%JC1f*7gF+S1Iu! zN$yEdj-LC{5p^#_5EH^Ora7RKq&hf}^ct&p`F|ku-6Hkb&S2)B3Ie(&CWL|_s!e8~ zG}-vr?AVgq0fCxjQqH5YE9wlQ0U7C?WKm<|Iob7(QDI83`_GOd{iOV1gN?(~FoY2Q z%eZC}?`HmWg5D{UIhW7N{H}cql9_)qqVL7esyG?tmltIIVuN7jpCbg)88*_rBlGvB z3j*~qjW1^YqliEeG4($ne*DsDzpW#7e!CwZ0)gVf_^9Eu%x}lp_1pRE{=XfW$IEZY z0w_PFGxO6pLLz{9X(3W5VB|~UO8IGAj|k}e9Y$sp6Y9k$!`7Oo%;`LuA#;sh@f70SaL6IMdYxp{Gorw#zUrPjh z0|ZC)BWR?dfQYS^M3KZvYePoAnRg^k{lRpR(bj*epMoJi)++Odjrwnr`EC8B{4^w( z`IkXHpNdoee@qkghmePYS--7E>ybZ-{9&Vhc7@2_%X5qTRKK0i)?fOO%6R#EWqw2s{f!U}N`lHD|K)<_)5aNH`KO)w}Taf=x;gjA`+n0T{ z^V#%0Ra>^z?rE+PVfBXUijITLORGC>UApC#+p0RYHf?V?czh&?8n-4Tq zt=_PDTXkFWzO_9&cWmxx?%cAqW8EFqEz36_s9JZ&rX5QZ+TNbdUHi5m-{zaDZry&% zEuC9xJJ#NK%kG^wAGocmZTHUFZToKBv8}shJjqAs!>NP((k-w=l11y?pk^y z)wkrqvX0t4>)SdvZ|SJ3xaG#$ zJsX>b3(l+xOHrRaYNa zvvgZ~Q`MFY)l1uUw^Xb=w6ktg5`K1;H*H_i(Y!xdy3T_w+n4Nasa@Gc?T*Xex;u7| zoOac&y|eky^7dPIZ0y)kvH8wjt5vyeTfb}n&3A5~eyo$cZeM$7$I_PgW!CPwSfnMyaTi@BTZ`&d0L+`N*LFu=ME%M1ZauYo z$I_j*-&(P@J+2Pa?l~mozLux0XidUj3zrY%o~ZwAyAXAc-{*GSbu*ZNz!)8v>R``b-UG( zK<wSc3{q^z+6$Y{?1$5m&sKOV-5Q{+PfO| zR}lP1d&Ab&{oMz;_Uvojcc6X6vL!0kz_>&$r2&>SEUBodtZZ1ZWZANYiiRcS z4O_bowu(P%n>xB%8kQC9B4NaZhceLMm zXGg1pw+P(5q@r?H!_tP9oG16})? zdYp1_v{} z&doS2QQ*nXEaba=_IipE74wwh&mvrwDfk#IWGHOSoE`qO_cQ)KOZjZzk<2e-{e>u* z(zoyg;?Dy~($o6%GfMg(ZHQ6m%$(iml%IYq^LW;u2S1nd`DtIs{fils>Sj$^m(Pa$g7obWYrJx%Cy2i;LHw2k@h>2r_&JOEDcJTo zI{`ochK&(QPy1XHZ2P=ELHZU-!&B;iC@x<71(ZYOM5oWBYuj#(+$#nQe}Ek zKE?M+jN6MmJb}LyACddN-fmGO0Qob$_gyM0qpo~>&{nkMPAou$T?%lKJ{xARie zM`U~+;-x5f%wZ*7HkrqqknvQnWR=I*`}S0?8ow2Cpg`r@yb&fnr(l-TCgTf?aypfA z)Vo2wN<8*eop3^lSJQtpL3{xHOM&=Q+q+E0n|xNvc(c7{We1t<9hUKCId8~#v%T*q z<*5AUOS_`-)%I2@@#@@MpCH~=4k~AsXgdE}DV2ue^OX1#O1zr>tP-!Luax?2mfxww ztLdLn;??r~QkhJC0y3WDsgB+u8Nbkouain|%73ejH|4)W##6mCx7(po8VT8vy+=*7 zkH~l`Uy6ihX_WD%Jw{|am9N_4X(e9XQsd>kq{PdihR44nm4M1o`B@}|YsTAc@gqH% zf?c=mzuNI(UOHoK@@mJQmG*PYh_}c47mWCbtRRT`DA?uL<8iqWZ;!|JKDeFU9*;YX z^tQ2{Fyig;m@#_+ntEk3p}vq^Kk@2}oDwg6X3C6kxB8K@Utq z5-}w9e1JX#b@+&O)6nl}6Q2Tx;4p$7B&PL}i%Sg6;H4Kdmtz?ify%&>=^z|bfRy}{ zk=udw4z$UIVo*XLq~LFbwsT;T(CPrjWdMycfW(m73Lr@b5CH@c$c+&oA674dC4q!G z5QCaz4nT|yETElw3ut={)Jz9ek+3!ZtSzkT07{39?0EwZ8p@%gjq|nu)^Up)=N7o+!)$aovhk;rMb2pUBpbNAH zViQaRLeFu-CW@)H4XPjA{|lh{LH$%@fbRZApggwtD}XjMK0q~tya8&8!t}%9Z$Hpt Vbh|-XDxeFV4nPA2W3Tk&j@9Du+r=K4DrtY~4+Gp@OJm=)M z!+j@?haYu*Cw%yy}|Gn^2^WP0W)OtRA{M5VQl$j5Q zMs^$+eCp|^2UF)S3C(HwPWZyS^}+qC469u#tL7D z4`>?UCiM;Jjz_88QN{{i1uTcR5qef-O`EHH zlKfBk-T-{>75L^o-@CJYujdB7cXSNjJ8JPg@=4`?0~Y^N{(2)HyteCyp-9Q(w}v0k z4`2AF;;U{uOS|~1;;SPsE${#JI@X_x`Ehm@yE^!ECR^=!$c!$S?);j@3M zUmM=f`e`09ZQzlWz(<~C{gHoW{g+YUA>URuBL#t2y<K*x`SD0JMLd z6gsc`kx~l4TgY(wOg(Opx&sWOPC%k}Y`>yZRHFtB}tq z7fWW>Jdo$Y-!AVTfqdmst`YD&0-i^}Gs#HuT?C)w(7zWlIsZ-zy$d-;UP%azK#q4I zhf#i#VdRy>&&C~GKMWDwx@t7f9z)Ib3Gv%pXg_3o_;j7L(;Pt<8e;U_+ zqw!&{cb-iM*O0WCYo;D@!9Ew^w`8YZ!vkpT(ex?sRLN-^R;xaDqwO6t7GQrskrJIvyHcDx{WdWI4pUx*qqFc+b@f}wn!m#_P>No zl59)8xH|7v3t83`$Z1Hw<)OlN4ws||7e1W5?;b${z7AjDF-Qa}1$-@cg||Tt?_D8Cqp!Ye zTiw=xx78=CV+D8@m>`&D4YV`_y#6LW>HLBA&i2l3KbjM+y{)6u+t!(P7r1YO*Wc<3 zG~5+thH1DgR$FqQTspY{jextrhDsc>&vNKpGli9Hxj@%5una23wOF<55P5{mL~eY! zmxkO+L+-S+FplfrOG7?q*}G}kBI{TIH=l;w`TH2PNUy^NYeDBIDw3HVC&v3>1>B$iU^x z&)ea1=Pz2Yc%FL$T^1BBW5r#~EsgGu?v74htGl(Uqto5!+uGdbbFZ(gT(z!l!}98N z<&|sdme;OdzNU0}DVtyEUgVzd4Ky^lJ6drb@UYj7p@~;IU%}@e7G8I-CjhrEbTIOS zJi1R8_qb*DjM)9n9Ou}YQki|L9d<$6hwY;8`)G6q55&pP<#cA3bvs%5a#6HyqtOj@ zow-eQI;%3`sZXlg(5-E1gz2kr%v5L7>~m7NP06lQ?6WaGjN(d$tuk<10p6yiGC#U^ z!4a$Qvb#V7gW-?A?9+vp*|OF%dy0dRclX$EzeK`!;ApEAuWdk}&^UDcSYEHm>qoej z5(0(%V)Sfdtaq0dVZQ>B3^bI+vbu%7_$0}3vg11wBm4j^1r+s8wtx=lQ~`ZzfThyl z{;JD1_KR%RyWbvRmjNKA8=Y*=zC=48cT;vMJMLyb>J~DvCxJ(Qy+KSQ38L|rck65m z@uK(G1V8tof-)1Yb53?RV7H?Miwqo0V?zsBZqRmL?c~KQq906RuODKwf(f?YOnGAm zuMcJ03xm3i@Qe{2@&zYGvQbNz3~gh)S-~VdN`Eqqr3WS^Or-t?4<)g4M9(;V643*n z8?;ZPvrf@J(er60=+Qs)VIy(;RUdHvb0MFB9{i+9dx0PHNAz5O8J!yOgSt-g`K!V< z;x{Qc30CCxO;C72A7+Gi1)P(J-_)~_BsQ4F3jYlKqt4}hgZJl5L(N~0{?PxE`&fkR zVRyiuLzV3QT|`R-@?A<}U+Eqn;pQNGt`qvv$8V^qsp0azkqvn#>*Pxj2lRvfbAyv7 zlfRAU`WJO}<|SR}r!4F1(!8)I`iDK7+{wuv;HNOaCQjnxi+b;_gb4cp0P+F&Uq-+d zA&-nK68Poa8oNvkXu>~M2J|$lGVNo|k1HU2-S%tf1O4j(8*>t#ltA_k&g8>K2EhMN znmv{L84IO<3-kkgMlg3Kt()*K-aa=7a}3vLlS#h8Y*zOC5n!oc{0l_?;LIzLh!&%t z<0Sn>q2#iX~5?R{IF<04RwM4D)5=Wdjx)V8t*?7 zb%9UT_H?^|$0ADOHwnB8b%9UNg*;m%K3np?MdCH!iGRPq*Dr*BJ@_EW$NiuDD`>NE zUsC=SoHK{SiQ#9>iq>D_^M%`k&p)&$ec91J=jUykUY7M7qjfbB*~{&p3HwG1=!;aH z><$0Ge9q_CD|>#aslVP`88Lm=w^{T}^V^_rK3_AqzB8o$;Qtv>eb;AWy>MCd?Q+HR zT}k?K#pXAkA80T2js7!&&=v%U4D^cim*xjIpww5ESzhWdx*iM_v9he$Ci%wFSMM(5 z1832M)~}pDf&4auzmfb7i@(`<{Irw&r)WL1i~NDshk9BsMSB(BW(FOc-VyUR2iFJr z+c`}i&7bgAz8+)#t%UtV5%bR)56Qk~h(NC2g~9RT$^LwN#EY`zTwYW0pm!IReigs` zB%dCWFUN^^DLyN`kMo&Gr`i8V|2AWMwjbY#d_&1sPx+3+`KxOmP89hL*)Ju+&S60Azhs{q z`zQRH=Zg-RUy?k$sQe>fbMW=~f2jN^+4eS;Gc_L8GbgqRBA@2#S+b3y4fhYoOL)HC zW%wD%Ti6RcA3*p|`fgh?&yO8EKep!v9m%u*F_P^he?oi6&&T8Qh1Xr=d?H_Yy%O?? z^&~~mYiJL3Ybu{f%m^l<4u%HEKdHYI3>U@QGlJt#H{xmf8u1CJ8}W%qw~Y8C)MI$d zddAl`yGzECG5x!eVZG-5;on006ZpqInw(UWe_(!|MV0Ie`y=1aBzF|?knBGtW`Fp< zd44YB^xU2_C!o*5;540)I)7@yil{qe&z_Dq-~S>$Xajrc3< z1-)pWMEMT#Y1_aP;FYg$X)fev?r49IKRqyAp91=b3-funGBrJrtxu);WK1B8kMVM{ zQ0Mzi^auN9lDqh6K0w~WU}k1Co>}=J{(CXr_4%Ro3jv)7gnyg;{SeJ3<9xvSWAb<4 zP4fYGlfMIRm=DlTZqV%i@b~FtU!L!P-;AJPK0sc=NB#ejM)?%{3x(q2i}(^E@-OB_ zo%DNdA?ipMs6&$H0`1=%2{HYWezFUQ z9XKjLv0=S9h4FU_d-&_$AIE-4@%A--zNhqa%kp5q&esp}H@-ffG1=3apPBLz*zZ7q zAwiM$vz}*M27g`09TX42-#ED+Gvzy!&-h7T%0DQ7Pkx383kiMQPCk?Ospd0&&b`tv z&u60k7t3e(`NX+2`wjCK#JB6_FMkUELwx(J^Ow>2iX-`DFVatuk6^yJ9ORGudWmA`9y^mJ<>0p+D1lj`i8Jexsg+^*TzA_E?W%`>i3(yk29z z5DEq^nIDgEX3~a=O`F{FS%vZ=C;^r$e#>iLPGj5!( zWbu5(9Y0@zJXvmv@6miEXvjB4=PQV(?yP9OG7SBF@_ZzV)>j@c(VrzBLA*5OBMyu= z!wr5vK|Yct7JQ1&qw=FWHyPtD`#_M{3BvOaoZpvqy*y^T zI%8k)n|Z&z5T7rMwcp;&#Zl`MkI!_ZMxPhuP&~Ph&-cj_A>~k- z%M_omewy~zTPZ#z^8GoFXULDzciVaXg#ADodmZ`f?u0~&=SdUO9Fz|g2JIBC6L`Lh z^-r7^9o@*{#WTc<^nd65w<&%hpV9;T(}(Ei4}_1#Gw`2|ct-o7NT$f|Zatov_8aE- zh4D@M4d~kx&p6(^|1j-0fH%drVV-|~!gzMs5?_6NV|I8PclDD?yTVLvmO zK9Aw+^ErIpxNBl;fBqofpY!umo^QXg5cHD``|(5gyum)1@5i}+{1@7f$Ig2&9`;YM z^DC_Xqxa)er`o&V$pr%tiNx7hiQb$^TbdlUQZ zDaQC#7@VLpe1IkPhcrJjCrpHe#Qr-Q`|qp?^!c1&|2>WNi-vgTqzy@I|Lt^+vH!+_ z3YFW5ca^uX|4zqh2?oV}742Q9NB5)1*CrUxn}E+So=1`YXHvc?_Mdz{WJdR&eE&`U zgc6N+lP5Db>XfgZUkH7=bb~%&|8(7=Pt$&6?ELi)INvWNnDbZUw`KcoExsW?GJHQE zjjg7`CSDBrEG-FgKb_ynA`}n#8`d9uUd+#X#d(I|Jp3q#ZsI)r7|z2%*^~G_j#Zk{ zEc@#;9s7OuH4t!r#{Tn${NeL(_=7lK&!W#y`R9>3ePEtQb@02|vft+WwWxefM)L9V zMG{i1uXb_}-hMcp`lpXuWPe)h{6FvJ;5Q~pWyQYQ-1Op z)FIWqa!&*IG;mJ?_cU-%1E0DE=UGt)`{w2x>+55z-(K_V$ov0X7x@}TrL0Z(%f6i6zP=Q8=Etx6 z{&$hL?QD6?+8M|6wKWNAYp@$ZNdM_wzrJ!)s`WUYxNBDv7+ZV6E#*1fKBs=#5;=g;7%Kti1w5L zXxcp9pBnGg1Uu@*sl|!(DoXX(wX)5V1uvjR3k9b!uCQW2ibjo5MLj=6Jn=EEyb4CB zFM5e~oVRgS-q4K2@Ox9NcuIHlrOtvYYzsb5!uM2DkRbMIiC{Te;^Cu;b;WN_xjZlK z_;@7$qJV*T{qtO(C_M4;BG&>Qiiy`g%Kw&_?v)MV(MRR+GqzUKl67c03gZy6Cl%4N zL8nO`i$8EGZcDPs-ch!M$c)>~pm#k<=shYojT8qTmD?2`AC=oJY`==tAeY8iBB}|-2nkZ+G?na!Ao`Yu2ZMv27xH{JbXVFp7iP82#*F|M*De-2k z&{HwmXv{iV6`%$^j=~c+q=U7(zWPDTJvaBDUfP`XY91S~su=kZu%dcBMbB|0{*JE{ z+9>a$vDJ!VT`^)jT~C9m-}oy~to>fyDznQlZamvYJri{C@X89*&uFO)BahN18W^>m z7>_eOwc2@wZ+>k_+Ld0$@URnJXgCza3;a`uh69W*t2(~XVyZuhx?+;(F{n;;7)`_< z)ajNleZYea6aCwk`cln|_gLyxmO70`{qrRP{nv^UZ36M}Z&)XfxYahFl)fmn)PAdd z#A|aiyr1DK>R6-QOiyF=Va3Pm^Rwf|%H3FX<4o+8F^GGaa4Yij8~Rne_G*&Q)EcwB zX3|QO8)3%l7^IkXVOpg9Myt)}O=5Su-FNXHriGgf-M#dZah!^vLVsu$F$*BPi& zSEOqiHAmHB`wW{-qZFmCw->n{OS!qxC>I$^H}P^6;`CN>0XNt+=vYT610+^RBc$U| zS!|qK&*Z(9JdgS@T3?J{7@1i|c7w!#zZdJF5BQjAjs7gw$WLR;&n{}lc$y%FNTXXWGtP1QoK>;_hPtt6&JBW#fQ;xhxVB9Ptq)_yKY_mDP9`F6i&* z{Uzt$e3tilJKOub>(?}&j5{_F>tj)G@2DM{2OboQ*?XBW>Zw`#maeZU#a^57vv2JR zypImUvD;Q<%n|r!B_8n^79Gi2&e4UBoA3kT`*C9XAM1bg$vBt5(7&$n{(BT}^u04< z{}Qjz{NLQx>RP!A=LYfn zq;}(<>hf}7j|%XF=l{ICg0U@S_}_B8uJ`;OQV`jKuJNAjL-1eH8;T7lLD8?)p{}(1 z2;lr+o>N`_Z4U2)9}Ss^mT~y#9>jCHxLcd(#dEqcOfafEi791c;z=(?qN#DC7R`AP&*{ok;OI5_oGuAb^ihJ_yrG%SkLfJ!J+j^q{Xkg&Vk?8F(_Wl*wK39U2j;-x!KlenO z_Qx#kRnOM;GlW9?9z`Ipr)2vNWSz#R@0wWKli#C@QTz|cDdIM?r(4xt+3^rMCYXni zhH}<2#nkw!owEO9XndpgijLY7owYs9b5eoDO&KG{MCr*9S87i*J)#i-Eq8Xv`knz%^-sbgc2v{BG{wBH+N{IAINxssB` zr(#hb^U3xe%lOA+d*#2>o?P16elaHRQ*p-s`eqQLxcNI{*721eRiJ$@+IuYH56Je) z|J3-Zoq|sVAYS{op1qwqlic zW8sdLiprXGJIhwIta!Ah$zQ#$Wo1FlqvboxYYVDdHn*R>kF3DY%gB3y|K2sV^i&#$5*wk$!{#H>1tfrvJ+#pfse8cWgEI0 z3YK-#c#Rt?+KbC~mew!Ykk?YSts**Z*|s&_qGj#N%eJkTG-aFGHni3*+OYkm=nC@3 zqFb@KHQ2OyQ6==zT1GN&-Aua6Z!ay_w$@v`CVx}ww&mN4s=GIBEDXdQvwUamCh%Qf zw&L-I?!sos+X7uZUSCp}=c_ID*SGp$pA8lDZEGK2O*UA$q1jusZgp|nn!JYA4Ndi% ztJ_O9d24F?)yrB+n%YY@ZmaSZm#khF+)gF$2G1tXU$l7f;zfCROF7L7Z%c=-PP~`Y;0I-V-qyTe9%H+5?&!xS z=lqd-*O$HoM=O$8%vG#a1({@kSrln@%K`!|HNM%4=a}D?NaBtjWK1mBMmG!Cj zd!E&g*#A87zd62^`VHp`qGzISr002{@n#!he^c{o7D2dcz- z-t&6iH5$b!@t!j+=y^mO)xu^NS0M#`j2|;x?t~N&S3z&wc?)DaE*XZMo05ydB*B&c z3w<&6O~|ESq6hI#Un(bhj|3{e4@05x=vQ9{{(H2KM^F2kc<_g>1OEZw#LpRS2gcOA z(yqh*^GLjizDkBm1e6CR2L; zd&xhZF2#$QSIk4O*ju;~{ECFzC0xlj0sSf8A=E^PpJajON%$1NRa*jmQNpJJZrn2NHo%Rk;xP%Q zadDa$Eyv{@jcbILN;#>2#T${Xl5lH3wGuwv(vQ*?^M`_B&pA=$IRo5CfBZV| zD+ahx-dw47E1$C^+$!&YG>}!^*CpKA&(9>>D(^21{TTWG$N)FWnyG?4p6KTq+0E`e zsH3{(wr2dyV|_=*tu<@wY84DM`#RXzn;f}y;6E=pHuYn^KnH#rLW5#*#l9Y0k zAX(7R(V4&GrnPa0?1gXMxN(0l^>#M!rZ%5^H!tR6hx+Z-JM@=b#RB^Y-}fSj$W+VJ z(-oPZPL=tGLds=YRT=H`2`1BmDBL6O&d4~X5Gr1&`YhZ~%H>f2dG+rUH_yp@j6PeV zcqhYgGza^80BL+f0 z?*0CHbl-c`Ai$p3Fcfs(TmS!8)&Hvc|Eg}?{_?Vw%XLj-s%Y#L7ON>JW0}3Ndk??$ zuvAdGSx?P?t?uX9ZrlH%7ur5EgR#Ryw(P!l@H%J9p8GGjp2zE7agU}G6)jz_?fOCJ zx8vB=4>#zc52|dTYc_WH|JhjI(GGSPv`2a}Skd>gJ^de+vbTPvv%@n|duq^Ly^)YDH7rp{j+n$z-~@P&ED!~5R*Y4|DUo8jgYm%@E3-veDyD0J}P;Pw*-2KUWB z9Ppu}NmxMwm z4h+sc`NUudJYSgq?eM<&Z-OuA0kZeul|1JQp$h2o!b6~e-cl039S*(wW;kW*$?%-> z7s1!X@VJ?eghmLz9K7y08Eyuj$#eb$?e-5=0JbmjWO(F3z_jcuKS}&*&B(=<0?B)_ zzn`4gQF`j4cKGby z>eq(%vwoUKOdEJ)1@Mt)S%2i8S^uS1^sDcbvi@`Tv8(@5$okKxv8xx7*wqx!7Gtgr z?@j0*{wLNy{D5}&;{VjIUVMig9{!tz(D1j|;foDeQzx+gksnP4EHO0v06YAz4}kWs zlR{@V?0JOd8hAc@ZP%-&xg^$iAN5Jz4cqSQ1zKh^<9QyY{CgE7#e{*&_(}+3GDDC=w#%jq!813;>zVW3kM{RS~> z%-9E>NLNjXxT5{XRipJ~?2p|neXb`&;}iW1&C}1vNQpnjwY2G1KCaTZkGV2ES5xMB zLaB?2L(VUhgvd5z6S5Qa_YU-W4*GK;eg={~{X5YAH*sBqS0(Oya6N$Q3B3L{?oZ?T zZ!|va_0F>iAsg(f{LeO?U3~}ocg@s8F4*TH{FdzWYuJEnegQT=Tgdu{U_Y{h^6B^2 zvi@)>>u37FBVDY=!jIpB&HfGci}?dPeEAUUYOrI|bfFLO(~ql)L)Vf@ zu8{ri>BFp#=iof>WB#JITdh~WTl)_hbFs?}=mbS)nnnRWDC-1UftLfF@4(fAR}x<7 z%tgbFDm#3EHeX8zs|F^QusUX*o{FokG23{XrrQ{^kHeBDi_OXGxc#!oYl{^^Xa7sc zB+0hai<`6mSzPrUEYZW3umo^4g(w$vG<9?aI=%I{Q`hcRQRmmXj;6N8AY0JZ-sxM= z_Sk~X9nEc9+hg^HE?IwULG-4fz18pSWD7cd!OjIjw7Q0_&K7S+N1eB&Wt+FN$rq?= z@OE_8No=RDt)o5Q?+i2-7Hn(vwR&5cx3%RjZqKj#_r+jy{PV~_||3iw*?3U7lP-n&AOMqho` zwz{nWZ>vvO#|rQ+FhMZQ8fa+PU$()k1Jo$a07el#asds|1Rx2-epE^yxlufNq7 zXt*oP4AXE~thVGpxpZ;^8Uc5K4V5@%pXJcIW(q6Ya)GXAU>Q`5Yq4tAA@T^BiQM>d zFAceuhTLgsVI0@Lmxg@KvUk(6Mb@zbZaxjU^Y<}okzR`p)`Iq}TRVK6d91Fk?XkKj zX4SG)bplhDhpk$(w}VITD4tp(f4Qr2vKvLFIY8R5Teg;^9G++evp?nV@uT1yv01nA z9lIIzB!}aq&{AFmJi`_1rzb}c)Pt27xb;YSknyq$aQD%)C|rWs*Co9gTG4+Oa2w7q ztR;eyD}i)R1$^(OYg+1_o}Ssv5!ur-+re)cBltXPC$g@1wtpP}(bj&34ne4j*Acv^ zzIuH{Sw*qC6#IXiqliL>Bt*IxudM{~cLw2$rCp6)H=Qmtw|G07+uPio?d>fc^W2!( zK5vK5onN?M(LDEhx-2MK%1XMLTN>RR-5s62R(ESxN2j~dx3#&==U!J;wQ_CU`eik1 zE2>u4E!((m+3K=oWo&+xyU;z~8)#^9ceLU>;9;*DLldubzJkv`EWGYuPXKOT=wRdt zd32vH?s3cP8L|7DInJ>)r84_gJM4nC58FlG_tEGK9*C2n%jwK6>vppA<>F}FMxz_* zI&+)qbXH}=Q=e3~pMn5oXD+2^Eko046r*k@yW7{!$gTV>$361+`IWqx$; zf+JSrWp{xF2E%{;vQHOYX3JW~>?saL-rZxv{bC8LgUc&V|l$MuOH!B zMhF!0i_x=T;*%uD$&T+#jPL`v6j0PV*#bJGQw8*?0hUUG z`>QY8*e|kK?|yrPT?T-dZgjFe`x5PZ+)df3?6{l#s9VUuo&+BK^#(DKB#6de)~&NG z#Eafz6a3tV3d&5p&NyB*kqvn#>*Pxj2lRvfbAyv7 zlfRAU`WJO}<|SR}r#$QH(!8)I`iDK7+{wuv;HN0SCQjnxi+b;_gb4cp0P+F&Uq-+d zA&-m{3jDHeja?=NG~pjB0(u%%nf5W~#}yF1Zu>R#f&TS?jX4QVN+A0NXY%1A1K@us z&7MmBjD^y_1^NL#BbYmr)=l^qZ=V~4IfiSs$t2%kHYmO9rd{p zc1hOj-_vFW=gpz<<`I9e&z^3_JU-vEd z2E85hSlfvHn3Ln}s0(~4@NVGUCcGPUfnNsvG~n|Deps}hhPuFi75GfxJpw;FjrX64 zy1*xEd%9h~V-cnCn*`p4y1*ysLY^%WpDp>{BJmpV#J^wQ>lebm9(<7GJG@yNfn5Q&@*8Nrz| zDgJSP{5kc%!qk87uIVv<8ajyeEx|_q#utUt3*lc_@1y=j{xdy+#+%Ol=MdR*depy$ zoLW&}T0+!cu^!9}PMb#bi7|hhmPvK)Z+p?+h4#wdR+E0GxlI1%0NyR^d2BkzJ5U$x z$sg^&OMioZL%%bl`c?KO2FTuTlfLO5Me3Wzvn&4`DusSJ1zyhQ(fXHY*&h0zN$OVq zLHg&PND&_6l|R44)L-wejF`Ub+bsH~`EAfQpRXBQ-x*SW@c)dczU#BGUbrm!cDZ8u zt|EQ8V)L8N544y1M*kT>XbS>F271N%OY?&pQ0gnoEHCvJT@QweS$WoMlYC?8t9KXj zfwO2r>sQX7Kz^IS-$;Ik#oz2ae%eX?Q?#DhMgBnRLp`mRqP>c5GlLFJ?}+)EgX@F* z?VP5M=1+JlUym{WR=|Fui1}xYhh*O~L?GAiqTu-PWPd(B;zfCKF0ZM0(7OvuzlvXe zl24Dxm*Ygd6rYve$N5duizxp1Z7>yyk6kiTL6%qMt@#Ziw&MapF00U^0o+uO7&Z#p|Jk+KPakBDW#F>pIz&$9LpE@MnB< zO#a-HMC)sLvJybp)9inwf15Er+mG)=zME$RiH4gKW(1Q_2SWqopVVIphKu6u8NqR=8}T%KjrauAjrc^QTSj~m>M^`! zJ>%<}-6iA6nEqYGuwHZj@Nc303H)OpO-?GxKQKSfqDuCK{gLlyk~@lcNcNu+vp@Xb zJUb}cdf;<|$&RQ$A-_!zxU<~oJwo*4PrDs_ zeZ+WaL=S(;H2Pa^Fw1RhU$oopruoL{xj)(*u#WIYj8E;O{`g@UdnU}0EOIuIM*J1^ zf?l*wqI?JWv~A!C@XFV>G#BzSceFpqpB|X5PXT?zh50;Om6{&N)~8Z^GA0nl$9Oqe zsPp|M`h$Hl$zA+3A0Tg0Ff%h6&#e3q|GgOR`ux!Pg@8^3!oSV_eu(CiaXw)EG5I_2 zruhK8$=`uD%m?TvH)!^M`1^FSFVAMn2~^kc((&K)Bk=aWmsmkU+q zc=jUl$K5QI{2Tr>j=fPu{*+2B{AGT-h$mk}jf!Et7)F)HgXv_y)aOtY@tpLXi0z0m zo+lIFaA@drNQo$z`B5YZ-&6XzWqGh)=j#Xg8(*K#nCxlI&rJCU>~|o* zkf2EWSrgTJoh4vL51Z=BqZnerXVXZ$2E=U(ZT z=QC0Ni{&%?eBxZ1{f7Aq;@fremp_I7A-;Xq`OD~h#gY877wMB%&{E_b`l>K2(oIhqx1~%ftd_Vmu<}2h6ar2cKW8^FD88^;X zvUtAYj-RhUo-8-T_h`NnG~^qj^A*HXcUClC8HWBoc|MXw>no3!=+BamAYPjC5eLSb z;Re5-ARoyR3qHl?QTfrGn~#wH{D;g(=9%Kx=>0ck7btT7eIUr}1mXDy&hN{+ULG@E z9W})(!}&V==XT;%bpMU@&Ai`Uh|d?s+HddX;;8kB$7ecHqtAQu}{rDk#-e8~1_v73@{tNBLW9K~>5BsOs z`4!gx(fjeK({3E!rf`3s8b7`veoUFl_g`v1K03ac&VO=)Q>R$sTkQPDy1&Kzy@~zy z6k~iV3Qo`&KEM+DLz*9%6DGn!V*j0u{dd*``h3o?|DHzsMMJ!E(uO3q|8_dZ*ni_d zh05*3yQ`8nd$0|)} zmi={_j{QFS8VI;QWB++W{_uG?{6U*>CgxT2wwKBl-CG zA_*zhS35ZfZ$F$){nN)SvOg_${-1Yq@|w>N(%2WlkJ!(m1DyYqcYhSiPw@GHDL?rP z>X7PQxu=198n~x{dm6Z>flplnba;+XJAiLFu%2IB`~8PMihT8@zUtp@{=0wpdF8mj zYW(QJ-$V`-vDtl3^}1@^eRFe;_4P5#qs<8*>0Dpx5N>FW_J>?3I z(8WAZ$LbpXW*NMXbEfeYklZxos zpwlFe#UD5owwfj3T%GHJv*@Vk#Ath=>*DgZlz1~% z=&1y4G-e&G8c>5CN8yPZ(!m;CU-cm7o|}76FKy0xHIEHgRgC-ySaH3cqUSggf5%q} zZIpM>*lIyo(Z~mcx46ZXSCFYkwU7JOKH$NIiT-U%eTinqdo1;8OP$7}{`nGt{x^yfZ36M}Z&)XfxYahFl)gB%%zmqV z#A|aiyr1DK>R6-QOiyF=Va3Pm^Rwf|%H3FX<4o+8F^GGaa4Yij8~Rne_G*&Q)EcwB zX3|QO8)3%l7^IkXVOpg9Myji%?g5Su-FNXHriGgf-M#dZah!^vLVsu$F$*BPi& zSEOqiHAmHB`wW{-qZFsEvlqJ_OS!qxC>I$^H}P^6;`CN>0XNt+=vYT610+^RBc$U| zS!|qK&*Z(9JdgS@T3?J{7@1i|c7w!#zZdJF5BQjAjs7gw$WLR;&n{}lc$y%FNTXXWGtP1QoK>;_hPtt6&JBW#fQ;xhxVB9Ptq)_yKY_mDP9`F6i&* z{Uzt$e3tilJKOub>(?}&j5{_F>tj)G@2DM{2OboQ*?XBW>Zw`#maeZZ!(N;5vv2JR zypImUvD;Q<%n|r!B_8n^79Gi2&e4UBoA3kT`*C9XAM1bg$vBt5(7&$n{(BT}^u04< z{}Qjz{LkFh=vsvf=LPde>H2~jJ%&r|M8);)QXmrg&6X4@cf@@ zNbSZS)#c^F9+lt;&;NOOIb&PO@wer8UGMonq#&{dUE@95hv2`YHxwIAf}&roLtSb2 z5y1JuJg2(;+Z^5pKN>O-E#>ghJ&5OYaknv{?#FJi(Nb#JmSU<{N z)N{HNEt>Npp3{}7z|m{;Ib9N>=%WO;c|$XuAJbXddt|*K`hl_l#8w7Tr%gb-_CJ>G zzGP9YwLShA(sITn`dvYPq>18?4QRciu@pN>Lu3zYpVY2ePWUFO7>bpa{6c;vE|yeD z`+eE|xS=iGTJ=u;L+w2lHX5bL-M~>H&m-Qg?JEe0uV;!&?ENdUeXay49b4Pee(s4l z?T=a7tDddxX9$J(J&HhHPs#Ql$U2Qr-!-weC%;D*qxc_?Q^aj(Pq(VQvg09iOfU~2 z4dtw5imCBcJ7xdJ(D+8}6&) zk^Q8{i*{%KSX>v~FRod*ko+bh)c+HJG(L(6HF1*wQpd(3X``TZXumhk_+OFjb0sB> zPsO4>=9BF`mhq3t_R4>$J-M{C{US`>r{awN_01qgar1Y^tm7*`szm!p z|Ecj+I|ZK#K)m*E$7xSf!rGpGEDBM_4>Ozee_6Jd>_;zF{jWv){~AYr)&8<<2{+~T zw~kLfK|0-1F9|J|OgW&ss`doF-F7^R%lT;;E;?V;yfuzTysCL&ZKvuU=^(~-93UGy%4@mhfdQOV@1LSlLv+VSV1_jg>3C z8;W+cR94ll-C4f8W%;AEP5zp-Eh`FYAFbF~v9X}0WpnHD{Q4E5ZAr`0mimn)dA`lO z-<2B+mUq>+F3kfE{`!K&wcATpZ*SaK)3Ise>c>~MuFh{PukC7F(Xta`w1JQE_2uik z8VZ(n(|C;=D%(pcc9zu_uFq>J-&PqNw|v`bZ}HOhW#!w}Nt*IaZR=Y%7OvlZQ*;IS zW6>?&+!}1!Tv!Etw3d_1TQ`&L^4rS_wyp7&tj^!mx^3C^;+pPF8;SyP$E?`7aTEBi zFJJz6Lw8X#cA*7Z&G zn`_!jH+gGo{WVKlN}JltHf*c*mXxkqUcNeibNRBaO|30mo6GSkT-CLCMWw&7b$vJV zSFt94{o{=*DqA*fD1g0d@;9}ut!>&Y?6tk5ax41IYus4bWwnd9)c^RV=0%Sve^b}G z)|Tc#-Hw8-g-i2&-h!oD^A{J?=PmITJnX}bufh9pLt#OFUY($8Xx{2=@LlgGR@U*~ zR-s>2!LOa%P9^UK&nC|=T(oFWVP4)6PP5$G(&4KU@1?c)K^dR7H7}UQ*sh#A`tiv* zf8^fvr7yua`3v&7iPU1OJzm4K-P5&cso8jt3w}RR84%Z8!~Hzm8~2z`(t=B6UF!Xw zXZ0iYKTrH`j<2PD!})^fndloifq2_J20e{+#6bU=x?kZuI_?|#3wBv2x*sSUuBX(! zg>D+)=M)atGId|(bUmwo!%leqhrbd|{qkbK=Nf|1Yzv;~%MAGFuOh8MH&0@;pBhGFNX+t_P5-*~!mL_;j%0Xks8}|()K7On)pBH?xkSMp6jf68|VHBsUxS>SmRJ_T^qmOx*W@Tq_sw~V_DaAT@?Ou}hg zoF+!gad}7M8sTM9PU>IrMx-kx+}h7Z37>B1N9l|DF`g3j82YE@oT&1g0dAx}ejWG~ z1KcQYuGG7g&)E`gm3KfI$SUva5^nA1XA*9e_m_r#jQoFOfE(q_l=FwiHO9!i>%fml zIPqg#&7PC+hb-^`1HI9{uSvL-e|m}z72;=#xaIGQBwUTFe6dNw)wl|N#6WKx_c;l- zj{8dqw~m`C=PiwEoG&g3A2nYjocd9`kN{f@aCyYT3hC9j)PGxh}o z+$dL*geyHL`W6Yd%KM^(Tjed2AxDjyC&yiBfE&jRO1K(V(eIXU>$u;OaO=2~w@{(n zg{G9!9mO>&3hv=G#+u|+!ByON&H`8QVZZ`cabVa2S4*!t|5W{}^?o+^qoUwF(tax} zaJAn1EpWBoGm&<8)YUh)c>~=HdxS>sN^e(NL(`-DCnuNTfz0@INB$d6^mC2uW_KRc zQC)LeGk)f=zN6#TnzePc3Wl0}9c=7Pj@&x%pO+k)`Y~Ui13wKRMQyGRK<;v1rw|uE zTtdHNS-!e=OmLw+Uz7Z~ zyRli^it*^@E9%v_*wupJZ*up?ZP3#0ZLI5TuWRIt34y?# z;8RqEEJg{Jw+9r#t&YWV8*jp9-oUnwx(>guVf62c2HG3FonE$pzYB*HR$2*3$~a1p zENJNH%wK%d+PFjZ!Z&Z+xIdVBI~#aYn@_%*7xS@0{dVge`pd3jf&GN_@Jk+N&ITKL)mrO-;^r&^vy3T*^;u(38BZ?f*Z8JsM~L literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k317bebfdef9ea155a409089e0bbd5030.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k317bebfdef9ea155a409089e0bbd5030.o new file mode 100644 index 0000000000000000000000000000000000000000..8106d0f423f259dca9e6122fe0ea0f172be04fc2 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZARA`l+13MuWI#e7UYl9bEj37>N1v8t zT85kmk#&s0V8-mkJ`-oj#FM3E$T&NhaWtE=kvBfXaWI25*{n|{=Y+@%kpMHe<8>f0 z?*0CHbl-c`Ai$p3uoQIPTmS!8)&Hvc|Eg}?{_@fl%XCd+s%Y$07OyENW7$3Ndk??$ zuyjzmS$9ppt?uW!ZrlH+7ur5Mjj=<6w%p!#@j7eEo&B%4p2O=mxJT3Rik7a|cK#sz zyD{wQhwJt52UWK4H5)the{HPyNIN?O+QZ#htmu2Wp1u!D+1tO;*`aCa-8E-y6MF}3 ziER z-s8t2k2${+Ikx|JWO(k0NblT}k$tO9MK10;6&c=lDf0B(_aYCqoQoVg`CcS#`Xk}t z?fVCwe&(5h^tlg*XElE(a(>RS$lgmojXdpqE7ElQVx)J)CD0Xx!v_uwY&*VxVDH>x zk;+rYBTcQRBE9?HiVQz=F>-$UMbIn?S2zz0oS*yTz}{8pXWw@svsy6z$+sdE=PpLJ zU83>+G#s9b@m3u`KgT1pTfQ9`2EOv#$;hk|=fFGl^C#iqeFp}fp8L!|_~h}(dAyn~ zy%m{#;zXpvc`0&!?m5sb3D0VIa-bKws5o~FeIJj6kDrWGI!{Hyb1z1oUPU}F4u_BL zADDgO$$>CpxvLL-M)be!1gAehzvh?E|RL{UinG#S8IkZyc|s3 zo%{XNH|rzU8gt7mKzdI`5#XGJml11Ic10sbX?V6(=G z-hdBi8sR4N4(g7_sNE69ie3XOkGBzh{SbcgK~-t^!}TTbJI~c?J6}I)^xs7O_<>EE zt$dRFPx)RyeD78G<~`rLvwg4o2EKP>6yG~y@jdcM<$wJa|5N^YBOkoB^M~PR>BP5( z9?%b+|L2mcZad4k@T-!m!!IxG`}G>umyY>yW(K=D^h5pX8}RGFTz2(L>4Zar_1d8` zf2Us?+Q<569x<)|(dEELpJRQ|e_?$WU)8U^Tgv**-p8)~>q6FdE`wb?pTe%DfwmZP zZD>zY-_Sp^zM%)SLl^#se)Yn;?9kBPCWVK-#SUGl$C^5h^$q`MB4Ekkp$FKZ-#h@? z|CSOyvwruZG}plMp=&!|GtDKjz6*ISfPa!@2)xpm?~zQ9@gih&LMDgn5#{=bRqYP>3O-;L{jT#w`Rzi@vF z*MFn&VXt?eOA6aySLJ`UvFz%*(7$WC9(KV#7vQ&Kr(eScWb^Z|`I&{RZxHq)J1C!i ze+}!4l(Igi_diNDRrbI9E!Yaa{2u)H5^VNwuwUFC*rCe@VON748>b3=ke_~BRUE#S zQgVgtcTXQ?eLN56fgkf1z1?cP`n}qJ)R>E1u16;*I@2@?@F7_z*mAra=zKe_9=uZU z%49AYc0}3k54QT7+gUX*`GnOm^YnCFb&c7^+BDt9n0*YEJXvf`X2P(vGfcx}vD%UY<JCi$Buw+z-HaX zckE`=Qyh+yLQ8oG@GMunpYA+C-~%huaO;uuAme2x;O-+CF}MV?uSpBf7hLri0%yM({b-PGnv2Z2vj{qOJWj9fD95ufupz zebu^(vWjANDfa(3M-hb%Nr?1eytWW1&=GnOYHM|Ow6!(2&v9dB z`@QXccj3Z$i{`l3(PduI5?0dL)ZE~1?`rSxx42t6+dJG1{w+& zP(b&Y;vTomo)y2pnd2N=V>+{MvBNHC`-olieIJd^;DI<9x}3@EvTi3!UoMW-Z8W-} zt~0l(PG?m{JoQO+8@jbkjxv1(j+yFgntfI}w<+0`ihU-=hf!RquvHdrE5X~8bQVDO zE;wQ}UUnB~U@-jWFZ)#CWwxBP%%0|8kG1jwFi?Uw zc%b^Sjr}5*_3X1p*<}ET=>{j;y*Js;$K9Bl&W^d+kGg~m>?z>UUrz`VNrGtnrCmDP zOuXpbHo?z*sGv;8>#UO<3fk>x!J_>KGT7jJmLIa+*ZuG^7S#`=us05}8KETGZzsLE zo!19*?L{HoMtH^u5BWmlqq(RhO@y{F-i%O+9-}{z!7_v6lg3m30|!&sS)ym0K85Ik z&kxzh(^;qJpXm8C6ZGgG`mm8Wf$9%9|M`&5Ko5R0q`klo`XhR-zpM_8_(5GK`2y7u z8}XYGngA8BiT+^^CwFqP2ly!pvhfr6_@dskGbzeG0Dyb|{+AWB zMad(h3k7~@m&Ps=1Df!U2(SGnh zm|;&Rf5t)?*bMyupB2iVPU|N8i?`1Y!5l-i+C-9XAeWWDa2QxB82>!cKREqLG^)kv zXE{lKQ7PZjE-Vu?v86wo_Oz3+k|(cF7Qi%p8|Y=zz>P`Q&1Q9uL7S9yhq??X7K*A zQ5X1BZFiRocr2ncexty%j*}KJNeIUm=@~ z`;zjv(5zV`P8>gDMy&oipD)}VeEy+5>C2A(IX~~%^zxkN8Lg|)=pJtWY}hweKwqrt zWN-Kf=5s!uuQZ>3k-EW;WWKZs|7?1pGGm*&okQJIf zo#G$&$DdRG%T4|F?3^0+r@;eQ-;!+PZ+uZWH6Q+k^*-ib7;Ju zAEbZ&i4@^6Uik})P5t%k%!=#V*J{x>&2NLg`Fzdd`p%O2ga2p6^zF;Vdf~F@+vSSu zyNdMXiqCI8KhR$48~ta6pe+ax?e7umFU=2bK&h`Bv%J(_Y&{q(X5~3EP4bPVub!RA z2hN}gtzUV62Kj9UeIV83J zfaE_CkMGbYdR3(70r`i8V|2AWMwjbMpd_&3SqkPBV{MEG&$BTT2?3Wg0XE7l6U$W1Q z{S*Gp^F@cuFG(I=RQ?gPIr#eg-&OvUYI_IEnHmr4nG;(Dkx%pWEY(KQhWiKPB|Km6 zvI30cE$RWD4gKXN6Kx2SfejpVVI(hKu3tS)nng8}T%Kjrb(gjre4wTSj~e>T$ef zJ>%<}-6iA6sQz8WuwHZj2yCYP3H)O(O-?GxKQKSfpi1_I{gLlylRJudNcNu;w?F*f zJU{1idTvje6VPW-Xo^nDEhGJ%{;#xtO-W{#qddPEoC^O+&5rrk;C$={^0Fa53h0Mu zzv1BilN#Hf$Q!zG#=*P4kV@bAPlsU>)I)7@yk5{PDvK_H2YBS@cXagZL}z z0ljFSLirBzX}=J{(CUq_4%Ro3jv)7gnyg;{UFUJ<9xvSWAb<4 zP4fYGlfMIRm=DlTe#q?q@b{@?U!L!P->i^fK0sc=$Nc}IM)?%{3x(q2OZXBZ@-OB_ zo%DNlKI%vps6&$HJni2cNpby>f21Z^^qWmfRlq0W0r-VB)cmZ%{7yn03XjeIMn2~^kc((&L1To=aWmsm-AKT zc=i(V$6YL){2Tr>hP_!u{*+EF0%ZZah$mk}jf!Et7($iDgQ;Y{^yg6(@tpLXjO~aq zo+l?$|0H$C6u?Iflo$z`R$FN^gynUUY?0c#q$|{K5;g~e#86)@$I_#%b&yl5Z^xQ{AFam;z)hPi}X|EBbaY42l*pEUx9y` z^OgC)8`o!AudMlsVf{tC$iaGzBzcy^hhNJ=UZ6err%Ouh*Eb zgpaM)qvk7oep~Vt+Kf{X{>b+e%Koq?&L6WU0vmNWFu zIXqu+C(c(OPmY`7dn{iG8S;&h`3mBxJ13T}3_*XNJRixS^_9m<^k>OO5HC&nhy&wI zbA#VckdNeu1)t*ci2Ufz%}2<8{=dve=9uEw$o)5E7btT7y+6e41mXDy&hN`RUl}!C z9Wli#!}&V==XT;%Z2yh*&Ai{9kIxrI+i&mU;;8kB$7ecHqtA=-D4yKM=ljI*ka94? zWr|N&KTZ4VEfk-U`Tm^8Gvr5^yX-uF!hRrwy@C97S5h*?^OW%!4$6m$LUsz*Nj%@h z`X|ndj;v>i;u+#a=D+j)+Z4Z$Pw7Ga=|k-E2g1kV8TijcJfr`s0k$105( zmi={xj{QFS8VI;QWB++W{_uG?{6UCgxT1-ACBl-CG zA_*zhS35ZfZ$Fet{nN)SvOg_;{$Fr&@|w>NGT0ZvkJ!(m1DyYqcYPGkPw@GHDL?rP z>X7PQxu=198n~x{dm6Z>flplnba;+X+mCNKuv6;P3_qb}^y|eR<_VzN?XRm#3`2ByWi++ux(pD$^b#GozZ*Lkq{o_~v z@cZaHcDA&3^|Yh<>e{5$wb+dyr2lj##q~n^;;5#{FD>#qs__Ts0Dpx5N>FW#J?#pQ z(8WAZ$LkvYW*NMXai;wq+grx=XiE^tH6H_gyy$t(WR2ATVADJz@s1%qxYNcYqdjE+ znl^{`r^b6N$&PwSdPy?9iqk!It$fo&!3(I-LcwW_E35>N;t^w1Q_l|(Pkf9kuYnQj zi(aA~=WUFYH#B2${GK!`p3)tCv2(!{wi%x%;d`oSNDzCpWUw47iSRMSx)QgiT%H$q zVmy+6QNTc={&}uX6rRL*k!t}D#UyGU<9~Bp_sRx|=wtE(7+bArsX8$?ZyvkIC&8awqB|CKnhXS(RiM z62%~y{Zo<5KGLJe5+=?hO^h>0cO%Y5&OtNhHr>j3LY?b^v)HKU#Ath=>*Dg(v_vyj z=&1y4G-e&G8c>5CN8pJY(!pw7U-=;Bo|}764{gqRG>;8fRgC-ySg}t})AJn3zvnB3 zHp07TY_+0TSByAM*VCZtH}MJ-Z@)*k%Iq?Xo5;2?&jei}ys`rIGg4~9$Ro6g21acs zC*n*`uW?@Cn_pXscBO|gJnV!Q8V&{V0)Nz@;Q-^ys*Z28nCefVu9zfx460KdMicP| zb-LwCAMjwqME{PZzF0HkJ(ha4rB35f|9pu+{~N@KHi7u~H>{IK+-jRoN?)8_X1~=w z61BM*-cR!tb+plLrl-;Ru;LT-`Pp$}Nkd9NFFrYDF&R{j&wt2{vJIDoX&ymp!EPXVvYBGxXOW$ohM z7B;M1X8Lz5?H40!HumSS)T=FZWUJR-yNs*7@V(LE+l`_DIga#_?=!nT6;&Qkq1Dr< z=NjsSr#e@rHYObz9_;{Z+BC4STGR7fh|Qkeq+<<%87n>B;=2OM;bgCF)eGv>>onA< zE7CQMny2dVeTGe^QHs;o+KXL}r`_CWl#7g}n?$(^ae6DcfE(-@bgUzk0TM5y5z_I9 zEH*)|XYyW4o=5!{tuIC}jLfVfyFp^W-;4Fo2Yk%5Mt>G-j@X7sJIGGMdglDc-2_dokR+ii=pG;={qZp{6jq#vPl8_3@~;XT*-p0}qPF>^;mF_0+6=Th~{WVXw{j*|&BD z-baVw*lnva>InR^5|8){i;iS1=jg)6P56HC{W!7xkM}?FWSmQ2=wH`(|J{l=`retb ze~DLU{%3Bhb*;jM^97nuqMWwlz$U>njIXw|i+L}7_r>uo?0d6+`@0RcznL*-TES1I{`g=Yf{ zq;}(v>I(8N`_Z5Ho?9}Ss^mT>sU9>jCHxLcd(#dEr{OfafEg(+oY;z=(?qN#DC7R`AP&*{om;Mg_xoGuAb^f7|lyrG%SkLfJ!J+fXO`#@O$;wyuw({{9u&K3{^Aj;-x!KlfyU z_D3!4RnOM;(}Y3-9z`IpCuRE&WSz#R@0wWKli#C@5&ZYdDdIM?r(4xt+3_GcCYXni zhH}O-#nkw!owEPqXndpgijLY7owYs9b5eoDO&KG{MCr~GS87i*-J%+M$Da)FK1*W| zd9zqvX;4}CAG#Xq1pY3(j(DQ}O+~i%n1B{11CG`g;=jZ~he!DH@-suniR>4EW8&}Y z$bQn}MY}V9Dz1y}7uOtINPZI$>i+>i8Xv`k8oyBhsbgc2v@y_HwBM6p{IAIN`I3^x zr(#hb^UL-g%lJoSd*#2>o?P16ei0_`(+S4^`X&&gxcNI{*721eRib?l+IuYH_sjOm z|J3-Zoq|sWAW{2w611l&VQo)87KNx2fSFDDzbxBJ_G1^T{@0-We@-C3YJXX_gqw2v zTgNA#Af0aZNkYpdTMnqMsy%^kw;hk-a(-Hdi_KRxZ;j&-uWDXc+o`%oI*73y2g&)Q z=+srsFLhPzwid5h*IvA|Sd`M0B@3E6>lc`WHANE;S>1$p6#7eTk@^ww#;x#KvT2~d+x2$XQ zZK`Q2-RP~Y4b&`YE^TZpTfep1TT;4mS^28MP322FH?}l)ZYsxX;mXcU%PRv7E$h0V zzlzm`>z-&>UfH~H{Q}s#rf_5Hn%c%q!d}}-Dz~8Tf`$#1omRVeO9M}AY+CfF@;7y@ zYiVu@)@@%{_=wN%+k&T-F7bN{i;BF93YHWsUg9tC`5KA}78cYAy85Oq-g^J_eqv=U z|7{ieRTcc&$?a4MZt!gK!i9?#Em~MmfS*sn&!{Z(Hn;obduc6xP{!|VDF_uXwlnXJ zetdG?pSX8^=}Rz9;k-g_BDEN6Pt-7N*HmpvdM+O1g5Qr+2E;Yra6bq4#y#d!wBS-% zn|{CNIsLHxza;;q<7?^PaK0#dCi(_WAl`P3LQi8IHqgJW?w31{jQNKCqFvUB?gt8o z>q&KQp_>BuS%t&3Ox>3`UC-&?uoIsD;je^Kzq}am`G#OL(}E}ZG6O#Lt4J%HuZ*Dq zUKA@W(KX)$o$B6teF<&t?0t14VM5CX$C+S44yEikidfLZR}TcRL0oF>1m}^515|vV zN}}gIujO5%QG$}_In#okN5l~=Y=&_aQqafwF~j9fNC9yb^v0dHK(^tMW!SkXxhPB$ zT=~Dy7h~UqTnZ+75byM*a-#Q0pz`|&6q<;B<#phHK>I}Ww7*FNf8;vw9{^7LoaT05 zOwB9pI{d$Y#Ea;wr3qe_a?qHG#(h)D>y{=^^Q#H;)X!xJS8W$axZ0$a$^P?XKh@G? zO3(iw`Nz|xcv17}ImzcM60YXc>$0DIX+Q;E4|t{;6g{m*oeG}f%i)`4{htJweG6Bj zFTgldPKXP88&`r~k#M_&EBVHuKjk}w8ZYq^Ebsyep9HvSOQ0`G_+-G1TgF`mxG_~c zD&aIPP7`D0n7pHLjqoxlC-tv*BhnQTZtZ7-gip2fqx41n7*C114gJ$|PE>i`05{Sf zyAJ${0dABxU+Ue;=S&H=%G)muWR>>~3Agt1GYPlK`%6PVM*crCz>V@|%lSj&8e?R^ zb>N33ocJ-WX3tCbLl$_yf!=7}*CpJ_KRrc<3h^^Z-17HD60XKozStoeNPx`-xIE(Fave6nV}INO%py6MlGn|t z8T*0(Zj`H0!j&EreY1pH<$X!Qt@4)1kfX*ekmIf}z>VXEBwUTF=yyrDb=>bsxOH60 zTc}X(LQ_iVj^Y{>1^4h8V~z5v;3{rBZ-J}$&~JgOI51>^tEE?+f2#h~dOs8VQBm-2 zX}{$bxLWT67PwmPnMk|a>wHbE-e4EQ9-+az!rR$e-}o5+$;qX7ATxg5k^jaM{ahov z*_{V<6p1+M;3(QTlJR-;jK0y+=`&0Fkh{-+hqF@^2_$mS{hjS_XD5Eeh%qB mm*wzeKXL`tUggmHFtBxOYI3HX-toiZQhs8Cp5$e1|Nj7xPad}b literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3399a017f924826ae7af74c8721d0afb.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3399a017f924826ae7af74c8721d0afb.o new file mode 100644 index 0000000000000000000000000000000000000000..661b5798adb20aa98942fd71d80f5f30c9064685 GIT binary patch literal 88552 zcmeFa4SbcwnKwM=gXH9c+cpUykVt?$Nzfam z^?2yE_^}#kp9P2ZXEPA{!4Ml_rK|bRJ}79d-z3P@yVC)dD&N7_G?^U!RI&lj;vGcv~+#% zuJ81}oyA7p+Uo6pbAzw{Js*4cKYi?EUpIRgX+O^pJ#b&$!VJbrHYCS2Fznt9bV{r~jZ z{mWUi~H|=@V@6yUj4xH{iyr=j~^RuN1f-_ zJv+P_b)CGL(p=Yn?y>uxzpvxL=jZ<9)bM?O_W1Cmv$Fsw?J@)Ky|Es?pzVAUwv#kHz{SQ2U5^3%&|Hg3Hs;>=aJpRmZ zX5BNx?cn+Ro>!2js-I-F=J7v!{@(Co!(|VkuJUJw*MxsO{7*x_8@})4&~V1w<^AVw z`polXr~d5ub2p)`@~;h_oA}e=HEBNrKd%m-pZ5yVT;IPs?XxK7-sjKVe{#4i`|;ua zXMO^{o<;k9H(a*vb)>nWfAvFu_I&>XCpnGcGd#r?S3iF5#<{P4y({CM;xid((Ty43 zFSe*(BR;kyfIjnhqZ@ZX$FA^1M$Y=KI7zf7FI(d11871Yu7JJ?ooMo~B~L*oJRYK5 zaq>lP;3mq~$JmnZBd&z!5oJ9Ay?JxPivG8@)<6#e-B;#NEiRObTK-HF7~C+#Xdt9Nk?sc95nRN*59#o^4zR}z2{ae{MmbVdHUbo z@9lqY1$+41*17(7cKiA-Hu|B*6EKdZ_22mGJCD8r-9GWd>VZHcR5b8+(E;|-#jC!v zCo=M_>mnm_{VaDXJG}8tU)st4^!Z^s?BpFqX(LT}hp%UTe{Oqz+IwwZ^9`K%Wn`pj z2EIS*89DKH)dOu0BJSsrk+vTIe--$4;GY723;1)up8)<5@JZmqz&8Lt0sQB{-N3Vf z{~q{3;1=K<;BNrOfNuhR%{y@7tH52r*8%?v@E3u10nY;d7VtyBTYx76e+~Gv!0Um1 zz+V8q8+aA)8m2wHVi5;97v|zv4O- z*MGrvGOkbKnv3iAaLvN?Z*fh>^=r8LaQzCdqu}EUxc(m3`*3|5*H7X42CjGF`Z})r zaQ!W=J8=CCuAjhl6XbC_+A$B;E?j?%cCE*?1@t1&8*!b1>wa7-JtL!s+0QJDjZNP2 zhj)4xKHU0Z#lyRPQ1S5YuT(r-gZR+`Ht;LV;iHq;z^IQ6T+CtvG_E4w!_BY!;QFQ) ze{g-XH+ZV~HGDr$-*ZnjzlHC^^gZWP^ZWSz4t<|^it+{6JsUGZ(Ty`A(T(koPk1l1 zZpi;saWwDFZ%r6!`bF$|R)4~O0`q?8Y~NMv<4^p$>2D{zM?5xp=RLgh2UkDb`jx96 z-W9w0;oUo~et0JMvw6IZcqJaeAMyE;oHsb1-vSSROFU$sYW^O+f1AE%oofC^eE$J` z&pg%qEWSTY-!nL$LpjmBJGM_4IrxbQBkk)ayhnVo=lZU?V+;7)X!8^XPi5e#3_O*A zr!q%o#0$xacp>@0&iOpX`S}d$icwvGQ_T}BdQL3Dr zzTATPHd1~5Q_Y{i_wDrEcdB_8zVD##-c!x{@clOW?vZ`TdNYoFcH>Qu{nPlg-{*ht z&{6*g^9)|!&icCAUJO3Gr!r&Uwrl+Zch%r?)IZSvpnv3Eq;2~G+7j~*G}Xkek3^pE zH?8u%*OAua&%tlZfX~<-s~xye#?3_@KgCf!D04NgnHkZI;h87DnB`#u#TiV@ z!S|$bfAE~o`?I+i4{!89PrVJNRx>sdn!rvqdpDd4P4M8l9#^!*=U z`kQ3>H8TC`1nKKftq$1f>ty;>GX2T~>1$7|&bHI9km+k=`f8b;JzQ4Je%1*7_mtNR ztO>G_dGMF^a_&X{674hjmMd=g7-$&{5Ay$m6U0AF>8Ho1{})Q1ApTz|{fzkZzo7I9;$Nin1@Y-$q4Wvj&r|xs z`1G$)`ULUAl>W;2^lwr61o7`s`l9&s@2?ohyobkY<3c9%gVuUY-qMG|p5pI#u-2ow z=n2gIZ*EuxpLQjD+uHtjcCUcXThssEUDf^PX8ovx)@h@Meei_?h|d5$6X~*$J{vyp z#Qt;e_lDq;{^o8r@XB>;tj*vKnyR1duJhG*c#AA1IV1HbJ;0qP%RBmdD-JurMf$6ed%*6```Hy1D_G_9~l`rj56K2U&RErR(w1M@cZ=ldI;=%+ZWT~4OlOY#UkX-mTu`QP90 zeQoVd`%7whY`p19BmGoArJw1Ea38lVGjSd7-Sutgo(H;zwHW5ji;X_$pC9@+p?}nu z^RocT%A8-_pS`%IKWF(0Uhb!!j*J9R_Ah*qfzyYJE_$k~=LRw&ST9|9F`R+8^yPC; ztSCMcMw%18!i)9%dgaB{GppzJ|FH1Pi60i6S&j5<6#v)P_O~I=KjCv?#f&pYDDLp| zi$^$~e&z(n(=MLic-on#IG%cO2<XDzr<31U+<)TmWX|h}6_d|ogzEaskbW-G^smS}lkrSsKVGjOYtM;Z-nk#;~5Df-Oz6DK%jID z=WVD((1#XC+E$|7pGth%W^eAF zglh%=eoKEYuI2pumi~#j&gb8^_GjTra%`KujoLXf7oW2~F6e*pvorc%erS6CE00X; zfAcS=_V*qB#Ko_qADa8M+)wsD>igKmzgp3F=C6PFu`}-%cn5y__2T~D{asQ2>;G_N z|9||ju>Y-}=l8!2yS@lJ9ogg=c;~O#$nRHp2HyQk*eUFOXb#4WEL<0n{Z@|*JwoEZ65e=b=&OV`UL4v$c+(k-E5MsB z4%HFfd}in{;msF^ZUpw-a&ZXb`-vakat8GbobYYAIJ5}(Fs=)TzV*z|Cd9A6^(M-< z_2SSh#IM+P2Kj)uT^uSPeCwH^d4zAhI5e5?_A`hF-hOc?i^}}E?z0Bv!+Tu z_pr1Gt6~$t(F{si-rd&S)78_w1K;fH!My_WYh!oYzMU2;-?zV~wS3?0tgPD6xTtYw&%w^t7G8)fY)5lT1yq6?YbQ;LN9f-cKlIj8`{zQN70a-tve3vZrs(?yth^Kjv3KhuEoI2 zTADjLc6T**wsE8D?AqV6pDdoEmi_y>dz$z4RQwUBZ%cFM-qx;`KZYlxudb75vb+B&gRDMy$dU>#-5hlt@|pLH10mo+_kf-xxJ&Y2XkFx%l?kW z9j)+Q8+W#1{@dTuy0a0kc4Jp-&jBn6b~f%f*vR9VTKK~86@CDWpvdMPl!J;6?A_OG zR`!wW^#8jRhc?ubV{1l?-*x)$zxJ{xmQ8zG_m=P9wX3_er-C&$?zsGF7 z6ciX;b4Pn~cY*}A`E@pT^|bf2@88G05~mC*9e&v#m9IJ0ggX~RA9exEu#Cs2nbGXy zJbq$a{NH$~oZPthA0gh$CbEYAB}f%AMX683M`gUdS&CTHw9Y?iIx zvbuJ4b!f%@u1@HZ2-vaW`)EsX-9=HIJr?YJ1?Jfh?K-u0G?Q_MdiL+{=$;?K=-G-r zqR_%c<%{Qsw$P<~$ud@RpuJ;fsQX}dPwU>$-UHn|p`ER}+V{1FHgDLlc2nb)l^Zv$ z-mt!L<+jZ$*VnGBWmj(qEec)T+||++>fYP9u;QBL5UM6#Y2OX^APipju+JdgzaYSH zW(v~A_elW;)!~7@+m=V)DGddGp8I_~+kgIDe&{0dUE>$^O(l*Pt7LEb*o8^VFY!Wl zLBkcq@r9WVeE~bG;d1tZhOcJNX?Q++O2hNmpoYuX84b^6r!^dACp0{V^=WuEJEq~Q zSg(dl*%1xTVo?p3FiXQTS*M0WtWCqktVzQ~Y@3F!WDOcFWc3;@U=a<^U=_gCj(n%H zkcOwRpoXW?kfNl|XBWIGzJi_A@D%o(h9|S9fZhJiV`nt_Bz8i>LDr|G&t<(DJ%>d# zJdt&3IGZ(TIEyuCIFr=_yZmIZ2(YVf0am5qbXK9IPh%lqSAG*%P{SCDHSA+i54+&; z+WL;MLc<;wWh$K;38mlQ=Qio7Mh9Ty-E%Xp1*6Qn7F+I(K97GP>2C{TxRmhIMe1Bx<9dlB5+9SeLE=7%nCV?Xym$HVf0@rgqn>GE3 zz=FRpd+rkg3;x0^vO{3OUznY46LN^2}jP5>Mmw95yI% zh~pp&eNm(jb6mut5?62>VrM0;;<$uGz9iB|I4)&<64!G)n_ZAt@CW_-3z1&%7iPT@ z3;x3Fti*yp=-Zb?dchy`Nn*ht^jl)VAN2PxMS8(sm~~1l_zSZ^i3NX1-zU-w{(>wj zvEVO=M<4`U@CW_-UjhsM0?b!6mH7)xSuf2;S7Cmdjp?wGvD5d%&&KBhK7*L28lZRn zAm%&||4$d@FT!k#;G2HY=g$nV9DI-Z@O`O_--FxKd-2&#sZlQiUEi12_vH0md0i}$ zF-#HM@3q$ePf{iXJKkQ4)QV^)UP9+Cx}DyR?@N;2reE%B>@LqJ`z1=Hg}{rsZ2CdD zCY=dfp2L24aFwVMV+7lg!(tX%Bt*y9v$>#85_I$n`$`UbxT|&*wRv>#!Cdy#0@iE! zYa5`aqi08Rsr+7RLhT6+=hU87@N}zibOQ2E%!i;~EM~{8+))zwqD11(GeF^> zTMW75y_ceA>=J(4W++0<7p{|nInd)I2|V^+X1U()xO z1^>>j5O`sa4~hf*ASD6zS`*M!2owqZdblfn6&V1xZ`l7!2gvXc0{B;&!@N*mZXmg$ zJ#`1Bi2N*?#a!7e@m3d^CvCu)c}pjqoap;dDFYRBsA?m1+r)L#pP}X=B+%b~&1LKEwet)5qgMJVbPpT#Vm?9r zkKN&S$M0P^kVhb$23%Vnu|rw*eiUzS?9e&`&(ZL4D-clUkJzEBGr;5^W@pR=__0F_(ST3XbxPW!hdwe;oxqT1@E$ zz1PYMMk&AW2fft)(u3so+G|Y-D)!@v9lFAx=LcPTdI`&Lw?9Px zy8Y!t`FTO+*k3X4Z?}Ivpic?9_PM~&2g;uxqzPN}hsf{h59Oa0bnQ{(clC|(&j`{D z4K=^3pOn8a7E2^T)}< zp(l|C9Bw;*uN4epCYCst%U_ib^{*=rVV`dM65FTS9%28EeZMU1UFDa`clB|C?oU8p z_4Nd>1#PdP4nlZw1NsNIkKmq)_7HutLVpPM%hiWc%lS4>_4T^jM$4-fG#4j(_Iq zwWj9s@lvjT&G9UDD4*|1xctu5_(lCS*-HK3H;)m6vVE}sK=k);hn@2E`80MYthJZs zdq$fXJbf|vomPvv$7zq|FYxiHu$H~-vUcwBT^$J5STkUhC@Gu+pAhtG<>Ex%^Q3r#bagdv40bbl~t8 zJG92AuRIV{>g%=UL*G0~`UP73x;@f-rQ091ztE}QL+uYU>HtT7=GKNZ{=>DjduF${ z*IHOxq0y^q*?G5pu|umFG?s$2x2Z~hE&xCIhW=fbi|NWKKXzytyP9XW_2*hm9wfg| z1PQqQ(0oxD;G1)@KYFbS=&#Ff?9d`jKE2lBTK1~TZ?APtE~Y!B{v~XoR{v77{+d#! z{#>zMCxOZKdJj#)(%qO7s`mb8#;D_jdnz5BY_CI`v@=5d84+xqZE6=cn;&me@~|@+JML zSL)~E4=KFe--k*xdG%U11fnn(yZ&BlIrP^qpT^%CjsC&-GwjlPt%6)k&kq08{`pG% zG;HFUKdeFQ4BSlI2tV6%L)x2Q`Zw z%*T^7R^`g$xRpK%ch}ip>$R5EMx6el{<${5H<=y%g+0MxwdEB%be$$2@;{fMJOs{f zueGF>suSs{zm^7~v{~(x2Ynw@?co}uyqRo?UEXtIeskL++UMFEA20G5H9(PvD=(4W z9bdRS@)#+oO#gJ(Oi>$K&-Zglp35VMlKT8~SBRz)1fq-b-0>3iAHeX@fbu4x{a?vp zPj?kk*&_Xu3xKE7a4InNSD)^hM#GE1D6j6omjYr7qc@BF`v&Nrx_?UZkLu45=H8Vk z0{LSW#xuoVF~`@~p;cgv2dR&`|I%x%$c4M8^lwBik8J&|*8P=3HJUy@V$JYJ?fg&k z@dv(>D_@dlEwN1U#QqWRjEIe~|IFpLP0H`l9LS^1mB;x6^1%E-{33|#$Gk-0STR5=+#6 z?AM|FMN@2Sx4%hkf7RISk87`RRZ#=iU#>4&`;7JTYyZxPN`(Fr%Em4dI ziS*y-|EIgkXnPRhzuJD?Uo`zC@&{3}A`jgkOxXUAuK!`Xy|VrFHn!VusqH^;iR~|! z!<%S7_FE1A&+LDyT!t5+A94C%_^*fN8vb7xhRg%khhA%5F25fk=Qmovk-QK%U6T*) zck%H-@lQxU=G*eXem`)&BQIM%QB^*jmnfe~a#s)(`6&KoynUpSkM6(5ARpa7jh7Ft z?_GNuhkVGPSLXAIRPs5Z%BSiQ<+D^yCyIO=e-ra--1w17KBhl2*Y#&;zQ%gj)F0Zv zyCnILgQdu4aVq)rs`6o%D4%O-Iz&+9AU&_a)dR>0YZTaCmLBjE=L6zSTO@8t1)%~@nyH?2q z9k%>bd5mZN*8N%1AK4*tzMiZ3uOuJZpB|Tdgg=Y%L6=9ZEf3+(#*f$ARC#Q^M0w;v z5EMurG(XAKP!u0O%=wJOm)xH<{a4!Wy8=1f_Urq5nm=#*a{;~mbc9Lq=VnZ?u{~ck zsqHVl#P-ji=@fz54|~zt5BrE4kInY${w%Fe)%`!(f1u@u+kSmN@DVHMkJ@y*{XxC` z)9m&Ne|{121|fhFo<;r*)~S|*9m7832@ixd@(=Y zMB`s)!v%g`iSI|_hL$aFzMuQ1#B}~CqgL80<>PH|!1f1LXk&KE6C^6=?VWV_09Q^!WRG zI3Mw8RUSEcQLtmzA3HQryT2AYl#8E+;6eJE*uQ_v&Hspn+vO0r$iINja~b#Zrxs9u zxaG&MXXT2Sm(Rz45{UFKxIcZwtq$%{JerTX>zm2@6LmlyWm-%<0=DWJx=^rt;Q--m#+3TVM@%bVsi)&8jdQYSz0 z5Buq}>&JYE^EPIEVJAJ^f0$K3?HA<}zpx*_T|e1NiCZ7`|BqWmvp7HUd@9vH-zksy zhy6Ti*Z&am&xxzQ!bwm3&MR>D$4TC>mo&BhGPi!rFUPIf^UV5HedG6+PT2KtK>Z8i z>R0un*IHCS)01dF$s6}y0&4x0ZvDLd<%`Vv`TY=8e;%{zC;98+G4*$q(|(fw)B>8J zUHLnHhN6W4xMUrGM3KRli$>R&+qdRj>1FROs$ z>+%Qv%$EG&e9UocX_mo{cK(O!<2UW{AL8Y^`bYK8DscDDsQ;mVIcoWF`sR1_jpR3{ zfE-|#Kj>djm7g*`bNO96$KW^3)kl&a^zS<^zp+F5cuf80>KpODpn%4ETYg;srl|Gn z<1fkA)j#4N`u3zh2U80DpfAU`o*Q2Vk9XuJ~g zr1nAIX1L{%JY4%B{)-D}dF9F<MzL`{>%$D zKP2Dy@p(DgFZ3Z7=@%8a>o1Zg>?NeuuZ+)Jzv9PdS6`_-=;y!N^^?Bq<0Hni>kR!+ z=4a7guKkGqaQjctU40UCS3d;Z?LR?x`%BPW`3t%$KS6i&|7D-Cp2B(Qxr^C0lAk<} zjOR(T`?uQq2J-qe&Og_lf&WhDvrCE~k97C`D$aBB`)%ak)BVbwbieImfmU8VtxrjQ z`u^0@T_imU7?uB#*5A%^_5mi!gZN~5B{?`xIXRu4P~zt)VGnf&wz=iMRUEgTGtYNJ zA1AxxFZJ&glSw|dy&Si)NBzLrQ*eF(<o3jXPIObmGt(4M1K5!`>bzN1zl+$k5wtShzl#;jrkm+_+P z@?tpO7GyC|DK9^iM)GspGt<$E;?u~@xb-chS0&9CIg^G~y}+;Q3;vt}9P zm5Z|IFE4Kat6;G<9?AJBO``{-Yr1HM9x zmja*vBr7ZmPhsi4D9#^y81U>+8OJyu&T(eRe2HUlPkskx0t1mGN2YJwb9=JYbW$ESX_Dttj)%2!4 z9<_p4KB(oJ_K87VqwD&9)S9N*Z_Jvm>2J(RWuK)*x_#>UkMYH@U%Z_|8UJEdD*K&X zq}p%%_*tmgcl>yh%ASibfT{A+$ETxKNVE5trH?WOwcTs5Kqi_byIt|GCC~BktcR`lRdqjSym#V+>z>PXTmxgqDynX8Q)b^>*&$C0i{hR(xp<(}~KM-%9`urUZ z>GnTY8$ZbZ(CNDU#w_klLna8M@76+NJ`aU-``7K8{0E(`kDoZNUfZPG|1`t?ryKSU zd!MG-KkQw@;i7Vhh5d(0DqK43KkpB2|I^g@|8(G1F3j=Te~xjU9riY;+WU;jv_5m~ zzhH8$g0Vj3_Lbw3O#lZ^d-*uQfB0QQx0R-6AzHG4g7 zP2&9V=lP*uw*T+W|E7Ijsm6hRBCmHj<^8AyS z|M7ws)&8Ff+``*8Ui&Yc971|qKE2kJlWB40;-L0?3Ef}i`Y6xG!9E~9*Pf4BS6-=N zvTtk$iS<102iM*E9RLbqy^j4s-Kuo z%F9#d@1s_6v5w1%bgZwR$-h+nefS&2rhTgPm=$k7`utC)z1;Bu^BEj^7so8!p5pDN zQizlOa{FIo*#FWz)&7rKrDZCPS=VUxYOaqH&CmM$UzMlY|50mpnTli9GNXLmJ`=4k zbo&qIsrG-=Dl1cQ%+mc0+MiI@FL?f;wo|wNLM*rt9Q(gAq+r-T?7L94|D~ms8m^jM zA+fOkaM^s94*S3M^3MOt^Z4BVFEaiAA{JKd|7_r9-k$N={|fDVEuN2^j0s7x|KiDO zHJpe20qCc+|KrvQZf`4``MwC^RmX#3*oTX0JizTAF&^;s{u$1#yWXFv+~1}3Klb~T z`{DL@kmuAle}y_fAGc<5er7u3L5Sl}((zzECLq_|j#@Kks^uNEDnsgc5VNk z<+Xq1{U6-^L#F+Q`151V`u}2J9dFNg?SGZ=yds{zj=J^@``TR|{r{`rZ@B&&-~V&YZ_xhs>{Y6L9<}DLRPk}^D$dVU z&Uhg9|6Tuo_H5nW<{0)|ewA8Y{Q7@^mOf@J=hEkYU@x#YH~^|W)9GH-{!6v-$DDs* zPuln}9{bVP|Ka(%z3cNO>`5CRVwP?{sqMdPzHaaO{Cm_|tc?%x{!ED`FY<5n`G5X= z-QKU(#*d@cQp5k({TVzTS{v2vpI+yO;Mo66;h{P9Us|y|q}qSD@>&g-Ev}MS*#G>c zB`zKIf5YXqf8{(VxBn8;{!94tppN|qFVX&MYMJtUM6Xq?JDvFS$(t2? z+^XUBRpZ!yHOJMC{gM!hqrv`b&fbj@>3uKU-sV@U z_If$&UoSt+*#9%_{{~Vn5fa+}ji6ml``5>tn037||9?>XkDs4)|Nn#9f4qNjdF?+^ zdqlVY62tyy#o7N2CA$4D*Kpb5YbCbrzsjY<{v(&y{+0KGar-Yd?Z1>iFYVZWIIx;) z-+1T$i1Iu?+1HJjZ`Jz&rITrg*0ukuCT~;laVx^@J>uB^jU39Ct&w74yex}_T8Bkx$T5Z^WS-C#{$Is6?-P~`8 zA76C)pI@%e|MBy)PS@w>)c${Yxjz5v>yx8al`;S8`wcYz)8lfs{$l^Hl>Av`{=bU! z+s5$!XIHK%RqelQalMA;FI_INu>bO^YhC)~pa16=`+u`d`=8Ci|7FI0o4KB^oU4~_&i{)H`(MkY&;M}yuQb;4 zv^rDmKWvN-rhmTJuz%A&_3%n>`Nj{hHCymq!~|MQow z(QtWHy~M))E3RGc(qaE~A87u^ex}FPAR5or^FNsXq0e)RLAT$hti5kb+z-I;h6X$1 z!RA0UFK>L~fo6|*KdyEj2mAL&t>xNzIDQ^mIgiin(>dQy{FZb3t9AC@mSaLu`}2Cu zevVsVo<5w^Udzk1^LMy^O!~_MpZ|3Ip!2KL9|-*XAN+-|IzE{5ImQ!hK7ORD96y)r zj(_k6AYYe%=+6`}Lkj(&^?4vd)gf^E$MRK4Z|+aiexkk~rjH*nYn@@wsn+Z2e2w#C zuqSoAG5r%7FLZm>*Xxg1Rq!9&_Q$W^7u53f@dom|!kCZC<}Y{8 z(_wxtuUxC)s>N$0KF`nNT)VX1rNbWmN$nB+b4llM`1w2K{dTs!Y3J`A2$00Z^P#EF z-=(&<`P%t5Zg0x-C;WX^&hsqf|LObF*bme8m*VYfIk&gvj(t{Y=jW)ub$t$p_4#o5 za^0TFjQK9k-!$zfpY|7Q{|oy$#`vJ?N6boPALjmc+b+hBhc*RGLRoOi8UUhmRjZ}`Cj*mt7yu9tfLEt$R1^V`n+_GDlg@2tx; zzs=XqlW}{qpQplj!R^g?e(SQ>n|59;k-h2m^?~he+DG=rB3xy-zwqJPn{obZdc3{4 z=f`M%t6HmJ*qg+Ty}9&1h`nj&%iayp9n8x(ziH>UxV_oW7uxpbJfC=3>`gl#mdM^N z^ZYjBBYXS9+M99St03Oq-1Az(-ZTt*li0C0m;MK_H|;#v;!CzS=e!kPzkP>WD?eX> z_1kf)jQe}`^G|rbTYDat&u`B2#tH4MOr4L8Tl08%^PKgXSiiaFElvMY_eYOf^X94L z#jnrw`Hs#j>Hg?a9H@5JV_3gw{w3}|8vdm2k7B)1sM=@z`ev<^A1-{pFxO+{Vb!0+ zex%`F>i%c^d8jo;`GOlBnb7|<*J~AF)t`*_FLnPjW?izs8CLztc>hxOKWV+Gu4l>L zG|pdLDNd}o{^nxk{FJaq_dJ#GHzjuL(WU=E>`^;kwKK4gch30zP3OE5w@2l?6YPN3l5U^o{RQ)Ufv*4Y=Zom>ue)Di+8aHusE*I6&lknpgKlp{n!TCN569b| zX>alNpzlZO_Ga#%#@nBHe?(|MR~bHkFEj2h==Nsr$Hv>AX>WOP=Zgw?Wsbd>=Zk_B z_3n8g*jrJhe%=V@3ykwc_IV@h$6&s@e9spp+h3_t?d`e1T;87X&v!VkC(idKxPL&; zW2o;R;r8~VYHvj;&hv=#|Mq-np9gl&^U?WZcYYy#)cwW0yt8*>f66&8l*-<8{fE8f z>E-MD>liPz`OSRZHMPCP+kao)!qV^+1~KHj`F@8Zf{Si_7*hG z&lDQxXHv~?=JRu=y%nXH->{Jz;T-&wevHD z^DABYA7p;3SM4nlz=1S`%Qe5j-jw%YaeI44vp3_sN}+LH<%8SX*z=odZ)Lpo{7*uA zGoM$-!%&*N>Hb`5d(-Wk)|0xuEjH{;_vcdEn>fvBw};k`y1gwm?9Ci6^ygX3^Z2Qs zS4(AYVZ+|^^FCy6$Dp3}c`K~fW*hc4$FR4`P?>xF1opSMq(Z|>OUt$MDuvl3b{$>35S*o3fq4zy#=TjgLS|20G^Cy#y=TCM26VGSjysSH) z;e0!~-^HYl>U{YK-Y-q-4O?GOKVHwLrZ??f-Cq>@yZU^J$1Sz|`gv4(pS`+%Og(>^ z@0W}L%b1mF{?p$N^N3YeqMtv}{Q=n96m$TBGrwJ-+8^v| zcAovdC*ePq6e{m~0zF(*AAP+Zr6qd(`tyuzV83RfA{Nj{rwg6{>yFpe3y?7M{w-Vd_PlZ z-fZ{03EA5e{ryby3+?wYiSwByMHOy(%y-$BcfT^h`xCYEDX#{m^Y&aud&3PrdpzX! z=De?yua~s*E3iN9d`W71GuK<@dd{4$QrVjrO}PjX+8cS_>Us_j(~YT5sq786|Lj2H zBhJ(6_Lk~>NU7{Ce*V<$tt_P5lfGWV`|u2Vo0`&|b$d(oKBPj!-gJAW_w^d*QKqT> zB<4T!eMlvFrS5qXn*XMRHC&c&zaL51Td2@}Ka$w5#1A-J&huai>}{I5o-4il_NKfa znA_XGtLwQb+W8gquXes9wY|~%P1W)93eCTx^T}h+Zy8#8@@JjE?JZ!;Z#4U>_BIXg zXH@H-uIVG*cWTUUSsFcNg`Dcd{qdQGy%pu_^OrvUAGNZL`E8<6zCM1C|Et?uNxp7x z=KK>h>`lM_LGOQc0v|8t82cyr`TG2&+bhlgI$fXt>3*DX9%TkP9Ko62`1}2B4Evgv zS?HcO0WQi8X}BaBjf$2Pj^x66mqbiSjrWe=var*m1J#aI*JL15hfm~MITCtX(UvY*Po5$SpT|4h$U*!zQ6&*D75^n-Ns z1mOrisLzYvkEtb$*OF5CXpb*K*h~EL_Ca64FJaIrJz6f(qkT&(9~*=?8MPnud~2e2 zGa5XR(tw_CWpm7HrTkq!L8F)0^ZedyvHlVHzxzp+P4XDc#?SmtO`EhyqNM1-^Xi-5Z?!qwteWTc2-ci?1jL0BsZFdKM?Fr+b9A#j-9&ke$JlHNvm8nC4AKr)Nr{i|dTZ=w8RKVgz}=Z=e1 zAIF^k9dA0AK!7QctXqJ%wxF8>4^=s@5Y^a*LW;pO0&U000C`uKkR z%&vU=fVB)LZ)Vq&Q8B(Dj+f8U#qXO?c?&@IUtwD)(Pwr|$e*C+pF;M>1J9owAkiZb zf8IdX#0;*xlK&ej#d>j+l$4h@byT*G;~Q!wj8ge7JsYYk1Jd76BKURb6GvtFJb%8G z<`3S^7#|~e`OGPv>dUv%ClHKo*p zGb;I!^2!~R{75_@TkjA5M76&HU1?eLi_5(JCyOCJVZU5{u%{a;b$JK6(lck+>3RRB z<#Jh49Hp<(`;X*@y?BTJnO*5Qa{Lhexg(8_*LHm*uL*2Wq<`7RZrGSUO6fWMN%RNg z;b#{FeaOc?b1TUwc3^FZ|)Iv{h7|eg9dT&+z-`pFZdt=xrSC1)M@eS$8jKZ`KG z;`ac&hva+>(;#^~*`4mFFOBCf9UNsc0`qA)($h~XAW(YfC(Y+nUuTHs^J37w!nTS2 z!Upz0#^@*+cp8glvY~X2eUJ~wD6fv@3m!!Ky_S~+|0g0y-jHuC<_|Ic408RcJ7~AZ zO&=K}y{0dar~Z3=59!Z~+Wu4n<|pTQ8Qq_%JGhxeg?}aVS$*Er&QBXOlppe|JD}#j z8|^)kIzKHqljc8@IzKtkN%Ie-&fhs!{%P#fsoHOimA^i9{^(fwqp9;B87u#p)cJeI z%6}ns{$pe153?hw!sW9r{Csq>GifA1#GkMSqr?=|#EdF~{8 zQvKZpu%}*jESWr$mB(axrDl1>DDU%^US43V^4i#!lJSS}=+fU;|ElKi*BxwNeIK;^ zX)6C_`Bm)EG0Pvv{mal;<%igbG0VTy_j#K92iRj{mVc@53pC5WkfQuceIJ=w{tN8% znEYSr`;N@=2iZ5rEdNs9r(%|Wf}KfLKDW;V&xem`pU2pDF15Td?X!suUTS$kRiD-I zh{lgFd-77t)9lYI@4}^*r`elX-Wm4PrScaZtG!3qGnZOk#aQKSX3t$}c~xVTSIGYP zQp<~sRo=^&US9oJ};~~lg+8;3ElqB!Q14@R0;QlE6n2_(Ms6?ycee!{Gb4 zk1}+(2S<;h@BU)W7r%}*`|p4MdmmiyFP|*C#rN(nfBu)F?~S}S^2@*c!SP?Fzn{+D zFMiklclU+S?~kUB7QZ_>IC_}>d-1EIfBE~-x8HvI@BiW75dP%n-~B(o`0jWA;lTda z*{i=;KKjTT&t|lZjJ*B74}9GB?RVe({rPV{?)mtq-v8u*kN-Sc{?pOlF8^tFs!Ajhx@hn_y6kc(cRZQ@qa|P>-FD% zkN$u6?O)vW`gdQy>-Az!mOt|%_x`K-B0#}D4{ob?veGi&;q-eZ;h{kVSNrL%j3+Ax zbh_Jd*mErw5!E|crZ+n?=N*3Y`>@w@G3H@eLNlV`!m4wMlr%n{nnv7MLOEKR!yYwF zl;3<3X(-*#Jf4E3;9?VT;f;C+M{o9c3qFzd4yz`{CS|gia)SslSbB~{r>*9(=(O}3 zi@q%dotAoIpxw2k?m}vb^P}+BBHEJ7Nr!7(Y?ro^g*IBMmKh=>u%2^%g66NELTPh58XHQ ztj1*sP~+d%hT>P#W@?m}+*%As-)yJUM~dcQ7$f zHD0gRqvDu>y^{@khK{`kjs*<5fn(_g-N3OlgKps11cPqinBSlqIF4@SH?Y^N-@smz ze*=5X_M6z?&ui2l*IuLkIC&ZM$I1J{t{?4D=D48eqBm}i^A@P~i3v25MPLjfkp0e@ zG5Z~yQMs97CPgN(*d!DxPRUV}M)yU0Ear;^Rk~Jwl*%8p%MY04`%yJ1g_>Wje?nve zi%l4-{7fpJja5bt>QxxAdmlw%nvAnVB)%qoD$}KCB#bQ}nJsKZ5 zksKDw8LRvWklUa*+k)L&v;6ExHj8B^FMncWB8yENt9)%Rs?r)jDz`xapj$)e)CQ=(y327cmD{{}tJUhv_=j zAnT!Wcv-GboR)gL(0OKNzuMzP%n2Y+7DJ9~mo zI^M*bQ+oP%CdZp}l|BZ4O*(%QE|tHe<5kQaF%9`iotmK1$KbE+1M-)2yi@B}^;M2{ zqdliwCLSBc!K3bTMm7GJ$)7Wa zBpCxQ(MGoE{EcUh)A$RR{5d|Wytj-wR?|^Ahos?m{p{c!V~lRn`1AQM+Z>|Sua41T z4)OX``WXDRT{?f7Z#K=8pY1a%TVI;|#G1nESLmJ{3@ddvRAXITy~C@nt-Z9z=lTjh z?8|w*_WmAUoFltSr+qt@4iAZkvGlZ$x1*4+jrcPpwjQ`_K@SwV^vm3R4$>2aF58)u zF7+rW-5c>@`w$B}H)!QcUm+>o?gux!&YzsiljgVQ+GP2~8UmZFZqWF%*8nbAuV1b$ zljaxlN#1_BhDe&<9;=hpFXWq?e`!Za%a`kmq;w(QZWue^C9i z=i+4bf7tR%)qfvUe?O@HeNg>RrN8z(o=pB9)c#ZTUn>3ipz=>O{(VsWPo=*?zcl+1 zzE<+}DxV{?xfIWPVgJZpkCLxLbC;)OC+0Kf$uCw@mX^jp;q0{P>}-DxuI5LXyXf9k zhCR30Wm9@34c)tP(%7C(opECG$Ov#(!jG+*yFIh%ixD5 z4W+f-v;@*nTAPsW&E&ep`#w>AZ$Rh{CMqSrDvhZBQl&xO=7*yj-fYnt>ia~k!FDFM z1Xxl+x_2UPElDFWJtIbzLYAb^FI5`MPvebk@1YM*8g6Uw#A&>x#3l;6&mPQ9NcZN5 z)>7Xm$`9Sil+sY>nly68p&RLBDfT!U{_vy`@N#Z3rNk1oCKiZg(YP^jFfrYm;mxMj zV#-OJKO;u;XwINYSEa#nQ%eKSI6^nj-*%~+~czZ&;ZlG`N)+D6k2FFAx zjf8Y>R*dM;z+gb3tI~*$LmE+%#-J^Yst;QlU@12?hBO$Ym5?r_5v9IQm|sYP>yArT zrNM&Z(2Wez4g2x$$cH8k+zjJ=pEsE2%}AgdcHd*8HX&X1JxMDe9XG9Vq@^fyRT|^c z4U)#7EsgpQO&VUQ>60UqV~M0;YdWNrkS<%xAgzRS%vmH2u3K*Tsx$^CCDM&}KNs?V ze%SXxllrmJZ*_cF@~{+N*7Z+)Iq*-jQuwf18hcJt{8#eWGQBjia(zhTvBk2uzA;GH zn~3hs;bTpd^f3`Vmcz#ylCVNoB{(ksRg+#)|5cF`_taHves{fiE(O75QomDJ1xZ2i zJ6$hd>Iq39AsxO=E=ig6%$1TVh3I&^cufjP{Z_T5hW@C!#J?w#f;XXu=1L(UU6DdU zx*`RI{=uY>)L*?EQi!FHLPEMCg@klP3JU#$Ng=79s!D;rkEIWWE^xbMv@T8Pktw>6 zh^|N>aemh#6#55~LQ?=!XM9oSw{Il~N!NEMb1zCxQ2_&^28c z$2wM@VUzl!W^0q-cqya{DJ1hVy#X$Tq<*HY3$#W_Bt=^axGkd4RVlDb)`g^gXtGwD z@o`eXx|FZdY2E31M6ND)(KzxSX}>BFJ(eTZrC8}Dq~k;)w>Vn&y7^TpjAu_n*M+41 zr`g(MI9>{3uY&Jmxm+qz5UYCzDa8`e?bSW)WhA0Q3ij$-Vn_#t~caLFBagtfLwHj-^!N<{Z6y5!O|UEoiZT+06WxO~n_ zl7cra=A~BPmXqQWy1IaK&_1H$mS|#nz-vn}F&zh@NLNVLl1L%p+#>e3+1k{>HCbHNvg@NNvmEq zC6hk%y{5{~KlBt3=b)e80HL?ZDLeWU=Pk)eyU1$r?Vj`qlvoWFfa+5P%P>@VgWf4( zr_B@aT%?T^d*4mu6yJNE%`4u`n>f~?ms84QqplET=ez-k* z4=RKF=h2B2nj<_HsT|L|0G-n zolf(+PS?kjUuz?l$Q3Xx#q0PG@mD(mFjFw@0hfQM<8Y zITk%$etQ0R`RVio^3&=M#miTthf~liQqZeX&?716_3?Dge@vi%8a;vhw381B@XgI*XiZqL5dh^ovrikG$fc7_GpcDo8 zFAO_Lsodof(RT_|Uo;jCo7Dwks6L-4LHpJ_J4$g3F9}Alf0>|DmS@-B;M6Agz1;Pk zmE~WM?Tsk)`4DFx4Dof;0YHVqfECJh&{Z5qCkHE6hy)oZwbMKnBvRRC8z z@}15?8lJ|28pcb%HJs1zD|AZ!E7(~LPht4cIfXu%;YU}M{>@`&H2NfVLc>AUr=`zj zy&64-MKwH;b!s@9HEB4DHE1}K)dRcwmqACcl>P{?Dh;Qz3N3va3jw?Go4|q^hM%Wl zA3bAr!Q-{{pBzLLlSipyn3KR8{CKxA5?7-GFi?2#(50_nlzG>BnXl33@h{9~eZOSv zEIz@ru&=p=&bUZpjv4&nLUtC0eHup|E?}wB1j9c4e;8}{1!Iw4f$23-=3myC)&~ap zi_|dcGEXgkV2W^q#AhX)Ms=b?J9zqS5<{Cfrcs^f5s8~5ZjzWrb)rWlZj-oAVwx3s z{StRdd{*MH#N=2Lf0o1{i7OMelf-Pcz-yCgI5bo?IYZTvD$7APrp!Lo4-E%>>$x?{(_if3ETWdmk4b07hEqe z$%D#^O3XM8u;6Av4@ex@E^v_J9M;(+@HC10b_pEfILMys5ID>+UaQhAa0SO9R)O~f z5`R@3m$0V01&(lB%6k7);ChZ{v*+#?SnwBSXL|(}{6V;15Lobsb>$ZY7W{=-q)%YM zUzqj%mB4~Oly_2K!5_*yEwJDZ<^4Yb3;s~vw*?mbg<0nl0t^0vEc&Ftf|468B2n^lOp+n8eRXZ1Wd+MbK^jPD^a_7y6B$+x+!O z+{eocu+Wg8Gl}~o4oF;aPSArK=deMEr%Bv!UeH4v2iaMP!yFf}&ff}p1;-)AUKO~C z;}X^@afIVi7W$o_*K<6Z^+_!FgMPdw=z>4!m&AfU&|epH!5`=n3;saAAn1ZW&?Of9 zf&L$YF8BjoV!Rpd3ZD@4^(@U(&_h3dS_HEv{pTM){{;~M1e)*Y`o6r< zZzmC^cX-mZ5cw$tkP}t$Igju&XX&pSf8i{L>|NAJ0Y1NdYk=c-F@I7poP zbdppDCz4)cl_>v@WWHOZKHC}0{F6aI*TjWTP(-!K43s7pADbOpa@!zKvrNi)M0Q2J zK{OyEy^}0zY&<8s{xK>{33mV4QKX-gKVq_mxClOKI4$$rad!Q7KD+;KN9M8eTe1MkPwCA3G>(u6U|w2?77G~t zinvmK8rP!&I)8_dnaW(@#Ki;=jg7 z$7Fs>vA;z9*NJf}U-X}}I8jja$Ko2fURM~wQ}6(WBx&n@y({dPWEf9Xdm6Xoxf`EC8B`e{fq^DjXJW}{B%k0$>B{q7<{ zi2rr}h*%eILjFHZ(EhN@Z}%V6l=r_`KiLh*^x~Cf$)XP1f3UB1eoFmc%?Fibwtb3P z+`qQH+E=wc;?=ebH;4YMNh*k%k4HVJr?Zi5U-s3`XVZ6AZ`x9~yQNx$73*p$I}Wrg zs_D3O(WYB&tM1s`ymjY+Teodo*s^!)Eqj*SyuZ16#kv(+YT8=%uIkyoZDU7E=cdga ztM91Uxn$%1>eY8_*tSri?dj>lZCqP@>(*Ot>D*M;vFfH z*0N~bfm;`Cso1`0Nk!|n8r0E>^xJRQzIDl+I~LtU^)0-0@9veiRj*&vQrUbKxT9|Onzqi3n>y+%Z@H;%_xk4QW&2mw?cTg|>ze&FtM907 ztzW-4MVgw{?Hl*6xV0H=+_duyfOWJSUw!ULq<;FXA ztWf2)WzCL#H{ZFA`mtW}x^>mTZHsm$F0*d;9k*5UJeBL$xOrC8Ay4J@?pwDe&9_*} zv;Nj?Yj#0@4m3k&63J}cKFP)Wfm`=>9N1om5AN=9-mRf_Z(Fqe_FF4gwI|epy4?q*+*k3m zm8~iG+sWkvxhLy?+b%@iWB0jTcYVv=Wp`}fvh22&RPElk=JuAlEh+n_W|2sD>%L7X z+Oz%O4vc5p7S*Ll*CEr@+`gmoV2ZSzGVS`?Zry5iB$2x#C)sw$bzv%*u577?Z6wjR z4Yqx5tAVX`SV`)&?Q9j5)x7mU;1Q>5`30db;i%l}W}g zCtVxWYwg^+>Hv-ZiEP1X?>@A5QO)hibjL}zh|-Z>Y)#RYEu!uuy6BY0<=nF)kl6bldC4SKjtIYkcL^wT&-b{rLEU&oQp@S}MoKUrS|5 z{#J8at6Q~i`?h4dx<<8;j+SI|cP;#;itW((RTb;*l>P_CrMB(Dr?{<#=DRHwJGZSl zu&r{{fgO7>uWZAdQHi;tcFmo)wl9{e7{(g+cC>dj?pd^G*|O$}h1cv_R=K#Ua%pqx zHO;%OS=>@}P36L!70tVLG>UXB?Yo*=S{qvq^mH_LcQ@``)VSAb+}*nO_QuVv`?~jc z?cUqEcYpiR#S2xefia0%N&_rxT)J@aV)3W4abZQ{=B@*+;?Jt)j_%gRg^l9MtgPD6xTtYw&%w^t z7A^=WkR8q7V`p~<2<`iJ-_pLXr>e1MZ*$|W{at&Tdz^A=n|qp9@u-?xH?M50Y}~Qm zYTVhrmkMsAkCW2ips}Ll+R$B}{3P14uzVqJk-f1me#FDbmyW^z%ykAZ>_I z=*+&d$tgecSoZOpKM#H}?@Lp@oc|ZoCe+WEu)2^ZBi^tD?S${x%czg#ob@EjkJOYq z+fIw`wRQ=$Il<3)zw9Uae1-1FM4_)$=wTy2@kKueZ-=m7erp<|T}VRt9{n%mv+FUh zgDA_-{)6wan>3soGoxhR$Hx;(>bcTm#e&Hv&@PX%`=hSke0+tfS8;?FZ3{!ECEC9h z>1o9pA8>~(E>fgV7O!mP$ebcQ?ab)me!x6MK6bmRWV~%hwqE>iw9AfDOZph%sho(6 zwCyJ+N&0q(HBmWJlf+-2Bz{wp_!kgQ{G3Jo6m0vvG6_Hbij5IUPy1XHZ2P<+N&20X zhNsm3P+X$;3n+)mk4mC#zt~d|a-&Tf1UqjfzEk=J8DGn(jBSzS^vQU;U!IrkwfCcK zJN;+bE++SBZTmbg%c+(Ht-p)%=ACvI~h?k<^F^80R*<>DbLdH|Q zl2smK@7q(oYW!x%fdZ9p^G2BToPt?Sn~X0q%IQ?fQSS!zD)HD?b;1cHUQPdll%w)rDD8^MSKC{q#H(|2Ly~w~ zIjEc&qUro^l~fvvFHqu7DDi6gvr4?0zDnx1S$?Myucm)SiC4?_OJy?o3CMVor#gCv zWc)lMzFsQ5DgVtf-jx3~8Bg`n+-`>|X(VJv_8v9SJ|^R-d?^y1rAfw{_867%RK9AD zrl3)**JuVA*lwVS10ILSpt`@UWK$C3K>)~@dM{rjXC z+IMt8CUvbnLKLig-~OJ~^1Ai6T;1K%+_FccS_N5cYVE1+>FH|UaiFJ_Vnupn)M9rw z*S6l?-qNbY3(2;&HSX$a-rG737I&e(9`M+>kJ7JQyJcOi5be7CI}ddH-vO;BvPTIH z%$)qhl=zbT_!O|iARYob58_S;7vdLW9|HSM2nF!Il51XaeoAUcW=?7f>26?9Ng=}P ziACwf@x{PSAA??TNl{5+5|FOU1=B_ODZpMGgC3ZKBw|SH`2c+g>hKZmrlH@{CO!oW z!C?eFNKES`7nc~C!Amb_F2^!30+oR$Q?xF9!5|n^fRy}{k=udw4z$UIVo*XLq~LFb zwsT;T(CPrjWdMycfW(m73Lr@b5CH@c$c+&oA674dC4q!G5QCaz4nT|yETElw3ut={ z)Jz9ek+3!ZtSzkT07{39? z0EwZ8p@%gjq|nu)^Up)=N7o+!)$aovhk;rMb2pUBpbNAHViQaRLeFu-CW@)H4XPjA z{|lh{LH$%@fbRZApggwtD}XjMK0q~tya8&8!t}%9Z$Hptbh|-XDxeFV4nPA2NA~b8MPW{t4wq>v1_AGmP#{(CPh#`CxL-FxBxxPzle+l#-GEtel1 zKKtys;X|)II{XNpi?;rB_?FjxGQ1eiRy;R7_`BiFk6s$S5zj4nrp&*l{~uqw@4_vw z+;_phbaj8rxkrZ2PWsyLrssY#eEB!e4)@>xyWz8s{0`|>_4hyUz=d1R-GAYprH>46 ze(v0G%i-sS&))x&;Y(lr-SFi@zeSpr{r5a@?}f8V@4wKGx-b9uk>M89d3oz|!<$go z*`<``y8cU#+_4d3#}bHn|={mt;b4^Wyl{g>{$|H4_Mxu@uB!-X6Fb~yRbXNOZNpB-)i z&zBFqf;6T5B&*Gj{`rM_3LY6QydQNHJv+R);K#%NIP|;Wd(RFHCofvtf9b}%FBG2p z^9z@5L|sLHJA7%0_0X}YuHfiJ&(yO-t97$gmOg+c^kFLWP3T0ei>-PZI^l8=ZR*(<-M$+s z-)Y8HJ%PBnJdY^rap=vP+t>HMwWl0<7#w}?*bmRB`W#I^-}JZ&O&ukDG<7iuUHmq5 z^)ZR@fNr08vTVQ?49sR;Z~6=t-qHW_;OP0k4-I_x zP;hjSm!-eyNj&?X9xrT%oxOc_;^?=o3y#*#NPMp)JMq2dr#u5^eia;T`G#w>|I1|q z&A)Vwp85N-ftGEaf&QNcN1Gq@4EWfcJ6axgjRv<5dCz2>-v+!6_$A;@;IHC&0MDI>FGBq1fp-Ak z2)qFJKH$~B>w&X@kClUO&j9!tZGH#%8a&^^a|NEmcrL^9H9QyNc^S`zc)o(?JUsse z&pbRY;5ie|=kT0{=QDUt!Sg9R)A9TPo~d|#AI~H_zk{a-&u`&520k9e^Y?h3#q(`E zzl7%-c>WchujBaup1;8}jOVL(-iqf=$m1{3j>UN1isz5fuHASxfIb)W8a!v?*^cLG z*XY=B_OoLcA8Xx3CmY?%Pd2?+eDc5#iccPVxcFo_;>TvPfq%z5J~o97jCt6=?^D?T zjjhOcvhI~1TwnX*53a9sXPm2h4WBR5=k#-RZ{hPWeNH=9_dY(qL!T#|qkO*881n<6 z9n*uM9W9S0ycgIyB$4(r6&*8FFiR2{FyvngU_5t@JD>UBbbh7@c9SyIptj4bNKuWeNN_l4yA=MZ{MFVYJ4hTv}Ieud&C!e{`CCYcZ1IzCQk+6 zsSrFBf~P|8RA|YJcp+I4FC;(MI-kopKX;?9FxBNdSN8xu-%Fp9&ei=TKKIe*#B+6D z!soxD&k2H`nW4U;Z+S;sUh{$%*dCYB#rd^=3YmTK@z2x_d!?M_gQtbyX(4!82%Z*d za(W5Alqx5yFL$H99aNw9T-~Sec|U#joU1#4&-L`#eXi~>KHoy0U7{~BUnggs+;Jmh z{|x?G?)ARc)8`#!uEFa!vD2N+FJ_!PRFXV!%eCHt+sg6R=N)Kyz&m;m(l-Ak+7k8- z)Ru>@4+bCi)^2paSCh>Cq>iy&m*6`l!*6T}R}9=B;}#*0m*S`%l(`Adl;qHkf;neD zpXy=*dC5%7!*?cfpYW2${j)_F6K`-qPu#I ze+zo4q;HEt{}$+LC4F}k`mLY`L6EDP# ze~R)?5&3@^g??+N-eM+Abnf?cqK1%#klzw_-`e!J8l=$Z;{fx-;7bty{_G99$2-x4i$oV`cr9=KZKuU8~W4!q{;Sd}ANd zBqMDK@}wfKA3pJ<{!8%vhTx;VdIuYL*Uj#@0{iWdOi$5+K zcnSX3%P*A;{2G4Quis|)>_QwHedUd^fmhye4ZQec*TAdr;jYX?{bOwOKN`vghVP^N zZ1ggGz28hJA9(*4Z1lJA1AjNKeBiw=y9Qo^&-nWK@_~0>PwW5p)Xj@t`s{}OcYegc zXAt}cM@NsN%sYYu?;PCRKk|Dv@NWAi$fc@(=3=iTHSor_+31_wxAwoj-i>-(sM9_0+jJNBa1UHga*Yn(M?AU)hOWUp z`q|(p=EH$2Gl8eE(INPfLkkiAHOL_y_#c4(0km}BuK_QFoEKAC;LieI1H8gLaOunI z`!7v%jSl^kjb56!Y0;(6BF#(iiH>_{uZQmo`54wEn9Gwfhg09azy0xkio;swY}~yi zm(ZE!L_CrI{q0X^YkATSwLCV_^!dSls-M!&aRs@*+nf@;j`xmz2fF8i?qO|)Idr7P z1O4+t{}TGgJZV4kp{$f8W&Qq@<^5@E*Yk4!^qJsj2Fm`WCphrf@!2D;va&_K4J>9OsOj;W+2~(;QD58A7|xEG$~&%dhB9L0kdFFJ9g6%iqxN z18p(kC#RnG4pxv2YmUPId4M^fclS@ia|!>vr#}@>l4J9Nz0}UpMfkh;`J44M=l|}>Pn>^$rhDKwU(M_P z-QUmdfBhe3_5a6{x&3ebJiGsG*!2kPbaba{;GMr?qrYG88hH1wVW+VBp@kSXQt@0t z_FFbO^bpx^`M}UTV2npY4+2*LPXWdncj(W6_W*lU6p-RHL&JP_YylZ6W24K(4BSRS9&pdhadDJs-#)GFa^pCo?y5x#$38=v9H~-zxn1koXXuLHs4*Z!*iEVj{w! zrq07ntzE1dlzgJP;fcBZc)H!pGs)v}dtA($3RhhQPa4BM47)>|BbJ#2H~YTeQ6_u- z74f0^qIkLwvBV%NWeMPD2Bj?OYVPXp?5?ZFC-Zr1saQXt1Kg?cGg9hi@(F zKGbsfK)a0_j!1lKk@caWy{)6Jn-z688Qn#qX*CT;x?AhIx@y{1)U+8j2b>rHFY-pQFJr9jkm?w zmpoDqoxB6ph(CfpG@GOAERX%8F=4ZBd4pngu$)xXvDnPlCHx5KiH!gF*bVvE4f&tu z7DlRmeoYO0P`L7Stu@6pKn+J4>uS2%mKPf}-3K=fc2z!}!?_(W$k z`y`K_6cPV*Hat z;QKZp&}Ry0PY*rQ{da~!1f9 z357!i&T4#NRGUqd3`kvsK8j;=dEQ(Z*DRlAy=GrgG?o8h*Y|>~(xjU6vsE@$j{ zY?f`?y{TeTSzvv8X9sji1ll~KeYDm1J3vt#-3IJ^J?7Z}?K-uz){$`sy4%}ZyOsnn zdNyH?D6o7*(aI%(-SjA0wT6`+X=!Z?bRF&LZfXm(9qH;0G&UV*IouT3wSD`Roi)2R z?AWdRTHnY?}jr72d{_N-H7)t z^D*3?V)N+pWFLd-@IaqU%cIYfhJrWK{ydTGKkorAbP@Tk@rwGU5l4)bus1#I%4Fu1 zcsaYG;bP+W${dTnj9t`l5qm+yOWE@pUc#Q%@M1Qo;X-y^!;9Es8ZKaGG`x_V*6;%M zIStQey&BGECp0{dg)}^u85*9$Iy4+$%^J>QwHlty_G)+*tJZKXtJ3gH7S!+zRt#Kb z$#*&nXgG&uXm}b8DN6cmcEzpYsqCVLv)J<*p2D66w);Dio!97-*%=LIu+v)lbk?iU z(^yEu7{xX0XSEtmWz`x^VO793KglczZ0nnkm1;PN6>I4eSpe9UUjoa}FvemHdsxWD zuDIN$zGJM=u#1J5O6NvG>9>2iO}fg^0oZtV-Gt}NG3MTaE%zFa%e&mqg58XT@b?)0 zyo=UaV=j%0R~x(qYe|#YMKoX#13&-+!t)r!YB2z3+=6;=P;jLc_%PzzamMf`NY{yH z7idS&=*=PKUDM%(G}st5qJvsmLDJ$lC~>jGK8YcHPOp-yJqj>I8} z_e$I=aX{kJ64y%nn8XDV4@%rD@$(WFOMFq{4vDWwTq-fFgY#oZ4C~=IBym9E6B5Jv zIK9`*pD8fO2Y2pR#uR}`9)tr~0#o}ro+>c4m*Z@KseK&7U`amI9*%PaCjL2|E->*+ zI5AH>W6Y46X`|$=ue47{h2Hzv8W&UB^LD~e~-v7>PLQw&H6p1 zQ!w|v!Cew%fA!R}^<4s?-LPT5f z5?ZIQW_mOJbd2;S-T4e3bB(cIA(kA$7t`5v^d%2-@NZEXd-AAU8=U~IPh(*N9Eqkg z_FOvX!e?XiL4P=no$RbgC!WU!A4q3UFJryN+=?LdWbEQtI+fpRcq>k5IHjUj!PAY| zW8AmQhM-=|W2cPNF%p?S+vQ_VZ)g5aHk$<8Q9bNIdEQe-QY!h=V~J=1%J&;=Hp!%; z8b0Jdr?blGr-3MhJnV8hd*n9X7;6DcLHfCAYy;X`OBvjtBmXjkcb`ZP{E(613juNd zU(RC#N4caB<3)Wg(BbrgK~8@*o#-$i3PksS?(>CAy2uZOmGswe`NvUSg%3p^5z+^yuN8x3XuMa4Ljr}|1{d79ym+VUg$&}A2&`F*qehPYT zh1F^78%M?}d}OFDl24Yv)PWp(1t!B)ag_s;A*=Li2PVT->3bcR3|*zyIxrc&N^f>x z8VFQ+hXd2VpwbNo4mt1%2kw%XhHKMR3G; z5dG`aH!c4Q(BFHuXz58GO!K4i!v@`ikO6pkw+9`M{-sU^c7skI`4$KC<78)II@uSu zPuTBEM-qhk`0&V2W9}ZeSlgwG_Ea9p68TvumHk%Kr`G>Tw6BrMC;NI!l*jEiW_iE$ z(U>?kcAS?tz0>0%;Xxlse|^(Cy$Kcu|M`a79UwY;Ac^$*{-fA0fWL#}scgu_u^0Y} ziBBA5WA4v&xS_;8_OgprnevzR@09oQZZ{W|kQeOBA7si)_u}~XQQ~ek(wf;nVV4oGVoG{13`wf;r{nHV+lFBYDW0gmQeWKP> z8n1q2@sIM5zw)RleW5?j@+*({xqVyt<-QW<2>Bm`e0W=Es)A=%9+-oFmL`qA3 zy~dV}G?AL}>@{X=#MrLjoQ=>F3x|8AIm*l4NE;z${&3G!2R&;ewgDCT6bC(XBRK_j z{>ct{#zwL|lYagP^OCAVsnFhe4E^tid(1k5=zsKw8@>zN6S7w>53)z7x5Rm%dsw!G zDZQ7$9F+74c(s9w6GeZ?@~D13qVW1Xj4-cH!Mr|>U7XJA<2Zq1ULVIv9P|1(PUe_T zfE;_E&)>Ai59sqLBgL=w|4Ad+&*fhXf8nWg)@S%rLSUbdPmo8SF)2BeCo!kPL4#_J zy^HiT|Mg6!?xDct)n{b-$ypNgbXT}%ii4izXC9OOAo5Rj(6jvvCMNQ42Ys4@p5rHD z7j&ABr#t8~{4^NJ`az%Rpy&EYL2UXgoBqReEEqJ$ZI|E}wv( z>%T>xqtU5Kk#f#FrkL-0S_!a^Sd78d$ww^!(++Fs({*5By*ZSyb6 zv-=}iAJ5TW4u8xQHn8|ECx#GS{A-N&*nWk3VA_A!n5NK4zp{kAO8pJ@WNY|jXKE^+ z&kIQ3QJ+yI%I4)0#$uOU;Z5Z9o=R=}3HPL!13tHpUSm>)PvPewW3xAe5-k3DlGz4e z1ercdVpi?=>w@U>3fZtsWiiy^l;A% z2R|6Ct^VcpZ}utmgTHJ>0=DFb`31->&pH3h-pI!zOTM`kY*0;~r^&b1m|qc4_@(yy zt@>Tiw?GBksLFd@D$O@`{d3qXU<4uGp1F?pE!W!DYb@Bv$3u(1{0dqbo8_VY%^6nx z9_k-#53B9ZsbCd0KQvy#LX`R!I@(uILEEop`Mt(seredm;hsf&scG|HsPRwo$g=oJ zfIL=J6f5%THCCjOe%a-fX!(1M)f@TvZR)21>$Cbh59O_d{@Cpg_gtg#Px8pI>Zks= zzJit;W_`UzX$8antc|ZrrSZkY;hr^Y6-meHAIEyH=UNB9>oyVtX8vAdE%Zm_Z;57) z;hrJ~E_UEF2fjgTU*!0uk5|1$u!3z<+qXWI_DoFv!aZf|dfq%!|H`%YoHC}2VY;{0 zr_iS*Vk1|ir~b{g9$Hz8Ped*Xgz<5pl6R`3pkiHi2jX=v@ zi#GE4-R%FKWm@~G{(LKc5-+d7!pT%#F(Zbo@`@|iW?LTNp5+W95(VsMi1BNQng6nj zmsf0J>>r@KY(@$!%cJ^NTJ!{-zSP1=ls=7-!YS$3TJ&U|K4@VVd;AExJBaj?!A~RB z7e_Ao>WDkmOY0A}KEKg?SA~Q;VEh=vd7Ef z*SIm2M%EQE_JgQZB7*{2aXF9pJhzcNNzF)ZU2z(lhhnO zKE@vpp6M(icM{bQPlQ3=To!2vi+qdHrsEg?LQHx{n(FJ=EG+? z*YWM1nC&;5_OAO|y+&#(t#|GAC2RhZ);>Ak29X|t`xhARGA)1LW#MnEHQR^%Od9We zCPw?WTKkQ(f0xJ4Q~3U5^%Z_T!q-n&(3t%68dwijOHAi2$@!?5g4q9a?za{hoGAXm z{&wVkVT!h27~y}=c}J}*gX{m)iV(WTv}B$kz5FF`lOD z>%U3bdWElNy>@*Vf712!)nrF{tbcac>3M$;!*cw=`A`a$Q`!Scf5_mq)2 zQ{6A4{VePtP>}wTb3HYKMjDIWYhXR;v+M6QlG3SvOnIC#X3x{hpQZ6n`%P+ne0{mg zE|2CTeLXkVQ9ss;emngsW5Hs*zWI*)3mx@gJ-FM>PvgH{U!kMEYtq&AU9W-lK!%v0m`m`$gjXlI^w23-=^A@+Ug@f&ZUnr^k4ePUdFIzvLR7pXFM4 zu+J1q&q8K@udw;!`j7SKe!Ki|&nhi{xM#J4AFMw=p{8HsNPn%C9`*aU2g~dC(0Gwz z?~n2IhtJ-hOTv7F{hl1N{AJkh&Ad)+ALS2N?V>?7AqPxI+at$kE~zLlQV^Lgo{ z-$H&50z*wt+T;{BQrBxoaCn??bAE`aw3UeIv_=CRRV&>=k z_yQU~(7z=no%D5vHa-ylw*C=6(6?eU{dUM(A0IKEYW7U+vGtAkgTAda^FK)X5n&&; zzLC5ZrrZ7v$qV|{YUUU1QSFoK(^`|hoy*_WpLFD3mTvnKBp>L{VVNH1E2oSIeNp;@ z@}oa+{ZZ&Y$t!al{TBS${ljsJt$%`U_lKZc`hu54k^JoT3A)`Mf^N%K&@K7B>~Z)r zIA2|K5pNIR3c&A!IQ@%L#$;{(=`Ng?R(uZgHJ#@+;{HodhP|JG^Iz;2WH6e~Y5#=p z7i9Qu=H=7=3F(Krf0xdl>6Gp;S;0jUw|rjLvth^50q(abLxG z9t?e1NAff08|)viBYiOODP!IkjYqyL?2o}-={^kYM>BaJrq9TSxW~p4M#T9;pD};F zg|VKlyyp=lrT~8r=f${>!^_K}`#LD^l#$4Bg)ATUPdMiHc|Z>vv*AsWhlv2(XXGXg z-hmqE{t&0!K{2$Km9DEET4+3vbHq_tFkMlIRi#DeFJ8JuAe^ti| zcu{6~VPhc_F3eS4l%IpyQ!Q`0Bfm&$=U>DEEL>&cK4V6%SIPg7k;peJO#KKOxE-#@ zKWr2__*u+~S-9CQFE=+)E6+EZ^vf=9mR8=YM_E2P1R*1t?n?pB7yhfjmp{!ivjh1o z!4rZ$z+^woX6G#680Y02C(bC4c!*{GE{ z?AJ!Mf7(yu`=_*DSn9(yYy`6ZiP&dgojsq@_&R4DjgO|haNogkewKv&ZJXX_1Oh;c zea?Xb+L-J)VA?az-z)F=E=s2$>=o-vZGF;fq*O3veS`aB++LF$_M2$oR8CK|>^+6! z6p8zc!tK;kxJ$()~SZ)1=z9y^sj`1Q{)nA-PC#(8L_OItp!D+T# z9_`=i_NY-4SHhksRh&o?>ud^~uK>cf7HvcBiz zgR+0nXQVjh(;UsdPZ@q~ey97pH2$081M;K0RsETy*&p5i(CvRls*0WSb=c7LpWh$R z%3p-q)%iFV;#KR{?E&*WmJcf3X>XDCq0h&&vHDf>JN=i;89LoL|8wtE^a&qtmOAF^ z1v$DsFVyT0_ibkAblo1qMv?4qJb3xb9P@Whj&9%jc!u$JhECW01G0BIE;98M_MWBM zH}F(H=fIlZbJGg4RC}MDS*YO!*-IoA_MS7N$fp07*gKtAcq$lq^8sU|(q}9rxQN zsrDUdzs~U?4ep_8U%LH=4X3>qY5sTESmv;oOib`b=V4=oqx>20uhsH(dyM99jOVXA{R8yRY~9}V@r>+!uWs+S@JGR#-={hJ z`J&8%Y}MY2bCzhhWJaOH!rpUdFRXxNXuWOE)7J$xjQO?7 z9v`{=QUA&NCb$oY`N_8DK4bcH7037IGb_~nX}({RVb^!cNEh~M`R~G>{T9yT^i0dX zGdRw$>^nVOx7V~Z9cO0hI3q*HCAa`$>uaB}66Z52UXAk?6)(W~ii(}~?;Kw;u{~<1 zC;wI*f5JxO{Hps0Gq7Ev=GW~l(jTYY6dCw@y4Xf~>$JCM^Rqr4%+65d z6E?1Kl&9NgwDCo^|AGuvzO=ufuP+v9nD$d^b$icI?Hlv^baCIun%_%iEXh&rePwR3 zhF8xnl33XLf`USu4tp>C$o+Xb54Yy`&-u!DcTUvaF`p{;yYT&kb=WYo>>1~e>napX z_FHYv&(IfZKDFlC8Sp<<`=2?}!nn_*t>3x74trP5e|imk{Z2b?!hUTK`e%(V3$*zc z`?n5z7vn>EO#5Ge1+1;lm>)7P-T@BfeqEp893OQ1C;d|Gk>(e*ys(j{(ZdGqhRDF} zYZZX%&u2QulU#@W>i&J?diPqhbm=pX8SfB5&U8Lf$u~CX99%uc*UZjVzO_P(T8pRaZM>oZngtJ9tH zZ{&WVK7SV#>+`p6&wWPewK{!`!~P@p5B2%HxLBXhb^Go!uDw>LU#FGdYur%Lq1$^d zIvl~W_gUy53uC{oVCC9e)!vt^&eU*G>1>ImyKG|`f9rNv=YVWhxkw|442fb(MIJ_XrR!DM}YE?TGB<0(V8?^DJ?&fh}I-U~3CsrIy7v)?|W zc#&Ry=|Z)9vQKrs4;w*9Lk4c|#SVKW?@P7!1=z4s>(~8Lj2D<-Rl3t(*Th3wfDl6Yc;%N^|cZUdoL=*7K^!`|AFlt>p7RA zKic^mzrQeL3Jgemf5fzZ^E@5r_mxNf)khxWM9=3m`@{F!wf%COr{VsecK@f>!1xfb z<%#=d+@30=J%x>8M*749^gnXG)%|h!gGJi;w!>fRGs?B|@G!owLgOg{KY#Bt7CYu+ z=lZ-*voDlauG`ak&3+&1EW+#R?ePxwzgSPNj}K0J4;xPVUIGPI?Mt`6NdGsUJ(pza z_UarDBIi%tURP$S_V|!dit)j2Z`g3!zpxe(xY$o$<@54I}H|fxKmT>z@wEWdVj*~3x^k<#+wPcAtf0Y!f^B38VPIubR z5={Sg|B}9{^Bw#}hkab5(Zj}E*??Bpsr#eO^F8t>^!cx(M76IzW36Uy z&h;$K|3T&aoaR60`Lx&}wCrz=SYcc9U*XER_IWeeUunLEi?21$tHu0RverDW#{H8I zJpZZpQB3>O&Z8@Q_y#Y+#QfX%^Bd-~M%&(SUsSn|N&aP`<2*RvI1kqC&AcCG+9%!L zQuRA*%+u@x_oW>B$%T&lVM$tg=l*G*G1p-)%OD%ozDfdm{rY@?`)`i*?tD#uqU|SV zX!o;L!yB;e*STNYXDlex>AHU#ZU0NRuhM{OUrzgk|6i!no%>Z{b|H-s=YV=d6^3-_mzb^FloONsj?s(!;<%)sXxOn;ia z6+7(FxjzwrgQk|J>zmWQ={#4rxAFZyj0X#Jdz+`}b2NL_?M?R&o&F%kiv_y9Ime55 z{$iqI{z}sL!+5bkw>N#f!21g#&SQPNKP-Ejh6_Rnmc11R%=1=ZUnO&k?DJOG-^zUR zd{)e7i3^nTTg+#xKXiN3&R-w%(TV=m%xB~Kldw1C`v&}eC3gJG^{+WUnfDpY`&H(B zW$XTxWpCnsv*~~K8A%TNiM;=)`%SybU$NdwQuPb>Ka*5_zz=D+Fonl)E#pYwhOjaRC_7T@2@o~z0y(!ZSM zC@<1K*8R)5b5;2|=RfjSRQ_mv;yfR9%y-jOdxU*0Nm0%_$)C(Bu+Kk%i?bJK=c9qS z`8FN)m-(UFpLQO4(T5BFAEN!y{blHnJdfn}AC>RtaC@}AA4C3Sw&T1~%y;JeX4-3V zOnapJQntQge0KP&NlEJbi1|#jN4k%%&X-R8gS~0?8a8x$qw`LE{-duGs{G_Y0c}k0 zJ5u$D>{G{)^M`Iv0j!Sg@^If0r@=Ok)L*B)M%uqV|IN);?F;v{vQ+G}hcLe1tn)8c zS6rpIf3VcyZ|3K#_D24Nio=F;KGnwuvOni}B@_U`n*V01_6YmR@hz~=AAtiY1sa~~ zFVxN}^Rt%N^pC{;wDZY~tF}L8-YQW6V2s*@AugA zpKfnX`-z+%o%R;sy~Y3G`Y{I|)%Du=_J;eZxId-pyM7-I^Pgsak@giiAL#Zq*KxmB zAO9om&1qke_N&`hy!*XQdvn@Xr2XpiS-ktbPJ46OS6amRpu@gqIqYkOFJPbd!M^6E z9DV~4}E?s#es))9$4ig4Z9luGOm4@-=~EBa{IErk4g5G={S$e za-7G-v#;^*w~>A6^I4i>K7)Cff!i0Z7haWpWjpLg_orZAxDKY;pL71oHEWmq4YM5U zw;7K4%4xqTY5IKTw6D32^5-$#z5F7dTnz{ObG7rhth9Wa4*U8*-9BZy@vNQQ+pW8YJ#Z)y`1fDE zRQ?S>y#9Zp^T(B(YMzfwKPuTjfoQzX-rbYNDkz0~->t+DGm8cxXp3jWE)$0zM7#f*+2v@%{`~#ahmb)29zeb^i?`oT?MM2Bkk41o zo-G_BjP~T>{p%EbS4KQUzw2n$EJgzIBNFG!q#vXYTt9wD^c>K81pO;%Y;tFwhmwvx z^B^z(uA}~0>@k`Dl{Aun9tl_0=e_G_2Iw;XT#|2Y9=$!C%NOU}_&o$S^#$~a`0<91 zdB#{PT1;#%NaJ)QW*wOwjO=Sw82N$C{_%BYiTB~MkQeV~^ZGicCg2NW2!#3XDR{X2 z>hXIDFXnk+ul#-O9rbVV_W}7j^XT>K6cX8s+nKL3kU%qvz)-@@oJ8Rt@bbVPhJ#v^ zNWF^woYR>_FLV_F_4zudC5iXVi1fbB>4`H073HTR{nRnRA0m1B#3Wphr6B3qStNfR zc={~7kW|G+4KT!N`MPwl_0HjQ65F>q>mQy=HlHI#X2sGqR0o zi5a&t@y-w-5B|PjSst5j_}yf^Uar4+u>W))r6+Tx_ifMVOc=w4kqF37{mb=<#$#Sy zPO7v=&M(o$crAW^TPRxR+0$xM4+7s|Wy zXa>*E_T~9mvkN#g*Bd~A7M_yp4VriZ=;^uMS`#M#r{#J>Do)Myo>p;ku6NMH^N~I= z*LzXo`p;m#9fNlXc?^d5cueP)y%a(3tAPCPxnoi`%cD0K!m1j*T%U;Ckqx0p{Udwm z$nLf<*Z1sB#(ytnO%o9JT`fZb)rP-6pUYw+?F`TzKzc`e_;I2 zH2l+ObmsH*GwCkagP(;^U$q!7X@21BX5nA%PfX(T1-^fSA5DWlabA3%kLO2w(B2G6 zKQ?$bwbwo0$e2uX4WFOjuS_?xx#~o3@cHJ&K_!|gxYu=&U*{!hAf*?jel#xhVaG_sNBiO|$#w#Ru5B&Z%I) z?7uvh)?es9$Tw5S_b!@`-I+$lRI9zvcUzu#EAY$9h~Eq$Pcc4CH!>XkDc&EGVfN=4 z@%?MwpWqk$H;wu)gXI6zgXq6Z-hWf2KZ*WDa-1whv)M5!Bmrg$i9Xk@9B!X;jcly$Y9F1#$g{1==K5s z2lk=zr`QLdPq_cYtBYkHpHt*X`e}?!5&ajAVIPWowf@9>i~Q(MDL>Q?Jih$5>++}i z!B3>TF&*2Z(Z-&h-8>Pg>IY<}_kU(*d`ynkvt>0gQ^ z53=7#d8aA*jqxK5N^6e)>8|NUN<4Ynfn4SrPpfxOd_ zSmlewxP2(}{&&^6KV1nm>+ zoouYHA)kBhNbp+nf7l^^I*&+_bzEY(CH>~6u*ZDoN4|(9;JT*AZNE;}SZ^hB4 zW&MNLzkJYBdDLu=ot}RCI#zm3Uq~OZo)tmlkGkI-#C{<^ze4|L``u%xKZAXNHVDU# z%l*_S`{i`unesz^dVc&q<>V&UoeB#lCs9UVcr#ocaGXetu1#ocSMN&&I1? z(;sL4ZuWe<{F=Tv^9R{K#mlehhckZ$dm&zaO&^^3FUHSbs_K(7{~7kLO8uX}{3hn> zVn*kE^n1lL-|C_QHbsW+aN%7vPrv@@ZwzdH zZ|wJ;@1Nc5VuQbW`-VII^p5x6e}C-3<%w7Dy^SaA|L*vYOJgsMjs5CZdA~qt`$gNo zvBT|O8vF6lSH0g{JC^?4w}0tcdc)ZFzw_v!;E!J&`@fI=X2$cw@7^%<%VSC3?0Nrt zZ>;^PP5$ZQUnBC%V+-GYF8R#6k3aXu+N53YfBMKLe;#^f?9eBFzUuVw*y&@h&#E4M z{qwILO*`}M?=OGnQP=;zqXwkFC*N!Q{0qORdGEckvB!V^`)|EC_SbiO@$t8ged&%b zzFPC@fByXFdq+NX#}|M4_yeCFyW?AL{NmsK?ccul_AigU{=I)dxC8A^^`?w)ue6Li zr#5ym7cL99QWI0&;q)gw_-dwtT^`!(bfu<&PPfx>GnZ@TYH+cWbHUt$V>h|nGe4F1 z4l5(ZCZ}LWS5_?^aM{PB2jkFb$u(a7syK97a*db2Iu4zdT;t{68;4FyuJQ8M#-Y=a zYrOo;ap<(<8ZUoG96Bwz#>;QSp@-toPsE}3a{9w1@37!kNU@$63wfdLgXZQe(Qk6O zlHIT@sj5gtmf@;wmDVN{v8=(HLVxaUca@B=##&n6Jux;ohC@b_C(_U!Y*e}|n>Mt| zDsN4p^2;jNTH9jFT(n>8+T=%_K~Wg~byYIh%GVXF^90T-$>z%}n{id{&lY0@a`O^& z%v)xU11=f|Txq_FcNhy%HDQ~`RAf`jhBVXuP@TwdT$jP~FqwsuwK}x&Q-UchoH9=NI2}bB_#Fz!iO&OS`Kf4LI5l?peq;^%$0=X$ z=VE6+ObSk7;YqQ}PYdGEIBmT0JtQBwi^hK%R%?HH5SzT|vCGd0;_Gl3j`Cl`JW%#G zq&F@)mq+>LJP;)QEnnz(!UXKdP4cw{UT6DFI|`dEz|5e^KQoxg!kOU&l|C~ze^nol zzu0zYN+1S*ZiPM$f7LpFo~xB#Y&(u2KWWDv{;dM)QSd(wf85Wq`}eB(%W?LvH3r1s zPul-P`K$ea@{2uw#Nbbk9~0%T`2+G7d;E#PpB#TC%3sF^|aW2r{Uw9mB!?+ykTVt z{ka>mv)*AD)mSlm!wIlchR$#&GjnW}bliT&LLm|KQ1W1Mbh;}{bOvRPN_V^6Nt8cj zP^G&XmSD~P8Ls;wAMK(!_b_yCNS7jcgYl%ON-pMST^eXD)~FuhHfb;k-=B>ljl?jg zlQg2y-AQ4uDUGP~1TJkZ4V$h?gL$u38svL^MAAs&t>t~25KS7QwY)V^>B&O3Fp%MP zV5BtNiClL=se>y0gGqxt-H%2Z;Z&<{!%k^S z9M{ss@KMMINpW-QfCQuZ7#Nw!I=H??^I_a+TpwfmFlk&L9X^aI!HM`Ny7XfDC{lVc zeG-!%;hRW30V}?!zK66L2DdjV-K|)S?RzMC=y#`+6v>mb>8cd2-l}5yCT1(+JtL*y zcPG=>p!*C;D<*}B&gZ45vviu9Mq-AM+8ved3v)WnTQ*&lLTCc>j;;$aeUf;sH|Y^l zaQodpl0qzB0=@>PLrP)06}FVzleiQ~N>S-pwsR>$7ZkcCg-P+}8(j)9eG#*j@t%=V zNDaHC6ciu9ZY3;&uWd-9qt43yFwyzA5S8xZQsfq8(={nfz+2a)5Yq>V*Lss4DTPEg zmjd~XHkY;(+&(TPvg@eyuurUcNSC7KPZm-nU9ssOObRi553`l=o)J>ODJMyRuVbUm zl2QsZI!2`@b1CrAFB%;yL#`|2kJ@xi3ZARAh?qV{yw;oaNGT);DPU_WY;$Sr0*(&3 zl(2OZmF||hfH@{Q9S1%nWk^w>t5S%*eifdTz2;}G!7<@~T%@}VQO~qaz$0@x_emOd zB)S?7VDH5I{uh&J%oA1IQ1DlH?Yl*HGP+Atwk9j3++Ut(KFdqJiS$RecQ^~>kFiFgqZr)N0m=BYJtVwZp16Ljzpl;tx= ze$xYEwW55VP7hl2-l+9cu_6o#x`Y1^6i0+Bf*#Pzw@%$z0N*e~`wMhBj?QJks{~w* zlK@-hW!|8pz4#$$$is(QS|sn|5%R{1S($r!X4(jw54s0-Oj~O?qOw8pDDzi)i6`7| z5q;tneWJ=d$}X0z2?-w#S8#_Z4Dx@WNm04NB?1|0S(N)hAHJ-lH4che<8p;`cI(dHE3xiv~E1Eo`|rcKTNO2=PE31-Qp_r(Q+zGb6j(YdxphU56oo_XU1B7+uFh|qRisH4%^;_7rXu9TZ@ zF3dZutlk}QXHT8QXh?G{rqfRvgf93xz`k&gy(&RG&?h43AkrAH}h`oR3@?*&&~1{nch-JgNK- zyS|IE{94)Gpv3g>F7q*ZSLQtWJlSV?ee~J%Yv{B2K22|?^7(+5QA54gc-h_b+#g`~ zesS(szW&9mHulZJH{Ras`J07H<`n>oXt?%&j!vJ7daiCcyx4DXt|GTFk*8Q#sI z;9`a!db~2nqAz0?HC)7A(C||ByoQ&sr!~Bot_LdR6|(ahUc?^LZ~;4`;f3t9h8M8U zX?Q;C)o?yLq2YP-t_`LBxeQ|=h!i9_1o|$9Jy~WKuH6E9D zxt~4%_qf~qO~!)w^DbIzjf*svTy5|ctYx(@{9;^}=zyDg5r2bU!23x58h;O?p1;BK z@9;c}=Oak>HO3zMI%A=4ApQSEqc?|`cTI=)G#bdi1dncEt_r@)CmfXcqNLNTM098u zPhTZ5w2@<)m53gcxLV>`iD_0MdPw5E5}%fsW+h&~#I+J%l$d5EUOzgF_-mFpATiBK zyncy0B(9a1W@TQ##D>JDB@RhEDDer2FG}1iaW7s7ZOSLbmB}(NSr8_95bm8NFtwjB z%M+N|OSmH-Ftv~H#W?~~dk9z06PWlXJeV&q@yqclfr;M~R|YFyEijXK@H&Bg5?_Qq z5`P&IvnqjeBo1s7I3RIwm%s%Q*X|d%SmIEvz@;2#vd0bxEb0e7B(SI-?f(;jMg5Q$ zUWH5ii25^G@R-1&e#kE*u&5vM`HaA#{!Dh|UV+W}L%jl@06&Q&-_Ht+Du7eiY3MrD z7nJ;Fd`sXei9-@sOMFG*y%N{r{koLDR^sO+Zk9NBUeG%veoSIR;^F~84{@Bq9+Q|! zT>KqD_nCBwg}l(e{~+i>UdS(Tz|8+$K`$`#ODyDt{NEFFQ(mVf7WJb)zOUAg*O5vr z>W91~7WHQ`a=AzzqW((&6=rq2n z!UFiK#$N`;s1W+iOK+of@&D<;y!>ry_<%s;B|YDlPkj55V|xF_7UZSiLoV`%XuPBF zF2-F7W_&=#FF}12;4d;|CChr`RWxV(Bhuk=#;+B{@N&d^*G0ryVjkgo$&cyZJu9NE zc+=0UaKuZSBH|baVT!aiB%d50=ui^2Dc7*)&Ngh8rBL3?bay{mbp+fx; z;^`$+W_zn8Egxe&#hda8%6LEI?ZK;ZonbAHpX!Wedl8?RVe@nP36Ufltj|$WV`%4L z8R7${<<6!;uSWb*B|ac&-7(5hvi?CIse2)Uh!B)94VdHAAl1Q&BHfLa|3@<4%~GGu z49@)Yo_TalLb(TpPPROpPau6Mmk={ubH8!4?T@N=# z3ugbBk))rLKj>hihZ=?u;QunN*~Ggu|0Y52kjbpa7i50hz6Hsde+Q!P#&Bb0lpkJ@ z`STqFXZ|@tARR$Rx_4#%=cT}@k7;~y=0AZ56cJJXv$>+cfOuFsV&*sd@j(EY0<}>^MR%9M8zaa~t{FKg_pT-dq0nAGap;-ckJ}sV;0+E^ojpZMDlk{i;w(YnbwM--_|)ErT$0$ zNhDb-Q&RmD9O7e5GJnuf|B%dY>M!M|A;Fn{CFBG5j`UaK|NOrp1|h)z744^wW`0wT zHY0x+`Gb!9FUb78N`9)}%xCKFOhiP>-~3CF(bQk!pN1r7{#A&;Y=jL0o&Sq6e?Syt z{p*zfPUQbgl=ibZ!IrrJ<_nEymjA- zoj2c7*1D^1Pven&dv`2vXxnr1p;b4v*Ojf`x_)(^F+bc>A z-c(nzG)6D8^5AW^l<_k4=$^Nc=dt&BWDdkzUZ|~*<(4Ql9 z(3xm5+j>~?ad=yCL)-4=`lzy9zjb5fw&nXPHyqj5)_P=rCH|ln`>Qs0G`8(N3Z2|^ z)AHT7H&$(K-M6Pi(TUxM>h`QUP#3jKyT8`&+0jwox~6#Fp3PnR^QW4 z&+0h+E9xhEiPm;YUqt=!^lme?d+&<&jgz)RrrmbSzCA{347po!l5K}vm&cRohK4HGMht!1ZrbPGa@bm{5u;wy z&Nfn6b$gCPZ%ak99RK#;*3i19vk~KT)N#_8F4nlDr|YUOi7|dz>6)otqjAs1BQ*X; zvjwZYhtb{@<+sMt9V^`mN=JIJCr(>-i@IazqE#N3bN7lka<1fZ-n(&P>Gn*tyz*GO zV)n1BcgKX%?XQ|xdHXAkiIrE`Jh60D6XOp)$ArpjD47_44JC2;+r(|Ha^vCsdt>S9 zX4OVo8)D7f74Vyi_e1A57H_>>`X3mVn)eHz;+Arn?{*hA?%jN3Z^_0Z^=+6}_F~Q` z!CX#&8pJXYl<6JT~pW4xVm9^UE|8S z6%92aT|>)(x`w8jh9ljrbzNOGZ7XWpjGBW@ZMW9!YC7E2-g&UCsja2iiN^>bk9RD(brHHu9+QeY-Y*x%zgarm>}s3a+6)E2V>jn&P?F29AB2 zmfp*Ymh%>wi!;~lJ8;52+m-9bZ{31ouA?YXK2ItBS%Sw(1%HkfFcfzAX9Yj&{tNH_ zPW(&XL;f$NeNmK5=^J?h@z4En($l*1FO>9I<|b~3e^#wke#+QPe*Og;B9xx?wL){vN3^y(pjJdnLZib20Wfo)jNS5fM+x_}|I+Dml)1 zWxUxw`cXnt*NBuQ=~Er?#WH>d;?2Ai^$8h26Y;XiJf=s9mrdp|XJkCpD_P|+=6*fZ ztH$qw94JuvCU1mE&nY;|X_oP`9p&K2lLsnCJq7Aj;<2AULn*O^n;(h2}3RJ$@ z-jy=m$!DpIceeMU>>y`*hh)67oHu2>v%T*s<*5AUO1q-+)%KPu@#@@M9V6Z=m4?c} zH3uuKRpMt_v5Y;U#H;CFR^rw4Yo&fW%kNg=)%4FR@oM=#sZ3OknqK_aozNe3^v;** z7bCqIzeOs&Q~rBpyi@+4knvP6&FyBWl14&yWbTa?%v3!8GljQ z&*vQR=6L^tBiTGJQE9!1Cy4gfmEL2cU%i)%8#y>T-)U`G> zbk|_r($v{q({-ewp{c8j71iUPt19yFkv73>OH&t%zi^U^ik8C%+9#0u)~3#`miEI^ z3@!DYkV$1zw-5y@I^5pfR8+a`=A~WTbq$9^s*RA<&Zh3N?(WW(`Xk*<6f4prqZZp( zSJ8B9OGA?uFC^R4TyvnauB~YTEFM68-Qcn2Fs0wJW%t$!A=<6&jYnEh{RCH(yjNP= z>l$mi+iMy*W1>M|Pw*+AXp7oHtexxx)>>)9;fD6crd=(qO^sL21IurmK-23w4|dgb zVbRBmy1F~N>+115+Qy%q?TuK~u_FGjP-4-0ThKyU@J*Pf|M@HJiEF_Bz-@#g(bGi@ zUERx9Tcek>@+cc4l47Z;V@cof-(XWgjQk($-*02ykuH~3P>?2KgwIGn$0W+>z=VwU z87NNm35Xb^e-N5g=K|QA7da9CEW{tVWI_C?A%`Y*gi`6Rk(*9ar0D~gzS!46r<8IU z!zU3Qll405GqL%kdsLL4%2vam%x|u*O`2K1i9`56aZVW)(*!GN{Db_*ng6G<{#g09 znENm?zsa{*=2BE6YBFQQvxoo6w;o5xnf7j(-`W0<%unl7H8}ad6ZB~PS3O;1ydpDG fGsqP+^PBzmI4XA5Ozx19>ibah<`%=@5IQ-;Ux@^zxk7B+lX&*= zqfcZ)(j%f>C!GHOR^)O14PDSZTr9-Q8E4VacX7SvEL!>ZxSz%KzwivzMOv2bm-f9f z`&owg{I@;E?5Ca1*-K6_^S_kGv1Qco5c|a=XJe;EKaQm@ zUo-p0M^Bv2tf-sabN2DmZ7&{=y#W35dwv;fKXxj1bo0lt$3Ofama(FC_UMyOp5BA} z9;-MOYyZ{p*!^R_ioI~?m$CCNd=&d&&xep%J^RMlCr-bBvb}NJ4`X*6Iv#uMkym4P zd~{0YS#cIJcg;T5@#N_b9)I$5+o@x*7fv0It$g*>*q)DJ=h#QF_NwBpd|UzGna_E^R7*e}kWik(08 z%h(IYKZMP*v9yZyv+s{Saynzh+S%7po`bOQI`q?*-#vQ}_RpjJR-AnymX5YMh_-3# zI2C&xZSqDrbXRPcef{jA)3e8q#bzr`#ST_{6iagyU3~x46J5{=r9E8qAJbB!U1>ip zicmlKoTAo^K5!V(u0Ht8tq$Mk?>KKgO4xmds5Sc+nC#zM;XksE?{kRS*U-lf2Vu7! zJ!81rN$-ddwLb-~gmt8?pTG}3?QER=ZBGOI%J=!DeJ7u{`sZB!_NmjcQu`eFm-ej_ z@U6ds@7?sRKWyKMUd6YLT*J4H*nEq8PW#shn}2D4yqb?)+V_*$xO4HF=bX3By#KCa zrr0I2SHeI4&gqFJ{i&o;pi5&edSdM68{=OVAkjG|Dz}Y2Q!HnE1Mt*DZXXuRwdGImF8p z*I#!2WR`gUfH~uua}U~q+F&RZcj$lFnP>++Ct2-@>y|WqFKd0{Yqp)H$suyYGh3jHWC(|YCQy%}iMJ7# z8#&OD%!g|pt_328vb`+r35LdkBa>n)kWxYoM2?Y(yI~0De5b=e30)cJPL-Tl!iC3& z-d+lfF-nN}BKf$)T}< zh^QPJ4+ks9rYggGhQ|iS6LkMxMNd_l5B=k#6M?X(3sHrQ zt>)`0f+TVkxaOKB{9e>k*6RkS#^g{S6dbs5DiR7#218S*5qc>U9JwL94fP1z5EU5+ z_U+y69}ERXgR*z*j6h`)2VvG97#Z0e3QP=f&`pHK!{g!U38)h`J~kN+jD@{7fcy3b zCPssy{u`p3(KTEbdtWkB4V~P9R>lp`hq^k+KFi!2CWLk0a)qwc!D>>mj>TGUQm!NP zQsk=lo6C@^ELCnUL;fI_g_pa2soy_57RFLOFyi<6LG|w)2>2&QYa;%E;VFM;42#~u z@Pq!rnri>#$oKxpw&jTkzJQvXNzw!fBt3s zpBr?xFkf8AlTZ#FA|O41&A*dR03@wHf6V-JO`7+=1%F|V#eT7b{scv&N1u{}7^j!O z_Z-PK`HC&xRPs7&Eq@k#C(c9cDUMXCnDk5u<^DF^vojx#M$3dd{&2L+&5uG5zuay_ z(Yg&>g(`mf5p$KZ9%+6Ou#6KAip~@k9ydPv$8N$=hXZPi&mb zh^KM=8vW6+V^hr(IF3YRUNKDVoii22#ZF{2Oc^d86F9QhsVb};qmd2 z$#M?{nP6Zt=&7oyTwU(jLAT1aIqI+St_gUMHI*BjCGM zxw)+WoSqlBb=UHjJJE9N2ns8Et5@LYH_4@#)IR6j2m_`kpS=QW+i2yw$>$?iMg5 zqE0;5DgGWD@r~lz4FHMCMfWe&{gS$Wf%{1UkjTHNPjoCa5fe%CB_+fW^CB*&!#aDP{z09Obm@YBt^-S|v$oB7MnhyV^{W_*;T$1+JFulgaJDKh@=?117OuCxsYLl*E zy2hlNnQkVU%P*KFQs%#b=?x}*57YOUbPv-#ChcR|XVNW9w-C+tTbXV(@g}A@F1Y@U zOgEbNolM_p(rrv5j8*yCnQkYV?Qdp!vx#qEnveO}{zj%Zn)oKBH<|QWrq`PEI;LqI zAnk8u8pD;!|6Zo=HR&##X*}e{4f6MF*W$=LmZvd?I)=2cpjr z?E;NChUhYq&jgJM6Fo$<8??tl=YlSibRWg5r-H@@NTB^YDZVSJK_}vSy^nAipHHFv z?)DL$h|h|0b9~|YR5uc>;tldOjU>f^4c*^z~8i-DeM=lr9+Fuq&?#w289N#Cijm43a zY{J>iiy_&H=8*3&xGQx+MKYz}5I6@ii2S3H?&w9yxwU`<)ca3A_xr8eb&d zK)!D&Z#9h#7P^MU2G-~HO>3rDOwfjlVkIG;=A>9o;BvH1;Kl|T3oP_KG!|IoduS}N z&^}+Xd<(@&LO-pQVkIGOG*PT2aAO|DS^`gNq*%4ksqh@D{GAl*3H`Jb>hFZS(MGXi z!P_ZTEcj-MH4DClV$FhYq*%4!nvvjz(0X8E>jkGke#;|;y_Dlpj?Y!FSLLk&&FRcj;*n`ln2UHwU4qTl zIS1^mu7W)n!(U$(#>?D_95h%tDZEz%I?qEijVW@ykiU!Q4x)MdD_BN!3DIaTX&>_e z*?-SL_SO?F=V!K8M|5I7&F>_-f%JL&ET|@$=V#NNTiQG05DN~~)>8b%bFiPm_);8N zUA3BYDe?Xyx0&=wMwQ=5G$*#7zk%q)d|c2&w2uVL{Bq3pdGvs&T}_x97xF`WT1Z~b zS6rVaqABK7`!o{GeJbm7C(%5An)Y+e`p|lh#*Yaf^&i`BBY8O=vHi_NC+4sGjYLxn zsrFh+G|!KweUE9sDh&TYqU{ygUvS7UtnIgCAEA^Wfc@wGeYtUQ4kYD^P>J?00z;)YRspYsf2(*H=m1_>RGEaYsx8j&U9g`+FiiEJ1_{i?$GsPBnpt}ohSSvVid+7j~ZQ{}gqIPxov(DFjzXkYiFA=`funh5P#8lmYh zpCj5kbYrDyy0OwUT`QXNqve4Wr{#f_rs>&A({dm|w}sLT3RNEQFkY8NXgXCm z?6-xo43CLZ{b)K)(l3%;$pDI~swjbkJSZ*flw@-N% zB;(nbIudwWXrV#zA6T3#je+ou*UO2%nv5fb zTgdH}?FAOvvwdpO?NA7+Jk&)WCfOQiSoBkb(AIJW%VY`Y*AGC!w8wfy2bN_5KFkncU@wEo; z&1HK)zQw?RWWnz=Xg{Ia3%JklflAOD3_cqneYT(E-;j6l`b5c7d{)c$hY9poTc~Wc zY%fi(Tq)aE)A{+bJwT)W8$$E&&4snTxw%cCIWeAk$4U^1_9!nm{gLsN1vGaj@RAZ) zelQtdk%!%>1^0MRGnp7KsKA6{!Iv)eDOl>~FT;Rp!3zsz`+>>&dBvV2yk)U$Z;j{j z`YTbswToo`Xnau(#cKkutd#w!@#>_l@fSeo%9!tFUEq-ig$EK zf7AG_x8G*Lu|Md|#eSdn2VB3DH5A`$AO5f*8I=#xFFZ=!tPSV!&2`majdi=~C!KS{a_ zt34~dNa*!*W~&vFOnDk2J9K?Jf#`<1qx|ezn!D?r6FE&fOddJc@{4t<&X7bYuZMNR3u5;*VhXp@qA?eP~K}%&adI1P5+=bNgwig zBp*LD1nj=8F^5zSMqLpI^TIPOdM1Kd1K-(65aY6g;f^2Q>BXR*dl6-|)}F z5k5}(UHqMRwp^dkb|V_kQ|E6vk%IhBp(xtQ2gB`ar8$NKbLS5?f#BxuN*if z&L@_Ii{Q@Dd*uFw&gb$9i;x}pxI8Z`H1hDs_2cucrR@J)3rXkEel8)uWC_s;+{h!_ z#H0OkUZgPL&(L2ODdhHLe+6C}Dk_xwE&2ULZ6v=4)j{@>hQH>22~!jI$^K|P7$wU{ z(%*m=G2Wo?v=Iqga0(aQpRH-(%kXo^k8$=NgQ^nc^1VghZww{LFAL?%`kerq`>(Lb z!x9{SF8EIowcSKljQr6;e@fr~p#J%Yhlgama(mG95tZN1z@BM83L|H-3(-F^KOfU* zuO)0>IApx%g$jB97*7R`{>})AeCj{(Ys4$;%?sr(;l89r0cg{|yAT)1KQELYr}?XZ z#v7EcF!B^>i-~(Cn5O=t_RA+6_UB=IA{xoX-!Db-@`x_Hi1JWc zsQz3SGAMw|`nFEfXi7wV$(f2Bg@r!AbbSsEjC1bGx+s=WiG}0he!>4(2@Tn5 z@pyfv&sCq7<7${MZnNZzirHvfdUxYyxG zGvG}u3z{^@V;k_~xXk;!;MQjjreE{Q&PuPhURsBE^lZiSvt7n6r*VC^>uGs?*Z4p2 z-P%L@7u!6@*25a7*5_pNFpblArPEyY_5KCt14$mD_33nslg)!PPGbot528J@A=QzS z3ZL-c3(*xKpBMN|aypAMs0w329E~{Q28;#X~{9 z;_nA8`)xq4M?Cgy$#Z}4?4nWeqm}>g*-q!bzrDQJ`})$KKb4)iX(W9W^7xwleqifj z?6`5Az~#k(=}WjS;2_C^gQ=~!qPSkdMPJU)?*}H(fmizdfbL1U&-L2?;+OM*7CE$o z|CeK{qSq6M1P@5+v{RCe|7(%*x@Vhgd_{`qufW&1{k{T>um1Zdo|=+O$=^RMxXkn9 z@1N-8KRLBmTAP3EYom>IawrT=rK_aEr|&64c;w*bQu;<6N_b7LfrQXm-`k#%u& z3++Th*lm*D5gMtG-jX--(=JQ?!vBM0D@H_h*)da7KG@rRoznN&iczXu#3R*sWl3eX z>*GsjN_wD^6h51jzE^Q{8{74}fIl*aepu<3nil1QUB6g1$b`=(_l(Y8H5lbjdj`Ay zqhLVfvhu%19gF1Q&xDU?ecg{w!7%ZBN>1O-DSb`r{I#C$|0yV6O@E^@K>8$S*Z&rH zR7IPcil{!`ECz@~`s623T-Jnt1!S9zRwg~KRSe2hC2Bt|gl6FW7jSGU6WLEnO5c~n zv^g2D=U|`8-(Zu|`Txjbc&`4Nz<7hHg8DDVYU0FQk~*W7F?3c<^-mH4eGeLr@<&R7 z{=8l?-;p=-Q^}h2cIY3NBmX~E74j-cc~*W|^NFC+_u2CQk=9okl0JpBr90U2YEgs|tY{iqH4`=Rf%91A#8!h=$NZR%3rJ0y@ zCeY1E?H^V8O5MEK+kYGM|HT~j*ZOXyV88VO|F?SN zl4g7+%o!M-Szj^*A8~KwoLEpsTvt*-7&ol_IleO(>&QQ)Z4XfWK*EKd!lRI zNaOI9#+G$^dV6|@26`f1soOi)+f((xz{XMJwQWmd)w-d+(b}nj))sG|XWd@pvwc_1 z4sTy;%Y*&Xwf7Go|IK4vwf(Kg1M&?cT_b&C9aCJUjXQ<|_2BL78rn4)8G-$_p$X)_ zvAVmxb$3Uge%*L;>+bCXJ#FI+n;vK#sNONrTRlYOxSCuO>Rod!nLd>98Zs>dqwDV9 z)zjV`s2^<|X=(2s?4`D?8gHuJeeeAZ15~%kT{}A-+&tPg)L*@0uy49#v_nuwb zYKH?owR>=lq22fNkB(IL?revD3^nc8+2QRU-7(bHi#|r1543G-k@dmb4@TBa!$;sd zwS&8QYWMclbPWx(j!gAUbd9%fdZ1-nPpx;$gYD?+t_ODwufCgn(my&f9P;m}S<@c~ z3{>@1SM_(&GZgRASRYl3UmuB~31i6a;InC zcfNy~R8>}See|I2c(4n5>7|as%p&?#4__>Rk5c`BdpR!aE`EpxZc5uTm-xPCJni~d zssGyjCz;>RdRAH{`2iM?AK$!&JmvMYMc!N=Z_GNJ@qOc29xH*9+$+~1hpj08OiPP4 znGabx-!s0?uUv4cIfNjx%c&`Wms;>L8&2{~7W{(7H)kEmpbVasBR0v6T!)-ix8I+G zu1kEP{-q!pb32A47xACBNkg{NF==E_vD~&Bb4H1^z4GQ+ZzEeh@;(6@C9Pmz{ry1svB`b%Bm6 zlATNb4OQP0Hu+)5lbs3GHCne?@uQ001e#Kbvh!=j*X{h5s{DF$tmEq6sB(H$7hjNO z#7SkxqdHLYd%$0yjX?GZ(3HZkPyEA*{u{{_Kfs;%w4kN*s=SFeaVP#-<-(04ulHD#--RpicdGcIdZ`fL^4+KSci8xU zYLU10?{kW8FaI&cr*@!!sM5);GSm5LU&N34Iq7^gf3rp2nlJ62DcSSY?UzCGVh72lpOt!pUJ+J!<&x1HWbYQB$z zkcQ;Vl-KjdGd6yX@~0CvzMcop+4y?A*ZX*#uO9EqP(DhUk0F-RMjKy`_Y*e09`A)* zc2D~IhQ|V-X~AzG+#J|D)<4wFuNZHR;5Qol&O!N};p^Us@T&}m$A?gqIJM#cjJ{_wbefD0xogyP9^N>bLvTEmv{kQVr= zd5O}2>*%vUX!oRlaw6D&`FrF;;{$;1CEbpDzizytB@ZMe2aO4#M?^uJa4+x2})f4!=| z&QF(H(+f~}d!G>Y?SG~vlJ9oe`<1@E{!yiW*rs4Fe-QXw{kJwK^Sz+5r~acR)%toZ b{WvnV=S?a(^hFxdVbw5H4Env9UH|_86OnE* literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k39138ce03fbb46891281e57dda50e68c.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k39138ce03fbb46891281e57dda50e68c.o new file mode 100644 index 0000000000000000000000000000000000000000..95c11ca5b91f4056eddfa38d940f1edf8f61e101 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAPh6{Z0m)P3`hth9-CRwk{Tq?qfbjR zEkjO($U4SgFk@z8pN%tQ;>pl5WSpJMIGWAb$QvKxIGDkjY}O}}b3$Z>kpO#e$Lm01 z-246Y=)U)=L4ZB6VJPUnxBmaHs{d8>|5e?({pHeCD|Ag`s%Y#L7ON>JW0}3Ndk??$ zuvAdGSx@zVt?uX9ZrlH%7ur5EgR#Ryw(P!l@H%J9p8GGjp2zE7agU}G6)jz_?fOCJ zx8vB=4>#zc4=Qb;Yc_WH|JhjI(GGSPv`2a}Si$$QJ^ddRv$uYwv%@n|d#cacCie~5 zCQ@HVdeT|J|J6PH!)T{#wzujq_Q^Ly^)YDH7rp{j)n$z-~@P&ED!~5R*Y4|DUo8jgYm%@Fk-UD4hD0J}P;Pw*-2KUWB z9YcfJ?CF#kMgmWSrFJ~7w_U6h|cj=oQXLnls!E1ajpq4}4>Ppu)ImxV$n z4h+sc`NUudJYSgq?eM<&Z-OuA0kZeul|1JQp>pW*!b6~e-cl039S*(wW;kW*$?%-> z7s1!X@VJ?eghmLz0=zzSGW?SV&!gR+pxyq#a=`W_o(zu=Ov}FVlfU0Zm+*lt8Y_4m zKA>rYo76X?J07KWM;R-46|fxMM)dVV_{j&A#i0*36v6L2SFi1Q?U>Pje=E+hld)p!)O0i zzc##|_0v3J+Q1_#fsZ`P`Xm3$`Y*krUwx;T^`E~S}B+D>(r7_&&C~GKMWDwx@t7f9z)Ib3Gv%pXg_3o_;j7L(;Pt<8e;U_+ zqw!&{cb-iM*O0WCYo;D@!9Ew^w`8YZ!vkk*Rex?sRLN-^R;xaDqwO6t7GQrskrJIvyHcDx{WdWI4pUx*qqFc+b@f}wooB-_P>No zl59)8xH|7v3t83`$Z1Hw<)OlN4ws||7e1WDd20lE4&SIc<%~98h!O$ z+v>IkysbWA9V@`Qzy!fGYoMhe;Pp50N#_r=ceZzS`_Y_m?QI>M-nP!XyTE-Ly#7{S zpy94CGfcx}vD%UY<hW}UK;W_%ic}P7Fov%xcM~X&fmwVMS2}JSPR;>Ztd`O=CQiEw#VwC znAN4L>jb7Q4_mcnZwHUwQ9QLo{&H94WH*XTbAYsAw`?uTIXuw{W`D}z<43_aVzX}J zJ9abbNe;(Jp{2YCc!n$1Pfv~@s0S-EaO;uuAme2h;O?VoQMd%NuSBt*ItudM{~cLw2$#a)eFH=Qmtw|G07+uPio?d>fc^W2!( zK5vK5oxga&l6mf0x-2MI&WgI4TN>RR-5s62R(ESxN2j~dx3#&==U!h~xoTZqZE5wo z^2#-Jr5o3mt|=)kVe>29i{10Rfrch`M=Q<)9`?F1H1SI3EBO4w!s`z91mO0C4o04k zNB8OC9=FV%5xc*c;~ZO4Dzk63!!Bt1uwC?hAC1o7fjAkuoX+gBZYN7$E{xV~G`gX# zGqFntw{nd)qseNHO3DcO~ZeKy92QC!KeRR(S=z}vJ`=12D~ zIARrEb{A-1F#P8)`*h)DwygEcp5kES-90wkFO~2eINEB(Ya0+KG!9)qme*_Y`Vp=r zgg_y`7(Lq<>)oYA*sp*j0}Z9ItZtz%K1p(%?D)>a2tR;J0Y$x&Eucd>RY0E_V5u~? zzv{A${UV$7?zczSWdMljMkm{|FVW7&-ISfmj=R~9x`hnvN#N06Zx9nnf@u8GZk=r* zUi2QD;O9P6P-fzF&dCl3>~^$Zk%5D0Y-l0N4chK|`<3Y|q906RuODKwf(f?YOnGAm zuMcJ03xc|h@Qe{2@&zYGvQbNz3~gh)S-~VdN`Eqqr3WS^Or-t?4<)g4M9(;V643*n z8?;ZPvrf@J(er60=+Qs)VIy(;RUdHvb0MFB9{i+9dx0PHNAz5O8J!yOgSt-g`K!V< z;x{Qc30CCxO;C72A7+Gi1)P(J-_)~_BsQ4F3jPfJqt4}hgZJl5L(N~0{?PxE`&fkR zVRyiuLzV3QT|`R-@?A<}U+Eqn;pQNGt`qvv$Jf@>)Npy<$cDU=b@HW%1NuS#xxvYk z$=}9v{fjy~^OCOgQbXO#7Jg;|d60xBVLWK>vEc#+-yFC6IlCGx_k50q{SR zW=|!5#zN`e0{sA=5zL)Q>n8k*x6cj29K$u*WRh<%o0UC(1XwB<{{qoJIP*#*qQ&Uv zI7xpIDPL(fmI<2J(w|NIz`qIt4?f84rHSzh0<-3Dd8qyAG|bO=5q1{sX+GsmM}2OD zU6S?s_q3V8d2?vIdBh*=v!~lJkI%RJr>C;Nbh4Sjxw8mAm*W?*C%V~OQf=b>SagO9 zL2m~=);6L)=Hz%g>H=R3yc>A83GYT-;7fs@27I2t4~zEGP#5^G0-p)IN8o3t@%}SW z7x-jtPqzzrETS}ilfb)B7x)BS$g@S_vnBsqBwhoa`1cEZ{X+QHgAbB?-2chHf;JoX zCFO6yIde#y7=G5QX#F)lU${N^{6l-vmmU3ce%`j}Wm(TLT2~{Hz1;qpuy3@0zEIW4 z-tZ61=UhHtX+HlVjTJqw33BpZ;LD!>nQSiVqx&<{!(VgBp4?wC9{JZ6B9ZeUBRF#= z#Xs(kKd1gzn)>hEH9h7}LkF?GCD_Q{_@Z!nA^Z#Lebm3mf2JqUc+4G3Ia6GO5n}Z72L6F=yzsRzslak0NML((l_0sNPW|IcIBT##n4Zuz{~kOTK^I)+e7~|N!`jn zNdNp3DZ*pCvgenX`s>}55z}{ln?>I=zYY54^EHF(J45OZ{+|)mcYQY23ztRTE>}$7 zm836MY<~0kf%a10=szO}Z9#y@K(APTX?}16N_}OS<)!|j>%mYVE6bW~l5Z@1_3lDG za28Ey{mS_h$Zs?F8_Dmm_?w-_PdmwfiqoMlvO4v^nG5@UbknDSg2;}-*5F9_A?9azXyeLb~5Bb?>`6X>^eUkYB@;A(%`NWSZs2|@6`#lZ- zNdBX-_zrzS9!q@J4e>oYPCQ2rOeT@~)dQKacs;aGTN$uZTMs{)~@~ z$)9_YXnie9Rsslnn*ERTZ!^Yc`|+K~H3%ZN`xJ%+ce zXMBCLyJS2W)4wYj)@$w`{w=gWfq(3y$w@`|2j=HlRLQ=uKl1%daz_yl$^KJf_J{wQ z=jTFB&+SQb0{ScnPSa_*Wu(6|e?jZlv_y6}!tF}@Q%&312EyR8xCllhMfPR?v z8xHP2$DWH$IFrSAjQ_}<4`c$e<#stFn7%wLa zb-v$3f3R;Rxr?9X1LQ3TW@bj?nUx>nzZc_OpC4Mk5YUN0__x{L57B%w&IhbNCVvOs zG#`LB`8)82`2hXo2F?Btf1ghF<@pZy%?KLi1LP%q)c-GOluyCGP$)jWh%X@`|6*>` zNx$b7qK$EY z0iT4r5%0pOhiXswbkvRbj0tA^1k_`A$jkke>jUyq-31PWer%Y}xntzxd~%8Sa-q^3 z&t63SxSOStf5V@~u{SEopHiuXzr=4B@#KrBQ8BC+!>ICjFrDm|`W&hvo|C>4u^lnS z^Tb5zU&Zr;pqNdp8lxb8*wf9>0ENy+u>U&Ljipu;o%ZM0AB#gw+8@5b{hhQE@fXSU zEP06}7yP9M^i(=AiS&^D4{0l+`GUazG{~KS@$unbz2HYX{=i4P6JAeXoF4+bj`_0N zfujNx8`g_c7=O30hrj;)aqO2AZ(rl*drCjIED!eUeElGQv_gy@Yi+RLGcj$jg$K^Q@%s_jGqLi{DboMow*p z;iK#InE48y-OjC{pC4+NQ=AUyxT`F&Z}%VWl? zqo#OeIA4eV+)liT?!U3VnfKcZ@%h47`|aIa9JN02_)JG?^m$PZ#gqH^e4jiKQVylL zOz{cpr)ht^mEuz(-=Fh%hWsdfx1Hxt*bk(!*O9;OPDrG9o-{GdLHSTY&`#kxf#J&?Si=F>i_qUk8H?iNI zVvKJE!3jFU2Uuc%Nb@6e!bDg|?7y?I|IV5~pU)Zg-_vNnXoz=C+K|Nd-%jTk`)?em zP`RCWS9u%z?{ut|U{LH=(cXo6bU%uGZG!Q<3HS`-c@+77Cgq!A|HD29(vl$e)A^k&Lh+ElVg149#r(WioM#x$!;gaKCeFi;;XE9aJ&Et*Sfwe= zvcFE#vEOH30|EDE>_2bFA3hIq}v0e*DVs ze;0Y%&Pr?6&N!y8tw~s0gWU*1`cG$5T+gR3j%u3x(ju>;8hdaK@K*?+1l7jbQ?Bp` zUC0AiHq!iH~vRRWL$* z(Mz=Byp6N+hGr~=-hFb*MmQV~5H zbeiO`_yecnwj`VE9c4?1%(&eQde@VL-lKBUNOAB{xn1$`QMuhh?s$Dfu zq8KEze=3sMM|%`m!o-=RiE;+%Zp7K>IcVnGrdv6Wt8-m&79ACx7;P_fU0BwZ5^u%| zJr$vi#;l`N0cz0WC_HgPI#{dgs~^PNb8`>srOjEd=CJ{*ijf}yE3DU3^c+Xx@Ayig zjq)xUTdgS86(h#e^)#sZjlTlL+V9n^GP?}p#1nJ!toV3+es2YF^mH&A3Di4r4_9HDduU+Q)lfWyph_%aRS-beR zg$-+$nf`4{`(?J>pCs{s3ay?& zJ=;(xJk_}}wQ;G)@Ms5M(`JB;wVIyeLTvWzAsuT7%vkC17TXn24kvqct6os2UT2_A zU6HP7)Erfh?K5mTjZ&Dp-d^Z>Eam1#qg-Sx-NegPh|^oi1>9iQpkp1O43JnMjgXE< zWwCK`J(KrZ@;vItXniq)VPs|<*$ol{{$8wyKHy`fHTttyBR`EXKf9>CX zsvBq6y%;Xmkg;_3N%2OV-;3eqRb0dh6(2^&9ol2YKS{H!?z(mLr+8u5$!XdZWHD+d zCnK&J*JPnTNL-?kW3kDWaAWTGCfwVH{$b{7yw&|y=l|_TgHL00E4T6bynPO>z5k)> zrkR>J`n1r>Pds|de#n#$jcOm=bMiTu5M4oN`9ZPL-)VJ|e&VcrR(ayltIZv)xS+qI z_m`Z1^I6{K?QHM!u3yu9GVa(!tdB*#y`y$)9(YhJX76RjsHbM_Te`lc1bc19&%U)Q z@IE>W$8KAdF-PE^m3YKwSac+7IY$>hZo&_U@5hPlf2{w}C*xcKL;t$Q`|nY_(f7`b z{Y$(;^FMQ2t83*hoG;LP66K5?2R3n@VWi#g8AeM~bIY}7r2ya~8whk3Cm!-aMpY$I z%A0xqPuyqwY<9D!UdVazyDv;^W#60oo8NA<{ne~_GxB~i{l|yWQp;OX7GlWH!SjEr zAhjERRF{_vdsKiYJpbqA6^v~u!{3(Ub-m~Rkb=k-bdC3HAAgNXr?Q=yxUkktT{mHlX#6#!~Dk4Us*peNwwBIpLeAVklNx@(cNyxL8sp z?e}H-a=p8OtNjN*SlP7$}EJ>9DI%8rN7F~K~9 zG?cTJDW=9(?Uel=L*pB@S9H{#=&bE&o|6hJZps)rCQ46^xKewf=@HfFJN{&V_gNf; z$eU&IN`uP6|IpP?C-8USb;J|(Zz{6A#{{%E8E~||5dTFMIy}Okm!BCrj%PnV920+E zNA{B*FWQ~`V{u(_zqn@MLh_r4Q2$Q=()cJQ)Wl5!NF5uCq>X~sqy64E<9|iA&y|!k zJ{61lm`}F%SjIml+bjR2_TVK(cWVje?YcZ z{-?%Q?G$_}0P)(t9j84_32S@$u_#0xKg?{>|7F=;vLC%z^}i17|7#riRr|}bCES$T z-#R||1nG23y(F|;GUb5ks@fCycH8kNF6XCZxafRU^VT>X@v7#9wVkSaq=OjSagdx( zicVeC{8CrdZd>8H+K$4~LQ#rW7cFY(YFJ#%AKqyEbjC&TnYlP_v_8 zZM(Oycy)1YQBy(;lF?_ANas0pxjHH0r|S-y7D#)_u;4YhfjH&(3j zZYbE%Qc+p6ZfDtwmKBfIH2JI7wX9rJ^Jw|b@{NnCTQ;|@$gf{1+7`7eZ>ir{l;_*b z`(3qh(TcA6*5!HN!C${@!gZ^Q+ScSXwAMD& zZ?0}H-sG*R@mDWzDQ;>n*|4q3TU5MyMcJDC&1I!sn_64CHkaYGcy-t2l@co3-4SrC@=WWdk<}tP_=Z=1S za?T&QcYWzgFi!r0d~PDO7;BGLGi~>DZCYwJ9^``Gk5mT4HP>)I5BJ7B=99GGQdysR zzvo%~i2ctK|C{4$so!wEAbKYHMou8!c8@_%V;wQjzozb2I**R~hW>(G)`{*13Ww_{ zb#I}Y2KYIJ!?i@+mpEO|>ff*vp8w&mgj2t~81T7==qz(8N~{1h7$DYU~8(k%t3Re4t9a z=RL3IU87N)67M^rdp5_eh}f`!Ez5kAC%a;J-)vc=WWti3fl9I`AI=PW+tVc3@1+ zEA2Y`Kaa$V=&Pg&UXyarnDNGaL(1!xCQ$RM8T8c8WeHbp7fHCXR8x*8Nctwo&*p5n{lTV(wo1(QnR-3GWZRXir) zG%ijPqvg1~qj8P!5-BJ3uXrQURT6IPXQPBqxAdd*Mg160iFyqE({oN#dCmYg(jUJL z{E7i?ls8xE-OA@|3Af5SAPr=d_jL)k_VY6dx61oVLqA6TKQh3L@@C5UL*p7_WZre) zMh`rG#6@O_lSO#x>3tmxPa+FA`4uC|*c_Ee5ze;^A@~F~FmL+yl%KIhc~y&8Zpt zf&p%nt4YF@9u$3xgj?l(QNpeAmdKE!#?6!Ct}?)l;|3*MjjQN)OSpC1?@73IT*_Oh zQ0_ugO6iW`8Wjci@ET)H@~Yq}Zaim!tN1WrfvY$$Y=Nt#SDk;V{?&Rv8~jmG@E&Qu zl@_>K@BJ3ITJM=iyF2RYo7=pBZiYQVqj#0JtF58wQT~&YrFbASe%+D(#uNQqBfHt1 z2X$1}+}4brd93f~xV2_&U9EzlW?u&zdy^x#4*cgO$EJSF7wEuGLr77Z>jRLx%-1Qz z#SfRz?^u?tshN-8lWW)^s8&E9>wKMso%sF9`mRnNK?OY;nxKu|65nIZ4L%cGXwTOq zf9`H<7Pn$N`uU2wHu`zIRjX=Omo$4@d<~te+Z(%DF#JvK{F z36cd39i910Z(197$X@v7jT`p|Q*UPjZ))?&ck^OCcBtQOy+eQ5RV=Wd@O>|Wh)lIi zJzbFr>QtG3D5PAbRh7{`pI|aAh{8Sc?u?9M3Zdeas?Wj=rCc5bkXQdsar2za$LO;) zigz*`M{}^h2av`$=GN5GC4;YrDu$xQSJ!7{d+OU*s#Nf9G(uzJD=49fC#w#t+60B- zSJ7MR|6PuMz4jkj5Q=ZrUpj=h+Jtbc0*b2aneoumAu6 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3a806c0edb8d0586bcf26ba141c1bee9.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3a806c0edb8d0586bcf26ba141c1bee9.o new file mode 100644 index 0000000000000000000000000000000000000000..b971e28a77d1dafb4e94f02974ac6dc5aa176b44 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAPh6{Z0i9+G9V!kug$FJmKr3`qfbjR zEkjO($U4SgFk@z8pN%tQ;>pl5WSpJMIGWAb$QvKxIGDkjY}O}}b3$Z>kpO#e$Lm01 z-246Y=)U)=L4ZB6VJPUnxBmaHs{d8>|5e?({pDpVm+P9wRMFTgEM8Mk#S4^*B%FtFpjrw3M^etO`Wy5}lrpTXB#W@_aaZteK+z@%lXLhQ}0I7rav4W z-f>{ysi&VFNT0hnJgfOTkqdKlqHr{+FA5I%JxasjWV z_uh=mK6x@y;e0Q0VeWa*EDg_Ud19a!x~MpR9DSdNgioA`R60*b!gDW0o?1;jFA0ZF z92l5=@`-^kc)l?A+mU^9-vnRK17z>TD|OZv!WGcvg@-@`y`?38I}(2P%}CnTlaX2H zFM_X&kulRB2@ex~Ie2~OWF-8=d9?czwA(*W0ocCelab*E0n>7?{3Q9SHNzKQ3a0MK z{eEh4cTMW6xgoNn$Hs~ru#u*B*ZAO56Ex4&v$hGngxjLCqL}@G6ZW3~{}Mj1MPo&; z!v{2ta1(k5b;qOB?kHnLuL73G+lan?2tWCtsxRTp6}h+zSn&N-#a>r?;W-H9{Hs5zkZAVDSy3@4_@2#!*H~8;#)%x z=!Y--Q^{4gon>77Rms)imzMSYdM)cq$NV@ugIyi^p?>vs`1N2eyLz^C!r{Ss?eN*Z z)vpchXMHq}nAZQu3gDy9vcBj)v%X8O=vUtu;lR21MKj>J^=TThatzi zki#fH$uRs%a(Ec>Ko@-%#<9bfpp)U3Qo>B@PAHdyd9KdF_^BG}qjlrssvm|4_8Y{k zQDYx?B3(5mV_O~_8v-#gIjIq1)Y_!&&~^zA_Z-^6tdUX{4-#`OTMC-C~;xIc~S zztQ-x*E`Q9g>A5_@;}>HcJ&?T-!)wiyI`M-@LRIeuVDkS`32bg>_XNz2>X#8luy6E zmi0wSSs&B;A0eA6`(OSRYz1F_7k>O6Z1!)kU)&$q;me0$SA!iJrwVlh_V_YkONT&=G7}xL{k0zs1|!w5_#paa&>Czb^&@)D;#jz-v)mVL?H`!Uc8s zw_s7h!-b2N)YTP0P9fJ_AVzzmH|TG;djtvk+x@}EAQ7w-^f%uX-Ud0mcZDDgeqZOd zx~)NPi(gpB3h>S|K`_ngX=w<01C4yr1%hoIZ5>?!G$&kJYkP;cwWHuJaNl}wpv51o zzbniP({Nd=w&Xy$baDe40e687l{jXf<Y@(7uU-1u@Y z4Y`+w+-YfH0@uHnhJ4Plchj;(*0BO^J`K6^_c3abUW*OZytb`d+x;B{tgf#0vAP&$ z)v{G}0#jFjty+_}ok#B&o?0S*xvO%r8%3sBK-#cdww9$Fo@@oPKjrYTBj6jcS-0^W zyBYN)hvTHsQeFf+%N6gZJ5Lb!z{)h-dL%u_c-aNG`)EcCF2U^Ul3oq1=)ViN4d)lu z5=F_EK)R;_zIW3#BYjVI_e|!9?&+TC;J1tse2%pfSyw#UzYc(CYd=GWAXLTc2wqfQ zy}qKXqS#%E{Xfo8M4>|xB3+EvRssb&Lh!}X&IYfWP8XV*y&X+$t?rJtw&wOZZp>`I zx83h9TsUvh9QS&<%qv>TN;;dG8{F+(?H&FWcS~n`hr7YQwW-zbURPDMa&6uEWi@Ln zs#e!6+qiDo>at~JY;Ki%p?j`3Sl{SwZ^3!M!(KOrCSK`$1)qOdc-_IC0Ng&`!N?N| z=sr{2+qtO{W5GO;IGnrl1?PTf8#j(1LMmN-T z<~G&otjdU|KB;a)x3W0_ffa zN36!n?g9-AhX4F!pDMh}ma~r8(;SSvyW58Q#S*>)M_Vm;Z36;@#-Z!S@_J2PKf<+) z5GdpqBWD|9J-f6h`xTI6pur55( zi{5P${M?5M%5=QWIoaW$-HsM4+J7*E4bErzA=`bIz4x=IelUf-eu&KoCE0#6>5Uz{ zKA3AS3h6e&Ge&sG7aAYUMJ;I}w2kp*gi`bv{mBfL862N9p86j+l)}ytJ>&E#L=Sv^ z$UdIVIz|6P&!?H7NB_`=jl>C5f57?AhkOQl@RK3!1%A*U(R2M}b!fy7>N?36sE*i( z-;~eKRmJ+Q*z9S5Wx6?bpx;`qzUt<|I5Rf$STa&WDfoga5${ zdph|u7RtaD=m+?$Q2umUH{oBreSQe$7^>AKl6(WXto->Sz*52Z7l{7B=~tprElxkn zN&1UQ`IdEInV^X+{n^A1{HrMV;Dg*=ni#JrIAa!WJzb7De7@a3HJ$yXlT8oJo(z!wPokZ3;zb%FmX@Y%q71b$`)?>`%L zflt-;bh&`XB1+>o3cL$-fltzfJX<6_SMt9_;x*uj|A4^z=EJ`pe30bh{!jiDve~#V zDSr#innmKo@iS(`>aX$n!tKH5AKH_??C78K^R`Vd&v}l~x*Cn{<@V2pePadm#i~yB zhJRo_=kxhW^Z6GUtmJu3kdyxcU;g~hWOGp;*`Jvn{+dtrXd(qy7_R8N@lYXbTO#bEo-Yx8TY%0e)P#5jV zAMLYK*1EB_oUg?>5&Ue4!{`j=?g9{Qh7>Q??i z`sbfW5gy}}KflD(U(c?rxW0X@7JbwFHt3tr*DS8@EU7>Ee^yN2zFe#qE{nchuDHId zNMEk_{O0om?WMlae^vk*@b-I zESk{zmG>u*-)8VPlHXzRH#?7?c9Q=Tt!H+TKhXN%qxDj>SMhCn$ie9yaes4geUQJM z)AW)232)`=G3MV2*iRHO|E%$l?0bd?` z>2dk;oQRj=v(o!GzZrTF#h-u;CZqM6=8w+frA~N!x#7pCXxEpgW2(TJvd)m5wug}HpF*bC;RgFj{FDyjE|1V zpSx3NeJxK_0tkDW{g3o-Gsb88@tw#wlzcwQcO1@NUHfpn$alzoX;F3#19JZ*``p++ z;om%8bjbXYXguh0KO3=u)KC6N{iR{J7~Y;08iTqKPt(_kPeR>@Pe!_B#HXMh$6MAj zzP{OAGM=F>zDi^HOZpiY+9-UJ`oSVFSMcNXBFmm66#QRZ2l)_#_<`z zr=V`cyKw5E+7mt#bt67&oEbk3^*A2#a)0IefV@<9fdio*8|HKVDET;_Tq3?)s4~a1 z7m+{iX6fYL@TW2CjVkh|bZQYO3)n?G`66mm4C}=ZsyrS{CHtj6hpLF@r0-;GM~v}2 zIhp!b@jNLcW)rKyD99i7bTKqQq4N>!zYcX_sTD=1{W;t-Sehi`CyC+$Q7C2~DW zT_ni`f0;o&olZ=m-DLkm+VWVwAn-p8ac5wBeE3%n_|c9(@DcBX*OM6MhXAi*zU+43 zr~t)=_2Lx9-!1GB@V!5V{gUGCYy5mq>F1W^!G4{uALMU*eLiEdr!_w_dRiFpCTW@d~-R-ANlzT{L`GT z%m?1MKGS+-%~uTTFXBZG)@v*$BHlxPru7`_vuXWCJqPP`j2`W=9>w=tgPM80#(X7w zY`q>eU*YrHlCRKaoQm*AzMoL`hdpurm^~5Ls0;J`^rx7wkUu2MSEh}Uuehh(IA6)( z`HDMnz5;o2+!Wtq`AW!;Z;Z@W5KrAXv3z9+`upVhNDi&9JYJ$dOFn{lY05_&7;l;z z{C+ql3iC3}xH`X`vetSMXUl?t_y_<`p)+ZjH=}3(}FUq5Mavz`X6URf!!3>uv zK4JYd?XR~|d`jl~a~{u-A7$>g^ZW_>feiLK^4Hx-$rR61#%DMvA1VsjDO@M8AV zI4?T7fhCG(h!>gv&iiju{6apZ2l=NDvCkg}AB$(;KNInc_CwKZk>A~VJTvV#%<&83 zoAw*fw<(@+ym|j&+HU}Fif=%V3_h3BipW^3N zSpP@v$0tv@aeSM^{e5!c_=fl~X)@n`sr~rK_+~o)$q!AQWQlL_^B?Q}7W4Ne_S=(; z@vSH{PG|T4OY9G6eq@gu4-1L?cP{qdIpgT_Im7;Y3hfsS@ymP8wUrI9PugGu9_uX22Lw;oVen19W zMTbqi81h+K66Aimu!BV@9`ZM=Klr?upZALM48wW&Q4rn4dH6A$hl8>^`F$L#G-g=# z*BLtY`|N8V;QoyL=MDM8=i%@NalW2IpP%y2BX#<~Jelg?ca>$o&G&0D`J9a8iAO0x%)t7p!f4lkb{^949WB#h) zqYHl%JygVI_CD3)s&)6y&O6rI%UGYi_Sxb0|G6&uHI7PKlk}Ipc|ELuwV$@D5t_t>@a%@YMLphgP?r!lUu5e4d2wsiq-8?9r0Ja;zl6#}w;I+@5lIUfhZC zNd83u1Bv?Qxjs>N65~a#1w0g!sC|t8Epgo|8ziES$rE5~ji#mQ&~yyOA!JV~qGy9n zQ#=-b;8fg}WRtxkYzdJWx0^xldWz6{Ol}$}0X`` zgJkwkMKb$Hk0MK$IFmFn&LG{5I2$(yn6JBUI6vPYsQHO>Dj4!J?zR_Z;KZ&|xlISt0PIVYf#2?h@ zmM?w4gAEh?+m`wg&5ZY0>eZGyjYs|SB?A3#6erpQ;^W`2P9AZqZ9XY|aeA5kR{KcQ z=4N<5!&lVNM!T7wM(e|hPt@mU$BmV{(dx#T*dt>Q_cGyDV9B%!G_YJ1bX z)Kg;f(k&RUwojyOE6wM0aPeiZs0IA~u(o*xk;kHl0Q(PG4s)c0HDMbE8o%GMa7@gl3uxrqti&(a7=GWJ|g+_j{A>?L+@Cb2ZWGeyj8U_M^e4F}ju8M19^qht}Tz&~?*H zO&)n#Xyqpny=6aS%7;d@kL@}6988L>Ahi6TSn2Pyx=B9?Rz9mdiRjhlj#gaI-_iR^ z&cFFA@AG!H_j%W^X+9ZuY$Ddjqu$;TJ2nqIC?2!-GGo+Jv-T}rUtNa1HsfdC+7);o z9fo7Kt;(n)@Xty-;xjBdlC_+p3m-S(`^ER;#P&bl|HzYZE`gzcUE}?CE8ggPXU6^| zUZMG)xvkN)3Kz~7Xg-N@#*PD<1kW(iZuktNMXI^w+Py*m@R1D!I*Stz`5>d}vKZyf zJpU)*vwb$Z*;6m%z4+Z1#<#HV&Hl}AH`@Mc#++#dKbiXDLmBB6&1v&7B=&}sPZJHl#Piey%>?=IbHF7l)tFw zbSYXi=S4iHD_enM*VuEqBt+522yXL+W;#Env$XffdVTBzWdVq<45Ch(fJE(oEZcp_ zqFQTv{4u2Ej7#*pg8oPo#UUHedPid^c9e$59@ajoUA3I>jZ`rdD=qnj{7hUdsgm~l zvi)>e`LCo&1N|dn{};NRzvPqe7lXyj$B>5E5U{6qoq>S7iHq2~s+?wx|8v6A9WM zv$R(|TiZ_)3JG`=fxMoQ?LUxp8lS#vVr@@;k1j^=-!G?#+t8kFReNQ}L+F@b9zq(* zS<4htViTXDc+1_IUTAU0xT3?9&5(^z3;m^y@3>_!3UjUAYzpo?v zNskxp&i=8uE}}<^q2%C2@|%cI|4#tY_$VgS_)P*x9UF_Jje*vo{oVxQe?_*>my|R< z6^r_qU$*yH#y=+8EB~eT^nY2lC!USPs{gfU|6dcxui9Uhjp3%; z{?_ryCrBq-e3H;|$(94Et7=c++il0AxSXGs;bQYu&0FJm#H*SY)^@7ykp?!l;~+Vo z6rH-N`K7L^-L~Sj>)VT$6^l~3s$@ZPXZ^yG=1mLN)^0CuUgzD=(79=2O<{e@hT0uP zYudcUrK?KUmo(P5EbrL7v8K5`uy$Q@#m?pR3mO4iTTA$|=A~;kZLDncZCGEhd1K{D z?}nls&6QQPYj>6}Z(jarZDXKjZS#r+wU1Wptk}4qrg?MA@m*J2rq=Z>8yBwMep7S{ z3P+<`zPTmTxOrg}^wCmIGH=~Xx+`oeTd-}7w`6tUrj~8XwinlQZQ4*2OgLu6&W)SE zx37Hp2m6_w4KHY|X>YYI2DuB~m{EbO(tq;f0zE@;?T*=e0qhrlv)YD1TGe zx|ZgqVBL;|-X#T#>kIr1z9kI>MN1a@>bEXf?DG~bDy%Q``Ta}l1YLd8R&TxkdOxwU zj{mj_{i+In?c{bU1vhv$dEvrEixw>`C|JU2mV2Aq{dMBKv=%=oFYnZlssx~D(7Y}m5??);F;+k)`pM!hj9`i|BaH*_I zzu)t$e#HLg$^Xsqwe)W|Ul2VLeFG;DZ@WjKr?HM0=wDO!E1XBid_#Z1F6%`11BJu& zl)AUjO#%Fz!r@w`?#rC6XZ3H`3D5uVSHh`ZUJUqrLok|Y!4rL%0U!HSq?OK>$It*T zh?SP;ns0(mb#J}Ch_-h2zB-aHq2+_)OfVsbQg$3gENJ4Z2Ljk2F12=o^T@*iDn3vp z(es|y@vhM*K}qzSX+h5;;)oVD!?+45=;QpD;c_RWfVc{J%yI`H44eIk0=-z0)Rd>!}?04ILVa62%j z=9P9G{+~zUMfBCu1g}XsXv{?8z9HpxOB1O1)dYI#=dy&WwhJU&ZBol*|9P^XYH2d1 z=f9WyqtZ#f80vE5WZwxLv}PeB;oc@*P5rm-q=5c!7jZ0$jBv&=)0qGT_E7<8A}om?|EV za2gkaH_DqY^={>Jri5GN?Ux3!%KN&6Tl@K$gj?nPrJ)}q{~sCPMtQU4{GoA;F|yz~ z@FNmV{1{iW=Op|g3%uVzZ?x}g5^m+6o}xpA_?aYb`THUXSK}&QY?N>{u7V#i&>P2n zPQtC@{!+rN*myq zeZc@X%GD_0N)L*@S;DRIz9`{VdCO$TQR5cKaaS7P#&JUuuEtgLyCvK@?)N0zIxgic zR48|$DW!BragBFez96I9Tnp$XdHE%QItRPQ&zh4%c7 z^5^bGXK^dWqo1#+Yo(vZTe))msbDOuJuA{B4fiorq z0(*i_Q5CWnC0yPXR0Ov=7RztA37dI?+uG~e1OEDvzbhJSYw&h>**yL(98y?mB_t{1 zC_yr>zP+Px@l9*v4%rLeym1r$VCwB`;7x5l`EFj!$9DDGt#{}zyNU(&6Ta_75Rs{t zsi!M4L7gn~4~3M=w5l@N=Mzk(1u?is-kp(gOd(XfQuP_Qp_I#`0P-5xDQ=#V`51k+ zM)6LD<5&*%_W;uP#@w1(x@7S6P{mNx`0DzsY)^d~OSKB#4Mu2ud<7*m@nqFub*rFI z{3?2D{lClcuh;$~3qtX&`pbmyR+|uRWl&L=uU6Xavi%78W&39=4Xph8flp*VhxDJz ka(J>Ixq@o1a_IdS*g7^fIonR}_~CIWKe0hi^0K!7|3i);H2?qr literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3bbc8b2a3a7b5c07d7939606525983dd.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3bbc8b2a3a7b5c07d7939606525983dd.o new file mode 100644 index 0000000000000000000000000000000000000000..c7620e23d3a357b501e83897a0000c2840b87eb9 GIT binary patch literal 67256 zcmeIb33yc3c{hG$(P*?UfDro#1PCF-AhCMEBe6(e1G2C|&?KYL0+P}8jU+R&>F2R< z9lX>ClGb*YggS1Dh;iuhv?xtuCHCXEj$;#^eu@9-|49jUTtQ-+Dvl%B#`FE&v)p^` z)kwmQ)2~f^?BU$^yyv{yXcyC>NWgccg-vO8S*#q`*&QU=~Pvh@~d}$v;WsAZ1mmT z&i;3HxcaZU*s))^*r^kp>=^2PkxgN%{%&4i;N1=E2mj_|$MRC6)fZejr!Kmt60I+? zY_{r0&cMJB+BqGr9|Y0&lGI1epL+Pg^GlKc;-Lr6|AjL!in@LH9c#Y)^zf;dP7fb= z>*?XAkT2Qx%J3a;eSdf<@)qP^(N}A>hFK#k@I()e)#+Y%bptE`ts@F=C+rI zPd)to;Y(k8bNKRsU!cy){s$g;@cgM|51;Qx-K7R^z9w`3$aN*{!4ktbR(r|LcOT*3J`SO7`P-ksF z$!hDU%L7J^M$AX@cgBl(O2Yrw@;d7(3hGg@5- z8=K|`j9zd}J4Lu!CtGFM18~9~rorBXP1HKrsuy4r4hP|;ox14s-c0RIFt+O3D4WmQ z2wKm;Zr<6kq5s|8<*>uR=+(QQJE_}qJo|jd~|uYytH7T_0onFKe&3gqyK|;XaCg=?AWElWXa6=j@XZ5((Isw{F`XT*e#hlG z^(&VfzQax(oa-6=`|AUvwKF|en{zx@o1S+Kocy=IXmguup#MjK(Waj`Mu(0@k2n8~ zW3>M(Wdp2o$bIq^;CHzOnx1fsp8P`DKvOZ=d>8TdwF?Vtvk%|cbR6Y}QJ#bHXF%hA zz@30UbPk;S2f%j$t_S=E;I9L21AH^!9|Hag;C+Cv2mCzXKLz|bzzYEXJ>a8&cLAOP z_^W_F4|qFZ7vST7?*Y6S@Gk=cP4@#n2zWi<*8qnBe+~IQ5Gs z1b8*z4S;h1-(3#AT?61}wCO#-YmmQ-ycGE`^5w|iLcSFFW#o&Izkz%K@*g9ghx~`g zXCXh2d^+-%kxxPXBJvF6&m&Jk{vG6r$iIc$h5Q@HW8mXyfcOu^jdHgZPu@w28$iIhi?LuA;{Cwbp$mb$&N50xI8avE>a5rLP zt+V)8gLB2P#*6-A`@Z8pw*LwLv2v8hX0d^vVlIzOVFNK28+bp34N!bVyJMj@zH?*k z#qZo0a%P?my@l(`be(ZJ^e(Q4={o&%=tEq;N7remsh#&l#C>11dPX2x-TZXIRo}KD z_Y3o)**$-sFk1VQ-WyrvN%u+2|Go<+7P8Mi^RwE&Nw`Wp);gCSYxvHxV~tNNJGQTP z*|Gg~%Z}xNKgHv9#4GU#{)o@lCm@u{SCTKIURZ)*WaP*DqHT^d(&X30)@$er80E_q^*K zZGOuQUf_#dM)14;4ViuRnLnr(_-+n7(6Yu$mun#r*t_Py4;2Ss;NKs z>CoqJy_c?Cr$hU2T}Ri>)1fw8-$B<7p-ar!NmK8yz8SKA5x?dK-B%ACcaJj1;Em1f zM7Zf<=CK1MNdtFW=N`DL9KYl4f#yftqYt2N(;s6jz3ze9^4=Sxfq!z>ZgyS`Cb9n& zV(hj{SU)CV?bzJAao{Fdwghe5R7U-P<`(41Nzv+pyiwl9-r_@AdF?;U$;z z2TKqaZ*ss+omHo|FgEH-V5dXQ9jAQ>4&>XBV@%F%$X%zc^=~=7#jWb!EbDKT^|!>S zUwL|qSJkhO^*77uVOE+5#M$wrrA-E<(Q zY;-BsQkM|#4SY55Qzd>;9Q;1ub0xlbEPkzH-dUozbv7F{XwEGg-ReUddtdWWo*$=t z7y2mR<(!s5CyeqU+4t(PeIK|Vc)!H2kApuB{7Q-690&hZ;MYq0_Bi;z2mS_$-xUXc zC-4E_r3?(dpGWyd+3s_3+WiptN{QbW2mi;w-x7!J8|4GZj?59R%eFY}J_P<=iN7lj zK4$~kXNmUr$H5l^f3L(JkAvSu?XyMuC*t7SsJ+DhO&t95)P9O+|C2cQJIh9s?*-m& zH@zr#OdVVP2-S~M{wGv_+SvMEqWW>lze@FU#@2tD>c=Vn2GyTFw*I%Mew^~}Q2pGo z^`EEuamrt$`ZLDXf0^pXDL+s3XO6A^L#iLA{Kr&(*4X-QQ2jXNm#O~jvGw1g`f<%{aLoIr1A9K2i=M%}_RfyY zSle#G8h2y=d;2$F?OWb|_3pC%OAEf+qVLyYhh113dx1|vy=2r+LAz9}iPQQoVcj=` zHR`|L!v@~Co{jzz>(6&m$_9Rg_2|2Y1EW7{#~R@a3H`suI(6jzz`!Whtv^jkUh*E+ zvmd;@z5nW0$_9oOCoQ@3N?>#dYqt;oH8A?q*8-y#zgIT!8rEK~zg9NzGpxye_G^Y~ z2g=y!8^0_Yc;lCjfs5aB4E#IRZ&zla{}>znkNUEK;fJU_8@-Hm-p{9$4}AEqZ1fjc z1HQSSeBkO=90PA*z4-Qq@_`TDPVfI|%GM>XePL7od*5Z?GXVYrqoapG^Pa%Kd;7Qc zkG#(YK4{+pnN;=<{gtbKsI+|GlB;~+*Uzxgk>Jh!?;d8Ozv^@hyu16B{&!Lw1Hb$S zHu}zvZT)X=aH1av`g9KbBEtbboCB8=9izh!5s!|6p*5I8zYrM3TsUxL7U1b@bO`In zp~WcwI^>W6_#XlPBXAjjzYcgY>*xZqn8$JS#s$M zsPh`uLx){-*2B+*ybOC2%;QOzze%qj?s%r3%CMI?HR;)sL)c4`2RYh*xZ~TFy*!;a z>GasO*3S?0Q~y*y&k^8j-KOODeO$f!8?ZeGY!7=g%%3Ac7wpds`%CDLxzc~&1+C+BZPZ=&+Qx~{(oZT=a*lN)B9Jwj!NXN(--@Qkx3 zIh;FklEb-YU*Pccks*xh&a(br@XO1MJ9Q|US;sYOBjDCg=I;nj0@=dUvHIC5&)EnCG9Ps9E<-85m3;fV> ziQ7%Mo6YtWvi)MThpqGWYpBjGMx7f#|9Yakn98^HUxuB|MA;0Ktwp{Dxi^1%e-5r! z;pffY(LWXWVt!rK??JwhU*FpAM!tYwSNA)S!)8!_XFu6b34TqBZtKrN?&sIH_h%q4 z=GVLW(~uYO>)rh+$VrY(i)v_`qf79+@V#067e7C<|MjCY`rmjoxBs0lPwzi*_;Vvq zBpzDw)r{Zof711tk-yv!Jo{JAe&+0lvz!Az|JuC%H~)5S|J(mKr~f~mo!$TLk8=8d z4Zj|NpN{Ty47~SOZ1nvNj)4!p3_pe64=qOANI_mo{#!OW^eFjn`M}Tuz=%gfM*&v= zo&p&A+o8_`-VN9V82jI$djQu1{w0HskpCPx_Q^w?C_f1JUjg3)_yB_~SDhXD9)q1C ze-`=Ss*#}}$~WA4cIa;jzI9}1H^J3shrU8^^~lf;f_I)pTmiguWT=AR+s+OhCiu3I zp_>4^ZXX#!d_Vc@?Pt-?z)9Dxk)cw+hmkKQ{O+?uJ5jy?`OVaB_sGx!ly9gxi*|r( zMuuh)yyxuDQiAu43{4?;?^%=s-a9gsLUev_WGE5&v%hzC$R+l`P3J{ifUo0RaT9!h zBj%tDhzI3}3uTDU&s$>Br}5&`c=2hx_%vQj7B8?q;_1)uFV;c)NB>m4dHXjUER$8e zB=An=aL^5uSHCA<`nM84FDf5E9>6aVzbuwXWX7Zejp4S&mQGd$Og>?qScy4PkvpBt zmF99daU;;3f@Qi)u5^Yo7zzS~xvjCSvpu{m)X{mc`L4TK8Vz`-fIC-| ztgQ={1{=D1IvVRa5lO5r)Y%wp=xhO^xo!XL&23$4gI%qm;J)^7YpBbhvoX{a+RTf} z_uRHASQ4yjj|3ZPN;?d0(ss&8-W z>#>ZmUB`=hrlPgd!_#M!p8jjUwdFk(r3oG670gb0% zIjQKeShedE>j=3LnK=7&8S?2ebo04!Oqqd{z$N^ zet%<|e^qdQcPQKt4mGy~yD--U>)Ts`b&XiP1{)eN|FzdQHUzQM4u%`My0Il_2-fuk zc{$UGuedhi-CzVHLtUVQj=EdhI_;f(x^((4x8tycT3XoJ3$A_X^xuE)r6!i0t&Oe4 z?fdq1Hg@@0FxYk{JVY%2cW&FXEhsQ>x=>4Vs54Fl<$fKZa949zb9)s zQ)|tMX51w}x^N0$PwRNO$6m}n%gfWol>fDp=wys3|1Qd%ERAKh36x*f$ajhIfGi)B z<*8uB#ZNS3fy{2;<;m7UKs6}O&eZs67gch=dLAwV0yHQtN4f7rhEXojtcuq+%Y72j zCFv*vbk;)Y0hGIN?qg3eqgB#gw#bkPGxL9Z#!F-HOXA{t8_)hME{fS z?^W5K-3F)_Cti5CoUwB_EZe?o%f>Bbz76f+4%m`Nba+PRXshwtM@1c75%~KC%(Ff^ zb!u)2k#qaH+S^+?i+qTljW{Fnttc&CS>)SAS@EiMth~FqrNP(P)7jP7>TB)p?D91< z?rUyq^xd{&$1OX9yEawt+_GbPa8u1~o3?M{_;Vbnm3x(^Oe4VYq75+6LAG#)9 z>D&$XAZ)xIVE3ckz1++2>=d^-cb1pIba&;l3Ytp%O_ z5_ZM{7qF8ScriO+ffupIEbv0sXMywC5evM4MJ@1r7O}v2tiuBPSd#^w$7(I`TvlU& z=ddaZJeyTo;8`qSfoC#5;4(wLGnmf;=dw%-Je@*{RzHVbaq93icEJKqW#=sL6!rpO zQ{UO_tOcLNPFmnhcEVCWgY{YP=`3o2(^!WEPGz+gIE7VN;AB<_*yJaP1pu4&=4ERw za3b?t>U)?EuqnR;mT7?zi!HE=MIG#l!>Q~YvBCm7Sd{5_?j%%yhnxGPqYMhb!Mo#D z&%wd{aQJfwQi1)T z(F~@yMwxqEhYO=&F=AqfIQ7zKIUJBMq`_gYgaZ<;lyH@VGbLOr;VKDtNH|x*Q3=;b zxKF}937?Q~t%T1=xIn^#5^j?4ISKnEd_lq;622ngwGxJP^YKI^4D07`RKoBM4j+*) zyobYm3dVFx@*?>pJBpY;OTZ)#g7N+b$J6*Zeu{u;yab=X+^OVEaMc_EdnLYao`5qY zJctDZ)z6jil?4JOc~N_|NWcXW&MXkHU&6jZ0n_-Yel41q{b#ca%LFX?&t|pD1uXgp z9uu(He>STs5irS%=$)w$@Dacs8qZb%_euB!Y>nD0d0mijrEKrJL*T0ghl_5pM;~5zH99i=ANC;YT=KWQ}Li1ezcwnK#sNeMe(}=KaE08?#xtvlaJOJ z^mEUJGb6cXjJVv%UY3sQs0-JtW%&WzrfbD-Kh*}GNR)pl^Q$udK;|n&H3k~le?B77ui{HfzHlqj5ODPY(;&k?G zPcjGD5oC4gtT%#+LUfG1oB=$PLTMp>Po%SBVK1Z%ISxLO!CqL-`XXuG0Q4BU5X&I? zeUWM2BMN>!gEW3MuV#WzTl%g(nIfFfQ&>ds26PAqn^V8X;9@r2y zZ~_nhmPfeyMSZ}JMl#(|K%D>A=dpnvHF`JrbpW63CPU-+mof;CXh(_gF5um6%x-#n zIE2K%g@<1(^jxgs0XPp&g5TZp7K@Lu{wFCdbzC zRW_I$UB}nhV4n@HwZR28xXA`nVAAP#*kB4&IzD29qc-@64epb017pwhU>is<*T-^} z3wtsW^i!r_vaicGu~iY4FRDM1NcyW-(j}_{|BJEqXQB5h>OXLcSs(V?8%b0K$NiV= zC7>7p{S{q_F|Kd=2e31efEb5s+VaC*j)fDF$(^~s!k=I3cJuuiwC`p4>F`gtFnI2t z^H@dqREUmJG==!3{-nWk`k%#kAr`{B*t>!r(I21QFT50EW3j`W-i&ZU0_TUq%Ig{4 z8DUSN0fYa1*sl+$7=!%4|HD18SOVq)#EcX+Y@;8zX;*VXPx8Na>C{R4TLGp8(@;jfwjwGQ!Y|O#4m%ZWO@g|D-jE^7j)1ps| zr!SJ~WrJpXd^p0me|Gq;q_FeLA#bnOkT>-GU4ws6MEi;!k`5&iFYNRyx>K#~e_zzb zzr6iX$VZkq@|F{6uN#zhkSTjcrU47Tk7)n4GAjJT7je$0va0sKF8 zu-J^p`_dr~KLRK4a=z$2RLWKe5+dFm+GK+(Eb!wIzh9pZAWzRWwC@N&p3jrKOZ-ud zeqUsxpA1^{*B9CBk181T&vr9Bo`4jEyzsCi!C1}-81bFpjW&3*23K@{(exMSuP>6| z*$zDK5ApeU#FIcXF84pk1Mxb6=06^<(f^YXkB2rnN?yk!i3!npvOXVArYEZCp?&Y6 zEaAUWzkQKxk3PQMLsM+{sUBmzy!|vA9w%~$Zc0AAho;-`xt^#6Kf}bsKS@4(g+THH z{l3Ud52O8oq2F0%eU6{)fu(5pIX3)U4>`Npex3zS{qcCj`%B>c^9h&3iReGm!{9Qa z{SJ;t_>lTc06yCjHSk9PdpuFCKcb&w)KBE?5pHDrBo5Cs;1mw?b2{oD@yEj7>%x9a z{sg~sjrM}RDZivK@nh_I0M=hl1duLfIQk+FE3#EUqCMi@ z&A7TE)-2G^lab_P-5#HeWMq(kxTQb$9!jzJWACAK6=0+yC<*=!&E$Iu1CA`V@w>>vFYL?DXu{+C?LdE-Uc?qfALE;1<8PWb z*Wi!FH`~jeQ~b>XJTC+8C)*!I`#E?}iH{ay%xeSK>D^rdBGsOCe3`WYk zFS5$(v&dt$pKsO-eXqpy;n3u{#>U@T>?2J6dJnB*t2BOS{BEOv7x=$E1K~iUcb&;E zk5@}G$R3ov^&Tp;;cxJ=m$d#OSdAP0hxNswWV`$;bB+G_`1AQj-5CEuZ-Hg}`Cb+< z$J-aVDTCsL$j0 zxriUc82m5tM`?3y*u!FreyIPcT6?1B(_qjmU}OLWy#go??2{s;TRaMXDEBCtP9s0t?j3yk&- z$m>Rnz47=(1dZ{m^V0T2v|kST*ZHFs`mon-x@JygclftaFa89(M9;^zj9 zAM~Hhs6*->FQrj^Q$C`8K(F5oa}A(=7V0;oV|`lX-NU=4)nj!GJo_%7H3%3lZ#Ay#(tg4Uz` z*ne5)XTBN`6uJB$UyJ+};rhqV7hns2fmgJF7U25{kRq(|G zA&)94kJm{azwqA9yC-`%I#wR`@kR9Uw8I|k61K3KGyltKL64Dm~0uZ zHXmZY3;kmQEA1cjXuMb<%K1;|f3^m5{nzOFAJoRXBhGknK2t6j=KNxRX(X;cvIcW~ z)q5z1Z59>e`0Vp*Mg~96G2&;MZ9Yw9KGpt3Ie*duLi!JlZz{1s`g}vqw+cpl*kO$u*$-d8;D!Q+>AWrJo6_FJey6V;#>z8$XL}{9HdpUthsrix?@`kSFHN_rQ4h4 zc$V4fFSqdXc%-8-HQE3x&D^+J_if0eEN8e9LXuebH@ zG55RFKh_67*6FRY(Ywyp|Ar|P-qiT}B3Q51>g{i|wZF;24~?In|3t~YT{yp3zd`4( zFB0&w|6iw9W}{bb;Rp2S_)hMp6DS_jahil_|DG|0)93Mz)<-x$`-x6JeGEOH(Vo`x z*xz-i`V_BcS;q5tWTxeQ0`;G7w5RnS{41*3A0_@g1qPn@TQJ33ZxDa*p9``+&TAfz z%w3?%hv@V9UCOU70)OdK^mlOjeEq}m)c?{c=KOsWFzo-LPJhu-JN=aw{$TGX6#eC# z{#pZ1@+qBS&X+p?!=7K)>5sMl8;tfDs1JKSqv#9y1T6ULr$p`lOhE9KseY;J2 z$l&~%_Qqk&J_O#U^=H+ez?=FK?M?X$yeV&iH{~brhP<#ow9c0}uU(?nD`Nh;59h1i z#}KdR{5j72W#4Z+9!bK21i@`8?suKT{7vgkTA$K-`-yb+Vwk*@67FRFcsb4wXM5;8 zU!5OTbU)>#^G;0gDeT=Hz`K)T_n7$J2+s6rLZHOgA9!{yzybz*2=@m#fc}7gJd(=! zNhN;x{5og04|o+Xda+#&KORX=9n|^p0#2wCE$pP4;E#ce<^9E)2lLACIJAdRO{GJdVt; z)pzp+kx9>LFW?X6X4318;6_z|#)rh^ndPPJnvxgkJ3Gp;hW>MoNErT1_}o4#?0(Dw zr;&YY`p+`;@3NlH;ylZ)U+ewAcI$c;_m8;0(|*L*U%%?z!uu2Z>&fjW&2pZI^{IAV zn8ELN8}@?p`Ntz^+&x>tQ2a{08>2Jifiq{?4$M&wNCC!STqTsXsao*UxV)@rBz<2Dgt4!(OsDoHf3^ zWa#!l@rCAFC12P}mNmZE^`C3Amzg$uvFm@X&0cbB_A>7iwwLK%Iu20d;qe3VL^AAU z<_i**_A>XJ6?VUBfpcv3GX3MnmvQXHr`yW~?dR+#RIjxHJ?w!{Y@|T zFF8Nqc@)cedtYSAY(H8kdD-JV?tgN7N}%U`5Wn%f3Wwbiw)=Z;#IDbT1l`_fezL+j z^mL)QUch{&&lkrdxv&>AJ{*tCOwi}E|K0(L36RtVl;C?T+j|*I6rhN=q;kj2N zEbYZTV!>PN!!p0XKP@oiXPs}xv4;ZP9$G$mdk8>Yx;<#;^<)pUzE=4E413Vy&GE>8 z0egTwA({3dp4c$px#E)}{5+cM$DM1b|H<2fU$=)5Zy7hgPcS}2ZT6tuA0vBM`=4wN z|K;L?&!S(?%mOPs*DqleAJ$s%pS(S+)$QTFPu?B|ZT4WfKRIPKoiF^4jt>EgemyfQ zt?=9`n?2N6@V}WoV1DG(CuFCgZ|3 z!AZH`N9<3NXg`_F@8|LT0oH5XXT3LaJjL@2r*;2N=M4$7Hv(_UC&jk^*Y!c?OV<6I zUEYwNzFv3O=Xb2HwDtN?&`Ywd*X`^3WE>w_)?X9u_w46W6Ura@Ao=qf6xw{8dql}w z%D+#-ZwUFjA5(Y^CzAJAwOOI%ONkT{Ba(o+dJx8&sS+bV%k@l3uSq6nL1JnTD3 zU++C0nZfJNFzRQ`u0nf7?|5XsW*-^QcP`?)8Gjy+Oy}*V%l4o@L#KZ{l0k2~Y5Psv zd=SC(q7w8$_Qlu{?4gkW!=5bX+lZI*_5SVq;l4-$^# ze!4HR%+q0^N79vv^P9kY>-(+VNS-Br^+k$35i~XV712}Cqxj)x zM`%d$JU7R7K0SZFb^pDhk<})l0G=1TD0eTexiJwOt zk1Vq2zZcJY>gSJ`KQheu*9H9*B~%bU)DVIRfSs|GkG+GX8wB!EbJoDIf9|JP)kh&+k37$}+y*$P&wV z`XVbleHQwwE%bX2t+BM{=gmIlZ#!VmOD*)2|0Ws!R^oX~qfhdK!jR~^6#j5%t*yUx zmj3MP3CL^tqSPEIFS;Ks&$CVcwD>>j`wOyH%K8Z;?-hma)rP$3{(|n$I3M+#0C!5> zeUbT|lM2TA4ed*Elh?}jiO~N{+|M%O1)is!U7+gI^I({d_4B9RLyN8B!}E-${n7lq z$YP(cSBpJz{c6uc;Jga=&&>W2zqI=$MTiMP3>qtW(DSc4z1~AhHU5cSzUQn~pY$#4Rh*C0`o-FQnWcR^ed_TA^3TdmU8L)C z{>YgK_dN#=lQkgOZp;pO8$xDKa1w#i3kzzl2G5OFFar28PfP6efbS| zIklhfOD;C_L-uyXz?1!Dc`g_*=|}emyZny=Z}B&*7p?l^{-!-|W0$wxzwG{TJfiD^ z><#`_W$J_MZ^Yo2;-S?)?EYxiPkeot{z3Y%`OgB~Um(wX-9PO5ApM#CkOKbad%Rd4 zDtjdTnCH7RzAOqa5`Qi7&)57XV~qUid0#U=W;nS%EYBMii0?7W_(uMcY4lI>EVSV5 z`m^eX><9X(HT7ecKiOxlM&GUv(w~^$M7)i!pL}V*as5S$XMDbaznK1G*AK~mt+Joz zxPL4#^+nJ7WO;lBzhm`5{!?J!$v<>|ApeNt58CrsW8;~9zL_XKNq^{=NZt!QevLo7 ze~~?|)$sB3q0cvVeZ=t(`}~rx`-@%vC-f6g^`9gC z{1)bu3#NR@KdMxJeEsz?=98#NZ>)dhdulZLWA*VH&nJjqrawUblguacJxv-vq>q5| z7p%`H{#oafg`N&e{VY$d0gv?`tNoL|=>9V{p7Ho~(UhM(9@*y;YkZRa(r31}%YUqY zEbv4O{`vXWgzK@7;U8MOqV=$KJ|F8J`JN*hKctV1R(*_#Ur|;6x8)zNoAM<8=u`cD zT>m&>(zEM_;?-js{ojIrAYOeG|5)HTsqsVlsI=0$-KYzV7nSY3Qq~-^)-g-gfm-J(vUyAkCpn3teE{HLQ1 z@6*I-Z+Ra0ReC>EJHJ)W2Oh!sY9`i?`SiZkJLdN@F3+=$2mWwhkK26ycs`zuk1TZZ z_ag@J{S3SxlzNCQq!Ty1-^0(BUJT=SnkO6+{CFOF3*VOz1$b^8?d|fh!uUSO0};_& zeh*?sk6c|HK{4;&<>L1XssGnASWaZ|!rMHu3HWh_m$1bI^6zhCc%9*;Hh7s0UT%X| z*x*tf#`i;9i+mvOa-#ni<`L}b1DsDV`d<~vcb$NE8GT;_a86_q=Pw^6bROjlFEr`l zc^=eXlu0}}iq{zNzV)g|@uDgXzbfKi zRBM4t7Il~~Lwme1AlnZ)2ri%#Wlm4Pg~Xo*7twjT1{ddAVL!cZq2Wt>3I=}_J^76A z>O4NKefCNFgucgoU!?nr#a^=`%(Mr{7vJkhMmP}m2>XD2;ssTL-{tm6c(Px7|B1uB zcdw1H7PKHCpuc>WUmq^HK0)uPgQ+og00s2Dr@C}DJ3*g_UIA6SP!lDWEKC3spFkvAv%was+9A4xR9z}S97bkLmG4MCK z!67B!j{;wiu*85fSY9~a_|C0pzr<5$!7q%7{!j?|dErHPfm8A4p z$b;}sZ+Nw5wF!gYb&KVABz}vTuFo^d9$+`& zHHm9Dp7)Rb-0R4HdE)#gCW!9|2t2o!GtwR|;y#LkL3;q@Cb_@Cf=Jt3- z+RuFW-#;s9iuX%;?_L>Wv;iR}siW^BB{+S!nL8kMkrnqoN!$f@1rLWjQF@PvJ%fnGPY90Pv`wT zY$ef8orVwby}p#$_;ZrIbARIYJ~_Xz53kS{_XoE(?DxWhD8b&AbN}J-pzb~XeyV*u zdEo@}JI#=X6Z5GtKETVfBJZja~TUZa>jrUm%eNnvfPol3P&cCTIzrPk;nfh91j)&{Z@3-n}9shp7 zIzwNcbtHce*Vl3eeR+y7qX~WC`*Z)F-M(~wi@|TvB>b+9kv*II78Om*Z&8tr-y)Ll zMEtI1+a7w4Dz2loe$ z%PwCx>z4c`!aiI{BEIqX4SRMc$?se7cun?4{@Nk^!y8`a&8PlEymRCso~scH`*Y#@ z;WBaiE5_+JB}IREiim$4M*k(rC8D~F*ZD5)4^+n6m!_0zc&r~88{t_}9?FOczB z#xGY)#%Bq8V=~@KdP|aIyyx+p$J0SM|6zX33%gvBAJyI~`{(fb@^vxtKdv9}hxsdC zpFj9~I;i4nI^t2m8Y3R@`Jpgb&Ii1Ik}vsdhm03|ejZfumHc;6vYcOeeQ$Vaik#m# zO#JHe3(@m(dOTjZ@87}-*0Dbm`Uby*Cwo!)r1pY73c0)&r^xwR()W60e-ghiNzVUn zia+bCg}zOBc)iAa3;!qi=<|0@B*~R|r}$PW#q;%b-Wc)A+Y?+a)&q*({d-tJ61!~S zKZWy;_=@@B{$_?3PO}_5UU2^1^Z*PJ+1GN&FM(d@k}%2BgZqg(oXFp=mG+7C5V$#~ z>?#DUD}(p-vs`>P=(J6d+38uBJukER2jb^-=k!2ZmYkz z)1*C1dWkV>JYSirtvBF*E|)PMaQdkk(%yM{(tkdCR?05{{Vnw9`p65fbm{Xy{4rhH zzp^j9Utsc|7fww~)nKxxB*4m^xPP9L{&bP#SDGRHN75@zEb=rK&^;d`wa8XkB z>(8e497g+`$m$~UA1))nJbqm;?KLN|aw*}tP7!bN!i6zCKCCad`>UVFGu{R7e`yx! zQ~3k=-%57B;a@qCQtrQ8=Dfa(zdz0EWBiZ@AD^Pf{po_VhdT1V6`9hWV4p_;mt+b5 zWZYi8;Z^*7dbBs}x0D5C|A}0lX+SD_%L}h4l=i{(N8{D~iQ>a*cG}<{{4Le(kL)F# z)8p$kH^CVk7V9lOQ<1N}l_uPEZ<&(D5uGRj?a+SRPZbH0ekXH7yhpJOQ6pOi^TQovaE+m-cxvcoX=qW zW2e8qPOLxpc(7l{iA+r(`{p!Z|MvA7*84NK?Th^@_({km|3Z@Xo=PtqAd!4`K!0g7 zr2ljKMSPqo<0G#Rf8zGcX-a#Z${sc30e#Knz9r*3))U-cc^7=VX))bD^TO#dJ$|e& z7w17rUuaK%>PaNg-e%vK>AJn}_{%Pd^RhbOA7l^8e&MgVa{eWIg*;rjMS?o%*KuDD zN%DQ;JPzZZA^nl-AMt;djQ>(T*)!yP%JI+_j7uhx5By~z`<5o3q?sbVaDPDm$us5r zuiCq;{z~4JCeHJCeD=;F`#|FT=Y^B-Ae&6+kL-uO|0Lt5Tg0oM$^MfNFLSxQoAMR$ zjZ={Ab9j3(UxL1T-Wr9#gWp-gUzI((Ec`;gvxL9$@$q<<%Tis+Ux+_C|5f_P3unpi z8=&yH48GpAj-RY^o*4gp+CL=4XuKx>Sy|HFIDh0XGXBbVKP!p+1Ao4O!|SWFVr09j zJ&z}u%3f%E?%5)Kl0PiB&M$oY9()j(65*--jM>tk0h9ksog?E9;VGW-=e2ngUO!=u zuxHhN8a|lC6Q`FG$>ILR%TV9GzsQNq%%bt|Z~k(;+dV_Hm-=z>=ao?edpe5tWA-Z^ zJwB3s%w(m8e!Ssnb45HA`hz^}`_Jk*0nfBqf?N2-w{`q{EDg71xj`%f4#xLRz>$TL`GQM$sz2P}?Wjy2XeLb+o zOzB^Ee}Lu>YkW$ute2n<4<3*giR;rfPs|TopYZ3@dE|dcT%YT!#q){W-e|p2p2qi& z%Kji<%lxyxZYtkTay;TWk7t?E{x8ybu66xJ{pssT_@7V4H|ihf)k%3WK1utUmPYo& zZ2xenIkl8Lv1!Z#ZRvj0ci_PCC&y?N{voxkR|V3jZ}> zW`s-fr@U}7zyD(5Q(|&`!0nstZ5sX_n20AyJl>H$I_ln~^HaXQr|((o_U7ZTTJNX% z%=vFQ#*?0B&X45JS@ZSz%Nx#IpvPPCmmKzr(SKgp>(%WA^NEk*8xrKT1LH|b(&snm zD^L2Xw8ymh+?VD02>dMIu}Z?^4|M;lqYn3vK)Crkci{7XrZ<5Tg_YY14# zKS}rlhzNNn>-^`1Q&Oz_dh=ww0>4Kgp9H-BKq)Hucl%_#l(2_?Z;az%?~o5(0Fa6I zmym3o&#mhp_*aU)o?1U&d>%^q!}@aDc~XhC{s;a0m(b$@bKJ^4)>mumCGx*IabBbR z@BVeHz{NI-`a?;4zV@;MCJcS!sYnxszj)YyfL~9Nf1jWn1H?TM^bdWB`|b4oO1j@} zOwRKDNyNI&Vfiq4P^#|HXOq_ii}cQ7!H-uIO@^f6oKw z%`<=}_Idl4@qSyy-gPVx5v2>;>|)J|A*4{_`FI><^9a`{>Vm+yj13uqWyJ(6PgupSbtq=>u5C`*`;D z=O%AoCELGlf8Hd<|Nh9y$@-@cApOYtL z-DLf%_fH$_@+1DgIC*>DMEz&7vy=5-Fj4#aCU5VbsQt*~?JeJr(Cv@L-@?8*9{=!v z%lq1KzbBwSPeSyDJ~};R9}V<=uud=Ls(`-_epLPpdx=Lc`qAkr`^g_e&r#9y()je$ z^I}n{Kg3V#^J4cvUb*Z!ZG4}>`?n&#`5FDaG@5THyzxl~`l#yZaQ5-~&+>RPK9#_y68N7a0Xl2J zPJHmkIOG_*$AP1&zJGhaq4SU4o%dnv>dL{`!C36=SMcA%-~QFTvCqdc@cWM+#9oO# zbI(2Zyz~l>X)4+x6nXy^*$ozxYY>PX~VHdB3n?P2baVf3<@R zev7Z|pScIj{5%-L-%YwF_9Oga&%}NW&SJlQ=KW`Ije@IJ@khE2{+f!O0mAc4?52Mm zee*3!Ke_wu4;_vack&2d*_H7Ho(X@OIk1IyqMON5BCjgF=bl_(>7qD_EnSM zY1uzf`6Gi+F<5vq3#jvN!r$s}Bst+((o|8aJhj|I-r=Zdksc?Ev8>*mOh4!L`$|Sw zLoMx-z8xEkv48_KMSD6pjEj#7JRRhfRot0O^vgD~8%(ctIOv?ru_YCK22e{RM+Jkw z{Ivp^ohNW#L4IFG!{f9)eXHrSb9!`on|$A-7T&O|GIvi^E)^?Sju76-E(BfTm1IOv?E+YnTl z%mxj6AsbsTbJ2TvYvoohz<`tWI(t)kjk~B;yeoi@j<}rJI^IIxYNK8|eFsd9nAY3t z{d)o)*6W#weiHeBI!4d5({~5ljFdDU{lq{b>rI@1ew64BD*8MI>ii`H5?C*kJRW^- zz{`5Q6VL~B@UQMk^BAeqPYa~6-n8-Qrw8yysM05*Z;25UL(67ePPXVc3F7Nb8o!TZ zu-BVBK_6Bd^4oPV*%&ecI9SV=h`tjt8dO1n$6;MYLI&gWpBczxy_q)p7ZFFx{*p`_ zwr`P9fcUrAuyuUQK0c@N5dZ{MYxr5?_g^`tf3Rr8WZXKj_iyOW@M5cv8a|}q zC+NRwlK#i{F^#^iAH&Bq`~>~ijP2i|pK*N@>elR!#nC2W+pT78LZBkY_2O0 z^L3>bOR*O0HP)15G?YyIVGAA>Bk&eGvf3G64=k9?V=cc&D`JVpCFtvT8OzL3+VFCm zWQ2u-_njjWXtSw7Tb(a^UdNg0d-mTXKZxTKx%~``I z=y(-%iAH&$$K$APfbYwvHqo}cer1XM=cq5j8uAaEH0K0dgE8j>JW@R7>rgvg>QDX9 zuW>zDe{R}I+SlycJgai#oQ>yg%4pL~yFy*}!u6`t}`-Ro_h?}2A0OyK#6I$qJcRIkk zel(env4A+XH4$y{-`^H(^2|?;_Sh9nT}N2gH;L#F z9|`gJNOdL?9q{2yi-QlO_7a}+5f2|tb<#MA4`;g3p7Rk;ZWi7GZr+yjmSnUwd5?Oz z3^{Mfaqxp)EGTtqo2FA|Je^M_H%+IS&Ofp*#O+(xX^#24c0bnMzx%)Ht@1#4!pvbsR` z2;!b?9l>f~>}sVNd!l{V6RBrMF=z2Lt?{Ha?zgG?b@wLG{kXDqQb9CmXn@7aO<3(jxP?E6E;1-)EOYvX7nL&{Z62ZK7HfNbWRm> zf+N#0LeqnqAw(Mu?^Sqn+?MuC;YCZ?p7T=;z?Ej_huCwjO~?HdA3xA>rr})I>B8}* znZ$S|Xm8Lz?3j#xFK*R! z$L#@`z7?0g$^Xyd(>LM>-hGTC|EPg46?mKd3>x^o0&n9#O1Im2x;HM~93Nf+iX*>( z(f*{s+vR8AUlI6Bn|-S8@nQ+5pKHYr8u*O@ZyTR!-voI=L4nUjX1~vC-3wb|et=^B1gCA4li;f+!PiWJr?ZEN`loY%iSV?Sp9oL;@rm$s zjx`a!U=qB45{L(R~hKGRYx$I!SN-E?!$?%`lkV?=q+Tn>8!&pqu$O!JEJSaUHUv1&z%OSPEO0S<)dDYL=PYm$ zd%*%PWrG&Dke#)_OV}9;T!42)_5K&L6Bc+8d&~kaWPKJmpB=Hl3+O#9Z9MZCUO>>` zJl0`>eXPj>&ttU~crL54z;jrY1)j|+EinE%p#`4F{D8|0`OaWI3!KX`Eim3qw!k^; zic@btja{(7Q`tESJcYdg*wlA6J8Qvbv6B`!lbx{C&tQEPd^(F-;52+&L+2-z)mq>b z`eP;9c#~NrV3VID765G8o0mTSsKJS}r_kzqm=CZizXbZ@ZW@ev(gM3!)WNPeoXY-b zg4JP~k#!jEB;Xxx?vsu(C;)-NfroEgvtrD7i<7y6E{A(XDm(EPj9tJl^9lIZlG!}g zP#lSu+y%2)2OJwux3CUuZ!X|>=F5Oj;P(|Uy*0|*>pEP0V35C1gQ&|KUUo*pGz$}a zPQn!R3BDrXN(sYyczcTagom|pxJtrR5~iq6_znrzNVreJH0u%mjD%|?d``kN>+$|2 z+$3T8m=y6(vmWnX!W|N>k}%DBynhKtB-|%qnss>p5 z@v=6-BoBi9csZM38b85NOa=tgcnPj86fl#pZ<&C-5+3vmI8(v_Ols6VSK;yTGlG2* z&b(g01rn}ZFJQlf&%mCjK8>IAw@JXF|7>=mLcpT`Z06f4V9`JDw+LADpUpCN2skR~ z;Zu^F{t>_)j4y!qwWz;733q%&zyVqRf`lt2T=k^DS4ntK!b)ELzZ7`EZxK5oVZm<^ z%lxXq3w|MQ2`hPBk+4tFul*}gzd*w0BlGhmtM@^eD23F%{7c}6YRl)PS!a^H!ud{&`+nO5$T^}EKQ zqmlktm!w{ZU`z_gl6uTzdXnZ~6wxCg@!Ee^w!2;0vua>(Pv7!@YQ`k>QIICtfa=hx zo?=Hyt_A|N(PIaFoch ziFbSZEri69L1gB?eE(Yv*O!8q`yp^MDz;+@-7rPmR)@OmP#nQ34){k zku0bqXGJ0H{`F)78U5{#!*Tk5bwIRI_D}s&vWbs1%Juo&o_eZ=czpyt`aIG{Dy&W%7{;Vp>T=|#ERlEIVJ9ky=uP+m6 z!?yC0mhSq}@|HcNJ8!?EtmU@Q?uPC?HPtKXTX)}nVAZYdp|TCzHtZ^Is&C!gwYR3a zrM_e5Z7o|4mN%@bZZF$%a7WDw4R@fcqpo!)+Ew3DwrBV4w|DHUXxV)8?fdrL+I~k_ z)4shGyIS|u?CNaT%G*}dl~u-c0?i*weay(;a2oOY2KQw};AF zjebLAE7~`f?7uZszFpSYyt1WY|JJ6C>YXi>CAZ&Pv44B0Y+d`Niv71W?B3d5zUAP? z#>(xjlhi41+*{qgVNVEStX|i$p{af2?)_Dv^78ggrMsF#WjnW(mp1KdC|TFDw{piM z{Ot9IcCTouZ=0-MM|Z>S75f@0)`e-@arxVFu!iJRSF!o7`kqzIduq0~)Ra`;RkuNx z+pev3ZMWXFjr3S4dELFar>3+aKFx~#2k$83ZA!LpHQQ{cK%0`ioqKkV+isuNv%9S{?wZoQckU_K+#J^iD)#qCxo_rmOByHPuYt=4 zavx9s$}dFU6YE^{U0L6{?%>{C>+YzZY}{>I@2s!bHK~5eOGUjsZ969!&)%Lo#Iu^x zib?9V$a>{>)|K>3Qny3a-G0ZO-I11Y^K%9;nC++pdi%ZseopmMS#4n>>6ZIQu*uA-% z;(t6}Fvi=4@s^g~IiBqp^-8H8*~RWj#%ig~35 zb4Cf~ij7qiugP)~qXCx7xpYRmrM# zYfBp%f}&o1^S)4hW3axvt0mOg8Eh>Lwnl>c8(Z%T-qzUG*&g2C+SuCOyn5vd9ku`y zA}uv-Kxs*E<+^nx!PP5Pt_-dWt_|K6?rs$SY!0<_HU?4OR^kte|5l)Osa%)dJ^w&s zxUI1z-WViC3}suWqw`?%U3axK8t_g5cdjT|TNf-1HgxrLG}d!LNP*Oaz(+%83lPn1 z`)_Y<>slM^Y7GVVwTD|nT?UHNZEhTQRpT!4T*vHG{H;r1)IN$T<@1{2pCV*9$~V$uw1J^?Tk4#^ z7o1;o|B>g9y^p4TDgDm{Wvbu6D~Nv{o}@nQOTVbq&!vMfN*$?lY7P3ykEK4I{wJA# zn*C=}zm)Umxe1js6Sjzh1>)^rcmwgzH`T`MGwBx`Lyc3e9pXr=s82g`YkCM^wk97nuC=mU`H`}V|ATR9LvQYU?`Z)EQ zA?SE?rjJv8<2dCz$0>gm<;2eg^iN6o=bUl)8Nz`G)u;0i_)W%0^C z1RbK^Cy~m3ahAf%tLU%>NwqD(HPx?`R5Ug(z+)z`p(aLrDsnVE;j$S`gmXkd7=#4+h!_!h* zd6hJJyZk>R%kA>7mF3hg&Fv~xN+%&dQfJeIYmwzdUy6h`>5%1if9#XxL|^yEL9JZg zO5=3i(aL40;pM3kAUZlfH%MXG%hg!is86n-`lfk`k}3~yh_NP_tMUude;%`yt9bva ztz5-x_AQO!u3%?( zeSKqRCo8VQKOc!?TX(Bqwz;vBO}=rGn~LVPeeKs!`_9I2XLEa-6hm`e7&577>=L42 z#cl0fjl~t)Z(r8g6{$zqhZ=%i?ZF1lm>3Y) z6MPC#j71+I_D*I6W3RNSt-igX@wVoc#)gm11Iur?hM|YT`#Xc3*z~dD&aQA*s1A8g zE6>C24cOJOV*al%VxhfTFhbh!!Gw90;`+|66|1kA!~g#K?rXP)|AvPO!n-?zBL8oC zs30Q$Z}#s$VBV2SE(;<5hXVWqxz15YIUOjF(K!Q^>1zc-2IwE8CVg=MSj~$ZiGLR3 zM@zB*etPnb3W)#la$M3fou){&22g9R%P6G&uoidx z9kM;`Q}tx$|6btZ=`V0qw7Mc2(=cc$s@g04J%jPvWlv2ie?|Y|KD-nRu|aPE+S~tM D=0Tuz literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3bc9b617ff7ca80a27e4613a56b743e3.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3bc9b617ff7ca80a27e4613a56b743e3.o new file mode 100644 index 0000000000000000000000000000000000000000..d8a4e44fff9e8ebaa807414d83843a1f2766c830 GIT binary patch literal 28192 zcmeHv3vg8DmFC~KTcY(q5*RCbA+?0C1(rlhLIT+sx89&9mJkAMW}BAOLb9IQYUGwl z;x_Dfjg2wdcz3cj;|#O$X45=eo~_i(dMk;^#_QOGWMitb>)G1cVKBo0Ha70@WN>2K z`*-`=lrj8{+GJ%lTGc-n$5;kvavsB2Bo-+xyFoV4}bKq zR7kp5xch|N`#%a?_FvHj-9v?p&7ZXw9C;7dIeWqCzsCJMuD`)ERF^4Px?kJ#gUH{f zuup&8r$s*LvPZ7j+5F$x*^$FDY#y>NhL^DF?-zJp`*j0*=PxxjUzi&1K4Z^3a@M|# zY`qxHVAcOq^SpKedYaAtju(0Ff91gGOUIr+{eH*6)2EIfIQ=J@=TpcY!ZrW)#4FJw z$Bsu+4!#mSaQSHTV#hC{mlppt+E)Ja=z&*GMUxJG7+rn-{peWp`RJ*M52MK|*F{cV ze*Sc7Sxsc@{Bx&UUpXE<3jIrCKaaK@I~6_B{$cdF58jWatg4P2Ie75&81j3z>{ztz z7ssQIP5vT!^uW)fmyTYJzCZQ>WY$Jbo`3%IQIzfEecz2faNv0K*{9!(K5+RI&+}(x z=OOb@;OW*=$D&719gnVl^Udhkhp=<(a?|k&RH_1+Iq^xW9{&I~ z&qtHWHbgEi{#U0{R;`b`jq>b=jklqnyz=45e%QZ+_FHxSXfzpZwI6NM+IcGaHrnK5 zDRftDjJ$pR!0E{GW6?<2sp$T)%h4oT!PSeWp6`ZEAnB=s|D2QM>1v4ly00F7<@xm5o;SW{_Rodq|&BKr96IW`HB*ep|eeAVMKVkFlrD&0}cH4Z?V)khg#Yz&pR%ko% z@Y~RPe{_EHh=K}N91Ayn@5a1d|{By zU${RpLN+c`Va!CVU3iGm82BQ}Hh*o;kM)@NvX$4#JfE&YdnDP|YZTX?cfB4V-rr)* zxaHh~cAz#GNyHucpLZGUpynjAJ$AeD#hwV|Px5H%*LWVu3+HhyJ^$+GT{i9`PUxj= zXow`2)~GOJ>)| zH$H+tbOHYe{_+w0B^^HSSMZ^9_}Rr3_Q-|rz*qLM`Lt#&QUSU2H1_F-@3DEv&cFYF z7P(wyi(I%rAwoVx{`4_?^TS6R;3Y=Lm&oVHm-tvt`D)>Zd3}pSeaXK*@2ZPjTU39Q z`se?*`l4OnTH*@h{zd66mwW$}cfD{v|A#$`-+%=8a7u#)`oiRrW6;mX88gt<3NwF@hQ-H>P#OtNR z&d>~uw6jUvwK10HVe44}N|ZyAkord6VtvRD-ZfZ!E#YG@8Is>xNm00J5lBJPYn2nypvOb39kf$ z{>hoCz;rM$T2;Ag!ap&!zQU{X28JtYi0K<2-8ETJGgaw!yKC0gRIKIKO$15gCh+=O zn((?&Pf@Qspc*qHzJPz|&Z$ViKjRO~qDJVYfPegs@HW)LcSlrY$Um@qmv=beoA8U? zu`+z+aU6u%pl^J9SHL$t!h>!)Fcq8%&P_v|u&K$Jpl>qhz60EM&^JBd4-DQB-Hfi` zb+Ps(XUd_Icc7VZ2lSz;PQ1@@?i~}ts&9FPZq&hYQZbLkN^eH2BgDev7N2*QAvam7 z++BwJPA&_tcl~0ocXTp{rMz$4>-K^g+&$#;&P-H=yhEe2-oPXly~DvLyu($M-kI^K z$FWGpWOsM%@;kQVj~nZ#cdU%tU%PzyH$Q4*@4*hNd}?@j#vgPuuXhseEZ5&X9Zelx zfxzprOY_a}DPPA)6N{cZY9V)v`D_(%J9gRDRLl8^RyO-R&R={T|EC(AEi7hdvIXR# zLj1xKeQUBRiT@tIN=1{uF^#_y`A zDqmab+D5nX>RML6dvturH8VF8^iQ}ZcFzP|L;m5>Nxy4rS66$FcUx0;&*rX9Z`1az zO`VNRjjXK8Rpl!41qMf4GZWql_d1^oSyNf*dQW7q+IfW37Nk;hl46pSS@H)B_pFo&cQSPAGtopvx+9BpS=akNH0 zW|Fs2%}g}U1hgFoY4+S0)7ov!?zP(-t4es^lAqivIZ8^{p%lhm#dQXk3q_yCbvTW& z6S&UcVmPpM;nG@B$2P81`HbF&Za@%8Dsx~(@s{N#>(grRwdXkH`BjJhd%PV1Od`7 zN@puWT0hy>*uh2Y&8>@+XAg zpOj+ganPBeQjV7rkK;M?WWrq>cM2wpPn~3J+ zw{w~gwCB1|AI52dMR_)IdZSK1!s$nJx{uR+I_=@KhiGoUnbXZW-ooh?oo?iGqfR$) zxCQfhC@y(pxtkdf`y`E@Z{#s7c zx8+gJN;J3M%jsSnuj6!`PCvxyha{aHN{AKU z*|7}%jCe>5FbBoM^uWhaCVd`4zy z5tpYiQ0OOcnvXh&x0e0y5`q-Avk%e{&!rjM_HxAY_tHRnKqCjDFA(hjjX8zr5|U2^ zjS3PyLbMaK%S2~^E)nzq#iy%`#sf&8y~)1|%PK(^63y$k!9%!+#}^!#p@%($8}V3H zs*fjJzOsR68DEgEY9M(YUvE%;UG;<;@mW(xbdAUl^6TnI-iYVEMTE=vNyrR&77^~z z%eOI+a2|i${v(Nm8}VmPAY8<6I_$Fq!g=E5ZE_IK0@by0ZzF;|a>wlTElWo@~h?T#iS;AI&11k4KnKR|JyEXfBLViz5E^ z?kSJ4b70W?dS4phrPP;6gkwBd5zt(&c)E~cMbq)59poDZt#y)b82U*o$hSC-{E9>U z{XAT0JOZ5^s*X|L=VjV>c~*wXmusNQY5vA|g7OrRFB+c&9+V$q;A2gQmZpJb<4HgFnU&A7Ia#sWiLdxXXUlYAeIB_`VAiI;Dt zSTXdIS}0Zwd99IR&A_!q6l(^a)IhOnq7&gcX88vx)(!on1nO@?UTdXTG2v|#D<-_1 zV$FniP^_8oO%$spd^5$W313gKZo+FR)=l^piWL+7D8-5i@1|HY;XM>`Nm|yolnWV7-+Oc4hQk=#L1NoZhVF;ZpAUvNHcuaUfzK)+Q!1|k_ zMW&K8$h!+zL0HI-(0u8HeO!^gWR)7NxDi7A{ zSWjR9kM+csD|UwbR)+HWDaU*spKD;R!d(HHr?W4xr{`E+7UCgsIW|`pY_PYs0`_1G ze*j$GLq(dq_afFGFviXZEq`wS+lwAwSfondHTM#p}~ZG{u~3 zp9Z3NpNjfCNHm{6b^BR*eX!o6Z=$ei5A`3n-%9dgKH~P+*By?e4^>;g#gx1xOZFZbt@JBj+kU-v&* z0DqV9FjS8AF91WPlWTyN!@p%b40Qr8FDQ@0*Ocr2KSc4grhH8t?p~w&H^<#;^myR* zbIJd8f7L?E+0i4bYw7%e>Q@oWL48j&@%o}YRs?gfJPtpSBm1u~pt*BZes8T|+8jb1 z@|*?#UlGjBA$w4g^J|Vq^1KmYziH0KzfjFItcTCTekczc(i8uUHxlrbm_O+`OP?Qe za%sm7Azr?g{Rn-YXg!VmvER+W^vBDO`M+^4$XyLK?d(#zrUi1SfI6Nv$Cbqg@~cC6 zc%t%LqW!-k#;~4%5htA&w9RHE^RhFS4gR7Phmx_pu!;RQ{IhK~4;y|;9LN0ca>e8M zN%44Lp^m>n`p)!toTgK={qnSUoTgVZ?oN%zY55>=Zh!6Kc$}6GhJJB~mJ0@5_zf+G z$3C?H#>2MRoRoO|6bx;KeA{g4Jvxs3ibJ%#kT}}cIird8Uxp?^dlrXiIyCf)Lo^+l zX_{`#G)*^Vnx<<>^ZaOeV8&^AV5Vt$Hq*2mFzD7mvPPlGM?8$z#UYwbB@X+ofi%sf z<5WMIj^p(6gg%7M5uUdq9#73ju!+R+dYxe`dzxw0tq}*1!gh z*3(A$XnAF}pLK7%J}s{d{kGZ6W$`#IuM8aRUmo9nw7fLzw+3o7T92#zTLVs~XfLqP zo^7+s?l;?Sn=QXDp1yBoJY89=)4ctQ-J*WNUNQFbdj7EA8gQ=_`77LmvDzqq+icA$ z-5%tNL#wcSGs@pK+gTEim(=MvFaO#H;_)U-<%a#%K&ys;M^gR!5Tx_sw{TR$|~NzXQtw!+vX^QEQCDX(eISuh-zvNXYhVx?^LY0t-Z1 zAK?SzS8HI4hJZm*xO-(WCk zkWBc48to@!dja=o9#96oQR8PLq|fcg`8VVpe0?J2DLyMj`@;nKt2I!vRt`3b;_&87(cTKr;_EM?eCwBq z{!#d{DvDPFFE1DUsqo5sDE{T0R8))ReG?bf7;|u;@!_Q;1 zK(?_S#Q8*ZC@+`Jiy5sqvEIdgBJUdHryH*;$zhwI9~!*RQ|h>&yQir}h)juMXwrKBf8xH1+QmjPSg_;h%>> z{5a`1u`{uBu|A>gMmUx&&)+gaxjCOeKle;*5rh8~eq(tI{Sn5`Bb-D#zb@M=15Sza zi50;DxO4asv45fSx$L|GWJf+O&I|LjY&`P%@$;>G?*F_Ng3hM>oFTt_Inf5LWs_~< zk-ue!@(h25{?bq$Z(r`Oz^elVd1AjM{IfcgQ-JCqd);4iK8LC4ouWTl_J_$blJGb9 za{=z0r_#9`;SId3G?#3OaI>(iIZOu$Px8B0XFZyyaE@O z;PK~x{}fQ$O?OAhA5HWp^#2^xKL_#fq=;AE9yEP~#s6nuPq!b2k+bP}=pT`vhtp`U z<=j5A@%&hJAdl}KV~N1g-zfo>L;c5oiFk#*MS+~5sl=a{#_c%&L%qVD#}CkL3#OnNaGRj-|ReI z9@(Eu0~!U8Uf-5E8civQ@n&hr)c(AEhz>u=|9^$@x6FyrM#&e4csw9Zk>AbYPwszP z=3G2Cu78Qjj{paey-Ajz=Wot$nIVtyFixJAhh`H>qWn(8FZSj!Mrzm(#+4WO6$Yt3 zbe@1#Td=%j3rUu@BFKk#aXy3g*fRH7*8ipeiTwoQZvpIo#_0PBvDiL4J7F1r?m${J z&fU+uEpWF5?zX_)7P#92U#tZdLmV^jiO=z|>B?_t;(YzL^ohj&BklCr|M}rZ*M2mG zF9m+*$Sbj^}W%p+FRR31HPMb50fsUe$z;=J)hU6PEhQE5R3>{Ob{HmlAE)aw0?BpR8IJnJ>^_sN6ss*- z#gWXH^)mGZqoz02b&<8MS*WYFKCSX9+qndfOk0E#guOQYe{-Vg>jXE&N#|zz=**kq zXzZKfDBw+T7y7xSKU4TFo~b;#@7WVQ7T<$7d(>u2(%>mv7Bp#)H`e3HcAaOr!L3Uj zPQK-po|ar^zLXB}=-G^^XRC}=PT}fq)l>57uJC{2ySazPxt-Z=`s;?ZQr>!-en4;>G;}$3;zRz# z;R>GxI7spn3ncb{`^1bl81tP!QTd4_r3pq7kl=lqL=^lrDYTB z`>X%<@3-6k!~La&?zi)Q`a*i@=JDh;$m2`)+W=1{&LeQWipzz=(ZjgT;Ls=ohfp3| z)3^@fqJPBDZv%FsXK(b|0BUIJNV+fdz5hIZa7E5+WAvR3kEAz{0up*ZuES0VHom_i zqFuwTjn)tL7XG;98X$e#1QljC`dyP!5HXBC!;s-ROfzPD(tp<^*-AcM>Wfn}qF^|>E_ozp5Tlei=_9IS zrr^Gi{!gXef1-0XQ?lyc0}RWD>r$A`hml-Tfn;bjR>e^*v>gp$wMlx1X_P{GSKRbZ zt1S5o`7KghF+#G-jtNZvw6=RK5bpPuViZdkaY;>@38?H=ef)Eok{aX$h0jK*@0J|Z z##a4q;7>20AC&sVx<&C|)h`qcGVQU*eM{vp8;tU&y?|B!889GnUHRXljzx0uPs2x) zzUs#pV3>FwDW`51q`sn6{z^~v|16YmroTxVAbpav>i-^iR7H!MvamYFECh%|`qa;1 zT;_zY{BfHNW+pwamki2OCTc%k2*tqtPvLb`CbA!wq`oJPX>l@OFT*~SzuqFJ^8dcc z@Iw7J4acPE0_quN6DJ-L)Q{v6hR&p^{&7N}A3?)W{z!4qH|hlQjJWBaQdXz8LVw=^ z`R|k!a!W~ZCVpM>F~8LJSn@xl^ks&mPa$pU4sLlZD)sCF`Ja#->f%z2S@TzZ)C&E* z(DzvKKO^hT$u8;C;5kau%_R8|!i zQtG;!hYE#c*5fP{&jQvzReY(t(%V(nv#sjUraF-tI_fLOcMn$8kMF4J>3ytje5C{CK0UultGdmR)s^)m0Do zwlvS}=&KkXYS}i|1AG0gkZGE!AL;Mz8Q<*d?Vawf9d8)zXlSk->+kCy8R`qUg>L_F ze_zGpLz^a$S5HSnMeWGIMD^@YOS9Y8SGybeY~4|{%{|c4{KVi~^cv449#^}#~rpL(or zsAZs~c4VlrW6q@0GFa6;*WcG1Tu^WG-GdX`Mh5!P$7u7R)}CfjAH2=<49FgdGtVth2<9jh7`tR1MSST{VpZqT>h?W(*BJtGq&PaCF!==)b`$@wW1J;i*?*m?-Oe?@W+*)18T7dbi!eTw%;x;dj?~*Lz9d zyEU-e?``PCYXJSe33o^htvnX@6n&}RWKr~>YtPrej+#`ISMd6%!QA#lH}=o@w!G8= z`qd75ZB1S7`KI=wMi6c zuNO`7`nq^i+M$$hYcKIJ5;)2I;5Ov2{p3HxQlgFeLt4f+wQuuRC^+QYK@i#HsnNiT zO?Zg~C;3JbensK!X@^rNgO|ieO>*P6A*a->_m`pTU?0n`48)`M_$>toMLkqw8}d{H z8$VYTyl{uO`y9mQGZ$y5`Gtsu-mkHhn}*T?X`y|wke7=*i;mrtuY!cUQJ#1{tq|gq zkWY$df||m$)YLwzUNp@TU-`diFUI}^^-?^uu@dk}-XodH@1tnYh2%SK!2d1u7m}y_ z(L(%nH{kyOd@9eYydM}-aYg^nSjf&FW8uc@E4x6&70E6ne^S=>ghhT7@?>XPc8$`l zl>D&dH-e^AB<=iC@>M(knJhnF>_JY&)xVSFbjvQj!mZ#dr_zo~cA(<-fxkom;+wak-;;dGS3ms)PxTw+YvvcrdXjxrHZno~@1kV2vtIJ^Ep|L6 zJLXfSohCl^ae4}w_-6S7H{ic(;+yMB-=3yqEvHlRt@RDd4zkvl&IuN>^P=Qi>-$G0 zJLdBLz{EG#_gxcT?X4)kD>vY8m+?dOk|Ds$w^Q;Tu<-xLByaBDmnGj?{$r9)?Lc3c zQpqVZQ~4@i#Es<$Wyx3dRs5?ad2@X~lYDD^9kMdk`W}*em9Jao`=aEl zd=>wKN#30A2a<2i_Y=vt=1c1uO0;&NkWy`@wvUSMAt9s@ant41yzwmyKSTP{2@7A% z0~aiOHQuZJyUJIM_a!JFCB?@O%hM(cUyb+E7QPzqnOJtucn3x&eStZ~-zeDb+dVlr z(#BsI-W0-bF!(zJ#e0KqdndwQO*lF^iq~`x%*=c-waMKRqClhm8FuSUkK8)&*Mr|$ zde$G9!K<-kRYwN`sCSD$D5{Hh$kBVqTRMBo@IK(dF(K8A_UQ2k>wGj6#?b|vUM}6b|!C=SK(C%?$ zf1A5O-Us7Tz9DaL$~(jhCMtyDiE;{3)WuxG=Ba=Z_@a5S;-TBwJ%V%CKzF)V!qx z@5cizr96ru@6$WQ!?)x*i~b)-^PZd@_4A591C#PMudzvUi_F(Ugej@~)%}}NpKP1c zA*rwS%L-HZD|!eI#Iv?xMxT(d%3J9_l=+m@&Jt&%_xr@nqdHWSpJMIGWAb$QvKxIGDkjY}O}}b3$Z>kpO#e$Lm01 z-246Y=)U)=L4ZB6%~H^PZ~gyYRsXB%|Es!n`^y!pSL&L^RMFVWEM8Mk#Wt$gu;*BclsXMEVz=j6AXSROI3lry`?Iyc>CZ;X9H0+RjCeoqQ*fHv9hY z=&l1pk3ae3Q2N4U;d!m!iJV_>Eb`d9KaD)@d?V6w{9>ek^}C=e4u=mO9NKyOz|dn0 zk436a9gnoMpNjM!cq1};-^IxJT^B*KG+gC8ICOsDBSVj^ML$n`Co-=MgXrgYWPaPXBcs4qpF0_ucj6p)r+)q@Jo?1Jp~n|KITSv5JaQhd zmUrKX%s+7=QssO%a(>}C(5wj0YkOpMMZya&Mjl^FJTDK2 zj~^JCf8vp$FnB({@Y|8c7QO+#pa;m_k5}rvFNCY0%k%ev26{_N{&pn%&Kr@m?I$Ah z&Rqas7a|j8KM)=z{7Ue8AK)v`q1~UL-4jDqfIXIcA~Jd}U|R0wpCo^^ZuG*7!PI@Z z-%m~MtxJ6+H$-;y*jTXxHq!LoIv;#$isrd;#x|v&aNBfN9J4=g!v0g>U&05rX{`7) z_<*JnZc6{K?s$;e9c8Te6~OX%8`0Mf;V19cl!ZUoR0_ZIT)DdE)fbHZo5&yEw`ud0 zPm=#B-y4MQy$s*H<9nZP-|M}O?;Rb-_l{b8k9<=3-=M|+l)qlj2e0n=VK`bg_05rc z^uy=>x%7(L&N43is`Sd}ixmUEUe5;7F+a}CVOK_es9$*vem$JauAC{Ga(K8=JACHv z^s6IJumPG!OdEV)74Xrg*+BGP*uce?^($|ev4OL9u`B;t#0Jh~uq)?N*p)QUmSC=p z>`xjP`DZpTa*uZS!vD~(TzH!u9{Jm(@W{8=;R}seQzx;3(H~6(EIB-K4?FyC_ki}l zrG(FH+V=p>HSm1+>Yi6jb4jf4LY@oYpJW*UuQcX6Bok!32pOG_iDXNi!mj*;<|^bf z%EeOIRS)Dj|92GwqmZvk$~6j}N5S(bcqSQ1z6;=U0{ZttCgXx;dz=7(W|{RS~> z+}H=6NLS6txT5_>HDmQ_VL!5i^6B^1 zvw=t%8({k217uTW|4ZM3t>DY=z>nXB&Hf$si~9pReCZJEYOrJTOra0*(~oLO!dFvD zFO&W5=)n>)LLU0xsV)U~Hg)cLicv$?$~#1^%8bom#x zKeVW8S4;c$j(EMXThs$HkZBV;CW{mY#+ ze#-$)8AFV8XDRkYKURh zRIF(bn1%vu)mprrJbK6Q)Drp2ZIzQAq@31(lH^lE5D|6RasIKQx# zC`!Hr(mfUMy_c>T>HB(n=Q2lhU+-K8zh#Wz3#^^Uy5iaXbpS+L`)N7^p(eg4) ztZk^+yrE)kc||!}SmQ2oFZ2c*o86slI1jkr>&DQ;E1j?4^A8KJyVxUu+ZQ_+c|rl* zXNr5=GJ971{zi^-Y|ZJ+zTFPHpzZy3(f3_6I)ew|Wav^Rv&*`jEPbgYR=3gUhPuw& zraGNf8S&I7)otk3Ha*Jp)i`Equxa*r>D;DdS1R_o7#~J)rNUNOxUB|nGtyZA-Miq3 zwRqWGpn<{gpTF!gg_qfKHZXgdgOPXl+Hk*2!gt|ls|~LmK%me#bp1$PugdF(xRw(F zh5TadY-6l%j}~RW0+I|goWXK>gueJB$#b$}yOX2*04@y_J||m5hjgldK0V0NY4AYp zB^&!iF6(>39%YvRAf}s~Y~N$ac0TTw+;n!#&3@D)WMGd1kN)~Xm`D;t<5%?PY#Z^S z_u2$MccFqZ8?UoYb~tFaqXml&9?W3Fi&=iic31D8PhnC0U$`Y; zIM-er(rtuijPQ^zG&!1!TGCW#8{^FhrRXvG6B#TsI5}xD^*?kdg`FjO#_3at9{Bu_ zeKMVOivEe7PcuP}{-F;Wi4&-OpYxv&`3&^nCqvo`{GdOg=laX)(ug0_b&@Yo8?h0; zDWNH_BDZgn!VCHcBfKlPkPAIbWCNg+0+f?BV22PWAvl#X&Ziti$<5y>Cxal)aBC`2hScD`<<7 zM@EYTeMOJPE)fHo@Q+nNJ%g%D`-t=73JPDh{Tli}e|pfyoP;MOkbOh5`S8&}@IRbk zPbYuILK)Zw{Q#d8%AZZ^Cj5)H&kw;IBlX%;l5Z%NRX%$JSSlF*Jkj4f`*Jj@#p&lc zNq0xSae28KyL>< z);6Ml!O8J<)CIl_csKBF6W)!wz*hi21NZ`g9}(?mpf2!V1wI>ikHF8(;QeQ#F7T<^ zz8)9wSVU?3W`TF1F7QdZkY}63=Su#!NxTL;@gER)-(vXJz4wxQ%AbluHXHXPp{+dtr5ZQBP%)f@6T5)hj zQp{hm9?TBSm_hW(aeteUO?B>X`_bNo_R8PZl745nO#bEo-Yx9;!c2~Lpf1{zKiYwp z{s#YserLz@3w>~V69Z)Lw@BY~k0SL=>xX^H}{$v}_Ok&n9&%{~-PI zPoxNs@hYEPZtAaZPgY#tzIKbgX?`2@&F5VI65icrJ^Lb6hgT6gj`c?c2kbHVv zzC0)5rTDD$F3xX;UQF>PV1vnM{igY&^LVKf9$#+w3&r0{phNt8QPpYGLDakHMi$@? z_As4xJ;^1LCPIETR(?qvU!P=tfcy>fXCd*U3hKvp!+s9~K=}athvM%`JYEkk)>Z}W6uAxYUDwIJJia6Ufj{G;WAf+T z6k1;^Qcas_iW-XKFmGXHIMtL_W>evs4>J8}1*Fm+*YO%L*`( zx3~{@K7jC_%)Pc$o*z4Ser(SVIa25TQ#98}{)G0BpO44q3$MG#`9!|*S`FkA>q(lR z*U%p7)>J-|oE1t%9SjYUe^P&G7%qmlXN4x9Zp72{HR6*{H{z3#ZW-|@sK@b^^^C7? zc9)DN<234{z?2mjuo7_>vL$d$0xc%Y(=J~mp z({p>$oPa)yLo;+*ZW-zC^k32XH6xi_it_wscqaTSH9O{C!;7&W$jgTKD4-vq{f2}4 zPiky`l9LysKS%owM=G}$`48WpxH)|e(F30!N_E8i3Hfbi(4FH(?@^*Bf7m$a? zAbR*yw$b16Lpg3^`{KQBH_bOr&;8NnfOUjFVti^J^T!V|*i#XXWYIIx4C1f25A>pa z3gtVgq3rBfJhSpc{P$tJYx6_v7XmsF2>&+w`yrZ7#`%Et$K>z8o8|-X zCVvOsFdv|w{E*rI;qNoazC7OnzgZ!}e1N=!kNN*ajq)k@7YfD4=kX;(Cd1l;yLL%8QT$KJWo!h z{#86r3W?do>M#oOhkZQ^4N&NO1pBW;Jy>c*(P@8<{joU2r2XOR+}}w%(LkwO&r+94 za=~9_P*0~5lV~s5|B$vamM;kWPea@p7#|=0)dzmGWA}W>JK^;t#`z(@YnU&49XKjL zv0=S9iShRedjx#%O<=#Ic>5|p-&6XzX?d_;=j#Xg8(*JKo9t=L&rJCU>~|o@kf2EW zSrfxqr!4vL51Z-U&9nerXVX96TJB%&{E_b`l>K2(oIhqy1vcu!d_VOG<}2h63Gno3!=+BamAYPjC5eLSbSzwA^WB1>bU7*PQ_kj?z6NKjF6I4B<~4%sPOC-Hn2>z_C; zI=YD^if4!yng7B2Z&Un2KBWiwrw_5u9|#|dXW%~*@r?FE(QJ|5-F!SV?KjNv3*(#i z8_>5Yo^iZ+|6$s10B?$KBRv29nDOkA$zQDT44)^(?GJu4ah^1IQ0fQv!+vHeeICQt z=d<{{anI!V{`_9PKj-JCJl}qOG3cin_Tz`}d4qi_-;ZJPJ#l2Di-vgTqzy@Y|Lt^+v;W3{3YA-l zcQv=L|IWl}2?oV}742Q9$M&Ph*CrXyn}E+Uo=1`YXH&i@_Mdz{WXJZOeE&`Ugc6H) zQ>QXF>XfgZTMT`=bb~%&|4iMYPt$&6{QUI~INvWOne$iVx0R3GTzo@*WcYqS23tdi zO}rTLSy~e0e!8%WMJXQgH>^MSyqKT&it`M^dH7Kf-N1SH3pfu4WpDC(I96%SuI&<)26D^nrOY)xqx?%YK{h*JAQH8Og`b7fDF5 zzS_w_c>9q|>YqMtk^O1$^Z$Yylh=HHkiotPe#Cwj9pL<@vggBieuB>rO!>*DP={1^ z${h{d(ZC%I+|j@t4SeDnpu=;7+ChBFf%X34>hC}JQS_@X_SgP)>)-$5&#Nc=P1A?x z|2BH4n9c2fyw6qd?w_CcLVrJF1NQo-N8kIGhUnKgDs5fTU-#$r_4lW-(?5Rs_rHt2 zWoH%j>t?;6ud7d5SC8EYLi$hVQd}>jFOF)O{L&(?qZ)s34)B)=paj(>*wZfa2wlPh zb-b?OZr}lI*CLrk5tut0diH*DAM86}*5NEfk!_xXelcDH$_HE%kgK@x;ft{0bPMzUU>| zao#3ac|$W6$L~+G;wjzHmphkSX4~+2627OJh6J%sO9soak_aDDtSfPQ%H?@+C&nZB z7X=I?>YwNOMBzz{7r7SjP)wrsG5)v3b+2rYh(0DyfU$L&ma0S3F&KxCJ*kME4LVKn zSp0!gaa)p2_K&e8L}uJ>2EFSkLhmuTX`}@BnB1|;HOEMekI(!@A}bl2l->>M<6Zqu!tC)BwvIE#&nPK>q}x-O|~PfIjog`P^$ zMq}2|Y5_IqaSWcgAswvK^)>fm?zy=K_0i_6PxIJ-RmI4UfR*_4G(FFe{5!rdx7>WWFC$DlgZVKfncP^VkI z^Z^exO!RMA>dQ4V-ealPTIw_&_0N|G^uJl0XcLH!f89EH#Lc$(xb!9I<@THHBT<_h z;r%pUQO6tYMtT~r4=X-VpPwE#R_?~D8)sslj6vMXgjoh&jh1l%bM>^IJn6c91Exs$D98UJ?R=uE3z0N|Nx*}cE zsClX$-)Go#8l@zCgT2J{P}+@+M!Cp%x=EC)5T`ej3%J3qLB~2m86fdO8X+Bz$zl`a zdM59+|YxJkFMt%}wetJ--nH6I7Pr9=Y{lm=FM63JF&i`AF2A{;}R&EpZdFvcnd;de%O*1ul z>}jEupG5SQ{g5di8q+?u=j3xRDYk;p@`GZf|Gd>r`bn_zS>;JYuQqqI;)4Ejy}#uA zo6qt-YiE0(b^V&=lX1r;VtqX7?H{va^T31RF?&BVMm;ra-_-TB<=AU8e)g?hhWF87 zICk5rj5`AVti&Te!=fWu%Q?F6aT9(}d_PWX|Kt6SJsIZ`82Z;W-hZ#+jlOqg>|f#) zn*W*GI$f)B;e3JSlPIU{IIv0Z3?uEj&oEl5np>{Ds{{Za*+8H(IPs7VGO8_)QQpY& ze-b|1XRDh%{#@Sk-+gX!8~fh;-~4v7?QiBRm{stTnLj?1kzUoBwirWx2A=;@3#r}s zqq>58*rOUe;rTx=tz>LlCH}S?uWLR3hZID%plhOM`w;w>^oC-?Nl@&ob*L-tJ^(mB znCDd2ew)Yp;73Cyq7@uIwg>T?F7DPQdhwjDEE9|>k77#Mn0V5Q5h#^J6+odylL)#y(IMfcVNF>a+<+)c(h^-Ipw? zwYJ9}Lt4(bM8B)(k2FynvH`7kG?rpVX^8A$?UUNo$_d|06+^Mol3&Qr#Kn>-X}>Sq zA2YP2TdUs5f2h63!bX!cxf?htiO$-d<~gar;--v|W1{rti7T}ynqE?c%Nl4h`d=Y zuQaGE{106Xbpn4EUPnAp|E40_drUx!lL1HT3-MoSp~EBmdHI>4<3#ogz%lXnb!0#3 z@uJA>eZHim@u^tU z$NaLr$1?s4vc2+OYELe0ZNC(g_wfYde|;;6QQZ8UG3)rskE+qW5A8jc@dssl<$r2? z)lR{u1CXfwTM63Jl(4p^AB#fN3Bb%I{a=#pCHt|9RsZYJ{y!&>U$ws^Tf$Ab{jKAZ zPmoTx`6Qv`k}U^RSJj@tx7vP3e->?#80h)-6Tr>vxv4Zt!ku>fW-suCTFfQ~j>ubsgT4 zvNdHJOPd?pR(5UOT-VwdSihmQYWK>xy+-HdidoGtbS+d+MP|C>pHh=Ui(jqKyTul{>0q<5uoi>n&N)QBk>LgQTh4(!Q~6bJ50~H$=Cja6Gz| zTiZg-TZ?L-kG4vZdHYt3rE3efwC$+aSyI=tWm9o5;h0ssH*W#ozRHyk zH}({_K;Bm9>S145ae;qxX~5U!hkZ6y``Xt%yoPMBYGaGHWc`}b_O%6#Z5x|?TkAT? zws`C719dA}%bGjNH|?nPmX@tqS-G}wYh^|EmbTXJt(ABct?Aynsyfirwy_8Lt6EpM z@!_Ub)va4LErGr33b(YcuW#Ne?6tGBdOP|qXxd!eZMBQHEb#D_mZc9Ue^b|nw$_$l z!>*#nW!}cZ0nBz=@ZVOU zUsb`co!m;L;5yGHFDzQRbZJpR!E#Qs(%ahUZxHWg_4q*<})#aMf|j%j;mYBSPv@gNucexx!WuK9-h1-LivF(0J`m&%6pyFE|q zN9_M4`M)~8mi`UrbE0RWZ{h^vZSOerG}aLV{j2JJmGkI?Z|Kk2Wu54Lpm4aJRQDFT z8GxTvI9$useYw;1wEhh{;rSo_N;vh)ivgc+2u5=)c%m;i;A6jvwA%U71RCHuvCkZJU?ycA7(bmr1Q%4dew0v-!2`1!F%8sLm1xrQS|(9(g!G#RsY+dfxK} z-ZdH}D2bjkE$Def9Mi&P7*`<$eViXNT<(Mv5LZEO+<6OR2QFELotu)2!X&|!{|kLF z_D#s8V4?@{PG2e~dXEGuzmGtniRjl{1O9uoPef1qn?&&YuL1u7;Ka{qZU@HHywa}0 z|FcNEh`v^u;8iIHjhSfN*QLB}X#zFBT0l?zT#|6rc8P?mO=`L9KTq~kD@~^K{11|U zJY9+xHLsqQe7+>%YCgRx`x%r5RParJXR1Nb(>m0t;3>WwzD?HuNr2h6a3%Tzj6>yw zxUe^ICHQ3tw@bK^ZxZ@bzC)~}m{cM)-nU;Q(zNjDLDN(PXe|pY|D$f|;M*3sdfL}Ji zjq>J8y<7R5E8$jo2c?0m^1deF)_#5_;Z}KnY3Rqu|Az*+QQmAhe`s7|j4ZeY{D_1T zKgQMU841770v|Nc8}0k5gj@Nir|3{2ex`|A{=QVg)ws$RnrSaomuEt8o?mUJ193`#lM_j!St970O*` zN-5n@T%)4k9$sUtSzZ-f#f@hya1|d0EpQbFMl5i(^s4hu)xTQr=Yl^f3f?R2x5@%n z>wUliSL;0!X?JIXuch4^>|xj=Gb@HL@Gsc~D0U zE$uD%nMYq|=gl>1?`{(ewfH;P_?sNLb>KfQIX?A6{$MA58bXTN;tN9VN`IFS7e8D= zzhhaswtgXgPp)y7pjruitoL`7bm8|WecfGtf(m*xG(nrZ<^G3S8vQ1?(4N0p{@mU8 zEN;hm^z#)B?ez0_t51gV1#qc+{`{Oof?eI1=bagZ|amIu|U{CNV zszMf{geyCOir{9)V);!sU^8!UM`uH4z~4CbcSVC8P2MgqTg2amLkcUch9u=2B}f)E zc6JpmyJ2m7PWHk#Zrp@Fn0hN4cvG8?znd5Hu~Yqa>*w^BUBv?XG2izih{#mS)YD~| zpiY|0C^4U7B|nxe2hL@qj)F7 zaV!V>2LNe&V{T0?T{8H3sA4E;e063;`{xTuF)h2{n9aI$NtCe=AY(GYR+5Ra@11tZ2;1k);A^qo)9G>h) guAthh9Qp|iY#p1Loav-@{P4JxpV*)$d0E^4KWxVu3jhEB literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3deb65d63c241dded8f9f6feee524cfd.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3deb65d63c241dded8f9f6feee524cfd.o new file mode 100644 index 0000000000000000000000000000000000000000..80b0240d6f89c9f2c40cb97f1803e8b868f0a9e3 GIT binary patch literal 51616 zcmeHw4RlmhmhP#aB$XeM5CViBNuf~#MhHJqRD?j@!?TFQYnhLO09i4bso~E zAvFd_V7sO(wd=UAc+WfdobVOT{58Jc#pj=K4rpO<(D%Rj^XYvDdVZU&hCccOZ_n9P zzMfBgYS&qxXXb}KwWqsX?V6qwUvu17NcA4?Q)>Q?seILvo{#>Z3iU!mpWb(9cdsw} z4?ijn?e9mK*W>$n8147$eE8U&S06rh`njFQc0T{`v8TPEA;{>$XV;9^o*#U0*51MF zycY(~JoLig$umD2eDL|#2A^B``rw&cUmtwsws!|lJ$z>Hs^{MuY|43WFzfPbdtPaN z@&uPpKw3YSfVXe=erZauK6!>^k$Bwrj-; zX9iE5eh-)%dseJ`_}Hn_4*|1x@SgtX2Y3ADXMhJD>L`497=JCeYtW)td z<2|9FKHvB~RMz``qT-_^>gT`mB1TZw=bNzSILh9!vC{jSM8#omXuropWrly;8@Pq) z_Pv*=I0*iwx?ZZzXtBkyO5ceko}VZF)z@?Ks?bnB;`D9A-0?isb7G>W=eRF0^W9=^ z&&er@V(szYT^1Sweiy|j%lP!ZzwJ5sK=IE%Y4!Ace79G}`?;_W@$5(Zr}ZrU^*0|2 z7pi-=&Qmp|Dzv|NlIPy73g`aNP@q=r^3(XaMBO_*YIvDrL;DwxG5E|(oM)#Y)okA8u;>8Ke^`g)(TZKGc7a}&Qu>% zUZZMeWQK+od%PdiUJKmV(9mn?>VsLp9vKsQta?)Dy)(+^tby3QWrx$D4L&u@xRlwe3?eK(D&Glq=UTAQ7Wy+4qDplhLE{$vm zI&3I(+wk-~umQYGV6%V?da@oW_J;8&Wc0{Z@{_lna`FHO7s!q4Nyryx3EC zy7nuo=6i_yZt^Sq8~~qudl3A-d09OnAC6gwV2P$~32`HpYs6?}(0y*Iv=`0?N^IM4NWuj&7B#{T}( z-u?YS!g%*L`ldeg@%uOT^cQ=E`n^RD{dQGo==A^1*uToZbIqx|_?kymcVx!TJ05E) zpY#{db*3*k@Yw0AeEa(!RYP<0?s{xa?%Kyr_yW7;cnkKN^m&F(=EaX?1_uUDtnv(% zKU)2ZotLCNbo16Jd#Zsw`&`EUkN(xOf8EB3f6+bo=)&Ogcl>YvuKE|J#(9Ra&_@3q zp`mx)$=J2iGv=YWIcpz#=MK-%639r*^Xz!=R(7>|sJ+IFB-k;x(@)N!ZJ$-*UZP#F)XXv4O)AkU*{{b~L^h7~VS;yMH+33mJ zb2i(v|LiZ+{!_CHdrpT#Ll1&?Dg!$7y??CxgGU#JZawKg^^SknNnhHoQ(yB8o%+jV zJtzMFw%r2X+~C=N`ZCY{Q>|*~)D4&$@;v+d>wtYo4W0D)clC$i$2#AxlXbvf=G)bO zw`b^Z?i-U&ZUjBfx2rE4zwu&pUdqzb>-5Y{-3J-C+~;MyVVf& zZzW=p`o4b^#tY)15ApCXJ~gusbC2fh^Kmmb+^i}Besz2C7&WI6W7y1XOEJ$>_nZqa z!Q4~T^GR!E&!_jjtk+KopPsXHd~#e;+}0sJD-k1D$L-;D+ybvEL|Y?bF+X)}LmiTL zi|4{{2QCoThH*jbfq+NR*ch?>X%~ROI&p~gy!rz0SkHcDyH1%!7m3fgNPHS>|2}-2 zj&Cf(cRg`|{^NQ}fInY@HFhiKGmM|PZCeiTKunJC#|ip{-4wL-xD?DZ(Q@w40vcSBCrQOc${#%fun1D z_6fcz7piked|2_0iZYHJA@DVK#f)T zRDIa7u_4mZ(A2J0qojmNy_p{6#frk~^(x;OpU3O-D1SE8wudiQK_zv++2^a|0STQttq;0lchJ$N}klUHn-JA)vRbkEIP}W7OvYAZK`c=4_h+Y(9+%-iA2Lq zP0itGYou;NxFOco5NT{~Xo=P~HMiC`G&Q!|9o{@wpSCwQwlp@^#=_0>QfG6*Et}W3 z%@2p$o5OR;uMLNzZF8+7o9BfaTk0EP;r6=LNJBU}x4mu!(bWxgkbDO*Z_zFfx30UV zp)MM(jfTTZTHCf%Mq8W1H5(!;s^^r4!{xBk*!2Z4qkThdq@n(cV@RZ-y&QJG?$p+uUG!M>1+>xi}bRb+t`RcSma5Ht0du7HN&PMz^$~ zI+eAyv`1@OqUB$J@U5$DYi@|teIdFTU88N0eW{r?b?OduGQI$P$mY0XR&&2FLzw$k z8#G!6+eyWli@9FASw|$7BIi5#avAdFG6W0Y@PDRd;RvDsHP*K4**jTIHL-sALaoWp zH!saVp$}Kf^0-d(u?r`TuWSAo!By%`T&(-_6}!7=yXIq;Lmbb7Uyz@Y6R$9Ubx>tG zPD2(Cm_F_UKiHj@Dz%X_FXRxI~2Yq3~MIFVa$ZcO;QN5xvxTH1GhJ6%s&?X7(mt2F- zdSbOjV~E8io9b(Ww7byQR2ywZtJ zHq{5)x3otanuE=o+M~hxhV_jt4Z)gKt5)6-{_4^-cdS@-TX^Z6HA`=+URtfLUKN}d zyt)>f5ka%1P#Z+kq$};OVBcBjx>0=#eE;l#qDY|q9{*T#j#K3?NS$BEc8-6&AAN#) z*ZPgziN;w0^^!*oj8%S1XQkfi;wqav~}Hp zrwaj;AicNZ<4ITDmd>~IK1&BI-D~M0OAnyKsXozJs9%mDN;K6cIzQW89>P#{mBS|~ zx={6E7cs|CfAs`~`|MKk2H-Pic^7==EI&HM$Rm5G{jN(=@;#s_DNuW+JWKUm=`7Du zt7kjQQ6DtL8rh%aLCtbQ8+#L~&Cisj6IMyUn#gONykBx{#@=`;8*^sMXEd(Ii&!6_$wbamgCclk3T<0 zb%X7(AOAFy34N0ec>GxbMHx8m!}&FqzY*K7&G_7n0vr^x^!+>g{i*%_#C}&(Ar7*c za>2jX(Rm*_?^kGz5)^J=o3zEYjCfG9a@F5&(Oy*Q3ef9v)l!s~lak)JkMNx_XlqPX z>eW2pEwLS#eN^AhL6?I*o~yhaA=58;Xz}=LH7TYNx`hew?D1l?E0XRf`xCv7a*GESe{z8soG0@gx<(jr}EV7n07nS zKJ||%-%+DWHI4Glm|yp)W@>u^DsB<#l7llSDRzuuHc_ft@MAiwN#O$TU@5WUWC%4yIM zeW{@XH0Uf{#QjGCq2IDCNTm9J{jhguOpW(|Zd*N&t^OfT?fnyfLN$Vs1$p(k>WS?E ze=o>HAM8CbTkYS$z3UIC-}s0o&*K1n+^1aa9m;dlXup#6p?EpMffG7PZg4$Mplui2Y{YGv={{TYzU7D+&ikary1_n+1brhPsMG*n~Q|M1R+WH54 z>IlWd0iTIr#KRd855QAAuckym`n^fNe<3YCMyx zz8~|Cp%(oL@%rj)#JATtEYA~Auji?)TfAOGu2Pex<)~+-A)eDhMCy1(`xOzt7y8AO zdL%;~Uxo3ImMBK4w-9Y0-X_H|H0>uE@jo__mPW%bvEb1RL#Jm@`%D+aGScG6DHuPx zydZ`CUS|y6HzBM2m$K)qkK|1%GX1apdHhjzd4V^L zLeER_a(I<0DM-&CI^iWcPaW~5>1mDg4b;BD+azCH|$5e>iA~=EbmB5 z)7|CyUA8(>O!Y$)pl$2WpTIB6>;^@L$+w;NiFk}S;t}m@|NH=OvagBq3*wRElj5;M z#3R~w#Ut{E?tdMR+8YBcGahbmdpff0QpM z9~plm{v$r0@VC#AU*K;u{(Z^u@AFZ<(Bpq3f5ZPKe;}V|fBRDTf%*gfrGVD{Mg1d@ zLOWUz&u^}RzpGJ7hxIqc6Gopo^n9I>LHg_Y8S8@|Z^H75=F72g8{%=haYT%TGu0F!Lw$?TU>}r|DYHr=UAK3S9G5 zXU8}fotnRp&tU(Lw~*!#pvPaHdIIZ>UJzQ|v10Y&CM-=!rF_gcay~}>NY1B0BucbD z%Q@d7A1~X~kz@R2)=RGS5c1o%V;TNo{>vx-VbNPp{)>-E`ENG-mz_`?&+BOX=BL&d zh`-JlMhm-8`@_%cU9Ddi<||tMr2LiT@=s?+j-Zp*$h!Wsk?b*gy+!%@_1WY<#EbPG z`~iREV}x0MErtFIaiats#(r0PX#46N@j>=sJac@&zTxqauchgKFrRnEa3SZ4kAR4e zOc5W_-fUrSj=7SqQ@P{iu*S|2rdZtS00he6QR8=MCvlG*j z`Jsx&`{74*eyBoodOf1^aoT6(hefzC1$bItARaK@t8_l4^$3msn5v@mk;=!5WPVr_ zLjATXn}2t7sUT?12g^2X_n9jY)4$#bjaMAB9zc7`Hic}yFzZ8C{G;3z&sW@bp7DTu zf_QMnAFgMXZ5qg&t{X&t`AEdirC6Up9|}j-N5_+6eEeSH0pnZe4~~D>L-`^V|6_;6 zLq)_FG9EJP1N4V)5!7p2Yy5kwb)WHXSf#gHWxIlUy;Tv(uC$N`KaxS~uXuVtrg_w_ zi1`D!#zybbdZ<{XwWpc*9Y8PCW8D<-JN#XY^>;R1f5y%HK3kPUvMGHe*Hc$U0%^p+ z`aXl!n^#8C0;FGpXvAkl#ET0893DfLL^AX>e1=_Lk$qIm{;Tk25FOX;k-h=5J~sLq ze;B-3&sG#U%7HI<#x$3h$L$BwX}lQy$$qcCzGS)}mgS`mN%Ug9iu$SR6Sz@d@9fpJ zp_0Bn{H*l~?9EEr3wfO#*1t9%b>5egTEAdCT;7qMT2Es8b0v8F9m&x0^?C>MyKBCJe!C)tW_?&>v{ewcmjJdL)u# zM~q&7dpl!hsi2)s{)>&nX=?rO7_Q%Fd?XeceIJYX_4N+Z7@x`vFx{U2a_qlQeom)& ze7RVCi}yP`o%(ttMFWKP5RW>4>gzq^hrJUW^K%L2vjFy6Lcr2N{woL=|Fs!?x{5Lj zUGwu)F<(uU^HrMhFU7~}$VZt8>||~w>3Y6?q9{MV7esPBGamCZxk%4f9<1-SE7~JV zBp%T5N%5-(6!jP6VSWhc^&#h{9jL1hif3HE#c=%-f_(DN@jQR$eOPWft`AFMIcEKA z%i({2(*Nkcf{p?&`Ir4)62lk*jq5>@mtTPWAT58PrqSM5-9NS*{hv?OaiIToJOs>e zCHfHhpRNyhy^rxy5F3+2`3>#S^}GK_WHN<24z6EhrZamzR}mTK%C`j_x%zrk>x=ff zBKhf*Dz$$#9Z1*mX@3Xft1B`&9oy@U@}hLwG17SC2X7=`;?2&Na=t3*7@t83;rbK% zb;TI(fpnq+`uf7`mlabv?U$kbv5_o1js%*7k5!WXFzDc5oPfU_Uk^ z8DH8TNH2Wg`nEGh`^TyLD*HF(?@sM6S3E*qsvKfKyZW2-J72sa9`;TwNcmgm*GZ}V zM}6eeBA5Tqmrv>XH8TNsrs8Gm7Uy{IsV9mG3VNZcBcB!o$R#$Py2gv{pS`2>5BiJj zL;uhf4EN8!?Rc5(9xqs5eb)GJ#q%?{j(CE;7;oL z97nNPPZZ<&Hxu_aD7Bb=zk+`Xu)l&xq~EXf^#SmiI)Rh_Xg=xS{R+&NplN@N>Id}i zUuOLt0bv(FxSD(h6>`wt zrMCf}<|Td8tiB~3YLZ^C==J?-_}Px9iUvV)Myd+EB;!YX+`I z#(HUj;q}Z7G5h<2E=T`&j8$|!i1C5-PM*3UmZPs1b^Ud;e(>s_9uOb6o_Q4e2bgcW zQGxV1PQQODrTatw{z>%<^>W12&o$TUm_L+yGa$K(*n%h&p@@-K?arwucU zFAgk<%w<~ROFE|L{edaEKl32p3(*#Th-g34Iv+2Jq$l+CnrT04Q6#{$Eze5Odd}*H z_Hvju?Zt9KaVj_KbI89jQNeiK-VID!c@_Hi-` z)$%i$w&mHhx!}-u4AZ)NY$R{7wr4=^XCa^FE~e07{l|7xPuB63AII;jkHVe>iECIM z`mZEbse+(&J_cP7E6UaWrSYfrxh`=n<4@$lU)OMbtKYR;Zs}aQ;IjJ820ou@Tb`Go zc+~#T^2ai5%L@{;yi573oM~G=PCo#cDxbr&F3*ot6(M8Ct7txj{IMNNC)0qLQkZAr zJ62h#^CR`|>-zUS`+GiqpVs~;)cJNXab2w2<=Oc$<&t_vc;`Xz!;ei7i>8LP&2sU&P9m1X0ZUIow_uwWsYbD|5=b zAvU#?#xtTfph|Nozm!q?n(n83Q;KpRs2urpQKTqA*Gs1Vi$PCH;MO4SZ_twy(_Hiv zo^L3hpnNLVxAAZZ*SGXEuCK@Q!bq@aA<3hQ0`%viNHNRPzdtXElq5=%?afB{<%tZ=$CVrpmcH2dDl8%YXnkaSY5$a# zkbg?8eigBjB_Ysyd||y@f$_JNNXkc8&n}FVO(lP^JRW}~$?;cGk{o|JUP`BuKec?s zAI4uva{OUEM&l)wkI5g7c+v5Co_IM#@qh90Li4}w?@{7~<|{|MKwo#f%;owf{`H0g z<4GTP{urO?Uv19>(Z3Ua&;BhdBzww|@#V-Lrhf~`9`4^sF?#%4sU^w&TC#-ttB~?9 z*5Bx#CC>h#{w901d~I*()O?3LSN|Ql$o|v*xcL5~^~3r4kG4y>KQP{5kL|xp^?EW( z-!w_<$+J42=^xOYRkQ=uw(b+k zPky3lJ<+!A9MJ*Lg@)F@|4@0M!GFy7!b*+*Nawde?9xlIT%Z)A>C9L#XlU#gB7QSt z)2BP>Kpb~N_uqN)l^&lLo3F}B zsDDR{XKgR_Ea}VkPom{ik`BbCU_(*UURn<ZN5%&c0Juh?{oy>dIZ`+s9gN?+jr{?vU)=I%V0obJxAL#T^iMPn@?T{9 zEQ&wyO#DO&XH))VdUIsrLu-)sm;eW>8MYk1H!P-n(iX!01>!9; zGI=t^Gt0}2Ou35I%S`XMe`293j$=wdR>FF0$Ndw=C+Q7DPnw*halhD(`-`R|XMrI z^U+wlUQXmcqSworkxQmfYBlXW3ViX@Bwp)VG%-nQeWzWLq_w`2D8Ov}V(4EywU}w# zPqO2Fs8dXhThmu`vaY=*dA#ANSe& zS*w8e=y=KLQ2P1+;{*4H>EEXo)AeFoHD(>TenNaN%S1TqBd&knLVqmQ>n-Y!Ba79x zE#nccaoVlG10;E%ClsoXlSX{SEnS+6@*+GeVCiSQ*uO1Q)u?LrZ)amXjo0xqk>UaO zufZR1C>=CDRsla=EhUnahW_8$V)AwFB-{XhBF%lj0Q}{?-w*q9_f1UQpMB{csq_Ax ziSa|fAM5$o?#EO_LK8#OKFu%3i~W~(6i%f1SMR4FevofM`hEese~90oCeroVc(Whj z9~-GIBzqJ2oqD|EdU|3UbR}I+p}nz@rG;eQ_`*l^{h-TstW`9>@$ZZ7{XpmarS|@Z z9jL4i>aSJwdo!3(+_54hga?Rz_X=Kd$z4?_OL{=ARw z_e$97(XXSu_@6xqaF_l)^9O%y%kzQ9_;ufpf%{Fi^pE~~Df(-q_Pn?s?tJa(a@*dr z9ZvbwUb%YH;Xl{!v!k_#{jKx1N9CioSAX&CjcQ+5kw4FGU;N_R`@QTtE80sl_j{${ zBaQA)(Eh8>RqpthP4<{_@*m2V?KtCzhsEfxSJhjhzjVG#$@kLlsppYz%14#ot@>Q@ zGnLLSDfyZ7``~%xoAOcRpAquCcn=Ydl>C(b7$0u`6qA2a`lEcfe-^3}7pnir@>BW` zlTYKNOUO6RDQf-ocuAA_0QU=6e?2ShS*811mybGLFm1T(8L#)RQ|(RA`_&HrVS!@X z8`(ZnK5BdA>Wr&BY2OI>?)>dxxdnYjZEv^h_nc*0K1s!Uued+`KOjw^`{nVa2fp;c zmmc`i1ONFvK))5@%CEN!v(%G6@_lRht^b<1Zp(*PefW)`bN79`;KQM-^?rimp^~&Z|pMLqvzxyq&8$8+mtaHi_Aa<_sm7?)?S8(y~$xhGu z*pz$dVZjvb@szpfN+0fybk(Pan=0#7*3UhjDc3;C9cTqN=zDq-E)Fg%dlR>Nyi@)t z{bN;$6-wNbrRvT}WTedF_b&Otx(TfOSFH7awbWzpw$k0;X$Nht+9%l>eJ@iAHfx^kU2 zi==<7f8Be)cO6O3_ZNPwC3ym#bLt%4@=5YysKOR_3+LVo^u^Sy;ehm=l1hftCi7^S_KZO@cHFs!`| zu5zDO>T{>7`~mcf+n(iAp3*Pr6PJFe@_hE0vs~Mm>mQ-VKFgEjDT5(zdWKmSg?1#} zczom@F&?5G8(9YRh@<{650M8ebE^H0$Vqx(gvcR1M~p1(g(;`9yi<>p$|Pik`ZgigNfMSXmY(%o`Jj&_KlP}m~hL};}PTOV>6ySBaJ8TWs|S?7%$`} z?@$U{cfdQxVd zk%Mtgec(BF(1Y+vg|V*TnUU|Oa?d#r(tF6GZCmNhs>FwUn5Ab56OX3tXXbj7V+)0d zNlcvPK-)A|Wtp+%$Jp}Axr)YC7LP3(BmPcjjIbYuXKtOhT=u|(VfF<4G~;>BZS}`t zw0DLdel_-^UtD+eNewrzf4)6`^9T!a4)Y9%>8-~V13Z(9m(fj_PNPt zo!-u5uOUx3<0)kHjGJbYdO|;K7wJH;?~0EHlKCp7zasrZPW@A{@c?22oi5ffBg96h zey@a4Md*2ce}1MfwLJ@xasKb_Px2#lNR2V$L-Olq`pFe1Blvmv7<2hCwbDL_t#lp( zRq%3UddL~z;LyB8bG^SPq=UyN zF{FcI?(oav2}#LKwT~4Yu@xK`#GOI-ak}VoKssN7(EwA$GxO^m~2w39$D9g zPY1q}X3){`RNqx6fR|Ww(#k$n$5oz=a<3PHQ{_?zC+(_Bvb}EE!`dhLoo@N$Cm|K=9X#>hRI z^BFzrclD@KCwcDDF&pN4$E0UFd^JqREMPi^`$5QoFLPXST;*95-F}xZrCv_jRacKX z<+6X6j@plIA80>1v#NgLfYXmz;fQwN3y-TES9v=5D#zt#jw||h>Mze-ap}WOe?&x_5On2!egw_C z=SO^A86lc@=$Z(6jND4x*b<`W-bY4+ur~-9+LnQ}P&HhYt5gJWcBnN*fj7 z>=%}CG>!ILOYCse`>*)Nr#$CjmAzV&zCoP=M<`(KB> z+w~8F>cQ_^`s@$CQ`oFtocS-mz0>#oE6b*rzcTHokLKm9Xv)G}b^Jco|H$uu?ZeGt z1NiL2O z_E7uuUR-XE-v8Q%cuAFKsiW69%Q2XjI?I6v?SnY7pWgqv`X)mgdlTxYrA_-NUna^e z&E=ux*o#T(n^5~e)9RVagDc$cckKf0ELZ5aOb0#=n&cUKVHw7uzE9HX8wXALb2-2N z^(gMPTa6n42M}}RuyHLu3bu^){#Oz~IfA}NdbjIJD!_sKVcvUUNL6jWZv;Qxo}cdv zmtT;fEpoxoP2dlg@O8~MPck_xnD>YmI9dAKyhq&J8R;xE?-5VQ;e56Zs!VqP{r23v zM;yT`j^um9EuZlMcG08e=AiVR)1~F3opW1r=MYvYhr-wHw4&||& z?w`aFuoVbHZLvD+A?gn)lj^Oe`2xoV^OgKpVks^%OsPwamu&sL$(rUw`kms3>W73H z^|s4{D5HZSk93#yF%0QwGheBHx7qq7mc>4n^=bd`TO-upE$XvmS%125NLz>j_WM;^ z|4G|0eNjG?^(pQlVwnE*b|?g`JgGnTXUreVe5xn zOc7)#qx?epR|y=qztEBLM(V#dlq0WG|JjH)fB5eV2PpacY(m8o2{vR9Im3m zyA6{BI}eLZvT9KOff3r@X6u(&B(+b6!=0LLu=PWt{oS@c$1l~Vke2o5WAHvWLi?eO zTD%IHFD2XOIJz12J5fI*+FxtybNq4pT#xxvz!<51Y=rtW>&W^QV4#cI5I-*eciH+@ z{p6SQ{~f6R$0OL!^#^QCgsJX-**=X4?bG&T6RwjY+aP~)ePVy!dOD=dLmRHPh~V+c z-_G_(R~{F1I?@rc0phHuSv;OaB_3b=&Gl;Vhx3+3BCQdNUy8r2Sr=^&Rs`E>>2LBy z8zL@`1;4SOp(R*{zrcq-x_NzYZ8%Kl;c)5Qw_hKu(SNJ6^lL5UWx=^+!8vmme7!8V zzNxh~x**&zzj@yJmKkNiyKh02x(y8u_#BNLt-`vd*1C;h{H4(*XC26InFsNWEzx;% zAy)tW&Xi0lh88XJSpx%VxHsGcB~!STxk(u_*1`w3Hnum{MxkDPkp8A%klfPR60D8f zy@~$dsHQI~C$2flg7nu&>!QFL>%snNZPO-pD&*m>Yl zG{zynrM|KG#(Bw_=EPB6->|u{4u8TFYTsCgzked@*Xt5jcPO=LQ#9DRKDe%RQ%iji zHSWG0e_^c_e@qd_*UG<6EZ5;^FWMSh*8u;UE(k_yTkdWMe(mbHbLL+=f5E(K<}W}H z@cFegf30ZQ3}5JGoOY|P%}KR0O#WQxG50*>2r^QT^o~Xv>l<9ris#)@{bTg4Z`|D2 z-iTNVuG;gGKbM+Xx{;*IlRKvbIR)L+MoLzEWjl&9lCj~{MLDaT+NNnLljRrxYUP!5x? z<87W+Ll03Q)qzLksj5b>uj6B{lLytWTrquTAL7NVuSN%_NUq=xK~l9lQYE~9Ur zygc7rgPo#jo~b#-`nd<5CkazR|Jb7p-%b;rr3s3Tnw(2Q|IPcj|4-BZEbv6mcXPjE zB-1n2^%`LQ(ZNlp%X3$M7YUwZZ8Jss-?!}bhpNxHWNlJj){{Bk&i(WJzbJUB@VgVf zL%NaN1CDw*j`Q=KMlb>T)d%{zSK?eh_1i*PU*e|0B$Rzop*bl&189ErIIefQqo(9j z6p$QFwd}Wlf(YXGSy4&z-t)K;D_@=SQyct_TYoh3_gq5$?sQS7sq0fiIZDk*GT*!l#RqM9tB@93cq%g zcJCiWKJK+p1?aCJ?4yI@^7&U1`%(CBjnXdQ`WT1u zY$ou{kw$-f#mNs^%w+Hj9QZr!`4sTSIrvZ6bCQoO#pKa#&q%(LzZ~|`L2?Ssss4Ah zk^IWf? z3oZXz@XhnOhCkKv*)AS0LCa_VFyFkK13rcvtC|*6>cBhs=4Co5e7R{>l?y!c>H3V0 z1wzjSmQQw34(4N}gMSG#@x8&p$5wFi=&*d!pKYST-Ijlq;NL}?q&V0=hEi&+oWLKl@;?!H=IdVpA(H%0{&~aV{Wg`ef0&X!ncJ(kb>pun@9`XxCIJo8Hhp7ksf`Y?Ziz_XqU1)ljKfoDCd1)llK1)lX> zDe%l+E%2;ojleU1t-v3(@lY@D%-a0~3Ow^e0?&3;3q13e3p~f+N`Yto zYJq3FY6PD7cL{v8)qkzPGrwNoIS+3Tc;+_={2O+DY!i6q#{{1J&>`^5j|)8eVVl4+ zf4jgRw)XB6c;$;@NCyx0?+(DfoHo;2t4!q1)l9XBk;^05O}uhtiUt>6M^SA z)Hi2fXmUPeen8)JkXj$^vw1AX@)^(ke1T`X3I(3|{MU3i-*OyI6?oGbcxCf#jleViPJw4V?-F?C zuNCLRxWKbrPYOKq zpA~qvYq!8Nf3Ls?ZGY_(c;>$-@T})c0?+&d0?&H(3Ow@<2|TYW4huZbbM{5u7n{cxASGk>kXvmfdOp7|RDp8e1y@XW`bGjqic>)9dj%#RB^>$y$fnZI4& zS1FWPXmld4}=44#~HC#xtM)x+vq>t|EbFzW!qv zXqWNbHlG9qp7|w0AGWJh;F(`8@YOax=LdN?G8Jo{ll;F*6` z;8)vtJ}2hQ^UDN&t?l1(foJ|gfoDIM_b8<13+7h~Jo{m}kk9;;0?&R}E%3~*5qS2) zodVDNy9A#7uvXxiUoY@oHeYQJc;>eWeB8!cRN$G9KZomz2lgBO45&UhewZH@c+O+n z1fKcZ1-{OzW0*?&-|wZp7rb&c;-JV@T})$y_knZH`#S-*R(j_*15GvDGC*l*^STY7`w_t~4` zw+lW$&(SORY>(R55NT;>Y7eh#Y^jZGQL`cqO|`SuH?}l}Ya@}`EhLmP0@xh9#BJhZBuyTy!wW9*HqMBGp}y${5kdY4fPAwU$_36 z^$iUT6?5m;t*_T2fp3heny=n^(@iV%gROToMAy_dwl~z@6lrZx;7GRSF_f& zx1SfLmZ0_OJ65i#2^(oiK}(>yd1lr&)3SM4Lllx5YMZTHjq4(?8eC(Wn$^-8ZJ4#} zwy#{>9<8n0Xizu7cnGbG;!(48XpFc9kBTmCeQmXVxYNZqu4>p2UQZ9|s+B9hdTVuK zZ4(|wy|uM|Q&WQ?4~OdwOS>i9Wb_KRw$!1ui}r6gTpw+X7&o)rO)Yr7xBmB)tX0*5 zqOD=`@TBn({9{ZvBpl!*jklaWys0(9fN*qkb9nvw#?KBMRZ_3$@liZujPEVY`a9A} z4_umNR~o;qgr(J*`8iz2+JxI1qTvR5Iwl`x?G}p$$ z&GS-cbHXi~*SF0NhufRObIPv`hofzCZL7`m5H5IfHLRc53`gg-*R3GBx}gq|@2G2y zG~A-~3%9Pj2k{oJjfTTZTHCf%MqyCRhRBNQIpyIn=M#RFJ_wBG{84TAM$yrfCq zNcJB-Xlk&XWIy?s>ofmp%aiSw+xDftT;E8C?l7M0mj~CEe2JwuR1;77kL$lF1jzR5 sPHvO5hRt7w?|3jv*asi6HC$KVDhIsQ>@~ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3e010db787a2ead6b11796e755278465.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3e010db787a2ead6b11796e755278465.o new file mode 100644 index 0000000000000000000000000000000000000000..91d9f3c23ed109564011a52167f73f5b82822e95 GIT binary patch literal 81032 zcmeFa3tU{sbw57)U|C?bEGvXSLa;)B07;04kU-A`BmsJ2Buhw50tk?hjU;|6WMA1z zyk5DkU$wE4#&+7I`X^~B6w7hjd{U7%HF92KtZFKiL=giERGjq&47{xbf4h0n;EZt&9my<7i& z@a+^f{?=~$;G5eWgYP-mga7GZXHOqx4}$kIY$mJvdTwy&t$Oz3U)b4$^F85~3yzGl z7ai$T)@PWXRsGZ+92!AByUp?AHq>40x&QgI_ucdSBHUlO_nznfAA4{dyaV`q@Zjag zM$bNTZuG|29vgiW_u{QT8olAQAC4}c0GgM@J8$&dXb$9c@Bg zXO|Gqs=-T--t+uDz4t#~{KIpj_x$l=qj!71KU$Ld?C7O?UmGp?A=R~Vuo(0b{2ko( zDCGa@XmRheqc=SI?C9VtzaG8ke&VSaymaq<&z}X)-KAd}E!p^$(WJ+o8BK0@X7nIr zzI@}$;He%YU2S^okDtH0=+V)V`%qWuGozb|elYrvBd?C$b9Q7jsd)9^rOkIfUvln` zpTD#jb(MZ)^wNwUjc#&%AM(61dU?^y;8`=+z7C$XgG~?o@$-ZCo#ix2PqO8XH9dBBOYtjT4JF;4dp^k(Zb|xfu15VDbg(Kn z`pjkzx9r0hn`H}*UvSJiOSE=7t1`y}X!>}Vh4H43iFO;SdJ1F0W+U3Hvls2|&6Mvn zV^!ZqS{~1%m-Ph3&70fn2jALVhjADjfA7}sosq|Lyz%*_!&YL9Q5uiNSPWt;{w>Dp zM`Q8BH5S8DjK$N_jK$NIu}EXo7>~o2@o0?SsmA1`1w)4})tCSHy<2UA?;f!azE{s4 zytKR6`OX2y;8>dz<2V6+)HS&7m$#gH17rKlcWZ~-!9WgkI#cJcNXy_)gX8D_COq`_ zHwMRxoh}QTMmBFHXhtQ;ym+d@RTCY_dz=a-jlUM?f#Q% zdNPpq1k&zA+EMVlZy!4Ix4^dmcLKi*{59aiz?*^p8Td=U`+-*hKMDM&z@GqK0Q?Q$ z2Y`10&jkJo@E-wh19kwP0=^A+Bk&u+p`JT|Zw6iq{337&_^Y_@$9)IVOOgJkz%9U= zfeV4}1zrwZ51avfYaQfv3_+gpo_By(;QkiwmAH@Mz7+S@a9@P`W!x9y{xa?haQ_+Z zxwt=%`&`_g#eFvJPvbrl_a|{r#r-?Dr{Mk%xF_QNP23&0e*^a@<4^pu{jU?=BN^N6iyrLy z&XNbazqsVV{gEXP9@w|!!TFHSkZ}b*a~UBY$@!v)9bC?DK!(3289e7Y{sEuANuN{B zbv%jB-=WXR=Q^In=cnm&5|?wt8}{F_H(^}+M8f#NZ3*v@TWpL|U6CxLKiE2tWn7*+QCEcOa-ZwCAD{1`&xz+cK8w$%=(Fow#~1MV zbM!eumuGJH)bY2R;|E`JLKfH_*U^Q7_D?})AAjNx+ee*3PX&-^A!J$znHEB(g^HeD z#9T_&li8QMP+tqx=RDW(34Gp5pB?8q_T%$D`fNYfaTuR(pwBkFFA=Ykes*`uX6XKD z{2jc<`QC|B&T(cNUUQJ04)t90J$Pe9($EbnokKU*;qR1l=-~a%@w>s>^I5bd;v8zP zi>wI-pK!Ksw7=Ju#QskQV>>Tl?wEvm-my7~T{9lYie~j`A`d`PPKS}v# z>iK^jhkjG-c=GL_TgOcV>9+KV>Gu1T;QVA||xc27ujhNSN!2EXo;5!HEG4HJ#eDBuU!AlFi-z%@xXg?u((t)|L8$3ziO-7y+ z@qJY(<@=6)lXqyFkPHuUl;HvR_Ypl_zs4*e(Qq;H)Jj{ouq<_Vup7A#Qf`} z7i))piFw&C-)8u1LmC@@`HkA4m*21rUHpM<=vSD-U73seqip;?I%|hU@1^`~{4(Zx zzn)n)^!~rF@mDYpe04$H(0gC94ZVgrFBK z<0nz(ZNZ^;4s046`wbg<_ec};(l|KsmyW@a%DSOTj=G_@pJ3x-ZJP(*I?2ZW^Qdj; zt=(G&-%PO$z45nf{LSrK2VbwZqaGXTv=6P52?U^2mE)Sr2>Bqcp>z>hv z?);6M?&kT^xlCujwruds$(dZ%GxamiCj}Y?OTb?Yp22$m`J`up<7ZF$#{#zCw~~-f zc?VCXjh(4aLwSw7yt5~LW64NIy;!RRAV-r~&&bVoN*}+x4r6D9ZM+D4BM0n5?)*($ zwvkSqKC)EMb`x!LEPsQ@zYzH`)_MLF#IwcZS&i~nQMn5#ee2+5jMF(t%R*W;?kjM2 z=WiR#z~?Ibx%0OVrsKYle{LRh;aWe^3S^lXW+h=f8ITqf;;K4r?8dUIbMvv3qP1Uc=3k32;7TQ@Ya z02uygCIe*sJ;p@gmb`x$nKk_BQEn_3w3GX-$zXH5tY@~tk z&hsND3GW;mSqJR6eryE({mgf-KaYBb&Nz0BjZ^}k#C<8zcb^~Gf%JOZH&eddVo9XpXOJu z|E7)kSf-s)8M8B+jUJ$!mY(S~0=vO{Bkn=`CE_oQ`Sc9am9F~7GR?K#|~vC_jw`nyXH-&ERvwD&r18dwf<7QX>Fay`g=Q$9&I~R*>*^4JJ5aTrZz#@*?suvkV0jMZC(Ax`?@=MA)>H-9Y?#{x{mgOaPaVf>kl68uWsu<)X}#8 zNa#>Uzgf=uj{c5~JgIKa&JAr9ZTpUBZCwWsQNeBWXL4F3Xe&cI^>%#-nsKzJBh=mX z;b};y`)GIQCbS5>6zcB%P>eRTqvJ!-kgo21#}2gZ4|N>s)_cdA&{3+Sz|1;3dV3Fq zI{JFJ(e;In^dBLM=cx0@;iLT>hx^Mu1jDzpqwi36sPjY7&FD7X7HePfMBzGl2g(T_ zf<82wqmEgg`k^slqi=bGChK4kRAgUlf1-tkdLMz|I?<06Ni3&TN~z} zn96tbww1L3bsp>LXghkSyi9BB?>x|bxU8z}z_E@{SE%D)Z(BcNU0dgo-nM<+n7y`j zbtC>A>Fn-m!&JL1)ZKp!OM$GIp}}Mb>KE z5mcK&6d$y%U#BRIg(5F@AZhY0orvs*xNzI9q2!Dr1$9J0K8{6_J{)I zm8HuT2X@h|w5o>H9Xr_D6*zkQXn*&iz@cMD`vYCw`wt%O4(!~%eanuvT^m||*aXT$R7Gx5;n^v6D5&Gw&jzY}8-`Bpgf`eu_vj8(8V9qdXPa|&F}t|+*S zB)&4=q%UO`6kN()Q1BA=oPrm#rxd)14J){WomX%%dtAXq?2LjJveODKWDhI2fDI@( zpWUV41uU%KJfC(fb}Rim$fT6hqWqr9&1)`HfvPyTozRD999NgYw9{Rin^h|~k(DX@E*1ce=`Vr#6bxUiU;l>;z-ObV}kBV`cbr76(ng zOTO{9xtv8o6oh98Z~|p;3Iq6SMm_C!pdL7mzCU8jcQ<3QLz?8cRbXhIW4FNQDURC(ZWq`maG$_E0*3|86nH@3 zK7mgQ91!?%fi;1L1uhc!If26hUl6!V;41>(B`_?5%UdllEQjL(fdc{u1%~Bu`ojVT z1#T3$S>V$Gw+q}XaG$_u1P%-QxWEGfpBMPFz{5uVnR@-Lz{xfz3#aS2-N3VS++*Mj z9rqa+&Vc$;6F88iW2)a}^RqsInV^@=)9G%5ACn)-?-STppyNz|876*24+wk(lMup1 z0$)JzCtN1*bIWvGE%0!)j)MX}j>!S>Hwt_@sN-gXzFxNBK64_W^;=1G{W4*8VA-4i=8fU@OGmD&*-CxZRLX;2wdm2;3)d z873o?UlTYiu+g691UA|e_!FJqXiuNOMtg<@HrnI+tj=E~%5N9AOyI`_t`?X*r1J*_ zZWg#v;L`#(3w%XjUEXXKd|1zK$U7jgA@2o&4SCBjPp9@9@;a)sm@_-ThT*gNGSLt8 zcRl5-QO4Y_r(2OabKphF}Mppe=_Eh-=j>pjC5m8i($wwBV9oAX{D>jq&&whj^rAPJv1e037i%ZjDf^i(pC(7K{BtezY=2mx&$G~T{4fV8f3Afd@M9!N^!YLR_fjqT zC4J=i8B8Q5&w>~~`ty{Q?++{V0)hM$j5 zuJ5H>zBwk{0eYsNk5|zjlppRx%0I`?$FGz>$D}7Bf3}~GXOSQM@8{!}V_hEslRt^q zpJ!ryJm#BNACIy2>fi^JR6X#2_7^@?Ja9wPJ zf1Npl=AThFjaQmK&G0aU*O>jLGjtzay1zhti{+>Iv{aFh`hW#>xv9@o_H~Re52<88 z&`%U8`Ww))Jib`{wurV!$sajUV&R|T<^IB4KV*7f8S9rNUHQQT;@e<>YoTyaVL+#Hr>rX)YYdmF2dCNR3 zS#EC?D<}Dl@pNLj(*6N$#Y}FWhCOJrrSU@Rp$yA-S?T5e-ITW)<3*Ntl_Kwew%Wry zvOH^8jUvxlMV^PWO_F^sMg64;eL!2P+p}5!x|x_BNcyew^7vxXgBU+C`3AIF4=qQG z{)n8YSLBUobEWogr~clQZR*=b{awOnz?uDBf=PC){>X_%YI!H-SoF8v!@P3+8(6KP zkBv%uslNke{SNeZ!%V(DH~R|<0m(l^G)xvuO#O#fjL79{c{-wh;g2KQ0(^-1(*bR< zhq-0>me7_UU2fWsAbsRh5yJm)`=ReK>!Ibek$*sg{lGyP znBzatuL7dus=oeS@w5Mlo?O!5pQX|h}BRXYNLzKH~8!+hEMn`_5^<^ z*$?THuOEcGkqPsYa?5<9awcC781{UkLb3mcv^lOYYT){Nn(GhyTdF=hBqG@()~nFR zoH7Gn#CRfovikG)zcGHK`38;ebP|c$kNp8^f1`o1KgiqPBaer$)c!c*L1}*~ z+CRrmP1W~DA)nHIvcKutug;&W^PvGPcP7n04g0drj~>#}WBqTmpC)*cJ!Bf~)!Uz* zZD6DQeRBJouc`fNd)Mjqq4ck%{U`F+7E+fv{?++dy!jUG^~c(;&d=huUy8rf{{f@D zqW$>>HrlVr?GH@Te(U;!q!SnS2ek1PisapNnAMB^P_@<;pAeFjGRx0?G)m<3(V<@@!tzTo@!*x$l_l(Ju6 ziv2M2Lj7uf;$P3!a0Y#U=nm{Rd7gqk#eNvpS4w<(Iy7Gt2cdqB@gnM{(K-Boj*qFUD}^W zq}?bJ)AJB|hRyi?G4y-);{bG_zpy_ype1=&h7nH*<8jY$EdHL-ocYn)M0-geS~g}c z)Wf*AJV{)hBznHXv76%rfg{>H77VhW!GB6io-r(!cOpSqKcCW)k}gQ}Q<~c?^CLfo zKjznBzv7`#wv+Y`4E@pb6SrO)m+zG3#q=b`AA!G>=@DgqO6^;K(jt0D<@%%tg>wE} zOM86amFZ~;J)#xzrt^Qe{uUzsM~V!L{UAKwlJn29oy%ckxJL7!H8QlFPhW6!!j(DxHe`+dQEjJM}%?bomD&sg_gu-{xj`(uXwU~dIM z2|uFwxxM&J`%dRL-L!AF8@Ww;Ci{>v**C3^xqZVwH{AVo0J>n`uaP|`dswDy-%bzv zo^0PKe%0PG(`9>mL`&xKB%Ah~z_C+c_y1r4+1}@FUtRZa=e3d(YrF!?bsbZ?e6oz`kO5 zdVBYJ*qm5@4QR0Ur(}Cioh94fDJ>^M#(2KQ$7}Rwj%v?Vf0&18EZ494vjJ_khnBC#_=UYETkJi>65mUG zg~_tLmt`(ia7A{Bz`DKXQD6~oipJI(`S$NG!g`!lk=r_EOFu`pA$ zcORF>H@UrIgGwGRP-GAdqzmx|@y{ohcSoCxn=&ntWdS*DvLel6sdI8o+8@B_^aCc@

~S6Ir=D^lsIJ;r+;soHxX zX47)}t@DQ>%&uj6u~I)hzoYFM$= zZtq31y;r*r^6tMndnbFtN?Bh&FP_GavdJ)-4uoNL;j)gH4sJ+{8i$&u|B=M@U_A)(QKr?lK$)gI@~Q~4JZ zG%Ngt*)kpLKgAxgp2z$-=8s^nxw5@s{yI;^%jU`Ua!RY7Cy#HOr<$jZN2@49Q_c@4-mB|<)gRIM9(De|e9HMiL{t66^z;3dmiVvwgNU}uBHwiL zduzP6`ajiQ(evA|YVS_jzG43fG=G)jds(Sdw)bVr7c01WB|MlE-}4GFgE7|ou=mx! zcY80B?Y-IE&E@|9?LE(8@6ve^Ztr1>y}$6m+PiGeWbZU0WP4ZbccQ)TRQNx#cRoDm zAF==TVYm0YK5X`0E!%sKyMuS=2Wanj34~!^JibfkrMSHhTkQQ`KUjO0?V0SIB9tM| z)x`H!h5sXa=fi{k`Cnr110Obf56bo)c7L3A=m%);g%*1^&)?B}AM<%U;Wq4#udgq} z@CPxUx5jr`Q5yA;z03aKlr}NGe@x;3$lm$zpnv|C*!#mDHhXWB?fqf*9^RoJpuHDa z?A<)iD(pQNvu|$iFU9Z&Vehg%Q+!{B_R01x`-6D)-mdU}Wbb@<&_Dl6?EUnI&EA`3 zdw<+buWGp3{rTzl>kz+7WP8t%&Wn@1r`w-XpI=(f%b~xN=j%A%gYOH;{H#}j%X8~_o>gziWK{MNGrqWkCjj7XXO6Fez;5bAm;hDMBez=dF*~6YQr{wkJIA z!U1z*Ji^~u&XX2d&XelrM-BUVL~~l6cNJQm=Pb00?}P+(zMSur<7K?(G4l9^eIWkD z?8$om6wkLT=SkG~h4c3LYW_rJ{KR`cY>97{^CW6KgT3Xe@lN$G@t%jN@ol-~Jc;_e z3h@bVosrvbwa2;&@fUju! zzr=FhGf5e*@$6Z(w^EBeSTE$2OFTFRd;eobvp=#MGSdy>8NsP>kO4wvmM#bR%T zF6n$G#W%Nlp0m_nrkv-@$(bN>GAO*zlm=WhC-?XALMZ)38(rCZKZ=2*^CPG@iF z>8icWnqY5geDhoE4d!B8eEwAc6q~?9Fc(&od|3n;PG|7JJiA$M7a|d!y)oE%ugSvG>`^c!a&-{P$G$rhDsH z{R=GXt(@CBEFRV-UmgCM-aDH;3a-K2WTNI;X{q{S> zH|70-ao%@_^6|!RtWRlwWRmY!Q+xXMT}fg1etx#*rp7Y&l`9Swr@Sh>k1&k4B{F*d z;l)&^N5e`E{C(}yS)mv7l%xPY8gx&HIjMbp%^@f0uL|{%c7=X{ z>yqg0lpI4I;-~j*^fmW6X}_!L6!?i(H{x2{xh^%mx`TEvwqZv_ST|37@wmK_R<5WKauOpC@-JZ->87=6ZL0k!9=+~ z5+SeN-Z0lM_#HSf&+6{*U2|=P8xB%tTc-lhrruuPhpeOO|L{5Jx)raxplE)9^<@)Dz)L#`!azP?oDDS0IXMxrj(?7L8l*q>;r(=8+ zXq$L_ynimH`ffj7)@b(6exh&T`)z`b{#e{-_E!(}Pf?@UUmWK*n*G&d13j?GERWN( zH>vqEH>vs4H^uThkl(jO&F|Tw=67#V^EHFFMh2RSkR&Gn?jBC)W7RpcxxlxAC#|c*r@Vv1Qz`0 z@0&tRjk_5OwmONQ`oFP3vOJqNH-f*xEPoj` zlx6v=faUVHZr%j`CX;^)HiTvV&A^E|(cWfYg}-_`__vw-*R53fR|3oOR99m^DA;WB*Q`?c zTY%N_@%$sW(&XQ{TIJsXtnx#^V2jDWdX37z23VD6C-`@m{MWBl`F8=U<)gyj8k2wR zI+cGNu*!cu_;(TiQ#AheRHMM)I>LCLR6wiVQ_$o`n}dw7UyD{p@q&@)FzD!yEUh{^ z8{@m$oIg}o(|G6h0sbm&RW)~i!rxRyR&l!UuXtZIx)(DZyl)yt$vx zcN((R!!go@?|0z+Cn+JehstNq34h#Q3)$Fh)jWRSyUn3JwVX~oB!8%O_a6F8@?T7K z2SU4-$+${eCeOFPzqd-J2SR_aT*a-eD*o6C8E0r!%glHS`wfH|8)bSR)X<>f_H`0Q ze66Z7<1@GabwM-Uz@G#{TenL5XisyqjI*>VKA)@N@AD`AkFS>LfzT(`sJNy^#(vaq z_8*tOrA4B%wV~C`v|k$S3-fpt2(4Kw(cNjGsw&vOK40Yd>+5BDAhdqHiZ^VKG1YI* z|9Jk58)bSRv}u!yo0?R-Ws8In&#IzuXklZ#1wxxQOLXXO`*sy?+os~{u9LB~e*&SE zD`k2hv~#D5ckEE{_18-n@+~I^j7yIvYeUOgYk^|=-L*^RUmL1wfdAI%7!Q7J#d4ax z8u6$~t6gW>8}zp}w0!SwgCF*qp|!3s$0zXG(29@!fke;J8ltpfZq?`OKEKwuj^I`c{%du*Z*6E@I~=5xzqwhWqdm*k*Mq(+R(>6W0maI%+K70`s)!DG zp`RwiH@2Lxu8+-#M{I?_X(t--LSNM9!~BucS8qc+VXIC4H7n`)cMW0a`>vy`aT&dk zi85io2kqGb|IOACo!7T+7yK<-XO`dGESK*)5o{nFr0=g{{)O_IR~zNweLKLxRrCTV zy0G4y`u!TzKUIwSLmwZxPMY6bp#(9%$Lsh0k1XHq&~5uMeuo|Mcy{Z^EH{ zS=zGL?}dQw*Op_mNtFK+d>^m)UzzEzFVp;G1*W@ZJa_rEGGH-YP+q>avRtAge%AoY z^wkwI{iaa)3i3~7d_G0_*H%h&$WsA+Ie&1mOh^851pHu)86WHLg;rBO8KgHJyGyle=GTqOg|I7Isi)1?Gr}$cF=HFB-)BV~K$Rp?9QXoB6jb zlIeb}6!OaXoAYHl<*!5jC1(DX0-5gDmZCj!{+)#~o${|o{!%mlu7xt)udPCR<^1cG zoAC?%P5Ck54K6kFw=N@pQf2y+HPDC14|^g0M)0qS@wbM-Z}<~GzCUa7V>}W+%~$#S zhmX%i(2poj(6@kILgjIKKJ=s3w+-|~CcOaqQS�o^R6W`L{}M0lmPaFNA*7@@YO- zNOV5mDlgv!ypS->SG3}C%Y3U?oo^NK`Bt$x-zq9H<29de6&IQQAM3lSNHM1y@dfjz za^g4UPgPpE94}yh>$tyXZsp8rtDz0Cu;w6h!ytGus8y2Z}B{>|az8Z2k z5?--F#bsqGUcFkyl@%(kt5b0>sN#(qRlI(^iZ^Xi@tQR%UfihSRjX9IZJUa>Y*F!| z5*4?!sJN*>#ih+EZpc^h&YddWT&Ut*yHs4hP{N3h1?%7s&H2pQP<{>kvyL&|)`kje z;jbjTupa(H!fV#R-X*+Y1MFAAO--;z9e3)3w zNUgsa z?APX?{VKf%^emG;7wuQ+t3aP)mQT;yRr(sx=bHIvq5Ud-9q8#MJp=7m>9wHGGRvQh z_N(-I&@+gR@l~a*Wnqk0BVJZ%O${~!Bc4)wH-LY(Szkkg84uC^ENv~HAFnmm-&#?? zjOQpH`*BSsKjJ_3Q;O91UdZEnks03$3!|t%wx3c|D9ck7DdKcPUc~nR@f-0yUn^W| z&aV)!8@2UCW<2NiR9G0M^2vWep8Wm*xi5V`1?|b#f(_)~3Tb}E<9A(?iILw8een9g zqwlvYTo|CwEGXi6;rswl4Cj-Bk#GSy7zr<0q~f9lDlRToabBK^mo8Ot$vhP=S)$_X zY!z2ls(5j(ip$GYoRg#Cni>_CW~q3^3KeH&s(8&B6_?Fb@v2oSo-;?qwY4f+ez3TXT6#vr;@>Qe}%rMHSxST@L#NqFxrcF985RjGu|)0Hk4Db4)m2o=k@2VhQDC} z!g@Tf%QpB&9F!kBo<=-(OuN3HKAzLt?+t&y6!Cl(u<(Dpe`cq}=oHWO_gfkJN%>lq zU!ud``_jqYXPNPQ?o65P$9WsIeENQ_Q9s2CpAQZ=m?r#fzLq;fru*?ebT$8cuS}=> zS;+4<^Dju1>3$6}DW!b{aDPVqxxD_l$UoD}zc49ACx1Kx^2qurPL}CZ|2*XPnE4l_ z$aFt{AEumtiBqOi{#@jrVdh_&Ak+QgeS$Ilm%C&-<)4rIUNe6s!imv7l3#!Sa4dg~ zC7%0y3y?pR@?$*u#ruglKm0wFUjVwB=)ivd{%$#5`h2kOV3JAK-*+nKr}B^N1Qb zeo}kkUxJBdeKj>usJ^~M`?It{9?uKSc%JD~-9`dm*TmP_+#v-Ll8OuX_X{z^ zgnry^mA+_^igQv_TwJW;@|1Qjn?qT+y4#pUHH&UdM}vQov>i4sP9bCN%o z;%h<%{HqjyU77HA5>A{0|0Cg?9CJQ^@wqmXoelpY(ev`m`2oi3+E8UR>{X&WXJNs> zoMt>u@FPC31QWYF@Yl>`;zTd}CrdPOQZmK=9KyVOw;TSCWfRupIliA^tj`lre=@D- zSe}_bh3|({64v86zWpHhM-nKW7p^elx#jl}QNH^A`&4%7I0~mLj6Xc@RDKT{zt3~0 z#=U{gaPa|r-z_o9^()S==)d0@0Ucg$6u)-W{k;tMtH=o#gM+5a z_xJ#P>;vKCG`d zvheUw+LuV;wNhPv@UbcxE7gOf zJRPEVLl>qEKg(sW;rDa(glztM<5u}qjNd=LyG+kc&zmpnzXyv6g?YXwALrYGfZRVq z9y@-&iR8!gYy4h1`D2HJm!SJkJa0?&?gT?B@qk_h`g8n=3EGG8l7aaG(H%sm@KAo67I0*iSh4qbuGX$g>dVHNSfHY)Mqam-{r9fS%I(1O0{m;qkGr8NX)?emp<2 zBfQ{w(Odj^5%l3BeKTl{_qrYS4A>ZN8o9;M{A3q=kkJwoxh9D|A8G0LSAQ;Hs6ccMbyyW9mnxwWETYgSG;uo$AbwU zwTI+&-f`Rqx~RX1pAYgNGwbvBVLYKfA}9~oD3999^|4PskKhhv($7=Vl|bix+@Y)l z@<%$R_&LXg4H5~@O(cJb3;aZ%!G9kG{pX_J-*bn&3FNo`ek^T-1K} z`wBOH5DxuGnaDrmcqO%mV_v?Ge?$^5f34jSrTSy^w5TXw)bCHm0);M7->hV87~#_E zbBAW8U;#pxpeLtLd^6|;@Mko>oqRmgdD;ZG?-$CVb>~!k-(ea&0^tN+!KhF;{ zKz{NMIB(3$pC3v~)B6XBu-7|l7|sWl3H}7mPyJ)qYvfjUl=X@6RFngKr||E4#`Nik z3jLtHOVR#}6ly=$A(zkLprObw;I;MyKAw2~m$tLDdCB_k)l+|hPVLWN0h=yA_*;DZ zd9=E4J_Nl$bI&LJ zxVeAMWp@_S?>`Ug{P3r`ey#|d!0z0`T(eoTkS|d^Uyc60{dlsQw|5WTGux8{oZ?On zpaPB)>3pI)IcQ*_``yXy23`O>)14ePF!86mlTRBs4S1G2dDy^|Kf#@RLEwEG5D%k# zdBe|Rp?$O;aGD$v?e9Zhci)De6U+_3o5G4-!m$L55ALrrZXx~i@yPvkhGt?KPkkBv zCg$TQ14^Xl`+FKerAv=5YeO^m?-lCf<7@CoIQ|?$4HT5}K4Ht!Tv4hIzkkL1oBuxi z-y(s|tMBtdew@#TLp9cKSz@}erp(+qoo|MkBht8^ms8A+ov;ajR_ocF@mI5d7 z^RbEP_?dR9AM%*;ga1za-U|7ji=?&ST~Obs%g&ooWYH~+mR zUEk|0`rh^d>KpNu^!?NXeW&R9zUvzG%`Eyx`MSQ{q~D49zQ>~P2a>tIATiyFC8!`(c*)X}q95Q+{C&DKNOJw+A-Y?4OJF ziS|$<>H9L`L1->A81eqM@;`G;dAYvz_0&ZF1O4*$sn4&_e}1$7rtm*=NxzE!iPL|m ze{%ibX|;zWZVzYX#^e|Jo=frcs{IdU`q$)tQu+GickuVA*QRg%_f?Ge0DaHA8hxk! z-t?UslV9jNm4&ZG-)W}4h5u3f{e<|MhM&SQ#`h76zt6cEf1h@3`j+@&lES~wBI2uEozHl$M&B|2cUAsG z^3S@yXG!{|`D)zx4ENW+7k?7Ze+TVX>rW_tx_P|BdKvKn^G}DV|BFsl|MdF}Nml(Q z((+xb-@bG63PpTE=o#mBArMHjElL<)j;CuXjluuN-e8RkZjE{z- zAVFMue(Wz&^|-K}f`0DA?_u)Gc0b?m$~rE{l!*Npe+XNlxT2LYM1SD{b|1d|YDfRLSU8#X6A3$@ACCEcB41gf0=@lq z&CZ@X1xT9`(C_-=Ul#QbW50B%t>L)Qo)|y-^yK`?c!55xzb~!)zO(WExmN7Y(eJ~3 znSKvi-oLyJ`=59{ByIRbPxAfSxcTXYMwB1=Q}aiWKRA7UdO^#i_4iGmpI$gIY5vpG z=cg~wPn!R^$@4?M5&V9$*uOFM`!Re(d9VlT?`t<4uV!~lrq9XBQ{HE${$6*(@pN|A zwU+0bO1^imy}Zn+%6p03cdhaTrYi3lHgK)w6-`y%*V$*TwY;*a%KJ0+@U@m#Jym&k zus^%j@`6*9*UL^{Yk7@RmA8|9`C7|so~pdn?9624!CyyCWDEZYf1~@a?5XW-F?)2f z@`SzV&jZZzEWd|swYOKUy}T*y?d$CEYn5+Gdpp6tey!zAX>aS8Cjei1d+q=CvA=kK-PqX@ z>=eBH`g`xc`>A)|f8w@JMcK`=5?|+_CrlsO{fweLeiy=xxEbf4=JUFQd87ed}#+ z&v-Pk|MhRldFfX@p(fhZ}Jp6j})%Py*nG^mQe>?g_ z^u51)^Q(EsMbd5W&<97OKDmm%@gssyID>D|J-RN+CrzoWtNp!u9~k+?+wZ>@jsD`? zFXp}<{q@)0c<0yee)IL??;ZQZZJ+;X+pqrfr{nMbdgM3Jzxn(xUVH85x4!QE z4nItD(rz1z*jS1_8p8UG)p~Oz9*0xr(Kn({j>2=&Ci8@8-wYS=(BF@3wzX%<=3X6)6#mX{NZWnWz*2_nucCI4Sir5dT<*0!_&|kr=g#ohTc34{meA<$ETs6 zpN2ln>0hjPhXub(JFVZ2hV3xdVfsuL(XX@FlI-vdF~cFtwl(w$Z=rM6b~=;k&%W)B ziZS>hTIYQ`Ivk~kd16POpDKz=7iH5%a&5y+$y9#rdbT>|acnl)r?oYCP-jrD5dYd5 z7;OEoB&^B=$s5RL@mKP3E?dLi4Bf{eH!nfOyk!d$X{3FaZUComrZ3Ed~&+oS|>2^DLk%;X^}W-_QW`RB5gvMZcAG%*q)@)22 zQN|>nj7c^bbA1%lrAsj-MTrH}LAdO5XWM43+@Ex3KyN=QvK?jRC zA~w4SOq5@yJA+P!ic@5|TE30SH*CXA_9W*|K-Cd+kd(hRB|ll=@dy1Z;-9LHWU9kB z2k5ib;R?DKDNm|H(vd%|j$|1VgNZDX81c*bl{SW{{$Zp3Ol$q_pc}6QR_m9`PYNcn zND^9TJjxJvo@en81v0Pdxkx6>2|N)T4*aiJuyPs;Fc0AU)Dz0d#&0i znoJwCS|tx0Z`B$veWKs+*gb}{alPb>$PCcQQtWZ*-iVj-lV6EP$I^qe$zUlGUDgIm zy;g0MeWVR(MNs{bHY9BXKhlPzjmD3(A!(y|GHuA7QvMvzt5VtoFKYKhuF8w5UQn8= z>RxnGFDtyL#mi3bMfF+#sTuySi%r} z(%E}TgjR-pJ{wO<*bC=t85#>Q`tPQNNoVhqwc46KQ43hsQY*0eFxjk|Yk}4vv01mQ zrHHAexb#G>CCjY-chkb8vvzB1C&Lr9fbDtG0=9_a&bF}&K&_-*fVgz@HK$`G7@v+! z6DprPbNtzQe2b{WaZgv z?G}47#dhh5soloH*ui@mzYpshq(SxVD5G6t^LiY!AG^8(NW134#?!I$pxZ3Fd$peR z&brPPm#uR@>W7(#OXoA5d0c0EI?l$#VHarJ7NzaS-;wKI#cPxg351{KU1X#4b9*cB222^9<U4{n&sN*MOeir`FHR{KGnZQu(gv^ddF?u$doUy0!X|n4hSf(4IzW zPo5j+a>!4eoS^S&qG$QNW2}JFhoiKqTZBJc!~9?PnPw;H1SbM`pV;kspTu^kS)8*A zAG}lnU7V|xs(l)%96?Lp|8}~U+u2Uz$2<`g z2Eaw%2Bi~G6X?T4Wh@ha){|CKJ^g9r5qR>|c>6ST{#iqcpfmjhoXFGa@8?Hs)aTnRw`O`lZkNotUO0ZH1HFN%y@0cF*VY zzWi68Pd~)IUh>AWiKdr342b#i`i2OUPMQPr1DDGc?B1<#}!<}&M0^xJFVbC_OOBr z*noob*d#~N!lH!dvpxj}SdW5pS-XOBSgV5Pv1SElvqlBaWkChcVP(Lz zrhcJr(nF7Pr(`Nie1h>i(OD~I(tsRGuczXvHte6^9ntUol&q4@6wR@Q`vw* z_p-2pXRtm6dsw@IQ&_WtlNn~ek~~Q)2pk)4ZibbDgcDhr!tY`M;F$gr@TOTABT_5a zLF>mWHoGzYDa6Z|<^?k5Mne4Co!lmEwder23fpzKM=CC=-eXQ(zjD>%N zDLMYG;LlkcYh`hOwaKQngW1;*O{1^0i$ z{Q}Zn1nrmLf0?nsB`9E1m^o|uoTt%P{$*ao5N7l6L*RsIPC@vBpi>MXjFHCqX-+{n zATY&f!a;$11a23YVl>gi0{01gT40LNL?0Gd6ZnF_MFP__i{uLn91xgdG_POay990* zm|`@qU*G|OPYX;jn%6Jz!vbFrxKUtoyj1^bfdc|J3mg>qjKJ*zKQ3@s;PV2X7I;|T zVSF*&(0{TG?~%!k>6`H50Uh^%pKu^g$9)11EYLCOkLVW)bWHWTY<|{WsADGRzQsCr z3tWa-Db?>2xVc)#nF9B%)p0=Jf%Q5r68P~=IxZ9VIoJi2S1s_BT{;d5?EAQm8wD=w z)N!-GeFt@H$lH8a$A-LpM|Et-JD};e!IC{d+oo9Oc_E-cAdA9@u4bv4QXF zbcoDxGi*ifk70q^4S6r?^d5mv3*0BL@7FqA6L>&iqdmS?bh^==0fCM7_+HiNMtcSX zHrnHRO{W(b!{9R&g=lYkOcpg55I-+IUIqX!e7}*eEucw(6JM*{)88%|M{OY$4OG~ zi~J^icwgM{wKtCG>u{9*?g;(Mu&-$2b0hM1bUkWGe@@_kMV)kEL>LbqIbQANw+ZRJ z@V#_R;MbFRI(;Y6xK>jmaIF#Y7_+@+^kg&LnDwr=qzlM0E#(;LXC~n{=+}BLMg|8< zn2@c8r;El*RG2XM)1(yr<6j|m;MaF1q<Y|>hh?)ORZys48iAo_fh3u8=tUW)X<>4|bSBYlaK9uWNflawPB{abyc z?u7~_TtSi2DaND0Aclhh`Phw@|NA1}^_VZnmo@)P5YaUgu7FQZ>Jb@;hhF7jh+}B3 z6$-VMNjdKlUD0SES|FowCqz`lc+O9yz<%|c`LEG`Mgomb${)0dae^9#E5QHN-!o3K zTk|*R^gh96-u_JFH&%a!j;;AykmF9g_RP#EKD;3E=UWKY{PT5%^aU+^?~42{2!&Jq zNoNk{eKfOPnBPj@)Msm zKlu?-0nAIe!t-<({+xcN`jhY%)}i??UcJW4tT$7{hx6i2m5Ph|X9-g>`74r%%2zIt zzg5uj+}kk0M_d$mlmTPR6Of6>|!@jPXzP(`8W~>lXQgmij*<@*Crq@{>!j=3fT= z+!Lq%-nYSwE5QH7@7jsGk>40cn~*<({6S0o10w%`l%MK1@@0zjxk!kYKQN{4e;B9z=S6;_|1eB>|6A)PyCI$2WsCx&B5^n7 z560cdPjdXW`7mUeVV{=jm|?HRU9OL0HSEHg&%o`XgXMfY>Abp<;} zT@431YxS#sYh6X}vChi6-aVB&uD_wScW1}$u48*zTgp2R?Y{oTs_Tw))YfmU-&NPs zd1zz*-qx1h&b}Qxdz)^q>#AxwQrmR%_SSNVc4L3vzC$~ZuVqW^p5519-?yWocjM;k z_wT*#$PKkU`}a2NI<%*C*U_#`Ja0p5#jgI&#+ItyP1{j-|NcF@tB&ofZ0YH0=)GxQ z-;S*tDz=q(HE2DZm0OSPsoYhzcSluOcWWK$=m!7Z>-X-ix@BMGW~#4z&!GbwZm8W> z*;&zXeMjvfv)+!{@+0dj4qVq!w@vVDT-MugU{gVx5_kre)y1FA9Dt8_1sNJ!(uCix; zS4GY7y^Y(ak!NpN$L{jp&coC3^&RWlUB17op(aG_jw@f&&8?)TeGMCL={#O_a8K*D z-qwniTlUq^-_? z_oVrj34J#1Y2CCRpHaSIL2htb>+Km>1y26yJvTWG$wZ4*s;56e@EOhWBpaXyQObmZ%x^r z-J6c?osRbq=RMM~yXD5cyO&SXzk2;-FY(%Lju*ZD>BikAYIkep-kbJRY&;lu3^W`# zF7&>U^Hy|EBVQNS5A;4+{~LCp*FANg8+A8!9;&%{@2;8~I;U&*;Y~MnHtd?Vf9fiA zzCDL`Ow*pd$M?aXwN^Gv!`CbL>TcRsaeNxyKEb=~hCREr-bwUs>PfU6dM%$$ryDvO zVH=Z-x9x^~w${PcdbLUFHSBC7mDREPSp2rE?-BlQ@6DaPHK8u}>9~H<$~T!`Qu&VV ztC+-pnS4D|uhzAD<1zC8@od3t?_spJvhJqI#*WEXNqjUec2ComU3%SiBkFZFzN*jTNJRMZGOo#kaTd>dM>Opj};g4Lw)K*LZdEVa{C;(s;Xi|^|IwvZ8~4)!TlYb-EE!6`g=Q$9&I~R*>*^4JJ5aTrna5k zhmRf!9XQl|=*Yq4%gSY}fGLRzrv;R^EiYfTtZi9Y&GNR2w(|0}jUBy5yW4h#j&+NV zN)k^mYoq@v@LzSCXz;ChH+F{(clX9?gP`b5+1k-}^yY)N+|t`^;vG6ZT3%7TudTAJ ztN(akcPCebP{_UxNYr(-7lea{4_trnaDR1M|Dley{YOHFI{M9W)_3%GY~)FGdv`fOLS8w5`BwAFS6t(xq4-~Q!uno(Jru1zAyh4`9$dx zZo??c$^L`yv0I5a!D3AkeLpdsM6yZy%SAY2OxF`=mrdIJk=JhyzCz_}=0A7wq77x3 z$^1jS{cD|{R;&{*JpGxFsPj*jE^X$BoH{@4%&6DBfPP6oM!Tv-x?xAgxcDcu%Se+; z`WVuwoS;ZF?8iF^|3RoVUOBTTNnbNb`i@D`UqCv^a{=|!W!UGuN#yw%Hb#h__POXX z>~rlT{9WLQCqJddOTU70sQdvzH0&39Dm=ZJHf?Yjc`NXl_}fMLdQN3*mni3Pk#6+M z^G16`x?!h35$*DcecB?YUeIMxP7nt)=rYppMS7aC(~b1I#gG|5`Tu`)SMM4{5XGlR zArd5oh4@tkL?lbZASqIukT38ef>A62%iiw9TfFQZyK^A{i&$u#(mx@Eg;q9#scf}O znZhDvij)?zK|o|25k!*&Q*tHN8pnF^C|cR=wB>or}TGLBg!0pVKC5#ie3pF)nb|2oYplrR0A7Px#iKRyK?p}~Q2@Vk3l_$&CX5*&7XG)=kZ_>P+OX zVGxJjYlDZl;cINu`QK?b9|7m^7$FKh9i_Fqdw;lk(JVjVV1!zR!&GhQ`~ShALRk6# z?fMCxJCt%A7c>n@|5V72vSryNe%@bm8|&x#7W{#j zj^V+LV85o<^>3tK^B)p_o6liLA=^z=GXZLXCPu=2=JrYGpR{j@U;E!8e%z~sI3o31l_KA|t4__nYi{ZxqfeWnE@IFt|t zpagASl@L2m`78JS8o$&2%BjD>{RRB~6=9H;R>uARpnq@N_r1*T^Tgn*A22enH2X7e z_{Gjw{JyFG;TL=RJH^hb{Pglu{!)tfxf4QM@f6{k7i3=jznp7?|A9Yk6(uSI^1^~i9w{%hu_ZYckCOu;p>@!8*#r0_sqIy z2d3sfJMgvbM+Yv)eadw&4BT<Iqx z`v&&@Zr{M!&m0}t`{GXqGN8L--;1DY%w!&V==dG`wjbX%?a6@~p4~UFcEhs+d$<2| z;Oy6aH*oGwqPZfo7W|yM`Jv<2Bk#SSn}R$){m{{YZM82B>_uA9iPvVf6+U=;YSBZ- z&wl3X15yX!z4;|0E_%+1eH?WP$f#0@2e`(;3 zrIgpznaJ0-ACE5Heta!>K6mF=2BrXi?laE}tbOUX1D~$@H_*(=Y^&RT{D#`k9RKl| zeFLA~yJz6~um5!5+>K8UOxgYt?k^5Rrp(TqE!c7VhTwzX=Si8)zMl@<@WM|9*4Di+ zF!hDsf@V%;+g{|g;i2O{UjOxhb9Io<&Cd?p(ERrUnJF(0d>ZnJOqrXB_$n_%Hhg;d z*_XbSjPz7~ClX9Aj~uE@QT@hW&NhZoM?NFHyj`T1kMt!5PxwdfC0ycsw(->k;<;ZL z&r2_jLwH`z@$5KnrAt~j>3*iD z{PnX7>YjV!K40ebjYj5;1!Cvf4@@y%yWO9;&|*Rl1JK)GX5Me^efZy@e`oU2u}J!h z%X>~5m>EdS1q#9*x{ibfWvmtTSWZ@BFUEa5?h9}qLwE6<_}Ejw*)tDt z8Q^PikKuk5?uEF|#661pXK-)#4Q3w@&)p4~v>KP~T=~OkJ6H8i+qruFw4Jvb({@t* zXUB>ICt)+PBgBDM^TdIyUmT$NM7*7A+7}J3UA<^D`;+mT&zMx^u6v^gTKD$OiQjzAZ2d~_9LV64 zt@8tKQ2APo$vao}PTjd`{?whTjj20t|KSv?Y|e5~Ie+aR!@5r8Tl>Po!8PX=4&D)5 zG@`o=+W>SyKoZ zRhMPf#sHV$H)&*M%^2LjqRMvuvE_gDK;?xs$IUltjs9f+(lh3nP&(GS0cC)WlFqgM z+*IYLK^;xn`Q-`Xxz&$P+If5Lq@D9s9Z@;5s(xfS%B^y=U-!c~t@D34r`?F`YhQ}+ z4HORVYrhrYYyUXHAEt0{AIltkZMxeWT(iM^BkWUr6@ag~4!)9_uPX30VdtvH zC+u9^J7MST?GxUHuTeH%Wymu|c?R~ik4Jb6h0T5KlMt?^uzz2B9l}#7Y)HOfn<6iM zdijkg_fh=TM9nu+F>_G(4$fIC`jcx;*{XFOuH+TsQ`~xP6+@X%xnvZK}w&V={(wu>-R-`#TViM zv9HZovM(O+;l3Dm1NZB2_wRG^)MhN&7wOUT3l;qWML&NCJ(cOTyUnp5#`)~ z@)6H>j8AYm8`o3$f|*E9$%Xo>jm$>KW!-n@XQoXFW$LHmCL_cyUeHqaY^Lz}^ zmkp&KPxQk)pG5SdhSJv&{V>n3B>K@q>E{ytFwf@`efd!O>xh1s=Qj}jn4$DbiGG;p z%ZYyMQ2JYmewgPUCi;q@^dBesVVN*`7)>c;i)fF8kw_WWq(JN5idvC4P+9J~s|WV5)=UhB+r&lDQUha#2=COjrKo8EK!+tjHNBDh-tE0ChuFjj5=$;o4 zcO`Z$xwkdabnkA$6(k0yBCQsm@p9WY#EPi_3OBr5^-Zy2N+dBj58HxxTQI;OUp)Aq(?&VWZo_W6LZ>zp5R)kBCKj2F6cL;E8+`(I^o42YL zz#E=v-s-^HufUt{Z#Zq{?IeYjoPFO%Sn<~O1BAgF@x0_Igdy*FRpPFb^jyikhr$kd zH-V1im?s`s-r~#q?Mc8==J_xC4oTjLmwCSBFYg8%0qs0r{>z7c2psB$%5?HaC{u%@ zOcm}j9Xf$B;h8SeWhm4AC{xv8=7r1T`&)#mOyIrhX@pgoe1C_qD$^MXt1|UaSe2=Y z!X)>;eoO9$@XRUqwjfVevGi$&dHzer{HgygTKq&{8jxE_`@^JUp4q^ z$%oD&p60LWS%ejTO@EKD*4bkSEB<`XA*}e@P0tm7O%&Grp&b&PlfT`7Yg--sUE$`h z?H|A&o@xGO5PyYS{UgGPzn}a@8nOxAD(Icrh~tE;IHbRIbGRCeu*&g2R*C$ z-w12{`xk_@{{0GJt$!5O`bRY2kL_$Z^bfcx+?I`(t4>H>Derlx`$Gf+$8Gy0`0jJ? zeUqE-wjYCUJkxx~Nv_b#s#BbwadUqRyK9^jn(Rs9zH?RW6^Q8#}l4}m{C)BM$dzsJB|8{%pHnqEX$ z@z?e{gcW~HFCnb>tNk~G6@Q25x#F*l!kRzd=$ETN=j87Y;M#)@{#LsAtNH=>!!ym_ zMDX{NiT>z3r_1$wlER8V-vGjjzmpVJ{2lpsgcW~n6juECD6IK=pa11UfSvrcy#oG@ zIrv-W=CAF$;1AC0pGtJ-C;BQ_L`WD2~ z{8eQUR{S*ypT%F54`Ib$*g%;0^ZD-DP0tm7O%x^>_J7Kf;YmDm^4Np8wW7%$b8d0- zc=GGO!b*l!6jnSQA{s42j5m9L zbMjXeLjJ`L{vF8iGEp4!imD?F1grG=EoszsJDeA;i=C z?Vf_L;_uK@gcX0gFGE=IxAk&_i9gIuswk}ZJ47^^Ka3H2fOGO!H4XXqIrw|V&0o`> z1CM8#zpKDs7W}m#p60Kq4q?S#TRp;xzorI+6@RtU5mx*iqUVagHVU&I+I_GOv`!v( z1Fr3N@c1(~k8Rt4$1~02RPfk>`Q1s%Q_oLkAgp*iITK;U?Jze9j)_ct9l_aiZA`Bi@D=CS9Kpuw{tJR+{e<5dW29%mt}dAu57 z&Equ)D>*h%So26U;PKp3VlcYIwmWU0amvznE%-R-kmczdK6`-2vs^x#5Kr@2H5*~Y zXVV;n6`xge5mtPL=OL{4+)d9FpG_1dKK}-MI_)E#JNaxw{MusW&y&APalO9pSy{M`205bO!Qh~n<~&a`8));_C(Wx z-~UfX-M`}IwXGX8c&2&PZRjNBr+95zgs|e(w-{l?Yugfp6|YCGM_BRNMq$OPkHT6% zVS}1(aPzqPCggt}vcwqVHu(H1F(xLO3%@PO%z7PbzJE6&lfatCYyN`0C;cO*{^1V= zSW76}`^LW)Vt+u<-q-zkdj}sY&J6nV_rCB*N#^v9SY}|s$jqs)NGW`R_rD4iFE8{_!&>k4DKY~XyOK6A++Dk$1w^x#H}X zZq5Aqvo~da^Oq|!zrQUnbHZN^p2lG9dF6XkZ6T*OR z1{}nFDeeK>n{mgwz?YWc4qxh*-iJHM0hPbfF>)W68r z_-^vB5IkTVd+G^)h0MRn*Lc!DVk*|K-&a{6rf-AZzS8`0_;5c4U+zcY)BOl!;+y*V z#t*}X{Gkl|z3l(RTCKI;Ld%}&1@*nN33ZEoA$#?{kj6Ywin4Nj<^!b-aAAk(L zjJgQo{xt5rkl)+Yvo34QBb`pa%wxTb^_VW~j3++-EqHNx(^~DN)Uz&=mRX}mUj8=h zjc1+yk1vs4^K(noTRzWvO#h1>WcdGLTluy}`7QQHr|bPjkMzqt(yj1FSL30c|Jb&2 zheuvsgMId5hqTPz^hodbn12X#Z`Cho8%X?pz`VHa_+w=@&LhwJJjy`J>>D2G|Jozn zcRkWY-vyZ+^T;LDl^*G7eBniZ*MrWBp4|nyi}g`|te!pKk>?Q}>9ov7d8DuK zNEi1=R|C3tT4q;z2=?Kt4CY$_a4))^j}3Uh|k$$a5y7eCE?gHIAt;-L1Mqer@{J<{p?b-%R-?dYAB;f)@7`)7<#FQu>kl}Gy19_h|_q5-RLFdn^>wt2xLJ>F6Dl&;SspYQW9-u+|eYok2!qV-4H+M7MvK^^a_9%b{SN4otU z>2&^@Z|o<$s2)1!5V`Sx;xXTkc%;+mw|Jyq>5*=YM>?Iqmd`)F3+?n?kGw@e=fxiD zG8B5GAK{U1lt;RMeHYs4G>^P|-|%LGw9F29q<`8Y-BFKpfAKEJ?1V>NIy};8nRR)j z@AgRd36FHNCjCy2J3i}?mkA!}w9F=Zq`%xFU4utDoxiqkkD(pC(=wdzk+(~ooBrJ6 zT=t(m()|i_UhM0y--Yq+e|Y5ilOE}Gy>IhK|By$zhdt7L9(3=tp6&6-%Pfy{T4u97 z(l>deTjY^W=dau9x6qE>X&HXNBX88lU&@@}w;t)wd!&2CBOPAme8*)L^(Zsy<1a;K zU-U@-C69Dp^+@+M(7n@o_I;1MP#=FOGNU=77y4U0(tX$?-N!-qPRp#rBQLpj<0~HZ zGdErc`{7m_gp^lA6c+?@qK=mdz4YNN4n7-=?Xp4nI7q0^H?8V`-T@hruR#a zbjLl?9q~x_1CMmy@<@lnT0GgjevfpY_ei(hBVDgYIvj50sT> z$qlR4bqdt6s3EKo@d*PvCJe(6{sOu0vC-?*wpJzcSC zL+8fi&21f>_pH75-gT=i`27;@oIYb#d&|t0m0jH(t8U{2R6^}-ovT__cCG_r?S|WL zS-YWYR!i6VwwBc!lk3~MtaKK(b+ui`kLG`9#iEuOE$tgqEi2cqr-WN(&S;t5Vv!DI zp``)kl;wI46r*!ZTXNOP_fA2Qt2$RD??#DGOUYI1-V@q}au2k=)p|ky<@Nw{6_e+10k8tKmJMzPGh?tY4M9 z?LAS=s5UN3Ze8+2WjeV6oe$mvb*PKu)>(e~o@v6mZn;2jRl%xLaoS=XuTzdA)JWv* zZ~r|E`R`%Kf3;!ZP}Q$#X~8%YL;1FKEe$PzZril7t)+AQ^oCSR*KN12+R)J0a{H#X zo%@l-MOl(LA11NfI6!=!~1VubaRWu zpry90Tiez-41sR59c{_3wOwmBZs4Jel?J7b!Q6Xlfco~`w3q_CAKru9>uP?EmBO6s zWBj}z=cy3izzgJlQ8eWF-{HA|FBFJ-rJE$7C~Dxopy!F4=l7}SY4to!P8M8EVA0PW z5((@7cky$0jB>6&$MaaRgU?=>|BWb86=)NRo_sEJ;hhl??=P~R%cnwolIiW|afQgJ zbTmskogd)2AAY3VD~qc}J)|&QD9>iP7v=Y))3qX;?Mc^$IVc1@zdSb)4J+B04Y|98!ES*V+^)NjX)vo;4sPuB{&?F~Sgd+4( zB)%xWKXv|RRQ|cT@<-?YnPJLL&vMHzRY&CNr1=C85sF}!A09sOIsEW$qgZmgx;wwZ)M&@k&*QHKrV1?-ZZHbF)4ysAZ)o zToPu!<0=d_7dlMVLx+i;!!a3`!d_gT=4un|2KiiN%DhJrca+vKzc^ndOa)hpBQAKn zIOu}MiSM}J3W1%hj`(B69v3`D^t<44@u&+PE&5#WD6!QAmx;6s9w|~TxKwnw;1Obt z3yz6a7hEDf=z@#IG8bGVmbl~LsEE7Zh#2jH!=l&)heX%~ zv$=D~Hz3l6IPWubeZeleV82NFoOrYg7wjW#bHryQAh_8Sq;tL|*jG5!eOQ?HkIf3> zrl>G)S%XB-<_M2p2Ua`j-alR5Pt29~{fI-)6IiUL`!aPuwpzmH)xG0(2^X*8@9K!+ zwelXv-QVK(nG0LRetfGZi|+1(x)&P~Zcz7S>fW#J2i5(Ax(m4fDSlkt6Y9Q9-OcF> zMFPTEhF>dw*h5K@6^SyGYa*Uuk*N>Ae*Ds~{rgt|2mbf30gOfK{rHJ>SYnuRg+Vs? zzpwwdB>>wRxRjq+dxg<}#+!z5FGj<|`<$X5okbc3z7?i9QHWyF~ zamQldLHrsj0pnK;T|Hlq-yG;N+Tp4)L()b;u-+IL9KzjFp77KQ*_kvISPHYDKCxE*jIQ-ISJnzMG z{LbUo@c>}_4&rwLKd}YjPa^)O@Oucq9r*PDhu^ZDh>u^#!?@G$vv~Gpq_ZEtZ-DMQ zi2ps+ zY&BLPC-_8s3M^qg;0F8}ed5H`K5=j^o-F{p#3x#pf({!d`7_s6?|U7 z#}s@)!Nt&Nicjf9efc6?VW$`MnW7(hPIyW$>Z=vSl@ca?@a+z9M-ds<{`Z4XlvjLA!WR@=tl-27i9ey>tqMM`;71iK zJ}%?$QE;(>zoX!|f{!S;LBY=}xT8hJKcnD2z{G#}7Rn#;QE;(>_bYg`f(6<&(G#A^ zmq6PinD7LX4iQY{4f$eX8QPtb{utUH!IU4#>x6IdJ#5&N%j#xE6puvvC`7zx9EIpM?tH@ttb8}3-{ z6QhyeR?ZKf#e%&fnE0joZTL703@$1k!D-krbXj7LqVH>!F!4+HV{MZDc?B1*m-*@T zolr2fPm)hs!LohBekiz=@nzz?f@S-Kd^gDSWP1&YRt0PQ?o+VV?}It`^EvqVMwOn@ z*JUmkdsSR8^v4AkiW4q4D2h9r>0##Jf`g(%!6YBjpMC|Ce5wAADVW+D$p@#XbA3{M zkp46%Sn5Mqv?}^3gX!Y0%&8r}1)eJ`es$Q|M=orx@sW%Afa}(J*_s(td=mRL?QaHlPQ_?NYaB zlu7h&s{0%2{<^v!mSn;J4bQFbl@Au?JlDRC!;}~DRuaGOuk7y_qKk5!h7AE4>UMR}DyCqzO+vA1e$$NrX@jSv&!~niA z<@23jSw9pKrz%Bn3iTwbN(g*F>B)QQvb=$d=qFM7jWWHhxSc8#Urq%BL>?3m6^T>z zVtYyiz6ZFY^?aT z6yre@|4#YA-<I#$WSM~&EqYTe9 z$$l~X^CGc5DFUR>9m`(NBRt82RH%=}3CXnwY1bW4}yCqF5c z$n?(pk)HYFJ>jfa3pCXS_{&d;B^19SF+g}Rz01r}eEtX6NExCEFgHZhPj4#7?UWvg zV@2XqDN3C#C=lpv7^%8oNYMqOg}GoP0l4G~VhPK8w3PoNDHs8eN_jPlM^fQ%+JcKj z?F0 z=}t+_|5p}@Z~j%*3{nT_1ATjq_~w@ZQwPbgDZcst0Vajg@FxM2LMeE(YQCHv%9ES^ zH=Ov>bG^E<@iZ8s@~fd_qWDnQbe7wD&V84i8g$oi9T z;0?I%*n-11Bc7)A9c-kk;`aSak@)qdfMAOhM)@HRYJUuK`>O}Pr6?+0r0-;Z-1&bD zIH!J*{lN0& zMd1Gg3B-{4NcKqd!6`x)+UFBpK?vyM13?4!<4CjEl?r9az6VRue*phTG8zPD%AP=f zr+k!R;zD}R53B!!4~6)d-Ua#12R-Epc~qoibM83L_7wI!M9o>Ihx-32>R;dij1u?{ ziQdg&fiPWcA9^=Oq&ygI=K9nGr2oB}gCJD&&p##VVXo!X%jN%Wr-)Mf2pAxHzC=`v zPBXd#9}FV>t4L`W_8k4m;>e3a>hov0edh8Hdm1H$()|(XSDaJCKlJC+--rts7gB%q zJnX?xe!@0Ci!Pa;TzzWN`bqVR_K=IG`ah`a{|w4EMYe9PKeSg@{Z(JQ{vJsMqg=mK zK3#unbo~V`Qh(Ckxa%+9u0QBYUn*iEIVnHXZ(k~$Z^`@ljl;<^NcI1xt6!9_SJ$uP z*REg1&ued6zkd`zx%KC+-^zEVez&Od>-z2Z)7CGNbJuTL<@avY@8OHo@9XdJ`h7a5 zep7$i`bBc?`aSqA*Dpqx(muaM`+f60UcdMt0heF5-}IlhevzEJelg;b{oT7{pRvC1 zw)Kk%kmP6B{`bGWeZC>5ez*Q<>lewn>o@T(*YBqHo%n}PZ4&uIRz4iH^5 z9(RndX}-qeYr`==svl;46bRV+=CJLWqy-%t}InxEBRKDz`-o0LDnIGjp63x6-c zM=>8y`FTEPl!9r%jhhXF=5vDPd+_}j~4f)#hNi0S{KE@u%k;=>dhjxEUo$q}m*bx~@A4m8`a#XAy%6oS znA<0_2!5ZuCuaL=9w&UEp#7E_-@7>q2MS;8fB8|jY+vjjKSlI;P#kq3J@S930xN$> z$o7{UsRzDDmRI>xp6DtL$A6N|7E}9w{#R7KJW4N{Fa5_mlll0x24Lk+w*5)4hZs-p zN)-eu2~Tbp_{*N!gl(Gr=uk=k!#}uq(}nQ+n87~y13k=FBP&&5*j^gB{-eHRxEM7j z`zw^!Wgn>gu!mXz)Sk3I39Dp~hfh2K|2MTi*k@ipfWI_KO)=XcVfdr+;eL|obNM1{ zFTVg!sv^e<%<6??u8czuVtL{)kKQH(fe^7qf?u zUzGL7TAv9Ny5XSIFNM!dZ+l9FNdG&QVf_mI*MrGW$kP93Q9dJ>N3b04%k`ZlnjZSM zc+=;?I0F|K{G;cnJw$bTQtctnrT@K~3+?s+`7Z9BEoki%{%iJUF5Z+5IMdse3a~z~ zzJnh2MR|Ckm4L~gDUPV|3gg4JJp#NdRg^IdRXYEp1C_|x)*KA9qHm!~f^ zgL`-zKRtZGMc)vf<$!mkf?S@UE>CJUu?%*{TA2Ud zS|(z8%1vDiUoGOImsEn|_ol8?NDW^l8bt3JehB)$)U+^lC%QblQd3;@OY)`_gzW3m zK0KVNhI%^TsTkkbaBpg&NF=C*;zIh1HysY&tnJ^c^+Wu1x%%Imy37^7H#NZlSJHZB zegl%`_Ax-|JJxSl-)pt`3k%?(AGN&U%d*E$+WZV-k8}A!e^A8jSLxf%WLcrrAG|>N zJj%BJGQTqN-`n=T|B~{1sykc8{fB7f_VKpsXOJ(AzthyClfN754~w_0Q0157ZITyt zy%@i%_1~&8>Ylj1&y)PVp!!c~FODV)Tazj3e65p3Aw3&`8pEccaT{K)dr<9R{j zd8Ci}f;wlj1s5cLSg$u^=E^=Li;FL({FMDIPhtHDFzk_RFBm_>Wdu(DXfmS4_lOVv zjwY$JbjkD{Nd|*c0daa*Pk1CXo+k`C{YO%xc>rd^F676w8kmHU{==y#GVqDK z7bX8BziTc6wvNtdFWZkEBYu|7ASZ+y1p%1SH4B5!%mi?h_&$oLwvm zXvIy!72~fV81*mVF{5X>;L_NYHhi{IRFOX5$MG%wqW+58n3 zOP?8zjj{CEC*1lB__JcBt*_XRrq>f8BWi!SuzNmy{}4R%C1?DXpU?e?uD?_~o*?-X z%=*K=0!u#;U-mx=pP!QNqro0i`!PdF#3kP7LjQL(3A@A>?xQ`hc-NBw~#xUn-*gL){6=zm?bL&EEUWtoRO?rAo$lt`Ycy2nyOCf%WLdYSAsr zEB!f*m`@%AcZ@&E{-0tV&k#NKE0v_Mo9h)2!~rA!#%he^bb6ql2>%P)fA$a6WBzN( z`S&=?fAj6f6BR{RFM zKtG1)VJ~B;a+Fu$0aJRgzljkR zel*JK#zUPFuX z_Eh(uJpKZHF62X3w)GI^ktDyd$Q>}XABGLQMMyB~vssf8V=qr6^F&Dbb8Au|eD(k|sy&B0m31YXvd2LALLnXxQu&a7SxWeecM&}< z?3X0|&_L<(QTb6`gU1(?KKP3zO?E(MXUq7pl%HO|5ux!ChW*1x1~zY&qz|k9oKrl~ zjPxd(#0vYzPZt%}q$c?tu!tuM63v&cgo;n|ce)Y^o`B{{S3%N@`B`3mMSs{CL1I1fe^tw=ivCJK`>vCFZOdzrnF4j_J#RZ zSt=MIJhySC59JX|GNkns&`&Zo{TV;O-D3~_p2jd7e+vd{M9K*u{Y!-(*5lnY(!hAn zwcq!N!kqDKv)bSJ089@4)Ob-gNa;@~Pla)w1=eWP{F>%p#J`?@x%RiCycoaQ{zSkv z@b?}hdTsAg*>19jZ2y_w;jaL^rJGKfxL|Mvwb$|fbdqI9afeCvdP{dcUXfRLtT*(f z@&Y_wwZ`MY!daBQ1qTXeJK!zd#bFUw@v;7Z^?^KUaM?W=??T?#pI4aB^vn;ABX3aj z)tKKWL-}m)wLdKx%%>i~@~JzIhtZ$f~7cK#8 z%PUxTlLIDtvIp433i=Sz7uufT2hti`f!s+3iR+N{T6X%cqaX5Lj zYCM+}w*{E~k^!&H7P4=Vo$0qgSX@=Ez|d3nCYDWbga zCl`LyLC^Y_(E0)YwNoC4FRm|cc|=$qD|PxTkLVEnSf}w)9>rF9r9PC{Fv-Wj0-d@% z%A>=Pey%*I{A(O|YA;<582T~7%I|QIkLuIeUby`&)AAzu>o4YNGTEZ89rsgJgP$oda(eWS@*>1%leRC^%(O*`-?|0pZ{ z!GC_}v(lRWpk(z}%QQskrZP=E4v_H>UUeO-R3FJ#|>g->cc+eb(L zf&R*i{(noyXZz=9Z%^+)e7C*i_SdK6gZ`e%|2;>1;*Tz^KhO_H|B}<5xW93R0lrw%eDq zx6~gx+Yigf(SDktk62#auj033FNm`}owDU64eJNmht?NqU!{DGIQXafd&U8~ z+QSiP&wy6_JL&)1+aGs-(M$l!V=*8K0Q4`4Z!_G1Y2E1>878Ryk{cs1-Z*3W2$ zs@G>1Z#o^G#xs<@JgiUna@SXK;g|4swe+A>ghlmu$Rn7??T7YXAwODAcna*k)qYjN z{}0Avy#ewtHs`z60~9+4vCq4{g}b^KlzC*nexoVdl?>9 z@TM4)^Z(O6_*cbj^3F>Caizij;56XZr9U)RzvTO>wmvV8fS512UcIH;m@6oMVhH`4 zwZCy9;MIlbzf9oKWC_0)`xEcObovMsoGMk0AME|i(2qz8Z|aE?2|?@O(5Fa>-bV*) zZ1D^DXX7a&FyWv72`UW|%;x|b*Y5q!Jnf&W>LU6yU|hiG!9Og->pv7H;Gf3&Tzsnm zSV;P3;56vO-dLg~{xF}2_tFP$=t>|y^tp>bl^*0N(@X!En9u%nQ^6<;(#QLp@zG2_ zSH{=*rEk=5xYIB5BYj;nV;Xl{;rP;qF`L*N6lLq?}WqS4SM;jzC zTnQ-;`175xsq>4+IX~C_5#$GdzCk;#vJI$zly73niGu5dQ`8 zZy5$TWO;oA{FJ5qh7}*@55&Nqwf;f&Dv8e#QTA9BB6~;cClu8CSB#`7*A9xHbD*d4 z5`A_r)-RwR;LlDU`PCr52Et4Kn%S6|!sie2dMfCt|DgEXe}JCsXPV&R4&Fc9*b92V z_WJA8DYRc>4yA|qDBl!Hf2s`|alpJD0=!;-W%)z@u2cOJ%l91CqUrr?x|X9nWZ#p5 z4IJ->hCYYxHhI4RD0sc75&NAvh9CI@4}OEGXTLYvg+G z8+yGrF4ubr=JnnRqb?3|f@^8LceGsZC3uZm|4k@7!^KA3GAE4n;#Mb&^<;>S)2pWX zgM9-yk^G~Fn*qa~0j}ZmNVvRQ9F{-Vx1^s$c*qa(VAI;M0*NwS#-D8CWqb=SVJ*!ZJt{IKzhZTztDOKkkG@nbfA z*!UxC{IK!!ZTztD3vB$b@e6JIul1J(9h2woeJ$ z@zFmojvN#hU@v*UHSC?WpGfs5HL0s=ocj&RQ;l)54;cS*d#_2&#K^XQ}W3m%}chb0u?D>{LteFXwvEI?f8)u(6%IW3^-p)hC6~G3(o#1(V!Rt3HOdW_ zmAT>knQl1mDmNUR;f5nuy5Yjh-Ecv@8y+*p4VRa@;c?^Ka7Be1j>p|_Wu+Ugu6Dy! zRc?6FBsV;9q8pw(*$vm!IAFA|v0>@}?fnuXaUha{VbA6!LvsI=wO=AoO!E(WKZO~i z4MR5U&)J`0jDo#!q(32_{2Mm?_$bZgZ8%s61LlAuV_<(AaCjW-jRVfJ_jf?Q<|d=n zus05T!6cfh+v(>|u5rL+sSyF1|B;SL|AWc%L6#B9UzRGKKzQDigY?T%C1VMuJtvYr z=GyO3ve`d{(&GKr96yk=zawbx??C&4|AO~l@uZUXgMc+Q&=X zKaxxTw);uoe{i2)R_XR9f}f+we0x6##!K*b=U+f+^m+*V&H1mp;7~r>V{89LBp+|g zTJ-SGWX*%@KMIRl^4Em3<}n4YAviy4o=`CSHSq82{SqemXF^H5|4QN{6#;+nU&Fy; z!$*^OvN`elA+#P~==}~R`Ae`~naB3p+V7zJmlhnz^zS>`|jQZ04bE;q3f8dhG#mfJ1GQb~+*86$1{N#E7eCbK}zb$>)0oc^*AG*AH zeI5Jl>bZPWUcEk`>wEb1Xpx_ThTp{42CSCt%Yjyabf&9TPei-ju zub2z|!eQ$?3)BbRpOgI3`UUz2@>h~;7XCekvA)6YV+QE`Ym6`X{MFw8Pv;x9lYbe_ z7R@UlEVPzSH@Xe%Ja2y&q5Z1sAWExb! zJ-jM_{fR;9kN7&(T->FX2Jo#2xD8Ljhqx8f1{2Nyp`pWa7L5IJ3(f9M_wK;IZo zU!+$T>Ko^6%JCJyU%JDc9`eWgzai!SWqJ0d@P4H-Ae7z~oSzveMSAl5Oj-})^&o!V z81=Uc?~ikTqt?fEC4=()DuU5}#5N1o2h1NJtY(p~;Ev@eKl>*>sF;=G2hI;ODw$s) z$5-2zV@Fr^7Ggw`PJ}0IQehn^+hLrj(mW3(qn(QFt+OQY@_|=!o&vzWceZt8~A{O z6E^X|04HqVg91+2eBOqw@duR=^8uH9YR0?mA?n}qkCHx|bG+XudRbpc-*~-Bo>v8X z9F+2-^bx-o=kHSa_ySDpk!b*|kI!!O^*u_jn>x2hJ;4 zrm~myX|1QL{+0SWNJn0ne^GrVEI9Xk@OrctoNYk;BefUqUxNW^3f#XvyMxaI=l&y% z*QlPzJ-@pd{e7v;|BJFexAQL&Z+#v((u4j||G?uP>Oa6Q&I6D9-#ZVy682v8ha}JH z!i$^_{`ZitWBdYry9xbQ8d3Rzez?Xf$18`N4{p`pM8s&P@t5O$;b&<*9Md^{{uRC# z!SDNi7aBw3SG=!5`jkenqm|z$HlDzFQ`Eomc|{lxJZR0o)A`h%LC@>ItbgDa<4gVm z1^Zh7>-A-m-@mIzeRR3%w_NK#^SjiSkMql@%TxUo#@m?Quf+rPy#pR^Kd$(_68iBO z;P-0&u^zfre*Zwf?}G0S1gSi7KG96)`Kt<{{pSHX&xk*Wz<7)gfG0`dlKazZ&|iIz zKNzL!D%c4J~H zxP;&^%cm&d;t&3$p6*Dn4tPQ5f$@5HI9Qif@E&>}S^$p>2w#i!@NK4P(bwhy9|XO^ zS5y52I6qZC2K-kzUuc=!FLUMr@@ECYX$>!6Scvy2yx+I=0lZJXpU;cOd=cM20DUm6 z@EG5??b#0GkM(!re;lpXhXCV!ZL%l~`v7BoU&I}FtpDSL92-yj0}>8=E#PGmZol~X z0IvB3?1`H{Iexd!zi;2*%RLWn@up=YU0l+hV*I20b@DuW`+Pm=ujBK-r~{-6;{zKn z&)>7o1E574v>!f?55;ABdE_zi)eOeF=r5oXc>mugo_rAc{Y&7L{`YN-nADwE^W$ww z{pvx3@pbQZcsdZo#r>=k5 zeaHSCqW0P64d7j0L{RHN*w6C`r9Wuz;6Ika06Psl?SDi6i1({0fwbNahx5Z^|Ha?K zp!1E~=Y665Vh1MT(|LFpfAjuMQV8pOrM|HI07&1k=iOodoOBPczry)_2I+rxjW3{| zH2z;kG1T~fOE*oIQ9;~aaC|lYf&Qa>@aF=L3L-r06W(tC41p1Ry3jrk10a=uOLr9b zxWvyu|JECj@3*S+iEus;e-4P}3wR$#t{1BHJj5?gVSNBe>G{M2oR>sN^A!Dbkx>r+ z5MLAVALD}wNuxk11dm6t9})6*rsogZ>D3cGpEt#E4imp7_^p-vuB7z1<-!Ak3-SVg z=wDQNkO%Ok+8hub`^7?5`Vv3V&5wcMkt=>#PJHPfqh9RkW=dbJw^{k4a$NB%a^hnL z7AQDA^QYFstoZ1DUGc}~#FyuLaQq#Vzgi!&;v;!i{CG}$`F^pTzgj=D;$!*B6~8(s zK3N9MzYqDV^-?Q7G|d&iCMUkV|3J!rVvhVVeQ?E}loOxd*O&6Er}C@yJS%@RPgnff zocMG=mgb-0^LjKMY5jqIx#CZ?;$!>;dbJ)4dW^5O-4E_7$(f2j=p*(wOI@V&uzzyf z{QxWJ^=J!k71zdNEcdqfgadCUFXL1GaVOp;K>nC7p(!r+2gpB$^)!s{x@l^l$9Jy& z2JJ)k*M~uYUKjoK;_}O>B)H`Kgx8Od9$I-=@Olx}kBSuxdrkYj$v*Oh^?AU`p5h7Q zM=+@#UkX&7v13xARA%=^NB6nHfK%bhukoqEX4Ry`mYV%&rm7gf1)Xo zsQUx?pQWw4ZIKF%~vBKauvI zV*L*1W08X(EyvS5{|NK`QaWD`@!=1X>sbr5KdeK%f%FiC&*SXrj>+-IAJD(^__-J` z(kA!}ji(C%L#GL@rSU*MV6;Gj+sS^*?-7mQ@;75Ya8U386;w!R-+8^Dgz-Cgzc8)G zvwRpY`|~0xKOgNErl|Z6_K3ziRDOBhFvcfn&$Rxo;v4$BmI8_|>3KZX5x>ji{kGUI z!|~&HaeY|n5sdw}LCBjgAM63-kM>B`Cae9Ii?_{E1hT!+eEakKM8diH`V;j%CA~iw z^5^wfq^IjcuD`xd!k9m@{~Jm2{P97o|ETttPxVLZA*y}K@hjF#lm@ErKZPm&CDeb~ z^*yk~-7ks@j`GA)G3@6O^Mpun{l!zIWS$-H2nfmvm%!FJVG+A_@2rF{5-PsG)D0UW z9I#okxze70K_9}M%34vSwPTo-Digiu)3dd{V1EuXsQ5FdchT!@>3PzBOY!}rojfF? z@#>cDqDqX|_K%4YyZkhM8?%J`LwsHi{LPE79??KG zV2v;Hk-v%$dqexV_#*kB|A?h3mi$V`$9N3qPc3;$!@OUA{F299@TetmL{;N$?B^f7 zB;kU~m$Yga<*(`zsF3tLJVJi;*dJw<$`yqKx3{of&mb*XX%aoeuj!jX?;jx#tN}gJ zn}GF3P#$EP3VRmr^@JQ3mj`VblO8&3vH*j}mj zFriBX?ELDHo(NF-L{9B>LYEPs^cgWQ3Lw_laFIdNrKF+gJ`j?jS>N0~Q4~`E!<`-lR(<5~JE!O&mdv>RV54L{wE3hljrMIF-~9717C&LML}=HUxNNyju%0X^uZsm|LXBB;0Vc% zzt;->mL&0QXD*JU$Nm>SUx^>d_>AWGpnoPA;HuSlyl<-JZ)xt&0rS>4#>1cKe*a32 zFDL=-@6rE*Uba0rFlju1^f@a3WB*8*?0=z}gl|lhmXZC|>s=^M=@L$lc)czXTUXVZA zH!dIQ2Y7XWS~{regZ&E~37khHJ!@! z(w{u3Pjb9Lu&giWn=G%?znWLcKHK_NgM}2Q{?)v~_3cj;XRQLC=K9e3$oAdVN5T5y z&_{*$D}9uB;z!mu_(6JrZT$m({?rIpe!%m78>jwp{F(uNf6j_;)sJI7&+S9aZ)jR8 z+Y6^>*PmbOV?fEr=11ul*=vVBNc>ypr}R~}FUTV|Kc#=b%lf#){G_~W{tY|*VfD}N z(jSs1#r3X#BVF+uQ{j<0`WI&XtHH;DATQfqL7!RP0cD>gjQqo_pEVt9p8)guG#$$^ zUIzaTeNypFtv}lSgjt_!us*=~S@uoQhje{f^ip5WQ~M`e~*Uj74*|(&(L0BCd%mb5Ptua&)+m-Za6a1f)O9{BU8NI9AYQM8R}mh z`Uw5l(v9Pl9e9^~fYFfAWW)pb7jz z9uK3x6NPg8ZuNJ{o^pQ-{z?KCe<*)r(xCS_()|7l@<)6|e=~{u9za)jyzokWUHhFHIO&ekdQ_*HQj{#0N~{Z%bZ)b$a-|4)>?w zWS~&>4{ZMt-`>9qdP+a6(+7We-#HHop4RfC^2q)`&M(3^Uqz;u=JtpBvhzP|=O4Ap zBkh44KT-LjvOkpl3*}D(LZ^Pn{N(&RoHPsNfI*j6*!ht>XB4=`ODHd&PlqJAz8jN% zoK|a>2j%DEH-Kk(G=d+5)4IH2BcI<7!87iUX#F0Ngguu1_qI)eKv3%k=JSLnd2}pO z{W)ObSG^BL_Drp>O8ITul_@bTgCN9a5W=<9E0%Q*q~vHVTs zFTnl}1MjaOgbP~Tj`gfA$~`@gH=!Xu=8HUjG>HW9CnWv+z6&*R2?yAq^CmZm=l+4q z0DW8hV{J~-e(_>G{9AeA*cQYkrzhqo3%l2eS12uUkoS`rNmY7<&-FJp>HVu}{~75M z#nAIBoL_C9uY&w({j!geC;AuYeIaapUO^9VVf}P*gsd%Hz@N)qKQ5&G4VBTf-tU0@ zg@8qef@wVi`(JtdPX5kcu)jS{%b#+;r+>(PgWUadx#g4VA3j6G!4lT98BWt|6@OR{ zajnm~_J3)A_XpK_h4gn@=}~;h!?m7_{bIa+O#9KS_2c~TSrP}oA29TLXR4P3|5jaQOghc^Lb~wJb=g2`OLf@3*~`-eXJTFv40)#7zxAXfd4OJ|C256 zN_?NLH24}As!;Vo{&?xn0KeepiByE~;QJ@!AQahCs6V9tApAEnKkx_s@x8oIE8%s17!Oj5WqHf`lp#-pm5TCw5a%7RJa?t?Xu|}p z2P?QBMCE4~{FiJlT0-*G{Pm@Z3)09=*MDECsDPTb4Hp)qiC(>bhxhjY(S`oy7xh$s z`83{&68|Va;0UFU^YR(qrC`n<>v;yg-^TTUu0xz^;r%dFUmf&48s2YufwIR%_NEf; zQJqh&zSl_KzhU`yD0yLh!@3}k-V}`n&r=g;`$XT%Qyy}bM_(!u!**}0JiVL47>Mwt z@FhXAx4OLU{hu_xHwxn*FXH`sY=3xr4AW!$h3|dE1Z6<_2!3i(va{#)Na*voHi^+; zV!4>=%V+Ip#P=%MABgj`1eY(4@}a&OQ`m1reXGP{{|}BYa>3CY7{e_WKHmvLpTg3& z%KTyfCgg*fq{}XU+%7-fuM5)lIWFI`IsBjY4VxaXKT>+APgi*ry)AEH^mJ)et_VpxJkzk5gZmv zReYQ$=c+Hf@8_zoBh5rl8`x!j=H8!IYZRmb$e?}6&5A{>-j)ms-S>sRO9=Acf#loLTtZP z{|oyCBjcpU_96{Ke;02$VSoP!<1@q`PyHX;AMgi#=hqoIKiD(sKP`O)ym(WC9UtSV z{G9oP%U(Q>{nWHyr62n*dB4ig{iSj|)g4Q0t}eFY$Z{B$!{$bIhMG|3djK`$YTc z;9tW285lSz?ayeO*l^fHW}wA^B5s93S%^OCH782tocV=Xf6Dogn7ReEA;Y z{=wK9czccZp&)*|@jK63saDRaH;F1dYy(-xs=Zue${#Lj{%!Eo0_ORqhxj$a_->&%v z^vAVdwY`w`XWRQ9k2drE>Lr@LQ=y#s2F}OZCO^L}=Nq9R?KOA4vGoAvN6WN+LqDE0 z?EMJX-?}A(!tPi6*FrvDLiP3O@{|urKi59U_OV55eKdZ#*Y@|qUM}9W z&L?Lqa(*=Ie5e@ZEx|UNl|=uu$Uh-{^}e*xdmqN{-$RR}@m~8-`-t=DM%NKJf4@7= zz<%R<31odj{uqxfwqVEu?Qt$I#?vru&$EEBwoW@N30_kNI3M`9gyHXi{}FfvpDM(7 zWhuYk$NP^D)A+^T4J?cp=|90$5T`yrs2Teo#sOyeu)n7!H4(#4jV}Ve60j3r3v=o` zUr1Czeu;^EK2@a;=gnX`<%}N(y^4?g>3dsfPKjDGt@4|U;iIOnhd&N4mFCcc|HKrF z-gW*F_#2P-PW~pKCUkl-zZ%4!tl~@gVn9UlM0>~n5e#Qle8`LVgL6cle}wv|Nmavr zXs3t$DoEd5o(j-Up!88b`16*M!_I$1$~%@S?8IY9DXFpTriQU9tuYjA#ON<_Z4cPexw!S=Bx)qs*}{&zrL zmjQOls~*E^ji>SIY(I}bNng19SHSRDc*LIydT0EZ7%p0Plz$rNt?~h$ZHQSy%vSv4 zdwEc1C;jD!?@aG1xaXYs`40W8G;2~f+WHH5)ue97(O3F9pfkPeZGA;~VyWwL^!e(T zJHHvo-x>cVr#>V8B3qwXKC>_(ck(+O={d{4(526@R3qq}=`X@pCYtvhb*1vl}2TS--K=HNY$S!*V|(p}Xn)Jgk3AP6bT( z1El>xdFi}K>i-lT@(A$;9M*4^hZ#t}na!FsUS)Ylmy-T!JkIZu`!VVKF5u%ydKrMO z1h21?zm6b24+#BdEY%F|RD9Go+DpS_Ag(8S13b<%y?-$XFvMdzta2Rw^llqcL-DhmWH zB>O_`dm-=(RD7&AG*bb!JR->wc^;T*&qSZVUz4Jri}S0gKg0anAbxTFT8XoL6FoKd zd5WI~Ngo$Yh(z>BV>Hzlrwaav{#pQYRQlMT1_Pz=rz`0^H?CinKjA~00u?`WqZf>>kKmvSiTQK_y9gg4}1#x0se5F?oy{b;wk?~ zt9|ZDnR%*IKPHuA^B&ee1NC( zxzV7l^QUM%h7N!Tb;FSwHyobghC}H7=t^+@kpE-`Ew}*P9bE~9&!5Dg1s9XM5m!RtizYB=!6Rh< zWx=s(2cGn2wxth{KlGo++u4q|`ktL!uF>_4`d9uWjhFkYYK%BbufG?l%M*Y(rYoW9 z+iYOaf^)ww8JXe6=Y9_|594*Z5-Ppu6%1N%3CuZN35747$)E*?V9x1EkbKZS3Sge; zN+|mLs~NQ5qDD7dh|vmN38F`N<$32UPt=bGeN*34l=?{LhgeJzo0*+Q-7duK?_EH_?|AzrBfdQ z^I1Gfe)wK;jy{<;I`PmibfChw7_{IJ5OgIJKDdBE3(iCLMpr`NqtvtEN~ro7 zslHFC^)a%D(H6c`ea}+k!`Hj*&ibR80<3! zqBdv|RylBwzJD8JyGV7ekJfFnz$?hS$$uc-BA&QSp^88}5$p%7v9EIh@JlnBl zi0SwH-KzS&s_Jj1XQYucWA|qZ>303Qb?er>w{G34vhof6rFu{LgY&*3AI*CcXbI`bX-~tu;{Ke(eKx>n`_^~F_;5dBH|sa{HT`20?ed-`>|fe1RM2}# zPx#+(AEJ$Kkom&+ZKhwxbmYhVUJ_p?{wNvGe3W7RavpgI>y4Z^{F3t$#-C^CH;Vf} zuwcT6`cS?D{U^J~hxYsT734k$Tfetojq&6Ce^eOauZ4dP_gTn(ti0a|{BR$YzAq%c zudvWVg^>7@i#^*sxU+{7Ski&d(Q0>7`HD?P`?wDgM-ipJvVS*P8&FyRp+E3Hru(zT zQ36zYiT;{WR)Jr@eYpPs`yG3GvEO3gM-VIGz2^!4D8f)VUP3RSzmA`@{=jg&;C~f; zAw_^M?kkl2N`cWoobSoS?ZLmu@gRSj^&R#L3Tk}}!(X}2X%F^3#$U&8&>#FH$16(w zjcD)C-epPsw=Ee$=_+kuqfgWIg3%@{q$UFAiPQX#}ej)m6@2AxFNy__T zJ=@V0yPtvkO7;9O^C6rcxIYp1cO?7_ddPVTAu04d=xOh}B)#w+T4&EKwtmn9`1Hl} zemc3IvRm#y(eEX~Ab`75--lxFzqI#%^8U{2ocfp!0S_CL`E`G&P~G|I&OesF&87n% z&QIE}kUz4Y%Kp3gfv>$E6!Ay;e$EK_^?j?nZ?w>}%hs17PnzkK*Zv8Vp?wYdIqSV6 zq@TmTGJahNy^{7dz4U#r3BCG)`%yJa`dEB9f3<-u@`vM*@UxjO8;Bd_E$+v#{G#d4 z`JYqT;I-wQ`GfwWKZ$%6{?B{!gYrTf)F_YqkZ<{zf3Y9I`AK;#?PI?N_$B&9U(4MDXkq-Tk(DMbHliKI*AJPBX z!P{)`$@q!*v7 ziF|mH^mXL%?F#AUzOmSTDE8cG+Y|nB*7%4=+>8vT7&Ct#ps;oGyQufvl+lvj7# z`lSEaM4oH@N#yBMJ7vCdzU%vg_56-jJ5Q#jQJ3bKk!TRg^Z6%EI_%KMCBC@Tc%=KB2Ghi>qHaznuAp^N^rdB2P3wCG!vcv-qUz z$^@`?V^B+af@>sxRWTkP@f*>;`;z@*v={BWHi4mkpdat!{MP!B`z66*v zaQh8kUE-M!?OoBx9kc!K#eI4kllW8UcXOUu$A5r-@xG3Z2QdEj(b~}Us2QKOFTU_m zgu8wapv12-eh=>H3e)j{K9l?2DDDG55}!6%yG+%Y#*gVWxF2KMAbZJ zLViI1PgLn`;D^8e?rVHct?PO~LVv_tPjudbj`1UXuZF%;T{%FqKKKgt5da%6fq0h_ z)pg8_5A{iWbXoXoT^t|1Ze1J?f%y&jTIP6L`S!Wm;5t7(+KtyGKMhKJG~P!u@zKR3 zUKjnt`dTg^Xe9B`TArHtRm4Z<qa59e*)Z8y<|?7yen5@Q=hh=Z9CJyrl>5LjZ=s5BUMFrbtNQ zo%5?{f8#Up&iO$N*G;^$A774m-aO--*9kos|GYxrGGXw87jLfHllNI|JTvICf)1x7 z{#oA-BlmwJ|0nUDtl<}Z-$`wS4R5=m(hb?7IO={4>WR zj$gO&&K*th?Lwb8-WlmE&v;<)C;3&{J{`wE zFT5{g;-6uT+W2S4lgWL{%>2;vpZEVzKEgjK`FTG&r?ZU@NBpzw-(i18bAe=k01Zc@hwRFv&3(s|A6JZz%up+ zF!0mo2iQL7747SWt%gtHpS8S^`M~=GAP+{1D{y|4Gg!;ViOw~O@<7W6#zPn6{y4d> z5BNM>>sUU7ROv7D3(j8@6z;E1&R;xJ{LfNe%YV#YUEan|?^~nsiRCN3t>#1hVLUbL z$ATX}-6f%OD52#u;)~6E5g7K3IIJu^J$-}q9`y5?I9?k3iFo0)ZhSQ3DMf+`y~O|A zT}ERPFAe$*!}zQNw(-^We%Jx}H!ise4e-O_M`bq!GE?Tq(R zbBe!)5$nZ6_ozh}e_b%~*Pn!Zu{w^?4cGmn8Djn~&gE5B@s&ciFO-e~>;} z>+Y^dby_~n8gGsK&@anG|HSdxxDE&$V&wtkFCtu%`7m+cV@Z7Vi&B0A^>q@To#*pE zF+Wf4Tb0Ng@b|=hxuiVfvug<5V0p$z)80DVWBgCro<;gKtS`354r93vSj$_=mqHiM z8{a4KA49k=OS|ucsOVq+zD{kgCGthi4_W*DI&+`3%Dk54e#(g|x5xO=exv+8_uL*p zR$o0>Ed}zuKaua>5&2$9QzSEfqv)@?~ji^fhj$c^FLFyD@(DyLjNAFT_qJ* zdY`D$JI)XFeSb(zeMgk9<}a;}Hg01R=Dbcw&xy}#Zn5+de1{FbuwU?g*ev)$Uc+DK z<6GQDf+zE%Gr^zz!+9OEL0sf{PS`v8JuYMKc=pNnx1{!m7vAaqG@mD(csQj$Bc?x( zg#FnP(0_*hlozT$J6rUp`F_la!<*4res=u|e%fsM^8)+h@)PlQ`ROAm`!lEFQ9+JxDOBPh&2@C!?(nK=+*1vm*u?cF!&FW0hdPU^h2GvuF9FOlihGyE;?EX)7W-?1c>Xr-Kho!IfAw`!oh0@b zVfXwSVfXwS;rRR;{MUmBN3ik>{)XXI*(~vwctqk4nE0P(@n6*=`#-&eU)ZSzN3v&z z{}Ax0IQSDDFDLyWyu;dGurCQ;yW6B=|NKHcH)iD(_-*ncxR34l!(TT%l9K?ltl+;z zVD@Lw_Ge^3_9NteT-e`(+j3{tpDo>RI63_xY|r;$e1Dg4Y)=~4u_q5*Sxn>==g+o* zgnw~9cJ#7UYn}aiHN34@AF4?neA!-y~NpzR;}H~;MGQrhYVp0_p@n&9(bb*qW*g1kX~tXB)My#o9u4u|#%#>>h- zjMs_k>jV9-rH3?1d4c^(U;kg4tgpphdA(GS*ejQt3TUrvN!2G?nM3TAx#)M`Yv=nT zp?)WRq8&7i1Ybs{y7m>=4GKcrE5Il858(dgx-N}ksrHJ~pPi}wDG~+gPl|qTvcJ$D zJO5t@`?GUy{V6Y0e~z^1PqV(C_(Rba=&-d{ocX_=>j&4K{My|5gDMwhe6XLGVjm~t z)7dm%!};giFW=4_>h$Le{bMp=#NPvZ9R9v|y$5;OU+wN@0DI#$>YKoyf%MQm_$2Pv zW_f%vpJ7iV{td_EisaL}3Gke9=&H2zwj1AoA= zJ}3UG1|0l{)CH>s21pOl@27+R5b*W&`9Pr02LgRQ;Olc|J`?{DjenxQHU6T%iT_A| z|A@w4^!e%FzmWQy_|K-Fi9hsn!atz@HjTg5*QdchXGvdM|57-=Lm2wHH}+SEzTQ>* zET@Ou2ec6Sx-ZbzBcZwzAlIQdYw@+52B{#?*TXIKb$LPcb<6d2IndV${zhLjenGya zvA!;=vZvq9pg%L{>)2ng5c+x!AlmX)b&c^zBRj9A$xxTjHln`p7zl^#i&H{)2 z@WEs~C;sr&QbD4x*R<5vs}@vWw_Lxi3iNe?ztPv)zk~jsuD*`fZwuiMpF@Ajp1wYV z{>-4Sv^uPd4CJPx$5iwWc?%h`W&es(bsYvp{{R+{p%9P`^UF`T?+Jd zg1=o~%YGhkI$eDouiq9zU$Zjv34Trc@uimQ>pAyl27Mj(XCd_U9Qxzy>oXXi8T57B zpVQIT2&Z@AJ#hY7?z6V>VTh;m_OBnPsU&_5`_*{AAlRSowdY@pjJHF4+Op*sZo5AX z?^GV-(B}^jFBajwiFm&p`NeyMcpr%`L;Pw#_PgW#a-@%PZZQ^*{%)R!F!uuk?qhow z6ibTk%+uemmh1J*el_k-?*km~R~tBvKQpk6KYIb8Ww_7#8niF>IRl?VgNWC(@hP9< z{V*e}FGo)hqou{r&5FZ?b<4e1`y!+ju7f$MI+ej^oj? zh(`ndHXeoekKp({P}KX^z<)$ZJYBMX9pFELa6SBV`uo?oPBD%@1AbcoPdfM$j^ocv zdK`ZS{I7K4Wr_bb9Urgb%YgqjgfH6oGTFaIe4|VMZTfzK)8D_w`Pw+%4ES9Q_{$Fd zgyVQKlOD&L1@TYdCmU}z*Y)*8=PLCK4`ekC%K2CDXBYREgg@oJiLNf~e#Lyk`NE0L zL+VK#$AkQi_U%aK?}_Ao>LTx_MmpkqcW{17dpy7J@B?ZyVEI6A zuYR5Xdg3GT^}vVineq~^%Xk_b!Pe(NpAPodwxWjo|F6DmZU=-dH3g| z{(;(EcTr^7`^yGuKl3wL_;2>|e&i>4J{sq-S-*)F=KHkb-XZ?D-vsxMsp|0hTYeDdk;#v? z|D(0Ht$5y+2Yqn=?ye;NPUwS`;(oyUI3oXfA1}-24E@mm8&+BU0xBL0s{z4wtZdfsG@e%qBuBga_(c{-&9SHmPR=2+X>N)!UgBahE;G2!{ z&A-jfKUnKa`Xl(3j`rB!aCjE=_eJ+N6{ng$q%`=NX z?%V5k+v~3`llOg+bT#0nqy8Io{klIm&x!Y@(lwcKcq(u_mSZ<^vaEX3HjNeIiG?2JkF2y_4p6gR*JnX_m4~e z)Uw@3kK22}-fw2-Z%?mVKkL)&e@FOv^bw@90n^?357+h->PWTpLwQ=L zz(>yO0Dru1@8FMgxesg4&4ypUjC2+-^$*wX=`8`W`g5S#C+KA^nm8Lm}}b)?(& zfKP{m5AK)#P}WZ`6`XYNXMdktKk?Q5-DCKn&ubs&r}5r)On;Q`yXPK1|2cUtf33UA!6!F* z6fo+u^u>Ej%baw)$NEcszvQG3)cX7V`tOeFNVoJJokV#j|7dlO;BV{4c!*Lj8^3{SDe}`l*zK1eP)<6=uWwli zty}&YtycL&EIy47_`@6jf!aqt;?u()|K{;sHsIFZOwYTz>S)c-6ZTs;{y*Qz`yy5t z{V;&@$$tCqk@xg&vii6Bp}r3heS`mK_3qxg9T?-$`~ZbkfrskJ0K!KBCX(TYC0-^c<~zxc|dWd&792&!^Y2V7|O7&-UML z+i#-RJ9A9GF;4F*`SiLq%k&%K{{F(UU+~($C1U#9;`9&pe$Y#Qhhq9$to^*$>pySaCmyby+s$-Y|4RCMB>g>c`pPb*%lRcqf48K+J5K*VC+YJu zy8Y5%?KVljEl&SAcr@a4wEr$ie^;D-r`)%nlz*qBzcWt1qu`gnRnl*b({C4tM^gS4 zNxvmd|93gR{5vH59dY_@u}_ooH%t1>ar*lszx=t9er}xpKDsZR{;!num2vvd_QdJ1 z_l9fl?T%r2pZsUKV)&~(?~v#F)A4z9tRHb9EA-WFocF}~)O)Y{#N!#<7o3gPXIM{q z_hDhZI#t~y@AumE;lzFAqL{(3}zf1*9qukRB!?|+5$KU}*g zKVLs+WxWY_^5eMf@^^tDeH!a| zzm>i&QIJM@zDDZd;}3wbJ|_9_^$L9Hz4Dx^c@K^BbNYw;5$=Z)eZfVD)Yl`WZGHLL z8V^2N>*^-G?EN<}eWoE#Li%HUf%jeYdd;lQ*?u=`UHm^^8PfO7oOJ2^X^-9*uWQ+yrANM27RdH5s;>9?A5}McaK1Jil_nv3NYTdF4Nc`N8`j?0rR1^@^ar%f0+j^|By+Ymk14 zmmclQCHyb^u-VJ+ov*-nT!;JjIbQaD39Juuy8((li}8j1wu0&A{sr~t_I&%z7?U`ccHK|GcpNSKa)`%yg&z!La@xx%p8) z<&BenRaifsKhpFv_{)4}f6@Oe{7Ds*mk<1L;Ni`De@Lkxt?l9ciUT#=T@SD^{eYZT zCoJWs>*BvU@cFICj~vX8^O0G|zY5FqgS=pV=?}}pUvuCOFGqg-!u%WAej&-v_Wx`f z(!oJYzg?FX`msFdd*H9PBOSk(^ep^I6=^JKj}lbyMSrt+?>S5_+jQuY1OHS(eGtZ` zK8(NI-@E5#@SmiQARRI^NtX*-lJsGuLpCSra-Ug}UPd}FO48-LV3NKD>9Ej~bh!^% z(%Ij`JxE`!)9)p|CBU!V)`I|*1QJ|lIih0 zn!lL$>23b^)DJt=z|lgU{IdG{fLZ?|xl5V8mFZ6%k^3Y7X7((9S@hAsBl+BAZvHRj zMw$O|$={dhIWFQVgZBe;{x9WoSGf7VoV$|wx9R**&tpOUFXwZ-&&1$UO89kB+V|31@HpW#pV z-`GF2ztNUYZ(roK@S|>j9qTVU?XP#s2kil$^O=8*#V4L$XrJ;D?el>r8L#yA<$hdg zU;3L{#`=1k_A&e7@@TK0=_T8qi;uLI%W*yIv`6uz`3>{Efcwhj%b-@`gZGsvKCOQR z<00$&JvRe??#~=H`rMqa{#o1y@U&Eb{f!r!?;H4Y+%GO4^(FY@0o-Rvdkytl{irX5 zx=F)0uO!#&Ad9}Av0CV4{IkHI2k#evwB#q}k$E4a?6;WvxuB0(ejfU`R+Rmm#zEM( z=-=KQ>Q1R%_lNf#U?%JP4o)I{aI2)E3F$BW+3exP#-C36kfyHP{~4%l-VAbTdzkzO z`FEc!hY$kUUwQHu)q-+A^B>i5r=>-(;}{$H5h|4Rj*A+gW> z{%_qnm;S#e?(f!1+5fFEd@01I+yBd?|Ca^*uLk}9Ij8?yHiyp{_hF;e}~im zZE^oQr2lV>``;XY)3X2XPVfJl1RvOrP5s}#eJ=ffSKQz2Z({$q$MBn=j@|yhS^EFx zp#L8U`hUCA|Lt-AJEi|`j{DynuM_s4`nwE-?E6ceJe2+Vbb5?wdSsgEQK`&@9`B6% zUl}7kDlt3;^TDOZxX@!fpvQ*;di*ppvTWT^jH(nW9>ZY;rJ8gEI*dbNRJ-w-^TXj zO!nouF+F;gkscBcC-&wt82&CjdW9an0X;q#(Br)hJ$eFq^v;7Go<5;{{lRp7(m!MW zOWaRy*?)=|FWOv*w?b(VC?Hs(EovX^dJ6R1d{N> zFu?wt@BKz${#oDmxU-wpZ-@0$qyxd5U50{!ur4Bux~dv4hy>o0G8Ao}e%mmBzy zAL}LHKOvyO|F>(!E(H(%TNw|F{_sA~3g|1Jz#Xbs(ID45M{CPE3B#W!`H$9mpib~! zx`7|67GynW_dkx-ip8>%KDn<`;#W{3)-&+O*3K)`0oeQx)z%k|IWYXs5+7mx&kxlu zFHAe>57jO!Jnz9<3q;@6hx_gGQ%?F+t(5A^SVPkQhT`Ij6R?-yYHHX0Y7vix#U$%i*%=fO?i zhKZHH57kPAvRB_5G5_84fkMSgf75b5y|=L2OTQY|b0_7G*2dwibm8l89ij_Qta9Oh z+^gSfhX<#V^W|vm+QA_GB1{N;Hhsq$H~pd7NTKZTC+geG>8$hPy{6h*p#PHg@m@=z z;^*H60-F4Iuc}soesuE>7k2ykNidTi?{U@MjO{}=f4NZi^S=oLVDhv6?O4yc`PUYX z`T4KLi!)CCtFH3%uPIFX`N!9L`LD$K(XD?~;dwv*b(eej-|z-M|H=a0`*yyAo)eeF z`Pttq#{B%t3uQn5wOix-dF0=Gjgvn@eLF^+bn;^*Ek?2gm>6w1E^ z1};AGLmuhHLfuI}S{t}ADZhpJE5ly?LgAQ~KYv4#U+RBz*~yRc+v%^g?M>CTZF(Bp zl@5Kb0z<~>$bTgm+=btO3E{$5U_!a@=7j(9#P^~eKb>1pZhm<$$Kvx(_o!|3C*lwM zfq_2*`r6Z9m|y<-b|>PmyY+ssd5;D9?|SaH_shga^fl5M zA5QC3(_h$v_xdjDaPx=p@U}mQ*OvZp|KHdfvOf~Omlf@+c=&RE z)XmTSHTDCaX|0b=RPo-4eA=F)f13P>Pue?C)zZP*dn|A2)AkbXr+4#D*4~W40Ri}l z{d9F=KLL*Rjd}P@*0$-qCVyo865kXUjE72n~*L~W=ng~zHuOq zuMhOXe44CXn()iBJ8Azx9`ZwfVSnL1c<4g||CQL2*NZ)Q8T|V)pEm3I#C|-I@ya8e z?@yE84)tBl7fypj0m^P*eg_#pU8!GA-nU+?#t_*uYRor1r_r$hgo z^G4z?{QS)$;=gm}t2dv}f6gEBhuzQM{E+gDHxho3>-3Ss@&~zrpPnD1Rn@DRQPTtO z!3lYgAxC;mRj=^kZ;(!O@zv%1xQAs_2D$O|m%U5qd8)4W1` z#>Cqo9_hNOzAql}A2R>1!+ezf^1Tt%ccQvEna|))=!g7fw7!S@jQKZl-*gcQ7a!#H zv(52QP4c>Pmp7gdR?j(y8$J!bf5Q2_Z@`{UGV(&E19FgFJ25|s`n?q1umAlc{{6@( z|8T7{p@+cvE)rDu;iT|KV+e4b@7>}4UC^T__j^+RV19Y>`9=1h+KrzcFYo^1Z?XRm zB>n&XP-YAK#rnhg5bcNk57(>ng=bygVm$HwOV{gxKupgX1YgCV1bn7y7Zv%um(?Gj z=YZ&wZFQmlgVp{4Zot^`4^-8FjCbWb=KR-4N#v7>2b-#Gl;hU6zNy-U{eJrE`elB^ z`G-riZ<72M^t&+bW9*`Riu;-SxE{xNdivPg??itfFS}@;T6{x#Pu2AKI%yB{k>e{9 z$mj$5=k@#sKOp^RZ4)MFVn07v?Hu5STw{Bm@FVoKmd~a?A%BcQd`f-B-*mLL0bo+! zL$$IfNZbF1YHQ^_eG8w6@Sc=>n%^F(t?}0XC!%E(Z}<#9;(V}A-){~YB7 z*GuvqqLl9iJre&0;t{bPOa(ca!5^bt5ic)g4t;geay`D8toS1;bDdWzdh^>b7m{D3RE?^CPF z+eQduesmIExpvz|7hb+*+i(oSUcZR+!XK3HgnZGh>+!^U<~1GaDJVkz4B)+V{@a|- z!2bJV>J$|o&y(Ul0qCz~9DlrTCGrOL?iKgP>yw?d2TZ&y^oz{@H){RH@rQhms#^PC zzh3k|@RN8Y!S7oc*#9F*dm;}Vdk6H3_N`D@pz8g8_{X>&AOlOhudD~k#S55D`bhcW zn1I5+N4OqH-p>MlUVX5ipwIE3Hu^8`6K#q0d6D}31oPYZOM2@0EgF{mj{cpIen$IV z{43;x7tbpEoEQ3vy`a}K1;quT(I*r4m5Nx<;^TT|4*K!%*Hu~aTmNEVTtL!G?1_%I z#4y?)-M8Wzp+EhNp#N{zhVyky@PhU~X5jD3??!=(`JN>7|3kIH@HF6Lz7&V+9^5lr zap0-iy1?IeZvJ^MebDoVMf)~+@|5kl^EKLM>|ff`=nvNK!xi9b@rm|H^j&OkWBor2 z@err;eb{NI{b=8?*B<5br=4`ZpZFOEj`nQ?W8ibzcjZU4Z-mi*PI}MqwEz?Rt~Tvy zeq(!|^U~j}-*b%RZAbpNm%ct}|4H5dPk8BTz4oJhZ)$4q54`-_o7#KKOJ9?;C-m+a ze#%KF{TU#i^zW*s_MUX|M|e+JETXu7=X?Ix2=Bec^h5t5*mFL;dxrVoRMOsgUVG%1 z9Zq}1Z`{d0RU675bJC}3Z^{3<14sLMllD!2uXV~t`^u(0@So7XFzolQC%_ljQeZaoAlCGCGAN%=7WdNwxm75r#O7bDL)1K{*N3u+IP9v-c;>I&tDzw zD>b#Z!)x!dq&?sRdob{Czb)`*Umy6hFZBG`5!UORKX!f0@f&yWovKX){_Sf5|Mu$= z|2E`PwC{?fK0~iB+xp~u$JQhr@|N~rY%hvFQi(o-eQy07;=e(CSkFrT?fC?r{|9;W z`FG@}K8f`QW8}2irzk}^HdaQlB zAz80{N7iTS)nl&yc)%!3tv`TIj_6wZ@4@QwWIY6YL;Kf*a7Q`e>bZ=`(U-~t;Mfy#EIqG?>3I+lAlHy_%rqFVsIE@-4B4`(?h1y`0DV zDkke|%)f_fy`KFYuD`55xM<#+$9t38lJ>UHel-3V?O!-iUB-AgeAfQy-w?y_2LQiP zV$WwVf3db_2BV)a->#DR#`%f$#1Ab+v6eO%{%A)~Oq(C@#?`X`I` z8T!rnX^FIa<4WYd8?nnsoF|; zpVP|IDeV6S_4ftxuAlYk^(f^X<$cnBtmm=6%a8JE3+7ul_E!xIdD1NZ`iA24lOoT0 zV|k1D67o03vtQ&H=XZ)cOn-ZM9O3Vb|y_|ICt_HM7tC|}{-Q_xoS}T9dd3&a zH*ObbI9G#)4*5p)jSu)6d`Ta>zYqJ6>H{D9C-QA`49`uzaXSm2DL;Stwk9q= zXZg0~Y|1wtuf#`sQQtMmH@QIn?8>(*<{{r+E^@y{+iz|87Oww(4ECGeE-~|K{`T8> z^ObLMzTWcRRPBcITzeGupJ%^K)i#~y%0H}MTiJ6aV1@H}3+*}0UZ#I3_M9xIWxgy9 zdACZYgM)7%?*`*^E$=qP@Z99xMR9(XpTE4*;c>)o&hqZOvnKC+`8I^{cGkaoJqvs7 z2Hif^)3dIBXSCNg%|o6&UDOIi+iRYEA>-xSZ{GS7?az088RnmLy?J|YlfThl4|6>q zte1oR;*|Ym&p#L1m*IN$!RjhcUWD>3T#ttGkoCFpBHVwTs=Zb2*GS~cb@CjHh36;V z)?vLlvdWb|v#w9EJ`3g79OP9gnU5l``eQiVKD#hZKil#O^CxA!iHXmT@)h!<+R2@0 z2ksq-;ZPnyUSR#n@B)0M{JiDQGP?hCd67S}@4wUE*1LVx8OR^kKPT%+_~+j0$P>pO z_Z_?5yv|AQO8jj)9r9=-PM@k36aO3N<*gS%fB4@d{tZ3>?3ZU4|CFo`@8tUMv7%-O z${YH-LVsK^p96n;D1So#K`3A53%8O-p}Yv?%Y)UGp1cU-S3LPD_v8Ei`!(SIS>+St!L0H|(hGt7bo^c5Z(lw#-hP(&8~7I>e?0$M zD6d+{C-}oi4n4m>&t`vF_p&5C)jwt&_L$%BSA_l~wD0-LK=1X(ZTySK2UFbGKP(@| ztAzex$j211NycxJl$ZH(pB6ym2jE}3P~=nNJHY>|;K!<3NBhCt*PXkT{3M^IKhN9l zBFyu)z=!#-2PpCXoKJ(oZuo`y8_|KiBb$W3ehB`$v-SNH*X)hdv}hERhd!L>yi3Q+ znDOr4mt800$@zfx4(-i?K@xr+tm*p!vU0v+3&wLpa$gbXgZrr0Gx}2G1^SQs@1yuU z1?ho!Q+~<$iJZujc@4}*@c(+sA30Bf^ugMYoY#>1 z{~@nNCs7|8B!8m)(dzI}nP76hVwma9`HJ-$2*>9umV5LWtz9q_Y9YVzB+{V zwSBDbClGwk7yC!%#}*ZicYGdVTJs;rzr+r4JW&5YZIyR^0{2O&A*5n|&C2^Vg1^1* zys9qX{F44+zWfg2Z?gz^&Gq9<&w9juRaKegh5zuL0`y0jx$(()ijL|ECgZCMjPXm} z>j3?{_b@jOWV+}h^auPAj;|aTv*S1M`IkgKNdIwOB9?bE z$Rpfuyn^z1gPFghlc*oqq(0xM_A+YJsp1ySomj`KHoA>TX{ZVE1!mI zgEAjPo>l`2%Z=6@EsdHug24wtl*X+%xXE?xS{(#@* zJ%>HxGTw3g$R5Uzpg)jr$ZPO{JUj&XsOknD=><&bbEC%hF2ca3Zs2d`h>!5!dZFh| z;v@9iGp^qkSXYg30h{@r9#ZRIuUQGMp56kuX!?gdPajbeSJq8t{Ehy4_7eH&;|LG= z0(z|N6Sd~Jrp}(1_hi&xVSepX=jri-{06;&-{5Z6XvZ7#^*Vr!5k3AF;Qj^TQ;_r_ z^@-1`0&BeFhd73U@tOGidjav&^gpmy6=Hn@dxrArd7BRW*njf7%=iDf6Z&Pdq}uXP z)y&U6^x>(YCV7~*_YK~Rbk=9zYuNSwWvP!kemRo(9#op6JvLg+X?kYSAEcxGoZPQF zj{JWM{^`7hw}s+M!5{m*P52&uR`LE}d!9FxKarJR#$NH}H}J)H>H%Ss=-($mFOI*^ z^Ci%8@AKxoMEd-<@l7(GI8QbPe%7DHf8y|a(LsJ14%-L4uH*dMvrF?k?xWK0MNNOA zLp=j}?EPD>eawHcD8LWGFT(#X8-0&>qeFYKJf%I#RVwDk;r(FOAzrBg`83?Cx@Elg zH`Mc-PeZwGqYqRC^AY@4tXlT3%XpUYKHpW3K6W*XXn~PmABbgs*bl{84)z4<;|F?{ zao$n>?dPvcd&rM;Go31m<6ne)!B>|_1^m|CDPA0=>lk!8_e5Th>_Tirh z^$o@o=Vdn`X09Ui@Z=Tf_2#PXeq&Ev=IcNW=aG4CJefm()K6nN8b^@EA}eKr=ue-FXyRoKdG$` z_7KxqzQFPVW4!=4it8(&14(_OUr>XwegZ!MA9Nr|AHe))?*t#%uLC$wM_B5|d>cNl z)NZtI-@6d}kuL4A{pza1YT~EwFE27bMhH1{{|2gqc_&?sK62h_ri;CZ@)sP}>vuU1 zjPZp&j8-A_lJlVtzbWr$4zB@h&y&GFvIgYD$NCY^T039tM}fg_Tp!3sAp-kqqVs#| zWkmz-invB1W|1#q;ep+6&V|`Rdv8$WME#%m(Qn_?Gk!06+8(=5i1*WU&E0&_{|h5O*Qs6(CE4)r}alrf9d?T}B!T)vKfioXQ<7OMzV z1}yJcU_S9MbIcFuPZ{TdgiHU8mc_ zeDwPh-rvIYran(%_!0108816O(VzS^s9M{r7(dvHuKnlF&yUFbWLLfU`K+Cv9ZmKc z>Nox#nV*;+>(+7jqciFmdARik=rBoAl7?B!jD^JaDXrTum1kS*RUR0p7`r9o>-5lI(jGk zL4Lq|i1`7q*q?TMM=k-30k50!hJKaxm%3h$KlJONy--(m0}pqIe4%MDBKPwhi8jD~ zlKSx*?LYD_P>B5z(08)J6a4`Bfc#m`AL)|3Z<@2ls&r$he1-j@35p9DW| zsCIR6eoOk(eefSIUtR`m{fiTaw-dq#dcDN;9JU*+KNb4jTOUy$VZV~~7tQ)Cl#i{~ zONC&)6iNK8O4dt}zh1(1Ll(am@xH>|k>VwU>0iWpkNDB`g!6229-98HnBFIL`t%ik zdd8bi;d<)eUThz*LB5iN`sF>KFQIO;zI`+7WByYb0e!(gBeA^z`+p_Qf4K4i>uXJW z;s}0mvVt^N8*r z$D8=T9zL+QSNjv*J%sm7G8nHS2xP|lNts{E*`WA~Q9ta#V!VD3d$dEv@40|Jux|>g zPW-T*M1InP^Ks8E+?PZCsdmbGP5Os;wF{~RBOibtw|(Y5Bh05f zD%0aF@&@Dp9GU*Wf1vmGwu8SxAKb^o;`ns``{94l^AqbsjPJnlqBA~_@6~)H8^X(h z0Q&!lK?TDE_Vgg;+sTYNxHmHMe=Gb8|2=bXZw^jDWat9?$lE!7uz(YJ?de14uTLCS zxnHsMz0ka0hVsInd+0v`eiOK_VgUJNKB9ap{i*dO^K1Hh`VjN`Vs^%UkmnD$_r1X% zVf{Js-Z8N2mYD?Bfs^0(E6J5 zOX?GSQUSzImp=jgGz|Nf^EI}g@aw8xb+oGQt82WT^Z>tD|6=A><^HBWh8WWoAg%!E zv|qG-13u`_llM3S40-(z;6^iI$lpQx3AQuvE7RYP`^Mm}Aiii1^w}c&|55{77wV~< zs87#M!Fg-1dH}a`MHr8~s&}zIwl6uDAN~ubb3WMgiM=9zQC#;ws-f+>F;*OZ@Z*lX zcXImTo2I`!nh@<`+=S_a-zlrCb?YZT zd|e&)>rd|2gS_|WFXz+c>WhAU@BX>w{C9=he0|LyAQPW|Ef zG>Jceydb?E{+_cQ5NY_`ZJ$=;|h3n0U zy~AhF-cZwhyMB8^DeWQb&*@+PzMf>i4gI_HUMy|U8?m1Nv;R|W|FPaS_5#~rqXg0jf55);#}E5Q?)cq_WwpouS95=n<|sa_&q97% zfcj{EioIj(&xOEG>}Qy?v}ez>KH|r!W{@8(={K?Wk34yiBHxnzgjD(Vk~ja)zI^-U zJley4t|w2QYa!pBOObE8)k#Ob`SMKrhxd^FTz`Q7=s%=?$I7o+=g(=3e<-iK`y0_d z?bDYmzF4o%Z#+&%{+~?ROIa_G9`YG^e(Am1U1ZnOyY}vh4vD?~C$jzt*I%v6@Ak^m z|0D-6jX#O|g}b?+HSwl{c>kE`5qGl%YDWazGxW0Tk4*&kEAzcJu*3f7X~;+V7YHnhlTOZzP%HuDLH|nSVbbo{ILqk8VseMQA`Dotav&Q5{e|3Jr zr#V03yTt!t{S`Vt{4LsFeY^Iz0M3)1@*X+-Em7ptH@d$=>`hCr4foSMQ-8mV0qD6w zwto!F@%>CkM@2Bkcw#>G*Rq(t_36fA^l#|-vxg2I!#^DV1L&|F|JYy1cw6uX_WQwY z^@hY#!hR^hKfE8sJ5M&GJ}*A!E(L#MfdK&cbbIjc6VL`C{P;rO8sb})bfI@{L$#>W>pC6u)Nn;$`unth)%g41 zpD%#jH9f!|NxImJq!0aLs7><&^Aj}j8jRq2Dp9R$P7k(jJ-=E|c&f{;AANz^oUX=L-d_#X+ ztlv1_Uex+bS_s;Ut~2->d6~55`wuODNPf_d{wTM;qS%wu4GE%zznJSaHy!g)=9^eq zkPprH8tNK+sjp-|U`BoS2hqM*-*J8TugQ9GI;ejM!iIS5zuFXNQ=m5ar}|*rPp5NalxM@!2dkn`1rRQKWOl`#;$sEebfm@To`_I<~%E6jae3l2fzC z7^`EOx<=3v3VlkgM^DH;(jojOwS!V0NHTIv7uqzh&OP#}O~qqz=RS^++4_U8fdL!8 zwdF_uQvK4Sf4=(KM_)T~%l>~sO5^|j*lYju(Z9OIXSE}LieY`OK~8wp``^zE4WNwD zU*>xk0r>Xg&;81GU;Fm=533>~TO)Ps*rSgeI~Lvl=&?3Dx>MQ&H7f5)vi9rzRoTAb_xHTo+na;41jo36Qa*EK8hnf0h6Do4jB z)%yDSM~`3 zj$2mXOzSl(R($HwqS^$?Jlpv8`~U8_U-`%n|JObHfB1tR{odNI{$T&#PQ1K@^k{tO z@z)y9J^r27e(;SgpKScY|NBorXbcU!^uxdY(Ki}HkN@MUoAx%Y{=I*B^+RA6@+ANN z#lQJ4kTTemiunJ!;{VI?fBY8zZ1Ui;J2ImDKchtn>%edR&(nYR>T{UtfAe2{@Y;X> z`@jGD?|%Ea*S_`IKkk3-+kf~!zJH}s>e2{pWHRRNIDrAu38X)ypTrOEao|(TVBt)6 zuIm)>;6Wn%mFX^Yy`td^1MZykwHbD7eX;P0x*7QoVrt?C=c{CKqe9PkA=BN9I=Gv4 zu=`5C&SjfXWUPC*E812roKiOdj|y00(y2u#AoYy(?8qhf95DFUy0g(6QTNylQCIjU z>gnozMFM3HMwwIC=P@_0Sag&0te0Ll-T8}5X7z5|CjSm#m4y_UZn(c3RCRa?`O6hu z8zj%cOdWZ~de?UGPqg}W9PJsC&Ae&G=pvBMehiptN_RTXfS;Q((dzf%td7)>0d11v zsD>dvOh0FabcXOSy?cgqhTJecJ3~4{YS6cfanQ&jjlak4R9%^gyA7x4gt0qvUEr># z5i$(sR#YzJ#%Ti!)+Wzm?r)}y1hYD22uqoHq`Pg-SU!6Dq03IGJ9mRQ)e{YPZ0aED zGyr_``J|@TnW|45jZOS=7yCc9Lv7Do-vb&{B#r;c+;J!1LIMXg%1pdxrJh~7b2@(# z>6UQ!m~gioY=G7*8(;1=vXwiHjpSm zda9^`pmq@3qKUsxpPm7xle*H7o~P61S)}J7jpc0g+_O07`Q(zI=ToieSz8?Rlt_84 zMO+!P$oNy<+XQSu+i-Vq@S`zeppfx?G76(0fZVTU! zpFd^A64*myJ-6hb)!tu#MLP^jjefEuxd zdJyX`ZJ~}|Fur_j^@~_vy>J<<*WYIAphXslR8Wb0Tt7A0e*r%<%cE@E723zr4(#lZ zd&MG-qIp?xtSjUZ-zs)irxo3Yg=ja0pMsT13MDh7Bq%E1^PmLtjF0pb?4Kse%IYUs z&0yummEH}gKq%_KvH}OZM@1Pd`u0m3fDzTaZf@}P~$DNaa?NSmNIX=F>93`t$rFl z_tJ&*=hgF8;$iUc(c%fmn-)a)TMF-cIJ9}-tlJW8djEKBw~*c*oF+TNInxSGt0V2V zR`s+>AKz3cw92#4>D*}u>r7TI2RpDD&e+wkSmcmJGw071-2B{`;5T#rgE=?9-Cqgw zr{OpkS|$C+q94f$taU#Qq8~~A*8Rxl-TX84WBiWGFAMt?-LY~xWImSFQG{&+EO7fO zeUWW3g?35)j>!(>ulG*(BEQ|mo4NkULR4V=y_LD;kMff_*1v3e*(~*E?%2HJ74;uv z##f5djj(;dOKokn*ll&YHWnrc>Fth(S|;PM*y%6ZlMoZB{Sp3I!nS)jKJ$goik?YX zPs|suF#lZSaaiU|;WXN${T8;9rcc5=iS9TZ9hm5j*1CqeDbcZ3@`w1$g}w>SzCN0% z&LlteaDwwp`pe1h>G_%RFR;EV%unCp9y994IqSR0+$8nm8R|RA-)^H1r+b!!{9Ww& zZn5hxZ@tm%k;hD4>=gET(8@~@N~Rw-%pI}QXmd$%Q55gmaR<)o5LryA-H>UFz8veR zvMk(>#%9| zcl)=k!%Yvmx6u9@jGrVWbXI?@RV&eshhFF0Y70rBp+>^ zrh1d;Ou;?bhqlp}@g8~8Y&(Y9#|C;4$J~o(%gSfD=@&83aF4 zyXM)b9PQ?@9NUV! zCEalNLU4EI4BV|Fuf5m=kGmOJolxqDR7xf3&6G^#;9zEuJ_#4=XbMfc<{86TltgA^ zqfnn52M;FPo1V@YAb${vqGzBp8+iXMW&>x2{lZ!6&Ynq#0_3+@(KvW{zVX#@)Xael zPa1@~9!(M;KK*b&w|7y|oPNlhLR3&@?2ckr<|2DS0@@RC(To>Y>K3Y79ba9+w6ANl zZ9I8Xy6M#_{2k)J8A&>1R7GCgp6m_iz%p^ajASZB%Wcee260CWw|2k3kP{45ZaW$1zpP*anWQ``ef1qFQWVloKpyDaB@^|Rieh?ls+RNZMwhc zlNGGvl#7K^2cG8>EWC6FOFXaEobHsza~93%PI)}<(46j+e;Vn!mos622M!oX7DS-a z5@k=X=<0UPhq%8WC?v9{N@g7s2OC~Ksn3SYbhe|2>+bxsAL4$EhmF^h`swg5I3KSg z1(tI-`{DAVoajl;g7DI5_gn^HX~j7U68{Db$MYlq9(fRD6Q3U;EM?}A?zTB&`P8!> z(`XH!ox4qEMe33)mv4*Sisz!1F=O^@Nc?+~^$=A8Tb&Vhj%>|@>31PLIeCRE_#(pg zt;#m^R zeMBlEQcl4p ztMho&T4JNZiki)SCk31O4O(#R=WugYxH*VNbgV%r29*J21F0I;9G2jGF#L|5qg&30 z%RiOYvAe`lH?~GJ z^k22GJKxs<9eSt7OP3eMOO@pu>G9J0mfJr`-fYCY_M(5TySW(lZw^qS&;bna;ZK6WHN&;W6Bbwgclw;ZQdZe+yw*{RHJsCy(o6;lKIE)Bk$ z>yIUKPHX@UilnnD`KN^jBL}i z$8l`hup7?9iN-0B=qiH5J%n56CT;8+cs_~m=h?p`zMsQ^FG)vO?yTttsY7Qg-zq(u zTD}$jXPACd6m8ggInG6tv{~E*=K-8QScS=Y0tA~SJyPSdrsvd-SR1H$r2|Dq!M7nc_cEa9-S zFlh7mM7W1ik6*5s*8j=4|Ak^2{S*3}>hbaSXMk-#!LLR7LH`7eEg$xOGN#`vJ!#{c z(C1%!Qu`P4?`u8i-}r{@P5wmD0)hQ8M#<`GD5VAxn`JD+$9U7qg|?KtlD9Hm#Ihc) zImq}>78A25S??F^dOz`fLQW;=rL^>>(;h|W3di%RZC~r{%oZw&5;ghuK&FRyks9&` z`v_wLut%J~#|NHWu77)WIsbiD{upm&xU-NtN{Ocz7JFSrGREKbgMXWy1?y*YeT->o zGv$lxrdQpqf0lpSy0pJe?|j&QyJ_|}i~&jcR+X76n1b#vp7L!Z9(IVADP zVY;<0iemX`j-rK9(lFIdh7@YzP+Bt%huV6y1)0=i;7(dPiM4XDQ>+!#>-^RstL9{- z+6j%@sf@$1ZkT;sENoiOS2$MA!Bi`U9#3P*{2_|VGC;yNsld%gvwI!JSSqH6{VYO>?3SWww? z3nH0HHKbT&=*+TO1Kg0B&Knp6dAQqVgH3Nr)7DQqpoSp?W*}{TK==0NKL0wqN z(bs_;ki5#*Goci5q!_Mn(~{8qMrrqi{mn>6!b|c*B5e)y;Wqb_XzLSKARMHsU6&fk zyrkkzj3)WAGmg|qgqP$v(iQL`wCpN_Fw*)vKJS(x-XL)-@ z{RGdbRx>u3aS_~%hobh3(=su+DP&?r6`rpQ0HT=(M&K z$#s*Jk!_s@QZz?*fI^GiPPZetC{}8%v|O0ZC>15KAI(uJ(>hAU^W=r)=L?(tAg@m2 zCGvg?>m{B=!Si6Y670Or1U2~uf>{1M}M=8-?g_?~&>FEHLG%pbyx z2MT`+;Z{2Vel%t3zWZXg{X>6YsGk1RN78Pt1`(h#1k2npU#B&(>wra zvD0C>5H@k;p?AcCQ+5m-@9vy-43hkF+c8M;&uzyb$-f{w26Ly1w+oP9q2{z}kmR4+ zu0fK2Zo39a{yFU$guD^LbEnD-p8cPQoq*G)$_$=-)%N`JkV~C!&vAoRz9n+5BV0Xu zaub}g*lke{-?ryp)>Dh^)jV&E()51d`S<2g+w=cZ=lSodbg4sKZO?x@0^B+QxR1Zs zHIemVEs5B;8N4EqJ(Vglr|tP)*u9~}PESuyw`ZJxt~*>c%@=OZI8dzZ`S_d^WTp7Kl`Vo7nPqfSfdpyMRlmy_WZX!|838| zp+IP}ohcm5xTqcT4~klu*p5I3DL{`sIA?|jxb_PV0$yx)>DcjF%6n~-Hd3g)*9H!& zEbz(3_Fmh!{4i(y-|F-9!N>@Hek;T5| zT~sqFl)51t%(%#S{@XGCxM!oI%U)Y@#&tK5?{I}9#XXg2D$NV?P@V7nxZO=b#AbFV~+28oF-1{HS_POte zv}68#+a&P(d$G!G&wtzVZ|wOqjr;Jdl_jt~2=VpcIe7lPdDOn`N9IvwfurFTm7g;3 zisULebtu>N{I|U#ZLf%-Kxng_DICl=>-ld-Ad}v0&p(K;1ni7OrNdgy$o7l_g*aku z&p%}hz7&qx(>`TG_RwDneY}-KtMvAaYtOi*ofMCQe)2v4bMfu9nnz)Ja*ux|U&*R` zn0}^k8qRasf$;D2uVnZNaPnent2_OJ{4MYF5AwIX(?7_+Ab0wQlun~um`78|-t;3_ zO=jbFZCl;(ALMU&$A6H&D`K(-Rq^EOZN`x9A(-xK%%Z;Nx_ zOs)=iwsvis{5z@ViEIVs{ND~_l~*@*sa4RH2OmI;w64Rp;nuIR&Gk=veHqKv0o$_P z->YBj3U-pZ_-=$ehK2!p_7~25<*qzOxhH;CbOs5{IPDVo*nC0Cxp&DS-O|xHo{81#nLQcMBZQ1QPBsm5Y{P z1tE(F2ks$p06&BDUMJnclAh`A?RrIMb}+LJBpvJN$&HsTFN~Ke%agQh#I*6!70da% zo2S{D{0aFbo4LLV+*BbE<>LvPkR!)OdJ6m_6v5w7Pgn0NU}L2Y7BZ(MGpbvct?R#( zlMacLq#tWa&n|P*r<>9b_PFWKH>GF2bj=$W!A*sN`zOICn{(?coBEhH$#c-lQ%TR0 zb?e#f?yX{)x*G!kJm#=Juq|L{_9a{V2Y4dtxp7qDm=JD%^T-@q5 z4|&bYcQE7TI~L><968Rj{K?$0dB-d2Kj=N8-swiOtZA0%iZZ=o2&;;>HtUtl9d}}; zcNks;w{b>y4|j3bb|$^s+)jNN_17{}%V$#bSJ;k`)f2!?)SlTTrab3wB0G?0vHQnv zQ?uM&owoD+jA&l;Y}S%ty057*12C$Fg_PMZZ&lx7w`fEBsj08C+yQ!6SUBUErLbG! z_2_iYsNU41heN(d86=Bd2~Z|4d_&a`5DB{kjcOX)7mN<+^6Tm_bdOz13f$_sjVZlXFWbitb(6^L1PYS`METx zWTqP_R-flFczO<2AHDhxufitA5E%J3jV|&_(m?G+p*?ykjHC_W6FhbBgqeVO84t7St;cW68uCFsUR#Hp zdi;ETT|1n<&6aQ6sW0iDU)PtNJdguRYDJW0pS6;mXPPc+gLzqeVT+V>;cxy5tX|}L zSL-sR9)sjV{-w6UT2h=8jz=12g&xYVOskba>v4y(7%Ojj+4;32)y1;Gb_A>(^MU5i z2Jp6wpI!;zT>(7n+)Z0YH;r@iK7m(f>veT2P6=B~!Yq*O#wq{Ag zF|Ez8{#J16Iw57fJeRh{u2oHvH*7z(y_EG!SkBR}utSplhjPNX9+!mW7aC6M$9&sJ z8&h}~Q#=^+tvStG(W(w-&l!8Z&3n$R%KNLP`i!eY3Y*CcEXjIK^;PL6|sO8{!jb} zzIHyV+Iub|-O7qf(<2M7(OdCbn4X4Jnr3b)H%E_Sn%y%eZJDBpn{Y_co@pbMrm@oc z9o8ShY4wD8969!5(bjXpDa3KUa2ib)LRW`)oR01e>j>d9)$L(@bJg`U&Y*?6_)kMEgCx!4&u%5;iYx$&IcYkIcATvzV%(9@I4W*D7TtNFH`#%TR`Pc+Y}{nExfoMG+tbecE(r)@nQ zaz$HjxAnG-&~n$;g&8zM$Upx!dOP9Txvi&LdDPwY^Eos0JZbavJnCsOdWQA$?XG|+SyzoBkG|mgLY6YjQm%}n??WC=#Q-7O-e>zQnhd6}robXQZ zdpFgSk|(Y1pVqbVLin#2VjX;T{nO2UC|80{W)ctRLg~+g-mb@7{L}7^^+0P~=G8xw zKici3NR9t~!nAt%cB_tuW=8kXCs-dG74m&a_`T(+JH)-#PkdvcILi z^YzhO^qs4ZTGxMiYs2}_Gm2s>V{%UCmQtdzT;0esJ+0cGZ@Ry$U&*ADUs6%>E7N`YJ=byWz69=;_&IwkMgew?Slqx`*r?q zzb0C{eP$m>>w%Xxjtx}1*#CZgD^ND%B;E=%KJYAVtl1&AHL=}(wOhzB$uNV-?m{v$ z$ZPl{7$cxXBI%E}0$Kj?c&Eq#N%9?^%UQtF;~B3Vp1hfv<7ey1v!*8@wMpG@z6#eQ z_b){ur8#E}*Gv&T(#miRIf=iybN4};$8Z&4NuMiMP1+-qY{+mG+l$tZ;Et%9P~E{y zMF)~FhkTrq$!y6Y-Irw7p%0of%PxnQ()zD4aQXPA$dQQITv*Q+QIC}xlrZkUqYHDu zk8&_me4oppU z{X!)4e^H(6UIOE>q%6cGK+oIUJBoaMuJb(Ov^~$byUov(S5Rp%)6EC?Oix5CDSi(4 z!@J?(9i`H`^)BCUe$g4-m&!bQ9!R8=|PKt z-x7Io&XYe&Kwd7X`MLz?>5ae7&oj(V-+pS_Q@%c$?|J$*Kl}7MQ++-gKXZLFZP!O_ z`7C~spK5!mEuUeiU~>fj6=w|S@ngNoYzff6EuY)+*)E~D{@j(xdh1V5-?aG|>v5kx z+X8x30(hy@(;NS`{S5u_<2`}1q~vqJA5HIy1nHiCgu8owy6=DC{qcT!^S$p&Kt9bf zpRmgU{|ZeMe(n3++V*?E@7{cQ8uMjo^LzUqON<}&38%Y+tpIMXS0iu#vWEQ-e?QZ= z*MHplcsBXnzMr$r&mMm)ZTZ~3v92wjn_dc9Qt>pt{nX~?Hb1xdIq+Y#{o}qp8QurJ zr2H>!djl(}_I{7=&-eMe&CfnP&f~+gCu+2& zXlrV;r~`anR+-GX7B}*J2A>0;#1HRr;8V=t!2hP@?ddCh!mSLnfEqlK21ZCx3JfN- zh%#-Y#gYM6y={f&*dh6_}9x< z=$NeBE~Mmk<3CbCIae|X_=WXu$KU+;Z+>jAK3N|Ln=97Q{7R;jY_OL)%HPy_DZ5!$ zgQG9{v+W-bz;QadoImOm;Z^)@n z@bCAO)u(=A{pWuFHF2A*Ja~=5`8OwhwZhpz2R=_7^WZ`C84q5irabsub-xF%RCNzt zp=utyTJgvC8D_;DcJ|IRyD~2GNO#|-n;scu zcbHjrmszPD+`<|*K=|B9)G!QY9)XT2#p& zBt?``644=%O0Xm;R5VycT)*d@?+fmIb@#^I{F&T(?sM;Xo!9rA_xJlf=W|Y9evMbk z=14fU!+%MC`0>H?HTNIH*U9X)<4)G;ReC_@V+4BBByItAN_P@wp zkHhPo;XAkLdPb7Q&)rUb^}Ca#7+?3}YvbLF_s7@s_?l~tJigAx*Y)^%9AA?k$h^9H zJ?W8`9-ZkCT=^E({qVZ~eM!>&V@cBS%Skd4p2hIYttQF&p9}7v59}``N#nl`Uos2N z#qg~AeExP?$NvAv|F=gV96I0i>v`6-*hg+1Kyl*4m9Jd;q8XWt92;b(zoKN()X zC%lFqQO$iitp98{M#IfP$LGWA7s7FPFRTmC{T~g-=r_XpVmL4VV3Hh{0uSfm@p53{ z{7Zf@9IxTI9iHWK@c--Jt%dD`{|L^%49nrUcoCleSJ>9?x|5W{bGzeCvVZ?hl6!C`841tz@H`LC{rsJz z|ARY8dfQeG&)Yvfo78oWBpqQ4?}Q?po^{1AHoRXSzIk8#`J4A0zkKul{BPX6U;oE9 z?~7r~*iPN!?spc$#U*{DKP8#m`qgATE~h^Qoi6W(H1t;PV=Pei~nV493JU+ zL8bj$jLYdyHB|Z^m%GEi0@LOG|0?s-e?_Ou#kidQUXM!qxSam0vC=*+r@zA?UEYt& zyK(tEE~me#AzgpGnr;6yE+^sKP4lPUm6ncQE-t4(C73RE$K~{=3e)BOa9*c=x<2jS z;&HZ|u1}ZKV<=sp_V+j4`b2X6uWJ2{aQ?BsbiDfGa@sy!UjH50dbYQHFI!H>>)l)L zB6~$M4Pha3?fdz5cy$jjmq*Ua}pRX`deF z)$M2hW^McQ*uPmH?)1J>y*}g*=flnVcar_@4fhTqUphbU-uj(M@_pHIIzMTD`$K(6 z*QfJS8SlYtIXxcUz4b8Z_(ryzj`zE_!ku5JL8+h4f2IAmvgP#nc=y)ll5(g|?El?c zP08Xf)Gl}b$=c=fU#eZ6{cmfRJN|O*a^wGSyYd?l<}3RP^Z7q!%W2+pe#^(9hJ{Dk zzjXQhe+~CLVRHHtZWsaZj zPi1~OAL;Sa|2^3<$Ip8Br$qJfv;Et%<#ayM<7dC6+HXpZzrS|*{ByO-$q&{pH-5f$ zIoDphy!if|n&YQCf2Zd7xYPN`q^_kQ>Hn2bgTnQ$9IkhDjo~IIT(`o1m1o^&(jzK+ zq}Q_a-M@{m|95=7imwx)efr3LNIKlN#N`(8?zr3+_L)Ay(V5;_T|XB7arTz&tjFb* za9@-@?6?2fH}jct#@S9eF1L&K$K{1^50*Z>{jT)h?+yIwyWY&hEm`G}Zs)!DK7H9s z2Q@st6nOeM0iQwiiSyaWUd6ug`OKzAJT4|{+77((%dNr!`-R8he2ugI*ZUf7eZ$zr zL8)r|xWdErL64uQ@ZTz$B!3|s7TJ+oyDHrBCAY)PH{(!_t6Rh6H06vN{0~LWIP+Q0 z4~K|%l26obJS_de3jfq={Er9ziN>m&Jz>M)@e`p-pRKYvQwo9S%IWW}seSxH;9scX zxw!hjdCPWcTmSl((!(K3THCfeEQTQ;V!rKu@Kw$M$8zy{ZEDUeR!It1bLs(t)GT>Qh?0Hy8uoVQv`kFRu_U#^DfdkSs3 z)A&fo&jXm&v9x`<{hCK?KT-pAkcD-tVf$&qACbrY2dmD%H_cx$)UEIlMLK_t)teak z^I5;kPqWwG9#*FPuaUy??+pW&8CvzptVic@CEM0wG7#GTM{n5wdAN^CAHA_*y8rZ1 zHJp{@vDm&_(K^F@SNh=iP1~m>!R_;5eE!%Q_J4IZYo7~k(ueoYarEP%{T~kPyS4vZ zdcg^gT0>-L2#0B9{=NV7yh+FO{0o^M-qt*tV|!j{pW@$WyKu-R3Fl|c-oipW zUwN(GUz(Nk!p&G7i3dn^y9(lbMvSkVU%b+8;M>KquS`z9oQ&RAri)`wOH;WA_jAS3 z$p~Z5)=odTU)wNr zsM2hD>ch9K8k-7(b<^r$d(K-ePK>^|5$<(%c7&a>2vW5b0fVWyeZ>Rp#= z*4%sWL1jk5wjbt0TRXd*i*_%y1>U@}gkgmL$hp7*zp56%eYSvAB6ncEK z^8N=8&B)jVDKE;YHO39_JrCy4P8KOcq{L zx4)MTH7&=dZ&GO%3NNbG#c^g1Mkk9m?c(rOx-xaID#?RKkMf<5YNmGTe%1P_uCn20 z&p)c3f&9ac>dAX@|LKdW$*=T;mQ$t4vC-#W%zeG``uUNY0`RqGQ_n|BPshg!xl$+{ zxpd9T7rD{Nk3N1r_M%i-zT1(~?9rYZt5lTWXGc)LKRr6}F-0eAZ*=0>M=!3|=L(~x z(OlLz{J*MwWkp3!_w=QE4|7kxT^g${PQkCrm+kLnQ{*C-7rAc#a~;k%9D*pdh%86 z^0oEhs#3MOdb|IAukSxzvSNsu_Vl-^16A#_<1yCI zw0U-^tKDstpN&V@79JhE`$3YV7eZQQ?^zrA^)QADFIC&i_FpL>NfIuVG|%3X{ZUw# zUM=I=`2Kq1cWXw!X8V=nDIM6#B{jCM+JBti>-9I=d>n2Z!$tNw&`td6{8{_-`Wx0~ z{`DrpInt4EEw0|qYx}=V|EMh`o7XVzP7(`e~@z8-0Ux_lvRE<-`p*PHsNOBdMB0lvGLt|`Q-Xe;@U{} z@`n9aja|CFsvS|nT!aHOK3>Zcj=zWDm`-m^w5v*sDnHJ5de?$~y^FAqhe>(^!#1+_ z)$1Q6lRwK^(!8NR+09MGrS;u=ouMvFJ*D;CdykXKO%32?b8ncRaJv)MN1sh5ckg#3 z;nwH1Exg{}cuN$vQSsvz?%ux_mZv`S23~7_Ki)A_+Nb_?gplEW67HU^H&J*{g`1JK=EALr*_x=aLzgby%f4$F#Nw~SXZdiGLz3x#G?ys)bG32lH zm;0_DRQ&7qVGi%#w6DCsUiUbu+0MdacJ$*G(rZUYQn^Em+lcS0)@N7N zs=VW)FA5W5xtEja-97!`m49^K`njJ9-*x$!TP+RWU-^1UGeUT6t$N=c-d9~_lJr{^ zDvyDN@9X|kx0mYv^gI7){eRZ*OYi++w&m0;2)U9hVL7>_TOD@_%rYQ=Z*ix zCsvw%>3e>$`MY{sT7tV$)$aQA?|rZTm%=}2)9r3mt*`rXTo`)q{Lb698&nsUW^2+cX5k{5I({5LB#xSc;ulwoe3Ad`guCN%sAEWXJFX?vD zb31)7-ropgaI5N5KZZ4Rvaf)p?`|HySlL`?q{r>B<8=Gs+e|7C=I7tU{qpq$#{bU- zFZFZ3FY~L%?@eD9$@W|M#6x>db5=glsNtFaHeU5%U~HB7=_h-&kCTv}W4n6IPlo(& z;#XlD-o!um7Qgc?yzrL&{=!@MuY_=K+V5xI!hilPyd1Xwi7+qemp0P}=fgjJOTYeh zaNe}v@NGzMLZ2voN*?yWieS9+Ht$fa$Eml5bOW%G-uHxnRKHYvbPv?#R8JzQ{kL?F>|HCwxwQ z{U}xO-*!#I*ORMwbq!v73+MB`G$)_&rkmS~r+Ui2w`%*{@l>?GR>iCR`gkhZ@2=w2 zez}TQ=Q*t6)&5cB>3CMxu#3olNPI4SFT2NoDRPhhO5|z&Pgjk@Wff1a2|Q+V*}K&L z3^2UTMgA!8@2}$Xk&}l$EJV(61z)W4faqf0)#TYh8od*s7=j67luynr z`S5M|_@p&X4ek`AUoydou%0G<9+nId$x%`vpU&x1F$JLZ{?4=>5Th{xx&eE6*V zT-^SgeE7V4`n4b*z9^r5Ey;&3%cox}^5LuUIS$w4!`J1{#{O=|hi}TKUt99w+w$qx zj(qs8eEPK~AATUeGv+^(4?mJm9X^&1Kao!zK9vu@kk2?=%7Y;r;Tt?hnX^56UO!kbL-vd~z1#!%OnXIV~SPBcGhJ^5Jvx$vH0{z965Ri}K+s z^2u424_}o}&Ncb)SMtfZE+4)jpPZZW;oI_a5ARfNWOn4kcjeQsJ^Ao``J6`w^5KW_ zlbGj7KKw*Jb^cU7{7gPM&*j4}Up_epG^UunM&&em}ynOhAd~zDR7&_`ZCO!vp#7Bl+Cd9m|KG$fsYY^5JLl>DRe@_=SA> zbtxa7{Epgs*cs2qy2z=A@Ot^w;RgBeM)}m?Ci(E3e8!^;T`hHnU@dm zlz$eF=Pvp1UiswglMnBg&pZyuhY!kMj`KAnA3h?Veih`y$K}(n3Hk78`Sfc>K73X_ z*Zn#9@Ok;DRJ+_=p{a9?OTH z$S3EieE6Asa-PeFU&<%vm3(;R2jN5MqrQ=|F8u-d@Sq;T>m$b}XM=orqkM8U$%p6U z&&KnsRX)5;KK*K!4^MvwTlyeB=TTlhyi@)}%+n`n4w?zAvAC9mt0t%cox_^5LiQsWWHt z;pg(nc_ANuDW9BI^5ONrv$j5vvmtWoA-qvOIh*9eo8^U>!~d{sX4z9t|3NFej%Sae<>e+C7*sJKa|xy z>KnW+a_*PtSG|0AlYIKsEFaz?pE{G14{w!E&Nlh*cKPJ&kPq*YPtI=n@E-Z(?3EAi zlTXfm`S1bx;X*$AQa*M5NABl5{vkPk1)Cud1Md|Ey^XXL|Y<&$$xK73w2ITz%^m*talMLxVNpPZ}m;cN2A z`AR-~T|PNCDRb?_=J4=Rg@2( zkx##7<-_OXQ)lMo!x!X}b5TBgNj^E3<-=Fx^L>3~`S4Zw{666|`S1<-jKijU_?CRe zVOu_YM?T}QD<8fmpK;ii4?mJm&SUxT6Zzykl@C9YPtJ4s@C*6myp#`5-mk51{qg** zi=6rfua{518sx(p<#V1k$%i+~?}>R@;&`Q+@756{adXQzC4mwa+| z%ZK;MCug60c)xsd4#N&vk!SK73C; zIrrtm59F8P@pC92ej=Zor}E)v@|nkT`S1(*T=y^K!>{DiuO$4vp7cRIgx5vRzfb8` zy?l6+eEQWaAKoIL>q1UGyj4Cq+vLOB<&(2RKD_xh)^QBY!RC-<1#Fm(O+WKtB9XKJ$JgAAT&K zc|Va4KbOxqT*!xC%BRj>$%pd`!>DiMtc$-W#`Um1a{SFWo(=Nh&GO0FA|IZU&pfuu zhquY6&bP~lcgUx|dHL|lFY^r71^2g0KKdb(A_@I1p4#|g)$R}q( zKD;QOoF)12Y5C-wkq@7hPtG~{@Ok;d`~{-(Y}26f&8tQ=TJWUSUz?BL_YjfJ~_|i!_VcD^FluS zQvP|&ep|V};eGbw*U|dzM?N5*c^s4vACga_xh)^QXFulLw;%a|d~zPjhabr&=dpbFiF|UN%7>rJ&&BiWLO%RbKF8ISe0Y-m z+VAW88P21+$T?r&_44auo(B2wCi&F)X8G_I`Q*&WhquZnXPbO@yZp|Wze7H}Q$BUR zOFq0?KJ(rqAKojUdGC`CAFv;f+d=!056P#_kI07?{K6PeGK73m~Id|m4cjc3F zPd@xmJ~@x%!;j^Y^F%)UR6aS+4Q-lyfmXXG>Qv-05!@)?Ik`S2zA)cIxk@D=&w zEX#+l%HNFRxh5aJA)lO^^5I+Zna6GU@E!To`Ca+&J^A!&Uq1XmKK(kB4?mGlzfR@D z&*W2Q&gH`|9`{a|eUp{<5KHqmSC?7s#Kc0^x_9GvcUyMKB zJs}@nl+U=8;OAmru?E`S3&e9dSI5dd%&_=J3N7UjcB^2s?ZA3i6aob&SG3-ZaiC?CEgpPb9` z;VbgVS(Xo9lTXf9^5N_9$+;mPzA2xaTk_%C^2xa)AHFYtHlAMx^5KW_>DQ5b__2J> zqZ9e?Q~BdD&zXGqg?#G#rF{66d~zn?@5rVP*SEUJsfXmOmk)1{zZ~;7%7-`0r_Q&? zhv(!o@2&FTZStA-cKPs5`HWAOe0aBf>U@uUc&~hN_Q{9$%U_G*IUpZCBA=WE`S5Z1 z%;SW7cu_uez9b($EuVhP$cN9$r(bjO;fwO=*OGkrvV7{yihOukJ~>zA!`I}K^Obz~ zrhIa4$%k*tC+CiQ_^y0%?#YMm%O~f7eE6|^a-PVCpUNlanSA)Ud~#mMhhNGk=aqbT zUHk)`)VEyx{`UIFsfX|e`Shz%KDm? zw0!uCd~(jphtJ91jpI2lAHF1?oXhgzEApAgvV8ceeCqs~eE2K*^lM!{d_z9{+LRC9 zkx##N<-_;nQ)l+&!w=+>^H4tgNIp4_<-^b9lk;3Y{6aoCFXh9p#I9me1G4cI3l%<@d+W z8~5bH59KosNAlsv@;OdULLAVh@AUF zc%yvkSd)BsvwU*4$cN|Tle1Mmyu*I{zPP;o$UEhevr9g_TRu5^_s8GYG$$XvAfGzFC?CEgpW}8} zK72(!=V@6!d{usZtXFID;p_6L^BeNvoAQ~LE&1?m`OM3XeE7cocw8OWkNi+Rb^b^` z{8&CYPvpZ-#-JLQwJOFq0?J~?~j!~5lvb3i_PP(C?_Q_{&rP9{FVHUc>b=- zhi}Md+&1OIx8!r4Zp(-7$Zw8$cICtO<#Qc5kPknUPtGIx@MHPpJdqDSmETJ}ln=j< z&voQdKKx2P$5qnxmVYlt&ix_BRlR(8lYGXZSw6f)KG%_)e0ZyT^0&!{x62=l86j+NxYr{$A# zMm~I2J~`*)!x!a~b4fmYSw1;eYz9yfXujIow<&$$uK73m~Id|m4cjc3F zPdnT*-&meW~{TkUC!59D3!$`{YyS z`{lz2DPjM_@aFJwIm;2 zmQTM{<-^zHQ)gbuhp)>g=Z1XvrhIa4$%pUCC+D7g_`ZB{9>|9u$|vWMeE6|^a-PVC zpUWrbg?#v>d~#mNhbOHqd~){5hxg0hi}?rS!-wQk=SSqj3-X!wary8G z`OJG!K72+#<1i~9J|~|#KQAA?AfKFz^5IMJkK%YP%ZIPZC+C`c_$&F$e(lPKAIPU)hw|Y^@~Jb&^5G})$$2UtekPxs=knoK^2wR}-r9f9 z!s{aE-?QYbmk)1{PtHd9@Fw}>Y?cpil~2w#`S5o6DPsP_@#XMbtNBO-(7ouNWU5)=l&4hD4#mh zBp==^pPVi7;W_!_Y?Tl1kWbFMe0ZmPa(2mwcgrVdk9>Hqd~){5hY!jp=a78(hrPy%7gj_uZx`fLvq&3hd0R2$MI~G4{woA&YXOB zt9<6MO+LI`KG%^B`S84a`qe2P-X))Yb<2nM$){ia^5FyWsbhok;Y0GtIU*lkkWbEW z`S6l_a!$*K&&Vg|tbF*Kd~(jqhcCz{=c0W0vV3x`$cLBZlXF!*d`&(%U&)8B%O~fC zeE7C}eh=P`eE6<>`n4w?zAvBi=s-UFP=04T-;dj67luynr`S5M|Prs5c*VZ?9UF6g^&ZBzy@CNw<-0#VU zH_NBax5$U*r_L|Shp)(|UuF65 zRr&O5O+I`>KK&faK^^KhMk@N3gc!PX$Hp+)L$tP#Ce0Ym|a^~d2TjldPQk#5uyL`S+ zwL?CY;r2tbDE`bMoQy@;R;+ zS8D4ab*wIO>LI*dJ~a~{pghtJ7hhwmk+;^PtN44Z~6Cc45 z^66Kze0Ym|`jwLpZ6pV&dP_+$tUN$eE5QVaxTh;FUcq8vV3@1K0l|hDj&WkpMJfP z4_}whd9)!PzA1k%p6^@o;XCrF^Skold-BP-FCTs&pPYyC;Yapk{$u-*pUS7spUH=x z%V*v%g=Z1XvrhIa4$%k*tC+CiQ_^y0% z?#YKA$WOkMeg1POAATgCejUq)pUCGtI+YJUlYbh|_jCF1OZn9KEBWxG_s#XKE^^nm zdimsRkPmN^-yFxYNj|(qK6O4PAKogTd2f>sZdAQ|Ei-!~5iu zvtK@ZKz>^s&q4X{A^ChRIwBulklzu%?`~W^ydLu!+Yg34t?_B{qpzY z?~fXg47_X-z^5F&fjN7<;_=J4wSW!N_B)>h@^J)3;Ir)pfE4x3Lmk(c%zx^Qd z7v;m3`S2t8+~*w2ho8u2UQXr1&*XF7oy&(`$Y)+I<-;3)Jgc9NZY4=e_O|l> zrpP1Hir3k3`S6l_a!$*K&&Vg|tbF*Kd~(jqhcCz%!#CuUb5lNiOFlWb z<->R6lXF)-d{6#n%)c)mek7lq$MWGP^2vEBAATmEoagf47xL@k@14Gs53lRX>KOHk zob{1Y-{1}M$=N6$-Xx!#&GO+b^7;GJIr;E*`Q+@756{adXQzC4mwa+|%ZK;K=kK5P z%7+igC+DDi_>g>Zj>v}>(H<-=deC+E6+_=bFPZpw#m$tUNweE5!hzP`UJAATU8oQLw^NAk&eEFXR% zpPZ-i;b-#seEwWM{6apT&tJ-iU&%j@pKm9>ueQFybCIX@?Yo0gpJ_>Jm0!)}pKY8|Ibn@HD`;%i6qwjzC?8UR;(aFitZzsb~KQ4^+jeh*%>G)4n)UQk@ zt<%vfjZK%5;h>bphM&DCh4rHo!{2-`cK>l<^y$M#Pdmr%QNSYHWP?!^zR-V@Y4%Pk*f^o6@hnEPOmMRy)h-kuf}xM*R#bi0M(1xXjPBl zS6@!v2-!jOwb9~NUOatS7#nytF;+-g3!&$wZ#^IW@WW^S&!<)L{#0qQH2NgGe*1al zb@F9lv^1K<>C48mFN{D>1^xzxlqkbfW2ND-^rQ_F`ttd3F+EdZ?C|}kQ>A+k zuPe(pD`g|yUG-TL>2<%Se)ieV{oKUZ_43!V<*9H=JsExwX7k&{v8R;{#0@)(EhLH~FNB+RTSBCEoKY2MF>dEtT!^02m z4|B?dp?^KXFl9Ja&)@jVUih*=_LYI!ZCrj_R{@{>u38>04>gbU*IuXWof?`X-Ep}+ zUWZuS$eZ|@USHBZb;o61X8)5c)&4)pNP4+QFEL#Anxpu&&^X=yC*m?M#kgPZk7Ztu z!fw*`@$yCUWHz>c7QcUh6|`gduZK5jCXeCyLPqS9pZ1>$Z@&B2Xm8)k>Gt0=e*JO& z7^ignep7hT9{!2YanEVclFOJseP8{s`_0(i<45~OHDLd2^TW{aP5H-T{>3P!`QHss c+LQkv472y0HmUy;+4xV#iAiI8@(u0(9ot509{>OV literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3f1dc08f97f6bc42e42599fb66daa0d8.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3f1dc08f97f6bc42e42599fb66daa0d8.o new file mode 100644 index 0000000000000000000000000000000000000000..2213a4ff77cfd7071f08b15ec7101923c318c18f GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Kp1A?+13jo8ITZ&*Jf6oAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi80x6|-unN)s{U8i|5tVE_Ls|6F4r}UsiLu0SgfX?jAi!5?mhh0 z!%{)%W<50nwz{8ZyKVo6UTFKw48{%**|PiI!RwqYd+xvBdLFNT#XXu%RJ3%xw(AF> z-;QHfKir^)KB%&VuG!e(|7T-;M?2VI&>rc@V1?hy_Vj;P!ruCo&JNE=?Ws9yo7^{K zn@D{f=}BjW|5x|)52Kx~+1{$d*dwQ&9;`fnU~tEKPYx%zQXB zvg5$uQ%^rVm^yz+Xim#_!WZTp5AS>Lr{SlZZ-$#sTnhKCd=GSmq0qsDgWFFW7~D7i zc)0TPiEwk<>2TkHH^U*lN zlK&~+8-VY<0^hvndv~_)_1wVsj*j7bM=icbKB@d~z~X<(UvK1t*LM9d6e*ef*6;)R z;S2v%eAR7dX%~N0e0Aidvi@JMW&Np`A7^K=tHVFkuf7hy9?E7{&z4L&Jk+2aKKr-& zwc-7&pXL$M1|C@feB@cyANgn2f9Vzd>N_Q@|J;4->c1>v{pZuz)eA}NY6@tJFxQ6n zCiD;g6YC#-Ks$W#f9h8+zQYa=|4l+@_*?Ao#RjaY6IlPqk0t|_7#eOE8 zp|czIJVJ8~JRiQc>s8ZS66?E==OXwgS%$$YjrlIg1Q{j|ZCU>aKu%ptg(JtH$JZVVTfSALChL6 z_JJqTRZ}9aX#a85Xnh&`V>e5m>j}~LL_b6G^z$)N;*W7HZTgjut2FLou1wF>lzE;| z>f)l1^9#ixvJKgU>_q*&1HGPu{#=Nkfn-nr4)p&`T-V@LiTfU058!$Num6qv)42W{ zjSqXh^K3%M2D>W%vyEq0-+}&JGxd-Q_PGeZB|H5ZHXxf{fX&Y?V*Nv~AK5|q^!saB zf4GG8GkxF@vZ=EFm$M3;r{|5WT{DB?5dZH(E+Vab!l=45u-t^J`s4Q(I$@Eof`+^et$6 zY(eLa=C-Zvv3f(7tUtCOdehL}>i2fC1)aWN=K>*GT|-xAi?^er&fC(m&D+`J3)D4u zJ38wmw$sEs490`3AEDsjv{%b|D86jrw70$tC*GN>5WV%4rgV4+?_<;=y%rm+1?^k6cKAB;SY2J)V|7u? zsvFzi7eYdG7UeSx~r)6?ZkaG`c&wJ34)>?$)l3PIsekYjc~=y{@Wi<=VRSWi@Ln zs#e#PZCqEjy0om6&98DVa?ke$8k*c4tvC;O*z3m7#4DYz;PVd)uRGWifZG>37B7rwS?icR#lgtCdu+I0BH=r5wAG5&HXu-F9J+oiuh-=DBV0=f zfkJ*UdbTmvyGx6(Uja!58cJhX-9le{lH@qq@tuhgegKyOih3toK!;KWEaY6+8}ZHzZ7n50MPPo}Z-z{G@!)c@e2BzBJI8K+Modf;<| z_K9@XDf%aRKFtI@`iDMjB#yuO1I~XgamAzlJ{0zaFqLC*es6WZ&RSK73>V{12ts zQ^}vPQ2Mt(Kfq@Mb7#`J3IF2lbAvF)aIH3(2}QH^X>lWsq8PEY-VuoEW*#__=W6=ZZ?-xn|MDKo#7(T z+d+@Djp&a#Io^)Cz?T5;2HtJLyHOYTGT^5HpC|CcqWv_~1^%nRX9DjL_}OW^|4h^c zK3Ut-?E)T)D2?AF@GjH^K0z1qY?1hE$^RCK*MKMf{Q_UV5dQVxgCrmKfAX)O&BlF6 z`CD+#91VNB^9kw{3cP)^m*3)ktJ7w|^$=8!ezOQgyO7 z`~&kjm(N$4&%a1x#m{SkoctH~^5=ghn~VDB{>=37*IcqE_g9QZ{dOOx@b@S zXa`>U8~hvkof*}yvNth6_I{i6P4_5L-!z_G`R7mx^wTNuaz2mNzeLOS(Em(QxAG6t zKmSCE@EEWB`K6}*dUs{S^j+U((KpR+gTDEE&EWdZkotrFXGHZ~pN;jxWzo0G71MVW z>B|+H-+X?cz0^1Q&j>O~r%WU0C{6{PL50 zdQ83?C*q~}tn@z4Z<=06@yBn2$!Pti`J?lAsS_SwZum3B-^-vw{Cr8(Y19GKyXZz1 z;16~)opwEmMH9wDel}WuNgG?AWPX7B4fAI{@uLdr$9KYhj{`vY0Q`sI@5|(n)cymK z|7a|}L!Xex65n-0e9w*(&yfR@Nu+-DKxQmn4=vPI1nd;K4e?#q$-X?kBmaRvQc?ba`FR#qvM=n9d_R-iQN%;C|CE^h;s56O zxscOyd(xbMJ`01>bXsm1>F>;6(E2qkkzJ1P{AOr6{3|&#>R&?(u^-6Eg!m|+AEy0= zgZoc%bbpeS6Qw^-`wd4jw-@;j-=DZSeHPIJpBqeeMEwc*ZF<0+xvyn97udo;N zqJ0wOJIJSP15bcgzP_coke|7u{Xzcpz;t~I=p!!7=i#c<^gy;gmFkl*fiOPC%gI8W z?>EsO?3+pM;-~olc?*M?nbCM=<%jt1#dz1}ht@9ybRrP`ZT9y=G@p#~0qc*+-+?#H z2jETq4!mJLKtH)bv;V{2r;~koz5{+Uf`<73c?lo&|4SO>Q}8boijObiONhw7m>YG{ z@41DjBVC{lNuCR|e{&?n^h^GcoM6#!CM{L|dJzx6FSMcNXBFmm0_sqBbp9u%$M9*u zC!ub{yKw5E+7mt!zYcX{sTD=1{W;t-Sehi`CyC+$T1#d1AM zUM$H4f9U}|l}=0|J!Jnw+VW_=An-p8a%W(CeE3%{_|c9(@DcBX*Ap1$hXAi*zU+43 zr~t)=_2Lx9-!1InuYZ3W`z6KO*ZBFK($6i+gZ(;RKgi$s`h3P@PiuZ=%12^Ao5f|qB=}$3VA%BRQugn-DUvba4alVqp z^A&ged-x!^*AfCFjqWQ`&^!Lg0kt|wYdAvk_mV5;9(v*)lFy0I| z`27U=NS0XeDL#+NkM7)jg#70}WIi&_6u(CAzbU&wk^AohL1rfi&p&W}U*7fdnDOeU zDP9@Q*Wo|66R)ECZ>(?T{q{n9zA)B)dp8$Htxr5Y(~%l|UX(-e{=^=RBSvKT6+i=lK)%18MAa&nB?S~?nBEP%!cxKvfnBy15 zH|;l|Z&N(uc=P_lwBG>U6yJt<{{0E#*=3WzSmPN!Pm0+e{HEhPY2cvL5A28i%w+mJ zhOf`(@Ok5|iLw3pgM5F^&rf;2{l-GjPd4nw58?9$`((Z!=l=0uXg?l1@4OzUk58R;F z<6B{Hg3j;(me?QC{K%Xz5f&2r?`-V9vnJ5zbB6u*G}N=B(eRs(>cce8wV;> zZYSPV-Nybq9jhf66#G@QccC8Lk0M{2U_5UEKErq(MgE^j`KH)^^7)V%-GB1^H~AAv zG~P{~%-pC`zIuKk^y$(K`h@+{b&Ebt`;oEp*FWHVzm#CkUyKdTKbA;Lfe9M9L{Nmd0Km1YTt1tCc|911={lm{I$Ng2~ zM;HDka;T8a?t7}&RqO7Xn{%wMkFkDx?Xx5A|8rgBYaErbCgCspa(esvQrMXvzw-Ov zMc%fvvf4E>j_GS^6V}vXH-eD<)7cc)^XZGDnkK)r$m^)a9-IUG6#^(hwQ=^8D?CCM z@jxA`YxtjK@IKC&@;hvA8QZHZLm=1c8R%n0&v7PdtOfv^<`Ic^6zRd8HZBqEDFe{7 zdAvV0-m3|A)QeM#6X{iy>alC(ne4d2wsiq)7?9~#%aN;^RfG1w0fJuYHvNEiv6I8^oiJ%HwBjjix2*&~y~WA!JV~qGy9n zlROrG;8fg}WRtz4YzdJWx0^xldXmt4RBjq64n8WkD?UCdw_C^^uaBr)V1#5` zgJkwkMKb$nk0MK$IFmF{&LG{5I2%0&&79kGE9Y@_t_#khqoNa|?S-z3%G*-n%~+wQ zVzkkib+l?g4SF1fCvHdwYjl0pgP416?m@k@IqTItHegjT@*`kH^?Hh)<4F7+Un#Uv z-bG`p6~($@#CW=%235cDSD;w?y}DIqmtov^wvBow=;Gm(6{w%lQX57drA;(2YCAC= zXL@Rl^9tYm+LE*@y^P^uC%n*bD2NyMrw$DV7++R(e51uwe-d@YB++9~o$4@}h(D;) zEnoV82OB2(w=MOhni=o0)T=FZ8jt$tO9cAgC{DBq#K*s3ojl@J+k8^`qSR9Rt@aVG z&CT$BhOek&jdn9Vjn#)0AFt2PjvFg?W7Ul_u~)_*?q$NQ$j@)+SMl1bNkUU=%=V^v zsV7C}rCTsyZ68nDR+`Uy-KaA?P7Jd0ACF$;0aC|)q^0Jy%UpjFcx4u`cG)az7yq`f zVeK;0ziny16j`&eKaZteZK)$$z5d!|Tbpp5sDn_Us`YYY5C(>G2lZ6;KW*dv&W`P^VsJpiW(p zu4&X9Rgdj6Y&wlnl)BDdAjWGvmp%Tc?n(F@j-aW*ylL5(EBTtcO0}W2QCwvsfcPjWIvFs2Pv@&yE|bJ;$mW zXV|?MF4mB-boNQ{MxEb_;pSCb#0nK3M#ml6W5z#8v#jpAb@iusVc5xO+7)CmY9}Wn zt{T^5p+87mqLE{<$(C?q?)N6#+lT&P=4!mv{Z{Ay?MH)8V{|LG@%p@d4z0caq3foZ znmGEj(8^Cdddq&uln;$+AKi2EIhYV#L1_6wvC`jZb(4PLtbA5^;?b+k9j&;azoYk; zoPYCK-skOX@AIx-(|j`S*hH+4MZLYFc5EJaP%LKeWyYwdX6;+LzPc29ZN|^OwJY#G zIt<5dTa__K;GdOv#AjG^Bx^ZG7d~#n4~Xx_iS2)^|IsJoTmnP?y2ktOQM}Rj&W!y_ zyh8JTb6cZp6)v1F(0mf*j2#Cyah_qM-S8Pki&b;WwP%F@;3FFdbQUKb@Xhtg6jT2dBb$j`y^f2tw1 z8~;?7mkWDTf+sxx=jG*$Z7Ik9mg9B3=l_s`$QE>s_iP`6|B~KNY&Z#uezgvDrQJsW z=Lhqg>iTbUcpv;|$V9Y^!$=%WO;c|$XuAJbXddt|*K`hl_l#8w7Tr%gb-_CJ>G zzGP9YwLShB(sITn`dvZ)q>18?4QRciu@pN>Lu3zYpVY2ePWUFO7>bpa{6c;vE|yeD z`+eE|xS=iGTJ=u;L+w2lHX5bL-M~>H&m-Qg?JEe0uV;!&?ENdUeXay49b4Pee(s4l z?T=a7tDddxX9$J(J&HhHPs#Ql$U2Qr-!-weC%;D*qxc_?Q^aj(Pq(VQvg09iOfU~2 z4dtw5imCBcJ7xdJ(D+8}6&L(x9^Ne{?m}3H)7n9q~l{n~H4jF##=31{|#~#DB4c4v+BXw1FRod*ko+bh)c+HJG(L(6HF1*wQpd(3X``TZXumhk_+OFjb0sB> zPsO4>=9BF`mhq3t_R4>$J-M{C{bEesr{awN_01qgar1Y^tm7*`szm!p z|Ecj+I|ZK#K)m*E$7xSf!rGpGEDBM_4>Ozee_6Jd>_;zF{jWv){~AYr)&8<<2{+~T zw~kLfK|0-1F9|J|OgW&ss`doF-FCE=s`+UdE;?V;yfuzTysCL&ZKvuU=^(~-93XagVR>&w@7 zH54rCrtum#RJIpa>@2Ncv_7w;d|PF7-12R!y+zB~%gVQ{lQiX<+Sa#jT(o}sP0lG;j4b_^$U8 zE9>}gtI)5i;MY!Wr;>MrXOrhITD*AiqP)DNoMySVrNdVz-b-rngEBsEYhEyqv0XWL z^y8Cr{>Z)SOJ9O<@)zWD6RE{md%T8eyQgc@QnT?O7yN#tG9a$GhWmNAH|{Z?qy?AC zy43qU&+13)f1dc?9A8WQhVuo{GtoD40`azc40;;th=Kk!b-%)Sblf-e7woc5bU#oy zTu-Tc3*9up&nX3UZGhMn;IAAcpB`sKxd&ou<2*%my}mm2WVUqxE!e0dxV z@Pb%riLT`)=v4RC>x*b>XYZ>c2@_gAIL-tUawuiTP{e{JzIq^ljp9;kCpeEh9H8O@ zRpLGGc^&T>jpCGe&zTnVJR**2VKa=Ykb*wOj~Om^LJEkhpf~Qk1+pEN48zV%$wgt3 z;L87nz8L!^cElu#6l!L~MH|`r!Ubi%XnqSSJr+zL=xN2J<;cAmwD*Mlo{Zvbn zDLwza75{*>Hu3v>carG_Da|D&?g96>mhkQo^nMY?ScnmVT7Js2}4gQIDa2dd`U|&l%uG z`s3GuUopUq^5#mtTlt(V;Z}JEq=BsRzAoX`etst5R(XGE=*P(aM+Uf2-b^`vXk25A z%)1W!h=dbA#?|aO34h1}A284x?faU9TluG_=ujbkrifeqzF5N5xXKrsBwUTF;71Jf z#&MsMaO=3glyK{~sdC=ZxW@V7lJHUUMZ&2c#S00r#Q>K_JY23L26*(Jdw^Li2UGI8 zIW=QnFu;v+HA%SAgQ9PdaI3s8O1M?tQWJqfpt zOL+?w%3WwmDcw?>pc@`cSl`)bDKBN&9Fyk^se-FwKX(7%71dQ3=d?+uRHSJc%q+c zWH-C>ppNRA+nVt+kM$iLx7Mt!t5q=6?CW4-Z*t_;f&aYZ*wl~t0v-5i2q|iFeE@Qo z`#OcV_~8=z9n137we#_Nat%8K)pF=#t*^7F6Td%M-__|OsGvtf6SUD=>U*rY!DoUC z?fIJI&)tp9;#Q1DKVMPTMn8|Ya^?C}rOnATc5h={XM0^E zXG{nL_5`1zDr7NAxV$}}2yS&OmfLs}HuDCyb<}nEeGQ|3S2WPx=o7#Nx-MpBO9qPAR@6ca%6$|VqeBX;8 zB2z6>Pgi7uI#uQ$3MrRqRb{l#CzwnNqHvGAJ0s(mLa2D9>a%b|DVIk98guilM0S)%97~p87VHY8AX2jnLTm3QB0=$*RNZ zHbJ5IRrJ>Sf0yH5ul+|BgyLKEmk!~rHX+=~fTA#8t+d-^`%&`C_Rm-vSo!w>AJ2Xc o=|7j{@MJ%71=U{V(EBm4b!=*Kwu9dB!{btZVuPOKWo`fe0e9>mo&W#< literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3f64fb35e6c6d45c9c387172cd70e1b5.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k3f64fb35e6c6d45c9c387172cd70e1b5.o new file mode 100644 index 0000000000000000000000000000000000000000..8e5e70b62f21c017b69fc24eacbbcd4141de730d GIT binary patch literal 92232 zcmeHw3wTx4mH)nvBsUL|Kw<IqBx(nY`8gh?X~yXYp=c5+Iz3F&$+j+yl$1-Yc&Zk!L+c$A5>RLxao>H79<-|l}S zosGS|)!qNv22cOn9(Lq^JnVR1H#>r|pJq91(btOt1FtVx%wT;LGr7U;haB z|H?>d$1@{0KJv^+|BHVdx#xZ=v$X&0z4x6tjxu*we08L3)t5(7AANcxt^VnecJTbd zP0ypul75oa+DHH9%-toAjFjDnwkn<;SzGeMk^dNeY2=>c!y~DsSM{I0_Rce9C;sNl z*=x~O#g|9UPWti4+T`zppBG17nD;!&T;0DW`IDe?_nEWz9v>;ocyy%y=^ufwXVAZw zM#|Q`f-=kc*F5kyXZr6u&T*8V>M9yt^XT0hOJDp-SLz)_r&5zc8&khq6r^ztc-SHz z#?0joZQKbRyTlb38}eLooN!HUw#d>4;DkP00(}!Y(d1%_o`O!eT!g#i__J=`wN$T< zu|?lS-b`Lc(0UAd^V){x{jYDWgB}LP-oEXLV@`dJrJt{PTxCi}NgtIi2B3@IgsxuH z#Xm1y42`3UeF=22&(KBEQKgTAhCV9&9Zx6E&KlTzc6rrL-oDM%|JHtY|J%#ik+WM% zy>IUJ^p7@rp~p$^qsjf({QlOXuR^zvJzhKD3-}9}*PAt+9o~5AUqb`mz9}&FjVl9V zrCyfxnkV`A|9HHx9d`WI!sM~0>B(;gHoWO=&r5!L&y${kW4{TE?fJE9?AX874m3?U zeD$7v$oo0+zJ|On)(-4h5BSG`KM(j*fGYrhAMjDYhXLmSehl!PfV%;|;~qHnO~AJT zZUOu};I9JS2l!gRKLh+lz`Fon3HV9Ce-HR$fM)^z2H*z(ZvmVG_{)Gl33xqV58$JK zZwI^z@T-A=J$C}W1@JP!zXRL__$#>X!gUk!E0F*9fHwlZ7VsRv_X1uFcsbxaz_-&v*#!}SGR=i>T2uCs9cC9Xxd{tVZAT+iToDX!1p znuF^PaLvN?NnF!${T{9$dCViiDg6jGF;f<;O(8g(j(8l&hliv2P8}>d` z6q<7I8%bkLzYbr`Rv+^o!@Tbw>YL3z`q=NAzLxYh@z~^^cck@u^N+NBVg8X_;rT~) z@0@>R2KZAvUQWCckKm8^{GFUPIG^7D4__l5GEOxAJHCI5zNepPeiGlmN8i&mNeGBKIv`Zi=FP9ed`wRxl!>{0-nmiQyF+F15ag^%!n6~ z74bsygPrqvjPr9R+6q%!z7x&&^96XYllAG&H;E|3YRTeeAEBM!Zr^v%%9`@H7`Z%>_?$H97qb>nNw3tg+mJ_BK*` z-V@Cq!}lHZ-E*RO7ryVL@9q=L`|$ln`tA~AiMcv;{aqWch3tQTPy0RIw+|iljxpEZ z)$OdWYtOUUM{cT49k}rd@4zi}_#E{PwBPR?yBlTqdL8&0fYY|Nj;PBgnWobV^Pa9xip`s7}RtLKDX{<;%uysG@Qviw?E zeoc(>t52-)sq*!*{3=;~WsLGGPOQmL<(JFyb+UY|EYFUV)v}*7g8!Qq*A1-AW@GcP zUb-o-c5EKjP-o%iE&N8{b0vOG4E!$Or%HT9G=7Pz=oHagTfoLFnuWDvYyGHWwlx#^ z#WC`?pp6oq&uLk7x{zNk+g=>q_Dy#JUn%j+V&IPgzfj^=#lU|V_$3m*J_i09z+WZt zTVmjE20j41l!3+flgM8o>wPRny`KTUTH<%bz<&n#>tfJ-zHT7Rl|9PGa$k&k?*M;? z#NQeNpSK+KCyV-b#=utqe}}{$je%cJ^{0sXpO1mxNA)HCYccRoQvDoJ|JO0_H`k7( z-2uF*H(}(va-;L_r}8oK|BlLE5?%faR6a)jaVnn|UH(xjA0z);RQ}TF^8ZfdW8{C2 z%1@0h|0I=(N~F80P-h zHmt&$b|u!fEBfEuy&P-ay8gFstL;BK>-!zFPm3J(U@h!Jek$;3D3^}%8CU~P>OYJ1 z-Z0jrf4H3uJbxt{dll=>*V1bT{s-&O*AEBAe!m}UgHI*(zk&7X=$`@uV_2{LHa)HM zO{`zvdS!k8+h4357@nJ2diKYGv0<#)-ubV<*l&Lq7<=}IwFAGyn(MjW)eig~Yq8(I z!SLOMJT~_HtF;5qzv>!z_J^*4KVaQ*zk|p*x6ZY zO3!`@WqyZs&|we7c6c1f$FL8<{GEz-saVTF&YI-ya0VvO)XTp1{D#!-b=++S*cIY5@DC%SKC5k(Y8+>9OTS zr%F)fn5STLHNRdqx@JagY5(H|r;a_Ie`*cN@1gvEzM_8*>ih>j$Cgh&br$2dIo)KeV3baWW~IySeW)K|QsKMi>$ls|8Azpr>@zYn;11V4VsDPQp_ zj`#8UxtyotUtQRL>~Id}_1N;9Q>p&?{xXy=MVbEPQ%~X75Gjn@~LkB z;mM=NmQMz~)tuh(!`Y*0$Va=_pZLMY8mpb*TildC_QEyLoy%QgB`7z%+dbebUdwqK zZV~w51roQFaM#-P>t+49s1IG|^)II~*I8w*0{ttA?p(@W*Z%_abUN~;A#Vw;m*eUy zUf-XG?~Cy96>sRz#dR+K-q4?n>umo0`hG92v-tOo{cc>LGbq2QpY*31pFMLn_fN*P zl7GLUKMU6i{(VdTBwWk+_pSZuxRMwo^isr|40 zCOG@PlN(bpyk* z0K*>*KLEHMa1LPXYllAxcq?EJVC;K`ZwK50_*I56!u5~1Vt+i`jr>~x{}y;i~i!Gzqsfx68Q_Pk9ZoyU#x@phyFCY zbi=n?ESsI1Pq~b_naf2tOo9@oyw&*lQ1~WX1NfxiGnr+J3i$3!H??)`YwPG{4ah7e zgc~a{cLuI*H}g#LxZEBW^QL2YE|VvdA@;&<7xxWo6~fK_UGON2HI<9zhGt7XZ(_*- zwuB{tqv=$#qI*wwPghU#PJC0>gL?(cuZ`V%_O%9C#lHPLZ58`&uIRa`ec!JAHr#SR z!Z%k~Ut0F>?QHI06+Laio(j>m#+Cy;9nIa{jeBbv_XZnxx9z>TQDQc??d#s(wXV6d z`Q&Wh%q=2bkaZu{nyjn$1i_XitW+xHUTM*3JK4GtPB(NEE@KZ9;`?`iI8 zYyI=-NLO2TTi4C#5k{%2t>e!^+t82ZKZ}mEw(UHyyKz@n^WHWwI%Y<5g_Z*|YiaK2 z*xl9KxrZBFXV?Co{bcbRYT3W9yQg_yPvxJ1`nEK8?rrO8`Lh^i3>)u@IhH(Ax=ucT z&Wt~UF;v~rb(ZJ;*_f~zTi&5~11u*Mr!Q9Zy2Uy|tysprUR;J;T!wtumW9!(U)kiu#x98C-JKDBYpskfMjzI=%Aqkd-rvlja^(i zec*N+nyaOSt(kHDOQ#S2-b+m^oA$Qtt=PY7S9e=aC2MTlcQY(REdMvHTe+@LU|@93 z9qrBCF$yU2>um1oY42&@zmKm#{$qwdQyeZYhu+L;=jSl;2BnE5ce@t1c}FOfS~V` z(4Rwe&C0kV6e?l9$Q_{)AO9-Kl$%5(tYqfB3K^oVw|JgF5#B9bF zMzeW@$%fQLI!bwLCa+r{^Y+NAYFCXFWfT1i-QF)`d!`OhHX>d)OwQP81k2WMS+inI zt$+Fcu1@HZNTe)TzCrnT*@tvwwd_cex+lvkfsKe^pJz z!gBu>x>PJ$%IXfZceMJu4|eyo?e*_H(B0#2ZQIqpug$-C!-nfNHEvnCanqU&>l;^Y z+q`o9ij^za{0;sZ|NQ2zmOcLNy^U3smpA*-H1SGtH{63Tc)f|;iG1$@AH$g`wv4_{ z_A#gqPxM_`9(|`Wl)O{y@8=W#^X~FO7g6tWuW0X5;)t>($_5 zc9#avVj&Hl$$}a@gLP`KpY74$BG#nAg>0J!U&b0VxPYzJ;CvR);OVRqaIGcZY0R&| zQ(3kKUrH{;Q9h5Ib35Tn*pLS2veO!z!=3_ckM|UIO2bcP$22&b^=aj^Sg(f9WFZZn z#5y%NgEeVzI&0A2G`1SB%}*)|0Jinb$ChYt3aix0Co?}_TYgC_TZ7??HQ2*KE_Tl4 zR{9QKp}{T|Vop3a5-PvJ%WcwCivd93-E}>#`4Q&64k7nOkIP$?!Ls`h>&GXAkGHhI z^10*}&EzeCM}!#f``y$>Ob+r;Y$Qe?Gv00gCIP~1LwzB6}{eB0#@{Tu`qDlHe#tH`3*0N@>ur6qP(C7{g<%fuUEo~zta*{{AGVh)K~m9NLcaL zCt*eJoP-siZ?!Bw<0X zfSr`Epa)~`7v%*#=(~gky#m%JVL=c2CSk>&A5R6?_L9k(B&_)Bld$4%NWzN0%C9*4 zTfjmRR{Wiou;S1Eh$yf4>y)tK@1%qkf9E8uy)tK&$DC_^A;4dlSx=E z&&E`UPbYlP&?iv|y6?y5H0G%$=(RUHgWu$K0Yx8gp%;HKtxX&Cc+-3g-W{tgd|xc{ zZ^CWrz4+{=(%=J$uJ6d}+w%IBye<^Qm>XqM@lkt#C#VvV%2#`l6(ZZpm!NeQU6oh) zeR0Yw{Dr=TZ*xW1Z$Q(*LqV^X_)&29 zkkDCVaemN#8onb@`#7w6kmVV0vJ-w1bB~qwWwNgwh~#?75M7|R(=T8eAPzGDlVLjH zN&_ZCb>f#8Fd42BA248Y3{L!N18y*2QW$6XCJDDP_SnIQj}gq}vw$rMvSI<=pF-nT zU)m$PkMVyoi!BIp^F|(E;HMt6q{G-RQ2y@g?DEi;a4=a7IM>H;5c8|5g8EfGUJNA0 ziv|s_8|CRM4=F*vk8~wvkhySsfW7?gfLOCb`aV{i3H_H*Bd-5Nto}f*sLw*_#4oj{ z%#G-O6#SywNZ|i7@C`#T{JsvJ9A+H6z ze*v_|6WYHH<{FY9d!FRywdHpvi`|uq_Ao`F&p!6Nhud>#FDg*`GYj?c1ikx$jN4PE z|6DpdL-O_c`1p`~F}~lo_(gq`uRqA834+AGp&x`lkfGPdT1_Ti{{hHD!tPk~F1t+? z2NxJ$6Rw{&%FsO|(Y~gyzszFm?pE~$-t=b=2lFy0uoH2^|AOD}p?ccRP~!ZB4^1}U zS`F?EPRR?wnw0=>&*X2<_PSrnk0AVbWR*2*^8S+at%KP#{j`HC{ zvkiQ49%@?j!iQ!V_?dYj4L?J}Q~Rr|_Fb4S{du&|Q2d1t6=~&rgN1n^4S$(|FUX@A z!>*ri;HT$>9C&J>j$eTKUWV$xJXA94lA{Y!6fdL}g}_01hVl+UL^RFU|C%#e+Dg%4e3 z;0rM+s`?LrevyIqXWIU01Mo8pJQj79f5Q9%`_uU=&UE@OvR@1TL>8AH?Gc22(0X+a zKd5Tk&s@7a4IX?FIc~F09rtSZVc>DHvll@Q6wEf=#j<@xL z>KA0%{!i5RTl`b~!c5!$iTY0cgMZ1i{hz4s)K{$6Gi`q->d&$0Q~OxNSpH1ZFR}1K zezw08^%Kg&@@H}6Vd3M*!}4cw zF2i!!)@MK86QhV&9|UJP>XZG`Q0+nA zbJ?J3{{hgSt`kEOY^JyG<+L0GTVU(&ZL@anvL~JG&1a?5n$=!~)?PS>;JP*5H2>ro@>`G(_iyn-w`dy(E%9ZU~^mvOf<6vm!-Q zUhemaKfhHzg^ypc1*cN|t9YPX)~E6%7CxPqFSlSm-U!Ir?NRwk3-99P7g{j*I{@*O zp!{U?@3~CuZ+`0g8{Xeu?2pJk&3Fj7x8%P8_CJ%C=6|?9fdA0^L7_H2G+x44D0%As zqBnR=K9>7Zp4WKDKDa-%^x1q%l=ZlhQ^!sH#mNO;s5!5fXn|H^l!3b{J~#PhM%ZUH2#OO_3zwb*@y%v@{bT9tWW%3LVF5u;&SwkqX}@6mn{oV~(C<9cA7m4Y4*f1yu+s0<&i)r9 z=)ZG(;a?#Sn%{Wh{ddX(`W`jjrhEeG?3A28Y7KvC#tSab$MVi0kMaHQL%~Zgb^23n zKEQqk{vg}*hXEy@=cPYXFxFr22YHr1Xnc8~Kjhz$z(3g+ykDKj;QmwdH)LPQhCem!3+gSZTk>$O*Ls6< z^ZEMUlE?V#In92AJd#a+Yx4zbYL#qvcyT#IDb{4ZZ$6*%fTwg6zu$q6=`jY$8 zbH2YuWe6;eKOL|C~V^Z;`mY(0Y^NKX(5o;7^r(usHHi{?n9) z=0CqZA$k0uORk>DD{%Xey`dg)eKquf?q6#5lZ~gRd9wV4bALm={~h}~ zZV#%z(jJNxtopC{$B#_3|FpeDdiVV=5d^I9FLl7Z!3FuGVRrw=@sFzi6rgbW1Jz&I z|8fPZ{wsgT{XzKz`(I7lZ6xY{7W@Gbm+4vfhp73|9DjYkZ|?8OAJY9YyZ`!rKYS>Q z(T)bm$5j6fe^_btSFSHDSnfwOfB2T~4o>d;{2}b0w#P{Cw*RI-G}oI|+Ilm5s8ZX1 z_Xcb7$-mn2skZmu-2XV_Bi5H#?!?g#=^t%*N%?60@u>;Q=L)&Jdbje?_v5DjBmYS2 zal8L|JTRty82)jIr4Mp{Zo#tunt%M<1p7Y|6DXy3+kbt3Ztlm)KhpNv?!O)njM;y~ zKl-iyivHXF(Iw(B%0ISJJf_ppfAa5UJW7xM(s>OlasNQ#aPBY4)Bzgl|IGMgem>c^ z-M?wZdU7f&7Y+EBnEx})^aBj~&m?XNo<&hhgo zy+I%1)pmTcH<+EvejwwKbe<`-Qr@1U`dO?ZD45}Y(O>Pn0O0vfJkGa7pC3rp&qKwM z?@8hi3G{k{sksd>XCJiCuq&QGb@fAL8rJIQg5KLzAanf3{J7j?q5i>z&T_%hGiIOO5*TwDx&C zng-C47vk@yo%}4wp#_G`&wPVkg~ku9cgUP^&qB%{@pUky>f`=In)Cid_)wKWuSVmq zH;DNA87DuN=Q!i*G+#T%2kj^G9iwbNd}y)8AHV<8>*VK(H2eNB)(?l4YV_#-kgiWx z<+7hS`MD;C%vtr1`cL{T`cLO2t~Tf`Gx!PQGT4)<-y5t;)9**u8ugbO{H(~uvmjP| z>c4$IF$wLjH0sxD^@(4<#cvAi8TR&5)jzu5nV)0F-$_2xHU6o+VyiygPl3JlDSkEp zE=*%kAk{y*fA5qhwO?Y@PeOUz@A`#m{{g_Wa>!%b^=C%4&;2hSZ{i>JG^pA~Jmt{b zsP-zY_K4qkIi#WRdA>RL0MP~bciyp}z_6Y|g%9H$ce^33l^_BRYn`6hHNdC~@=Tv=a&*AU6{AbQJ`)}(r z@elp=tNNt>y1%FPZG9zv7vx}N=+J-Y@AIlYweRrnT>fPX%=T@4rv5;GD^>jmP+#}w z)cy)jvap>{eXVHAj>13|4=YWANga(!v7LK@Q+Q3{sWx8 zZ70DhJ&H|1mdN8%?x$KFrUc*8! z9R85_o0VhlzllHS*UNT!I*(!52iK2Pv&{BweItIMU!AHx>8I`=;oqX{!`3h2e_oEg z-y{B^ZzGBx>5Ic(O8cQ&`QKcoGluAenI_Wp{-ANuv0s!#1X{3Gvww13p~ z6XP3F^+`YV`RJiwq3Pcp`Xu$WPt_Ot8nEfw%QX;B9>ncze7A-j=t(+w+6K zTk?vr|EVu+VwRGsS+9*0a|A>`~wU;q+q1Q-08e z`%yg6{+Q&MYM)1<`^$75fZ`QIuPhViA2Vjr6G8m^BlNrez&V@#>(S@iwdV==eknP_ zkM`7hrZf?D54(Cex4+YaWoGtO;U@@bID|q7V`I823M+%O;N|)L z^1-fRuD?n?M}v7-t~kqIs+Es@Kk+VF;&FOMgVX)a`bUFPA#NwVX3)aK54TQUUC^74`#pE;^!{*=C5!naJC)9>0Zt)*hG&#G z{KCiC)ai4Q*(^^8`ru-Kiwh9o0F3k89QGELNqCsUY5sB>f3}-VCVjvM`ry*fFQWgZ zz0|7s2$kDV|35SxoVv0^1&AJa1ys=kb;ADSu1;brRMiwtRXK&nPB- z<^9s$V0s4mHw8Zw^a=a2>}e8*CrOy}9i<)invUhJ4Quv|A3>ZXr#(J|^V>AunU?(ve~=_$vR7UIA;^s|O-wS3ac=cB>N*go3u zdD*uX|AT!uLEojnfqhQV?XNJ$2~XU<_4)EB&a)3`^RYSKoA#~S1MJ7J?|G1()4u66 zh|_+}^}|%Zj!$eK`g}X5P`BT?n*Gvzuj9@6yaN2n#OLD$#(Z3YC&r!pmm2g7wD~$5 z)a@@EwN88TIXxucz(fgUR*p+VB2pmo_{0{tK`!g zyfh==fFBB8!tMDI%f6>_c&cUJc@Up%&qr~-&}fs=)Z2?mBl)~O4CowPjhsxw^1IgBSYCB`oZ_cutv(hkZ}G z;PbcNY2U5BHqQU~+V}Jf@}R2!h@WMwaKL1LO=^Gq9PN*%LmWhIBDVUkndu)Z+tTc7V?zZp*YG`~6b2eI^D z_YW0*z5aa7zQVx;2CVy^aM0XeR~z%W?r*|D-9MQ75!!#~_P^MezjgZ$2QN4HgS%6S z&*!rZdpGB6(;w;ngY0{?Zr|8XP_pbhpU3BH`!262o9?vl%IXpgUbuLUgw_0AY~$av zePjG(bV$T6oaY00eDku)iom^ko+*5&fC6Zgd`#@u(H*Bh==3#n^!fP%vUj{~P{hOV zzKi<1Px0E^o^k$x+jFneo?*Y%{Y*#vOWI#JSi#6%c*1x_+p})JroO`7jQCf%Vb6); zVHp{^Jy*ir+4dX`R%!P3U{{5gW=!R;jt2dPea_Izhl7iyd~o6VdO5ym>vO1^-5%Bl zm@sYF+|QEzIM<)yV6{QN#)yZNlso+q*|T1sM~ejw&fikQKFi9T{tNN23O7ot{?mM` z)8o--2famxeU>8{W%uvFU||K>n@vymPZWPC(d{!?tmrNKO!4#9toT^*oW*v$3;W}W zl4=e1mMxL6w9oQ|HXing2kya7toYbO_s=+Q7t{W=$lKb_mKgD>G9zB4$EQ@h+MX{;ML~S zaL}9&7h=KZY+s)*Fuxk>UDN+3vX{kWy1nT05A5Ao@9OhOto^1w-z_O~+EX~VRI{I> z!D3^*tFKqdUs{RV>jJ~R$`Ji^%15_P_?HCh-HfPsl~@s2_LlCPA7Nh$3mx$%z>EDj zV&vG*E-9|m;#FmHDr`LLE#rN+H!Xhk|9smow7ps9WoZ6`y`9wT&4?eB8S$g{Zg2Yh zbQ#{aL_n-)*Xz3}$-OM5@g?QKxAHzS@? zX2f&eyS-hU-{cI$=btKLz4*bO-^_T9VQ-g4+nXJq5%#9Rus1E91ADXa|10*U#dogr z-7vxVEugI@pOAwa;^*08#%CPoBWeGv$7?YES?4clKPUEYc6>(IpM5`c8jsIRiyNOY z_h)9jX4*7;{wp-roBDnX@fK};iS^txeLbnKHxsW{E42G7i1!%#L4AE{?q6fYZ}j-< z;tc0{6#HG{ev7_7rTwbD9~0Wm2bJ4vscElxpQF8AJQ}Pv?ziac)ANb%=;t4bGo0&D ztj~@6E#~@GU$4`7Kr=q$=bf?Eqx^j@DqchD(TZ|A-a__TsmE)USn(Pee=St;8k~oK zy%t=s^{6@@p~YK*zUw*v7jVBfWY2#*Uixc$epLInTr1w%DPf_-z#ZQKPYc8~XJ!{w(W+>&aU|;Wh{N&<( zjt{83Kk?z)&wXg)C(6FGc!~eQ+n3{cCO&^pr{@9q9tbPt^0` z;E%6A1GfGBZ^TdZ`S4==j3ZEueEWpI0I(CqDfkEiJK)kS;zu-V&( zHl7l1e$(PB#TVY*9Pb0+_SU4?+xs40xj4UZ2l}CnUw&xgEAi}2i?@8lw`hX%+l1pS z=WKh68E@&d@$W~xMW64?cuRqC9;HaTpB;T3r7*#H6mfT!i z^C;%?`}%nY^Lb)=|CM`#sJD04OYrT62|^I z4_&|Y^Ow=j_soFp}he z;?bZ_TR(>Jdnmg7Mz3#m`zkTkgZlHVc;A<{ehdfG4f@gkLANh+J*e9!-an|VAH%^( z8hx6-%=n4({SbJ6g!w#AVY1(TK8NNn-y97tNmK8O5%X6;hU0xP@5}t9#Y-wLynU(n zWh(nR>9ns*BVLj{!RL1}@wmIwzVJij0ckR5kDgD}_iLtoiP5s{3$L%3kbO z{uA3*^!r#a;7Hc|Wj?<%C3(92JPz!uz*nfj{xtkRC4XL3*jIMO92*b&dQb0T(SHBo zm(joX0Qxhn`7y@(Sn2mDumNG6JMnuIG3wL%5{iK5?|b;g0{#7rJU-(!-~Zrc+weer zwt8RVIp5n6N{9LT+|2jGed3_Ue!nZ;@A!i*FFh@U1gSq)#HIy35f`8(dH9R=rUX46 zg7Lo66#6}mj6=+W359-7Bc0*>FfSkUGU^Xw_!T1jG`!iK2A?Uy`z1oWw0gh9w1ZN= z?m{-|PYKR;^Zv{Bii5M4_<@@37VQH+`x?NCAK>Q%6#l6q26%Qa;9^;RQP4fR6XIbE z_2Dn_g0nrFlL1mf|MG&x9HtV4cXxT%Z2C?wv3@qo=kA(oz@;|)T$T^?i{T)UIDPM; z;GAN=2A33z9i70tyXM*TL4Q%OtT^B({~gR1B+m%_>Icr(ySwHa<;#m39Od0z3p70H zyM2!Oi-Hx!OZjt|Dt}S zU*XZ8`h(9wRVjt|^Jv=pN8{;tlhkkM#{+mjF|ST9rN?;HA9zvIcj&9FKfs4?L-`J9 zFbM_q?+daF(r2Z=&}Vc_!YKd9E%?2bn~;%)8ap$gPadv+9?~c1dx88tEZ?~i55xOe z$Uby^^muwvUFqK^4wicuDIhg+ER%iWU?t!}4~L)4q~E83cP4-E8s0}-na!Tzh^MRXEy+zpCtCD1Y@r3sz{a*3JEVrSpfP- z5?y&~S9zqI!}x>pGrE!^vV93xL}dFC zUJ#M(OSrsT#FMT1$)bG)FRc*wQx$y0f-CTqC(d7qPwr5`FZ@ftgyj30!?$@O^!`VT zCm;XfUxxa&G2%duXMU%gB zj{?vAWlBm2-nW8a_{-TrZwig4m;0k4cIQFydj-jtWM{DFX0ZAJ2fujVht6*y^e+*= z$zUD)#pSpBy!b62hu>t@#00;yp$|#EVzkE?=|iKtIOw(I33<%uDv78b3H>QclRE&? zC$Fz-PTCx+KIu2?qgU$l7Z~cNl_YUI{$PBk1h4cq;VYLvU~{}aaS-!Yue6V6Vdh<> zsbc?#Zu9nLr*ePJ`NjMN`y%?n-m<*h9{Xhdo%ns3+kCdZ!2YQ`$(P$B=n+g4X6Meg z=>6fSztF?_B>yuRFOL_913${&FFvEII3oGs{c#^EvB!hkOKHT~A&LH8Y3UQk&n_1H zgD7BXpY*SD=bMuLoKhiAfnPQ=Ekdf!_2CIFzbi=}OK5yi-`7=~PInlPqtj4IU#4Z>jQN*}aDIv-V*CZXtga+N?W^)0y#L$}m?xB<9rRsE`tKwC ze|7=r%}FPEoM+h|AMmO_^=Fm#V1B^wX@ze~)5fDXSZeDV>YMtotZQk6{Do}~rKL>- zTlQOK%b(kCX{jY|Zoj3aB(E~d9@0vQ|1@sD3&?)cma;)<4{+nwct9Vpp40f}_Uz=p z0`_e3pJn);D!)yeqyn$(i{wN4sO-nr z<(?$whY;v(V0W%z@ZSM9=SS*8IxjEm$Jdq4-#04sa{>67lqu#Ds!#q#y#J50#Nnif z|t4TQiSMn81(Pl&fq`$WPd$uS)Idwi}oXuKIhNZwIEB(m!iM0$0XYxi{L*qJnW?O zUs&J3zLIVGqVcPqBJU6ozit{2TG}dm!+bwk&bNeze@>e$=3~`fv6tv^n}_|*=&Fub zcO1Yk_rE9AdL$F=O`am2kQMcPT{Ssky};!K`jy#Yy`kXrY;ngymCw$hH>xq|k7jgD zuAa#B3z;;ZRgwPl#LK(g`uZom z(y4!BkC3(2K3|{m^&#}DPWSJ8Jvb=kFV~Y4;ZXAv`E#PL^pWJ1#)iD2f9c#_r||O1 z-X>il^&R?9g#MOrf6cuz$p`w22wjqrkFP6}URX^@*uzY>rtdSlCPkd~L;6R**Q4~0 zSZ-KAu2NZ_}~n}^@q zApOas9PrQW_q19+WWs+4{Z{(v>$>z(;SU7A%Qoh6|EBOXo_VY(*_PkrY0_TM-Uhy2 zp|+GgX!;=KSHt?O@ucyg{hgE_6KBtHJIugOmP@^$6# z-xrp6FX~WG|5KSi zMU6kvBYCOugZ!p&SyOq;PuxEZ$??K|)YOMh9L)8wA*qkg`_TS0K3>zPKJ^FsHk}P= z?Ga9`cQJnSf?Fh#R}u8Zi}*fIfERH8I*o;_`mo2#r2Rr<)E-aFfw zFPRO>`bpeB`#GNPpFq!DB>bZqPY-^OjS|5R0MBL!{Z)AHEa49bE`mL$XzQgJU8X-t zn#p}Fr$_4--5<^9((Qdlmu}CnAHIL*EOGnu@Ym(qFy6GU62~X09}t)N;^n`us>*Bn zM(f3DPlR;G8vi;G-?IEW^ab-F|22&K)>bhgCV}M0P$%i*7q_# zJ*~^l-3JeX(Eg~)V1atbd|4R)Jh$QMGzE12Q!AsHj;QgmOx$fuuOgkvi zlu7;VlH*?k{^1Y4)0M>I6{yeSO;2>wZds-d{C=9;|3qpaPrQ%aeaOqWzlE!yc+?Z! z@^vp%<>5^ByMqydryf0P$S zL&w-~3Wq(A4~Ido{-Cg4)qi&o+WSwOK4XyYii3V0`eazM)9-87A5{Gj@}lw$%@K^_e-MV(W-u>tPlC=^=}9N z*Cnn`H;}3QnDv8+>j%cG|8U~^bigrg`#+6eANmtMlxN;=%415Nyg$zSO(c&D))P;j z@#tytHR%N=PEV7!Nw0I_^fdXK^!g@FPmKrHM<=}qA8`I_}x6W7<|Y1aRA;`&N{G41Jz`1Rp0!iTc>`WPNby+1#5e0@GQaeA73 zo#RX6Q_K#<8=rXmY4SGdHB6kIl7B1tGbg=kz~7GtuuqM33j^ zCQ46}uSst;>zgP&P2MKGZ1(U(>1pye>HT8j^tACX>GiQ=@%TG0eGalm64lq_?evFa z4*_;EQGHFmW_=&~dZPN8Jk9#26W7<|XV(8LI~A|}ccssr?AsHir^(kG-z98tqVzO* zoAeN7nJ7I?{wBSrCQeTq50hRmdnz7(=cUhk*wcyXYw|Yrp^}|WR9};?S${NfeNCQb z{XzCqNBbYad?Vshh~i^EQq|*$&~}LLSD~K`2k&z?@bZuI{efAY_LDr(CYP5FY31t= zzQx{B?F|~g*ZCI!3*lV+dr<-xC2&y!7bS2}0v9E4Q38KC5}Bi8X{`AV*W060(h9bX>oV$JW_R+}qB3Y55cdmJh z1Lq>@-{TSbH#!{oL*$Kj-g)QdTW&+bpFe)(Vs8s8+rAWul(^_$(x^vr2Oi(S3+Cfxqay?pL5NR{QI{ay(#d+$G-aL zAE%!l;YGgwR%GdqUoB`Di~PrTmU_xWiQoTb>(38NIwomGuKB~k&wF;fb9>kRXF{J5 z4TutZ#@_hP10Mw{^0_}8%slp1_{9fi%^v#3JCVp6 zU%fr@SR`_L!mw{E`WjbGpP z%F8eFKQDjhjbH!jSNQX9|Mn-kKjBLErj7EId@Wzl+wdLCg2c*w#%7@=0x=X#er|6|E-vRYy)DUG3hXF>>RJbXz4JZ;g&t6!M_ zPg}I{>Mu!vr!Cre^#cj;w8b5-{^|tyh6MO+3GhuE|Ap!|S>R#1gZJGCj_s$h0DY%K z__~a`bjQ!NCIf8-P)a0MJ%gtGivpR+)AcS_Jy}St>HtdY$P*SNcrsz$ zGs9|JnZ6ZoG8Up{!d|yagD2~-+rYcCb$r->-DdqMMtPS38+dn)j(58aI8}$kJ{@)& za5%-l8*n(;z#DMbXg};pP~NDIa|yb94LFKlgMYW#Ujufh>Ga)3`%(Qf+K-Zt(SDSC zjrPL^e8Jm?yz6Gy<=S2xPrAkO^7)#T`^d`GJ!?i@VAk|GZ?ax?Jt=)sAc=*O+^(=Z z^M@=ApBzYL5S+(}*XW0c{-C1Iy|j~lW+0P=Gvm`w38XLxejNH9qAz1sjJ$Yl{tIau zk3Mvd2FJ;xlZVtmD#PH!Zv$ruAR{FK?&+QN(*kKM4CRVPKRuAn!s+AC4<%6_@vWfaC&L{0IWuGxP}f7A#W z{kQbTfgh*+0MXZNMC*T)etE$fO+I03gmfFU^vi)Cr+r#Ua2wHh&}9&%gV}*>MiW>( z9xNS<%Y)X(3G1LG1IHLUbs(`0YVF6+!SVf+)xLFHOdkVF2OW5Q47B#6br5T4*q~Yx zN9mx|zC*sw{#!cez>m{@3?0og~QXZyeMI>`YmKKowBi4(b+CN;$jo6LnQ^Sw5mgCWQ5h`Uc(U<_H9Tevf!F47eXZBP{a68C&+}v5Zfxjj zqb90~`m-E(VP{~%PHrb&_A@SC=t2DYY8-9y@MDhl#W=>Vuf`!>ec=b<*H?BNufDM3 z`1O??#H%ml6TiN+leqE|^GN*qY8@7@eIeiY^`)J}ZC}VYetqd@;-~IT#SN%;ieu?bwgRcMY zLjGc}7k@v_=ZJCUN?MEX*l0^D<}-VJTUVBx%s(X=$+a07-a1^(k3DyJ=#D|Ey5pc$ zX;j`W!|zI1WzhK<*xku*_hWWR6Hz}Tza zU@sc28{t&0J2bX2@rV+0-AEsdjSr^?-M~cWsQ=!iL95O8M>pIVV(e(_lVV9DBdmHG z6Yrj6TSBb*?$mH9=^p8hjn|}+JW<{7zkkw552q_`lVeE3oe|DZmJ$;`DLjellcmMN zE8V~%%YjcUjc5-IH;ag*=_adT@F5P5YU9IcLdx(#4!lzeY{F8A>v1oH6daz=(FCf8?9lCf33S0tx&ljyjYq(l@Q5D9#^d0pl%fOgl*0MUu=*smkHCj4 z{E)!=bvzVMCLB(o`XvV5s^2H@Mtz(9peI)Q0jqwKz#IHC3(o}Jq;F}}kT({80h|A% zSnb>W3yJzh`ythRy!MyZH`}+)6*nbwyitG9dY@-s%=QCTeJ1Lg^0n}TG3le0On$sy z594FEc?Mo#?YUDg>Mr0N?pG~7Jg9R z4gNwFe||i*7Omfd7Ct2KCVv)wNZ<|m+x!L6IemjaX7SgT?tF^Lg_HeJ&pkOjF^nDl zZyudOqxGC?l<2tT`S4Vfgk8*4+v4NTZ85eP)*+wU3c<0303fO7wxaNZ9L?VgY(9H6 z-hJ>A^4a6uC$CI^Uy=YHNPu4*jn__Q$KX%H$KX%H$KX%H$KX%H$KX#pnHYmV4IhI) z4IhI)4IhI)4gW#suOWJTHU3!wye|PhI{|)b0=z#8ukUxG`P1>y{OR~;{&ajae>y&z zztBbbIqt=O806RBqZc$43kQw24U2^L1oIa)W@X$F3Y9QlI=`FO25%2xGr^`Y7eGnVb)L`!}UdalwOe zqL~H>)8+kWZ%ES5P8RJ2BrGo81wOt`Tn5lP*=MbZ=(}1!(Ra0$^iFYn-{r;167??k zvODRz!_V&dhnWxm%Rl7qWnV9Q^^I+wFU>8VQTZQ*-+v%0V@*fe0+hK_tm>*6en;)x zWDBlhc<-|Vu4Kc2XIS_JY)FGE*e^7AK0B?!cw(t-(3$DZuu4PhqDt z{A6}agR}A5JI?Z1tXIQlvXBN(Vx1bC!J0HUoi%818e0w6<_Euq4A|B;A6ufqDXdZ} zpUnJ#ZTTg!Yz@XrRD{@d7I!t3DY`_;2{aqyiG8)hu5cd8o|&u z4%56ta6rNV2{%cY=54}&Sn)UX z1pzDm*f9Ys{`@BdtoRFjO~8u3)87=Z;&15N0#^K;`>uc$f9(4LR{ZrnE#PgCU$V=~ z8h#?+CJ8secAevWTEdl5-hmN;Un1d?5)Me%|Ej>RmT;ehOE{dz{I3bTpqIz`BrNCw z|33mR=m9TbK@a-0W`la%UKhKg$%v(^*vJs3AErj1)#MsbU^apxh ziBA(gr)i#oUVHJAtmq#7=O1q&CN!C7zN4>rUlqU7Z)_7x?{=kY2I^Dtp(d?qY0jhc zZOomNRQ`~JUqqXf=+{V`@$ma(@XyTuwq(W3ze>OjgW4I`RF?95DHJ=KVH!aB$&MWH^~b5qKa8R2iX{!sHA?BYLBcyWwP*5>?p}?gFsE1MC>j(6srxS0rKB*lq_m&oR&lX z2oZB6HGV3K^polb3^oo?$B_K|ulUVj;@zyjM&LVTG3)XLvAfimlFME0Hb=HIIS z3t7L|%86Znh7d?+z$o{Ytlukd*3cM}e=*y?3mG60#s7VUf}meqEFDqx)i^!?!ju<~ z<<#Y*tgm3zzN)9j|7KJkuYOPxp!!tKtWSP~L;&;BQm9CRUlmuXPkuclK-B#hyBGGzfSlq{5~3u zU$&D(=;s3Z@-^Z*4HxRaRtWGl5FE9Sq@m_qDsn0QLMO-+W}msy{00EB&SR$t9Wf7a;?)5rRCr{5y+v{v{h~tQ5#Lq5faT=zp)Quf`8* z%E#YqpX`QY+Ei|itY|{5AJkRVr_%4X9#ocD_9>dN|4Msh^6G4lcvW^`SJ1yTNdUX!)inM%PU3JHSmYTYb?KPWjxUsflbMw~L1KYQ4tZLc2 z^@f`kUBAD%cKN#HTk7_->|NEfW821#md;I^JJ#G%*Scuq{@OLSY}i)iz}?i-xpVI( z)Z2Jn?e?uV+|aqHzGKz3H|*MR{r($k_w3qHzh&?CZCkoq*Ydjc+p4$pw5;B^sAKI0 zwB567`_@GVcGhg%(^}tg^Ulsq>sD5;uWGFi?rEu6cVK(XmdYKQ7FD)wt3w-YD8J)| z9a|UOy0hk5YOiYh-rXy2tX*HzQr&z*bM0QM-R9b={VS?>U*BA}UY1$4u%mwW+C7~c zH+8J8zTw*X-Rqlcm+oI#zk74**0uZV*4(n9ZT0%S3Ch&9?bx_~`Sxb?apTgC<$Lz8 z*t)x+xvp;i%9<_h&9$4>)z$3T)mpvu;EvTB67aL5vUzJ&N6WrMu+60mTZ8fd2 zY1Z$)<;GfGr+WQbyUy}@)T!Rly?txkdJCmIS8w08b{F*LKr?hEmdw`elYH!3U)i#E z%buMvWxIUcs`~X+JL*>+*uJ;pz>a!+pcgw!MxFF=^W4wS4Qw&Yc}gE4Oc5+r1-E*}c5%{^qS4Z`!eSaf0y`?UTL4>bs>c zqWwgAx0d?7t!Br~+pAZ#$JBxP-3O)ISMjpdZ3*~m-LiM- zEjzX>y|E=xzxS=Zxut$f!ttrA5#_e;+mxU`I}YxIKigJQpP*caELV5)&gz2+%67`K z>u=n?HP{hH?v|Wn-yzqkL^563vKqD#N8dIm``lIsTk8nMX;<0VDx%fA^+4>ttk@&{ z-;P^aI+k{|!cWKalV-Vien~Iay|X%w|FX*Mp>~6VxM;uYS+@m0Nv&{9(;;Ui4b3&yT;B>V*8Q;kH)4YTu4+@pN^q(?&X4;?3PFux_f{ z0i9n}x$aiE{(xWFvqP*YZmgsEZcAnBwzUVgRj)d*b1&wVZJ0BvF;}cud+YZ0g;%R} zTI1f1_O8a8YIZGNxNB$4qPE2?i(3~iYFXM+v*hxs%d1;jFRyH?+PSDvlxu0<)!fq7 z*m9t!qq)1gac@oI-eBYIw!JquZf@Jxy}xVs-nPB_+ZQjaa>5!gE>bIL05#Q(Rn^r? zmNYJ|TDY*Wx^YqC=B@*6;?Jt)j_$U`sz!0;`QmG3qxe_VxI}jNwwX7zb?s~Gh}8#) z5nWl=+}V9g`>nTjv{~>b0e4qbFWK2x)7aW`u(Pd&3qlHHXEWGn?d|}gec$dI+V}M= zY3$kC+_-Ch*WTtHi_VJXp5|3Nt8V+|mFUaP{lUi8_Ps|N=Q0Cud|NWHD<$gZz zAEqX)o}RR(fY&14e(I?IZMA*+Nam0$UVW6O+8L@WzOPU;R%B*>%KdpS;hkGCOC}Ed zb&mQaMt$O|!-@CG0!`!e3q`2j!|$S=q#kt{1T8Q7ZysAWkvMg1X34RS&L@^!&bV@g zVDd5a%jJmrsC^p|_VAsl-o#y;v=WAnixsyP=4W}Ncv5Nj+t6b6cwe|4Pv zO>y#nfqdd;2<=l+_IX(xetwC72$iS!7A0k$%i@%8r82yv{)h5n<(~r`q92k-Wxt4} z2)R+P21(Ve#&;^;AoEvnEMr?Foj#eb#^sFcuZlw}JN>Eb7nAYX60acif}~SgE28i0?-ED8b8c>kldqJ9=u8(~=YN+-Wukl>sj<>AN4~TC zkR#t&eu>m?lYXZo-&y`?N4}H3S1Oaqk5A^4Je}U#FZ1Ua`KzVUoATc*^G*40lljyx z&Fv~Jkw!vxq+-*A`-sdZ`cfpkN|Vet?J*?tiN4bwPdf7Dtu#*OIY+*9H9Y?g;g4_4-igw#-{i^Z|8e5qon94e_Uzg zD}Suw->SUw$DKxbWvs`HeC3asSaf$c?rh)J+;xy)UEbP!UGsr`Eqkuz3-y&jRvcs% zU2PrB6+Lai9(KMv79!NxzOTK9@o%l|%^htmJ&oA6v~~3~b{}YIY3uH06+7`~LL~Pc z*ejTAZ|i1>H%@X>(Y|lj{_`k(b6Zz;`~H1W4DCC+Ad~vG9w7==v2TA*TSfi)8|HWS zG`HL&O09ydHnsKC_VjeM?>x}cM!BLqDr&i{%`4h&Zf|MR@`YsE_B8J5YTny+9xU!c zdp+Q>aUYex?z%1ORtV9q+uwSi1I?f3j*^c`$NuKl#-9C+t(-B@A+RU-6rkvfvxnF_ z*#)d!)yjP>`&-*Kw|BI)PMil;+UN-PnyyAFJr@>FR0TiR;0={Mxm@6}viC z!T$;+7Nd6^dPp0-^XBQpztcW{4fqg-5h}z;SG08ZR4ul=m$dR2ff0$Z)bs-wR1lv3 zz5V?r<{i1@(g+PU3djG~a-E})ayn2Tqc{WQIoAq=4A38>ZO+9AVCS4n___Gdk}QCa zGktc4XlaRDuhBA{rbx90P;0S&0iH_AY1|^wC)+jKQ?Rr@;@HmLZPAe)b7L`y11T;JLpD>u6AowB~!e|CeY yPy1A7GWov)_*mn2`eUNvIa!&yK}%6pUya{mXgI2El>5aNQGZYhhM>$C_5VNI>^jc? literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k40f506331a47b27d6c476c5df332cfd3.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k40f506331a47b27d6c476c5df332cfd3.o new file mode 100644 index 0000000000000000000000000000000000000000..dd6ea60e90b65fbb7f808805d46fe270cf32ae91 GIT binary patch literal 57064 zcmeIb34Bz?l`mYqsMT7Kge1g{S|C7T5n4gm7`G%qfY~fC+jG)tEg%_fuO;CYR>tin z<9Hie;CxA(WD-8dlc9-mn9Tefl*x>hI6vF59S8H9$@rW3W};xn5yT54JGSH)_xqn( z?xvQ+IC*{(Kgw?IJ#|iW_QZr!~{`!>O?0y2HG#O3|97c+1~#5s@TiFw6kM5$)TD-N7ktej!fe9 zMV8LW{>AR?A3{63&GE7iedi}Xc>dG__nlvg>l^pqcmDsidq+XrhtIL*OHT}+dhzt| zp|_qGejL~Qtv?^W?X8~;FU7S5*G&(-IlTFai^DhJx&_z7g{vYzee3@7x4nM<`Q+s* zBh9BDA3inZyThAa`q}WM-@Z5;dEm|AQ;)w1y0S>*!3WRZcKU(y_bz{Yc=JoAhnw48 z8b0;F&xS94=gr|uhu#29Y2@Aq?>m2L`2*)8==;(Oj}JGa&r4fh8s3DyPAw;z>mwH* zzwi8gEf1d0|Jmu``~KpI;d@h_9bS_9((uLm-x^->GwSQQNIvjO@M+%mIOPA%aDK~6 z!?!*D(s1OB-wxmRAknOjT)h8*^QS;_Z^3tmmu&dINS`GFCBUvH02S} z)#fMu;{3gNj}I?-0DTp_IJ`OUh2ftLy*Yf}siEP7{56q_H{E@H$?3m1fAJ>tRq*ZM zi&K6+yxH|EkHs>@A-@OpBi40{KRnN#h*d0moUCJhnH-98#HSp zn;!a$^N|NmaU9hr*s@19J#lYM{u|#3B;1vKHo+CDN%&EAkml9vU}bL1nav)mse_G8 zvw24c9n($`uGY@VG4E_NvHxdn#dv-q9;}{N$w3o@3ePyAInDWus(|$`-w_#qYybui4_K%N9cu*y4#v zZ1IG}7Rg4HJq}pxQQ7ZAHhFPDf9u7nqL;7SVT*j&ZjW54V#hA-$#;Hm&=DE&Ibp|f zh@-B^jlVu}{2kc#$)_v&-CoZ;=5(gc8VJ=y{>3|b_HRS|KRo0e&3CfYcO9-%zjHX@ zJM7evd9KkXulJ7D&T?I8&T?I8ddAUz@|WJx=I_}?Bj2j%Z~710=*hpW=x^TU=#Tu| zJKFSw1H8jE&5znfz1xSJCm#k)0optZ+;Pwzt>~{!KYBw`7V4ft-QB3`1kFeG{*&Ja zd<1Yk;MW0v7jPTkn*jd;@V5XT0DL{*X8?Z<@Mi%p0Q@B2hXC&eoB{aTfWH8E8(;_E zw z3Ar8pkY}{%1Hh|reGk_XT!(R8f$LkiF2(f{u8VPf9oGf8{wuE8xSq##Hm)z>Ivv;N zam~Q>8C+9w{V}ddxc(!q@wol~R|l?7;u?hVk2Z>YWS;~V_; zwA22#aDR#JQ&0Qf!~HPbr=0eGg!>QZe#&WT=gvdS_k?O@dP6nMPsCmEY#nkwmmNy) zc`|Oa_CLZmu#G33C$aW>22U(xpLy!nwcm@oLNeCcmmX{Q@$zGhk1juUAiVt8!Mf$g zav-0QaTUo*GD1F*^HsTaa5xx=%dqe+l=`(|rP$ zb0{H{eq?{#Xz;Ufqs`mmu8>^p+=+!pc0wGQa^4yKS!qk`hwEsce-$(cHr~O~T{c*Z?o%VkN_g|&^I4)1qywLHU_nf26Z#f|g ze2?pBaAEDAL1&+N>d$M3ol;K=A=6^Wv=}lihD?i1dU_SPlu=LGT<%7HHPoN;wEwfX z-%t0B)BXdvucLeWX@48;Z=-vgm`kkJ32Db`Zi4Qg$EW!|=at^$&QWF?xM34J5oo%Q zcI;4bLjP^oIs0#~#OJuPzxhGu=)ItA`Vz(xcJ|j+hHvnCpK{i2uwU^du>a|2Y}ZBP zjtR&co5Sn+ZIXbG;hLBbs>#ba^|d4$>(5SLVjaHA#dE@o4*Sdbh>17a zV5jySr#CS+>WO2g{r2srJ#jW%x8aI0*|*~AIBllia(a_f(cdKLH%t0WG3Ym*-sD#F z)slXLq+cI{e%p3rt zPXP7HWZx^t_I>DX;0q;wZ4CTz;7cWbLk#@4fiIW%Z87jq0>4J$cgMgV2Hp$2)PW}V zGpJuD+kG}hyI%mmQQ{B8z<&w&Eiw4MUfG{$OB>;H*%qVSN5J1D@ke6dv#QX3s%U?A z4159bcS-#582D|}K3%kbGzR_&YA^BMi-CWJ+GmLN{}BU!xMDQ%F5s0dzeMzN#?rq<^fBs(iGJ=_`uB)FM*Rmw zKW{AkM^*iacky~tTu6j{(0;Gho_j3Mmif6LK%0m*NobploOnv)B67bWd5cC{iDcVf0dM&{{iyZ58vJvx$>=w{-MPQ z`4@lg9UVgc_R+t2M}PIIcl5#w75%Rw|9b7!ivC|CFZ=cT4EHwFvC-GxspxFbkO#iGptAqUw`~1yA!mHM zsci_JA3Vz-rx)^jM@Nr>=bhgE4-RgQjQpPUf7reWdf6Bm`fEpI zsHC$0qNB3^{ioRIi0`Jzdq>&m?>cS$@9o(Vc{j<{|IYW>=)2puM&7QnqaPdkwD-S} zYJ(j1{!8(;(c$|^MqB^TDy*Xqdq=Sz_FtY2csd&$LM}P981>(U9#R4SJ>b6wE*0>1 z0WXH0ml7@DhXJnwTw?FP_^qnQ#p$-up`WwSiwid8UwjxeuOcTp>Y%e8elFx@*q2}} zPrw>ZbNkWury^8`z09de&z5Y$W|~~MqWwqPe`MOrlYJQZv8$!e^+u?FqR+8;dA{3} z7`u-vcl-dhXM^owZ-zB=#OHwhIbnZsk*Fi(WjA;wE~|(nmsUnn)>Lu6fA+k0G!4A} z!{O~eb9CN_t)e2|o#4g(Y3@i~0_x(|;Hi!* z0ewDbB30>U6JGR=o;sR7;<0)EAp!N&Hga_8$jPdy;J1CPIi#2-{g@8_fgV&_R2@J9jgeZKz)0hgL}39>U$Ex38A%7o$CFowr{_G+Q*9 zHQ;|e@m);yTO*fXr?XHu6LsaduEN!wyDgH1`!amox!WU|xGv`RJ0dPz7xMd?BTifw z@cWvG9aq>4=yyiQev0vFTC^)N71u(3e@i44*8+aOJ2C~=W&D0mBnem2W7DF&G|tg{ zd@=JR0Ad|LxSzMILk9I`UUlzO#S*^sQ$< znr-j@?RT;xZ~pDP$lHHEH}aoP&xySEFIkcI;nySZ)6t!_{ty0|jsCvM*8kx*;ivHX zp~Z+BNw}7f|5l6+eVP2XvVUj+V8o-LhX7Xt&H#)(?$8$i?*Z%pj6Lwsoq+2Bzr!#` zxc(Mb?3IT)QGYw&e*=6Y;6n_yyyNW93k-IO>(jU%-7zxcLw(iFXNUfl;G0K=_7Gfi zcIaCK*NhBpCwS*s#1+6hM~12i-gS29D8ai%hHeDxxMgGr@%`k}x12>k{U;r}M}|rO zAH{VA;rE;!+KKuqTyLUwdq##9puTGFS+oPZcVuWb!TZh*EhTv0$WR8s`_G~t@cxmZ zB;xZwMuy^Xefoc#9dd|$aMO7a7vSqSS6mO@UxziQ3h|&4aiIe7`59A8x)v|4#fxk4 z;#$0zEM6c#;-wbdoJmO4WpSi1oWrm?#dE|8g|M@K z6f#P(%a4kij<1WW{Sb3`SviY?M6-ypptGs7E70Yy!=1YJv>rivsfp#6P@0Rf40_~>0y|u&N#R|F_gIxt;XukUHt`>i1r?0id*BbO4Y-~O3 zlbBtNZJq6bt^SVA+nbLZX=&8podWJGDlV_{mG~OEdO8~GITOjO&fnSSYv^nNqPgwh zEzNCR<-V>~zwbbMpw-`{@mc5Z@^9c(mHT$B_Z9o<+JnA^=2l|tqmM>uk-%4oaf)$$ z3Wm|y+>N8MJn%a@fG?2)ps}eeVwgEg+X6e{lUhz!ZP2%ZhxR5;BRj6bz!aZ)wj3!>Kc)~ z`WhOs{X`axQEz8c*arro#)} zE6<~QqM_nU*YB_9_|JL330p+FRZh|0bdrd%V)m|sU7pIE5-wtwO>iMed^tzMuV8~F zxPZN4f|s*%CU_Zp&IB)I115M0J8OdT*%=d@$4;8y#q5L$Uc|m)f)}zr6P(NLF~JL1 z$OO-4K@*(AI!v&KHJRXSR%?RivArgEF56*(=dg_?csBEz;90B?aD}GdnapE?XRtIA zJe@*{fj)~}wj1GTY|sQ}vU4UlgFOdWpYL>b)`XwRPMY8}cEUuT%KA+B6c#eUQ&@)y zPG+?xIEn2r!HH}mU|pUB<^`yhndTxUm_eG87 zo#LXkIP>PvOtC@;m<9*$z^88udLtg&A@t@DbFRK@8$b|?k`Q?$shgdXaG`_;CG3{) zWeJx{7^COy(67G=jUI~XJTr1%| z2{%djgoHaJd>PB78gC-T+m|k28Xv(YG6hWI;c%9KNq!E`5HQKh;aLJE`8bS)jQEp0 z1o!0$nB;NUT&#APfJq*LcVI4z@`NO;6{K@k&_9(;;CckE1-5 zIp^fE6R-ylLi7oI&Oj#~$mzwW7N0Y)6EFI4rU8lh#{c}|oCjk3ic zWz7H+K|i15jpKH}NzgtVoQhz=AwJ%4ZyFyuYB>M?9+hS|0Z(OL4yL<8Kq~pa9L#hP zLmf_Wg$!_C(1qcpvFNWb-sfn1@vbClFXewE=yK6`6^#Di%~YsAy!_R};LRqOOcq7p zpXn43|Fo|j%&nkw7}%ZM*I#7 zOpa~D@3p|>=tevlj1lgTa06pc^+eMc!F;|}uGBHf z8?9&|;xXhs9*jCkAJKtODtm_D z__$Cu!29kv*Ah6)12Bh^ zIE?g3ws)BA?I*OpBJ{mpXS&YvILpK;-UvTzxZoe|0=nKwu)AFVB zo8zVcqu_9Fwy}S-pO;7u!||BUzw2hnSU|DS5d6czrIzt6v5e2-W|?|_;oclJ*Cg+J zi@Xb_!+dplf<-3&=syJ+h1Ngxj|tcH-xplyrsY>3PoYJgTsIpq@-H#*5BC;Z=$E@` zIad5={ApVMHt1)Wn|W;-KHR&)qQ8Pfqd$`T+yS}%9JD?!Vl*LgzQVnWE$tU28sh== zKS%374*kP^vXp#Sj}b56AyoMMhkMg4`puY5;mRMZIO3YqA@;^ytx{GvPan448;%r(L0yTBxyBy=UL>C^`A=e z7V?deCNJ!5mf}zP$YIMld!diM;DYJ2z#I4_Yx9BimH6xW<@}2o8Gz(Z<5{N0Bk48$ zy=G%uwe^zLpKP9glmB(>xXkk_tOxKnU4MN3=16%dKTpUst#1Wb|Fr^tHm!GnmmBfO z*R1Q`7w zui{UJ$)91LQ%wAkk9@_bkF1ms@T$LX?=%yAxOcib{g?a~WEk!F%R!eD<|*4}BOW`Q zM*ryxmb=+JeLQ`^>obh;G2FY_(%*Gz!y)@y<7Q`!@~+4*+Gn`;221-JP4YexTszyi zKjZ%7(dD6dZqyIObGhM>{lkBrGxonK!)VXsPv-X7mj350F!v9CnQ!br$I^bjsekxe zB0u=h{0;GRP?sm%yU@~pv8De-D~$capImzWLGc0cG}kE4atpr#lRSOFLN^;R_FrV7 zFR}DryvowQQ_FAodRgjbi;Vr3S@^9q^-uEi4QNQj7r@SPvTq%Btue#2T`=L1z8hhu z#|*o2&9HNs8FumQx!xatjKlkn5s#|#2jzxY!KJLbT1=7O~3b6l9h$6ie`)%aQ zPj#c7@|&r+)}^rL0~9T&AnoD%vj^NaaJVmt{r@Y_-Wer-aMSq&^!Z3Ilf#)34hONv zBL}4dJmrI4FY_w+IQ%twC;FH9b+|Xq6i<+UyJ$bD@W+FhnFB`pU=ni+Zk+#@11U~O zs>KuJck!ez6Da@C3{wh?yLGyL#Mqv+>Pm(?0Jf*t( zhwjy!AN(`M`Gb|8Y5sAZ5_A4T`uy_wH{|y;{}lf|$@w?h1I3g7GipwBO#e{KIk>vvK-t#_Z~{QtM}f9MBApa171 ztmglmgw_1NB4M!}{NeLY8#ZnJ|H)m(+5Zv!pW>f(o`CqT?dQb&d(of5e-iQkzn%XN za)s;jKO|u_|Cc4K=6}RQ|A)^%?N_z=ul?iCzt=MV+J0Ni|2hoklbrwmcK#3lK+)%4 zoQUK6RIdMN60hcehKc?UpMTo0YxDnx8z*QeSMUEGI)J2ng0KHS#6YNs{O^&V$k)>OfPa zuW%6Qt}1*yNzB6@AeD0d$Ag{(V}2VB=2-X@xPl)A>@7s}G&&{jaq2ze?}_ zHQs;0{1nx{4g2e*CjQEQ)qbllSnT?Wfj{XF6NWElR$MN1M_8uVxOe>}f*od7?f57Os+*GUBn{TI(l zDVObSu$MU||G{~iVSlm$^3OEcBb~n+?JeA!Z-BRx{h90&_G+?6o)6F0&yUD|jrIn6 zFvJf!pUz{8AU}$}e0nKr-lyu~VLs1-K68!!VVw_}KiwY^Apd+9rJKqgX@1se^3(W??_*t!{&NlfGgkj} z-f!?f8~FpJx2k`NZ(==B>*W#?Z1oR3pL0<ZKRscyZJFEI9vlu_omE^jaQ2?|WB&zX|7kR{2fm{h)t7faQx9=2CgewZ$Bn8zdPb27CzW zX7oN7`pXKIv0Zp1Qid9mCo5RWVR|M>cza-#!14FHQr-5zY72b54&(i^vf#?nG-@mO zJIjLQr5+P}U1^~Kwg+y|+h3seYf8Na{6Bn-m6Cl$lkt8M&X??g8;$hP*V@t@iXQe3 z`jw<#PLKE70GHGAIRVEJd>zRr;CO=9(DN_BKOX$!4E(bpkF&HEcr_k-Al`r{d0nL) zCVYYczXEu7X~=|6GT^CwVricN4|&!iU>W3@VxZrS@g|p^FwmC;OH0q_FpbAa&*Kc^ zb9v0Lo8p-Pj~9?ka56ogGvN7rDg9LUq%jZh%HCu8GZ`@XGwgrN`*PivP5y2554`V| z2TK;X52r z)`A|1>_7^CU-BGo%gBG=Z=dhMX)rkq8PUlUz~z8%74U@=dOrd`G(_$20!EDd>@3IO&P2djENCpgcD$TYYX!Tz zM?AlFO~V%hUdv$B{RVk2S>-Lb3VByYNgw|^D)D^%0emC1 z=LP<1vbQnwNA~8L#NGsdT_5qTS}@n`&1JE-_;|Cu#mAGq;d!{Sw|J|)-H4Y7mA&zJ z(I?|WT>;kz*|Ufbq>oZJJzqv4ec0i@3DUo~KU2IQf8p^15&fe z_2z`Mr$_OwaloA<{DtP5>?t8p#B%}X1Xe{QKRMqkqmmzo$-Zc*(KgZgSQC?aPCX+W2FOV76gFn8!iEM|IcN*d= zX6KBwpGWJs{m}gA_7gWn`nR-)WW1oy3&BnvpUi+!f1n(Z?+mFzP`v(MyIvd?6ZU)qtcB*(LmobTB&@)eW3_T1nf8sDt)#*W7Z zdHH&0l-Gqftb{*Jjghb5`y@)g^XEa{K{x-!+Bw z>*D&&rhE`z&>NKU+F14fmj^5rv0o40VU;&-5_!e@fx5hLakb#4+k;i!xHz-Cad8%T z@qEWDuM=Nz68g2l{}Fu$)cP?M`9i`}8J{WsQaqk2@&Ss+55d2u(hFl$r2oTv29*Ch zflp5t>jBZz`cala{-)^Nft49z{Sf%@9f|sU^87wyLjB|frbh9=Q55he|EBkOWxV44 zOi$#bzaziT$fyO(^~e2r>QwECrfg3~4{HBp?_H()8}RAD^bAeEynRB{yneay1p{9A zcyj_7ru+%}i*%ZgRpg(deS&d6tmEl%V8bNk@@ zR=JJq@!CqEAB8u?C!T*9)}#59-=Kt#CosWV@7%{hO66SIt&3t%fB*oeIY+2 z{nPlTZ!af$TzcstJ2aAGsee=_i6chbxztV^Lh6sR$7kVcM$D`y^{SYdvgb-Wr-)q zf}Zx*QhquABA-MP31>z#IUr!Hx6_U5Ev;Y1^%U!^Ynp*RCom;bzW>hoL0{9!{u$r@ zQ@$qFBVGl4t}QmiPqM!nLw?KcFEidK@7l^KQR2_-1^HiAa9SMA50`<%t6bWAzV?ua zKZ9~UFVK8uO(A?12>|}&-_vp1QTL&P&J#Z0gV7DD`4o8C>g)N6dxq|h*_iKynR+}+ zWi%gI_&pFYANW#mAQ4~4;f31+^+&R*{JkFRS@VVQc4`^YOLC!k$&UqWPJH->)(H3-QwZC2p?nZx4|_&olZn z&Cd+{zK*f|Y~EhvcbLyPy1%EoVE=Qpcp>$jiQl&o?c>2ed9Kl)$v$T4`XhgzXY}8@ z^I7Ri{2q+4{~4TKoL7*4oAY0*e~a}WP2_w|i<`NrYVWk>8)RS8*`TRCjU;P3%~B8>=o+`&lmU@Fn;)N zu3QhDjITG)Mw+yrI@14?h0=dCd+~s^vKMzCWs&f2t{)LE)8zbNy~+tBoA$3*U-|k+ z;*or0UrRYZtY5Gv5zoC+-XxCC;dpVLpA$%-FHDQV<)5A_;uGl``O+p@zdB@l8*e|K zw^!%?&V?eri21?!-69bm1uWv@3E4i5jD`)V0NbuVZdCl=~ZOu%5 ze+PMle=sCHIG@M&^=Ul0d^|k<^8LX=4l92#$4Aod3|1)Hi}_i^>6L!P@6n)2`gcOU zA-}+R2(;xf+E?NdV}614Cr`u|nh#pP((;Y@0*`MVqkT=BFSr9qOZD}a?0xDIOTIB- zzF>;i(4RH`fP72!{DJ1z>hDST)3UOVJyaU}o%72yP+S**{5U(E5a55gn9X0ZVi zK9A#ty&zv$Cj9~SyMp^8@(qn1{elT%9v9{Q-o}<0+Pfxsxw`(g6`2hKUjU}IeKC!;* z^mXR_1M(66eOC++`UHRYpOb|oJ>>&#e1Vl0jGyQ8MFxKZKC#&7UliZs*csWL^82I` zqkoY8mw}y%?*uRBu!z5u|G3zoY%k(R0mqB{faq6nn8!yRPYOBA<2(9eD-8LI%^h$Q z>i$mt=`7OyeFf&zyx)TT6zl$*4gAFZKh|GeCAzkN@5Py-%biL-M>X=wx2t zc>(?BP2c~Whu>?$_idxV(j+6Fpx8-=X%7DE|U9eSaAF2E8vMzn_cu zhs66eT)*^w4bDqu2J9T<4_WE_1i!yT=Xdt7`OF@Prb^_u=>cgUB{8ZS*n9250f^r# z+t~vN(Wo68ViyY~GW-!cf*sHghrzG9N1e8a@!NwK-*zq!|Gm0(Jx$ab$^iepxN2ID zc+u$#C!jak3-oDz|Lr01eMN@-x%PdP5!3rE)#!f!=WF=AF5dsx?>igm@5Fcp#?q%v z#J`;V#LD0F{YfK#ysy^t_*i<+MEpJMIV*qD_p_}0zc`k@b|U_6_M(CRtv$5RP(jqj z^M3mMHD1V{VP89XUbH{X>7VBMw9=Em5j}T0PCsCxukQJf(QiBQ`53UgANhX)%yoI~ zpKBVprh#i3xTb+?8n~u`YZ|zwfomH0gfu{BXV^&((643>{StS>cYL$uiSt*Y-rs*S z`u5M?exv8-KdO%27>&MtWp3!5JL9|`-Rar*(T!=j(QNneXsYw4(fIIdcV7S0`}==1 z`tLvMIsbO^)15C&dF!uI+C887+|b$wowwNE_)awX`+s@rqbpZ_e`oZA_oMHB_s(ea zmrs54J6u1C;`8^>=$-HXKKk$f9=-Bs@4p{?`_9{=|Bl~O_PrVXKCX8j{blr3W(#1XzDso)zE9k!*}u+_9Bxe`C%cw6E!Gi;-sxGgCKc-m1PwOxmeO-0JQ zM0;{#$_M;~`J+zTNZ6)+fHDLsDmvZ~XT+1@%=n`YBR(YYFWYRhSEARQAUjPFnLfI1;`ZpGA)h5ChPJ*YO@|vi9`6PJS>Q2<&I|-h)z7w_II0>G% z!V|UMF$tcw#uK&QI|-h)$`iG3ngrj$@sAdNz`T#a{Me7^_$85f#hx1HYQR@C;Lko( zw@AAdlV4HqOr(!}+vkf%SVJxCx_=ZMh~m}6L^dF3=%ghkUK%DHTUAsaP9**n>)0CI zH!5s&DrDP~j6S`B8UD3ZGg#^07+|wZH`{F0e-7e@yK85B##%t2A82|YE#rP)RA0qw(ia(DKM){MxNi3W+KK~f{vDGnn zl3uvS)!7pbw#Zi-gPqx6XWB|Lb|rC`@M#3+$RpK)IowY96BCEm!Ax3! z=^F9U&Wy{A2`}w#L~Z<$8p=sc^nrn=3)H)g&mIH zzntfB`6t5!!^z{fpW>au!c$EBSIoYYXrFxn(Z1rZNGsq?bA*5)3q||nWM^ewvWq?j zd!Woii#v*5$MeiX!y~NmQn>_w&7{5owkO|FLTzPzX|eS)kxUeSc$~KKkuoY-sV{yX zgFjY=1UsbAbvaRw6p=pJ9w|nBvOQAX^2zo{aqN@rks{8xJ(9I3iJA8-pDl5DZ?EJ5gyWJ}W8k2{~^$riJ3Z-ORU%)Ubjnrw!? zO=DAY7dFVIY|N^=@nnmsw}DCY#&p?Y%8s!q*$lFeiJ(f>@no}qoH;eqBNYq^u-E9q+d$aZ7ssNP6txTn20A5{bcWpGB* zsAzAdXK+VttFi3b)9jfdCfVG!5poR`p@;+U^HcEqK9#up5%Xg4Ht@`iGdG0d0P~;1B!^ z!6#E0l$j+a6Q=bvY8ulHXo@Rt5@9H5H8Tlx=au@GkxB)^3} z)9`r3D8~4h#=lSC@ry7beu(kw_Ipl@uh5Lgae^!!C+BZzugkwf;H~m&{htwdi~Iu` zeo){o`q9T9irK%8#~-DQF@9Y>+!20U`S1sAC)Do=fwzpGY4V?o*}pCy!=F+c+dpb$ zDHM2%e!LpKR^ToDhctYjz+3gF$u}@b|CeL(*T?6<`(R`J2U?Y7hrnC(%QXKE3A}ZD z8vcyHTkJ2S;RglYGJe%Ny#^Uez5$H@2UoH5ukr5?_;KYsA@JkMcg}4*;j|%N0Bk-Z z)t^V)g~bLr#Zo#MqW!vUgygU-b>jq8!Zv2BsCV--UdDDIdKk`lLjX|J8SfbU07vuh zby#JXXz$IC(@cb?(}0Qa{4{Zb6Ap3SKmG{_?TaVkPrKNO@U-XL;l}G?_%m&y&^;9g zv-Q$7HTkYkD37_LcZKrY{1$JYp_EGcF_%9RHT=)^I6aso!NA$qtC3QB@vAO^7s1qD zC>+YOcIOVG`z*qwX-CY*sgB{*UljUw6_pNgVNCY#RWPnZ_s8yUQ1bW6{&pDp!=tnn zZbm1@RromZ4JZ|J=wA6h-K$e&XS(73fRoWsovWPeZo2OGu>1aU{$qdhmzk~X@0Pst z{$9slEnb#W_|tjMK9rigsU>j*Xr_zv*J6h65g7i~dJ(>hXoL&tkD?m>_WBAoXo3sa zD<*h3J7OQo0Xg3cvfhlcQFs(3XL9rP1pn@ z(wkrh?J+Og?8^RWi7>*nrWj$klYqB7xlh_EFadB~8y-bDW=ENQi=8=q4x6(mnVrD9 zx`X(n9f5!4&*4#oB1x>`%$tL6kuv6iqhncT1NfY|oiT3^pF1G*<`8qPzHGzj#0%+& zyUgY$?QpnI!nG2nb%XFB371RwgoJ6`ApC%Yy%HXjFs&PehjnrO8zl^D74c@ zED$isPjKxb0h7E0pU4+5$;aE{%QV!U-1k=0Tbe)6tRFF~eHHB=B%8RgFDysgn zgg-=|RIuu*hqD~MRW4N4`mak-to3VnE5S$G7ou(r4Fcr`C68PsUli3^y;`l;S?eV{ z!@@^avy4Ud`*JQuJ!}8D7t~g@#(j{swM=zp`wqBj8!o)0ZgJ zc)gM$7yd%^)+{0!qBzuFGL*D*S;OnKZ%AvHy{J!5)8*-t`lcTn$OCjM z%DDse%MJA&N#8XN9|P-;bxYF=6^tofSyQhcbn+T$4jMguw=P!uXJxxv^apt{+tYW| zFg0U}`VxMV?2u^a+)at2G`ANDwaP@y?vYcm(So#qjOix~Jb)x3DbgUs9vVE?qkXLK_ z9HEd7uZ8YI***<+M}_8?;)}KYJ*WVSG4dzRqE@hR{;zH$s=b=WhrpQXypm2`3T1l* ztNv9xHUEdvc%t?}$$;7uowYs15mJF1ChlaJEw7|t6l$MaAIy}` zzg2$n8`A0JN3||YH!6QnSJj^6__*z0vWzMJH1!5R`m4Gc`{O>|VQpZBYo&o0+wrP# z+A051SJm!d#m?Q;2kR?DsoGju+|pfNQrWVvWallnRkZB#?`i1Xx3{LKzID$nhsthl z_g7SHt=e7LRNuOxYyaMwmimsJyIMBgUfEDq(_XRZ_U(I%47fvG9d)fc(XM7o#lAha z+|seLx@E&nw;b4ibNg)-O$YW@?{3|)5$gK_wOt#Y}{LkJ{m#4|CarG%8t~P+(i8q z?Q1=_{Nz;Lb;?+I-H*TLqp8bXXJw+|`ZIjV;bT{lNI?zzPIzZ!&Dc`2s_mZCKsy7^| z?;plta~=}>@8`C&9nO8?YC9%HpSaE>usv4 z(WZER=e|ATwkwtT+_-P=<^!;wZa-`$md>`eNjciK71p=zZmNr^+p4V_s<##GuU_B1 zueGIne>Fa^i~SoncQmx_?tx8iy18igk%o<%TlVcKHrT}OL;gKw2mCR4>ho2#r>3K> zWp&}cJ)1lCPe$9yY1{pKY7Xt+vvQL875$UH#2UM1FQWg+>~1rSdvD49!~2RiG{>}o z>VrK}?;AL6apNTNHE{hv@8iwC@(a=T#B;9t-dNwd`u6?1SKn4Y*|^&_AFi+7J?Z>Z zmI%6iZ969!&;Fh|#IwC6)sxV*NV>|yb;UiC&~`}LZMW^)6Kokr@0y>*Yn;DS4X$!YWY=;XDe!cWxq97MYn(B)$!Y39lSbz)lFANxAE%aL(Xwk{OXIZ zPQLo$N#)zbeXV*!+y1@d+3IGakF?Z}w|1{X-c-0BHou{8>k*lMATBlS7dgdkm9*aN zE^OGlxqEN%hVHsntSfu5W)x$sShxAezUI;!RCecUZD|hp4wV)jC@Wl9Qc~nET~$}S zs$pe)>8h3WWeo>PN{Z_bG?e%RU48Qbe|@8`zPqc%-`VMFE%CJmeFqy`5BqjCwsp1# z4z@P7wl}XVEi%F;U_zvc(gKv0`&Jf}mimgyN>=)ceJg#t0^N<`pAG(&&PE^TEj5Kc z@n4az+^3!j-ZB4BW1y|ECDs@uMhs=EzoYZ^<|9X18Z~&QfIEwd%jdYf z2&s@dKjdiWYyqOV?cgoVZC&NQu2#SAKzpFo-=*e_?8hUQjc z?4yrHX_3HJIR84&9iPKdE-NZ1;!~(Lm$oA{cuqObHYXW>y9*e#k0PR6PAUGuu{2Y< zpqj7H28PP65k$+5XAU{)cK~c5BL@ZM^oNr`E}e7WeBE zk98?&58J=N|2ESy176dK0l&q7&$F~Ax#-UosltEUh=AG&#sxIBd+cMhBi6CX0C+js zfAX_+D;{qJS*zsSkF6(>Y=(2?zQ;tS));m66@Sr z(DR+@m=+F0zX~bnb$QJ7hC>{Q5cG5+WiIyvOgHF9jjLSND?d_p@y{5SsxxxJFZOai zURkO9CuJP^W+*xqpXuY&-!M-7&T;BrK|PIYkk14D0=2x(9Y>xa9EcD-oo`W5{<(G> z`UcR%lAr2g)qey&#J^7>mH*-_h1c((!x|LTwix$BUn}d^aV%rIC7(00Ud_vSHC|b- z{PcnxSDHMZRsK05`FN)b#;X2))K66>+(6wcO{Ne0slHFbm!v#T;Y#%(ylP108F686 z;!5=!Wt?-$dNsZb@Kd%%s7#5UW~ncf^|Mf~+EUf6vVJz|<&b$zuc2NJnb(|@_0+Ev zmDi~Adg|9$zYBVxLj09%1e2XpvGQq>_46!z@HCzm;$wUY)Mu#Ies39+Ck^#R`X7!{ z@5cO6A^yhk;z#g!vC3I4>#gG*loMnf?~tsw@_AR*TgUsMfsaxCInu9)zj3_fhI-@L zyknetr!*SkV?6R-W2iSC^*wH=H`2dms5jEDk@jun-({#b(w{Tb8~MAXF%cgleX^`4 zeHx>8uB=~bsox@v-m3q-vfiryTV*}4UR2OFi6kRQ?6uqrLGp7=|Za8!q^xB6qB ztSA0Pe;hE>%SUOP&%1_t8ESaF`VnE`W0Yr&RF<_~jl~Iias|~lty5G~y_Z9HpFmz! z{h;)puUP6;ynn?~ui|tX`lF)wsCZmxsaNq>oqsF(K{@^o3%!cRCoT0V9y5`2clzp@ z+x&qZhP=GNzs29(R^N0JPt?~3@wYfwL7=h4U(nSU>|$5DVIlej*`zxOS`|p z*VXQ8;DU((K|CR+0L560BgEcGC(!mv>)Yzv8ya^tw=_2V*JWV24OcOAf8b!JuM?X- zR?yiM=R18)R~WIFy)76aZTMiqoTQ+>v#V(3RZIA%zwf?!d-xMP zR1n_X=@ZvK(L)6h`TwwgzmIiCrd%dQwZ9PHpUXT)A?0$QKt|^bRA^e@wX N28P6-?@3zQ|1VfvmmUBB literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k410baccb2ba161cb240e3f170f960314.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k410baccb2ba161cb240e3f170f960314.o new file mode 100644 index 0000000000000000000000000000000000000000..8712d17ea8eb7a850506636b7104c8312ad8a232 GIT binary patch literal 75688 zcmeFa3w%`9buYZH7W9yW-P1V}t3#6tobMo*B8gp`qhBdpY8 z;kwvyjUf3NyN{GFaZ}`)gr>a>O50d@{9-o`hxofq-SoQ&u^mS^jzbm4ksRA||7)Lp z&e{8DKqPV8-drDBocW*i*lVx7_S%m*=gfVZwr_Si9LzWz?DNb*##P2H=2%xl{BnpT zgEPpY)kCiEce8@7f2BWg^=C77?nPJDlP}@#WmndsAL0B8{(g+l5KT9D>HN;U|1|K+ zBsTVDjdS3QO4qY zJfFOD)j;c|r$(Nf^x2Uu&wXp;)t`KGWZ;pXjy(C)Pr+9-F!1Q3&);+Dk>?*;`qaqQ z=Pr%39(!)&$w$65a{X_9I`Zn#*TA!K;Gst!e*Vd&k32tsvS0niQzNY?^VJ>Cjch?# zPc9{%I|r^m_3-l#w>|oN-nTA|Jp9L>8F?t>nUO`Q&y8IF$m=7EzC~r-F^~uPBK)=P zdBz%R4v!?}tsA($?ZM|4 zUHaqauWv(H%RW7FebTo_wtAj{J+FZ6fh}kM`1yfHp5!#Tp6JLL-SU}-s`Fm^n@HmQSyvK0(dxu6Wogu| zAr~w1q0Jo5XmtZ(Y^ozPcFi^QNut#`S&v|e-^G0Raz?(Ith{Mp>JNJI|q8!ih;`0rcW04V~6puzMh7gON zN34Dji@#s77`;U-UYH;jFW6#{V$_JoAzM5e@q4S7ygqNR{d!r!ciy?zG4OVWbKspa zcJ6vjp8KsMu7T088*!X~Jn9+P@S{`be~#F`_?41DUnn?-x!tJ)e^)Yi!=0hAx*48#S~EQFw0zAqc<~3JvDRa*!GV7Z zjkSE=F*bbP{pVXh=NKFKRLLN#7ko4YoW8d=j`A_$A;7@ZaEk80X!%z6{s@ z47eJ28}I_)j{vU%E(6X0zPA*1y9Qy;Sj$_$t8so4=M^}Q;Jh5?*KuBq^Q$;7#Q7DR z=i&T4oU?HLF3vM?ejev(I6sH;6r8_~b1KeX!#N4(FXQaR`HMKaaQ*_$G1&1LoPUM$ zr*Zxz&Y!^f=Q#f<&OgKXQJjB*b05w>#`y%!yWx*NK|L1Zd;;fhpkDiMZUlWU=wY1a z;M{@pD#uvtH2coI$dPrERRiLgE1Ey z{8bVgr2GoGbM>!$d41iBUtVADOuJP7IzGQjpHnZ@zlqNy^f~2H{k!=57JZ&{iR65p z$os))_4H7*y7e;&?*w-YyPwU9273M>VXW@^ed}4pMfXLF|G{e)=CgxO|ETVB3Ga}N zbr3;7TIosVbSo(EA@AC=|1RR1VGKTMy!m+JonpU=~0 z&!zfL;Pao-=LFrJ>Cy8&Z@R}?Uw6Y6^hNF?^t(TR&kjEQhjk-v;ivhqX(4P{2%8qd zriF^1Uc!7z_LJF``%qprmFK=xe+Zus&}Y}B`os9#K%bqL>W|^`J@nb3w`#pSKc-`gvP9E~&1_$EQS#@a(V`IStcB$T3c`2CSzb8$T<&h>pLBbQ(2x=cM0Twfx}UNye#qYr|fFX+W_=;uLSDd?Ny z&_50O8bRL~hyD+suM_lrap)&N4}mUxVA}mPT;C|<4#koCF6b44emD;OPe9)uNAD}8 zgMLTaC~wPSapc|w{eD3|6^EWt2KmXl{DX1m%Rs+h(9g%A?<4tuF8|3m^kXD1=%0&2 z{~F0p(dEA%hkl}D%zr=V_PFW8bw~R6>yHwDoa=u|{8PvCe}eeqT>mujXN>3n4DrXg z{srQnHlF{B#2@GSmx(`fJpb2-KhE{96aVz_{Lc}8oa@gM|BUhc-zEMy*S|;nGspA4 zLi};Aze@bG#`C{U{Bf?25dZA){BII}oa=89|D5ss@0Jbv@8{Q*yx>QC(3-E#nR71J zk@ZgwtodjxdK%-}8pDKu-jp zAN)y>OU9gd(!h1h`-U+`{qcQl@Rd8+*v~Qld?Tr3@E4emzIi${_M;BW5k8tQ@Jq~7 zM}HL>9K*czhe`gtw=kc5`)4}`-uYC?;PAr4yzAc%jSXY&_U?a##(wxxXzayrlnlOv zx!21tl??s}bFv@(lHs!h*Vx!AKQ9@4<>!vU7r)^c{4wTlH)f*z7#sV~#*)F2kB~eY zdlmD%pG+woeD^=t*lU;r{&ZgH;5(mk48D%};?K%T2jBi#%D@kkw&uO`(M^orABXI$(!$@Kvv4Y~&+kqhoM*HOA17 zhQ=@!4&ImvJdKSFV;(uY5Z6BoKcoWxBk(_hmJ0k?;Dzw>V&VnzL1Y$ZGv$;5#u+t>${at570HNWuBaHZ^@y@ zON$3*$iG|pC1ov7`%SVwcDwvJp#dtN__G}$KG$vW$1mfZd%u9#b0GGxHpBQi8g?Q6 z+=#z~ftV}hJ3i?0FDV&FURgSjvaXEl{e!QE#?qks`>xR7<EX`qg%2|@&>*#>&nHi%)GJ%{4I3-ukRRWfy}?)?_$}E zD`)81>FJ|qIG%pxBFCAd7dg(n@+`;GMu$+30={_!>&TV?~vnvNKo}R*Oy;wHoN@B2lU=jH9z%x)5xRUtI(Ablw1EWDl z=${jDon!}2PaeHkHW~UVxV|S(r;Yk?9pz%35`-OF%yNcLI_dh@s~ZqIs~uyx;2S>T z9Q5UE<+cqs>h$5|f>uMcZI*nwkY5OS#5$K>O+4F8o^{ZFC+S^C*LMuOia4EtYtwOU z4bH1^_T}sx$iU|!{P}V!2hwp~$Uj#NcyON2KkpiF<2;Xlt{!mWjFT1cKX08 zAI}_k<1eQTTsVDb^poD+yiccoc;E@w-J^e17QXV=U%C6ryEC1GKlz)ifuH`}oPnSH zH~%eT;Fsvvqv)q&yB&jX{WTl=RheV(?Y~4nMZX_jh`f=6^9t&}C1b-M zr~X?yI6My+`Dpkoa5?Z4V61P4{|LAS*aeLB@9=%Vjle%=Xd|3|f-~01!(F(368JxW zHvk`Hh~=s)!{1NI(cu-qr*U3R^qMQfyK%h?=WQfc zGdesE*UM_JKn}Qeba*D={a1z;6W%{MJcaOqE4U7PV01W%^xQQ%?8W&jcU>8F>1*JY z=k>gRzK(sxP3Zd@F$R?(ACw|5lpsHUP02|g$};q?q;ofnO?PUz%p@o}T+evYl{!C7r1>tLblf zzdsd;GZmcxum0Y2VWVw%g>KfsVo;HDu_4!`&m+W4j7LE`7oNyR( zP)z0P+rs%_pvL1(_2I7ef_yF9-FT$=SbkCX$np9}Q>4DNE!>T^0of zjPb9dvAHRXsdhNh+Q?Z>+8g?%t}`fs=5 zP*N>TZ0#3rf9mvq{@TkJSa!EJw=e5Be7LK*JD-Ka$4;P!=+pn*J2veI>lAdl`nJ~k zt~d-v_v@^Wbhmc5b{ylg5>p3h$Gq(KYOZ;!5jPJ+7j^;c$69{fW53J}^6QhvUH@Ar z=}8@T{TW<$vPrD}tWL@o962ui`jEIjB(5jJ6c^vo5EsO09l!2ZF9Ox#dLYfRCrWo2 z|agjdL zf%^2)IW_tIXf&7kV)sXLef%S1bbX0kiNJ=>+-q<_mvuiePbZSxI>ASu>7a6=ebHQd zF}^+&n?aN`cwIlv(=|4imp@BfYY}IoT%)b%GU

%X>kTXO988j2$oBT+Y~4Y?kfZ zw`Jp&l3-a!q!Y2EAGCQ!`)I51cbG19c5CSGWf*6JwCmK`R!^Nf*xk|5*0m&v?AeSx zqF}*_Wh<8i_t9xt(OOn|ytS<<*wxe3-P|5*Ki<_HY-&E-daOCPr?PVU?(n`%)w{P; z?hJ3L-Lq-u#!Va9(#qh9;L`d?V@t5BJzS8#x;}`a$yVBT!*>uiTaU5_aoxS#$8hfy zyN^Ck_A!JGKj^d3^XN12(BTePpKoXT&wbd9ScKeaw_e^fvWT%l_J)hyn9SS)7qA-& z&L@j+WSjKm?3#j?u@@A)lwDQu685Zu7qcM+FJe~|oX0LJIG0^i@IrP$!3)@93ZBpU z6`aG)D0m)=DtIo_6r9aE6&z$O3eIA63ZBDi6+D|&DR>sEQ1DC^Qt%9x4_spUcRC9y zIFqF*cp9Y?i9dtgaLRZpyQbiDc2&Vs*t5V^dk5GRg+7^GRB#%*pzx=%eubXGq6(hG zIu)GE>J*&Bsub*J6~LA~i7W(c#hZ_2FLvZqvvp3AuxIt$Bzjd68M6^RRUiWxK7~90^h*U zP5!O|_B%W*EuiCCfiF+dG2__JI;ZN`C-4wDE6JyE>}EHn>o`;3AO=;U2L-O0qvKqG zqd^_#3w(L5j>$iMhnro)B!cA0AB58u>X_Y-hdrv(4gb^$91{FP0#^u}hIu8~uiF=3bpq@51(<8i zH0GWaWKs0V8)*9w+Pfe9|1$or;x8Zla|k|grzP_*`y5b3f9^TRIyhX@M_q2ekEP&q z)P>Kh#Py^2x~?66M~E9X;Gpxn;{1*{zb(!ybvB0Xz{h`MF7#Cr!eLxD=42c7%jR_f z&2JM8e&hOuoA4X-_v+ktuOr5O0Bt_}y_m|T_mCwxvhlYpg?**R&mn_fSwjlz)4-^^ zjIrlZK~L1_^KtrQ3Og6^!MpI^(4(pB+2yQX3-~A&W7lG-q`zNF_g$3mbj=f^xs)>l z4t+6;J+7t3xGypg?9W!RyQ!>v z`eh(GqAvDoDtqcA#9IOV;GdhqHua!yk$@9)$S>D;Gwb}oAJ>vSQ6O&b%UNu&hly#Q z8|67bPxDYfar!q?iH<;{gXk{MJ)WpR*X5D11pRf~TxZi){!p(@7WVdO^mM2=UR=)l zda0uPfc`A_AGOaE^q0w>$tXXK#eN9?KAQ^vO-fDz+3?Tfpp!lWf2~W%(rHx+``q!E zFPS=K} zXM4OV@-HC%AKI?)Q@n@(=s$@1HF1pkys7Ks z@;BA@nvZgCEOwgfn;vnI?tuElGw0= zV>kMbfzR}S^-8kKN9DilU=@b{MSryW+X?-=e&iqCA60*D2R#ZBKVBjKxh?-apUTc8 zqCCtoP-h=|#ew|fjz!zSP30rMD(#8-pnv;l`C#PFM>OW9@;ie!lGyXhS^06F&-6F) z-7}{B&r(uidEqpx?0u@>j{!as@2LGKK~ zKWP8QHMiRwFOVNG9=W52{MS?2mE$apdnk(f;$}v|m{w3DLZ1u%?YH6?{yndG5-9+A ze}H}Gw8TUTrdTfQ{aPw}LQD5>EbKq8O-+mSOPq)!$q(|Xui$2pvpOsFN42wS7MxT=&Rp?n39sZ*Fqq~UulfA)oMssk}-fWAX z$LHL1w46krXQSt&Qv+G@^A$RkmtmHdz{^MX6yCFTR^6kc@jcgdz5@R?@HIM7~C zXnwyOpMAYaO8@z|HY+U(R=xbQ;7?7>2lk41?(0o45`rjylCl4#V{NHqFY4F3M6nO! zAKYV%e-vL+eLkbUUrl9y4}bco3jI_+2el_2V=(L?{%l{G!O!C_6MR;F?AK>;TFir7azc$y+Xpz9_ z%VBTOhg>4rGtb7KgK@{Qzpr;bpL$#6ElhO+SmiBH?C;kWxua%zE#gKk)oa4KDOkeav{pcuoA4{}R~K z$I(5jz&{!OcsT|0yH1}zW$sU6ubzd!;{*B&_diqp6QnB8_$vIF zV_?j8xqoV9|E!t7KkE3|r;#<{;QpcA2y6a9`A?sWP<=L_KKAi5HI?sQnEhdrGGFZL zO*Xbad<^l2`k}$p`lXWzR6nf0sQr@-jCj~#u8(NFf&C@yAJKfFv+9P6VLeiwD7JIN zdTANfbLN5iEVW@(ALR2MV@tSzzMgpi>nGn8*hlNTxpT(F*W+4}(tjV<5@Qq(zI5z= zVLq13Q}E%fk9(s+ZR9#`>iE_zS+{oR0W@b2;el#Mpfn zy-!O+Zzd1m;O#Y;g+eT3;@$*hzK8uJZeKFl2mXF-)>K;G8~xSZe?A^bbIbJeT1xVe zR6guMb&)UM*4KM|8lG&B%Kx~Qk>-^2e_YGt>ouc1=tIRV`SaQ&O!s3k@2H^<8xwkM z^!nN6Gs=1x^UojmFt57~1@ivr;5e;C#TlI{&eUUq?;k*Z674NxPcD(>uh+X+0L?)@ zgui(I!cL@Uzb7dFJYcmC;?cg}pxDR#jqTNtUK1y#!eW<+yQK~<+ zkKW!HX9UK65YaRHRqTFD!5&+CW5dE|Z`=A0@nvsMb-$1Dp>6y@|D$|}!$02^!~p0FQRe%vj^Q`Us>q{IvM zp9yk2B0dw;_;V-Rc6?!crT9p@K>o7gCvZ{48J7hX@#Mau(3SX6;sgCx!SJ6NFEpR{ z4dNwJj+cJlhqyU3|9Ow%Wyp#L9xu}Rkm7~*W8Tkrk@FSu%YS{mz&|+5d=q$9#Ti!x zHsa+4h5r7>OHhuNS?_(kQ2!M3JsvO8`kdk=i0T{k|10y$HKqN%fngPA+z{A^mr;fO z{>MwM94{Ar^z{ZNci9@nP0v#5P&+0E$z>-#=^KL*>|a{SaO@iTt^757jXC6YeuSIPSyeYXAA^SGZ+ z?%%jyVCotHZU zoDdl3u=@vC+0QWcTQ4Z>=S{h2U>DzC$+&Fb1bsj5ion><$No!V8ZV6YTlx1p0ds%# zW!eu*m;8Sg{VieYI`HfE(|);M+3&>uv2A}-ZV%eeRQD%ozf|@Y_LnBf*uI~Q_Vdg8 zPkq`XrM_XZ^VbU&Zw{9?;*pV`u=9}vno!x zYWP>|f4`v6J^J@I`u?_39`?hO{_-sCPf7O=7~fx9K>tWH=KD9S`}=Nl|7nOeq-lSf z?^ot9g4TYe$H(4~^DDkyjmkK~$KI6jG#@)-;m2`5GR3#Y+^?L%ajL+$e{5<9bR+&B z*D^Rg!{ndF@idb^)3$%^@yPx6ycV3I;#9cT+Rr?%&CO8r^P*`ge;`xkrza?__@Vt& zIX>wAV)Nkn2I0yc@bPQFh(G1~lk-|;P}QgI$M)P>p{)eHCIMgO2ady`2D-K57kdaD53Z$R4QRSGc`%f^vVM`<+yMOT8MHEq&;& z#=-R^v3&LbRYBMn3|jq-@^QAJ5A88m^~XHLzqmiZ8-*&fJlvm1w~FH6N@&)=Q#&;RdNm1(CyMX0D9tYx?^7}%? z9^RghS>>TV`MK_uW_|4O2719F|0**+aQ}(-GL#$fhx?tpS0qgR!y11QFdtLfpW;Q% zm(V{uBRNOt_fmffW~MC=I1%>e+Uk?1)Tgg^kwjk({>2KN;>Y?vqmuHc(%${rJnyrT z{d9ks9G}#_Lni$!gnDImLc*E33{+jK* zW@4(Z+8(O^slIZ4pm)4Y9< zzl{8*w@=96|0=bQJl<4}>kqf?PkOvt`JD0%=A&+_y{Law8RgM@Tgm_9$A`H6W9U}x zHP>4!=^x)7_VG(?550dF4hy{3VSSBSo^E-^H_R=oL@Jbvi;$QU2=`PQ?Nz0^MPekotC zm~@-}vA;Occ%sg)C|{}T&pzA_Z;dCZn15k?VwPtgZ>T(VeL>^l3ktuy-XMNCzm3mN z@CWjZF&;Q5UnAd$`4gVk)#F>lyN;FlkFp;Me<&FJ+kXF^a=&k!=V5XGaJqRvDZYPK z?>Eol_piIre>xlV``7Ky(-vtiXB5wGFnS&u?@z$}?#?Lj*MXn%&Aqr?wFwTRZ86+W z{si{3X@}5y{x&(rUw}{VPxxdC%g~rBdIogx(E4AwKOWzYfX>oR3Oe?Kr)w<6?@*`r zI{*hj=k*zaJ;0MAZp^N6?b=y-UORbmgkGiue1l`?OK_Xzy-fPO{2e^JeBkNE{iW&9 z4;;Y#Cl@DBm^2po5-&CH9+J|Fq51z+2?18?%dr@>7WPM;+ zkOF&r-VidFTh-u%GN9(ax$)ZcjjSxry7Te?Uv{41vxu}jsx+P=p@s0|7Nkty*?-DhyQr}P=Ee-j$Z#%H|$9y`_P|Iek$`u5)*a*p_;sY zS|;BQ=J8kQ4rnKV(cl!noPK<|MNb9&ZeW?-jmM#}Unk=CL?po9XmZjM74ngpGCiPm zfnSzCgs)Ew`Lhi20X#1fy4!8Y*Wm$WgHHYHAoR)d4cM->=p^3_`7Tqw2@fz^^neyd zd9wUrc*mqWNFFI5bkLM<(Z5d<^$TeAut%1^N8kUl>K`}+`LHS9hUPN$!~RCtE6X24 z^I3E%zYg;CrhEs&+oA`wX4FTP??m^p=p^3&`9@Q|2kFnE2ecy;4SM^2HI@6j3H;5( zkNy?VT2Ws?2PS=oK|f;BkD|URy#@4ElirT{s`~E%{isR*5bCSSw}IYn(odkiD*YJf zA2R9pqQ0vB4$x1S^j_3gmG1=oUZNwOi!`@0ivA+{JMtU99bdv8XB6qsqI;dpX=3=} zRG1CIjMUZ`EGJ=GCfeF_eYc$)KX8)-x9z>x*?uVCikY`et(OOeCq>+O``t! zVrtst-hfA?Cnu})bgxQJNs;O3eW@`#U=?!K=O*Qj7%Ac>L@aLa7Km2olPi+7JtmnhS4$zk@HT7}& zo`9K8Iepe_m0lGv^DCz(Pm<`!x7!0|zQscN{+@~yvpkMBrkMIU4yBm-I9`{c%CA|X z%CB6a%I7ap<#T7L{5i8!{@^T?Ka*a-r2B)HpPsDprzNZWOO{yn(fBfJmWq>;P29le z8`5~1%HLnrPYx6Buc|=&zDwn^clmtIsnPHgjqy#)|4Gj?rs?CUc%COI!cOt_y{gau za=K4ZINp~?@sE6yL+^(`e>28gU*wd|Peow-3Pz3}$1CS>geX3J!N}crtN7$e6-Oc} zK6p^Yb#*c>jvT)m;}45+)`BL^b~LAUI0Y?SFmT7+hoIC%a*zUXeI*F_llA{aS*SmqCCZEZAP zPtrJlS}=0th(w2cgg7M^$|6n2JC2 zAr*IYsQAPQ6?b;3_}+U}+|wgr*q@XXg#CWg{!_g&y*Sd+it*KKf3#m4b-yGrUKB?< zPGJ0#=n*UoOgiGL=M=^(osRxh961!m{Djp}ym1_EYyfT&cye!B8+FK78s^LJXK|$N z2*!WbV)EA?#dyx@MZBKrVr_TR4K9>~@O)`;q~SvtuUUu5-*_*^U)E#lk3>xUC_gRI z>}m$xe+|nt%&(y@QfKHJb}_sktC?=7fefA(a80nDrSZt+#010`kKFWpVTcM8`N~{> z>hJTzc<1Ba-}CqXy}aC7Uy1c>dE%&&zmT75rTLU6pd|rY@tmU_+b_{EU+w^w>9q$W zI_491`zaqLne*l5yCgc~_k&-SKd?ulll=L|~vF<%G2kViZdKk}_UzeoIGJ`R7$`m1)?^}_&t zKF;}ffiC!gNgw);KHui_Bk-@RuX?wnkMpCz&@Qs~t5p7O&{ciN$I^Ve740GG+rC56 z2mc>BgaSjmNgpqN2k5H47SQ?poR@bL?IY{kS!vgY0z*4YeQlu2`YI2}^ZDsoJKi6U z?nU!Olz$>pS&0HdZD#$#VMa6dA$@!gXzlHQM*qb8_)w)hKMrVh;cHM;H)`;Q@m5su zzir^yA$@-PfLLF4)=|DQ)}Pb08a{tLME3CdhQln9I!eq;Q?{H0Q#zu^Il$WE`)R_9I zyzWX$zxsR!^BEuf#r07m@cB*qF*qDYNX&1Jb*Q+bO2t(TDsHG&@t!>@-n&=D)kjq9 z_p3N5NySIDtGIKQia)el!Wa*>A8SW#tohWgCW(&mdG|dkZf;ib;lnE4w@=0O^(t;U zs^WVnz;T3xKO;L-e3Sy*q!&ka?nOQ^?ak5lIH~=N_(FY>{J`7A_!o@S*2?nJ#rqEQ z_sPHgS&w`s z%?BDjMB_iFBVI9|=648XnXeDFfiA*V?V|hw!@qc z?7R#4p6xVo2}-v;I3ks9?-UeUzWdngG@gW*|{AC==mJ} z%h9%PljtbF5?Geswo#&^{GB^tU}&qE&o_cD_|d*6B2_zSyr?8P*S~X&u(B@f2c&J zpNQ<)0|P?kq#yMQXeFS_`uFWI$6Ky{PpK@A`3|tGzqCxE!~T8yU_hwEw7(2=S%2+5 zbw0Ciog@$YYk+0_>({rMC`YgTxVLUC;4%Cu9Ydk-|>C$+qCBCiG8r_D;%(bhym6z7H*Z#PdAW+K zs#Lsdmx_1qR&i;mip$DWyl0P!_wH43c)g1E?Nf10jf(HOOT~BJt>S|lB#iNI9iNY^ z6XRb%^KUZ8WAv9|w9g@rV0qs#7*-(!BC(Q>+%aBiGem8T< z`wP`!&^XvvO8sZK)VC{vizo%G9B{n4M-+yDIdfB9Id=arU$e;p--0IvP!0td=nt5BwsJSGsKm8Sge6*4`b zt%AL>{N5!po#Zh=2o;&~HA`iBKwFLa$nyJ_$#jyZ`Ohj-{;uURJ)q^IzOwv5AG(LR zULbjz@2nYUg@r1vsZsIvr7GULSH)YGsCdPG74KZ8;=Arr@tWll#(1@)82MQmUyJy7?4x)= zycI{5hLAs{@pV}#@|n!f$KxWCe>opdt?{>_0{vg-M}8@eY~GA~BH``Z(Z40UW)1qU zjuGFmZ#~unY>7D@FTJA#c&Uk(tweri%S^m{6^-x9ObWg74NeoWv( zL81c(`183c9TTijwn?9d`m6NypwBh+&qDoGI`Ttko+&>Y^;hX7pwBYtb5MVkj{Fgt zZR*cL{Z%^pZ)lE?pRV1Z-%lvk*Hpi9(6fk+_$$)3ZE>JKit!imzh#RTi0`js{0nMJ z{N{KM{(!b^n@QK#^Gnq6KbMdHOU&^&-cfS^*g?w zyMxACzCMRN6Ud*+LZ;6{&c^0u|qRr;78lRlIhsis|`w+5Qb1R9rYu#p~Crc{9e|5~qLyrgjQKHY{_mla>c$%Xn*yazj7a{+%+2;5=Cx^!4IVL~W zk50xC%=tss0_01UW%6gwMSfz1gn9YFAo3l{7xEZ?*Ud7*IHE(BL0Vr1ATXkU4%Qz8B+nrj*}l{00{JP9MLgTXY(~rvn>w8eeD6u;^L1 zAJoIw8`k)pHPfO~ewzV)Sw1^mrbFICYg_{7h4R@no5f`}bgvT>jF4Oegtl$fuj~%cscnfR+J! zW%+_+nNIQ;@Iq5f`4y97dO(|o`pEJ{DKeepF~JCBnDVPrWqLqMLw#lWwYKrw<5>v# zX(W&M4QP{5f5DIOn)G7=7fK^Ka6k*7{wf_4tk7hWJ_Yqx=`_CznEL5{geIB#Q&E4Fj{X`-5%SZu1$_MWn&Y>}gM1T8B|0C!?_4X5 zr-*m&0vg}=c!~N4wM>sWe#2j&d!wKm``K8(XX@j(?l1bj6Aj2R&GFmM$75^!&h$&= zxr_QT_3>KV4~FquA5X>ofY`rWYt9#7Z;rOs%4e{zLd(sR$Mb-ed8f+n@>6~Adr`bS zb7+2v{I-@b)u%fV^LhFEcaL#DJdM97FEP`U$NV6O?+>MXkMFB}(^Z@$j|ZUpXR7ph zw(%@;s!CsI8*gV#Q|b2kL2!ynUu>H%%$cmx?el}VlT^BW{LkUPXM>Ot^O1?>FSBjq z?Ti_!{%qTPARRwYM~9)`Yn!iR&19x{m7@%`0L3;ev|O5S;#jM4hGTxB|K*i@`;4!&PD&$G2*j0 zl9Pk}D$(71eD|5-X~GPQ2aJ#R=nusaPdf6mKE877&7l4;%jBONKz?FD!d!n^8uFbv z{$u>b^FhY<=?>@%BqJZOxu*P-6yz6{Ls%cbaYwJas`yu#xXX;dn7pHkYlheuLhv;apujBbFkZ~aGe{1sD3M>C&MMHSaXRE9;`b*pszIO684))oz5Zno3Hm^Z-$o6^*M2cmmhKI zzt2Pb@HhC=DSsR5(<1!dWl+JH>G>9B|NLQ=PX3JHM~Lb7h=BF`6D^#90bQo^@qzQ> z_d3dZs`#z12xvSU&VL_AuPwb_4*nYV`=!U#_;dEsPxIgq{(D5j&$I*L_on&phob%Q zgK31{7WO7I-YlXpeGY}k&bhT9(1y+qP!NuIlw|+1Dq7er42e9(FDp5 zzQ`iaA`=rmFM;x#PABYgV?%|G1oqWRmL18B>GHUO`esKK#dLWc`y#UvD8J%>ev(gj zV}eZw_AAeV-<3=CZq@d|(P`$r6VhBhNT?F3`oX6Va~WMt^|)#cG1UM!e!81-?vjP#=W z+vBDAC=T8p54JKl=}Ml8_liOv#Vhr9(H~fG_0$;ogWHezg*}SC;?fNK0H;+y9nd8^{r%p%xylVc)qxGYHaDGn`&Hr#9-;h6MdDZ^x^^1JN%gc`V z@_ahK@Q)gQh=(M-|1oZFal@<_>9g9~ne_>%Pg4f`?QAb-zdb!PmK zzldM*w-WzUUy+}XqUZ{Y~$mP|58Z-yih+WYE!{lJ$Jd?S;Oa zMAFCm7>`di-(+Z+ynk>R&>!f*_QDOh|GQF_apk;y640MhM1MwmEl2z7`Nb%o`hO17 zO#j2b>2mwQe^4$CZjVQ-7jX@k`u}{^Z_4{3OOlu9>>{4#x@`S7RnLdI|M+-%L-beV zZ)*QU$XoFqOb&8=R=f-SynLQNZixOi*NgUAI*Ha7ICy-LKAMV%{)hII&-DJt4S>DH)q1|Q`sZx^{G3T|(DSQB zpA%F2mlq2eexQ8nU(U(YJ~)8S0w)H@zdEM;;+Ta7s=6okDTjY4*TX$ z7ULN&&lgz`5aWp%4|CZKNq*rJkv}<|%BQKl5l@6^sv*V8EVchQXFEaX2alh`Il_O! zAHFO#Ke1GOe9_|_`A7f#c`;u@{+TTDkFY-@Ao7{O)24|2#WCeS8h@T+7^E7S#E1za8Z( zCtG7-#7iQpvM}<2kG1F+?^D;;_oXPo!RWS%esuAw@P}jeB{<4>Ghjs?B-+}p>v)9RZ z|0aDeRI*1BV=*V%$HSt2hWkGVyWk&=p|8Bh=pB0fotl$HA4b{K{_yX{o{ggXAs#>F zJw|;j{>!)ESKN>gg(Sm`Em9A_vF7bH?!wv`jp?3mG#|+`nB9#AN)tZFCoUS z+x&ge*OdLc@}8G}d;QAqQQGy_vPW*_kK46BDEr&4|C_(Pe&zS>?D|8j|F^Y2|5o<@ z{co>d`8_SW{kiO8zpeeDTiO5TzrB9t_j>I1r?JOwW z=O0-g<)0T6eRpGjp674uG0=V{jaQVq`FI6;t9sI$oxDC@;rYtW@6^O}h4L-ucPsqm zJ#RDq{s+o;L$>$R{TE=W+Xu%79{9imA9&yc4}9Q(4?OUJ2R`t?2OfC8JV1M^*bf@I zj%|qHr)ifW^anrt=(qNM@5iy9{k7xWpZ(<9v0wi9m(TqB|LDp(IeRqr!~6dH^RcO( zXJb!zzVPPL_ka1}AK?FcX%XT`a{KkS zah32ziiPiI4s2sNl05#m`0C|;C%%=GaEX(4!5m2`pwn*FX~!K{C3gcuqTHo+xeh2_Dk*9fxD)Dp;`_*RYf48j9_5 zIApp$X{y6VNLrpg%SN{lWzFixbc z-(sOBXs7v0G4u#O8xq}u5q+1#k?2IvHm>PC(^1|gdOLLkTx)du>Cd_Ifx=PNR7c(9 zOR=FCi#ni>I2ivW&bV};n>Nx*%1`*I=a+0`>#U67aL_)rV@op143R2+ILaCN>faiW z*=>U5<&<+usA~Lhlpn}2-WW*rv12OP%0n(U7I~> z^?7cgAGL%1Vo#f;+4cKEKGp}P-b{aD2tNv#cnke%|Ka&nw$C5(vp)aL^-l^-Vttct zrQb>ZF#?+BUs->0D4F#o-&}tZs@|7$3;p;a3jU$*km<;^`)4BiC-x8NA2Rgk+x1W6 z|B3yhK9z%KK$9vTR1Q0TW5j~Va z<60n>YFI+&I2QbRB~ng(e@mn+`~H?ldH4M-kuvf7TO!ZRG~cqv?(ZyeC{w?`b)?+= z-j^6BbGa#_8`*|3wvoBUXRkh&F~f_Wy)3$+?{+hnnkS-)O_ZrfkC}<2oG>l|;^YYD z%xkntF-9;YQ(w51myI%wvsA8W3(9jgrl-Hf(yAB>Ih`IF95IZ*6~?F`=(xEpA$xeyK+m|40bSB2p!F;+e$tH_##DS#8r~4E> z!yb#_a~`{TyTI>JANMP_N98emr}Cqn4BMo#tulu!+oUorI!iQd>a)s|=$1XsB*kXl zM)r97wfq*cY>R6yFGSHd^6knd}h5Y-SOL0qFa88>(|iDZL|A~ zV)kaX#f`V<1b%~GZs|7`-?n&m-O9GOZ6>v8T)#{dZ&Ay(_dMPvYSW4Q7P4%U8{?eb4$*xI?O=9Y`OszX z5wHHYk)P?h{AbY(KC6td@^1IWH zR@mZK_Ml6)>pKb3PO=k|ChVwuz0k$d1}Naa6hrun$3h6xeAPCeRdzfsf z!zkOhBN}~yubj;18PKE5Z{QN>8@7_Q>3;4Dai-7wZ;jhiD*e%STzq!Tt+nqDk;PJ9 zPO|qu>e|jUXSzOXbEG*&Y0@CusnD50C%wiY$@>htRZqyTNh!={!O6t$#1{(q8ZK(p zTc_u$bZiuhBU7gbReIE<<5tQz`mxa~j*7T+YZs?3Zu$JVEv`X}TfU{gKfZpmei!4) zGxLkcp}6H^BU>CV#MK`)={IzGn(7}znjSCV{>fD7H0{@qpiZ~@*C6BZ5-x9VAB2s5 zROxhE`&uJLi$ngN#zDQy{!Nb+`S+tT=oyP?zmL|hj#0AHvDk<2fdq!Sl13l@rV772 zQKx)U71d3mrxfU$s&VuoosN(iXM=6^tMq7nk9DFwP=&6naA`$v)34BJb*<9bxLvwi z(P>ZZR`lQm^xO&Pw7YaG{j?W!D|%=Gdc`<;RM~xq<3EK?{=14_MvldD^aiGPIPn9S zxR*YpCf^^8<}zRG{%Ed`f5ctaI93XN`zdW$zrtT%><%Fabp`I|D<&(+uM>zq(?R7$ z`=Yt_;(UE5K7%M}X2(BI*BBld!(lwd+$r=KGrqLD+QhB%yWVw&$ z!TWI4J=urnfb@et8~K+$8@bjUkUk%F^Bm`1?Pd>>{DB~Q_%G%@@wb1G-p>Ah(a(Qb z>-wvOOS1F-WzI8aQOGO0du z8E*8Ka5n2yaFDeqIE&ROcn+&o@N8D4;90Cf!82J%!82GsaEa;P=`5(=OqQl#JOioV z40Z$0aa;0J*);{Hv#Sc8!kz`T+B?9mDD=tfqJq=d1%*GA^(*ugY;EMu> z1ioz0@j?jVuLAZvJnRAus=U*P^C9g~0j4peTfj>#W{J27MtKh>Y`jSV{1?F+EFQXT8|1z6f9 z9bW*y2Wy3X#2v{Kc2GPCe38>VZ0H+8A9y_Q$Ivw$8~(}vp^gpzToza_-_P=2)aiQp z@Rz`bf2v;6>4tx<2^3mtI=L*bzq+`=R0vrAbVgf<+2?_qo0#^u}|6`r5+ZSM$ z1=j5gFxQ%Cc*#hRb)rvR!?d9a?S1tg^l|(JkD^`BKQF)sZoHBQS%m)Qf9^SG8*xyd zr;l{^D4pqMQ##YLV{~r2M*q@7#11~TQ-7yp)-`@IuZINQjxy;$3mIQdar{j$KaIOC zUJ^L&x-q*UJ#_D`ajc_C;8-v0F=i{@)GwRYjak!1+jRk1rcIA={o+md4f?Np&Vzu9 zq2t;PbPZfLW>FULAivI+$JRxB`dj2m{3@?wS*LaUy7`6!lTeH6fi%mW3u1OiFOSSNxe#OGGreRxcwxMqRk*%Xx*in#-8a!A zss4>VQuD$ElbQtYtT%!0S`H;3YIm#@a<2l2WEQ&4<(LCELW2zL2w-65SJ8{gYP{xLHj z>F-^!%b&pos2ErN$HYr&f7{}T|tmAobt zkUa6(nlAszd7{UWxyl*0xYy^Hg7obrdhr;8c!Pvz5Lx4&7)hiv7)E#!^(C3#8- zcKMa?&%<%b|JaLQ#u4QIGG;H|h@-8L?}L2ER{j+s-!I8i`G%YkzcXNn)m%Rj+C zaSmhtV4Mwkvg0?Eqg-nAPkWI@e>Ki>d1R~6FYIy#t`h+w%kfK>4G#5e^e^LV$Q>!! zy|4U8V~Kv0?Im@({y})ZFNCod(GaXMY}rcOUibX?JI3* zY~S2{pticLv2*vHwk;=1n~JJCO17M=tSyjeN4q;4+IK^)dV9(Kn!S5FcbB(q-nRGf zfn6QeP0^`_72Bx1g8l7BHr-RQb46oe z{oeYLcC+02l7fzng-3SPm+llin^(4#AKBW{S-rchqHyoF@*_LzOV)O5DnGKPsb*_O z>6Viln=5vW>o_YF9MH*I9n#%=>k%{Ye7575u;bXy{KocuEe&yfTef3!`Obm^<(rQ0Z*Mz(pd5dQivtx~ zJDb|~^&lp<>?+uIs;OdY+y0tDDJJ$Et*L)dcOUmrwmAUfs=j(aWDG?zU3BYgZgNvA=M0Yup$pKhh)ozM1nDHcwz* z6Za4NezW#B`h{Nht=rrvyP~mu?a2fC*51=NQN53CJ<(XcZ^HH|U7_>sKel^<`W)zK zKt8KoQ9c1*o8T)w(NNek0dJ?^-FeUc8m;XnemDIj>JGmaOytu|jTPt{H;K1OqkquM;xN&YhVTBuyD zsb=$W%K!2Df?3~VsPBr>6E}+;lWzs_QC!qaP?vpr**A$rQy=$p_lgPpT+aPmyZQF` zYHn9w>CIxrXkSro^=I$ES%83ExfOl zZ$5UQ_GYoVRqi8gjW-*+H)7tDe*iJRIe*6~G5HdiSFSf^YvJ~`)=2p1%7Xlc`o_kF!iM^SRRuV&%x_+CxL|eu z;kB#sR}`!a>wJx^hwB@g!;Qzg+v>Zz!tE=rDx5a3HoPK#bzyi_!OE54mEnT?@Se!=X8oVd^=)0vVep$*_^14^{$D|OO}Jo{ zsPMgWk2Xh+HMhm9gP`bD*-_uwb+Yx;skUYl@78fwLE)N)@QQF#cTZ%)gTBJK6vrk;)U-SwOKrPBR-HiZkr4INszskNOHhw0Dc zv{?|&pL<8}-Vc8mbtza@z-weIZXBno@qO_e$E@TmzRPEdeuDqDz!HgW`b45{m*}}Rd9tfbrn?10-7V?`7Lt47H;@zB$DM|t%gz3i@2%U3 zxO2jONwod=>tvEc+E=dnsV@37>gAAjeT;P*QC)mxs`DBDp5dwuXBb!YbG&_Pou5{! zL$ky^L$Jrn_;S^n}Dk zqyMDbgufMzj;CkZO|GxM$@Segx&8vKlRei^J{?B?q?aqlD|Z+hBE(PoTXY!xv-l?b zP2h>AhpxrD{x0;8{(eC;`Y-lU`1LBRMKGocZ1!#^-r{9XcIPAki&!X#O-lcSl<91Qq_P#n!ZKI|D+>z?3*Vdh zzVl@?cw&`-+r9bhoA>6uotd4PJMOO1dzN1WpY~UcnTMWDaJerZ#9wgfr=Q_h1N*a+ zhk~N-(9@zX7H|^&iuiHa;U@kPO=g|+Ghavki!Wu&7M+=|9Sz_eICmT7vl!=KcI4j) z(kE?=i%g*0$%s#ZpC&G`VYC2#hB#dI=$wu`EIBK!4hSMxcF11Humb>lp@ zb560I9{9OP4}B}bC)T4s1zJ~p^n2GNxuLk0f4dJqO!t=)>(~BX1CQ(31dsjQ!3~Q2 z-35>Jd;^dD{h{>e`kzI=V*T3RrsDe9d~Y9q7>$PY(8u(n)KXl36tJndmfuxe%eTcBc{Ydtv z`AQ$0^&=#{z+>>ZKdytbe%&9p6~{+uUe7nhG1Pc|3>ns=>vJ8!in;i5h~&8|h;4pO zaT4D0Ib(V_3*SNic^z>X?>|Lc#_0yx<0N`yJf4cUjK{M7E%H0?|8XQQ;2>Ni*Dzp_kH*?yKdyW-5aD;8Fi7G>d#f0q|o-*#O} z>lRzNv>tZ5wysUmp~F9^EQiaXvW2b9-xp3;R21dXU<~OewyKLki7*tMiefTjT?mCq z%7L@V%-si9YL|5%1*uyUtNYg7aIPvkLuZ)=c`CSv=GnA;QgrPe9wKXd=~9&~+cB!R zMD`rjF)dlX-JZWQ9YT9&kPrK0e#{l6zbpMgmZxrz=DuRVA*x=eXUGO$w1>2I`Xw-0 zE8Q%+gWS#)eVhM#9n8gi4A-;jVV%~r=rc*}DwlQWyt?eq)gY%;ok{#LG~#gg+Ts` zd5&Df<3O^8_ZgV!xdLBo@qtoLPcA4g&qaTc4wvY~Q?eGlbozcOSZZRv#?y3uM9LgM z=3-w6vLqg3BNdyl8{3n-ocViHT%L97w2ktzzLslZU-E18hFKiL#*t9}SU%3bhyDG^ zw`Cs&2BqG}nk!@`SP~M>3lo2$zstTr`PlzOl;?G-PI3L8(e*#wzbT&hQTmF)>;_Lo drM%p~EixS2<|0l0{})Qz2n^@=H>cwAKLK4h_cj0k literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k41f577a6984ff8bee476d34a62caed8c.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k41f577a6984ff8bee476d34a62caed8c.o new file mode 100644 index 0000000000000000000000000000000000000000..4d2a600cdf4109e08a985da352757954a12ca8fe GIT binary patch literal 75688 zcmeFa3w%`9bvL@_p_$Q0NMaB|yhk8F;wc2;VS^Y+fFxcL;vs;GIoHzzY6$W@zB8683{1JK34687y_Tu;rc-sb>}8O`uv4Q9(sNO&aXZE(DVPdGc4KKjC73cEPzZl;1rE9~Rao&Qnf6mJOe|r7l=MTN| z@bk%wR`j=Ad}8>*xGxWvKli=iSAYK9;r>T{F?`{PU!Yt;fB&P8K7Z)qBhNp$=!xM? z&s`jDIr7}_g-5c|3_o;XXgD!Pa2j1tbj-X_{-p=2a$oymB=NqPmlHkFs>E;1)Tm!W zE>_?}pE;b-s(Sd?L`P`ks%zo}qSZQCf$0yR>HaVg{-*mxt%DUj1D|j>h&J)Ui%#EW z;ycG!!8dSi7U$7rJqf>gv!bN`t?FX}1uOF3*Kux!i~ycH#I8&&c2185*gb;(50v!}D(Qw_OA0e;gWVIpP}V|JTq+ z^M5!-hE7M%w)~A_r2lh81FUSweg6BP?{*C|f7UT_{$oW0&5ObFH1h3BSLf6QPOfb} zi|Z$GJpo?6 zD&Wn)bAcZQUIAPJoB@1KG4yr~K%bH3w}DsU{1(p3a302a3C^$Mya4A{ah`|sD>%=_ z`A0a<#QBFfPsRCpoG0P@9L^JPeir9coWG5863*Ym*^BenadzSSRh(nc<4ZXI2Inu} z{A-*)gYz$O{xh84!1+;}e~xo6&OgKX7|uIjk58Z-3vfP$^LNm$-8eUZJ`40P&NFas z$9aWgBzBVh;2z}2TIb?3jn1WKnqJI1bKskKXAXWg?@TeS$ELD@pI|JHO<)5t7aRCZ z5*wiW3cfRSuY7ZD?Tg=BTjxx>SobG_fvKe(QM2Oa)~bV$Bf_jP>! zDt%76Sodvw{w96)U#xo$pP!}AiCoX2anZo>y$K`Q{)Ca1Z3*v^UhK-bImdTH&niQw z9OyJ3I?abp^P$sx(`KX#*@|=_`yt->c*ga40Cn|JUA~KTkK*$~^x1o{?i2WYmOgtf z)_n$_|BOB-==w~Kp6!0iJ<{^J8@eDCxs4Eae*~NDd-9KKhuy+XbD+~a=rj*H&4W(! z6g$0y`IKxYvoCj}zACEEeX(voKJTT^u8VaC@VTBoJ1^E9!RJHt*`fC(#_Yt$9;n(3 z+dqrHmWSN$_MCN(FvsB97IrSu{9@Xf!}*B=hwg9>oG8ZMS@%H8qwbLhQMUOLXiKkq zptiVoZ8Y>XckM>!yWvFkkvhh9Uc>w`5p&0u-t`0P#I;=Txak_z1DWMG`xB#8IawF} zJjuZZW+pOyEWXdf=Y`i?&L89=FRpXIPo0$)%NZL9Ca{Zj&Wek{1P9LBa7LS)TXA+> zRLgImLSxzsh}dy`%Tdb*&`jYB^GdZwT+9!+26n0cAxZJNeL zOqnx^Mm7b(W3Ou#u4l)&z8iJq@atTbDJO#K3q{>4M%R7#0nqaVy)X{_Ea=MxePbN@ z7eHSn=-cAZ{|@vug1$Qr{TS#W(1i_5y}ym?>jmHbID9_@y-d&##G!uz^eu7ZzEV8k zcck6meR(7f-+Q3nC+H{Q&@)QFKVIj5AP#*o==TZw**Nsw#2?W4KO2XBg!l#hZ{pCu zP5cvd{{M(WKUOs2zYlb~-}K_TBYpJsN2z?A>wiY&Cyp-v87d#=`WL8t#^~~2qVjRB zf0fEl8eRVDR6fr2Z&LZp(dEBQ<>Op`mdZ~aUH&;LALsh>RDR0n@;{{VajyS}%1<3# z{uL@8=lZKue%k2ruT%Lr*N3V6^wH(tqVjRBzfI+5j4uCP$$|QqT%AIWFmzaOPnN&3JE6hjVIvE=IX*=c!A4}-} zHRh=|eiIrP!Myb+N&eioF`s?sjcxt!ey(U>XkKFOweN>UhA?+~@83crKY1xM^5S=j z242G4>*be<27Zb;*-wAX@Y#WDY~+<+77e`eOUJ;A-*F854D+|^Q&E46jr?ar(ZKM- z#Lq@v#XRrl6N(4k`!_c78s>n%m|Z;Z?&ll>uVcRWMoIC&J8z`)|0HQs?n@ur(Es++ z40?v3e`sXnBxK$j8hHEQrv4khVFT~9m%}Dy{X>7{>K|HGJaElbJn-u$*~pFX=Ki-% zvXNhPItJdV-qQbOl4IbPf6qqVtk~NBMu`*kI8di^;I&i-^l%Qm>UE3^KTJA028LE* z4EbXVGZ$pHgautdG583q0CE|51n+;UJu_F@-eJUFpei;{HA_=uj0vmx`wsPg)#S* z9J;?WdvFH-dllbM*7CI9B+FyBTRuC~PxVvzEJujXb({V1>v;E`ufq2n@I9=}Fn-<$ zyWoFr_+LVQ%$4#3A7uF#7WF4DFYZrSQ^Mu`(X*kEG|2uBS7_kU$r(2sMMb&3#1Pg^ z({JP?;+l6&?)j3Lmvd0&yldKxGJc+Zqdco9xBsbWm(M>n^>R7NH`DdMx}(1tJpY8h z^CeR*pQ39gC*L^5@#M?rInKOsp5x5R&u~2H#t_Xec2oO zeV{EM{M5wDzU+;h?&JLFT&D|PUfzHHdx(lL^Q zazh851HSA{T(_YHoj$Zg(5i{H+2Su1{PVyMU+4TQsmvC$%o@nQljP2$>s$L@g`ZBr zwaK`)3g?wL`?9z7XW(-I{(RXL{pmQ*k{o`<6$Uj&2C*e$XY@S;~?HtL)-_`F-?SJu4ru4sj zdUF3OpUmuk^Did#pF6q##%H}fxnD^AX#eM3ci;HSlJMofdg|`W?@e_M{QQeE`+xD3 z8U1hk!}R|Dd}><%TmPET|7*nc4aDil4#&XTf5k?AQ{otS=PwYai2I>=$QwyGFQfP^ z8X5W|#c%Pz&}?AjqoLElrN9$_vA!Mp6X0rK7ckbpL-ztV0RNJqk8u7u&R8c8b>jMQ z;C};N2Yi^pmn$z1eTTtMaefNtla)7y!nj_t{qoRP2yeeJR86?*^3dl9SKSz@AiU!; z@(S>d8$+dpcU~SkNqFatp>@EnT{ni1-_Jj_>oV#YIPcnhV`v%hNt~Auz54Rd4qPw6 zc{A}<-x!*W>m@ap!3SJ(V`wVjJ(q_T5Z-fRXaeEAmvJ3$o$h1~2YrF^%xgNPf6MUaL*c_Xhw$gc-*}crVn!W@n<7V= zT02=KDA`1HVkYKH#@Xp)u5m7h)8%09BuvxA;YwlHgJJjT-y;?o1Sk8tuA?Y+>F4@~ zyr1f4=V9gvu~jSq8cm^+i#wY;yCPk6_4s6*yW4ck&*9GIBaIqce5AdrY4MR`i@OfD z968W#;fA9EA6smGXlQTisOw^jyPCAF#d_1i4M)3L>pDBbZOg)KTKHg7+p(~q>})#H z*&f+i*U@>r<;02BCKK<_ap%(fRrTRz;l{4+j;01KLud@)cD91ha^&EymLpxO z!d-23;REfFwz@7;&icBpx{dr&@t&ORMY5M(R46dC+x4+Pm5*;yG$)KhoJ%ccd%t55RpJ>N?t*A`O2K-HdMIZL#+yzbH~C z??Cy&A3z@(%~5@pU;Tp#VWV$(gKpNrVo;HDvBB4=&m+W4j7LIoP z>~I)!P)z0PTElr^poXK3b>YsorFmMotKneNk-UQN!J~DN#z z*=xA53FBXTLsMfIQ|)l1sp}|~1dZYP?l8a3WZ_G1Pxw(N0+DrHkb{bjwjJrTSN7r5 z>A&5ILrJwXv9({g{i)Oc`D-s@VA;{u)V8?&z=6)Dt~?eFA325y(Wn1Awr)NK5>p0A$Gq$hYOZ;!5jPh^7j^;c$7+7vW53My@$2J8 zUH@w*$w?h`{b^ixvT^L(X`PfON@TnA>sO5HVmBlis<`-$hAL_ezwTEr0@dJpAWhPT zzRALo0oAi`4iUgXdn9x$!S&#|RP(xi75g~6ymdV&2wg&sA;6IHAg;Tx?_)n=II_h> z`b-Df(?jRfWTJ7 zbL`dldQoi#QPN;_{Wwe4*eqWEG;yt2oQ-;ozM{({|NXA-1yP^f2k0_(yl`_lV^^?Q zwrzL$`tqV+NqeLNzN8=IF0_xf0)GeSQb(7DcrU>?8>C&Qmex87?qFAYdu!*yAhKr@ z_K1Q@mn~ktFu0peiwjn>;-f9Cjls_D&aS4mVB68ou3%%+ftDjp!JQQqTXuwZZ>ZW) zUa>8_p=RfXZRYSVAq{8p2)5$IGtTl@C5b@u+`rIc3Gj1XXh21#?C3_Q(2!vPhn97 zk7FGQPG+?VPGXe`_OmizOP@p*0=E3k$5tuW%kq@+9u@?)?3cjO6pUP~U>A!z*mZ}~ z@OR`21v^-j$#fnhRKCK^W71KC4#38{V>`}MW6Zh5#av;R!@V?(1+l3geUPz1{9VVN zJ2%L4BIU)K9qybUMFdMjkYC3ijWHp}37*1DHuxKS2o2a2W$xARODG#7g_6PG@KH-S zt`r!S;MgZHEX8q+z#)Ot1g;dgR^VEJGX?GtxLM$+z(Ilg1nv;{oWMB(UlLdocu?Rx zfv*T075J*as|3C-@F{`e`Mf9-72^k(*ioK5qNNdjv2>(7M-YLpTJiTtkhp=9K#c*=r~j0JPfKt4+>m6L&rG+ z_XTyFC-C4b9g}_h4mZ1wNd)neJqQQq>6qG2xHebEx_$w6b-s>u{Q@kCNd@trLwS$G z&nn?}26n)|Is`t?=^l3Vgpdaw5A1OrzDeZ``{X^WW5YgCf%W?R?25p8{eG7Bh+f{X zPoKbseXa`}6z!?}Q@wnSz?TFz?34DWPB-k+A#g~Pzans%z(LF_Nq=3x0P7G~*Dt_a zt0pn`v>=PpV2r-5#NRo@{~-RZ;xB~wyaF4z(~|j@eGU|(KlcpuH4fM08!ors$5QY) z>cZz0;`(8HUDt-cgH#$i;Gpw+;{2{Sza!4e^->-A;J&qXrF{JtM_hKrW+)bL`$im;^6!uiNpF;+_vicO(tD&H7 zGRB@u1wB!x&%x=lDeO$d2kXLqgO8@NXO^%&E#M~SqM#%+-SXMd)GB}bTtI-;X;*ad#~<61&0%^zbPXbgGD8uL*3j!Mj#ewfNi zCtm`hBkE$Wrm`oFgS`dNkMgrp*oJQS3o$rB2mca{ce7p|_>)?)Ckn*%eR(Du=w@Qt z=SF=F(9=BRP@MkVRHDPt=pecabdM)$&~<*X*Z}=?++1gqSNdR&&Mx%r)#&L^alE*M z_4ZIh`2qPeVSltfQ_x=~dnTj)G#2{_?E6eA>^Cks31q`QkAqI~4E*g*!AqwVDeP~K z#(c>XY!0$dx{j&CICkrpf=|YjOk_;SLdLZ=Ovyv0H`{Q~hC6IH$A&c<&a>gD4O8-w z>px|~Asg+3wng{%^$iU>Euinc_bGrmt z_h-AmD*P|N{~z3{&gne@l&|- zct*Uxbkxn)Z_wVyvQwB7?#t6bs@|T`qvwosO}Qu_QL+;Mk4$ zG4QEw6uq46@=^USJ6M@ve-V#%dpjYYw~y?@<59Ke7SN+0@#7V;pWCwE^Qr7qBI?5& z18w%PR~*Ps?pU-9rKx`8SEWDE9>lkgmJde$d_-ezs=p(6J&8TPgq0rk`AmBw-#u;W z4|(7(?dH}6hweXi`K3pbRsN51&FLS`f12#0AI^C4rr#r|WEJ$c9hXkd5eYJm|NbbI zZGF(-*Xf_vynafkF(<|k=>Np=G%J4}NP#`vUK)`Md-V29FxKK+U)U#cE9e~|*a!Xp zxaM}7;|1~~#v^ys;D0uiT|UavxP_u<*+!p|9);;G`cxZzS~^WSEc$dCeMWjzq0hAFupik2;Unxp`UcY(&B?`h z-rJL9m52U)+N^Z+q(q->qi3g6hgtk{6gt(HVb+(x>qoeX`n}v<3>~KPS8zI}@PeKI zdS-gmq@M;(PmfCVk$i-oDUUFw_yQ+#c|j99IG$zTmt7p2`n^0_Ka!Vilz&l|XW7p& zD!$e6o$@7mznBEzKz`}d5`7HAg=sGh3mB7L0e?w1CMeurdV4ZdtntpGNuA)I4CXh|k`h6eB^1`o|gjVLH|pN&1q1 zFI4oycnI?t<01LaM4!)S@3&If-@=|gYC<2ygM<1Lk2M(jQ28uhno*wnUna^~`La)& zno9Z6v|o=JKfOIu_*wu(guciYrhR$+(~`~lQ)&FpV5FcKpBe7xZ!kQW@i0lTAL7By zXwkswOQ3Jihuk9RGuu`^8{?3re{atmJ{7m>o0sYYu8lD&F+7AW}L-LoK{0@@u=bIg7f5QJ>HS9tDIxm%v zXTqLTf1b(j0zKE4W6G}phW{z{hyR5veW?D0CVv9jKS%Ko=(CEEKvN(1(`$x4iE#xlWlA`fWCzHrzMa4b6%=CKbFP^#0$5-G~SYZ zg46^WUxhuh4UG9Ow@;01pH*YnM;$+VHPS{L+&;8BVa-1%|LKzvYVSI<$3A|hrt|O${b;>6YX;h@`q$%Hk`lj< zYl$)P2VXk&!!RFL{0Zwx^bcAi`UmUBXCoBNbYO?`M>DZLndqhUysLa58;p3hx9^jY zG`CDYtED6lO7%k@G#B~u9ew@RtKmrpss2xD8EH;Q{wK9ezJ4?6gFH0c;y}QrfScAfJpgh*|=nuc#KYd!lL|V^C_8xat>c3uXg3@1S zvEH`&gZwSSP5x!|?<8CKOgMl=KdaGlTJFEew(@Q_ZOB^kRDZGiL$9Xxf1j4-qx5F9 zhx$7ZrS?<*=>45>N?_~{5k0d{#qP%x?6LJXHZY9-wypo*U-te~_x~s#s`;T8@kjX( zhmT?Zptnc=4wPd(A7H~x_!IW$N{_lFf0{X_KPmo#{khTyYpE_Adj3NHE9)2R$0+L+ z?2k;$L}|mGy_%Xouz%0}ErIs;;lJ3Q;n*#(-G692mE-@cmXIL(Bl_2^Vh`=M+%(?L zYV@tHNzc9X%f25rwds+SyIIrT2O9Bgja$i>HivKA7f%sK0 z?5Fw*%_n{bf60{nrO)>ft`5zA-Y0(Hb3N`~T7TC8YGfH&CYHj7poo)F|{1 z-d||{(abMf-hY2F_Cxvl9P1-XHybsu4{@dl3TBZNJfo2tFblChwQ|KSOztDcN z=`Sz%nkc=zpZO(Z?AMTgz+cS$aNS>)f586o-){aARr=o>IHlr@KAXQhrqF*sf5H7r zTpD#b;#pZ=WB=uGZGw9y^$#t(IXz%~->2`#V0&BkpIXI#M(@Ako+_h8l85~&dH;j< ze`r4m2e0oe?(dW18}|!LTm_;z9@+K-J{d`I)Bcn}A8WrA`z6=_$M`4qW3m65V%vXB zq5Ww~ULwAZkoR8@I3X0E!)_m7Wk18%Z#}2afn9umCF7ET6ZHMK%K~FRANwzf zX}mJpZ{^?b1kC-_muWvNU9$ga#9P9|H7Kv^Py6M5Wxo^q$F}`Vxj$$>Q{A7W{ZiRp z*k2kaWBYzK`p+-#KlN(kl>Wy47VaOj?2rENTl-NJ@DFu=jrPxEdn@*Zz2P6;TBE&& zy$20@>ie6?&!{-%ieX=||NVkO_vqi>==H@)3$%^@yPLeRtrv0aVpGf#nV}BR)(6N=TB10 z2Qt<2^hBlQKeV4J`v=``Y#uz{z+Kt>K7I`t{-=C@a#qU>s`AwR_&#mAw;9ZaJ?Q%+ z^WgIF1}%0!EwOk$IIB$ws{W6su|h0l(DD7bw*$bEN7+UkTwVeTvir;U6|V1$pd2rB zzmqC&kyk@uOCG}2IJmqdmdEa=CJ6n4K`Y)UA7?4@&>yo@d(2ksi~9q-QHV0@!~KeM z>nIM9zq9A7{Bsof$Y-hgH_Q}YsdP(lVrFs*^SYw^{z%~IQ_?t2B0BaveC&qhzo0Li zfsBvxoDRGw=r=Hr=TtNRhc4gl?|s@5?oYbm~$DedX)S+3B}YIEny{fqh+ zXD0`QKDfWAXN6L}PfPL9<7|3;$QRl?#U8X@EBia{@A39Y`aQ0t@OVoR@kaNHsP@j8 zulh%>BA?TOhW78)?`To=X{R#Jvw+;J}zXZMKC3%Q%jPF*yKzw3+m+_1grM*3K z8SSs(fIm}ypQq@<`|~lYKC~w<$GzNakKNxuUpn8v!t@W^f8xCa;fDX=ekbo`2~&Jn z<4*$SV`~4CzsUI#@~3AcXAAjWil<;^+FXGXp?{98J-JGIdVA(e^d%_2K%tZWSl?$< zQ2tcc2YpS?fGo>qC9U#b2?_O$3UpHlsc`_DN` zKfV8q{&Jwa$CeMWyhBjV@E5y3Q2zu?dIk0W?94R7e*BS(>?v-;UI~z|_8-azXZRPI z{zLwF-IPc6n(n=7VrsA2AFBPSy>fmae^K)R@1OUq_K&uo9DkIbWdBWtJlUS7p&#<= zc}pJo;|)_^;+OrK-_OqTsWJai``0ll-t_)4;>(5lN5x;36>kY-|5=v*dQpCqy=Png zo5=H(57SShf2lty{}|&1*&pdjV6yM6>`(TdCB@fiVAcM;xc^FuPY3T0X5BB)i~OgL zpL#q{{G^%n>-{0q?fs|r5BU%J=aSVw_V}WFm?_D(_XqW#T%XDMi$ua4vu>WnugVVCV8c$?zH6Ctl zZ@OO=_QvD(hChwA_iS&Tq@O*$X#89y(c|@pH9k^2{Wjwh&quSY{uph4*n76sKlFS= zmbXmOhx*6bpXU98{AJ`fy?;VR`EODG$m31LsCc+_Jn8;!<#Wn6n2)-x{-XG(H0q=I zwvzuxj}LLspC z5iif&-2S8EBg?Cq`t$o^r1_d7&UmBtN1X9Pn!oY*kn+{-#K$b}DM=sd4{Lule*TIY z<^QYV!)Nse#Ydk}-|re9S8m=vy&zHy$9#r?zS=KZAj{$0J_d?vqt-HrI^sMqgbw?AK7pt+n;Jio!{`DnaP0r$H* zqg1{Y~K5;xb#$Sj}?^F0}3d_)#D|!lal%e&% za(_I&9|4`E9T#-$2T#^mjNhS7?|T3afX>@92z`LZN8FfQ#>wB5xdzp+U8~2x{Lq6yM+)sWT72y7NFN6IvuwS3{xLzmSKU)yt_n$_&Jy1Ry z_y4C+eY$;^Gvd*occ(=Fug~o^+XIaEQ;>RiUI2>2e$+k>u!rggEeSF>!J_>!@1)3lC^NapN`GT{M54xd0=|ij?l^tB4 zfaY>jX`}rCEx|JgI>+G8(2}hF10Ekq!2PbEUoq_MR_qP_gm(;iRDUn@L3v(3w`T&q z&m(HI7y9tqmAOB{KMHzW{CRlLQ_245&jiz+cz&ZG;^Flh{LV-Mi(3iCAz3T4LnElC6c@}qCdNY-PfIo_g~_KTf2te$ zB$9rJC)A(HyphC2-F|2$Z=aUQ_k+3rRk#D%abR>f`7fs*oovxlLBAVVrg!0SXzbSs z|2-B7@Hd;CYeI-EM=w77r*Jbc(NikSFuk zW4qd-6Mq-@J5ByZJiu(x16mmM$@~Xk9h2@Lex!iVK9j#$|2|E$FQC;yADRDMCVUxcV-DS#${teJq=0Ae&v*=WRE%@t9{&u*xMGt6AXphX_f$*{D#9t5o29v)V z>Cd7Ew1eagy8XYE%I)2V@=a78@fFZo&|X0YCV2-yKWNeqqrED<8T1yD-iG$7@(+Q2 z*rb01?N#|(L2onZ$IxDtegyQ7nDl$lUR8cO=*LWY58A8pcYuBm(c#Ypn%fygyoh*5 ze&e^}OW5O#A{|HG(g{P74#`h%C14Vi~qQ0P(dSXs_01xSg zcs?25lY;yGEjse84-__z;`7DSwDCOwk4jHYR_WW(G)70{r^g<@x9=!haWVL)+ zvRZ!OLQ6jyU#3k{adNVW>-l^`8c$RC`>guNVB&pNW$@qksDAbypU*ip8h)ZNzKQwI z^gLslKAwu_d6FXR1n=J~`us1u>jb&uy@}-i$T!*aeh9>yG2Z$jC-m}E1;(#neV+|RttxJ7 zQ*mRXikq4w%nBoSAHN&*9X0FQzh9*v-G}i=(q~_nO0TV}Rp^mUm0s7@D$%pGRu{}` zj@N~e#->Jz9*l&;62EV<))bo#dXku51|#+LGQBXeFR~AGOaF!jnO>kpXm*K%=MUtI zt|oe2gpn_Tkpl2mFDY78ZVy~j2t{D(ZL_#Pn-%m{G}-=4duJYe~^!Yk>+Np zJp7}jMa72>src|=6(2dG;*Wep#qI4XK6XsS9UUsZ=N=VzcS{)hCnW`;zu(mVM2}1_ zj5N1kd^P(Y{ntw2mn6oE!btlujDHe6f`x%ehktdSz<8z85nqLo{b9^cSS|S*$Ki&0 z;6{PR_q4WBK*rKAUxqykBee%H{#0uGdNY$4KeibJrsrcX)6?be`@ke$@7~{c~BW-AnHJ{qvDA6%K z?>MC5rY031IH2O)yH#9Qr{c!LDn3LGjw2-O8QH4h!{p#5y)d$E7xIazZ??A6N&RQ| z7uu8L2i_vazhI=MM&_R^-gltCUzX-0d-llmU}W!J72kE2itoN##ryWDxS>JC&CM!q zX^}AOzx4o(uiXBKhr&oj9rBqpAE^HbjsKhuf5m*Fqg}xVBL@|Hh>!n9d5nMfwgUVV z-;f{BcJ91xU@mX#9*i$+t2rOob{FzJ+h*d5eKh~AF!A;(Bv@8&;z~-;`g{WMhWSAC z7R(Q@iH7@uXueU!=SK&rJa6BQMkG{rNbsjcl9FnHcMgg9MK=pq8vOeFB5D6`F~6Ys z#8n>fQ5PGc_04Pc`9YbKPicMtY~$9;`qNBd5DjHulnEI50F3aCp z1%J`USFTT0NaBY+JAh^Rp(2@nEV6Sa6bO})e6%m16@f0x-@VfuZ@K)P#WFwUJHWF1 z;u47t{dez%0-+*P{}Rw;`8B)M`ONM$52A1WoSu4|zAs<13P>HGkTF@2wH8h{8 zHtR2hKdE$75L#>UuY*6SbWA8ig@Qg=+h-g9_tYSsL+ecW;eF^K1y%W{>9$CQCzV0 zFZS)@@~M-!zwFtQ2KruozL5rbbrQySQC3FlX_!!qS6jELc-uA=Z`h#X{UH^Hs#NUv zt2il1#Z_xmTw5sP!pNGf$S3Cb7>pDZsr14Ph);>WWs6LotZlH?Th#xU|H$-Uq`X|k zn>ML<^JW#VU$5eg8&zCds^ZE@6>s0J;vG9wTwJW;k`fi~+^OPSyHp%rtK!|eRa{-I z;=Arr@!fZ;c;7k+WBgmg=Ob&x_!rRp8_e+-@luHX*>Cg*$7?H)ug(6!{saHMitj(5 zKMEu3N|FDh`9N_A@`+sD&7AW7LRA>~P^W*3_)GYBZp;q~BOyw-tjHYC*KWdi!q%F& za6QHYR%qgN+mXN7IujT1`B*VwF2A^%<}X!*p)ckGd)64^Ii9~l{k6zXtc2)@f6(`a z4Ej(4VJz5<{e*-u_Y-KmALI!J<2~vNV87F}FY-fa;#EfL4<`0Ev$Zv=tbAO7@qaC_ zoUhldw&?VI+G=2#UUY{<$9UmhL-}K^xj#^Hr%VrMcc8q?zhSvd2fuqAlF4ey$Iw4p zD=m=e0qsu6lljY6$aLZ_0{BAFi0R-!#J|DMG%o%m_~ zv%=)RYl%z`XnAO_%)ifv@G#d4#82~`mBbJK4QNZzeo-FrNOUaNLwQ684ruvkze>k^ zKD5-NFGKrP`gYLsP5BGaeuF-f%g1~;w9MpRg!UVBz5ZRGFEr_k(SDU)4f-Ne{t~ob zrSAcKG0`y|9*ZQo5r3g2glRpi`IF4?9T?-eUmefgd_4Elc+SUrx10Rk8o&K+b^P{o zx-otm^9j7)i_a&pK2PG~tr*V>w4{A9e;K}CP{;E^Ydq)iy>Fj1pGd>}J;}rrKV1p5 z{L{yK^k=qKSQs_>8~MN&Stag|Fvk0YeQBT()J}KTG3l0UwWjKJjq&+KFY3$RgnXp$ukiYpZ9~FitEjv_o+srQ<%hh) zU*I>#bMI}9=a+au8|!!b_>G4T5V~T&2jh1Fu*i2Wo<_VU?$pVW``EWkU%lu2TWjgr%Nk|U4lus%DXv^lv^nkVi@?`#kxiX#jk?=$F zP5zbhWO_i$Mtw5>>Yz*~ekAj z%wIH3rV~HaN9Nx!L#7iyCK#c)CV%NnnI6!BXs^s)ZX3V-eoPQT z^N1h*7|^oNeo_8q8t?pmOyEL6q5}u`^SLS=6Rc2{NuQ1OtMs*?&obptL;F=a@*M?J3pw2w|1sY4{V`+xj_>F0pz)Tk&!JD57Fwu|_kMqr^f>6Qe!8h=>0^7>bDf`lKWe08AvtZ-1_e>3y*>UBBPaD!+t&Y8XdI@aHU0 z@uEd4rssd<^2?T~c;OrsFI}qQ>}(aUT&d#4^Hf|=pyIi6Rea~2D$dJN@#@torsvyb z{nxEiasF%-uU)I+S+i7JQljD&(^Xtlq~hh%RJ>t>idW50acQZFJu@YY@rvhXX}p}C zgM2BCr!yAP_+^d9GZ!PDNH{+q@h#!JJmd=ruULWj)iECrJs!lPM4!g*C-aN(G+Rrs z%^#-ENB(Ei&GC0eHjT$K%<@=2IvGna=MOXIB44tZX8EjH$WJVvFt0xtM80Erf*<4W znrX&(jNhLrjLce&e8g6m{Igdfzc3GBef%yA8s&!)Y5cw;WRBm7@5lI^Ddl$>zkx-* z)5q`07M;fL$-oAk#@FdnEc#5`59;CT4Qu?KIn|<5ew%{wGJjUOOb5S*#_!1@e`ag5 zC(87IHWl(@{y7;ko%oUPLsLxtd6Q&%0Qc{y{JBV1=KKZYlgBd?{8LT-1>uY|`m{@+uwqBQ(yGpNjUYbi`{YMetA7 z=JN5|YmVO@5AsbYmFRr@zH_xSp2FX~b7_3z<0aZ3)G|Hh_zio3?u~+O>}O;Bo~e)D zy1nT8PIMs4G{UaKN!YueLNNS17iPjwK-pazS-JpE1yBX zGA$=l9?t_>=ACMJm!H~$-;3h?nN9OcDR|$A$&c~O=R^LM#$Uf5`BWNzJ(1R*jPKJOkQYcsK4P;>{s}3_FD#p|K7QklUQvF? zLF4z_9CQ4({oVlLNqycdm7VQ|Fgl|6dkv4QbP^CpV+wmfiwO`#|1H7u>+0_d>A&~b zi!#V6!}vXt2V*||3KnD!}BBG-<+6UOPKD5;Om`<6Ukqk=KV}gC-)zsqraZT z^I0I{K-$Oe_u~G=Ji1`y&*UEFq76sq*UdHSAeN8tNXW9Jsh4Fg^i3vbR`uzPU z_Vy?k??ZF;r0e`x%20m>v?UXx7*5hjKlGRKd#>o8$=#S9&~XZXv*`C%@jfrY#EL3~i!oryfuuzdD_jpikm-@JGb&PhwPqJfAZnelJhQSZ_L= zgRnPW??K-TEuG7A;tH=n;?jSghswj=D4$OG+gP6#;P)KQQBR}Z+(SBHTuB*Qnn!J*xVjvxk0~ z2ZylVBN~3T9T2}i&3{i6{f{3^Bm5SBpBB-n?c5(=|K58R##k*ls1WLd{keVlcn&)5 zccnzr1Cl6Lz_=p+ZLj`&I?ZisZy}eq2F& zvm*0jI=_y6k!cB(UvWS_@u#~n!KMTIm8VhOl|$|22d8IvFaf56?`LL3a$|b^xWehv z>F0lOM2YUmilo!e1M3H;yFA#ip@Y|#6`7VR^cDOQ6GQ|OzuOndOu&K#2bTwZ($j^$ zqCGAcQyg6WWDgc7IE?ZWrwMsNzQ;@X2#2UY!zJ1y=q@jfSK#M#-oIBwf59IL>(l86 zHF3W<{y}|nH1`tnkA))svmz7u`^|8P)44xg5&aK;g+2NGBL+P~o06V(j4`e=9 zEQ=0*^CV)uLkG9FFXHuOOZb6u+&aUynf!(Iy8R6MXGMaspkAKaJ1deE)9c4Yw4cY{ zRS{n=((}}FTx4%vC%5;kn68gr9`WO!tIHGm%ubrkvEdI3o$RX7pJYF8kmeu!;PO0T z{)a0ZXU~%Tllo^4J1W|f#QTTFGoydVKlJ)^e#ECwj}PkqC3?QNYWXAeFZn}XYLHP{W7n$ML<5Q=jeO@(x;q_%je7Qcoys(ezfAEJSJ^mQix3GR%jO1DU?ah|$U09Wf36Vwj zMSMwmd_#ZCH^|=8*r!bYA$w7I$=-_pQ+q{z;`%|KaUve!4;8S_aDR1^|I6+1 zMKV(Kd_w+mniUp%aU-UsAI~S(g+C%6Aik6J_~HJAd^d5Np07A2fAvcKig?rG6C$~O zqvJu(PX-$z*EKFXg7Z?6C z%Vmq-R6QT+_T%H}brG-i`gooQe#_s3$w4m9@^>Mh*U$6EbrEl~yy&k*<7j<>gZn4> z6HP@#{GtD{#) zndR;Ng?yp^{x;9g6#vu5i+C6LY(_xDx4<(eh-GF<(PzZe`0`>Q!w=L?@#P#({euJeG;m^o?5ktSPreCcKLck) zX2!&LZrDqSXT)zRtw(X_{Q-YUW~)Sh;8h0b4{BRSJ)a-a`#RV-j^8rpdswNM{zCqo z#lj}PFOseQ9yyo41p3VxFUB)opD!{uAjT8ZA7-%!B>s65ME>M-s-LFzhCdOesfOe) z)71EJPIrRN5AHvSGlczwJ$y6O{KQiA@kRG{;C5T%JYk?h|?+0=d$pA2VFmC zz}xThiTM!6g|o0!#lig#<5OY2l#ek!6c#&UWIw|{AIOEDu#0B@(D*|3-%EV;NbeAzw@8~t#|nE zqi!`jGs@Rny?9@K2y}j+z4yz%cUr#<-%H~C_0(c=BCM~^L_GX`AXs1X_Zche<157P zih0Pu{P$8*us%jx!LP69mta4D@%;cM_B$s>oHSk0U&2D`ee^%KAFc17#rtvaeiXtT zRr-2rX_qU8mNEVh*0Ynv`&{^@?Q?WL;oT$-hu%J}PbMJglMQ_!pQ!r&fG(dfm(PDc zHbfAI&X4^CsvZZT7WR1n_TY!(Q9i$!+%3p-5#ujjexduvGJSvIYj_`u&Vlz9qQAe@ zX+{rYttsrR{yrNPTLyb#f5M61hot_3y?wME|5hh|T>)KT?DHw?rS6zcrxxLU4|V;E z=P!F0H)ux?`yH6CIeVOp$2ZA)p@Kb<7>ha4KOPqKGu;0{*aiD=40)y9Ms(=ycWO@7 z_Ymr){)c_ncdr-q4|4x0?KawDl^?uCdBtB~PxXCBr}2I{>Gu$g{ijOIZ|L_tUZ&pz zkk|h)=o@ACWB(-UI>#^&f#vZ$w{yd(>`u!F; zo@OGR9{KI%8S%;Uk1UV!&kKsYyRkpd^EbmTDfTmIyrR_2$1CVt*`4g{;O%*e=PP@8 zrzWN=ly7-?rj#%3euwe*KTy6Kw7sA1e*h+1et3Lnfe$V4p#?s)z=sz2&;lP?;6n@i z&#?gQrDFeQ@KtO(3_VG^5}`kO<7403^`oD~eq7;t=g0s07qK_K|HkRB9R6A1>vvp_ zJr!Hk9$S#y7b_pvHF9s(m7m7`IP}TbB&T@qN4i|sF2p|i>|eair}MGc zPhwk#>Eo}z_|^AfUoQI(aXI$O*uAlz-&@=F8~wt%SnOY8u_cLD-qZ2_qSL*J(bshN zE}z}I${Ka}#?LsJ1zo>-^8fpE^y|OkT<`sR4AS0>#r~cC`3HP%zy6!pS3dpY*T440 zZ@!NIfBhbeg z_XCGx>I!JF1J|%~=NgP{cQ~i+_q@%DNHHf~c4+PI5+jzXx1!Tl(yi#U)Vmd(wvui| zPaA_?I|iM$Qf?)`c?>#jmE4NIV+?xE7<6q6I&F2_s{ZI0blS?e75}L*=%F#_ePhte z#-Kko2L0R^^z&oTFLC;3^WSEn&(qHGH)4ngKN}Qb!HB-y;Yf5MvW;tcWI9S)MYK~G z;97&*Pk+vB_vhbWjkOde--r#ySkwV|RD$te-i%8ZvT5VIsPvehBEM)oTVrJmhlBQ| z9p%ZWGen~J;V5N@)xS0%v+D%SODX3RQPcS0DBYW3ydjX>o(beijnnO6d7-_LRAAvp%++EbI(9S+CRS?lr&VV@{dw3gJf`T~4n|SL8=Y{-7b> zNB$}EC!lIFs?0CbJs}V4_1r=}S_l2bcd#tYF5egOv0fPUX7Uq5_z}RwTgX@Ahv!#W zKYz&2di^(-KQ1(m^^Uuhd?(q*aA=-?W%Jmp_*M$F`4>zbL;KO?}m`_>i!{@6pFfAMi0i?t`(~AZ&QMF~G+N zS^ijk9M?uKBF7i~CH1KTDVh4f$mf;R2Wku|?PE9XBO^)?_OgDa4@mT>H?O~Dbp48c zH_rh^6xixFbBslI)H946)=^aN^mD&Z^%~XgqU2qC5lqR{=Wb2I^NO2$G?r-ued z3?ndwF=_}pZe|O4dpX@5a>u2!q+T!aJ3@}Q^iY!12YS>w=#=O@FG4q~o@kOY(a^(f z>S5|E+GCX;OzP$3Nw4J5pmG z&JsDZucD%-%Ph&pV=-; zcKkk-=$75$+BI}@-Rw3apS_uGas4eihTUM7TiT7q*Dao2x6&WH+f#-*>x7eHypV$Ff^AzTL>T;@R$Iu@JXzxliN9 z@$VF;qvz`S{Ki_du!O-_adNT8IV;6;gH>S_<8p_XQ(p*)f71Y^eKyx>+Gn$`(UL=M zl2e4UE~^;a4rZ5?2U$irDy#pk=V!Vu|5lm8d4)?9f z{BMzbo2~xs_5QE5%72}7^g_8`*@G_HqVFU~JIPK=ny{ntY`%-7^^?PYBL?>ukAV=T z`KoO`tL%832YvLMxC*-m2T-?h2Q*@Vubj;17Z6eAH*g8`4O`LbbU(L+IMZkTx617) zmj38FEPrNMk9MlkD+Fq3uj_rt7mdN1EdXO&Vl96*@EMB-c13exE_N+6n$u zDfxLUIG)No@r44uhKpM5*6BGa9UH~szylh@A5`g4lfF%-+vH=TR~$#<(yd*bPsOcY z|6xxO@l@RUE%~p;mv6Rjyd$3e%={v9l}@*}-?Sgz&>qizCj9}Oo~GKzXgxhn!uc~* zI!*iamPo2X%TTPIet^t6wFe^*z?H z_COW7vcjbmy-mJCr`5Ge$3SV@rMnfK_S9}g4~{|48G}x{OSh6wdqKCNhsL0njiN`D z-G?~#Q|M&BN*~?(kgazxy|alQ$izMLAvO8FXf%iUV)sRJeEcKsvc|Dol((PqhV(1^ zb;a%woKP3wKKF`Ql=#;OM4#!P`l7wj9D8-XUR0k!lr%H&&(bx9N5pW@Q^s_3=qF~> zj|~m;pmOhbeOHC_sAzArbqR^C6&c=~i|a(OGg_|!d#Kk3Qx8J>r{ zKHkJj8J_2oa2~^t$z0De=}XvE1uteVD0mUOqTq$>83iw3g9@I{E-N^fT~crkJFnn* z?3{wr?0{*!Gg^ABSHYlW{VuRd5okRIs0w0bBYcVpmb_43n8r*Q^C+b96>c7rjv{mbQiTIw;=86|y1E57 zKZRWm_fot)>S;W~`(wr`@fXFPJ2z-%N6IMi4tGwFe~I6Ruet^?(a1x9pM#v>zcY4; z{;r|{o1)CUy2BNPvN2K!IhZ+o)H58@m`Qk0V9L>iuL@ivFg%BsPZJoP#&NB{A%SU( zBmP=}n+1*v92EGRz#Re)3Y;VGRe?2uX<9-0(-=qcg91kd4hc+S9Is#CQvyc?4heit z;68x|1uhf#s=$v4Oo^20KPPZd;PV291ioZoya0mAR|5MT9yW*v<_XsboQ4P72{VrU zcwLB&eF8_bb)3ern_b8Jn#yMiT#FZA5Dp4_WwDNP1P)M)(V&4cf?QFLH;A~c~1ARtKSjw zP{sp$T)nDe!#*sxCs6A0NpR|GB-{FOh`>AHRac12)azW{TsnuM2%1X%|Sy3;XZz~A*li1EX) zIsPgUpM$W08?WF&7NP(7pL+%zPaG8U^pS24r88~|0f$i*ulqkig!BJ zT;(V8dab~%sFM!#kn!ad$KUnx)2Qp>rGTTZ8?zgdL-+0)#~Nw`j80AibEEaF#Pvne^`I!IjJ|4H(PhL%4%^DG8tA@jos2 zb_str80`ELKtR`wI)Z8XrDnlEW$2MjLmk6%H87}MCUKq;T~THu+8`sp6DlfdToGOW z1PPN4qyLOcKS4Z{)S|I*7*blvUzm)mW6)rzuErLw` z7X*K{c_lu7mTr)akgeQ1g8wlyAL;K|vDbeJ7a(Gk{vQ`Fr3nhThL0HhMn9egfv$x_ zIpcI$@Ee%;DZd$fM*klJ^R4(bA%OU)oSmQY2$=xkr5w>|0-n*QOVREOrjLtrME zCFT|J;gUF0rQ+cES;BOg@)hxueB}`QwaP`ClY#bm$RquWZRMEY*QEHP`fb&-4mc*> z$3p#=CX0yvy^dbb($AA|qV_9=fY-rrR6h=dbmV{O7b^u(M`l?AGJ5ys$vE{7{zxxk z_&?Q8hh6_B!5^~K|Bm1{{FnGCCD{3w!#)qissH0IqA-pi|Ccd){e~ZH0)H>~L$>-a z3;sTdpXxXG4F8>q3-S2NUeXy2|E2mVCE584Z~>zc7Q3qb2L*pn7i0cw>khU9{C^y$ z{Z|FQ(SJ@zp#HblPjN#wz4UYZ6X!7I560QxCp~^wKFXzLeA??2@yf(mu8(vz;=;~m z;9B7zG9SNm$tac*pc zYO0ntv{mmqT(G^puBc>d$?oFjhPI7eduys%8#;FEY%M=t+*nZ6UQ~X(qGqW?JKWV# z-?jsMRa=VoRPWl=v7@wgz7qEH@tMAycbwmEPrH!RpbHlQ&NB1n-ows*KL0(f$G3sbS`MtaL zRu`P8U$&X*Te_$1;D$p*+mhYUR-{BeN)-CwlT^SH|?!zFWFOvHdd`}Eop9FUwyE$ zuDG~;!?N8ibwxY27B6c)(3rovdv96A82aqZtE*nx+Hhp7avet-tCt>VEL|O;cE{DP z{CEx7slIgMiH7cimOVAwT5IyFPSlskcH6zF{>b(dTd5z*gs#;ayK9y;#+O-o@c5x3 z&Xd1wlf_d~3ZDGEoqMWp%C}tDvusbzrUUSwqjm6^csAR5MCfs3TV6xk?&kWqwk_GZ zv2@$gy`>wD?rCd1y0;X6@Qb}=n>rfXc6Y-k%eODxeWI~!Q|q4Ue90$vAFit|I8YZ? zrqy31)m0t!t*i6)RB!6sJ672?Ubeliy6W)W>J?-3uU6V@v-{+?lH=Ch_c%b?WxvUZ(?`T zPNMCw>(a4ox}l*Av2hcBt1#lTrWmo-s@@!PV#S>(UH#~WH#M;ei* zM`1~ zTd(^jzG%whcJ5j>hMh~foohDUUb*Vql~;T-UorYu)LV61<@T1{p1i%K+U?0JZN9y7 zWw)mv<{Y;ruOa{T^lQi;Q@?T^Yo!~H?5(+(uWpiKq_yE@WA}Q@oAUO;=QrkUJt5{F z$V<(8^*P0%VjAyu=QY-BI$D#z@o0S;#+4e38TlA1)^9qor)BwCW40D60))VQi4te0zOIZ)To6mB@$)mqot8E#t^ zZqvdCo7#?rcQzgAY>ymlYiet6S+RVnj1_Q8qEgZZR)?3bUY#Fav2^+J@bd7|yztJ* z(I)+$jdiV^O<|NbukcTKVg0|Q;Z@UK<2OZ_3uXj?UvPCr-3B znRtheJD29Kst+#4G8UJP6IJ-W zc!pzI@=U(VXRM>BP&O~A|FaM$`J3v;XaPgV&gAK#k2yc?{=Yn*@O?7*Gbx|eB~$rE zUO@lnkul2Cy7c2x`ApgfqoX5vdaWtn|5)l=>xM;%|##Q|sZ{J!kPb<~YN4))Py*yu`Dh&E{X%k0K_3N}ESC5AQ z1Co7=cC8ZEjW{y=;$P4%uYaveaB6%zkutc z&sEe-lTjZ%EXgH*Wz7&4{}KU|6kqJyG9X2@hMV7 zNpWH<{2*Eg1|&-miz_TnNKC+s1dOQyo4wtMw|LnNyK^A{i&$8Q&6WNMDbv{qNo6aD zh2>!ttSnNbu<*T^_b%_o1UFda!R_As=FR)snb|jUdv`sm}#wpOWSlCwuRUze}!FX22+e~@1_+Mk^qae|IXP6Hk+Kw|to;8%Eu8}N73WqKeV z@g9}okvvQLa729PvMYafCxycX@p+o(NbSJCt00g28a5fHcGo<7n(!Ndb8HY=A^axb z-1oOn7s&VbkZ@nl5#heSzl9uW|1BC>8UmLe1?&l2@*fIZ@;9h|`|`I0F8RL+T*@D#&V+I#e~oaA zr<}bf3BT{*i`41;_^%M|$A6V@)ax<_7ovF@Bsh+EZyIYlgrj^4iEXe!xIZ3ygrj^p z9uEYL`_@cH0;j3QR*g{siOjm| z&e&9!>fZ||T2!RjT3&&BQx|2LXB39CRX|MUv~i(OL6#dG%sqQCRhlS%<+z?ete$K0 z*q9=1b&W>U@k7HQG>K;PW*X}u+(lNm!?hw>*A+}$1APW;3^U|!HkTJ>Txb{aq}u`Y z6<3t@s&w)w2~8d*)-dM~nCI+tMdyq35Y|p741=}OY!>HP@2L-EjVwl+jEg8u&PsmJ-SZpu3i&7#04L8VapQ4rGeFG)hIc>i@ffx>gk2t zRB)ra4Bh#HZYsE$|No93;l4vD*N~t)x8215mhv2~q}u@(749=2CUXU>Y~TT?Et3l< z^L^1)!r>mgFePiiOVaUmr>Rc)8m8%Zi{u=DbFrTQA}8I(1|s%Iudk2Ge8%UfuszC_ zgwIqW@%q}8PV{qm2i~Afehh1uoc(Ms^ZVs5r2nk;P2Ptg0c`tGJYER zH!pm3=v4n#sXeY!CHeNhf%?;~-!vwEK)dpee*k?k4y^zH literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k42514d5f2dc215403c9c11ec79f6625e.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k42514d5f2dc215403c9c11ec79f6625e.o new file mode 100644 index 0000000000000000000000000000000000000000..139aacf3e5fe84aa21d1958ee0b39b87d4d54fc3 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^K-kR0Gp!dwG9V!kug#2TNevR{(WfPu zmLVrXWF2ELm@&Jt&%_xr@nmTkGR{tB9L?rz^Oc&>rr|V1?hy_Vj;P!ruOs&JN8;?Ws9qo7^{K zn@D{f?n!5b|3~-q52Kx~+1{?h*dr&O8LT|Fe{lPyX9icCdS>vOy5}lrpT+CYoD<&; z_Z>eLe$4ru@Ui{J!z1%gg!|^74DVZYDtvL@sqo0YOW~*IzZZU}^<4Pa$@ju3Gan9( zY~Mfl^fS*4rp{jyn$z-~@cDVi!h0|MH2k#lt#I@4i{ZW%mq1q-3LQ8wxb67z}+3is`QD?IYh#qjy<7eTW)RN*`@cz*tqgL_w@pMBp6&uPW@C*KNJoVys_ zc8SLO<4|Zm##?m&{TvU^ZT)t51o+BxC&P12oCELF&mV!fKLi@+EhX{W;m~_;g;Tbi2+uip z0eoEukDK{uXoT>~!0Tft!asTP9NPT}+U*;x0BmpKiSWpSfN9xRev>%0EV`GI5*htfRYU<%rlQhrOGqy>6gxjpM!l?a$6ZW43{}Mj1S!0E7 zzy~yqaFhCmbjM@V?g(RruK||B+lan?2tWCtswDK``eOK<=jyecuOBt~Z#;kez^2Vr zK1u$kd~X20_bPnzp6}h+zSnaD-#apf?;WxD9{Hs5zX6N?DSy3@4_@2(!%(DT^4r4? z=!ee#bMaNTouys)Rq@r4m&^Kpy@vItVt$;N#jXzjP`~;H{CX&xT|HAW>CjMvcIeFC z>DPw$v3{CIOdEJ~Iq;F^SbyYSSpUUW^{eleu>Q06v8(^Oi1nXKV^_~7v8ySdEy7$I z-jmQj{Lid^_yO(Eh5w;nz3?tOH2k*-q2X__Ll+vbrcPk}BR`r9SYl}S0e0x$9suos zOA4J?zxz>|YvB3NwVkh-=8{<7g*+F)Kglu-UTMttNG8a55i&X<6Uml5iCz5(%~i-} zl#3;^YaYmR{_o2AM<8E?lxqY$kAUY9@Juq2d>6pyIP~v@OwPa8LhnJ2kyjH!Baq`g z$YGS9WEgohF*E{spo{+V6WF1P(8{0t;}`nRM1Z{oTduS(qa;JP2z<9Piq+@Her z-)Vf<>)q!PLN?e{`JZh(yZSEl@0zKHT(Hjt_$}G#*RTQE{5))aW)bTjg8j%2%BSC7 z!}`M|te@!vkCIK5{V#tDwt_Fe2S2_9oBccN7xM>p=<-3>)nLb_=|Ugmryo}pg{~zP zUm^S5(}!6f&%=4($NWWaw_2}$ul64`=3rm$P}T{y94`kt-;S#XuOz(E znTv)URkr&AZN8QcRt-!pVRg(rJr!47W47@&O}8;-ABQDR7MqjVarP(KX^+(#x@7(F1<{*^_Ex{QlP&1<1v?iA(drtyI$OLQ9d+K8maX2-CSRbg z!Q0VUC$XKrwvP6IzcbLhs99^@2q-34NFGsSHa1ttilSpzK%0k6M_Pda~~y|cZu+mGgiYj5l5^tN^8-39L3;Ptoq z0u6VCnPD0(i`AAKD3?xdKqKHTu%Qyi?6Vwt*Gyq$TQ1P`3@n3+aV=KuIz%2JGm#r# z?xi93(vUkXEsW#(_tKEhS@v#Pw#YhGz|E&2cm6&`Ez)bS!CKJ1WlM*zGmq8PwLM-J z#jGq_Stl@cdDyBodpmgaj^e2$@|U|RC%aK(nggT_yJc%x%HfGtF#9tOA3qAd0h@Ij z-?5ufPjNU-3N7U&z%yL2etL2QK|NTRfm@HH2N^Fr0e2rsi^3(CeO=P4p%wji0k`4& z!dfCIxe`eCRKWKxx~8S>?&+D$9Fg5UvmN}FF@n#tb|UMFXZzOy5N++J=@5jfcpb)z z>Z{gOlvWhEOR)dPIf^KBNJ6Ab@Y+Hke`gTBSkl$#b<^oWbBnjLxxLNZ+1}pLG0%;e z?eli{-1&gaSg`nEK;`P^%(s#dJ2TUS=I zrlM+9UD<}UWvfcdO4_agUvZ=j*c-O-BkfQP+q3{AY!`3gS&u<*K_Jqfsdp@Wep zkK5Q3#-$$b}cpy%OE~hiQtlP=bmy4ox8;x$L z>&$Jc(^-`fPkmC|hHhWuGp*%$Bv5*;5>hyt~JS`y~>-9YTU~WT2rmmenov#V1LQlO5ZU7~uzSDWIr#vITTVrwZs(11yyW z_g7!Gv0r4f-hK86y9@v^-RNYy_a@r;xEr%m*)cc!QMZtRJq0}a>kVQeNf3=+)~&P6 z#Eafz6a3tV3d&5p&N|tlfZdK3EHZE)jSVeixk1}~1Akt?BKmy)dZT2+tVdAzyG}BpbDa$ClNjH zxk39xI_nhu6Fr}1f*$=tA2t%lU;P2+KNs>D=)q5#v={h6e?-sqm(i&aKd9>@pT9b6 zBYu;DlVC+|-vosh^kGJLSHL-m_)R?%Nn(R(tne?;Kk8iGH+g@~G}Qd{=nwrrwU%lKQ}me zGWpwhu76Qyr(f2Ue#*1HF3k&jqJP-K$(@|+0e%VtY~mz7zNq)^Oo*@#03aWL|78Sh z5%S2$B7raK*4SlYKokD4JfNphm1!Syep~_J>$YD*ALw5X*qD>>qy(~Wa3&u@%Fhvm}9tBn@sWzX0!4a4g*UC8Q_* zu#2)@e@UAeoHvKYn@9Y?KD)ae^Z0zbe|jqWYbTo-oI8u~b2)w?d$OC&CDkV0k40y= z2=sQ)V{Ie)qfU;uqb~3zz`KEWoA7Sb1-=aUX~5?R{IF<04RwM4D)5=Wdjx)V8t*?7 zb%9UTc6Ymg$0ADOHwnB8b%9UNg*=-jK3np?S>iR|iGRPq*Dr*BJ@_EW$NiuDD`>NE zUsC=SoHK{SiQ#9>iq>D}^M%`k&p)&$ec91J=jR=pUY_+lqjfbB*~9Ih3HwG1=!;aH z><$0Ge9q1^r(Lg zIkm#Tw1lX?Vm+7{oHmW<6J!21EtBfp-}a!r3+_CD}Uh;Q-8fXGh+I#Z?ou|=C?uLe7Ah!T&R&`mWE$df~F@+vSSs zyNdMXip_66KhR$48~tYlp)Cjy8R!-3FU=2bK&h`Rv%J(_bUhd zNj^O$Uyc*;QhZi=ALlnsFQoY6x4~qze$)KXdA!sKk1sd;h2rlO&>?=ltm-uC0P0mjLGoScT1@&V)V815-pnL%SL-F?&a!6|b z0m*+P7T=*y$YY7`x*@)2$BF02fypFNzj`1u7O#gEYRdz5irj|yuIpr99^aAwz@PEa zG5K>(60NW0$w~lWPqY7#{%ywiY(KUG`G%6Op7I@s^HK z_D}dX&lepsza)8hQTa!}=HTn|e^>cavh5u#XKFmGXHIMtL_W>evt%1Z8}1*Fm+*YO z%kVRjx3Cv@K7jC_^j)@Oo*z4Ser(STI+ADqQzY9-{)G0BpO44q3$MG#`9!|*Mit}} z>q&~B*U%p7)>J-|m=R1y9SjYSe^P%b7%qypX9UNgZp72{HR2OcH{uhKZW-}OsK@Y@ z^^C7?c9)DNWBPX$!+OpA!@rsKC-9HGG&!j#|G@k_gDTk<_D8;-N$x1(A=!UQ%>M9y z^ZZ=M>A5{=PC%c9!D%`zw~X|6`mbpHnwH2eM|gfSG#&nxoEi16p@rBF**o8}v*=l*DOz&gSoF+R1A`s0Ub?Ab6!vdEc88u3@y z3wqH$iSixf)3$*p!7E?i(p<>T+|m9Ze|liLJ_Ym<7v}SDRcd-5Tc1kx$(TSGALHd@ zq0aZ4=nwYIBzN)Ce1N=#!OYBPJhSpc{P$wK>+?hF7XmsF2>&+w`$3vd#`%Et$K>z8 zo8|-XCVvOsFdv|w+@RV2;qTMQzC7OnzZpTpe1N=!kNW>bjq)k@7YfD4m+&P-vfbK?Xe!k_FF@mdA-Jb zC46+f9y4Fz^V^cI&}N*9@JGI%Q1*vCasHS&8Q6#m^ZnGPn6HpO#LZV`jFGRnXWTen z$>RBnJAS?bd9vIT-=q0T(2#G8&Q}mm-C5CmWf=PV z>WC>`8P3GZ|42>LVUh3)_!{z7e}p6JU-Kr8hu`rL-FK3KHn!#gp@;R zE>nEM`f1u<#3vyAl#9o+nLAb5K517_?KkPT=`2 z)<1DxbYwk?7tatc(*J|^-=_G5d`b`SPamS6KM+0|&%l2=;u-CSBAFt;yY+Zx+HaWS z7sfa3H=u7*JmYxt{=>B20NxbehI#(|3FFyilfPKw89q;n*&qC-<2-5LfYcA{hyBcC z`aFiO&u8&@;4w=_a^q+ zQ;hMgFgQVH_y9}n4{3g6PM8P_iT!sr_TO0(=<_+l{(Bnj7Y*^wNgI;b{@dvsWB-i< z6)LwA@2YNN|DBH25)6v{D%!hHkM2j2uT3zXHvykvJdYy(&!l`)>_7Q@$c*kk`Tm>y z2_+iuCQoK=)G1#*w-EYt=>~nm{^`0!pQioD*!k-paK2wmFz2txZ_D@IT6{x(WcYqS z8e2(+O}rTLSy~e0emcLCMJOKfH>^MSyqKT&it`M^dH4|!-NbqLQJjZ^vM2F<9IG^? zS@zdyI`;eQYarnMjQ!^g`NQYo@CR|eo<*OZ^3Nl6`oKJq>fm>!Wxvh$Yf<@}jO63z zizK92U+v@|y!~)G^-mwS$o{n0`G4Nc$!k78NMm0FKVmxaY2cm)?rGqj20nES(BU~k?Et>zzKDt>F@vX=au9B zrtzcme;YYi$Y%FF-Rr7#_sz{Y+SkWezrFUkk@x?lF7h>wN?D!o*L^v?eSInH^p9Wt z{qG|0*jZWa>KRA%)wKz$Yq1+aNdM_wzrJ!(Uu~RYxNBDv7+ZV6E#)?fKBs=#5;=g;7%Kti1w5L zXxcp9pBnGA1Uu@*sl|!(DoXX(wen4q1uvjR3k9b!uCQW2ibjo5O+7zEJn=EEyaq<7 zFM5e~oVRgS-q4K2@Ox6McuIHlN1O#$*k*j5gzu@QAVKWa62Wq`#KT7w>x$o=a(Q0d z@$pFhMF9ix`scYmQF!9xMXm)r6cev~l>f~!-76c!qmRntXKb~mCF{_16viQBPb#8k zgHDq?7JuMW+?HgMJ)>+1kr}s}LGOB!(0f#F8YvDwDz_^>J}S3c$Q`ebs9a!#WL1)3 zNECx)_D@AJ`)H3MOPDy5G*QkV-HkXKJqOL4+jJ}EadoZ>&Z48D6Qk{gu8YdsQsT{6 zp{HWB(U^6#YCsKo9EB%tNC&HRedU9gdv5MQy|g*&)jT#}RWb4-U`6$Mik{<0{2gB@ zv{BwgW2+U#x?;q5x}FAAzwuX~So=M?Rc4oA+<3N)dM4=N;guDrpV3kqMjoY2G%#vA zF&<}nYK`*>-~8H=v@5-g;bAAd&~PY-7x7V8Gfwp0=$tpZB^^XL_6%WaU2|y~+cmj{QhW&1;vr{uJ=aEMo1lS=KK8 zZDGUOWu||}(*6--&Bp#bmU^|Nj%@Y%YnO4g7rr-Ee7jLJAjgqj@_lC4r=!XPDzthU z^=w0(@Koo@)W)SE!=oL5O`8EWR%?2W3$fX=n{=!pFk_|1TWnWAIh^d(t$IP7dYyqf zbw#?SQFBy1w$HHXG)htGT6>Y}@syhzjdGE(bQ3REAx>{47jT1JgN}8CGC*R5G(tKa zmBq%%^-SJt$@8cmqxHoIhLM?dWH(3*_RTh!i~A#lW=bz`iGgT@mBX+o&UEV4L*(0t=z`z^Y%Hk_Wp;i zn`Ua_=+i=gGuQqqI;)4E; z-d}S5&1ZR^x3j&^yM9gc$+%+^u|5{{_Ke!GdEh~@n7xM?qn?_zZ|nN1QtY)EKl|3M z!29Sh9J_5*#vFlvR^kz#VbPJS0S%@J&2habh zhSYBSQ(ayz>`@7x@cf@wmNB-u9RFL6*Y%$NLkc2W&^6w(eF*+bdPA|{Bq;jTI@FbR z9|fEr%yX*izs=!&@S`CU(NYc{-Gg{e7k6tDy?9Pnh6zTMr!b{#Og!nuh!oH1iuI%X zr5>!%VU;Rq&Wm_XSEd3-uhHjpNr?jS9J*<6FyJ|V%o2X(aR$B54`I)#_ zQYG#8W&2}>wsdRNJNXZ__gL6ylqPorM}<6(c(=B%ASAw?DK4@1ugLbf5~Or&ZBP5T zC*!m~YH6=}wzi)k6yo{U1l;8?{$-)Sl?9?P;Eq3M_8Q7&#_NPmZ`!d!p$P)#y9^Wq|it z5{1Z{N92_Tm4*MKtD#Qd@51YdC+go+WP6VZXmK*&Xni66i!F3`gg-AoGjtr!ettM6 z{=bgwCp})YJM+ily7+!^&BBG`HxZ%!rvTFUC??dzjRHs=8;hikg4Uw_o;c%wMYhkC zlr%mSi~5*Pw)a@ZKPuZR|E2ci($@BiF?pYkGyc~%ff&Wj-x;%xul%SI?R(MQV;O%y zwpaeA##ikWd@2C(+P@R0JxvK~d-|~`L>)iOY|{T_* zoKK2QUDfl?c^Zm7v`XkA~s zy>NBAx2R-g$-3gEhSp`Bn>N(6H2Bx7ZK>F?tf8O@ur;-WFKt=6dgF%5ruy~k@-}U# zT;W|`xV@#as&>te@?|Z{9;E5`$Fc5dliX9s^ zg75nBWluD87dAuQ7U=4U`jWyt--cp;eX9@lSyx%#w)%;cWP|1Fn!QD9Ru;Fd%4=v{ z*Hpi$roCjNx3<<_v$Un8sl9al)@pBY$;xHrtMWIMmvwDyZRy%nj@P1!L+s^Hg7Zl{uWgJ+ZHFIv2K@uIxEM>x$gZ%c=-PP~`Y;s<4X-qyTe9%DOm z?&!xS=lqd-=a;?&M=O$8%vGzm_({@eQrln@%K`!|HNM%4=a}D?NaBtjWK1B;I zm9?q&d!Ew|+y6`Ae|3B<^&8F?MbAXv$O**Tt}*CotiuNS*VX-U=aF&W&|kF6I???= z;cz{v?k#lF06(j6xR$E>Qm5-V{Tp_|^MCx6aO#&A13uRfjAmQ#L|9YYZdn)vF005*zCt)1XJ@^FBP z4^)ZwyyvyNYcz^e;yq_t(DR5ms)fxku0jg>7(ZsX+zBZlu7cjU^A^Z19tl)_ABIBX(XYG?{P$=dkDm57@!$_%2mS-ViJ#Nl z4veXJrCo>r7m#=neYG^f>rxIHGv2swN_pMV1ZsXYgP!`iEa9qcfrP6~YN_l$NA^=K zO{Vnx50Za8U5Xbqubz{9z9QjjKD{pc8IT54@b!SFt3lDzYSgLVDZU)OS=RqafZ4Zj zCHg#!L*<0Hu(xp~_!SAaOSqD60{TaH_DqU^={>JwuD>d9gqgH%KL_dTl@K$gj?nPrJ)}q{~sCPMtL*k{GoA; zF*5Hu@WT>L{1{iW=Oz3h3w*#pZ?y005^m+6o}xpA_?aSZ`TJrCSK}&QY?5#_u7V#n z&>P2nUc#;8{!+rN*myqeZc@X%GD&{N)L*@MZ&G}z9ivRc}r!;QRC*xaaS1N#&LrZuEtgLyCmE??)N0z zIxgicR48|$DW!BragBjI`<_4b$ zF0|)sl0SDhHj7&@9{qepT^s#8-ij6LR+ctZV+x_-m8#Fe4W(Qj1&~+&4sr9m z%*W`nHHvpK97l7oe*loiH|EyV(j|khhbo4m##h(pWP9q{SgKXwhW7zh3)~EC|K7>MtF_TWvzP6#+$IzFKLw$@ZhRD=#h0R>@@arkze*MuZUB>(I*+KhQ? z6ImA&HMs7M?lLpcWp)UI^D*=JzLj0SXV~>Q=%|ljeDnNv-Q9?c!#|x&mT`#7Ouyf$ zs?&Y%NdoA+w{O?ag3i5l>Q|@EId$q(-RgU9KX~)9Tf82R;^a|(t*4rp;{$oIea^V


itcw8v6Jv-rh4S ze7(Q*shwwho~gg~sl7cNYUiZ9#HypdVygF4zfx5%Q~8QTy&r#NA?k&Pe*3^Hd-{Bl zpZ#@d_`q?Lc|E?LM9_Zkj>nGdedDnsr@pu2$d2b9JMyeIJOm!y`0SkW=JSJ(RO}nf zE%^T6=|{gmc;fVr2OoL<&B5<2du#CY^0x+GyZwW~laHMqy!QDI2b=Rg9L%|DPVZ|i zk005)?C~Rc*Uj!dvHbgkuRp(UaOvZ34L|g$P@Sg9# z3GC^?dp@1wz-UlO0|dtXEO-VYx;(hGSW*|$%(^+?NGgS{;u z086^fg+7lTnfe+qr@yB$+tF6e6Nj`sdz()WW>1;dd*ZPjNABs}aiq6-uhHxDn_91> z-#_EK5jFO$_DFH%4G7cYDSW&OT!dyk^*ed{ZIA0?{~dBX=h9x5~Rqu#)6 zRJZ@bWc5MNuhjL@bw=|o_EkD27kPe?{5N0kiEG0{$Kj{%!RL+^sNQ3jdU}ug0#iRI z_4b~apvc#r`rYL45b!(6KbgmGANZTz6AzXCQd-J)>r4-Wlk{upe;PL+I**U-!hKD}#t6yCQ?A|fqCu_%tKk>{}tEU0~?J?oON!MF^hWb^_ z4Qlm7;Kq&#A1JycJX9WZ$e!nx{a_np2Mux@Sl}7Eu?}@vk1aRi-0D#`59g!kagfTO z$F*ql!GqVH^31n-K=y+;XFc9qpm`6igN)OGJ!$o*39HpNLXQUv!h>}+X+5?qRI4uo zE|YW!Idmv?>+s$^&;hh;U~_;Cd2${t^@j(keoC)Jx^3MjTWWD`^?L7SoI|e#8S36c zx8NN8v|tPRl+LNYK=;r&^s55@Z~tA@se{N*CVBfNpL%bxS{*>&_)+#C`X>SZ2%xMK zI!;;wxhfrUPjt&&w;tu7vE0>=`{hj70QFeznx#0mepPoX&aK=BZ^JpsO+K~^=T`0- zI=6DaM;ORG6|#{}O%zU`K`ReCWG-{d>{$ny zL1USxLgq+Dc(4XMSmtL};+$lL?dn$HoMpZP=hoKcU>W`QU1tTbb+V95PRV%nLll zr=A7w+nMTDJ!DtdIRHA@)^n#?-9Ia*H;lM(^6re@Az%L9_k3fgo{ePop7j;&{f)0+ z@5f)s>iyW4x%U^jf!<>U*}Wf5&+VmH;6ogvISuD}4&r&zi~K^&N%cz4PfxAI9H&&G z=+7|D`+dHhRJUifsy_brnBP{Y1Em-*3CPutcB!oI4pp549*AW}vWvS{{rVTapG-KL^4C83zAvzIhPP<%6xBC){8G=*65qH-KUo|e zdVhsy=)+%r`^h(xFAv@Z9w)|nhPL}k9zFZZTYHZ`p@#DPJ*!sucdU9s^+c!qr1QyB z|Cn{))V00?1CQ7KwBL8x)Du3>5Mt#Im(9BLFV1`~>p;)o6Z68$kNQve#`TgMN8iud z+5eYU?97JDMUcO~;I1ckT<(8#QvHO#IC_U?Xx$UFKV1d+j^=uXj@IYw{Q19m4kTy! z4y^L}_WID?4$qiJuT^6n9awxx?{EI3sQ24j!ao=+@C=>F^&EI(#<&0cLFCW;R2_Kq zUjJUoZ}b%G-EnWp-u@>NH-(p<@Sl9&zw?AIW9R$Ri+fK+!b3Yye)6lHp_2>3LnprC z89M&mrM;)KJVU2$^c*-f*>m7zn;JTKBj$wy&w=ChsQZu_I&mBF5n$_mJCEP(8QPQg z_GSH%#7#fHcH)$ibHL|OV0^&V!xoc$J5OxHaV(B)Xy<>~Mb`z-f&Q16TWR6C_zCSPQK9=yo@ zVgCQ)Q{uUwZoJ6;eI5Kq8<*@;m7$;A*3G0Y)bWy(EaFICX zzu{BjS)PBs$a*aoiDP+~Kk55v`rzL?rmg#odSOzysD708$U-dr1!Czz%yqc#`b<3T z{fzZ6$9|^vzKnVo>cii6T(DiFg|by>ab7HQ=E=iNIdi;3>lG z0gkWr>BzRhkz%4!SWIEeE2+SnLvZEWsPD^XHLrQU3h@?vG-^?H?WjL+lsd6YjF;@jlQSBS~#L9_2y z!v$XT-?e-U3pbmyzD9HOZd93JHCJUoqRR+b(XpW;7LC=d!xMmjdMHpEp$ zYg??bqV>Lt*v6*T^=;{V{U)2guOf9)-`3KOosNoFV?0)2REyMaiZ$1DbVO`6*4Wz7 z7LCRt&CM;5SX;DyL!>d@-WY9aX>5(vHMg`iG&VQ2-W|DbratXxX=-h1sf$NiW~I+& zL|X4#-(D4obhJcfRL+S+V(l}nAzNlenpztg%sXBBHpG= z9%);5Ph)*7QWuLv7PYl+u8Fm^L{@KzF0GwW8HrRvPovl8K#h(Kbsz-SK*>ynAO)cH{TttYu}&;U3;`G z))w2`j_Op_*4h!PYmHTY4&1lCuDzu(TK~D|W^|3#MfRm8+R&*x&`J0l^dYO`_E}B+ z+zesvTdmM&9c(8RXDsG=9cCTDOOf*)zgUKRu?)ciIP#xqSvZ30f3Xbt54L)Dg)QnG zNx1lB$bbIp7!J~Rv^2IwszNYD81Qq)NxLEh;D|UC$HciKFhe*3YFA6&NR2l%j z4_A|L8n$@Abb0{vP)|XMZpmtw#dFh){{iSe>|e-~L@KjHI;RA=JLy=EwKC`n7aP%TvLdh4cd57hp{Pg`<+W`m)3+9wME;pk75#SlF)w1 zY<$)esXZ2lFD}~DP#2=zg{J1ZSW{bTDAv~2+%YYLk=X8c1PsSn^)bjbj9tFo7b$q`S#kIYt?lt zLbF2G)nPLtWVRIQLTH*~rTrD`KMPqms)s@MPY)>ag!yz{XwGq}{6*>Wi`mZculLh7 zlk(5;8@`uP%hdLZ9yL&?{5GGZ_8?!A%4ezs^3SI78EOOab*a2xEkS-uD(_QaQImtlfK5S0;CM1=28MdS%K97G!xjI~^R6jnI*yQiVM`2engwI-R zIETiAei_d3@lPqyzC!*YlfS$~h01}&XDvSM`1IhjA0O|nsC)f@ryCBD1p7n+e>|D0 z$L70je!tCcv-v)of7<2;0;WFYb5Oq$Lyq`SeaZ*nmz3x7FovG196F}+#i|dxg5;}Q zf8{viC!8OE4V>lOum$xdmnS9|ex#3<=khdv59H}Db(TX|3`%a_b zo|Pi~b394aZSzJSAg?v$rv0R9w|Oq_7vMmWW8yhcvcFOI;7Z*#Cl3 z_?N;BW#cznI4wgAus&+;*Gf4hq-{S@9 z)fMXXC>&cS81^3u)Q+ecLk4JHc_vpKDNu_y)q2UMuwJe@S)iuJwRtG-BmCm+t97Z) zqr59lJKQ#DeR5QS^RE`D?sx!_237LCWPv(Vpq`0m2QbuBGLz(`d;qrtr1RbJ9CV!% zpPS>x=j1r?uXb=_I8M!1Kirhe3?vfd6alrbNau6NX(@lD*Ob#BapZGo5ZHVP_n$(7 ze2X_Dkm&^dq3@2k8bj^3uN=r#KPyoC{y30SO(5iee_6iz#ww;}Ee&GWObcKkV}zWT}1lf7b&UNp(A9Co!JOSKo;T{M2GV!9U-a z4uAIQB30lCsJ9B#md#!-JXNXjSLCVZu7Lk$(UJfORJ30mRbI%KQ0nn4b#w*%KO^#J93Q9LUnK@(IT_yJ!O&-SPC@aH_V zM-|ZjJ}O6EW$QfbN#$|nrRwbq+f6)(dy|CsQ~QvY@Or!z4cc<0%QINMct!@0D*3vV zFYfmfKb=Q>sP4+J{bl5i(Qf z{duTVl}%Jy0G}6m=wDdm_v&~QOk}A;(~-~6_Eg1a0QT&f0Ci?setYuOmhl<>RR15% zcgPF-gI`x1y>DDr+cT}t8QnkQOHBW3`yPElT~*}GAlLJfzZ_bj%8FDL<&$2@7pT|0 z+2qYgXch9j?hTN;m^|7mE6U2C{#JyCy{n6qPdgCtL;m$Eb*aI_o-Zvj?hGk0zI?;R zw`=@^zk9qd!1#5HAN0q*`!kY>Z-IRP|JKRzQ|cZ+=-(}yl{aO--TA5Up~nOCr}04Q zX#a=)!vFo_i@K>C7+(HOTFV5~EFp`NMT@MIai>_yhYEMt$199QO95;tSbN z+A|d|IG&hz;Y-B}pDz_3-X?qd((%FYn@tUB`@&wYM~m%ffmgh<0`{&&DJ5&~coxkp zCPCh3;#pY~PxX9_^})-VupFZKve2vNyTzNiWB*dHy`I55?=ZR*?OIdXY}{gFRrcB19x>ud5*cJlv>2 z$>{I$j|S2&;P8*>&?o572jg{^e;8NLEHR&V#mC~*=^p{%AKAh`E~Iasy`cm8jOZW1 zU>{mE{xO~O4F*Ug?H^h0_ziow;&a1C=U>n0cwFU($5*7{aTSf{LnaF)57#e?H$9b&OJgL&myd=2 zT#5AwaPkX~8-!=D^xI=MedUD9>+9~^6I!h6W7GBCfdjPEu$kt=?PS7p)MT#}#HQ`g}7lOnvI_`(0LiDqO{gGsv{fWDZ( zkvzs;Wzj6HewJNd!9O*D?N{y1qI^QPNAd>DdfCXU{omrvdbYa6Q4W03a~Vl3-z?K! zb<|7O4@fqT^}Df`$rmB))BKt2!+I6)yIv#aBkdt0tV?12dnVDWsV?2=kwF{BIV|`K9nWfh=u$Nw6d!zRHfXWfys-u3t z?r&?)!f2+Srn@AszoS`x>L0z{!TgTv@$~padpo1WS$e%*V)$>_9LUTEUSE%Gzmz-zheG=;$U=uV9EG6J~76pv`+KoKjCE3M^1bEQ? z0Lr^2iur1yoS(Dse6CLVdidq+WN~UfM*VM;IOi*@XD)HhS01eIwkg`vN+uuD{z?9; z`B8tp1%2S}dVR?8X}f96_%p8GXg?rKIvabr*4K!?WpQ)8&gHPbnUBcc=)WSY*Jylk zf0V_?RCtlc`kU+@EQ-T@iT^yE2fqT{KeinGAEfROYLH z?&xKi*xq)Omt>N^YdrQtyipTB?0h+FzAEb+tH-ZiZ@oGl-giAPmU*-KA3M@@v3lKlBQO^nbbj7hz?^s`sk7pN^f`Y3CJ$|=r&Pm&=*`xMlAwD;u z%e2?l1TH2=UZjfEjzhzs6X7_4>anUJPMu!u4%e9P!LyAKAZYdw1#f zUH*vn)8$};ysN)SzVrDj{9)gvj_V&CU&p8WAN3JWOI-FpUp%Gj*X(48(_h@8GoMfcCjQThkthx9}L&{X{YX1q*yj~A@3E;K$|{`_3N!=E58 z#+!S*lwQnu$xn?J)DPPEwW1&ENyy{sf7&m1jvve)zu)*l|G37F?w|Dd!F=HMckFj; zKWD%7hQ9AkAEv)ceyk7NDBwgSdZ)V z(c(>c>F=XX>-kXcr@8i{C;-s>h4vP2+7;0A6J0N2KgxVxvJfp|RS0|0ejKfLy!!hw zte@j!XQMW+ZxD}y98YyTEbOe**GtKuS$ zq1N(0O7O?M(f&e$`{x<^eG9(d(Du49o@drqRDT_<2fVZOdP(=E z-cQQZUm~lu8c+Mh1Z@vOYUo?mX|A93{w(A#>&z+E>qnCB*mOVm=?zTm-|O!a%Q`Q) zgapp9*FXF2$9{1S5nx}~|EkWMid6fr7I?7!_tA{T{ZZDLTS4omT+N^CKR1W?X@5og zS4D$hL^6`So`C;cpF{HIl2_987p@Nz$m{O|X#7VD_4S$VKa>~6X9c=pF#bNNu=57J zpiKt5^!{-duAk_70!iCreso$=x1Wh7kCq0iJ7>+J{$qR3k9xUW$Lsmg>-6`%7!O{G z-{a#mwSUhd|Asy1M}1sw@qr|Dkj3ZBkLGaRmS-nvJz>iO^KpG|^S0cNB3rKcXXx(- zxc&Ljyd)+_?3fg~Tf7}@jnCn%0z4#=?r}lq) z{aO|;$LnR3nnmNC;$=Ko9-``Y|6+W+Q1^e3`XBR6QG5!GELCXY6UKLFCiyeGC7{al zk)Mb>D3sqo`QQ|Ny+C<<|8`@%1bJItmWB4Fl!vLl84vzy$g`I7kY{{+j&8p^)&87v zYJZOHuldp9WC(fVAEofmOOlgPdGgnB$tzv?OOumb`LZPKPwM)wOb7m|B<&BT{q5=` zo<%fy*n3{IxOf`n>7wjt0sQ$IonI8?YJl6jvu{<_^CCj7xYhJXh%)t-sPiFb7 zzfNTNZ2ikwKAXRS<+J%xmPdboJTDpwQG6**$yVK?AOkT&6ak!vMm4n}Uv+tj@VD`4f79icrTxvuE1ExW>ziJ$&W}z=`=7S=RMyYRe=X~$ z<*$xkj=NDQnfVd^7b1RIpAXFZL-DeG-B};q?}hkSLi4qaH;C`!rw?F#Z|5V#_X*Q! zy>Iha?@X^q06(4PANV)TXEc7>*I~aM*GrVA@!G!5d|!d~G0WKYKQ2(&@nX1F0{qN= zdUm`7?vc(1;^W|6l*hCTdj#T_VuQx!p?@Gz^GhDl+RyQ=T0S z+V!<5-~M1>kt$7~3t$CUV*7(A<~-J?+0nuximzxB>&sH)#|9~W>pbp{+5RAQ(o5-* zgzJwRo676@#f3Cp)Ah%VC3(~JFDV8dmAU?<<5GEDzoa-Nzixm0IMPqo$Nd@G{zT2d zhKaeJfqmrl6#9F1HPE{MG5>GP4pyQrMK|0qP5Ht~6>{c_DplC#VUMlZV=L8Kn}ofB_7-n<@`wF4?@;fu{a_FJzTkrGb&uBG0;3<5kJ?_}r*Cgm{UWBl z^Xm8Rr*Cgm{cQi8SHG32-(}AXvtO9@j|@NaZ^&0}|Cmntm~yfo$`@}x?eK>M=&y2h z%*9{fy$(oe{)+aKwf~LG-;|Ha|J*SC2)O?j`7cq&FIIkx54U}^{nPTJe7Jp{9>#xE z`9C#(8ZV~>f1emHei;w^KE#vr!~;`4>UdeAPP^?c__4Cp7^yu+mk86K%@#b@BfA16bNB;-N zlj*-mUs&J^3w&XLFD&qd1-`Jre=ZBqUOx8g`hJE@(vp{b4=-8%Z^`R7|N7cb?jJh) zz$c5Ye(+@JuYdWYKm41Q|H=QoIgd{M_1PoOJovS{4?O-4_fEgzfB*cQg3`Y|bM&K+ zPA%}O>6g>S|4P~#Knnf_OQzpV`}IFRaR2?k`R57Blk3kptNbXb(a)=-^Z4xrwC%~w z%=tu@AN0@zO6h!!XPPVT@%VL~`EALi8;EKeRL)O4o(Z$T-ner^ngC)X>lP5Pn z$2?*0^WFeWym>kKKD~y0$m=-^{(5dD$#@n7nIiBQJHMwOuHF;q_f4o6&v5_GyQ^>)T_ytUR!}!H*d#3!M zVdc0*Q_ExPd!;;UT;&g;U)=gEq4KnRDVwvasfo9fL> zk0sxTW0A0F%tK!38x*l>@>cpu?}b+_EcQ;gb>>-BGlj;8M^T4o;0qecnYR&8MAmXr zWS)|Tcwy!8_2Kq(4by*g{V72|m6OzAPoGEYw#=JTgAds-N6!?-hNkW3W_nX&3x)7W z`Zk`Dwj;(?ju~5{`Ut)rG-GQ-AEB9oM&8JA$E{Cz1bqU2n(;hmw?OHz$2-Lje%voG zl0$zLCZFW?M(!JDeUC4J3J}dE+R!I^d8VY=^r1~3^^@*XS3f0SAS^G!h5`8~^;62; z)K?hqY1<|;Va7h%M5jcyr9P>dFjx1B*UaFl^?_I0oYn_j!`25hF1>7X6VJN5cxsxH z+n=!eK9ZayE9pft&`j;}kB3rYEG@qx`NK~6X&qyH<01G40)tq`ketFdy8QN86q3~~ zXX3o}yr3<$K66uk{_k#2vLj?jk1=CIvg-xyYt|@;?(* zjQ@Gkt4aK>TEfQjh3J`s!{VBI4dKfOI|mMGJPpPaqs})c^Z?@+pI*s>-hiXy9IZtr z973E6;)`#Z@#*{ObV|>?Bd>>`sUF8QMN#-RY(dA*_@Nj%f-Pxy_HpQ-u`F##V~Xl} zQt=ojMnoq47Jk+1{@6L^d@_3fKwW)SZAnm@Ss9rb%A1RcBE4dfbzS)E%khM{-_-XDJG$D)!b6E&7uSZXo*yoNdXl%4 zhx4ww+E0CGc;x=o@xtlXewe`PbG0S-WvVWp%O~}CqPXMclS4SC%yN8s=jP8;Dy(lz z=)x=pd>^eUIMMp-LRW<&#D!romge`LU`OK4g=U3^^&PG7$k}jzEv>@oyYkTybiH(b z1VzL1BR(q+tRJ9}VH^7v`}tgT5Cm zQzp?lC7<$~_icey7B3Gg7a5A5_RstS_+wug7^FFu)5W}NZ`gbE{vf0t`R0|ofAG!X z7WKl^fBnrG-*>K=HmUNpD}MAuLEh5l9Ndu0@4oww{I0!z+-Nm`&wkwC8pcg0p-DL6 z6Pk>7ilG}0Uj-{&4P7qquD$RLO0C7G2cJHC)Q#YQkM~y8#k=;}5x(iVn);dEwMQ!t z%9EEip@%8Y<=wcU$60>3#F;NvLFh{} zJGD>s!y)l5y(``2hhYndkizf3#_%J3sD9tnG=F;6UVDYJJV&jCi?TfQuDyYI&hkvP z@+NbS8R<{&+Uv7ut=%*uv$9f1?*F2J*Yu_DO1=I>dnodsba!A6J3;! zRR2e|-hUOHBkR+>cwQ$%d%i?J^oZL4Z3V(mTPzM+i26hFBo-_6wpCHnQ=;)nvdj8QsSv}&9MXTc+WKXd#5R`o=^E|f z5$f*|^_jD*Kgk%RJl|beb*)Hli!2IF!`(OPzYImQhx5oCs05+2IrTz;tj*j8w zCO4l3C=!+pT7MS^v;{B)@T3Y_PHL@ zCx9?g{TD~5PqU7!Pd}6nS8a#?xa@z})+ZUAN!tGo)c>Oq^ym5mwldsQ_rGkP#st}f ze<#esQ->1UARoCtkw0raa_Q9gOtoZBD&0qCdn7B53yT_d2$N7q&U%`(#L(qm9$$Rq zdaLonLpMjGZBbj{X8bl;eXJu?9cr(m-!zIfMmwCmLicZIYz@(`!Qe;BZV0W7MCd#c zDZl$mH-uK}-#RP*YHQ`R(9CI}88hd8ZCYr3b6Z_(ZltlQW!C!EDbqrC--asn8yX$( z85%oUg>}ts^&2DjrM6~g9q@0R1@=v?u~{?0R{#E4nkN-QiqjWk2ZG_G}K zipP|-P;_}yM@wA{;x&ZmHqcPO=DQ!LcBKD4fFQ)@#AHSYcrexargKSqW0)yl6C$(0=K#o9va8exCa z1)*qN>)nl^uU>->GiJM#LuOd}GnAT)k1;M(#c%6{9F^R7 z?K&aIV&M3pK)@AyeA=>U2I;c2^pl}8CkKD+INF)W6K|@50x_wsQ-19)4py3_lb`kn zJ$|?~r5uB8Byrv4R^-VJryRy#``aunh907Vs{@bzN!d^PApB2Fu1d>BXAZp94_%`= z?5Gy3bwvk0#ebNN)R0YSUg`eeGCK0)<@x$5?EhThnV46q-}}Y$Bw@^tJWpxp zFOEVFSU(^+DK2p;wwoJ7zl@1EmRR~6(6MEgN+C;Uy?DIf)eSnaeVAVEz+2HYZkhw{ zq*n?&$07aFS=1!{glGC(i=QjxoM-8z7sX&sD;@O9nTTVvgO2UsREk+T>B~A%;Vw(R zR?r)=O;xrJGumM3vfrC6o#ZTKFpgUtbf-M8IQWzQbKE{;>FobZe?#C0Y@9zV@JxSO z;DfD3sCNXO=>q~^)n@Q#1fJ<<1%9o?pA&ed>+kBolI%(P`Ym3+WE6SUm+1i;vKe1& z@%&ag#xuP{;8{-nQUC{kriTQcuceXL`H9vtBWQXF9(Qp6%Ih z{h?FfnVt~*2P}T8z%zY^z_+*S?o-_Y&-AARp5=U6;F-Ql;MZ9GJp#}4-2%^Y?h$yV z?-%$6%l`#|XZlM5-)-@I0?+hU1fKOeB=AgsL*QAj!vfFrw*{W{dPm@y-Y@WdR{moG z&-CL0&vKp?c%}~s{4&e`jKDMftibQF_;Ui!bp0}AM|?PE{Zs$w5$GI$m>#r?B=&#S ztJu;R&-4<3XT9*+V4WDx^pL=_US$H$^m2h`y`~8~)2jrY{cx_pGd(Qu1J<6k0?+g% z0?&Fa6L_Yt6nOT-)dJ7-H3H9i-6il$ZxHycR{jkF&-7-2=Q!Lh@Jx>he6jT({WBR3 z|6zJU;Mopa1%IY*6L_}64uNMn-g2Q6`)A-@lYdI!nf|oEvtGLdp6RurOyGH)TrTiTpC<5qc05)JJk#e2e9*>+c>>S$u)woDYXzR^O9Z~( z%Ck)1nZ8osYpoqt3p~^B68LT_|5|}(dV|2ToErq5>CFPqai(41nI02(mNPE!Oiu_r z&pTTMp6NRTp7rV$c&0xk@T}L<0?+ha0?&H&2t3nw3q0$!N8p*hU*LJ2@PfcI{Y8Oi zIbRZZruPXvuM=Joc%~l`__fyG-Vk`Ezb){*E_z4cncgq(tk*GtXZmr0XT44fJktjR zp7lB-@Jv4|@T}K4foHm(zQjl3_{M%{Un-hfA2OYP0-Esyc03kaJ{%93ULyFjUK0hL z=^=qorZ_nO-IETdn+a1)k~i1fJu3Sm2pnEAiIfmPov%uM~K;!)k$N z`Wk@`+jx7Iz%zZVz_XkU0?+gf0w1$-HVZt{I|ZKQOb9&Fw+cM><2He3`VN8TbzQf> zGyN%nXT6>lc&6_Xc-E^&;F-Qp;90Nz0?+gp1b(gUuNMWL=`RU9%h@OJOn*h-SfZ0&+(1r%(HaHGd(EqEN8L6GhM&j(h+Z2&WQrg^pN1sa^lrsIcsxQdc_2u>2ZN)z3?+WF8)m4D)F|zwn@CD?+|#Fvs>Vq{*=IT zTzy*LnZ8TlgHa=QkH9m1kHGUfVV}S=eZRo7pT8jROn*_}+0S1Rc%~l`c(%hE0?+ir z0w1>i{IE!~yw$1RJCh$zJ6nK`iO5mA3 zSKwLBc>>S$u)y=YSu5~N=ljsv{)sjt|1yDR`WnHX?QoaCGkvYVvmF`)p6MF|p6$>q z@Jw$Pc(y}K;F+Efc$Rakz%zZDz_Xk?1fJ>L0?%?jCGbptTHsmET>{Va9)ah6+%51- z-y`sqcD(Nsc&5J~@a;CvzbNobe@WokZhZpJ^j8F)+-XZjlgzs$;iSm2rdj==N0 z{I0+={g}Y7wfv6@Jkw7LJnJZK3Cwkwj2KQ1fJ<( zf#-2oEAUKTBJe!!mI*x5*9bh@;Vyw^`dWbxTl+K!JkvJ_Jj>ZE@Jw$P_=NT6n7}hV zF7O;T@H0<3vHvhVA@B*ie%>nZOz#$Ww!>2b&-AASp2x*5foFP;!1Fp`x4<)fkHE9t z_6a=G_X|AR?FE5n`ilb3c6&+SncgSxA#2Z91fJ<{3q0HH9f4>1y8^$~`dh!iGyRyr zvmK5LJkw7LJlkPF;F*3#;Mopm1)k~W1ist)vwqkG9?Jg7^t?jjw;bQt4*ZBBKGd&^jY&U+sk^P71+XSBD*bc#;>D>aq+sgBlz%zZ9zz?(=eR>3*>AMA< z<=i9iOy4K)Ea!fKXZi~QKVap5QQ(>0C-A&Zdqv=xen{Ziuig-NrXLn~_N%uAp6Txh zJp0wV0?+g_0?+p`o)vhepA-0jR?}a6voHG()BQzm|75)amd<#l1zd^_k|h+p6TrZ&vM2Dp6UE^@7%vECw^93C-w)XCj@_%bF08JeVf4Z zx_5`bGyQ3SXE}EXJkxswp5@#v@J!z$@GR#(foJ-Df#-4kg1|HVMS&06@%WO!GrdpX zLw0<H(eZOJ}}(&$-{y88_FSFul^|Hwb#ay(xa1p!4$_eS*&Is*R1&*2dtZq|l{f65#^yWBzbj#vmjq#Ymqotc- zjS)PchVr`R$i}Lfv**lNH)m%3%sDe=%$(I&xxS%sM)j=g=ggixqke9~Y|Rq*rkGlN z=ki-_S*jmwy`wR0xE()zdHM*&a*0xw<#p2t)d|gMZu70CI-2&yov?hi} z&DNnYq8dCZx~L6xwff;s7u}e&aYJN1J*caeExU7hZBtz{9!6c>*08C$QIUls4Thx6 z5@|MaMcP{H(b}iBZzR$XYl|8)Gv7_Ec)qvc_vNfb)r?|o5%ch*u@UTJR5w`|z?n7H za@z2gwx|V^H7I%{6weLgxVc3iqiysMrD3(q*kT!^uhqoQV2;>*Es^!>n<9~p##p40 z9-~HQw6#R=>~yRnG8ZOh#08xfp09Lz5CUyxo0htTWIeal$Ssq^ zZP;}g8)LNb?qqAnS!UzSrg^wLTdA4#RuYqq*RdUvjj776XKQPR!5#)Rc#YutGD@JA0z zhop-|dQaKC|D(ud{}o-(JyOir!Z~}Ix-G_pAGU5dFI} z_UUg1wCE>2_UKhRTliZ$J9c!IEkO49$TC*<{UXmRziDD`|AodDiqj*#XYJX?&e>Ox zt>+_ItnPnmo>wkFPqW$I_9E|tFC98_@x-%dKIl4p=Jd%!XZ~38dtttRw4?jbnPU})&it(WyRoM$PR4$A{&ei(>7U1rpZpLu z&&N_KHbvjR^yHbeH5;ODqC5v-<4x$Nu6{6j5cV&k{nng69!o`A9You-cb|^Ei8gtw z9J*^ZN8dbu=uGtFiCDDabnIZorC5rs=*s)2pY4TCFy)D&|D2K>=}q}@QJDJ4V`p_v z^np!_^bWyi?zVY8J!`-F7-73LR;TwbFwwtv!+%5{@3yhJH_*p68)0`JJEu82Nbe|P zbw38LgzE@fufh*L>1m4oW}p#%<@xmLzSq8I_Rq!q?Gw9gt@1hYFXdaO;9GwN-@D^m zzu&$Uxq)vTy@_ufwfGkKobsubij@?0wb*v* z!8f7z!EDkiAN+*968{-{<-_$!(Z9OO7CoEEe)#@A_R9P3vW55Wb41^7Vhf0o2QLh> zg$ws4N6E&88jP8UwF?h08Uvq4*%q$u`>`GqU$*i(ndj3rXpa;ddxhfq^PX3u#QPh} z88@AK&<@lFqsh2K|MMQB9n_p;w#RN&zSI+?{7D{d{R+<`b@4o|W)@!gyvN3U#0kBO zO-<3%@`h-7ZDW*t1~(#|BH=fv6(I?BWcWLqY3m%Cnx#G7JNF911-rsxG0Zh zEQ_+eCXEMzQ-O(Dwhc%rp&BAbOUGT)n0=|;rr8;Dq@g=yvS%;{9v>I$!UhFs>_@y# zO6)Anz(~7W#9bR_$sV?WC80!lBw0B-IvWaxd_#Cr_xVXd^SgI;bjlxQl~dE9K;_h2 zWoUeCYGm4=hxbZ)u2O#(o}QfXg;-@M5Dry}s(FX^h9-Qov);)XZ(wpL;P*~V2PeG} z3j<&%xXI_8WylSd zDtDG4zn9CxYhAz8>m8d4VJYvM@VdRAhWGk?-r32Tu-88}=M7F_(K`})%sWz3?VX*N zeiVyjOm=tHF283>{)DlPddJGR^|i~FfAgb8_CD;uDyK(AW&lvxTMXY_5P@ zbclfTEH?jkegYt9_4z~2PgSIT|8ww{XPN953+N9~M0)f{Nr>_EGWf2enL1yx+1pZH zWv%4TgKx)qh&9ELN+pw?DWTk7qkCrh6Ol+6bH<;DlsWk$W5h4F8j-YW14jWELf5fV zHYMq8S|p#y{z#cM8|Sx>Z8jlUI8+hoUlNa%@chXQN+deup{}aOHA1YJP#9s+g!#Zlrz2w%zR=k8lq)nnJuzGE!XOjy%?4am zHI=pHt{rr%tlP*M_l`~YU9`Cw)>zur-|CFAWIaSy4 zXX$Rtv0TVNj5#Eo#L7Sy>$HQp5@=@ zox*h%7sG+A3zyc8IuN9pOlbv`pInEDAQ(@uXll&lQGqxJndl_sY z=tv^*_^HF0?6o4cf1eg-XThK$?py}+opJWhIPGMW8LWa1VO4_tN;Cb7Oz0!Oix43F zl1#QbtPPTVjU8UX-g=HbG4Ig$kdj-N&dxQn((uXxs#=`zbL&`kxFVi4$5;h;RQ{we z{F71~Jq|iMT+Z=w;&D8Wo=mul<1XU8NAks-rsfg!E>7={2 zINhYv_jCGwoo?qe!dlwz;B<$McXGN@r@J`K{|&&~dkd$x==fGnZ`J7yoZdh*FaJhP z)4D*k$2Lx5_>%O)oPJoRdpX^!(|w%o)9LM;-cB^P-_PlO9dF=tgHAud=?5g88%~NB z;n}eQ{)~7?4>AYE!_3gH?I@pZbvW5ULvmb0JiL*~(!8(t*JeZWc5@h@{==1hU=JGt|Krj1RRuYdl=l)FgMiJ;`N%;RLTax~q6Y^Qv z;U!$2#z3K;#A!b2Al}*zzC#F7#Lhm@c?E#G(h`vCy12pCoqRU7= z9W*LP^eE9z&@L051G-GmLlmE`3K|a}f%Yc?A|9V}WQQO05N^a{ zMY%qnaQW&cqGfzRzNU%fd3?P_`SmmsZp3GO1JU&&Kgh3dAbBI62bK^n<0mOQ>{&v% zM=#&zWWss;ar+M?6K=$xJ&AA;znQSlk_hLCm$$`1IFENuuUbd69G@UxvX12W_{8n+ zswA9`Z|{-4?n=Uq@vXd^Xg+>&dyS1m8{?0|K{WR-j^Ce2_IUihmZ@cjlQIeC{=o63 z48mo+BfsPf!j->ehntrYF2@JpOO_ID_*ZK>;jL6*UjDXp!j17|LmJ^co_T#XrV(z$ z^OjV?<#+)5TT=-)#)Hlj!ufc@?RTXRZj2}GPQv+k#O-%D2{*>0hJ3;ssC*oMAfIp* z&)MPad4!wBgFM2G@t`-Ca5bHjCU>idFh8!yl5aOFx3bS2H-7*9~167ogEp0g{8#GZ!w9e6 zgA}VOzbEFifXnfM<63=vg8mBNa{LhZdWuCuzZAGjj32^&55=;9mjRdKi@+Pn_f6%k zrm?|9*U;F&^?CcIv`{P>wC12#G2~P16w3xqqpKO$Hq%&O$ZHSLSYVPLps~b6dpwEq ztrRPUeo7m~iXpExQ>+=dwuEBMz*CwiR!wv=JjX15KgGJCpOQrVZOCiw6e}jYgJQ*m zcT%jG@Ggoq6TXFF)r4=QST*4rDArB*Mv8S4zKvqVgg;ENV#0eV)=YRG#hMA zT0h0A32$gfq#vMI=Q!rqgO8_ZtcZ3j*rb$Zu;O5$CVCh`Zxs<<$cj8Byr@ve&lO?) zP0=D#Sq9|YMXV?y_0S1)^ z>vgOruz<&UV%udqLw>8n1%s4hA&<{>uvg`-0?pI8=h&0;tRM&Rkh~I`s|z;RtF3}P z7{lLC2Ij+@iYzo(IVrqX1UlD6G>s`@{>|G(bT`p_e9d1)bP3UDFJT|^0oi}gM)o!l zF6L)$ubyaQKF#YPx{>tx{E}ZyG@qY!dro2RoQ*9zR98pw7tcDJN#j#-u(qm}bSd%u zL{1ColZ-6CooJr8{k+XY8}o7g0MQ;2(DTdE+vlkRtgeotZC?yDBzqx;3PMlo;N&Z$+$o~1kLUJcjfB5Uc z$BW?aG9LVuX#XNGWO{2I@Jjf%j0b-=@XDgf1bkhk?*D#@r*)O<5^(oA-M=~RUZ=+c zx1Ue`uluVOUdfIhTf33Y52$`sp*+<0R12>!+GABH56k1oLwU0Qii4UvU*-44I;PDd z)FIDV@c&hz{5-M;B{{$5X(Z1Z5%yc=ZTt(>Ov8HkJnV-HupvG5-*_VdUyb>bo^$m1 zF)yEX>=5GRYdeV0=ZV(S$RGRNEKGmA{Fwin=R@4pVAIYnW@=h6j|!;cIrCgud?3FD zl!qrO&qdn*JK_xM2^ewGc|pfqPAV@ubJ^f8T4^{H%L|*>f5SgJ<_fUkr^Ip0|1MV| zUYL@I7Z>aJYozbYOvGtAHQTSuNW^J+HRJB|M4Xln66f}7mnP!0d@%G&!?av5=;Ci^ zc|7*11u!0V%;lve>Zf67Gvqtw%J0%~6Ko$6R4nqCPDL%=%e*%SGvA`Ilq)XT~!$QGR#@FW+5_I*#(U z2dgz&{~7r=V!4!{pOaYsCJgOne7Pp_SN3UnB5~e61z$-lKP^`befVc~B2LQ}18)y* z(r7(xl#iBIX8SpJC+gGk%Fyqa%U+R))AGu|(f*Z*?MKT?!+v|PUZeH6%D+A6bc*%@ z3+>r4x8h#2{f@cHdlKn;RwvTcr8>>qztk=2C+wACKdD9*osS`8($7 z*XZ^jUm9M6<(pCdj=Ao#M7*p)$9ef{?@Pp6FqIqj+k@>I0v<{A?*o|1&Gy%>P1J9I zpjm&rh5Uwed7?c)qyC$NOYolxbA5AinnCl#@#Oo~f-u^nyj=H3j<3z9x!b@?N<{g=`qO%%Y~YW#PR$JOgJXIu+Sr6p`W)31F8uxC=l%jCfCm`b|v7g*`mD_p2OE)M)@|Z z5dEX@6*UyE23}by`cvW6cTxNs_&xWC{#W>2%gB`syuM!ei^8+9pfls$-NN4#e)qli zm~iY52C?Fi>r?8Vr1ccv+&=tabI{R)2}025@6ExoHmWbjIlUZT-8b?hdk(dq;qA3- zX}ZF>d{!FGZ-#zNic`@%zx!4eC>r`?kLt_!FWlZ&Hd1?X{nrKV#?nIUp9Eco)t;GN z!PNRYlXdCqHL*U+4d=$c0(+V~PiX;unL?V@Zx?8Nsj-)BplzU$|MGBtE{!kve+@s6 z(So_gdJyLmb>V`1Ixl9l-o$zr`-y_9ke_M3yoCL)Os4J66KhwwK5jdxdKdO_o|}gK z2V;@P*?+}(G~^qx-Q|h$$9l3YWeY_rlAx{Y>*e8i9udir~(Xhs6Gc&gXIqijW=oxHvB?&~ov}>&MTx3c3IDS_nFq_H%~(%9TVLxRy({ ziAVmH8!j;X8T!k^1-yN^zXGod78Qv7mhjKIa9$CrgY0#G&HEgtW_FAIXge4o%SgiC z;Lk-IZCg2RWMJ~?-ba){|bs+ zT!P1+1O8J)Z8y^!BY!l}pV0qvQ2#u{!{Z`ed3(_G5fT5Nfj!-R1V+wf7NCDbejZMv zy;gGj%*OL$xxoUye~c#sM}MaUSswKt`z7KP_Lc}*#~G<%KbTNnyr(|L$Ck^#rHS*^-_a)$`;WBKXaDDiA7A~^ zG`j~_#d96Pp_ZR_v<)xKZ>^L$&HBlQZS^+*FhjjIKVt&MNQ?=lUCwP{w>Gt zniZ!#Yr~M1Y8zrGlaWj|ZC0H!h}qK8Ax~wpCu4_Uq7QJoAuE{LkelUbT*~e-*-Eq8 zqE#Hpd|5A3UodKVQ(c!>>zaeQY8x`DudvHNs+L zA&$nrA&vsx5O<-U+Xl0R@8X%tqx+sc*<t$Rn9F88vbry$4SvZ99 z;F`g86c_y?hJG8c8$Ely-v&@aQ%BN$vG4sC@PjLIW*ejLY5~`w1O(TCnl` z6(O&Bwb{a#B>UcV_zJh)*MRZGf7isPr36#rcTLNfDxa!^_+1m7@FynmyQaGnnV|k0 z!MQxkus>5LMAYw^l!Az1^cjW>*HM}=6O;bCCdpRviBeyjq7end(KX2QEYN4HI2&+xfJ4&Mz(mUd& ze_Cb9U&wEf5{eO)U3NlX`lq$sHv-{)Zz)EpbP<=-q?v%qZq>&>mno@1PEh!4milhV zQEhD1?*;zkBKjezU#eRa4_5tR(I7J(i`=(V{<6U+f7%OJ^`8O*BG;7v4eD4V7ymST zMCq%3d=7?*=aF*ic0uYZTIH|wRR7OG`9}I%qyf?=IjjC3fJardxG9UMW6WZJNTg5w z9KmHy_{tx**<@zY^9ISFOl6|>yW>`)h%V$7Pq@}qX> z?}xs}lK)w$Z!SOOuk@7v=7X`Aepa){c#A223Q4Ozy$lnx&J4Oaq5XGCeW|YBtnJ?i z{eQ7Y{gu8`Drm4k{cp{m;)dE(y~hiW2`NYFtGm)C_TN{JTvCtEgxo+BONP9g^P{q= zxR6rU+&ok$B(ol8sdyH!{;A?i-Id;+hQ1v&54SXk)YR2jJ+XJVrg36dO<(^b4HMga z1OC0ccJ@{cPY(2t*F8M#YiR0f+R->VJlPr=+}S%ZJkz&*V(X*LzJcDyCffEiJkn4% z(%;rPziXgs!r!)Iz7O^W+ac33+c-Md+c&Y**WW+WyK$mvtgESY=wF% zBZC7~kNUSvBCo!#rmBsjLz8uL{k@Z{2Vk z@_>BfMDN7VRQDV&)0Q1$z6S7i_KxnF3{Sv*`{)ev-%{P*(YB}C*RXNArESl4|3Le6 zV2CGM@95<3{M!jopCNqRG-bAL=Ke_RdT>~BczJ|%RiPnz(kwI$Ps_EwH zJr6(9=%>2P?%LV?Sm$K>=y3Ipk-;t9vk7(WpV;ef-sPTX@9CFyXdc|zNPVza`KKRg z@V5=MZ5;JCcg>r0+J?|5z9`jHX;hMMY;stqGRNYwmA@)5 zR5cu^^Ld5b@YslNIB=a;;%(>e!c(urFj3a`-kBirraKeE^lrPwxx$#YD&Vg7uJ@9@ zcYAPez}wW1*8m25lkTt@T6rw)EBR8t$)eZ3XE? z^s60yaRGdkst4T5ahZ4SqcmVs+Md4B^G)q}$G=Mc*Umpo|8~X;!ZOMGxq$fi>P_S+ zujftj`nq^a#*wsdYcKFI5;)2I;1=Yt{p3HxQlicJLq^s&wQuuRC^+QYK@i#HsoB6w zO?a6FC;4U*ep%t28AsD7gBQd|O>z^rAg9!=_ZOk-V851M8AwF!iCYQ|ih8KVHsq-Y zHh!)wc#$q~_c@5qXD-fA^9vD+yZN#QV`bozyhk#X-^b9Pi^+FghyOe1FD6g> zqs92^ufzWV_*9;kc|S0w;)?#Cv6!8IhlLxjuj~R9S0uZb{9CfVr!4YgkS9AcvTKxX zwd6-6zZo>85^3j`lCRqNPi6V}Vh?gEuKtZIr(1UMWo`vuIhA%?vI7-=0Q_aj2xOlH zO(_KX#6KeGUkNt*4(`OKg)F7l#f`mYdM{gZ>?`cc96BcbWX6Co#!RrTHil1 z*)f;@2PVF`zVDd$YHvmPUA_)~r;Hz}mka@3zTJ|4pN0QNCV6xJz9{+D@}H1=Y6tql zluAySnaWrBB7U^blgd}|J5BQDd}-fI$(rw1CEuFwA(I_*yd05yYrMQ{vZKmE`JFZK zX+Nvd3OO!NzVv-um5xfjs;^4|FG{|uui{@Z$(!r@ndDpR>yVYP*7u0yt9;!u-{&P? z<*WD?O!DS@Ka_lHzMn|GHD6lSP@=U9g_LSLwS8244+$ZSikmL4=8bPz_*v4QPFeVB z9=KrPtMOj#-&MY9ye~ufC@DUMSe~|6_-eeLvGCP+&&0BO);lydl;a zP)y+Zea(TnvEe`hpWeXes|<|F*A?Hqh$G04Uf<=NqSqUDcJAnE9`j8EhC^M`{=E~( z{uXzGybmU(eSU9f+Uw^96BR=7L^%a1>SC^8>vT{Fe9^pEss9%G%op4<>z$nm3}5?R z_~5kP7xJ-6{wW+%^in5k(##h+7kll!c~4&DHBtZQdwda_ zv+70sU(&bqi^26D@Oc1}iCl6eSh*4qqUJZl?f4G0OVyp{g9GXLx8Z<6&_`KfX%dKoHj?GwT}|A~@Fyjx`- zk^0v9Ztk^u>UK(Lt=EvZ2QJ^HjH z(=y~lh^%7_1~X-_1#?frfM&9@k$H5HNWV1e*oD(85j0Bj$9j^n4 zaqsunqx;^g1_AcOhNYnU-unN)s{U8i|5tVE_E(p#Sf*eOPG=U9+)+|JTO)j&!htpgq)+!3w{h?dkuhguU}?ogJK!+EabTHo0%e zHj(-|)RWE%|BvqJA4WS}v%OP?u}4lmJ6LgU-{7`O&kn9Q_3YrcbeLe%$%p@UeZz!z1%gg!|^74DVfaDtvM8sqo0&OW|kczaM_6^<4Pa$@jx4Gam_! zY}+^Z%(Kr9rp|vjG^gde;q&v3h4)}+u`Qp7sGukE`hEv6xzRkaO?4XgL~#5 z3s;;v9&T=3fjyvx<0rG!#0% zZ*cC3rv^jd`TYEs!+Ykx4ZffU$lixn@|-V)%Aw2i4}k`HOG$h=9D4ulaLVQr;W_6n zfUgVTaWfwajSzkrczx(ZIQivsX!mDmw|B4{usw+g3n6gJegKjTJgzBTetAu7^)e(mYqs*e3N6Zj;UmqxJ_**nblIOZdPhjTOEL zAJ8ktJikCan$I)@%-^an>JVZ zB>A86y#e^%Yw*o`zISK)Ue67D@5mUwcf{g*SmUiLS#aBmOS=#@bHLO1s^W)4cc6Ini`qek#*F)Lt>Y0*B2ZtK8gJ=Fm zzc##=_0v3J+Q4JWfsZ`T`Xm3u`Y*nwUwyBH^`E_uUH#|9tp8jZyLvu}T}=UP5$4+P z?u7o~e`Ni`4`>H3{15%=h4 z8T-H!>8dFaSG516a(Oovcay(|7_#g)%T!(*GxU+f_*N)Z^=%-fepy!=V9|Ri&_5=>_>J`KK;QO z)*mim{Y)QtjBKjxfB8GG6@2-9`0*v!>|bHOm_M+Cmk+?M20J!Q7y2MS{iL!ebS!!Tci*=`=3H4 zNw%q8+?@IM;;L_Bi5~VSO8`exh;l(kQ%7f@(_4=_b?t5yb$+euXliQ=vITAJoxTNa zPb}!%*4(zaJyvh%lJzGRL~k0}Tm9ZnwxH7&>|7v3t83`$Z1Hw<)OlN4wsjW#^$F`(0p0~B2&P#BEe!#$zll#ef1tgyy|deo=7ei+>*(~hb>`g#?%Uw?xB3DN zcZHc@8ZL{~mK-RTPHsRW;4ZMC636Vb9D3JGVP#t`(De)~gNku2R_!`O9w9T48(;3F zA@|adJ1s4Y#`*0O}d6RlwO=NvwM6ucIjbsOKY zn^8}5I8F*JxyUlHvtfB?WgGwgsONQ!i(yw z)|Ho*7r9HY|HnCsD0E0dqz~h@nLz%|Abhc;tI_MG(}m_1Z)bCRo4d2Uy`^KG8#CMI z?eMwt7cVH7=Uzvb1%*pkaaVIoqr0QKqtn;wZtd#mbT|4oH@ErRYbz^Ptf^bKw0cc> z<*K@+wQHBIDqUL2=2yBGyXSiY4NdNjR-6Ys;&o$a;+4)<@cD;@*KO=6!0ihij65Na z?$gCRZkatJc7HR+Iku)$X5VawUC{OsyXgBq8lAxdaWZr{o!MpGPL{r06s_B6bVFTd zZd0Aks*HH*lj=5fYnvKj`U)H~)!8)roK$X8vMUw)Y>W@1xRPP34BS?Lw`r-&kM3P? z#45b(F3`YW_|ISV>B7rwS!kEkK~qICT9)Ua!gP$GDaf z0)_ly^lW3ScZU{XzXp;FG?d1&x`n>@B*}5IW7`uW`~WTm6!lKFfDY+Y0exzKrPAR3 zs>?R^%WT%W*B)V)0U)Luoov^hL^~gMLv|`V=4Lfk%J6K};kGqVbn@>ueM8 zqW9PYKlh=6G83<}PIfS0x1$A%4D3&1Lkn4M(01Qp$3zy<_b0J853pIm1lwwlUtUV3HoCKas}L0}~U7Ix=`5iJc{S#_5xY9{Aj# zeIlK8ivEe7PcuP}{-F;WiQ}*Okn^7l`3&^nCr#Q5{GdOg=laX&)QBI{b&}6t6}Az- zNx@05BDZgX!VCH^BfKl%oJ9Plo{1!}!8BI*7w8{#F7I2sKW7?h{(AI>{-55%B4iJ{ z1NI!MWcTkNS}Kt5VjBBe_xK1m2jO#_(2qWTT}@35m-nq~$U9jlUy3-OAM~FaoIIKQ zZ9Lb%sI$|r=t@6jS>KfAg+0+f?BV22PWAvlg#k8k5+7gGdv_#6*oOd+55WI20=5Ww zWMr|xFYVUYWnw@R{;@our%{z@pKyL$0paVm-#{PeUk})rlklVjvTtxEA3ibw{)f`+ zspQXCDE*tDAK){Bxie|qgn#k&xj~p?xJH{y@(pIQvKI~kO9kVfC;A6xUWr7s82ub4 z=`SMXTiT6ff+n`~XA?j0ufo8C4|02HV!Xn@tT|jBYJVyX^K)K=ok4q=PkGZ(pBrHp zWxf8AHZwSH4vjaD_=A0Rbvx$q`F8*GRQ8unHZwSP7UAb|{6h9rH=9eUO}rnA&TtXv z?V!inM)XIW9B)Tm;7fpa1MfEB-KY!vQsAcnpC|CcqWv_~1^(;6X9DjL_}OW^|4h^c zK3UtVNB^9kcWruE)?r5LY9z9o+dmWbjTX=ssXEyk z{(`km%7`I`fHx3K5Y=^XDsU9=~E zv;!~w4gL-N&W!3;*_#+3d%sKirh62rZyL|8{Bx)T`soySIiE-CU!rAu=zk`uTloj+ zpMN4nc#K!}!lS1CdUs^R^j+U((KpR+gTDEE&EWdZkotrFXGHZ~pN;jxWzo0G71MVm z>B|+H-+X?cz0^1Q&j>aZQ{Cq{#Y19GKyXZz1 z;P-bkopwEmMH9wDel}WuNgG?AWPX7B4fAI{@uLdr$F{?MPXa*s0Q`sI@2ljH)c!+~ z|41ypL!Xex65n-0e9w*(&yfR@Nu+-DKxQmn4=vP|2kaEN4e?#q$-X?kBmaRvbXsm1>F@M^rS)rCBD);n`OVOD_*Zgf)W3!nVn2|R3Gq=tKTP`# z2lt=k=>8-tCrW>g_8X35ZZGm5zCUqu`YfUcJ~x=`i24)q+w_1t%Z=V6L{I*-)4|tA zjF(3A@TW|pzvTwA+{X5WJKb)YZ=9a{qs;;92!F)*)IREuAEmM9!W_vWXCi6DUturk zMf)VmcaTrp2A%@1e0@uEAwP3R`-A-Hf$91b&_`UD&%>3e>49v0D%B@r0%3fNmy?A$ z-*2Kn*f*2h#ZU7A@)ia&Go$g$$`A41i}9|{53OGa=tLm=+wAWLXg(R|1J)mtzXNZY z55SxJ9eBfhfPQj=X8(u3Pbd5Gd=D>6iQ?Il-deOj@e^^&%dCUuZ+k&q~bi1k|DM==@JikKxmR zPeR>@cj44SwI_T!>PCFV1T%gD>M=az<^Ia`0ePwJ0tZ4rHq7VTG4gRfxkP+9UulkK zFCl;2$x_L`;ZNh(Tb1Nbsno(>>bHw{@@3Sh7}kqnRCzp@PWDSZjH-y|r0+y*M~v}2 zF_HRL@jM|YW)rK%D99gnbu%!zYcU`sTD=1{W;t-Sehi`FzC+$T1#d1AM zE|BDczx05fN+%|f9_?UOX>j{kWLx49iUv@fh zRDfc`dT|ot?-cg%*MBgM{fgr48~l7v>F1W^!G4{uALMU*eLiinr!_w_w-sLz;QL#(X7w zbiE!kU*YrHlCRKaoQm*AzMoL`hdpurm^m5Phzs-m)MuElkUzxDS7wZnuefL2IA6)) z`HDM!z5;o&+!WuV`AX1`Z;Z}Y5KrA%(R^hX`up_xNEWTHJYJ$dOFn{lY05_&7;lCf z{C`6u(CAzbU&wk^Ap`L1rfi&p&W}U)J^NnDOd} zDP9@Q*Wo|66R)ECZ>(?T{q{n9zA)B)dnXr1txr5Y(~%l|UX(-e{=^=RBSvKT6+e=lK)%18M9{&nB?S~?nBEP%!cxKvfnBy15 zH|;l|Z&N(uc=P_lwBG>U6yJt<{{1QA*=3WzSmPN!Pm0+e{HEhPX<)z95A28i%w+mJ zhOf_O@pOzUk58R;F z<6B{Hg3j;(me?QC{K%Xz5f&2r?`-V9vnJ5zbB6u*G}N=B(eRs(>cce8wV;> zZYSPV-p2kr9jhf66#G@QccC8Lk0M{2U_5UEKErq(MgE^j`KH)^^7)V%-GB1^H~AAv zG~P{~%-pC`zItvU^y$(K`h@+{b&Ebt`;oEp*FWNXznEaoUy&gq zRQJj~4cybfJq_H`z&#Co<{F^GbA;Lfe9M9L{PNl#KKgOw>#y`x{chvm{{1g1#{E^} z$LIe#a-fjS?t7-!RpaiPn{%|UkFkDx&GRE4{8L@z8yuCgI^i$-a(esvQrPLAy!MCR zN8Yuwr8TQ(9MxCXB&@E%ZUiCyXR|4;=hGKQHBEkLk=Id;JvayWD+Ex2YUAuFS9pXj z;(L(D&-IDI6CW>fE#RS;c{qa}VmJ%~`MJu>q@!kskpos@GHW97p2s`AVUU z@-7-%tti$NBgWJ9G^qNGzXHYD@7Ap{yA0#Tvu)HfK^G6NtU&#YmfA4#C~cyFQQL{} zIMY+Bomcqg*OsJR>17NLJK=?fLqWX2A9ZLr!1$`F;~Ooe`qQW@CW#({>Qsl(MEpUW zZu!y&JlHVNziX*Ks+sW~OTEfcr}3zNzC@t^T5+OHAU^&L>*Nu)+UC>J7p0ckZ?%tj zZElA5(|ko8YqXo`X{?=diM=ugaW4~YMSgxmzlzshO%j?~W41TV zOFbz%FWrIxYx{WGw$gmj>qedFabl2_|9JE&50E$ewaaE%yZE<- z4QrQ~{#{G^N0Buf`}0`pRhBxk)$6Za#?@Z<-dOSNM$v#AM|#N*m|dTaDi5g8>S@%o z4Ryj(ohwrtmx>IJb^tbQ2H04w={YXMX3s9tv4+5ml^$=gT><5AvRAk21$F9m2I|xm z>6%8(QT5n9!=}?HMX77;MXo1OZf-QnMaI%iyj+Diy_H3CEY z8zf6qjqvK z;;L~?7W$*aB^o&vn`{X;=6-j=y?y8(XRgLu-EVdN-+nasEJn9-8?VpX=g`{wAG>av zsfnXc3$6UbqqppbO!?5L_R&2jpMwd}6@-=_6f6CmRyXM<&dO(%Cmy}p+|h~)`a61m z$@w>*<$cl4_P*%)HO(jEj!ne+Sk&7+YRBe*2gPFcZf1;nYSzA^>#It!*Jk|eTe|}9 zqr-6QwpAH(1pZlxM|_4wN3xc4bm8MB{DAm=oY?-y`X7BV&LuGPuWP*j9>p7d@66c0 z#49xaGq=^cR_?<20?j8;PTO%{6XzL5+6|v!RG^w$u06{I03X>vpffn}kPkAdDveU! z%=3TZKHF!bn?3Vl&P(5Wabhd`{@maGuGaQfv*yjn`|0$b97szoZ%J8*A-@36|EYr1 zZv0VQUM}oW0iN*upI4VLwy6w%TaMTDp8rD%B3sZk-m`rO{wsPzvEd{r`qetrm3AKk zoFB|{s_Vba;eGIArEE+*>BWc?&*_Twqx?lZ zr%TbIIWOWlU6~3Ty+)tYB_WDFN^qMuG}HMpou$1;)*GT9C<{PrWe|1R1jK9q6WQ)7 z7S&qYGrnto3zb4z~N|4gAwLR_Uo{H1{ zsHMH?+1h@FP>A272;}voZ2zIG)A;mV6Ki|&dvr01{{cBg+=ljatJ*6&9ze$g^AOTd z&RC|H8eg?j_J0D6Z`5AVQG24Zwx@YcDzLaIW8|19Jvrh^?TMyGRHN_slL6jmNfaV) z9+g)bR2Ke+u7)~+zXz`)o~VCQk?lPupvB36qxFUOFSgL(5&pdV+|Y46`}yIR`1?At zpY(Xq?#!QxYr*~EnuQC=Zz4kd-vLPDqnJ<=HwYkgY%G#C3R;WyyW@=iHQ7E_QquTT zEb3!E+1_Ir|EO%Q{FmC3OIzC)VDdf_XZ&w&1Tl)6zcXeXU-?l5+V`Tp$1?taY_I%J zjj!4%_*4MmwSPBGdzuo~_Vi;>h&q0l*`)u=vb|(Kda>$%4ch<5IP$CZmt{-1DYw6M zeDVp>>85%~Xt`v{0o7HtC-CjI<567BPs?!8`KsovaXjKx%?oQgRrg2-F}C9%IiD1r zx~lo5uBzRZqBZL}ik23MQnIpmQA=0D;^LMKi`Ud_EoxcoUEkQXp|(1|p>=)Dw!+oz z-lCF~CF_ct8d{fiZmg|tY4ERE+fu%LS;L|xz}D0dzO-e@>J7COP4(;7g~Twl2%BUoP4fw=8L?uPx5=ZRGu~ zs9m(ItG;zf9(eHAFM7CUYw@bBjkVPs8){cQxuSJdeq&ipSL5=Q?HHpCe3Y##Ti4aF zXh}DX*SNl-y|{dPY5n4Lc`apIDx%|-ZCT|lTGGC>Y|C0nQ?{XPU2E;)bz5(WZc+YN zbjvoj2Aei6u7o~X%Sh(U8%cNh?WK#hto9bK%HPnsW$D(U>h2Be3j=Y-EZ<(c0esh& zEqk(|yRaGZwm?@;)|V9K`D%;(^{qbGXI({o+v+D*k`0!xYxWkcSy|k+DzBk+T~qzW z>h_Wi-kKVJ^^%s7ruNeHTdKUpB`cSet;*k6wzO+QYfIP0GQ1YA?Ao}z!r$1st{eI* zU!A}1$;RasEgRM^g1xKrH?*y(Y1%04wY9inGy2YJtgYy>+QnPqe{w@}!DGtb)U~d) zr8!Wyt)Os;FK=-{L%whG=7I)qeM4crx3Te&`o@CBhWv-UzB)nI(7f5(;Je;WtgPj~ ztwO)5f?qqiol5=YQnT?O7yN#tG9a$GhWmNAH|{Z?rUjSE+SL0! z&+CWm|0(f5JHCVCQN$hdFmFWP0D=zgehxSmw^ z7P@JGpH(^~;L^pKAz4vn_a{FE!w!zlyZN`RX_t;6<_0 z5?#wp(5ddN*O$=N&OT5_5+<~KaGVJyFNV z^IG0D8pSE`o--}zc|;u5!e$s(Aq9PmA2VF;gcJ~0L2ul73uHSk8HSykl8eG5!Il3D zeKGcJ$faPS2k}l{Dkpl61S-D|L!t5LS6&DH2egkzPy3sA@JFr#{~_SS&uMN4#?-vh zuEYNeNW6%?N}AveDF=-iZ``+}yl!a%HNTocPyJk$aMgB^gsV+zsq8;T_ERNIru6)G zl7Bp1iWfDno|k;SD&cBAy&?M`Xy`aFz7 z<%GDfcW@>66$!UXxRP%I`cu9`sEHCk$pX)l@F{?+wgmdJgii(BxMkdFfE!cAqY_Tz z;xsW@j>$V3*9b3_a#H__HzHjj;nsd?C49Q2AEhtq$9PKAW9XlrbE3*&1Kdb|>^kr( z2Dnk)T&Z^}pR*<0D(`?akX7C{CEVK2&n4U{@2?E~82SI$05{5;Dd!K3YmAY3*MT3B zaN@_fnjMz#hb-^`1HI9{Z%DY6e|m}z72;=#xaIE!60XKozStz;YFq_BWS}>WdsxD) zdQ3c}wFO=Zj0iN6i-rr+ySKB)}#ETpsaoxegiN(Le41ra%s+|CwqOOg89&g2pbt_Aoy)C|m&Xw(rT`d^?CU<|_1}*L0#=6e-x<<~J5D4rE zK1Ef?Vw7-Mdq5G~>R2qd@g{8M4Q%PC>+t&;M*ps8puN%C>17M}yKqQhr4^8*l%oX6 zf`*RH{D*H^8+XWF_~wlp_XksNX9I6)^XYfrp1UeP4%Z2cX)9;2;W7}&oS{T%pCcuot(KmBI7?84>n zj`wK1KMsZFW4yJ8(9g;6+?H>Lhk-A@a5_Ba)CKTP{rpjA_`soor{+IB5ITJ_d=amv z_udT8J#{Kv=6o-FasCC+EDz0Td19a!x+uGF0)3whhfbajmpjjdL-Q|(pIS>iFAIfE z9vql^>WP66c)mFQ+u{B5-vnRK17z>TD|yZrLS@k9#fLxxy`?06I~;oV&2Y-LQ{g!m zE`hI0;c+t`2@Ml|C3xL=Djd3S0qy<-?G6l-0k%K!RCxG7z_jeEKS}&*)$pa40?B)` zzn`4gU6uT5c987ov9SUNY^3SkRdw*GNt$QmoNZDs;kN3mAZmZ$g#9PMzl0BL)mXvn z@BvLD+@#(?-SH^3JH}YStAOS3HlnW|!cRV^C=PwNsR(}O8M(Imwc|$rjpvUa*tEII zC&~Ym@Abp?UV(4k^SwLU_quQ3d&kD`y<-;NBcD|M*KhGZ<*zsL!E3vJ7>X26erxCf z{m8|CDjIRyS=yyv6^#tPw4(3V>sena=Eu2NY-H$%`pE0>>%nX`a;|vNk->WH$hp7O zuMHhweKe1l*8j*V;3LnnzQ{kbzRR!ZBkvTmzVr97k$+jt`Yxogk&8)eBn7mEm}^7( z68eVziS-RVpdGpNKlPDI@3137f0Gaz`W8EKsUB&&A)iq$ zmdvhsAkW3WUC}oT`O2hR!{B)sJP(6sl9A-Q1U|>1e=lTm{+$+j7jg{0k`Nk(9PdI7 zqx>Yp@GFU-VaNkr^j)05j$DRLhF?kwF|9kUTn^?LnS=3@HP%P##>W*u3=!-%h*@LC zKJY}kYD~lx?LV#}Kh6Js}#O=x1o2K0ZcD{4uVjO~3kag~ol%mFXEtndb?m zE-4H-zfcq++mKDjPSoE!(Cc~V&xQCINcQyYME~E!bsb*ixbMdGAg(9z`ro)egX_Q1 z_^{VI&nARyu&eSv+jute4)pJusfS#!&n5UR+3DA?0onW_Y<_Mr>l=jq$PUV<-(S!A z!o{qQ>HUw8O_lwxd<(XMFTV>veh)VLH`p)c5A4X5!?3Htj*ZiWKFCi$t|$y$ODeib z_PeJKvp$}K^T3b!i{5UvUiohIKWNOwuGFIw6rE`r1^A$>6KoY;4s^Z~R}WrEc%?HJ z4Lhpr^aWad&F!obm|Vi@n0b0CuDZr-<87L5W6VAdOP(w?C$r=B%ObBWR0y5@FCmj8 z+gc}X&i!X`)pxQ)4_n3(z|j<pMG|z3uI_-sa}*-i}6Jptj!I z-cc*D9lqA~wt&AQ(6o5b_7-1@x4CJ1YyQ%<{MvtC3_`r==m_^*8cK=MS`Xv~_g((VTE?t?eD&){eZpzv`84Ft-^Zv$dObE+3);4AYxi~JvD(_!$7-XP zH7nNC3QTPtwrWk@b{@T>cxs9K<*v%fZWNj30BOT+*;yh*z<7GGC?qg|DxCFDWOL{f5qW>=7Hk@Br zO9UlX0_mO#_})X;wA8)b-Lsh^vbTG-gWobn@OjowWL@!W|2hDot^F(=f>0H&qj*t$ z?Z&c_vO;$;_Ww9X5rqy(h;%7l+X&?E2*MYOI~%-iI$daL_I5P2wYodn+M3(vxiPbS z-gcimfANAP^V}QhvY=o&E9z`&Zg97EwRiYh+%28$9qtC-wx(8}dqYLV>h-l7S5&Pp zt5{pRqGrR2wIwS`*!&9jV)uM+puW-F-h%UhhrMnLO}x_i3O@g^@Vb*d0l0mkgOMlX z(S5qO$1SsG#O`nAILFqQ%Iw?hunXEgY!`jsN24=%AWnv^q%*s$+sV>b3Zr!!jc%yx z%x$XES(On_eNx?qZf#Q|Oka&-rdpe3pOeaMN_M4UpN;Wh6jw5Am4VxG@HQ=#`O&=# zj#!D8-31yL4FCDdK3#a3Eo%d_r#KjScef4qOC@|Kj<#Cx+71K?jYHRu<@K7peuQfY zAyCLKM$b0JdUk6O_A4OCK!a&4t4rvMPm&xbJFzP2#eE10B5=})Dx^uWY~iPZnV;UspR=ozO^B6{F+ zgZ7DZ)+zcYdOpnrJ^F_}Y$T4q@&nF)F61-NgP$~MFYtr@h@R^&qeCNpP}fO5e`VN4 z{3Zn_!HV3z2?{Uh!;J8*fO8V@n|dyi#0Jt>!Jnaj)VaKG@cx`>sQK&AANqfCKZ}q( z><-v-u!7ycn`o&(zRPLsD_!Fw+#H0@bwWS-_>I-o)m+{;vLWweoqQ?cfPT<_ZgBEs z^0)C^|Dw*$zN9PtlxBThniuv&|FDOXJ2}|{{1gP(#7TU7QSaHE5MdtxKt2Hf%Lv#a z7e5V^@d)P58&EfSyKGrhUx$aRr30+kOpwpnpAJV@|@863D*6nSA(2KlmR^ zv!{|jW1;kKg?@m~2_jf05`PoOv}8(PH#- zoTR^qly5~BmI<2J(w|NIz`qIt4?f84rHSzh0<-3Dd8qxFG|bO=5q1vkX+GsmM}2OD zU6%E__q3V8d2?vIdBh*=v$xAJkI%RJr>C;Nbh4Sjxw8mAm*W?*C%V{NQf=b>SagO8 zL2m~=);6L)?&Nqo>H=R3yc>A83GYT-;8y@Y4fs5P9}?}Sp)T-W1wIpakHF7P21B|4dJy@uqYCIZXDP9`&z5 zr&bV{mJsze-Ea z;2fIJ`jzu1kl$wTH4WaemYE0*XI=8%##)H_acN$4i~?_;SOaDgIst9pdLps!pT!quxn3 zvH*Xmi|MrMNi3W&9`du%@=My-`XuuMtR4egC-s+t;i7nZMsOVJMm$YlBR&CjBR&!7mJy$XdJJz_ z&-nUgcgc7%rhiv3tk>K>{99>%0{_@glaq?_56sVVsFHnQf8_g_<|Ap z&(DRNp4*e=1oT-DoTk%q%SeA`|AN-9X^HGggy%Pd)8SvqnNj~5T!{TZPA0@h0sRo| zHyqr5lB4^RtehzQ1=?>olDWOefB62y&FQm<9{Ai~vLotG$ZyjF?kqQYj}Sfi(;f$3 zA2D7U(ZioIjsBJ!%yJvs7wmDnX})oK?vFMHtRwsp<5T;nKYp0To(Xd#i=2z35q||e zpcn0vDBnRoZR>vmyz=!e&4v8T9qkYDrw69%Q$QbaVLlI4q^1Y5^{G^!j0uGCFU_V6{$Srsau+|%2gq9x%*>3&Gb=yDe-FmHK0ma6A)ph1@NcufAEx9v= zlYY-HL>=h@bx87Dr2U&CA*NsQkK_c4eluyQ^4E!Y0DhqjH9spbzY|c0!lUy)F+GM) z13n3LBi@Bm57nOV>8Km=857L-38=^LkeB-_*9YXKx(gf#{n#*{bH~WX`Q#Guj0LqheSuhEV14U^>|^^*K~UJSTl8Vmo4t z=ZT5bzl!GxK{1dcco%;(?ENC%m4(I6nk<9rI<6 z14jiYHmnz?G5#K54}abJK-mWMRF=aw6hA^k-Vnu|AvDZ`8A}UPtNC9_vwTzcr|t*K5pI z!bjKZG4mBZzb*L+ZN{kxf8_fKWq;Tc=Z~3_fsME@-_LxC`3m_%+Pso{wbF`pV-a`m^LCh?k~(#DVc< zxWVry$Valof=}^zRDN{l<|E`k{~_~{d8YU^djCz?1&Z8%9}F@(L3sXw^ZU}ym&c4( z$4v3caJ~-zxt(|w-G5_!Gw-(-;`4>E_S<{7IBI?3@tKa)=<}i+iYNE+`966fq#R6h znc@@HPt*Q-8^xzYzCY*j4Ea&|9y`yUupdZcuOolmlaNU9JZWN@gYuz*pq;{X0?&7` z{)zLVW1CpKc!qe9{_nj1HpMUGQ+j}Z`Vjs6f$-6I2L96#&uBjs$rSnBt;aLde#0EU zFurNO0ezd|8ONLVAEx~V@TT}S#Pjb@7|*Vl{KXp2@Oe_q{@^zq=SlsCq<&yO>}Mv^ z=P`VJK9A2EcTbG%&mZLbbAEox^X)elf_}1LKYkdWH`pih{W$lJ|3drm*m)1e!~Q9D zeueda^nQHmv>V5_Dcs+u#*c4^A5*6C{g>L0kB)Dq^Pk+{)G3zu7CZm3?r$-FZ(_ea z#TefTf)jLx53t1kkmg6`go&__*nekZ|D82~KA$t}zo*fD(Gc&Pv>}P@zn#u8_TM;A zp>jL%uHrWK-|1K_!Jyc$qP+|C=zbLW+63cy6Yv?v^Clo$}QS3!zV!ZqO&}pRQZ`#lG|L5JDyyo+RH1jovo-|H{-azt~y~|HFhHi=|7!KaXp{DII3y#ON+dYYV5%|z+WYR5>y*!Pr1q? zbRiGavATx8SqAUpoGHJ<_Li}I+HwSPt&V{{R`eWaqQQ+-c(y(Vj8@ zO`FI2Q{%mwU`M?uwJ4EZg{dC9R=Q=f;04rZq2M&eRaOK@;ixexspp4?CqBm2SHTGN zMK95g^ES@P8=A2geqV|ePw9@n%(>_)+ltSV@IBQOB#1p)B3O==c=)JdUGdvfF3*cQ zJ|4-xC}1F7|2)?x3Qv5z$hClnV&b)r^1n5vdu4-o^ig^IjIGnOWF4B0!Z?KNNk#N* z&}ov#;t!mP+mdXuZ z^i+g48ncd838+DjqwvHH>0q6%uXzx2&&@rkhc;(Dn#TsLDn@<;tguc`(Q_P$zvC-~ zHp;tbY_+0TSBw}>*VCZtH~tC~Yrjvo%Iq?X8_%{;&jejOys`rIGg@lH$fLB021acs z#^X#+t#V%Fn_pX!cD08wJnV!Q8V&{V0)Nz@;Q-^ys*Z28nCefWu9zfx460KdMicP| zb-LwCAMjwqME|y>zDzUYJ(hZVy_zI6wZ?33 znwNS~bY8jz1J?HOv~8vNyw{C7)8oV-EC2E6RURO9>_=K^Uc1coCxKUH5o?#tvUc%r z3meuhGyU6^_REkp8~gKE>Xnu{veoOaUB=a3_}*CY?MBgn97lS|_nBRvjw%nR(CS&# zvki5^Q=KbQ8<&aaI#mo>IHS`bq4Cx z73rEr%~AE(KEtNdD21sT?1iq!Qf_WE%0|47E z@1w(T?6y@Ia|HfbiAQ{fMMtuhb9CY3CVap6ew^6;$NC?AGR`G1^sj5Y|8B(_eecZJ zzr-sv|1-CBx>n}G`2x)+QO??NU=!yVM%oRZVYEaww_Lkd2>?E_fk5YQ;vpYoR9OplO66hyY5YrJRs5d4?)hGN4>Q1q*Hs4MM0 z0ysaI=Tz5!o5TCyM?)r}qq&E zdQO+3MRQ)nbGkAWIC_mfr%OT(NFkF3{6KTsBc*vcU4vQT2 zmn^Eaw#OesTF$sczpLnvG*KL~0j+m5mSRU~i0onQliF3v3ExN+L$T75U&znI#gZy% zzc1ULFtnvxtKP|fsJ+L+MuRlD8#pTDdBnT5eHkI~^-OVzy?;fv&y^shV{3cb&pi>R z{c%fs)w8wz451LeM-j;DY1#e*S*P*oyC&B58dMhkhpvV?fxio{Bc7;#Q<3dGCZNU1fTQ(=_%E{1;Sv74{LIjCJp1|KnE3lT zvY+&L(eB(Ii|dm6#Wf2TlHWvx`hNnD#z!%sCTeyH$Z4|Tt?f1nQ|0}Y6uB4># zsaVvduB$&ATv0EBGkgSh}&Z ze$nzS8n0nfd0SE0u9CXN8}ph=x0gr9E#1D>Te!S!Md|hplBRTX>&BLv#T$3r6y2iy zvFKKAX$dxNSzG~qw3L#}+qRJI^4m%lZC~dtTARPQW&4U9g;iafHx&fpj#;*=W;6J% zD_!|`eOEyfYai;CB*EM1$wrF2E-=9cEpEv0xZUemc{Rk^>RWn&lg zSGF#HO*HBa5X|;>D*#G$ErX`Oke^b}m zmgc5F?an1j8@AOI)GuA;D|mRDcd>VgcUj)@g1m+$1>StGchS;XL08|j&0Fug-cPJ- z;J>Xxzp8>?JGq@o-VL5jp1*j>k|m4t@|JO$mEPudU#)mAuEr0__`EH7!92!x=iJec zPtN%x_wFx!3C77^kk3t|7Gv%4DyHq3u1!nL#)Dk&`;p3kxaJz}=i%PC$9$3&Tq+w< z@Ao{bAGQB^;(v2|E%h7D7evoQ-@pmP+nzD#X{@6L`q$L`D(A6r-_T#M%R15hK;dvb zt?n&!(*QrOaJZJJ`x2+?S^XP!!t+1;m2m2p7Xv=m5R7J9@I+r?z(;=-X|?m^aWudS zVx=Xz=9{2X-CM6OqOG01uZ|>4X!+nc6HLgVlpRA63!3=qfdDp$OSPTgJo0dWiVswY z_q^u~ylXUyQ{p{mTF~=|II4xsFs?!h`WQcExZDXTAg+SmxbqgsHe50cJ2xd4g-L=d z{}=jV?3<8F!9)+@oxW5~^d1RRejkQH@QkqQZ z`R^tFc)AoXYF<4n`FvTz)qHwQ_R}v7sNkCbPgjGYr*)`P!Bc!We5CL!aba)aO7N=^ZkKQ+-vsohe1}jIC4Q0xo+sf`09S1Z^hF7u3b=90xW@oDri#ZU zoW{jzVziu)cQmdMULxhB{uOUTx>~}m{nSYKbW1-!YTP_I?rH|rIhX{u2E5N53e!SD6a~x;>L3pxQY+`7PyK7Ll(GNde!-->R+w*v%wz~1@D&j zTV;W(_1*Yh*XO z^PrAun_8RjGmmxc?YGvfwX;Pq)Z}YtV{dZg)`9=L_Y!PMK?z?<59^4+|ckL~KWTkp_cb`=ZkCw$+BAR<#O zQ%_fAf;v^^9||d#X;o#k&nK8n3!-q3ygMu7m_n#{rRuYALn)U>0p!)cOWZss^D+8t zjpCgQ$I%?@?*XLojkz_objjfBp^BlX@zwQN*`E3~mP!@88;sD{_zFsB;>oJR%2q+4 z_*L}Q`hS<>U$6Z~7KGwk^_LFetu`Ut>VTp!U#+w|WcyL_%l6M$8d&-F0UytP4(UHv lq1(=!l#xKcaEC&XzCb8|F{5t-AgU`r1#!5@)OLu>} z_q}X3^zJ5i?>lQfy_Y=f#J_pi>63@q38eiqHk!@-MoFaa-A4A4U%J_eNjb4~uXx6u zKJOVrW&IfovAI8UNBa7a&+YR3q!o2n4zSCc{zua^}YY}=I39& z{q@hkoHJuy@7^;{44lsW`oQuRe>8CMw=WFzKJ>eR(@*>k>E`zKKK$^@x1V|FDa{SN7e4>c%cqg%f$FahR4)DMK-QBl3Z<v$f#~8cL6&ZTPbKPl@wYk|`dptlU#=~_OZ(>Zex!Bz2FeY3sl3jQD zyxYH$@||RC?)Px5l;;s;J&kem&f3P_cQ-X)97cvN-Tmw-c|4~YpYM2FmDU)g@o0_3 z2*%>yW2}BM7XP@$Vr+!5crwjcJgJRE8l%>DJg1FEYy6HhCNE6xJ8+?K)=w_o?dtvD zkh}L%BRg?nQ-$~aJ)YjdRxieJ2JEP>ck!?9>iIp!_Niy;`}~n`ao^v?`q-<3Q@*_~ zGW5+GBSRHlmOqXiU-yp3clzHvUT}w{DBz;^)u82FpOUjW_)`~%?Az{i0X0Y44==fH=7rvm>y z@Wa46fb)RA4jczw3H+wJ@6?xpqrf);{}lKyfOi8=2L2}SBfvKUj|ToK@aKTn0DFMH z2z(FlQsCc5`Z_)fd?)ZC;8%eU0)GwXZ8)#T^-5g-3*Z&ND}hUaKM!0D+z1>3zPq8X z<3mp$+B4MgKJfK8zl-y1oCk28iSwH{PsjNp&eL#y9p}k7pU1fb=U?DF0q1|gc^u9^ z#d$Q&&*Pkr^LKI1#`$k?&cykvID2sZ63z*<@51>loDbss zTb#G!{2QFN;k+LDxC8x|j&l^}AEIAtaNYrVG32c{kH`5C&b6+g#Buf$1GcfvU43Gw zd)A3v=W9;v{(jAgJzuIh(SYlT39Ro|@ZpKktS{kVeS_JokL)V)ooIjk`wQF7e}7@S zJ9wu3O?W?Txd?)N@d z5)0k+&5WV8U&I%(Wv9HS;P=C?oSee8KK<*qzsvB=+K_Xm{k!=5E&7~&ru`rB`TO)aaHjo5 ze14ujXYqFS=fy&I-kLFVY+J_A-ZdGQs9o%(lT+@z8SPwWwW%C!szjSA(WXkYsnXUN zwS{y=Z6W=E=iHC+_Iws~#i=g;nf8bA`9b=ed8Yk~_}oLEeP`PL8$SOzea;Z=DU9{p z^{#hl@0(t|S$EcBpIf&Qx_=&jdmr>(>gw?h zG1s|;d)dio$NAuieYIJAx6k+X-Q9q{9&g{?hrL4&AZ^DN(U-WludN}zFcNv%+qTqw zsWppzs-3Y77Z5jQA!ghgZ|YlYUaLSJFI}U0Q08)+16i?k<&#eTMYfCem1HsDgYWh6 zc<_S9{gVpV!^JL)Q+Lam<%|u5GuWAS_u4bz3>VI8a7Lfpt8w<6QPZzFv)pT?UumXa zVWwZ6BK@*6%l%gRW;6X#GyRej>6^|h&#}@sn&}(N^z~+XcA~PL{iGG`-&ftxw<5@f zrXyb37pfncju`3!?7S^s0r?nHUYbI_4e|n0UYRVv!BujW%3D#yhU_xO*AJ}-BadEJ zF|L=TxV{E;l=JJnEW4b8xL#$}U7K9@zRyBlW6Bq$kbe>KIi`GR3i($duQTOqQpmpv z`2tgZa|(GB@(ARn4(xXS6xW-~e7jTRdlT|yraYQLeh1{MQk0w5*cWgG2l>5>r^q)I z@_SAB!ztv8DE}yt|Ibs%w^4pmemaHx4$2=A`M;S${xIb?<=;;s|0?AlE%LvZLcXnj zC~z<2`ncJS>#i}$*AG(q6xZ*f^w%Y){|u#1as5F`KQ=l27b$&;>tCYuVmC~oU z{jRX2VsiR-DSe9T?^F8X71M*1AYz`4B_ z5byOPCjHGltnc+3+0gG1cfONd-}i5bL*G3f8T$1h#0Fo;=zS0I>EItCeM5*>f0Z4m zcpvfW2XC$Ez4Yb!zW!-h6&HRS8R|#O_Tj%qhJN*GWa#`4>-%0s%=Ox<^?koaEcWa7 z7(TmjjSap2`})4ufA8u$|3g>bZxFY=JpuJ6*wFvkQQtT4dCJd*E+W4B?dXQS5C4@7 zy@A;8cat0XE`8b6_a@@Rw;CJzK6opy_gC2~Dqj7&o7DkF%kFJM8Lvchjogcd}i5 zzyEtS^v>GVy>B(TQI89Cy8GV9ccC5bzKfZzp@Gj+8(n?<*TZ*yAuDuwa!TUKb zJbQ}cg27W97o2^L<8g!i=+~)f)fN7-rrrRqmDBa<^LqVdOM3l~O(*=!b!Yu$OS#<7 z^N-xVg6y=_`xIbB>Eofnuq*PNEL^9&y~jrl zo@yM0@|N-PP9F~r25=qqVtx`vJC@t^^xx^G>q8e8W9(e-8Y)M+{ypwKf7uG&w*DPL z-apfnZ6eu9Cx5e*tn>WWQ<_zFnguBTMk;q2U0>aM5#w|`t`*{19nRO|>@Qo> zI~Jek;?G~Uws#E9)A;9>ULVd=_~)B?y*N+ipV#%eamJWI`t`jueroa8F?B=lD4c8f z=Z(GjI9Kz}H}~e^T*W_c>dnTP^w=?VGxc+*0)Maka6<3-&yDYW?UBOX*B>qDedn*n z^`1PwZSYH(T@_!=|8(zHJX;3;y0P`_-#oMB?1vNFeZT!$N$>CeuDJKDf0)?&f1WAo zefMW$d*1`E2f@>!^{&46|Ar0yq0!a%!C!%=;C=rz*o|zQXA^(xhx#8S{xlP z>VE{d8F(}>=C%Ew1KtGe0mi(y{~q8Sz`tj>M>zi$XUvcL599is!2b%o7`!^9@ceek_gx3xBuO+gLhOP*|+!L?n4gVan!_j zRNEhR96He1euPyY*<~E57F}!IarDUk_QQu;56o^oV6^Vpb>NOxQ?g;#!NZ55tJ^yd z-?{g$yY}z0@p^#|&#JB4-a5N==aFNbyLRwG%)+*}AKulv^YDHM_8#1`aqqz+b*)DZ zw72d)6g|*>#4e|){Yd*#eyQP>4NF>UTelxFT6gX}Kn1tbpPf=`L2C{ADf;zC(2c_# z?a^I3|9CnQ-F0|Z^bYh0cPYAS{~yI@LqFR8C_1up*Y=}(T6ag=59|_mN59Zsty}@k zcC_!`zbD$>*}+NI89j955HX&k9fuAcKGJ^hNX;L?@ZHhgd0B$c{a`4%W}Ylze78A?QBJ;-5T9>DJ6pFOYvtFOT==ZZEBq)Lf+E|Gpd3_m z^uWQxdS#!4PXE`p=CP-MbI(I#R=0TMym=hKTTg z{puyFTZIIqYu~@O{cwr|7X3Qgqeu20*?Z_9k4o$^sC2|-e^jyNNDj9GLJw8}^y3D8 z-KSq>Tlw|er0f5~P37b#UH<{DyIC#^KQBZzW{NV8xZYx2H&;V)&=e0}(NK$8z^@0? zi$I%kJrs1>6Z@e^G8RprgpUyc?Gk6=diZ3%eccqZ7M|X@9ySF>%yO&(Xsv~g2XNhk zbszmm;Na^v5YT5j(4Q_k=jYrTiFz#5M3-M=$@_}JkiyAFg896fv_ymQy?y$5%N zH>_Q|YJKa?OV+JlzIILPlFb{ItZ7=(#Ad7w&koOMkM8IQA3o4JtLFOlFsi1u(z+X- zL1=B=$3Bbe-kE-eJyYx+`aH_dFm(7qpDpImXG%kdH{^W2oaI07ZZF0n@?Gy0^^Kzz zF;>go@vyf?F|Ubdv9}dmLoI%Lk}aRfUQuv0`?-Q=u$L5E#hz2}baqa`mF%p7E7(&C zE@!6{JdK@H@KpAgf~T-<1(&h=6+D^66kN&-1y5p~3J$Xl1(&ck1sAi;3ZBSX6kNoX zDR=^lD0n=p0j{_8Tgbu+E?_|gk0X;Jr60@QcFXuW_KJeXu$L4(nmq^X-0u)OtH?*O zQwk2UlS=x0)~(3%SWLmWtW&``tWCk$tVO{AwhY*5PZo;+JLApI>J*&GYLxUo76x|o zm%)MxhAmdGhs9j%ZI{~`@30jLcCi?f<(wpxeyx{t(p8TefQ5J0O*l_TF!w4C^R#+g z-dV8o{}24ecNlBKUpM}|6=B=HlWk0Oc+11Y2o?m%-^L&Lm@vvY_YaKKd>dtb7adp; zW8ND&edq~GP=m-Zx|*n`9JiPln&8-PVrYuv%_feRIB4P)6StYT&BO&J?lf_SiDM=X zo4DJ=ohCkM;&Kx|Wn#m`=S*B<;+ISuGw~}Xt~2r5CcfXq82P+E5fg(M9Cw>IY~p1m z2D7;QF%w5jeA2`%CO&22HWNQ(;_k5`e+zKH+TSt;axjsvW-THqkZd8`w029;O9agcSw(Gm`G9AcgG z1unPnB7tix+#qnBg_jB(vG59kmzg-WMqtsta`qB9NcD;Km9xl3fkpeu*;AVZ7VRr% z;VlA-_LZ~lR)IzP${E`suxMX7>)b7{XkR&dWv{@ZedVm>fWV@CXiulWqJ1G2J|eJa zpPzN#DX?fCqJg^w7VY!18t|U_a}xb^f!{4AJ_YQf`~UYsjwBp+!!}U*7Oo$RPZMu8 zaqwv&Z!>X=i91Z(ZQ@QVzljYi|2dIAX5}~W{Z@VxcU$>Q{Fs&hdusbyOl-BU+r(D; zUNW)OzVP=&e#Xo5vzUqf90%AdCKme4W07Y>`T{P8=Q44a;}B!d3VFGS+e}AsShNp(H?e4+pS^8j(LO)()Qw}_ zqA-j3;LoNo_6q*mJ_CD&zmxcT1wN`1W6m2yEa~F^(}{VDz4#aYv2f7i4ft6eKF2)x zJkPwo509%4;BOD5MjLU^`9t%3$vl5xp67^U%#Acz__O8$xLy3Pu3K}GChfWj?WgHh zdh7bhVbWXjYjuv_?Mkp;qBKAL&gZkjW7HBHlkiua$DTR1ltXquPTTWX+(1u+${2ew zAM#N`j{CsAl*dj)o0d{v6XzbzXV1-K-G;ZR1!FVuN+O@i?=~`;PD!}X2qiL*-aQry zJzv5eH}Vst+EHU+{GVIPa-u9mH=?s;z=QnW$BnG~Oe)`p7NfizgN3NYoh_I%{4}36 z7d{0I=@zH?d@)z<+Zcgq=?t$Fri&=7!UrcH!|C`ucs+HILK$j@qY;W8M@bSto zo}bC$T|Q=x9h6@J{iFAEujqJ<^cg_?L6-Ox^!r>s^p_pThRD*-Mcy;^Wr_MxdING zA7VV~<5kK3bBzB7Rw?P@hToa{F6C_{sr}^j@k)Oq~*&Cuh%Un?TThYIrl%K}?yP`av zf7tTg@RRLLB#!g)3Zov<9gSydpT98bCG#OL$}ck*3lnA!XHxqgI)?S7F?`J>oAtXm z_98|Gi#Wdj7?PeX^E6TYuesPVOaCSxb$!n|B2*>xMSN8CxeE2iP(MFjC;j;x{k@#e z?$1Jfh_TUpKYQK9?Nbaf8<3yut8zclpKc?kiKfq%{d~wkn1$q>;kUEd%QIQ?QNN$x zFVZ*c-4E>cqdeqqK4xiOj32%H=A$`k{!fdh^A;ihBcva3xKouk5%Hop&=M75kw%JAb@uq9!lm1h>kMcTI5Q&*rnUN6GQvh|zeM4E>Gw$1o(E z^o34(KHduW;+`WPr^(0qV_=*kzfO~n;R}V9oa*NelGpEnKO3#35BXytgOwiRd(=!c zw=;tAjrO?V|E5y=xqTs=DfSoI7;a0EXR)w^seWz|&HCM40ah%@-55W7eA?yZU|%5) z8|N|_j}Fdbn1V|VZ5~Lg|IP+Xw|?l%L0c6wi#t9#VdAzIGd9f)0ORPrI(u zg>q~v$TztCUI zmW%vp^??bP+@2(T*z#fY0RtuHPtu1iA4VStxUKx%Mlc9ZY~uWs{B`sp`0w;j@ZY)L zg0D`0JnH>Kyh7u}X|E{H8IPH2Jc045&L@Bku(5$|5RUVY;Qruo5yDf^ACixj68?&@bSbwVrhkujH7ow$QKP{dLk_F{uNe9H zHNcs6`aCNjG3(D|OCZM~urXa~FY52ARN9C58|ty*ZyIl-n*3IOp`UL+pG^^z)-B4z z{cK{uR(raQ!ls~=9`QT)*2K=+@tLdJ7?;oOr8(c#?OD8QEMI$Y+FPNt7xg#fNcE$= zA-57L+9 zyU|#uzKN_57>B9Q%zhDGXu0OITlbQ-_ebIP;aj?f5=^ycs-9EzMCU#D)PmRaA zjj|^8y4=2L`7|N0+DG~uW7qGY`Bo*P8)mn63SZiA@KmM!-A1LK(S*a+?{sB+b{jLA z%BB8v8_SwlztbM-e}P>;^=D;1+=1P`c-IQ8zG{C=s;}Fq!gzB@>1QhSt9&GXsq&Bd zUu4(sqW+ij#c+H4R5XQ^_Ln!&Dw);aZevzcjUumWVi%qM#k-a>j95BMzTp9ceLu-Q zjMc{9jrlY|v&xTm-N0t>?AG|1ujqsH7mgqSA3x+TYW;i>&b%MpMh(WV)82U3Y(+oa z#+)YhhST0|s7EH zB%h$ZN;Z1_E_;J$JGDw2nw+6&%!2Jez4jP|Jp>4R|O{fQ<6{dy(M>d zc?Cc1`ZBEem6y-?^`@17Ch|`f>uIKbY5Xjc>gV={2E28@yGj*(bsLNPG0?@TzuQ=V z@oSY&_II&%e_($G9C^1fAs_D9Za?+EN~)jYxrLD9FzwBFrGCa_$cr=Gd6q6Cb1XBs2ZvMrH!xuD!n2ZQRflvF{h% zpZR{im~4+<@CgB{rLTC`jf#F~es=@P!@=9%ZOm<=>O^|FU-SGiT9mfS!+1X@^Wl1} zyh&`XRo+X&zd8L8{d4%n?ZsF|9gy_l=u4z`>v}}jaRxFl=1wiG2R=q@;}e*4`L@rzogG5YBK2)>qo%jBNoQ`GuPi{ zQ-5E{gFe3D=;LAveZYTEdvS>V7zkc%drtZpFQoJ@Uylwb_C0Rk0kWbmH6EdO2ql{O zNS@#6@zA42u{UO=$ND<>iIsb`|3PXJ^&jys`d>W8!dCy=8<`t@eAGm0&1eu|FK0D zw))>8_kY6`_5b;3DY;7=d7PhdL)OP)^iS(Q@k!B-9&{=gol;=eZjpO03X=1ue;>#bV+r{907+&s=>d?bwrEq?8) z(Bi*x5Sbr*Jaik=^ZD}#)4$RDjr4_s%N70Lc^9`2QhY-EsIv5d^?u;7w!SR=#AN+^ z`wI0_OW_I*Nk3A&naq!L`cdQ8IP{~&r^))E`MtxZ%g~P=-=x#e{jz@UyF&fUGu?@# zA3NTJKS{D5>GY$=GZjue1N%zzU48t}`r;MohXO3gzRXFdpKe({tFBN#^T{3Jko056 z^Zy?G=<#EP5H=!ez|T;hW8 zN6PvJ`CC1{lgAU{jU;=d&PS5QBlsuJH^?55hr29&VLW0!kZI2c@Vu1Ur)8@C7Fhbj zeu9+tsYTY`Kl}eH@Bii6@1^8!aaj73^>I1=Ta9OF{Fpby^w$+i{7U+v^~%fAkBDbs zA5?ubS^5z1Y_h%HEbHUs73w1dMbJU|Apd0chAwgYq5CsZpR-=6Kd;i)^C-T(4mq6u ztLuA8Ja5HwezpI!RY{8H#*eYE<*(Z0{&!ws|HqR%#XntnL_SJwj{HGEV)Y@79ULa zM3?CPll@Wdw;sRdtgKpk^0BSQ;v8(N&ZB{Cq8@q?KLs~4pS5O{Eg#a zj)l?x)%NpyJaxKQ!uL0k{p)Ob+r#j@yg7@MLkG$8S+yY${}=Vo(6Cw_@>i6x%{;Nl ze;=OjH=TukY5zfKG3-~CvYzpL)C`A7f4`vdBH3H?FWzOp7vg8w-+=iiHu0gnAkNqy z(tPx7Xa5HJqvS_>dyKkF+G%G!FHBh;*$Y|^;N`vKf1J0U>|sgL{w;lf1J>v3@g?I)u>;f(Ls zfqox&inos6pEqdzH@k_w_3ekqZq40rpM#$$>r_Iru}dn z`#AQQIBt}%NQ6arrI7a+!JKn)d3e4J_0qxf_ZYdkvb+cT)l?jWy0D(~f#~m}(c(Zc zGF$DX=SBVq6^Db@7lOfX(v!cH<#D4_q&3sWjmh{F7a<$ZPi&C$Pty8BZ^X&+af&={ zlxgKp!O)AB^NYOv0w3wq>QCG#)bdZ&@=s%77GGxN$9}+Ky1iEadyMfuuTmb~o{{t) zH$>G|{=h%FH+y7s9|1Ou$0r`2G;8(=CKX|@fC&wf3&vgr^f98A= z`yn}h0yf{X1;1>*kA*vP_}ycSgZXzb@tx6Zf%x7}eE$Xj9l&oA-*cPTyAHoO-|us9 zkC7L6MdByML$=Jn#|^>v++p~R{4m7M_=y`>_>S-fn0(C5Rm)fTt@BqOudp8)pXq6j z%-2$E2axNZtc=$lBc$O^URF@4;Hl%QOpN#%>5Gc19Xa?P9+v-jo^|E?rw32`@YDXFcvqgYUM%>Z$DhaA z{P$lW|G_Wi`E;K0JR8rq>k3f4r9V8cuZu|7=9kU?JkI|-oBsvK?(n_GC@hq5dj97( zu|bFLod5SbnD{00y~haW$(Z)PQ#_JCEdN#itn;}C`z7S^bpM|0pH=?}{$jo$^Lq+f zBlEvHtjcF5@kf>G{xKc@Yr-o3XKCdZY5rXGm%2S8|110R9-}s_@_)8gezE4?)%YrI zB*zP?e-DRM{_F8fsm6cRe$w-Qn(kTS7yQQtH#%(o7i#{$dVE=d%>SCAN(I*zmz!Ad zKU_N1k%RwL*OvdIH2zEbg#`bDd_H9JKkPrj`**qiAN*4Ge_(&kIhpU{>uA54r9V7h zrU$Mzw)tiAKgjtOwD~_CPm^T+Prw5%2dC%%s3!Kl!*|aAha61&QvGvTQ0D97#wgw% z{{9R6TbK40>Zc+WR4j>cSG=yN!u;G?}yHj~fxruZYe6 z#T+lT`CkO}I(+XjCQg*`W%2(5hwq&K-3|tSBQoC+4=$GRwcx*8Ub6q6r14+%$B0KX z{!fSYWdC1_D5TjqaBiHt9c{~tPh=luUO2V?%(sQTxs2AQwv z`9F24%CBh(pVRyQnM(S&F`sLn{{ueu7}Xm8Z-hwZe;FLCJib-_^%yl8|JC?F-A_XE z|Ir%%%d1rWtNcLxuJK>BKXD_h^e1ji(&GQhDwY4!75?=Yb2R>|_9xYPrpo`SDwY2; zwDRX^{8#M{@!wGSUyK`$!;b$;goU&DUsf}}Smu9uZAihDbHXMT{I8mab$)BT;6H@_ zH8TG@{C~pxce(s;*7o-<#r$99f4GiTU>*KXs%w$(<3=;*d$Y~|r5rD{`5z9;eD5(P zO_K3t@gEC>PW`?n_L$87(q@_O@CQp}oSy%sSWuVwH(B8`@n0@4+5b;h(#MSjT>JbV z&i_h{|MWCm=6?w4mFpj^jc*KpMXrv2-QL8F8#Vrym8<;M?Mt=Be|7yKZp>5q6F08c z_&>E=<-h9R$$qJF-G0r{%Ac#P|CN`k{7<%Ds$8{S#Q&Je|FC$1V)K8Jh}dlYhpHBY zZ9Zc@P*y!(!BcDInON|@ympQw{}17Roy`B3|6Z=q%jG}bPm=Z%bsP1{{%5>TQb#jD zhkujnHc0q!V+rT`5}W_^9M{|YCw|HPe=;6eIry^p@7DPLmoop$mb`KgN~HHA1>fpr zzNY7YnKu8PqVPGr|DU0xj~g^QG7sLKDvkfkAyoK};WrZht2O@X{8a5t+*qjbe`=-5 z|73oua-E-QJVW+HjsMFlRemS)QuomwiC+%idyFYq0C#YD{*P{AUXA}Jo%SLA zUnKLr$EZ^L_2b59-oDYpuK%Zt|Eou<<@=Q9`*A~$|5tGB^M6GAkML6F_ezKq{%i5S z?*G%p|Ej%-8_PBRUv2!KY+qFVUv2!KY+qFVUw!=Ft@6K2i~px+@qefWKb~yG^YH)W zwaXM-IcI^1t@wYwBmWQK|1z2XPx)w}X);dF|B5E&llhPLE5G8jug9niss6u4;p^i@ z1#e%4&Hwovmzx;*HUF!|^Ozr3sO8U}FWZB-FNFt4(G z(EaBe5JZ-%{w;2(_9kwu)c9XjqxyGs{?KF0o3F~%`LiC+kbP14UtOd6cQro2ezf_j z{03!yqtBNpeo^^f0|O}cUyV%Kxdj;W+I0e;RI(jfwAbR!x=p zUo~%qf~)J6nON|@X8r<44*uf{(lEkP?ff-IeKNvsC{ywb#V?0-20nL6t zQh7gJJP*KpQnm-r`j7JRF3%niA761J1MSDb_enKB@C47z_v7&IFIk}MhwC=LN59NZ z%+F6b{G|KokC;VpJ}iJkaqj2iM!CYr%`0l?-pqMsnqAHSs$BI)sr;$R zr`lSXPdfj5j0Fo+`9ej1J_a~=a zpPZ=rGyVC*)viw#Y3q~f`vJ7SN1pEy|JC{2Rj*HK`xEr}Vsd;ZuXkYmCEfa@wm(6i zFUspHwBB^Z>yz64By~PW_SjJUS+TZ0S)#2^7R_1Z>^B16us*3^tWPTYldwMN$icVr zVb@D8XMOT==5NY=q@rujw=o*u&dGc$*7gHcYx{xH@s0K?I_nqoKAF0HSu;`fXFA{X z^+0ueV*NnllfECINPAwW^9{=bO8fNn!3J*K_&+!D&Mf1j}H5Jz$9&bvZ!{Mv%d#? ztDMuI;Hr7nej)KZpt?@lFZ8kStxooD8~p9OKi6ixg!{L#8sDB(_@?c@snYh}TrJlX*Nj`QIN;P9E96>G7mG{xP1l=L72V zs2-zK^MC65Ea~IPB5Smo@4HXb{9D@R-I{;X{f`)(yb{ib66N{X)yI=zt^JcUzNLNM zt@*d)cv9c*qQ#R_w0N@A-p?YQ2UN`~a`vP zO=yyQ>dI;rJg>Y)*{zmpq{{5ig-Y;dczZ>*#Uv@Ap(%IZ(t(BCHuY5T#|_svs{ zS2f9m^`absjYK5CSgxuyQW{%HJD<6HH4JFTD0(EOeH{SA*A zm1XMw5!D{i{u?M9hdp1@zrSftQMI!_1pF#1u2pb(xW>d{J+iX2%8`R#-eLI#zFhf! zk`&+PrtBwKaK(HY>G?K)KeAZnledZev&^SbWxomRv$7vV-+%SE;pOq9*WNE#TB?pW z{e3B)PhF4ndgXXB`TJ6^y~;YEU%bz)jpqtT<@wrF>?fD~nf^Y*7z`g-ex0@+srpYE zkE%bb#Q2u;tM(pz^Qv-nzL<_bh#xV&m%Ct6X|>&tMP4|W-r9* z1RpYdKB#{`S-S5hD$@9-+6&@ao4OuZpv8}c0baQsKbD5n?e(t{M$#*H)TJ`6Mow9e!1@>DDMOC{m0+nxvl-j+`pA-eEVNA-^MEYP4NA8%6^da z{*At`P`1zG6#mfrza#s%EG0dC|EPU%|K`{DRsyl?---&<`ql9QzG{5S*2b%_*1S$W zUMFjO3m2$-Qtf|_k)!#yT&?`6c6Y@04XWSoiFl$w^FtH;-wFy;|5d28 z58p4V@lCZC-9}#0_W{xOEld7wf?vYmQ&FJU*&hNN&M8%JS!k+>1>XwBNZ%g>zU5tw ze^Y)hImPe$qWlcI{9(TD7z2iNZWrHotp7gnT*Kpz;pdMS{k}7P4?f4h&VNeZhV(Rk z?#Azt?sU=bz4PCz|BB(&eoq+YWFKVod+_*up|J+@#O}xZ2bbvgij?nD#{3d;7QEAx zW50Ew!4mu%k?Hs9fkTk<{+vU5fJa5W-2Eng{s{g4bk3-#_`1+UKgTF9!)w>~8kL{P zWL#)uBxw6}#u$_jdC2g32T=ju7tW;L7aME%GJ=nr?LqtIMtQs&Z70Qq~FKl<&8yoc7KpQC(4_P@0Uiyk)HaOfuugFA9yb6BW;`drusZ|e>mpt z$z*nWP+t6QRJ|X0sS#82&>nwg7$42{c;^}ceBZNq|B!xe56Xw!YCpAyvUav~^7e!b zkC#$g{SO%#zH^XsjQnGbZ0G&~kBVkwz+kb&OP0R9ioTIQ;{&Tasy~kQAU&_2>obG; zAG7+4_RtrEi}8r@F}KS@1`P+^-{#{ZBg57wzTbXs)W_?$^1GuMjHf64L!V`a*Au}< zt`DBy8*EjkH zePmH;&QI_U?aMNM&lBy*XPMEgETKw797cLM|F$uX{1M7O8t)fHws@`lZPz(+;@eh~ zC+FWT{Htgm&wm8@58L^7;`@Q&e@%JFXhnT;{@u8nw%kSa!z4tu+W9+jB{}MEM|l((j`5;9JPpi~gE&U@C7n?-w*i#TYd-ntI7{T{wZ62H~OoVe+cqBY&m{_+>)E^?}Yqr zl4Cs1HN5T^#;3`D*t-l0UnJ~v#}u6DW<)r9eBCwr7|QFi>&xYI<@W>n9M$DVJc1Vd z{1N{B5n&_$t|`kyHzEtSFQdA0GGuufzAq9ImP_{He133LS02{Coc!RsUzLx^ROLB2 zvOHuY@)P*Ndi)-^$-lfjSw1&9I!nyAd3~@CW5!5w_?O%S1{i(6t?)0={48$2ME|{W zqchB3I6!{%H)LFwOMJJ+;}jz_n%c+hANn&lIx!0a28XC`ZZtOw1QZzeXNr->e}C0Z zZ}_j{>bKIz{nrV5#q+~HZVuV@bh``rt7h2s!Cv2cY(vPluUuXLe+Q-sc3LmSKE;$nWQe9J`Gy zpjd*(GyjbDJ#n`wIj-^F!^n=ZyZHTlNyLL?NA9BL-;>4tCB7>?dY6z>7Wk`h^ytyM z=(FIDKOEh%Ma6gCsp4o<#ap+kxUEgbi=s!jz~8V$XN>zplyAuLaJ0Q$#ryZG_`m@b z@7$^4UArXA7Dcz*xdrtdwd>oqO_d+r3jZUuXX_DF-qzlx$fJi_a%Zd$*iEWbEHh@k_SB(+9)RJ$ocM>`RpY!igzIe6}l_!lxrNUSNO1 z(T)x&y}!)J4$y$gCOPa&IJ$SQEDuL-zg@-q_Nn;bK^1@MQz|}mNX2*Dq2kU?72kcg zijN(WF#4079Y%Zk{o(z&t4o$IigxVnAh|hS%8WpO{LwDqFEE}C-2wk6$@lLkzVZIU z{w|6hy9<<+DCS`^*>Dfn-8$WFiGZunz%%q~9~waZ8SLB#Kb zJ@kE%u;(Z*+Gds4?;(u&2Y#@Hj@LYCo}Z9^%68&o#KUC2A~Zlu`(@8hTL!rOD#Y*0 z(EQF#{_nM!L3MsX^Dhsde{uQG$-ceANT1BKZ@533rFhGScooK+=|jM> zeDkf69P!td0NKl}c09f7Hc5{Bw;;Wo|JDtXobqp>`00=t|CAZqTV#32*p2e!{5#jl za>|bmL~gP3cdU}-A>%gGC+FX=QIdNp|1RX;ZRfvzjU>nY*@*OV{sTNdv+UI!(QRAM z!N_fP{w+64`O*JYU^)NIny*yMHxry4x%Uch*THa2`o%nk% z^e>mU3d^B(`*?p)VPw7C-_?+-<-wjv@%ld854pTGYo+pde^Fs%wcWn`kjv$*-Db!4 z=x?FXmLPHp{*Zpxu0;ir{gj{lxv_1n9S;K|Ufm|*)%$q=3yn=Y{@iA--)!1s$CpSy z*Vwkn_AjsHd*^;>8<#u&W>cFF^;`Z1@y1#aZ@k0#9WvSuAQBOB=zE#5ZJQjwgp32c zeEj|owFl#C(<~P%w@N#ovf`Hf~h$frBbOcu2*cTCd_mEh=7ryNX-3tN3;r08;(i z*U5NMbj?Qa$@VAV=(;_U9R1zkrv5qpED(_7=+D-zD&Dh7#ho`v82)3`!2?QtH|><< zC~wy;74P1y;+t<)aeKRpckWYhbhV22(SXMh;rAnCY}n8VWR0(Ibn|9ej`nU1K)%ZK zhvDcgx5)BB{JxRxUvU4!(OYko<>BaUx2br`78SR)s(8l^6?b%~c<){bL!Ya6!yeoB zuguu!VIb@VJDynE4*MqL7{BDdw_6x|LA=pwFzO3-ZQMxlMZk(Lj@Hg3Cu_=11G4L1O=D*Y4> zyg}vx2k=GW|Egt_WzjAPg=Lh{yJ>jcB>s9ApS?Z){2z$dwgbml(F;WoRxwsuZO^4Dlj z$k@D@_`>ZEj~{N^Oy%)-fyWQGak&*gAYQnY(zBPWc;VJoyF9dKnX&n{4@u6%dPkXY zo7WG-=UdNV{2$py=1us2)DQV>Fi`c=6q{*;t61+Uk-cC7MlJK{j~uV&vAnQ|Bn8Ju>NW5 z7yNF{dWCsWAG7`cf;z{Zu7&?!2rS#%#Wy%|^8Ysg%kuj9k{tfSyMXM+Lfij0;@N_2 z|IwcLNH6DKGDnspzjrand4ug=n&-;$kZ~i*lk+d1C(9{6?nh+4oqy%^vOHwa*L%wO zSJyc5nbdwS9A4x`JOA2Qj-2#47wwVr-&8BhDL?u5Iy?XR*|I!j%tL$S{2QxeIpuFg z{<(JkO*3S9$haQ;k@MeDEz2oC0<_3HJO6DnWqF9dk1OZj>c?=l*Iy|=#XHwi{%5Jb zv(SGtJ@`a&OxPneBnJ-h_n%Zb;`zudTRt28SLKM8Beiz<^nGrYyo8sJcsMfK&OZbF zx8$P!jgVK_a(drUm2ZN4hF$(l^k0?V0(mvb!H>B{>sH&}V*L&A##Y;(Vm^FFbdL8n zq@QWm*Q)s2uo0MJ#{=Lm)(Zlv|MznLAF$U8ykfugIrx9>?*m@De(vuBTyFV$D;~+9 z{Ae%y{~Ye0$^UbI9|(L%bhiEd9L?Wnkhzuo{hT>5TyXsTB6I(QJR5XRCPeViiYjQ1OBVDqgry#f^@qH*Qq% zswx%FzD30?)hfR2HWe?LDPj1pstD|}@W0%D&gK5oZ~Ma;G@+3Eb#)W$nVg>c&$)K` znM=vPTJ!Nm(G@Gee+e&J274mmHEY0c2`^d%ehLhGh4wAPe1KKi{&mKD_yab>#?^J? z|Eq00a~|v^n`z@&wJ>;Wg^g=!VDH#6!ssvjaYmKppU-EazS>o=M`C?|*FU=j22aEb z!k^EnvC{WvQvSIC+n;A%mOp>W^5=a1uKRCqxn$qrzcYYMdw2d3#P>5Q969aBsQ|X* z6#vhxbmU}Th5gB}{r9ZtvOL7UKU&T|yG)iNe*gwMQeoQHGGp!(SspT`qdYnP^;2az z<%hwJRNDD(m?q0ZhWLJcYkjNCxG^ltDL?LCWV)Sy;Uq`C7WP?uAHS1-aj7Hc{SV;& zMat~_^^;|Jh`;|Nx4*GSmQ(!*5F%6T{7WXv@{sv`;738!@)?lP?0dnfkOtp-(r?$$}u62OtR&Z(SKD= z@mZ-|zWBXExqR>~GTF{Ae*aLG!#+fcY&q@sSIejPX`)?z3HqH~eSVs0!Hr8|ktBwc3_rz6^g}rTX)7 z?$4`ie_k%WU&i+5RpoYl+@Du*x#iE{zxnerYyOVsar5o@Irrz0stT5kSeGFIY+bna^MN`u(ESHb#H^Q6K+)ZX_|+?`O=Q^$e)U^xuIB6??ItSeA?Z zkWTtpvs8KcbQRB>sbcy)6S=&(b5&e5Ma8pctC-&Jk@MehgNm!CsrdTqRXla7iWe?a zam^$Z-*}^n>3s*e{`z_q*G^XP;>9X1r5BEIL`;1zS)$^36II;UsNy;F!lRVGxmm?^ z^ngo z!Tv15_yZp$c@f`V7BKy3nUSHz4-+e4|Jg*_{}z{#KP|S?WB%yI{$@LVD47a-$x7_> zlS*NqSS?{*e>e<#$7;;{@V^U+EPs4H3+*eN1AD~g+4(2e!M-pbVd1|Qg{}1cS>(Ux zM{NI{buIjNfn?vwe*>HLj{EP4g^s)=BV_)byyHJg#yfJd?>;_%aQyeA39>w7h~KAn z@=qQk%aPv)hZQL_?Pr-Wk_9KMDCK*!gFSlI7UXhxStjHKU|E$rnJY>-OE^_|aIkKGcBVdVK zXXl@rE6YR1IP^!(e|?@Tr~K29f2^JVhJ0BbGJ@!@oc~76fBSq0&?4g~KgMqezsF~% z=l+&%3(huxpw*a=)WolUn6;D{z79a_urYe|MvMPe#s{}_umU|bo}XBjF-%*w!cJw z!$yJ6_TSIac*x9*fxp)L7XG_H_-~;vdcKJp$O>%#9pL_$k7qs~F9?d=QRK)clFRP_qAc9z99(cYz72e2V6e3$9b;6qnzSgD{Yi)nq29zX=~b@6Trw z?4zRUVvpmtmpsIrTpHF`27tZI|X?V`OETH^U-(wmtZ~xbK)Z(R3E+{rInx-j~Ypgi#x}f9Lbv^#15m5cBWn;PaW>oHmlvd@6=S`sc`Ti!idogOW;*I79uzb%CJm0|mC_D!DM92O3 zL;dcqeAYym%H#IYof`pkll??}Wro{bLt-<2r1YFWlpgw+Yvj7m(d{wq{a7PEX9E)C z;|j^g8hIS^T2a0`nk!_KfalN2BYSM+|IVjb9_c5MgB_F+}ZTT1@Rx?l0T3lgjiS7;R&<>7uN^s*CXHtteErTJo#aITiX#TR7U{hi2q;aQ&n87D z^8Mxf67=nlX7~^=O7h}_Sw8o_lcGflvwjnM`THDB{v1E40tf1Q1m$NGX9#TNFPbD_ zKi)qvM>KEmqFOiqK9U3ts`L4-NzoE+Uw9U@Z&8Df$4{ml@0)lh(ETjnN&tqB$6P=7IcUjWO=v9gK0E08fhMCd~SI`#!szVLhOY`L7G+FpO7!G%o<=(_!i} zKbv#I#B_g&Pu6((Yz5=@N94zChMf4&N#lj<*PTi67~}VY^qt9H4>7U(vo|o@uNt#{ zk)QNujaNV3Cq#fO<jWO#jq?8o*+-7x#z1 zp9}lpVf6mB-8o^O;)a(+keVMze;@3V?SHmcvsp(nxIPnP6kaP~C!)Tk-2R!rSFG6o zmngp{;iJK4`Y--IG%TL*Pcz}4#QVg|jModJZcZPr1lZe9)Fa*>;p=ZD{C#A)f1T^Z z`#WyK!|f9{((o^Z=KIP#|BLiId<*Xi#v8Z4ygdbGd*FYmd`ckZ2cmq!ynN`VIV(aC zhsY2AXbvBYGU(^ChRJr9ef!De)KAt@+>@6i{ z`}0_jcwd_(W8L;mO4!~FF-o}7IpkG*;b6mg8$E^MceNp=Z;ii{vR^cV&w&3?Tc3_$A&IFau;@!FPNU zhg|*}TKR)%%Foxz&(q7#W5JQie}#37*#1#k`9Zz>AS)QD{O8#I5z8N~l^@c}53%q_ zi3UE=z$Y5`L<65_;N#N(Jq*E8|GE3{P^tfEtQxb(XWsh4k2d~GB5~=`CD+@D zd)99JFQ;7}zV+K5CqDe(!w=qC`>Pk;^5B)gME-}?e~Sn4bow|h-1AD}d-UHupL%oK ze@m?W*?%4V(yxE>%S}H!n%h6{`?tRK+i&@f3=O^aKaXzpeCvY`{&4YIPr9~#=EF}P z-TJdw_2R^LzxCw4$fJLCPwUp7%{}?UA3y!|Cx1KM%HHwo_wxVsH{M%6^Yq#8K6&pK ze>wBXukSng>-RqY+ux4w7`k-+{crhh73F+UWdAj0cssr!s-Gch z{no{M{%YWFK75bIv5B`6y9emgrPEI)5}TI)LR?M!CXsmW55K&pzWWd2!eV}^2>;WE z{KH#6rqexHu{StOT;fr!XPHiElW?69A;5qdu=#O5^d!X~e@A zJv8S4JLW!@xXI<7u+8^AtEYyI3b45JPz6`895Rx; zC5@b>UnAw;oJLO5t&#G#rIFM0YNY%fY2@KF^3F8!@-%WIjl3p}JeEdYmqvbn8hIp* zygQA2SsM9cY2+u<$WNt_KgH!=s(qhDzCr{0dkHWiz|NVpU?jiE<;rq{+153|Ojq-M zlkLccqke}sK!5Hv_tjz@yp874-%FfJpe&R}Da4cilyb9ddKg;Yd`EzoU*E(QNEX9I z&l+9Jb5Li5isFZ>nSrbS!-7n26I$L(HmBa^tgHFfvBE~7H00AU)$G_I54S_EJb%;s zrpa-;-7Xo&HS8X($+J}K)^Oaf$u%6$)Z`kD`!ufJTy|h%6%!<^_CsUX7OyL z9;F?5kvta98>##Z=y3lSUhL=@^d4(ES@`3`HJ4i@77Pa7*N_M${4;5 zaqmW|L+Jze&GNk$IvBqEv|~%<2Kny#-H1`;I`SiUaGbByk(5S^ajhTc;e+e{nmFNN z>KJP=>!EUZSx!vsOK&gKHV_sqwpA-tleX@MQ@NL~p^j8W5v_}r3w1RLF^w9n(cWf&4qgXA ztZw=KlT@$LFI7(CuAb)SJ9eUMd#5~Wz|TnYzw#VkeNK+I0cZJ7ibrS-G1FQ@wCc6e zO6h1F(N1T1_J$pEYU$6tW9*psS+E78$RE!*hj7h*&X1kVR)0-7CSaIN`XFa;vZ>|n zILTvK=Z2Ac+?kXgPDaY_+EIn|h(B@j_Pc0>QeU4@wWzX}*pW_)mPSTAtJ=r95tGGxuC>O1WET zlOZsdA|IhP1XDui zLhdZPwNAWY-CJlYrQBi)jjfdVO{UP;a^zR1jb$I1HryufM@L4-oz_a)Fna@SrIcg2 zG>@1@V=HC;oH&=$*mC5uHexBGFA?X;Yh;Kxq&4Yb<5)9pwPRR{u*_>tZ0gCyd?WiZ zV%R)6ogBkD8j1%5yJ$S5jM%cd)tYI&}(tTY}|<&Wpt8kXd;23blCs`0C;yj@TyO8fT7N)b1~9JdGAo$|WtNluKHWbWdSp#|XsD5h{CB*`wA5mfM8Fs0m#zK4xxO)Y8h z?A2*u*mzX#=}4bbXhB3?=w&>W7TkWJrQzaGGx8cX4z;v| zi2DU2(!`Gp3zlvtVv@#+@FB6xbE)MPBQTSb@?V`6hK)bvo{sc6NegaiEh1Gf-3Xp3 z9>_VDBgKu97IMYx9JBkBa;$UF>^(+nAx`MidzTe+ zDl5~M6?0zE7{a`jR*o4~c2mcUSQs*Uky@TDRtISeIlXkW;LhS&;$zT}D_Y=tTSs0w zR@dJzE9Sg1Eto6D!>y4@T5_k1#-KdwUZ;|~{mxi($Pm?xDz5$jvdvxW%fY7IFJK6h!JZ;-9S z!yS(|f*~Rw_y#R$AGgPS0}MgZv!I-am&&QfS(Md)B3w)_s|ICR=_sxEYqHNIlmDDF zR$8Z?hR_>Rhy9~+Mv|MdnM6*Ro?z}+=X34Yjk>LGECL_+ zQ%+~slk+x3y@%eO;99<6OrRcr^tqwV%m3(o1ztOE9a^i@PA!)B>(e<$ban^bW5g30 zm)|u=%Z%3Z8p;iMU0!XNh51*Zxv{yMx!tJ5JlcdjsLGl1v{uLqG`YR|tR`jtm{mNs zx$^q8{O9cY&k4EKK07=gytF7nR{o zPN!$#FoeuCNbPh@_hV1GiBVkr4nN;Dk6-F?cFo72*qCS%ed`tNnop9S6Y>I_^fst# zM`%8td0IMZ;UiRDpH$_W)5zP>$UD-=JJZOGH1b#)`Tc3+-D%{HrICO9`ok!a_DN7I zuiamgpR>j=MXQ=T>6y1GPksul%9EdqjwCNnu3wpp)TEKurIAO{$d@I_p&f1gU2VTA zPu8C*52h)=C;x(^{G5AZv2ynRRd;Q%jUDHCxT3e}MwMyBvDdraHFv$v($22T zi?pVKIHjlyDOu8Lb>R&z$is_7?(zna)JjT%Fm7UOlL8hRx81b1A)pSRhXjy=0P#Z# z>#0sr16>p;U=)4OLm!MFEmWjHUks%GKmR|}oFmed-99w4%X7~6-{zlxE@x&)ozwf< zZ#7-L@)Dk!2x*Art*65g8vE?9{uK5A05V`t`PHB(^}o_4zve4Vmo>ZIS0MYxkmFE4 zBoTjxG;vwlQBP^DvOSgmyyh7C`+uF^0oxCJnTi$A@`Hju-z)GDo8N?w{gKwQ)8w_f z{~}#IhSNTvU9Z>$7S;ZOUA&6tgj0O?&C74U^X8)^@zdvi^WU+aA6~t7$^G{i-+%4l zzNsRBy!1FRS>=y8_(8EhD{`2C%ga^fkCVX8yFyU+B7bg5A z{CSUN?>X_V2|p|LOxPuMP57$VG2t(Y`zHL1*f!xS;+_dVjn9kJ^1m$L!7>`YB&sIt z6nPVVN~BEqqKKLB1u<{JPl~V!pBDiWJ}2COM_ciI0{*gB$7hAZgy9*vCj6Kpu?L?6_}HIywnj%B zQQ9TD%PIe`XB#Hr4`6{2fDV2DcA8Jy;BkF6;kXKO-sz^Wk5o@L!CCpxBp8E*4IUrQ zc%Jb-V;tj=f514#7+Q|h$1xuH(9$GMF%B@sF&_CT#(Bm~#(0mE`7^FE-e>G$e86~v zF;1(|e>Y<%<0j((V?W~*<9m#oj022!7;iJ)XB=jH!1z960Rt)Kzr)zcc$aa2@gC#( zOC&!J_>2u!druRNF|J-AoML?N4Bw*dex|U!Xmw!~+nEZ2zoh`!TpG)k2lQ8+`5(nQVO#VUeze||>gH`5hgvmdb2)s_1{Bw!a4+xWg zF4268F!=}h{|#aC&m|u05+?r!#NLkylYhP9!A}U2f4$=HXN1W=z|ikieb}&no+G>q z#n!rkD6(#==LK{DbzvnEZqGHbi{#5BQA9 zKj04&pZo(pWAYF9H;7OE0iQAX2mBX^PyPX)G5O~b4wyh=`yl@Y#5QB{uU9xmh)@3Y zifzW^U$5x#KMoJ=cfwELh1kCV{T9Xx@wd>w;WH1PJ-kOjo3q0!0pKo*|K-Pi5e8mP zc+bO2e7Fgo@f{|3#;2O#`PW}WEqti2P0nrceuwD|3w@vQ2aqQwSZ%1Uy(Iou-o&!f z`K39>Mj1CYDuy0m!ey>l<2%eDNXcU$HQ(`j;pm)qq9rW7fY9Tkn{#j&Xdk z>fZwbm^haI>lab$sQjvpD1B9ruK@#T0TkBKyR5HBDu1P?>VF-SKS_Us4WK^CS@m%o z!4N?AGNtBI1e^arXViz=hiDRP9X|qPv^mzI@WrqBjG1zhG)#CO5Z0eVH_`kJq5r@J4(SQ_VEIi#fd3EzhxtP? zvC1&#!31Arn34qwd>7Rpx54*9Lq1rF{k3*7G ze;5dGH-gO`bNu&N-w8T;vc`WA^uN=l{0FSB>JOT!tbc3%*l#cz>u~hX>%H`;&`l~wY`Ju0>UmR(euTosWx}GGd{%jT(s(-1o(pw!}jD%J*qm;&G zgWh5z;|mt!zQyI+qs1j7nr*~m3!Y3Vx_oE&R>c?{n;nY;^O@3QJrP?dW~z%z#i_f& z?C?Tmbn5QSm`CH?saMmbMbKNA8I4Dmm#d4R;^a%qD~Vf`+oSoFL?}{<$0D`twA2m7 zypeh)yf9pxz6p8PSK`s(M%uTK&xVTY>FVO_gm=!94Q=EzzS&0H7jY*RhuyhY5c0@@ zd}28f9o|a&Uc&r5@zUzV?a?`3#%nAaqor29#;B(r; z@=KxBIb(FBG7(x`%0{Ov!Ku6Bx$s=6i%c+=Sg4G}4JhNnNO3G*8IP{c8^K^@!WStR zql>daUw$R)9obBTZ+7t~;Wnb4Vy4_puG+{(JuBJJ$QqWrt$$N@V;HA&XmTsFIb4Xx z=87@z!d7}rk6UCqUB0z7i}e_0U!#+oF<-X5&Cu%I+oMv)J2!o#GZq3JZ=x2DcGMf< zc!uM#=@n={4FlRtyO_*)pY5xHYThgXcYHjnCSEV@ul7f0Oj=yWa7O|~RuD@JtTP9l1v zOZ`&**k9U}z13bQ|8DJW8p|E?CD!BK$wJ#U5L(^jxKB!1Z?21fSs5RQd#CzW{etrT z4bqce9IZ zm1XB{$DmBrL?#Se&nD595pA?D%Xps0 zzr@{4abztE<8<3`(kj?%T=#W}b4bQHHhD6+=t=DbJGB*6znt&FiR2RDli5pzHcnZ%X#H&}2Cg>(o}K^*&O}bh>ws!@S9zfHptrp55a4 z2aHSk1kEXK2l0Lvac5)GjhJ_`kuJe~B?k8lFWf7}r?=vTp%>M3Em6)N09+FIn11uI5VX$)#Mm zR#{suATPx>^?aII$D$1-;t=%nbZ54AZyhym_@%qzAUouUH~{pZ<&9zwCRf|4p)t@>wZBhd=EikLyzSnx&N-xDke_+IJzgDt!GPr0{lWx_AdtR$}c z<8*Y%rtK@wkCT({LAh+&u8&%`q2V66GBsy$PnuT2F!S2&Ta!GlRF5b47VH?2*av1N}pwqdy0bKPJ^bFLdzdUtmK7<#B%tlj@&e?I52;87XN#Agvw!5bU7+CNow4 zg}oGt8*bJ>Qo3HaM)`TBk4vr)5w^3#bX70^#N}1{(5j#QlFKD{f7WFuL%(7>Zunsh zCPn`q&>vAd+(7>mZZg|oAL-kSe@pqmWmhHTM@+|jb@~#-0TbF+z9GhT zj>&2#&-9BHJ5|k&{w>h9Mu+{>R@&9*y8O>O(0ieNF`=J&d54&8_0!LEYk3d2f~@8J zi0M{4|HX7`c|X?d=>9*+{R-{t<@IZH{oXv^fvy@2+Bru>mq$N0CZwO&=(`$Smp{o2sT(!un&bn)%tMORSq`Q_(7s|%krhs{Q)|fFGb35LY(D;99O|4AyfS zb#c-a3mQrm%7wa+SJ{G5%w_6HShwWX>d9IolgZU;VlWMdPi3;)D3P;;TupReIPs#Q zP+qB=Kzcp5Rx4D>9EL)A4Pp|?)hQHWuw1F<219ep*K2hnbBClRAy$jI`e?nrR!BGM zIi!+2D4M8QV?4KB$mC3Pifk^QTv;`%C& zl(kZ<7};dKlFUlSs6gNz`ALwYf{|(TjU|n~f3Xw3?}9Nnibcu&F>J|NrjzGq~^Yl*?quu}%a3cX*zon0z}>P~biT z()76kG6Q&k6w@aofYp6bGU0F)J~$-{z(-FX@eRq(^EI5N<1JFn0n}XV9pGdBd>e0> z*x`Jw`6;Z<_&X}pN85VZV|}&0Ry>uz!cDk9+OZg>C{k8=EB;5$zmxuq+K1r`mEQ~n z{w#rVQWQGR+-XNWF8eC$TgyMs`nXQjlhyxk0l!`SIUA&S$jVp@oQf)aRe$e6#@4)1 Q#Z@BxeGUww{xQLz1f0(H=iF z!O00V8kk7Ecd}CVorqLD`fvEW7O9%`fAIY}KL3v2Kpk5ZkNiLK`S7;y9eQt+I{WS) zgb)2{W8~0>5q00MBB5Eoj;KcuwyXOl7pFI$h>WFj_a9ek@za#Q@#;hG{=sULOPu}i zwr3wX5~=^+->FJGbrN~uP~@NMQU9R>cfa)L3wOVC`dbHHI`G)tFFh1aoCS@0@wsp2 zj~?r}Yu@3WQKgUfy!o}qdrrOi!=Aey`%%xg*1y>E=7tx0p8M46J#XCoX3y-$-s)*7 zeygWw+LA-h?fUvlkFNjvOT}}qICN^m<2}zmcDU!-ufN!H*IPgAIrX)-dhUAsP2klY zI`s9gzx0L24!m^d+`~QBz4%zq&fPEeJbK_qJ*RGYtLKed2xsx3osS=Q>ADy1e(6ro zc=P~p4!qd&g~x#dI$wDGG;o$5I`#V3Ub+jPQ{eB7TORKz{L~M69z|U{AA7TBR{rb2 zx$@AVU0;3a(YbfOH1qXu^*lfKv7YCCgu3Rw(DPjJ+dW0^5S?X*oviB#+m;?W^`pCA`og0JUOM&Ww;+eZJ$JT%AMkYNTc>;OI`vjh z;mj3>o_p)SOV3lgc07(Yf2-%tUC;HrvEhZDL+jt}xeh!P&b;c-&I6Fq3wOVCYWCrt zFTDL&&z*1ou;;px7tyY_d+z$w+ratcq32q@`qH)F_tfDZ^&C13+M%lVPo2Jd^QqI{ zN)~okJz1Ea-dy-hRf_ui;F;dq82T&}PH%2h>CNLpiL=Ke;~u5FBk%OqzI(O$=U2ke z5#$|@jDPe5^8U@}%8|EwYmbH#PlZC1r_)b_W7kvJ<8SrWK7;frT+T0(O}9i>>AUyp z&_DP7Hgf3H?8MoV(9>6;b0MAFICDeVN)aLnWT7nh^S8x5Dpe;%uxz z-4`Xhxm0~|)pa_p5V9y5mpHpPRP?pK%un>JLb<~7#8WFrt1nKj!tX--E>vF(#qs;o ziDmTrRr>vz#ImBfiO!YX>WhiORbO63X`>TouK>MQ{_%>_-ILXpWqFCSE5E<&^g~nB zmZgP>vsK?;dit5EYRj0>iKk-aiL=GEi9;nwpHXJ;a`pj(nHvCPVi{FO7{qyh} z{E<8-({GY3_~KQ2;RR!mBq{mbvKIQ`IK3JzbrBH8N(_o8Ji^B3+EYj%ZB9Zyp0>oDKtCjWKbX;_>nRtr%mKN>^M5 zeLD`_{qUU-mycheYJW9Ko&D7fiKqTDJm!`(bn`gq6YWH;sy+Gr$bE%h*|+)4@2ooX z#iL<-*<9)@u8Ddpwk|@@8lwN zmc|nmI&#&?I{;4~eTc^w^wG)pD(+LixMu~mhuU`>?fYd!%{p$z6U;66J!V#Hk*bYF z)u*dQs|AJV8`v%8qibNtYhce;!>(7uzE>Uk@U|cDywn@vv4CYRvZlV+2AQuyyD&dK zD)ZwXh16KopAi@KfA2n&A$os}zUdF%-}MvkAApyOPpLZ(JfQCT^6~uxd~X83B)?cl z-It9Y$nC6@uS7iq!7m;lzRTYa``NBf11}r@!aK=F=zpf4bOm4J6@|R|Bj-QtCqMrg zc;}L5VLx!}K5}{f^2coJb04!@0rctLpW6n z80+H!;g9PF{_on)BKKDz_nS~(f9zp=Kk=l0J^}oX)1Oy-%yQBu(xvI3Q597S8uHYZ zKmMrqlK!+0Lg*L2Mg7DRImmeM5ctA9(WAF~*!!n6=k5Yu$FYX2%2NvxKkD5=I)(M+ ztXPQFy)me@x}Wg20Z-PG(hhu6pYgpK=603FANt<()@S}Mq{>w4e-WfqScO8gxK;+H zpLO`efPD+T6Zk-Gs$7-n0{CrTw=^erH@CE_O-QV!jBsH{g)vfx!yy$J9SMaoyG2KV zb#p~Z6vj}sSKn8zl8mtW2j)-EPJUZgx8rfc=PfEfp_ZyVaP$enoY%giy(8Jt(1>61 z-MLG{{9WI^V|P1b(aZ?89CNAvFX)?~7yzNKYXeMf6@+m8C?R9ka$=dR}69StqJTAP|%cJ98p z{?>*3vwhdj-8*+Rr0RDq@_#L;-+k-$w#D`J?Yrt1)GVp5?`T_S1i5QbeS33D^ES|L z1(l}umiikQUJr0hd&~7K@A}rpFAz>c2g))-Z5_$&ix$@3ysLRveQSMv4ce*OH4K{3 zzM~=8+%$X|l5B2oPTq@!A94XZ0%_6=xjrA%4^---qEnTqh=VW@3w}vUCqgD!=jtfHQW}rFFDZ+ zo!o(T!Z7GVX--gQId#~Wu=FiAC|d`cNyQ$ErChsSN61nn=l94mWMmnF1+ZC#4UJ`C z2Gt)~hWrOxy$6OZ>K!+s|I3h}e;*@*^bNSXnb*2~dwX+7jjG2@RK1t8;hGKgI;Xy- zzM*AjLp#sjUOtt?`ej(H$#P9hGm#m=-LiWv<8;7iql|F=kwVE@9sdT~{o zMEXq^`0k?b(&FxPdWMSicBf~=_@`2oKF1|uVBtARJ1AVXGJ4Z;or#?d+H_IJuqpDX z&tALc+EwwZTa#_rN6{DUR?vP4_Ndf$N^0v!K^L#y)6@{B-G!Yk4IMjMcgH(gTU*-a z#9`UZ4eibG1&ih_o)iBpea)*~rdIFS+0qnm?`-dA-WA`qr@bTI)VzJ??&kQGjT_hB zQ2*I$Hs5gV#!uB>bK{n4KDFkWHEQn0_@emS25d&e^_D_I990vqw7-JAMu*p1)K`%n zogY)A3AA?;E!V%XC$FLv{_p;5=R~(h(I+UkB&z9NLNzOB-wvhDl&h$LqpB$YN7U8; z940L|)9vAhR1M%&9!xaVnP*f43`ZtDqQdLLD*Bl;?;D*#2k=f28bsf5`JS%fGv<3M z41n@+v&P>wJN5Sycj)gU=DP-)xP+H5-&^l7cz%=fW3!ApH46rH(`b&B-rVqVc1 z`_xv5{}@c9rW_lF_@E2arV7YpiBe@(DYX@!BlyHWsZ<)D<6nb3pw!3ksrhdxgU=a! z)G7SN=NLX^Z-53q_u_L5pZICe!RH`8$M8{*Lk&J#@wpeD@O99KNyVz>B;AhO}>%sKZ~1_40;j5lWmWQMIX1 z=p@(w^#XOG3i!}fx+dc@h4Axw(_h5z82(KuMgBU7lzo33XOC5vys=*J{*BeH8z|)g#;-Eq8Ed_rO9_N_9 z=Q*DddMO`pZ8EHwU`WH&$#9hTix55J7bg|NZHY4x^)JA`Jsd`Th9AI?6S2blp?=^~ zzG~ar6QTMmlHq)kSMLgs9yg$^5%gXwMSlW>-my}B0qw=W=FvlW&_joV-e%Aves2dY zz>4@i2KwJny-a_R@E_&=VR@V=jn<~FHu{Ks&0jwAw+Z#@_G$VRZA!Nf_(b2xXId(J zHQ|R@9`jXqrwWrQstK1Bt0{M=;*_ePu5U}ckM>qUp0OxvB=HCS=kcaJ6~IOENUg1@ z!q46#$VdApq*gHhHRI80_@=wEqq-8$+o zrgx$=96pNv?Csqv`dFQsS3~IQOn>hGzM2wuF1XAU;)mlH+6-tBC5%mW5--H^;NBgenRO$&dMTL?7 za;aK(=T-o8X?(Ib)d8@W?2+uVs8_WECVPfGP3sDCd!WCi>S&d!?uyh<|3hC(Nq$@& zUCH<;U)`nFQ-44oN=e=*AEWUL_JQ*AseEjdlt(`BuPy+rG=3ELS5tY$C;c<_rR}j) zm3Kv1-+Rl_sJ|-8^~-p1c?t6K!aQD#Esc@Bb$3Rip8kBZg!KCpT0YQk$a6o|!!=%i zL{xE?;`YK`MycnkR6$BbpYOSrMlAIIMuJuBo`+hwOSC?-{<3{R|0|#`qz|kgkatDc zj335cs6J00UW=$o^pAHzo~s0={z8AwmF^`Z&kMtNFt!i!R;&Va$J|Z!90S z{}|ka?DnWi9uK_sRE+oP1HJODvH3Kfu>Jpl#-mHd6Mr!+Kje?gr}0IR{R4kmSB&|I z8u>v!E`MX_A-ByRmV{paVf?8qB-rp%-c_dizwCA%|GGQJl8=I(pGe=6d6im#y1jw% zjQP)xdijSs#uuc%lb{@zphWu&k68{_Eb#CqxGihweqQd+Y&4f(2s^a z`jB@;MYNaK9~M5#Cn5YU$R~V^$0T3KdsLo=6~lq~iTJnYC&M3&k1{_Lq(*alL67BC zmsAT_zi9l0yb4mK%pZ#Ic!_-Yx8tIG%#Q@qP$=-7d%=Ab4Z--2@elLSTFRID2%p-= z_%Bzf{dcHIDjkP`BMs}NOK+phwfJCsKGS!846JufV!dz1Q(*b z4VR(34aZR4z>O@gFmcQ6$N0wlCZ=k7=r58Fj|U{**XARi_%-=h|4=^Zo0j)RhIu^V z{y z`t=Ae2=#gf{jrYekxZ zG3z6y|AZ-T5%i{*`(HDk)xrLCKIB8=5AN?)0-MJ_j4!pRSlBC%e5@z^ z`%8i=VtRpX=z+hrJbrJ|?YVOgZXc_tevCIw@Zmyz+Q6tjtnZT%Q$CIHWj>EbR9_p7 zr(A#7?5pVUo9;(g|EWHVZ+Gs&{y~6#crDwXV){|g=lMmCpPGIoNIy*bG$KFH*Y=%* zzOFwzeNq66|M2?*q|f@EgMVsn8_8G8CwwL+eeA!8e5y+zZ!_V52FV}&wE~R?J-uF_ z_Ur!9^4Gy5sgCH#|1qFtlDIbX-TjlBcxct+A&3Yh8_VEhCn!kdiD%0iDrhK%5=$BJ@tQQKB z`Q?Vb;Wu1iVBwebP4lDeuL=fW=3}mp@R@(iKWHC!Pd?FSIKsDsW0>EKkWYA$P2V41 zgx|a|I$!h4_RjV3c*OP9GmQRZc}={G1ahfqkJK;nDM&^yHT6k$J_tSNXSjwyTC}|$ zI_Ld7<421w^Hv<(4zLmUhJQQD{1J` z`cp;v&D$$dUu~+gGR+F5=NFo-{Z!4^BeGMQ|A7b<|B;v7+-a^H@+6f z_=cWf9$#@g8pm&fc|8`3u|NpMdJO%k_qQ8=f%``u|MRIqZH+i}h4$5^qP~6o1Tc2{ zv^{J2pnS2$M|tcg!X9y4gf7mH)=p7oJooTRJo>Qz0DXe{(&zEHjrcpdu|M>Ayk-9T zM?dECe@}nt7Y&gAakQZH=F?pA!TN4pVV9!yM{n;PvVVLD$(ze#{9AYDlO$YR+Fq)S zJ{WzW@#zwhztji)7|ZC?-%FR{z4@NnhDMjlQnC69&(h)@RTQ z=nK`)@}czu>#L-mBhCSXcmjOvN2mx5a76ZGA;aN>fw4ca?$6SS=L2I;=i1+2 z*SU#FiG9&}fCnI251b_X1VelQeuJN<@cw0-_cQn6{tnMyaq&$X+8Z42F}@@t{{A7_ zhy72sFMhtF^_d?J@cj5X?Jwu?^QSy-Kh@f=JXsob_b0JG$EwBqhu}A`-}o@r*Qq?S zp9d!2*hp}m_xuFy!TuKSPnr2M3Vq@AHP4Ulma2PuFf>v7j^mjv^rs>m^`AeYJn$EI z_VPw4m1lKoD{G_u!QgrX{CW6mfvDTBCV;-!1MCgQH0eTrzl-wUh-v{7jQYaKSa5tQ zd?Yx2rc{hn)7PI^%+a6Nf}sAyyys7^6`p&20!G7{wlHpy#v z>Ki}*V0|(AMnhK)ebe|CQQt;+`iAGvcc@sAj90Jq>wd-DAB^Z5jmH;ZeB{zcTc(T=_q4@}?8{vq^@)}JHe+4-J7`FXLeU)U+lrC%EVBI;MEqhBRC^(%zd z;G+2v^Ed7nC-k0`{noz>k-tREDw9v^11z`pVnFBpFV?>mDnaGVe0oP`L=$9xA;(_^ z_qVmFQpew+*z$MSi1kQG4u1zKUuOF|z`tQ>DX*~Uc?#Aub;)J?e3qZ@Fuc5!`b*&2 zR7H(mAY=aJ_0QOvG;LUz`{M~I#r_3Ljeo%%omYi;K|}5NCj9%1Kg2T5pAY{Y+kb-i z8(*Uz5HWo`Pr0%bC8d7yPhfKFUx54jfPVzm7sy}A4=`xGz;OODf{p(J=v`V`;nPEV z=a=#grPux|O8J3_hC#2=U%&8rMEfIDn)U01R1vrLO4Huj)TEj;P1qz4#xE+O^`rPx z)Fl^}PLleu-%;IFbS3duWcZzs8qM?;6FsbVs=G$>c+2IP-sr`IKic3=>slD0>U@8Z znGt$^y2$VsPR>vZ0n2)9LMp=amm7ZBAFB=d@R`m&+v~}?&V`W4F83EZ~eyi6Zzbq%pdLdWBo|acZ@zkU$j15 zYWOApOj?dgduaX1cVOmzP5fKt6Ta35Utdt)gj6B7{|ZBIT37s1k3W{Lx|G{ztc)rTtOzNnYH($&-ma=AZjFHs6y6*$ezF*jzmMRnPb8NBQ!u%N_rd z!YinJq2>qjoWfqZ+&^63v}yVdRNG5=*L2@Mh2=SQsx8lezY6&2?p$&?mnZ+n*Qx(6 zyVB-|__fQIcjX)Zlzqnk&h>YxDyow9b9)Mm{|TmD<|niN%4J4=mh7x((Zx<4g(r7o-C8%iVp+T^HF`qYHTZ;tK%0(!cA zQ#3v9KU*H%owH_n@@u62o6G(&5zjwJ-_H~YKdE?4nk$s?t-SOyk{69P&-(gQ&G=QI zS8e3|RU?l%bG-Wg2gb`}K|r6%yQUcZLH|&D9e<(nu4&>AR7mzw-Zho=rC==j2lbbC z6*=;+uJ-D~_+6JAR~i?7L0{~%+}KlKKLGjA=YTxBJEuESt2*C@aLfFBs&nZEXq z8)xhp`W=vOcjrWSu;U{A{brcTmwWch@(h<-`cl^2dAYCekYB)G4fTP4Jv@2#869Vp zhyF~d;jxGE`FVM1Dc}Ek^F{tRs-OMYFrG|E74Ue&{(hhzxWDE4^78ZpcF=?Vg8u@l z&mV7N{{9M=&*$}#*gyKqm+ueeQ~7{Dp6D<8r=dLL&GL(pexm=;UOgO{_R@H2?|+f~ zvph|Hc~{K$FXQoJv@d_`$GGh)?<(Z}Wq&!ygXr1ge|P64*ua1@)>6J-X=MK~+Mii!`+FgOna+p5E!q!wxrW~rf3r(9{s+9j3;%>Ap8wYK z_mICrz`qOhmuY-F4*`FGjX&3KOnFsh{Lg6q4*C9FY4ImpZt;ixDfkzNKQH`oK#%t) z+Zx|zdf@j;oBx7T{w%{EJ&z1mY>ox}wYt=$8vi$=NS}b8Z~Lc#{*@m8@P`Ba`IbDQ zfUnT7_Rp(Gjq?4CAwTS|&GFiIVxy{9JxcEn`u<|`HGS~U^?Unqy8T^hv8M0&`(2^& z_5DBm`KnF(!rWijkEaUS8b!X5sb2e;zWD#a&xZOpsTON`UVlzB{OaeozJFgn;iG>p zH}sU|*XwWQw<1-*{%-R84fM_aq4u8xZ2SSC51O9w-`4z8Y%b^}d$RoR*k7ZqQS*oL zzW%@-0SEdY{(%1eCC0OgRFUr=i1t@Z$@ljwQNE(0$oB^%I28@}55k|&*EfgU z@Xspr{0TWfe2J$IBp3(mp5KK2txzpF`|82=uE zrxB=O*8j^Lc)A0tIUfFeDnHlBpXb2ygYYQ5zd-fFAJFwztxJx|GyTWUUjgg;XRm#u z3haE0Z=(q|_8$XWs9}uvSP%OCt!W(4#d^20kAG{aT^{}Gkn%g*OOK zpg#T>j~=bBQ_A*dC4Uu{{0dfP`c(s@KRHN0;y>?a{v)n@rUvPU z%?2yqe?vZ)Fj!$2hW~Cv`wO$a=<#zU^*5cDAo{L+v7W^I0}?0?{mu1&unK~I0{jgPKWWS-T56fHpr$Gerrunj-`NjHx>K7ZA%2{a==xcsUKA)$9*T z{=(CyX?&Og^5GADxq(rh{ZZM!OZD*=*6Sb8i_&~Go$1X-`RUXB`l-C$51FRxWBEMI z{P6vc)#x9{hP7y7k;$;X@6nAKH`s`pTOSWKh4|0 zOrPP2bEv)o=9m11)f^)amQP^3W_bnuqpAN_V(Zw*+wvFoL-8U{Xu`>@a@|D zk#8zc9^;cg9)doe$LsU$vi0J>qE%P_%mnXFN6Ou z_#so!{<{70A2##lVDOW{e;EAG{io)qAO6GM{5%-^><#)4nf_u4bAJ`|$6v?H|AWC# zZ$^G9Jo)s?f5@ya28ExzO!~|g(tmw6>zDJ#4?i#LKR;%DbpH6^=ZpR4$E?53A3ywj zqyPMv_1*d7C+_KUKm3#I@eT7i<`2D|T#HeW$78(|V1H#=AKbyGGxd0q`F`A-7eM>? z`TryO{zS&p4Ce*f=#Ibp5Bs3~5lk_(eyzfJfgI-tFhA0G&FitPdcQ4jp5Wzo zLc#L{^D!ReIzLbae?RN|0M@fP&kxZ2_T2p>eO#D7FrNhd{|FvAKY-;2F7zj@KW%?v z$PfBAa()2PVSggBpONzeBj*RiUNHXBc)~Zt#vf$eoi&`x`xEF z^8@TJWcxpoM&|IZ)%e-{AF;tp7v=|jen1PD;5SJBi-P*!_xwN+KMyeRy~=MdDq06> z*7=j7LKWBga+39@$Ui^OkM)wbKkBdF&UJoZAvK5XwHN0BB5I}wlRrex^852AlM z9vGMZeLFw!d*{!s_jgD1?IN8Y80P-nG#KGB{@wobnGt=vNczV9eD-`w`(-)iQ;nZ} zK0WW}@ydhqX{=1vWBofH)n|WiWIjcQlfUPMIsZ9^?k8-0qy5kv`lj(OqP~rB^sQoG z`qpRvc0}JUsJ^kkv8`|HAC~>RvBtlM`Zmtdx3L4$w?6yxBl>nh_3aXOd}DvX?E0qh zFQUF(;^^D>f$3YH=Px7rc0u)x&s*8!8~cxD*EfxS5%uj-N8ct5OyBxEUmMZ43#xB; z-EeUI#{R|G^-benM18Ar^zE{N>06)Yrz84yLG^92t8a1AzwG*^@h_sjO?LDxJ}`YV z{+04Pdp}xBmwsMA=gCI)cOPnnJP@A<=LvrIctlfO{o?Zo+4W1~Uqtgd;$oca}F z?UDU9_%}@0ix*O8|Av)_pBPe4>;($#$DP#v40~~4#gC`(GkzU|=-(Zk#vPjd;a>~| z-amr#thK2PPCSKDE1m+*Ya?EQB8NowuPD{;1L**0-e*#qx+)M)p*B?)h>uX0EZIPh zHq7}F#2cvGK>Jk&M*S-T@e^uOpA5u5p!Z>Xf*j_E{}U1Kylsg&ZySi80RR2K`PpYF zK0@&M3Gz$WpWi#*KGAC=-{K8*0eI4el>*~=tHeK;Wcb1P;0>RY ze9)^+eZr52fcM+fCC6?s?+c;xU7$a613eG(<#QcMCT=#o^^hj^QUi! z2g=*$bE*8)4fgrmXnd`Qpndet;EU4EZ=8U*z$F`apk@Wgm#Qkk9dlMV{_? z>e^(X)xXrAG5`=l>BeqR6(#puVzxaR2-7x6%6M$2YO`6XQ4Z&5suW`|Ivp8fc%ybL98WSo$>G z$P?p(ufNZdJ}$BJD?;-0^-Jr+bWb19-fa9^@i(U1@)7-zOJrG^8y9)8Tz+2b$N}V2OgRqz%y#nd_Y*wcZjF7p_;Pw z{EYd$Y6BnW^WbS4W=KBxm-s^*&xiFvzaOiL`oq$XX{LV=pTnOYUgPzQACCy{C!u(v zBp;ctu-@VLJsdBR=YufsA6es170K6s-&8)|@B8_&X_PPVNijcgeWv~B&!TCzKDqIT z(0|&ztp4KzlV&|po0?Ul_&_wv7xPcSC^|~y)3@Fyh5ifUyg0c|OaF^KOMKP38upc%PK$uW!$Jf%&++YrH>Ra)0P}Nt4B1Q*&z4?C`GnM?4XUCu80( zb&30aDa#&6-z)t23HpWp^5ZF?y>-d!H_Wlx6Gpr#R!1Yh@~$y~@d@@hnfiCd-eWKrg$@=vy+yC`m|LgJSdM;qr z)6zdY-a_A@4|@4W^dL`n{e$;?`SJ#T(m&k4D3AAd`SDX&KG$vF14CAS2iKR@`UCjz zS18f(2zqopg)UV|w-bn`F-pC559sUliH!H%o#p=fzD`iQHXVQKIwLRecb(II-~OmR z++S$=h$jL5(LdLD^3n6XFAv}&9>GE{p2PDSt@s3Kj!#gVT3C~wKnel*LEdPO9UlYZ z&FT%jzhLAMkRRw*q^|YjQJ{QazCnI@SBcTb+j)IlvHDu=uSoGstnp~IX+P?pK&r^` zcXj^<+F#yPYSvF!Ux7aOS?$PcwI>hcgI=ZXA06KU@-y#C)A3OdZ)JrQ|AF3TR;=;a zf4?}OUz(nIe;S{cnqPX^+6v+KTLd~ z=zLvXpU(yUOpUMOqg143`u&IU6{~0Z{m1gM^bPuj@`hhMp8M}B^y1H$cr7NLOqZH& z@r(Xn;@1a#0KI@c0G{r(pY*9>iObIwc6%yP^L>3H{aQHR*B`)syqPr11M-`y`O)#W zDpKQo`=Roy$7Qh}JwAZHshU5D7ZhFO=@aCG{*v)S*OydPHhx8Fy5D}bZ++aEl3tx&*Pi-Nk+rTMQR1rZS21yHPw&zMErR1y*&Lh@yoP6)TWl!DBdt*`dF`D zS(C0Jg1SD$Tl3dDGa}%BjCKDL!T3_S;R-^h@dfgf^%Cz#0S?4l)#Cy0FL8WTz25fs zr+^1C7Blml^oB3Dv7qK3JzRG&e+MluZOKAQn4(MMv zIZE+cNY2glb7@Q&)>gs+lT(CO|7WWFAUJ@t$=<&o^{EJ z4QvmjkKn(rKFE4k+AHhXV0$b4^=pN*9`pIv`ds0y&oc7+$r^gSrq~nd1ILdddGL5} zsegYy#koI6d+U;A{{A@6m)Z22+C%#>dOyzHKPmIuQ+AH_lx1xX=M=z4`Y#kDh+tu*8E5tJgLX{Zg+zRDYMc-kVQoeo3n9b$Q4W{gYvSpFrcK z#EZjxXX3;0dWP-~R_pSR2bV8hj^n|5mG{10nIF@WB{xS2T(A%Ksyob<3{U#n@n&<1j@hV4OtJ{wpCjKGf7ml;*gMLW-P`-byONImS22tK!e@&qEf*)V5 z{3=6F;+L5DNym=_y)c;$&)2NKV^(VXG(SJ0d^x0t_v13!pg>D$}0fE#tV7m*+I+r(Sc>*IpdB{NNXxxWza zD?Coxr}e?yFYETBJ?8!0etT&7!}sT?ALW%DFA(xJ@AH=ReO{4g@1S3hs#;>#4}25v zRj)UxziIfF`+wg)v5evJQ>QxRC(wQ<<8yxWQe9t~+lTmu^NqYibpPbvKVVs-+dCo7 z<@I_V@e1{PL-Qf*8~hl1)Aj^?llybS&*bSIeb7TZ#wafb&HW>_r<&m-_ld_1pJ*fGZZ}1;=a9r~OeG-wIMQi#`5;Px>`IK;Ma%YR5y>^;PKcQO8Rq z{o?bAzC6wSBkK#rhpW)zG4~I_Wg2Gvqxa3vw)KJJF^Bn;`@7i^-&EUwmnzfs>-%l0 zf40THtB*vF_6sCFC-h6!!>oUlPy30!e$F-VQPW=jLe7`@XuiZ}75milyWf7J4>F#3 z@pO4Vryx~p;&FQS-xFy4&HQE5=K}x!4)QNZE$S0bbw0|Aeo}kr{)NX+=rio0z^wQ4 zNxvz+DX%vSzXho!CZ1}V+Y^YFOZ{!z!?U_P^`F#7?+c^*6B*B_ep;XV@?0kIQZfHi z`Mmrkc6mE~DYtLA$Vc-tB^b|?=+k;w<_C&jx-38s@c|_sD)`M5zm(SVK7V@u3hM*u z&**ZY$Lj~ZUqnfof2H@wNPd_PN&i*~KHxz7(nfy2;aZxX%=>;BKP>n!&!>1Pt5rH~ z&Zn>H8&9nt@k@{3(UCIe^VUshBI>xb|5-K4(|oPtv(WqQ(OXjANsd>d)RDNZAO0PP zUkU8V8s_+HSUo2U{0QY^^_4d8NrJIjIAY)#M30`ZXGikK^DoV>$>(^R;V`fFxqmhiJ-qSI(8KzfV17WQ z`!7T=KVS0tGaLrTx_<88a2Om1U|xS&_#`prm*e4^czGs&e>-)J@PF=jUx9dfZD=NUcB9m^sRWiB!9@K3gnFZArCA5F6N&=yj_gvIpghOeZcekacBNN zVqjjsS@Cu;|A&*St@)bb*A@BmGxRZ>TNy@KR**O7kn1sL&V^?EXE{NKtbt#_{1@p4iA`s;Q1G~W*( zzAmpvuGRH1KTiYSZqF+WYyJ>#*Tk1bBG+G;PxQ0L|BdvC|4R!L%@5=Wc~wEQc74R3 ziT@jw_{0+b7xiz@{A+ojKm7RYnx75CPl0JqL2{t+e=Ap0`L(7z#lt!V7>iu?uc3Ir zWS>6$D33Q-^s_t)lHt{cKZa|Ql|@8P#xs=X^;y)=Kfm#S&%GXp{QATLMoYZ*)~2lY zPvZUY>k4_a0D`{%LjUE64@~hTNpb1Y<2l7ErltG<;_>#051gj*rauDj!yGVwtr_|A z;|2FCKCfA?>irPXmmq)E`#1;8-#Z!k;~t>9g4g>MueVQpksyE8`#uNGU&i>o6<|&G zZ@=`fPdthsf7bg)2Z=w5w=e+y`ou2?@@KuzbddOC_e<{Ie#x)TdG{cH*855ai9dcm z-hcl3oR4lJ`IG$)INta>8TF0bA^OixpYyYW%}<=#*MENe^Ocytez$n+ndLKOVEmZ- zKYu(MZ243Vj30A;0%4@%-8Ujp&~UWqLkUsiL?aHSu)U6@HsQ zT;~1<@%$?RwKe+fKS)m;y!Fk40esvKReJkd9M69&zu#HEe{??f$1YhHM{&Mi!g_$> z<<}5so-fe-7VqDf`2H23NBciq9__77j@SF+A=)qhF~$2INBc+oeC!0>kBu9b<{L8P zgFfxw6L}rKnf6NxbOP0f{kaVB{>SO|>ivLq&pklJaq0D3-+2G|ir2H|`AImLr~Neu z#{SQ~&RCv0j{QEmDa8I!Z7S;DFM_{XdcQ4^H~y`)se%F<-q(ra!PubU^1L6U-QgI< zeh}_&V|2sa*7yrNPphKm^ZdS~w#Hxc`?$IOG9Xa>zfzgv@fUmLk5d1rsCPe${T{r3 z&FqJ0d4hh4#s_`cFYD8uyu1Xe;`ZSA-!P2-&-*~miK=)!Jab4|BrG1YklVZBl48g@f%Sep3iCZp#H@Bzd$eQ^~Y=UzBNm~aQ{&| zCF(y0YRXPEe9eYNQ0e(T)y(_oY?*bnb_f`8FRy`MUk_k#`n z-@QFex;j%|ruNYI#{KuP+Os}DpZg<@BxBzgzt~?`=>P8R;n5KlX!#L+8h+b0ne!1| zf06x@J?8u43;YrHErYy?1dadDms&hc*ZX(co+qW!$N;@p(ZA3y(hs69{)xp4E&am& zeC<-J{~^CgOSL+-ooNe^J_qcb^bLzOo=?npNaH*Gl=-Ioc8uMCt@!OKqT$^KV;pXr3V=Qf4^D^`Yb=#AKib}kV4b^brSYDGfkPe03)BCcdsFk$MbhMp8F~f zruiht`v72nef^QG^WiU+hH<`<GclV=N~1VSRJXSNwkJ+sOL?M)d7M z#%mvWKfs8-T}XW!c|X92zFkOt8+kv#h`wD&eH(c{z=*zGNPQc5Kfs8-T}XY?@&0N5 z{0%;T&*4*Y#B1017crjpg?>LkuKnEH`)?!rA#^^J&YSW1Q0$Lmzxwz0{Q!90LC+7) z^*)LbeS4@C``;t)3((K6VpHV#)ocBF|8O7w#1VbN0CeG=XK{R7`~6PUs6NMhs_`%4 zd^#=oyfjulFn#Oee>$RX7gXOkzPGJ!m*>_ujeimKZMvgxmk&(e`uG=*=-UO=H;(6T z>)VXn`lj(OqQ1>^^liq#^sSHo_=vt;P<`X~JJ|X*E4RLB{EMh>vmJe#H86eab3R~1 z-!7=W&5e?NS^FV#a_gJMzli!a*U`5*1Jk!Y=RZdD?Skstd{^H%9$EJH*=hWXsBiNf zeVaEhed}|+XGGsFsJ<<7^^ML$<+wl9_!m*%7CHL1aA5k@=ls@)zFkm#t9A8_<1uC* z-!%S3)VEqk-xd!{-};=-8_~B5s&7kNedBnT+4W81UqpRd;^^BI1Jk!Y=PyU}?SkqX zzaPfFU!e0MIp$N1e-ZUf3Th->w{( zzV$gjJ)&F))2I@fpqe+QPv+k2T^ClEEKjrK?{Y zpF8__rtvSLeyw!$>ytV4D+Fu7MdwA~{}PDr41W-M9})S-==g~%$v=kAPw{#J=V3P# za%ue0X9%w-?Qc@5 z68B2hce^8+y5u3iUaz`_|Wz$BQM5}=zy1`uY^Cv zgE_wWoqM5wH;FX22qjVVRIZ!E!i zWVN2{o!%#mcmaT~=6Ll4BR&Dn#}`n10e-*m(T&KD6i~dEh@XEomFM&EXdm8}rsZYg zqm*~a`@C=-9`Q=byX5_F`u*d0Uk34qBdlqTXZzJoRD~-x0k0ha{~KgHGvnvF&JX`O_u3i4G@BH6?(|cleTkp?r z`|baH4@=v(KKwBTd<0pV{cpcq@#A0P_tQW9=}&+A%1W%8U;C$?8!wyw>U$COnZKSj zHZQI!-dm|Qedf{CpH|`c7xCsJFS$xRGV3!B%}}9xmlizOv$ywOf4%qbcYpZu-+!g| zr>9SU*!%R;y}h6O_YXn*ws-d8>mF1*?b>e~_~|cK{1l13|KMFefA7OzDwX)koO}My z7c0J!rMB~#Y`3KH!|#0g*YEuME02Bn%8#Eu{qv{4yZocQKm6Bszx(eWw*F1;hac`e z{oUVw^z`2EzVgm5m-K$;!ykX;=YP|?w|DP9yz^1-X*BA++o)-O^xo4S_4bzb_I~&` z)z^N5{!du(<6qvilJefU<)@E-{=2un^)J76u77yE_W`But@`1%&Ik9saz2|oq-clI zk8U1!SGY{w`ODtF|L%8xd+U3>d*A&$RC_^K#Z@T8ulggq1`IHy*Jc`ipAsmKuF$Ax z(fho#S;YaC!c$7}-^Z({lv))^=oKL2(Vrdo`1su+3K$j|RZ{dDypo(&UbLaTx_EWI zj}s~m0Vg~qmj9b9G;zaY(R`+SP5?jDhleKMQyr^{6H)naoikOG+7WucTctyZkZ#9j zVBewKc2F2Ap%I#`iVC^?q4z^75t4MG9comNjemug#vz+kkX?epV1d)%1ql|dDqa)S z?QKM6K_C9(NY(kwTiE5TGLRqXQy$rhc=Ki2S4LZbL+#QLvq&OJeL55>45Kw*-dw91 z!$s8EV5^RB&@REDJL+J*dFzuwpe6CST0?cn=UkFuemrdnMO^7@Jx6`SmS{z`Z<0)< z>GAFmYyohYfmel&u;gN*5utP+Nru+_;dpH^C<|R9Irf#Vi$yIXj_hyyx0`3f^xLH; z`F(39s{KuMsZv?_tSUW0ehxHW2y>=38rXxJjMd1pJXruLvaK6FFWwkP%s|@RsesK z1K%joAr1Ybh^99kzkds+slFoyW*z~%_=64a06%2>d<}Me#{xEo@hvpa8c4?mS_b$c zRa%ksk!lv2Im;OBa;5*I(_3vS63{lhOFIDX2OQMCh@*Y_-8R1V1@T{Y$}bHC^Aird z$$?$|sm(Lc^|qV{7VVRKTRSu5bNOLqlXD;iyVV4XR9q@BPW7I`^Y8U+lLXr~nJItx+2rj`s|gmV47Lrr^oFlZk`!&5^rV`@ zCbO2wVUr%+A!?I%IC3Ibw2y3(VB01$>ZF0XOCxS)$$R-K4Z8B5-@UzJSPOAwPsSLIay7Y#xO_CICoAjib!zQzq z$zhWo-63j|cRO+-ShSC9l3?2=GvyCIo4n6yHNhek*M*so>X(#WY#h{Y|^7U zL~ZgRM@|Hb_K{5zY};g}{601rMj`FVX?c4p*V~hD&X->>^>XkOcRFy&yV$FHHGbvx z4JKMlk56fD2L2u4-_=K$G^)QJsh_^3jQyL^1eR~XlW$2UA7AnLT~*=xHBsR`Q&eI%On)hZIhGseSNsK zkGAwjD6HG=@dG;~dQ!gI_=HQ_SDAdlcYU@6mhyw>!_844K8^ZG2Db++iz4I&EiVzs zccchQI=?>PcCU*joR9DR(qyf3e9g7ieHvPKM<1SDiQ;dO4Xn z9!b^XVSnY2&C6djhIB(d;_**;5>dkbEgGIF&&G3qf827}>iNBmeu2o0fwdUH7=t#kHZqhKH|Bk`!&5^rV`@CbO2wVUr%+A!?Ho961pz z+DA4?ux*o>@`s;IUh1@(V3EpT+n`Ht_}U~%(Y8rXsyS>jYndE2>CqjcHhGyNCxS)$ z$R-K4Z8B5-@UzJ&POAwPskk;Y*zoYRNs^*%lb%#_*ksl+Ic(CSJ49`Asv{?YMf=Dm z3ASxAQ~vO?$>~n32^Oggwhg-UhObSM6m6UIq?*GfvzEzWlOEk6YLhb@IT0+{M>a{Y zZIhYuho4Q(aav8VNX50G!G?#gO_CICoAjib!zQzq$zhWo-63j|a~(MmEZRplNw95` znevC9O)hX+O|VF1ux-$#H+*f9q-fiuC)FG_nYBy~oAl@oQJY-o$cbRlKC(%IZJW%L zKm2U+3a8Zsi&R`28fU+lLXr~nJItx z+2nGk)dY)F2HOT*dc)TyNs6{jdQ#0{lUd8;ut|^Z5Vgq_j+_VplCO0~*CRn60P#YU4{_wR)lA>*so>X(#WY#h{Y|^7UL~U}DBPW7I`^Y8#s-Q%d~K4XXxpSG)f_gNwM-71^ym&zoBWI;CxS)$$R-K4Z8B5- z@UzMPa?4HSR)+9XNQwn71GTY%;tyY&Bq`c9=}9$*O=c~V!zMktL)0dl961pz+DA4?ux*o>@`s;I z?r>U7ut;T~Ha1ZF;cJs5McXDlsphcBtYvc8q(^s%+T>10P6Uhgkxde8+hnHv;b)V( zoK_PoQW>a?4HSR)+9XNQwnU+lLXr~nJItx*<{jb zHNhg4f!f$W@rSQXk`!&5^rV`@CbO2wVUr%+A!?KDj+_VL#~rvP0MAfmDrVu&P?ZiGcir3VkM~ziSZ3o|+o3)8F@AtH0Z}>FqnD6u}Ar+~QgrYO|dA&wGcCoJLjD6}L-W6~R9oaTP zsj^Qf^#nefCPEflkw#sD&%Ha9s`&!&@TmzyQY9gEFFrM;A$1I&wlaMFC8Rd}Ye*f$ z=L|lHp8*G-Gx%)!|B#N)8GLGf4qSYmz~>A;HIPa8I_SZqVpa41A`fsJ8W}AsR%uj4 z{X(DUW#}|>k-X^VN9OxO^Zfz74;VC9gF?4@*BbNdDx?QO-g_9qA2R_)pq1{mjMMv4 z_&lex$p(tt2!0|8LUykGoIcJ;RqA1+SCsj5#5x0Xnfs<^!P!t#NhQ{7W^RA zz#-}jvT^rHn1pSxDf=RP`x*rxQ~7@~<^ISKty_K!$_Gb7O}(yEf5%J#!lAl^9gz_A zhf9-6ZYra{ibkQp<>HcF;1w2h1=x7fls~NVyuZ>> zUrU)VP5tHn-Dk>In7qWguVzy|f$7dpf7X;2{iX7x(r)?1Fy6Z|)IZ}f z&1u{ab?X;Bx(?-UNBM+Pf1N2W`X}{EIZ2<0giPhPW++dyj$6JK37B=-(9MDNr%ZXn zzW3$I{|1!*FB#e|<hMn!Z8alS^Xl{ykpviGcZrvSkNZ!0>SM%-;hNsq0 zs=nsLn^^+D*DXQvXB%4fh_FBp{NKFW%a1oTbTq_uVf?kqa|R<*O;;_BH+FV32ZRZH zi$7hTPl{<;^maGx+;!C=PWAs-`AyBY?%dW~PtCk)+n%Ndw|o=ln69AAje9!ct=r>` zt$TJi#R2OsoY%gip{<$nKUed4N)B9BeI2dw#%A<^?uK}>VfW3=@z2d&xM1;;#Y-1m zv3Tj6xc7Sr&10Wipy_>Xq5ht_AkcoL8vt@m&8fJlHCP{aW!&(o(dRL`$D@VM@#D*=%v?q5AI0rY_fXIDzDp*3Z^cEPVMRN%GwnM@dNy`oFNMv-PRw_&wyW( z3m^Tl;0n!);d?3+#^M33RSFikhb&?0V!vt`oI_vUjQje_LX(QCuswnC(MM6$*6%s^ zZ53A!(*R7@mf}kje;mF)`rq>ZB=)Dpe^K)1n(`!MH8Fzzvs8Z9m2CQpsHU|~((qkZ zWPqDWjum_KiXJHbvy%U-?7vq$IQB2b|2gqSbiZd%=}X_;TM1cy{sjj3sQLxp*SRQ> z;8L5jug2f7l`gWVjJPl@J_kaa(||9?@6QZs={bY#d8x&7RvA2*~t`JhaMXCf}{ z{dt`6AUNmxF{&Z@ZM>KZ$|KYLrKUqOR`Am4JR=1&07bl6*3Z)TGQ0Tc2~7MU3tuoa zzI3vW{~X#Sy6C4W^(C|`Q~FsTWlE<*44Kj!vZUXgrQW?+=vRSG1>_t@`{)vVeI36u z)%!oPq(6~`pRZ)$r!`CZ9qcn>qhfq zBGSi5BEBCqzllCxE$Lkc%`c*Fr`Lf`x`>Xs|Kz3Ryyu8 zyz8Kqeu^c`X&c|pTXba;7j_A z4*vZH|0V}t(swxc!e@(vFX?R#e%X_{y&Vp|q^BHw;j_!Zm-MuQFMRHE@Fo3T2VeNS z-@%vk2ONCi^Faq+(hoZL!so*dzNA0m;0vFJ9ehduo`WxZ9&zv`{aFWJ_&n<1OZp2A zzVLa>!I$)(IQYWn%MQMz|IEP`yEyLPOZsaLew)#slMcS5pK%sVv8PsGvIbi5V&0r_viKNF&y5K)(@QWRMNgwOrA2#@v4!)$v9ein5wSzC|GaP(r z*Bl35(rX-iY1d*0U(%O4_|mQw4!)$XaqvGd?X7e0C4G~FFYVgm;7j_A4!*SOCI?^A zw>tRJt|kXx(swxc(ykTflc@_%As4l77s=mv;Tc!I$)(Ir!49;|{*0zvXF z`yKov2LAyEU(z3R@P*HV4!)#6?BL5d^N52l>4zQsM+`qtIQWwOq=SFh%%|UT@Fo3- zgD?I5tb;Gx}f;ESG2a_}WR?%>Nj zRqfzQ`V0qOf24*tsqf2@Nq>609M znFr$zzNF7^@Z*1{@7Lxy_>x}Z;7hv}JNS~m)WMf_t#I%qJ>lR>yVf}PlD^)-_wJ(| zd`Z92!QW)sdy|7N>02FqX;+hjFX=lRd}&vUgD>fA4!*Rj!@-yIE(c%MX=w*v()T&| z!smVmU(yda_`>JC4!)${@8CBXc|PFaOZq_vU-khWcJL+r5eHw!jl&MUq(9-{%ee7v z2Vc^kbns=|_@09==|>%W*;jwT!I$)74t|?iFa5;9m-Lq%d}-Iu9DGSX?%=l>`M>7i zOZpiHU;6!54!)$n@8BOW^UDVgzNE_oZ5iM88GfSXLAQ*Dk{&bZ9=?%Jv5Rlg$2$1Z zu1W`A(yJYO>6aM}zNF7_@TFgB9DGS%?BGkkEOqcDeT9R6z{n@z;7j@%2VeBC&cT=T z^$vcAp})z&m-L$)eChYC4!)!}Ir!4=I~;sTZ*lOY-`gB~N$+s+O~0pHe3Rbg;EUYS z4!)%CbMQrO`yG5qKj7jU{lC}6H|dW!_@Y;b9ehcD!oe54`nH2F=}$WNqF3K@@Fo3- zgMYt~+p`Y7r2oXh7d~Hh@Fo3c4!-bt+`*Uh*BpG|^Q40>>2Esta^HQ%!I$(89DL!E zovp3?E=i9XzY`gMgwL2sm+?x{iyeI7v&_Ml^sx@U%sZ71zNAlb@I?>f4!)#UJNO4p z|ITpmC4Id~UxHtVhZ|>oLo(UWsp`p9aec#{-P?9td=RCUVWbu_0s6kJUAbTrq) zbq)D&^t@$pbMu1QrQ4S^Em_*UaQoth?F$!fYgoFiY5T(3nnksZ+nFTrcXp^PpWU!_ z?X~RadP8%^=7yc^%}r~Qt-G$ip(DAeA=%Nc<~6pr=R{LiqxLm7tlzw)UQ_c3T@B8) z53Q+Y_pR%iJ3zU)VV7yw&c-BKja1z>HE(xoNAtXOpZd()_Kt>ax1enux9+OnzJ2Gp zTcs(kg`mK|st)+~HG(lp)wEGbAhoGs4LgJe(zW=Tchqku=U%mb{bx6<+1bznm){Mo zO?z6J74=4alTKn0*0<<3)VJ>5hS~<)7xnc`9j!^-B|>-4ZukQ?{jQXmRVLKYTCbf! zwTvJi-E<8?05+(lWy^4FYtkT8H>r8;9m$S{Mtpbf;_qZDIr3^!>$NP_gZnj{_%XyG z9bVtw+)>|5F4*Xd)?M}Rx9(`KUkc6V3g>NW?|?7oEyjHGH&xft(B58ezz%k{PIlC{ zwCt*fn78eyC->##PVzWzXxY`;)J%Tb^|vnMpY6L~6uTNy^}81NzZTT*zIA)s;`;jb zUG)oUmOvxh7MfajEdrCI9Q7@&pwiUdQhx)(>jAE5Z@HeCt8Z=m0^u}tpe!@g){)%4 zXkq=$yP9{^x7OFom|}*AkM%|8P>hRJp!|`yF`(hf7Y8+yt5XetudDE(G~@Q;WhBh6 zE;F7=Mw$FD-!x8A6$t~&SKAA^rX!Pe3QdUHZASiYYD zQ7TWg?dzB+e~($0N}iMxn0(&o$eJs|tw%Ng!mr@FFwx7@{u(2HX(zRxWGv++{UMX) z*1yBl@A50k+fozzGXIl>tk^u>UgkUo(T2g~#T2h~u zWLk!t2$6M+!C=PB#y%To$i$Oz%aCz)GUI4AXCrTXh~r=eYqD9NOwI|B8Abx^!5yyy ziE;1u*Q5L1s|Eq~#E+p4t$XYL|El_5RsUbr&Ane(yO9 z2>*5xyYl`fJ^Ws^Eqv9+4*x$J8#vm<4ukecUluF=Uan{G{WA8(uXJ{Jc6wj!8Qb)M z5!+Pi>quWFEB?Q_XK)nlbj|igBgP&(`NVM5xdX#H-g#np&8a7bzo~n!fc8mz4$nXF z?a09KW041)--#SMa6B@$=tN{-(aFgEb*Ca1_n(T4?SCio_@cKX_qLyl96R}TByG<9 z;jtYDh97_8iQ)7`_l4*CzY{sX@K|KuJ3oy)?tDGccKl*wV9h(AD-MSb9vpt?_<`Yl zi;hLAP92Z5b)1R}9C$r4cJIZ=`5hNQvn*WcJUD!Q(WArr)}f#M--*m`$M`2-k5rz! z7PYj1o9*>;I zr|q5BBMVNPh*Ub?iJV__4m2ym^V=UC9)K6I5zouR z;o}E}7o2!>I1HZ8FZy<5-=f#S7xVzx2k=Rq|AlZRbb0<>&_Hi#$={BI-+n!kw(Ue@ z{<#a_>q2DGoCm^VgkJ?-A3PCBO*w~le}Z=VhbsZwmwX~Jb`M}$?&Y5(f3&^J^uF3A_|!DbbLEU}+5q9U>Z~|sf8d1ur@_C34{X&~ z@vHCwO(Wd2ff3#DAhkQnSn@1z^uTK1qffEH=s&Z;i!bR{-YR2*XYXKF{$(i}JeR?)oKIm_(m-2+ zxi-2tX>jzP*x=~h+TjcTQ@?WIEp~YHZ<4~J-(rU^G-FMj!Uo5FG##+y@aWy_@W0*- z+P_W-pV_qM0h(*z`S8_UFPr9)Sl@*_7r;NsG74U4%(qD<$aoPlIw2FumO71H`3cQc z$Y+#`rLwCY$aDU0R}PLrzDgy@C_l+C_EK_q4Dvu1gXgEP!xy2Gu@_RpOzTT1mxFn(%*Xht8XKf_y#wbv-^+xbMUD0ItXJ`QNxd zh3mi3_^{VoPbGzIu&eSv+hlg-E$H7hM-RJTp9}C?veU0&1G4#f*!;{=HaG(NksXvz zzq^4AM#|VA(}x})n=1QX`W9>jUw#{Y{0?mPZ?Iq7AK2kbhhSHO9b0D!eUP7iSX~mn zno@e1>~~uqW_>&j=Yb#d5B<5>dd=H)|DZ7!yVQ(MP;{ng6yQU$PO#PZIMDeHTs`=t z;FHN*H0-#t!x!xE`MX#RF!_YlG4u3vTy>4vCfhXK#+ZE)mONQ(PG-mLmqlJ%q7XX! zUqU8HwzWyzocYhUzczNYcC>`p;*QR4 z-{Ovk7kBSy>)6&AuQ&I|`ooK3H_e^x0dF^3-0cf>FBYOTHurS5WV%4rov@igSKzmHLi^agCO7I$vj*5&IiV2zC( z4>!gzYgevq6qv>WY}MMlT|9cn@YE9d%jYU5yIy3P52OvdWouc%;mKAo`%?~|JPzJ~ z&AN^6*v+WNI2jF(-2yN_nX;1bNfF6q_KivGKR z+i-qiEm4$w38Z@};CnY+Gt&3;_040B=$^iL4t~oR!53OPk#)th{p$dTw)WF>2trkS zj^IP}b@i3yl_l;n?Ei6&A_^Un5b1sRY$H&hI|N@W>uK@2>2#sZ@9l2u>~ME?cKW*( zx-qkT-Y%cJaOvV@3*Gf}SzNq=mG-pxTiji}UERKRcY9A)x4Xr+t*yi7-dJ6|WR#jxHn+OF+HoFmzt@eSiB~#b!S^2)UU#rZ0k=h#})nSGlbc0t?w?V|5HXmkb-#L3X5OlFsLJ6ZZtNvv+8 z(G7K-xlMICt1{xLPpaF{t?kYz)7RjbsnMp{=cjX{q`uk1OPGJ;$(aFCENM9n{(6IF*o~BuaJQ~20Z%f4`Cun5RJdG zS7%#^7roCW__+fWlsWjEb+W@jyB#f9bm(9P8(G5gL$*6S{$du@52mnJ53#wSB-?Lh zytaeaM{@1OA>BrJ#t0AjLQ|u;s3lE@wlUt^P>LR-Kas&QgHw~HQvbt;QrKCdXPiET z=z-4<*{9N3r|6&P`7{&s=pXv9kvM^x_c;Igkk3F5eln!Jzz_N(dal2$ZjJasT_^bh zH4z)}n-ZD^D{}iLDZHSMFv7cn&S}JN`k8178_rM^O zEK2sUJ7CX|YIfHyqNM`)E@rT=^iGa)a}Yk?3H|7k>+9<3xV*3BLf+{*`BKyY{h9&J%g%D`;hbF3JPDh{Tli}|9a5IoP;MOkbOgQ`0&vo z@IR7aPbYuILK)Z!{Q#d8%AZ5)Cj5)H&kw;IqjlPJl5aSdRXlS9SSlF*Jkj4X=W;Zv z#p&leNqi z=!}+t-VS=KZAAaPljH5E3w#;yZs6S}yc>0aUkUt7;0pwPRJ5Ooy1;)G_-x=k0zWT< z_n(crz^7_^dR@R{5vB241>S|az$fWKo~;s}EBW6l@fz^Ne?Z`ymcYO6xrgNA{!jiD zve~#VDSr#ipHJe%@pI?K>aXzm!tKH5AKH_??C78K^QKL&$a$L4x*Cn{<@V2pePadm zC8|#LhJRo_=kxhW^Z6GUtn?X8kdyxcU-8V(WOGp;-=CQt{+dtrXd(qy7_R8PZk$z{oO#bEo-Yx9;{49=l zpf1{zKiYwp{s#Yse&@vWtL#k-kiFj|ebYUP)HjW1SN=It2K{skyqwSD^)Jz~J@h|^ z)UEu3^v}PMB0R>ccxJh&zy4iWaeX&+SoBTv+n{egU$eNrv!wpu|5-78H|1iza9Q;2 za>eysP5N@h=Qp1pXfO4R{`zAL?h^P8a;Q~U|oU@}_2Y5wRuUh0I$j~o6>@%JL=5Ia>L` zB(;B!N&ZbN+6b+Rvy@5q1P z&-m(?{JAfM*4K(uC4jJ}+5brYHe-CYAKQt1L&?`f`HsW+tE=x%75NU?FD=T>VnFV{ zWS{H%C;Xe|iw>Dzl03Yq{3B>{@b&qBsQf9__9m7yH6GS8C$2o_{Tn)oK%#5V1AxKmFx@qBj3*^cNFoE>^~!J zfB3(7elFqk+@3TipwHsaOr4fnM*2Jb7qouOOlFs&Jii&41^-ITj``Qf66^=^vLQYS z=tpV4;o$z08rz@bb4^NrJUf3!JZ9pR4{pW4U#@%;?;WP~GG^h`8^ z_$%%Qy=b38`3~}F+t8!nm9K9ZF63wKSbvZ|GdN410s5#5^Ley7Ju{fA-%0i9m_QgG z5BAL_cL~sZfV{<_?Ce-Pv+_gy_hYG$js)R8Vwha}H=+P^uH;`$~3NKLZnH=CBKK$D0E;1}9Z^RpWBI|+3tJU0K6 zGvoLS;8RdH;$1lPQ0)nyiMkP=HN}jdf_fYedAYxGeL!BSyTF0aj}7xVe}a6RPc9K( z&R3h`*>lJrce8ZzZ}`(B_F6UhQ#!Q>ln3l0o_rBCDu(r96jdG%W|95UpGH;0bJBM* zwj;)Po}5hmt9YIi60?cbVie>LdwLlfpwRgU_Fsp3vDAv9)BYU$V{wQ{`@`3`zms;N zfl|4ir7n}?g1^k5o=ztw(LS>OA#GJGUl918hPX2@K0f@bAN**??*4#x!s|(l^Fx4F zF<*8&a8!U|!+LQN@Qf+Fo_J*go4L(4%TZdCnDZMf2Q>u>$7S7Mm-1Xb&MYEu^z?uTO*oz zy~cbcd~Cg*Fkj*G+mf%)W}J%fN4}p>_J=)j{+K--*r*Hh{nW>puaG|^%vWYlkgvFB zUq4^T;rWU?alQh1a@-W(WBE$RkZ+97R}fF#Ik9|Y6#Dz<`A80}uRLC&KTAG>cxlQ< z92jr58~lERd?ZIK_!OVVn`){mo=Kb~(e7`W!etS0;N3BmhKGTsJeP5JE@#GFZ-=|N7 zlp`4~Q+&euY1&_Jqxh7}_vbvGAwSC8ZRhzD_5&I0RphU`laeW(r%cUoP(D-~vQxND z;`uJtKXG1kbQ4Py&k!#%|DE^Wruc5Iz>qz<(y<8SRIn*&@HY`FLj9 zZL0kB@Jr^Pl|Coii-)Eq?xE-QQyV z-oSo)hB3YshoE&i9K+=KK};ZN>PZxHxD8)nm2kQ^MFXrdH;ylA}9)1)=H*g;QJkG;G*_Zq- zj#XMSEc@#W9s7OuH4t!r#{ToV{Nej>_=7lK&!O*6`S+1JePNzVb@02^vft+WwU~TP zM)L9VMG{i1uXb_}-hMQb`lqj3WPe)x{J-GF%4w>5BE10TBv=}i*Igf8KMI$qcCf0n`fBxl<1u)Sq$ueJh#Tx(*Wj~6}9nXIu|0Bo8^B;GNk2Y1?} zWVELYK+_iT{?vFcC)rUiO)pKRPf5DRu2pQAE_eYoS|~V;aha6@QZjCg8tVC8;)#!O z`DHLdebGm>@`h$Cj^CSR#Z$VYFLxGQW?S)n5`L$eh6J%+O9soak_aDDtSfPQ z%H?@+C&nZB7X=I?>YwNOMBzz{7r7SjP)wrsG5)v4b+2rYh(0DyfU)(Oma0S3F&KxC zJ*kME4LVKnSp0!gaa)p2_Kve9L}uJ>2EFSkLhmuTX`}@BnB1q}x-O~c zNJ}(hg`P^$Mq}2|Y5+CpaU7nwAswvO^|kk4?zy=K_0#68U-Q_2RmI4UfR!}qX?mU` z`FDJ!(8hTejjdJ`>xvQQ=~^08{U%<4;_dh9R+(LfaTD1#=9!>NgjZIee#T2}79x+weDiBd(JuEhhKHT-Lc^gTUf@4ec#oxCW2w`4)IVP$(0_wC(IyZd|GIVZh?{NmQRz$4 z%k4MYN1`@2!ux5yqE0m0jr258A69&#K0i5btlUjhH_pU<8H2c&3AZ9Yzph^;YOf{< zO|1#ro93mS5}TK9!GN`WB5hk~KJ9g*&h!K^$jW~rdX)!A9S4w>n%6FK{W0K`S;X39 zv#eeG*TROi%S`{KrTubb&Bp#bmU@k)j%@YXYnO4g7rr-9e7jyWAjgqj@_lC4XQ9di zDzthU^;|=p@Koo@)F!1P!=oL5O`8oi)@yp63$fX=hjgqVFk_|1pZKnTayZ$mTlIoE z^*S4M>WXwtqvol4e4k;{X_S)mjrJ1P!)Z4*8s#Dr=_XOGLY&@AF5m{c1|91NWq`yB zX@qnzTaQlIKxBM(c|a3?nn^$Zn7r@V8?<^Z_3;t7iMBIOJ z+*s{7QQbJhZpU!3hD@ZhkBT?y{B{gCui_$BsQ55G?$91H@kyFxb=R${KgA2fPEOM< zBa2ZxIT>-)xF!qzLE;jP9FI-5r0a9PH|h31^ba#v6RqwyJO6Jz8hjk1Te(ft=dE*S z?fnm3H_g=K@u!7WeiG4J_Cuz8Xk7c)o|DhPq}U2V%MXf`{xgL$Lzh#81>YweM8sRm1D2X z_}RC18Qw>S;n;1fGT{jPyAqH14vUUtE$8UM$4&Sl@%uQj{g3xQ{$!j>VCY}hc>jHh zH~QU~v44qIX#U^a*6UiO3+D?opF}xr$AL|PXBcVMeTLC8)!cIJTP*Z@22#86pXv(oVUH^Cgy;XfxQel@75Km9_+0DxKcpbC1zi(8+lS!4q(3M&oCL*w zT8FyQ?gN1HgLzJM?LYH*AN**@M6`m#$M+zf)5YD|L@%Dxm1TlaJrwbu6d&ybcgF46C5`cIlD4%vX#I~q%|qclYJu=YvqYUG4(rHY|gX~{3- zcj976m9*cN?T;DS(ydkR74kgd-P*pAkob9~xWxZ{MYhkEAf;n# zd)m)EnxOsjmiDS=Yx~(kApwsfkk^y4{d=-bViTXDc+1_IUTAU0xT3?9&QVSg(;m^y@3>_!3 zUjUAY|6fP;lO8YHo%v&NU3Qna=HNo|n}|^VPXN;RC??d@%>qar8;hikfi|N3-UQ=+ zMYhkElr%mSi~5*Pw)a@Ze_pm%{!8u2rLFCkVe&qnVEnId0Wpf3|7FZNzVf3gwC_iI zk7fKJ*|)jb2DJaL3FKGp zFUgj0Q*M9j_~a9$)2&UC&~nL^1FEZPPvBc^$D_EMpO)cb^Ht4T<9NiYnitk~s_u~v zVr<7jay}_Kbyf3AT~)j7B^&CyN>-MLQnt3V$luevwA84v(8O8gtWn_7A{H`Eq3 zw{NQ3QM|s>TT-^RtiH6hxqVglmWEn?b6~?pf91|q%|)$%ZKxxBxqrp_%?(wpO`GZq zwlq|&@op;K;jgN$+px1@m4DTPb*+Kg4gS?dbq`kVtZXQ%^>1lkRoJvzv@P|o@HaJ- z7WlUCe%CYBt)jd?Y?xB{3+OEwF>mFItzOJyPqOPZ9wSOnZ=l~xT z^%eC!%|$DEX}p$ARh^}kJIkAv)))9IwpYc*t=PWKTe6~aWySW5lBQyFM}2$4()x#P zh_0w`BDz&u+C!~dmR3U_?G+^Rwk@Q)!p`!d?d!dz>k2owZ(sRPNp0`uO~t{4V^;2L z*bKg#Dpozx+*{lRdHvAUBTZ$+1-^#TKvTO9_NlLG>RA8CTC&0F`ZjOLhP9;~>k69N z>sy<))OMC__SV$}YFGHnT06@(ZLjf`maSb?v954S#mb(|?f#xE75FS&+p}eLRiLH4 zz8CtdTwhrKNXzOf|K?3auy<|Y=8g?@ty_e>9xAQchQ1408mfA%cJY=49@*Ts>;dI( z>e|@uZwof=C@#2vd0|UIp>M^C;uQtn<;}~Bmo057D)ioWe^Ft}eZ`G}uDNZSx7l~C zSFCL0?^dC=s^D!Ww^G5YQLg!ll)|OUmMvRaP_UfStn&K1e2wC7Ssh*|2 z2MUMlNp)|bn+f#`EP(ub#J{shqiY1t~!!1q2+_)OfVsbQg$3gENJ4V2Ljk4E_HT-^T@*i zDn3vp(es`+@~+V+K}qzSX+h5;;KT`$Y7#zexnY{~GWg08adz z=5}CA%`5F1{6B-li|A{l30{$M(3pwFeND>imL^d1s}1zj&m{?0ZHpvaZBol+|9P^X z8fh}6=f9WyqtZ#D%?qE5R>IxLv}Pd{fY$@*P4=mH25Ec!7k^09>^t&=)2APQZ;@#@zR<6jq-!MH+E0Uo&$9HR^hNy`Pl@^r{nK+!RC(F} zH_{)w2K=%CZj?7)>fOrcJPEhTJ0uNcmG@N%xAyZh3Af7oOG7_K{y#9ljq+y8`9tFx zV`RZK;725!_%W_#PfPf{7Wj~X-e})fB;3kBJw=BK@iRl*^1sU@T#c)Iu~ov=xC(y6 zKyMuPX$iNE`%4M8j+-v$EsblOFD?lmH(w;2`cb@)09y@kdBnrzI%0sw{&Np7%j956 zUN@&^>*XM~zz`$6aH98^;YvxEfc{@0M`uxZjg- z>$sG+P@&v~rj*hh#WgAl?%_4YTIE&2Ror;m0$1^2$O2b!VAKLvORqZrRQ;>a>x2B2lPmE+X1v{zzvGEs z*T`;k=RqAcwso}OHIGeQT{qXPqo-Xk)aL7A6K`_l)`7oXa$@R-eZelg8bXTN))a)? z6~1mEE?!(h@3E{{SGNf7lWX1~s8&HA8+_d*-FW|GQ%|>#pn@I^P0$u^x$oh&W}gWz zwC8J;ue+O=#cddmUSHAJL9fSKv!;G+d7IboYwlj#+0x_3@He>o<2LYjdRrR1I~!X# zV?rRXC-@XqA&XJMRh>aaaI<5v{FWQAnK!t-tFbHKYaahz(O_qbx7*7W^S{C&g_YJo zl5&m`B#WE7x(n~SVQqXy_QE%A+=MSoy_F5Tsm({<&5QZirQY588GW;>SYSWm-@OPT zGSxEmbXg{-cgp-jA>}fys*Lve1e0k&4DOM4r)3;d2o7u`Q@wk+q*q|qQS=;}Ai&YLy literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k52af4c03f6b8e5990e9d227e1421165e.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k52af4c03f6b8e5990e9d227e1421165e.o new file mode 100644 index 0000000000000000000000000000000000000000..7dc058528f890f3e8919fd1d4da050fa3c9e027f GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Kp1A?+13jo8ITZ&*Jf6oAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi80x6|-unN)s{U8i|5tVE_Ls|6F4r}UsiLu0SgfX?jAi!5?mhh0 z!%{)%W<50nwz{8ZyKVo6UTFKw48{%**|PiI!RwqYd+xvBdLFNT#XXu%RJ3%xw(AF> z-;QHfKir^)KB%&VuG!e(|7T-;M?2VI&>rc@V1?hy_Vj;P!ruCo&JNE=?Ws9yo7^{K zn@D{f=}BjW|5x|)52Kx~+1{$d*dwQ&9;`fnU~tEKPYx%zQXB zvg5$uQ%^rVm^yz+Xim#_!WZTp5AS>Lr{SlZZ-$#sTnhKCd=GSmq0qsDgWFFW7~D7i zc)0TPiEwk<>2TkHH^UtRF#B2+)xa^^IW~Q>$PJ>|BdI5AK0|H z$|uSHl_MhhGn6v#V!ICLJDX&<>yd zTm9Pbe%4R(h-m|ltN=dpEbEW_GwZ+fihlK-64rn2K6dqA7P0>GY3%BSBz83gv_+U} z!+R6@hyRK74?mzCzW6`&s~6v4hll?rAvF9gcKBig*3=2Cf8jR+u z>!i@x4SOD;xdxsOU)%MnX)cNNUC470{F5xh;FZRFmt=yBmms4PGLdY_li1at&|HOl zM!8rryXJvB7yh=ae+2SXNV!J9^9Xnz0na2O$#)Tajzj-m$mIMxE%Ywr7_XBV;lA=r=XpnUrM zwX8o}!upv$@Ceyd+5hslU@Q3YyYS=pV6%UN{bK&W4qrY5yBh4+G+pR}{Pg3hqR_RZ z;wxmod-^cz<2g7F{FuM!?N;m6@7Df<#$4=j13E#`nWj;I56U{hR^a78=R0uq;FW|| zI&;ymqsk6npv~9P!K#7DC9IB_r>EkoYs@y@rs+1u?BlTH$zpRdJ8r)$^4cPW(AobI zGD)&6_2TC2e->AL2TSy@r7Qs)O(Dt!9ZembflhBd?$ourRn+;luA`}~F~}CQwRid! zv^}<>n;$ZqsbfaHQqgf1biL7z+;dIRtor9?h0>%9NxP^kVap9 z*S5N?0dK2MSjP(RE-*nb%^GNF2zdQXeA4*??VatN-F`GDTzgwbr?;&$?=EoP2Cu)> z7ihRE%nZ|TS**6?K)G~s0~!H$fen>7W}oHIyJiY2+j4=fXJ8prjBBxK*CFxMoq4RTuI;h9 zC}vgJsyczG%fnW!+1tURcN9-8k-ywkIoXXO(;Oge*ezSjG7e9)g4v&P`1n!qjo7T) z_>SF-dXmF&QfMhJ0-oWD_0y9h2&iB+D_dP!R?6mAxfi+Tdjkzk?v7TR2R!U`V`$=)&R6jHhlST2>6U#XWABJtKC1Gsii$rc`F%YKL9W_F=o|`#u_-!2@wJbUB^bW!+AezFZWo+h}w{ zU1x4nozAL^cR)V)_smzb= zU2w!|yzDN}z+m{}FZ*=iWwxw!%%0+4`CC!UvCf-NrGtnvTmJi zAzt(zo8ad@R8VH(b4dA%MIG@^A8lTh<-4My?%(z3MSZoGv$pP zygrm|FAVB7!ZSvA$QPU#$wn<HnJe0)F5k2GdNkk8P zZqPoF&N@Z^M9-(0phy4EhmFMXSAW3y&xL#jdhnAb?FD|&AJKFDWprx959&I}=dTXi zh~K2(Bv_H#H$mYAeV7s66>v@>epAmzlGtDxEBrI`k2;t44c?zK4K;r~`a}Ou?qdtEE_nU{2>pYp7)OY_2>=pXiQawjKyfSdM&FY3L!5+dva0LTa6e;EN= zggi2`NZ`x5HFlX8(1d@i2{BJmpV#J^wQ>lebm9(<7GJG@yNfn5Q&@*8Nrz| zDgJSP{5kc%!qk87uIVv<8ajyeEx|_q#utUt3*lc_@1y=j{xdy+#+%Ol=MdR*depy$ zoLXUET0+!cu^!9}PMb#bi7|hhmPvK)Z+p?+h4#wdR+E0GxlI1%0NyR^d2BkzJ5U$x z$sg^&OMioZL%%bl`c?KO2FTuTlfLO5Me3Wzvn&4`DuI4F1zyhQ(fXHY*&h0zN$OVq zLHg&PND&_6l|R4K)L-wejF`Ub+bsH~`EAfQpRXBQ-x*SW@c)dczU#BGUbrm!cDZ8u zt|EQ8V)L8N544y1M*kT>XbS>F271N%OY?&pQ0gnoEHCvJT@QweSb5fLlYC?8t9KXj zfwO2r>sQX7Kz^IS-$;Ik#oz2ae%eX?Q?#DhMgBnRLp`mRqP>c5GlLFJ?}+)EgX@F* z?VP5M=1+JlUym{WR=|Fui1}xYhh*O~L?GAi!r=JvWPd(B;zfCKF0ZM0(7OvuzlvXe zl24Dxm*Ygd6rYve$N5du3n~8iZ7>yyk6kiTL6%qMt@#Ziw&MapF00U^0o+uO7&Z#p|Jk+KPakBDW#F>pIz&$9LpE@MnB< zO#a-HMC)sLvJybp)9inwf15Er+mG)=zME$juJ4gKW(1Q_2SWqopVVIphKu6u8NqR=8}T%KjrauAjrc^QTSj~m>M^`! zJ>%<}-6iA6nEqYGuwHZj@Nc303H)OpO-?GxKQKSfqDuCK{gLlyk~@lcNcNu+vp@Xb zJUb}cdf;<|$&RQ$A-_!zxU<~oJwo*4PrDs_ zeZ+WaL=S(;H2Pa^Fw1RhU%1=truoL{xj)(*u#WIYj8E;O{`g@UdnU}0EOIuIM*J1_ zf?l*wqI?JWv~A!C@XFV>G#BzSceFpqpB|X5PXT?zh50;Om6{&N)~8Z^GA0nl$9Oqe zsPp|M`h$Hl$zA+3A0TgGFf%h6&#e3q|GgOR`ux!Pg@8^3!oSV_eu(CiaXw)EG5I_2 zruhK8$=`uD%m?TvH)!^M`1^FSFVAMn2~^kc((&K)Bk=aWmsmkU+q zc=jUl$K5QI{2Tr>j=fPu{*+2B{H1=oh$mk}jf!Et7)F)HgXv_y)aOtY@tpLXi0z0m zo+lIFaA@drNQo$z`B5YZ-&6XzWqGh)=j#Xg8(*K#nCxlI&rJCU>~|o* zkf2EWSrgTJoh4vL51Z=BqZnerXVXZ$2E=U(ZT z=QC0Ni{&%?eBxZ1{f7Aq;@fremp_I7A-;Xq`OD~h#gY877wMB%&{E_b`l>K2(oIhqx1~%ftd_Vmu<}2h6ar2cKW8^FD88^;X zvUtAYj-RhUo-8-T_h`NnG~^qj^A*HXcUClC8HWBoc|MXw>no3!=+BamAYPjC5eLSb z;Re5-ARoyR3qHl?QTfrGn~#wH{D;g(=9%Kx=>0ck7btT7eIUr}1mXDy&hN{+ULG@E z9W})(!}&V==XT;%bpMU@&Ai`Uh|d?s+HddX;;8kB$7ecHqtAQu}{rDk#-e8~1_v73@{tNBLW9K~>5BsOs z`4!gx(fjeK({3E!rf`3s8b7`veoUFl_g`v1K03ac&VO=)Q>R$sTkQPDy1&Kzy@~zy z6k~iV3{KD)KEM+DLz*9%6DGn!V*j0u{dd*``h3o?|DHzsMMJ!E(uO3q|8_dZ*ni_d zh05*3yQ`8nd$0|)} zmi={_j{QFS8VI;QWB++W{_uG?{6U*>CgxT2wwKBl-CG zA_*zhS35ZfZ$F$){nN)SvOg_${-1Yq@|w>N(%2WlkJ!(m1DyYqcYhSiPw@GHDL?rP z>X7PQxu=198n~x{dm6Z>flplnba;+XJAiLFu%2IB`~8PMihT8@zUtp@{=0wpdF8mj zYW(QJ-$V`-ve|u4^}1@^eRFe;_4P5M%OTI@y;(tkRe;(9)Paa7afmlk;))!2h`fWJZjC8##eo^pjp z=pr7dV|5MxvkczHIa7Xz?JZ+_wPgt8T0H}Otmrw;M2*z|VADJz@s1)rxYNcZqCI5* znl_L3r^b6V!H#-yYH=dHic&pxt$g!j!3(I-LcwW_E36ohqETa1Q_l|(Pkf9kuYwWk zi(aA~=WU#oH#B20{N5BRp3)tCsk7h;+k(%N@IBQOB#6CQB3O==c=)JdUGdvfF3*cQ zJ|4-xC}1F7|2)?x3Qv5z$hClnV&b)r^1mgfdu4-o^ig^IjIGhMWF4B0!Z?KNNk#N* z&}ov#;t!mP+mdXuca$w5GUIkL=v_||dXLIYBgMf-<#xr#N9A@4x#RT_l?#lJtV%Kr ziDHn<{;5c2AMH_O2@_|MCdwJ4yAfxj=b)K$n{MSiuFiGAS#(r%Vzj-`by0a+O1v2> z^i+&C8ncd84X8nnqwvHH>0phnuX+%3&&@rkmo{g;n#TsLDn@<;tf*d3(Q_P$zvC-~ zHp;tbY_+0TSBw}>*VCZtH~tC~Yrj{w%Iq?X8_%{;&jejOys`rIGg@lH$fLB021acs z#^X#+t#MxAn_pX!cBPjwJnV!Q8V&{V0{_&Z;Q-^ys*Z28nCefWu9zfx460KdMicP| zb-LwCAMjwqME|y>zEm^gJ(ha4rB35f|9pu+{~N`LHi7u~H>{IK+-jRoN?(*(YQNP! z;Nkd9NFFrpJjvR{rDBt2{vJ*pIZ-ymp!EPXe#ZBGxXOW$ohM z7B;M1X8N}+?Uy2JHumSS)T=FZWUJR-yNs*7@V&9(+l`_DIga#_?=!nT9aSDsq17{} zXB+B-r#e@rHZBzz9_;{Z+6=Ia#h|Qioq+<<%87n>BV!Hy$;bgCF)eGv>>kQPX zE7CQMnxpEmeTGe^QHoO6*^69{rQFG-j@X7sJIGGM3IhDc-2_dokR+ii=pG;=|~;Lwn5lCux?|UAM0O6fX=rIZeBQEJp3* zWW-hDnk@7OiAywcEH>E^Zp{7OgnRqYKg?W>x4Pfz{J;HZ@M(;0#vPl8^|7e8chru}0}qPD?7hqw_0+6=OV?MIVz15k*|&BD z-baVw*lnva<_P?=5|8){i;iS1=jg)6P51%v{W!7xkM%$LWSmQ2=wH`(|2>K~`retb ze~DLU{%>w;bgjaL^97nuqMWhgz$VT!jIYf{ zq;}(<>hf}7k4o@_=l{ICoUtwC_}_B8uJ`;OQV`jKuJNAjL-1eH8;T7lLD8?)p{}(1 z2;lr+o>N`_Z4U2)9}Ss^mT~y#9>jCHxLcd(#dEqcOfafEi791c;z=(?qN#DC7R`AP&*{ok;OI5_oGuAb^ihJ_yrG%SkLfJ!J+j^q{Xkg&Vk?8F(_Wl*wK39U2j;-x!KlenO z_Qx#kRnOM;GlW9?9z`Ipr)2vNWSz#R@0wWKli#C@QTz|cDdIM?r(4xt+3^rMCYXni zhH}<2#nkw!owEO9XndpgijLY7owYs9b5eoDO&KG{MCr*9S87i*J)#i-Eq8Xv`knz%^-sbgc2v{BGHwBH+N{IAINxssB` zr(#hb^U3xe%lOA+d*#2>o?P16elaHRQ*p-s`eqQLxcNI{*721eRib?_+IuYH56Je) z|J3-Zoq|sVAYS{oUGy%4@mhh!5%hqh#SlLv+VSV1_jg>3C z8wz){R94ll-C4f8W%;AEP5zp-Eh`FYAFbF~v9X}0WpnHD{Q4E5ZE?%8mimpwdA`lO z-<2B+mUq>+F3STC{`!I?wcCqVZ*SaK)3Ise>c>~MuFh{PukC7F(Xta`w1JQE_2uik z8VZ(m(|C;=D%*=Ic9zyJTA$ZazO6DkZuz#=-lApgW#!w}Nt*IaZR=Y%E?U3+rsxXt z$D&)lxi#3dc~KSg(OOP2Z{19~%Wp3&*tW)7ygGkV>$bA(MK#@)X~mzKU$HVtuo>Xzi-vw$*tJt?QfW zH`lb6Z1UFD`fHZ8lr*)MZrE1sEiPHLynJ>3=JK+xO|30mo6GT9w5n_Kib{WD>-ui! zuVPL9`o|krRJLr|Pyl<^ zO5I!NrU8CV;czWg_oYtPv-&segy;YGE8)~HF9v+BAsEfJ;EBG}fRFwv(n{yc<7j{v z#7awaEjK}@y0>0mL|Z$1UmZ!9(DK1?CYX>zDLaND7BunI0|9Ilms&f)dF0^$6(6V) z?|IMbc-LqYr^I{Cw4mn^aa0SNVO)h2^f7+SaJdswKwJgAapx_N?YLwZc5X^83X=p^ z{x9^!*f$}Uf{7l)JAJ8~=sgms{5}kY#-m?#9r*9jJ{~>oZ{oopz7G5cfD=DwxE&Z% z^Gdr8|IZ`wBKm4+g4d)RG-kYU-;naUr3uvhY6d;^b6LVw+X4w!o77U-e~#>@TAEDh z`R^tFc)AoXYF<4n`FvTz)qHwQ_A?+2sNfp_PgjGYr!}Zk!Bc!We2c9AqX4sS;Y##* z7>CM9aba)aO7JTZZkKQ+-vsohe1}jIC4Q0xo+sf`09S1Z^hF7u3b=90xZ40Xri#ZT zoW{jzVzeBWcQmdMUMl6J{uOUTx>CZe{cM!*>6U(!zNjDLDN&E1e|pY|D$g0J7y<7R5E#X#q2c&_l^1d$N)_#5_;Z}KnY3Rqu|3?P6QQk~Be`s7|jLf?Z z{D_1TKgQMUISGHr0v|BY8}0j=gj@Nir|3{2ex`_9{=Qhk)ws$Rn8=EC*Bax;Zst zUogOpay3b~(u1OJk#MWLFG{#o-clKI)VO(a+?58naonJUt8o?mZV9)J`#lM_j!St9 z70O*`N-5n@T%)4k9$sUtNnRCP#f|4Ia1|d0EN~SEhAnWl^s4hu)xTQrXM;Z~3f?2_ zx55Hf>%HFsSL;0!X?I6meRG>P(9N(%X!Nf1cC|G$J<5M_vJ4Mo#;-f_-*}>*Yh*XO z^PrCEn%kQ3GmrHh9khEV{dZg)`9=LM&^BdDN9Lld;oTk3nPxxr_G3+?%u zuW&Ei&!M?YUt*G4~&w{qqBRi(||7GFc>s`kdN77TxryFYG&mUeGrU1xh;BWFwq z1oi}_Y!PMK?z?<59^4+|cj~(i_Tkp_cb`=ZkCw$+FAR<#O zQ%_f9f;v^^9||d#X;o#k&nK8n3!-q3ygMV~m_n#{rRuYALn)U>0p!)cQ`|f!^D+8t zjpCgQ$I%?@?*XLojkz_objjfBp^BlX@zwQN*`E3~mTDEe8;#J|_zFsB;>oJR>NY{4 z_*L}Q`hS<>U$6Z~7KGwk^_LFetu`Ut%7CIUU#+y;W&2U`%l6M$8d&-F0UytP4(UIa lvuyt%|a<+rs@x$X%eqw{3Qvp^>RZ-qSZlRdm?{?b0~Uc)En|25D01)Nw;mP`N;eCZ z_uIU`N_X4-jXvP*$zbf@X}NnKZs+)?^jsE z?`^h)FWT6_Kib&Q!|m)KXurs&ux0;}?&*8KguU{gR(3EWK3IOrmU8s8EtzC}kxgdH z{@v>78vvixVtd7lwsYd|J$dw=yG|~|_07BQI{D95&oF3v@HyCc?(xB+FCH7*^Y-I| zkKvlL@fU+Pz5VmSg}651y6(QW2G>7+cJO*!H{cpOcSZQ8Z{L0LrZab+j9M4Ele2FsJFI!J8g? zX)yfe9|rHbmuOan&)$8{$)ljTGxs}#3)Vh581wjxgR!MA4mLvPb9>H!W@VUcwf^xh zoV+vpvB3rRpsm~&2iIr+Z1AT8Zw=mcbYL(h=j!m;>u*1~;Mf;Vp1mGzbjoz4l}mwC#ajld4#MJh@1B| zmxSN1C`KH5hA-ay%n^M&M~lz*Y?cKoMkyXuEP4=&KR~QriN#M=ECw$Ti-*UF#lxmp zq!?B4*l&tQ6~C8?$+MY#&1XyUU%7a*CH!HlHGHvz9Xwl+W52N579R525yuY9qt5Vk zzu$l8UBvd0r;GYr9`|f!w)vG@1Eg?-&%$T?h77j{GMev{EebMRyJTi z@(bX1*!mhCwhSM+zo@Sv7d+2lzJ2Y~-0H~(u4y=g`U9v>LH!BHxE*ji;E$|*M}7c! zKj2!xX8?Z(a0}q;0sji{HvsPf{3*cC0{#l%&j8K@`~=|p0Pg@i1@NPQKM!~lU>o2= zfNuf37Vx{CzJ}WY?*n`-;MV~A0e>6UUAS&VeJ<+10=OLT^?>IAz8mmzz$JiF0N-2; zy={HaXSm@4;1#&Ok82^WgSali^=(`i;(89(`M93JH51og1^uW+4#>q%Uv;rbG; zQ*eC&*F;>O#WfDsr*NHs>yL4@;rc^dN1(^!xc&*(M{)fC*RSLHF0Nn3^&MRA#q|%k zhH!lY*S)xIg+0E6ek{awFRnjBzjokS3;Z15y|~WCwH4Rpmf?{D?3J4_M^;;N57t@p z57wW~JGkqqyo0+R&O2C)`jHu|?{`?sN2aj85gY6KQylA~`4xNzeP^D!ruy_#*Z8bS z$9!+&{v6#W9`n7A`$4)-IOh8Z_ZR4X;xXcLy?}Y&9V}1t1j`#AcU*LD9I!v19-Q3u zgk!k+x1noT*%A8@tpDy)hv%|SpZI9_fOG%>@nX}P&zQmdxlk`ye^|1X5_%z;Szu0}qKFlor*W|Fn{)W>@2lo`j z^xbroy>EXpK8NgmjrZDz?*whbm(Z7xy|20$`WN-ttJhjDdSlrC@G-XSEb_+~sgBx#%yqcN#stf=Gmd^G&cgcAW0+WrZ*}s#@T|@HN)G15>nw;<>y~5d z7#nsw*fF1V^D(!>g6k$+(I@LhTy4jU^c#+?vn%@RCH;Cyzb*=W*|BvlMPDlE*Gl>| zQRr76TNkhBOC)`UW@xY+lc0X>$CiUnJXJ zKDzBaw*#Li@z+Md9|C@<#IKEle-!wY62B=5{t4i(miQe}@Oy#x055Hz>HRF~S4+On zMB)1t@MRLeD+>Ngz;B2m_e^nLtR-oP`?4hp-$%gTD)IZH;8RM#KS}W49tEEZ{H+py zC<=ZD@lO`~4@beb5WmEKFADxy;-4b;e;WnAw`e%_R^ZKX6GFWud361~L?5O8%S1nQ zH2v3!K1%(gM4vL6{&Av@QvXAupEjEQ$3!2c{wbnQ9Zml%(MPF&f#}ml)4xRYQR+_; z{q)iFzasi5^}i8 zmDd~Qg;>M~?fI&$SqHN%=}%g)=cBdg1lG0pHm^l)y9PP#>hOi#CCGh?!xwKZ3ZKn< zu1UXN8#!P@ZtMa+26VBYj{{#ka^i{Mv&j1fkfXkF3+p@cDK`8r^3V6;iu(SDeDwVT zp5foOB1gF25&i&q>d>D&eZ$CGe-{^A6hwp3(U;IW<-@yEsoU_013=bf8`{*~G z;orUH89x29qQ2LVd%gZzQQz;8ll}e!hIH(X z_`hn4`UdYNel~m#dEOtU6!(4f8#eqVa=^DTi~BBq!_xOQ^2K*biu*o%Cn5a1xb-=& z-M=P$;W-99J<#7XJbVB$Z}Ie9*u6eH^e5K$Ve2~Bq%1t}En9e?u(j7Ri?j5-`vW%o-sX+rcS@{i$AUJkeQzdOpog{Z+yu+;;N7I7 zrEg#b*3kPs!&nRZ&d&fmjSUYVj~tkf`tQIFiGcqV@V^3=2>3gI=flnmi5BqvfL8!6 zwDz6-MoIYWH0&XM!G_OfuFE-lKWJV5F9 z=(uM~77;HEPF%tN(dM5R_VRSzq?gAomp;oAruK+1EIGk{G377kiFL;KNAp5sAPv7wavxh82MLDh* z5B5#7hO%Q&H{t4>BPHp**`PUMn>kd*ud{~MWfbLvpPt!!p6vdt+Yo3?Dr( zdB|^Y3NAFB*-h{@{S%z8j3|d+QmM_4L#Or?F{U*QvL9`>kvCDEW_EL z8`y2_b7igPx((C{{J;{4t03I<5&lxiKOg*vb9-xxlJIGv8V zG}Nucbp@`jtWDt*+%Ln&m9;sXjO%=Uza{L%buPcZA#BGrli!zzt+*m)K)*Fi@l$|L z!@O8)X(7@d`|srM)>sSr-xs^FD-oL zfzRp_`qj|9-h#h^JwB{!;jc*9QtO7xA$95-`M-n3~S#XzMUR^>-)3A@BHhm z@P9o$GyML)r-VPixE{hd9o}l`yYMYG{HGF2--rK%af)$2Fdy?q9Ik~lev5_&9-#4C z+&7R381vD&R{*vF#{PHU7QnTD-(~O-u7AK4`{aRk)b9iQ8^G5A z-op^fTY3k6#t^5tK8@>vEkgrd)R)}QJMeviZx|Y=Ah^7D;2Q*&4-IT4cxx}_6~J4E z21*Iu);n;3;B7+#*8#R|9~!{?e&p%xy=bTJh;7HvKq24*xGo`lMeo2?)R*9TJ@Hix z4P>Ieq_P)$fGdXvW)QrycVHpGJBJ3Q5M0%Zdcajf192qhABF}d;QI7G^bXj>9=PG8 zm=`eCajv)qV}CW)pc2dn#h4e0Fh4(Qn3Jx|7gy$sEAz#b`C`2J0{IayE0F}@fBvC= zw!F3dhZdH^+_ww7m02wG0OiQ%1Wf;y;o}1F9$Y>6Ou%OnOCm9&%AR_EOMO#2+X74$ zVXer-tns*7t;{yjX0h5V%pQj{T^3sc!x;>_MLb6=QV1*ivCvVH_1q?IlKxR#t$Uc$ z!&Wi}G@4G7x$O<@9sUkq4er#nt69MO>TPdmsSB{&me!8?+?Kt$9eWyEcC|*}+D-}Y z&DCydTbtW_9W1w_KG2aXy5_Cz>}c|}w|kolz0Cpd?)v7vUWwUO-_qXd-{@;=-`BW* ze^b2%ZxwKRe!YJBbW-n#ZCAR1eCZ*OerSn2I(_IY=;`kQ?n znw-_X4&Pc{RlIZC8gGHOrZwQLYiuULUixU1CJnrK=%?t{C!ibc4L*N;-6y9b{`&TM z|6cS6Uh>yBeIlX_{qTJvI#O3()4AKb%kOKh7v3>f_;L+37-qG;rl#F~Ut0qobZ!3D zj#e7+9I9q3Vt?W_j%= zW(X_a@(zvVV7aL1=VHazF7gPOiCp?}B@MZfhWx#yg`-11%j-oBid5d$?t*J-$>aDBC`qx@pU*|=t?e*7p zbYe?T=dJ1T@_MEho_~45JD~_f`Z^#74RtoRw3{2dk~;m(?KlimEhV<*ipx)({{Ht~ zYGK*hT;H7Ax@%W^eMcVidRz8lgoyNi>&7)3y#j+l=WA;8wMQXP!>`Tf?`Z63Y;ECL zi6(=jBQN_z<(ijTadUvQ;S|7JR`GhLxte{N*H0W(|6MD|NgP%G9O|uXBJ2N@K;=n_ zESso5FYDE5NIX=r@e>WBsH=H>tg#YMCF&<9>GZiqkfcEM4BUDIs7YLcdiUW(tzM$p z7ET|jcS}Tvl%oXDSqqgrQE$V!kGYJXWXVdpr-J@;(={>v)?hH3xkhdcX1n+;V^qJ$ zEJVV}X7-h+5WM!|2?9xcS4%p&r$X%nL&0owGp-PtO(9GYtS-tSs$+9_`!i)-gS@JC zm9MCp#Unjof?~bG`QUzt*uS%i`v2Zp&M(Yey2!nQF1gEAvEt6graE_fS9?c& zv%9&oy~ABszpJsO-o0(}<_%lDJJytMUAK9YcTMHCHJetiSk0bnb>{5GGysu-j2@U*ckTcM7L%xS!-=2pwMNUX47uCmJgD$&vfZIsUWn zvLhD3x56&kn?@QjR>0n~vGbFdUBdb7yaCQ5jn8Lj_$BO=0nTNw8sNq3Wdpp3J#T;) zvVH@+fb|;S9Cq9QXR{*)cs@I9fakFX4e(snV}P^R9R@g)1r6{V7BIjWtjz$sS%U#i zXVnIHHmfwiv)C2`Jd>3f;2F$gfTy!Oz(tyU)0o=;r?MmiJdLIl9eoNrZ`H$7*(n2@ z%w9IYQ`qx>BmSPudJXtV?1%wQVuua%iLA$fPhdd9{ zg81~Wz&$2_EmuM5^+9G|b>8waCb1Dx;y7+BE_O=7c@jP^VV8u_TTZ`H!mtL1lOzml zao8haSd+u4681>AOu}0v?3Qq~gtthzO~Tm{4obLE!aWkMmhfQ-H%R!ngxe&XG)d?~ z{f$L`k53jb^^f5GWC2ruIGhsEpTnsFCVe?PUBILthp~{6JxL#ek7o;*^l@73>@dEu z#O)0lrzMuP!57N@M1NSqN}rb{tn^{XRfwMKN%VOVR{B&+Sn1OvVWp34Wg@fB%woq8 z4<1a>{rJ3$6v6{NtMLiq^D^S38tvMXfTXZMd-So-Mlhk+(uQpISQn!-AZWw=a#_Cz zkBOS`*-g~Y6NRoH$?HXV{ZL+)3Ni*4@pVV7!OsvAimF$+z-m#g)k{!&ilyjP{oyg_ z75=Y%4c%-RVgCuKE__ZWva~Ll6J`L;O<+%V!MIfJz^5jGg#sWHreo}-MBrs&H5d4Y z6WBq&E1N`(^xvDvo?pUx0ux*mMCl(8-ZKd#1t6zbUJ$|kcfPY zv4j)^%JZ99ykEpl+m=Ba`0Wn`;u0x;8F4~m$cqmMo40Mjn*FOpR+`ogER~>*ol9ho z?X!=tMu4%PpF{dPd0cQ3@P`5zT(S!!FVvmHO#wWY-`AyjYX>}uJrI~|4+5$5e;|-- zCxH<--X7G!Jpnt?mLxXvJM{l~>Yu|NNBmO%hXQsx$Tb-4VRTcW_V6-T7sKc_zzFaN z{A{~`_`hul?0cOfE<1yPEzrNlE#N!_GXc{;<#-9x#Hfd9;Ob$I38sOp$CsI48rXXL z786VZU5}@UQ4iC=*TWP*dYA%;;5x=mbd4l2g1Ntzu+p3kX%ON6B>_I1Q3n|QP3@9s zy1a^bxpRZ85cCuu6u-Ri_^>mN1SvlAJ8XP{7@-z{w+0+g1%H60 zyw)k!NC@M86pwa8`=5rs=q3vEdtb=o!*5J^Z@OsC z8W}mj<)!&;HkxA)Z=|0q&2M*TF!av~SgmftvwJ6y{`YiYKZJg?Tj~BGR0pm9)L3W1hXOGzLWFosG z2JK;eLZ4mijE#@yU^5t~{W-IZ{)D`qK(dQH7@1%02{1ms+T7>k*h#Xt%frifG?p z&tE69jdv=3fk*!mzJjVY5A~1qyD<_^=pW{vgs%dx=BLN@rAFq%c?s+g<|96z(tM5n z(R@Vh^ZDu7MD|F)>0p?UN2~&WC@{euOb1N#z|;H|)X5KZJNW(*RotFEfp|XQs`f+O z6HWLeXAq`{;3t{zlby79M(|Ti_!MW*fS(q@Bc6LZSt{Q*21%bu9Om}ru!F;VLgnxT z4)fCqf?to}FrSb)9LHg#S8xx>vl-*>8Cq|g5&I%OLjkO(#l(Pe`aY~L@nGbI+vlKv zT4FZ9kyNtp64)~a30Ne4Ljf$T8bA3XHe2tHPO(__Acxzv1{i1jeu86)Qwa>{uE=oF4xCmrmxyf{M5Mh5S%pp{aihO#O4a*jFR% zg}O7?EUvEP@1A4QH*=a*r*9~bZ;+4n-9WF4w3!} zb{lZcUzr^jW|+ zCR+TYxhQ=R^fVq3e+UT0AL^cA(sz2Setsf-vo!e>kBGO!iXY>#dlpOM>MDI_oAgbe zM(nCRvR}mhiKK5HBLy{m-7fa9l27)?V2gF~djgr$Xa>;9b87y;dQ0*n_T};m7zKco zPyJb>`XlKz`@L>q8@2V5)}wTuj}vp-mh(K{!u*f%rt~9wz<-ZOnDX_QWJcmKq4In^ z7x{ozFrE$TAK=A$Jn}d5dN?}Y$Lx|NC4&Evz?5v`{6-rCUgZ2S(N(SapXd`z^KGJG z{-k)53%;6vrWnRE;#2OZBp&(6ZF+m8Bm~jC@(<0odiqfJH1%{q@@G!b$MXXLr*8j| zj``Sb+JCHcu}AgoeQFB%C(_<3Q+rpb4Tx;-Y8Sgh&%a~}`76SIjfwv{gT4<1uAQOZ zukrD8BqERIb7Oy}o7%_txhF#36Id}tAI~(NjQr`Q_Gf1r+sAmpH*zBSXPEfs7~02p zi{%F++IYi!+N0;6YvP}8YJc7mefuE$gp$;&n9(-X*Zu`lZB^G)=H zruGX~nA*2%`3+w$m%7;FdU?xC@|GLgC;j<`G$`gT!2B@_i@?sS5g$63whwyPiF95M z+ucUknPr6Si;S={&j{O>Dmcn~RGM>2ZlJ~davSn<7xpjEo}ULi6o?r?IxF&BU06v8AI&urzm-#p4&-(e~P++2yHk=B7D3F}oucr^hF_)0W1Ej1yg%`J^N7P^OPSN^@)Xh5f2iN^Ns}le1`Lm1nO@@UJ9K~jyQ*e zJi~tJ`9$_c7jxQ$ez1>)!wF>&z{it=!;`ld;p9pKoFblHCdIJR4&filf5<p;C~z> zKtEoPuO34FtoI+{C6?k(`SVa90n?`*PKl)jB!YKf!HoEm+NTYi!iNxVTD)NYNB*5$ z&0E*}o7^B_j7P$!v`P4kfMdlPuIL>UuQ(%ve=WxS6#QlOUy;f;$Y1BYA|5E8q|Ne-7x&O8O2d&?66KOsBIQ{=OH~tSB{OL?S zB4OqK;}TZ>?={f><^HGrkmmm{x-z(US`Yq84F%Fo@)tX|fK}Nm z6hOMG3hz%GbF&bsAoo}97=3;l3S^k%UBRUQj9DELrdPK+@h&NRq_ zeJ~wF>_zAIT7Jguv&31gYwu7XHPzrRoQJ@HR4|{CKjs#5T@xyYXn z1+wvcFVg;@zygQ9|L1POs}z%ynbQ{J&wBtbnw`SoIBE~)3r^~+W{>Wr3<^@=`h>cd zMfAgd1N@8K*fK&M=cjm$*e6}=m-Od7fgFSW{CS7rd@0}AhQ=c8FE_QnBGUfry#3rc z395Yy_SXvy^40iN`>mcpfis|!PxhnvTkJpRdBjRnd#fVt!9Fye^!7*pm&}`-D*Nxi z{$I=oF_H0U82_Np%T8G;=_io=@)yJ|*V)fF{_#B18H9Gi9%Ros&N~z=>|ZcFVWs4^ zAYNt~#t)4z@}J4cb-P$S08VP2i^QZ z=hNA29`r$x@$EK@Cms*F^S;sk%XIdDIT9ZW4Dv$V3w8QaJZCu{)X|f_$sTII8R}kS z;$LjwkLFMP{0ILfPoF$b^FQD@GgGoP|C;A>;4^2(E|B;b=$ES7KahXtI*%xQg@3c$ z35zuU((@L*zbHNqEBe!$;je6+zsP?^e;NHj^NTTFLfs2kHsqm@e=OX;y7S4Nz&z)1 zAS3o)q@z!VJhT6!__rsLPmT8P9A~di{%C*Y>e{38Hsg2`{!sRJKp*4!3;f)fDa!tQ z{LPyktLz&Cdh__pbUv@shy3N%;B*?lbKD8Jntv$XUe@pwf62~%4JQBS$Aj7aDqZ@Vr@fo=oxgs-`c^hsN;jvuqVVGqndSt928~_<7%Zk@uaj-#6=jTbI>v-ak1IKapP5!g)G>-{}Rv-3kyz zobTg(DCCc%KkzRkvdMTqs+wS;$9Xovq~F9Lz&QVW8}AEIe;C7aD!gBq7O?UB+{)i) z9Bogde(7vk_fjiuKN!RLIr5qKZf2wT7tc@Q$Udca@~0c95&IGC{meU1h4!r7^dcDw zf1eERqv8EBo)Jqp3;j(3I^Ih9r?Zs6MB6qz(%_v&J_CGw9{u|JH7s!5(-&Dw}$P_ebNTeMf|S0YhKH zlP$Qt0rU5F_Mk$(FN%2J_OX$DfS*b4zoMj`7tgnepPt8Z`G|+xy5g;j3`hp6Nnp0X zWd7q-Uv6M}vq0sJ!o_X&88UY{Y2pPu(| zevIeay4;|DQqVULoCWxVI zq5;fIiaR`&$1I-kLvM{+V|EypdjEJe}wOzYP<-DB!mR;z#5J!1s$T|0KJcV}lO% zOrkZzKhq}pIX=?f1>RohX4?pBLWEIRz~3%^mOV>Ichd914F9B=*iehY`#;ekpI5_P z=&zqmrunk1T*Q;hKgCY*iSYscM0c3JOzJ(9P=AZg1r9;PI-c${0j$3|6=^#p2n6Ey>lwQ zs6g`kEMDNnc**c5jA$niSPyRNO2T-P^9j~Fmp{#s27bf`mv4?=vIn2fxIM^U4yQbU z!T90+vI~BQ!2AvW*=0O&{WJUydO#=&^d-EAhX}p&zs4{89l=X~Gg?osoh+V@bNc|M z{@dAn*?-%p{!@R*pS+mu4Sihx=}vLpLGuB@GhJf+VX#L!;8`(pyvhDL9D*JaA)_v?;PwSR1?MkGdl~*V8H%*`bhb$! zZ+3mWx%~DC^4rJMAA2MoF(2{x!+O>u*DF51WlWesvPJx1{t*7_@G|4NBE7Z}u{7EVvF#c)& z)x|H@Z<2}+KEJvAla+rZJVp7J!^AJ=6B!Tb6PEDy#QaL}L;8vN2lfNZ*Pj}(o)f>A ze?TP1o1A|%Sj;~PpB`zC+aLL9k6e%EP=7OAa=nCoX}+5iqhC*4{>)hY{Dbu`DHW>-)o2;L~E{{J`f!jJH@hek432PWYeCpP)~6iS?Z3*ZTlZjS=e$ zpKk%D#ESVx!PDYId{v4lW)32xC&k*qliGqKQh(`ryW{P-Ja8{OxHw9;8$oWU(*RS_z zKT|t#7wsF?V`^XKLmIzapCz1`shyxpemTA*J~cz1&$#?H{<<>87yL!@p*2Q~AA%`f z9C)9D3c)mg+T+OH3eNDS&C%8e-rn>~{rs8XpOGc~FWa}J()`H_r?;j>=2!BMgC1b$ z@b($Td?8-NGxCGY&}X8GXPGY~sCbreqKan@Uu#Vl`$cjBZ{KFq=aU(Jt5u)h(EO^c zubdt~Fr?G(_87J_o#7=nzCPQqe%bi<(fIuO4A!$bR=$347~}KW0^R%zd9=RS*jFR` z#k%zm>(}i$4Ee>=0*~+4+`ft}b+B_0cpa^elPtV_e7^?cf${Mn-={Tx-$s3Zq!>NG zs`)nFhs9~DC}O`z-v{uhD(H8c{{1)1$N3$O$oH9EzYgEOab{C{>U}fAe)VPIw~g=@ z0Ok90lcAq{KMniK&Dj5n@AEO*|6;yN^IN(5`S*_iPgdVYqW$y>cwg8r^A(x@S_3xf ziTFMh)~__+3FQ1Q(fh(T5{M%D$MvBBK>8p*BKd^n`*kUwV4h#oemcfOAd28O?*HK* z{(T9cE%);LKCMfOW}aV9_KVl^Mhc-n{Qo1rgNgV~zkU zkS|(sp26cELyzq9O#1|FIIu?&*lS%Q0#7|UJ;Ah}MZVI_xIy{%nJ~U-zt6`n$$ND( zyC-I3#EJ>U$%3&Af8?29+VF4~@=CkZY$N(_rSp!fpe+po*mreT16Q5V-y}E(dv&S) zMCj?e*kjQf;sx<$d_Vj?@xBg2KC3;i9nziOT!;3KuwRPrukrTz`{cKc^qE_Uq_NyFO&}-DiBC(8l^r=UJZs zSakl%zgH}9#R69>aK!>wEO5mFAEyOqZ-#uozZYr9zzO{G`Ultk?;F4Q!HDgxTRu8* z-HGqNam|GEiz6S7ocLfLdue1U8y>mUJv`#+`S2EZ*+Q@VaB{<3lK zl)hhzf6F*{O6M=dUpWq*()&yCH;jXC0(GMT|ypqs9#Lo<&BSlSZ5+ z!L|AVV?h!SvRIvZ*lLDtCcNDUhs>}8c&)tye1sRzEl#%1FnGX59*}z}{*^N02o8;& z0Wzo0yMR9{a04lA^;lWR3e_V6)vCukJPsCeSZ#W|vHcioUmgDN5vu36d+aP^A6vfD z<76S{CFBQ5e!r5>N3dT01kVH(nlQF}m&b*dj4vfWmgI{^{fvJh-_ZXUbS)G!wtUgQ zOUTFbc-Tjs4DuO3ufOR3*z!gHE+Ici^81y1K11l`$9dvdC~j=|QS4)>VHin~!be<< zH8v89Vzr6H*#V1wrHHbU81QN(F~)I_ULWaKgOSfvnKOBu@>6XaLOf(cFh`12kC%QJ z6EEU;Z2klf{zO-TF3K@9^zsuuc*QDlY@Rvd}~Wf6+3!F|%Z5?7yJQtwK;@YJJ=Fg>cqWOo<(`GqRN2M#Z#qXoY$I1|6 zg%%NCT&hKykv`rQDGT{{Tcla*<86`Vv5&V!nsLT#k)lOK%%vhpjfC3d6CdJu=n;yB55|X_#7{aTX#9MHMYEehH|7w$uu&*R z^NwjGSobkntM=$rCg zNT-{MF|+N)(k-gp`p2;wi|7_rcl1r^rqg}Y461Y;OE>GE<5Qj8qUwgeMbT~a%q>Rr zBIM|wv(h@$Py1EAuAQP?j9S<5=V7rI0iH?&W`fb{P&wA7U04;=Qz;B>zQ%}W+Qe5B zzuKaqxNL;9Uw&da9z+FLfMzc2xYF1iX)43~@hU9wZw2f;T^zmE+4%ZL^#4 zL~dP~P~c*2o-A4?BA*Xg`c-B@vZCN&cUk5~#RoOtZW4HAlCR|{ol*Jwm3Tas=j|sM z`GXq%X@O5Q;UDy#z7-3da>*9NM#_P0=rRHkSJ__FtVJNBin02^RF2ykNE`_J;?oxQZ zJG?~3D$Z}l&S)vmxs-g$N-l+`9Ae7^I_1ma{iin+M8UfSm;6PErIOx!Y5NCJ!~b-W zodQDqR)qc3PK}gWxLQK8IlMVy`HOy8qD{(E?NchF%KwY@PD$bYvcDeH7{C4{zsE4@ zw-&10?5?lPdE~oaOKxWWvf$khDsA7KzbGT`r?a2CFEM^yQ|uBP#a{V625$E&zsJyP z_`QNFzsC^#^M8+_0>7nzPY|EW`8|dx2CF7(7_R z_)904cj@;SswJE!;h=OzQ@3U&56V?w2sF8@zo9H%OS4Vrsul!bx}`QuUX9&){Xe08TLVkN>`b zTfo#Gg4;3$O!^aie4cP+!QcES`_sHbY zT>m6|O2u5SR&c7<-Wvto)d&qL*GPTT3i+a_*6P(t4HxmE=q1QfO>$KI5qV>#SNQ+# z6(NPb(c?d3Sq0F>w2WFqewx?wv|X#C8}a*OsfIcN#P5?$VyeF?GZDW}Hi;?wcqFIz zeKPG$Xju!pLPeYUMy*z>exFRI&y519exFQ{N;fj``(&f_q~9l7d^l3?mW~ zQF8y>E4dfb$Ef0wHMNnAE3c8^pwY+SE*k%Hl5e}V*%2&eehfv1>2p+Z(X}!gR%KcZyHq_n6ecp9$y1EiOE= zNp~~B<7}b6})*LEs{%<5k6usFrWm946`%6|UNB#ZKOhAV zKhc@_X&xaHjGAF)ppwcp;zIm1uLlLF{W}0=l3AkF$eT)erAB4p^Ro`7YMQTzpX3{g z%FE-IJ$sf>-zi9qz!+eJCYmonv^df>@0aeM@ud!V6{u_u;cO48z z?V}h_zit~292(DBoh zWaeLnim7NH@nf+6Vae~7{OV$I2ipq%&qe8fzvNf`Lzr^^o7<;xLpJTXO|@XWs{BD+ z6+h|m=koC^L5)vibJ}=S<5u4u>8i$snN7jfGC=fvyh@HIMW?Q6e5tGA+g-GEN9pd` zB2h{<78f*i))p2w?JV57{idR(ZN7@S&YhLz`L)dz+xINHq19JZvaw`GaYJqM+K#Hq z@}}Cht=pQ`?JKTZR^D2)Zr|q0d>w92M_WzvR`8W?DB4-EeS6#1(x$c7Z{JmQL+edN z4ZEsJcQo&;+|gdQp7WMg7VPM#Eh}Hvw0<+%?%1`nVp(TRVR=JcY17`CHvF7!!KVDW z(m+FP;l|FLg*)=9wl2%7uPjCz^`NiXURAMde@)@_)L#D1=G|*n*^s#(ZQ%OVX>WbZ4e8t7BYYKNX`ii!0EG}%=Rada8tEz1CIQmrO z`6}|8YFoymYwN75$lq00y2?-ej;i0feU)UVn$orVYrB>;?yTI@R9R5IzotZQw;k(i zT5j0Ck^ERDb*)(2RasaUU1sU-eK!?xo`Oy5BRnOg;3=qT-&rvx-%@GMvYnOdcOiZ{ zeTbQ8Hrv=D^=R3YSKGX!p(d(rOE#`8-IQNdx~6kyb5mzkDL#mcsa} z$ltNQu55kN&WZwEOzha>t5~+n7gc7&UnLdgZ8c4+@^)6NZ?77Uwwcqm`YOuzR8=e= z$G@U|8ZXiMuEmRJf4sO`PyMbetlGP?U~Oa67%1J{CGEbJ(-zc^qhB4j59~gc|JAq< zZC~2ws_nAc=2iQucC5OocD#PKtlwK(x?^1b6c-A*oh@6(=}%Qx4d%1T!qRc*nj~HE z-kO50acJ8l?WUV{Rs@>Hu)Ag_*>~7Ae>|J6sV&3U7$e>`tMOS`jIq`f7^7V^&eoDF zUqxs1zN~JL^Iz4z+NM?hI?U5i=Sef&So4yRuDzyUjQLBWYoK-mbrownY5tEk7PS7h zpudI1d&i0$jjoXBC@w0->B|n$_876K$>VnJC>+PmrQFVyYcG$k;&SB`j}lwlP+RL;v23YV(A75X z^3~RRYdbrdeC_St=0b0Cz`MJ?d9Qa{eM@_*e|K|zb8F-BrTKc;09+Djpfmyb-leNn z6?m8DFJ00R#K=I^W*|E%>jwby$=ZmP-iivRLKzfx}7*v)hH)cafNo1*nWVnkOq z`r6v}HSXWvRIkBX1>Bxru(HNm=&kGMYOAm1hL8rS!B4Dv>)M-uXl&WNy|JZZrMIKm z=iSxnZ}xR)a#s5~d~11C@y=~)yanEx)_}LJv6%#W>7!AaH1OulxypU>XFrQ1J3lv{ zr`l?BW7%JhXOy!oGvo2M(|}R?C?d+@l;WR7xagmbK1drFD%;{`dG5D<(f&W3UvfPV z|Mi5g3CToX#|gwg_l!eN`_eD!=u_z+j7nSltZGeu?1S+SC44#QE0e#P{PmQtr8>%{ zJJ!wQWTe|K`JMO&e<4Ja8xu}h#^MJBamOoK+^-f~xJgL5-}-g_x1$#6@S06@_zgOI zwuztgqCXF%3jcCF0o9l27tqA_$e+PSqNA05$g;El;^*yVye-O!q3EY_T7-jCvgpp0 z^KKWo6LzZSJ3?Ne_H6{%_|8<&sekX_qQMlUL_4<@^n9l}iX+;2wxEybqnJd!?hr>J zL_M8I8Oz-OC+qB^`n6Klt8t{_;@{9ORi~G9BkDuwLqDz)jlL0vj#f`UZWyip znlb9Pj#2+A>PeqdXrGE2pR>l$=hrw8A$mIBqN2vbwD@mTus_}VF%JIlQ zwxa6qM!oh6rI3H844EFtr}`cV<5eMEPT*gv58_oQD$j}wdka^pFO&0}UDm7q(T}vL zSR+)j#7{NV=gIo%s8_sHb)&4GfqL0xUem3smtE#HM`S&b#!X)z@!>9jK6e zr5nK%=Tyvc8f5)!lbkl49GtsprAJq<{rwwYkLc?4^iPga?}C4+ke>Sf;*YfOV%Bq| ztT*@flys1}zXP(~EayF0Z|?7hIyrj%XUcI!^7Z{)sjJto&0EH($L})pLUQy+{vKVu z{;2P`u3k?+psUx@dt`i@h<#DWiXK(J$;g_CwuB=?`&DW&{V%g2EEz- zH_Cdm{i|g?wJS0QVXrbdNN60XvvI;S$$FA6O~M)4WW9Mj_Q-mYuOE;7x_bF2jmvpY zS1+d;ULP+3lB3t>YH2KUz3Ph{^fVMy+q6zmQS}}U;W?nZs`^uM{5)u?SM&X=rg}9` zC!swmN{*V3^Gx+>K33=7ivE=Bf18P3&BsSf^=dw5BI$1T)-<;G{9O!rd7W>Aud}7L z;d-8^uL|p$^uF=<2U)$luzNOyZ;cf4%t*viw zXSp@_=W;heTu@7Avrx9NzMYM~agv*g#+F^JmqER^-rwHX+9J)+SmTFHO6xm>DOhex zYe#)<>895*lKHiM^Q(IzpI7Js1tOb-va0dmC%(4fVpZ^$p%# zeqVF_WvI9d?R7v$Zwt|H*sx>cYGK-qt#zGEX#O&Hl-!t2t-d;MN2|AvD<(Pw^@N@R z6n)Y65PPQxfwos#(^A`7SHG>XslM*NuLH}fyNs^;{JYz|?b!6O-1ZKChpz_Lu4aDq zx7J}-$8!0@GoThWASfL=P*LtDElpj@}$4HTz$-3BVaS_ z?ULWze~;v+eX72g^}iMPX#Pw3f#5nXnW-C;iYk8PzY}QK+%~bKeuw^L^jGkBAvNgl IMwVImjD0& literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k53a1053964f40eebd57ef8419af84352.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k53a1053964f40eebd57ef8419af84352.o new file mode 100644 index 0000000000000000000000000000000000000000..07473b2e222ad52cb7a7e3db714565259ab0c62e GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+mFh-3m%)>k+glw3JXIc*sk^u>UK(Lt+EvZ2QJ^HjH z(=y~lh%93p3?^nb-_1#?frfM4tE%kAoSk$!2{rIVVJBhy<9y9j^n4 zaqsunqx;^g1_AcOhNYnU-unN)s{U8i|5tVE_LrBgSf*5Ls5vSs(Zi`Q9O_MCsi^&DQm#yy%&RJ3%xw&Mq( z-;QHfKU}YeKB%;XuG!eZ|7&A?M>^O+&>rf^V1?hy_Vj;P!ruO+&JIpb?WsOvo7^{K zn@D{f>PcsX|3~-q52Kx~+1{?h*dr&O9jrLFZ*beCX9riDdUo(zy5}lrpTq0m>=Q4A z`;H$AKkoca_}ISV;gNYK!hQ2jhWD;I6~4IlRCr|XrSLQJ-U~n6dMm?fRk3*q(7;n{n^m9Bsr}d@q2=Eo>PKIZnI0xRTpFawX?A<^3%)Dm@Lnn`i&*Rm6 z>8e=UB z4kYi&{(f>|Pj&KZ*+H_S$HodBu#u+sRM*3&CTX6lXKa)D2)9XRg;DzhC+t57{v~{1 zlg0|)fDdRI;U@JB>5j*#-4VtLUjrR>teXr*RzIS8{-#cRQJ@QHAe*+f(Q~r7*AH25XhoMNx zTMdf{DmaQLqiLc`x?2QM^WO`X8{M}9OJu*A^tgY4k1 z9|Z0HND7@dS=8{<7g*+F)Kglu-UTMttNG8a55i&X<6Uml5iCz6C znyZk{C>Kj+*F2Et{NF6?AAx-3Qmzs3JOZ9az%$86@?8L*^F#6W5z!4M7nB9#1-v7t{kl|V}I;s>2p0H8lUK=Yo2~SMoRoKuBA=8@^Piceaw~V zxtcQ96G~lF6mou{I7GG~n~EDL_zlG~+yee?tgX=zAkK^@^ zxIcyKzti}z*SpUrglw>@@;}>ncJ*E8-!(%IxnQ3Q@LRIeuV4eR`FYsp@bUrJ)nLb_X+j_5ryo}q zg{~zPUm^S5(}!6fhv7W%WB#JITdh~USM&E8bFs?}=mbS)nnnRWDC-1Uj+X@ zXa68%l4P6e#m$-jB(C~4mgr%NSpqnkLX`76nmRfIo!)xfscUzusPk)GM^js4kj-yv z@AS=Ydt!d)w&u3Y?Xh}8m#jZAKYG*9-s<;uviY69VCQ@xT3tg|XN$L^qt4sXvc=ok zlQDr%g@Wp zD_B^Ee+w7oJ(~XrAa!|=Q^<7}h|$sH4fq=G9zg=W4qxC2NCYbdd@Xl{w?Pi?T_H%L zufA(b-R6L|)hDcD1$gJ1Aed$iv@`^~{w6-@{DJn)_RelUniHsSFdpN8D|`xv!IufYaue*5Ol9lp*yR#(^d zL|qiKa_P!CfvL;GR;}6F!J~H+Pc4zZ+*LW*jUv-*AZ^$!Tgws-Pqc#BpK_-x5MYoFPOh*u6rF_<`*tu#a+!UjqZ-_j!s{zyS1yM)7|LX+}!4KudS?Hv8Ha_ z(&{zkm8P)a%C3#4DYz;PVd)uiMxj!0ihh zj65Na?$gCRZkatJc7HR+Iku)$X5VawUC{PXyXgBq8lAxdaWZr{o!MpGPL{r06s_B6 zbVFTdZd0Aks*HH*lj=5fYr8+f^c6T}s`xuRX#p13*kSI@zwL6YYH54cV#en4A5mTgbqk0v`SK1~HK&h{j*q zt+P$Ui{4`s{M?5M$_%{DI@!U1-HsM4GO$054J}}~LEC+vH(e~E?@wZH9AGnp3AW!% zd2<`D4`tg6gSw6Ij1eC41t&(bQA?N%ZDYKd!6ZFOeg2PP&=r2YpFB(bwZ&p3S& z(F30wv`?h7PSHQn^JymN(LeNIBXRsyA8`J2A)kRB{G>^Hfgkio^jv=#of`3jx=!-> ztHL(oHz_y?R^;|gPMsw zS%mCicfg)QmF$5XL`wzoT})$N=^h{9<{*5o6Z+A|udAu4;qtzj4S6T)dhd>e2>Sp4@&Wi? zM!*&!kBk%u{L*fXT_y%J;UCKbdKy)k_A%$j6%f8|`xW$o{`G*3ISEfnAo~Vq@Zlo^ z;D0F1o=X0Vh0?zX`T;&8m^*{kP52jYpBscZhHJFRB;Q~*EBpE(V5wmI^F;s9j4P3d z7Nei-B>hFCd`r8rOwh!Z{%qn0{#6)w=pk+|O^jCT@FOqO8|n(q;tb&ZhC^5`VDIu5QO%KHnagmdgIp$z}xS%q09Aj$gp`bhA06+QbL2 z=nNNu-VS=q528Qn<9G+^ zqCNSe9eC+)@NejMMpVDb-oyae`yJ9Z-J?i-(|C5}pF<_kPp81k`8-!oO~;@ga%gVQ@={^sEN zAb&fn>7)4*-pbcw%)jNZpD1GfS>qwu_cRg6^}8@QemvQqkB@j!mYmCLDjxLiz|ybc zm!IU*WAf!V5iiAOrT1}u)AT}$KYklbM(a1tADzcbo$&Z_!&eo5uYeBm^JP`1Q3p`( zq8nL&-`~x2+VvzBO&AaP*=YGCZESs#`2q4b%%6G0k1D7i+Yb9Z2>|5-@E?l5uaHAh z`wvL|BeD1neL@~feAf-}Jv&Z3M-EITk^0pGnX!01v_M-Puv6qV#CKgM`||jX{0IJw zkB-Tody;5s^MQk-UYy!1Dow|D^Az^apPVy(Thx~jzK3{m)1igm#P`9S?nZ%4>GU{Mxfc%sCOTlnaygeg04s|1*rmqp7fVvT%h;++{PeMJ0 zx2$J;eY3k{JQ>r!D;d^n?jQb5v_FA=JWZ36it-Q4&oii!ePMs(`rsL zVcKsvxc?+a_a|97QTlVV-*6;zdy)U}{fV2?XAwQ{xxr*d)Sr;wrU%?vZuA}@dh(~8 z4!%BOyfmVRKV=&IEjO6uHnuO^>2}k6+9y%IgM8XHum`;I^)1bX{LCHg5Avr6rs-2aA8}zm4_Bt92eS41sXiGK2;*bC zoGjG&eiQw{zM14Mewq)Ew=kHQ8I5OFeu)2GjCXy0X#GM!Cj#N$W`93G^T{|Lu>P3* z9eC4x0N&*9z#HZR^phJj`#=1B8rhfUJK#4XXqXR>m+(>lzo=0@1^+^!`1m5egoyl$ zxlt$mo?U=C(go^}k<)H{vrUnDG-(kKrLN_gAhD$V+t>I1u`=VLs=Mk&pAqCF0BZ zN^?AW5&7dzmP-B&e;UW$tR#O*r565Dzg@(WFQP`puwD$K%HzQ_vR~?9R7E@|eJ5f& zVvOgBiPXP}=LtbEn^-kQLH@9-o1p;;osVGub)XwdttdL}&#^xihnTcKe3Sb-X(!?@ zmg`yaB1tazOAqL&bYc?eA^RWDmPPXgf&WR6I|Jk6!@qjLk9O?Ak9a4%p1?Ri1b74U zWv2s21t>PG7bh|PPGJv!{rltCFDTx=&d>Lher{PF?AQ7FLH@?q=hG&8TJtkgJ_7sg z3os-o(tg(SjLYD!>zIS$A^007_hY7fhw>Re2~7D1CqnRQEb08q?y-i z%vZuk*XuF!6+XW$`3h~usR)1M`w3-#*c0cEnUjHyxG>*OeTw-C`9s`%W%?NTihKHv z^OY=~uejsqE08D4P4PXNuLKSG#^`(n@zk9a%~yt@zfYczWYPM{<0bmDYvM#54!xLxn**h3f>K z?_&KE=S4@>vv~0g@gn^{c>isRU&yEQ0RQwM`uPLlqwx&^06uT9Pv-k^?jL_n`|;R$55~j( zDRzE^^?&q!{Qjvoj&D=AzuzA}z9D{0xu5U9)P8()d^4T@IV2n&h*cQ*FlSrh2mP8wUraFPugGu9p1!sChWyCz z{eU#Kk`9}AG32wfB*^`AekY4iJmha!fAD!RKkpUi8HV%lBOtno^YEiM4+mvW;`=yO zX-c!~uhVqw_u1D#!2KEf&l~cG&%@yl;(R@eK0oE3N9y!}c_P)p?@G&loA1}6@;MpF z$Ill@NU^@!$w7Gg;dJVsK5miyX|ePFyqlBPe14F|z6gHAeij|z{HLt@qgZ}|&ks!b z$!Ac9RQJj~4cybfJq_H`z&#Co>KdTKbA;Lfe9M9L{QTPQKm1YTFJA7e`t8QQ{rjI) zjQgv`kIw&ft)%Q%VtH#|oC+BEiA7lOYn&(H}|JS<6*ElL=b;4iv<@EOTrLfaK ze)adii@alJOKVn7KdP^;NmyNj-3UVZPi9eE&!;brYMT7gBCn$wdvFf$R|ud4)yCOV zuJ8z5!~=D#uHkQ%!TUI8%I~nfWo);$1c6+uXP}Q2J;#}-v1$NpnnxtwQKScV+PFlt zrwl;T=JNj3c&{bcQ7=v{PNY{+s>iOCZJaE40X14EIE`_I6$4TjkEHGW-Nx^onpmPx}z_4F1*4v;qxSXPc;PzVy~76mZK#eKB`z({PvW~ z^Wu(=NAfQU7>L(D&-IDI6CW>fE#RS;cq@!kskpos@GHW97p2s z_)4LT@-7-%tti$NBgWJ9G^qNGzXHYD@7Ap{yA0#Tvu)HfK^G6NtU&#YmfA4#C~cyF zQQL{}IMY+Bomcqg*OsJR>17NLJK=?fLqWX2A9ZLr!1%JN;~Ooe`ctSYCW#({>Qsl( zMEpUWZu!y&JlHVNzhkK{*35X1rCw#J(|FWBUn0mu|s;wS7EoTWLP;b)(MoI5Eh|e>{4X2S^?Jk(Qd*E_3}U;FVd#+GVq> zUHsd^hPBH~|Bj{oVr0$6{ydg?m8Fhs_4;d)rS3o(O?A5J$L7jS? zjyiQkx~5TcR6VxOu<0~PQR-TIk?V<+n;VUCk+F0WFIOQ>ZzUISgI$A;b%ZiNVudt9 zIv$n9#>w?e-fPM8s2`*C#R!IxnRR40NDTOUu^#$>kD1o!&ti@IG{*ewqGmkqKRa%$ z_8hBjoMHE3xL8BR(%C1)8+Cp!hMQM$5i3-D7#(+Lj~V|Y&9b`d*43Zlg<&VBX;+ZN zsGXdQxN2OJh5jIMiAIjaCR@Udx!;{|Zy)-HnXB##_RL;Ikfiv zhpwAuYU1eALMuP<=q>vpQ$94ReRR*s=U_r~1)=2!#Y%st)lK?|v+`NxiAS$CceLVy z{*K;Xa{kR{d7rnlz0bRTP4mgPV-vAH7WH+&=rA0+ZB@n`fqz!w5uahvk*wt$UHG^OKOnvzC$|5w{zspTa|sOn>l*LBNAX7A zJ2Un#@e0lV%x$%!##Ce91cEe{FEmF-b*Pi79fRAh-&>5U~$Ojo! zl}0IV=J`KypY5~J&7OH7=f&^7FtL?=Z_aOiTWkBPnRBP-{nNA`A4p3rZ%J8zAwLJt z|EYr1Zv0VQUM}oW0iN*upI4SKwy6w%TaMTDp8rD%B3sZk-m`rO{tJ3TvEd{r`qetr zm3AKkoFB|{s_Vba=6&#^ArsLO4j9rEE+*>BWc?&*_Tw zqx?lZr%TbIIWOWlU6~3Ty+)tYB_WDFN^qMuG}HMpou$1;)*GT9C<{PrWe|1R1jK9q zW7+OY7S&qYGrnto3zarb`N|4gAwLR_U z_QYv_)Y4w{Y;8YXD8%nk1oC=Pw*NrZX?*&wiM2iXJ-Qgh|A3q#ZbN&zRqd4>51?a$ zc?fAJXDm}pjj!4%`#*ukH)^lws6EkH+tWNJ6*|lo*Z$d_C(Vos?m4+$pG)O zBnpu?i{+IDm4*MItD#Qd@51YdC+go+WP6VZXmK*&Xni66i!F3`gg-AoHFO-$ettM6 z{=SavCp})YJM+ily66FM&BBG`HxZ%!rvTFUC??dz4FX6V8;hikg4Uw_?l|LrMYhkC zlr%mSi~5*Pw)a@ZKPuZR|E2ci($@BiFnOPeGyXR=f*8fk-x;%xul%S2?R(MQV;O%y zwpaeA##ikW{C)u9wSOm0dzuo~_Vi;>h&q0l*`)u=vb|(Kda>$%4ch>85%~Xt`v{0o7HtC-CjI<567BPs?!8`KsovaXjKx%?oQgRrg2-F}C9% zIiD1rx~lo5uBzRZqBZL}ik23MQnIpmVM|v-L2=85f;BZ;i(1xt*Ee=;sIAU#XkA~k zt#Ea_x2R-g$-3gEhSp`B8*8gu8vJY4wv=yQ*08V%ur)P=FKt<}dP8kRQ~mmNc^hjh zR(RJJZfmKitXZ?YY+1{)$7`DW)oWUoFRXdIe0zEA!s?cdt;_Q3my5Q=ElXPJYm4)I z8+pGgY8Ni+s&8GA2Oj+O3m>W3TD)p&V{LWEhT2t6u4rAA-&j`D)wsN6JH}`OA7$&x z)^#;3T+&VBHLkB{FD~C+T3@g(ucd5DMReS~!FI~80wYPXx{)W~qOScwPcW+o<7>GM&`S#ik z;JdzT*^>?3h0Tz+1-g2&zN9eES6l3_Z}q`G>niHoRzJCtY_NP?v$tr?%Hp!u;Z?A*Yvb|?e`D*q zZs@Okb^f|18<$tKY*@b#_O8y~(6*+gX``^$*5ZoI=sT~mwxY{w7jKFG$qmhm9#j6N zu63;~&4IdYg$3UHyuyMdk1X1}D9`7sZ!CP&w|ViR{3Twz7vOhG1YJY(W^aS4X!+nc6HLgVlpRA63!3=qfdDp&%NjeudF0^$6(6V) z?|IK_dDmzZr^I{Cw4mn^aa0SNVO)h2^f7+SaJdswKwJgAapx_N?YLwZc5X^83X=p^ z{x9^!*tZ~;f{7l)JAJ8~=sgms{5}kY#-m?(9r*9jJ{~>oZ{oopy$<{bfD=Ebxg8i& z^Gdr8|A&!y5q*_3!Rt~E8Z+LwZ%TRH(gbRLHG`h|xh&zT?LrAxo77U-e~#>@N}5dR z`R^qEc)AoXYF>R+^7)E{tNHZ0>}Nn4P{G#&o~{N(PpeUi2i&-2+-ZOtQ^lha zPUGSB-|?RfHaU*-Zv!N+RslV+$!%c4E-4S|HuG0%9|h`g@jwjO_lSO#x>3tmxPa+FA`4uC|*c_O$N9;;^A@~GQgvM+yl%aIhc~y&8Zpt zf&p%nt4YF@9u$3xgj?l(QNpeAmdcQ$#?6!Ct}wuj;|3*MjjQN)O1O30?@73IT*_Oh zQ0_ugO6iW`8Wjci@ET)H@~Yq}ZXCA2ReTt*z*QU=w!qcWtIj`F|7yLT1^%cgc#pK- zatmCo_kIgpt@li%-5quH&28R5H^UyG(YwOi)z;ATIRDAXrFbASe%+D(#uNQqBfHt1 z2X$1}+}4brd93f~xV2_&U9EzlW?u&zdy^x#4*cgO$EJS57wEuGLr77Z>jRLx%-1Qz z#SfRz?^u?ts+ou1lWW)}sFpz=YkZwWo%sF9`mRnNK?OY;nxKu|Qr{EJ4L%cGXwTOq zf9`H<7B^!&`uU2wHu`zI6)V=QEN%9-_!>G_wl{XQVECKd{c#(#w0j%tI@{|SIb%W~ zuqXHwRUwN}!e#9NMR2QQvE0U+u$ecorK7IH?`s(SyP|>iMsKH=&FAmJA%&GzK$239 z5+w5*Iy&sxis97@qP)NB?KEY&K5QTf>-Dw%e6hg%-RiB9)O1V4=Ag})I;^wf-$LO;) zigz*`M{}^h1CYiy=GN5GC4;YrDu$xQSJ&rdd+OU*s#Nf9G(uzJD=49fC#w#t+60B- zSJ7MR|4oj6z4jkj5Q=ZrUpj=h+JtZ`0*b1DKf<;Q#;t literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k53dfccd97af20f0564192076fba638e6.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k53dfccd97af20f0564192076fba638e6.o new file mode 100644 index 0000000000000000000000000000000000000000..663318688b81f6392b5106e90fd9c0c29c85e14f GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAPh6{Z0i9+G9V$45Nu{eOKOlnk3KEQ zvYl5heFm??b54Fc z+;`%5_)+I~!p9Gs2#?G^8Sa~ZD!hN~>F}lfr^6%r-wQuA|K0FIt>?qXPrVyXnfY*N zWXFNQr=EU#Fm?XY(43a!;}U3=gvy);2QSQjVsPJD^t1mv;W@1s|J0k|vh$b1 z+ux({{x}qxkMY(XL_a6Ob6dY19s$1m{HgGqljp%Z_47xek^Kh;pPK*lVCd9|@CCe@ z-+MDW_vFcNne)Byh56?}vph7X^@+hg=%VcWarAv696E6-T<$y_4$Z$5erhf8yet$t zabR%n$tMOw;Q7M*Z-@8Ie-nH`50JeNujDyj2$eyX7ajr)^p=wN?QrPbH^V7gPlo55 zzX-lAhR4l(Bs4JyprCW(UcR9vdrgz($(hQ&kV2nxuKIp0!QtBit696-4b1oUs2S_?Pg3EgCC$ z9X_CGgqzeiq&ps^c1Ia2conc5-bVEGL-@%D6~&@<3+10bflMW9xXot`K zt$uBIKkKJ?#I%7&RskP*mi0&ene|_KMZfw^G3!5fAG`W5i&_8qG_XBV^nA=r=XpnUrM z^{hW!%=(!=@Ceyd+5hslU@Q3YyYS=pV6%UN{bK&W4qrY5yBh4+G+pR}{Pg3B!qBy( zqAO&-d-^cz<2g7F{FuM!?N;lR?^geV#$4=j13E#`nWj;I56U{hR^jD9=R0uq;FW|| zI&;ymqsk6npv~9P!772tC9IB_r>EkoYs@y@rs+1u?BlTH$zpRdJ8r)$^4da$(AobI zGD)&6_2TC2e->AL2TSy@Wh?<4O(Dt!9ZembflhBd?$ourRn+;luA`}~F~}CQwRid! zv^}<IZ!U0+<->FU0_2cj@f59^sbr0%C=mf>ls)E72{g0+I5ILLS`a2 zzT8Vg?xi7jT3Q&#_3x!2pR?@Uv}}=etbm(OL+<>2j9R4EV}rGzee2c^UuPbxt806# zE{a*RVojaE)a7BT*6i)z(L0K#mdIc3s+{abk!cQ)Htd$IWjTi@TEXm3Ieh#mcr7;T zHojvwqn_k&oD^Ehi-2diV*T{w2!eXBG6T0BNe?n!b^-1_nihpiF#EcsS3@iM?*eYa z`GvJaP;w=Z?x}$9-E>V$-P6-En>iwTdS*NLEn@_qXYEAR70>ps10dSk&(I+VRq;B4 z7uDC+l$Defx{I;@$2p29bVx#^OYz!DAb)2NzF6GV=ylWSLUW6^v$?&^-PzvW(lO7C zneFp-_}ux67c80QuA$31%bjc6D^R8+}`w+kEZ~6&0)3*VU}3 zT3=SNwr)l3h81f|R+O;$74F6E`QAW7le?o8=K&9U-58p9rSlbh{$b&D2YUi=`$7jJ zPspSDba9VcX3vP--^_82ttpk+x7uMBw0+nv`o51wXYfFr3|&rVc3HQRr7sso>oywQ zP}iB;RHw5lBcA%Cx((ghrbd{)8plj^HqAaKmD`l;O2s}Kk%S}OCS zdlwwB5-+<8G%y(c^Ot?P@G@K024+ujF!Jsm8}64%_zoOxwc@o62oxHJt{=nq-pLlwA)P9qPYtkC z8r)xb*~Wg6&3gCSBkVE&#B`&R?b(-T=i_e5PG!g4>_^=~2KFTI=&v`3i6lWZ{)%p$ zZ6RLt9-H9jK2%U<;&sl+4hQUZv|y2egK2DNA`%Ke>-Z z$R2hF>^W4y?%zeUR3P7_H1?J5@eytg!sj}nAANjHb#*nD_l<1GJ6R`Ria4Mj^q(7? zJemA$JlDUdvokO0NTANJr4Q8{_=Z^qO1>;{J`UhuTiA1y* z{TwIhFCyhz(T!z-Cbslv6F=~;g200ha(iiFyn?{2Ib0rUe>x5Gb6$jlcFg1R?f&Vh>@S^cW^nE-!q4USh3tuLHkVYJcs~}M z;X=^cL65bK=#M!$-j2G!7X$AG-fhCWQ5X0Xz)u4{PvD0|`)Q~P{8xd`1l}X?v(tG0 znWzhVvbLw&1w0l}8ox>4U8oCuf-dCQBJtUh|1A=)0Z;t<1-^bE{OiF7Nj~oXSS;D2j+7wpRY8Zf04$Dp4S99`7iLL&;Lv|7xmHond#xLxnxi7uNaT~YYUOc`H&Hu zIg{ca_s5@8|Eo;>_wJe=^QWPMSl<$C*ZTiI(l5|Cyw2 zUmn$~E`TRh8sc-b35rnoNKxCj-tiLorxB;cUvdr>Qf6?_|sF0Or%{Iw5mcDv- zAs;x4CbWL#{0Zc@8T^gpcUb(*&f}+@;Pj4|zd5)* z$luOs`e^=yxAOHE^KTXGCyJPV)_6$vJwpU?{VoWOA5Zq@<0D>_Cg<{+iU+;Bu=K0= z{n0z@-#7ps6>3y8vG`)c0kKYE9(fUpEN9XZUCp^B~@Mns@mqCa4`I4&Br~{~X z(Tyy?AM9p2?RpXmCya;uY_$B6Hnu*=`~dkI=FfcMM-|kM?}Ysx2Y~Vc_z%V3m&qZi z{Rbrf(O7(kJ|T}KzUzkgo*gHiBL^mvNd4-8%vii0TBxlG*eP-w;=8VseR+IG{sVu; zN5|yPJxR2_mL@9!ggwpvNBXxJkVnrC_)y-kuR0hq@6@)7OYkK;4K>M7m|fC!rq0 zTh=qazS&(eo{Z_=6%6Y&_YeOT+MmEb_R-{|qWlB%^DL@lU)Ue{ekQr2h=*kVDKY!Q z|IPDrA*bi|q&We776hm1wA?b%-2nBOf5iCIKI)Gjrm<(j9LXYQBWc86 zK`-b<`y|SDkWbqNo&c|WeM@s8KXXU>gZ$}%>G~AVM_icC!xgFNfoy#$)hA;DVSJ32 zlZ86pZ=yfgHdA78|m5Rrc| zH|nI{a|=;NxX`{7+1e z;nRRmLfwdW;nYL5Cww~UMtsHuGkyZt?Md8zIK2SPtK%;(%O@^L=7M0~kW zVUA}nB7fY?QpvyJPvh7d735E;)WTolw~Ki4MbxMm){9|Oc|4d-_Dg*ZRT0lg--+0c z7~^?jBK5E0c|uUkCRT+}kU#9{W@vyy=OfsE9qPtXD~eA0bL@}BAtvn)-{AgE+KKp! z!;&{g^4=p?tvr;)yiYZs@pJB# zetA9<^}kp?!_OzqrP*(ozaYL{H-Gt4_#fiiXPv){&Q~1CFME-GihKm~&E+6})k_-MRS)`OkmId}N*}evRILQ+9zO_umJC%uW!Vf8hMSwCm+D zF(%9?BUw0=YQan$ZnC75-s32&kaGk*O zU95lNyy)mg7B8M5UZnp!@4rp)3;C2D;GaH3KYt*6G@gO~bi^~-4@ELXes}Bf%(UMy z$1jX;+HXMLrg+Bj=KY6hzX7}{z76yI`xC~q%O-!Z#xs1L6th40O~-lCz(J`W*bn=e z$@F;)U!Tw6^Tu5hWBc<5`Tm@rpYnYBjfJ3}Y}k(OV*78WbBz5r z4pgYzPQ0tQjs15zR!cA__N!>`LOr@4MZPw{c-{njhVeX#{6CZOO|k#v^C2_3|K$5` z@+XvNyqi3kxlyNl_54EU)1@2q3Hzt(7JZubBV*^Uf57>EDZ!k-BEK!&cWdzt`H|uK z0cmUv9X9b|$Y*Iuko)QUP8Okf$ltL3;PYaB-Yd>C4CmoTL39)6;m2?u4$7Xy_i?P! zlxEpqr|H=5v#)`G`!n{RH{=hWhr=Jl`Fa+8e#$?O)ae8BM5=?|HJ1H0->*gGb25^T zpD&V-VtuufgYfpl>C``c+#>taV(0&PHz%+8{2+~e5&Ve#EIPpXPigl@vHS#|ADHr! z&!7&e?v;BQxTk@88n~x{dm8xEH9&{w2(<(FmILeg#kJpm_@l^IU+Sy;?dHGxho6^^ z`>V!}F8od8Pyw6W_f)T|+TAxd=U87KWBvB(XGh-u=eo$(I4Wgb!e92~^!D|ouroh? z<@djfylrPIs@Kgprmw3`SXYhR2txW#XH#6yr!S6bn*7otucI1!a1QWS2%rSj#@SP@ z@CaSV19hyf;cu3~`#5LH@36gPY_GN)fn2L+ppO+j$C;?HDgbPnMj7sYHA>xUTaphGo zLVeLowBx*uv+{;!EQa5kV#QOsqc3wVy27^L^CWyvH3bP`ua*dwqa_|bs#sV2_LR%> z;*O6;@-GS)h}S>Q^@+k0A1`t(;Gvj!?W6o}iRoV1ARc{G9zSF2G%Z<&rlT+pA$w90 zJsWhI-RIghJzU2qm16`dGuFLYg4+LjV; z#tJ#?z)vYqS4CBVLZPYVC7Z0zjK>duC+A#7cZK8ou z+llcw(^IRQSNP`FmZV+jWeg8H;f01nLA<~pb!a%i__C_w8!e{#lc*~ui5`RMREN<- z{6U>=`O*hG*f7z*ZK*HQ%y^HbUTLY*c+@{%BG7-WIMF5$AOD7R@`zh)^GWFoQ%mf( z+DE)LH^ciGzM_sb+RgMdRv%V;ygolWZmismRX5JWUKxY9mkGBbKfj@0#cQu72~Djr z+neU4o)n#zZoz=HeLQVjX+H0Dqt5g=G04h)JbIM}NFDo;mYUZtbNxx+m085vWwWeZ z{M*8YwaZNZwx#_tWX;C@JeGQ;rH*X%`fHbQwHLlOR(!irG$6;3Uh;ir*QcY(11hw7 z2K8)1o$yrW%GAcCBEzE{fK8hLHr8o+jtjBbvxju7Auwa5$6IVyKslW3)vbC#oqC;t zI(0?5rcrZLJ+{xV=`>1V>IQqE>#>xZ8;x?2v2+tJS0PSsB^PjmU4xEwgfc*4g)~As z9+kz$$@NU$YsvGdAEWif2!@fFb!0b44ETGo9{PZfnbzpfVvYPX#{BG}W<2gcJ8rD@ z9II}eVfSLVSVP9r*(b#tb$%~~n^$oWD^z?K9d~Gt8UG~BvbyWm)t};pVJD|)SCGZ1 zot%ugYFv|r{vdIQMvlcMTf&XG-+|+GwD$go zuA63R;^@;tD?jn*E&Cx;J~XO*bkE7>U_x{Sq2&j~N`I%-P5Ozm@>%7HN3S+_wBmyP zj^1B#{>^83pSQEU&%1t2^U1hl6R|!P_4bb1v3cM@v6#J=8Ka(>wQuSA+7j%w89)2h zuE6`~FdVyWRmL2Fe^%lVpJCCFtmPbC__zr_Aif_bw*RsIN1u#y2@L(~8t=bH@kZY} zGxjg>3eEq_ZJn-_xp2Nf^GTF5b{yEmd4`d8!)F*RQOzyao>c;Xk8B{&S)6#t2N_kC zL@96P`9E==?X%g%%PSe%Qi{JV$Lo5}{~-mDE$ABW***mSCB328a1s>#Y8~oI zyN>|Q59T@5_21_3KKRj)iD)^8kM2P{r;EF_iC#RXE5ih%%9EHHL_^(%vKM4bcyj1t7LEh&pWo;pYF46BQ`XfyghipLW9gU^fQ5qt9So@@Qm2$#2QN>WKwB#4^GjXw` zO4{$s_Qwrv>DH=u@*isNv9Qr7P3{Jc3V9y!Zf##iNPIn0Tw?EEk?nIONa@(xp7wK3 z#A$!b(q8p!Z9hXO#P3lA@_I_P|3KDheEP15wLSSgx){a(fSe+3LwmYa?Ufx5p<{x1 z2x%y1EmKU5ui7d5KZeFPYOm<1J<(a)(>x~?SlpB`a!iz-9C4-gMAIXx(RcjG0PnLn z3XwO<fcmkdyffdaWdd&eIfpfEOdB;KQBKsbR5rqemExn zzK-lCJzlgs`^VzC+ZlniAIb^kY$oI)0ehr2osZy<|UnvFd+4+W*%$@~ifjWlOjz zx4(6K@(I%EmU>BOxn#-#)m618@a?waQC!YX%W%>8s^+b6JmOW&3u`-7_ecjZw&Nf< zpA?#MgHwrucjZ0y=pTb19?y0Lmk z!Mb*DVey*cnxdwL)|H)`YpYrs{OdQglJ_0@zgX<5E*Q*C)u{l=QS&9&vL zy&DU5w3Js=uisg^vSsC?)lL4Y^)0IwRX zyx-Nei&l2kw=T~E5B~Z^ORKjRt=-;OTh+0tcJ1StdG1|aKX-#QO zSHq&^-85d~#`5-}vYjRMi)->)O1G6q$1UBq)?2u|eMRZE4U(pGQ(H}I?c$p4H$}H7 ze=NF{n_GiTn-^C=AFZV%^VZFzyZrW&McdYSi`M3EYTdSCdtp`gri}%GxMP;>tlb2@ z>q}QY-q2mp40&6itHn*|@FJTU5MeW$D`d&7~{4Hnp~NZ7#)Y@tUs9tIGY2tu@`y zU)j3+n#UVgmA7o#xCr*H%HPzszPf3%u-EpY@~!AQud%kg%W4;IvH$T+%}X9p{-&;V ztu4)gx*Y|J8@DzzG%kPGyLD0C*1UqHOY)a5%6oX})_U*K#mjt4>jYgx^Hy(z?|MJ6 zvVs4$3jL}Ie(mITDtR|}HhKQyB}Z1LB%%xSxl6;~w)#T5zds zNWI_ltbWA)=ZXK#@wL=%IA0Jw6MZ8m5O2H3pr^5p80cS9_p6*o$9+S8!7l4W_XCB) z^_04|&`ksUoWkK+qV7wau4nac*a^@7@K?gAUtSFOTthILZNU?Li2)z|RixFgTeAtG0_ITy0WIWdAv`pGs*m zrRTqw{Nw3Tyr_BgtmN}$30L#!HQCRAG@ycS1Uy|0ik{Y?P6bc#pYk0-O_caa7I>b7PXS!DCD0cod@A6^E#qzj+?XmJ zlW-arr-{*WT;9>RMtF&olloV@5$S3PxAs#j;nOYsD1A{s##5pmL;v)g6IGrwz>W0B zuLHkgfE(q_m3p`GIa|W5@(xG?S>=6Q!ma)MOv0`5{?gEok^hejaHG7La{kb`#u%A* z9rzImCw`2p*>e*9kOe+qpf}q0H3_%!PfyXILi|h-xBPvHgsX9tFE&ZI8dt%O80d}T zJ}2SUaepb{)^StiyrprC^Tj3Mqvnf*Q$LCq5@3q~E{}M)Tt^J>=pXk0vqTQ2P%@3|QbQ4h&o1YUx$ypQ?Yg-p>YqR1~~N z+HaKwuGV|M1+Lb6CerSXy87leZ=joDkI?8{?d@u7XnK_Y=0Bdp^x>x&caUo{$zbur;nh59t};HJl^WnHET+my)C|m&Nc0gT`d^?CU<|_1}*L0#=6e-x<<~J z5D4rEK1Ef?Vw7-Ydq5G~>R2qd@g{8M4Q%VE>+t&;M*ps8puN%C>17M}yKqQhrPYw6 zgrfw>f`*RH{G~UojXPv7eDlVQ`-7>svw=6Y`Q*EKF&{hBZ@1o|zw9a&*iZPr7ePd( zTBe?^$OLt&%s&)TF4L;YXrE6onHEIh9(i|0#xaFZ@k-Ta;f7K!j{?Z6f2X*4PUd6u z*&4+=8IGel*xv(4;~R5pYUz@}*FzOUQRA!Yv$8$)Z7h{4csClMvGEm@(8QBfhm~!D zLh-BUt@Zyd$G=|tk1Pnqx9Tq)!dq=ZxYYqgVZK^vx6AgUk+kZfq1|$UH)wE(H4M?EJtdV3! zhO7vY>llN<)Z80;HBL$6`Z6-4&TU#pZB`>UUc_-Q#cFP@SDLj#WJ)E#E}r;0kQmSX z{{5Ia=V(BH-Pmy1OEc%}|Nr;i|7-96-+M-9zFf9qnXYL}6^*^ZVl@S2EVDOu@8P!| zmI_KY>!}&A)%`5nZTsK!LffZjFm`CjmfiObUT1CDbN?OJb9ns+?$LCjqNVHAo!<}r zW*ocn;d(vvL6t3Z)y59}UmNQ?(!maa_Ha)IEBtP@r~kte_SP?Tc4$UwPt6(I zMC$8sPdY37Kf0%X80~b;_EsIn9y$5cVCA{}gWKPGYH-D=rv|^Nd#-@?X}k{2Iq_n+ z@A$FsBhGJ!kL^Dm9+`h4+&BMZc;Bj1;fwoDg-7ih>ob6UO~K0ohRc<*~Z2|ww4Gu(XqVz_U`d!Q=}g$^7T+;)8b;NJPi z!j-3vhnw3@h5Pou86J7yV)*>_i=Zh8RX7g}o}d5t;NDf}XWzHOb6PR}$v49l=Prh~ zy+`ByVJI{oM(q!ru>U0Zm+*nj8Y_Gq zKA>rYo76X?J078SM;I%76|fxMM)dUq_{j%VC7}=37sKy7SFY}S?Woazeed@6y`Jm%-jOkU?})|s$S0Nm4Osk7`Rnz3@aoPVgd!!A-x|J8 zKXm?Ii?6utEbYQCi?58lRM!8iHLO1s^W)4cc4hbn`jyw=*F)Lt%9)Z$hlU!oLudY0 zzdF2+_0v3J+Q7rhfsZ`H`Xm3w`Y*nsUwNm5^`E_mUHSLLtp8jZyK+8>T}c6L5$4+P zo`nA4e`Wo{_i2YN{G)#5!aMBH@ZThahQGxQU1-3XI)U|%{BSa0iJ{^9*rETp548U? zDRgH2?uTivf#*Y4cfM+xOJaQ&@>~G_B+D>(r7_zis(7zWlIsZ-zy$d-;UP%azK#q4I zhf#i#VdRy>&&C}bKL`=**N9nT z#y;>wx@t%$4c6k}}T| zN-Zb~IloXGBHNHn$WGMXJJ9P{=+A}t8A$f@Z%6;%#C0`ZmALQ0bw94h@%m@npThM& zX?)o0oo5n4HrQ49pKUz5@(%RxnyH6eu+IhfE!pW;umRcpJZyeuG3y_K{m2f=r{7=0 z`okrxpXmb+lTDTVFMSKPf-k=dKYkB3`%l;}<`3-9rGv1m!H!MSg+9nnKdveYT}>*! zO!m9053@d=h4a9V`HS9ewqE^i?LTPD#V$3V6BM0k8U^^EtP^ZGUJi7=9aj%tNqD6* z7Y#eAZ1)A)d@UWU8kk(d>X>L}zb#w+gz4f?L*X~wP=hwQ9rnbf)ThP|t>08kD z=z`Ae&23xSWA%nES$}jv^roS`)$i?O3p#zl&ILlWx`wXK7H>yKowucBtGBbs7pQCS zc68QBY^SfSqdnm73^XrZw6)dO>TPM>+Lr%ddw$)2Ed~SB!mcOb|@723i^dUVjsxbpAklXM1P2AI%BZ-qz9SZR^at1KhX4>u>c1 z8tw=)!!%qLt1UTDE}h(fM!+3lLnV&cXF2qanZnApT%cTsd$cZ! zSy{HSPGIWtuvKgJcJSyO#Zyb^eiVEIHtRON zV>hFo;BcH2TFMK6XSibh^yCPFdayDBw;o9kGG2B9?mm(hg-bB|x};Y_EBfyMZo~P7 zwM0;IC6Ml^fbU&&O-tR~(=(enBD;HLJNPYQ1fOT^MAjA0_OAmV+S*UkAqZ9RI*b?9 zSFNikttfJrVE>PE6jA7qgh(I6YYTz=ok93wNmrxSO{WXZE#A)N_BMBCdwWaAJU3>x z&)ea1=PzDRFweb?E(;2mvf{4hmPU6+cSonM)!o|F(dlmVZE0@vxz|=ztyojHuB>KF zMb)ahvJGp?R+W~OviVi+#qRmuKtq$eqZQ`?4|&}fns}x26@30-;dMKE9B}(W2P03& zqx*Dmk6UKXh~3}FagMDimD#u0VHdQ0$S(T6hel`cK%5L+N@sRix09tW6-Dbd8r@LW zncGySvnnH=`lPxI-P)!`n7#tXOm#NRJ|~silpCma>c5Fvtgdf1AfTG^X7SJJ`Dxgmduv8k{ zUwz5OexA*G_t_)t5&*<>qm%94n`r0bZp=<)$K33P-9iTT1n}suH;9QOK{S3@x6U>b zFM5wn@N*9;C^PXo>tu%lb~{?I$iRU#HnfoC25t9z@1GX1h<+f6y?&6*3MSZoJ>`w< zygrm|FAVB7!ZSvA$QPU#$wn<HnJeb7J5v@>epAmxlGtDxEBtfxk2;t44c?zK4K;r~`a}Ou>}3(M zhur~t4pp&xcM>fX$agV~eWiPRgqwr#xlZUuAHS})wwBBLMmFS~tdlQA9MBK?&katV zO#U{W>tEE_>6dh+pYp7)OY_2>=pXiQawjKyfSdM&FY3KJ6C&&b0LTa6e;EN= zggi2`Sm4XLHFk*@(1d?159n!BW!lG_A6G#5y6soc2m03oHs&NeDS_-8oXLle41oWk zGCm6Od3&Yea0xg5WcJ>Jdcl4=w0#iBD@ z1bRE@v9=NYQ76aSQ5X0U;N8HxO?Wrz0$&FFG~n|Deps}hhPuFi75GfxJpw;FjrX64 zy1*xEySrV$V-cnCn*`p4y1*ysLY~bMpDp>{Eb$uf#J^wQ>lebm?!TYpWX{`7;O^}oS0$={zPi1paAKjms9{!q3_T>JG@yNe66N#J;8Nrz| zDgJSP{2BGX+|+;X&gn6K8ajaWEx|_q#utTC3*lc_@1y=j{xdy+#+%Ol=OEd0depy$ zoLXUET0+!cu^!9}PMb#bi7|hhmPvK)Z+p<*h4#wdR*`XbS>F271N%OY?&pQ0gnoEHCvJT@QweSb5fLlYC?8t9K{z zfiq}A>sQVnL4KRT-$;Ik#oz2ae%eX?leC`MMgBnRLp`mRqP>c5GlLFJ?}+)EgX@F* z?X0Gc=1+JlUym{WmcxFci1}xYhh*Q=L?GAi!r=JvWPd(B;zfCKF0ZM0(7O{$zlvXe zl24Dxm*Ygd6rYve!}(3q3n~8iZ7>yyk6kiTL6%qMt@#Ziw&MapF00U^0o+uO7&Z#p|Jk+VX&%BDW#F>pIz&$9LpE@MnB< zO#a-HMC)sLvJybp)9inwf15Er+mG!)zMdTCyIQB?3WT@XE7l6U$W2j z{S*Gp^F@cuFG(I=RQ?gLIr#egzf}H|YE$juJ4gKW(1Q_2SWqopVVIphKu6u8NqR=8}T%KjrauAjrc^QTSj~m>M^`! zJ>%<}-6iA6nEqYGuwHZj@NcI53H)O(O-?GxKQKSfpi1_I{gLlyk~@lcNcNu+vp@Xb zJUb}cdf;<|$&RQ$A-_!zxU<~oJwo*4PrDp^ zeZ+WaL=S(;H2Pa^Fw1RhU%1QdruoL{xj)(*u#WIYj8E;O{`g@UdpgXKEOI82M*J1_ zf?l*wqI?JWv~A#V@XFV>G#BzSceFpqpB|X5PXT?zh50;Om6{&N)~8Z^GA0nl$9Oqe zsPp|M`h$Hl$zA+3A0TgGFf%h6&#e3q|GgOR+WgS^g@8^3!oSV_evsyqaXw)EG5I_2 zruhK8$=`uD%m?TvH)!^M`1^FSFVAMn2~^kc((&K)Bk=aWmsm-AKT zc=iJF$6YLy{2Tr>j=fPu{*+2B{H1=oh$mk}jf!Et7)F)HgXv_y)Mrr@@tpLXi0z0m zo+l<_iMa<$^@1Pm*nJ=IPIx_maefH!I_AqR z2aXC*Y*;T&V*Fjg9{&3G$FW~fynT(I?e7LPB4+lF#IQs`-qcbFcKv z^O>ms#qt?`K5;h9e%<^9@$H)V%b&vk5Z^xQ{AF~$;z)kki}X|EBbaY42l*pEUx9y` z^Oc3b8`o!AudMlsVf{tC$ijM!7F{y^hkOJ=UYxerrfGuh*Eb zgpaP*W9BP-ep~Vt+Kf{X{>b+e%Koq?&L1-;0~>K+zMuLO^A+-kxcSPAG4d7njO*ts zSv+5H$In+FPnMhFdo*7O8uE?N`3mBxJ1d&63`2jPJRixT^_9m<^k>OO5HC&nhy&xz zaD(4ZkdI`E1)t*csQl>m%}2<8{=dve=9%Kx=>0ck7btT7y+6q81mXDy&hN{+ULG@E z9Wli#!}&V==T_oXbpMU@&Ai`Uh|d?s+HddT;;8kB$7ecHqtAQu}{rEwA-e8~1_v73@{*3nHvGX2`hy7FR z{0i&;=>7QAY1fZ$Q@FoRjUV3-Kc-CO`!BU0A06LJ=Rdi@sZ%WREq4B6-QQyV-oSo) ziZQ+w1}EqYA7F|7A2@_!|eLiQ{e@~&i9K6=KK};ZTa4ti*Lw}4Brn( zV=L*fi5EjYOG|>>Pv>{C2*pGGhV=)Z7xVL8ah_o~4?hB;8#oU?it}(#_9VWKW0j^f z%l)TEh?Xrk$n7o zk%ScMtDPK#w;xWY{^{cu*`F3W|IfQIdClhsY3z&ON9fZz2Z^+3db2dtJ5ezPUL^`}!E`x7R*1^8UZoMZU&SDXSCysxPOvuP=q2{?RMH z`)%ZHJ1eVQJ>#gpx;9~TEp{Ua=|7oGaXp{DII3y#ON+dYYV5%|z+WbS5>y*!Pr1w^ zbP*5KvATx;SqAUpoGHJ>_Li|d+EN5^t)78CR`eWaqQ+_fuxTEVct?>Q+-c(y(Vj8@ zO`FI2Q{%mwU`M?;wK$PpMX4UUR=#Pn;04rZq2M&eWmXJG(Wo)1spkiXCqBmISHTGN zMK95g^ES@P8=A2geou-OPw9@n#JT7)+l z^i+&C8ncd84X8nnqwvHH>0q_4ue={~&&@rkmo{g;n#TsLDn@<;tf*d3(Q_P$zvU~1 zHp;tbY_+0TSBw}>*V3TsH~tC~YrjXg%Iq?X8_%{;&jejOys`rIGg@lH$fLB021acs z#^X#+t#MxFn_pX!cDa`^JnV!Q8V&{V0{_&Z;Q-^ys*Z28nCefUu9zfx460KdMicP| zb-LwCAMjwqME|y>zC<(QJ(ha4rB35f|9pu+{~N@KHi7u~*R7LB+-#dqN?(*(YQNb& z;Nkd9NFFrpJjvR{rDBt2{vJ*pIZ-ymp!EPXMpXBGxXOW$ohM z7B;M1X8N}+?Ux{HHumSS)T=FZWUJR+yNs*7@V&9(+x4OWIga#_?=ibR9aSDsq1Dr< zXB+B-r#e@rHZBzz9_;{Z+6=IBV!Hy$;bgCF)eGv>>kQPX zE7CQMnxpEmeTGe^QHoO6+KXI|rrg+Ql#7g|n|Qejae6bkfE(-@bgUzk0TL^u5z_If zEH+NAXYyW4o=5!{tuIC}jLfVfyFp^W-;MRq2Yk%5Mt>G-j@X8^gsKGM3IhDc-2_yD{9nii=pG;=|~;Lwn5lCux?|UAM0O6fX=rIZeBaEJp3* zWW-hDnk@8tiAywcEH>E^uFw6RguDCD-_Kl)x4Pf#{J-^R@M(;0#vPl8^|7e8XVi|(0}qPD>^;mF_0+6=OV?MGVz15k*|&BX z-baVw*lnva<_P?=5|8){i;iS1=jg)6P51%v{W!7xkM%$LWSmQ2=wH`(|2>K~`retb ze~DLU{%>xpb*;jM^97nuqMWwlz$VT!jI`@M!>B+tw_JOc3jjW{fk0<);vpYoR9zaS zypiYs#C^8UCO3QX`J5NN^ZdkC_T9O^{>=v4U(cF1Bk#x4e{?V{wW1|uA%^@MJpZQ} zQoHd_b$Pk4MW0 z7;t_t&#A8cHi!4YkA_S{OF4XW58^pp+^tRY;yGOzCKy$oz?8Bv@uU|cQaqv6Vs8X%i5y{f}h3 zFIiM;ZI6G3w48B?ewWifX`(n}16uEBEX9t}5ZS}pC$+1V6TXQmhGL~9zmT7aizQXk zeowYPW@t;dR=tz|PzU#bd;f}TpDRI1$JX|=pL;w` z`=gfjs%LBa8A2g`k0OxQld}B>vQFdEcTKGA$?ws{DElIjO+nri_teqV(j5E43$@9#M_H<6j1NpCwU< zyjdczG^i~6A6*S~0)GcyM?6vgrXt&WOhAj10Y~c#@n3AA!z288`Kh7fc=q$dG4cO( zWIyTgqTQK46xV`##Wf2TlHWvx`X2*GHVRsc_Iu)t{}tIjS5ngW zR4nRaKH1)58ULtkul$$VlS^CM7hv)}8E5>jZvrujo4+$=9bfrTCEE9*y~i^CfNZb) zPmQnIDfma{v_LpQ!xGA^4 zb$s#((&^@UNocub$^q3?wI}ecw&PJ;&QHs5(fO+8t#Lf!Rm}@)J5~2c2QjweAUU5D zow};|rLL;o)}l4*I*Q7QL@8NWyr`wCVR3QG#>H!Dw-vRl^{#L1+PI-6zoB(~?e@ae z?cSo2l_l$nn;Ke|b#B^F)6(Eyv$myT$Fhb+O@OVbC46bi($yO`R5sPGUzfLOL*)wZ z`oirkl~uKCc9bt`S@uY6lfPz7%ko9Fk5ufa*s!RkWmD_2{QBjhZE?%emii6FdA?1& z-xV7cE$gapU780T{Pl|-tld_;YFp!mnvRVdRz0?&bya?2d2LtY@|GPKqYZqNuPa~I z)v#!3H;vc0zOuczVn=EH;&pi~r=EhSCurR%p=dy7j}E-PP^zp1>eYh!Cm*QRp37O(8uw7k;a*t)J8 z`m0!-zwWWd<&`ZP*Dr#-Yw|a?t*LF=BDmA|QL zU298opl*9%K|y0d{n9Oqecn9$Em^YU!G^-dhWtEV-jarbMfG)puAzC0x50OpYu0e{n%U!Q#BUC7foNx23~ZC*DhH@q;oxZ);vKkFlLOxAo(b zbN;}+^GjcXaq<`Ba}%k>SbMC7X}hLt(^9kXAQ$|8q%t6`xrY0BxHs-GpP&Vo%G%U> zJlFD`VHsvqGzISr002{@n)=qF9c{o7D2dcz- z-t$`CH5$b!@t!j+=y^mO)xu^NS0M#`j2|;x?t~N&S3z&wc?)DaE*XZMo05ydB*B&c z3w<&6O~|ESq6hI#Un(bhj|3{e4@05x=vQ6?{yVgfM^F2kc<_g=0slVW#LsDN2gcOA z(yqb(b4a|1zFL~#H7N&;8E@P-q`Yov0yV#yK~Mc$l5o{_k%X&FYN_l$NA^=KO{Vnx z_mY1+U5Xbqubz>7zAWKtKD{RU8IT54@b!SFt3lDzYSgLVDZU)OS=RqWfZ4ZjCHg#! zL*<0Hu(xm}_+<&VOSqD60{T!%A>r13Hc0q%OFv3q)Q|C$sK?MhJ?BJ~XAN*8{jqDn zFB{-Sd2^-St$fawaI3rn(m+;uUzc!eKR=altGvH3^kd}zBLmzhZ>F3-G_Em5=3N7R zSi*@P<7)P-gg;<`4;bi;_I*vlt^Ctdbf^$NQ^YNQFOYCGuJXku30LDP_+bORaolGm z+&bjp8_&GU z)KOh?TQh#iU!&ny`5gRfWHfe6joXRNlH0N zkSu8E=*)lchP82f2dG+rQH_ys^j6PeV zcqhYgGza^80BL+2{$zk)0Rsd`G*-k3L;*!fF}1d}cbFuDMDous0}eHJ z36brBqJryg?N-;=rCUO=OZ#}Ag}v@=*m}E^QXfjay*_unyAdhnpPFW=C9Zbv?>*mn zXTEt8AlmJHwysCReD8aH=bZO_&wI{!&zbN0X45SzZgsm{ijzzItx8eMRjQyrbsf@| zA(aOtusfzGwez^AbkCdkJK-sv^$UES#NRJ*4QO#`(DT3e?}LY4>ibQu8h-b4?!NOY zJ$)Z|)XwuB*Q{TA)Sli>wR4KUd)0AIF_n92K&i^_Q~t^&eeZs5G0KI8KX~Zn-Tj{M z&;PbGbl?>7+%C^g!)U*6`(wxU9C_^6*>7(@w*7_2jy>-V4Wo`7_}e-2Coc>=GH35l zZs0pZ=N|pe(CKqO9(v@3pA3C_#VbSS?s#SBM|YkaI`i1Mp=)1wYpBKl)=-&Dx z`uMRuD;_`QpM725={vqNbohn6L(3n3W$2N&emr#g(YJ;k`OZ1us`~mKfBe|}FKj=y zZT8-w+g^ELsJZQxp*`DwGIV;~TSI5o5vH8`+|aDdlfZnsudnqh$M(#A z?AXkc-yS+V`-Pz&{RC~zJ~H$p|Jy@3?@&EA^!*6=d)|8NSRdqhWba{$`J#(RrdF#<*eJy=MXYSZL z)CXJL_5#|1Joj&Z8k&3xzER`%x7{n?q_tFjN4#;L!12NP93^qI@u zy{bWVue!n&8XoXmv4`^d-$_)xyF~r;U)=B!+ii$Q?T1a#5bm zKkoM3PGtw)N>m*Lf2uB*DswU4Vqc}t#1hv}6aVIcuA$*m@Y6TobH@X!@5EKEzT+O> ztdphgzS9#F`Px&zofaAfekb`S*YUwaf7f^V;nJVJ-|p&r?>@Ko_X}YU{Mift&*;1P z7hirdT&x~AxIopOE&gP><+goeY} z>Q|euQ}r{mL&G<_+`oGMdf*B}!*6@luVw?gColA5?fB4pu4+|Z4*YX@p`j@^7<`8M zRn4bW{Y2o#<%JFuT@f0dQQ(l>pDz2scE}DMHcHlcsI;U7Na zl|siU%OF>UL+**`a!1x6A3T=33UYrx6E;9Omb+#-u8m(sZo{>a`{3=kCb`MSR^Zym zT|?JK?l%Ynxo1H(@~J6zaKB`ot+`XxUkCsCmw&kKY-FXa!&Q+`-+1t5gd8%LrOWJU zfXv{r%(EbKI3qMvgF0B|=j(7yGQ)O}Rk&uE@4~gQ^^DKr+Q@vc9@j?Z2wg+wnUMKr zSMk~BfqO1f{i>Jj3OoD2C)>L2R`ml5a{5A;H_qIb(KqbL-}8oN+^h@X%)SerqCLOy z1opi9xvai-J(+ucnd|F25y^FJW3;B>TCYL8PP#!a#+uZq^!oJFYOHZe zbr=0P#`%E9vy;m9)~l*hKfwC7QXMG8c`tMRzIj943*|F^69jku*OV9AH z|Mu#grykDT`HLl?;S-(-d(L?#{P=-H?axB4{+ryv!J(XicaEKY(=+@d=yB?-XLtra zhw|@z^7NbT1E&Ms$4-4e>%ghAZmg%{XU)rB{p6b--_CjNqCGQJ|In$cT*J#eS3dgQ z($MgmD_z5Hz4P3YKS_Ll=yud`dYo%`o2TT_3-8?4cj^f>obT;jwbHwN)f1}Mn)%bN zC(r&<)`7FvdJYUeUi-5F&(*U|dtAeqE5Em5!Bt;B|Lv>;y+cnd4Bc_ud)jklAIWk2 z&8(dRe>Hh$He@b>{EdNopWHsl`{DpBD8!w>k9Pp@3`ne6H)jk$KPk#U{|_f2IyR z`ha&2(d%4+J=-5B*)#A&_f4TYPJ7S1>D_tSld<#7xy5~F!=d5r$UpN%*YKH}L&K*( z=Ndlst>t~^vRuPwZ*(0vJI!_AOuHIBb0gM;fa}1iMwES64WGUpbQsu1&(2f#xrTT9 zU%h%D+N6c_khipIyN?N1&92JzxB%Q<@tc*M1rZP!_6$%n-IKO{Z_ zx_=Wk&cx>r@%hy;+K%-r5T(DipZF@pT7p+{)#m!;k$K(43ax!Uyy^e1`Cc zzEFj#fQpYu>!Ma$w53zkA*YOT-PtbXMr7f3yOk%;<8pgk%9{)EP4eU`%*kq#eePGo z8E*AIw0sQfZLzO{YwS<=I+Ynx)hYuLT}{Y2oohQ|RxHwhEB@TrYSa2N+_|=`DX!+U zwa21!+BVFIt!r*u)1IOm*Bg4noa9AgdusbEpR#Ef#5MZEuRUG`HOs-Y{QZcD6RRHMd6M;noGIt9jwJ z4Qo0o!{N@>@Vtua!{J!Rd}GMg1>xqlrf59e+1PGHm(yi!v=NowMfC03+Tr$w`=gDq za3mHEFKO@CSQBe+4cD)=mef7gAjvmu^^C!D(~SQv_#T{J$do`TGfcXs1i8EDGvMFc-91D35}Uhc`1GYx z#4neXFl}V^R)b-e^irjhk^4jG!!6ofh2OYEMM294FUUD7&^wC8} zEDm2>vc4%2q|=4wmPo9*y)77PZ*S==4`O6TBc0LUyajVA%Y%2*$DFDg)Z+EcElt7B zjh(S*Yp`{FXDrwhUDMna4c4z*x#F(y-M6f|Yx&AM!?)a1f6JY#-c?c-MH*Cn$Hl*RJ;} zYMI*J?^1(>%4_H>wHtIzlFn4!pr23D8EP%)NRsxdWuP}FX^#qlPSxjD{YnkG++4p) zb%RdjlMIya!Qqr=LPEJ$;7BU$ad~H!=>8`kkjXovL=Aij#qbxLp;QI_R^zV&f9~6$ z;|;4_JJ87qRNdW;e_Z*h*U&o*z0c5F4c%|(rwv^Yu*(xZ2jzp~Y?|_kuEWV4)o`%0bpUe%9*!|l0TUe$}qgZ$ZU zKcNa{DLs?e?I%>Xp*eq0n%^TfT(a2b%Jk_EdY7cVdCs2hl6= z1wb$K>3~e$0>A174(8N<-pN!7eLMaomp8|!a&X=4!S!{9zYa%{IM;OFr(wF!>FT%W zbo%_je7=v*T4ErP-ZUnD->K&zS3>;@TJhlzH*}5PXj+_t&pG+(2ODi2)HR?R^3^TK zuOQ*v-5$dC#4!;Xh&mDg9>$`y6@M_?4>p=Pcsl6g`O4ij+lB|w;_ab;xVWm*-3AFbRT;%@(K=NLfueMvtPe$yh8_ZS50&3~{ zTHPn;fn0SapytN4iGhH;{|EWo>Tyj8H1bJ)>PMZlK3>&rNcF>j+7Zu$I0Y*4Mk1gN z1=QE#S(&-WPh^t(L}zAFd8T*7y|9K8pW#i%d%aHl4?DRreBPh0zPCP+pV{3_j^$J3 z(>3iS2PV3~YvVQ38G3qte=YIgn1H{Th~s{N>C$YSKe? zX%ZOy?IL-oLM_iL)W5|E^>Z+25S8VlKl@d-E;34gqJHwn0m4%KuTlT*g(8$}efK^H z{k{p3^n?E%w!FnUoBRp>*7J}%q3(n%RA?&YXJD42MEL>r&3I-W$(O0%k4NUhf6bZ{ zaQVpJH*d_sZ6-+LC;Lg>Dk~!k!=qcN$Fr#Z@$t+=DYCPQ(FXiyeB7<+45H!Bg;r)} z7V;C-Ph{D&$F2R}&_!`?W;ZPk{iyFmz*`h|dmOYXjF;H{cNpa%uisKxwmpYD3jF~( zgN7^YtH&#~Pwi!z_Q%J)n)VP)<>M-gn(J6RWYZ|G$5&B@rVHa959KGQJ@`-6lO-@9 zCDIpkkH?#&4+qpTtE7b1Waav;M}dYsv&kbn8Xymi$LvsAe`R)(v`9OGx<9I{tl642 z_3wSKAfcWE$MmP$8m}|}YZ;AK(qGrV+SYfOl~F?aVz$j9`)8p)psAf~d+9ZY?c?k8 z*)bG_cGpZ+1y)7|%6Dg;f|-E-rsd6}tOWVPp}A_8RhU-@I>QYb`8{0)JV5mPP!;$2 zFwa9e*ym8G%IM6t^&Om|{kzKYB;eoO>@RN1r{z(k59BSgJlcE-H& z_#bGG_7|dI-zv-P(e`)QbeWZ*^?`gkpZo#ERmbY{+Wu8jGl=d^_TLN-z9^#vJlad) z;ZNwVsuD*&^eyVKn@cR<_A@iJz${NZPy45?5Aqeo^YSP^(chhA&&M&{8bq+?eG;b#x%yG+w!LR7wx&BkJgXu1G>x_uNRovq~Gy?x5~=POX`pO>2bS1&|i9dRdsp2 z)PIWlORpzz-}$D!j(p4qzJ$8Y(BH_%_{{N=D2W2di~cLLa`k+kQ49gF-lWebw4Pwq zrU6I^<6q7vRnR|a-!t02T3+pcZ|nIaM;E90O}noo{Dp{85j zESkF9q~ED_Uzz0(5Uu?M`X9Ekb$|2vdDyb2E1utg&!YL&PxV8dD$Ac?uedgi@@4_X zeE2Tb!y$})s-M;e*zcuI+nu2`{N07|ykAiqlSn+Q`xEnt_J|cGw^D zbD=d^ji>s3XdnGq7N4ldU&fpM9Uq^d{Tb`6?*A(5YJwvH4}a}>=*pD;ApYoi2;l+u z=ZVR%M~-9u4|sdx=*MJ#ZE_v9a#Qm^=3Ce+h1Ybp$j=n{St8%D-oRecAKcbdt*;*M z#{<3s%x7Am0X^RoyRB(y{_eJd_}&?b{QtxVJjTbAG<;Rp)TvH?cUz@t`EF}c8eP>D z3_9_!?*!c**%$tV@h}x(KgL53#?wk)p_L!BX^ao(hw(9$Zj6{#`~>xn5A9D*uW#}M z-0OhE^#y{2pF-nF-jW29&`2j zxst|XE)5tx9*_TxZ+yH|=LabNWWZHsO`AsfW_$SVgDlJXccM|rtf*I3*XZwziPto{sW_^czfuwvV10`rLsNp8wnQmnk zCi9O|{#7Y?wEte2l2^+&X;QL%^xt9Y3VXRV^_5vur=;f>Yjg7aGu`ryOSX4P&nMI! z~Jty|D;07pOnnUp@B|z{D$_&QT|k8zbdOVqg3bH{aa;C z%EGo3iRr1?Q<;W-(BFnrds_c1Y%xFQ<;hG20|x${n+l@Kj{76`Lz84&F|E{PH2DcgIEu~ z1^m@bsgWeI(&hPGqe0o1d z@nw!ZU+eKf@odH=z+8QA41)fKx03Z zMct>!ew0u9Uyiqr(|E|x>uZ`CGeFbWc}Nntw@uO&ecm@2l}E zuk1(jc6}>VSyyfb&Cgg51M27GAAS+<=cW7w<qGke4*DQI#dytB)Il`9Y5u*8em!w5E#bAi zJ#jj%8q{l)_K&V{_JU>Ri=HlTO8%pIKDbhEQ1pD@qWRP7oIi1XC+rn>*bC<==x@EC zU({YXsrE4cYWYGWERyzzV!gg5;|XW~>-;NH@i@ji_BWUC2R+~ABi>GLhk-?nk>`IqMBbb6$Hebj{0zU+?;qxs{)5%$#q(MPs#$z|+2 z{=?gs{qrLB)fP?K_rv*Tfi>HnKWP6nx__pv?<4)wmu5fA=lVR2=Z~JQV&RWaKV)~m z9xs>i2dp=G{^j!~xgKB6KhpHm`{QiAo@CSd_cGQ~tPd2w`G~gXgJP#Ymtnk9evX|_ z>-m*V|5}>;OjF*ap5P%ZSqhG%%ovHm)N*Kd73 zh0;1fUmYJf=Bu5S&z85t*a!2I8G*XKCvm<;U71*D=TDEjwLj_oF677h<8}JmlW}i= z^kY2yt-wbD(tLK(h5iia^5E$DmuBxuogd-6>5h%a(TU=HoPTXBXNHcKwvYJ!$8Y-6 z{=NY5M=GA-_?`l4TF>!)2fi;#+3(C;jGw}C-Tyj%FZJ5rS1d-G&;s@{)W zu4$ZK=jBm%==clktG|m}6XU(;yBwa`$2xxzfu);&B3>{sILJ2 zfc~k9&z-B!&vZWGm#%6(VIbZ`{#>=lnw6mX(Y{-qDD^F}W-zV4AHw>f{kNRjgS?BZ zYw7z58xMIFSu>fo@m6*@^?#nh!{2;6&;jQAD&Vh4kgMD6;ry-Z20!(0b}^2rQGZp} zJfG8_aUP)S>j+`LuiG!<@*dJ}k@cyB`99T6>)rTxh4#OB9et%;#>b4|)I_f5mz;pXQg227Dhx{lW2*@t-HLU)26W^F8ca7GL01A*JR~f7AXd z?p+W>dF{WT$H%L5e|VGgPnDPStuo{L#&}7EK2UT+j2ZbDzY{^D1)>`e&#rQuSLyNX ztJ33(@=xjU4%*;Pcu{ZFf)F)tj|a?e3l^+S(&R4%35qA}{w)Pvn4kz<)9{}~R#Ae^ zx9xnQ$0a7D(f$OT2iy5`k)M~K1HcqMKSAf&nuh%rS_O2Q3f6o4bCT*yu0Pse7g;4N zkMW=JERUhDW_fgfEVTUoWPRGdC$M}5Kau4#^dy$g`0r$v&*ao@vLoOliJS|9Ysjq!q%|6u)HWMy;v#y&aR zzOhd(w{PkzDWUqbKDs`C%Kz3-eO}hzl+R#$89I~YH|=Gy{5pT3HPKgw`X`dT1_Qo@ zme)&kiLoE|$AaW|TCgD9{=P&3@Y?@y9zyz2JZRHo@0}G0=IcN=#qq4k6{0~ zz>iPRe3HUnjRQbKYyY0;n+jUX3x8Q;O`tEgZM?RRZ$WarB0iw;XXG>b7qWcDK1D2F z(%w|Rm-N^5c~PGk@7mtuSbmMy@g}d|z%R1=tdFTbkM%L_=d(U0Kfv;v_6ziS%laK& z<@o*%^U*@9#2>L{}}vabVbU( zG#}6x4BEf-{8GyPWblshpyxM-zw7bm@ORyxB@?N>ls*ps*Z52Lx31se-x}}mZ`5BE zFPuAw^L-QFco$key}#1q3I0*7P zEayyPz0I~Vb^H$*V2@JJSw4zqG>!XPwr$GIRHbM>Sw1H#N$c{t`hK`n`J610H&q_D zOO#S!Za*U@N$c|7To}y8>-IBpbbLVa9G}dN65Q-Bs;uhj>xk0vI=(OZ+7@-5zTl$!A%^6B7U#oV z9&8!;hh zgnqici|O~GqVTf_-#nLi@0Z{Iqjxtw`|PIk*S2jq^X_kc{r64NyQlvQYkx6zZ#uq@ zW7U4=Tfh0|hu-;n*E~1saJlH#QJp}8OufJz^aWyYzAM+8b3sQEHMDtBDH`u?r?R=3 zIq&Iwmy2%jNzt1pyK zenR(Q*QcS0+nMnm;**{yyV!WxG*RNVx*Hz=MUbWp*Y!m8slQ^D{4~#b9rSjOqD!ic{6(mgUs=Ltf`! zs!oqrmWiXxV*lbyC!T2fb6tza674>}a1bpLM(53PXWDXTIo+~OJRPdBHU4_912;lm zJT+?cp?W+c>bVE`@KZdr>w;%*0!GtuL2m^Oq@Bpm@lG?M^)}nkl&OzRKttR#{A+@) zcBSWs1YIZSdz5}+jhfK%eJGmj37IZeW-Z!)huBHtYy5Ry$@_iN~lF8FPERz z#u#+^z2YyTW<~K+6)BQt~Xm>*+f_5bD z+4}~2M>63XC==R6amK+!dKtZAns)Uy@8RbVD9TpukXvm*fZR`C!P$voP^y>zbc`?J z|IL6>KIJ*0n-O~y+E1k9dpLu3OjIsa=3ylxBE4UL?lq)8c31D|(EZ^I?5;9K?5?P; zOW25ZaJs#8o4Cl15KSk&3v}AV8}h0x-kyw}RHT;P2D&1LUTR|}?gw{z8*XZYYq*Hr zdot|1Uv)*7YU3h$q_y$k^+@*((nI%WnjUa0=&HYyf|O&@L+?rT9_}*!;PH;tA5wnc z@wzU!z0waBPhVV2{$k6YHfGZLTI&d`Yi5C$&M#aSYCIuNzbC!#()(BIJkyVTIGh(H zTrLbj?pWgB&+YUdGp4*a4B+HA|H!4X+P5#^x74QNd;r^_ZtRWeEMPIJ+&sdrG+(5~ z)W}`N(>kBe5U#rJ6eWvOfIX>wHp`wQxt3u<#Wxki>Z)2n)NFsr6!epxmR@4;jLm%&r{v@%}Y8{y7DLnGTr!+G$^ z6yQ9g;G|3=+Q>rLp6s+X$S%%uX>DafCg7cIN&F~nxbgmZG?daKtxgwi`A2@HBj4G+ zyiO~dmFZSJzO=dtFJ($Aqjm71(Ud%lcly0@jr99V%0fQjPM1Zri+RQO(e%{*laba> zmz0HGqtlad*|zk2>hxymF&tTcG}QL)_S z;FuWS0u16G^IFCJrFMq?cP#HC9PzHfF9g*i&raR--Diti)r+(K^*8r;zWJ&0DHT7O z{Etrr{L5Q%=Aywr@b?k+VyW(@<$8%yLHw=8UpM~Tw?Q|&kFXccPEj}Y9lei`oS$g& zI-&;*O>rI3AGch{kD9lA2j7N&F>>ru+zS4l?p+hAY{1M`v_6=lJ6R%{4V+j2_wq9 zH^HWB?GJSm_?hPVQXeoq>LBgX(+wa!qtx&v&DT?9S^@h#uSID*BJB6Pnnz6?xw~MX zqJmUCJi;3w$`l8V>`2Le&ufWIIQ~t(=hg5S?|^)yOSnArADKdi)u_+>Q~88+M)(l2tF$H?84IBU6Ad)E0}w7NY);by6jDbW=CdTKgmUuf&pH zWZAiCV(d5N_a;l)7t-$JKU6*>#Aq^I9z-4`@;s7VmNyfA2VtCPiz#1bm~3NNo_2j- z!R^G(vUI&$l;?a|eu{07jt~RP=Mht0A8LR{?bCPZvOM`csu{!gS-$nNFcF z)w#sUr0ZJ4phit1`)MJVfzR*4>!>`{pO#GdP#RM>8S-d;A^8^z9JjyFG3AZce;vrj z{q@v;14(_isvYp3H2YVW^0pMIf2IBJLis-)MSm_oXo|Y= zN&PR|Cmz|9e}@d0X{p3C$e&!E*dMiA2DQNBGp#icX1wyJvptfP$Av7$bjUb}vz%tt zG9vwx#}|KcxqAGt(JhwMZW;J3`0caCSZA;**b$-MOo~OVPG?QQ2iHd1g7j-M_))Y^ z2Umx~bR7=QxbHKc4%X}6Qk(I`wu9TZwp4O`_|JB!fAS11##_E9;9E3Ym5PJ zYY+C_k(TvrRMdxGFSxHQnICM5#3DhvF#dJ4XO~Z9X^f-(wx;IRn-FTJB}aZ!bVGAv zG)!XO)QI=f%koV+C#^e_TDd+JY+n;>XkXvf6hw*pK7(JtiQvbskiJ;)C1Rx|M|-jM zU_%u4x4R%{McVF*2ERCa{=CZTE2|e=S6K}w;OpyY{aW6(0k+W1IQ3RvoR?~6ME&z2 z$NWp=!^ucMk~?NKH$~IDRjCp6H8pQ&?rerH1sgU7=jlgGoC=P>&%-+d@n;(ss^SOy zf{p^|ci-`^n4@oG>21;s9*cr0xgCfqM}v@9{!ZHpY}ocpPE*gl8vq$ zcFMs=D|EvnW99r$GZBXp#OOiI<2>JQGNPhPxSUtWc?lgX}${!+TDQvWfoj4$w6 zjz6agyw0n7|D@2Lx}Wm?S>~VnzUu$W{AcZ&Cj#H(D6gHH=wp_3%{7cKc^kLHq$zRnk5vHrnr+^k^ImC(>V; z`U}R{8BTkQw*R*4(JOjdO*UVz*lBzsJjo~|m1X03^-b|Z2#n~}^PKp~#^Yd8q-UWQ z`=T@JcIY)4e;Ai#ZfGbFpoHM6h2S0feIWqTO zgs)!$g?y5e<|R(W=5T}f_Ir};9P`q7bv^j>J_$}igJ->Xyp$O}+lTox9C#y|uBzOD zck(L)zQQz2`!GsWKjE3LA8rL*E#zEi@T3>b!JO(G{7KBjXN!YR@5A5}GknsQb)vjy z4F6ieZ_2hyvVFLowT3VIy~XfJ&Qb>BbBlxTl;>qf{p9~VZyz#z_J8Id5%@YY&mR?d z=D#ZN9R~lJz%zeP;Gb%@l{hc(%)cP;{RaQOz%!q|8=^$^Bz>C#oD&6}`9XnaIm-l|`7;Eb5=0 ziUpqeB?8ZSO%!Die6-&k%Ult6bojUn%hHht&el{E)!c8GF_WJoA?cJnOYW z;F(`1@a%{60?+(=1fKP}SKyi7B=9Fp|E?8y=C=qu&%+%8&-|Fc$ISYuUz-fSV*g=& zx4^RMS*9% z_6t1oUlMp;H~R&i`7aAR%XvuPnSVs$jhsg%-tb=+c$RZO;F*6y;91U70?+(&0?%>| z3Ow`AOT3Zug2WrX*Bl72Kd_woC4dh9WWIhGCHRbIIq|j|o!Fn5Uo7fpIZFhd`4c7H z$QhJ)!!Hwfj+18yJoC#1K4`{!g}^hvTHrejexblKKP2#Mw_1T`{xX5*xM+pIGrvyY zcNsg>3q14h75G6T|7w9}ev`nnoNEQ1`7Hv^^Gt`pGe0KqEN5Kcncpq&yzXofc;;^x zc-Cu&z%&0TfoHv*7I@}ABk-(OufQ{Zm%y`Ly9J*4`vjijgck*#`TGT)<$Outncpw) z94EXi@XS9X@cqW$jtD&SUln+ci(V6W<_`!w>vcllnSV;)S+8>f&-_7wXT8n~Jo7II zJnQwoz%yUJEZi|4vLE{N%>}9To%#F|(>#B2TvROZ%r7x~#J2t4z<1fJ#W7I@}w5qR#$tpd;d?E=qn-420g{!;?adOa=h z%zsAUS+8D!XZ~J+XTA0bJo8@^c#bRf3q12*5_p!gU*MVlvcR*PhXkJaM+BbZilYM0 z{8t6Oz_R0{*94yVCj_4Da7y5re@@`p4ub;E{PP0OcDNw$%zt0t*$#Xt&+#7f^~X(@VLbCo1fJ!bX!wk0eo)|9PQ2PzC&n|sLf|{h{8lOO z%&!)B)@z}_Ge0Eo9M{zfJoA?ce4X*16#~!vdV%M4_8x&}{%V0|z3}S2H2W}rt-!Nh zEdtN{4uNOAVgk?nxWKbsc#VIWJj~xB@n*i=D)ENDUEo>H9Rkn%rv#qo)u#oX`OgS^ zhv~0gfoJ}1f#*13ufQ{ZpTM)9zbNp`-!Jg&=PwC7^A8C;+u?}7GykZ-_nP(eRe@*z zYXZ-5zAo_09}xI0#-1kxp810U&vKp@c;;Uac<#sd1)ll#%TQDMUtTx8fpq_5zR&P^ zeq+7-0?+&cfoHww=VU1He8v2rz_VUu0?+&z0?+emxxh2OLf~1>N`Ys7wZOBS3k9C} zA%W+0vsU1l&-bCTJqOxtyQ~m+=HDagXFJ?0@XTK=@N9=BfoJ|&foD6k2t4yU1fJ~> z6L{u#3p~rYMc|pgRp439?E=sI9RkmCJ|*zXe_G&K&SwOk`Mm-BztXZ}k9&vxq$>@H`J35_skx5%|4E{-XlV{MQ7Y*X7p*p7|#P zzTeb;O5mA)PT*OuL4jxfd4XrWE(kpH-xqk+i*I`2`HK0z0(-pj_~muYZ}^O7ezCxF z98x0i%(s6GIyFDDUO|CpewnDB)>aErh*e}}-c9i9?+=07d)JT9IQc;@#CJjV&U1fKc31)lA;SKyhy zPvF^ZFA6;K_X|AR?InR{e!swX8+*Pi@XUWz;Ms1k2|V*(7kFN$1_Yk@Cj_4Da7y5r ze@@`p4ub;E{PP0OcDNw$%zt0t2aP}Lhh5;I?4Qi{7utTy^C8=TA5mmH^NR(Z?NDO) zjA#BtfoD4e1)lk30?&4sA@IyE7kIWqg}^hvQs7mGt$($^Grv~g*$&GDp7|>Tp6yU4 z@XW6lc(%hm0?+(=1)lA&THu+_&mD6AvK`h6Jo8&b{T0T4Is~5iae)t-=K{I}p84GZ z&vxVI8`*!Dzg6IQ9@{SJXZ{X>_ZoSg5_sl6Bk+9g-7E0S-zD%Y=Wc;#{$7D+Irj-X z^IsJBIwSvnfoFcdz;m4TvcNO{kifHF9T9lu9~F4^t5*e{`L78)`_=0L&;0WO&-XE2 z5P0UlFYt9H9^jjO*?*YtElQu?STCR9GoJZ=fsdK`3k06|L4jvE%LJbJGX$RHEEjm@ zR|q`ISt;<$ua@`+Z2cEXyx}hsc$Ra8z%##2;91UkfoJ|b0?%^ZEAY%;E%1CU)FklC z?+|#FGbZrNj|)7@iJyhniT#23-2%^YZV`CqZxwisd$$Wb^Pd)Ymh%~bXMV51vz)sG zp82~4p5@#t@XX&Q@I0Hg}V-KF|pe*T2kK{{IFd%KIx!vNIpPJW5u({lt<9QE*}3w*jSo$KX$&gs?D z&imV|O#$Xt7<#SX512w*1)rbi=ofr$SFMX$ZPAv_a6@xj#M-FlSkacqoHfmD&Ebe; zMK-E1J)*cGvc9cx?d=+ROB_FnX>hSH1hSj0e<^A884%tE!4NUDps@P<2CN z^!m!^{KkfgifCii4NVPA*F~DDZ_rf&-yBo*ci(aAt;_X;t#?IZt0K*v(WYCi_SPkL z#jLuB73)-U8ag{K2~$hZdhJ~+R@H~?+L9G5f#mixv!i58Dq<+BSlHY}IWN25^$LwVXD*wcRp+vL;23gyOkjd~R&jpH@3P zL}^#G!nVZ99qx8r%I$Nr?AYKe4qh7B3h4_aHXa7#;T7+uo1HcXEpTg~)jXr!gJy(vo1XNNb; z*O#5G7!<9Mc(`>z>S|uNZNr+5%5b=|H9W84`fxbbG2gV>x&ZEiCs)JziOuD7SsQIc zrFS*9ThZIKXyNvT`{8QgNGu#)(%!MLCI(&V*ILVK=T%_9tcYw3^PFO)h>!H~$C!W2 zqho31edb|Cyx1%)VVX-zH-RFad_pf|P!r93YDi_asZ1)GC#i{0l4D<-dc^+RVU(oX zU}&o3`J3nd`TDuO^k=@P#RF{4-Qa@>AW;)M3g;I-T5+&(zT~Tmk$^8@~1x zyBy2QwMd%e7)}59f457po}@q7n9DQ&dBc`EWI<9 zc+!4c{xwkn%g^ja6dq0g9Y+7S0h0c7t2~!y{U1ZavTZ7||4-;&JU?a(Lo!l`Aj|(h DfPNdp literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5768d8426ce95a796c57d497ca754f7d.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5768d8426ce95a796c57d497ca754f7d.o new file mode 100644 index 0000000000000000000000000000000000000000..5981b02223e72817b7144a9b10186d1637ce5433 GIT binary patch literal 28240 zcmeHv3vgWJmFC~KTXO4RNjApO%Ti00K{k@ydRhWwT5nrVBUzTMO}1&NTb9ssT5Zw7 zBsBIci@|_3VRqQsA?um2DcaZ+vz3}zw#dfWump&c39g!5W^1cp4C6KC5!1s4lVEzk z^FQvreJwwj*_sMo|E1gap7Wh^{?|GGtJU|x)~+_qW@9SZ*k3V&Qe4KiPZ-Z0{^()p zkaV#~-wC_-zZbgfe?u2^4;L}Ec-mfg^j%!<*$X%PJ?>|4{ck)&b(NB(`^9}PM?Xtr zpZ{i1i+7p{lpg7O$G-aLY5OX& z^;9H_)&397^U_)9X*T#X=i?9dA*V<}I39NTc_gV@g>J`+1R^>Hk9-KOXp zA3c65y}T|uaptj89WNY@JrDhJ6TgUc9y=L3+Vyelu@66prLC`x9)048QxnMV(eh)l z&R-sn-8=ou*z<>e5j*$%N3jnkK7>q7^o=u*pL!l;d*hZL#BMuuJof0ruf}fs=%mQA z{0wC7h(6l$#HkM+d*W2b$z!qSPacnLc=grT#K*96?4wv`#>bGkJ-XrHhf%+WPd)nT z4`OM_KZtFBoixZjddr#EW3Q5(hG=Kcp;Jf851smX*$-lmmLHG({LIPNxs$($J%9W| z*gO+UDc>A@fBC;YmA1Y<`Z~&U5H?jJtpc71asPI3fBuDyEeo`2we)8B^ ztrLA<(;|Jt@R^%!p3mR0-+Yv?do)(7_b)Kfzc<5wL?7Q{W3{iLk8L)>Za#Wib9R#6 z5yonN0$wrK5w?B|Klrq_Ir^KyCis=-^NagldCu&gOZnTUcH0K!bL3yjw@$#f{tCW# z!?%9FeJgSW-#T&?-#TLPE%G_#Unea7rTp>Nn5YZ(e-=UE5-jgJo`j zfBv1_9__^a!|$`jj}EX!`0--$IxU)lcsO@}y>#xUZ1LSREqdB+TTEHbK2M=oNnsa@ zY$xt~9eN+kCB5{)PuWZHpR<=f+>{jk`CSG(PQH4R$eFbe7+v-kz!*nQCxr7`{O9_ z{sD8wRp%bG1GT|eGVaj-ve#$_H7A+vv1^qt@kA+ql1E#=#Pdj9I**H)1sA^TwQ(PD zLN8-;b2PQAF`8b}6eV9d55M>Xena-iN5~(Xu)7}D37h9fALIVlsDB6U6SxlHI*!Yh z$}W;`{2Kny3;fscmtVtQGT{S%2Or9WpS{1<9zFYg_{sscn9-(1D z#Sd=Nq90Y;qGxYSijohJKYaq<{P->hc*#-nCGt7)B|er@zFOo?Uf*I-U-GXndmEz{ zmo;6W{`vo{zGxS?mbfCge^q+>M+5)a#0@OZKpOmV^@Jl4Ql)*jy+W@(trj-4~_=&F|j1v1xypRZP!>0u|Hq z6`_go>CqX39@#7D`3n7EWM*pC7h)BmKsZz(s^%Tp8=CaZ&3UJ)y@9FWfZsbk6P)r& zFcg@cn+eW_g5z~Hd!_!2EQW4>U(fBjS>7?=wL=TRf{QZO)iU3eSn;kzy>;tve(-Qyh%`lbS+ zcdQIwMFIz5HsYI{+!OT8j`5(I4bFsSLJPA{Cv0YVF65gIxvvBF9r4Xh1%e~jMK_~s zcwMZ0$(eHKjrdfi@7BYXWm@7z>%*y|sk_Xel2=p7B+?;WkK^3F}p z+=oRnCc7JJm*2A`f5KQty<=ru``YE}zxh!kdp~wy6*HrwbAgbXdA-wcXSx3F?{4k( z3Itw{U7Bx>Px(4dnppH)R|~l-&Ss3Q2X4nM+nO3UKiSG=-{Sn`m+*h4(b>Xsc0NZy zS#*ei^f)&Ec76gNX!ZFc&QDdOe*atWS7w>)7YXQ3P(*t4NJ)tC^dk7KBbhp1ve}zb zUS+N1&wy{od5AT|kxC?!o++W+570d`{h>&tlsV%MMM|Cgkulm+q^_Rk^PZUYc|fOk?m?ivT&#((jO6z74!VbJf$&lyQJ-jjguMn6s}*PKYDg< zYuVQ5YQ_Nu`bs1^sGc!3?=E5Ko@XZBW zmDLqBWv-octEg>YO?$^D{jRx%xlmxrHMMsx{ob9eef`^dd%Uf? zcC_}iw6?JFURSlN+!q`fbInb8E8UxXE@Vw*rSl%XMu6;9Q) z{8_pibF36H5V@wKDaWV-{s4QM93qczP$?KoW^TsLuV4;IC$Un{MLO+Zt_0f7vJ+^H ze9R4$|y-ai(?InB8l)Io6l*z9m1oT5^<@vLk7XoxpV-mm5Wo;Ce9w z{f+B9t}Gnbx^Zb8sAIl^y@U94WTmqR3P62Nm!m*Lhdrl;IZ~YrrdY&|=Q_!sz!BdR zt~~&ds9bdaQr<7h`xm&sLI4u^7xjsXg%)BWDZZc>J0c&fe1-qNy6@j-i?g?3jU4QJ zCj03E1_DZt;402wISVqHia-x%u>E1wT~wd3)0ynpeaUftK$;2*e@^m4q|MkmT<>MD z#h@dJ#N#KP$Yie+viS8sbi8K;Ic>f52%ctH@yGbO@^w>{poSUuHrd`JICR z=@)0RbzyCYimtIIma&shvWFHljSne#1?jA~mz9Lq2wvRt8}`W)73iN%IQ|3 zdHETqi4^&7;q(@rzKhd$>GU9{2X)%RX^&2~ak`CYZoi$=?K_|j*74gp zeY;L~a2jDO%h$>2PNKQ}E>3spcsHl{F+aDzmD5{wd>f~?>2y7(>vg(;)3go{_P28y z!B|)$*&7LG#ZjcJP@ADaUL^dFU&LySMk}VKX9;DGg)mg zDGA?iBEjApnQYm-!@&nRVu4;6v^&<|G5!T`-kuI_j}H-`S3Hnq=ke#WVJ70cEQ^ME!40TO8cUW)JXD$quJZ}t!_;`2$g-<=-9 zjrc4t)5jOy9#zdm%Xou)bu-EHc;o%s+eEk#uXT+?*HQlOQU7deBzYsg2bU2p<11-( z*t3jqkB~=sw62dhJG{<#+}8;*BKF z$184scLm{myyNBVsUX}K@5;)E=Hn^1*VII`F&;S_L@R$;9lkx2?D6=1B~x1+PRb;l z`vcc+&LCWl2PjW+2I1T%INq|Ha5;VeU$&fZ?*ANbODEhk{-hIbj6d~hgv)q`enT4J zJl?tf)>OjfcmaG{D&fX>(Un3tA8%eEf9g&l+!${^(4OzP%-b@G|Pl6vAI4dClcYq>Ct4G#yXbO}=5!S`Yb#p`Wsr ze2e4AuOvJ)#KTpLUo0nF8>ha{Uu~1)SH&6)bOnt^7*9~1V)8}9p0=4{lhd$=_^;5% zhf!X?J1JIGeh)2V0hi+i$F;h;1pQUO<@h1+O%#iUehF}w7(aylUW#P{F9j~g7lAjC z@0-e7MPq}BuBNep>+|+aX{A^+Xw5;fV#uf1DV7afjMfIOZK1KiMBhbYfk}Rl#u5|l z@g&N(QLGsHDeV+1hP>87v1Z`fGKw_=PidxDHPOlN9JBoG6zhh5N)q+AA+L2%teEgl ziWL*yMX_eWyD8R8_*RNl6TXdN)r8kmtefx#iggpdonpm=-%YV%!uu%JOn5)VnhD=Q zv1;Jj0L7{aZ){AY@1R)cIOf-b52k3W(3Q>@HZ3I?tSDHZi5`Z~8-;`yutJXsFD%gU z(}h@%Q?$rbngMxtAuFWa5bTZ7eCdRLBGLRfoCN#sLbsBCWh3IhAkNN!LGeFI`ju{0 zhyxAM$NHVtAG98r-G1KAkl(s+{t)F@z~gfx>{YreLGyI_N%rsp%g;tUB(K3{>#Pm- zYARt5#_%_sf$=iCJPQq0Mhfp0g3fUfO=F5!ALQ;Px`${!{^hMDx|nFRm#~ldfb73# zBYT?(7xOc>S4XrlpXT-w-9-9){LHH&n$OR=J*TjD+QwEKs;#B?i)Ud!gYl&(SW{U; zx|I0-BD`BS%_t=EUvdo+H`dZ_=n{SJ~B^AWe-MYJ)0_(D6B}pH&*EW@27a$Sg|nycW>1Fo8#_{ zdOUFZdF21PziQz%?D?Y`8dxTbpnjF1T-5hOE3YrwV{IrG%j3vhxw8L?f|@%|<@ee~ zrY#`UAk>@kRo^4)Z5H zXY2E0ZXWI6A;inqeh{J06RoF_KlZ;aOn8+2Ai)NjMeD3!B(~z&|_Z^Kk$`iQ}06 zU9LpDASDqmD$?;+NZ*;6h|_dxwqKEvh|~0H#@*?OI4vI}&h6JMPsC~YVCa{GX}Ms~ zMc>hKdF)dQU_9)c&rM6zPs7k=$al_{-K68luOv*%3yGtBopYLK|5a!rv}Z|}rb9!& zBuvwxnWpK+Ow)8@rfIsCG|!Kg2WFg>2WFb4XERO90fX)crfL+be8j_eT@t40RN}DT z5zNqBI!^VY={P|@U+6>VJ;L+XCgSM@2sV*8zNT2mVZU>}c%|Nc9M8aV)3Dz;?^=Hf&^4S6(o8+dWCC_k7S zU!Q~BsR?(vP&1J@o>z_u$AlLYcqAKD=I{PD!l3@ihl#Y_a@Hd6u zeCsVH9Q%WzZ0z^>{(#pnX%odaw-0~V5_I%pf)F(NdrPpio$AYRPOrqj@*DY)J%`%Q zaC!S^?G zp3JmhjVJ4JMwXHUYM`t;E~slpKld#|L3(3bPnz34EZ%{h&FI7hins% z_RBfpe8ZoizcQT9+n4(*@Y-NuzSwVx?=Nb@xrL|>vKKV`HTO%Hn!QK#NBhADSw<56 z2E35tO%hL;6tD@Wa8do)o)WqQKa2c0&izNDsziAC-lFd}#uDV$26ILIPJqq(FTc>m zC3yTf;6H`bcC&pk@<$W>DSiKg`sX4Z9u)D)+k>W$i1>a6_H_FZ7&)DpkNy$)c{q*s zTEp!#n~3+EU_ReJ#*=}gzte&&m->(W3h@ei%YwOUcwf?@0JQGkeTWO>pA*cD)BKf3 z;|I0p+3kpuBuOr16ONZ%#fh zkL=HtL5%`PuW$PTji!{uc(XEWYJc87L`NRv-(R8p?F(YGQSv2W9uJ69tp;vepK8@&Q@7|p1(Q2Rfasq!vuL=9-2)kiSj!Uzu23{8L44Em{4BiR}`Z9 z(0Kw{ZOQVIEhJgq$`Bvk#rX`{WBbB|?Eggp68|a2-$K~`g34sp!1Cuq^K_55#XqWtwI^oeBa;or)?yzlMbefw=p zqU{6UiVqGT*>JHVM~@!;?DS{x&pwF%b^JST|L)=@Tbd*F0;BauBR_tyr6s5SR?xOG z%<2YiTV~^Q25{QE9k4Y(&A=a8jsNmtT5(451^%A^2bf2!pc5*m6+4rE%W=DA#c7Y) zIIN}Gh8fCaB$G{>Ri_kUwzPD}Q=Wd|fY7e=+JFQ(c!?>za+aYV{da7uY>3@W`}f zI91qhm22B96wsB8~!H5qF`t+lN*Q|E8XYyzZa&WRJx^Ayt*s= zpZRX?q5Z2Z9%SiZg;VSEWbrVC(|D!QQufvU1?K|^9-{Q=bc`p92PvG!5}sU$_VlJ? zTUIiB!i6&olnge>OK9ArB+5ckG=|V!JxeXf@cK9jH~W{+G;_E?wtX3mN+p8OM<_e9 zJwW`+{Q{y!%3E)d+XTl!Lzm+oe8`_cFbRM|b~ocriJmD@evyB|FCk`yIi~XG->yp@ z3i1_yKXA!!1BP7e(PxUE{flQ-O|c(t_=nGS+5f|>Wkv4S3x4)wX8N|t)Q!mFYxetr z?W?il#&rUh8waK@;yRCmBo_{*w&RN6dJz|WIYYl6m_-L(?)L+#C+WV_Zv%*5#t*c} zp>6zsIkroBGl59(fP_vvDcJbG79p>Cw#CAiBzx{Me1%)@>%sWyzklLWQ-Ue+`==F` zc%JzE6P^4gCh_~Hn-iI!{uTkcGRv?ZsTCsX_fJYe#4-8?!-ne!&6EBQpJ zuTIgZf+6XWK22+J-zrfZ4^Yr8i{eUGIWCDKJ)LXEF1 zsq9vLeCbR{4RnISXN%N#OO9$|t9~EwhnLU~N&OPtqIj_C7l{U$^;qPdR{6^Yqx@;l zVAX#F42WD({@18ukzD-K@DZi2`teB^CZ0#isoPnpuV|IO(o_9E59KTAZ#myKQtL$s{dv&-e9Vr{>w6(IB|!dPRnHsomEr)6NEtDg@&X2 zkrJRkZxqaT#7+N{vO2v3`UjTC{|{w_+)`4Um0!|)EFkqgmi&LH^ks&mPa$pU4sN*y zm3nlE{NIxu>f%z2S@TzZ)B*kd(DzvKceV&cbNMNMrKkKi4~(Vs>!toyQ~nf^R(*PD zCT5*kbaO)cN2I<~*KgML?}z@sTB80+-zgO|SfKv5=1*}$ZK~e)g~x=HE%nu1=@a`O zt4A)W$7e!rAc`bI-p%<@Syfy}sY`AiDio4gkF!)f3t0bD@ulucPYLX4?B7{^cWa|a z&D~8^lY2+1nt z4U^5|-OX(c6GMYTWB$RgTj&mr4h>e`=ifSoy!yMFD;vg!r)uZ@?QL$~V8dSIvtxJl zPWNzo+x;U8wfFjwf7f(h?MOTFfPB+r-{kOg&pa>F)}7-qlx=8AcheBGcxdYPfgz zVCR6ZajJc?t#e>>h}yPtrlo4n-S;;6scv(-clF%gHPtaTQnhn*Xlu`0LR|+Y_xf9Q zyC*w(2V@;uhITbkA1qb=nR^@k?ZfR2WB!)z1(QzuNOj-B&|q6=Nxg0N4WPXzcK6qg z`vz+#a80A#Cq||wtA=)U!av4ZcJAtNk4)_x8y-R*qs{#t{cWN?c>BR*!vcH+zEeB8 zd$4xzaCP69zkPCkc(!k*bK8Aw{e!jc?)y8@*M0Zz9PBie`D+?Bjrcaz){JiQdxhM{_^59raG6)=?cnduQ?Jl4QP%h5nIQ3I zJQKt8F1)3=!kD)*;I8x5dr9BBBe*xZFSXz#-(tegE4(Y?NE&7Ej2N*=Zt@!Bl)Cl)EOZ_0 z6ZtO%iKu;v9ge7nYHUNEieTfr1;Kl+TiksP;`14dGtBeMQt#u~!A(PHiL})ISjfx8 zo<+xQ%2z=`-Y8EZpH>QqNyw)pGC|GYT4`z@RWF)uiLd-$v=?K4hI%QU*;py~B=3<- z<@Yf(=u-0Cm*M{o`b){vK4~fbrpxeO2A|6FBJT&rR9w;bA4}Q!_gKL3`pPa)aYeFA z$-g1%d%_|=4tcUOE4xPNR!M$D@>@VtDwcMBCHbnI|5BD;ZH`r3{aaa1x9sBc!VG&w z+HuJaRQy5kS12QpeH=8U5bP8Gu%!P+u-W%`mN>UoTx+-t@M)3HFrl zkc!&A-E85zrM+D6Rof8jo06XgzTEn9&I2aCc}x17Yrcm}cFgf| zSn{p$@}lIE9aR>}?;R7L_O&XllH&r&)Bo$LbVTx1eO(fGR`OMS75{=s-dx`=B;Q(J zhpddXzK11W(z7*x7r1%(OdD?2>tMPu;!dK%x6U**7 z@9_AvFSx+?8wk66d#6XnI{7QcTf_K`27l+Ec+c=P??m{k49BO(@mlZUxw)^VHobRB z6lgp!$F9EVky{7;I`OMZ&j*5Yc$Jo{>iBRF^==P@M0N4bIeKq-d(S{Q-XAG`MseTub&r8R0zcr4yzVNWZ1J$a?qME$ex@kMOTsTcQuP2cJ-2G@VW=K)M6a>tLNBln1hr{y|}z89o_h~ytPjVd*ELviNw2A_Fk!Pt$#%7AGRo1%O3!~RR7fnM84-`_SApWq)K0{r5{7a X*1Sn2i@r!hIxHK8ib20Ov+Dm}l<#kd literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5c30d6e150a9ec66ae0a094c7fd60913.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5c30d6e150a9ec66ae0a094c7fd60913.o new file mode 100644 index 0000000000000000000000000000000000000000..50882b4be6e03ccee3bf21756d269da7f8525544 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAPh6{Z0m)P3`hth9-CRwk{Tq?qfbjR zEkjO($U4SgFk@z8pN%tQ;>pl5WSpJMIGWAb$QvKxIGDkjY}O}}b3$Z>kpO#e$Lm01 z-246Y=)U)=L4ZB6VJPUnxBmaHs{d8>|5e?({pHeCD|Ag`s%Y#L7ON>JW0}3Ndk??$ zuvAdGSx@zVt?uX9ZrlH%7ur5EgR#Ryw(P!l@H%J9p8GGjp2zE7agU}G6)jz_?fOCJ zx8vB=4>#zc4=Qb;Yc_WH|JhjI(GGSPv`2a}Si$$QJ^ddRv$uYwv%@n|d#cacCie~5 zCQ@HVdeT|J|J6PH!)T{#wzujq_Q^Ly^)YDH7rp{j)n$z-~@P&ED!~5R*Y4|DUo8jgYm%@Fk-UD4hD0J}P;Pw*-2KUWB z9YcfJ?CF#kMgmWSrFJ~7w_U6h|cj=oQXLnls!E1ajpq4}4>Ppu)ImxV$n z4h+sc`NUudJYSgq?eM<&Z-OuA0kZeul|1JQp>pW*!b6~e-cl039S*(wW;kW*$?%-> z7s1!X@VJ?eghmLz0=(`x8UD!==h5y@&~E==Ibiz|PliVx1We1m@{`1`R*zhKDUiG; z`}@g>J=MvtW(UcR9vdrgz($(hQ(X_AnxuKIp0!QtBit696-4b1oUs2S_?Pg3EgCC$ z9X_CGgqzeiq&ps^c1Ia2conc5-bVEGL-@%DmBpbCHx$9|JXf#ndhM9ef8+V%2R3c4 z@=5YP<$DA0y;tCy_k8cp_Pw4P_}vgO@74zflEOvGHhx*mm;nzdi?CRO#Nr#6Tw8Llr zR=+m9pY_u`V%oqXD}j$Z%lae#%=$0AqF;TdnDw8#k6rzj#jO8)8oPQSiCs+rZ6W5` z@ZN;};eTTN!w+bOFaA&c>cw~1;o-kY2n~OW9lqFrHFW~(ANkQ_z!F2l53s}k`T%JE zIw^E^!=6WIu7T&n*LJ;XnoDAR7xG*L|0K&Wc%?DlC7B@OCCKQ6Oe9jX;if zA%{_Zl40bP#Lx)jfiC(lOkjsEK_??GC54#Q6IU(=^IV;S@sl;yPwU3Vl|Kv->^F#6 zW5z!4M7nB9#1-v7t{kl|V}I;s>2p0H8lUKAXr6vPMoRoKuBA=C@^Piceaw~VxtcQ1 z6G~lD7;=80C`7g)n~ED6=zlrNwyee?tgX;lYPvG^xaeo@u zf1~kXuXmnJ2-#p)<$t#E?CLwvziXx*a=|_q;kRU`U&97u^9!)~*~P4X2=*g8D4%|R z9qSJlvwo%zJVG{A_P_is*b2V8ZHt8ncbJX}XOu`#3Clve=x=j@vJbytYswboRf5 zOpu73g46+4n?VY{_ zZI3PJ+|k^&wLMmE=#urv7DR6v+FSkJPPU-a7wlXhM5}A)>TL0LbkuoUTDEyRn|y(~ z25(1aoy2zf+B(_;{?0)2;ziq9eXZV>=51~HOWX77{(UhRpf0~)QQhLDOX~9T^70lh zs>8oUOY$DhU%ITWE)Q}Fx$XinI-0xzU*p{)NWjxoZSeYA zeSwC%!ptxYm&IyJ4wOqLH=q%47uZmVWA<4Ny=$hhvMm?rdIpw3#kdx$b{!&*keSGh zFZa@rduhm>mKMfw{d;N1=PY|SEn8$AE8ynSkUM`LqZa9P*kCPa-@3KK*O|xa>e?Qw zi(*!nuC5c9x;$*vn!O!7dPnip68Xzrm6P2lGR*sBH$UWSU){Ef}kF(%)qTj(u0hbU4XlfrbXcr%)Tz^)zFImyMWtp zeqk*Ulw1j&h$F)Rk^rU%IBGw1myCbT4+#_XZl8+#Rhr4|v$?#?Ztoov+~Y4-2n5*b{);7djYu zLLS|xi+kKMdq(X3W{z`gO{vVj)egI$?ZbA__kA=vg9qYd=yE!<%etK`eYr4Nx6$Z^ zy3X9DI-OM+@zf{PZRplEHNx~&IA*G|Y4$m(+@@q#D)!kJA4YK{!&Vu%tpIP+Qkfs! zyWogbc-dW`fx+;fzwFb6m)Ww`Gkc1Ik$3mlaKBW-ci?EN6|ZeTpwKvU{a9YF$?HeB zmJkAk{9^QMW2|?V7Gb{vk_7X&%Q)EA9quBDm(6GKk61TuqS~>f4xCWBnhJNOS^Tp zg?Q0>Y=WQrP(hiA*EuIU9I)Hbf<*=nrm>-gEH`Mo&)reVBKpB3_WB_D=)q5#v={h6e?-sqm(i&aKd9>@pT8<> zBYu;DlVC+|-vosh^kGJLSHL-m_)R?CNS%rRV}O(yvUvsu~mM}Vb*@h=elgEOy0B3g`o zj+68kk@A&xW0|0dE&bWV5B#ej@Zf{oUYZ!MATVnVmxtP)PQ(117hz}7p5{~DbkyfY z*dNE zUsC=SoHK{SiQ#9>iq>D_^M%`k&p)&$ec91J=jUykUY7M7qjfbB*~{&p3HwG1=nGYy z><$0Ge9q_CD|>#KslVP`88Lm=w^{T}^V^_rK3_AqzB8o$;Qtv>eb;AWy>MCd?Q+HR zT}k?K#pXAkA80T2js7!&&=v%U4D^cim*xjIpww5ESzhWdx*iM_va+n%Ci%wFSMM(5 z1832M)~}pDf&4auzmfb7i@(`<{Irw&r)WL1i~NDshk9BsMSB(BW(FOc-VyUR2iFJr z+c`}i&7bgAz8+)#t%UtV5%bR)56Qk~h(NC21;O#-$^LwN#EY`zTwYW0pm!IReigs` zB%dCWFUN^^DLyN`kMo&Gr`i8V|2AWMwjbY#d_&1sPx+3+`KxOmP89hL*)Ju+&S60Azhs{q z`zQRH=Zg-RUy?k$sQe>fbMW=~f2jN^+4eS;Gc_L8GbgqRBA@2#S+b3y4fhYoOL)HC zW%wD%ThI$UA3*p|`fgh?&yO8EKep!v9m%u*F_P^he?oi6&&T8Qh1Xr=d?H_Yy%O?? z^&~~mYiJL3Ybu{f%m^l<4u%HEKdHYI3>U@QGlJt#H{xmf8u1CJ8}W%qw~Y8C)MI$d zddAl`yGzECG5x!eVZG-5;on006ZpqInw(UWe_(!|MV0Ie`y=1aBzF|?knBGtW`Fp< zd44YB^xU2_C!o)Q;540)I)7@yil{qe&z_Dq-~S>$Xajrc3* z1-)pWMEMT#Y1_aP;FYg$X)fev?r49IKRqyAp91=b3-funGBrJrtxu);WK1B8kMVM{ zQ0Mzi^auN9lDqh6K0w}rU}k1Co>}=J{(CXr_4%Ro3jv)7gnyg;{SeJ3<9xvSWAb<4 zP4fYGlfMIRm=DlTZqV%i@b~FtU!L!P-;AJPK0sc=NB#ejM)?%{3x(q2i}(^E@-OB_ zo%DNdA?ipMs6&$H0`1=%2{HYWezFUQ z9XKjLv0=S9h4FU_d-&_$AIE-4@%A--zNhqa%kp5q&esp}H@-ffG1=3apPBLz*zZ7q zAwiM$vz}*M27g`09TX42-#ED+Gvzy!&-h7T%0DQ7Pkx383kiMQPCk?Ospd0&&b`tv z&u60k7t3e(`NX+2`wjCK#JB6_FMkUELwx(J^Ow>2iX-`DFVatuk6^yJ9ORGudWmA`9y^mJ<>0p+D1lj`i8Jexsg+^*TzA_E?W%`>i3(yk29z z5DEq^nIDgEX3~a=O`F{FS%vZ=C;^r$e#>iLPGj5!( zWbu5(9Y0@zJXvmv@6miEXvjB4=PQV(?yP9OG7SBF@_ZzV)>j@c(VrzBLA*5OBMyu= z!wr5vK|Yct7JQ1&qw=FWHyPtD`#_M{3BvOaoZpvqy*y^T zI%8k)n|Z&z5T7rMwcp;&#Zl`MkI!_ZMxPhuP&~Ph&-cj_A>~k- z%M_omewy~zTPZ#z^8GoFXULDzciVaXg#ADodmZ`f?u0~&=SdUO9Fz|g1nm^A6L`Lh z^-r7^9o@j<#WTc<^nd65w<&%hpV9;T(}(Ei4}_1#Gw`2|ct-o7NT$f|Zatov_8aE- zh4D@M4d~kx&p6(^|1j-0fH%drVV-|~!gzMs5?_6NV|I8PclDD?yTVLvmO zK9Aw+^ErIpxNBl;fBqofpY!umo^QXg5cHD``|(5gyum)1@5i}+{1@7f$Ig2&9`;YM z^DC_Xqxa)er`o&V$pr%tiNx7hiQb$^TbdlUQZ zDaQC#5S*Yhe1IkPhcrJjCrpHe#Qr-Q`|qp?^!c1&|2>WNi-vgTqzy@I|Lt^+vH!+_ z3YFW5ca^uX|4zqh2?oV}742Q9NB5)1*CrUxn}E+So=1`YXHvc?_Mdz{WJdR&eE&`U zgc6N+lP5Db>XfgZUkH7=bb~%&|8(7=Pt$&6?ELi)INvWNnDbZUw`KcoExsW?GJHQE zjjg7`CSDBrEG-FgKb_ynA`}n#8`d9uUd+#X#d(I|Jp3q#ZsI)r7|z2%*^~G_j#Zk{ zEc@#;9s7OuH4t!r#{Tn${NeL(_=7lK&!W#y`R9>3ePEtQb@02|vft+WwWxefM)L9V zMG{i1uXb_}-hMcp`lpXuWPe)h{6FvJ;5Q~pWyQYQ-1Op z)FIWqa!&*IG;mJ?_cU-%1E0DE=UGt)`{w2x>+55z-(K_V$ov0X7x@}TrL0Z(%f6i6zP=Q8=Etx6 z{&$hL?X0wB?Tlmk+M0y5HQ0?Hr2lj_#r1sp;;5#{FD>#qs<8*>0Dpx5N>FW_J>?3I z(1kou$LbpXW*NMXbEfu(qMjcjp7Qiiy`g%Kw&_?v)MV(MRR+GqzUKl67c03gZy6Cl%4N zL8nO`i$8EGZcDPs-ch!M$c)>~pm#k<=shYojT8qTmD?2`AC=oJY`==tAeY8iBB}|-2nkZ+G?na!Ao`Yu2ZMv27xH{JbXVFp7iP82#*M((mDe-2k z&{GlGXv{iV6`%$^j=~c+q=U7(zWPDTJvaBDUfP`XY91S~su=kZu)=yhMbB|0{*JE{ z+9>a$vDJ!VT`^)jT~C9m-}oy~to>fyDznQlZamvYJri{C@X89*&uFO)BahN18W^>m z7>_eOwc2@wZ+>k_+Ld0$@URnJXgCza3;a=sh69W*t2(~XVyZuhx?+;(F{n;;7)`_< z)ajNleZYea6aCwk`ZCRo_gLyxmO70`{qrRP{cjW}+63a`->^;|ajR`UDScsTiTzgl zh}Y(3ct68e)UigpnV!b#!-|jB=V!-_mAkR(#+leFV-WW;;a245H}tD`?bRfqsWoPM z)4bG^qVv)%7_hdFr)?|E=e=&!nI0zwS^1Agukrw?V?WYT^V(&uKMA}ti&(pCmbHt2 zTiCF6nd#rQv|om-+1Q`QQm?Yqk*!{T?J}SHc^>s+w7wX@Ffy}_>;{Pee=pWUAMi2L8vR+Uk)OtxpIy|9$Ngu=jn$rG z)r~XkUJMs&$XGi2qqYqHQEBreg&vDjowxH0#86YlLp|1fhk-s*m<^Z)jv!KX30mD_lI-ad!c-v7{b z(@aeqeOhScCmy|JKV-^>MzxRbIr$t+h^`>C{GeFr@3gu}KXFz*t32`O)#i>?T+rXq z`%BKh`7H1AcDDC<*RN?l8Fy?V*2kjW-cdU?4?HLqv-dJ%)KjzeEnQzzg1t85XW!Zt zcpn{xW4Eo!m?QAdN<88-EIN|4oTCdLH{l1w_v6I&Ki2=~lW{JAp?_WD{r4!|=zC|z z{v}?a`JcJ1)wOaL&KGDtiE_q{1DiO{Fw$=L45KBgx#ilkQULIg4Fo!i6A$?yqpFf9 z<;^_*C+@R-HoMtVFXX)V-4`advhU6P&2Kl_{%Y2|8F@dM{^LVwspTyx3o+#9;Q2pQ zklKwus>{oTJu1Kxp8xam3dXjS;cv_Fy594DNI_%^y2g9955a#)ZzwjL1Vz7Ehq}`4 zBY^XRc}{ixw>i8Iel%nvTF&94dl1j*;%;rC7tiU+Fu|zuB&L*&i6^}nk>WXBv3``l zsONMkS~TZHJf|yDfuq;xbGjr%(MJhx^M+JA6jS4?cFO*bq4AB{D>`aVbk_DX&q)OqH)V_*6Qw6dT&X?L^oVNo9e*;w`z($^ zhb6(ndk+(SC27@xLP5=SoT% zpNd6&%qQD>EaM-O?Unyhdva-O`z4sXPsJJk>zhH0;^yy+S;tp?RDt%rXz#I%KOoyH z|5M|ub_zZffOzfSj?si>@3x3g?T%Zf*9n*7!4T2?Nqd9-|I`Nl=nEt^|c+(GC;ICh_v}SwJn(d7nt2;JrT=V#<);0N!Wi?%mD_eGAj5hF5R$Erv z)v#!JH;vc0p`yL0d}m4h;@Z5HvTYU7am%)?@fI#`FD=`)Uec6pYO8JCxVU!vP0=mN zAB%3q=GI`-=EarJM{60$ymd3_F2B8G(YCeTqBZ%OTDO&MFRbp~w4opncg*sg8#jUP z`mz;|H*^;?L*5qX>hb#Gf;`{GB7c3W5B908sBc^Q_-eAj%Gzdc;kwmDZENxxT5Fr? zH&?e8Z}QgE_^X$<6gRb(Y}i)iEh=8UqHIn6=CabRO|30mo6GQ8yt-@i$_jsDYi&35 zSH3pC_VLD*6)l@KEP}nO^Eb7vt7+OS?6tk9Vk`R2Yus4TWwnd9*#G#Z<|U6Pe^b}G z)|Tc#-Hw8W#d(cOefb4>-sQfArAxiOJa69eB@GX6ZCskSJb!VWplfK}>TU2{?3U5n?uTH!d*Wd?beBRc)U>;+;a_;EI zC+GZ;d)Jq~1mol{$mb?fi?Q~2HPd!a*QTXr<3TR?{YYg%TyqWg^Kft6V?Id>E|vAE z_j{hzkJ$e_@xM8~mii6n3!-PDZ{!5xZTA@TG}aLV{cGxerSs^xZ|E=BWu54Lpm4aJ zQuh|RX@H+oI9yBAeTmccto{u<;rSo_N;vh)ivgc&2u8Clc%m;c;G@5aw95JNI2zyu zvCiw0v-!2`1!F%8sFk1xA97X#zFBnn6$fT$XUvc9DduO=^kkKS%acB~7OE z{P&W7JY9+xHLsqPe7-EjBH(^}N2;3>WwzD3smQGnUEa3%UY zj6>z5xUjcyCHNHyw@bK^Zvy&LzC);q5Cr0+--mxQ^jKv zPUGS$34I-k%K9D-JF`S zFBsrPxtb(g=|R!ANVrwr7bV;(Z;1>!YTP_I?kWS^IBrnF)wqg&w}e~A{howd$ECc5 z3gs>|rIhX{u2E5N53e!SB(Dmt;>L3pxQY)07PyK7!xp$&de!-->R+w*v%wz~1@Dpe zTWNu-_1G9_2qdS&9cTb@HL{!C zc~D1n&27#2naBE$j$3Qi*3~K)YW8)ou{Sw#>%f0ra%}3ye1Q)9G=vnjxjq27%Y2Nke{f=eXnwt6eJ-LP*f@%fyvCh|7*oohttncdd5meBlp$XdPE%80p+~708h4y?+ z^5^cxW^pUVqo1#+YonjXTeYfobxE_g#n;fey1lWh1;gLu?vLA`rQO?D*V$gz$Qcs? zfjz;es0vw(60T?uD1uuZi{&=ngw4ExZ5?$TeqY1r-xUqCH+nn0Yyp244k@g(3X+s? zlptBq(9xN{^rp3OhwO!K-nempF!gpe@TNAOd^a!VV~6_f);si|0D1NA6gSVwe2hL@ zqj)F7aWn_}djM&CV{T0?T{8H3sA4E;e06;J*K7Zg1)=y>{iQ>At4#>EDxfILS1avy*?yG#vi&ob23G!kz{j(nL;BBU kIXu~qTtT&0IrM%EY#p1Lob8}@{P4JxpV*)$d0E^4f0>>e4*&oF literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5d54e1f60cd25f1f4f8d7e0a394cebf2.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k5d54e1f60cd25f1f4f8d7e0a394cebf2.o new file mode 100644 index 0000000000000000000000000000000000000000..714002ce208889b4277c533c92e394c4c9b14689 GIT binary patch literal 34072 zcmeHw4Rln;m1e!yEvY3$t0jIcVT>AC7=-ze0AZMkXIeiXBm)uxfnYN$T2g}q`sve> zOv{iHA+n5ZFqoL#kh5`?Ogvs%hQ!&)jHB6{jlA(8j)PgO$!2{rIVVJBh=ehNJ6;D8 zXG3R%}$M>HIkIX$8?wflmyl?gC@Wp+n!z24Hg`b}LUijhG^Wo#C-V3M9cqBBk zegELo&pb1jI(Km>ujM=83v-T#pStw3@YBvW!_6lyhWl1t0$pJ!bl||?wiEjYpPG9- zTygqDxVi0gxNrZP;gN?ghA(Wt2%1Hqa_51;3v>4lKD8SC?E6kQuNC8;dNW*p{$hCB zB^vL~L!r4CZ}kE6b0R#u^~LZA@D=Azh4W6H2k+F+pM^&D9T4nEPV*skv{0FX#cX_u-YC_k~b7bU8Z_G|*d0;)~(XdvAufEk7B~JO6I@ z!ei&d<7PY>8hIG_<>2+Dli{BNo}Bj=p^?Y-4wmnIF`RrJwBS8i%f6EQ+kMq}=U)mW z@6P@~a$-+)@~hcFvZKew3LUVKruS6W!>1-`o~vhVlllm^S!ab&`vWKJKMDRNd|%X7t~9{`i4S zo2`73{7?Db0DSKi_~t#|yR&_-=LWuabPV4+YVketN#%b77XMTJdLtjaw)01!NXg{4 zh9A@qUHGTst8P0>d-vDHS4Uo2*8iKetUnd=!9@z@saGk37fvBmd0$FTSE*eW!%=pSzD;{g;KT|9l#|dLfBjO#y8Y z=GyR{g#O`wV*SGpYKPwa@A}nu-(iP_znc&m{x&=GZUffT39NtQ$CCj|3=KcX4*m8) z(Eit?(Af>UAEmhlo)2By`KoCyiS=E`^Dg)&S%$$YjrktQ1Q{j|-Ln1>$X71q8UfEE;CTc*lZ+(ayWn#i`u9R6=ih6g_aMi}D+!?y z$nhTJFv?FdjJ%Q<8i72}MgN5f?9fH%WaOo!5Yu|%%H?35t9ckdS!4aQZhTz%qY%M< zhnO{H>;q4vtENO;(f;Gg(fTs>$8MHB*At@giGI4~>E~ml#2@2Y+O#VlS8CkHT$!G$ zDRVrb)I~)h=NF1YWE-*x*@^ml2YNjR{kaf71IeEL?dbnoxURvg0{1<*?#J~6UjGyK zr*ZvP8Xxw0=edNC4R%%jXB*G1z61TcX6PXo?DH=CmhAK!*nn()0X9Foko6D2eq;yb z)9o@Za31(Ef6?2m)~nvD`A3bp*yRRvf}%4`qW~Y2b%L$H%Yn|fZH(E+Vab!l=45u-t^J`s4Q(I$@&1-A# z^v!E~d|v1F=C&>Ev3f(7tUo?4dehL}>i2fCd7Zvs=R6@=T|-xAi?^er&fC(m)!W(R z3)D4uJ38wmw$st<1!8nGc>})2yGM|KufrF391_7w0bk2q;cbw^dshh3 z=&SGATDK+OZS@K3SOMO7CJ3fk11${!ufK^;I)9+Ov%Rz1kLHAHZ|msvwsq#;1@7D6 z^|$&04R?i^VHz%r)s`G6mria#Bj7Hup%TaJvmAQYOkrhPF3|N1EQ5-1EmrM1L>?hC zksDv`r6KpykUK3cjN|(E(vZ(t_HJ6X$U0WQ&8H!E{ys)6(rdB7n%BN%ONXyBpVig1 zJzf{ZtXj6JPGIWtu~lpKcJSyO#ZybjF+8&yN{+t;S$WgA?el7ivD|m z+i-qiEfJJl38Z@};CmNc(^7Z$^vq(8$nKt54t~oR!RJ^zk#)th{Tl#?w)Qi02trl7 z4&z1j)$7Yk%ZuD4*#F}kMHD(DA=1ToZ6T1qGYDTS>1y=4>2#sF#oO83-sbLXZ*S?C zkPbh;aTTbkQ^?sb)wE7#Vo zUsk=gymEEjvf6daR+la-WpgXt3*B?Qfrch`M=K5l9`U*{H1SI3EBO4w!s~Xn7jXN0 z2P03&r~7nqk6UKXh~3}JagMDimD#u0VHdQ0#4h^2k49(kK%5L+PG@#mx09tW7e(tf z8r@LWncGySvnnH=`lPxI-P-PtFnuMCnd)qsJuj8plpCma>c6>)-gdf1AfTG^X=FuUYDxgmd zuv8k{Uv=5WewodB_t_)tG62MMqm%7^D$&lz-I$%qj=R~9yM+wwN#N06Zx9nnf@u6@ z-8$P$yy!hP!OwlDpv=JQoRb|2*zIV+A_E7~*wB2I8?@c$`ECk}=m(P6>j&A)V1n&; zQ{LFl>qFW0!k}&=JY$51e8Gv4Y}67aL)#c{W-v*Q(w|IY>4Av}6RH2fgGuZh(KAk; zMD)Pt2JI8+tW)$)^n98Ldh`!{*hn0I)d!saT*zmj2R~`jUf>7)5k1#mMyE#npstgA z{;IHz_)Q8sQK&BANqgt zDHb7n*d4IvP$hd{C(%-Yd>7N$SGvbXxH$-)>x6#v@#|}9YPh^_WJBJ`I{8w>0sWx= z+~DNNbXO#7Jg;|d60xBUkCK>vEc#+-yFC6IlCGx+e4 z0q{SRW=|!5#zN`e4E+F~5zL)I>n8k*x6cj29K$u*WRh<%o0Wb2FtAiG{sp3cXvURD zM2peqIZ1yJDc`bgEE6=br9YeafqxYS9(st|OB3T224?1Qd8qyAG|bOA5q1{s$%XT$ zp*}mpF3NiSC2dAH`1Qz-I#Q5%^hY zy#Gwp1wL8Z-R%M%iztoXB=9cO1wKI+@@$s)Y{~y-iPwN9{`~@9KOg?}&_g62_kZ%Q zpv}g8N%>naFOS5D;b+c_)?efEh1-MAKeQ)(+0j4e=WUx_mUV>Dx*Cb>;r7pjeWL~R zMXFBrhJRo_=kobV^ZA!)toZAiASeF?zU=Eim(4|ebbn@g_-iiNllv>iBmde=Byv7v z1ZT{k_{aV67u5d>Q~$j?r^Wnf=m6HY1RMDqUldNyhks$ckNOw+&$I*@ZyNWXgJjQX zQU4lpYK4KR2~mH=dN3n6bt=&(#{6w+Ce^vW?Lm7N+ADusP5PbcGWnYWc(<_Uv1uId zKwY#af3yQH{SE#N{mzK$SJ|5wAbY<}`lfpnsc#z3uKaVT1p4U|csZX(>tCW}d+2`# zsayF6>7RchMR<%?_Vp#E{(5(2#PnU?X3;mzZ-c)1e9hqc&XD?p|7S$?U7wBh!e!C7 z%N5giCF#o*o8NqXpuN;L`p*bLTM!^J&@0wonjhSNQeRnSd8xnXdN5SP%Ccse{$}U#(@yfAqV>!!@&{TU>S?_c?Nxl65p-~RN6g9B>SEr0=a${2FH&l`}6S;FUpd0c}>NG-kn(b zRs8ake0og294F$X_^k9k&TpDtNb$#SgUM+9run1uc&QT}UvBuS;_qe9A%4E3>NM&A z>Rogr3-AZJnNGW&#G(n~AwL@}zod<=PclD1{)YK8m-tZy_2WBWzb62od;tDK@%LqN zNNWE9$$vB!-=R;)V~Ov&A--qFiRZ|H$s|(0dLT0vuZQMqD*|?k+=lqB>ttUZ-;w{o zpYhQ#`EySat*>RtN&sO`v;UF)ZN~U)KfVL`hLW$I@*Ris*VjIrDDoY$UrL0X!+_j> z$v!vsPxv>_7acOcBzbsI`A5L!;Op~$Q~6V}?QJY)YCNoGPHYuKKF!y&WE({r?jMks z@O-_?@H3LPuorkffbgI6UAAPNA3Jz{Y|jlkl4t#|NVb#w3GE?2ACJ!$UU!l6iG1bt zO2{YHlN3R(p*_^CseC3eBbbaj7#bk|r2bMcToiB52#!PDh^Ogm#3!I`#3v%%GUAg^ zkKrxr8DHP*E*Ve8^zTZB^_u&Ke>3e*;2%%X49nb6wpUpn9swNsp)}i{eG%X#stFn z7%wLab-v$3f3R;Rxr?9X1LQ3XW@bj?nUx>nzZc_OpC4Mk5YUN0__x{L57K-x&IhbN zCVvOsG#`LB`8)82`2hXo2F?Btf1gJ7<@pZy%?KLi1LP%q)c-GPluyCGP$)jWfG;5; z|6*>`Nx$djqmFceIwW~6(EiPl5YsRDM{$EY0iT4r5%0pOhiXswbkvRbj0tA^1k_`A$jkke>jUyq-31PWer%Y}xntzxd~%8S za-q^3&t5?OxQnHdf5V@~u{SEopHiuXztnFR@#KrBQ8BC+!>ICjFpcb&dIVJw&q?2j z*p3+Ed14~1Lwmb^%k3;xmrdMcfmM0&{n2eswVd_mxU7Ua&r`1tUzUhtzGfAAyT39lzG&JO`z z$9&o4z)=B;4eP}zjK53R!(adYIQA=wx3BT@J*A&pmIwQFzJ8Fu@%8zP$)48y%#@G7 ze)|Iq35v9z^*rM;`0G0Epm+%W#>xGdDc_-d#!muM{z3VB^0Q1>Na*W!@|pZkHJ|Zw z?v;LdJ`?r7SU$tgC(fnWZJoQQZ2{h8Kttk0(P8}%%#*HL=3$9fdoZw+bY z^&0b)@X_^p%zTBn ze&c*4i{~ru`1uOt$#PSCkLD{uL%uOOUqL)|XGQasVd(FZ=ObCPzVdj9{w(%nz5k}{0!8k>_XnAsAUyxT`F&Z} z%VWl?qo#OeIA4eV+)liT?!U3VnfKfC@%h47`|Vv^9JN02_)JG?^m$PZ#gqH^e4jiK zQVylLOz{cpr)ht^h2m2p-=Fh%hWsdfm!0QN*bk(!*O9;ON=T%5o-{GdLHST&&`#kx zf#u)EmdQDcs-hk00LuVi-vgTqzy@I|Lt^+ zvH!+_3YFW5ca^uX|4zqh2?oV}742Q9NB5)1*CrUxn}E+So=1`YXHvc?_Mdz{WJdR& zeE&`Ugc6N+lP5Db>XfgZpAUVybb~%&|8(7=Pt$&6?ELjlINvWOnDbZUw`EV=T6{x( zWcYqS8e2t&O}rTLSy~e0e!8HOMJOKfH>^MSyqKT&it`M^dH7Kf-NbqLF`S2kvM2F< z9IG^?S@zdyI`;eQYarnMjQ!^g`NQYo@CR|eo<*OZ^3Nl6`oKJq>fm>kWxvh$Yf<@} zjO63zizK92U+v@|y!~)G^-mwS$o{n0`G5Y+$!k78NMm0FKVmxaY2cm)?rGqj20nES(BU~k?Et>zzwN?DWeH+?z1eSInH z%uinV;~ye#+u5?3HPesjYibhK)L=J)kp8n-6xR#ri=&z*zqH8fsKy?g1N;>NC_%Mx z_LM6;LKpEs9jj~jn`Q7m&YAKDY;PIcqb)@s*XkMQV@1z#CTgr20Gs9!iFXv~!JRfP z5$!1h(6l+cKQ-Q~33k+rQ;QSnRg~(nYh{}z3tm8t779*dTw%q46pb3Aih6#Sc;aJR zc@>OMU-S~~IB(;uyrCJ3;rFCi@s#f9OPmX?u+8{93ExvqL4w$;C4%K>iHDCW))l`! zq)cM<12P&)6DGOV*+3D2zkM zo>WB72Aw8(EdIc$xGl*hdq&w3A~S9`gWmNdq4%iVG*TRVRBl&%d{l0?kUL%4bi6g|h0 z_y@jHXrsJ~##Sqeb;XGBbUh8Ke&eq|vG#j(tIRILxbbWo^-R#k!z(LLKcl5Kj66!4 zXkgTKVm!|D)N1DyzWKE!X;*p~!^2K^q2W*vFYrel8V)eNtm^nii>dx3>WWFC$DlgZ zVKfncP^VkI^Z^exO!RMC>Ps{;-eaj(S?V+%_0N|G^j|AZv9iQpkp1O43JnM zjgXE>CXsvBq6y%;Xmkg;_3N%2OV-;3eqRb0dh6(2^&9ol2YKS{H!?z(mLr+8u5$!XdZ zWHD+dCnK&J*JPnTNnE0lW3kDWaAWTGB;4DF{%PiFyw&|y=l|_TgHL00E4T6bynPO> zz5l7}rkR>J`n1r>Pds|de#n#$jcOm=bMiTu5M4oN`9ZPL-)VJ|e&VcrR(ayltIZv) zxS+qI_m`Z1^I6{K?QHM!u3yu9GVa(!tdB*#J)?GP9(YhJX76FfsHbM_Te`lw6nkyP z&%U)Q@IE>W$8KAdF-PE^m3YKwSac+7IY$>hZo&_U@5hPlf2{w}C*xcKL;t$Q`|nY_ z(f7`b{Y$(;^FMQ2qif|ZoG;LP66K5?2R3n@VWi#g8Agj#bIY}7g#h3q8whk3Cm!-a zMpdOz%A0xqPuyqwY;v=wpU-*Wd(Tg7W#6CuyWiK^{&wb^>G?mM_LGBYspTyx^D*S- z;Q2pQklKwus>{!XJu1Kxp8xama>h27;cv_Fy594DNI_%^y2g9955a#$ZzwjL1Vz7E zhq}`4qk!{+c}{ixw>;hlKN>O-E#>ghJ&5OYaknv{?#FJi(Nb#Jm zSU<{N)N{HNEt>Npp3{}7z|m{;Ib9N>=%WO;c|$XuAJbXddt|*K`hl_l#8w7Tr%gb- z_CJyBzGP9YwLShA(sITn`dvYPq>18?4QRciu@pN>Lu3zYpVY2OPWUFO7>bpa{6c;v zE|yeD`vckjxS=iGTJ=u;L+w2lHX5bL-M~>H&m-Qg?aK*?uV;!&?ENdUeXay49b4Pe zer|7^_Qx#kRnOM;(}hC(9z`Ipr)2vNWSz#R@0wWKli#C@QTz|cDdIM?r(4xt+3_Gc zCYXnihH}<2#nkw!owEPqXndpgijLY7owYs9b5eoDO&KG{MCr*9S87i*J)#VF9!jgMkNP24De)UmNh+9+rp+V6=o{#RuC zTuDjeQ?aOz`DA;KW&C5Zz4BjbPcCh3zX+4}={VzmV-tu`-29y}>-fr#D$u?c?LC(9 z2V{HYe`0R&90n)T^nnw3mRHC z)NC(Y)9x)QSyi&WxT&FadFQ6u>Xruo+I21EJC-*rXaa0)4dF{$maf@YThUa%VSWCl z+KQFl4TalVDk^K%?kHQ{viz}{CV%zXmK6(X9xLBbUb~>WWmD_&g8CJrZE?%emipS_ zeBUPC@5cr?$}tn`iYgTs|y;-YPuR%wCunbZQ!G9 zecAf1h6PKzX}rb_74604J4)*puFr2N+gcGFw`}WbZ_(2BWo28}Nt&{aZR=ZW7p~uS zQ*;Xo#-dxksWsTNX<;Sw(OO0_Z`nk;D`+oWuyu{Mcy+f6>lv5IW4Vtuo>Xzi-vw$=F! zt?QfWH&wTnZ1mRD_^X$;lr*)MZrEDoEiPHLyli#Brm|&S8(UktHkIMEa8=i)6&3!* z*7e=cU-_DX^-naesA$=^VFB!2U9hojZB5fAVXtk)6-cZ0(66fC*G_Jyl7E9|lP_GfXwky_eEe7nenn-ux23};-%D!ndon(6Ykn{v zLha1CqaUA~^JngzU-}Z{D418kO{5lM?TKop?V6@dP0hxGT=4sm94p>(4fk_!Z`@-( zNeeEOb*T?{p3@K8|6}5Ra(pfIo6hG&&qUwI3B=p3G3aTm!v^}-)cp$Q(Q)6@pSR08 z(fv^2a6P5&Ep$@>Kc{fGma6+wr|UWWn|8wUKm3(&>X#P-KGzV8W?Aq=UuwWde-&w^ z^W||g!1H3ICAyZIpi|vjuP>mjoxQJ)Bur@e;5ZXZ$f1-SLlFy_`09ZGHj2wyJHdJ6 z;Q$pMs1oma&+B;CXcVW!d(O0==MiyK3!7nFg%tELe#~&W6H-821-)_SEs*WFWEggC zN-heM1Xunq^u^e>AeVxP9>hC+shsFN5~%z>428y{Uv(Y$AJIM@J?(Ge!5_H}{D*)O zKWDfd7*q2~yAJ#B26fbICeO2=LvV^Po^qTBvKpIfNHvpcl21QS6P^W^Y_;UDWSwAhn z?Ay2!eLlvaa#CE_TeuSZiiF!GT*)^9{VCre)I^D&WP#^P_!PiZTLOJi!tV#%xMkdB zfE!cAV-il|;xsW@j>|h5*9b3_a#H__HzHjr;nsd?C48EtAEhtq$9PKAW9XlrbE3)- z1Kdb|{5tR}2Dnk)T&Z^}pR**~D(`?akX7E-CEVK2&n4U{@2?E~82SIm05{5;Dd!K3 zYmAZk*MT3FaN@_fnjMkwhb`~{1HI9{uSvL-e|m}z72;=#xaIGQBwUTFe6dNw)wl|N z*g$U__lSgB$NiOrTgOe6^OnXn&KH-2kD4zMPW>oeNPx`-xIE(Fave6nqkr53%py6M zlGn|t8T*0(Zj`G@!j&EreT#%!<$XcIt@4)2kfX-Um*cK9z>VVuC0vcG=yyrDb=>bu zxOH60Tc}X(LQ_iVj^Y{>1^4h8V@>j^;3{q$vA|V)7_h)q92mC1)zYiZKUM!~y`KgC zs3>@kwBHH~T&?$h3tX-DOr+f%b@k0{-at3Q9-+~@(%aS6(DWGp$;oASATxg5k^jaM z{ahov*_{VokgAa{mJ^SP9H%9JsO&zjown<~08|ymT z>l!&@LLjgw_!Lzki&4Vm?Eyt_t7Ea;#+$I2H?Xy%uEXzZ82!7Vf%Zmkruh<`Zw`#cb?QzubC<{;sQ7UO(~s z9t07YYMJ`FA`{dsnSUsxT&7i((LSGGGA)S0J@W31jAII+;+3k;#0{lf9tDtB{|<3; zMCN1k*&4+=8IGel*gpVB;~R5pYUz@}*FzOUQRA!YbFw}4Z7fwPcsClMvGEm@(8QBf zhgEHYLh-BUt@Zyb$G=|tk1Pnqx9Tq)!dq=ZxRn7#VZK^vx5@US#tYHJ+n072E((#`H=r33`(=V@ zYA$brwqKjPm)3uqw{>hx?d55qZ`;r&Uz*-h>(j4qp0-bGO`4+3#h0WBq>1zW);{~3 z+50epX_Gv8`_DOP8{oDN5K0jHjPz$BiA?ddJX%<@2%`eo-4fdtS#%rIa?;l z`V{l9ivP6whDMOjYO(#okFtwg51cu1|Gj4x;{4)$_n!IxtiEyZ4&ZmB?ZRWDCw_Wz zbl=O5jXsKV(Wak|-tzL#Mi=7Tj`R8lUmD%;*!j`xao&h?^1Kz1AH96vnOk1C?~H5l z@<`jsM@LVjer0t1Gd~->@aLb7M(%%U^u(htfv+MGdEkLFx17BH%sq=A9o_KE$<8ejl48^?}?Goq@q=k^Vi>frugJPojHF!$}0Kd z==s#2k8W`M5c<40dST%U;JG%k-th^@x#!IJ`%a7&yB-^j{Pbtg>lxJVrP1O|uYhND zWc`ExbS85D2~MN*Bunnt`p53sTJ++VgGqPh4kkImTa&(@8=`jg*;s`WZDz5Cw>HAZ zrdxdDXKm9@5Us(=Dhz)BP4kE8@HfpT8Z4~hY50W2LbT~8&RLz;Q@&%2ReT?5`8v8zaYg=j}ukWaaANt1MzWvGLvOmZ3&)000V%Gm&(>4_OPv3a!?=0gZkB5)8ea$i+ z`9jqYs~fQ&|2gQ}Z9}b}w~QbEWYth>3G)09@%Fj1^BO#du5CSv^g~F`Li*#7aW`-e z@FnZe@oxd&2HXVv0`OOWJAtnU{uS^SfcF4j1N;>5zW{#}crNfafgc3k4m=I`i@=`% z-VAI5J_>vX@H*gEeM7Bx1K$d~8u&TjAn=!Q-h=Ztq?aK5Ux2p)Uk^MV_&(s}z%{^G zz_(XJZ`%;`8E<_Pcm>X{`2SG(j$9*P9_+9_CtnRq|IL3eP*<TRp3jN#gX{TC=vrh5RoAHyIu%2wV(3&1or(>ckuGE_ z(uM4Y{?5lUuFu^ltDnkpo(w#I>wD=s@nqoBxIRkPj+24U;`*QIIziJXCw#Q;b^CbR z%Xa93zQ}Eae)qp%vyVLf@rF^mu+u!~v;aCSfKCgb(*ng#&tX0#+sSCl?I>?6m1jR0 z_$aP-)3xnnU=OYv>DqcS(246?=-Q&SCC2QeU*Em;df5KQ__f_@fBWE3`#7@Nfq)4}J3=WW(s z6d^8NXMvwu>rbv{Y}}i`P6n)7PI?n8IB&)ob+T^4*>+Op-*|GpUFW}E@NW?O>*Mg( zom}tK`D+FLI>Em-4*!~y>s>m3jo_~q{8fUV9VxD2zwkrkk^YJ7tidCXFPuf?H1Mfs;``F|IOexPbR`A*Qye$$V1 zOXkG%2Z%pT`acu@^ojhRCH^?+UnKsliTsZdf1LDh6aQ5c`M*c}angT4{Mi%vpCbM^ z=|3j^oQeF;5PzKXGsHh*BLA<5KTi6uiGSuq{uhWpPWlDnpEZ&HW#W&MK1%$vC-T2e z{BhFXB>p)Q`7hNBCEv-@mAH@$|DZKrgSFsDp(Xb_7OeSbEP5Q{+G|_ZVQ#w?bKEtN zH}}?H?pqys`}V5H`ME!Am)C32LpIEfouDUyFB$wP$mhbGI5lz}^S%+xQU7!Y8+zdy zHvTH+pRc7<4gCf4(bo_8#{bxbIl?CsB5z=xI`&uJ&^YF;e@IC#dK2^6w_e#CdHV}h zLn8~4iq8MsH$H;7+oj+5#{ck~Z~WX(s)nA!-0S)0s)qiEIoTiIV7Rs*jg7zXYSqvS zuUdxA{lqf#C(Pe2&P4f9HvV5tRYRlqQGPal0rR{+Ppclf^cyz*BIbZE&8;4K`wNz# zmoZ;_rKWo5tyj__e@NL-^xP-cM&A4(gPuON4l@iG7Pk=GBg@xSy~hF;&XG4fi9W$4v!vGLcoY>K>6V?{X@ zlxZD$G2H?^tV0(PE#ss2k&c$3krfz2Kj|CCSU7ZXCh%2kd<65zkp)Qq3ha;${2##o z0a`lnSAZA5&I^eb_>;gZfR|c_&VQjMa{el;A%4!r&(B?7bpDgzc@FcTLpIv$;rl{P zhP4UC@g$7j)UKDdJRYGmtYuD2xwmA|{H4`_GxA^B@_l73Py0=>Ja(o01-=NCPyBfn zAD`>CCdV)1?c2W%-?PB?ur|Z^Ip(*)|LpL;ghOqxRzB%(pJ@Qx&Q0O zzHv8X|IX$cI(2Byn5C+!$eHBBx@q=UVG`03R}~$v$sH^N&vDzVu{wU9J+?ltswncr ztij_?%p6<~{#Hu=($$extmAYi&!5S4I`Nfdk>iJ^ab1tsOdCw{ z)<%lKUj&{=jb||Fr@rwMhdg6mi|@NhNT|i7$UT;fbd-yAiWhpU zH_92g)k^8(7p{Zvtgwt1f^TH6b;wz;f$KKXq|rx~2-*&!T_4L|EAlTue)u}izk+x+ z8a%5Y{~D6JfYLWbF2GM`AT0-Jl{l}!*;%kTl7;IE{G0__BAGZZ;MesL2hQ{O^$ig_ z&U5+o)`%5n_zd{BMaX~3@oSxbVGug3q{!Xgr zcoBYQe=;+2?h`X2&p((GdEqnJk=H(VRpi*AkB)sl@nF#x(?1q@#CG%8zt{K&zx2e- zgO_GnhyMKK+{jB`pA&iIKW0b%>xo&B*Z(ss@&@|#82ah>Hp|ePUt;5bt+5Qf^*QuY z^!t$oh#M(5FQxulH9qng>c7=PBXfZfk47E@t_7Y3jP>ovCxCYV+kmnD9k~Oz3HVip zHp2POIAfhW(u4F{fqw&h9q>K|U#=e<`3Zxc;`{{8hw8^h{7A34VQ}Q@gl`xd*+F>g z;K&yUZyg)iLU`LC;tKG#v5{KBHx7;*B7Ec6$aTQBo5n^E-;Y0W(;&(jI&RxOHnJ4> z5Y9`8zGHA?8`5iVzMk^!7#o?3^qQT6$OpW0Y-A?kU4tVF3GW&knMQc`Aku+%kBy{| zoc}R4l8EyY|6_2(rmcZn&uDQ0eI5IXYti@DU<|53Jg7!os6u>xN{LAy#)}W*#fS0Y z!+0@Oyukd3A9NE7^C14Cf9hYl>Dw0OW~ZOh=vHR2&<&KG0#j-oeoip&!`X*lB7Pan zO=2b-`&xpXE$uz59+U#2S}_x|x^T8ynJv|3vDz%mo`Pw*IBaPQdob({?LK0aPO!4? zX*vov-~Z7rhQFtst^1h6$0}I@G@3!2B|WV@y}{lhB2o_qKE# z@C(X~EuB4G!A*hgo?F{)yRE&&z}qz3Q&wKt=wIq@?(OStY2reJu*N`7i@&+29fY>d zy*IUW_E!3PI|BYaUBQk(uOVkmpf|9NCspsdajn1H-`ExMH@9_=U_bo~PLl@yQq)tc z*FQitdRha)mgaw)iUeDFT7m~qBeYVmrTrhm+fa|dKSV{ETN?N8_3sG=I$E^WF((8{ zloWKcra*i9-e91+m3O-CU{`M!^>~h&x;lG$1D(C4{{ZgW6zJ|~2{!#hG&7ow*TvkH zJW+&B-hgt#KR_Gm)lq$xr~boqVZCj6g)Y~?Vo;G|v7WC-n@5P5$mFLFry(CsL;kx> z3n#jMf!~igD5mm(c7LfKsA+$5z~9qRRvPm6HtlWcEUobG-5&@x2Lo;G{$7lA{-&;W ze`5<~um0v1jDKBCEzN#Rwf(`C-u+k-H2WL-{5+k>!pp8q_Xcz^yoxP)ieAbHbHRo&Ni;z4U=)TSrSrN!Ok|JuSVZ%;gyUDLn z(CGs0ZGoOR40`wL4g`DKdfU1>`K-i{LDDfV`-hrqPB!8efoQ`nfcaR-(;eny_7R?* zIwAe5R+5uGA^nF)x3W}ru|p%33XTGsmhSD;63&Wr7gVwF9SxBnPOEr&vYH6A6X_ne zq)(T|k_FZCaOu;aA#n-Py~omxbU|kIoIjTC6@*?PM;AbQEp*(2bQ|`4%tsUl-?ssQ zuIWI14$?W@b!RwS$ehtT!-Y({KW^44?(yWvaPL7Qi^kG34YJ(Sel8$y4t!8q%sU8lD80CjF}Z&z1)&mu3PXAAa- zyk$#EmM!vbr&CGAN>;tUt-aaX)7R76(&6pc-_z@DZrRh;+2Xx%%a)DX{M*-V-L`(q zX8+oqH?G~hX6+ibc#C(bcX1%t)avc&@RyaY2zXI6=}P-GmZ~hG(bv zo}N9!$>2KtpliM7(KYeVVfVzYuVnkrzQ+z)PDmas!R`4|TG;plFJ#0{+XRzZ6cC%v&e>xjb=xHph;8fPF zU>9ppa0;tea5Ad{j_H%ce84e(bFxYWC$dt7-@&}VG5aMjw}KIi6>MW+3%h8s>i&*c zp*LIYsq-EsrYGo#G9(Z+0ko5fz{V#g31&*JB9Lq99>8ZnpR zV!Xp%=%wDqeCXg_boelS^&bV^%h)OW&O(lLLzvlDcH2-Z7A0loAcMt8wdQ!Iz_1F( zPJw*_HwauWuv_2;fm;Rc7C2kru)y5{4+!iP_?W;UflmosDDberVS!HzTq^Kcfe#CO zQQ%5};pMzO0|KKLaO@Kpy@cb31V%67xK3c7z{dox7r0*F27!+Y+%51afx`k13Ope2 zuujj=%G(Ky=K`5~nuZz2$?SBdhMfXm%+j!1V0Vs&vjr~2m`&~L6}W!3h6@Gm&ed?K z!1Mwa@mC6bYOaQT0-v6zVX{ZEC7E4Zpkb;%VK3gy(*23_A6}?os=ovD2KXV-30vTA zVS&jW4)E7Ei2UFo?7l_Ax;+{M*6ndhVBH>cql@Ibg+BEH>-IP%ux^iw0_*nhwQKoH zMg9SSb$gr@Shq(h=9W~RZjZ3Q^&HBc2t^>cl#0{Nr(D{-$zb(#h z;k-;^Vdxrs+}GzoPf`XPdb&OrTcaf#=>i&e5p{k&{n%yrb^5z~?!VmFntx zY#@~BB>#(^ji!_Qfl!ijP{KJO7NxnAEenP{m&+awrANsosabIFr?#*k2UCFPK&v7D z!=Z%qM4+ex8b6!P?hmpQYOL=1(G;TJA7Uw#zq|fo3Ok+7YWEL{w=}~k>_R$w^j1fd z!ADp!^5>_qwS9~YQwA%d3-T`s@uo+DhCdT>JHkL*|L1erP#+W1K)aKjv#`&EJPvX) zPXB2-(Z%9|=r+(Dj<8PG^22NC0R3e=Y-f^Jd(T}OnaYR0{UI)1OM`v-54w3%Ap!E2 zfR3OCcjw3RWM2pLbF=6lVDG2NUWpE}Ah#E{pThxl4E$7&mX-gzJdJ&Af7I!q4rd|z zWNMfikYl@sDPYLB!GtM*$n;hd&Nkt06ZV>L$b>22$nwJ`OaVxyA2#7i6CN;Op9w!? z!gVHm%!KPr__zt5GT}iJ9u~O3&K~cJVjuwK?VZcA;jhMl_N>w|`R|20;P1VtIvtw7 zoBdhIkN#78&qg6m_qRmdfq8%H580Vda8Um!>rISu|Dk_?t)T?OOI*_y0{rht(2-2x zf%kvJhv)X&`Ff8z;qL`$%z99Sk96|iT7Nc!L37mZpmPXK!g8?kU~ z&yL~4ePDfhj?L+W{?A)joo;{8U(NQmLO!n_*@ybC$(|cQ4}-*y7s!70nElSAv%^U! z4-0+N*~wn8Ab#1SG)@9JDIP2BiTVtLTuxek=c|47K4Le8U?e>9Yl8qO8` zynK&4O!>ul*ne;uo+;u7w;#q!Hx(G8PdCxC++l@&m5HA1rpZPue~yVh!yQ)WGh=l4 zKh>WPV7&fhuUVLQ$?|5$__@E$aib-q@^VddubVbYV*2DMbSjSzh`hW6UOu`Xbqd%c z5#@W_;3O`O@1ybZEF90#>Cf9Z zHuQUbqJAXLtMi}J@}&Ifmi~iY6P;U=i)ZH^M0wM3eNmf(p#2^RB`3@N+J7*Gnu`uDpXN7VfLQry zdI#6aKgiwR;2#L(IGvKd1ECpq_E&^s;qMsF(0uxMM*fwJ*A!&=GnMkc0sA_s3In8% z1^T1681~}+>13m^c-Mb$Hp?Mxg#H7e+;nsop+DwJ2j}qle~g}Q4@>&>ADnA0-{UgM zPp9$N%cz2e{5+>ysjn9cGrc_0ugICL&jGZ|5*ExKg@QP8s$-a(SA~WdHZ328jUJ~4Yz{b5h59-p@WFG#2P3+KmrP_-9t&r-fYB+57U zZ;GEjrG7K)VWT~$yR|Qynpy){ndEee^8BwC|`8ZGCVT)*H^JVLw_A<iT#i$*~6KM{UzwHjBn5n?SbYN?Sb{gkAu|v>A?QeuXEYaP?m$%yZZdUcK>&r zH*q@J>-8m|+Y_UA#OVDY4|*$E00-GOG>!Ru%xB<(SU+>1wa>#L7uU~4`hkBSbX6A3 zCw2cb*Y7hykDVrSI{j!U%{45Q4}DNw#KX6=^-w?7>vH*@31xe%QvT0`W_aZB3Gz_! zSpK7-R7_Z-QOB4r4-0u81kl0#N1ZPy>&GY4*`KiAWpCizd_G6m)2ia^ZWYhae1fmH z!JkU|x7g!KqWR$Y9+pY=!;j13?dR0{QyToGHP${CmQ2{+Q}l!Wv>$kinv3>Ru-{O7 zPXhoQR|3DSI(#z*XHNb}EuP;wUaNWDF%zsmN;{v~hU1lr$3c@Kx|9NPt+;2)T; z%lVIn5)#z*O|zd>+8guvN!sUV$c^DuZck4F`KRt5M?;wjb_LrLXa`cKoBc=e53HxD zJ>6ll7qy3tW6xm~XATIA^%eJrhZK5ptbJ|j`U(Bj++OPb*L7X| zzvjGw7fMtJv|2W1C9_#^_2cW^B-w^xSW5``hdn`?jPZI-#><9?a%$gSl>{5NzS7Ei1*V!K5+2^_Q4VJ zkEd0fd0JrIKb}?S@4bI`W&d!!`~E@k)|fA2d@t&bl9^M`g`vmwEtj?F9q+ue}ql`VeD^c{!#co`^N_^et4Dkc6bU^oLMTc9zQA- z`g`vmw4rAB$0_G}@@Bpt@Gi|yhfV&W>?bB?Eqb5*;{z8zd`f#eJasD0tT*|`PKEy7 z`v>j68UC^A-S-ddSm@&ipPw50hg$q7eV_f~0~bFUl=jB{g^Dw~P5u#5=0-{&8A{)kJXkihqArTq{*pMv{2 z_FQTodcIQI4|f~;;e0>jx6UfcE8@w-{cb$trI$eRLwGmK$9y~sFHha?J{oc<`_;7n zDeotn_xnB*OtaJeU~GI!$XX>~^M3cykOz+gVZI~wzdsXlrCAmJWbB8^``PCGad(=$ zA3lKRDI4_s&xrl(VWoW>X-})zby~&lXBC{RJ&Zy7*{1zYbNf6^`&ZKbI-cj^``xF| zpQ#yn`&@Jy&%a&ae&#T3SR4D@1&m;9zZ)+#4a@yG9WO%Z_sgh1<9?)5#+g|@FWOYtm*3h_?+qT{`k>QVU8*<6Z*#F z(fv8uzxzY#XbX}#f~tt8uIbHoOoLm>wB_KKI$ghcdjBI`^V%kIJEmkctkca+m!|v z4)f;&fs1FjIp)ug123F~2m?CD1ECVfVWs?X{OKHi@bdc)E;Z>>P$>HgGd;B{u0EO<*K~U0!2UEAIg;b)5D=Ou1}iLe@h(?N&5C5Tw&6uuu$$V z(5IxpT`t+jBi$b%`xKeVPfL^gmwCK^ye0GP%cT0vaU7HMLH|blkNFGwH{!dDXQj#d z9Ss$rx%GHK_jf%?e}+Aj`=j*yg6yBv-_`yGd%N`Zr}oWt9GCQ?=Rs)y4hQ^;;{5_e z9-gOgvY+bhl}?X*mlh^3H|jgFJ-m)n5T^SNp3iU$N|^eGJpS|_ESAcr`)3Q4@*fW6 z^Y+d+;;Y&o&}UAjt3c?Jfc%A$JgQHTA`kqz+3xv<{&?O~y8l7;&d-3O~Rz&J$wD3mbOcLpAqiTNXcUgmVHz0K`Q?G=-kOZUs9 z_OxI==jH9?HQH-dzU<#xd+GKk`=q7G{TuYT*>2r_37~tuYJ224Mo<~2H z-Cq3p-7K$8hd;xA9T#vG5c?Uyt$4sgFhYo za({ggSZz<D%s4#fNs{+}z`AN^D6KNIau&(li%)57iViP@j*jSfi%?+zKO}E5dy~Jw-oKs9 z-g6zLl718Wlh;ux(c`s8Y&@j)RNI5%mC4_w@rS<>l^<&lieIYzVeh$e|Do}*PSS_k zC$=BX+s7O0Pc(m;K`*=?LqxxKrioa?H?nTZy)l#oFhJ`p1hg`%ml- z`Hn$DUoHNr0SKeTvtjm6weDMMX7(#d=vNg z)#uB#=hf|?cQ;;4VR-(1R>%ni=8G3?ERMX2&>~wnN>7C2{o!-zc2|hm!rwo3>cO zI@VA9x6hBVX2_#$H|+n?_Y@K-FMUq|&o_GL{cJ!yubxOaBghi?eCF8)>3wHcMleCc zBOF7%l^NwfpKi|zIT9oc`Br?d4BN$-kdN{`&}R$4HtF@{ z^-r)5gU;*E%P+PX_Ii@+oq*@5K|dYWzSKSmZ|U-=e*Mr7{JK6Uzjz)H{Lx0=!>9;y zdxUj*L5LO54)Q`PweJ$De}Yke$OE1EKk-LNO+ zKaAJIr~odXpZH4cdj1Nyr%tE- zu#CPsB|8i`UHzI?Atv zK63f15gB51%3qHBD~$YAc>O0v_k^y2zH`j;oKwo3f>rh{rZe`V=*BbQosIM&FZe0s{9nrbJT5Z(d zV(}=!76#V)BjPFSAbnLAzax zR}jnVm*6nyXdiDV$vSVn=Nb}8?N<>@)IPwW`GXzt%uf3A`h)HXMU$d9K@J>Q659M78X@PeJ<702HBM*M1oy#5VbAH6-5`28fw zb8b}Sxi+fu+#3yf7}x7Un>Jh`_1GoMCvx$Z*W1Wip$DWylk0_3qp&F z2y=f%`+0-cUMtbr>frnZ^NHRa=KkRg`g{_d%ESFKI(#&~1D7sQaY>1amo8OtdAW*L ztWa@9g^DXHRlIz;im$mw#Vc2;c-1NuUwyTTue(mgt5>VIrbfk8RVrSyM#a_DDz2?n z@!GX2uB%h=x^*gEzh1@HU$0_xl~{Z5`5wK$UaZZ(+!&7+()_Jh#EaFzvSk>LjP|Vv zEm>^zC-~>;;4&ZLuR%xtLN8%49(Y36TIuzB-ha_QJfX#lDZUklcz<+byk8<=$g8Nt z_#?@yEXIU@U2EvGauvo4iEdp?@uE<~tJT4(HHhaDeaRAu4*6^A)++SMg=COQLq21O zk6M1{i~29cc+Semzv2I@gQcYlfae>yd__5Mg@Kn|gYl286gcCc)k^k^y74|G&%f+C zjAv{a(YgGJ8jMe@iZJYXxQ8LW4_o=14DUa#4lb{)27Rp|f5m!?7YO*8J{bp>;48K< z`R;?2m4u7v{Sw4a$Xl{Zmp5V|T)7Fm+a!Y4N0YC&sJ#}%#v|xQ@%AD$n25LfdO>{O z%}w)Zir3xs_{J^j3%o!ZpGLLy64oyit~o#Go{$w!7efDG_`@#|?{a?2sJ)nYAG07t zAMrEh4-QXgC9v=(=u;3{25|-*^F#U~wM@4o+#7U^e?`klzm>-PuoAV2(J@~KzntFz zbH?b%UxfWIZN9?GpKqF9T@871{$geRMEOwxUxG3IE>h-C9*nQVD+?+@zZZ9!zAg?FHwF1varsEo z=9?}T?8QpHh7i#(c!-gaL8*#Qb96LKV-Sui~O26)&R&os@sYDiv2&s`#3# zRqSx6I5APhNl7YBPF8Vov5J>0QE^(Diqq3oyox@9E9sw+q2g=F0VR4lIh=u42c3Mp zkImnfFIVaFi_kxf@zfh!xl*QkgIC*CoRFa6!a@}l} zxU^Ko6%`W3_-S9k?_U_>ZRItHpIUw3f2)IzM8repG{#r=62uRQo|caOFJVsx-`_R( zrO$6$Loa5Sp3Yzcumho%JI-KJw~Vfoaw+a-Hz{v z$??~jK=B~c7_X;!Wje-7@XPtL@?`pfV3L!@yXhjH7KEHwZW{7Y{#C$o{`vV59q~3f ziN@>Lc$t(f(UCtJSk7NKSEAE+NZ-r#U1gM?oFmbZKL=ROzsNM+x>9I7%{KC<%uw^s z02cYt-UouIE)4j-9E0D5uUyOZoe3=X;r|DMX{iXvz8MC8>MWIi7O>2po`wP3H`Cxx z!}39vKO0!q$DNJ=-8akNPoJam&jFU@XW(f~=9_KsyK`0kTwsMi0|UHo4)LQt9{#+k zTz*Cd`jao$py$Cq)clASKCeM9Li?+9IG8WbpwC77s`AzO%R;`N%*SJlp9g~Z88m;H zOZoYDUb4snf9B&U#zSw&lW&aIpksbiVvMJtWBm50<97xhzwx}a*f00k!n!<+pPo=g z2K5&{p2HqFArGH#bAHe(Ld6+I`FwmY=KRw5o=^FizQ2Iyf8f7b{DQoK(1Ha#K8f{G zT_~ei9q;q=!{FEV3kpIW`aq7B|4ADEi!pza$A9ZWl9z9c@A-=j40||(#l>77@M!xP z4hO|&m{g4S)2FL=p;N_+u2OL_p7xXT=j5o^5~lG7&$FX_#Q5P2CM8L9?9XIptC-KHk;5nK zJ!6JUUmaYyXrUr6HC3j2gR^F-I6Ym(bLOac+B6k=>9hGbe4_mM^Hp3_Bw^Igk&_H= zUO)KT>Y!^T;*~Uhr_EN!b9XNJ8|R0AuMQ^Xp}$Lb?p*X=3EMMi{mRF0E#q5S{i91kMGvjq&vY;-Sq(nA0i#9bDj|_Cvhm?+5%iXt#1m-~Yn* z+p!;kw_yc8?CUzntmIGpd-J$o{4&0utj%}u-M##Y-)C0pgZpK*_mpbljs1**0xwc* zB7RyA(x=UEi2ZsizRyPf$T8|u5Gp7j2|9mPDAjr+IxzVV)wd#KD>w~%@c0dVvqCOT zrxY4*tU;T`OHRi5t$aPD$NTSojOLH1uM0a)m=6ML`&EQ7pRDcsj;Jr@dl&fk)3|z2 zlGlI2_n`5;rMEFW^6zuw{Uqzb0;PWVUNir`G`A=GANGxE_Qm(Bp)b)%cl~?+xZggH z8mPN*jPD=89yYQ^fv_LZX*}z0e4Xg_*!}#sI6ukb`RVTkK)z1TzNqa--PM;}z{r3U zB+#BDkC)&0Cho_kv%C5-K`+qgjcG=H{=5LEyFoA3=*QE*kB8$=mueXIL+|SI7O=BI zej&-51+2@X@hzJ^KTbyi?f*N2ISH5$3QY7F+V_k(ov_W02{j$qe_6uvg4t1#pSL&m z{}V8w*66T@GdSHr@eBu_e>j7<|EpoFH?|;uPE?e~^Ftn{6gOhJDHxbFyuW5`Kbxqw>hTr5*!q- zH9G0znU4Jr?cn;kb3}cFJbNN0xOBjt>Gr%}Mz+vbl<)Cq{U032&)fg3Xg{<+%1fmC zQ8>80`ftbWlm>VhIVsvdFPIwpt9Cp;{Nt=>@6Qum``#Ou!0qLAdO6_wSe(JUq`Vkr ztE*F^R9}2`Q|MWPxQV)VSI zD4)}_LTRb)1N`fUqWt`*D34>S1*TKl-~<^jG*h@9#;X|B3!IBU$t}foG-&f99C< zrS(BK#V77xnf!B|n!a10|MVoSzlBBqtYoeK>-ee^&A&AadnU;71NL;N@gOx-?*Dl~ z7cD5YgWETO7W@)+&;nn=si`9V=zja_xB_j$zN$)eior9d!S_ zyNKpT?S?&9Z7XA9PM+Wi#$c`)VgVPL%X1Z~>$2sA&!d<6GnIPTu6-9Ikt zwZ;D42j&kspi_0Y(VwCF&o}ehF)@GP`jEqrK6oCDGY-cR=M*^`!`J&hd|@fZB* z^n9Q7pqQ^=D53e!lRff{XLOzG8;+c*arJJ7%=43_UXMttG^ZV%TsMUKmYqmYx?+~KS07TpI_DX z>Gg^6)8C{+d2#s_e}O&K=N%u6KOa}VM=a0RqPE(;XW1j#{Eh3Q%;#ahe&zji)Aw}Y z5BU2Y%KWdkuaq5sr}C8VyUOyegFWwh=knZ`e;MDykmX^1-Z%EnRcIc#uc`t?vt2 z2RQ!|ynmbdtsx%HwRpq%+ZF!WzPI?Fo51?c#)eJf)dvt$-9J1&w7`cJ_|O6$THr$q zd}x9He_4R`g0ONQ{slH^M((iCR@Lb3pKE_?>nqV$zGS)d``>&ndg;Z!pMUpvL*sLP zzw%=4pI*A-bs(oz@v%U66V6 zrB|xoh<>E<_DkOx{{449cJ@EMG4Hz@KJoL#3;%e{;#J@JZT&0XxcJ-Pp^yFVFU+i4 zc;WF|-`dY7U(x8LJ1+e$`Z&OYI}(0Gf1nJ<8?WM@zuXZn(C{5y_`^VVypqexkN^I^ zGT#0(Auzo1IzCtPc=Wp8jKB2VH-2~fE8o2pGdxR*J$a0;WUKfDFowU+ELc)oQXI)| za{8UtMGCI6(H#&=N*d_2(miCk8XFu{&Db}0C8xc~H$H84%UHh!e=(CEVeQ1K(&k7! zhZG(%Ivpk+iy}wjA--Q&EHjrwi*3kv$P%^;M{lrLXMWW2CaWUFa32Ob$OopylhJ9J zI~kppnUm2Qrl8ZZaPXB!Qo6PqJ?V5d`wz86$kM<|& zAo>j!OOh3lA=YuIXIpC9MYPa3tD5Y|^s{cht9*<#H_#gJ`_bVj3**nyhczDB$BIiA zvT0MWs`fxK$*)?&R>dNY#X@^-mh~=_>C=SaUrQ}RU;nCvRh^)DEyXPH`UKChJ4=gk z$Q=jsnwjF5CC$0!O{UesYU#Hq*skJ!6SgLp=q7A+nCK>KO*GNXc#86@PBp(3{IQs< z#@n#4qjje258(v(!?3nr3I_6*xijX!iT&^!h~Mh7vVJRs8XPL!=CiSWTfbeVEAp`i z3V+jgES+Zg4xfYd!zglox%@<5BI{3_OupjJZnJzRitdMzE+;<;?-KSWO(I{lPqtZp zvJXQ}^5x}=`b}b=6ta&PHu;}A5-Kyyvshy z#~fMy-(;h67&EH=oqPsyROw?osg2b>uF&6&7-DH;7ztuWRiibTx64dfFV(J8Zk5Yr zuP$^sVlmL>z z$SfCS@iM7g`4^|WiR1NTB~nCue@mno`TmyZecwx@IQst9ks|B+ zTO!5U_qRkIeK9wj(Cgl_-a)bW{jDQK=XbxviLp_Q*%TYiR@GyrDPGzTFKy;{NwF{% zGx=KI;4xxmoLFU@d6w25VjNN9^{L4$ak&5abHIedLd=`n$ZKIVzPhX|<6HnMWY?XPa-ZbVIe7=-^r>=|L z&_(45r&yDShxBn7JTV!=DgB%deem>&ULTTS{3Kh%@-de+S=Uj@7qb=FBGH;4^s^=z zvSTua6E$0sZt>{Mso9crOE&Uzy@XCNSw1JXE4P`6m+CUXc9L#!%N}+bx=Cfn%411} zZvCe@U z*)6VJCuv8sUZP!NWlL=u*WbcZ=r%ls-B35ZO{F%U(3X1Nxm=sZ^*7X8Z_~JTk?baw zE!91y+q>>>sJE_LT-~R#8|wC+*)1H??QgOh>Za@VuGTT#2w;xOKE#5eXy6L(}cAvOr(B>eqm5a}cm5bY&sYMP8vy4#?(c_2~PO4qi zj@VrTOA_5Bu;!=X>)X{}sKRr6`ua-ar_ad9cMa&f3>MSMPRqX)84gXz??(PAm(Nb+ z(vPmqKRI2$uJ*h*pZ_SIe*F&56D=ld@wZCV^Icy0ggU%Ky)Rz^%3PnU?61m?6mW)CNaw^pW= zXEQH7GS2$3S%q6rK7CHi^#k3j(!=5jD!inkI35h((M$Aq@NrX>W&WXc;3tEDN33!xtEV zr>e=x?=moUW5Z;3879z&HJ!6brs?j5xUy&tXs&_%deLddoRj6E974k_4P^RVP%Q0J|!WAw6JG5nI3ED zTECk^*Th4I9dB_P*L&=Ys%polDDS58ZZEs{-|`>%>c3@nux}K<`o>P%zb{ynSNfwl zKYTFVwZ1)h33#s3cInF*-qX35Vc;@`_Xs3h%J7oK#XN()gq>Az342z-i`i)fFJezC zcp*I(FUc!rg9>&luV*?5<^+zQ6KHVV#S7p-#s>Flh6 zGudecPoqB`CAF`I4Jz~uc3i=3c1+>NUnEfIY1m+q%THz93U;vu1*foj1t&9n{!Y>- ziTQwI{^rD&++>`{N)>(w^8&~0hp!1M7$cQ}ZMX%_E?TU*|I-jBV;U1>jP9i2Eq2}~ zEmdd$bW#g$xY}k$nRTO$+59$(z0AeJC$O`H-$nfFMP4H^QZ$Ko*bBYPjgEd6w=D42 z!&n$U-@oE7H6CZo8^P~UD8C`h>?^x%15lJd_>K6>EOXkzpB9+LH^LVMZV(t= z#PiemMs#=`$E^a_3ryo1(Ypoi7I;8l8sCV1O5l*drv;|*jh8QQSYS6Ag7l~Hjh8R* zVS(!frtyuJFYtiC0|NU5J|*x&0-qMRPT-3I9~0P(AWG%e3tTGjae?WZqeMR?aJRsN z0uKm0EU*tR%@O}j;AD%Nh3P9Pz>MQ$R`1oYQ{Yo`HS8AHy+Fg+0(Tc{*emeq#TqUY zxU@{er2-FNIHmF`1-`gK!#;uQuhB5sBN-2UoLa46sy|_GwT3l)@dcqZ8m9U?K(B`% z>h`dpy~6^NJshZS_vb`@=s>t0zDoSMJx&X3*yE%|*X?myVBH?|k7;z>9;XG??NR>~ zjjr3{w7|MO>Q8BO-5#d}Htg|Ljjr3{w7|MOd>EBUf88Fo${c2&U@liipyoW(DE z3uDLdyNKT~`e-TYXvb@(h#>So|Jmoj@x(!Wo-Xm6A3xK3^n~gAnQwiWuIME-3m?a+ zuha3UDgCs-zeRa;V3wtC_j8P|7VyKIezTZBnA2Bj>|72#OQ|2Lr~){y75eD2l%HzJ zM!G&*g1hiT=NFJ=o8;)}LVZJyp8mID?c?LAn6n~2*z~`|DAMReU$3+=;|*GavJ>eZcTAts zuWO_%sGetV!N0`y5~O>NP1Lg<>5HXwui)>!j2uby-*idM3lmH@d?KYu_@mAs+(C!j zcjM*%p~!br?Djp+W6n?CPC?U5IJ|Bxsa0ek9(u)6S4X$pP8ifIlX4ywO;Kkenjj;; z6Dlfdoc53u^k3~{{HwR0o#|J^6G@szpPp3qF9qZ-m`SkWbfXtKS4+#O3pZLuADUOf{(7lu+jPZdVUD}EA zW9VRE4I2NBAT!CVF;c|E{o+i8ii5{z2~#q~E6Pvul|$rjP!df}2HGQ#NBUQr_?XBa zlKNk~_M>?b-v6iebIT&apVZLZx!Re&IY9MSn1HW?;i!BZ3d!+xEs@s$@?nCEzLj$* zPWd~rH%5o<|5QF5CiAftk>6)3|5GBr?!T0uLV`K}GT7(dIOP|92h2FU{9ne*BVGKHYz3A|YP>XGMP9f2n*5N#^_&NWf@B(|g7K_21PB_G%~N zU$gzUA^*qY)W2Kg*V_+n%G=*uKJ^>2sqg<7MJN~g{6RnK`ALtzH6L7-8U0f!4M6l) z{VbP9y6XMHoK43K!a?MGJW21*I-@x2{YyXV`Swz18m5^Y~^cVovkF#Y6r7b(FQAP{+ci*&oN5yT8ORuN$%64_^U3*K_=A}*LftvzV9Y(oz1|G?%mMZy>(lAUHMJd*Y4dMs9M>zws!B0%{w-9RjE@3{J{zjY_ssj+t5ZB2a@ZM$}EZr@qH^|r2l6G1b$vURHpiD)yZ6>xs(7CA%^PBQYHE?Ee0R^T9hc3wOxUw-*Uk-l;6M8V@R@iv z+texa=-gb|)Umy_F|KWEHm$4OT(-M*?fzXI?fZAv;s?LjUALjTxnp}Dd~*E_W!rCS zuG`SQYe%`{6WjL%c2w*M#FZIqubLfOyBphAmhRfIp=bA0yd9jkE3jkhzTG>PPtm?w z`P5(H)!pzHt^BF{ZUfbO=hEE=c9pMdi|Yfmd;5gl*KywRmMQdW=JtWzFW3HhztGB_ zyv_Bp>zX=N-nx7H%3GSIs(0sx15LHtr);0d5y2GwzQ`vNFQyu!oW&CZ6-amI%qp!7xE>o`F&(@Kw zz>fX#>$0X*#J}CQHnpz|HX}~Qjgw}+%f%&?ucxv6GV#mcYo&5S%{$iZr}!VQFBtXh zM17Z5AGnF8=(zN}2f3Ye6raGJmh z|FY$!mHy>r%a-|<`O82J?r+ilSr=&UY4OV`oL1`B{wwoW;y+R0+w=Fe1Up;W zw5n_hboboacH3?3Ee77E;hwVc%0~ZEe{*kNcS{pDgfK{B0D3g{w1d#rx%Z~F&fZFY zZ%4qtrz_YI=r!c53G@cm@uccqH?H-U`y0DL{^qt066~j+!D-UKUz&fl_x6u{40S0h zDdRQL7dMvMw&K3@9Lp?MF5l(T*HOe&z&W*l7U3k{p?-)KFm&ALn(h0f^;7o$?fA6w zGp^63{hKD4_?tO{_RsxO@YA~VQxbo6iU8fN*$syL{D9S z$5AhfwCkg<+lXr8D^rb6|MxH#ts6s6)z0zutu=mHsZKm_$7MpI#(%kVX%k1})c9#f zMm_EWII$sZ6zz5jk3 z{FGi#n>9G}yydv2^ahc>hEo~aF65jN>3X}I(d#SH^?v$GQ7`dFWmutI6Ldky@rjRj z>go3(JwxB&2J1b-Wd(skQV(#;}$CelTfdCEa4U2IA7l;a|u$`z{elt)E6l`E&;2s_Y0@^#$^ zlb_RJmeVTI=a}SlOLFA9Km$^`@jX?@Juaoo{NK4ux)be72k9x-cbP~x>scw%&GkJi z8pvGV5s_||^O{IE*Y_<+j;#MI(XU9pTwnU@x4QoF*j#^^bf<6{l4E?w76N=yx_m3( zl$0*>k4Whhx?UGM_^B)CWz#rCho0`^h_P02*3-|5{_~J2U61$An$q<+?M8WY z=yLRUTxv?!+$%wDP51pOq+D~_#4|g1HnFqd3keSV_<)0Q|tA7 zqP`Y?NIb+!f-UWVlHQh3FT2te3kmYKb++{~e%0I-Xm4ri^<&-A672Q&>~Csn>FHr5 zjrivsiR|3pp()$e(!-`+IEh6?Tj!pxE8srR66|T~>J(;ZYYf6BwJp7xDOgEoS8q#6 z?dF>n_w)vu_Gw(}V5@B{y;Z%v!M4Wzy)Bfg@gt*>+8kKZa-glLMM>8z+tTXa6AW~; zT!D&vP+l)|^mh{f#*N!It-67$rG4cd@ZWG7p+sxxlBS;CvgJnf61_am#)zO8ZfaW6 zcYgpj6|~6z_x<}Vj5}h=r3gATrooSkd5%sL!+{PN?K4oCJXav1kN&~YDo-we^>L9C z@y`PMXiDb8Pd*O6s-a3TU!!R{4UzgBAX_kh6?7_J3}Ye{&q<1M&E@G>Khtwml%HhF z$FRt+udj8QUcQdQxS%w%4Ue7EIL!QJ{Bu$MoAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi80x6|-unN)s{U8i|5tVE_Ls|6F4r}UsiLu0SgfX?jAi!5?mhh0 z!%{)%W<50nwz{8ZyKVo6UTFKw48{%**|PiI!RwqYd+xvBdLFNT#XXu%RJ3%xw(AF> z-;QHfKir^)KB%&VuG!e(|7T-;M?2VI&>rc@V1?hy_Vj;P!ruCo&JNE=?Ws9yo7^{K zn@D{f=}BjW|5x|)52Kx~+1{$d*dwQ&9;`fnU~tEKPYx%zQXB zvg5$uQ%^rVm^yz+Xim#_!WZTp5AS>Lr{SlZZ-$#sTnhKCd=GSmq0qsDgWFFW7~D7i zc)0TPiEwk<>2TkHH^U|PfhZx*+H_S$HodBu#u+s)YQYLCTX6lXKj=E2)9LNg;DzhC+t57{v~{1i^d9H zhYx5P;U@JB>5fOK-BHF0Uj;0Ow-J5)5PtGORY~Z>4aM*~&(&+YUOQ&=-+2D`flZsM ze3JZ6`Q8A0?-lswJ>R>teXr*RzISvC-#cpYJ@QHAe*+f(Q~r7*AH25fhoMNxvs`1MdWyLz@{(&3>7?eN*Z z)vpcjXZkL_@7w+@B`Z6i~m!J^6szIP~v@OwPa4LhnM3kyjEzBaq`= z$YGS9WEgoRF*E{spo{(s6WHNP(8z!v4LN?e{`JZh(yZR3F@0zKHT(HkY_$}G#*RTQE`~qx#b`k3zg8j%2%BSC7 z%lgA5te@!vkC083{V#tDwt_Fe3qO7jHv2c&FXj*I@a03WtHF*<(}h0BPd~0I3SCPo zzC!l9rw_9}o`dtikNJzsvJ`PKsEH)>z$t}YL93c2nAF*=&O0bk?YBS^s4;R`$liD0FGujQ`rHpt<H z+Zyn;`h<0?0Pg}51kgw7atBYb* zm9451n7TY{)tbE>JbFj*)Drp2U6qsFC^F3f(uUo#wJhWCL@Sv6DTj|A1>cCxx{dGH z&8R0i94CdA@*?0Fu2?@kIf9@btjxfzN793gmtBCnkETW863o6X>DADR{=0zNaDHJe z5tLjBqi~$h_A_({LRGwu;6?S- z>nlntirgjG|Kl7*6gng!(j|CpC6K=}2wyDeYV^A4bfLM$+u7XS=I(57Z|Rul#?1D4 zJACf^MGF?sbFZh%g2H92xU0FP(cRJA(dlb-w{~@Ox*L64o7;TubyZa>*Ve5st65u7 zwYsis-VpMO|*-NBv!+`iDk$P@DD zK3&}7mf166_cwE#V{1xf_N{i<1#KU;i@xup(HT4tCqtLhnO)ZHWa-OA(YlRBH`I0J zHr45@%7~{vscu8Jwy6=Ouf#D^olUdPN#!;ryHc^w#`rLbD;c)Rz-=XXo0iJ_=-vfK ztj5dk0u2m?KmM{$7hYz|TF2}u4o2SHW5fLt3EzREtya9Y0f9o}(Dh?^y(X_8;aW-v z6!MGFvyHLdU0Q_w3P>{0P#VkX7W(3oB*)2)?@WyF1Gp4W)H~S%I;2wt^r-=sN`w2W zFWcBJvRUtbdxTvEfS7J{vOW6}?R?x#*{STfoBgO;$iSWi9{u$OF_9#Q#xLvE*%snO z@39Gf?n4D-CSK>9>~O$tM++7iIGDzU7P8!+?Y^TE^H@Ybn8aQ`#AXE(Y`>ZE#tvQ| z%C;8x&AUbHR1<#o#gXZhi$}f zQg9Ni$nBe;@Pa7S z8_)GG>g>!*y3$X1*4L$ZVNdiAdpNn1lRdysVSr7X#K#x)-dzb1_5lFo1Mt6$fGt8E z8CfLoW!)OPOblqkKUM_vG^#S~W6qB&Abj2SYv=?0>j4{c5}uSm_6^SD!$$_d|4^De zmHZhCrGE?b1AIm>cP6cy@GsszHwbeK*J_hVzQJr({`?VOsbKsIME~H-E0Ks6qo3m> z{Y9jFW!+dNXktr$Ht_@hDhxdMAh(w$#w!fWn#1Lx_NUV@Kj%f*S+uA5ls6sqxe<0r z*6ZKXW(McYq4DMsf3VM07 z9rRe+i2j(9ShSypy1;)G_)OqE0zW&A_n(Qn zz$a^ax?R9m3w)EnyHFSS1YO9pMdGt1|63$p1D^Qz3w-@T_}7CEl6>6%$-ja&8}}vU zZ^1cpNSqjc)~sm#H9lXsJ^1`Xd(xL3{d0caw&~?r&oNq8Bayw_{+Y0Ew1B=y)ydxQ z56tIWK3{1*|00bQKd%XL@?YS~pZ}R`F6yKEGtmk*-oyae`)$%U-J?i-(|C5}pF<_kPp81k`8-JR>(5!H8nHr5N5Mc*!0Oy5wgfVLfwVt03}ezMdu9DB5uUfV_m~>s^MQ zk-UYy!1Dow|D^A>CG-5)!SiE#ZqSiD`yV6OPVy(Thx~jzK3{m) z1igm#P`9S?nZ%4>GU{Mxfc%sCOTlnaygeg04s|1*rmqp7fVvT%h;++{PeMJ0x2$J; zeY3k{JQ>r!s~Fa6?jQawv_FA=?4!v^MfnHj=UG(AzOX;?{Y-L45f91!Q)2dq|C{IM zLQc=^Npk}FEDTQ5X}M*jzcYV9>({hIb~(cHqoL{WujI_Ae+@0fejq0k;-i3mnD!eE z?mx-V{Yh3%l>R*JHyp{_UgSS~f8yr!Sws(fZZO#q^(W-F=>d0^8@)$}p8RRIgRhSm zFOBHoPnkx4%ME6^jqM9}yWKS3I6e1An*-Jn{)q9ZebgU6Ok>Z4Ig&-rM$(AC!d}pe z_DPiQAfL7kJON($`j+NGe&&w$2l>+j)AcE!kGL?OhpST41KIjis!zrQ!uS|3Cku7H z-$Z|~Zzj2mpXLMPEevL6M&p^4AL73k<6WO0TE7s`i9q0kC zgt`&$!l{R9Pxy4yjrfcSX8Z)yV|d8R{gvwj@>1Oe4upPen9sRmq*GsfEAPZx`|8i>OgCtQW(m@^~F~;-6 zMCxC~^Ms(7O{@l^Ab;4?&Cmda&PTBSI@FD&RurA~=hz>MLrmHqzQO&Sv=i|c%k?aI zu_PD#r3ds>Ix&g#ko^y7%cJ>%!2dMJoq_T3;a|PrM?3z&N4yhWPhgxM0=$m-vfF{9 z0u&q8i&GeXx3Gu5{{3<6mlSVbVL6(hM!NIOS9iFe?fe^ZvOJ8@IS=2&pLk@ov%2OU-lyX6!{3|o6AA|$j?{cpXPjJ zA@Iianbs?7zG7H^5ihc^USl~C@gDj!t>;*uP3t%6Sy->5^k|RuD7N1k(#-2M<}2Z& z>-CuV3ZLJWe1$gSRD?hB{e-eV?1}To%*nt;T$t~tKgE27{2^|>GGmN<#XaN3`AQbg zSKRUQ708q2ruZJsSAvFoV|2cPcqJSz^Jb_&h2I zcx5vA&u2+Y9me!dUz5-CP{CKJoZWM{4wWQ4YnE`}lmHJP}e3rMXP; z3G1h6f4!CBQzGA=^LU2*D1En`=TF!Vq_NkLzwS;*q4;~vABtp({O;D{nQ6aaj$atx zwBLZfP4SH5&HE42egk+@d>iKZ_a}^Jmred+jc52gDQ180n~w9OfrC;%upjm_lj-vq zzCNGB=Z(82#`fnA^8GnKKjr!M8w)`{*{~l!gwGr7llgv}`^SHw{dnxW2jgM?6g$7d z`agOQH8Vt+{UBXhz;SV-)@v$6lqnn0h=8TQ}PXuoKPcTU=n#P;7#=NS8M9H>ya zop@Ju8~g8ctd?L<>{rp=g?e;9ihON?@w^H64C8qe`F|$mn_~aT=R;<6|H=2?iLDxr%N~J6ZTKnE&4R=N5;-y|A6!TQi3^uMSfep@7CfQ@*~6d1Jc+k zI&9*_kk8VRAotVxoh(A}kiTL5!RN*NyjPrO7|z3wg6JmB!;j%S9F#qY@8ejdDb2FK zPSdg9XI}#W_h;-sZ^$1$4~IX9^YtwH{FHwlsnZAMiBt!_t1SC%zF&*V=VT-wKVKvv z#rkR|2jT69)2V;@xJCA-#m@ipZcbkF`9T`{BKQ&eS#*H&pYraHV)+R^KQQGdpFtf` z-7EJra8CpGG;mJ?_cZXSYk&^V5o!nUEeF=~i)+9C@JErazSLL!+s%LX4?nLQ_g9S{ zUHF^Gp+Yvh@2Or_t-Eh-&au8e#`^8G&yKwR&vlWnaa78hgum>|>Fw)FVP}5)%I|*{ zdE3s)YS+v-rmv|@SW}DL2txW#XH#6yr!S6bn*7otucI1!a1QWS2%rSj#@SP@@CaSR z19hyf;eVFF`#5LH@36gPY_GNqfn2L+ppO+j$C;?H8USpXMxUTaphGoLVeLo zwBx*uv+{;!EQa5kV#QOsqc3$9Twz=Ac@n;-nt}weS4#xT(Gm|IRjez1d&=c`amU9a z`4u+VQ*m38P4YxoM<0_^8~j`1q*YZXtKPKB97g5t3C&h9OZ5 zlG#5M$?T&&iY#H`OwvR-gLF6IZ1fy7b8geEoX6F^(e+giV(z)Q2ldkCtXK2cfK|oFkAM}`>nVDUBk^~9rO-xs z7mck}6zhr+QNBHtLz6i-%WMpngV6Z5Vl!HqpSS?ZkMT z>8Ul&D}3{7OVY0NGKPnp@Iu3(AYR~~Iy4+$d|B1;jTTe=Nz@gSM2|srs>5g^{-92` zeCY!oY?$cZw$ztuX1vEzueQ`_JnEk>5$J!TIMF5$AOD7R@`zh)^GWH8QcLZ(+DE)L zH^ciGzM_sb+RgMdRv%V;ygolWZmismRX5JWUKxY9mkGBbKfj@0#cQu72~Djr+neU4 zo)n#zZoz=HeLQVjX+H0Dqt5g=G04h)JbIM}NFDo;mYUZtbNxx+m085vWwWeZ{M*8Y zwaZNZwx#`2WX;C@JeGR3rH*X%`fHbQwHLlOR(!irG$6;3Uh;ir*QcY(11hw72K8)1 zo$yrW%GAcCBEzE{fK8hLHr8l*jtjBbvxju7Auwa5$6IVyKslW3)vbC#oqC;tI(0?5 zrcrZLJ+{xV=`>1F>N#>xZ8;x?2v2+tJS0PSsB^PjmU4xEwgfc*4g)~As9+kz$ z$@NU$YsvGdAEWif2!@fFb!0b44ETGo9{PZfnbzpfVvYPX#{BG}W<2gcJ8rD@9II}e zVfSLVSVP9r*(b#tb$%~~n^$oWD^z?K9d~Gt8UG~BvbyWm)t};pVJD|)SCGZ1ot%ug zYFv|r{vdIQMvlcMTf&XG-+|+GwD$gouA63R z;^@;tD?jn*E&Cx;J~XO*bkE7>U_x{Sq2&j~N`I%-P5Ozm@>%7HN3S+_wBmyPj^1B# z{>^83pSQEU&%1t2^U1hl6R|!P_4bb1v3cM@v6#J=8Ka(>wQuSA>Qd~r89)2huE6`~ zFdVyWRmL2Fe^%lVpJCCFtmPbC__zr_Aif_bw*RsIN1u#y2@L(~8t=bH@kZY}Gxjg> z3eEq`ZH=x~xNyEe^GTF5b{yEmd4`d8!)F*RR?RKfo)rRsk8B{&S)6#t2N_kDMk#OR z`9E==?X%gbWp78vimzOiPr5yiTj@R{`|3eBQThKM$vwaBuOL{}G;Up;f)jHIbb{_$p zAIx*A>%Yz6eek0p6VWmbAKim^P8WA;6TNs&SB42jl_xQ!Y)m}q#fTKo>5BEE{6#&d zOVOe^FXA~}nF<`eMxWCqA&NdqaGN(Y)A=!-rM*Yi8=@a53qWjT5OvxF#B2X!+3rgg z)mq!*pCK)0T%z9<^iP^74%vX#I~q%|qclYJu=Yvqs^x@lqKctdX~{3-XX0W>m9*cN z?T;JU(ydkR74kgd-P*o_kobD0xWwMSBHQOmkkYZWJ?-b7h|~U< zrM>Fe+J1&mh~J|KSI3H-eVd6m~5~7m)es{TiY+jBpiFb^I{1N&lB+d&z$EV%7gzwEwShG;M;A-qqv-(mf@oFRn1%Dc*Luk7uI&F?vV~+Y{x-zJ}EkN zRr5<-Rl99PYu9%al@*CnvZ}bCrK@34am%JfYiqX`wXE}QXzbdwu_nKvbwllr!Zq#Q zqLNi5>x-KjT9Nl*<+q|)IrFTQ& zj+V-*+O<2&m$xi`w6@7#v$kbLLG7azJ1aI8)U<4FU7laRLbNSzS=Lg&u{h7SnfJSL zW5M#S`qpK6;K5&Cu%vc-@#^i38*4f?ZCw5M%GTBSjpemnjVoGqVvIKMQNF%>eOE)l zvTho$aYJQ$amCKk`bF#WTFSRoM#nARw%S{?ti7y!+d4^8zNu|}>&8Xvx8D?9LH<~D z%Qv?Mn>H`1f<9WyN#?DaNq70}r3Ks8c#Bu(Z))9Ew!NsPd((!(K-@7ac5d7RzU#}E zKi<$?*bI4FpsUC0OA7OR8;kw*tv=XiePw;yn#Wg>4OXmg_7<&ORou2Zuc38)Q~lK*0rg%rE7CJUW-I7Xw^Hy(z?|MJ6vX1|@ z3jL}Ie(mITDtR|}HhKP{#fuj&%FA2IX_k9iI(&8Fy`&aDDC6_C<^}T@+m&-iKR!9< zkKDVy^d%T4e?dMsky?zk$7`6jd%89)H5(6d!S6>Z1LB%%xSxl6;~w)#T5zeXOTFLo ztbWA)=ZXK#@wL=%IA0Jw6MZ8m5O2H3pr^5p80cS9_bZ%7$9+S8!7l4W_XCB)^_04| z&`ksUoWkK+s_sjju4nac*a^@7@mIpBUtSFOTthILZNU?LsR1ATRiu^9m&efnFNl?v z=vr=qPIYg+zKFJV_P#ojFrnpx<4iCihf;P7MJ#CIs|NzuC@!^jg7e730V+OFCEoL% z*YU2=C{Bs@oM}PNBjTtQHp93IDd=PTnBj6Kq=2{zdgIPpAlq@tFznovTofh=uKZu< zi?MG)E(H@khN@b>qkTMj+TX;3KYShd4*(~A&Tu<0rskD) z9sZw3;zjh;(gd$bIcUsygTeAtF{Fat~RNqvi}^}Pqj3e((~U- z{_%7vUevsLR`U6>gsb`Vn(Svl8c@ME0G_S}MNex`r-G;Wa`+Zm|3?94-@=vX^DqvT zlj6eO!j<4xB-}3HO1=r`Px%g^CQAGy3p`K4rvR?n66lK(J{54|mT|WMZcG)ANjQy* z)5K^wF7IeuBfM0~N&PF{h;*fdTl?84;nOYsD1A{s##5pmL;v)g6IGrwz>W0BuLHkg zfE(q_m3p`GIa|W5@(xG?S>=6Q!ma)MOv0`5{?gEok^hejaHG7La{kb`#u%A*9rzIm zCw`2p*>e*9kOe+qpf}q0H3_%!PfyXILi|h-xBPvvgsX9tFE&ZI8dt%O80d}TJ}2SU zaepb{)^StiyrprC^Tj3Mqvnf*Q$LCq5@3q~E{}M)Tt^J>=s))WvsezMP%@3|QbQ4h&o1YUx$ypQ?Yg-p>YqR1~~N+HZvg zuGV|M1+Lb6CerSXy87leZ=joDkI?8{>FsK3XnK_Y|CwqOOg89&hE!^{Yyoy)C|m&QR2qd@g{8M4Q%VE>+t&;M*ps8puN%C>17M}yKqQhrInDRl%oX6 zf`*RH{3SQ7jXPv7eDlVQ`-7>svw=6Y`Q*EKF&{hBZ@1o|zw9a&*iZPr7ePd(TBe?^ z$OLt&%s&)TF4L;YXrE6onHEIh9(i|0#xaFZ@k-Ta;f7K!j{?Z6f2X*4PUd6u*&4+= z8IGel*xv(4;~R5pYUz@}*FzOUQRA!Yv$8$)Z7kI)csClMvGEm@(8QBfht+L@Lh-BU zt@Zyd$G=|tk1Pnqx9Tq)!dq=ZxRn7#VZK^vx6AgUufPL5Vzb7egF zqAQ(beUt@Q(Z4%GgTpB2bhy3~M%{VIk3Rq8BM(2n80XhM{P6StuQN0T-hTX@Ykl?8 zBTs(w(#X-*KRxmk&Urh&J#x?M-x^tra~sZE&i-^{>!+`eY{Pjw&i?tU2ma;t4?lm; zD<6J7dFiTw)=N)~JUQtzBU_&P*2t?r`R2&LBR?H^@~NMKuV`T4(MO-Z=h7q3KeY6z zk*&{N8fiWD+{lxUd~4+T-~M#u)uXS0XXU^{k3RhTlS?0YegJj9`i-YXT2be#JDwZa zg1VkuN<4QCTz~4}=O1o+^!dDRT^f1#Pd+{JP|7nSi&CE(x&GnTM;3jH>bhef5A;R& zYu))2^#9wDytd~??s@9Dk%8BKGV<`F#Ittb`iCER{z>pWwCpn@i#C61B=OVV9PyWb zbEFkIzk2i)@T?giTW$UHpFICi?o%U+9zk8pzB#fr_ZuVsGW^q#ho2lCNz7X}aDCf@ z&o8?4C(mEshPsx0YUKK)Z;x#CJOh1R8+mo{E8tl_u*LJokn_;<*FXH^$fD#=j|_bC zThQw{wC|@Qi+21BJjDZB&i={s1CKn(X>>i&ku|#I(+^eWz4o_}#QU?ZBzmINiC@an zs9!@aR^&sUIh@hz2Kd-iM`-MtYwD9kt8=m<(;q<7{b4HnP4|g92P=9OKH+c>ZR(RR zI(^%y+y%ypzJzOYc^O^S)9{-&D$54mtSN;bhQ{8x_p29We~#y$Z@3(b3?C(bG<-1x zU;I3L^#fo0{qn`=Eqw991ipB|=8NQ`hCdG3{L%2=TlwVmd4uiO%L=~p&b^L-w>z8z z@077~*K6|JZyj+BjE3Ft;{?P}&%lNsojU(>`1ZxGlnnYp!8y$BP96CBlEEu~A07Os zqoJ`pH%pzyPFKI-@;vzqmm6b;J$Z7DXY3#D42{*z@VwKS;d!U!Yp%hIKM0Mr9&-&2 z{CjAu<@=7Y;iseLTR-a<8~9|&AgdU5U;H-c`&@%9pKy#_{7A`Q%QBRC2J!Z#YxC;@ zr`NZf$Mw^=o`LI6L&k%^UBK@;2QPje_!Mv>@GHQd0X_!24fwmjp9DS(d?)bNfd3r$ z5b!+Ue*iuUybpK^@TY+P7%=?`Fl8L;rv~kXX5-k&eL#y4(BO2e;wykoWF*163$=7*^BcRadzSS1)O8hzvEZH8~5;HNTjD?(moM&mH+h{<%_IkIiI*KSVB%O<{vE z7aROl5*wuWigM@bU-|O-x);B^zTTO3ss43*ew99_UaEf+pGW9(%BA{u@%b(KJn0gZ z^IbvQ4@Rq}hoaT3pH6ruxMSG;Y*sYT^A8DQb>Ht>&nhmuFCza3uU(kW4nF;(y3Z!O zLps(u7oThT^3rq7pICbCaNp8%M;ex%%Z7f2j(3o*q$BhrJzo;}gX{SZ(BZSBL-M8i zFXHnT=yTGg`mf>hm+7C_X<-pS_pr{}i9k z(`V16`j6xDpV8+8U7zXE^F436$68-^Ll=xiZX=AlKZMN=KK(~^BW_`*`Os-0bXo|V z7DA_mik)7ukU{)g2bMgHiJ}zV4$Bf}Stv#c}B8L0>88 zo8!XUEf1K-ICjQLv{9hyfIM=^U{L{ztKS%s=u0K!wGsg3Om-ypc{~qzr z9MAs>@yEITD)G-6&;L5{$GJX2{IkdNze)UYuD?b6bH?+(TQ=ywpI=wvf*<}tYrZ;X z&beGi);~G0=A&HnH1gUTm76iQ-Gn*r#(}qvlws~$I`GcDB?H&zJ<}$y*J7t#m>c^* zPXwPI{7ER6j5+b7f$Ny}4P%b_{MpDV(FEAf{^K@wJM;(|Wd?aDu zmzbxH{wg#$hI#7`ll*yaVLtoz&vp*H^U0FI;f0BL*S{Sa8^+x2-Tw@Y{qUvG*o)sN z8GH$Iua{pc8T=9EWIy^P!)FJsv9VWvUNZQ~&mDs=e#0^NW6a-f%tZY$Huk?7O9n?i zOy$|wtC;8gWJ>AayZ_0?Uc(&lr}Ih&-}$6t@O8`=e^ypH`1a3I27Z{dHSeX5Y#Mm$ z83sK=&_6Ubb{aD83k|+?Wb44_uh`(*9a~_Nih<$3aSaTwC>^}+Djodg(`;-sylvpk z(`@V)U5>#wYqk%(k>nWs`RCc#8|9uX-J0BOfLm9fQNG zkwZTc8bdA|yfG7a8XFtNJaTv;u73u0NCo~!;C}=y75Fp23t{KQ#0&fp;MKq@oP*at zSvGKe8rBfsW@FdqZOOa-5%9c(`Os+>?e*||As@rq1bI9W`J4Ln-O8s2=o;2CPfobE zFUnUfBZv7P|g7cMP7!>jo_^&b$C;xSInKQDEXUJEhtaN!3zy~jayAb5aV?jwFJ3j^%h@#G z18p(kuS~t-%h}B7K3+bZ>-6MjRt{V|J%#Ifv24nf#9;ZrBJk&dXP_)_CGne~u_sRl zMuU#fKPTckl^r-edGuo0WXP-F@}4}MHtNT9)QfdW5PEDe>lr@jr0Zj^Zh-Hsc8ukM zZ}^CF(3i87>o(k|(}$M}S`E>*S>?+``GqJCU+3jl6VG;&XC36 z5PvNT_6$tMIiG*tJCKU=GX8nrz$BcP@Xs{^NjQ@oTNcz(JIC_yckLT92VVT+83Qk$ zoj&l&$1(@r_^W9H7fv4<{e-tS?^CINFz|%y?$N(43t#!0uiSm*-I>n8pZsmsz)$~v z&cM(9arVIf`pT?NI(cu-qr*U3R^qMQfyK%h? z=WSH3W^{NSu9wwbK{?>s(czhd_g@)aOnCq3@D#!auHZWGfzjb4l5^MSuovgA+;wHx zrLTcop4a07#ya*DH(~5=L=GxLJSat6C_#Mwni7*fh!-Ekix1+(2k~N}c!Bv5Kj=#= z%!BwJ{ZsYRyTzZLlNf%z!TA^dsqH<_i8m~qF^=E$+; zwk}o$N)Az-n29-)adtYHYm&?1bUBzi3Db0OxKbGQVAy^7_lPA1!O6a;>nPYR|3v?g z|0DhEJjy&FwuU7@qZ!1xtgEG~JJMa>fKSG`r(MVV9PVm4)}*mz$2z*3mmNE?tovx| zvBMn}ZagmViDl-8#*X&R`fj$YyIJdArZ+9zc)Yu*LRz8Hr997Z|0Xu_wU&hE(|wxXyK;Tb`l(>Ka%C*YP`^yMf??KJ z-_~{{Qs3Fa2VG~Rqq~DfJV%Wk$GW=fk9Ft&9=LB~eP?@fr1AHno6&8&E%v_T7e(mg z9VlP;J?KNDIjYa{tG_ox*yvl{pqq8DNGftHHp+GB^9V5$x%KITX~+lDkpI)Bh2vd6 zCmhBc6jS;7ws3wJsPTAHeYmT=AYTi2Hy&v|mR}S;a=bp$6sd1*3wI;eg&RBC!VS%s zy@s2bk^edxo14OzYKJ4u-N&&cXbLyxgCb9mrIw@ap zC~eT-$bjnEIEM(}pgj^gmg9QxLaKRPzlwbv&Tm~03PQJ#V+b(hJcR2m?EBb{7>*oq zkv`Lb_Vm&@HTnK%G?)2e_eXPm{3By@eTiL&z=qA-Yj8m?>waRMPNZ_{1Rs5-gX)R) zMRV=d`1(+722s*rb^SO`*VtTM|15E>MVyU#jlQDGB>%mx?*&nx-3RD0cD!(NIb&C` zS+;ZEmW^9Vf@K|%PWY03(B>KKqpiZEAMngmBAIkrS*}Wx_h~g z;od2BAAO$eV{jdQ&}U=h(P!eJ!yT|b-_G`*`>-3nh;pmldVSMKBgP8Z8!mQZGII-D zz-}lwpESOaZPJ&sYYJY*UQqB-c2&Vk*s}^=%!U-ah+R=|9=ojITy{~x3)uw)FJO-; zcs}b_a1J}8;CU>n;JHjwa5n2yaFDeqIE&ROcn+&o@N8D4;90Cf!82J%!82GsaEWQ( z=`5(=OqQnLX%tc<{tR}*DdVZ^nu62WRRvFB&jMTh9bi`!`eb%d!D;M*!k^0e6?zJb zDtHp>RB$q@Q*aWiQm~&@09*PbvJkN4Z$7q0!CscH@OxMg*s@;&OH(jnv4UMJ>R>k< zPQ%|3D-`TtQ6|&*AR+!rHy@LZ5_CWn3hlyqW{f$vyO=BNa<~i9SQ<90$&zb6L?7A ze1WeD92NMQz-t7)A@CW2;rYBhA%QV6IPMo1BZuP(fibc;eoWwyz!wCr68NIPbpl@& z_y$rp*}DqZ@9?m+fR1YgzC1<8jAK9RoT_7=z(W|UR6dPkH@h)i$C&~LkyMEu6u4@R zj&lW$26db-@a4HWCj0muZgvfm2r5tZAe^>P$JBnpRe3tr^$W18i*&5(7hs*3R8aW~ z;P*KEECjzZumk>8C-6m1_pqy{gglh-z#iA&o5XL}C-`9<8}{iGSg+sDh6L8@_p{(5 zI=^9`sKADOt_d6z?Fs#f&Yvsr1%VCwut#;eVV^pILxO)u;0l4$Fs~&2b^QXYPGDWX z0CTOG#@w@lEQ&FC1AQMtfA?ejU&h~6{N-bO4#5WQv}FEep97-k&piiG2Zw9=sLSp5 zu@rocy6}0GxPBC0*R|vC2ysIP9CUtHoZk`Wx5as-&c-ktc>Xu$LSLmqIE?GYoNS|h z*}N{G`E8=XZ(P4{6MlpKUY+~yb;Q^YAkBxr7gO2v9?}FyHvX2Su&?y^Ib^UaYe->z z8W?qxG4@<4=!rUgK2D!VVdo+~SQqvidNh?iyPWlF0U!Bd>{=|9W5`R^n1}c~t1xH! zZYnFEei?|4sEfUt%APuj;;n#w@Xt+Qn|d&|sDKl6lwYp#Zr1sMKc*#nqCi~Vm$TSl z4-?ZqH|leMp5`Hk;`DE(5*?042hm-idpuEtu9ru|67<(`bDd3I`9r-rS?JrR(bJ*g zcyT%F>!pVB1M;(Af3!YR&|fBdCZql|7W*OW`)n%gHz_#@WWzp>gHG}c{IxDomQJfu z*k_N&e91J}9Auw#9aD#K?A9?2J{eOmkue1e8Q0k`1rM3tV#7fj?zG`t8`f+%--e?$ zOuL7OmI{AG4&)gBK>k6ruZd%{ z=S^K6FMm^culXqU#$uhBERNMg?~XXVFz zKGWWach8vmLmtYP_i*c?gzi6f`Q^ux)$)J9HK%`g`Lkpn{cy&UH~U^eC99yn?YMk; zjtG!({P%mQY{x@Jd7b`*=JiuRjX9A&p#M`R)2#S?I0g1_dnqFs_UP-KVywlvzOYYX zIq01s*a!XpxaM}7`2z6~`N$nL%6~nTT{+IuxP_u?PD@Or2|z3t`hG2yJ)xz0$hi#t&udc?qgjF<^nNWPJxb+8KJ4qA#xO;t zgX`a~W%2=J(Wl$!Gt#3lwMCz4qt8mGNrpwAZKKagk1F&miw=8H`!QTZ`$^wmI-@zb zsc*K$&;4_5I(kl`&$H2U(y4>2^79os)t6z`m%!`C@D%lXxxE-VNH1T>=`&1v0_d6P zQImcaI6XZo)kpF%+)R0dF{Kwck;@C3*un8!1HbIz*wpXk@%oXx9E1NwU7lq>$GGvV z=5vaV=>1X>fCKTQUrY2cqzBVp8Wt=jy%O_*bYnuo{iUxrL&X|zosgI6=+oeQe5OqD z$v^#m*+2Vwla%rEF>O{_6s&svXThJEnh)$12JjP(?L;TskG=rb}Uncmh_}H(_Or>~e z+OJm~KYhJ3_}m{XLSIA&)4sg^S;=Posg$oX7%6Cu&kT3;S4c%r|?vyFddYJ{9;y62y5+ zUur*|@;CjhuXiac;OZOgU7m{YBFS5(=-;Q^Y18ixMZZ2}{4R(7>UgF0=a}`o^!Xum zm}o!gmuu1~-od|UQOes(@q+xzYA=nCe5rp({z|jFgXCk8Z^`f1;D4_g_8@;{(*lDS?%xZU2gL~*aw?vR{t$dL2RgY+p!eFZ@liANMy*Fu4AhKaqa2|MvCf zS^b6h%>9i7n)Rc9M-2V+{#D1@>Qqdp%>E?(O#5Q~PW8#-fyZa+a3P=gO@BqcCVtER z6WG(oF+8ilKNS^)}*-5)TRU*#N4;f%ft7E#+5xyzJ}E zmhvODH;9V)q3568xir0@Ltr&O^z|C^XQD6X{YmyQ_^ChD`LL89FkZO*nQEUPHG%T0 zuxE~eG2i9(sg>=sW&-=D`Lj=0sWWSuqqGb^B!YMcmaJq^8nUQzAMm=)^~H~puMVpJ+37wVD2#sbsu%r3CEUZtadTIS`tWV01zu-I0>G1D2mxJz3 zjNNC^`?NHSX0iYd-d~ehD8xc0?oCkUd)QCn`X!To;P2OFO{Mj{5nt`?`&cB+Ez{3y zDak`p{m=)^MZA1lU+?v4c(OsN|6^K4np2YhF)fp?*Nplg4-L1|d5YR10Q0z>oDi`ok~xPrsHhmDV$oy(e9h`max$qV(5!tgo&9 zAb-nnlYd$LJI%(Q2?wy~SYOlXP3phtHh#C8HefAzs=wI%p-)r$zh6u9QFt@jL;W3y zQv0cY^#0B`BQW-Zh@RQ6V)tVT_SpIx8x}@?+tz>ZFMEHg`+XD-)%ehd@kj9xhmZCb z`g-;6JUQ0m0XEo#KVd(v{J2~4r>qJ6N%0r#$CW=+M|I)Q;}`m0S-)WaMOm-(Yo4i@ z;5O{pr>XG+`}N%45@j@<&={fF|YJpRvX2??@4qJP~g_RwyN3GGzG!_ZMk>Nd7|mG4H3p$ngsC<-guvU>_W2ya_z3 z;*6^T8~*ZwLVy4LB`EvLtoPnuXnczK9`_e%eNO%oMDq>$|CRCOn$rK?z_5xlZU}7n z%cw$s|NSLb_Lqx3`g#Ks^7k&jglzs|?qAS+)H5{)wY{JI^4~7LU_9Zl;*0(bkBKw% zN0^QIs@I*V@W225Li;bY9_RCymG8a3L~Z_J?l0;7lKVdU%YVE1OHk>5Zy;C28TkSm z@nwxdfB*eujqEQ&zE<+~_cFc=+5E-aPt*N{*7yHc$Cr@O|K31_iZiNg{!**Z-+zCh z{YNvtY=7_l#n=z!>vOD+%>7;6U-I8)fBA10U+R?p_Xb*2oY86X7fqqR|NcVz$)>-& z;A^Jv@?OT5kg;Dw{sDh6_rrC6S@}Nu%YVE1OH}E9Z{UoIGx}}*@|Z&Z?feDzD{*Pm zb zBpkfH^SGZ+9^bfMVCotW&3t6r5BOLl#ZCKD27RLaR_vEx104BJ?8jpNHO037nnL^2 zmOR}5qU^t5;Do?HhuuEF%6^8i-+DpmKX1xK1H1VCO2%aaC+PceR|Lj>KK5S{(|BdH z-^#z=37Gq=MJ1g-r_kB_|}$5(v48kKQ|kG(15X+Czw!jI#AWb$v1xnDVj<5Yoh z|Jc+J=!XA4u4Qm~hRHvT<7p;;rfvV+B8V8q`#PZn#)C8elFldc8ipSZCJoLw0)gJQ{`{MopZxo`;`fz_D-71Ph z#P6I%YWew!e8jU<{TpT)U#WBpaAIb13iG<6{C-H_*)!5OP9i$?JA7=^@?X%G%t6El zKc@pP4f+kt$8#zgfJ2vW_xFBnx%UjHmi~p(_q8;>S19f2>s_hP&ua@7$^DD^m*pe} zgg&^xsCSjZ->;?k=S7g6n5HAKssjS@oek`MK_uW_#@Z2719F|0>fzaQ}(-GK3rchx?tpS0qg1!^%Gi zn2)LbPyQmuOUR#{k(?vsducocGt(9boCy7MZSBcZ+SAv&NTM$X|6+ws{$qWgQAzPr z>F<7Rp7&Wvf4aX+_D|~HA(MU<>7CAHAhP@IXU+q8e2j}>gnEpfl zc*B%O_L}XzW@2iu+8?U@sl9T1Ab(Ng0q>u8t@e+%pFI93KFR)@3VE_U&q6=M*Nc`s z^2bqAUn(#AH?42f`H$Maj&b8n?;m4)xlsSO@t19lw*<2PT+4sG;2&r2d6xes@_6OL z^wa2H>Q9P4M!q2XBU}kg_Pv$;$=-9N@pTqhwSQmlJb8RNcz-bKet|y3KRtiy5sQ={p?EbI!&s-@UsN;wH+v*=a-@te< z#>e>aK=#iy?Elp)>d$Pezw~?+H1!?dA2fb)O?oQqe;eb$Y1v;LPh@X(JlxvebiXX@ zjmPT^e;RM^dER_UKl}Ki{JBP=$LkL(Khk*mb@CIBM{}+I7;k^rd!E%l^n66Nw?fi~ z`p4Rz=KX{CWyCkVe?kWTSE+yGd{a4YJluLb>HcoTbBZ^ZkGie?qVZ8>)JOAeCH{}k z4{^uG(5?DwuD4c_KfXWg`Ah8&eS8@CMUP+de2x6u$`?F-N%>-Y{38Eok@TVd$T!9d z=1Y)={A}bC>W_KePKAHAx6Z`l$B*j&)L-)WIXkZZcz<}T{;-da@#9C%A7Z>bdvp7b zA0OFX&D5XYA0y4z9C7lE+8=T9hctiV<3oy9w=+KGde2DuP=8qatMTJk)ZqVJjSrvI zA2dGtjrxAm@p1L${X_ovm?ZyqFg_5ktb76go?v`rdoN1*P=BmZ$A>iE(#OYTg&*r7 z6aV((!#*DPc=22Pb6ex%1Ti*&q9h6Xg?iens(0U4QoBet0XNq+O7$>iUB6;R_1Cyxt&w zIlhgLPp}8#jgb!=6t59)#QX`*>+1e3{9VV&{72ajg*_CE@om3mN| zpA_G}tM{8{@%z`^7(bm2`u*$n=V^;HmotjzHyAw+jrS+ues^b-`0Kz=@#bFKuG$2H z(Y6@wCw~I_*|bAwJiq$X$>bP+0Y1Gy;S(t=Lu0P!8PLH)>wo3`czizsI!ik#=-3aQ zuCW-uL!I9502~0Fw`U0Y08fs%F}udKYiH?s?c~W3dYKaN4UQo%!EM&}GRgPyckuA~ zfu|eym!?BLZ~*s{-vAN!zk3<%pMm}Qw8!;2>HgWG2*3X{%IyLE9Nhn(M)m3TVa|w0 zf4-d->7Zvop4lE?ypMv^!}9`A9QLF3d4N4sKX4K1BWnwLQ++P#AC9>`cpl%-2lD#v zMb&L6>jTSz6zJpghVW77<1W(txSwD2ANY&TLq6z+{-h5T?X2qL`UEtWo4Aek2ebsw z5a=AEe1?`}^&jx$NCNJ61^udFZ?|G^luvlukVp0RK_Bq*`nf$5==~j0qrK3F->%I4 z5&lus>*CM9gU+9iMt>%l_QdlWMG+6L-ze{lBrt9-vOnyZqq$uneB}1v{^jzR_9OZk zcLF_N?sn?_l%plOs6P!nU-KoByfmVJeL1_oCll|xz%hGHGP}4FINg^_1qctP04MvB zg97KK0=s?5G=U>}jsyOc=p@y2`)0Apy*?+&hy8f_(0=}Sj^6%MH}pv){V<+Te=74v z5)*a%p_#mWS|;BQ=Kfdd4rnKV(c$F3oPK<|MNb9&ZeW?-jmM#}Unl(cL?po9XmZjM z73Cu{WqLsC0>51T5WYS!%AaK@AHee>p}XBi`8qtHY|v?Z9fUl&d;_+tEjpF&M)@wY zd=nmEw&(#ZjQZsAhhZI)?x6Ar0ilCt`4;{AG||3*Ru6sT^7rWbUsn4AhfqFjmTyCM znew53BlMNaA4B(9bgI7&<@Rm@e>3r8dtv{%r9N#0@5kC^nMXs=3d0ln3vx1+tP{Chw@YSKT1_NwLE zKyNqcC(vG%ehl;vne=cAs$aA_om=} ze~XTI>jQ;NqVf4+YTD%9fJdb#C#&>yuS!ozk?9zHsWCiY6&o_fPaq)Eiy~9-fIl6) zKE#Lgbcv39GARZJ!yMry|9ykK>IgrhJY= zDW*J**QKcC*DO)XuUw**&tIaJ&z+_6=gd<1gR@lrOnL#6ZVz66da}x&maOtGSz_r& z`DNBD6(=W~xPi|%qfHNgDS48lW}q6_}$1qEY=z2`4RFpnI4SP*Q>a#O~vi)DsF00adWeT zS#jjwg&yfr>Gkbx52o81W() zIeb{=4`^*|G+$5BIDc9&a^#3aNBIbU;#AP#FU?75;O{2?K|BgZT3RH2_(yB2ito8c z#Yc~-_}DQOf9OLh?&whQi4!XB>{Ri+_o}$3N5arQDJcm3{ign>dS!ZXq@@-4)$D)t zUmFd-B#|$QBOND@|0H?@3j>o5|LQq~e5KPdzKSDDH|%0~KUOo{P=hjfUcfcMdY1B$%ZUkyk&oQ; zd|`+x6!FSjf9mh^L%#Fz@9+8h|6X2ht*^v-wmflEiC>6MwbFdb6VQ@?Eq~6@j_sG| zm@jt#%kE}ESKL`Bhyj-5EKmUHOtpIgOYsA*TFB!!=H&C@m8PT!~ZZJ zhrML^RlDr+p@2Rg=lr`s7yQ5^593FlZ*%$)*jJWUy<3vU`B7nL7wP*|s(&}=syxJF zX};Zx{*dKu-yz9^{SO^Ng`wRfkJrBgbX8sp=zM<8>pP16k>%~Iw97+{LpJ~2N&R4LDo16p19 z8bsBN8vJ3r6&3t%8#s1IpWi+p)|Z`i6z`1n=X9-x&tDIbKD@o*Fy#{`f`z9Wz1hcaftI9`3v)xN`3x<_kCi18qjLmP5F>lp&dG8 z&WBK6j#g7+%BT9eD=GZy^Bv4*e6SanN1eduH|@t@a2z2qzd6>S;*Kg6S2d`(p<2az z_NaL8UKLj#QL*2z;-n-MAK9+r&Rr`0&~6DMA8bF?j@DT7sa;JH9r=0pJt}T)R`KD( zD&DtG#r5?nZaS*sd&t3YgoHgKJ5+p>9NeTAM|SQ-JTdjn(e^m0{|x^^dy@RX+eQ8h zMrv#2^3%op4)piQ(tKq9ewiMO95|rjyY5o)-FK__;6W8PHmbO#Ma8YH5{CVE9H#ur z?T_(L9I32FJd@@F4IiTX&*|`2%qKcK6nrFdM8Ws){BQ6h|KZyTl&A3x`2lUuo*M?{ z@^aPTr%WvB#(NX`-olr2e)r{vGK^Oe!-xHClos=&siO%Kk z+$5EU{7t|ze{~fq3~e;|t2V3ryMSeVHkV6u)K^_i`DK%-PdVta{5{q17d^jneX2uJ zdFZnnSe73uk?AKQd-gzqP&vs*`vO`C=(7BMd(3>x{}<5hyFFdvix=HW%>!kBPb9mGxc8&x+1@p<})>B{l)Mnm5vHR z>&@~T;7=+Y6UtDrpikEh+VcPYT8!t=22+0cpqcMs&*^yo24TE^gW@sr|AGB#{;v;% zqUZZ($^NkeX1?d;Yk7V@Am$sG-_`TsXy*6YTFR$99wDC=X$NXao|VrJaJrPw_Y*(6 zYUK0%)_j5I^V$RN5?$kjtk|;{LLK ze;Viq^!Y{_r+4;zJ=7hpJWV_p3N5NyXLcR9sgq zqCBCX7#szJ0q)pRR4P)?3v7nE%N1U}Vb{6>r_D;%(bhym6z7 zH*Z#PdAW+Ks#Lsdmx_1qR&i;mip$DWyl0P!_wH43c)g1E?Nf10jf(HOOT~BJt>S|l zB#ivGj?YKdiToGP{F}^tjPX*8{yAjy2gmCx5wFeu!TtmPzKZWZpg)Qu8_E&?r1?N; z8RCh|?`BSUf1x^zc&O9AO6ALVJ~!qE#gPyNTvlS{^YvSiPuO}B7jHy9V8tfhunX~< zZ7^{OpO2Li=JHExX#P@782Vy9uz#JA&++^f>aRn5Vr4|f_y_$!$e<4=5XOSt*iT59 za6f_a{SXf@$oHr(fc;L>zK9RyiPspdKbY9x%+c1ZvEp$h^8b2ZIbLsAYtiZZw6(x8 zz2pvwj(p)>NAY95xj#^Lr%VrMcYt3mziFjRM|t-K1e3KCkD-5#R$e621KORCCzs!{ zN~Tl!5|qEgEWd5FOb=))QJ-9XN4`a0PWro%;6itrF(1+n+t{130&39H)dH8QYD?s}NKgJ``v0xA76CF696{7tr9rO86fk|J1_N(+= zpck6*m!SOyJ&Vi7d^ohiEWZ@(H|To(dqH1f(wCwAD!m5urKbGlXunF|5Bf5qBOjiK zB)KvELdyx$dRFr%nfV=e0%_q_@e@`+oji2rWTK?(z9{ri46&FX1 z{zg3TMb?P>BaD2Xa4-$@Vl&?-@%?Ink^eRqAVA;9 zfFt3uG8KpJRB=g(ito5X#pUHHE-g~=rcEkdxl+a3wyAjIY87wUqT*GnR9so9;vG9w zoS(1a-Mdv>uuH{-g(|M8QStVrD&D(S#aoxCc*TAd?_8$hyY5o)n&lEkzFJa@_$=ku zBA$gkv;-{2fmz5%($^1MY7n%Iac|NuBZ$$;hzs`^NQXJX58SzBI+qYwU zOL)y1j9(qYzoFlHtOwWODOBH*z##INV`q~$F=I`Z;Ejq=+ zMc|jq7v#uvl=mkgIOI`0rTC+*m@m@<+G5C)%NH$>=~Ny8KeWg!zj~oe4`?~4PcFYU zDATDt0)A+*S^my!nI6#QLm#>P`nfWl%A@~7IcE6{^JIEJTL69K@+GrmI+aKNhvu8* z%Vx{;fVL3rk;`wIBh#roCK#awX8H0gnI6!BXs=v;i!Fcq{g@zx7E*cmV?fJB`vw2Y zl<)k0OyEL6q5}u`^SLS=6Rc3SNuP)ItMv7t&o$-GLi<%Z;zMYjS$;O!uhL6EpJmeL zp#3Tx@gp?bl%Iw6t8|Rt&>T^Ix^{r0eVXC2Ic9<@tY!ng4ThV~}U%|0TI*eSAND38x$RANii| zj~VNCd_Q*wV=8$nV=sjQ02< z9{zh^U=jQENl6|6m`LPrf1ZlzeX%lKzu(c~Urs+Qj3Xq<=Pp+9(xob<=YM7X6)RM{ zWWI_E3RIkvqvF-8RlIDWii?U=ykLQf@4QpR`PnL7yH>^Ye7mgwh7Bq%oTuXT>s35= zu8PabRJ>}oic3mVymFR`H*He!nmH;iFITZAOTx%kJU&bLa&|7_rIb(SETQ~m<>Rbn zh$j**EX4ShaDG1Gg@jkF!uZuO&xall#-l`^#qTHci+q}+CD`TV@-xlyizmx;+`k8X zJ)mVkU%7livP`G)NO+;CX89G9WO_iGhW5zii&A7dmB$1llwp=%ohs7&ZlXlklurh&K`r1~L96JZBE#iG=6Q#rW1S z{IfWclY{Xp(cL`1`^5a}Usw)dJ%8hlUco=?p!~fc*UaCx-y6VqQlB?V zW#@Y!jE*S&Uc)0RT?E9@l)@g=Vgf|bf6MXwy88P<`tLjTfd^4#1iwe}P|U}FtQtSt z#$Nt`LwWy~{=P2{yC}c!>kZ|7U-swY9_VpNzn7AN?<@1A-v>^0(({>@A&x)4PwNLQ zJwFI-czophn^V*42-E!#e7!SuD*21kyr0SG(5G=a%16ZSPa>;9p3fN(zn7cvmd@olafR0( zap}L$L;SEe_|qwV8|%{|{N80y!I|m#7H0qa2P~cJ8N-hd)9(=h>-Q&GI0Fe?rt|#3 z`SE)lk+*9uDWfkE6Gi-Y*Axjr;x5D3YW+0Z+MH#Bl*01!>Ehm%L6^Lz@MS> ze~dp6g^mRF)k>Bf$&KmdaRu$ojx37l<#p_f%u1m6iUabg ze7YMGY&x)Cc^3SxTxu^rI6cFI2{0XeKQlX$7t`y<6;7W`Klh6xN_0ndB%OW^SU))3 z<-vvx9lXBm$gDh}uP8q?L5x5u@AgG96R=>x!R0}p^mL)GXpalV6bF|--Gc=R4ugN{ zEFn+G_joBD;Slv_xI}vd-Q}fxh4P%v`}eBoFZe@oLpuGSChixov>2eUYFq2s-p3`#7n*+P~R}*%saX;C|+wiZ^;0`uHMq{c|lk{LPby^$s1}-oA*} zmm}c^x8T+pR$-PeZqV&#*grcGj0JUmZtv_!c1*7y7tww`{;rAf^&&k_J>Nz4=5=y= z&yDH&==>Nz{sp=`q0hXec^n)5u*Au(8U0E2^9E`D!4EFaBj$g&!g0=A**~d&=Cd}@ zo+REsl+TR*A^*_p)5~Lg`t@$2JLr=xvdHGbsL`cXePzbA?2e>f0t$R4x2>Uj40MZDql zWk-B@KAm6KNA*AWLy|uJ7}vMBVOEUfS^e$Jk?mbvornpMMfXK~N&5JP{+Mr&y=Sq0 z(|^cb#4p)f@qcQsh)-NU=rc)-NBBb}>@z(@k1yO`-Q@prdwh|M6g{4hzno>orC!{K zY3aw~$qnI;hzA(o$@=)={)Koqb&?*hI3|DfO8$!RrjJjEU>;`==UjGPF!SK6n|>ALzmM!VP)+yHb{M;kSFw6TQOOlu9?82Ysx@_Y&RgZ_d{dhjT zA;v4>H}!ua%3J;(Ob&8+mcI-6ynY@(Ziw+V*NgsII*Ha7IJke3JerD#@rVA)o+RQQ z*N^-!lf5nUN7@HJt*s^=TSUy`~1(R4_UXZm>L3P9iDYCYas<8wBDe$J#f z=<(H}&xxty%Zr5!KTto7FXv?H9~{7EffEB{Uma6?@=YQ888|zV6%+Z~u$MBPF@95N zJ&Hr`5BO6u3p(_CgjX4$Kd5b;4Saq`@9SWbIDXTd?_s54`V0AUHrr^H_eFB_-y`Sp zmqWk#lSMw`_4y(T0wSN7{xFxdNaYt!5%H7LseYQ;8~#L?rW%sJ%u>gXbG8$7esKRu zoFnWf?BUB&;}c8O^Na59h(G%8&x`pQ;?HCee}w)S0TIsxo;F2{FODhxQU0;&_j*PC z7WF%w!X5%ACJK8P`uRlr1!ktZ>;C5T%JYlth|?+0=d$sB2VFmCz}xThiTM!6#dEP# z#lig#`Kh>2ipR(g#ih;|+0XFL2lL=3>=CnnD8G+5q74}TvB*4O-f#>)EmDwTJ|JY-=0d#Nc{AET`( zudnBqV?TiL{QxHRJEupSG+ohO!b0nP^gp*Bt?$2%_v7IGD1xX)t@Qug zZ!cf@JuSQbx$L9At^T1~>HinMy?o{OdhGhAvBz$vKOavM{eFu)p0Y5W9{Kg<8RL`3 zA6Xv7pBEH)cVmB^$8UyRQtW3^zM|00^A+^1>Pd5U^7ed%$16L(Qxnq_inpBKt?-xk zyv_LgA1K}p+1^k0Ux2A?9~>W8-~$VMV1W-T@PP$Bu)qfv_`m`mSm6D#0PU?}KWOMW zwjqX}rd^8AAO7qk-`e}VAIE<7H;#9I_LFbNe);2HKJy>{qbuv=?9tc{@B53-$EJFo zjXmM{!rM=$4Ufc1FMK1GbpNv0a`(6k2VxI6vYNm2quA;n|MS0Z>V5an|9I(pZ+pM^ zYHQ19x<~*2&sKl&|NC;+HzxgJXVSNqu706r?HBIv{A|K%V20RP`hix5YW+poWk ztAsC7EPOw6U>nPk31tSZ7to3o;Cr!ZUQ=Ot=vj}%LH`V zI=NN(&I#zb6VSB@=(P25tNNo8&}nPqR^`u3Ko3no@1KBPF#-Lt3FsFlpkJJTewouh zQTP@MJwZFoUy5Nw_}P#c7L4e-9F9aMMz(QHADNEwHZj_17~oo?+fRSaoeva_vZgv3 zCSQsT#aPq%^7%?RbLvoQZ0A(pQSHUdO1@ZI7|enX-Q%K>Jxpd`qFnzj-qj zyoGxfc9DghAt&o|I^BKdw|>ki(_JC_$fL{YmFbH7D9Il(P?1 zx|Mt<*~f5b9)D%|$)RM{mwa>iNoaas(kL~Rf@yxlRtbA&8^qCSpmqZbk5Oa7Yr z)PaypePG1%D(VAu3@YtoH|-;1lw#P+`k6i;(P!Sg{@U^NEBf6$1{kBjR=*izEV`qC zA#+$qNrThR{X*4iT)Rt>lij7c$sQ|4o=%Wsq)~%KcX?zw(g#0CM(HAt$I;jXJ*?PR z+PJdN{^w|1f_(EwPLdsU55=6f(2;DzbCz8$)o1*nUi0$=^)b{pKHsYvz=04U4DX=` z8dn3kRznjy$I;-|tC3>r`&%PL+4r|bio5S`jTDLB-x_&rrumlLcYkM|Ly`LZZ6n3* z_rAtBk;_dH-H0|6v5m+zK6>@Jj2T}1=w;CjdAA$6)Ho4UbfQQ_a?D61#e{Jl5GO`B zXI`UKijl#TNPXc}RyOK1&QiUmE~wAhn4bO?ORHimpmG&JsS`BTRgpPrCZ!SllnBST_*CksHNL`?r#(I=|px5 zS-Q#o_MX^H>eJuIZc?AV=XR6&G;W_yWVdL1yOD3jv)#?cLfpFLK8-t$f1`0aey*<1 zZ>%*7OBjlkCYO4gb5lH{tQtANSLs z?6UG9%itqk{cj^b({=gJq8ofx9i@TS7TolYQ9#nJ}I-@g=t`-{gy2-AGkHlI~?JT8Jh zeokDC-GjrZ+qfedV}Y-n%;y;}qRemL66hPYlC|l6ZVPdy&-`zV+fyq2(RW;YcFnbg z_KG&7u{4&G?BkDywlmF{uFu*WX^v5vG{|}?bY{>=u5n1^eFoiXC(5r$Da>cV$;9u( z7Yg_qE^4(~r{}74Y!r(lN2do>deo#>=yaQWZ1jqwEiT>K#p#b*KYwnEOZZ&e`Yrh{ z#FuZj@9ntdnfXOz(829vZ@+23jXK@dzNkrW(dlWbeT>%A<0V{vrb-_&c7X6W38&lb zYYO;Roo?3;ZlfQgI^Cw9l`-(hx-_~RG`j5H^jMyMKa7Quv6%MzkW}Dpq^D!C58nd` z>|lqL8?3SC*@(u&?D zU!l|LTBWmbyL7jr)1KO`=)no-xf9T7cj;F0X)owj^w0$KigEO)vilInehQuJcNM>k z9E%}X@Pj_nfk@m-=hWo;qtRUEi`^g1_3@9m>l(*O!EZmM4eM9<>x6AF9tFN}4(F&(k%AN5*i_Q^$04>L+H@UnMY|i0{3w@0yTb zC)yhlSUCqG#s!}F0hCY!i`;rT8J=QI4c z%#Cc5zMNfC@G|y-f|s(Z3SPpVRq$dqq~JyDih}dlWd-N5iwa)IE+}{bdrZOeS-*mF z*ck=Sqvy$__RM9t(O<&ZtW&{3)}r7nR;S=OtX9FZS(Soku?hvxWFZC5VEMo$rhTWg zpn@}5nu75Rq=GZp4Lrwfm7mJ4DL9>7Rqzz{EU?wz0d_^9Pi7YtoW?FF{Hd&8p{HPb zLau)jet}HJ$*fMnNvul2epUf&>63_EMY%tGY>k4wEMMXGupqG2{sjCYuZ)q&6zrlo z+YQ{DApD!-hKXu#GebFb}mMWHO+ zKSnW{IegSJ98=CDJR~s1Xu{V7t`!)b!}-$$hNp2{CvZq$%5hY_PT&@SqXGv7z94X? zz(WG(3VcmqO<7!2Lx!Cj0muXxv&IlRXG`B4rUjwV&{f4La8K z3$VIU9qal9SlT8XUjV-cYlVLJ9hE2SAb%40BBy)U&^LrU@OWU4p=&xe?34dP9UJz! zEU;d`pXI-()AjmcFM$pFRK29r4f|XZI4IiF`La&W6`1`<$EJM*HtZ9`1cKTV68x71 zt`Io?$2witFTgGftm_wGt~Jx}l93?m#F)H>X+stI`|3Ry_ z2>s9h+;h-3;-GO)AL;H z;!XGs`mcM=Ljf1Vz_lG18n|xEqAcP;ew{Cmt&8~dcZikvRbJ&9foE{tiHDYr_8POq z9E2#kZrCRzN+v@U7Yhhvo!0T|<{J)7LM^Tb(ky*0h}j{%JTlwlf{%&M^pfr1h4Fe; z;rdePdQk9p-$afi`ZxMW-3t?pJ3``0BXXI12zM|qCE-)N^3RBJdxbw61?=UgfPk(U zcLdY)OIWJ&gLvo>PD35Tah;jOVRJ^1nDC8PGVw5-f@hk{*EhPAi(-l$Pz*L^% zn^Df_{}U*DtMZxi*R%EWbeyRD3KQ@K7>?@4p^%RKhkmh25Orko8j#VuI8Vo^f9QKUi{bxNKOJ`c zn??DMt^T(~dBcCHJcR^%`IWHG!*S~W=!;;+5#;|eX0PAyqpc|4hw>p?{Z~Z!eyKdw zZE)l`pE!pxe=yEQdD7!IEl07`7@zhkjqz%n<@!ihV_ewF8Msb3 zh+K|ex@>T$XJdRBXQSMalHL2tk2IF(N7;_j!nWg$D@xn;uh_l!o|3jb^)*e$_t#bz zG`835JzBJ@qrRkUN7=s8md5tY-3Mx`+ZsD}?`hj|vb3qFx}#*v$;#RSiFUNRv!Q)A z%2jVK*^Q!E#lHLlyNmLhYfDi_Gx!heJy270 zs$s=8s;^*w`;krel}jgm+EKdY zc4{c!e5$dhsC9qs&bHdZ>QfD6vfcJ=Z8)~;)DG&$3ZZMw=APOWP4Q)xA31qX2`^K) zbE{RRtQ=(u4|MIXxoNqT!k!iTYquVT{~WJ}&&0FYj$=ZPV>|O3+xN9J#I%c_3?VPuxzNY%{>99O*b`GU~JsP-ztsqSzC&+)~4O0USphXCRz10$K$tUV~dD?2TnG&t&KDxPREUt zcD|d%C6%wMq3|a0%j9dJdbOsS&BrPJ#~TY~dyk>LD@sq?%y&$_6~sq=Q8PhX_UU!s z#1~C@+|J!ACa`lkw{z|0+vBUbU3sN9^A)3iMZMLx#do0M_T(KX*KSWn~yiNBd^pVXA~k=Y}|Tkf9uNi#%wLz-qsokA6=Ed_Hh2n z`o@A4MNI{33!0l7>sK^2u4z2HGJjRk>c+!iov*R=aD8KQxbb**TYXnoxP3*qT?-#+ zZa)#;(|oL}BXXp@xxJ%x)ye`HE8vzyh0_KKR)$xu%3l*+Rj?AD!Ug%^J(1(h`ahfN z+q#;=C}3XUpYp@{e+A(+;R4a%d*>c)jvQ-li`NE0(VMcPzO(CO>#0+1%_iQh$^=k8|%C4H}gxS z`}b@L7ls=;v~W{vI|&ZcpUG*{Ae=wd# z#yX0cog>;#rzQB4zo~wV7BF<|NuC}0i1VZF|Ht#EzKjCmpJ#+Ba*Ms{_NP0Q}m7eslgWuFfo4z^5=Mm8~9slG9A#5yhHf|TS}=N zI3wRY>%b@C*lm!{&^SkCyYZa|ecaZt$S9S&;PDCKmw>K8>on!L#G>KpJ^uV_wY(hQqQ=pFE-QT+g$t}U9{Kr0g z815q`)Gy<`OWgOfO5Bfkk8Y43?>=!~&mnO?-d{qG^#2O=E7ULJtqLxm&2xSDFf|&~ z10U0OQ%!LBQNXs~Qhr}>DPN=Z?dx9`T*`kHTvyb!$z7##fE};c@ov5b{01u+ll(rI{a6b3JTqd z0;AWB8g#{iio!HYjk52OG-~PCgs^VWg$c`c9P6@FK?4paQkk_^oU^Ge)xQ@`w5Uk4 z<$M6?HC>cxo>4WVjRNXqS{qjtD#&u9gXt&F#!C~$FPzjPsH>;iJT#_A8*QVJI(aB~ zj3&{fUQ1(rjJulE&2YJhR`dWaE`vP-K86{}*Xs+jldfuK^Q7GZ^8>CZ?Okc*Q4*Rw zOsr!rAaKw5>9Q^s86m8l{s?r}N>f>!Cwe|@>Ez@-s2j-up-08@GAv=yr-ITHCThTW zW5u3}Jb_i63hdD}Vt4oI5Fsx3pb1-%ATG`AdnfSl6fU&~Z30g2G;b=n-d%?7d`dSJ z+{pib#}Dw_p()ofMc1#nf&VqlbNC`X4mfM~C6LLv0$!})0a8;=E+Eg(MSBqrH{pd- zvKqW(I=bjARcXG4({y}9@*IHYVg~?Gk{)AS729Ojx5s%t%H#Z1Wx=Q{iMPe22Mq}Jm0?^FznmLB31nV7f8EQF_^>Oobt>62Ku4^-v9sr literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k60b43c6114d06189e061601bda5423fd.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k60b43c6114d06189e061601bda5423fd.o new file mode 100644 index 0000000000000000000000000000000000000000..5c97daaab4be58c6581a4279f1b544550df5b02a GIT binary patch literal 69944 zcmeIb33yb;l{Q?xNNTkjNeCeW5~u+ggutRD77MQ}u}EMR888UONvpMhWVHQ~z!zH? zx0@M{x7dQ@i|v^t!UPytOM;0r&%_@Et=n&%s#B*Y$KFX} zBX93?#@^cEioNe*hyT;Xjvwt}hmrQvY#Lkf?fhW>+pF2nf9GU}=lUXB&$x1qzv#*) zUY}-JY{jpg!Tv$yb2?l<525Tuz6VYpzyIFTi*bGRzI#vqZ)b1>X?yWG+!1JW(78W8Hx#@7wV~sWzJ_!wVzCDvIDN~B`%m9<#iK(ToE-Sfb`r;lH8|LGXYKKGMHhni95xlPXxtw&kMub?zn z$Id=_@9BG69yq<|rzeK){p4dq_oP2PRFv`j(AoQ5A1eAOm338Y5$HwuG;e+s@_%h; zQOol~w>V`K)C`%=?Ry1j z%3`Fe4Uc{D^gV@-4i(*xvWlM@+EDnDp?@8GZRpkqSJE&+4ud1_-~Qucsy!#N&$nETB59*!kJ1){u*L7fRxjD&N6Qu? z6WHR>No?_`%@)Z`~h9L^gSLet+xP)g?cF|8__0y>@5p{nhO7*`15r@9uTQ zhC^=HaT5B` z^JZ6n>}SD|rpH|U9(L!}<}Wx#f?EdN$3BHL#mMtCXh)FtPgauD|f;_gCR z7t(y->_7Hh;M;)ffnNdsI&d5C4Zyzw{vz-m;H!b30{$o9j|0yK{tobiz&n7a0e=Pf zAAmOlyMT`X-vPW1_|0H{(_O%~0$&IG5^yK**Kpl~>o&v}BmSR&w*ub)ya4z<;FZ9u zfpdUwuYlaHe#kS@^e*s~xW0|+GF*pnU5e}LxGu)^9Igv-eFfL~xc&y$d|XfCnv3i6 zxL$_qv$#&f^(kC4aQzXkX}JCX*A!g8hpP+M@8CKLIUd9HPq-e(^&MP4kL#Pb{v)n$ z;Q9crf5bJ4>mP7Efa^Bs<1?toVq6d4`V-V^2d?#?&jUS#>l|F$ab4*c89l^)emnZd zT4(X$24~6P#uo#J_xvbuc<&bihbs_2n#=nC9pm`uG}b@rV*P(gWBt^(BH!WgD?hrn z_QfAv8+K-%2)~Z+=jeOJiSXO_K1AQsPlP|f_jl?0loOQCa}<5PKeBaZFtWAzv84C? zn+Dy_^9BGx&alz9*jue;(iek-jIH^5jO2 z9DLh7()_v`vcUJaj?P?O`?t{9$Da86+99{l)8&w9A!J$znHEB(g_@pT!dyz#lhu|x zP~KK5&wV2NaeUuR-(4rdd+@!EzB^BZ+wlDs`tC5>661C1Z|>Q819bl^KF#;K-|so% z9$}7wYn$28&ZZYL5AQ2Y?Z4$JcmJ&w_#AQfH$UJWxd&;RK7+bM-Tk!{(QAXjC)~B` zobQKH+5Zz}Z2MWv9aAxHY>uw!zh1;GLLN88Q90nb9#?N_WNYEvg3#ntLucGe&dPtZkhfD zk$!_nzdk|wsuSxyGJT~;zfPoIn;`v~6YG64{c4fELZmMj>Dl3;a`y8OSBLV#Y=t0nh4lKE!Li`$$@8b#b{R;FdLEn>r{u$6WCgA%@MZedPIn3L#EkV8y zK)+MaZ%aVWS&jTt&HQ&IpcjLFr=TB6K;KOHv&{ToNI-v#@(cPm6VRWc{L{?*zfC|t zP(I?l6Lh=XL=o@E9vlAvrB4w5kCcA;*z}*L^a%CfM!Qnzj{^Jg;_h>A7 z0^{0STh?J-yB71?HL-X1uExB#BKH36<*~E#pKejtYP6p)ddP*ju?J~Vk=BbmX~^rt zoOnv?EarZLn4|vT4%Yw5)okQV%t7BuEARhL%t_xq6dd_|JLU5%6=~HWC?>@~SXAtrSM@9~T z=N-ZRclU0H4gZPtzt_GVdZ~&He$^ElTvpM4)>YB}&J%28ICMko?L%zjKf4_LZ|~d~ zdn?V+|K@ku$Xi=B#ok!$L^%$W>Fj?s!vQ&*{pV5~BSZI*jE?@nD>06KDma4iu>X86 z@MUad5Oc}Fg^2$;^pFAkFTno-S_bgfffqv0izzMer+}{nUgqpS`^D9&9<%t-@VJ&lf(!C{zX){g9xFY`t zTYjjm<;gx&e(XZi7X)KeKBb@Q2=e)ElQ(f0@8AAC*q#HnhqW2T(BY5^_UDHECB;Tv z=|A^?mv>2d%(uKEmVV7@&i8Mh4US}j_itUn{*#C13_Hrp7kN^HSU=4kE=)yS$~B9Q zt&s)u4o%~-9$P)FFV$ZeD?<83NE2I~)tCBQaOC)*tYN<+`2AGGQ{LF2 zsl&%sPX)g!&hPl4%waF$Q7+ahe#o)jDrfLkC&iDPyB@Z4rDLQJ=?3>Y`#l94xNL*< zCVgixTm$_3 zO|cAIi~08*u_?GN;oo=0(r_g`HZ7>3dX6l@=gd!XV=w;0tk}yB&WyeC(2Uqy|9n~O z=%J4fe<7u3(N{7)5&M$s=HV}|4)uNY$(#E=$aVJr@oV|9*S<9;_Qt==j{V1zd9k;D zofCTpemx979ogpSfA_0w*1&^z0M`S*$2;lU8%SKrt-_$|UW4iD}mytQxei-flh4{jm6tq=VQ@V4Q>O2XUw1`iS5 zK0J6mu1QNtgPIOZ~;A{^C-9FD~&zo20K9(G0Wh@C2&7zdW zT}@rxo!#L&e3RFMttRHzP*+o1gTad1+PfQz+YS_W?`v+`(;mn5`vpEwY<;P3Z|w+o zv*PYXqr2FwTBv@1cT2ddE7ZCy)M|wGHntuJ3Ci}ywyyThP2rBNTbpmYt)PKG@M%&zT5jb>XhYP(xP>2+eJKZ)$GqE(>+HhC_SWJ6pru7N0fY?(jMu zRk3UP+E8hzuH6VVG`AAt5PhtaHVHxj)YGijN1z&AP2tYQhL27~IvcwhI}e~nXr<1^ zmXCzBp&sFnL`51J>-O&r?dc4+Hkz$tj|dlQF>tf`a7)YH&TvN)ce;+w_U?A_c#i7Z z+q$~LZQX&7fce&kJ6an%>pv3BjAr9?v9~3U6kR88KsDkc(1x-)y3O*~kBke;w&fKX zuYtv&qV~lyUza(L5Hpd9FPEkvm!=_qZPUWBre6>WVGfF^e7GeP2m#gaZwQCFT1x^( zsJnh|V_RTFXz%`TXG3SWxh2$%u`X2K-V&;7#OyWH(1`J`y}q#_gsFC@v$16;z_P8i zv9-8;&z`Qv?f?sg+77@&%<2EOO=~xWObVPX+|nHGN{~RhUq`sJyScl$y^YUGEFQ!i z^RkcBTyvrkcM*s#>;l-sDjuI~k7gg^@l(dcf5S<9GRDL|jd&-U!fM-1Qb43AaGCMF zBL2LH_dygF-_g)nUBlzOdL&Q{;mE1D_)RUHBTRp z_X|R|;3FB(UJHeL5bwgik3Ebc@O>K)=sN|}r-!Z?zB?n4LgpF0Gg9c`U(Ga2>`Vlf zI&)ux2s5wy)43**_)#kTXSV^0#*P;bmos(>n`N7KtY5Rf+`qcLvjetd2HHHMeYBPM?4hWRZUg?l z8sn^=cAc7A!sOik?)LVUt|fl-o{iWe@|P?tUcSV?gD%A@RY3ZH@lzTefW67TU3P>$deMY+t*1&Du5WiY@+S{wu%h*YL&fw!-lxOw1)GykDyAVApgsX;g2jJ|z_+;*3EP~HTbh6G3 z5$0ai;YRJ)C<(C=-s|vD?K!Ry7~104BQUhfajn3$0%r=`A#jtx5rJn2+$(U0z()o4 z3w%;wL*M~{3k5zUa75rU0tW;>FYw(0!)fF!U2KL zTx;SofltA{i5?VqV6BO(1U`vFxkRs)^i3wV%lc%tnCS~ed9?xu1U@NnnZT^oOdk}uTHq>yj|yBZ z@Ognvd1tX;o0(tA+bgh?_l&?&-T>y-R39m?t86B7=M}I4^hX^t&=ulyW-jWBb_w89 zO;_|$184_#CguB$t{;f& z`{MeZxGp!7F?1Px-XrILKc-9wGG5L_*O<{(ynxnSRGD7J9~~#Xq<`4g=dOe1fUN}Q4dMZcKO9xF1@hwFfa*rwmTek$IjsS7_U(RR!2lu6!a18ar1dQB6xkHMw;1n zl;UO5cQU*rla2m6^!-c*^p}#16qbG-0iE$%I4(!p6Vh-A}mj5-`qdyOD0El zkiN1_ObyDh+r;G9Dz3F*a&(p6WW(h6D!s#osUuM7h7A|maKwfKHhi}Ym)UTy4F_%b zVH>Wp;iEQOZNtZG_@oW@+3FEJ6PqSZb3Hf z$!_0T{ukgc_iWVCM~xI|AH08v8g3Sp1){u?ZWjy<_C!Vn>@=Qp#{NxCEZC;2=xJ6*hk;~4T67>xOe&K^i1`R_kCI_kuH3dz&h zpo3!<#vh6AK8U2xtaSOP{Ffc9O6p(uuU+3I-KMHceZik?`rL@}BPgGTS4e;Exc*LO zu)9-Ho;gkTuvZ-D&)lOCjJI5$=%2OrM16V<8W7|C@BIdIQ~PxI&!@4|OQCO%hqo8$ z8~x+cmi*Knu!nV zJbx;K9Wk=fz{&LI7ncH0b4T(8=Jf2W2*ewG59#@MH65>F;DPJ=h>?>;1)6mDch6-u z`i!iIMxSY;&&s07R6KvKjh>ek(de^n^f_5@2PJ>LjqcBaB`Nf|ar%!<{m1o7`k0r+ z;39E(=Eu{cJ&za#SrLtXxkjhn{85`lpfuWpgTa%$>R3O^}Uq0 z?<|W>`DbKtdll_L`O$qS`DbNu`&IJKvFIttpO?k$S>#9iXL0-G*wlyLN}tNhpJ(AT zjteYo+Go7JrhUfiYuabLekr#4nEd1NoBZSL;n4d(wAa}A5*EW1BoM-j*Q0-(HiZU& zQ4iTG^_Nq848v>Of3q2ykAm4>V0??`r}1ftNu|HYA2qJPm-AdCj(ajR|Bv=$$qte8 zd&HPFjrhy?*h5Bfa)hv$FGqW(;w}UaJbkZ`lOCsU;r^4V(W`t6L!8Wy`2y4}=L=N7 z={}EAzg{ET$AW5k8_fPj)VJ5TVH%AOl3%oEJ%a`*2z@Nl^zm)jmycA^E9gCin*M11 zpJ|obpFhnBqO5Pu6I?hmnXk#) zYZUlcvLz4b3$*g1J(u(QK~lb6W8pNKz({$VSPv8fvka_{VDD7_Dyw`K%@+&##(-7- zMLxeGPp?tvqt&wH*K3sc3N?CZI`2=d`UYSx@$}K2WmXf4*K3sfXgMO= zBigfClQ(MMMw(UsE!5r{@+^HjsJ)9A8JyMLMVMsA%a8Uf*7^0!vgvP)kEN^SuVv+$ zKGtdVrS|q)<-5?{mDBio+-ff@1T1-B{{|)t7N+*YD@IiQ#yk_zU(7F}#(aE<_ouza z5+C!Z@?F8u15xnMegx^GfLMtBhxwLx1DEoH@3MWV{A;ZAy!~b|G62D^*MPsF0cHAL1OAgK>reKl z`w!KR_HWJlQ+r%FjpmE-{8yz%pl}m!0ezLP%94-92lz`cULMJ}#^RSm_JsvtT)sz) zq*1h}HGXk_%98R?`|AGGYg|8##w#fg$*bzO*QiM6^BrzKR6p}M0b~*5U%8K#(=var z0slb6m|4du~EzG#J)jB%f`6rzarq)}#KkUYx4+y^@ET%I%a-SVw-O)o5t4z(w}=uHUEFen3Wts30!~Aa{Xa{OV@{wL?nO2 zdKLPZ6_EHv*c0iaR_Nnp(np@6j|a!kUxfY)vAu>OO?~M0!}X`> z<1$-+sLyX`{sRXT_O%r9=<_43{?dMEf~D9)Hi<;_$Nm7-ze?g4g*`N>_VB6)w<{<^ zVtdf)pMm<%a`NP~v(5WUwf^LPlht3JKiTI)y+-~tntw|FqW;>P+@e01?>=N?$J<}l zpC))p{b$Jfn)T1llUUZjL#_Yg7gc|~y{|X@Lu+4K{d?xIji$oP`KxU{mT103_YC6o z*XL&m>#vNzXzygdtgonlfyA=@hFbsoC#k=E{Xx=+z}FwN@fII1X+AABq(y!8@xK@A z56Tg*-_-bciuI9JKjPn$$!NGoz`aMK|4r8Z5@tc? z^7(!}tuH#N&-49v?6+V(6z|XZd_PC@-?X2fx)F>iQ2q>7jQuhzKtB6^+Z8Gu`N|<%lKOhY@Y?p5lJ))<`#)j>jO+gqBiG!|6Zb2k zn18ACXwOWoKjQm$KUVdT<&8ibSw5`~)%73kXHkERz{}(7yBTqLdyUI-_4S;-ABOQS znNbBre%!xM)?Z6u59!+ahvwgWzriXG>nj}Rkn-XFj~B}e3se81%bRbL7wf56s=V`a zse_aJqCInM`RCcn$9n2HwfqHMO#hYqm)r6$w3UzbP@Y=;;#^Wjy!=JB{6)6%F+ar( zC7Hh0DE4CdX5nbh6}J3K&{H)Vg1><#n3IjTI%a`}FNH6EAP_?6n^!F=%-syr*b zm@X;utkCkqU(@6JFXZo6#^vMokNKfrmFMbQZGG0WO5+#pxymLF)>|*A@?7U-P_Wb& z^_SE=VvP^`b)o!gZTzm+&jNr(*_iULvGSAr zu>T@iKFNP&t~KB1Pb`L0^RZr!c{Am41{IGR(#PH|Si5!0a80jnb561bwoj+#2 zXTLv*^ALRcE$yY(NbytsC8qNq*iJFef1tic`1y&^(fncae!vkUg{;NISl?9MbCQHb zz$ox4-7oWELxiyzbUp?0(S8Ww%xsC>9Q!Ps%CSeJKVqbN&)gBO-x16w2Q=(Fa31F^ou^$=?vY>_0K2R40F3%AoHC5-A2l24qDDK~&JamU~{7|RcO)F}tA39&+ z2@*R5)DQBaq2lSISdXgosKKq%N*^^A;d_*Z9Fcz>x_>qQY+Zir_i6Ppig;F;pY9iA zvZz@Im!}|=^dajLH7>X1=iXuRKzh1A<7d$-$?u4El}_wPd9Xj=CVPnMbDGBQ4>^f*~`Ov&rkiM)DPBI{-B2EYWK4+|H1xbJpB>F1KU#l zH5K+1$K>CbuA2Uh^ z_g~%LqY3;^_h%Z;RDO2KkELSgAPySx5y@s zt^YLm=xLe=_wN#$f9K_?`lb6Zdj0hNjP57X_NKIN__v4p1L@zusjC0Oe+$x!Jml}v zzZYaJ(Qsi-vB20*C4bKg#Od(&DdYM()|Ynw`!D%>rt0rYJmtJTv|o84{*L?W@%yis zPfgYC=k^+2KdnEdzTlt!H43KlprpSkmj6xVc&fnWcoz5nBSsqRJ&vhAV6s2J{#4~X z-$n!l^arogd@tR$|A6DH%6j;ES!>5{oU@LrM$KIAMW4FCh>3GpK+e2K=*&WzoYdSZRbn-g@1c( z{+?zV-{Ie0)!*UY8ZMd@5ZLthyaFtNW&Z$w&;E=1JMCv%boPt(R*I!~qYqw!qDllyxXZpWzhZuj@n1v-71-CqiI zy4_zUn}07~p!@R*8~?mQov!;+G{JbQ`}@iTx<6lO<6lsy({+DLG(XVi>kAg>{%r3L z3o-jv^{e-1*iZL$=>DE=^Y;w%ewjJH$9P>jYe~B0zrf4$u+EkK47{=+Ah7B03kvZZ zhNQ#aFZ+x8yI=M9n?0+@-7nbR@zAbvK7-%CA^nN{s9qz>?+3Ryp61`E;75$hxIbNH z`TGoxXITDD_8#|doIjmieD-44VNePg8hBA z&ENBK&uHU&(Ioz^&u?)a(>8uDh4$3(y%gK2s=eC%y#Nla((V4D&nIZTtdH+2al={7 zZ}-OqMLJ#gr>Mbq=O8lzAAeWa*5hTfbbnrD^Y_9cov!<1qW(np_gS-af7biEBSsNc z*J}Or`33ereRR84+AqfUEStYiqw!46@5$c_inA>LCI4QqM8jo;%LNwm`JzCa4u7Bj z7x(u<)!(~4E4cg@HolX;VdiYEukk!a55)P?dQ@M}V|bT=`12c{Qz74FyVY4YUuq%qVv=G_NQ1ErXe$nl!S_m$|e z6kK-I5`j&BFIZ3UME<^AqhIoO*?pMfwSE76wK$wn`$nzG2F`qvZ#~+5jPd2{O9h`~${br4R$=}I3c=*4>-yi;{`FoY> z@4xU|&KvVXoX=b5_k_P&&-Vy_561nQ`@3~s@V{q#ryZq<{JmDAU-EZu9`xt`5`RDX zQSwVm>we8o@{rC8O8TRjN`*HUD^$&f&4Uf;q z_w%qHi0#KXjv9aN{@Zfx{`wJP<#cs_F>2`d=PrIfu1veXe#E$Hy1HK(HU8ZFIsN`R z&O1%l_dh4!uRHeF0>*^Q_oU4-}PY*u7+>cK$W5xHL}3e)@-g{&kH1c~t)kT(tk;_Ly@2>~UWExc?<9 z&uh?l>sFrkAb&IcYo6tADSUjLYvE*$lPzrer(577hW&hMN{T+-CMWB-$gPjB%WdaV zS0<_Ui2GT#^QndQ^QkyrAGa6#c^C6MzDiHA`QKQ3)cr5dcE8&`{!M-!MZdpQV7tF< zzn^6vpT^#g((i9Au-(tLkDvPes6_Y6^!wZP``7mVMDHKyd81l=yc}~rm7bSV{Be0+ zY5cq?-Onyi&YuF8Ezr-W78WVzSK*JDANKw1G5(mT`eUtUCYS%h$3v$1Q+#l!7)BRz-X76MD)vx*EGz^|VV{mc8Cc0TlB4<<|qAELim&;JR3vz{j;f6KF-&-B~QXHMpC zldt#m@h{WnZ*UhG`237^-_-M1llj{`zwS@^^H1=%Or37`x3T9tbbni5^C!DMdom~X zx3TBJbbl+d`IFtBy_vedrD^^!`T5L={ppC2$zQ-MpSPm-yQuz5&tG)t{$@UZ8TYpg zo4?IV#&b-{d5r}gywF0yMP7M+(>%Z7&(zOne(29%j2Ykj>iBlbgEs;pe2D&LJ?|>~ z&3b;8{4LXVJ~G>OK5{aD%e2`u4weUb1^D`8?D&>$^EdPK4bQ>-jaIuC<8QOFb$_y- z|HJz(wE3vr-^|`RUjBUBdd#1#`;*@PV|+{3>Gtu>T>ZrPFSM=K3bJ*7(&w9a{srfc z)%mjB-^TVA`uLWet&d;RHGiV#OLV&ad>8qf{d^=E9Kjmj`1|oB#&|X>8Lt>s{LAmb zD^(R-;9a1dkIYUljMFin{ImQ`dpA5X)(8L2l+RP1kH5bf&sW)>-_oAXM!)rJr`enF=c)0&umZdf-w*lt z`>gSPt}KJGTEYW(zctFAX*l_-uNl2>AJ0D%o%Wli41*5;dL8e7HaX_;_vPEm_w_I* zCY1F49{&Dfou1EDSnqq`&ojA+|Mjx`U(k6`x_d(*!r$jJ^B_&W5$?uk9=#uY3d0R$ z@jhh!er{Kgu#@EH^~3W!ke4vzrSx!MGd=3J0zaj63hHF^K5x)-j496TSkTZr45H$rrW2k%=q$5W0$PfErE2M8vf+c|wyl*h-bxt%$q@(pxU-rUY?df~1ac=;^F zeE&7&UkbV#`{i-^+|HyN@*f_|{MqDx5_>v*cp(-KB%jmMIVF|bu2~;;U8%XBGfI`< z`7aw4_2c-uiX?6iak>wtD+1DAH_wzePM@+zq>m2K{&%fn&SY4qncumrm$L(-?Ts4zYz8Y{iovvk0K!d+|CUC`vp9P<0Sei zM>BBjo}#vwr!zUtoR6WrALm1V=Ia)?{@_p7mFoKTbf$RC^T8w^wSRJkxu1d}D1C+A zexyIl<*fS8?Mz7*{fF6}b;-QHmgSMYlGuQm9^=z>TT@0UJ*WQ|?E!tb*?E(W_XFIu zi}d4naQzZ)CVf|%82Qcj?^l2Wyb|?C``~#?=#SpdM&lcvt~Z0|-)z5Mpp8d(Ur0%J zQvCNJ@cSg%??HgSHeQ^f{=+rOUx-fc8$o|XSTwE!@ zS)DEy^$&b}dv+=PUJS<_TOmKnb7DN8{s!;YXAI-vOwbAP{Lf=NN~P056wLZ@dC1_z z?@JJWqMH2i{yLGL^M`&aQ-efCF!N(PCUFp8btwNXJa5l~;{h(u%!7hVk?7B}I)yzg zGyCW7cP7#6ju6uE{>TjW<1Vwm=20zpKT4;tpJf;;{#5yU7!fZUOySu4Jq%#* ztEBOQ2ebZ8!^t8|oIn4)pfv~kMEL`Jystbc>l05;zct6(2kl``uh|RqslQL~LH+)j zygych@rj;aMg1%HzopEl@cSqSkFsOt?}Ko8wEJ%|KW&gw{+I3fN0I**ljo-mpK;4~ zvqvW@pBAv==3hQ}ewvVvo4;ZF{IH*BkMjO|Qy{Z~(T0inOgeF zgYPl=Z9@FMm5UA7=2L$K*sR>8zn3&{NduQOa7hD~G;m1+mo#um1OHtbpuJw~hYkDz zo05ZfI6e;e@;lf4zc+vRgX@Qn7h&h=oj2b9;Jv?n?}I1q_}fwT%A9x7hwnIV{tNZI zHTvM_=o@dezfR#huf6^{JJA08=ofGQgZtapjAlIk&Tk!8TtE7Q?>)9J_>-SK@%6|4 zIP28Vn{RyWkKarF_IsnQ?-6-)H1hiQZw>xr^!lT>zcKsN&}hcsZ*Nag$b;AXtSh*9 z^o~_;d^Yl#(Ukx1_BSG*xg+$kU#~d&``_*S_5LYMBk%lj|HoXXPyF+%Q~zxU)8IR9 z9sTb4KaGCtbH982_21t9#`oVb|Ij@rN1EF^%xB}}d{VApmjl~yjHP8F&P#D3qtqQPQ~FY8rDR(w^p! zLroLmo0Vo7;`MWfBX=dF*aqH*91+LB=#36%?#GkgW#uH8%gZ2x8nD!xh+Z=ZotAbJ z<*%KDPD{Co@;6OFr={CO`8y_|(^74s{Kh2o!b#|nN$7z|=yy*-FPntkI|)5F3H{+o z=v9-@k4{3bo`imE68gzW=zWvW2RQu;rSG!fmuRQ`hoccE+;xDyQy}_{4o9jJJwx1a z$g>@lEuyzDQ4jIxNs%OYkP+WSsOA28{3BcZ(VfS34} zuVL53dmM*@_URq#eJC?%CdOY!C4;YjL&3UCki3%mta4c$4E$mrSR|JMCoxQ#vh8^4Pd&f;eYN zf^;Ho3{Q`(E;j5;)#-`kw9BYsl264Xn~J$UF0aT|#*{Q|ED(0!%vJ3f1Co~o&3i;h zK=wX=)`E9&5AjAy?+W5aR9sPq6R*J)L8H5aZia%>RJzXJLHwm}cP;?Xxq~tG8 zEAVP%WCgQWG;5+Vyi^7qIiQefFC#ga%t(1k8H$dw66#3RK}s-%MN^_#YJRPb5h{N` zmOsN@z9;Cx5i`Afm49k5l|@rgLwPR?y@19auBhnHZ|Cm~V%YXZ9cq4+e_Ak&MbjqW zA4wtk<$dBpJAYr$$D+RR`A-Q>VbLiQ@h2M_kTw>u^G^@rhqKbh=bsV8jh~E(_$L#8 zd0(85d20PLgE-QfIX?fX!Ko}dbprmn{)3ReJolV#qkq1RTvP|KTg!h!QuBd7Uq@OS z@NxCe*3pX?7t6m)J_rBSY_x{jI6k*cPUvI0jk{5QbS5I$ZAA7%QE5f+5mkPn|74Y( zJHC9i7cC#|FDvacem`c_K$lPSVaZN~K7o8qntZOL$@QnpH+~IeAZUy^eXE>7p(*J zzCJ>JG$6;zlUhCa+Ri1{h>(~E#m`)@6N?T`CVhS$AJ6%@G-uGsXp&`;#?r-D8LtJ^ zQ^_yJ`*HJ&@qFC;Vk{juzZfgW%`e8var29@aNPW2EF3q#=tsuQFUGoY^NW6Z-27rJ z8#lku@3{Hpd}+M)6Z#!DznqJVmtW|2-28H`Fo*2ih;6t4uJ~@IGb&Izu1C*M0R3=+ERV5kDhcl$^{zV!c-G^SR}EOkS0~31gVk z+TW53PD-bw!8{?JMvep(^~+1?T;^HdY&0T2O7Mwh8G@Z!}5NHX%Kl>`bBja9xG2Y9r{nSZ&b!nXUyluIsB%(*B z@Y+(_C(Iwk&X=$qg|2F2;G(rbbFaUYHq5pozji0k1{SeazmSOTOy{*_7^D*CPmYo{ zxVGZ?HEnn<);4H0o=_WdmyCL0`CVQU#f|MZd{G}si(b^3uos;YokDGkUNkX1J(^DW zsb5K$Kb32fdQpY`=V>EwNgK*eP}wDIDB1{K(uSgqs!Q5Xv{5~tHq@R{{Vsb_`}MD; z7oBX@Kklp^HBK~>*JtW3++}w9WACJ${T9Fnf21If*j6j<4*ZUZ+#cITF*;oN}#_knT)%(#kP1 z!R-B_wJ`4N-Co=A@K`P2W)amAn=#6)TG2wPsRdg9CY-f}}7`IRSA$1iP=wK5iezABTPaq<(9%qka$eRO0vj%rx>&7Vh@@ z*0|&6jDE=ITv3IuBDs83wwFHiy`s#`KXj6ZOAw>YF$XW->qDc#c9C zI$RXLe`C^T=ya$}gn&u+>+}JOUYnS|%#It_ynGw~pp}0hA%AAk&ztnZ@yho*IRAi7 zk68Ju9qMTc2igNzKYtN#**j4f^uL(y7|=-R7$!cB#U7l_5E$&r>plGZ2xHr8X@_*Y z^CM&Es4ImT_}FEbXwRw!Ryz^Bb`m`5uJMeiRjcPPDH1*{Y3P@B=oXL=)prCuMrvP zI|V8)5{(qvi}OTLd=61Ety>yLD30O5e+2n9q7K1NmR~I}T`1j$UEUeNKkzm4OHg3C zxR-huJ*Gd8zEAZqIU3P-*?ZG>`6$0TOZmRX&8VX8E8Xlay6*O~dp|etOW*ihb}RdK z(VOqoxW2q_$=txd&UyO54Bz?|?^2|}PkvG;Wq60j`KcBzVR+_V!2x<<%K5n#eJMMm z;bQiJhOc0!G`xg8qv6GL+(O}3#QHS6h@I4MAv>nwh3u$?7qEvld^zjYZ~?np!}Do` zR?45p@UWhO=dunB`&pBQ^I5Hi=dc`@C=rz zVZ6vg!#V7{Q_Vk}ez-;9mrXyEs^Dq#!{|!;X0bkvK9wEQa3(vdrN`UFG`7y$7(g4#;P^!WmUj&`@@^HfaCV&VPzUlp*fS1KAHJ}Ut$mL!aNorC&x522&X=YPvoTsWgfc)W8j0()bfA*GR znk5iEFX%L?5{9*Kdab~)E{~MO#*iaOrt8%dj;+g_@uxzsuKN_z=pu*1*TDz zmk))Lyb*!vMbd<6RORIhe7C?I0@J9<%NMv;;FAIe1wJM4!vdcdxJuwmbe2@!QGo*j zR|{M%@G*fq1U@NnufTl*pA>jN;8S>Y9r3RL_B!0GI^V>#0uT62+$3-S9Xh4&5cueP z6B{!B3KLWL#BV@gCh1q3bdSItus`CLDR6J4iDw9Wa+8Vu0-wS|2b8{0;Pbmo91u9O z*2HB32lkjaC~$R)iK_(e=rnP)z$Xn8OL@bBVA2hNBLbh~7-zx1D9V#` zfn|NzmrS~>Pefo@ANFOFF6$E!xKNh=6_Xy2`~@zP^sk!qpri|2CF#dZdbOkr%s9qt z0%9gz${P__%F8g-9LYl|Nc_&JnEf!+cX zKM~E?M)aMD;(MrHMX;xr^EryQ-j{`R*H8rzt`+jg`Pp-3v=uMsiEC`}0hsaN9&;) z@mDDEenIXYhmW57;t@43R4^t4MNGZ0N0~sFgM{366Xkze(dM z>E09hdxgTOjj4aJ=f4{fU@@lr`}0hOe)D44n9MKR@j)=AxS&WUFDFHQiDmgRpKSjF z$UISg4COpfeoAN0PyGm~0PdxQ2p0KVY9xK{6%kaZ2)ut* zFhx^;MKTe8Er|TJTBOa$K>HH-k^B|5bh3Oy@xMgvM{`BIH#tq^+mQe76V$&+!yq|>rr z;!lDg=MVBK^OGF^)qF5nCjHZ1r1V#LRm&q;rC-?dNn9%oM9s&eWJAh?;wt@1US&R+ zY;XCt9hH0Q%gwNQQ$=aZ{`zGVExVR&yXltlmhIu44f}W1Y%Qs8-Fef#6*snr%U5q& zy`!S3zI9#q?wYMF^&Q)`x2(UlqG83>_VV?&ZmB6zX#2W5>RPuU-`0)gyLR4mQ^&T- zmUTDWv}gB??YERS?b%(qqjgu!j;@9cJa1)9>5lIDs;w(pHf%xJ-FtTJT(Q4y+193p z%9aCl9osgoE!|wwP-!&PFWa<#*Rmaf-P=|K8fz+0MkCVizG?T)6}QzbyMf9p+10vt z?Jeb-m(`bsZwi;UTIGhzOWN0z?!7Twv00>9x4flt?}nz1t=n3vN^iQMa_{DF`KtD{ zm3y~0?A*{^vHsRIja8dlCrMM$xO;2+>Rn;faqFs<)lKbdcJ8eXS5&mGUACh+T)u5n z#j>V74W+9N?ylN0i9EXl;hiNd^=*@->)7A0vt&<0<*H7qcS8Br-&#X@s;gXgTm8Wm z&AVzgx73twy{&Gws<#~*>e_C+Z4#8WQuY+gGJ6G%pC*&D#uhl!ZcGR`3 z3hdgsp=P_VCWF`*!bKIZ6AP<&(c8s=H+`X8DuZ-3F?6&9dDGc9pJcPG|#_ zdk+e|uj6S;8z+&kf$Il)AFutTUzlZ2+~%_Es`}Pdx9;At>X!P+>fN^CKz-$oN!zDl znVD`^+qOySv-@Bj`m>s4m6N1v5$P%p)Ri8bByERCyZM$~JB^lc^ls@%)E#;)nM|i^ z>#N`!7O+f@U<3WoN}d~ts`FHo%<8lWlfXl|90P6-?FN+0sVBse$t+9ynabf z*Hu?KPXA@4YocPsh;Z$0<5%5`nKYsRzH4XTf{)Q>lIufe=2up2hNF0kn~G5EFTYmK$3m?w&7GlrD+6`Qm({Nq3SOXIhwUjnc5?WcZe0gYjVAaY{X{ZF$&i#$%pLOAuuEtPF$h`7+ z^J^ev{woQUg+zsKpSQ2Ev#qfuQ5^)utjeZvN7t>*x82s#XyI)p?kXuQs|zg)HFO{B zXsqXo5DKXaLym^777&`-_TJRo)?F6rZViX_w0E|KyDdIz!rkF@JgQ>X_O+-@UAqx# zXl^COA^KP;Z4!h6^RDvW{)tbZE+xe!yhd`76C ze~jAW3FFMkm}s+}W8+CA^~~4{zcRBvfqFTV-5+)R=He?p63#zdOw$BS1sb3{%vJ*~|3a35fnq90kWG7&HR$kZ2OzeK%coXY8D#1o&Oh?M?= zACF3yz8Q*6#OJbc;;$Vie%mkNdKfCV@Xu*Z?G{!>1m&fg7nYp#!26R zG>Q07T%vfoS55qT1yTAh_EdO$HEr4;$h@WaPU&k!{2ESWY=_`;QpC%4IW6le;-#Pd zLewi$?9)pBJSX@B#jm5t`1=q)Rqk}7%zK2%^nyRd_X>PY$nylQ6d&=L5l@Nu*F=1k z=;z!bUe<3K_(@wMQnsK^x5WoU{4B)FycBh_h|fj5s4|b~QQ}3FdCV~pPvr_xdCa3C zp2}6@w?hvUh`*GLFxfc;JD(;IKgY(WL*b*|4eC|mv9D@{V@kZ5{_%0*J!oGFB&S+m z{3IR^b~(#LyuH3>L<8CDJ1FAqeBKiA_WHi3@KNQ@6MjYf)%unx@#@%IJx;t^7!C0; zf2YQ*=QT>ac;7oGJgUU2>0egj)%4d0`?mA%R^rw4r<8b=zegAo@ln(JL_FzJ?Y#>` z{9;@DMq%`J{nv=p6EU-icU zC0^W8<9yyy;^RNWggJ=-#7C9q8lfzEysV2G>B$vj*)&d3knur|7;6$&8GlCj&%?HO z+26lliCcGzci7U){`iYCfaod+4_ z+2i4x>#`?{#+Exw*9T9u+5EKZ1RPZSX4B(?PH&U#b9vQXRhVYuk1I_h~ zTD+;*#-`Ao&Twnv1&Fu@<#j{GP#dM+xN*m(HKuAewKwcvJ$i+PSDis3*)M*9pDr_L3K z7^FW4P3q(VSdNRFh(8PQp($AqA2rk+fiSdj@8|(SG9V!kuclXwqyY)^m^G5j z$dDBwa)~h*OideSHO?iCo6E?MI=5*Z^=38ljTdnoOtG4q>y>7$5SdU3FvSyJ2NL7C z-@hL-=Nt_Pup1l7hM9Br|NndM|F!r3?>(b4UtGFknXYL}6^*^ZVl@S2EVCze@8P!| zmI_KY>#pv%)&4BoZTmm;LfdDjF?M*+mfiafUgvDtvww~2dA$A&_h>p^(bDz$&hLkQ zJBD5RV7(rCztR@EZexf4pN;h%ZD)r;d!##q6@E9{)AvCMd+V1vJ3KA5yZWqcV(*}B zJoR;?JDnB&U)|F;gm$`Sd#e^>51)EwpyK?2f$f){8CY@pnSpQUo@=0e7O%szPJS!g zd*XQbap$+g#}Awc56?Xr?wxxoynof{@TL8y!^8V8ho7GNZupUw^Wo#C-VLWre>60_ z{lLJ}&pb1bI(JcMR`a*R7v>xf@4Nhy@YBvW!%ZhHg?m?A23=t&bnxK7wi5>i_RT#W zt~h-n+|+tH+aVlKlJRJ_ry%c_W74f_{6gqKW zVD`zq10nEyVeYrW`{uq0zMu!l-iufAtj~qYq00-8fChR?N&Hqg^zNJClr1O2v(8@x zUl+q;rau-MCj2t+x*t63J&$&Of_D1{$^qM#crrZvFko8t)gLE*xqA5GOM&Fw+22b} z?5<9JH9JUl^w?OT12)q1?&>=D)CA3Q?W}D=FX1-ptT1YS;Dr4rz`ukKY}Q!e>+k_h zBiw}ELEZ5JkUkog>mfgCFo@MrKAwky5q{_V4iEUFn+Se`e@zwsPYFPg8c?DYt+~W zo=8`XiMXQuN0lS>W$cgLDt)deMB@|vG|kh;$4H4k#`UzRS3j!MxR1FqJ=ap^ctWWQ zi$c!N6^F<+WD~Lz_4f|+dJg(?A$|suJ$>8J|2J@5jaLQkyKy~$>j}L6H||g4`tLM8 z?Dfub2_YNos{GG3mR)-X`gcv&LoV3oBK(%@^efnaY<>YYKf8eS4Z?n82j$c6tzmuP z64uA`{>R9s%Klfr30uLJ--RDvhRyyR_KW!gJACC3>}s%M<5Zy!^3#thi$d3vim#IW z?(4&>kLTe$@MHd>x7)2(y<78-8gsEL_2>jeXPQO>C8pL zjwsuGfmUC0JF5aFm#{i!o}P-Ut})wKo2J_svyZ`&CyUL=?701+$ZLxfLTCRZWRhf? z>%`63KNVMfJ4^Ji#Vi3FO(M#9?TzgnfevpS?$oubMb!DVw!N{nA;{*nwsrXCwLUSg zV|!EUmbO^EzEjqpm>0dNZ)@>;JJ`GqU$A4I5UsYpv!mJD-d^i%Zr%HwA zwG!LmYi(}}_&Wkk3ktTj_*%TpONf4Ky7|uLG8kVg|+#4d3g&8YVogN zVcw(pix$_`=0Q#&*F7Lcd!sktYq)m=3HaK5fhQmltQ7Dy-xJ;jIlT9TAPv5{&aJgu z0^Sy%u#Oeroo9kzn$^?N5b*jN`K0p)+B(`gy8LKPxVF~z4sUBm-aX*H^H(wvRG}&fpY2O1~dZh0UIiD%s$JZ_skSlw&eod$iOnF7}sLeu3h92G84J^<$fA+ zKMlFt(!w~de?JZRjAiepWs9t11>AZXa`*3J)FQnG8?1S4Teh_OI`UX;ZR-=YQOwGv zD{BR&HV<31CT}~B-cdZYME-J5M0J#Nui~@2zZ7o)=zhiAgBW?({SsN^dRG9C*ba*X;HWYv#&{dHMFAt4&XMNUsy{7 zC07FJo(lNhMc1^{-QC?YnIp2hd!~cmGDh$@)=p$y@ofJZ0HUq^3>|_{6|W*zACa0x5!Y-(Gk73QhOVSDyR6&E(pQS2bsLRtsO!va zs?%AO5l?+m-G**$lOs%Dfn%mxn`WPt%56$^rDC6n@nIBKGHjKB+Y0bDC6)Qny$g<5 zg_qq08W;@!`O7|4c$qD0EwiUM7Z0GgC zY+IAhdyj1j=$=C&VMfCGth&dG-)sJgZ_w~>o222BYsfVNj`s7*hc&& z1t-9Y+`b74FX+RJ@UDP!0`Z%AHj=~!(pcf2qkq)7yl?RSoN1`}>(C$ie`+6#kUi`U z*mJOwJ-Cx-sX)FB(0^`l;zaVd zv0VS6&d$7~EB%yZeNCDd_C)`%hm$)w*#rC(2H5xse0)*w*_jYw?*l+S0RPJf*dpYS zkp%+3v`b@GhyhLb$MS%lMpdSL#QAXrgs9tp24Rk&8f_xUH;~QBzIp^$Dj5F)(LX%>Y9yk?=w~@e ze-SC)(k?6$G_j>WoA`l$6$T!DnA=Me;}r&G%;NG;`_pNdpK~JYEZWn2%A1P%>vfm4>A^X(XuLVZAMCTc%Q1(~w+E-DvcGn+>A~4E2tS+S=d-=8oyECU8oCuf-dCQEb-Zr|IHGw0Z;t<1-@=R{OjR|Nj~oXSS;D z2j+7wpRY8Zf1bvQzp4px@?YS~zWP(yT+~PQXQqe0=8`?RzhXS{ugydv=R-zt`gDqa z+#i2N{VzB5-?MXS%%27iVtq@nk-zap;q-j?7uNfzf06%8O`!3na{oC*_M95^uR*6) z7?_d}^;fJ1(}Po{5Pf3I-=<_zo%`D!w0EJs^0!r_-zhGWzd3++3ws`$%JB}=MSJo` zJMhxq;NQ^i^r(K7y@>&`_uHgzx<`@vrt$2`KL<;opALbS^LeEHC0e$J{-=|=m4A@_ z`6p6@$9QF5U2N*FXJyQ|&+H<9p!K1S)=SY|#kc7}2d8(${LR7jLH>45 z(?{|ryp^xVn19P*KT*W|v&KWR?-?SH>vv&r>{zltA0P3eEIF6gR6OX}iKSn~FF(nr z$K=a#B3_EmN+00-rs;(gfBZI>jMi_OKRS<>I^pr9XCvj8w6XO`<_E~%Fn{I}KdPX9d_v|?F962zVMCw-$WX9t4;CyX)z)q3d5Z`s3?91aj@*ns!J~}3U z?oOihwJcc)Ana-OKhnR=7@zIOcOc(T^3_qk<8c1+`Um4hzC-p)iLi4Rkozy$=jQ$i z|K|CkL*|zx4=*bJ2-qBaeg2;+e@eE!jpa;@hxN>ft%As>`FfUYqiDnZ1M(7{uXh=K zM)DT+0M7>y{*%7Tmdx{G2hWe~xj{$r%zut#JISBW9`f_?_sy)&`I$S~ALLIDOw}iWKH|cB9;!@D4`l0;sXh@C2;*bCoGjG& zeiQw{zM14Mewq)Ew=kHQ8I5OFeu)1bjCW&xX#GM!Cj#N$W`93K^T{|Lu>P3*9eC4x z0N&*9z#HZR^phJj`#=1BD%qFkJK#4XXqXR>m+(>lzobz<1^+^!`1m5egoyl$xlt$m zo|}(4(go^}k<)H{vtKnepRLkKrLN_gAhD$V+t>I1u`=VLs=Ml8^JrCF0A4N^?AW z5&7dTmP-B&e;UKys3d<%r565Dzg@(W&!a}guwD$I%HzRQvR~@+sET+_`cA}l#2C*L z6RCd{&l7@THnD1qg8X537efOSIv>IQ>rfY#T2XY`pJRV44l!we_y+fP(oV!*EZ4K- zg_2zGmmbhl>BJ<`P4+*eEsN$00{@dBcLv7Chkx~eAMN-NXJNgL(xW}rqu73HP&2RBn6HG7 zuGgdHD|~)i@)g>QQxX2i_Y=zguqVzRGbaKYabdon{si+C@`t$j%Cu4P75B88=POw} zUvbCJS0GQ8o8o&kUkMuWjgk2Z;;B0;ny(B&e;+>|$)fd@$4m5Q$wv?`P5Foe<4tpe z-;a@xWQhfz;`507=YvM_%sLQLxn**h3f>K?_&KE z=S4@?vv~0g@gn^{c>isRU&yEQ0RQwM`uPLlqwx&;K68_~a=!k8hK>zfX=I-w;10P3HSAwI3fD-%RH}xxvYkEb%RN{$t(WV*cL3etVKJ zz7+9C0xLq1DOg4|E%cd!V>L;i;K2cH-7^ImbDVK@&z3Zh#$4?l+Ua8Pz9zK3I##x%?R zI!(uZpM3=c+@G=kyeWV9JRJTY&eyZ(^HctLq)s20CsG~!uC(m8`F<@bpOcY%{Ctsw z6zi*<9E7(YN~iwm;}+SU7CZmXyES>u=Lc!*^WaD9XVC%9f6BT(jO8cz{J@l-d+!ci%!6aKn4r>D0!g`N4)E5H9; z+ zlE>l?oQm6$Y_eyBEg>@Fb~ET*PZD~M%1tB1!AIqG#m7hGb_==V^%0c|jF7BKG7O1g zkj(yxNM;}DQDg}dXObq$8Kk=zXCvpJnRAV&5{SEe>56&W7w0BqVcu(4Xxb6kkcp53Hl4S^XeJ>Fuw0?OfJuWr=~>eTBr)Tt}d zHI161>al%>O{Y>6~eBa{IWE2I(9@rW!o zPOfM2UQ3=w{TQt;Mlg)btRuTYV!+>z_0R`=%(O;-8f)YyG3KWiHREyr>2YJV=V*1~ z47(r0#Tqi2&OR>QsPp?V+`Nj5SfS#>$hbp$%=jm1mepOiuKpA+3_CeZyNWDE?c`*{ zRpXj0^aqJcG;%C9*%EHf{hoyT`_MnkT#dK7-|qas^Jwr%jBe#NUY~c)p|$rvblo&l z6GxsFTKS1bZ`lu-@}Uv!qkB$12NR+z2rWM-R{FcGZqiSjmCq_qJbJacqZJqQclG{~ z^KU-O`>dVqeb)7Bnoq_Zn~3$XsJCatj?DuPipA_b%oz35tbI$@SCwL~&G^~3b`{=7 zhvC?5t1{{c{Ie2|_za7VWG(0D!pBYce)0V{vHg$rKk{UpOJL|<*LeTkiZ}Y+nX!L~ zS7`ocZmV^z+=cT6nopvfvE#rd&NGa(n?A#6p=xfqb}ttId}ITG&f>&FKFFx5G)j3Z z&;N=0Y@bbT_Vf!mFMj8R@h$ATvw!p34Yt3TF=txdkEj0VP+DqvbIN=S`5AcrPZgwg z(NFkF3{6KTsBc*vcU4v5G*KL~0j+m5mSRU~i0onQliF3u3ExN+L$T75U&znI#gZy% zzbD%tH?*Z&tKP|fsJ+L+MuRlD8#pTDdBnT5eK{fV^-OVzy?;rz&y^shV{3cb&+U!V z{+Ok`>eOTTQ}%xXjc?Xo(NTM%v$m&sPAagtDP!cADBU^YO6`fJTU4X(_>%$NXGs(y zZx+id4Jr%&Lsvtcz~6z_5l_^=smS&o6VT#hz|s0b{1;p3@Cbiiero7Ap8foAO#FQv z*-v`BXm|FH#dYC>;+lmE$!{V;{l5T6HVRsc_Iu)t|0UTzS5ngW zR4nRaKH1)58UL7Uul$$VlS^CMFT~`1I?niC+XP}1H-Bf$I==Fw3bgM*dyi%Oe%W66 zpBi7aQ}D?E#B2X{oc1&&tnKN?q7ZfbFtbVjS7dw1e)MA1{~EOaZ*k;T?XSp|a8qu7 z>-gjoq|?oHlF)L=lmn`(YER%hZAWXVnxB^8qVrYFTjO}dtC|s{Z_xp6~vetpaOn(c+F z+q^|3D@)cDH`cc->)5oRy1CxJW^Hr%j%D=)jexDGA$)1`lGPhGR5aGDUzfLOL&Xa3 z`oita6_qt>c9bn^UiNrRqrZAh^YVh4$IEw=Zz!m4-qf-zzizo`Tim>)xo$&oo^KQH zcg2Q+Wu0{`OY*>jzph|W&9>rI+Zr}hw{P6A>d6%?tMVJlYC0R1H}Ak0t>B|+M z`hq20G+x8{inikN9i?>(*5x&qZLNroTefwTw`fV*(z2~RMMnxsq(Kd|i{bXwAyv)>V1+E$bTV zHdVKkZ1mRD_^X#Rmo&DOuHRbaEiPHPtZY^Orn04-8(W$?HF&fnO&rlxU|u-CTYiY@3nuVF()r`0ar691DMn-)H%{7qeJ zTbi2!wc8iv)fMJFx};!9p0}YOZ}H;7f<@l?hWx_(`og*;-UVA~1zmm97H_@pMnAE# zmjAX2{i+In?c`1>c{h1BdH#Zh3l}cP%UjH8mU)}oeYN7fqy|4I0mL|Z$1PaR2^(DK1?CYX>zDLaND7BunI0|9Ihml`|4dF0^$6(6V) z?|IK_dDmzZr^I{Cw4mn^aYPH7VO)h2^f7+SaJdswKwJgAapx_NZMb9@c5X^83X=p^ z{x9^!*f$`Tf{7l)JAJ8~=sgms{5}kY#-m?(1NiUJJ{~>oZ{oopy#f6DfD=DwxE&Z% z^Gdq`|8GeCtE35DlXB3Q@y0!XM2-4)OB1O1)dYHiUy*Rtwm`zwCbd-dpQFZ=CR2L; z2gyI4F2#$QSFcI>mnB@yr@fM&erZ4jUl00pH7I&ojXD)P#h1f3%lbbHF#9I1M4yNL zsGJlR_7<)LzbfH&30LxsLx0M52sK{fCs^Ql5R7*cfU(}EBl&IU#KRxF}mFEp`BmMCk zz^@wMMtO6k-mQGjlyIxO{n9{Id0&@sYd=4gaI3t(F!W>O|3d@ZC~u~mKQyi}M&{iB zeni5FALDBFyo5huf%hBejrM&_!ma$%Q*@{hKa<2Qe_tr!YFy=ujS{ZLRq!JQdgHjy zOSpC1Ur4xh+*CPlX$34I-l!GaG-JF`S z&l%uGxf&&0=|RyqOSo0u7bV;(Z>bD9YTP_I?g|6kIBrnF)wqg&mxNo#{jP*t$ECc5 z3gs>|rIhX{u2E5N53e!SD6a~x;>Pn9xQY+`7PyK7Ll(GNde!-->c2|bX(sriqTt;U zzT5&=>%HFsSL;0!X?J^VT~n(!(8aJvXz;G^cDB|xKF)t~aw#6jj9+);zwtyr*T`;l z=RqCSHnle4XCCX?+i$N~YiEmKsL9vPM&IPftpopg$+K#r`2F{oe z2?#)6kNLg_K}4on zrk<|K1a-2^KNM0f)2hm7pHDEE7DVA5d3Q#}F@;d^O4VoJhEgt%0?4a>hq!rO=415P z8pS&qj-xr)KLAMM8*^)F>5{?MLlr|&q^}mwi->ChE7KGwk^_LFetu`Utih!apU#+y;Wcv~F%l6M&8d&-F0UytP4(UHv lvuyt%|a<-k`@x$X%eqw{3dOX_p zixbgDCVvqWU3->UU>HGF_i7i`@S1pd+0>;=_gM|*M4|LM=&>^=qH9i{j*fiq?&c_K3#eu`m+mXqLjJ7(6wrTA=6MX}1@@5Hi zS8s~EapBO}$cf|8Na>mA!O{<-Nw)l}m(D!f4V^&J6Z!u+DKXrg^yB;x^^?cWs-5Tq zn-=aKfY01(^L%#Be(zDjc5AF!?_Xe|fA59=h(6wJW7V&tk8L)>?mc>5bGDJ*5yq;2 z3|=AE5w?B=Klrq(A@Zxfdia&+vupcad&TUZ3;ElpcH0`|bL3yjw@$&g{sO*t*SCJR zeJgwu-#T&&-#TLPE%G_#U#BeorTp<`K6Y*2k0LSq^0zPA@14K&u5G@+!P3{jKmW#V zkF??b!FSmFhX>d^{CGZbl@>`tJX}7&PG0^An}0V&i=4OH=93n)&ypxslGwEZ+o^}% zfZqEv2`AtG2|F4489VvG`h>{e-eZfLOJ_g4w2z&<^e&sfwAK;1)WGHuBM)61Wb+sA zPmGX_i;Nw@LcT;kN4~_za>`c=Kg8=>DC$f8^+i`*EQ7t+0R<$D*=#g{x+G1!Ex4S>g_dLn4IEauc}xmL|Mz z)Kk>!4yeY=h%ewDx^pTL@Xz=Iv#1ezDc~Q!BfJgu@ZAv=8S)S8-Qyh&_$K_KcdQIw zSsVvpHs~84-xKgnkMN+I4on57f^*YQCv0kRCg__Cy6*t@9rR64_ydD?L^q>rcwMZ0 z$(eHKjn0L6c!aFlQ^(Yp} znC$MZU4F-w{BdI)^^TQs`)il4{^m!G>^<0ll}!x~&-jCG=Jig(o#pzwr=zLED-d`+ zc4@vDKIQ8;X=2fHM=j)TI-4=J8n_+1Y-_6J{6s68{T}BpzK;J>jm{PpvvXMja?v3I z(zDq7+xZEApw;IOIX_vE`u(rKUz%aEUm&1ALJ{fFBPAim)Azu49ZA>ulFi?}uu8F-fLD!Ifcy!Y5+TPXG-s9cT)ZMeStJB-Gb9+-~ zV^bq5?Q&JRN_~OB5!cLwx7@wn=R($0RyrTT_am0Fjrvi?hYUgA_;%b%sY zF~?FN1CiT0nsN*~;1963-XZe%8kK^vMCNAf$`a;~bOI{|U7*tr=8B{3EHjSQ$j40b zHmaG4=9z%D;~>qR6JuJtjoH0+n`3n`?_2VdTO~(vF$j#h$xa4SENsVfjpUEvh;IVd9so#G zF1mjq@7Lu0bKGAe0EzsI`b5P-BQcQ_Ur;Eke6aEr{{QN}f1fSJ{uh{UOv{RG+c)>FoHUi7|danhXnnO7c>F71Z8KWAgz=0*S{? zJ(td2%V+!dX)$&V3>xCjr$OHtW53U7Co4;1rF00Z66}|m>0hKnANgH|0O=Q|vsEE2 zfr_rN=N7Ru&$B1yG>s1_IeDq9u!|Li`tt~n5q^FhTOKNnWy~^G3Lcd|Aq4-V6hn`L zULGppcnR@1o=s0C+{JMh@!li(0!|m`^e#^CBKkGLJ2~B{<10D6Ql|?!U8vI=InBpH zZvP=p)8W6!uZGh!E(!bVIlW%TyExsY)AgLL*XasQSLk#lrz>^3iPKF)^YSxJ6DjiF z#OX~sy_wUSb-It!eLC&ov`43#Io(V&x8K6)79DToG>;2j{{~Js==cMien6*NIgK!u z~G^VhAWx>!<>Ft zr@J}btmzwcnD(^+*OApzfSBEjC9 z>1@%g!@&nRVu4;7usizk82cPJZ%+re$A<{eOTLj|=ke#*aaG|(Q<$bsmKL_0uZjv>03MEu20TO8cE{gBc3eZM;Z}bo@;`4d5-$Ne4jrc4r(Z?6w z9u*Bl%Xou)Wdq6cc;o%sRZqAPuQhc<*HHfNQU9#3BY7jf`xX%{<11l#$g_xWkB~=s zHzg9zcDVa=e0k;X0D%;}y5RtBi0y z-tqEwmJx1@cO@l6^YN71tFI^87>^tdqLsfa4?U1h_IP~1maZ)iC8QJ1{ekN@q!BL1 z1C%E*jd1Q09B*7qxEw!#FIr4E_kWHzrxI=&e^LoI#-9x-gv)q`er*cjJl?tfmSn=^ zcmaHCGU3K}(Vj#&A8%eGf9gmg+!$|KorKHr3i@qM!ufc`?bqcHF5?~egE@rrc<1=` zY{L0?!R_~E6VAsAj(2AfF2@___hb=nj5pgd376v)@P{)A=i?RT*Oh^!QkolM)S`&L z{rk#d>^(4OzP&Gv@Dl3FB*I@OdCldDrwb@nG#yXcMZRIsS||C2p`Wype2e4AZ&j$Q zUjto6;|<0O^-eG^?tv2T)ZqF6I%%|WqX$S2t; zRt;Q?#s;o!qF6W4n<>^!@_jT`m}rkDUcQ-P!O%}?p;$2FwML311J@Q&EE#xG1I40= zPK0NeJV|5uu2jab zDJe>01%X^m^e}|p%qKjT<$Fx{id-E(pO5u6MT<<^ZQGD|biU?rsUUOwn77tu6ki1|Nz z7tx(W^YJZbCDAkn(0W7I$9PZn&)Zm5C@)6-l(*a_^)ok;yqJ%<{TiZ;`7^tVXo^vh zUrq(ld_LCgJ9Ya@4(0J(U@Qas7mPmzfpz&+q)X{Mtq(GrNS|U>*2hjXPkcPi-bA!9 z|K{`&?I8g@zYM*-o;twt))D5!h5TT@ndHU%#Ou>YG{vB7uLh!dAM^Zj9w3^}m%9B- zy*{+Qqw!?gL;cC^x01Y=f4Kd2qK)|~dkfJNQ?k7_5Y6X9-M&k=UmnaOcj4P9vcKez zrcv9d_FJ)!P)g`8>^JxC%Z#y$Aj$s<^VlCA%o7ex^@sn$AM^D17%C%w14E`a*8wks zf64e5qWCGxFN?$1mFe-o@pWaqeMyGvyVvRd&T;oTJw7;|L-8Q|cOZ>vp%v`d(KWSn zen9mr4`!pjr(m{|W+{JBRY3^!hrc%^~z5&uQ3O8O+Hhdr*?| zX|_i4yb)o)Y0k#~p_*w}|6YLoP;NH#{~K>4;HyIYJRURk`7%3)&c7hU%hz%cLjg~; zen$S-?`B|n@IEf);B;19HH9{bb+7$4havs2>r zQ!vyS@@=yv_vkqCD+Cwy(=`2f*Ggff|;i2+Dy~(z@S?L$r^6l6kyfv^?qxcZzr~GMo zVU)jZHZLPypOyz^{fz9TqI9zSOR@Yj<7t{GKfHpM@1A-cNBLU=6&kGvjr{Aee2UZ0 zjIVzKhITW)R1^6t`?Oq<2mM?}r{BwCcPRkhsZw+kJXgzI|kCt0z`(g?}&~KYvzAPT6<(7e?{mbIpkCvN;{nkK@M(c5ve`~<$6zv5T+OutT+5KkwZL?+f z#nbn#il-}zbegw+kz3SH*ek;RUC$r(TLbPYk-x${7^{u)x6Rh9*6l&QD6|^OIivh- zvz^89cyXPM^YT}%jmMiXl^gb31FaeY9!d4@gP6+A_SdY5*RO-1S%15R0YT|Iib5N) z{5RvpTCstn{0=M!4EwEtMy)XprRt_5$wF zJfIAElg7_NNT1t}^KZyI`1(Z3Q+!s4_J;}dS8JfSO0<`v*Q^oktLW@((H@{t|4o5K z_@=^K-^|QL&^&QGaqSuqMthW$=>Ev@H90hQ8+c)%C_k7SU!8^Bs|k0xP&1J@o>Phm z$AstQc_b|KvsYq3HQ~9rqW!?+`dJ08IJ|keXm5pQ^7WTdz75Mn|0sM}CB>_Omz9bB zRCvWb6#oW(-+iM06@JeWawP+=sS*C7@a0(0neom};cp7R_x}4#IQ9$uSntUW>{XvWVL3f-!?{CMd%v2(F>u|A>gMmUxw zzAs=Ip`7ebp`UXuwusLE3;tk54E+(t&m){fJHIL0D+5l6^NN+he7JLXv)I4Td0kd+ zKC&Yp7w3byS{5F8{rGud9`}D<3qfbme$J3zv4Us=*Rseq@o0aQ70NaI8Tv~@xx9V3 zzXGogL7bT!(X$%fT`)-qCZ*=hRHIL@HgQ39Iuyn(zt+4IE9Pq z&z7X%b@)Z($8qjI8dW9C%l8g_pD_|AzcP?5>URok-ha9IE-u02&jJ6*r?#8!j*>r` z=uheU9@IY@@$k5aSKb~peT2pLF|eoG55vg$^j!3h$j`%RwATu5pV>sbX9aTk{xOya z9Q~aVVA<4v>@CDA>@5mpui$-2ivrNPe|IA;kbhPnJ4W+Yjy#Xa4LwiVY=qJmN^&Cjq6{g@*}`OWNne< z=lPrSTV}{(JdBg)<)PVxk|@6u@r%8AjFB4lgK_0Weg#3Q51l8V)fOx-*+P=#Ef4bH zU7XLLJ+{q#p83BiKw>|^_?r*=pELS@g;?x>ot?6bKX)K48t3lk-4?jp0(V>BZVTLP zfiKqriy@Ag_tY2o*mC7JG;x0X34J2jj?zw$ef)oZeeEx&@K1r?Ir?hs*ioeL(W7hF zw%*=f?)&7ovETkf?C&1`(Z|=m$g!n3lCLsakJRzw2wO^G@^1ufE5R&p;7=^FaXJk+ zZRQTxYN2M}bvFE$57P?M60h?A1USGvVnt2m(UVr_O#BVU?V1&*J!{>7mTVhfD3g&) zHf>g&Vu;yNQXx-evL|APVWJOkx-J6;$QirSuCkR>VrIL#pYeaJgobFfSgbD9zawWY8%ojuCm=r@W`}9@U8te{#`rK z^e+iF#YyL8`snbR;%Mxf;wa!vaTj{KrGL5bZ|Zr->;7p^^jQ28;_P9YElGnnaaqu$ zK^|L=C);)2=LWYfbvXH!S9)4MyH!ugtGmMgneXNv+P~c5 zL6#m?IJG`c77tT6jaMoyWMA!Pa9$ASAxfW*kz~N)K?6Z#${>cg-bf%4 zJRq*q&ImUC&qc_qo^7=7CCOg70bk+P`)V-0{O_Om)D&k*{QhYPQ{_{Y5Wjz-Zvx_z z`2Ewp@k~&Eg#cZeVc6eYEkxAspOk`#W5&eqpJ>*MPx|klBwNYHOMQ8YMimT6*CmhS z3}U1*Cw)|v%oN-g(*LQ{`%iQpXG&K6dw^jHaa{`2IWm$PP{JXqVkR_c2!#VC?4;u30nWl3eX>f=jjN@}1J z6h0fJzFTrs8(a0efj_x`eo*Qc=@!Lky7IqH z9gF1RpN5YpebtZ8!!YqYQcm42N_|DE{FR>S|5+&COn-|sK>8$S)&D*4sEQUhWnp#P zSpX1;^vO@cxXcOvD#&d%n3?pvK{6;)nW+7EAru4mKZ9danaF-zlKP%Frp3vCy$t(Q z{(6g?%K!T&!wdD_G?pneO+Y=vY~s{|f;umkF?4oK^^X$*y%`Nh`6IYxq)#Dj>JDzX3YB_#f&AZ-9qQsz zj9K$ne$)#6{m}PV@^>~0MRWNnf2F7VHwTP`^fyTTEvEb_B(3`N(oD=c)9B{7_76*a zsjlCw?cW3af3-mUmA+FdXs|&2Z_S_LhT2rU?+cG{DO2jJyV582KUR-iQjgEL+&~ma zhP<2eqq3^FkW$y(JX9zovmR%ucowk!sp3oBm7WsVQ`fVj^5LdBks3PcE5`Q@R@RU2 zs_f}~q;7n>uWxAYuASZGgA;wdW7Q8&`RW=v8g|r=3{Er$`*(Ja4^H=NAK&_Dqpz>~ zvGJBYb&u3l5BIh-&+Y0fA0KMjG1mio{jHE`nyDY@@9r7j>g(;D?yeng80~0it{v;| z>mM2F3%P}E|8Rd_`J+QyCXiQ8M?-n-$iPJP>`+Uy+t*jS7x`@8Rk_1G(9-8q=q zYHHcDeWs($OEEkhMMru!>Ks2n$wYec;(ZzVH;GTuU_d1#{ckzIXly}r7MmhtAc z-r;^~+w!T#iaifMQa?m>o7uIq^Rf1c){()A9mD-wI%neQ+B?2?sBxEjytS)W)}gU~ zXFc`7Lgk-&q;9BXprv+XsIgZ|kGIXz#IIJ=LSWzUnbtlW6y` z!HMyT{+(^`kCDb5J3HNj6FWu*`q9T|^P$$BW>Fu!{b0Oy4n6|ksUF_dSG{+jvU_Bx zWqfvEx_hc^>!Zy*ebw%c$J)@>-H+`Wt$K)j(mOFe8t{(QxCd)$hHCu775-{}dDU=r zMNMUuzsg-zxnZDoXhXGE$PJDT`v(0tc!l0}{_Z^W3JnuweNUbV5^u&cF--5mTbL`1 zdCUFo8gIFm^u5~yd;Q*qUc5HY@0)Ok)DX)FQ`(-o!t*Wd1;@Wh{I||OO8s`)OTseA z4{-tU@%3BCQ(iBal6(r=1 z^2GCLr4XNld{R6U)D*6zruI?wqUo0S%Kt@sG4>~@m*Sa?6@yRm9?4XGA4P*MB;RoZ z{%@hbkUZ^^7UHkJ0sjZ!Q+d9}`++ePSM>eILU#TG7I3`2vI|sPk?cb9Z_4_fvdE7@ zp6pD^u2H%bk{_1*M$nWBrJc7VU$yg}%krzuv5KpIC(G%UU3^8DVXsL$F4=*K-v|B@ zWdyR%f~FLNec~UM^e+XQeFt~q(}I@LX>nt3<4*hy(xv51Z<~8yPx%h1sO{Um7QS2B z%LZSy4Y9s1`8nXrtuN<%!^AglNneqC%2z)L22b@Hp~-;YtU+Swra zc@{e!lO6ME({2+V`#C*@OnkHap&RhuG4ajyrT@03WG$yt@~!m^%MP;Em(B|ovh#xE zTkHGBCOhWx|G>mI*Y_P0U+u9dzbiN3@09UF^^zgL%ePzd*IM|0Y?3$k@5_>JE&p-J zr*@!!uu{n>GgJ90U&N34c~bc*e!EHDoGt zP=4o3eA?Hlv`mf*lrR1Nu1ZHFU)9$oftMv;)mQPan&i#({ao^`^>xU~SnGRO@>RZW znePjdukuy=izaz-bl{e!`d zsiD2&$o@8WgS-#Mr+h=+;FNcW7fe(L#S`Teq^OIzhRss}CGchQVnsu@(PzHEo*D1V zw14pW_s9pPhI~OEE90NSAw@5>qb7|U5+Y@TGr{tj+is1o)ZX~^yLh43?2a&TK}-rKWtI3mfsJ2q5j+87x`Y1*;D^flPZ0+mVO2qTk|HB4EiDs Q>9A}VDhB=j%&Px?0g41`=>Px# literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k624b4bd74939b58bb85e6b0a212ac48d.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k624b4bd74939b58bb85e6b0a212ac48d.o new file mode 100644 index 0000000000000000000000000000000000000000..a423c8e3b32cddfea07361327ca811955453afcd GIT binary patch literal 96584 zcmeIb34Bz?l`mYqsMT7KgaEM`H9{6*6GBKJyx^8t1enc$*~W=>kc_t161cI+xZPwN zZ?Of*8{0FJ3@`Cy5HSu}UV@xtP~v=!CvhCYFE8UOspd3lrYbfIeSh)?(KW;UcF(B)8Syo;b7lj7G#t%w)UV^>gUCNmX4Y} z7TVh9s{UE7&-K6PgIe9Q7(05wm3#6fd|r0t&iggaui*0=d7lLtuAGw>Tob9S zPqS=R_H(DdcK~!whwCTR=zDJZ!{<*v^uYP~IKTGb1LyyT(?5*5-S`}Be)Xxrlh2b1*y{`K_-&)@#agXhy1 zF70VP_2l5mabFr-_ri|`U;XX#gFO%ZZt&!jzeByUo}PyvK7aeEhtA)>@X5jTFPs`| zZhK+yCiy!gf64ZgbnHPk8Xx&Pq@&YxWP(D@$p{pt^&9Bf9PuWo!{a2@(Oxsd8y z-*fTF2hKmx^6>e&KRPw|z$c#?yg%dngYz<97`*u4>x1)tM15V?GZ*-I_%v^N68wL0 zaBj;BgSS8V!eGy9za4zwVXCvd=i-A8oj-{>_ZNL>aNe3P45mEw{9tO;^MlRc`PKce zpiX%Y$!h&mpFDqm;gf^&9ztJ5&kwFI{Ndog4*YKLfs+G+DRXb=xp?!v=jWaJvV|eE?4A!zAdN(1{ucD|-$);cyUc(#Z=>@6AMaoUyX+ zpe&Emh_=o`Z{FOzvgfUxmC!@~@a4O{cS68PQPeKvhG^minkyg0kJ_2SCKKe>FDqvxG=XV2x8?C8avbKP(6 zbM*{WyP?NP@S~odn|^)gvEM_tPdro6>-GC)GPgT(8auM}%vVFb-`ejV{`&R);kj;> z`KHTr@((UIY=@n^bEapwW}4@+fAib!<{Z!Ere|HfCw}E0Zu+HT_{3K$dTX+e+}PBH zvY(^uD=7PXMQ_t4z&{53S-_tHTm<<0fR6z_0yqcoS-|%K?gadfgl0eAx7F97}o;7x#CfR6#b8}J&y zzxVex-3#~*z$*a11ULZri#YGac^k@$Q2zITw*tNyZ~@>40WSr-5^xUSyDGuAs~7wX zH@yvb8P0FvyaeY#oEPEzI?nTPeii3AIKP7PY@C0Ab1u$5!+AQ+=W(8l^9wjn!1)I_ zXX5-U&S^M*7w2T0{{?3k&QIeU0UuA{{3o1G;`~RPKZo=0asE4;-@y4{oPUdR80X*M zd=TetkjLL*9P@ELi1QCIt{phn1D^+cHO@0}ZpV44V>ohz{p2qA#u{hQ(FW(@qm37e zkM8|$@zH&sD?VC@^2l`7`yYtmkqN9f;$ppjN@Km`S3!5Q_LcA6SaadKH`Y3{PSw7S z@2}GL%u}^*;rk$c&p1{4F227_-^ZOII_cBq z8m{?e_(rz&g!=^IzOVoIwd^Bje_ivHq|3x(jdT9dhVL#s+W5JJNB4#o9^F^B@aQb? zXYja;cqJaeAMyE;j2oQKuY-rL5D)36YX1e_pQi6=r)rQt0CcfCY^T5+Q@H7uR%`;_2ypXJj7m^?B zoX0WF&%Nj?OnrGz)jo{x5777IQ?-AK@5ku7=Tz-y@%``Udy?R1YUtRZx7@?cue-s^ zfRs`HwKabYnSJE!-_#7crJSw>PjkT29Pl&;Jk8PM^b+zYRZiwy?m&NAsXzCr+K=M< z9{TP&Rl678>*%}lRBao+-%j5hVlEM@Q|^0k>&=k;5AbPzzjd z$Qx6TGd71;_1+}Q=7PpeWz-MaT!(XNN@#20tdoDA=3u?KDNMxRdptZJyy$ZNWG?*S zO%CX(bIYl9j1Bve*r{6Q=2N~T2hN*t#+aNNadw^3>u)%<&TZ7cS=L`K>#vJZf9c=SmJE}h^y8h>=evI;yR6i%W{!>&xM)}iJe{yvFf1&y@%D+qXr$pC(mg>hS z{{hvX8eRVdsvo2LJk_5TUH@lPKSud4sQ&cm`ma#^80D{0{Tb2qU#I#p$_J_bHPQ9o zqWUq)-=_LAqwBxBvN!b}Uat9tROkn-^=h2?M++Uf-*RBBN3rNEV*i_)*C3}|jofxs z&)fS}BIm8_xqMee&&Ao_Z=rQsnnN(A0&D<{3`Oh-%hCPefQtk@N3BZemA?a_wwf* zy{{urd}C#0?>lc~^!!KK`nfNCYIV=s-)G>{5B~kb!$;8O-TvOU_pR?4`V;GYr+pn{ zv9@R6pItozODcOWx+;7Bc$N(hRo~q6))6-RhfYWDTRS)OyqV_c{rxxC@SB@A_Pnvu ziGCdD)7kr4rUQI9dtXg<3=cj?JUV&@mLYb2%0G-4*n4R@;K^)w0Quv<9F%_va>xYy zO~Bs-E)(#V0MCJ(=Tlw4p8~uL@DgY5#m}$oxj5M|Jn&;Sd~x==xfef$Ixis)I^v?e z9lj6bWmtzGey1RA(|o?W`D_oBVeN7<;a-wM=t+|YXVAa9`8(R$o%Wa1_Sluy&-eFG z|5SgL!_RZvrqtMdT)yjR=$-?*hqV~u=1{c@`sarJCG|vH89(u&t<(h-J?W*DJsCHw zt(jd6Ce0kXZAC2hm*>OU%d&sv&=DEh!w zp2NRy>G9xvE&qN?j~nON{QK4(C(h6r)Zf-a`cr~WQ^EG0@i-Ur@3;13;#|bP@8}ta z^8)^TXHOc=B*&(LT{O<&x%l+|aC*;$PfY81`H`tTuRJ=X=gohd+;jZMM~6O_e0c5` zGC$VygzL7Ue_C06=AWOr?aaH=oxQ*PVs6jxzB;q#jc;Dl^PkVm=y~hsIX!=bT@S%d zhqpO;-~MMd{HK+U-go{Hb_%;6m;=9&hVv4#--_XZN6CIGdk1C%hCdp31aKAL34pP# z9ry&`oq%0{vF;tX8*n|~-!se+&cDSO>*Ik=l-~jP-vHkPct3+KZ#gsYLk2y?`5By# zY#AD;M)}HH&J28&;9G_Ub`rex%)sXf-a0g}nc!__;8y@|8ycu0c>9@wBLr_B8n_9t z>(-$G`1cdf+9eadtYHYn;pB zbUBzi4e7Z&Tp0{|FYIn{-LS$yIN84l9%Z#%H;V3FTq^n8&pdus&XT~$jrF_{*;rj|XJd6kXA2O`ZToI*ZtE(q?rN>A-rF8%t?e?~SykIryM`B4 z?%uw-x}>_UJy_k)+)52s)5ol7XzXkZ9K?t)OM%9g zt3ul_j@qkYAPtRm2liF(4b-+airKLj)D~$aFthsFmX>{i+Kwh}bRB{Au6D9`4%N4} zb#~RZbroL)>RVsi(b^cOzbd8~)5gbQpG#gST_>MFwcsk4Lt{9)&hpZ$#)OTz$&BoR@p-#q1-zd|Xudmz~s3W>oq2QSM~pSn<;WRV-`dyF_`7 zEceOXkaV!(;yW68Q#bJPRJ{<;E|h0yS^R{)DXQdv^;!7p7ogeVB9!}%XPV^_&9?CR zR=H0ix@0>>1GLvd$NeaGVc*AoL~!tZ8zAUA9T?AHI%lTe6ABeFZ{(g(p_hLZbr#qg zk+2~%cR30Kt$Ra}KvH`*$a?gh4(cZq4i(zF@rKcD4q>t&b#WY{GM2}AoFU7aH6RNQ1|*4h>FcH5#18wrFrFTMO9YCx!U| zTl(f@qMakx;z zehC*FFv2tSNA0IN+{};gMKHBTaLsrXU&=bNReUL{nV{lPf1-*9JW0iaK1aoa9ty7Z zUo%C;gMO-t2mLe^5BljU9`zBpH2$wq@t~in;z6IQ;z93I@qlNkc+lsmc+k&Q@u-hL zul9esM8Gs3Bu}iDePIzDupjz3qe4bQN)Qti=Ip3K}c@|hp;$bT(kefVrCL0tMcYyh@WjL&hzsAA~7 zJ1d=E>AA87v&khfqZkdm$5e1RFXqC+u?>>}ViA2{l-x@8L}L+#=Q; z9G6da80n8>Qv2P(8To|@JT>TyP!Ikb2=+oQdptNRLL$z~ad_Etn^}5*IcY*VwhX#J z?|wYEd={;rA|7yz_R@p6!+_)8bg`dhvZ|>yfanOh*sGcB$vZp|)(kKe_4A1TRIV4i z3h2X!-HiCf!P^T5)1XMAfZFdpWGs8!z>jB-1~WV%ppEf78qD%ggBI-dgcP_t=)vMY zi$(qed&8dfs#k&d9%$fLSI|ao5 zwq>xd9Ejw5$nYE(U!70DG$9;j0wzOLVRE!8?6bi%fhxXO!VQd_Jrqf01ao;VVpVey z9Qh&UV^NTs7fJv_zEciKG@V|={N2An7Kr-cU^2HIUI_gN2N54+4WeJ%<-+~}H5{4vyqpPr^dynGa(#rpzI4FN*WZwam*rI>^0~I z`lG?{1MPnV{9>3mz~5VDPbBddkJ<=vEcg-p=$Y?&U_lr%PzLolqbnAm)YgLxJ!si$ZJuQ zJU5{K5c=oGD|_Oe%8urGHcg#3t)ANH@!C&tqq^yRZY%l;n< zGB?dnhwoAvJHHw7_Imkzk-TBA-#7V3i=eMM#H9-gp%3=7Z7WPf9VdE7KAi-RRJleg|93KDRB63g{?h*0fPn_^}~m6 zvf)=0U?d9uMjQTy0-Ah_{(2k!x`L2~UuMHEEuhUKi$2$epIH!6@MjLNwdEwQ5Y6X! z$S>Je-%~)wVYQcH!+Q%t8a~a2Pc0yaV9}4$@YH{)*}nt&oL&&p>St>8yMq}8;8baE zybYgK5Yq4yZ20T~N-PY1sQ*<;|KY=vZ1pD=gp~T>!;@|JoC3r?v%T=)sW$wSf{=!v zuHmWwN~Qnq;IsmGBBi}+wEBqe1+Zij4|~t627QMg_7450_WVkF1Y`Lq=hwmUH`rh- zH#Iz#qZ*9mvIdiD75XCD;{lTEkAN{;4G;6wU>#W5Gxs!I8cY_a8@tv-21lj1+Tn7eT+?#1s9RJj;FsJ?8eIXCapKb;}d3mb;q4_lB_qyQU zYCpv`KWVo46#l{*zl-Blc~bkT{0X-7H6HKI|Q;rVC<1?3;X#rS@SyD*WD%+dUJ_%N=miQ3XX z%+06efzkgH@Gs=gjC|(N;F`4|f=&N2i!TLHWcZf?3%>~CDa1lvtv_GuANrEdzH9V{ zd>{U1E*irj#(TI3SN}}d*&SRooBK!E|Kq`=NG{OEdJ5|iOuw-n$z)%{{N$5Jx~V+} z!E1ab?4tF{dcM&q`=k1W2L6>KUVniBUr1(W4?x_so*Iw#@{z9|DEEGbv)4`j+hL#Y zaB#jdUz;%>+!{CqyMXx%F2Z+pe&*@(dAI;~zw-n9lYI2~>kclN?F8DIk7DLC>n}q6 z#oGLY50_Z-d_I%L4|TW6i&!Liy)5KaY`_;FFB(6s7is)YWbl068b8JtwZ5LCjGxwX zWisBO53ZjiTzOm|etEh{1!DfA^miWP`I&)7{8IC4(hpj2S@IZBpRhhJD^v9+e7Mw- z|8rbE$nSLdnED~*W9kRiV~|g&sUKWE=e&>e;gNpCl8>q%Bp+2jAdf51k5#;VDX$p% zLH@&%hpHc$BkKpvk1CHJyeEAr)%E3g0(sc=#ik!o`e4(S!}APxD*Z>aesKS5`mYi6 zt-+hBE#U z=G(6CV*K&#Klc3F?0-sv@xTB1=j$hP{L%3N>7g?I|7!am@cwtMukrlh`_~7)el+C~ zZU0D56?u%H4Z{yz}=?;d&l@Yj4jZ}1;PG_K?d zBJ+a+3E;%@A3Tyyl*s%5`l97e(es^#8p8h}o{pSf>hVsDe`ZwtCzh0aa)AlQivKkU z#?QC2Xhay#O^EmKR@(Tbbf^aRPafO&DSJ`#$LRS_M~(O7|Ly)z#D9bnJs-*^7S(*w zfJHpC#sA`woAet;-1(+ULsvB4c6}H2A3guXqH6yp9LxSsDC6IJ1>={a zp%D>3LDJTd#$RAuFqGqup8qVgV2xj$f7sLE5$4}6k9hV^EGzc!HDKgNd_82W7r8zJ z6?uF(`at8w5mz6^18K?QJ=niF-_rg~IF|iyQO5h>%)i*&F~%D`|Ae>3JBs-?^xy99 zjro=9U)y|J`vZKweaiShocZSa@gvOl1a15y%(q?Njq%(4zd7GA=iiq9U-Etu{R{v1 z;mkk3{$P$jdj3g1?`i(+^7ugPzfY0JhocYt`im)#=zMUJCXW&HLFE6V_5arXAYcDK zri}N)nSXwL$sDgf|KY!thx|Z` zfAi>e1UlmQ_t~>218I$CmTf=T#};sM&v}j55f0~4H*7zqj|pqu1ev4 z3%VbP_MJcpdDfyXsHboT1^vhsW{z zwfKEB=*hCKU*_WetgI~cdS!QTdZv1PfUbX225GbxK0Hm!m+1a9cG}{nJ2;cyFlO<0 zjg3Cn#vkr?!Ot0t`rX0VnIvrk4j-OnqtCPTkLx4PS@m&!Ema*4UBA`&pRe_g{rhZ2 z6Kcu}*HeC=^0O$@z8|i)SETU+efLm!gAPmi7w5BnmA{fy{rc}>8-GhQ{&4?FK6_r} zXIZAcA6=%=qkp=i)9gQdc&Wx8?vKf5Ln=SlrRvw4muu~X4_|NN=Z1XxIf^kq-NBnO z$-E7i#;;!A46{Yo1@tMKz6i=M_$_9f(>g#3K9 zjlN2wCw{H_3F!JL?Ck|(eB^&7XX^XiIU4`epU>=%?stK`Wf}a?^~he6+MnX@x%_9&w(*~3=`-;U{XJ*&Px`O>d+OiP zSK@b3rnSFA@`wKRtNknfoy&jTB76UqKGS%hzXL}9B!AtXQ~#?pdBeXWAQ<`ryyCxz zzw0w;J#N`2^z&s|ANMmo9*ojQkEM^q5Bwv3Fed09;r43T3(0GGrnSFA^9}vGptfIP z*Ehvq5`WOQDMtI7!Ox17cKKNPMf^<8wAPa}->{EY4SMcRJeEEYf4P~~`kVNJehpal zHGjzUV@m{!!OY%x{fBPx`6HqsN2O?EYQR zC*mLa+HKJ1a((hc9wOcfeYMs@#4q&qlB^#`UoCscM1APf38Q_XuaRejv9=odiz{z=rYI?!*m|5o(<^!EGF z>GvKaPhn3qzsad2KLdU|=!>`k`zGS|EolGo;7ktZOZXUm4}#i}Ylis!VBjYi=5N8P zvED?w3KuKnS#|#j0CWI;joM4%?Rn| zKF5ODNS`cxI5BII5%`4}PDNNBU*8hhO(A^YH> z_KOAl>c?2Y#6l0d#zprVI~d@+oOv9^{o5RNPgx*gvQdKKVHV(mpdX z^!BIN{6UgdKOEHU2wGGPPudr_~`2)q?Q!qg$O!l2M&r9}h*mptp0u3&lSR~Hx{-W$1`X{50@NcU9^ZY6^GZ)8O#@8r&(&FoF-UHksuH2r?_{Qx?iEp^R!Q+{>AB}kYoFz~A zH(Pw0W{YonJbOIo7V+4OZ&PgXW2#NRlaloKmg83KCmbx`qve0Nf1ZP{YJN$-*Ntx< z$AWp9y@i8O@y!>J(<9)?9HJ*_^#mZ;3 z`1NV;f8*o1a(hb{zqIQuh+qAxz1i1;WARJZ$FcZjv!_41_+?%nHu6C&e*M(jI!b#= z7{AQxX%x>8zlKzM8jD|C-1LXX&$0NW`je|6ensVrTKq~MwY?>bU)b<8)+>B}#JnC& z`;}wyOU+Nm)?ZiK`fG!?`GbyM=KUBleidRpBk#W&TYu^MnPclOegE-lh+k3pq87hC z?rj{Uy(NraKefd#ziMw|@oOx8F+W#1n|>D`ZKov!u<_bU;G+*{q>IbBcrspgz@V|Tl_NbM;%*#ja~oJ?dvZwenr`n z7Qb#7wLK+_U&{T^JbvA$=7VGLYb<`L*B5pBy872&UEbXvbo^3&ufX>YkE`}Jw*DGh ze_hS%udjJ;9i=@bj9<#{S$O<1e;+fp{u;Y~isiOTFLsYTQ4J+OOB* zSA%!s2OYnR-}hoan#V6@+27dx8}lJ;>;B@g`#1jb?%#;ACvE@m3Gcd5+Ec>#r98iY zufOn%hq3qtA@l29WAW>1k6$CNzy963dX)B-aQ&q`pMl4(V%6Tp_7BJ6*I)MjVU#^- z@oUDY?I~gWQl5{(#r97{S$FHDjPhC8+rYO-zyu>TjTqOY0C53uH^cc{=5x({uuq9HBCM*VD!)5us?4j zElquXz_H-_e*T8-c^U8f`5SPL>it9spTCjz0Y85u?R`FfBTe1U_-J=Sen_E>|GA@lzEpO$YH#%XJ-^?|5AlA0sQ85m#}W7W8&f>f zwC8Wk%_>IVVw zGzR|s%9nFlHu!UbKRnMdnW4X_K^Ec9)5^hLyw4#0Fr(K|)B6b07@n{B`XM(HE#M(v zu&>t-3Le5{|lM!vS4|sPlK;3EmmM>;6{rc?Uw~_DD^A&Z-0!H()>iy z>3!}F<_z4V)@NnG6{TB@`gpz?>Mz9u2Sxn>7s2J}CY(g@b;KWUZy=fA8^~To`^jiO zNohY9{J2YNfH%hD3?wUf;@4B!q2W^${377Jr6CQUrr?P_wX|EogP#@j{2q&+aZ3Ho z7;k#%aixA)u(b5F1=D!kGYon$KK^`MWxQU}KLt)D52?WElo2T~pDzyM{mxZ~CNdxJ zhQ4dhr;KOcq|jvFu!l$RJX=mq55;aOqg;WPr}ZuE#F1pFg_mnVt!1;KI3+y9?~8< z;Q#Qv-SA!R2&;hvQc+Y#@+;;3Ekgbb_3?bNJpMij`F?>Ff$|9T7sJ2+^XHV9d|8L3bI++s)@|5xe&gJU&c*m$Mos_?^V6{&ggN+pd6L4;~fy;qvQYeUjhFu!mB{wn+P^ zOX2oF^Vh-cf%NmHWQu1vq<*`Tuje52_qjKqZ#P9Gp2s7=H}UzWI?xYI-=gM^^v#n% z-$eV?{3Uy8(6ps*GCwi(Ejc+CcuU`slS$t^mcAw1_3b9+a~S%@{lymPAL@#@Jh;Cy zL# zr!S@c`N8em<5B&ACf}I$oSZ&Mcicwn;$djsxtK7X9Q zMTk%3=~g`2tlKA^-yFWnYl{bI3G_$sXX#H`S`B))^e5e>KWS;YeWs<6eWsiGyi1wiY1n79{;2#?JV?ZEYJ@`Fhskd$J1+Sp{lV;XNd5U-9oHY4AIrZcjg$5* z^&uV4Yvl*QZtkBujC=*J6$m8r_Ya|ltiODmnorCMczAxrTj2VTgj7%-T;HzCQ1cCM z;QCC{pY!_Z5z=pda5$OoH(IdAqvjW+k0ehuU!v<7GGF5P6ZPLw_cqU;kl!SU{!z%= zcc+W~4LCVj&1W?KHc0xJuIMMNpE45YXEGm;HQ$*THR#IH&kUP>isv<3@gp;n^fSZM z&vgDiv~-bQI+3rWLtZYSpVMRHD;mGf{NVfbZ+3oT$2)~z9`96sJ*-CZJ3dCff_&V_ z2lHlv-}D6XP2&1y$#-13E?>L-j~l1k|G06w{d>ld{d>54b15J6B(XlpuY*{J1a0DqUKLnUu4sKEF=3A z^eL+U@CKHN_o4Ci7sX3&M7C$d*9m5P#d>N*Nfz&)>j!UdnOBX+D=LM247}!_c>blt zqddxQvu*Ouv#qB@ytnjmLbfSSzMje!?=Ls=uL+jCkROu#Y5duyfAUVS;wQHs@E2`;CwBi>KW<5joRSMZ+`EFVvJE>p+@`NTvspLjXb`SC+o z2zDF!GUYcOJpZ2`Qa{rtiugnN_z2f0df$|c2kv{9v(ja(U+{yKSRH~fr*op z_y~I$H;MF-$@#o@9V;zo{v>X1;E(7@{|tTe1}5c*_XP-hp!K!nU(UbCCwY^6z9&X# zf60ImZ~6PD089N#N>bw~;;m{~{h4?XM_N{3q$(Rwcg` z{3NUVuBaRrq4v4HKtFPVlagqDI1e0N<}v5<9>yn(E#!aoUl!e34k{>G56-M)2yf_$0h zf4obKpY#v>$o^d9AG{DQ?CO!_g^k{{XoOx1qxoy$s>vg4Y)PvP}x|Bn2-uqPD7v2PLoE&Qa2 zZokp?t=cp8+w%?5ugR=Wqo@49 z(yvKA;eW{97l9w|Ea88I{qyyDpR@Cf*yEFkAQst`8V*R+g%7EAn*vguk3E z>|OA?qEh%X!+t6Mk?nE)Ua?j9H;%`C6Xh=?U;c;d!?cO!{6b#fe}eSKoF4JOYsLd! ze@Y6$WK(2Mq(9S`FV&bcP3@G$@n`^}f}(9L+ffi%uZ{i!4QkGoddkEt&{G;QdM zH;_>v?3>F6`UmdM$@xRPniWXb)~~YyBK{G3#2?MieBK`77xYQ^^N{2>jpJu=yx7m5 z708Il{y072PrmR^ByZ$P>nMKpOL_;V&*SvQ{=fTL;UC5PVE?W__(uT?|L9K>@=7B9 z(^Y>#{Lbd>8S7#9I5ob*e`QGj%lXsv<9jr}les($eL#GeBkY;W5B@f1j#$5gA6#Fx z#|xUziR`%K&%styPT~2GXm2z4)&1d$ty8V_9qEIxk91>vG@g7u9`1kn`ruki-qfDv zA4$GbSdF9?^HZS7*Ndy?{E+sYGFQnjupa_x^{e`oI#10nFrS6OztDV8{K}fE<_p}v z^{e_da=zdVq|LYDFX{XEdA59G#C$>X*N~q*{{VmUt^9%J*KY4=5!t`chf2lXd3zI; ze1y;UB+K6Ue5Of%%jq?H2R*m9ONKlsekk_;Eamr@jZ0GAg&90Qo67p+{0aM;$LA0F z4F4nj#W=%XkbcbJc<2Y=p+7k+q~QxWUieq!3k#$@Kz|o;dqlor)`z_7KI{+U%M|TXKH$aMVhs85 ze15TFZ@{OPsP;wvEs51gddlz9mZHv&6Dj zvM=2}X9bwKeM<3ZR>0+Tfq#Ao{gChXK?#SGQdNJC^(5{Wln3xD^2GIb?w`5;cSWc@ zBfe|;f%>FBjQO&}{KCJh`hxjiVbc$?M;8l;`go3n(2sr#=JV|z&)XwBpTFY*#{EXu z+(o=r;)83*=pXYZ?l;;Z?>91&A-P`{bTdDZ(1H3N*WQ;qlP&H_662$b^2jHS20Te* zkJw-0?|aqn=OKDm#6t@%dfzVc4Z43uzW)~Y6N&q6xP0k;8|;@%4LCW-FS649Nx;Rg z_w)U$A1q?&0eSs@EBFV0&Y+7%M%Tai{YX=RCy>*>fa{wnw-Ja#jF0m}$rJH|`<K9uww)IRRVs!H(_8i%0A{ZX`ig9BcP<})a@!|^cZXX+t|rYgv9vjb8;mS8^z z^ZRYw4NT9e7i@-zDOxI&_BBV80FbyK#QB`)Le%dchabLw-8_-Qa&9 zK0V|gHtwGwd5iW+nZXZlPrcuc+MB@|;_(x&J&iwmdmG}nXN*tdN0krp6HH96^>3#? z5uZNZcrrT*K)ObhauYdIIjmY0|_UQ!tjZS_| z?6J|dHzI$9>>m=eH#+%U8hv{s@^_jYPr%>kOGZzbR_s=X2U z+r|14v^P5W`PlbH+un%$^=s|jhV>s`@8VZww4Pkth2Ib0;6FM-yxVf9$+?R6{|sN> z+3VAO06%C@y#8%k{i;Lnuy+jp`!e<4BVC2C5XRWwF$s)GU`zsI5*U-fm;}Zo@PAJN zbS)RxXZk*gq;=rz5&R_WV_)fAe>rj%M>6Y=oc+g>^KNy$7J1k4ZtlC)Ly=2&57EDo z&$+pmBkx4MjVAv0|NbQo{CT?Vt~>7f`2lvY{j-tpANq~^Ywx`Cr&phT%5mzQyWc#x z$^XN%UwZ1d)6NY>e)4b29SbA>^7K>t{XdLUwNpi2Z($_&+no2X{a&6zo{t&r4^7hZqp1nKrMs?)f$h&`dBNF-3&(Grgt4QRHU)}vj{Qc|5 z*>C>o?62N<_ua_lH(o#cO{A`0yd3#D&h3}}6#43Be)amdzxl`4fBDV3-k>kPyz7l` zD}Rs!JJQ^#Lp-Cd;EA;bJF*T+ugI90 z>P%0~c$@zq^@z(c6m~GVAPGsevhXgqiWe8^@Iz&j-sM#BA$}o`v7a~`)0d*xZM@fz zvoAuu8lPJn&gmcZyv-_zsgzWP{?!9pOC#awQp8AjTB?kMr>)76@V*52!UXtYj{jWA z+syw2T}k{-B;dR>9hU$KhaVCja#L026e>(bGuUo?iljw*&(`mzG+Jl*1ORFQU7 z7|erWPma(=Q1cdan2*fVI9E1loFl`#>TSkC)QvO4nXSR&Z18v;4rl4G(*}n#ZFn0T zw)Gdb^%qXo>B9~kb~<%9Y=fOH8{P&x-8Q@pb|%^IHrVO0;cc+f&c9c$?@ZO-C)zbU0D}QT!(0C#t@UzbO9g?b%>Ee_`8r!uIh(-j&x@PP9DY zk@1cpXSUs=R!o02)j3`B3hCq(B37UMPb~uj14EX)K&(YySegTE#yRd$4^6+V>l-TXt94RZ4N%{@&DQBfKC1%_V_jV{}e=^y>al2q#`wtr) z!sdevAE@GWAEe1Y#{83g^cg;qE*fY)&h&u>hqnGrAE@9*=s(8%YyHO`ztXhU}MP8N0|23%SH?|J+x^|B-XRE3^L(ZT|HfHNO8dJ+$thjT~Fqpm5YNte~xf zue9r&sn#xw%S~%uWj)})dcbY02fz_OEIjS7Son~E*W;L*;s~1M2j8KXJ%g_m_?`ID zPOMR&e--cYyI9zTPOL-2C;5|DI7u0$jo#z;u&^gSeX>89g_Gmc3pvK87i*FD^uiCv zrg#7kB(J&c&q{@{QlWkZ*kcg?!`FkN1yfi1Km!Po#fBzVYc3&EKE9{w5m#`;~tp z`3e1e@ALP5^*7P{C9=Oicm014@)vs@@%KZ-E_Qr8<>D2z3tiuUcxL6-mGe9v{wYlN zRHUc7D{;1el-PyX7WNvk&DbQS`c@r&g~6<2_@K)A(h~YO>vJZ)&9b&I+U!a~9ATI& zSh5jgB_2OO!)6!ZL#chS@lJ=+L-cTFvFTl5!iUoORJ@~p0rsZ7>-YHH< z#PUT>J>%9gPRb|}NzpF7n6NGlK3b<+juyPdkVZQf9W9GxDe^ra~CY+02vl}mxFDmFgNltN5=*kejT!K+dTjoPZl zbm&9zNn<*M<&R-CV}9s8#kDaV8q=Zon;VVk(3lR5>Ch-v8Dsg-SU!Z?EtHk&SWazU z$6G6!@YuSH_7BF^Wh3ncjjiozhh%JRKhl2U*!~dTMH|~68rx$uc1`#>?q2nO-5z5f zeqk`Wo%^wz`p+w;#%{cWu6KC)Jjy<@lGP=LU39$zm%3x)y<)dNo(|&n27bK+SCZ!!XQNI^gmy9Q@QC3ga z-^}L`D!=w|F#0u5WqPHDJ{8L+rqaiN&2zb%<16W`ek5H~Jt|l0Dc|XLIeb(-*r31) zT^|+wZpFIG<%nZB+#-IBF;QHba^UI>U8XXwR8b4S=Vf??SS7xFhxwUteUPJAF+hRe ziUOCzj;HF*@(lbI#wTua!Ovjmr$XXcqN0uWK#yBJHq6rqcfv)057CeI_)+8G_(GlD>c3CmZT+KF;}Cdz`=&1p;aQmA-`2k6 zEBgiB);=@a_X&KK&Yz{f$Hn-Hb-dr~A3w06{%!OY|HlR1&cE4zO-d~O`%FB3ml9)s zLT3L=;BEa|cwcHPdW*l1z{lmUU*O})7e4@sF@9$9e_Y^g^0)Y7>73ryKQqTyBk;EV z{U*Lo;Ny>fT&(ejO#U#_#DCoW`vu|cr z@}pJmNO*onye9vUi{Val^YX(B`e?*579q!7LF26_5U z2lW>UhYIc8dBf;FhcH>DQ5~Z)ymOG~=*>=abcho(`rjg9I#In3y1#zeexDq#U&7+# zUgSmg$=FJK+<5zvk(1GPBfp~W#_uNG*~<65Zbn0OFLSed>Ac6s9{5b&6JP$!#8&pT zdB6YTF4sTJSum^kUuS;*k<9dUEvbu8XR`PedkMokEiR2W;l&Kk@K@ksHUM~*iC@I} zHMoeqsKE=_ISpRGp3~s@oXXY$w)jb5e!!N#d0DvzC(|yLQs2XTfUWT-u`CTnrmMkt zFNlL(aySkBrxaO*DW_Ipn3I4vySYs|Dlh?X6%ITy!Zkg@oEx0XRqb-P7pJpJm+`Eb zcNwd}$31t3=^4qJ*bjH%4AMll1;*V?#*Up<_6}pm@#)6XJDux8%)PwB18IquVIVFs zM?OiH!<5Go+$~|sD+oR0RZ? z%snHY`4Nv!FF}@!wA}}vb?$ozgZTJhFU1t6K;*{jkfHwcpMTsl-S{RBig)zwU3tDN z&-5JW7LK8pFF1HUOYx458Mbo2gg2l+I_RxcY2qvY`wxhnblA(EM@)-VeuF6H_02tb zl-)oBz;UDG$H>s17sY0|u~JxNE0-XfVr$1JKM|+CfgkO8E(l!as=%UN&dVv8(2tJ* zbTNKOPeXheA>Sq5imH`=ALaD2Cu6+E$|xUlrE)_)eo2*%d2q38tqlTh;N|9tqnE7DGHXG<&#VLP^^GSWNAqQ=HKIrUFcW6EL7pHW2mN%Vdj8;5CNIDGss*5+`@atbXd;ULhv;R} zIDGuCr6UHtF^`X+VJh>>ddBIrq&Hxre}m4L|AQbNi9RSBAbP52rzbx`B7k{mM<`c- zp>K*a(bGOfNPyrz$ zX{*O1eNeH#So7BbzlG%x&7ZV5(NO3masJ#YamJE}k6){VvYQ|{>K})O$}1G5TO?9c zG3y#2+Yb68G5YVCD+mq!r~c`%iH|i(dcUpz(~{oMU!o_MV5cvId>)9=zrR%U?*k1T z_Wlh$S`Yd#=>4|-Yb1R)rxx_ozd>i{?{pNzqQ4~R4gID5$tBt8%TR!5)B*j`5ch=n)3O#_fK|1GTn8fZK7olM*d)&4SK5mUg@B+%&Gpg;Y*tVl;Uwwr*R&J~;X*p28 zq_SoAl5Mx%UeU6>c4xzZ-Mh9fu5aCW>;AG^+G{ITZd|#evZ=mxP1l}XTU+Wowry`& zcSmJI+1B=ob$4vuwOGOJ@9L;)-3Gd?8!C41y!FmhUu&I8@ z#sj;T>?q!|t*p3lS0(yrMEyOt?%7#(XWf#UslUa$TlcNLy<*dn`jXmPYb#pKerqch zx34PMcS~*MCRt}qX-n0<^-UdHx3#P-x%K9%eVb}4mbb62+PA%7=lb@_b$6_4T)U|? zL7mFRJzLvX?ykicw=Qp4+0?#j=e{kqm6h$Qm+WY+t=P7)a!J$PhLYun_N?8UfS*0Z zwL2HL)VC$7*Kwd>=iz#EgRk`h0 zU)Ofaof~N$*GgV@t~s=8Nki;5tM=V-dj+Q{*|grGSy=^|l0BWfcgCeFmGWG>d)NBC z(4Paf(3w~=+t?=gXxmg=-@2ozE~ac(Zd_BfY4M(_)dzOBwj9_~g%9*%&)W4J4Xry4 zK_}PUvUtax4Qto8?A}?T=){iwwL8o9*2Zkpny-~Rw|3OEEHB=@bA9KYM0Hzv-S*m@ zTleqTxirE2ivG!7VvXI@7twzry<1P?-nC@U!QCZmnq%rf)xJYg?rV76lEwu5HE{Vr z?(yc|unW=m$a8M=y|%t}`5k+9EWf=z(YV{zAFQw1k#K%0mxy}1+qNYb&z?hd@MpW0 zR3)g_BI{KitSdQ`pl*k(yXp4bJA*B8(|0I;^^CE!#;Oa!q!@X zar!mvYz?(lyYoQov8-y6{%_A6^)1T-4e--3{iMBKJinyZ>#Qq@}gLNH2CK7|RaPcN|?b+v9TXT9QD{Rb0-y)?8V=ombjk zWjtLm=2!N+^@{54S$k#e?Wqc0S$kDYS5|NBmGOt1Pr&xw~pId)ta_F zyW;8Udeuf+>f^=kRmhu)_dw^@6mPs!<{$7&O?yO6aeF1jyB)<1yVf7rRkG$lT`S_s zF2sxy#EMnx@7dZUq#RkyY@2deikEh()lt!r3Tx_rsr2ZUsztr}=DoG`jn(xBx>{;GJF8olRJR7J_cgX2tlr+(*4ZA|*V@?H-n_JQu?lOz zh)Aub4Ja+IUR+XAUS7R)acOCF3I1&l9B34O*3`CiHdZgL7H3{Az7|)De~YWj<$&+X z+us;yYix-%28j_v*;w1rc}Me|ceXT|@HPQ=E-oprt6oyw&~>Pzv7QS;3Z$+UY&3MX z0MXpG@7CtFuJY=x*4pa5?Sa)W<3bgpd*VIN$=0JzeQ3+g{!{Ln>uB6LuNMFk zo*r)q3T`S$D1~Uq2prhUyJ&*VvRmrO1}nH)aNTz zqllkHxw4rfv7(%Iru5@MfZ2+CjB$})pmM{GguEF0Ul^BBrnYn&%Bj9z78>@G5vP7L z#2Tv|+76FZPTK~t%D2TSe-Y)xPe1yn!?4e5;_&keY>ZHS+UKIfu+J57>NijwUQ_== zWwFZX-UDhsB$0;wVoycLjW%s?7_=q$PW89Q@>Lwm*bdpwaanH6%Xv9oV?Wxk)1S(5 zF}Y7$=oSsVD%&Z>O*M2Fr$cE+y&BZ5lw)7jJWeR(YW;7;DfeQ2 z=^#GU@s`SRJD=sU+&>A0vIx8az_38h@E->;Ob^~-Y);`vfM8JU9z0|rPyv9<I%O*ivIeRYuK2U$^|v8A@Ct1;Ncu5`sh4OKU{HFq)ot)aQL zrLn%N8tazEKv#9=f%^K!&Q4ZThd-lgvh6^tV79rjlOcDk0j9?F|Q7(ESyzDEX|k zwAVINcePhHaK^-dz@Fe!fMP7_2(fmuDwwOP)ou0d4UOBITN)ci&jZVExPqbA2KIGU zcVf}UiaNUjUA1*MA8O_2KzjpLb*za06-q2-Zv#e13%)Dn=`Vk!edQYP7r2d3Bxbs( zzO!rbQqy}$E03`;A~B|#{uyp6$dK^9PTxS>ktvtfPM-`2eZscrQ*E$NN*wShDGH{cMyP?=qZC-w;b?e*>Ok7fUM`qPqrgRu`IJ2d!~ zAX|uTL{CPEIKR^vt2Daqy^`KOezsT8(>hf>?EK#Ye60C9*DeSzNn#oXrJ@GCF@I;# Xaa7-^_u@W5-zNn_5Iw8HPXB)ZTYUU+ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k62d97aff61c66ea719a2978c9072de7e.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k62d97aff61c66ea719a2978c9072de7e.o new file mode 100644 index 0000000000000000000000000000000000000000..cb48675f74263df4bba38898117d14a14845dde3 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+mFh-3m%)>k+fq2ZsGpz>*$$*4Fyf!nUB{fK(N1v8t zT85kmk!6g*U}APd&c<0X@px$&GR{tB9L?rTNEkD?<8>f0 z?*0CHbl-c`Ai$p3u+&laz4iZpRsFB3|F7!S?JqA~u}s%Ari#X1VX>NmGM3pByZ7*0 z4@(85n{`+B+iHK7?Y8}IdZF#JQyDuvXv^+>2e0$C>{jk`ije9g5uW0FdediBC zza7J_eYjo^eNbr&UAM8r|JTNPkF~SIpgq!^!3w{Z?dkilguV4kogJQ<+FgClHnDfm zHlF%A(w)u<|BvqJ8$vr>v%OV|v4>ATGf;8iz`*v)&kU?M^UT1vbk8-=K8x4knWtV1 z_ntfve%$$;@QDK_!^3kG#4Z(-wt> zw;vdI`k7}2Qs*oT&20Wo_~Psn;eD5X5`NnGX1M9(rEu?x%b+U^g$^DZ*mm;3z`i*r z!WCyuhMQW?gnJLX86JN0QuyNbOQ2a0Dt8_nxHxC;z`j-JXa9G?Gg~nJ={LjW7cPaj zU8eE=I24+L@m3u~KPSVpT3!qf17C6Bba>{e3*epl`J>SA{(}Qg&v|AbboylYB3@0G z-we+>bt+u$yd1tb=K^S!gl4wv9q5HF$}gNi-zUSNlc&QK&NJcAoJ--SR}s&PL!pxg z24@C7|U_FlY_XMQ164qeVl1P%0-lK5gc^xm7{ZOcxDXI^+W zeDU!M;W5)53k^RC{4((R_^I$dz>{bGNoe@-y#wWYUkoQ-04;b=*0QfA|9XG*%nL6C zl6Pl+KRL0xI{DS?AlcDlV}%abNYlHk>)=xpG|#njwh6t2+pM#~sQrNx_MZU%5V)&is+V!2U9XI-KJb(Pa zrp;15N&csNuOGhm3Vidv@7>+L*L@S;J2r~%9kciz`K0o{evAJpf4!LxUf=n{P^4ty zTSE`)hcEt1@in)drM>&h;%mb%E$#c&8rGMJ`EhPKyEgPg{o3pB>%nYx?Oe%(!-MtO z;d6hZUmx1f`e+_8t^cv*z(<~AeUX1k`p!SVuKnA5)^{O|UAvgXuBCvs z2y<;{PeR|&zp%cchqc4+{xAL7yYH~WLw}tR8u~Un{BAwg)N!nD_(u}~OAHM?%ntwh zVbK1Mq|mwbyC0*u2A&UJ-}$O(E{XMB$n!4vCs~HTD~uFQ2eq5<>A9H1T zuBFWOgi;q2g`8g~4v}rhCS)h-?;Ys%JoM*6{0t;}`nIG0Z{fNcuL|6E<9YzslX(3j z?$6-*?=(K__0Dq%Asg(f{LeO)U3&-mcTLknF4*T?_$}G#SFi!u{32|AZa(W9g#E}4 z%BSC7!}`J{tdHsakC9E4{jYo*wt_Fe2S2_HoBccN7xM>p_{t&J)nLcQDMBCQryo}q zg{~(RUnTq9*N0gjN8vp1WB#JI+pSl@ zXa68%l4P6f#Lc+sEOePV9M_NLY?ZLxZNr>s9QH+oaw*5dbeu(=(+V8>h`T5Wx2N3*xRz1G{@yw%&$ z=nK@=d)qr|CAP!Y+TIrMcLbW|7i?|uwRoGGwzlRkY|F3xkHuhs+Wf+T+JXhO`FVMH z^Kn;OP_Q6xQU1ckwfVJqkWW2}lGh1$@o-gttKs?>!+% zgRic0YwebRx5X!{V+DBUnjn~F^|Ukuy#7W$>HLAVj<$|2KbjM+t+l&vNKJGli9Hxj;8Euna23wOF-l7kPxt zL~ee$pN8B|L+-Y;FplfrPeVRu*}G}kBI{TIx1NUF{recTNUy;LYi`??E$zOJJXTxV z`b2FMvvTRmT7jv}!&a@y+s>nR6i+RYzuZ$f+07!;OdxI8EnCYH4o|d#*`IOv*b(pz z*sR<5j@^uUiocGlW+j+*{U$w5hw7ke&g8e_vQAD9b5+YrQ*A@c#JA&}VlFkONn@$&+n!O!OZLRK(wzlT> z*>22ipSRuT&YwSb!EE8jGDrEE^6d%k;)H&EZ`Zg0VXz#^|3LldubzJkv`EWB=KdjYr4 zb1?FRJi1R8_qb*DjM)9H9Ou{?Q<;5>9d<$6MRw8m12j5=2jXPtN;@!ojP06lQ>@zSvjN(d$tuk<10p2F3 zGC#U^!4a$Qvb#V7gW-?A>{Eo7*|OF$dy0dRcX!)xzfi)r<7le|udP6!&^UDcSYEHo z>qoej5(0(%V&rUNtY@bdVZQ{D3^bU=vbu!6_$0}3vJ*QJBm4j^1r&8oHkS_RQ~`Zz zfThyl{;DfB_VaAkv)>+JR{$WU8=P$SzC=48cVl)cJK<(O>Jl=rr+`O)JwZ$)38L|r zcIj+0@uGLz1V0a;f-()S^G2#eJ(#3N=})Dx^uYLp@znpop(J*m=ozO^ zB6{F+gZA-s)+zcYdOpnrJ^F_}Y&4p`>I2SyF61-NgP$~MFYtr@h@R^&qeCNpP}fO5 ze^uB<{3Zn_z>3_y2?{Uh!;J8*fO7)zn|dyi#0Jt>;aAZ=>RjG8cz@0`)ckem5B)#2 zk44BHb_eV^SjisRNwicT-=#G6m9DW7ZVtldI-wtZ?7Eto8ZPe}*^qalPQDazKtJd| zH#l)3`P*2oe^F;=U(%I+%Cf#8%?o>?f7rvxot*3eehLF@`~*I}sQ2tlh_DX;ARmDL zWdv*y^2o@1fnVCCu`9%YCj4W0Ku@D8(>~_>xB|l0ZNGv((7ztAF(=_k31r{kG(LQ! zAN&ud*;C1%u~7OqLqEW01aqg+x(WZ{?Q?@L$54$nk>nf5W@TSL0xT7bf05`PnRYc2 z(PH#7out2rly7MlmI<2J(w|NIz`qIuk37QdrHSzh1Jh@6d8qxFG|bQ05q1vkX+Gsm zL48()U6S>>%i6Ty?3py)Y~m00+1=%s&F9-gQ&QPqI@z?~tm%ZG#qsml-YzzaRGat^ z7M-CY(Az|}Z}21B|4d1s@uqP9IYjoH z67{b^r&bu4oDlU_tOwJAlP42>V$9zrXHuQ}+a9!cp}q3ARixj^E|b4GfOiXf9-qST z4%9_^@<%)H(%<0U(C@UUewDq60kZeoq;I-Mk@}|b?8-j}OQ4?)ftT}nr2ZvZwukaS;KMoiyztrmUL{5I&D&({pD?+mFw_tR4egC-s+t;i7nZMsN)3Mm$YlBR&CjBR&!7mJy$X zdJJz_&-nUgcgc7%s()89tk>K>{F`Zi0{_@Ylaq?_56sVVsFHnQf8_g_<|Ap&(C?Bp4*e=1oT-LoUGGw%SeA`|2wT;lM~sM2+wZ@r@+6GGo$`BI1l@QoJ@$1 z0{S7^Z#cOBBuDorSvgVq3$)*GBy)R_|M2~Zo6~0zJ@C1~WJlDWkl&^U+*xk)9wB=2 zr(F)dK4QEyqK7|a8vQLdnB_LMFWlvJ(|qIf+#hWYSV#CH#;5jCfBZ0wJsajo7C9G5 zBmN3|Krh-SQNDwG+Sb1pyz=!e&4v8T9qkYDrw69!lRzJFVLlI4rltq7^#`dw5fcdG zW4xR!)cJlB{lUJO@1;ooL|KScA%I3KY7 znEV}h(|iEl6<^%MT8#Mbr{Cx`9m*+d+HzR1650IDeQUAZBQ9cF#LZSHh0=|TZ z{EN9!C;gtEhdR;)>X78QNc%TOLQKEpAIS+8{btfq<*yU*0Q^E5YJOH?ekY&~g-7Rq zVtNdp27D6gM!XBB9;!Xz(@{6#GscQPihJSTl8 zVmo4t=ZT5bzl!GxK{1dcco%;^B{YC%m4(I6nk< z9rIK-mWMRF=aw6hA^k-Vnu|AvDZ`8A}UPtNC9_vwTzcr|t z*K5pI!bjKZQS%i(zb*L+ZN{kxf8_fKWq;Tc=Z~2afsME@-_LxC`3m_%+do_&ES|5pPso{wbF`pV-a`m^LCh?k~( z#DVdqy20-!$Valof=}^zM1FMl<|E`k|6k@KvrX}9;d~wbb0_gCy8p)dX5Mel!{-a5?YDPvan$<6<1-zp(dR`u6i*)D^L^rY zNI971GQ}sXpQiow7K%@ae1Fd48S)@S?i?T6pFhI)=luMX=i6`01N}t9e*6$VZ?I3~`*H3ce@^@H*m)1e z!~Q9Deueda7(c!teoT6h@4wW3d}Mqxo&V$pADm=~Z?W?q>;4w= z_ZIfslZ^4LFgQ+U_y9}n4{3g6jvEgPiT!sr_TO3K=<_+l{(Ca*7Y*^wNgI;b{@dvs zW&e!>6)JZU?<((L|DBH25)6v{D%!hHkM2j2uZ=UFHvykvJdYy(&!l`)>_7Q@$c*kk z`Tm>y2_+iuCQf8-)G1%RFc124=>~nm{^`0!pQioD==tj(aK2wkFz2txZ_D=GUVKA- zWcYqS8e2(+O}rTLSy~e0emcK{MJOKfH>^MSyqKT&it`M^dH699-NJeJah!*PvODp8 z9IG^@S@zdyI`;eQYarnMjQ!_L`NQYo@CR|eo<*OZ^3Nl6`oKJq>fm>!Wxvh$Yf<@} zjO63zizK92U+v@|y!}u*^-mwS$o{n0`G4N6$!k78NMm0FKVm=Kf&h* zru^hHs6(pz<-P{)Yv8^H?rY$_20nES(BU~kZ9l%{z`B2a{r4aKDDoFC^;Z3M)8GF6 z&nm|JRl`RY|2lH0kj?0Qy2n-H?wyr$ytkLJK6}k`!|(rVZRBelm9jeFFMD%(dV5pY z*&o01``<<0wzH)*tEV2 z>?v1ygf8NNI#$>4Kg-~Kj5FnT*xoX>M_YnGuGKNn$BLfgOw?F405;7d67ML|gF9_Z zBHB|1plP#te`>r}6YQuLrxqvDt0>iD*UC0c6uf{MEfk!_xXOwFDH<_G74`fm@x;ft z`YITqzUU>|ao)yQc|$W6!|zG4;wjzH7ds2Cvd#EB3ExvqL4w$$C4%K>iHDCW))l`! zq)cM<12P&)8~BOV*+3D2zkM zo>WB72Aw8(EdIc$xGl*hdq&t2A~S9`gWmNdq4%iVG*TRVRBl&%d{l0?kUL%4bh6g|h0 z_&dH*Xd}Ff##Sqeb;XGBbR!L_e&eq|vG#j(tIRILxbbWo^-R#k!z(LLKO?0!j66b{ zXkgTKVm!|D)N1EdzWKE!X;*s~!^2K^q2W*vFYr$t8V)eNtm^nii>dw;>WWFC$DlgZ zVKfncP^VkI^Z^exO!RMC>Weiq-eaj(S?V+%_0N|G^uIxzXcLH!f73d7#O=2Er1V9p zrS{wHBVL7V8Gfwp0=$tpZB^^XL_6%WaU2|y~+cmj{QhW&1;vr{uJ=aEMo1l zS=KK8ZDGUOWu|}I(ta_rW@CRIOTEfcN49$7wad8L3*Q?pzTGSukmE=%`98DjQ&8mr z6|YxHNaMt&M&es)na9`~Oe zH&%O&RyWSD`!QUsA*1Q+lj4m!zaPWRtGI|2Dn5*iJG94)f0AZd-F55gPw~RAlhd@T z$YRt^PDWfcuE|1wkhnx6$6}K$;pW`$Nw~ic{lm=Fc&q#E&i^})2A{_0R&L|Sp_us8} zqwk#=`PCVp; zjH*hbl(+KypSaKV+2m$VKcDl$cb^~M!oD}_H^1Fr`>W}*r{?|Blph~TOD%6snTH`i z2habhg4AyOQ(ayz>`?)p@cf^bmoc`v4F6k>*NvY4Lkc2W&^6w(eF**wdPA|{Bq;jT zI@FbR9|N2p%yX(6zs=-*@S`CU(Gm_H*@Jja7k6tDy?9Pnh6zTMr!b{#Og!nuh!oH1 ziuI%XMLnlW(V{so;yGQJ3LL#gpVK8Fiattkhc`6S`7xcPy+_vTqaP>>Kx}0Yb=m~P zYyV@}?n@TcTHE8FAuVTIqTl88PnswW*?`tN8cVUGG(`5W_DSuk|kLOqA{%ai#V|(=Dpecl^r$ z@3SNdkvEIwl?IiC|D&s+PT=pt>xd`n-&ACKj|pgTGT>-^A^wXkba;e6FF!SO9M67! zI41tTj_fBrUbH*+$Ktx+A#u&Zh2%F8q5h`;()cJQ)cB18NF5uCq>X~sqWzvY<9|iA z&y|!kJ{61lm`}F%SjImt+bjR2_TVK(B5Mi zzhAaj{-?%Q?G*e$0OGZOJ5GC=64v(gV^N4Yewf*$|0}Y+WIuYb>VFN||K~XJtM*r9 zOSmbwzjb`_3DW82I!S1`WXb{6RkbJZownmqT+UC+aMAgy=B;r&;#JKHYdclt z;~+Vo6rH-N`K7L^-PWQt>)MN!7Ku`_vbdnRvwnVY^TzpWYPJ*00Ok zw4q{!cYWdZ=8DRiH9N|dH7|R-rqN%$rg?cm&Ew@e$~P2LH*acLmS4AAv@LF4(p4fEG+ zyCu4U{L$!^ZE6WNZkk^SeYBL3%v&~*?(*A83%0KI7O%?R*s^u$wxa5;jq3{oamOs* zv0)?lt}9#iWPMj*6Xb1%uAZzbDa`Y2DE8O2_+X!P6?LtvpIk{cSiY{wTeN0naqFtQ z`j&N#b(^Z&N;Z0HYW&qpnoAnnO4o0#@)nn@TvoO!e^c4g&W$b2otw(=n!mDh)A9;` zL(94@=&yWr{<K zC%PXf9ImI;y@hTv;O7+%*HU#~>U2G)f74EQ{*S*BPW|#?z~>r*(F_Zo=t~Xw=&vHJ zaK1c-26$eqv_#i@3v{Y`>-7b+wX^rtk%S2?9~@_b2|1LqV<=)l6JI?Lzy@(yV<$L| zJRG3n16ATZ?|Cin8ja$Vc+Z&@^gJStXkjyqtB`^|#*Y~;cR~t?tDraTyaloimkh(s zP02-JlHkh!g}xa37UWVe(SvxWFO?I$M*@}KhoR7T^eb-w|2^8rqo@5%Jous;z<&TZ z@pG2jfiX3&v>Wh$6p0tnS4k7RCgq?pE>Z_8^mnB@yr`KdZ{nCI6z8>&&H7I&ojXD)P#h1f3%la7s zX5Yq@=<_fRl~dxv-olmOS0&so;Yz-7=ui0$p~g%61PeS*!Y2W)+7jrC68<3I#x3J6 z1KgM@9+z+$7pIBQazftGxJG!Xl#}{byb0YT#c*X zM-24FagR#4b=+S_xOLoAId5rP<9u;R_=x!;;na`fg#_4afXgEuF4qwQJo?W)z$}o1 zDS6$Tnz1h!;6}L`C0yx2(Kk!DRo)jQ+$wLW3^{7tJUQ+P1Kc=nP{P%?ihh@bTgUyL zgj>g@yoCzoE;OZ-?kKKNQE(5hG1e%r3a;YDQ43tfhkgrO#epFUTrIuo{8RO>*83UY zkBWkKOZzRiz}0&1x4_kU&qUhYUR&4H>J4-;>=7EgE4-bp^^K47pPXEZ2QuT=9rL+}GcKkGi6t$@? z0J+P29YS3Ea0&g6W!b8lIru%f`t5>h8T7Hn*HP4g-=D1O?C=p((4(OV+TbnqJ<(L} zGr@)Se2wzw?nY;E3&x|Luc&RMpT}FVV%^HpCU3K^zGG!uLuWIFzs21jw?T88x1qM9 zt+s(PCIkX|f=^KuvKS>?))r6%w>uWgZMX%Sc>`P9Yuo+4`jNjY8fa_qc6ixb{w^F+ zSZM_$Ddi|ZGPl0HBY)v7YvbF>IV<@FQ4 z??Diesg|j)t1>~&lKF>1%4J$r8SV24Cewl_+#~PK$~dMFDqgAjblgzN$|Hy(+e5?M_A-vTlgj*3%6y~dycAIQJLVnr)SxW;e|32X3 q+0P;U=ZYMj>_@Jk+N&ITKL)mrO-;_V(>s26T*^;u(38BZ?f)Nue;Mrn literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k64480700cb22a21c82f517aa00c1365a.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k64480700cb22a21c82f517aa00c1365a.o new file mode 100644 index 0000000000000000000000000000000000000000..c2a995dd669d25977d8030c2887acaad200bfee5 GIT binary patch literal 86568 zcmeFa4SZD9oi~2wCCSWWl1zrY1IZ9VNJ0o95D19zCa8ddMDPWzHVJPAc}oY-i^e(( zihw8~wpD7ovVGYSDwgf~>5_JLO>6(wwXIS<&wsa{-RIv@wKhxAFd8}zV~~6=l%C`&fI%*|5Zz`b~+r)JRIyFSsa;H8LL4Q|=lNdYns;lJKYxunGDyjM{&TruJJA8*|j+vIuAKv>>JcI?PDb_i*|z{auqXG^rc_ZG5W{MyM5P4Y)poN?tHd)1Xg zd3}KeS^d8|wY~w!IUTNFw4&@v|D&gm9oT<*8qRM%y#MsyI<;Y>?Z)TOhI7vh9{btx z!HsV{Gx#*lmCIfkyz{M}4^G2*6V6Kx{$cQ%XU+~@i}O;PeN*Q4{_9&0pT6^rhfn*b z&*|N8{OQ4C8Q&RP^779I&;9;qgS`j-FnH|gKOkLwZ||dzp1$+=fz$h@KRtNO%f|;d zY0{FmobE;0=YIP1;0Ba=ZrRI&OHkIa>6GTO z-m_2dKfQm`qo*r>etdBMBhL)(%lye;Ro2UcXCHoRuc1QGKJ&9d-=d!lZa|&SZF~c1 z8hWX(u6gE>)B7r(9;`ZmvTA-dcumDm2mf{84}<%U4GelK7xbRJcF*amdJv%)S+sZ+<)M-BohZn;Kc+{b7kty$56Fa7I?F z#vIFaXv1e*xyOiWbFzACK7cdkLoVi z#>neA%$s+YFYJBq=4Q-8ZTQ1`e|%J)&&lTJyDmqSIY((ensZUZT>MAO)k|~n(=`_( zqs+x4W6Z@Pwz)`i)SQp~w)tqz-_hpe*~xvI&n}$#ix2O0^nS3_+56!_cIfQQmG1ZN zaP5&!dHkncJOJ#G)&D|6LnC`45r4f7+-GSGrl& zyROt@|K)N+ci6Feic^QbcbPWaR+ReThP>1dJ6>@09sRX7yy3f!;oh$|^>zHpF?{qN zn))_e=j!WyMH}vT#?|LxyH;%Ynqyd7KHxt37}C^0<|p8Kkal-dUt4hZyb0xCi)t;H!b((fT^}0N)MV2>cpw82H;bug7^MuGirDlfWy0uLUj#ei(QT z@Iv4`;Cq`x!TC=(AH(^NIR6dK@8JAp zoZrUzQJjB|a}?*_;d~d)E72ccLOZ76d>76?MZ0doc@6kd@U1u(x;Qq-=ET1AK6yO9cq8$ zN0+y~`lHL+oq^-+Z{hnn`kr;X{XKjir0+dsnh_vw4aagy^Kfz1y^RupQH6&s#O z`7pF>!2M!LB)Ic?DZ_2QieAnZA9Wvv?+=|hGKJm#+;7{yoAM#mvCTQ{(ApnOKeX;^ z(+{nWPCsgk6jp?+o^=i)oBBkD)>d`Z^$3%-A!zNa5=e*xcr zMBjbK+h4}_m*~5f*K;5v61?Zulwtj|DZ?ACOZkxM#ZDcWa?eeu=L)k<6{u4c>Qsd~ zRiRE*R-aK_sIRCl)PK-*?#p<6_MogNmE}3!{wTihr|)UU+rNbGJ@h^Gc>CYr`pidb-aB&zOSb5&g1P{@cmBu?l8s@{@VM*!z->u-@k;< zhW+jjyL#Ni%+Y`O26iOe@oM1E#yW4`o%7v&cQ@nHHcpEBmex*D4|7rfXCV@?3(m&l_1$G3nTo=?>Oc z;$?;(-j&K@!m}>tFDhXZuW(?VI$Mq}VQe^*!j88)mmd$MIB>oWXSB(=3}@GIHT}}# zOWbDqYeo8NMEWI3(l0)~#ABvkB+_3k(qEM%{gua;_|5bSMfzruzDcBKhpL*`FIrLm zjdPm&t_iT=X^5FN<~0pZLmYJ$w%+1bfX@+pc@lm-_yWP#B=QZ8l9QC*H51sdm1l9& z@HHXG*vl%#^|B<_Z$cRr{5sFe$|sEL)uQYz-!<|A6WH%0oSh-a-U5i_ix}A3x0hP{!8GOCdv1W=02YzFvQ1l zOA@({!0!_LJxTbyg^@?T5BKST0@|85fg1(F|U$p0z{ ze^=A6Zx?v`youtvBPa3tqm(|$^)FNU+{E;ML+O)TKSt^E64O6J>62Xl9;MGuO#gjK zpXB<#Q2K(z^gp8XNv^*@=?fFnzeMSiTz{F;7bT`YP3e-4d6Jonvh>bl+<3(B@WYQt) zM@*d2dls?Z0AkeN-Ou{oxQq?IgBbMP^rpW5LQMMJZf*FtTM;KbmeTu2#HvGo()xxG zzy2oOSNT5T*$>{nuJ^;QH}wrn^;VvJMH?PK{Pxj*YQw*IO&fmor%ipYA^v*(wWhw` zA};&w9~r(oaE%SW@lI3U8}B&!Uj3<~?{|pd&W}g=F*f{v)-?4EK1}j#_#9%r-;ZnV z`{+N}@SBJO|1i0^@58S<`rbm!`1Zo)z7O8c?EOvpHI=VDc2)2DKVhh+hWcy6!@H5^ z{aWAqcU;pu^e5K$!PX_{m&LsU-*WX1%xdmC>uT=%<8y3ysP)?3_ja@4|Jvs0d++9@ zz3-+w`ri3RHvI1LWxa1NbfO#w%5?U4|J;~=DZMdQ z<}W!01+o4H^i&-e2$X~O~J{VSK&cVc((kfW)o(&N>zewsK`;l;JI1(in^ zmYl3Wnxn1>LyP%&;?R;wO_jaRPdIt>`SB;0Abkg2|JMB84#@l~K1UZ8o!m>;b{7uq z<+$+VQH~3Sj&fXZ@nXdc*8^@E;pcNtddjZm zyobx@@H!p)&g|Z!yT|dm9$h%@q&Kvvw+iVik*0TH@TB)=+VHX6!J&{t`)4n%lWgzq z?4hFzvytCop5L+EfgvBRqgWrVB3vuPwFaE$;_NBA zt~U?g>+$iFE$_|2c`Ejw*a-*~*B_uZ%R zdynk??9kWJx+?!J>%QJ^xNaNz=EByK-+KPGlOK(D_Wl0bCB1+6hvMG1e=xE4|2#jT z_q~75>-{72dI)+tywcJ4{UIaW27;D^thk$Peb^&7zJa9ko8sK*r#t7%%x5Sf4J;?T@+9mE@XDcqMTBoS zIk21X4MPK00K0A+8i0L2`uvS2QBL1c*G)qMvw(NwJcIa~PY$fa^@TWJOL8|44NS)M zg{w|N4tUkjz<9#9oE(@&_?Dr8afEL@iR-|(4h^JJKA#yHNW=O0&zu}^8SCJV(}rDu zu47;ED(L=|@Ieb<51L^YnqZ$_P;AmAdvVELT(TFJ?8R920`U<)+VC&pApW6$TK;h3 z_Z=+2`t#{3V@~F9&<`k!1*U6@@$n$>Mw~T#((uV<0Yd=a{axix~LUnPtG8y(^*!{-uh)pKpWdCB+Q6#I)Gm1NrCF;46rE08! zrJzPdl(J@9$F}X^?d_}a%{=eiY+!zF-PW;Xt9n`Zp*f<;brZe z+wR_Q&pn&gS$L&^x6Q0;Slv3Sb?x?@o$J={Ohjg@+qbQ2UAt`)hz(orxN*al?G3Hl zH@CO0-x}WBzTL{_%J%K;SMy8Fx7=`5YhCN=t$OR)4Vx+BR{B^eZ8d1EMLUgleF~bf zt)o4>ZtbV1A>nn~)`jmvi!e&zb(=mFvkmQN|5P+&?Yh-F?r2>fZr{Al7#;hC_8R31 z)ND=rrcHN*+dDhB(shQnZr@58&(WH#TefX)-?F{-Q!stkw0CY^7hdzJ7-kF`Z;O2_ z`9)zm`2flnJ_TcFHbx23ke^^P6w;kDuR4VzlG!`HR0 z*}AE9^*That!vl8|7~5fZfz?W3 z9hWuK~8b2Nur3DSjK0Q)hI zUr)7PX1DX}841__-bwjnC0zdrt~*%&RPClkw^ve#CAn0Jp8MXrrMr~z-FJh8*sspb%#V^%5Q;4 zN8jn7aw5@4g}oS06vgHd6F{#UM-N?NrM!+4#I+7_Hp?}~iY`=ZW3uDfZ;l}nmJ3%7_v} z=!WIXm#%ER>8cegmn^@o^{Q1jTy@=*S6#`bFAvQMO>Ymc=?HDx+&Z&%ZhHttQ(bA_ z4ZlIy>bjBb!FBfx55xUa>}mR*?O~WY{Gjiq=FxXbLx(#U|9&Ccf9~~e%tgq}bsOd7 zQ;is_WAD1y`E2GEcqTiq;99Ej`AHT(gPl=u4f~~nr?XQEu4XSPcpB?ha1}eL;7WEv z!4>SNf~T@03NB|)D0m9%R&W{HtKi8jqTo`dD|iy?RB(uOD7b{RDY%%eQt(99qTmT^ zv4Y1lO~FO17P!gk-$E8rZ~+S_IG;?4ls=E0cgi@Iol$TOJEh=p>_y=Ccn8@@h0kV3 z6&zqkl=NAwTj4WVM8O%XQ^9`Lrr>neqF^6e3>>eImubN9`Q~8_3Ql9SO8Qh50*?1z z3JWM0wphU~7ICoi4yQTaVJj5uU=b$sTuCVXayQpWM-v911wz;3JU+&pOI^&>>T=hWAz_BJUG=<}CfuTtpFBVu6_z8hq z1U@2go4`i}?iBchz_sIy{91s04mZ=77h!Sf3Ud}fM)3j{t@Y2c8+{nHIxA@GTr2CfzONWFm@ z1nx#aNA=YNj$Cfw#U_urZ_YoTql}%o(!fUj+^l7>fsOjPSwEiWq4Y-m+^lxFfsOjP z*^w0nHtOeQ5zKo^e+1<@Fu#uod=xkp{l$(LJd$u6__l#tc>Pemz^erA7Pw8|Qv!Dg z9DuW>@;Xgh} z30y1kKP7O3z@hIL={12n1zs$0zraTQ%2?opk>04Eo3#mS)X&XM2yE03{fW4m>TlE! z{U@+dKg0_S`OG~b#JXXVYB5(M_^_Lh9v=;#HrS{0Wak2Y{$nS_Tl8@kBX1n8!XcO2 z=V6)n9&zFO9C3Xkey-e%&mEK+b-+RAkHq;yasEJ@XB){F%ma^q%vkWpB!t7fZpNfn z8keo>0$RVOn(58!M@C3*@}JZ>daonKevRBb_`I6M3U`W}C*fR^$)4ZoH7@MMX>}%x z>PTpG8DlSJfv29t0ljBm%VdYb9xr7T>wh$hy*Pt)>nWZV%#GNYSQh2qt*3g9O1Mx@ zjggyS^gHjXCG4=C9;2?!%!9JMxSaXJER_bLvt`f)dG}#`Svtkfu~gI;`T2F0N`2Sa z@~(^hTNYbX_$&||5f?j`#h$*~A7dK;eMn!L$*w~C$R{u-c*xJt`7j&lfgjfc{s<7S z@9QP3Zzm6vam9`D9N>fW_^QGGEQ|O6NWr_n`~4AL=m8e{4f^-REc9Q7KOMB$KZn6nf0{Vj6)=a7 z=l8-kkr$n2XR`0^hK5D}!1TJ&4=XS;dj4&VH5>{a9AB_j* zW0Q$#zMi|EEvnp33=U&{+vl$$|4Zn@zNJQTYkv96@#pzRb;J-N1(i2*y9*Nu^Mxi2 zuv1S#SL2|IuV+EO)4ZhqT)(0JukCO{wQzXwUs)z|b~%mwvuM7Suths^40#qwXK!Y~ zez=wLZ%6yql04P#JtIGEUPjFCO%K`PSZp`XuQ2R#kve03Q~f-JVRwp!QU5aC=?oFi z9!;bAAJ~cg2EkTwVtc(P4;vY1vxmLmfW30ZBAbz#%7?vG#uM%7)^lK9Xugc%YYfk@g`Ge>m zamAS|zlryn=}`-ew;eAkJR!_ilKJ}|S!~%pQ{Ld8z8mJ>wBP8D!@AEa+qY;}K3isp z@cKo&#@TSTg1hylJ_d&q_eXiYh=PMYTJyyDfG;B9Xjg_UKfjMAkC{H&m15)FzKFzQ zyfSB?zMUGzi}b4tu8-=^^(Bj)+`&@0#zv^WCv$z`4n*javrFvWiS$nR2RHyr-ih-0 z2uu0#{CR#Zo&=GS38B~uy?$06OFNfnCK z!vDXJ#d>t?jAF)ed-3W);0$-9L}1Rl4S)Lq(xX49|1zjO%$RRIdZyuTEk4V}!(U^n z#W=ql^4T^X{u&0s;K`qkv+;TUh{ewUpKs#}{7^R`PxULb@kM^jREZxS=YMRBXS_eD zKPLETfQFA=lUSzG5&r&|2a1L14~Mg-l#v{ z$K~?I{KFDc@QL$}+Ap`4aD4t6`Ni8~rrzUxi+ex6s^nE&aD{fKtu znFg2Jw{AVd`V#clcz%beSwF--sJ@(@RwVnugF!wBMxBuPx1P^n|^_z%$SovY}M^~{_KPtZ@ALhWw5Ag!Z zFU!$zL6ndBh8PWi)&7u&^~cNa)+c!i%=%+|on5J{42fA5)-#x2-Fh`P(&hTpDCI}H zrt{x_%>1eSehrA1?|}S_d@4NN{&F9;Z`OEEwT(}ur$VYvw_c0LLN32fDZg8vWy~j$ zAGN>0D&Ix%YAN4bu-aeh*)wtnyvpFK6>mI1ZtI zSLDN7OYLp;MW9h;`=dI%v(-MDe^+{F`##QNL($@wBmbaMezfaqrM}(zqI|gXxO~?_ zTYDFKSWwEp>l&NoQixV{(h%_vbGjYpZp z0~gmy81o~Ok-|yoDU?VC3q)gnpg+5;^`e+>mwPl*-ms6Ua(%mX z%y$?hYkqZIq0EnH*H!Ft)E9@)-+2sM)&h(AX3FztzHRL0m$^rw$+ z7EApp_aCLdQti)7MiWFzuSqz=PnxfG`$PS|lqWX(%N`#zFf3>3K!1|`ny2{tZvC=+TA#$-k6E_8sAs+8-XOGU)@>r|6HO1{1%U!gznI68-Ty^~Xu6KMp4857{1-u$RI- znnM9N^Zt?7G-b&hj$-iT~Q$uEZMfQ2IlMI;AGiH# zycg|K{UgP5q+#*)tMOja_LD=F#=p#LuV{a*iOu%wa{Grohp0JhBzvf?r(OzIQMiV~ zY(L*kG%j&_K-w$McYC~;pHEwp@%H5^@ger( zG2?zEL;eBmw|P#Yf5m>_VSTdV?_Ua&H>3mo_bY0jq;K56OY1xMOU&0&{$pkUIhK*V zJ|EvtMSB!^)VD|XrqM59eE){WOG)x0e?k6`=eJiH|B{5?SM2+Z-Flvv*6-%{b?f=w z1_>Y5b7+Cca&oYKLVd9R$FWCX>^ER}AB&|88T&^)SP#pXe(#~*@EPAfNB{0S0YC@( z7yFalI-W_)iN_n=I)3NDg2k*4_Ah*~`$hZtd<-GFp%KTy>yyswlTN=^aqQ;UEpQb3 z0h-1%Gd=cKa{A@`u%C$TrGv}&=;`S)kNq6CiUa5_#B*Xlqu~M_ zs1NpQpg{5T^GNo9;vGwW zpY}Y$JA>@^h3GHzN810f?-zCJ1zw6z&Hn4w3%z9DOiaJ`Qvaq~`t9e~Z|Qddwijjn zE-aKW={F;9MfyEJ`u!{b9l&pqerI@Co=v~^%KDv|en!$q+JBYx4Zo-I`eclt-`G!u z;g#nvHr_Q}0iiz`8EXEjKHK%xKHp(45WdLyPebO4eqy^+=I#0!wCU$KW&TF>M14^8 zv%FZ<)hk3N?E)`-r)3G+3o&rtNYN3X!08FKwAZTXL-&$YJvXWI3-Sk`B|{`Tl0 zMc?iAx?9in(DKHduh8!T3^)!;-wS2^Cw&iA6_CE0`dwaBt>B6YH3DNjh4jUt+Bgq= z&mB?UvH$Uj=sU&tmcIYkvyZpuLiHW`B<-hSzgXFS?beIDP*|(Ku>SID61Mco()UcR zUzwJ^7ttMOl0P4h4YD{MTi>%htROyK+@E*L`X0!T^}9ze%ark9J&V^TYeaol{jXhr zd-SXql)u-9?y{t^tce_0-x9PiT&q&{CIWOsZk*wd) z|M52eUOS?-|pW_lz70dABZ>bbCjI_bo8F=-)F-> z$b6!{sJvZY>X4r}xc{AHi@)bosQNzFrjKPdeOL7B@Jdyt%E&M(FHS~FctnYKm>J;2i zUMsMn?-f-wasIEO@3eno>HE8$U3~a2)W1WYq~8a-bv)EV^UwOdrq~-oesNslT`XZs zpDcY3a{UTg`d*Cgl=Z!&M8;$5`#2Aq5FanD??+^P4`s{x4gVgL@nL-&ug|y<^*w~< z$@(@)(cd0@+&DSEME^cjNpIJW9=#kJ*mC|e(R;GKH`x5UU0G* z8k@e)Q1T~zr|nvE{zAWtL%h4Jc)rBg5wX_q4HdP;vcAu+nxo(a)w2Z_{=KFy&i_^P zy+YP^|3&IM^hx@?o%)Aqvc8ABn2=U~;r<5iQVCo7Wa)b@*RNbl-$|cjeV;T*#$)Sy zsfQKE$BXOxQCZ)s$Ep6eE?3si!+I&NPw9yIUX2B?tdG+b{q509OV#}C{=HsFAJrTA zY~uewKM?Phs`+05O4j!(OnA9`Ro{AadO$(uRsS5-X>`QF^>K<#-)rhr|E}sokKSO@ zch%k`+aFZ@du^TS-|g#>`8Iu5?M<@%K~>-D>Qw)3Uym%X>APxg==Xm5c|y{6@hGyT z?~@FRX6bv?oW_u>?==l&3a*`BF0i5RbqgxuJoH_QVL0$f+^_qj*6+2lzL$B7TXZkX zzmvXUXUy0y!~NEa_Wdt0d2kiKS9I&sl>1e>b=2xiqR%}#-EAW0XZMGyJ+RxCsD6b_-|I?c zeTV(A>9?vsQC(e6B(o=~zRxL@^_}d8jH9~UK8AQ5l4cB-4Bp=srw}&3bj$`P1|0k>Ov;%GCPx>zVThjL%6uzcP<}c~{C4Jxf zY14O2*7qAcJ4oI6{S=>&zDxIia(%~3ZA|^+{=MI(?<|f#3I9&^*eoxZe_y5Wm;8G` z`Mvy-zIT7x^nJ0c@0&cIBX$48^u0#8AG=$h?qTINeSa~o-~9OmEshWCH9UT=vEujX z98b5fJ)W=Te6`@a^(rr|uj2Y%S0npxtT(*3Dg5;5GN0(*7gYD(kNoKQa=yNe&)4Ag201@oq z_t^{7cz%v8p0~&E_V_#T{vI`+Z&)D5?>)MGy>5@+?eRCQ*2Ka4r_%O&+=2yaJl{C# zdY#rYn#MGqBl4s5`xC0Z(}oESE1sVw$M@KuucW=bIKIlFuz=2Bc5N-SP|#{ z?De}o1*0zx>JRyTJlsEbSf4Vb1ho0QR)*!2HYT;H)?w%zYm4(~5og!UZPy-l*5dh{CGdfRSa#@aux zRrK$1y%FpAc>Ymc-M_>Bcpf?&hox`%hQ?d#=lSKc`BinktG}m0-);Ia1MQRj8wO9&AJxA= zA8h;Qs{SX_FEyTR$dmnBR9F2C?hmV0dHe5OiSu37AJ~sP*}o;~a}^x8Y#;3VZH9N@ zgU+%_%qUA2aSBZK#+X zzkdw;{Hi+T{<6Gk_5QM)Pi((xBz;rvFMHRsj@SQ_(zg^3tG4MI9G+?aF&}OBcU9Q# z?=ti&Wd#2PdAJjKJk{TGd-N1Vzo0+L{n^RxPg3>CjprTY{8jzz(d%GdWnSH{?a_UT zKcwe7H7Q=Q|K3dYT-CSPw);KQ_!0Nh!{5l|tMfV8{TQmg&9U9@VYe3r%KmXwFSOO4 zb_2!1_04{N_E`6Olq&nj^gO8CKlHqQo2qX`^6$OycjIOKgFe;yr2DmiXXn+%@82SQ zEHLl)GUCaKQuBT<=qLP}_b;w*1+ssu{^a$o#-?urvc47B?yoAf-Cs49zR~@j@_ZYw z_$$&k)xQO~8QTu>9LZo4%F% zD&qG`LBFc}>it$VdFuUDr3KRcSAP|K3(5Mn*3-`0^9lO5TARM%0j9WrE3n<)RAjrq zX)JxC=PBg*P?(@^a{P$-W79XNi+OPSY~K$V%fHd^%jK*0Cqv(C@upqh67^lxxAG!Y zpX~bVx5b-weM{75Ro|+LRDH7RbEYlc%(CgT8h_LC5URcv6{-3(UXiEg-Be!HXVN$O z{Y@Bf9G1S7*z|3JuQYzY67;RyU!mZt%xdNSrlP!>IR97Cw+dO`zT&x^x91bow>q1? z4axeJXS=_sz;=JpSo%iKC(83ZKSAGA{}!<68{J(k4z6#wJz80xjiqlzHhs6xCtpCV zpIzS)^I-n3cc(P<=kxc;1=X)4AGs-r4@XzIxrwBK#e>c;6!4*NXSEvIwPbLwcIO_a2C` zwb)O@t_Ho|kiXw=Bfiu7S-+N<@Vs=M?sP}?Vt)!r*o!3$*crIn#CTt=?u^kB&`x@P zES_%+>MkdbxBF2(aCX?~qWZ?p9HjR@`m^!ANRH2QjQmoZR{nVYJ5P7HWL${%t5Eus z9OMtaP#}&Xzf4a{gTkRbynpb% zUD{m2fi|-8Oz`Q(^HPGx`)}RUzRs4goBFRFZgCwtJvid0UlhOzE)U>KbR(uM;hvDv^T>07wKIzK6{*~ zKjs^-TLVtvn8q*rLE_y4PtgO}G(QdyerX2VwbKuKfn#E^pB=pp*c0F{s3Sa(3G5D} zh6G=#Ozme;E2! zuXFQn@%315M=;!G`g(mLJ??JdT{~g#Y5}Rfs9(MQfbo2-7|-mkyj*H8#ygAVTTstO z{1TrZK|MEzLFdi+lHE0ducvu>nlD`ie19vBiwdp$IA6@y17`l&cwd9XWB!&Q{?Slg zKK1vjS%K`X&;u$zVS>!3g>(7yln@c)TTo!}uV)3&pE)WnDpGNAv4v4yefauZ!@is2 z5ei8>=HI?t!($tKeRvrAKSmkfn ztLA@z&lg@_jNh)EoAz4ycM#5LvGjrCiZ&~Mj?3Dt{5Wpe8&B_me5fssDL<`E&A(&6 zn%~;}YJP3|Rr${Sad{W=Ygnqv&t9s^*Dh7%7c7m-Q+=+)8vw<4@cJ+AQstL+sq(jV zsq)(osPg&&Rle(hDjzvu$*)f1?>pzVoqiAG`tzut$9sG}E_Vm@gTQbg{d~SbzxEl= zn{&Pdd7O+Ahpqag0IT&uxS_eN^k2pFu<}QJQi0X_Jc;xvR{E#X)bwen$PtNz)*YJL6|=>t~!ZwA%;>3K^v|F0u`ww3-{#`9BF{^Nkv z^xs7KAf6=*#PfU!watFYLP%j~M>fNq;W^5ixi2h*uWtz574H75rrh7sY3b z@b(+BpGd>+zfTV{{J{borwsYB?fzb%`#t3EP`@&L&mIPU&GiBKd7xiBJ;_k}!u$3? zp>b&Re3mB^-k+i3FW?2iGXLR9SakqHGk&%C6*u8%rvWs;_gn!-tAgU+y zmHN{gqZR8z6feU+g~DG-QS*BuRmP3slmpObE5A_qQ8HLKG~S-ZaB4U7QQ{Ah0mGqj zzFt4DpNBWr{OI=5-|R5$TPS?!keoiK@81svU=Qis{-J)Ec;Pf18s&%iao~W&qyM`f zf&$<#P9Z()PbmDzBT{jmO-};t>(O(Ze6avfRY2IJ{ zaWdZ+PV0gFv&ORw<3alHkcjUZ!(aLm?3ctJI3V$?G5qB(!~PgN^rJESm9M~FNZ9X( z{!7?>@KInl%~#Y1^DPtphNTeZ@~L~_A6Tk|(|q)P$21Fn;R~?8>`MY?ckSO#8XgPa zel#xs#V24t*_Vmu`F|+@d&j;)81rfGHiq^^ocyKj_;}K*43@djcgP?3f+>&p1>x@%bob%~R*>QA_0iuCLi0IAT5puN>G=SS3Ka2ywO%*= z-U{@0FJJ%izt4j9VSSO}L~?WezsNhpWc!8nQiEjAQ-gXMa9sb&bniTghrgN!Ec5B} zB_8oZDqo+aS@A{&Y`5i)(4P57FUx1mmU+mhA^_0lS^gutUgm@PWynvKA2&zlNgfVf zn{UZKI9KL_`fQXZ%Rf{r^CXV|O1sRG-#1g{gL*ycBg;RGA7tX~BY6Z!+H6byky$by z)aRhSviw2g{zJq5QU9gF0c-V^{GrQL`MGG1EdLlDU{uS8gVp9(@?X3{=7V}I+AGUv z@^DkEPkH$dLVm6#|2PX_{+N7FpNaku^3WHm??XsmOX)G+f;!#LDR^Mw;cv8=7C#I9 zq4Mw_TAjr&K!2$DQ~Wo};x9*ksPYHFFR=JV^oPnH0)M&1UxEHm^M4F{qu>km0RFxy z4vHs4{b5hED=c}$%M`Ko8~#10ALjc9mcPRL3$1vC`=iG*&mgPEhYbG{)So;INbgVM z@d)A>zY)(o!27Qd@5{>v?-%7G|HI_Z82LM_e}lTu7bAJA{3oBZc-WhIy#J2!=kX22 zJ72^5@Jx*QAYS47h3wS3VtwIzoa})apDfnD>bK$@q%TAH0q{N}p27P2$tNYu%C@JH z`847k#4{d@w^1G@C68~sUUU}@P52w%JQe$Kf1Av^GgN%pWhzcfQ*lPRis#Rlabwu) zgFUhQ5#og`iHALRU&h-n?0qPlK3n1uA7p2%I8d+R2Om^%aIT88#;JJD92M`QKk@3&z^0?zZ~z`1N&g`KPEmk75Z+ar}ZD%gSft>rJa{>V>mM%3uqP) z@jzoZYYyx`3tD()9mU5Dgn9n8wXm0LwrDTn0mO?DC;o^RfBv{JT(~=p6oHiihi=FJrT2tPLFWg_fi1>oy3HsZ|W;}uR?zP(c0{$M9_jOymkc#ax z%imDE04(e&w->&N5|4Pnct4P7AIo%qvBbkac)7iZ#|xPyG9T29_m9No1K5tU$|rd* zHUPAVWS@EY!9tl2>c;y@;_~A6Y|GzzX?>&>Tk^HzRe9rmhH?429GR!`G5%VKB|ke? z=7YNN{=>MuxIa@WAM;%+u;lRv6Q%YAb>sbNarybxGEe0PAzx_8FPJX#LEU&CTU@@e zM&?N#1E>{Q@_S~;d=UGs*7#!nQugP}pU58pl9p%j^u7mG9s#PBZ}HR7UX@3Hsa0D%J^w9uk^chl(=C1m+N;Vpg0CSS z^Qm6n%OX$@?oZ)gsr`Gv&k*tyZ<9USOa2;sNYC(E{vZ8=_0e9kXZ-gF@bLE;s=rU? z{vPkEjPeqX4^vAScMgElc58uzoNLT%P2ItNF4E!F2``3j$>JNW!{60kf9_26A z=Vr+MyZU=`R6#EIbFqkJs|OEX)2$Ysp66n75n`vPA^t*FsS0fLKO$_!?0Yx zi~<$c)~a~?cok2WpyI3|70;fn;@n&n*VU;wJ4eM04Jyv(f3G9npL6D@{J1<7FIb@B z`SVpgy;{YMjVhj1qvAb#RP3E0Vfa_yMA&P|-)5F%ni%@h7!DS~o>~64F+6@e?2UwT zb74OuoSzSSAmN!aq2C6Ez9ar4eU>tY|%*&rg3sjab%7eeJFEITz-%pr79rlXN zwB#?F1^dHzJOq75`uz=Ndc(h`&Amwf-f#N%0ouPz@aGkhy(fPTEbJlvK1G@CtCV=y zhe}|RC;R5FlK5o)e45M$`SU=se4tF`$-a_5uN3yTObK&8SPi)Dc&fx^4HLQl`jTA*^+NU`&AwjRMRZJ z8SPhj1Poe}mH$GtU*&7SHxmziuBZ2xIT#N3FJSBsWs$mw^|lr7EEMGx>UG@T*Cq4! z(AQD@JN)0p`g6s9bNiECNA}8$Cl>1y?fyKAH%GJw@j!Y6>CJco{v7eS?9b=e{CVMQ z%0GQXe?ESW#V7UWm6ejc$NKqV{r9wKs(ga~u2Ols|DG~M^5V!#H{X42LG-r)ryjZfa8b3VNVYN}sE# zxW?wc^P5%P9^ZHuN<93ZuM+lJ^0%4OZ2o+T>d*70!oEoUxM&jWg@lVIL;odQ-2{D> z^5^Rhi4Tq8&nMaZd8y5xW53eLB!50R3-*)cS^n6c4SUCmEPtLcj{JRb+@BYQO#h6( zuY&!6T-Ya8ZOLcl!`{gLe7v>ZA8?U>ubXK3bJrN}3(w;B2aMs*uaxXP`Ey`l{|tYA zmBhoJUj=OP+@D_^=Vv7I=ZoSz*|V#WUX~9mmU;3QO zUxWI}@}b*gp5)1&FR|oHZ(8t0{=CZa*Gbpsw~-|K}wN__Pg z{(QR4pV!#@c|`?N{P~QF_UF}>KTqNQJcRm{O8y-7W(0qpLjJrW?$1*`34gvyviIcA zfrb4u{P`^s4}X3Ou*q|OerufHn#`YnCeD*TzZL0a`M?b_PyPt=L%T)TTln)EWj?5X z2KmYIIXB5X$&)|7)soM}WK4Jc2RFKCf@lE-}1K4ZxjT`%)N{YKPBmY=Xf=1Cs* zPP@U9FJ3A0LH#DwSC$WLl6jIRe}1DSU%FZ5gZj;Ak1SudMdnGK{P|6meEC+H59%#w zuPmS3pWjUK+@D{M_KWnAKW`zP`|}lOzsf^Dwd*Z@CEBm@_vbg?tnvx|{3eyR`}6CsSNQ~g-lFn$f4*|1 z$|v~q6)JD{=bJaHe1bpUr1A;=e2dB_`17q2PyYNCn?L^y?5E^!gEzvSNjUFjn?JuE z_Cn%|SK9pf=KneVJnbU=d790ir+pIsyhF10n!=w z&&hmHzXR=&<;%h{Px9o?*IV-C+hjhdx1qhVd~$z&2g!4P-j4Q*^pZbsBcA*7)o8!U zlRs~__%&$1%9B4|ZROvI_NzSk^EHHyTIKEj{BxgE`2>I7sqzW_Jgo8w{(PIn zlRw`8du{k**bD48Z?yUIb+BiWzs|eE=Fi)0{(OzipMMVeEam^-d42A^NPq6N`E&0D z`STNL=n(Vo)q2m^?}yti*?aQmz{36+{(OhT!=LW}HhJ#P?~3zpJK#_3_ru*C=gFVn zh4iv~K$m&)N950U2zv{EzEkFd`rXJ+me09I=1HFX`CXQL-n}v()OD06%NKOXJjs(k zzuS^8x=-eV`cBkGmY?u>nJ0Pj=ei|de80>G^?OiXSw0kzd6LKc*LGU+rMqN4sNakB z$ns@TnJ0Pj=l59h<-27*sCS{gvV3xXey=58sN7!?)bB%o2zlr)l~;iBx+p#O@1IA1 z2p*Vt=&N?0#ov$qP@w?C;s(c81#Nwmq50!^K&~{n;ZuEzm zKl%5l;0yH!_M+bUtH@-ye4&zn@3d z`{ClhpPSFud+ooEKY;mW{GN{c4QXWBTtN zcpH91@b8;h>;j6fe};Ck)g@%&zOEQN|B`}tCq6YsmI4gqre!tG;jP8;EVte;>n3iTXrZ*zsc4eBhH>d)wT zBiL7e7vuM@bhezw-#5=GSYDlN->0T#^A_ue(@34VPGNI>Y$}mxW7s-r5;1 zjAQgy9`1K1d6J;|Im50v&+~KUkiQW5;e9T7IUxlXpnsSzF|tSIe8KajPPjijKl+|{UQF$8;;FAXR}T?S?~iiw{8qoq z@dW-nJsx&FPw)H5Vh`+uw$p%6qgQ7#N>5ekT>ZW!pU&|IF4_-yU}p&F7aQr1W&*=H zoD%YQKhFa@D}V)GO7Bl8$ff!g@cY+Fz{5J75$VxhPk3U=MDRH9eD4h4;#9JS2B!Ph zC%DNT3GDWSOVYSL@$|&I+#ya!Aio^goifS7#HXZ^{l$3k=Z7bSC-Ud%`6bM!N#Wv{ zD38ytN#O}GQ69&ha8cT1Mcysmw~32fKBbu32TH@S3x8c#92}Q$`^B#Sd%|TNvZpvu zpM$KiZajZJI7X8UJn!H0qW^#!nY4`^#dYd|qCjKG7Ax3WkpX^4l|y6^zFliBfa%SUWJy2Wp==-_YM4j4%$~pAJvh z551oY&$k;q^-mG&w%X?j=lIEA<3RoKx6Z@4 z8QfMH^BeO$FO%X4gEz(}o%LJtG(HsnivH#LPUVU5a!(59#6=pMbpb*~wE2^w0~lD+@8_=EDp-{jQZb& zJEDy89NIwg-_1@L_;r_vXPZ$2cxluh<5|t^Wee8F z^nOWWF#_I*H=??mX+Zoy`UeniSn=xW8a8u#N{q%oMo$0r5_Tw@>ZSY;-|@fqq{NG- zNFMKFBm*JhU5W?kZHPuZI|K17i^so+w+h4D9Z)=Kye|^(gXH~3@#9N)zhxMqAE?e1 z=KFhG)G~?}`TKPW!4u^2FVpicw-Ll)w2#-P0Fdf~cs+>xiRJYX`4i^(qkk58HG(({ zdBoFHJPv3D`ezT`*TE0RqkMiB?i4g#MEns9i}^B($HVNO!zs*=$i#XAujt!mc^k$y zWwIW;Z;|)sEcEBk5Rc;T6LI?oRj2s&$J^348E7>4mr>mOnJIJmz1t zt292}jPWQ{{~p$PW&OKo=UTRDr12OjKaZ#%^mEY;dwv@0yx9B#qvcn@wqI<11*7E` zVEV=87aA?Up^MK?c^|jE|IV+VnX!U!MulKRvk*IGYjc>zf`JG@7Tx|Wc(eitO z?Y-Fi7LS%+gdG?)zcRail#Q+*oveGL{9q5GU1R;dEc^J@vM*e0exvHg(8cFBs($pd zCoZ;rqw2?A_LYmxZ&dwgVMnC=Zo__};c#miJC}v{$Hn(wc>II;-Lmrw&ZRv4^L+l> z(>t*r#SikAJpEoJ{i2;8F#i5K(x3h;b-(bx0X8as>EBB|aH$6_^}wYbxYPredf>0w z1GG1X{f&Nx4d8+2*bK%ZAN*?Sli&VdvA2JC^NrXUKNd;ffUGCH=u&-hqN8)!~6He3@842hr{cHW(!pTLz?MWv`J{Yk+f-z z+eaVgbq~}Hv9)cqZu?=ZKgRkUKG;$dT!}@_k`) zoa3Sn8AnrW*sWrx9lLD2BkE|b@TZRSk;`&I)=Z~dM;i|;}HP3dJmB@dV8 zVl+r{9&#O0wNw^O9i@y2mCC9a{d`w28(8lmOt6he$$55+VjuUGFdcpRg0l#laW5E5OoPR(IuxMc9 zJ_>3<77g0+e-(bL>6`gQ!7^&8p7jZi;f!!ERboquNiMzn!)|H$JjY~Y2~Pgm+kYoxKpSJkhP zZK!EMY~w5Zs4E)h9jh6lyZF(x+UetVZTuOEPlaBqzV8UtTbUOsPkP%#(dwGD$eaIi z&kJa-N=@Y-#6(SgzuSzf%yZmU@b#j{j4908CZ;2uz^$rceQH{T<~6hH=;K_Im-9Xg zw4fKf&eVPfi+KCJ(G+I3SMUfroSrE03{EIH?{cP*ystkwkHCU>h6yUkJJwVqhJBFV zo2ofzRcnvwN9#qhE@Sng*(0N#)!v}idZTX98fR*f-hi9uy@A#w<()pGH`4o)^9XZ1 z)EiI-NnYuVl#A^RvSojv-thC*@Uc(Y8whHsx6s;XGTxbCw3fy`DesMXMQ=;|pVu2? zIsZbv5pNCdp-j}2Xoi!w#@~-_OTs%dd29LDC*^%npXg0V{?F?TveSQ|-pF)PZK*ZU zWUWC!$y<}rkH|DxZ)7>sNgly#a^4#yK7!tq_&={V$SeG1dIRBk2GusJKMO`T$=pbH zW>RahR7lD@{ZZn{3Mb{GerFcR!wO5h+#4)#v38?&wBDdS0P;zr>;+g{q7_Z&W6uyY zV}~+HZ}^>7Zzbh3oK|lotk$mh-0K45%$6! z!%FdM7GilTE*&L)Rnm_^3>Lsr{3-Qtmf>f}PbHPla{9?K zlOKxnat~fe{K}No|A_JHg;-lklVEYN{ljCAk>XCL@Ye9QQrsEu1=Cwo?2;@}i>4U8 zIC9*|E+B4Idtt=5_2c(~KkB5hqP0k}RyZ-l;CalFj7J2=*Fq!4v%V;g-qmt(pGwnUZ8btQr;Iednq}e%6o~ged6+eUN4LopMLCKz;7a?5e(Z`Qrsx%eH!lt z@;^y=raL&Hm<8XK7~Y5L*E!oYG zCjc1#O@6my9-M-Gk0$;b{|_1crcZg&r^&Cm>0{*Ij0|@%d;YaZZ>BTPro8=Z^2yF- zdCkE$%dHR9L&g{QJ#k+oY3>)8n_PqBIa|4Z%=%s`LGFPI`Tl=v9BAh9S^Y8Qj@+)q z`s2zHpYpqS(^6ySRQh$>2{(ruA3j)8^XJFVt#chK;8IX*ti#hLd3&GrVj@}c`|_&zVM9ZZ~3I#pA$RjGDc zLZ-_ZaON0mUx&vrM9W}vC2Ta(*^pURz(UzPuT(TW>ac!e#3R*=_<+i5@f~A>52-w} z__n0-_#smq-3D(jAJsOF{^a>v{Q0E$YZf1J@cP^Gk63(*!Q1MOQpFK5_zGM5Fb>9X z!r*IFUbFZ!25;*>Q<@&xOjf^sD*;+U{co#Z#Ns;*-d4YO`9}=iRzGId@03%%*TX^D zV84SU?#p(evoQgu(eDv(ddx9I`8cL|a4(6#xTURWjfdau!q^RM*4-`<-0dQ2ZQkvY z!1p6F90mB;>!9w!qx#bC@GV?Jl{KQmuNs4I8-wo{gYO)J*Aw`N2lel_Y{M8ltLvvaFU6Duy^Tc*VDm?tFk!H3?Z$EVx8v5(Tz5{b> z07eFGR$vggOeHK3f}%LQal_FSiAE~y#d)GAK987y^|NjdU1Oy@AM?#3ogzQ8{1$=f z^p_~_jL83tXs;%)adOY_U?MQ~G(PTZ4>NrVeW#mJ>7eiCUXeQ}eP8cp)KvFeH`_z! zTSILBS4+R~_g~G~%$}`!=Z~vg-<(=KsrFxse{wL(zhsjScSyVQ=?w>r)v7}YSHY9%8Fi`rQig}mBs__ADmcVC6kNjE6kNxmsNg z_sjr0@h8S=GaRfFpCkCVD?^rbG_98X?5+s0Q&8|D2r|#WiPDSqPaz)-f6@31KBqDf zb6pc*?s=W=Zd8^(SP2`<93FN;;1+>T2~0kM`11m<5*Ray%ae~F9y5<)iWvyE2uwbL zaHqf>0(T2cK7#lY0(T00N?`I4#Ge;f7dU{yp!(MeTq|%y;1+=!1nv}gufW{`YXYAT zxLe>;0xuT$yueQg96&!)`9}n<75J#YEdrk~<>4wQeG9P9;bxt9i45UY0-q@{aGSs_ zlMLJ;aDSuQPw9`KJO}1a-~$661x`hOVN!7(Ukr@5 zmQwl_UO$YFz^ep4BXFC*Ede8ahrs;;cM4paZScClCj>UyTgqyK25+>tl${XRXfOD2 z25+<%yue0#!RHvf(O&QZ*Yfhf=Nf#2z$XOOOdjtur1ma0d4Y}km9g4W~BBrk-)reMvPb5uABLoajTi$ynb|q z^d^6?=Sm2;tmrT<4YPpjqVe&|4C;*Qd}~Cy%GlpK={l`c)#E3)?nHXCy=H_=FN>q= zX8&kH%n$qNV)Uj~^;p2KTW>V8$W^!=48-dbDl!Hj4^^LJr9!MKzR$q*(2>M?w&417 z>3T?{-#$V-5dAwX&9l?rBc%${3?4z=e=l6%EaEVhw0$Y|b)Dk{}DC5HZK%1k=U@iQ;c{3LnJ zR*f!d7>*GC7w@0Ond)wrUt;i`BAIpiija@%w?WzE>6ef_FkhCS`0`63UuGlh@{^1X z>C|lLJ`nOxhz_SQCi`NS--`>#BBA^*Al{%O#Q%-Y5mVkA$Aie2u4y8jc{(ZNO>CBL z%9-PT7lcQX*F^>-PwDLPWJjnApkB%mnP|Ysca1a2lUx%etN}Q=s zad7)AVY*EAisUJOUk;{W3P@ZxOBoAc-z$VVZs*~)Ja^4*d=m2b+K z^LIQhB$Izp$eZ(*$`2-yuZKLmQ77h)(*K%>GeVG&4}1Twg#3d^+TSMR&GExD<>PO! zKj{tiNiDsfI>}+i59ZmFr_>);j%=x^pY|e6eKpT=c~nb;Y`%K?t*cgSTGP4mhD}TEZeClzVr$cqyO*z;DRCROcdp*N5^^h+ zHr;abjW>3#T(s%xYj0eC>-AgjZ0cD5|5tZ4p^aR3e4ILEDaLHWQcCkd2@4IhF`AL> zSY;DzS=Pu}OPiH#eGCh;^CfBQnUThlQl=_9IIVJ?x=}y|stH?y1*? zl0XhUn9@TC6k5okhf@0g&-*|5c~)8}?P=d)jo$D5-v8@;%zJONjHXuoM!g!^#kj6o zSE`X!sH}O#8=yOCHfn2qGhJ!hs&`^`_e(oUS+>!y({6DPmwsjXtWTiN#t%HegjRW`OV zogKAxVAqPB?CxpKDU|&goUEf&I@=8ce5_==?RF0zK(^NM!+P4DEK_Zr9&N>SlydRB&bA756fJDjrq)}fcorJ8_GSAy+n*N?d!qgFZAYU?xFm+8mxGRgOGe_{G(^}8baT~BK# z4JGeR>;r0RNO8}{T*aATpB={s;y$hY`M6-ZpWNnLcfs;Ar&=|0WX$_GkE`4NaKO>S=Wb&m&&;#8ieexLx9wj~casHzn@lIFauVS9w-UcdP=8jVb-@ z1|OgGEQ~d8FhwsPXL*!m)cTWs$+c{`B$^Z%r=F!nb9f798MX?jZlqpIr7TlX3`Mp!l;)cJnqdGUr`OjEo$)NUX;_YK^&`&+L)}m7 z{y=Xz{)v9z1YxJw@*Th9uCK~s{zB%7g%_BKOh#XqS6B7bRAyaQbUCFT^!g6_$Qxeh z=(5hP=XgqI|73kbm&xI`UOskuf#Xf`fl8PwC8Hajx}%ZjjPv^}AIi#xsi$>28g?Bk z4g>|nG{6o!^q{~ET8C~BZRnA2=*>>gH=?neoDmuM7|J#dcEB66Gth0704P(F@{P`)RWzmlkrc6nm?A`xys!7rd-z2E*1^+@_-mq#FLQTk_mzMd?EU%ezD zYWv4=G-*LNSN;&5BVc+6ek};6K78MXWnYSSrugkV{U{dAgK>`Iw@K&LjE_6j$J3OP1o?Qrg^7>PH>U7y;7pQ(*d+Wv zAP42&p+Y`>;VdP_-^9ZjNW88B-;sZt@LXIfNmY{bfNP@bh}qE8+(b z2TUlR+lG1U=a`am+JwK9kkb|9h);p;2>AH@HX!%DfQ$SOrr?*LeKDb(qQCH%G)_r7 zHwaJqdqxeE^!HQ3lX5;MJn8Sh1v#SqFVMI``J%rY0xsU0@%IRrkT2qkG(jOgeh(G! zt_k?_aZUE|I{_E@e-m(#e~tQgQhp@hBL71H7v*1~&V+JA{u1FBPjU9XO8B=E_#Soo zWc=%dC*yyMaMa6I4lE7@8YDQ5_}Mg;c!Z;T3Q1g{OL%fT-XR?2i{tT;faAF}PoE1o zO*L`R5@k@1XwNkYRubpF;4wW81+E+KQ|q`D8ahiHuj8uV70Hu3-z4= zzUy$M=Xl0S}H(%ry~r1+cd8lGcCI_z7VUZjV8%W}d{S~20{3yBQ+J~P&J zLTUEKi8d8((Co~?K5=@X+X*NPuGxc_s7}N}A*}?R$XQX#hp&c_VI4D89%8lcL|akR zbIpF_AjF^9lBW+%%sUqlaK?{rQW@0E4}t7AI{uIJca+y?2YJ%{T?uNCSc zZ2F{?FzQ8y3D=w=u$?6Qgf2ei)Ng>*aMSqaY>5EIu5Sh$HFklNzn z0`mO67#G6Fx8M(#WSj6Ora_Kr+Mx9sF4OTA$=3jU&Gi|8$fVoYSi}R;o7Bhi{EEM$ z!ulv%OpmBO-(T}GuAk@MhA)UEV|XXc>`(HO`9G2VY4!K`ISeV}c2ib*71S_I9AVeL zyfF!VoclwnpY;EX>f=6DOiBB{59O2E?*aW#=^s=X-N2At+Aj#wvfoAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi80x6|-unN)s{U8i|5tVE_Ls|6F4r}UsiLu0SgfX?jAi!5?mhh0 z!%{)%W<50nwz{8ZyKVo6UTFKw48{%**|PiI!RwqYd+xvBdLFNT#XXu%RJ3%xw(AF> z-;QHfKir^)KB%&VuG!e(|7T-;M?2VI&>rc@V1?hy_Vj;P!ruCo&JNE=?Ws9yo7^{K zn@D{f=}BjW|5x|)52Kx~+1{$d*dwQ&9;`fnU~tEKPYx%zQXB zvg5$uQ%^rVm^yz+Xim#_!WZTp5AS>Lr{SlZZ-$#sTnhKCd=GSmq0qsDgWFFW7~D7i zc)0TPiEwk<>2TkHH^U(#d;zcK z_udT8J$W)*;e0QAVg7m0EDOzPePXZ=x~MpR9DSb%hfbUdS2|CJL-Q|%pIS{kFAarG z92lH?@`=F^c)l?I+u?ol-vnRK17z>RD|yZrLKV>Eg@-@`y`?06I~;oV&2Y-rli@k% zFM_X&;c+t`35^hbIe2~aWH>%0EV`GI5*htfRYU<%rlQhrOv$jcngxjLC!l?a$6ZW43{}Mj1MPr4p z!v{2taFhCmbjPFA?kHo0uL73C+lan?2tWCtswDK`hGO`g=jydxuN^b`Z#;kez^2Vr zK1u$kd~X20_X>RTp6}h+zSnaD-#a>n?;W-H9{Hs5zX6N?DSy3@4_@2#!%(DT@>|0X z=!Y--Q}I=|ouys;Rq@r4m&*Eoy_WT-Vt$;R#jXzjP`~;*{CX&xT|HYe>F`j4cKGby z>eq(%vwoUKOdEJ)1@Mt)S%2i8S^uS1^sDcbu>N!Rv8(^GnDw7eV^=REv8ySdEy7$I z-kZ=r{7iDXNj#IF8?<|^bf z%Egk|H4o&u@V90CBap8`$~6L>N5Jz4cqSQ1zKh^<9QyY{CgE7#e{*&_(}+3GDDC=w#%jq!813;>zVW3kM{RS~> z%-9E>NLNjXxT5{XRipJ~?2p|neXb`&;}iW1&C}1vNQpnjwY2G1KCaTZkGV2ES5xMB zLa7BsA?Fv0Lu4DW3E7GIdk1u37FBVDY=!jIpB&HfGci}?dPeEAUUYOrI|bfFLO(~ql)Lf4Xt zuaN!j>BFp#=iof>WB#JITdh~WTl)_hbFs?}=mbS)nnnRWDC-1UftLfF@4(fAR}x<7 z%tgbFDm#3EHeX8zs|F^QusUX*o{FokG23{XrrQ{^kHeBDi_OXGxc#!oYl{>@Xa7sc zB+0hai<`6mSzPrUEYZW3vIKB6g(w$vG<9?aI=%I{Q`hcRQRmmXj;6N8AY0JZ-sxM= z_Sk~X9nEc9+hg^HE?IwULG-4fz18pSWD7cd!OjIjw7Q0_&K7S+N1eB&Wt+FN$rq?= z@OE_8No=RDt)o5Q?+i3AUbL;%*XnI)-qx1Cq&>gx-xq@c>hcR0)h#MqQkS2Xm$w+V zb&D1isSHa1ttilSpzK%0k6M_Pda~~y|cZu+mGgiYj5l5^tN^8-39L3;Ptoq z0u6VCnPD0(i`AAKD3?xdKqKHTu%Qyi?6Vwt*Gyq$TQ1P`3@n3+aV=KuIz%2JGm#r# z?xi93(vUkXEsW#(_tKEhS@v#Pw#YhGz|E&2cm6&`Ez)bT!CKJ1b!&&OGmq8PwLMlB z#jGk@RVOfYdDyBodpmgaj^e2$@|U|RC%aK(nggT_yJc%x#^H%pF#A&uA3qAd5u0@z z-?5ufPjWa;3N7VDz%yL2etL2QK|NTRfm@HH2N^HB0Cyivi^3(CeO=P4p%wji0k`4& z!dfCIxe`eCRKWLcx~8S>>FJrx9FaXevmN}FF@n#tb|UMFXZzOy5N+*e=n#aecpbru z>Z{jRlvWhEOR)dPIf^KBNJ6Ab@Y+fse`gTBSkl$#b<^oWbBnjLxxLNZ+1}pLG0%;e z?eli{-1&Sa{vRo&emw(80(P z^5{NY+~bznGh+8QbDU#qN@ez~cGv}NAGV9W@1xNfJP;>Cm(!VD*6n2J%SF+;jYc=r zb>=qJ>8#3#r#`7}L$|i65vH%iF;ksQv(HK8HYK}KvCqc%Fp4V~w#vY5C3u^b%KYfw z1xKvL%kBaV42D1cvQHOYX3JW~>?saL-rZxv{Spb^fupTfytV;>LgUc&V|l$MuOH!B zN(dD4i_x=T;*%uD$&T+#jPL`v6j0PV*#bJGQw8*?0hUUG z`>QY8*e|kK?|yrPT?T-dZgjFe`x5PZ+)df3?6{l#s9VUuo&+BK^#(DKB#6c@>(<#8 z;zjSV34ZQF1!X2)=bY?tz-~ti78y90#)cNM+@S3~_I4hN=m(S7>xbB^V1n&8Q{LFY z>qFW0!XOU*fM<;GkS{nfl8suzWM~`X%?c*zQTmfCNS%rRW6O(yvUvswA`M}Vb*@h=elgEOy0B3g`o zj+68kk@A&wW0|0dE&bWV5B#ez@Zf{oUYZ!MFfeNlmxtP)PQ(117hz}7p5{~DbkyfY z*d@?ng zCh7v8tnKM`0becfO#<&iUEmXRAau^$X!&4?alpasMa(3fgSk zmz2K+=gc8-V)$9JqV?DKeBt)s^AGJwUv~7*`FY!>muEf4XkCp&_Hz4Y!oJZ0`XW^) zd&564pL6+qrTP4eG*Vd}qk*Yub_4IRY#mS7`)FN#AsjBK1w<*_D3|l|Vn80x#$DX#GpHY!CgaTZKMoi!JZ5Dmg{5I&D&({pD?+mFw_J% zz*#h*^(*I3AivGvZzR9N;%{~yKkX#{DO%6$B7dOurJmMH(O$*3nL!7qcf|b7!SzA@ zc23hr^C!HOug92wD_}oS#Qd|yL$dD~B9QBMVQ~C-vOgam@uECAm)BH0=-q{-U&Sv! z$*0HU%W)!JiqA^#gVHRGmg0K)s7@ zWC8wQH`8g?lUOuiJmhDi<(IUv^-1Oj$lox3<`X}vpniNO?DseTln=mvDE_`o4oU4l zAo-8R;yd&Sc`Wf=H^lesIPn}gFquT^R}W;y;`Pu%ZAHLNk=qd8b)D?X<2&*n_%l8_ zCV%cpqV=^rSqUKQY4$(Tzs(q*?Z0B%#8yG%(|kQkwo$a<{sDOj&)2&Q zKO=bydx7Tz2>(gnZA<3)v4iKw_S~Q&dGvYq5lXb<`McznL_x{I7o9dF)_}pN!BkE7cZ_@+rEH`?O5Iyci;{40s6@en*AUCKAr5#^BwS;5j4yP$V>RB|6kH5pMrm(P<(t5UqVFw#oVZq ze$OpL9q9sfNb+2u{hK2preE@pS)t>O_s2lMa6U_JtsK@Y-m-{Q%2jr!?3mgdj*f5`S$H>R|pAH%le|hChvCZ&Z;#rBVxjsoyT*$rn+hVpuPRQRVSqI@vGvIaEbFCw(VkJ7SFI ziHX#|isuPIF`HNoMnV3tr<3jUQb}09|F9N`Lf%A zqXHBg){9dZf48uQzyAGk?3Wa8U*qR{N+>0tJ+1kfDIbCT4g?qy z6lp)}dB$b%*LB=M@eurtllw7KzC-zpp9H4-gYx&}XPB^%(AVwcGkKqCKI7-yEB*3( zChC8&e1@M-oJ+IcFn>XOyKesSr|>_-x6e9%8J({aZ);GgDv zWg+m!^_kWyYrbMwe-SUTuwG+15%C`SGp*-XpH1sG>RDK?qx5Kx^(eOA8q&<`HRdbf zqwDpU`3j%kmVAXa<5Yw{^8JLeKkSL~$IQvVMqHThr$5Dfh5R9IzA|Hse8oNE#`#JX z&sW^>^A*UG<)-)^%~yhkd}DOJf_UoAismcB(BCJ|N3v*rr}#W7Ke}`C5%Qn^kom|wQ~Vme|EBB$Mee^31eu*6JpaJ?eR|LKTlv>%FOiu~@@VvT3`JSk>>@SBeFq=AD{Kd>M6Gn47_ z7`{HA!{?2=CdT&X5AyvvKR@O9_8SX9KiRM!KZMU4?34L^ocqUrq5XL5ya(f9{}emF z!umgYKR$KZjpN%C?(b9M$2Y`}DO36WOYO%;$2ZgYPi}DP6ia-Io&Q+(x0t^-vEQCz zjBka(2|B|ESYm%j^CNS@L|91dzq7Ib&YD1<&l&dL(`dhFh<8rfki_=iPUjf=Zycyl zxt(}dbsPKdbgY(OQ0!OH-i3N}KZ<;9g7Lfw_zdHD6#0K9<(p#v$>&36bpOfs-{emy z(Rep`GIOI&`Re(F(5Fi`=o9u&*Dd-q?MKGWU;lvf{ZfKCe?@*9QRj^ zA6@vH$e}_uyYH!9SFO8mZqBj3KF0d(wa<>c|Ic-iuW?k$nuNdX%jxawOJQez{L1ft z7kS&x%4*llIHs?uO;}Tl-3UVZPiIqH&!;brYMT7gBCn$wdvFf$R|ud4)yCOVuJ8z5 z!~=D#uHk={!TUI8%I~nfWo)mu41rv$XP}Q2J;#}-u^IqunnxtwQKScV+PFltrwl;T z=JEd2c&{edQ7=v{PNY{+s>iOCZ=Ni80X14EIE`_I6$4TjkEHGW-Nx^n_|UNx}z_3F1o_D;PWJWPc;PzVy~76mZK#eKB`z({PvW~^Wu(= zNAfQU7>L(D&-IDI6CW>fE#RS;cm;pa}VmJ%~`MJu>q@!kskpos@GHW97p2s_)4LT z@-7-%tti$NBgWJ9G^qNGzXHYD@71j`yA0#Tvu)HfK^G6NtU&#YmfA4#C~cyFQQL{} zIMY*WoLBhf*OsJR>17NLJK=?fLqWX2KXqt0!1%JN;~Ooe`jeQsl(MEpUW zZu!y&JlHVNzip{6)y#O0rCx2R(|FWBUn0=|MscD|AU^&L>*Nu)+UAqe7p0ckZ?%tj zZElA5GkirIYqXo`X{?=diM=ugaW4~YMSgxmzlzshO%j?~W41TV zOFbz%FWrIxYx{WGw$gmw>qedFabl2_|9JE&50E$ewaaE%yZE<- z4QrQ~{%uS9rO29%{dp|)YD*p2>h;$y<7zK_Z>;!sqi8^mBfaGN%&t#Il?PO4^$hCS zhC1P?&XuW+OGSo9I{=$D18l6(^c)vrvu6+KSVLgON{_eLu7Gkl*{fUif;#m&19j?( zbWNk?sCsOlVbf`pqSSTvBG+RnH#ZvPB4g<$Uamr%-byau2D=6w>j-6l#0qJIbUZ4H zjg#w{yw{TFQ9nlOixCVXGwaB1kQngyVmVMa_8Je|Fqh?KxK6 zIK%G6aIuDrrL#|pH|qRe3^%XhB37vQFgotg9y9()nq_s@t*bx93&T!M)2<+kQ9C&q zan-mc3;jXj5{(>-O}2y^bH6v?-ahmXGgsrS?zcMsZ$BD*8lzjejo0Vxb7<}T4_!CS z)Wp%Jg;svz(OdRIrhI5r`{>#$Ojo!mqsaX z=J`KypY5~R&7OK8=f&^7FtL?=Z|-k?yV3Snv*yjn`^oelA4*HDXh~U!AwLJt|EY%5 zZv0bSUM}oW37+u$pO=?2wxt~ZTaMTDp8rD%B3sZk-m`rO{!4m8vEd{r`qetrm3AKi zoFB|{s_Vba;eGIBWc?&*_Twqx?lZ zr%TbIIWOWlU6~3Ty+)tYB_WDFN^qMuG}HMpou$1;)*GT9C<{PrWe|1R1jK9qW7+OY z7S&qYGA272;}vYZ2y6*)A;mV6Ki|&dvr01{{cBg+=ljatJ*6&9zw?i^AOTd z&RV9J8eg?j_J0hGZ`5AVQG24Zwx@YcDzLaIW8|19Jvrh^?TMyGRHN_smjT{qNfaV) zmdYy)DhvNdS3{k^--Xu^Pt?Du$o3u+(Bfpk(fUIC7hCA?2!CFFX6QJc{rqrD{C^$U zPkOv)clM9Pwcvhn&BBG`HxZ%!p8%xsQB0_bn*@+LHWo=61+7E-y>Z6>ifo@NDQSEv z7WFZoZ11s*e@wPl{!8u2rLFA?FnOPfGyc~%gBZol-x;%xul%SI?R(MQV;O%ywpaeA z##ikWd@2C(+P@vAJxvK~d-|~`L>)iOY|{T_*oKK2Q zUDfw->M8-ng-*W7EdfkFRW9o!?kq+ts+DWhcgH10Ut<%hz`` zELzr0<27!mY%i|ZSz5n%eO^oXw#w+Z<=a+!iVU9qFasiI)^d`0>t@njetYSnZEL*6tMfOtZY$efRMWj_Lt!B9m=!xWZUW!+ z<;x#$=q_xAye-hxtGYI?sPs3suJ4Ba zD%Rw$f4p%;Wy_`wi(v1X{7r3ZYnwI;du=bS+={;Q8aGyUS?%I2@jt$)x!@7yZ|Yjt z+R_}T+p(lzX~D9_g1n{q^}el3wl4O03zvA8ZOvQeEofL&kpFOTU2{?>u zfS;(4e)achij?2FLk<})xTjUJpadE38#K}G2n9z!DzMxPxPe*eDqh5RytoEM+3Yd zR$8KKxd}Sez4iJc+S=Ls>PW(bmJg0I!Gs)2*)bHcpoy;@2w|_hBeB9{sB8z<-bS@#txP6A%9Ib>Ke$ocKAz?ZB9t zSK4*>e;$b!(N{|oye8$KG2@Nh`rG#6@O_lSO#x>3tmxPa+FA`4uC|*c_Ee5ze;^A@~F~Fn$+yhL398AgU=G2UR z!2mbP)g<9c4~o7;!maYYDB)IlOJ&GWHowC8J* zKX*4ai(4@s{d`4T8~r@q%9ZO^l{R}@d<~te+8euCF#JvK{|Wh)lIi zJzbFr>QtG3D5PAbRh7{`pI|aAh{8Sc?u?9M3Zdeas?Wj=rCc5bkXQdsar2za$LO;) zigz*`M{}^h2av`$=GN5GC4;YrDu$xQSJ!7{d+OU*s#WlAG(uzJD=49fC#w#t+XRK; zSJ7MR|6PuMz4jkj5Q=ZrUpj=h+JtZ`1B$|YwbE{v?MKNk+dpGzVCCNjd_4O(r2kx& k!;}5U6;ykbL+{7H*0HI{*$#Tg506Xvi4A&^m$m)>2SX4VFaQ7m literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k65494c2c19a9814c6b2344a80dc7d835.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k65494c2c19a9814c6b2344a80dc7d835.o new file mode 100644 index 0000000000000000000000000000000000000000..1fe8196e65a59f6ba6c9e67ece2be1f612ca4caa GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Kp1A?+13MuWI#e7@z~6Ymee4D9(`Jp zX&G`NMAk6|gBi0E`%IiA6Hk_wA>-_1#?frfM&9@k$H5HNWV1e*oD(85L;~!=9j^n4 zaqsunqx;^g1_AcOhNYnU-unN)s{U8i|5tVE_LrBhTA^ziQ$=I1uvkq&8O!XA-Fx`0 zhoyqj&3dW_Y;`}+cH90pz0mfV8H^nsvSs(ZgV#A*_S}EP^*mm`!9AKzRJ3%xw(AF> z-;HBeKir^)KB%;XuG!e(|7&A?M?2VI&>rc@V1?hy_Vj;P!ruCo&JNE=?WsO%o7^{K zn@D{f=}BjW|3~-q52Kx~+1{$d*dwQ&9;`TjU~tEKPYx%zQXB zvg5$uQ%^rVm^yz+Xim#_!WZTp5AS>Lr{SlZZ-$#sTnhKCdJlAkq0qsDgWFFW7~D7i zc(~&9iEwk<>2TkHH^UqcgBRvMF}QCH`q}@T@SIkRf9lO}`T0xX z?eEcee;NwS$9QWFqMsAtxvk$0j{skB{#1C*$@Acy`uUU4$o_+aPtAXNFm&of_yS(d z@4XqGd-7zs-1%Pk!u<1~Sr(ep`ov%#bWwi(IQl*j4xKm^u5g|Xhvr`jKedKKoD>k5ap%j1|5LSPpL^`uZXK^*LJ;j%;>-I{P6>u zHdpy1`JeK=0r=i4@XdR^cW3)v&kcO<=or3t)Z%;Olgj@FEdHnb^+rB;ZPyP&k&?-8 z4L_hCzVOe*SKW4&cJWumS4Uo2-v8@$tUnd=!9@z#}Vxk37rzBmctsFTJ8)eW!%=pSzD;{ntgT|9l#|dLfBjO#y8Y z=GySyg#O`wX8pqtXooNU5B=)Jci7?KzfA}Ye~TTy*nl;40_z|7(PY39L&FcS!@qd| zwErzBbaum@M`*5r=fl@_y=t0EVtp6#Tm=6l%P@GQG2bPbAmb&-=!8rpTk<4!^(Qn} zA)iq$mdvhsAkT%rTi!nc`O2kSBj9-iJdc29l9A-Q2tLQ5e=lTm{=F7@7jlfek`Nk! z9PdI7qx>Yp$SaAV5y%5w^k1034qt*!MqWw^F|8-ATn^^BItSw?YpkEvjgKpT7$Vqj z5wpgOec*|7)s%=U+J9U*T3^Qg*v-=CdO|cl(a+F4{d|m+_+wm4n||fvN{#!NE7Nl| zWu7OLy0|Fh{6cYvY(q97J5hh{K(FVZKNsR>AlcKu1O0y!*R^<6;Jydf1Gt{R>wn?? zG_LJev@*!LG{xY~$J0cc6dQOg-d+eJ;Xp$xgq94anvfVDqz!SpN|0M|My? z{r)=EA1-12OdoiJY^v;k`CG6ReED7Y@q4h@zrlVne_)3%AA(&Cc5Iq1^g(|5ab;2H zT2k>9vfn*@nDy}-oCki)U-WjX^{RJk{!wEtcDVtapy*7~D8L70onR~Ra-j1axO(tP z!YiG*XxLF@hcD3PYw2KBz~mBE$IR1Han&_u8*kHe8)NoySn_1CIhh@|Ulw_7kwWO~ ze+ro-*_L{7bM`-qtGyIsn-ZZqg`n{cOL8mX+xj=|k*U;73;_c|D^R~2X^L94* z0(A}Ej?OxX?ew*Ev+13#r;zI|5Tm2X8}K#WJ%R*$9lpS0kO)=^_*(7?Z-X4(yF!pg zUwzlMx~&0kt4~6 zzttCLxGT&I({Nd=w&Xy$baDe40e687l{jXf<L@(7uU z-1u@Y4Y`+w+-YfH9M`{>hJ4Plchj;(*0BO^J`K6^_c3abUWX0Vg7&RjJA9patgf!@ zvAQT`_43tq0#lcVty;6UgGcWuo?0S*xvO%r8%3r$K-#cdww7fao@fQLKj-lAqu?8{ zS-0^WyBYN)hvTHsQeFf+!xihNCr1#}gOwS$^+|xB3**lRs#7ugYd_-x5MYoU$kKHJa;W!78EXH#a+!UjqZ-_j!s{zyS1yM)7|LX+T7-Iudl3JwXUvq zdG)&T$~AS%H?Cj4rgV8Jn_uZ(DiB~#b!RH?qUU#r30Jkr6 zF!F>vx=$DPxMlW?*!|5M=h&K3nSHArc0t>R?V|7dXmkb-#L3X*bY_=zJ6Za2QM7KO z(G7K-xlMICt1{xLPpaF{t!-+A>8o(eRAk?y=#1iG=UK(N-&7+kil!ap?N7yk3*n zk8mv|1Pb}Z=-I|t?=CIEegz~MXefte3}V*^bdX5NF0CF2b}+0$Y-DjKWWll;0OH?J=b4Gr$+psu9JNJ zs<4gtO$tte6}f#A6kgDW8R1<4=Op4c^=u@G4W_Zezd-+}b9vw3{W;T6^Vg$4^#9~O z79o4s9kAz6CA)tY(NckYm(ti*y2nSjIS8NYgnsn#wKX+0T;4abA@5|Jd@169e$anz zaPnmGxA9#6qR!5|q$~ZDWqn!sea!iB1%$8Lehqz~e?4GhPQsHC$iBgueE7%! z_#aBMr;|C4_O zZ8q*p%HM)>=8!ls{H$5g`fGf?aC`9ihxVi|JNoDRylvCVvYunKu0|qzx&1R?-)I4S zk*bru;UAdKxqQCTeEvlmD}G)R9_!rjusDF|FOi!TkrgQ%}ME0B> z^{*kPRv4I;5cOBA2Q!1yrV)K&%-^PEQl0zTUbJ_iz4Es;q~B>SlfOBDcME$So6hkL z)J1#pM?3J+-{9ZS@64!vmA#1pviIAhZ@Nd3`lj*h%0Gunpr1~Gm-BhF{v}$rhyG`h zx|M&B{`n_TgvWSg&o4Fg*SjktrtkVTi@s@o8}!ZRYX;YMhSVSYKO?H|`fRKhE{nch zu9&_nNnftm{O0om?WMlae?}16f&h_$Ua|ht{NM(Z`pPoPOZ`RHgP|f;mNnZX-&p$U z-GzMMESk{zmGfti-)8VPlHXzRH#?7?c9Q=Tt!H+TKhXM6PwS;HLJ{$2(h;^#}MPNNQ> z-bFXE0DrKX>9p%fESfML^0U$MOWN4_B=ZC0Z{6}N)9r}bkmiVq4;(KPWI*T9r+LZ z86O>!Kldcj`dXH(1Q7N#`yc7wW{l7F<2#XWDEaCs-*Gs9b?w86BHtnVr9{{{49NYL z>~mxPgn#pV(IN9ol7|0B%#8yG%(|kQkwo$a<{sDOj z&)2&QKO=bydx7Tz2>(gnZA<3)v4iKw_S~Q&dGhd*T+{Vg|`*b$O%G)2Q>i`~6A0sD zyqql5`F<1q!M>T~E`FL1khd_HnHh~|R(^>8UW|8rerWwdKqms>-)4V5MDxiwAF%$I z{2h4Hd;s3$@4y@81N4&{H2XjNeLC5f=R4pxBWRcpkeBdL|G%VBJ_Y|mq4@YBzJ!SU zi@8xJ{hnKhI?@H|kmR{Q`!`2IOuytG$q5$yX3|pSuNUzE{6ZUQepX_BC!h|6N9TWH zdJLZid=lzLybGrusy*S;Q8(f8EDkYgfA|LXchXM8 zUo6+N*DgE5CJlL=E^@IG4ug_;p_O#|_rhEkU zI}l(?UaW2h%!~6yD?YjBPpTqwU-#+X7WpuvcNPgLi^i$*`m~Soz`6EAH zfq$Cwm4(0?*JoOdYe+M% z*O;$_kFM8a<|}-DTk;j!j8hT*$oCV<{;(&`A2TNd8*yR2pZ*l{74nC;`O1tj@)h@t z8|N!oJYR9g&sQK%mYd>xG+zlC@{Q5?3gW3dE1ItiLw}z*>p4`Xh`{ap` zawyGZiceTSP5bMu6rU3L{+!1%h)fL-g|p!bjs7_)kYXqy11MQ{;EI9?wks z4Rid$_@?~^^lge~9B>~6Ya-i=RFt? z`={9X71saJ`|+vMZXDmHaDSf~KfWP;Oqt5}Uur)-I=-3Ce{zFUr&!`!?EJ^Nzs3B$ ziT(BzV|*(NPS6=Xz!LjInje`HCc;8u|DBEfch&^@e9o}{o<{pcL%eg+h9tKCb~?w{ zf8#)f%I(Cv%G=m~r(?AQgJQpm_Ab<;`%&a;6O89gz-JiGqsad=Dc=kmFJ=I6cQJi~AveiTGEaUOmQ=i#92Nqis2 zDotsY{dJm-{XY8|2)I9E|9M0H@Oe1=L7cB=(dVc9^GKaOFi)g9_+4$;Z}a_HR6Zvo z`S|%F2`ScBJ2?n%Kb%he)5k5cKP`6tpLcWen$Hi?*cZW%*w3N^od1+{e-z74@cDr$ zKlu#mkm_E!r-6GKxTk@88n~x{PhA6ac#cpzfNwdlo?l%1!-qeLeD$Tis^4w?`+xj- z#kjv|{OH2pMh+FS*?mv-x@z2gb90XM^)c3OuX%Rl{eP*8e2t@0)+YRQUrujdUkW?( z<5&Lh`^ejNw!CKTjAQ!RnuN7A*o`2h|8zFR^?dr`sHVv;E%G|5u?Ob>e}w=_P;Hz& zKguL8N82kru-h;TgLWk%Mi%5dItJf(Q}-M8mk7trg=o-9YuO@r;STQ zd&&SbZ65DWjrVGT9rfbW;zW8CrF!gI+2+ZD7f_>xg3}mRSTP_)qsFMBo*yEf_!w7S z1tZiKy+k|C+c+z4XvSjry(v~ar91jkXTcS=1)nG3d#Wi&5PP*mupBM%@KMFO;p~Jd%G=z(BnId9F_sp7?l?YXJ|%#A_ere@jgF$_DZ1qw@F}TdQfwIy4=HaR}Ly zis;#((Hlru zcr#Y$sTgfEW*w~xP=g*v;fWj4!CGBk{UGL^n|n|%ZO(c%j}2H=jQj{#QN5m`=Qt97 z&sPd~9-#~K1NR(ia}b_JBf$zI*67u2cO z8K_fNq-z>AN7ZBd44Y1)6s4}W7r7owxw+9O7a2=8@p2X7^j2~KH`q1kSVt%WBvwcx zq~lRpY@A%r~+DG@Cd=4f=R}fl$P^|QKTHU0dI4hr3o_O?Xb4M#K z=womgIG4cCzpnBAdlYZ< zy)$F~60gww&)nAPTDc473pAfZIb+9xO`K;KX*Ya^(PGuya_w0u0QkrT0-eQ)hkTGx zRcVy+W}g2O_t`$1-R!9sa$fxI3lm$}_vZfgcN=YgGi%kU272pZa|9N=@V_VAbx8-6d6dO*0qF=2; zU1|3b!1=*Er@H>z9Nq^%8Zr?r1w>Hs>=X7P5U{rY$Q_9A~lU|HS@tm$$ zKgwU!bGj5Qn)4!_)0L^f(QEWMT@s?`qXf5kLo=Np(^=YkWW6E!fwBO^Rt8b0O+dW% zKbGyjWKpfPJ^mQda>gb4T}gkWiQO3Bi^m;%L$3EXNpVg{VTG4t^_F^Tierq z?uj_c4z-oTo>Oju35N{{3asQ{{w(DK8gu7agzX2$HpRQqoDO@zcJ>__gKas zknNTKsqs}i1)mB)y!LO$X-`wa+Ma$a3Q@-oGn@2(S+?mB@?ky@=T~b@z)X=)3bMwaPmInX2^)2N)S2Pqf0k*D&@TD!w)^6HZ(Nw>oHgEIB zidEhXg*#d*Dr?s5EL+jC;?bHWfAzYSl?64AmhUXzSWw-vxphT;{YufcxMf*O{l?-v z-)7$Ls*MFJy6Rh(<$(u(eZi8N?Zs=hH*T!%*tBuY^~dvW>B()vZUc`aqzDx%|-ZCm3lTGqb2Y}L-Mwi;VIc09FKa1jYA@Zet;$Z1LB%%xSxl6;~w)#T5zeX zPrcvstbWA)UlRY)@wL=%IA0Jw6MZ8m5O2H3pr^5p80cS9_bZ)8$9+S8!7l4W_XCB) z^_04|&`ksUoWkK+s_sjju4nac*a^@7@K?gAUtSFOTthILZNU?LsR1ATRistUm&efn zFNl?v=vr=qPIYg+zKFJV_P#ojFrnpx<4iCihf;P7MJ#CIs|NzuC@wX2g7e730V+OF zCEoL%*YmE?C{Bs@oM}PNBjTtQHp93IDd=PTnBj6Kq=2{zdgIPpAlq@tFznovTofh= zuKZugTeAtF{Fat~RNqvi}^}Pn9&8 z((^w^{_%7vUevsLR`U6>gsb`Vn(Svl8c@ME0G_S}MNex{r-G;Wa`+Zm|0e-v-@=vX z^DqvTlj6eO!j<4xB-}3HO1=r`Px%g^CQAGy3p`K4rvR?n66lK(J{54|mT|WMZcG)A zNjQy*)5K^wF7IeuBfM0~N&PF{h;)^NTl?84;nOYsD1A{s##5pmL;v)g6IGrwz>W0B zuLHkgfE(q_m3p`GIa|W5@(xG?S>=6Q!ma)MOv0`5{?gEok^hejaHG7La{kb`#u%A* z9rzImCw`2p*>e*9kOe+qpf}q0H3_%!PfyXILi|h-xBPvvgsX9tFE&ZI8dt%O80d}T zJ}2SUaepb{)^StiyrprC^Tj3Mqvnf*Q$LCq5@3q~E{}M)Tt^J>=pXk0vsezMP%@3|QbQ4h&o1YUx$ypQ?Yg-p>YqR1~~N z+Ha)=uGV|M1+Lb6CerSXy87leZ=joDkI?8{=0BdppSLF&Z18I{$zbur;nh59t};HJl?8RwW~{;y)C|m&eiRWT`d^?CU<|_1}*L0#=6e-x<<~J z5D4rEK1Ef?Vw7-2dq5G~>R2qd@g{8M4Q%VE>+t&;M*ps8puN%C>17M}yKqQhrB#rm zl%oX6f`*RH{3SQ7jXPv7eDlVQ`-7>svw=6Y`Q*EKF&{hBZ@1o|zw9a&*iZPr7ePd( zTBe?^$OLt&%s&)TF4L;YXrE6onHEIh9(i|0#xaFZ@k-Ta;f7K!j{?Z6f2X*4PUd6u z*&4+=8IGel*gpVB;~R5pYUz@}*FzOUQRA!Yv$8$)Z7fwPcsClMvGEm@(8QBfhgEHY zLh-BUt@Xc><6p1+M;3(QTlJR?;jK0y+^T@0Fkh{-+hzMv^2_$mSQ=RQ_W>Wzeh%qB mm*wzeKXL`tUggmHF|c)PYI3%N-toiZQhs8Cp5$e1|Nj6PqY~Eu literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k67c2ec2335be3981cab8732df48e2960.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k67c2ec2335be3981cab8732df48e2960.o new file mode 100644 index 0000000000000000000000000000000000000000..3c7e392aab2a58da03a3df24a1cd93d5a838cead GIT binary patch literal 336336 zcmeFa3w#yFbw57${m|oHU4)Q?kgkvbNjxOLvLV6ZinlRX;$iHhNhD#wVDpeK!dclg zmSmDZLd>K2)$yL*;n(he^07f*$9Mj+ zDs}uKcwUe1*$(7izkJ`ByI%UpnaQ|r#`Ul_HHs z;qyNkx^y?mT$kAdJeQU~bmk`HeFSp#$n!%Fof_Ix_u|kIq!pd;_ROBb51wg=J#^-U z4}W2(;oC=tGE<)(y7c&|p%=dOB50C%Tc&<9@_PKCGnp4ZkN8K2_E0%+?)k_|LwDUm zdCks5zOeU9bn)IZn}GS!-Jcz*2mjKCpB~!u(vOEeH0`I5nU&cyZSR?-bss+ScNdQi zedx$zLpOcl+e4Rberl+G?@PG8I25U$lX;}20$ZP9EXZ~)>7ltlPgMOAjGqkkr?}sw=FAjYO`ia!HWFnrbS0YCSA8WC>{}T*QDss6CSM+D4x^4d-oBMNNp49MJ-{>R6OI;pm{`q|I?DxH} z3GmMPN{*Zd?+3~@Qa>GOKIu&z_jrgW@1OUEZ>G4NFO4*x!2R(Y&x!MTwxn&7t`}ov zuf8yU+Ox0S@5#Kn&6{~`zBus0ZT0?N-s#J{(&2|424J_r%-kQ|ckn-8e;4!m5|O@7 ztn5Gc%7kcN|H`lIOAXigMQn7uZ{>B!|IW90hQ*S{{EvMwHT*)+q4b6sp5gTkp5b-l zJ;Po9K5%2}@)!KS@cEAX(&s%g?DL>};>eC~1ct@wLlZWpJ;RSZm>TYRXWsFRj|N`r zS&Hjc+XU>vmk!xOU*W2-iQrbsny_;o5@hDqOF}wGG#4xZaHGL|hl+T8Zm?T+8Vyo)zzY;)jpT z1ziIA?YJgzy#d!kTxa4M#r4CuuJsI$JSd*M7dq+mPCl^i8&eOgADDVz!@Q{n?(|MQ zK=nT|RvbSEpD{8@9RGQqI6mSN$EiLM@4&{j3x_vtSU9{f=o#L47p|YJ%52(-YnsB9 znN9cNdLMFiq>J@m?)gyk_{O&dUb_@Nluj8{Z`9jmbC-+C`2$}$+d7qR)AI|4H(pvW zd{=Pc@O|eIo>-dMIJ97R(@P76@BP(+;rqP{hj&&2=lf9ppM!rNuD`_fURa`Vi3pFUu>VI~w2mZ+CeKv}4J$w++YlE%B*1vaTwm$!!O!wE{ow>;) zhSN{IE7LXb2bpyby)(1nkKd6Q|89}#`NZv+S?fKSX*YT^O$&XQ86JP8x*(9L9UaU} z92d$^x;o7*_qGnd?>o0;?)p+{`0n?$4&VEdH?y-+3~#=#b@**7L}uzce3>nuO$|5K zdoq)!doyqE^=JOe7d>*tbBJxGuugchrrk%e(Msq+iP~Q1bJY{JbmVsmyOw-iW8@6?gjE zP|mF=A7Q?$GR5U=-a_RIW+MGXuQbl~X0|~uo4-0QGqpaHX`JrMOhdU}@QDKrt>W1Z z)WzN7TZf~b#4D50?-}jXPf#kSt|>Fm73h@ymkDFYla>am~EUYxlQiE`}Dr zN@HpF_-ACk=h>0}@?~C_oEpA}arrse_W78|To~iYocDzrp0Dy|UKlI;lV8=Oh9Q5z zoyN#FLl`@iA&+w8H5PeRAn$P)GgW3f`rrfqRIIKOspC~;o_qS?59|D?;c%BYK>n|4 zoVcg@ryjl+K^dZ>Q^Qj{(GP#JAT?Btc#)#i@wo-!9?vm^BM8HPo}=)J)a>Z3so8l` zQ@wKo;-1taOYZB8wBFbHMeqtz!wpEQ!{dF~^ChvmAwc1#mro#EA4v_*MflvGaoS35 z-OK%7cH|EKxsnTrmrwkKSUtKRb-XU_!k=&A@45v1xTo!bQj*bQSP*{T1|?bYUwt zv7LAx*lry_&)< zeV+qO`py#%uI%vS{kVTn>pcJE!zUzm!ZO#h^5xcVATDI*dh%cH`X+d&8!A)(lPFV@ zt4tMUnYzxQOt`1ZG#O=j0A)IXc)Co7{{~?y6R@9n3Sm{I!+(pgD%0+h2&*zxQ&^Sh z1j(RG(fx;Py?24{*1PBLfHCF5SZiYJ|2JU7J&myu7)OAy74bC2>Qe|SjIG~7SYfPw z8ezf+Tlah$VZzui?m0~N6~;VJNzud3STRQ6~1#Mqw%2~lAIe~>lxtdbm60&3*Q_QU+Wp*!#$0!3HYjjulk=kU2gN9=MYx-s{aLHh0pU{gcZJv6ju29DXj2S zQ`m*?9B4Pb{(l9&J{P_PCcge>fDiXHzUja>7x+#fp4Q*t?;%Y1U}q=(4PkA6-$z*6 z-&usU{n34Ge-s8j_OoTMKgiW{TlT(u0%5{RdCx`Ncbx+b9=GlDz`fsvdzp#ba|*a| zPvcIKUSXFf5YNW_1B8`cPh3D);XeFBgq2=*Kaa4&T}@$y`vl1Vcl5xAowg0WTeqGc z0plST#?>aq>ZgDa_cX>DV0-`=TMu`nde@tpu&ciM$-vx)D-SAh@rG`@+z z_kHNl^Pik9x9@WlR`@)>Kv>~BM`6NOgm(1`!i29C;~)wvd>#su4#hrOht)A8^Ws-B_^W-D{qzm6iO?-zx2R`m; zd~<;BlOgy+#MAf=7a*+gohU?D;X52dSmEoBBdqXMQ&{0U0huuH=`pXT2>CU+@O<3F z)AecaaZls95qO?}zFHAakPh)HW#_tnG#IrGuL0DrfMOb4jLs(-hN0=~TTz;7DYm5{o zjJ3`f_8jiHG4>;FUAqh8<0i(#9|s@zG{(0B;{(8Wg7Q@T$lw2v~Sl8<~gcZi! zl?W@0)fCqCN-~5|>|c50hC}Ik9xp_H+zJ^t&U2vaI$b!woQ3ma;Nzagc|GZp#$r{R zzHGxugcZ)}YJ?R|&v=9t&WjXQIQuEAa8^?oI3>PyFP{VL#@Ani{JUNFzGmV(_fhb1 zPve_OdZe)!;@NtvMOf=`0>WC46A{*WoP@B}BZajdNk-#q#aOJ)#CI5UU7rizlP13E z1K{JH#&;d?eG>T2A)dx}q8?#|?_2}I3g3yz2rGP#O+lFOVXo3jVTJD;$xuBHK4`a> ze(>Eotey%C`&}5HHZit-1bp1n7;gZ^5n${>JdLq+8o~-=S0lm-V`~$_3S-@LgcZgU zbYEfYqA=T{(;s^v>&AE(blo8r#(y+1cI^Ql_cX=^VCUvJ_&0lMy3>+wt96vOsf%lAx-{qKhi?p=Wqaczt@ zAgnRYLRe#*jj+b}HiUIuQ&?jp8DPBhgcy!4aePk~WZb&+yd5|OUAnxGg|i=g+{?z< zig+4l^&Es1&ej_dRyeC$5LP(Da}icJ57T{xvz5Yx^RIx@7lT~d6;Af=U)P+YrI_z88^-o zpzF@I9{x%1>c!U5K#4>RF7i!rHY2VTJYNO$aNjT@+SW zJrvgVc>;6lr6$J1%aH$N=n`X*JA_zWh4C@Tyz=8%X4b2C4*a*HGATR<`K7Pm$T{EW zhX4An0z6+RJo4Iq6k^Xn?8vLWyd%S(jAw>@`A43AxF~aBUm`O!e{|;j=j7U?Pd@wS z=l>prtFaHji|agG?+__|*4K&s1s>6tSdO*FvonJSXuUMG)YJUxFY+_5?rS~%N#E$Z zvHrRGg-@-?{NRt4Wq$Z4t24j2BQJB-R|ZVwcpg3W$j{qCnTwO6nHN5F8?KSek3My4 z=4Ulw(91ywalHlC0IqGg;#t9`+HuAD>ZexVinY{Ft;E%X>ndDDrt_7NhmmK)1j2-J z6Q)B8Jw|7JEgG=)EG%d7>C)<@(B&#oF-x zU5G>J^+j@%`&;NQm-HUVF}+P8QB3RXVLan`y>u&)=bOo3&&sea&(s{`VYkr=Q6K!0yk&$_IM9OZQSB{|m1*c{V^pUDa5PlIo+Z+d=vE%mI+ zq;=MuBbL7j=Z0CQ|996YukqX(%`Km2a!mgdIq2|zV_W&l9QC(2M>$>ZH|Hqdo}=9A z9OY_rsOR6bt=yF(mS4g@=i-O7&VH4nyf4T6Ly&vDenI;{!grhh&3(sjsMK1Qc1#-FA**%bZvp(uKwX+9v z#5^iTIjyrXIm%b$D3{Dpt`>4{wa%{35zABXt-08P*4e3Rl-GU#njC$_-{+WqmG8}U zn8;!LG&%M{l(bZ_W|hzhZoPEq(R( zbCkc3quj+D9OZPqAIwqy;T+}u1ai6fl$r1;Z?(?8m?P%a9Obmm z7Un3wG)K8xa+Jf%3~#y4*5!!hXBdxOOWS-UM|r%rm{Yo*9C5D5VZ8gB&ez7|h(+6v z_O;7%^n*Iy=W^7|<2lM5%TZ3}uW@6K;hWl_dk&Eu|64ic`v*D7>GV5tlwX&l+{PT` zbpBdD|NL8Mr}yQEEeg3@{IM=WVUF^na+DjBquh^v3+;4jj#&QMo0|{PIy;f0{8Kr~ zoyt+}|Na(qb~Z;W-8sr>o%Q4>-$du3w4=9LhqvX3jr#a&nKS%2 zNBPS+%Ka=yIlSQcmg_8L9OXU_xwqQR{yIl2)W=_o&S;L9 zi~O1#<=&a2+2&)a$n9-4j*vI$>$x)QSOg( zl-rx5+(3?U_~2De<=C2|+`1g)a`o};Ii|NLN4YsU%FW18t}aJ8+E{HYKiNOc!U?bCA?m* z@D=zxUY|$!^U$rR!&fMLmIwe*IZ002 zzHxg`x~FR`Li*afMbdoj*uHV=I!jF3x~*sZw5|6}>$!W=)(zVn`i>onzIU1(xMSOv z?yep&t!KT}Gfft)EG9W!TiOz*HouY}OigmTJq{SFjk`^K*H`gOlM z1xc^pzCL{~N`zWUuiyMTVQnZ!*Y89@)~#Q=8>s7W$k3&=$dBSK{LCfYxCwi z(_P&gxzTl}xAknJR?pNO+qQ1+>Dt=U^gCd_cXV}cS)ab+ccPk6ZCsY@y5yV6b#eu| zZ~P9_p)QVTvwZh=rU~o1uo%rphS?VGo~7Xy2A(7zvV$!W{~m}!dPpQ(1TZ{$AS z`hS1IrhB7RTh?!xwr#_P?dyA*L`TO~n6sKQtXjTsd57enrFL!J)V2L80=mt1ccpta z^=#UaK%Q^Ks{-F&|w>#h)A#LMG85xe64 z-{QU(zgW=p6S+*1QWCXTzM%WvS@-W(_m8Rj?X<{(qZJT+yh0*r{r?`mkJX~A<0H7A zh`Vs&;Wr&Oqe#`brij2nYc6!)eG!p76tnNkyF$F5<(>OU#mK01bOJh^uj9TC>qyzh z2##8HlfrbMJUi(c%kS^&s}tdo{=T{}2Zf;fQ?e6LwAQ767H-J6{_y>hN%61-$Onb# zpnUoU`s%W?3HKq}65_<=ViIB3U|>-yyCwzg%fI@T;)x$362TRIlrzIx#;3l=UAQ`?d= zlT*9WcWg{<-_kL?>4vT(vL>ulWq6*Hh4pUn0o?aDh6S~xdnsHLX50vJ7-r1}-PhiN z!n)=86HYjnrB43_KiUb>zQHf^9z*ER1_c&4FINk{qN~J7gRT^V20c!E#h@z$cCEVN zj}?y@bh$WW&}HId23;!l8}t~l+n`HCpFxinmO&SbZi5~rHX3w7bQ*M#xZR-RqTQfl zVu?W)ij+YYh*<`mFPaQGPt+N7R3r^LB1#Q9EaC z`$V6|%}2X1XpiW_BD)ixjezJjzaX3Qw8Eca5#57_@V>E+3GcG7@OAh+{skLFJcXzu zLc|w%#0a{cBsRg&wNqW=i`0E}?bh^Sc|WDDol7LWUtLr9kUyuhOkVxd=TIr|gDFBB z!g&nmAcfzDFwV<32M2)TL14hSALrm{k2s4n{w$XH>u34MFaG}cy$1d#Y9LRD!2i_K zUpLDKk17V?SlZF?bmFAGb`URR90TY>7~<_mo8MoA6E80x!Y+rtar*^6xaxr#F@f*=n2q+NdF`* zbp9RgpTfOwK?Y~j)1YyRZ{zy!Nb|>t^CIN`1L^%7=PO7Pzf+d#QP)g4&mOTxc6y~IQ<#q`8mYHxgX~-oTqUfdJI>bXK`M}DKfx^Gl{ba zXFJZbk0bA+;QcA?<4ob~#My_l9ZtLt=YE`ra2~@si1RegvpB^UA^&BhjiaMSzCHObNCRj4cai`1fD`9woK63R`#2B%J7lm4xEtqboXN8Y<2;Qs zc@ANmXK@}nkMu791J2|Rfeq(boR^=+{U0GeobihY<2;5l{sO`{kJ0rK(!<&LW60y| zdl6Th-lcuQpXwB6{}z2Gpp)JWc;Xc&04*O@P zg2%iV%aN(Q&4^AM z6BTDmo%Ha#9pWLsHaYJqTLfsc%q)TIHqWd?*V;znG@fG_yT@G=O6V1MG}5N zr$_uIMH3$4H}%W%re?_Ur4-$*=ypXPQgm`a%AZzrlcLY+_=>))<11Rsl<9XWIU z@=FETufs1D$*81te6neWk5RUwb$qgOhhK_5#Eeh2@9=>~(Ko`O_C;ihZS<0}3k_)y{trl$x2f9l8= zi$3^Phb|Y%K9la=WzvJYO*#%=?2dov6CSY&*_~#5Xm5ukO?at(4}A#^1_zao=(B(3 z5rv>HEBWM~OPcT!zv;M?C;bq;|BTE}x9>qklfR($E&f6BW&1{aMeFw6t!VNeWKYKw zE!!*fuW8noUo`!rOs`YX-HO)se=LiCR?)J)gChPXsQq5?>FR& z#h^i#i_2Mb@;P_<<)S@{?#rSFLA&iopuck4hi}#>;jhStox;bUZ@oFfKy?d!n(=)Av=Txp>Q zKR{K6evgl8{GV1RzS%oV(i3o5TPPlXuYW{5hj0`z9xW6D7TT(;4vzo$ds{|$!yNH1 zQT*95{%%~x$HZPs%%UM{cjx6ialTP}#`4di=zj4~Oq_2MWtMj&Kru?7xbs!wK)P}^ zau6ef;Gc|nygT0}vx}jTJQWkC(j}xq9I!6nU%caF7{huTeR<-0`^4U~DB*V2{wnf| zi3FB!s=!5hA)5+pmwDnDioZ7~s|@ik#YBEu_+=X`LGoX_StkyG=D4z3jW9geVju5e=HQt zLsBEO7;_=!FZ)=K^}#gKYyKD+id#MsHs!-%EOgwTj;8*ft z;y(hul4tsOFDZf!toCAz44P81>D`YAe-NF(2b}7yO*AAJQ}F*cJ4+`*q?#0sUsn??QPhQ67{A_45$?8Rbp~{MmP_ zP+zlL^+o>a<3TKe;E?t8_f?9(R(b_NCcGaT$626l#Y zbl|H3U#=T7|G90VF|7UR0GI!(+XX&riO62Wf38SWl=gvwzdm=F{2TCT8vgq`&~M~L zA?^2%xIJd$hyRR{0cpHco+Q%a1MO+Q1%HS21m-`7_J1Xw;BSH_+kRTL{ZswI{L9>`=I^^{%T-nzIO0jzx%CFC6bf+1Ao62 z%(wM@_4@UbYx(c1-*8k6==z0ySy&rT8Y6o~{pJhIeRTZ}z@K7yk`DH#Mt!4yhy8`A zzR}BKJmks2{w^B!mtCL#BkKF~HP-horO(%?ZzN~dcb}5~ZPoXaZ&Kf1{$F0-&s}4E zKlR4-jpWSw9{erWH%7G5f4@%q{)_+R^?mjl>l-6}*5~WAZzN~)*`H%XCjIAC`|JOH zf4<~R>ia7i5A{d?3;p@UYpn03Z(QF<&aCg0*6(krzCZLP_5D?i=l}ouru73n>cIMe z93N-*_cVT^`8to^G2UJrq1{Jx$nhJE=a}Dhc_O7V?09x5;`!Rq49t6TO%Vp z{>OaLyHgF%V;HYsJpl7FA3bLfUd+!`# z0_0<~zENS(#6XC2B_04^ym-pX&6@-;Fs-@@KcI$Fn4?hquswg^^uM-U4*xOc!y>N- z`nX_>mzpi4KaY}^^GP1R+2y7D?DTx%yH~}hq+b^wO)m;stT0-ygMIb)dar9CfwIbcu@?2W z<3(Oyf<5zk!8dzV@yKkj-Z)?t8utD8d*jZ0_dN0K`2*YYz|JtQt0T6YFIo{f|EE0J z-V52@3&}n~)BMq#Uk04{$jL_V3nYF@5A(T$mLE@^FkZ=*@eib2{oQ+boX!w2}e z&d+~cd~eo%F#f@4$-zhSNs|sp`{MQ5E7yy#UX23MdNtP1u|D&`bPx;De1yd_D4#bF zAzIfzy>ZaRM|<|W>4BYN zO#cCW*!o3!2d$bQt+(p*2X~m+5~FS`(pOs7v7)u&$TV^dIRBST`UrAR8&u z*JbgWu^v27$2XZjBa1&1_B}vVtkP@B;!hV%Vql}@@3$H&X}R0c|Fj@wuIWdtdZYgK zTQxPME->&uSOXT0{9RDu;Zyj$MA^(f@@Wne`3I91YK=!{^uAgP& zfxV33_NwahKsp|={XgbEk61<7{yxr|QtbR*H1d1H=L67JcK%QFj=;U50!1g=TQ;Bi zi_6qNsK2-jeeY>f?U(use%=+Pt`*}=ejc{ps)|!`T>ltvE#9#kwMU2a52w;eqx~L8 zha!`}9bx}VN5s0Oq1gdHvUkU&sSu zhfc76*YuBiMG@&2&(I>`f=68XLlN{}kLTv}!~p#@*+17O^#{NUent6p#7F%}`O@MU z2AwEuQZ(Yfuw7J<|H6sq8#EsKv{C=yH`;ux$1m>vIHQ4musvbDHhh}81Le<8rDIw4 z63DU_zhN)HAEM`hcz&wvF)#e2E4>5hpi%$YejNYrdR~F@(|YnWiICd|>1Q#eukqZC_RPeVJrD2_E(W*e~B#nD>v+K z|L%fHdK{|LH|;S%_T~>>kLTdNKIoh2(&Tka!~U2qu9#uaiSi~zL*B4Iq8}6mj=cr0 z@PFC<&YYh-K;t`O{IuUv&!csHSpn(46`h?P?1lBA?Pb3;etb&n7ykW&=?FGJ@S(?B zW8C#m_C}pC$LIC1G0u1j^Pj~#+Ql&0H$5N4^FYY+28QmnvcA#3l<@dU?sp(Q`wvbJ z&gkD9GzVi36rdcJz9QB1OY%RYbPNW+Ld zfA@2L@1tE1NWUoU_p!hF?mnL1*IGXRAd=GZnBOd;^#I`E=WEU2Q)xJT#Lu^cND)8Y zjpvtG4;TO+w9KE@6PEc>kYC2@2}JvSgW$SoKRP)0BU*2Wi3!#mF8>_HyASinra3%+ z9EbVi#X@g?@02+$6oKeEnqSt&6=`iBEX+PpC`q6o)52VE%tXw~tWVN=hx%F0T!ja{zwc1-rKkat?w2x4K(yrE5 zCRq#G&H7u+^|yfc%aA^=Z{qP6rFVXxKVjVf8VL>(P5h?m4nGF|Y|w80bPO;te&hHt zac_FX^;Ca11p8F{mZ>hjCt=NjypG?9^k>X;@hSca07%XD>-clwo}Kg)){T(Y@d@8d zbniCbPw@dTHOG#>WU`Bo{X8kiyW`&s_ie}L`~fg^BgKD`_W#TWekG6jE2R$rsTA=+ zC#-qE=T2`Lx>qwj*k5YCEx!oxlq+l5VimM|r@nCp_Kla(`RGn!1Wo*Z!p0S~bO;U$eDx zvQ48r6kn!~4c|z=*18qR_i6v)2fsn`)67p;Yo-!CGN{|*J3^qO|9Y|w_H^s5i2s0{ z|D(ts^18g}f7Wn%?Q;K1tu;lEJ?Q;KVUC}o2%KN@mw=ZEt*^7SPUX|1*9 zHsWJE$n7V*y`8@)$eY5@zbDcwC$FaTUe$Q(>%*YybbCv$p3+MEIhwx)KRt;(w3=_- z+DhHeluGbXzX@xV4+?hSTd_J$wCyiirx4#~>GXWJy7{Q@m9&1N^`A&jo&pEw%6~P6 z_fGptSPgY{{ug3#|Io$V1xpne)fk+o(GuMf>qc++(&xP4IhVPDDg4L7v2 zaEj_PoJ?PTy-ClWZPHEDo8w@4=+o!3(?kBr^vVS;KGIuI>!Q(qYHy$woRRK6Za>NN zoH-_c_UtBuKYhB3kMwT5(WGb0aM5TVQ&zL-~_rCfZ(pE=W{=g&9kmKKwq zH_xQ!&Nb6>mc>7`3edf76QzWHX8Zfi5? zTW&Gw-v2;4A;b{wrB+%Y$x7Pnd)8 ziER(HR%>e<@sC_yCxitNPQHbW_^`m#!FHj zB}&Fl=aR0BBbZ=?@hP!e13^N-${&e@liiFgLdn06xI)o z`0W5B$J3~9!h;G+O;q-Y^xFZ*9zSDzjQH;K@(Z%l1HjbHcKj79P{FBIMYp6IZ&|L( z!}hft@!jba6lSM~0?6_61&I#^A?GKc6V@$|cgHV`Wyc3#IbK)sVL);|#QC>D-W@-N z;fJXo0F>ip6(0tcTA<34uvS7|#bQ>QJ_~8GaWY_~Y;#;?q+i>5uUN6`dd3uRq|Hzc(F? zLOgY=jqf(_6+V^^U^r@*7xo#ABJb1CFC6D%+2L*@`non?j zg`&t%&evH#GiTcK1<4N~gVa0h{Gu~QC_at%@qSIMb^A<=@9ptA^cS-Ahxi!ZM@e4K zM@p>dbka`~?ooZ9eH(ZZ*6p_=Jvm=M{Dd`qx*Jc4mDgzZchFCm$}!!oNnU* zzsQQbM!Ku*uRw)1N?(&DNdaMVez{G+4U|+mGE$sQkQ*9Wp%JDB_ zQsZ&18^Xx+jW?QfGHKE+yjAdRaGY4+G^6Jr6xUZo=KOLne_bmCS6``(hC-t z^w_Z`y>OvP7Z;oKqD3Y>dbCL|UTo4OB`zBFuw)4g6h}(c|Cli*eG@Hcx%lz8NiSV$ z(nUokee=yGok*DUvSlVcYLrPYUvAR*`6hkKEhb%1VA5@ECS6!)(koY*bS!4lt5%tG zd%H=mS!2>GR+#jyx0>|o)h2zz4JJKfhDqOcn@RiqCOv14Ne2QZ{f>8-bTsJDjqu0U z&%PeD%j0GEmln)_lbVKqX-QYNV0>?n*IUx#=VCnXqH9tZf4k_42^gQc=yA0ePfD88 zuf&ECH@|9Pl|h%bV*KQaUp5cp9~WIdALAJpJ$3=cC$9917Zz)p@sD1F@q|mhWHH7E zE;_sf16UV*(@k-Mjx5D^(3O7hW{me-^2?TCyyl{pFUR=HMd#gu@sx{Bwqd;FqF1iO z_(#&LpH-{cHO=@}tigE2CBOPsbf+$Q#tk$!(es^_^li6M|KZS~Iq2^t&FR;^0|O|V zUt-0tr~dUu+@tY9iB(ihG}W=B6E#Frol1JtIHIWzC7oYIG}W1$pB9u6O?5>59s1Kc z(x;qnhLC>YSmGBGpYw~2CYtI*(%~@CH`#O~LNvxCIz1oh*GVtBjQBLhmig!95uGGj z<}YT7qjrB@V)<7Re--gL|G)~O@h9cCi^()z;E(g6|1YtEGl+j1-DCMsi0E3Pv7dJL zcESBM%U=xf_+fhL3wD0H@Gy}?F`D4?ua!QC@dzJ%_>bq2+3&ZS&qo`De4p|C9_sE% ze({dSWZkO$BF6jUk3NL=0rB|naqRbkIOz-fLuYY*_I|K@`Mi^!$IAVcdc4Z}w`jkX z8n5yDV%Yze&-VA2+^>QCbey1mK8^hdfjP*4X*_>Di08Fbw|c)4$)_|Qcn+q+ep=sT zKFf!$kYC`EPvHGqmZ$aH3vHM$nD3JxOy^w@f0-*j#%}>?FL?i*o}Xg8>kkZq#{O%b zFO*?_n-4TUj|Cpc`>1O1Jk?M35Tf@Bp-(A}76*w0;l|P=6ek z<3F*hx2TBQ7QJt}p%CM{c-$V}A-!F_`P7`GeLhwQIvnS*9yN-KMEm0+srci8Cq1gj z9&cg4>`!1%nGxm zfY9|v>65<`=a2mWfp!-iN{{CK7PtjG`#8N~ z1oJq(2<#*3*azb=?c-oN$e$(E_$Yl$IPCnOAD*vJ^0c0f^4mLNRr%$7koZqlNqwgf zX8eVu4>_JEKE|`U{86P}qMhX?5eNcUCd(!3A_0RT56-xTT zeh0F*ZX>_ZZhK<=k$$ATL|8u?UGh}^YLcGNKiNmZ zX&+Cuu{}EW?#r@w_@eYEcl+^>{aLQ`xjlE{hO0g7bn#(dF8iAH^w9n8rw;xLH%dYgy{vxYAJlV$j%=Wi~ zuK0w<@t-l{;6b z*U$6RU%YvHdD0ah{xPfnC~~z|s;_*PeZ)N3{SoC){iUoQ8Xp7$r;YRr9RJJhuh8vZ zk$%kGpOF7``Y+T6w-+)9(huJ6q5Mc6Qs1)t`@NBF3yQlB0!FU?2Y<)!(kyS&f2 z;!}J4op&N0(%Mgh5MxA!(9#z|cL>zee{h%O!gw%dNV}F=txB7jM z?D=YT`q{L1XPnyujEvK-wF(NkzSR6~;QqQ1aS|DjQH?M3()$`vu6$nucrf1+Pk`)B z;rY?CTR(-j?4>JKGU&4Ok0i@6Jn6F~KSZfs){1@`eSM$s2w0s|n zMKIr~P6y=o+~@|%SDg-`s@1`CXoQN-2h*_;Y7d%jQR$7uafkVl5o(`&fNoLgJNz&{ z1?J|<{2ab-gxV_~9AD=iK`EecSCu#l4KSJv>tTX7Z1oL&G zsgC9PO^|6WBlAN{a~VOmq@!5qK>JnYskLGSR9?J~%Jcu0v@Df$?9`iiB(a^R8kHXG$&2+M&@J4_$n;VAzW0fFynoV<{D_7vBR@(X@7K%txQF+zlBM+e1>#fs zNH4jYmW%psW*YiV?qD1GQ>S1(>C()KgH66FnZd@e8Z@p(gSPuNcxu|jB?XH$$9 z?-NJp{bAdl8eu;LA=91;3QT+A`f%G5+_Br93dlXX>?wd>LBc_Wgg%o40bUPeeWO0A zd$_)3de8^#h3emJKSaCi$Dd_C9N%R>Z0|1naoH2~TI?1k{4ex<%sXdh%BA+{HZ zkL)8v8v<~E&-M`#Ogr`w%CZms{uSE?{6lv8H{aip->Yl)B6Cx^g~`6y|FC^U!i3kcuQ20t>?7N5NOAoc{!ZEx*oHk3?Xo9wZ{*K#kNpwbkISBz@3JSCeX%{M@q}$pJ~B`o zDP>PVa+r!Hd-BnS6iLIMO|blqe*#`V{4dMf_7(MI*%v-2r~J=_E8iEp(!TKRd>SxP z{anQJi`kFM@tU%)63dq#e+2eHBIM804Kf^qzk)r`crD3*5ufeD@B4g%_HPfs{(NkI z_wPQM&&XXEzk-kPh2PE}<)`OS@5Y{-C*P*?*&cuY5l%-$(Cr@^}t-==(2}zl#neLG%7_@F!Sx z>@Q@05)+GPJj;h{uL1hrBpoZE59+Vdsbd6y6Rf<2*s z^NWu*zqflHr2P4ariR6&6oLAFn{0c#vk`4@$r3yH_X4v;a@fX zQpxwc9{#d7iC^yUuUh`G4*#n87fZhP_3|J6Chp{Hw+FDogH4Y3dA30$1gYbmzQDwXvQCF>@TPI z0FAdOk1tL90F)}T*>56Li>l2egfY!aF3S~)S#Eq-|q`=pXk~ji}^Ci(+mAH{zH8y(zU)ilBf6c@O}NF z^dujRe@5v2HqbO)gy#`;^nM=dqZ#kx(D#z$d`FHqh?=DL2d5`;`Ed*WisnPLwKU?I zMDrEi@9X3D`4x@vc6Iec7EaN44DXXAlO|nLW76ZtoAkJGCS6fs(p6O^U0G?;Wo0H^ zT58f`$C`9`xk-;6ZPLZXCOu}1NtcwEbWxE>$KxhFYLrPQ5++?xVAAsYFfehY&j}Mu zeqo_WMq) z+>)*s2mfvF-)l)%Rly&-=+ZLyR~J2YthryZcr^T@EB!HJ%>9t@BKF^uKj$}U6#SPf zz5D`tp5*K=oG^jc_igzCxjtm$FU*5Ka>*C*e8h>Lp!W@IKHjHD(t0gAO|@PdsURAi zoT8(pMAM#6x!#*sOf>D`B)mmCYv~0odw)FUYjNV!Gd|*DJvE0A zll%P&h|g0{mJbyXO*)|bigxL47e6+aOgD{D=dbnjI48yPQQ$G=s}u43%Zk-s%lx#=s*TauknNA>Z?5?&(#QM!wHPbO z`6$LK#(Wk0F!NJL0SC&5`KWw1|~-mq0yi`nA?NJF*1T4%h3{dl$3L_zaG)(_GT zZ?6+HzvSn?;|sw@{%{>yf0$2Ctg=!52`jNhJ5 zC6gvyQ)ANO$D8!HaVA|+VbWDqCS6%+(q&~PU0Q0=W5=3wdAUiC9&OUa#U?#wj7gW2 zm~>H*Nypu8DSN>k7^ZpgjFZKIZ_O&;*EH`}KC9wCO6dpr2RS)?^bgo?Yotf~xC@_azcs#xMf-{E{nPw?tWetC zKdoqc|1{Hj|FGO|>g*qu`%US63+#_|<%j*k{K0&!ABs=Q8*=|~1o_+hn~{Da@J9k+ z(yy(LaF#woM)_R(p%q?xKeUaP)~ogV1*8vJiq&{r`;}FE*aPHofNtac(S%3hmG5g| zfgi?8`)6hOX+6tb{!&-^*pF@CC4E%5_=G2E&=G&Fi>CU#7JXpFig{|NSf(OMhE3%oy;>JJUb)+f=_UZg&xerUYn-hUhdt>14m_8YIe z;(O`dtNnf?=^F>T+@E0#)zwjSnzcBq?tb4yOeGr*{zXR)OJ3bnI2lOlV3&*^q zf7Y+vzSaCd(MEn4?~s0ZJZ9TxK(1dZ+K`7o`erYsj)P{&@Q2v1T0;A`M`(X4{Pjz{ z`u9+Apdd18?8n1?Jq%Z!`HXkxD2a#t zH+{d`{9b(XPEs@-(x3Ua7s&l{l;3%JzQX%8?ft0DI}-`ACp_O_|Cqr0AXMMLgYjz{ z@D|FoSB1A|XGwzfAMZo?8nOO5CPDT!sur=ue3*5S?oOWCZbJF+2W+M&zgVQ>pw& z-X{i)_!u>N)fU3L!vDPI&8XM>ozNaaDdJ!3it9+xyH(c_ekS#Cwn?^%o zyp2DRj)Wb4BbP5C25f$I`Lb!*-U)z*`}@82?}6ic`uuw-=J)Ww*(+O**o`Qt|LpX# ze_uHJ_fH0}e-Sy!1NMdY*=2jD@dElQBRx63Fn@0Zzeh^XNBKbe+lAi`_7{U9HLiPz ze;dv7 zXvqB7UvT?%$EWn2_;V;S4jI3!mu8phSX+qqv(*0lnN)tlBi|PoVY&^!hfb;C@I1ut zFFd)O;mdoE((hSI{AiTykN!HX_A84*{{6Ba^p0@(5CMEVf`z@=$9Rkf;qMr4l72s- z5$XBR*Wlpv+CcjeQPJOtdG`ipz1GU$-yrQe_KdjKR2 z8lH)X1-+ZaW;TF9{C+}*Z?X30F8O1Ie77NQ*a!4wtY4u1u^x!IK8;stKYOES-tF^B zdxL*E74v;ykGPI%prs7^wXq+!tXGX@C*UTYY+-y{`BxO5evb~nKfRn97avrfZDMa) zl*sgd&cD|%y{C#u`F#ie{Sc{7l$w9f3+b_asOM3B+8P{volr zYyonkL)#1e^gIqO|2BL-!#mOkl3#iP-Xb;M;qqerq}j?Uua#9^?3Z!lZ?>|^Yh{%e ze78L2fBxWUU7kHO|MPp#YMSSR{@`Vk_KKNGKZ+kWX>Zb@?eQCx8T&;H{lFfx?cLsA zMfPwh2LHh3N3r?)%VbYv-}?C{`Qtv~(gFR^?*X!Y6>aKwo5xAd&*?cd)B=CGf43jq zeO!#3!~Bc<>+XPG*$exhN5EJ5V|qWoui=)r>qp7k^~1E2pDa(&ULSu8$FXnvIT3BI zcpuKmpPoO%UhVwl_ha^8JlCZ4-EZ0>`tvV+W~9>Jj^Cg6z|E~qiZA`2k;>rhn#L<% z?+3rrp|k75s6V7<)Ze;k{QDN-7`oCwY#1D3;-;<~Q6!u8|pT>8x zKd8cZ-}U=am>-aRx$}QF{M9;=C%pK*PD+pCzixUjhUM?g=?HOVXg6D~`8 zy*3T}_+E@ZNqkyg1^&Ig=zq}VpbyIv7kx~7OC1{Rb;NJY^i@*+*dMrEJJ(i(=_lIT0?BLcs_ zD(3^z|C2#=V_|(1V{Z$jJB?_76Mse0=C@(Kh2?;M!VQ{lVf-?^XPnc20`DaFU;e!TwSLa? zvrS6=`9kphemB249iqJ;vcJQAE}m~4(fMOMwcA(G3AVp zN0k@&;{5*5NPHKKU-3Ps0MF+ucs>{th_42mh>!2bkbVds|9&p;((hIFp}Hi#Hl$a? z@8d;^F%4vW6!ZKbO6i@9c_!@k`fUAR{m;~g2jiha=`UDcF@Hb4*VNL74C(h)H{g3u zdt+W|@jg}l{@y@w3uxx^`xDWGP4oK`;e;K(j^39Dfe*lx-bSK>;JayGzQNz)EwuS- zY5WqYAba88zk>h3{-%h3@Ot_+GkjmFn(07F)AN~*iaN#rAk*PKO-DQwKP2iEKj0-g z*r#a^%llF{C_ct#55jZ@6^-$Fc7KNVMUq_paVXzfkIBb;81IjO506j$jXrvR1a#6x z=Ywu?(P7XjMgM~b^#QtF(rd4LA1)h@F+YX>$>#6s^^R1+ze9gvyiYE!{5{?R>E3A7 z1O9XIjvn4$3a_0<@6%vBT|o|r=L@hm8gCy%b@F%{`^CJw5&be~>Mt?g`P`n75iiCE zs6(_L5Bo#*$24C+d;=fvH$z$Z(EYb+zKysE|5pqDNef8YzdOHIEBk{1`u$oO(9wbP zvEErp!xL^Vz{CBC{=KX5)3AQ@M!)CV#(ctyL$&`Tw-@&JSYM#|AM&0bfjo-I6^b38M<6)HV{zW6=X;dJ2txwbYPm30@J&%RG ze)<-wZwOF*)B9I#Jm2H-@`7&GukriNsK2Q+F3<<^yYly)=P7+*J@9Gr2MgwLe&ioe zzvwUJd{o&7?MEC$f6n%Gk^9ri2S9eK{A;bb3qZ=>!-BuZ?=LK{-(Q1&ZMGIJAoJw@ z8vYOb*`g15sNVy@?+wfavyW-i-{MHV{C1=oKkD^My&jTnFGV}2Ipb~2FY)`aenAFD z%aG9L;vIG2g$!Un#%pb+y+c1}kJAVp>wj$}eoxt7LMsB5>i7MqeVwHbuF?Dj^}nlk z3gls6RA11?uHFXFoszDj_e<+Q@0T?E^RC{Bpa+#a(Y2t_MtFUdp2t^v<4B6fk54v2 zKQ!Ot=Tkgh#P_dJ-HL`jYtqS5GFZ0i^D!^t%jfrGgusXQ6&vjI&R0Rcri>g)5+}7s z-rug%WBHlP8YPePr#R`w{JVP5XvHjrpTy-Ad3ql^#xuSqwSlB-krk^c=-&HNk6o53fHz=!h9{t;HW)ytMsV8%O{xm zPbs0j;7<8y|C_t~wZ#A|bp62oYMk^5--I!gx5g*iHzc^dQhha#MuPHtuWX;SZ`giO zzNC|W8{*6S9sQOkO?=4`!p8W~K5NqBi>W-MbE=OrYihNV9_w>7wJjMe&p56SYaqwwDK zuTvd=B<*z!rIo}9{{(xW_I6t4$L+0V4EY+3Pv`IGXDVEQgBSM0-!tb`Tcj`BKZ->4 zgZ2b_ar_y+4_9WDy8H{-Pciae;;PT7W2n4N{bunM0Km}ANq9s50|f@+_Aq#)Mp9ue@@0<%l!e&&B0fb zuH*eqjz7Zhtr`57@^|HNYCpPu@cmw=z2o;fWq*z&$^Kx^M*q$DrWTPdsSKp=nshyX z&mKml=s53(r7}qVJU&>m{i zvV4yIyw3h#>_;rK%Cr2**iqzfG(Py7toEGDr}hsesr;**_Rr-@BnZ1M5Bl4z_FG=) z_Aj#i_9^=#{nevsJLRcKPacH~OJFOTy$^sLrH(b(TL%7AC>c z`YN-^#+danDMJ20`zPqzwcmUn@K(ntZCyV2r>yZqVIeZK%Y*VGc?U1yqq6AwK>r1I z>F}jL-7U+*?I#{3JQ^?bA0ho~d`KVLiJkZuZ^y|0CUH{vrGNa2ogVb-_zyhqhd+o> zdWE(;^~bkM`SH{qCcyQ$(@%zzkfl5*{W7aYzMmJA^se4Q#J}i@UyJ7Ijz1xfd>QFn z#`n74FGqWdM=5Px-UQx1H{;iE{Uq)9Nyneyc`W7+mVU+mj>9{-o=sW;+=|Ae~ z|H*zwar%T&rEh$10P{imV495|<8?G$cluHO9+2`EsIPFAKa=_=3=$u`ui=g#36L)! z_apiA{)C$!4pQBy@*(}G)1Pwvl?13QJNWam{84@g3YB}NxV`yeF8!nY0sfw)uD^su z@3Xk`j|K^w(m&>7W`7NTgb$qA{sesb{((Ebj~Ao@kdfu{#!P-VME*|MpMTE-SU{;F zcY;6okD7EeM7~|=2li4DqW2$^{{kHd5q8!O+UK4JLSZ}p#zN2qQJcnk#-0a!B{qGL zXn(+_$J6&B3-h_GNR!qttKknq6s80Fwe~!aR|t8iRN|{n7hpe?I-Y6+KjyXh=2$RqcqL(pw3;RpD@I``V{RUB9#bU3VdS^x5Rm&U*Q5rOZGHu&)LN3WOj zda2s)0(%Sd`|K=#s*U!Kj0D4uylId7m>>3QKK9b2-SrXkP<(&B;$IA~yf?1puYBGi zpD#R#>^<`LT+04G_TE0ouH(8ByL0Eh_wIZ1=6%gzFoUmo7`y=nAb)tq`oX$FqV?(jY5wv0r?`D3Lj@#1Q0Qc0iuM&uB*Z@P?nNavZB%-QpIay6&8zv z6d>dVqoGSu6CL9s{kW>- z{}ST*F%7@`AYkwZVswZAdYO2?HUW(EmyJU%sf@bGf)+|JU7*m;G(wFXNAr-M@F} z4}XEl;$c+8PthOrnP0{F_@3_2uVH^5;k*A3V?_acWV1R?{&W1T&y#VU;=Kp~tj<$T zgMOv?RbP?zg8wQVTaT*aaN|$pM_t~%td5fq^cagOt3>yx_#uFOb-v8=QS7(lJhl=b z>3n}1{x3YQRPl|gDeJMaCDj z=coOT@SZ~Yei7&g{{)FQit*O|E+12C7wr!>`x%74Hsd$6%Ex^cKJZKbOP-(OK8vq( zyZ9fThW*$j{_^+ok^$(C^QcaauQ^}#qf**`!TWw2<;i&!+A|!NWjxCIeEW~>czihD zc^>i6F`jL6o`&}4CEgy}&;FSAB=0u;Iq9tz63~C=w{YH5!Feyfzr}eE<}1!O)%h=B z$j_&bhk(&w!Xt8C3>X@i@DbX_b?IMK;@M1Ld~3oQia*4D7QDp!zH}ORU;GILkMkXz zXF)*d^Q_C{kA!~Y5A4@DPWsV>H={aBV$^7Jbm$nB&;xYcre+%Q^cKnql zai3sB@q_3K+@GcSu*dTzWPUbdafh&A*b@$3)Srd#tiJz?J10EfHCk&v?t#e|BzQk9S1$b-;MeV*il{Llwr?t$uA|ZLfDkoi{h5x&&BF+7JD# z>`&F6@JZ!wBt(S0f&5*n{CDL18*ov<^1d(YFRJo?B>3>jlHxbk4|@df?e6;bWEeI7 z#lMND>ANt#7|(l;`(9fXpx_t4NqlD}x&(XmXa(JAgMNeu;ZH>Q-rghf{T1l9qugFI z_)*B~Aoah%dev1q=s)u_J&Ol@0_OjJzVO9AH2Fzy&UaaG4$&T2iu!f_BPd_)$9orqCqTb7 zQFnuRK}6wS*3a@qMo4>Vb(XK|!}v8;(4SE5OM~9Y_-J~DQaPNm+tsg&zNoF4tyI}wv7ymE@}o+x?9(;4567~T{;2rvy1&)> z)}sZY6u%&UBd)LbbJHKdp|n@VSK80^IP*!$lm6w%pNQYK;=i@>uaS7#roV#{?^o8I zT0W@#4=6{|pKq4CNIJ2pr9ASV*Df0TQNPpXhfvD5=zFxx?(X3p1lzObrIH93vtO|W z`8l4#pK_jATk%I$dsjyTWv+kayEKLVkiWflPSua`@;feB5u4^~RdnJQhgGN1o|dN6R|7{~#aRuk#BJKIz8~coU3O21pm%D#iM+ZKIhBcOPaoi^fmT(A@g}VOfc~WLj6877=J0p{}cUn{7qpGpnO#w%Ru?C z_fmgQDZk>BKZf^0C{N7&1=QEQLh6(82nb965B?4HfxyzggFlx0G3cM%&j8EHc;Y_m z5*7#8C%tihv!P*)FX{Ii4LKFe^)Bi9k3%{X!iQfo3-+u?a)q9mm zdiwW(9=udNp>N%4=!1M*8r@sF}k2rK>pe<^w? zerx@R@;v1?IJO#C(zkvXFg@;D8-5e{kM?SQ!#YCfSv35n@pHK2*)Oe$PGJkgMVmyE2-wnK?SITcHKgR>~ZlHY9$COWcvV8fZeF?vt_GQMe zX!u|7s88`<>j$!a04(~|;tw@H()K8P!XHW>r}Y;!zheFJT-oz zzbtvF@Cm;x5jf?SdaCbGAIj^cJ~iV5xW@AP^$47nSC#apJdD3m$NWL$6>j$P?^ND}Od$1PeEELIAmDDUzlEMI{pzr1#XcWV z?|qT}O1^=fir>T(W)(_al*2cTyg>P36!t3ttoYTRDa!t)v~ZI{YyUH=P7an*qbN)LGNddW4UdW$NTsRy#0R}JhES5lj zo1St$ho3;NL5z=~Upl`Nd4m4o{8jiv_GeID7&ZFI@A!=Tk^3E*pF;A7(jRJlZuWQb zq=$@W&eSjSwT}Hx$~)F9_UrYD=p*dc=F$HoJ@%g%4g~W_>GxB|8jYgi zC#>&@SM=2OBw)pV%AUvmH{TaQdH62j{Op*Y^G$xN-$74(UjY1q@jZ2H>C%e9!#_Gq z?pw8USGr{yCw} zp9Eh_&m#rzr{_z8_Zi?qeTYB1`Y86p|J^Pgp@=*SasOQKT;y)SehcFJs`Th3=)2}9 zmxJv+jCsR&-dw+vpG4-MKR91i@ip)p<>Q*DDC-fq)S>WV-wF?cA1A-1r4{!d-yX&O z4tBTy5P2yjai8Iy<5#?mtcy$Z?`Q6BApc(MKgs7KKk(*$MtD{63;o$An4fs0ACG)_ z?@t~<0e_I+gukdSfbYZnWxQi4Rqltl`=h*nA!o_d4@eJwTSfkn`HcQ?|C^+f`(5BC zoX2$L+hh8EkKFg^^tWjEQYX$+nn=x0wO{Vv{ab+gyoB|MK9KtMVE;H^yhueK)W7v; zPS{uW(@;O|vkCnJ?x&-F*iX~<)7ttxM4c}c7(WE-aBc;eSB=1AAQJt<1&2GOAm1-|l?ezx?;U2t4mUfL@pxdsP99Zi~Hv(|L~qnNIc{9XoFPtr_bQLPULNPTJBq)$9t**>-=-^KrX2d`Jcmn zIq~2Zb$$_h%h=uN#&pQ_{M;{rcT(m;utGVBe@|3(gqRYen3jH6ZcT7wC zugPQIQRfkQJihchFdu&t33NQ2xp-xN(mwFh?95WN7iae4V@$*er04g>AnMPrl>HPP zF9`Lk{TB8I`2kD(b-Z`bMS7ix@&oKIKH2XF+^^uaz@Nwc$CTcKxQ|lE{|y?nT#v^r z>=9)Tsrhz3E((5`rZ4J~_c^BYqIJNIG6PZzaEn7erpj{kijTSMVZ=&+GQgekMQfy$%kd z(eo1Gt6@Ko@=W+$(}NQP@6`xB&cR-z>^1j`E=JU;sIs zFJL^GAMe|7ei(XTyk$e!%%90)Kkg85LH_>BLC~`z_kB$~f-{I;CH?`^x;=3z@XYw2 z{Mk6!ofG*#8`Ijqs(zIEixq^^u;!ya@742TF0RQ1e`%kjuLo_bd>`jqqU*H2!2Myl zzm4$|dy4V$lKvt8A03~sc&w1+WBhnuw++CYr+)32*Bi2RHI66l`^M!^&JI}KD_9T2 zr4+_~>{z@k1NUd(z6{)(fg33-<>OsqDmZoij`9)5rm11@dwnLS=-pKG=KN)Hd}!|= zs8O`FYYHVRvzasKH+ba7qkJ$ct9U5mOq)_x;;tz zppTsNpl>$*xy`Glu*bm^JuOJ=-`5#W?GM)||{E>ynXitj? zY338$1t9Rue4IRX#rtc{$9crBfINl#SV8?|&MzUamv#U)@`d^@;U~!F$zu~~{;Oi| zIrovd|DMcm*o*XU^0(|z@hld(zb*bWZ68(Q-`i-9ZBK#!CXc-#M;#EE5^tjn`&9Nj zbUe@o-dCpmr_MX}_*;(#^BfMb|F9lnda-BuJ~!5*KHj%w@5P=5{jmQp?MD`_Z*oj8 z@>a(;qP^Uv!qE0|+1M9ce?uNQ?|(r4WBuLfDg7kt9qhM?e3$aEKKSpRl=*-{)p`i! ziG3sXGW5+@=!<&c6!y@s#Oo94?HPt-SH?ECkkM66$mke+(}O1w9=kK-lf;XUVs zUN!9RV?QUK#OFf&&^HalqhNp4{_G(7 zBmRHT_`}<^eP7V&HGHY)wO+_$f`XhfzYuTazejn8js36gr_2758ZWP24hy2M_`WN~ z)8TjU<4M%i+1BlWy?*LgPzY%7P=CMWdz0%&2;fue3D759pP+vOc+Z~0EAKO29>xAZ z=xKWa^M4BZCXfRsiKl__Ks`?Bkeli%>>bKlYNVIWVRIR|sW#9w&)3Gd{8 z<@uep9eRA$xb#AK+47Y?+q0h6MEmvpllh|k8StJxw(l@uxL!d1vv@x~SwDb({Fw<% z9DLO0=9^(YJDmI5k8%C;Mkoe>H;3{@7qtKB z7Oq7>E8kV5`T5< zCJelz=AZa?PwV-D{Na;L!^C1WL3_7j9pb+v<)gkv zbjJ|aKjQBIe~nf=G}H!+^@7-28-cLmT;U79I1`5l&M0(z#?hmql)Ypg}6@PMF z9`-N&Li_LMJ7qqzKBNbEjDH;Zo9|58Bkk=)dqFIg->Q1GXlP&vf8OyXmMy@y;t%>0 zfHi&4-@LR>-hUMQqmPo*f|vR}_(AWg@TcU*{Wr8Pzsb|4xwK;7=p} z7$k{}-+|?V!L$6`*nUg#pN`6{wOEm^OWuzhMR_T{5s#xhJO4J6m*SDXyaD;w2W`JFemk~H6Fl-4L(KPy-3k6Zu>MngBi;-3k=9>~M)sk@)ZYc}hc)AH6_kOR0@169!7ih zcwQ#|@c7BVe4pXr<^xEo`a24IAI2xex8i=7QwcwS-`)FHt(ezg5OH84I-`a>f~zl7r{^SdWAKc=ktQSE5q{!=#p9mwC8 za3g`k5E&*8B>5l8%FF-%)^`y22Z#s#i&1g!A&xKSi;TyiLK~py3u%8CK7{looj!~newh3%AMXD(<4212 z{HjHVKFIPO)$$$lO`S*U`G)vp`8M#T{ATpX>PMuo3E6*WMjJK=zp43(_b=D2bKyJh zbm2{#Y6_HuY^=@8Ta_`>2Ee(3ph}abNh6M83%Q>}GK~{tox^TX(jc{9_y4 z{Et4G$RnZ0__&Q9k2Y=E=H%b8!@{HdEnCJ69{kW3Z`#DCzk~+>k4K|BMlJk!G(P?z z3+{_YM}tr6^r*iWtyTA(G{3?AUZ1oN@^c*PJIY_|uZh1vGv2T^;V(QD_|9>=KX`9vLqc!#5B5MSX&=fD zpWLWoC1`nw_k$iKF~Y=N4A&*;CBNLC_BLs}(?3=I2Ie0%`J>UiWxsh_qD@Z!bUYvL zPNqkDaGu;bf!=8Q7x3sPn`Yo5uuccdrfCqHS`%wN20eX4Gg{c-WKSsPL=AdnY|NsaOw&o>6hP zjF-kYqdV@f^N&Z3y$!Z6nNKZ=x1jrn_SE~V^daJF4uy!Awu<}xmvVf+X|41>kw?7m zZ1?Xy@3HfbNA=B6cvgL*{WhNQcGq0kpyRC-K%f4l79RZ9dXGInTR2~K>su=EK+^Wt zmRR}2KD^(Q@F&K*iFhnFK8*S{+2h&7`%M-e_1Ba=pU`V~*v0quCi7Ry|G)=eP~e-8 z@$HM3)ZF^Jx-2}(A0FX?Bpr{fTdndTzm&g@>^DgH+qYYI%-7-JWWEZ%*5j7nzg7HO zllJ$NEqpj0E!_?S$EyGS`)$0~Hz3fko-fU4|9%(W-%t9b?JIX%c+&5FnNLamrILk5 z{XP48ocgO(8{dcfy>9<16&tVZFKU;aSZ zY2T6uEj-3&X+P!9(xm=}9lfVA~7r*2n3{pjZksnx$p}ZcJ2e>cpE+_mW zc)9QFCGVNV2M#3km-I`PB>e#ld-#*!Y^9Bz&{A~W2#RZ@sa~9aw1>&zC)>6 zvg?cG{=e6+>5Kh2b-r^D>kqVt@?-tP9a?^LB7av!=*jkB{Q~^9k}3afoS!58x(g;f z;tv+T=e-T{4*R=u{?GeV9jIUSx6r=D_kZqUmqdTUKgJV(8m?E69_KNYX|`XVZ}NRP z;hzLwsKgc?@dzgX;sY5ae^_}2kjan!%6?-SFWC1F125}w>?h$pA|3D7ll@qn=Wufv zpR@=2AzciRr{O`sCGAgQ@I{HQq0&!bzplG`6;hk?M8rFmc*N#?C-*%h*5B1d{=+Bs zApDKx{;+|u-|tgk>-J#31o0Li?`6{o`+vA^S+?uLdkM0C2!B@DKdaAlb=i2*_b5s; z^uhhzl8wiENH{xfae}|!s$9qMY{uZqLTg6XpwAS<&`(u6+uo++6S1#J+kHvw% zI>G-w_rv_51m<|T?-^nLDopcNfX}{vWWVnu`~T{KtsZaOhb|=fFL6Bc`J{iZu)q2| zkNeS>uka_a_Javq?}3B=ai2PGmxuE~pZf*rcvR4Pe(rGB1AX#NDhWca&-?>AvVjWYzsXD(vC=PNUS&G=;a-)DNxfAyYL zm%NWffvDcc>caa-1$7>typQDa|Jf*qic)^g@+;!kRP6TRJ+aE?oc37!jPp6Rhw@du zH`R&u7b+*6^fo`Amh{{YGVf6_Kih*_m*{_i@25HQjr4fVDv$GZ&VdV+7c3a}LHsR% z_4wvEfAhp&gC3M`ThsJ;$~UgR7Ud6&C)-PTpy*#jd$aT`YKOxmqo}9*f4gI|d%=zKUKS!QnzGe6ga8d0SB=XoA zf6&97-+a%`o!?n`O8Rj;6u%<=c1FGt&+{y0qLm--1v>mxpK;|Y+uw=wYCf<$OP)%3h01xSJpG=6 z=1)i7fWD4AMSkitn_oEJzGRh0eR4NoMXy1`$Ft-q<;nSsul7TX zzI6G`k@q5R?fG~b?YG`HKz-@^2Fl)W^a=2eKEV8-Jp78)zO24ksDD*F2xI>^`s0Y| z@7JyJ$X~8}cH~EgI^Q<-4%_eAN6vf|eFbyP=$|8;-|qb7cu=3H`9}GjuYApF5AoD5 z=KV(3J{e5pznRZB(mx#EKgg6%eW3WKL+JZ$gGc+xpOgniel!1T27g)c6X~`1dxtEq zP`O~z3xDM+uUIhUw_U!=|5K>nmCu}iu09De<7L}Br-eSYymRCu-UlRoS-)w2hCjtV zpnOu})8X=u^0yKM8;zf5d#ak-yN5@Yj8B7W2bh-%Qbe zY?a6Na=gQ6&uInIzgFOkzX|*|9DiwyZwib5z3ZRyU96|bJmy3CmE5fJp2h#LYQcFr z=F9x5mYm0j-pDxS5Bzm;eh-Gj`#rD2U)S6>g%o>b5b#n`3LoG<8^wA1${;y!2mjEY zvj!|4F-3o5{w^i`MP6b3d-`U~# z-+~^>Tu9cByT?GuL>|x140AcoPx|wv-`Skz2Y=1R>tw4+QlS5{Gw*ZK&&J!lwjl8z z;$!SCvI2er58<^%1u7V~ul0y1xS{-knB-OApMc4ykNl1JXFebA*wF^8^oRK0$b1!l z9K>VC`d7o?r`JUOOS}+dk@^dIy$*WG{oIR@gj)s@`~d#Ci2LJ`R?`RP+2Zdb`@7$p zqCeL#_vdB2#DD6b<$u11@BIuX>jU7)kAdV8YrvG@5^&eOAPW$)H#Var$bbX{h{HG}2KwslO zJ}Bdn_>Ut!@X|iedoF4%7k@GR9vJe=e!kEPc)){eSYF~Uu4DIU{+)|Q;6JbH7ycSZ z`3wGTb5SrdVCn-N<*9fBsXzD%@kdYm!2=mTWYPZM!E%|uUR(QvyZ-m*<0S$8!3RkX z^q>Clw8!wte1N~W;MZvUIrvWu0x9oHs`jM*;m8kvedV7|e{b}k{`!ZJUi`ZiJq-V9 z|Kt1u|9#Ro_16cU`Heq(>@AV;N&L-oQa|`x`f`>6W8 z<8#_S4E`rto_3}0b7DTu#XVKZ3)A1ZxL&0^N#RIu6^B499=^vi& z75et8{VDXfpZp@_Lq0&>=RLfaDECoB-hVFqKY%y#9`Z!x*Z$n{l($Gyg6*5H0>Ww z|8wsD8vWS9dsjq@{}TA!@h`{t;=N$G&mjJMm@imQr2gE%V?T%C$4~&)1L9v^&cMwK zyetFvr|>H*uP=qAyxt7l$iN7XB^{9Z#UDJC2g<*k&BG`36#6w&euTeoknt~1>81V4 zv*oG%Pm~A#aOkVkEB<1As(zk_;M4WX^XqE7DPIzOsN}a(-lY!Qm+3Y7QS#r&KFs33 z{FH=WwfxBeJO1Z@9shH{j{kYS;`*N>T(0APo^N`N|9O6y>wli_hrwa_gXjBP|MPsW z=lGw4pPHan;(wlRc#i)$_{-qM{~Y@T2~7X~A~s5%b-FVJ~C-02Q~s`#$G+Qb!?-^*%=_I-wf%A?8dUaIavx+#YCniR(d#*V zAByvX^J!Y^Pr_{P^pST&9}|}L98vH45~hKO_(p<15pRj zynloGpq9}G^;94HK=eV))dwrW{~h_2KG?J1JxXW%!4Gp$b)}qVY5Rcw^;_6r`q%#u z<8|r+3&RH(<25_8QN$~LkoKR6cBc9U>leuTjfuQR`IE<5WyPOrJZKLLa=oYDTjTqm z$tU`VtKU=~or~XJmG!vR|8wzsYqoy680|{!;~(Mv-w#B&D*nOvL4NJn$ttwI$@YKp z*v75}?R(6u2f2Q++Xwk6{`z+Nx-#txcalKqd?`BZr9OS1>aftyu|Iw^x@i050-&_N z)1ALU-;JdO?SEXiU)EoC`}f>j`yn+Fzf6GuM!SB2(_g`q= zte`z0>qVqD`x_%y)&9oWePNZ;(b&UhM@kNC()V9g{<4t2jLeYmALAq0_SLz4f$Tq2 zX+L9rK>kQP23fD6{%786&Ces)AKq2e_hm$%z+V%~L2R(W{+09SFTDr$yXha|myXRe z|fx|#RI*G{fqn;w@J7+ zi1_DNSJuk_tv1-+F1y>VVb<>x-pc1b*BGk`-^FNK=0w5>b^;h^n*Oj z+DA8nzq|+bCa!xY_xA|f_7h>-ej=RMPq;6*9^-4|+hZ_Ah9q1GiUd6ukw52m=JTYF ztapSy1x;^|0rWUGl27S#1@`&~&KE!rvCjbykEs4ugkP|}T2VlNYJQ5JzO)p_`G$Yi z-uiyzmq*dR35yA$-q-3*UEG!=2tU1E7bIm*+u`hJ-Uh`$&(4^6WhNM|#Np zW}%0^Z*TVZ`qljm~Lr>`%h87Ci_X`xfmJys>9-|M8h3=wak9`9nP)f(BfJ zKl;Ov^pN$j(4$As5BwkayA{IzL;vMIG5I0v?V(1J_js}1y-|O%_%Bz{{tKQSKit+k-=-U59lk9|A**XWP3w?-{{ z3pe2w$uIFfjC}=rY4TgINbc0W8c_b0iG8(2$wRK^m3`HJ)AkkQx4i$VYf$Tba-VI# z+?laow)VUB)ppCi8eLRhZlOTDwtaGK`|2gOuW4Gh=zav^0c6?-cB>ivhW5lor@ga? z&(by}g8r_3wQxSuK1}x`atp>A_Tjbdt4rDTPoY1nQM*mwMcUuI-S(eel>TY^7uqXm z|F!L_A@2X6G3LAUCpXuAOn7DA-YWg89gErLY5Nzx2auhAuD$hF9a($DpXpWi_eCCK zJ%IISVh>Rs^F9su)z~kN{R2Gg7g^63{e2?pN$nNju^#J5>=W|WHhEqN6ugFf+^6JY zYG3rZ@^L5JX^s5=c{#RIt(Q^1*ayQqmx;W57xn_f)u!@r51xZd>;=e6!il}0;k=fY z)CUXfg&XvDr|NHNFJ$z=h|&jQAF#iKMZ)O*j`T}^#a_4_`)ITQ`Dpc*aAFUje}ofz zK*RT0{k81@>c7Sf`a7cfo7w}yUdn!?{_970KcoNI-~LR0`}?H7qW^EFzk>}oo&w=S z-=lwo6Me7YXRZGJB=tS!!wJM6RO>mgIp>?L|AB|Sj{qLp-sgNH-i$x+=dgZB?0w)} zd*R0R&(F$U=)ZY;A>E(R_QJPB-pCsrPhIn%x{m;s~8`}%@jJ+_T zcKc~B+(i4Ej=sHx_FuzZNZNlp_JZAh+6y<)e%D^Oh4x>=UP#*ild%`%JS4RjdfuJA zVC`>+yOe^CKRWw^KgagMyV+m(SDfBD z-paeT7qa^c&7boA!kUb|u-aL#UyNEev=^H1%3c@|>dAk^URaMmue&~PX6yy0y|TZc z4P4r5H)Sud3Ab%8*zKpia1-rsI`+aXwEr6RLel=*u@~(2(_XlV_Ph4NEwuj{_CnJB zpNzdA{uZgd@Qb&h#>p3Jdfzl*b7d3Wq(0iI5%xCK!4x9y}-=+yF`0o zHGu2dJN;hMu@`Qk{nxM;P@Zc4?br*)#%7TSLed!dQ?#a{c#*bDCd zLeGu%7kVhZ#UF$FquKL_ctyIu3j6a!wA{Rxqx~sX^ol=y>QAveIj>Ou6f2Vb)t}aW z1n1u`(|&~A-(Lp%!tU>XO!fuOYwYt1p4Skz`|J2G{QR6pyqo=n|B=)6r?$UPQ}QQ0 zkN64uFZ5*Wg}Sp|bna^{;@?Dj0TJVWmi9s|V=p-EmHmaEZ+oGZu@`Qk{XZA>LM>x2 z+(P?*Hui#CS=IXs7ZLwarYG%P= zlt1zf{TTxB_e%X48frV1?WO$OS!}re42X9i4sB+?p;qK}_U-#KApU>i&w%#__5e=& z88n>uGiW&RXMn$-?a$Dn|D(9;qe!kN7W$8$-i!NMc;WpR+~0wG^Jn&@@0VRn z{ozCH4_}VGee};peR7|2@?5XC6R><3ANX_iX}rE48}!Nh8v6dui8LNf=zSaU=Vs;8O7aj)Q>)`v{qu#y8J3Qw1I#hWz+{fbjEWCnixu91a@#}@2Z^(O{ z0m{$43B1H>dzJA|J5~Ib3y`n4&oz1KW5DuL?VtS&{Hwo=63DYdtMDG13MVFm4tmqS zq{a&Ldm4y07rZk+UzK= z-b11}Bjbhk;ypKshcv#jR`#xRYS{M_oYQci;ev+q8ZK%$)NqG}OBy!il{IY2t0)-p zq$Z26dA$E+;_+quZCwAtD&NnOcw~m(?m8u%RpSMIn>_V7b{?Oi_rmh*^n-l=L+%&r zczZ(stIW>v$9Oc-`_Bcw$2ECsBeD3*c;Y@uGM*L0<5v#axGyI50op6~Go`&KkK@y3 zc8(A3Gv9SefTaDycz=`pDgN1f|KhkO@&43!2z>#Y@iDL&9|P<0fq(Vnse`(LYth54 zAM{Z7=Qy7=KVD!G(gW?CJk`<}7U};Hz87Kj|MDQ>rO)S@$T5GRE%#H{{|PBU_n*Rz zAIr!1x&5#D*X=+1S2X@FH`hOTPa_$xRcLQ04uvvakZ-78=9k=e&e9+EcNg-LUL0_~ z!hilb!~-FRrSfy7C-R2+8s*ocl`CaGXaxPo{F^-XnsYyLEdIpG7fBC2zt_e7N*y0U z&-dmNONhlxVw=VAdgq0ut?U8r~KHu{I9{ORG_oAQJ zuZT~!@jdv{3XfXsq>@1TE46I*XmJ^#7o5cc)I>vmz}>JmE^vO!Q;Mlst?ef zfw&>%I)5JV*1xy`@fxH(=Dl6h ze#BEsgUZiOs`u~8PAMfdQlDr1bcrzT|K*v8w-s4T`FMQn2k9b)tCny_q zd+Sk*f2nN&0nhKi?0r`lsSq82T~31>1}Ffu%h2 z9<{t@BK%P{@kfL{^(bFu^Gtir#$j;bDZu)D8@yKygEPhBy@Z~_gcJH!W&H5|rO+Si zpOT9&2)?ZGvv{vsdt6HUATL)kUM}8)6!}w+s{M?2VA_NC7(%-|r2ho(s~P!Pj|yep z$4}Fj8=3SgS>JTyM{FPB?>)TouuM?JA1qJdBWXY2k5j^!@&R|l`~u;POyakX#k*I&Y4Hc>8`$L!#5JMc(v-e+8;|lo zqT2Ukq2IcA$!@#;dQ@9lL;DSVocBV(56Ca^s`Y!z&{v=AVRzE`cDP}|UL)MK3nR_= zU&22apPJpCv9BUO@P^;V{uKEwnD2i}{tM6Q{C$PPz#D#Ie>8ohKY)vByjB0$UWGr} zL3uV1SA;%G68aoD%=jkVnm>{LVKts+{QC2m`Xrv>bOVN_$b-rftV9!fwc^J=$?{g} z@(`c0WYHrSI|~@;O@C2e&gN%~XZ0|!R{tKb`KuH1c>;L(P=70KR1O0$>G@uxQ{Pts zOJYU;vG}7-JQL_!-%8K%eOS>857D9gv6qnE&i|)KZ^204soJmZ&$2%%|2u|%A5id7 z1Iu_%AHsX{C?Dx}Q=VG$N7-+UAINx(#gEkO_Kn347F^iR+c3*h^{M$Y7LQkLJmMSH zEI1gu1la0NEB@F=IeuMc{Jx3$9K3o;Uf=%>#{LxTbMSSq4cOcd9(xJxv+#)TSh4E& z#?GRB7QTUa4kACbe8787kFY)FzG@?`*70H)-olpgthM5U4-#+g$BfOQe5*X_8_pkj zpG){<|2Wf|_nQXd$3;IquHT1m#h)k>{|VLqpb>SKMW5`F`odQH@e=VLPw-tO7yprv z_>Uy`V(8)@4v2p^!RG=O|A0^Y0|~xU;zjHCvQYnHW}>6ZU%B{zThu#5jy20*v+mpjSu0QZ+e$x|iEc<_)j~@!%{NW>R{zHLP ze=rb#Lg@dAg#Hf&ZvNmwH~#_O&3{DlA4&3mK;n0&{rAVM{AeH6kLaJ_&w=>klK2I7y% zeEA6H%PX`uJ}B@931h!yAbwcjhZFdqzy}lf0f8S#;Qa#cPvG4G?B%p)zBmDT2TY*%bGIH<&*_-* zbMbbCrr`rHK`x%54gUC{U_r^`QAhh(eL2Dtw)tPV9N8OZE8KmMU#}L%u$2S z#ao>E%4jd!XZ9;{@kR&lNAlbf_)nFWi$@%ME^axn9}PM6=i;V=&&AyRz^D7`N6Rwl z$qgw!7ndE_kNPs{xf;Ny%j0^aNl>pxa(J&+zNEfJCO>C8K8?@r@BFTgH&nZ#;=}y0 zDen*l1^Z#oM*<{xJBZe@HktHRPmRx^8x$4 zDL=kGi2X-*J(=<^!gOPNv-~~#*-tt9kRr}O9oAY<8%HGQeBAS%1;(^Wn^Cwbz zAwEOv1*bo~EHBwl1ASUAxb1Dd?4+-y<*WE(%%AMfkzOxXo&L7I?&P=Hr{d)@f3hF< z{c4bw|Ko%{_I|esCHU#|Af{!bP?AKMK4a<{*4Sn01+ds}Y| zy7(&&-fLZ190($N&0`O+A$Wdf$}87D1M&(->m+$K+iXx7-V`?{-F2tzgdCux2f`mIy|o5AsN(n zX^P{yj_F0dB3?)~-a>Z&WcIUGeK;R-<`?*f^5E}`{*n2&E~O9VOK03A`+d@YOQ%`C z^lzH;3F)8yizdczmul}z82^(`Nc*Hee~JE_YN0>(Df}gwFQZBOS%3d73Gim*>+CJDm15URt_j4;tJ1pL0G}OQZ+uGvzz|NB?F&JDtu~@E7Uv?@jr@ zw@!|x<)i-L7d~|d%M*UNgm_2s9o~cpJk$q%9zHp(`X};h6!fdgdy0yFB9A2hG~)-7 ze&O)Ozh>tL`vdw=LmT>l-m;%EeMs#`HD0PN<^GiP=W-S6t$CIH!Yt!;1m%~- zazTARNZQZ-&R=*&weOqN0n(?R^toIO8(&@8O}MP#&Ui2#&&$;g;Foeg(5{d3U6}?{`=J$=)>dmqeqKBVlO#%y&x z%NtScEj4DBRyUaZS6N?m6Y(2~$NQG$lMBjo`TeDy>SmK))vx$Vj}Pi!p45;2qyDWd ze~YRg{9CO;+A4aHJ}2G&rTP)}R+gW@pIQ6JkuUTgIBDrSz_2$`{fP6ur%!Fi{C`5t zkMk9_Z;b8RF72BlT-|Z4_ASz$d`!a{U&lWc{&eIC>MzIOcYLycx=8-XQ;_BIwdMSj z{NDs@&Retg>#xGU;593K*53VN#$Vt^`!-u2_~*IEll^A2r;Gj;fYtj$4=MOW6qYC- z@WH;IKYX!EAMwJ!KaUrGg7?j zFUH$hztDfez(y5+BXpAMBdkVULf*I_K&$975|9_{6Bd5Vb*)J zm!*H)4@CM_EZdtM@V}Gvh#v{@+L6RF;&~?JS0g!(@ML@Uai)Kb`LSBY66ZHme#CQx zcti!CB>bwpXV{5=-oCuYiugZx&#lJxW54Qc4e5f5?8cMUTf4dZ0Ma;~yIYF7F{ddaqB91^mHqzBWCeu55br2|fB!dI*0UVSg=p{Cb8S zhZm(sPUz8>&?9&2{PEtKqer*!$5W(7`TF!&z#n{G%BBa@sZEcH(4&&lL-<3^BP@FS zT819Kx+pygLXS#9kA?T2Zpa_^-yA&#gdX#xNB{Ndv4B4YGW3A?VAErn&|_Ij58;oG zvA-5QekDVXk1k4&qR?YmLXU;_scuM*kK7zRRw{aQfF3KZPmcxsu`)vsh&r1dtArk_ zQhEq~$a$1Sj}K?)@u5ZOA@UUOcfJdG`kOaLkJTwXn%Aet0{&Q?p$E)8n;t_#kD-(v z!XF=Ge=T}EnxV%d*P+MIEzl!9zgPZUj(oiZ{X3k}r*(b$EZ~>n41HkU+Voi~^jVwI zNBBj~CoTFM%FySb>(FQIEz-wF1C2d)yYyL~(r4uQ^jW|^>ofF$`EJu^gV1L~N+02$ zgY3UWpQ#Le9=r~HHryh8)OzgQ@z17|J{zx3p9TE0DMKGDPi*>Z7W!;X=_C9j=hqf} z4rJ)_f$Pv`^DWTlTpXtF>A;>L{~R>-E!RggJ{}qafAM|Fi`*~u{0XzaA^uIi|E7V_ z-qc=J`>TM(-&FReIA3pQA9jkpyfvY>zs03D-fwj5&fhTb>(M3 zz4l$05%`DwHV2RSFj;&<)`v(W>*I@2=^rBpKA4^0x9ocGV$}7Y*!b*u=4{;SO|UM0 z$dB_7@kgxa^SRl0i5E@?R^GqEdn>_r0Gjj5xwsL0*MZBy8xC9z-gMw)!P^eJJh*DX zpNIdlhb3gmQ|E(chOTDET4us2ACOktYI|!ls3P z9`BR(TKRGREx7E!OM)v7JQ%#=z(c{09T?lwAZJRSmK;#0@NmEb?i5}d@Bxz)-W%|~ zL<&#H14=2pJLq@t_XR5)_ya-Hfma7D2VNJ9IB+xAXv5LSDsE_+`p(7y_Osw`_>LKm zUDyvu;n|rk6%U!SROpNM>McCa!)^F^_`5cp^6$ciQCi;F_#O<04KKs`A%*|18b1pN|{DuBQF7h}YZ{v}xaRTz_=R+f@=g%j@u-kDJ2(CO_&c zlmAqGoL~22{hQ{0KHdKw$L8`v++J00NMC0;e<8bXXDA1i{Bg`w(#ixT^MkDq)#XMe=qFQ zw7l7wfTvIVNPqwNcw^Xh^WW1HH1p?>|8Cf`cK%(S+((uCIIjagt&{X-JV(M8-SY0= z;FPy_uUp=L$Ng%2XwN99MyloyLP(p1KeL#^v!v|AJ}feXX6zgaLexvC*1P)!{A8r zpXGa67JZ+GJUncrKO2`Zy;J(5zI|J){LjN5w{4{dei8HQDbQ|7*eydMl0^@;zn*vHuK z_2D)R0j=y=oF`5md#3o?(xJI7@9L;)f1v+adQ2WW7;?b@dqnoLFGf~8bEL=nZ>wOw zCgacdsX%LfBCjt-13I1N(Kcx#0KEdyoKU4Sv!0RalJn84sV+V#SwZFh`1=eofr<;v; zJL5MSzbA#?_>&IsgRd%7{J{GhP>}pIzsUInK;Uto7V}H)rwTvI{Hj^+AItotJ8_a8 z@0a3&7{;>(dj{$GUbp)matP~_AIs-^$YoitgvoozRh%Dae!+SK_r17ZY`uqkcQttr z8T6sQ5)2~4kAU-Xz$fb^;6ERi3i>{a_`{ux@0RzH?_AW?s4 z-tR32oWI7OZZ5_HT&6w95Bd~}z>9oQ^IzV}OW*HBdgj;nm*=9|-TMfq?)06SDws&tY>)s#Q81sc%AdPa6bn?{52tu zjQ{8TdVXWQla3GkTTuUAmM8raeoXEo2t8rH1~PxOe<LR&ZP3^YQ!SJ!jmXkoMC*jqg7r{)70lqP_F9C#D;!y_jDz{_N^p=Pwbx@EDR^X&`7?QJMc01{K>S@1j{@brkM@X+FZ!32?+%RdnLO6-jPIFf z#fom^pC3`}r9G(Z9n=T?cP82>@3qY*?+tGn9AtX!PkSZ`^Am(W%=}YGU&z0ga8s3c zF0SFh(j-iFj{0}vM;Juy#=bIV7Gf{8;{Z@Hrq9wsy7L4&x?@3aB zVtgRqLt9^*jYlzkjJPhqAgKqve10 ze&AWGS8RECHirFZ$5qL?Z>n6W}r>~dEQ;CLEtClVf|C0z&7P`J$=A}!C!&!EB!|~|6_!^nm_`6fN5_i z0}A#A_|f4v;BR$*CY%1c`!yKvY<;uwa;JUI$0Hbzw12Qq3+Z~O&h;X!n-q`rM#&xz ztaocJ+$xE@*5mVhJXEt`@XJO%zisG=`o#Z6*{hvgA8c02IkA67M^L}Q!$s(rS1aAq z+lI~rKNbl$H{7!gFzhp|-vIZoy=RMtbKs|ZUgVc<5AZ{+d$wzQ!GnEV7X6^hqdjk@ zr#;E`i~PMp`IFc8$CSJ|f%nvz9v|8x@+J^@Lw}36U_V5IgCcJkPZ0ZElgF;6@9zSJ ze!cHI7Cj(;Nltu}zf-^md+qkj#rG?D%>I14%Jz$YJoHL;a=o6?*UUE=>p|G z!sM}!rS)S!8uoo^KahW6k7)$}`S$xAl)oJR^!~!pa5bBk*u!$&46xD{Zhp~sqR_JX z4*mq9|2h9<{%ZTw(PxnNyiefjE9<@k+Kcq^zL?mzxNiY@7kXTfS@$14A??NdgFjGq zK4f9|uaw;W$@EbDgFOKKs?hAOz?OagXS8=V2W=2M^pt$3emjiZxX&u`d@fp1Abo}Z zkss^J%@+)e{$PE&8RmI1Kd`>tJb?i)^9B5p*^j&r;M4Xt>*M~T zDbL4vQ~wBF+Edzm!ND(+`)Mh@w%L{kKK7?LK4RaX{XX^&lKN2I9PXok4`@>#^p~@L z>d*A0^}nL}Z_5*Z<`JiU_W!a~9^Wrj_2K_eKkK{V;5(JQtnUM|{?09eH3`6JRkS_v-Gt0WBr+3nfC8++RygfW$`QNcaH`0eOQ+s>i(|b zACBLM)gHEwSF6(YZAdAJR+v!p*N*<01Uzqz^LnuXD~1{F${*|6$)a`|JM9{3+Sehc+)rpEv!7 zKJ#Zr7T{lX@Ixsc@*eA_^!^(3GxSNo@m2dBqsjV8?z2H3&qW&@ed6dF3Zk~ zqE9w@o~=(t!e2-E>OS4aVbAww?EfW>{h!q@4*Zu=J`Sk3enPi!e*^0mtNexfDr^7U zDtznqA{sU!9oG9LkVh7LCfeYvKQ2bg)%K*e2i)}!_2TU> zfW~LWvk*)GGU;(X`!KNvPkTRIUv$vEb?kH4>%GbP0QI}{=#=|#i9HTH?D=r>{tfMd z*RE0?JMrnno>C7`s`dWL(qA6mgW6u~Q{5+w_`y59w#=13Vfd@k-o@?ZTa`E2 z^tUd5TzNAayYlEvG!nS`ANe5NUl~RJL&cA>|3dw;#G}JvDnC|oy5dv(i}?le+w;39 z@%YU6X5|at%f(tWDG%rMBd+}DFJ{ZL`6DY|xL&p8$0G8M+fn#Td#g_QXTd-Aev2y~ zs2^)s(4_m<{>%dTH)6NXkr&X9oR6tI;{HIdBOie8PtK>_lJ&eb-*c2t0rQ*nV3yvP zpO_f@*uPQA2b!mrJgH9!TK7LIzb-}%Zhzxr{wY^ph`#6{`)fSq;R1Pfz{az`>G>Py zf7b&x{siK6v3>Yt{o%+TS^p0V$b9B}1pO$l8v94|{Oxq-qqMJgzbX#x$NkQW(Gr}G z@?(E!Ut03)2cs+x@@YjPKOmow(4=2fK8@J?bun5=e|LNjYWbv(8wq27K*^_6{$=G8 z?Db`dyhi&Rc_aO~j{H;fDD@ior{tO3S26Mo>-khZ;r`C#vA1>n3CO3voLaD+&&r!y zg|920URrEF&6ZEl#|z}s+Uv+C%A*DLjGA70yyxQk*4pEhoiA9=I`V0e`MlPaN3J{^ zg*>>ve9ForNB%)S!hUh&&q8@)?Kfk*v+^k`kFxT{kuR7}uKd|cIF&che}JzmZy;Z2 zzTo40V*g!v5BVYD+{llG@})=AhnbIy$fwe1SdS$R9`lOMSiDllqjrp?@PjRsZ7hCY1+}H~RRF=@*qZYi)h< zZtbD#${X0v=s)DieYSmc19@^eR0|T?JHH6>q;;$Egyc`gb1vTAdYAH~^$+qSwO=4l zwx#r4RGuhz9+^*5(D&|o3-d?2hpG9th&;J7Eq_sYvc<+PB2Tv3c*mZ(X?fDR6?w8b zsZYt1(FBIPxe0j!eZudW@}%{%mnYkAL7sdsR0iTNiaZ&*Re3^lR*w(&54R1yOL;Q% z5Awv3CmU1xE-FvBJ%f+)V^MiBkm5N$3)WAYZ2ThYpA9yC5qWZljlVs4GIT5QWF)Ci z$&>X7d~@boduVo-}V&o^bl(Gx~2! z^Igi5=1)zYfZyQnsrFO#e(x{f`e$uQ??vTHUy5H;zQ}VP>3A)WFYD6$i`pwK8^4Hr z8Mg5Y<;!4Zza_8ZA*=lt_&?;hfD?ZWy+7R8-?Ze-pxXbG{ZXDD^bV!<;r*hx?CfV? z|7A^zzbX4gyH696bGlH-1{wI0f0KV_r$ z@5F&o#p{57s_HK1BjP7qjB=@ei1J6xF@^C*gnmiqgPh;hmA{FOFEM#+!uwyX@-g0h z+(0+}Dfb=^ytb^!ao-E=r~jQ=Kr4Ui0o*T>Z#Djl^Rab^?{l%dzME4&i&I#ecT`TrTy8#r?|J&tkj$W$f5-mt_d?nof3#|DnXgdlmjzr~I7P z#(5tH?J?s0&Nlv(`;V`UBEC%CdGF)tQ+|FNcyEC5*;IMVYVJpZKB>PtelhJnN zFIq*Lz_0WN#qp^p{_IF!f3CAL@#h7c#*0+xIe$oAu~X6G($fp}+vGl3v5e9QYaKalcLGzb+sCpLhT(t?zuCFN!~q_7AE@o$9@cE9yOt^LX#b#^ZcW z`48#zkavA75ApwH{|NhaMVW3|KLOvx{CJ;1@PNgC&&*%u$6hP^ZI9~n&p_h2oAKd% zWjt2+;|l!<<9#E*)7sxA$9OY|e>mjn1d79_#xEOhG#6|0l2inJ7RD#l>WO&p@7zo1 z4@#5vfPd%CqCNv7|J*FvYr*H^Mjr9&2Ky3!tHMg5cS?^u59G{zIUm<^Opo~T+CMA5 zQs}MIWBe7pwLdTDl~ef@J?XzA2YSZ8D9Ww0`Tcxc3s^qhbCU8`=y;^zRTZxb?}tc# z&?NC^J{MP=`{SsO7Zj8BV*el~?bGdP#Rm>}SP^J>Q#%pmB;J~*{iEp*A_82u59bpq zUX!lRpV^`PNz{D6d@I@h`eQQ(4v>ElUs2kR`DflA(QrAi)3@UN`|bA38h;vnKGKSJ z?6B}Szjo&%_mA}fm+;rv%>MoAfM3-Q`p9{8Ui(9j&Ft8rHXsdN?XTdy6v$ z^VpxaVa`96H+@jWGd20ctLptG@)zqj^O@t%4H0|(lHc(jip(>iDC4uWgv=5x=vO_a6uo3whhgcn^T@WB;aG$2{+Q(3k7?5seS! zegCg~d2jC9njc~BJbm0RNdWZaurI!}%A5TR$m8I*=f^og{}%Yi>qx&XKh7Njfe}CT z_?G;5(}vgP$44}Lgy)%gHWwe{^(#|=3pCFZ40{jry^Hjc_bkP}3&k`t{-D5jCwTA= zEr8zB>m%T{{}ySk>By3LVfOi zMHO%3FzPqsGbyu|KajpGfAaw;@2^R(PQaV?yYR*x3H;d2#3%E;t8#*>_+fJ3sJ=g| z>Z{KL!>drc+{d~&i2GO%;)2l*Wlv#z8IOYJh!YBDiidwk52X1IiBu zSssZ5JlZP_kogI?7U$*tozXY(ekAOVe{sxvIDmuxsyfU##P>z|`(=Lfy(`4;C`7sY zSrop2^e;r?SU~V=1O5|?SAJaUM;RZ+FJ%4DUkL2T_seoh-}C)78NWjAF!0bXjIV<6 z+l_dJdi>S+jqCN7q3<{eXvS;u3y3~DDheY!m7VwuLvysq2K#rybo>|7<&@$-@u{N^UQk_h$njo(;NG^6+bl2`(6@{ z2;m-{rJpWL@~<~!13KJ9{}O73F~9M|q3tMMb&b544cw=W~l z8#6&5_S27*zG}te(jK#3^JgA4{(xdH@O{z4>=5aP@^D{5Ad#1lkAI2vtNLN1+)vKd z@4}NqVY~tF;(c|AhX($r&wS`Z zi9A7j!baKbe~A7L8{K+5(JaANM4zHa4Od0qrf|*Jp9;Tih1{P4UhIi*+sY-{9@Fru zKGHwM4;lwdjc@Anc$NR;lir6`c)fc4$o`Zb<^2Zf585~T8C<{o5`NJ37v|SP!q0ap z`ae#-W{*1mtEgJbG(3->i!`AGsod>+1BvO6~K6J*ju8@Ck%I~wuXlg%LDIOXnBGD zOrbvo;YW%026?|)%L6nS_|wD}n)p-rLGWc+uM!Pc#-LpVm=VIM^#;e76V!V}odhq}4UY|IzQXiwdr!ULykDEx4?4e(cuXQ6 zxkw;?X5{7Ss4VXlN|mq5d#^Rh6QS?hc;EDRQ|@n~JqzU*%1ikb^oRV|3;Cix)qb(> z%zLMxKlz>I3x6O#<&pfr_P!Ja0olO)C362 zPuG{}AI9U5%s(anA)n@_l>YoL3VhFI@_69%uRc>bDDp_(k3IENwTulEhzv*xnGanR z@1Wj4y2AMo;5~N0G}56@yCh!nQ}f=NT>nf3U5FP>``(*?eg*$GQeX)kdg3{d3Lrf@@Z|wTZ8)99@%q3`Kyoe`d}OF$Mil$D=6eHpT03GdVE)6t6a8i2@kqVjicCK9Z;86-@89(Hf}ThpJ{fqz z52P>b0gSKO@A!_gN5iPw=u6a3dN&#ge;|J|%8|bKCZv6*o`QHDq;^vDME%Nsm~NuI zwAYpXmG+@Mr;hci4F{$lsd#EU4x>E4m=C9(!u9RJ3kt@3IQ3Ky;HH8f4*@p-Pbhe# z2)F{ct?@o!JZC!iqJoELKlJvJnWcWnqf<}ey7SEcf>vgp5gxBPt5g+;Cp1lxhKIlK+Xh@__CEM z^H&mlH^`shSw4ny-pdjH3hQfN`wMY;31hzT{W`*k*QMZSS7Q(R$M=Ky9ze9alE7%s z*jLB*pg?@UV|L?ihKDlxAN&P-Ag9`oDqcGc*wz1&-})3Q$A|jg z7JaJiC&*9emuJsW|Kt5}qyJyX=>Kzz>3?*H>*Gr|(*LJ#p#Pux8R-AmYnlFk-{^nc z{_{U8{jcdO`hUBlAG7vk${&mCfAUjP7!Kok1^V{X&FX7qFQ@t%@AHbj-srTaK6CKe z`dajJ*TFrxdH4Bv#(_lxLj;D3txI(ZK~XY@7d zLu2p{^5@;_>v~pS*Ij*GtK0e(`lpuB*R>4(7ffH5O0K?caKVL7(f8C-P~VQeR{DHC z)z`8As;^54jPl1`%INDa zP+#|JeO*|fud!b7Mo}L82}J+my}D6!A0PBR%Zu-j{U?DDpJB3i*}K98-uyq2_2NM6 z4RgOsj8NErd_R}l=fWR=v0vlOBEQpVjoqo_DbSOfUUM{apVa1}yyr z4Esx+XJ4SbHHh^l&10lzpMl4IU}xfwf%Pf&Pb)b%sH|fAA*?qh&&_!1!U5=Ubr0yt zviT|cfZtkn{m%OPhoI;28AqRG_3L@~6HtAc_`VJv_5tJ3$#`I?!1iYM2V1e%&G9$y zBeMUJAvx~}C~5Ga|0vJdPrQ=ZA6(=gg+UN4*b#= z*5`QxSH=MJJK5<~a6YZ=VX-%S@3QU>_yz4#`$N*c!65q{AL_5RF(svbXkZ#~``xs!Ao27p z`YmW5_cqe@s_FbF*3MF@Z^V#-cKNc|& z*}ls+KJUZ+|6ZiT2m1i?bqwok%Bxi0v_yUhjQJ0FDDRQU`N$&jV6yl*kMjH)`UvY~ zI!=JzoUcY7HEn&QYB%!0@i#;IGv<8k2k>{`{^^V-=QH}e2Iot|!%6$8uZlapH?8*U zg+G(}jC>wpcox;3%T>P-uU~J$kS`lHBrxdHih4)ZF+Jr2*5|FLvEGGi8&Y`C8;*jt zBTP^EiuMmj{@V2h=6P7OBq#ESc$827HjyPXYB7I@qqS={r0KDL+|#M{b4>b%CmNjT zU&Z)E^>q|gX8h<6x0WFBuVX*;IH`|sHU2mB`hxvn_=|;!JiR;^;Jh?s`JxYzfAaX- z!Di;a#(r>C|KR*Q;}0j{ypUh)$5pM`M3Lo)~{M6CaoHK}EhK@ip^mKa2iJ zpf~qFh&TQw@F&a0+tmJI@FxNNe8jVs>ZnfvoAqr!&ttIvF!r46&!I$=kM$4uuZt8F z{}13N@fy91pVt4>&*Ffr^fUL1GX8aVTTliN{D}Uw;=V5USIujE9gNL%b*)7CQXly* zu6k`i=6sL#5XW2I2ci6*$@PhRXrg_u5Uy4ed4c@YkMBi#IqyLKupZ8N(_?D?`pPKa zZgPr}A0u+!!wMCw^fB-q3VxaR67U+v{>mEcM~Xf|A@o0~M?oh>+MB2UAL?6!^9I^8 zvgQ%{9^`?6V&1P?6BQ-kCpKKc|5NC1uB*wv)qF#KuS9t*GV{`D*rPj%#RvLfzi9R| z2!Ch!KlzcE`|A9L^C>RN{!^LdVZStMFD`PPBkPyxG}rc-zWAF`-Ac5L%cugU-`h#t0QkS>1*I@d88lN)zI%*#1mlJ@g?d!Zt~O% zl2yh7{dMeJw5N~pc};!b53K)Wd}aKBhklS?#`AnXRm&U7Ka|(S@m1?neID}|&LyIp z-v4gPdGB~s#sXINAO7A0@lx?m=o5bw%!epY{)Y4~0Y6}W9`S&YUhGH8&tCEWGW_O8 zUEZCnp0h&QR|@`~`b4Lf^Sidl7g?_ve-8=r1^R;CvYwau26^vCCBD|2l*joi8cDBb z#at17u0(W(mvFW^u4?=nIpeT`{&hq zXcy=U`8XEk#Glsai?LaA|3IZ@f5@L@wvXlEz6eKL`*VQ41?B&t{K;yd2OQu@5A5H` ze8YJ%(&yPe@E7V6|5H;R#v{KS<9{|5C zvfu5)-_nw|@NX#YMRE9)zM(!jEE5*<@h!9`_KJbv*}g%^iwxwWvZVZ!2t17@~9Q} zcH8=gox`21|5%9J?ss^08A{4KqkC?_&`l#2{M_pZ{w+Ojc(Vr+MstfxQ zGM|h-k~6wXi|M1%4fIh-^pV*6KNEfQVbvEs{_qzpEvAoTd~Zu1;dt}9`e@Up>+2(k z0;3v52mWnP-^h9n^_le#&3UeWU>}M9 zt>_!H7y3qgajw1o$?BW3(tX_3LH=RHD+^HcE$ADGC!Vf<$`a2FMVR#u0}i1B;BW57 zEbwRhx?^uVeSG7h`bNh0r=o8*ZoIy}$z}D;Mvjksi|ZTlN4vJZ`CXm4E$hAP`UmG3 zeiJFo`rTO%VSF0!Z=<}B^$+MddHjcR-xKy4;)7+@Kj(d{e|%qT8~SI1zuooEZ^1tA z5q;B0^bL+uSiY=(=Yy88{F|(e) zep`+81zc~Co|*N~Skx);btOH<7w><{-iKMwjLo7v(G(Of3Vh+=w1U)eyFm!r71bIKq^F;`tbUkFXN92?!ZAdjU&% zIG=IPZ=^r_xIbXVZ}Rwb%AeS8&8E-p2de!(wh2_cTJ<)OKITLJ9}vODAJqK$pO5R= zcoIM1{3lyJ=hp<2^F#h;j{EbgV%d+E^C;#23LMAd^6z(^(f&tRA7ed|iH`;U6231a z8?w9>bcE+im|q}}tfwKrd4Gg0lKUeV&jGB@Wj;{|0&YOR;4hW_;wbr5;ORr^Jay~Q zPC37n|6u$E%>7vA=YB`=W6X|E^`HI8K|i21a^Bd$d}o2u9@x8_4}_(B*)N&^#82hN z{=NeB`gb5X@5Rs9FKWR5P|CX#^9AjH!k_z-$G_%Hh(K5V14zHho6zYWV|+?G{ZpXtW$zv<{p`%3w_XuEq<^90 z-)HJ${vTTPasBj;lOO48-i7yB@xtx;m_GFOIrZh0{{Z<%)tC1UUaLN~_plc``NM4f z(EG6IFXo@q-ueHZz4rmH>pIJP*ZCv)SXP|lLXclvFgbqc}>(ogBGJ;|5<) zQh6}NP$iZdH;(7dl=3*u%t!Y&-ROR}D z0H@Xk-viV7vA<;e+roUw_*=t#$@q7M`IM3UPyc=ST>o257xQxej^&HetL5Dh;&pve zuZQG7d+~pNa+9rZf%3i(ll2XAG<-hyduS5g@@(IP83^2KFfa4@h4SmHIT7aR_Xs%t3FiIx^A@F7>)$0a7>?iPhcrKf@gEdEWrml;r_ISF z@wH}TNqokf3h=!Dz} z&)M`L{p;=hV|Kp5`C;d$@_0$%`Jtq)KipWjA9>RT6V9)>#4kU4W3?F1S_{_|GK@4pHAi>04elKEx0{N#E) zSbm*`%J0GO^GsrYsD8=(IljNzBqe>Azb-xg!1GWpevQwTuZ!Q#n4{b4BJ2-5Pvzpb z*F)MqT>Q3vTy^|`=doP;8g=pC8`Mu^Qi7#dQlZg7L&$|}#Q|f={ z`7nF_3Tv*-e_fvlJ})Nzak)M$wExKVaErbF!`A7D^Kb60uu%D#?b-t;US zU+8%=T%Ze}byn&=&)>HlYgxpWEwE4@y67z|J5%Y;tG@l4RuN3O<#lEG_ zCpP%cH%+B=KH)##BzJCx`$L*fY~t}KdOjidGhfU31fJ({^NFgW`9$D+;rWENhv0ZF z?x^{Ngb&UyHiYLF{5;hL&p$A~kOh;_{6g)qeWIK%G{0EX^Upk=sKR`r^z&4a{byi4 z*vaMhX3Qr7&udA3X#Gv1y}I$;A z?0h&dK1+UPWIxi@Z8yJ&_pQPE^W1zq@H|>%eJnMfsG#{o@cA`6f3On@yPj>!BQU>+ zJm13OeewKZ!!pN@!1Ha+zCwKzx87g9-0?mnFXT`M`qnkM9&+PnMW}M6I86FaCT*?d_h}!s|8n z#CZMT;ZMw(vw`)KsQFEY_TRc+qU|TRUSQn|t{;`W9xb-#vKe1d_Vk*G>t>I^`kHJJPz-d>Gh{QbpGJ_TXoy( z0U58_J~5O`+x?8M+Wm;7hd&kyulx7UeB<}3ef?Gv{QN9vaQ`L+GXdXbJN2p`{ngMUAQ zoWHK`vo^M#+o$dO#2;SUX1pAi`2SwhH>d9lJeU~Et+DSn{N-%OU(P4fX*u7Y+4Jf8 zqAX9UyI_y=|E~AE@NYD_@bcxwukU|6IG30Br9A8U{+G4q;}x){&hcS;ih1vS6A#PX z+d=zfKit0m%|5><`5|sQp|I~SmHQLo@o@gekE}C!F$np!@rd3Ymiw?R0|~$JcwE!x zxwU+cuUnV37E5_cf4M=*Tm18Lqv94`+fR=cfU7_E^;NHZPUF}6Q8fJi_4a-NqwklL z`XTp!>G|f|{XofdUv>3C3BG1Z-q$GcrJlV(ckT=8MR)9v&xt!{Vp1P%eAV9jqBMTB zw>PEsKB#jPeV@<9{iz#tsZ>O}9w+ygRoM0Gf;$n{TQ2OHk(9^A{Rw$^&^=#c-QN8M zzJG;%9?aG;slRAH@_W?Y{SU$C`^Eo0dH=8^P!6$A-F<8Q_91Sm|7)*^Z;w82uJ)RTwJMMmuq^yrsOLZ^IpL~Ds zeksWxDQGEA%^%yJ=<^#n?|lMNp8CEgTbAR}o}|3rwtsE4mKUB+UnBWz>#9Z}g!g#Ok?eFz`85IZ)1lat@Sp;QJ^CbZ#RvnEVglBY%A2;Ei$fy+7G`=U-0#;(eL- z{oC68V^!~4|5J~T-SMvK2Rc7>$EQE}V151YpMUU^_Z|7->la>}oBPAAUr+tJ2mi)= z@%O**Pc7M34<2}9P9Hv<(>o47`Sf4R{ktnK%G~FmTrmCa+>5Wgeo*AOgRfrkZb(#z zPM)sJ9lZRbug<;NH?#Tjo~Z-l|NZ%A*Ztk6e(Z<7`OWFSiM?mr;HUp`>#U&vSw62x zY=8W1iQ~V|eW%LKsx%Vs_`NF^RJmQ2BMx5q$!W;$`>*D{tAEAJl?$_POg#BJ|HoJ3 zzjV0gbN|eacqU!3|szZQGy zji-HsDcZ2}!rV9K=JtJl?zz`1=05sw-r;$^@#5s`qCpGFlP@={rt;6 zedU!4lmGtk`Tup#g}?fvr>^||hadf^_SiS({`@-^4$fV8WA2qt&b`nw_1POI|FObk z|DRVc=yhw-)SGz5WX07G z|GZ$S*kJUx8NB<>TVlsPV;*_Pe5opx(Vxgl>N;^y4heE+?y(Ko zFUR9Yv@Xl}w^4pjf?$jWu!S-a^3psPAz;ZtQXWn$$ber4@^_haHcP=G~ zFFY^jx;tKPL*H3tX5W~*aBue2t-Zf>_E%p2=G?zXt9kv&D_^+rdmC1k@#?q!bncBu9*fCs6EEC7SYPw}^LGwxuiEvgq3qpXy!%VrcYX2s$G+UQ>&q`Z z=GBH>yDEO|?psVeo08-klZ(sSdoG0jdr=l^=Kgf<#kmK+Ha9mt_XR<6wJoy1G5*`% zoV)PqgVK2aVD6PG|J9fy@#pWBe3S}c3o^MYmV0IHC$BV*&-DH2r$1eJ>rY5-=w+&Y zJ~KRcRYu(iJ6*g&7&KVvGlH69*tFO$Qmt>ikc1WHKwiop@%$4?p@7F z-Hn-tcD2{ErDOKtk7($Y3RAP|(MMY&O1Y6Cm?x^zmnUwH8LmtNG4WbVQh=_sE3&IKv! zxR{D7)>z>!#fBW`*@B8-LPrNdF<@f&j<(FTs)E4Ar zN7iP5v>|Oq0r~f(3zij}w-7_2b{PMs2_CNma zUsq;r+)|Nk^sm1D#T%$X8mL9lj4rwnW}uvJSKc1mbKZ5m?U|! z=oDMGUU*%?Ieg;-60IM;WqpH$Z&|mgD!|0vBTUQAcol!*)m7_X(?=uZ$+FnYp_p-Z zsp6Er|KV8BKPTtV<6E)VZ5>r>%?IW0!?B#KAqEv)67iI+RU0iMKSC4B&$X<7N0ol! zw>=?mO>5C)h$>cy0XXzwh$4Txh~Wr%SmDuF{I;L1yv@dtlOJ%1`@sd$$2Zjf=6U?} z@eK7}H;=zQPNDvZdHnTp2}$jb%1g6hw0yU8nyT2t-LAAfkCwg3DycDJ@;JRNI_g+& z#GQ9;YqewKF;UA{&4sZGU4by_UaEiaob!f{|9JnF8PnOJtC}y($?8rvR%LSHkVDsp zBK@te_jb*ehkw3G)8Eo+w#7bBEqRc2zGCp+PH{`ab_eHC?BNgH>a`FraUQR0Y~JCw zh{FkP5%RV{(^6fX^jm=xi`2bXx>C-mhh*1`nTg+)k@T4_YCE*)(@NtGt#6M?gRPc! z)gqeI<~1hPV$wE6@s0KGd`+^$*05MMW)8*CL~M9HlN7C-M*dHk8FyxNi>u%kRwT8g z(C%!ziWk~l)<$P*QSmwDceXrl`){dpN@~`|CH-D55(HCJ~1oLpLd+NZ;%SB~gEuWNhXv0CzTYH9iTh)dU-ou8k1 zQ}dI?>z3;Fr1ZKUu9O+pCK*X%G3~HbiHq_dL}}5n!}i=>`xS?bw=L_FRq^V_*3Wn| z+tAF_xRHoQknnj-(Rs2Z&l8>@8`ncGe4fRpCt~z@LSDb_CENb%&G~U1@thsen1RUbw`e{=8W-oaKO3R4$!dzcxq4BryTtQGlnc-Pi*s^~<|O8} zdYz)Vy}iD=5}%@+wXo)D}v+p1V+Ur1osTyRy=1 z@*ODg%GiudD&6%gYOU{|SQEUSjqtAshDVwL;U)e^F1koAs3dcBb{xf}8~*tc7F8RPrM*BsK{3pY1p)gOI9z`s&%LXhy)`RYjj zL&-z-wG!3Ypnq&{L+fkiQ}$Xrza~Eya<9wJ1@qtLeJI}(k@>D+{In-h!L(mR-iOLA zBJU+{{jx{i`sY6!Dm(ax^FAJv(v^4mNZxyyWHUdMxAqsqCGW*QBJV5Y1`!D_c^~O7 zd0#31M)E%3AIfW&_X$@P*+hP>bZ_IYGpDR3)vhYP*JC?nlmRo-2^4;}1${)lHGfA}FT z#Er1fgB-LV%l=V*9A^{*4c~|t-_Z3N8oq;m_#1xswlz4Ax5~x#J|X3(i(uZtD^ofa zFHp8?WF{o*f>Tn)-k23wM6wZj>yDwBQ%uIiyi6eNq3eo8@zWR=Xth<`Y6rE|>X@f3 zckg48o{va+bYamuf~#Pi}aeoC#44DFpoCYSzE9J*y)rpk*Un{;!XpAlYi!p?z&q|r1n~*wYc}xwct1y4-*IfgSA2#z$H_(#f+LEJMyK4Sb-r=H} zG%WSTw%Ecv;BcI4n`K)!szr&q7*sFMP2_&iMGEG$DPvC z6sJEcr}qw-X7UKc^&`+uY;Ww>iscm`a-X7$(|y| z;1h$E&^^zYA}gFyuwrH&E5@A_VSoP|lac;~`eddf{r&LzS}09JIDS8TTKv6Kh6z6g zeGym07#!F?){@v-l?d$V1{K*m(_LQSgC!>GOQG`7@{=-`3pE`2LN9MmcfLK_Hk-EK zo~<;Wz@e{zTWHU=L;S_2MeN!7Z7gQnSYQvgUrrd@PgS?IBCv;B&}0PHecw+BBU`wI zeP4CkFiY?co^#&tk$bdxIh!14xAX1sYP+@La$#T4>tC>^8b9UN$4#GQyg%upvwHbSFf?nt9 z{Rss&yndK-bxxUUz0Vz^Rb5>}mD7dytX+b%wioxa;T7P(nyB50_RA+xREFBwKN`6r zY}fHVoRVH|ljNl!y4hWD%a{>zKNrg@B`dc2CI9QiNpMfx+XD~msYmSzi6yju?c07> zZj1`ozY(nmI@5nb?sAd4U+mRF(aNh>BP*1y`0NWSIsdlHS_5l=@i(E#DYmRs2;cHV zoGtGY%NJ@D>aW)>wsqN=M4NYgCP|SQP23KjlEHd-GY!u_p78vUT3N7(^e^P2lu|@^ zzk`)WfTH{h;bp}#m@}RpNO>YlWu_@}S2({R#*3?PHTRov`~88tn$s2EbW`Q_bhbvz zdwcq4Yw+>e!}iK=dwQ$*2^Rwle5`t|Aw5$0|(7{ZcYt5!;fM@a<~9%+%!HfIBa$*88BA zmGlMOc?rXeYC$F!#fZs4MknZ6BW*j={?dJdH9%7-Ihu~wcej%D-j$ zKN^p3Y;2NoFRc?*xgkz2=j2ggxstTCSenWL{&BsdaRGmM@#TX4mFAbC{1Z6;Ofu^H zhYI$+mW(?8Az8fe?4M0Wp1CzJmX2)sg8d&EYurTIg$WMSh4^ zYRqE#T0UdT!#CY(`(MAmG<@bB=@4m0I@CW$&2wem&|~5;xuiNMcG-D@yaOWSJ|b>u zv)&=RoFuMpot5gmL*4qP&Q8@WW`+0(ayQF%Y%o5(9W>}}Blm72<%C1x54p9pSS#g@ z3KOM8#M33V*mSzIhHPQpU+sRWDwoPH*-nY?D$mz)OFnj0=i~G&>HM7jt8l;B=i~I2 z-3zp}rPjx*DF3%Q{}-%(y4$qeG_~BcUw5P4>U5URzi>S-=g%t6pT)G(?Ke%-yxHw= zG3Q(iQ_Ai)D~`pSCyI&EN~MYYuSY++n0mu@TyOVc(ixa%u9FqutC?s1K=pT5S^soy zfSVlkI%ipVwn!WFoP|ZKQs-P`oXfKNTG((c?7S9ntdz8@7CTDWS<3%fl-Xk9SPWCj z)>8ga=ZMlcp2aX6<{G%KY2PLGEU8~|{6c*v$JZk_9fsGW!tNvDUsx29_+{6N)}*}m z2>NSMVSa+bH^;*MS#r0LJJg5rT3ukDQ;f_N?hV&9Ci~GRFXV6BuLfP+_`C2OxaSVM z^EZ|`s~8(be5r`MBqM{2jJLT$W}X z?S1JEo8C<kwE%?oTaex}7EyyJ{&*q&8QvZA z&hO_h?+$wDEBF`f4tn7ub_efDC-v<>^1M~0sg(P5X=gCdi~X_Gd)pIgFC{(Ds@fxS zHizaYIwDCrLK^KAhMqcgJ@2ZThpo19(z^kCw3Tl8}QLCxQ!IfU(tF1+?^!$8FTTrPMNi8gH@AoXXFt5+^*SxOn zI9P6B*ET=PEevCfUt`0ou*+SgMf~M4%$sv8H$QpSR35{YaF+~?S>;)w&Vtbg`aQt4 z?PFKln5EZQe1BKoxRfT)<%h<&kgk%fqj=+?5kvK%DjJg-lxwr z-Ra$DAIs9-g+BgM#QXa;X~bWQ1YFYNocEdUz}^QGK@^cJ4kvQ^#LR4V*({oUS9 zs0=tx&EvP*JTBfJipt}~#L4G~`75vTES{-g|HZCYayWfLl{I!pF`DL^mXKzSvy?mP zoE+wA^3mHTsdZ0L<&wqwBzzX#H<^dG`zA}^?LNv9c)PE{d@1*MQJXFmqLhDBK5O|` z&GYeFUyPbv5E^F1n``K(dx!pqxK zFmL@Q-|XHO$D(;V4{x88TLN$Q#+Ja_C*_zgT2{~fd&A_6E4@{6&$-e| ze5L1Ss3*7Ozlzrxu62u9YJAkf;@F?^ul>s{%*cXH=pOFn-ltJ+Vb?Z4-_kLvREsOO zFv%e;Azdkcv);r}ZeisX7PxD^Jcj*Xk73_``%|Sxou&FOW=%Beik#20^c6&RFFVib zL++^E1plc6ed>bkDBevd@Qd6{DDaEiO(^hh(r&`y^DVTxA1~Nx7PTAT^NZUJ@cG5< z2KfA&v>UMad<(7I(>w5@>f1d0;#T_o_!qa*@AGfbN`Gjlw8F$WN22mM>e&tsSKdL; zEqBfp+Cf;2Pr&dxSg41W^se-Ev5>vx9RyoPSN0nhOF!`vYDeWA1Y4%pdbR58~p9Ae8UF*Fn}Et6=1LOIr}nMA%I&ERG!n+}~jPl$E|7$8(dFJ{MW(`5Brk zm0MW&X?AWwrQGEfCOM>iV0j0@AG69WYza@alv|kOz)E}mVpWY{p@(#d=w?lDGmUj>|yEs>92O)Itbom(s zn_pOGxn9rshUV<$9R!=_t9=I{)Q-wK2)0bK*Z6)}w9CjRwYi_W)^1vTNy#tY_NU4_ z2<07w!kk2E+)7_BeoI?WsTN5sERG!nd$X|3Wt+p{_p0~Nd)MFU_pFz95U%YTkwSB& zatpgYI|$_#Cf&>Rn(eQu_04h%E4MId3*|Sd+V@%Mq(fT7HGKVK`8Ga1or*Q&mhbg^H#mZD0J_1c?Y4qgK#}&O{?I#Bs7QNIZo6w zlk^s&P&+E`Abj8NAXLh$-})=dI|$_+1nn^Gu?l**Rk?#uev6S*uk!k)WJ0-xExh)^ z^A>i`w?|iQVdWNPhp=)BE3vxCEsEXc7A85OvyAdnK3;<=x`VU4gTQTu-HX{l;4pzT z#5JbEtYQDCEBM9SgqnUyC>ljR+R@(zO5czZ~%(IZdY%+1XmgnOku@BfyzAnsM!O)V^rXAsKkn^K*w zSD#wGL%RHy_VQcWwZAU6Fv*1S7*^h+lWI_Y2BG{6LNEtb)V%R(b{MYbiZ*^RIUAE- zSdJ&x8h3wqsoz8(zT|Hr5MT0*M#Qh!od9d+p+q6yO6>#?U-Eeb;!FM}0`V*MHxaCz zaX;Tu^ZW(}eiMQCl5aF3zT|Hr5dS9qCc@Hp0<4{f;)VKF>KBoTFZqke#FzXwF6b}sS6R?sUg^A`zg)#E=r86j z=wDsJf4woq_duwUtnzYGmw_b?KQCih<7w7I{y|;_?Kq@AOSeEi7Wq>?7Wq>?7Wqp) z2CepNdo!$F@T?{(uS(@pusrPRc?vZZHFh!IE!Ekiew5{JXwW%6gvED|kKJXGGbU+b zGc5^|$R>UFOYvMh=x&LdusddALI0Ob&IH{tnP3Io&n0qi3%X+oQyX+Y7th@qbjRZ6 zmXJG^yE*87&g5?N@&xHx)c3kgRk~lK_ovWaYg~UT8*L$TZ^H0z*2mUqE7CH2&WLsP zO?wN)(Wr5(tClbo5+-rO+>O$=`R;0RuOoLAx!00AMea4^t|WJY+!f@GlRIhM4|lsX zOA_ReQD)R6;|+58VPB0APbX_$GbSS`OQYBJSA;D;F8q&zkF8r@^_rI-@jEmRtE-dS z(@mAz)7cu%Upu0$>=paa2*xAyBHUdj87uq@fc9@OJHjPyi|T0>5#Ms{Suniz8*Ia3 zH4ew_sCbxR_dH=VPYYF$!$i1?%3!TG%ebmCn8&6C^ObFSv*MWNJVp7jVR?0O+2wJ^ zTYUbvEu%hg*^`3x@&oPx+I%8&m&ixigMNKi4R{@FTiKK3inb| zxKy5dq@whDz2J%&iHhuVBW2z77%Ate#hhoQ+}EN^mNJsDJwiPWb;>9Aa$T|Ef!td(->+7vUoR#mLoxK~vz2(C(NuQK1hL0nezp+UsaIH1-JmSKFAC@ z#`jNz{PTz-e4ZPEaZE-si4cS8L-tk&${Z36|@Do0FMbJKMYgj(ah3a7k z;uePLYdjcV4qS0Qhq%EY7qoSrzhiqFT3<7t@}63V<>!i9deOd3%EWZv#$~Q)i`uu@ z+bKQ48dp~(=cHA)=splyx9kUbdk|ZT_l0V68zla`-A@5j(XNAN-G*8BjRU<-BIyIX`wy1l=Q(;A6!Rl+}q+p^Mj)ENNs2BTD!Hbah#Fb zn_x?k9TDf$9*DW9`H(+)9G0H*P+sZ8EAPyZvlbCzWSMz*c$8Nu-diP`$ zpVQ~<+^!J+BKT1J*xsGreme&!#6KU;;iK|ov08lZJKrh0_F2tqxj|F1Y>nBF-4NIN z6!hS8h4|DV+4;Bpp=x)J)`EQHp|yyAK7OI_hf;DcptJ9WdCnJ_Hze|N9N)Ee`}f|x zv?jkMACvf3dXKf>y?4R<%gem0XqF|_L`PM$t@I*hS%LN=bHYRG&_46FhDn5lYp25@ z|D2o3`Thx+qx#`tgwEgsOehX-rA+QF2{GaG=ja`Im|(?(%ZJc#=i_iBNf=&ntDFuSAzwD6X%S z%Hu#BdOqLguubw}9(zRC6guDg19>!GT$O=y&c}Q$z}Ws|WzQ%UoKY{gzm2-`2+i_M zc$ROZKUxKSFz4C#~K*JT31xRm_!gM_ns(nClfUpC@unA!_X*;`)BER}YIxFWbfLMXm#cYGz>M z)EQyniYenp{A+Ozjw32hm|^!~!r+QZCmHd$%x?3s+)eqLpKkJWl5H z4Oe-jsXR{pppKJKwI@nj%y@J){rvTcmuvsko+oO2U&%9mr^1fzf)t*+Fo$Hu2)o zx;)C-393$0?el(82X(?K|6I2>GKJdh%Dftltf6n=m1C4WS2KEZ4|6qfM2((NTGYtk z_Z-)DWW1UY=4#T;`B%yvl?SE5u>aMhEjaSX)h9+n*{F-Y`YbiNzJ;UZq4`E3&Zj`# z`o6k8jcWVB@(qrH8|EwD;C)x+QBYS;R@57>%cJ08uD_yM!nNqduO^PD5id%M8UX@5 z)iu37yqb~XYSPXU@sXM1?6^=@+hbid_6dyqgG_2v{2sa9Y)uA zl{TvF2W<$9g7HP&MN}RI%cEdX?T@NuQQBfgD{fhf>1h`eZu z++JhUHDQz%%GWpHIx%>@;5FfVPo{)M!-ywKLjK-0<4S)LB-EGs@7$EvxyrVl|Av~w zF}cmOFay#*GIED{%ess_2~qqByw}uwk14%3H0D2PACou38huBD_snfmWgslQB~2H=*!;+}oPv-M5d(Te4g3 z(v96%I$vXQYh-^{UB-;dGtL@DdFcm6O zmu}Yj$9IRfJp6Rk_Pvi)#`bCz zdEO{_8cSD7{CY?XQm#dIQvKDpK5sDDFY5W?Ny$3-RSAivaAE87-W~0O(Glyb$x>=mZ?)$i&)O!nc z$eyg)hNCDw!SsrKe*P43jO692dXhx!4UIw9+T9}kTd6oh{a-4Vu=(m* z%(V1>Hcu1uYms615tERi%pJn(6PD`s@00O>ii*Fg;fE zrWBs7awwj?vVVBgccT-An)0)*Iy%cFJ~aAHkh?h+_Ro^LjodPY)*(Fl4Jlkc?Dnr> zo{Q^M4EvXIFX}z3ORATp&REL7Y*Fb_`~k_Ml|Jq*_+}HS=jC~E__|_Q*BQmDORFG< z7wkEsM*S$wrZap6*e6$Tb8~YC?Z_eld2QbOy#6-wF_71)1@rm>7?;=TUM+bY zFO}EsS02Obgtes@mWVoR(+ za+I$pIUc_+e9o3Y>|I9o2W(rOiN*3J_O6tRFE;a>nBKBCRTcm7eyLaO9nw|k9>koy zl~$g~_YAL*uSox#=PygXMg1*3d#G<<%NMHizfMU1{x@4*Gc8}z7-Uavjr!v)x7iZ@ zJBs`zaq^}eO`#t7_Y}pi@7c{ueY8hgarn?1;&vDNhu+h>TgIQT{gMPZKjQBw;a?P< z-UD15J{PjT2Wz~mXpN8ZD^y;E^n_<5D|&_^eO!5Nehug5s~|sj%WpHf9z^=XQ2%B0 z4JKLa=$7lVoeyVCcol28t6VGnT0T@iR(X9c_s=?6@m5v+JwEc) z`dWVF>(#*e+$yin17K3-mbxxVYOq4pQP z9w_(E<@s~bGbtpW{a4=aaXUQHoeg+)xjPSOz z#=ZLk^;2j6B_)afPv7CLsJkq3Dr*DY>m9bPyg{E9?q;)CRduV^)h|;xb zmDB{3-2An=TvwJ;?^oMM%rl8_&zmd1eM39N>wD$8s&?8|Vf`-WZ}>`Txpyjut~_Ks zDKR$M8dHAu!JE8FXCnEr+!9z-HM(>SDLp5-MnvhFnlZ814@n`}LrO}Vp?6tKiSVI- z|BM0w_gfm?K8Y$%qk3;_%ZOQGn%v**+%j>GaLXkU!r@6x{SIoO_=Hz~$(vER0((ZY zNYn4!J>>p5aoe}PiN9WQ^q~@N>)u-&A9@FG_{>t3qJ%N{O6Q5*8Qtd^M%-xJf^Xxgc zMVv2pC|A#?OXcMBx2jKCFRASDA5}AV+ISkA*5kf${z2zXcE}%VH9)@xvWKSYTAzQ~ zi`wK}`UjnR+2{XEMEX^8>92F=>9O_8n2uY$Y|aN>u_y&`Ps-i`{_{d7H>3=nveduv z#FGAPE16^LE#vz4O50=&K@H|TFGf%^YM($=|Hi=0q;iP)p^ecP~!?@XD* zhZC`6YnPeaD?#Py_w4bwnUmLUcKnw2oju`w=RWU!=iT@C7d-dK7rpPy^WOKsmz?{* zc;DG!?>qfF-gk2IHq#-7YlkeUH3RY)l+W}ZiM4+$=aZr>L8t8oX>-!%dv208 zc#E_*nQ3;&FxPXtup7m1leEzv5nVom@0M^K5=P!3JSm^?1M)d7W6f8^EuZ|DxaE`k zhQx7J!hFk^#&hz*-ERvcpY#O@`?u2G<*vb=ikKOY&xm}+<&&M?Qn{$;-#A ziJ3ODMMCmN7Q%0q z(c$)l$-FCJ@>>(8V|&6(%JA2*Plgy7Hb=VU@2BO{lQ0AF$;)S2KAE4Bu=^8cP(FG2 zOv=YRE4qBL^68Mzpng6f;XWy!1H#E?T0WUyOqe$L^!z*NpXD5klh446u&;^xbveI$^76^d3jc=0 ze?>I;jLRo8Cz^aF<{}CCZ}K1T+!~+yXMfdm>-lVcIk$#acaQWb!T36U({pQnX?|+G^y9BIgI@~zr;XNC z-(Sj4`XqIG;cJc7U*Eshj5xRFpEg>jegCwy8_$jKTF-s|TGJ?fn;*ZN{{umPIluII zzJIOJ_7Ds|BK@GcwS1~#DKqj}&)w+UY3UQyU-L`zujlVPx7LTM*!xY-KX`7fUsbUN zlaqc^`Lqk)kw{p#vF-*LlRUS_*Oi+MbyVi94dC)EO=`VwBsZW3Hxix>Z{xqKV+^9c; z&W-wGUiSP^f70Lf+^9c2(!mDnPwxNq+{VT)_2*)cm-_Q}LHFCu$lr(E`F{+%$EU;Y z$$t*J+x{h7e=_Tmq56?%zS$(}WDVd?Wd4y|JL&;F?ev{87uBOEezz#Y`dODXte^0rvGHho#BtV3mU*wF zSJ!Z`BD9lqtW-I(3OrWOT3v*Q2k{S6Jkaas(TC@W2jQ0beU}7Cn2Mrc4_FQ60puZ4w! zXPt9zDy+L|QV`|(G&20xUAWI+vC@U$@HdMtMOJj|PJ2?$1yDvMn1h_#Eg8yjYRCy! z6cgl0PNVhFnHVx-L=BVUr1w?$7v%>*>#c^*k{NsSm3wkz?7!Xyf!faDKj8WIIIQq> zz=hx86qFH%e@y&;Nv2zcfbNeoF8n6);P7{N8Pby_Zpwx4aOI%qmrMI1`;SP35=GJZ zcl?MKVaEF^)DfH?<@i~NShcL<@YUtQ!;SMJ9LoRaCGbMwhg}33UO5i0{flOSG_TN+ zd#7`L*UKOcul;e($@rQQZ;kVP5b#~+s*|479h!gE2nh16{g{T=_(O*a-w_h2l8N7t z#IF`^C(gL=y94bns{DGS-@4T+KUdOTL>(JF`(Kmm13lg*A5DJ<68D3WaUK!&`p_va zMU&1`P_dXXsOQ!X^@Tk8UUb_Si-z53-nTYdW zlzFNi8T+r7e*F{SQI8&x@P{OPmd>A&d8{58mtdS9;ZT2X6Co;mz07m<7`Fd<=hxAQ z!`~-@j7GBh63+i|7e3>{5?R_?L@0Lu@eA}yxxzpv?m``q4 z=JDQpw}@#^JoR|MPm&Pw?YnnNIG`lR2J3tFBlCIfdsZ^b54E)+!?E^sKmN22Y`J%9 zvz`@EODXg|^>}x{D$Ucz#sJdw#K%IXzqh$5%m?lFT^;6sd+OPbKHj^xYhP!kSGtglb`8%woq2r!lOOx&zNdSwdt;-j z-qDcRXL}Ukz1k!4zkR&>W9WP&zQ?c&c``?r}eS* zC3Id>=-uA^Y}eyG`^--_{!9jE2*ls}Oy-Gw+8e0fPjB{^B7dzV>c7SF55(QO|EW$+ zelWkh_+8bV|NOE|wAmOhu~MZv_w{7}%XiowTl5*<1~1gRZ745LfVcT8QHPE|-|uoK zctn=AEVZ{BFo{Mufs6y@3P-!h7}6^`08s8|uQR4e(xmNL=8Mz>W7AqXQvAdVj-L z(zVqYCtrd1yz|>m#~2wZ5#0M5w%;#)I$Ank_je)kjiKBR#c%7AW~?1jUHHKHUHKOL z{Wu?x38W0N1%rI^gZ6vj?f4765QVgnMuVX^7*EmpZLjWkfPR=@T$_zIW(Q;Pj4F;# z2Xj;45zEHSwu%Mqr~a|2&^>zNfAP8Wj78^@$${z!UlnR_zRggQ-$hQi0tO2w$k5TcytHtc6ZxLj-uwUOS-y zUpR%FW2@1*4aZR&UZ*@#=Peryf4|xPaSk0(9^SObvIxC#bF=g-&+bL&jhi1Zb~*$B ziI?vpd+bC>+PqGm-Yvts8$&j3X*4otDvF^fJmwb?x8+?78@Fs0_p=hE9r6km6oud7 zrdl?<@&yyj#w}ZnOuGtE*uM+$-z$ELh+oRZyQ|=u__BJ(x4TSec@c`fT_|u);`!dh$*NG_N!*|;K5&Pt0aiTCelVo^hqM1Fek}QGmA{_)&()t_|2Z#aWjh1m zFXO_tr}~FG3VhY^>fc!Zo9W-W=|s)vZ~a_t#g4i<33DP4t}xu}Sl{+}xscX)PY3*K z-{%~A)=wwqi&qeh_i`Xyb>Q!X{?uEkljfi7o)mNP4oQ~mI$rRnWRQnmk5|m=Iz}w8 zpV9YcZXHCvG#;(%dLX~=kvxb6RvVG6%E-=|eua+D+31S(xSUwym)m&kf%1{fn>iuJ+KJrd_7%Pl)}fT*pIp$ z^!ahIKT4mKe2CKToQH3oNAH~Hyq}s!|DqTdb>2JX(GSj}e^-k3CMg-&Qnd%#gWM(& z8UI)2iRaVvq-)na_P>7~{lYx)%WV;n_IOqsBe%ub{Qy1zC4pswYshsJY@_ow|g>vWDsZjZCaJ1Cy^Ks>TOUO4no4Lvme zAM*aR|LVQRdZ-@kgXNaxNFD2p%In%N4(u1J{&tA+opnu-`Qd;b1zLNfqa+==sG6gIF|ALQeN(2 zDICW$-k4%P1r=#%s483$=T=>8DKuT4%z`-NUl{D2$p8i+# zN<8Y}Y2u-uA->0@cZ_)GXNgC;#)*f1j(DVNf_UiXiATC7iHAN#e7&>(67kTdiAO(t znRw_k#GiBH`z-O$Q=7bg68$0aq1x%_KcT0I$2_Elc<8mn!_GS5p=XHicloo4c<7D9 z!_Fq+p*Itca@(zTO#=xxL!UG2m}-$Ojo)j>S;PU11{cM%W0 zn|RpSLp=0e;$deW@z9SD4?B+%4}E}m*m;b2=*NkNohOKgK1e+593me22=TD<6!FkU ziHDu1iHDvi9(JA~9{L#Zu=6bO(9aQ{{>y~z049isex7)wYm#{A7l}tZnj#+hCE}-D z`M(Hxi=CIeH-y8$1L&CTZuzkVp?4AwJ8{1) z;)mW%Jj$_$;)mW#Jo@=Q;-McP9_h*v551pwr0WRr(2o(1bR8!i`U&FE&kPa|eTaD2 zIZQnClf=W$5#phrA|C4`qr^i$O?;1=59f)8ewKLT!#MHK&k>J&m>?eddE$`|lf*;6 zNIddkig@UkiHDst#6zDY9(I~{hT9kPq|-6qgPkejp;r?RJJZBNuO)ukl~*0{&@;p% zU7Lu9UQayQX#?@l8;RfH`jaN&p>HF;!|_?-p|=tbJ9iKdeJAm-vyFJ@?Zm^*-NZxh zAU@^B$4=s*cN3p;>FpsNdN1)vS0C}v4-k)Z<%oygPdw6fgm~x!#OIy;$B2i1oOsxI zf_UhI#KX=Z;-L={Kj`c^Nj&sZ#ADnaB_4X7_?(OX4Dry%h)2545)XZxc%6#@Ty7^aLKZ*Vj=}I~s^9SfD;*qXu;-S|N zU+>yuE%DInh(|x1As+fB;?WP+6A!(Kc;rJf@zA#sk9l2|c<8Of!_FPVL*GgKc~_s? zh=;z1c-YxNJoHZDQI1{2L+>X3h%2ui;-U8vk974B5B&h~NLP+{=tqf1x(0}cevEjG z`^Slgeu8+|IY>P8A>v`@F!9h&5f3{@iHClgc-WaI9{L&LVdog}(9aSNJI9HKex7*P zIY~V9i^RjuDdM4DA|7^56A%3|@vw7-c<81++`pN>@~-!jPDlR+Jw-gyRZTqfH1TLh zHN-=&C4SPi_d4RCZz3N3d_D2d8;FOUjl@H5A|7@&6AyhG@fnw&S>mDZARgoRPU4}r z5sz|jCm#B4;!*Bm>2eM~FwdP7x11Pdw6fhIr^>#G{`%OFZ;(;$i1G z;-OCv4?E8j4}FSw*m;R~=+nf*&dbC@pCKN0&JqvZJ|Hh$BIbLrGpP^G%Ypt7dbQKR z!_GAE&})c?owdY6uOl9IW{8KriFnvqPdxM{;(J{CY9=1~HsX=4Eb-7=iAOuyK|J)G z#P4zTv=I+|H}UA__Ye=ggLv54Nj&r};$deu@z8sS?|1h15)b_V@#yDs#6#~V9_4<7 zc<4uoN4XCW5B&u3$cI7Vp$`#{etwvE=qHJXog>6UKSlg8m!G4=Lq9`2>>MK=`dQ*p zj^o5bKSw6;-Q;&dHo*xH6q_9 zPZ5uPzM6RGY2uNt8seeX5|4D%5f8ndc%-X=c<7D9qn~La9(ptJuyY&n(6hwD&Q{`~ zw-FCJ+lhz1n|Ro{hj{26#KX=`;-PmD4?DYwhu%v(?Cc{R`T^o$XO4L2{lvr0Bg8{L zN<8cwARhV&;xW%3Bp&(@@krM&@z75ak9IUdJoHn0RC&@U5@a-Sg{dQu;plmq=C@*(AP z^oP)^iAO)5CLVeX@vyU&c<6P+PrLlg5D&e9c-YxUJoF~wQI5^TL*GU``uQyJ&|8T| zx^@r`eJAlqR~zxr_YjYCbr27|lX&zqUBpB0CLVV75D&eVc-YxTJoJ9zVdoLzp&un4 zb`B5^{TT7E^EmO)PY@3~2Z@J%l6crTLOk?S#KX=};-Q}=9(Lx5hkk~5*f~Z#^l{>Q zT>CmlJoE|Tk*@Q^L!Trb?dT%$(5HyM?CiNjJoL-Nqo1E49{Mctu+#jg*Y9!vmUKG$ zL)e)j9(pzLM&6&SN1Ax(wZvndUq?Lj4Dl%UO~ga5Cm!YAKs@wj;*mewh=-mf9{qeP z@z8e=4?A}f550}}w9C(S;-PmC4?8=Fhu%dz%CVbx=sm=vpYJ6edLQvf*8$?8=ZHtT z`iX}=Ks?fQjCknBiAO(kf_UhI#KX=Z;-L={4?9m14}FAq+}C%Cc<7_VXIyL!Txdc3vhP`V8@~bC!7M=G|VuhyD$A zCY_G{5PFJu*jY_H^cv!`uDoiAhh9fK(v=||`X=JhPV0$>-avelv!{`G=*`4WyLtFF z;-P1Whn=m&L*GF>?A%E_^fuyQXFKuG_Ye;|JBWwgNj&WAA|84-@vyUpc<8;vx4ZoB zBOZE=c#MPn#6v$yJjVS2;-McS9_czxJoFR9BVB{ULmwg@=^7>;`UvqUcYo0-;-QZc zk9K;Rc<6cJ(N5104}F|?G9(o(`u(O?b=(~xBoqLFf-a$O<>?9s~7xA#On|SDb#KX=5#6!;! z4?Fvbhkk^3*m;zA=mW&V&SS(wKS6xXwXZ?qp$`#{bPW>^{Uq^dMrpUPCN_y?VEgep zNTe6#XkX$N;Gu7$_|ea2DSqg!#3Nlhh=;zDc%-Y1c<6hGM|wMmhu%p%`k5}`p?4Dx zJ9~(S-b*~}>?0m}Kk=~h2=UO55)V5Eh=+cRc-VQIc<3jHhn<7OLqAD8>>ME;`YGaJ z=P2>ePZJM2^Tb0xLpd)*k4Wjq_aOwJoH-P(a+Zr4?ROX%6${@(Cdju zxi=6Gy_tCA&o<(rXNgBY-%3369mK=Voy0?LBmT0>&vxRWcQEhXchSi_-g8Af%CVbx z=sm=vpYJ6edLQvf*8$?8=ZHtT`iX}=z`RS>G3K3qoOtvzCy0kWNIdKuA|CoM@v!qG z@z6(^cXpm;-sySbVdoj*p^p&{JI@jieVlmMd5(DKlgv9iFEa1+DdJ)0CE}q^6AwEt z6Ayicc-T2hJoKdZ8t}sYZ(4reQIC|<(H}yuCLaAjnt13n#G@V65)ZwO_!-yUGsHu$ zCm#KL1M$!siHDs{#6xc;9(Hac9(tDedY7NA#6#anJo@=I;-R+_k8LVb{-%edXD&Jm!JK_LqA45%JDey&`%JLavUTc`VjHx z=ZA@hev){kYlL{{r-(9(GO=5B(DH zuydMt=$DCyoioHkpCul48u!C3=-;3xosRh)>`W04y@q($SxY?hI^tnxhIr_kh=-l^ z#6xc&9(Fbo551ZAoNHg(h=-mf9_eZ&9{LXA(T;W!550}}PG?U$@zD1WkAA*`c<7zP z!_F?^p?4DxJ9~(S-b?&(XMZ2@&~wD2pYJCg`Vrz$?njA-K0rLm{TT7k2Z={M3=t1~ zn0WN_Cy9qXLOkp|MLhIT;)h*+o+cjp81b<4Eb-9CiAOn}BOdw$@#yEz6Ayioc%P*jY_H^fd9Xvxa!+8RB8* zCgP#j6AwEZh=<-tJnY1eI3c~zn~8^=+bDkMJBWv!JBf$hMm+3nCm#B4;$i0=;-PmC z4?8=Fhu%dz?z`wF9(oV+u08fL@AL!2PdYtEJoJ9zk#9$chklfJw9^6Np&ujuth47h z@z4i}$2c-XJoI7WVdqKWp^p#`J5Lc0eU$haXa8y9p`Rfh4}BZ)NLQA4=&i(~AKO7Z^qs`R&Nkwq zw-XOLcM}i2lX%$KMLhIw;$deG@z8sThn;=ILq9+~?934l{V4IUbAWj0$B2iW$BBo2 zf_T_DNIdi*;$i17@z6(z&$#w=ig@Uw#3Nm&iHDvi9_{E1@zBSJPdR(e5)b_x@#yC# zh=+cjc-T2fJoJmi!_Fz+GK<9{LRN=;vpNhi=?&GopV(xhI{@^Q9E=DEDgO zq1O_Re5fNHdWLxP^P7l=UQayiZy+9eBk}DnKbwe$o+Tc3wh|A02k|J!oy0?LBOd*H zJMqwW6OVN5As%`M@km!E@z8sSN4k26hu%j#`k4d7L(dTpJNt=;euQ|~d6anQ$BBoX zCy0kWNIdKuA|CoM@v!qG@z6(zhn=U0hklxP*qJ9D`WfP3=NR$O&k_$i$BBo2j(FHP zK|J(}#1FXkHAOu1OT;5x)5Jr+Og!4r4DrxsiSKjv82Me<;Po%`l+)24qMxrO9(tO1 z*jYn7^jhL!XC3j-GsF)&`!^8}y@7c2^NqwqZz3M$-b_67ZN#J8v&2K+Nj&nQjd`XZwJoIYfVP~3n z=(WU;yY^K_JoF6lNY^Ifq1O|ScGN&T^hV-OJA343YeW4%^likWpU)Bxy_I;_xr2D< zJBf##ZNx)wCw|J=znggI9mJ!b?<5|27x5_fZsMW$5RY>2B_4W?c;rJr@z9SDkAD6r z@z4i|hn>fWhkl&+8JC|Yh=)E*JnTG4JoFLbQI4mGhdxR?`uWquL(dbBbe$m{`WW#@ z*IDABPY{oEohKgpB=P8HE)oxYig?(0iFoMK#KX?Z#6vd^c*+e|_X5wMzHsYaYiHDu7#6#aneEQqo z`f3~T(A$Yex^@!}eGlSE;-NPX4?7!)hu%ay>})0;dMoj; za|iLzcM=af+lYtWPCV?~O+55H#KX=G;-PmF4?BB^hu%v(?Cc{R`T^o$XO4L2{lvr0 zBg8{LN<7-#0P)a|5ubJM8#zuq^g-g$&kqp~eVBOU+ezY~j}VV`dWv}Hqr^A6{^~UG z(9aN$etwL2=x2#Xxr`GJ{T%TqmkHvbUnCy+FhxA{OT?p}pC%sqW#VDy4DrxsiEnlJ zY5sM%KZIWGbo7U?Gfh178sbrowZucOBOd*5hIr_kh)25WiHF`mJkr%jJoIhEBVAeI zp|=u`eqaaj(039KJKKnd-cCI1+)X_6PU2x_7xB=$iHDs%#6#~T9(MK-5B&h~uro(I z^rOVX&H>_~A0r-i9w#393F2YrAo0+Lh=-lS#6v$xJo@1g;-Q}+zQfJWM~R1?XWq5n zGt4`EjCkbRS>mCO6OVR!j(F%3#P4wDJx@IJi^O9bnIazgCE{V{H1W_c6AwFQh=)E) zd>{J5ZQ=e9ddlhO4>68Z6AwL2JnB^q@z86DN4=^e9(q0T$cF~vp*IqbaiocO=*`5# z&TYg)&k}#c^x39^daJ5=P>cmPZAG1M~H`hig?&LN<8$_ z#KX=!@zBo_4?D++hklND*f~Kw^z+2S&Pn2-UnCxOP7x3NGVz11ea#RLeU^Bn%e*(- zzd=tr9sL{HQHprz)x;ll_N0l2UQ0as`8wjEXNZTLn}~;APdw~wARc-n@u!{rO~gar zMm+lYEb-7=iATBbARhWo;!*Bx#6#arJn~@=@z6VnM?c?5JoGN&VP`k-(0hmyh=<-xJlfGV z;-P1WKkMviB_8@t;?d8y5f8ncc-Xm{c<6hGhn*e7L+>Ph*4f`hJoFyo(a-l15513g zl=}hVq34K4x%U$feSmo6!!hEaA15CD{0ZWr4-yYMhlqzhOnmhPZ~yBg@z6(!hn=U1 zhn^=M<#>j8=wrmApFc}H^l{>mu5-jgpCBITI!`?GDdLf?OTQ#6!;# z4?A0lhrWY&*twH<=xxNq&UWIV?;*a;y+5;qc<7zPBVAp@L+>UY?Wl)%=)J_JU3>2% z9(s;=^z;41Lq9@1>^w?5^a0{w=P}};A1A)f*?)p~=tIP#pC2Y3`bpwZ?jyuQKSeys zeUy0UXNX5Wj1dq0Eb-{)$BBo2j(FHPK|J*H#Mit0oFpFlCE{V{H1W_c6OVG7As+fH z@#yDGHhldHJ?V7J50S1E@zAS@N4nC)L$4zq>BE9(p_RuyZ%@(Dx7zJ3EMn-c3C0>>(a{FY&Omk9g<@h=-jy z;-U8w4?B+#5B(_d*hd&39{Mrn-TQBkGw<|4;z8(D`@zAG;$2f9{c<9r_qh4Jm z9{LRNs8_SZLr=9Bd!Ro=K2$p$^F!!q;xUfY5D&eUc-UD-JoF6loi0B&5f8nQc-YxQ zJoIMbQI6Y)hn^)K<47y<(034zbnPS_dK>XbS3B|0JBUZRI*EtgMLhbkZsMW$5DzP!_K3`Lmwa>b{-=h`f=i6=LzDW4-yYMhlq!Ml6crTLOk?S#KX=} z;-Q}=9(Lx5hkk~5*f~Z#^mD}LT>F|J9{PFWk*-PNpxU;{8c<6P+qo2bcN33t-$Oj~4&qVnoy0@$As+eAOFZ;G z;?d6^ARc;-c-YxbJoF>P=UsjtB_8^5;$i0r;-L=`k8&I$9{Mox=;u!o4}FAqr0W#% z&_{_!x=s@heT;ad>n!om$B9QjbB=iE6U4*L^Tb1+Bp!BNBp&)S@v!qU@z7_8hn=&; zLpQDA`5x>{IvxEX^c3;1vzmD5wZy~DI^v;ch=-kB9(p73u(OGH=*`4q z9cdf!(6hwjKGjy@q3;-Oz89_2ViJoHP%V;q?#9{OeC zk**oyq0bVJbeV_3^F!#>PRINZ=}Hq1y@q)7W3|LXuOl9IW{8KriFnvqPdxM{;$deq z@zA#s4?DBOLvJM>cJ3e^`cC3uXB+X*_Ye;|JBWwgNj&WAA|84-@vyUpc<8;v!_Ge9 zq34J^F!!q;$deE@z86DM>*CJ4?ROX`uR=7L$41rSz zdL!{jR}=Bjv&17^t;9p$K|K1Ioy0?LBOZ3P6Ayhi@vw6b@zA@7|Nl$7W0<(^JAmUa z6>U+A6qLG>+j?+R1_sx{a5jhJfi~D74Ng%^B3H8XIJk2_J~(vuoSq{F1_lNX3=AF^ z7#J8B7#J8_iNS*_F`x=nv_%zCJurA+U|{gjcf9|Xa^FO$npm=r`|^43e)oTO@6N;l z<}Az8XXKePk*C+?nR8a2J}1wd^YZjXdFEV_r$3ix&SiP}iac|!%G1~6nR8v9zA4ZD zr?4eYe<9EP+Loukl;?T$N}j$We=R)Ucjf8(@|@=nl7!{+yU^p?B+qdimZy)%bDn=7Pk$)S{Th|07v;HM6Y}&#p8Hjor_aiBo|%)U z&&xCCf;|11JaaC})0gF$b48xMD$ks2^7M6i=G>5{Z^|?0mOTBXJafL1r|-x!=dL_` zPo6pV<>?3V%y}qJKa%I?ti6$^AIo#UPUPvQ@;r~;%G0C4=J$Jpyx$9*_j~jX5CfeP z9a)yAugJ4*tMc?UdDd-Rp1vW^x^2qSx8zsCzlVPzPv4Pe-FD^ad-9wo_vPsa@~p$5 zJpHvi>u@AbelUkuO3E_wQGdDfv{o<1PYIt^E6d*OLBEKeVm zXU?KLeL|i&%kuOYdFD*y>2>+}@H#syPhXH{&S&!UMS132lBYkHXU=7L`ilI(-@|!K zp1vW^oSX9WEqUgAAy40yXU>=M^jGo+Vg4O?`o27K9>~)V<(c!fJpD+XIp4_BkL7dW zb@oJ_9(~+=o?*_G(Ea>vm1oX2d3w7%b6%3CUzT4A^Iwstcgi#WHF3MnP zye&`fm){KY56IJp<(YFtp8i0dIUmZ?N9CEbC{Le|KMwPk<>_^K=A4zM&&f0AygYqD zo;jb%(--C2|CybCmgMOx^31s^PhXQ~&UJbEhCFj_%G0;xi(&p3^7L2o%(){^-<4<1 zJ$d@RJaZn%(+}m>!u+r0>BsWSc_L3gm1oYk^7Lq^`TWM5EunLMqqoYRg!$X#>6hi1 z^NKwEsyuUc$kRLJne&=Fy-S{-Po0ye_scWqfINLro;io)>BI8OIU-MgAkWV~eJD?# zkY~=aJbgx_1*=*{Hb+$ z`j$L%zL2MH%QNRodHO4P=G>8|@5*n5?}NH0Pd}7r&e!tvBYEb0BTqk;XU-FO`l&qM z|NpH#z4c!6`HeZ-Lg)NOZ^Pw$dv&YV0wFVCE}<>~$M z%sC)WAC%|&qYlZ_AILN3LwWkBJaZQ1=@atIS(c~I$S?dWd;gQj)92)wb6%dlAkUo7 z09#5`9hw)Er0&~RpjZr^31s>Pv4hk z&I5V+p*(ZGmZu-dcZPq*_(q<7D$ks6<>}EU&F3rTYzdw76}?rSIossv?efJi|0Q|) zRe9#@kf(RbGv_sVdY3$N=H%&l`PDH0ZF%~jJaZ1o(}(4mb3~s0K%O}t%F{>X`S<(c!PJpGkCbMDB~cjcLLPoBOnKM>|Wkf$HXGv^z5`msE7p2*Wr<(c!XJU#kD z^ZAXh&$oom`HkKx&)4VMg{b=I8|q!=y9n! zT`Je&vB~K|^=WjyS}GT=KboALj0@Fj;b|0)KPeW53QwlTCq6!-cj{3`J-DP)Pog*_ zNhzM3PSVd8%JJh{z2n`b@$R0Un`5P(zWj~x!dSkyr@QzlmoIhq-Re5qW$GuBXyo&I zckh05r&_I4@0XHK3zM}{@ou#;^}+q5I$WqGwWx9PadEOxE{!K~p;jwZleqR|e7sbv zMc2pDKfY}j(Ff^d1NVnM9f>n`ej^{GyFI(v?=*f=PA7?{dg7@{DIO_J*DBR7rb<(l z8{PSyuHNp(@4j45?~Qz}@!^doa&mPpd*>7q|Vf=BX z?xx2-UAx~VNp*58UDHO*e7a#%Efxk!UrvsfntJxADow$ajP_{-x>Ae~_i{D|Hk}s#?M~^1sxK>KyQsar39;C`tJkxmg#rgD!es+V` z$7{)rTjyts$7iR4#?4)PCWLx^Uuu7v{OYT6>HPZr?0PMInvBIg=@EK5QyM?pLD}dXOHV2JIGHNMk1ExvLUO*J#+^-9FHOJuN&i}z{mrs@R};VZ;aPY;yEy-3`ts29tMP5M z8|_C;HH!Mf_1ocfh)=ikLAY$ZzOETH?|KiUu+&NhVB0n-nZ}xw&VJr(+?Umc@0Oqvia?qpY4C1 ze(>(MY;W)Djs1Vq`fXp%4jk*$Sic{qzuBJtUV6uUOj8z3g!vmEU;JhJIBf6rWBYb( x!0ofoN9l&&H2;2>e=)$u{69&5vpw^#r3>%dZfsJ&k*$9{9GJ!!Kl_I5{{u%nkiq}} literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k6a230f0482598b0b12a92ec687a861b8.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k6a230f0482598b0b12a92ec687a861b8.o new file mode 100644 index 0000000000000000000000000000000000000000..0151acc958c7bf37f0942da9f0e7d662e4ca8e5a GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Kp1A?nbr#-8ITZ&*Jf67*8l%i^}nkAzp7iezg)I*xvptU6^*^ZVl@S2EVDOu@8P!| zmI_KY>!}&A)%`r%ZTsK!LfdC%Fm`yzmfiObUgvDtbN>a`^LYI$?$LCjqNVG#T|Wr@ zb{xC<;RZeQL6t3Z&BhM@UmNQ?+QANk_DD|#EBs!zr~kte_SUa-c6df=Pt94| zMC$8EPdY37Kf0%X80~b;_EsIn9y#^&VCDG(gFD`PdT{0Grw6~Od#-}^8N3e9Ir;5y z--+YlN1fjZA3ty+JTm`exNrWc@cz}O!$2=EI?p z9R~)Vdiv?X)cH$7b6UO=zA*22c;9=AQ@6ve2B?CkFeVi;DBd(f5gP=)|dTrSo(+H2+fgsnx{u(opEc zfx)>apBM~*=L_?{9o{$pP4ERhK=wYolIMIOQ~_OHcnCDmTT0@$!=ZQI45w^88J=_g zBKW!(9yjxm&rkQXk>A=&Uemf8d1uC&9mj4{Xs`;p^}L zO(Wc-z9HT5D78DvSmCRHyY8{$HmPnVJACoK>sK$n!wwJsO+sk+TkP<~2CS(QSpUe6CIglj8h(Hs{?`XU`(Kkn zXE*G5gytG}K74K0tERam)^{P#Met9u41-r1^Ieh&GG2m=PRK;EB~M~ke?oH=@)_k~ z$?Td3@?7}avi=dsS0UvZ0na1gc?3L@j3nPh@Hr0sdm)qa@3hdnkYnVPgwP1&co%XQ zMCzuEDDk_dU2C!1V-P{|oo0as4+M zANG3Z*@Tb{c2)jo8_%x31O2;Z>LC~Ga}j<^cKS7JKsLVso1b08`iEdYvV-#J_t&!i za0%;Y`oJS(Q)U0l--4~+%kRRE--FHm4fc!q13P^A5bSEOW7Bk@5AxHGtBOL`l8Ud8 z{qE_*tdHm5Jn&=wqPJVESHD~P4;pi^%MIuRMQ55u0X`_}1Y3cZ1D)@{)q__OUg^w5 z!;UIDe1SG!O9!h4CYP`}W}cpktFAHIc$=o%7_*PVk|&GJ$?UlOvdC+T6hdeJQ^+LA zw$zK8v;RR{^&KqH!c&QG%qUH*6M5Zwlr^R%U{x-U-$2e!2osng#~qs9$r$HpO=@nsGtu2 z3Kr)*oWEpgU0ojJ6ms1KVstcl1HQ()N05N8!xwlA62VFVU&~$LZIHuzR|wMRtMA%Y zw>98x^$F`(0p0~B2&P#BEe!#$zll#ef1tgyy|deo=7ei+>*(~hb>`g#?%Uw?xB3DN zcZHc@8ZL{~mK-RTPHsRW;4ZMC636Vb9D3JGVP#t`(De)~gNku2R_!`O9w9T48(;3F zA@|adJ1s4YTCGY%g=3ceAWbsOKY zn^8}4I8F*J{FX6-&$D(S>xyUl*8vc1?Pur^gsONQ!Hepv z*H@HQ6uC>V|HnCsD0E0dq)YJHN+5q{5WZN_)#!E8=|Xdhx3jsu&E47F-qJD8jhXH9 zcKF=+ixw=N=Uz{j1%=C4aaVIoqr0QKqtn;wZtd#mbT|68Hn;iQ>#C|&uB}^NRCRit~Vny>1Lmywdp!KL4=rx`RCdxP766ktgKQ zeY&{EEwg9D?r-Ke$JUg}>|5=y3)((x7k%GHqceCQPKGY0GrO$Y$;KWEaY6+8}ZHzZ7n50MPPo}Z-z{G@!)c@e2BzBJI8K+Modf;<| z_K9@XDf%aRKFtI@`iDMjB#yuO1I~XgamAzlJ{0zaFqLC*es6WZ&RSK73>V{12ts zQ^}vPQ2Mt(Kfq@Mb7!JT{>9tp24RljT5U4PH<-=JpFaXD6^ws@=pUSUB@)qM^mCk~ zzlfBttQ*S&O>F7UCVt>wg@Fej4U8oCuf-dCQBJtUh|1A=)0Z;t<1-^bE{OiF7Nj~oXSS;D z2j+7wpRY8Zf04$DpVtIA`7iM0&;Lv|7xmHond#xLxnxi7uNaT~YYUOc`H&HuIg{ca z_s5@8|0_)W_wJe=^QWPMSl<$C*ZTiI(l5|Cyw2Qf6?_|sECzk%{Iw5mcDv-As;x4 zCbWL#{0Zc@8T^gpcUb(*&f}+@;Pj4|zd5)*$luOs z`e^=yxAOHE^KS+0CyJPV)_6$vJwpU?{VoiSA5Zq@<0D>_C+G5-iU+;Bu=K0={ zn0z@-#7ps6>3y8vG`*1GkKYE9(fUpEN9XZUCp^B~@aKxZmqCa4`I4&Br~{~X(Tyy? zAM9p2?RpZ6CX9#tY_$B6Hnu*=`~dkI=FfcMM-|kM?}Ysx2Y~Vc_z%V3m&qZi{Rbrf z(O7(kJ|T}KzUzkgo*gHiBL^mvNd4-8%vii0TBxlE*eP-w;=8VseR+IG{sVu;N5|yP zJxR2_mM1F#ggwpvNBXxJkVnrC_)y-kuR0hq@6@)7OYkK;4K>M7m|fC!rq0Th=qa zzS&(eo{Z_=RSfGj_YeOT+MmEb_R-{|qWlB%^DL@lU)Ue{ekQr2h=*kVDKY!Q|IPDr zA*bi|q&We776zy3wA?b%-2nBOf5iCIKI)Gjrm<(j9LXYQBWc86VK3-K z`y|SDkWbqNo&c|WeM@s8KXXU>gZ$}%>G~AVM_icC!&Ry2foy#$)hA;DVSJ32lZ86p zZ=yfgHdA78|m5Rrc|H|nI{ za|=;NxX`{7+1e;nRRm zLfwdW;nYL5Cww~UMtsHuGkyZt?Md8zIK2SPtK%;(%O@^L=7M0~kWWsYYr zB7fY?QpvyJPvh7dRpd{p)WTotw~Ki4MbxMm){9|Oc|4d-_Dg*ZRT0lg--+0c7~^?j zBK5E0c|uUkCRT${kU#9{W@vyy=OfsE9qPtXD~eA0bL@}BAtvn)-{AgE+KKp!<$9L9 zSdt6=(gS)botQ*=$o_}4<!;&{g^4=p?tvr;)yiYZs@pJB#etA9< z^}kp?!_OzqrP*(ozaYL{H-Gsv_#fiiXPv){&Q~1CFME-GihKm~&E+6})k_-MRS)`Op8C`N%v|{2IOgrtAVm?!ONNnVld!|G@cudDqKh#;c>I zcx5vA&u2+Y9me!dUz5-CP{CKJoZWM{4wWQ4YnE`}lmHJP}e3rMXP; z3G1h6f4!CBQzGA=^LU2*D1En`=TF!Vq_NkLzwS;*q4;~vABtp({O;D{nQ6aaj$atx zwBLZfP4SH5&HE42egk+@d>iKZ_a}^Jmred+jc52gDQ180n~w9OfrC;%upjm_lj-vq zzCNGB=Z(82#`fnA^8GnKKjr!M8w)`{*{~l!gwGr7llgv}`^SHz{dnxW2jgM?6g$7d z`agOQH8Vt+{UBXhz;SV-)@v$6lqnn0h=8TQ}PXuoKPcTU=n#P;7#=NS8M9H>ya zop@Ju8~g8ctd?L<>{rp=g?e;9ihON?@w^H64C8qe`F|$mn_~aT=R;<6|H=2?iLDxr%N~J6ZTKnE&4R=N5;-y|A6!TQi3^uMSfep@7CfQ@*~6d1Jc+k zI&9*_kk8VRAotVxoh(A}kiTL5!RN*NyjPrO7|z3wg6JmB!;j%S9F#qY@8ejdDb2FK zPSdg9XI}#W_h;-sZ^$1$4~IX9^YtwH{FHwlsnZAMiBt!_t1SC%zF&*V=VT-wKVKvv z#rkR|2jT69)2V;@xJCA-#m@ipZcbkF`9T`{BKQ&eS#*H&pYraHV)+R^KQQGdpFtf` z-7EJra8CpGG;mJ?_cZXSYk&^V5o!nUEeF=~i)+9C@JErazSLL!+s%LX4?nLQ_t%Xd zUHF^Gp+Yvh@2Or_t-Eh-&au8e#`^8G&yKwR&vlWnaa78hgum*`>Fw)FVP}5)%I|*{ zdE3s)YS+v-rmv|@SW}DL2txW#XH#6yr!S6bn*7otucI1!a1QWS2%rSj#@SP@@CaSR z19hyf;cu3~`#5LH@36gPY_GNqfn2L+ppO+j$C;?H8USpXMxUTaphGoLVeLo zwBx*uv+{;!EQa5kV#QOsqc3$9Twz=Ac@n;-nt}weS4#xT(Gm|IRjez1d&=c`amU9a z`4u+VQ*m38P4YxoM<0_^8~j`1q*YZXtKPKB97g5t3C&h9OZ5 zlG#5M$?T&&iY#H`OwvR-gLF6IZ1fy7b8geEoX6F^(e+giV(z)Q2ldkCtXK2cfK|oFkAM}`>nVDUBk^~9rO-xs z7mck}6zhr+QNBHtLz6i-%WMpngV6Z5Vl!HqpSS?ZkMT z>8Ul&D}3{7OVY0NGKPnp@Iu3(AYR~)Iy4+$d|B1;jTTe=Nz@gSM2|srs>5g^{-92` zeCY!oY?$cZw$ztuX1vEzueQ`_JnEk>5$J!TIMF5$AOD7R@`zh)^GWH8QcLZ(+DE)L zH^ciGzM_sb+RgMdRv%V;ygolWZmismRX5JWUKxY9mkGBbKfj@0#cQu72~Djr+neU4 zo)n#zZoz=HeLQVjX+H0Dqt5g=G04h)JbIM}NFDo;mYUZtbNxx+m085vWwWeZ{M*8Y zwaZNZwx#`2WX;C@JeGR3rH*X%`fHbQwHLlOR(!irG$6;3Uh;ir*QcY(11hw72K8)1 zo$yrW%GAcCBEzE{fK8hLHr8l*jtjBbvxju7Auwa5$6IVyKslW3)vbC#oqC;tI(0?5 zrcrZLJ+{xV=`>1F>N#>xZ8;x?2v2+tJS0PSsB^PjmU4xEwgfc*4g)~As9+kz$ z$@NU$YsvGdAEWif2!@fFb!0b44ETGo9{PZfnbzpfVvYPX#{BG}W<2gcJ8rD@9II}e zVfSLVSVP9r*(b#tb$%~~n^$oWD^z?K9d~Gt8UG~BvbyWm)t};pVJD|)SCGZ1ot%ug zYFv|r{vdIQMvlcMTf&XG-+|+GwD$gouA63R z;^@;tD?jn*E&Cx;J~XO*bkE7>U_x{Sq2&j~N`I%-P5Ozm@>%7HN3S+_wBmyPj^1B# z{>^83pSQEU&%1t2^U1hl6R|!P_4bb1v3cM@v6#J=8Ka(>wQuSA>Qd~r89)2huE6`~ zFdVyWRmL2Fe^%lVpJCCFtmPbC__zr_Aif_bw*RsIN1u#y2@L(~8t=bH@kZY}Gxjg> z3eEq_ZH=x~xNyEe^GTF5b{yEmd4`d8!)F*RR?RKfo)rRsk8B{&S)6#t2N_kDMk#OR z`9E==?X%gbWp78vimzOiPr5t}-j@R{`|3eBQThKM$vwaBuOL{}G;Up;f)jHIbb{_$p zAIx*A>%Yz6eek0p6VWmbAKim^P8WA;6TNs&SB42jl_xQ!Y)m}q#fTKo>5BEE{6#&d zOVOe^FXA~}nF<`eMxWCqA&NdqaGN(Y)A=!-rM*Yi8=@a53qWjT5OvxF#B2X!+3rgg z)mq!*k0C8*T%z9<^hcT~4%vX#I~q%|qclYJu=Yvqs^x@lqKctdX~{3-XX0W>m9*cN z?T;JU(ydkR74kgd-P*o_kobD0xWwMSBHQOmkkYZWJ?-b7h|~U< zrM>Fe+J1&mh~J|K_oBVWGX8*Uul!Gq zui7d2Q~=_&e>+ZlniAIb^kY$oI)0ehr2osZy<|UnvFd*<+W)6G@~ifjWlOjzx4(6K z@(I%EmU>BOxn#-#)m618@a?waQC!YX%W%>8s^+b6JmOW&3u`-7_ecjZw&Nf(u)@ULChQn7P+LqQW@YikK#+OllTrj3 zM@wZ@?b@B?%UhN|THEBWS=+Lrp!U&&ac=h(ijWr#cHm-hrW$Wtv#`4;(#uY6)F-9BsC|_T`zN?{N zSvQT>xS_JWxMF8%{i5}GE#=!PqvMuuTkS1c)?QY=ZJnek-_*9gb>pJ-+i!}lAb%{n z<(pfBO`8{0K_9KSkyv3{YH??jn+g?=Dy=g;XAnuqIJ2!3u-}U9o zA8+U`Y=*oo(ADGhC53sujm7@@Rv+xMzOue;&Eu=c1}oM#dyCetDsEey*U-AYseW@! zd&wqmZLPm%SxZS%d+CO4)!yQgRm;m)=Wi}A>)O=X(zUr9uSKi6Hm|7kH@2?thW;wn z-_*SL5#?{{TG!gr z9H`r|#9Oc^Z)@J-r3Hn{me%Lh=NEXF75EyKEPdF!bV+_aexZcZH8gMaHu$dh6D#ZZ zZ>!L+s^Hg7Zl{uWgJ+ZHFIv2K@uIxErJQEDx23~ZC*DhH@q;oxZ);vKkFi}jcl6_v zbNq}pPaq<`Ba}%k>SbMyNX}hOu(^9kXAQ$|8q%t6`xrY0BxHs-GpQHts%DUA1 zJr002{@n)=qF9c{o7D2dcz- z-t#)%H5$b!@t!j+=y^mO)xu^NS0M#`j2|;x?t~N&S3z&wc?)DaE*XZMo05ydB*B&c z3w<&6O~|ESq6hI#Un(bhj|3{e4@05x=vQ3_{(H2KM^F2kc<_g>1OEZw#LpRS2gcOA z(yqh*^GLjizFL~#H7N&;8E@P-q`Yov0yV#yK~Mc$mT=X!K*H4~wN&<>Bm1eACR2L; zd&xhZF2#$QSIk4O*ju;~{ECFzC0xlj0sSf8A=E^PpJajON%$1NRa*jmQNpJJZrn2NHo%Rk;xP%Q zadDa$Eyv{@jcbILN;#>2#T${XlyGZ58zp?Yr5~j)>c@CW)MMzMo^ztga|XDP{`hs^ zR}65Yytz{ERz7D-xK-W(X&|e-uS>YKpPxy%Ro-73`Z4nVkpXU$H&f0Z8rK*j^R5Fw zBH_f3aW#8R!XL802MqK^`@SaOR{rTJI#h_CDdLvDFP3mMuJXku30LDP_z?rWaop!5 z+&b!50~qR0UrJ19$*&B!IZpiPR-aC z3~-}dO%ks3py*p9+$!&j5^j~ZRE8WiZk`->r2%dnHz?t1Tt&ZI!mZikpnuh#q7;E#%e_elG# zu)x)N@3+9!de21K-BDNH+~y5*Gwcx>y(_(4Z4FJ2@}Hb6!vmS|>yG?4p6KTq+0E`e zsH3{(wr2dyV|_=*tu<@wY84DM`#RXzn;f}y;6E=pHuYn^KnH#rLW+3A)#P3hmcXj#*D(KPB1a0(|`W|a;@R{I3d%h<5 zb9ZC2xE15k&sWs7(a+5#*-l9Y0k zAX(7R(V4&GrnPa0?1gXMxN(0l^>#M!rZ%5^H!tR6hx+Z-JM@=b#RB^Y-}fSj$W+VJ z(-oPZPL=tGLds=YRT=H`2`1BmDBL6O&d4~X5Gr1&`YhZ~%H>f2dG+rUH_yp@j6PeV zcqhYgGza^80BL+d9a*5#RZZe50SiP87(d-lN5K>OeR zd0FDf8HD+LftT7*eqG+fCk{UJz=>J-`^JM0oOsNi7)2g~xc6^9`_#z67oHs1`qon; zhw!)V`u{ny<*gr$%);Mo_`7o7n0FbRuug!p!C; z4~?At!l99sFaBuc;2VE6l6mOOk+X;1K)j|*=HZ7=Y$?lOvn&d~xL9 zLq8gM?ZG!j&TgeR3o>I+^RiG2OcNd1o}&XP*o~o_9~$jQ4qPc( z&IE}s;`qtx&btS$EKmF-*!dd+S9aWsFyGND4?LI{-SoT|J+M15+OcKe%6L$;Zu&m( zeZXAgH=J96NYz>fppguj1)zpL^0ck%bP z@b@|VZN}e&_`4i`zlOhmiof&4=-6)YQZMSz;jh}iVPNL|jsBVYZy%`He}}(j|KoT+ zc8NH03gf`oL~-P&x#GxJKpdeqLAw22*M9%Xj%DA!vTGpc$*vplyo#PfPj=mc=QZ>k ze6s5^c>V-E2cBdZ`AXvb13wAwPn>+Yx-1~x_8t2{Xy?7b(Yt>mING@c{P@kp_14v5 zzqRzy-?iTCdt}p=;3)Afz5CgqSi7RNW4`2B$DQdbJLXDWb<_`BN%iRP$5Dq$)S+_! z?ccB1f5$+@{sC17qW@7m>pD=KexhWP>ay9tVsul^iqS3k_*+wy*&M~+B6=>&Y@UL@ z)95*t*)(It=;jJM*Oz5B)zG^cna%b1JC~lPXEx8r--YyCn%TSrf3KqFf(+OFk5KpJ zA+GzFU)BBH!06pS4~%xc+s}1Ba~0M7OS0}~1EY4mNBXbqxbx_h9dGrsd~}@cS9LB$ zoy$O8}!vsvG1+!wjN1ItG@J-mE$%jcJm-tz#SzgCjj^tI)qn-4A@?R{kV z=*}-LAHDZ?myh1}2g^tM9!K~>0ePc%(=wa>1o6I&=ecX9Z$@9=jE(yhYZt-zk)Pey!&=gK4E z$*#{|^JIC%hrcWE*N?x;@i*|KJN}ky+cRA`ZJE1oS(EALx-k>!FFf}fGg>nhb%D&5 zt3H!yx=myjUE|BdL;lQFSBlYwx?m<*|LIKq%1~y@tBKM2d17?SY5e`~IT?4p+YjEB z*;~qI30gMCP(4U*?{LiP5=r zV)U)RgoAGe@(#Wc$UXRaAbN1i-zG+@ekUhWgR-yqbTCu@g^*EBH)tUG&SH!s8s851 z4tBnq7;PvQqaEMaX!mQ&z-_&vCU9HNG z_D{9bZ+)EN7LSX&^=m})dEOtS_$A}xKSlA+^ZxdwN20!I#=9zDX%`^L5zw<5%-qJkN zg)yt6;*$Lo$A|R`joaT_E?)X+>$O-fT!VGPip;xrEXP`6S?2xwmS$e7{L5{$p4DrP zvE{y(#{MIad2Lo=^bFR;r_kP~3qv$)-y$Esi=ZCS$%)a~zUYI0I3Y1oj&za2#F48D5hsb~2%a&1o}%X) z6N{raCl=?rKD9%vtwX%4eg3z7_lUK*g^AIr6B0+Zd^M5TMDN4m?KEiJihOoO5*(-97k_(c zO01Xv8Lt#dx=hIhJdb5OVMp>DqWn9d_t`=)L^MbEmP{|(Q& zp7e}*QvII!4Lqa1SC@mI^i1^*fiLt7dgp@9A3rnqb&ALJ1#PExsrphK`PzX6$J@^UN2?Q=Y2Zz5JaOdU*RROfZ8gcPPx&0w2k&%!=A%CC`H7LokO%VP`tvIY?5J_YN)u++bHl=1=e6^?x-f6naA4J(iH~6~rAv`O(%fEnUq8sn6hw-fFevIBL zx|h;3=&mEV15bU37s7K6;2 z8_z@+t!sV<&qUW!$R$1NHl;Xro0hyz@pPXo|DtMB^vR`Pa@w@>QQ0Rys6X~2ociN4 zUzYvR_j|HGF8zJ<#|~Fr7npS|FGXGPPS>>&buG(DjPxU&u50Ii$FpjK{y)I8s%z&T z;#svp%MhNauKgH~=)J0IKRu(ai@^7?A1r!(DdLr3EE&RBLVeDn_mGdr@V>U&MduYB zI*UOk-hs{|(YnNuI$t#R>QG|j6qT#mioi;rPt4+J;Sz*$1Y7*$dJI%MNz_%ZVK@a7r>G>+2i5?oGzlLW`&)4y+={bOB zO%FY5dMJ)9BgL;jhOkR9Ztwd3rLRAek-RGf?{?+LaoIu`@zHk>bPu}dzRsk(GX}cx zPSXuOK9~uVo+DnCBdw<0_YFKN+RG2&S<&AAB%T%R*vHzm`{-HAH^p($-ifd#(D%LX zH@)8fN0Rme(EdFS?av^rcF0BhT9fu?azQ)ZY1(Ij_FXi09Om-m+(FMo_hj_BKf$x2 zyYo-+Omw3ymVOh@s$ZU>_gYrz*`~YX^=A-IkG1`OruyYGkkxPD8S&bEzPF$GbM#Tj zG0lOeo)&ArvHOy9xxibhmu5E6d-uHAzm(&&WBx4vw(O^!eW=k6=lAfeNi1r_OT!0S=G_PtbeZRiABUeahcO zINo{dv-CLA!Ls_;OI!}mubux3&$_+-8qcb|%KsM6sy@&B4W3nfI`M3wKK6Kpc(o@J zN34*zu?#X^@iOqRgHIC{OYQK-{|-3j;I6+1jyt&GAAlPje422=!H@qV@b4!^X+QWz zVA{b0cPz;)?oEs?>BD~QbKu<*iP41*VgCQ^1oUGi=W5%Q9w|e+of1fQDDi%6hs2#e zw81wM@8@If%d}%nv1dfAjU$fra_47sypum4TdTv5Q8>~>DZCr`O!s~E6xHWD;0M~R zdi_yn-r!r4`sW0>O-@7L>dWRVOw~1o(%r5Bv z0-rc{>Mx+%`v&gc6Gz^?5BDc=uLizc9Dy$L$eH^nUfa2MD)Tc3cDJ27v!3uL&z<=s z{k`$rnVabEnsaAn;qPwztt5Q&xu1Q-pE(e?>D-y!;>dx(T1t21z;2{lP4V!)nc{uw z+?mS>Bm6SLx12i@CXBzigl|1}ra+7izB~3M?4jzxr%5Nlf8t5vxM=#g*Lw}>4V`rT z$v`pJdFYsEIu)2$UmHZfoFL}y#Ta%H`%CNv_rF%Tbo)3&uwPt({poV- zPnTuRhFjjF{U!9awmn759m(Z-$lp@PH|#zKwcY3YIbxEQRc~DEEeAh`G?d@^V0Q6( zvW&MolwE!SANn}xu%MgyAUeUc4~+x+)2Ob`H5VCwaTf9CWD&3Nb~Za0@<~r&i_o7`C=T%s{FUJ%axwh*IJD`1WXV&_ zt32@bgT_tWhpC?zX4wXlv&?VFFDSduTx2~@WEube%`)CUWEoHMS`;LpX#q&NNU zEX&>H&u)Cw`9F|lzPq!`_qUK=Hgfe(zn;H&4fUTrf4_}(%%**Qnq_-yIXahR{P(kr z=gU%#NZvoJU;U!<9z(njlgo?M?PMc2?lC@_J>kXf?flO^YS;dQH8PCh7Z+^sd7rUO)GW;mYBfC&pP|Q1e+1OgyM$(SvgAMJ-Xm;(E z?Rb;QvVM1F8E-?D@jj7dytXXkU7KaR|J&vHF^qBWe>(Wk&&Iw)@>HFrJbl#u^9sfp zIzy_*8vLd8l1Sr@;r8K%Esk+oVS3m0=FQ*sAx5pALMe{V2U!(|eG=2V7Tgwc*owai z+!)72p}k*JAQAOc?oQn5j*7Bo(z=VL$59?bn7elJL0{J3{$6eA7 zZK)3Mi|;bcI@+U5y%5Fk%3}RnMNUF25+P7DjbhI2+0@gU?CtErlm5GNyTtsry=T*% z8&YEKop<$aoO|cpb9=XLzVr6G9K3#q!gtTLAJ*Tsy}Pql%YXdA*1mp6@3zjK zp7sqJ*Wa~aWBVN&?`&5gYd7B6b64{E&hDOjHs5>iwv9HvN#dS)^A~lsU)sK*cW3v; z^_+>ytgEwUWBZ1lZ3x(W=N;E>yQ{Nz(KQRMzx3K0+IzQmw%>kNa(id5o!yGg-p*_J z)w0{xUei9mz3Z-2`-aWiDgX9)3)|=6)x35)I;d%HL_KA_J_gn3+0>cbxZ&edk>tjn zjmf)FBeYU-j`rJ=o!d9c*73gRoNK&6H(TGiZQC8m&hAaz z>AI75_1;B2p3(Zd?(FI9ytB9QV<5ikJG-}UOs@Y}G&7ow>*8%oeyN5|Zb0{ok3k#i z>Xj=FRx!~VlFGGI44EaB8SvX$wtJ>Q)-`R_$ zeCM|I#&)3fJ2rH-_iUfnm}>7`k0ozoQ~Mn|I+Gicotw9{_hPPVUw_xO_O6Xsy<%aE z`R}gv8#lCLsokF3*t-K;f(`9mJKOoaaA!U*yUcfh5M)mrmH6K)*>1JD9^FW7AG9@;+7^=W=T z!Shy#KZd#^APUC4|4Y30<4ZS(CrZ6Tql!_5!_V~oqasQ9EKp{oJ!<3k1Zd-)c69FFh2kE?)Qm5(M#=d%~@1FRX+ zImo|bxT@4EdZr8Y>7%~|dHv~ht+2-W)3p{q3PJB@n_1XM^DGGUkKnY4;PDkQn9@N< z$QM1+MdhRi(zV`VtUV}p3WdbLb$LBP??eTcU#i}1Qh#;1x~=Fn<^Rhr?`2h_8mRF@eLbq-+brB z_}bRi>uzel<(f4&t!%xa{hD=aueo8xH7mrN*7&9IIi1P%o8mp&+vhc2))`09L@Vjt zU}y5sx>bB0?}H7Nz&R+^{{;&z?p37Uqi5=Q`kCVJbCHZs&)KN|g0}}Di^%V?pe%1H zQ6xmEC=U96S}uYLUm{`#o+c~CSB;mW^295!&iIAG*AK@0`_KXcOOiARm_Tyh0+)h{Xr z4IB{<8Q2ng3>+3|qkcRD4Ey|=zaf$K3m3*%Vc>vB``qCe8Vw9Rw{VAZBT{~?L6#Zc zQj9qe3Lg%71TIMn|8)f-&>rvw=M{@s4C5N^4%{bSLpHb@&!R!=s(jRUe|=q;1HsxV z-w>vbVKhodE{wczWQ$=sYb?YZLU*$eY21Ujhj7PQ&{uJbm55I_EMOh@klGOL!?=&) zZtTFYggf4ebhyPfjAwmF3nP9UcO&j0+$V91T}Xp_@B!SvgLt@yaf>e^58OkzkADea zFuZj?4tl-?dT<}ceH`~N?%1CrPuy*|2XPPKK8pJ|?qS^GX~e-D$32AmDDLC9hjG(a zrDC|_xEpZ~<4*r2@;ZY!xW(%T$L(L07Qw|Gg@X|8F}$w*JV~)7h7@j8_^`r@6h5kO zLgC{Iw6>d~`Na2LS2Nb6D2jzEI;SPnLR=8W?qY9@LKBjOY zEYlxXxJ}`g74A^@q{7__pHVoi@UX&z3ZGMWNMUimZ12MgTM8dlIHvG%g%>G&QsIQc z!wRPq?zSZTX@%1Y7b~2I$ndzrZ3?$2+@Wxr!rcn*QFvJ4L4^gDk}SUprxk8gcu?VW z3J)pVp>P~y1f}1kaHGQA3bz%>@O9u{6#e5UtchG6un+jS!W}BSZL$n+RN?U{?({`s zc&a-+GB0+A$HZ}kYdL*PoHW9V#NiS%{ZWM_eK9dO%^e;S!wT#0Ml2E?`eP!Yundoi z*mQSzRKyk5=?@z?DiSz2LHX(Q$33{C%nXmkX>p?CkBK%99tNiTikY89;%K=;Pt+F^ zu?mT)e^7f2Ryz4pdkiT|{HO5a3csvyyjsSm_M-Tr#>~G>VVVClF*MVie^eY-nA!*5 za}pi3GF-+7Gwa>)OGTr?I{u)-I)3A9Gk%A{I{s0Gb^OC~-SN@54HC=vG10BCj2{(8 z8{P5q#IV9LepGbNli@P{G!eVh9luz_6^<+Zbto*`2mDu9mlt2)&MztsdvK!33_tF{ z9ShCyVGmAUW`@TWnOIyYF^yL&A1fuM@rva`VH%&JzNkpAmf-?v0cX3;f)IG z{EjANIE|-7|L`t}X*^|lPfJYWC)NLWzr-}YQuwgKG=5V0p$BAmtqM;(D6!7}=%W(r z{EsWF^Y8e5Gyjtc>-^JL15y2T{=*8(^yt4|G1JGMkXVN|0%HgkGQTm=?ZGEieBvL~ zM|@SLC;m}=8x^MdbA1)2{3$&BH5p&0Kd!J&e^TLfoF4L&k?}XF^n(iP{7)*Z^N$^r z@jF!c#}7%|tPzm|AVjUPu}Hq#F)tkVzuy$ml_=}#(5^hJGnBK8k5 zT*p88j}q(hV*g}@AO7DG*Q)%66{h-8dw2ZUj6ZlvVqN}mg=P6sAx^vFPZ6*jnZ-Vh>$^m4rcc~@A@zhmO02e-W` zPSa854sF zH>z;)Z!-L4g}W8j=}#)G)3^Op#@FFT71rU6KQqIJ6xQLfcg*m#!a97|6W;OfGQJK! zuCOjYan1}stgxJ4QU6gHuK6>lu+C4sYo_n^;FBKQ_8(^aqaNJ&o*6#m!Lj$v@HB80 zokMRmV?2_=CVY{$b!8U>!d$aM0KlJ_ziNk3pHS zj^7PT^t7n*PAWX8aGPI-H!6G-nCQ{*8|fGxu#P_jds0t*g?0Qm97rht4i!GAuud=Vu?`B?>AMxy=}!V+<3_Xin*M_@KhID!nL>;oS;%E3D(6R9KhahU1x(pDzEX!a9Cqp$ymg4=Jqk zj}@8pq!qU7udt@S19lK9Pp1zonlFN-d7`@zGKI0cgU0h%9%&RI+42s~pcaBIup}M~ zMlF$#=X3zi3)TCr$a6dHJ17O_2VC^`=j!kK>hF8_yFf-^8X}0Jb@IpZ%ojh{NuvXWz+*>&@qyTfQX;1vQM(hvx#C2FXx{O(c#YDFbgnpCAWC*J5TYD^D=2@o z1zme^cM~S=TD!P*&HH{PXvPBrYI& z%h0})WO{1fH(JM)|7p;>f$~@N`SASSu&6(cjqR55k_-l!9umV>3apZ(-*02kUzG}l z;s^qN!w44-?Hn5mV!}Y2Tye(7*kA16Jv#;D>GOVz>hnrKwAlP4{S#>r$whg5p);Ul5-qQT#hmAofJWn7@=o#uBgkApa|5Y?ANO`Qp$$kk6ms zndB4wD=#Sm)Z#49$p6qi@i8`dQFsN~t53B@0p#a}22q__v4YAAQvILb2l?3sM3*BE zamM%$@-~=?B6>`Wor3(c{N+Y-5v1ks%c&@4AO{boVxIVgF*AOQhbo<(Rg>Tn{Bw*M@xYCp*ThXe}l`npRI&NZhD!gd2qJ}r#v`q;-KZhF$Wv{8SUL1S)nTLw4cMz2DINVz|XrSKZCTm<@!l} z2ICq_eg^k=@Sq1j>cK-Ee87Vbd+^g9eAI)FIe0vO)_n+nE>6BlUehYaUz)rg*YqoS zeOY5AuO~hDj0X>U@Hr0_SbVwUGirHo%!3!1I7o{pSNdSe!Q=TIAIEQpU%w=IOzZNM zJksJ((Knu+MIL%Wh^j8kw?=znKErs2so-M9yI-JxZBgS@T+^@GzeVAC+5SOVth)KX z$HC+48~4=LEw8^M{txQ%75`~5{(=0j8b^=K|6{)-dKz{4ik@{EEBi^ueWGD zXYqniujeoyOlU{7<#0`)k~i3a_U1GIlQkS`R*y6hT^Ab;qz? z2OdmKu}EBX{J~U#MdIw>utm+GaZ}14mir*N{=Yw&!}%dgUe6AuLKZbA$47r`N(Mu+ zd|I!xik4(>3=0Ho#C{H3K$zEiq7d`}H>GldyC|I3HwB_F72^HH*s*;aemS5mocAA~hxQj~&i{0QRhZ1l0T!G;_<{9U$X<^j zKdUE#r5Y}%3Psfn&=-lI{#ZSuJ};Ds5W;DHN9S*7zfqOU4H6#nBOL4hnaSJ;t$!?v zkNlfbRt`0vuHVdL4m%}k`iVZYPg;jpfq%p=i}C|KppW_0-SN|a_i>H%Eqy z_yZ~E$kiK)|GAykm?h95XuXg9rau*iY~Z5(4#Ma4hM0ccpY|uCdDOmg|K{nx$0@() zSd#?9QGKGtfNDQhl(a0C*Q3;4Wt5(=lRiO-amn=0Cu39`F5W+5KSI>0 z3;ZccMx6Ese?FfgX~pGxI6(WuXM0$jeLBtZ=3r|@S{&ylz(@e*vi3)spnQ#dFVT}BmcT!`~CF}2}{X29)-2N{Xh!r~_Z`?tS^8UHm=7$rX&MtjW z{PU0RNK2Ux7l=8Y@tyiB_Yc*+*Y73T=%VuWLB4{9ypn!{#(%vZEJ`(tGGLZB=s%iL zt3+ClAB?X>pc+rme$bcj@XGCd0{iv7hxnW&m-hzuCwM-)``;evFRl31zcb>VkFdUq zNOmC&^`ArcM8{acL-JUY4}C@?!t{|o1N-0a?ZDr4=zoRy3wgnMEK1#9&L7lYf@2gv zG70=T)+!2#z^48 zlt1XD`GxbN`31Nz6^4cvEPEd1AwTdN5;Qi}y{w!2XA|~YyAZC&-$LMaif_+9!Js|= zAit)8FwYkpAM{iqzC9liy{bH(?-1US%HwdJpAg=Ja4O$U-!~zTh6>Cd1w>y@M{9}bGwwh$oBeY6@EL9Um@bZI48&Z zmZZhw73kynwrnCseM0k&?>PKA+kpHFjr^NZd3o;jj91%FE_?gV@OML73A-JTdvThRYVo>`s{Ue#9+BK~uGy|@edsUrH0j2J5h{}BG=)I_Jh zQv6iRqWp7tebGw&Ee83K{SEa6Ka1@CO!Gx*l1F|fg?Yra&m@u`q_6Iq#PTr7 zk%xlGD9@3HqDhWCkbF#>MB_ip1L$ksvB_z_;WD&eeo>r~EBP2q5kE@xMgC2wue<&4PsO-B5QXLG{$w7vH)G`YNHP-P`R)qI z&;7}m)c-2}f`1{Ke<<(EsX~X}z(uK;hkvm!<>&OjLgru0=HC?NU(Cfn9BIQP=|_K> zV)CzWikp9s&&l~_e<||xmzdqY++S)Pesh1xclDPU7|#n~VKVuhN8~vCE~N2G^Sdw^ z3Nd|)KriZlf2zpC-@-!bPq?_fh`&VuV>W+_aN1wVFZl13U(^rsuKM2@9uM$Q5!L_B zc;uJsm*=TpkyoBzkuvLNk8jVFjgzPG_4@_nhwJB(7sw~ouSnNVj(1M`y~XwOwBNAx z66ojof!ieRW@u`=c|z6%^3=L-Vr`^PkxtBdq@}p!to*vnQPP;qfeJ z#VPqm9M8tfU)00@@#7cMXZNqDGk&rB zzhd;P{S~9(qL9+TnCk#lmr#rwyPkbdv5VEqt{?`N;Z z{{0nKeq_I%XzV`+`if96Up$`n?+mYq6QuF8%ZKv>YX4t@ci4Yoy)e=jrK^7>BOKL`49CI40Wfj(j^U#wrje24;RJ%iU655KuXOq1b${oz^I=lddJqRtQH)Baj2WN3aY@O$^uyniBl7Uj1A`wQ%!Ilir*wLJS{%W6md z&i>f4?ENw1xeDtIB3|n^TNLK~3if+deG!Ysd%3@)_5RBa{n+2qeoQ0~fi7Negd?<{ z;{1RQ-BaPT*CyJZ1jD3)(fi+lzGx^i z)%BzOO)%I;^*yTBmxb0K$kF^JeI@0m*OLQ%6S2YMi}^c{nr`;@RGlDZ^M&%6|CFJd z|7pd4^!I^Otr349RchkCdZRwiCnvkwqk-Fl_hat%Abo;rkLJA_RR5IwdGIeH>67&> zNrid+#rj~-Oa0&EUyVy1h@VvzRAJ4Z=aVensGii<4v;=J!L^6{H#qtvvTtzy?)E98 z^>LlOUpn1N<4t`TlI!v2%c)DI+Zg(7WBkB)-@M~Gm%LDZ(@LHEsK4M><|#kwKXSgB zMeE=0Wh1$Sk>50-$B&A9*ZL=&N8yho>+9|L3gg*e@vOT4pyj>lkIg$;obunH@=NkC z0*En7?zd^Yb@xZ?N2i(n*OaF(r%FnweO&c7_<{P(vg-$W9!bSY)cC>mBYyIFa*-?_ z@)woPix30wANGSLh?|?^1%G1{RqoL-7Hfc z-m`v4)RMjvNn*_C@{BAH4G5^dHcJ_Mhc$f5?BW zyS`|jQnx&K+aLP@)jyzrlJdagJ<9_y>BsH-us$>7V>a~X)~s4LKaii(-iVL(aO4;H z-JhJ}v@h^^oq>7d}v4Y90CI_Aup#jrPzu>J;*8hQDsIRm~(s~H> z#d?s>PhkDQ_C<`JZ~|59bDj^izSLPC+4@oH@5K?%_;%8v5A&h9K9uW^IwD7{KcN5g z?1$XzMdDYUF+O1ajJoGf@A{PNn`vYpQuY(bm$SYBKaBBF%CE6rGR7y=|I4Xd&-@!@ ze(LpwDG$6pcgJv?YdyyK74UXY*t$znquaNZ#5UZ0*vzv|IvPM>b?H$dNc&8?Ddh>@9Ob^%j-7c zL%-zkPtJ!Cd%Ol7&yPsN9*@CKjK2myP`<-ogd6mqm!3^tdf5K#+7JFG(_``z{P{1Y zXWg$Df7*V<=xO;CqbKnzM$e*)q$k%ie?>k0J$E|k`?b9l{oUDr$o`&dkEiGD?~MjM zc{rbj!*23CCeI&Nh&biX_EysWU2#PRVbcF$d~fQz@(Q&+;{APdWPwACYiZp_izBlOaTWg$;E=sjE^Lk0?8wXM`*ZkH1dLoe-lwODT z6&m_DvJaAkc9QkMX9hHPu`RIy@6DoD}P$w-84p%U9&pB6H2vkQbDZ( z{uHII3$y+_CDVse%Z%{CzE(^yD4)(ZA$};8Fv3Hrs|;L~y6!rfuk`xMpSoPqtIoIj zQ%gK}u?H{m;LALCp$9j4@B#xjCEKoSV+v*cic&Y;MDn5e+0=L4b$0usez8Ip$xEJV2`W*OXebR<2)A9>)*fa5I&!&i7TNuFrAv z2l?H61BKK1B*Zt)_aXnLzU!~I>BIb4l3K^-2ZO3T$v(11rswn2O{rVgwG$?JNBWlJ zteaV0DSxD&nY#5>)BbVmtt211f3zT+u(p3RrEa;^rEd>b_1$`_Z4W_y(3jsr!pQn^ zIxoF<0e!*>m()L(r0Tf;-(sI1t7X~cm(0KVn@$qL=)51Wzi+l^=VORjL*}J z9Zk#nq$(YG#D0m|mv)1?ef+5kC!Fe=DmO5%@5o_->?_ch_)~Esyd_!9=kHm5IDMHB z?oZ9|;OQP*YT&9=b#&xA@CV@Yz(LO6 zssB_XzCSg^gC`pp`O&9=Mfbv9AIS@Fkn>;m4(CsJ7KP7;KLPV&l9ApId#y=7_M1k1 zF#Z}CzFGz@@Zfw8p5Vc-uNdibUAPtPsm61gpK#DM;#(dJe*q&r$Ad#295itAjHCCHw1JJC#4Lz{hGKZ*Rd~jqw9R7BKcl-t*FTBk?(n zt0d<37Z&DA<`4ZG7tRA)+~0V<PUp|v@e2w}oS$#vi4#p+RAl1GlO@JyRA7&@QaIlx=~eVh zovPx8Bz;q+n7E|Gj9*-A;!7?u<4>Dr;_1`P_|Uf*<(HM2@n_61ae28JKOQ%6Wu+Ov zqQb^6V*bfI%hCn@3;O;*Q?kjCN3;jzZLOT&Aa8R1YNE#N?qYcgV*S`e_M7f5Ii577 zW;VIym%@iRT#skfO>TLHd^XX1+Yvmixb$4<}i1s5(DE!$1ad&bt`(rF7 zjP=vq$wln1xX8wr<&i(ZWj0`Ar61aH-s1%JQw`%q6mXA=@`uvq+YwarHu(zs5^0 zm-J~|wLs>t@#4jjK8g44nmJRHL3dB*H87uB$%U6m`gHuJCP|OP7%y(5@v*xD{(j)k z-d!~{c6yv&+PkZ|+QtV6S5?{gbUx%^=8~CqeanCsEUvLJ$=kw(a(#gMA8Q3Y)eB^N zP0xavZh9igCLUk7eU9x@<-3P<{8_QoIvV0|V1 z+t^=D`-}Sb&}f+^{kfp^(w^O-wMy((=b3#%`kPXVAb+H%^CfaV!XM=SP|sM~Oo~qt zfc>c|toO#$`kePKd3OH-y%hhpm=oWd|L1ltfIbA_dC0#uANJZsS6CFEF!%-ixy2tV zbk2XlwNdR)9>@1$Qepv>-@T0F2j}xK|47X9CE-|KHC8QU7kL`5L}oa z!>Jm@AXz`ezf8uL@(H}e&L8zFN-n4-`GG!(`S0YH=JrJVMeg#8l1(+#U`~9yJhF!c zi;@egNxqzLO`o0L&khKdv9@i~hs{VdsB?uaVX# zvc5&h#WgfOaC)xql4=?c9DI2djVC()<{d-U9|QAx|E*T+&z1D3_5PbXg_s>DXXz@) zlTX;+yCI2!y^-5Hh4$t4ZeaV2T8}Y5AHjMUl1b&Ueu&N^xUsZPDEUBsutz!cj3B@J zZW|NNA_b8J`O0Pa!FuPoln-FY4`Hr9)-SbePtyCxN3^|0ozJn7H4A7x$1FsDu##20 zzT;QG$Pe;oDM|6ZA_3f_hwjA_*0imhaD zp}V}5@Yi>Tqddr)Q{Kwtq)B%8D+R?XlT)VH80#m{f0@jW$|rh)LA!i}BR}*Pn;-E1 zpCr$N691U~kV8M@4f!p0=ZEx{yVI|Pzc!U0)b;6^BBlUyd!YQ4(8t>0OmA_qjj8@U zQw6UdsC`me(utEX7%k5qo3|8h6|9)8gNzn1LDJieoR zt7n=#f2a7d68>>EJ;aZmO9ZcHC>-*(GMS%mhjaZ(OKr^cD-k7#%KX6iwlXZy|c%|y?u#P3+ll|T4f&7Zn++M}q*4o7`fCgX7%Grdu% z&rc;8{7Ps)5#s!azcoy%rWffaPeOQ5g|ALdnvL-}jaP!kLuzl*C+hgCle6bzJWJCn znm_!>`B)3$Qu>iqs{h~(^*_t%DZ$177q_PntCF*4Q+;vC`7n8@;Pn-ye}&4MDecu1 z{{W`ho_tzK$n<9@ynu_6^yH(yIdd`orzwhnf1TBnhp7;k;?L^j+(wM&X?jKZuTEN3 z#1Fn`f6+5R+5;$lMLy_jAxyj%=~ll*W&(18L&JT$@9mmysMItT(%e6{EXJH ze+0*GMSFw`s6H~keJFpTJYP@gA@5fuEuL_S?ex>4BoBO{Jmf!<)@u>FJ!c4B@2m1x zCFjm{^Y^M`l>OPl9KS5MI+@Swah;#lQ^xBZ<`>ozR!;?foeGy-pWKPGK4*HdzOs5M z1@9Lq9P3T12jBQN;^&mHy@bl+_9&F+*(pBfUy9#H!e!?luB7su{HF^#-^1aYe_X^h z=KALrQhB*{eTsPhKvAUpl=G(|Iek^WSdkLY&v*F60~8(ty^R9XwX6J`=|oQs=hp!K z%uxF$ykz;GC+6w+m>ANmy z(-#JPv&C#3j{H_9i;B2C=pFDQJLv&e8aLLP_IiNcRZ#z!Bj(UE@wbb{pG0E=U{#(ahHRwkFtU50S(gWTWe@^61}dghO7{6zopCmXQWbMjxA zY?<8xNHPTa{r==^X}`0}pM$+0E`?VnubOlfAonqpr`Eri?eXu80w1q;iuHPj*6;6$ zS7?E!&ToB~J)kLh>n*oZc}jmi&K}U`vNxc;#@PeMz(}9t*7pOiqxyF*?Bo0>{AR+-GRkFGEZ=93?Tgd8rZKL`ySZ?l6lfzMeR>GCC?X0d1%G@-pb?gUdadK zh0lk%WvkUeBX&;D6Y}*4YiCKS3`bgRS(*ESp_hG)V_2X%!PxI>gy!t?`@6++M zK2T$=@6%Z812xwAKB3Cf`aq4Vt9d;v($E*~1ASFh4o0U-&73JQJ{yMqVx?+oOsw^V zIzO#X)VQwB%&)fA#5iwF>v1gUWd8N_CidzZz52*GbIkl^&o*&GgPGslxh8h$E4hAN zedfG*W`2#0CcgAiGr#%sZA|uo*N$oVWcvWlPdM_4@|s}p_j3%$C-!?dKkN9PL0+Lh z2%p1jpzQ_tK8O51jWeEwiFw+7K=Ix70>ZXF9^*gcGqi@{-+nL21C58FwTx|hL0E=M zdjXZ_?jMLBmhstsfc_H>%k`(!e^@CkUuon|@~-8NFv*XW4~@0_Xsq?+TK}%~a!2^vHOpzpDo`C+w!ll z-G5kLSr{U%OgLEqW{{eM2&EAk0reRy|r0`FfZ za5&*SwpZlYIG5Mkxi*fny&`Jk2(OhW?E2rAU5j`t$SM^^f3tJaT;k zejY3E$?_fjKJ+vE{Q*tCm5i`|n#QQl4YXlneKp2=Nv}%(IN1mC^Fu@)(TndPlK*TO zP`+&+aFzdDEA;;ruKIGvNVeSm70HtYjI+ddG``xnIF{mVFe z+WG7SVZ&ZP=R?Mny`X{4JB_m!L@sDA$aCxk*iV0MXUwq|fPWXX7o28$0nS&Dy@1x| z4K%;v`~_j#K7jKN+MdAkFV060=JOyFALlO!)B3L!_5+@-!7$dZK%btY7>HMFPvGSz zh4X$F>2aQd(#!KBz&KAq*tS>Tya!=PPf;?D{hwIBj`61`IU$GS(b2!>%kP`n_6JLs z@0aD<^eJr9NBSVpYtyGN`VR|0TGQ*X2NYN|J~-ve?{zu!P0S&CrxRX;?=`t`VUWHD zz~LD0ns=PBz6>n)1N&evsAU8acs=h=20i=hkg>n^ku3OShg zd&tG#2BJ@%?{Vmp=YJG^WG`q#RuBS`hhp?k!(LAI19v#aw|q%2(a-fMaMO?S&|e&V zJFq;@WS3V}W#l)`9#B)`3bzdV3(7V{aySZ`$hyGgrv)+Pk&#H=>a_2 ze(3uaF2n={=Hnvy=rQ%XtCLHXV0@6_%)cAruJXWdmp=D2(OXgB3J1LrmwllX;qm#d zaHOxNpG0->zp>F34*heG_0Lj&P404 z@CW*8m;QJk>OV{HdX@Nx^^4Wx(zl)={4|GX-h(=gYLPxQF-olK8u4}*TcJZ{HtKzx^diu3pCzb&@ESge0;h5mVx zW50QY>gUpjb9pz28<4+Kp9tG0q&kUybw?c6!i*`Ds4s({mnaGap*Zq`;(ySd-uHF z#`mGH|7pYd3ijVPZumc8J9$FQ(>G<`?!;{Cf^|cuJkOr!rB0`kqr7ym!J)|0nVPYwBxhQT*WZK=7~A{E;{nSxNp>O5YFvswEIQqNRL92BIQl$i}@Xt#36NQrAmeef%o`}m|%h(@j|E!z1zwIM^`*k$F=<`B;d_SG{LwY<H zt-StVUVpK;H@|WIR-C`f--`A1w!c-mk$${?o68>#^6kg`;HKXXe_9iJ{hPf0Pj3Gx z%+L6Kpqbw5U*-0{qVm1|SziAwuYZ@<|I6zi=Jg-*`j?shXU#i`tuLUxC)(Y~+!g?j4?5|j_W#Nytrkavdf8p`#;{CCtJURYY5%`kMPu#1 zBVcc8K>6B#hcMBj{bAC|KBN6%G`9WIRDA8q`8RpYj&!krf$|Fqs@$TE%p3447GMw$H6kaXE*?vmlRWh9YuPjG?)3C3&<(DwEpSFK$ ztnHr?6TP;*7y4cJKh~J`SzEqU``YrYur1#T+w#rWsW1Cq(fn2=>towfRsFR+Rrd$o zKeRnqV{H%CSlfd&w)=y}o~h-{?hhV&qYl^of%=ELKM*EH_| zc~#i#zN> zgp?QfJ1yY$k{IL1^|adk&-2H+%;S$W^Mn1do+E!OxVw}odubK)De$*o`wy8cs6RvB z+~ls0+aC-2A^4NQf6TGhMv^tG|CRDe{$tgSzu55xoX5OE=BD%y;E(u2;*b6NdyPi; zyxuBIFKOPc6Ft|!el$ce{IF*M`;&^>1$f7}N$L{`WB!8w?}D`6zhV5w?=eV!G@6v) zF9ZMf1!+9${T=9;IfVGyUx)l{1#hUZ{(6~y@9d20d-9&&^F#SdK_6d4ALdJ`|5}Du z*rj3rr~T7vlzs*NlM~7is{JQRQdMkku2$h*`)?8EA4He_YbejFPbx}%f|iTvDrKKG z{K>F?g#Xy;nzZ9ThWa}DWwa-Se`3%H_tMY)Wt86SUq`CIItqGC|(pP>Ij zdD;7yfgZ>I3$*Pl%Jqrv`*^CtfV{$$FZ&9;BTUx)0^+W*VtUxx8_ zyg!*)Kd1g_Re$YY27Z7)7w=DIS^Y>V_baW?Psr67M26*+^qaFgw0-si{K@v{_HDrY zGGPMgA7FoG{Sfb$=?7YAzr2C;m#*^8?=NQfhrxab`Ecx;S@???{$Uuu4F56Y_aXjb z)sDXv`-@dM{$gk!uRj;^GyJ!-KUfuYaP~iv{$LIcvcH*wtJr^pG5NnFa}58p%HFVx z9`+YoxZcbu@G1BE69Jt?SE7TdZfRXL$7H+TnT@V+hlSjZ_(%6IISkY~fc3-yCN)A2_`xaogJ z@lAiPRj_{&|8Ob#r9Yix|I3-i?MX4XeO>-vsLy!+G}FHg^w*f_O@B(hxQ4)@xVbIaoha{SL+{#{(Y z%bpDSV4sD(rQ79y<~qN?^$oK>n`8g=`m?#tH-J9mcV*DdZ@m2+{$dq$evB{Ve?fb1 z1@X&ePw%4fR{D!^A4UKCaDOrH_~rH&Ye0IhznD3GjrSM3cJ{R(p6igEzgY2jf3am) zvf!Wg7pv9l8R)wr|LFA>GwcyDd{2+obGIn_mcOscuy>HZlWA|MaWLcs_OF|7A?CCF zVV~vio9Tq3J^X#O1~%*u7=Qe*7rFC4pMRG-ocyIsd|`VK=1;c2bxVJ)JoqaF(?qYb z*ZBJ`&R&H4)=_<3{#?>OVm_}2*k6M6&FXxnzwhGgOa8t|p88Gp;3*zF)q{&YxWt2} zdGIA3TGv={+5RJhF@E8D6TJS!ksOJ&|A@x6|Az{<{W%o2{WBD{{Vf#M{wG>Ls{Ok(w*7CQ zzuX6UZ2O7QXKHC(~!nQtIMm}#Nz`+TM{9l=@q? zZ?k`_w2zQJ0{KCnrG2807jkP>#tmRwxCoSI^Yx&ez_gBqN-Cs4< z{Y_)t-!!(z8ztYm{}8`vyt2n16|Vb}4!6f66|Vb-4!6fA6|Vc6uCMN2x<0x;AwBe^ z@Ryo0iF**)BlZF3Pn~RI3ZF1d%AdAZ$jUyRRZy8Ol1U(Mw&#^)_v{$lK}=JFR~e>IoC82hXJ>iLVM zr99aFVj63IF^#pqn8w;)Ok?dYCh-URi|P2vM0E{#O`(pCfy~ z|KI*&IlrL)SlIC&!}?{szgP~BC(?e=0QuGa!z8cJmzE@0S_SD#q0a|@JoXD^k5Ky5 zLYyC^t*djsEBGPy42q4b05|4~Cit7#=cVvHYs^oU$DSem=Xifi_6*BPQ+isjqdh=h z$h2p`f9;oQ&j_)-BIg6`89@#Yf1o{sf1WiRQud00`&K&sIN0yd@2#RT>7WYt+B^I> z7C`$k<`>3KvR_!*zHvt4;05g&@PE_xj04L5P1`e0tMj}cWY4Hz{wn`BT7R?>zm@-+ zod1B8zZ>R%vL%l>Zv7ANE1tpBEzgvEvUmA!z!q z6+}sNv_Ba1ua5uJv;6%u$A1d+z`xS*mx8=O-#LNSD}2#>%l=J{{Q_9|>&f*N{8jk- zW-GCshP{RSg;>AK80!i6^XdAq{s!rlzaIF5^6))2K5rDH4LRt8zPg6i_l%*RE=uOJ zL#D0|>D%4mPX1};pP~PT{Df0dzFhTD{(+Ky@JIP?sr8EcdtDgcijppWJn)13wOr*R ze4=Ih=ONtGPgD82{*pcyJ*0mm|1sq+Ch0Ns(-mZoU8VfNPy@s-N*0Al_rd%CzYKpA z#4kz~T5f;9%HCD0NT06S1NmK{{K=Re*k@k0HX=W14=V6A|6MI3$mg$)zhH;9M@3A3 zLFmueKh?HJl@UKL|K3@iy!d6GWD}wdlveK+uzLxdyV1mX4t#Hf0Tbg|2Lz* zpnsshu>D5&7xHH#`C6s))98;Lf3+OLUyc0DT=Ho8(^ZhX&Yf$AbN?zWwK3a&y#8D> zX4v70SDI&d;Tv?;?AO`}=A85Z~ob#PqoI?PrKSuRoUedulHK zE6zXnQu1eFd(Cr1kEy>kz6bammG9Ema{1=>$-qDMuVVi*uD?s4you=Z+5=qQBjoa3 z_5jY`t8b68zZLtZasDoQ0rSsg50LzGzURjIyY%so6Mw_yanzsn?KnRGe?j)YlKOnC z|8^X(IuWkV`x^RtoF~Eh#5m8f8s8_ze4XYbkTCG6+};*&zms^z>n4F&m^B3lfRo&AN@Tzt>0JvW-k4{;crIz z61P7a%13=`<^C2WJWKo!N`Dl^f2+ss??&}6<=-!6dXS#y1AX2F@`m-w0^7e1>o>!G z&*i!N`Di`XW7@;XpYB>@9;X5@9-01eWG}v0|F%gk`Nj8p&*$IPq32uA`Hx=jd`sV( z$aB6of%=d>o+<{fzsJ!c!l*LR3XlW~4zOr0P3^j>tH zFFuM42eBSO_;0H481kd^I8Qd4%H#FQM@YYS*ma)kYSFy2)|DThuj2iSKA+XU(^?Uy z;>cgFJ0RPezOPL5K0)>Qsr6|Jr|+?yRq~g1>`V9_6@KWEEc~##7AHeVH&{Cc+gnNIF4D0u>^skcZLBs#=80=jt-1$AK z1v&QbQK3IK?>#KA*2g8+nGe3pTYdn?TO#tL3>5TI@Ml? zPvccBDnmKY_<`}U0_V9UMtxWH)yL&|UVO$D`^DxR-*M<+b@He2#9ZI4>Z_`9hkN(atNLcoc8445IWIqWymGBCc)oS5 zCwRWDalTi|;WM4@kDjLU6qD-h@gL_ORwbv@PZ8?Ee%RR0fgc!;>!rLP4(4Z^=WxdB z)2)c_I?n`pS0$&_+TX_|{wH(lX+z2vW2;v7N4!CMtxA^GNc|=7EBk`0k~3=T`HbU7 zYojtXjmLw~6GnaM{N1E9jlX1{ z^CADDDzxbgiIjE5$kEBp5*-M<^;c%{b|8b5a|a^w%=t9L%&??rY{ z;ph)tp8oLH-tmLZ15TVs;|KPq>U;9)_jujs13OfI;qn(a<>P$ddF6kI$|w1sf$+KG z<$uVLf1JOmZFKdILHM^YJ?i(?(SPc(en?Y}G#`2Wue|c_JwLyyucpS`Kk$3%3#ff? zemR%&Gx`^`ci-H(?)b*{d$~Wj*2A@Oy^j7Z&%3Dp)S$8IPc<5If0#7e?mv8<$92Aq z<$ua-yFcOg5YT^S+x_Qs!+H8w&TO-PT%dnJ9z6XE`Fs15x4*di7xfRLe*qi)3E10T zT>b0W29)pWUtB*|e`4(FKj0VluOZdHUJc9s1o@Hu2`W?f_lPiGn0SWlPu;pd;XHWc zQ-bV+uI~$HKOa3o{cn)ZTk7?K_q;cc|8Z(?^uNa+9w)EHe1!SL7|*d@Fy^Q8lxM6r zIQ$8N9`;x4(B-p!L;fC^I?oAxhy1;;P;WPj(r2Ah+JpH$Tqx zQ~FZr{}@TumCEs5+N(gX(w{;f3;qslo#~`MP2ZcVneO)IL44!;P*!qdot$6NSX^U% zPHw7^_AxBGFdvlrt4pDeOEVKd@4zSF|Db+J+sMa=|^4|aNg z68^E2nW}#zIbEG!ll6)F>)C&fe-!?eRxxln{LgiHkOwQdZn~WRkR|A`0=GHe|LFof zMbq;^zg-^eg=JKf_E(B`%`BbCIDz?fpBR``?d1Oo@?ZZc!QnIs!=5s*8tP|VKE|&~ ze;)td2G;@l7x2^hz83gzB|np2#`)tv&=;Ry%lZbc5BkHvC8bV&Xy5WRbu-p!dckk& z(oZ_ytD*WVn^E=|9ZvbJsj92u{1cSFuiRfi=QY#(ioz?ZYpU(=eR6))^Bd&Bx~bjy zeFz%QD?j1<9z+GwFPkw#$_wRRCg$}@^*i?;%paBW^Map5{w#yC{&Cq~q4PiKdt2*#_s{di%MA^qWJ~+h|BW}Qa&mxCWNR5GmY4PS`$C*e4qbu zoO4VG`FnrI(I3DME7>_cj`+Cndq8N<%O^SX)quZU(_?OVDW4yTcWQoOeKW8X>TX@1 zF7U@+S`g&y5{f@$KU3oi+OKj=a7sI2dWrJq^-A@&9)GMrmqRb)3G}M(rOEN9vM88C z`!y~D@`7;rdxYxyW%GJ-obr(Wz^us*eUPX4{3#(h|0Y;I=zB7pK=rrEWud7Z$Y0Zk z{FQ!Kj(66j1kZj{Xis(C6Bzupk{e3PDQ}Uc?-!y!loet;O7knK-)fWpmGg6osVa(oALvovbJOEv zX(ib|bbq&!>!+*rF6SS@@4@m#4MOEr#LG9h$}3+sy~r&O1D8PEuG_zY=)JwPk~6?~ zN$HCUE7rTx<7-B7c|IY5{97@eOm*b@Yv}LuCr&i#my6Fk>-@XWerw9goc1fj{G(P= zie9U7iilJBG(Oe)A~$3Xq& z^`gGGxIa*TsR)-)i^%rG`RqW@Ns>=$^<}`ZQjUegO27f8a^VO!)`y-G%8`rw2a<;jct} zeeBd2?@tch6CF#RQt2_@?cFsE-@BzFma@Eul46X;zV0Hd@6jH6cU^+-WqbpNsGHs`M2F;(qAQC>)n1(_2M$IO<3GHKsR5hesyb;a#L( zD9wX>%M1GN`+J0MV;s-|mysU74-~Zbzt8Oh{WzYZ{WUQk?cIgrIU0jM@L$QX_qWf{ z_juthtiw4z+=VqhP4O*k%+&S&wf8pQQC{br_cNoB7=$baStbI@Fo5jHj$#We9677t zfyg#-aH2RQQ5#xGAUR?!KoPP#v}!%ZN}^`zT6IZ{8T zclRG_{`wnS{v>?O z*s4`}KLp;dB>T7S{v=_f{VLzt==i^p{j-Dbd5>MsnEUg@p8C4%`;AR`)l_PF<}mrc zZAI4K*eloT@xElkSH-jE|0e(Px*PcSoAxC2nGr#T)F0gHUn%VyZY`fB(q5gr zQQC*t|Km+!O}0K`?)h`kTz&U6`~DSmx%Obg_qO={@fyE9$nO=(^8%y#{HEAfX42c$ z>hqOVetS@R9&dC0k{=BpPq^?>9`%ASwK(?DM2jexnDr3h&%f9-{~9<$c9|e06gFhI99;e_Z^*YxwQr z@7!x+(q5|?wfUIi^mF2_xJ%Z{_K@*W{Hu~#_dV)gu@-L0-`^i-31S%)bbsc9;=i?O zt2*_N^`Y4>nNWxLoBq^M@rXM0knv-2`lhXCUpgd@sJZz$9eZEcUC|MCultbimj1jd zTYq*vD8o}m(l-0CF`1v${WF2yPjB+Ga%HyuF6#R)l2YKA2gI6~jt|01bI*pC{5-bu zqvB}}u~+z{_-~Sr&)fJVd_28||ca#dg2GlOx+?fHeN*`bqYBYc~Ij*GL7*k&^N?>(^ZQ%YNeGbnGL3d{W<` z@=lICYF=nA;Z6L>p$}}c=Wi09$?uX}d5N3bOLC~{W*s_g`8_B0-~5=LUUKMzkJ=Sr zHa>Zs;(YB%U#AgjpIV>U`cIBjRr%$S9IC9b_n&I~T7ESj4yPZlu{AnMn@g7XL@8a_ec0zeYPu)?c1~+@{l!In4gs-P`Vl+qCz^_FPfZ+o}75%^z)s zzqEI`U*!WjeV908e31K7J|M>0-$*n5&5xwSZNixND(%`)VBua|3U%rm-|s3*&<&}7{lLf zuK(Nht(M0FG1vag{JUxB9(%up3!fa?RBp%Hl=R2v?EAj3Ki#0^k@%=)#T+so%YNrJ z+kZqc`AQg~mkrfF zq~qgU++H7)>j@u~251lQuaVz3(elLl>3u`hA6O-B6NmUOm-lxiUUchqam(|^kLY-2 z;t=yKk{^t5cEUEtHFz$+^Oek>6_>o-Om~+eZlDqVJvpr<~KPJC_qT|uxsHU&iOX6#- zJt9i_j|^3=&8-i$e>c?Y>4LG*jL&Q9?-RGN$+%ZN89J7q1)~I)}34KZ(Q|m`EG19{`UE+T=eQh{^MY(mwV5O1;&=ucZ^!&ElCb z{aNj=yE!TDy6pR?Z#MTUCA9sW(e}8)xa(|tOl&;3TsE-vu=RQSNL@n5!#W+Ggl|Z% zT%Hm4GL28_yJ2K`Lf4CPnx35RtVrK#;#=w3cf-idH*5Ii>M!9pj=1wpb<23Rap>k1 zx*ccIlkn1C%=_o&QW>)!y=>@hb+?M|&FwP5$@QF19|)G~*pj?0Q59TTlXtz7->+rf z2c{iF_xmnM{l3qR-ll%Psrf0dRr~CGv0C;E!}S*`f3Z*MPiw}ue^Vdf`dg8%G5JUT z4%L_N4e44_AJy6VsLs`g_=oCK*2l|+YL@C+%CsNx=la$5r|qLw%UkSgkoy@j;?C92 z(p>$>_;mY7t*OsiH{YyC+xB4W)$k+L)mk1)P5o$n)zp}=($ufkS9P^r|J(W(`)ZAS z+4{Pu%^zE-4TW_+^a$pZ2K1%qY%hx-GMkD>aJ{7HLXl9b?aSH7Y0 zllopiyu=hI>OWLJXA6~Yjp-jXu77MCsjSSk|9iAPsC+9;|1jf~E8pyRX3BR-jqWda zckZP!)430=UWd{?B!p@%JB8NUMcw&T~FjK4ASdjk)<`6xP~+wF2_`HFq| zeg<7TnfX%2V|APJf1MAFdr7$tmoasV{W5Q9xXzAG`mjl}T#4NJ}av_s`FyFX;}vuq@0+QVET zBlFok@_v5%ylr{yQZqm6eu_C?(L!7MfnbUJ9;#`8Df|4^x{(UApC)x6_dn}?t8PzZ z1j%52Gy6r!6*9kFTwJn!nEQLO_wTv;?XveDN&TgRGE;y0`yX1L;?|*Mut7#a{XLMy z$L#N&xcZa(V@-MDep$ZX=$Qi#wkIXLxxabmGPzzR+nXZHx%=l1%KK_eel>s6-d;>s znBNPjSROHMbH7pi)As&l>Hp8C@2Hr1+O$V=y-n}G-Kg>F{n6jPNA53n>sytieaihy zi&x86W8eQY_bPPms^!jbKe!sao?|#yaL(%Ai5}%n5|NGMD)}g5V{S>|bUF==O^ZV7gPUo3$`&q&ad_NUs7KC8Bec>rPLnR ze{}pdvhS&wK4r+EZ2IN|~3meaZdGT}!S^sh=zgl~~D~vfGO1szZI=94EUzz0>X?yhA=VEm)&$Z7* z(*EwxjaOp-!Svfh^`rG?+LMW2+pAljnf5OAIU>LRpnk?J;r;WozxtrK!@qa*@XsD` zkGl`+GVW ztS`sY)p5Q34Dmmo-n7Xdf1gjcwfXLY@){a1f02XfwI;LXKPfK_FWZ^{&C~mdSkK7U*DVyetxKOqYn0Sbuxea ze}>P^KZc)-=Xi~OvHAUJ%)jI5)f;s@Kz)s;TdI8bi|Lzcv-U`RyeRuAnjiT+Vl#h9 zd^Ntm+;6g4?UCQRHTxstZn19J|1jb0{9mczFUkBgl&ZYTOaJGGmdJYENP23I ztY2ikqVw59sc`wWr^4mCCFPgzczVeafBqV;TfNcTFYW46`g5vrt(%{8{Ag^c^8KG5 zy5IcXwt1dF(ii@rq~#?)5p}oMhuvFNdTtpH%n5?DaVf7z;rUB{k1qTCR<~DQ+P`(< z#T>4@{q|v=|B>xFfBqZr!mrl$Vd|rJ`xpuFhjQbutFOhY^;%4>y=+|Vw~z63YfIKG zE$R0XOH~k_zMk65Qi*{^7S5>5o;u{n}nOHOvKdWwt)Y>o%_T+sk-kYfIMc+RM8Q zpUJkD`@-%m8$7qP7ug^Zdvfi?tnaP6T-%Si!4&S$`sbpqe;T~{e2&Hk_q-U^r)GR; z@%ZZ+A6iS)cj$eUH#8oY`o2G}zQf}~rD-p@`o6#FeyZ;q8y~{;U2J^Ft}hx}TC?^0 zH@WdZ{Im6|epK@MC|&r(e-CjpO@nAl>FJ}GoF+2z+K;V>w$vvg`S`3 zcw)j|355@h2Xg&7|9L6*yi)f0sb$MdeP`X??_Gbh(0t>kw>l^_p6L2ki@ESjA{SD`cc}S8*ecGY&PegDK~$O%lV$)--S2x55}k5{3Ev& zNcu8grTp>d`Lx*?%|34?>(h#C|1jf`H-BjO8y$}tz4674C#g0WFK)?>KUKN$#9^(9L-P9_=KLP>U9Ww9Dc3*lk@0SMZarkr$CroC z$0xk>UubFFEgYqRU4<;K0- zxeun-gw~foFYiC{*E=t!*O~T-@XyQjovEj#KBnycfRuM`d_67aOEq-9B=0K;x6kEz zIWxO{9+&>+w|ANELhHX5(s!*5&zGx~=f>03(taD)hUd?EbG-!f=L_kEwf=l6>+R(g zVu10N^XpI7x$@Rd@~39}T)RA(n@?qay=#?mXWMUMec0Ww(sN6Di}~$M+}?a?)?40u z>5s4S7`*HcNq;xLmn6TBu|(||k5`7)hYvs5yg1vwbic@dUjg=q%y=vDnf92<@v=Yk zne-wzpUD-pzv`0qSz*#M^OxM;`~1+N`*lChtXItb)1pPI)vfD)8-GQ%{WfTOZ}-~o zYOno{n)@y5(-{dZ$OUMeRnXuh}yR{A9qOm*7on8 zA7Q;`*7M>o<7w{vPwHFxziH3#k2Xc^W`#>%?8}{x%J}xe&>gw;ow%zS1M|KD{C>%c zvVWoL{g~@da(;J5xV^5tBiugab^hM^uHHRuJvcjL&hMX>{^0kos)=&_qm$+PV)C@`*W+3 zg7`~%%bKDlzovc2{?W$9XjJ>F;l*G2#~l^2VzA@C-A}7FZtX9!1-sVl<|}XVC;ZAP z&tKeim7e?g^qiT0=cZ);nn>Qasq&+qn)8+h3&PUDusrUTp zJyQS6mSx*B-iHv{PdyR8Q>?Y?8|`2EVhwYk{U-wK@0tGfQiy*(EoY*F=lV~9{adVu zO!zaQ@HOc&@As_D{#uziAJP46vwuBlt#SL;?)QkKMM-CGBdOfLJZ^m-Z%X~c|MOM@f){^-;{4k zeD%nf^~8T4a<#8_0806+IH0MyL;6pR>^GVC?zPuTJ}u*wIU@hoyx;Kjesg)qYBIim zX~_NFNl+aR_JbA5F)D>2Hct%UjB~{XlEbsNTVpDW8WApnP(C@VDY!gy;M1Z2Oh4TC1|Z^iFBN z*+a_v(F1U4{4WQ0lmGm@mS?%VSVRstzUzFGEnmrxyk93)-uY?!ejRBa;rHvT*7xgJ ze;o$I|KZU4apZaT16P9vZ~Yk^iBo#zUV0MWlRfnP$>q=1-)k$+E0kWjoEoD%HN6;@=WA%6`u?U=&=hJf<$<}rn=Q|nT#63W-)l^7 zM_zil{AvBY-uX-CrT50>FY_j+H})o{H~uE4cO);pTz|LyA#c2{bK_?(kJsjXXt8qj z*YQmJ!{fE~hnc+ea_zH7dW|L0Q~#prUCm3cWd3AzG`*^znfiOl{56+I zPyLIg*P54J$^4x!k)HY&O|LO8JaW=Rj`}M&zaPmjT@^>*WJ$e5?D1Y^EW4)yIzv}#5%1f_gdA_Fnwb~z8b^C+5 zUU_5l*Or&w8=JpFdFj2e`5Vtm@4EY6Z;|mD{ZIV!$Lp(k>E-5kJ6`9lU(h~_rHB4k zFuhpNO5w&kT)vc0^f{H5~JD|vpDEHPfIf6?)J{KnIJP4oL`UV0_# z?=_X@@SB|8$eWzrn_ZrV-sJopev{KX`X;A$>`hMZ#G9PnNlNeSwExoJ?Z24yV8h#6 z^?k~6PgR4=M}H(cUD@|3%lZ9(3#@;_`ak?f{`-)1zo$1yh31Fbz4K|EkMh&IJ)`N# zHoB(wBBl4apuzYjtpCF=a(eCZgKm1*>zn#}n&I)fKzhk}-xu9*yZRR$ueVWp6_lR8 zpJmFkyuy^H+!w9$-+%J+C+B0o7g+y<^?&$3`T2WV%Ck1;4CT+?pD1!ZhVs<(Lggvx zeNOi4)L+X}{KMs`=YQ9N?ofJ*D1UmP%dLOydbeWH{N-u=iKqc7D~^1Ke$b= z_7;;rd;S{UkCgoB`@T*7tbhJ`sWuo0rRSIDJ!b!f@@LCS*YjGQKTGwe`=2+MUaox> zNpJr0jFqc@(eex^f7j99w`zZwr@yOz(f)om7z~x?b@cafO;5_Rfy>kECpX-#{+d7W z57(cLFBgIlN-tNQMaGwAmtNueYq(wgidf6{U81f z?(au}u~7c}_F1IAXUlUbwa=vyvp=6LPr3dR-k*;=e4r{g5=!s=)c@L~B0~MW?fodv z!u4nUi`HLTa5R+Ob@caa{tC92+FP>uE7V@5DLubD?S3oeFIGN(de*;a{?6y67jB;g zuCJtB`#i?^`=GtPqW+qm_=oFH=ZD!FPj9S5dg@;^y(>4K-u32B+EX}x+TWAGM5w*^ z^;e`nbh`W%ZlAJ$@A6kLy~e!sa`jiFy`VgcrHAq?nBIxJ^m6B8Mbhgokw5h>+CC@q z(#z%Vxx(|`_151pu0L&`+k<1Tw*Dl(@O>q zclE-0X#9*Xp1=OAf6@AD4^9*<&p(M2ynY)mu|J{ynqDv0-vg%qouTw{t8gz&Ux}j>HYmuz5X6G^;3L(X8nt%*B+du_L8eVn_hN566vA-8o0ff^5lD|4ve|hOWp!03CT>XpYZ!|Bxn^0eJKSrqjkiY(U+e@@u{fnlT z&P(sQ%d>F*D^Z^PdFhobPxObp@_gEs=e+%4DlffPpTB)Jy?NWqWL|nD^Ve9cy}10T zf6@B8mX}`1_A*!^J@qe|-fUiaZ)|zyr&qK*?fJXyUj@gPr}ccLME~m!rb~=3dE+Pg zUtas{vgys+|F-9)XWp-9?hmp3-Q01TyS{1rSHb+b^yY1!?Rn{yEYG6t)1^0W`#hPK z9=Dgg@+{guU3&Aj&trM%nL7f#@-%l~URQZu&-^fvmtM*8#QG(#y}0zSei8rt`KUEG zN9{9w{jb3M-chW-yY$q*Xn$|aORr@9iq@Y?Z{GT=52d$5?mx75Og9I5Kk@oubNjKq zA6D+q>^KmQU)BfW=e}(2hh#n~?{hGR-VbT`Yw=4SulteLP@Yrt`%6EBZd(`&|1Xrl zLJ2ICz(NTul)yp>{2-Npp1aAD7*nJ27)a)?H~;e2&o(^&=m&zS$V@cz*U$X*;OV(N zU;nXR_?f5Ye)%u{>HqxGC!hW=fBUW9{eRDX?dAXdg{LpB{WG(%j4caqc^FbF06htD@iizI>gV`C1odwSVZE`@Znc%_{v{bKlcHHmTqKpYQyK|MimtPj z_E*32t(X7#f#3hw$))Xck*|Gw?zfE{zdZQuj<3#r=kvWUKONirk^j;1^7p?rw`u>} z+}ZDa{hi-=`G3uQv`1rsKgS9*q;EfZgEB; z^3@#Te@O)UN@`KWeN_SH9YmJIDz64Hac(vX`^b`t%I_NYGm)6syrpVOMUIIC5z%r? zq)Z+c_1w>t>BEkTR+4-Yt-Z7w$Nid9O!&IGZjo=QsNpVNF!IkySzm{Y#y!e}vSt3?=fn=m)v z@26QbeDnjO8?FYu-STi(@XFlOT#$-LBea>*r_2iaXT(Dec_b~^{P0tin9D_;=*_N44>1 zkIoYQ@~9f+FC4zVg#Ta(|8xoeOuoPC9X1Q`68;JC57lzPI+Z@^9di3A=Q-prx4X{k zKUTtjyoCRe68;k<{EwILpDf{js)YYk3I8+V@0WRjzE-dn!aZv`FO{(W`4av!CHyay z@SiQ=f3<{vATPa{Z)Av<@K2QRZ_V=$w_@4Eqx8f6Y4P`KsjOg~gu`S?#3>h+LUBGD zP4OsOFqLr4Ci7AW`)|$j4_hhQiSy=sjI6w#oR61?Q?{T(IlrDbn@hymS;D{jRixQp zBF;hakG!u+)*Gqph!}bJN0-|*lbyXII|K309lB=PT$PIXfAJln-W9LZbwv$)S)hMbMnN^Ra5tyJ6fW7i>DM13a8>yR_D4%vL`H!GvJ{?!Iq zKm3|Ny84OA(lCWZM5l}*C?ov!);2MYb4udO`Ah$D`O>)cIId|*T|{p^BWsq#ox1YU zRf~T#nvtuxP2J*e@-|N#%^Jtr>WZsH)?jL9u6%TzR@l}pw{NMjJ%Ccm<`L>7S&Dre7`)cgf@};=T z{5XRGwuZ|q)CTg(L(4QAe@f8g~E)fU2@+3T}e#Gs>s!YaXxQ{-RAk`3dz!^ zD?$sCP@kZ-tF=J+3Z-wx(|P8J^Da+jOf6;SJoy|~J6!pje8gl3{B-QP^MScR_6l@w z;k{B=X5-0oo2vB6a&e)u^myGr@XEd{#JlYR-K#))=JLJgHWyeuH``-)j9d9_I=_8M zT+@=bb}w&mo4k4c#&*vQ+rS&!H0^NRXS7dDq0MW1&5c{nRAwr3{aVA?@P6A3k6+q$ zvs{6?HAZM5%i*_Cvn;Z*`9H6X#=~}M-^ZJ(i8()ivQBg1 z^79rBTOx1KVtES|h39|Gn{T`H?M0zFDUes#7~=G&-rtw0TGf#x(lWRc{BbBKP}kvn2cfw#>B_f2I|y-g$YZ!1l$pX>G~5QX40B^|$c7+X z_6y-~KKn+L%PY~;qABlOWBxw8xvD)DS)zLuI`x;;$S=Lfx)knb6Jd85xu0FC{<@~r zc(mW>C-hzCDm~W^wWDyG)ZI7}cGqJSTjUej^|;uRYw;RC;2)n#t8F(sFUZEK=>z#p3U5hG=;kZOInQ~$-8C#;-ns)Axl?{FOO{Qt z<8o6aqNY7HblyTG9ObLw%k;E$7}l3T)ty-)ym2(-d}uIee=?4`_b?Chz7?0>wnj9 zzFWWJW76vmD&QYVUvzbUC=z*JrPNkR@@NmvrPo5`?XS}6=V_t3$E>X$J?i!Bm4$ky zXPfDfMS9$n)}T4&>6K`7Z*Z?H6JyG}UR9Qfq&)XCWvNKUbH5@jamsT?ViI0Fp8J`Yg!kN=Whju*>^1hv*VgMIwFP^f^eC+_zrBTe`i=HNZTT9f>pj_coo?K& z_o26;r|{oUPjO+;i?$T$QGS@vckQ9Bx2X+CV>GdfS=2fl*LM^nwT;$s$ zHN7}0V{3Nw@n?Iv?ImN!O6ddPmBdtytRym+KQl6B74pX_Vm!QxF!9beM%ycJ#a9t} zH7z&3yfs!4dezXcnayRXycvAHUSh8H{R^xT^2RExGW;>e&G2UYk?*0gMaMelZ{qU& zv*VSHfAq(cS#^v5@hTZP9gGp;y`z41$avvvMef?kh(qx-4f5Apr_`lG#aW>$+e zh2|||d1zH{`fGNN~vNkZ|GkM)bzl8?Q(IMV<=&)%Nf8t?YNzsBc`iTjzAM)?HcYk39v?9^Cw| zeKo%?p*NaUiCev7HFir3H{90{hZ`N{8wqvv*U~W~0!mE(-r)K{YqV4!*R<%$SGm15 zRxa1ZbWARAZL9!MDjjo^3`2*b@>x|MYlsjl#4lT@}o;w$ePgYa73fbUe4cu2NvHV?O(}9Kva4Li=%@ z;-77oOA~e`k~BiGuCP3zqs z0bYqs8M*ZNAX{U8y#mA8&rUp0E5(-`6&g+qYaEK9?zbn{m^=;(T_QHX|!e*;$hA`%Gn_ zaY0ui&Bp(g#l`#^jen%5f1UL&8h^t0Z!Q`>Y5mKJr60HPFDmFi&-kTnTJvY~ABhwU zFaFN{@?!qBd}P-|x{QD5xT^8H{1pxF%CD%u&41zez4lP9^{?r<@+dgY=kohX!Tjdp zE6c{`@?TLXe;ywTx5sCV+naZy9R9E=AH7fCw7)$6L*lPvO|&&B_mORuc`T*V;R?Tu zlj6wE2W~tH^MUt+;X%;gcw%rOYocy6jPCc_eb6wQ$v#`6aFb#4Y}DHXvxhtxULYSq zC?72anW<1btp%A(D4sT9ytUaPbB6Eof&3I2!{HbD>&=s))vB3iWk<=Z$#l-O<2RT* zx4X7s%dyARV}<#d7BSZx`c8hcwlQ^`W%PE;(7|(!w&$z1|L3*$Ag1MRucPGdUD_Nq z_b$n`9KD7ETZ{cgo{IkMj+yI0TlBB@J%5Gm)r$P$X2w{i*3Eaf=8Wk9+7h?RN7&YC zx!TihUZ`-eT&@O36z;1IJ@AZ|4jyRj6 zJR;cXb8VqrkC&)`=o@=)nm%kV^kk{ipHm;TV#&{^*GPD zgI!M+zn>RtoACM~y#9#0Yvy4_CN%H-Bc=0YXg&?cVXpT#R#oKg?J8R&_Y73xJ~_;e zTf$6*&q$XwM=p+MW4^yk*6c6grI^jxh7@F;e_6TjpC5mj#GebFjepk`xd$;Yzh!o? zQmaw6G*V@9YIxPA@BH-&Zgi2}o;Z(xMN}?47xzCH(`Tl_{*hfx?cWVP71jL6)s;El zJtgO#;d%(yQ-@ir%V-g*t1vUJ_ddvZqJB14RmH;Pohy(0^3?l4Pns z;rhx*eVKVk+P$`Ycdk7xRcktZuI$USZ1UQ&$D8W#crA-8kFxZ58IPoWX3Or`P+vjW zdAy0jbNly7s=X$N7hkpX&86HHyzu50Kaao3=L^`Y-O0DN%I6E%Yc5da;x7|EI-fme zIL`6UF7f#S_Lx?lSQ~1_M*gr+?4swU`DCz5cw_lO;jZL207~#21zwzAY7YL7j0k>%i z9&g&c=SKd(n|AB@qdeem+MW2nrP`I=FDN#L$~@eLGNHXZT`!nChTBY<+SCtP8+uFC z3D-v{ccHO?!(0!;Vc4x}4%P~_A7*l6@w3Zh>`oMzd&2E6JNK1^cyFvP!atkvBO#;)HAjg#RzS(s;~4eA!p6PhD)e0UC)(cK*L#2=c212@Pp&+bBH=g(u&gq*X=sqEs^Vwr~v+rV+M+mG+y zvV+yR`0W1Be0+4-L3xcD(l5=+yjxX@_vh72sd-oSlh{5#{FH2Sv441c3D3*A-z5CP zyc|9Ue_hYn7sh-uuKYmlqi`R&QFozn>PF*W{(8@=a%22v8R1ja98Ye{ev6)Gr7Gq< zPsz)#x1Vp$qe_JL$A5DkbpyOPk1E0Y<8i6+T=$pr@-xrgf8)Yc_~JBedqPJOfIRPP?SLif;dz54guJrm31-E%h|U+6i*`S?hI`{?rcP`}SC?6>}a zod@dmR<5%fbr)I--e?>*$`@K!_-#K@9+U^}d`{1(^a4w%H2gJ4!M)Aw-fegd@`G}o z8CnA*C^DrocI3D7-z#d`rz= zI!F6?$SjiU_9gBI);;NbCOq!yo~ZWO@O+;smwnSIb0Jasie3xb+#8s83Xk?`*Bd&`qEA)N6x=29Nqz*e<(bzk9hv!_b$1;n4G`fTaZg1 z*Go2Ui&lo(d0ZIT`-rFX8i+kY{*?#ingsj{+-p;**DU7oU#!M1To>D>dRYv=j=b={_Qog3TaKHKW4NP#vI(fT(RqNNy6|B*z6Jgup<8;Q%UMpv~R z2bNGe{56-i~Ot=k$YkAkU$N~}m6rBm^UbfLV2Mu)MIMCmv>OU6+;mF|*pyn%Me zdQ(>hdi!u`JK`nB!c@t;jFpU|qhK7Fz*?$(ZrsHoJ3?=@yP@7vI~Z@Z@(r)x{T73E zX>zo1f3Q2#H+!nD^}d3I7X1cZQ(9=RL5iMLd9Cd=bT3))5d-toLJhp`HLx&ZEQ}a- zyH5W9;aOwUxpnKph_NtY$ea~^jxp^1LA&p8VMKldN95vbjQ)iY15dKP-mAQY7X1dc z=!F^Xb)VrD&Px8J=fs5(V`0SjAv`O=Zu-KQ{ko6Y3nRwDi19->Vl3=@zG2UJFN_!q zBgPNmh|##PkFl_i@fO_6Usw~r?rY-f+{YMP7%>(`jD-=yJ~tkI2j%Pij>@;x9dgC5 zl`Nc(AVGZCa>(|?h@t5M_CU#`$Yzw4ptWe#Q39Q8q6EP7>4)_rS`2)S1V@sL}7Oe7Y}F3a+D z!PPQ9{M&-rkb8M>sVwHje@ieEa@Pju%l!Cm4yHrlYl1T&|7F3HxO4UULI7yr^= zGUTofj)&sADVQjW{$nnDRd6Kaza)@fo$>NlslUGMxfciWt23TE9*l+Di-Jrje^QU) zABm#8%Y#hRcgypzA$OU)&ENOeuI;65^Bn@ZI9KNvG@CbC%)lm-&{ko>(h6GXQy z4a&O8BC+*#!MOa2&4her^w(@gBSAtw9rB6Y*`PPKEF2|DAlBFrnUo4iN{1YjHv|sK z!>i4=$OFW6qRD6clTtPENeqcDpGo-y`{nzve3wt~Df#=0;{SqZ^68Mzw0=Jl1l=!6 zSosY8hH$6FEuV~hCghX+Z=%bm`42@uBcJmU$2Ucj&!l{&<R0e<#}C3-dh* zw?dw*kk71qk}IXIjRb@88JEwLd}id6sF!DX_{FP7ikxUS~zkAt)G=6<1yHTF*(8txMoLir+ zRQHT?w>ozqPwp$9a_*#aZ*}gJb9Xp*w{v$ocgDH(*7oqNW))6T8y zF-5a_(v8o^tM4=RV`ynM#}fv~y25_j%`@a_$-Do^kF=&aLm#R{Lk2 zJL%k4ojc{+f$Rln{=1z!?%WyYPB{03bGJJ8lymD^NyE=Lw>~YU?hcuMb5A?>xO2}sx2_NkFK_YG_?w+OlHl z&Yh4kLiv8@Zg%cL=bo0yRQ)@p{3^x%+1sqUQQQ%6CstW^xAUKI?q=uTo%F-k1etYy z_?lqCxo!N3Ah_N4PXtNlw*C_S4&PtGJ9jpGqwk*xCY&4oQ+I{^XPn#m%R^~RzJF!V z?c508@lNZXJ@+=x}Z=fAybm?(=ZBT3+i*`N4OGw_to+mZiBq&`S&3G-9V4SxK=FhAqmHoq%_@rSHG_}TaQ{K{bP5kLM!kb1w* zR|bQR`g~O|?c6s0%3$IHKEE>P_>j-n1%u9=bmceg+_rwC{2%kfOMQPh>`rVCyMvE} z-LvGL`DoaG`eR}DRA<;d`6FR>qQ|;*ywds)p0sWque83C&Ydy-l|e_^`s?_m{!`Aa z(=p8!w;UfZXI9M zKlNwUjrfCaSvTTOIyd5N6?<+I_PyVJxk?O%Sa zQT_W}_>^<2{Tlz2b0hv_-16Pdzk7*w54!ME&JFuhRpImpog3*-IycfwR9hbDbuSC2 zH@n5UVQ=ES){XR&?baPP{a?ll=T13$XPtY@`42v9`3~nlC1TW6F$E@nEFZUM*JN=W!;E>*12u|DuTg`^+*1voZH%;2%2S`uI&fm6Q7oIKXKdn zc23r%5wq7|*BjE_KV#ile{v_r$e-MapQZT8o%k&IlRNQm$e-MaL*(z=cK#6if6n?h zn)V_2b8b7|iT}T~{>|`r?(@!_c#gt5cQ*Vu`8#*k|L4iyxwHPylfQFk{eOY{ojdD) znEahP>;F0HZ|7IZzjJ5H=NGL%_;GS4ULb#RCytOmxf3svKe-dXME>G#koMSW`j6Dl z=dC;A+~dxjg#S_7{=komTiY}E#1|rFPXzq9xOF_w`hPLZk2^Q`#4p?Qw<3M#9!L5U z*1sA4;x3o?!6$yj#t(j6+`6j>KJly8o~*yP)qc1W|IWJ8h~K%#kp409ckWL3|C;p& zKknS!@c+LkymM#6|2l;iw~T7$y%LoXsh{K4KaT$6+$qHW8!kNjox216zbShKnqTKX zal*P0zWH~o+m3H@!PM_rcbyC0@t>``-??Z0Aj}Vb*}9Qla3<_Oe%87hUHIfztb5S8 zC%+ozo6lP}(w}s0q~H8!VSduN5r6ZaQ~b`2{5AiD^+*0Eog4NyUkHaUYuymU>Z^i| zI@$A)eU};e#1e8+tf#9|uKJ0s(iUY8-QLn~-*LRLCec7d?2Uk+2iut>{rC2dt-VWC4I`d&uWL1gX0;{j#NOecYFHGU$f?zc;vJ_ zd$c+IJuY9^eV3)EIiTYzKs)Kq8Pric(2p14;eVrl@00*#W)umByx-^*VNqN^X7puw z`tL{^C^K)*3w!;c=+R(Vfc%a-?*=JX)lohfr&mcmlm#{6NW|4<^jq@uPSI-;IeRAl z(iYruF@23kqgCD`dUC=w63@Rw^mlrC()kS+iN_22s=lg~u4#qBr<`k-4t(C>nnPC6 zK2Rw9AG&ZKk-bL$;P5w#r;deuBxT!1kA4?G8QDfM1N8?<+oy(4kr^YJF*%ave|agC{M91i@b9z!gAU97IqJgWY8=X# z!^>8Io<8S=KkvdfWMi~nIQ$x0AcHC5t|9z#>!IoE_`>1GM35-*(of!H!zJyXY#Tv% zG^^(%V%1WP!=IE34>!_BIMn}B5_rDwX%~TpSB}H$IHE-$-79>&!#QuU1*GB4{1B#s zOX96@zL)eMWyAReX2xeV!aH=l((oF8=y2h?Ln1XY@jE8*X-Zp&L;4e5|0`5~gVHdi zinaa{eiRcATl@d!1*@)*Kbrp#B<=^K;9O2aUZ4KDRprz)bdWRHF5#aokpB4@OQQX2 z`g&0D1${1jicG8hh7I4~+OLM!A%VlUNcnuWK>DrUvEh>vMh{LO?dTB+e^A1wDE&?s z{=5lo!)y8ohxU7`2!+BoU$yC@{c8F;BysqSBFJnsDD5Xy{>NSTq=fMg8*5Jau!R3q zf&8Cw;Zc9mOilfB`r2=_Opg>C#bum!3F0rptNd!i=~x<|f9iMtaQ!uKfBp2-R`d%$ zi2uFYH4s0Xkz9ZDS^bXwg}(^5{lH+~gF{36hA6;;>A}8T!~2sPT9Si3Lp^)?hWm#0 zdra~t`up}Kcl8Vm?B4r{cPG2|^z7c-HQcv%|GuHFp5C58!5csM?&Nl(G=6+c)dS(Y60^ z4z*sad2;vO;a0KYsS^JCo*ecpESRVD_enha`##w)$;XGDD(h@uH9b!@GM1y7qadO1`AzTUwiv_i3Hw>dWK}A@&dV4Er(rp(FuQ z!6vu+2M2Z!tBt$%J-JufMP5pN(ew;UBO2blZ*SNB-Fu(#;@8s2#ku#%JzazS2lnrl z7N}8cjs=$DQZXm_uaUsmn!>*?*vQ*9ZNY@=lOsjfm%Jmso^qS(1-&-$+Y zVxXAdEN@DRxFJvM-903ot?QHPyLRs#eA1NalY;|&AOEpCyV2zy-MxQL&#?56-sG@! z*`(_G_9lCVKJnxpDKO(+*Q}~Nnv#8{2McfepvWKX8F&(%UE=E**!_vUS^s2j&u~xD z2A2Q5@W#Wyn$mmL-AK6yTk!GqMdFhTeSCxUE12H;`+|?( zJD>lCRt+b1n^sH3n{R*X`ujI%0C#xhXY;qAxy3|S%zu3ltdox|Y$=M~zH}0i$=`5% zrs*X=dAFtSHniLeXNOW((w^JpV4WD~2I(w=BnrhJiccy|*=%^kd9RJZ3vb$!WF>TX z{%2ypn+ZJsjk)l7@tgK2l`MzM61hMwh{sF)pgc_*H90k5kVlEd z7ZP@(uxPWR`cyxyr0##}$|mCyRz0PERr@~=$;Gvhuak~5F;E8|b7 zcXD}%|9vgN$6Ipag%@|up!wpz_x^h~NON4b-u4X7UT^%oH(PCp-^}U->?i*`8@%}K z_;YXbyEO83>unOAzbP-*jk7u6MFN}Pg?HmyHmLcMA3kT|y7l)4w64fTzixetUpLZg z)aH%y(XxA&d|Ksmf3R*tvwU)+jAxJO`D(JH`l{7tuc^)kH~%+y-<#ySH@pSM z^h8OI?Nzs>ej$1|_PL5*jK5Iz#p*BEnA^mw*8ujvew;82RNdB{xdU#n3Rejve zX@2KP}~0D4s7C(SJpZD8#QRqCZ_km%HO-|3vD=)ff6jw1}PmwMaY@Mfkl% z?C&a~f4xZjy6sXZzk-|NX7Ts0^Pt6Q{rGy)@he5Y*?rePo$mXsqSt!*QTJWr*L`*z z6Yh(~@9SG7UwUXfx7e@d{|={fJUg5&w@-)S>Gk4qTXiO$QBQZ5%#1$a>He5}s)*ii z`n8mSr0b6X8K=L4^cGj3nlJ46X~F}ZzS3A?|I__fJrF97ZNFz9r&E9k=>@2b?qmiBmGA0fWkrJW%j`l!=e$<8sSYrc}6{kVRs`WjE4bd68- zwVr;=RTAr$ov!7e{S@=ptkcndp z4?RhIv*R0xhu%p1m2w+@6Y4a6f~jl@H5CVte}-$Fd}HsU88pCTT5JMqZZR^p*=BOdwcARhX5 z;*qaT;-PmFKjZB0B_4V|@vw7%c<6(~!_Hyip{I#IN<8#7;<3(55f8nc zc-Xm>c<9@RZ*le8K|J(M;$i0w;-PmFk9zDS9(q6VtKTJHHJ~`PxQ2^zFp=yYlQL9(p(NnD={$hu%*-@-;v_^g-g0uVLb$r-?_tMu>+# zN<8NMG2)?*6OViyCLa0`;*qbT#6zDT9{D;(JoMwlBVQ+ohklZHw8K-xLw|{Q*f~W! z^wY${&NIYAKTG_m>yOjKLqAVE)-4x^hdx6*#<7dUL%&3Pi?io4@zAdlk8-$1JalZH zVm!pUCFV9&F&;vXI~_ditRfz|zC%F|@Q0oKwZubD5)V5Yh=<-tJnFHDc<9Z%@7a$BJt>_mxzacnfN2lo>}6d zUnL&vqHDxMkGYFR7_Ttz#GQ`u5W3u!5gOl+uLSYXYl%m`>WGJ4PdxIKBp!Mr@mLo% z5f8nEc&rmziHF`sJo1$y9(p_R$k$flp>HD|`RX7ZdMEK%7wsS(dN1+FS3mL42Z%?$ z28oA0Og!?HCLa0-@yJ((c<5uqqa7Y19{M=(u=6nS(2o!gJC70%eS-M&u0I|l9{LI5 zu`Ze<9{Ne*v*RT3&|e}xJ5CZ0{Vee)hiT%WpCcaSaGrSR7l?0ll*RUgJgJo!@L%AI>tljapGZT74gs$#KX>7;-NPX4?7!)hu%ay z>})0;dJFNevz2)0ZN$UQ6!Flv5)V7K5f8nCc-Xm}c<7zP!_FPVL+>UYcJ>kveUNyJ zGsDC~PZN)PjSvq#Lp=J?DDlw8h;LqOuXh|G9{OS8G4CHC9{N$@QEn5&LqA45%I!Gu z&`%PNayUgi^p}W7IZP1`{WS3yXU-50{Vegw*EI3a&k^75%Ktp^&@U2?d|e_Q`eowL z&S!~-eua40d6jtR*NBImxXA|N8}zC*fjO`~gq;begNI&AJnXC^9(q0TuroHQ1=QW+gL*GGs!nMzC;-U8w zk8x~(c<6(~quhpxhn^-L{d9zQ=o#WWoc*K3Lq9}3#<6kYp&uq5^>T!G=tqf1y-W}f z{RHtShe_h0pClgpPN#^6{u1%9bBcKAr-|=%<#~p9=;w%so#%;%et~$@;|%f8FA|S& z_!9BZFB6Y^%@Pm&3h~I-RpOz?))pS$;!eLMV(%M7eyfPbIFKM7dM)v=vyOP^^~A%@ zB=OLjh=-lc#6xc(9(J}8550|e*qI_8dOPv3b1U)CJBWv!+lhzXNj&V_K|J(s;$dem z@zDE;hn)k&Lr)Vw;riDI@z68GBVVJ$Lmwj^{pb+!(8q~C;-Q};9(JB09{LpV*ylM-JoK}~kGuR%6A%3y@yOSC;-Oz49{HLf z9{NS%k*`a{L!Tuc`#e{OhklLtNtfSXogMcuzCn*Uo!5JD;*sAf;*qZe@z86DN51Na zhn^&U#@XLMJoHB5VP_NZ(3^>eoh`&eZzX=p^{+PKp|=x{b;wrYp?44;xc!ao#6#~S z9{JiqJoIkjk*{9jq4yJyd<_r}eVBOkyEO68GsL4EjuH=jjCkbh5b@B*iATN;6A%3e z@yORv;-McS9{uh(@z5uU$GZ9?@z75Zk9@sEJoG8zk+0LlLq9`2@^zMY=;w$(=KAA# z;-Oz49`o!B@z5_4k9qbI@z7_9M>$*}9{N?{v97*GJaqGdP#G#Q9>UI8LwNjz9(OwU zvu=E-A|84@@vt*VJoE*{9Wp|=o^e6c<95#!_GAE&_{@eof+bxj}i|%$B2i1h`#6zDT9(Eoh9{O?OVdn|rp-&PIJ5Lf1eTw*`8-Gp{5B&`B$k$oop-&SZxcJWz z5B)sxb(K*wFkB!W`bFX~&R-%P`eou_=PdEiuMm%M;41OZuMywu?APD!&;#p3=y9iG ze8ak>ig@S=;!*Fl#6zzm9`#;NJoHB5Q4USSLvJP?<9rM8&|8Uzoo&QJPZ8hl%Cnt# z=pDqv&h5lQ?<5}exPy4;-Na*@?-d0P)ZViATPMiHDvc9{Cz29{L#Z7-tR< z4}F|?*m;ew=vNd4hQ8lf=W$lf*+mMLg_$iFoK!#KX?h#6zDZ9(JB1 z9{PFWVdn+nq0bNxJ1-Is{Sxu8^D^Mu~@hhK|J(M;xVu7ARc-*@vyU(cUHz z@zAT7cXlS2cX}=HxKE>wcJ))NmsNj&UqARc-n@x87*n}~UY`RXMedOz{V z*8uU*hnaWR6VuE)JwyDI%kL=h(8q{Jz77!&eVlmY>oD=qj}VW19VH(6G2(F_$Z_JK zPZE#&d`=P%{S@)Y*Gt4hpCTUlI!!$EGsGibXNiY?j`*O;uD8z<5B&o1uyclZ=og8H zotKD*ewp|g*M4S+hkliKtn;rC4?Xsd@c!Y9YqxQyWBi0(MLhDAARc-x@yJ&l@zCpu zN4}E8LvJL0)Y;!eJoFag(GFXQhu%g!@|7YUdOPvR*H+@8ZzCT0>L4C^C-LZaJBWwg zOFZV0e&V4I5RZHf5)XZtc;qWhJoFLbk*^H#(8q{B=i2`v;-QZdk9Eso;-McQzSqTn zlz8aJiAOn{ARhW8@i-4UNj&sZ#KX>)h=)E!e4QJwP7@FP4DncZoh2UnH1QqoeGBJ^ zhdx6*%HbmM&@T~>c5<0`=(EJ*JnIVa(617Ya=S)6^gw@zLJq9QP;N1&V?2bOARhUu zB_4Vm@mPn~6AwK}JnU>B9(p73u(OGH=*`5V-dl)=-b(z0YY%P2L*Gg~%3&Mv&^w4n zIcz5$dMEKHhaJR2?2czrpCCTv#;Zxl*RUgLj6z0 zA?&PjI>tlj3F1+YwZucOBOdG6dg7rciATN~h=<-tJo42fWhkl%R*m;6@=##|5&XdGLKSez1@g?G+PZ1xu z>z=2HhkllLtdpmShklNDl-qgYp8@&@U1{<=W>Z;-SwHk9q0}@zAdl4?C|B z4?VaiJRZW%nA0&HLXQ&&3Ig zL%&8m$^k!=it!D4%>6(f_7Bmn;!Xz-y^46O>k`C6uO%MkR!2Padg4)TN#db55RY!lK9{M=(D2KzuLq9@1%Hb&S&?ktWa{I~0 zh=)E&Jj&rD@z75Zk8*g4c<58aqa02X5B&`BD2KDeL!Txd<#3L8=;w(~y7kNj;-Oz; z-nF+&%sc%u@hG=h;-Oz59^=?m;-Oz7KIz8uV7*;8VmyQ%cRI#TtP`q;hn^rF^-@bb z^g7~EFZIMjZy+AOE80jr^d{nG%I$ionRw_e#KX>3;-R+@-|EUUMLhHl=H2g^ZfD-< zoy5b=9mGTLCLVV75)Zwf_-)Sq0pg*jiAOn)5Dz^=JnS4L9{L#Zu=5b{(8r1IaP}W2 z9{L3Fu=5!4(2o-jJ5LY~eUf z3&ca8A->buf020Tv&6&BE5t*;N<8emMm+RjL-_m`cE+5J@eO*M_#MvvD&nEn5f3}- ziHDvf9(Fbm5519i*x5up^k(9_o&7DuLvJM>^I9A6&{M>>x%k_Ohu%Rv%3(Y4&^w7o zIqV=FdN=VXhhE~L_Y*(n>>nT=`XKQrhhgHOr->hT@sAJ>eT;a__167^fOM^-&=P-xn?@_S*PpooBR5- z)Ae`B-7BUHf5GWkx1hbvI342v^ozu!yt*fsR4?TWwcpQVB zRZeI76U4*LTH>MC5f3}-iHDvfzQwIq8;FPAOg!vwAs%`w@vyUvc<3qOVP`w>(6$#6v$tJnVdlc<58aM&e;-6Y-^ls)|`S&vK^g-fb=P>cm)5OEh5#pg|h=-k{#6uq=9`_F)A|Cn?;$i1e z;-OCv4?B+$5B)gtu=51*&?kwnbNfywiHAN#JnTG8JoGcf!_KqBL!TxdcAg_1`g!7U zedGf1&@T}WJ1-LteU^CGd4+iBSBZz6*NBH6Yz&WY^==&-b2`R1=yBq)9`HfnI3^o}J~I79ll)8mq!9*7_I1W)u0 z?d==b-?el1-kzZY!QDfB13h;?zI*TPuAZTxo&!PGt|xnYw)H%@cUS-W4fDX-h1D5UvJOOzLtA; znka?eJsfQR=%Wuk^u7m&hV~78xNrEep56QVdLJ6vx97bd9vj_IeBn2apZXO;U+PzZ}Q`Pbk zFr@bOwD&!=dskmbx5eGp-}U%V&z`gpZdH)N{-#rovlfqgx_uQg^e&!Du~u7`S_ z=(EW@B<2sdKG^c;y$^k$clT54Th`yVenV^P`ujI*Xl`khuj}W5*Q@`A*4_8DXwdMV zR~0U3=nd}PKRh(tvs3;a*rR{Dx`v7)C4dxm>#sc*B*W}8%WyAfYS z%R$mz`}>Bw`n1zZyWO{^YfyV?S1S^{d)NNq_4j2*mM6@xp~;yV=XPZvHK@~a=_JvRQZRkd%u+G&aQi<;T{<5+hrngk?ri+-`CZhPENBCRhi_Y7{!ZXiqjlWR&UHYm`ALXRww@5w+5B(jIdCfl`v(A3j znH_cEx%?15Ndb^P;(c2D3)vsI_BZZ8wZBq62oL*@OTls48YJ7N=9nD+{s<0{)RpqIR zxwOpVMd>sqsMukq?HRj+-A=|L*h9X@?+bk6*TD`?LJV=x&z^B+d`7exTMZ^y!z38d zJMUV1uY1m2RV4*6owS|9L*29A{qDWjUVE>#_gUxMbML!;To3WpTAR^JoFaw!lB5EDYSpn3lF{YrCkrdbUFU--}TT-PlS{2qmJG9|LDN3 zrw3=f`}E+qPJL_e)TX~4e7p38!RO!o(cowP`q<$9vKIy)y!EGpUwZn~;Mey5bg+5l zPX|k;EgAUXyI*^0U-zz;%4RPfh|k$S_|8N72XC5lY;fPH9}PbE(7S{0>^}wGqJe>j zAAadePw#r^!P)xbY`YVE;oeef8Fdxt>{@zsdF7{N|4aXKqG4`>CEQ1|FRK&`ZxZf90jOQOEOd?jM}_ zebn>xG4%PTXq)O;I&c%(dFR+eFAY4se{k0Kzd1NC``F++(8+_d-yQrabP=C<)xZzC zzw**MKic`yO{inv*rx{vP{*uYM+XPq{mJ0I?o;4gJ@EEJsHY6|bU!t?uX+FA+duyC z;8%AYgY54P&U*S?aIP7cdBay;ivPp61|R(Cj|U%|gL0wTvk$)ewcEPh{gxNsUVAWJ zoV_jnTy2KNd++Joq8P?36wcnZ-ehl^5K6v(A~Ing5hIPrMG^8|7x;*Kc;M|z4cGBeL-Kv_A0_{gFaN#J ziyy(?oa+q_{PM2Iz}Zw3zN84gr+6UMXzqT!*{o?Y$wReMLU(V6o|~e{_hW75QPOMe zRCD)pJ1iE5oFx;I?=J|IeC@t*$-yU3E?${DbnRGk_x9O1kK-KU^7QnV$>)++ zl+I3WB`g_V{fFz~t$)}$7kOio?=M7M-}u{wKMA#%HM5G6@26VLd(Yu~W<2@+wW08P zug^1UN-C4@PZ*m#bkh^bfo~L>_p*@fW|T{X67PlPn>EC1U0~MK7Mb_9FNBO0fj*QK z_TlyWkPl4yuo!)KtvEUO9LhnSL)V5TY zySclL&Z&RICl30x1oGwncF9k+BQFN|bC5d*EIOt2m!T^lLn0t!b)k%*ZO8`(84s1r zOdgsUDk;6TC^^`Qaw22hl{j}Yw*DT@os4x?;oQj>yBgyC^^4A~!8yq@0Xn!A=ORxl&Pg7OeP|iZNuE}7_Y;JR zJai6uX7QL-|71IOJQpyYt=Hikb0PKZ(0gsmaSk5Mk?q%GjzF(8_D|ek)|BG>OVe5h zI`JuUWE1MU@4qJBdk*NmtII?%u7!v1aVMm{j<-&N?}H^w9fUq?CVU+eGV+|9GlAK={e zuk8yscl~?r4xGFGwb8ljUkK;HdDeO-+FO@A6e}{Ht%a{x_9dkI{B(E|(pktmHFWpu z#9a<~F`g^_h55ge@aL~bCh%S;0>Al7?2+Irc~29x&qdg;bAP3uq=yaYZ!2_zy(VO~d(G;I zsTSR6E`0g!9VkQfJsmRDA6Kt%i%{m@p-eG8zmLxk!lSGI_EGZ2Lk`;O#IP=oB==>| z{rNc91HUga&mILIbLN8ToP2@&vJ2#EJ^v7T8A%_~Pa3~tqvT2b$4ANkJMcy`KmO|| z>$_zX`k;N?g^aWGN#^FZQObPZD0x!<1Eb{c93^kpD0$Mp^sfds^{eOy4sldt`b~_( zXvSXT`@K=tD}3qCTca$O93@ZJPjmfz`dWN}bMCVj$iMyq`B#jBKN|lQ^oNb3s58j_ z$M$s(L%s`{Z!&+@Uf{gI0&`LATrHo#n{{-eXV%PiHKFIwk>HF`_!&WGc*LICDC} zCKSrxbMHf(SyRfoeN z6B!!`g>eNGEyZ+pM`WDAzS!Jn?`&Y*PX>lv(ywpqDzQ5 zr@ym5?WNn+<4iucZnkOuO!as6v}eqmp5AoFoSu8;q&Ic-Z0z;Z8@9OgJ#+FW8+teQ zwWZCRbVnvV$965XVN1HZt-n9j-m#&#y(4v3M^CD+&1>uK?&wbSchG6NBjt6pZ`qLU z>g`FnQhjM}Hz7tI`fhrEb{V z*|waexvsZoW7l1s9c}IHU7P34Uz$qyrrWwxn^G<4r|s9Lpd0<2ZP4$hrz2iRe}{Ju zdW2E(I=Vjxv(AKxP!`tww7-kF&_eGB-gU)sG00x0i!5B(+3g;}NpBfXE zvE>epG{A0A3Ho9w*KhX`cH?oLzrWjtU{mzFZOA{VZQ;{jTMo{$-E&^xJW% zHK%vu#{Q0Ui%F$=?!jf6-Fe@B%k{UUY!2o-uF%^0d9lyuQ%P)hK2=+|k6p=TA~%Ao zYkeSI2^^7Jb$pTWu?jWj2e_FTHe*f8=WW(%y4c6@$M}r!mA_5P-|sMv$cH-qf8bRW ze*M`mSi++a5%ErrOAz9!H~WAm_Lk>y%QogmjxR*vp8y^~97G>E9E}d7b2=dRHu@|t z+n&ufnOJUnwkgJ^#t{ChuEeE92T|P6F}7^9x)kx09jGQsl8pHyIwL8@?9OA4(%DX6 zu?O?HyYjgY(HXTEe*|ZoqQH*jo+bS2z`2f-XjZW(FHraC5$ZWpP){?)l=^uUP&(+G z4$@2Z{%n(0ON{zAk#Z^!{ISQM5N7JRY>mSvLq0l0Pcl~sLj=j}!T;k3KHd8Ho0i|y znz*jl>%-i#2Nk8b6z&U|jRf_jGce-oux=$NiqqBIM&X!5y0^Exzc~Sy*wNPCk(jq& z&cf!z=jmh4qASd@EnVI1iT5k2b&0G4@iS~|-T|FI%HLF&wygl{#>u0!=7Mf^D%)o3+HiGlT4&Q_@ z<7U)z7cy{AYxMb%`~1*-o^zjRsuTy!AJVzZx#a}n5ZtC9r{W9#NqydbZzyM8Lv1v1 z@POcytxIgxGw?BIoVowraL&Aeb8?OKc#k#+hlEFlGw%^Wib(SgqN)4=sXz?xQsEpD#C+nW(|NCzCt+2&ZRcqS2L|b1?`ue+tr6ziqGFtjNTp zwtm!iwA>W;7aKhHfIC?U?(ZV6G>3H7wI8eW;A@f1^+WzL&%~)ieXD;Sq4J9|VK+xv ze)P{oXvDa^qqRs!aMyc>Ti_#-?wfsV!luLkqGA}p>Oc3GDQ(q0F^&k9!39iH1#9Y z-;*>xi3n=X<^C3YtG_VG-?#cS&$mTYRvtt0A%D^2*|TZ9Wc(Lxi#X3f^J{K&(YD1l zP4f-)Lp~T3>Z|Hw{rd8rvGsdlYA>6wFB4s8XRz(>luVeXW|-tXhW77IxcTekX)2Vb z>i?Qe^TXs8ARqcD+h!8f-@f&yOKJZ4>7R4Dahr)ye!|)Z^c6{*CVg0anX-&o*iGfD z`l4JO=}nv-ERyt>IDICc?&9>%DQy;EJdlKaSb5R^*$XLu8S5iTX_{(%%LYk5oF>=P zw=5l!G|h*;Wj_y@m&?tK53WRt4)}w2%Hgl7a^_7WoxhrrS;X^sp_|W*nfMZtANJ4V zxBtr}Ig@5S`?D#TI_4)Fe_Ez&G4U4~%m1@Xl=EZNp%~kfXU39#WxS?s8&}5qGZ=5o zx5jO8I2LYC(q$xn-08P++x&4PYOZ1@%@_E`vM|lpDw6N}wa{0*JXn9FS2nJU_{p!@ z`M+(!II^dBb$JQqS9PT~b{y?i9?ToKhyfI_Vo>aJtb4g`l%R=zWMimGa~=j(f3RiVBst0o}d*LO>cukQ%- zT~YBX>3cxyd!nyz)?X>>yJR%_#`-u^-zBW?l8@0h_1B-@)z}5WKa4!TOA7Tpr`p%| zBhdFm*7ro#_bSr&ghG8!te%MaeSOcROO?RWaY@qK;Ind9rb2>Py`_$%rAkkzl@#iGO34)D`}%H<`uZLNeNUczVfqf4y|eQTN|XV*Kozbner7vvXZ8Vk~uUaXe%$Lkl$ z-}vBRn@{V-sd5v~6yb3Xv}d+sJ$R(}`mJHJH{kE;(iDt9K4Dmo>b&TDyFPQ;lxAH1 z@w2?YiuJqdsnNH_VgD30WDh)O_1I5&xc0P^Fxo@9&Tg2X5!x@-c_r8dy94c&dbBGx zSws5|_;cEi(fIPk9_HhsOdQE9q=P@i{<10)hdE{M?gTBr|&TKud8VRqBQu+w`R-=qMO55FYYrp zZY?Yy>yO*=1fLvd8q$=^ewS??`@uTYkH)ij{KL?C+2E1mUVS~+ZIU#eu@!|C_>tpR)G^`BxsblCY3 z_e={}thJx=q4wiNTZl{y%=d~q?{YS9n=U8&!GZaA-0s(7{WA>LH=_N_m|@2|OZCBi zraSvV+S<>I=~N{h)_y1*u%9AjKgBd3(P^=tVzP&z{S;gKfqk?66j|L;{m--gK(zt; zxv6me4`)BS$#(Mg^ORXmb^XTeCrkRm{9T$u5AyS~#k(wr{Z>Am@Mg>i*hdWZGChz+ z`RP>WBR#yrQ@rKhBSm{UJY4{<1ipr^6xn(Hy>O z6OgZz?aj|OlpiQx!uBNj32!Rflb?3iceZ}`qiNG96v8sYU!cFXyZQXaM#^{g6>IPs z8Ys;&f)9JGr*tX)lfQwz)=`>l+NPVDC_Ocwo;j1!(R_N=EJ|~u7h|sjL+o|VdF*u# zn#)r2|FrGZUQgF~=dstB=eO4+CXjEh&(g@__)V{OS!X{>bGSaq&wsL4T#w|_qp??% z&)WytE6V57!`UnL=eGV4>=pTe@4od-486WiCx66;U0*5v39PT5oV}i=3&{NXdfZ&a!v40{>yyfS_z_diMgg4a{z=lQhW z2G(QbM{}1-dF-DPp0-~@`GN9<_M7l7$X=sE*VAd!X#HoIdA)3Gq;zn7rF3w8rF3w8 zrF3w8rF3w8rF3w8rSxdm*Zp6HIoyK+bUy>vbBi)H3%M>EKjHfUENea;U04jdz21I= z<7pLPBx!$*{T9VvseIph#9y(0rTA(mrK?WowAi))=>VkJ|2HclWv0XRc@c+tB8mD@mR!HU!v>f)l?Y{yWhY6-m+jkBs?WEmE$2b zBgR8;|3~7h(AShqIrIJaY7NIjYOKBx56Q<@_K3fV?3(z<3Zz>so?lj1?wO8%MhQk8*Cfj zt@J8NXou#R81U;lKc90$#Rr+ zpYAsl6&32gI!gY+*Z*9;zgEQe8&*O86QbmMi}Lpyg7H|o-%wr6{_g_rHx%gq(E0RV zaDNB-FH-k+0`njG4|@T9!v9XoEadU<@9zZc0saH?y=W2oC;qfxe5`-*=R5nOD6a>B z@mQGGH`=S**6feR8*{_cGUwC(WTpSv0sYhco+(QI&CRUu{QaI^AwEm~=fNTS(aYzv zJ}6!~T)d|+zEkJTH_RuWM*GDK6{dI(;wjKqLAgYiwoOJ@h;q1ay* z6~rSG(7*K$N$MZ=r*i}G)m7Lp>iy|FKOTFeJe2UF(`QrnZTtlL&C+1J6y@hmpIeYW z){n1Zz9c*|eO^I+c`%-e{HfFJ{vgToE#Z|;=e?n8FX0tWr~9MQKK8%U$v*vb$#fbj zm*)77?Jpp=AEh+Q$mxa#N|O(^`8K}n?>Fl>p3Gy){H7-2`}@_I9H900tFvZNKFf45 z@t!{$YOgIM8{gj;jlH&DbB|+)y&^u#_Bz(Teq?(sv;4fhjveb?Ka#zcVZ%sA7V%QH z*Rf=ZeAw$pN*7!|vaT-JUWbd%4r8yw#AmrZ8gyr`Ez@vLY3P6X)}-y(Kb@Xm2fEc@Fs!Uqb!S zGJAcIB>7Q(63H9LuPm@p{CF*TSB|KdnGd%MJs z-}3rdG?p#}{Cpjs9XEDdLH>~V>{#Ag=J&ssj75L)^(DOWvFuOt`O&ewm(8cgy6Kc; ze@6F1X+Oa-v;8$T=J%@sdu5q8ZTG|ee%IP7%fR_|Kg?@9r|o{2W#F{k53>-X*bme5 z;(mN~9LHM^LtcruR(cc1kSXOVIDR;;(yO-VgpD^*JeKws{CwSGwE70g$M{|2O>e3zNY9uxQ_^@o2j#09P1ruq zb22o{^DZFMLGeI5k4f?8>}I4X{=B~2Oud(L@dpkC@+`_s#up_Bv;EnXG5kDvUjonj zzEmE*aZ5E`ok&Nvl`5d3UQJ&X#`q9tp z+_WA!&>tKALr1svi56EmiE4Ge(h1U-L*=jS*HAytdHgw*Jq_+|Ht)J`t=o>BdET)|8Kc2q_P!Mk^vhfyN zUy}Upd4HAw`i*5Kxc&lx{ogo+DyGAZC&gDzP=G1Fp1f=ll+EkOe~1hhU*T-hGRId) z-e7zsO7}Ypo<}1FkNL3vhm9?7pHsH7C$j+pjyILoT6?qo$9yZj(D}wSjy&JEzLDpf z(-)P8e#72aRu%u@`Br8LG~e3k`Jce^GWdQ>;Q2aw{vP>(=X((Eruli8R6zE}`HcnF zr}}w2tcR0WmnO;bw0IT%^(e204gT|Wn6L7D9Qjn#U)|u%AWJiTdOANZ1@=1L+NpP6~?mr9@A9b-Y-+oEn?{fS2{!*P6 z*Y;~|mk6V>UmH!3^OP6)_-6=(DRDa9+#o|`o~&% z^8SJ@|AYR5){_E%QC7IVkHxq0-C@^vBm@3pikt7UzI!fKF7qG$;`o&)PigiSuh?sm z5!ZWH-uD+I?{hKFy006Vx(!kZz=E}1@UNH?-o2?kMcA9^^E;XrN92Mzqq{qa?(xS9|Yss zwWx1uePO(+R6P$|RsZqv^ETF z^|Og<^$Gt_QLw*&|E!MttWV z`T1u`!-uebWBqgOvs!t7L62_-{RKTwUEnX=_f=$nQEBhBQGM($Dlyyve=*Gdg4YkS zAIw+UUzlrgPHFZR$LtG8t-ly%e-XFx=KTdFgVzUq?O$+xfSN~rec;<#m+tRioZXbq?IX_kTQrv#5L2IuP(gzIeh zh1zTI%IhfYum2Uib!HpDe7eRfn@Y0rnhO10gZLk{#p#oj9yf(-!B3a7{G7)1KgD~> zO!#BsJt36FLHo0JDBgqjA8bZ?xOflZAqCGPt{-O*?*{ccItcY+p195nmrB-oy2ceErQc zr(pk{uMh7}`1Qqmus)qH-UEHayom+t2lkV8?wTz37vImDg!@DGaQj1~KY@6U_;>lf z4Vqy8PLH!?*<3i@Gt=#7WIsszzvbZO{rOrq-gLeA$ZtH}GcR45LkIZlsub`0mG*x; zKZ>Y27hn8@_kZi@`a!-wnPq#T>l2uhTaRkPwBI`pcg6Mv|JUNx`};-syM#B%-~X_` zj!z7%N8ktIN&J1xgn$2^*B`#Whz63b{Xl#K^|yGJ6vUGnyhJb@G zmg6Im-{4)!-;eaum&9o=<)`c8v{&)d4IE!`X@bi*^>pl8Ddlp4VUb@~Z3Z!X& zNJ+$#Fu#Yjzrghz+n?;;498pP`VH%U;QFmPaQ!xyuE#hYB=RBtGXeLH?cw%kNC)h5 zii+>h*T1t^A|xJLisiUq|5aLey*81*|IG~`o<;E}I_LJOf0>Eo4t#qZ=K77-YqD3j zAKSeG=Xv|>lsKi6dMcOagN{iU<-B*j~z1@T_kC+?3{(^L@ug!R(c$)3f0u0?*~{u=SJ>cI0- zf%pvZ=Yn5c%^OtG2lUn8O(6GUIEKsn&uZRpveW1EC4A{8`&(R(O=R1nLe}0X9k?E= ztD`J`|65y2Y1)&5kL%0JN4_3|e&8>8f6S{cx1UH*QPu&c?S7f}$B=*I_YKJUUSGr4 zzElaX@Ac?!z+a5w`t9&aCjei+-D&O9`io)Ld%pf7GvfO@pX~LUzQ053J?V;H%Eaq^ zOx@p!`uCg3|Ka{2>yrKqyS`6W@!+yCw71OB{X;JQG4Y)B!>{M?A8tLd>v>`PcXDt& z$Mv3G&nK$&yvDy@gY}l~=kQ!*`zHJ6`xkcqfccT|CQstMvCYTzWx@BkE3lp?Xn)B* z2>qk`LHYHM>#v-|b2neUaDM=Mn#6P6<>UHrlJ%fTwvPs{@)D|_uc=W#{LLhewK#cc zzEi$$FBNr^<~krh_7k;~cJ@Q}mt4Fg@1N*>W?4|EJ?sbSDc&S$@=uhs`vGphXcEV6 zsD1P=){n1|QCYg)=le@l{QevEft*j7I4Ga@3o(45fe+p<#Jn-iACdo`3;hN5TUej# zys0t1p-n}szo2y9U%-Dkcan7STxPGOX;RnI^YY{2da`!E0Qo5%Mf%_a^;c!0m|kxG z!1?1RP`+Fb;rndS!u`Z$rEH&6f!*&@eC1>J5A6HMUQj>9bB?dZIi=YjtTlH2@&13< z{R5ZJ`-3R;=esD4cB997<4g9KkN04{!=A^` z(7W}H@81OCQz##JpAF{Y(EB$N$JqIiMD%tjA+`ZOPl` z`0;JQKMmeEe*ce)=U_c9^1si__QLmL0{XMh zf5>`|c+jQPp7Rf|rwMMq01*1e^T_O2Yy8U!XPU?;0;Qp8Pw{38ZiK%HnoDOMbkPy2knNuh)1b1lX#1*|9lk7(~9Sl_#k_KM?JsQUQY2k^?X*W z@Ode}zF*7pR=4EqxAN)dt#m%V|3=-n?+b-~>HUBeazE%z`hIT_Js)X`CV~b1AYNBQ zy5#p~^7B|ttPYv41@#5_w_(16-+&*01~(<+5qrRTGhU$aJX$PLY_D%+zDD`__PcDa z%*nRWIHyW>W|a5Ut9TmgMEZe4m2)sPFu~SER9jr~Vt3 z592uo-zTO6lqh}Qy2_i%-?x?bf8u?!HXcFm@8tTXa#d4lyqTXXrTXJ%@qS9KKR1;` z%N^du?_0(5dsAuo%hB_DkhezE_iSH;=F6B&F?WUAhdkgreOmq^uh_;b-TPrNzc}5u z-oD?A#)~!_GG0@>ILj9&`LJJD^hz8*3`Ym#M|)Ft+BB1Ry!ib-S#uWoXusfjM9c>o zKR#%_wPT-?w;!az*YD2^!(R9x|BL!pZ`~Vxk~RE2_=9Uv|2aeNhR)?4h|u$v1@-+i zNL%In?(gq3@H-9sP6HRN0g4^MpCA4rT@?-d``%xC_{R_4_u5ZC_yt``_BLOv;Y1-ezEx%AAEEl@Eo#< zkL(Zc!-p&PpK@(~H~#(ma^;T#yDwK;kB`9TYm_?p&hn35%iVqw=ReE+`=j_kCHC_d zs3({EZtfc|{$L-9Y)Fh4Ta)iXvVJj$OvdQQuOPp`GH?s&5`0yR_0J?8MH%t zlt>9ZxZ$TCh!v(IaY|#hmeAM+^-3dWOc23@!10^n*}1U{5`;opTQt{YZpx=W*CG+k zFPv+Uu+C4QzERD!^UUIlula!?GJR4qf6_Jorsl(F+x6#7Ye1v<3-toS z`-Z?OKg7F}?`tg|R}%amUOEombRFL~42R%;?hnz%Jb!$g{m5!Se*fF~5}^zB`|MTS zPL|Gf&urDJV|206=-Q!kIuNGbMxP`7THwOGT!kkmw%9cP9AI=EZWg+q3|!ZfCsMsv zxfGq#LG5LKgAsLKU;Xp?C%^sY)tk*Tv;NmF)<*u_ zjOMzQA71u@N6O1?>Mog!%dp?#->%zx9oFjQ(8V42{W|>b!zhJshRpcwx&t^rj{hV5 zb{%PuMuJuPwxPyYzng4iOHipTARt z^r=M*=kbcgHr+h05MS-m=Msf@qRytTvN!qj^{jL0ZSHQnw7b`(n{gMN4#8KubPe`U zbkI#F_@Ml8%$;5B^erNnm_pD+Zszr*U+-*T^|$nUU90Tk!M{thc$2^Ahcf>&`v#2LabWLmvS=x_n`2{m0#g1pz_4i<-Z7w zx~Qh9dlMuB4`D6j8 zCK>XcMExZHGR2ejp9xev-1zljV&WMUYONwz#o>ONDrTXLnSvzszkr1FwNM$v$eeW&GtVcLIcyQQ=^CbdsTK%JcKaOIP#{a0LhnZHz?ys$1$FCG8t zFzuhbpQ|?s`@#LEQzcfeXM{h@YQ2PKB$S>vHLmliKY5aBjWH+RfoxAX7m_vS} zT0W&b$#Jpe$d}rOg@!gL_9~yj_DEK-i=ibgl5_(cEXNROXqZFnOFpICnyq~u*Lz;C zr^;NP>Fd~#?oV8i=xg)ZHg}{uJZEW^nD}yMM^9oyTX%O?&s|q0)_EQ6TQ<-yai`K9 zJ^j62s;#}P4~3d~x1_I3tYNI_j-Hn0#Jt7$KY!^Ln-d$md)w0U7SC)>^mp8y>INsk zoCiWzPkO<8WN+A#0)q&tyM0QK0U^^}ZQZF}KbzY^Q!Pt}Z&Ql&r`yuOhDpSHF_DtR zXiKAS+{gZ|o=t%=BnfwhY3ny{o|o!}YC5hIv^fEA{t&Re%Y%|rch5_8_4I9FDYo== zcii#Cnd{VWm|ME~H@BrXbau2S(ipM?<@fd^+Pu5AY=%&rZfGIY9?gjk9zpP}@&SLo zt$T|MHtK8Z?z*cdpPy)NOSdI#Vf+o0=L{xOOIOcZoLIj#-BBwq;|)((?o4D)n(+ISupu z7K1K8$~FWhQJ_7Rhb}Y4!h5BFU!BsY-(Q|Lm^1+@p9bo;;{(4!8FwtOV}8d%f9wel zwC#6@GTbqwzWMX#&0D(AG`O%)AfNlAqe+4MfITe8YfRc_?>0>TRsIke1NFQ9F5wB> zK#Ikod||$Y2DzU1{SR9zm1$Ul|87zT>fgArYfh`@wQWso?DZ09@_+?4$4H{T>kqBh zqnZlsfl6@x)xHCiO+x4oZ$sSdvQSM~Ek83q*%s%&je+xKoCkIMWY}e{DVv)7dU#Lt zUlsrB*q@gD#kfDWHB;I4K>3VpSinzwJ^Xm|FN*)1*4P^S4{kZ~Hoaop>c5EEHiOr0 zD`@jX$DQN)^39e!R`zG({!PWdt$MurFI)$x-jhtUf64;qA9w0O&Zv2hV^o?K;oVaq zdL5GC08ag;!%hQHlEvsRsUgHWCW|#aY(GCl2Gt`YB>JV7Z-n@&)AD3rG9Md7;-7F$ zh<^VE(uA)=IXXn2f92}u-=+c{)JG&B4;R9p`?+3MiGc2NxVVhv^Ny~7ZE@bQ4THxE zFiQLo?aJH?W_||p4~M@G(r|d~2>eAO;B6zcTQ@@e_lfr3 zfbSSV&f7+i^STl6?~PFZt0UA8Iv?YaPcb1mWF|~dc)}4b0Z!Lta;$YHmjV})CFpU7 zQ~mCel##s-r}~5V3iOK(s%N}C<-e;PuIste;g_j;+5`0jLlQdz_=G_HdjmLs_rv8K z906}}3exf<9j@hB?r`dt3&C;=6o;XMaOcW6-R?|FG;tT$y;-7K+rxaiCgNiS5KCAeGA6EQ6C;t(}7yPK=i=4+4 zU-09KFLJ)D_=3Np_%bf9D!$++6o0pq|D@szeoFC052qDh@H2`pdN`~2f}c}-(StZ^ zna_grk27Hk$ov;Q#GG>%zToc5@p-<;Ia%=quW`84FLEXnU+_l77de|0U+`wd7dcxL zU+}9Gzv59l|C5R@_;SUUeyvb^!B;B&DK{ThE56`s6#ty#uT^}(*D3xwCx5%*3*M>t zB4@Ya3*M*rD_#9*#TOi}qu@ivSLEEL_=4|H{I#zBor*8`ZpF_!{vO2_{4vFsem$=E zg6~y)>DQBrFZff6Fa6rD_<}#9_y?T)2NhrNXBA)MJgoSFA5naf^QhtreoXNnbMhQl ze8FE){Fvjvs`!GRRD2niw-jISQ;IMBI<5GEpHY12*IC6E{G8%TzvKc({F~r0ci|)R zd9CZOdkeJ@zTkLSTEYB3?)Z}(E_}gj6kqz4P<+7~6<_+*r1*k2E57urMezk+s`zOq z|5b`FcvA7j4=-1I!B;50_~Dg`FZde87d@<1e8KNjeA(x$Q+&bO6<_4+RD8j^HQ(u{ zPxBqVP4Pv}tl|s4L-A!CcPhT%yA;3rPwf2Lt@wiPQGDswV~Q{MUd?y?dQ$TpzF+aB zUk4Ol@MjcX*8PKuFZi>HFLEAMe8G=szLWE)<~#fq#TPkWReZruD89&fQt<_UOYud{ zQ;IM6Y0Yyiv(h;rLC8FL<-!OTStaU+{&BFZ-^giZA$8ivO6iucYD&zC!WE&#zQ` z!B;E3$hk)G1z)T9;s@?je8JZ#{sAX{yW$Jpt@yHz_bI;MX~mau&nUj&+ce*eZ&vdi zzDw~%54#m#@I8tze*Q7V7yNO>7diJTzTi(P{t>6orxaiCXB1!LJgE4BKdbmMj)xUr z@FR*Ze*UQ93w})TrC-MtU+|X|U;6cm;tPIK@ugpHDZb#R6kq(zX~h@(jN*%&XBA)Y zbBZr=+85^HaRL|T5xi_%;r>nJtZ=yS1+P|ok#n-*3tpr6B46?U(`||`cvkU`JNb7gzTmqQU;OZH#TR^!;>)-^ruc$C zuJ|%8dlg^s{faMoIH34~Kco2KhYu>g;Lj?)$az@t1wW$sPM=3L-{CJSzR3BC;tT$& z;>$RmP<+8pD!%yPw-jISQ;IMBI<5GEpV55Rud|x(@ThyfR`zewub9Kdp9o&2_~HjD z6kl-n@=1Tc_lWgVlNDd&uTk}%baEy%-{FmlFMgm&@da;I{L`+U7R49*D#aH)Bo$xq z<%%zQSfTiWuT*@|!)nDBe2wNi`PXW`!`l^K{etGJgc$ zruk0Jtg2teeTU-9IPO$@!FMVCaThP$t@wgJrud1k+xhvp;tRf4@kMV>D!$-PDZcoD z{faO60mY9xd7e>x!Jk!pS*H#wzTig`U*tTh_<|o(e3A3G;tT$=;)|TGD8Aq)6kpF2cK3w}oNrC( ze8FoJf1SGyN+`bIjfyY*YEpc`n-yRB)uQ-4 z6<_c*iZA_ItN4Q7srb^bb&4-|yW&f~Iu&2=KE>bb`kPjK!83|4a&A+6!Ly1la_&%k z!FMWt*2%L=@de+b__A+&Oz{QZtM~`p{CraJ1%FEMrC<9MU+@EpFa3H(@dZDq_|mUu z6<_cpiZ6C|RPhBruJ|$!UsimDMX67yOLki`|`7 ze8J_R9Pw|mjzlZ1-xL2Pc+BC#mwuHgzTg##Fa4@ke8DFxzVxd`@da;G{607Tn-pL0 zX2n1Gi0x;K;tRe|^WA=CspdO;x#Ei+Rw%yUD-}Pn+sd-HI>x9>o{E zJ*N1AKd$(qx4nul_W;1o`)5EnmtzUD-~bxHLCvOk6Qj(#TWce#TPl(DZb$CiZ60@D!$;|iZ62ZDZbzt z#b4{>-=_G2?@)ZX4%(^sg6~p%>DO+>7krQ6OTQjde8C@AeCgL-#TWc3#ZSBb?pJ)l zpHcjEj(Ka_q|E56{9 zRsGVh8pRhpq4?6TM#UGrS@C5awkW>fOBG-I%vFjncvA7DU&|F=@D++L{aUH`g3Eh3 zME}ySHHt6zovMDhE?B4dg10NatUH~GFL<}&%evF2_=2YuU-lIl#TR_1;)~vPDZb#l z6<_w%dlX;r#}r@s^|;~-zE|mc_+iBtJseSd z!H+7w%&TLHFZglAul^J37hYC;!Cz5)(c7zvFZcdIlqKB;F3%*0~MGrd_U+`UuFM8Om_=4|Ie9^;WiZA%% zihttk)_(RXzTi(OzO0-36<_cJiZ6P5M)3tdsQBW?o>hFo4=etWyP0qc}4LBe^v2iTuvyy;HMP7>0vAXX~h@(jN%`6@v*atFZemd7dhp<9pVoK zkGe0>iTysd$I26PxbOupQ+%0M6^bu-wc_t}^-orO!5bA{=1G&{3*M~whh6HB&?yxZaQ zJ-b>vwEQ>Ya2a=*Z`&L$<1Tns@nyd4P<+96D!%A*m*NY)Tk+4hdA>*S1%FcUMb4)b zU-12kFLEAGe8Hble3A2@;tT$)=DU61Va<2=F~t`-k1M|5FDt&t`HJET{;J}OoF^1t z@RN#v#O)v6QhdSBD89&fR`CTtr}!eL{h>DhIY7aq?n}HfKSj=%!)1O7UZ(j@{tC@^ zc#YzVoc!aGun?(V@J7WKIh$1df;TI^$l0R!f-h8jx&B(J_<}E2e35g7;tRe~@kP$n ziZA#Y#TPl(D!$-%D*jsM$JQyn;O&Yp{;*T=1@BgTBlk5FU+|3LiypQqzTjEK7d`Ay ze8G1rzUX0>;tRf8@iR_8dlX;rClz1hd`j^J->>*0=K;kR{29d;IS(qn;Lj@l9w+}{ z#TWdD;>$Q5ReZsZDSp<~e_ZheKONxHBgXQ*_%i{15I^g1dhbDkSHQwQcsV}a0%45Q zE8lIWAKMFlkAA5uCU}cWcPjjZE99mr|7agU@}7?aiZAblRir`C7% zw0T?29IvChZO+E7o~~3Iy=%s#=+#my+qU#<===;*ug{pqjN_#{GHH_nCEbBnbKx~+ zJ$Sd+ro|hU&YM5KWyuvAmn>MaasB-HSG2dSU$Ak>hATGC>sY#UJy!{SSK6%k{4F=! za1*~+?e>oJZEaos9ql)Gy_>JQJ?*W=yUO~_ob~z=sHMl z-`VDx>AB~|jx=iSXxr@i)wSM3uYuaWnK?bZ>5e%!-uk)Ocp=b+O_p^7I*&?Q(|G&O zdNf9;<)dH$+TOODU&>X0Ta|WnrZ&^mGpTA`}-ob`9n%&adz6CGwAq}V6EyQX` zbz8Yoy*(Sy+6C)3m14@DBYW~@4T`)f_`k*Z4wFyCat!D zI=rFRbA-lrGp9f8rQ6oybL(dMOr_HIY);{QN9UhQn25dfA}`zQN~^b(=;?9>FG6+F zG+w2bPIb^L|1fa9n^W{AsQ%Q_ix>`=M|WF)e`=(6|D|qd+tgv-26uyf)n9wpJ@e)* z#;efi?PmCx7oglkFZ*MH2ImE7!J6<6E?XPFJIlU7YnYn3h^ssA7dElclODMb?HwC> z+dEQsb@Y%z@LI!;ZYYXQ;R5^)x>9{Sl)Yr*^!LGBz?b$DGH zC^eM3alw35+99db4Vyb*Z6wWgy*(Sd?&|DlYsVY&psiH8H%)#hB~H}&zKi+zBRE&Q zVrj^c9JJ7D?maC+fLkv(u11P5{pO>jW1VyBhfA6YIdPw~E>IImmzIyU&e11ZEka*^ zrgcEpE4fyC4j{EpdBG#;UN&>qt%s7Q+kXNXRHxR#(*3rC=1cohA8pU+QOuFbYralX z{loSDvG;5X(ogC?=~&7OUXIRF|Mih{@@rgkJ6-u9`jhgifV405Zb1HU@*i;W?{-9z gpI&JrY~w^bNJiUh3EbEuZa0bJ zEw*s-VtZyX!%I9FM2th0m!K?z66cFOiQ^D{d5OQwyoAAyBZy-&isML*alikms(WwM z?QS*l@+RS_pmp!5Q>W_Gsk2no-R@qseznu#U`BGVZ!iloav585z{>UWY(Gm!NgoSs z>37xqJlE&?U-Uz%o;=2moOk6Oe+j>rUAc3AgX1gs{TAmSooJM$F*l=ozvm^X${(*lm778;}1P>b{>weKKQ`d|Kaoxqihd;M_MjEJ$U^26NCF+ zdwTFG9OrEK$>8m;{djO5j;%PZedPCp>z=+acr%XcaZJ5-Y45+j_TbsuUwQCs`hq3B zEhnBDJU;GAgKJ;-@!-YZJwMp{(C-J2KlOW*Tio0G@WW?sKk?An`xiVlxbB4$gDveZ z3?6^z$AcHX`1`?&`(8zv^4|L&e&Fo!1rMF=McWsD^weMr+Pt{og~7FG>-YjHbA9iH zrye-_KV?7HSARG7z{6B#dGCb>A3A#+W$rKj(%{_HUl>ez`uV}s>gNYr!1Kj@ub|Ab zUXs|N{m1nRl}?1cx956(@0da(ETAA_$K(7)dg&fV}j z%B<*J`^YEH_C9o+<0wDHkvp{Z>HD|LdG(8dlzVbdrFcSHQofrTq;d7T*kUin%;5}e zsfUhDa`=bOxh5SaT&b?x=(w`iC#y_5EY2KF89}H(ZXnhK`ax8oKC*E`9^LI;M+PmoA1z z(8Z$(bn&Q77fDAAeeAdCqoKbe>Ewl3eQg)27X9?{U5?(jJDk0jtJskXJLb6G+Ux2a zs&PY)li)`^y*K^l&PV?M-9GkAMW5I2o7wl(P#=3~=-O}Z^ACUhdjIeoH_M#N4sUtG zcE;oO{)pzXI{^8nbIDf%0eC(?gea#Oe?-%~z z=I;UiJm3R>e**aHfIkKJ!+@U!d>rs$z$*Zs2K={xy8ssg{s!QO0XG270Q@DuVZb*7 ze$Cl;?DK#FfUgJqbHINGcn{!NfWHp-5y0C4PXPP{z@GrT5wHvJX8_*~cs1ZZ`1_jg z1$+nK6@Xs?+z$AQIDQz%t;nB?{J#Ud4)D!@^8r5yxEOF1;B3HmRrWQ%+h${K$0H_aBJikqN9X;$nS6X{?X@D(H^Xz4E;qYtMi0#yV%#iMrQtev!^I zPt?7M^Fcb#I8pZw&flW*aVLn*%kJ5d;tOq=<_~RYc{=H`Z^MB5x!h3ponKEHuKiW` zMz-dd`xxTB@7&RA*@sU5ruHjImx;$(=e#40-&=5`>2nK?>VAOp@6mbciMkhX{zE!X;d~CHgtG70 zoiu##!%4#}8*U+m1$Ywy?&KDQV=6@jO@;At*+nhT!hnld9^NLIuP$q#nU;~3}X zUbGdaw!A0m9>)0tbe?>o?lU-ll+HaT>iz-ee@o{{f}g3ONAG;oJ>2q|8@voi8J)Yf z_M?#5hfaUIcF-;5bS-$A4W4F$r`h0XwkD^SFppB@WRB%_w6}%YbDyaDFwS?=x$8vT z9-PnvN8Nob54(r&N7?4jpf6!}Uu|XhM!)~GyLPqna!m^RZ*`1qyMTFP3g(P0 z;gx+i$-FtBaZ?_(gF4sZn3@vWQj~Z6@6sHsFE@pW7<`Y1&j&BKoIjlde|VDvdg|PK zVl89Cz9e>{&bjG?FUf)9MjX*6=LQ^IC-m~`Ppowtwupq@%b_E9|nGk#Ltb!FL&ggqWad&V8dpe(<+A7`9NcDYbNpwV&rc`8$~>y*Jakz zj{Nzu?b7JB_uUJ8iNvpnf&UEfdmO?OLb%o@v#iYpeqFa9*KWA27U$6j~DcR8w3Ag zqL=vNG4Kb7K3mX#JqG?^qL=vZ#lU}o=qCvJ7h>Q)Trr$_5Ab%qsYkwJVsw5xm5-5s zCzYQRUH+p~K1TinR6ZxV{AZ|qjQr10`N`4czd+?<OZkCA_o%Fl={{~DE#kv~Y~uZb@ICY6to{}z>> z8D0LJs=m~Fc)sQrQlTHT)~j_E94T_-e%pby9>t>5i2ZMDT8%mFD$H$H_P({Z3Ul7d z-phAY^j?_tY%8tPB8Od=3wx2D0(>gUrJ;N}=D_26FJQhmfH~=J?`D0kT+fF8fO+Q| zX%&5c#60xP!~WslbYO1qsifXNVSYOFXMf)?=BxjamOAGx%wOMrePi$C&sX#f%ubne z;V1s#0nFLn`8WUYf4t-$KL4YNzLzlPdikY_zTaRj_M1O3oI8-mhF|$ZMc*rbaP*!3 zk)!Xon73V;j`kyL_&*ye`UW2)dNzC!^Sj?osO)>^-`Mc0nEU;HR%PGi&pY~F!#wfz zs>;5%U(e|MkF<4jUi#Fk-nX7*;L{KO{lmkDQRm(MzPI+S>mB+t>wCLnEo8B#ci^90 zy#r;HeHUDneSbR5hKFiy?tSwx8~$ULqwmcf>wDiwbM*b;8*KQEO&fY&uX3Ut2ikP@ zy_)F&AI`pu$&TT{2Z={V-@sDD&QJM=5d-@!O$R)g4G&=cI4~ReUxFMm0e=hdw}8t8 z{3XD%A?JBi7VxJ4F9lrY?7Q&!s@@Be9m4}ZVZ#?@t(|k>Qz-Ki=0S&Dw70|efxHZB z5ybBl#BCbScQ&2wr97-%jwjqpatJ+X_TUKmcQ$=jTf5W#l3E|T(((oVUTUAp=Q;d* zj@z6XyN%0teG|IpfbL-}hPXLY6)RU6yzn}Fy~lR?x`Y_Ip&%%w1ywA8CsiHF{k&L z8K;gtGyT+Bly9c|f4;7_88rWj-?6G`r|zS?!&8UuGc+@>h%IQkKkt}o$?l}=6ElspU8PS{-yHXV}~bjUXN8x zIF;h7?wyPBb5N$YD*IH*^Zw!EhqH%#4*z#jkWaL|hsO^cs~V5`*6{j{AI=&|MLyca z`osr5)|%}M+~K7B;fpsxca}Pai%@Q0ud~lvu#WRK&>-*w3ngv`;cm9*t0nzx&_mZb z{ZcBk-Yjzi>c5`qolW^0dM`pxry*}D@|NMa6i08t#@-y9FUHSXu&H+oj5{gP=0GK=}#$s&4t@~$KzPS&u{I`#Icy4Z|@z4<9vR;qc;sl zl4En>PU`3I9Q@AxXnODYPfY85`H`u;uRJ!T_l{m*A+^uGCvoZdgdu7_Z!!&@DFZ~ZeH z{&SV1@9lqtox<)1X2Wl!;aEoYTQNNF7};-S-@q)u@J9oW0Imi+0Wj9J1D^oA1F#D) z*1ZFF18xBP2Zk}i@pm|4eLT>G{5t^u8{nG&?_<#A&8G%_#Gt1*K7-@o%|in<$gjHP z)WBB>zGY}&2fp@C|Gx1Aa|Oz^g$ftvukZXFtce?Ruj zt*6jV-!a$rp@A~MhjCm;_#LMPwj#d@$D4_6$I!qmF18t&1%z;7Cgx1X(dlHaaW03`+rhio2VF-NmA7H4Xc_ zTkE>IYTC+b+JZHEo7xW4NX)jT_O6b=hPuwKJ6i6%v$e^Dw+gsxQR%Y!nzEY4?t`68 z4ZIRrS$$nsQ%z%6D-bR1dv9%N?_O5Z-Bwq#rz6l-*KO9bvaY*sHP5QtwQW^RX-$1c zu%@x4jVi98pIOqzK}`wzDf)F4bfc@eF3{9?^>ifA)YTL?fF5C#0!^(~g|?v|byr14 z8k_3(@2%MrsB3EyqhrsgE7o#gW({?%t$PD?oz2|nIs+Zu9c1wwYUpV1>aJ_=F1ZTS zx1p}HttrrORSYwRjrYYqmON9sPCkHY##JzeMt5|b<+)dl2^(X}I}~q#Wl&Llu|d}* z<`Kq>WyIsyG-PZV^4B&kj8^@Eni|YOF_o`tttqJi)Udy?uBNMPQAx0-yJ2rrd&%OO zz5D9|je)wB)|zg_x|)WL)|&bz%wB66n-Kpx8k!nwFx9RJGRMasx?&VC%&)U9(B0DA($UUmC1xE|JLY9q)m(F= zfIA0B7j^;cX*tjL*t6M(c>cJk{4YDHp3JEHXOZt@<5EZQiDsL5c`M&15#6#L zqXOD%p>#j;UD)@rrw9_?w*i9ADWN}y=$M&)PbgHxypelCMP7a?%FMS{B4I;j?q$dj zwC?rA0!j7VAj{D?C2A)W4i(v(@rKcC4q>t&b&(#WJeJRSoFVg?<3tj_Kr?7a$c)=F4El@*uI_id*`@#5vIa(_!}qp#~=S9epJuWf%$lcyU$tfH+D#j4R_)xjYU9dPE7^igzB1o}xbW~d@0YwFJ$L5xR|}D!3)?K4W7@Q)8KimUxVkeQyM&n zoz&nWc1(k3v!fbZ$R5|=Ygvy57qI&@coqw3a6Suaa31T_U>|GN;9OR#!86%T4ZenL z*5DazjRsF=ehr?+N&r`w@}0_j8a#z%Y4BunDN6YqcFC#2lh`>8p2*H<@C5c8U~9ay z*(nV_o*mQREOt~YpUHYOdDU?HaBxsg!$O>SvVk93LPJ+(D7$h>UjSe9p7`aj%S;6{N^n>zW-JopS4TJ zhi=pHXKHl3zd^_MAP}qlW4G)0&22iqzf;F&b?f-h9XkHZojTtCQ61lNkB;yBxPZwX zJPr@re7}In9yknJBz_Hh=#j8t59c&|IV-`P)l^>Kp&t@9=uc{RjQ`VW{pD=4gbn&0 z4Ub#cHN2bozNFK4O4y)3so~x1l7@$UpVaAlBy7;1(eQ4T^<}kx&~FKg{*|+n8or!e z((rCp@)eyvBw=9>sqBn|4SVo?Rg^dEp;N*FpTYWV_^hvq@&Edi|Rv zY|tOI;V;?n{;%uwJrXwP&)M)L-_ZFBN!Z};j1BKQrPtpnVWa+j8$JtnzLEYJ_*w}Y z^`Eri8P-;+J-U`9Gxv-F=0`m8U&~lOew#}Xmp%p?fUT6^cN8(I1bXkzO6M239cYez z?wQbMB-hj-mpj$VGH@Pp;e3hA--pZ8ZTRh_(r6cnj_=6hWqEvC9?L~B3UPeiYpemj zPee#YzOfcrDYDIc37S{Yjq*nR(KzJ|{QG?j-{pv~U!zu92%OJkQxEz@6L~lkXRv1u zCJFdH9O^SzI9LFoiY)d*Ch!P0lrSdja~bSNpdbk)Sfu~qO!nMD))O38Kz0~87s;gh zdxA3xiWGQi&>5i?{5cTp`CRryFfT$P&d+gp*>js%dVo1;Ksq-Mxh|szev2k za4-#uBr>S}zJtcH#|`{=_E<2(69U@k&tt(X4^?QvUQbAYdx9P;{jiF&+zp+`Nzj z81kKRP@?JZBF69j^)f@04+oRE?eI+KM>vT1AWIPaqHY)V4dN&I`ufs-H(!548eUeA!JLPrI%m@O=d$Yk69qjBrLk9`A83y{!xQTN5cowm zk-*=ZqCToWetoZc$%jNDhk1Qd11@5h#+&%_P7S!ZfpHl17X%se5uQDqO#DA|5c?w& z87W~JEqe)ePqExD zw8wDX#VEOU||w%v_zcnhv-lE&|zn9ST1WY zxmKYsqCOrVx&8SMPs=%$+j0 zKYTP8$tO6{AL9PwNN`p@_a`OrCqKw!p9tnh_(m>m=R&@Z24~F*Tkk_7TEB``Ktd3A6jU`m*ngIXps$Hny>q# zG7JBGp)dKwEguhxC#&;y|5R<2NB?_*>++df?eEPt{QCTmhTov!Ngs>M_L8_h&Nbm= z)L)cu`I}7~Ki|YB0S|ea{_7ErFE;T+4|$q+LC>d~ynjSroNxInL64=5>_5?$fI^VKiLCGFVnn*LPK+vQ>U!xsd-RlksjnZqN%Z$n zF|PiZu(Kz)a2EHEvi&E5NfEd|b3KLi2!`KSk7Tm1VSEZmBt2B0gW%OZ6L!)1WgXw> zlt!LY5(7RDc~Spqy-59kGK0_Ot^T8bQS0m3O8;p+w^+tIw88b0ge%i| z;+Ic1DMO5Zl>W|Te12x&5x>;=HR%T}xGZ^$s83j*FJ7$bPxw%|CI9ERd@#S$%qB;|59ky5JEXYre;kmOJQCVJt~}}eEi(Hj=Ew6*Sk8~N`0xFI`^&fcl>XYs zKW$cq)?Z_P+mKI+ZGDhxUmqkKZ?D;ZhrT{&r1imz-Xp{vBY#i!XTSbt*LOL3M8dEC zkhfO#J9@lHzmejOH(eUKqVcxtyRiT0@h28l`#0fO_J2(2|E4SGzw8b5i2FCv){**O zXk0Lq{f{31EYx82Umbtg)1eW@-!6}M_D?J;_U|=d%#Zkb$XG9OeF!S@_+a#b`im4- zAI1Y|$>Uwvzd7F0{!KWR{cl$K`@xLA*xWJt8$JGnxB5GZ@i+9}?(dE9mFr*Icw74e ze7t>1|38@V=KJv@jQ0es|09gIUEhuV+x@>e-Z97DHvhlm{Uq8K{_lese}4VJ?0@w5 zlYHLQ_}k_2zSw`iB99M7ANchbQy$Ut!AY7tM$iW_{~xXYxAq75`u}mIzaPx_^Xp4y zfA#SX9~#--dxB}RNPjH-xPtgUD*JC%`X6h&u^Xe!cQM^C5|4kFA87G!F5QkmDUN@i zHES}ER)6wr`^i2wpOb@L=(9U&en%`&Jj4AEe81R$;qNw>_bcG46z;d6`;llrx^wd- zet%0(&|APlM*F$g|4uJp-O;1HcPZJnom#??0gX zowBt152EK+@P3`NLyUy_dV(nhK1^o~`8*j+wcW3ft*3O%8uf(_P1EK}bblH$s{Ntsw>tmxwDz%opUr4MO?lyZ$`4h3 z7G~P_!}a=#HGZJ)9tv;JVJZKT0(MU2uQXM^{=3M=Uzx@q?q4Zj&#U|_&D8g!7i;ur zpYG^1+YcXFqVb3OV+z=i%FlJF`t|1JT7BU|*W38Hp@1GoF~+AScvB{sw*gcC)$9A= zLpR#$TVdnJU%-kSCVfw^GF87`U16iIvhlMLj|W-w)PJ%sA^#-I&sW*#t2KJ!*Sep8 zu8+drUNHJc{%3NgzTcgr@lWmf%=YMh7uZ{t!4F-JoR-S8`-7%OwO?e?)AeZB(^;ea zM*!z$k_R{B8$L8Ms(tQ{xxNwquqVIKKCXuxnibU^zaqfP6TkB^Nnb_%#4qgWXGZ&( zpr32Chx-ej2q)#m;ve?2(`f$@&@YT?f0@}|;`Amw@`kS`a1AY2;t3J{fZ!@R#e&LW9b+1Gda^*Ptth9J}w&c+@E+XeIovHGp+SE@dy1Hu*z%xkn6|lTzmVL zz7fCBFFdYj$kXm0;oqX{!_qI}e_kfZ&yXkAw^xn&GBMtYzm)c2`5WpF^ebqTC;g1} zkGg)AWm@Yk(og8uYX&{Fr}#(S|7icH>nFyy)}SZ-)Z@_;!D)8?uILl-4}I-1=ySO~ z`5_MxZ-u^E>mlM7`g%#0kE5@aJ!GOh^y!#UztC69zY)F)-_J4RFa6V?T7Q)Od3+{+ zB3@bgFZ9hCAAz^@Pv9;65qL`<1l}4ifw$x>@RmIayx-s_#(JoF&QtuIkPs+e-#iga z)9%mF)+g{sAH)6H1-PFb(HPHb6k|M-EPs#t&-wi(G{2?$P56G&%U*HIcbxrXl^**0 z!@xDfEfT!snOspH?PpPab2D)NWa>5a1Q5S}66LG+pR?+JGx~md`~B$jya&ls*b|L! zaw^HsfS(BZB5uIGiFm#R^*<4u$>9PCKZ@r;P&#tW5WgP`{3OHtEqE2yo0zV`#R_>= z-+v4MCBUyzeM!7NFFilQ>rdfux`ZDMx@Si2MvZiTDDdGR-M&dHBqYF(2B#(UYxr5r z?`M7spXJl=kT1kViMRJ?FeOF92O(HL$IJS&;QkR$-WKgY7RYz!18vCX(O@>FPZmBL zoS@N%gIt?Mg`EF6IDg2?GdTZ+5U)l5XmBF7Lo7HPoUQdI9Gql>ar={zc>VM6z|BLV zQC@$}G?JIWe>gbVM$fHF)WGQ%;Qp(J*6<8YKV_O*p?@ss&Zprt`X3HXwbfT_a3lBs z7vk}xhnlVWrcI*1bWRyeW9 z!>)1B{l*RkI6r4DhjITlhuu@=OBnM4;4`KbTlfoIY#iwWe$WR;)cuckd#Nz?Pq;o) z{IFdQQtj8Uf7*W~do7^rN$Hk-#(3Vv-d<18t=wNo_L;8TkBI%6RPuj@{IEZgO8(A( zp9rRMdvcrhn=b6jv|lfD(Xv;v2Nfp!rS%iHUwj0i`u?v0palDUjrhsX_BVQhc%1HW zt9{sS>N&+e9}VWDtM>LpFoW}xVcNg&2T7LwYxWTi;=w0BXF%F#MuuMh6q`Rt(#nT} zy1hIa%z$IH^pW51sM%*e!l{an@-GNqDt^4Czu{oCed+dDn4{aLZtw6fnBA%MPq5V= zZEw2$7Uih+8xGFV=;3ds=y_HNjBVfK6tE}B>@;a9l5=S(TF@UZu3aqS)RH@p2s z**o-4MjzqdRQu=itIW(?aLb>M#Qm2vAE5a?l9w6&z=+@6AI$IymqPP_#POQ!@fU9)HS7lZ><|3~jooK{U z5nl^)N!RQOR?EoepaCsugLzg^>{bm7T>1j=Jqn$K$S>(k!<#`|;S z_LeYyY1dm2zs{-lW?v7E#V=hS$KsdGp8n$EmwA2Im=9|4>u27!QQA|&_+?&Cqj-k+ zHKf|pSp4GRrawG>j>Rw4pIi;`D{8)|#jo^H+grl;g$++*y~6iL%o0vjb8P*k?>}A*@hfV+sKu|3d7DOQZwceq z&usC_uiD#K{2Gg2%+FO0^X0$B{lh4G($-(!_tuTlo)WIVl>6!U`pdjOkLH78>n|>D z`ZKov!u<_bU;G+*{q?r@L!-2}gz@V|Tl_NbM;%*#ja~oJ?dz{Genr`n7Qb#7wLK+_ zU&{T^JbvA$&IiZh*I4{guP^HMb@i{my1l#J@A#!WufX>YkE-@Iw*DGhe_hS%udjJ; z9i=@bj9<$0EIfXh&&Q0dzsBz0xccwkh?*~I`-iW3ZyBY%C0u`@fU!T%n}X@oVJu*S~vLjndu{uD_J`Gw}FTqT1Wo{^3~s`s?05jIt*!e$5!QJtd4^ z%KK4x{4(EPGPZv>wtsl_?;l3Z7q#`*HgCoI9lw+JAOsk zlNP@|?Y(i7_LMMwDetS|@hhm>)7bqRT-@|$?Eb?T_aCPH-+q2M?frg!IcmPB#joeR z%SUN%Bd@>kyt47WHNJnCro6B1O0IwD@7tjFkJ0n2Y4UvmqksQ~{e2s0Y3lm}9u2($xKo|M%X%kv?jBN*KS4_X~1+!u2on z{e)ys6K(I`u)kj_(fcN+CT?mgK({XC;kK=klg7iEz-zo3`4|v%P;~rtSS2xf(s*$CItwPo{>C^?nhb_I|j0+xs=5 z-XD^!+mrqMAw@R+=Zy0GQrX$6z0v#k{C;bGi0=bL#V-svQr!1%Oz}+9-oG(3!>7U4 z`#I$MrLv1GJl-$$p5mAG{`_A?zb_y4&oJX-jQ3a4`}`?hbk^g2{xRsE2qs(avwU%( z{{A}tzAEMWMB@Df9#(`GbY&Uuue;>E9HDfWzpu*vzAWv1n0UX{4+G+D4E+6-FXyss z@aF`7c%Nf3Lwi$$EW+QXm4mXI0pk4u|0vIuPn6F!`{$(o;r(F1yUw6|xvUTJ zK(w>uu^4#2Q9t;}!S`mj;SHOMQHJ=*36^u1UdhM#4J;Knn&h0%^g08}ZSeIL{Bovu zad1gFy-$nte?HT_IJm6br@_~imng6^aHB3i-D<_z4V zmS>BDE6O(;We7Gmxy{iC<57r-n~a@C$+WmWMQant~_#)bbt$4}MnA`+F>Y#wq1Dp}*oZ|J-Be#&_E zO$tr+4SRS5@3ZCP^ipi-U-kZ1HG8+)%R@nyMEc9^mD|r^%YFdEo+G9`1AgibPlW9Q zL8@p^voF~Dj38+TUY2)h&_(tF`>^Z_`0!n>2x~=)RBTTM>X zWI&`;$1(sf16<_d@c9h--ieFzTmJ^Wce2n;Fj;eF{hRdt6AqJYb=D8@@1b}DH@HzF zB{zNF+#9$tNvzKa&UAYND?BSqJe)>gnHw!4g@Atq@bV$ z^4Hn&d({3R!#^%w9741xCm7>%ZP2}#VF-H-}Wou*MnC@ez5#{SgquDGVGz8vCYyx>QlHq(D-$7 zdm#P1DVgFKlGJZ^^7R~q{yz6Q^zEjI#PfIr_$EI7R0jH?>08wJk-m8n=$okD8oy*u zEvmNkP0mkDeM?Tx1>VxP` z_IOl(pvgC;JtrrSui#ISZ*nafv*hbVN0Ac9*E@oI%UQoE-^BKuh+n&XB){+cde|k? zoSEkt};mTtwP zO}c&J{mtRKyta6dmOy_5f0q8FrPZQ!OMlXB`jeKX+hwAz`KO` z9EN>H>yOGW#e+orrbZ~#eUSX7vdxm;N${^2oldDgpR4EkL*rxl*Q9aMzNJ2-<9)3> z5$xvvxzm`h;IjgOWd8jjl#u0@k5lIpc>xcfU-25aJ|tl(C==JW>oU~&hBt71rs>al z`Sb|sH%}Z+=KGBn?D44c3(`lDr#fGv>lt#s#OF`cerNq#eEx*_O_FFInY?~?x@g~k zlatl?jOO14Nk7vS{iO9%Mgsjz=KZn8J2RseO?KE`|n^KoN7m_HNz zrYDeZ64y6NzT?t$`P%J&+&JC-$BonN-!qQv-^1mbOY=cb5<4gPb+GFHzX(`LV*MVz z%g%370)EB&b}fFBl4?<#r4M#~lah3Plag%wCfWJL2Q!_*UpZj^@V+4Q3h(*9vRA;bRhX~#fzwulE*p5n*i3G(_OFYgU3nIPha zz=!WjwZ@ar?N*X%=sv>j$rIsaK81D=LM247}!_`20(W zNBK0r&9=!q-?p9-@!rzM3E8GR`Fbi_e7~IJ#n)34EO}vmNb;xtXPf@XJHd*d+>w6`J;r>Q<~JU^|DPwRpXn1t{2_gOgzFQ1Z%W1k_r1$m`BHq)liLgSD@fl+{|x=| z1}08Y;v?*3+$7RRCdc#MwXA#@J1XggzLNeK`sNKx$`RiS5cWXpYstSHe=(orRr2wk z7@_?o14g{%-=6|3^)D$&ji-pWo=LK}(7$mL1%JpC@|rBJHyZ7Wcx2SSqEzvpq<>qK z`K{n5S><;{<+upd&-DfRkQ1DgMB~GG;P6t9Ii4>+BK*%$IiBZfJafhoK8F}U{Uq+lplc!MqVmz__-$nXRE9pi1#H%rX@CLG`3jZYJ34b|F_!~pMcKg=-3Fgau z{>Pg{|4IMAk8ICH{=rM~Fz|VSloY`qa`6`a9k7?_{sjCe!%wt4zRidp&84R|Z>=ug}L< zlnQ+|{1uJQG}f=lm+G?YCFvT=-X0-)o~hb1jn5QzNu#GLT-NwZo?+R0rU(9FhUqV) zyeG0ANuP}R)2~tOne=0-B|ozFnX3KXJBO7o!55^2ehK}U!pqbC9r<@*Psoa6-y;57 z_(>7nexvPMwP)O}iY!Ci|fA0<;*Y_2bihWc6TUhy2=5rhTYxYk1P^!#N zcz-4;^YKS8zJg!N{&Tqfa~_~SIl;-?{yC4-9^^HNZIP+Od!B*L8uJ5T|0m7<(Rj|}c)ov%@yzl`dm?(beZE2ZHJP2$ z=xKgn>DMHm@IPel3&D>!Pxv2U|9rmHFYN*G3;roz_#c7K3uNO1!!mLCW^qyS9FiCO z!8F7cF+SKP2LIRa_W6D=^lRco;)f?r?;WqtCsODGU?$!hm@rHD2d)q3Z&sG7Z!7Y3 z`-H!oCG1`ByP{I~GsAvq{v+$-`n_U{@NXQC{U(~fkbL=A!7KJ$2eh+ohr;m`e&-!zWT z<9M;3pBKo8$o4or;!lC_Pb6>5m)27J>Xh^jPM^=|js1W3wZcD&@xlIGq419a7XI<5 zq)#IL(^Y>#{LbR_8S7#9I5ob*e`QGj%lXsv#N}a3DFEE}(!oSdX zQ2ff8qs|w&e>0H2iFihkg(q`jf-@HGC1r3;&Aw!hC5D(BFmJ9x>lA%fsFl zSo0Ga4-dXJ#FOb?=J5Iqd!0Prnvak^q%TnC8}LuX>imMPx7qB`wx6c27c~3i<6)m4 zj69!MQL4mq&QVi0@YUdVT$X`3V0$GDHG-qJG$)n}uX~ znh$vKwHQNwd_KQOu{YpTOI7aG3jdw8s`I^B0FV;3~1~o%F%I$g+2eclvq@`crDzZ!Yj7 z+kdpZdde(&CHvCtGcUl*?Nf?Rc>$N#1^#&w`XRsXgB%VgrKq*=%C=>82^2POc z?w`5;cSWc^Bfe|;f%2q3xR=Q*^v9v=3&wwiO+UyUafga1kM~Fj{phq{KHf(yn2+BX z0pot7YwjXmEAhiMWVDa*6Zac!miHT(lOefZ7j!c}kx)YUk7?h_oyit;CyD;iMS098 zjs!eOWRKWi;@^AK@8=~P{WjPynHq3%kY8k_ z{gZ%;U+?GpS3g|H(gX7P{}%8M{+vM4-zBY^%D3YCz|y}(${^3@>3Y2zjsXd#L-P54m$lu% zp3Gn`9gGM(^$7bj&O`7SMtRI9aX$waXr~YNXJP*2JmeGxK=r-2i9M7Oi8wJn9u`Vv zxW9#97v#fXd4CF3XY}8RXAv%a5g^GM@?Ckbhu^Bfux4ereq2rQh$kU0D(`y=8F9*M z`T}`wI4H@vydTl;cQd}flZAN4zt4*H_P?Rr?-FrTA3VisvEPRK-8etm{WJzWec+4e zAwQk|Zty=4pC0lL8~0C;yhVNG%;1ODr`~Tz_03?7@%V{XpT?iPzV-3zGy13TqsoW) z2_~l3+PBjmi%%b~KN>&w{`_q8^=bUs>vOZ_c=Em?`IIE4*V?zsr#3$QyOPgCqpwfn z&n};D$FJ{- zSqOQ18C{==87~lz5`RxotuGh#rH{To?Ri3bedprW$NMX$tS3QzQT-i}zjLFnZ$$o% zvQH=AZ*=l&W{;1yz7hE=Vt=2YzR}6=(&+0Ok-wAdXafF5C%+(japO1ZiwDpb1-#M+m+pzu<`+L|`!g_L1H=Z9r;tw4m-fceE z>|Dv)e}=E`?B!`cfG6q{FMpd>zWU(X>}`Yp{!IOOq^l4X!WjEICV?>tj7eZj0%H;w zlfak+{_jbEuI1v&PX8w{X&pFy7>}fVLc4RqlsmhV&NqtM;9rHE#9xq+#a-?MdPgkrC zJFdgTydq;}sxv(`<1PL`>S32-DC}TzK@yT`W#L_J6)!H-;X!4S-sM#BA-blB^izjp z`VzFd73YT?A!mPtS~Y&RIGocz?0Jh-5K}3s4DG84TT3J1=~Bc08YIBwb1TZY1P{sQc-h65%TvS*oMDRgSC3uV`?m($Bf^W2Hl^v6far z-;MM~@aPMuMH#vb8xt=FhAxX%R3AvC`YTql8!X*$IOwXQV{JOx^oz>y*HO(dN?%rB zou^wIj%w1b3WIr2?9LI|2x?w~4)dOw8t2L;jdNspSH8tqh?;R`II}f)oDCkY!{IC) zcG}=@rVVd{!?yOqw)VowI(^uo!%n9Thi$OaWy9NGr`v|N!OkQb-Ud58HoOgX+WGhD z<(;WI9JayEG#lOqJJW4=J8U1XY`wg5f(|EYKZ@T3{6v+v@fXFvy*?Xk=PzvQPuSjH z$h-2|%88aoJUrepGd?bpQY2bigM654!)F{QnDVKx=<&{Ap&V>3=fW zzj3=@T>B3j9>V5>3?HcCbswb3KgRfzee@eXk}eu(KF;)k28XuxO&_S>M`%CB_-pOQ z@4r&N>Z8s6D|o&AME0+>KkEKF{Z1BkA_iPp`|rB{{QYPY)ok)N`mf^k{=Yxt@03?O z?Cl%5#fLCV!AK3WobJY0$&-Bo`e>UdW z$_9m_o?!)TCBD+GccxmqEG{>#d6o5m1M2~|u^s?NJXv_!VX^Qb1Fy$1H^mWD%M;(B zm^Fj175JU_(oU>ipnVnZ^1E2rg+{ET;gkGHES#kD(njy`dsx^LpFY{2%)-g>>4hBQ z(~Gr8e0t%BNB9RpB(u)b}(%`*yx@B*{TMTKWgb7c*jfuy;D3=y` z8xtQ+64HWeSMaJdLY`RCNThdh^-AiIU8fX1@nRn2iKR<6eM(Ww#nC575E3vX7*ijd z9#eub@nNqi!Pxi|u8#~7RPd?<-$#9lVTGg9qm=#>Wt=Uma;9=AkX6OTr&dZLt{ELidDwgd}wSw zgxf8YmFn1>+P;prRy5(Ubs6m+jIGN?+6x+6+tUuo*xG)i{lu~TA-;i@bu#(q3uFuI-lu{rf$)|?u<@eaD);pz7%`^d_yE;;O?>m9h%9UJczyZ!NW z5Vtq*>m|6lq|mE#>exCYuI^Z}V##=xh2P74j-K$4_dE zJ-E7LykU)I^>qEsd>^6mw2y<)(>#^wl^*(4ET5Q4KLa+8%iJ7aNk{c3>7vR}zFJN> zr{{8buX?dTffc&mD|&9ly36ITV>#R+p2nCcu1z^`^@c7}8CR;P2H^8EJVUG!XWwOh zW?UcSC{_Yc;I|;d<*?(axN}(so`vy=n_Tc13_U6&-X$vPco+1z)kCI|chx?GB2(?V zqQ@HwWa@cW^vt;1oSq4K7yY;RzogMS{kHLmlE0J7KV-J=!?#EI`10|)VtP{xHf!{* zgzeK;kiw3BQ~v0Q(*BjwTm3(&(jQK!e_^42ezW~^?~0z8^jWwY*p%Pl*zz;+uGo0h zKE(L+o8#k0{lHH#>}Q;ipNa1jc%P29?CWSue25;}o3yjt@iN*Ta5PoCOs2) zTl*o?7xu@j-}04sy=RR2nOT32z-Q_FS^9fM^shw6<5oABd~RO9jo#vav%uT=H`_lc z@OJr__;Uhp8y~Cxp_uJk_^hN@{kQnT8ObLuf1Ltv&R81&lxLu2j32aj-nRSfz4KQ8}}_AI#% zbMleoN2}bC@cfQ=P5vPl-Dys?$J40k;r^5-0!H8d82y<1tfz{vM1RgC)63urq{7lU zCA?Gq5FInq?+Jy9m^X4ysL0Dt@g_Q?a#`Mf=%lyUBe}8n$=rSbq5}81i2{LaqBp)E zPv?}Vy-+w*WN*$JM)Ns@$uf=VQOd(N2Z@fp>_n+k9GKDmW(m`Q%Dvz1os;$V%l`T$ zEDr94UPjMER^jKy*O!br7o8jPA38Uln{;O@=X>0Yy6Rr)X7|!@w~syW+59KJ{Mm_Z z>}zxX@TZ-wf0{i%ujF56KKn>!`r6jig(x#wJjGth@J)+L<4t%GzJsR1C2Rn2o{3+` z&S`Kldr^ZIurnGwpFO9+^T^dH_046cGsli3;mgM>V*RJ+8sm;yY|={RQkk z4W32s$XD9WXL$Lc0_U+#4fe5S4bH`P{M7O@*-j09+2_F}*=qIsr8jSBEY48O09AIm_v)L&PKb{@a;4FqksZu_Z z^=SAEhWMzoKMvo$RN-`1tHEh(vj(TKHGnOCQkWmGrEgxgOoNkY7fUJcVLrfC|C8{Y zS{24jSA$)&$9Bo#H1waQ$SO>8Y88e#33!v6+oYoc0{~ayz$+tM(<98e-pO1wE{A(j zI=gfk@0xjsv0D7xb7pWaLY^d6au>}YO=O#4+*npIhMiXSHe*Ne>%rSQo$Er(y}Z){ zX^D?vU|zr+1teV#(>#{o9tqRDg5Z-9rgJdyqg_eCtw;6l4sV<0;af2@KNX%)kot=@VV_ezW;A*_*xsj$%en(hVQW9 zyKQ*9OIBmfr~mxtp6SM! zNEGwv>>YW$ERXaa=~j-Rk1sg*e3s%Jr5Uz-zl7JLJxcV|sx)!R|NcEtcl-={Ol}+pUFaS^pW*EvlR48o-ahc?m?<~dRw%(A4rvFM+bbhpwRP)T-r`bHmyCMp>pA8wWeREDm38tfR7+X;c% z>!ix=lS8q_hO~i<^iHy?9$@eaJu+ zQT#tdACpG%@xPXi81%+CK7xuV&o9dvhm(@tfQ|MII%E6~fOsVOpsaxCshpjj{0NBv z=B1@jt^h;d5=Ww^eTt9(&A$U6raG(49C`BZ@<@%!#Qn1ZQ#Sc4qNn<`B}i+16WcuV~#?x1(|YuAN&JHMH%xb>HG!I_fH_HdJk|Y;I^<-MxF~mez*O zt=n4H-ci}OcuPmc+B-JwT%_Rkb$8acZ3W$y^%c8z+zDU0l+%vl4AIq5SS!ckftyXMNet)ZU_9ZF^VUUa_&Pp|tMSx{5Zl z-MWfJ9V<)s-cnb&QI=U<-deqPU32G_t*vWHZ@syC@5Z`{>(4MGV=l;eWi}o~DFAq?^WAeB5j-4c@`s&qpHXK~svTNtY)}5tW?yRp; z<+goYefuqUZlG~oBYEAi`ryv8#@KaM@4e&p3QkkHah*j|RSlZb-CetO#HA~j@?5iP z=ej-6pZ#^vnOHL0&@TCC-&oSnw!OJNrfjP=tgha;Xm|Ch{kz&)_wTO84|=hC&AQIU zw(SR@lWT8TwEfP;HS1b;?I=}rV*9?j9gFwW#jMjBuc{qeI_q1Pm+acHu4{LqvTeL< zN8OGs`*!bGl3;vA`(!V%`florXg`tOt)qVLEZcoxSLy1Om^x6s_n?&fYF@UqDFJ_t zTt1L{yzw{eLbN^dm>X@cX=q!1$L{URZ*NG{@Ah>E8mhM^9G}WEQEpfJ)&%|8eXt(> zY-d?@f^x00T;+lK(t`=gcFM9FZ{M{e*cwOfrkrHoA=gETWV)(h4QwNhzHKt>b7v)N ztu+{@UBk{+Q(bjC_Q&qa%4X^RcHhy^x;)SbKONIg+RMfBOM1Dk`qDW5%PiMS?FJin ztlm%lKb9?+{cT5o%PJ4V(;c&18I>cw*pZ+w+eO=PbkVGj%elKOft;(koOiCivT{4F zw7$xCx?+s4YpB!8)v^TZJ>Vw3Hu5765?7E}n&O2M1On9q+yB3u$tFI}mY3x4O+0?)VAq7%j2R<6R zT7hV3-+OCId-t-M?zXy`Jsp9zx^A)~rAE@$@G$ z{x<9Hvj2YKKjeINO46EXNo!|tD&p;D3jMc@Wz=mM=N$3qQJ$!AS-?yyjkA>*S)X$L zgPZX46}%}E1;1Xw7uo2EuT~ZBmIZ1@=og9*-IMQvPSzfE=to^{_MdXsTu0;1Y&`>z z=x203vE)$pmxopeCQqYZ4rTX8UB9{b3RQ37K3=sU3>_D1|5}u%6>D^IDeW3qQJ$|< zjVyi?`O0RF#EN{{nbOmP0J9bO82wr%^9?%^@?z|Np!q>c6h9O+BS%lzco(&i^wN_&Y^uuhJ9WWho4_!V}#1nJ{Kj!K3Bvk-$-S6N&OGy z#mc9952*f-L>l&sJryB0+O$D3XiITUKiaU~tgleyK7&sGstCBz%!mF?Je9$`8r?-yg{Q-^=_p z($C3e&Hmw1p1DUwBoieaF41ZEB{F{+@(tNg)@?F>I`X9`c+MduUv`=29FzIfu4I+x z82k3ruA09Ma-c-@8@v%rdQQn+PqWOQX{)DGsYksU)T88MU)4;RX@_WlmYe0x1_$b5T$-&X2T`JW-}it1PUyG+SfV>8_wWb{kT zH`@c$HknWLOObGrTA6RR$B@jY z`c->8spQK`YP_D8m3--Hc>dc`38)^GpZQX__I#r+xWxlr(Q2FG6eT0y&mqQ|<!CGoPrh3bKMAD-JZZ))jX*1-sdmu2`s|nwIvKZpP0VTk2Yy z8oFz+ZfOd1*L3Y~XlUx{V#W3NGpZ`v_qPdVTbjC9;)Ro3RJ643>9~T@2buz1EgkJr z3@!Bm$fUZdTZn=cw|8_m6<2S(bwO8mUBf<6YBglFwW+(JyF1WQzrVYQaz%L%YPpSd zE1M3qG&E`XLb6TGHG2YeZB19e;vTfu4IXRSsr>r&+c&HfqTSHZxW5(6U*U?9k4kGt zU1LpmM@=JVOmqnB2|fiV`l9v_YbUFKxmH@$-q6w5w5_GJsd4l?u!6=b=z3jXZ&yth z7JaO^t2@wLSC8YtHhv6rG-6f9iuqrm#A5W;qldKMyJDXH`d8Xlt^t3A+X%&Cq>CH6 zx)&`my_dA|7#kxJW2&ibN#FSwu&E$J!h1S=193-AxwLxvWkC42oaY!w84e7P(LMv^ zsdEKF`sokSPIWQ@SdGbqpN$_)$^7`K>G6G{rDbxyM$>c(k;WXrm~(v@cq%EwxLL(f z*{;1k12&FyKNZna-D)~1>5cWZfiv1S;1EtI&o0Aby9NLD@^<(qvVA-KNlCxn*oTn~ z8hlHTEkHA(B_l^1-)@SP8(sEZNpJ5z+au^{ovJ1~|MvhNYy8f12*OK}n7TnzQG?zX Zztdpr7WSpJMIGWAb$QvKxIGDkjY}O}}b3$Z>NPs=K<8>f0 z?*0CHbl-c`Ai$p3u+*V-Z~gyYRsXB%|Es#W_sh#xF4r}UsiLu0SiGj7jAi%4@4fui z%hEyVX5H2Ow%VWPx^4fPK4|;QG{z1O+H!l}!RMSUclN*HdLEzO;2urKD_Xi<+x3I+ z@5ZpJA8ydYA5_}H*KF+Y|FyB+qwVZ4XpeMfvBK}=diy>sVQ>9PXNRYycUPaaP3#@C zjiS4^*5#FtFpjrw3M^etO`Wy7ww*pTXzwtdrl4 z^qx2#dDQux$ngUwBExe}MtbL-itJx~I&x|M>B#W@_aaZteK+z@%lXLhQ}0I7rav4W z-f>{ysi&VFNT0hnJgfOTkqdKlqHr{+FA5I%Jxasi*F z_uh=mK6x@y?tCwDVeWa*EDg_Ud19a!x+p(?9DSdNgioA`R5(vZ!gDW0o?1;jFA0ZF z92l5=@`-^kc)l?A+mU^9-vnRK17z>TCw0~r!sXEAg@-@`y`?38I}(2P%}CnTlaX2H zFM_X&kulRB2@ex~Ie2~aWF$4^H)!`~Xt#f$9I$=KCnLiTo{ywzxmSLY{MG8=i!TLJ z_vC&*HMzSw_0`-E+0kobg$~$A)4Qwd;8PPc@71%m3B82dqO-!7{ectqp8)?7KCne& zg|EX0G>vc*dIxpKqtxywV}-8*mdD$OzJ3Tl`Jl2S{NaXT_?`FawOy|rGx~2LfBe9v z%~n21{-=DeAHMereDj{~-Pyj^eFNV+I*RWdwfG+Sr1HOhi~lKqy^#-I+x5e6v}EF2 zLl5YOFZ^@yRkxjGT>Mq>)!~r2P{I6H$~9r~eu^>z65U@p6Qwq(NL!FuiR z*}v1T4ee)rG>@3p|Hum9qtCLw=)bVOORwlx-zi~z=k8-y|8*hjJDUPxhA(?DB< zxi+*nsc-0?S>MnD+Tn};L%(|Q9d>x=ZY$aCTEmh}xozH%woFnAsY&%@xEWF+}6g3mGN-v^nTf3Jn#g&f1LB!!0| z$GecjC_l+C{7Q0o81g_DeHX^D!+cA9x~NH74VV_8(V{)R(b8cC+;P-Y|_%^wTtNA0Hzv@fg=Kre68DQsX}6%JyDO zo8t|qFDeQy#wbqzihxbMdG0InzS`Cqs{ zjqAVB_^{VI&nAU!u&eSv+gNt>9q8XRT@SlppNsHYveU0&1G4!A*!=85);9?IksXvz zzrU9CMM_v7)B7JGn=1QX{uXQnUw#*U{2pxfZ?Iq7AK2l`hhSHO9UG?#eUP7iTv-&p zmQs9$>~~KeW_>&d=Yb#d5B<5-deyr%|EMt+yIhY>P;{ng6yQU$POugDIMDeHT)p_D z;FHN*H0+47!yjz*H@CAYVDbs8W9I4Uxau0SjkRgIjWPQeEP1lnoXn2fFN?gkNFj9g zKZQ(^Y)hTEIr|^QRo}sqy=)0f0!NdGa$b95dq=RtSBE=w?P?Kqeywe9Y;6d!d97_7 z{&}sB&Fk3F)Vj4TUa#+z^~dJLZtB}w0=^D5ufreem?uQ5t?%q;_O-Xy`kI@!`8pc? z!Ppm0I$f+b683p^gr zLfqCaSg^?RaKYjwwY46|Ddf5f#At8y1^o?ok03#RyFd6CB!ZQK{^q;F+aQPUt`MZb zU)Q;DADR{=0zN zaDHJeQIvcMqi~$h_A_({LREZ@ z;6wG*>&r{ai`*sH|Kl7*6gng!(#7~}B~YLv1Ya!aZ1B11bfKx)*U{A0>h5T3Yi^(8 z#?1En+Wqc=h4U88aj&P#yuzidxU;Fb!QI}~-r;X?w{*65xEuUin_B(ub(NJX*Ve9I zR=u{oa&_&pjq8@JE?rj2=2p5Fy65_W^^NZK7Muq>>~mvi;+4)<@coB{*B$H$!0q!L zj6A_Z_nG1zx6GauzrUH|99v^Lvv0M-E@=C(UG#k)jn3eKI2pQ}$?URjCre*0iq&m2 zx}mN!x2aBNRYpAZNp%~#wM~vPeI<^WYHgZ*RywyS*_DcYCdP+RT&b{C7H%uR+mv(` zK=&>vIolZP*`-C24Q8;ME}<_zN%EZR_|D`gKY&XEMV*t)qeD7XK%X9D z=`?tt>avafBA4~-w@2A!0Ep=ZC)=|x+0Mt^l$*|uyV;MrgbeIS;L%@C2op(yX#8be zI@?0L=-oEK&wZ$%OvmS(lN}D)?P$TG{RcDH;Cz-JvfamiyO>4wgDLFwLu^JU$@bex zZ|vap!CZS`2nT<_Ge&sG7aAYUMJ;I}w2kp*gi`bv{mBfL862N9p86j+l)}ytJ>&E# zL=Sv^$UdIVIz|6P&!?H7NB_`=jl>C5eZcw8hkOQl@RK3!1%A*U(R2M}b!fy7>N?36 zsEXK#-;~eOH%XqU-|z$Oqtm zSwUNrJTkga;Foo2>@qQ+3IA9T)HA5cw2wJIuAuOB+pnPy^sfhP%t?4s0@*h-oev-F z2mgZ^_H^=RER=yQ&=2rgq5SEzZo4t%S=sYPfTe=*FA)8M)2~FM zTAY5Clk^vr@-6GaGC>nt`m>22_*Y@@!3Vj$G%;RbaK`%Lflt-;bh&`XB1+>o3cL$-fltzfJX<6_SMt9_;x*uj|A4^P&4+(I_#ny0{h$0R zWV3N!QvMd2HH*ZF<7do>)nDWDh1-MAKeQ)(+0j4e=WUx_mh&8=bu}8@%k7^H`^F0B zi&UNL4gbJ=&gb)$=JPKySn>0kASeF?zU=v*$>yRyvOhCD{57BK$^8}Mk$-I=5;-5T zLer;H{Nw)kbLxMEssEl`Q{(p zpf1{zKiYwp{s#Ysey7LutL#k-kiFj~ebYUP)HjW1SN=Iz0{wIdyqwP?^)Jz~J@h}F z)UEu3^v}PMB0R<`dwz+jzn)!LaedddTJ%lx+n{egU$eNrv!wpu|5-78*X3fpa9Q;2 za>eysN&0fd=Qp1pXfO4R{`zAL?t^P8a;Qv3a?!Z4ke`i|U(&|cCz&50f5ZHlOZ=#U`thBx-{SyKJ^=rr`1>+B zB(?v5VLDzl03Yq{3B>{@b&q>tNba|_BNI?H6GS8C$^~`P zfB3(7e$MCg+@3TipwGh46rGk^M*2JRUupfClFTkgd44lE75P0HMT#=$&1mSr~QT_mD`K_hwo3^oIZ!>fzJ=6I%58W{5CV_&T*snDAAKY z?RN0>5#wbLJ^U%#=x_O<9JjH3;cmB^<{PKy{%CW+I>H|@KDCee5BAL_cL~sZfV_pF?Ce-Pv+_gy_h7v1^F!+w0y+^0|2F&kA(~Id`GEDu zG#}x)R8Vwha}Gh+P^uH;`$~3NKLZnH=CBKK%IyO;1}9Z^Rp83I|+3tJU0K6 zGvoLS;8RdH;$1lPQ0)nyiMkP=HO`D5hk6_jdAYxGeL!BSyTF0aj}7xVf0TTjPc9K( zE>xQ1*^9^@ce8ZzZ}`&~_C_W7Q#!Q>lm_e~o_rBCDu(r92vr^rrjq^ApF>r|bJBM* zwj;)Po}5hmt9YIi60?a_V-(~Md%74JpwRgU_Fsp(u+)m8)BYU$V{wQ{`@=W5zms;N zfnvFyr7n`>g1^k5o=ztw(QdN;A#HgqUl918hPX2@K0f@b2mENqANYuO!s|(l^Fx5w zF<*8&a8!U|!+LQF@Qf+Fo_J*g<%6BNA36Q{)e^CCO`V12m68gHGe8%&s<}(4# zz0xnwXJY;r&u94g#JLRn4f7Yox9jFFe-8gceEY2Pmy!93BlTq;(od0(V7|E=^%wCX2kSMK6A|yBKht`S_1Uz3qn?BHI!2H7SdZfStwGJa zUSqxzKDJ(uny>KrZOK|IYhwQ~W|cr3d-953%nb2p@}Q;6D@bjP^s(Y?0sHdOS1j zH_Y)1Ie42 zer6(lAH&z@bNIe-*ZBDU{6W4y=jW$9-+p60=qDQXF42u0K+PhGX?MIQXjWeD%0iR_&k0SrirhHTEKlyyfj_p7B z{+s*>B^K`{PGoM>DPKK5ANq9Z27SW*nYu-vrv1q1`RgBXzF$f*=dZ|b%l6$`d_#U@ z_t(FHq9#%?-)jgYP?sI?5G!~7bnxFDBWw<$~I3Fynq@l6r9Gm!ioVY8Zkx{_52X=#K*Yu zDj1=@=p)*3-o{vYLo*h~?@hDfDc#YRI2T-DTkw4ney5s-1hGd;2FtOM2p?0dD{*_u z<#}-@#v}O`1q>wWpXd5S;Z2MexfbwHOrrKN{AHO zsfeBpI!*Cf{DD((Tar!oj<6*}X54NDz3VAL?=iV)qy+ew+^)p1nh+toTHIesISMU@9s zX!Q*0xrRF7sm_(DjY&s_M>_zUHVtg7(eykQVzYM-=~zQx#!8Pr@m&GsaI#mo>IHS` zbsFl_73rEr%~SRGKEtNdC`IY(>_x7}(r#`v%0))gO`=?dIK7ozzzuc{I@S@&0ErjU z2|476@1w(T?6y@Ibp-xhiC27wMMtuhb9CY3CVap6eVo|-$NL|7GR`G1^sj5Y|8B(_ z{qD@zzr-sv|8H(njIOA?dM+JDo^M77m&e)bR{NHkXuJ`;OQV`jKu8E%QL-1eH9~2u-f?_|d zLtSb25y1JuJg2(;pIN*Qel%nvTFT)gdl1j*;%;rC7tiU+GQp_wB&L*&i6^}nk>WXB z@qU!QsONMkS~TZHJf|yLfn(R$bGjr%(Z>jG^M+NXr?Q=ywJECruQGY(VQBjiuO88X|jG`=oYNa>6%K#ZauY@i)~a{%A8PNlu+bn*?gowuc^>g@ZC_4E{5(@!;(xy)+viJ=(y_HY z?dP6I(EgaEz3SQ8ewt87z^e%4^^|P?fvnT`^ji~ad-8j9F@pboIYr!t_H?V-D?1)S z#{}~d(ooJ?rkEOEwNv(g42^HpUeQr|qO-QAc}^;@xG7`gm?+(O;!5p_rdw2FfAL=i zc%LOPh`d=MuQaGE{6D%H>ID8SypDLH{!K-;_nLqfCj*Yw7vjIzLWf8A^YSx8$BFD0 zfMeqS*OC3C$BTAn|5RKT-7l^=xRCrNBGmr_fHXde2{nF`08+=sB57lwb!fjg!T4X1 z?eirijZej*KIWJ0y_WHh$@a>BsXe*0wf!PY-lr0b|MkrvMsf4Mj9JH5epG?>J!tQ> zjNdQYEB{mDt9A-L8GuCX-%ik;ri8UUy(|h*Cjc{>^nY2lm+Z$bR{gI<`~RFke%1c6 zYza5z_P35tK0!L&QYQ&5muxwpx~ldBzTI{_ip%+F87?+o)x0&1N4%-2iC4@F5kJlenBH(YikH!+Prklri~SibsN@u zHgBw0>Dy4aqq(B8X6??h<;}|@#^gj8>`zlZCw5M%9hmy4P`Z*4J(>=VvJVs zQMSHpeP{iGrCl^$!-k5s;_{uPbqm*fn#;CT#KtY#w%S*;v~5}0wsn%GY*Xv{mW>P7 zZ@($J1qGwgE#KS{YTUfA68dN}2 zeCNhZ;JdDD`Q!Cng-wvR8M=DBuB6c8-&h=|Yw^QA>nrM7*F3(8Y_MW|ldov)s^Zqw zp8A&cjdh!=+e$Y1YH9-2OPfm?+e$ZVtMV0>tXf{Sx?pqJvd&E{&7GUe@L9O3bMuOd zKts#=F6gg(O~Lxd8&*^_Z`!Z`_O342)Vj8&akH@3_Tq}I=-bn@l8#O z9#Q_LuC*=AO~Kk7i?{lh)O$ROJoQWc{)PUBJwDIkhnM(0i+o#`6f9l3v{um7H*NLR z`>*$km392xD)d$ryzS(6DxMoWo4jD*qD6}qdOUc23SOhK+}GUhmw!uY@Io2Cuf-Gc zFt#i2j(&V{-k-R4ed$XuPQknaZX&f9YmZkmZTD1dN_s9Hd*dGS zNm_8JtV_S&`>cM%{$G;+(($$QZ#Z8NJrjKcClG&jk3vsl9Wl_qrtVibkB<3<{(@cB ziS7pqhwCYIZ=st4_&J5cwN%}gI$h7|->?&&|Hoemr+#@c;PVZ^Xr={E^rZ%T?5#*E zoiC4}0bUR*EzvdK1fA;MdVLXX?d*MZBw<3!2gjLULJp;&hL zhXYi6ph}|WJ+I?kqfvsA=sDAZo=3zHEo_Ey6;ja0`7y)gPDlZ974*iPw?MYxl4aPr zDY+<25?uMe&=+Ihgj@$SO;6DJI z_&LMvz?hm>+I9GU9*GyxS4k7RCgq?p6OH?Zl-DgypypQ-=&7H}60X`VkZ`q0EtUP} z$$qM&$&{Y|LGq8MOYx%S)w7b%mnB@yr`KdZ{nCI6z5(z|H7I&ogE|#F#h1gk$ofAC zF#8s+MDM{kR8EQudka^BUy*RTge&>Rp+Dt2gc>jL6D)9#giiuowI$FOC44gA#x3J+ z1KgM@9+Pky7pIA_a$MfgxJG!Xl#}{bybgYu!mZ<`%Xv%V8t02k!bi*(38#J(FC@Sg16&^QaJh~c;IaSQ1I!{h zn3C7csTuo%0dAD5QNooT6n(RVTjhOG!maX_%8;YR^~iBo8sNrpLlUmWRrI?h+&b>} zB-}bKL&OcTEYQ3Kc z{-`K;x3u313tX-D0SjEM_e`YS?X`7Ht-fFv!yci*x6;?yTHp96f92#dJdhc0cjWJQ zqSrODo85U(N3~6@O?b^?U3>ejHEZo`5ezl?+u7)w9JzJiua_L1`Z0g79j}IvqBhk9 zA$OU-Lx_tPm(Y7G%U0LS#rx#ycL=KG(8pSTM^OjfKUvq=;U}n|M?({|!B^^ktf}5_ zf(z~W8|CZnMrUy=#-rC))V9*=@m8)}zpAv!*X*zFSk>0h*^J?Da`(q=(A?&0sO@O0 zZQzUvfxw>NQ&fd4MhTa<1r@=qj>YmDZo+21;I{VK_JF^Bb+6V%Bv|4>M|OsguReLlftS`dSK<=q(>#}q=vD^;I?8%nvn3Lvk6o#N&> znUB$TYZUKfIF99D{{SG3Z_KT!rAr21FI5ahjjyiH%J$T^u~ezx-C%^q$5&876Hith zR<#NW#jm2b*8fV5f4%k}SrCeE)n6urx7vhoD}#!{e6`YUm+eQ$FWWz3X<+5w4}2o~ pIi&wwmcx_%$Q4w3l|%2xz}B&;$=P=Lix-be`H2mBl9#pp{{vcy7(f63 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k71abcbc4b33e413259fa590cad8fae89.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k71abcbc4b33e413259fa590cad8fae89.o new file mode 100644 index 0000000000000000000000000000000000000000..423bdb1627a151e1ad642ea7b9b3e6be0fe58a7c GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yRsARA`lnbre@WI#e7UYl9bk{Tq?qfbjR zEkjO($U4SgFk^OOpN+F*;>pr7WSpJMIGWAb$P*vpIGDkjY}O}}b3$Z>NPrpK@j8$g z_kMppy6?Se5MWPiSn8^Oc&>rr|U`5}{_Vj;P%HIB!&JN8;?Ws9qo7^{K zn@D{f?n!4w|3~-q52Kx~+1{?h*dr&O8LT|Fe{lPyX9icCdS>vOy5}lrpT+CYoD<&; z_Z>eLe$4ru@Ui{J!z1%gg!|^74DVZYDtvL@sqo0YOW~*IzZZU}^<4Pa$@ju3Gam_! zY~Mfl^fS*4rp|vjG^gb|;q&v3h4)_iY4~a9TjA#87sGukE`hEn6gqHVaNF_ygL~&6 z3s;^x9&T^m^{^!#TALnn`i&*Rm6 z>8%lY@QGMa8*e==*p$bo^wv(s?Qzntw6;^eW!fKLi@+EhX{W;m~_;g;Tbi2+uip z0eoEukDK{uXoT>~!0Tft!pXlmhjxF4cKZe^0Na~*B0TaSU|RN-pCo>@X5_-lf#luU z-%n2LsY!kzRS z@BvLD+@!uC-SHT;JHlAeYk=kOHlnW|!cRV^Dh++Oz65^fxq5Bq>qm|L8_yprRF+a}CVpoTMs9${pem#`UuAV8KbZDqSJ9Or6 z^=rfXSU=4prVTv09Qep{tUvNEtpDPx`qg(!S^wGl*wue6Wc}yT*wyn%>}m>Vi!s-R z_ayWW|1;|!en2~P;eYB^FTBeR4gXC-X!u*~(1ix9sS{ZL$d4uimKYj-fF1hH1EBry zNue|AcRxyV4Ll#Zw(~X9ToUWMkmmyUCs~HUD~#MP|LxNL5y)2|eER)0 ztUp}J`k6lPDA`om|MItBEBNwz@Z(Fc*}uVlF@Io(E+2$l4R&mrF7!cu`f*ip=vq?A z6|&zweVFy}Je&u9%wP0&tM%&lYX3oFE_S&AouKGU(#cKQ*qTbW*cwQbQ@##aai(Xu{oI?w_g@{ZLvb=?0*TF zB-!SAadYNBh^xMxC3@ImmH>{X5aoi7rjE`)r?(z=>e}5Z>ik;Q(bU!$WDDBbJADh< z9$(P8y}4~md#v8jCF_qbh~6}`xB9)EY(b|l*ttN6R@czg+2ZZ!sPne8Z1r|F`2uwf z-j2>XiS6{Yb+iZkoq^`Uf~~E-R&Pu5*0%hI+w<%GeK8oIF2AUtu3+&ab@_RDd4;&G zD=1i$_elQ3i|gw0Ag7S)E)b)m$s6!B-aUc@d>y{PvvO#|rQ+FhMZQ8fa+PU$()k1Jo$a07el#asds|1Rx2-epE^yxlufNq7 zXt*oP4AXE~thVGpxpZ;^8Uc5K4V5@%pXJcIW(q6Ya)GXAU>Q`5Yq4tAA@T^BiQM>d zFAceuhTLgsVI0@Lmxg@KvUk(6Mb@zbZaxjU^Y<}okzRuh)`IpeTRME5d91Fk?eV%O zX64eAbplhDhpk$(w}VITD4tp(f4Qr2vKvLFIY8R5Teg-Z9G++ev%lc*@uT1yuvxe9 z9lIIz6o=!a&{AFkJi`_1rzb}c)Pt27xb;YSknyq;aQBh4C|rWs*Co9gTG4+Oa2w7q ztR;eyD}i)R1$^(KYg+2=o}Ssv5!u}{+re)cBltXPC$g@1wtpP}(bj&N4ne4j*I~S< zzG_`XSw*qC6#IXiqliL>Bt-f!URwy{?+n5hOS>AqZaQ6PZt-?Dx3{@F+uK_@=D9Jm zecleAJHK$jqIvFhbXicegq3tPw=}vtx;r|3t?t&Yj!t)@Z%cEV&%L&)YQ>tmbxUj3 zR8+01Te@NG(p6!XccFW}H_*`J?r6n%z$0Eah9+L=dSa{veo&?;!(80(P z^5{NY+~bznGh+8QbDU#qN@eyfcGv}NAF+$R@1xNfJP;>Cm(!VD*6n2J%f->UjYc=r zb>=qJ>8#3#r#`7}L$|i65vH%eF;ksQv(HK8HYK}KvCqc%Fp4V~w#vY5C3u^b%KYfw z1xKvL%kBaV42D1cvQHOYX3JX3>?saL-rZxv{lgNz9YTU~WT2rmmenov#V1LQlO5ZU7~uzSDWIr#vITTVrwZs(11yyW z_g7!Gv0r4f-hK86y9@v^-RNYy_a@r;xEr%m*)cc!QMZtRJq0}a>kVQeNf3>{v|DGJ zi5I=cCiuAz6_lBHoprK90lOV7SY+Tp8XH>3a)Y+}X8+^EETSJsVs9K|vw{h>-%fdR zJFgFA+lzv_jqr>S9`XezMzT>$m<(-Wyjj5{JxYHfjim=BCQPLM2M;E(vqaB0eG<_F zpBuDKq_a-ZKhg7PCg{;W^kE}${M8?D{&OLpfgb#%Nqd1G^hfkue;J(`@q@Zf^7*U7 zHsUuaI0;td_DxWDK_6y>cLki2h~LyRkt8;l#)|$D{iDw1eUta+Ohe6IkN(jAQ+rv2 z>|uAno8CvF>(aciC;EpyoZQLD9^j`az$Q-Ol`Pf8&B250i&BLm=n zD9xTq{)~mvzZv=gJ|mbrlh#f67jK^%ggJ(5waFykU^Xj%;V`gNF#dU>e{klNNJNX# z&vBCeB2vDk-B>1QVoQHE@dN)V3Ox8Ax0fcyD+n~|DgY)Llc=L!q*k^aQV;-Mx_fJn{f8}H|gL7vQelEu^WKVXpxun{}`?2T@ z7lYmodaP|kf7HqGcGLyF6nHoAZWG>(y1*|5ej4z30zWL;PeWbczY2UN@E(DmoyPmm zL|x#MwcXt=;IW9(_)P-uLS5h!bRo}XiO-h&Zs}C3|vz#dzdjn~6luhm7FN znH2xHKmMHhUvBEZcjxq&KMftg`j%iLf8&e7sfF+_toKp>BLA75K;up4{&SG*IX&uM zLr$$IFfAeKuUHRe2B%FU`ox&OP0OS@_qRQ0??QX!Z>vbZ(_AKha{%ua_B=YB;~l7r z_T-Ot;HAI8zoFlmQT-}=69Z)LcSzrKk0SL=?Pl#jHGQwn@IR^wql) z`M?=8q4g{0&mh0e;BO?q!{Tpt9zX3Q|4CZU>>_`l^`V~DOVM7%x0yi)r+38s&B66S z{&rT=NAoAVm9NK`f6HM%QN;YS#zV62X(Ev8cTsTsc(OkqAMv6*IhWT|Jm}qtrC-G_ zKgp-Zj33)8>T{pz{>^SipIWU<->Q@hB#^UwRLT!1#PLbOX-*uhr%i}xpANVsq zIwpVaNuu?&JXr}K>}mEt(!b3ZpY6wXAm32()lPCDb(k&xC3H2D> zvYzqv&F+%%WK93AVpy-afA}}k{sjK9mnJ6_A64J9I%e?M~qMHqyG3|8hbX(kt}j1l1BU$ z^@3isPojJW`Lu1|N$|?ow=@^>Gk3H<$e$jVu1^7d#D)1hT$P$0$kwM)eKIBx#>aR$ zS*Y{*MlMgEjZE&OGEyND-WM2(7Jy%d;+^n%0^|G;;0?@| zT@D--pxCfpoW%IMggyNA?~h}@quIw`&rL3E`z_WV-AXk;BTDVkD2lv%4hr}Fy$YVzb8M-goT8@ZYQ6~`&9E8Kj&WQ zm*+E4|BK}_{CwhUn*E0P3*y^#^OwJX{~^A8*7?ime8rLciWljp$VV{WTn_R_e!c?# zH0LV|fj6$tv|d^B6~p?Ac#(zm8q0}@_t2kdJ;(ZNTE9`x!g?L0M|-SCvHjMNW?ru` zUkM*wugA<+`24oyE3_G>BK(o>CzSnRPnzjGMy%3)-jJ4n1#l=zU6OYeyq(+|?#l@Eiswla(;So!6$R}St`m5^ zi}g>O7adv8;>9z>i}Ziz{kJK8A)nF%{L_c%=MRLB#xwArj(A4+@NB-nesOY=8bB-=FjIQ=V_Xxe)Y|4g2wf_`JbBneWHBfBZ+WcHzD?o&J~e)PL;RRBmG8gQetdL%GoAnB2B%K3#JAY_k9B{G`Fj)l z?J36iRur6|Gkkz0_J=e-GAB%gg~a|l8~g993H14#VgEgi_KSvi=cEluZ2#?ajAEdD_f*-M;MF%+lDewL$mY?AB15d?j_cU-%1NSs=PXnL22I%k{p>_b@a$r5bxb}w+e-!!Z%YD_q+w^z;@bk)X zf8F@e`M-%AEMl|!p6+$ky8Gtl9PR64tlwVy+{pX?QWyCeN2RPz_^ZC0-oCyRcKXM! z{^9qLckFCw?dlmv_0_crt81|vK}i4UY>Mmo^uVKSpC~->@gmm(9*T+AKFa^*nC_Jg;?YOt@iVqs(~@;)Itt?uvL_YM zvq7gx9*aM4DsD@%$(~WRgvgBB&7gNZN$5Q)H;ohrAC=n`A0L(5E#!{ZM^r8_Lb58! zFeHjWGW(|@nSHcJktIx=Nt!5UknTpDjh=&M&TYDt^SCl+Gxx=S~Z{sJ&wW?H>88ry1w#3%sn^vpkCUX^=cj)u&NmO5wPNVJw?xPB>tYS z6xt~7qOsMAVqGy}JY7$Ns^9o4P^|qP-72%oFm62CMm-aB@$kwD)X!+C4I_`zCK?#E zofwZZJ+;Ppg>QasN!pcO#_+HcUT8QJ#0&gWhlT@;FRMDf(PFAUg}P#r=rO2Hbr?;= zAJplVFMYs+4HNx4mil7NjQ3dT)s{MqNB#390{w3gC)xz!cfhU*XL))jg`Bx>c*MaD`OD%GT~O_=Qs4LcOTN$S`gBxzK!sLM zqn>T36Q1f^ncBEiWO%d#uxT^E#%fK^aUnK)c9V`Z1ZJ%Cc#G`{D2J21x>YZzQ?D~n zr>;oXG-{5j$MzXEokl56U289PJ)UxNqfst0mTuzZD#YonkVZ(y zqq5jIxt_^;EqNaGW3;{)!7wtjj_d}B0e>&nLm%)l(;EF*tdXC_n4ewLjK}?F$BosV zW7Ul_>|P8PYsgqS`=ofI&hN!=^C~W4g^CZO;|}dHF>0)7tR-GK8bSLjsu%G&oI(%_za^(s=4Lbvs?i1kqrbogA)(=AfxKC zDCNyO|0nLVeKxt-(=X<{^xYRHwzBWd{q65I*#3Ifycu~vnf~L0X{i-0DGM>==ivE2 z)sWhaf2zyNg*__46Q2L`$}+|_m*aoS@w(pge@H=O3%bU8whzI7NpC1NoCHO`T8FyQ z?xTS7gLzJM{kJ*14}LUcB3i=Xqk9m~>Edo}q8HEU$}qvG@)V|&jfp3{7?I*RU9o0f?;(qE4HDcYVLY_ywTiaI<5?{|0m)QGPWcyqRQaZM_r~TZM zaoQiXv{yY_+s_aR@p}}3yq=WpKah1ApT28iZBKrWE=KV`Ag74i(4KBpdu7Li=$K$0 zLK?~$%M?@Nt9Hu%kE8L8+ABJ0PjuGyG|x!|7B^*#922D{M_j2r(e#LF^d0{)!22wX zLgdY2d8I*R;s5Ars1x|R@H*m&`ZpEX-eUq}W?e_|(qd6cSC$mCbTt%~v}`O~Q@gFWWvzF8W7ozFHTey#>ua|c zt#0=gm#!>bSJKqbx~y~4hMJZJ|C+Te6+4zS6f^<0rk3z!ElXB!+)&w6zkXfbrVW)V zyz7g$w^UZuuGvw(tYz6_wN3t-H7&~vY9FiEQL&+*re#y>vi$nxqHRgbl9u`nC3(J0 zyx$cY3YK-%w=T&85B~ashikW$tlHMNp{8TwhE-3jXkC@xSYF%JxV&Wt#%Kc{89}-*H^ZeRO~3LFI<<`Qogk^I&S&aRo>zy?MutIu9Y<98{5{kZYW&0?WX7o z^2efEwy8DPw5hNP`e-dDnYU~r-Q~BJ6>MGYEm@Vnv32XxZN)X+8`l>F;*MFdW5Y)9 zU0=TJiH7c?X2{zDT|H4>T9oJ8P~xv|^}#;tD(l-;Ke3W*uzX#!w|LFUlD1WO4Xx{% z>NnN2mu~dd*7|Ffw3Ifrm#yDg?JX%?xvYFu{-*M!T^n0lx;B;LRk*Ti)ACAxW9zzZ z=&xdR{<j%@jm?W5RsN=~ zb*(MUfx7LFKhi-7Z&;!s!2}PS7|o&@YRX;(pvnWjL+Me7tCX9XU-k{ z_~e{Faqs-nmtdUy1^L`WYBAQHsA1Z!>Dsi^Y&^&XzaOa#h-kKc-u7wJ&kqPK>xbBU+z3I?i>1xc3CI7A1EBI zC)K@$ZW`cc6%N-jbzkOmJ*R)ePI&&0zYqY{w5y$k?X*J066h;n%jXf zHLtYm@c#l5FQTuOCU{-SL1V@n_f09UTbe-4uV&CwKbIw3wJnfvwMi|L{pZMjs-?-4 zp8sC*kEcuVqUP0elFwHpT+OG~Wj_PbfC|1I@N_jOdRmP-6+Fe4!#B(NKM64V7Oq5} zhjFN!5Eu3~t^~g#;dTjE@=ZX0%6AAgQQ{|A;CT{01#s1tKwp&bsel`|jJphQW2$&m z!f9NbCPvFKc}L?K;bl@z>R<6jq$?!c+Rp|FpKj?#>5KX?o)YyK`lsidsPeo4ZlphU z9rzUk+$e9Z)Vr0>*%EG*cR(7*D(@Q-Ztdr15^k0Emxg|f{C{MC8|BTE^M}SY#>l+u zzz<6}@nc-go|o{4EbsvXz0tm}OSqMPdWsGe;%ADuvTM~#~&$6aB78^;YwxEfc{?~-uqxZjg->$sG+ zP@&v~rj*hh#WgAl?%_4Yn&ef%Ror;q0$1^2zyeosVAujzORqZrRQ;>r+&b`|mmHh=abKVVKMf&8ZLSYM?s8wJ z5EnmOLce2KzN&UUeowApyP#SIeXQ|y7I)(JC+oX9eFPQsXlQ~qddqx|H#hi9aG^b4 zll-~6v02=L@#yC(>e}e%@m8!@x3a9++v02JT-n~()q>$~a`(q=(9-U0tm|yAYvhax zfxw>NQ&fd4MhTa-2Nc1rj>U2tZ^CBYz}Aks4!^Hq^zVuW+8e!{UbcY03x^a|S^-JQ zI7*N#Xz1w7fB2@gafj@MZ{E0Ze=zlSHt?o4pL{nj=3|HY?bbW=mtDmI`w8FoAc)9R z%hb~qnV?RU`G-QvWm;7k?ehsH(}F16BkxYjIHnLPUa9&l+)&EpQ2=@M?+`c7%Y2MJ zTcda&qX=u|dbSj6d*){2^>(6CWF63$SCyn3UV3_gJz7c91OP zNG3;#S|5R%YX}5L=`DSaHcm?K4-??g-^;y$CbuySzZ58;lwV%UYt!PmJYw_ds1z^( z&ik)5v-X}nI_FpdVONT_eRR&6S+i!%nzd$T&t9|kfy=jS3Wq{Uu8{h^@*zo8>d^r| zHNnXVH4&Idy>q5gcO8$^J@9Y%ycDTh@c-faWqf{#-$0#Q7mxfu^7-JaKRoiz1aV2N8AGTanO$|A?pu4tJ}&W|n7m9FI(Kzf7@k8-u|r(D3>_> z!B?NS|7axn58tm#Jaz(k;ZWr7lc@j5ZMQ%Fz%#c$|LS*dd;YeEZh!vU;lydsxD%hd z=Ku7efm;_pI545|;epq_{_w!b*Zy|k)`xyN@ZBxX4!pMY*@34%^YXwex4$-U(L=8f zw3oj=P&Q}fk*7Mo@%#f@zVUqdqKl85-1_jq(+@p3aK$&C9k})Nza2RF_16b(efTxt zH6A(gjc+{v#fNTt{+2}#4qW-{Lj!yFJv;EgZ9g42dBf`iuiQX5D~{}a__pV-eD?O| zZvl-5ZUfG3&klU?Vc>wy7hirAIBSlaeEI9o--^#k@b}6M4-b@n<|hLWpsu|Sy*99* z_+{W+a^y(I*Pef1(e2OAfBCxuPcM3C;HjUYu0_ubJXQY2K-rr_XZ4Y%kpIBzw?BUb z{M`EBg9D{+qOBdz4jk!t8MxH8RYy+#^!De!_`q$?pM34Rki&xmx3q&F@N~=TuMXUL z^7Vnz`D>3n_4;kkKTYl0^Dx@{-GN&=o*H;%>oWsKw!ATLC3q^Gf9a9Ew?RhF-2VK@ ziyj>K;u{YQ-15fX4qRFBEZX(Pz^$Ko12~^P@>KiRp1%V8o_z4914kYN?NHsjCttmN z$H`Z}n=0+Ed$hDTv!nF!x-|9o;Zxbh82T&}&g^JWnH^I?iPOg-Qy!qaqi<##-@Z)! z{flAf2=b0arao{SdH-f~<>(vP#wWvx$3h{>)9J^R%ptzcX~?z3anA-c3fJ$5H5g(UI(r&-?r2Wc8)~3)IeeD)Cs|w9uFO6@Et( zr(@0Pt|-~f`RYsSuheO!kVV;)#OW2Gvaf%uI5Dsux*CQpQ(1PE=rtU_v6*C zetVYMxvDgAy6(rTUVVJF+Bs=r;;~qD;&gdq;z$M3=T%vJmOFfI0-s0|J{!Pi9r)}A z9n|;Oy3nK>n(OhKcrHS_=is;D`4<=BH}Q=0C+Fg~!}Cl^1J9R$=j4wsdG!~7iLYdd z`c({R(P``7Yo3St;}(CDgZ%YxKt9rhzop=B9r*h;$_am=1^8`b`uIioZTOqH2)_+~ zhZf^E_#=7Fq~9c4@clU9fX4;IW2pAk&=R$C0pywtO?`D{gW5SC@>{oF{c8PE%{yqH zM!8@80`UBX#AEY|)Mu&B{(8CExpqe4^tXPz_SJ8%KwiSn`%a_Uxw0NQ_SY+44Xso= zr|(Z3NkFH6_2bBqH|DC-zlcm)@Y?soM@SbVup=5%@tem$8K=X5H(*TMt$2LAdl$x7 zr7|^FLf?)-cRzSD#O31`tH!q`sMBv9y@NYm z`~LbPuj~%p_3G83yN>?x_a!yt4LcDdh7(o_me*;dh996`~1ZDC;weN_WJUYBky!i znE&=~smESfQgP(ORqFJ~KihEROiME zbk~XH>NJffDs=SH6E_2%JMjpQFX*Ea@6_C-es#xMY7e#V7~1#m5w+l$8BZ{`;P<2j zvE`~U7FD0Eo2ZtSqHkcgn2#=p9dCp^Uk1D00Q+8l4R z{F3})A$3=o>eHM86_!r(uK0^PRA<`9mkyjM*8j75M zdx-q}SKysVo~1*;vHQs7`70l>txtW#awX8GUqZJYhi#Dl4n)sgDTHQ@}yMZU`NofbZsn7Uc4RgE7;17LofBl-j45=!$=YJ8TR9J;V zw75R?KQ&BgP58useFMG|_&{!|T2=W7faSWOEw!($y<2TZLOmshOG7G*kvbd>smR1g zD2&-HIsvSkD^j5_hN=VlzH+@}gw-S&SbIy?bPiW z1I_5()0}E+9Xkz4wRN|pZbXaFOR2W@F`;c}NAs9yNNZcm{_B(1rJ6h1bnmzc&5Hvm zFtgpw?d{j6n!EO}(RHOddpdi1yHK3+I`?(=H1F$a7z66NySb~QEwy`0bThh!+v4^m zCz_#?JJ3!T1AQpX3F<7Tju{h{zU2nx>tHjf*kiGj>(=WCS&9_=9$$uxFGH{ZHmk7F zu`JA?`s2%x|75H8z_3NV<0cG!88Z6sV`PxN8h1B~JFmO0yRD}|C2Z2MiuCy5N-y0ct9uQ;R4w>_f%FLWFWf7O ztKKBiZ@R#DAAMJr_h&NmR4m({nHS@qN>TbkmxzId=P2!_o?Csna@9FGp z?_L;(Ww$kVx5bw(U%X;r{B!iRxN)`GuzzoRYrMO+yQi%q-m$;CC*In2-QIm|@txbY zZMi!6xyyH4eZ{uVBrm^q=jET-c=<-PXj^=Fd{HwtBjS2Xp*fDKiC5ZR!Cs@o>kaB_ zNRKXwDbfVmyNOoo-`JB^(Hj5vP_}cT*G17MD7P}I>7GY5D`?*yY%Eo)sDY!ZH2_D{ zt^gb+EjZQh;fGWM;PoC%G}WocRRj!2rr)o^Tf!=OO@{Z4PN4%x_Q#+f^i2(;?^74* z@1t||cN_*l=?!3w^UZhH%^J>_?_;-{eDhs(hlY3kcm3UEzB8|Zmt-Uqoqv#ZiuCGS zUeS36)pm&gFqufTQoHauich=-@Ji?{o@m7{RVsteF?yd$r3Y25zZP(|Qe+IY|3Xg?jMIu`F+hmZ8M)3e}hn zg-&q&-zZVX>wphkrE4ZWvk1Q^oB0xc$MA1fCGs~@u2P5aX{khhG@I$eZ<9YMMfPH@ zr`)b zx+6OO6c}s;|0#v>QDr#)_!70WC(N?VQpcdadhkQ_Wy2^Bcy5YK95krCmEbR;N|4X| zJx1|n2{O)6YW^2)CD=y3zOT0rlmO7tf{=p8H77tvk}Y#u$72R(E+=AC|}Q%4lQyGNX^k*ZdVTf7?;NZl9)K)1`F#fKT*|eCDLX zml1xL<*`Kd_o^_dqMCL=xteveDo?8h>iVw4yJ&A6yMiPJEe=%*^%LVxIUf3)N zIGoY)NpEVX1DHJue8}U!b*m85k3m6>ht#z6T5ey%R7eH+)6!eAgn!v&z@QgSgBUSL zOh6y2Q=2yvd8T)~G8}#q{hQ4m5dExAFK!_8&8B}3{a7sytGJnfLW+hFJR4w$;yCBc?0+W8eg7%~T-t@*#$SXa4t*1Z7E2}Q=g&xiX1*6YRsm3gx7s5V2 zWPOHxs5c=H9zQ_8KCRZ0K5QcW15ElpodiVs{&ErWXOKR5`bqgSD8b}!E}fy`0Db+j z^ko+FyA|xhgf;!fbXh|NEE*W&L3slYfI46IpXgr;m9J>XP<^I+Q)(iuZm2zEuhFJd zB@b0TJb~A50zdfW@~jelXsJ+5si~^MFAw_-v;9sa`+eeDfQuNeB$(~HnBfV=zUTDK zV*SDP9*)rct-XUotBPq zeN(;q!V{_aakdw>k0OHEout8U`)uD^FO`5@@|=QynoifXk0b$bKj7xP~n_4r5otNW^1 zKh)$qV(JNkOQOVok>Q`}$9e;orbpwQS04J(-y5m&>c5c67lrNi_xBbTd-;24{8QHW zS4`vE#BxyQ@eS)4z?B+?mm2s5JVC<)8cwN+jg0>`*hG=u%LDYIVUIrarKTpD z<@JSy&+xTW#TY+q+vaF{#Pk;6FwNPU&DIlid~d|OV5`Fd&7%8{1cVxV6R%Nj>dKQc^3eV zEanN5=lZMX{MZ8(PV25w<_ zgo#^jAI3B0H!)k&Lw`{F+5Sm>FD*en@oVz2zM*{5CoRuy4DpIrJMAg;+XHzm?ZMIj z70CT4ji=PVT^sax3V5l8F+T8k9A;FYgy{aQah@(J^9}GP+T|N-lwQD^`3U)yHa*-w zlO^K9`1UO5%`*412EPgRt@9xt8eed~whq`lzG3`mOvl1rdE{fg>)$UDToZHa-^Anf zcHN#^_TzT2j_SvF(h8p})Ta%M>cjdy6*1*A7(bTqctiDd(Rj)Ahs{2T9-rxcgY}>I z$M|*2e(VPX=!ZA4{VAp&1$~}B^!TXhM}qXjwC^JF1$}Mb1?cPg^V25MwS2;-3ev}ZipZzF0`fK!3TTl0p)YIEc+k`90cyYQA1&WQf1swTqp(M6 zH2OO*J{WyTQhDkx^atB7;oIZGa|Yf+X?DQL;OtU^#Dj{exQE`m>&26{VKCR z5*Ml20&M2{DA~u03~T;MQmRUq&zSPj8lqoK<*^VgC-ytEw)6S` z>jm(`8+u(EP(SuV>hB4&>hXRI`k%%h8Xtrn^fTN*AT7#X4xMp+gFMXmtJ_EOTNjn5 z^)mJYHuG)_B>=|#ePMp;a`Hp0XnxK--+~`Ao{t1S>vHl#rl|QTIa)$9jiKWFN68eh$R9`x%|H4N^XukY@Zf&hI^AW~-jITP|8(+&~d;?D~kFU7>jN>=K zydI0iSRe#rJ%;|&`_(PKil`@e{4b^kb+zCW724OBj{5fXGr-u*)Ap?8gYxAXALXB1 zLhAt>|DcQWqm8qa8P7fZ3XeYQKR}=0!Ss22?jru4+%^>YJl-<@L!%$_`M+Z*^vj0H z{}ftKdh=-^`CxswxwKEw`XigYS&ug_2S&dpZSP1L?)cm@mosR>8@=ZPo}{H30~yi!T!S)ID7 z@I1?1uYo@ge-jXO`_(ki7khxc;n*cz=%UwWzZ%{iw$~(wFTt0LuEY^bj$ROX~~mug@Zo?d1UYkMQFl z4O4$d{r!H(*U=}eKkW6x9rV1;-!FfX%kzGJl=S6kjJK&slo!|}ud%6b{Jewp#poLi zT?O<_lz6?1NOh0q#YG(Teg zCVz?SY1wc6R*3u~;_FR5?3>nq2QZ-X{uk@tDwUw}W^R%*q>mP@h7;s_tFqAXsA8kfxn*de^|}=OW?l~^p|K%Pi@c- zh?qW}r(9Brl2Sj`6M8+({sg$c5BNi1eS!Q{`~ZX23k(;pCfN8pfZq9)H9kGGcS$AR zP4{8l1<}KLr@n6@kGEW&=}lZg_!ABOoW5lds!scd+yVUg z5qdtl-0&Ap%~Q(&%X(~DI>PkV7=GE`stft#AA;vmSij>A2amoy|Bo2^dkFuH@MmQC z>i(wn+Ug*_T0{BlAA#q~@P}CA&zw2jenY>yZ?5l;0{$9PvuD%uGGm{hZ~3R7e1GrC3%NY` zGrmmyf59bQ{Y>BWM=3V`C`a|@op7ZO#f4!0C*NlJ9 z!i8RcTu1#^640OOzF9_JAb;@J-@9OehmY~KDOFv00riI&zpDG@h`$f_$6)!*W_>A{ zjQ#@u(8qv$`g`l^gY>6V#x43{uhqt$Tz@3!V?bX0y|ZWe^za<1uPo4if$<&fo0iu8 za8nHZ{@x3HeMNh#`z8eX3+1bW_8hME^y#I_s{Y>T@L>u*;`|Br5&>j`=zZ3goe>9YbytqGNq`xn5d-d>S+Dqf9 zz5hky7t7PLkC^XIhViSvccLXv_Sg0A-?_e0?jQD#5*`-{>3%qrWT3;C;cKKyCH58yQ#ep~#@&e!JpD1;QRhv z8TKcufj+(->pwAvE&y;eJeDq{q<13IOy*dtvB?-#Ba@p33@-# z_a9rL`Q!Ew{}*d|@CSr``u=%-cG{i}6-=*gsC5 z=K-9VJ<^KlaB1f37~&q|1E&Kg=gJHD$hkAL_?`XTTq@COtjS-;i&$man#l znwF^V|3~FFOn3Xo_zQCXkbO<#@*A|hz#q}yUj@H4vx@!wQ|710*JqMfpPFLpOU;fV zSHGw6{ow}bztR}l16d@?+w&KFSMFdf?BDZ+gG3Bz<1Mzmxc<`;EZM zdjtAP`UC$`nt*M8R9esQ^8@aW^QYVT0D492&}R6nNyFbt#!u3}a3r8#CFwe8pNt37 z;!Iz+rzsUJIv=p~N5Ef|_*=Je$fI_)Pw)*iSI$kFiQ^8dj8tCxEIsJZ-Cr|qKJaOwSvv+ut; z!SP?6?)a~k{ipL^_56=bf3g3M)qi~ZZ^68vWO(HLS8whu3-JxG*&stCF#Q1j* zy*UJGnDzTY2cGM|YN3a}gvu{+@)tYsk{~=m?-x-0xSw|YQ=3u~icJ6U^H#w6e%Wi^ zgc3U+ydfV<2vVNL=X6B-2eZDQKQNxIL%#1H?8q1AKQO<51n5EkbNDCw z{=rxuG^NAZADHMv-sI04sUrrk%3*(1tk;;I4CV{+Uy0CpU`?O&!}u?I{l@IE(tI_S=`Df(?%cV4{Zyac|Cpoe zWBL4u`SJZ{U$ppxf3VRnBx-((i9Y+!V!brjpE>OMGb@s}=11F~_7C>!BmVe#3G5C2 z(!9K3`V3EBNcEL4zvLgR7V7qR_iO%m4gOqzXts9@D6Frvo+o`^*pt^{qQ8jfqdkx> zudj4{L~qd|U0;UVUy@q9*woK3{DYTJdA29$7s~Ve#q_|>=K0#4jn`iR|6tUQ=R@^} zg8sqbo3y*5?0*mP7rxn?Pp3u5(ENqX{HEJW>j!!q!22KUpUT#V1ie0le2o8cF8*@( z3xgk88fgFAA^8iN`En%q$>A>ye(3&E^D_j0VQ+pO34RU?_Fo1+EMe}iGoQ!y=l_x5 zC!3R>8c#k$@)t7eixJ_cD3?C7g$!Mv&HCk&vd1^f=dd@up4@~{k;h~0n#umj$R~eIy8q|##J#`9_>S?? zoDV?z_<8^R`u;@5(;Vjmy6BF}^S8S*fDiki{SQnrw4SZQ`G5lF1u#F-c+KmvU3xzz zaDL#0H$%bm14}R-6gn?Z2me0nya3j-15$o+j@U@T;I;od4VzR&rO37F5}OQ^|yIGGp=vvNZ;6h&z?_dzpTJ~s`2yB zr=R%wyXxS48mp4^*wD^Hndf`s`gV@=Z4%v2*!o8Mp#}6!;*y8?-+^u5V{m-_CQ#H}(h2uWuUv z9O~P7j=oJDp1uuw{xYs_XI0<$Je57Zv43cOebe~oP~XmX^lirQ^li}dwQ+qrtNMo5 z1qat}>_41e-!%R?)VDfE-!2%Qz72YQI<9YLRo`a1`W7es%dc-5{~YSuOh@11!_zn8 zUy1uuex7{@t))vpFQD^dW3Qil(i{1e)r;Q$V#`0*27;MXmP{@vVb+@aYY{@IZG zK9R=sRwsT!r4>Jc;%`=+A%4Q8f%plH>83zDgr-!*R(jrI&W|9zK{dj(+fA=oo`h{zt*cC{vm<>MtPhk=jT7npU&6k zb*X(Wzu;$DI?Csf`TZj}zYKa&qR;1znO^XH6%wzb-0%l{et(EE@~iH%&I6PG3gkon zM%sP}R?}Jk*k6hGFJcFFIo}*OZ;bX=_gUwK!7u9b<7a?B#LFlrdK^EF<4Z)#J^28? zzjv*Fp7=tFCt$@BLq5)@mwEi4KG2_G*$3h)6mz^`k*9lpx-nI1^)K~j%s+pP_Je=c zuTn2QKu{im_A)<~e9*rB-Zeh|91p?bU-zd~Kh;<4*N^gkf7}FpC;mj9h-ZNE{k^Mw z`m+eX#6O>p@}iG?-k$YwMIb%{!b8^Nl{Dzb5?cVg)${Ntf-e!K>b-?1vtK8fGR@0YRkX|9nc#s^=2pCEl)Y3WyljKJ`^%3S*Fa8Yoe_7ytQKG-T zJr@P$m#q2nXX6njlCY{;;~yXGJ9LnNMzd4JS-?)#%Gdmw$U@#iP#7y2s@ zAE_yI)mC%flh=D;#Ft`qH1ezNn-mzIV4pLozqr3yUl%uMhc(R~b+`o{AtY5A& z?dk7by*`ogzQ4EHf1lTJiqEFww_R!E1^%vd+V9&R)rb2FO&{?jz(3@7r6(Uf-}~|a zKH?25^Wrl+z0Hb0km2|Pjp=0#nQ5fpkRRlY_So?>Fy3s~%KHmO9s&7*eogucKi&n( z2j&~(SNByIeY}a+$2A+S(Ef@PkHi{}HkkIK{%NF&>_4abKhXZ_zDl!x!uksI!OsRq zUK>1lARqKG6dglFTd|nRzam9-7 z*Npx2_f}T$ctg)K5iiM=7wRwf^aJz|Z)J@g?*jcL@sx|`{M%r8mK%BAhV>@NccQE( zIX(yCPpr1uWA&%R_lTJHvtb&f_Fz4(`Qi9H6klV8t=p*7WZe2mJ(f68kli%KFsQxCO zUrT>Ez9Qh7^g2HthPKau{?_bR7l@Am{or_h{`{yoexls(p?^&L7W9v{uYi4{eSW+X zmS52&w!Cxb-+bNQi01t$^%3Ql=ck}#g{SqWa7p2sks)v=>L^|eb5Kc3)lnTxnBE8 zpK4aR{9J6erzX9`*C*1iWlMbh0gV2W_tUaGAivp~A02P2COyTsA1c3LN*?>s_5uE8 zYyR~4CCGQVr%#X%U>QGjeJNFE&F~V9#~(1 z|2dj}=qK>|d*@8`#+Ty=d(o%rbpHFSe~4#Tkl_P*dVPZS@Of$Te#mGz zwOp;Gx{dwUq-Xo_oroVVo|mV8CSIA=hsN}(2E`j@Odsp@OBynDL{Qg<_-g)oXI=#S zPqOZR@P23PPhL#uG`>KdvR>l-D8PaEs*oS%civy(c&d86?e9+kKQKQ*zg>H$@z!1+ z*DRau$FBlDt(Su98@Ye_8pk^fr_T4sOW@1>6z(@Uz9IB)k*BY$AD%w)_>K4RZe1nq z#r_ET!?rKH4_Dux$op`^sYQ4JBrdOiX88Jq`UCQLmh4Bro=V~aL0Hl9;&}jUFFRuG{2& zT^{n_@|A0FJotd}-oGpJV}{2+;7_CHF+vagY|#AZ`;VJ@KNaFwPBA>C*%MMX_iixp zy8&Z9^Vi>*V$&badbU)D(>3FT-(ULvq$XYNuOIOKt(qO>{`x`pkC`83eY{oCc@G(X zQNM}rm*M$(a6HS2SLpVm4CvL&8WeBR#4}|7YL4GW8#v~DrC2YO`13FHtp@QJ$=+l< zC;8ESiy!Z=bc$Uc^h4r<^8IU5DjbL>i1P0GYZ|Q={P=Ozml}E!Z_><9I^H7ag-NtL zU$g#BTBq?d{QQXW?f8G_Ulae-yk8sqZ{_;5e>vj)HJS1fU(xIrXni1hcKkoYCw10u zF@OCAeh?4Rx3?z%x9D&$B0uQ2iKnR7$A}*|-?W$e3-P|fQ>1-bAI$x-Za>;%-pB2? zhn7Ele~$W5UfJ;gA#d}3ZduC-#)R7;qtR* zJLRX*ekkK}e)N1@UxwRPl0JWlkvHP2QvGuOfMtzt@3c6V*Xw!2Bh>Q^&4;jW@Mr8z z+Y|In?#~TBGv|8rK@af}qr4n6_m3z)t)A&a9*`GcqhESFqf z`=ca1XNIXSO#1^hv&sVY4E-sZZPBCp=Slr6pScyD{Lntqw>cI*-f!;f6V-3u?*Xn^ zRumksL7(q^@O-jF;;)K*>iOMoztIO7PlEB0OVW)d-lli|J&o4i z%wJA@F7fa0Aper|@KdF!24@UPVGM-WWv_ALcxmx0(V*Y3QD_&`rx8sv?`__njG(WR~ z@k)t4t%qfPp!lS#1N0CdP~x4E{pE;HO6z%_KfQm2^?~$fbdAvC{?_|Nl%)ArdVh@M zhxw57Z=K)+4#X#I;rAJCqWQ_Z-SaM-4oW=+V;|%m_4TGhB=EHoO7lZ5XSUqXwQu@(V}8D9ZZu z6T@!;BP7vdeKGOtNZxqfrTI1a9A7gW=Jh@I$2OveHyRpxST7UI52SQ|g$U;7NnZbj z!{Aug&;1z=gW~|q>n#hPB*y%5yn7QL&*UHKrmhtJ&m7+?@P2LdC;L0m222_LoTv;F z%mJp!|B3q#@$-m24T9W%O;-F;>W^AKekso2 ztnVlu>b!jMP&K_A@iWbNThM=!_6x!wL>Kl8aDRsQoxEO=`*Dnq=DZ~6F}@S8vnF3W z)ExA|e-8TQywZ^AoAV|^rf<#*44J<1_Z~8R8}|k{Y!^Qf1O7k>zhD)UD6-ho^$9E;@{H! zCD!z?_cPfutv~E|x)_f!9&Dl;0*()h{gpucUC={3SzfPVlE(P&#M`C%3dPrL!8ByY z<7N6*yj|?yx$$bLJzjiWO`rV>xP8pO6~EWervY8!k&^sqX~5%o9{OH<-67Ja3C^ZZ z({CaAUVPo0^sV^1B!9@Kj_7a4_(bbl_(M>C@P+v&5MLMLdBOO)SRe5Ge$1Kwj~bZQ zZ&rL=%>Uuk25Y{icy(p|{0x4=sZIWT519A6NC0J+PkFya9!1!8TRjJx5X~EE> z$%@5=Wc@>EF8Ir_bbl<#v+&Xnd1MFefspHJl^Q>g!(tU!SKiO zZA{gc5j`2tP@dOkQA7WejsJV*^*Ho-Q2gHvv1{7fn6}<8iTA^AF6Gez2>SjD{Z}9! zFvXK3#idJ+=eWPK;sFmS{_dc7z!@Uv@n^m7a@hQJ=H$fP7 z-pt7#_W<1$yga1%yo2J21o^Yx=Q(`-a>nbe0c$$G@{sgzQ2dD?f7bg&M~FX)uP_Y$ z2E{7~@@KuTbcFb0cT4WyA<1vh`Su`x*850Dh(CTFK6L&Dormrs`IG$)JHGgvIrWX* zABN7)p!2dL%}<=#H*|jd^OTs+KVJOx-13<1Tx9zW*(@{{9-p4SYc zzs&vXC&$mc;qhbck3Ttn77ULcbN~Ix@k7r`hS6W<{{EBWhyAmMZtrIQ;FIHL+3@6J z_9s3$e%OC?=<+f9ADHzuBc z4d~JS50^)K8&gyD{&cjqAj`;plbbIxFz~-m!q2jpodTwxh|6;}KS@ZlPoGQ}(83beh=U{KFNFBp|A3fi} z{!wE(>fbMdzeakWERi?El<#|(D+W^ayAybQ>1GASgNCwm(d??1-< zul1SjlRP6eKI+5sIjtVlpLpLF=taH$c!~CVE&am%N8_xh|2zryGdP~SrQg_(1ifOP zp01za?b`kW^l`s+X8IX0N*C;h_dCJA=%e0Goy_~ehW^KIPphuZ)R(J0G`_L?+5Rkf z4XHg_0`$2*;z%;~jq!{9gN6RbZx4@-s6fk)=+p4qwcVVD@cQdl?C*r}#2;Vae|T^; z`OR3R)v@bT zSBU11fW4ExVX?;Zi5U-Re5ao>-&Eg(u^X@zuU$n{T>A^?cjdm?eugq}0Y*MO@7_otkLPc4{Py)8O!G;B`2VoK!T!hAdGKc| z!#GdL@{sec&<}q6&d(!XCjWAMUS7`Y`sdLzmEafW(@O&9QQ607JW@%5%TCZq(GclV=N~1VSRJXQ~YM?+xYtc#`W!N#>*amAHcZ2olSike;>fO zzMV~d8-E|bxW1iDeH(utz_`AhO??}GAHcZ2olSky@%?H4{1rZL&*4%E#Lw3F=P-Wu z*?u2Dq5a&#`)}j>A#@&;&YSUhQ0$Lmzxp@#eE@jgLC+7)^!|x)efxGN_P@v951^l4 z#b(L#tCxoKzTrXsiR1c)0qE>K&*FHv{QTC;r}e15zKn&j%s;+q{Bx*pjgG#p7@ocj zI-fVLZ)a8CR=WDe@h|i1o5nwf`nJ;1w~L3TZ-dTXj_cc5)i-`0jD5dA=S2$4ryBnp z>f35Z-&PGz-v*tp9oM(Bs&8xD@r~o_<{#fQ{yEgQHIBYrGCX}7bbfkV-_EMOUFzx^ zoi{HqzG?h(sBf1#`nGm>`ex!Y!XJjuvybF?tZ{#k9R48dT>avB-1)~djeidHYn`KC zpDw6hAy^A8Ixhanx;yl_d2asEr(EOmihzG8`^3PIv4|dD* zd1sUld+p)!UVRb8n!*8i2E z$-}N6=VeX%q0IR2_W$v^@__y^KD52g$cynKI^ZSgE8)-bV2*Eo%K@lkT|hrrJz2jA zf0ga7=WStor}qaVUI5_BI9@%$ zh!2bN@g)>rfZr$lq@$WYPCfST%;}RH` zz_(b?QI1Ua{r}FaE#(c;Ix|&wupJzy99qA7ryXdi%q7ve|E8 z&D_@ar@wscLrf;`y!YN$vcLXiNM+u?_JY6txBB;gPaS?>=kI?4e?Rcg+1Gyg!F$wIlRyzsiIXA>EE0z`j|z?Vzw&LL+pMDl6sohu#gTL`c$!cBn-`HvSb} z6^CrrLv{%Ys|8Mj7bIA=zIkEC4=~nBUR@!Z(*0W+CY9}PWTK z4z){1;3A1A_1RFUG>q1Od2>-qgDpDB0l*}m?#_ex<_%H?ftJ^2Y6;aLpEF5?`SCO* z6mcc9>kRb~TcQ#9zDX*zrnmb;umiwV23{XJ%2JDoHiR;Rq!`-#+f%i*pe*!kuvA?P zY87!~f7`#^JR7FpE%F1VbRjQ0W!sST)9%*`^s$#NEC^Y`RtKJnp zDV?l{)2omUDGrtkrDAOC!dtay{$YH;c*|`7J_Oh^85&DfI7~1_)#I@hF#RNBAs9*< zz<w7L> zZy47?!>xgIY`A5BFH(h22TL^%&4Oi&cDd4j)ak7*6$xk?-oYJ!4*?EpU&PTq{f-=8 z`-1o{IOSJ`g82yt-tNFI|J3F==z3dD1dH}bzO9|P^11w&vdQ_7g57F@MJlcoM;hKU zW^IzBXzQ9M)dDt|w@d+>^yp$N7>zbb2PM4c;Kw0qJOA-ehIepcCchuA)TaCfDQ|XkyzLZB`dq>Fct^TmvTXZ8Qk{OBbq0(T zGBk$dDD$W(BfT{m@40Vq{*Y{nHoCp@}Bby}H*3Mk{W6vggoK_PoQWNl&T;Y%*_|0ygQ<9i=u&e*U(c2o~)lnRV3Ep5+n`Ht?Ajzr z(Y8rXss(H^ZCqjfHhGI9CxS)$$R-K4Z8BH>*t5yoomLYpQgLl)q~Wn^lO#pk zCOxSZu*tk-3fQDaca+-XHyk+;EZRplNw95`x$?)JO@7m9HNhg4k+wmX-q^KClA>*s zo>U9iWZp6bY|^7UN^SBF961pz+DA4?ux*pM^2eS{{*lvaf<-E>4UIHBc5RZRXxpSG z)dDt|w@d+>^yrRKo4m)76TzZ=WRnEjHkm7b?AheKPOAwPsf@G@y7b1bO_CICoAjhw zz$WvSDPWTx-BD_jf8xlAV9`FZNrG*g%#}aLCUf}&_n%dtVCv=I>FrG5bT^&5x6Y79 z3i$;SEvCn(vv;86&0!ZGiNp2}CaoIEN9w0LMR(3QK)F-3G6G zOogb8RHxJtLSQXvH+@Sv_qUW2f9d`R=>+?*<5z#N>z%B1#>=mA^P|uKTPC51ZkwF2 zAMC%a{k5e(LSfx@k000}(UbE1#vfeTzTV^uzU!|ou#_Jm|80%}@mbVO?Y}8tRaKBc z(CQL_d`E&|SAsE=m);sp-$Uh+xzY2^Ar`6N%#>f9db#M$V} z$<6V2s{ZL-Lb z6TzZ=WRnEjHkm7b?Ac_g(`tf6Dy|KUG(2`~lB8(cq$kw^Hkr3f0h{#bj#8T}bL2#@ zXdl@m!M082${%|+S>d#rV3Ep5+n`Ht?Ajzr(Y8rXss(H^ZCqjfHd*P&iD1z_ zvPpt%o6MCz_H1&J(`tf6Dy|KUG(2`~lB8(cq$kw^Hkr3f0h{#bj#8VP?8u2=(LS{ZE~6;CxS)$$R-K4 zZ8BH>*t5y=omLYpQgLl)q~Wn^lO#pkCOxSZu*tk-3fQDaca+-X1&*8u7VRUOB-pme zT=`?qCTBUVCRn60(l+SQ8@o11QnYQ-lWGB*%v+{_O?q@ksZGvy*so>U9iWZp6bY|^7UN^NqHBPW7I`^Y85W~RBq`c9=}EPKP3A3Az$QJqqtqr>I&vadw2y3(VB02h<&Qm^T;sHw zV3CSzLn953U7I8++BWG)wSZ0LEmOcIJ-VaRCf7Q0B3QJKY?5HxCUfPFJ)2B8ttMEc zGSW8a(i^)rNm8_J(vxZdo6K9LfK7UIN2yKH5mZ}Fd?XeA`}A&JvPpt%o6MCz_H6QU zr_}`G|M*!geSQ(T;f^F{f=QQ1ingwKQY~PUdCL^ANssO*waHD6oCp@}BRLXm+hnf% zv1gN4I;|#Hq;e+h`?&dIcb+6E+BWG)wSZ0LEmOcIJ-VaRCa-ejM6hTd*(AZXP3FoU zdp5bvX*Izjm5*z4XDUB-ZIYyD+oUJe0ydeqOaYtp=#EmG-0sMUV9`FZNrG*g%#}a( zY;vd5YJx>7XR@h}n?H7KlB8(cq$kw^Hkr3f0h{#bj#8Vv#*q`jqJ3nO1lu;5D}U_S zQnHg~4-W7j50indL9QY~PUdCL^ANssO*waHdTP6Uhgkxde8+hnf%v1gNe zoK_PoQaO`Necb%9Ym+2J+a^7!7O=^@WeV7&M|YIk6Nt7Pim3(_55RF%8-N=exK>pK^5YKN5P;{YDiyQv=c!r;jyrIJiuv)! z9}l_FvB`r#$E%>58aOAAMJr_h&NmR4m({nHS@q8fS$@AN2RRW8Aqanm|@`7ydAT+GyNlAO#sq zVv0hAQyZey`gbk;uCab!7ggvu63R3sS*Mdu&%dX0MQTwT$52UP~|3Ahs-*)UD1 z=kV$Jlu}jGA-beeadgRHd=h&h8+N6TT99Fr!>O_X6@Dqoc<{Dq2;p zGUycQ7y3jmK&P3Dz1E{^1;ziQ*Wx&-!W5waHuX} zMuJ!gI@$ zos$S0HB}i?f94Hj;-d0|lflQnPM|Q+yv$Cd-%%}CuGHj_{A`%OzkqzHJkbwarhFoh z=qMTTXndiXHaK`v|7#ZET>aMt#hj}9&uF|RbgD%2zxX>E##F-X4)3_n()-QT=CCcA~@(HK@Ri?b?pVTkqBz-y(a+Tknqdd(zZuv$eVAknEHwW6k z$CNkhdta{nuSWUb&(VG~gyoy~|MFKw)MnUXyoFw)fz*b7^OTMMt&fc!J%TuY&l!JeHx~pw>Pj|dA z-qoCH?r7_2OLf~Q@h|Ub+ZW&6+}^%--}RTocO{ebJDHrWxum9eqDQKbIJ(xIumtlJA-tLa(p51%eTH`%va-5Po_r;r2*YEFW+t;_8J*TQ%kO91$~B}o2UbNhY~7U+Th>-Ty2@z&;^=D045zgBt9U}UQ4(iQQR-k!FA zFoAFJr|a`cF-?o!zSg}RmoDd2|Bsd5+IHjK-EB!~=B2y$w>G=wTRF#c1!ZpA-xKe= zF5c3)e_v}Hu{`9;f?u2{KZ z)$)s1tXdfNey^l??DI=Cz0WVx-?Ntn+OKp2K(4hd9XGWG>*KDB8$LDqVn+9PwD4J8 zs3LG#`C1;VLrNLF^cwHMJ*mB|R}AlY{XJe;yTTxTfPa=VWMQM*`UCno z@EZ!@qd%5jta&kf&xXQSJfO8o!2G=GGw`?>W0q7+;|?=1FhPQ}2O1^lt)w4f-pN_R%H!`Z9j! z;^%+lNk5o}pB;Jlxhzlm%sljeoF^SyBsuv6I$spLmhNZe5>1=pq{j`yr;twTKe?_o zKc*vnk|g5$Uh|vihi98OgbGD$M%ca{bIoKVK5@eacF|z)C-C zh8SwEw289sGwByO>5o|Www`>?q`Uoo)T9%ibymGUv(jyTCY#L`;)nEK#_d{@F8VL& zGaURo%{U)-@Fl(8!GFZy&vWo4J>lTr_kA#iYomiN=}iv)BL;tqgD>gZ9sC}Hzth2& z^lKgbg9iT#4!)%CaqxxDb_ZY5yBvJ=sBUkMgD>f62VeN?bMPfS%sFgD-qO>EKKH zGY-D+`J96<=|6Muh0hlpd`bVggD-Y*%)yuRmmK_EMt@E?_>z9g!GFZyzvbXd`nwLk z$nAXxU(#hzlkud_(2ttIPR0{SkC}AAzs=y6JNS}5*}=cp;MY3%k{);PrCs$7zNF7{ z@TFY~9ehb|aPXyFD;#`DU*+ISyVg4RlD^Twf6cVF$-$TO?GC=QYo~)R>DM~=(ylK! z_>#WM!IyTmI{1>l$HA9&wLAEd-s9lEW!jr|@Fl&^!56#BIQWu&(7``w#`8lCzNFvn z;ENpYaquPmUI$<1ox={kq~GV@3!nEp_>%shga5vf&m#`Lq(AE53!gu9@Fo4IgRdUb z^U)IyzNA0t;FlZxXB>P|cKyu3m-L@I_|mRp4!)$n=HN@aPC59J{+5F;x}k;LA8)<={*DWCwq{k$d|9ej~PgM%;WD;#{0!zu?~($_lpB8P;7FX>GVzVNxl!I$*y4!-cY)4`YYYaM*y z^9v5Xr0;U@h0j(8U(&lAd>My(9DGSnJNVMBJ_ldYGY)>A86OTh_>z9e!9Qg5=Qaml z((iWgA2Im%IQWu&uY)gq9(M30{XPd@#+myad`W-M!N14w^N52l>5n@2_Zs>9(7~7V zqYl3G`x6enq(AB4zi;S2;pYy%q#tweWgdCS!I$(C4!-dDnu9OtryTqz zj6C0R@Fo3S2Vcge_Z@smS7r-L#>3~#{2MjtG9F5Lm4h#GnC#$7daZ*mdNRYom-M)U zFY{ErgD>gx9DI@6LI+>c8ytL*+X@F?(pNe7BDb{;zN9A{{J80_jSjw~?{x4*Zr3{a zlKuq;U&hH@4!)$fI`|@oJr2I4w>$VEhb{+S(t8|ykwez9FgD-M8?BGlKeGb0J;eH2S(jRp2MGlWR_>%rT2R~ut^QeO_ z=}$QLB8Mj(d`W-C!529^=ip2F&m4S_!wU|+r2pK(7daer@Fo2v2VdlH!oipH*BtyV zV_&Bnd`W-T!9Qm3-*@mOT{fL%JQTS_&1STWhmsyM>4L8e{c;Ci(yJW&Ck_5&2Vc@> zIQTLT#vOb~pXcCL{iVKNSm@wOdV_;6?ONgBOZqAYU)r_S!I$)egD>sc=-^BG76;$E zk9P1S{aOb_zO2(S4!)!x zbnu1GLk_;A-{#;8pLaU=l76>?zthO`9tU624?Fm>4{)D@FX{I?_%d!h=-^BGBM!cd z8{c#ACH+wcU&f6eI{1?Qq=PT}>d!d%lKz~7zsszbe&*my`U?)ewCm>%zN8;>@OK&c zzvSRc`Y8ur`u!~jU((-o@cYdC^1g#F>GD8Z#)=azy@N0PGS9)6^o0(-^h<+-FX<~BeCd}}4!)$Xb@2O)d=d`6q;GWaMGu=C zd`aKp;I|t3+Z}vK|AK=r{l3e=m-JQ#U;2HIgD>gr4!-nzmxC|qJubfK_q2;|()%2I zkz2;Wm-K@UzR2y6gD>f~x%fu^?{x7^`uz^R=+%P`zNA0m;EP^;&%u}UM;(08s~eb9DLEkxPve0 z^$z}>rhn%-_>#WGq_4y>&wlQ|)13MXV9^{%!{B=lf^jzC)tEwKWUOPQ) z(mw_KnL@<+#<3ZAr@(R6;K?(3auc=v0mRQp(&f4T9z#pgkC{S;oOJPjIqIZKz3PUx z)V{X%?qti}ea)#}wK&z*-n{raxN#<%Q>o@&l_XciEzSG)?cQ@0qh6j?^=X6G)0Xa0 za53HA)0Twm8uH=jdBe)3*R{5`T-?^OVrgSbYjfMGb9O8&3n7sS~sOSJ1)DrC$+sf)zhsOw{&+GL{pcc_KjC>*|9UJ zsdO0a#?5U#pxoBnVcNB~C52WaRkuwo-q+dFws`Yru36OG)4clzjkO7l z2hsIC@Y!oYWt6J%Q819&+PsmSJOk-kq-}eW*O9}n+Op+yTQ}})ZigH1*3Q=b?QM!A zoNU!eES6-u<}2B`Z#QZiwtSPx)}GFk7PHXZzYjjXtsj>%v&w{eI+NPLQ_BeQ(M{JN z1Ym<&TDA-~b*2O%*>ht@^1AExo>_5JeXCmB-IMBRZozkN2Y;tJ$&puAyG6@w3tGC7 z6F-9Jgqvh{TTil$T(Hq`ogGQ|TlaJ)S3&c+!o|D0d*I7?gE1fdP1UtGcXuZZ*u&1& zsh(tedq)yovU^XG+?P{($>X@Wy`!_Wjr_EeH!kC!-5oHBj^=c-W4ZrpX>#9<*LAH( zCc8V5OB+@ulRaI_OsyTu!6YeCvb_^jTD#kmS2Mf?;KuIutC+cDXUi7}r@04ZnW3(p z)OE|3C9m&j>qvGclQO25A>t!_(K!_3VihR=$Qu~Ya21P#n#t9v2Ef;)_)wa0`|&ao z=2xE?Cncjwewc3>C#i~rf#qx0GrFduGj$40mo!w$_$_n7GB8h=wvQZ@IBM#3>l0YM z9|ut?PqgjpIaB_CS(i$lloObI-ss4hE5q$iYW{^^!FOSzm#h78BY$ZpwVz}xw zCe5w?T2sHvuarkIzHsc5+b^(uOL>9iTenIeoU=3KUHM7*7kpIy<rp1UeP4%Z2cX)9;2;W7}&oS{T%pCcuot(KmBI7?84>n zj`wK1KMsZFW4yJ8(9g;6+?H>Lhk-A@a5_Ba)CKTP{rpjA_`soor{+IB5ITJ_d=amv z_udT8J#{Kv=6o-FasCC+EDz0Td19a!x+uGF0)3whhfbajmpjjdL-Q|(pIS>iFAIfE z9vql^>WP66c)mFQ+u{B5-vnRK17z>TD|yZrLS@k9#fLxxy`?06I~;oV&2Y-LQ{g!m zE`hI0;c+t`2@Ml|C3t=4RCpMCCeQg3v^y|R2H5^Yz#arl%f9-P#IIHjUwSE!yf^#% z$%)-n$**Py$&MZyD{#O@n%-Si2cMdxc}C9JCiN0-tIi6d_6JVbe-ivl_`p_;6}%1~ z&@{qL>K)V_k5ap1j1{~JSPpL^`uZXKPh^5x1SCUHVng$nZ-m`hLBh^`&BdoSVf)hJL7zybiw}%w{9!iYFZztk;g5`&<3m z&;iy*^N4BvkE{Yd@+|9%{4?vj{E9yEPBH5{e;*t9m&L5_LK+*nn8Zd>KwF5pHncCH zZ|I*`-_QfvkxTzmAG!1nJ2Lb)38A5Hu_Kr2v8GO7eZxPR3|L}l=mB=*UmpPNUnhmm zZQA;@C??RqS;Gbj}0RS=eZ!EiOv*J3o`=EnFnA^zNxnJkUko#R=@lW$0x1rKAwky5q{_V4jgV7(ZEKeY9?TT=ByY!G437HD>Gs zPo%5HL|oDS1zN z!jSU|MIo{c*@Wyw{k;Rdo`?Qih@XLEPv1`T|4m%i;Z=_NZd?!IdJ?byjr%jW{u_-C zd%g2)LdXWYD*v;MXCv=G|E`&O$OZdcg5Q#zehnLt%`d{{=N7ZRLD-M%pnUrM^{g*k z%=(z#{|MPs+5gJ7U@Q3YyYS=pV6%UN{bK&Wj$AnmyBh4+I9=$2{Pg3B!qBy(qN`-T zd-^cz<2g7F{FuM!?N;lR?^geV#$4=5Jvu?rnWj;I56U{hR^jD9=R0xr;FW||I&;ym zqsmTSpw-vh&MJY)C9IB_r>EkoYs@y@rs+1u?BlTH$zpRdJ8r)$^4da$(AobIGD)(n zb>imSe->ALCrk9OWh?<4O(Dt!?TzgnfevpS?$oubMb!DVw!N{nA;=cAwsrUxv_7_= zV`o$AwzgQkzEjp8TM)geZ)@>;JJ^B_U$A3=5UsYpv!mJD-d^i%Zr<+gX!HeY>%HwA zwG!LmYi(}}_&Wkkix+Kg@wIrHo3^*+FKx@O{rAOSfZF_mMYZ`4FRjhb%gbB5s22Yg zEy;U0f9bN?+C0c9IZ!U0+<->FU0_2cj@f59^sbr0%C=mf>ls)E72{g0+O>;3LS`a2zT8Vg z?xi7jT3Q&#_3x!2pR?@Uv}}=etbm(OL+<>2j9R4EV}rGzZQHhXUq>FRt!;g*Hi}uZ zVoj~U)aGHU*5qyH(L0K#mdIc3s+{abk!cQ)Htd$IWjTi@TEXm3Ieh#mcnvn|Hojvw zqn_k&oD^Ehi-2diV*PaI2!cAWG6T0BNe?n!b_4D{mKKFeF#EcsS3@iM?*eYa`GvJa zP;w=Z?x}$9J#iwTyJtK2En@_qXYEAR70>ps10dSk&(a|XRq;BC7uDBp zEGsE1bQfd)k8>1J=#Yd+m*TaJK>m&(e6hH*!Rw~eg{Ee2M^js?yQ8hGxqY4+Gu!8F z_qp>IFIY0qy^$^p3YN2?&Zg!DcY9ZRhp)xm(%IhOZt!huYW2A{R8*{9U%PQd)%voE zwY4j1Hmq1%vZ92|uW&DR&-VuE8{O?KI1hN(>&DQ;E1j?4^A8KJJJ}O}+ZQ?*c|soD zr;B^sGJ8hs{$`GIY>lbRzReE1pzXtU(f55cI)ew|Wavsdv&*`jEPbUgTDQ^YhPuw& zraGNf8S&I7)otk3HZ{WZ)i`FVwQ2S_sobVyS1R_|7#~J)CBs%3xGe{7(^8oq-Miq3 zm3Y}*pn<{gpTF$Wg_qf~HZXgNgOPW4+i<^B!gu0ms|By^K%me#bp2RfugU92xRwwC zh5TalY-6lvw-#Z)0+I|gn8vcYgueJB$#Jq1yAmV(04@a-bxyW`4(U_@eQJQE(%}Be zD>n9vY}RwY9${AiAf_9fZ14U=J0Ev*b}BpJW(Ld4iX(s5=KlEWEar~7ZaQ<^4pMf6yq)B^$AM{7`Tz?rI8u5d=PV)IH!#3hK zDL4sM|d<$Z(q=S)M*Ux)tC|C9S!gzRB= zz@CE@?Ec+EO9k>>PGeu`8Xw{2AbhS9`q9U4tgf!+^1hJ`c_-`SOA!b3gZ^`alP8nE zjpzCob$0e8UFoMZ>+90IuqXP5J)GRh$sXXRAiyS0;^T{Y&+dc>`v3s)0r+1=z!o8o zj4T%T6j4{c5}uSm_6^SD!$cP6cy@GsszHwbeKRcn(;zJY93`utH~sbKtzME~H-tC5Hnqo3m> z{Y9jFE4r{uK()%BP5i*W3IY#4$nB+x@d^U7=5Tqa{h2h(&v_Ac4((|^Uktn(c()1fMqS`n06z`*Jb@n)?WdtG@LvT!6L^on&rak0XQD3f z$=cp77w}j_Y5YckccCuu3A&JHtHftZ{A}ilYa$mHttKx z--2`IkT@~?tXa|eYkaVK7~|DN5`WBxRF2HZljp!3&{x&U>>fGP`e@iz27E%(>;pRH;rdk{yA6-{d5StoX?~6FVV6+^gol-t^9-Z z&p(kOJjN@1ewnGip4}NSeb=>G^iA{Ipl?23Gq}Dpr2gRl8Bu-LWn;Z?S@i94#q?c4 z`f|nQH=iG9FZGT7GlI|-1c>zai1nA|2RES9SC&~`>Myz;3>LD|tl1{{#?n{MZsY^! z(1g~noIip5HiN&B{0@u1*?Ii5ll-S?J+q7af!2pQS}#R=72jqC9h}|~^EU_A2l?B1 zO&`sl@K(MaWB#pz{X`M-&l(TOzGsO*<|75c@#D$V(Ia8~#l3_cG`ZKVMRH8nqwwPP&l= z_(NSxr(I8C;e_#!pN*DZ(#F;&nI9m3!~B^~{HTKZiCwVY;{Z@T0RN%*`!YEswf}(R zKNgGc&?n@v#CP2g-?QVybL7Be5~*K3kQs~DgA27)0Xs!*LwwhDvM-PC$baC^_~@AY zxjTv0*V1GqfUu|8|49EfV|=!s*oAyU$yZ1Dj>Gw@YadP&`3~7HCBn{QK<>X}pBwup z{F~>C4w+w)JiMsUex5^>>6!DPkKP6^=_`i96 zF68vwo-`+*&w}7Iot9fh`aAm><4l(AwCM|hiJdy z;Qo^w-JfLTMCmWke#4Q>?M42>_a|;npGEY*=LVA>tFkPPl`iKkjd8i^aJ&>(WrTSz{AdHXkaI) z0eF+Y1811D??||QopkY2hUcyKH|FTB;6#NT?;^T|>5+d?1=0=_L zdwwD6NEfI>lIJ4r-y8`s{gQtqCs_2GNlTT#PQ(N73vHEKUwS}Kr4y4#H`)KNwlbP82>efj+!+`jAO6(?ezX%0e8fB9^#sQGA;9aHFMAv~ zDnPMey*Q2W_XvCV>)s#7eo686HGaOQ^mEJdV871S5Arv@KA$z&)0&@|@)6kYV1OY( zk@mBmXIue)ohKX=55eC!xgRs-JCx7(Nnpx9D1T3Wh6xJ^ecetzllQ6SGk(s!(l5_v zqW%}lXZZQV`84|t^B2Uo>*ggYq(fNua`DHKCPmzyczPTLakNkWE{%Ouv z76NZvpJ~0a<|~Hv7x5ws>ot}W5$~Zt(|V5e*|dJ6o`v-~N{{wfk7E0+LCw5gW4;nT zx?Yc&ukiV8$yaDIPDS`5-%lv}!=5;Q%$y8t#D)2O=2Og9$RFb7D>KH(SKKphoUdf@ ze8n9j5os# zem_Azk|h>=iqE6+qdPYrA^-UgnUBmf#jnx(Z^|xEl2U9bfiX~7v)eqxsT8H$rB;vV4BMm zpRj(K_Sf4eJ|*(~Ige+^kJ9(pdH#g`KpJ};`Rks9M2hE06Vn`&4;2LM6s{9^zKiuw zoEIJ2#Nx#>#EbNQ=l!=Sej%UI1N_s6=;sfFkH$0bpN@D&`=Lmt$nS1Fo|*O==JOUm)1N&h=Gnqb* z;p_8xeBQWwVr+l@Am5+!^HZL0zp)VXlMVav!}z?xKAG>wxqti@+KIIKEBc{ysH+d_(-0GL`SY)P8()d^4T@zA5&fd_H7G_n&I7-Jp4G$!$H}d_&$zR8q+NM z>ogtvefBjFaDT@B^M?H4^KkfsIA71A&rkX1kve@~o=A1@yT-EL=KHm%d`?F4@$*Fz zQmn6bauD8rD4qJJk6UDaTI~Ek@8;w+pC6>LFM=PjpG5~a|0(VID3+h#^8-_U@)^`2 z)xC001NSs=PXqTfa8CoDx(4X*9HF)!-*RBxzqt1M4}TQ->Px+qzuoe8|M2tjaevkD z(Z#=s94=t9d!OoYRl9rV<{a1>MY`Sit6O_N_*xg@6`l5>P4wViS#N=_1Lx2Et3T=phgP?r!lUwB0vg9jZsNGKSVt7F|NJ}MyM}( ziFTZ~aaP{YjK%Q#QmlANcl2e>MOWEYe4d2wsiq)7?9mdzaN;^RfG1w0fJuYHvNtufsz8^oiJ%HwBjou(!0&~y~WA!JV~qGy9n zlROrG;8fg}WRrcPYzdJWx0^xldXmt4RBjq64n8WkD?UCdw_C^^uaBr)V1#5` zgJkwkMKb$nk0MK$IFmF{&LG{5I2%0&&79kGE9Y@_t_#khqoNa|?S-xjOIuUo%~+wQ zBDB$%b+k%A4SF1fCvHdw>vVn1gP416?m<1YIqT6pHegjT@*`k{b$W`P<4F7+Un#Uv z-bG`p6~($@#CW=%235cDSD;w?eY#aQ_cGyDqP^Yd) z*EDL5s>k*jHl0Q(Ox<8FbUl`GbE8o%GL~-Qgl3uxrqti&(a5pbWJ|a)_xlp=?L+@Cb2Z-Teyj8U_M^e4F}ju8czxbJht}Tz&~?*H zO&on%Xyqp!y=6aS%7;d^kM247988F=Ahi6TSn2Pyx=BBARz9md@#xj&j#gaI-_iR^ z&cFFA@AG!H_j%W^X+9ZuY$DdjqTaqyJ2nqIC>FE#F=NzIv-T}rUt5B`HsfdC+EsWT z9fo7Kt;(1q@Xty-;xjBdlC_+p3m-S(`^ER;#P&bd|LBu(E`gzcUE}?CE8ggPXU6^| zUZMG)xvkT+G8fJlXg-N@){X<4IL|QBZuktNC91jQ+Pz8u@R1D!I)@Vv`5>dpk|^cP zJpU)|vwgO>*;6m%y!hQ0CbqEe&Hc@9YixftYu=2!pG^Pp;k4AU=9GmP@^kR~pGrvW z#vj$?<-#82;0e$Fd3hycTTAh`<#=81`9Gu}vISk^J==%izoa)58%~0vU#&x3Y4;Jp z`N2G=y8hc7-UmM#G7&B3@X3_M;D{`@0U}=ZD>!os=cz~VRTF|4pbcohFV6U1k?nIOC5=zT zqCV!6?LC(9kIVMTf2lpWw6*;bOx~yBjQ{m5AVzWXcgC#aD?chn`yRCSSjO*{?Unzj z@l`top9(;{_HV~&PgBC$o_;I}QO6H6oAiH0wwLTjFIN4pNBjR8M}F1*ifjot<@UFZ zPd-69-C8FJEtgC=pt`E|1isyNJc`TtX&Ej$U)8)djz_$zd0}m*>K^GJ#&#Se=aZsS zS2e%XRkhn*xPD`M;fg|0iq{k^YVNFGT-3aI@%ri=h0Pnhn;JSd*Hq=#w`{83S+K6n zTUflNcwrtY6d!*!pV1mozV5x4EXgv2N4Gye&25tG$~F zb~cw+RIlGvy0UrYqt%W6s`bsQ7F9o5wyUgWQC0JnmX-N+t3=zP=H<;wSRe{@iAP{%VvRyTs!FOHh z%E#-w3Ys8qGj#QMU2#F4ucpXf*W!bHHkQ}5u6uk9*THASs!^XgkRHr8#a zYAfFCt*-W0EpIMvY%AHcz0zA$yk=$T+Wal0D>^r~G|~Z&+2{ym`|i*t;r!bL;x*#x25LJBrG;q3^tgn(|JoUA)Eq$2T`Ec|`e}y4JQd zHw9{UKD=moLqUDPwu1ZuSZCSt`lZ{JEL~ENw`8ez*|z$J@e3uKuD)rTx88TXpIF(z ze_MrqRRzCxayym08$6plfANwfOBUzlE#ovRz0K{uTJc_7jUSZpd0X;=d5rDOxuYMS zobyNS-Cz0=jFZ10pPNW6#@gdmOxrVEo0gi52f5()Bb5Ph%{AQ5!@Y5j`6Mm4R5qmE z?|D`~YX9@Z|K|8w>NlJ(h@OeQffI*0zLI}MZ#6vMG~$ysU@=i9NAB$G?~)# z-%I}SbSYlcyn0sh`Lcwo`ShCXr(YUS!8ZY(t_DR<>rkhHr}%RCR$2c?0cPLAmFV*@ z4wX~l!rsD_;8!KwF5ybP3FuGx4xuJW{3HuJPr|1FuG$jlixNH+aO0M7j{$B>6^~0e zjf>O7XgMM8Xj~(_M9NA1E8d87wS-&ysgdyMmVT7Js2}4gQMaLgdd`U|&l%uG`V-fI zUp2su^5#mtTlt(V;Z}M3rGc#SzAoX`etst5R(XGE=*P(aM+Uf2-b^`vXk25A%)1W! zsDu+g#?|aO34h1}?>EpJ?faU9TluG_=ujbkrifeqzC^;+xXKqBC0vcG;71Mg#&MsM zaO=3glyK{~sdC=ZxW@V7lJHUUMZ&2c#S00r)c}`AJY24$26*(3dw^LY2UGI8IW=Qn zFu;v+HA=YBgQ9PiaI3s8O1M?t5*c#TxOsBi)dsk6+@OT3aTWa@3Ac{>JqfptOL+?w z%3WwmDcw?>pc@`cYAGJQ>!=7#jr?EtXNzE{$=A-t-sH%w1OIu+v8f;P1={h`5K`2px&Y)Z^>qkw z@xvwbJC>zutLNkQ_Ky6eH?563WG{U4#*O=fskgI%H?{fXyLmAm+tqKk-l4zjDi+vJ_`VN8M5bD% zp03ISb*jui6jCnJs>*1ePcWGlMByHJcUHzRg;4QI)o0;`QZA1I$g6*sxOq)6!fTsytvhsUM-#0EXd%i8|`1KHLWn*aa+ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k730d971c25c04dea469c917abe6da00f.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k730d971c25c04dea469c917abe6da00f.o new file mode 100644 index 0000000000000000000000000000000000000000..48301eb24873382214a70f48023067a56c7f77ca GIT binary patch literal 28240 zcmeHv3vgWJmFC~KTXO4RNjApO%Ti00K{k@ydfEbGx8Am%MzSnhn@rPEw=ALOwA!MD zNoed@3WEV_!tPMDL&`H@Q?wrxvz3}zZ;_3&VF?f?6I?a3%+^-H7{+VNBc_K9Cc*T6 z=YQOL`&zbPW@;*U{FiRud(L;x`CsS!uU6lOS~^=bn~kYtV}HdAN^u$6K5jgF_@jrV zL(;_}y{GKnzb$mx|AsE;9xh^R;jF#z=sURHwHL1Y2i(u$`oDOF>Ix-G_sjcUihi2L zKKspp7X75h9=&X53;)l~jvkq13y^&}vW(UKu+VezH%;t~w=}j;lpg7Q+rIMXS^EmI z^>ieQ)&4KdbMid&G@JbmFY-S4!l5%4jz4kc{jMj^oIY{r%%5qV&memk*TQR)FT{=> zKM_lN@`c!;50Ay(>-uHv!iiU69TmTb9eUw(Eal0MV(ZSmAN%QU*${pG z!za$9m)Awd&pm#o{e=^;W6-}a{)<@0@zb%Rogc*>|KR;t+S=Oa(I=lgGmiWoD?c9V z_~nV%eN(@T9Xs@k*o9*s#@-+Q05Ubv*Uvq1<`~NM`fWdm-Ertd?6F5)iQVzxX_067 zImp}R2G{`-6+qu}|uaKSkXh-*j29ES8G4I*7Ju?>-%S4Q=vz z8Fbffj=py8(3$9oDa;Y4`V5|!b|U+exesT!IXy!|7%Keq&MX!g<}~t4M+v)EW3_t!0u%jvEBr_F@x3-y`zrd_W+Uv@qh~c|2k9MQ ztoA416>}Y7>&NhePkNf7zZqzRUwJ;eyzk}b&HlNVzkOo2ty4Zn{-u2D6nyKi;CnZH z>-XEYBG>S(BiHe*BNpExpHu#I%Hm(jAFt(Om-qcR8n>@}D zb@yw~dw(|RPu zTR4Ava+GYGuf~{(SUZ0gqcQMllx^YizMtqZ@pUV&lX*T{i}pydv6B?nU-bMqO1yu> zoN?W`2kk&@Fq(`z^uOpa+Cj}pW_#?*%9nVelt0O%txxhiQWwwTa%RD$FM4d;N1V{h z*whqFEo+FT*EB}SS1!UYK8D|rJ@OIq2Pf>V#dXT&`O!zX|2690j{7*SL%2@hvZb=i zn*@Rv;Zz~94%GT~?Mt+q$ce;>YbfGuRSYSBu_WoEF?K6-~OKz8B% zJGAJB)wbyQ+moW?L*!2%!#6+L;s7r>O1?xsN4~_za>`eW+|BD-Eb2@C^+iuZ^zxF% zOVmIA*6N#t#(*F4D}wu1rMG|B|KDsZi#^i?Wh9N+Y&3zoBu%`nxSYs=mSi4WOK>e? zS(NP+X*>{|3QWwh?LbNh)et#aI_{dr>`U!7&CZx34c#e|J%c&$_>fo^HYh-2|DD%K zi8(vP)1xiou8p%~58H_PdRQ(=R?LphhJqpA5T4Y1ep1lx1UCue*iRuKw>LlvTG-r>EW3E%9jce2_Wm>dfDy;IY{Nv{M$ zfvMT);7lkuR#&rUGB7z^Tj|w#LnD=S#Pm&!?U|~qo33)Z-E}o}l{Ngjh9HSt1FpTU z39lRV6!p3Rsxdq23kLi*PDO%&*+6g(H9{{10~0rdx1k=s8=@lqz|h`3-jSehG9Y@# z%J5Yra1dt0zKMxFLEp?M54xG)bZ9y>KLd5brl)2@zNwJ=25{eD-^^qnIDA8NGrES? z#oCvgDThwpfo8@H(1)rzi9XA@H%thtzU39VS_jKX#XJ@(y;-r2P)m_(-fu2LuCY|P zxeWQeTozvG`XyfP*i;BhdEbQB?FBWw*YESrPF9D#{;@f4a0-jwkqfKyL|mOKWb#}!w#%sdSqla5OOoGcM9$-*WZ0zEnQxL z!0WL~^Ud-pU&l!ki=G>5A$QH$jIp)A?fg_BQLvu#lM|VszQy@Vui*boqqBvj>|&09 zvgi;2=?QH9?fe8l(CYI?oS&*l{r-30FV8aBFA~t7;NXKEJyH^4JiQ3M>qw@~mu&Wi zlvi0R`E%ggaUNn#aikK-q-RPf_k(oLOn*2MDP_+1!;w-ae`JjKWmY4SR&C(O2SeyO zzA2lM^foP$Ph@|j)S8X+8DzVXkSrXki1f$AW5qmwGEZq#+^%SQV&i0nJ&o&E=#TE5 z+nTpExSDW)fxZ%n4tc1n>Trz^YbF#%STtciaM9_=*n}@MHa+DEO;1nEmboy<1bnjr zS7miYO_^&a-70G9S>xWZ3BPN0el`@CbWQG^4Y~Y*k+G?OYe!E{XP<>BZWEB~#oDN}Cg8d3J{mV?~Bfkp} zApPP@wkE6%QqeW`(_1*bMW+WiJ)qMbPJ48^mD8<6bNg+aZqxB*PV>0n^>5;IlaAlX z={t40ozn^%OQ`!m_Ag`k(& z$zD?WQ%)go=khey2s(+=d=x@?+77-+2vWq(KFCCTmu2y?u0ni2p8?tf8aWVso@fVX z%rQill6*R7RG8>dqMe{!COR8*si21_UR~uhK0pHP-$U_TUIp5S?@b=UMSMPm_Pg6d zxDlV_W%~HS+oP(9Xc=#iuWllF9&fyVdm0Hh;(UFx3=4J2>G_rMauWqc*A z411Ok?h*1R@8)E}d3wcFJ4db ze7xfJcUKV3$2(r$?h3+<@vf|lXg;2DdyS1m8{?6~LA3IhmEk)x$sUjImov4M;iOE$ zxj%6IrVPU6c!2UGXAsVPg5%9g376vs@FhzL=l;*})^x&6<4-!_#`v=_jc^(7(63J; zoX0!Y-L>2CX?LRt)(RJH@hri_zM^waqjZnCLAu7MSD*Xe=?& z9#5itE5(YTpVCILV#sUF6l(^qEumO5@RTNsRTG^I&oRs2NwIF|rzBB-8}eE^#fk~< zpja{CofK;(yo+MZgm0x-HR0PRR!#UuiggoSPqA*ow^OW`@Ovm$On5KFnhEctSTo@} zC{_(z>!(;X;SCLm^j#F|9LM~6@Szlq6}r+H!=|MqgB1k}G||HldcBbF0#@iT;e`b{ zezp+naf%k1N;4quE@XwY8-l%2nlGL3Pb8Wjhmv65UFcTwFRw@Z7sS~)Fev^x10gM0XR-$G^PQL>Ci{_7e6nACUcb zZDel~;bMN~_Uec>=F{9Bq8mw{kDqx}MDzJsx91f0&f3_rL$$ROfAK8rXE44L1#2p6 zNS6}dUu3tCKFP@P+ll6h+t1xhv@sv&4G`@i0X@Ggy?q`%z-nstuiULfQw+)W z+DJ5?A9edK-F{^V{)0r@E3&`rkfxd2Z`D3RDM0}H&;9$dGk!%E8yQU9{ko`gz}@S0|K_-R zy&ezTejfS1?yp*S6+3oxT|LW$5!A0Tl#BYFYT@-od#n!SVtE|dk}LbKD5$yfRDQ3n zXWBeM9rBz7|6d)-%O!hIlJje>M)JH7VZUYG#{a>ZX;=@RgZ*$mHmIlmCvPO+YcPM( zbGAM|=H}539zwi)Z3hwhJkfd@`D6dkNLlOKEz!OHtp;}rltjRsen43J{~L+rr{#*F5C2@5h|}`L zz}thHG+Iv^<)h`5*?#t|iTbp>GW0v2TXDPDe#czJZHe@4YZB?I5}oGlU*Z<^6ZT55pV#w;{q~@{M&z$>55{Vv z{2g<3Yjt~&FA1;3^35oJ$6R-5B3{~{F1n92?N?ZI{p0gt5m_byE3X8Y^b zCF(ao(5$~h!+@al9VOvSSniwgQmxd$QGN%O1BU(fV6)bofYVCCtly}?p^=a+&YLy| zE3rVN^$|WWezga;YX}%5g}c|3fHLr24Gt-AoFB9YJ2eELpn3mn)i7WPn&TTazBd=` z0r^%91Cj~9Q=|QaY%ky*%>&AyH*5TCg!H-n1pkJ-gRf7dJjG{~Xn&YMf3*ioYeah~ zdfhtFzKYJx73~2U_1_#^f^RO&_07(12F(-4lkZpu!f21OGTk3JzAlgEZUZka7Uc(% z<7;!UJ2l}h7iuOF$MecD;h6A(0*{1+e(q`vs3tr=U$h^XTtBDCm4LUd6z#3>Y`*?7 z%C~Wa=pTizsHS)|@QMo2p9-(Kh2r19Z@W$Ozrt@>My_Pwb#=mD6uuG*Iy2tgE&NU4 zx88o63CI3mFdO@QzCYmgOWHv3&F#Y>j={@xrcZKL{foYTwkulz=SWY3}Y zGu+;?rRfUi@>yv#zZv?~DNaT6{O(wluW0C#J*ux!zHii1dvg6(1n$PtLhPReU5eG7 znO?!v`a6?#>FYJIKFbN`#J>T1nmkWw0e+D}n$~aUX?>}&7j2+zpi!RX;k+CgU-116 zohLIbm}9I5aXwKS&d;OsVs$=@{Y3s{$j>xiT*Cf0_ zA1D1b{&qZ5tWRjW5sBx>^S7*UUhXH*&wD$*guV|b`p&92`Xhp$OE`&kep|Lz7Mv32 z6RSgoaOcPtv45fSxt#n$WJf+O&I|Lk96a*+@$;<$?*F_Ng3h7+oFTtz710K+<&bUS z(SA86oNxFu^p}V8dHZsI1zsC0%oqDD@%=?@IJXehLH2@%zvg}cQ#1F9{%AWGAR_&@-zl(p|K%6D zxCDB#Um;#$Z%HtB74J)06oA(KyBBeR{Bwf2ahkvK zXuLuB^21M&Hk;Wp!)fY2YQJ2n%Lwk0-BX8@aDaKl3Ob>3TCp?vw;Z=?R-E>z z4MSS0ZHS>vMl#v7S#?SwW=l(lJeA3wj2(rEKEUaQEF2hT-J5ZVt)>z)+ok=C|6?UI zM61Q)4e1_7Lr#{XaVfjalwX=PKU&$5%-8iY^%tY=G1Ya6wXWHytF|$t>Jqzm86KIo z1g8r7ZT!1*qUm1@u8EV*we->9*Tm7-*Thl4YvL~ScH7`e;osErkk|dwp6s#sC&bx3 zHd~4YZ{o6`NrODL5l^-&yw448L;6VSb+62f)CTjVbcjdKW=uU>Wvp@vS9hzPl2>`_`Uqub zwg-uSwO>HgNO|ima);nJXy|g>ix2rT2qpn=$nGZGDbX_}$}jTI_$9=QFvnE>{M&WO zLqWdc?+32yVAyaSp;A!!HY$cy4 z_0=gFRWKx7kvx(!jFHNm^ifqZ({Nu*|7TL~ztH)cDOvSz0fr^S6)8gJ#z-!yKr%EU ztKz5@+J%O&+9bUrG*ThGDQ^0wRhImP{|}O-7-89E$8}BdU~Tt$sqe8AqeQxhOQ`Xc zC6(Q(k1w4msew*V_-vN?Zpl$?Y}M}t{>UQwA*o-YTNDpg{UXsIGaifFGb(@CV3a@Y z8Law`f&q~$%Ks{LERu_V8a|@*RX;uj!^HDQIdwZP^%br1S9+@d=b(Ho{jJgf>64sQ z|6AZu6)kScBIiCThMTn zKT-np=M94Swz%n^QdXyTK>xrZ`TvoukXuTMv+^sNj|HT@$CCdKmA=f7^eLoG-N7x_ zpi+-5lK;E1LtR{oF>C(HkJ_QXANn3k{?2BhXf8kHuk@7v=7F)8{zj?4)s#Plq*b3@ znu%Fw2Hl*{{t>A!)%BaT{rjN*?-!}R(sxP)4Hl^Xt@%^jP@Agvec>@7WlMc^SNg>M zr|OYQ>hYP78;Bywkau%_R8|!iQtFDEhYE#c*5fP{&jQvzReY(t(o+I^8v1rt-_z0{ zQd3uB)x_T6>c)xP)qVZ01NPk=F{O*Cu34hzp`99biY==zCY~$!)Z{Nf=Uw{8h zZ~a8mSXWbP{rKR(;HZBf>=wF%BZC8#_xra_BCo!#rpo%!p~>1ge_N~DH&DM9`Rv$T zz0*C^*80HkeC>UH&UeFC+qLq zJOHeB61KRD1DT2yb_{rza~@!fs3 zW4?jfaa>bq_wnJ$iK@X}9q^CQ=AFB`-NTbRM~4Q{$7pkZdta-l58i$-Q9lnKf$!9g z>>jAyJ5=2}>TjEv8=C2z?$~yJYu`YvyX%1t^mXq8yT@woCZF_9PK*V;;~T2o{`w7- z!&SAz?izoJ=I$%KDx>6C~b@ zXJVM%g||3Y81q&J+;!esFX?-C1osBKP5pRnV8A!&4yz%Shups6ul1WRitlvo`}Vg{ zlgf%pULQ56+aBn}Ub?`RpI%76>fsj`z(8m{7)t+|zhvfh0 z{FC(WWjre^lf0h`h>vexN1pO}+9a>9kGEzVPWztrEFUX@liW*RLJnI|{xdBl+N?ih zWPMlr9)IP6L(U-tkzJme4ZOsJms)U=Z#Ll<72cU~B#kn7R*cvrH}NIpl)Cl)9CRJ* zWBD%yiKu;%9ge7nYHUNEieTfr1;Km1OWb`9;`14dGtBeMV(;VF!A(PHk+j(USjfx8 zo<+xQ%2z=`-Y8EZpH>QqNyw)pGC@t_T5f6|RWF)uiLd-$v=?K4hI%QU*;py~B=3<- z<@Yf(=wk9+SK1OV~{8QjO-evTP67s$!`WtsaWRwE6G>w{Fk!)YICgO>SZbKmR)>Nm|=%x zd0ess6@LKo%ajqwJ^`9i2=O0F8@m=zPY|{n)qsuMfqL43V)Z3AF7uO0bag)CI1c!|4&Ww z=Kg(7@~!1RF8S0B^bb`kIb~)lU*(JVQ9n;AU&Ze<$(!?~{WB$NzUuu%)_f0{?3m-_ zu;g3g4@a3`nn|WoaC$eD*h#tyt%%gOTM+f z4p|v%eGf~%%GWLPeOmHWzKVa|ByZ051If4M`-$XR^QCnSC0e^sNU63{+epRtkPyx zfyM%}?E0G?xpm;L6TiOnTp&1$S82(rjtvD-@3uflR2T1@qxY7#b@!L!{lUZILaG(* z(H96cgz&=gp}nC1afLh-6S#g~b6{?4IFP`nH#Yhz0;BTv#@8?62=b%XeR-$ob;zBa zJG+|4d=r7;P}j77?*y{{lDk3P2NTmizc)1P_49&>3ZZzSoPrc}G1stlI;aG`YF@0w z|0Vj&7u+-Jot+5`U-=&S;I!Wt^05m3DI8MtQYUKC%poCCF+3ZptosVBvDh2s&D+;X zuZj9s-wuD-Uj16_@kMOTsu%ZvP2cJ-2G@Va=K)M6a>B^xio25*z3)mm%`Yx`TW zX@N5|GGWXzAUpXIn8_sml1$=uFmaYAkuph80v}-rA&#GX8UAFR#4&`(4vR-HL?S@{ z@2Ok&cGXo|G70c7`MJ=(_pNj4)TvXaPAzxQ?dvwJw^}SrKP>D!EQa({#;V?qUHkZ@ zkGa6AVxJK4YotMsU-S)l6 zpHkSw>pQKH*S6XsZ`s(<|F*G{$9vgP@P3NTV3ps>^$ovX%YOb_D?6I!3U51S%Q|`9 zmPxWc#XPL?KdiptQRK5)Y(Mv->>}6wXHVXD&)LN|zj*IGXaBd=Hv!&3{EoI?d}Qq8 z)2GJvzx>G9!#Edh{@K{gFaLCGG0q)0Z+PIPv5k*h7`qHX+QPw*vZtdjcs`5r(+lY`1DxhzL&;MKKv5+DkG8m?>~Fieo*%yEE@K`&_ytw(9u?;Beq(`0}ORjr*tQ|UE-2Vc2 zsv=~ojgS1}*}DrL9xJ{NWtBWVwz2RhWB)$-(%3yGN5_(iR!1&ef7jXKQ~!AO!u2St z&8+QTjx$#IL!{;N zJesUW;Ww{st&P0Cvj%?Xn|SNC$4|)q9M3;rvssFDA0>a(ebEPB{0@BeLtp&c^2P8J zzIc2ZUp#K|Me9!sEjxN)XOaDleYVKB z-wr=cKpb^MuKWG1$6keRpLnc#*y;1mV|IIbIe8fQ7)YR_`K)%JvK_{4906YX8L;mCjZCfa^unHYT_e60N& zmWjw0tA|0Jbhucb!=f{Y*&z)P)=sA3C+cBgcMtT;~ABBv& zfO~=8who{84)CqO&A=}Je+{?`_i#RXB`30QkKu6?h%jV|1N%D)2V0-=OQ%Q#8wqcbj*ZsEM_Ych^l0lBmLAnRU=fwEI_53z;_y+0VIu-Zm;t{Xj<5F%btXZ(8m%c+P5UUMS8I_#~0kP1A1=Lbt;5T#n7o3Iu%2wV#8*n z3)zZvA^V}f^YM)9a~H}Qpt78&0{7$k9=c9E75Fr+kI}W`RN!;C{wKOl(Dca(AM1bJ zKGFWN9lD?|avP!F{XJ~<(MLbgIA#}iS^%9EL8nE~X%Tc%%@~K8Ev@(#dQ;1TTcbLaD6jfTeP;sn4R>iyS7~q+dqY0`#tuz4jr>kFw4lb z?d*84?Y#Tw{_>>Zo3FMH-&%v;G5c`){q~8w!Q1v})Mdav+*mVkZP@pHd*gcRTmB^W z{{$Gj@dDte z9RB)K8=N|So#0WRgeOI<*`H|X0y&lL29ap-$N&ldEO$@D5q?hwh_IEPIbGUruKZ1f_JxvYGo7sN^5 zfientI+tb02_k)oD0{`^viIKwda0nVi9EABX;B(5nP}OC0*QL0>KCJL1p} zg6;!d*uc>H38b$T`92mW-!DP07xcYx=${6CQyjT3)C?zE+~d40yW-?~8}vH_{nj}2 ztXkyH(DL6Ehh75u9fE!=4t)pZ_h|XQ5QpAH`33zOap+G_{ux^S-^8IGte!}|19Y?B z3?SW-IXV4);*XR5PsBfSGXLj@KTi6Wi9c&H|0Bd7C;hv`f7N9E9}s_>^dAv__GJDi zh(AvHQ^cP$ng1E$kCT3u_-9S#|0VIqN&gk`&z{Wx0`bR5zexOZCiA~c{BhF9h=1;6 z{?~~=PWl_fKW{Ss+qJ{Vckpy2E+oT0XwBDXEjU_e$^D)MYd#u_9>uu!+Sc`$+pfbL zcWva2eYKeT)4Rd2B=txX?4zwgEz;gboGKVhCa{%7Ct1m>;3OGz$z1M}H8U)d6Q z>xvfuxS;o5>UHu1u% z)x$5mY8gKN6U*=)Fn_x=8|6pY#D6tc50BkT`Psxp%=7*@qh|Q+U$co9F$a8Ue$DV( zU$hLrjQQd#wKc-n|N*O=Ey6x zR+M8wnbzSK(=E`$I(#wFGBI{9>1Y`qU5PRDlfDUzg~OL-17F1^Mlp{ZU4- z-xqQ+tW7YECt>`ic71#6qY+BOTIS@mdrKD0U)mfvBmdi5f2geGX}?L9$F7vWz!#zN zi9gTc<8$4%<BIBJE!EXU&Lkh!O>@T!laQ9Uy68l0?oc6kPT1y**Yoq-@eO&^ zMUltm44rsv_Rt3Kw^906u8y=J&%fh$qITBM5lTCpGk%2QoS_pOXOExYID6>2l3eBVz(I^~TV&KN&Yn*n+CT;9pU?(t-#qg%gX*o!%!g(dm&VntGEL>OO=PcM7$;5dPzix;)a9+T# zZ;04&p3kqhMXWf(XTZNbLjF^ZU)#bPBN;fC^6Q%-={T40>m89)oR{$Hoskrr$&PIc zcTqhjits!4li88;|1c}^`~x|W7e136dF}I8MUEf-*!UL`4;6hm{o|1@**-G#;eJ*Z(6c@+b7`arD!P?Uvy;zQQK{Tx%JA z^YiGZ==Yc2I^qw|3gk47H=t^=L{jP>p4KLGCpwgF@PJ9;~C zGw`bnZG`h5amG4%v=`~O0RI~JI^g{bzT7Z0`V$5}#rZLu4>ydD`jK9H!_eqA3Ewb2 zx|8s>q0uiA-ZnnEmGJf<#1-J}su7=`P-4=D@#4dH z@nO99FkVa-FEBsi$7k^`=0W^N|1`XG(|0Y*&CWcb(XGs4p&KZ>1g6w_{G4FkkFyWI zMEo+Co5V~y_O}MRT044K11JSVwPGe_b>VEaGFz(6VzpVAJq6QraoExr_F&lU+I_@o zonU1@&~y}RzW<|LjQl`5TlX`Ek5#b*Xf%sBOM2US`+|LeCS2*~{!R_^v%j~kt0lxr zy1M&XOS%r0^zCo&+S?t&%?AWNSYli>cX###`dCR{YpAb8tD3+0Kwn3ox7XiU;qMIj z_qBE&^b5+3tzEs{!Oel5-doyly|ts&z}q$4TUK7xu;bMy={SDYs=qGMS`upt-*t+5n3tO+VMBxZKy}!Z=xbCtxX5^`S%6`ovm8y zm=gjeN(#DJbD*PRUogtb$8o+v^mZ$LTWZ=enJ>Zm@;Q~zeVu->-3LYHe`F{sF~SkKq1%_GE2Wa`t0(~u9R zA^+2+g_B*s!0*Q#6jS*?hriSh)O?^N;P34$D-HSkn)kJKl~(%q9S8(lf`Rr9e;>v= ze{*+-zo`|oSAR<@#=q|7))qge+Wuf`-vKNMTKrA@exA-`;bm7Q`~Vbz$UqTF+TN(JfG*YSHD6nbir6T=|NOwUM8{g4TMXl!P$!a3dE~I(DK?>&(??}kCN%2YdWZ$@IbiG zT#R!7#by!34XbO%F-l|kJntNl)}|9pZAHl>|J^R{Sy7(Z2PhdkUU;~iu`}2#+p=TB z+6~p-+U{Tvd`UZK^NjY={l5*Wm|U>phRIiH8omCw6@$+kf`GcK9Olt+Z?9T}2u(R?c3tu}c}uE^ryUq~KE0 z_)?xhU&hWUxP(2c;HB)0f|sx-6}*^@D7csnDY%H8R&XIZq2NXAxPlk52Nk@44Jx>R z9Z~Rn7FKXR3n@5{^(ffO+7z728WlW`?Nabu)}Y`ytX{#hnNPv9SSfI|Vc#6)Rd6$XRs%MW9{u>Lkc~Eolvlw9as3%*`PvCV_^lSvK|Gy zSfhedSc8I-Sv_z}pCsl3j`^FDRVg@;l`8xW<^_(~FM+ugj99E-8w*?5C5u(}cf<+> zTUeOMblypbf2*DMNlP^v02}X?8*rW-W!6nLX7k%D_A(bcjo^3*KW{twS&`R>xfB=U z9ri*m^)}{12lt}Ghw*Ft7-GOa#zyeF1Uc4?VP;>|V?(W2l$4o=3>GKVn&Vvp!zvs* z1@;NtC~$+oZh;#GZWFji;B0}z0`~|!D6m)H;{t~SJ}q#ez#{^O1wJEislew1J|gfX zfvW_Dm-G4z3XERBu}@(15{@4f7`=$&dVzfc9~Zbm;0A#k1wJ8gkHDt|4huXa@SwmW zI?m9_+Xam00-1M)h8f4n>|CaXodPp-TxvhJz}_4UXAA7Zm`!xAz>RY?TqtljSHqfdnCy{k!JMd2!&HC5rNtW7^u@iY#TusiJ3#M&ACi2+7Wmts zz+?{x>f6{T@`H!4_ht?2_UI8q`OgTf+rw6s!|Zds>@@XB5A1;7B_H%f|M%h7gZ{}- zC%fClZ}eFJ>1Us(&HijT<2HM;lcgaoY{T^mk-i_d>N@e;N8Hc}2c6#*=eNZ9O`Ml& zEDT+PkNf%@=yA$`Lr>S|&1<#fSh~RaT||Rlq#wTwzes-9PXo7EqU^U2>%{MTI?L%t zr=lYdzmhceSbs8y>1?_M$K%DJ(CW4ibjWTvoEBvqd_Mu3>%KzpU$3K#s)*F zPV&F#xoA4c9}Fcqha{X6Vo{n)*|K2R^SSJyP|#25_!dW$A%?SL z-rJbDT5W!1^JhSc+(?6!=DMc9bq7@|MR(QxSxq>pxw#NTi9np9tSxYr$3!e zbg{S~x(#%PBdpW4{P0>jKz|tz+nMCm-F>G6YfvD3#9nM1b$<#14 zAjftMQ^1gMqX|<0k?CzFoNdBAChRrgkO@=3k>!U?m;#VYKVrgFCOl}uJ`;Y>gzHWC zxCu9y@Cg$>ZNft)JR)#`ojuwgbu+@ey>nSM{M9(np4B=g|GjuS{JjrVr$h61vp*~O z(SPdh-Xz57{+6gaFz;^zAv^O44(cCeeTh--KlBfi zAFNN#vpJp6|9K0m*X=L*tJ&UG$mjJV`%wQi*>e-S-e z`EEVGrk@heVUI6`l9DMLMy(isP#@GkWixP(597fTWFLDHkB2lK@%U)x4#DFg>^*dV zIk*MFRG;~<*Qb>JG#E-r*+TL~{$ruk6zZJZ-%;K}pp(OKc_{BoA#8HNnRI=Qg)&n3 zyphI7u8+qZ*5x7pz@Zs<#)u!>o`a!GHz}aE=fI(vCVG}TtkADA(X-t&`H1DuG0|ta z!wP+NjE??5_TU2=w+Gp8j+>#vF?n-i{Lp_eG|!Eek?6T5y4OvcC$ao_3Z2U110*jm zftQc&NSy@sNQ8Wkn>vG*e=DbF8T16uGu>f>{s6Gs9hS-?`RLAuJi@aKoW$kveKlU5 zh2wcT{dpV5hJMda){o?Qb^i03yqNtgllrSVo>P3lqNIid;5fe({*#_wipwP7&oqBx z7!G3g%hKZ|_m_cCo|He`GH}RiqH}9<@$CEqC~qdNFKKfUwBMIP$;q<64jf9+_qw=z zn(u^>BUXNz-r=?K52fn+VH^*Ja-2>{-@(uzu;UQ=mgCIy8^32V(SmdsxzM z;Lv1emafMUPcu(F+i2Q~!p4ysq1e+AoI{L0C-Q z>~s>Y*Kgp^EVDdMfg#UE`eZR`5JR7N_OM}}T$DEl%NdIyZ!k31>6O}V;83n&FVeTr zkZ(bLwEwuS?^fuW$FPk+hw#Vw=?GVb{DDLHlk|k{e>37JU`ik{3XUu?Hg+!s&A3A$|#@r zk7Py)k-jgGe=Ie%7uxd;U4ODq4qHIhG3qxvosXxI{;~EWf0*kmHOiy{PZdHn`I9h?LqAg`(c3B^%*!cIbO#6k=mR5S=W#1H#t7?^}Ep@)%ZyI zFOmEk{^(|;pwR!JP)0PD>Pzc$J|D<))*JehJ@V~*gGI>a{lh2gug2SfLux!k`BFTi z_Ahca81kw8i|t{!t8NeOubANInDm$ZANpgvJEs3&2>r<}+sEv$%hG8_M3N`PCkt=i zn13c}@r4u=_96X^_PChD`D6BkKO8{#+y(v&*sq25FS4Dxkr@LX?O#~c`2zZrY5W@q zQR9f?eAHqd-zk1myurcS$2`6dV!mO_kNJF58qeXsyuGFIn(AYW$CxjYeU0%E>!Z!a z`hw=`xqN?y#$Uc)gZVbr6UzQf3D!f#f%2D{uqu!8e~MuQ#G$Ph?!@}T`6TQ|>!bX6 z@Nap)gwM|v|HS@ClSH?H!hxR~oi}t|!;;A6@embxp^{ZTVER^M-^{+nv zuRHKP=S`fB_IiC8==Q|u?J@d5$b;TW7QjLF4b5OaAM+Xb5Z2QiXzlY*$i?+@k$${C zU6n=iN!|a<_4`cFW2ecSPCph(bB##lLmyNZ@$gM;eKdgeyIlTfLYW?`l>alKY>zxX zK^`g|%a8R8?V?8=1^W+m zcQ*piLHAJTlE{39FXy`TP}`pXBvKaO*Q$NK*X6=$9n7~=`|k0FIF z`G;wJJ*|Jx{6`ueF6SS#KA`cK`$zcQ_m7cS`*Z&=);APilCx+(;{EiG4_y3!eQ?D5 z<4F}~o)K90k7pJ7d+#4!**{$GzJE}>HRj70UySvn<{w^ESEs)pb`v>hm7~@O9yYC-ilYbcd8=8Lu`v+~P8UAtFxq-Zy?+3g~@ngi~AIg4Wa@Laf**`vT@x!OIx5HDf;>-qbAJe(?FJv45z=kJ9(qKR$5rqfu#Zho?=&nLQ@|2r2aU-alx6 z)9{Zco$DxkyqodEXYvnY|6B8q~M$v=$!b>#z{QW_N_#szCsdqy+T$Ad865&Pet3Ay<5qI&*gp=9ib%KO>o z{c(4iydOS@=PMia@}3d<*&|B(IMSX}vFnVA-OnmGS$i0R_Ong8p{dGLw z#rL~Uqd!wK^7gspG@ggM!u`w<+ORhEy9*e>*nT%&avG8Qb2?s(((jj1f5!btr;IbR ze8{ca_o2{CzQ64e``dJXlRX}j_fTjCr>7hAS<3$TL!oR=&ldD!p>(JhYp-LWnKR}7 z{ZJ@}^XC}+nKRV!!Gp(tg##1R59mLInt^n>%Kuu?Wa zDvI)RmHfv-UaxAOJjFhPp{0&6vKsR7yqtJ@73+JlPd@4<+jqVqAN$ATFgUdPMR;U3 zG24{}7Y_611%ZoaxjE*~lLIfFg9rmU$Ah5~#}TFca{MVBe(>@K4po@+DJYcv1@e~6 zL&VnP(f#;LCrj4*3;J_t!6bdXTs-MZ{cD+H5LmDOL!nIW-S_U{TuOJ z#&gnS{f>nS(A;{wp!>TXr9ZhEfQgS}mP`&0YoI!;LX(eog*e}@D9 zMe%--A`j0~IN8&Bd!^GO;H8DhD~$S1ZV#{HG=%B?GZ@Nq3`v;!hdlla94eN|r~7A% zmGU17<@5H=H{z?>9?)lArmH~clYsn%l02$Uks=TLx!LZ8hW>coQ@Z~__Ri0j+j}rH z-|?iRKRsV0+Z)dVIYxB)6I;nX-dXlSA>RghnfY?OI2KwcwJ+V@_bBa&{CRVdiw*tY zKl!r%@cxq~w`YOlj9xzPzqyWQB}~ujEd<6P;zyx`slK!Dz)Z{^`SUVoV(o2iUuv(I zyj;3pCbg#p^EoeXFR#&FbMj^X*4j(AKiMZOP43^I&(C)2_DcZW>s8w$&oPS1aR0*n zeef5|vg`KZ&+le=bvpbR{_D7;W8AL;f9{;*C5C;e{m$uhyzhec@~Z8X?Ty)gE9A|0 zj2ryv;FtUB1Hfu~^7eAa+6(sf%Jx?KANjxRzr4RX*)R0=Ab(c-%dyZLuWVm)drfYi zT;4vpqJ6aX(EE?p9Yo;Fe^1Q*WN&mxI(UD;^K=fkQNCt> znU40#mD>yU&Xw(rc#8ML^!B0kjCp*@b7V{M&Fw+$DVO(9C|>*IOYvkIk361Cw*S=qN%QwuJbwW4F<;1vwHNh|2E9C* zUz_6p)a@nW*SV?ME8nq8l0UgU)bY#Q9d#HE7l&=KL+*k{?h(2dinMt ze|%7q{~q)Y#H)AGKk^(WBz>qos?_#Cyprcv^Bt!ZeyooS{MYvnbARCRYfh|vuBd;! z7_6@)S64pYGRG(6SH=%59_jO6ZNBxSq%XBkY`;>Q zZ;cpqll?KDo^Cu*=T{GaU!I?lee$LCK|1DNSf3c>5j{<&BfmP{(|Gu-!Y{8^NIyBg zO^#2Ref9CcLh%~$M$Dh^ek(BIzt|tqFye>q|EA~rU=QHc_}_eb%(a8i`| z2hBHee_wsRTzg*K4th@$-XF#D?{h*!clr69PbmKPq({5%oYZJ zPb2sde{%YTcOF3^o+lr;Z9$ZEz@Q}d3u)T-3HIZfzE5yIjb(+{k}!BlM_T_Y&&Okb z9(3luMbNQ+8n|s?l(j$}ZM$Lrm%gvSUwEePE8zJ?552Dq5}sF2B%Bds34A{D>;v@v zvnwN*py5%DA>Yc3@}Ez)XN4RI5{7&$zF&sz;!Ma#`5x#qjsm!Rv`<#ZZgoE-^n<>Y zL0){A+Xwswp#(h8Ms#j3W)0fu^+WA_a_Hv@vL)D%$NK^xkp1wyG+{fH2V4pH+SfXe zpWDNcV3Y^>S*WjJAJ7Z1U++bJQJ%dL`<-Ymy}d|3UY=+_R(TBa!LRE}`j9<)8hY$n zeG!}V`ttfG*hfI;_2=amTMc_XPWDc~^VFc9iECeKpM*Dcc~rjv=m&mXACzCb00{nQ zqwix>2Dv@LI=vvo3TOv;v6b3)8Pz|*s6XU^&U~Qr`oq4JK|9w+&uElky+I>tR#?mrwF$7P*OE;02*f?1wY9knmHv?2dkSCU4KV^IYu2{lNB2{z5U~ z(KKLZCT|eJh3UYqOb1Qq2HQO5sa?-s z3HQ|L)E}19SBND2R=^xFI^{1x{t6@iD!5mS?g`=Ja7zBG(OnF>h4SMKOy6=N|FwAC z%%G$ED(EAZzXp*ZMyLGc$iLFaUyav)VsuaF8t5zMuR-g_=#;+_`Kyfl>u|#&M)!nP zqds!}_1b=#=rh{jAN_~;SAl;u@uPovLO#@2(1D45HR#tF z^jg$crC$rW&!DeGeO38uK(96Eb*QhJzZ&$l2E88jmFZSi1A3i7-+=nc^6l1jpw|WMP{Vf)c5^P~$y+0zJ!XC1GhkZTrZ!qNBZ7{GAZ*uTH6B?Z1eG(Cm6RgyobpJ*C zurC1S@dM+N^$=cn14b!)zomfQ526}V|Al@BgI34&D32NXI(U3ZBs$sy@zm)<_5_XJ zofx#+wRi=wynYD|gO2v`hLWrcHh8Wfk<@;b!9?wY9GXAa5zp+TKd(RNo=`L?iWjJ& zBU*dsgq%^_0Aoq~dqL@JL1CfX0;=3p0bW-_xeHVll^7zK-(1+gVX`lkc_kIwM zHW~4$1^R?GI$=2+s2|3=JNh#h8uT`bPuUBN_{H&@g$^&+DPD2xU1-FwCdeDu$o0|N zQ;FY?lRW1pRi0~;D$l*ikcV--KD2q`+oT?Q8}o@=e1ByluES9tuTj6cq)e?n>HQ>p zUn3>RwDIeVHvcH-vu;L)+c)y~j7ASy#rTF)ydUcgE?i3f$lK?9y3-q6RI1{#G8Hdh zF5`mG(jvm#pV5Ba;I-FEbhakAaM41d_k_8Bc!NHlM5pp_zl;tajqkvv%T!!aqT-4Q z6_=N*c;!kJS5~UHs!GKxR;c)zYgD{ym5NudR`JzWtN6O>RJ>-5ife0CTwSf=wQE&e zQ={U#Iu);5r{em06|Y~f;td;AeEsz*Mpuco2cPfJ`|HKp{L78;crnf2ibcFw6D(Vf z@yKZ3%Fwc`PsKG?M z)z=H+dv9)WOlLHC5Lc)Ae!kH8;(fq0kmpN!gz ziT5!JLiABTWB%aqgjN9ye}X;*q2&-~&@n%xFI3BPJHov|$M{#Yg7jNu%nz$jn;0GQ zb@0pi9WZB%j{HT~57Xu=y!?fx`PJ2sC+9C#=1-I#74Rh({^Y^zxMv1KHe6D)>tWC=>7R%u&@XU z`W91u_^&5ag!+s6qCF1=QwpJgFWHb^D9?zChDC-ZBhXQ>WM)~Ej^@L`>lv2q z`G<@6nV5fgLasG(`Sqcc)YyE6`e!QN3Ry()^-18q8Sob|UqO9APYol1`xo?g24~FR z@{y*^H(f5+i=|Tj=hHDhrTQd{`H0gA1LE+B`NiVJDqgry#YIIbUQP=-DgVmVDz2(h z@ikYg*x^udVxo$Zl2n|Wtm5Kg6)#(+;CoUwOhp*87jV(98juXIXRqx*94t> zypPS_R;*Cz3yaV{jq%hQT(wH3dxKZoRh*EZ;=)1|FI}qQB}-IXMhkXBKX0(4M8y>q zDlRQmab=~1F@D-t^7|LYcw2Q1;-^+0_}`kKBN6eCIgRnvy$ta~qNk;!|4Z1D!S{C! z{u$-Q{O56!mza$Br_B$v_{+S&-2Z{0&nlW9*p2)N2{a$5BFy=nN#x&FF@IVUbT38x zV{U^#qYUwjr3rp~ztCmJ`0jMkdXMG{9zI|35S{BYqZ09i%^-~a4gOSofKub9`9ME& zreHpRn9lc0r<)J(@2gUWr}2N3#_Q*qI$kGa#^Pr##%p>$QjUj?nK3%;=gb6_>2`cS zOpd?K1d0cl#&|u$E7LJvf?v*`l_%2=29umL-pv&8v>@cfa?_BH@~;Ax^DoSo=!mz; zNi<%^#>=E^iH`i)z;gb=`4XMRL;8NM?<%AGV_C6R)bz#8wJeS#eTWR7S`oq{Pct} zGN`}s`5Ww!6Y}u+Hs=SuGE|&ll+VZaV$Ls(@A;HpAKwe`{15zBi(ime5L&c|$0xB~ zst;uptK)rsei;1venCOVLm$Y|@;^@Fe=+7y^7wCEO!D%L@jZX3fng74u(+7(10HQZ z!{MO#43moSe&$RSFLtVU$yF*&#?yXs{+t{YyId-sJzK?TX)2yOSH*6(igR;SoRy{G zygU^b7OHsud=)1os5moI#ZEk(C)Y1AQNlF-;CXhmj~G9^!K5UKj{TYJY!&nQG;;Wa zy=Tpm>1%?Emn>G~rKZYsZ*a~W6{n}Gc-}k}&zPZNuUEwj7pl0ZNW!R}BPUtc2mZDu z=$egqC5_){bJg+OolE}4?E(K@6HLxSf0yw5`RKnAwrA4%m5<+CegZAn*kZCL&+o{l z@!w(K#91`qbs5+>2k}rFzd3)>Jj5?;e22edJhsNha~tVHfA)YSQ+{4v3ZIY7C#;R< zcwkr`&ux_7lVObKlixQ{@2BGaF4pt>2;*PuFDl;;$Nfw5_usVd`woB^QRKy~>?>c4 zI{63SqSO9?z`4M-alT$aJha&eb2`PpLyPE7Tj9Wd4D9_>(2nRt2fPB`bH{!J z-i8(Yu&?V7vywmY@6Y3Y@yqz0vNqqrclYuqf3I1o5AK)M-Cat;abUi5k>Y1TffvXo z;-~cxecB9%*sr(Zd#mJ+9HTzOPZD(gtWc`;Ms#5EA*yd>$X0L$_Tcdw`eub(oK7iJ zKWosY@sg8qek)&3>GA&iAE)^v>g&Re6Xt`!+I|&b%qQ#mzbERe>CeAsN3yAzQU6Kb zhsO7o-oo(6zt@fTldOjdl=|WO&HQ`Q+@A1%*cTt{((P=#EX?mUL_SpUWH#t9*$Me(Q4}d*%diEu4KkCl@>;gsxq#%L% zkUU;~(;K)So6hd+&jh_dqc^1)`T6q#obCp_Sfigv13w;)KU1n<+z-98-&?@W3HgO2 zZw|07kH)ub`W!hO3AF$34CW+YLMSlNXKCLv=5)d~J0{e0VE<(q%L`^lMSkAi*#A$! zgj%D+9?sxQ2gNfSeE#7K;{LCOvEJB<{5erk9?uVXc=}BouvcDiRumIV?ErQL-3c^+ zMUTbvH z$1@ZAAKJn7bLWWq2zmBIOmOLdJ=5)Z!HjI7uPEQ+(fU6)ke|2zInjP-f0UO<_oHxd zdkx%%+bNCkFmh6~e_k*(_E+wBe)z{Z(cWJmy7s*{E`i(2>-2KK^|3gEc}aOO%+}PT zMybB|>ZZ`oF7B6U*cr@E&X@R|!TBlkCA_9QBTDs)<@ZLt8lB5qv&|DFd&TH^QBgjp zXNA&I-3NK}6XoYeMR^><9!@L(=pcJook2$uCipt0_DaSCUc#;vinlnp{qlnIqe5RH zZ-&?((fo({8^xcVChQlZy$Dl0>}k^a?}CtZ5&4sq`a7O~$O|TT<@U=9I`U{f!w>Ec zsrj1!AcbSse9gZ!eAg;wC3`t$;*ZMUeM~Bkfl3G(jRz^rL( zk&>Uv=kW>g4DD~%@(cdm%<5rV41V-?r|7Tnci!KVME?{0X;!l6ZvxLw5&q0E=}YT_ z9*R%gzcTsfIyHT_LjRdbT7L_R{8`Cb|JU(VDVl$481_t%;|J{NP~$;rs@(tcf-YK6 zY6rJ(0xkF@?4Sj{gi}*R{L$OPCHfb~Yx1@J!u^xxH|5&%KpexKYib;LRXOIrcNNk6 z2p=gG?YCx|)PIP-$%Uaz;zxTYu+M7zqc+}t4)$0KqkU4@#Tag9Ru{{#aQXOtp*G*C zfkear)ED+=++Q1T|COE>qsbmWwDHmW_aN?+zh8>@p#FW0>(sxm0r|@O0e|o2&VF0e zK?dgEKTX5@1a-rLNSp6$#rjUWpTfj`uJZRWFy4EDHtl%?njc|4g8MNX_iWSdAD8vn zVt@Yw^M@SJsXE-~&(QtnkMP_FR9Iw%Bv(zZLV#k#?S+|GlNP{rt}#AYp{h59|8% z`o#F@Z`7f@xcrL0z@F;!jt|72k1O9Jmgj3xTV4ON>`U7Gjq9V#=V8AA<^6Qi_jKV8 zbqCb<)9d<6*@<^5Px-#9Ebltl^UilJ&yD$)@jVP#9_Hu$x)zK&U5 z0ULU!`ZY{tzXvZbPhUSXIltUr$$vB1_auEj(yxtA3~%MrdXwTiA_wC^9{snWzt1|z z`5)u`+stnb@o=ui8_wUR@YnUf$^YC0)^|2GVj8bL0GR6f;qjpbKD5Ax7WmKtA6no; z3;h4f0<`yomHNn8Y#NN-ZlSHIvD-f1@yNDUqQBj0d-KunTo--o|9$myw|&3&)oXvy zd+xetejQD=Jsu>ho@#t-T9=WCc`dj~3+UNgn$5XfL33Uzs@87ilZus|(KNr`n9Nh7X z+g};~BmHq79B)P6{C@OyT;zT&`PiQ*B^rI@5BTSgQMTcA4x_)JCuM(g9cyWL^tt(O z{0Hr$BlmA#ibhlaK6?ANU;XuMuYCVczq##|w=K+)Vox6DE7xj1;fvuPG7FZ_mJ~CHRj199 zcn&K(WOO=AJQhWc#6x^Pw^(McfEL@4@319o8HwItvCjUO;|*3#is8NsbdV2BgQud? zvUVyuEi0#@H%>#RW#m-(+oqw@vT>^XJ=4%>nK)Jc&@^;f7EYBvJPnk!dG zUC5?Q!0Nh#$t1seEn6LnI2H@-u~{~_P^M24hJP(}41N9U5>|DB=5-Xa#Oo3~$DS-L z#vykc%xh+fW0o}M+BcY12dibkqF}p<2Ta(SV4|C_)nTHWur<*{H{)r_vpUuMR`ADS zvKnv0!j9J2vOk0q;146({wNs8U*^wR_y+#C<_*Me^;y|~6+#UTm2UId*nn-oF4GnH z*z<(H={uB8vwVlo!3JOyIlo+fqA!sRBu*t?@n^SLz7s_cz(|*qpM-Y@2a=|cui7Wu zEI-+YAt(9r@ShqtL9ssQ+c0Y9CkV??w!v7@@ln#|i}wyc+GS1PyK z<+9flx*V|>XiLC+(yq@-^?o4H9Wq@X7fD7B?hRL_rg?d}`DbZfg7M=M z{25%IZ6s=)LPv^!d<-(nMOnN|Dp&r+DR1(4JynSm5#QeuDMr4(C3@fY5-E(kvG)Bfkw;(54JY-w_pEnNEPj9MNYVM-FL82gRAV;9MzdA*SZRuvHpD}lIbKpM zlwu~W;|(4oX2ywC*4gK1?IFeyHC~^d$`Y4raXB63S(`I6-(cMhw4pZj=|?k8F1>l z*bQA&o^XmaiFimKm%$U0F_JRC>CgvHo9Oi+8OBetMJyk4S(9}grF=14ku4If2|_<> zk|8@LV6qDt1a=UVynRux#lWZsH7PssXr=govcC0*> zWau^!D=$Vz-SjrJrl`7;K9Ws&ecqAXP*=mp;@K@6*KU$-Q`_oFbsNB+2NYAMm~1wU zZj#;N+I5O{H0veWHCDFNrg8l(JdJK6)7TAl)7w;P^GR)~_nphNX8FiuhhJJV-uh#-`5bn6mO8iaX0(Yp-eEhL!YQl&$Xr!VmZg*mw?` zRj1fjpwxz-Ko=KdMNqhdcEYK%aMk?oWcNr$pN5G6D8$YuS>JEwOwc38x)cZ1&!8`3=M-GRo>lNtc1FQV*pmug zOwYkf@`~Azf{WN`1sAdt3SPvHD|jJ$P{9k>pn?n75e3hu_Y|b^^BEq9m2e*GQLvY_ zDL9ukDtI2-rQo@&LBVray@K)AyA(W&l>%2A_RV2l1!pt2g7I#Rg0t8qtDJu(JE!1G zc1FQ7=#NE7?dxGf3O$3JP_UaFSNQQ42o!o6Hdy5HQ(2FKU93^TDXc-k$qb*nlk`bq zKH!+YIq~H-87Hz*h2O!vz%l#b>%j`fNTpyKZh^B)7OU?6G{ni6#zYyTJ85{Uo%cyg zH5ve&)PftXw%Ji;-DG1nzs+JVbFt$m89Rrc`>W_*MP4H^QZ$Ko*bBYPjgEc}w=D42 z!dMu;M$7?DpJ2=v!S7)xzcI}0t9opMP?SIDjrhwfcqLn48V?Dd5tzm|!j}YY6c}E_ z^V9f7ba)-dZ2~t4Oye8Tdj#$gcu-&(--v!%;E=#)1g7zgmoIQwU^g0q^r!KSmoM-U zfg1#-@r{=+@Swnh0{a9$E%1W^pAooT;7bA@7uby;O64~QTq^Jhf$1BgL_aNXkHA9$ z4+=aYa3fxlBmP~$$rd*|PG3ajo|0rk3Ne&+JHLR@tP@3@cG;rZ1#C@JaJH;r%QT5kk0hHAD!tbz&XfEhZ8wb z1n=vFpGNGULr=dX@S7-)4zz~8-On+;Qos*$`bWeB!koTZW9M?{SxWs_O%=d#t3-t{-diq~_E<#3|F*AxuS;f=!*;6e2qdYw$mLRUa zjyQ?0XiQ4~G19GgsZ6gg&CBS(oBcdpw~vpfV$O>AVAKEdqDZ3?eXY{Qj5lTt$}XgP z+%bI`#H=t2s^=M8@F#J-4C&tEll5#s`cf&~EBN~^BS#Yb7hO{G!UU5JpGavI{-`qu zchDjC-FW$bEb`qHyM52|nDfs-Dw<}};dN_CZ6X8l&?}X?I=ba{!JuZDl=Fyaih2{# z1R43AP*G9i484SpLv(8m;9tG{^aS!x%I`C&fj^7N53hFOo=v)&^Ka1TJ%Y_ReOBa; z^>2-2&QE)Hcj5J2Bcr%@R^%@*5zP7XG=uc`Onh&O{GSrWp*E)YV$OdA2@o-<{6X>3 zxmU>5eMHZ%x8nmKP?}Hh>8H~ozmE0t^?Z8!A4KM<@`r=~%1?ae{1ivX1n6GM5vDPM zj&AKl`6;f4HE8@hip(Ul)<_W-_lYwVDh?i>B}~Z_uP8ssR}PWCQAsp88E9XEJkr0$ z#K%PbkktR;wI9uk@V-B_pIa6Y{-lQP%GJ*FjRC5^!UTLB3`gbTP)Lq%YKgS|mk$$U z^lhBOamwHIEiISs|5QF5CiAgYk>6)3|5GBr?!T0uLV`K}a@gmdIOP|956n2c{9ne* zBVGKHYz3BOzY?XGMP9f2n*5N#^{ONWf@B(|g7K4d2%a z_G%~NU$gzUBmXDj)W1jM*V_+n%G=*uKJ^>2sqg<7MJN~g{6RnK`ALtzG#^}+8U0f! z4M6l){VbP9y6XMHoK44#!a?MGJW21*I-@x2{YyXV`Sw+B-%+=(xmr7FH`kPR9B8hn z>DXPd{id6%J8lf@Y&o!d*S50e&Yd^yue_lN*ZK^=#j~u6#>bOI@g~xnlEy z-4#1Z_iV2$ZQWIaGFrjE=cYY7D{pP8xSq-@+ugZu-ObfoDw@j!HwCIYjdBCkW!-Db z_uUYv*&=w>FYl<^x3R5f+xCw7@|&)&+qWf9y{da%-M$-Jc5dvh*>KC+*7_}-)9}=^ z?%CE|yE}k7Zd=t++t$5y=e~wOO-=W@iXH8N>g}6rD%$q8l&|XFQ@?c@efE?Fc9wNC zcTLCFbD(8s+1{4ARY9tET>UoOvWx80RJZ=t=Kjj|-MhAQ>?+@OYg4Ukw;dasx^B32 zGqq#A&~@kf{#_L<@nzQSyXEF;o~L}v##o-(I^-$e)4O};W%Df;_N?E%YvW${&w&7Z zCZ5eUcL_bZwv;w^?r3X@Yunn*>+80Z?WtRLV0UN7fjxEj!7ujIZ|rI5+|du8+;BtL zj$2#mH+JmaSuXj+j{Sk1m3sqmWyad8cIUR9rjAvmyLWEv-7_6;C+F=B?A*40&(0Oo zw69h^^_O^cH~d8_e>%V0NcG-TvFG6K^7ZX;eV}e%zp(py&RgC(jeafMKCt`c+F$P% zTG>;#xn6dCbLXmC_Uu@7bMtid?%H^;xo*d_?Nd{s@$K%~K23f0^fw`%?W(AohOa~L z)f{Xp@1KUZNAPaBdH2px$7Sqp*h$nKb}gIErt6yP(KjyRZ(H^LxvK_!ts``qa`k?; zo@51f9*AF;wQVB)?YX77V^y#PaXN0CH1l09E~$LIP34z~Uj|`fqIQ>g^8h>ul}pZeOvyOvVbBlBjT+K$(B} zs#WFw6=lnp`ev1&^HStsSif-mc-^vhu1Xe}%uLufM0YnHxeFq$vPBT6#M`Xz$v0Q+rommA|hu z;NROF>tC`H>SrZv zn8O)Kw_iy4zo#$EK9Y9Ma=HBAr`#@`7uRcb8EezrpR|6?PIU3Xa!xU9BGETV`3p_? zNv{r>ZWj!VQ`8GAlMt@%F7Xep;zcK5)lnQliFxxpZk0N95G_X-7sq z?giu@a2LsXy{bgI-j8&@_`j%^o+eBB2+~Q8PbBL7C+#x)?J#saIaggK{o2c&Y^rd^!_>bGWz@q8zRI{`&)GA{geLqTRc5mz!NW>(&DAl{aBJeD2RIh{RH?a zy@57saOio?R^BHdiyHzhf;{&PgXBKdNC>95`D`paW;!)4N)!f8m3@!eVo@JZ?Nt$@=~ zy39W+rOW(2;ooNYAt_zv|Amw;%cnomqSs62cZ+ngryRWtMfze>dV_F!v;99J(#`g7 z6zQZtjqUnTFFFbJBYkh0XdNP*KAyyJ>?Ff|gwTAlGm9AJwkiWgFy^ry$mi9nLYjd9;>z3AFpTGA&b8~BNFDq%n zKkrCn*MUw=+4k06HvPg$EGpW&_I6(Z_rca+Z+mx_FhhG&5H_i6?bA%bO1irHT1)D- z+_bc}FVMVS<5~|}ZEx+X?&}M-Hy!9}rBsa{8I{zQz}nV>?ai%9x@OtdHvisWptJP~ zRNRa5`k4E2U`4n-ToG?m{uXEr|GFdtuAs6 zv380v7;B|x#iEau^!5e&0!=vgck*+vy9KK{ zR>J?)oLFnUO{gI)_^w!||M@HJEBAo^f!hcrT1%HS_x6>oFrt^}<#9Gf1jTSu(~`dD z1F)%}MgG6;-|t}D5mPQj(CKjvenQN1bfOpzbjWC*fzsr;0ug=m4~{l@asjN5i=2pm z7U4%zG9P~OapYAERf+i;P19+J)aL-%g8A#9Q~6>T6RCJEP?T#fPsjS1o};4tBwId4 zM1FmJt<&`KbsWY8rI~Gb{Jh3t<~QS?iSjR(f0MosqluejKl)?Xq`&!ikrp^{4m0mv zBEPx*mqdPAr^<&}|2sgB*M1{^(sErAnW-8y71i_W?e{1OHkVCVo_vM=W%M*YKS+%< I{LK0PKVlFt?*IS* literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k76837390ba7090d03755f38f335984c6.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k76837390ba7090d03755f38f335984c6.o new file mode 100644 index 0000000000000000000000000000000000000000..e78937aac67f56e9a3bb65cf3e352702478c8c84 GIT binary patch literal 76584 zcmeIb3w#vUeJ_6Yq19@&NXtqHiI)~3KnM^*Adq;jBtQ>dvcMpy6G=z{$w;8P5_qsl zywbS{ z&di)MT1n72{olF_tY*LG_xqjq?{Ut|&fL3s+ZLz8!HmPf{)$=1NM$T9&q@vR9UD^3R!}&FQeva=DO*eSy{Qf;(?|Ubi zjl8|r+4t5?SKs?C_V~ZM*ts*^>~ZjZhNZFMuVjb&->zUk{1+#Ce6BxQcg2-)?iE)$ z<@FgBWX1pB4EGO0&gpRdumNQ+@;`d{+#?TNUX1hW4?lGIe>uY=;O)id@%C%a44r%N z{LtYyo*8-?=SACpFm&e|-yd3xa|h1VkNt9}=9#NQTXEiovv2)=`0_u!)sT|T$;k;{E3``Y)O9%@IK*S5bjRE@IEEhV0{ zeOI4;=<-7yk6vE%{qsW){r)pU52k!)C_nY3p{ozSF_iy(Dr-&OBGB{kY2WcQ>i@-| zMIA2<-TCxOLw&FRV(6hqiKn#h>cfv*J_nu$3%)dzzva({lAd{S$XEH|P&?{;?eJ^h zDe0rWs(I%3FF%;~^icjID68Pbp_;t!4gKxlFNYpFH#n5EXkFjct@mHfKmYreuWm(I z1%Eblb;=KhYCPXTeO@2Bw)i#htnaJ#{2uao@bcA%&kg1KpBd_V@%yOPOK9IOhw`_- z37!po)sOxD<-SMGaT=v3IkJbVpLwuu(d%D~B;A*NG079HOZsNEM&lZGv0^XA%;AjI z9l#u$<_M2maZNi%v_>Z@Hs=Fq`h1v%`KHf_Mh7c?0dvCPAlkHZuQNOw z#pncc@ysN1@r-RQ(i}DBD&0TyHEcLbNlRbW&Pf8Xby9`Q)jSPUEe>1M=t(lwEyde!y}8_ zEcGpy=iI-$+|V6%?(R9Bk*}@|k2KEkyx*STdB5#>SO3|636HdY*)h`h`Lh1DpEyR& z{$*Kz`wmxs-w(ngZO^#+z3jfa_Rl#+!aE1uXFm;|0?2#^wA0`{Ro34aJhi?p18L79 z?S7W5X8_+*j(WTLQJ;~vcY#;q{5H-ja2~>W8P0Fuycp+eI4{KcHJs<+{9~N6alVZ6 zOq^fBc{oFB#c7dXdo{yEOaao&ag_ye?KG0w+v{vO)32j^zcb3kvvc@EA;abD#Z89l{* zcn|DjqqE@g7U%NETVE-B{NT3=A3yZD!pF;zK01^2|113XXd3Gub+P_mC$oOCt&n@X z>9udIZ+zuj>zkZ`^G$Ey`!)KWdcNsxd>^9kDd(F$!1s6Q`;_w}=RE_PABxsxg`;)t z&m_Db+CJ!hAv+p8@zsQp#-GI2v#PW1v+(_)D`)1jk3IXd#xEzlPjzf`E`Ge_TT35r z{oK;W55|^0e(1o`$LFGcMjcn)v@EyFKUqv0hOm*;|Z~7a2{~CQyKHu~_ zzJH6p`_4DLgzw*_?@7F#gDKJA-TM) zqfYs#Q@+_}R2S+istffWbe;P$UZ49>R*cH>o^N^--yfpyiRYXC0N+p3chC8z&*J+Z z()R?tJ~N}IPrU6OX@A3wx_`B${k3IXT#v!-pr}?PULeyy?>a-AbTB!8X zPZ3MW{bY{i9+X!{<+;x{eH`ES(|6bTri1u?fWAA=Hyy$EJL$VaA4~Y_q#r+6w-tT= zU3}Ufa=(A_w0nd(2G+N;Gm*Ae0*@aqO6tFJjl2J@a(qs^``aINk30z8wm(2yV($LN z^4R)t_*r-37U%m7N$h_#F}C|EV#g%JjqR~b{ToHvBFMNYjmklu)j0c-qIG$5&;3!d zgY{=8G2IW}=ixEoRhRRJi(nHsIxtV2wdboD8wn+_^G(j3=R*k&oOj@iHaWNB>^iUV zZ#!S@Hu$#+{u;qw9f!Z_e6`o$uN3@S1pnqZ{F}~K`wjjI!Cx-;%LG4rJim>}BO3JvUDJ9+Z*C(|KNIJ`toZ5oNC$TlV4mK`#{a4RPqFL0>88TjJ3F4D=E~ z-w}uYRnXT7`kpxSimxT8;J-Mo+mA8+*Z%1YRFxk|FCy zOgyFUDq_Du#Hc^Nm-WB4mW})hG3Z;#W&Qt-nDp&a;gO#mMV#>IguZtWs}BD<+&_Z& z^%``(*7`riM1S^wa|q(xVM5FQys{Pw{=hDZMOr{R%TzE{@&Q^a4d{Iulxa%`f{wN#yx8}0`p@&JHja);l_lvah z{ty04X>32(`D`DhVJ&lR(!C{zK4;oII79w} zo!?Z}@-#o>{Me21=Z5>JeBz($2=jQi%@@Cn_wV@{=AHv{4{I~{(BTFb=ARq$FQISL zmGVO`^71Vy>+`QH?@L)%!Snsy?}kSL$onU*aQ}rX;y`r07#xA>f`m~rs|N;{P` z`~b&U7teB>IeeDm%!@B@Jbic&?K-=#V39X>Q=boMd6d3*Ri8I^bDtNq#e|=mcF~)= zh10!UKAqR;+?Q7NojsMt>w2~#?P5}>vM(R}i@?)Y5xki6VtC}-so-$P5&lLJ(n+@O z)YRd#6;qL470>V7slc!g=_nU#l@RJtZI&~5my^;*u5HBJS?w6f1K;2wXTLYMhSzPd zS*H&!6STcV+iJ;I3i*YQ$6V*~tBGfu$+Hgmucdq!Qu_A3YnZ1qkd}qC5}a4#?9JWL zmx1ra_;_=7_NC*zkbkf3^WZ$6f4{BIjq^PIy{^xRGv*BVclFWyDZ;01!S24PI2ZEo zxA&#uT)@BY>6?P{68?Q}Uoy_rk8KO;shuN>@VWB6nSHPP-i*FiAIs`{?TO63xBln! zzB8vjKK!}FlZ*Z=^%H$xaNRNdrxgts|NOZ-E`Bi6+5d|#X7~N_FX!~V`PZ}i{_VM0 zeQ*CmM&CQo>tX2W$Sz0!yMN9`eqG_{fA4=nPoejN3t=~sab7|CTQ)NI1nF;i|KL1e z*rUP6fGdI1fU(9M{5{~kz%F2{fd}seZU+7p!x-WG3!Jf59_&W?UBLecyb<^?!(6Vt zIQTt=d5ZIMIG?H=9&A8*#cdY{|B~=+!-IPX*IgX^JmI?G!JUM6U4&f$-ZeZ}NqG0g z!Bd2H4-akxcHKTa2>X8ax!W(Ioc^<}J;Q@5fKTDPjOcqW4(>vF1;>W@el+4Q;voK^KefNS{c8>u zU<21VnK_xmK^IV}1g6v~e7s;jjB^;DM0}>Q0A(}gINTaJ(%R9@YC*{*suPiz(~q;$ z$y`%h4yVh(+{p;l#oems=41$yWjb2B=R`{m=<-(wTb{=M)Fe_mRsL>4K zEa-0Q?uqm?9l$r^e41=8^cr?=4)MMte zsi~)F3r{NFw|jF#QNw|wT0={FCuQ6~ACuEogN8!1Q*YNtpc&n5O_A1?k4{4(t=+AW z<7g2^Dbm{Uk(h00N7F~5AuX*3jvZ<^7-{Nk)knvk&{UwLK+T$)Iyw$Tn!4J!(sf0S z_8cXR=cxJUk?x+RBRz#5f$7`a)YaJbG7+c<;@dj8p71Sw%#WcW$g-&i`JkXFy8!m1l&5>_$?Ri1eae{hzi?7MsbkW=gLEgG z!a@)0q(Z@w>(bL}jdZaa;zw0nd`CmgY8_Aasfj@KNDl_A`a~bmSu#-dx%e8^p_$_{ zq=(L=n(2bfYB|4^9ukBek&lr9?X}SHAktmf_pu+NIQYH|2=tu}wC5z9Q~mcvqj}6b zdS5ip%fIS8OYE5lZ1kCX9TId|_a}37BIQTPbkKJ?sGMjlnrAP@8$+=fL7j7=HK1cI>I%R9zOTI2!4~T+$Dk zF0_xf3ZH|N)YYRw-z(r}L$vGE-qA$L9qKuHw4-}T2-dR|dqkn-D+*RF3GJa%L2)T7 zKi1yS66!wD-P76`>O9un6KZKa*nXrnw0q~yZMz!wY_8i?y>myy=K9^6cWm0ci7nk3 zS`k{>6lrb?b$2!_FI?ReLeW%L+IPb>2wPnbv-^?mUgl-Ee~KNT?^C@DQ->e)-OxPx zPCRtDgVy&O+5U4MbYm_;Znax4Z#vb8u_E@Ci(Q|}+yXCW*A-kyHNHOAq%UJv6kNbw zR`61GNx@6l3kqJ$1{9pnE-H8tyP)7ac2>a)*%<{dU{5M|KI>I*E_*=1^H@~DIZRXV zT-K%F5NlI#HfvPy99FO3*{oK}QP%PG+?V_OU8p zt3F9A3~bFeFDp@SA}dt*JuC!l^XX_!7dhcudT+G$ra=4fKS@aZR1NdCW$Gs?I+FY`W@eX%h2#mNs00l3E z!uR5H<{tQ-7{22(@JUp@Cd%BUU2e3JjZ$S+L5m%J>IIJL1xBxN>=hWj%yFZ@jRFS* z?h?37;Hbcv0{05sCGZ)6LjqqASQB_a;5>mZ2^=Sl}vwYXv?daHGJr0(S{~R^X_>7XB+i(ZJJm+-Bem9d{WRicI6F30#<^V=CX{2(qZaOwhx#b-LH!N8m#80f9sFb(|@1 z06`nkLjtpTs&s7B zH@a2FMt!gE(6Le9#yTAv^}PT+pzE* zSs^w6o79yFTSTAR8M}f{U=L&A7~(GUxjTS30)ZCATJ&+xaZ|4`SJtr0?enq}e2=>D zeU(T*j7ycB_#7f`)DZ`rKM?2l#rZv)SL!Sbrh>;kMhy6E62M`k8?opnJ=shb(7cLj z@Ehr8#^E>UANDzRk7Jbm3v%`1^GYhqIw4A!i*rE=d+tPno`A7u2U1u}L*MG%#@I`# zpo_KBe9%9a!XA(K6DW!s9e6aAy|9e+Y9W6;=ELZf(NxO6SDWd-DBIm%1OK*zq4 z&7RVxj8Ye;W;nd;`JL>$5ifNS4vZS)pVE@1Bm#|kP~(?V*&`9GOMts-hmwi@h{m`; zSMBv=b}5xr9>Xo6B+jDA>{=>&`Yz8XYX|f}J|~53MtjLGFej`nCp4cY z3dHOGYBuXX!7%&iaC_M+4)%l=@X#dV^cPc!?z8AF&^?~0LD%KQ;tTR`;ATTxew7bW zN7I4wQQw%xv)7Z*Ke3Z;9=9U_zEW;#*H>i3NF*zY?u^Y zrgzye8338C*>Ii>M{T&!h99uu5*zNd;jj%qX~R`Ee8z@rZTPGWU$EheHasA3uA4o3 zV${b7^YPDSnTEd7kYPN_3{3O++P#?nJ?J_*^!aa}--XoZFgziN^fk`NuRj z3kwb^Z+VXk6Akl;3ITR%33_>y_-ZQbfhU2?1-A#V7e77b=IdYP#r)4rVa}6IJ^xgi z&)KMNy3?%h>omXJO8Fl{`&y76uix8xe%u_5o8Rj`vfrblr;-`+%Zj*Mr0$siR6lQ4 z#GPPb)IV2qIzvQfk0w(6A31^j3B*xgPiBJ-j@|G(27cfK8+E?0%H{W>{;xV%mC^q~ zf9?IfyvNDAO7Ab|v#me3X=oYB=f`W*e{QS)E~l~wl29I8DcbC1uQ_1P+@sM>a8voP z&&qhBJ-r$^L(BfcKDw!YyFzazv&+lS-(H{D->^5|G3$@~AYXZc_bw!Kdt}eQ@|a(h zM+_tmF8>(%M_^~X{AMGD(ht-E<73B|zmBP#A|~k^Z&z@y=1qyhL0SGQJ%t9}!oifNg14p61UKlhlgYOHd?`_hj`2uc zh6Z$n;jc)4PU41kY7o~4!cIJ@z`)t;;xXnSSB*)6`p@N<^K@N<7P4Ub~r2k-CGT1Jox)am4ZrrYS5!Kgydve9P*DVVb4XWHnqf>DJ&+eV)g zggQv_**1D8h?ykO=UVh{r`r0L`XeVu%4pSRp2d&xJgwyhqY8b#LZ|Vxj~|w0VtieY zPY=>&o6(=d5A!4F4$w1#e17u&UdHP;!=#gZW{}TUF&-oj^C8L42=e(W$cuw{H*rq`C0YX^RwoA zf@=RTUSs1WEP~6aLO5P&fPGDyLhW_E<-EG7RMh!OGGt1B1R{z9K&SqI48uZx7Ikxs^r#V41+TW|?q;UIW)+gj=0~Vdy zpKg}#K!4BmXIkyWc+FGl+pFdJnMbbA0;T-e$@%wyAbmWuTf z<~y~&$}HbS@nRm|kTKi8$RCpG)2rqAX_ai`*Q+h}=PC4}6mCz=_7-BkSp2b*E0p$7 z`!mh*6VU!rf1#4!N5nZ+duhBwX8A6RcV!x1pPS=_g@8F;WFHYMn3%>7j~J2j z*K%}3e_@Yf+B|%*>}jvI#Lv8P{gyIVAUeFXA3^<*OIZl}!}Xsgyyy?AZ=OUa{v{Fy z{~SgNAn_NPbO+CWrHQ$IW$+D4iND08)BIoV51Zqe4P3-;d^g6I%3o*l^YNR(Xab1* zdNt@92GHQ|)u2BCqy04hRQ;j$h0XkEJXWVsylBa zAAf7S#Q0N%B!0R7)&2wjF--S*@K42j{Xz=j@7>=20bxv7>`zhsZ2LPW3yrJw@VB&H z#A>LV3y64wM^AbJ#UCdFHhqGn6{A7=ZdbR8{ia!ng zvd2eHXzAAY8||k6Ptu1>qrH0j(`OmjXn&X7{(+lnzuMl7x_&6*Yis|>9JWpGFg<>? z#bfc}ExKM{wO@_T;x10M* zh=Q(V^Zj~SU-12V>~CQ|N!hP2z(vtHuahucPoeq;(y=~3 zeV)>CIra;DnqMy%9i2N&>tEWhNTl5<6Vvq(x-QQ6elq&^!3zL%VEnLO*Q+J@S%%?X z3FGq5faQ-*Ywq09dqsPxKeSnhW@yB5@cJb2`Xtfy9*(^nCkPzVX0vdZg$@4Gns3U0 zoZra=Wj%dbOG>&T(NAk$ugnj5r~~3{v48PIWR{!u6O8_&>nUEnG+w{cS_;Ayi$4bY zE7N02yh`nxhumVii{<@E7awK$Y+HK*;Fal96?#ltz?;th@cvtfcp#Q%VC)a!dY3Fe z%_gt28vHSB5euV9*>xo&$BY}qxF0t zi?tc}3C%l;=AV&&Ov_T_e~#;47!VwRB)XmkoTytg9bfwdn-j?Ku(_@%=BI-J&q`mw zG4@M2c4y=XjQx4gy_xwI{c1N0LEUJMV17B&{UP+HU7ytJCzI&2ZV&YR1yg@FdE0q= zD4x3!{S7MnHTL}&>_5+^{W7Efpl|cT5`Ibxa(xM!`kl^kx~bn@FQiR9W2(rTKV$e5QUUaO@Ts_5tCg zJiqMv=kv+=VLviK#xd;2srsnSce}pR{I}^Rwj1U0?fRJl^B~i4udQT{;>~|mKXWow z{hX)BpVk7}NvX=F|SS(OAhFtzKo4y2cRQlxlsOo#+ z995r}+w#xMQR%8a#j_`}zdo%MA)t`!zrvRPtQ?iD>SIh(?IYPk*jCqVLnWK210EJ(J^1Q{QR6Tl#%k zOND+}cyfKm{Qpw7AXC-%0IyGAe0|4;l{{b2kzw9}BEAj;r0_hUQAnVi@#ScQ1!XkrjK*- zRJy89F>S2>RrP(<0#%<^+w#xJQ|YQc#=AbL>iYsj({lUm@k1V>Yni@CDW9(A(e{m` z?0 z_3v5A^@(0>2DT5;f9Ca`!2A&Mv+&gUv>#>Qr?f1tPg$nE&)|55sqZx3E&V>NO@)40 zcyfK8?PqhW@#6Z98z7DHVgIvKeVjEz#!qRpd3|P&ukSYhnr+kfIkT@Q{yl#ZeOKdO z@=vP1()D)Pzpu2#>vnz2fdP@{vt3_`ke@hsJiNlD@2hO_x?LX^QhV`Yfn^t zFR{hzrHa16KIW_XuIghvd!p+5#PRxREg!3AxqlWb<@ai9{Ip$Z%oq6gfKA`0^7}_b zd=GtJnX@Dy>-(w&MG7v-D->AQ_gVP`79IM&=C`izd9uEL!Ml$S|BdN8=^Iwc`uZ97 zBUV26BQbf9iu>vKeptxrFYKo)*S~u;=wn#64~VyaWa;Bm+8nNL#(vmSS~kborvBOW zaTcdr>+3mlWc|hc3-fbPLu33-YuVYVKF*%4^3R)JtMD(FCDXC~Q}hw*dBo3_J%YYw z%ld}+b+(FE&X)D%v{o`(p5M5?YPLEb?e-)A8*Wy4v|lL47xDB#wHJjG#s}o@)%Ct= zkLZ3MHU3{UVLTAiRC_UbyuZfg|5bYs)7IMRH(7jd_xE=Dr`jvJo*Py5-7V`k^gn^( zSJ}T87Pw`7U%6_Df=kxGf=T{8X8|G@W4#Z3U-w(r_d;3UzwX7)M{sbtDZr0+B%WPMllcdWkeR`_q}JIxM$i2b*Zy1qZ~QPcMlS>J!;y^Rm% zhp6wlHhq`wpW^yHVAJ=1`f&AK)@Rap@=&tA%l6>3Hdf#375-cL&Zh_c`ER1{y&pAw z56k-gx_3Jt&<|1H7ufXOydRI^eZ=#)!foguUteFb@P`r4%lb_EPAf{IJc{RKdk|0G z?@;(}={uhu^yj~czCZa<)AuS_-`yX+zUSHW-Mqh5=zG}GZ?5mJTKL1zcUhlF-&ZR7 zJNf#(QQ^O(@8}Nw_-~@`XFh8BUMuVSEN?X*%@51J!++vi$2U#si! zl{xBu;wt=p%PKFXsn-L>?#GSY-?Qz{E7#XgYisa(A36WY_UF{=m-hSRFkZ^_b==>B z-w(<8PrhGw_U81xu*kk&C+){!|Gs36v45xU&*v;K_U~{%-EX{Khy7Y{H>e(e$@kyl z_r%br+1NnVe_z7y$4y|EuylBtasM5Y?ZFc7W=_6wdw}p<+P{CAOY>cQzJCw>ULxBA z+~4}TrSD_!Z=J(vzVQS8=)ykCy?*sh=1@2{iiLmb?H76Vw<-|YL**#B4bJ*JU$69<&*@)Z4hLMz1VxALd^ zXXNq2ez-@sAm;sTiNr7MkFS}t(z-tt`jwwkqTqrBtCag$3-gLBI`r*Bzn^uCK6zz* z>hPBF`ro)dG39#LSGcsLPtczNS)Xvd3pbb>^AYyWc0XyJ?S4}I{!v4Jp3>a5>s-^WCz)jCVaop5M?9_&-aZ?DwDI`j+i}64igzPlEBtRsG$3 zrGM!6`HiZ7^J2hpnEIAv)3+jzdjBZyH?i&)h5q4w6Xkx=yj`CO!9AD^?uHRV4-q9XYQOL<$liJzP>5< zbN-XJ^uyM-BAdPq%lejXyPtB7?S9J1^esJI)wgM5^iB0|L7TooU5tarU-N;i_-%6k zmSfW=)&4`@@az-WKiKtctiG%Iw!o%Oc765+RsUw!x3T)H>RY}|pX~bV3#$Gt*`{w} z@7GlQ+r;`r&(Bf)o2t*GZ(XXsrC`8unEox*rf)eO{BF>=e**fpz?-Mwe4lauq_}@N zXxv}- zM&AEs^ew}t@6(m}2z|r--xKMZZmq5I=h@au=Y$vGzjMxAb&XpQhRRCnZIttH0kOeY4-shyll8>f2PCzGZmu z19nN@=6G`yoaTGS2Ut|v46 zUVoP6rN%Pv_3JJcC%X)hKJheuocbL4oCW8~*{n)$z)3hZ*pI1B4VTAsSokw55Jnu|Yw z)xc1qQ9s%0=v=wZnM8BKeQL+MUC<^w9*8Vj{I}MPi-^WkN&r} zrxg8#`8wocFFc0)JiIo1?;eDchS(=C2a}W6s^kar-@OyZAA`_l>!TnEA8%-L;12RSj^!|W9@JR9j*iSe0 zr(y5Vey>DF|N7u?MSU=y#gQa8*?*%y=WF}<{xi2{)Zeg|m|zAS{of#--;4UAzvgRq z@clEvAE4hi^pb#hK6-JiftP302ljBkY5xw8{3`yuVlI#VxbH;z9hfk>J!~VKd538~ zIi7WgDbI1}4%2>eoV!Yu&s(L+FIlC^7p^kpQ6J1F;Xg2+czxXa)%?8s)%*hcO?kvM zRod?~d_eVQA0Xbz#`CTB<9l=z&*$ayr!vWSJ}7?gf#;zlN7%=?Kf0vP-`t*6zXQ&D z_j7-ONffyQ0h2iJysS`U)y_LO3{!sIP-Jzjig&J2aczZ)*OaTce65NrHmP{+78S2w zui}~-3Bw<4DVFtZL!`U{_Lhxy>Hanp*|bjL_hxAoqqM>)rFiO<)Icb*u1cnFh^()H zy^`dst7UqzR<(tzyJnqB;?KX)H{iv#v=x9$x zDd}qsjR)p?D6)IE#P7}3N=slMF=;@DK7}H;-!9Wbk@|WS@7pI~7Eg1lj$Hbhpf zrYlxff34mi@ngIr|G;tiE5-nv!7G=5v)arE}^_HNrI(NX^P?JBOT zQ}M1{D&Dh4#e4Tkm~DvctVKLwwl^33Zia-9+fqUgJR%|Nr)-F9+zbE4 z>dpMB@IYRXAIb}ytf?VfOuq+(J%W7IIzt}MuK}+49eSV=9az6j{JaW^7m6)^i~5kg zzD^Tf*y}896<_Z#-Twt>eL(gH@5gXa`+&KAcOAex*C3yzt+d|90eVndh0Pn(e*ok2 zBmBN?(LXU7PbQuRpR3{JZB{(w;r`L^_sB0-D}nbg;-zfFv!zyi!|7|}c+P!1lCYZU z=QiWnb!d$tPxYbqTUho0@if%cqC-9b?m1j4;)`4@Y>Vgg=UW-}Bv&hkJ5c2*9`%^= zn-ITP^q{uZil2D>X}VZ+Dj)MHoM_5d+2ZN-R(!?fYi#j!Lc#{fuQlbj+2U#a`5spN zcfg#f<%c1^-juJ^e}8YxtH ztJrAzGw3_jpW@{-Q+`hcsVgJ@1v=u>1T#KGdvdisd&pmvQU8H%#fvfCeu{6+c=83J zd-?jEw+Hb>G4>bC@?TAL7oSeD*7v9%;*|tFUcver81ftC`c&b0QD*$Z>*Mvx^~vo? zpm0-)ZxZzUSd=jy->joopyCLN_-13GiZ^Xiaapp8H>avN(WBz3Div4o=eb+urKYI# z%BdNT_(yRR>oU%j3Qv)ilt5b1xP{q4n8JUs(zLl;~+`YW$Fx z4Ex5)g#XUfl2?X+4g0hq;-3P0#Qeg4Z@~JE;)@cK-ya7cRmz{{VK z3j4!0oAN1BVc=P%kjMLCyl(h=SW!HW&V~L+t%ki|X+(#4QpI^;d5Zg`rom$_PMmP`+7(~+MnKXryoCwUm4aE2+LHdCet z@%%DXe)TMwPV#W@;pwJ)DV8rr{j*^Yvrr$o{I&Y^Pt#v|JaA~?OjCY+wk1#YbO!1x z%Wn+HbSfVXKAdIBm(7*wL2V}5Bg2{4lF#A z&-3`<@9(Jo8qY%%*Ej6`+Gq3EKD)oh z3+!-Q(*1R==A{5ww_k`4s90{A(7jR6c|3*sz=5{i|0~ z1G(C|ttN)N7yZG@GyQpDBH251k??=sX(~?R?>n&QDKk}iaVYtWS)xMZWZg-N38ZvovPAj%#iVhNTQeKhqWJ& zlqAzbkqmkTJdUvF-zig6OwVJK>8n?(n4a$<)7NtUWaUTqC&=`T8&w<%skov-#dLqV zDG&dZk^=i-wg>Z#Ur#ai8#hFJKG;{C{%!LA9zX1z%)fy-#r(s3+kpLU*aMx8`MDty z2*AEbI6WQuF5!|A=&OYF_oXG0Kjq~m&VYSnDW*S8$e@7LXJSt#>^Jk6c58j3lK*A;E!>Gwc)=LsJB|Nz~CQDAbD@4>CbKND?7nt&Ci)DIH%R_mx{In%9o#e@%FEr(+FO}&* zZ4v4t%V!nHbdo23o@dI>Tqe_lcs`OUKf6$-lRWwJMW%fAa+w~~7Nb3K`E!e8I?0ni z&o|}gt&r(Kem}n~Un=?YZ0@hgpD!kP%*UX%6zv!M&?m|t0bO_r(Sd_n0ot$95pac< zn)GF8ze-0y6fQ9HFGTxQIs%sPGE;sz+ON_PV1x@zdJ)>M(z8KdZsxxN?N{k@K`$aY z=6A7%$e25D!=JA$<&gZl@aG5!!Yj=3N=q?6_4PK|pM~FNBmj=@&)2Rs`TtY=xo4K? z&r!cztyrcHH1u9NWRk3f8il+_@?)DpK*h zc`Dw&Lc-*q$$snp^xNcry>#KKja$OQ|1a27W>i8lP-Sg#(LlC6vpFQy55iW zfFI9C#v}3te+KRs+zkerb3`xJTd3}>Aa*goZsmu`)jN} zzwrs?r}kh}aX*oR0Xxb5bByarl_zG2_UiS=`$Bm2_;{lIW8SZO>@DSeB=+{y0FMu)!tK=imi$;sAYaWY#EJtREb^@q9xIdn0}? zH3bLCdkp*ub0{9+2j_R;>C6`PMy4e3xhLwop@{L{zm3wQdL`8{HXMyA5oy5F8Q$|Jke16|wjZ1Z` zk@=?KIntP~-bjj%_B(Ni{!LA$sc&E!Uz)(ie7S!Sy^RyokIQ3NL8tyPu19(!ERp<) z(f{5^qAzG+=l!L)l@l7O%O`Mo>OW(?dLzyx@&|_e+(-&u@Fotl2m2ez|0!29A8toOygnytPc9{DTw8}CQ@yrsc^}+8hD1V~r*W2{`3G@8XKb1*gf;e<}_%ktmFbiosaX%YB9FKDQnsq{uDUsS6 z5%XmQ>W}gLMkIk=kBB1$*8@`7x4X$&ilZZiosQ_ghaFu3fBGW5KV0}rOm*@<-|ptO zgXscaNMS!cF-jAP4qT5&1V6nh3kUJTe!?H{4($q&zrg)ZC!LJXZ=%1vlRbjF*PJkz zxQoxn5DyV{p?^3=ewAdu`JuPpsX1BiCwcyi-q*S5#1};Q1KdAXo-o>D@efSEugn+p zr}{py$JF@9cwRs~{0BYHAMLMH{N=rAu-qISJ;m#zywA&!ryrn_{Hu2PQOIAK zT%LYVH*WdR?#arh9|(;rUpTou{or9-`Nr|((Z4agZ%enAXur`POymcBu)hzj^2Ccb zpP%ykbM^gdl_$=!&y3eUcNxHopn=Ex^8gv-wS^Z!atP z-GTMG?jNxV#QJf0k4ul=C}_a=)Sg)4?BV4<$Nhz!pW+LCi1lCXiIBoydE!0B?{6i4 zGhmB<{u^Mub8h{;)dRPB;8qXZ>VaE5aH|Jy^}wwj_|NJA+RH^KJMdX-yA0mz_&DIJ z?`-%#cl_xG8;8&3W2fnzH{bu@z2ANBgJ!ejB#(GPe~&2oRNg3lY8Fef3h9PZr?Cp0hIBDfQ22*fvq}6vd8zX zPIoL(@Hbtw)8|M|0iAa2PC4kwjAbp%=k)th-sKluPB|UJF$bdytT>|lFL+YsNIWjL z%%g8s+IdbnWS%JB(A0S-uOB)bGgqM&yO94WN7OMedYi*J^W&a(Ss7K#m&C#e6>N-H*=^*-T z4o8v`mcddS(ria%hp-kpXIZn`M<3^oPZkZcmPXnO_~z)qD2qCfpUy-3<8kRCZ@Sn} zR(ae<`QxV(>nw}oaL``9quP%$!+K`;>!@VV^}mp?S|`-Jl5AF)(HuzZ&(LigqwZ(>=Nh&>FJ?(XrG1X7TR5uy({+M5$F8P#X#TSTq;G8MXXE-Du z3+vaBz(DhT-i!tB;=1B(#P15bSj-i3IE7IG5GNe8V#_c1y4-&(PES4Ax%JNDZ zqg4KYQGTYqd~eu`$9$>f%lRkacd4-?w9vTTg+5mDhbl^a7_#T@3&U;uVh&ke&ObSv z%wowC`Wx4(^X&Qi!+sX?kDvdP@DvuCGEx3C#|Df!R%p*ZC7i-yDdXp#8ph^# z>O}c_D1ZHC7~F4d&p!~xV+8}_=RY+(mBpq`kiXjhVdP&n^O`zGfAcd?fkCZXsepSx@Kh#IBz%Q2lDd`;aTbZNvGzL~|>xrvl>Ku2Y{V*otu#cf(hhoMo zve_Zg{~nn>bNu=>-gNzN4O$wX@$Hz|0@Y3m8|HCJ^a<+Mrqs`sFnRx}^&8&?O6AMr zuiH3h!VT81>o=_5_%?E!`sp_EhU=%@bp6!%JHCxIG}Jc#bU)}!h*@;U0fuPII?4_> zecW!$yh8D$(3>dD`vIfX7pUG7%nJrOkHMN-khfv6aaT^h$HPB}bISaFw-FZ^XG<4d zSRdjxJK@faz%LFT)}t1mQNQx$l|}S%HfN;2%L27{er2N54gVJP4#2}PqrHNTJJFnp z&IF<-4Iuo9M|Z@C&fvu2)16Ka$@>Omx}$jsV$M(T`w^L!R!n_??QbH}2~y)kgpy+M}gx6~U_Yg}>WhI2}c z#+JdA!HUG0J5zXTY3$?Eu?i)5>P<^t?v1eLX6FXk!QW1AptXGLXr{a4w>Al_iTVfp zap}$!y|w7A`1HgW^#<1zD}T8+qA54q8+qgP2CZ1h0>@jq#!DY-H~db&QQNpya!PCp z=rp#@xb&1*3dxgQiATrsgL;!eQzW|F8!Yu^d!z7HZ%C`Ml3TqY^+xzsZ%DmSb*nd| z-l!d~H)KmGe~)KXDQ%1ub^2pBWJOggD8;I}6&=^g3M*=}vXfg;J?bBSH9BroFIsMo z=&fk}6z%Z&ZXlZX%Y5TR^Ax|P=rIVkbYgM+h)?(0*p7%#Pv(9mI%OcfJR&~smEqMT zd8G$Gyl8&Bd1Fr>GmmJq%kZXFTq(^HD^hdDqhm9SVjEkOF3J0x+#8cOx8&ts2)l09 z8;={s+gm#x9@`68u!)#Kimt6z#IS@RI>nn-D^}@JOyS=sJ z;jz7dEqZDNwu$0K+t>x5R?;p&Tsp>@)3FkaPshdy+v6j;JV#@vH9%oL9XPDI?;nb8c{fJmP0)W!BdS@xjrNg$DV_p0I zCZlWplxLYg?AG%wM{f5qd%lGdAMwhca$L8eJdAt{nt{uMuP@*q;xWG8)~O@`}n^B*wj_(tXDspSut^$$5Xy-=lF?W=`dv96OlXa?A? z_FMMhJ`{#ASWJ7tFeI=@RE}e@7uVedMs{V*UVdi)o+jL=+!+uBGY;dYh(J4-th)y1KB0B8^ zO++u8gkCZUJv<4WR+JN!Uptmg*LZOuTO4EQ%JsYdjQ*|VZ?DM}720YwbYbhtj%#LF3eQ!U{_h6k8lpKH>Wu`3ELU@t3p zDZ8ZLCF}(SFQ!Z7QhxdDqJkH(3kuF-XBE7Vol)=t_N0R6vt9+~vIi7AkAA}~m7l}# z<1Pu$WnBsmu{H%~vqlBaVf6~0&1w}qi&ZIjCJQTg1}g+EGy6A-g%q600t&`+6cn66 zFJ6}9r?D#vPG^@CoW@=Nw#GZiE-Lh??5u(V?2N*n%6b)g3X3Xu3hPp^pEW8tnbj)T z$Etv>`ryrUz}9^8;w?}zPGp4&zlVi@t^P}30R_WTE7(Ob)pdu{nE&MBWlS-FjG<0C z-s$E#=_tbhz*IPJla&i^X>)FKF;|1j;a=`%wJ+g!-}nsR<6acv)`l$3c!xVL#6nPX zde(m-6#EP*_j8OjzKHYpk^Tczz9!1trCshbs3`x@BJ4kN(9?x=Oz{BWD*}`KCyZIb z`6(VCj9JAo`B%bWf!hRb6qx)g(W3%)34BIi@~=c65LgrViokgSQ;0ZWNgOD=%N*UV+aDO#YRZFYuEBUlF)UU^1Fi{uzNo0@n&07Wk~djRIc~I4bZ( zfzJp$An*WwfJ^z;1N$6qJp5M2jRIc?>9|ecP>znf1n!-uV@=@e^L0$+dmKU5wLr(r zkYA!>ufSn?;vUKm2;5kr<4l308+05J_{=69=LtMeqvJw>uV7A7ekB64y*dsH9Qv4! zs{{@=>$q0nXuFP$`ZgZXu~FY>w~meap3!t{)OX+$IyUMXcuvO`kiP@-tybWRz#fN( zg`d~ys4~Z)OFFIxoyMbA;6{Pj4|RH*z+D1&34BFhP2k4MB7cJ}u+g5z|E<%F_FNIz zXiwvhbh^==D+1>k`Tw0xFEsKOxWu6Uy-p7sbb+f3`akIOT7xby<9HTpMDRr8Y1H?M zz(#!=|Bp^L>U%|CqrR?^EasjSVpm|3uIDn=i1~RbhVS@djai32mj*f{5Somgnw z(>q{W>9D68ksYO*&p`s;I%))t^`bsTq$Y|r(~XD`D#Z_jUqF^=%g0DRE56wI4f@TV z7eT*iIKG#OD%g@z2q$_OWdr!!J zMszrhG1(Wp`~yfp7Guiq6-(2Q$k&)7hP*M3kAXmGVZmpdE(m!88|52v#`qtH@I>;O z$bjUD&n{1Pgt`Fgr5w@OI*dM|pGltVdQ^wzUpyj_=UHKXJZ+9nxjUqF6T|8g526Y})BO5=z(eqETDcwQilU%=p_bM!BD6v@-iSvXPq z6(-<~=r}4Lhe8_qM?I1DK;**)89lr5RGjiR{j)A*%zrAM4qNxJRv{m@l^+%I#{4CD zG6{Bhdin4}amr7B4a_(~{4Znn@{M^^1Nj)_!?yBEgnX|gPvskO#{8X$gn05V2zg`v zQu$<(?DEA(fH%TIU+w?!&-H>s`pNuj?|*vx>Zjtgzfs5=PkFwTZN)q}?3KO#rA)X-0*FiIEtYMkZrsIG=y*yRk|C?<$3$CC_wHW%%wWsn>bD4frY%eeBIM%$Pykp;rUANy^*0H;3Z_BZL^>xdeJNMpxxcIiCO=T6^ zEB2JPHFs|5*_O4x( z9b2~EesKS7NAE0aJGj4cPv^e+J>4xeT(+{lXirabRb6pM%}$ivb8z3@;$sI^)U~x# zb{s#@wQKw4q8-ayDz&!e72A*PTd}8b|E}V~*7|ak(F*?kx9{IueD{GBTdBO|`#KM8 zzO!t{isquG+ndTd&2pQ{mLJ_zbm+FG@*RR_%gT<*Lp5z(b-OyMif-Rpd1yydS?STu zm4|k>?5#OkUVYc5)~X$ylkk+c?yoyqv9AejtSjxPXgj)T@1feJ^75mbSL|tTD%-Wa zd_~*AmZH)V`>S?NQlI^WO?#JjG#{Cauj^RL-sJ~dDoZ2O?zr`s#t3+M*ZaGoEq9uNwm51)Svy96W?WnP2Dk>pU zw7+}b-f`ttivFzHS6_1w^XFI-=1jak+kQmU7ZmFv2*tfSxniG2tH|;Gx*c3NUYrHD<)^#1| zC@tK#x2AjlWW1f6_h{4Jy2JbTu9{?g_3}wy;E`At=*BqnZKt81_2tmD4sD!r4L#dJc{S}l7QZc<+Jybve^+xyX`}^qI89L z<=0$vWA$q;nzVk^T-PeM9NAw#-dwGbb)=(tJm0+uaZ}-b%=s;a+wT_f2kcVYem$nR zvz+|hp2C*;nq&1vTaF#*gkPzL&nSYg*i>`(zV?;tjc~1@v!gxIaCr5qk`=30lolRn zT3uLL*iyJ+b#d{*6(t8(tSBxmS=qd*LFa33KiJgV+R%Kgr=zL6yPbT*eBRkf?Cl!19Jw%U7;!SXo%Qs-dW1IjE6it@@uWO> z4a*z!vz1=hp#NLmP|{E$8hlUA;nv8J){b~>5EQ*B+nc(&?`psM?v7Ry@6vJi@}iOh z4J#U2dQNn;HuH`U9de)v^=Rqt0HOWJq1)S!^prI8bT%~{JR0e2>M`@#)YQ|og(sEo z+r1gBIdD{KXld`Hj2q}`Yr2PgjzHc&e*pw3ZwDYrWqA!u?W}is(Z4y1tCQtS1km+v0&^ST6z(R6g_z%d5 z+{c^-ke8eN8{b>E6LHsxJxPrH*mSCrL)urqbX^yH7VUCKyFSLcji@e^Y(F00S&vQE z`DsO}9y`RCDD{ugF0u=hZs<|25c@l{%Se-R zx&!IN9~Oy*{-lh<-;R!sm(TQZ($|lZzH6NHmyu5Oxq|ZPF!XcwIQ984HbjV@_P6LT z^mD^F{4L;#SAR;2mwp}jQ2xDwXz1^!z)$J5v{{40kS)S@N^cbDn>dxRJtChABHb95 z%SL-ex}m2(6733z{n|$JDP88j zB&Eys@ruc0ua94(Q-8|VJ6EJHwxw?qliuF{^&;Kg|96OVDwlk_aa0KEcov&*v>EU6>l4?iYaak?XTgbgn(!Xp0;CqzXpa z_{wt#WT{eEh z9-pXXX_1A@gU0UK^+^85dXGqd$w=V zyLS80mSY_#{svc+d{jD)HnlYL9BpXf71JAp>gn~=q23m`g;+aT4CY#C^O5GGEv>uT zJ6c#c zNKRw09V6UF?54K`$4MAsyR`o_HMwaFiLXms#}{+kUN>!T1aTdVAH`~tBG-=Yx7ObK zoU@NG9-REs+%)rJnDd?eT5IjKUu&IpwBxF4t_}tQLVp6{FNKTrT_H|=Kj?rYUnRvj z4^yZ}~mmGQj-skoo4z>UN zUzR5iokChL5PG>C?bqc$eEh&e4;-I~-!~t8;P}2^as+h@;=5<#>8FPey!h1c=C_|7 zeiFZR*Z$ApO>cj1cqV?g;CIFDw}w|f{rd1K{9c3KSoI~D@4ofm@dH~PI-Wm!L1yDq zPY$2{+>^sAUi#kffj9qVIP=h3!>6Bo6Zx7lnTH=fzUirlj_;cNkqy;e0nqGnV+e9^TFc}e0kUL@4ore@TTYY53hLod&4ih{x`#Q54}D7z|(Ib z&!w5yU)*{8fh`Xo-}Tm0!w-D!X{5b0TwnOoaP%vuhU>pac@}5tkY11P#_OI0{a+of z+w#)zrYBz-&b;}?@B6 zb>N}nv4?ja|L)eOhwC4DdieEg(bkJche7D@>1*E}j@Dh4$;^J}_{!HFJihC?{lhCk zcin?849ETzyq`i_v))3U%QKm+4qXzNFRGkkjH^TTz|zczgO!PCfd zMJDs&=ZtElYITcb@!aKDnvxn({`x3sR(@9WbKZGC-N?aS}q8_2wOXE5{rGO_2~+e4YN z?P2I-1Uj6Px$DV-Ic_J)YH+&oT zUf`~Q%Qt)r*LNe&H}Ja|zhB4i?f88fzu!q7+HhOo(Eg7DKMs5Ye*YN1tML1W`2Ah{ zK9ApK{2svXGW>oSzyBY8=ZTTg`^3w=;GrW}wP)SHj6LgvGxppzP_yUuV9lP#aewq8 zap(l*fzk2e(2w)Pq0x{yM16vCd%CXv_T?Q*zkPYvK+aQL*Wr34T}Phkx(U~-={o#W z*Qap(F}e;t#X9oEikNemgwUxft^Jo#eIFm12*z z^Yvev9e4D6?@Aey7lN zJhNfi@{x^|xUMhHY^b4o(=r?D@jHjEr)DI&lh^OE<|p%I7g;r`1z?l^pT$J_m^A04Or70+ehxg0!~ zgXd}Bd78nq$?p^%i_Gu9vXKoBFB{qP*<~YlKY;5mmu5D6dD+Ou1ItEwA6YiC{STIn z-1CRaM(+LNWg~r$BYlB@zEQp@nGJuAeBZ$J9Nhal>Ucby+3+n&3o)NZuS^$zcXGOT z%e>^sOvwBNd{ZlfheXT%@P?Zv@7er+at>_@1`lnH1rH^M_EvAXD>SnCV@Q9e@IBFz z32&SN-3(fLHrx{$*?2tXQ1qRht5=A<_C`B2vI2F9SN2wKd^hJ%$8DkaF;)sY>I3h$ zN5!W*g;;YM>sAzN)~1z959xdLC=;f8_}zqZEAhLz`S(}n{rgw!j=+p*dC~OhmWr=_ zAsP^e%AuIHGEKLN z%)+YznM5R*x#Ds$(oh%9*!7>x)USwSHocY{sh=xGHl4)pf6K{u>)m?b*39ObMqC47Oe0JWENX7=Om61@Up+B6N=c9S%T#j5$ZY-dk1?%8C$&4h zC?`{0nhTo8pPkiY(ola@Fth2W;Pq$85t0|x-?$y@@)r z?55+%5t4~+hw66Qn!xheJSjP{=_TlFT>!j`J@re(%kAjv&3~92iSZ4_kLb@ z@9J1!BHe$+d!KYZ)<5=sB2fNK$`=bvI;-oh)%W_{dz=3l?FA-}Y41waeyo48Q-1T~ zl(%F|-pyYon$L0n0OcQSasPSBKXpw0JnT7Q{e0i(3rA*wX^zaCd;RXc7-UQ9Oc&;?j>?Pn zP@VwxFEnpIyG*?NlX>G_h-|SBgntUTmE!m$$PI)4{Uw^ zUhGxw6MNpfGlcfSXg7lPbFg2@&5V90_vQ8u@#zzv7Heun@=$qc;M4t>`)b3<5vx<| zp}DSns`zyO!T{fkf{)n5^?m0qfwadlA7MOpczB&}^181KyyBkbwGq6Q=Ol*%|BY$n@pyu+RUZWY1lNk!6LhWm z;OJgl6R-W4kLX(Q8lY?Nx)5?N|IWfUjsTZqF4-%jj#_jd`cbhDbas2_yv#>u3FyQ< z(0M3UmpoJ#h~-@wNe*|SobHn)kK|1vfonx)?U!(^=sZIA6`h^9wm_#t z5AxNfJ@hO$>FJyZdT>w6?Fz{42f3h!>R0pFk}u<$=%G3KE4bG5?8mjHCxdHE58c=F z(6yr@C2v$9-^G}>cYOQeH@*>AnF)nCToc_B#i#fEIj$Amfv@43 z=%z7A*F-msNxD{aSJ1UXcj+6Qf8oxv6<=3lvOn<7lE1_?^0fy7?{q$c@riYV*UUp= z%~wNZXY+tJRWHeGp!?o6GqCKPe&lJ#+Ii#~YR&wnS~LF&ydU)N{t1)!BN6b9dz$xp z@ZL^qCdz5vPkam4#5?u|fy1~~ymubKwc>rrw{fld>;&Ccy!X>Jcsg+3MQ7zYQ~E{) z^6GW-1YOfU0sIG^({obbor?c~brO88s92IAUTI9PMBkGAr9F`M#uB7o2|4vYujZn^ z2A_vLe6I8HIf``L(|mFtV~humGJSF`qHE%F9Qb?z*P742!L`!OmKSlY`K0@rPrCN> z@e$KJG}=HX?F$;-ZXk&b(s7w$9g zas=gcpH&>^cI4Xjzi_R1S@L(dR(-bb?{TeoSwh!}mlMciffr}K`rl}8D0#?=i0jLt z=a0Vve8$CH{{SqOIO&z&2adb=B;kaMAOA<-Mi+Pe6L8YSmH!O8L55AUs zzW{q*rXB0N^>1QL0(q=gIzOfJ4gG9%jZSZ&bd-rvdN=Bs8u)Z4(f>`z1O0aNAH`qn zU3Ez&iSyj+_luDep~>|J8X7Zy@$9_JPeMfp{`>#DIP)K{{ry`g=fF3AcYfxXd*^1( zh9UgWU$x9Yyp1yD5*{_F6&(b*}g!9FFI6vc>?b-qSR=~au%lzO{{H_)9 zywL%>c|fF#9|A9XZxAn2*~CAHrtzWi4?gpqPdyV|(lxCG-)*6Z50-}}Z{C+H*6hFU z*0Y)WK6duq8^&e!g+6}v8=>pZCYB_o1@L|EJ~8se`))k@_*W(!()^Vy^J$rVHOu@5v&{FkEc4Anz7Lbx`SuST zS+=t_%Y0fMw8+minh<{%2X%tMhA_4`nIKuYbh)kD~q$n+FzL;C}7;3(SAj z1?K;+c;*g&#Qtr|Qh&7G<$lC=7G&8@KI+eAevc#Hhw0^o+IF(h8}A&S&6)5*&-TCU zz203}&Qsg7%%}C_w=OWhp3k1lQoja1V*S6Dh2Bg<-5;h~UOmXBkF(hic?4*dK)DO)bHA*f7#*FQ&AfG4 zmU=rc%Y3z2<|CcXX5WIzH>-THEc5y0^tF#z?n3udT8AG+d*mA`4~w~nFBx4U`$+oJ z0`R3B8p*ETvYl^IS@L&Bmig9YneSs+=4;C`-_=>>`>(Fg`!L5L9_ql(pNxK<^r<>a zefp^5=T*!z^h~LqYw%0^C6UHAj&A^8_~IC+6{dS_Z{7Hf0CKbhD3$UA0?<`)jC1oX z(SjcfHEhOj5?{>YqDWw;b&k*G_4Xa>xAX|;lc=KfAfA#BV$});gCaC86bQom7S6+z zsy?9t5ys^X8JD(1Cj`Z}m}Z^rnNiu?!V&o!+$?gEVxfqDqA8SfPS1v(Uc0xm3s?Gg z`&Nnhx4mb>9qUqJ&K-C5uAg(qU2}RjZ@lBSJ6*hXo5FX^aW2;0xwX5qSIp^MpX!|> zS!-Xrt#?ajPfz>0^=t23x4!-M^>?(Zlr`(`=(*FrwzIqE?v3}{vt_-5Z;-fW?!1Lv z?H9MN>)qbHel1s`D(mX(S>L{{XA2TG-f{cYTkh=aU3k^}YcIa~y7u0!o$a^XX>aZ9 zb*fw5+1q(F-&%U>nycF9wRhc_YG1c;E7jjVcR~AH+?w0&WCu0vjo?%A^=rUJ&xTHW z{kmTtM(p)H>+QS15qim9zvb6L+rUTXuLUFP)^}~Yz5O=3bL)E9JN_G;bBsF}W@|gQ zY`NX;?B2kGuG_w|_f8t|jMm2B_7k^*!O*M6L z2YPS(8uX!N$JANA`)f0Vb>A{WAJoBWQSr{jx?GRkN9e7{d4GSi4f)MB(SDuW>(!H{T2vOS?m(%29=fpXvT#b^oxspAS+(Y|${Q zx{U9`y5sxYg8RjBm!9-^+0G=8UV*$xm7i=R^l8BT#NI;ZzPt-N4(E68CsabOsz;Ng z>)D0-A+`+Z8szUNK2_=#UDF5r^wDo&et$Y$E3DD}bgjjgLeTwLW)%+7G7H1}Be-lb zymPrsrgE?m@`tYJL+zvo(zX6(tet3f5~ai;b@_RO?uklnzf9fRpni3`y07Rq)&Glb z?-kXaUkB(mvHW}Z9)q#$x|>!kU$G>y>`uEIIwL>S%VbA`k6heFx4L^%81Kul&L+s} zv~f!(4emtmop)~OnU%ooxgIv6#N3PL%%7FGiGJoZEfPz&ZQQah(X+j$cm3AH)@?n# ziFNC5+jz(N#G2ODYi?-2>8jN?tZ2Qi{i<8mTy@>@tCow|t%-{hvpemz8xlQR+vhf3 z(wRWhL@U|d;AisDx>tdOouCp=!g>MT( z7g66OVcFhfqDY7`Q4$XRxI%;#zDUFkJVjUrE=3jMOp%knM4SzI%TE?(3_J-OhduRA z6hj7{AYKW1>l-hQ8Rd(`Q6s%b90qpjjf-bO!c%{tIB4JkvEQg~oY)%*{=_YxFCI10 z^C%?9!(U7c8aOH*GO#6f8aP*^4gPovI1>nJ`9?%KC_ET*g@HpN9q^_@mkkU%xA3NO zCsKW_Vb+%x31Lg2-g99MZhqCx~bLzi%IrYCUHD?+OZ-0Q zz_$(G!}y-TH}QF-;d^)===lrKgRl5H?&F)pw*%iazI*WJ(SmC6?#}sZ+cu3(kh0iE_i^4QfQhOZ=#}(e7a6;j3g&P%4DV$U| zt#F&dmckth#}yt@xJ2PI3MUk%0ZH`iR5-5ipu!1-A62+f;k^nc72dBf?LVl#g9>*j z{EWig3LjRuF<0h4s&G=_V+yw^{EEUI3J)pVt?(&@(+Zzacu?W93hz}|G|T=zsBo>q zhZSyA_?W^Q6mG);N&Iyy+@Y|ga3U(x;|ez_yinn!!byeO6iz98OyRV`LkgED+<~nl zwVzP9Tj3Uk(+amKEHDR9{#z7|E8L-QW3ftyd}5^k@D@`3THpZiL4`Y1dgDZy-l)>W zByag*acr`;e6bi(_!cf77Y9qc>2YybVVPbm2C;a%?H7x^3hVOSQ@rVMaZF*I9xpS~ z6AH`pm^fpkLs(N~eqDahz%h})gApu0ravYQ`f#$`Og{sRAunWoqoOfk;$y&6UkTR- z{wgFUdB*~A+BR@HjR)$F?&)s*)E{YuN&b|6P+=M`ghjQ?PyI#thZNTJH`aLTpCZyT zy!C@Wg=PLJB3bLrUnYj?y!p#Syk26RzguBRKhkHJ`I8Fk{CgGF`3L8C^G^{&3d{U) z(bgc-W&W7h+vv@oFODfJ^T$NnTyOp<;tU>~(ft_+K>03~IHBa1R9N;8$5zU8ny)B5 zzDi=6kEs0N8ziRriqbpQNUZCRcS@}5PbjSGPp_5fy8a=Bb^RUK8<2c;{d*Obri zVwP`HSf{5I*7e10nV;rQqCdGqVwyjR{x*eaz9jw={W6{AQ%X-NO!Fm`KeJ1w(|k(l zgAbVX#~(H8PbjSGKllZ+{>DF&Sl54SpTxTU{@|H+^(JbGguue|{W6I_EJr)=h zhkdy530a=xPyDngO!6as(h3v*%&)>!Kcyf0l370SWr=n9Mul6rJoIV5Out2yA5vJ? z--NtmtlK;Ml-XYLpv1bp!DA9@diN@< z+fTkM(=8tF7{3ZPDtZqqyi=t&{x6w7snXL5532bh{)$Z3<&z5Q@@M`*wWrEAeqUmu zFBZrb9SZCGjsIk}*P*aZ7yn15m#F%Z3KM_S-v<@e`G1p6F>eKnp_;APDX8JK7PX4Qze$a;#XUz0Lh2?sIneQE$Ud#G1D!LUus_>A) za{V3^ZNoCZT)#2i71re&|4pXr@_QB5>G2<%>1l;^`Wat($4^Z9jw!6`PyWifqeWyxKE1Xp6XXvp*q|5nwRCL57*5!{Wtjj0!WV)rwA5{1jg%b!s zq55?GL4|dDVw{=Yt#FAdKcsMn!fkl4+*{vag?0Olg)&{YzgJ=1ejJZqQh8l}T47!P z8HIKJjv|@g;a_1*e-eHSs!x{>Eo>CwvV752Bt#l>_d%M!@GcpCKpYW3`pvlCw! z>%Oy{q=ygAVdC)uQJM;k(s&n>Aef`&;srY#A}C18Ckw;*DG?$byW7r$#4FrBMl9nz z@%M$|$@{~jVk2(Ez)@9!`20>QoCX{n8Z8tj%f+ExJbZ8mubMxD7g_IO&LuvXi~Qj* zb+pbukb+=jx5BWt+bn+)FVeBCd=eexf%+T7mr~)JuO2v1KPrGh|6{50Q7Rspgw4v) zLf~=bMDJ)0+J*d@qrE`%Nr%Kw3q?bURYhW?muUG>7Vz7=>l(Vk2l*l2fmB#CK=KiD zd&8`3{Maz?K#HCeRv+MC3j1P$WOshP5cEgLbtmb6u~7VQTUfGdN%;l)A`%x8z2)fN z39>x(@0+b-+W#czT}SmRem=auH!T_uqoempeX+w~riaw<)k3S(4h9_z`m0irNCHWa zZ!XfsL)%A3!dC%<)vgoAz%9+0; z+T#z}za|CF6cn$K{K9xGh?3v&La{R{MuTOnGM0EP0R3MsbCZ6bED%rL4gJI(ll2qh zE8i9&>T%X*)c@q&iBWbqk^WYqzxq^v6hePqY!KC{<;$tPF!BGwZs^YzAo{rakYJ4e zpl^ez7_!I3=n3dQ>t9|h4@p}89!tfrBDi=k75C*YA`_Y?Kh3}{CjDc5!adm@#*fin zBy5%s^LW+mKi@+I)n{FS`15U}R;(6_8!d|`DBB8P+-Bl9ZI(Ut z#XEd>gAaH6aLR|%CJtLZ9CxuXp3&dU(WGMD?LSwZ4d}n0L!Ng@d4_2-%KS=sh7%e~ zd4_lT@SqPr>ce||c)t%H^x&u zV5-m}b#`&CMctusQ!1D%&na?$f4`l>^`T1MuMVan7Ii1*$9QbA!x7m&?MGTgiyh|u zt_b`LxR5aK=R^_c18z#?gm)0RyniVaMX3m%A4ZSv=KM3Tzr+4H$oo6wF9!Y@wk~|$ zp!$FnzqvG@=k&zbJ|G4YLg0J>dg%O+=K4<-T19qF4zS?*ArI`gBF=sb^;tbpY{l@2 z<6AWi^hKlKAG>4l^J2M(Af3)%?5M(dqRP$-6CMpB9sBzkc3zbBHx}hb{Y@z=hq_Pm zH^a{1pg+wI(TDy?>+~wfFHGgBKF|aDSYF*7KMslSqrOU+E&{6l+8kIE=tJ#$^AF4X zC7yK9Q#_Ev=c&<)RsDS;hxLQbYfNuWuG4?0FEgAZS02{tl*caTdUb&NE1#+Z$o}^?2H2$MUIv<@wAveveUovC$YAjHo{B$BR>z zMZL%TpnpqK7R%4#^C-@ku?3PbbVSHry7wkAShYz2Rao!;6)Cck@wxe$ULq4BP zk+$L!csNAozvp^bo#j37|5tV%xzO<`zrg=V|l&2zm4gi|X%Akp7h;T|P6%^}W(69*T~RTIi3MNXGv_3yR2P&a?5z1`5Su%WMHzo7YFpZAJW&7vHb^$qrorqoK2*7FDBtC6VY z6Z9YKAv9g!gZ@5_^Z2eO`8h_G$D2H!5LxUUf9qsCvXWQ-_NaF~!u~2M#YI)3{wME_ zjj{oU^s%M@_KIkf=_7ju&bQBQ!|yE^e?|C(zFt4;nAwQOX~k0C^p46-9Qg zoG%w*e#d-_`7g-xDa~KdKg=Jw#4q!Y@nftXNQZo*G#<3PN>hk+lD+T-;O|48p$Bfc}V&ic^4W51{!eGHz&ebcfOx*Lvt_aCf(9ZLUTFKFKOs4K724XD4!sJ|(dpYPqz`1O4tHGyrcL=x#E^erEq<>(vH zQ7pAqS+KUk2{QbZ`tBNJ6NbhVc;Pr6`})56H*A-)l=wO8T&ilUsUxKMo9kLUoY)|{i&GV(;`Moq&_{C8t;x*)K`^?TU37@ z?=M32{iw+K7hXFZOh&M!^1@V+Y5>lR9xxDU@D(la`vaRo-jS6k7)m)l;H7% z`5}+SdsS*;B#rzr;urNdfnRU`-=B(ee;^C%)BScn_cvqI_lO;h@_Kifl;<8hF73NY zzK~zUkssQ7ELG&n8@Mo53&zN_(fisys4!`~^N{xW~3te2@Z znm?JpV!uAY17z}7MA@0Y@-g}}mcO4vf0#dyzCb^TzhcdwobTNJdz<<5_1}v={^Fkg zBY%wrk0E`e{tTujoToqK{*u;Dy}n{RO>p{`=Z}axoH=fVZZvXIn7Pb;p{3Fh1WA!iQ zlmFQHi|KR5SInKiSpV|e`AX?u*y$hN_>%e;bLNjHXuVzM@^`uw^R?yUFYmni2K~x& z^b7qzmOuaa%FFZXmv4TS=WCjunLkrsNdNFuoZL^y`6K3=KU{rb{#p$F22*+Ph^zSm z`#Xbv;QLc!`eO{eS@(zBU;E^p^wE>=%2(3ku7BsIXUzKc^U<^LSB##O%fj|w+e-)f@)bWQ27d+igS@}f z{>_2DJSl%wexQ#8%OAF{U_C^Gbe_O%%!gmwCZ@>rAnL<;gVr-T|2X^{HTW6mi;D5O zKD1BgYiW?7^|3JMKTq@d>8Pr29nKdxKXZP^K5O~T$CmXh>UYn_mgSs}ah@Z4nR6Zn zZUL6(8EOBm>Wf-5-|O>k^R}=P{9#43PY*Qn^t&PqDoe_{qB~-TvA@=aXIuYXD`kw7}z&&mE>YV9=otIX41)ZZ83J!<@6edF^c6;S;J`wbox;q}9! z=Tk`DykAu5pHIaq)5t8_Yk+-bs%xM5ek=7SoA(exi26hF{7Ci_?|GKalTQEcbmeop zmD_XrSKD96-{|Cj)XeXbC+r=$k|4Psr2V1yJVfJdl-Vc!Efju!@H_w>n%DeMe}}_; z#P4Cfzbvu_L5`L;*(<3&y`LQD8;=7ff4KgE)Kqi4r|N{%7vw?vEPtxdEB~~TKgRn& zs@BLqkSa5AU%kQ43pW1B3V(2F{(b}Z2cO5h{XzBw)gR5f)~WF+&-0L9RMIE;EluU} z{)_FwpqIwKDZd(zK9D@CDv4n&pBHRaZ%_n!s65$YlafN3KOB3KI`5nPQ%?KiI_JD} zvX$nW`f?Q4^UY(ai>5jl_HASSz}y#_^a+zAB`WmUd^QaZ}-yS zJi@4NiqP{%Wr1h^lg_8~N9_7~XT8FFHdr#V?muXIuf}8Zwko&%pHlm!`IrI3s3p(a zG~as1BhI5!%<*gL(_^X9QtBTM|Asuk-%N);(DO(tUaIB~=8xpb`^kl}edu3I1_#RV z0RKU)Qy%;o_A&Gy%9oW;``svy`OlayAa7Iu%1QrfT>T4>zRUfd(m&W!@ycF9{&>Pr zowsRxko;?D{9}E9z3=|iEZ=zm_E_Y{N1eB4O`Qe6=P}U#SZeAlM?X+s^R|DAewVZ3 zy!_J!@KeU^X?xOOs&1x_o;sgClsWo<@#)%ENFT~sADsO<>qFg4Qy*TjyzOJYDdqE^ z(ifU9-0?yBSyXmDeL(-lo&E)X7!Q7ZaK{hmLI2J4_CNH$*2^#Yr_8Gl{{F{#K#dRB zpQJwUe9!s-O!jf7KJ3p7{g?&&xiz!aD-YD?_BZmQKV1DqefQh5-TnnWhkjN0^s9`| zZ%V)JPnEg)!sB(;EIh8}oY(#OHCA7C`t@bDGd?80{`rF**znFD7pN~W-~92JG5W&$ zg%8x1T33IWK2v{4U(|ZW_IUq#h5o3O^vnJot54?qf%(QAKalT-=@0a8wljaw_^@h? z@d16D?dT8E4gJCV3;l5CC-6u5=*$;r|9*SsOlLkgx4!u2qhGMT%=hbyq0g9K-0=Z= zLI2M?zr?)bsd-!D7=7XM$Or0+=X`yk`oiNi9y8~Ym^+`KKKMU#$$rK5Q{?|J{h;~4 ztAB&3GDDwf{&Mu^obyY}KfmyLuiGC?)w=QreTM!5-=CW5&KHF3vNEqex#I)-1L)@? z_S9LJvE_V%{pw&zt(4G3_7t{K)gQJAa?N^Bnu1ybs?0;Jicr zwf_P7w0#)syKny!b>}be+hpfPX*{w05Ap@S(jQ6tA@Ga+AU{8W{RjIOF@MI)^-$YO z-TjecAEohL67|h*-*oB2dTj0w<^H3N$f5H+=!5;!cOEkL^Ly-kV}8K;8S}26{{3kc z_V;O2A5s1jV}FD8jQLUOud!b;<|ph29!ur<*54S*Q|~WKec=7MXZ_;yQodIo;1BZe zkJ!G+`V9J5{(67(W!jIF`t;XtFP%CS768xw$g_V&ea+i?qmI7(w3Wt(YfnWw`FB5q z@~j_nzwPdipCElpyZw2(75#aUv7e#+1nqy_{>;ra&sR}n{xHu24WK8U?~H%yA3QHZ z>ji&+b${ln{&e+Mu4n%J_2*NS#(Bl)Z`8-*lN88%KGWx8^f&s)wNGL{JZMGz^N-tK z(4ReaiJ=d8-XP}PpPs|M>a%A~o$8!7K;Jp^snXpKu|Ab}^$Gd{dx6?dbAL6u{f+(t zJ%&98`s|-yEYsd}nbF_${E@Ri()Qh>)E?U9_O3DV!@lIoPp*e7AJ%W+vGRyUo%tB@ z#QbZ>1MR!=MY=)%Iq6yMr-%K|p7Y=@nI2P~kk5ZIJ@bFX zdJ*%Due_;bU)TOkj4yY;FUMD&Gk=|Pd`&Ru$;b0xc(_bH|H$+8Wg|f-MP2+(eCmrh%J+6czZ0{wKq}u;KkcV(7^TIfSv@ZkNwsiMLA?U`6sNAqWqWl>mXD;C8tFxSt(aKR zK0Qx_{E<}BNROnhFmP4snrmo%(EFWW>QYHBJ=BEq!PH_OUgX0IefSa|Uf{z`K0M#R zO?KPmZA_u$uQ+x64Wu7ho=tt%T;udV=!>Uva_BsB;}9N;d#)9D%^C+oA6x9}cx%ef zr?LEUZeod*>Yvg_b?Q#C_bvIJ`>Up}V)U`QY`JlcVucLH&9tio3=jl*?Q{T1M zI`m<^Dox$O&+CO%f0939wJguiFE^!bzU5iMr0*!-V$Zyh^_A*J`5CF3Z#Ml6H{Wdf z8*aXt^o5^~qVbWs3F*%BlHsbpn{Rgf52z3J>6=Iyv7XB3jd#r#w13j$r!-Z^$O5qYuyZ;dwrMv4LAsb$ni7{$HYa z(`ZpS$kX=FmQ?r_*}wGs*lz3xBef*2o7H}>I5o=8uZ}r1Ao%M&ymmBH9)HENS>ceFQu1Zx`r-@&s zKSAuTdA@PRFZ@F^|2VkRhf53${ir7W@9x6$_x${AnCo}>pKRn0rY8CDL<6HfdeyDy zUeFsL`2vTz{-r)TAHhJA)t0zXnEZlz|aDVc-HE9_PdG zXBheOJh&D8spfM>o(Om~@>@QP=kty993PJOaM-}~yvbidK73w(6VE%>(dtAWx!%3K zUB$-=ynGe<_BQ;(m_IOO0pmR4f1bPf&|WUB%Qx8z>Z|z<#xu_!^B1W3i^c<<$F5mG z^PlQ};Hm|3zToj6M*l64`tn`~>#+qqU*dPEKMB`Jd{*NsiMjuUh4qr1tuOp-o(YlCZ0G^V!T!a{x2(q=gB0!ik``nRsM*iZ_*?amzJ9OOG-?9 z(M4wdDN{^5b*h;k_AR6R@^Ul(v}q=;s4(*<5+EM^Jf6F|_+6DOwisx^# zn_PWFe?Z^X$n_2SCfBbf>fG)w)~7J`k4@w+>F$#ANmFV@lUIK!{S2q;`K-FhtIyES zCR&fXZ?j3>q|Z%Nj8iOs;2ARgJ(54A&yeXq<^BM!mg)DX{-O1P^&>3xS?hNi*sH%9 zYyFm(=sm^s@_dN#*3{$~Z_p1%zma=W zJsxO$68(C-XspMB#(KPH?2HE`A7{KM?2HG6o$;cu9uG7=X?*DMqQ|S&ca8OU(Ad#u zRiC5p3Oo9$u%pikJL65s!x@hXJL5@VXS^w_$BR(?p~sWPdOT>X#|tpVN7TM_!X)UU z5dDyU1MuRB^7&}Ol)k7$`inJQSPBc9%1`N+l+k%bmM6S`?KduNN>m!@^DCx%@Ll$$ z9RGMJw@>@4ym&d%_5SKEdvS!$Ba12hxk7Q5y@=x<77@n&=`MRA#|JKS@Fn^5{Qe~l zUNDZ9(ghA~iWdVnIe2~%J^wMEF!OuqtV@Ai`WMe$Y~V$678!V9!$JdJ(s+r17tF;g z8S&BeH_dA@@cfJA`BJ7^_Kd}{eIfhTu30GC)3|zptY72BmrD9Hu9`3F*Lcw)NuR{K zcFdR|%3&*~=OeJ5TlRuWBz-!6Qvuj69jZ+@a3+>ubUG3og zgsZ9?e6j%gFk|ryhu?DG`HN~CO!~H9f!rT}|D&y-r+U83uj!dT!%I)pZsPfc<#}|s zYM+kSnxCpBA3qDc^1yt=2M#`;V7{01s`A*+fWG+-Jugta)eN>z&Zl?+)(=nn&(nFG z$xF|ehV=vbSemNg{7sY}^%tjZrRLK$l#cSv9Q{B7!2U|clW}~Sj?WD4q}eh}_G&@< zrJeVQ)+(_~#p?uwjPIl#LH#IC&vVH22tTO*p`OvU8I+$g00&c5*zb+1{W+gs@}2Po zdMW>{aW}ue{?BZm4|@pG^HG0o0sNy2FS95=VaN;iYD+ZYJmNlY72}ta(fQU4|0!H-FQ_K{a?>?^PJNe} z^<7d!q`e{J}b5tR@3-&({+8!ANa+1Vuf((zb?>0`xD7;vAw8<<_9j%{4TDh z`M|}OR?&Q->u=un8SBr0dB6X5E8?q^_Ne{-Tib=0m7w6{D(F){IPZ@kje>uV`+I}# z?*{gVsQnnr^AYTap_$Yk+lR>i@5Ow6r2u)-57gJZtzI>!^4RC5SBOS03vR z_B+RVy6n^+TUj z{U|@Zx2EQ;)A>vAH_gFcgEhT(MwNpxzE{{);aib}IgQ3!iDicuc-vcn_;_zR+JnBi z?X9pUOmNa)Ei74KPnzUl?4LmYC9*zhpXdpPo%WHA`Y>J`c_7|?f_(mx775 zc`WkQhw_(t%dbFuG_@br{PawsS8~h#K>I6Tk9E?S-jWgr6aPJv1@9lIea=5+iYFcX zgco|{GriYCpX3MQ*OiaeQzGq0sy@i$QZM~JdC>X4mi)IozoUJtXNr8@P03>g;?W#> zNFF^G3Et08I`nOYT~OepGk;}e4rcyJMJckfeqw%GVNaXpNhkgmpN~JQXQG&>%fD7I zra!HoDj82q^t?v$j>kRqL%!Ag>NswnTj|q>inzhS@STTrRf%}A3=K__Ja5*`^ZW) zesG7zpJnxw;$VOe_oomm?OC&kUwq_xXkRRNe?{eArS@h>|1jm>k7c%}fOZnH{3%K= ziSl4(9(fWf9NAp3swzr4Sz_pH=ppM$G4Fx<&P`vaKqT2Y={%(KAl^11NuG z0qASw`2mROzrrqI`FZ4P@%W<(Sf7jK^S-LRE9__<`)3__#%eeof%CVbKXMC+A6egS zv_D=x&rao`?^oCsFE}Mm`6)5d2mYWv)IWpvYf+~^rwQKgtM*sgbLM#Edxah2__18h zUmjj%7w~>u*Jt&V^L~frh5dxpQ;EN}gpb2d-gw%dGribfSv}JQpBE?{`%SASF5{=2 z{5j?9FQN9hKZ>xlsQS46GW@+Cg_)n+>C~QE|5SQqziglDPl$xZoIkIK&hvQ=KgE20 zpe#~0YvQkq=AUF^5|^w)(D$I-h`*+ej~cH*d#+gI z!Nz)p_Ey+S=PX4!k3k;qbNFvrae0=HXa2$&x>uv=!e0Fn-Y{$S9aCH;3! z`?GQO!$;v2_7xMZ0OT=-_SF6tt3Cd`Ss38`j-~fIw10n3yh$=?I{2K)T}5x@V$?_V+f zFZy{G#~*-r0G@B43bOt>AFlP`83v~J6{Sf$Qa)t=lJ@&9*n64Y8Y6$OuiA&J42<$Q zUVA_AEyREKfn!M^D}Tcb^IJlTgQ`S3&^o?u{#ANmq?hk8Fo^G3fvV1fv9y0rhly}gPsrbytE=kNz)p5WY= zK=^$z@L#+BJ@NeD*%9H;#Qgt?X%jd!OGPsO^0^zqSWztnGaoYkQ!^+TJHrd)gkT zadkECheaCp!rh>+s>;O}bg3CLB*tsHFkY-wO^u1Qy-?Su?TH%K)tU9x)|wd4H`9I` zTRK^PeZ7hO_C~)wa`tSqzFD(O+|XdwH)oECJ@!iG&u`D1JJ+nQvC+gAUu@PlZ=Qq6 zKk)idt)J{4!1EKXexki5`1^yL1Nw>c9-dcq@QEpXHM9 zwEuwed;JB39eX_Hf9PjqHRZqU9?}Pz4IAb+mR&;A39pWIxz z|CII*E2Z@-jrvL7wf+$%{n7fNvDP1rwY^;1-?hD5V?91J*5gNF4}YXjG#)iSZ0~gW zWq$#uOMbM!z^`AORuk~G*cjhBi-ao$dc+%zd_(6I0 z7Zg^}c|h8;cf(#}x$^JjCw%$o6oL@)lv1|X%o5lNAwuiSOJvN*7r;dF*+Bnb1 zZ_T~fgYU8n`MilY9l_q(0Q-Ld`zs0vV}E#;J&w<><2apgKKm>39h}Ge?K}s^*k2KI zaFqAYQ3qS>udoPXd_?WS`LuO(`6+1Py{eNwZUKzZp7i`nr&cU>-?6TTOivP%0G~kFWb{N!tb9(0hM0B=RX&REm^-#AE(~;A@Qyq zc>GKI2e6;C>=>Uf-SSa>K2>APUq1f;UVo^|?YkI%t0g`y%FhNv|sZIQa((3L+@X^x*x26t7wiRN(jrJncW<3j2R9k(Z=m%R&tDLB`~!IYLHiSU{l)VU zg!y?8%8%zS2-E(r75)QWt|2hCufU$3qXdXs>`&nBC#Caw7v=Fh1(lc2j{xI&3c`-R z0?&I8mh=?c`5bS=_I1oZ#rC)y(nr_+ULfBm=J+2h-F{HE@6e~PLm%0LK(9le!WciS z0BKFH&mT}|(fr`HFW-;l(lmCPw35AFKELDo6Xf$f(w{)_%dWrTdF+SmTon(IhP)ei{*m@w z)DQU>{){ff&)?hT`Y+Hv?9;hquW!R}gg(N55Q(^$-Nuj6a~J#*=Ot z{ui_d{-Yj#iGPng$iETA3mEC+(5K&L#`J<7`8<;=5Ba>4LtjaWr+w4^fc{uvPnqJR zV|=c#Yik|M_VF6>-8lBS6^O5Q(s7>5g8pJ3{o}pzHT(~d#|p&vdFqG1XWBFeGd&)E z1m?>XuwQ%9haiHPmz2)_EhkDg`MZKCnrP#MO}MuWG3+^?ai$y@39wSee~Nuuds6(<~aF@9*@0~ z=`sCbuqcb&Kn^}GB;**_uu z0c5`}as5Tm538qDwCeoazQ>;h`9t1p|BgBBm$~+DZr|hI;`VC=uQ*CSAn#V9N8015 z{mSbcdo{OTC+c+ju%Dau__6kG=okAxq(6$}QKil|sD$b9*tapClf0^^e)@2Ge*1o; zj_8&CFQTtjoriFf>+{?9OG=nOs*ls>2|oWg{N~Ii`@HlQ5Ph@M`3X0<{fp^O&dUB_ z`pU{EKYzHs#d7_0>dT$mh(zfxAo>!G37K29f2mwwDL=-?O50lCj3=C@Ay3zTfcHhC ze5F$!^k99ONA~myR44Xd=k*^vPWn$TLdJ*wWeLI!DRwpT7OOZe5>3~ zdhGM7;9rjG`Qf?UD4$~`QIu^UuNDSY!XEDOqt@?k^*jRp9);E8{Cy@xPXq^He25;% z7yj`u`6t+?1pa9M!@Sa_3;G?jBhtUCFxgZ0AW7R(>HS9X{D&Hm|GfK`uZ`Wm@P0$n z3;7xQbJSfxBEj*Bk#7_s2g` z{1u;9o%pCE7uV&R>{XYO{ItKPD%HaFxv-1}i3j`Ir2iLFCAC+Tx*YHC#C}f2ijNERzs30y_33!4TPS{s{S`)h zRk$T}qZ5g(|_ZQbbi$I7bE`aeDPLQeK)!BRVd&54_n-LD)?iYQZv2rSa|<) z9Zyv{K5uTXKfWrma)<~x40|xvhtkwc4!xSG;<1`iH{2xkP1=i_QuS>le=QG+$HIP* z&U384!M@o(JjaI{e7Mnv=lbwGAHLYY#i`Xi{;pH;VNHn7N-rS_CA}@Fnf(3^PrR0K zKGN}7H}H7dP4@O{XnxVp3kC7May}2~`8;LD8~Njr5N~xOuMgV4n)1glnek6NAJZR< zapJekc&V}RR{r=ZfBcmhuNB1moV@gdziQI|501C;$AkIf#S;Gd#>883{hoL$w%0rH zRuxA1vGHx5csS^H5bJ}N{vhIMP3(_v^2a}U%JY!T*}1bog5^mE;A*W38^H;xt}( z{<`pZEU8a!JeI_6JeI_c|FueDX-{+gtK<)&c&rQaA6Y5w@6rAx?N8BI`$sg^{tk`3 z{x3{o7wrGi{wM7Z)BYQcwLeBbH_%xMY z$M@*+I$lQS*YPqMe^`9ahsD#JKmNx`ohROg{H>Y8tbM#iKTZDB8bcp3U)At@?8aZ! zNdGG95B!jC*sYiK{9;;-@v(|Nr6Huk4-e3DFOe=4O{ z%XIdiQhJq4=lCnj)!#JyD_;F2O#P?*pBiibr^H0B+z(q9#0xO<3;tq z6Mv?#GaeLn;?I;n^-?zYOaC7EzZU26e!}tpE#miIX#Zc-UYHl<5v2WvmlT-(#RY|? z|FNl<&KGX^`QypHuQB4U*#4vAuLgYQDa2o~f0g5}&>x0<92oIfZ2#eSES&!lkH!90 zPdpaecRcY}>~HnNW3j*06OYCI*5AZq38^oLcbd=rB{AlYYiYOp%Ztal#21e>@BFkrzhVBy-#?J?XtXFHUZyEEe|{R5yuSiJpl8Nj?P9u0`KOI|GMpa~KennS?Z%IRU-!HW{we)qgKoN? zevU7r^4|C|%I}RYOS;eB;{1d^Rj1BRuz#Vw?BmNoj~o95uGqh-)*^GUd)IO!~A8$kD_Yb8(pC&0Ji?@gs~W{>9EQ;?Jh{=6dMic(H|Ec^LL?jOXdSIXO;z8PVJ0@!ueQ zgq+8Q#+PwA_n(X>Q~AN4<%dC>z(xqpW9dqN2K>t{NbFmRu`70@Y z!yA884tiv~mrJkdKU{%$k6UGNrEfBR%+2rdCsq=FG9FFyqvMBM|7=wlx+p*JkNExH zI-W@8-%asP-2W&*^0wfQ;|E@(f0E?k@rOg75MP!{@{#4qpW}%?Vtw_*Z$X}`?3xnp zU!n--HMH;gFQLyyd>8nGKhupzL%JD%M)}Qnua)qBlKk*d^vifU*Z-F@h5M6oaQ}MZ zzrfGf_%t)V4fNNT<;{4^pf{ck^`k%Bc(3OuKFt%4Sxf#c)4vaYGe7?)`W36Y?_fC5v0fPRlZoZ{6y@r=yB$6t^AEYc@A`u!Z~gFhb4>)$4Mdg8mdK2Lla z)-%X^8r#!}CQkSJlk+MPNOI%PJn>!JzQ>;o`rx01zopR=f9839f%(nlcsAGn>yKyi zJl_ENP~YWYr@pcNbHs~P((_~dLH+0T_g0d;JpS}9nr~&i7>`km&kv6m^Uq)2c(DeQ z_s5Hw^VisTv8!iY4dR)H?Bc~r#>R^+#g>Kl<1${XMDJ%{?~4ADKVHo6N5t_yJ=)LR zr2JdKzAD4tLGezezoo{-&=>f>ZoG+v&;EzqRsX+B);!wDa*@Zm}yo^D`z zpZ_-QAN;)%{Qao0@n4AdN>T{&e=|kw4-W z5D(_~A5=V;$Nxa~TuWkpo|kZr#PWRQ##fPj59^Z?KSCJu7v49)`%gTQBe9Mj(b$Rq zQ0Y!Qhr&*LhQdy~g~B@iMB7Joe3!;f{0;1vyFriRKT-BfZO=_Bd!x4JYOLc|bbUI0 zMPqHh7OFmNzt&jWgLQq{9;~qw@1n{(@hb{D_HBh7`?kW4Jz3H3_*WEm;#pLFCq6}C z$3CvGV;@)8u}7==9RCd4W1$bqA1w8u`Sll-{^I8q*4GcwU%7wP^Bd+n*MAOuhkYR| z{TcH8a)@u_=WS&ER zTkEsNS|2sm`mM*4)^Clqerl}8tCpu8uNv#|rm-Gx8awli(r-O}NZvGGIrEQ7*W*d2 zJM)oB*W*K{JM)uD*W*p|tH+n-M~^3zhrJZ>Qj;d|2qJ&QZi=6oJki0FK5mNCKkd)R zzo^tm&zr{2+PU#S<@~+|$3GHP&mZdY*7OQbdBltH^PHY|F^*UB#EbFsnVxttj#u-< zi}CZ8o_H~iSM$V+alD!*UX0__e)ZzT(o!F69WSP_ju+Ed$BSvKM8$X8q%h-6a9G*|4|APE` zI)0e+75387Bx|c6dnxSskdM!Qq5Kibo?3+GN9pM5KHnAo5Pt^c#-|WptQSp)H*=ns z!uzbTK3P70hK!%%^D+4|EGtdrX}^yC0DTeDpMm(bU#LGL!v2b!5Aq!u(183rqVqPDva-uRjCvZ`z-+ zU&X&^f5u7mJnskjGb&lWD*lc3AFU*B75^sJKVTK_hV>u)f%q^o52g7Z`!o1;n!xxl zj;VCEPf>j;evS4IyJ7FokC6Y^jRzYSHsjX{W28Dd9t`$ZH-73let()9KLvUaU+Kn6 zLEm8S97p>V{?K~M@lCG(0$9cC$^91MRrvj8D{!2KzlGw3*uKjc`w7JJX@1y#gYqg~ z5As2Kcpn=-Zxp5jIp~AEx`y`mjA5THwhK5QQ}aXic5k{{f12fI*uSAaxhbh%9)47O zprjx2QSn=9zv6vg7v{HO+Y^rmc~HETr+uW4x19Jqq?`6>YG3m&>GRM-_D6~zQ}JSw z9>YFeN&eWCDjp0RAb+u4oJ+P3mIvfz#G@d8v0Y@*^8#94(|cF0BzwB*57c*=iYH@w z;Gg-XwI2DUKd3O!{13H^pr5}w@q!)N9~Cv@1z|tq_*BOqRZj9eUwmA6dhe>sJn}&N zm1&>#`?D}Uyzy=U_-l-KH^bir`J?^w#=jZk1>*zbh5a{ryihzF>DNkSpT>Cf#jE8Q z@oE%r=Fvwpp01MgbHqA+w@pQ&`9!u|(DU@H)i}pS7SY6b9 zZLKFA{sPMrAGRCm^X572F+HaJll&o`bSlsBKhpkvC(4(UcFh(CeO{JJ-q*0-<9QP7PmJd|R^k1` zSg+Gu1p4NWw>9kbh=(@fJ6|n?ziIw_(pP+Vy*RJE9`yU;KmGB76mRLR&+1WsFNyb$ z5OOhW5d4tvuhNh382AVHuC2!~V_i-*bDOcs|;X^_c!}il@68l_#hH%tvN? z9QlhcG`{Vu^+~Ery)W#X@ogP?z4bl+(d%Dt>3tLV?)xUd582~MHLxGI=y~;E-$ePo zO~QWpz+ct8?Tky$=?3wYZDPViJU=q3-Z%WoT^Kxn@F>xX{Rq;3N2SM6AC<@RWV5I} z-miRw^1IG>p69w!G;go<)W^?P@%cqRpVhzJTArZhC|<5RB>S7*uT1nlLHxYp;YaBo z#s8`IZR34Z_(SFN;ph2-Ep{&dJt+8t*xtqZL;D%35A$7%og?2*Me$IOC+vSYJKXZ3 z2>4mx(WAS$eDk)yb=!Zl73Fg+e%?UEyUaj-=kG^Rf6)8DNnaIx1Ni$ow3X)%(*1vr z4Ey)oKnKqI`hC`@&xwbk{NDJ#qwsejosXLW{4xFWkss$Z;veI=dDlUKy*@s2zge1^ z&+$$3ReaNvcVoMC63G;Hiuuj>d!I+Ir1^GyIg)k%7=MpI{{2x)=>EZev3c8mS3alz zWX${6-`<`O;RGtA_fu4keLo({OZO-K{to&pCR$W~AwSJmwP1#7p!oyyVO0DyLT#~`=AFGYY(lj3G1dar`m*Tc;cE zPr-a>;yH4B-=fENgPgDQ{6h2RwlY`$Fu(fO1Abp*2Q`lI(B&Hsf9sz==y|~L<7xiD z`BS}5Uj6-B@AH9eYP@j!C2sq8KJc9Szn9u4{hx;PIb-#IudDxf{-(CkGd>0p-@^2$ zzi*E5Q;+>anrfu=$p81S{QB>IetuDXO=lW&NIHC55Dn*`u*d{KVH1!i^hjBzJQJK1neI#p7Hfu1KRhDFXqoPo)~+^ z59Gz;Yp)t#ujR_|1pSfY2_{qT`-re!n0T5TPaS$Z;d$`rCj|KiJ?|IJem#1E#$PW# zZ^`?y)2-)NkKnIQP={muJ^t_*eKpo2tS82Nj`h}9pU%;qvEShIod!J|uh^m6XZwcy zdtmB$PS`u--wQh?-~T(e7k>`|AN}_sBdL!Ols6!MgPF>2S9$@yLS zt3a=^pTZss`3`KJ;g&y1@0+Wc>W$|?e&hX6mc71Ct}kh9uCYGb8)~F~4BIZO2Nl8U zvh|+%Y2f3Cf6#wlqY~-!>hkM6=@ngNQ#W{VWz9Iu4`~)6^NaXc0*e2rJykuwCizJO z>p6ao{|Ni5(7#I&f3DkuK3MiGQ|0=HDp8*my48LEPZ#JZo>~CUue-&-%xbs(S5g1kPY6z@MHv2+fmJX+>-I5!O%LYt-`ijwV1I!;-S=xj z{+9izf^x1O|DpZFyjr$5Fh3X%0~eLK^`U<&R@Y6tMbis;TNi)aeP0dnvvgYdr*t~i zx4NpXitA5O{eg;LAw92|=3A6rSzS}@r0lW020MzY!A`$f_^0A^9xcxrdN)O&Rxxw@p15Bbm3 z{DS_QzB)XqoiN=*`*VAxd0WpvR;bIR7y1Nx)%()q{4>2coI~efZUg#)bouuP)%(ll z_U5?lq5grH6J7eCPlBIO&{u4_F*~SSr-?&f3Kwq z`~)MpUC1B7m-GSo<=;zG^KGJQ+ElrJOj3KnfscFS3;v>ZS%uth$o@lrs^>j{Az#a0 zS5`rFi*%6WRk6Cf2=h^zZxMg1O!-frms3KlDEb4SN4?KY&yQu($^W6pyJfGPs`k5F ze*}LImOs=X)LvzxVx6bGiltMFz4|b45zOtn|0{{!+sdYM1(+|Xd~s3bT2Fbr%_t$C zPe`KvR?H`pUH$$t#{0bS;|>1u@Oo!me;4|1b$Pklf8|(z)NV@AYfYad64XA;PxV3Q zbBSC2B+4(HYV=1%uweRa9{Q{sCLlacmG46TcI8-;0eO!FdK1Od`1xKr{t5#T{C#o$ zQ2Vq#jGyT8L;4m>6!3XoB*AY3)-$}{fC^A2VLVvNirl}C0siLpf?s@iJkWTl%q^uJ zk^PD1vqN#0ze>{QlJV01l|=o!ArE!>Q1Y0*IyXuZVj7@-(~FCzko;wRkpIOvBLK6! zAphx0i({mZTnp%(9vpuWame|PmScV!pGW%4Is|#p-@CdVSlXw^Kab^?ADKih^grwm z%^{?6yddzCcSlFlCvk)91E8Pc1Ba+G6(6{32bN=99`YDOyb_J|(G#P5K6&!)*l7BM zDv$MU*N!Q8-z`02DcgI<7Na!xbr)lQkN()T<08BtSEm<<`|W&AN0xk<4*#u_4*xU! zuR)`H9^Tih%2yVO`(Zz$bbKsX-YRy|!5`VznBE+n9v$zbcaeReEFbzUKd^t_-y;I+ z6Mz<^jPm&VKw;R#ye8J8T|4l2j>eD=;#YE<^X>EWK3;?i>vYbKaAA#4 zQho~uGxd4C@c*{=KJZaq=e_4UqmdYdECyL70?RN4*^v`s3oL9om7RgeMsaW=CnV8r zXeEK$mRt zBw?igD&N`Y_`j9wX9pjC*zRZK`jga;^sl$8@uay1`Tm4xg{ngox(m%xiA6+NjWZN_5 zK7THnYww;WKfIzY*B@;B-ex~MUgP%%`MyH=yg-|NepCD_Gx_ao@%hRszdxuyPqn&m zDUZgFCtQ4~5Bqu1Kaj~w{>gss(EdhwpRfs!YxxiDZ)lB=Ia~UVzAr|m%MY6QOUiR- ze^U5I?dM6Q{toSL7v9-YABXnW%lnG`^y=jK4QKajcwEB4Yy52z?(8)&>919d+I`G% z_64yk-X;5Gd&vAK;Z@13eV>L`tbr}%`-lC_K`f(!UZ44}gtu3MhbL(?D_JOcn(Gj-SKIz*so>yht&+Z3hddf)N=J|z~ ztj}uyWT4N}oARt&nQgzz`u>ZgRCwln;*EK}OnB+;+4xePkF5N-1e!zq6+S89kI0YD z+w`SA;^{j%Jz3f^)*m_lPRjeiB>_27c700Uc_*c(yqDgsoSrQI8I(`1XS&a0=_0a5 z)+@O`BMHipGS~0oa{Yx;=>uHP(9S8Vg! zJ2|pdCP))*?w@4uw`R+~_)cj+IZ{%;X8)S2f4QExI34?#pPsaLsJ@dUPnZ{)OMH`l za_B=_?fILeXUe-IS6^ate@PBi-KtZEt-lw<|64!e=a(G%@Dp|im`zV!r#N4K(${H( z`lq&Mw*8YMRaJg{B!?<%?DMCZzSdvON5lEYEA0Clk-xq!FqFTh@AVIT{4blojwfwz zFaJ-t=Lb<=b>;yzTYhbC_kK**KJxwbTHpTo(e%9h@n`255tWMz`W*b@GD zRc=46>DTCh$%f14AGhjqWDaxv?cS~T!Z!VVu{~GR{C4Q|!KRNlAzb>qJg@R0T|P`2 zGC#=kDIXGN?RTV^@TSL8Vw*T7y-K@x)bwQhB!^m;*zqQmw0CkSWgf7y{(l(vON7?> zS+CD;`1l5iY7Vg<8Cqwa7cl$JvueM`K7XNrgv;|Nk8hSgO&r7DYwrKs{jJu=`(m#D znf3RPq5JIf5-xsn=#g?e-=<_dzF^ zVI=<*={x7N{vUJCPs-<&KcwrsiDSa=Fk5@Pe^RcOZnf59ui)YzT*z9PwbyQFjW1aRbrboBz(ENzbo;w+pmi)pFe(F=QERr*elYj=X5`g z{F5WeR)2i1NY@+x>Ye||ku~r0!)wy3S4Si|@=uPeeZ&tBZ2ij>tot>$Uo~}&7ccky-LULYWM!ojpDCv zUvv3&1ZI86=J#aqF-_aqy8mj|4D?&JB)^U6J6CIY5&w~)YI8nm$}9Dye1`@*K6Jg* zwBKW<{m{P-{fgRkHIW>-Gv&9}#&pfCCa&dU zHR)xyO3AbB+mOCvRZ{FbT={F#t5$pI$@p5mwp+s8d?fSJ+O<1!w$$$(cVxsy{r=&8 zo4a`_z*E)we%4i5JtloAe{yKmnyyg%*7|I?w10A_-hBURF26hO+@9kl|723HzqxSX z@4Qp5hvn=wYxMdN?7a7-j|M8*9 zHM#wvj_-zgJzX$9n)!K6{R3hfpNt)=*Y(EuB>m@+p$79jp6MfE%lqP2>FGmKezbo+ zGIW=H-@3Eq`Nmc6mA{RTCft7hDi^<^UiS;8j3)f#=FgLwX6?I zYqdSC|M$uKT2-mzwOq%ylt(`AxqMnDD9z8`d4ZEzZL7lNMO#-2fME=(a$$+S==wF>mkpZ$oZ*!e%ha}_49V>Kh^`O zuiK3O%d^j;RL%ug1*nuRDD&eo^Zjg6KNoHL%jXr$d>6CxqukF+*uJb?R?4sGYu_7m z>p^rxFXzZ1TSpB)I3n*KRa<$fk3;)o@_9q)pZb2K-s<2B=|pw61SZUQR{!g6O^RKY zeINC$=6R)r_P=x5A6FQ=&i2Q|#)He{0=6EuJ?|Q+OXz%9r}LBW4e6E3Gh#2(^rXET zMwTaZzc{D)$@$KT^zA0Sm9Bp`jNE#w#$T@C5`W`}JKt1W=Ch4Mx3193aV9^BFXP3$ ze{L?7G1sG)4ZWl8b_u<;O%^!0pYz%M!E&8jl6NGkf-86C-S6c0Ynk_f=>XB|eOIJ? zKVWBX)4t!-@|4%Ae|EiCE!PXf?H8(l@lV=MTgLW((;ngWTam6Y<;VCAwU_V>=~~ks z)!Fu_&b5bxhuTy2$IFIlmg-*0^dAZ5_SOEU{i9avTl{N~=NU3$=h|m!u6<;FzH6k` zv}dhbZ&svje=z=P{E_Nvt&gRqeYCx5YRp_|+E?4Fy4vplZTpLVwZ^|}d)->A?OB~` zubNzYt-bm7dMMmpE7CDDKiT=@eF<(at$!V#m~Yk(+wN>OnNok!eoM6dij6;Rujoj1 zX#CY^{bIh6{?F~j?IZ17os|4;Dbn5>hiaB-{b4?g4$1q2)ztmmS#6J7=50T2AK^{= zNq#cEbUan(+DF=Vqg-#vwx5hAo!_;+ZppP*b+)}O-*rRnWnSQp@pVtTdM}5JHyK}2 zKXO6W%};B7GCaPde@933@~<3jep`_aj|ZL~L+vBwlm5OWDbeGuena&q?Y(|@iK$Ms zf2e)V7pmVHGd^nE_}Dm7S()qq_i1}j{Z^XsVdg7WzuEcB)bElSy}sbpuep9~=C8}| zxS{$@=msA3yCN-y9=3jE{t7hM&R=sf|HjPs4Qz4iQFKHvx67gREB@*G8FcSt)=Qa> z)i&qHWHX;iAiO016$bQmX zKiB&&!C#A@mp$b9rRVp%uT$0Q)xS|~`@`Q$xUQG7KYxC|Y`>DqOSr6`^R~aVukKg0 zd}cl4_LlLJ-`+7APnD*HuM zv8F!pyez+8^1%K_+maIBJm0)yncOdv%bOz1x##B&%KK_ed9{4f-(F5vnC}azSROI9 zdEO}gn0*H){I&{Q0-;ljn=w{#GUFpYr_D;??q}@$bKw z=a=O5Nbl_;)b%8W%l#d(>sMM^_Rq$ys}0*V3D5q!eL;n`xBN(bl?7tI!?(@zV=31@a(zqh z-{#t9wT$oZ_vdUJibfxm^vrtr$4jH_Ls9$vDf;}o_-oP|bL}VTwGYKCZ@v%Y&m!$Z zkIKy)IrMpU=`S1Q`(vazpx@Z6|s~Ap?!4z zHuJBWzYnIfcXQ17=1cDRNAL&Jw^M(r3AO*Cp$hvzjLBd7+ZuB}8S!6AC#ZiU?fqmo zo?l8==K714Pu@48?Thy(O8>3O_9qF~{y63JN9j*hjVonc(*7mSE03k?Vi}2#{3U!% zjSV;bZ7h9TY&;yEOk~5$rT%K|`K~bLd?@W+zw2yCuf8(NFVg<#_0Pp>FVFSQMbiHs z%FS2e|H1S-L+zvOXWL)%)BfuAXQsbPdydHWKWN^@miYep*^P8kC^Qd>SHV|mxJVWZ?F%h-$(vD9I{_dKVs&qM{4bS^>Vt^*sadqXunSYvF_ z|CIjY^}iRSf36Jmw}a`1P<{u~cLj-zKYzaW5p5ro=cV)mkA&yD4UhQ!^QClCYnZ=R zuD|&4*R}e5`12#HW&K(5&}!xNd5alZf86}G-gqqQ*Ei>a&ka>>)X9FXPS%hAVffto zWB4QS9IxpwHs3#u^>-}2dZW$ugH8cl&XA}m;dL6mdJkIl}GaXX~|Fhk^PJ8 zS9E@PEETTbwp6%&H>dpi9ZN4+;;&z0b*nf2t6ooX^N)<@RO1@AKI#0?*j(j@zc}=e z`Mz!Qc>>8__{Wl#m-0l^ZmSR5n^$_a%m?NKLHfAVSETUzrQb)F{rpy&*Iqil^}>re zTz&if!+idyD$xCz#y9QV;l*FA{bO~uJ`47b?ht=CH~+f!TD)5C#pL?S#?^lR7)!S_ zXKksU0{vqd^^c9VKVB;j+;r5sQa<}^9+E2>YoGqWu7eiGK>5d5PrTs-G zbnq{w+fq8==IqU>KyDzx9+UGAzrRR-t@8cX{_;q}Tu@hL+jFdL<7&UZj5W42XKmMC z-fQ?ww!b_Owl{C^Z0Rp@fk^zx^%t|hw|2SqAM=1I?9l$_vhIK4UVFYk^Mm`m81|>8 zztnmB4b2ZNCE7dmzRH`L4@`SMl-J(j`Ju8U+};mWJw)w&bMr&Ey^GBc+5JUhb4#{; z|28)tNO-n=bw2QHGatY%(7xV&A!7HRb#6SH_BbVhHdt@<-WaT@l*d!ChheBsqftUdREStYUq4P-d7UtpUd@j zW_JHPCgaWT@3P*7_J1#>-?b*ZUanf6n@?9u|7~0oUO(&2{SvI7FQpsS`0J_cx0hFl z114O~uRmMo>RSiNSIzvnW_dEVp33_Au2sg)_TR?(u-&lIv!%bq{Qf4kw_ck4mbYH| z^Q(LeUak+xc(1hMU9LZhKV$LA(Ef1CGqsDe{Y$SG`PUO{|B44@zLoS$e{9I{a((Eh z(~I1CCU?;Ox=Z?Jh53FavtO6zdtV${^pNIn_ABQ4)1pPI)zw(hVq5XVRUSnyl7`f}=vR|w8_jd==i>QBV_i=~xZ|(p7 z`4RSeW6)u1AFLypF^V>^9cjxwZVplZ=QP)0he8$Xp zin;M5=XZC9`|HZP!~IiU=kM+B>fNWU2WN)N`TdJB9{llDwZ?d z%hpBB`x#9Ckn2Mm8>3MjuZEX!86S67$d19z|Mq%XwXt=)$R*e{=4!t3rhLM$tn$Le zuB-Iy7t?cQ{hgbX>n9h?{B7PhB@W_xo3}og>u=tA5Rv}4&bTh;XL3GjuAh3JKfO=d zf7!BZf5!U|Lf2Er;`fNRc7LPeOJA&E4)p&-fa`l^e2s_r7t?YkDtK=E6u5qi{g8=& zG8DfiUFLnCwYk1lrum!wgSmb^VZCwJuif_%Nsp3TrG2a9c8Q)4<-R}ac!>YQ{n{Jk z(DvVPuZ*8qkWyO@iGOH+R9=7~4XpPscIf@2a_t=M^D^Iv$b4b@mzn?X3tz97`aHD1 zY_38dpm3i*sYyqpnt;3i5?zswn){Vez1}DNcg0XpZu+mfC3ekFnR$H9@N&OE_D{wh z{D^s;Tb_?Ev+b)F^`$?q5I!6KPxbnK`2MMur$TS|s6R4a)yVZ$P0&l<*d~2bzbOgT zBV+axe>~(`U!MS!`dP7Gb90A`pBlN|WYW9e-Y@yA%va`!{JeR;;n{uW_K?+Ne*e88 z_kAZpbv%&!aW4+VrM|_V`(-^9J36AS%Axa-)W=zQ|4sHdULE|+{y7_N#=|1LUuEZ` zq~5QR?;jZP=QH^{)JU2A?3||e)ei@U7thU=N&8d;nabeGB4bCTevJLsi{yPT+IV`{ z@<-+S%#+^~Q|nvmw{3rY(5S(|q^X}R`%ynRzGEJqpSQF9SK?}`%Jrr9NdL_qQr}PP zho$L%Ik=a?=jXLP%jLx)a=7_j*PCqpN_ph{I z`@O#UtP9qK`b)W-8lyfnzZlo&>*$~Q{-)l%{LKB`Y<x278|Oy~WBkyl8&%KFFIbUsH+vG`whj+w<}( zSw6X8I&b+jyl8$~^YY8hFGc20v|q9H9qm_eeLs_zpFW@I?)d)|+sEDiD7c?M z`xVS@^5*kzmoO&y2`gbFTXdpeBF8Z zy}9L^$jk4|E#IlU{BC&s$$T6h4{pBJ_ANTz>w^}W&vNU#9q)Pj7xd3!`Du93{2FgQ zKlIOH`ORCt!MyxR_LtXHzTUk2N}eAj-FPonJ`FEgpQmmiV2`i_3TVEzWQ9Eza-UTb$n%<@Zjy9@5}l4>9MP4exB!_Z!PIP7Sg?{E=LF z%D&%N&bR+lV8atOe9IsC?wNts zzkIUZ{!U=S6E=Lym;CZQC-vDBbcV`TPW4&f`Wfm|^9$9d4*bbqb&`7_jhdOgzArwz|Pzs=7t*FTHoH-CM~2L$HppY>F}8yN2` zIv(a3?;2ioyk7|hL-lzB<2|GKNqseNeVXgL4R>j{wx5KD>r>~K*I^?WgNQRWKf^&l|YEk}i>-h8N8*ar61zX!)c+h0CYoy)`et@P48|eYU&u74Dz0 zat$xqer4n!@Hy+S_H<(|pd`0p@`xVO%?N>0r?NmPZc_MQ^NAuGg z-J$t2zIgujv*AVCuRAzdv_AhVQt*Ciyu|ea4cGj7x&7W}#@{*0FIS&M@@sPaQ_>5! zpX@(%zog+s^J@w?zg&MQlHX{F@@aU{{E~V3l`LPXMENv)p7twVKJ731s7I*3M9atK zEuRf9n&0p|<)i%mVX58^kDB%=zCW|!Mf2+p&QpKMwV%x|dwmf3q5T@TznJ}Cv>fdx z;rZ)BM_zup_A64pxNE<>{NAViGg_|UMawsym)|XDFL^#9)P5*m=e+$TTCU+m^E;B4 z-woGi;qh0ZJ`d&PSF%1a9`fq*Ia{Cej)zNm`Mvh??X~&M+h5M-(RF00*&|Z z{kQ__d#c!Ycll{}(ed7vmtV>96>UG4-@NVD63TD8JP&A}U~UTZ`QG)z=5b*Ae5yPz z*|EPPepNr9KKF;_`9$WU@_q$#=<|$*Zwv8T_FvC^|J@{gAJV^pZJk~Cf1w5zYG9!T z7HVLj1{P{yp#~Oe;QOTp^xRrLxiUE{ADhYi&8A=b#`%U9pZHKP85xXBr>9%$|LM7} z|Jc`#f8!f-|LxhI9Qnyh-}r}bf9LAP-+p!PXa3vjf8Fw}Ph9)nmhXLT>FP&DW;<8^ z#Lj>D$KSnp?bR=iSC0SAJAe6!|Ng?a?*8GqU;m*$|Mu_x_=CUx)!%yjfq(k(hyVS# zpEmf+UwwD(kDvR!-+p!O)o;zoes}G4+44U3RsC~r?z{Tm+$YX|`FDPP*`NH@^)Ed4 zcOPu~&6h?;cYpNpzuR-+ThB@0PV*&=fA=*7-SOe|b6;8dy|4W3-~88?9{Kdow14e^ z`#<{lJ+EXQzB2da|M8s{e&Xvt{=NS$<(QjWrg?n%U%o8=Bdd@7pMBr^>O&QO^1kaA zzC82`qYY2~%I;tJh~#zc)mO!KV9{k-=6!)U%2|}3&|7z)60MRwYl$p<5T}I z_iJC8`@jC{_kQ>K`>ucIuYUixe@ckhICBB@LB-9*NxkfT$l8_W5WK znVj>LvRIxX%|AK!WF&g~k4j#f^h1l$%HUuqvqt$?>3rexF@t%-TS|ndN`$wU2=B-X zk80=79-Sq^i@BiSWS^;pr0Lnf!3ufm{jVCBhRD9%|)+Z7O5bJLFMaE^{bc z9u1y1e6&RPSc&kXCBnx`gr6u8K2ak4bcygu3HR%?z*s6+%i)o8E-#mG_*9ATizULR zON3u35k6BQe6~b*ATJ1-Z)RxB3lH}~xj;(s!{PlB?zci&!8#Ae87z^e+_Vd&`Fu3R zvt+?s!Y!A`%OxD%k{2HKQZCibTk=ZrT~I)O|q^Y|HH%GP3Nt zH6HthyzrErqIKVp=R?lNYNa;qh*qla`eXMIn?*ek<$c6C*+*=;{hO81+y8oltp8sS zNOvz$+1jPB3Fwqr0Cj}l-rgj}X--O-xo{a@u3Vb79w#(UX^ZIX=VZ^3xJP$Ry6f&PP=Q3yHD@OgJK$$gmmy z&H4G~a+dM&&9lL#dt`x?X-3YLq<2STVbsy1pB^bqc;~6*`U09#^tYB6v1fpAi{ObhWm~Tc5j^OIJRTm zZ_gk!r=AnWjE(F$-1BjBUVVGWuTY#(^-IpXzb%>RTosu;80YhD_-#JRT_HvKY(;2e z5*icKceNF$T%r8Ud^*oMp}ScvlbKUX`8iKH$J7s3|E3%yWw(RZIE+)y*BWE zsVws{9?v#S>DA@pLUrl!=Fx*3Ul!us>Aqf{Kz`=Y2+uY*uRWXnF+Aq2{63xEza*_G zDO;OYHrS?YUbykyv*8p*3uGB>U@u8r^a-SGUSeK*S$ zXj|ii6h;rfkD6_fmCgToeKa2SvoYjnWJzjKute)KzkIuWf zl)SP$XK%VCvpD712P8b>*{?|Wq-RH}C4RLRzgpsZ_5lgcc=jvR!Q#p4%{zZo>UW3K zZ?+~v`xCxyp;s)U_f*Ms4Lxg@D*(E8#Yv?q^5ci=^`N})dD`AwcQn^DHcLvImZ$v4 zqKx61hUY`1ERZi?l|Xl>51VhPz58|j)LgxLQ`O4K&E9YPt@@^=)%HjK#*b)V{NMiH z8qc@&Z~U12x`PUYhw>L)?I$CV4^&EPrBK+sl6pN<-dHQWex4qx*NSxp#|U~OM`fXr z>G@_xq**-*j7n`mbIda;(co|Ls4Sa|$nDAODBV<+iKIOHKv^o1@$6TmCr)~HBqs6Y z3Jym3E3r&u(z6f5B)(^FlBqyux7Rr;U*D*U)D|3dGNR1R#v56ok$$tIP zv^w3`ZuDWaVWjZy&`5D{Fp4%88Buj-Dp!E3XeFiC*Rm8lTw<}O6v2CGF=l) z$zMwkm}g$qbW82kbnf-m`eos@zVO3t>>=QZvHLOry= zTIBm9ExkA@b8B|?@mG6!{3K)NN|`UiyM)Oa*-2!uer9CODin@g#8`M2VbYy%j<$Ew ziti%yu2^n3;j9At?*|4k-tN8i_Uc} z+@$4&XXh)O|L7;1ugk|z)1TikZxx=i%)H_}_U147p4s_#33^}k%^s)zIKE^wz7Y#Mq}fKZC%sNUUy}sKlc6~2P9#;XPR6gb$bDpynvzN)ct|$Xqt_kC;K+B`^$b2iSsXJlZeT)B9 z&N$4{pL?D_&Tg~wggqhA^G}aA_Zm#E(p|wk`!6k%yGQZAAs^4S_eTm`>zL1ft%q=) z8Lfj!)Iqz1XZz*Sgk6aw57Srlw`%&Mqe6b?U1!n$_Zok+|CyDpKtGJ!#Px>GyfR9I za68{*y=lAqGr%jc$^2^+C9W@o>pkpKhpTtJc9pGbH>UIJI1!$W3hYs{?PS{PwYLe{ zqtM)NGi}nX<>b9Y>x%RFW%`Wl zIOWQc4BrElh2{m_i8Pt;R~8owZ#3bNqTzKmylDCf6TYcv{G<&pE0%xUroX6Q_&oEM z_GvAjEq^3ZFusI4|I3So+xj_B)IV3gqVZk*6%DuLFPy&DAIi1;wLGpq3eNMn^1f2A zyt(wsvgx_rRy8ACYw@7k(fw{IG=UToY|c%JX2GWF1TCG`T`L zYxYRWU)lA*%|~HA@V?|c2pSwu3{GTCx{Zd>>;3jRXqe4pKjWeBI>YAqsE_+)5Bcn2CC7T@o(4BB}O{1G_UWxFXB4y{R(JP_&n`C(j-+50;{9HTgXWb+{{qRvVJ}JW@ zdlYELdCnc|ezN#;x!Bu;_ZQ*)N8H`}4l`0c-oCU+*UQj)8cxI9w{NVf@SeF^B=`O+ z@w^-AdUKem@EIAh=E$Y-e9RA*#hSw>oEpyB{9U--!Hq7`#|-BQuZYTx=i=c9WBS>paCl_ry0&izKNZ#T z$laAW-=387&u}}0+bL!C>M~n|+A7TG^Ln@@wW+Eq7OwAHedO1tKF=Xv<5#@CL-kpp zzC-cz!}S@-T>jbi$UQqFeJYoKHr#8kXqm)MRb(nAD+;#P!T3S>BF5t3(YUn7eBp}@ zMmgMXA8xOVw3k_jq~B}bcjwxFCQWMweXj1yv~KeHvd5d|@OZ6@ERVYMc$tr+e`f3M z`OsKF-Fdu8!n6Hn9@SqH#Y?YR#^zG)3toKlsGi5);`0Uk)#2p(Tjlcw{53Zya_N@| zADz!1Gacvn=a=|=0e?&{&+%m;-t%Wrrg8b`>T~M8UK2@^Jt}r~f|qb9r1@--@yFR3 z)s$cC6!90yZjmRj{q*#SY~Su6yEBkGdN}l4RPrdP%{PDOS|Di*%5~mwxx@CTgl980 z<&=*eiyygimzge|D|eW`Nby@D;qVmM9b|{fDEaI2vs044@yWA|6P|6xjb~$AAU?(g zY||Ay-t>FVM)|;-e(Qy!J`isDorJ%w`jtMXCO(JiJluyep=)`%Uod41_n9>HsqeQw z^tNgfZjVxSp}B$M+z7*Q*w#G8pHKMd zdvkq=XaCra`umcEtkppz6Q7J{_a)4;?L7-xovO}6IUMQh1anZoXQf2?J|A7CUuQhO zKR$mjVfMN6`tR?9CZm&c>~APs_C*{Xo-cyX`G-E)CH&iM&3RkT7lp<;wUNG3BxM#% z>4SPMy3j^%?*6UNJQ;42g>_c?pzg8dzqe*j&GXpnwF`~A*PZ4KuDQ82CsLNG4$3&+ zVr#J7Rh#pl&>ETJ!)vh2?&g>${m>d5xJib2whPsrzm7!{a?UEJvWruTWgRZ;1EjKDS74fuIHQcs1ou0`QMyJ-2`vWqe}4pd|YZi*Xzr9<(cQ&f8D}-t?SjBdX1Ia zqtIBNZ%x&Ec9O=zn)+6+sopbkg`T0~cJ-gRI}pp{J##l7U+8m&^YM`a&(Y=ap>dyC zxZe7Gb{?qrT)E9|)-JRcyxBBvmM^rg@cVwGJSY#``JA3n=?#`rdH8#fg4Z^)*KWgm zknflC%+MYnA@?0q^8K7QaZc&Smpw;SE`I5ALOF+Aax?tFM6@LD?>SN>&LJ}e&JjcL z?K$S#YW>nR+Al+9k=(a0@jS3zlg?+t^R8YK)iE1h?=$6c-E`92NR+Xn_rf;y2IieN znx2lg@N+WHSIWKc?7O2d2E2W`K5OH?09P)3=^&IN7hV>Q?*K166d(6Tyzub*lianK zT(~}4kjo$UOEztdR)+d{To}3b5l`to5PO8eD-X&&34|AT)}~VLS?kfj$z^_BS`8r5e!wkwk@jT2os$ z5|>AfX0;#W>bOkX$bAW^L~@aH;ie68;rVT>ZIlalZK!$b^Oq*iAeg0oS0-_??+P!_ zuR`JZ{l>iWEE^yFBKJIIB%<#w*L6|*ig^aqKNnjpd8f+d6h}^SjE~Z%y>NNpF`&UZ(lNrAy z=8gPk>4MO^hV+`6zrM<|gn8?0exC~WRqc!3Q>XSlsM&YMA29Wv_fFF6d*~0CNhIRE z7c~-@l=q~1??n|C3%?gt9%-DE>X)PNdr>7kV}i~5)bo7Ez2hYNUew5r@Vj9*$-7eN zb6ZmLI#P?jn^4hnZ}#whwLVG-PebMOZyQoIny^FSQUYIc! zW{e-e8Drtf=bQEk?u8j+VaE6YoH6Pau461*$9NlF%U{?Nzu|l08@!Iuy)a`e%oqzZ zhTi>5(Hke<=r>Tlt)7r8ey?QVtYqP=M7PexdkF5o7bkGPtvG>RR-C|Iw>W{dtvFFS z7d)?6JPlmeD^9=%)v|}XBQup?3Zgu06pV~wJ#^-zWZ)Hq1)p&nD6$`YWu?J(pXZ80FCxf5; z{F>wc@$|2&H#RFkTk@mt6d+;Akj(Ng&@SAjrtil>7wyV$BAAod^1IG3M6`I(WQ)LG&Fv*=%upRY+8S48_e zNl$*pETYjRCXM)&6 z$z|ix`StVGjq=$J{V4jRv-OjcYEL`6#o2*;Dqs1Ovy;wlcXrCz9nS7{wtmW1)5|zp zKbNcaxU>76J?ZR0XHPpj?QGqRY5o~!C!HO4cFNfaXHPg=KMSb&O*vaXJ*f7Kvqzns zkjY!qA9Hq-vyVEv!`b7`9(48zXOB61!r2qfKJDx&XHPo2TRx|(`JHoi#@SQO9(VRd zXHPnN+S$|2zT#|sPqwBv*w!F90 z*zL~lc6Nuer<~pC>=|cwJ9|tPOHHp+>ZeltnYhE+jbcZ{o^p1#3m;!)!<$@qGU>;! z2~unQ_%%Vdvu*l`U{W?@nx9QS5llPVhD-dr{cwrz>}>pZ`r(P7yV2SR&%7%fKJIKA zE}dnaA6^+GosIa3_xR}ve{a~HA$wZ(DY^Qq3}%{i^MU%D3!2u4?HRF^uQT;o6D01p zw$?{wBoSnsea_YQ%*I^$+8&7qtgZF0?bYpU9WTaivEkZY$~V0?oPP3$tZma@8FW8v zZ7qL9UuWQK>(9zy;t`*(4>GcLqr8zweK6&0@R`kFzT?p_KkjVs9gl_ii4XYv%Ao0S zYg;}M1RwPI%An~9pRWqWoNf8ap!-8UzcNTX>GO3#le3eq{>Geb+ehX{XP-0SmBI8! z{PZe=sgH*3$!%eK;$vZZ{NrJJ>=R)-(;2p>yR5DAleX`atkGKkIzMUq&Ny4=qe^MN zoPOfvaQaQoM*7po!s%x|Z*8RCal+b2f5O=| zzWDRSaQq2pBRu%caC({l-P$^Ts{fgwIeJ$JshxYW$S! zOU(Fl;fagk^r!wloc@fnk^bO6htp48vvxLp+0*#xPdeMim+||(aQtp(BYa$J*{+%S zy)rTvOp`q*iEH}WzFM9MXKVeL@;Y0~Z^DC!;Wd2?Z;DzQ@u!@fjqmJs6JO>t`G$GT zPuCBPpOEk2Qyb}zIUDKEIJ?8;-&AS&ZWn*d*~mY!#D*jPl(Uh4#@Wbky2|p%uPI?| zA+PaCrAFYuCB-Qcqi3%ct!=b2i+r-xA;1+4w(C@tvIwe}TfCoedwOaA#-3KS$xt&W688 z;m*#6{{n?O+peclz9Tj~TR+Y|hx++Nicfa;FHtzz-7ir%+1*DeT))T?55uf^OLVw8~Jto=WzJ+A6vW5#UK1LYj-<4 zxDe(uU$r*!558t?0jS-zrX8#UzgwOB(5HDi6xf-%(WP`FFhaD z1IJYzu)17B59BmFdz^fS{S)cvwJklila6n&KCA;$4o+u42T}pOzO1l*-MVAaxy2&p z|5m3z?8=2}=jR;*9oO*NNJmn2ln!*+LYo8WK!2^j?~#aQW)=yj@Q~4UHVYp=B4$~z zEG&xkAEghJnK#>ogT5kqRCttk%mp_{y{eA-$vC}A+Mz56eNDCrX*K$7j>`S&6ul;q z^JnJkw&IqH>v!fbGta4cpXkYP>A-Y!i2fc=PrC5oBI!`{n>N);*R?`%Q_k9{6Q6gu z?vPb<3>1q06&LSga;?xmIR32?sBWtit|{6n zkjy~)opUkd#*}ONlbV=!{4cL`lHXb-9RCA0e9&RpU&ma0 zT{rbW9dmp+_1Ak>dGRm0_zfm@?mv!yr>&5|R1PtJU3c*>yRNSJ>-@s;M@5h%^75b7 zij^a2{?GRx-RcENSkp{7F8k|U*}u#m$RF|0{?AI}`QoQtG8$hwj<55GR)Gw!(2;TZ zXKu9>r12NaPbNeK!4(PCG~Z8pkh1YirY_9P&+3GC=zOK|HT}@x;&+Ec>SPjjT+-8= zHWP>Z$G!1asQm_|V=lMt=SG}OXnc$Hzx^d^uaI9_{tzVg`=sJrNkc(b{?e))F3_fu zWnl)}B>wXS@^6{85z+rOe?6%Bf<6~NMXn9I_~^eHU#A3)-z@d>vjy_+{Fco>DRK1R z{Lzmdm-q)Iev0x>yZ9GPY#U$mM?Cc3+eIi8zvES#Kl-ocuTv7o-zb6%(Lw1yq541P z;wL4Je^7G-AC>q&RiOM=Tzs^jbW_v*oWG76t<%$UoU+Xo`v?3+e5GG&Je^B1K5Z_! z|1n;dXI*1?7NU6f*!}!8)#M`!iu_YLjynB=_u?wdYgsO?L0@*|n!@xNpzCy+d6+y*-11H-7rP z$!$hy{N$ddb;;&+$@LG&&xV#CS(kijU~kWGOIKg>?)#tGbI-cu(_K4zcJ}vqG1d!e z=weZxz4<6lckO$MW33l&p4qi$xJA5pwnX^eXNElw3zlj91Cq|ZzE5`zNYOobgJdJQ z+`mDBcRtf4gaTb&9jPNAb`0#@`E=KC&yE4VN;a%Z?z!KiKHW9E_vyYpT~9yTWh3m9 zw%XU%HQ4j1zOLPUyLU*N_3s)Cxgkk3dr3%y;axogU3)!OrCd_;4@p}+plz0GFH<(e z*f-oW?5FI$~=egW`hod|gt+4S8bkt|1w0U7udxwQJAdGp1Ib864>Q@IMgDltz%v-^l3vfi zu21dBh9`S_hI^7Wvi$GGHvtCLoE~2PKyt_a;XZ#d5Z?38=9f##r0J!%r+3%xhntO> z`_BvS?R$3D&b}@!=EFOm>Fwe8y(T0yfds*apBYZ>eJZ&_x>#>gVto4Eef>RyeZeQ2 zend^bJi~jFJNl$k*uj<@>e=(DzT_w0vtfPngUv1XKhWH=E}8xNLEU$GWeYyJzDRmf zpigeFVFmMB|3L7``{xVa(4z6gZ_{h3dh`8nS^v-mjo=Qi{%rX+G&P$9i-oTbg0=Ev zD_g3fw=bPUV)8qjp6PnYPv2|ldkrnmve~KB)wE~3608*m-6Wl5kVT>NL+MGwDVvRt zH1D@5c=1htlA?qTFWmMQGsWkS+)9vhZLQZSb1Q)tzA+a+FMZP=rIF>3RU#M3Me%rP zAJnJmqo$-L4$3I0csz~2=0d|S#GCP9RuXwM$*n}}P)^Tvx3v%E%I7aph`ZM7P@aFL z{{6nFsdx(*qUREqzr6U_`pEgDL)Ucx7b}z=>gl0ek~g{(!kHWAz4}?7n_xZvP5HHJ zU}fBkOmgl~d1d~2$aIw~;iYHC?*q-jC!2Hgg_m~Dp=@lfPf7p5mIs>e-ymb$jtnn- zZ~hd(9Z27->IM9#@H`*9^qv3fn{_y?U2n7S!cBd-VT|kp-XyRKUVNK=)^nRge*Bz^ zYuDc&(7qy@{Mz-+e%r{XQJ*);&ja%Fp!~GR&qKl54NdZsn`JzI%*a&hNlQ;d?_x9c-@Rk!8w;^)F2&hL`6RH5qU~_RiwUZ zwYk<*XQP|{8@#{eQJuh>-h$(LqNK;RsykA@5Iqw6#fo2wzf|?B)t|R1w@Og20qlSM zIAIv5x}!VGS01f;x%!tAzfyCo_E(pGzOKA|W%&o{O$7b_xEDV=b+lJ^{Aw&P_gT(* z;Z?Yf(qA(D^2d||G|SbVe+*LVjsFXhGEfC#5dzT`ZXP#Ea|DQJ(U?dV9|A$L|+PQ@G z`d8sK-wt;Z0PXN)aa8qjx2E~^SLph-%~#K8^N+H;t5QK=d2=QkiePo`Vvh^8dS!@J zL!SN)UoddQr^J7Z0YB1hMf4vkqRS%(1=INnsmDU;|7sEa*TsoK{GCPg=ZffmBIT`- zb{TW+g>fN|mK03?4~nESUWDIM#Q&}$`ZtQCKU*Yy!OfwUBmMh4=&{;9zMgdaO3`n1 zf9tPK_xJ6h*LwOf_qV36*V%E5yFWC2UvHOk>7nV|X8$z*cQ~EX+3s|C^f{DHua}No zhs+*_Jl)+wGhySN?$61mi|GAkTuU8DzWyALar(PSZ*~=`<-#>T&3M4mS7yDm*I(5{ z`sk~t9FO#&;}#dE-|waX)&2BmJl*$4?-S|4e3k3pT>te>=lV}NUDLVUCTadpd%9hD zY+%1@4CPmO{D9LR6n(X)4?7+HpbgSaN56tT;_eJoLj( zZy`U&oUY|cx-|6HaZkU~)6clhr~DdEKjj*U_3KX8deCu-^$U%S@e4iXIsy1Wx8B8_ z4jy_H@neop5D&eP__lIetLun|-b8$dq><6DV`o+7@%@omII?;sw2ZX+Ig zC-LJh{q4j&J!BtGfV zzeGIrY2uq4f0=mbSBW2U{0#BXuMrPFXNiY?o%j)#J{AegKhR@t@xXiqKjTgZ4?RKr zs7t?=cLwm~FYzc>Kk?89i4PXp@ik05^fdACbA)*48RFsRA>yHr5`V?j z=V9WZA0fWQ@kfb=K2ChnmG?OD&`%JLa-Ad|`ULSP*D2zmpC%sVIzv44v&4_O@}46e z`V{f#hZl&4evx>T>k{$Mr-?_oE)x&^3h^k{RpOyvBOc>!mU!sq!ienKFu$c-`^Mbn z81ozSxYNO-TvfzFPY{oC)e;ZAj(C)-o_OdD#GiHT+eke0b;M(xZ6Y3eGw}^>f7L=f z^fuyA5ADQ5-%338nH|JK-$p$A>?9ugcH+0W_U$GfdOz{-bAWj0gT$j9hlz)tCce}8 zGeSJ{4Dl$}A>yGQCLZM)BOdxO;!&<~;-McW9{aiz#6v$xJp7y>9{MTb;pb`Mp`Rrl zex4&9`g!8v=M?eKFAxtuFA@*^67leJnt13}iHDyv#6!PEJp7y`9{P3S;b(A%J-NmF z20i9<%x~~BPCWDk@z___5)ZwOc$BN2c<4#uv9E3*9(p73N8I>YM?Caq;<4_x5Dz^? zJl2Ia;-R+_k8*7#9(o7ym^Zc&551FklxsWj(0hp=b?Zex@z4i}$GSgEJoGg2DAx$_ z&@;rNT!)Bybj+X76U1ZRQcFDaI^yAHJ@L?!#Gi8hHxLiKiFo+gOg!`! z;?a(+#6wRJk9|uU@zC3eN4d5V550qUlxrLD(7TC8xq69*-cLN%%>m+}4-yYQhlz)t zCLVr{5D$Hnc=&mkc<5uq!_OncLqAG9{5(cH^l{?h=W*hpPY@43PZ1COH1Y8B4Drw> ziHDzOiHClUc=&mqc<58adGue{i!0^BeTIy9tE(5c?u|oWdO7q1O_Rb*GMa==H>-TuI`gHxQ3s@L+>XZ`=SBj zp$`*}a;1rfK0-Xol_4JbA>vW4QR1N=CLZM)BOdxu;?WO}5f6Qwc=&moc<3jHho2{j zhdx34tQ(J~h=+cLczJoK}~qaMx?5B)sxsD~-ypMp zJoMwlqg_rA5B((ZXqO4%p`Rfh^)N|1^s~g{y3;x0p`Rxneohe&{Q~iaTzy_79{OeC z;pY|Np~Zy+A!Y9tpbz$r-(BJn8KCE}r9CjP9OPp%LT{VMTTXJ?3qevNpnv$MoQH!tw?#xLq2 z)?mjG=0oUlr(?WhUtL8!^aSznvzB=1b;MtF^FuxH(ANw5VdCNE81c}L5D!0(5)b_t@$hq;c<3jHho2{jhdx0({5(ZG^wY${&ojhBpClfB zo+Tdo6!9rH|6Cv*`bFYVu1my2pC&%((!WeR^ee=-ERLFq=PL2guMv-VewKLX*NKOp z!Ci&tx0us0zhNGT6A!(L_zvfPf_Uh4#ADx5PdxM_@o4u3;-NPZk9J>2JoFagQ9rH3 zLr)QpdA^N!=Uqu!c{hu%Uw z>aCS{=y@PnnliP@g-bpj`YGb! z=V{`hpCcZAo+lpq6!Gx$0`br<5)VHw5f6Quc=&mlc<3|4!_RBPL!TuceqJXYdhpJ0 z|A(J3r(=GD9w#1tRuK=qmiQ?*zUqjFUQayAl_VZ|1MwI~jl@G=NBo5Ir-^vzEyQD< zY$YCg8}T?VXeS=}R^m~v4&tG2BOd1koy0@mPCUxhO+55|;<2s`5D$Hjc=$O?JoGg2 zXqOSD((5sA>yGQCO&)Jk9g=uh(|v^N<8#q#G{{&6A%3)@u-Ih;-Q}+9_!j^ z;-Q}*9)3;|5B)6ju0GE(@AM1A!_SMvL%&2k+HsnA=$DDdx^{(l=vRqHxn_umevNro zu36@t9%~HGhbULv>6j0pR}qhSH9Y<%@=v#?LJ#-KceH-(x9y*zKdN1+tv!8hA1H{A6LE@ng6AwSr z#6urp-uanf-sz*nPq}tEOg!{4;!&<6#6v$yJjUrU;-QZdpLBj6Cm#Aq;<0W{5D)z{ z@mM#{5D$Hlc$Dib@zBo^k9G4r@zAH3cjdjnywfick9BjJc<7gjho4u7hkliKw95?f z(615S;ryH>9=iGB6z}{9>t^g8J5ORhgdTS~&Zp4NtB8l5VBYyt%e>Q*#G@V>h=<-t zJl4&1#6xc)9)30x550x>L(b1u;-R+_4?njO550qUwBt77p?4CGb#puM(7TC8xq69* z-cLNrH9$P{H1R0c2=UM}#A9APL_G9S;^F6E;-QZb4?m9(5B(_dST~Om4}F|@*B_2E z@AL`cQ4gnxhklxP)WaF#p-&QzdN@lw^mD|c9?laF{UY)3^AhpUr-_H3mx+gdg?RXR zm3Zhg#KX^P#6zDY9_@IYc<6z9&o1V-F?YW;=5)-5(DD9J@RN>D5D&eU(nr135f8nd zc+3My;-NPXf86!MM&hA25s!7MnRw_e#KX^4;-ROAho5c4LvJS@er_cm`ZnURPIVFw zy_@(+S3kYPL+>XZ%K4*!CevWvQ>pbz$r-( zBJn8KCE}r9CO)~uUJtoKJoKx?!_OJwp5B(_d*WCPhjCknd#ADxboOtLbh(F}gKS@0F)5N15&JYiMl6ag4oh2UnIpX2x zdE%i@5#QqGhYQ3*zeqgxU6+W5K27{!xt$L$6Ayicc+|r+;-SwHkA8BUc<6zCQG^_r z-*BE4b2|2i(Bs6T-l~X)o**9eR!cneB=IO$1M$!siN`*C9r4hch=-re#6xc(9)7kG z4?RUZ+P#f<=JeUNz6!!YsCM~H`? z8RDTIA|8H@5)b__@$hquc<4unho47@hkl%RoClpC9{Ne*QLYK%p`RiiMC6OVmYl6dHi#N#|@9r4hch)21aiHF`o zJj&HdJoFUtC|4Wt(6iHDvf9_`pbJoHB5v5#FxJoF~w zQLbj5D$Hj z_(|6vhKYxsAs+Q`hFRTkc<5>3v-?Kkp&ue1{cx0c=!c0%xyFcxeuQ|G>nQQij}eb@jS~<3 z1o7E@Bk|BDh{wElig@U!iO0NmhIr_c#ADt&OFZ-o#G~FW5)b_n@z{4w6A%3|@hI07 z;-Oz9zSZ@&8RDT|BOc|NB_8^9;%D6bVtlb3=0oUJ?hEg5y%F`0a5{MCwZx-e)e#T9 zo_Or*lEg!AARhJBNIdj)#G~Gth=<-xJnF55c<8Of54!tDDdM4b5RZD>Mm+RR;<2yT zPCWE(;!zL1#6#~V9`!IlJoG{0Q4hn!Lr)W*aQ$tBc<7_VqaF?u4}FYy)WZ?tp&un4 z^>B=M=;Oqr9*z?a{RHu-hm*uZpCEqPT~9tmJoHK8Q4eQ{hklND)Wdn=p-&NydbmJ5 z^ozu!9xf3NeVTaG!)4;3Um-r__Ip=}hklKDHy&r1clveWQE$O|yKm(AE#`F0hnUCW z#6zzlKIP`~1o6=8h{rymo_Od<;?XV*#6xc+9__M@c<9Z<<9kJ0h=<-veBk;yY#ma551ds)I%@v(EEu;Jq!>JeUNz6!!YsC)5M=}{*Mq3JwrU|;SllA zM~Rft2u&?ksTJ)9yQ`f1`(4`+ynK1uwH z^ZzXI&@Vb&zi-`rb(@(mE;(Jl-`v-yovz64?W&o zc%N3~be=yG#KX^8;-S|O4?pXPhn^(9*WD*-ARc-X@u=r!;-R+?4?kOphn^xHezp-0 zy`6Y`UVba_&^w8TpWBIt-c3CG>?Iz0Kk@K$fOzPG%)94uhnaVJhIsgShH;-Q})9)6xA9{L3F@beV$&`%Q&KhF>keUkVVcirhM@zAG; zho2XShklWG_<4zV=+nf*&&$L^zd}6jk6a}l`YiGA^E&a+gN@<&3Vz0%j`<3DoOt+I zMLhHb@vUwjTT49jI^wY(t|uOPlK3XKKWrc#dMoj$hZOP9+lWU!v=a|~EAgm@4&tG2 zBmR)9pHAYT_Y)642Z)D0NId);CLVg4c=$O&JoF6l$DRL&h=)E(JlgRv@zBSZcj+Ht z-s$5Wzf`(*+;;EaxX1hY38$}=^C0(1OBcibiioaP8j&u}p<+%4+~WR#?k-0M{iILZ z%a*|j(mR$|;T-8>PLE4|dLVuH6Fl8Fw5M-iU)PRZdwPcU2lo#34fNdm)UG|dx_X9& zdiDohJD=(8+1m5Wo}K+4H0Y!0pdszJ;lA{6&?U%lU)QcZ!xG*z(Dn3#4{mI3+R@jy zW5fE54{q4`RCCkL_011#Tz~&VJuQ9r_qCWLh2J$CZ2R~Vk3IH*M~8;?4t=z5_#-{L z_Vx8XHney5`#(B7)X_6Eyf4t~yLxx^4D{_B?&{gMuWx9$Yu__FclPbu7u>r;{+lm} z;QiuR+ef#4WLuZTvYEVJk~P^P{+?&I^bJd{eLcHfnRo3Nk{S`!mOHq2&)(s_d$)Y( zWAE8F+_Up(OFbqPBZ+Pr9v<4YLo!p<@)9wm_V%>(J-cgXUr4vr-Phms)KJgvzF_Ot zk3Z37%lgFL-e(5-!o}Ar>l(0rb?x1=Q*yiMy6@`h9o{=+n*i1N%$|Y0J-x3tWz$k7 zq2aw!!3_CKXLzKAL`xp?E2>Q53JwN z(z5=c4I7%8TjbC6^T6vhd_&8w2bwi%_}6QS7c}$+_wE}W8t&O4zxVGpzlZkr_6+yf zn%ru;!d7X#HY0WoKfAl@si$_89)qu;3cIqYckSyN?&{M)EB$uw?yf-{sa-8HDH!MO z-MMdg{R7#Vy{`8<2oqdp_m2Uf1yMp020%4(;w4&L*#QBz;Sy&HrxOqg@-i zcI-{d)UsRC?^^#r*LsoGcV%($rVG=sojhO1|KnvrApi1?$^XYr=y#TfgWPnT6(5$r zRIA7;_K%daPr7vo!3q4~e(U-Ylb@8c@$2*!n}7GuS*1#Tse*N{&Vf%zWX-?E+4wc% z@=Zc;zmY$FPe|c4zRfKMCmyhW$8QOTARfrylRwlauEVj@Lnn zaqsunqx;^g1_AcOhM}PQ-unN)s{U8i|5tVE_LrBhTA^ziQ$=I1vRF+)8O!X6-Fx`0 zhoyqj&AO}lZM8qocH90Bz0mgA8H^nsv}O0ci`O|@_S}EL^*mnxihDGjsA%bWZPyP% zza7U$KHR8>KB%yTuG!e(|7T;pN88z9&>rc|U&dG0w zdruq>Kj!>S`1pYn;oHh=fqnCj zhs#f&2sgE!4)-2-D?I$rrSOFvmq4>5ROUQ5aAE$F1N+vXpZ(tn&uPK` zewoJm<4|Zm##?g`{hSETZTWV182Iw@r^0hio(J#L&mV<`_a7X1dj2y5p;ITq7w~Gj z{8o7G$&=wS=jHH)`R75iEHtO($$?(zqU`)}^nD^6I&mso?mQh1&A$|WdJXZsG!!~< zU|{aaCkH~{`NI5fhxg5Y3w%Kjki8eLflq8G|$Lc+oWE?ZP8gl)c(K;`%i*@2_M*^v4S_? z1DZy-Nxg%*<1uP?l(B->0L$TRL|;FIpL|eJ9Qtr$5&X_Ga&6b^$Bh0P&mTXqX>*lN zlK&~+>xb{X3g5ivdv~_)b>G1Ej*j7bM=icbKB@e#-{ODDUvK1t*LM9d6e*tk_Rs_R z;S2v%G~%|iw2Qwg8X10hdEc+svA$HykF&Gb$j}el=DNJACnf>LVB5Wrv6UCLuKREq3@~J=W9-tZ(>7lL1Q%4L!gP|LX&w{p+OA z*^PT1rMU*44`18$nrSYH^k1Ku{BG_*bv&M{l z;E8nAn20Ode_SzIU&j8}&C=(3LNq?n&(J)5e2kR%V_Zv{e)Z!Djr*7@(=(DX&l5^r zQW$c6p(sSQA)An$sK0li*K^RH3-L3M?CINq{=bRqTD;0}-;L`5Tu=-)L{54m8Ui||{r)30FzviSwr{On@ZHwgQY9h6VMzmD~V zi&-Dj`yVBnD*Ipg7HkDyeh+?p88-Vj*e~V}?C_OCu&cq2jnjob$WK46C=6XoD!NMc zyQdGcKAwm3z>oQh-fp#C`Cj!uXw1c~)T0v=ooN~c_@Jy4Y$aX}biM;u4_--lr85@| zJF4vP1zLU0?W_`*T*B&@d3q|Yy2fndZJKUl%sviFo-8&ev*Y&5BCjn}2%Y^eA(JHA zQYUWC{%3L3cd$ecTgno^(G;Rw(B9bI5$N#N;Z9w>Yg>nJLF?lS zI(9U*Zf%Rz>pNxr@deSF`nDFow}UO{@C7>-2+?ZmJ3E@a?d`SR=H_kQjz(Xgw%*&` zQ7f?>zSj1(fWITqw0P0B7GI0ExoKN#{=;qgwg0{t3{ab2u&8!%L0)ZsUS8hfMYZ_1 zXi459`42Cxt<8g+Law_&jP^!vz}Im12omtM`vQ+cB3LQlYrZSI4RUzz3PBotb)DO4 zw+6f|K4Bdzz`MW%!8EIs$ILtBV;CWoAg@;S@iP0JQp#|pUlG~~|T$EZbm9X41C+O}?O_jTm4+S=B~YonOe z%U9P5Ol=;vYE9mD9=)S@YKi>iuFA=76q)7#X~S;WT9$Emq7}^kl*7l5g4bZPZsR+4 zGwLZ0$4Q~3yaafLE7nhUjv%N5D>HEGk@O(rWf$P?qiIpN1hcP8dNs77|1RJ*oL^W= z1SMAj>7EMs-c8rE)IHtZvza5Zr+c=8-!ewdLBTRs)Y;VB;BN0~@9?#_TRPi2+zr01O|3rn`ihEG>uNVFuUc1D zv8Hx;&HCkQN|u+f`4#TP?)lz8eWSa*1?K^ec-`KKx8{@+$u4LFM1GnYiZCWbxqk9(| zu@W!43p6ko{_~f8y6`ev)_P`7aWL}kZX50&mhc@o+G@dT8xSZo4qZQ%*K6|n5w0bK zKq0>vJ=+-T*`-C;uYe>24W_ZIE}<_zNphU*_|C)#KY&XCMV*r^phG%UK%W|5sWiC1 z@`{c9BAfN>w@26&0Ep=ZC)=|x(ay)+l%2|syV;MrgbeH{;L%@C5EDs)X#C|}I@?0L z=-oEK&wZ$%%*5-QlN}D&?P$Rw{Rh+7;6j!gwB1+v<02N(4<@lU4zXFm1lw<>yt#wd z2ea)3LET1p#t0Ajf)gXzs3lB>wlUtUV3HoCKbgkT0}~S_QvU;olGr(-XPiEX=z-4- z+9%Rkr|6&P`7{&s=pXv9kvRU!4>H#Gs3$9&Pl{?>e)yV8%SdXe}?{1=kmVE`*WtD=C4D4=>MsGEJF6M zJ7CYj3U>c4qNM`)E~T-rbd8U2a}Yk)3H|8fH&j!O4@! z-^O$Oi#j{=vaa+~n)P*QUf2`;!yZoV;D0d9 zo=X0Vh0?zT`T;&8m^+i!P52jYpBscZhN`v6B;PU+0|0)PP_#n5JCdMlW%$mdHq4uZKFhA!-*jcov`II*u^|=vt zN!IHwYcqrM=FoWah(Fk8PnTmJpKte1Pi24UWHW@=nNHt z-VS=KZA5>}$?6t0`Edy;1hHq&lZW#mi%v#cnx^s-!JfW3*lc6K1lL$|0n+n+HBmH zl)nY%%pq}N_*t`}_1F1);r8J35A8``cJ$BrdB>)gWvbm^_?$1mQf6XO(a(~5mBeM9znd;LMp6 z|F}Q?ocdpB>c3~#^q4;l9>n^VU?YFyi^A!J@Gq?QQU4R*FS ztspQhA?mMK4`v3ZO(Ximn7>WSq&oMvy=d=3d*yFyNWaruCVz7P?-uqvHl5=gsEhXG zk9Odtzrnwu-f(e@0Z_b=g=iTo!%1TrquD zkiJ~8`OW7C+Dm<-|BN8C1py-cJ!1W(`N0h+^_69om->sY2ZM#IG;6j=zOnSxvkUpa zSu~;bE9Xxjzs=xpB)`MrZ+0F(?IiyxTF>kvf1vfDj@C=jUd6YWK?kRI#Qe>{^+Eo2 zPSZ#8C%l!f$C!UBVLwsC{IkYGvhNupkn49raQt|(KOZ0QqBJ>|*Hk>{*@dNF#VWaUx!d&r0v({HEyz6o33Sn2gqMnm;;^mpb9`<%T~~{JjD?#Lt&ioks0Py_0Ta z0sdeY(`na}SU6!ke`1BMZQD!ONp>^7?Arf+2_Xo z3IFE#qC@7FBo8kt{|ML|e0}~ODt}70y@TaUjfeHjiLHXjr}=u8Y@=wy{R8q6p09Tq zen#>Z^Z?HX5dM?C+m_7pV+YTV?YTim^6Y<%WIM^9&>r&h@%VhmD>MsSuMe+8G;5gKcc$&UOd;;o5d?L~hd*T+{Vg|`*PuO%G)2Q>i`~6A0sDyqql5 z`F<1q!M>T~E`FL1khdV1nHh~|R(^>89*lQ=erWwdKqms>-)4V5MDxiwAF%$I{2h4H zd;s3$@4y@81N4&{H2XjNeLC5f=R4pxBWRcpkeBdL|G%VBJ_Y|mq4@X`zJ!SUi@8xJ z{hnKhI?@H|kmR{Q`!`2IOuytG$q5$yX3|pSuM_bA{6ZUQepX<9C!h|6N9TWHdJLZi zd=lzLybGrusy*S;Q8(f# zdkOjDZk9^^4SyQP-mD;hN~IS562D!8EDkYgfA}W%chXM8UnJMF zALSMI&&*Xio`HY`)uk_3F znW+E8@)>?UaW2h%!~6yD?YjBPpThqT-#+X7WpuvcNPfkO^i$*`m~Soz`6EAHfq$Cw zm4(0?*JoOdYfv+<*O;$_ zkFM8a<|}-DTk;j!j8hT*$oCV<{;(&`A2TNd8*yR2pZ*l{74nC;`O1tj@)h@t8|N!o zJYR9g&sQK%mYd>xG+zlC@{Q5?3gW3dE1ItiL4Th-AIYNimB&l;XURtpFHQM~1LMtb zgWpe(k7S7jpW^eV{OHcjN63HvL*^s%Oz~^<{+qH36uJLC5M*|O@caYk_obb$j2W+v zn&Oq=d>#ICJMk*I|Hk@e-fu6&=L=))w|8@K)cVBZGaaeX=S4XbPwwOMeey&|Ihf`$ z#V4$vrv3F+icg7rf6n6>@}uoTrA(7&F(!?|erH;!*pxW7-0AKwr^rcCAgFSQ>Z9p6mnKe@rFQ!Mc!A>KJ@LlWD6JDp?fzj2^K z<#ys-#ck}r)3I8DL9t&&dl%}_{V4La3C8m#;4_TpQRM%bly8dtC!Y_Q(fudif0I9< zMC0A$$;^#9<*VlxLZ2?(pikI8UAO4dv>zEefBgf__e%-p{1y3a>AqWwZ^(}f-w#M* ztLdA4;eG>EjmJpB6j+&$~H!&F2Sc?2F(>>}Syd&VNd~K8och`24_>pL_;& zNOiB=)4)9q+|$544cybfr>+4yJV&VQ$G03<_b;yf{=*+dzWQ=+an!H-(+~@vFc8 zUF01*TVB0(#xZ?ub;8_!mMe>$7udOm$|RMX^_7I_`j*n@L`ze)fls5Z`?a+OEu zLLR7Nbq#;B4Bp2%Q+|i-En|DNWeDV29Rq!==sC_rja30)(>x;ajv_s{)5ayDJ!Jrz zHjnqG#(OQnj(SmQQ6jwxQ$2RAbn|4v3#idT!D)=EtO$_8QDan6&kqq#e2lBFff4G9 zUZNf6ZJd=iG-EOR-V`gI(j9%NbJ10{1)nG3d#Wi&5PP&lupBM%@KMFO;p~ zJd%G=z(BnId9F_sp7?l?YXJ|%#A_ere@jgF$_DZ1qw@F}TdQfwIy4=HaR}Lyis;#( z(HlrutkG_!r?L95;^X!C*>Pj#Zmha-Cich}#Jx0k6y=)-Ic6?c(1S zHmqG{`gbhtmm+I6_UEzGD=l?otJhz-xr!nSd7d7K?|JiY4wdYuM z;|#kO!^Ijhmd-vY-l+3?G2Fb0i&&xJ!|1p}d(8MJX_nPpx32yaFAO_5O}mOLM(yNe z#8u;(Ec6G7OEhvUHrWzx%>CYkd;8Eo%v_DPy5H*jzx`q^mp|B zlJjpq%lo{Y?S0<$Yno5S9h->tv8cCq)Q-&q4~oU?z04T()U175*VmL_ug&<`w{{iY zM~C6qZL2co2>i1WkN6CWj$|$8=)%WM_QIdXo+fWxpuD<0DNQvfzIN@Lq5o;vLs4* zGtd8t`)r@hZuazxIWK+p#fdHKdvkyD+Zx+n&6+nO?U25E9Pa8$_ih<9uIGD70(nc@YJAmB+5d4gzEOKcN9~Eu+Meb)slei6O#Zid7 zSt_qIs4V;sT@7^te-~ayJW>CqBHMdRK#P+BN9zmmUu2=fBm8;!nW5u&_VdFr@%MFP zKk4zJ-Pu1D*CqFhYZfjfzljL-{{$e7k77bi+$4b1v9U(Ti38>(Ktc#*tsOzam@0O}YK8 zWjpGroYF=2|sk%oxh_M|9$@!${ z)K$$dbye-Q6|URRUbwtal;YJzi<&#@7Z){eTD-1$dtvi>@5YACO*K{d^(`B#cNDB` z^A;AbF5XboSl_auV{=VabG?7v`sT8oE9w_D0=BN2@FmU5)^4gPZ>-z6A#Za{`6}Xti-&C{aiB&CY@*7I4I~!Iu@5C6b;G=Xy>4whw zMa#NqyoQbCZAE1}OX?PH$ZIa$RvsOZ`giQbc^!G zqFb@KCD^!maRv0zQc5y!-Aua6Z!1}}ZLPOxP5!2qZOgY8R&{OKSP+OiX4%e~P2jt( zbj1_(T?I{$w;8&6qOQ0g&sS6AuWRwaJ{!vGTGu|YnryIgLzA~~-Rh#&HF@W;-tClqvH@1~*+*au=Dqg*!bWQ%|(&e3-TADjIm*TZ}b?4@l<^G124PDS* z+1mUKPc*D7Z{Dn7XukA(UThVu3Lrr<7)h^y*{}Y>aF)(?Utm%aq!VBp3=(um_FWP0D=zgGZ zxSmq?7P@JGpHnzoOVoXd)AgMG4LjlaAO1=>^~;L^pKAz4vn_a{FEQYwzlyZV`N}vN z;6<_05?%96(5ddN*O$=N&fZr?5+<~KaGVJyFNV^LpMj8pSE`o--}zc|;u5!e$s(Aq9PmA2VF;gcJ~0L2ul73uGHE8HSykl8eG5 z!Il3DeKGb;$faPS2k}l{Dkpl61S-D|L!t5LS6>JId$f;7Py3sA@JFr#{{i5{&lzq9 z#?-vhuEYNeNW6%?Qkvj(DF=-iZ`?Pfyl!a%HNTobPyJkxaMgB^gsV+ziR?c|_ERZM zru6*xl7Bp1iWfDno|AmOBH?O2y)OIdmj+bujew`CLDAD%)T!Vpz8t|h5*9b3>a#H__HzHjn;nsd?Bz(H1AEhtq$9PKAZRnq#bE3-g2Dp*_ z_;uh{4RE8pxl->|K4(j~Ro;GSAgjD@NVv71pGmk?-d`H}G4lVB0dABxQ_de6*BB%7 zt^+?J;lz(|HG5vdAF{yv4fIC)zAoWb{^=1=$j?nD(_1YZk4x0h8#6+o*Z|T0d5>ODB)^cMZa6Zt>b=A!mZ;{ z-a>_P7n)K^cNEvCD7c5$7;BVQ1y^z7c?(>{hkgrO#epFUTrIuo{8RO>*8ADukBWkK zOZ%;~z}0&1x4_kU&qUhYUR&4H>J4-;>=7EgtGu19^^K45pPXEd2QuT=9rc@S7cKkGi6t$@?0J%$j z9YS3Ea0&g6W$Bvg`S?A#`W=F51@y7b*HPGk-=D1O?C=p((4(OV+TbnmJ>FFBGr@)S ze2wzw?#5?(H2kyw>lQfZMX@Wc>~+pYuo+4`q95D8fa_qc6iwW{w^F+SZNg` zDd8wVvY@`bBmd!>*2W#O7ruGp#{I$6+u6XI+I;feyqJ&e>bG0(&|h{H3+yL+--{q3 zQ!P_ZS7m}aRpuWGDVJ$gWwg&Hm`n?zaF4t@BjcDtsCcF7vv5Nxmq!8Q)xT5RJTLPx z`fQEjoeam(9PIA_r16cpHMMlf;On7^p{ViI^*Py|`ZktI6}%ga(AfA2N@(KAs>8}w zL816n^w#=+m*Zcr{YMsr;#>8X4&kjfA>68fqA*{rwA*F-QS!_7&srK-`S$@I&wdW+ nKUd`NWIu8R)n4V$`!TR}Y-)10o!;@o<5GTNgP!DNZU6rPpc|U%h*WAyOke7!L5(wl;h!6sj@Dc?Dc>-tvYeOyV+HP$U1O)+!cY@y3 z{Ku2%R4S9ex}ojXMe8aLRmZxnlg_n4nL}qC^6PK&zt2AZ1?n8)%P_Mp>1v<<_qxtG z_gg?7YU}*@+pXcbulu}R=Q`K9Ue0ySeeb;b6W0U-0imye_)FnH`l%52?g`pJitkcl z9C9LWf3*;MPlc)u{se!ogsSTQ6~C|I@858btff`S(Erwd@7?iK=0Eer=)1oW%)HYY z%Dfj6d*2BK>VFy%2M4;u-s*z%^`}CmRPL)oLM-?prME86y!#u=P%bt4-W^8|42IhN z;V-LFhfgCd7zn-8hW2L`+<)TW?ysDfh2J;s`^t%jf~ir|(T~5qo6bBjeDH}+#J<WLo>KmXd_4bR;D*6>%J zcoTW9%)Iu(y(hl1<^B`9-h6!cE5G{$(q0^{FM4q}_UP&1`X5oAD>5^YUXQ;`pL`7X ze|vc5mKTR_e(c5J%p0!{f8~D4vncc0eY=4->Y4MW!}ZsEYdH4A3&Zi27lts z7+wv$XWsYxaQxrV_tR)=_M6DFB$L^C|A{lR?mjUS?VOo~w*Kw!@cZoW%x7O6K6Bq0 za{@(%tq6ht4y!5!a;9p14ocDVhvz9q@QkxSZtyKl*+Nn&(U zFf!U<_ALplJRK(5h``{I1NWsyH$E>$58Rm=ZNC}$!=g2CeCU#mPvhAgc($%@NynG_ zmTde1;Mg!iU~UcRI%B;F1jdnU5u%i+=6emgwcaYf$}@jI#OjHP0)(fq*I zj5mAsZM-==O7u$Kc{D86u4rwahdiKN`z`4u?Q(pjZfF? zy{WHeZ=dP|;r{^cbswltKT$kMec2RTF}g9jV)W)h{7z3~HpTIqp!<@{rZW6aqWj{^ z#;GetH`U<2zACeEIz5}3*;J3;IdnfIvuPfFFQfa)%%&^wdllUmWw`HuioP$8aNn~* z)%Tx;MsNE?Xtd*Jz1;WHS5e>h%f6opjoSSl?nR%`@3(rHPukD)syXY>Sc%ZvwD;X(=+_ujB=~-ySe4}uh0L_N6q%YjH&st^z|#Nzy0-C zKpd`$3D!r+80P+w&`V1LkO%nPgt_usDEQJ$$OwA2?(SWzaPy^N85|P7|zuHZp710CvYJA&OpWyj&Fwo2RnY28f~Z+ zqwPQVBI8eZ$9fYZ^NvI^CUC8Zgfb1i;mkZA&a>u<%?SV`^;ZWoH@}O%{vtI>^rHG3S4A^3b|8Q6_>uZ0;mplvfG6rr&h(bO z`9x}zXrkMpx}CO4nLeA#QlmG&2%eo8K;OmQ`eyM`8^(I`*HWYL-qMj-(0T6X`Qd@< z zkIB3FTZHpDo*$(A6=U;1LHW<|{Kn?P@xY`pecb##%FoX|G}-)ps(

{6C=l{M?)W zY06(Y7N6%S|2dvNOZlga$)68fBi_r;J!4_hEHKrknf zV7HLm{^4@*(mSo!z)rXt_QHzH&u&@{8)8}Jy*rvSuTA^gEo8@Lmj_(WQal+ef;@ z+IUH7bmF+w;Z+Z%GJhNuZ>NFVYLsgWB;P)Xbjnw~U97E&h_^d#1K!21J`{NSaB?R4 zfM>c7m!c0pL?0ePIpA}6X<*{!-rI3c{V5T*K4jru^{00S?$6Volaz-3%mH5ia%9fy z4}Dqn2RI$M1NZ1p8}K~YgT5gj;T-sFvDQG}sN7QIdkAp+l2oPwdFzs?!v`N-oUzBE z)Yp$%^aIayKjxtyZH1}fUeuxc(XkWvsvo^++^c?c^x_`+r$2$;ar#qzx3@n>0Ec3!jG+EBy2hl^+m|C%=nI}{ zyaV^3ZNeLPRNsqxg?C^V?g?)^KTY=v?_RoBcvsUs@SX|0iKZt3M{pknzt*K(IMjM^ zI9UlC@J!=S{T0-saA>{{_X>yV-MCjc7{7~qg~LPiT;b46_bwcIkuMK8ln@SUTsX}3 z;t-ew9Pmt!X)VpK0*ooOqjfe)d?p*V&18Ky&4}De6x0An*`PT00(*kdw9x6qj@J#oq0e#9w zQp44Mz_{>y3+%_e9`6TmugCjA-0ShCd+Jj!`c3zGyy+f&S_t}9J+tujlYp!8#M=Xq z5j3|9dJaBl{u=P?bm6(!hi5VH#53S|I6gCVcxE7;zZ7`(qMS|puj5|f+53mM*R=m5 z+$%g+egpRk&y)0A(;oK*@U-zkzPhvvpA}wwI>rGXJkxZ$3UvFIJm5q1YhL(c+!H<+ z^WFj6YkVHUy~gKZ+-rR3xyFa?ZC)sUy&3uDL;mi3dj9Jj-;}hg1nqVzd21kz==c!Q z>iS)Hf5MCRNdtJ}nZ_G*ye|`~93fgBL0OIaLyzEI;eO;#aIbK$eiZi#cjEx=748qw zbA@{|-MerPAYT*k`{C0~uUBUz?#00Uhd$gpkXAR~!hNk5_vSEg$1{!lEa1MAbOXw1 zyn7Ggp75Rk-SBPPE4%}bFXU&I5M^RPnAslv%uTU zhj5R4qzgO#4D%9vOmnXHJ7Vp(@0>i65BTOwnll^mtM%b#8ms>|#d%0C9{F=QPXm7; z=V|lz(Dy^GzWP<69U?@xZa7ipmH+edyV`xiI@{0CkA zTkq}P$^S$;p6UM8qknDS$!3(({i}W&_tZbg*5+q$ulo1UQQRw@y!UT#uliR__o{zK zC=dE~Ff@5Y>deyDPa<9S@u6pxe8n6(^0#6wjhkOa^&SI`$6bBc;_XA=Kah@R{ytPc z$8Ag7<$2swAD|+2p%y;s*R;f?_4{@v7jg|J&0 zchF+v>tbyZd5o7kzNGUF{UWMv{nW_mPt))9 zBd2HK_fGsyBlt5TKmBGfb0GBTk<)jI!v{iZsodcMccR?2ln>8aDBo|5oW7D^q+d?( zXGc!w5scq_f`4P=bdebSlb>bx!~dxVol1v5f1*k9InnfS@ApmUH`b5qheGAt=kyWL zbTTx)zAlV;IZn*o1Q|92-w=M>-q)rzZ+)k=1-{)i@bRvOueSm|<#PCx%Q9#3R{orQ z$`JdD%zv8ih%avd|22cZ5u-V%V>G*>qEz#$KQH{k{<}~HYqz8G!@1@AQI7dmM{=w0 zb(-VA!@zpV`S6^YLmtG6Ms;>C(*I9m%`3o*Ge*`r66O2U^zt7@#5Ok(- zIUh}QU;ieQ8=ZCif1Bg@J%@U8Y5#bRc;0%E^vGol-Q#ny<2fM*d}zJ+LcHn5LDN`| zVQY@GS(_uRb^V{svEI++SZ`a7_38F?|8B(`zZm*_9LDsYa-^xI)$7QAp*c+R>}-zp zzLR5pSA0O;9mo+M-OoSEG5_~+%=i5q^Xc|APrd+Nz8IeLmtU7-yEg@MlaIRoO*z*4 z(;VwHQC}{6^*4bF)8qd}{dyX7&V_!j=QtjEoc}$?{O{zLZzM-NcOu_~`2K_Hv#;lf z^EYzLr)l@5SF`uT#mykaf-uygdd@eRxuCnT_ z9At%?9v_Ac^$XyXOCJLlN#n>x=GXdVL5^}fo@4!cKY}!WC`X!qD@Xp*e0?CtdUbwH z^MA~dmXH6E_5Tp{Uq}vIc9Hhfr!F%8)fbt6Q4aYp*50nop-%Vw|FC`H{{nuw$O1qA zwB*P?y4){*$#yQyv7He3G#B}vhkO^}ODrf|Fpqo8a^V|y{#7ITNL%C41TBx6hzUm70GZQS~D#M{a--z_=jTc2aT zFXWhSO^*4l$uZym?fm=@WE{@%9Q=o$X7>|6eJMwN`eof0xi4Z{%2!LnBlYb>)dbSdO21;zkZwh~q=b}UuI~f4$ zYx4&4mJM6F1o%q8{zZk@?Hxq)CKwEg(6~?_2x~2z57DZwP@xFpcBed7)vObO;s=bQ z&Ncun=}LZA;kj8vQ(~cr0HaBib57UBu5PotV;%1F?~bh!=HIrijkm0~#GG5Ub#It+ z%WZSIH*dP-#%&IK!*&JVHpjlXVcXWuj&3ohdxO*!v1^@8i>U-QYf?yVhdH*Pbxc68g-t?20PxP~7s`~2Fg+vc^c z+h(<`-?WwLZ<~8r+g$$7&JJwa8qrVLua7}Dx;A#08`gh(I%00<+F;&>9$}Qs4O>1I z+=hO1d@MS$e#5%$H?`epc5K}sN5}u5V~*zurr8Z0TejR}c64s!N!Mv^>)u8)o}nAI z-O|Jb>;8AA>Q}-SP4)KmFKDVLi6o zq4N!}QYvm)tjl#tJ3^Zx@B8~dX2}1TA^)eDg=1O2wykZ`E!{BXJGQhnwgI|f`}&Ty zuB~$$t+wtPVDdILwcWJ6!(4B6Y}(S+4PDoE!?rDL>o&l8g<%Z+x9x@v>)T*zx0xHd zx5Fh^-?nZ?8$TEB%ID@*`F0?JDm%JS4;tFO^_DJwWB+41edIh2kEkWX*8d=P)9L?w z-%Bf&PjB6@bDFhuOWU zd(*aC*ebE>pxR-VeXMNF_fxntkr=`%fd4{V2G6nT=exeh&&TEA)1M1cI zO%Z^L)=21T!1Lrlk^Njg6=DtNcb+FzLbs|%SD@>`K4#!aD?{EZi>p>WqIx7Z*P1hNGD+E2C?XALr6m7!P43|xXhZe|WDu=Z~zUZDVYA4;7uJboz+>K_- zD5V%wm)AaeCTh6-O7(1`PV|ixJ*N79)$P5k+Vk@OJ;sWc_$vO6W3lX$pIxawogDZQIf{I|=Ez0c%9bx%1~N zn4SDA{mf}vB$jR8v}Jv=Ye!f2hONo1+q=4x>o?rE>6Q)2wXLn6_;lN6ufG1%t6D$V zcJ=4hUj4}xSFaG4wkGE%FYPdI*qH3v+BUcG@{S~$CR}OV4e<;guA9Z(cph#r1ok@d zI$*fO;90eXp5vZop1!9%e4mi{={}d`zwnJ=@FMEFJS^LrNEiuGDaymacdA8L!IMR? z2c9Gh4_rah!+8whIpMnpO&TrgyX2M&pJz?}|W_P|*07VdN&M5+%Px6zgeG(+ZqDFWEc5t^J9 z!A}&4P+KSvo?9-4AbQW@udx{G)fvb*{I%mRJo5#fyEK144p+GD1(Cw!?+5YP^YMp6 zSwr~4g1RWiUpr{jiNE9cTeA%L@ONk>eyLZtCGh`)igcn*a&KuYaPD32F= z4dAbT1M0XL`S5ofe`m39-?;_4#6nncu>Ke z3O=Ubw1ST-xL?69D|kS`Lkd2m;L{2oRPb2^A6M{*f`=3=E|K`2Rq(wkT|lr=fBF@i zR`3G~?pN@Df(I0AsQL~mxLCo13NBahaRnz8Jfz?{1)o)Lqk;t{5aF{>!Nm$rDLAR% zl?rZD@EQfD6#O{_lf6LowJW$7%81~N3Qj7xQ^Ac2wiKLFa9Y7@6nw9O+ZEie;7$cU zpy0HE2Nc||-~$RCQ1BrIA5!p>3LaGOpn{Jp__TtD6ns{}XBAvGU5U)6cl@#e}2@^g<-{T72 zi1CdFlA>{%Q$Nu=rC_2zn{}pGu-tj#32Ps zd@%TRGF{^{Fw>nsDUK^x=Wnd{=085mogd6LTf#bjyMlH7sX6ZaNim>cnLjRy8{G99 zqQB9dzf=q=SmsZN;<@hprQ#4a#OU!11QKFU!6c7~o?^aCm*W!_jS80d8Dh`}uUX)( zAN=HlJDa@eWVloNy1srNoV?tde#i%>7JAc<`{4FP-t;v~Bz#)&$MIznCi%wmSHUFD z;sNl-a+yx@j?x=fNSNdq(Pwa_gpaB6ovS3Q>ldvO*7X-FSl8bUJC6FV>mO9GuK&?eqQD$`OEmXNtom>_3yZXN#0WbhuUR2$zw_v9TFya zOXUYPNSNd?rKfI`u&%!!_7e3^*FT_OUB9?RrtA9K6)fwIi#4XV{1(#j{JQ*rf_3>r3O2aB zA(EENU#{y{uwB1`b^V7FT&&t}yhD~xs`6-LW;Shs&x!Mgq89$8+u*QsFL-dP3f z_73%X@f}pKZohr6Ouv`qg&__pnE0LOHKbsJVMC;TPv#e@Km7_WR{78Fm+88E@&O6! z^2Z;PFwHlrzxZnsCVIvLagkE6&R_h8GF`WqQm{@R`XiZ6{7dyEzae4jAHn?!*7*kq zWV&u&JS1V+z9AYFEb|wL)Hl8L_bXV}e^$YE{oj)Lb^p&QSmGZQjgQK7UB3SyIqQyo z72g&jLj2C-0e)Ao*1tnOc+F!nzfK?Y!Hti5_4R-cE`GwBp7z0KeQ^7qdh;Lm!Kp(s zUFw^xIHX{yzp^6vXEJ>wk5^XoD_H7(NSg1+^iGxDsbF1xNWr@Nnm?EMb^4%!b$a7> zz3BrA*6GE6;Z08~Sf`)$rMG`i=GW=R6|CD&{iQekkb6$+M3fA?}+ipqT zX}Hq|5BcCV-}mMp^udjPj)f7Y8$ zFJ++eQoe)#|5m~}f2V?r75+mC*7dJB<}E*{U|qiPIhn4@4*;hA-K*qF@$(Xv@+vFR z3O3YuomH?1W?Tg-n2>-^$HnXcRGRIpAT zQm~ZwS+VBtW&WhXZ&1NHf8%kPuG=3_ux`KjC7G`CrxmR0KdWF}fBOk<{^JU^`~SaW zy2k&If_3@O!i6GS2|ZdOL_5j;VsLkm77*oMcq(xN>ZnMD!*N3t;yxY1{blO;W^5zf zioctv1at{5`u&CaeNX-VT>UPP*{BI#yVh&;N2w4l{anX&R>;TBa|P>Nu6BO)eBeC! z)#Ho)>AM4y{$Ehcz+X+FsMw*ZsK)Oth2ohVarxk0{5)JJnyf%zko$Uno_M}WBmfTq z&JF^8r3m?ng1BfBQ+-x2OZrumfiTA!#PepDx`Ye)P8EdMq5g{R<9R}j4pE?P+&x(qaDnW42f}Kn-WqP(q>@}lt>Ja1eLXr5% z_6XZ6pa#<;*rVVgz@SIV_EK41r1QlaRb$%!BJf>L^{f7P>Z>gj-ukLg-x~(;cs6@y zzCeE}%y5{-GfPOlQe;$^p^yy&|5_^&Nm9Cy{IGjRHXDY5K=ynJ^bi~>ci?+>h-@TX z8^Y)r;+2S4Y10$*JySJ?-gCP{Ogc%w2JT-9?eT^7KVhL~3KXvr{UUM3kH+&vk+?S| zvY|@WL56rW1b(lTxrtv-6^h4hk7mVBa8LXOelIXZkVc&O6ZJoKdu5h4q*Hng#;Zrh z7(@}AzZY7?C01>182Q80|L6CBf3^Uki~1w^A;~cQIaVb4ttikd7TJ?v3&0OrdC`0% zfxZ||f}>I52MzaIcvG4uf6VaaH}DE3CqMBY^An!Q_9#;9Zf{(Hw|qf?yZp0VR8U=y z6pHU`&xWI9d<6)<6%yvfJfcTf!cVGn4G;R@V?Ow}7dFVarutQRgN$=GeA)+}^}!=P zSU?%N$`_C_cEkM+?3vG?e@kqQ>bx_4UV6I6ul0iC=b`8K6g|l}WqcGp$#`|)7bRus zhRGOk!{t6W>4WQhaH9`i=z~)}c%=_s7ddV*a8K6=&xT={tfB&6+Yyg zx#$%oXX?L`UgW%R`wBmDE*K7up_iY3r(iu%UZC0=e=O#}KCv&*w|7UR7UqZ$6;lgf ze+1@s!~Vdf?Zanwlo2SzCHPxc2>zZNCUqh0U*h*kVVdp^ToK;rfy=|hj!yc-FtMux z7ln!38vYY2TK1obf4Ua-8|;y7q7U|E9wQ*_k$x*LOvANC3X z&@o)a_^X`?|0XXFz8$P@lz*X0M3Bz?$Nm}t`?wbNF2UI#;a4SQn4vuK@1wFkl#fQp zTITYoe})+evHeczfC)bsU}>F>`odhE>O(s43lYAZ?RZ1nKOujOOcw#91OKMFNJP#* zYTuoISmrNxr6Yf$FGBPX+4-ve9uXmq>P*4ELVI~um{|>v!m53TK4`xsHi#s;r2Uhy zLP7F%+2134EfFI6v3<<>ALHCQzvTaA=pwKt`h)+gaEtaC-UtK++{CDk$ zX2b9gFi5Zm3*kRSJpPl%Uqb%9rU5S={bm2n=bs&?`i!hW3szJg=2HUtf<}w`gYm4e zqRh`x_E*W@i9~q(=pW_pxL}bm4Kg@n`R5yG{21=MUia^}kV==NAN9v52T1LMpAu#? zO5-o-Sw-a;cFL!aN|!AEyqQnU;bQ+8{tIEFF3=}o=3zf0>Z9=nJ?^$*GBI#}i2TW; zT@f{%@fiL8Ycs~{6sQmWOOe=T=4aU-n2-4jnEXM4;g9nCKH5e4UtO@5ezGGg)2Tv~N2P;K=`Xj6-LY&oit#W6b|&Cu!vvFm$S~?_*+GLv zm+XHK{_=5}tK=WQSR_{LctH%}R?r@}GvL=9JevVK`Du^S`SPDzUZX7WD>jB*>XX;D z{7~b2-7adIE^2QN_zUvK#joJ^;5eje|FOxMfx%Jo0qY+Lt4>rQoyQ+=lQmPMbvnkY z!kW(fIHMH!V14Czli33$F#cG7Y1#D%uh(+>@9_Kz(h7_-A72r*fHgfl`sU$6wx3Y5 zyudGy-EL&XYXFHqrx%iZk1&4ZkHUZc;dT+`%?Txd!9Vav4H5@Zf6)91XDNS#@P&Q0 zQp!7!6Y?7UDbec(3nA~(pC&65+R3ehUjbu2hN=Hd56X{pTR$Kj^ym4={X_Z;q}%#} z(k;RAw{ux1%bN=5Ct~<8<3su+6teXR`V0NT>rL6`fe+;AN-NIktY0ACYJ2kIBrhTF zS)Voag(F#wFVaaLG5uJ-aeu;uzLxJRdqON9(SPti_;dSH#_x0aeyqJqMzmeq?kB1nM@_Z%ufh9oT7a@}06;>Y8KdSosh?!r=^O^Cd z`4>i0xM=<%|0O-qF!2lY7xypNGtQaspl3->VU*=*ak@x6+W`F)$s>N&{VVAy;`tpZ zE#msXUxl_k1O6)MG4Oh9(61QvqrH2(NIoV>o@0EP#SeEtJ<)oCXy^3~<}dR1iNDqt zf?m5FdYx&2y;S6>zsZWnD}3HsF7cuD z48rFO#%tFaFg#spxqfp8)Z=lCI1#aCD$|?BKTi4r?fLDuCM%Xt{KoV`I?pdXo{*m_ zVQ(=1kUfL++MaO{8V}~T7k5IR6%;f5%H{YxXpM8`BjvXWqf~#4?T1$4pF;4DJ%2#Y z0(-vJ5dS26{F4YJ2tAvB#xwsUh<`w@OL`K_9|__Qj335-+<3~b?46RHV(hx6@dbZi z{*?5TBuL&fKLEd$?F*gpJ6nbED=2pJN53_Wm%6q6N&1GwJM$;n-))I;#NU|j`6M4| zt?`jGV1xRL`e9ENy2t+>E3WMs=BIniapUBC1wWy_ebDDLblJs{o_o#sI5&Mkzlcpg zwD+Ku@6a1?!YcI9uMlsNz-7~~i0M~o)3222SLmYO1j=vIuaMsNLB3SHwz@fW!stoeuLi#>j@uP~pS z@dZ9)Pm=!A_6M!crvdBvaU%WJM)J&=kLVBNnL{s@XHnmLd>?&?@|X04c|NkfV|g}i zoHS^qJSl-b^v=h;FwMttq~FlrmhIin_^><+N4@ea-zU%1X{YnfFKHiS?fGR~aDFj< z_WUw@^DE}aD>;7fCzXCX_xyq#uJr9Aq;Egx^zTe7cE%x-U--_WyK}sHie=#2%{AoT|{_%*9 z!Phq)GJZ(Te7L@R z=})QjKKM^kf1y6CpFm$0^X4B)f3-+_ce`MF89PDH-Zbe?lmBD)=P6Hr{Pgm!rxpci zy>Tb^ANmXY0sWZd&1|Z^64(p2{|tRM1F-EcgI{U^+x{@pS1Q=?fAgYl|F>oPAqT%V zT0viMV~P;~eH$>oUV6da$Yc4f?A^z1pX!XyM)F5O5n5u={%&7S!Z#i;7vK=+@#wco z!iHXdX8M<^^$$6IeLdp^`!i}hK!2=cpn!t0KBM&^$M^SJ#?&;mulAEXZ%&}RxM0=) zQ!9;^!Tv}Z>)WS7(my19>GoI2U$n>XE{8tgM_NCxmK8edC!N5@&i_?!e#A2!dh&V# zuU~6>kL*9UzvJiM{Z^@`KUkl^rfc0l@?S!s9^j+tP5fT$q9@slS)^-wv9HI+-~RcG z^(*2ZzAzr_&r^kNdZ$@G^LkMq*1z2O`>doF_KpAZ<~Y}QG|+gkzl0;9&Unyzff|pN zUF*RJbfwu}$NEy9Y){&Ev>rtAh}UBF+WHdo zLir@k?`)?m-)|+8Zu)uT1?UBOCP_m%_Sh_+y*@ih^9Pr*$7WT}0>Jb{`_N~z-2LE#t@h9iaJ?VEf#WH>GUfmOy{3Cw_Q21{IG>`uls$|NrAD&mKkOkbNH67^nw__1NP|84{L&zR2sp5hnM|L*zGZ`IAP@nQU8NjHCB zd<}bik?xm2z`su7kM$s=kL4Gm-pvo-pM*^h;{P4lq?&{4jgA2VbLZhTVVY zTR(q*{;v7W^KZ6?U(DIFZT#SKxjJ{rP;rY$>1o#8% z#eRNk*?yP9FBq?}{4yh9_ZRhp-ZMTJzsyM3^g@4Iwx`GN3y;rv`NbvAP#*ef48JIS z#_QMT;1?zT&dV=y{Zh#f@QX)Yd~kko$vuK6taMa@rJ-}vN<#z*U$#7r9>);F&C#r!gR48O3y!Tdt|TmAB-)RA}0FC`^5 zKLaND;^r4WKY{+GZu#QpC+LeAZux@vW`x3vS|5CHekpb2>pA$P-{MH1ZI597mfG^o z&p)82XZ`}d$Ex=6L-j0BQ1S!p|NinBrSAFq{_-gz<*Caaz6bipE#GMWz&m6=IO~5d z`M3@8Dc@)R#QgS;TyG@%hxOf2<9C3sqwfr#{S$HQAJBv1RkR+(_A}@SdPHn_2716A z#3zK%f64@y?X8W-Yr4KOFlg;TiAkC5MQk3MLPAMds&D889S^dCC~%& z#c=Cuzx@RImU#7DNr{^uy!^!Wyi4D)e}&iIIQINKR;iy~tI57}$*+WM4}FXDoy&d( zy;`<+#%z9hx0UEw;+`+GAL%aC$NVAfZO1<5^@Sd1JkP+Nvk<*WaRFXW@a}hz#vdiHA>v}2E{8$UhxI|{p0EIuheH4508Dl-S~godOXj; zA5$HBi21{B593fZ`QNa2c)Vsi6^f6@D$X}z|Q>K~+d zIPBS;YTh3Vdxh7-T4HBC@s{2lbDaI;57YTex&N2XUvm9Pk*508dVIoatRwxT*Ms}4 ze2@J>@%O3j^qvCGdT9ddWoc|MaLOBkwy&WG+eg@6!Tw-&2>a{MexDWh)L&uE&yxLJ zr}j@pta>iW8{ms!jkK^5v#_RJ{JmHwGaEXq^Ci`wBJnj7yB=SNGf|cXjMu4 zX#XhEgVt0ZJjDlBdf+Crv6{w1ug52>dGl%fbpM-rl1ajc#|QPVG?!%O5Xk!jTd{v* z$&#G}6a6us71k2opSXnjPvdJXUPAF2ReyM;b;S~C56b=SGpxmn?f4DqZ?dkqg1jrv z5B{q~I{P!K{*?;W@th{>%EhkuLAbVO@nSo^1A7JQ|5wuB@%lgQ=iW725Pt}J|J#DB zI0}r&NMZh3$n)h&d;eP#k3ByGehF*Q{2}T9?N=mv_mp!{?f(RODts{Bu1U?&ehJP$ z*#ob%7A~~u@#0dl?+U7rU9TVGyg(E6e=(2u9*pmJ&-eu`#FvOa?eWL@zX!(IAs4*T zth$27llFHJK30(@J!lnrV74buEB~~ao~JzRr}ovig!W^^!%dnrF z`ycV-590i^yL`*`9~=J<%Cmp|4)!NkNbb@6^VSY^URGe6KIron;=ORaF+~BdD;#s4 zre#+<*QU!inN^Euyt#ioU#3(szesvw|MbKvn(wOrfG1RyyI_Z2?}p&-8-S-s`r*6< z&Wiw^=%OdV6C^DAZ$KV#e?>Y%``Hcb|8-z6E%uXmVcH)}@{saVdS0G{ag3W`I)6av z8peJ_nLkYNRKqGLa9}E5RHX7Fo#4VkFO2UUppKmiOS^ig2I@V`yGbb7pU!Q2hWii^{S7x_ArF;rt2cOD{N)J47(nJkI zU&PF^$t3?cKk7Hk36-+Eko`9+r^xbvIlZ(@me=VMC(7~?hCavV$kD%c$uGNhPM9F7 zuy#c8!n#7SYiDVx4TIl7ze>3ul;~4M^qVN-XMk~@#4yVy+U?VLn-eSD?Sq~yaG9Qv z-@A5}mEk}EwgjTRTB={;NA=PE{8HY(Fp>7BB7ee~K-Hy7r%L(U5|gtBk3!-N-cQe> zQlHle-b>h}Nj|f_y7x}eP%5U$pxUW`kmr4f0;nJ5{rdrScV!zVP=3mQ{sgU3$Pef* zmQPjQ@<<0g=LEKpe3SHqypP-b0sQaUQ3-np^%X!(G#1ME^I3{Nx3m7>^*lOnO6!N9 zH|ndT_Bxl1V7}wLjU}c?81g{2H&wzc4=BA#!jP}vmpqGy2LwRa8uB*JDpLeWQ-1(Y zP%!roFvgn+h9cmfgjvbsiS-V7-oczwM*QZ$Qzx*!Wv5q_a(@X1el6P{Fyt!|V9(6+nddxm6 zU+(u)>k+Kqjw}7f>mU6NKEn+npXm#EcAWJQ!Jsd}a{R6}FDnN>gBZBbpC+rRN$PXi zziZ9%N$9^yhkV6+BzeJ)kRSX%MZ$Ol81-YkJndn;oc30m<>d>wJdH=Iu@C13-2I!@ zT{hv(GX>2H=pp3)wC;%$-SBGjvdir97=NRyT-w`2cC^3BEDcjSWB`qSxnV}C-1x0B zXGgv1dDs9$S6Y=XDDb3rP2zX6v(bb4r*%gnZhTgmg@vyCWUnQ`M)1i-7nPtUxI{Lc`=jm*i`qx!Rl10|X{Vz_~>99YI zt|FewjQ?qZv7GG6f0Y@&l<>)Rro#gJMU5_X{){p~J{n!c?Eg6BOE7hDQ99RGD)$#C z`d?*6Vsz`&H$kvGwd*U8_N!ffoZK%>F!moAU2(EMRDU7@`0)e|6Mm97D^-+# zTmjOf+aJQ-vJx2LgFicaUO4#Su?2yw_N5_s|cEt=oPn8 zA22`Emy-ua-6#3FPYNr+GaCB9mPeG1{shf(dd03?K4@0J7I4Fp`Fxzz4@Fu(V7B7_ zH>#vOJLbsyme`0mO9GqsBhq@Q*FR`7ue@?G)vxXE9>ungf;|&u-@*DMAA&t)9@yg_VEltU6Flj9f4=P>^kBK3>bL6~?}M@3#gk5o&dCq? zAnkYQ9~9txb0kgacK>N3C@#Bxy3ue0@B>fVF#Cr~sK1?;+5SP^9D=offc=l|{6VZ2 zxMA8qw4Lgs{VMOYp2I(2e*o#yA9xGrKW+a2^zxtAr2SseqRyy zm;Hs9`UouR8~6u#`Cfk^lJD~u^0|NOY=0ppVcTDj{g?D5e?ie(>N^AHF{J%W_J4!h zo>%t2ZI3J1w#OB0+xrT(?R^DLn4s#1ylO@LrKJvxv9u;ml(10td|8F{G)V8mIU(7Vv;B@Tsyz)srfMoxYJsz!??1Dj0pT7`e z`4?k*zu_GIg2DcXVfzPp?2qKxaFqR#sO>MrDrr5>!N-`w{A#D?P383`f*~J)Ulr>M zRUh=dA`pWQ$BANo9R zp6xHNJC+XT>@UF5AmQ)w7ua4WB(a}17k?q*@fT>lA*=j_25))L7xcVEEbFzX#1cM<0WK$lsvvEJ=R@uNdg!#;CV~;;~+Tqm}G+xgO&92Xehc`3D|VD)_rIgjS)WAv+8%CX`@R+Sy|kw#O#ak&UG!o9 zMXr}P_{sGYh2JXJUyZ;9gT(ee>`fbHf1u2)ciHPNw`F~3#=AIgs_K=`aiejDce zE`I

E?OAm-N>XrphVSK;B8j ziu1?t<<%C9@=InWO=A5@{^4Y%`f>{fzcB2N^3NB>hp9|)@lpoTs{N{{FySk%FyZmz zO?du%6Hd_R3YX^B-fqIBr6%0bVZzCz2`^Y+!sEu7@YJa$9E+K7JZ{1zB__OZp$VsG zH0Y2&5;5U0kN55TEnaNm>+4OpsK|t)Q4?OY$b_d)H{m5qOxVxsJ9he4U1j0}0TaIZ zY7<@>bik-@p<=Wrvi_hxwr9p)fdPP3U)wXm`D|aReQM9Nx1)dMz;|?@KjeTHEI@nb zfTvDHdn94zUlB)p;=rdCrVMyaBy7O-i{bwr^le2^1D?JJ{@203W(lglgZ`?k;4dAp z)Q4_wE3%0xi5uiOPUJr;vZYM~PeU9DCmRTEAsF&kWXH7c=B(|Ik$gUrjLlBl1^G z`eu5(PXbuf9ed46Z!0Fhh!m-_JQoV@0zbZT_>cKBirU>@i=d81NxJ-Y2B-tG}@i{h`;q0`ISI zf2|7Zv4rpu@Y~uSBX=YZW5Rsk*8VUK4Nx%V1Gn~L zIx(f-ddQO*S8BC)*x&JUlt1~W2*3a4!+dWf0CTp=CzL7W`Di?Xyvq?ki5Gy>h4Khx z%6R^tpK$p^{OAx{l^DYZuHTo`Mrfc zavCr75wr8p`mpQAQ_I;NZF~0>*gJetW}LJBc*y>;4*pypy70hJAGSE~urG)G*D`;* z{!)G~%pSBoN&6)G3kJ68^xi@rLmDsj;iw<154(Oi^zl~qK>6Qh@N?9^ld67^fvEab zm_PUfM}3p_XWNUcPxm?aQ+_S`c%JHyqdj?x`tm6qpY?Hed+`~KKVN-usgKduj)lx6}T)e&rBB)vp5oJZ63Rw1dC42ll7WlRaIyKYi4}5B}q=>&w%2eR(VY zNbSMn20usrIT`nGeIbKn`-r1{Iok)jen|g{GlyW``yKqWJxKcuU2y#&eqVO*Guw-T z`tam()@Px=J?Mx}^4R_}Zut-CKdF5V2fnK5*&faMZnR&z{_I8s)(^K&w!c>Gr(ItH z>|ejG`NN*IJ#hIvPwmC|>&ugl_-HQ*+K*C4eWmh>IqW0uDQu4@eQGad`B48L7&v0^ zFSh+J*S`{{e+B=zvppgIYqwu053Vm{5TqaMN2T;gA5z~k|Ku-Z`$Or615X3C+viZ= z83(M(Pqsg#FSC5y=p{+RjE%8=NA*S zM*ob)qcXqbKf{3^IO0=%{E-1i0w*1CC{yHUpGcnQ1@%v;f9CXOBrhg(W&iGUgyjtb zmi|J*M6df7L{H;kr~Ib?>+&KzCR9~_OZF#Tbl}Op&KPhY@QMSb@}uF&;*umd0_FLaGD)*y`1k&gIsX;*-GI>8DjZZHQzaON$Kk{_kQ#k%CLE)zu2sEgsyD9?ph_>xhw zU%r5ESNyT7QpJyqQhnl!@shud598<)Ctjw<_$P^36}C{dA9Y6V*ioXywCz+1?Li6E zUmw}~6e5BBeY{_k5dZcO9D0WRwT1rI+>aO+?<(>=S4BQ3FHvcnbdc(nn?g!M4u= z|BT_2_RsY-!kcN8zB>+|4SOJaC;!F#V1MMlA8XO}yNT_!z<=BJs_dEiyV{;h(O)DB z9=%ojbL$-kpEJJ}AMIzi z{dbFx_+x*z^5-orDZsqm3VxHbEiL2^f{d>fS5f}?lJz@4=ReVfb3v&-LOwd`81mtL zO8A@59jz=s9{I^1wPpOR9K(n7fQP((@u{|t^eh1VkEs3Z6W4+j>-RM6Urn{ZISTP6 zR$MUujs37KydRce%ztA)Y)gAe;hTuQtfieYMEJD?CtBLmPB_-mzQPHITiUyvu)n3f zN5QMe|1@)ZTu1ZyPc8>MmEn|L56FmS@}0UoG<`kckzeS8_q+7m0H)B7j^AM8Wqb=S z)2n9uj<<|I={)iAe%U$GueI@Gre7)XG(H$p{;KoDpJ3z1j6c!FkC{I2w|qM8eXI1# z&lA7G#*dl)cpE=v`Ux99X8ckcKW6--jUO}qI2%7^{Fsd&Gk&p+A2WW5jUO|9+{TX? zKWyX2j32S_W5$o#_%Y)b+4wQz^ZMC2>JQ$J(e^!N{Gg2=Gkv_Db#{E|Uu}By2bV++ zir3&T`F&9MBdhHW2IgFmPCqt(_r@m1N95;J~fr4vu_ zs}X;a6@LPj{%!tM=x>|xCt^F4ji>k!C|zsCFDrB6tI)qU`WU7gobhiLt4X4#7n*21g7Qvgrufhu6r*x~o4+*E%>AoTxqqAXC-~rbL<{ZT zMtQVlo15uiwHA_ItAYct+<_;(eScC=bz0{jrm@zd+6hC>Z0ty1FK! z6=}*3_GhP3Cfv|q!u9ngJZX{%S67>GZLJB{)R=Hpl?hi?n()MlCOlz+3740fa9NoN zj~{Qs6%{62T57_Hgb9xuXTr&(2^SZea4crRQ>U76Nr?$ZA|@OTn{ZK)2}h$Q90-`O z-*3X9kO>Ea4jA<|`@PU;s`yrZ8m+^X(PcU@_r&$jR|Tr38}>D814xK6#$D z+TVltJfFw$Plb3tS^VIL+)s8gg#HG33%cZZi^g|50O|Se#lUyz@h$diEr|>jj&B|F zFFP>*h~c0$zWwN!3;RdJA9Fk^_fx$ST|NkAvqf~n58jmvG8gI(+taChS;?EwBns^!C zIUY6fGQM*>YT{*l=Xlh_%lOXmsEL>Ho#RmxFXKDMqb6R)caBF*yo~Q0kD7QH-#H#N z@iM-1JZj=)eCK%7#LM{3@u-QH@txyQ6EEXC$D<}*#&?cKO}vcn9FLlK8Q(b`HSsdO zb3AI|Wqjv&)Wpm9&he;;m+_tBQ4=rYJIA9YUdDHhM@_tp?;MYscp2Y09yReYzH>Zk z;$?j2c+|qff71Mi1-~uh(aK70udVTF;dr#H%*0>lcr-x+GRJuLLdT;q-T-Tlm#eC* z@eJCF^N&Zv*f8r%|9s<7bG(W5IQhQ5rJo9Pcg*qDxyPf9@urg39FHc^HE@pS&Nm)) zj6c!8XvBEkim&*edpznGZz_7|$1xt2=gH{tQ6s;fX66?HJI14q@h8g@gr-dTw;Ye6 zKh&6Q_T{L2)ca!>K0f8OXE{FQ@#-|1o|ofM$YYFGr(nG-8=rO|WB)LEjjUPP4|NkT;xdnc^IOVti-!m$1L9 zG20|)JjnV1|8Q@Md^dxP8skqI4?^9R{vclyiK^F&aUMf6jZde%#^XgApEl9@ff}De zemNd(qVXu?GsdHUk$)aVFyZoY6D}(= z;ql{5xT3;@OG`~Skuc$L<4ic2G~wc66OP49cr&^J!nKGKcNcu0C+;=&-@yOC|MGbm*84nIpKXP= z(eLx%eAkP}`|t=|($mbJoL_{03vEgGY5b2XM(3I0eJ4z}@%)~8|4Hr_gMU@;KLyHZ zxIAjV|Kxv-VEg?i{~HDz;SY9cJnwIYzIo6Ne~$AGAMD>W_z^xq?TPiijJ1E)Pv?<_@cyjw z?|^ClqO~7a!Pb6UhV_0_=l(^SFL3PN^oJekVgD%~7^>5!_%yvC_p?Tjp0!^V{AWV` zNPy;Fbbd)cAp;H>`E%^ARq|T>kLIhjJ_)A!;@rYrl2>Nf08b?#TC1L64mV$7#){&aXc^sDx(dP)DR zUsXR0{;02{Pws!I^F!%8UhXfAMUVRPt$yN$YchVTs6R~IJ-h#m{l#*=Q}Nr0^V4X5 z@Tj%lRDbUX`%NGIhR2Wx{Q3KWKLv04o)`MRv_F;f&-+tXT`zXx{5U}yCe!d|SZ~+g z>k8BNx)6WxE}Tbw7z!Yxh5zvP#^n5BDkW(Dg;$@a%l>($NNx%9`;n?X`sw=@s(vuP zB<4TK9&~;6_ZG|Z=7}HtTN3BT_$@$2dQg5qL} z|H+wQ?2p67IJG|x=S3#r|1A8|lzu?!SLKK3i$#~*Pj@On`|A=V;-G{Pzp%W}U)KB0 zk)PVIzc(ni8xg;z5{e)7!9;j}9;YAB{toMJGns!tta9K(e4aAvQ||9egv2*3d|~>9 zu&gi0Fby)G@B6IpTjD%UK5yAPPxD*-(z?VpM1lSb`4xUIukifjL6kpIk{8Mw?-xt^ z=ktYknEYh>Y@Sz)?`zTeB43bqE53*4F9SsH*W!DMWZxtqpRbAb??LRBBpBlxwuc8Z zfsh5Gf0W!Jw|jAW5BNHoOZurW=#u)P?eLdigzAjU?Z|E$mNZ`(%OgV7@C z@3U?z_&)2lg733#^MbpR-sh$EpN*2f=L$UOk73BOZ9C3O#mAfV_frY>;^Wg2#(4<@ z_h>j^zn=mg;$(?1zJGe+ezBx#5n*s?d!bLM;^O>o;_u_;0P6D$O4WFQ^NaZheBZ;& zZ?+)6*l*&Lzb#vk-)upCfp^kl{NxYr*ZJ8_<0rotAHSCQuG=Dww|i zl0#VPcaz8d9*CdM+p}S)1^#4+_p3X;&%3SQ`@Fy_{jt9tg1wP_S>N-u?NRz;+a4LV z(}S3hxAZCo!`vu=@zh}OZta%o}bR&`(44*x_(&SPo?%3 z`-BPfl3?Ty`?38g1oiwieeah(kb^eN`u?vM?U9C2Uq}7t z`2MfapFw}HuslqEJQMo%i!{>b3-#ez`0tzoNj{@LbF`Mf2kfopF%o|dc<`>&sQ4Nz zfco3@(*Kh|^mNJdyQuvke)jwZwi6&f^88T#UO+kN1MM5PKR7>&%SX+xWB$6D_d@Xe zFv>TC_lZ<@iO=H;WxRY|7^KGeCOfcS+v}s|Mek3VoK3O5Q*!>-gYTQ$_KNWo`8@OdmAnYbIx7oL-pn|I`kOuR2VKkN!>RpbsxAfqw#c#GjmvR#O5|re9w0 z{qc~Pysd=CAIy)_^ZRqi9?J*&FysUOfqq}Coy4W@3$4Ze_wC*i3x1UL!-u05+(i4| zgVFY!rYAT6JQASz7{B3rNWd!?!+_Ho_8@gWC&_V0MNA^H$t^1ra(FK);8%hBGT zOkiIgu3waZu3v_Hy#EJzX_Eod1$)H&Qw=qD zY>$wK{gXa_y?(}dzAuIPs@n&apX!6de_%XH;|Z>S)PBI;u%EG-ij(Uz#sf1em3N~4 zAI4kw9$boL6ZgaaF9#f`vth!k*WVaV--F9By5~4L#_%up_s`DwsJ+Gan)rQg9689} ze_v8{%=kW>p|c>TnrDA+4E|`Kf8IQ{Xa2s}^;F&k^MdVFYwbv4r!1-W&b>94v=zw2IaLRa!k$#ss6d1KhooADu386 zUEF^w-O_6J*DyZ8_lx|342O1*aTUg61@(#aA?t^< z=SMJKM)NgjKX|@qYkw2sv@_lm;z*%#KWI%?QaAO7fjiSZbas|pFID}MhD=K(87y1%iMY2aJFUrzf1=jE z73crs#b8T5TFn$;p3q8M-?T`LVw3kbf(_j{s|z{7;a)O}xLD@(g|X8!{6n zT!|p1LHcXR%qnk0&`$5lsZM?kndXT|K%fwmz8fby<(2jViAX<458qFyci;mJnJXt! zdbEd@(o5od8z%q83QAwe2mGwbCcdeR^hq+v@@{T$@P|GpvpeNVNZs|-tH^#*2}+M-Q2Mv) z^OKZ+uVeqz4$wF7`hH5$C$p)7`cPXF-qIcZ=LbJ(pG$51nytP zhrOR&|EG;3|C7Q`$~Sp}BY&QTOfpIQZTT9iobsW(im7grOfr7G!@rS#ruvQAsO%l( z6C+tEO_^VMAIZ*dLuTAKN-L@Guvf=^Hn#T?(xt6$dLPLdzoD4wZc4?+dF2KEA;#(R z{&46|`tNN@KJo_>CR+JtdzsAVqw4vxt^G;l=Y1A^Lnc;CGo;jJ$@~{P-UpkB@|hZk z09N_KpOuh&N}sUzSPAJv$4_QyznU{YCH%ch;vw}#``r{C^}8xtJl>L*?O}3+>JF{J zN_yHq=j2x&2Lqj+AN9{szn4SaSd^MqZBJ*{?}>3RwEQE|xAzM^xdZk^pfkOaFy)ov zKpym8=>0DzKFs+enWX;Wc75giOpH){*ZDzu@_hxXeU3yZ|6wbB*ipaZu;1}Tv>(jY zFKst)rtghX*ImgsM*WrN3-S8~kSCdqm74J*tgnblzair*LIT{Uq5L#tiUU+P9P-)z z8{>_JOr(fp)A8Y-?DqlL{)&wH40>MA)bvTbKWNJ1?1BY5%5EU#UO}D4jmeTeaI0&aaO<^ppnihy9?cIph07V2Dm6b9MiQSEqE=#Sbo%a z_)nb2!|QbvoyL1OugA;02*!CN;1>iv)B}F?L>~=v+Jt{@7?%Y-VE$Nst4%xj z7peIm;F&+pZ_#+kzsQ1_znnj``3ECZ)%armVdOvM;2*-LY1PI2ebAr6Oa92Of|-Aq z_d9S5mS5V7=8yFToENmr2P5b4bDb)G#)CiMeSU>M#p@HJ-ms=Wdwo{U4?KzJ9(mq! z!TQA3{s=x0wT$LF!g9P2#>|6)n`+QLgeKKgYM9!K;G~*H1+OJIFsY{13HvA2T;YU$ zlWLY~cn0XblWJ~I@Wa=`z8vQ}VLiGo6Xy4+q&~<#L%jYi;kXC(%kNJq*bDpR`l4Xa z%k@_qmfug4@J!(K{2-njhCchnByZO;Uk&9S?=PKu{4D1;$zP5+pMm;s==)U~Z!Y4b z(FalMt^Uy(q$lXS@p_t{!2IGjP&V8@sKNdRY|r^IqR{-~Dw<#Lirql-{Kv`fV*N-? zfAabU)@R!?u^P5N`F1rAw^`;k;4%d~eL>+?N&H9zy?W{h?BjKZSmIeX>BGUd+$dkf=OAdtxT` z=Xs^?=K25cqW>VjS9u9-nu5OQ{Z!I-HR%)ON%^<(%XqIh2Yx)?19>KA!-oF8?L1!u z^IP~HD$T#<^n4WL5AyjTBtPXxf3ISp&d>VN%j z{}_b*^W{N%LGsZ1c(~+xIO^B-c@nSc*N6Ir_%nVY>sw)c{x;T^_)_P;K>xfSj@KV# zc?5W_<6W|RK9LDyLY=P|*~2F?e!XEKM&(EQg%qCWi)j9c`#MymfT!=_URe3(_`EXy z9_4A|um5|37a&#s@IC+(A?EX(1msEK`=cgI>yu8r<9slt_r)*YJ1;G~Y3`q-vfh{j z2ERbighM!w%%aDB_$0p{uhRF^`TQ{N`FcG&tk3HrCDfPD`C{=uf3Rx4ScUmZ^oI!k z@2ZGAI8?42^p6Ye zH|>8IJd?XZ*QaE5;>2H)fgY6pV{yM%q_{%~dj$WwKI$&$_%q?3z4rPAr$0`iXz?7#$zumA*u&1CRD~0{E--4Cs&Icf3E`uKK&?d2%Oq z03QmQ@iASe*F&JcaBq>U?<5cOLHvH6-}?>CkJ2apgeLuO?H`Zy)p|)i`mf+mcrFv< z595I(#f8xjJ@AQA;-}(6etdpaueYqC@`y-(t<%qz(9rcD*E#5)_)~nFf2kZF5r6pW zFu!j^`pWTsPuN4yX@6FE6MxdT>Tg)`@%uzNy&Z`p^AoxFSGjx>4TB|d$`83i$`9G6 zBjK9pdbA53wr z^5{!Y`t!BD)MX|FXg+~2?Dwk6RE1@K08dUV7Y&0k9`CW&hyLp_;}blVr3mQXK%dpL ze?XTH+MfxOR?jNwF926bnEeavjgK>%uk!~ye_s*b3+DQr%#@9iJlOxn=WEnu%40mX zm+5(vnQ@~eALB`%ndB(Rqhauq-=~!R68O?liqCkGe>U?!NL>Ib@49;CXTh+~>L(oe zsY^0{iHH3x&G^cA?nnHZ&p)a2GJh-n(v087b5Dc^f5U_^|ZqEHl&XM7WmQk zjS(OHJ7fLaah@@M?*!>t>}WmCi|>*115`cgPvHF37?r=?Uk3g~*xmsCT%LY{Nxq&H zuXFoCFwJN5tayX>FCsnlJDZ9Wu)>G=eLNci;w0ffamcp%SZh;O;7RwM*1Az zPR}_$X8q~+H(1|@&+`|Yp5#}^U*c{44BPwz_V^q5hkbC_QejbX9fG07{SuMEc;XRC{OGk0$$o@ z|NfoH&tuvFa~z zc{=MK$t&#v@)AAzKflubA6fpgJjgy(d2xIyALyU>lfE>6(y!$pSU#(MX!+~{lk#i( zWIXAI=StC8|kt9j`ZF- z|7`z-`IYClbNbRg3+(G``{eqszV}Y@DSx88u*R!>KH)8xi$3gLV$n0-T z{`g)!pXbc>3wxveY8E}@qx`ZxH=gmJ{L%k|zkOp#hzCK>ll_$``$M(h@=7RslQ6|+ z|JvBY{xX>zmG*XU8}En0_Yt(cs`%#nS`=T}dk^pD#P=PvKUMVR`(8vZ?c)rUFV!EJ zKKXm(2j9!k{!7I--yZ|LY!8*b0dsrj@NW`t+0!_NZF`E_?W5EmeNY+x6z!R;A0^b6 zbnugSi(isqn_mgjYq%;K=kY$?hvEJ&D(_V5bXUl0Y4%l5B9iop2?Kb)t> z^7DKGmxrnUQjDj$KS=U}9`Z~3BmK$rd1yjO5I&qKE|={chJ}EW*gt(t`zOXHRC_D! zk?fEC(ZT!V`#hOo5&Th(=hGxTuSeh!*DvoDyeUR$QGS5mzByPNtfdFg(kT!67m=vt z|GEBzqL#jxpVi)5?YY(8kp4a*VXM91^)0Qht=L~@>Gxr53Oc)gAnS+hzmUEn&i)PN zcSBZu$U}J9{>c7rIFnTU1=(KUyhJXa0M8dm{Y(EZ$K&Blg4g4qpCGMYlxCt)@=v2w zUfezu$8G)Bity|zqrZdn%4PqP%gZ0m#CiP@EVw;t%M{akSx%P+=7XwfzCica!rt=n zs{b$B1K_EvtlC#U(Mx^m{$ng^%Y*fg&Ho>J?*eAmQJo9!v(G-yKK(j+b?enft(M$s z8`KhR5Focs>*1DcSuZ3*7!N4IU&5V_2|dRa-WvBsM{UQph;P)^WXAR+nFvRo7~_xl z^6`B!3AyeNhbW5UI2kVSBuwlmiqRzmD~1aKKhFIBs;a%K_Gzgl83zd7AWQ44daPQt zYSpS$tE#X*HT{S6plNR*Zm*7qLw}%s@V|0@Bcg}xCmEMwxG}n&`e^T~$c%T_s@c%yU|CD6G!|LhpsK7spe^urG zXy4(3?bxr;`$dwzy(DX4rboQo;e*BYX#Ym=#dbX&HNF-5M^&)U)YmHeiOjC@2d&Zm ziR8D|2RI+c`|R`ov{t#iK~Me`oXliiEKsJ|D->11pXoUw7=WLH|EbO`DDE1AoCNC`uKhX>y}HMAuo|UQ(psb-j84> z@jcAZ!qxm2KIH!<^>13<>ofWa<9#vPD*6QNqyL+%^XdP_e8l_S3KbJin-VxUyF5{eA&PFTY%iZfIPB3jt5oWI{1$lDc+JlsBan0yO4tmqx}kB{H+Z+|F|9g zv9#D{Ii7z6J=|a50aZEgg!75b{A6YLG(YLzqdcH};@@-9qdbS-u>TkR3NxqRw8LL# z^?pu0g&x5F5ns#lC?8UuoPXG=+Nb^XYX1lMnI8SDIC%zE*C7zJyG331&#uJ{% z|0~K@=XZOW+9&b6W4qLTUEec!kB0s}ei%PE-}PYFEut}BpNBmf&Y#cu2mVMopO5ps z-JJeZJbAl!Je)oMvyfNpFA4r(oEO&bk4yaFgW>E3^)c=f%Tqt`{IZM{?ccr+`)?wT z`6Jw4LV5Z;v5sGy47>3Ds;&?D@5TC!>4kqt4}MDc&ghL2w73ulm{K{l4yO8#Zp%3+my=deMaF_T?cD&p|e%)Qmc8RW_OAPG= zJX^tafgcX@*_huw;9oBLLl87y_s7I|UdxxF=Xh8Ue7EK=>XY|$#>%J<^Xr3nkEI9n z;r=n9AN-a5>#?$u|GV{jI2f;ZFN5bh)OqE`DfA!OdpG=%nqKK|#G~f-vb}MAIgaNs z@>}(%j91LBGQJ>R`G}tyFa1-#wIKcb5b!LI=NZ5c)Gzl}Nc=wPAKkw@?l0`0Qr`1> zMgL$wv<>`e367xx-0v^+PTZbtc+X(0Q7J#v<{yQ9N8Km#uKB?Mg8R~h9}mKxEB1&! zKYOxPvc}I!lpmC6|KR=&Ezg0-E7nH>gTDexyfEUM9sd;Q$9Z=-5Zu_T=$pj+0)DCe zKk$S4RrH>z!BO`gIJVx>% z_*OZvWy+fjJA%4ZzK{JeS-@)h82hDienj49ko=hp={NbGJb>qwSRuzB`VVf6OdWvP zkNv`aZ*HE{b+A8aUzhSS*OQ-5(Vth0VXQ}o!ubifD*?|-mhTtFaoA%?rhAhLq1%9U8bJ~mZ zC)1TD_D)=0P)FgCAMxtb<&XPM(G$^!`N&BR{wD9knF;>fL2<8DhRgpa&abt50{ayg z4yrDk-D|@oga>Ol&HP30I+H&`c)qvQikB2 zz<3k`%F*eVa}i9e6_W| zx-y_W4NsG@55X_^8)ZF*U+C|V^&$NW^4^~E55?;}dwlagWaxjyKV>{MKgoIt`yU9v zlP~a|?(sf-i>#NRp4IvW{F4A4=x>}4!g=q!AHR17`>>4jK+NALiGIO-fV#AI>=|789af741?KL~&MK0NH- zth&zx<}2tWJ%}fg^$_>7PviYEtgp2FC-H^_vtQ39V}9;?Ae*ClRQFfm{*qE&;*r$+ zzS5Nk$k#oP-ppP@Zwl+x&ir1n{Z#1?Z30D`-FBMXmFgE4d%!>G>b} z_%AVk6vTg`*E6Zk@q9t%L%A;-`tMh8KJyD00kj{$U!13ucpi!G!TbY$<|BUMeX(Wt z6eV6o&PQT>i~ASEzhUlQNXol={tD%N5%$_7&c{i-9pvXhO#kM*;zz%&`fF0Jw@xi} z`Gxi-%a`#~|7Fh}Pa^+%eB(Z_p~(J6e!L$!1nFXY0uGAU9{ek-#f;y6Ge-&VuKg`eeyh!~~T1N=01KO{tE5e+Q^}wUj9+Y6iO=(H`8lwoVKR=0Cj{_^jaN{$8X<{9vyBI^6rTPXG_qb}Ufi7ck1lc^1@< z#xZ{{)9>^v_^0MK#{ae*vY)+H@&|q4jvaO2vHlQ$9_-mH%a6`8hdX!512p8X)W2;T zS!c?_c?{62>)S_u_oHT!N2K45@|OXQ?~}rN6);Z>z8(1GfNlJ0C<+_D0v1oT)$7F`lDj{1N4o6-6sDL@Mi-wg_D2tAo8yW>bkwxAbpH4hZ~`8 zO@7i}1svm1|K`=mzbeXqEvBCsUk*o6Uz8v9Ux9Fd2%lnpjF;*)QT{$GKO;Qo--7aV ze#i^ck05=O>18}v=SMVtBk&s{`~d2+>z@O6Z(Kgh8%6qSb^1@#c9DSU738Oseh{jLF|pA*%$7I-^7-N{jUpXske`ngg1^)P4b_U+oW7WjD)eiK#;c6rxX z^2PI{3_r8+*W2<1{8n53z<;_^BYJ?}Zp$C=ZMJ-&eK)eg>h%#n7(Qpyd*h8;fo~3H-Gj~J~3CN{tIwF?`E#2<-Hi_lS*yr5(pr^ zs;uX(!2JP{{=j(0`x=d&pj)8E@c zZwKT@_b2M>3%T88!`yC*`t=7v&$Wt*EZB%CCeQciZyj=nKfp zaBcPIYCC;x;}XnI_^QI6xwtQU_gKuYL7XQ6JxZUT|3J_3h#o0#V}tpP{*(C~2}Wq| z0bdTU!R?ZozXC7sXJ>5C_8{c3f&q%JD)poMYwYsphO1Xc^hkaB{cq|&j*naVy?$ng z7E=0XH8fwtTOHNE0^-&1R_OW$Ya>p3hH8ZcS>{*vFvfR2T>&Kam5k4k3RP)DUu}88 z)!&6e9ayp76o2xOy(ap5xjLV#%g^63BC1*FeTe&u*85&+e5>*PEY`D%U(3-QNU?*kA4??{-=g2QGW^}T)yB^a=XALAUVpuX2fs=s zdYEH=Iq#(b-zEnfV*J2>&ClW5^;@rZ^1JVw;k~cfj((}FTW9B|y%&vF;ph7G79RCq zF>r-ben+cCKlFRwx;_WLeY=H6{(0+Rz*_Cwxzobu=Z4o`@9N)e+iX1ezn!WfZvXq< zXXDGEil>V8|Bf9tez>-6=Qbz*o;@~xF3y9y^~-tyIjW-1hHE?Cx5LWc7q-Jbi1jz` zw}|+M@h}(XnWOS#{KK9`g6p+Dn;XuGIRci8!dYn%6MHuXz-tWS{Piby}s z>mxibf{V1@y;)jc;(bgGr)a)G{Q5SgZ;kMsu-9Y!+;H>eNM22S$S@}=U(RcLgAu%$ z>j7`Gfomm+?~3l5!TAR1e{=s#t=I_zO~p5|{8~%9v`_0V=!Y9h&_9d0ABp?#;lI@J zXwawMgue*-W7%1)KhEI1C+^EHn)KhoemwBkj9stw3+^i|eAW9F%sKP^tK$!Yyie;t z^460jVt=E4aaf9f66*`@SM4SJ+TO=~K<@#F4{StxPS#r98^I`_{#VMcyyp!V@h`X_ z2_O0O6yDbeeSsGm{)MF?YBuqo8N`>tA*VmoUk+P}zQKP*?xU0ad~<(2<7?6ekNR*Q zeM^1{Mdi+Dd5JR z1b;;-s{d3Q_P=sICoF@v?vI!D;C@jaNH=)U5Boo&pY4tAtJm$1;z#^6%sRTgIoMaLWt)B~f`Fr~l0FjbPFn-S;l^VZE4- z^CP}0z8_wHR+XlSTgWVjsuy7sUJ9 zPtiYUy{9Pr!+xJt-%!}*bA4{?JL~-djJF=}8~Y6=J>sFP_Y7n_O8l+CLmnOZ{tD$4 z!`t8?FP-CUZFc(-@}J`U_fUA)W59`Y=qKz?w!mKzjW6P%fMGr@V3J`jE;TJTWV>JI|e{hQ(Vrad2z z?~>02ml?lF{@}cdPy1KneMdeZPi!y!yK#TG`mLG2uD(m^OZFGbGxR0(8}{4%caR)e zFRX+-S^5p~?&!yT)K3M!%H)z>_EUXjpD24jAwS4(={JnWc0T#mAvK9 zeoMb%JX7922E5TPl(#Vp#(kKMK4tx!AC>%PP@bhjrPkoB{J=wpcp9Y(GkA(9J^{+0E_JTft zfuFD*r9Pz$qwigLVEcV+py^NK*P1^7v%f>2P5Yer;1u<_JHI6D2b?#y_G39d z5iel+qnG{X&d;px9;-ge3-w_v57vCLkMdya6TZLW+E2;xiuf)+uAli``-A<%`9+PN zl*s>`7Cppseo^`^h5XKVlKF@FUyb(^+IQjkhwZ&LQ9kDbBY%{y-!ORSH}cmWAI$%N z!NWc#edO2l@5Qh^xt2+jUivrN@@p2H9DnZk*e>m{#~a6=J)a!K`A56INKdT)GT>MA zevZ~hmi(S*m-(OVGy21o*Ok&f>MJu}I`Rv9h4oSY$oM*~=2P19IAw(SG!@@p&L1H^ zGsPFpgumc>T^Jwk`e&twc(cVG?@Ragf}SG^W<0LI5}z8!4yuldDyUCzbf0 zV|n_|!6doQ1^TJ681Y2HkKQ)_vD@Ar$L~U3z@N2}ocxjYs{3G#ew-NB@#f-x{zMb)4i{6RCV*c-duH;AluR}a3?sEbp>u2;2 zD## zKCb|{XCcdv;>m*^8NZwsQUBewK>8E)>3DKCe*btlFJL@*h5SJK8Bfpo0-wk?;>`uW zMB^XS=kMgbA<&EZI;FrXIv%h#h*E5h18WGSph@TRL9P8Uw^&Xvhk4e7|fcP8XFXGMR z{EEbjgZ{~|vz7Y7v={NxtyG0E+}2u0RR(tA(-A*BH113MH0;fGp6@~bNxZs=PX{dL znKDtlx`N|)bsfJa`d{fAR8RiMedR-W%>Vd|J4)VJ4^IImzi&JvwKJ{ z+pp#W#4m5R&a-UH{}J$JK7c+^`NvfLiSa(MN0314+b&5h{~`Yo|CNYG=W{&~e;7|q z`yHRkk9cc6J_Lq+6>#s$@Y9+9F&|?7@3Z605&yhg4&dwfa>lQVzf8}UePK3NL1hw8 z1$lSk)6u_pKS9nzhy7c zsrr?F)6^f#qI^pKQNKj-l9eRrw6TgmD3IFfLe8T?2C-I2EJoIsA z1V6)g_b1|bckI6)|5DD+=6*29|AL4f$lt{Hf>_@G9^Idh^CMm!^16ry65`kOeTC2E zm48p!2c!GH+G@i%f3%3_!`rp}a$jwpvcID9ByF+%fbq-mDC;|IuW)>DJel)egh}t% zzH?%45*B*)srTLp(?A^Cr_OKP9llNS<0t9AsJwmZJllhKUxDj!(f7}BKIlly2c7YJ z@Uon5>~QCUr}JM#el;JgYkHr^>3`Y}wbr@RN2Y(}JS++3`A*y?FnV|j06)%8?2oG? zQ~^H-{rA<@#Pbc-Gtl=}Mfx7)kM8d)Dfu+(4Zksu>p!l)U=Q$pr0A3RC<8D)AN64P z-d6r1{2;yIyF2Xp>11s!*^Lja_+F1U`Lax|sy*mG=&!f0VHJA5Nw$A<|5Ytb^ldcj z!z6vspYpz-KkbfR;qO(&Ci*vP z`X5TtzwYAbuf+0jY4mTOf&S+6Dx>@R)peZFdKB`pRnvbeN&mrV=ugVWnl;hyIUYhT#P0EjZ{-6>0D>k21_nSR7iu03aAbPbc-8Sx?;g zZfkt*gFk+4LC#Z(JuLgx@UJ2R>|co=dhA`W-%a~)f9lY9S@!dEde{rmeJ{BGnCpwK zPTRgcx!N4(p?_f|=xTJS^cZNSU=ReRfnKUgm<%%#}?BOi~=!*;%r zhvk>bDk?=-iFKPU>d!zG=cXopxM*kfLKgx2wJLu#5(5#=1 zqkZIuoOc#}==1l+f7hknqZ4@{Ke}*T4frAVIYC~BN?Du`;|F`|MewI5NVIfV}Vo(cY3PkFW%*{f(T;R%Z$gpIul`UG$6Q=Esqvta2{%7>4a zt>hy=rjd`Xe4hM}^{((^mL4BUKJE_lvAs!tZXgD|;1;^^2pw(O>2~XvamK2gP}- zuE^d3KS%eE=l`?tlm6J1mc50Ocj#ZIKESlEs>Z&8y)^n~Pf70BzUubgrhd`()u7Us zT+b`}s_UZdE9h@|zfISm{H2su+kSaR!hTuR<=R(QUr=6XUr7M>`R$YQ+gD2J1IVBenKzUqZNAA%V3o%+j*qaO-Ftrsqp{`S;#_IXTy^L_Qn z@#ETCFQt<9ia$OZSH0j9{f+ei)}OSGg5&o+E`vUrsaAaR-_=5Ju4DDT`+8f&!*8IWt_N)0q>6Y zuC!O|h0AGgPuabXfpBCWpnZfR`#{4#DfR)!8|??v|IpVDhL(Q=_+_^Ja5?;WlkA1A zi?!BZi zChP^fy*~@~g5BP??1fosLrD1_lm3LCRD0o#_!mCS;VtoAZ-#%N@+SKimL}%oCC+;N zWUX&Td!a&y(q-EV{lY!@5A4f{a0UKp`#+lOg-XI+aOf5P!ll{^B;m5{1)F}_3l~9u z#jzJIf&TN@3laU7V=vhB(_Xj;`dxeB66imVy%5p=qp=r$>ir86(DgH9FWBw-7)9+u=pN|`8-+;Y< z$*pGAUolBtqJoZ8b^$Yz!8hgR@FU*?Bzc7p9EAbf9?MEP9Uz?w@ zKX`X-fqB0N@e4dJwQx4$*=7F~@`iYe1yQ^O@K`UYczbg{_nWbQ!3O*+`6KN1{w&xR zc6T|s;umHm?1fHey@>N!Gx`^B ziS|M@%a#kE4 zkNKVP__PPaenULsH1Q1ydm`h;H{ksl*Pe*p161*$HLtCJ@=bgw;=2~kW_|j6!OZaw zfy9%=@egI?zGZv{_3fHM*^Pg|{Rt8yPGm^W$qW0mRFSuPnszr)+P39+yz*cuLMM{e`^K-u^ae z@8!ooWZJwa{sHY-2{?*>&~Oz0py4R~0qwQpAK2dV4DIdDN_*q@2mOAD>EA9DPb=|~ zY;PChN$vjaDoOn8<;OqZdAPcXZ$W!{07vl;8jj*0G#tf0puKi{B-`uH(B7_`v^S2A zw9fmnfAe^5*Th4zy?GoLwA-6k@v{;S3H(g)EJ^(5+VNh;l+|9sU$*3#@KaSC?}PRd z{>ok*|Cl&GV8uUPY&^s0{ttM6!2^tZ?*jDW{LZ3xLEepiz={H%k~w| zivZ^NpFcAB)%l${oPOo}DD;^>eog#-+R5hg|E18ohVk5}PtH@0K6tZt6=3;D58e;c zc=KM;9C@EZpWnfG-uV35)3`rvys?<{HD0Z^^L>WVLcgZB9=+#=`+T)NX8fklmjTuJ z7}+1rrz>c%yq|Njo$q7h58}Wi&NC_c3i>^4eST(hn91+ufyxx$|6YmmQr-(>SPJ!_ zybhel;(5>f863+6zdCUrzVP!od5<$d`I#4hm-jlp!u#LS>OPn$=u4dE8a;dyu>2JL z6SpHC`*SFPGKia>SKX?fuN9>aels4mgB9ra?{I%ya1O_#@~6prEzD1t@$g|u&S#2zpKhc|cu$GujPw`i#rtw{AKdVwj*^G__7v=U3eIRa z&~Q$}Sq&F7oY!zl!$l37@=6*u<+Ujo_m7SiPIx^3W$y30aC{-|+jHI%Y4_G2md2|7 zg1n6$=4~SU6u+Cxv(xwR{ysThtlwJ^{-0%b_CNZg9G`#A@jb86!$!JIf8snz)SrAm zQw4P6e3ZnmfL@Vr;Wx@-|I}GN`v>Ql*B=%jqJLTF%X=FVzs>hH@cthI)Kz~7e*v5R zF|g?$1MB`leD&zz9lC<^@x!ej{7~oTIG(jUPB97j0eVLd_vsANw11zxPa**o(*Dyu zxX-YWsUSyVsxIeK*#4@NpxaMlh9ApE|GDjN_b=FfwohJfWdRr0K6$Sr>My<@k&o$x zenb5-zSR3E3I5~!ZnHe;#euxP*h78yPD)s;KefK){EPBCYl{|%Kd2w=$M_rF-<9VM z9OxJQJrr(V1j z#q$Af)daD;zr_0+>Frj2c~67-uAtrgQ@Sd@}w}_Qof6mRX-t)tK3^IOjpF(_}i{$s!dqGkD@&-4*eh*sb=lw98 z9s+$oK_}j?Cmh#b*kb3$eOHShN%+Vg@-wrjE?q1z-;>UB{xSJ+A9Z#wVST=&J8Y5m zmv#MDgdKd(WFW@33EuD*?-Sy{6u-Lo5Ai-hnfE)z<v!a3&t z5aDNMt*rn{ILgoaEl7S=-%rvV7Nq_TO&{)i_7mlIay_Bn7lu5PB>kMIJ+oTv_K+UZ zKUT&i?$SSn4i0HU|53cZH+2xOJ}-y#+j(E7(TAP2(j1s)QF^>j(7u;&++OwGWcgJo z-@H#eE2h6Ck$$_Pr)Ko8yw5bI--E__tYoLh`^B?36&rd{f5FCs-VHXr=()a2x7m1-+ke=8;LjP9fl@QpH&{;vlsOq$#->_>CJmX6XPxM zd3Y&rPoDTiNUZ@~3?lza=1kON93ZUHk^0_ze+0zr)3E%hpl9rZ4CY_lUmU6Se;aT-adKm%Yx- zzd5k;`{Oq{=O_EZQ9hq+wJ+G{=3npI`MbmIQvdd-{t>zVIOdPf2bSn>5q;Yi!d@q= z$9s3URq}6*^0&0O_$`9p65;d0f0MtA^2S)+lqo+a`YXn_sQWJ?d{+1q1sSSr@O~$ByxtjEnb$KQTTCT>Q=*b=0r>w*%{=w2Qy~Iv4NzE`HNS z8$V>;3)c<1_?`Kk9zVmsRf85D?}2U0ZFBIuc2GV$wR{fMuFqcY;8zb4-x}dp zd8-`!jvXCeKJLRjIg&SDfj!~{v~xEHSb>n z{z2St)I+>||B{FM38MGJ*W|H;#>aSz=KJ_y56OM==Kc!OL;J|k6TLsV1V6(6#P^jM z-frxdh(1j(-p5|drJmZaKM#E!n!cNPK6Rl*-%8wP6=dxDPwuW|9eWVt#h3QVc<+I~ zU_hn|<1fH^!@2)t)vdh_J`?tXIuZ!3qQ5C0q}RyD-QYLvVSHkLuDW%;li#l`SMv>( z9`dicRZUl>e$XG#-vawY`iuIof}}EEL7rCKs-{bmJ`?ri>MMa> z(r4CFnQ)<#K7;oVokUw>%ZU7_oZ~4 zQTE-R^ZW|GU0@2<%lFr^xPVB%hYkM``YY@G+R$xd)w6P+)4kfh@-_gL`yeO1|3wN= z>%r~=vKaB?QAWrk^=Bjx&+z^9r2Lrm1g;4~e@*<#O^$!UAJ2L8$C>_jZDiq_el9*wUiHh-#s2KoYM3le_X!0{|ES7odTuwc`)(oAB_3K`!!fU6@r;R z9>_>~@drrxEh3LH-#PTC`-X%blOOG$_=CN1`8Z!4)1&Ugl={qi6X*LU{`^C6dYoro zcF#V!zu+d)i~9=t9%xbPS54oc!=-)0B=@z&C9e87w zNTR$4J3ctf6%5Pk1wWs#@}I%`qfg9>>nA@>Sa`t8p4dB&?AAxi({mTso+lJxDx(i4gLzT6`36H0p0bHXhzmM#y6}84sn~(@}cPZExHjYrIfEDF3*B!8y-JedqK)=(Xhy`0|%qmazVc)PE!o zxTQC)|I+x$_BQda5Byv6BjjH{_*b?0w~X}nS^Qfb)8E8Dv~LB|_pA1@{VfA=dbhnv zequalv3`~>)h7N~j0ah7kL`fJVw&`$e}=Zp2QM1rGdVy0k1)4a`#TkUcdaNF8ls94=aai@Ejir+>~F|?OnhDJHRiYHV^yBo zkl}oc``mC}T0VMD9_<-D@QC+Oi+;$b`0I^*!u=bWzvvI9zr1j@q(+s(569jGecfS3 z-Y>>|HezpMzk&V`wf+PD%h-RSH^Qt39s7;?HwUPe<{}nhcL!0;t{6v zhwJ8wKV0o+Mf?DNN4qbMADzMvzMm9yT$mr$eMyQRo$LXNA5iBuKgz-nTT%#HZb9DkkU2iCVa+UI6{Iuv>wN%&B| z>__1J2s8gdzNl{|Y5y`6=JPV%!{GYgyjKDJB+19LeeLKgjBo63q+@%xf&GpkZS8je z#(Nj0exyf!vFFVFnt-+auIy{TYCo0tYkm#mealS!nHK%mE#sZ~&hK*j6Yo`BQ2+Ve zXl^R9&%-qCS4o-m2=wcG>CgFbe~SJSf2H&%md(v2 ze>2Lr;>RN4hrA!h`_!%ZAjyx6v`5x+GT!|~34TCT+x%E8{8$|GL+&@fmhH9pu_?ii zjnnc&^dIhfHuJ&E`tN91i2XfUe$Rqnk>4|rmld!T<%2wboAaaM#}c7;o%k=P(4@V{ z4|%Damp{S}r4PmbtoXA8xovrY`DgQ|SNPK#@khzWdba;5&X-C)MyBP5=)X*F#E*;9 ze@^?mt@gv-FzsI^^c+O{GwQy5?cYQD`(|jr^lw%5C)exC677fiXt#g4w10V`{p%9# zAD*`TqVF=xqxR3N@2J0=_7|pa{|cc8kq^!7@4uM#ua4WlBGG=B!*=@zr2PYl_KW}8 z>i@xM+h3r*&kRKEZ$7V*)c47FvP*?8`2Nd2c`w27C$5bAHyB^1+I{l>7$*$Q9~gho z-{RjL+dU@sbJ9Ln@0qN}jD!EfU*X_S(%yYqO;6h1#d{_1Md|opmSVh4z}~DfFF(L1 zYsC*Jz~J#FaaH}I@GU=QZ~V2mk1ROuz#YLO7JM{pZ6Q5oKBYg8_H&%SEqKgO4na^Qu*lmjmco^s%x;DiJB22VTi(%@MKULJhafv*Uj zbKw5q>kfQP@PY%c4Ng1o)xjADzBc%l178<#e8>D+5`5Qz`+{>0ydrqjf{)63)Q!f_ zIn{rNCn#1d{&7E)3m|;rzdjlUu>bA!vpQTD_oZ9-`@=5r7aIDGhSk|N{%E)X&uPTv zP2fJeC02d+haK4}V3U7he4baa@JGXqut(zj_lK42UMKy0ug^+v1%)xJ6=;GI8ryTtCum|k?@65hn;lFXfD=QB-!Zt8Ij)q&{UbfS(&T_b#_91-$ z0mjGlc;7PIjQP_}KbWn!>1T1gsPuS`EF4|xrhj|3>ZWh&iqfZ%eiP;=yZo!Nd)@R^ zbg)(a`YYV@1KGNj{%APTZ{yLP4HYXt;Ek$3*gn#G4W4JR>tCLovdTmH8{T8(KN_xG z>!$C`zTl?cx+Y4`_Fa9on|@)I_PE9G>-k=h;$H^wyMf0E3?Jp)zd!8H9<$0n8unak zmk0W`td`_#PsY1HEN81x`8^)+n+GlY#CVqnf#rwtSE9adkMGGS82*!Rezp?jr@YL& z&aUq$?{jqcRm$$Q(xbl7w@Yz`o?^Cc;g5#w{+QoLzv(I`J>IKv(sTLYq+dT^rAK~t zI6hS$;A6%IbgTJV3n#p~t{3jTn< z&Gw-)pJF_^_QH0|m%)uvoSsje_afkLf> z_tmNWri*(2jPJ*F`gsrg8Hf*zF3M zHNj7|cxSh}_5|9Wu)pxh{4xRmShea^ZxNvwKm<59PfnO8h^^Scv5mp5y{}KEM z;`!x?4@pn{xb1%@7GSac1o@Kq0`s27MEG8(eNRpeL3%Q>}wD)9`zn@XnVdtuzqh~JM(H!{xqaa&=H@?-fuke@BG z-q;$wAGdX@_IF6U1LS!N$B*fc-X6p9~h6I z4-bZezUck9oO<6(@83^`*CyXD!vPwtFVuM6!rqMEk3)Kvr_Xat)^6BvE#dh6xEqKU z{xLt|9k&vW-j6$O@sskG*MqzgYzttwM>rhI|_f`Kf`_H8@(wkSUOQH_S5dJWv-R>MDN>WHYm6*@7v*hwhZ61 zga1+b_c-Vy4F8qL2k1l7*&k~E9OFUBE8mkteRtQc7CiPJ)P6|#Ho+r{$ouI2Uu)R| zK;mU^eh%eb{Wd^mKO?Ds9T@ya{JPVB_u;?mV6ulGUB@YG;zdZLt?yC)`_u-@W z)UKQt@eBM#ypKF^+4unSZ>PKlw3keI4UfU?q3MrZIyRlt-G||g3&(%az88QH}p?ZpC1jc!f-J5#u3PyiuW$+@q1sbP~!Qk z#@cq_FV2Ii{cov%CFHv`Bl4!{nLs>7B;O)m<%GWP$W|N{nCGZPz~;uI{q=Ga0_4+lFC}K~b6ZQvQ3e+=}7*@!l~MbW}g=QFWd|x9|RN zxjL_8V8nC5V2sj3-b$1YiHAY^Cqmp$UBwUl1V&(2RUm;!_kXvrQ$W}YkXJ_@k^WNs zC&}~|^rt``lJ#-DRslVRzxRi|Mc;y94`yO}u~Y2Bc)gM)+{5u}>`AN-i+2CP-p&&r zm%lLKe`p=x^MeMD@~-NO%EJ|;`@Lem+Es^94tXpP?(MG*0*1YJDh>J3@pig@Cc~1v z4`s?j`pWX^+f{nJXAF4Z{OW3*Kj*=oZKeF;kCYGnW8QO>{aNRHANuERt)Gzv=|$g^ zMBh*!zX_ti&ry3eD}7BE z`jGR9*0(Z0f_{mYSM=jN(f*Id_2E3EGrvK769jsBhWdalGTg{1QPcd-`N8 zAKMeB+EdB=us8G5&0l#ve_wEZLVKo(KaA%e&34dr_?LE0=k~N{{iC z`GN)ez6gSXDZqdvYrmTdnyt3KBMT?_Wdmp9dar&E4! zTtDc8{g8Ru$*6|x4Gewjk6+tVUe?G1?0tW{+JwK$!4EXy zxgwSJOMd-6Fvq*}H|TZni{t#z=UDHIM)L>aLor_{`%=~?+oSc+DCZN*uamWYXT9RA zch0c974iJ>vdkC#-V648@mPKf_+jGf@2iy)_WUfzo=?sn4*YE?zpU(SbUWgkNB19d z%6HO3ev|gvrNT4WcN2(rv+d1^u;9S=)s{Q!hm*Cgt-=n@cc3r1UO8FozQM+0zh}9v zzbC`3%S%XLj%nLho|qO5r6&3+5%L}&(v?PpE&<_=1_XHe?omneQN5X z|In`A@qa=7j)omh{^s>c*70{a{RMf$ehr3q?5{bB^NaR+=KfkKUJpPXT>q)q%kmzg zvX2FiiwgMBADiZUSE>L~?VL`uvcN9X9udwX`jn_82hU`kQ^V%!~2aVYb|Q|L^v5=iSm)ZRIJX| z_CDy>?|U2m-d!sN_Iefa*P+S>o6&#BU+k#nt^9d9Fr)r)-baSMp}jA7-d_Y5`SV@X zttvnEUtpgr|B|d1(imS|CGoF`|A79U?pWU_eWl)Gi_QZ*kl!Q(PH*2{s{Tl(pGluM z;{p9e|M7+OOJB+LPqdb7{|DZaFVUb@{*h_*ff^oKelQ-K^*Z!VpR0d1_Sxef^3)pR zIbN*u^D>@iCG=CK)s|=MzdGl=`CfHYzUY&(lOOXHs&n*1VCx6xyc*;K{)XT>BTt}* zR4;|Pq*7Aq^NRJ0A`KkRJ=sRx6NB+Z;wOOKGjlM|o59_(jl3nu` z@p^JT-kh(;_*K)Rj5n@_@Sc}^9M9W9e~`8H0mdhrgO7NQ=WL(NPqt4DFw&di_`N<# z4}QYGiT>q>{8%4j`~>A{pB_JH;5)93@G|~8MZVLvd4^vlaDnZKME{}SJ4#=kS( zmA-+0A#0C!+3zyxAx~YAep2m={R`+1=uh##%6=63ljFamx-lC6IDaJmT+l4*73d#* zzFonNemIT&ROl0tUwt2%qc2eY1@*;o+8+<(M}$Gt7tQ-o$?<=w@CEe+_R}s+U+_Kh z-Z!c*)Ns)9my~zti{4ltP(D}pM&nuf^QT!~j8GrEQGL-JkEfNiKOB7veX%~qQ~sLt z#fAtk;~DSc$#-6Tu`$MrzUZAnU#RIX(nsbvgsm@nZGBOV%2)bA4F@eBKbiW%_#;#B zFaF!>3;G|`dg!M?UsT?xzM#0_Q}b0)-k~oluD%$oyrnN3`$C7)kbl$ai$R;e)98y; z5nkyFoL}H~UVSkf<3(RoX3!VwqV&qXkZ=LlzNpyxVjxb>^+o5E5&RQtUzq)mACJDE zegD&=FXmrRU(g>R4*zz98KA#U))pyqTK||Z8NM#<|o%9ZIS;%tw-kffB_QE zkn~^Z^$7h#KQ4W7VSfeJ9}C-Te+Aah7+?G-uk?qO+7kYYHrs!ZT<=VVS1q*l0pjlg z+WyN-Db^<|-SLh3Cc{zXuj78NjQ37wygTDt(q~GMe2acb`Xf3L`SZa=`zKI8#&>Jv zpExb!KlW#SQ}+Km)B;TTBQV}dar=8P9!jx30lk=CXGQ!yP5XiIA~Aok-l#-)q=$b) z;!$!wexPg<5nc-0;&UY!0-*Pj9U7smZVdJ9YJ{P0&0jSuzX zd=vc@9aTNPB_5%?FzOG)2jG1k4i}YP^h+y;i-OTU=-Z_KqNC{eBV2s}`eJ`Xq%X>r ze#s~O7yZvh@dlKyAim#S`7h#l2c_@wrRr7MKFZL4X~h=^Z22>(Pv#~38`gcuXdmJm zM%iHTk5Ha#^XAill=ukvTSxbgd;cT>bL#$Q@R!WN$NnY%vvK^v-+q|o#qoS7AO0C` zUn&2r>_4@~@pZ^Qy8m(Szgp#YAJEegd$o755C6>^I;>6n#ij#+_pHnaI8Oxrj_&`K zw+wg_FM#_-=SaLF&$FIyPBu})xr=rH1=_98x+LUc3no?pj5<-WZWD{-Gn)}sKL`*jZ; z_OfCicjLZ`Q%s*uGpn2@75&_ z^Z>?wgWNAw((z%P_onUk;5^>kIWk_<`CiYj>3CW)1^K~Wp7&Pq(@)_%{vC*(sP9tk zJ9VhZU!>1xAv1Uce<1Gn&Z{0%@fz6wQT6Hgn1{n)&W(Ha>@oK@ovgJ|J>ygQd*b%R z4ezAp_r5lsr^?ysc^&}@lArkJa6i%9g#grj1wCH!z9z;~w@7l89`SN&K4&~C+S^(4 zW(&X1;C@)N7x93+KS{-_0*`nR*&vDO?TF8pL0)nGD=YbRJPG7o-#@P6BO&i{zohBU zleIRwWAUl>BE2Xm8E@o2^0UJcU#a6I0*Plg`F)&^5dP}@ZH@=t=Oyv6$nVz{sQ5ni z=T~@sbbfaQ|D3Jmb@C|a0W9SKKRJQ&4J_@=W?obBE&KsJ89IFBJh!wr7Z5M^-z$3M zLI@d8!1KJM%s-kw#0$-}_4C4z(vJIP&b}=23;J8p+p2uhk9b?rC{h00T=utVKhDGA z5CeW-drq}?nDX*x&q{gN-L(Jy#!LfU^{ zm{b1ammx2Z50vLsP`RYB^=Y|`-ElugTJ{N&QAlz7f?UuYD;_t*8|?qyezCN zA%7N2|3F`ai{(7Kz|U;w`N#$2hmHphu06n__+k8GKI3`Hy##rl68vL&a#;EU`gK&I zFPVVp8|gy_x+Y}qvFv;KX!*pCH}l`HhLZQL^Ct__z;}ne5`Qk|r;#7ySw6&@L*FyMI{$(3tM4%BpJ&qJWdir-DSjgf+dF~sCB4W8?SX5k z&&PIrkNgV=EBTQ7_5>FB*rD_n##@8>*y!)RuwU*ITdw2{{9m!q?w`KU8=(ErN4&`Q z^5w?>59sux2X+U_!KLO;XT5^-$@4`H9`WuNe=9`as(AQdb3$K~$Aduh>8nORu2@kA zf%92E#t+KZ-Kp%k&hZs1B6|q&l;4KEr4nG<5c1&E4}Q9EkM}zV8do-O{$R0(SGKU8 z&6o3LKR+P<^W}orr(mImi)MWrke>YN4p}b&pON&dJ6pAbP{U(Jnf9HYh z3a_ry_qPGgAU{NwgZ!g}1K>6MP@A`;mX-TtNP%~v2XH2<#wX#GfYtdu!DGL8^xc>s z<*SQ5H*q_<<1sYY*uP=WzZZBT-wPjM#gPBAV=3|<^K-!Q)6Mo{Jac^0VbtLJ1Av!d z|DEz1bSQYj&u@XB2W|?MD|+O;2X}nh@`nDyd^ZAlRiEPj=;05bf&3JltRMWEt@x$n z8Sih%dp?b+yK=mrcl5vqoccS*i~XWc`j!4Ud=T=}!z1<3HzL1^y++? zq;K~s@DI?wlJvk=YAu0UFH~hco5y<{jRxxD2l>$X>#Ba}rjS}5>!8L!LRyrrn034{iezvR{yWUqsMd{6gCdMS@y$G$eun~%%Kcn1CX zh<=Q(JjS~sww~d5tJGR~{;Def2VAJNHpG1M>M|Y=A1t+k|6o4;kMaw(g2c~&2H<-< z;0saz2tJ1wjrc|QoYzywPbPnmOW*^qr)uH7Ac05!R}}tc^i3@=_7d#1m0oiGsvzFy zvyTqIWJQy}uZhp%zF+gc3glwx9$Zj`WNM+ypq@R0)9bxppSTt zSGg}1`41mNxMk0j!b5*9d!$Y9^1j?xRsvrHzM}Bp-{FJkzMi^*&j8-vq8{X6`TZ;} z5By$*7Z~EuQ&n&;^*{az6{6~={A7VYrt!Q_821YTQtjD}^jL03>A4<&I%9%!(68uU z&L=F;+<%Mv0&{WtBTUcIybRN0e3io%xt~$WKi!@xbNrh61YeW~4T@3u9G|*;)Q9`( zWx~j5{Ln+Id7ULbLytZmP53F^mqm9$!z(bo^cUS-4KHixsSOrc0PbHQefe^2RW^bl z519Yj27#CNmQLg8IIJg@0haf}K+jOPY?;!?s{WyHq1cl`5AQoGob}Gqpl#eC^Ci{~ z3n)Kg-$Or;KV5{UkRS2_dkklVcPM-8#RBxb4=g{W-!VTe;4ks}RCu3^0!2LR%hAJw zo`5?2=>AGU*%QiM?Htc8pgk<-kwO2eTa|fd);nBpQ~%5Q8~nn0FN^oa&_&X|0QuGZ z>zp4zFV=5q>I?CYJ<~{W{Xl-mePtpad3m2k_?HuJMo^FcWj5UPcieu-o<_7I~BZ7){_`5kT}NZ^aMd`R??} zd>?wRg8bs&LjMf=TK`jDBYkJ7loxx0{s_Qx^R11J-8AO#eG}`2w1)|BP|JT`g2jw~RP+s%-d<9jv&^`ve4n0C|TTHK_ z4*_1QE&6c(#3K%U;?FxD|0;g5XwirI1Bbr8e4#*kg~_ZwtA`eLfd-lhMH{L<@r+()@U-rE&=sQ>W(+qbOxM$CQr zS|2*&o9{6mI3ed%Fbby~-^uX{e|N$^F5%Y^zmn_Iq<@?1%|YpK@Eh`lc!t=Y4Sjk% zEXevy{Mmr-g}<5hit%6IJ`nM@82>QwGCs>=a(1oh^ z&+!F&sSa4iAJzl(r%@Gj(cg5s-R~UF@GfC>zW*uuOG+xfU+wpByvzMGvYz}3^#R-^ zgV0xO#F>?VX&!5R(a-e>?5DbhVL#n@pau9z~#Lc!MC)~b;mKv^$zW$3jS&Q%}cQ!V)x6M0q1{5 z53d2l5B&QU<{!QXYW7!PFZwlQU*bHc=*wkcQO29R-+A)4nteP1^C4SvacR`Ghd z1cOgLy*X`X-q-8>uHrD$bG?6@?^|Z${yVvy^=01%v|8^|p1yj) z^|SadK99DMVQ6n;?^7s5Ub?{D*#1NLYee3;{s+J3;=TpW7xKYg!+N_*t?%#);ADC5 zZ^-%V$iIR0IP(tA(f*CYFA&71{osKI12*>mrj!0(u4mEzSSV8890eY*=x6i?_Ge(2>HQh#w`Jjc zsV^`7sO_LPm@o46b3Niu8m_fs{zrXcFRTN7IbL|d{Xe~)$hJo7N6gPyPvl;w{ug_Z z?M;h6+vk1;_yPMlO9dqUap)KFi}Z}`zBs6q(}X- zpRLPN{<1pxq5MTYx>3}7v|ow+Iw%mKANiLd{(*2ndeOgZkMKvgr?cj(^K~8YN1Q$q zl(GJ``>V5-7Jp89HYz~=yShD}{XN``@d*EL(%h|I+=W z=UeCtHNV>Pu{Hn7d<^L^^Dq1#OKa`oZxfk5MSWD5&GsAn!I_Vr;(VOXQy)+~xjuB} zW6-~}rsiXgaS#5KPUut7AM|&ly_k=^cs|~a_46B?JP*+=7n+Zus@?fm$G3z3|0eUXvbWAZAD04U&z)~Ro|~MH z=O*T3|khrO%c1%AtX zj{U=={{i{eP}j@)1pPO9U~@ow3FWuYpP5XbT#u{uHq$Hn?q1Bdk$>QscJFPX_=(2{ z{w3GLHz?x^NHw4S@d3^b`0l|U_`j0%vwc$yYm3)l;{a_+Q|^jr;X# z>b}0lJIF7{kJ?nQK98vT>;>P6b4gnjTwVD5(>VCV5u?e}4S)_)VSNP`6Y z-FV<}y?&m;`uW5DDEznTi_<^ur2m4GJ}G}UIPoGq-c5iNKL*8L`ewtpFRbWQckBDY z79*bYT%P_Ob)PWXH|B-108{%NNWa*t>-3N(H~l*B@0d4krRRNUD-?tBysudD-(%`y z{_k4#aee&-CqL3d-P`GH`k4MHC;bHOKcc@0pQ$hFO`7}|A5MLZ)aZdPdB+`k^U3^q z?*Y@^ru=E_w|mbz`GaKsz*_@-!uu+YC);}v^!ItEXUbpI`TLUjNq;{KQ0Xu6=P+J3 z(f+u8=`Wk#D^dRrj}P(~f9|yT&s#tFzXl3F&X4}edb|)Ix}PK2Uyplh-Siz7PT%is zbklc5>8F*y(8C40_#%0Y%Rf*0-ER6B%fHD@KV$hfyXj{v|6^`?s5`s=ll=UWcV{xa z^B%FCp8frtlYZL%Hurxkc`L;Ffc*_OpFxTMey{h4n?AmORi#gq?|F~8 z=@a)E+T}lq@=qkw_j}HLR(5*Qv)TKCTYlnxs$}}z$@Gc)XYBOrQ2tHc6K?r&`r}D` zc~H`S=@QdVU1IvDE;0QHD}8c*H;zv=`?24({K4e!xc5~nKlej-h&smni22_~JUY>4 zKHYqaYzHF+=hJpSIX^CN=ErtF+TYUi^XP$Jdjs|L%urv?H1*xGEYgLnpj0U+m$_ zF5ilO)8&hQ-zpz9MfUxr@@MqGRgONGMqei4SCaa2tM_$FpIlI1e!EFNX3$^L>C255 zU*9u`2XyrF^XUug=jYSS`nlr0V71SYPxJ@dyH`5E%n#Q10^ZSoP2=Op4CCX;ruZPP z?@qsI^vpb7oB8uq?~Fyy1^d6L{U4s8|3BQM@3}tvmQ`PJeaZbH`Sg6?#FJq>?l|y5 zutU}>e=O^bOV9tTlb`#|bSMXEe*^N6^2WHp*XOIcalVpx+26!`Q^xr(x?{|Kpc8Kh zeT?`PIlpK2&xR0R$mR+RJoXDSuOUC~qc`V|asun~sQ!e6D@OD4=;51@njhp7`@NXo z;O?j&^uGKZv1b5-VT7@NO?kzB6#XI5{0{va=`Ym3Fg!x6H`a4j&O413b}PKLXW$PE zK<>fx3HSr)k2$>2!J|G**QoEWr9RHLFenH|^l1ElSMh@nUaj-vuWyRK5$~-czEW-E{Y$f74Os7AV}6JK&-Y#a z`uDVWv}X;!mfpkgmHLO-$K>bI2N=GgVfOC>EiF}{82%=Vht814@9R^IC)EBA$Ilo$ zK+De%;!zJ-{XbMw@w#%J4Dy8iG?;7khTdNdI*|S#vS-gUWIri?OQ2#yI`CsJ&v)b) z&t3OYm_SHR6d$hYUwSLugY|VXCn6qnsotLw9)aIWYi%XjuVNbn4)ESK9|wtG@h|c> zg-?q^Eym|$KUT`4|4)yfT(p1P5|0nyhicgt<)H$7gb}X~cBt_H7~`SkbyOwsry_54 z){Ia0C-&C`zaHa7o zsrH5FzidTP2!Dg6wUWZu6@IDSpAM=5%l;T;NBc_yiBHA@Oi~}@3GyiX8QTSU1PuPr z^;$Rjg6RoL{4nsaZ%ZBwi~lT7`lEg@PA`%brAK^*#5c+K0=>!kKbp^UyrEyqdXYT{{)`@|uyTB+ ze%X$yr#I{Nggnk|*YEmYq3=nMN-yQfNQlbIA0RiT8XmJ@e2(J0ocdIKIFBXqCbFLZ zebG$6=C-DP)vXfF9@)1$4)hhbM)Mulx39+Q-Kj)*KL%Jg_N~9(;=o%Rc#8vXao{Zu zyv2dHIPew+-r~T&3l4C%1Qz<#-wfa%2O5h5@8w^AaO3Cy=#~Gede>uvrzhv!wea&_ zp4@O_SM5Y`L)*{&{E1Khmt5xm%YNo-@B3ee?)mg9&p+8{{K-fEkNhXso$@~Y2fy{N zBh~L8{F2(wgcIZspU29S-%GFj$1nWdXFvGE-zCt_1D?PA@h>)>{L?=^^677^|Hd89 z%=yj#Va>{$en0%=UGv(X{fE>q|KR_6t?~RXK8b=3evw&fum5vV76}B@_j|8D&tH1r z{XF>k`_++AgKISYw(;Kyq`cRkfAzKJe(^v5_8vmcV1uj_urY-XkT2< z9r|x*>b_TQ`}S8~ZoKmB`~OpV*K5CIXpf81|N24W!@uzQ*pn|j`4@lskNT%qo@~7I z@N3`yXMiv94wNVVfQZ-r@D;d!{vQAOMq~KrzuCAQMgBf$oL7Gdk@PSB)++~Jd-C8n z&OO{nbsQR)^QnP7dmer?{`v>-gM6Fz?Cjh*XU?9^-8=W}-no0vo;eSHYWAKz8#is- zy>U0bJ^1N=KN?KE`qIaL=fAwzc;)@!KOXzqtG{&n&@cYQ!GC`L?zuNsUw*Ro{g*)F z>y6v~@Zf)~eD_atXTOKTi{E_ZrFH0RA4|Gahgb1-&%xHR=U4Y6|IVmD=X)L+n7HRy zZ}HgfjQ8-Ty#75wH9(Gho&WLnFjd{#G40>jo1Amsr@ZSnc@Ov2cV&W(6cYOt{-?Sp zRrgr?w0|QcWz%tF>8ySxe{6RuwGmaIKK^IN9^_)h)Bf#wdpemV*fHSu@2Mh8qJsaS z$_unp+U-^M%$|%$?YxvG?%5f)kr`iqz421_vD+8?{tYj7HX1*GBsBiy!_~7N+H>3Y z8;$qx`R_0N_HRCUaCr(3kL{VgY3Ct4$nnVTJdJ5h_CwYKYyq3{nx(1-hT4+-$6K@|Iom( zN4;ZcIlfe8cdGt;4L82N)gJ2q|#=48gx$=g5P_?yO)U;LL}-Te2B?JvFh`oDbQ zw%dL$MLkyqJK1FZADqMgt)HL$r8GxH9S#o{ah~*#|39xh50bzAZIH+TM6qW6S6=)3 zvuDr#;QJpx_?_3k^W;n4|N58z$ zNZVCNKGM=xw9=&BgES*s{bKTuUn;i!P!6M|{nWXMlxGf;=_+IV9Vr`M*Z99mr55Zh z7QFY75@>{}hW!geFV!#aEH0HaaMq=!8Y69si~Qp+*oWIGM%WDeihX$y5Pb+LYr^ws zLH#i(Z)eJ1@S*GiDWRS#dcKw2lQ2BuXPpPnFgxN~&Vy%I9P#P%;2HLY*1iqSM?)0; z9=X{orba)eds`t!J`xlmF(+`Wna<`{Aa<0fDGdwNI?WUIZ#oY*sF5YgLs;@$0^Y9k zy!riYcdkC?-Mkk9=KWCvUc+h%x7`51M+f4xc)_Y$9Y#h!QDpl^HhODQTT0+TRbxT$ zM{Wj8Z>0K?JW7pzaIP95s&Oj4_(~;1JRQ;9wT0dw8&rDzx;<_he%ken)2|-WGsuS zCUh;T!TxTqmdB_+uRxs&i*5Rq)Y%)TqBE zp2u3}lKGXNmS1tZ($p7myVffbrsHQlHHC&HsYt~vAEs|UGH@K^!<9w8}!VN zm7g7##?L!{O#FQ6r-h$gm&VVp{FwMDw@<5CqiMYVM9F96()gKIBkSLMyl>X^{=|>> z{!8QMbw4J4e(I-%pM#gi&wGDN{N&vS_@AAT9(Tr4x8(uV+Gzl^j6MZdJG?VPL>+?9Z*XZGZbe7W^ZW3{MOkS?zL&dD04 zsU<|ZEvYgG@;0@kw=H|_JZr1RxwhKpX{%A?9OLod1r%&TsJRKXUbI|08o} zgCB4UO>jMz_v)=vt#Zm!@g>Ilxrr=&L#+8DeCzM1zE89>QjRY0DXu2;Jt-+c&&M#)_qY; zPc>%NHc8K&AI~PG|hbJ<{J1j97oxf8;}1 z!u;F(N#f1#Chhs~rko4m4bGHh$~Jk;?{+%#o1Hd^pAMHSbGkIXl$FVMF+SSRp>o{u|&?X?wnUo zVXm5j2__$XsZHf4=3LQrAK3+3e*v^Y?dmU*iq>dUHhT+?=~2lM zIfCo@{WWdy3RXW&S|`$&r6<~9jm2NHPKd|+^rgSyQd3DyLYhZfDn+)PbZdL$BcKu( zu~%q++-%@f3%##&tfx~;z%#L7yD<(+I|9GsR#@^&fuq~aIQ#+r+R$zIZ`0m{Xc%ak zSi>ADOnG&o#b}?){X6Dl^FD`Zvu^9hK$_jw|JlHAc3ZvTl+xmtpgE`GN3%AizufYB zi{ze9V@@~P&2Q#!qXF!{i4*Jh5;ek%BMjQ0a2{=-c2=CVjiU{?bCN8dG$$=l%bzbJ zpJ~=lqaor|*d3|l$aw@Z!#tLx1=FSBn3X?RS&o!8ie^ZTW4YT6+CjvSLj~V?(o~}~ zdp*zAFuII-grCBQOK7}gnG;;rO|&DW{xG7aDYF`Bt$owtBW>(`nRQ;`eoVAiFp}a9 zV^d!xRvnhs1~Eu~W!~&?o;w7;L+VPiBMOTpRZWe2BwytI5<&;|k~gydteJ2 z)P$c>Jn6z;NtudiP`j+up5z;Ll6fhT=i_k)hp#Ln+L=9+<81R+vcs9uJDgL|4yXPF z^Ws#4`;@r5Z`ys#v)j}@ri*9o>f>b)e}#R_WC;e3vl^0u`>f}hsL;PeRGd2x6_29~ z*5TsgI`AaUH26rKyl1Dy$N71sKy&$V{CQt3_|TApKqU-*FCuV#8|mlAFnUD&^|G*2LVkNu}^I2 zI~ilvzVW9}p}mJoSg5e}jicW%V{)(gSCGcuHzq83E&*@XdEWff?h8+$HhgB>o^M~7 z<5uRugnprCgZs#q-p*I4NUZOpP!zO=;z)U`Uz3prvuP0dX?t8ZVaWjS#P>t zPRs8nyV)`S(oKCr<-K!Suc!E`KAeXZ>jND~LU&^XV-9fOL#ZjIQvO>Ik9~CPoNG!5 zt9^89Uw*oMWKfdW1Gi{zOUlOMpxyTNinZP3T}0LJ!l8corsZF9E{s1@LNPI~b`-2YTMdmw)P zcX0kcaq%B*u=x9({o(FS-0d1gcy%0O7F~slxn0F2jupDiuX^htIY!+tszPN@PKZ>*W(rZBLoHWx7XKdG^IvbdQ;bDuoi8>dl zY)zFcIQ5lRJ{r3Yvo~rESf5x}*|NRtY`BAvT;VVyV*P*{<$2zyo*r+E)yo6!PQ0*Q zx)s~vDn%+AuZWh61xvYBf@@y=k8GbPJL9VA58lhy+IbhvJ3PYq<>8On-AR+-o>owk ztqCS-t{u1*<-UW8dw2G!ROfU&8-gy9;nff4BO)L-=rYdI$e-)qe;7TDL2( zgTGtV-^stWf`7PcP>eqlCO@dDlz&g@|Eo1Mhub@B+-q>1sohj(3okVUT#R=Y?x%{+ zuWU1Ij}>QZmHHz=W4>Zu?IKN2WzQ4D&+|R4`?2zG57_&(>8*y}q5t?%FJHIw^0;X% z_8F&Y7Hgiq&p7qUW%s81nWp^bT=)aqE?uKE!bnhmP_miQ0bniD?Xg}S%Ne{!&{Vd{OoR90? z3pPBi&Ch^;+`g?+zf>tch99#V`jTr`N%wX0kFPD+t{T>^!rw!Rtz9MgRfO+MCH=E- z_raa$Gr6W0+fy+k^U?l>+duMm2fH<@_$R#IE3U=vVN;msp0MPd#4Zkn)pvLwT)FeO zINdq6i(7uOL%ctS>?jvr=aipVeM^sjr*Nlk>0N`ob1VLQ2k*;`?p-N9|3QDZYgK7? zEaq>|6_oDZxl#^aY7wqiy0zoq_-I|Y18VD_!9Xfp#Z5fvRC(ttR#RSdyKk`_w3plt zTFm{Ta~JQ}c=siH_TeS%aOJ&C;Z9n(YwUJ}?c<+uDel14*4Q$xr=u^o1Ly7pv|eMu z_E`RfsbR~Cx!oOKdREB!hqE5<|KE#uwaYYje{MmGxfO@}!#KjbKuhjC$S3ZiY`1%= zd_4cMa4t(1TU~PZFnH1JTIx>l?RF1etSL<3;(H^udw8w5dsec0==NTA^most><~VD zZe$04x4XDwc-uYP$=~fR?&KdnGqltB-7`f+|BU6j9p!cp6ZuzW_waH2le>p`w|iLJ zopa|;?B2oOJa_aj**!$~lHJ3|zp#5%vU?c%*A)Cqb`K-}I_=(->>ft(Z@+sOg)jKq zKvph&+wUGm;S2uQJ#_oSb;K*rS%*E7DN} z=P%|jcMqfV75qzf52Ns9b`L+(kalk%N^PbN1PAQ?TPuiW{LU9SIHliisyLeb2kE=3@J;oZQ`lEECpA`kUL0)c>REXBq!$nI-q0pYL9E z)IXqI#qVXZd78~ebUTupjaZrOw=!}wx7Apk+w1+I4M^ckH(W0((?$~I^{o9@lJljP zvo8k|FJF5pyhyb?v|}IC0C!dGOjQo<^%CE2Eo!gl=Lgz?Dz!+ou$pXmKWMdug)R1Z z-Lq9~Vb6A6QEg!wV~RC4xf1)dE47NhI)+7aj;GB}&YG%Y*kjzigE6Z*D|EA9^?}76 z;Mw-E`)$l}*I8?pE8*Hx=^f?ab;I7ivs)#qJ~z>4#P(U5w9k^UQ zPe|p{rF$F<;htF9h7b2V9)l0}Kpuk+_e987a<5cuUzK8@f4O{i<$q|K=XA=|*C^i` zw#j$*WEjut@_ifL-qG?{eA+!Z_89RWbWf6zujGFEd@q;JJZrgj5#?=NJa3CnwT1gu zjK$^cHhlOL++*uAWx_U9Zf1tNXAPu3!gYKRyA&8BU^yxAgAy8BfCA>JCC!NB8#I87n{Wl4wWO9fYt< zvv_0X(>|5-fnGaeee+QEP%v?*x`PnypcJoVpY3YFzPdKt2P4l1+JZO}p&Ko%Mmq?) zzai{X_Ii!r++?rMMfQ4rCgw`j7M6UPoh_)6yV}Amhg=`%?lphE-uqo`Vb69>P#wcG z#uR5=)g6R=^Y*mr4nn0-s=9-~XCMkI?bRKG=+|MYI|#8~*-9h+~*g>d1gYf*HjsHNe9kKdSm#qsX>Z&^k)g6RrXtw`t z-m=%%i$Bm7#CaXvXkj(lK?s-n!dwn>xcUr2_^u6?A(kIcyDM8=->j~0y5AJ{N413o z*=Ky;L$!rfTUfP)x&EcNJGy5?J4)wwV~Ch>JGwF zymJLDtVTNs;le{W-3fEJx`R;NL3r=?R3G-*J8M4BF{;v-R&8OHL#~9X&mj2A$1*jh zI)+upuv9|I|z&;_SW+kf&;+|^e=Y>zneA0>YI@4 z7ua|4es$*e^USwxb``2`Lh@sJxb~80QPo|D(mW~mTmoxx^({uON- zFu!ya_`L3!NVM+iGYDax@AtPDCE8JS2O%ue>@&WbPwjFq>>#AG>0lyV-9f1CAjIcw z)zz2k>PvO?<+-iC*j%>S!m2H-+QO)jc}P@#-F3a>pTYy@E zif=>$zh`$6LOZY37V@pqZz_PV_(mk~6@OC!{GRRY8>R0d!17nQ+R{6%H(&+`|R_xc^3JHDN$rlF>BGhD4n z)ulEA`!jNdlAE-A>g9u*+x%91XRW;fs{ZwZ?o-qG^3ly;;KMFlO|3n@eeFYw?qaFQiR=cCWz0!F{f7`Rz(cjG7(Z9Ze|I0y8I)iaa`cRac;XU5*yi8@SUx^>e z%eWoa+)HJ5$j1_YKIQzSiz(+$eI`h6+J#tZ^9MHjSgeh?Q~6ZfeJT|s+&>8NLEPUiXzLU1nxHZ6 zekYZGCGJk8f)B^t?*#b|E!M_u-S+R7S`NAUN!-ABO3Bx8{>BPwMN_ zuQzlYe7zytSj?{w7g!8M{jI_r++0{^)M<_es)98@7p~#IB!=%Drhfc0K7iJ{hI`S=B9< zhCjC!t`ezRwbnfum{cWF``a>`OFkE)1 zJ($|OmI_{qPMPs%78~QzaV{IumxtSYQs>&be;9l>yeBt%aPhs;)?Tt=Sr@of&#cd^ zpuOCx=eZ&44PFRqWLPU*t9GlXt~I-lYw6*N-|^B_)u!OXF8*+{!~!q=N|{wv#vlD= z@Lr#PB*ypMKA~8SE2ou=eo>wz&Pj&ua`Bea606{K`9oG-i{VnRm6v~#VaM~jo{Z;y z)^1nN7-w1SeXym(jxy&gmnY?BI>jER)_N~%xPAomjlcB%kHU!Qde|Z#?fZnQ^NF%5 zcTU=0#=Wm*S!VxU?^1uWW8qy1ctq zP2IH@eSFtm+9tjP|H3shGd%yb?arT= zN7OF9SIM5o^)KFi_k#nA2eK&(zt?-59q%lO=a=n!zgIHL3i64)<&9m;f{OL_T{ka0 zq>;hqYvED!uXXdwg1?;!{^9(y=pT6pZU$G3BY6(ZQIkyaJQ|^rO!8cTha8jSl5wf6 zCz)hi*UH+RjAx=OlZ+>PdysL-b0y=^BNj;}8IN6IhDVY~#xqftNyZZm3>EB&?fneXj_u~%4Bjwi;1{YgzK`OCIR z_nv;<_;LGF@~;1(mYC(cdp+!`Otbto>yN@b+uLvFOU!ITdy{3l*WMdSFSCgKEVHmp0 zUBi+&_2r$VGsHWDwXq}c)2)sJBHETa(#EXKE9k*?A>bJ?pk4$ zE#taJJ9w^gagd?ATw2LIuDe{CE5-37wkLYV)U`rS{iNehF~8kzrmbmI+Bb-=Sf(F$ zv?|8G-DtIo(Wx*(rnVUkifMAAQ}LX;9i1wbTcQkNqveOUv8Pgel}4}RXc?4@mgV}V z>gX2s6?%l9vj%U$Zmy4r1y7>juGNUTXVrFR4Q&2<{cePWz17jp6)bCiC3m@zsZtpF zmy4r1x)mB)#WMMTN4J{l=vFMN;)wnJUb}5~2MgO8-PO@88qO?Vs-s)tS%=b^lDzww z{*~P2uJX%iyIFHk0SFWdz^*B$w;#h7aubjrT()}#* zk8zeuGa0&{#P-B>Lvdx&ZW4^%=fiVUau>_xsjl5>UfAaRvG^0Kgg=5SliS|US*Z+m zKf1khaPdHOWzt{Txf$nQvTL_WHL2Wa!7$~Ra@ynW@jZ)P^-1C=J1UhK-O6#yn{vH) zxn6p=8sjhahT(k@75Cj3{z>wOaXhZOTw2TJ)8odQ9Lb7zLu@yaJa+trXK&mdXK{aO zyL(a+_QdwwD(`=}J4A}{Z#S~h{`mP~PbJu9WBHLhR(BHTdrtdp_6Ei1?v(AGQizXQ z?k;+!FYaINZV~d8-0}Unyp4m#@Aan;k}G$Gx6z$C;NC`8eEZnDC2udxSnhMBFI9Yk z+m<);@pm@9kkzgGerTL^;frsw3*SuV-T>g9`z-ff&BAM?EPTx0-W2F6V^kkC>iV%_ z!QPMXZCgeTK zPBmOUbm}J??ERB{>6(Vz%bUSr3u!OFPWf9ITtri^ecOFSZ@^5R-`S$0B%kByHTzvY zyEI4dFMGE(i`pH&k@S^^pFZ`<)o63unR4dZe)_T{uAwFKt;KkLxb`{=?@wf2m~QUlNP<`uMfuGy7JCdwu?} z*Yh{g#!9cq_dyOPQovU2uiO$$IW0`*K6-L`1?vgPYu3PDvvPjo#&50|xfP&Ft2^%3 zy4)wW=1{}yZ@0Q+xwv5K1^>{l@WP8czlI=9>o0Ci;Xf5)VdRe1TM8>w@9wz9aIM@L z!wzdAJ!YR;Bf7M4f+`n*T+UtVLNk9FG;bvMX+nStkEj`5h}0r^Uor!rxl*A&ah*4-?d%aoC;6TuE;;`~$IzvjonwX?+^E0ma(b+{&2Tc<;3hbpzb)Jq5Lw>exLJC$lP+c`()d5XW+@lKf5)aL?QHw^3OW-7H9q&`>Y<4bM6 zW54Kf>RjESny*b*`uhi5J+!sS_w0TmTc?Q<*l1s6{qvE3lYP;K2 z*7~l*`wd=j{(3iIzJmYt5`Rmay$8dkFg!~9^NIKNOqTdtjj-^t?`Elg;_Zr^*0(0@ zx1zN0)>gtJTfzVF!zb-;*Sgt0dT$2uD^XsB^z5V2H=Hf(?fl8Q_% zXVt6q`MIn%RO|DzU1!~?K3Dr^HyMalx1QI#7pnF7{j1N_{<&J86aC#jpKnzA=W2ca zQL4|?{<&J86ZJWHeU)7QaBJH;`MdkI68>d=9V6i%l>WNKz8)XS|N299zWHqDoBJw1 zll48>o;LRXdSdlG$kp*9F@Egp{=53VXSa`BU4Kd5yS>-%b4j$n@Lr!nA~YC zj(tMxxAD&GMSl@{H$wF}>+1Ru>c>8;-}|aR-f#I^xpxm!RJ|iRJ_G(s^<;bH8uE-r z_X_h6tB08&mHG)yOw4W0#2Yi#P56zN|E2?C?hiD4_%xS2(G|VDCu3%rX*NQ8w>@xE z#%mzM?k~XoSIr&1&Bu5*OZQIZ{_Lg@^<=< zI-iC@_;#@6Yif3ZYJDK!DPby6T$U_`=dcV;r{VpJmGE(#uDz21S1LeE5Sg*{o&wr!rkC* zn~K@{p`a(>emTe{+&>m{CfxNwd&2!v(3)@`3Ni`zi$PPuT^BSY+%E)m3AfdQg!_P9 zOa>dN8kL_t-Is9N#bzq*ANDXY-n|Pi?r+s4a{o#?taGVTc0%hVQom~7ug(TFy@!I@ zv$d&o_vv8Z-!Rmk6wUr0)?3(1((+?9;rHZkN8f!{yxV`T==c);_`iVsYAM%s!L$S+P&>zgigk4A^JVK6Ccj zuutZHGuA#E_6e*_Pyb)WY*{?^8MAgiXP*sQY;3Ph1&aq$!PpC_prI}mOdc}rCHuC| zhJCX2=4YR2`>eRX8|?F8`}^amVC@t3_a{@qz^CkQ`!pQ3FfBGL{ax+pPKuho4#y+uUeQtH~nAR()|x>g1G~=!Q4k` zgG|Qykx$hIYo9jv&)DZP_P2c+K3f}P?K5Vd#n06S18udzw0+j>6a1`&vrpDO1NNDA zf1k9lUpD<~)+ha@>4VlI+h^bn`wUxrH*14Q`^^2l+F<4H8*|It_Svvc-yc|<|FQ9Z zWV%hv*6cH2Gq;*w$_MG}Krm_fS!-G6>h{L(7XM!SqvGFz<(HVO<(dA@TfXV<#ZMRi z&RG6QPoK;l%$o^Tt*uVlAF21;=pDM<)9KE=XkS(>0{1j}kotJA=H2e;bLV!j^LI~^ zJ9oyr-P7aFz4(d9-^F*x9;R>j>0*3`%s*>w!-aS89ZIzZ8{X~gIh1M$2CU6Ee-}Pw z?n!Gm&h6|;S@=2ccIQ8oIvBKD8&Z7Ezt6j!JM zgFlJ*Iq%MRw|i5mPq=I&!=CoPUG0 zMV-&(SG#w+@XnpFb{dZ_Yj3p;c}tJWPggIC@gEHG*W&&Sfvc-Uf17>%aopeDneflz z{x&=G?)EVLhQQV7c=#3Xj{KVfSI>+7O+mv8kz3(i-7osv`K=#uZiRR4q3Cbtx4y-> zUHQ25O}-hqUHOFRo$=v){%zcj+^#+xN_7VtHkna;hf<#nHvWls`}~?)irlV#xca_k z{Zx^EC7Apl;_jwk<9|xJ`~J_Qdu}u7Zb;Y0;%f@}eyTS3oF!*2JfD^4CnLAB*OhO+ z-POLR{yF#BvBDu?AcSr5p&i_{A zAB6tap1oV`*G=bLd{KL?3&y-#^>xj=RlobrN4)$Oy<7g7zZUsNQDA}Be&|$oOiqS9k#!}5&5hBWPUGl zyY}kpPyQY64$H&p5qpajJd3@vcrAL=UP2;rQv?4{fgR4(Sb^Yql}g9aa;A_}h%@Uv_6M^6lO|dN-6y_$}*G-NAU; zEKTmvdEI=>9rx|p#2wNDuXov;%^fB2yii_$=(C0Vtq2d}u^zi{EJw)-zF_G+WU5vf zdZBCeW7&rkuT`0D=sJN-zrsVj^ww?kSr3p`xX1mrq2AJbpd{GIP(N6r|5m6+@Ai&+ z{v(?T)&}lSFPeAs`uO+2xvpXg6l?2)C(;92iH zSC`a%H7STM*--9*^8c0v-YNX7kHCd@9K*Z*#btptuf&mW z@y_`ugIsvmALqTSFS~pV}z`pKD%)W8lvt`~USj(f3c9k4t|7GWRc7#`%;cyuRkg z?=|nKh~W(SE&RJ>&fjixMtAi1fbRV6K=uX0K71CYjr;Jbzb?G%5*WV2^5^^I&i}3G z{22@54xV52=$9<~H4C4``TG>!Cs^lKIMv@*O(+-M%|YEU8~z`i-;G8L|C$Ll8jTs9 z$p5DCr2Re`m6e>qH!S>j%cQ^4had6wTQv>KpXYb&#^q@yADyodvHzpVhu62lJNZe& zS(Ob6?UN_;?KSY<@$)%b)h-xJ?g3vx;^9I`?OCAzR{PSo!u5Y=Z1_u;FJHRsxi5{6 z4WG+h$s9YH85_Jjcwsmcav$^4m zS1w)t`AmQK-TIqHj&`vU(9uWrY&>j&2so_?I%xmob2d`frCfO z$9nuoM@L)c#PQA;Ymt`HCtm9~d7|UQ(HJ*+>8)7bv#UgO>#EIi}QpIeAkUKv(8aL`l z`)em%iiC{3@pj_uM~-zKb*Ym7@YtnuBe4jM9*Kv(^meXz@*|xc$DHRIVU8R*ZaH$< zfZ?-4Z&+cgWD~LH5?)7+y%v^=o%?u)g&uz+o-Zel9*L*z^0$j|p6EOtow`^SSKhpG zt_U14vx^S*KP+{IW02uoi-o7w=F&DZIw>vdD+A_mok79gOeN6uK!O^$X z=~#S&qvzkcSn$sb4dw`C4WD(%{KC1nhXxsbDD(*nK5^cl(7S#9%E;i@aPXV$zm?G$ zV)5rLW!@Ney@B)l%_9+0;_rG;=YKTvkHww4e13?@kLOnuzpuU9pFao=AF*xBFtJuw zhla;9g6%tLPgwMsq76}~&xfI+KruedUyC|%#QJ`pI|*-`&*hKCg@{Xp_c?IbGN{n` zBi5KbQGU3wBX-IXe;ZX|-tl;Mm){-MS6Kh-;y7%TD>f{_vgj5U~I z7Cx$vzK?XlqoFQ&+87_@hs7mM2|WB-5V(O*A)~+LYw0@fjkB*;S9wv@wkoz(V!iVDGE{=pBv4C0iy2O!V-$PXXs2ny2- z#8496Wk&3Xg_EgPR#t)i5er|apT{ig{(p!KGJFZ%9V2pefY@9Ae<0MP~z>o#2z35Zv7v={0C9YjvWet=)Xt0pdG9KseuB2=w|)jd-*?T`1`NiYW%~G z{z21$p5|u5Z^gp7vJQ`)m;3(EE}&h!cf91jn!e~AXI|c@-7a23T)dAw>b~N8Kry-G z&$lg|>EQnk_o3)%cVAEnoH_P9S3y@EMm=6p7vR<}Qn7uEMccy*5BqZQXsWLK{!Pn+ zRBV+|+3LdBZdw#J_P@Jrq;$2);T~-EN&p#M2*(=cBQBaxvX*|0^7mF+JV~&u*h@ zebL#eHDwpsvd=iDyITu&WPMX4KO58Ie4THKeJzl{$IVRi=Lyq4QBZA+^1AF% z8m7IjdL?}(#`_`-W1Effas4*<_K>!FbHGcV_j)JnT=2R}m;Wx`iXZRVz)Nojzv1}~@X~w1cYMz}Dm(hXOYaAt z_55k@($9cTdwu}C^daz#o*w}(JqKQPj)Rwe75t=+KM!8|1bEqb9lZ1#;AQ6|crJ6FI;7?*uPB3ts8!1~0t_ywcSRUV0yRrK=yj z^wZ##t~21J4}qWb=^X(teH6Ux90M;s2VQoLgO`34{Ft{V4_^9p@GYLd0bcqP_&(3y z1TXy-c%^F^y!09HO4lrS>9@fvU31{2FM#jy_TK?7{Vw=9&o6IE;o54_UV z4_^8i@JiPJcgTt>OAmg!eE*jAI{UXe@UpWWyma?Wm^(B-9P{=! zf|uS5zQy+^8Sv6uz^mL_!AoxguX1k(FTD%A$}tOGdN+9W^F83D_kx$5ec+|{gP-vE zc^bU*A@H(u1ibW7@G8eK@X~YOmwb7RgO`34ywa5iFMR^M(sdoY^eOO4*G=%!Z-G}o zGYww)40zc&3tswd@Un9by!3hSS|?cmFZ~YqIX|CS1TTFVyz*fMy!3nEl@F`nrQZjy zd{_f7{Q-F8!#a5BkHE{$P4LpUz{}3y(^0>q{!Mz?>*}v$XB~Ly_26Y^19<68;M;wB zX$CJn177KB0WZB3yxM6Sc8b}W zy%GF~Pj3@=>CNEP4`;wjZvn4Nk*dhpU4z^ffK zf|uR|KI83a1~0t@y!!c8@Y37B%g%Q2(mTM*&Q9>syTA{4``x=>+@buG-UD9acrSSA zec)B@{otja2Cs5I177+Fc;&+=cJFkM5o(G@v`8fez`XqSSIR#$& zP4FtmTi~TngJ1If{|tEPv*4Ak+u)_ofmgcb!AoBRuXNo7FMSET`k7_$(pSLC&U@gc zuY#AI_rXhF2QNDxf|tGlUUohLFMSid?A!t`J$x|U+Nt_O*_n0^%-f;%E4|+9;$>$8 zcCNC}X9m3V7VxsO6}R>b%K}P1zzdOf|uS6UhSv{y!2l1x4b>> z{c`dAmwp<&`uQ{9r4N9YokQTIkARn*qu`~FfxqkR&w-bI6}5nipGUz<9|Ny+<-kiH2d{Q?6}%F( zAH4K6@Urs(c7kKGe@UpWTy!1Zsva=t&^wZ#F z=Na(Q2f)kDA@I^iz{}22@X~YOW#>3}=~uza&OCVO6X0d%b@0+}fR~+<;HBRJuX+A7 zctE^h;MLDJfS29~ zUUoKtm);D%-REZpy!1Bkva=n$^bYVU$4>ClyTC8`a>;_1-VI*q>H#mk7rfHd2VVLa z@JiPJc7C$}KV9IZXThtV?*=cu2fXa;1uwl1e4o$He(=%2rO$!aeSP!br7wUV@b%#icvl@H6{rLTZjKHLK@eHFa&;XZijYv7d+ z55P;`053ZqftS7sUUqJQmmYjR>i5*Y$@? zUV007wbNGc(%Zo2yglvUrFVjF_wBt4y!0%1+1U+VdJlNn*$ZBJA9&f>4_^8i@Un9N zy!0XPvU3Ey^ilA#a}2!n9QY}p|Ks4L=fP_noB%KV2KYH&UX$RZPk~puZi1J73%t@b z4PN>Tc%^F=y!1KnJ>LF#@X{B+tDW8fFMScb+UZ^J(pSJMAMSydz6xIB{(bP$*TBop z2jHczgP-yB=OK9Mo8V>V7I^8w&n5dqm1ElL>JO#YfnW0FRS#Zz19+vY5xn#!@Jd%R zc;f-63to11gO}b1UUv3_mwp<&>^uWr`T%&@IRsw% z2zc2!3SRm+c-eUsy!1SH**O7T`gQQK^9Fe7li+3N6nN>kz%TmtH4R?+40xq$7QFP^ z;MI=iz)PP8KkMyT055$Jy!!dO;H58tmz~StrLTaOo%g^?Uj@JF?Y|FR`UCJ9*Ve&H ze+XXXz5!nPBk(HsP4Lpw{)H`?A1WW}ysrLGdOdjc^9|soH-eX)P2iHQJFkP6egnMhoCGg@3cT#R310dw@Un9ny!6}PW#=4t>GR-a z=K^@?cfiZeMex$^f|s33;H9sC59~L#+;I=Q^i}Xm*M0EP*TAbCJpeC#9sEPz-XDUO z{s_GK`AzWBx4_HJz`vrB{af1W>JMdS9eC;W;4?ly8^B9%0ZWJ7>U4p9L>FZ-bY< zK;GMVhrHJp!OPCO;H58tmz~StrLTaOo%g^?UnB4Bd_dmo>)>VQL-5i!z{}1@;H7VZ zmz`VSrKh9UfEV_E`+WPV^Sb&&>Gj~%4>W+6-Uwdps0qCEX7HWfo(y>Dt>D$qw}F@5 z4qkS4fS2A0UUqhYmmdBmwvC?}FGjrm-QcD7B7XJrec+|{gIBqq1~2^#c$NDAcgRLdrH_M`omat2&x0TL`8fez`V@GT<4y3=Z-G}iPJ@>|1AfW(|Fhtw z-v+OA&4HIb4_@h705AP6c%^Fzy!2)8>StEKOTPzRcCLb#ejmK-TmvutA$Zxj0bcqe z@Un9gy!0*bvNP~M+@k(XdfMx1zp}Fqy!1x!va<=i^k(p~GXq|F3wYVt3SN2}c-h$w zUV10^Mc=-1V*JpC14(eF(hl904zV6uj&l z11~)Xe%0GQ4qkd5y!!bG@Y1h?SGnH+FMSfc%6$sF^l9+QhZ*qFXThtVzYSjc9C+C| z4_^8L_;sJ3cfdqipQzp0-|yB{vFL;a!jIjxT?a4y26(m8N$}FAz}I_wZi1IS4PN8O40!3Y;AQ7+ z@Y3hN%g%Z5(igyYdi(Eymwp$##*roP(wD)jUaf$aeh<9r)hc-D55OxQ*1=1E2wvmJ z26*X@z{}1}@Y1)ycl-Pdq8~&l^oP>xy{`UHb~b>Q-Uwdh*aTjBGx#MxzGT2lZvn4# zwSt%4243lE2QR$~ywa5gFTESQ`mrAH(tE+n&OY$c`@zf3)8M5KftQ^l;H8g(mz`ta zrRTuQ&T;V4uY#AIdGOM2fR~+<;H6K2mz_7kOTPtPc20wrJ_BBM&VrXd2Y$e}uX*s& z7r-lBcfdA0=?}r@z5N^D zrEh{)KfeWDdfgVghOK$)#I~&1EZvrnno54%ZfM56aw}6-4244MqJ9z0G z;8pIO;H7tgSGi}wOYa4*eCPu&y&t^#`P1N~p8+pB2f#}o0>A0=a|FEf9C+C|4qp0I z@G8eVcL+bj^a7ejB{{nK|&%=fTU)1@O}E zfR~+%;H58vmz^u%rQZWDJ6FL=zYkt^u7Q{S0KDv62QU2*c-grLUiubz*{L7ORR1PD z?SDX1{hREp^SXHH_26Y^19<68;2Zu{wEoi!UU~++($xZ9dMkLfqc-r;!yo6heqQO_ z^6k9?y!0-_uYNuYUV1lp+1UeLdM|j{*#};FKloOkpQph~9{{g@eh9qu5%4PaQSj2o zz^mMI;HBrmD<3AnOTP|Y{rnB^(kH>o&MENHZ-Vdi`FRVx^jYw-^EPdINab*$7^G6L{I#3|@LGc-h$oUV1xt+1UYJdM9|< z*#%yD7QF221~0uAe4lS$ec+|{gIBsvgO`2=yxP$Kc5ssxo@|1b zz6E~Ck1xT`NByMwL+NR+Yrdp>s{=2+9=!6c0lf5P@Jd$(y!00E>c?8aOK$@&JKMoa z?*K15JHbou1}{5%z)SB1FFX6dOYa9SJ5Pg`eg?ek8~`tU6uj&l11~)XUUrUymwpwz z?979gJ^@~KUI#Dz26(l*N$}FAz~}w@E4_^8L z_;KG~-2pHCE_n6xOW>t1gIBq%fR}y`yvk)2yz~d)l@IISr9T9(etrYI^he-j=O%dR zTi_>reg=Of*&j-;_qzH++1UVIdLww1V-tAk&ES`O|C|9Yy#>6|)e2sE8+fIw9lZ1| z@Jd$}y!3AH>IZtjOYa3QJNv*(?*}hCPlJ~}1YUNIfR{cBUUrUwm!1PJJIBFGzY1P< z=D|z90bX`af|ou8UUuFDFZ~vH**Ohf`V4s4ISXF;ZSd-c=fF#!2S4rCXBNOqUnK9_ z?_KgfS3LVyzJZrFMSLA zg8IX*WPd2V&g<$AHICGSm)-zg^{NrP^d|7CSIyw1w}Mwbw1Jo24qoF(2YBh7;ALkQ zcC50{=L&e~_rS}}Rq)d9gO{Ca;H5tRFFV)4OMe7@&9|>j@Y1)yD_y}ClKq?XwAaMcCLVzejmK-Tmvut0eIQD4qp00@Un9Qy!1!lW#=Y%>FK{3^?T~yWM`e%wH_$F z9=z;q0581}yzFcOFTEMO?970d-U>eB+gBTS>FwZ^t`6|hJHe|Rb%B?j1z+#&=>{*o z7rgrUKJe1}!OPCm;H94dFFOaoOCJK?Eqy4?pMJ}&x2RFPk@&` z310ay1z!41@apGpftNlFUUtramp%)=*XQSL@X{B+%g#ICr7wb4Io<^?eF^-M@Bf#< zOJ4!6bln3leHFaYbsxO+b?{2pL-5i!z^k8m1YY_kc-grHUV88g$^KAwroFEIPfS3LVyvlJCy!0*bOTN7X+2r-F^t9JCKUBKvz)PB@kY-U43zOe=WlZQx~RJ9z0G;ALkgcAm1(XCHX!{orNiY4Fm| zfR~*E;H8g(mz`tarRTuQ&T;V4uY#AIdGOLFz{}3-;HBRHuYH6`@Y1Kq`}f$~B=7ZU z@Oj^UXTVFJ1+RR&4PN>jc(v1c@X{B+k9vFVfR}z3yvC6w@Y0vT%gz<>((i$novYxb z-v>XX{t&$Mb?_QT9)g#?0bcd$5qRmF;8m}-z)P$&WAH;HCF~SGsz^OYZ}(boGOm zJ^)_n8UinU1ibpOQSj2oz{}1YcgTt>OAlTz-@m23&i<_qyzHz8FTDZ$s<*!py!2-9>gO}y zrMG}rxwnFs-UeRf-VR=R7kHIp7QFOs@apG#z)SB1FFX6dOYaB&z~|>_@Y09C%gzz- z(nrCo9LK;*&w*d^{r@<4=~uxkU3u`*C%`LR*TGAl0W;HA%imz}fV zrQZfGJLkYlzXM)&E`pbS7rg9T0xx|TyzE>7FZ~{P*|`c{`UCK?a~-_&hu~%B26*X@ zz{}1}@Y1)y%g*47QNO4DO?sWzHQ&ql`;qFwOK$+LbTxvP-UMFls2RNU4ES~5-dn&+ zZv(G>z8$>u4)C(G6TI{;@Uk-tUV1n9N8bJ(@Y4IhtDo-&Fa0!lmHQd+(g(n++=swR z9|Ny^$bpwW4qpBIRq)dD;AQ6ocdJ}lr*$iHK2E6QS0WZB3yzFcPFTEYS z?Cbz9y%W6Fk-ET3&w|%|s@>qF_kwTm?Y9rS^nUQlx6|OIp8>CSIsjh!5cssWX9T?T zG4L8ka^R(pgO{CG!AsABmz@*frC$f%qW%!P^eOVbzq(1@>$kwGUQL6SJ_BC$Y8Jfo zdGN}I1@O}EfY&&(2wwVK@Un9Wy!2)8?LI$Oz)QamUUsg5m;L~}%5fdM^oQV=e7S6Z zm;MO6(zOX*`WASlEBI1!eki@(>zW@bT@B!+H-c9`)&yR9GkDpV0WZA;yzFcRFTDf2 z?Cb>L0ueF(hl904zV6uj&l11~)XUUrUy zm!1co_3di{y!7kfm987$rB8xaJDLJ7{U-PhZ_h38(r3V{pPvOU{Wf^nIR{?)Jb2l; z0ABhX@MpaJi{Pa%f!B4!GI;4L;8pJTz)N2RuX4W+UivzC<- zfR~;HuXJ^Tm)-+j{Y)=->3!g3XFqu9r@_n4GvK9ZWJ7>U4p9L>FZ-bXU4_?nHEP$7O2fWg?2wwVK z@M=d(;H58vANB2h1-$fC@apI9gO|PrUUohJFMS=n?0g7b`Ud!MZ~r6k(zn2?pAWv0 zobO3bdtLKAm3tj{>Gj}M?hW9jH-lF`WWY;r0k3|(6}yy&wFN@BdGOmwpDk(lr2H`Ve@fYXrRX9C)Q`9K7_a;MLFM!AqY2 zFFUV;mwp4h?3@HI{T6uHISpR=40zc&3tswd@Un9by!3hSvU35v^t<3?=Ms47%iv|_ z3V7-Fz{}26@Y3&tmz`_irLTk6KEgxr(l@{>U5~&^-vqCAv;|&z(370+UDtfi>zeOL zuLrMwz5%@SM)0z;3B2@X@Uk-lUV01oDQ|x(c*xX1}}XMyz*@xyz~X|Yu^4l;H58v zSH7))mwpet`pH%B((i*;KCFS4{s6r4VI92mhv1bD8{nls0^e{ws&AX%rKf)}*&iw& z>b$Q0P5br(4^7~uH-lF`WWY;r0k3>$1uwk~{EBan?ck+%fmc3c!AtK3 zuYBkMFTEGM@}UpB^nUQlhtuGtp8>CY7yvJQ2z>DUsQ!$Amp%r5(f6M@@Y2V@E8nhy zm!1c&ery7~^y}bfy*)R;OTP(z@tr8Wx4=uE2EXy$h@Sy3eHOgR@iutrbKoC%`{%(+ zUjVOiyaQhPB6%PGUGiRE2CsR}3V7-Fz^fcr!ArjnUgfw3Uit&@_q?6!;H7VX*F5JD zc_64+(q32pDZLK7%B3E>^cJt5NCiQ2bgJlF7T?)S@6=k!7G1yz)SB1-{||{KJd~9z{}1d@X|-X%g#~o(#OEd z&K!8@(OJ4;qJMV*+z6O5X+y4N(^bPQ`^AULIo8V>V7I^7FZ?b=rooTPDf0JGZ zzQezFx*ojrCh)Se8NBoic-h$kUV1Bd+1UnOdOLXiK6M9p=~?iyvm3nh9`LfW7rgX7 z@UpWXy!6xH_4}u1z)K$iFFQxUOCJL-J9FTrkAs(;SHVlqgV*o(O@Nm^30`(iftP+0 zyzIOMUivh6**ODV`Yd?;{?u*o(igzX&O6|xFM^kycfm_v0xvt4!AoBOuisC(2VVLb zc-i>?y!3VOvhyK$=^Nl>=OggaH^F!M_sDI5mtObPWd9~R>%Ff2O?m@(+1UtQdJ}lr z*$iHK2E5*1-2z^EJ9ydI0bY71c-h$nUV0Y1?Cb_Fy$8JBkJ<}f`f2d8^9*?D1K?%n z5P0b$;AQ71cx!eLTeHOgz zybWIZ9C+C|4_^8Lc-eUey!1uzh4-s~m%ajCcHRRoeHFayyboUb8hF|H0KD{d@FPC| zAA*;@30`(?ftMb9E!khm&a~InUrDb6FFWhOOK$)_&JOf_(0QkVKqYiCFMS2P?7RnF`YQMdZ~uMp($~Su&WGToZ-AGbkHAac1TQ5Paz-xWJ9=!Ah@EO0p-3VTKdyM~(nN}B>7j(q>}db{@+@!!%{Tx51khyMo!e&wY|ScKkC9|@b#Wl!+U;ma3? zN3WcHoD_4du=gwYv``o$VD_4TgzhVFF6h-hwv#k3YyMyxZ2$1ZE0-?6bz%6zr6Wf>kF}pT>OLnsj-5Et+2Q^?l0aT-KY6m_ zWJgC*J=%HvNJqzsa~+c?XLuI;sqQi}-!+ z#m=4`%3Zn~)il+fw=a%f8XS7sn8OMgMwGjBb|_41lt-37QPLx)WkB2+O!>B8AD*ZR(OT0;*{`1x~Jaz~C^L5Q!|$_qWq zH^XkjnHFy~UM-~i%D2z|&hLy47u?^7+*hngy>a%KmF9QGhR=l&_=w&ZyfS=t=*p;h zoWJxZo>;Go|m(CtN`^Kel>rXDYh|eB7dR9%w zvcA-Wv+9O@@A>3k({_78bcaGRj@hMjD17ojQn%qDak$rNJMCaX4T7w9clmKh-VOT0 zf4lL;ohj?x`f7g=S+(-#k>Z96H^gY%Yx3YHEwDTPE8eZIygx6`C%3-m>@+UCAHEbk znDpUaoJ{79!U_B*_J^}6N!d8~XOTi)_AC64+aDj=Av}4yUHs+pFYx83d~*5sf_)TT z`YV=jmZ@REthe9&9Y4r^(}!pNDSQJ0==_TFO$%7g{w8n#qz9e-hwP*9vVYdHndf#v S7S^KtANK|0%=k!|@c%!a_sjMG literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k89fbca15f9847079d4dfb2ed49d66e4f.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k89fbca15f9847079d4dfb2ed49d66e4f.o new file mode 100644 index 0000000000000000000000000000000000000000..e98c96ba0373001be37e741e3388c74154a4adce GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Kp1A?+13jo8ITZ&*Jf6oAt@$oDi8|B)|;rcpXTL zd%wRP-S=KK2(TwMEOpdFhXNPB`_SBrUP3{}A zO{Bh#^rW-G|D${QhtW>gY;V_L?2%K?4pyE&Fu3FLvx6&7KRfs>-E$4J&*61=&dKkD z`%WAWKkoc)`1pYn;gR_#!+rBlh4-&M9lo^xba-U{ zb{rUd=GkWlQ|B)U&1v~=_`!*kAG z1YZ}!<7Pe<8X^30@cQ`4aP!0G(eBUCZvS8fVEYnJhDQjdWncYi;#X@%F1{Q{-jn@< zKoD>k5jv&j1|5HSPpL^`uY+4E*LS^s%;>-I{P6>uHdpy1 z`JeK=0r=jl@Xh;KJK)}M;`adsBFHvA*~+8glep=@^TY{{gk$+Z7Z3K8tLh*@LC zKJY}kYD&Zv?LV#>tuJGL>{jV>Js}#O=x1o2em+J@{4uVlO~3kamBxL{mFc;bGS3rA zU0f7$exW!-wjrC4ov6Qeq1SWJp9}FbknHK-f&RaR>l(Z&ao>aM0bEbu^}ld`8rOfN z@nNrbpHB$cU{~dTw(;!RyU@RDrXF&^J{RG)WT)T224wRKu=&|VtbYjhBReRcet#|N z50|ihrVl(uHdXe&@@?1(zWg5i_%dwvudrXtAK2k5hhSHO9h;^LeUP7iTvZgho>Y95 z>~~)uW_`Q>=Yb#d7rotXz52b{zt@lkC_2+L3h+T$C)f(S9O!%pt{%LS@JeSc z8g^9K;S03+S~^%YFu8=)G4u3PTy>4v#@jUA#+ZE^mONQ(PG-mLmqlJ%q!2p$pF$={ zwxwR&oc;IWs_$Tl9=4PvfTJlyxuBz|qchOyt;e0ZcDIT;zt(j$wKWFWg0}Wf--5O$ z7If}tZrj=(t2cDX`V$MHHx2Eres3pR(CG_yE)b&CHFR~hcsn}kye%!;yq!(HKwX2k zqq9z8JAG{(?E!yhpm|ZjwpL%Ox21VoTmF*v{JMWz3H z+Zyn;`h<0?0Pg}51kgw8_sEcA& zm9451n7TY{)tbE>JbFj*)Drp2J(ZK)EHcdj(uUo#wJhWCL@Sv6IfsuQ1>cCxx{dGH z&8Vk294CdA@)F<~u2?@kIf9@btjxfzN793gmtBCnkETW863o6K>DADR{(FGiaDHJe z5tLjBqnlntirgjG|Kl7*6gng!(j|CpC6K=}2wyDeYV^A4bfLM$+u7XS=I(57Z|Rul#?1D4 zJACf^MGF?sbFZh%g2H92xU0FP(cRJA(dlb-w{~@Ox*L64o7;TubyZa>*Ve5st65u7 zwYsis-VpMO|*-NBv$+`iDk$P@DD zK3&}7mf166_qTGKV{1xf_N{i<1#KU-i@qPA(HT4tCqq}#nO)ZHWa%qK(YlRBH`I0J zHr45@%7~{vscu8Jwy6=Ouf#D^olUdPN#!;ryHc^w#`rLbD;c)Rz-=XXo0iJ_=-vfK ztj5dk0u2m?KmM{$7hYz|TF2}u4o2SHW5fLt3EzREtya9Y0f9o}(Dh?^y)LgG;aW-v z6!MGFvyHLdU0Q_w8b~tGP#VkX7W(3oB*)2)?@WyF1Gp4W)H~S%I;2wt^r-=sN`w2W zuh`fxvsv$cdxTv9fS7J{vOW6}?R?x#*{STfoBg<3$iSWk9{u$OF_9#Q#xLvE*%snO z@39Gf9zX?UCSK>9>~O$tM++7iIGDzU7P8!+?SbKpc`TwIOk!^wVzYt?w%<;9a|f>v zW!npbx{dIR5gzgdCq}YSOPCC8W4u|xBt1%hGL5ANCMHaz{s#{wv2#SvIDHb)1D_kT zPo%R>(Ld4iX(s5=KlEWEas1UEaQ<^4pMf6yq)B^$AM{7`Tz?sz8u5d=PV)Jy!#3hK zDL4sM|`O9k>>N@HK?9v|W6AbhS9`q9U)udS`+^1hi3c_-`SOA!b3gZ^`alP8nE zjpzCob#~@uUFoMh>l@O%uqXP5J)GRh$sXXRFu*2G;^T{Y@2-Rh`v3s)0r+1=z!o8o zj4TrPvTluCAqF(zA1eZS8daJ0G3Un>5Wa5v4fKKj^?;2z2~SEO`vzz7;Ufd!e<;nK zO8$(6(!T}z0X`#`JCoK;_!n=V8-zKAYqiNF-(WT?|N0SNsbKsIME~&2tC5Hnqo3m> z{Y9jFW!+dNXktr$Ht_@hDhxdQFt?W`#w!fWn#1Lx_NUV@Kj%f*S+uA5ls6sqxe<0r z*6S~8GlTQy(0KERKiFqaw__fkZx2pSWq;{pGlO$y5q>VmFJw=3v$>?�RnH3>Sgk z4tlI@M1Rc5@pjY&z65wT@NN^{jk>^>0Y44+Jb@n;?WdtG@LvT!6L^on&rak0XQD3f z$=aT77w}j_Y5XREccCuu3A&JHi^OM3{wLa&d+_;(_M|U6`se(-W7EsCUSPDYMk0H;{WD?TXaRkZs*}Cp zADGX%e7@3r{$&~~{<;^}oW@fA6m8F@G95i1jVOM*hYZh0_b+Us&&>{zd*XJ%PrX&i&^Q*>if-zlNMz zVPIN9)L*e4%nVMOM)Zj>f18#`b?$F_(cXpj%HLL#ey6!i{^kJQE$n%0I>$Ru7wyR( z?Z8WagMUN6Go$)d_9h0%-tUmU=^jPuo5r&%{~RiTemVtS&gaqkmuT4@`kzVaR{lZy z=buOs9^;jNeW|Ix-d!0neb={H^iA{Ipl?23Gq}Dpr2gRl8Bu-LXJfr^S@i94#q?c8 z`f|nQH=iG9FZGT7GlI|-1c(guiuIS~2RES9SC&~`>Myz;3>C5Rtl1{{#?n{sF60Ae z(S+8ooIiv7HiN&B{0@u1*?Ii5ll-S>J+q7af!2q5S}#R=72jqC9h}|~^EU_A2l?AM zO&`sl@K(MaWB#px{X`M-&l(TOzGsL)uHS{h@#D$V(Ia8~#G^_X_9`KVMdL8g&5mF1nEg z_=DX{r(I8C(S-4kpN*DZ(#F;&nI9m3!~B^~{HTKZ@tv^WlK@aY0RN%*`wBTEwf}(R zKN^eg&?n@v#CP2g-?QVybL7Be5~*K3kQs~DLkqPP0Xs!*LwwhDvM-PC$baC^_~@AY zxhIL%*Yac~fUu|8|49EfV|=zB--&!f$yZPLj>Gxu>mN=O`3~7HCBn{OK<>X}pPTz9 z{F~>C4w+w)JiMsUex5~@>6!DPkKP6^=_`i96 zF68vwo-`+*&%)p|ot9fh`aAPqY5kg($gV_qels*3{*{~=^{=6Y*bn4nLVOg^57U0b z!Tl#Wxi>~Xmh|i!XGg{wU7GahiUA&Fh{b;*+?4kSJ(@B z(LRat9puxtfv3PLU*FPP$j{u-{vdyPV7fj9^br^4^KeyadLUb$O7+Q@Ko}q6OyoJHc%xFBb@XWB4@S zlTbI}T{!hn?FpZbx)Gl-!Hl1PdJGSFxxaFKKwhf5z=6<@4f8p7jC`C=E)icYRGH)1 zOUNI0vsChL_|rJ{W)=BUDz)&J`t2f~d=WJ&hV^0?RUQwfll@X(Kvl$Z(sv@ZBgS~1 zm`MGrc%Bdxvx(JU6yy(kx)~av(D?}VUx&J})QY0h{v7*bafnI#!#BCVlXfEhV!56r zFP7wjzx05fN+%|f9j{kWLx49hUv@ii zRDfc`dT|Ql?-us(*S|lG{fgr4>->CA>F2iP!G4{uALMU*eLiEdr!_w_RCht?tXZ)OdrC*-U zMEx(8&+zk!b7}UQ<}ZkEH_TuD9R7#+_F3mIqw^I<@+)4XpCTW@d~-R-ANlzT{L`GT zECk-TKGS+-%~uTTFXBZO)@v*$BHlxPru7`_vuXWCJqzn~lpgJ|9>w-sLz;QL#(X7w zbiE!kU*YrHlCRKaoQm*AzMoL`hdpurm^m5Phzs-m^rx7wkUzxDS7wZnuefL2JYUJ; z`HDM!z5;o&+!WuV`AX1`Z;Z}Y5KrA%(R^hX`upVhNEWTHJYJ$dOFn{lY05_&7;lCf z{CNLjLprWj->`6u(CAzbU&wk^AohL1rfi&p&W}U*7e~nDOeU zDP9@Q*Wo{R60f5BZ>(?T{q{n9zA)B)dp8$Htxr5Y(~%l|UX(-e{=^=RBSvKT6+i=lK)%18M9H&nB?S~?nBEP%+cxKvfnBy15 zH|;l|Z&N(uc=P_lwBG>U6yJt<{{0E#*%gz&SmPN!Pm0+e{HEhPY2cvL5A28i%w+mJ zhOf`(@Ok5|iLw3p!+d|v&rf;2{pLc@Pd4nw58?9$`((Z!=l<~@Xg?l1@4OzUk58R;^Y}J}`}@@R@eT1~%2dApQv31I@y&GplN+2m#S-6Q=Rel{E#~hn?6;>F z<6B{Hg3j;(me?QC{K%Xz5f&2r?`-V9vnJ5zbB6u*G}N=B(eRs(>cce8wV;> z?j+t--NF7l9jhf66#G@QccC8Lk0M{2U_5UEKErq(MgE^j`KH)^^7)V%-GB1^H~AAv zG~P{~%-pC`zIuKk^y$(K`h@+{b&Ebt`;oEp*FWNXzm#CkUyKdTKbA;Lfe9M9L{POxAKKyayt1tIe|8Dc&{{1g1$Ng2~ zM;HD&a;T8a?t7-!RqO7Xn{%wMkFkDx?ein=|5IJ$YaErbCgCspa(esvQrMZFy!waV zN8Yisvf4E>j_GS^6V}vXH-eDalC(nN;^RfG1w0fJuYHvNEiv6I8^oiJ%HwBjjix2*&~y~WA!JV~qGy9n zlROrG;8fg}WRtz4YzdJWx0^xldXmt4RBjq64n8WkD?UCdw_C^^uaBr)V1#5` zgJkwkMKb$nk0MK$IFmF{&LG{*I2%0&&79kGE9Y@_t_#khqoNa|?S-z3%G*-n%~+wQ zVzkkib+l?g4SF1fCvHdwYjl0p!bpp5sDn_Us`YYY5C(>G2lZ6;KW*dv&W`P^VsJpiW(p zu4&X9Rgdj6Y&wlnl)BDdc?n(F@j-aW*ylL5(EBztcO0}W2QCwvsfcPjWIvFs2Pv@&yE|bJ;$mW zXW0E1F4mB-boNQ{MxEb};pSCb#0nK3M#ml6W5z#8v#jpAb@iusVc5xO+ErvRY9}Wn zt{T^5p+8DoqLE{<$(C?)?)N6#--rHj=4!mv{dVX7okxRDV{|LG@%p@T4z0cavFoOp znmGEj(8^Cdddq&uln;$+AKi2EIhYV#L1_6wvC`jdb(4PLtbA5^;?b+k9j&;azpM9` zoPYCK-skOX@AIx-(|j`S*hH+4MZLYFc5EJaP%LKeWyYwdX6@U$zPc29ZN|^OwX5(x zIt<5dTa__K;GdOv#AjG^Bx^ZG7d~#n4~Xx_iS2)^|IsJoTmnP?y2ktOQM}Rj&W!y_ zyh8JTb6cZp6)v1F(0mf*j2#Cyah_qM-Sinoi&b;WwP%F@;3FFdbQUKb@ta&r?emeaphtg6jT2dBb$j`y^f2tw1 z8~;?7mkWDTf+sxx=auD*Z7Ik9mg9A!=l_s`$QE>s_iP`6|BBvFY&Z#uezgvDrQOE> z=Lhqg>c($#cpv;|$V9Y^!$?jS9J*<6FyJ|V%o2X(aR$B54`I)#_QYGyV zWc%ZWwsdRNJNXZ__gL6ylqPorM}<6(c(=B%ASAw?DK4@1ugLbf5~Or&ZBP5Tr{c6f zW@)c_wzi)k6yoJcKlq zvz95Q##il>{hvVNo3&ST)Sl?9?P;Eq3M_8Q7&#_NPmZ`!d!p$P)#y9^Wq|it5{1Z{ zrSeLH%EJHA)let!_uzHJ6ZLN@vc1Oyv^W`Xw7wAk#TGg|!k?F)8#<0>KR+B3|6fP; zlO8YHo&8gBUHqW9X5m8en}|^VcL37(C??dzO#()iOY|{S~*oKK2Q zUDfW_9H2Bx9YpK||yrG~8u(h>>FKt=2X4A&Xruq%*^EPj+TQBeDM#mE5)VFc5dlik%xbf$#eA z}$+lwo=qVK%Mjg?(iyLe0dPi|^n{Fw4Lb**b{ zX%5uwSh{R$eSJlq@im`~G!OJ!Z^ zgP!O0BliE4_@5nLOZ}$vMbR_SH*x~;wtEbE8taIG{&jV~!g+MuH}x0ovQBhAR5)Bu zse23EG{Da(9ImD6zSQY@UjL?@@cbWtC7k-@#emN>1f$s&JkggL@X=pITIqac91ZZI zSZRr_KE{t3E_Xr-h^wGC?z{!E9hVHl&P~ZhVUpm= z|AoF7`xfLE>UBx~iiE5A^pxahKpIfNH-J7}4T_%DpiTu(@#XL>vi=VO%)X5)(dVH* zDksH-y^SluuS&RG!j*gz(4X=hLQRzTNfvmXgiirnwI$FOC44I2#x3J+1KgM@9+Pky z7pIBQa$MfgxJG!Xl#}{byb)2C~ZghJ;)D`MHE!<^7eRA0z)C8Q?~FGv)lDag8xD?*{NA z5>EUWSF;x+{1FR$z(8-b@9Pq7<)5CSLxuR6B5wKnVhLB{Dqn1pa5b)iA2HAy$9+M< zt>gYm!mZ<`%6UuU8t02k!bi;)38#J(FC@Sg16&^QaJh~c;L(5X0cNorOv&r!)Qo+> z05{6jB;iUAioQj{t@6Gk;Z}J|Wyn$E=E-qa8sNrpgA%UBRrI?h+&b>}CEPkLF96RU8<$z}3>L&OcTE)zVJ0!5A2+q{8phCM=~ccr(ht)c00{*#kscpx)=-I4#s6a8ExyVacs zbyU~f){LKdtncW!y=HA)t%9LuUk4j|lOwkd{O2XdrhdW~=)g}yNKu>X1CYDi*D1us z50}vISeCD@osZv>YuF*EmO~$FeVs*}`2ETHu1+681w9&?ppD*A-xJLZJ`-GM&(|b> z?rv-rw_-f{`HH$W`gy#SE7z|oZT7bK8ah|CH+HpP_*>llaT~O>dmHOI+v^%RV?rRX zC-@XqA&XJMN_n;P1jAg_Tx9l2VQm zBnuikI`fy@hBo5*?3E@@-6ovU}rQI&ukCI=uf6mgt%D)fzc=mHh|G6TE jC;O2rsP-y{-j9K;V^fo}9rTVL9+&bH8}uYEYy1BPd|e+4 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8b914d547c30519c09e45ca3937f65b2.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8b914d547c30519c09e45ca3937f65b2.o new file mode 100644 index 0000000000000000000000000000000000000000..7013d9a1c47df38ef3bc6976cea0084411ddff5a GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZLN?6Av#l3GG9V!kug$DzNevR{(WfPu zmLVrXWF2ELm@%`l&&C-t@nmQjGR{tB9L?rzVH-Je^s|`f4OYMGF{V{DjIv0#cB%5SY~hR-otM_ zEESY))>AWJtNVGj+xCCxg|^SmVC>M4ExYesyw2LP=l%<>=kWSh+@tA4MN8LfJAV-R z?KpPz!}WUTgDP9-nvEU$e>T>4q=Ov-?ctsbR`|VaPydG{?CoFa?9hzVo|-eZ$$dk% ziPYEOo^)3De|1m)Fxu&w?d>{@J#zAy!OC;{2e)5(W^l!+X9mBid#-}^S-cL-Iq~gq z-|=JN$DH2@AKQOCJTm`8xNrW+@V-^2!WZ|Q3Xkl&6n=XCd*O##&xMbjd@r0b^O4ZV z_WgrTKl99B>ima8b6UO=K0ohRc<-g3hM#u66>dI$G2FM}66gv;p#uj7w;kURtz5|0#&wpkxbnaUxvdyc9k^{~TzRhUT*lN zlK&~+8-VY<3g5ivdv~_)_1wVsj*Q`ZM=ZWaKB@d~z~X<(UvK1t*LMCe6e*ef_V5Gx zq4WP#eAR7dX%~J~e0AjIvi@JMVg0F?A7^HboVZ|LlG2>c14Q{&Q*U>iHygH3hUqm}|p( z68eY#iS-XZpdGsKKlQ5@-ere||0W?c{4I9qLIc*+39NtQN0R|d3=KcP4*lx`p#AHl z(3$nSAEmhlo)2By`I>1iiS=E`a{>I5EW_ZH#(a-tf{Yg-qZ2ZbY{`?@)t}H@g?vW2 zSTeijfjsB`wyb{y@>NK=M!@q3cpd@IBqPap0ep@_|6a)C{5vi59^@E#H6b(tIo^XD zM)^sGkyjH#BajEW=s!P!9l8jejJ%u_Vp>mJxg5-Mbq>Z))>uES8y{EwFhsE5AZCpj z`@j?FswojywEwtjw7!h}v74pO^@M1AqMxC8`uP|s@yEE9HvP)SRT}p(SElD`$~;df zb#YP1`Gw*T*@kRFcB1~?ge=fw&K(eQQJNo}7uB-8?#C;F0`*A&v*Z;=-DO~@J z#)rM$eJ&wngI$&X*~YW0??V5snR>_t`&@wElAV4H8<5S;o-(bI(Kd?iW55lenJ2p)h`XE33xT+|0Evfhl z+3%h{%=&m9&I3Q@FM7Mxdi8s?|DZ7!yWD_IP;{ng6ySrhPO#;8IneobTs?Rt;g!x@ zH0-Fd-4|%{wREs*U~&noW9I3pxau0SjkjsKjWPQ;EP1lnoXn2fFN?gkNFj9gzl2PZ zY;(Q1IrE>zRo~7MJ!}a}07p}ZazRH^M`xhZTaP<+?QRuyey!_hYHJL#1#Rt}z6EWM zFX-If+_t4XR&VH%^~V=PZyMTL{oYQtpwk!ZTp&cNYv}51@pg37d0SexdOMqZfw~57 zM`xYHcKX^n+5`U1Ky$&Ot*yRRZ%gymw)}_N^XvY7F&LmOzi?4q0bcoed3gnk>hN#T z;=D)lA6`;dmj^k8Tz7#O9ZlYVukr2?B;f1t1s;b)uu{O+a#wg823v8&wG5ais-ZfKL*_I1*Jp;?2VqA+=yAF{@$V}wMmwRc* zy)@)bOAF(;{=GEhbC$iEmMyZ56>#%u$eq8BQH%5%Y_JxzZ`sn}>&#77K~N7?X5iK%=|RTJPQcwq(xPw)W?z@|YG_6OUBGQPzp$1F zO0ERbJr(f1i>_&@yL)6qun%=URZ zeD3^$1&imo*U@D`;Zj!I)!fqP?&$95^tHNMyE;1EjlM0-Z9ezfs;U)h>eiLjtf{D4 zRadrQZP}{QvQjp`%3a`|?+r9GxjR~M9`K0QjiHHGI$y!(9~NG>vnK(!FLW^Sggm-W z7x%bj_Kev5%^c_0no^m4iyd}B+ehr8@B3(U1`ou^(B*VymvuW?`f^dUZllo+b)C6Q zbvmmu;;B!n+t96TYJ}-4aLiO^)9iCnxlPHgRP3`cK8)f@hOIJiTM6E#r7}Odcfk>> z@v^%>1B2l|f7z!CFSBK>W%d*YBk%69;r?L>-;SfLR=l-yNHWk+8q4Yy`r?x$$H|WENR03UxD-&-JJ|v{q*Dd-sR5QsgZry5 z+t@F%S?@l3gk1)Jm~M2k-Fp-5eB6!MsqC1W{is{Wz@7pg{q+VhktB%5FYDIXX5vNf zu?c?eLj`3fUT2-`P{3|S3lMf3wn?2UtLRxrW#n<;N@=k=j% zdtndzur{bh7HB^$>*;Q+lb$! z;3Qa)+c!bs1$~$i-W708B7RfPM3UHG8Y}!W^p851_f6iPGYvI=J^DlcPwiz9vWMLP zdk$5x`*#v8707onjeVtie1w~W@VQRtM<2hgwzihb`(`%eovf2DMI6u%`p*qco=pBW zp6g%K+3A;crJwSwuS@g7p6DO;aB?Rndw`$90Gl|8k1y)II};-80|3Ye;C~qbTZB9^ zQXueU-5R@03~0hXmIw4Ssxs|k&W|e~eBJhI=mY)h0UL7?o|HiL4bJ4lM+U(EP?|lJ z{22?Se>3z0d`2*LCas(BFWx>k2y+bAYLiL6!E9Fk!eL;kVEpq$|KQ9kk%$(fpW`I` zMWlRX-B>1QVoQHE@dN)V3_SQCx0fcyD-6tn~|DgY)Llc=L!q*k^aQV;-Mx_fJn{f9Yg1gL7vQelEu^WKVXpxun{}`?2T@7lGam zdaP|kf7HqGcGLyF1b8>_ZWG>(y14sOp1Tp zAAe5$FE{nyyK{QXpN0-#eM_*Bzwt%k)I#_d*88Y`k^f9jpz)@2|2atZoF4VBA*WUt zn3fRrSF8s!gVUxFePYbtre#u{``aG0ccH!Vw^gLyX)cq$Ie>Qydmf$6@eb5Qd-6v+ z@Y3Jl-_Y;OsD72bi2<_rJEU*AN0Iub@$AY!hf1KIPJx&6d9?l|TDFJ&XOg;=e~|w9 zCsKsRc;zoFG4+svSY(>r4R=HU7ue>C}RFu<00AiG!e-4yD&I@JlUU*k9bj@oXcw}9`x?S(y!u|pXAeH z^5r-YFU4o2_i=vH^g@b1ej7|i>o?6GoySX^@c44WpDF%c0UhG!%c@SJ4xrveH?jbK zpquHm>q#t{Fdp)=(eg{$*!m>%1LSX*Kl6znRZu^+1NM6Y0Llm8KNNpoA%~>)ACUY< zV(}gNggln`t{dWecAR*Q9GFZZ^{WRmWAS=up|(6=r^s!H@48O*d>o~#5A_B8t+>EC9I&-PM7rGIDd8R!-*o_A^W97*jWt7{g>=>WB-JI z^L)`E^GlM47nOemY!1FY{|}WvCEMP?a;C<^dgjDdLFChXJxjJxwBh~%c?r+gy9_@g zc?)}i=K~1;N#A8l=J~OM=g0Qkpd)$qKSr{hoKNH{Z&X1(v7V#| zdJXNNZcXJgi5bCU)WOgI`6u<4g5jcgdq!{^>P9?GUn4#Nbt66z>6Q_ngnA5bSr+4Iz_^AJ1)F_{Vf1yx(dsYFpL55^$NA(E@#TD#Ii9_Q z{BajcCI5y$jbm?Ckw2wU3xBEKF5<}-QKMp5FNRU&@nAaHFZFp;MLZ{cCt^EdjOU4o z)W3@72|+QNSPe!&{;<27p#ch=k6{0Gup3LQC_3%Wu|F1vn6y89llwbqC*m)b>sj(* zNiO(H59q0MViM^g`ybSnMe_xL|7nmr1LNbvzk0!scI<(VcqhD`z&Jkycmwlgmjg!y zC^oDYCo%pmVGn=(`{URzDc-)$&-avmZdo4e*ZKNE{>Inm(oM~cKEEyb3T?)z2!G`J31xrS6X%balYx!6FyBvoiunrpL)?62#u)jEd&Z6Pl`NjG zxZ~$5kSEJc@jaTa1P%Gd=zIn7)SVU0SB9a#Po9rt(fZ2cCHk}EBZ!x#e8hqAX1Kxc zC&)*##DY)pc~pLM=jJ2iKmQ@~k$I;0HG2O|*#(N+fA0@6J3)BLjv9p^~{2c&*rKkR2F)8{dK zeLjoN8+T5O?av?N`*VJN%Jc0v7lMAWVLyHlpEuYi^ZhvYkN-mZ@z{9}#>4(8c7BER zfAoHQ>a-ijw<+A;r^b(Oh#ym?^8J_EkB^RTrt_cN;M6IW_!c|=vF>j%e{W*HJ;fN` z3WF1Lh7YjB{*dNJ=7fo`kl24`WB;8sfj*xz?7yece$f!`oU|c{?Z2JQG4|g$P@!@= z@viDN_TTAPEy1AJucEyR_2_;S`Pu~Ic@yv%#`7rh|4hm^#r~7ehs@~ylkdODpHQOl zZt`U2MxFB2a|@wQmu}D}?4Pb%^l93UjGe#!0q6V01atn1{I-1Wt;ILwM~3eQq_LHB z*u;w=pQR;1?x*uRS%l&tf5ZBN&x`qauQ<;zoQEF)(M_C(AH{h%D0>p$$FWLNnq_~T zrenX)z6JvB&)9$7kUxAL4u25m>sj>qDgQiDrw_~%sSbWuTK3y~zZR9x$w)qazDPof z_0>)e!rKq0Q~&gFi|kK}o&V?EoV@1qgEaO<@FVuK=m6(G<=r2}@)LZ1V9HNEgF2+T zSMF)xo(Aq|;GPEVY2Z`W03Dtq)DGZV4y@-F*M9%uk0M`vxv%=SoBr+}eqK55uNpr( z|2L6?g=}`;)4i@*ci-HcqkVmh_1kNo8+req>mpy{sFc+Sf7zGQ+t-)EPXGAT-~TT1 zj-8d&uAXsJUtOE9x)!?;g!G@zrnsI@UmVpm`K3i(M>Y1~9N@1IKnbdiv!`6)5xR&6 z>R4UF-zA;?N%)>>3KGO#EfFk7OFVp3v99>-DVOKP9UqV6 zUlcG9uYaEF6NM)}UgTQ9LoxB%NBQ3z)4j4mJo>0Se#Ta7TCxsJM`0X7_M{?uHt00T zWAO)0#cfG8*)z(P5Seki8T76v3B5<{qjJ0Ax##8{)JvPQUd>|zRuv;Z0#;P7r|3D3#NY9iLL22> zG`3n%tSd&0r|W4@^&5W$inZUPTV-|`#*Js&sAqyM9$s02`WY>?VdPQTL<6I?6XS8F zr`9;H@XfC+NxRa^7#?=Q3k`>Yc!59a&~Sk9WmU&FT1@q)P*+S6JqFdO4x@?qgF4;v zr4M+pVWNM>QeUE(@g7UP+ESc$SnAc5IJk1Z?Rng<#4iBx9SCT>U9R{)D`KP zM$J+6*gnIi(4La5l$^eNK(g^8zR2Ca2 z*E4yqCC{UNjMf(;7)EB+k=-CM;P1tH=mS1xTBARUHS*IK^RtVZ@wor&xUt%Eth#ZA z-HYL34H-*kpA>J@`Mnr!Ud2VMQ1M}O+@U>Y{F5}x>aJT?e~K4|ot&m!K^CKSax&to zaZMKbgTy5oITo922{-0`Pr|)@=pSaT##`NQb^hOeH25?|w{jb=&)etF+WQ~6Zknly zqfZO1{KTWT?1xPG(5Uv&Jtv=o3DFgVmLC)={hd}f=_k(0XO$-&z1rN-iVONXdVk6J zH=pHw-p=+u@A@^(C*zJy#QIp&+cRp%=79&rV)hS8-4H0*uTUp zH2*WV)w)*U!ubNtCs9t@abOeY8AjR-pJB9EHMd-QmJ0wrvVlNnaN;2!WK>-mrM#Kv z|HOT^&n7o}`o)}=zWd_DR`$KQzxnM3+h5I^HzV&S(|>$0Ew!R0Wg&+A96bN08dAIQ zM|F9*utz0$!t;M#S;pAra{O&MUe|m64=IRjLDzWC_96H$=?%q(lc4BV>rhwPeH3th zFwd#3|2BvB!HP873H2p2C!}G4Z4qBT_u4E7p(l7xkPj zMT_RVi05==Dsc1~eNLBzDEcVDZQjsK=f`xG_8wVph<>0f0I`)p)M*nCul5_|uOY@aJZO2^jrw4Zx2PWz*l z_Nr%V`x!zZevcxM*ORjS2eMA%(|1j*?aA-a#VGy<

om+S9FSuk3gb9TUt$NJBYe znPO^u)lS*}aWuYBdqqd>iO$-d<~gar;--v|W1{rrh%2=xnjTS&zT-~@c%LOvh`d=M zuQaGE{106Xbpn4EUPnAp|E40_drUx!lL1HT3-MoUp~EBmdHI>4<9PP-!!hyqb!0#3 z@uJy$<5RJy zkNISKk7fL$vc2+OYELe0ZNC_k_vtv}e|-~(QQZ8UG3)rsk1El=7wtWk@dspk<$r2? z)lR{u0uZnLJ8|06l(4p^AB#fN@x#m}{a=>tCHv8fRsU>eH9S6zzr0CRD z%`bIT?Y0)JS=UiiRwPQv%Hl;WT@3}rEgK8g)NU(kS?gWj*tKy(O@2e``r7S2XMcd+*r7iUvit~J%c)u$) zELzr8-?}spJoxJuJzTr3c-6MX4K*DbH>`SMMeC~k#`4;(#^o(LFh(2rC|_5;uB&0u z(ry~BaeZZbam9|(`hs7uQxy~V5YH@0pq+g4Q5y>WeEAnuqIJ2q?t-}U9o zo@nSUY=*oo(A5+5C53su4aNTYRv+xMuCl&u^%E<}2FuqqdyCerEN)ws*U-AIseV&U zd&x#`ZLPm%X-i2{d+GYE)!yQgmCMRkr4>)P1b(zU4^uY#3bo0eDl8(Y_PLw^;k z^VdDmxV*AuNjP1<1qaUA~ z^GEKTU-}Y^lfNLJn@BCj+7mTQ+cjOAmYR(Rx#0IBl>u?hHQdj`y>XBE6fL+^)~4R? zc}_oU|MSHE=J;CbH=Hkuo{7GZ6NtB6W6;xBhYj?vtNZ27BjdiIzi5|rqWgiu;d)Zt zTj-_%epcadEmilWPSl)zSMw^{wmT6=PToAfEUF| zOLQ$aL8rR6USC36J9}RpNtn>`!Eq*-kV7duh9VX;@znzXY!sJTJHdJ6;Q$pMs1oma z&ue+tXcVW!d(O0==MiyK3!7nFg%tELe#~&W6H-821-)_SEs*WFWEggCN-heM1Xunq z^u^dWA(w)Q9>hC+shsFN5~%z>428y{UwIw)@6kRUJ?(Ge!5_H}{0D#&Kc~4J7*q2~ zyAJ;^An_vlYH5Pkr5rS7ym8-@^17u7)ck4&J@s>0!d2Tv60SC>rLzAV*-y1JnbPy$ zOaAe6DPGjPdQS5BiiE5A^t$Y4KpIfN*8`re21QS+QKy2Z_;UDWS^q}?X5Ye<=<_fR zl@sE^-o};SS0vmn;Yz*<=ui0$p(aZFBnv!G!lwYP+7jrC5<32Cp z)^UF+;ns0e<-DbFjq}AN;iKk@gi}9?7ZPBz0WObtxLk(~@aP}+0JB&QrsQ>VYR0}` zfE(p%l5nL5Mc*RfR(W5NaI3tfGUTXn^W?ZI3~=MPK?zsmD*9a#ZXNe~5^f!r@)jzT zyU>(Ux}&&8MZrD1##ocQD!7Up&s*RsJ`7mkDh>==;A-hr=bx&7wcgJLe^eB_N7`?> z1+Lb6zXh(=dnVHEj=K8hHgBMtVUN)0UE%F&YiN3m|KwyD9>|PecjUkEL_gQaZg%HE z9o03rHRER<>pMDbtyx=Ft6-?v*TKf#f2bVRq$>!LSy4AD4~fbs}8H%1cl;P z(Oc{PU5C jll{mQRC|>}@58{>v8l-_1#?frfM&9@k$H5HNWV1e*oD(85j0Bj$9j^n4 zaqsunqx;^g1_AcOhM}PQ-unN)s{U8i|5tVE_LrBfT&`;xQ$=I1vRF+)8O!XA-Fx`0 zhoyqj&3dW_Y;`}+cH90Bz0mgA8H^novSs(Zi`Q9O_S}EL^&DRRihDGjsA%bWZRZa{ zza7V}ez-voeNbr&U9+)6|Ifzyj&!g?pgr7^!3w{Z?dkupguVSMogJEy+EabTHo0%e zHj(-|+>_1<|F7=pA4WS}v%Ou1u}4lmGgxtM|KRpZ&kU|S^~~Tmblae=qz{>$&i;lkbI7W1Uo9Or8I5Xim#_!sq853-7)3)9};Ix5CZGFNXV8UIJZVD0JY!;I`xY2lviD z7OpsTJlx!ND%`jKt?_;@>G{tLhE5(2pU12D z(p%xVCr*URotMJr=br=3($JjNCkOkWi}G{F(D(6h==jNSh4WN6H2-4w>D9#Zl2GXQ z{=vB?o*WE;=kxQw9o{?tE${_BK=wYolIMIOR1RI9e+V?tTT0@$!=d-y3a4y25uS7I z0{FTR9yjyR&3oi$fcV~Y; zIkBfY`L*mI+0kQTg$~$A(|fAx;Zu_|&($-wNqvOdth2(X{ectqp9KFBKCoG1g>S$I zG>veR`i6AJW7O^lV}-8)mc!eKzJ3Tl`Jl2S^x=kL_?_qKwVkgYHTrKnfBe9v%~d{0 z{-=Cz0KWGseDj{~-Pyj^a|7QyGKTLRvG^YOr1HN3i~lKqy^#-I+xf##q-65j!w=|( z&i_;KRkxj`UHDb;)sdH%_5XS;>rch}I5Ue~9sZ$y^$qy-P&T`IrexBgp$6^HnZMPq z4ew+9G>@1z@aPKQBhRt^$Un3Gi?8Ze-z{PNXYXTI|D}NSpG#v`&nK~~DWEOFTpQk# z&_Dc7tbh0c?a+n)sb9VDE;}^*HwmHPZ?Quc8nC8LVErRMnhaQCX!rqk=wBZI?O!K_ z&TQEID9ttSeCXQF*GzLstnWge3*et*83wO3=6fU)WV{F&osfxSOP<88{)FZ# zl%Hf6c{MRK0(qc|{__*qp^MPT$jeC~ruD>?%fUQX=V1I~jrG&I@p0u3Lj?N`V%C_k z4?K~sni6qE`;RL}>&w_5yIJ~NPl(1R`Wc$1pO29ee~fEs)31D7sc|23WqPir%=3g& z7Z-(`UnmZdZOA5MC+hEA==Ci0=R*7pBzyX|qyKN>x(2Tb-1p$RAJ^k}{cqf#!u8*1 zeAw&V=Mq9T*j4$TZ9KdBF7)r3sfS#!&jt7`+3DA?0onXKY<{MI^$)>*WC!Kb@2_S3 z;S$!*^npjorpo@8zXe;tm*0aQUxLm44fc!q13Ps2Anac&QG#4z|+Ujfdwlr^T%YV2%zwX}`g8}OD3m4TDEXu3P&&$gzSX769 zix%fSlK=3My1G2bDdf5f#OP@927HZok01eGhcECrB!ZO!zLvYf+aQPct`MZrSKqa@ zZcD)1>J!$n0=x@M5KOZMS{edge-oc{{y=+YduO*F%?a1u*3s#0>&&|g+_%B&Z}kNl z?g}%*G+Y*|Ejdswo!o#%z+GTNC63u=IrOfX!pgQ>pz9e}1{LF4tlD*mJVIt7H@@6U zL++&^cUoE)$Mx@}A)mAC-L!0xb*zA!PebnfeT-V9*J6XUpnc1h4qs;;tE+2!ye^7a zwQNo&e) zH=~~7aGVrc%1eM}xMKbE2CCGX>Rkm*Hu=oTwAw(S@qiT z%GGttYS%4WUAnB4&98J9xaWHV4NdNjR-6Ys;&o$a;+4)<@cD;@*X`^{!0ihij65Na z?$gCRZkatJc7HR+Iku)$X5V6mUC{OsyXgBq8lAxdaWZr{o!MpGPL{r06s_B6bVFTd zZd0Aks*HH*lj=5fYnvKj`br!#)!8)roK$X8vMUw)Y>W@1xRPP34BS?Lw`r-&kM3P? z#45b(F3`YW_|ISV>B7rwS?icR#lgtCdu+IWSi-mCXsZ>ktw5mAICTA3Ua!gPN4S;} z0)_ly^lW3Scc&I%zXFmBG?d1&x`n>@B*}5IV>=Qf`~WTm6!lKFfDY+Y0exzKrPAR3 zs>?R^i)_}r&mLiy0U)Luoox5sL^~gMV|FS#=4LfJcA5K};kGqVbn?>ufXe zqW9PYKlh=6G83<}PIf3@x1$A%3>-*fLkn4M&~_hlKFT8cfh6|EK{hLxVEfIKH@EZp zP`14=h=V`i86!O83r>t=qn0ok+QxXZf=POm{zMu}4@^v$Nc|5UOk!t=o^kplq6a=V zXrD-DouYrD=hIBkqkrhbM&kIZKH&W4LOugM_(_xY0zc@F=(+wfIyK@4b)Dq%SA}iF zZ&Gj)tjO(~pzwk|%n0uaI42Rmsb?ZdY%q-#{u%m5oy+?s@6VZrn!g_Xq5r4$vIyD3 z?tndqD%t%ziIxiFyO_ql(mg)H%|ZBFC-kF_Utd#G!{vQ58}d%p$(JGy=m-7h1}9G@ ze;d#BFY4^{%evA}S=QI3d0|iV4|_PdlaoEbPho&foW#c$_1>Kc5%vK9VNB^9kcWioD*7J=37*IcqE_g9QZ{dOOx@b@S zXa`>U8~hvkof*}yvNth6_I`)-P4_5L-!z_G`R7mx^wTNuaz2mNzeLOS(Em(QxAG6t zKmSCE@EEV`g(ar`dUs~T^j+U((KpR+gTDEE&EWdZkotrFXGHZ~pN;jxWzo0G71MVm z>B|+H-+X?cz0^1Q&j>bXsm1>F@Mk(E2qkkzJ1P{AOr6{3|&#>R&?(u^-6Eg!m|+AEy0= zgZoc%bbpeS6Qw^#`wd4jw-@;j-=DZSeHPIJpBqeeMEwc*ZF<0+2ch z#!Dl5_*16Q-*SUlZe#nxU2ZqcH%`y}(dK}4gg;_@Y9IB-57XGQVUA>xGm$jnudo;N zqJ0wOJIJSP15biizP_coke|7u{Xzcpz;t~I=p!!7=i$oK^gy;gmFkl*fiOPC%gI8W z?>EsO?3+pM;-~olc?*M?nbCM=<%jt1#dz1}ht@9ybRrP`ZT9zrG@p#~0qc*+-+?#H z2jETq4!mJLKtH)bv;V{2r;~koz5{+Uf`<73c?lo&|BD*sQ}8boijObhONhw7m>YG{ z@7aZ@BVC{lNuKkxe{&?n^h^GcoM6#!CM{L|dJzx6FSMcNXC>x$0_sqBbp9u%$M9*u zC!ub{yKw5E+7mt!zYca|sTD=1{W;t-Sehi`I!C+$T1#d1AM zUM$H4f9U}|l}=0|J!Job+VW_=An-p8a%W(CeE3%{_|c9%@DcBX*Ap1$hX8M2zU*?~ zr~t)=_2MMP-zDtfuYZ3W`z6KO*ZKLL($6i+gZ(;RKgi$s`h41CPiuZ=%12^Ao5f|qBsZTLqA%BRQugn-DUvba4alVqp z^A&ged-x!^*AfCFjqWQ`&^!Lg0kt|wYdAvk_mV5;9(v*)lFy0I| z`27U=NS0XeDL#+NkM7)jg#70}WIi&_6u(CAzbU&wk^ArcL1rfi&p&W}U)J@?nDOd} zDP9@Q*Wo|66R)ECZ>(?T{q{n9zA)B)dlwf+txr5Y(~%l|UX(-e_>{=^=RBSvKT6+a=lK)%18M9H&nB?S~?nBEP%!cxKvfnBy15 zH|;l|Z&N(uc=P_lwBG>U6yJt<{{0E#*=3WzSmPN!Pm0+e{HEhPY2bj=5A28i%w+mJ zhOf_O@pOzUk58R;F z<6B{Hg3j;(me?QC{K%Xz5f&2r?`-V9vnJ5zbB6u*G}N=B(eRs(>cce8wV;> zZYSPV-p2kr9jhf66#G@QccC8Lk0M{2U_5UEKErq(MgE^j`KH)^^7)V%-GB1^H~AAv zG~P{~%-pC`zItvU^y$(K`h@+{b&Ebt`;oEp*FWHVznEaoUyKdTKbA;Lfe9M9L{Nmd0Km1YTt1tIe{dUve{lm{I#{E^} zN9X@0aq$cIQMqZPIQXdCuK4(<+-@Ovygs6Gff15bNroX& z43gPD70K+QJ&G)0;!M&+IfHaJ;%xLBG;?m#t(?czxh^=1j*3oBC@r@Q!{VCKHlSGd}b*jT?BL1LG zw|waX9&DKC-?7w}XlA^}Qm?YqX*}wmFA?a!R-9-Ph>w56I(fvcw)v#=MX9CsTkRuW zo15YNG+$B28trC!8mkX0K3<=n9XD3)#;O}09(oET)~KOVix1Eh}qNK4IYm%08F@X9P=?Xp?cF8*y{ z!`fx0f5*~(39@Eme;!M{%2G$Rdi}M_xY`Tf8!NuuC>oICNH6(5v+L7QpiaHcK%KfG zUDK#Jsvg^C*mN4DD0Q8^$n|*2&5cI6$XL3Gm#Yw`w~`CE!LC8aIzkyBu|gUl9goUl zFkr@jXJ*h!rY6jE+0B$BciHW?9{J>*`PO!myLmv@6JB)J{%D zTs5xALVu9BL?g#ylP%%K-0w-aw-5cp%++|S`>oFZ+m8mH#^_dVL&ffS^2E;#G_Z6J6dr;e@E{x zIsfLfywBU&-sfGvruk&tv58n8i+X!T?btl3GE^U`-;oY=~~H}^Nct+oBtta&r?elq>X2h&o^TT&Kc$j`y^f2tt0 z8-G-nmkWDTfG0fv=auD*Z7###mg9B3=l_s`$QE>s_iP`6|B~KNY&Z#uezgvDrQJsX z=Lhqg>iTbUcpv;|$V9Z1!$?jS9J*<6FyDB;1o2X(aR$B54`I)#_QYG#8 zW&2}>wsdRNJNXZ__gL6ylqPorM}<6(c(=AMCnUa}DK4@1ugLbf5~Or&ZBP5TC*!m~ zYH6=}wzi)k6yo{U1l;8?{$-)Sl?9?P;Eq3M_8Q7&#_NPmZ`!d!p$P)#y9^WPtZs5{1Z{ zCGtvx%EJH9)let!cj0x!6ZLN@vc1Oyv^W`Xw7wAk#TGg|!k?F)89I()iOY|{T_*oKK2Q zUDfh{?v~H-`Ubv>+ zTU4^DWPNc{L+kR+O|{i64gR(3TFQ4UZ&=g>*xDMxm$ocjv$3|KseZ%yyiK(gE4>>E zx3^SO)~wx8w!CHeV>M0w>a{H^7S%jfzN5T$QFY6v*5D@5DkmZdHAwZ(b9O}yWg zwTqT_)weFq0}uZCMGx0(D_*^=v9`KnW9{lER<^FrZ!D|nYFyE>17oy-kFxb;>$@5j zE$yc98aGt57nkoStuI)g*HX5%B06r_*45skrR~egwyu*jWgFYpx7HS{-*!`Ui}J^! zTfV6^*tDsj68dN@Bbm2sBHiV;moD16##_8Pe`D*`W!s9XyEkqq48$F?d`Im@@LgZF z{E3F{!e+?Z0$n{(Us9Oot1b4|xB6h8^%eDPYo1s|HdwK~*;}-BRdL(uyoT2GP4%0q z+eE^E#WlFy)7NSI`LjogCCUfd0X>>d5rDMxuYMS zobyNSonQJAER??>pW8_-#@Z9rOxrbGo0gi52f5()BMnx}HQdj`y>XBE6fL+^)}`L> zc}_oU|MSHE=J;CbH=Hkuo{7GZ6NtB6W6;xBhYj?vtNRtsBjdiIzi5|rqWgiu;d)Zt zTj-_%epcadEmilWPSl)zSMw^{wmT+=PToAfEUF| zOLQ$aL8rR6USC36J9}RpNtn>`!Eq*-kV7duh9VX;@znzXY!sIoJHdJ6;Q$pMs1oma z&+B;CXcVW!d(O0==MiyK3!7nFg%tELe#~&W6H-821-)_SEs*WFWEggCN-heM1Xunq z^u^dWA(w)Q9>hC+shsFN5~%z>428y{Uv(Y$@6kRUJ?(Ge!5_H}{0D#&Kc~4J7*q2~ zyAJ;^An_vlDrth(r5rS7ym8-@^17u7)ck4&J@s>0!d2Tv60SC>rLzAV*-w=;nbPy$ zOaAe6DPGjPdQS5BiiE5A^t$Y4KpIfNHvpcl21QS6P^W^Y_;UDWS^q}?X5Ye<=<_fR zl@sE^-o};SS0vmn;Yz*<=ui0$p(aZFBnv!G!lwYP+7jrC5r13ekS2od4Fl>$H@Oj2Dnk)OgVpOTw{#PyAJ%Y zgcCo;)$Dl*f5-wKFwh(A`?`c%`KPDoP$7P%h+F=?Si;q~$`_j?T#c*XhYj?`ai5oP z>$ty^aO=3Ka^BLo#`)rs@KN(c!l@s{3kk5<0GCHRT&}|gc=V5ZfLSaDQ}VhwHDg~e zz>RV>Nx0I3qHmFKtGq8sxK-X#8FJLPd2-y92DowDpoFV&75y#=w~qTg3Ac_*c?%WF zU1&-v-BDblqTn80W2{MD6WzSut#Y0uJm@bH8efOe{yme9>|PecjUkEL_gQaZg%HE z9o03rHRER<>pMDbtyx=Ft6-?v*TKf#f2bVRPb&zLSy4AD4~fbs}8H$1cl;P z(Oc{PU5C jll{mQRC|>}@58{>v8l(rQ~W64r}W3)(@1 zw(tZ)43<5diJg!^Fiv7QCdWD9et}$aXdirtVH{!%_Z;rId=o)D#25oc;}DrK^!-;= zt=@Z=~pFCn$Z->}2IVZ954Nn2(d#GQCdC!o1#lpT1zFLWVp_3ooxo>CE6aHVn zEDSw$1TweF^HLb~_icLM(5?dy9D4hwn+|Py{DDJ{x^Z~ zvw>sZ{n@}<$Np*H{>Ohi@Y7{44jfzl;=uFQy*Y67fnx(R9)D+`Ip>{$tV=HId%oqt zL%WtecqnJqrG0NL|JlF`kMAD1=D`;S?tkZ>2HyJaI|KLs>=S}uY*uZo@VXo-wYx&NhU9%oI zG!5mvFzfMw=YNa3W*r!KKIh$mtoPmq=JLMhA>Z}R1Bd#c&;7f14`jZFwzj-D(AV-N zu++BlzULdibLjqeHy*lg+U|i}yB{Cu8-a3Ko*#H?`MU$Vmc2WWIqk~6=h3EHH$8ah zzIT2)aCF9Q$PNtL|L#8xTmyZMe)lLaSM^PM;DJN;XQ8f5X#0bG1ATqZ4@{r=yMbv( z-X1vmIF+-Y@4grBKh*bp-$2tf&kamNTQgmSgVUydXJzJ_KaFOtFMKl7pIDi>w=hoq z-FrMWKY%`Sxf3f_iNwkaT%nWwo(p!7Ecsq){s#-iOTTx+MT)68E$o`;xCHZb@ z{(g7pDVK|6ivNZ?a4qHSeg`tG>7H@w{}Z&DHneEpF+r zgJBQ+(F_0e^;u5y-Rda*qylPRic^ z`7+=A;~(*f+x7}k`yR&D&b6VFSA!OAQF{T#*)rGQWsi&yw>|3;wbf|r9DD-n3U7GN zCq@;za@PK1Dpl(SZlpJ{lJmZfvff8o7lKcC=GOuHJj%OTm$wwMQhe;Tq3=skeyLxK z3cE68`CgPW3guHhgx@tcbTWHPV&$#ZiIcl-37x$4o4BsVbt$f4T$kYbbzCcP{VJ{t zaJ>Q7D{x(b>!r9}i|dWJR^eKP>q1;-i<8su%9*gVbJ7jr*WI@*{rPvtzm0yVbGcqx z3Y!&T+`qeU^h>{Yi7me!lgQ6cY+2cO`GUS%*H3)%1?W|S&qjQrqBkn4ANS6}_ZM8^ zsdiW4lj7ChOIj)}dNR8HqQT#0il_EpBcA&0--xH`z9vrA?VWfk5&E^a&NcB=a+>%k zoGEULz=j`UzRN_L_g@ows!;Q;K$&EZMqIy!PoP9RwX-YqcYnrw_hnqidV5zch2OmG zaZmr_#|vMc*qd1SW6&Qz^T^ge6(&}W@QIfKnTeJEoO))U|Jx&eaO0PE{-8BOymaS# zcfVYX{9pBo>jUqHwhnaX?)v3jb-y_1hW;7iyZ<;j`v?CApKAA}mC-5Uma@kFS4uBijA=U7^1#bc^>Vx+d<*8~w=l!|=PM72V^*@Vn#fb-(yY_lR8~ z`0rBq@1o?=Z{@is?waULNdJ8yr2KblhPeI*;juR?(YU(1?|yj)uuH!k`nv&-clrxo z%KGjF(@nf9ekG+TzZ+cK*vRE&-BtJUdzsIFKkKEgt&K}UCj)fd73gcM44ss` z1 z`3XA)Wn?4YyvOGDQNC={e>ZSGoo_~W!64O7dEfRVwr06RUskaHPy+p3?7m=7RKM-b z?oS=caJl;;%a3}+zWz&GBNA5$U*ak%FAL>`kvIP~rN{K{3)1D?Iu&zA{~r$#cEkwL zSM0uUFbm~m1qTn+QCTR*UdE(!85gF@p!$z`5-a@`$|lsNT(l_*Z7Oz;k+$VFxh^0b z#FiP|V+NU*kNH6I++))`tYM1RZSitOEtq!U`oi)FTMEk+eq0*ga%1SJ%&Qhm8xQ%6 z?s0=7G-hZWPPQhDfS0x$)|OioFB5ua zT@X5Xr7P>ZkBkfr?ER9c%^Vea>T0YtkD`psO!a;JGlkR&!%IJQ^)nIp(A*#<1Fa-%!tqcL=*&4Cf zdioBXr$djhOT4`oIHhyW+w04f&Uw(c5PB!j-vMfyPrN_*a_ChOdMe-(*HirP*cG5d z7QO1rplfXORiNustg#y96hf!Y$vX7H0v{$B7etq1?0lV4FKYOiVrzu(E<3;q=` z@Q*wB71g5lvM%gFvNQWah(89Z#L3q^$d$4c$H|$|C z56C^I3)jhrK@kV{RN}f?sClOjds>%Bm-YSmJ%8UvZ~OWVBj!Kx|9Sfcf9L6Y6LI5_|LyKOlCiY! zU$O=5hfltB@wI)gkH+_(xcXi?mHHm$tLYP{F6d2l^)D3jKaaA1g|e~lncnZ2D9a62 zi21L3Mo%wBoLM*m`yTY=i4`|su3L#Y?|RIAHGTbKt+3JjZ(f2IbKj6TsSYIz6e6 zSXbJ!oefWWinC$tW%^M6k6;HsuFv86b?AO38?!FoI7eG>`KN$CmvXT;Iwzd(9QEY- zxqWMp?~~Z)FJbeK&$Z1)U(Nl5Hu%A(tY_Rg*3k^t0(F z+mZitW0J=J`SYZ6Y%>Zwr}AiyIhS!_>bvJ0^F4NsIJ5k1pE6$teD;r+SN39VzVB0( zvF=mmTYHZAXfFFSHopP+K8ugBZC;0Mu;!nW4rS+vD?UfuC-L8Nu~F7J)@%CLqp$(a z|99;{oZv#d^U+7CF%*j+?kx>Hld4r?QN=_g{}be&_xSwxY~TJ=Y7Fi1SZ7>mJ`gqU zeC1azkteocU{Z1mmx~U!gPJB@H9i5{Ie|Zm5I*o5F-qhiWE`5-Hbh$+nma@dL`5Wa zXS#$Nv!UDV7M>9vm)qkK-fXz7PM%zW`Apoc?(0;rz%BkeYP3?|QT2V}_tn+CR`^4r zT=<~TctXzZXzYkZW06((#@BT%ik8=KM`LS!T+D85i#5z{y?u6UZBy%-Hj7@}sp;Ei zr@yRjYiW+NgW=Y_)^E#cCV%fjJUdzm(5 z%iPlNnz`Xy8d_SG%v}~~X>W_hsz6*T?Hq1fb!)@wSU3_3hijXf8|s%YuL##RMwdm_ zwROgBTz=iPWu-_Z;S#h{wd)JejE=@gw4wfs(~xLGM?>^>v`Y z$^7swosnpLG}45&Vyz3WZfg#&YM@BBz5(lB+v+p6hV=Z7|KCKoh_{$W@YC?r@y)#hhwUxz}94(d+79^fA7T$%MC3JEwkI!tm$Zo zm54Alei$Jt{J&xO;^ko_!JvyYH$^&z0r2o^k3?flv8J|G8I`2VpwbbSeW7B_Gc31a z8iXF40+{J?$;YX?k#N=q{0M_9#LwNtA322o8}Qx0_s}5-Qi-~VBHZ*Lep2(3nx6wz zJn}?CGxX(3$;auGk!}J%KhM%9ag7382-PR!Ye?hijD*r0@PoZ0(|pCnIgZ3z{Gb-Z zbUCaNmvbNZ9-RA_DTP#|ne?3!+S5hXkvZ!ViDD5*txpsOoa|T*f{YL)a-*eEW7T;YZhHo5nR|7ZHLdOL}y!cj&>Uip z2o3IFtgWrNV`dPuXG5f;Ay_(h_Pm+F8|gB8{^g>wv#Gg0*s-o7*3c4c>FkIF>l@ZI zwKfE6SFBieL-@wUD{r`F#dYDuH`Ok_ZqedJV%CaaS#VYahg3mz)D#J#YSNX?-EjZH z&~>f&4*1?V0fGB^Vm^Hz6%cS8ndm!@Jo-);O5S|y`?(zddDnR1i^zAGSJgL$G!kNh z2xW-llZ98)<3-Xhj=S7Uj}svWJys-rcKH|)a?lrwq}MLLK!hB0fe3)M>KiRidBpM3 zbp83_xPu-gj)5MPmgkWLZS+X-nggFJ4m#)&;(!C6BlbJ!Y>@60Zwpsn`9-VS*{ zB;0m+h6sVS^7}>7WtaOz0<aK>%e9{;lE_sV5iV`?o9h@|QMz?0(XyXj;xI z$lt5!axLGZ>5!(Any%I~yiw`{lPG;OeUql^G+n3Zc1<^GI-%)yP4{RzuIXM)Cp5iB z(;GFN)O3%g4{CZF)0mLRe!ZIR*Yr+JAJ_CAP1B@7_$M_TMBfmd)byTQtG!vSfH;Wd zmE_d^ESFdGy-B!bo)5F z+$%~76-~_}{p&PM_Q-MtMFr zucSoLy%IlG^nckd2Yr>D##E;1q{NRG{UMq#*#0R|L*vs%Ck?u!(kbsX=)8qadBUKN z8+6?wC;p&8hZZ~Kdo-==l@h@vN)DA_uaxM~G|P{J#*mY18OiIa6kgdkB@Tjy%S-~`NBVk7U-3|tOm6&3>n6t#`xINo3B>1EW|6_yWcn@~m&5$t)8JWdUc2vv zL4FS|84C}Lj>AoFnI_}kRk>pK9p03@0hk5E?p*QgI=cOdv=N``ub1qV%G*4W^xqO)K87?*1&H1(w#HL_GI=W5 z4SJ3khw{i5#D(AjzcudjQs1PK56b-8<6~2VU#{vC-SP3txhN0t+vENe+1#t;o1(%; z^|#j?&sO#OmSZ~6^^X)eap9%M7CG9Q~O9CpIyEseyvyP zKcL!&_TaHCYR?Afy8`%e)DqB2|DNcyv8XSR>QBk`Y>AE^gBIB2%f~FY$x(j>EA3$BU=Q^8Bzb^vQ_|H~{HCK>CZG?rZ%-bNN-7b)YCEJmLfDZ@`;ClE2jFitok! z-VjKIp931P7AoNU--~B?lMXt^i)h9s&-I2dSy}izZwL-!(fQt_gD&ufU;s-#(VKM8 zle~x?)AFa*2~d>0Xs-wQdA&)eoJK}k4m^3fomN4s&JQ_F5q5c=L1Xo!r0N46tDS?^ z+nTf-k4iY?a=38*u{>T0g8r!*7ou-S=#Qf#;lI+{inV`^=t27g_J66@;UD|SpMBnJ zNK?AMG2WEFY5KVil~w+Gd#?CzovHCs0T=m4k)p{$L{IT4np#ZsDzBnxK-lS^K^Gfz zi9weebjYBqE&5#SLE~4Z{cWF*J!()IwePVhs=YU9dX-zzbq3vN(Cr2tH|T^xZ#3v0 zgWhJ*raer$sxy#H;~oBzP5$y5%b(D{@42ehX?N59H+JhV7Q{^%V``mjIt#C>CFe&q4o9j}nFi_CR+q`>?;hBV~)7ydmyN%{5s zw>?@>LGC5X(|9?b_Z&l;d-wOq7{^;MS>7&{o_h@>G zNBQTInufnR`*-9J`mfDjsxtW$$eQnhXpT&n-#(K)dA>`cG;Pn%W}g+}%;RZ=jF%EL zzA3)J`pNN+pW+S7-xTlAf|V*qJb?C5{F0#Yq1XQ{@v&Kljc zjfzi59v6PfK5YC!_KQv*2meo`$5Y&w!gf%_Kghp5e!+C2d3^PBkM*U;4~-Z5{DAo2 zH=l8S=#EcN;}Lbze6stFYf_10M8IB%Cuh^#$w|tqQ@t#g@o7~m__hRKexi7FWJ)kS z1?`DWn1J{wJ^xf?Zj`a9%C-q4NBxYe4&{KIlXKJD1AKBYkE54C&8$zP>k} zA>(gAa+JRQRJAQX+LIyYZ;H27ez=~lANucz`ur4+`>1}*_f?&*;8i6`Sl>=i|50m@ z$SAVE%2J6|d{6fm_Ad@PD<#^r9Q{Y}e<8>*I{x1hoi}|N=u|+pXG=80FZUmMKhoXh z7kUMi>&@19NrrvD(cP87{w(dYBU)BQ`pf=AeBa&W#dD4_NqKjd52`Xf2lg+aiKUdk7(^5&n!keDBc={EBddhQ_!s!B-Ow0uM9aK$^9R-uP|dCq|)C|es?_A z7+(dB@pZ=bO8XV!(O|8giYMgwX|Iv{C#*NwX8V=D=6{f>{6Awfj_QG1b}Xln1Pa=v%i2lJ`cAMO3%cj!-_WiRA^rW5YQ_QCp5)j2Ovit=!xfcS0> z_18GrA2fb|$9is+C-)1m=1!SG2>8b{>r&c%mH+I%W2&{kgZ;@~gQ?WV>Gdb-9YOV_ z+PQrbN2JGxv|ru`>EG#iXT%5@@9F-9eq(6^)n2LWiSbq-`%~k$L?=#^<=>>rhrf;> zf4BS(`O=t$(>T{*m=)cQ}IYTr+HcV#-t?}-NjB5u_`WPFy* zbok%)@>!O@p}r?NMNm3yPf*s^6VJ`H_vaHGqG%!-Fn^np13x8?8k9FVAP##3Cb$JaU6$Nm3R+#Anif52@SIlo~+u>8xh{;KtJPI~>*>nHVwUSH*U zO!JfUmvdNONxyV_um49r4#JRkMMZQ~#9ag1ls_v-Zq z8Ki&RYab6-UuZnY`B;w!hdz*>eSR!5^zmBrk`2Cch`Va9~ zRc4K~A4Ry*RI+_yI}pC%tk#lz7OYuRO1i@gwGE zY;kTi=0^_-=b3^7O zvH;qTub_BWpSKL?{djpMIv7dKCx~Yq=Q9uE{sN`;BkF!^cYHE>U#DT(gYi%2H+M^W zVt&AQROiR~{@<49#a=uvqRt;k9>3&bkm>&48vhc}3Epq^#HSH$>wj;G>~nFte|?g^ zBu)Dz{iQS=kn}W7L%)I&A)zW158JKAvykeyhk(e@F3cg zrk8~L4mvwTCs0z3^OdU3UR(eDj?N1l{^C4uq4_A&@*mZ|RGjh$8NU|?tp1&Y@@D4H z4ct@>`_t^4i-=C2|18Ln{!*;u-SMkMAOrv<>5tQM==s%DT0X-er}IG!Z%T6h=#I}3 zp^%!lm{hKcDirIF$JN{*7 z{`ln%`j6{G<$?sMSAzA_BZ!{fW6?+N5;JFer-(dHFYJq3>UW8<5`PiVm@j-pUpg-^ zHBDRd55_0r3CPQ@$ST(I4DnWnSW5Pu5D22Y4Cnnd%r8ChoE*7c>-NxkH5vH!$`kTD z7V||;dOvC(@3pZJRG)LdJV)Fc&B?L%ucaU2OPpVB#eQ1aU&jZyUp6^iKg!$kn0-8~ z5ZiYe{{JKR9VI%CeJq#e)5+?52Kti!b>TTv34k2uxmBG(xBa{ZEq6Ncq#}J+}CFs+{kw`1Fl7)Mq=dlJ*P8^$A7D zcmVd!CI83sFUtN=JO?jnoS&k7&iHsU?46sd?46M3{g`j9_!#pm#jCb>*cq=%`I+M3 zk?Q%U&m|sK`jI}>Xb;E3Tce8>(fT9jd)VV_@i6++aUSHbC)Q*5!}7Gf(4T}?>y=vH zEc?QqXus2cL8t8r|A9V;uNK+;bxYK5tpGZC~Py4fyJD!Kberrx#^}ftc{*U!KB+J7; zpdI&*p)d58&o8L;c@F$B;Fs$Yjeppq>OMIcDSeKk{c-OYIX)?VhX1=Uf6FKG==lZE zu%{ePyr0R(^9K@t<%1al^Gj7{mwbK<`pNwY>6gd)p}+pn=~eBc@dkUjyWEPV^5ywo zfz^MG=TrsSWEqDL*Mh`xn$dF7EN@{uFfo`-Nx*br%xi zZS)83cTi8;*58Dqe_ufSB;kaY{ozyXAtq=y?x!uI@3MWMY41Z_NP`se(wMp*DJKQk zwF0Z>zkcU(_!H`L`0op(pRNC2m_yGqy2+o~Yw^^)952pwgXI;6kc zzI1<~Kgs@HoBf^kR`z$Zy%8T+{g;RP85ob~A79%39>e}4o#zP{-`#h*Q(_M^kc<3m zobTXH5~VNUgY{M3-&l+9>E{jbIOb7JOS$d-2G+yfcNAFXJyS;Fd0UTL`kQ>-_F~K@ zLC5n4SkES;&tLYhkn0m^z|Rvn`(rllw-;PX8#Z;n8PAW}*EiS?^M3*5m*+vCv7Q*` zKN+_3pUt$MW{^9m^B-E@ZTCZdO!q^)biXwr;{&~(UIV;!{^PUCM}O1$h4v^stq*v< zfjpMa7wCQn4b=&9zf9|o-&+5rzCP=I3d+NN%o_xr?u=m3fjuzmr*BYKg1ULu3a zdvOKk+rxnokV6GI-@^Yy5R{%jy!mmTC?+~7=M!%co+nG#LRD_{8v`kUnL`SGbzU!mqh@BBC%l_m&H=XH5;fCT;0@%iNOlNlgLik0cGN#EeuilT4d>);~n|vO9pX@&r(69Oj{yHE2i{>R{`_W(Xqto0} zgQTCEgZ`W$;}_YVFOq+bmG(}JRr|^9@p1GU1L^Y)JpU!nm(}y}G`~-m?`+BS5A|)0 z`yJ1p(|H`3Uyc{ZvEGgqy65vydvcfYCJB$^LBFllN|P1 z5cOO3fxj(?W;pDzC@%0u5>h2Th%e@Mjk?5HUcl-PnLpFff0Q5Zx2E$ih-Qx_eYt(^ zsM21>yU+*tQD}+6Bfcb>mJU_FPsJ;ez^s6Hz3RS2gho>F-zc0%srFHO=Ljl)Dv2qc zhCgpu=fnLm)S%;g#FI4q5^_8uKHPX8%@=s)jpR5VfqVwdKT1yZ6@#M0@g|jr8VQf_ zM)hF@O2_ZRm+#l*XUg*3fLrBL$567DkM`Tjmm28(!SLk=RQZQJG=JoiJ=J>fUtNn4 ze}||+Zj^rn^L>{-kEQcU%rEmnPp0bWejDP44eMxr`7nk0g=ox`_7mciqIVKa^U=qO zK0>s-T}Ao|djAUg(KlsZ^oOM{e!=676+}~aLH+^g%|z331bWaTUK#1ycn|f5uAk=5 zxX7m*?R9eg#QKZ#crqZZcksWB|44YPAKJI^7eupupr0igiL`=!X#JMo<$5^>{hLH? zO83a;>mB=L)L+5-7qZ_`%$H0jP#*lz@xBW7kLvvb8a(8$D1Y2>e5a)HqdoY;l=l9A zNJ`W4sPLrM3%TD3Ipp$wTN7A19>;u+8*=FHMm65IM6)iU`GxN%(ET5^exuc}7tY%P z^uQL!L-Khn^eEGGKC9*@;L-l>u3Ur85#(P;QvSX8q`Xi%KRvIOmzQK3=WqX4hnOS} zuzCoeA!*qm7M*9v^W*`dmRGIYrfSpn+kM9*$Mb-QkI&kEb-$F`<9yx??U(T#w_iTL zZ?n(2*n?aPN!u&muos`-u)T^Vh2YPsKC;)ONlB)ay^1V*_{d&Gmc1lB$+C~4<@0UQ zUOqhUFHaYsj`FuJguURv4tp^@#9ki5UXr%iOVO6SJkIkm`1>z;d{d9&J=5_teijspf0psugo$#$n?66r{^uBS z)BOJ!J&!nc>>iM^{TRQF{VnR>eaA)C^LuaP;=E_<6guxgj3fOK_tVvWB)uQXmhVro zzyBy2m_YM`-2d*IL+5$&{79ZZ(|i&2O~Cj`aQ*j2GvxEdV9EK!@q9n@!FhxZ|I~Q| z^2_^+GC%A$^m+e)d|yP?kM&`==l#?5qkP&w(|EJ?&wPH)`q-XV1m4^~Z^iS^lwY1V z%l7#*VJ_`ov>&%UkBR++`8*@+Z#|z0Ii80^Xl%vH*pFXC<*D}pHlzQa5btyU<9&dt zanARL9Ot>{Z^wBU^e>GGxgX;5Cdc_M{B2nrm7_kHRC}%S-9szH#!PfAs>GBmpLccm zFU3>ip4OSweDOlmM~^!rsr`BSyiVV**&d%D-$#)90qj5cd=Wkra6G^5t&{$T{Tu$A z3W}rqHHYQ>X#M;O-H+ak^CoCUX(K-HUwXeSL^LI|$6Sx|<8HTpUIF{}Ik3+e&f^Y` zbdQU><@?ue+E2s(hMvFqtn)W%Kfl~x%JcC3mYt z?JM~_|7NK=dPT+yGCz$7nIGpNz{~NLr2U=BPqdXk;M*QvES*oeX{yYn3cd zwl6hTod|sFzqr34pLaq2k?MR{A9v9CFzmIj zL&tZNAN}`-brdbjg#OLMe%Aval045|A#%EeJTGVu()}vL<2cWi=M5x(b+g!nx6j?M zCr+KR#F0!%dte_)qr58YXH`=6bI0AHCM3%j;zjHi*5#@C=x^2_9;@Q_YqY%ypEv{F z^e39%(*p?YorUp~PVk52>Ggp0dCx`dkrtNirTxA~*+bjwC!NCMuEBgl{*HL?8~363 zzu$%;dys#Q_{raDIW+_Fs{4*u@h9X}>-LBPYQB)|`}eQ!NpbKmT3>QVeQE!Iy~&<5 ze@cHse|&a(?*9wdcknM<->ZM&`uhLE^&R;O*LVCcT;Jf3`u=_A7a?9T#`|O*Z+iR% zndb3_0Q8LW8OcvOpVf(1jr!z%%r<|f*AK1_^`EgmlApG|r-#&c*7@Z#wJ#w09qm)+ zQMUQZ>p=t0&>kgEx96C#9yA)|+t$<5&6j7c&x4cnGuqE=&p7r6S)aW9W1pXW-rh5| zPs!8mGuOK!al~Oi#P;d>R><{0o-eEQ)lKUG#go#WJIwWUg${3)N(99dbU#-neSTQ8?x6b#iGNnE$0oisCXZ`WkPiQ?S+~!DuUdD?>d&MR z&;AF{s&VK4J#T^Y7C3K#^Ao%*)#p42l}pLl2FJDUd%zw=s4;D6tJ_aC0UcVyv$uEFE4 zKl^Oz?n=G>a@%_!e!b$6_gwv{k3V{MU@&#~ z@jHHV&&pj#n@{28zW;l0aQ)!OeRl_UpZvE!KGIT?O1<{t<~!f}^`HM4|J?E3ukRMF zY;V?}+*w!14YozQgbTMYT-muS-vO%3 zyR^XGQ}?(M@P0dI`0>!KXBkybFD_1hb;@@Rkz>BIr9EenKMDEg-}bY&(y3+gQ)pqW zJVRHOeJGDshO_sQwcb?Zhm7?Nk0}b*DuI5tl7)8-!T*b6z1us=Ebsqy$~~t1RjSWg zf3yAYEmfafe_ba2V7k0YvY*?O_dDc!vrPE`hrD7)e*Bt^sJx`Iz|zU(o&}vOdVLPP zbYsyKZhFbUqSw3WDOD;1J%JaPFyvF7gT2pn$REKaxZL zQ0|HeN3+}$-YAx=OE$iy*J@cxyq@A6MwkaA$@_ zq!DhNe3;twnWNKlqH}CJ?22l{kTFM3NQkrtAH6OFp)2Oa z;nxLwd-i4~1NQcyh(smnZ~Igf=Xvd)(~M)s+JbV1%=wk8FD;{wd-Zs%T{V~i-PzsQ zWLQk3-P9PBEO~d1n;zIl2h%Zi9je%&EKELSUJsDNUw{o zQDRicW$BGjW0X^GYyEf5wd7}3w4zQtD(dlLQ%1D!p610Vh#EKR5y#v8jK)pHYJZme zOemLe%IWGOzo@vpAgki?3xWFw#2qyk-lmJ}{Ij%e{X0bsNeCMSUHHK4KFr zy$Jx5Bo*?n{SGS+skLq_FiIXX@~e8_w4$P_HH+FiWIUoG`QJ~@lLm33Z!g}%~v`q=s$gF7LlT(=)P zi%d86Xx8Wf=b9{IH=p(yncwCEUVBV$i>>Ko%q#fhkcK{zCHK(DTw0!F`NaXm%wl0s zf-BvKnbYyIWZC7eO2ph}kh_uJ?LB>dPo>9{TkX0hWB7J?XwQ(5ZkKilt6g*=Xvvc- zx9ctS$~?VZ)GqfK3Mn(|NFE+1Lj-d{$$SpFZ}u8ND@S+0xo;j0N!SGut2 z>$W&(*!{6Je|gR_f2|)P@5jPott+LJAII}eqjel$X-AljB=CaD%tQ=?uVouw-OY1pqTN@R;R8zGh+BM z1kQ)aYpHK3;yQB~!y`hf44eki{_qTCz&XjA6yl)#?UEu;l;mT!_y@s%7kxk7@4(Mf z@FD;9u2n2Ag>2p=<{0%0@7|tU(pW0@N=?ysN>pFsGyEQWyDpFG52b5UZTh_H>(}M? z>-N^z>iaCep+4mI(Z3cH_y2Ipj-UUqphY|}{a^q5r=JDxU%%;x^}o1v%43fn-usho znECu~s7vdAk$*$|cpaJ;!LA0MbNLN*`t9@}{)k8&9!^NY`43j&6fKMg6INmHFHOkb zP!IVrHrQ`n_>FW;)9gaOp`Oq*4P~NxHBGxYqLZ2qYPw(3#hMn_fl>J-nht8ZT+<;< zhcsQM>1s_M)U+I1sE>b>{3b2O+s(54I!)6beWCgqHC?0Wc1^cyIt6A49$BuLB8WfQW6}7Vj)hKI1nsn695?9x$xivfDNZ_hv6J3|MZ&7jFM2O=(mhk{ zv{xkOC>rWQU+7n&XtGC^D=2E_+T}2Axs&GKJXd(sw?N5(M1H)ZT;Zg9;NzqpCHf8Z znx%^F1)b#@D}s0}*piPG2ft~j@f+%zPD=cEaS&q#3z)VyenWk=lRjwBHT8zvplcco zxk1;gG2{kabBiH2=$b|)SN6hhplcdUK>MM8lT)4q?S)L)6Mqu_RwZXT3EBsFug)J@ ztK>;d^KYp4!cM3A4Rt_eOhH%r8&`Ctztu)p`kOCwU55OW=vT=wZOJ)`=u?>MC^0{% zam&JrkC=fsvyzd6{-b2_pVo`N$v@_@@apgNDf&vyze?-FYy1Hv7R={ZW0Aqvr1~T8 zAQ;d5ou|RG{F7d%A%jQulr!h$lJ9piL48B=M-Acs+)d?-9m4+&_--7la(hD>QG~%k ze6~-G=F^Y!dGIHL%tXIp;1Tqf-<;_t@bmL5eM*{@{6eTc*#;$Fr8EcpVDC^pYrvmn z;|C>Lh}db$(c({iAvdADi>ang1FkZ`ZK&D^swzXJsN zY8Z~{M{45(c<&#k{$t-$`Ge{r%=)R%C@n?)ZsZRc_2+GqVxgNM^HcqtkNtN77{lf7)%n?f zss8+7^3O+ptVZqdABX+NZddgO)g}F}(H-Ijd|)8>QA%6){RB2Dmk<*U6u(v`=>aCwPWryazePqG?^&%xu1ubi)T zU3iePO+Ye!qO`0K4DGlQ+A-CMBJ4%|3;)vHilvg?ou5=h&*)X`Wp|~~85t!T7j%jtW zSC%(*v_xX78yo6_F?4B=m~E}WNc5J@mWI}tq%STZu1Yh54YG%USA7NcjgjU~b_SFe zX>PiuH7yU;!*)TH8UNbyOMwJaO;^EVR;`OQ*qsGdyLznZI6291bVMiq2TDZB1}hTW4#15II^ZW_L73+8ZeI*Gs-Z zEJw1{7i$ZyYJiuKqs=V~MkB4aGz7mstE_b1W%J7CUOKOQW-$HzGFqp$%co0v_86i*wbE~CD%45IhC2Nf3Bl`m>IUfq4&k}@u8nzMzYn{ z7-_9E*G@P>cR8_qrLUQ@tKDY8m1F3hOTd|#Hsuw*F^kU6|K4S;Z($5aGi;d{eFOYtOd;4oQpz# z=YGihU;KX`_+HLGTk|Xy3bPX30uA#+xIhl{TnRoUi@Xj;V`&8KZH$j zE^h0m`D{aFd5g~gTIXGIrB|GtkE!cnK&c#_I@#XiU;~%!9Dvu;LKH;!eq9CI`+p+v zk$fHJvOVwD<#XU>w}<{qYpkNY{uh)pG&)p};m#!#o+1%vQhXU{I0TAM5wtVyF#|U( zE`$E;!}c5}K@7*ICxeFLPdp93`ZV=UK27;|pN8Lra_A2!2k9UHiT(9Wd>>AqAD@Q5 z_B484ei}WepN9YJY07{0H01*>(`fjYZi1xYhB94X@PnE^9(;=DIo+hcOaMQhnYi*; zr}B9ylem}uL89{Q{A%b)iOLzRzRG`VG~X=eX3d{ulv8gjr@&Uu1{?nZ8^71aSAQ2@ z37oT`%yqZ^DhHmZ_y<#PxDzH zo;M0KpY6>2u{OMJqAX{k4R7ZM4Sb128dOmF6ncr*R z^M0h-v%|nMf2VKSJ&zf9<{vlktmg>>&-_6H&w8FR@XY5)pZ$~d^yncr;SSfssc;>%m;LEi> z{RW=-hYdWp>xhA8{&54(?K)xLna{y1`wzFvtAkpe-sfE$ncryOS|-*={`sp7{?M_#Um#HUrQ6hYfs8kKbMc&-|SRp4ZLY2A=s(82E%P|49SS z{HF~(w<~GjnZM7#^Srd*z%&1Vf#-G|H1N!S)xh(*`I>=ee!qcdJr5gr<{vTetmiQU z&-~*CzE|7xgn?)NDFe^zCU4GpK4iXEZ`yf2n+-hsNu7abe!YR`b-B^NGr!ruv)$SaJo94)p6wPl@XX&};JIBJ4LtKV8TdRs zzx5b+=09ZMSv`P3GyjBvXFUfEJo8T( zc-B+CE`)s&&u`54>cdB#54j%$n$LLV=NR}sJRcf(=8rY-{Te^fz%xH+;Ms0P2A=uF z2A=1EnFgNuB?kV0)@Po9Xa1E2Ui?IEH$}+6Gk=kRXFaP8JoA?sc-FJVz%#$rz_Xq= z8F=Q`8F*d?>kT~fn+<$W+o#>YGe2hFxm|Gs&-^X}&+SSWc;;^~@Ih_=jRv0iJqDiR zw1*5l^S2pz9!C!wc;@#S_}$t*I}AMYcN=&<*WF{_ng4`=Z`bSQlLnsoPaAmFGil(N zzt6zW)b`wO;FXFc-_Jo5_-e6`kptbu2Kv4LkjXBv3sml$}~bDn`` zez}2XJ+CzI%nup(MOyzw2A=uV2AY2euo`wTqu_ZxU#M-CWx<{vchyf1jgz%&0< z1J8DQ&A>Ci-@voo4jXvpA2IN3w_^sL`Ns`>jb4vV7&-?-d&vqDV;F&+sz_T5K2A=sv2A=IuY~Yzc)4=c1{-%FiJ{=D;|4IYTb_f~e zGk=kRXFF6Ic;+uN@N9<~1JC?g1J8E2$-p!JW&_W5s59`)uQ%{DdcW3a;F%vY@I0@^ z4LtL^3_Q=P2?Njk4F;a=u+hLXf0Kdt?o{!4kAY|Y!v>!FvDd&ee}{qRe%xu`nZMh> zvz~hlJoBG0@Z6728hGYEZQ!{dlLnso`waX+?f?4?Jo674c#iX5G4RZP)xfjeUNi8_ z?>F!~4<9!0%s*n_k8Ayp8F=QOFz_7b4;pyppEB^=FZ>V;&xg$S>W5=^{%qI!1T>%V z%+E3K+%I_sp7}uo&*P}bz%#$t!1FkoY2cY(V&J)5^9(%m%ME-$kFP5YJoBp!JnOm4 zz%#$bz_XsU2A=sh8F<$7W&_XsIs>1h^{+SZ%x^dFtY^%?Ge2(NSDNh=FH4j~RI8A2;x<=LrMP{6PcH zdY&@y%;$&pc)psb_4n$B`FOr!en9gX&+}4_foFc6fv?f?VS#~Xe!1q;^MU%ST(Gay z1U-*u=Z7>w&(RgCrCt8JO!HY!wr7pzbGw*dYv9?QHyL>5-)!L7o^=ME`Sk|=A$@Ms zXyBP2H}I@ymw{(~!oahh8w@=2HyU`>bCZE*evg6Y`&kbec;@#Sc-C`=foJ|s1J8Qy zHt@{fW8hiOCk#CEpEU4%{`RziXa0Tz&w3s(@XSAG;91XC3_SB+HSnzGYX+YA{RW=n zhrp(AM=KU?c%z6O`;EAvY<-DvRp^~1ltCSO0toHY3S{QIE6XMM!lhG=U;b4Pep zQ)?u;PRx!rG)HEyX=-f>N21ZlIuWK#ZdkUmHmpiZ7qk$Xt9KoB zHLbT-HN;SIL!?Ewt7%mftp-=MP0Vg>i#5!yy6!8pI%1L4Yd=-nV$G3`jbhhyz!y6alzmQroEG_oNiBe8IJNn~w< zdK+(vd_8mVyybJ3Tvy+ujIjiL4BJ)2@SfHx*p#@c4A{rPt&c2{@8mi7%5xeT!)xgE zQL${uS{YiQOn1u7@?1H*vY0t!P>O zXDwN(DhrCWh1ENE%0{q{YPup}06VFyWw+sywkQL_vD;h1Yt}TKeSCf?#UiV4 zUDqP7(KdRoO;x*0*=-qGx=1oVh3QzEa7RNd+(2&xqvP6I!g%{P))6jG&kM@Qr~>KM znpCgL{uQ`%ZBuhY{qp4%;o8RNvdFr&&e)C1ue-La6saW4lc=8iKFwtt-si#pLsH;` zV*TFh6k%F-q^m#?&+p(05z=&yUKdy#jG5~(llRs|EKzvoTaYxP6aIuyk6<$ zP3(m7RKH#}6d3>3=z4=XE@pv#yqrn#D85mdrVZNnDu{{a`ndo)j-^*cPM6=rn>3X_ zT>I<(Rn@@lr1q1IIY0A9qVd#zGsU(3Cf~%afZ>Q|+K=<+85y{KE_W$14yS)y>))e+ jq(8ld$N5?RUKq}-o3d>DAM`Kw4BD`y20c1!=Kucz_}vWY literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8d2a1751722203553aa1a28872dfbb04.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8d2a1751722203553aa1a28872dfbb04.o new file mode 100644 index 0000000000000000000000000000000000000000..88853ba1d65a503a60d5820bc7a6431b1cbd6dd1 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^KsL<8v#l3GG9V!kug$DzNevR{(WfPu zmLVrXWF2ELm@%`l&&C-t@nmQjGR{tB9L?rz|5e@G`{l9~%XCd+s%Y$07ON>JW0}3Ndk??$ zuvAdGSx?P?t?uX9ZrlH%586IEgRw(Hw(P!l@i}YDp8GGjp2O!~agU}G6)jz_?fgOL zx8vB=57+CV52|dTYc_W1|JhjIkq&kUw1;~#SmF1wJ^de+u(yAuvqLjdduqud~E;m@W}iV;lBAN!~0g93SZoJDm=39Quyil?}Z;~Jr_Q9^1X1%%tt~a z+xHJX{me6isq-HW&1v~g`24(M;k}oB8h+aOR=D~2#cOQ0(Zg$^7T+;)8b;NJPi z!j-3vhnw3@h5Pou6&`u$V)*>_i=bH?s&F0{JU{=*!M&@{&%W=3=d@z{lW&D9&Rq;| zyF}ytaVRt&Lc7{ofSsy51g?7B>0!`fz28#d;>n9 zX@r~9H>5irqjpCaD|`*G9NtFs^+Wi{2UR7Z57!sN?>twp?R@>H(SPIl;|DfvuJTFp zKjnJ^@V!^zoA-S0&i1{Y8~EOlF?{cc#rMc3mH!P`{7?DojePLh&L4&%C6nJCen3BT z{-27ky6r6O!mo<2j=Wsf|LZlZKNa)i%q(_w_=oz{H{jPp+3f0>l1Yb#8ni=a{#L&> zypQ$MJYw3wqsxJhJjePY|IGR?zN%k+w}kbdy^me}mqo1qTpGK2K8amT0c{cH+VGx) z{^5UO{lgDvhc5h2{py8x*`eXTNeB&piygYqfHidj>mT{iWWW+b!w;}S|M~!E|2ipj zX8rC*X|93iL)Uh`W|~W4eHZdv0RJS*FnFag-y@kI<3-5mgiIt`@+5ZkCp1?fpHVKB z%&vJL&-uSC>mPx96;iGd@H_&ZN5C`5Nb+3(pX1QK7cx2jP7A#UIYwSh2#r9F_aKK+ zev)D2)x^*U&K1NFXF|MUezw&XF#(m6{>A9LR&l5^r zToiJCp*TdgA)An$sK0li*R#-{3-L3M?CIZ*{=bRqYJ4hj--GLZT#w`Pzj1#G*MFn& zVXt?eO9Ke0+w`sbKG5a_yd9v7?%#PbHi@df-A$0b?giMlb zbG^7Z^Pk04-_8;}Yza#MM^lJ$K}S;8Q)7@#h{u%NCWKd&x7FE4KqZtDsP z7Uw;Z|L~H!x;)4!IZ!U0+<->FU0_2cj@f59^sbr0%C=mf>ls)E72{g0+I5ILLS`a2zT8Vg z?xi7jT3Q&#_3x!2pR?@Uv}}=etbm(OL+<>2j9R4EV1u=wean^(UuPbxt806_E{a)M zwz5uO>hiEvYxZ{V=pDsVOXM$iRZe!J$TSB?8+OaqvXsLUtzh=296o*&d;>P?Hojvw zqn_e$oD^EhOMqv%V*T{w2!eXBG6T0BNe?n!b^`7`k`{$aF#EcsS3@iM?*eYa`GvJa zP;w=Z?x}$9U35)L-QCkOn>ix8duBWMEn@_qXYEAR70>ps10dSkPtzd?Rq;8D57k$# zt0=7~a+hHLk8>1J=#Yd+AI4`3f&868_+m*{qt{KR3(YOw&gS+ucV~NhOUFDnX134U z;dAFNTCjMYdmUXC6fR}OUCk|x?vCz`PG76LwX37k-RRrW+~#wyt*TnFrfywX&6)|Ra*Eh}a7tK5s+^SyzFCU-|G&I2Ctx-m5IO6M#1{=>rScJ?IT_Js~co{&fP z>Ea%@%$^auznSA4TT?2tZ?VHJX#0p=^nD+V&ftMK8M>U#?6PhrOJ6RE)@?Mpp{_Hx zsZM8AMm+UNbsM_1O^q;p1&*2OY?^&eDz_=wm5O~f#)nZ{$*@%hZY#mtv{dFt_bxbM zH9mG1Xkak>;~)EU;bpe0walL4VC3CBHrzif;oEVv)r!wnAW&!=x_&IL*W~phTuTXo zLVhuNwlUVbQ;V=)0Z9fLN@H2wLSKB6fn_ zWgGiNHtXGIkFd)C5YvrLwtH`)osYXQJCz-CvmbQ}8Q4?6qrcuDCXxiv_+{NX+f2OZ zJvPD5eW;+!#OJJ&9SYd(Xu%=_2h!NkLY5n}-Itd>n?>{kN$ib-Y*sMA_M0hhZs+x( zY+IAhdyj1j=%Z?&VMfCGth&dG-)sJgZ_w~>o22IBYsfVNj`sd*hc&& z1t-CZ+`b74FX+RJ@UDP!67idQCX&Pk(^%o3p?}o5yl?XUoN1`}>(L+je`+s_kUi`U z*mJ0g-M^D)sX)GqY3wWA<0IT0gwJ(CKl=D}wY9Ze-Z!%$?_`~PDdK>B(0^`l@?`S2 z@m&9+&Q8CqEB%ybeO;Ot_C)`%hm$)w*#rC(2H3<&e0)*w-I)+!9{@l;0RPJf*dpYS zkwpSu)~&J2#DFIJV|hSNqbk!r=KQzd}ILp52e{t z$)B-M`Zq&Az-I(=XVSU}|Kja)gD}T%tu~qD8_Z_qFB}Gz3dTQA^bgLw5{YOr`Z-S0 zUqs4R){SL?Cbslv6F=~;!oY(Ma(iiFyu!e&Ib0rUe<}_0b6$j_@gMD^)JLd8EcK`HL_LojJGdOn^;pcMvLiS`gn@g%qydR6sa1rS3 zpvT%q^hcc>Z%19=OMrI+?>6Dxs0(}<@Y8_L6Zm1#ej4fm|5e~Kf%gdf>@?ngCh7v8 ztnKc00gpwL#%~gM7wQ6^pbL36OMJHEf3w7Ez!U#|fv;Z(|9bF2l8^g8`B%_pf6mW4HoZLSc}DAMB(jIwKNI$i7SI=|I@ufk zf%%-v=PS+UU!<|(7c@am{tJBh3qO<1MSXOCW_tK*F4>d&E5;-L+Ds&JK4b)E&ZPLq z{qg72|8i6Ry*sDJ{AuU_*0%&3`5RvpPA!CgVZD#~7x~Zh1R8HT_n(7g&*@SB8ggod zfoTa*f5mz*GdOJ;(I>|IZCWPPxxei}dl%X(e_KWRo#rz6n*(^au;tP@J2PVXu5YvGo94Gc-+aDiaD8V;{lWh;qWZ4S#(Lqh=-cIr>AQ;b z<%-R3K0nZ2>KpxM1feYm5Eo3g@Za}H8EVI1SUvxbfDq`hXvrY1irLW$d$Oq1# z39Vl_e**b!27e>@9TtDH^Z02e`A^b%W*7Mbtq=9IUW)cAzRe6eIK3n0Zw{^x^0%{^ zKAJz_t$aPk{96wDi6Z8oH6D_EPZNP$zYBxo$CLf}_=p$f$+^6y;z93DEd45e`AI%K zCSQ&d@lt$OdLQREO)sSQHz9pbR!G! z2fCR~yPm|N3F9F@8!f-2jjc~IKS2J5`7@vRQ3ds5J7B*j0HAyT{zLKi6>>;w{{hK= zBo^PHPsn45@46wrXUB=>$brcuQonj2GZwFh7HZ1_c8c7F_^#_@Umo9)|G=N|)iL>V zPZF)K<;h9_VNbLFk^XJQ_-sG61NnxMub%Q9hx1q0KAb4>9kO3agq_8J+<(bFH}+5X zH_sOxGQT8wcv1OBz~8cz$fp4LXu%|6?TEN&bZPke`po=L@g9$oWLR@ef^~lb8`qMjZ?dkbhEtDHtw_w`T;$p>D*}^flrWP&eWek!~6BNvOy0mi3IU zZ+4fACu90|6~lVX{lmYR_9yUbrAqxT5WlRxcp@bwYn zr4c>+Dbwh0xxp;Av3=n#x0~i0r|14?bHF;nA2B|)kNV?>Y3$iBN3zJ7NE-21*b92m zK8f-jL;Uw*yzBEr>lXq#5eWY_`};wfPsaIx^~dDzz?_XI$E>MRg&w1LvITB*}CI3iHu;@3FmMVX}hzH;o+EDYe3iCSwbtpVK{}a<=_%z^? zP&eXTIQ3BN37?L-5uY)^jGusd3=es^zjA#*UaGsmfzXc)^Er2ne4I}%5ns+%nd8|@ z$RBsHRPt~5(>V5K75P&tweXkv?INCh5j84?^M85*F_`3Ux32fMM_{CrR8=a%Kcex0u$QcKW55zD4+3@z?6Sb{+|3S6BZKsx}AI`?^DfZ{G5BGU!KoI z{V$f!@bihYY4#iDFNkl~&0qc${)hPXS?4dK^A$((D_*3ZA|Jtgb2-Q#`S}X`)10p? z1m3tl(|TpiR}AYf;zbtLYb+-s-a~(;^&IQ7Y5hh$3+r{19__In#r9i6nt8p(d?kEz zy&f}P;q%*)uh3?kittCipHTLPJ#qe+IT_f93-kTdrSJC}9);IHhdm+AG7;C@1i;JVyCmx^aNR7TP%At62AD{1&Cql}hG?ytp zVf{4iueVTqO62=<9?y^;rSG!y{0aMkH1-Da*IfyT6wi|;ra34dDh%2wTqp2+7wexm zFFLZG#fxW%7wP}b`)^bHLO!Jj__q(y?;i*sjc4FL9r29zLy=68-`#pVGwnCb@eAXd z_8ZW*DV}k>dH-SBZvbzKZ^JzQ{)F-DvdLep@eJQ5#q1A$({Y|Oa6sw@_QQT=GJPMz z*XOhNzH#To*#7)MzCY*Zr##<&b0O#_8}{P|@qL4RGT)DL|M)MoACH~)U_9)fV&_*_ z|3~k~r%t=3Mk*`fKo;LxXVLXo_|Ieg+Q|v$ae8`OMKl%Qf{0Su* z?yCEV@;_LA@O?2q?-l16hV$?vAi9b3@S`{n2W3y<`#4r* z({$|j+1Eh8{Tchu8}f(m!{HC&d_9Z4Kjq&?>hy(qBGtj~O3Qwm@7JR8IT^{v&lgEZ zvA){LL3sP&bn2hJZjt?IvGf1Do0Hdkevrn#2!6zV79HUHr@Z^4Sbl=<4@~*VXHbV! z_sTsD+|$544cybfJq>*78lb~-gxUf8%7OL#;@a;&{88kqFZWgdcGKVe!_Of z=l>>hu#nB}d%D+E>+YMIbF{CIv3`5)b0hEnb6wZ@xLR@Y)Tf{^~x*%a6F>4&45CV#ZZ>!`*aoCEw70w_VXarTreJVF=o zKpm@V_&>|weVjApci7%CwntlvK(5s@(8r3N<4n|84FERHBNFc@(t|r~Tq4?22B2y4 zcz8Z8u@#<;?Y0Vx_aMm6>P5b?yvxbhkpp}y!N z+Hu~-S$RV<7Q^pJvEnJ+(U&+2uCUGcJ_)~5O+kX#t0jWvXo-i9D%KUhJ>~MexZ~rI z{EGqx;`PsSeWLKh$BSGGcqk@b`zZgLW4c#1h({ll$IsYmO-t6H=_rgt$evV0&jy_) zc`W|Gskkl4CVNKN5+XBhH-p~wB%$}H+%!@gd{k~%e0)@Hw~#wtA5po$2+67>!;mNj z$?TtsWcJY>MV2seCTXIaLAo1pHhK=4Ik)Ln&g1G_7o0^$MJGnv3tbnLx243Ju|iM9 zXrnRfXw`rk^f(Gn+>j1d>-x$EG56fugL-Lm)~k7Jz^Y>8N5G2e^%OnFk@!2lQfQ;R zi^f(figm?^@pL^6s($0IK(Y3FbgRrR!?^Km8}&@k#ltHrP(P!kHjF$5g^{-92` zeCY!oY?$cZvDBAnX1vEzueQ`_JnEk>5$Jz|IMF5$AOD7R@`zh)^GWH8QcLZ(+DE)L zH^cjBzM_sb+RgMdRv%V;ygolWZmismRX5JWUKxY9mkGBbKfj@0#cQu72~Djr+neU4 zo)n#zZoz=HeLQVjX+H0Dqt5g=G04h)JbIM}NFDo;mYUZtbNwmcm085vWwWeZ{MW*U zwaZNZj-~w)WX;C@JeGR3rH*X%`fHbQwHLlOR(!irG$6;3Uh;ir*QcY(11hw78ue^L zo$yrW%GAcCBEzE{fK8hLHdbqTjtjBbvzv6RAuwa5$Di1)fO0t5t6TMgI`ujOb?S&R}981VOEJ@f$|Gp*5|#Txl(jQQC`&3N2@cHCI)Iab{` z!|ug!v4)JLvrmdQ>ik{|H?QI%R;c(eI_}UOGyX}MWp&rBt3SmH!%j}qt{{t1J2@F~ z)wm`L{XyaqjU0tlCKN@@*qg%O+*XQkXXzl$ET{q3t z#L=gPR(|5qTlPbyd}vhr=$@0$!G!1vLdy?|mHtkvoAeWB<+I8Yk6vx=XvGEn9lgKg z{F~46K5u7xpLhM5=96*9CSrXo>g^e|WAng+VljIUGe$i%Yv0!ORi)T#Gk*51U4i$} zVK{c%s*E`T|E|O%zQdv;S<5-P@NpAR*VZ2x2Zk3Jda5*Yf|HQs-Z;*EZHX6#?$ z6`KDyx7E5<;llX>%_mV#+i_qM=NU%Y4WD7OST(m?dzK3TzOsQpXK>;nA7oTr8l}9M z=l{fgw$COvd-}zkm%jVr#8&pbxxe}C2HRiFnl~fwC)0m?FfFyBC1oLo{2V<0ry5eb z@t^AQa$%23@Pz08yt0h3&E@#N<@j9h`9Gu}vISk^J==%izob7XHk<@Se_DsS((a>x z^MiR#b^Skccpv;|$V9Z1!$GN(v$g#Up%A}E5yAK!qxM8+ZBO%@RA6ya#>g>IdUC{-+7nHWs7C+dzYOp`OQH~Y zvqWBLP+9nYbT!lo{9Sk*@kITbifr#O0WD4j9IY?Jf3bxQkMQT^XNHdB+0PHh#Q(1& z`$>-%?autMxGuh5T(fW?`AtNq|0e)xd=wLE;zj|aj*UgqMnP-Qeovh7zarb`N=h1^ zibZ|QC);~0;~$mnmH$$Ea%pS(#hAQL#~J_Yn?Q`>=6@Npj<5Wv6774@-eVbmK(<%@ zr^Z+96nrWG@!G!=r#(#xYkPWG6rzqFW;W^nvTQHek6x_$UxW7lHIDqM{bkt_Zp!U% z9iM!Hbh^1-5?U^qazJ%e?FoFl?RXTI^V2e1biS&2YaEYwRrA8yPSriqL5%G<&#sjF(YwP?+{j-s+6QA$=87qoOWEGllr&P^L?S{nRo*0xmaSk_R`1lXEd!k4xzUA=KbWmEn7b$OdMRIc!@ zFWlZzSyj7cNBOdrWslW1`D@m+EH9{itYSyShJu=wO|8rF>z9kR#Vt!)>NgbU`8M%> zS8OO))>YrSG!H!Z>kA&P-B!G6TjPeBj*S~uJ+Y#7Reoc6ZCB&+mK_+Q4SbZZD__^u zP_VR{#%o+(*Ei>jcH)^d`0%O=uYetT)b*45tPRrwoRx0Y=ys_EXizAzAX%!(ZwHiGZ^ z@?}pnbQd;5-WKTUiTaYlJl}?5e|@VD_E}e1-?sXRm1KkE>zchqYgQJwt;%a?UDs5< zsiwVTqqnx!U$eBOq^Z4h{nl!4ammVM<*V{Hm6vsGY;EbbSNa=U*L6dG z6|3{tJ<+(lvSs7?0@%AIe`DL4+NMpyUfYT*x1jI5#toHSR=aph{7-CbUi_%?H+8LR zZD|hFZC}z@;LU%eF#nN)f`Yt7g@uc}-h6Mtk|mE6G;XP{&s$t4=o*^0cpH4zd&SCH z{%#d|s|wzBayu2g8s)mLNXcKcc=6&zd3j4X%`$IGhp$fjEvdx|WqjV&ykH(iaAg;NF`+2xG?lGUD1((X& z)cZZp>4)uqp7`G!UrYUl^F`4!(Km7e@n_c<^fcCC1O4mje!26=xNqn$+GU;SexPu; zo>cc1x@mx)RXAKr)qSbc^_>0hy_jj^gsX`#iiCxa2|O$K*a~D z#CzWJTHZAp#VPTgGcD+OL>$$^W*Ao?1$~SkGhFV36cATIZ`^qcWIHYyhMk*|i^3$q zmH!KUG4@T!rC_25@t1y7PV^oLRDK_ZLgUe|ybk>LXdjQB_BZk1k6Z`-1Hg%&)7%b> zsd=SchyNFlcoBWIG{Ngq4jMDwxNk~%-O>bVel>%h`nfFOs%?RUt4(UD>_12LQ!P!V z^!)dde>`1^7d5Y*lYG7+;c7m;F8djf22}9%fTycL(bHDgdG%;F^$vYa?2rrd#QvZrKB3&Wj)_yif_;gD@N?+8E@sy~?&_6xrM3v_aa3lS( z>%gxV;6{0KrQWT4&X#biyaUoeR(ao$aBDw5lW?oNzcln?i67%?_Pm5YWPuMD=#BP$UBa#W(^GV)5IwpjjP~?4fMuw zpO$s_M-qN_n`QnoBQS(K@sUO7)39#7!mq$EYuEPd+^gs6ivsezMYqR1~~N z+Hbi9uGV|M1+Lb6CerSXy87leZ=joDkI?8{;q7W`XnKslaG_wl{XQVECKd{c#(#w0j%tI@{|SIb%W~ zuqXHwRUwN}!e#9NMR2QQvE0U+u$ecowWF@X?`s(SUeQ2%qqoz`7Vy8qA%&GzK$239 z5+n;6Iy&`?D+y+hyZDi+vJ_;(M2h)lIi zJzbFr>QtG3D5PAbRh7{`pI|aAh{8Sc?zD_!3Zdeas?Wj=rCc5bkXQc>ar3;)$LPB? zigz*`M{}^h2av`$=GN5GC4;YrDu$xQSJ&rcd+OU*s#WlAG(uzJD=49fC#w#t+XRK; zSJ7MR|6PuMz4jkj5Q=ZrUpj=h+JtZ`0*b2iLC<6#xJL literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8d38d65c9e1a4e267b294dbf112f052f.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8d38d65c9e1a4e267b294dbf112f052f.o new file mode 100644 index 0000000000000000000000000000000000000000..474759cbd27b26f803f0a856811e2fa886703372 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY5sR!clAVJtPWFc0&P1hQc!o^8Dlk^u>Ucx`4yOKOlzOX|~- zOv{iHA+nA!7|htaan8nBGVx?-88XgJW*p7tY~+m(aU9HGO*ZS3$vGi1!$^QVxZ`zD zV%+=v_2|C$szHD~@nb3IzPJAWud4r5_5W4fy8Y$K)hl&PW2$KEB^IwKC}Y|E@p})y z^{{kMx>;ZSkgf3-d2ZYPrWe{iHJh;`BeuMOx9~b=%UkfzxSq%BUvQ76(-bXTukQX~ z_;*v-`1_mm@O!nk@KqZ-@_%h?;Aj^+0@`Q#vRKLY^E`v^m$Ns1t+OMu)BEbr+GY%l z*rrim&-7)olK-Q721n6O*KBVzV(hU~PY&0dKRCSeohOG^pMG-qTe@c)v`^u6Wd6zT zL%S3 zkL^4-{KS(_4yP}=KRn<6-N=Q7$0PgS`B~%%=j)NS6PF?btKR`#NjQAy(D1`24i4{M zbUadX`b4Cy<8)-;;OmjG`z}Q;?7Re;W#MY)q2UXQ9vj}j7X2LfZe)Ht#y|CXr271& z$iwf@cz+rWFT!|h522qEkp=DFiHreXbN*Ch{>k&;o%;Ec@YsPv!%r-FayWeIMC1Zq zZSTAuS#a`Xq}usTVJQyA${7Ue8;AEt2<$1LGGqgJ}Tn*U%1M3@w2w+1BBbAvyzzoffM$h4*wE9uuWqn zufhj3jd0TkMs&wR)b1!_B`*V(&)bN;egr>xueLn={-!ecooD>&?pKZ({Wp<6zGu@G zD4!(%Q@%F@-+KwZdE56s+rHO#9p5`TiSHe?_#XMB^1mUA|0#dHo)2E#{iATSe8wB2 z_vlA1{A1a;+s-mB{<>^@?1jp~->heY>6jm9=d$t9AL-+-!mmg2*!bD<=|@JIwIgT$ zM!!0GfDO_-V%pGytALL_%?6|Y#0D?Dq>sN<&IZrj$;SVADH}YW!NxD7u<8dpuSG51IcA~zF{jnRRFYttEe4?MNc?S6yX^F?Unvr|u!&;5|m@C^ep0>~v zPG43Uc7CBOOtvALke#T%x1iT^(4PzOGnncb+=>3bh3h)JYH;6&>p@&k;Pt<7e;U_+ zrSV~}x1LT4+hAAaf3`c=_*>AwYmOdv!9Ew^w`8Z^zy@UV3$XdwrEG8n_9HtepMG~e z8;q2*L8cEqNH$gWzx-|33cma{{P-Q%>|bHOxIeHXmk-0P20OOq3Vo2Depp)?zM4{Y zh3t1*A7*_#3+I6!^B29{Y`yO7hQHUCi(PI;Cn!48Gz#z`Str;kyd3C!C$1j6Qt--T zE*f@1+35>*`21b04wwSM>X>iv6?E!B$TiopnbuSj8H8%Hj`@LOVjb6Whhqt@c7i?_y zc6B#OY`3qYt1}qr4z?{V-qG%B_xjs*bQIm+S=9J%i@^YmMJ2_J%a#^578Mp2;(~w0 z%L*STx_^0NVXatRcrHh@#r1HQ%mG8pR1hgdXZ^9kT&d=tz`v=CtJbn&pG^#3GmI>tlRjG z-Hdvi!*Nn*DbE9*<%;*ymoEsKz{+ggdL%u_c-alO`)EcCF2U>@l3oq1=)VWJ4d)lu z5=AMHK)R;_zW2~IBYkgQ-#q4s?(LiB;J1tse4(`ySyw#UzX5<~Yd=GWAXLTc8N8^z zc4Ku#b*Z}?`+uCHh(d=XM0!77+X)ot4#5}8ds@71I$db_- zx69`)TDo}ILia|xEG}8W%6i)TE$*(~u5MquyS=BY+uh>Z-qzuBZ>X(Zy}of{W&Qf< z+O>_9n>SRht*ESEi)!6V-HW`z=2mxCJI(_h@VYTH@k-|_`254d>rVC<;PxdBMxIbe z_nG1zx6GauzrT^=99wHTvv0S5x}mN! zx2aBNRYpAZNp%~#wcQnE`f40AHQF@${B&+pvMUw)Jd6*cxKd%OEZo+Bw^`{dfbLyz z#5%m}F3`YW_|ISVT;XN5oDIyL=3wOAeKy?RFX20JwAGH+4j@oy9J+oeuUF;u16(Tz zfkJ*Uakeqmzgvs4Ujs=78p&Wey+U7nlH@zt@m!B0MV*~_L09^z%K3&gFYJl_VGk#Fa$v2$Gs=odVuv9Sq1){%q&Xs6Xi__0{ zlK!GnzRF%K6Ev}@3>Td@9UEeL<96 zlJ%x{v^k-L^J%<=#2@Ulx7V?d&$qjC)7f7-*__aVxrAT9@k`iay=(!gHu-KWI-{ka zw}T#Q8_^$ga=aaNfiDN%4ZPcgccU)wmB7yezEI#tMf+K(3;b7s&j#Kj@bfZw|JkSu ze5$s$*9ANlQ5wHh;9aN-e3CBY*(ULMlK*WIuK`c|2L!%p3H?~BIiR^XwDpp zf7~B`LH(~X_20icH||d(hp@gS*~s7cqHuZ%{0r-S%)iKga+7GhT<$-I$)33}{~B>> zCBa!qF@ME+Fefx?7SSii{cTn@)w#dzLwgt6D}P%{`km!6`I`fHx3K52T#k33F4~hn z+JTq;2LFbB=fw1@>`e@iz278#(>;pRH;rdk{y9<({d5bwoX->WFVV6+^goBxt^9-Z z&p(kOJjSc~`f^i${kyZ``flp5=$q!ZLEn77W^sLIN&Uh9vts&g%ENl$vgq68itD?U z^yP}rZ$3ZJUg{hDXN90G2oN3W7wa$04{kuIuNpCXxEpgW2(TJ+eew6|__2HpF*bC;RgFj{FDyjE|1V zpZij1eXUAW0tkDW{g3o-Gsb88@mL;lzWV+&k?)ZG(xU7f2IT%r_PM@) z!oPXG=#cp($-|4vKY}&~U!VWG%AZngZ(=!9<6%8>VyhtXX}+GN+9=v^|A4%N=j&Zo zfRVf<{lN19g#Tpjv8D3-*unE-dqK#NI`1E%c~0^tw1@nBJU(A|-9^qP@|9O>A)i=J z(geMR_E5K`@|onUP%7$RXo&oi`b)!bF}yu1GzE1do~Ex6pM<&*pNw?Nh)+R1j<>95 ze0{UKWIUPFziS!RYwjO`ZL~juf9$8pNk#bw=I2>d$-b~Z^8IXbM-dOn{xjqDhyR=B z=MqlO?MZV2`YZ{}(rLM6q`x!&mDaCW$?S5J=Qks{@UPVDn178d!G0h=8{(sYew6kb z4(>mxvHeL-evJM+?Kd2$++O5Ae1GER^f^Qid_gGH5%VYHx0ykAjvKv4iJtsvkAts| z7%zk9;ZNB{e=7*(xQ*>g_PE_N-#9(@N1FrI5&nqrseQ~J-_KxAML3d0&qgzdzmk5? zi}op$?;xMH4Lt^4`TCaOLVo6s^#}PggSq-l&_`XE&!e^JnZZ2$E~?MK1j6_jFFyx$ zzTZTDux~cGOMvDBEC5&n5fvd=F>zDi^HOZpiY+9-UO(GtEUuZ+k&sxmyB-Eks*!)kza zPeI*?cj44SwI_Tg>PCFlR5N}m>Tx{e<^Ia`0ePwJ0tZ4rHq7UON%C<%xkP-qP-~87 z&m(`_!_vvW;ZIZ8YqjK0>C_@n5wMGR@ip{r0-;GM~v}2 zIhp!b@jNLcW)rK&D99i7_A)d;q4N>!zYh0esTD=1{W;t-Sehp%yeC+$Q7WpX`B zT_(u|f0;o&olZ=mePsW`+R9kIAn-p6ac5wBeE3&C_|cBv^8xRK*OM6MhXAi)zU*<} zr~t)=_2Lx9-y`f1XnJ=F`xV98SNQp!($7uHgZ(;RKgi$s`h3P@PiuZ=%127(Lo!J&NzQMl|z!jrmIW z*m^x_zQX6XC10V>5nmAA%94iugsn#UvbaAe!h~! z^A&gEdH+dm-)y-Q~a8^|EBB$Mee^3hM1ioJpaJ?eO1qklg6u~ zrg&vIUx)wPO1z5gzp=iV_uEVG`NCxT?LAx^wLbCqOh;<;c~L&alRNo*pD_(mj%2t@ z@d@jvX@9+);!`r;pYwQz{3vseo##*34`i@ck-zRqN~U<8GA+YF`A|v7PT@L<=et<{ z#Cg%tO)OD7L%hiRH{O4n;urEMJ;*%FQi~R29VvT3`JSlE}@SBPAq@hDnKd>M6Gc)M( z7`{HA!{?2=r^WZ@_wxNYKR@O9_G?Q(Kf|yeKa9^C>@)a&ocqUrp#6CKya(f9{}eyJ z!umgPKYrJ&>&Lg5+~4m?9N!Q>X5Pj3Uur)-F}|73e+oi(&9ua~`1y}@e~bBh1N-fn z#`snenyNE=fF<^aG(WPZPJ@NS{yPu*@0_Xh`J7?@J&X2>hIr?s4M}|e?Q~AE|Hgp| zm0O8-wYRYU&ctd72E~3A?OmwH_M^zxrW((ifX_0XN0I+$Q@$zopL{-K$M&Cm|4sgc z5{q{;W-vGEl&_v&0)4u4gFa#ZOx>bS(|%;~{PmAG-!CPZ^H=1zRr_x)z9By{d_N$A zt)asvUJUsxEeUczUDVB@6c70u)*pOc%+Gtpd4}OU{3wWS;5_^o&ci|3m;5e{Ra!GF z`|AuH`+fE`5O9CS{`0#0;q!3#gE(K$q0dkG=aD*nV4h5M@Vmyc-{$+Zn0!t~^6~RU z5>l+Mc5)Ejel(N%r;l4?e_H(fzwpN7HJ=}3urGoiv7bc;IRB~Y{UDy7;PV4he)0*_ zA=T}2TLZT>a9abnHE>%4AG-$V@EoCb2;XvGeZRc=hxdOR{pt$?b-&yCw}1bOnkj$P z^1+3_jvg*y^9G*icQv>N7UUlr7+`GB-thF;yZ_V}{TfH5txNjLf&Bh~fi!mJColcs z_t7`)tg>O<>|^@6hNN{3*o`2h|7;$`^&sHVv;E%G|5@dxJse}w=_P;H7m?Fx_3 zr94o_>l*%M8N5$%ru`nr%g#ld&&Sb zZ6WVZjrVer9rd#GvSfOdrhDvK)z%q;7f_>xg3}mRSQ#Lt6UL~cp6?@`_!w7S1|!rL zy+k|C+Y~ErXvX6BeQ8!ar91j^XYm!b4WB3Bd#Y(j5c{=cupBFi@G-@@61S&Zo)>pw zJd%G=z(At@d9F_sp2T>OYXJ|%Bx)bye_LGl$_9z(WAX$TTc>HMIy4=FaR}Lyis;#( z(-e=zA2=1aCD~-(1Y1I6#_eX%yPhKS9+R6!N`Q~a?MjS~$?X<$C+Z_67Z@Q~m1Gza z#UPpeW0A~0(WA%`Ce9>Hj5A1gJ~exGiY*<~0vk!@q13A#jhWd-VIqSS_wCukE5jM`34 z#F?I6@4Uh{zqSqP@{9O)%LV0JwhRUS~G)ibE) z8R~?mI#;GPB^?lzx8PFag1)|Hc_9q&Y`vUKX%kAWn*`4=(ysdqqh+eO<=VGO0Pv9w1Uico5BVUYx{4U( zjXeJ+;j?|Vy4e%Y5{1YVN|>g+I;x$>EIjYJb`i4EZT|{!bmG zcH@ug3JYM58t{bY|Gc=8v29iO+j6|F_52@F5ZQvRiJt94@L$mziVY`0v9H#luC)6g z;QU~oQ(gOQKJSAc4Vj2maQMU?#B;j1Tbt;`bGoujFseL`DP?2gNiRmEcurTmALTFV zIbDht&3O^e>B?5%*fsW?E(uZeF@jsXp_$H)=`8I%vfdo~Kv@9dD}$)hCLmG!pU8G! zvZ&VD9)ApJIpY%juA)EEL~+OlwBFHJiXEjPvWK-#YF8&Gd@EH9#Y#(lAwLrrORA*( zfoy-=(3WnkdME#(_8toxEz;y};HZ%25%1Ra)r7>?GsPwT{uSB2K!TKxt?g+)_gI4V z$1Lqt&(`*{g+c-zMIf)IWc&AIoyMo{npoSD-=m8O{13?~;x@FWTh(6K@i00jn1_&t za@I1%)cC5Mvi~D!e7*LHj@lEQwLQ&qQh~)y86(F;>B|>aYELwMq8fY0pA7Im%VQ9E zvs_+jP+9mNx*F;P{vN!Jc%uGIMYi{tfEFhMj@B3Azsy31NBHydb3@08>=%Gz;_vIo ze$wMbyR&~PuFLKg*Bo3(eiISu{|-PJAH{^4wnYG`V`GuDG0+CI-<-gRPT-w@x87A)&3C91%RuH4O`8#9Q@s%IdpnX5udo1G*$@a?s z)cC5Mg5L!|qV{hlXirnZ+Ma$a3Q;EjGn@2(S+)ZFzd5jegTH#$%I4x$z}7bqzQVs^-ImQYtxcOY7H-{Kv)a3< zWT(HTwqgCQs+Ing4>hy~>eu^M6*oLoy{me2alL_`2L26%ho>Jvbnx%%jUI@u5Mpj)Kb;Z)3VCH3uAPEkE)GT8+)3I zSM<_&Et_gO%c^%(G%ej&=&#yQ6C1Z`$69aciq6Wa9UCM~)s~Kp?VFcweE5dwii;+r zTe-D8)Vg(PE%eb|MKW*SO1dlRtSH{G&Re#&XiNK!%7;tqd$(*V2__t~de`PH;Jc}6 z<)h8LC2f$`4_!UlR9;f(+guiCYWKlD8*7?6);+q0Y_Mu$o40iRnzD|yh0X07Tbs7l zcb0GQHZ%n4SNO|YJ1aKrsPmSUuUT2OwrFcrWzUv&f6vw`yq2!%*}AGG(9*uK7y7GS zSG4ibmQ^+WEt`sA@A{%G9qSufw+eebTvoFkeHXTDuIaJb#akYDbW7W^2bI66Yh$~= zE!enodCSt}E%%o+ukaOlm-&kCf1s&&#j=*B?L|e!+Y3vIw>Jv9=CrOTErTUuDSoYSoI`n!CM;=Q~9KPcn#wikv98QY!znSOk7 z{-3yaf9XpwPSN5bZX&f9Yme45ZBMQ?D?JYna>4IMDg)wLV7On1d*dGSaawSxY)HS` z^R)hq{XZrDXUEslzv+BV^i1?EoIt$onS`Fkdd5Kiin?FrJUZo@`g3+!C%PXh9ImI- zy@hTT;O7($*9vuC;dDK%f74EQ{)fL3PW|#?z!w;T(L4*D=qn8P*smh3cD^`;26#@a zv_$8>0Xo&a_4+*8+S$A6NWz4c4~{dzgd9rQaTKwjiLV|AV2ijk*a^-f4+p6DK$S$# zd)~miMxz8J(Q~E+J&%YJTG$NZDx{!~^J9j~osa_JD(H!q|;6DVM_&LMvz?hm> z+BNupP4ZtSP4J48gT_oW?m;AK)W2JrK+Uf<&=dT!gsZm260SC>6|(<)HLf(7((~U* z{_%7vUevsLMbf`0;c7lTCixkX22}7(pwCo;qNjDJQ^8YwIeeR}|APRtZ{tezh3JpU zNpWFs;7af-5^k4pCErx^r+kM{(V@2NWEM6oG0N{d55Hdtn$7p;nseBF5y;re`V;$$o~fhxKZA0Ie%zeV~i}k2K*Tb zCw`2p*|QRUp9MZ-pf}q06$!WUPfyXILj24WxBPvXgsX9tFSbg!8dt%eG0+>weOAJ) zdQ4c}wFO=Zj0iC(IWKr+ySKB)~QUTpsaoxt=k=V}INO%rZHclGn|t8T*0( zZj`H4!j&Erygu|=W*t54f}u8F7n^*OBexFx=Orhne#95-!cRj;QQMk=kh{v)EyTqS zm(cH6R;_JVgx`~E-YKY7LLcjW-KE|5{mG`DZXZDfJsO&zE#3;>BW=w-6I^J|*D8PR zZgLj4V?6r#ipCE5dA!xDH?FB@^ZI?w-D^5qdi)sv26unl2L4WOOJjFuV+&_Y2n6;7 zpQ0*cF-o|yGpGn|b}UxVasxK=26uEdb_IOR6Mt7U*xBOk_OiwNT{xt$(rQRj!BK)_ zadTI9(fv1}jfB52dMi6#Q=5;z{k>6penxilVm@}M-){Yk{<5oBU_avfJ_He&YMFYv zA`{fRWd5O$a+y|DM*Dn%$+RE__sF|5GL9*PidU*W7dMo0c@#ii1G~h{voar}&(ZZ)2%b!Mnu>jgPOOgeIP>I;`su6pCL( zZ>|3qIsUcUe_%l_@Jk+N&J;00y>>O-;^r(K~*4T*^;u(38BZ?f)M-u^RCJ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8e653ebb28adfde7815602f8b4d83f31.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8e653ebb28adfde7815602f8b4d83f31.o new file mode 100644 index 0000000000000000000000000000000000000000..4c8261c0ed9ae728fd426968b38ad03f228a87f1 GIT binary patch literal 51616 zcmeHw4Rl=9b?%uTNuwW2vOKosAFMG5kd3hAAAta4k7OAeFxW9l%4$f*vLs_+{V^Il zz}t6v zjj!($pW1WU=b8I~PwnmPRC{LUCDtAF1*zV{14=FaF_o`b-uK}j)SzB?FO|K2Vbn`pIC+IaVGU4BL1@td9*e);Kr!`DCX+VH*a{%rX8eeVw6`^+ieD*O5#c;Lt#Pwzgm z>+*fWH@x=raC6&h!+UqXI(&TdyTd0o6J~MWoLBEZa>Hx)AKA5UAIhE{&VKJ_!*@LM zDzK-9?|A1VFxT`Qf9Jj<_u_LL`kmPP%y9P2FAncTTg^|O8lIc^4lq~u^|gNE$llBE zKQiZ?r-om?{ORGBUPW7%zcTz%-fxC;-Xl3z^}U4hz3<+Cq!0SsyKkRv>)zJahWlFI z0hVm5f;|r$nfnqjr=HT7U1%%k!9!Y~y)CDPv*#@7JAVJ}BX{)eKGN5+*VuLHRc+Vx z&zu@Qaq?YYuI;;i_5DXqoV*X1eZzMQK0UnaH$NNBnfJej=j{H?@Z6oh0cL66@u%;H zO?!v4Z+dNb&aT1XY){FVIho&Bcl_j2(d_Ld&t+#O)@2_oiBo^~4karC=rfNuv2KG( ztefl!j|})G@1?T-_mY(#E>}PQl@~FBvH{%UJ$*-gfw}LL zc>9jeP!wwq|L&sj2=IF-K3T>mcl~YO@q0>s{&Bmf@1xtjI^NGje28a1;yherbSYLB1B&jsqv*)hY*o)|u`Y=XgOXX89u-C4g_ z)n$Jvd{xeu!&l{A9^STWj=D3Pz3PwGXIKC6K`NUN9=QU({MApcIJv!2)y>Tak3_Q7 z`_)&dx;fe5k!2q5`}J1>H!(c&YNmRB9ZEIzp%WtTwTU-e&d^6{0bE`YxAq3l!CAvtn*yx9LWd|dmw{-UB4FRq%(XsdmYYM=UZ@YeLYx*bF1@qI=4E{rgP{#2RbkF z1W(p~S=D_9aomLFO0l#;`WTb5nt}!w|tZ4o{41ko$+Pt{jJZx z_k)tGzISp1eQ#fr-S_UY+&;Rs@qNEt6COGFPgw`n_;;^6QIJ^opz4jz`FYnv zEfv%L61vXyg@ztFd71CP;Dc&pVZm(=Ey!R0&@o?N&jN4J-s3*c$nk>2k?hdW@Ub4xD;>0A+NDkT< zyd^yH_S;!|c6%n=w=i%0LvP>W8Ceb)$wj^cH>wHueP?_42k(C0zh|A-zxUV-^~0kL zIeXsD^^6RyoZR=yg75sr-6%iio6eQM;m&%b9d0zWqR_8e~j{vzL= z!P`9}hw@*aeta|NNxnS;k;Jvf+dU&6UN&>iiA%5RJN{ID--%0C^bP)09XNhBWZteu zsDG;wi`4gnYcO6A4+DsYfAOig1DJa>U!PB!yXgj18Stwwl}u0znlXmW+_nPqOl{wp z$a2g*HGLnqSNDB#*Nb}nl=SI2OUEb2CBdQEIvy77#WBHrT z6VLkm-*c?jdX6~Ohvk#K=hFxO-Z^ciRS>#ffu`im~+Bm zt#nQ}83&KUPV7_d&HOm|2(86`2s>$Qh4ue}f&ZSYBY)$Xe{R4-dl7-X@WI1`dlERh z#%G`4n{uI2SVUpHuX^#J@3rrK`ENa{Q1!$>;P5Jshc-KkEls>t_ymC7jPEc$=mRxT z6;kz4$L6MJTT@G?T8olWD)nZ2lou-suh*-56MP=8&!hagP}?59e1%-B?l${;)m-3J z|3mA?Fx6?!5-sN2yIEz1Rh7zsM$-v7uX9spEE=odfHVHy)@o?|9qHWE))-gw+S+4H z^V+t~i*0Uh+t{9_8@5<_>%7!SLwjpSeN4@ZHN|7|jA@aEEwPsR&d!J>V@++H?a^o~ z($dlziM2-?Hbt7^9Zk{Z)~2>teM@V5V^d3W+wGC93-xJdYjaz3YkfS@x+s0NAkwyV zW5?o1q_Z`$pyJ9%B-XLeI@;?L4$SD>R3B|>{QMXaZR%`_ZiPkY zrD#*j=OWr*NB!r*kjADBTW*hRjMld{nck6%`gtx6hFL>>OUv!i`i@O{&~-%HW9_kR z9jH!a?QNa0`nFib=OBC=>N{GSq79#mZbsK=TV!8qrcIr?1D%Y|K_9X??wHlw&&?3# zzSRbe*THsDapq#K*J;)fsinx-PQF-%e6b9{0yy%YX<0Z%=zp;c`46^wccm@r9mzQN zWypX2>ljYbx3o63&THSev9l>wp(415iljS$0?F-( zvVcBSiumPHh^1L)e-#*}uKyIaH0Yo&Y`Cby*c7?>E7#XvUmaTB9__$BiaBVLg!W6W zz-J?|I%09e;_@wx^_Xl|*GHMh5gV(slMo#i2n?56t8rqF^#^A?wfzCvH~Dz8#C zTbf%ML!H|?V@<80)-9c}P-D}^=C-C#-I_J4Z;5|{D7iJp#2{IM01W)hMIR2JorC=6(-83cHCReAZ*5 zIW!IY={U#7Kc_;+3dPI09{#clwf-`6Jw6G1dhi*-CsYo(_=NH4z^4bF{rL3b#I5@9p?_GF7jo3oX6h(g92NTe{fNL+Eg-Pjn9IS73+|P4$T`%ypNCF;rdU z@JX5us($Pu<~i!GoucrZU0U7{eC90gf$yB~Sz%~nZ&Gzw+Sr>^1EPHG0)yxB{h*OjtbdZKa3Sr9rs)pQ z*uzeh9~R{mi`?bCph=#w2l5sx1(&7h1ZdKq%Y9XT<)2xsD)Nz2iol1z3SnafKK=Ok z3-eSj*dF`w&oY_NH|=?kKPR9l11Eeqzryl2WBaugpW9J@gJPDxe`mixvELut?^-Iv zK{it^`1d-x??UJO3awFs!UJs6w%L{u4{Bb%`ulC#i%MMrdPBZif$|Dc(wp!RzB>+W zjmb*AQUJUqb^^1X>f1T!V$etPmA5Nw`lSFZ9-XhI#Z^+bFa@4HTB7zuGyP2XzwnB=`TT&zE|X{-BC4}9NV$>bgnv5pjK|F^->?9H*(d90yRIb z-A=Sm{UgeE)#*}Aqr5xr*L|v)+Ma+)ScH19K=s7^FsM)^-%1v!Lj~%a@eDtPu1aQ- z{zUuz)E?74@c=Y);{dL{s9r@~qTauZ63Q7;;mtU;u01XnNH~39C z4LYJPG<1Llou!Mp|41P8Te%I1R3ES(_U?|W$sW)hYlm{xKNP5af9g-FW-xLfuQ6Xe zvNPcC2bmmzy~pOO1G~6){Q>nGAJOD_9H5W-l&ifL3fwf>uV#HHUQWEHF1X9oCWX~d ziRW^<>A?r^q#s34$J!5bN&kIc&rj+bAf(@g`RcK_X})7<*u-B~(Bv(O2;d(>f6~!0 zIOJ0=Q#?HHGZBnln6+_Hz+<1>G~WW`@RnUd=D&YAMxLPS4L9Z4BbhM z@8+xT#r+egMZZG4zA_*2?KKW7@C4Lr1#0^?uNRT4)U;W7>btWL&lzDNbv&c}%BbH9 z{Srz&kfn~U!Fb3>mY~!dL>q{=Y4I#g`-w*UPmE?{(C|xEJ(y+a%q(i3>7sa6Mglnn z<42bl#qlJqgPs=mC9{CfBzb5*FRHvmBOZJT{Ri4ldA4J%@$bZVW+qinzHHjdiz_dY z9W@u4gUk1@WFKGB{Vv*mg^HX6WOC6dIIz#)< z7eoV;pJu?kg;w5^`D**L41cQskLEk{g}))MI}Y!gkk$T6+jH7S@}?D={@4CI`k=bF z$eTf-=cRZ#v__Q{Wo8kb^b%d5UiN0_X^r#c%U(Z)3l3<3_DYMgGN^wu2@ii)7G?UV z|8@LgY17`GihdkH#2kjp3%P@W&tGf@d$kpznQ7{hy93e9q&56VL#$k$2a?D zRaZua?k>;oa@ETvR6k4s+OYxs3H-|JUQl$HeA|7Oh{vcS9?`z`&-Vc*`&uZ!ARakB zDIU8-JfeM9JR*PS{@3xS{o$ql&7gRMJ=?eW)A0v+m0g*bUsMA2AfD$t@~MeOSANC( zNBM&Ck?}X;KkD-dfBPKy1^zbU-G=!!4EFzc8)*&!di)irN3h=L2chL1DN)aE!P1meD#Uyv=VRoL)O;F3 zqD1@iobxU6@yab-dB$I6z2sUCA-{byp5-6qze4gK7QKz+zr=*J|K`Jgxk<(Gyn)7V zVS0Um`0I{iw6F`cKm5Gj)%x{dzM|z%+Fv;?|8#fd2|9I+tm{7!&7GjvTa>R~n@|2j zyjcIiAMjTpMws>23h2KC4@%Hs?03b7wy)6_%0EO)+!KV12_@tHB!Gc`&NxYT;4rY1F? z9h;TP4>dI24?U>!Lk*hK>k*xgGd?9hEX9KUY%G{JX11g+OyYSh;1V&s>3+{`E#_yyBqs0NPu*C2aGBSs%LMALXuizU1b! zj0fZs#DgpTa6Pkf%TV@g-5~PIha!G1#QFsKP&l$aI-VTk`pg(*|p!zsR;qMZxzjNvOGhycU`KmOUOX(xEp1L#| z$RGyR_gS>wyfm5-ApMd=BR(smUR)61@EE!@nx(Jdv+Vkc?4x4#U!^yT=!9;M^bMHx zvC-G~!{E(&wzAk!4t&vfO>@ab+Gcf!rPtTqXhyn!w1542KR{W3PQ?BI?Q>Pw3Y141e4G2J{EobL}@E zzrGyJvm-{YzrEdYvsBPdC;!DK;WWK|cnH^TG(M6`jJ^*={rY-`X^c;02AFQoe=+u7 zC_iUXJib_>zQOw)o^E|TlBNMddx%G!KlSw<^25HVj`_J1^H~7<5ur&rHVrOfJ&%l?Us) zor?CzlF55?d{X@C0Y&`U==$CNa&$U{I}Wa2Y^F1NJy#i>H?3a~LIqjFB{fW^Wyp9B##xM4lkWVI?0ix~2c+5`k4Gc~jRjg*otq*-zQvq4AoRjq#e98n3I=Oe)v;tld2x5zk}~jX0ifUG`x7yYx5t z-KC!#PcC~f-?;RJy{>!!`P~6*EQd(}9Pls31Kgh;4^lp^ckF@^Rv#@u{?NPFroPS} z*!Xwk4_tp0S%2&CyM0?u+FvamwJ!_lyBSsNepXLl;&j(|&J))sdG70zM*=xH{itrn zE7q48DyNWCvezHzj|pDxk0&U9eyq$6B#p;44u6>ZN&P{g#r@&(m+mk3`Uv@yc6M+W ze_%g0D-~bbA4o5J;QF>ZPW#8{{3`o5?eA{wFIPN5Ub-A&K)d>z^gCO;A|CclElT@a z=htcJ{zrY}(_)wZ&z4W=`ZYTVcc$ZI`!?rz@To_Ni;DW8sw1Bk1;`~fpSs42?w@_* z^bh)r>_h+16%6;!zwLOL?;bB$Uwzv6aK-a?^BwU7eKFqLcCbNpca{`-v|T)(@(KJUdxA;y(laTcyr8))P1J{Rur^VE=vFGAL>f^l$fF)6)C> z=#SF4T9!buSx=PU`ZpWTHz>7??q9(_Mc7|KB+~sWeSH9Ywoc&WKblXvc)tSkC1~1T zqxu1T|I4i3!=Uy09rIsN+!yEo<%^>}{IzmRUi$vlDZSp+`*W`SEy@oxA0pqZ+;WUo zbU4iZmbw2@0~-25US%BnIC_O*Ek7RFdksfGJU;6_JE!i&(-BQR^NSJ$JOL_ zQ6UfQU3fF_8D7#i!|GewrKafxi(cQ)M|o}zmFL=W+7CiY=Fs{C0%3cjWTKZQ7+%j@6SwytbUFIJYoem-L5vTqcM8-s@jQLKsOxW_^@CU6dO&>Odgejw zA7H-iMFrC5DBb^5O81Ap|4H?OdO2e1=bP(w%%4iF*Xv`f_kBA5()twbUEEceP3t%8 z|Lgma&?lSLPg)=NPxlAg|J{7GG`b|Y80Cl$FZwTgY4mEQ_4vEEYkHXE6>5Fg_?Jc( z(}tPFmjsqZ7c#B!rCl@h{=f{~p9PTbg=mX^foMO|Iv+2MW+wIZnrT00X*9sJEze2P zdd}*H_VSoE?ZxxM2`V@1bI89oS;=_a-Ze~Hd6oKpa$4RZrfs>o|CKJ!(Dw&G+w!cW zUeA)gT7EXuwmg?M7aaOdU|N??j20}@_6+I$EacPtWfVHB|HQ7^={mj&6S%+nAnd71 zUcvIvf2Hwi6#}jEG3cUralZC1jX$l=Rmm$Ef2;uhx`OLl{jTJ4OXt%Cm(_PZ@P$m< z@`5DAqxOfEKapu$UX-NeUD{t2OxyBF`USvr`2wbOd117s7#TZJL-Q%*PwZMTod(Q| zV1bG6cy+bTkJP`f>HB;3em?F`YkvfFzFkK76ZD>FPWpOY=i6Yg2MLPvZDm(5Nd7|p z)cXm@m#+Q2Z^m=wem1%T<1LjhpADGtHg>+m{=#G%&szU!@yR;>1Wi6H^mZ*zU%wn9 zeFGUff7Sw`a-(Hl@@`IKMEseMfkQoboeIS^Eie7ZDRoTTd|)Bh!)rzP=d5cfCe>B(6x zdIrxoluuATlk3}fxPa?hdKTB$<9SImRJ?@b(M19Jb7{1M<>~v+OQWU9vQ&HXQGRi9 zPKqY{CCRx=qyCa;aj_mBDS4MBFXM8pUm44@`pjW@w!OJ5&z4`t^?j6Yq3_JAAumMo zhYI{?ZzlP-IMv=v9#7UE7x4J8`C}H37h7J!&F2+J947 zo{z>4=|}m6^+PGwxrF1x`r}fL2TPxCd<9F%KUyCdU)n!q zrR1M7t6ycjba@!G9$#25S7Q9FCzA3J*0W2ZTj}F%h&dn%`9}tbM@Z~=h=VSALrkHw0<~S|Iv0S_Xoy1?6LiKpxqsHXNV4f4jNkD|Dp1r!GFZ~V7117sPkJOe&K~!E>Mcmbap%xGBoxJ5x?2- z*|VK=Ab!zBmiCf9ffODy9Poh@UU>{X9s6x*`J}IDU+a(Yb(Vak`|m9IN{`R;%~$26 z)W2iKv$mIdmh@%&r_pjMMF-+Du%W1FFRcfbM9WGk9ytC3@u{{x`(ui&4}aMFM(cfN zyi+`I{mS?S_O}7dc%lBI^?`jJgZj6|(70cP_5<+8*}(a9kM0Irc}f4D$xjRt3*dM_9#VvmhmIvB?EB`V~|3u>;|3xRy zqxb{Q#7{IhpYku$TccB_A|O;*O8#^#FSI;ee`|Eg6gXJTvE{hmu#ECaM;QATh_~$M z^yw7OEH67c<1$(=GrjBXsXml@nDeRiLW=Zqcf+Y!*P&5*HC*?R0%yA4t+2m zO|pP7C%+@b~{v|U@n8x!Y zyY7ZMCDb_nM}EqV7E^#4{EMVts3cXsfy!rHK3CR!>#ND>{S6&2GiFhLb9+1QQNbxHVV)-(^eCdIhb(>6XP;-S z0p6qIC9g~A>jR7rJRhd-Pc5VC#g1CcI&%Gl_+FWfaMnjc-`_%iEYs^P>W`O~sU6!U zBU}@-TY(oy3P4W@s<4wrd?hShmXGpcyewepC%o9d4XRpHHT$>oF`g#tc$rG^fallX z4>*(#8Xs$bpR868NlHWiZ)`L9I)54-fIpVueqI3nazF2f{kglQrk~Hg@Q>74?`LBC z(EVdQ|JwbS%4m3MnA)fL--r zZIJ9u7Iy3Lj_c{E3DDJaJ%#orMpp#MzRAG{_4A-Bcdb`6zw!M=_kN)B`BMA*!!A_T z2ldw)y5G#}nE*Y1uAd)rT~98tH~h@?Bt7p=;{n%Ox_xh#&piJ``$5Q`*q`^&^Il1N zJ^D4Ym-zEX0q)WFGr#{Qw!9E{j9>Tj73OVM9rwdck2aA#{zm)rJM?sCef z_A1mH4*$9C&yLp~_P5U39+i*VUgP<DIx!)^+)+=|143*&Q<@h<)`%@C7;Gi zkC1QPQ`GwF@sc6)0iGAI{(3^%vqtx~E+2QiVA^omGge(`+ze?Xc-_lx5T4}9T)FFf#t z2mbSUfbJFI%CEm1v(%$M@_l2~P5+v_YTE~weem^>Gk1No{F1v*lzi~sPyg%R{`i;v zr>?y3q7Tj-`R3hUyZyie|9I#8tN+ih-zX^gm%n)PZ|+`H6zHFjcsfRR_HbYWCgH&{ z$M@X(dMf?Z@3KC*@$Y{7e}DPQzxyq&8$7xGoHNP~Aa<_sm8S7`S8(y~$<569$dr5N zWx+J<@szvhY9F4Cbk(Pqo2naC&d)ub8CO8cEocP~=zIE;E)Fg%`;%YtcxU`k=0~a; zE0lyMN7)Ax$O1aH<0g~49Bri!ECOXY9GO1orI)kSrZ#{=;{sL9FqQ# zzPopi?<$gB=nsCRC3ym#GwO`O|M(|$*zyob7^|H8?7c7m53WNU{1pdm(K(MN(XjWz z{LC||25kJ|yC}!6I;iU0(wyqNu%AEPLT}LMkk%>W5K8g_Mzy!eRqpdj zeQtG?--CW}+p~(w)B2@+;?gf&UdTRkmTNon{bTglr+Jb*WiaGT&nW9+(2k@VkB{9W z#zWL&W6Pi(anv8>A@X2#Uaj8|IVmrU5jmvin31KuFylmycgE3$7=Pab5J=>V9Mz8| z^r6Q`Dq`(;FcBLXPp)^ycOfr~eIum~Cf)M%c*J=6$c!h?SmVii(ez6`!VmJ3_mn^E zS3CS(k2f6!HHC{aJsT88TKcG#c9gl>)p`>&FF#AR4 zC8wU=uvhKydj0)o)0i|nZ@;?QbL)TJezW@E_f$v~eVj{do7m;p9NSN$d@W*zeC_b1 z$)RVlZ9HFMx?t7ym?GV3@@g`5+}dnKG<`gmsT}jnq329BI*GA0bG@04u{3JD(pWS$ zQ-;*z)hA=ajJLIZvb*NunxJ>a4GYhx>N%*2pdckPa5I=FJOQ;G@{p0Wo|KvARkv4uj!BqmOC zplzC~a?IHBV{G~5Tt#CmhsTzU5r4NcM%WLdGq=uLE_-0YD0>2an(;hmw)+z>+B?S& zzZ(0|FD|?aLti~Jjh!d;OyR6gFReJ*qs?scmuF6@OEQ864C&yQ zJN&Y|*HwPcs5YP_V&@R@U?J|Z(u#ndok<@$#u3wwHoRlCQC$?x`TzMLKT!^*{ung} zssyEgesrffwvW*lsHbvR$@FN!8w6agvag}GH-0Owlbh*^QNxSNjY_D7PsP;*Iw_85>Nyy1i-4o8H zV}|njTspeyXJlqeKf21ZQ7-&;Sh)}xnK|U842Kt7gS|g@^z28dt#KXx<}o;F?NsUeM*n| zT|Mg5NuIlO%!T>h37NSLUyaf+2bk{Beh_lt%RHAHS9uOax8LPUsh5*>)zza;x$GaM zqxPfQ2ilL$tg7EQ;PhiwIHDc+!sBYkRh~({%5(Xd?<|qS_4-abge;CC$^yIdXVnukBLYSg09@pj-Yw>?1;}Q zBTN$yT@yi%v0F(PTf+3-``Cyu_68wC+cGxd-$Dj!N*<%@(9!-#WN1CY8RH_H{lYR1 zXV9K&sU41b{}uoEl;^C!VN1`Xj&%&&Xt;iQh{ZpIfAaJ%-}-xRNWstL_g_c-w(B2; z)V<%j@QELME7+=@o%=7pz18=iG9q|0DnY>i`}W8^UKl z9`Xv~!I;o&eB%?k2;1Q3gu@rZO_#uy&-nYVJ><`{NAahDV~YmZ)qCML?5;^jXR=;@Wdb0Cz^h5^)McH zBbw?H9YQ=f%X{z^n6vzFv6Bv}LimTCE9d&*ko)&t6BoM655sqKZ-Vh_@t7IOCwr)U z`dwUZkADAk0P&J8&ryf3a+YH-uW*(F584NDWIz4>>)PuKZR|~|!^?6QD_+u@{zM9O~yJt-cA+q(7JQ@4p_# z({^j|0N@Z}t^zi$$49}I@qYi6L{N^P@3DT{bu|^>K>jeldtyjcYrn4rKhvI{?F*M* zkfAMd!O%_MkCyN?%{Ol{IV+go5ifGG^tt&Rar0!Pv(WsGcv=qUvkg#Xwgc$5=jM0B z5xn9^{f@ZhGhV>XoAjWD?nhBGZ0QEtYH}QecOJPQ#;X5QTkpRJ*UI|z@+QThJXX>3 zlQ;sl0%52vR);-A{UK#iy|px7;MiooQvXUU#YL7Wb*b^3t-mi-)0{}ZQ~Xfe#O>* z)HY0Cln-TnihGC{rGKLx3Lz^`>d*c7APPuk*y8wWrx;4yKG);^--^m-tIs&9PdHhh z#yPn_gsDw-O7U;#Q+>iD@Npi4s7x}KJDGIOiHjPw2l-Dc!3=!=08vNvNxti_^}{Zv z2r`sWej)v91diKZ;z)U8^|pI;71@8?Ls9;PJ}e z&h|)G9v5^v))BS=;;g4xJf1}*9$);;_3H44^HxNo?NN(gfxoTU5bF$8hC1r$Z}P>O zqArhxzP_ocE!2R&z=uD&d39)gBtqwrNZIXQx;j*+|5j(&SKBJeLkr783l>&=tvs}` zrM*5@6=_=Bx@cqDobu4^H=;_zrX~k`fyRzkVM9xM!{!M7(rAmb4&=8jg81gP*rJ6H ztN(syS|$}ki?+qAfdMt$8EJu%X``fH>OG2o5$V1K2)WeYnM^6=MjZf{GKhZ^f+^&wLk|2o<;_0vTf z1pk{Z2u17LZf^>G_40)a7GJrzYS9&os}KZy zekILc*SBqjFLX0byVX}0q}v%KeW%hK9k z1e^JA$k&0RgNY*X+Ql#+Ce3x~uj9obN{e*L)A69k54Wb2W3Y{-uDjf-e32t4N6FXm zwn(d?hp3S1z@vXs{?jpt_)`~&4HpMq+lQ`Eopw|U(bC?O{82ViL-t6^O7{ns(Kk35t%oybHqr&HJ$b&oci!@JQab^S@;z z(>vCU8esm>!A+;D^4ESF37%x_Fh%;`H|_O@YRtP}eM(-=qj}%V|BJ%EEP5>X?J3_P z-AL~9j(T~H^Rt~sF#-D3`}(?9;#{}-Z6U2M@z7uj%D$-7oHU;SG{1Tr*SozjQ}PK4 zNDil3_S+93g7^bgRLVc7HrC(G!o7YRaGW^)%E~uC5@_~R7&6xNw~?b~bcX3~!`Cym zi%$ez@EXN4_|g0@7Dn`^J^0`=gU?}h>ib);i(@gx#L11YYb^c<%EscCjDxQnhhINV zyLXQxKM1>u&|e{Z)4}oj4$jBY=dZ`%?-)m)b>rx>d>sDlapeE~IQ%!qX%}#POhS1s z6Zqyxqd&glgcs+B;Uzj1$*frIYD!(|6ObO zQqFCbKTF7IbjY!nR9ems2j97*?RD_Y`_ZQH=f>gdA8`eQ^rXDRM;GKN)V>)Skg=tn*2t4!Y`izb$ zp=XullUhruqKE*lb?S9MW_-FnL0>9tp`9lKF{8t42u*Dx1c;*iZ{OK)5 z?^6QL{2_r4Z8iAQ0?+(20{?=Q|FOU`U*83UNb*1V=XHzs+f>f}VSb_IGoJMf3Ow_R z1)lZPKMI8QSU&SZ0?&HtAIWjxnO`RGtY^8GXc-C{Z zz%zfXz_Xrp0?+*Q0)N=XL!-bmf0MwoT`dC7{0@Qdwen*E&-}Q+@3;6afoJ{>fv@;F zIu*xGfoJ}1foDB?1fKa13w*Ja|ERz-|1p7QJ$nV7`A-Ubsg=J^;F1)li>0?&FL6L{ti3VfcGe@ft) zKP2#*EdI2>Gyh|Of8NF?FM2rtFkk<1Jj5gCH@3_Es%9!*F~86*o*2(|1udWDGrw5i z*)IG{y*?Pv{E)!2U8Mrg{BnWkIIIwO=2r>)e(TRA0?+)gz_VSo0?+(a0?%={THu+# zR^ZvLI)P{YZ316w^foJ|HiMM(VNxbEMEby$SemDae zh4TsX^&cy7OT6XlA18F+d7T^* zc;=T1eG)dGmkB)cD+K%% z>x5ecp82;4JnOk$;F;ej@SJBh2|V*#1fKQm5P0Ut1)k@fE`ewM4uNO8b_zW6cMCk* z)g$oCe^}tzu15u)`Hu-a+tn-Z%zskgd7ZFN;F-T);91XS1)llO2|TY8o)>uL_X~W; z#@h=5&-_;ep4UZ(1)lkD2t3>Mroc0QK;YS~V*=0oL4jwxP6<5ohXkJOIxXR%!7%wH_f#-GIPJw6sZh>dJdIX;N4+}io^{Buz z{|SL-yPgzy=I;~uknOMi0?+(s1)lYMPT-mUyuh=b{Q}SY7X+Ty6^8_#`L76khmD8B z0?+(61)lvdAn?pTCh+WsL4jxfDS>A{3<*5*PYXQz;f%mDpEv6{|FE8ZyNS>Fjrjq~ zXFThfC-BTK6nNHC|46bU-!i{g;91X^LO%2Hlau;jeRy40F7V8+5O}t0vA{FGO5l5K zJTDP==7$A-zm1<-foJ||f#-R4t-v$?R)J@`ZWDOsuNQc>t5M*Yze(WPF8pa|eQ-Q5 zzeC{Ju9(0xzf0onypWK1%ikgJtmjUFXZ~)1=e*h@@XUW$;1ApWdQ{+<-z)IEPIyA# zng67~bDZxJc;@dHc#iXD1)ll+0?&SULExExNZ^%?=T`)t`G*Cb^?Y66ng52sAG7{^ zQ{b6DDDbT3DS>DHkic_4o)&oKpAmSTH$N75=Hqt=UHOFV@>@Q~Kl1|u&vxYrJoAeM zp6!|`@XQYhe8~1!slYS8OyF71a)D=lg}}3(iv^zfRRYiR<`RKtepukcR{vUoXZ~7& zXFt>lJo9fAc=p3>0?+*Q0?&SE6nN%u5_tAQi@-A5W&LxTz%zfnz_Xr>0?+(S0w1>V+#>MI?+|#-8!>@req7+gHlK6}Jo9%7 zJo{m{z%#!`;CWm;Ebz>KRN#4?@R-0ezgOVdZ%+t3^Pd!W_S-&zXa0VHXTLow@XUWs z;0J6xKQHjie?{QgZ-)h*`L7Fn$i~|n0?+(61)lvdAn?pTCh+WsL4jxfDS>A{3<*5* zPYe878_#D1p80-y3C59s*bnxPwx#kR^Ya9r{ZMH6oDZ3gzx(cz&weNtc;?R(c=kg` z;F(`4@a%^&foFcXz^}LcTOshwUn21A2lG1$>G^{BwF1w6SS92$f3?7~AJz&y^Xmkj z{cx+mGygV$XFseLc;+_>e2>jnn*^Tu9Ri=Q@fH(!=Ht)dy5fQThCc(U4~`$^Cj_4J z*baeb{!W2!wEFB8c;-JW@VxK+sK7J-F@a}2dj+2PPY68g`J})zf1kkbxBBlFc;-JR z@VriYUf`MEFYp{!F99rr&u~WI zng6lCvt9gfFvlD7{q~_@UZ1>fa5_r~gwZJoft-!OMbpp@)TLqr?g>Dmg=5G>s*0V+6 zncpGstY=K%nI9K;*0W3CnV%4NUia=0c;@#AJnQ+ez%&0*foDA*6L{wL3OwuigupZZ zNrC5ay-(nozhB^aTt6%D%zsYc2kiKIUf`KOVEOc1da;=z^uL3aPtSon`9qdZ&rydQ zcz)=g#x-q~@w452`!WEHJ10M2`Sc#aY^NN1?g3vv$8K@_%(byYI9Sxt*NCmvZ1-HKDtfKi#D~?&)eAC)*PviM(ek!2)&}Xx_(Pr z!=@WGbVZ#0=(@qhn&L5qS4+3Vnj&~X4dwMMkN6gEU#z*jvG2M`GfRi-da{BPP_NWDvHY$206z>h=dt0mij<(ZFlt$EQ!S48f=JuejU9`jduwDt#g$O5W1$h>*}4eff;U$q`i;#oGkUC$!}#eOPwCL9o7z zmo&*6%l<?a>{eda%Id9wWq+rHG7>l^9N9mbRW^5FWCFR}Cs)x?wj5+J@&-UMshGpASCGY_Ki{GBIjv*Z>Mr8f}2kMHwmH+?% literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8f7c307820544eeca13d80f89acf695b.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8f7c307820544eeca13d80f89acf695b.o new file mode 100644 index 0000000000000000000000000000000000000000..315d658c5e7b9ffa58d866bf0b85ee222ea55989 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZLKtS^+13MuWI#e75Nu{eOKOlnk3KEQ zv!6z~GMeo*rCv`su-M>Yl5heFm??b54Fc z+;`%5_)+I~!p9Gs2#?G^8Sa~ZD!hNq>F}lfr^6%r-wQuA|K0FIt>?qXPrVyXnfY*N zWXFNQr=EU#Fm?Wt(43afI6XE8z)8W1YZ-z%6x)i>!;}U2VhsvD?2QSQjVsPIY^t1mv;W@1s|J0k|^7EI% z+ux({{x}qxkMY(VL_a6Ob6dY19s$1M{HgGqljp%Z_47xek^Kh;pPK*lVCd9|@CCe@ z-+MDW_vFcNx%0j7h56?}vn({H^@+hg=%W1marAv696E6-T;V(&4$Z$5ergTzyfhR# zabR%n$tMOw;Q7M*Z-@8Ie-nH`50JeNujDyj2$e&Z7ajr)^p=wN?QrPbH^V7gPlo55 zzX-lAhR4l(Bs4J=MvtW(UcR9vdrkz($(hQ(X_AnxuKIp0!QtBit696-Mn3oUs2S_?Pg3EgCC) z9X_CGgqzeiq&ps^c1Ia2d=;=9-bVEGL-@%Dl_jAMHx|S1JXf#ndhM9ef8+V%2R3c4 z@=5YP<$DA0y;tCy_k8cp_Pw4P_}@<3+10ZplMW9xXot`K zt$uBIKkKJ?#I%7&RstV+mi0&ene|_KMZfw^3F|+1AG`W51+4#k8oPQSiCs+rZ4u_$ z@ZN;};eTTN!w+bOFaA&c>cw~1;o-kY2n~OW9lqFrHFW~(ANkQ_z!F2l53s}k`T%JE zIw^E^N|?`4(Uhh1c5VFCp%KvQR+0}QTf7eVs8ZHt8ncbJX}XOu`#3Clve=x=j@vJbytYUoboRf5 zOpu73g46+4n?VY{_ zZI3PJ+|k^&wLMmE=#urv7DR6v+FSkJPPU-a7wlXhM5}A)>TL0LbkuoUTDEyRn|y(~ z25(1aoy2zf+B(_;{?0&i!J=)gzE*Eb^R~A9CGGij|GpRuP?ul0s4l-?NnL(kUS7eX zI{aI-IPc;7B}?n-@*t;>>n;$ZqsbfaHQqgf1biL7z+;dIRtor9?h0>%9NxP^kVap9 z*S5N?0dK2MSjP(RE-*nb%^GNF2zdQXeA4*??VatN-F`GDTzgwbr?;&$?=EoP2Cu)> z7ihRE%nZ|TS**6?K)G~s0~!H$fen>7W}oHIyJiY2+j4=fXJ8prjBBxK*CFxMoq4RTuI;h9 zC}#EY)pY_>mxrxdv$ums?SF-dXmF&QfMhJ0-oWD_0y9h2vrE@Q=A%`J`Yj_!_5U#q*dtE1E1=-b-d=5w#FtX#FOZo~5G zb>)?7>Xz58U%sYvc`2J;=`L{3_XZl8+#Rhr4|v$?#?Ztoov+~Y4-2n5*b{);7djYu zLLS|xi+kKMdq(X3W{z`gO{vVj)egI$?ZbA__kA=vg9qYd=yE!<%etK`eYq%Fx6$Z^ zy3X9DI-OM+@zf{PZRplEHNx~&IA*G|Y4$m(+@@q#D)!kJA4YK{!&Vu%tpIP+Qkfs! zyWogbc-dW`fx+;fzwFb6m)Ww`Gkc1Ik$3mlaKA*tci?EN6|ZeTpwKvU{a9YF$?HeB zmJ$Mm{9^QMW2|?V7Gb{vk_7X&%Q)EA9quBDm(6GKk61TuqS~>f4xCWBnhJNmv`%I z3-O}&*aScKp@K3KuX9dzIAFJ<1&a(EOk+a}S#HpF-#|EvMf8J7?Da!zRxrW#n<;PX z;Ps(wdtp$w5uP!^L%!g|NH%H-lc8;lH!GN=N9j+dvGl;igo)Ju;GraTj_4VuPa=BY zbA$GYbk-^QCwe~31U>qPK5Qh8zv=_de=g)R(1V{eX)o}D{)nFIFQZc?_^lBitN>&vimS`uGhsH8ot`H?kq`WSx8|;(&h8e{OK{ zWb(K1T>qlZ&b*{6{gh>WU78p6ME|gdlRG)t1N;;Q*u+VEd{OV+l@MVc06;zf|H}y2 zBIJ>g0)b!Nt+C6*fF}H7Wk64(D$_pZ{I~+b*KNOsKG44&urVj$NeN`%;7mSzWB~jR zrP))-pRrK-w?IF@X9RO+(z*%%;_Y*TFvoC>Hksrb%w}cJ9|4vM#=k)H56-+2iD)tU zIZo1FM9R0k8_NVuZ0XM?e&An)fd?Pt_R_?7g@IXfxIEPUbQ$;M`e+pUd$J*%RGtE~z&0ek?k} zMWDBX9%~!XA9HfN9d&^(0p1O~+k|(cF7V5Np9XxMzz>V|(@+=quL7S5yhq??r}6$X zQ5X1RZBMrgcr2ncev`nvP#5?FUC6UV;1A2ZFh}%;#J_Uui!7B8?S4uL*MUU*OB0|Cwwq>ZAKJ)5BkL$)4O_F&_EX79x@JAtN|* zCdEJQk3XmWSDO0o-8DVtPeTW>z9rbm-}s_%dLjG^>wVO}$bY6M(0J3i{~RKFPLKN6 zkW(uROiPIRE7pUV!D-WoJ~8HR(=w^f{cSJWyUVAl@Zf-eVaw!G`|h{=JPd!>pMg05B{GK)pvb1)(e+K-!4~7 z-<6~5i84@ZIW*+ef91_ zK5!OIX#L9h6Uc8f_#4UZu=tyu$4@)Se~Q*KyT~7CeW<7PQnXj`ZD!EH=^ZhDb8vl- zzn#PT^Jlcp6t)ZN4zLY&gC@~4|;cD=~wZ~ zPx9$8`Es0ym*TV1`#8U8dLhLhzYQj%^_%98&f}#{czn6x&lG# z8(Dxq*v)j>^&}Qe7!Uc`X!#{=Y<-gX0rEG@pZUa(DySdd3Hv<`0ObSlABw*(lS5Mb z4@mx_vG@*sLLN(e*A4MKJ5D@D4ooJI`qcxOv3NbSP+J+WQ{*p@^7xMY2mXwY zj>(^Ul4yM`OI891dz$@^^lvl9XZ!J;$TyUH^_1^8oWHvE;Y5+|ko{63>>LK<{!8|` zv46t9dA{h7`6bE2i^@L&HV0py|A)$-l5KBeIaA|dJ#%8KAo6Lxo+aBT+Hn7XyoBfL zU51~LyoJ5M^8tkar0=#R^ZeMs^J9B%(2+d*A0ydL@+Y*1{CqqR@Pq{FC}i!EjN$JtH^{bt9gpuMwYsx)Gm_Ev%6$G8PmTj8P;p=AO0=0KY@Sjqsd7{`3L6bSyaisus`zsOmasN56S*hV)lps zo9E|3PS5R0a{~G-3{KN&xn-okGk-zr*R(`-Il}Xsq3Q6iOL=Sv!Fxe6HC*-&30e6-gy+??i{Ass? zua6injp*S|nMQxh4Q9EG?F)Ci-8A1gJ@-eO1J)7#i1Dd?)E_@gW6y*+l10u&(ulvp zUeJs7NtEv(pSBG=0bcp~mgYi!=8pCU`O^c_^(ml_xG2i=`~=ivc*x8BmFolYQr!g(gnn$8&$(md<9u?7_;R7r z9M4`v{Bq7xCnas8KPj7sIIXcrcypm--y4BA%1J6R{mJ z#`DBP>R-k4grJyBtQw;rf7sK_&;W(bN3j1o)QzQ96rJ|x*dL2SOxhp5!Tp`I6Y&?z z^(=X@Bp3Xp2lP}rF^Tk${SRp?qWOZr|1`**f${O-U%lW*JO02&yc1qeV4NQUypH*@ z+kvA36dTryQy71@u!q0?{c-G<6mMVS=X**&w=56#>wNtnf8*=(8IwJ&`I#vnf&C5y z7!nj|KkIqMW$@Q^+(GdW{Ed_QF;l)n`HY_gru>8Q_vB}ou#nK#?c_6gpK3ni=iDp( z@_Z)hf3bXqpHG}iv)?d(L43Pz{_>~rKg74sI)53RuQ-xl_9Fch`3UBl%R&Cg&sX4| z=6q!#@W%C-)+=kiVpx9>FS4*+V>uD=9{Mw_=UAUj>o@9ISg)h>Xpi+Mw%;1k%$mt zN*2#o-0|}j$dl!!_#Vwyf`)u!biRUk>duPhE5p#=C(lQ+Xnp1J68%~75yVSVKH|W5 zGu+_!6XYXVV!@~QJSsoBbMq1MpZ}2g$UIa08omFf>;gsZzYheNogh5_!1;Yy*UMwZ ztD~lPWjJ4l|J+WzitfL$zM1#i3-S5FSo`hWTpYDN@%T(fYV>(g4#kuE_!)ddy_MoqBHy3$c!vBaeYc(GPuLHnvDcBm?oLRgc%C#d%|ZE4VbD(DI)UfA zSpUR%(b0`8UOYp*NdI@JOlsfh-b7Pie!rX?$+a(X}@8P zUl`xC-+;bN@r>im`w!E819($>8|L}GK%AKA*$qjk_ks_U8}s{W(8B<@xp-3qe2Gupd8!&l~KM`F@=H$A6*yc(saOxCGe2bm`SogP>zc;bp zo??t|g~16r!v|Pme@OEqbHYSeNbJ9}vH#ARK%dVU_TSTJzi5bePTG*f_TNtD82fJ= zs8G3`cvpEF`|otDmS9lqSJB>udUQXEd~Jg9yb1UW<9QVMe5f_D|O>`ZVoF#?D{=fb;!Qf;oRhep|Nh*5VuTBg6Lt z(%5P`Y~sa`&(e}0_tW{EEJE>+zhV8s=f(WISDa@U&cly_=qAp?kKsHVls$>><5;CB z&9c8v)3M)YUjqU6XY4<3$R9orhd+q(^(^}Qlz$$n(+B2>R0qGSE&FZ0UyI7;WF#Ly zUnC*L`f4W!;q8ahsek&oMfRt~&j0gnPG0l*K^pra_!0YAbb#}pvhI&!`3XKhFy$wo zK^;=vEB7>TPXqTfa8CpGH1MfwfDX?QY6tKw2iEh8Yrp^SN0G0-)K~S}&42d~Kd%_~ zSB)QC_?yU~LN>easa{u&yKipJvA#aW`t3E(j=cZRb&;=eRLa_fzwFED?dwZnXMX(3 z?|&D0+s>BPtetU8Ut5!~wg$Tqg!G@zrnsI@UmVpm`K3i(M>Y1~9N@1IKnbdiv!`6) z5xR&6>R4UF-zS?~gCv`}yw;|ePVq-fL_Rn+rC#1kLm%Bx_6 z`l6R;$9WrPAv-N%)>>3KGO#EfFk7OFVp3v99>-DVOKP z9UqV6UlcG9uYaEF6NM)}UgTQ9LoxB%NBQ3p)4j4mJo>0Se#X{nTCxsJM`0X7_M{?u zHt00TWAO)0#cfG8**nUX5Seki8T76v3B5<{qjJ0A|zRuv;Z0#;P7r|3D3#NY9i zLL22>G`3n%tSd&0r|W4@^&5W$inZUXTV-|`#*Js&sAqyM9$s02`WY>?VdPQTL<6I? z6XS8Fr&c?!@XfC+NxRa^7#?=Q3k`>Yc!59a&~Sk9WmU&FT1@pPQCCb7JqFdO4x@?q zgF4;vr4M+pVWNNAQeUc>@g7UP%2KECsDHjhp#NHNqD>$^{tfHo5x3grlhPNZmfCN% zk9civhW9gkMICFjo9St+KCJk7eSUV_Sh*XkZk&m|G6r!k6K+L*enY>C*IrE$np$JF zH_b~uDLOCRf&pv$c-pqoeBSFuo#}C6kd^;<^ePXKI`$(iHLqRf`jfybvxv3JW?8%V zw}lOBmzn-;OZ%nBnvMN=EcGf&9og#j*Dm8~FMMyT_;#acK#n85e+@m;i=A*sf|lThDSR9n>GV%tkv`!7hWXwtqvoi3Y@cD%X_TVW_4XpyV<|T`8s#Em=_X#TLY&@8F5m{c1|91NWq`yAX@qn< zDvOPi>zTaQlIKxBM(c|a3?nn^$Zn7r@b_Xp^Z_3;t+*s{7 zR^2$m?!|DihK!}NPl`9{{9X(-)xF!qzLE;jP9E(l1gd20eH{sqs^ba#v?o|DhPgy;%F%MXf`{!Xi#^b=?0v&s{XUTyAZ#RdHx zy}#uAo6qt-Z)ba-cm0~?lX1r;Vtp*??H#pa^T2~*F?%mFMm;ra-_rFprPymTe)g?h zf%nm2ICk5rj5z}Tti&Te!=fWu%Q?F6aT9((d_PWX|6~1+J{ji{82Z;W-hYqcjlOqg z>|f#)n*W*GT3su5;e3JSlPG8GIIxNH3?uD^&oEl7np>_tD+K@_*+8JPIPs7VGO8+# zQr^t-f8svdXS16<^+L{z-+f_XEBoHu-~6`L_E)p!&B*)7^dBEeOD%6nS%@J&2habh zg4AyOQC(gx>`?)p@cf^bS1`7v41ZgW*Y%$NLkc2W&^6w(eF*+bdPA|{Bq;jTI@FbR z9|4>n%yX*izs=!&@S`CU(J~Gn-Gg{e7k6tDy?9Pnh6zTMCo!dLOg!nuh!oH1iuI%X zr5>!%VU;Rq&Wm_XSEd3-uhHjpNr?jS9J*<6FyDB;1o2X(aR$B54`I)#_ zQYG#8W&7iXwsdRNJNXZ__gL6ylqPorM}<6(c(=AMCnUa}DK4@1ugLbf5~Or&ZBP5T zC*rg}W@)c_wzi)k6yo zJcKlqvz95Q##il>{U1Z)8?{$-)Sl?9?P;Eq3M_8Q7&#_NPmZ`!d!p$P)#y9^WPtZs z5{1Z{rSeLH%EJH9)let!cj0x!6ZLN@vc1Oyv^W`Xw7wAk#TGg|!k?F)89I()iOY|{T_* zoKK2QUDfv~H}~ zQMk6zyv?;0 ztGpWvceGSg)~wrEwxVUlqcu(b>UAwE7u7skzO%e`QFY7a))o2nD@EJlmSrvVwZ(b9 z&Ai`LwTo7C)weFo0}uZCMN4Y77q8jgSXn&cBzo~WG^6f>{-J3QR2I7uczO!}{ z_^vNo@pwabVKd}yfvz5}FDcCP)fW5fTYa$4hKl;OwU4hR8?4;W>@8Ziy0~pkUPJ4K zruxm*?IoMMH8uX~Wi2I5?WG&HRe6g`R<9^qlfSuadDo`amaffZconSf+Pt#D-`KjL z8~Q6>o4?`l#+4NH?jy{q##wXLgZ+AQp~y|`j4`p#>tt?07a#arTkd{gt{N0h&* zYh7zgbD(a=(yb3S6y!a;bWvX6;>A8+gEzmRacSPxrOUhxTbC>=tgjPv4b5A<4ZiFB z#L9a9+bZ;{D)_aN+o|N;;MwH)1&bFiF38JU%4t@3TRMDo;=QB>KPcn#w&n%%7~7R| zM?XF}=a1aGzVsy+Cx1acH<4P5wa2TOwtKoZEj1eta>4IMDg)x0Yq+0>d*dGSNm_8J ztWUk)^Q?Zv{^yDR&GEI=Z#Z8NJrjK+ClGJD$DpUNju_})Q}-*KN5_3bf59&6ME3)Q z!}XN9x6n-k{G7t!TB`0#ovvr~Z`cXX|L|ABsb5|U_*_FUnr*=oeW?K-{Z*t@&X>p0 z056D@mgrh;f=+dBy}pRHcJ{tHk}#p=gX2svA%{|S3`Hzx;;RP&*eEVFc7pTB!vQKj zP$k~;p4aoP(I`%d_nc`#&m-cf7B<7U3MuGg{Fvc#C!~P53VP$tTOiwU$uR8Plw1@h z39kHK=!>y$LM{apJ&1SuQaRCkBvAQ%7z&L?zxq1x-=lpzdfMN_gFk#7_zwUle$H?^ zFs9~}b{+noN8&~FRni2nNjYfDc;mhy<#kIFsQJ|ldg|x0gsZlTBwTG$OJ)B#vY#qx zGNtFgm;B@DQoN{n^{nLcWeHdF={4EUfHa_jZv;GD4T_%DqD}=*@#XL>vi^?(%)W&y z(dS_tDksH-y@e~muSmFE!j*gz(4X=hLQRzTNfvmXgiirnwI$FOC44I2#x3J+16*$9 zIOLdw)3`WIjF#i_j>a{@OQoFDzv7KZS4p_FpIQl@Zs|wqi~2F167?ARr{|og@|*!~ zq(6Qg_!R@(C~vORyOqz`5^j}uKpMy@@9Pq7?dN9_Zk6|!hJK9ve`J6g<;|4yhsHI= z$h_;ok4QN2V_eOilkkTu@Bst8(Y~)qxRrl;iVhXxXNtJx?~5f|jjMdINy62*3Vy^u zZyfhI3Ac{>O9{7*n=0omjcc4QE(sqsUnHFRQM`}F1-;;3b zxRke0q1=V0l+qo=H7W}3;WfsZf)m*auV_;pAA8&C9e zjqGN39@J4?b6Ych=CQt``ji`I`E&D9Gm(vU!Vg&4IxEst`9)& zGGC_<7e8D=zhhapre;2VPp)BypjrWatn+mib>jCY>$^IA1QqmXXo5C+OMQ%d{JFcaS=@^8=;tfy+UV!;R;}8wy0qEb;%n$!-QL*Mg5hs+_s4C}((Y}n>uj%U zmB;bu3~}xgztM1 zL}aRE>gkG1P^ZfLLm}ldt*VUn`2>?`K@{$hcV}cAQwSBWRDBk1DCP1ffV}#5iks(T zK1QFdQM{AkIGTg~J%BX6F}J3cE*X41R527azPdgu+f(1hQl)}-qY)Y#UqJ~?JXv*E z)g~wuzlz>k|L=1A>$U&Lf>3;`{?Z}5)h2{n6;KrBtCe=UY(GkV+5Qh)uAthh9C|+nwvJ6r&UVl{et2BUPi)YWysYj2KZo%g-2eap literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k91103515cccf539429183d92cfb39807.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k91103515cccf539429183d92cfb39807.o new file mode 100644 index 0000000000000000000000000000000000000000..0c7c09c85758569702379680ae992c173fd2d3cc GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yRsAPh6{Z0i9+G9V!kug$DzNevR{(WfPu zmLVrXWF2ELm@%`l&&C-t@nmQjGR{tB9L?rzf0 z?*0CHbl-c`Ai$p3Fw{}^z4iZpRsFB3|F7!S?JqA|xm?#Yri#X1VX>NmGM3pJyZ7*0 z4@(85oAuNT*y?_s?Y8|NdZFzzGZ;HOWXtY*2d{Is?79Df>v_EX758X5QPI-%+O8jj zemjm`{cwXG`k=}dx@KdC|DTQZ9qnL;L3^YpgB5)*+tdGHDSPWzIy*cgwWsE+ZF1j` zZ6ft`q$iyf{a@YFKa6&|W_zm+V~?DAda&~Rfx#W`Jw3Sc^wWdi)IC>0`wU)(=bZd@ zxbMXA@T1P}gpVIM5gwU;GTb-+RCxdD)8R||PlreLzZZUL{=4CaTF-}%pL#c(GV|fk z$c_VpPd)wgVCwwEp*bzz3166ZJiPC{pN5}uz8P*laVgxl@;%TMg+d1p4sJhjU~u33 z`BULJC(na->gSI_Bl`~yJ~jX8!O*D_;R|>* zzxQT%?#Ywk3g>&_3-ix|W@%_n>l1^0&_%`hHEIe&t7`v)rk+n0DUJn|r5TK1KnB!0DKfaUNuqOTvqPd=zB4Sl$w1b*kadTrNh$Bh0P&mTXqX>*lN zlK&~+8-VY<0^hvndv~_)_1wVsj*j7bM=icbKB@d~z~X<(UvK1t*LM9d6e*qj*6;)R z;S2v%a@B2TX%~N0a&_dTW&OWi%lcC>KhDl#SBHP7Uws{ZJ(SI^o-Lhpc&I@;eD-hk zYs33lKg}bi4Lq^}_{g)YKl0D4|I#b^)ptr+|GE3v)qg2u{pZuz)eA}NY6@tJG1rFo zCiD;g6YC#-Ks$W#f9h8+zQYa=|4l+@_*?Ao#RjaY6IlPqk0t|_7#eOE8 zp|czIJVJ8~JRiQc>s8ZS66?E==OXwgS%$$YjrlIg1Q{j|?Xvz6$X6ld8UfEE;CTc*lZ+(aMesQe{d*yk^Y66KyO3k#m4wg;GV)SVh-p1><#I63)j1eHS!4aQZhTzz!w|uKgP1jD z>;q4vtENO;(f;GA(fTs>$8MHB*At@giGGIW>E~ml#2@2Y+Vm?QS83eGT$!G$Df2v` z)J4T1=NC#sWE-*x*@^ml2YNjR{kaf71IeEL9q9j?xURvg68Am09>DbkUjG~Sr*Zu^ z8Xxw0=h=ji4R%%jXB*G1z61TcX6hjq>~j%*OLqD-Y(O@@0GppJWc@?1AK5|q^!saB zf4G$OGkxF@vZ=EFm$M3;r{|5WT{DB?5dZH(E+Vab!l=45u-t^J`s4Q(I$@Eof`+^et$6 zY(eLa=C-Zvv3f(7tUtCOdehL}>i2fC1)aWN=K>*GT|-xAi?^er&fC(m&D+`J3)D4u zJ38wmw$sp{JgxpLfqCB z6fDYnIDheyy1G2bDdf5f#OP@927HZok01eGhcECLB!ZO!zLvYf+aQPct`MZrSKqa* zZfn5X>J!$n0=x@M5KOZMS{edge-oc{{y=+YduO*F%?a1u*3s#0>&&|g+_%B&Z}kNl z?g}%*G+Y*|Ejdswo!o#%z+GTNC63u=IrOfX!pgQ>pz9e}1{LF4tlD*mJVIt7H@@6U zL++&^cUoE)$Mx@}A)mAC-L!0xb*zA!PebnfeT-V9*J6XUpndDs4qs;;tE+2!tS*XK zwQNXEt+0_Vmnl@LR?RKF`{TtSg@FUk5<6wV$Cw5US#J1TU(u zUSCmGQS2_o{vYQkqR=4;kuJt-D}nr-LHJ^6SEJWWrwh$3-p=OsHg{)xdrQYWH)gia z+u?KP7cN*d&%K^53yPMqlCI{KMt4VdN2jmV-P+aB>2CCGZEo|q*Hu-mTwAw(S37B7rwS?icR#lgtCdu+I0Ea5wFwAG5&HXu-F9J+oiuh-=DBV5Y} zfkJ*UdbTmvyGx6(Uja!58cJhX-9le{lH@qq@tuhgegKyOih3toK!te3}V*^bdX5NF0Cl2b}+0$Y-DjKWWll;0OH?J=b4Gr$+psu9JNJ>adOY zO$tte6}f#A6kgDW8R1<4=Op4c^=u@G4W_Z8KSTehb9vw3{W;T6^Vg$4^#9~O79o4s z9kAz66}x{I(NckYm(ti*y2nSjIS8NYgnsn#>uYOkxx8;=L*B_c`BKCI{h!sea!iB1%$8Lehqz~e?4GhPQsHC$iBgueE7%!_#aBM zr;|C4_OZ8q*p z%HM)>=8!ls{H$5g`fGf?aC`9ihxVi|JNoDRylvCVvz}wLu0|qzx&1R?-)I4Sv8t23 z;UAdKxqQCTeEvlmD|ucM9_!rjusDF|FOi!TkrgQ%}ME0B>^{*kP zRuq_)5cOBA2Q!1yrV)K&%-^PEQl0zTUbJ_iz4Eu!q~B>SlfOBDcME$So6hkL)J1#p zM?3J+-{9ZS@64!vmA#1pviIAhZ@Nd3`lj*h%0Gunp`T8Hm-BhF{v}$rhyG`hx|M&B z{`n_TgvWU0&o43c*SjktrtkVTi@s@o8}!ZRYX;YMhSVSYKO?H|`fRKhE{nchu9&{7 zNMEkl{O0om?WMlae?}16f&h_$Ua|ht{NM(Z`pPoPOZ`RHgP~$po;BMf-&p$U-GzMM zESk{zmGdW%-)8VPlHXzRH#?7?c9Q=Tt!H+TKhXM6PwS;Jh~{$2(h;^#}MPNNQ>-bFXE z0DrKX>9p%fES@kP^0U$MOWN4_B=ZC0Z z{6}N)9r}bkmiVq4;(KPWI*T9r+LZ86O>! zKldcj`dXf>1Q7N#`yc7wW{l7F<2#XWDEaCs-*Gs9b?w86BHtnVr9{{{49NYL>~mxP zgn#pV(IN9ol7|0B%#8yG%(|kQkwo$a<{sDOj&)2&Q zKO=dIdV%Ky2>(gnZA<3)v4iKw_S~Q&dGvYq5lXb<`McznL_x{I7o~rqgoENPlPkg4VBTiR^NO=Ql&s;a|y_QU4lRi2XoLCd5Yp{V?q} z9Nd4Bqx+MroGATy+HW|LxxL7L`2NJr>9dF)_}pN!BkE7cZ_@+rEH`?O5Iyci;{40s6@en*AUCKAr5#^BwS;5j4yP$V>RB|6kH5pMrm(P<(t5UqVFw#oVZq ze$OpL9q9sfNb+2u{hK2preE@pS)t>O_s2lMa6U_JtsK@Y-m-{Q%2jr!?3mgdj*f5`S$H>R|pAH%le|hChvCZ&Z;#rBVxjncptr$rn+hVpuPRQRVSqI@vGvIaEbFCw(VkJ7SFI ziHX#|isuPIF`HNoMnV3tr<@~KLmIk^JTXK zM+GQ0tQV&+{%&CpfBpO8*e@yGzQ)h@lzwhm9_-io`a%B2*XJ`Pds_1|Q$7Ow9SATa zDAIn`^Nh>juj{yj;vx7OC--Bfe24NGKM73v2j%a{&oE&jp|9J?XYxMPe8$hYSNi4o zOw|8k`3ygwIG1L>Vg7>ncHR8tPvL)vZ=ZGkGCE&zB){xM`YG}e%r}>V{E?rpz(39T z%0l3c>ocua)_ldV{vuvvVZFw3BH}&tXIjs(KAYBW)U&W&N9oZX>rrgKHKdu>Ys^=| zN7w5y^A$e7E%^#<#;FK@aO&`|_@r$Bb7; zP4UWbz7GGnop=@9e`9?!@3$A?^M$eY+q=0qYJKAInU2)x^P(JzC-?FBK6xUf97=PU z;uF?S)BbuZ#ivBRKj-lb`BD0AJI|l6A4p@bBY)kUkVx@7X=0j#@}Z)jox*hj&v&u@ ziSweP8(6$}hIoe%{?if9Xg?Im6#3n)$1~G@!yLab zzG=S!eVgJL$D8*bru_!+rua6@^Y2d>&n}z%#Tw7>c~Z>&;5QxTNdpI^eqcZBXC~9< zF?@YKhtC^#O^of&ALRRUetydH?Kc*JezIXdeh8m8*eCP-IQNhLLi_R9c@M_J{wa2T zh4p{*ethb*8^^aP+~23hk8g+{Q>OC$m)ehyj&G*(pWNWoDVF#aJO8onZ!v#wV!u7b z7~hJ56Lf|Tu*Ck5=11m)iLj8^e`jO=oi%|zpEK;gr_p}V5bvC{A&Kq3oz5}#-#AdA zay#*^>NfV@=~yknpxCdXy$ki|eiZrI1mk%V@EOMQDDwYI$~VRSlh23D=>C)Mzsa9a zqVaC>WadVl^40STp--1?&?oGlu3PkJ+K-H#zy1N|`=tbP{)+szeBZ6bH{?f#?+2u@ zRdm?Iiy@z-B|+||^E+9D;vs*-`h(Al`FXE6&oG>a9|h4(oQEI7c{nJ065q$MN>iF; zf1Rddzt6q~0`AY)f8LNkd>#&e5a;Vz^!X|OJW{6*%oC{&epgxc+kC$kmCwmYK7PJP zLW=d(P7cD`52sWA^l^*qPm7)Z=iQvV=JSIz_C@d`_Os{!=Rf7$AI0($e12fcPd zq`Ft`Y2cm)?rGqj2JUI#Q`Z0;o+H!_;9Cx?=NH$0|KX1!Uwx^s`nQ|^?jL?$Iqt6- zKf3TYkwZmncHdLIu3C5B+?->5eT?5jg{S#X7I!RJZ%o@xpb#9l2CEJsT`d{nWn`0Xi|=fxc# zkK|tzFc7bQp6e5ZCq7=}TEIgw@!CiE-xAZkvOzrhs62kg)@WL?4oyd4976V_B6>FH zG|6M}2TsLpNjBL#%9aqBal0Axt|tk-N9CrG;^3olyW-=ca=V4x@%o6$1x83#B^ic9 zF-T_rR3x*H_9(K1i8DzPLGT(76-IgZ5N@s&aw zX|?b{WQvXWOV}f-W9jS%LZ)Ewy3fQQAZUqqY;{ zai*u%IIr-{uPsTt(#segcESq{hk|&4f9lY1fbnHj$2VF`^(RqROcFf?)u|4niTHy$ z-SVXmc(7rjf7?=DqM7j?OTF4sr}3zNzC@t^jp9U`Kz#fg*2yDowaq7`FHSA9-)bN6 z+T0B9XZVUb)@V1=(^!32@$vfn?6|RVH&)#^6MJP0;$9})iv0YBeig61nj|!}#%ynz zmwHlkUb+PX*7otVZKe6V*Nr;UM_OuLyUg_`fmdb`YnRQkcJXfu z8`dr}{o9uIOOQ1i`}0`p)s{N4)$6Za#?@Z<-dOSNM$v#AM|#QknO&cbDi5g8>KW9t z4Ryj(ohwrtmx>IJb^tbQ2H04m={YXMX3rkdv4+5ml^$=gT><5AvRAk21$F9m2I|xm z>6%8(QT5n9!=}?H#i{G;#jeLvZf-QnMaI%iyj+Diy_H3CEY z8zf6qjqvK z;;L~?7W#w4B^o&vn`{X;=6-L&y?y8(X0FCt-EVdN-+nasG)A{_8?VpX=g`{wAG&Uu zsfnXc3$6UbqqppbO!?5L_R&2jpMwd}6@-=_6f6CmRyXM<&dO(%Cmy}p+|h~)`a61m z$@w>*<$d1H_CD|WHO(jEj!ne+Sk&7)YRBe*2gPFcUS^DXYSzA`>#NJK*Jk|eTe|}9 zqr-6QwpAH(1pZlxM|_4wN3xc4bm8MB{DAm=oY?-y`X7BV&LuGPuWP*j9>p7d@66c0 z#49xaH@7vqR^h_=0?j8;&e(Ba6XzL5+6|v!v`96#Tzggs06wyTKxc8{As=K^T^6Oh zndkq+eYVeLH+$-ZoEN|Q!o*hgy}7^n?MB;Q&6+nO?UMrm?4a8$_ih<9uI3PR%Rnc@Kz zW0v-+XKVWzLLq*SB9PZpvi%3LPUF*eO|0$7@6p95{s-g~aU0sxt!l6AcnBR6%tJ^+ zIcu3>YJAmB+5a&#zEOKcN9~Eu+Meb)sleiCqBHMdRK#P+BN9zmmUt*!dBm8;!nW5u&_VdFr@&9#X zKk4zJ-Pu1D*G2b>YZfjfzljL-{{$e7k77bi+$4b1v9UWjpGroYF=2|sk%oxh_M|9$@!${ z)K$$dbye-Q6|Y_2QM{~Jl+slt1ub0-g(WSU3fI$l$&T|xd> zbjvrl2Aeh)RzV-Fr0FBd>c#r^{qbGXMJUT+nUE$kquU?Z}t|iT~*SyI3GzN~9gYfIPWa=Z#xb!}cz>2GXZ-wpj$ ztjS;hc;kx7mQ5Q9VDFmzO>Jvyn>Gu3Z7-?ZioWw2H&%98?cy!*;#ShB(ysde`JjQnA+|iFu z&iNzvt}lHF#>roh&rPHjW9{)8rtO}tO-s$jgIw_Yk;;I$<{Iwj;oi8%e3BMiD(h13 z_dKf~vHyAEe{*~-^&8F?M9)Ou$O**T?lI_TtRn{c*VO$A=h1QB&|k32I???=;cz{r z?k#lF06(X2xR$B=GN9YYZdn)vF005*zCt)1XJ@^FBP4^)Zw zyytbiYcz^e;yq_t(DR5ms)fxku0jg>7(ZsX+zBZlu7cjU^A^Z19tl)_ABIBX(XYA={P$=dkDm57@!$_%2mS-ViJvpv4veXJ zrCo>r=aG03eYG^fYf=sxGv2swNO|4T1ZsXYgP!`iEa9qcfrP6~YMJamNA^=KO{Vnx z_mY1+U5Xbqub!2BzAWKtKD{RU8IT54@C|^ct3lDz8q}%aDZU)OMb`gOfZ4ZjCHg#! zL*=Bnu(xm}_!SAaOSqD60{T!%DdEF3-G_Em5=3NJV zM8b(5<7)Ptgg<0~4;bi;_I*vlt^Ctdbf^$NQ^YNQUnJpbT;+>R60XKo@FNC#HT;qIkN%*MwBH`4J;)Mj*Vt~sd9xm4r13dcAJ-{rIgDH94oSLyO z7~n>^nj~E5LD9EJxK-X4CEO}+nG88<+&nq%N(0h`o`hS+rM!g- z#&Z_9iVp)8xQYYA7Pwk^)%mCDU#<7E!58sX4oS%dRKb8+8UZ3 zppNRA+nVt+kM$iLx7Mt!t5q=6?CW4-Z*t_;f&aYZ*wl~t0v-5i2q|iFeE@Qo`#OcV z_~8=z9n137we#_Nat%8K)pF=#t*^7V6Td%M-__|OsGvtf6SUD==6kHU!DoUC?fIJI z&)tp9;#Q1DKVMPTMn8|Ya^?C}WzF6eUqk1r_QtLj41bfmKW>ATc5h={XM0^EXG{nL z_5`1zDr7NAxV$}}2yS&OmfLs}HuDCyb<}nEeGQ|3S2WPx=o7#Nx-MpBO9qPAR@6ca%6$|VqeBX;8B2z6> zPgi7uI#uQ$3MrRqRb{l#CzwnNqHvGAJ0s(mLa2D9>a%b|DVIk98guilM0S)%97~p87VHY8AX2jnLTm3QB0=$*RNZHbJ5I zRrJ>Sf0yH5ul+|BgyLKEmk!~rHX+=~fTA#8t+d-^`%&`C_Rm-vSo!w>AJ2Xc=|7j{ k@MJ%71=U{V(EBm4b!=*Kwu9dB!{btZVuPOKWo`fe0h8zvxc~qF literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k91346849ef2de43f93d0e96a65664eaf.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k91346849ef2de43f93d0e96a65664eaf.o new file mode 100644 index 0000000000000000000000000000000000000000..908f413d9c8472f2612dab701916b1ea75b86409 GIT binary patch literal 34072 zcmeHw3v^W1dG0=EM$$-#(MUWj0Y-x?%)>k+kTA4y@8|(SG9V!kuclXwqyY)^m^G5j z$dDBwa)~h*OideSHO?i8o6E?MI=5*Z^=38ljTdnoOtG4q>y>7$5Sda5FvSyJ2NL7C z-@hL-=Nt_Pup1l7hM9Br|NndM|F!r3?>(b4UtGFknXYL}6^*^ZVl@S2EVCze@8P!| zmI_KY>#pv%)&4x&ZTmm;LfdDjF?M*+mfiafUgvDtv;P&>^LYIm?$LC-qNVHgoj(Zu zb_~1r!FoOPex)sR-Np|8KO5^k+RhGx_DFXIEBs!zr|*Lj_SUa-c6eH9clBA@#NI*M zc#u_Es&%9zONVK*jk31KTe@GqB?HGXvkyJ=Z|{EMA9ao&0vV z_r&q=k)?;Y<5ZhllrH4nIBj-S8tV=flTOy&F!M{%B}; z`+aVlKlJRJ_ry%c_W74f_{6gqKW zVD`zq10nEyVeYrX`{uq0zMu!l-iufAtj~qYq00-8fChR?N&I#=^zNJClr1O2v(8@x zUl+q;rau-MCj2t+y8UE$-`?|R_a|t#f1n(&eTgT-!w&_;dh>E*LS{l%;>-I{PBI8He2~5 z`JeK=e)!%i@XhW|6|L6k37fvBLBkrF1?~(d#8l;oqK>?``3l6?|d4&b|HyfO95>W=GxGn zgubDFW_?2sX@@WVFa6rZci7>fzexxUeTyBwSdTSz9P1nY(L}%!LqiX-!~gaWX#Yo2 z=mv(FvJIw&V%y+D~Y%LO!Eh zESX*RK%NVKyR>f@@|8=uhQaeNcpe7NBqPap5qyq8|6a)C{5vi5F60<~B_T8nIo^dF zM)^sG;a3tv!;lBM=({kE9live48N2VVp?}xxg5-MZ5GB))>t2{8y{8vFhsE5AZCpk z`@j?Fsxc8)wEw7bq`r*(v0J6j^@M1AqMxRD`uG?r@yEEHHudU9l^XXkSElD$${bH9 zwV){E{9JK}Y(q97J5hh{K(FVZKNsR>AlcKm9sPd;*VTAc;JzEz1Gt{R>wn|^G_LJeLr%!LG{xY-8EAcc6dQbUoyPeJ;Xp$xgq94anvfVDqyJS>GV+M|My?{oWeZ z7cOCaOz(e;Y^v;k+l6T<_Xbi>pMG|z3uI_-sa}5-i}6Jptj!I z-cc*D9lqA~wt&AQ(6n&D))rrjx4CI+YyP6P{M!Fm3h|%8Y4fq=F9YF%Vc3?zXfrj_cn~Lq22KyJ^`X>sSG|o`&50`xv!IufYauUfY%}?Y@pYR$JTpL~Rta za_P!ifvL^IR;|g~&ZBn}Pc4zZ+*3K(%_7q*AZ^$!Tgws-Pqc#BpKNy_e|!9?CzfF;J1tse2%pfSyw#UzXpJ4Yd=GWAXLTc2wqfQ zwXVFhyvSXG{Xfo8M4>|xB3*>n76SP@g7C$X&IYfWP8XV*y&X+$t?rJtw&wOZZp>_- zx83K?UpTK|j(Z(l<`phs#hp#f4es`?_6}c*yQQg3muJyl!WE0k_Y0F!F>v zx=$DPxMlW?*!`^>=hzxknSF~Lc0t=m?V|4oXmkb-#L3W=bY_=zJ6ZZlQM7KO(G7K- zxlMICt1{xLPpaF{t!;9I=__!|RBO}hvr@TD$*xrFGci7l;!1|CGH_b~-ln87Ke~6p z5v%aByFdej;Xi-brwT8#WvylQ6bB>k?zZ86k%Vu@(N+syTY*5Kap?Myyk3{r4{ zMenu=ejY#tWjbExoa}JGZbu6i=|7mp2IsTfpzQ(AH;Y(AKbXW`Kg4DP6Kubk^u~5x zAI!EF26Y?Z86!O83yzOuqn0oc+QxV@f=POm{$v_U4~$P3PyG)ZN@C}To^kplq6a=V zXdh2!ouYrD=hIBkqkrhbM&kIZ-sk-1LOugM_(_xY0zc@F=(+wfIyB-3b)Dq%SA}iF zZ&GjqtjO(~pzwk|%n0uaI42Olsb?ceY#@yl{yF+boy+?M@6VZrn!gVHq5r4$u?X42 z?tnc9E7^lPiIxiFyOhSh)HOE3%|ZBFC-kF_T~||6!{vP=8}d%n$(JGy=m-7h1}9D= ze;dp7FY4^fOS;leS=QI2d0|iV4|_PdlaoEbPho(KpTNf#^`4yx5%xX+?bpx;`qu+C<|I5Rf$STc&WDfmga5%a zdn)-e7E1qS=m+?WVD5BUH{oBreQpru7^=}Gl6(W%tn8~tfTe=*FA)91)2~J%T8w^{ zlk^vn@-6McG6B^pe>U+0|0)bT{4lqdCdMla%$UXHq4uZKFhA!+*jcov`II*m_1O`2 zN!IHwYtw^sX3==kobV^Z6HPtoW;%ASeF?zU-?%lg&kaWPfIQ_-iiNllv>iBmde=Byv7v1gB4@ z_{aV6=hXjlQ~y0Xr^ftg@F3Q=1RMDqUldNyhks$ckNOw+&(s7OZz}hnLuAjXQU4lr zYK4I*2~mH=dN4gWWeU+J#{6wcCe^vW?Lm7N+ADusMf#oMGWnYWc(<_Uv8f#IKwY#a zf3yQH{SE#N{Z5bSSJ|5wAbY<}`lfpnsc#z3uKaVb1p4U^csZX(>R+N|d+2{UsayF6 z>7RchMR<%?_SMCv{(5$1#PnU)YSA~%Z-c)1e9hqc&XD?p|7S$?U6+mZ!e!C7%N5gi zCF#o*o8NqXpuN;L`p*bLTM!`9-y_yvnjhSNQeRnSd8xnXdN5eT%Ccsf{$}U#(@yfAqV>!!@&{TU>S(9B>SEr0-28#2FH#i`}6S;FUpd0c}>NGo}F0wRs8ak ze0og294F$X_^k8+&TpDtNb$#SgUM+9run1uc&QT}UvBtw#ox=IL;QS6)oIjz)H~@$ z7T^zdF`afjiACeaLVh+c@A$eoq2G`2hTf;_u7kkktNt zlK*HdzC)jo#}eOlLwwJU6VH(YlS!n0^+0AUUJuUKmIv$ow?X8$Ao+l=wqetZY=4JBV4isq&{}+uK;q)Oc9WoY*Ree44Lk$u^2M+&>^M;rV)( z;b$aoVGr3e*;2-;Fa#B(Lf%$nBRkAPak90svj3!*{o()S z`8l7{b9>U9fIbU@Q*>Hx8R_rLf2Z|pN+P=w;rY$rRQOkNX4Jn1=VL#RlL_%rKtDwL z4F~t1j;0u_|!h?j~}G5XTu!HB4;CM#9v_# z=tcV^%6E`Y+xqu{SH8Zbxsac^qy0ht^uSbo66hl?%;%xX)bv2MKAGwhF@Z2X#>>e< zo$oi%AMBe+?&7EU0C@|8nVHdeX61+Y@4S)g1_{Do=PVsk#4g8A#GVSUl9182DvjZK0f@b2mENqANr7Y!s`i)^Fx5wF<*8$ za8!U|!+LQF*g<%6BNA@sq%me^CCO{45g|68gH6d?xP`&1d|ad!=8V z&qVz%me26>iF0Z8o8~WwZ#T?e{tW(y`1Wb%FC+66NAk;Fq@N-m!F+Q$$RGLn3jEWY zugnMDxIWW*WzAO%>o4L(7S?MlCnDZMf2Q>u>$7S7Mm-Dbb(9|Mu^z?tTZ5W;y~cbc ze004YHDBTL+mf%)W}J%fN4}p>_J=)j{+Kxt*oX`B{q!f8uaG~)%~z(4lCQX@-8^5( z;`xd@e!c>EvfLEkqxnkEkZ+94R}fF#SSc@`A8P6uRLC&KTAG>cxlQ<92jq! z8~lEZd?ZUO_!OT<Ie42er6(l z9>dq?bNIY*=lIzE{9(R7=jW$9-+p60=qDQX#QlA8{P>3WF=;a2f2sZW$oOVD|H%zbo@9w{vGX76{ucB17WUhd zjPb27I8JBy088u-X?|pm8xISK{dYF@-&y16^Et!*dkXCr4e`!N8 zB^vK0PGoM>DPKK5ANq9Z27SW*>AFRqrv1q1`RgBWzF$f(=dZ|b%l6$~d_#U@_h+QTd#V>$_x`0e@)eFsS)K6B-khG^-V}D`$FKbU zcagX4Y-!EvX~*={H3_S0up2>0|LIJM>-qG>QB9LyTI6+9V-L;&{we{KpxPLF%2ghr zi+G@p)iwOhGI$^3O!*zQw~X!4mLQO8bqw^eqUSghHC7FPP4kGvJBsw+P8*Ym_LKo= z+8o}W8t>HvJL<)$#fkJPO7+;avP}~OFQ7&X1*b8tvSL7rMvPHKJwHM`@iDHx3Pz|e zdWm+Nw=q`U(2T|Kds3`;N_X_d&IMQ5W_+H6@2RFBLF~~I!E&_3!$%eCir=1cd0yP{ z@kstf0R!>+=ea&nc;e$lt_3_46R&-g|IIPoD;vb4kILg`Y_+B(>(F!*#vx=+DxznD zPLn(qf8bQymSmGXBWwwg8Mm83?|PEZdsJ>3DGoj=w<|tADz{t69j}k5TwsJ`Rgz&y z6oX{;Ped~NNRJ{*m^hO(QO+RU%{Uu52hE(@bSvj^b*>A}qNAb{qwR&Ri^^J4;>}p0 zr((3xn02%&Kn;2vfhTTA2di~`<-?eJZtg)nv^ne1JT_oeG4dl|MRj_Lp5sXT9bYN5 z5#B{(s};q%V#Ij5kp@-2@mHW&`#ri9nutHB z(=A{6fCn2U`nN6h#hMxKvDB+9bsCTQ=Su|o-ylx33BR0jFt4TsrYt;6p zd8sEw=cQXPU~L~y+g6&-dfliqJx&a=@*j_0@{Tx2xe#LHEP)7!}f++f$BV;!LkkXRv&kd8-W zv2k)ellNNkJnF}2eKCSzWM&=N4H5(XeyoQ+;A5sW`qNk=KZ!9vy{H+F`%jM>t35}n z8)w-47%tY3(RB84@kX8BkKyK3T*L|$A4bL<+GEB)Nwci(x^?xZcwyMdY1&m}F={6# zBd!|PWT8JuT%wU9*uZ11zKU(acu*{6?_tKMr)KS2y1uFudu_(gzO}3H zJ~|A?Zd;X6N8q28c*JK|bR=szM;AVB!uN~s$BFHKtpAZG<6Htm|GLKe?^e9g_s)#{ zOT0q!KXY5HYvnGSFVK7v<%}H%HgTR|q}}uxMg^+5<=VYm0Pv9w1Uico5BVUYs?sRs ztvvrH?z4S1x!Kb%V z+KoS|%gcp5D!>z-|MT)P#x|GXZ_Dwz(er;uL1YWM#(TC8!GB3_C^nn~MZa2yy3+1r zfb)ZSPIcqAS-cN^G-M)L!r>!(5YOr2Zf&9$&*{oA!Km^Srj(6|C%qVv;yGQhew4qc z=X5DrH0MP;rz=x|qu1zjx+Fx=M+xrmhGsfHrn9v7$a;PB17!h-tqh`0n}B%je=OU5 z!J=Agd;Bq^<%~=8yPW<=6U8AL(0WH>DRz{G$R5@{sa=(v@QqY46e}(Hh5SrhEUA+A z`?CFULtDDF>Yec4z-sTnio)*DPE}eiISu{{=uAAH{?kzfl0GV`GuDQP5hn-xFv2FUj_~l9I-! zVo@LS$@U)0_{U^><-gRPT-w^c0F(FWIOBh96NpjV{GBoD_{xte(7p%lJ(ltNWqaj+ zYJAmB!6yR{ul?I`+S8P(Ti38Yta6`#gSjNzam@0O}YK8 zWjpGroYF=2|sk%oxh_M|9$@!${ z)K$$dbye-Q7Oh#=UbM7Gl#-Rj3z|FY7Zx{fT)3uYTT$~`@A`($jT@@->s!{>Y%g5h z<}E5&S+cIUvA$(l$EFR{&Gr5@Yn#h=EURD82-un$!k0ELS-o*XMPuFib$OdMRIKo> zFWlZ-QCYKQN7=IGWslc1`m5J8FJDmec=?X<4GXH9H?=IwuUjtK7B??xuG>(Y=i9{l zU9n-ovd+4eC3)b%U$rI+Zr}hw{P6A>d6%?tMVJlYC0R1H}Ak0t>B|+M z`UOk6XuO8?6>Y`kJ4)*ouFGpK+gcGFw`}VwZ_$#rrDa>!N}95bt?ODgEL^wkmgpAb zk4Cp_Q%kUM)51#Vqos^w-m-~wm)};pVC!mc@v8idEnAmvE2{3=xV|tDcg*q~8#aRP zy0T?Y)^`;)LEdKQ>dCs2!aU!GVt-wW5B6DCQP;Zq$(3Y-oZ&}w^ zx2d|VWTUsH#$UapxumhJbp6&UZ*j@WWo4`KHoS9uY0m#c}4Ta^$TF{>imtZYib%d343iTuGoUU^BOi(bXx7=E%86Ov8muOQm5-V{p)tZ^FRERaO#&A13uRfjAmN!L|9YYZdn)vF005*tAjh)~;@^FBP4^)Zw zyyvyNYcz^e;yq_t(DR5mqJ_;cu0jg>7(ZsX+zBZlu7cjU^A^Z9TrvzhHzgN^NrEf? z7y4rC8<0!EL=WPfzEn>19tl)_ABIBX(XYG#{P$=dkDm57@!*f%0R98OiJvpv4veXJ zrQLx4Hzfa6(gd$bIcUsy;~qewM*X{`3Do>*0zJX6NVsadK*H4~wN&<>qsEmcQ+ob; z$v>Vh#fzF(uSxosC0xy?y^^1PX+Q;E5BhX9D0*6rIu$&{m%}&9`acOU`xdT5pNIaa zoD>)K7On)pD&cksSMrTRf68|VHD2N;Sm1dQJ_&HumO!7E@X3H1w~V_CaAT@?Ou}hg zoF+!gad}7M8sViIrMx-kw+}h6u37=}|N9l|DF`g228~Uf`oT&1=0dAx}egpVb z1KcQYuGG7g&zTZ#mA79S$SUva5^nA1XA*9e_m_r#jQoFSfE(q_l=FwiHO9!i8^Di9 zIPqg#&7PO=M=bDu1HI9{uSvL-e|m}z72;=-xaIE!60XKozStoeNPx`-xIE(Favd?iqkr53Oo1Fs$?N9SjD5}k zH_Fv0;YtsRzFESp^1dkHR(VTh$Wi0w$#GX0;Kp%-60XKo^t&Y7I_~!*+&V7hEmSCX zp(&+wM{$jcf_r$4u||1Sa1}S6x4>0==(oUC92l~|)zYiZKUM!#(oQqM9~A}fmhj~k zxLWW17PwmPnMk|aYwMa?y@4)yI*K~*`;&E@9X^5zdNeda8@#2yCz|SgCb-a^uTlQo z-RLZC!FcrZ6}7GO^LQ&(tXo;y^s)i!X(gg{_V z@F}W77NdmA+5(E;cE@754Yy!3Z(wVCZM)xBKk|1)18oi74lkR>--Sa8E3JSer5q(l z=GC`%d%c@#ii{X4|X^D-Z!&(I;?6H6pCL( zZ>|589REh`KeQke->ScK2ye9s;Z_6`h52fw-6q?QkYBcc*3!VrzYq9$_H#)8xgv)r i`;jZC_9}-_1#?frfM&9@k$H5HNWV1e*oD(85j0Bj$9j}8D z53h5!?79Dp>v_EX1@~w=QPI-%+O8jl zem9Pde6&FieOO@&U9+*n|JTNPkG8YJpgq!^!3w^g?dkidn7#9BogJQ$+Ffv%OP`v4>ARGf;m1z`%~n&kU?O{mj6(bKj!>y`1pYn;oHh=fqnCj zhs#f&2sgE!4)-2-J3Rc*rSOFvmq4>5ROUQ5aAE$F1N+vXpZ(tr&uPK` zewoJm(@)C;JD< ziQQGnuV)9zjvgB;aKJ{I-d$A(pPHn3M$XzM^%8E2&I+RT2Ts_368uZ}z!r@aya^xB zG{Q~l9n>9qntT@#J@g9?%b8 z_{X9Vx1FV3{B_aD@GHyvezT7CrDA@ZoyA6mex#4Q3BMl9W+P{dCmkNF*AAck8~xhQ ze%43xh-v+gt^_{v9P5kx6YIP5nm+PgG3z^b9~=4S#jNjq8XLKg#70s;TZp+fv^Sw| z=pR|%&;#1xi~mC(x%eJCJoMKIp`q`v!x!tZrcPje!#|!3SYl}C0e1Lb9suosOA4Lc zu;)>lYvB3twOy~9=8{<7g*+F*Kglu#UTMttNhZj62{Jk%6Uml5iH-b}<|^bf%Egk| zH4o&u@Hfl*h9O^>lxr9~4}<4n@Juq2d>6szIP~v@OwPa6LhnP4;nxyE!;s^B$YGS9 zWEg%eF*FQ$po_i>6WHNP(8=&CNg<|n$Cb;$JR@^3ezL~;Xx;dv;zuEZ{T4B6%-9E> zNLP)CxT5_h6{GcK?2p|neXb`&;}iW1&C|!nNQpnjwY2G1KdI2TkGV2EBPsJdq0}XX zA?KHhLS!4V3E7GIdk=a&2mQGaKLg30z8&cQ+qka9s~q>;xE{du1YZ9O_os3FR~jGo zdhfY}kPUWK{%0G{M&5(|T{HEN3--APza=~U1~wp@Ux3ZeE@pj$upik$`Sb_tSYNo9 z^)bEwQL?GB|CR5+R`BKb;m4O@vwwyCV*bDmUpWN38tm9OUFd`S^plFh(6ywZt7N}> z`Y`L`c{mUJn7`=lR_m4TSO2}nTjm4sJ1bJ4J) z$_`(k)z{q4DuKx*td5zdr{bz>%r@Sp={Cmf+CqiU+5Z$WNwO_< z;^yqX7gv1;OZ2d%ECC!%A<6~qjqM$Q4sRXq)U~Tc)cLixy|J|+$QHD=b@&#vKE9x1 zM^o$8wphKsQ`R3}5WT5yYw>$K*n$pUuw#J`t+u|iquJZuUh8dc-sbIS^aX0`z3m;f z65HWxZEp+sI|5CM7j0|twRoGGwzcLz+?HSaZ;QbIwfO~$YV#i{sLjvI%Uist7XKD4 z$$KRK;ia{;d5}}cbr*=y-slbZ8txuJ0={-%;BiO4)0wdNQ1Agb6f4! zfVagbtYZau7nmTJX7#i*1ib!6KI!~{wvM)rERN%R&BIo$$=lAOcN9-8k-ywkIoXXO(;Oge*ezSjG7e9)g4v&Q`1n!qjo7T)_>SF- zdWyqwQfMhJ0iNNC_0ydr2J=P07kAqkN_jMr8I`8$H}#p2EeubWO6nwq^GO>M32j<&Yu_IYm1Y@fH? z=gwceV97jp4P6!#EMrBTP0bDN_OA90UyHk?v%SOJ;M>~N>T|EJs93eGwq|+Ny0VHj zwaYiIU%sYfc?p|e;a=>X?+w&9y4zcD9`K0QjiHHGI$y!(9~NGBuqOexFLW^Sggm-W z7x%bj_Kev5%^c_08dI5ls~vVh+ehr8@B3(U1`ou^(3Ny%mvuW?`buH6Zllo+b)C6Q zbvmmu;;B!n+t96TYJ};laLiO|)9iCnxlPHgRP3`cK8)f@hOIJiTMpi)r7}Odcfk=W z@v^%>1B2l|f7z!CFSBK>XZ92aBk%6E;r?L>-+`m87QD6rfkNZZ^%Hr$Ca)jkT0#gE z@{7^4jj^6xT7>-?NHWl18q4Yu`r?x$$H|WGOpNdYxD-&-IoSd_q*Dd-sR5QsgZnG5 z*w`<#SgJ^K>veB4dhsqDC${kTiWz@7pg{q+PfktB%5U*4s&EyRo7 zZ4><5hYHF}yv{k<;eg$a7A(?#FpUi^WVu1xeaGr&u!w#ziM@G<%?c*iemmu@9lSo6 zZ7&GwHo`MTc*qx=7|BL0VKTIh@n!{+^eFwwG?pHim@tw0A2^i6&JjK1^hrbyd~VP_ zkzur{bh7$#1HB^$>*;O+lb$! z;3Qa)+c!bs1$~$i-W708B7RfPMv~Y-8Y}n<^p851_buL^GYvI=9r{E6Pwit7vWMLP zdk$8x`*#s7707ofjeV_ae1w~W@VQRtM;~8PU0u!PeJdOCPS(knA`a*W{pSWJPbPmG z&-E|r?93~=(obpDH>G)DPxKFaIJuLPJ-|;vfK8ml#~1aUT?rBPApqn9@V|_JEkYg{ zSuF6&yEJx%7|?`&tPJRBRAt&HoF7*}_`2;k&5rgOj zi%9vFcVU@;YL!2m_n~wV22)iWf zb(gi7!Fh9Nym`bQ?6arKF^|u;`=_U}zjU&h!MU>tKbPYdvM0OPTvBb~{aAE{3PEoN zJ=Qj&Kj!3kJL&>o47?k7w+Zh?UEr4kKMnXifgcj>r=c$JUk5%Dc#pu(PUHP&qAu{s z+MX^K@K{7?{6>Lyp)T+Vx{zm!#Ai$Xw@ADOJn`=r__~GguLmC_`MCd+e+6we?n}zw zf^+7OI5GUJS<(6%e7(f2FDao?X*p{xosx}2{EaUPrx(J%u--@gi~MJL0*yDF`_Cb==k%z54LY@g zz_f&@zhXU@8Jsqa=o4f9HZ7Ct+~4-1y$kJ?zpWwtPIH<3%>ley*z?$Qj(4Cg+LJ%p zftUUU|Au~NM)j-gO$?B|-z9z1J&M#fjb~T>Iamz+bO^kh&!hD((Xu`CKaeF1pC4#1^^N{Bg3uNOi1hb}^_S)cH=xv4mRVlvFS;HK7P8W;*(UkM(pS$egTImd4vW9pdHl4K{HJI=vy1$J)`vP;FGYJ5-)06KoZb=hHwV`T`P(^7 zAI+ceR=yr%{;h=lL=p4P8V||7XNW-NBL%_nL2J4J3oeAjidFOToYf8fvf=$QPu zJBilU(qtuou&3GoNdGote6}CoiF`xJS4a7d!};rLA59eb4%sgy!p>np?!RQ88~Z2x zo9BxTnO~ATyr}#mU~};G`M<0DDcSZemNPXT)-xxz3L>B8>shjmq7C;C$V+&>-eve1 z$y?9^JRd;#Px@|KGS814JU_PQ1|7+>{~?m?B!5DC$j`^)^M%)4&T6bu)|+cSdWP&eXf`Wo>Gs2lN#NVkmmB-CSg%X-Gw zH@i#5lQI3ff?>Vp{^8$3`xE%bKAN0Vlz(7;o<)`H3;QGA&m?yg@sR94C1!v4zj=Nx zbrAqxT5WlRxct@bwYn zr4c>+Dbwh0xxp;Av3L;Uw(yzBEr>lXq#5eWY_`}-l9PsaIx^~dDzz?MRg&js4QITB*}CI3iHu;@3FmMVXphzH;o+EDYe0`ofobtpVK{}a<=_%z^? zP&eXTIQ3BN37?L-5uY)^jGusd3=es^zjA#*UaGsmfzXc)^Er2ne4I}%5nnD;nB&V541^H7dweXkt?INCh88s?~^rBE^njj9Cnk|@vi~7%MKoU!_@4#2GcZ0r{Hq82XvZJ;n0LbK35@eYfHyH;b~|uX zfMUaXaSG$_7WVMheK3ywisJ1X{CrR8=a%Kcex0u$QcKW55zD4+3@z?6Sb{+|3S6BZKsx}AI`?=#J3{G5BGU!KoI z{V$f!@bigtY4#iDFNkl~&0qc;{)hPXdFL;q^A$((t6rp^A|Jtgb2-Q#`S}X`)10p? z1m3tl(|TpiR}AYf;zbtLYb+-s-a~(;^&IQ7Y5hh$3+r{19__In#r9i+nt8p(d?kEz zy&f}P;q%*)uh3?kittCipHTLPJ#qe+IT_f93-kT-XPB>$Kg7*fW{i=qxM$orU&-S6 ziaUP30(r9B6yKxyO3;vRjLugOPu*G3d}Rpw`}Fxp7Ok&5UZOusK7x2@%10a+Z-yKE zeu{h~ODy;lpGW0KcWypH{`3E3J~Gb~zeexBDZ4GKX86u+WG33@#?54 zUK!5U;Xk(%ucG^JtZ(N1_CkEVFxGy1Hy1~(Pdq-;ks5toltc04K0ew_ur=Yg?vg6@J}D2pFa>j8qdIgI^r4aha#CGzq|E#X4-F<;}^y^ z?KhxrQ#|8%^ZvuM-vHhe--dYp{VC(w6_dYM;~73rirF9hrsF)R|De5$vHkgje1Fc*PkFxm)4Hx2G86 zTS0Jw&hP=2*dNmT$eb_{783jKZ0x_YCeY_|hW+<6+AkX7os%{svHiExImZ4Q2P#x< zC*D=u#{N4Ut0fo|`&G1ep&s3jB43+eJZ}O%!+0J={+~(trr3Y-`H&gifAak|`4dVs z-c6p&+^AE&dVV4F>Cz4Qg#FWXi#|>Jk+JjFKjM79lwi(Zk>8f?yS4a+{K)YAfHbz6 z4x4x}IS$o+JF2a8ZVhytmBGtj~YRi6`@7JR8IT^{v&lgEZ zvA){LL3sP2bn2fzZjt?IvGf1Do0Hdkevrn#41UCZ79HUHr?l(iSbl=f4@~*V=TL`K z_sTsD+|$544cybfJq>*38lb~-gxY?5%Yk+O^4cFh`f=pzuk=>_Zu8&%{V&SL{Z+%q z7ydeOsDRDxeY(e0?e3kMbF8&X5zO@3*S*HMi(8r3N<4n|86#zEPBNFc@(t|r~Tq4?22B2y4 zcz@(yK7lW7kSIPZqp@8Z8u@#<u+VQ*m38P4YxoM<0_^8~j`1q*YZXtKPKB97g5t3C&h9OZ5 zlG#5K$?T&&iY#H`OwvR-gLF6IZ1fy7b8geEoX6F^)%Dd6V(z)Q2ldeAtVi?MfK|oFkAM}{=_z`SBk}iqrO-xs z7mck}6zhr+QNBHtLz6i-%WMpngV6Z5Vl!HqpSS?ZkMT z>8VxDt95g^{-92` zeCY!oY?$cZwbYktX1vEzue8)@JnEk>5$J!TIMF5$AOD7R@`zh)^J(b|Q%mf(+DE)L zH^ciGzM_sb+RgMdRv%V;ygolaZmismRX5JW9vOqUmkGBbKfj@0#cQu72~Djr+neU4 zo)n#zZoz=HeLQVjX};)nqt5g=G04h)JbIM}NFDo;mYUZtbNwmcm085vWwWeZ{M*8Y zwaZNZuBH7_WX;C@JeGQ;rH*X%`fHbQwHLlOR(!irG$6;3Uh)HG*QcY(11hw72K8)1 zo$yrW%GAcCBEzE{fK8hLHr8r-jtjBbvxju7Auwa5$6IVyKslW3)vbC#oqC;tI(0?5 zrcrZLJ+{xV=`>1V>Uw*j>+zJE8;x?2v2+tJS0PSsB^PjmU4xEwgfc*4g)~As9+kz$ z$@NU$YsvGdAEWif2!@fFb!0b44ETGo9{PZfnbzpfV~zYQ#{B%EW<2gcKW?n{9II}e zVfSLVSVP9r*{8)Db$%~~n^$oWD^z?K9d~Gt8UG~BvbyWm)t};pVJD|)SCPf2ot%ug zYFv|r{wQ&YMvlcMTf&XG-(XHIZ>+|+GwD$hTuA63R z;^@;tD?jn*E&Cx;J~XO*bkE7>U_x{Sq2&j~N`I%-P5Ozm@>%7HN3S+_wBmyPj^1B# z{>^83U$nEmFS>qB^U1hl6R|!P_4bb1v3cM@v6#J=8Ka(>weRTqniA}_89)2huEP81 zFdVyWRmL2Fe^%lVpJCCFtmPbC__zt*FTNiqw*RsIN1u#y2@L(~8t=bb@kZY}Gxjg> z3eEq_ZLO}Axp2Nf^GTF5b{yEmd4`d8!)F*RQOzya?v(<7k8B{&S)6#t2N_kCL@96P z`9E==?X%gGYo*N=q$kPFaW{zW~qwsf5&S z{83$AF6>bbp78viS648$r4)Z#j@R{`|3eBQThKM$vwaBuD|$n*;Up;f)jHIbb{_?t zAIx*A>%Yz6eek0p6VWmbAKim^P8WA;6TNs&SB42jm8USJY)m}q#fTKo>5BEE{6#&d zOVOe^FXA~}nF<`eMxWCqA&NdqaGN(Y)A=!-rM*Yi>!Tki3qWjT5OvxF#B2W(+3qVA z)mq!*k0C8*T%zBV^hcT~4%vX#I~q%|qclYJu=YvqD&>T4q>7m9#&Q z?T;JU(ydkR74kgd-P*p4kobD0xWwMSCfnyqkkYZWJ?-b7jMM&@ zrM>Fe+J1&mh~J|KSI3H-eVd6m~5~7m)es{TiY+eBpiFb^I{1N&i=5d&z$EV%7gTwEvHBG;M;A-qqv-(mf@oFRn1%Dc*Luk7uI&F?vV~+Y{x-zJ}EkN zRr5<-Rl9A4>uTBymlujsyt-&nb7%eHqUKGD*Hv#XY+mo((9pSQV^w~A%ZBP51#8>9 zg~h9jYl<4{TUK;z-dNRK?_amRxoqc(`bCX^t*a({N%OL`n>Lm=)@`WC+q|)Sm3KqI zj^^@;>UBFyS2VA9th&)(wXS*PqUy)Wc9v~iRMot>Wkr77O3}8cd0BJa#-cpmX5R0r zjf+-v*0n6l0}uYXMGseRFIuy`VPjSMrj2W!SkY~;)dG#$djdh!= z+KM-MtE>H0%bJTD+e$WUtMnEXuU=8QCVzA3^3F{y&7GS|@mjpPbMwk_e?v=67xY)Q zHoxYHhLz>bn>H+hy{qy!wXUmf+$`+1y{LRE`p#?ESl(&1i?`VS#HOYtk1Bst*V>lm zraT3mEebZKNz3+NIv9g~3 zwhH~K3V!Y6b}IQdcs64IMDg)x0Yq+0>d*dGSDOzx;tWUk) z^PGOf{+|;6v*R17-*UbvdM5e?P9WZPk3mmk9Wl_qq3%~YkB^`m0E*oUe|f0bUd< zEzvdK1fA;MdVL9P?d$_}Bw<3!2gjLULJpZcX#8(dlut8j^?F8qMhXYi6ph~>w zJ+J3oqfwj^?>WE>N&~hs}io}(;Kp%erZ4j-vD^J8WcUPMV$(s;>+P%Wc?omn0*IVqR+!P zR8EQudk0s7UzKpXge&07p<+!|~agFd2DJS)>cq7tP5^n8hql8bl^rQ4e{TNS)x()r)b52xw-T*h!AHNR# zssV14H&^Q2%I9nex60cu4P=$~O$oR5^K%Ke%KIxrKSusPHo%SYX3F_P;~HaR-gV$d zB%Js$u4d0m_(K+Wzk%Lp-!~-O%0E3thYImCMcnfDB@(X2Rle9L;c8q3KVqOaj{CfX zTgUyCgj>f=mGhRyHO?28gpZmp5>EXnUPyo~2Dm)p;c^`@z@vZM1I!XRn3C7csTuo{ z0dAD5QNooT6n(RVTjhO8!maX_$dIGP&6DG5k$W6$SV38e@&}s^BVaJa2)k_|R{Gt2i)ZfvcrgoqwwS)p|c0{83TxZfU=j z7PwmP{T8@d@0m!u+iUBZTD^fThCM=qca^uZwZ8E&{*#l-@jzz$x+DLMC;GWYcC$MV z>ZrDogF@c3VJj&K^wd!zQ>#DeI~fjp082< z+}+qLZpC=?^A)wN^z(SDR@JO7Y4SGv>N{4qHFP#(_?z7QaT_$Zc^hgw+G-m(V?rRX zC-@XqA&XJM6>R}UaI0gn+=iR5nK!Vly|&%&s~`QlqJg#sZ-JMtgCX>Hsgd*PcmZrmSCy`2rbsm-U~&5QZiu711q4*g|UvA}-H_q_-rGSxEm zbX6v(Q)T|4kaC$;RYv=Kg2}WX3irsnGct}Tgo;XKSU7wTfsc&PcRKdH!2#t-epoAu#tU9c06%>kJ zMQ^SD7dig*+J9_8D85yH=@8y(6T+u-MZdnZuGLHroZSSRV!-uJ)O z-urxokBeHTkIx(p_pGzO)>?b*wb$Bf?{l_a^O^~#KFD}v8OUGb;HR(F_rt;pb+zZO6ko@((nKNVw6jad~o;k2l@jo z|L~U;iNmLm=Jy3&X+itb^By{WaMuIJXW;j(2Ol{8s6R1+I(qT9XZ`7?hYr5<)X>Iv zo*sG-ldnN z*FW{-(CI&Va%lO>KN~vu*53`KcfCDy`pLJDuOXd&=%M2qp4xSM=cP{$t$6vVq4l@D zJall^&xYQ3@U5ZK8!6Ac^z^qLJpRCUcOHN7t*3@Iyl`M>`O`leI`+oj4Nc$m&d>u- zzl}Usrr&t!{^Jj9dg%Dhx1Sn%;E$e0+RH<=1uqXpo;WpB`!mXOMS42YYw@@Kvrhv5 z?+s1g^zzV#Ctn^)zxC$O0}oN21?e{)+y%T*&#XTks=fBRLy@On8j3c(G_;;@-uOE5 z%unw;xa)ZIq5F@&xcTX!+Fef%y>UI-dgnh4KF3OIp9$vvBaX6jrf1rowf<>)Zt0o2=T`sJJ&)meW}-NJ0^>h3 zRvdmWR~*g+#9`_Sl-slV+GiKGE`E05>YngZt3QkT6?7kZYW3%Fe*@hIpIZGD+<%eo z15dGRd^_>MfnNmoBnDr-q#_{R^&NdKwC%3o$Q@q|jsC3Ak*8kWMth#OGY-_yJY0f2XO!0vh=#|E*V*WaLGvL-X$a3zP@DSu5TF*Z&S}aQk*H3_K`i(v7 z?h1^oKOR0Dd2ZJY%SB&ueKIhz9Ce9feV45NRrql0ErAa(W(rzseIK+$#O-ZDtU8VP zDuTIc!-~a+^|M-(3DPtCZa}#e_}$p_$2a8u#}mm`-?YiOk<<;#D!=#5h)*1@hzOpK z;t`DfBY{^I`JfN*yB=fZje!4^>Ch4MZ1ug@J{9p54Od3QQ>*X2=Bdhv55G(C>&NdB z{05$K=HGB_OL}#9W%`cK-;i!y{ke3cyJ+~Ulbh33(*x-ZSAQkl@CA{ce~m945Bbwq zFBBtn(}U?`?U&QF%R}i6uO~)oXN!>yC-M6~!)a%|n-6{=y>V({Xe{fTY z7+I5lO?uv1fBKfMFG*jqIgpO{V#BqUiuC0TzVw}!B}QgV7bEWk#vFV*kazH{K<>de z1JQ#U{vk0^{m0?-RJ487mxJlrKMJ|pX$KC3-)ZPL!tvdJ?_k@n5+ik$Vx;vaUuOIX z??`uSc+QbfItg5>LVu(Yx#4oD9et9@OZ5#4;j~%XE7))bBb|X8vp}NsMfG88SQF2inD++D7q83;KHFHxeV!?&9GYnDd;^bAx?1M1AAv`7_Sv zB=gbk(a+<)iXT(HsIO#L*IlEZb=%K2ehcmS#*b=mk!nBMJ>Dw6@iEF?1iQ2U){O{HpJre2X{B7-Q_9L@={44_e zqvH}IGkwtq9~qMv@}XR$C~oNK2!5_tB(SH9e9TS>*G-d!$3GiWVhC#*axH zUjAqz{YY56n*wetP_D%nfA<8^DPQGQv8o~@-fg=Bco#Zo$aB+hVmfHRGfl&#py8*W z;Zc+WK8F|i#&7Jt6Zb?yX*%YB zj+XqyP&ew(bhK^9y`rN#g?mLuTQ}}O2lYo|5AM-Fiz?A~^ql%BByR7fd*D^ee9nLK z2=a0N0M8RUY+jk$@CcJ@1<^8x< zc>8wZp76%=Q*^KJ?xuT%cO~5e@9DssczObG2=`&gYfZv|L$w=+6Xn1G&omB|51<}} zL*s+ES2$Ge!o9-5_#@ma93G|T3Wr9zci_;Cd^y0Oh;Ufxz+t8v2j2wXfM>c-t7&}Y zp--tDC7W#z;hylqSnJ-6d)=qKxF@_Y?i%;tp73f!pVD*Pr*v=iY4MwsPmej@UVF?{ z{tr3k&NklKkVdlm=-1?UJMqUDZ>J$_*?PZ{5MYkuE-JZ_CZI`*fQuj2zz_Z50SBebL-Sz_T0WEZ%<;_X^MMKf%4`{r|+h!gJZTaIf$@LC-bs zac=-m3m@dGNjdOY>c*#S4Di7-&9|$;w|~w7K2*P!g-38t_@K|b`*5%Ec@*~=pT}^o z@uBA$AG)_>q4dp0RIOH{K@<;EiV* zZ}9QKbfA2gczFb6HSUi-j(dgsk?-JM;a>R!?iKFF0o*IxAEoCC_eQ#R;O;}d2H^M8 zXB*zEOiSDgf%{KAxVIs#rq6-M?RVh+x`sW5^_vq zuKW99)%SKx9L@#2;gZJmI{fPSa3l5AqY2JK^Wu>|m*dp;7jm37{t&bua?t(rd*e@(9;$5g%=^Xfb|Ve^efP*uC4Iglz`x%?-x@c4C;kKJc&6#A1$`}$ z$wrjZ^i@8Kd!i4zwedOJEBYRN9`{Nn@BbUzEBY$wUeR}i@_@dBfr-O%&MbcO1kyE) zkG`PvE5^`~zZI*f-@H1i`zUZ6aL};HO@r^>k&b6x8Y*Apwx#WI4EIC>=E%mEaIb0j z8SYgd`TidFdR)DXdp)jjZ-54^qq>o=W-xKs2#L>CK(76-06t^G&k`&et@NJbfD3JS z@&5uGw_*R!0oU8`vjit>xaWTZ{*wgOnyz3!?yW%C!3CYL5xWv2SKNcS^YO&UWxFu9 ze+_wdlnvLcY&={+HYap_&69}_YFZ`S<^$hHowU z-Ebgu@Gq9kP5&}bbnwqR=B58V5I*>mMRU@RU)hlUcJ5{ACoi9${$8MLm~8$4e1TuX z*Tp@rJ^S!mi8W}fp7$=s@6AH`2CZ1z_K8&N0npHQqj-hN_P-(;#s@q5GZ#`t6({(ikfj6Al( zOh3M_EB(Yb?@W)}Dn|Z&tQa}99cy-hFAkshE3C(T4}aejhktc9{=S62>i}OV4rBfC z@Tt2g-^$_lr{t#(>{vN`>Slt!G<@ny^!vHtQ#aD@4a28q;CBaprx5&m!@vBtKYbu@ z@a9SoE*&8Z`z;Y46;ZA}p4a*UY09y)9gz9Ia$ zJ#S2D-27g16MVaC;p1HcUvDXV$|dkA7pG6>Ec-S4lmYe^S^l)#kzC#k`D=uHBSv#j z$7ptjMX{DuZ(jI?z4xFD)^11U2D8ievn=zi2xV8_n>5COhk^B!bKyBTi#&)Gjc9r< z%_2X>>;>C()>(Ry`SUI^e+WGI4*DXD-xu)P4?a`BoQo%#*1rj4$7fyt-)7l=FQVRT z+8@Xg&)YAOAKCPwvwtqOKgVT(53LtpNH(1~Xddf6Y|fH5tFq*^uK)8{)_YTy^|oYL zpKf2%cRR-T#qj6T(5L^HB~LZ4-bDTjjbR#RXR@sKy)5gy;v>p#UzYf2I{z%o{6EYx z-;c7)r`y*u`4VLLVr0@=esz}Z-s;azKkE9|XIbwrv#d9X`m&L$zwup|AOAPbucyH0 zZ20$Pmi?jo`QNh4|6Z2)hO@+TJMvvf?my~$_RTDD{#KUxG|wK(GXE1<=KEfj`Q{+s zg?M(p`Th5^Z0Czv=F|N63i4+on_qyA&&Ec}R#&}~g|2Y&y>who{ zx%pl1vmC60{VUeNPGCLr3t93@_lxH9@()=~w|xX{U-2R9sQr-T%0Fbee3ZM8u5gZP zFRvcYWX5Nsldj5ACg)_CuO`cUko;)>hIo{4m<1m`O!Y z%pSZtvr6`nj4@GsX=o(7e(T8+Z!^n$w`G}cO_uq-m}S0|S?0So%Y6U0%k!hqaX80w z@E?Ae*+=s9l`Q4y!;YWVpl9^nM*7t{2frA)LZt9lh(DkDBbfd)yz=ee`;iYh`VP|* zA^gJUgUpH=1=F)-_%l#qBYqS3gFY8UqEHsVz4O|*HhJ6HO&tPqC1C%eLhSbTBYNZa z`$b?(z~_gx7R-feRaYQi1aZ4vo~vrq34ZYt#!+W0d`33c`)3NzjUt>7^F;_4O`x2! zI@Wb`COg|!<4*r>+bm)JZRuEd+Zt2Mx@}A6+F7^VF{^Xq`rB^VV#7CYRq!3Ntecy+ zY;JGs6tgD<=7_GYd`RkpgVV{OZtj!j5df7`9sZrak;Isck@*U!E7 zvn`#Q+gfhflHA>j{S0&25`D-I{D`U&n*4J-MZG3ypY&Zr*ZRM`zn@o%Npr@x8gNee>Gn&7X>H zMz=97-oE6AN_BDvIv;!r`cRYOmRWxKsTsn$Z<(NTb+DRLoVr+->yUPYHbu_+``>2B zf14ryr4&YNNK);F}=y0tC2CfT-r zQ%fi2x|W-_Y-(A(7S<~aW6XbBZeF{l1*UdOa&6~UxCCojR&Q(J=fYX}?CdJv3Pez4 zTPNy4Lt8iB*5PgJzfGs#Igi7oYRRzmKFHp5`aj?I(i6*#o7ZlhwdIyuI@Wg9i6Yg{* z2Cxd?y%3kdbFBJ#t}pZRF`lPF{24a$21LQA=YName!RJ*Z$Kv1s~pwXAWY8(^mDZu zk_S`*yrSW*>MDMYRUXfE6Q0Kk9rzrV8A^b9C2kWcKdq6_Rfp&Cz5?sHd@96B&Tl`D ztAtKfk48z?gMG}vvrreeelxhL6-4)Rfu1hb|1K zRR71_-m9uTuME&*tayp9;cozoWuN{0@}yj?T54 zEESY+_1T1n4wPa{v?Qy#vL$^3Mm z&GKLHmLOyi^<5s6?TsglgeVuKLH~P|BB+kUsUwE;E34ef(>!M3(gTK7k#V(&iMSAzaf$G3kM8c;erDq<#VP( zmR&H`yM;5IJCW+c#%;7Ee2vgK5DFhQa|9-)g#R-IBG3}>1!tFv6jX07{su?|KY=xC z{H?@aaQaI;c4_>68m?f?OCo{6-%E-h7k@aEmBL?Np%91gw{npXt@!K5Um}4t{PkUn z-|J8Ye`oO5{u$)OU+)cg{w4h4?+pF~CbwSvrEW$YP=;8568-oaTn~7I5cTay??l-< z@%vTq8h?e`F`wYC7k`KF*N?ve{GGvHVG3#ZtH<9V{I&1I-*=FPzxV+>$De;iN(2|Q z7S%&oGD3(0Ye*`BBB9_z3SOz;XB6D3;C=?o+U#>N}+1LIw9LxKzOd z3XUsyP{B0{KBM4z1=9c`eC8{-P{9cW#}&Lx!Gj83so*mTzDdC}`A~hW3NBReIt9lS z+^*ny1)B;^C^)6yl?uLJ!L17JRdBn4pHXm1!Tk#ERq(Ka3k`{%xJC9)T)_{k^g#vp zDfo5TdQ_~O;A|f=&qQbWsOVF$EN_T*7-e?-hDa$`*Iziv zS)U>LJa9cWDA@EGqTd6zSCIXx`H>NaJn+i6EdMa`GcMvVrZqmq-*yGBME^v6aWPP3 zw@3UQR50Cc`zI&{6s+58oh!?0_>2cm&2y(0 zHn`zF4_psJoAA-~^?Trzm%GywS4#L9)!+RI36s8}{vJ>;>8GgA5QB{}eORT(7fYD* z6OZR>CH$~TZ*7vWuKx_?bHY#8FRqucu7BlcC9LZ|q+nft-wiTd*FT_OSswc4MtAvo z1?%)y1?&1+e_!S&eaiU5t|0sd6#o4RCjCkD4}3+YlYS-soKf&0Rh|yhQhCy^lwQBa zU4LqWyZ&AU>-x`Zbl1OflZ0jc&~LD9ob|UWSe6fpeg$j%#O*S_PLBg7eIo12h*l3g zpz;&{i63VaO#CBy;z=7nN^e&%)lcb(4wK{e21^RQScW$?|7Z`E~{C_WKpA+aFZ0ZeQFb%j@=96|CDERIqNZ?;$t7hZL;a zU%6YR-_PS8`d-2P3f}<*8w^9=?2-8gRC-Fmg$f@0W0|hY7k*vBx_tjW2|ujzi-#pl z{EYfAKPp(~7vGTSy1jY@>-2$d%Jc)OzQR9|FwsZ-ky5bEf9P8>UAKQm!KXOCA>xn7 zbeTU-)IaL3Kc!$@|Db}c`X871HT{DMmiUK7{5vvTmrora=h&Vv&j5yrDWtstc~47N z&d(V!;DHkd-RXxsaQu7j`MB2uiznUb?H+j01Fw9_oxk4$*FP=O<@}KmeF~QIQAQO0 zsZ3wV{LYA!g5`Xk5oZ*9lS*$rB=hU?0}9sV6MyDTKcrxt9{;{Oy;s3HUHrK_y@&_`%PVZN+#;^V_-06J^mh&s<|DjCR{7ETT*LTJPxBjKOz5x%Mc*dQ6$OFd@ zyVH9;u=tTXy&W)2T=9(J|Db~9e3TI@|4OF!sq}upB)>X;{f{NA^Y;OUiz9UY!oQa5 zS~@)in96JTjDmIk)}P4sj;i(s6fEb5j7a=crXN!2hXB*~(fQ;3l0Kck7cfjZYd#T2 z-0AIr2_Kz*P{G5h{*}+lbg6$cqF=!}z5Y3w{;JB~r(m7G@Oha&sM1pk*5%JASeI}8 z8+UsH3fAQlFG&2Q{)WA+;C@wq{BLEt)K3}FtKb2ZE{-bt72K|1U4Br(y8Ox)W&Vd% z{(c3YQgHn-cYA#b*6kJkojX0H-~+1s83mtFaO+F%`~wQs?I(UF({=lY6s+-$zbw;r z{k;m-^^3oE*Wa#Si+%-b{8tXh{JMN#{u~i3hn-mn8y31dSeQqvbHWE1qQBr|k|vZX ziwA>IgO-)B+K2ng)bowlhPfGkw^DA*4Y=s{H|qBT_4{k}i-10hx94tOM*K5cHSkxJFUnvK@l}a`Z_5|YZHr3ye*8R^ zFB(jrub+AKP>wiOA!2|B0cZLFzgB?!)C0Ik2~d5eKSTOLlz_2E>%_5SkXV8X`A!xD z^GxiJ1=PM0Z!UPXKzT5L#%(n4d#cxOTh(z!H=e`#j?Ce<%+i|Mz#NC;Jb#~ zlk~XitIikh`YKT0TLz7hOlC)}06k^NV37JVLrA?=V3Z{T0SgBH)n+IZr*t9ZVb``y zCWr|S*>fq-LU5qehVS1dGNIHOKYGg$uZ6@ii=V*nbj2us&+ZH`>m>i`n7#zs;|uM7 z#zfC3P`pn33q=_}>d)f^;{J%p1j>01GsNow$a}TSP4ap&Up#qdI3s?6dy+TEdtOrb zsmED9QU8;7mS=cFIHgyizq;fazbJq+_)@dD#H_9kB7czRKeik4vk4GgM30n*IK%ko zXo2W8!$7Z4WKKXV0Pi<*!nsHSe_4L_o8d5tiiUel!;?Q^xbqu$1(KbgbW6VeREtN1Y))7B~5`IRdYq;M7AN9ZkZrC7W zoa$HQjX@85$^)PAz{4I`V6t=6mq(Mi6YjNP*LX&MH$|2z^6ma}^V8XX%@^!H7e9Zf z_({epZ5r(wlUnrt2Th6_D#sRxdG;2IBH?}6ug;DiTWX2YZTbJNGlAClLU7C4sI zi<5U3f3>`>)c7m@-lSp0-&PO2&I7l5VABJqJn;Pi>gkeD!JkRsSDVaINfr zV?d`Xn4AeU-u_GY&x=i@{}jFlZ2U&!qvkKi{9^Hc)co=x_0u8UzQX5K4NHE72W@yX zesPa{c*pn0Ag`x%`wG7^8uq*GGw5?Kev|Ij!V0dv^@x}S`@y=9zk6G#8s>r!Ws~z@ zANXc>!al&I?YHN)l@KVzCHPyN5BZ)LBy}$BQGh zbO_d5i67;We_2wd3*T?pUTJ{+ww>!E`=vk>nZeK_fHOyTGd#@<2EVa?Kb!`nqrPgu zWWO_v{L2&!4~x^o9R_S6T)lDj2O5%r4B(WY@+S+7qGTvU>0CeXfxQKTU)fuz&*;bj zVYuM`iR#JlKXP*5yTOD;`Ijn02-*uJK8z=R(Rq?AraeL*fy^&uVj1qk2vR=h#&ACbRGrVAg^fqz3yC?v-p zweQS7B=eU#(vd&b6C!?y%v@D}mk5zWwI|^JpuHS3$fAZvLDjy^AGF^T=|>V>(msiq z0YCY&?2i$?rU(%K*nZ{wPjYUZU&{YtkO-`a|B(L*+@gKPw<*%+%HLxeE`Q~oZQ)E1 z_B%QW`YIp(O~~cnxcnK^zb9$HO9#E|UwOtOHo@Gp&Bz$EKt}apJjF0yP;W6k=+81U z%<>#&f0F!>P>B1F{!#vj0~WEQK?Z~@f2@xBkKy(kG<|=7RJtVph#sR9Ahi#9iY3Eg z>VL`43M$XAT|R+Sx@7rd$y{m<7yGyHF9;iTfj_Zi4)*(@KI(7q<6bi&6MYW_1jg(0 z9U(QG@fh^~buz;15U3CSMS<9x%+0X9J{RK`F!^f)!(Zg_{d@=MQ+2`q`Ng)3Oecy? z<{-b?HLCo@ye<(Y4BF$6x04k}4@(D{(qC;ByCRuP82y1rc2wYHyabc~#xUw@+D4s4 zm!#hhf3}^*Dml3?7l@_X@?#4SC(hqZzQBzwbQx&#tdgmgMzJfwrMT69VoIhy%1T&OBMEJr!TPF1# z*a>|NdWy7vKOg!Y^fZ`(z;^$%E{Ve^KeTu)gB{5W`ZRFM~X=2JrnRK>EAP%whhA6}^uqbMtvTGyXLG zf@lgCjX&hSq$?aGdBOa}^!dBS*yA1iEb7V+vpy|M6^Iw=Fn@(|NS-x)MO_6vzC*QUqxL8ULOtq6{3E$cYg=z$5ztk=$}UM(`}fZ$ln*OY+tZ^kv~rIwI(0@ zirf4;T?c!qz*T>P8I8)dR;lk?>k;60k6B2|x$=U%Kz_h*dpr_81u2EkJ4+-!wBA7Y zoJN1`TnT}vOX`!JyD>cupvQ@cRg;vk)E_KwFK@?umRHF9E0z7T-yCC)N6K&JhpGMu+YikoKlzX!Yy5zpdDeKX zBKe7V)K!RG$<)7)4~(Cp zuA&&}dzJ^_*R=H|10iNykoPkc=)b%|r+oC9WB7#!+Ww^ZhSW97C)(d-iZLYL81K2H zAF9oENk+#w`3wGq zEdHUr{bsJsZ@@7#-^0IrylDZK#lHgPU%th^V&-4IgMZ^Fzs0|NdLNF1e+9)({y{#+ zVi%foyySVtOVsLL9xuCWe)D+AbBveC(AT+RNZy?M=J8DWK=M1+$#1fs8Nd19H|W35 z%=7R!H`mMGJgVQ~Z(gp&U)1lFU(gq_=sU&wAZpQfisu_!ewn^@4}FDRdCK+DXW1_= zRE(0R(e!-;`Cd`JILd_=C-YWbn@V)Y;FD~u<* ze}NC#lQjQn`-9fcQ-JmOag8sG-}{n<_IOeF=WF~=UPJoP?jNMP#v96e$5&J2B{59; zD6@m(8yIgMedgvD^h>Uc2tnSCwU9ou$0O*0KC}77`Yh}jkLQ^WQT~#yAdg3$?^vIW z86yo^sZWY9AG*h5PLRgq7@FTeZ_`%8?jP1?!LVDO<$Cm)It_Hr@g?npj5WTD3yv?w z&l+EbXM9C$eI@%3{-n|$XCGg%!_|EIIL)_j;d5hjL0_M4hQ1Da=!jLuQm3KqVuhw{uc;wwMDo8%b|AIV`e@*&G+DFo!r~YOb^6|u0tRG=K zXUKl8&KF9pQy;luUiN`FeF z_rQOW^B3yF`U&RCLO~T$`s)Sa2U`W(%h+Us_NGXGn*1M&o|P_oy!>*nrxy5Wy>UBJ zhxrTgf%!4co3T}WMX(nhLAt%Z8V+0jGUTNi>4coUzA9nc|IGZ~wf-UducvFQV1GvS2l$Vb3=~i>)@Ss5$np1mrZG81?W_GB$CBe{`3x7V`hRSu z{xaAfNnw5aWI+0dG+#RXRq_|D{wuZla~k}CeO4(ewAW8=0zOv$8h3ugCvAT6dIGOs zYkQCEKc~OrmEV14v5Ow8&jji4sOBH|9q0l+n%^Yvg${m_y_i9|wikQ4Jo4=w&se`A z=JJL9V1J$}bn-jJ^E0m(^F{~(fYJp4`F|v;$@MTQ=jZN$JzQ0>(v;~(w-rCfj)2A`iudU zj*ING3e;B|x9lUd*K5|+Q2UH8#y9MlTC#837Z2rB`p=h7G87x)>6#*^z2sP8^=hR0r?8K?1sOW9*HDrW#-ej-2S zvl&kM-11#P@?FjNv3&bzJjK{PxAjdi+qXrSKh}^uY>L?OPzQSAGc9^q9%8jldVqh- zBMfwB{R+e$n{o`6f2q!iVP@$M|A-={KD1lYU`;gXM+x zpL+F6v90e|UW$q=c?L}S#VIddc>@27o%+QqPna*JIrR(Tn<01@)_m~M<)zrxuV;~$ zUXvq%mOTP}Q*7xsul#_YuJH@_KC{vz50x{(K`9Sd59=+RR_q+F=hsgmsZSmD@O_wn zocfLS54=bAgT4Oe(2rZ7pK?9+PsD5g$n{3De|WyT$(9GKZ`1tkq`%h;+4c|kLGdbD zk7D~7`~*KjmOcYNU=N0r{lW27wzsr>#Pi`;#V;+dV`CKmuw&;8hW#&Y|Ho-Bl6~UPN67EBNB5g!$2#o^if^V+eNe?mM$2zZ?uTLflk{nk zZSOqK^W(YwCGZ2|#c>ani#`0U_v^OYTN&1Q};P35ETYlwyhW>Eb*Za)LE`P<_|Ikmx zPW#7YzidMP8X?=hfj)=*?2;F(e-t|9t=BBH^{dqXg|7ZLi|qabJ%}$9TJnJLxHnl@ zN%~Xw=UL=qvTYBse0c3)9C|1J8}<(OSAV3NnMeK+`B(1s8u(id{ZdEvQqWpYV*T=x zVVCFrvEvnb{Z{J-&i_xY{Lnx4`oH8y$juM$dU8nZC(!;vWhKuq)_R~}^B??j;eS?q ze(uGG{ZWTM_zC0V<|p{`UyRT5zhnIQ@$VQuKlmNP=gHqOd>+3@d?FtGXL!bYq>}W5 z)_<5!v0w+7=4UzHBbL5B>v;bY7e0B|Z$=ALyuVV;AH1JJukX_OUp>{|Pw`dQM_rY? z-xTv7udh%%pcLtP{j7W2EPH?QV|3n5?kDB*c3gj4q>!8I2OKl&YdHQ-^#krPb6xWp z#d9Z9ex2Tx=URV@Vf}^QVWiWsAC$I3p$N~nh}W}!kr~8(Ev`T6s=v&fn<0CyM(xK5 znYCP$H&iozA-Mt4l=y|rDo^@sOb7};*h58KbrPoiODJD$#u3jKBChN!zgZ#iqy30T z_nVVF@FWjh?t&YV^_A2gdVM!$&Y4U7r}@*+6^|1>+&`#)S#n`!7JR@QqqUa?Tx7jpmfH1n#ftat$GZ!oX8f~<9}5As)y zbheLG{mT@r;{y%ml~*~|ql495S6yYr3o!p-J^4!NJYG+x{lPnD3K`+i`>Ez-#Ph(2 zCOwv~`8-~(wDwaqP;qNNe9T-hcaR96{cptYu2L?l?P-5knFq$(457Wj;lQF&-K)|kmzlnH^k!u^@q6r#n>Os_=a5h z{Wve{EZ?*>W&A%V&+8BGVZUpciVv#w{dcyh^N|AEzQLb&5I=;!han1hb^fUH7)?7{ zxi+EW1Ida7)ZcpiG?`kgEqh34`6-T06a`!pRwBMHWA?2s^oE!<`*e=7gpPkY* zjQwvie~{vHhMAXV!&JVYK;=g|!TI@a82h_i^+ls@82+7AKb41l?}EpURryu@v6uzF zTp%LJNqPCq-)_in9pLgYBw9icOzGnbWjs&AB}H=op@hND*+i_4SKX1E6eNFt5|^(a zdpVc-SC8jAlI5)b%PAfH{~gKk2E}W~TX0E)_KTHR?N2J51lY!>ysX>>kDoAJ!kWRFf}uZZ9sjV8}c8S1$J#5`QX)f8%951~ASi7|D|HR{M<4_;P3a;3q3w<|p*` z&g~^7*r1LrW@xXP>eu*DeYD@ZnB(E&X}=@#$INk5U8;Dp)JIJbIePFYAl~Br*{mw{ z`49j7gk6gCGtXD|?+|sx;%2qq(I@2j5#j*qM|p32c~?iKejMee44}tv7DInv{$l-9 z;VzGK@N<@LCFwWGPw4xoB_F{5o^9o@hfrT0^lg2txq>sUG)K^aJ zwJ#pVc*pq-Q%sUD^nq+|vV>V5P@oL5YtCOJ! z-P-ky6Rc0I`tqdxYLy=&_umqX{VYaDl77u?_@>rx@2)N^N;dy|Hr7W z6+So=+%Xn&k%pR%l}ec=M)m&C6b=@Z4}I{#z&NH0V&3d%nQlZ6O# z`*lI1qfC_P{OF(Sl6h+W#Y5ym{rt+B<)-6)?kmge6QE??lgPMxo#Dj=VJ@Qe5B zhGaU`S4{2k1^SR*?w_FtsDCNq=h%?q7Y_i&`L5;3stHxN02{pPr|eVQzTE#QPF@_8$ruI3kvR{W3mnXo>4 zSnD(B$H=faLkgSsx6%5W+dpVXUU}tJRKK>rJ^qEi%jRwm`e|b!E~|dJ(QqB`gG^g6`-cmO-uBBZ{~%`;!P-B-emZAC zo$90gFz+>=#Xn$w0O`^ncn9YRE&l-g@}B>r{X_DS{wd-!w!caKbTXIUPs8-Ge;rXD z-Q@WO{y|Qz96uTk<$C;uT&8cezhK$(u=nen_PnzHEqh$S zmOZXu%idS8W$!C^+&F2UlfG(3{l&#LjJ`C-kC(7e_Iyc+8@B9yRi9<=E4Zx8U0-Rb z8=g4PUEhQW7R>%aq}Z~rfnOwPu)%5D=Q*VlxC6=lC3`$vHqiltpB{f9!ul`5_I}-2 z`~`#k5ySEia@ZfqvEVTKBVo&5h?LWMo{f(&iRINw&zVf;F*qIi5%^W`e4*;Yd~e9| zioZ#N=OYwQ{z8z?b1)40jAV|AZ_50zFFpPO!kyY*!2GmxJKk5NVaPxDhx2%J5x?pH z=kR__hL55Dli?hRzs4uWkoa5e&7%El)q5tE)V{KpBL~K4BKC5zkt60exp7< zFDw0pI)dTflmB4(3-GTAw)_S7-vn!afx8C`(EdVLF(jq@1r8Tcx`NT)%rHE{{EPGI z+@H385aIKiHXPOG_?b z_a}4g`55v;{sw*5M*15#vfjas5w(Kidv1TDne27B9%B0ka=k?P2QGhu?8#&f4mi`5 zg1poL4(an8fO-EE;)%*1S&jYUch}qNlW1St!}V<6H^aV{_OygyFKlgf@Q3{uxn5%9 zC)ZOHe#>Eh)dL%J65IQ*FI?*tWS=|JjpU{3ya(aW^yu?<48wmYcIEGI*sI77d%VPv zeir{Fcq!TYiXNhm?SD<5*IvI4>kSxsKn54|8T^w%NBywx9rio;xjZ?m)JlJ?pmaIx zO$%m!pv1O^8Q;PLUM)~7(9tG)MuKkZ-8 z`U>J8L{} zx{B9Sn^SpkYJVeU&h*3s$6#Qp3-nMt5b>h|9q*$3Dr&t$?T?x|)g2F1@e8T~qqo{4(Mhp9v3p>R9cpYegVB451jfj z#RFsJrL)}ezo}EHefS$3e_WQV%Sb*o%}KQCuL?G z#{=uEc;MxAkLYxc2hOXL{h{N5mtXFV2hN*E_6OI8`Lo)*{BoL#xqngrG6m~+V0G7J zmy$-)%tIPIrnl$yb$qK6>nAX)!ChxZ>c=bue_djnZ!@`$DCP%z-Kze z|DYcz-b?9omHbgW5c7k>|19&s6Fl%l7u;aZo}FkT3T1oC%(-(d{=7`_1o!+5|GJ)X zH*o*MzctSv&-2Fb{H8lzw=6k%9^2zqe9#>~^y54m`$KAdgUA0-Wv^6Ge9#?_tj2z@ z6!6pIadiCB9k277-gur`&sFWgf0|74XU7MFT)*a*ALqfG`1wt5ywM$xM16IeT*59K%l7E)JU*g~=`TNBZKAXQ!@j)wor~GNJfA97G zFXZoA@kaRbuutrGr1I~rc%p)>c%y>7{=L`#x8jv5zZK6^uoX{KuoZ6%()*ndA7lr| z_D{=mY=4!~$D?EE(*Ax)uG9YpKWj*#==E&G2if1(@xdM~pHv0r1H=c}-{<%sV3+^E z=`tSah!?W||2q^fyp;6OCm1heh0XlM{4+XUI8Mbg(Ld0iO%b_Ts_hBPUt%}mOYegN zf3Jc6*AH0uVQ?;ihqJ0 z#7Ae2OW5&IY5&^zaJ-Y!X;~HX2m99p_WC#G7ko75Z1GN7uW-gYt@R2O@09U6;9miD zV*LXB3wz0qcc#?(gpPNDL9TdbiumJ*cYHV!753YlLQ14rD#xoD8`I!9!M|+58_S`++UQbw^pGADL)DhpTfV|eTE3W5T zZ+sQ;Oiz4sT=wxzj9();!xc}Q(pk*$OWq`PvJUG3<2b%4;R1u=q0&D<|3Ticz7QlB zJ|nMZc;lB?Kbb-63%We5f4K6GiihUsJJX@h>*u0=tyj}?yqWZoBi@AeUH(7D^NM6FM)l!h z{uv#P^#Jy-&$nJd@lMBjMGdV#IO3yR-VqPw^$f@Q#ADv|398Sr{=(%Q>not|>f{7R zd=&OVN39_LnAaOX57tv0>l5Vvd)Fr%@lm|5-xc5F^^+Nb&dbX63!=xdzEVZ?%k>Lh zAA!As^$S)YR75;a_{Ze=f7b7upXVRB-of!mWiKE<{3oopaQt$DieFOvmG>7>d8Ris zgXoj%6SUq^SS#&4Do_3Ej#qluOR!!6|LrCf?+j2pls>^JFX+SN$#`iTkc@v8_|Ch2 zf&RKsd=v51SQYByHlUxCzQ<%k^N(8ZIAH7drpQrmeABi5SZH2$IqC1q2H{2$y@(&W z{i|^vxWtAb@5_?Gd6yBj?9bG3Jkwp??XS7~nbj1}bo+1gKHwCpv-}yvSKalw{W*#k zy3^JBIWQE79_SP4Z&>lnFxRj3E5$3_~9xf-QeZ z!IrH%g#3?=XX^8$#D5x(3GN-XAJzUw zuRk+DIV^vs#?BA_pr|WCwupQ`D)>d;??GJ-|0YM~Cw~U^72YQrvDT;HPhmXfc>EiF zf0Ue$$-l|TNpTI9e-n25H<8H4>)$xv|9fHo2IqfW`|W8x80cE@V7VV$#b;rkl0OqV zuRnu$toCQ156|n*KtDo0RXmpbnL2`1Jl67OR6N%5XH+~^`!hP;no|A?FIN!D70kzL z0ONo5XLLLkIWRvW9;@H~rQ)rl;;{jmAME(7Hy&GU{2tP!Khq3d>hH2Cq5hY#%J*nW#wO?e-ZyZZ+sR0vHdpq{I#9 z<+V1T zbQOQKV6^Xw@3Q^xh~J(fd)ylj4PGD~>h)io`xkekK8OFp^*j8ZDze{0o_MGup3V4q z{h1s`yp`&oAXwv)K@9p_`!Qd@N44ND)FO%XCD%9B_CHQ?ywu)5`9d@5n<%F1{NShF zAIb5~*LD-W9RHO50>H9Qha=vL@#l)~%KGj7pPb(v??wHzom^ft9y=WIUGV!t{tCrs z<0#EG66_g@&yv5xb)MWUe-ML;z=l@y% zffTZr$=_hy2nPM^&uDx&KAY#jrN%pcu9Z(+YPt?$07`V;w~A83C-iZKK~aABH{ROy~Us zVu0)?tY>xQiUX$v$%Jr@{L@ zke);P(FCQVKKFb@^OIw~sv`O6v-$N-g$SF9`*;Jk!bi5J_h;$))&4B+{tLq_1$n7O z_=9~{2mi4Q?is-lEA}T#@>M$Ko83qs56hj?1>OhhYg$gH8|aeu8T;$41UurBt>=zU zu1Mw)0`hvTBDf-%Uy245OzVk-C6<1DtpMq<5_x_OS-{T~$zUn#D=P7P9pG@O+@A>; z{G|2oyjIeOzz64jRwRuQnmYO7`l2P=KlA|OgZH-pU(Wy9==Jg!o|8q{$5=B9*DT)&s*~h>@zZGN9+e0bxZA4&cP&LRKp)SnLdZ~rfrf8cXb@(+Bq z{JY-2kN$M+Uxj?1bv&H0#{=FcR!r+Xh==CV_rK`-z}EZov0eaul>L0c^Yagk<-}gz zpVoYq`2zbRtogz-o`An=zQB62OCQ4Cam^pqG=81!@%$1LdVF*GQ3pO1Sg*u>j0E8) z=PUSg^8B!xPrUY;YdsU+Z+6f3732@f^T=wx3Yokc5()rXfW{Zzf642KTHl*Oo=;Z# z-k+Q+=Vw}PLH>|gPsmy2XLl~-2}JKt^6C$NGT|yeyYp&S{+=r**#0K$0pO$dFaMh7 z&q%UN#Rug29@5o*=3lAw5`MoHuLnZ^_RJ`?>pSW6hv_^j^ha`f8Fw#z_MG9%O5}QB z%1`>ca?ZFy_%kV{5$Qb>5ntEuW2i#<>O!N;kv_Xqy5~$k=99{*q9Uw!TjeWnD2ZVI zmj%b?Oc*E6Vjw4x9f*|8guIer_&^}x@`h4 z;j?;tp*$ZAd|n3r*WvY5IvxFI+&tIbe_Z7k@f*gK$n)38|I%*Jb0wB9bbiA3###1x zLh!pXHa>>)dkVig&>I`e_KjN4owC{pQoNJrV4UZR$BNZ?Zq)x8{MY1sejgpGp!Fv7 zXDk==tNPIYV;9=z8>{j~d~qTB$3mby?3EDgVZPM(sPvB=i+H3gPxH@`+0wTn1lFLu zKM&udp({b-!ylg$mAu11kO{t2Ep%JC{FQ&wz`3p4+ z`i!|(+wnW#JLQH*h(<8mprCKc;%Im_V8$Bu*A#zlK|+^5+AOBb$<4VMF!K=ls%h&{ z;~K!y|K{~v0~?BAGU0#J2Y;f5{A0cUq}p6CjmC?PFFko@D3j9mJMh`LJttz7Urq1B zna1DS3+%Kpdso2XU(6yX!fwcywAeaZVPrzViSlJ*mV z|MztW-}E?@2Uf^W@MYC*`0^SBqrK}8&$jG2>^G$l%;!1WvWFrm8O?_ZY;^7}6N3GnqNt@DiBpZ?@HDJx2U`;#St+qdBHcrgYpf-!&L zeFK%nDc=8r^7{PHvzRRKKm56SwiO0~GTk8l-?J?k2==S}GNWrKW61X*A-}XYDV_7r3|jsI#slr2#PGh3>F55bwfhfn3Eq!dxZfV1 z%0Ged5&qb3XGUhe^y6I!(aZTMV&-1zh8NYk;dwKhaGCm^ilO#D(fr2h<|OQ0tdA@B ztAfrup*_HhkPcY-pC#rT8$XN>!;FF;_OCdeM*7@QAI4{p*B=6^y=wCsJ|CyXH{zLB z&7t_Vj`tdfZ=yU@NJ_=$mU`&R?=0NU(CgeWq$qsmj-A3BA#r_ zMTNK$JYEoQxsuN}_e*^fPcB-73aoSk?^AHLkND!$YmnN$QpNK+8bu>zC!#Q3ksd^Y z*8UuN-x!Izq4w9ulk?`GLR<-MpX#Hzpgktf|28DApzIWWI%zW~2T|C1_cU)t#-`@-m$M;##RTa9?zUdRW* z6c0!H;NMcFm+Rjx;$3r=5WS#>-}fHxYHYM%&cApu<45_A7Wm>_i>bq9eokN1AXfyC zPV;GU-kds_PB7$W&K#QmHNWD?xs9`&aMz+m?)n-U>z(PzS&M1DROL}${bHJL?fPaf zO4u;c>lV@csnh9wV$g(m0Ra9zSx~qlSv;qhV7gA$1-%AtwvcDxY6pg>>c#>bp8w%;&{;3~>EMJ=MoN;dHsb*G`Ytll?92Gmf{&_m|=s;Yaoe znIj?(wP3!*`zkoTqRYGdJFb7cJfEk^KaTT|)OLu!XWC5h7Cv9B%QtjQnk3~5@kA~k zmiyJHJm}Z^z3lwu<<9(=Pp3^oR;9lwUNZ?V1on%?cnEx#B_q5(F3;aoV7+?kl{A0w z`v}2r2X)un$Ll6li;g3|gZX67zvM|FDq z$oNSQ1QP$h&fk;HBimW+XQuN7S4dduZ=7G6ipoSTjR*MCO_5eWbisbbdIQ$S9rkb? z?eC%XE&X1G?`;zQ3~Rj`{^47ApuT5YWPKkP^C9?y^Jt83P>m0YA1i&q_65~XebXKT zRp6IvJsxnGIfnV4NAo%Q6ZRP9TkxCguhseDJ7zS0B{fg-@e+OSFRy~|%~S8UZ0Ipm z{1^1X{=@fEjTF+Eey%^4>#xxELYGlNz676Vf`3?MI==6T{MB6s?;phX6#0834Os8y zu|Y8WnTDh>hGCjNP`)S`~6%)FB6y@KdP|CE_-`n~-#x<96I z`KeSM^m+LY{@DD(`tc=QS8#t^q0X~4bWNRV@#}?VH$Q<7_;m%_x5O{h-;fL#^u7?5 zf4qMT{JVn2Yv64Bd+m~}_;&`Vv%g0Cn@*#NE-n87LHb_%gJouoeV!2gNA?ZrS{{E2 zX5C8j6~kfPN>D@avF&8P$nT-k``mbaZxuC$OSf0U@5lGictn4b|7pdKYVyU7ZAJ}e zCw~zA$KMlv8&7Dw-9qWn8W9If>Chh+?mxL-N6Y){&Ky>aJn27ww*Cxw`ZLV^$=}li zz13X-IJoE!l7GlE`YXWmo3fvbj*y%?@{puqO~5&>jQp5mG_&2 z9@vY18n5lukQ45|ugSSu^&jGQJRZ(F+D`bj^UQ@x}He z)>~P>hDz|gE66{ux6=3IC|+7c`vq!4{o7goc)n}edcctOHTE04U*L0n4;%f1`t1EM z@_oWV(XZkk)sTOz-&*BQqkNa+dv5yu!-g~e^RE2+&B=^!knn~4Vn0JIs0n6%4S4|E zgZGyv5+cDAGF;(;_u{-_#z7CoYpsj(=Yn0z1qO`oA>{G+RNs%G@+1qoy?tgmuRm!1 z?lWU~JYFe3quUgS^Z6-;8rn{)hTI8LcIW(D~{6FK&5= zx%x9F8HNH?7t71ksJp)lJ^fwH^RKR-z7J^iKgyH-ruy{x6UZNbpH=6l@q825tzpPV zbaFf+%1K^M`Y_(h!mTLpNS_*$>5!iQ(o1-G$dO)MXr+hv`x7)>@dbQ(%m$vHG)(en zSow2EzL1O8j6MGY{!@9UzM=9? zn7()Dmd|RJeA4$^Y<;tv^^sHFdhq@M>*D^NE(%jl{ddLHZh5{847KHJpE>7hXMOw3 zdig$Yr7!oHv%qj?{+U-f^7}Af$oGZv`>^?ax(g~OKfz~sK61<-q_4ha$n^%yKc_SF zeVzucpWbIbq~U(l>{>J(wulMdtCJdVYb^Cjg5vd>4_j6p| z%X^LHlWP<1{#crD^LKH=!C%OKB(l;WkG-Xn8l2zH!+1_i#tIqFH>5vm=0%<9dy@+m z5dOTs0qrCIwbe2|=;?wjCc3` zU)Vnm`QZ3(RFHi|<*TW`TOH-;`;i9w>-@cgv&u)iEB~&f$Oo>Ze^q~lCb{LMVG`*_ z-Cw=t{CO4({+vZVVjlTu@W{vfbIJ$myAS@a)W@kj@}Edv$l%fKgT8r|{FA;&Mi=n> zMf;s8Kjej6b)A2oximov)`k}+1OikV#`n(*r@VmQ_WO%bKgrA0H5pNcqjfyrlr~Iq z%1dv}{CO6P_DNo@;QT2|UM_RPi|06C$csVzcgRbD4Tng65a8A6_&ylz7u5C);#>Is zprb!`d-Q|akLJ=3HIDpQ=?6Xv%KQR-XX)>v3c^pl-~XcY0qVQYTrjG?-TJ|p@9ysf z(FN3=qrWdsKe+q*T>8PXFKXt^xBB;WTOT0Z>R-|a4w&_UhR@Q!4toLLBc=X4pvE)z z=RX;jf$vJ;e z`HR&@H4gnE^-1C?=nEWzQ2Hlo>l-DH^_*_Y<19D4c%}n}JUZ-Go}aJ*L+cl|Z+XE2 z&(K~D)-MO_?+v2=VZS)${|X=Kn`oE!Aw6Q-4=?YQ@~!7PmT%J~pDurx`+JN%-|6o) z>-mn$-{UI3Cz)s4L(-qd%B6EWe9e`=&LnK<>fw0kezVGBAHluF@~563M*G)sF3tZi zzEv!rRhE5}$1v7YdAzbe<*+aJB*QLyXTLdx+|rJVjh1fSyknh zPueb=vi$X4eBazD@4aTO%l_ME)>OIY%gHYPf1f$E$~}L2{r~DJ=X|=)oM5j9ys${z z%i0dyqCtZ7aLkWYd2xF_1zbb^T$!y;rIn`Q@)mZ=8(_zL$@1)&FP-`o-?PJd zHpV-@Ul{Mt$R&Jwja-*~*tE4eV(DM{-lTJVK+0E8u)aloYiRzRXXnRxi2JZ!CG{JY z$RQulzq!`@h4qfT$;bkV7x4VW>qn8%-=os<%k|H9$Zwsn<@X-?K8h$eQ>YF4>x5kI zgMM?_^Zu@&j2$5qOZdRP&vn{&^gYc)LL|5nq|^R>>teXbS=Lty^9v)Vk#p zuQzwtcjf#wbMWU(lGi`>LjOC*S6-E4JW>CcwA`u25Bg^x;u*Gm(F{0GeoxHu_oBQ% zNb3tLetJK+i1p%z(m;^x2lkiIU-v>6^F=~}_KIc(_&_7u>sTLZs0jpUymEe|V|>&a z(9lx<%k{8)hDqN?#IdTepq8$Om_Fp6kmpDTTpD%2d(DjeUSL1gPkvrO_T^Z6Jn{OO zV|+2}m=7Y5_nADD!6W#iJRdmxS%PzH`ycz0*}k#XLutJ98PgeG@@HwiBMbwGFGI-j z6}If#YRZ4el^^|Q&%ftdf93HBdFS#!c9oBre#iS8<@z(r+e#fjI3%%A>(A=^6nKmd z!+6Lug{Z)-{@zMct~qj=k@&rQt6WAk0puwbU`bDKV{pfHFXhv$sq5C$jp`Y;=OpE9$&tP zcz@~h9lSkAK7)S$YrOwh=0MmG@yfkPBSS5qJjXBnU4$ww(ud&3^N62V*W>x(7uzy2 zUD-d_Kkldc+T-LOW4~9IU&<@%(>lNo{q_&dVi$H72hcxZk%FI|!>}LxVVLYG>{pYQ zq@TX;b>;~?A%26uOSjDwHJqWJ`P;Nj^T*B~KXC5+E_nezrTn5S^(xkvE`|JB7t*gs zyi;CO|8Acj^Br?U%@;I(@c1Mc_&ouCC4}z@^Z7dTxBGlu1)Z;x>4nHo36K=5Iu zKi7Embse9F=5!hlNO$XN8V_Bg+H>n`$9Zkt-hm6WN9pHm?`xylyC8iqn0Y=r7k!j| z&i497wRb`K4qbpgNK8WwRb^&ekA%{8r9x!m%f2f?frJ?iwn~upTAxDszTMpVHTQ{2|>v z{zzZ%8r9wf$Geu#bG4_(-&xyxcvO4*zDLU*49N3m&hf70^IYwnm%hPK?Ojk_J`#OD zAJyIk=ZC%v%ny`)PJLZ1x;*lE!S+7V{9Zb$z2C097@u%^g`aSHpEN(?pD=wjpKyEi zpKyEgKjHQgpKyE2JnfD1tiKdH)?au&DjxR?*`L@WRgL*5Ekx?;55vOg#rh5W!|w>0 zUMSO>(q|m+b0>Rn4R#aLrPq6`^Q6k(JWG3cpH8L-@fB)szo)%FMthWADASwvd)w<1 z_juZK>(8^a*LPm~x_BI=pHp8S@U)lX;pgb}vmDJ&n*Ut<#Qgr9|G&NSfsX6C>-?M1 z*q$h=Gf9;q8n>AwYOAzrD%*;qbe%GCl|(5D42@bu<1&&hITDejsWc8sF`J2PisT;#syn8_#mr)5Y}_#%G!OD#tUjrQ-NZQv0ssd|#?^ zzH55f^F8g&1mbxe=evBrScUt^^|x!f_Fp6KcXfVQ^-fWJg?Q@qZHe=Jq00HL>1D?Y z*H`KOwLgDu*@}4T`B!|toAk2hd$%_iXkYRDI=Vk!6G1$+zMpXCPyc){>1ErO^X3Ef zHPQXn_wf1mo~Fw2H0fpQJL#Pc)K^F6d-46Hx+?AS)63SkO!ZyI`Chrc$QF!Goj(Kb zcj)}NQdD1Xe3qDBHthJE%E1*cyaqr&-W7XT(R+#@`C4s>|gx%t86?=*4I%~U!ni>{hSi@A)e*x zLp)2?H(gZUO}L-r-`_1!->SAx9%Iw`{@gv^rG0-Y>$|4MO!}jL&ikbcMfFu}ANsFc z`!GI>+NbA3i+ASR)_~wGWF%``Pa|qpIwhMy=;9gMfDZVhcfla^HUY? zmuq_2`r<|PRc+t0t*`MGnqO`S>G#b>G+pZ}Iv;*P;<>-5zC!zSe3t0HSe5!Ty<~lN zN`3uB^%dGj_5H2Cf06I!L4DkR{`_gu%hs3l&eM3|_T}GyKz-=H((_AA1pQYOPi}6@3QZYRf^|SQGI-WH@`na`_^oI<<199FB{JjMfF{GJj>NrF`k*C`l`k= ze?IVddCtVM?D?SmcfP2;SKq!-Q(wjQoh_=bYWwEPjThTKO)uMjQE!>fhpOXcy-IzW zUbeogMeTcI<5^r^*?8jpv9jl%ji;uUZQpQFJh^@O^AGi{Rf(sjm#we2sJ_DUHfH{$ z=ZVaGWk0VKT;FZ`qLu1P7S&fZo;vtBa=fzEgT`NaJFx)SUAa=G^1c^&g-Yf*hw+gG;#Y<(5` zuPIR9etDkIKi|9-PoEDCeE)?!Z<&1d{MKea9xwXO|1qzBFWEozc}Snn1^7pwy;``x zoTI-N`u~A#g0uPWW&}1Puo;2P2y8}RGXk3t*o?qt1U4g}5zuS#kmu!oKtIO%Q2O8Q z{+Z4_4~D#L@@d7N-1d_%-ulhwzWT|Z`SO>q{n6L1PJHX{U-=&|zjXQj-+F2E)BpR< zZ#?=>-+$$mM_>7=TkibO(T$!vfAoJ{ed&FBzV)N;oevLw?I%9;$1mUVS3mJ*&#isW znODC5?+(rXZg+xclpW{o;@N8P;F<@+-gkwO79N&2N5g{o1uJ%O%dQef^eu ze)6e5z3*!e{j4|k^_O0{^Fa_;_Mlks+8}GgU8$bRFzkOt4>@PnNZrt+X4}Jcn z&-W%@`MVmgH}Q{Omd}puoc}+6@graT(XV~^+O@x({l7eCaDc z@^@dl-hVOm4tXIh>*lSsFZ=1whnvOTSGRAgqt$HnneDaa z(}Fdj4@yztx?1_$mHKI^v_q?pFw0-+KYxSqV)dV<=m>xdY@70DL9kxDN z{a~b4+O_0G!rH^%lp&<;&RN^f-ENpgSF?Ha6QhzPpGR#!&kdxzb(YN^{+<)NHoX2` z{oLpa*K*h7Pj5(2Oyl&k0wvRPk|2kE^tAoaXKLG}{p}sz-a(!+sZ$w%tQzhJt2M6+W0 zT9x$mD(M?l(mnaid@#QL$By&8EpKU7OrMf;x0g-7SMF1J?3&69_PIRfT`^s5Q&vom zRY`BCk{+*;-c%*MrAm5RmGnfF^iD}Hj=8ItbFFeOB&*cYQzgB(O8Q`x^mLWeyUDl=PhmaW|Lvqhc<&%dRj^kB=E ztE9_gAUE7{c@(T-%jb()UVZIYDDpGd&Xp?ZtFNNYwJK$u}ey6bZ zKOfSiUh^A&LK`oHa$-mA9!>0UF5G@FTB~~t&+9JiJ-l$O{^>70u=glR&-uyz-lNEe zLJa)x1#3SLu0{LIUSywS4uyFyvLt(v_B;NjHhjnb+$DR6U(p)Gi;!#&6WB=T8jLvU ztFyN$DlaGH71CuMrM=I*)+ezK-?1cn7X6cTFI@9<+vOZ)&r;qWUGsfEDwN^xYwq5@ zb;I9-)r8m&^)uL((!TC{*S`7^@zL#bLW=MYjSt38=-yaF_v1Qp5}|Au`HR|IQ?1R} zu+)*0;gqiD&UJ{y!Lx# zLX}HFx!Tfet#HWGvqwMKQ=KTx>9NvGV%CY@NSR9-E@p+shCB}1L9O&4R zvX-T7AC`(y7Hq$5PP*^AW5YM_zHeGL^}T*kP{J=9#G7$n zShIPrY3}-!AM<(#FLbXL(fcQ%jnKB-7H?a@Zr_&Ol5p(jwz4O?I9pVnGW*_A|L%%Ew_SesKKtXBPc}67)&ITM+lq3(FBE#L zRw9+~J=K4l9*y`Tz}bOukr>X&)vPlT!`V>6v7Zko zr_{ExZ0S{+ML zhtFQ;Jn!~)sS$>H^;LT3rm)Np`7_O(ALKEGY+<((Tq$mLSBgY%rSS7rnvs{KMbw*t z9ge=+Q8?GWJ*$MiE8|A4y(Uq-N>msv3)-f_ioU(nx#^!DdWXHrYIHrTgMZ%XDnjRG z;2d<$Nt~a4x;;1b%ql)VRdrSg`Cj*I@cNeT=S$09?=kF-(bqmsF}mO2aVl#>)k zJ{4v$e{@wS!;g`aVefhVvTi7KSLwV5SyO5>IlOfZDNfqYfiqXX9>kd>wA0=6Qr9?@d|WLNXSckP`7laeQIo!ch6{e`mWO@4ZN*>rhQK)bNyyrurdO?sqU z{ic2G;Zo^N{K9^`{O3lsK6!q?J)uOImU}+VJX%!VNlkBy%QFw{a{48710J|(8<#SI z=~@3H68&jr$(BwgpeAftakrEoM`CO=2oxzbsWx9NTn^QG?Hn1$^#e!AWt zlY3Qie*5>zXioL_qxt8c+e2Hz;-|xC9fh6u3)>PDSH11ypYwEui#>t9nl|^@xNh~= zmO`s#bQIlN>oE89dLvu+$xrbv+HZ-p%8ATYjZCGdXI7<$O}=hFBH^~}a(AxxR`ukV zK8+CGCS??k;;XmnrN6e4OOIT=TN%E)$&O5UZoM`zFnxzBdA1hF|9nyYxRjuAG4V+F z@d%rE$ddoS3fD+_R&JZ+!tywXd15bY)AcU(v8a64YK8RhHhDClLi({tIKt_ngUuZ; zdp{P|_9V9EuDNzrZ0d)=zG+$d$)8JteHCD0mT9l6ixgsYJP?cGSRU8s5W?J_h5W%- zmWXvAe~I+KGb8!FD7?R>eVwmApYHV4v9P{Hw#}0_p-T7FP5OvT`SkEj(iaudrLQF{ z!TfIDB;+|=*Y>2Z^o-XruRnGl{i1ZMf5JPl3q0e2*gL#075L!$s)(h-%Uo3cJVERn z-Y>$j{kKmX+t1?Ezg_y`79I-@Fa6-wfBY>jU&3Gi4IiicI+riuuYZwI;A?~rSMtZ7 z$_xDQn_a$yKmM#$;A;ZB<4@WPhvZ{G&uMdYFFir_q}ch%UD72ea=Gluw#V>l>#u(M zB|Sm>3fXP)G@4spg6t&OJ)XQjf&uW3J&j&d&fq< zg7?R*la6tKbbs7Q`kQK8=`#Z2b0Fq4hKlwIej9^hW`f4lzqB#*rs@;!k7{@t5B>)C72Et}nwMkGbVLbA5nk|JcSJu}LPK zaLnu>Z_3@2--K8@qoi zHBScnL~a^6&*teh`DUNIv3*op-Of%U>T+U!R;#W!8O?Pcjd+9bN*7X3#@xFxn@ml^=cEfvJRK_fy-puE$`T^ z$F>*BGxvrn@v`o|bXZ_fGtdH*l!J_?`XH)@xfr*5oAb4eVf;T8n18Tdn-IX*w*)CS-VlY)L!sL%eYa#)V{(U`|?Cm)pLdGboU@7?`{4I zzs?No0b=sLV^%)ac>~v!ZhpCUTUW?`UDuE|-v}>zj;g%7=g{}Ci~7rX9>I%LPS2VA zZ>sf6*Jw8eIep(=v`Xu=-jgoD1m<16C#vTxl@s`G8@)%@F5l0noR+8O8$BaaEANFz z@qP#HwK)6qz%w*hCY*GAwxE#T0)_Ny&*0Pq^V@qdh5X02%KIhx=X~&fNqbkgHZabk z!pOZB^K6F9&2qTu;aYi*B%fa5Ih&ip71P5v9ox$J-9F9Ivl)6X#7)o1y^laYMJ3YC z=OcCRTq#}mZQ6&<+$L973IBT<0{Nube2-T9KNQ<4pJ3A74MpWKqYWJgg&1oe9jmP{ z2BZ(1^x{4arrSQ$I`yl7{W?8wBRKAk*GRwSpSLLT8pW=RTiP=(R$oJX%W#n|$ zE>nv$^OvqA>y%Nl7WrB_KB=gT z0I^W2jF?@;{B^E$E$%s2x)%R&BzXd{R@zdsmi&3E5i4DW8;|P5Mya;QM!>{FW2&F= zDiIHds9wg(tE;8S^<%Ya8E=>$fq0bm!|mJ2s&iqkWLw-a^eVJC>MpTdm55nF?jF3^ zo_u;!-K}`D#kc&9730!x(dM(jxCGSq8eZ!&1e+uJ4SfG-bG&*9`drFu9j}4s#>&qa zSf@67;C1hT%^72J#xS?$7FA-@K2pc^_kQ#?TovLEk3#dVjmzo9cO&^6!>mx23wn(N9XR}RS^xE^;rl|j`VOkAd^l$7L^I$-Wo>=Y@?Y=zR&DvS zw!K+vo0rH3qW)jM=@c_^Yt-9nQ}uWJJyQK$zXaG1)NRi1ha#RfHS$2j`;5MRBJO?m z3wO``<`?P@duQJJ4=?r9{K|Wp?`rwOcmBcWZmD~0xb_3Wd`F#M!w#<@625X*-oC?& z1?;zb(SZG(lGR(kHP1JA8#QkJ@9@?G_S?L*nut^W?cQp@uJ=}I-16S)EeG=7<}C%% zZ}oCw7yONRi#488{w>}@z~1i72g-YkH(L|_$3p%(Zzhm_v;Gwlr+u}a{A~xvzR8on zLgLs_Z!%!Z7ybv@C;cetp)letU+5opZTakLz?Ki1hTL>LwOw2Pf`)7Rov7u1Z^R!n zp?0ZI21w}pnt4Y;_B0C$rh1)$ZCqLfXJa6G0^2sXs@!lEoBKMxppT#=z z-|R=E>AXhOOG|Kb^4D9Y-YVxzz2~*ajG2(1^?vz)?||oJ<)`s{feZ?PvY0b{?GFJ?@DQuUmV~+A;aYa`k83+AY@Juy)ego~+WEKW%OO)QQ?t)}FGq?uWEI{rsJ#$E~fO z*i$=UZC!2E?zMKz+F5HiSbNsmack$S-DK@mYqwZiCs(b%&DwEmC#;>YcBi$Mt(~;? zy0v?(9h1*TX?eZYZn5^DwUgFPTRUy-32RSTJ8SI)YfoBx+1gXqUbpt5wPW&)@LGS) z+AY@Ju=cFA^)qoA&z!YqY+Pr={kOEr9H8C7+cSmkk~EO*72j=CAW|);8tWdhtd#T|Ve&ZIdqf-{q!DerqHD`n%n9`S|;L zjE(fwdxPoTd)@S?*ZV#<gBpzzTS&H?((;Jv(`p?ld{gEze1tgytK7-zSMXwTif)HjDOh^ zC~xvdz4%9icI=6u?R5w3b+T7K7EE9Mc+k#$f6!j&3EIm)XzWGX-?68Rt@D-kcZ;=k zeyWwPHt~i``i4zUTU+NRZSShA;o80#o4$|^mLGpMSboCVC_nil!SdIwjq(Q}b{;ai;zHDuj*YXpF*ZEZaUyyx;mcMNMU$(Z+pW6P}X_Kz=tHx*5+6y*+ zTJ|KGU*}g%Pn-&tzxpe|^1c6HY?L3LH#W-8S=*Ff>m`5Pq@(-=Yn%MiU%wH|pS3p9 zb7Jd!V#*Uw7lQVb*y_LbuRngRt?|?L#DB9;zNY7_t>ydazZJ}%vo`XtT6@JWKk6lZ z$MCwIX#QSnYx}kQb!(&iPbEs`Nvc^(U>Jug}`3FZX-F z`qnQR8~(=r%Gjts{uN_S`scrl7i;INzw6eH`nJsPe{J|>o4#!An6;CCW73g-`LeN* zKlgQG>-nbfn!RFd^{-aGt|k3XL4NkCu~A?8m0-I3t-$>F)$-!$-L-|DSe8~LYXuO1P9 z&H8)Ii^W6!wY%AG$ogsReEMA`9rhI2vAZc#0x>Bm#jt!>tGNpCXgJxI59KL5Rx-`e@~_fopG^Xc!SbZh6+n14-tQ99YN-IPvttTp6+m(n8Q zZ*8+4iU0ST^jV~ft>YJbY>%EqNy@2!&n(+^QN^Bkf;A8(LIXk%?G2RwkV!Z5 zuhehteENq>`U>J>ZSb-0GU=;Gw{|{%g7RBCpTC{*i>>}o`t!HMcb~D%{3HHZ+ssdr z{$Z28jQXt|vv&NEVE%=V7#sN$-)rm{%P&7+>@{om_5}I${l-Rp*}h zkF#>WO%Ld+*1jzFeDsjJ+WD{QQ@c)l(F45R!_wod+{4oY`l_`rzt!km&ROd^>%-DM zJy<_$-0^j}49_2O3i-!Fwq3ZJeclq#aaX;AbZdGIR1S36a{33#f&OY=@0E--{wf?S zK`tdl*V!z1d{WFB@7ACw)^AFjYP^Ob{SQPB3y=0rS)x%+WYrO$tkvtbXgRkPm9y8^ zZ!gk&M87Ro@F)JmCU9>P*Y9#@jLZ*+9-l4tGb#G}9X)RIX3CV~6#c4xRWEfe<0D~( zgR(ty4?7(4lH7_ang0)LzK?Coqy8_PUrz%&v;$x~X3RmGL77|R2kszpJrYnZQ*%z* zuIQvhGCdu4*2RDu%eL!JYhliT{)43BQ}ZXtjidJbj!T*zGCX|WIsapRs(hOc9j9#m zy;esbasH1=`e)=8HD&i!Z2m?{;QV))fTR<|UA6hscBpIpI=^uKNfD%oqWXt5uyVxx ze+y#-`O%M`lY&)CSPm~!Ha~3Ck9_F=XC!l_{1dhS&95Bi*Lg%kAg5R0$jUuD|M(O0 zqWN!91;1#ia%iRY@aJd8R$1pO&9CJL50k$)C>D+NIVn%ceZ*ln z&N}B`ss2mLz?3CP`!D7eF}uywzkSN6-zhKJ{s1KQcS+!EO9M&2{i2%mWW@N_`t_jjc>^|of?V5c^JDyKew`9Hf2+jjvnA?3{p+Uwxa84;>&H0y zh~z&e`4d$CvdzEZ=QjDZe&oaWy+edj`R9Mb)Q|D2_1{(^|NW9*PSLcCpFsTM3qk*_ z8|W*T-V>7l$4az6Ve_N^WSIK>&;ILqqcM#wn)>nu+h4GMz$@}A{c7{+T$(>Wxk7us zdiK?=Pkk*Ofx!ZL?ErD}8L4#paDL$x`MRG?4?H|JHacdDe|RE2a4<6--_;sV_l@-( z9>@%gjXPDxe|T_UBz~}Ocz9^!sSn0`5BCj?q%s2|}rG}-^4!KL}kSccXlH`L&QbH(@a-ySN zfH*KbdhqE~rtiS88;@Ph@sZsQ^{G^5^yz_-)YH$TOonmkrSXAOy6>rh)Zu}{2c)+K zhtdHzq=;6h2+5Ec>KjgtI<9J?n&WL{+J+qCnZAr$vYSgP@O#&9?Ra{4D5E|e96d53 zBciA#x8Z#m89tez(UH{n(8$wH`3e=v96540l^%R{d`QNg7OgE7Si-lbRY#H^f$_}P zP`|`4uMQ6M^{0wd6GM_skqkeRDpkZYwi~F31BVaqOpS|!;)3M$$+8s2)iV4?OwFdwZReO@4iRX#8+rMoyCcct%duxTcPd#QVmcI&xUT=G*tQ zsA`Vp_<(-~3vbRrkw4xyd;}-Cl-D;r^wdZ`J>K7!>5H4p@~@NMPw-)_>7ku_;s>71 z47ihl@Q#0`zCuxcnNE2l{X>Tz+U=`_f1LFGfoFye4y3f14;?(x-^cm;{gl8Q;(6bD zBoiM!6h9yXtv@a~K6T&tU|)K`d$Q$|YP#*ojK&WP$j~sSTzsr=y#({n)>tkv}dGa)PsCJlw}8oli&A8ni@EqbTc0OS-gN0Ci}cg-aXnV zdlV*nC;fh;2IAq2H|b5ZefcMA0zg~DA z)*E)HpcjVQJr5Mx=PpFZdyg~V9RK|IOBV+YXPH9wK)RDZACH1h&H%8v0_7o24;G5N z-UScNTtDx`XJ=sob^Q0+uR{at-zX|6xaandIX~N4yeC@=^MRAMP+_I~JNN8vYi*Hp z+>Ct3A7?!7&UXvS_t)kU{!x074^H{Uf0?f|8u#opRq#^IRTQTes<>z8ZjV;?eD-^G zwz{#CQLS~}FF$+a=K=X?lb;8@dv-}Ib`@p;$6tRX&`{p9YnS|(3B*l5G<0A4*jV4Q z@k66y@rNA@8UIBU0)iV z-<#!kXZrEx^a4eX?z(p*ekME}`TW*@7ya3~FK+*WDS59X_4|PNrxzRb;ktM9=K0#0 zx}V$r?_)oI+o}35zU>PQTRPvq<*^-p2K{%=$)7*9JGUpl81XzU|D2P4p~)J9RmR4hlDZrS2=6di6@O_$YP1+3@BH(#>IX=O&9yG;zf3u^ zW%!XY{-?_5UoBJqMw#*j_m8(qy8HeLMy&RatH&)bj~sY!wZH32kNsWWKB{;0Q}%Z) zU+-_@n6Ea#~n>W@1I=#45p<%FdfoEZ9NpzR?G(U##Ex%{f4$W?{tZ^wa_&H*KIqh;|_1T)Gw{k;a$Cn_!e7tGx5;zerOx{*=l)hSKRTx)6ws8 z^!;}5DSx-4Cv7LOK4W!_gPx~YzfM^l=P&eG;&XPrnmBj z62EHslf*-xA|8HD6Ayic`17{>Q^Z4`B_4jx5fA+|@hi6c1>&JE67T6oTaL5DL(dV9 zcAX<0`V#SI*LmWhFB6Y;T_7I%MdDkm|Et79zeGIzTq7R(W#ZxII`Po25Z_?^*&rS| zHV2sBCT)L3?4|+p8}vG>gGax_h=*QJJlfSjJoFvJqg`?0p*Iqbb~Oh3g6U0NGBp&;alf*-xA|8HD6AyicdE39In78^I@$hq=c<85zM?WqQ5B&`B zgSOs9;-Q}<9_`8z4}FPwTmE_Gt$u-cv}=WU=og8{y1z<1^h?CU&o$zqUuNF=xz4=R z{TpAho?yO$pAmcUgZT}5)au~jXC3j-W5mPHdg7rsFmL_b!MxR*h=-re#6xc(9)7kG z550|e__>#O=n3YnpB>Cwy^DD4yOP91?LDKbe&Vt3>Lng}Kk?_RpM%6hPZN)I zFhe}_qr_uf$Py3z1o3FsB=OKs5|4Rfig@VL#G_p^#6zDY9_!#7@z75bk9B{6c<5({ zN4pk@hkllLv@1tE^mD|cT}#A6UnU;w;05BLUnCywS|uL(CF0SpHR7RPCLZltCm#A0 z;?b@R;-P!*@VD3b`4HnUVs*@i(4)k|&pP6v>u0U?!2Ah6>xqZnK>S5}KJFkMdL!`{ zEZ;;t^k(8QkLhQ?^+0=}w=!?bZ)4u-9mFFJoy0@$A|CseB=OL@iHDy(#6#at{AKHB zFY(ZaiHDzQ;-P1VM?X#w5B(_d*tcYfhkk;1v}=-h=qHIsyQYYTeu{XsYnFKEbHroa zoF^XoY2xAM0`bt#5Dz~WiHClUc=)+QJoNL#!_Q^np%>F9LOlH3ARhWv;^8N5YV-LXu{XOhU%}6))xkrrBOdo%W5h$RCq8MfzZ!^#9w*+* znfoD)#6xc)9&u|X9(oJ$I8R%Nhu%i~x}8t<5)ZwDcpbz$FA$%!<9~&C=og8H zpR2?}zeGIzTq7R(W#XIc{IE_u^bO*%FS<%RbiB!j`4IDG#J>55`4D>4>frP9B=OK2 zh({cD5Dz_0JmSzuJoF~wv9D+*9(oJ$Xjd!o(A$V_v+>_cJoFCY(XLM7p?49FbuCFe z^lsweXAkkv_Y)64dx?jhCLVrfh=)EwJp4RLJoGH_@bd)m&?kw9pC^fjK21FQoFN|i zDdOShEb-9ih=-r^#6v$#Jp5cB9{O40G0)_PhklNDv}=iY=;w*Yd9+MC^b5pi?f73I z9{MWrSobdx4}Fbz#O*Ti(ASAa+^!H0-HZE|efjHC#35pJ%x}=6#3K%M#6yn}k9nq^ zc<2qpqg^|QhaM+>8v8@yp|>z^@6Wd~Z}m3fG0yiA4?RIV{OlkedMEMlvx|7>J{~AXNkvonj;?iIpT4iE)fs?Jn^&E|7GH# zuMm%U>>}~dSBXczTp}L&8u93t%fv(9VBU_atIS&+Umk${8}2(r?(+Nt^C9%8)xpEh zI^v{5(lK^eN)u z=QQ!qPZ1A4XNiYCM?CzTCm#A~;^F54@zBo@4?h=)hklOub$fm-5fA-5@o3jF@z5_2 zkMn4Sc<2|2zi9ngB_8@3@vHX!*k$6OuM-bHuMiJ?gLwFPm3ZiQ=o9lB{EXO#L2nP4 z=RBd;Ssgs?^TddU-atI=d+Z<{dYpK)tC4umCeARg_SBp&({@fe5G#6v$tJoa6)#6zDW9_^YZ9{OqG(XIvJp`Rfh?OG%rdX9M9 zhdf6-^d;gk51%I<`ZDpDhc6Hh{UY&*!z%I6FAxzeAyr# zdh{OubSk=DuCqGkL--#f9{pHPJoED4 z3F5KtcMuP~lX&>qMLhH*@$j>oc<8;v!_R)=p$`%dKZl8jo+ciCW{8JAK|K6CN<8!v z#KX@?;-Q};9)3;{4}F?=_&Gy7^i#yc&spN3pC*3F?q?Q=hkk~5v}=)g=x2#f+wyb7 zLqA9SpgrH0h=;yRJm&ce#6w>p9)4aV9{MWrm|oy0@$A|CU6l6dId#G_q3#6#atJlfSuJoI7aZM)LUTRlTO=9vlN zp&un4erAb>eu8-TIY~V9Y38k;Gt6846!GwLmU!rM#KX^d;-Q}=9)2zm5B)6j*3TUC zRzF8P{9Ga)`g!8v=Q8ooFAxtuSBQtcN<8MtOTW#XZ)6OZ%g3h~f4h@Z9R z`&HthN8S~j-!M-`t&aH(dL8kITa0+<^~57?4a7rlWZw3B6Z2MYCLZ%-3-QofiHDzU z#6#ap{23e11o6SRckb?o**9jB=PX`B=OLvh=-rk#6zDW9)8Xf5B)Up@NgLt&-D)G?0 zcL(P;oJSF>V}64kC4SYO?{&mOuO}Y!WCQWg;&769=u^aFU7IEz`V8^#^Az#WXNm8y@th+b`U3Ir^9=FO7l}td zo+Tc7j(DtV=ZJ^CL_FGco_OfX#G_poh=;yPJlb`Mc<5`yV_v;XJoI(q;pY|Np>Gfm zKd%xG-T%WGG9}{u3%rjXd5@VVF&{#YS{-~2^C9uj8;D07b`TFePCVk!NIdi=;t_{t z;-R+?k2tgv4?RIV{OlkedMEMlvx|7>N#fyWH}TMWh=-s1iHF`#Jl4%Y;-L={k9MVr zhn^uG=jjCT(2o+IwDU=pc<7VFW8FMSJoIVev2M-~5B(JJXxA+9(C3K9x;alg^wY$n zT?@oRUu53Czjl^+tLKP^pXZ2&zC=9wC zewlcT^L65(UnL%K@S1}28}x|PF~4Emj1mvMj(GSPBOZD^@n>v28;FPANId*(A|84( z@#x1E;-R+^k9D(+c<6hHN4pZlL+>CS?dl{RdN=WCR}b;f_Y;qKwU>D4{lvr1LE@ng z6AwSr#6!;zk9Bi`c<4uo&tX0!9{Ne*5r-+_p-&T!ILr_a{S@(t!z}U8=ZHrf=81=X zhIsh7NIdkj#KX@V@zBo^4?mZPhkl-T__<6x^b5qJA6JNnev$Ybo`)eG`Wo?gZs#)b z(ASAa+^!H0eS>(+16PTM?*9QBXMcFXj?c)w!TA$WxPl9;p9mJzuoy0@$A|CBZ5)Zwbc(kjBc<8;v z?V2DS`cdN1t}OA;PY{oGO%e}%ig~+_nr7bWr-;XM zKC{F_pCcaankOFmY2wkY1>&KfAs+2oBp!N>_?8#V{j_t$Lti2uex4^D`ZDqG^8)eE zSBQ^Ue=ZUa{Sxum=dTeDeVzCg+mBa>hrU5P+I5w9==!(U^}zT?yCPP{^)mD*@n}~a z@zCpu$9dO4JoGs67>A9-LvJD;?P?|-dJFMrS1a+*+lWWI_7V@hgLr%nv6Fb{N#e1N zbQ2G~hj_GWKk?9eiATHoiHAN&JlZu(JoF6l5j*}Th=+cZc)3>J;(N=ZMETHBUVB)5OEi1>&KfAs&7%5)b_>@$fT8 zJoF{vaUFD?c<2|1$92#O@z5_4k9MsR5B(DHXxAF?&@U5@cC8Z+eS>)HyRH%sJ)(a& zL=Mb{xDJY19rGvjI^xl;81c~SiATE{h=;y|c(f}{JoF~w7wmX$CLVeV@$j>ic<61! z!_U3MLr)MtW&Py0b_; z^s~fc-N_LT{XFrA!!q&EFA$G?@(S_LFA@(wSBZyyiTEXZzN`@s{R;8$bAx#3SBXbI zdha*$9_Bab5vyZ9#6C7kJoGx^(XJTr(CdjuyBdgx-bg&!)kHk>X5z8WX(1kZEAjBN zjd@zA@7kJ--)B#DQ&JE z5|4fIS>mDRh)3Mc5f6Qdc+3OmiHE*S{IZ>|E)Wm>BJo(KR*8pxiFo+AMm+S(#KX^Z z;-Oz59)4~R58b=Z%#)Z8u}_Xz9rGvjI^xl;81c~SiATE{h=;y|c(f}{JoHB5(XJ-q zp|=o^eMl?u(DxFLb|r|1-a$Ot)k!?`F5=OyB=OL@iC?$lvxj)-y~Jam*-t$5LEh{wEflz8YTiAUU~h=)E+JoYU!#6v$tJlZu&JoGu@>&}_?FXoAd zewui+Yk_#^XNZs6`E!wY=;w$>9F~ZOex7)Yt7YP$Umza)q7~wyUnCxJTO}U)CE^jc zHR7RPCLVEHCm#A0;FTaoA5h^j_i- zhkoLr4-(&E_p8IiL!TfXaX3mm^epj+!wKS{PZEzfoFpFl6!D0|H1W`9h({bw5f6Qq z_@rHr=7@)Wnt6LZE--KPGsGiqi^M}eOFZVW9P!Z45g)Vj`4aKamx;%^d4YK7E5xH; zE)oxYm3Z{aCE}r9CLVv+Z=HDPSBOvA_nS6|hkliK_=!J+i}?_G#Qp(9>?fOSJfl_z z550kS__>34=yBrVXCv{@n}~;>&BQ}*A%3s*U;dP~f53m}3F6^r2l3E5iHDzE#6wRK z4?nw!hu%Yc!ur3Tc<6(~!_Q&jp{I$5pBdtzPY@43j}i|(OMHj*{{->Sr-+B2)5Jrc zAs&97A|CoI@$hqwc;DDf zp|27TKQ9pveT{hdd6{_V>%_y)E5t+JAU{d>puPw4mq zcG~Lt_k~@3#_IZaf$dj``23vJu`Xl0&08Jg9QtYEG2Rx4hkk~5#B-5&=x2$~;JFv# zp)V5;KQ9mueT8`Vd69VNtHi_4OT?*2@zCSM!_P+Ip*Il^Kbwh%-a{5(lK^eN)u=QQ!qXNZTN zr-+9>OFZ5mnIj(h0`c(k4DrwxiHDzOiHDve9)6x99{LjTb#@%GsaflKRy^i>98=n~Q(Bs6z&qm^* zHxUm%n~8_sLOlFzB_4Vk@fqv?UgDuAh(|wm5D&eR_+DFn7xB=09sU;SqNwR6uixQa zebDM}m+K(=)nYb>c?pRgmS0gWuAw4U2i#_Vfo^Xjc!Q*`YGHEBn->p!;$|Z(k)E}B zRO-_MJpA#V9vB-L7#>d@7#istd)B*eY+$(WzC%MJL#e*8vA$=$)WIYDeO-M=Mh*^s zw~szN;WbWJE;BHZ@lt|h22w*K8A+YS09^CUl-=Y2k_qQDwIPhTWgZ+md z*mLOqwu1u$_aE?!6n-e|X;jyvNu_p#HAMG0&ALxH%Z1nJlpU8|Q`^GZk zo>rggAL<((IG9QGjgJqEWm4lu4jvpBANTG%ApcYCJ_`VKyAs7E9)QfPZ7Gd6TU zDpS?)k};t6_jL?BGjwnupqp?H45ki^^&KAYy1G98c!z20mYh6jSp*N~-#jbEwJ zk%Lm(4aYu}>d%ahnJz$BkBkhD_VvHklKr0Y3(Ab9`u(<=7)g9g(~T(sa7_~}H-?Xl zjun9JfsygivB&$;504xi?H}kK8XoBP8vDKb#xrA?z60|5>|uRPr83VPP8~WlRDJx( z;2rJn%k&v{x=cK~Bmy12c*7x;;Xf=RGj+IcXe5>K{Y~}t_od~0NIlZ`^nhu?BNCcS z+rzDo?|$Ta`iGv`xpU9XU2VHsTHEAhXC+S4ceM@eY1Lf8msg+fRgD}(I5j?yNe$>p zE8}+ba4M~5YN`!!z3^eDD@9$Kc=1<7)jZbd>4=Z@X)6}J#6D8 zUH*iz$EEoPQoCi~KARpm=oexOJJ2^ikm?^FmV}{^r`!mqGKc$8hepQ^_hs_MYxm07 z5^3KLbU&Qhl{zpwAydm?EkCt$Pim+CUmh24QkaI#laj+Sa^ceo0t+*seoJj^P*ks_RQceiGKkOXn-5rqxNK)X9r_6tzHueSKUq zD|wr>@v>m+jYF`ns2{KYD1~Z%)7k>OZ1b<2Gkzci`CxxZeo>z|hV$Pr`Iv|Q$bY;1 zqU(0K{LHhh<(G+BQcq9RmFpDfO2%2UzT0k<|-bjKGR_gKK zI(VrOq%VoPq&&w>7-Afn^h>1FElQlvb{)sTd~NEleJ>%{A%b|JvSUlcc;4@SmOJ-e zjRaYqU*eBr5BL7he>wm2U(R;#XzpLXX@k{bVX9czw^sKq9`9sFLHn0%3M>A{EbqYkW$e|TTG`RL$)W1=w#<_kY#Ai$FIhS( z{tv5nU;ggU3 zaQM>K-WtBV?@iFGh&=rGBj-*od+b~UZD0Pu)5FbZ^YZ3bhAYw5$z?=yTjbKykDPm? z^pOOHNw?j&d)&infC;tgLJPJHI&VR!k; z-spS*`n)-OdC5i4+!3jCeim{bK6mNSlf#RXpBaw4{6pyV3i|if@Z!zyfM#u^^2lGD zi#&FcFqBB&T_}#1^`PFM<#V+{FVhvT-A;zX!yrbuB zQ%@4E*2;>tcmPhs!&JnZh>2PYD}E6%VX+Wy>d6aM*ImSSoU!8XqHZ4N5wgx8Zr?@jFpWUYbA9da117)ob@zA|JL}BiG8<(MvmX?N|5OA|pOK;@E+4)ET++ z=Le6yi`YK#T*-jT>zTvs_SBh!q3Xzgct`uc85($YpLaCZ&QjmAIZyt|W=HO@lLzNG zN56HOceHk<^ICJJ^IFsMwt*8r^^P`w!!jEAa>+o`Pb{M+zF9KRyu~&U`H^?D=@}bj zhpU^Pw2XSU4%tt94m5e-c>%a%pgmMFP@8_}j;2i1ok870sOtpHN7jK8-v)dTa6RCQ zfWHp74e(une+>A`fcF5t4e;}T{~GY80nZ2gEx<l^)jxDaJ`7@d|dw%*DPGm;W`V~S8$z%>r1#!!S#7u zQ*r$su1UE53$6*ceg{_@uHV8n3O$~|_1Czb#PtJQzl7_%xc(Kc@8J44uD`@JjO#CO z-H+=w*yHo)#}ZukiYX`3Nz|RBThwB_%+i_iK8I2xduil68vDTV*w82_%wDCgz z(LLYGKf3qH{G+9)kIrHPKf^p8ox%p9Ha76dF)57T|hDgQ^fze@L$P7$9g2V=e`R6WBRs&0P9am}-N$o^thD81)fj?vnm zgzsP#C+sIM_j}GCU%>wK%+G7T;kZUR)>@YwZTQ}@qm554JGv*l?C9RQWk=^iKc(X; z(v@_Cex&E?a_-=IehWH$gLFtf<^LDle~0dqPWhk5{rBkJeaim|?q8z&M6TyhQYii4 zZpUcw(~i;REskrX7dv}=!NDESvs&qt1DzH_r^V1|F?3q2*^G1{TahkgKjb=}%eX!d zp{+2rq*$0{*w~syy+NRH=FJb#YZE5%pulJ0-c7yerFOmHZKV#c3VeOcRbz^gQ z-N2o)E*Cs@s-t!wvl3T#VyHT2?#aJSvao@yL?-6p2b_FOc*$mcH5X&zofgEYwdzzQ zW1}7iJLR`-J>_v&aNU9{`efaVtL>DLe$%N+yQ05K(r=XXm2v1RPF1=TeYvFHAnDh~ zpf;H2=%#zV1SVvvLSg+x$fzOcm zg>mqEfS)e$d1LXbEm{2}Z{ut>s>z&FGP=M5b*gDzcvp381O43enTAmSAbtF@mu2HzXkm562Bu3en0SD;H3>Ty`M+@I?4Cx zID9__zCz;n#KC_a_)T%-UMwAOThd0jFWchqeFXdi5`QobKC=w`lLh}naqxM-KOphP z;^4Osf4bm*G7kP3;+Obu#KAvL{8I$~PvYSBmyEg}0NxxoVbohP#@0Vh^l|F{is+|~ zrT-Gq$EiO_^qFJnpCS4<_1_};X=CZXL-cX#|ApwMkEQ<}(Z{KOp6F+crGJU&_5Vrqv&Pb2B>Fh@mx+G%So*h#K2H5G(cdzb{(Yj4Q-77{=ZvNQsBFOf z0IxTU3vR>*?e}V}*++9MSwr;?>c0Ja&>mpb8mSNpn8o72~N#xS} z7h3dtH9Aj-9;g?9Xx-pR0&gtP?)xhd3FX1EW~K{w&Fzdll>158v4mx%TCffuTi-xtD(A9Ua2@ z?W6zlj{fX*@92delnlI%_17D(mkj(I>$0DJz;JIt9UHy)ZppyKcP#@Keqb5+1=es^ zW}*Ek8~yM4l7ZnziJy&L##--}Q%VOu`Y$&6Cf0#(%`Y9e_GQb!+gLNcQ&u|g;X5gj zpCxU~ef@LmBUfKw(9;Y3y`!UtAoG6jz}3AQBO||N10S|m!Y&n&p|9E^Lq(+nmu#g2 zADm&MBfh&L?;m2Lzv{FMyuWi(-zz`NgOqwj6q9C@eAigqk$(>m~Gss(yj2QDXA zMu#6I9W4Vxt1yp#&O3_vaNx=;z|+|15Z01Ii%|b{*dZ10KLP$H;8Fp99q=O9c?r=1 z{v6;{fQzgHm%dyUxirl(I`ktpdTD-T?xoLx=5?%z4%z6eho1|%81^NY%M&q&lW#xT zdL}}3*vp)}?%9$>#7vVDSMYzd^}B|>JjI7z9=lQcY;T0xC;GV-FJJFAx#PES?Y{3I z_AH1!?9DKTj`(bdKRe>j5sBJTUUfm1dud4|c|~a?<@Pcz_s?GPj;2BOPi)?S(}(7a zSV~HAU5Q@opKck+Nkm=3?YSq)vifsCbHX-zq=H{>8L6CGk{fw$cK?azX7yKszKQC; zdTXQ!JpYQ%iL#mfhpFz+jFH0}p3#4T!_!AjaCmzEiyWRdGK7AeSd^FR%3c?7qb`T) zm#mDqve!pkz%3#8xvBlG>+R zJF6_CIiMTbYaMW9Z{)fS)eHR4a*5kXxVvKf<&u99_z~-ze-+Ve(r9jn{M$(GBC6jU zxr{iSiMko6TaD`~TwU2)BAK`^#>bVtHIjkrB7R>LapJmw-`^dv<2s+;S4XV4B4$9p zEkf~Ah)>hP?UBj2=JWe|BB{9M@%tT-Nw_ZM_d6p=xRM>47S>QdM|1Hx|ASeP3!j}C zdE>~8$i**AkG%JH(;~+YeR|}{gx=h*q<$vyli=k#b>Np@%Zj}9 z%{h^G{^^#;e?K=n^8SBhMm|7Zk04J+w^;_RewB^>y38{0;ol)ok@rK3Fm5E_T15F< zGCK4H%HPs~q4|I@9t|A+J_~p!U>jiUfrst~To3qN1|Q-2OI)#69_mE> z0l@zS_)ft47-G4qf9MAcaf<76xE`t+8S$k0xLtNVw(OmOwc&{l%C z^vMn7 zKV%d8;HGn8TtKelTyZ^ee;wwaGK>eM7#B(~K0j|5lWvX|H^+;cYaP{JofX`%>MqMk@f@*4A*}3QgpQJ|?})fb zdstkp`uhRk2(rAk_O8afw*7fs`f75oyZmik`JaIBt@n4dHU{cH5pIUtcwfxE?$FZS*2_6HgQ{^k~67v?%&eS3?qt`VzOUqd72zxMjZ z1|OE%zCdGFH?{-~zPcVCuV;GU1ve(V8;U@rzYB8EP4z z;>MRwfBbtdHL+}KZEVeJ-?OK)u`8eXd~N%YA!7NzZS(rgK7m2f`CFR(opA_M`gQmN zUCmw1?QMKjqRAlXSeJdG)|wN|xVb>ua0*~9Yk0lWT+RNJ*H0Q#{|{D@lRBpU1=L#^ z=*t8uUs7b-M18)jKP&5#p^A;4Xc$G^&g z+f}}z>i-w*y(Zf;#{gC1#0xK%GjZ1Cd&t0qS&^~(ZL8Ho!I(val-_tBQAw6$aSCLQIe6Nka5n?K~$f zC2&9$nC`}PR+L#c;mF-*v)BugS(O(r$KW%Fk3Dxb9}j6vj92VA7*UasRZuV=3Eqd# z6(oP{ow!FE){P-%UvtHB7K2!nlo-K{#l_A`IA6k7Bqqi>O9W9*ew;3cpC_$O1?(SR$5L`4T5 zPh~GIXMMp*E{db*`DiM8Bb8Zuvw2fk^166?!PQZ?m02@k*b7+T&8K4{V z^GJUOj|)zM{)U4{E)R!z|HHj$LO-$y@B=++*fGJTqUcxjm>S0P;J?u5Xl*oE~g$eeU;3^YLiLU3T zfl&|FnqUebJ-$Q24UC=XiKa1vxxbdP;vmZw;NuD8zw+EJX&}VIo2hJhkf%5500Te0 zN22NS8t5P1Br8Pxgo6nx9(a6G{CFh=@fUR2`GCQL7O*wQ!-4TF9pe3Hz?nnw&GP~I z@_M&}ufHMG#j;b74{{gD^C63scW0ohR6T0%hmWM#}l*s^6q3K|7Vo)g8vBYBWtYjY$r ze*ZI-ZGKqs3q1N~UyiB{FZB=lch|(?iI0cV4dda_pua4Ps);_$FkXgxC!6r; zPMSQ`_!{n=V!~%SLk9e`7#{K4-_54;ePf9HF`2{YE(IB22Z#B9%HaeK^P>WS-$>*z zACNhm#9=J2B)`oVZ{Mf+COKx`C$}KJakf)R3@8_lz+Nd3%nOgNqk(C;IRK+BvM_Zyx_eREnq&S@e8YHF{co|H&X>9r|%0+ zcF}Yo+p|0soHmUl#N{UgDHvFIA^Fz6V7fa5Q1!nrI34wxexRR$cYri{z%v>5hwOh}aMm;)zg!=R$6H)! zvHk&`?VDvc>j++vO}JWBP_Y3~_QhgY^C#)Q*wjA{;zqCUTy~2= z-+3l|=TAfU#`Fmm804e<6s$0`_F;c4pkns#3odZcbR6qXzDb{KY!LMFiwyF^y@e+F zWiFbYl|1Tyn%2Gr_E`%5#qi(x%K%-`tt_aMAKb z^q=w(@rQs=_6YaRGU+?htskFA-)v1j#UJ8rNbw^-dT(JfxVlQ;IVOFxrqOt>+9Q2q z_D>~!^BF0q>FaT^%St}kV=h~!liwGdKaJK0I(f;OKQOB>nqF$$Ty`Q*#rK2Q^K@fPs}jPZ+V#iwF3Ve<~zX4 z^mwe-%=6#V2kf!&m#_CRy0l1%;D0JOC5O5b8{cSSzzdHb+OO1V{wMkr(|DU|7(XfA z%WTu2j9~qzF-l+!qaPKs}lmv<7pFc$(&tC{S?FdiF zpM~+*ZrA5eUvRaHy{&KWwkhPFSbJ+s?cJ(2AhNyNUF^7?fB6)Be1?1PF!A4M(D#Yp z+FAPj8qX&j(8SuGVd9@@Y9IM?I!50YTs1`>&y-Jk`_gzWpQdpAp9s#GZ)_j=G8~gf z`C{asXJ{Y!=H>?@n*C`!)$=bf@h>v9zi_#}edLofw*H`ezw({VHi)a{&vy^Q{j&VGcpGC z^uZ+N64JQ*F9cHT&{WGWtnU&$I+)ty^Vx@T{Yd*+gFbG!7x5tRIPXZ&k7qdVNTL3! z{5=-T#BwYeb&d#mhW*itsq7axKe2nk!1Kq#;gkvmJ2;$PWrQvirQ zTK|xL-bjHzzV8zA9sF_m$e8)Upbw9CyVDEtLjNqb9`hH57aTw#KHkH8^l&WzDk5I` zf;cqvD*CO+7YFgh{yY{;!SJbvGu?GvqQpIw-M|XbGV{+P`u)d4F0tk_fznf*?%Q! zy+QuE>=W@o>q&|SoTrp`Z_>rX*z*UoJcIvno)Xvpp_qTU|8?to@_!Dr|2X}xj|Uo0 z{=fSl_CZnpbY>iuu=0POgq8olC}H>;@$ftNKTG(Z*6*7CUvuSh@iZR%p8P);^Dp{x)933G!C9PsmPS7{VKw+w{2U7= zChF;r1!o%8Yfl7caQ+#RAM$4z>8DPGG;KW|4$g|L5BUB=K3=3Qa(D`E4vQ`W#(L4P zo)2&DFL9$;8M9}CV& z)UQ{=!7P*fWzH(FDtm>4SnjIA`;){x>@caoa)@q|R#g=v~2}AQi4pxVJc_ANCvIU*^G<5%M@c#cRwySz^DWKko_V8uaJSJC5te z*8*n;8jH2R($xN{So?4A_VebYsP-+`UoSDpSNW^__`IVLv+0Uu|k{ zO{_g0UwB(hw_is8moH48F8lAm{$Gp-iLv}N1b23**`UJ9H!D9DHo&AjYkLQ=p z5VRBaAbZYp9#*ihf8orO)so+Wc$sa;AIh&BMSlV3&%qgncnstGT_11Z-dr7ggyPQ- zpNLmOJo5Fh?mUh1S08VP2i^EV=hHcCA@o6!@$E6>6ORYodEZ$7&3E>JITjy_4f4Xh zOLY3v_>%4XqK=;YE#g_^OSpHbiGP`aKb}AJ;~)H&F*ALk=6}HRW@qMT{xy&1z|WuK zUM%s6eEf^;AIQH8oF|mN!oS&`l%<+~>3NIZUlbq56+Qfm{MY%5{Acu+(H}Iv7~_SW zXHmSMkbf-Pzq<3uzTiUVX&__vU#g?ef;_YT+z z!XL{14(MZ?e}P{xJ5$-8=ikCPZe`y@(3|sbzVk($K9o=S8k|M>JI|Anr}>BC?W~5U z_{(q(YB2dnpATmHlfMl4W*O^0qkXx54ClAT{33sjpKlazc;2i#Pp0^LP1BdgLt}oJ z{blw~e1917W4E$%#quEu`pGUQ4#Ma}~zYOPj4Syr{&(`Hn z>KOZndl%^Q56`O?I?1AwoTf`sM z+Z2zQeFcp8mhq1DH}Lag_A{NgVZ1S)j~UO~Oy?GAWv_hRoi;k?JJ|ARfY*m*zb zUkW(r@fix%d2eK~8A0(r(rSAD5a-qDc>jrxgK^%Tz+Osa$-T@*;~}1BCeiy)Z{vL+ zOn@kOZiDBG=JpKmIXo|TIK-*dequ(C%*Mm027h{Rfi(-as=e&sg4G_t3#_7j;1}F! z#4q$JJl-b+{enKg*^<6EXk8G3c^JLFh4o5iaDi>R6Cf4zFEg0UVJpGFTLU)n{G1+| zT}X9V1B*;>ZVblzVvwJmM!Z74y*Riq+hc%pvh#JYHLxVc5BbHx#o1mR{+~a?vMC;- z$#|a#=YiJ1G86sM>?$3-HL%=(2fvl#L-7{}^RjCV^aXl)ydRjK-4Vkx@Gqq2ahxCL zI)HPie>!+E9*F4Sr5GUea2`F+($VL86pa3q_nb$lQW5*fxbaLcEK)op9>%;c)_uhg z--ypxywG~|KBQU;{7psxKi>b!A@8erMf^s!_ys+GA9f!I=&^TQ3cc^gjPfJ+k-tz@$~`dPeweE!XE6wcQ-<_tsK6PLhl1$_d)rN z_j4~~*$JlD>!^F*M(sV=lVf9jl0S#|@eQ{=$zPYqT!E$H`5(P?jP;Exu*@Nz_YjP; z??9e25AskXygY&4e~ZDz_IwV8fIkBI0!M)c6aN}#p#i=%0dL5l5Pd3}8(13qo*9Ro zQQ1DvpSgj&sBB-t%cHV=2`^n*%;n)9!S599D|k)b8Uy~;<+ti!S0Kkl{zHL27*8S+ zqIjbBL+`UkSuHF~+hF)VJD9-z$=}~Zdzi27l>X}d;I1Y>+$5VR+dKka<{$zl)nFzzqsRJ@4)##~lD zpwk!c9~$+Ya|8N1`Kzkq>$~)Z^j$iEzE14$g}&IU2OO?!^hfyLXUz`UWA=nS<_2~v{60g+<-28o3#h)I$&0Y(D+dq!K9_+#E zy2oXHTtHq1audb#4T=Y9Z$aV$E=;sX`9<@S$hTBG&&Sh}zYhE3`&_YjLH-jx#Rtzv z$Rn60$d0-X`TO<{_N=h6(^CI=IOhVM-Y4j&8{zNw%nf8mr9M2qtyur?`H|v>$9pbb zu#yEZ`g@xePaMA>Tj&oOz$BmMgO0kZQvSkRVNZcyJI{?5GGH%?-|q`~6puONU+}vE z*-1-)LXq}al(dM0!ai#YlcLlf=Z8NrKf2TQW4+e`rKi{VBhs%Llx_6u!@G@JDtqHx_T;ADh2x zqBLH_@*y|3mS8P^7su?+^EWqFvp3J*++4EPVl5xsxun0F=kIdLU-ueTEAzpo^9SMq z>otS^JfHRY=OLfX`rA$8&#h4!-{R^o_0#l4|E;}N9uJJ-3FA$AQ0ULcBR>9&)&Ew! ztSR!@#vaOLI1e5aFy>=dz@AL;g@X7v0@yK$$Ey}EPQ0+J@Tu&fHO#e;U6K3=+#Wd4 zfIn$I6Zt~%sPe}Zuuc;51I61`_E06m`0cT9ePle)akI)FS0E{w_Wz8}H_M^VBeyrQ2DlYp3q0-&)Pyql;m+3^bhS~7(c6Ie{IMg-S{oqk4pJmKUZLRs+cdyUOXQi zv3$(JdLY@x2IcsL`3?EyjO7>kuP|NA-&|iSU=_@^@~aFe>G7{rLDhsMaHRcRMgSeI;2(t`A&+qA6m%z}tuX{4}xNP;gQj?MGwu zX;W6|;JJaxg_Ajqf4F^$qEa4*DSj5>8;<(;!Olg+k9}^SAgW(atgW6LC3(spa|1<1 zG*t3J{y{u1z#A`mdE|c?zp!uaWU>bz)44wRc;QqQzz_D|ySHbhKb}mX`K*BApBGMV zwHntyN%{KtqkPO}Aw&DLbd>pocrlLed_8zp+W&%>uV!(2F+b6GPV!YelD*vczOCqA z63d-);Sp z!DIHDJVWLS+S|(4E7X?suiX$2(tbtwzPI8h`_X<*#hWXTHeKci*_+0P8GJ3k?RNzF zWBqkrt)D2~h5xC0B40(mDL6AYmB$l5qviHW<6TjBjQ7*|SflyJYQ^{0MFG7lkUo>* z6@~jB`88ReU#`HESrlIqZ>RV|Thcxb=$DkH_s`lf@tPi$^11%#|12?INc-#akLy1L zKeHnX;LPB3D~%7+sR}UVO9sCGueUGtUCoDNZ+bqZ^1&6DI+L%Bc>lLT-b@}Z{23N+ z&ppL3ULpRcxY+oSG+X2g*((e2VxKMZ3-EGA{+NNk=b-qJkGGiC7jyLM3$hQ{ zGM2x|w}|mX_yg;KIU-*L{F|-^#`@p99&k9c{O9&Jt_L!4-a!85hi2bTwX7{Q&;Ppd zn(POE*_g*J`k#RHgIKTdB*1!fZJlvFK=yLu8~-tTC1mN>A7mdoU(t_u^nDr?U*s=4 zVoC6e@o|p9Uum9Les87udv0v}5b^6^rzO9Q@^79#|B!FHeu6w}mKYzEKW!cwf0!I!Tmd#$#IM5J=Zo=x;4I{Gf?>ThH((x59P{{C z%jMDf#W)_#4H)x%Zort&$RECc=PGgk+c3e)!eP8wp$dl`>BjlM&c6@9Nq{%R8}e~& zp)E@CH2;^1*PFHR9r1$s%_02fh!Zcmc!PY37aQ{m`XL9;S9l`^V?1)O<1rZi!4IAZ z_zj2req|{%!kG!=A-+~q{_^ju(0Md1?s<{tbwNAx${M17Q2+jT-4=ZR*Ae^vFV2(c z`@Vesl<#vJ&V$bqzb(pNpqKBTrz5`Q``tKSS&s8)Cf4`zJbFgJ%FWFWL1=#zu-W)} zoA!Gz;rliL#^H`?oTo!SEB{x5?57!UqA z;1K&&mcq}QzTZi^Wm#HM*fF{Pi5B7g@|S5pXlH!9#Z*b>QQz-Oki#!{o=Rb__e2Gr zdUPRyf4`bbBzla`I1l9p<=_8Be$oDtk2i#WZ7X{$F&ecZLGf2F+zdaGK(Gz=;V|Tt z_lWFO{kI0KthNMgQw21`@`L>E50v+){s?;!J^d|{@#qcl0(%;tpBxeIyE4pg z+WSr;y7S>Xq2FulW$f?qeQ)0W*!Si2^ij~~v$JOUw29=O7)x(>|5h)b^qI~6%`D$D zk^H`~^oIBQ%<}E*HM9KMiR1?j^q=aXg%*{3Mt?7YzMoDI|9(2+tEy*(^;s_eIc`rg zJ>?rOh z)`9)X;8HBYhR$Fu_2N69`NqJ;52CiW?*Hh_ooBxJi#rmsu0=nLp84PadnG!RjYc2v zj7GhEAKve&_~_2G>}ZzjSoEzAf3nZpRsR=mefEK_U%mRm(e%@& zu=cw4%b!NyiAH~V=33_m(O*aJf9IWlIuO18gXp!-ocVS1zy8m)=ugjo5Z!y*nQyiJ z8k+%2lHEPRSA|%pL$4V80JC7XV@YzluX4P_z0?R>Xg6d@N&%ksL5D22g1;nXj@z2- zPPxh-(jT&0M#2{LML#EC0gBv`$HD5WiU#u zs?XgP>#R>Zud)(S%IaolUth3Xo(P|R9Xx#?UgPl&9KhWdjUIsH;#gN5AY9p z$&kfX+n7;c{>+6}@$~v`;<0+IENr#fbjfPf;~icH3p>IFyrF&k;U+|ZI?&=-s^_;O zK*IL%<%|B0FJJU;0{J15Kd9vMQ9>_2$(zK&N#o0pV;@T$Lng%v z&$v3PTNjIb#?i%@#WG^mW}C6IiOqmlvxzB=t<(>djC{13KbyxXKN_}qZOmW=gsvVh z<4iy081OQV$Hk|3@n<_z(1}7BO2+)FO>YA8(5^Mt!_3(x~E@1uA9M*Dl5X+vEK%5ybW76s$MxrUy2WkVo2cm)x9w1( zrkk#9L*JC|!aChljG1jWo^EmNHh3Mov6yagbw}TnZaUq^jG#)_@pQBPHa^waEv|0p zTO8fSj@%-nmtu|n=T@4Byf*Av`Mh?Xb~18aL*!!Zp~m|rj09unp=!)cdoU}gCv8Y= zKF7%SEGX5r9sL7|v}5PHp*0!q5`5_1@L6pau(}eS?&ExDU(L0RP}(BN4BGO2*}N2d zzpIpEy_lr+B_CjH<#x!j2GgxI&6**`0E^2qLMfp}1i@+L@~Q#YEM|Em-nu#k`#29@ zm|1BHhjoSaJ`lKm3x0G;7Ca);@mu4zujzLvK3=C^7OS349oDDcybcx+>2xgzi$BOU=Liw8S6$jAGUpNB@}oP<;7XRCQS6>F3q-KQ8d5_GA6?3cSgFA*CMuj0ovx?jId0@q(}Z#L-{d z0nyK{5Z=^2wy~-RezScwd~IC$cs4IfpTL{-)9`pjB3}C%9((n8`YZ8xttlRVNV7kF zxP~XRk&}?cFXRx~GEoka|PE!>_7! zNO`LLDy=noW9s+c-gzniitMjf*B(0lJNy>enBN)O?O~7n?YyV{;cqiq**`9R_k$YS z-!ED^H~(Mfyl^Bnxw6H*9PQowEi!KZo4-ZYZ}_dQo4-XC`mKM9%-g_N2%ppV+{kZ{ zg)j}#?_2iKp%*q6Rdr|@pYx5Fx;BRRZ;{a&p1&}LahHCJtV+W967G;N%^QU8lkjQ@ zpO!Gq8-zbAVXuU*NSNjg!lz*pr1mN#j3pO`Y2M)NOSnqH9TKK_gSRi?8VR44FwGmh zeF-;7_=!tN!YLJB<2Au*W0t)E|Ot=L?whC;0fnn7#y`&lND~$N85DnDpWN zB?2aW=(oq%1_6^k9Nr^frsyTC^cj?}(uXE3(qHKlk}%ni>my;Mj~BB6(JOt_Z;|2I z1?#|z;`w+v{5ZZKaNY|d=$emDs19+07#Z)k$Uwgqh=C2fZwRo1)jQx$|kL9;Onfd8t=^+el8l${JI+E3ee3EPQ;{ z!BkD-74egNL#Y*zyl5!WRbQThJgO_zSJHj0sNg>*C2r&T4@U^}dp6|1G)cxUG|y7m zA;0X1DUFgUwNFEmnZFnn_&ypf8x8h9F8Mt|jP|eD{@cL+=W+T!DEXED5T@M! z=JsKJZqvRaS`&sFwSG`n#ZP*WFMb<7zDiK}X=n|=I$0sF`u0dyl^2p~%%xTah@OvE z)s7+M6)L~fRq^dD*|wv6Z+(dw4u1Vy`=KM)|vtxZeLeNUF$aRRc|WU zwey~PI<}R!Y`E*5J-hF2zqh1m&+hUat-ES=bT(|{yyZ28JG$yCs*77TZbjQ&dv@(C z?yf7UZfYoR*qb!KV76&E30-cI5BgR-E5hQ;IekLBIQ+-8+j9))n1F z?G@~5-MjwYk}XB`h5mc|C9PV!{*r?Bb%lHH_LpvvG#gg5l<(cx)KR^yrK0ejyUO=& z@t3S=Uthj=d&AC+?WL6m)-_gaX}u0jY2)td_Oe}m^s#zPOIcIx*_Y z`%AWME-h->(@?mkXLrTc>*%vP-@mh#@Gd*I#@&QrK$V~nS) z96W`)J9q6Imv4o%XT`3Xje8J3-G0POJezH9lX|pm$**tS(Nq`Lwq=_)ly526UB141 zS8Gf6?s9w(7rQGqb~Lo^=s`?Y-d(WcU_-^mmR&mwbuqDHpMPiZ9)Db!F@KfqtnR35 zS(CqO=f=+6*Q0IawC(<#)%$kuTzMV;iuNgA;`LpN7t#Lp;%+1LyQXOO{#}I|n&ZYm z`Q9FB_YIu3u<<(jHE{dD?&JAi<%MW_VxOzFE9zU<9N4{M&As*4>v!A6{q^NLuIr!D zB0;ySZQFJFv%9Ab<5^8n`E}@8BwgwLy2768&~`}LE%)x)8EhHH?wXxs-(lB+>)CXD zeFbu3oOs)+^0THCxz-XKr(Kn28%UOaXLtO*tZS0v-|hqTEo%Y|7^mZolV-Z{#w8 z(RJ)x&h1>Y;l}88-l)9N@nS{!SGHSyLv*_1N zWBSz>URS?Lo@?bB+IH8B7poifInq);-rT(o>!$qOi1`iqn-9wM2gaqQ-C|8~Zz;`p zJMtT9Hg?w(Zs@LS#k^93IinDB#k!3LcQvoLL#^6;tu4&~-@Y})b$j;YuP#_w=Pzt1 z_UHRo6|Y&*xOz?g%9Zu2iu3(GL08|r$6w#*tMBe=@ppFmT8n(GLEqlS*8RTijcuLn zfxWGbt?kV#R}|=B127@dKxqPs3VkaJR;=(96c?@Z;oi4B(A_Bh+2C*KZ1jQNRFm%$ z{}uRF`-)_T@0+);G0@i760Z*uBf7HL-_d!X`QX8pMh)I3;Ld`=)pfoiUqe?e_?8hUQih z?4yrHY0|)#Kkru0eV_RZ`cjZrz>e#G1E~wn{$zF|EA-AR&C5am2%!P9zW=bJ6X}HD$uq>@EL=X3bk(|z{Yo`f=>N=n2QEeoD%QcTF~>I z>KG22u3d!`^f7&m^*CfO6+urYQpWNqz;vB`RKHfsdX+~iF8&?;QgwPspF%y!@ybe- zKlJ00@#vdj=y>(?Fp`P?PkM^mk{JdoxeTHx#LiBXLMMdT3+HvR` zKod_-s*6|u5#*5kK8aNMi?bA7Uqy#CD2lfb_e5VS>(_BC&ex=z)3RRq<(%rTtXFw@ zLG~+6p3kcM9FcOo^1M#fKZ^Rv{DfLG_^=F_KFFu~J_%oz`kcX)>O=A+wC830Te7}F zj&pWdulh$nQmkT)P#F?G)l{D^>t~`~@lw^NWc@7E%P#YpUR}NHGOsxy>#1GseL2*r z^LlDmU%wr8phEJMZUj@DQ!&eFlJ#>;ayoQ!w7+8p{yts3`1?*=EIXmA*V8{cPCfou z5-+5uzQ6dZzPyPPQvS--?ozextY+5R=M-faI*$$HYC=5|#oWRg%Gsk3Rqwa9vsFHOQ(=-D?Fb3XRT zdXlft$3a~^om;E&o~~XFH5`>J0g|KF=XPmabG_<|9rTn6s%@I5sHl1`hZt*;S5<#r z=Fb;R^=iC-%~Y?(=`^%QMafa)alWZujmPTzThX7F{qHc*tMT}Rsa}o8Of0%PeRa)k z{y-1Iy1c=^$=}^p-*gvWsIL#=Z!xpHKx2zPud6ZG#cp)RLV|qFZOvVb-!(M*TN>-T zeAu@%2D*Hm-Szd2ot-SN4*&c{BHOxKg|f|!o$UG>C%LI;Zrjs-1JwH)1D(z7ZPE$MQP6 z0$u((TzgviHPGIGT^-Bg|B4_M-rIy8(uNNq%t`XBRy0QBmeLA@3%4U$R(Fi&~pO(bGgn@NI4xSkkL5<)#=v?g!IxsC{6ms2w*iY zawPs)gbyvry!hzLYyT`D{>L$KOUrbcBGno|t;H^*lG>Nk816(lE!#D>r(kuZ_o#@U zWb4bIZ3-JqqY;#dAVgNDs*6U+4P(7#Ok^A)59 Keb36w|9=5nRY)fQ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k967c39595b6f81d02c06645d911c9961.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k967c39595b6f81d02c06645d911c9961.o new file mode 100644 index 0000000000000000000000000000000000000000..b558f561b4b1d4a4d5c6bdf90288919dab1bb936 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Kp1A?+13MuWI#e75Nu{eOKOlnk3KEQ zvlz_T1m#dLEyD#XXu%RJ3%xw(AF> z-;HBeKisH?KB%;XuG!e(|7&A?M?2VI&>rc@V1?hy_Vj;P!ruCo&JNE=?WsO%o7^{K zn@D{f=}BjW|3~-q52Kx~+1{$d*dwQ&9;`TjU~tEKPYx%zQXB zvg5$uQ%^rVm^yz+Xim#_!WZTp5AS>Lr{SlZZ-$#sTnhKCdJlAkq0qsDgWFFW7~D7i zc(~&9iEwk<>2TkHH^UqcgBRvMF}QCH`q}@T@SIkRf9lO}`T0xX z?eEcee;NwS$9QWFqMsAtxvk$0j{skB{#1C*$@Acy`uUU4$o_+aPtAXNFm&of_yRu7 z@4XqGd-7zs-1%Pk!u<1~Sr(ep`ov%#bWwi(IQl*j4xKm^u5g|Xhvr`jKedK*lN zlK&~+8-VY<0^hvndv~_)_1wVsj*j7bM=icbKB@d~z~X<(UvK1t*LM9d6e*ef*6;)R z;S2vOd4 zLT5Mbd4%Q~cs_h>*Q=(vB-VE!&qeT0vJ8V)8uML}2{K-Sj84czvL#PqSARlt74jM7 zV#(~92l8C_yXE~Okgr_IH3FVT!1D-rCK*Y-i{Nt{`u9R6=ih6gcOl2fD+!?y$nh@Z zFv?FdjJ%Q<8i72}MgN5f?C>S%WaOo!5Yu|%%H?35t8*}Zvc~#p-T1ijharOf7BOqg z*ax0SS51kyqW#B}qxEI%kKHVNt|vs}6a5U$)6d69i9g1*wCPtquGF}XxiURhQ|5U> zsf&w3&My>)$TnmXvJ>_94)l5s`g0+E29iDfJJA0(ab1f~1@3!rJ%H;8eEt{iPviP; zG(PP0&a(+28|X>IiZqEKkan*OQL=Ria62Q?EqFm6?)X^E}^w#4}UAtRFonPxZn%WwJY(ZOlr*A>q zV+%TWG`DSSkJTHxWc{%P(VK?$R=>BCE$H+GI~NGi>KeK_Tf7||b>5biZQjl%U!bnR z+tFDkv7NrQj`o1RGtj)KU|XxN)!Wj%tu22^dw$)&F9rkD%$)Eh?zP ze+7&49?oB~w5~1>atgWb0x>$8ya8Y1-6KfA*Wn9128m#$fUo7Q@HWWdy()S+dA{^0{3n3`dfX0 zhP%SdFb$W*YD*53OD8v=5pWmSP>EyqSq{Bxrm(Ut7wCEhmO;h17OQq0B9D-n$c-=e z(vW*;$eorJ#&P|7X~^d+dp9jxWF0Hu=F^Zne;=b3>2=s(Eok4mwZqq$$Li|Z9;=ID zRxe*&Copw+*s3*qJ9zYt;;AL_m%Az_yHRAC1EdYRWouc+;fYo-`*RK-KMG!p&AN^6 z*v+UXIUFa2mhvLt8Ln7AJvoA)9<0p3tw+*>jF(-2yN{+t;S$WgF6q_KivGKR+i-qi zEfJJl38Z@};CnY+(^B{J^vq_C$ey0r4t~oR!RJ{!k#)th{p$dTw)Qi02trkSj^IP} zH51y=4>2#sF#oO83-sbLXZ*S?C=f=$T zc{_aW{6z~E&vS2}%YwpXthlSWrP1Bd-O=f5b+>kPbh;aTTbtW_?)8Fntw{nd)qseNHO3DcO~ZeKy92QC!KeRR(S=z}vJ`=12D~ zIARq(b{A-1F#OLy_UXdQY+37>J;lMuyL)W7Un1cF?zN!*1JoKuwMa51{z9ZS=~Zke3Ili+3}r;5q& zf7N9h`$abE-EWVu%K#A5jZU^_U!t9lyD2-B9e1-Ibqg8Tlfa|D-XJED1kw1*yLGmO zc+q=of}i_PL79oqIVU?Du-nmsMFtM0v7v=5H)y-h^P9;mq906RuODKwf(f?YPI+So zuMcJ03xm3i@Qe{2@&zYGvQbNz3~gh)S-~VdN`Eqqr3WS^Or-t?4<)g4M9(;V643*n z8?;ZPvrf@J(er60=+Qs)VIy(;RUdHvb0MFB9{i+9dx0PHNAz5O8J!yOgSt-g`K!V< z;x{Qc30CCxO;C72A7+Gi1)P(J-_)~_BsQ4F3jYHAqt4}hgZJl5L(N~0{?PxE`&fkR zVRyiuLzV3QT|`R-@?A<}U+Eqn;pQNGt`qvv$8V^qsp0azkqvn#>*Pxj2lRvfbAyv7 zlfRAU`WJO}<|SR}r!4F1(!8)I`iDK7+{wuv;HNOaCQjnxi+b;_gb4cp0P+F&Uq-+d zA&-nK68Poa8oNvkXu>~M2J|$lGVNo|k1HU2-S%tf1O4j(8*>t#ltA_k&g8>K2EhMN znmv{L84IO<3-kkgMlg3Kt()*K-aa=7a}3vLlS#h8Y*zOC5n!oc{0l_?;LIzLh!&%t z<0Sn>q2#iX~5?R{IF<04RwM4D)5=Wdjx)V8t*?7 zb%9UT_H?^|$0ADOHwnB8b%9UNg*;m%K3np?MdCH!iGRPq*Dr*BJ@_EW$NiuDD`>NE zUsC=SoHK{SiQ#9>iq>D_^M%`k&p)&$ec91J=jUykUY7M7qjfbB*~{&p3HwG1=!;aH z><$0Ge9q_CD|>#aslVP`88Lm=w^{T}^V^_rK3_AqzB8o$;Qtv>eb;AWy>MCd?Q+HR zT}k?K#pXAkA80T2js7!&&=v%U4D^cim*xjIpww5ESzhWdx*iM_v9he$Ci%wFSMM(5 z1832M)~}pDgZwswzmfb7i@(`<{Irw&r)WL1i~NDshk9BsMSB(BW(FOc-VyUR2iFJr z+c`}i&7bgAz8+)#t%UtV5%bR)56Qk~h(NC2g~9RT$^LwN#EY`zTwYW0pm!IReigs` zB%dCWFUN^^DZVSckMo&Gr`i8V|2AWMwjbY#d_&1sPx+3+`KxOmP89hL*)Ju+&S60Azhs{q z`zQRH=Zg-RUy?k$sQe>fbMW=~zpMNy+4eS;Gc_L8GbgqRBA@2#S+b3y4fhYoOL)HC zW%wD%Ti6RcA3*p|`fgh?&yO8EKep!v9m%u*DU$6Ze?oi6&&T8Qh1Xr=d?H_Yy%O?? z^&~~mYiJL3Ybu{f%m^l<4u%HEKdHYI3>U@QGlJt#H{xmf8u1CJ8}W%qw~Y8C)MI$d zddAl`yGzECG5x!eVZG-5;on006ZpqInw(UWe_(!|MV0Ie`y=1aBzF|?knBGtW`Fp< zd44YB^xU2_C!o*5;540T+|m9Ze|liLJ_Ym<7v}SDWomjLTc1kx$(TSGALHd@ zq0aZ4=nwYIBzN)Ce1N=#!OYBPJhSpc{P$wK>+?hF7XmsF2>&+w`yrZ7#`%Et$K>z8 zo8|-XCVvOsFdv|w+@RV2;qTMQzC7OnzZpTpe1N=!kNW>5jq)k@7YfD47x5!R98lcem2=-rxy0O%XqSO8y`(ts4N&CY$xWAKjBK~5z zo+U4qKfG+z+-p9Z-zFg`x~s~7xe#~=8Jcf#uljPpZ)*D+sq zJ8)EhV#9iI3ghn<_VCxgKaTy9;_Ykvd{61;mgT{Iov$C{Z+v||W3s0;KQrYcu-}0I zLxLjhXFboj4F0-~J18E4zj1OuX3BRcpYfBxlz&kEp8O0G783fpoqQ(mQ_W}moO`8T zp3g-6FP6{n^NDk5_8aCeh;P@;U;Z5ahxqnc=P#r46-V;RUZkHQAHjTcImjRR`3n5g zoUbed-nc%~dS%U54C^oAMHbd;EGHt~Lw~0A9P6`b{YE_t>vfbK?Xe!k_FF@mdA-Jb zC46+f9y4Fz^V^cI&}N*9@JGI%Q1*vCasHS|r<)NM=KJYSF<&8nh?}p>7$aYC&$w~E zlEw2Ccl>+>@?^OwzDM(wpdsHFov$FCy0fDB$}sfz$@7scT3>m*M1Pii1o6_8k2o;i z3^(}w1o=poSnw%6kIIkk+*X=y z)lpNtGMul&e{LsUMfcxW-^}~%h4_A9to`g@#$6L*`|}6+{+yqm@_hS^g`l5o*pDB=_YL;Rd_T_p<3G`UJa*oL@vwi2 zonK-7AH5%+I_<{sZ3_4Isqy0*;>VP!eE+5PHH@*ICY98zQxXetovKc-<#NP zPcg=~!r%m*;R`IWKcxAQIbk9!B=+Cg*nekDpzr4l`|oMAUo^x!Cv8Y#`){XnjQuwb zRH)oeysNy8{dYQ6OE4(*t7z{+J-Q!7zBa*l-UNJx@jQzBKa=uJvH#@rAv3!F( zCzNQsn>?AhQKx+M{6gr{r5p4K`={#`eVXFk>U3N zX>2teHt}M}XK6{0`|12n7NK~^|6u*W_r?6YSDa@U&cly_=qAp?kKsHVls$>><5;CB z&9c8v)3M)YUjqU6XY4<3$REBBhd+q(^(^}Slz$(o(--E6R0qGSE&FZ0UyI7;WF#Ly zUnC*L`f4W!;q8ahsek&qMfRt~&j0gnPG0l*K^pra_!0YAbb#}pvhI&!`3b&1Fy$wo zK^;=vEB7>TPXqTfa8CpGH1MfwfDX?QY6tKu2iEh8Yk&CgN0G0-)K~Sp&42%oKd%_~ zH;o@%_}j>#LN>easa{u&yKipJvA#aW`t3E(j=cXbb&;=eRLa_fzwXQF?dwZnXMX(3 zAATQs+s>BPtetU8Ut5!~wg$Tqg!G@zrnsI@KOEIG`J+W%M>Y1~9N@1IKnbdiv!`6) z5xR&6>R4UFzgY(FyH>V&vfu^OXrbUV#uZi!NYSVAw2N%);=3KGO#EfFk7OFVp3v99>-DVOKP z9UqV6UlcG9uYaEF6NM)}UgTQ9LoxB%NBQ3p)4j4mJo>0Se#X{nTCxsJM`0X7_M{?u zHt00TWAO)0#cfG8**nUX5Seki8T76v3B5<{qjJ0A|zRuv;Z0#;P7r|3D3#NYFk zLL22>G`3n%tSd&0r|W4@^&5W$inZUXTV-|`#*Js&sAqyM9$s02`WY>?VdPQTL<6I? z6XS8Fr&c?!@XfC+NxRa^7#?=Q3k`>Yc!7V^q2U1I%c_oFw3zBoqOO=EdJL*l9Yz!J z2X(sTOCRuH!$kkKrM^@%<2{yom8DMOQU832K>xMkM4Lc-{2SKEBW|_LC#5e+Ew$fj zAMx7U4DV<7iaOS4H`CKteOU4F`uyy;v2r(7-8d6_Wenn8Cfthr{DyuNuf3WiG_}TT zZW)W+b&9Zj! zUke-7E;Iexmi9}LH5>c$Sn5@lIJk1e`31=%Hd?MZq*Cw)awk? zsVmYojhdtCv3-V3r%{Sh*V~I+kEPt)Xq1bLrJH!U3UPWXxqut&8g#59lmQYeq!H5b zs4O;4u4nRIOP)vl7_BcxFpSKsBfCLjz~77Y&-h_Mm&_B*xjkmhr>ioa`Xz*!_Zsj&!pSRDUwf8@E z-854ZN1qm2`H4qw*$xgL#q7Pz81>YweM{HZlwz;V_}RC1 z1>Q%8;n;1fGUf>UyAqH14vUUtE$8UM$4&SF@%uQj{g3rO`ed9-VCY}hc>g_$H~QU~ zv44qIX#UUK*6Lch3+D?opF}xh$AL|pXBcTWe1_3t)!cIJSt$Vc$_4_R#fgV}kWp1> zl=5bt{}cDwKAYX_sTXoy{O$`ATiN&K{`PmZw!fJ*Z${ovrvLa*T55Sq%0dkJIe7k0 z6{L3KAJygM!X6di3D5s|c?Dxz%J6T?@wwjfe@H=O3%bU8whzI7NqHL_^(%vKM4bd-@1t7LEh&pWo;@yS05eA@TD}af$u?ifo@NK}yHg_OzdS zB2N2bmiDS=Yx@~OA%2e{kk?bP{Rgs68njsC?y8Q^`E zL?QBKsl3vlvhaWCYN!+VyYM>ViTXDc+1_IUTAU0xT3?9&VhbG};m^y@3?0X_pC68i ze_u!TlO8YHo&8gBU3|Z|X5m8en}|^V4*=5mC??dzO#()iOY|{T_* zoKK2QUDfzyv?;0 ztGpWvceGSg)~wrEwxVUlqcu(b>UAwE3u+!M-&tN;P~Eb*bwz&tO3}8sWm!voZE>D& zGw*j*ZNZAJ`qpK6;K5&Cu%u>t@tW<8wbdP)YS%ozs&!3%V_8jC6{cUb=BxmAANL^@_4J`J2m@cWr8I>DpX|&!W{`n^#u&8(TMY zLx1IK^EW)+xU!;U)5ZeWyE=bU+q#;j&B9*Wiz~LG@4Uv^iY}{Nye0m}H#IMQMERS# z*0r`Y2kLe#Tk>$jqGg543hS3_U7FvRSJ05RWXa;f#%1~W4a=4-!OKfHT|@I$Z-ei8 zuUJ{n->pJ#Rl(a%Zl{7*qg?kDDfx>QFJ8PTFK;QQS>bKz@YRXGB{g`VjL+Me7tCX9 zSI!;%_~e{Faqs%lmtdUy1^L`WYBAOxuV&ir>Dsi^Y&^&X?~hal#5LD&KM(iDJ?4|N z;8IzidcWsc{fPa)B>tu2YpLIGz94!g`bJJ5{_Gxup2j+2pnpx>uXG+A_YM68yQ~x4 z4-^j9Q|jJAHx2M}3WsZ{x-WIQp4Go$Cp`a$zYqY{w5y$;p@PE066h; zhTDNLHLtYm@c%p#FQTuKCU{NCL1V@n_YEnpTbe-4uV&CwKbIw3wJnfvwMi|N{pZMj zs-(%3p8rAekEcuVqUP1JlFyeVT+OG~WIqGafC|15@N_jOdRmJ*6+Fe4!?(!#KM64V z7Oq5}hjFN!6c_dut^~g#;dTjE@=ZX0%6AAgQQ{|A;CT{01#s1tKwp&bsel`|jJpkR zW2$&e!f9NbCPvF~c}L?K;iXbe>R<6jq^l&{+E1;7Pq*}=^hNy`PlfOrcYzeo@J0J~YmG^ZCxAyZh3Af7oOG7_K{y#Fnjq+y7`9tFx zV`ScS;725!_%W_#&q?@07WjaH-e}*~B;3kBJw=BK@iRr-^1q8ET#c)Iu}Q+!xC(y6 zKyMuPISIFp`%4M8j+-jyEsblOFD?lmHD4s0`cb@)09y=jdBnrzI%0rF|8Wm6i{)TS zUN@&^>ATc5h={XM0^E zXG{nL_5`1zDr7NAxS~Cv2yS&OmfLs}HuDCyb<}nEeGQ}ED;j8T^mcmL0{&Mxq_EN| zNK(pCf@DEMM`!+$o7To1vKPL2g{acO>I8;ZeGmC4)yNVJM_)2VuAgHfA=DY z$W+VJ(-oPZPL=tGLds=YRT=H`2`1BmDBL6O&d4~X5Gr1&`YhZ~%H>f2dG+rUH_yp@ zjJ{i=cqhYgGza?!0BL+k+fiTR(Gp!dwG9V!kug#2TNevR{(WfPu zmLVrXWEtCFFfqF!XX7lHc)YX>8D}Rmj%ITv^2CRD9L!)%HtUngIUzDbB#ar{@j8$g z_kMppy6?Se5MWPiSPHuDt^fb4>VH-Je^s~oez|PLGF{V{DjIu*#cB%5SY~hR-otM_ zEESY))>AWJtNU5D+xEZdg|^R5XYA0BExYd>yw2IO=lmP4=kfYA?$LCjqNVG#oj(Zu zb{xC<;d(vvL6t3Z&BhM>UmNQ?+QANi_Ha)IEBs!zr~kte_SP?Tc4&HPPt94| zMC$8sPdY37Kf0%X80~b;_EsIn9y#^QVCDJ!gWE4XGq~dPGlSpKJy${dEMA9ZpL{Xg zcj9>XG3R%}$M>HIkIXw6?wfZiyl>U%@Wp+n!z24Hg`b}HUijhG^Wo#C-V3M9cqBBk zegELo&pb1jI&X1kcFT9d7v>%hKXvIR;isK%hMP}Z4EL?L1iHdd=)i%&Z722*J~i)n zxbpOgaC6(~aNqtn!y^x03}4uO5j2ZJ70v^L7v}99d}{g6_>dkP)`HSIg zmuS2{4u$4nyj2I#&x!Dy))&Jgz*n9>6`pG^PU+DojMV|fLHUS zH^XyIo(xwwFNH76I}e(rq1ml_2m7Fliu1?O_la=m#Hnzl^K>{g?_&7rRmAg>Q0T<| z!8s@Q4u-(QMZyeDhfSCW6duV(i7mjcPV zv%jC5*i)1IYIcz9=&`Xv2W+J2JvH_4sY#mW>RH>QKEiF*Sz*-vzzO?Lf`17g*sQU_ z*Wm-2Mz~3RL%QQJYIl^e!dC&y;cY};KZKurP*oE8aD6fS&U5wJ&ex6^{WqRJeqht) zD4!(%Q@%F<-+KkVdC&LmY~Sm-f$tq1!}pF_e2;um`QL!W|CGPp$Oo_O{9!0kGWo6H z2lYc2{-yY;+s@M7{bljhk(bK)f3=47r(%Aboyo2a|4_gBI{bPln_WFyGU?DzgLdfb z-{{wd_pyGOM@$=dbUE;m=U9K_Us?afSM;mzl(7DD_pz)0wvhFoPh(dvB(bY0pe@2& z8{U)9Km0GOfA~S|(7XRjzxwVw?9lLECxnK-%?`cWfHidj>mT{iWWW+b!w<4UzkU$3 z|05}McKz;0X|93iL)Uh`YMM)8eHZe)3;s!#Vem?0zDF`a#*2{A37JT?AP;oWe_;YUbP+lkc_}Hxw4S(fIhg0_Y>c0*v3^=NKCb#cQzEWt|8do5eHr^>H%p)E3DNjOKV9?m^D$E5k8v$++Le#1H11=rOwZMn zxt>t!qN0%V3&kO_4cUb3ME$)3y`F>qT!^26WKaKg^#3hfSL0QQ`yO2P<9Y(Gf5iQ1 zT>qWMhrQl;E+J%tU6udY#w~8ddLO)ybHf2JN*haAe&!+&Cf1m{X?)H*+KdA z`)gQ#xP*ENV2Y$?7^meQD>i25@USlqHxdEM^=uFcnzz1cWV9W7xp!4mxdhklZ zE1kJ$*imJ>FVN;|>0s5sIZ!U0+<->FU0_2cj@f59^sbr0%C=mf>ls)E72{g0+I5ILLS`a2 zzT8Vg?xi7jT3Q&#_3x!2pR?@Uv}}=etbm(OL+<>2j9R4EV1qTkean^(UuPbxt806_ zE{a)Mwz5uO>hiEvYxZ{V=pDsVOXM$iRZe!J$TS;B8+OaqvXsLUtzhP? zHojvwqn_k&oD^Eh3xH?1V*T{w2!eXBG99-bNe?n!b^`7`nihpiF#Cq2S3@iM?*eYa z`GvJaP;w=Z?x}$9U35)L-QCkOi#Z~@duBQKEn@_qYwbkV70>o>03h1h&(I+VRq;BE z7u8p-t0=7~a+hHLk8>1J=#Yd+7vr^sK>p4ke6ggf(d(wuh2|D-XLEa-yR*H$rDLue zGu!9w@VWCB&R;awy^b#P3zxFuuI83TcSm(bsLRt zsO!vas?%AO5l?+m-G**$_eYq%0>?~sHqAaemD`l;O2s}4+YAW-i zdlwwB8ZWyGG%y(c^Ot>^@G@K0T4qmiF!Jsm8}1iN_;ws^wc@oE2oxHJt{=nq-pS_EA)P9qPYtkC z8r)xf*~WgJ&3gCQBkVE&#B`&R?S3lJ&d1%Doyv~8*^jz~4D3nZ(O+*66G?(-{IYJH zZ6;py9-H9jK2%U<;C0T)4h8IXv|y2e18Hn%0m}{A?i=`45{u{ulGy79+00;q?Ke~2 z*v{)i+4jPqZX-Nngok{=iIHs75++027;k1UNsrQ>Ok?SRi3t;_|G|Sv>>SZEPM<{d zz~=_-6X~o|^iTABnhARJ4}I839Dnr(oc~(L+jfAT37 zA$!;zu;)+}dtfKgQh|II)7V$K$49t12%qbOe)RF{YHMq`yl-Sf-pM-oQp5rMp#R+9 z`i3+w?1}zi4<~nWvIqDn46uol`1qpUyE7reJ^+Ax0REQ| zutmrtBMSw-tXpH3i2+Uc$MS%lMpdSL%=vKzgsQl0U;e{F|X4;4^}`Gss)uU%Y*85at-J)h3gCgW0V7>xY4*g7Gg9{X;XZL?T*@ zezueJ7m@Onbz_;Ji7oxv#1H(dF!0br++La(uP`ukHkXImpH9R4oEu?h(Vpg0-Za$b zMA$`HufL?t2+o~N&5}z&k-z@PO@Wj7g;OiH_zaDys7SXXGZI<@%h5-!RH^^lfLZepY!v!O)t+n!f0KMMD}p|XTrYG0{S9V zCws#`FrRbze5LvP^E6icbxn|y{{mnB^`FY-qCUDmGd=t@m+Z;?72}bAZ6*>qA2Nb7 zW>Ea&{`fQMf4Qmu-ksB8{xoy|>sx}2{EaUPrx(D#u--@gi~MI=0*yC~`_DnL=d`GQ z4LP;Kz|@4OzhXU@5u7@e=o4f9HZ_y#+~4+~y$kJ?zpWzuPIa05%>ley*z?#lj(4Cg z+LJ%pftUUU|Au~NMD?reO$?B|-zI(2J&M#fjb~T>IaC7ubPBwj&!hD((Xu`CKZDe* z{Dbt*KanCl#w-8&5>tP@J2PVXu5YvGo94Gc-+aDiaD8V;{lWh;qWZ4S#(Lqh=-cIr z>AQ;b<%-R3K0nZ2>KpxM1feYm5Eo3g@Za}H8EVI1SUvxbfDq`hXvrO`hrLW$d z$Oq1%39Vl_e**b!27e>@9TtDH^Z02e`A^Y$W*7Mbtq=9IUW)cAzRd_aIK3n0Zw{^x z^0#xEKAJz_t$aPk{96wDi6Z8oH6D_E&k%uJzYBxo$CLf}_=p$f$+^6y;z93DEd45e z`AI%KCSQ&d@lt$NdLQREO)sSQnp?!RQ8 z8~Z2xo9BxTnO~ATyr}#mU~};G`G2bXDcSZmmNPXT)-xxz3L>B8>shjmq7C;C$V+&> z-eve1$y?Y9JRd;#Px>xfGS814JU_PQ1|7+>{yCEEB!5DC$j`^)^M%)4Gs2lN#NVkmmB-CSg z%X-GwH@i#5lQI3fiebIx{^8$D`xE%bQ#3iLDF49xJc}yX7xqWKpGod0;vw08O3eQ7 zfAjoY!0EX?X-+_&g~6#hEw_yHcjmv-`ZYC?U5@blW@sAxD>*ajUqcJ9AIQms_$Z(s zru~M4`%iLof0C6Gr9V&m4M#G!7x@q0pSU@F7SRKr8%%aY{R#PPdcd9KM(+`#Cx6=I z;OisCOCx&tQ>M}1a)ViJWBbBgZa2+0PS5?(=74pCKVp1pAN9u%)7Z0Nj%1Ouku>73 zuov{AeG=t6$fs=sd%-JT-_l&j&)m`eAb)ybnmz^e5f|q3a8+u0AX~qm>XR{nFh0i1 z$wHm)H_;#Ln@R5Cr}+SR3xk=N(RgO%hxqTsc-QBL)-MEfA`t#<_VyOFb zfj7+u;7$GxykR~-Ke<7(|HI#>k$ri-1Aa4thWP+_2_NQH!e{wJo# z@M*v&p>D*xaO$Dj6FwbvBR*q-89xE_7#{L+f93jsyi|9A1EC)q=5y{C`8c0kBEDRx zGRLzQkU#EXspQ}Ar*Z6!D)OgPYT+;S+eJM2B5G6&>%}mtJRVFV`=uU1Rm5}BcOteU z#(18XNd2pLo)8qXiPc~fM85*F_`3Ux32fMMQcKW55zD4+3@z?6Sb{+|3S6BZKsx}AI`?^DfZ{G5BG zU!KoI{V$f!@bigtY4#iDFNkl~&0qcu{)hPXS?4dK^A$((%U-0PA|Jtgb2-Q#`S}X` z)10p?0N%Jh(|TpiR}AYf;zbtLYb+-s-a~(;^&IQ7Y5hh$3+r{19__In#r9i6nt8p( zd?kEzy&f}P;q%*)uh3?kittCipHTLPJ#qe+IT_f93-kT-rGKX86u-u3dB z@#?54UK!5U;Xk(%ucG^JtZ(N1_5ysqFxGy17Z*pZPdq-;ks5toltc04K0em_ur=Yg?vg6@J}D2pFa>j8qdIgI^r4aha#CGzq|E#X4-F< z;}^y^?KhxrQ#|8%^ZvuM-vHhe--dbq{R!jQWs|>H;~73rirF9hrsF(m;DFQ*?1%l# zWcoaYug~Z3dE?HBvHkf&e1Fc*PkFxm#sbh!Htfd_;`0XkWWFEg{_*FuACH~)U_9)f zV&_*_|3~k~@1J_(_%?<6`~C6b8{)^5`}zJ$?Z-#QH`Dn~Zt(somiQJs|FQ0GF@JAj zzdgkm-wJ~hbcPSG#Qu=xN9Kfyu#nh)XJh}JHGw{#Gwi>o(tgnp@0_$DiS56g&N24i zI8dQ-JMpgSHum4?SS`Vz*sr3!3-#!J6#3c&<9QSC8OHM{^8ZZAH^u&w&xg$D{*&*& z$)8Z7@ow^D=0=_J)$9C0xLq1DOg4|E%cd`h@L;i;K2cH-7^ImbDVK@&z3Zk1h4?l+Ua8ULnzK>&- zrZmg`I!(uZpM4Dk+@G=kydi)1JRJTY&eyZ(^HctLq)s20CsG~!uC(m8`F<@bpOcY% z{Ctsw6zi*<9E7(YPN)9q;}+SU7CZmXyE%Ey=Lc!*i{MA>XVC%9f6BW*isdKx{J@l- zdUmeNXqgYTbQva*p-&G1hOdeQxCaf31srjiXXlC;Vk!PH$gd3On=T zSAPGy$lG>SR=axoF@1Gy!s=S=MiA0}GK=DRK7Dai)8v;Hc^%c*59a`Xg#b!WZJa&j z3Xjl5JW$8#8vbS(ypMCH{0`e&#`b7S5y-WA2KrdhbDW79s{z2Kc|_tJMS5_jjY~v( z$^bNNF7Ho`_iBP2_2SgxM0ypadhA;HrpbaAP@{!{(->D+F(5^w#;B&AA10po7*}2e zBh(kYL_5ygI4f^x#$xzADONnCJNgo5!4Tj2Jz^l^7t8Bt!c?RG#!O;2-%a0 z=-HstB#*@(I2E@g*<{ZsTS8>U?Pk!so+R`hm77M2gOAGXijR-V?G|#!>mw=`7$I4e zWEc{~AesGBk<32YqsS5_&LmBgGe~zM&PLBcGv_wl%6VL!>w>fBsOZFKd!g&1^0t(C zGgj!S7;Q9W9jzKrgC0lWi5t?vYF%IX5aynndr&WJ&U!VE4Omr-{0LZ4y`G}yI1+!y zR|;*EchT5tMX|0JF`llcLDg^k6)4tzk8YLOWf(V}ZKIwEx_EeH1?p$C)P|8qX%h{M z+D?qenVwqXyuvrXwj}LJFJt(z6JBUI6vPYsQHO>Dj4!J?zR_Z;KZ&|xlISt0PIVYf z#2?h@mM?w4gAEh?+m`wg&5ZY0>eZGyjYs|SB?A3#5GUFM;^W`2P9AZqZ9XY|QEI9E zR{Mz8=4N<5!&lU?M!T7w#_Ge0kJsmC$BmV{vFgT|*ehcY_cGyDk*jHl0Q(N?mI&ay_1MbE8o%GL~-Qgl3uxrqti&(a5pbWJ|a)_j?lV?L+@Cb2Z-Teyj8U_M^e4F}ju8czxbJht}Tz z&~?*HO&on%Xyqp!y=6aS%7;d^kM247988F=Ahi6TSn2Pyx=BBARz9md@#xj&j#gaI z-_iR^&cFFA@AG!H_j%W^X+9ZuY$DdjqTZfSJ2nqIC>FE#Fk{qHv-T}rUsZ~|HsfdC z+7);o9fo7Kt;(1q@Xty-;xjBdlC_+p3m-S(2gLW|#P&bd|LBu(E`gzcUE}@tDBkFM zXU6^|UZMG)xvkc<3Kz~7Xg-N@#*PDkU2mEZ}#|MT)P#x|GZZ_Dwz-tYgAg2)zhjrZF=1pfuSq1bQ|6#Z%) z>Powh0?rTSIo0*wX7fJy(U6H~DTj~lLHte^cWV>9_?@l{6O1ZPVoKSV_@x&kQv6O= ztRLks^g4?{Ina+>tEbTqA-VpsjSpZ@ygQ(Lc zAYS_)%XVL~sMgvZe++3k;}ZQYr$5p}amWU=-qBc!9i<_%hqX^?S1l)e6IBewN=tqr zKNA;As-*qCY=7L)mTs+jC;y@L9t#_d(&TR7sF3Fo@7DGegv8e~#U=Ls71=&lf|QP} z?P))^H%|LwmiDS=Yy0U!A%2e{kk?bP{Rgs68njlSbg z26&$(QHZ=*BCj;4Ec_2$4Rr#47hXp^QU9hQ+j~qvi<1FI>kIK;Y@x$X`1A5pL&x## z=Z9nB@9W5Z(&I(DvwtkEiyjcyEL=!_6A|iv3LuS-VnR*aD1g+lu}In|Xf4|Bi8KCJ zWcyr6N#j$osE_$%dyi%OW3s*SUusV-ZEe2@llSR3<9}llh*8}9oiXe9%8x41z8CF1 zmhlHeAQ0D?*|}W`?uq?rzv4=PY;Vi)bYd2CjDQQ?Irusi&g(?(EdNikzcjH zEL*}&x&5ua?!T9Wob+O zhT=TmCf@Ig4F$`(>RXrQfd_wm!Q$F&#jCb8Zm8+lxM9^3D_U3OH^ zNBO$)bzKbwOS@^j#`Tr$#T7eB>ld!eYboDa86CHL>nd;2()P0Qt!pJs`Np<&ts54u z+jdiQ1^HvqE!)%@Y}&N23i@a*Cz-cwBHiV;mlkYY?JZuFzp-^|*|wsZ?v3jU198W! z*s)VKd}yfv%pYFDcCPZ7BBFxB6h8b(QsPtDjg&Hdwx{*;}+`WpUf8 zyoT0wP4$~<+DkTiYis>AOIu2s+Dq4Ot@aj|tXx*UDt}XXS=Ywamaa|Zcr9GnwP|^! zzp-^)H}qGrI)B|0jms-rHm)y#y=(F}wymjc+9d3?t+;Xv`p#?IP}ya*i?_u8#Kz`D zk1Bst*Sgl0=0M%{rAxexjf)te-f|81b8&Cn zV?Id>E|s;Z4|tx_58MAs;(v8~E%lqu=S9y%-^dBX+paO_X{^Hr`q$L`a_7-;-_)PC z%R15hK;dvbrS2_sQvpAxaJZJL`%iwEW>X6HLgVlpRA63!3=qfdDp&%Njeu zdF0^$6(6V)@Asb9@~+V+PKozB(}JEy#8EA5hH({A(8u^O!{ts$0dW=d#+|o7w&Rjv z*tsdWC`=Mu`M=N?W8Z>Y3MP6G@ARc|qW4Ik^7}9p8jpVEb>P28`*`%UzljHb+-wuIAHgvY!EIKm}h9c)A)CJ*`Ha3Vy|x!#B(N zX#r;6#+B&vFb0Iu2+=!+75Kj6kK z<1PbSZsj=Sn1s`~I8BU}~75G4xNr zbE3)-1Kdb|{5tR}2Dnk)T&Z^}pR**~D(`?akX7E-CEVK2PbJ(c?=KAf82SIm05{5; zDd!K3YmAY3*MT3FaN@_fnjMkwhb`~{1HI9{uSvL-fBF?2D#Xteam(KqNw^wU`C^lV zt8o?luz}t3hC9j)PGxh}o+$dL*geyHL`W6Yd%KL(ZTjed4AxDjyC&yi3fE&jRO1K(V(eILQ z>$u;OaO=2~w@{(ng{G9!9mO>&3hv=G#+u|+!ByNiVu7ppFkpeJI52F1tEE?+f2#h~ zdOr*NQBm+7X}{$bxLWW17PwmPnMk`k>gt=@yn$|pJwl^*g}1A%q3JRH$jLJNATyrs z$e-~YCe{@tDW@j*eSv*4EW37;5%)u(3Bea_hh!FF7{# zhZI&?0ZB?ZN|4NN=;+K}eAC*vGxowaZ`-)fOTC@_yQ$45-pq^H*rA@>dWSyORV=Tc z_*1ePcWGlMByHJcSgoBg;4QI)o0>{QZA1I$g6*c zxH%&8G5Tzc;++h~(H!jW0HpDaxiz(P$>8guilM0S)%7{qp87VHY8AX2jnLTm3QB0= z$*RNZHbJ5IRrJ>Sf0N^1ul+|BgyLKEmk!~rHX+=KfTA#8t+d-@`%&`C_Rm@xSo!w> sAJ2Xc=|7j{@MJ%71=U{V(EBj3b!=*Kwu9dB;BhHGu|dD&Wo`fe0I@p{#sB~S literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k99c15121f2eb0d1777f2be633e7e5072.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k99c15121f2eb0d1777f2be633e7e5072.o new file mode 100644 index 0000000000000000000000000000000000000000..574aef28ebb8fc9cc9de1b35cb3ac3bcc6f0403d GIT binary patch literal 64016 zcmeIb4Rl=9l_q-XM{<=UTR)cnKt`n>8`;RV_I z<;JrpgF9^!1E#}FlOEC$-Au-EOb?mGOzat6M|4<;LmbBN==XT@SPw&cXdosr8c$+^ zq29OmIlJzyEeXpG>7?mgFRSkP&Uen)XP>k8Id|V%x9+}b&DG&h$jA{g-!*7OV0G-=Af1b1(pF%P{Ssonn%_Oh_049}6enLmhkZ|DnM> zPYlm}=ZWEOpZNChiQE5X_^s0Cho5`r$HSlgnUL(ems2N{qGFFec%LmO9lrY zc;Lk^KC$P;`{o@OzV7)ahP!&6AKt&`C&O=j^~CVow-aaa;G7?P`NjRKx;K72JZCHFIY9MXI(Xl_`(J#n`O7c9g*u*l+VY93CEg=cmK_yH9}gslm7IM?Gbzr+eS<{^kS2Z~f#a z!(Z8R1hT&~Jokxrz`1;I&edOjG5(L=9=`8qKN-GnKFWn^Pv7^>S8wck=i6R$#j?W~{j-bCo_nxa)5R&#Vpc zJk3R_U-(~`bBD{bRfCtT9Bf8?6O%h{d>)^l#{a*L%sgmT?~5LeWUCH*f9Am_A{U+R z37bRa=)kP*5_9O=_tqZl{=7LfaOKQ1#M^?t?EmY_2fu={U%bv7DnY-^vjejR?yWhK zyy`pA4Rg&0ug1+C?Z&J>0egzWmb$M?9{Q%ksQoBm_&k8R*WhzvJbUBrx+nf!X~-P< zDP*mUn?qYdp%-dH=Ajcv-`{3l_#V!)7zavk4b_|$>>j~(`q(!G3(d^)b_urA$G$9B z=%PIDK45R5jb^m@O|&^Tgz?7s4*Y|8Ve5A;AACI$+h2zAd82!jkcIf4m{y_Q!1tkT)^;-eT1C&A(gx(@=|9Kes6PUaHN!`wY(K#FOt`5emQi zTB})KQki^j%EaWMYadGvezVxTn}uxGp$h^CAF=VU=^r5V< z53k*gd|=XtrRc+}#mV7kP!94Ox*{~?_Q#gsocdFQJ}(67$aE|^EzI!>&(XT6Z znmg*~occ$6;-Fv4AYbn9mi=@m@?wxb2f1UwqBGil6}l8MBmy#47s?pgfqY<)@leT} zV;jz6kdfqJoq^ZX z5|F2+P@ZQp$Ok4mTMnJomP2P$zv%38oRd6Lpo1%LF7mYDoaDjShgRa8(=0$ z`dw-6XuBTgGFNWEx$F1NwK$i#LgzA9h!gBL^78#&7wGqA&{y!Fw|95mXzk(Jx;AAC z)$cxoJmQ4b+y1qE7V@tP^sl+Fe>=O84@~;E0{#2ugyisRC@1}E`$L?&c^3L3oV)(D ze;((qf6v^4bJxFiI(PjG;XF9c+HOUA8#0&z1PxWUL?EfIU;lWUHFs zzm8!)MZR_53uejt*e~53ft`oU`ZKqi!DErh`(MM_`8w9oe~3(*dpt6I|Nlh#jmQk_ z+p6cT{4n?D@U?R%Q$0?G4Y!yj<0-ocWwB1qJr=o;Wgqw>v*fkNe)&qs6PkObcje%R*yH8CZ+&&KxF+Si@SI7^>oZtfVT%y*BIC-vVuPX6w3^7f3AC+$oBYG6~pj(*?}$2F$k!Z?g) z>_xsm7-zl0m;Sst&T`3d@^t+)*UzS}rRO;3K68%ztIm;s={We~@ozzY*f@?lgZzJL zU-tmyJD2$;^XKMsoEKPN&a0iv$Jx%mMg8OP*?%#P@xX@RtaCRwX5*PRLH>B=(C_*h zIRX2If5e{THSB@@@ADknAD(A9Y5N(peegW%*ms`gX#Sj!z3f4`v-!i|9MS&t@my}& z`Lx?}p7L%$Iof;IMosI`w{q*PZQ1=j?d@AX$W0rMy)7JP`%UBI(SCG1dVn(+7q4WT zJUt%dr^nO(^R53e%x%Q3_qV^F`*YIw-=E1%lX#M}i%*iltb6CP|21SP%yc+&I>IIt z%HVV0eopeA)%Y7jr2KY#Ch;FOWGYPsF5yNVw|9Cyo!$Lr9kQ@~p~85`gt4m*heIYZ zF%k;n3Mg8N>FkckB!hjix!cC^+ax1w{+43%`C~VXfR5Og?I*4~i<4%#DS|{75OaS2 zmj1MtZr_MA`P{zMruj3~zon-mW9IktraR~N+&MpedsolqUO&BQn@it0KYy~RcWYmJ z+RRUPX43O**HW9drMuhv`%@jAn|eDsQ@3^Yr25*u_U`V^?o@v#ou)feUT4R)P3f-Q zo|G%qm-aTdF1>AQ=hpV_uG@NAm!^PpEn1i&$ep9APOobdrABf$FIuQdJ0z95dh3?< zRV>Ywy*-<|Zrjq?-qF#uwRPe0RJu3a-krKV)q;N7etiPE(Z8h~`u*f|#Ov(u^zKBD zFiKu$_a|bup&#v^h>mo0ZrpZTYO~kAwbPD{2HNKrATYB{a0a(|?R{I===!`~SmE|Q z6sNr2p8j-uPrBt3Fnu?*_igR;Hhm(78AHQ;(PPP=bDcbZLEsZGhSHtFIm_rL#)M^T zxkF8@Od2C4`wP{;N zd#Zoy!j_b`sW0XAq_Fu+^>_Dv3EN{VbHCrU{Elt*3%neSjt0iRYx(DY*&{yvCR}RG z@7=t)zcbxpQmLLhahYa!-Zx!;)%7WxgZYjtwDx{p?DP3l65E|m)E4d|SF$e&C6L@;q+Y#{Agvg(&=Ez$1u*=p%=t(SdYM2jt#CpXFsc zv)LvS%k9iI#rV`1!e7yqxU}dXiW@q{mW@`IBA&7f)kH~>34cOoB*lcidF&xN+YKys zUp{wFKKFh)qZZ>2;*3)i*sOMI}J*NulX~vjRKUV;ygU;z7 zy=3pnHfgoQsDD!_rvkwrd;A$;rk=~zIBYuPqeJv0bA>QOkj%sQe-y!|8$NgKs%zU4 zSN3{+m|OOsq7;|HeIc`%puThlMtmjKtpr7Jy1Lsb9Fs`*_ICF-C*Tr0+xt5ct&8R_ zZccoTKISjE)U4dr)!mWk-`=0@+?v?Btv{XU=-k}Z)0tSmcI}#*QlGo(#+$BPdqe7~ zo7Z1;!>X%RnR#myixTtNy-iyZ{aaJ5Ez8;yXqsfDI2K~EO4i#Ag|DIuVl3;Y=)5u} zQ<2Z94|GJUEQij=6GDq_j$;0z+_I=`V;V_eV5u#}oUAlamo73*CVZ+cpN^V@i3ZXU zGrb@khE0V3x6couXTkE+29=MjgZ*5XGvPG|^Q9u8=$z{Oys)#w{*5+Oo1q1WJ3_1{ z+wi&4n4!<2kJq7|8fXoEB7y(WiZYW$-88QBKYEe1)X0p*CR!3RFdLJN;C!jWZ%3GM zE9$uo891ml`uxy+e&9aOxX(0Iii744>0IXADgto`Zc~s`@df|5KJU3JlryiQHkvqi zKyb$PWwz>z@iBjrx#zBM&b*Fua*g$q%z-Zz=Qto+f)X1infVm7m4nN>nN$bxuZSPZ znLBYF!@t)jnOT`IW(xK5feGeFt=YLf9A>XR=$sFmO%4ZhWQVn<}&zR>PG)poSP$Rp z>qmWu%S~~AvB7f>xRaIO{x0!Kb4X`h`>{$7z82YBKjbg-Oq@E@x9%4aD!(KXc5{^F zNB>NOMvU7#T#IxhSBtc%h7>3tLwh_v6~G!FK>27f<~Fp8@)*CO{vzu~PSzQ7+(3U( zYR<6!j@Oz-ue8+Z8}dJBF7}Fw$cCa^z8vX!Mbtm({}L}ck1GgSeKdN}I1OhwY~`(j z{N^-_X$Wcbr_u{g;PMH|UrYUsQvdrl3`filC_fV6{3P@ZepRL@!U~}B$IGMWe~zYp zg!+4&#wQU$?YZ3Ffp7H}Ci(l;o#gqpq{_-;NIv8*nLckGjhBr7k{uD}8EAejh%VW& z)TU{^p?=5*gF<~(eXL(!-cz=IFHG%a^Yvw-i|q`y{hg5svueX6?-8_rPr}V#Cr?wM zJXQZ`9?cJvTZDY*qilysP=EV2o-C#L>!*Lg>Bb!D!)bCo zeJj%;Nz;7jTltHSd8yo7bKe@I=zu?XyBz+iDreq6()p_ynI$}*7rXh~n29eV`CbqmJ)xlvHY)RqMRSA4#n7>JTsB>E8{h5$D}gWpTT%z zzBTTM!?AFCk}f0p<4(VoI~GnNQF9f$X}-WemW64)R*`%^sD-}b<-z(Zy|PKP!se1+ zvGae&qDf>=@#^vt%&+Q7Z{j4{t;Sb97^d;wxqab6wqNqETS7>em2r9|(jDbyI{az= z!1%ODn*1yD$^MomIHli&zQ3;ZUG3`|{(45HlJ!$D9)156@wL7ySl<<6>pNb_0^9MJ zwWF%yEc&h})OSmjukW?ccQxy~n)SVa^gX#y-xXEWXwTPoYgM7XE32j;-`96bi?8np z^j%T$Yw3GX>wBuNZ`NNa>$_w;`o{V=Qr{)4?~;$uH}%(_-__U!!9R>Wze@`BJ-^!5 z_k+;)RMz)Y*7sV{_mo0?PpzJc`h9&bpi7m&{GL`#H}QRa&!6w>y9oNOp89L)`;gZ6 zbYI`BzarLmbUga5r~QMT-%-{#uYa(&QT$1q)*D~nCDB5C7b$;I#8+@dd4Ccu@%3FE znw6Q(`kv1EevtId>#@{d=}j-0j@EpA7nKz1dq&9&nE_e> z-2SaKJ=fnaKp)}ECCJR?WdBx_X+jzYt$LL2)!FsV?eB^*^#%DwnZ|;2r5CFu{qg#R z^4HuqWb+FUJ8lnASomYZgushIRsYknF zlQp#efIp}G7>zGq>|s7G$;6S&LOS?E>@TY_ahOx~ZtgGASg+$)y+eITJOz*b*LlT7 zJYG}>_Or3fB}h{LaXb(G{1L6bbXfl%#(t99bNUWr|GJJAAWDP3YJ0}4Cb}8IdU3b8 zW_w}zSbyA>C;0dz(~zcQ_Iqse*bmmBel(uN;~$3B%Lb3+$9$tdsDI@8M5`kXw?4(d z_w`wBTD+PZ#7meqaE_N(w0Mn_ZgFXo@apTiZj+?(jJ0?rIjom?e!`2_1@ci}G35u! z7v=1X}7T! zjqb@)O=nwkJRBuJ3I9@JF*|Q7D9EhQC06ZFlqejg6G=>?_vb zH8fC~WdtAgT2JXx{3m|{d#$51*|beJHBowIK0Rj+rK9=u+_{wIM$gAy2S?cJ{Il5W zd^DG(=Kp!ytG%AC^Uh+gbIxwBNlYN$UZ19s$MM@<@3PK*mgjJNlAr%%uect`r^jQj zD4(|vvR9PPr$@6_?9XleW7sS51La5BEB5aJ`9|6+@&o1nh1u(&5%$`A7JF^RWW;2^ zF|PG>_F2}~*|Q7R*NgS~I-C3vA9j7E^vAHieth+4aof`$EEvDeFTxZcXo ze_C&m4$Sx2!SxjT(~EMr{>t;Q{sh-wthY6kpD$1Q*$afv_IX+SGVXtp`~|P4$j|d> zy$!6#$dBeOl=9d=Cp>MxhVldD3+*@Ios+#rN3N%{X3_f3GV^-b*huN%`bz2G`bz2G z`bz2G`bz2G`bz2G`bz2XuCE8a1ar6(1?YYTuIH9yY8G={Hh#kQ16bC4I=Z+RbbG!1 zAji`x!bsBo8v8AZzf$?WjflTu|4Q-IHI%M8nU9}-n$v!K^*T=b@zrZN&GA*Vzj*Nk zq~-da;;A*1zp{_VpVFk8V0@L^r}!%3;eI?8@zodUdU+jHhQsdn@42fi7!L{0$jsz; zNX?k>5ZwQf_$u@@BU8?NKfYSS@sJv;FT_Lg@fE%w6ZR%y^5YwSo zgLrEV$v-I&kD@f+-@)}H;;%I{9>I7kU0T?9E5%1_ytT}akDN8$N^`}=TWik}Z>9Li zxyM`kLTG^I2VZY#{a5(MfyhkP~zKcTkigvsoc@R@j})&;@KFFs{DA+_+=_MK2^c`$Ne(Ir`Q(V ze5mv)N@$1X$ER9LeElzg{wqr;9_+`bg7Me*dBtA~^gpQekNX90aEJMR=nQ|pl6`%& z{x_2TtAp{su%7QypRa$qQ;vh?ztca(n|=L9t4aTeCyt^2`5ZsA`d>i$uP)So3G2Uv z^-uR3ii!&LUmYcX;p=|^-(M@@`weTM|0z-Oy+!%^4Z(OU-EXL_X8(5%_ZtfIf9P!b zFSx%0{THeGJAwHR{fE7PKH-07Wft>z`1f}L_5lBZ`ChaH{S$v$Fh175`176pQIyw% zz<4aq>l^J=?r8SMT0Pow=}h6+=>2k{i>t03OfMDfzumQPL^;}t`^!H@6NQoP6Q zPYvRw3n-p>)_4!=13^G~eu3^^qkTvt@g54F7sPv_h4Iqh^&{HPT!P^Y#Ct}GmvVb# zuNZHNmuBm5p5Ko)<6a^iv>*NNq0!={T%Md>BgacgUKcN|{SCxRX9nY&h?mX_#zV2c zEGmdcCZK=oAClBR>`xa2;;U=1U)1~4RzDv5Y#e7M4W_D{qet9sSiu{?g?fxLi^DW_(&gQ+LYcJsy&!+pM(mwXTv&laFbjfTQ zDwpQ?kL@oYw;!c6%gE`521=6;w)r-`?C&@0IG)U7%KWA#;`{s6IUJz%_p5W~Qa;Oc zKJlKfkF?hol8x_gjK^MEu(`)E!d?-dWqY0IUq7cDep6Ox&*gqRm+duLM*B0duNH4|FkVacTBhu^OxbH$z#ehESX8F$wao5ES-vst zmHY6!Xs;vVvzJf+cKr4_-!T6i?3MYyb$cBx-ZP55juM~M_Ij)d>dM>edehAM`)#kU z*}OdkV-ioMPZ#&<`t*C~Fym+f_O&>q-c1Myh4*Py+%c;z|dOMD6S zN6YN>MUv!4`Dr9?AiuJ}p0U23v%R7|99;jH_R2Q>5%$VHh4cT3*VmEp*-Pwv8}EL& z+0M@~?X^s;uM;glZ?A#%^mSTa1M96_U(4+N1}e7eE2Te%_4VVk*S(~$yuEJ0li_sy zuGniuK|Gf3KL+EaE#CBC{1)~yF}Qy6^8|r-EbJ{FTz|2jp5(`K*OYEA4>ZPmYMCZu`$124cIHo#A&-9_V>HiURegtxBFpU<2h~j!z=@*?S7br7{`8? zo)`Dyvy(X9Is|zo-dgERoj|6PtKj(Iq)M;arV~hG|3~py+F$VVvHzpiTobpW0~6xs zAusnD8)5y` z-uK1w@HN}2@#;i6vZYktpKa%aF$4I-^HATGG{HkiZ=(};oGdp#^z%7BdyPD=@ARXe z*STg}SwVY0{MS+$O`rc)G^gkJaCd?RGcJ7jTFNOwE|u^)LXM6ZHHMo}cINInNJ_55DPCp9Q9RL0$Ke=dj2-> zJR10Md>;T~hv|p)=DD>L|DbqKmSsdfo>!w}{&_XTTWtC${5RyuKhIAc=Ib{uX%ztZ zW8O4;!GjMxZ%yk<{&}^KZO_!(^&1?!zEC=_zBqm5uiwa;vY2wV|9Ji$Kta5v$i`c6 zeM$1W=lxj%=r@*`;`$2&_J7R`s+bNto)lj>MggY$dh(JsBT<06tIQD$w`o^Ab zPG3|W`VD&%SylXp=UbU2(0uEl=YImv%i#Mlf#>V!`FrFCp6@}to95>bsetT{^BW7U zPxbS5SP!SME=`irq}08~o?%Fkj{QIP$5ezq-M@m@Lis>Dm0ekV~7mS4tBG zN8UcBVC~5=1bOj%CB;i6P0ycE(msF2@&}$L!u^c`d!hN_p10%n;NN5YCH7t;*%RB( z10 z1op$opFNxMdCu{Ea@H(LbDf-SY@{^ziPINeNNIonI%5W~4ZLFC5%j4vK8X8C*mI3wa z^_u(*>bKYHoL^sRjS(Bf!)u|R;wgdsKJ2A*T0QFH!}z`=l z$@>es{15sIT2Bi6MOoqcJ`vx_cZXfykqr2Y8E(GI`tG?{xy*m~i=&sJJf+!Rylk&U z#$4}RdEZ}!1Z@l|Uu95zX*k6vvUs&t&{l(eV`%C+K4H%z_6s0q@!#1qf1OFv0eiOl zRoJV)pV9V=`9S`MY?b@y{0*hKPtM;^n$~pkuX;a$^?wHClMPZn*(0UJe_{Vnu-|}v z*Uuxa)hGNzMZx|8{EQli7V({b zdeRKC1wUQR@^c#3{}k^j zGvSYj_k>Uy2kp<^rg#tDf3Owl(c(RbhZH=IxN(v}yr+n2;CK(N-zXkJ@^U=u=mPW4 zJ^q8|6KhGAee3(!e)#)IpWpm+p&m6a(k^La;|5kyU_vbge@uus=2Y>7Fp4N0}4jr)Xzkt2}TKhkq zA4OE1i!Xl6`@fBJ{UG0;%(6Yv^$E<$tw*(C+V35OyJGu-|7-E;{rw{RUBa8@?|;}| z$EOC?Bk%+9B>p~T!oUB|>kr>wL<33Jejq-A`dhpU3gSr(ULu|Vo7XJ)%-{5fYCr#d2J*|0*rKUYp9_|KQA4GeB=l5XGE#8RzVf7rEFl^rx z&ne^gIdB@^H^Ba{8v9$9kNX>5%~ZR;llZNfiff%LQ`p{$ivnq~H%bQLw*}8DBHkL< zk3v7Oey+*h??5~k`%7ovNs6~b3*x=7Puw4^rl}zQ3G1b?lRb<1T#Nj|{Waoc)q&@w z0`VE*F95%|nm4GV59q7Gn?ml#a159CpVhqIWT(&R3;5Dc_P4kmo65FFg{-|(I&eK! zS4Ua?{+x;@{k0Jlq?;DWyy}pL8 zeW?;&-|NxefWH{W_1n-ICjei+-D>UA`ioK5d%pf7GvfO@AMf>>zQ053J?V;H%Eaq^ zOx@p!`uCg3|Ka{2>yrMAy1q|W@!+xvw71OB{X;JQ5%HXjqp#=iA8tLd>v>`PcY1I= z$Mv3G&!?*OyvDy@gY}l~=kQ!*`zHJ6`xkcqfccT|rcdL&vCYTzWx@BkE3lp?Xn)B* z2>qk`LHYHM>#v-~b2neUaDM=Mn#Ob8<>UHrn)RSbwvPs{@&c-#uc=W#{LM6uwK#cc zzEi$$FBNr^<~krh_7k;~cJ@Q}mt4Fg@1N*>W?4|EJ?sbSDc&S$@=uhs`vGphXd1_E zsD1P=){n1|QCYg)=le@l{QevEft*j7I4Ga@3o(45fe+p<#Jma4ACdoG0R08_TUej# zyqPh+p-n}szo2y9U%-Dkcan7STx74MX;RnI^YWA7da`!E0Qo5%Mf%_a^;c!0m|kxG z!1P`+Fb;rndS!u`ZWrEH&6f!*&@eB~qe5A6HMUQj>9bB?aVIi=Yj+-&Up5le8QVBA+Vk!eiDlX(uM0E#+U3dAMe3@hdocA zp?B*Y-@ggOr%*odJ{!!(k@s(=PO$SM$@2~NIib)#OC|*ExdHJUn%ZLjSdS<0+LE`= z$&)Fc>*V#P5^pZ`?J?%zeV<6P%J_aOzh5Qqe;T|={Qe&o&%t_J?fwt9Z6qnzS(>iMi# z;qy{{eZP_Ct*+14Z{^d^Tj_j!|BbqD-xmt~()$4`H67^*N01s$#!f$)gPzzIbZ)Z+jsl&A-=jKQ(QzZ1eNP$*kdvEzb|R!yC$CT z`551-%td&;=-u{xV{G3S(fW2p?hMTzTJQD!g&N8ae7~!vra)eNpPu#$@g)AdUS8AS z+3(xOvz*5JR^xd8HO7O4LjAZu!|!i1N8hFOAzGCwEy>G+`91^nQQ!G}uSjG6PW?A5 zAI5V6zE4aCC{g;pb(J@hzi%t=|HS)dZ9IbB-^umO-Ucr!m&O7+K2o zKR#%_bzq;9w;!az*YD2^!(R9x|BL$9Z6Am}&Kmwc{J|Bd|BRt`L+5h$M(BCVg8Kdi zq^)v(|M&MA_`L>xuYq&d0L2dB&xgK1S4D&Wq4$>`{ONsnzxuQH|I7P-`_A9?{t{SHSUjlHljE6Y|E%m22rWHfj#>9(>}(YCVn_;2&&Gjxf;JqB+3bX#dMOn9LG zQ7td+FT?fJAJ)%^{v?o3b<4eya8AI7dFW91dM$;9GZc2`+U_xAXlLLcQ0J_ieLg15nrFd)v0~Jh0j>tu8?+ zQ-XS6ktTx1lHqXuiiM~7C)-g&D2#0UREBMW|BfgHiKWz3T2#!0XgI1%jjlakqGuu` z5wS+p52=PC=PXskFXg0);K#dFU(PC3^dqIh`f~2N8givR8M+G2W;^lx+1xhjZ1(MA zbiewUe-ShN?6bUWJht~S&hk+uDwyHpk_g`5+ru1qZGalGUK=Fp2Yc4{2%ML z>qvt%LiCA?VE)vH~I7RY;frv?ryuZJK)mIxQkAQ;Oktv274$v z=%y2VQ2sdL&OYVzFIGcSgbu-N%9wrbMB|dLKj#2A_upY&sEH$xG+ zPAd0E1^va(E2>PDG&EnxZ{R2pBjJYlfX&PQ)q1}aIaEHW#5m&0Cy+-6Eea&NE`Kd$ z!Y$el|L+M`p03~M5FP9CHzM!Lqm-|4cG$?J{Xe>VomEI*(x>oQ(v>fB3ZwQZW~s~X z1^~(k`Pbm-5;_w6cclC>jt`<@sx#?$^3mtYOIq5Oax(sRqVPwRU+pTO^2F2SzW|K7 zsHUlV7{M6%-DE0H(P`^zTv(U!OK$7}jKtlQoD3034QIPe6 zEMw-F{mB1y+P^CaaR2F4%F8^u4&^~5`zh#u44tCmN!MX%U&_h+rH6DzD}Qj8Ex+1+aQoyE zbonI!uA zpHiOWINx&QOYOr#BO4TZmCs;%B&*oP$dVRGx&aQBV~8{~$|3e8pHgo9_P)-mJg?VN zWvvouRgd}&K(PhwMhcXwCMZI>lBc%2>FHqkF}r_!B0 z{k>kQy`#Mkg_?S|r7ugYXRPU#o|fiB>r(t*xcm#viOt=;?djH~bD9(Vop+?V!3i*1 zLFnp9FItG~P1{mn5FvGkPYE(0WV)-pJJsuFb6d@c0rA=mr^1l3(=lN z&$x&ET|Ku4ijW-K6{c<6y0tac55;s|CTMd4;KC7LN0$c`r|xJ?b@lXZVw|L3tJX*0V;%JL_V&E8t`9^K)xEU#rff~ibc0O zIx`6==spAG(LPVL1j%a4%;z$}A^|>+XFwFD1@i59vF#comYp9USE-Nt%4t~Ww-_{e zDcca3M1l5L9=fa)lkSxQesxNpet&u1VA2Grd>W|Vj?eNIbIT$--nT6F$6d+=+V(pn znfM%0U+dCE%NMtp1{Wp@QOnxo=aP;34 z|J&G~m;L3WZ`hitY=@)JM@BX%;3vKoel+@*#owScwFdu#Ta3I-ub#B-FQc~2;I-On z+APs=>!iMXvn7v|eSOkjRQ$WDN2~wRb%5$U&P4mCEO7o&ryk^tns+%SrFjwFEfu0y zA{h?g)NeZMG!P|OjQ)}uLcBw=SknXc^8;j1Ju*U~UwYX_h_5m&Pxc|j*ccN3m}^4x z`=5{|d;`kSA^QAlS3mz274V=wA^~~05dPf5^}0#~bf2TeVJx3_a0P6O^A2qkJYIlN z;u~mJ=4LSS(~y5O{5_CH!)wRjFBt=GAEVv6G3vj24E!C8OBM7#;8Z8`^^Gz3-x>qo zHHMrwjv?ojW8mK(qyAUMs2_AbCL^C>Kyt`Tn4<86BU}KSuEXTG*_~VnTuhdrM;%V} zyGu|;1{_ZH2l3VD7adg3WP8eg*Ew9*bF0HIQuTBM>IsG*b_MV$f%*pmIDgl}X!?-atjo9p@VSe$~fKOOoed%0w$xzY;pK9;1@cKKKdLk@`&F^ zJ6!ZEcqYJiUF6z!1o%NbtN7x#b}7E#yB)q<$+^ej)GzYcats9UnF8YT%>a%E)$>Qf z1*55+Ailxjqz9VM;%_?~F7sLNEsCFX{<&N61@BY*4UV5ye8Kl9esr(x!d}G}{9(nf zaQsIUU+_m2|D^|Q{R4_G_~VLy-0}A*zTgKHU*vpR@dY1J{0&b2XBA)Y!-_9*9#MS3 zk1D>%`I6!b{<7lBxV)nHf*(`-tdsw^;tPI4@kI|O6<_dEiZ6ONt@whUQGC&ZIBS{D zg7c3tVG79n7d^zBa~Hng?#u9bzQ{RU@ddAOxYRFlCKO-rM#UF7n-pL0X2lmdTNGdL z6^d^jvhzQw_=2xeeCgL}#TR^y;=khN!#c$me7)kIaQvGUU+@izzs||uq4%7@?ofQecPajASO0Fs7ksbc_c{K_^paB`O~xiZA#c#gG1(oqu~3U+{+&U;6ck;tM{Y`L17&Yrew|D8BUT zNyQiZDaDs{|DfUv{|tsN^w@-=z40H!HsMt3~kzU#$4D?^>?-g0E2g-Oj#}iZA$T#TP%nM)3t- zr}!f0dc_y~X2lmjaI4}AzCrN^octY%FL<}&%R1ht_=2YuU&cM7_=4}yd^f&X&3E`7 z#TPy7ReZr8R($dEk0`$2k1D>%IiUE0Kd$&sIeqR^e8Hbme3A2@;tT$?;>$P=DZb#( zD!%yn!-_BX5yh8&9aVh6Us8PO*UO48_;JOTe!Z#qf}c=)@iQkCU+`0kFLIt%e8JBs zzQ}1`kc-C$T%1SnvPp&eH<7c#;ldZZTJc5B>54CSjpB=(3B?z@QSn7i{PYbUB9GvU z6<_3BuK0qlP<)XysrZ7gQhbqfwc-oDM)5_?b&42!E56`;iZ6OgE56_v#TPr>q4vzqVpd06uu{*vO0oG&ZB;IAmY zjN>uI7yP*5iywYd@dZDj_|mVFiZA#n&3FAet@#d*y60(S|0ey4Ib8gS;AM(0exO3} z1$Qr-^!IxYT0b>i@kRa`RsTy)&V=SWyixJR4>T#h;LVDE%+=GP_=2xce9=Qv@daO{ z_@al^iZA#Y#TPxSQ+&bKYrd2JX3ckahvJKzTNGdLZp9Zl`xIaBwBn1L@|?EJAHjEM zzLPVn>X&ifrT8+AyA@yXJ&Hf%;!%4QU+_m1zv63lem<)Bf)6OZ=F|lpH_TXr-l??@MjfY<&nmvixl8c{ z->vw4PM$rAFZjcXFZ;$v6kqTG#UF6<^Kr!&e4pY=zYZwA;7=;P^y?|b7yO{&OTV60 ze8Hbpe6hR3iZA$4#g}>blHv>gvf@j>UQv9(k14+N>$u_z{-)wfzfLH=;HMN{?C!MU z3vMc{-xL2P>qyk$vc3x*Q+(-Hnc@pxq4?6TYQ-0Py5dW}Y7}4aM#aC?&HpCF7ra^V zUwY8?zeVu{U#$6VyQ4qv7CqKDOrFZde8uh?tlS*Q4duUCAL^Jc{t{8q)^=k&8d z@dfWteA#zxQGCI>6~D#F->3M3?@)ZvLsszx-=+96Pj)N5;CmEbuCw+kzTgilzUb`{ z#TWch#TUH|D8Aqa6kq!Fq~Z(yl;Ur2`aG!kfVwq~fo5 z$m(&G;tMX%!-_snI(gP8zToRs{X?$)n-yR1TNPjA+@Sb^cPPHdxkd2>?^b+~vrq8_ z&nW(CC;tw`7kroE%XQFh#TR^!;!D5wD!$+kE57vW5ycn$QN@>j4Jf|g`xL*!_4k0{ z3;vYiuXFr^iZA%niZA^dQhdRmReb5!VZ|5xh~i7Xjw-(3FDrh_!*>3^qWFRzQ+$#0 zxZ(@`rs9j7Clp`slZs#A@de+l z_@cKxiZA$H#g~2c!-_BXBZ@EmdQ|ZRA5i?&&b}U3e8KlAzVz#W;tT$y;vaMV@F~R? zd`R&{56>#T;D;4o=G76!7yPK=NB_)@?@Nj=_{)kfdV59j1wW?vqPOFUFZi2^FM2zn z_=2BQ{DhPLl;R5>bzctA_8W7!_#e?{nc~a7u0rtzuU34~!*sJzUU#T_=2xee9^;d#TR^y;)@>EDZb$A6<_plv*HVWtKy3u zHYmQ}9g08TuGh9GzTj!a7d>PYU+^7@FM7x-zTmqQU-YnB@de+b_@alsiZA%XiZ6P2 zMDYcGRPm2~&H9G{#TR^^;>)^uK=B2CQt?G^Pbt3O2NhrZ*wczH_>khSd)UhVtl|rP zMDb-`bX4&Le@XFWTwYdu!Cz5)8JA;sp%M@ScRfXaUUak0hUH#J)U+_l7mwD2p_<}bp{*$i$ z7R48QvEqv!mMgyCD->V!kW_rZS1G>eVYT85zDDtnI{DWrzTh1Wr|%C2zi+t3;q-mI zAl~h8`kq~_9a{dIakz}T%(op5mvI+7tN1eCb}7E#yA@ybxkvE@->dk?-8_F-@dbZe z@kP#kiZA#9#TPlBRD8joQhbr~pyCVuwC1~g;E?7!{D|U{* zKuLGvWn6e|Sr6VNcKfB5Zfaf9y0CTg!p@B?9j(ijE!(_sW9QOEi#nHeE@@e|kgEj0 zD{a<)?)s~*zLsC4c2j5i#`doM&W@|S-mO>Ol=jx)-DLe{{>J|PkEEF^(fX>J*4(&0 zWoyee}d`*FbIG%>16-bm#nQZusmxybx&9?Ur>l zI*&@*(s=vNMl?pK<)dH$+R?s>U&>X0Ta|WhNo}SV!kIN|K6m{pymJljFuT6DV;f%L zLmEzXScuh<>b7#FdV4mZwR6^QD%Fw3tKOPb$E5J=Ln4*W`2L#OSf;t=k~4q>GjgfxolNytS;6-`c({`hdR=CZcX9! zNh!PzFP-Y7*Y;sBdbg(N?N0rv<>#SdO!@Bi{{GZh@9#@p-F|zgeZ$+;_H}(7U3a#& zF2(Di=#6FgXbn)Vr5E-wL4&P9TCk?&EnJy>9hUt&O2u5lRZZO#MV!?}^M2uWHhR)I z_qU^SQ*TFS>bA}vQXXDo*x3!G(kWbkuMAhJFYRq^U3%Nr&aM1jyw;^DAYF?VrU-K9 zsH)TJ+C-_5+|7#?s?rWgrLNw(1=dE=T-n>Rx$Cwqo$VcXV;(e{O82J852eJ3I^TCb zAAbVpidXrJI9i4+^s0MLixA+}3yx1AMVNl`QPQ!_xb?#&O@*AePg)nKiKI)*$7<*3 zlWi8EuRqf|AnTP}tNj~*)IQ|}kEDCq%$sgKlsw)3W5}R7wGNsN*bMJum-= z@|v&HRR3uG-~Fy_LHbGkCml<9!OPKk>c2i(octQs+%2xW)?dE7Dj@Aky_=9fn*0M! l{=JS!^3yAAq`b&~KN{9;Q<0V@=r69x-N29-mH4mA|NoO76Ab_W literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k9a6d1706c5fcdb285a9042f823e51edb.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k9a6d1706c5fcdb285a9042f823e51edb.o new file mode 100644 index 0000000000000000000000000000000000000000..c8ea072f4989137753600969505492e53e700051 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Kp1A?+13jo8ITZ&*Jf6oAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi7z(=Yt^fb4>VH-Je^s|`f4OYsa$VDyDjIu*#cB%5SY~hR-otM_ zEESY))>AWJtNVGj+xCCxg|^SkVC?XaExYd>yw2IO=l%<>=kfYi+@tA4MN8LfyM7S* z?KpPz!wq`qgDP9-nvEU)e>T>4w1XW6?U9}gR`|VaPydG{?5$tv?C^}#o|?0^$$dk% ziPYDTo^)3De|1m)Fxu&w?X5bDJ#y;l!OHUo26w#o^x(?VPY-@m_gn?-Gk6`IbMo8a zz7xm8k2=2-K7Qatcx3*`aNqn>;r**mhcE3v9Uj^LUihi`?}i_0Js&=P>fLb4%!flG zI}Qv!_4Lz&sq>eF=Cph#d|}@4@V@tc8h*<8X1MvprEuTM_dr(|3LQK+xc$U|!F}_O zhbvE?2sgK#4)+~+Gd%LprSOFvmq4>PRN*{0cwzn%gZoyapZ(tn&uPW@r``-#oWB&_ z{vM6@$Dz=CjJNtA`Z*Dv+xqSB2=JBXPle~4JP+QfpFawX>_0g8)cmIhL#IxJFW}Yu z-kagMCr^eeobQD%%s&sBWuZB(PYm`!7ZvA^qwf>p(1}ywO6TctX#S<}Q>%&RrJ>M? z1A}uq&ll!@JG^iHo8Sw2fb4yECC~Xnr~3Hwiie+eJhqOro) z;RBjRxJi9Oy5mu5ca*WhR{_i6ZA4!`gr9s+RTBDeLoxi$bM@M;*Nz$eH=aL!VAJL* zpCtcNzBd5hdj-CE&-d2OJ)7PUd#GZF+a}EVpoTMs9${@em#`UuAVKKba<#iJAC$U z^=rfXSwGDqrVTu@0{Fu37FBVDY=!jIpB&HfGci}?dPeEAUUYOrI|bfFLO(~ql)Lf4Xt zuaN!j>BFp#=iof>WB#JITdh~WTl)_hbFs?}=mbS)nnnRWDC-1UftLfF@4(fAR}x<7 z%tgbFDm#3EHeX8zs|F^QusUX*o{FokG23{XrrQ{^kHeBDi_OXGxc#!oYl{>@Xa7sc zB+0hai<`6mSzPrUEYZW3vIKB6g(w$vG<9?aI=%I{Q`hcRQRmmXj;6N8AY0JZ-sxM= z_Sk~X9nEc9+hg^HE?IwULG-4fz18pSWD7cd!OjIjw7Q0_&K7S+N1eB&Wt+FN$rq?= z@OE_8No=RDt)o5Q?+i3AD%jTQYxTA?Z)?k6(w<-U?~B0zb@_z_b&HlRuFKEM%Ue`X zhkpf&^B&G$vb3%)4{{2*?gBA7n!EvDfY;x|C!Igg-r3&S?MHLMwYPP2dfPhl?gIC1@cLVQ zfrh)n%rFg?#cE3qluIW!pb>Bv*ieaM_E`?SYo@TWEf?r|29`m^xE8B+9U_mAnaGVV z_tKDiX~>mMe-LkbTTMd1?6zAow2(2D-MfZK3> zVJ#7qTnVImD&TuJUDHzc^z_VTj>w*#*$#fo7{TXRJCSw8v;FG;h_?1KbO=IKypG^S z_0{VuN-K)oCD{Mt97Pm5Bq7ozcx@$+zcUD5Ea__Wy6JSGxy9Sr+}`HyY;SMrnCHgK z_IW#e?)*gy7SD69r^|xEWvsZXxuwzF(cRJMYjwAFb#%HLeOsH`eC~BsRV&xltuL!t zTT!*Tu59DFvel(!rEGqcdy#v-H_*`J?r6n%z{6fQh9+L=dSa{vRo&emw(80(P z^5{NY+~bznGh+8QbDU#qN@ez~cGv}NAGV9W@1xNfJP;>Cm(!VD*6n2J%SF+;jYc=r zb>=qJ>8#3#r#`7}L$|i65vH%iF;ksQv(HK8HYK}KvCqc%Fp4V~w#vY5C3u^b%KYfw z1xKvL%kBaV42J*wWuGp*%$Bu|*;5>hyt~JS`y~>-14mn}cx?j$g~p-l$MSkjUO&RM zln^N77o%qzW4*hy2>TU~WT2rmmenov#V1LQlO5lg7~uzSDWIr#vITTVrwZs(11yyW z_g7!Gv0r4f-u?Cny9@v^-RNX{_9fc+xSO(5*>N}fQMZtRJqbMe>kVQeNf3=+)~&NG z#Eafz6a3tV3d&5p&NHnJe0)F5k2GdNkk8P zZqPoF&N@Z^M9-(0phy4EhmFMXSAW3y&xL#jdhnAb?FD|&AJKFDWprx959&I}=dTXi zh~K2(Bv_H#H$mYAeV7s66>v@>epAmzlGtDxEBrI`k2;t44c?zK4K;r~`a}Ou?qdtEE_nU{2>pYp7)OY_2>=pXiQawjKyfSdM&FY3L!5+dva0LTa6e;EN= zggi2`NZ`x5HFlX8(1d@i2{BJmpV#J^wQ>lebm9(<7GJG@yNfn5Q&@*8Nrz| zDgJSP{5kc%!qk87uIVv<8ajyeEx|_q#utUt3*lc_@1y=j{xdy+#+%Ol=MdR*depy$ zoLXUET0+!cu^!9}PMb#bi7|hhmPvK)Z+p?+h4#wdR+E0GxlI1%0NyR^d2BkzJ5U$x z$sg^&OMioZL%%bl`c?KO2FTuTlfLO5Me3Wzvn&4`DuI4F1zyhQ(fXHY*&h0zN$OVq zLHg&PND&_6l|R4K)L-wejF`Ub+bsH~`EAfQpRXBQ-x*SW@c)dczU#BGUbrm!cDZ8u zt|EQ8V)L8N544y1M*kT>XbS>F271N%OY?&pQ0gnoEHCvJT@QweSb5fLlYC?8t9KXj zfwO2r>sQX7Kz^IS-$;Ik#oz2ae%eX?Q?#DhMgBnRLp`mRqP>c5GlLFJ?}+)EgX@F* z?VP5M=1+JlUym{WR=|Fui1}xYhh*O~L?GAi!r=JvWPd(B;zfCKF0ZM0(7OvuzlvXe zl24Dxm*Ygd6rYve$N5du3n~8iZ7>yyk6kiTL6%qMt@#Ziw&MapF00U^0o+uO7&Z#p|Jk+KPakBDW#F>pIz&$9LpE@MnB< zO#a-HMC)sLvJybp)9inwf15Er+mG)=zME$juJ4gKW(1Q_2SWqopVVIphKu6u8NqR=8}T%KjrauAjrc^QTSj~m>M^`! zJ>%<}-6iA6nEqYGuwHZj@Nc303H)OpO-?GxKQKSfqDuCK{gLlyk~@lcNcNu+vp@Xb zJUb}cdf;<|$&RQ$A-_!zxU<~oJwo*4PrDs_ zeZ+WaL=S(;H2Pa^Fw1RhU%1=truoL{xj)(*u#WIYj8E;O{`g@UdnU}0EOIuIM*J1_ zf?l*wqI?JWv~A!C@XFV>G#BzSceFpqpB|X5PXT?zh50;Om6{&N)~8Z^GA0nl$9Oqe zsPp|M`h$Hl$zA+3A0TgGFf%h6&#e3q|GgOR`ux!Pg@8^3!oSV_eu(CiaXw)EG5I_2 zruhK8$=`uD%m?TvH)!^M`1^FSFVAMn2~^kc((&K)Bk=aWmsmkU+q zc=jUl$K5QI{2Tr>j=fPu{*+2B{H1=oh$mk}jf!Et7)F)HgXv_y)aOtY@tpLXi0z0m zo+lIFaA@drNQo$z`B5YZ-&6XzWqGh)=j#Xg8(*K#nCxlI&rJCU>~|o* zkf2EWSrgTJoh4vL51Z=BqZnerXVXZ$2E=U(ZT z=QC0Ni{&%?eBxZ1{f7Aq;@fremp_I7A-;Xq`OD~h#gY877wMB%&{E_b`l>K2(oIhqx1~%ftd_Vmu<}2h6ar2cKW8^FD88^;X zvUtAYj-RhUo-8-T_h`NnG~^qj^A*HXcUClC8HWBoc|MXw>no3!=+BamAYPjC5eLSb z;Re5-ARoyR3qHl?QTfrGn~#wH{D;g(=9%Kx=>0ck7btT7eIUr}1mXDy&hN{+ULG@E z9W})(!}&V==XT;%bpMU@&Ai`Uh|d?s+HddX;;8kB$7ecHqtAQu}{rDk#-e8~1_v73@{tNBLW9K~>5BsOs z`4!gx(fjeK({3E!rf`3s8b7`veoUFl_g`v1K03ac&VO=)Q>R$sTkQPDy1&Kzy@~zy z6k~iV3{KD)KEM+DLz*9%6DGn!V*j0u{dd*``h3o?|DHzsMMJ!E(uO3q|8_dZ*ni_d zh05*3yQ`8nd$0|)} zmi={_j{QFS8VI;QWB++W{_uG?{6U*>CgxT2wwKBl-CG zA_*zhS35ZfZ$F$){nN)SvOg_${-1Yq@|w>N(%2WlkJ!(m1DyYqcYhSiPw@GHDL?rP z>X7PQxu=198n~x{dm6Z>flplnba;+XJAiLFu%2IB`~8PMihT8@zUtp@{=0wpdF8mj zYW(QJ-$V`-ve|u4^}1@^eRFe;_4P5M%OTI@y;(tkRe;(9)Paa7afmlk;))!2h`fWJZjC8##eo^pjp z=pr7dV|5LGvkczHIa7Xz?JZ+_wPgt8T0H}Otmrw;M2*z|VADJz@s1)rxYNcZqCI5* znl_L3r^b6V!H#-yYH=dHic&pxt$g!j!3(I-LcwW_E36ohqETa1Q_l|(Pkf9kuYwWk zi(aA~=WU#oH#B20{N5BRp3)tCsk7h;+k(%N@IBQOB#6CQB3O==c=)JdUGdvfF3*cQ zJ|4-xC}1F7|2)?x3Qv5z$hClnV&b)r^1mgfdu4-o^ig^IjIGhMWF4B0!Z?KNNk#N* z&}ov#;t!mP+mdXuca$w5GUIkL=v_||dXLIYBgMf-<#xr#N9A@4x#RT_l?#lJtV%Kr ziDHn<{;5c2AMH_O2@_|MCdwJ4yAfxj=b)K$n{MSiuFiGAS#(r%Vzj-`by0a+O1v2> z^i+&C8ncd84X8nnqwvHH>0phnuX+%3&&@rkmo{g;n#TsLDn@<;tf*d3(Q_P$zvC-~ zHp;tbY_+0TSBw}>*VCZtH~tC~Yrj{w%Iq?X8_%{;&jejOys`rIGg@lH$fLB021acs z#^X#+t#MxAn_pX!cBPjwJnV!Q8V&{V0)Nz@;Q-^ys*Z28nCefWu9zfx460KdMicP| zb-LwCAMjwqME|y>zEm^gJ(ha4rB35f|9pu+{~N`LHi7u~H>{IK+-jRoN?(*(YQNP! z;Nkd9NFFrpJjvR{rDBt2{vJ*pIZ-ymp!EPXe#ZBGxXOW$ohM z7B;M1X8N}+?Uy2JHumSS)T=FZWUJR-yNs*7@V&9(+l`_DIga#_?=!nT9aSDsq17{} zXB+B-r#e@rHZBzz9_;{Z+6=Ia#h|Qioq+<<%87n>BV!Hy$;bgCF)eGv>>kQPX zE7CQMnxpEmeTGe^QHoO6*^69{rQFG-j@X7sJIGGM3IhDc-2_dokR+ii=pG;=|~;Lwn5lCux?|UAM0O6fX=rIZeBQEJp3* zWW-hDnk@7OiAywcEH>E^Zp{7OgnRqYKg?W>x4Pfz{J;HZ@M(;0#vPl8^|7e8chru}0}qPD?7hqw_0+6=OV?MIVz15k*|&BD z-baVw*lnva<_P?=5|8){i;iS1=jg)6P51%v{W!7xkM%$LWSmQ2=wH`(|2>K~`retb ze~DLU{%3A$bgjaL^97nuqMWhgz$VT!jIYf{ zq;}(v>hf}7k4o@_=l{ICoUtwC_}g;4uJ`;OQV`jKuJNAjL-1eH8;T7lLD8?)p{}(1 z2;lr+o>N`_Z4U2)9}Ss^mT~y#9>jCHxLcd(#dEqcOfafEi791c;z=(?qN#DC7R`AP&*{ok;OI5_oGuAb^ihJ_yrG%SkLfJ!J+j^q{Xkg&Vk?8F(_Wl*wK39U2j;-x!KlenO z_Qx#kRnOM;GlW9?9z`Ipr)2vNWSz#R@0wWKli#C@QTz|cDdIM?r(4xt+3^rMCYXni zhH}<2#nkw!owEO9XndpgijLY7owYs9b5eoDO&KG{MCr*9S87i*J)#i-Eq8Xv`knz%^-sbgc2v{BGHwBH+N{IAINxssB` zr(#hb^U3xe%lOA+d*#2>o?P16elaHRQ*p-s`eqQLxcNI{*721eRib?_+IuYH56Je) z|J3-Zoq|sVAYS{oUGy%4@mhh!5%hqh#SlLv+VSV1_jg>3C z8wz){R94ll-C4f8W%;AEP5zp-Eh`FYAFbF~v9X}0WpnHD{Q4E5ZE?%8mimpwdA`lO z-<2B+mUq>+F3STC{`!I?wcCqVZ*SaK)3Ise>c>~MuFh{PukC7F(Xta`w1JQE_2uik z8VZ(m(|C;=D%*=Ic9zyJTA$ZazO6DkZuz#=-lApgW#!w}Nt*IaZR=Y%E?U3+rsxXt z$D&)lxi#3dc~KSg(OOP2Z{19~%Wp3&*tW)7ygGkV>$bA(MK#@)X~mzKU$HVtuo>Xzi-vw$*tJt?QfW zH`lb6Z1UFD`fHZ8lr*)MZrE1sEiPHLynJ>3=JK+xO|30mo6GT9w5n_Kib{WD>-ui! zuVPL9`o|krRJLr|Pyl<^ zO5I!NrU8CV;czWg_oYtPv-&segy(0mL|Z$1UmZ!9(DK1?CYX>zDLaND7BunI0|9Ilms&f)dF0^$6(6V) z?|IMbc-LqYr^I{Cw4mn^aa0SNVO)h2^f7+SaJdswKwJgAapx_N?YLwZc5X^83X=p^ z{x9^!*f$}Uf{7l)JAJ8~=sgms{5}kY#-m?#9r*9jJ{~>oZ{oopz7G5cfD=DwxE&Z% z^Gdr8|IZ`wBKm4+g4d)RG-kYU-;naUr3uvhY6d;^b6LVw+X4w!o77U-e~#>@TAEDh z`R^tFc)AoXYF<4n`FvTz)qHwQ_A?+2sNfp_PgjGYr!}Zk!Bc!We2c9AqX4sS;Y##* z7>CM9aba)aO7JTZZkKQ+-vsohe1}jIC4Q0xo+sf`09S1Z^hF7u3b=90xZ40Xri#ZT zoW{jzVzeBWcQmdMUMl6J{uOUTx>CZe{cM!*>6U(!zNjDLDN&E1e|pY|D$g0J7y<7R5E#X#q2c&_l^1d$N)_#5_;Z}KnY3Rqu|3?P6QQk~Be`s7|jLf?Z z{D_1TKgQMUISGHr0v|BY8}0j=gj@Nir|3{2ex`_9{=Qhk)ws$Rn%HFsSL;0!X?I6meRG>P(9N(%X!Nf1cC|G$J<5M_vJ4Mo#;-f_-*}>*Yh*XO z^PrCEn%kQ3GmrHh9khEV{dZg)`9=LM&^BdDN9Lld;oTk3nPxxr_G3+?%u zuW&Ei&!M?YUt*G4~&w{qqBRi(||7GFc>s`kdN77TxryFYG&mUeGrU1xh;BWFwq z1oi}_Y!PMK?z?<59^4+|cj~(i_Tkp_cb`=ZkCw$+FAR<#O zQ%_f9f;v^^9||d#X;o#k&nK8n3!-q3ygMV~m_n#{rRuYALn)U>0p!)cQ`|f!^D+8t zjpCgQ$I%?@?*XLojkz_objjfBp^BlX@zwQN*`E3~mTDEe8;#J|_zFsB;>oJR>NY{4 z_*L}Q`hS<>U$6Z~7KGwk^_LFetu`Ut%7CIUU#+y;W&2U`%l6M$8d&-F0UytP4(UIa lvuyt%|a<+rs@x$X%eqw{3tk^u>Ucx`4yOKOlnk3KEQ zvj@N<2 zxcB?((S7e#g8+MC!%)zDZ~gyYRsXB%|Es!n`^!sLEYmfOsiLu0S*)g@jAi!5?mhh0 z!%{)%W<50nwz{8ZyKVo6UTFL548{%(*|PiI#p|psd+xvBdJeCD#XXu%RJ3%xw(|#} z-;QHfKU}YeKB%&VuG!e3|7T-;M>^Oc&>rr|U`5}{_Vj;P%HIB!&JN8;?Ws9qo7^{K zn@D{f?n!4w|5x|)52Kx~+1{?h*dr&O8LT|Fe{lPyX9icCdS>vOy5}lrpT+CYoD<&; z_Z>eLe$4ru@Ui{J!z1%gg!|^74DVZYDtvL@sqo0YOW~*IzZZU}^<4Pa$@ju3Gam_! zY~Mfl^fS*4rp|vjG^gb|;q&v3h4)_iY4~a9TjA#87sGukE`hEn6gqHVaNF_ygL~&6 z3s;^x9&T$-zG8qT<{!^nE-WI({--={yw<&A%9adKK}!I21a* ze{k-JCkI2|`TYEEhxg8Z3w%Kjki8GDQ z0KP7S$IW~+G(z}g;B`BASauHW{sisz4ORfQH}OPx?=P>{A$g}g_i@#yR*Na zoY+&7{91O9?C7zvA_r`w={+^|@Tp0f=js{Tq&~uJ)>%>1{=f~g z|5Ls<0N;BRzIo60?rh)dxqIESbUFsQu*J2#s8GQ-pB{9?fhXVQabtV;Rp0X z=l`kXs@u-eF8r$G>d4DW`+vQL^`~NfoSDV04*yWU`Ud=ZD4Sh9Q#$F;P=j{p%-`zQ zhWD|4nnz3^-{ zl%Hf6c{MRK0(qc|{__*qp^MPT$jeC~ruD>?%fUQX=V1I~jrG&I@p07;Lj?N`V%C_k z4?K~sni6qE`;V(e>&w_5yIJ~NPl(1R`Wc$1pO29ee~fEs)31D7rEwo~WqPir%=3g& z3yVX}FO-DHHe?gB6ZQ8l^m-Qhb0K~Pl0E&~(f>DbU5!^I?t5_EkLz)~{x|MV;reeh zKJ4}Ga|t0E?5h0FHlAI57y5V2)I%=V=K}ne?DT8cfNXvqHb1k7^$)>*WC!Kb@2_F~ z;ZoMm^npjorpo@8zXe;tm*0aQUxLm44fc!q13Ps2Anac&QG%qUH+Ujfdwlr^T%YV2%zwX}`g8}ODiwf!%6%^Fv=jG)sDyYN1 zg2KE<@*iGYSCbthq zZ3%c=eZo3cfOmljf@#)3OGCixZ{m~AA87Av@9g%YIpNydIy${=oq2bG`!;y}t-e6R zU14UJhRb5LB?ro-lN-r@v~Stc;p@y}b#-lz*F`Za zm#(Z6n7TY{)tbE>JbFj*)Drp2U6qsFC^F3f(uUo#wJhQAL@Sv6DTj|A1>bDADR{=0zNaDHJe z5tLjBqi~$h_S19-LRGvD<3;sV z>nh4BiruBy|Kl7*6gng!(ueWdLLh%<5WZO2)#!E8=|Xdhx3jsu&E47F-qJD8jhXH9 zcKF=+ixw2lbFZVzf}$m?q^r55(cRJA(dlb-w{~@Ox*L64n%jKtwN+Is*3_+ATC=92 zYE|9R4QrRKDqC8{=2y8Fx#xQW4NdNjR-6Ys;&o$a;+4)<@cD;@*X`^{!0ihij65Na z?$gCRZkatJc7HR+Iku)$X5V6mUC{OsyXgBq8lAxdaWZr{o!MpGPL{r09Ie}EbVFTd zZd0Aks*HH*lj=5fYnvKj`U)H~)!8)roK$X8vMUw)Y>W@1xRPP34BS?Nw`r-&kM3P? z#A>|kF3`YW_|ISV>B7rwS!ktw5mAICTA3Ua!gPN4S;| z0)_ly^lW3Scc&I%zXFmBG?d1&x`n>@B*}5IV>=Qf`~WTm6!lKFfDY+Y0exzKrPAR3 z>dQ9vi)_}r&mLiy0U)Luoox5sL^~gMV|FS#=4LfJcA5K};kGqVbn@>ufXe zqW9PYKlh=6G83<}PIf3@x1$A%3>-*fLkn4M(01P|?T@mEejtgxagfalCfI&6<<0H9 zK9p@Q3hFk(Ge&sG7n~T$MlE47w2kp*1(WnB{fRV|9+;Rgk@_Dzn8eN!J>&FAL=Sv! z&_0pQIz|6P&!?H7NB_`=jl}U+f57?Ag?t8j@RKI(1%A*U(R2M}bZW#8>N?5iuMXRY z-=yFqSdrT|LE!~`m=WF;a84qAQ_n<_*kBqf`ZM&8I+yoN-k&oKHGe(&L;p|hWf8K6 z-2rb*M?BJ2YI$Oqtm839{_ zJTkIK;For5>@qQ+3IA9g(9@{Ow2wJIu7L1$+pnPy^sfhO%t?4s0@*h>lMf#m0RKa2 z_Ehp`ER_Du&=2q#!Q7d&Zont`m>22_*YTj!3Vj$G%;RLVAdQi54As)hWR-!!p@*Q&8NKSsLzeC zi?UvSNt+p*H;2ZXNBqG)ySp9p_H_~&;4^{u2>k3c-hU?Q z0-vny?sfr>MU=*G5_lKt0-vA@c{WRYw&Z`a#B0D4|9*k5UkLwt@IjJ~`#uMyjhuc3B_KgiBmde=Byv7v1ZU2q z_{aV6=hXjlQ~$j?r^ozh=m6HY1RMDqUldL)gnwbZkNOw+&-4TuZ#ws%gJjR?QU4lp zYDIx*2~mH=dN4CMZ5q)h#{6wsCe^vW?Lm7N+ADusMf#oQGWnYWc(<_U(diuTKwY#a zf3yQH{SE#N{mzW)SJ|5wAbY<<`lfpnsc#z3uKaVT6#D5DcsZX(>tCW}d+2{AsayF6 z>7RchMR<%?{=#BYf4w_1V*0Lcv*?@Vw?W^0zGiTJXGs0Q|1+ZcuFuAL;j-x4<%;RM ziuC1*&2K(G&|c~r{bvNBEeH@9=oRZP%@1xssjn=vywqQGJs2uxJ;wZ74*Q8B=AShll6_ATfn2|fg5$@N{rUKa7v;&hyr$wo?@lcJDt`G% zK0PL1juY`xd{%lN=Qm9+qWI&t!DO_4)BMqSywnMgFE{*|;_nsEA%4EB>NM&A>Rogr z3-AZJnNGW&#Nr9#AwL@}zod<=PclD1{)YK8pZHM)^ttUZ-;w{opYhQ# z`EySat*__7acOcBzbsI`A5L!;Oq1MQ2A4`?Hw#+ z&GU01r|0&hIRSkZ1*hq>+%nSN>A#@$Yg!_^9O3!R&~*4$a%R-Oh8AK!kdq1VQ9wUT z`wa*8pXBKNBr7LMe~$JWj%02x@*loGadY}Cq6a=VnCyu96Y|^ifIG{L-Xla0dll_+ z@bwYnr4c>+Dbwh0xxp;Av3=1lx0~i0r>FKon*-Jn{)q9ZebgU6Ok>Z6Ig&-rMAC@A zqF&I8_DPiQAfL7kJPBU;`j+NGf9`01kUu>zU7rH_hzs+1xGFV0kgZRp`eaNXjF0hh zvQX#q1O36indB~hnh%h-D43ZUjb~PVi2q)UcYS_n{X#$|0^#3ge?Lg`$v7Xd{+Rq7 zc+-3U-sJDV8|DM_lN&VqKm2_<*_Y=#;5Q>^m=BPb@KOK2s8K!z|3abo_!7Q^i2RGW zQ78SLU5Gl;1?rIGIZyjHM?y@$ks&aHq`vA!u(D^9SV=m|HSkd zJ`MOJ)Qxx-PCZn6!l$Ed#Ai$}<0qgV!$V%~uUsFHm+CHXAoOFye9j#sALo-x#Fz6` z=6LoJ^2c2)mHZq2G>*MlMgEjZE&OGEyND-WM2(7Jy%fpLBa@CN3~ zE(eYZP;6K)PGbCB!XEzm_s6kcQoMbgpYJLC+_F5_uk-bT{Ee^Ar%m>>=4YmS1oqn> zU`SA;{jBF1m%(4xF$cv%@HbBG$4vPS-=SOzT!xJ#f$V)36;VZDyhqdnH6*nVqBGq2Z} zuY`}T*JI`@e12Q<721qb5&p>c6UzRuC(a);Cj%RCVZNXG6!R7Ghq(F5j4|>R_lz6o zD_J~WamUYBAWxQ?;(Ih-G3Fbi^A*HXcUClC8HWBoc|MXw>no3!=+BamAYPjC5eLSb z;Re5-ARoyR3qHl?QTfrGn~#wH{D;g(=9%Kx=>0ck7btT7y+6q81mXDy&hN{+UKul9 z9Wli#!}&V==XT;%bpMU@&Ai`Uh|d?s+HddT;;8kB$7ecHqtAQu}{rEwA-e8~1_v73@{tNBLW9K~>5BsOs z`4!gx(fjeK({3E!rf`3s8b7`veoUFl_g`v1K03ac&VO=)Q>R$sTkQPDy1&Kzy@~zy z6k~iV3Qo`&KEM+DLz*9%6DGn!V*j0u{dd*``h3o?|DHzsMMJ!E(uO3q|8_dZ*ni_d zh05*3yQ`8nd$0|)} zmi={_j{QFS8VI;QWB++W{_uG?{6UCgxT2wwKBl-CG zA_*zhS35ZfZ$F$){nN)SvOg_${-1Yq@|w>N(%2WlkJ!(m1DyYqcYhSiPw@GHDL?rP z>X7PQxu=198n~x{dm6Z>flplnba;+XJAiLFu%2IB`~8PMihT9uzUtp@`n!MldF8mj zYW(Q@-$V`;vDtl3_quA`eRFe;_VqE=Z?AoBe__Wwb+dyr2lj_#r1sp;;5#{FD>#qs<8*>0Dpx5N>FW_J>?3I z(8WAZ$LbpXW*NMXbEfwi%x%;d`nnNDzCqM6et!@$gZ_y5hH|T%H$q zd_0nWQNTdF{&}uX6rT8ak!t}D#l&kL<$rTb_sRzG=%e!Z8C$Js$vQM0g>eYklZxos zpwlFe#UD5owwfj3T%GHJv*@Vk#Ath=>*DgZlz1~% z=&1y4G-e&G8c>5CN8yPZ(!pw7U-=;Bo|}76FKy0xHIEHgRgC-ySaH3cqUSggf5%q} zZIpM>*lIyo(Z~mcx46ZXSCFYkwU7JOKH$NIiT)i+eX(Z7do1;8OP$7}{`nGt{x^sdZ36M}Z&)XfxYahFl)gB%%zmqV z#A|aiyr1SP>R6-QOiyF=Va3Pm^Rwf|%H3FX<4o+8F^GGaa4Yij8~Rne_G*&Q)EcwB zX3|QO8)3%l7^IkXVOpg9MywWjB|5Su-_Nyi!jGgf-M#dZah!^vLVsu$F$*BPi& zSEOqiHAmHB`wW{-qZFsEwHLb{Pr13#C>I$^H}P^6;`CN>0XNt+=vYT610+^RBc$U| zS!|qK&*Z(9JdgS@T3?J{7@1i|c7w!#zZdJF5BQjAjs7gw$WLR;&n{}lc$y%FNTXXWGtP1QoK>;_hPtt6&JBW#fQ;xhxVB9Ptq)_yKY_mDP9`F6i&* z{Uzt$e3tilJKOub>(?}&j5{_F>tj)G&!`=n2OboQ*?X8V>Zw`#wyv)#!(N;5vv2JR zypImUvD;Q<%n|r!B_8n^79Gi2&e4UBoA3kT`*C9XAM1bg$vBt5(7&$n{(BT}^u04< z{}Qjz{LkE0>so~i=L7XW-@1A)%q#6v#FsJbjl zc{9)diTiAyO>Xw|i#acS_r-~=?0a*6^VYf{ zq;}(v>hf}7k4o@_=l{I2jIqt-_}g;4uJ`;OQV`jKuJNAjL-1eH8;T7lLD8?)p{}(1 zDB%2Ho>N`_Z4U2)9}Ss^mT>s!9>jCHxLcd(#dEqcOfafEg(+oY;z=(?qN#DC7R`AP&*{ok;OI5_oGuAb^ihJ_yrG%SkLfJ!J+j^q{Xkg&Vk?8F(_Wl*wK39U2j;-x!Klfyu z_D3!4RnOM;GlW9?9z`IpCuRE&WSz#R@0wWKli#C@QTz|cDdIM?r(4xt+3_GcCYXni zhH}O-#nkw!owEPqXndpgijLY7owYs9b5eoDO&KG{MCr*9S87i*J)#;3be!?Oz6r!AZvM`gb$sPVm1y6K_8!al1G2sH zKQ+E;r{Gfoh}Zs|IPGalSliQ&MIq|=VP=#5FU$6l{piK2|21g;U*pKH+FzC};ilaF z*73nL7YEK2Fhl7g15hD9YU8yBsq-B#SP*1Nv3YvYER{D#)`wcCqU zw|k3ASC+0TX=-R)*12gzO-qA+&DxfV9m^UDngCl?mK>vh1-{2 z8VZ(l(|C>RE89ydc9hjGT9?;SzO^zsZu!<#-r^4I9CC zefhE{8oG;`A#V$G^+bJXQJ!x@iNC(p2m7q6tZ!TW#7eTk@^#JL;x#Kv+E(Q?w61Ha z-&E6Hy3t!(>#teTQrgsBwtj21x1@CCvhr2=o647VZES7n+Ek9$qLp2nmRI^4Ti10% ze-*3q*FDj=ys~BE`U2RyCVykwn%bsK!d}}-Dz~8Tyv7ZcT~@nzOZ`u5Y%Zh+Jeo>f zYfE#WZo9YO;YaEV8XFq&3JMGJ3yU6ZXe@f9@R5SX`i8}i6gGP61YJY(7H@;^dOxwU zmjAX2{i+In?c{bUc{g}AdH$lp!oo#)d5byCGH*+VuTH#|*5U_ceBRc)U>;*TbMENJ zC+GZ;d*_$F1PkRa$me!ai?Q}Z4byf_*QTXr<3TR?{YZlqa}D?NaBtjWK1B;Im9?q& zd!Ew|+y6ZAzd62^`VHrcqGzIS*{{F^T@bw=r7u3o#=j`aJZgS z_ZGTofS*-3T+7sbnbY;0{tY|f`5*pDIQ7em0iSCKMzbw=qAxSxqrZx@!uiTL8sJ5- z(h^r002{@n)=qF9c{o7D2dcz- z-t$`CH5$b!@t!j+=y^mO)xu^NS0M#`j2|;x?t~N&S3z&wc?)DaE*XZMo05ydB*B&c z3w<&6O~|ESq6hI#Un(bhj|3{e4@05x=vQ6`{(H2KM^F2kc<@KA1OEZw#LsDN2gcOA z(yqh*3rM_(zFL~#btwmp8E@P-rMzxw0yV#yK~Mc$mT=X!K*H4~wM_P(Bm1eACR2L; zd&xhZF2#$QSI!3`hei_jf8*xR@g{ECFzC0xlj0sSf8A=E^PpJajON%$1NRa*jmQNpJJZrn2NGQf?g;!z2w zadDa$Eyv^?jcbILNja&1#T${XkZ@~18zg+Xr5~j)>c@CW)MMzMo^ztg^9Hz){@8Wk zR}65Yytz{ERz7D-xK-W(X&|e-Z%DYcpPxy%Ro-73`Z4nVkpXU$H&f0Z8rK*j^R5Fw zEaAkDaW#8h!XL802MqK^`@SyWR{rTJI#h_CDdLvD7fQGqSNUR-gsX8C{IG%EIPUWj zZXNfR5^fzgRnA))*EnBX5Q#1Aj z1KcQAlY}chDEbx&x61pHgj?k;lOacqn&_-ZPPQchuE4w|N8I410t|?+R~MTSL=h{3j=u;(^Tgbw~aiPxNz*>}Gc! z)KOh?TQh#iU!&ny`5gRfWHfe6joXRNy<1% zkSu8E=*)lkrnPa0?1gXMxN(0l^>#M!rZ%5^H!tR6hx+Z-JM@=b#RB^Y-}fMh$W+VJ z(-oPZPL=tGLds=YRT=H`2`1BmDBL6OPRlr^5Gr1&`YhZ~%H>f2dG+rQH_yv_j6PeV zcqhYgGza^80BL+7aM~@VJzp*q|qQS=;}Au09$W literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka4a690575a315b7d9738c20e7f242a36.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka4a690575a315b7d9738c20e7f242a36.o new file mode 100644 index 0000000000000000000000000000000000000000..b04e7aabb9eb820288853ee6c1248eca925254f0 GIT binary patch literal 49512 zcmeHw3s}_GweOw>!|)Ig5dj|ziV8lEhtK9UfQlLwgAb6Cb{Jq3BCk7u4u+mg#x(CH zXnLBYuj5T~TE}YQN&BVHS7?(jNz*hnemy<;dhThhNiEtuJS0uP#LWGz{h0ax{|sVI zd+&Ff>rTc!Yp=&%d+oLMUVG1fnA_HDTx&L&m?|dreHMXKD`QPRiM)66w@#J_N(b{- z4O(2k$#z)&jb7mG%VF%`8B6xj*YSG8lD*)MsL$f{C%gx1nxdup(yc!VevrUM->)_Y z->b9)FIm{ZzgXDO!=3CPXurg!uw~!Nb`HE>&R+e!nH|hY^jDp>q#r$FNh4WbVySG| zznh%{L*O%;EU&uIcYfjnr;gr#@2Q2T-@NbMQ~%TK90hG3UI&}cKQVmt<>SM9-hN{E z80!2@zZ$;f?OzTrMBRdV{ey1}Z+PO|@QtW9qK==tBKY&S?>lwN+51iy#=~u!QcZAoVw-s{ip6(d~A5bE60bM+g=$ydjBtn z&wb~u;q!am1kKXmJrCS_>geM8PX*EU`Cl9xZbqNyH@z~v9(^5MOf=U7&mFt>)V(ba zoXY>@@!@;F_{8u%NiPg9NPcDb+C+htDrO3!3YK>uq;I&ON8j-FI|&LE;m`!Iysty((Y}rrIHky7jpuT7Qn_pYK^r3zUzNKPq2z!WX{}U%l#!pDthYU&a>?PvDD(4ZcV| zs{C=#;E&3GFXxlz<_xr+D=&KW(ygZ8NA2d|rE+%gTy?(n;%-ZD#ASsa$6y_`1#kG{ z{zLD=w~stqI$(D?W-+TZIcv~g75sPSX#aQp13%j19L=|~}92)@9i)HK*GyH9c<`IP!bvX!BF1(cm{r2bz9o8a?ve(t+m9mVw}}oTE)oSRgx4 z)%=)g)LA)XJ@OD}3c&LMaECy9pmd-v^}uya>1cZfZFi!r6Eq*22abFn@P5DzfX@Pc z5^x*f8v*|Y@HYYP0(=eN=K+5e@aF)}0sJ)J2LbN@JO%K#0N(|8GhhqgLx67sycY1g z&Vi;o0q+BRJ>b^?djWq3^)A#~(O!V|uL7TJ}fP)|qw3hHU7UqU?v_4BBcQU45e z0_vZjjz#?g)E3lFqYgojCs2Qg`Y7rTP=5pUyQsf{`W@5{p#C%J0O~)X-ivxG?D1uc zV3#BX_xpH1 zOz)G9yFbSJi}ZfdapJS*Va<2=t1_Mbs^%wRE;%+0SzpZdr}jJ@Gg|k%z;$fh5$h4` z{f^Uz=d#Z}^T)cUVlI)6b>@W!8-KR=py#p02X_S)AKYEP_+Sq7Q#!67T}emiM|!?4 z_YSV-)6n55(joD<`zLt+1A3ou-2FV>|BT+pA9ufk_b<`=IIic=B!BAu+L%$_=VC^i zH^*Eez1YdabNBCno>fYxJm|ClIxT=s3!u}2u+2ypvK8q<_Cu`my^QN~C;AFdU-skf z2k`!0dLMh-{bjsAMDK0K-QU3buh9D#u20h}|Dm4ut)tCvTcHbLkK5?<+`4~+%|83g z7wU$s(oS=s(|qVOA3Dv4PV;qkdL6lx)=uHM+=2e8s6Xp*_vi4wmfl;AyLaJzJ-s&{ zcemmFE%e?b<`VmLoc-%nH^TNW;njSv^-}L4>nJk~Ue|OccK z0a@!&$H)1r@^X%THNnINvg4T8gKxL-eDIvb{Axbd!y8QSQS+AL>lqt$#IWOTbLDYI zj0yE-)EJL>6KczGJ^jYx>#d6ZMoGUx(yxy~zwY>YyP~g<^lK&knke+EkFQTu^yQMi zOwyN1dUkL@DSOog{r4;{8`zM-Mi(Mq+LK;7x)3?kIjr+xd=>C%5c!qJ5FSD_uL75p~PPw z1%C+mr4qk33jSNbuax-BQSeU#f33vth=Sh>yc2k7gRtJuqkXmH`&<;h-vGZ(;&(;C ze;N3VQRJR28;CchjPSW^i^BIY@V86+{wVnLa_~8;-89wf1dcK2>#zi!S5{{jlUguqu&J3Zb}>5{s7TOY5xk*PaRAD4Wf_Iew66b z$I?GR^ikTMCi-b(>3=}kJA2I zqMtsN{w&c)X+KZ&Gse=tP4rRPhlzgXSo-&gK1%yVqMtRE{^Rn2_}h8AZe56nf6!U4 z&YXKN&y@Wm6V7_H4?TlD?Y+vi$Z6Lgw_P2)xVs!VZ&~ott);(8v*p9s5p_t$Y z$W2E+bPkLnU;RTueEvn`uOGd$Ie6)tr2|9r$yQKH_Ji z=aJj}c}m&9$N#}b-$dT`)||3|OW!mNyp0_3o$|7QkKRcN{vlyQ{_78|30{1GK~E?2 zcaDx8fXv&R0~dF12#$Qn20m(E4_mAY4t?7a94aXrIA@uj^U?k!?2rujhk$7ZNSthXAhtTw)$L_s#O)xoM`+ zpEo9^Z9Y*GeK&@+2!bjcS$DUCrvig z;QzStr@FH{-7jh7u`8v|bq1+_qR%lod4AgzAH9!DxBdXWXM*qHEQWn^#ASj1S>b;% z!H^~CRXb$GFDeZtE-ectU0crO{^Lu|(GE-B&KNnu;f($lIXrD-2;(|3zaZb9yE+(;wmfQIxIAdjT@$new~*jxr}o=( z*K)j_^QUp0jy}0Gc;vtouIrKVDgAMdir@m!=Yu9#p4uPxvUBw4fz%O)$@$|rv=eXe zz~qr5<&z3F{kFMDofFb(y5{(ei)hI%f4e^bzkdJcbI6*QxU&wzewko>0@ucmq1f|F4f^7q?= z$*2qX`yIhas2B0~)xiYRWXGm?H8jrAe7sKoVtVk*U0K059?T4$eK;fd-ak(Z9zO87 zk;h_t^S_n+`QYQ0n@9eo+|~c>XK(KRc)EGu&)>-ozV+Q%!FPT*Gx%T6&IrE$@9Du0 z5Z5D!)6uP_fs5Z}qaT)=20r>{#3|x_Xg=1B1k@!Izonx?4^#Y>4GhfzjP+>fLBJJ& zrvS#8cIYm^)qpL4apoPm4R8bCcNyjg^`BAWY&_J7_I-f=1Mm%i_b~YKmj0n%F!(9z zXHg&6GBV^sd-+ZML*FI%rjem)f~)$6zDaP^$WSH0Tl=xD0Ny$>R6+2z{-FZ|ZyOo9 z0kCEJ$Pm`|BhPN{M?V8cEIUSqN&p`~y@c@9{X<*PUXJ=k;;SARnuGT8ntt#Bt{EAc zPVmnDp@jtR92uHIaBV-@0oRTUC6Jtd7#WI1{p>&V4_U;yx9OBv7ZB^XM_hy0UyVJe z9P2?D)`e26&(G`Dq^s-2)%D`)dU18Vm}tE~e#A>1{zV?dU-ZwGx3>Sl#8OzwT^!BK z%w(buOdh#Y!1V7ryzC&}gW8E#EMAjY3W*t0_ISK)o|aCw1(;mInvsc_6H%MZ%rePh zGFwc{nt=3N7E2Pt-3z--d~R5(5N7rhp`#@8e?h*vP1NQ+%;sb(SqwDFBFci!rp_*J zm%ASCRNd1mU|zdAo7x(Ete~yE%Tv&{x1eiJbK9==2;9&u;k^anHx2Er9qukx(B<)U z6^Nm^8oIk$+?}1S))H5%&$Zjry4NK!+dOTZ?cPo9j?R6}`}ena!thoBcNP_|tap{T z8oPQrJPllk6jtx<^tc*3TYzY8+r7QHt!t&LtJUq=)$VO|cZKDwc6Yhg@}{z#+t#>> zUG?ogS7UQ433kydOli=-RfutlaeWGg(b?qodKy1H1Mzw~J>I<-5oXEjY57!m8^+=O zR1BoiQ{TPYwae>n^@!OqHn zU#G|;WCC*e>S`KtH4XWDOAE)key+=f92BX%yTw)L0@Tpm=yr9s78UwjT@AZEZH3ER zySv@qMz6cM#npwq&ehP~;;Q!`dv!H>u>Z9;cp6!@q`M1p&{215Tc@$JtEtofb0-enQcH=gvEj;7r@#NRm)cmiwt89%+IQ{h^mG+6 zm#b|rB1EMBTQ{xQ8%hT#;?e6SyG=7M2m-Xr3Tz0n(p2Pe)csBFt;2hSWgB`3%2WPW79XyNG=-`=biw>T_ z*6HBs%&CL3SRvrjuzfR`LkDND6dgQ`mJ|(rI=f)j!c*C49h}Ba>fkBtMZl5yPG$W% z{A6}S2dA*ZI{IYRr^6>PzYd=Y>N(#XX^k*^oe6mz!875vz0nHmKEyg zZOj2UV!s%cqJyy(>tGA>o7e@DS@}EG3LR`>ex}9qAR+onE00N2DJEbGm~KKnJ;cl# zapmr^n5;#K%(0BII=uSuvgXeSuer1?8jCe=23d}xg`I$sDG2hx66}3TAp;ARd4r!> zS6wij#v&FXB?j>_*`b<%QzVShalBo^7(ItGBn<0t*dbw9kHdKqc1pNV!W~#(s6Ub) zZ%Sf)*iH#1c?2I$(BlWOjOg&G0w#TleoLBwNk4)I)3x|4wk1Q0&tlFjEk22zn6AYq zF~>{+(|AZ;3X&7*pT=XO{uWApwAoCltZ%u1)p$-ySdAxTg&w~}!V2FfVKttU5?14J ztd#w6eX>}Ggatl}ozUTv*aaOvi4`KZA^p{OEGuJ~bw)08!XFMJWpN}ihVQ7b!+CH4 zyiU^#e$s()SW{qp_yhjuFY7Eg5{e~r#A1!NGq7T{!~5m3eGfh+YQ<|eaiCumsy~+X zC0TzY>!pH>T#WBKY7c&nm{3%^$^}-7=5V_N!=GX)dewe-9D0TSTi1bGO(FJsNTrSV zOft*tnJT)-K~<2%p6!vjqPY0*k3pH@2Z(fM@_NnPG^qFk-sBc<`OdgkxK!354C3@LB znxKv?!xqT1KI)5^M)^$0h7mx1qL0lZn|5r$9{!tTR*`u#AS!+fJD<#s?X!hgGr)Mz z&!+KMxu0+n@CSUz#Z`JraBmVf4e(t4Ko4K?09pZ0W)J()Y<@s$d=LAwY$PxOr`Y@& zxX)*U^-@^q53t9JG`?6HmXEOhkNRvj8m}4;`on5Lh5Ez8VLkZRQ4dRP1^)R?f#v^p zB(bNuLv|YlstNknI|NJ>iV8Gm7OISsI z7rjRj_Fv-T;U!?$H=~Cm#XqlMzV6v58w7p87pv^g{ekAosaU{Y)MeoT%bf(U*%t#< zQ0PeYMl$@-7EkWV{Sp59dbgG5zYuC?xk=32E8RI6F#N5eJ5BI2e*$}x{K2Z}Kbd_N z`eK+U(C>XAkK~Up?@c?cA)(L#E-%w-rj(QBm-Mq|dM)Jc0*3y%KE@mbvj<{H|NDDF zAu~JxYY88p6aDdm{x_n3 zzXaJ?vY$0#zf;NV?l|;^gAc}RXJ;*lCu_*x3I^&Q@voat$m{c^+1UwkoP0ujt{{1&8nlqx_2QL_PoKmGXlBL9&l1=4kR} z-m2)K1?JlbS7bUMh>GZ6XV2dzvrYFXeu017ml97?8!}`4gZ{_%qDS-raH^;@dkP)8Q@qQ{HV`r)vkB2XWSCV>u|!J&y&4pSp6FLeZEA#z^V0!)*nBH z%3;v=ce6ZhVLygBG?~MgUUCAy-VmOjiZn2dPvI-Nw?*Ow^LNM>!h;u(Kj`P*iVwE!w5dp zfX_+_j}P(#z0(c&SqOS9f3^YdNb>9OIT1YKk?cV^gcyGehmnd%IF`c*PYK6ym@n95 zzc&&%JWJuC|{5)see<~5LFjg5cpkTe~^TpZOpk_S291R}v-8+ltJfN2T z<cIDLF;@d!RR61)Q9(Ogn8a za=uIt`xfN+(L-eZrmFM1q|dVYX}J{hZ>Ic#IkoiDb^Z7GQta%h2)@rZWg7QC&i^3B zKbd8sKNLCsRF1*V-2oH|wMI&t%Lu0zPC1Wt>3ne`0EQ%NdN% zHwAAa^A+fg*UxV+)D!G-{5d*%2YP2S9{*hbcGzo~c-K9$f{++#K-(^D<%`w7iwNQme2(dV0< ztY4odVd>yS;*o#e9qF&nm!9eeT037_`athA{Hc}*|HE`%gTV!THqLw2NdCm_KPNe| z9?*O&wX>%cJ;5uIBl{8jt#_GDzd-MD-T1LyX7Gy-=^uT*m3H_>Sl>S1!eo8?tTM>E zMyLOyzRyh8=8Jqk?T^R{^ky3Pvvm3(pR}__BJ_Q}l4QNV&ouC78~UF$N8i80&VCS) z7wF9~@Xt2%pKE7_we)ih^z#k<&0DDL5AkK=2V}p9w?1Ey{(L=^`UAEuCHIWLwv~F=nxThn zk$j%7FII;EpR0$hd3xBkNDo^J6&z)Ks>uJET>r&-a|g~}_LYO=&A4wV7_%N5^RGvJ zaUt?=dm8S0PGY~Le2MZ0$_H@&^pe*`y`!M{x7j#9Ot#Sl06#w<9xJ-@>~lFD`{Vmd zfVaklZj0apzEq_DGD848fg+3DJ!vO9pwEaJf^+~&);XWyq?4gl|-66t%$QQc+ z*pO{R$}_CTy7My5SAW9&l~w4A@t8QAx(*%C3`sbBiyqFZ(ZR8_J>d>3j_A*IvJ6^} z@dCfh-)Mge=WoBZ=McHfSB(3OPCss6Tpv2sc`Vu&tK=8}(k9)*m5I`0>?_)mVwe!>Gi6XU74&i)G8;Zgo@+Dz8)y$u7rh(pSr3*zxZ}gW#zErrI z7C%MjkA1$W_Kd;Ecxe99>o7pgd}lRD81YQ_86A4q>eInV(>A%JUTEf9o{aALg$qwWAaU%e$bcVxlhG2 z?pJm3%I}x7@k;AM%w*c%)O;WwK4H9O7~(b45Pw+{#H%sB{}06LVX~iCUupg_jz}2y zy#!~S(8Jb#9ju903$53hcpZD+WSoyN@v4na5r2{WIhW$EPcz?ti}*`3#9!QG2ZpES zJK&qD^Z!pCe`#3mBkSkb{Wrr9e{uN0OA~)udl=(Q?YDz+KGc5uqJqWzrJq!A44==8 z*AyH}a2(J7c)ZE`BNbo#{%Ezv-pMBYekS507vhw?4*3!c@uAIsC|{!CM(lCOm!^r2 z2eDpHHN;1{CO#^`A7|LljQeFg4%k}xmyZ{^-_hn*#`V_duM`idr`xhbI0OE#oxiwDz5Hoh{xm5+;LFs{pAXB0Dzu-|=dJ7xVG_XbKV3eHe9EEiFW}R! zw>S^m)+>zP)qpf2Pk5K&C;u*FcQh#&^X1Uydxw0pr|I+09G!f+uh+&$ zpRd4Hr|A#;vvu|}+;1H6-5L@+t#6xALw0b;9sWW$LCR+`ctn00iGSRuY4Ty*2jhIC^~XT( zT>bcawf8l&-{czC8=XI5{x#>DO6Z5ryL9(Q^ThYPd45Fe=A&DZIJ z^#B`QWITs_vr;4eM)ENKk3{TYoG;vG+kg(+E6}^h(BI-neF(R+MC@?}`9$wLJWb6D_CJ!R_pd9O zpTkPu$bPEgsn0jpc0!{s`GdAU@^36B;rXNZ)6NggrKZ6Q?hURD9W$kaAzsuYALCDkkzg%xM_mw;z-~f^J zVYcnXus*c@=COHzQTTjDi`KY-`Vui^CC>Ze#{@hvd1#pAu=+E4X^EJWz`8fW3n#D=5ne^x5<9~ByD#6f~==uF1 z(PMne@KplmATbDhx^Jp^n+@ZmCg9V3X&mOg62I3XaC9s5I^O@69 zNS2UqU55KQhYrq2D~!PG$De0uWY16{KE}rLbb!q?J`J2spVw&M43f`b+;3O(ID|c< zy>!nf3ICDTqacI*#yo%4eL?3>M*9YC%?+_S_ySET`0;#KJRmW@7%%3V9;gVjKcpr1j{>!R2%@aJV&;8wU|3nWtW<4Np}9vp{gf@sh~1kV8cYXRpa z15SfLiEkn}1F*!Szq!6#iy5rBeE#|8(~RJb`uA}kUui`=GMd|t`Vs!TF2Q^}^>{uP z&+ih<$5VeX4AXJBqy8g;ncbTclLHvV%8)PEy$-8`g9H`le3K*JF6}z&kaL#~!z5veoXj*fXem-v1Kd?Q!CMLyZ5< zRV;TtPN>$Yc(5MwXufDXVt#2nQ$tdow1*{4t{@z~b3Mz=VRbrwgun6!@Y|;f4@D#7 z+r8=WV*g=eKj=3tVH!{<5{`?&+%qVBY))u`d(jV5ApRw&R33i_8hIhr2De?d~lsXpX0Sq z0ubg;_f5CceDN^=&*QB_&KF>^H~Ciw#S88e@943cS;wG^S2Lse$zp?H{p{WuiNfE> zUfHnc%t^xE2_}2aN)rCfVMg;aom~j?Q-3rbIltEHXAA$2*k?va%IEeWdK!<6Z}gXm z2TrN@(O)HC>m=d-1ShjQVb9rkph4M_`04XP>a=la%pazw@zb|(mAwh38#1+C+P&svvX8LG z5^g^Wb4Y#@_YYdSBlff7fgoP2?70WoojX~s6`zm_{VcF=++^v0+&+jOJa8lFx&0ER z(E5ua=Bt_I#<0_pKZf^5TXTnIy(ZU>tk-O^SP%SCz9mJhPXuREyz_XW4rsn|ym29I zJmh%e`TnNx@SiCnUWtDR=o9c$2(oZ~b2jZayl~iJ*RDVCr&JOD#9s-1^>`aNIK4GN z>mS$8Hit+apMUrR{3TxRFNXOoqmRBL{zCq3#`o=ny-n~J&HQtJ80$aCkOV!RZzKPJ zH6t+QF9zQ?8emt&c;G)G|E1@{DE|!`SLAob?@!#Jdp=_p zzE2TDk5h;*3B0jH|NRNb*X5UZKI4uaONc+9LeF;~AE5kFK99lk!%QFJ{?GGUsR$ccT0q>lL2Q;KlR+&qp$Q!rsR7vsAB~ z{}SZ4nBO0JW90LYNqC+!nLXDT-rX2$Nn(e*a{fz@kK=g+bFYiZ-9c!odvO0f9; z2w=#o=uz2&7{3|$W8rcLBzwcYt9xGN!^darINZTrJ!(7=dirkWc=S4dfj#xlGt&K! z=6POq{z@5?pXZ~$ZteLiWUA=-9y>|ThvScLlRi=TGbBIkr{}*7`oB3bzv@3Kzk~hy zO8aj#_J0HVubG%1`#FDpCfCT1?Edu(4Su_ zdLIAe@|1quK3aKXpZCU>r~Kox_Nim1#~YvGSN-Qk^RHtQ^E;${qt5S$j{oK!k-zc3 z-~oT7@2fEs{{1W1drQyP%+GOspXL5+q^I?d7m7zt->0Lm==q4zcT;&h3>wzwzXMnd z?&`l+EpXKWS1oYW0#_|?)dE*7aMc3;zgU3ozHq)D{3oQjL$~3|;GNGuHL&5s5W1^Yd?=abhtg$3F~x_|89u+Q0u{==Y~T2<^V+ng9IH z|GahI2g%HoV2vN)C+t$5sB7RD6OQ_(1Y7(?j=$Z!NC%gi>88MxkOVxPzz>){gOht{ z(yVxMVtmp?{;7r~#xxQz;V(Dv;ujTumO$YT*tB?Y(SeWF!gz~Si}&-35XN3LnWisC zuUjGSfQgv~L)5E!-DEOP|D5e2DYnJkFG4Uwx5@F_=we`~fHp9BY6rdN^R9 z4;b-QJ-ykchXY2~*q`03iG~C5rV%)#AsR}jpN}_B*Ue}w&8WEbh7kraD|52uUBssb z_yc1AoMsjT)m zoetyMAUq5DISAW8gTIUppV$Vm&R7iR_KMrOiU$)6^`RY1n29Fi~t!6n- z+AYAW#ao;fhM*r8FUL79ev)$%3rrfHKgo$dv6ZCTf^_}MSQ}Tq@PqOBWvqL0CpZ&WAYuIe6P<}Hkf`Gqu{M4@(t1{!m}o7_OSEZxfieZFS{s#DY4DUO zgz+l!Nk-WM{Kc?_M)+PN7Rwrz78_rthDF#XzF_9EZG$26W#X0Mf&6!jT^%8G8a)SMczE9yjg20y|pZ{T5CkHmJT1b*5AfbIvxM-Sn6_fHNKS& zOyEn1UTO0G)2Wb6U=t%$2@4(;CVk0 zebA3T!5pXbi3#fy>1Qw@KzOc4ViY_}G{+MjdYC6g!8;Spe7vMb1kd$|W;2~`%r1Cg zD|1{l*>-MAu3I#`-!5!Oy2VG~AGC8Dl8wv>Vf-i74SAcI$B5ln+~sw1#$8@FXLQ}7 z*^GQ`Ji7&=+a;{qIChEX7O{(NY~*{~w<7p(ENFfI6YGX?lfTfMM%7Jgw@BX+-TV=| z1tRz;dja2VB0H4Rf>SL&8JwoGUewGw4=cgg)ual$;V$emD!*4W;avs0?`mu0oc~is z=lZh5GF$1Yw0L^a`|xX})mEl?(dn9N6l5qcx|Mb;wZl-Gd3F+>xURxG3!W`ddlO+$ z0BZBWRbVuynA3#6n(U?#a$V)Ul7-7F17KoCdGwcy%`20N@h4+=x@V^EE}Bh);fFp7 z1>T{@htEE&PT*7Yc#Z9%%J+wDIT%%bWCy2hw(V7M$tcvrauf$c!PeC{tpYh zK|dDO?vlfKm8$_$d6kvu6~3 z9eyHy2lU&5zkw6&_cKhF{r!xkn5k&xbhT);A|GMaIyi~^5sktfVRGv4S_#p6Di&t? ztA0nLL&{VAZwdE?=Os}7*8QE9@(XLk8>fV+`fGlBV$APK)H>L`U!VQ>zkWTfm3?o) zyC2k8{$>86oWh^adf~z3#Pu!lOVHod-=5&szxvw~{kq>~xcb`@e*8Jc$Zu0{C%MAk zo+!k1aR*+9@w$@Vp6J7NSBFnc4%_hcFI=WLD;YbD*GVLE<_&)S+Y@!zLHPq*gU|$j zvcWH5+Jy){EMZ!;2_BR%?P3IaZSL-d1qn4VzLkKn!>Ek28->U&8mJPb%1R)ukbakI|=oD8}~x1;2{cuJ%-- zr_W+lxt3Bxl? z@3BSn>5w^z`aL#9NO?LH@q27z^`zfpTYNau?%>png@2Dt!b*+=|JxN!uXHV)45x$| z!bd&PAl)HMOCJVC<9|W&ZI4`Pa~>l28Ykt{A0uI!VzfK?C-FNCYT(b_^WqQ{_iQqWk$=6wcSy2u^)<;K ziEn{2^3!s1Czh}DF|98~{=3lt5o7vKj zBJtC@?#D}8hQLfR%e9U4zD71sr?T+%Sp!outyjcP@^wY>*XbG!P6piLkVpEL8R%61 z`1zG^i8g;7SaQ?F{K<$D`ohW@une&xT!PfLQ4e<|#9ZO5%o)w|2ReY&h@$D|%x}#!uL#Zg`o63q?x*JN$T6UIf-F{1H z%Qkm)WB1OQs-lM0>g{`$-PG-Uv4E~{!UUB9og zrbvU^)74Smx)pp?8%uXqZ{OarwW4M1joWwC-qe0eY16LSiXE*xYj$)tZs5EXHN`u+ z8rD@UYuQkVzPonqtX|e#UsBc7Skbb#z5_oWT)ercvBKBXP_n6eXUUGj+O5k9JvC+M z!vp%-?X}g*_Scu(Nc|Pi@U6>eND-ZW_RhecMg@H_<$Ut=L>tTd}5lXKPD$Z3SNNi`sP?IvQJd^uQw@#bH}h3Nb8bFTVc*U-9ZU+s=nw=_&N z?zRnk8!C28I6q}2f^KKq)(OT_+f$GAtfr)50=gDSSGKplxMu>|4oSQDmYvnUmT~MJ zwv!w?>{>LDP1iK6Lu`!WZBd`^^mLu|#pA4BVY(*j*VkCRwwu=fXt59;ZyUy2Qnq(I-wD%|5FPnN^#o(t zA^IN27sK+nox4gVuyX~sbIsZ-qpQABd1d4Iike^9Z`Bpi)vmiTd9@Y3E0b5zbY*nw zu1r7V99JZ-q4>)5Ybc&jzx6!UD%Q5u){N(?8?-Uf(lFlMy&8E_VJ&=qZQ-W2)eybG5cKdtH0nOWn&? z6)sz`%w1BntbRq~sud+G8;T1(D|QtxEq0eIcL}1r);wfbDU zJ*|6P+dOTZ?cUw3p4Rr}U7cE`tDq03@it*nzZ?{MMv)0|x z>2ZPH&{F6U{}s7bx{Bn0Z=Joz<8AY_L>q&|h@ot9cXaM+-oL-a6Na}6xU;BuWxcD! z)!5b3;c4K8kOry8Ps+O*J6nKgZri=RxvguZtE<)R+STrDb$5m3taf*~*Yc*ao!i!6 zDD~|=S7UQ433kydOli=-RXF=Ij$1!ZN06cdF(P%iG3~FyrnMy z{UggF)Y{KSAE5&bm2HVLoe!D6Wc?@Gm+cQHek1AYLNd`easu(s{S(mBx%5jK`V{$8 zJnKlDSr?Wc|48DaNnc6%YU%Kj^4Y$w1NqJ9qrN-yroyu zF0Z6{%Msa5{Yq7Ni@L9;ezon}U^m%OD32eDOkZw4Va?Y1(n0 z8ZJjP?OOUDjnl6F_!Y_5j&~{Mor+POVuq z70K6*ccrFXyEktcr`;-@hU93k{IAutYp?o_Y1*~)Z)n=J^w&!NHp=hPv}@^4YTC8( z?b4Y@j+Q=Awv#=zt9Pz!UubAwC!OAC|826}X#X17PW_6^LHNN+86*@(>TaAcH_LXC zFHOQ3>SVhy9{sYNYMf{ zDyrSdA$$%fYt^pyfkzDOYQ2BW(C(KVq@X`4N{(8O3k~gRJy!SMie9b99R_+8tVay( zYCUEm>F#vZH@CUHJq&qyqkE&fyRD(=MxLmz@zLKJF7SF<+yz}8Ul+U56AKA)HMcc) zG5)Tx+1=u4=yKuQ;_-I5I=dShJe{4apdSBxB9U#~twPymPbZuB;3N+f&278duYh{5 z$J^Q5-X_h^THrID|d8k#; zgHhMo=w9vF+uY#MwF}F7nq0fQ?pDtgsJILLbwNj08_{pvxMS05VcJdYjomHi{t8c& zd{$c8-HooUc2^@;ObiI>2|WcU#-be|&Q1}6@L6e1TSI%JXIpcNr}2~Pz;YX}VCZh| z?oL-H4t=bkv&-A%u1DR|%4=_XBTjX!fd4C;Sj^r=jF1j|aA8hT(9qdcwET)S{QEz5 zU%5a09bPJk=V3T!D~I z`Uj;(n_K`^`yxl;pZR!EO6J5%Tl&8%pp`OTqcolNRh0u|NaFKor2geLE?eTUHHD>x*(Zp7?g@Cel>s3pkrg-#4_?E R{ma6CNQ2Z!!pq42zX9k~ER+BM literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka7d5145bbf7eda9f6df50f622e00dff3.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka7d5145bbf7eda9f6df50f622e00dff3.o new file mode 100644 index 0000000000000000000000000000000000000000..4420acb7d6e548abc8d8cc8b42701ef928a30426 GIT binary patch literal 65848 zcmeIb3wTu3)i=K9lF8&g8A43BWfB4;0YVZ81Vx-AKmu}!fZRUY$>aiwCc46|pr;p!r*Xj8v2w;vn>3RSMNT3<4bp+PF}b) z+e@f99jF^&qvPx<=K(&J+F)$fA|&fm50Ok-h2AS6Zf3HbK%1y>z+F?(%SLd z$nkrAK636GuZ*1E`!aZzgzvofuG7aC-g7#Pvd{nQ;gMF9d49ulBWqFC@rA^5RruV) zcb&eg?cURcKR+>Y*FQZna%bw(BSmS?jhwst)sdo~Q(0Gr3qdc!uXW?Y(El4Fg>BD` z-1zWwBjK0-GIH0w#Irnn?(Ta|9|zB!#ort$TJyD$gh!qoNvwHxq!l`!-}@4HD#B!| zb&veh={pM^9x1v9Wfea=vaaA~BR?5_W#q2o!y^fWE5hflz2kJziGMnM?pl;p{I!vD zDZdz5=Xe_WygYJ#{!8GwI=t5LImo&5^tro_j}#?8G7^6F=g{jp)bEv%q7AQsXJvTp zegAYie9v)Cqx1wz&gj}l?%Z7X@;3qrx8)2ZI6|8fev}iWcJv4%D`z{jRq zyklo5$?uYw)^5P1K$kX2>cT8H-S5VuLb@!@K=F%178LF6!5gp99_q zYy&<5d@Jx8;Mcu_Eq4Il1iTXX1>gYiH*nsK^A@BRBmG~1Hv?Y_JQw(G;HALTz?s0e zR6%dsAoLk)c^h~c&Trzp80Qh37vcOW&hv3TkMlg7U&47d&cDGq2j^eoJRRrLIA4MD zb2v}I`KLIi;rtZNNjU!)=Xjidh_emnAK)B;9*^MsH=MtQ^IJH73Fp^w{%4$D!}(sE z|AKQL&VR;vKh9fVk1wJg^Kssf^UqMPZ8$fAo(H-Q=UF&+;=I%{7CFd%c?)7>y|wso zleP44^VyQayMJ7Ac+Xc#4p$*PGMx?n5o38|3LA{r*x=ui*dWDMBm>spZ)RG zeyi(*|5aR{r|Yy6{x@+wLf5G${1|0)-_@-4nEe>Wf6tktbJ(XI|5N?9#g$-H~o0w;pVR_JiNPa;o&_E z3lHZ)KV8QwNmtSl`jMV5i1CB#`90|HZPFq6g#U-Q{sCPlo$x<}>mSo~;tBtAxc(_! zCvZK7Q$y~XcEybaKOHyLx-sru(u#O>%4tXE5*RQd@>q}t&)6dw} zbC^FSVD8x3w`%YjkyeO2c1ojiAagCwi3y?21-ZxnCCS1Da}t;~7T@OJ^TKmB>n{rt z7q79vPpx$))-pEciDM`H)=eioaTc66;*2_3H{fhLq4KXkvDU8hUn}_63I4S)_-jwB zb?W>zf`5(RUmb&g)rqyqI)Am`uM+%~f}b5Os${?PLI1r=s|MG(*w}o`oAzc_j?Kqh z>Kx*|LEj8|hM>=lLEjB}mY^4pr&m~V21wqz8EnjuIjeGPodynp40;FU7xZt(pg%?Vr)c?qAA`QXaxC#S(9M3+hjdHE z`1E^;KSugL6aUol{9hve80lXl{><_Gj}U*1^dAua732ATNc=I$ zCI0O3{Lc}8jP%pQKW#k!uZcfK`frGT`gs1Ah(AX9dE%cjp8r+ikC8q?{4>Y%ze)Tt z(%&ZjS>yRHRu3lL#?zI!kO=>vHDA3o|8RjN=TQsRd^8q4j&bdcO=~c>U5z>Js_@%; zsxkMi3cq_xW%%6er`zQ9TI8S&b7Lpy3E)cve-iQ~V@{kBK8JbVFy^R#zLgEWbQK$W z9rMpOk}3!Piuvf92fbr|>ckx3^Ks#~Fi#!*n|E*w^VUBmB^JJo`RqHdZ4AHr)yl!) zc?pH*e&HP(#@y}VZ@puGe8D?*_GguYFJSKV;tQ37f5M#XPj4|?Tad=aUV6Q9@TJ!+ zgJ*wc8T>QmZx^Pc{0JNSug1#3k-I5B8#|AA-e0Cv4PN{$8+#dZz*lBh4ZizT%iyb+ zFTPe?HTceJso_5+tt))t^Q*&eKh2=07y5h0#tuT}t=_@6_pA$#{*4X3)43KlsSOW* z-4-5RTs3&kRyFw6<7{lycWwC1gKX@t-Il>Ox33Sskz^Ts{rha}jZGWEuT@)7js;~} z2VYLJKo9HS`FP9N$lau)WpH>I#?a4u$1oNSUYHJi1sfa2JaTv*(!U8iqyhg2@PB}o z2K-Ind9d?*;syRZ@G{`V*1>aMtqz~N0&9q0u(5Nq*A|}pJa}HfeCVK!_Imifkdt9; zf^j?n<2SYI#Z8ZgDGh6x;!E-;U}gJIP=$Vx|8S6;5r@u=92KSgHyP!$Ev3cBzS7V zMc^+4Pq^AWknpT`?D#?VsK?@cECK11H+(RC^jLK|$Z7@39U#^Vf0Rh8s2d@FGFmPPA*I`D;Y}dB_i6=lPcr&w7Jr z1>|2va_3R{hVXg#=`^HeBdr4GWjH(YH-Y z-8%S}Z{&nu`Od8HYyUAb{9jMZ2*3HCnc=t4uSd~O$F^7o-~Kuq`&+eT@SQKCpQ7In z&qLfu!g(?E-^#J!`>Fp{4Gzx+Mm!q654Z++3NY5U!=D4*4r~L)`gizN;6~ur8QKWv zzu=5@@^ClOZvy@;@HN1D8GN~JVEAVYev0!GI3KJV9rhu;`nrMP?-0Ijba*@A%>%<< zCA@ibcoX3*1BffYTSkX#2yY!2K1g`$=9QN;*)qW zS-imfh#&R%7xN(gqkrmNx&8+h=3=Lw;$&uJ77N`#*c*5c;`^In|2_{HOw&RisB z+_ASg(9zu1&FVnOC#n@QF>5l;Rx7im*eq6?h1ruZO&5nPm0=Ht-KyP3tkels_Crla z!RGxx+QrZhwX=0Eb9h+=i-Shfh_kr6rMoB4<8Q!~e(r78FhBdcTRNJ8thl4Ir@6Re ze{s*=){fnsQQWvs;QhtMMPp}sm%oP<_cRB4inXfw8u#_I`MbM)?TdZwLEoO{_WeFV z+1lLE-5J>6@9Ms(_2!$~nhm@~!`-E26%D?{zNVhuuI5HAL}|Hq;tP0bDa_V{)O{O!$J>zEV##Yzgg zS);$LZBM}8)xtYnSD>?}lX^Tyjh!9cJ^qfKl8=G=Hu}5Tn*)s>i)Kc%@w%AXk|&DL z$s15k_!zXIULDnEdFsce3+rvmD|D#_7K4f$i}ifn+B`zcL?%9cG7b4;8uH(5S~%YI z^L;+dK{1v0xA{tZK#lvF{J!q?(vqOBr*ThnM@hMF&pv;kDd2By^Yvh?^EGz1`5Kxr zd-XLnWBluEY;N*ls_hFj_w2)xpvl+J>*MK67G8RJ!uLTDi1hbB4hq`W-qCF??31a} zM{dQTgj$-|niDR6>h#}#?WGSaTiTo3i#vDk?r!cWVLo5Se)JG+`oCqv>J2`Pf==ge zYxQ@>V9>i?mp{8&Fm|Xpyjn!OxK7ckCN%2YdWZ$P+zFPT#U01 z#by%41*>bv5lUluJnsyV)*{Y&xq4esGRc3x%X?mwXZ8U~#*P;rE@$i%Hp@0{Tf1s) zrKh?x&;?)84%$4UeYBrHO5&F?K-u#`KfbzdOADXx)*p5J)5ydFm$RyUt!+)7?%wX6=5|l}zV04RQ}gcDj%Lr+O`Fzl@oihZdCS^O8-1&H zY+b!^)#_Dj;U>>w&q9BovBlHf?kg=>=J%jz(v|kza1X+y>t1#T((Q|!49`ySJw1E6 zlfiZPLDzcEqif=!!|sk=U(WWQeYYLHh=^~mXG01uVgm{;WG59|z>X<+9y_Ywx$FT2&td%v&S!@d zJe!3SoX3I+&ShN+_OKQO=dgMO&tf|iJd@Qacm}Ig@O0)?@HAEeTxr-hn|TzR#as%$ zf|#e1{xsIF&{J7R!6~du!O5&% z!AY!6!HKLEII2$q^8!cx&B-bh9M4J=eh2daN9`BKTna`kRmotNS$Y9>z}McLs8->q5-Fyvv4Ku?Q(M3mGg^HC!NYSB{2D1g00b zh`&PMle0DK75LN~4U;_*Es5;HJPlL*348EnmhMla|ImC5Q~e#F*TWBqPS^r}3kgj2 zaDczAUgQT4Vb_fs*6mR*ux^i&0_*mm8(k#dCG@EiShvSffpvRa5LmZ|w@u4mBJ%eO ztlQ&^z`8w3Ft?=gbbEvZt`qr(1lII*FUMK=|?ZYuhZY}bKfnN2>S!1(nRQN8q4lQ zr=lYlzv5K(L~kO8><~^3sjM#uM$K%DJ(mW$n8U-j*;i88;eZnsh7E`AO=Hh2V*SAk zC;4CGOeBru_XiW40}{>-vIxzkY?(0Z*&Oy@FfBqhNy&tRKedVdG>`;D2U-pJ9}LE& z#REkg(D?Z@c29sMQDb%0jU*BMo*+x2{9SbylGv#OM`v-4!C$zAp!CifsUXD zcjw28WM2pLbFs)DVee$~`S1B#;qN`DIvtw7oBdhI zkN#70=XxPd_qTZ6fq8%H3)-1ia8Um!?TL?Y|Dk_?t-(0NOI*_y0{riAz>!Gdf%kvJ zhZpwQ`Ff8z;qUpW%z8kCk2Lb%9O#>2HS~R%{N1jU|0&e33G#SepyKH6c8C2!^}f=X6F`G z+^Of+^i$$F?D0@AA(6si#ES6;^+Ekp76Es8F&;cc_OU1Mcu3^m?8&lK^4+YjTViwca=r<&-Qu8=~%!bHz<(PSfeZQGIe1I+e!^cQU$8~VLC zUO$rO(fQA6@}l;$jO(B3_)YNuy9ZSy0LR%)@Sn7_5?m$-|DpL3L->!{FH?_?++X^F zxl;Z#OWy&HiO#Lb#j~^bp}eWMzM#!P(0&gE6BA{B?K_Y}%|!>7PxG4)K(zc+y@PAz zA4t*nzc}s>W;>mdzWu>zcJ?=fW8v=@&(M7Oct-w}h1V2h`O}s1zYF_1sS5q1j|KXp zw;1-~{^?{R(RkN)U?$53I}V{wP8zz4&>!=q1GD)2KT6NDha~;_4$L-}?@l(#PoweJ z!>EFW{9LC?sjmkMGrc_0uh5yL(DR+_4O!nge5xIlH!sZ!pxdkOz+6S&zTiTo{(T1) zDD6%5$};q|q5L8{qlJqx|1NN{AzA)>rM`VZL`TEEG~Y~Ae2 zwgDZ&A7`f_Tp9BF4&;r~H`;!2r0)X00b%4{;A9td`Q%@7oCSt{i-70a8BKtMeM!G) zdyqfC|Gd%q^&Oa})Q9^EEqr-?vIqG~l%Lu++CEg@LT80hKJOoij1(e$Un2ilXlgIC z=i9peWS?v{hpc1NZ+aTVA6frs`;kA)be0(9QGL;VQhj;*VSwcMf11Wf->5xB`;!2p zJhaz4x<2IZRf@{T4|64S#eoQc&ps zU@$$BL-nQgIG+#XI%^I6$sT!jzM&%I^ZwzL^;hF<-vKoqlK)2AkJ`V`S!c+n_Aj!B z;I6toxW8h8qhr!P+Frc z{ziM8PvHDfd%_>~A$;xte>&{fMEez4&Yj4N0gv`8tm=FL{mC@`^#!SM#BnxaF^}&Q zzbW3};O%1`-}^D&Fy_N(zdm_9hyU{Smd0zUk1-x&zDV{p#z(A&HW=#(nxE(J{TLd5 z`Th*%*H}L&`!U5>{}>0#UueRrJj(wCh7k~lwm!HW>ka2Kuph05@@7GQd4GiVKav#x z#C}MG?BUG7{u1<8#y9AP_CRxs_P~1Lrvd8ybYTDKH#zJ`Fw;TnU48yvv+q&o^_-6O zdUFxz_V~!HQF>p{jowNYz(Mv6PGMdz^BVX7*3TSh?ekzTnd_HK`tkmBMJCNBb^kNh z@BV<>PLnyEek7QhJS3G5eNbJ*!*{gxP#@Ooa{2cMv)ooG|NX&fZh3ryJXAcI|41+e z6V^z?F{;bMLf#7jba4Ms=L^dE@yRsyXY6;`>p3^-XW`i0qT;MB6;IQAg0HucKZW*h zvB#A_`OkH;46+}7Tpn*fr{14Z;V&)G_Bp?3-2R@TAM~gFz?0Nmw4Z|ghMGI;0q7uq zruZ8t%~xrB#J+|!|LhMYW>SyT+k^V6Y;Wve^7f6R{Y{kjV9?I7UEp#4f%&?e|41+{ zPHo>*`x&LZF`u8HeU1cO7+&S}bjOi@>i%&gm=R}Jusx1;Aa%Oge-!_~dYanT6(W04 zd)T!0J*47{eu1&R;{Nb}LQjmguT5P)p}(5jOWpsvM(zL5CuaY43ID);RL#COoY(Q< zDcrBCjT(j)0%(eqr4B(KR$Bt1N{d_G=4Z$oT0tKtIsDL z_AG_|!TSg8KN#an{`>DAA(MX?`x}~n6nx13@sW!k9;Lk-+B?A(MX? z`|Fy2RD8((@sW!kN0s(=xR0qgto@nrmdH=gm*OCb3{yqo1^UY>=Qr|x$j2_`H1)wKU9?KzxzngjmLp7-x2%Y_Xm?xtqOl4_Cw|UZ1euOD^=bP@5l3$ z^?Ls2#D4aW(msyVXH=YgO2w|{6`ZI&j6wU^ru|NH`#eMYSJM7Ep6BBG-6zqXsTq0u zTyPrCzg^~j<`8XI8~fe)j9_%X8!t5t$^AJEFGA_}%cwu&exy^z8JS+>*4yX7;8ec9 z?H2pnbbpgQ8I|{7a0;iV8T4t&{`rH!EKbi7^drGEs26RoBf+Ut<^KI(Fq`vd8~hnl z)bYWM$A9Gfo-8^4kzf`ckcs-sk>Iqc>i+nQsp|gttf}(;_>o{iwkj_J`bOo^{W;mc z`-1eenmBlWNdf3{@)Tg~ubb|-WGnp%_dgviVAk#37esf}4=xX1tYUq%wB_K9_l9BceWxQ`^V%kIJEmkctkcnD>)S|jHmLD zAGm0mi(~%$IPm-#h%lgY+#f7<9767B`DOUiIsD+|_Z?Vl(kH(__7})oFbfe|lSlXC zGn_0@?=R@j!8zmf^>Fc|FZHiQj(%Xh{0D;>+`lt~fA<|&V$!GFq|e*}*TgQ;AfRHOfvI3AGn?K`l{q)$PC++Uzi zalWfevX5K3KSK5?G?ky4D)%q*cma8f=GvD?^_%55D(Qp%jrbq+7xZt$cNxz}mGwIk z%tv$U@q+H}x|RM6dnoru>G=iOKdHZ~{SEd`*4v-jH^*^I(vO}8q5V4?@Gpw@^Ave_ zp2EqV)!U2vZ%IMoQlq}(+r#5H31Pbb^apbt0}`hGA&)HgV#rThnjdAz;z zjQFay2kbv9BROB_6Nmf-l02$Up&}3bIa#i`hW>coQ@Z~__RhtMK^5@P-EHdQJmoPoAHy0R(h#v(Krut6912a*72Tinh18eW|^o@^a{Ynbe*Z z%;!A3y*x&H&B&AeTWc@f{$!ujRJnhHK0C{$+b<4uk4J5fT*ojf!~F~Q_rae(&92*v zKfjyl(dqDK_^;!Fj&Z*Z{5dlc7Z~=X_B*4~@qP>1%cHhemM3ceO^`R+F>3IqfnV;g z_W`Tz$=k~nZ7dn^QU8udeRFb5{IjF}n*e)x98OF>qw&C<8NFXX?K6wiH^u|l zAIpKLKf?cWWc#CkO8sZNz3F*csef9y{oPUflfBU)>EQhV&(k?vM){ikWjfj`M{Y0J zJ4d!R;wjz>)7yvEGv@Ir*O4X3H@641r(E8H!C38+C&iO-{lyb)AH6@w{YPydigyL5 zyw-nms6CZ&!&$mFL(Y$sgYy>iA`D57-m-pKv@u{EGGm@GIkmxqT>JwMhC*_w=c(hz zY)6;EKg&^XU}~Sq;@M@4A6e1%Fxz{4|B>U9=r1o`+WzDFN3J7i=+E~Xl=0m>{+Qe2 zGWv%UuP57RUdi*T*^ZM6Kh{SE{>S@=xj*puH6z+Sm(@RBj@o~Gf5>wT z82W1QPaPlB@!s4%=Ki6@tIMBnnd1}kE8~Y2kM#MkHs5+i(wEvNx?icyw}uS5$^Mv6 zPd1*Y^Q-&7FVD}&K6%pmAPw^`tWS*ch@L9bkzXC}X*_&h;g{Ddq@Nt$#>XekzWR7z zp?HmWBj!(dpA{JKU+j-)81X~*f7A1Qu!ng+SbhG}bbma?`=WSWI3Ys)gXWvKzpp-D zu05}A2feG|LK4IC?=ylvM=JWt+t%bW;ngF#8`S5md_5$we^eUIR5D$5MA1tIW~jFq`O@$y9bvGOC34}M)=(ueHXRo7+L>WkQ<*O%8n z&OQV>uRkxp$ZFW@NwRkwo~H)=R80F)`^3GY%cJ`BK|k>8`k?%xIY9778+{L>JizS{ z(&_m@mQOp#^R3jri>Ur_M*SfVbmj$}*B|yR57@ardVXslj#)wHi!{(PgN1g(o|OL( zUJs)JxO|c~wa`WU0?!F%U_YF(xrBe3!*1(!W$^Z#IV+hRyBFA=!CxpQJe&&b%-{_| zxF8KUIm1B{I>LGMKB}F+znvuBf6WgTpg62QxxZ|(yMv2>(SYR7ynYMuLa0tp1HA-T zrWa#rK{!)A;C`xw+7xVRORDKz%r_=AF{Pg~bx5TdJFNb^Tbm|XF z=&M7LeoJAFD4p^bBmZI}|8lrjlFD?zU|=ryRXn!ghCRR+Bl^_A&XRt0*EL0^md%JS{j)u7iBo%^fRM*S@s zkK$}0V7)&gp28loe20Av@~<`I+iftg5pS~bJ`)<8;e8SjkK?S=opk?2{IJgf=J5mL zll1^zcLPQ#e7_~1-VdT0Q~!m2`vX?TwJ47n`Z{=gi6=VR1M$@9MfNz2-xVLQ+qHNF zvAlk94ug*N@dOjBbJn`AB9YX7<$-wZ100$^*b&d{q(84e=ClJX=c%It#P@v=kJcOUsR{Z7 z*EwN19H<}0yW4s*<{I=CiceW{jrhg!jJXaE*ePCd?3ruCuLj8LTgUa$+f#|(Pm(<6 zdR1QXdR3lly&(_ddTnsSx{IV9yNLNj4!*at4%eXwkJqSQO+tp&p7egw%V{hrz_jt} zls5m!@3C$`hFjP1_>4vmSjG5;RJatOLVp}Fn8WuqIZS3e|Q33uSBQvaKDTWFOBcOC5u#CT&&{7i&b1!rs8GG zR9s%J;))6tFI}qQtFBV<^5rUCu|maHUa8`1u2J#Il`5{TR&iyeidU^taaEOyYid-y zdbNscYgN2vjf&T8uLs*Ol?%>r{di|dFU-S=maN$CVZ$&}gA6*#l7fBfM$}2Gb zNb)L*Fd<-98~Q9?f$>73TNhHiC=l^#WuS5u;<-d$v`C^u{_5J*3cX@J8KlCH&luvP zmLK|}{);i5vr_VJ`2Wg4Ny$9mxdtvp(>X;X-=91o0E{7A?``4O<9TY{2d|iJMaHy+92n;;p`35Z`xm z(R`ZXbypp}af|u_&(p@I5pBJM^$Ue-&JVgfXvNco(0>U2@GHc-?B6kJFDBl{%n#B> z{EYd7!yQ}>Ec^-j_{egl)DCnI}`Qg9r zU?J)+>WlW=A4n>I0^USJet|r{MEfi>&A*b8=7PVF__;qUGR?n|k_tdyXp~nPoqzHA zr7H7Fl%G@t{zZnoG}C-5DQN-dr3QU5>?gOE3+s!hJxLx4^rji*r-QE4Hwp1Uo9}b| z%VAGho?BUekUSLVO*hIfi>@a$`z4itUQTq>uRQ2oX~bLf=ly}?BoyE+GxB@A)E#XU zPeJztQ<9DO7{-6hw^G#kRuZ3Y;rokX{~(EfUy-*r_{)PSNk(~?FO~PDaJoL}M?7RsV|;ZjLi~{EscGo{5_YHa{au59 zN|`bLd6MMCCnEl7^8+pZG7m8Ke_-gdoaP61BY#{R%?Bz7bAD$6`L|WfpH>E33laaA z%ivEhMf_r^f*;>2OtxcucP7(%kLC+*K3{PYo$E8D9PxxrA&mYF{uF$0QsbxjKreIR zgLXE=biQ9Y*?fS1Pn9}6jsL?mUcboH@j5Oe8b5O|UeoiDay)cQjnZj9XDYBvx8wU^ za{P71Q9Q^n#_K5_nU3)i{Br)xT$#Q{8ZmNi<`9UX^n}&Rpe+96de{P;cN4!l; zpz%67UM6HobmY$hmh%_PmgqDd()V(`R~Y3dW=nMB&jyzBFEEX_$w@SxW*PaDrm6X- z0gL=-@BM+4WDNM;Y=b`;U%8g+I~`c?!~gdOQd1C+z0(Z-lo=}j3}Bf*EfoW}ce=r! zisgeWeJKL7$EGRpqPmm-&1@nUBX9Klcan(rNxO zoAUGVym)~H{>;ZyjEA0}JI@%eLC5^4*ceYi$N245$M1ALe&cy-v0v`Cg>-osKi$Ff zbm}jBJcm89gKj?G=KP?S2aD2;^7;5)#QCN1J&*D;eSZPZ|G_yu|S!Fls|d=l%W z+F*K-I^O5yg}|@x7vu-s^nn~L|C2QS7h(P+kN?*BBrnex-}4q481`@mii)^C;L-Lo z91e=lFsT^tr%qMze5Z;RT%qDbJnbjv&(2nHaLicL!ZsZ;T7f2ovY%)LJ6aOj_gEm z^ZLQxRtA!%BVI}4cj`=aJa^@gzj1!}_sT$GF8aHKXU|6em9Ra7)~|g0=JMlc!N%s3 zJ$Zgd7LET71IJIJ32(B2oih**weg$tC(J_p(#CiAJH}&cbUe3_J|3FSBvO7}UJ{>= z%_gjk=XhXPAJ1)+-<@uZ=i}cuQSYbX{w~(@{0QM+>@OwD8Byfr zP3-GmjX3!qEMP7tJAV!CH%spu>F*Z}$nUSbp}fCho^RaVYn8tDm5O+i%sePftUs;z z{;P8c;`sA)JU%8T*At!g4+PEuwvF=j0^*^~Mwrtn{vDW?Oznqw$KMb5X~1sfkiP$g z@3&(=0&l|#e%Lqp0JD-m@$b#!e(|gLezG>-!FTua#($q#sSoa#)!bR4i8uB$^7B1N zv5EL;JwTr}!y)$Tt@u70`6I`uPkt~zpCst~nZXq6R&-$UA*yeA(3XD+_Tcdw`ep`` zIh|5yys-vs8ZS8+=eP3plpgON`wYz=QQu_jIAJ~rtnF73#(c7-_fb(_%=gao@27F~ zpd_#VxbH#ZdrR+Pc;w&b#`{Uu1Nln*@V#dKeQ9n__&@9$(d>)wS3_T-lkWQW{&By3 z4mD6$!zkZBf<0_xk9=W2qSJWR)$k_K?a}-B?{I#S$Me(Q3xIr`o^?UnkGj1#E1!`8 zDM+9_Nggl1;ceWHO=GwBW`Lfr(Hl~Y{QP+VPIrM`q|uM1f*%jZpDNKX?uXvq>&a(l zg!}@MHv?FgN8?)-eSVydINJYr2D0NYArzSC)3omyb2?$09TRFgu>Z1%^fBZYYWan}>2@+bVZZ`oF7B6U*cr%6%#-+?f!Rs3CA_jMJwo-1=J!NA8lB5qx!D~ddqwHF z5m7#;X9iPKT>JUg4@LQT5m6q;u!j>106NGXR%gJGfC;{isl5_0ftPS{62)5_+GyQ>cvkn_cX0APMsB zSj()4`bvH(pT{S}Gqk^5%P;tMGOHVJ$m!*wzdJ>Lg}?Luo*??4=uguUMSl}`dXn&G zj!9oyA9PWC;{KJvKi8@0y9xSFP0;#VNaW8<)cU`UuSnATTf?wtoE$%3Plp;0Qc~pp zpBqS~1*LXy`^M3NU&0Pr;7d3qMZ_PyJ(5NL;&^4A)?c`P()^}Odme~m*mGr-1FtGa z{r8SSnjh5~_FTDH>OaKakc^>B;zxVOv3iYv*v8w>!GckY_DR9V$9ek@y_H$2W;B&2sA&!d<6GnIPTi4 z-9Ikvu|@yh2j&mipi_0Y(VwIH&o}VeF)@GP`jEqrK6oCD`BYemdcKZ z_zV6tdcMzkK+M-Ll+gU=$!`95fyVz(DtnuEkqyUlY=eCq3>flidiCC+)!&Nw<;hl_pZ|TORlWSrA0T0f z&#!8F_4-8l>2K1ZyqNrozrddA^N#n$o{uZvBbMiD5nD~~^XwsQ{>JrD=JT*$pYndX z>3h2H2mE~xW&T&wTf&aLS9!|!U1fRKz@E3icX=+%zl`r;$nr2h?;U;b@|5pKn&q{z zhu>>`mG2Fj<(+x&@|5q>ndNO|C*P}nB@@~2nfESF`TmtzzZLAe?^QqLdr4+_C*Qj~ z<@-8jdHHPMz3Nvtk^LUHv^;(N%;fxXeFzo5Eky`wste6IkEb*pO+w`UqmG`zOaI7Wl*hpIG1% z3w&aMPb~2NFALCK5LWI(zr-fO@U0fwsv5cF%WaQrel7Ca*DV+S@Y^p(F23CRi^qOH zI5z7K%P-{o`ITEgYZ-}D9sN_}v)-RYl5YEANY){Wj|Y1y|bW4u~Zw6?9tZ9<*GE4UWnt?3*ViroPQLK5cf(XrBdtF_Rx5?Zhfm zXGuH<6&^A=9VQ-&B1hsOzF%4_)0aYvEy#D!60!_MuCrLDf7 z_9y8e`gImdf)$Y=+Ht67TWZ=ww9q&!8|{hovu?b-Y?L+C(;Dzck)a3+;m^{CG#=W= zib)r;X;ZMWW`82duUy4eL?e#HLVIqOwaF;cs|mxumKuh>{w)csIzjUqido|I37%tD zrWWInI|k-8GsQ7Ws&m!bOsj*{(q~byUB!JSY>hL~P1x!%(M{MIZ=##=B;{G1YJMyD zqcK^Hw;^Fi>vY*4LUHhiA#J}D4CF7fr_FsE``yK)!09 zEVKMXFNU1NOUoDao4`IvWFIkX@;~#a*k9EDlKPAKU0Qy~O6^O1pN;~v{-XYuls~Ed zs(nh#^8aD|an~L8(eKak@j&*E$?Ts5)Al7z(7tLs^g{p2>E{!bHu#5qRBZ!3rpR@8 zpM8{%IkNn}%SLB0W>o$=`3&Ny(#LjC8>@X>p}!w7#L~bp5=4*625Ta3m+7)zs$Hqv z%H(8wRY9^N8Ut+(%qQ*oyj1T8QTjoJu8)f(qpHZ^urxM74}Ld?D^u0Dq|E%YG%mpS z@mc;%E{hwASSQet;vXM_%yLl{FO$lZe=*7%KVDB%B1Oaxw?vAOA8v`B54}W+qaSV^ zDYAaJB~q;Ya7*OT7jwgLz3v0+9TbZ{+&WTpe*a4x9~;$}O|j8zRXtXk;-wAo(q@jA z6bqvblqsT~wY>k~M*NNS|bbCn{qosgKj451u~J>q9b(pJa<@K9+1v)OD2dMQug4h_}WG z{j3Rw?5K>Pc+HliTP!+rYPKZZ5{>*^FQHRZmeoupgLvWJ|8Zc^FN z@>qhQTVJ%iC>?dv+t8Y%>Q4GdHs$qsPj*9H4IhhTw@^&GNxDsJtIO4`4}UIDOr4^# z*(AD2c8h7(3EI)DmuS~$*;1Rv^taF?x(!WYH`GmUQ>o3zwWZ#7F4d+n{SEcj+cc(K zB)dsvOLdRx_P+ZY>aFV*Q}@a2hPr)Vb_+#y`@8Ihy6L*T?{-7I{~^1fZn|z#o6GUf zh|v@Eg_v!IdKIws<>IntXpzIhETa@e^f;o0 zld9w@NA#|NC4ue|So2cx_3bJ!RN^^4eSM|z(`RJly9V@K28(HBr{&*_3m?6mW)CNa zw^pQh>7SWlhA3MKT&>Kl}|*cHS|REx^Z;X{D_65*qG%2 zU8E)7777(GXXLg}fsB08dM;j)SHuPsT*yu;xPTo~@H}=@!E@OI3ZBFI6`aowDR?%$w;+|D$MArxgmYP! zf<3H7!8xp6!L!&71_TRUpE_2=;`d3f?e#W!jHd5pwLsX!6KKR!nzck%<2`K#Of5B$ng0) zNuLDf1&;cg6JK(ZaXc$g_#Mmx9JL?5CahqLR0_7?7C5_LvFiR$L!69hOq4OYlZH3h zd7rdYq5;rJEx6%on;v1-^)_bn*(~;Ab!N7?F{pNvy+O;9)Lw^fS0+ zfxjNcLilF(nep28r zfd>Tc7kEfuFJ77>{vE)H78eWAS5kl($BC@YqhY7OCueKeC9rFrhO-3jD$=k=;8P1V zTp(~ssfJ4g?#FOS=_? z2S3#9VL^L`1SWepP~WaEi~P`ma2^Z=zspR&w}HLgZeyO;yFKlruXOx)Auvq`37ClOKKKA zj#FQ!<6%?!DS>~7^60=UOW*G27+)>mhdKQQF@Z3ruh7`J9D0^gKUPo$a9l0)(Pt^o zYRN{rK3jsj@I&VpkY$I-O5g%;&Ut$z#bfT|U+L-YMtwGs=bhj(2 zPsw*QQYKW-HMrnk;(8I%Jx9muS%>t6Qo2X*_gq4bB>L~Vq~?VQ#vNXf(kT2OP|9*W2+v5Gc(n`1I3BkzdDp`FcLR{r4mDMEQe40Ocn>bAE~=WCCP z9E?%^j_+!@bpNOF=`fj(HH-XSQ~6&I`E~!L{1g(*`Io>xcf}~b;88H+@bG^bGncRX z(K_VsLw>KR{92K}U&>GA>-lv5osNW9`JWg0b^oREDI}Tmmm>kA5l!zE``10D73|SY z#=mC!Z$bXg#;AXn$gj5_+?2P!xqRw3WK-||F^W(w^!bB+*7K7de{VjxEHnD2QW}8h zuliXok95`hg*ls!>xF~J`FN7vpLIrY*87)!*7NPD+_J4^Ph+KaRBxy%YunejxTmiT3-WDVU%7Mp_1AZ8scBnt?e)8NUDtVIWy|heHQU;E?%3Ad zw2tSk*-^Hwr?GZ(dE2^8D7$C(&h6#<8WwMEX{u@4-_W&X!|Jk)rA;-#md3>!_U&A} ztz_4h@{;BqRVbqw{JXB-wY~i2hQ-%Xd8Iqs_pH9La^vF0GXM4d%66k%e`RUssDs)dt+wp?Yissw^j9wLTwSwgYt#01omFdZTGd>;v3(Mrs^(pr zJF9p4QOC{8+p1eSS8d-@=dY^jT)lW(tG{x~hN{IayPL|E_wK6QG>JaDO8nbP+ZsD2 zJ4jZHkR(HS-o#(d)vNUHTc0VcGa%yYHHus3!hwjUFo)) zn`+m!?c81_`NX!p{_W+v{V`=m+pBu}=B|deMW zCvJ1S?Apfm0~xt-B^phaS4Chr1#GqRp@JN!Aq2@_p>!5 z%fEeJ?7FOK5%F)=O^t2K15Jq2G2^6}?^1C|I+7F zJ5b-nRr@dHI|kok;v>JH3 z+cI~*7udb8( zNL%Bj#_m;^HWStvBD?)@=kQ%VeH}$i`J7YxX8}&~9qI>Y0Yk^ujTfboc-;OUj z?@#_x>c44{iNA?6X#ds7jpRjIDeTfbx{y7+)Orx-So=ID|c_t4*wPe>nk8iFi4`!Bw~ZYJW12Xm5W`|;_dl118AKJ%28 z^>Nh8BJKL<>o%g=_{vn{)BiohMeD}UQ?+xfeQS-MR;uF<+;JI~sPSJaUE0JEIW>OT zkx`Gk0r>~rMY3M63X!h&Bi%3lFY2YI$&zkBI?3^hM7{r{UV^_BhK?oYic6$leTnof zmq>pe>7>sYluw7=KWAP-pWk3Zg!pNHiw?bi(jSA1rDqd(Vx?1BtaQ2`OY-{#QSZN> z1wW zoqGD+NKe;yxWRg-aG8F{r}Tb-&kKDX$C=VYiCTi_F6p`Ye;2E=i7j7pPxKH@>F|xyPh*ng7vCq&v~Rbda8MeV2%Ivz`?q-CW-@ zqJhly9Tw?kId6z`bA8{D!s*TSzd@v% z?O!j_Nq-vK^`lmF66#0#-ZatLL^{bACgEASM7p^@_KS3qFZahGDP7!3<8t1R(v9x~ zgQ9;lnB>U%tPsXBr|WgGgP*#BUN((Wbm-|`ju>kZXFdIl=sypb()D=%yeVCe(=L=p zhb~8t$0eq8Js#`(-#Y&pQU5LzzaEc|nbP%m%(O{&x38hK!yo8nn3p&C*ZcQ%G`3vJ zC+e&5hs1-dIMCeYFYaj$_OQ!cv5+8NYe#Dj<5x|s{&HGy$o0W9Uvdt~N-2s1l z^JS>G8|C#tM_&i=uV24y!z#_R8#JDZxfwzf4ly>}g0e$!=C-5=P~?d!&(j}>?K1bX}pIQO>mbD*;c zt2$Q9|J9sWYrXZTAuaeWTc`j2EA7kofd7Wu2*p}U7dLkIlrA-*m+0j&Hbw-+a8uKg zzUw2fsh~yvf9&7yW84u_E=ADEQ4M}p%yV?27!GvEXrF=7hgtvIK#$daLvLxhE{M!j4VsGT`StdD90i-prYz6AM*lK;8lNAe L2K`+YbN>GafiE{% literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka910e7175f9a30bf6e713cb86932fc46.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ka910e7175f9a30bf6e713cb86932fc46.o new file mode 100644 index 0000000000000000000000000000000000000000..c17e31cb48df7b86fe93ddb2a54394095f6ff688 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY5sR!clA0Y;51%)>k+gfL{{+13jo8ITZ&*Jf6oAt@$oDi8|B)|;rcpXTL zd%wRP-S=KK2(TwMECt>7*8l%i^}nkAzp7iezf!hhnXYL}6^*^hVl@S2EVDOu@8P!| zmI_KY>!}&A)%_ydZTsK!LfdDjGj?dmmfiO*UT1CDbN&U_b9ntL?$LCDqNVG#oj(ly zZXCP%;d(vvL6t3Z&BhM>UmNQ?(!maa_Ha)IEBt=8r~kte_V%xJc4&HPPt6(Iq`o2B z1nTQ>PdY37Kf0%X80~b;_I4e{9y$5UVCA{}gWE4XGq~c^GlSpKJy${dEMA9ZpZHF= z@A$FsW6tk}kL^Dm9+`I{+&AxJc;Bj1;fwoDg-7;X3O_yXz3{`W=fcNMz86lJ@knT7 z`~JbFpLu35b>8C8?3V9_&(A#;-h1h1;isK%g`1CG4EL?L1iHdd=)i%&ZO8Wy?wxlm zTzTqvxVi0AxNrYk;gN?ghR<)m2%1Hq3g>~r^Yfk@+`9_>?E7wbb}Pm|`Bu2%+{N&= zOElh}ghKN$-l_xW=XiKd>vzHOO9C)XGz7QJOcVO`8dCv@nP96`R$E*3$ zTj4n;PJ}C*m%``godeC%(CpSH2m7FligU-%_wjJ(_{ngk^HexA?_&7rRmAg>Q0Vyn z!8s?M91MZy^YgwF-aGFt@C7|U_CCClXMZtN0bQPd7&OpZO5%6Iq4(Yjr))V9o_+2D z__`1tH{;RJ2;rB3*X<|5$xog`yFW#{eS;N%?M*xp9(f2bE&Iw(6Ten7a^dAb^6u;( zBq#RNB)^s&Bs+R+tk3})X?jmhJ$!1S=DB*tHnERzn{`$gwLfse{uAL}!Ur~Mtndx^ zfTj^{V&9PNc#PT|VXW{qz;bvS(btdQCm&Rmgg#tf48QYSy|(l9qelOY=Z_!Qv^mNr z$^Vq^4Z!zag>T;Ty*u0YdT!u*N5=5IBNpEypH%)gVDUfYuQ&3+Yde1wij+)xd-y^9 z(D{EVzUsEKvMrck25pb)!`rMSKokN4`s8fXG$g>8fwrEo%vh+ z+VDQsPxFXr1CK5TKJpyvkNh+1zxb+t_1zNIfA)TM^1iiS=E`a{>I5EW_ZH#(a-tf{Yg-qZ2ZbY{?VZ)t}N_g?vW2 zSTeijfjsB`wyb{y@>NK=M!@q3cpd@IBqPap0ep@_|6a)C{5vi59^@E#H6b(tIo^XD zM)^sGkyjH#BajEW=s!Q69l8jejJ%u_Vp>mJxg5-MbvDLN)>uES8y{EwC`7Q|B4&*l z`@j?FswojywEwtjw7!h}v74pO^@M1AqMxpL`uP|s@yEE9HtovCRT}p(SElD`%3Mz< zbx~2s`NiT8*@kRFcB1~?ge=fw&K(eQQJNo|?uB-8?#C;F0`*A&v*Z;!(DO~@J z#)rM$eJ&wngI$&X+3sUk--Z5NGxU%P_PGGRB|H5FHXxgyht1Cvu>K*~kL;j)`u#Pm zKU~84nLh9+*;Lv8^0#3t`0{)3<4dsFzrlVne_)3$AB0^Ec5Ip^^g(|5aaB?1T2k>9 zvfn*@nDy~IoCki)U-WjX_3HO(|3PCecDVtapy*7~D8L70onXuHa-j3=xO(tP!YiG* zXxLF@yD!k@Yw2Lsz~mBE$IR1Han&_uyU(WSHpcAZu;j^Nb22+_zasM5B8AY|{}eJw zvd#74=FEQ-SA9E6^spr?0US*x%K05l9i4$rZ$0kRwYyc+`L(X2sjV@{=C`$X`sTMi zKEHE&bK92oSiPZ3)*qiAy=iD~^?N(n{7zr6bG{I*uA!^5#oN(Q=WS`(>g{au1?n2S z9i4R&+v#iTXb<>11I+~sx3>CPy)DgK+wvE;=hyxFVlY5me&NEpg1kj_`FVMH1qi z>$U{Etv+ELE5JM71i>_Gprs+;^*8ZJ=MS`Zws&^>(VTGYZ5^H7w$8k}zv`84Ft-^Zv$dJQ&M^V_#<>F{;tvAVjp$Lpe) zm1QgI1g0(zTeW6y2an!SJhepra#!VKH;PQNfwW<_Y%NPUJkbhff5zeWje>8$X5Gej z>}J$c9FCJhOL+-B?-AB@*a0zDLko0P3MgKj(Z8*QM zmIz9&1kybf@V$$!X{oz=dS)?4WOvUj2ft;E;B&2=$hzX${tW;`Tl;A`1feQkhw-BN zs&y5m6-Dk6?Ei6&A_^Un5b0vPwh+kQ8H6vEbTxY2bh^;o;_Ym1Z*zCHx3_f6bz^4x zyd6Gwe!=`jbKUFcGQV&sEADD;X>@mVcXaw%-K||6o$f~8mgY8}du>(KiZylX%4*hB zRIREj+pxB5RcTo%n^)y7aL@Aw8k*c4tvC;O#Oub;#4DYz;PVd)uiM#^fZG>17ktw5mAICTA3Ua!gPN4S;} z0)_ly^lW3Scc&I%zXp;FG?d1&x`n>@B*}5IV>=Qf`~WTm6!lIvpAP9%0exzKrPAR3 z>dQ9v%WT%W&mLiy0U)Luoox5sL^~gMV|FS#=4LfJcA5K};kGqVdbRb+(y! z(R*xypZifknSs|?Cp#3d+tGqW1`edLp#>~AXuH3MeSt;v14-t9Z*J%H zp=^6$P`44DF~UQ>;DksvY6+8|ZHzZFn50MPPo%N*z=VVe)c@eYBzBhQ8K+Modf;<| z_6c;>Df%aRKFtI@`iDMjB#yuO1I~Xg@}-Ca`a%D>!AXdhgDJ2>Sp4@&Wi?M!*&! zkBk%ud|9`~E)xTq@Q>vIJ&meN`WoA`l$6$T!9h}%mO;}r&G&gSw^`%`I{pK~MZ4BFFt%A1DzoCv!p z>-Cqk8Ns=;X}r0_AMCTc+cB5Vw+E)BvcGb&8NoR-2|tJ97qBP0*&I@B;saQ8hKoRN z2R+s{qCe{7csuF>Ujn=vc()1fMqS{`fS(F{p1==__ES+8_^$z<3A{((XQlD}Gf@}# zWNml13wSJ|G=7u7yHFSS1YO9pS>m%L|C=RV1D^Qz3w-?o_}4=Zk$l|$$-ja&8}}vU zZ^7BKNt_sd=FDjQbv|FXJ^1`Xd(xL3{d0cavFYVm&of$ABauDa{+Y0Ew1B=y)ydxQ z56tIWK3{1*|1ymgzn}?n@?YS~U--FfF6yKEGt<9G+^qCNSe z9eC+)@NejMMpVDb-oyae`yJ9Z-J?i-(|C5}pF<_kPp81k`8-7cYT{h-!#7s`sVXBgX=p(>JR>(5!H8nHr5N5Mc*!0Oy5@3WA0P3eJUN%wR6OY2iKSn~FF(nr z$K=a#B3_EmO7G|Vrs;(gfBZI>jMi_OKRS<>I^pr>V`fd5eZeT5v7+J8Xu zABn|x=o9i-;=68$@7ZzUIdWh!iPWzi$c)A7p#|FVfSn?@A-?N6*_X$6=D>Yk9H~K-kmlf24n#F+SUm?LfYvP z{>}46hs-ZY9$r-b5wJP<`uyKj{*-Kc2g{in59^r|TLqC%^Ytv*M$v}*2jnF@U+*&f zjN~os1)dKe{3m^vEt%)X4xS&|bAyiLS^pTxc9K7#J>=)(@%h5*E^zmyr<4l(AwCM|hiSjz z;Qo^w-JfLTMCs4be#4Q>?M42>_a|;npGEY*=LVA>tFioEf`iKkjdAKSyJ&>(Wq533DAdHXkaI) z0eF+Y18lIJ|_-y8`s{gQtqCs_2GNlTT#Uc>|N3vHj22+!+`jAO6(~ezao`e#ATB^#sQGA;24$FS{H# zDnPMey*P>ScL{s=>)#*8ens*2b$-65^mEJdV871S5Arv@KA$$()0&@|@)6i?e}ExD zk@mBmXIut*ggYq(fNua`4unHPmzyczPTLakNkWE{%Ouv z765NtpJ~0a<|~Hv7x5ws>ot}W5$~Zt(|V5e*|dJ6o`v-~N{{wfk7E0+AQl^D$RFb7E7QlwSKQNYoUdf@ ze8n9j5pm4 zem_Azk|h>=iqE6+qdPYrA^-XRG9Q_1ieIDm-;`aT$o=>JAhQ#M=N~w~FYkI~%y@Og z6t4{D>+ql3iC5A6H`X`vetQ8vUl?n@y^D*Z)+ZjH=}3(}FUp~KazCH%lO{mQp){8% zK4JYd?XR~`d`jf|a~{u-AEocI^ZW_>fi(68^4DDni4@P1CZstiA1VymDO@M;d>8AV zI4?S~p2dr2h!^Sq&iiju{6apZ2l%HC(a#?UAB|_=KOOOm_Ct|Ok>A~VJTvV#%<&83 zoAw*fw<(@+ym|j&+HU}Fif_X_|NeyW?6S#Utnmz=C&la!e$#QDG;l!b2lm5$W)giK z!`J7t_`GrFgxLQ4A-+H7=chd1esclnCmHtR2l07>eG=b~bN~2Hv>%V1_h3BipJL}% zSpP@w$EQraaeSN1{e4RO_=fl~c?#cusr~rq_+~o)$qi1KY>98N^B?Q}7W4Ne_S=(< z@vSg8UT63KOY9G6eq@fH01JuzcQ*FlS>x&RIm7;YD(x2y@ymPBxUraFPugGu9_ug83Lw;oVen1*q zNrz3m81h+K66Ah5zmr8M9`ZM=Klr?upZALM48wW&5fI(PdH7MBhl8>w@qHYtG^JVg z*J(QT`|Rr=;QoyL=MDM8=i%@NalW2KpP%y2BX#<~Jdx_)cco>&&G&0j`J9a8cy$WiS#N;_1LxYO_KyKphgP?r!lUuVnB*UjZsZKKTJIFF|ND@MyM}( ziFTZ~aaP{YjK%PKQmlANcl0IBg;&^Se4d2wsiq)7?9~#%aN;^RfG1w0fJuYHvN%`x388^oiJ%HwBjwWcNO&~y~WA!JV~qGy9n zlROrG;8fg}WRpFkYzdJWx0^xldXmt4RBjq64n8WkD?UCdw_C^^uaBr)V1#5` zgJkwkMKb$nk0MK$IFmF{&LG{5I2%0&&79kGE9Y@_t_#khqoNa|?S-z3%G*-n%~+wQ zVzkkib+l?g4SF1fCvHdwt95} zP$xXqxiYnJsmSnX2Vm2tgN@aip5sDn_UtAdYY5C(>G2lZ6;KW*dv&W`P^Vs}qfT9s zu4&X9Rgdj6Y&wlnl)BblAjWGvmp%Tc?n(F@j-aW*ylL5(EBTtcO0}W2QCwvsfcPjWIvFs2Pv@&yE|bJ;$mW zXV|?MF4mB-boNQ{MxEb_;pSCb#0nK3M#ml6W5z#8v#jpAb@iusVc5xO+7)CmY9}Wn zt{T^5p+8DoqLE{<$(C?q?)N0z+lT&f=4!mv{Z{Ay?MH)8V{|LG@%p@d4z0cavFoOp znmGEj(8^Cdddq&uln;$+AKi2EIhYV#L1_6wvC`jZb(4PLtbA5^;?b+k9j&;azoYk; zoPYCK-skOX@AIx-(|j`S*hH+4MZG^ZN|^OwJY#G zIt<5dTa__K;GdOv#AjG^Bx^ZG7d~#n4~Xx_iS2)^|IsJoTmnP?y2ktOQM}Rj&W!y_ zyh8Iob6c%z6)v1F(0mf*v>gXFah_qM-S8Pki&S&VwP(2i;3FFdbOt9L@I^emd(mg9B3=l_s`$QE>s_iP`6|BBvFY&Z#uezgvDrQJsX z=Lhqg>iTc9c^~{}$V9Z1!$?jS9J*<6FyJ|V%o2X(aR$B54`I)#_QYGyV zWcy==wsdRNJNXZ__gL6ylqPorM}<6(c(=B%ASAw?DK4@1ugdnh5~Or&ZBP5TC*!m~ zYH6=}wzi)x6yo{U1l;8?{$-)Sl?9?P;Eq3M_8Q7&#_NPmZ`!d!p$P)#y9^WPtZs5{1Z{ zCGtvx%EJH9)let!_uzHJ6ZLN@vc1Oyv^W`Xw7wAk#TGg|!k?F)8#<0>KR+B3e_u!T zlO8YHo%s`SUG#vsX5m8en}|^V4*=5mC??c|jRHs=8;hikg4Uw_o;c%wRkqKSlr%mS zi~5*Pw)a@ZKPuZR|E2ci($@BiFnOPjGyXR=ff&Wj-x;%xul%SI?R(MQV;O%ywpaeA z##ikWd

Z+P@R0JxvK~d-|~`L>)iOY|{T_*oKK2Q zUDfte@?|Z{9;pcVRQ+ZGoZL6PHNj6x%uGw3(W@T~Ps=S8Qbxrk~ zYT8RSdTVR_HA`Dcn%Ya(Z>{zgm#kb?zAArHd0E%S)|ReK<#-jW?Ao-v(%;y+t{eKR zSe?J_iN@uXEgRP_guQF>H@2;*ZQ3O4wXL{v3;ND$+)&wNwTrjJ|HQ`TMUN_fQ`frI zmgYd+cJI>sJl`Yvj}&fM>Mh8t-?A8$f`2!LOa%P9^UK&nC|=ShQ$SL0;YxPP5G0(&4KU?3mW2O!SSMK)mf5gPz7ZY@mN#-7j|@8TU>7MZ2sM-47KG*OThr zLN^ugvkHf6sk$$9x}MX&X(v4Y!(R!fet9wAa}B|0mIY7rr3QTTSCLjYUl~UOyeL*$ zqHDPcI@P`P`V!jO+5759!i1I&jx)i897@?S6tSR*uO0|sqqx-C3C<%A2dMZ!m3Ys4 zUdy{iqc|nrbEXA7kBFmM*bL(;q@a)SV}{F}kOJZ==#4vXfo#Vm!?1Hxa#5HhxblCY zFUGzFxfD$FAl~UqXLUKLnikInC|Bn3`AG zb@+b)i5JmVOB1{<<)AU+jr*pQ*DXz;=2tW5sh`UduG%h?aJ5M-mHp?)eyXL(l%D@y z@{gxW@uKF{bCS*JVEg(trxS9`JNED0*6rIu$&{m%}&9`acRV`!=pbpNDa% zoDdiGHm(G}BH?xkSMrTVf68|VH9_JhTHtvSJ{fS;mOx*U@F{>Bw~V_CaAT@?RKjUo zoF+!gF?mPh8sViIrMx-kw+}h6u37=-^N9l|DF`g3j82YE@oT&1=0dAx}b{+T? z1KcQYuGG7g&sh>~m3KfI$SUs}5^nA1=Mrv}_g98~jQoFOfE(q_l=FwiHO9!i>%b37 zIPqg#&7PO=hb`~{1HI9{uS>XIxaIGQBwUTFe6dNw)wl|N*g$U__jw7o zj{7SKw~m`C=PiwEoG&g3A2nYjocd9`kN}$vaCyYT3hC9j)PGxkLT z+$dL*geyHL`W6Yd%KMUpTjed4AxDjyC&yi3fE&jRO1K(V(eILQ>$u;SaO=2~w@{(n zg{G9!9mO>&3hv=G#+u|+!ByON-U3(gVZZ`cabVa2S4*!t|5W{}^?nxkqoUwF(tgV= zaJAn1EpWBoGm&<8)YUh)c>~=HdxS>s3U60iL(^mYCnwABKxX{9Bma#j`ng7SvpWy! zsIIxK89(z_-_dbv&Dy$J1w+lg4mS2CM{XVX&r6O?{kSjCfuDwuqBhqDAa}X1Q;3Tn zE}`GCEMHYS55FhZuw76sgFe>yI*U5-`;+xuoj!sJdNeda8@;8z$D12`Cb-a^uSx#g z-PkN{!FcrZ6?JX&^LQ&(tXo;y>}~Nibgpb~>}tXAH@W-cHfU-0Hr92v*EMp+gg{_V z@F}W77NdmA+5?KxwwjW=O4Z(wUjU5DS-F#2~z1MQ98PA{9!--Sa8E3JSer5q(l z<~MY7<}bc!ZQLPy;hQ&Z+#gK6oejLH%_rZ@i}~20e!KM!{bg6Nz<$E_JqRK))iU*T zMJA|IWd5O$a+y|DM*Dn%$+RE}_sF}`GL9*PidU*W6E~D{c@#ii{X4|X^D-Z!&(I;?IJ6pCL( zZ>|5O9RGUlKe8Yc->ScK2ye9s;Z_6`h52fw-6q?Ql3%ue*3!VrzYq9$_H#)8xh#h# i`;jZC_9}nQQk+fiTR(Gp!dwG9V!kug#2TNevR{(WfPu zmLVrXWEtCFFfqF!XX7lHc)YX>8D}Rmj%ITv^2CRD9L!)%HtUngIUzDbB)}fr@j8$g z_kMppy6?Se5MWPiSPHuDt^fb4>VH-Je^s|`f4OYMGF{V{DjIu*#cB%5SY~hR-otM_ zEESY))>AWJtNU5D+xEZdg|^R5XYA0BExYd>yw2IO=lmP4=kfYA?$LCjqNVG#oj(Zu zb{xC<;d(vvL6t3Z&BhM>UmNQ?+QANi_Ha)IEBs!zr~kte_SP?Tc4&HPPt94| zMC$8sPdY37Kf0%X80~b;_EsIn9y#^QVCDJ!gWKPGW^l#nX9mBed#-}^S-cL-KKWv} z@5J%&W6tk{kMBPb9+`JC+&Ax3c;Bki;Y<5Yhe!6k7k+x)yWxjh&xenndN-Uhb%9F*)87*UzmG5{M36t2|w+8Gu(XQQn+u$d!Q=}g$^7T+;(FB;8XLC zhbvE?2sgK#4)^VUGd%L}rSOIAmq4>9RN*`@cwye&!KYTCpMBp6&u+!|r``-#oWB&_ z_8yJ*$DzlPAL!&iBF>=A8%4($MVIy@P$wMaB8!==(%CbmCOF(s?=@ns+Ju^eWcRX~?>54!!$kc-yj*;o0Xe zhA%vJK0I#5qoI+9fnNq*FMveR`i6AJW7O^_V}-8*mc!eKzJ3Tl`Jk#K^x^tq_?_qKwVkgWGx~2lfBe9v z%~3u{{-=Cz0KWGMeDj{~-Pyj^a|7QyI)?8ZwfG+Sr1HN3i~lKqy^#-I+xf##q-642 z!w>3*F8oXJRkxj`UHoP7)sdIV`hT^C^`~NfoSn(84*yWU`a1l2D4Sh9TQcd;P=j{p z?BD3uhWD|4nnz3^-{6;rC0Q;@076qbN8{U|F)3zpHE{~FC?+6DWEOF zTpQk#&_Dbytbh1H?a;;lrC+`H4m&jb*9oEFZ?i)e8?dHMVErRMnhaQCX!t>P=+_T| z_J1UW&aU76D9ttSeCXQFS50$CtnWgei{PJR83wO3=DQ>lWV{3!osfxSOP<88{u9kr z$Y+#`C9`WD$aCRu%KAqjUxk!w1U!#`=MnHsGLn23!RI*i?}bdxztuwTLXMGF5<(-8 z<6X#Ml%Hf6c_lG40(qc|{tFY>p-a%o$V*8fruD>?%fUQXXJh&w_5yIJ~NPl(1R`stddpO29ee~fEs)2@76rEwo~WqPir z%=Ls)7Zrt^UnmZdZOA5MC+hDV==B`*=R*7pBzyX|qyKN=x*D%a-1p$RAJ-Fj{Uh#A z*WC!Kb z@2_F~;S$!*^npjorpo@8zYSZ#m*0gSzXzNBJM0(p2X^T4LD zB^6&G``y!rSszE>Jn&=wqPJVESHD~P_ZoAt%MIuRMQ55u0X`_}1Y3@m1D$Wj)q__O zUg^w5!;UK3eStP#O9!h4CYP`}W}cpktFAHIc$=o%7_*PVk|&GJ$?UlOvdC+T6hdeJ zAY_tco9o5R+5aT2`gWG+VM|y7IGRF~^E;Y4Is={FdfcgNcdMxLYh6cETVs&TZ)@-L z&2M{re&_b)wk_?kdPA42KR!Qt)6m}P_ja=RoxWh_d?8w0Lsw^ux1*!Z+tRYt+u7s` z)HQfJI_o61)7RF~9`JVtnim#qZS}Q!Tbj4FHoN(=J9i85`&b+(8eH*;~ zR$rjut}ru9!)3ABk^|+^$qi@(+yypN;+TDwL+_d?tZd5#x}Je$P%*B>s$GZ3BV;CW zoAg@;S@iP0JQp#|pUlG~~|T$EZbm4K`Tw+qZ1#@O9>~y1KT< z>!O&IWh?6hrY;X#wPtSzkKR!{wM71MSLI|kicGVCv|+bwElW8((F$gN#^K{f!8c&D zZsR+4GwMkW$4Q~3ya0HHE7ng>jv%N9E7Ni7k@O(rWhdb7qiIpN1ha2QdNs77|1RJ* zoL^W=1SMAj>7EMs-bL56)ZINjvzQ~YyJwbz-!ewYZ#p9nz@+`qThR zrNRBxmu>9l*{pY;J;E*nKuk9}+3u$j?R?yg*{STfoBgO;$iSWi9{u$OF_9#Q#xLvE z*=FKJ@39Gf?n4D-243f!>`=gNM++7iIFQDM7O>o)?Y@B@jAs%3KoWcXAe$LXu>EGr z8{2t(DBE5b)NO=kjPQ^zI5CoqTEb*#8{^FkCh1Z7lW8nHFfm~w^*?wpiJc>Q#_5xY z9{Aj#eIlK8ivEe7PcuP}{-F;WiQ}*Sfb*XV`3&^nCr#Q5{GdOg=laX&)QBI{b&}6t z9kvm_Nx@05BDZgX!VCH^BfKl%oJ9Plo{c22!8BI*RrHTKm-h|cpEC_Le?9s`|4%-} zB4iJ{1NI!MVh`*jS}Kt5QX2b8_xK1m2jO#_(2qWTU2SbGm-mfq$U9jlUy3-OAM~Fa zoIIKQZ9Lb%sIxOK=}JH4S>KT6g+0+f?BV22PWAvlg#k8k5+7gGdv_*8*arZR55WI2 z0=5WwWMrYhmvw9GGBKbD|5zT-)2Pa{k2ycCfbezOub>a~uLo?*NqABM**7?Y4<8u- z|3hi^RPtvml>W`o5AYem+!?fP!oPU?+#t*`T&qnc`3AFD`PUBvO9kU!Ao_=9T!}=q z82xM~=`SMXE9=HGK@(g0vxy)0S7G3xhq%2oFtxwLhJP`8hYj&Z0fdr@U#X z&xx>0vR?n5HX}H9HjOuz_=A0RcRS|t`S!rHRQ8unHX}G^CgJCB`~tSOo6RBBCO&{g zXSfLTcF<#OBl=@bj<=&O@Fl>zfp?qmZqx<74EU+Q=L!6pzsBbaw+EkpXixgGqkqoN+cv#C>j*@z*s$PW}sg`PY9cn~VDB{>=37*IcqE_g9QZ{UIN zx@b@SXa`>U8~hvkoe|ZqvNth6_I{i6P4_5L-!z_G`R7mx^wTNuaz2mNzeLOS(Ekil zxAG6tKmSCE@EEWB>q|`i_3q4w>ASwoqHmht27UASn!)v*A@v9U&xq=~J{#+W%c5_W zE2i%%(w8eXzxn(?d#P{qpAm$%AV6fGSFFD@Kez#zQ5T541ki(|Re|tN1n}=-~8@n7=u= zKFHtBY5HjXgtzkb81rvA>?ewtf7WZglGCx56hWRs(_)!J*<2zu#Cjg*)0RBVq_hoWO zYX1Sre>4`~p-;$TiSN20zGugY=g5J{BvQY6ATt)PhZbnd19pnshWM`QWM3ZNk^jJ- z@zF8)b59bjujR=~0AWwF|B?P}#`tVMz61G&lCPff9f$Lm*FKym@*T2YN`#%mfZTt{ zJ~#GH_&3iN9WuWpd3aIzN5JOb>+}Cq`BSp(Z7gSMJgjF>|+e!X}_K=^C$L9;LyU6)OzVdn% zI)innJ3$DwY-)ATjs6Hqte6OnEi@kyx1 z@Rs$AuWxpjj3;CIcNN2W&Hcl_nf52}kEdvIQc?ba`FR#qvM=n9d_R-iQN%;C|CE^h z;s56Oxq#Did(xbMJ`00Wby{v2>F>;cr}b-UBD);n`OVNY_*Zgf)W3!nU_X$P3Gq=t zKTP`#2lt=k=>8-tCrW>w_8X35ZZGm5zCUqu`YfUcJ~x=`i24)q+w_1t%Z=V6L{I*- z%fZ)2jF(3A@TW|pzvTwA+{X5WyWDP?Z=9a{qs;;92!F)*)IREuAEvQq!yL&XXCrCE zUturkMf)VmcaTrp2KItizP_coke|7u{Xzcpz%+dd=p!!7=i#c<^gy=D>6iQ?Il-deOj@e^^&%dCUuZ+k&nnFC1k|DM==@Ji zkKxmRPeR>@cj44SwI_T!>PCFV1T%gD>M=az<^Ia`0ePwJ0tZ4rHq7VTG4gRfxkP-q zP-Tv1FCc&1#Zt+?;ZNh(8&%{_sno(>>bHw{@!zYca|sTD=1{W;t-Sehi`CyC+$T1 z#d1AMUL?r{f9U}|l}=0|J!Job+OlZAAn-p4a%W(CeE3%{_|c9(_z~}f*Ap1$hXAi* zzU*?~r~t)=_2Lx9-zDtfuYZ3W`vt|@*ZBFK($6i+gZ(;RKgi$s`h3P@PiuZ=%12^Ao5f|qB=}$3VA%BRQuS_2!UvW>r zalVqp^A&ged-x!^*AfCFjqWQ`&^!Lg0kt|wYdAvk_mV5;9(v*)l zFy3@G`27U=NS0XeDL#+NkM7)jg#73K%Y0<6DSnOKe^YjWBKP0>gUn74o`2x{zP#(@ zG2_)yQ@k>qufuOzUkKaG_#_??m_xJnb$2Y`}DfjdJm)ehyj&G*(pWNX6Q!McKJ@LlWD6JDp?f zzj2^K<#ys-)otv*)3I8DL9t&&dl%}_{V4La3C8m#;4_TpQRM%bly8dtC!Y_Q(fudi zf0I9zEefBgf__e%-p{1y3a`BS$R-;f^} zz8{drR?=Y;FNS=UmIS$<&hKOqiii9S>kmFJ=I6cQJi~AveiTGEaUOmQ=i#92Nqis2 zDotsY{dJm-{XY8|2)I9E|9M0H@Oe1=L7cB=(dVc9^GKaOFi)g9_+4q)Z}a_HR6Zvo z`S|%F2`ScBJ2?n%Kb%he)5k5cKP`6tpLcWen$Hi?*cZW%*w3N^od1+}e-z74@cDr$ zKlu#mkm_E!r-6GKxTk@88n~x{PhA6ac#cpzfNwdlo}XX){f9q_{KZRs)xX{Jw}1b$ z%5i_y_|b*GjvOpxv-+Oyb=A82=Hwjf>tn3nUi;k0`~O-O`5H&1tWNmLzMS5^z7%%m z$FKbUcagX4tgLqR^ke$!+Jx1$*o`2h|6~@$^?dr`sHVv;E%G|5u?Ob>e}w=_P;Hz& zKguL8N82kru+`uTgLWiOA*MmdItJf(Q}-M8mj@org=o-9YuO@r;STQ zd&&SbZ7%OmjrVGT9rfbW;zW8CrF!gI`KHN&7f_>xg3}mRSTP_)qsFMFo*yQj_!w7S z1tZiKy+k|C+c+z4XvSjrJtp~Jd%G=z(BnId9F_sp7?l?YXJ|%#A_ere{)Rt$_DZ1qw@F}TdiryIy4=HaR}Ly zis;#((HlruP=g*v;fWj4!D?M!`4Hxwn|n|%ZO(c%j}2H=jQj{#QN5m`=Qt97 z$5#q%ly}kCYDKZG7%`r%r$N0o2Ersud2n?1Wp#~K1NR(ia}b_JBf$zI*67u2cO z>8Mjzq-z>AN7ZBd44Y1)6s4}U7r7o!xw+9O7a2=8@p2X7^j2~KH`q1kSVt%WBvwcx zq~lRpY@A%r~+DG@Cd=4f=R}fl$P^|QKTHU0dI4hr3o_O?Xb4M#K z=womgIG4cCzpnBAdlYZ< zy)$F~60gww&)in)T7?Ve3pAfZIb+9xO`K;KX*Ya^(IVB{a_w0z0QkrT0-eQ)hkTGx zb!n9HW}g2O_t`$1-0bP+b6)uF^AlUy_vZZOw;ODKHFNItynmYZom+S9FSuk3gb z9TUt$NJBYmnPO^u)lS*}aWuYBdqqd>iO$-d<~gar;--v|W1{rrh%2=xnjTS&zT-~@ zc%LOvh`d=MuQaGE{106Xbpn4EUPnAp|E40_drUx!lL1HT3-MoUp~EBmdHJcK<9PP- z!!hyqb!0#3@uJtCHv8fRsU>eH z9S6zzr0CRD%`bIT?Y0)JS=UiiRwPQv%Ho2Su7-uhEgKiEsohr8vevu4v1{Xon*4^= z^|jjzSGRkMN>-MvD{g9NUDmm2LrqJAf6dyKiXF=u3Yq|0Q%m^LmZhsVZm4XkU%xJI z(}v0w-t~psTPmw+*X$@?*0Sue+9rR^nwI4SwU1TosMt_Y)3T{`S$_R;(YCl{X-oZv z;ym9b-tUSH18y2qH zc2jf(`D4*7+teCt+O)6=`e-dDnYU~r-Q~BJ7HnPZEnbzsv2|FBGvsZ7uAZnbDa`Y2DE8O4`e2`RmGy0_pIAvYSiY{=TeN0naoeiA zhSqgW^_yziOE!9IYyCA#TS}VROV@9$_7<0{Tvom+e^Yr`*T&YCu1)25EnL~PX?dl; zv2|TH^jEPuf87&}%PU(pt}lSSYw|a?t*LF=Br_l3(B8tzTT|dt^yLVWW4^mPLz~)(N_X<}Ka^ z-}QcCWi9`075Y^b{MyOwRPt`{Z1RPR7A;zsmxmuq!LO(+^R{&OK zC%PXf9ImI-y@hTn;O7($*HU#~>U2G)f74EQ{)fL3PW|#?z~>r*(JTv|=t~Xw=&vHJ zaK1c_26$eqv_#i(6LhM3>-7b+wX^rtk%S2?9~@_b2|1LqV<=)l6JI?Lz(#RdV<$L| zJRG3n16ATZ?|Cin8ja$Vc+Z&@^gJStYGE^stB`^|#*Y~;cR~t?tDraTyaloymkh(s zP02-JlHkh!g}xa37UWVe(SvxWFO?I$M*@}KhoR7T^ee9e|2^8rqo@5%JoqEmf&Tz- z;^z#v17m7lY1iTZ2of)%ua+iwP0B%I#vAtyDX&|aK+Uga&{IE`C0w;FkZ`q0EtUP} z$bPD&$&{Y|PV$eZOYx%S)mJ5-FH5+ZPp`>-2BZNMd_CalYEbmF8g(joiZ6$6mi5yD z%)X5)(dS_tDksH-y@e~muSmFE!j*gz(4X=hLQRzTNfvmXgiirnwI$FOCH#KCja$ZD z2DseHamXB-|?RfHaU*-q$7E+RslV+$!%c4E-4S|HuG0%9|h`g@jwjO_lSO#x>3tmxPa+FA`4uC|*c_%?7wU;^A@~Ho&8Q+yl%a zIhc~y&8Zptf&p%nt4YF@9u$3xgj?l(LBg%_mdcQ$#?6!Ct}wuj;|3*MjjQN)Nw{^~ z?@73IT*_OhQ0_ugO6iW`8Wjci@ET)H@~Yq}ZXB_|ReTt*z*QU=w!qcWtIj`F|7yLT z1^%cgc#pK-atmCo_kIgpt@li%-5quH&28R5H^UyG(YwOi)z;AT82`!1GCYtOzwXF? zjRLh``HNMWGPW=94eOITCpn@I^P0&VfsqgXT z2A>HowC8J*KX*4ai(4=r{d`4T8~r@qiWTctmNt7^d<~r|+Z(%DF#JvK{*1ePcWGlMByHJcSgoBg;4QI)o0>{QZA1I$g6*c zxH%&8G5Tzc;++h~(H!jW0HpDaxiz(P$>8guilM0S)%7{qp87VHY8AX2jnLTm3QB0= z$*RNZHbJ5IRrJ>Sf0N^1ul+|BgyLKEmk!~rHX+=KfTA#8t+d-@`%&`C_Rm@xSo!w> sAJ2Xc=|7j{@MJ%71=U{V(EBj3b!=*Kwu9dB!{btZVuPOKWo`fe02zB60RR91 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kab00779b3ea6268c3fa8c5384347c979.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kab00779b3ea6268c3fa8c5384347c979.o new file mode 100644 index 0000000000000000000000000000000000000000..d7b7c9195cf32edbc764219be7019abc3c92176e GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZLOf>T+13MuWI#e7UYl9bk{Tq?qfbkM zmLVrXWF2ELm@%`l&&C-t@nmQjGR{tB9L?rzf0 z?*0CHbl-c`Ai$p3Fcfs(TmS!8)&Hvc|Eg}?{_?Vw%XLj-s%Y#L7ON>JW0^g%dk??$ zuvAdGS$B26t@h{HZrlH%7ur5EjjLagV0s6)jz_@BTsP zw`16~4>#zc4=Qb;>o#`e|JhjYv37O@v`4!$SmF1wJ$)aRu(y7tvm?_|yQ|OHCiV{6 z##3KMyVF_W|J6NxLujXKwzq0A_VDSa2P!Tc9N6*R(*rBdJU#GD-E$4J&){`r)~RoY zdrzJSKj!>S_{71J;o-Tb!o71(hYzei6TW=lOnCUfd*P?%z8ij|372^(-((^ zcN`pe>glHkQs*uT&1(Km_~M)s;r;LZH2jqF&2ZDn%i-RY?}4r`6gqThVEf5~1N-Nm z2v?jr8E$Gl6Yf3uW_b9K%i)VVE`w%asN8vI;NsjT2KKK;KL@@Op4Ec!Prn&1zi>Ic z{XH7*k3*rk7;p6<^m8&iyXD*AVc;t+oDR=AbpgCnKYtV&K5%H@sku)NgifCfU&O2F zy*I70xr^(A>-6r&bfsOG2TO z2M1=KdSW01o-fY*c6k5XH^CS50NH!-N}lzFP&ssY@e$BKZz+l24u{@-Gn}&ZRCv~f zOW^BLc+B)iL&Jn$4qgwO3MYfl@dic^yf#ki} z-%n2Lu1N@z;1kH2poNYoc;kM|kFlv9`g#9PLzl0BL(OBW@ z@BvLD+=Sji-SHT;JH}YytAOS3HlnW|!cRV^ED3$Mp%{MWxpsZ`YsZcL8_yp>em$7YuAM8HaAdGvJ96%C z_3J|iSRc(Jru9F%0{F{<$Fi!j%Q z_9gTU{S)gOdPqBR>3{0iF1^E!4E;?)Xy{w)$fbI$spDAR@Q)?}mKYj(h#mRYhd}$+ zNuhHa_C88;4Ll#YzWY_vToUWMkmnNkCs~HTD~>lk@Mi(7TXh_?3jvFywd_ zav0?&8HQg;3=Kmb=%Vl9ICkVRbTa%>Qiy5YapiI_&$U??KUrgav~GM{`NI&weuJ1b zYU~3~q^rh6T+#mH%8~jq_Q!6OKGzeX@riz#=IP^Oq{JWNdfL>hA6IJJ$6T47YbkR) zq11&%A?Fv0Lu4DW3E7GIdk1^)bEwQL?GB|CMjSR`BI_;m7a6X8#8J#r%OCxpEkGHQ2Fns?Z1d>Bp5tq3cP- zSIK_&^b8sH`F@Mq9?bfT_t@#Iyx!9F@bb_KYO``xGly!owz{`Qoci`&5D+#Z3 z=AvOolpVf6tFO78RRNPrSRFG@PsLT&m~E_0(`}5|$6(2m#pYyo++sEM z-8rvgM^o$8wphKsQ`UFRi{8|?wfMaqY+i>i*fCFtR$JfM(d=z+uk|)JZ}WCE`U17} z-u8}KiS6*UwzmcR9f78T1>0JDE#Bs)ZLRr>+VX4veK8oIHotH|?ZUi8wfT8@c?An< z@o&Mxyv6y8mekhfK~5ppJs?JVqc`AdxOW5z_}YDeosbAt3iz7u32%cO-g`oj247w0 zw%V-$Z;MY@#|rSyGeI!T>S<{Rc>Rrh()j~z9c>+5el#asTWfoVx3weh9&q1!ufN3? zsJ|!74AXE~thVGpxpZ;^8Ugo!4V5@%pXJbdW(q6Ya)EASU>Q`5Yq4tAF7gPOiQN2h zKMlE`hTLswVI0@LpN4$SvUk(6Mb@zbZaodT`}Z+wkzR`p*1Wc@Tibmdd91d!b!Tl9 zvufF@T7jv}!&a@y+s>nR6i+RYzuZ$f+07!;EFf*zEnCY{4o|d#*`IRw*b(rJ*sR<5 zj@^uUlEZOQXeloOp5co1)14y->cGl0+ylm#t?0iCxDDqQ z))GO1aqH7#{-clS)@i0tj2>EO4F5qyrd6IoY0+rJKgXlp-9hagnN>nL7S zU%kG(w7ke&g8e_vQAD9b5+YrM*H!}gJA&}VlFkONn@$&+n!O!OZLRK(wzlT>Id05s zpSRuT&M%m^aE^ODUFH=oWyPIM%?y~Ew$+uGFXbFZtcT)DP({j%z{ z<&~>zmu+0PY<20fQZ~2JUErSU4b(Tf+gorRu-NOy(8Md9ui*0!3$Hua6M)<2I~aLF z9^I#ld)zX6M(qApj&p2{sm#9B4!fZ3V!P=30UDjb1938RC7s!2-A-f>XYg=bZeU&VfsoOGu7HO`>a%MQ?e@+`%H`vqqve`s|?&$fVU~B%#ZF} zaKtLS>@Lv2VEE5p_Nl_lY+37=J;lMuySr_;UnJo>aJ1Ee*ES$fXdJqJEU(w)^&?zM z34ua>F>& zf7KNm`$abEIbe^lD*zDF4NkUqf1;g_yD2-Bop7@sbqN{Rlfa|Do**WY1kw1*x^%XM zc+tCUf}aOaL79%%c_%v(u-nmsMfwk=vBCK)H)wm{nVqXxL_d_oUO&uc1QTq(ne@gE zULVZ17Y213;Ta=5Ss2OAm}s7*G8V98O~AiJo!#B%%jC zH)tPEXPu&dqUY00(4&9o!$#uxt3KfT=R!UMJ@`qJ_5wfXkLbDnGCDNk2X&p~^H+s! z#BWk?0<6gGo1pN5KFkR33OFYazp3XUNo*jE75*9eN1e<22Jg?AhMK<){h|LS_p=Du z!|s4R2P@fwyNQ+x`S`RPg&O2rFmgb^bdPDxs#JUz)xX-ji12B7xkXq2@&=I0OSMkzl?w_ zLLM0@5cp+X8oNRaXu>~M1oSkjGVNo|k1HU2-S%tf1O4j(8*>t#ltA_kPUpi%`oaHT znmv{L84IO<3-kkgMlg3et()*K-aa=7a}3pJ6G^^-Y*zOCQDCWH{EI~Y@bs&Zh!&%t z2FSDZu9m{E%or1$BY{D)5=Wdjx)F8t*?7 zb%9UT_I9~|$0ADOHwwH9b%9UNg*;m%K3np?MdCH!iGRPq*Ug82J^V1q$NiuDD`>NE zUsC=SoHdKYiQ#9=h}K`@^M%`k&p)&$ec91J=jUykUY7M7qjfbB*~jgl3HwG1=!;aH z><$0Ge9qTY`=JjV}sk=EJ|R-bek5{AX$cjW?D1&tbCX)Tn;Ul!U0iVm+81oHB*z6J!21C6nsh-}a%s3+_CD|>#4slT4x88Lm=wOaH|^V^_rK3_AqzB8o$;Qtv>eb;4Uy>MCd?Q+HR zT}k?K#pXAkA80T2js7!&&=v%U^!JGMm*xjIpww5ESzhWdx*iM`v9he0CizCwSI=(b z1Lx3$)~}pDf&4auzmfb7i@(`<{Irw&r)fR2i~NDshdNp>MSB(BrUxCI-VyUR2iFJr z+j&hN$)E66z8+)#t$_VR5%bR)56Qk~i9oL3g~737$^LwN#EY`zTwYW0pl3Iheigs` zB%dCWFUN^^DLyNGfb*NC7gGH3+h8(UziIyHJYMRA$Cn%aO!4&Gr`i8V|2AWMwx8IAd_&1sNBNG!`K#+6ju-h3*)Ju+&SOCCzhs}A z`zQRH=Zg-RUy?k$sQe>fbMW=~f2jN^+4eS;Gc_L8GbgqRBA@2#S+b3y4fhYoOL)HC zW%wD%Ti63UA3*p|`W{;{&yO8EKep!v9mzBQF_P^he?oi6&&T8Qh1Xr=d?H_Yy%O?? z^&~~mYiJL3Ybu{f%m^l<4u<;4KdHYI3>U@QGlFAKH{xmf8u1CJ8}W%qw~Y8C)MI$d zddAl`yGzECQT@A;VZG-5;on006Zpq|nw(UWe_(!|LzV0c`y=1aBzF|?knBGxW`Fp< zd4A64^xU2_C!o*5;1r#fTSodj`xmr+O-W=|B0RqtoC^O+&W!rk;C$={axx)43h0Mu zzv1BilN{ZjWaUKZFVKF&k<9Hy{=@etZcd*?^uXr^lO0iiLVlYbaA&#EdxYr8pY}NT z`iSw;h#vlwY4o?;V3yn1zHpD*P4kV@bAPlsU>)I)7@yil{qe&z_Dq-~S>#+Kjrc3< z0ljFSMEMT#X}=J{(CUqjrpPV3jv)7gnyg;{V>fZ<9xvSWAb<4 zP4fYGlfMIRm=DlTZqV%i@b{@?U!L!P-;AJPK0sc=NB#e@M)?%{3x(q2i}(^E@-OB_ zo%DNtKI%vps6&$HBJJNC2{HYWezFTl z95^aKv0=S9jq&#gd-&_#AH#l0@%A--zNhqa+wx$)&esp}H@-ffHQCdepPBLz*zaI~ zAwiM$vz}*M0e_t*925`1-x#?cGvzy!&-h7T%0DQ7Pkx383kiMQNj{VJspd0&&b`tv z&u60k7t3e(`Na7&`%Uu~#J3yfFMkUELwx(J^OuqNiX-`DFVatuk6^yJ9ORGudWmA`9y^mJ<>0p+D1lj`i8Jexsg+^*TzA_E?W%`>jFEyk29z z5DEq^nIDgEX2yDcK`F`e8%vZ=C;^r&UM#)#)({7%x zWbu5(9Y0@zJXvmv@6miEXvjB4<|~M&?yP9OG6em7@_ZzV)>j@c(VrzBLA*5OBMyu= z%?*A(K|Yct7JQ1&Bl4rWHyPtD`(TjS3BvOaoZpvqzC3EY zI%bMjhVym!&z;1p=>8k)n|Z%IAD=Ibw%^{v#Zl`MkI!_ZMxPhuP&|2n&-aPrA?0A2 z%M_omewy~zTPZ#z^8GoFXULDz_t<&5?_6NV|I8W+7B=rOPVLvmG zK9Aw+^Lc#UxO;qTfBrDvpY!umo^QV~AM_Iq`|-p0yum(^@5i}+{1@7f$Ig2&9`;YM z^DC_XBlqKzr`$ZgP2&DOIevUY{FpSE@4wW3d}Mqxo&V$pCr`4(x7hiQb$^Tbdkg#R zNyhkA7#ycFe1IkPhcrJj$Bl=D#Qr-Q`|qrA^!c1&|2>8Fi-vgTqzy@I|Lt^+vj4_` z3Y9yFca?Xr|4zqh2?oV}742Q9NB5)1*Txyon}E+So=1`YXHvc?_Mdz{WJdR&eE&`U zgc6N+6DKk^>Xffum=Arrbb~%&|8(7=Pt$&6^!)V?INvWPnDbZUw`KcpFTNo^GJHQE zjjf`?CSDBrEG-FgKb_yfA`}n#8`d9uUd+#X#d(I|Jp34lZs9!qIL^aC*`4@4j#V1d zEc@#;9s7OuH4t!r#{ToB{NeL(_=7lK&!W#y`R9>3ePEtQb@02&vft+WwWxefM)L9V zMG{i1uXb_}-hL>Z`lpXuWPe)h{6Fv3-s2`pWyQYQ-1Op z)FIXVa$f`YHE>@8_cd@|1E0DE=T%V$duQhy@9kx*&tCKF@caK<8~GYXrL0N#%if%x-rf{;_Q$XM z{&$hL?QB`inrX-NH8lxqYOotYNdM_fitG9G#ZgU@Us~jKRAUd$0sblhl%U!ed&*TF zp^JE+j@32%%`$i&<4pM-wzrJ!)0QHTYjq6tv7+ZV6E#*1fKBs=#5;=g;7%Kpi1w5L zXxbd!pBnGg1Uu@*sl|!(DoXX(wX)3<1uvjR3k9b!uCii4ibjl4MLj=4Jn=EEz6wUD zFM5e~oVPJn-q4K2@cUA%cuIHlCC&v`*%o}Bgzu@QAVKWW62Wq`#KT7w>x$o=a(Q0d z@$pFhMF9ix`scYmQF!9xMXm)r6cev~l>aR;-76c!qmRntXKan8CF{_16viQBPb#8k zgHDq?7JuMW+?HgMeIslMkr}s}LGOB!(0f#F8YvDwDz_^>J}S3c$Q`ebs9a!#WL1)3 zNECx)_D@AJ`$&%>OPDy5G*QkV-OV@~IS0+0+jJ}EadoZ>&Z48D6Qk{gu8YcAQ{v57 zp{HWB(U^6#DnJc-9DyfpNC#_lebvL5dv5MQJ+wLN(L6R_RWb4-U`2I$ik{<0{2gB@ zv=QD#W2+U#x?;q5x{(G|zwuX~So?jtRc4oA+<3N)dM4=N;guDrpOI1`sYgo`rjx{v zU6HP7)Erfh?K5mTjZ&1l&R*o&nR08RQ7$r?ZsO%C#Odwi0&cKt(6NqC21u-sMo7ma zve-Dep2>SHc^>s+w7wX@Ffy}_>;{Pee?QhkAMi2L8vR+Uk)OtxpIy|9$Ngu=jn$r` z)r~Xkehe3D$Y?tIq4(&1HpQKqqYqHQEBreg&vDjowxHMzoLaIr$t+h^`>C{GeFr@3y*0KXFz*t32`O)#i>?T+rXu z`%BKh`7H1AcDDC<*RN?l8Fy?V*2kjWz7ab%4?HLqv-dG$)KjzeEnQz-ioG`DXW!aY zcpn{xW4Eo!s3Y*tN<88-EIN|4oTCdLH{tum_v6I&Ki2=qlW{JAp?_WD{dX(g=zC|z z{v}?a`JcJ1(Y10H&KGDtiE`GC1DiO{Fw$=N45Njrx#iluLICiQ4Fo!e6A$?yqpH#< z<*hvbC+@R-HoMtVFXX)V-518Uu{oTJu1Kxp8xama>lll;cv_Fy3zB0NI_%^y2g9955a#)ZzwjL1Vz7Ehq}`4 zqk!{+c}{iXw^_Uoel%nvTFT)gdl1j*;%;rC7tiU+Fu|zuB&L*&i6^}nk>WXBv3``l zsONMkS~TZHJf|yDfuq;xbGjr%(MJjH@P=kOKc=&^_sDvE^aEu9h^-8wPMd&u?SCxW zeaWI)YkT}Lq~(lD^t*!oNE5{&8_;@3V<~o&hR7b)KB--mobZiQF%&B;`Gx#UTr8=Q z_WQE^2}4`Dwd$SxhuV8AY&1xdyMd!Zo=3b}+m{m(U(Xbm*!x#x`&) zk^Q8{i+1P!SX>uAD6Uz!ko+bh)c+HJG(L(6HGY!-Qpd(3X``TZXumJc_+OFjb0sB> zPsO4>=9BF`mhq3v_R4>$J-M{C{X$IMr{awN_01qgar1Y^tm7*`szCc5wD(xX@0abB z|Ecj+I|ZK%K)m*E$7xSf!rGpGEDBM_4>Ozee?_*J>_;zF{jWv){~AYr)&7cX2{+~T zw~kLfK|0-1CkZW=OgW&ss`doF({?V*C&#zk{+7>r2ZLZr`oafuj z`(3$l!Sc?!mZf>%!C$vvQO)+^)!Q33R=023xcc#xEvxez%4#|rRy6Oz7_H!=Y<=1K z&iVyQyJ)m*Itrq=Z>8w=KNza_c_ z`J>S--`o;x++0uzeYBL3%v(2;?(*A87i?SOEnc0!sb$-;?M2mHn>G{%;*MFqYvU&H zU01gJ@%pa9Cdk_iT|Hh`Qkdu4SnRKB@xeapE9zR;Jidx-uws3aw`lFE;?~u9^)2fg z>o!-nm2C3X)cC8HHkUNEm2TKp4JFj75MW@v+-V*=go0=9rs{Bn| zYg?L|0<}B5b$NM<7cZ?V@Oc+4ShS?RV5@gYePO|pg#`;2*DqbXv{um7H*NLS`)>3T zE9>}gtI)5i;MY#>q>^`&XOrg_EL^y-ATMtTr&;c8ZuixS_mUd?pp4Jkk{8TlY(wAVI{CWA@L~1eC9y{=^^Q#H;)Xx6@6g{m$oeG}f%i&vO{T~IGeG6Bj z&%-!WPKgV93s-_)m2kU+EBVHuKjk}w8ZYq^Ebu%Dp9HvSOQ0`G_+-G1TgE*GxG_~c zF5xsTP7|Z$guJ71jqp+_C-tv*Bhr-;ZtZ8Igip2fqx41n7*C114gJ$|PE>i$05{T~ zxB>jC0dABxSL)r$=S&H=%G)muWR>@I3Agt1GYPlK`%6PVM*crCz>V@|%K1a%8e?SM z4d6#5ocJ-WX3t6ZBNlkSf!=7}*CgD^KRrc<3h^^Z-17H@60XKozStoeNPsN{xIE(Fave3mqkr53%tASslGn|t z8T*0(Zj`H0!j&EreY1pH<$Y1Yt@4)2kfX-UljE*5z>VVuC0vcG==Vstb=>bsxOH60 zTc}X(LQ_iVj^Y{>1^4h8V~z5v;3{rBXMwBu&~JgOI51>^tEE?+f2#h~dOs8VQBm-2 zX}=W~xLWW17PwmPnMk|aYwMa?y@4)9Yr1Z{mHt{4j(}UJsO&z4c=1U&Zc^w2`;qf zYm`5CH#&=3F&_PVMQtnnJl@Kc>sOUFd7FLp9jn?JI-4>4E$;rf4Vv4$4YeI@wGEsx zArROTe2S`&#VFzOwtynI-LY71!!6j%8`#!f+wS+(kNjQHKwE>i!^`IJcj1u2N-H5r zDMtyCdG+lb`HOB@8+XZG_|}aZ_Xkt&WCL$%^T~JfVm`L3-)_B2f7w+mu%GaKAA*QX zwM;!-l?m!(nSUsxT&7i((LSGGGA)S0J@W3XjAII+;+3k;zzwBb9tDtB|1NR!oXp4Q zvo(r$G8{*9u)hb8#y95H)Y2t`uZJpzqQ+O(XJvcp+gPep@NO_dW8*6*p@}D}4y#%P zh2mGyTkHQ_j(?-}A6XEJZ`EHqgtyv+a4Q3f!hE&TZkO#x$S>PJV`*UJ-v@j=`#Gfl mT#>_*{m2zmdzC{Uz`)kAsmZx^ddClsOZkZndXks5{r?9Wpcn1{ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kac95216b414b7faede973efab3c4be64.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kac95216b414b7faede973efab3c4be64.o new file mode 100644 index 0000000000000000000000000000000000000000..a4cb967f91bc8e49dd0f545783491feecace4f69 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAZ%vh+13MuWI#e7UYl9bEj37>N1v8t zT85kmk#&s0V8-mmJ{xDr#FM3ENSvL_IGWAb$QvKxIGDkjY}O}}b3$Z>NPrpK@j8$g z_kMppy6?Se5MWPiSnANaxBmaHs{d8>|5e@G`{ku8mg$O1Skd=$J$)aRvbTS&vqRI;yKBzaCiV{6 z##3L1yE9qQ|Is~tLujXKwzumr_VCGP1}e|(AJ~5BnSm9jo*DS2?zsxuXYn~S>%_Ms zy~mG59&>&ta%}(c$ne|~k>0r{Bl}jJid@`xDl)w9Qsn8m??oPJITty0^1Vpf^oPU4 z+xHJV{me51>2nu{XElE(a(>RS$lgmoi#+XoE7ElQVx)J)CD0Xx!v_uwY&*VxVDH>x zk;+rYBTcQRBE9?HiVQz=F>-$UMbIn?S2zz0oS*yTz}{8pXWw@svsy6z$+sdE=PpLJ zU83>+JRF{j@m3u`KgT1pTfQ9`2EOv#$;hk|=fFGl^Jn4VeFp}fp8L!|_~h}(d3>5K zy%m{#;zXpvc`0&!?m5sb3D0VIa-bKws5o~FeIJj6kDrWGI!{Hyb1z1oUPU}V5)L2V zKQQ~mlLKM!e17h?BYWq*1-_sM$li-j>Z~t>E1=8s4}k`HOH2NCB>di6k+dx*BD2n2 z0ACj(W2QeE9wz)U@cP(^NNVzL(e5wMZr?x!V0)8KM1~(c7fID}ulzLmt2M(HUJj=2 z&iz4Za(7MYYq=q^qsPXI9I%n5ch~sfQxi1L)ibsUy@cDWv!a;&ffM$h0RIv`uvue8 zZ@>pMjc^ls2X)6|)b0plMXv#t$J>a$egr@HpsF*zk1=mB=hE3X^(^$~Li`M-diu7b|8L^D8lOtscjLMr*W>v7FWjHP z^=*Y3cIfg!*wtXi#;HOdim=QR<{=K;B+gY-QJ;IW}(Ildr*WTFP5$y2#aHpi@JL-@K|#Sn z+}15vu&ChS!o`o&)fGTaA=h0XMth?-=x?}t1PS`v{lUi}5v&ySH{TWB206TUg&+-n zU+31kEkSRKUs%Tq@Xj+qFwN>|X$X1)jeODtf^8jb9bEx5CtO==dxy8Rqu?%Z-+FJL z#UHG{E6fbja9OOj|E8B8`u4iBwRE%q}YS%9E2$_l8 z_;N1|xtE6AX=z~s*T0vBe9p3W)3Qa@u>x*B4Y~98F=~-sgALZawk=!Q{T&6YuCDd* zx)^5V(v@`rQ&)hkT9db(NADP(S|WeBt8%g%MW$Im+OS);mL(jXYz4Ewyx~BrZchNN?eRp^FOy-F0?w;x3w~P^dj*j-V-+x$m-Oio_+&tuCG^E7NuHA(+mRgQ2XJYi@HyE$I;2wt^yxvC zPJ;)kFWcBJb6L+mdz4)UfS7J@vfX==?R?ygx#{efoBg;;$iSWg9{u%%Fp(sP#$Vc{ zv(3bd-fa{7+=mLvbbQV_*`c7_jutH1e;|Vm&S&`{+kM^Nox!5|ffV+}K{g|lWc%Hu zH@EZpV6MF=q}vG37~vsbXnZsmwWNvAHpZI~O3`EVCo))OaD38u>VM#13Oh^mjMJwO zJ@EM<`*=F*6#WxDpJsv{{X-u%5+_jo0p~v-@)_vCPlmJ?_(6X}&-Itpp%Fi*>m*;G zI$|S!Q$iD9MQ-0Dg%|V@MtE1yIf3|1KNC%10~xI7uh2j0T;4Z%f6ffl0zUMI{-4^* zqGS)d1NI!OV)ySPS}Kt5Vg~z4*VrgG2jTOb(2qWLU2SbGm-o$F$U9LdUy3@QAM~Fe znmCdCZ7kQnsI${A>qEzE?C`O{G)|KjcQLomlstu~S58^~qlFB}Gz3dTQA^bbzI5{+tc z`dLoWUsTGsvF7UCVt>wMZpIjt*gMD^)Ip*;BcK_6L_Sa4}Jv4g;;b(LFeD-7)n@y@sz8{Ou zP%-H3pvT%q^hcc>Z%19=OM!O-?>6Dxs0;j3;HLmzAn-$?{S?#%{;R-e1Md;|nHjwQ zY}5rlRomU=0v?Mfjo&EnF4P4+Nf+{LmiS!B|7MBTfG7R~0`Hp-|9bF2l8^g8`B%th zf6mW4HoZLOc}DAMG`fe|KO6Rq70?%} zI@ufkf%%-z=PS+UUuLk97c@am{tJBh3qP05MSWy{W_tK*KG~D|E5;-L+Ds&JK4gWa zPpA0D{qYyn|8i6RJv*ny{b}$3*0&@Z`5RvpPR)mZVZD#}7x~ZBBpPoj_n(7gv0 z8gyz!!6`{Gf5mz*Jv3zs(I?0KZAv!Pxxei}dl%X(e_KWRo#Hb2n*(^au;$mK z`3LEreeI2pC4#1^^N|sLeLfji1zo0^_S)cH=xv4j#*ynFSZ^G7PIo4nI`#0(^t<< z|CYmkqKNrtjfZ64(?lTG@1oGyv1ETfKH^1rYCf;2c+j&GOTUU= z0g_LT%a`XwycFM+-pBdP(2FSk1Z*%Ft=}|%bRI8t!sEvcf2H_)1$2m?FRMC@+K+lC z-N*v`fi9-gt|z&8+*ruZM#?W~&wQR6+gN4%qJr04N`T|4{sWg&dOF ze?amdiN|;76Y^N%yKac@xe4Mqa$quv)UO`Qj>qf4`P%ZJog%j(zUw;Km&bSHKk#RK zbxi);okHttd8!gX*wgHPq<@<+KHHD&K)#{m^HILzaQ^z*hvP-QL-tFHva=YF`!Ct& z#{LQa=J}#S=9eT7FDm~C+8lg+{_iS(O0~U%~&j%3xlex>5%JX9f&yVf-AxG-We~jij$)C_3^7HZdeBpH$IiJW^-l&3n zVm(O{^cvbj-I~g0lCwgosDq(?@=xk74a3Fo_N>qt)QxzWzD9f!>PCDr(k&xC1@$=I zvYzqv&F+%%WK{pIVpy-ae*`wu{sjK9mnJ6_A64J9I%e?M~qMHWB&MI275Nbkt}*9nnC;( z^?+WqPoaDV`LwP7N$|?ow+t8ZGk2^%$e$UUs!sxa)P?yxRF$3?%+)7TeIh0h#>aSh zIjHmfCi;VYv&mfoG#?;uQ7Ah*7SF8w5dS?G@A~}E`h|c_1j4_~{(g|=lW{&^{W1AF z@TU0yyvg5zH_QjPEZ^ryi<3;WJS;;-Ce zb3A(q`Qt8@PW}yl8pGbKB7aJ!7J;&WUBr_wqDIBAUJRkiBBJQFPj$V}C3TF=>DJCii#JPBc&= z*R#|`l3ehY8PwD1#3b5H_CKgCi{%Rf|FaNx2FAySfAxSL?brh!@lJR>iE(}i@CN3~ zE(eYZP;6K)PGbCB!X5$N`(xOzDBix#&-avmZdo4e*ZKNE{>Inm(rwL+KEEyb3T?)z2!G`J31xrS6X%cF6M>DoFyBvoiunrpL&AJz+9>&od)kfj zl^mY0xD)3qkSE7Y@jaHWgbew{$b1Fy)SVN{SB9X!Po9tD(E7^bCHk}EBZ!x#e8hqA zrn$lIC&))~#DY)pc|?A6=jJ2iKmT9mBXdmgYvlf$vI`Ws|K1;Bc7pKy1LybUov(}< zua21FmEn9H{&PF=Dz^W|`exp5&&T%*qwTkMadFi8#N#s^snPdEc@$6XX+^%jax$$Wp#;~Da!%w2Y#KVd(R!QMdrx+^J};(5yW3yMIk$d>m;7< zV*L~6MMu`NMDYyqBJ|iY4d6}jZHVXJpD>3}0Z0{UObd>~Z5^A+i6?#r`{I9DP4$*ndx<{h}e>IcY-@-+w!uqwK$N zphD$#;$78k?7uUyT7p5bUqyQt>aqPO^0jfs^CsZ4jOS70|Jjspiv1^_581K(C*OaQ zKcU3p-NcE^jXLG4=jKD7F5RF{*gsRZ=+m?x89jgf6VCUGN#^_&`EB{$TZ?bVj|{&L z$Y3k!u!$E#K1)l2+)o#Fuqeet{s-$1zAxtIz2ZE>a2|dHL^p9BeiY~7pzKb5AIB<< z8J7KZhK~I{`x*$iKV$!SL;mo6IQ&7JujkPBr~La!oxU(nraJgtY1wb{{aQ>uCnNdz z`63A^)>k_@2yZ`>N&VB;EwVo?e*Rx@bMl(c4>H&n!H?L_q63`&ly`j;&rk6Efhj-v z4C;{TUb&}%dm6Z>fqNRbr-4sh19W(fP}`4RIk4_uUi;&RKaPI&<=*PwZ~D7`_(kQI zziIgB{CA@Vi`dNGr+Zwr?%vsXM|*o2>$BHBH~juT*G0d^QE97_{<=4>r?)qao&L$I zfBZxA9XnfEyL#GDeRXZp>RRkZ5Ym4(lj3?I{cu#%041n4#-4VC zN9bZ6sN;1F|7RJzk8!5`0oz-~_Gn8G$Tc4WeZ1&-&SZ_%0ASNRBJqwPJ-E}xB%?iL z0Gc+3_ov2tEy<30NqR{#eTvgPcCCEVM8ONF(L%v#j4P}Jkm3We<29p`O~l{YkFar~Y%E1uFF{SoJaD{M2qPr~n1(~uzcXvttXRubW3ighJ!Pq{oV z?!yH;t44ACuda7$1|{E#ywrM@%j-Lb58! zFeHjWGW(|@nSG>3ktIx=NtzgEknTpDjhusK&TYDt^MpFr1!u8Q(TUOaLf6IRt!as7 ztk6>l+Gxx=S~Z{sJ&wQ=H>88ry1w#3%sn^vpdQ+s^=KX&u&NmO5wK#Po~GwHlK;S0 z3T=dU(b#H5v91_#p01}s)oSv_XhLJ~T6Ag^o zPEN#`o?hd;!Z*LR6zxh6V|dsJFEktq;syRwhlT@;FRMC!(PFAUg}P#r=rO2Hbr?;= zAJplVFMYs+4HNx4mii-_8Sk;wt1WdJkNW3J1p40~PP7Tc$G>5nJmOZ{d{X-2^fLRc z_K~Q~&G3Gjuc)Jqb~8PV)`u0JsL#)i8!LCC)r~W;N5&xTWx}n<&u{2giQ214LQ`wh z_NIBMr^M!^TQFd4pGezQn$LUPs53o546^c{h+gFZQpW+LrRKHETz?99Wfrk^*(_@p z|Fy7T?K0E9V`={gvSwp{9!tI2Qb)FW{k6-u+6&(sExz3-8j#~iFZltp>r+wX0To(3 zje4%3PI#(wWol#6k>Sw}z@|+D8>=-v&xP3R*-bju5SX#j<4=57KslW3)vbC#oqC;y zI(0?5rcv`$J-*Md=`>1l`dWLj>+!Uk8;x?2(R7n2S0PSsB^PjmU4xEwgfc+lg)~As z9+AZ+$n{L#YsvGdAEWif2!@fFb!0b44ETGo9{PZfnbzpfVvYPX#{BG}W+LuCJ8rD@ z9IbAgVfSLVSVKnB*(b#tb$%~~n^$oWD^z?K8Fy%pnfN5lvbyWm)t};pVJD|)SCGZ1 zot%ugYFv|r{v>gUMvliOThfiW-;;E2ANr@6tBF?kTb=*69}PZ@(XHGj>htzFwD$g| zuA63R^2pOdD?f?oE&Cx;J~X0zY|qK(U{Y)aq2&j~N`I%-P5Mc&@>%6cM6Wh?wBmyP zj^1B#{>^83pSQEU&%1t2^U1hl6R|!X_4bU|v3cM@@tD1b8Ka(>wQuYCsxs`g89)2h zuE6`~FdVyWRYo0we^=rW-(k^_tmPbC__zt*FMb~3eEqU+iG2_aN&G`=94I=?KrSW@C+mEhR-lsq?%i<-OB|4U)eyQGdS^(4>GDQ zi&5Up^M4XP+h>!TJ^fn2}!5oHidreh!}h zQw^!z_)m2O`LIVNc*664URlQ2=5qYsa(u4${2x*f*@CW#p6x^MU(p{F8%}~^KdnPu zY4=gU`N2G=y8fS8ybpdfWFlI^;Ujwx&*|cBZK4;?>B=&}sPYu1l#Piey%>?=IbHF7 zl)tFwbSYXi=S4iHD_enM*VuEqBt+522yXL+W;#Env$XffdVTB*WdVq<45Ch(fJE(o zBHMk*qFQTv{AWnZ8JFmHIsGS16o+g;>m7}y*ijlHdszFVcGYsiH&Vq=thD48@;h;{ zq)OT!$o9t!ZRyskck&-<@3FAaAWiNDjtY4m@osHjK}h^OQ(WSIzarb`OOVpBwLR_U zo=njGsHMH?+1h@ZP)NX|2;}voZ2y6*)A;mT6Ki|&dvq~^|9&||+=ljatJ*6&9z@3k z^AOTd&RC|H8eg?j_J16WZ`5AVQG24Zwx@YcDzLaIW8|19-Ff0l?TMyaRAYbfUj}%e zr7?)Sc|=}mP+9nYbT!lo{5^Ob@kITbifr#O0WD4j9IY?Je~E<-kMQT^=Z20G*)IUc z#Q(1&`$>-%?autUxGuV1Tyt=6@Npj<5Wv6774?-eVcR zU$$5Nr^Z+96nruOiQ2!Dpgm0qYkPWG6rxT5W;W^nvTQHek6o<#UxW7lF@gN5{bkt_ zZp!U%9iM!Hbh_Cm2`!gwIiR|#_5{A&c07v9`Dqz0Hec1eHI7HTs(E2;r|KT*AjWna zB0J9h)}PG}i~#tZlB?v8;YUBVcQ4318N{Wc9`km5sjj>k2k) zs9fP)U$nisvZ{8?j`C&A%O0z34AiV?UcR9Av5FlP8y3_wZ)#aq=vyw@mNYME_H8IB z@NeS%uGp|(S*NdMNdb5W_!ca#-Bz+{Tf>H$_Kh1>J+Y!?RbfMUZD+&s<{cQL6?~Mh zD__@HzhFrhjn}ZgvaO_IN11Qox`O8Nt(CEH%eSub7B6XATE2Cyq$%Inx~^rz!gbqj zif%#SXmraqwS*csEv$k*TFOc0Et^Ppg>7XEwyyS;tSa2tvUTaU;+n3F>x+U3$E?_~ zVI%nVl`nguzN@GS@-{zcg9YgU%Dt}3W+ zS=Z>>RMS?v(OX*^s9Dln+SpdMervV2q;%!7@>PYK%9nO-Y-#S?RF2QWm7SZGR|Xnd z)^$OD6{`!^J<+hdvU%hB1+aHb;l|cAwT+vEy|$H9Zb9D#4I3&ut#lgX_ix<@iy85Oq-g^J_ zUa_*4zgvags)Dzj+)kz72G1rhT)1e_qJ;(Y`V_oIWtq3R-7o)^*5ZXSes4=bsDQDZ zd3W^Vlk@(}z4J?7f^iDx6><})#aMfyhH1N|YE#m4@gNtxKT;VG*L=hM9NZiCm`~Ay zOJ!~P{hsIa!}kA_{GT0POaF%RMbR_SH*f;+XV)n7G}d7Q{p;#}x%0@FZ|E=DWu54L zsBpNRRQDFTDS)3iw0v-!2`1!F%8sLm1x@_)KmZ%WrPfYx9(g!G z#RsY+dfxL|-ZdH}D2bjkE$Def9MQsN7*`<$eViXNT<(Mv5LZEO+<6OR8!lOfotu)2 z!X&|!{|kLF_D#s8V4?@{mwr@E^d1RRejkBC6Vb1{4*ZX3pNO9JH;Ld6UkCm}z=@yJ z+zyPXd8J*4{}+&W5q-5Z!Rt~E8Z*(jZ%TRH(gbRLHG!V`xh&zT?E(o`o76Jdf1d28 zTAEDh`R^tFc)AoXYF<4j`Fusf)qHwg_R}v7sNm}X&s2k=r`4!a!Bc!We6y_oqX4sS z;Y#!c7>CLUaba)cO7JTZZkKQ+-#GNAe1}luC4PbhULfI<09S1Z^hF7u47hR2xXS=H zriw=;oW{jzVyqmKcQmdMUMA(F{uOUTx zM*3sdfnPDejq>J8y<7R5DdARm`=x=b^1dPA)_#62;Z}KnW$4Gq|3?P6QQmAhe`s7| zj4Zeg{IG-*KgQMUc?o~W0`E7_8}0kLgj@Nir|3{2ekO@q{&$gtt8tYtHcGe}SHTY( z=#Aq(FX7g4e*myqeZc@X%GD_0N)L*@S;DRIz9ivRdCO$TQR5cKaaS1N#&JUuuEtgLyCmE??)N3! zIxgicR48|$DW!BragBW-IwG8@LPi@{U*53 zp1)DP?rwAzw_rSaeMMa>y&iAHighc?n!L^a`i_-t4V}#x{w8;S+y>2U-iEr4wz>w+ zm=Fl;2|h(t$YPXmSzAyM-0D~?zu_iq<_&IbuWJwZ>qovQ5k$-MgZj>5$^t&KZmFMRXHP58pp+u6XI+I;feyqJ&e>fNn(=$l=|0{aR7?m-Zd zsg|jyD>6ZyEb|YAl*_cLGTP@8Or`}fxJTZdmT^oWRJ>C48MvX8%cB7D8rUIjo|pL; zeYZyOPKM)H4)*r|()h;Qnp(PK@byr|P}KP9`kZV}eH%-)3f>JyXncGHB{cD5)nRq3 zpiulOdTaf^$nme&{v!)Q@vZvHgz#3I5N<_KQJAk*+HJD^2>E6EXDtn^{QH4VWIu=W mpUZN1vLCsEYOiwWeHhp}HZ?iZPJi*@aVbBsK~M6sw*P<3xFMSW literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb21b47d8c1c250b3a20cf80112ab527c.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb21b47d8c1c250b3a20cf80112ab527c.o new file mode 100644 index 0000000000000000000000000000000000000000..d5ffd45365e7e5729b35331d4d82ae5c6ff6d66f GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^KsL<8v#kdR$$*4Fyf(9QPI-%>aHJz ze>aX@`EY|C{-D|xzG`C!|F4bp9qwQUL3^ktixqz_*VF%D8GHLzIy*Qcy{GoHZF1j` zZ6ft`s3(&Z{~z7cKa6&|W_!CHV~?D8X0Yn)zQG+ApBY?v@|nSJ>Ygj0eHNdCbB=#I z(s%4=_Vh(gP|wBTw%id}{tPgW(g$BIodF zzW7#T?(yT1O6SGMx%p>7vot)X^@+hg=%VuMQS^N*5D2v2*^V0pZa=j@Z3L_UUA!5#`#~BUKx3LS^uxsvi@|;kJGc*mEj-iSKfeM59P8er^_ZC9BR-G zp8h-i>hRO7pXL$M1|D7keDpciAN?2Bf8kaA%DZK(|I9t?%D*mR{bw`Sm2)ZVN*ZWO zFxQ6nB=ryfGwUC|Pdj-2f9O}vzsn8||7}uu_*?AY`39`16IlPqk0t|_93H-p9sJFG zp#5(t;nN!)f0*VPcs_V_*K4M^B-VE!&w21qvJ8V)8uLAp2{K-Qj84czvZYR9SAIfs z74jM7VyWz^2lAZzyJh_&kgrn8H3FVT!1D-rCK*Y-^WbwF`u9R6=ih7L_aMi}t4ZMz z$nhTJFv?FdjJ%p09)Ud2MgO@8?BE6HWaQb?h-T1irhhc*K z7BOqg*ax0SS53*dqW#C!qxEI%kKHJJz9&rM6a5U$)6d69OFYKajOmv@uGYAZxw1W1 z(&l->>5EIk&M%aP$u?vYvJ>_9F7$c^`g0+E22(x#JJA0(ab1H?74Ca*-G}QjeEt{i zPvZJ-G(PP0?sG|D8|Y6%uw&D7p%3!YkE=_< zS5rzall|`M!>o_z;XLqT{-HlNTd#Sq?jJSgVwW1w35w1%jRJf~)(N%(9|t<$fvX3f z6nrw7i-sLlcKCyB{+13_157?)b<8|H9amjrw(&Miw=rfPhb2!Io0Hjb`(=^WmMDbI z{-=;hl5O#co74YMT=g9+*~6BwByco^C>L}zb#w+hy*}KjYj>-t^J{%aQ(I$*Eof`+ z^e*GeM47gi?^er-rLf$&D+`J z57sw$J38wnw$tC%(H;zR2AdZZZEN+ndRv;ewG}?tUReL{i@^Z(g~dhnMT?iz7Zwy0 zEW&MlQPJXphYBBDQeR&HIfYzzfEXQ3-k`tn&JiT&@9+m7g+#DY(BE=LcpK#K-VuT{ z`h8v7>bC~Ht$tx0E5N(J1i>_Gprs+`4K(pd7YMd@ws&?1(427XZ5^H7w$6e(zv@igT2-^Zv$dM!3s3);7C?eKRNu=@J8 zN9$vlRm)b@3ru|hwrb7Z4j#Q@cxs9K<&Mh9t{0i+0BOT+*;}J%H9FCJhOL+yh*z<7F4%?!y@|xCFDWOL{f5qW>=7 zHk@BrOB5wv0_mO#_})#|jP%EQdS)|6^zokA4t~oR!RJ{!k#)th{p$dTw)RtW2trkS z4&g)f)$1$ED@)vE*#F}kMHD(DA<_r&*-D^5X9&Jn*45~B)9FHUi?_46z0KX(-rmwN z&yAVw_jdT*g^Ly}p66aqmj%U3S!q{uOQXA^yQ9Td1o=yW&ww>G!=-Rr8WSFWvJ zzpQp`W%cU%WgFKmTV1}aoXxLxFLKZK1{<2(9j!PIc*yI<(8Md9ui*O+3$Hua6M)+n zIv9CE0o`Ycd)zX6R{Z`(j&p2H>CC>>4!fZ3Lw3>kJv2Il2jXPtQYN#@x}7Y2sU%jn z(ddS{&fKOtomCm})F;(#=+-th%Jh{uW~#So_BrX?res$t_SqO8MscOWR#~{M0&mmO zSpeO;;D|N&*j=E3!SIiN?9+vp*>ct~dzyohclX$E|Dc5Lz|mGKKHGpmp>gQ?vAkZD z*N<>5Cj<)l#pv0_Snn<^%6u@kx^BWJh-52X&p~ z3)Dnx#BWMy60FGWo22l9KEep^3OXkdzv-u=DQqx<75@eLN1e<2ChyOgfm*nxKz7zV<$FHxetK;&%nG1O*>*PyO2lRvf z^Fxy-lfRAU`WJO}>SbN&ry}R;(!8)I`iDK7+{wuv;HNmqCQjnxi+b;_q$v9U0P+F& zUsli-C6A0Q68L4^8oNXcXu>~M1oaH6GVNo|k1HsA-S%tf1O4kk8*>t#ltA_k&E&&J z2f+VOhCQA984G1#3-kkgRw#cat()*K-abDBa}3vMlS#h8TvqYIAz-Os{BuNq|IEwL zs1~Q6<0Sn>rF_e}u}sj!mi}zw2mVzYy#Ib~FHMYB9Go?W%R}u?W?+8Ki?Y*bPxGl@ zI_h(y?1HTOE^0GF^XAZa^N2s#=kadGJU-v(s0(}<@NVGUCcGPUfnNsvG~f#aeps}hhPuFi75Hr6Jpw;F zgZH0}y1=Juk9WI($0ADOHwnB8b%9UPg*;m%K3DR;MdCH!iT{AW`xe5#?!TYp&hJ9lN z^d+iJ_J)68KIilKO7r;_8LadLO^}oS0$=gM&t!8^AKjms9{!q7_T>JG@yNfn5Q&@* zS)rLTDgJSP{5kc%!qk87uIX`q8rqNbEy+gy#utT?3*lc_?_>T&{xdy^#+%Ol=K$Gr zdd$CuoLX^kT2jnku^!9}O`Asa$#H+1mQ8i;Z+p<*h4#wdR+E0GxlI1%0NyR^d1N}r zJ5U$x$sg^&OMioZL%%a)`c?KO2FTv;kiO|2Me3Wzvn&4`DuaGH1zyhQ(fXHY*&h0z zN$OVqLHg(4ND&_6RlKmo)L-wethl~?Z5Dmg{5I&D&(|!j?<}c5_fMEW;53@h`jz)*kl$wTHoMlv3fNB+G5@UbknDSk2;}-*92!5K?9azXyr@Xc=QR}%dUs*z zSMe)A^67E;@|=j5;=9s&IKLTsF~y&N4JM=Yo92(s`6X?9eUkYB@;A(%`NWSZs2|-4`#lB#NdCj|_zrzS9!q@J4e>oUK|DteOeT@~)q~mbcs;aGTM@KVTMs z{*14V$)9^tXnn0nRRRcmn*ERTZ!^Yc`_Y}qH5^N&Tf^xES7^6&i=S5l_?Ch)+V@h)+hkWyGhT z9>-hOGrqprT{51G>EG21>oxa}z!ut{z(4lVF}@A?3jNIEyR8xFB{^c zfPR?v8xHP2sj>Y@PF{@uEbTWOsoY-VKYV}U=JYv44}5+o)e-Y2<+zRQi+8)-G~YNq_eYxp))D@Q@u_{xA3w}s&qg?sMNdaF zh`-`q(2MpdlnzZc_On;%-g5YUN0__x{L572xv&IhbN zCVvOsG#`LB`8)82`2hXohs^#Df1ghF<@pZy%?cUj1LP%q%>OTFluyCGP$)jWgdZUy z|6*>`Nxx?nqKU&Hjipu;o%ZM0AB#gw+8@5j{hhQE z4V23DEOoIY7yM-g^>jKhiT05F4`|C{`GUazG{l{O@$unbz2HYXdf!L96JAeZoF4+b zf%&rAfujNx8`g^x7=O30N5J>~IQC15x3BZ_J*A(UmIwQFzJ8Fu@%8zX$)48y%#@G7 ze*1z935v9z^*rMe`0G0Apm+%W#>xGdDc_-dCO`sH{z3VB>a$E(Na*WU@|l89HJ=G^ z?v;LdJ`?l5cs|3=C(dNpubaOhzFjka`E&Rm;@fAPzl_dT9I3B(k$#GN1oO@1Ab;fN zEAUTqzOoQ_n zas7NHhvzHq#Q6&3$#GMBkL4>NL%uOOUqL)|=fv`rVd(FZ=Oa0^zVdj9{w(_l209AUyxT`F%y# zD`Up1!=`v;IA4eV+)BKP?Z2_UnfKcZ@%_SB`|aIa9JN02_)JG?^nFnt#glvZe4jiK zQVwOfOz{cpr)ht^mEuz}-=Fh%hWsdVx1Hxt*bii|H;}*XPD-YDo-#4RLHST|$WGxp ziRZgm|HOIG;SDTNJVU(5{CD1eo8lMpDLu%)eTaSkK=@cZ1OJ(bXS5%RW{dpp=Hr=Z zzhRDF7~iztfWA%fjN{Gw57T}FcvE~E=K1$0jAxfj{$h=1_&zCafAE`$^Q3|OQa`XC z_A`^|`xw4HpTYNyyC%l>=lApdIX^$;`SzO&K|k5BA3uQa8|;($ew_Qqf1>?({JaO_ zVgD3Azry-IdOtpO+V$hx6z=a+6UR5ik1136{!8u0N5?nQ`A>dm>J&?Si=Y2k_qUk8 zH?ZHHVvKLap$R&}7g%C{Nb@6m!bDg|?7wrd|IV2}-_IHL-_vNnXoz=C+K|Ne-%jTk z`)?emP`QD29(vl$e(}kTZO7W2Y!TN*mi}`u4IL|PghaU#f4V;G`!Ff0+dy?PB zu}V{hWq+NaW53V71_JKS*neJ^KYSk!e-P*EIrRN0|2|TuFU*sv4t`fz_S<~F7L(7( zNIrhPNJ5JB)lLq=+Ye__|MYc>>`#lI{}ns9d;uK=|7!KalMd!II3y#M~l3UYW%@Dz+WbS5>y*! zPrJ+`bO{gC@w$fpvkczHIn#cR?JZ+_w515-nva1#Ui3U?vc_rwuxTEVc*l?)+-c*I z(Vj8@O`FI2Q{%msWJkR;y)>CVCFvf!RDR62x9D87#+2B797-uEgyr zm*>Tu7?0#%6flsef1c|Tg(opy!Oy|O_f`j|Wc#@1+Bst!%ZU>rjB zq#}AY=rqM+@dr-DZAmuSGs>0_nQ^-r^sc7}y~pIHkrLoza=Q}aV{*HN+==>#$puD8 zRwWsRL@`Ka|5PNikM=0Cgo!gr6XOihU5~TTbI{DWO}BEMQ0KbfEH)}SG1^|}x}>5l zEzyh>dMZU5jaf&l0o0(!QF!8pbg)L(SKW`f=jI;NOPjM^&0_;r6(c_aR^rps^gKuM z@A*ohjq)xUTdgS86(i2mwKS;uO}qld+wak>GP?}pCbDhJGeMUKudG1*jF#Fk@+fVh zfl=GZi8#~KYn_+*=GT^@UG8NJ4?E$7hC@NTz<=t{aDee;RmU${O!X&GS4(NS=+vej#^UB=a3_}*CY?RwFG97lS|_nBRvjw%nR z(CR7Fa}9OEQ=KbQ8<&m@k9GhyZ3fs_qv?4r#AeUqq+<<%87n>h#CHXh!^vLVsu$F$ z*BPi&SEOqiHBZ&!`wW{-qm-ntvzNFYO}nwtC>I$^H;Hl;;`C;60XNt+=vYT610-HZ zBc$U|S!{w_&*Z(9JdgS@T3?J{7@1i|c7w!#zZ>hJ5BQjAjs7gw$WLR;&n{{v;{LPa z#%j;8>c$y%H-?KfWGtP1QoK>;cVoDD6&JBW#fQ;xhxV9>Ptq)_yKY_mDP9` zF6eLT{Uzt$e3tilJKOub>(?}&j5{_F>*G;x&!`=n2OboU*?X8V>Zw`#wyv)($6lN9 zvv2J(ypImUvD;Q<%n|r^B_8n|79Gi2&e4UBoA3kT_i6I;M3o+#9 z;Q2o_klKy^R9BD>dsKlZJpbpF<&15q!2d1B=UUJIAq9~w=$h!+J_P?I{XwzeBq;XN zI@FbR9|oKs%yX)1|Cz)4;73CyqNN-@x(D%`F7DPQdhwjDEE9|>Phv{hn0V5Q5h#^J6+odylL)#J*4#fcVNF>a+<+ z)c(h^-Ipw?wYJB9hP0ewkN+w7o+$ekW<8MXivAQy|Uv0 zbWAW0Ar0lUWs0fsRXb(>N749t?G+uhCpv3;n&+efi<>e=j)~HfC$7|fcmkdyffdaWdd&eIfo!Ep&KSKP{-eVd6h-|O?m)es{TiY+jr0y&T9hsm zY~EP4(z~H}M@v<8-P)ZM%UhN|Qr8ryUE8vvsP2)Yh79Z9s<6i2kW+%uHN3bv9@E=#?_ClY+YU0SW(y2xT0kz#%Kc{ z73(Y3cQq6(?WXY>H&nHkR_-kKEm~jDQn9TnHg3hX)!ve&?aL~*t&=nro7&d5Zd|l} z`wh_*6^=!>d~<84Y4f6L=%clQWZt@&bXVA3UbJnEw{&&krq*rCwwKg)Z`x2COgLuc z&W)SEx36OPV-4NK&5*YRx_Zo4R$SoUSQ_xP`eC2-Rlc@0kF6pbtXSXdEm^y&v~6`k zL+kn`-{#u(vQ6H)xqyrL@5 z*t)(O`m0=1xc;%m6;&;pHWb0$wS}A7*48y`7WUd+TD28@7c_3H>ayC!TNZe1Q}g17 zmA|QLeQQf|uzrWHsL;3gp~fW*g$+f;1-?bzqJoC4O9~1Li@d(#qK6vl1zkh)R&Rs< zTCZ4H$KS0&Z&ktDPHv@AaGht97cN@7c=4iw0=zy2uTfdA?&;dJ^jtj11@Dhk2E;Yra6b?C#y#ef zwBS-%mwvD3IsK6Rza;;q<7?^PaK0#dCi+HBApYzggPz7ZWT1at-LG&S9`_CXMZ2sM z-47HF*AwdALN^WYGYW@mxw*JVEg(trxS0q{&UD0*6hIu$&{m&3Qn`acOU z`xdT5Ux0C_92Xb%Hm(G}Ea7$uSMp6jf68|VHBsUxS>Oc{J_T^qmOx*W@Tq_sw~V_D zaAT@?M8auYoF>M~QF%w>8sX(qPU>IrMx-kx+}h7Z37>B1N9l|DF`g3j82YE@oT&1= z0dAx}dJXtx1KcQYzSO&w&)E`gm3KfI$SUs}5^nA1XA*9e_m_r#jQoFOfE(q_mh*?k zHO9z-YrqdlIPqg#&7PO=2Q2Ub1HI9{uS>XAkNxK!U>3{4 zl)P?E&Da+VaHCvJ60Y>1=vyS*D(_1YZk4xOh8#6+fgE?G0d5>OB;jgYMZa6Zt>b=A z!mZ;{-a>_P7n)K^cNEvCD7c5$7;BPO1y^z7c?(>{hXD&*#erc9TrIuo{8RO>*8ADu zkBWl#Nc*j@z}0#mu)x)N&qUhYQSWPR^9H*a_6UvMmENwlhNegOD<_xXfy{WjBY(#e zy{?hn=+1*Ws&8&<#%msZ9UV8eNBzMLyc$A^+UyHL z?h1dW5Em~lq4!u;tgf4n_sKQv5LC;dkG1~Jl1{vT(%04LC#axDLld;oTke0fxxsIO z3+?%vLnW^pUVqt{o|x6$kIR<2yXs=V3T;&13&)!x|Eg5hs)_s4C}((Y}n?`*Gc zkS#s8ePBp^$Q!R#is(e1gfeAO`oyyHhfbDTInwsy+)hlyZ3#Kwbko#m)0F zAEWQqDBj6%9LvG}0YDnxm|Igzmkhohsu+qIUtOP*?Wu2LsZqhZ(Fl!?ub_k`o~$~o zX%iHRUqx@N|CJp7TJ1lwAQa!Kzf1^kwF%)?1{H<*YNg#S+mDi8wtv>rz{ zyA3lgV`Gdao}FyXIKynbDK`(7XDc9_;g24c z3P~3WcQ4qz|Fghl|2w*%d#sSL`SbRI6Yt=9*IuyhpK!l`>wn`JswHvfO@?8NaIHV@fn!pm6o_X|8P{icDv^$U&77p8{0&)G9ioVTwa zThD|uSoPm)o|i5{PqW$I@*?lUFC000>C{ta-|skj_RQ%cXa7?3d;-~HxaMDg8V zK6N&=tR^yc;mNbDFPx5^g#M+mpGDhFor#`k|0w$82k%Ey)>cPO96fq=4Ea4#b}HKT z^V87>CVw71dE{r&OD8{!zCZQ>WHvdiRHCc%DBk zy8xMcBTsZ5J^TKXN6)sNITby5=5%!3tFK1KK7yT7A4c2KK7!0Wk#&zfhWb5r_K8=& z7fnh0UUVJoq(JV8J1<0^e3k6fM%p@$oIO!?~!>}7tTa4o%vbxM+`-wew8$b+pMF zrO;iwCGz@(BWEM0PemhTXQGG8K8z;W3a-3&=BaMz1d<*t_%BI`;qIg#6@;jtJa$&? zL?76+aQ6UwW~I&Z$vOMV6NK&6She22z(oJ9g#U;>-fLskuc41^Ho{h(IIlU|Nbfjf z)jtBSi0cSjFT)Q$?rMnqrmr4;<@x05fmfb4`{!c*_OaczPWc@9m-4Lz_}1UR_ip>v z@3(J-Z{S zyFJo|`v>1;^B*2!^YG*O#5GzZ3Gs005PRv;kJ zdmVc3&m_F`{*T#9v7fS+KG>KL`KLQn@?-jA{CHJPh+2a^bVVc?EL$8 zYmpDDY>|t1B}B-F$e(@%-~8x)2Y87Q@+I;)@+Cf&Q@&dGK3?ATR$uraTuWSG(=+jIh2H*Q?|-$i3^v^X1tg8xY&3yhY3C&W*osdla-k)e2iFo@ z%UA|waZMWY2PXaFGi*DMVnQ`Uj+TnMrZM|cyG^q*=14(z%4APt4m>_0)`fKn(AeMd zdMUAUGy@~;Z4!5Fj3s*5CYFE_<&b3g%*aeI5cCb;N!{lr1kLZ>nUTq%5G$XY3i`_@ zXUl_Qqm#o^20gf6(zE6I!{F4!v@giYgZ@ykTvW|FxIZ}Vo0;)WRC)aq1O6fJua77IpUXKd^tFcR1ji z@QdEDGJNH69E91RZ+v`Tz&Aa@gKj!76`TsrO+%fqsmYn3Z!+lq3b^l}Z+gNX82n0f zGrES?#oCvgDThwpfo8^6pbu4b;(eBLzcL}L`j%JddL1k$74ulE^k&35LM%LP@_Bn1 za)YJH?PbXC<+AWv*Dv;ZM<;_=%KOH>ZZD|8{X;(Q%tTelJ2X1$4NPLuI~;t-J6u)i zof)5c5Q}6?cDL6qzh_JSxUr6U$I7_%wab@(^P@)g9_+x%r-p}T{6RPKdMDw|a{b-Y z(bVA;2)rJ|9!{HLsTh+k?oB5BnIjyy1g zu4AWcO48fBNIv0%;Sy^$&Tk{zOhPhns3OuI6OR?~{K-6}5plbw?TL+(8TJgWH_;!R zyS6oMt8+Er00Vs`5*_kTSJmJeCf0N?gs^D9eBh$fk2A!iT*yG=mX4+z!w&cZ>}_<2JibPyU@Vcj8N0lUIV7FHN;W}k$+L2s90zuCX(U{im~JJ!OB2X{|X)Jp#pObu$8AxLXL#Vr`K4a(8*{KH;WBheyWKfj)3hRR|Yvy7Ed9GWn!^RbZIzmL;&_%HIS;WUj)!v021Z`AQFPIu{aJ*Vq+x{}kCI$g!-DxGfPbQ96M z{B}n>f9RXkPwW zPSd(Tw8wT%WB8Kv4o>gT>26MU>vRvNdvtmyr*{&~?e}uJSI6r(U8mFca{69LXNMAE z1$cI>fIlN1QUlCE@i0B`D?7?(TN6rj(2yL{5D%}Vv(ylCV0em6*H1fy|2w!n9^$~4 z%qAk>WO^f=Ee|9l_JfRl27TUMiCmt?9Oz}=$VlMv<~*3rUMm2-ECJt-vL&fcJ0YKu z8Ct^SX$%zl37qDm2I8&d@Y{qSh3)Kvbi{LM2DiNm@%(%mXb))QK=ehT9iTC%5M4s@ zsi09oqDP2!f_9ncEYKx_9-#PimC<+r3A8u)cVSs2=t81-{Wg0D7xDPCBQtcLhj1ev z%S!d}gv(bp5G~^i@>LBa&*SSA%CD=Qa3elz>WHop`9XeT9myN<+_!{q89xb`AmOwa9yu3{g!g;)Ndi8pu<@f~oqV*)t$0u%o zPdVXye0!Jdb(RxujBll-MDy{J+pDi9+8BQv4x+h#ar~Zivd821m2@pLl#ot1_XmzQ zq!BLT9r-1u5w83#Gt{`0a5+8zU$T^N!@rtS32&wf^YXW(5^jtyn^Fkp@yzQ}n?kq| z&s&oTm*WBKZ%ZcJ7!TT$2T(IMqw;b5-dw^} zJZFY><`8Zg4{``M#)Ix`!sU2^^7UjBZj2|}vj~^t5%3*Zg!AzT^Xck9QW?#KF=|o7 z-@yarG4?JPG{4@NMtCXpWfI{S4^{^>mn)twq*&2(JZTU4hCyqc7%z}t5&0UI2ab3y*W-Pd=f9a^(ZDl+%lPMb(tQ-0%KoEsD}c-KKie*k?{yh9hlEw;L zsf=L*Qk=#L19_V0VFzVl-20vHtE!=zu}W(7E)pz>fnj`ac-=vXgoziem7Z%rtV$7ddo z&-JiZ;4V=1&Ogl_n`3!dh>yfo*gRdd!QO@f*n=_rVLy-=$|}o1gO!rby9J=LT}0EE zA?DYdJw$gB&BxE&)kM=6z|da8KE`{pf8NHjL-{fCr~FKt)X&;X@?t*b_G^eX=Fgli zqA5m2ez}!I^Z8h}@6_!tJCe_LeX)$A=`8@MEVr7vOadAdE)a` z&K9DL`8T(ZXb%bK`DN(s_4px{zn(BBF60OM%_J}8CtjaMqA3Ptdo>Wv`Gi?-j{ft6O?#+6x&2m>7xNFd-%hkKU*&8inqo?}*CwL*e5l)Z>Gmsv`Q$Eq z`$YDa9nmyu8`XZR4iHKS{e}JI!2?+_b`d1`Ur|2$!^8Q)p{f4xU-)Cb9v?&H1DZRR@}l(GdZx`G^dZk_*jpXU z%^`bGlJjYfM)JH7VZUk4#{YquX;}YWfc;Qj4)p&!ZzSMrLj61*v-J5gCzsB!)C-GvwQ5OYhKe*8+iZSDygn@t%=#HQ%SGvA`Ilq)XU5YsQGR#@FW(*YI*#(U1}ZgL4;uN` zWBC-PpA}#K1`O?He7Pp_SN3VSB5~e6d0&e!KP_JjefVc)JWk6Q18)s%)@VI#l#iBM zX8TzyUYYvSq3Vx8vgU+fn36ZVR+f7kPe{nmhcgUDau9*osS`P*h|*6Q{kUmRMC z<(yIew%N{-c)X-e$9efT+#QcMVJbK5w+31@1U!=J-+M8Yo9(Y#7q4FjL9_l&4FiJG zcNB*hekrS=hGcq0u>cBU&d%o!T8l0 z*sdX9kQDA-Qw++$yEQnZz_Gt=4YX?rKtc2V*{Wf{5H!a(X|#Tq?E(2_4Fi&izDJ|| zg@JoC4=97)qVcm3Di60G=ijjB;Oi49Pw`nP+8-v+U#)?X4Whjiy>6XoUq$ESi1q-D z`fmv=!8a4;`etP{g64_iiFdC9VYEkSsqT*)UzbaBw}BTGiSmQV@wM5Q!A-c!g_?=P z@!T>@I3_$l-y>n6pR*bRstM1_6YU2k*Uv6=#o^7FqP-QK#n)d(`8KT({iE;|RTQrV zUS2NxQ{k0&Q2ZPCop*}#uL<~N3TaxuT_jp#F9KpV(7>06aMYCV!ey~MBY`%Pd8p(!u}WJ59Wxqt6U$q99F#x`#8T%!Ty7>@FVQM;rto$ z_1OOMMEPSq*^;!CA{9x{*7f!BP%H=8Ka%}gnCDl2*rI>f9j6cZY?6;HTu(kPPF^qH zS^N@&w};4&>!bd8p(roig9(?};^qI09JZW6O zCY-`W^=C^`@EZIg^5Zz#b9qRks)Tv@-lp#}M&jgG2XaLH7Qp8HmsjB861+bg@Sg%| zyXo#I`J;*cn7+S({T#%@BO+dTd(iX|7T?Ffo^C%3Bj?le&_5zS52w*ytGIn;6aAkZ z$m9FRSR!!rcS?ZeQ2((v5wEbfB#^U;_a!X~Km7sjYqtHv-5a)WPdIXXcRzt zeOu;eG^HfQo4EGp?L++$euRHth4Qz|iP1*M7l(K}AWo6rjp9%4e_Q5UJU6a?iT5`Z zH=MmymY?^(IlmQ#JjO$m&yeTkq1l9zD8Cc&i@kY_ks9`capgsRg+ZzhohP8x7A-H? zLXzdJ2=d`woX?;=w$HJwa}*%4A7lJ2fc?)H{eMC%cF4{aEaT6=0$McI?a$jSaJvO= zx4`WdxZMI@tOb@r95eC4ReWr{{3%VOFaM4{kytNnyuuIu@^=saVk77sy?-C;rR3@D zUCEw5apKYsuUw5?{WSLLmw)lwhhs}^DUReTjMgW0{CL8al9>Fdplzj?^$q;dB{oi{ z0jE3;*=nI?;B__!aIGjU@e2RXfJ4k9R@jg=l2+tQ{FLK%&5F|=wr)U6whb_p$w(%f zHmgnv#B3?4kf$=)6S3nk(T6x)mk~&=%g%7rFJ*U{Y^7Lj(W;JQzO0wg7mqS(dZVuP zCDyuTp|0Adw8|@N?=n0xZ3#{k4%+zl>O|AOAKVZpog3++GjE8av2Tc@fH%Zl=;xOH zOyRqDrt;{%XHWE4d=KJmhs~Cx!Be;_Xwo2WtjCk>8qacrTbDYVeA6pEExFEoDIMa` zvl&y*RvD|D!qwfXr{vXL;s3&Sa}OQNw0MxEhZRn(&y&T&6t4D4B(s=(wRgeUz@j*v zjPYdgAZ4G%37%Yt_SE`BTSg*$!iDn-lnge>OK41_B+5ckb*UZ)-PNLjUh_JG0&J*9{w_y!E#IZozTT(B;^R5BU>^H`fUOpOoKbNzpSU z$}jxS_?5%7w5Rgt->mC7e$n3sT=Reb{Vw*zvqjJS)w3%m*!S1{`nS97|LLyMLig+W zKY2Pmb=!FIdQ|pH_S*oK$(S40G_G^F_Ts>F0f$JJak+3XwH;R&7kx29zYXx9XRr6$ z0BUIJMY=EcfB&EHgEDev8>9c>aPfmJ`TwM#(fe^7c7_Zg-6!Q$uQm#{&X**6{yKbx zTkmVZ_~O57;?q)`De=3eWlWV%RYLr(iBA0EllWcJ%6KNIe?V|9&oJz7uNETecTGw` z#4vupfg!_noMz1Ur2no-vXy+i)EB2{M8R-$P4Y<2AVw*3(nnOuOu>CI{hvs^|4ip@ zrexK>0~nSM*Q79=7bCf(0?E*5tcs&rXg3`_1VuWOu zof4S-X>IpfAf)fH6r)(Whzl4cnhB`vR(*WgOi2xLg2HE`)OSmcYGbQ@H}J<6(GN=f zV%?&6u<93z2ATF)=_^06`N?-Nk(=bdtkCaol zi&9_FDu1P?`hOP6H`3oK4Uj&`S@r(_JgTC_O<7nSYZd}TB7N%TFfMb#zXWod4Q3`i zZ;}kkR3>UaUI@j&{jcD4R3@??m!!TYj%jf+V9&ummA~F1r}F>4$?#(RHx0+6sRHU5 zW)lnd3hHm=5{AyGss3?7pzlY+QT|AA(8G0tc~0E)PbsU@^!vy|i{!soR>&FFT=#FGmUPJYyaI+U#jajYy0;=|Bn`_ztVR~1q~Lc z|E>8`+)$gU_jutkE@er5byxbt{(sdYm(=4kE;kT`k|FQr{HUxdE~L~oHxCsG$*jj& zDxL+bf2#OWccr(lu4h-(j;1=18anDL$M+9b)sOF~>gj!;ZhWV&Z)pFX-Q5*~6Mem7 z)jOtqbqyU2yXr>w&%gR>(BX)Q|Lc_l$4z_4ZD8*N!)gb~H5Cj`jETj|}yN+(Ng1xWBLB!J(}a$g8KL zp`vzVV4`|#N<5e0J`s+T|W-X?|#MuKIx?JRn~`-aS4r**VM0 zv~}00uMWK3-6MM@LgTRCIx>y?w^sJHwe0Kk)zwZlwd~tD)Ym#yzwN=6p~_v;{goqB zjvL7}qTW?ElNmr6Zz9t?G*SD&p1!tTU)@B@cyn9ta6h$e#Z+VEz8w$L4^iD__U!I_ zsC}Y!WUzAAaR1iMnYg<4j_)68+~Xc^?dp|vXzbrzPkpdh`KKPJ8)_M7sT~<=?3gp@ zv z4f?P1O1z!?U3lu17$(a4-a8W{-gIYTnBHx-I9C|+R`}gD-U=`2dv^x*`@IdlcnzT6 zH{lMcp_Rwtfub+c;*#-?g7Z<=sv3kJ0 z6qkADK0*UFrJbp(Jm1ouar{Z*zjyv=>UYwf6_!bUhzp31uir$T@_NQ3udj=@rX5T9 zj`l1cBY~6L4{kvY+fV*8EG62gKcr=ROZyIgg@QxQ9R!hGo*E6j*o2o@aFTB{;g=QO zo_0KiGI&;u)Fd~43vx=`dVdbO4)!bgF9h+ZJ$_5UK~WFY*oHh6!Nzw9f*0-(cb|j! zeCFZ|HNOzC*!wkha??;+BrUct7V>hDXVI~n@>P(KH_8*wrxikc67os$Oi)v}mYdo~ z)r+QC;w%3b?Zwz%qF#z;HdX>Y$$KPI`F#`(x|n>&b@;!7{$ld9KU$2x@jCn;fKTOl zk@o{*Dz51JjK%Ey1r~0+zOoBcT#@Wz@^8rcE?DG8Ay0OuW!EU(O34pPej{i~Mbgfj zlCRqNuVwl9Vh?gEuKt58r+UB3Wo`xkf-3E}WCtpKANb3Z5y(CTno*zWV7ec>hUo*d0)|2e3vXKe;zKfF8&L+vvx7hKR?3hoP_L}(E$LT3# z;+y4?`wc96BcbWX6Coo6K9THil6*)f;@2PVF`zHgiO zYHvmPUA_)~w~QaEmka@3zP*xvw}t=bCV6xJJ}3Fs@}H7?Y6tpyVYP*7um?t9;!u-)AIW<*WD?P4eb^KahND zz8_1zHD6lSP@=U9g_LSLwS8244+$ZSh?_32=8bP#_!-il7A$-<4_vhH)p)P=?8r0 z6z>hb<(&wBHR0&wC|=V&Ff;ST)F$^&hysoJXV}d*J#y>7Uk`qB=~;hZ2Cv4FRUI7& zpx!P1pr|h1AxG~aZ|UqU!~1{-$AnZf+M~xGtPA1=;sg7Ge&PywD8_Mze2xCu(LsM4 zpWeXeEBBAc*A?Hqh{MQ_Uf<=Nq}Lm_x9{p`9QBR+2ZJ3`L;J^({VnbWc^`~V`G&l~ zDen+3n5YnnC(0>EQ5SO!o2LRw;EU$Piid8Y&wPP>Gv1kL|KPRng%3;(`GP)H&Oe1i zie73*O&U2QM9K$ef)zEl+!|l1z45Jg@nWyNH}AStGLcKJI4f8Dg1k%44+@mSwIccX0P*DXt@AzdDZ{cwQ}dP*ybBMsl=3Ksyie~H z58sySEc!l><~=z*>gN@I4JPGpUSpHycA2k-2vbt|tNXX4KG`;>V^Ux3mldY+SM(4b zh-YoX4E&mql2zVH|C!AHditAX{e?AS-dFT8RNmSrgth-0B@utO%03|Vt@Yn7^^aK; utmXFuU#$O}(tj???5Y2#NtM1@OFxN>t$CBmIk|y4CL4x|LBB7v>i=ILtpy~b?X@J_vX$p{ z<2ZPU5hN4alT5-(JQ+1H4q0A;vRR4!*&fGn2;aQKFY{iaV8;=}3!`{!$+4~XpQ^gI ztFBsz#ExIa_L{}*$;j{ikJwC&Af?8sTi+~d#V_kv^Y;@{!;B7VQeIhtk~v~+y?*6&B(NM@t2 z@3Kc<+vbSA?O;d#!@-Up>tIJf`x!Qkt@_qnU;pdt*w24sXGiis8UcB)u!-rwd4YunaEU*A;*KlF{hee2_Wsz1l`&(|Eb#fFcPKN`O1gD-vuzWSjrez1Hod=Xzf zHi<7Dv-l$UsNs(T7JoGS_hLSIZhn8uxpl=qfBRNj^sP2~^zC)*$hlpMoNw-PL`VEi z_;CXAs4IHy?`}EzNBDN%W2OBbpLY&(I@6-xEbTw_&2azs_xnZ{IayjZJ6QFa!*%>W z98SazJAU&V*XXye^o`ccbiLh_<$AmE2}gh5Z+xRot&aZaFMXqpzqXAI-FExYrf=9r zqhBuVXB9)vzMlcV)6w7fCEIAJOqm3-ynJ$DM#X z0Ka4J@B0qmTL9Mqei8830Jj3Z4)8Ake;M#zz*ho(0`TVme+=+^z~2UZ81PQO(*S=3 z@FxLp1?&KP6!2|;Hv;~nufOq5z&8WF2JrKMgMhz^<6a!MqrMRJp95S4_&UIs0lo+D zYQXCNX92#o401dAA(3}{5_5baNG`kd=}$ag5v=ke}ZxC#IX+edBFQ|oP%QV+YyK zZ$*x*u@@exw-+C2I9qgN?~jX)?E6yDkuube&0_t(#auo%jrET?SpT1rSwH1hv^x@b z@yA!!oc;0D0ekw1z$-XEPv>bT0QLpCRKFBrC}X`AE*^<@~|r{5E9x2FZ|eBJe|;f0xdaPXwO8`H$&5 z=|tdJoIg$HiCoU1DdCKp_auylK9(@rv^C*vl8c=_w&3QSkh98=sQ@x9hD?hg(_+Z9 z*wh)xLb@VZNPmcTKA&-U?nGY^>dSK?a39X^rgQg+z-MuOl+Ilz0$;%SKhSxCkY`5t zX!q;R(WY0NkOi^Gb%eP4@6g#tAN@qluv6-30c2VTnHEB(g^+2Xrl;qzo>KK>+HxoQ ztD^p#CjuYC`5roVoCxg2c`cpWPXtlEsB@Ir z2Ci;m$AXP#(~s;gN$kJr3TOWBF_GrvdGn8-w&KM8|`oV z6WMZl*^+=OFNVz{ax@A&7EZLEK8A`^4*?JmA9 zJm;|gd=c{EwKn*vz4F8+#zwsf>_ou6?SwbMhT~QoF(&&K933Zg`s+_@avJp4N&3x_ zep4L!iW8eW27S4t-ze!f#Gzk*VpEDizfRJZN%~Sr&yFlEWk2^r{{5@V`ZuSu(Ir?n z?awM5U4pgLIplj2Uj=-o#9tN%zZdu%i7y;CE8{4AK5eaqz9wUgEzI2mb`MpC;P>Iu8Co>1fjJ zz+3$$f_hu#`1<>ZK2H5V5dHM=^j{$QIQ3s4`mFKvj}U#F`tK5b_IUaq5`CQd9}|7f zc={)ZK2H79L_cFZ{j)?Lr~VAl&m2$x3!;xx|0|-OHJ<)OqK{L5p6GMO)4xLWaq5SO ze)f3!*NHw({hLHTXFUBo>-v*!=k;1%NP>URHD8TA|44yt?)PoD=A*gjQOs+vZQF>o z?FOuI*GJ#nw+?IHvgq5lmPXIbf3jJbkQ2w_Z(+{x*5@qUS%oA^PT%408G)zi)K(Ab8&9>wk0K=IF?u zSpQpXo1l}5=+HkqqC+dn`p-Ga`rmkzjgI)Qi@tu4js8c6t^f61*GFGVw)Owb8~U{Ozm#T!9QOY6ZrkYaJtU*8f9P_|p`Z4RVlM2zFbi-t8y&(ra%dsy zzXm;|0sbD~?*W$v_-lX{LeEQx7VxJ5Uk-SMz5m>o*G11|;~L^;Z1mjxO^eQb8Z^&i zJ#^4Pdp&$#$ir}Lf_Xd<^EcV`oo$assSejN$0yxevI&1_bm55h?`->#b}di)O)5Wj zsr322DD_YDc{U$k>oz9E@8j)TzYE{9!S`@&hWT^E?|}a~;eQFyF-PjpJ>ZqJv^1Ks zvMic<)jH1izd!98O$YB^JAC~o56&5}m6k5@B>HgOG<&2V5q0jX7WJ*0d#V65eU99b z3VxhDvMH~0QS`CgQ+I$fS$?B*le?!y*+!BHxn|{iZzmelTynQB@>G;=H zM*9v<6D#_`!@3ug&*^MATE;=)tKYee0%zUj^rP z{9yV>66(<}u2Z~_W0Tp>(9L$LA3c99eCKl8XaVSk_SySA`J1_HLv;c_v|QqL5$?KJ z`*PWSA=<;&dHc(W=6aLnD)7IO_%5XSEz$Gv)0wE7fx0y~UXG(De`_=g=d1AZO;L;Wzk`S<$nfoEd%L@Qmn-_vb`k`(k$V*ujsDe97Ij=qqU- zk3Qu1$jCpf^Pl?X$3AlEomuw&KYVp=^yP2PiN5;1+0lQ0EI0c4FSDX=Ag)Icr=#0# z{crv=8~xKdTmM^MM4Te-hZZ7lB;&Y(;F;yw92r^x_#lqU3BT*q(00_X!|^(5w`*i*KI+$1pF%sp)gwc*2;O~a zXbHi)M~0>myyq0^0q+?ZN+v#kH!|eL@v*-Q}_UO=p4UvUFse?8`)b;t*0 z$P1;&&rfJM>BD^SVZQh~@5j-HpBulaES=blEBhOQtqsi`tP+@f!rHMCv!~!_w=>5Sht2M=F=sNC z>9ROd8TMeI*@f?kllK29j`sb`KrPOU zV|R;y`O)9e*jgWAg{^I!4TY@-3On~VweD?;!F62{K2T_$)U~y=2Rd0{XG5s7Pz=pq z*VWk^=;-jbtnjyl{QDYO4)`TzM?-5zTX0LDz2oMlTW)D?FyZY2?kFxZx;B(vH;M}xn%)_pfNwRW!YceVuld)tC7fliap`aox3Bd;pky<>yF#9!MM z^4B-D5Mw|6OiGIc{vwQ1jOzn1jE=@Yu%Z5gGmv0IM?>%cMg%Jb8=5~5-iC1mJ`e+` zZ>a6s=ieI)v@{6oSStdBS`EUiF3{Y(FBoWVn9yMDgkk2NTk@_}Z5ksnZ9SAD?W(NbI#@^{wlYiKQ6<=@v8 z2-XJ!P0jvJ%ys^{wq}2A16Hs8`UcE@ZFLRxek`^9!G_K*ToTm#YrFlto~g`>FO7K@ z1OdxHC-|VFu9nsgYiA!Wo&M&oIJ8tt3tMZ&r7xZS^Iv-z6U+9NhL*y%y?Z+vI*XX! z-+BNMB9{N#w`|zr7Z?Oxpt&i~5r@DCzxF_|v#GPGt(C7zOdiA?>#`5jTJvHv?jj%^ z*afhbwY=VCt!5wP^;5>xf89=e(#F+4iF!Mm!is7Hsz_4gJ4AhrtoO>@kQ9jG;5!wnpJZ8SiM&wIwc>20qwO= zxeN6U?E6^D7z*FF0fNq{U_3o^OiQ^v94=sTFc71nwsmq9o-$B4K3c5u8vM`eZ$_S)&}p6 zZQHKj?%%ngYWt>bTm2iVcWl_Ye#3gUY@2t5cUd4<*XZqN@fR0e9`K@Tl9l$|a1X*F z>wb18>Yd9y49`xnm2^JU!{9o+(76$LbWSu>oEfq6OWFQ&?sdW!(e84m=r5ZjVyuL{ z=3p15GN*)#*#!+QB8e~LnfT>wP=gEEa~iyioz~!`>?sXi!Ui;WF*~Kfi`YpGE?|8c zypSE!;LF$p8oYq@YH&U~q`~u9ScB)WkOt?mb`AEjMh%|JYBYEbtJdJztWtw>S%n7A zVm=L?$%+7%n);o=yc(Rt(lr>Nt-)FBf?aJtoegSmCOfUc)7VphWA@Hqr!@Rj)~CVg z?3hNM#(Fh;Dhq4y6xOc6DXd0=lUbz(C$S2^F?kZ14{*%iJZy~yyIGM&?_yrSG5sa5 zbPYx>)?fz<+t>x0-SBth3JtcgFjMh7NQi!$lgFg36b8V?yX^)XXN@uY_1JRvJ8aJ4 z6gIFM)O#2!!q2%V*UY(;7p=uvkV}!m@N6R^#x?Y-MlQpT{T+nf9A?h7?GB8Fjgb&3 zAhvmEv>dLKFtov8kA$H;4p&RKQo`vHu90w!gxe*YBjK=w8ztN;VXuUbNw{6YCna1U z;QuAqm6#IqZ|LSHis#M)Yueg@k<)en7&N5msjKfK+9TOwj$0Om>a|E2uVR#=F7=+J} zaM64LdnH`6K)?kO?p-M0A_)&*@kQ-PKS?$xyRbySqz{6K1p^pIx8~RB9h@cn! zK|c~U^l@6krarJ(B>sjzdL>*S$8$l#hCV7kCg=@)oRqMkk95DlR~Y(`u#hi{os_VU zFN?7nL4S)*n#uavVK2q)3xBYkEqcv3LW2($G7G2EqPoi$QVL|&-*nsM9tXa z;$YSrYq9l~dI_3$5e+^@{jmw?4g7n3jNED)W4{4c4}NFU*o^M!qKiBn3RBr*-E!@I z2yksGi-bTZRL9t}X~4_X4`POWDU}@wdZ1b0UP~${@HsCW{QwfPt}#yv1*VPklPO>_Acvg-ra)C;N<1o@Zh>noFeMt5zR?1E zEpWR9F0jBM3tVJ@!xoqlmD>Ly3+%JNy%xB_0zY7Zk6GY83w%<-`A+s|H!MLgMIh{% z1AjCN{IAr2$v@BE#>y9UO0@82tG{aPpM(G3b-hMU{*rEV0REAXllc_5xYIqx{e%7i z-X2OoPQ*EF5x~EW1YJoK{ye@B|Ic?h`Sl#6_ptm_W{3Oo^pGaTQ{I&cRZ|HkvzJ7F zYX2X_`054T!Cn{qc>4+Yz2u>sJT`Wa^P3U0Q@S93Ciy%wf=)^Y0*3tgA;!D}v-{j6 z|GnK~WA;qGyh~<7HV!)xKWNV3L){=fmE!PF|1a2Bg`s~Lk5+x#!Jm(h^uyy(*XQ-X zhk@edMbe)$roS_3>`)^5%jG)ruorE}PtLJ$3#h4oRfzLD>qH039Iz#qD~cF{ukkClIUSBl>L<6Lt3hqpgW`Vqw*kKgQD zC6=@Tdt2f18Tm3n#_``@rLiq{8SMrBp^!U?66%;8^9kgC_~w*Y{@$AkeK_4TLmB#r z^rRWr=3HLrCviFO?LO!S_J1(sRd`(C^WZE0Kbi#{=X$5=u-YXBsPjyicUDheGKt4omruhNdQl;VYaT_})-P zdYIbF`7zQnjh4Gqxct4LoOJ3ihM!@<&rA<%_*oWwZaOVWV(n*J@N?3`8h&mJ4}H=2 zdA#!Rlf2$^h5^U;<;Cc^f6hyX0g>K*Ljie1hctCGh?co>KpAt}g}y ziT2w#9!o)qPXInAJ#69+15Qs5EBz6Fgqz8aU|b$ZIFa-7ns73Q=NT~afhOMz4i3lU z6Y}O8^k>~1AJbprxOmp*bIOmfei<=9Ih$td4JCRQrUz4BAuN7PJgo=PjRgt!mq<^B z4u|;Y41VB07v~qm8U*%wD3p|>`e&pkS&N_hL%Atov=IFt27PvR5n#9U=SWYMkq~76 z(~bQw0f&|;@?v~FOEvj0|3N**{73#Z)#EY7`*<4r7VP7pA@qv)gFW$DgCP&m=Xuf% z`mU}<0sNHli`Ykij)`AhpAuQe{QDPe;Uo#8T`W1jL!^b z_)nOQl=$Tnl8K+^WOVVs@pB=M*MnT5$TMHlcW)>k^Nt}u>1zSs4>#oN4K2*J1Bmr^ znI?a4Xt6VF_DA|D@UVdxeQ#)yCnu&a@L!_!Px?+b$7_SXVD)I~gZz!V3nVb|Q?bXZ z@aqk|_Tzlz~3U!f*{By^=k zzAH5OB3k@n{ced@8h^gozfG(U$zZa7lCQwTQ@(?L(Zwns?{ev1vGG!T6e;#0{wvM) z4&tB0FZ9jv!T-)1`XGN@m`(m{=#%;{GTYmLU*st;`BS{X|1|x>|9mldsQ;yA`vi=C zf#x5OXAL8UrabVcmkfEx|5nA~0rR)64+r|cJe!t+hP=I@Vo!xBAC0HP8HTzHJjr9x z_Z5s3X7&&N@frP7JT3FAG5OJW3!Pz8|3aRa|B!v)Ujwo6M|zf9{15uUhFi>jOS1X( zvl$pinzyw>Z* zWhx%m=ho|;@&1PFYqSABS|1(5@Cni_QX_KwjV*Wz< z$o0_eFGHT*(3}iR_a+?a(XV$RJ##JnuSiqpbDqyeRDSc*)b-Z=A!lsA5BqW2{vP=^ z;#<9biy)qLc`?5}pz^;gBdp1*Uk^ok7Fy)JGEJS2`TFinmEV#yQnsNF%D=iii!JhC zemWLwPwO3>e!15FgQ3C|w83J`FNh~+?D~f6f%)<+m0z)iUx}rEQSOuhrzC@$-Dl`*&b};hJ@- ze7zwb(t9jkBR!=Ser1|G;OA1WCyBp$z1kbfPUHN|^^ z(+2D*y5Rf0fyu{2%`OoWWn{$EV@1OpDEDq#yWelFEOa zf7AA)9Df>@H|B2~R{TTYy*j;OKY@?gOSF&aU*Kc<7WkO{1m4sa_CNIb6Z?;g?q!^x z++Vx{`-PrU$p5sTId2a9OZo9U+gYGXgHylbhal;g3Udwn_Z&M9NJ#qhL#!@GZ+g7z0_{pRyA zAFtr~9Q)q_%jc^HLz!H@Op=e!$FpWq{)^>XeSG%^Gn}-XGVn)3shI;x|Bwg6#r45k zVt+G&{dTqg`$Jh7`uvlVLHbenVc@a$M?;CTfE{y>82qpi??VGBd_4N~ySD#}{g&T% zGq|<^8<}LrL6zNz#4~ zhEh44N-*xN_KhSEH=yf?-}_Rk8_ z_{mrBJwZXk1Frlq4?L_N&QRn7u=63zpF;{Ci5BWukGLA{)4t(*Bf%p z%s~r7pH_ds{c7%S33NXj{*3z(9Ck`L64L!460+JmAwl&=*w?AUF1q`m#J_HD_#^x& z<`3}41l1pphH?{h|8pkLg7=dA3+sLEKgYPrV*b;o!?`CVEd9xOO2cdZqxlEs0}Y1$ zbbq1i;lIFNa%4W@{xayn4kHTXqxZ;P24eoe{YBXyC4Zsy-TUb;YQ92#`Ty@P&<{$? zU!Kz8+|v>^{N*_f|Ni@nSM`_X_ugM9-sJqn{bk6Q-)Md5n(4*x4g7z}{4%K7-<>n0 z!?_nEZ1~HFhJXM4r9kzU5zi_v&U=_&d=`H(_wR+jUuU%ZD8*{aEk*&8=U;n*H55hjci%*Wxb^X!v*MFL*wfbAt`U{-JVx z&G+-OopZ@PRL%DnvtrM?i2WEmUZwiaNsoB`K;)ZuyB~Wrl&S5f_J*d+T%-DnWj|p2 ze(c5fUx~lEUvhsa$Fd)rYuS&TG6mXD_FtxJ`)%~R9Nlk1;ri0{GmQP$W19Wk8GQ!q z;QK2%Ck;4(!?~v<{DPa{DYQQ+_TSvJU!KM9hx7QwepXk#qW?6TzQ0NPrwKFZeufcm z*sr$iZ%$YC9}i=HG*jE(wC=yq{-?UXWQ||i4~^|NZ3Dj~%YLW6A4dD-7vGOc)%L?9 zA?yD4wAlV3f1g5sUQ)9M{F(N%G6xKOi|d1|r*t^yv<^F;Gwi|Xlg#~atQTRA41PbH z>`_nqlbMP=mS58TWG@+*_Oto^WR83uL_B}+V93Sy6I0}VVsB`MhaEEVE4@!6?ysbq zcsy>T+<&M2Aa%ZcFf@bnn_<#Vo9Wa1W6Bhj9`=NLW4$QvXVCjQa1fR)+k^iMoqie~ z2Th8`~=p*^q=`3Lr7><*6sWB*gT-+}#0#9yrcqoMq4HJ@AeQ*l4g-3VrezehqM zdJO)N&~&^naR*%z@&0>5S(g3(Ia751o~y(M&0ovh?Pwa~Puh@$@+I!~c-b8lyaw>m zkatQf9tfXj;aBJmfjGtwx5JFW`6aU=b_WfK%X>65Z;Bo-^EH0Z54Hzl`ig`WX!^(V zj_xp68T}!ikFGLj^8Xe@8=$UF!Jp%1dHy zY6f#V@YFoU2YBh6Ob#a#jQg|hLyCNno|W43a`e2Q`Me48E7~vf;*K2n@%H4eG5t{f zb9tEZ{6{3TNRuD)i{*KfVt23Bzy3UHq(^yv0sGhJzi?iX(LX(3yF}w}#IMVYw-WaQ z3V+fc42DAYSLpu78q0Xs#>UJ2LC>G)fB9vpS#tdJyn@IFQ)2O{#Xsna3Nlwp`UH}{ zc(HS}qCY+U@jiq57^D;aLi(KN?lWMa|B{@fHL|@8{+_GF55*VxhiJbX`pMS((fT|L zJ>O#5_bkmHmudb9f7Sev+gEvB=4fb+7H{weC4bQUgaURM;%B`6>3MG@{_s8#;zN5Lt$>O7 zMC3P$mvpm#(&u6gZ?&IpADIu8=jW{YC;!Y*_*?Bk_EY7FZ=ZbWzj5P5$!C1NMZCn~ z$7&zaKjjau|HpZJ%#YcNo`279drf)A+k@h#z{ICn;$d6-cu=0#8=udt^UcNblZ*%R zdA@l0X@R>)k_=mkB^Uh_aQ|dvd2o@9^>*?*r5Mw z#fR~Htc;Idqrbm2KFsHHlkI&`rUpWy!?X`ISDo*>15Sz~LHw@r=9+-c$_Z{&*mA3yZ?mxI#AMQV- zV|~k??|Uuw{SZ9gXC2QS-FD@9^;CR6B-7!gc&6`%;Qg`;+z+TBnEJ=_js%lD(?^-=@gZ>Qt;(H@W2v=cGj1Tt33)Td` z&i5N5x7x>8GbpGL_!B)j2)<{<@vsm78$#2iJ?MKd{0|V(_p0!|LvD!o#yl7BeJk95 zyQ3RVkCK6C&|iw-b1sPY7l}VvkNX$p|62>k7%j+1sI#eP??wA8(Y}EA;p%I^E8yBR zr9XUci}zOm{9=J`r1m-ZK%#69et3V<&c|Q-7T%wn>m-Q%!U+ufcb?$vgxL@c&))-v69(tF=ymZJJl*N#FkTaVJk6FD%pIfWXL;fDnF)CU z;_dT-v&RHKRC0X2cprv7xr6UT?K3YGk);ukEC`sH}& zCtv6T9Ay7OJ^_1zuEZI{UAFfGXC>nXlL=o_wP1|w$Mu8x@S3uG{@rK}Bi^s675Xsr zcjs=lbOGK_HpWZiL;A${5D%cw3Kclh4=}MtApLA&OY?CDN2RCy)NaO0rbqZ6#m8Le zCo4(p4--uKnw}{9nZr)fcLA%=_>sO1{pJNT$8>+r8Wa3L#Pu_MOpOno9|mQ-bAO)U zk^Tq!E{8oc<@{%~&&iVcRoW*jQO2Y6_vzC~AG~n+XM4OHAb;5ge}gv+%J``zf6L62 z_L2Oullfd^=8Kt$B7TIwd4h>aGJa)$nVEV%n3p8tUC`$RXH3_}?->*PAr$1hqZ`pZ zDC6f#sjznv*PmhUykKGuy`RVnAJ5!m5wDE*&-1~cjGwbK9*;-*m(+J+qRc;XeDjk< zdDTb*tWaJe z`6t&Y_gC)>g1w+2|GlO*cZ~Ws#!vp6hnMYz|0cj6=i+VyFVbIUr^tLM{dLY1nGZR9 zM>n`#h~-O)7m_#T{|nP(yvz2kRN+75@5|AjQ+&V6^oN8gBEQIZKsk z`B}_Q(1#0;vdF^yU(auO!36$&N=^Yh*&~B+$avxY?sDMm^!{O8=+%G)>M|WKSo}FJAVX)}EH$e175M%}$f~(TVjE_t#8z zL8GUs-tZsjKSlEA^1(mSWAp7c=z~pF=0_Xmn+$cn^#oI=sq-cIC)u;T7VA~6ACE`1 ze_qh|914gyJ}F6^Z}Ni4$znk+@UT~UY`&rRn3AF9cdQ?$s`HB{XrHF$UmBm9&nf>2 z`^WmX$j4Owr~YYuBICu5^~hA2UwC|xJTf0i*gZ|me@1-SlW9H0i`0kSKP@~|d{UAw zkKL}$r`OCA@nXhDiSS1bQ+_TJuP+<%a_1tpG=+I=M*h2|O8BQqUu);z+coLeGQ2SF z74-Pt4*4s-d>w<~e@Uz|1|uKZS)+jQez^GlV;Kfyp_vYZk*o?sS ziRF8w+WX_Rg_iG;&egw13VmwV$ERt&c8s|wA@TRmQ*r%`xoQkQe*M0j%?L4(&*^)^ zxSrL%Cki+t=-}TI=JdGUrtc4v{o1R<^;L1FBlf*iT+hz{o~*}{)U$ZrF7YD-qKNTv zc_^Sr9_&XDf5NK#3FiEvpYlW>fheNAwx59U-ihajc(L8b<7-B@>0SK#G9xJMzXI3q z@P{7+6QqCPy=lOYcZh7rc{Hc8qjEhzwgUSX*pIN|{aWttNQI=I$2;7@{i(r2sqFdg zF@fjz-`$|6SN&0l9{G{>H%LPMeNwDf?LBsJy-oP%wy}E?jqkCBlNkOeIKd9+$Hf5S z`-$ar{ltqHzddAUr}vO8aqr zTlx3Ad;Z${f>!<)-aUWq{W2^6diK=2)sOZ*l9m5+@1DQ!V*0|m+J_g)-<j3+iciM z48Qg3g`fNCw|+MEn{AG_e)G#Oj=lP`R}X)4|L?DP<%$bqkByCt-1hdhW4GOQ+s{7w z%dXks?~F}%JvFvzO6TZpd8dCj_6gtpW7+nPzT@|dtN7MCiH;vuxnH>L%17VW^P|!K z`_t|-uQol_@slYp|9$Eg|8D2gH}46x_WzGxH~qH%cdkDzF2B6@V?X%vleSy8(;pYO zZS2t>;PA$8ZhMv1)~!@^@Hb;$eEF>duYUYDZ{YtQyz!3Bmh4O#;j8mfzDQT#1RJ)6 zY{{;qH#z=xJKfZc!KHS3O2w9(3Ow!B9JF15MSf{Lo=r$eN_~?*EZ}h1Mj|%j<;k!# zP-*HMh324Br6I*qvC!C5ny^AcbU(M*X03)4+tCg?V~&Bb8*KJjA9KCQN=Y!>tKp9i z5L;1j>2fhVZPi>1PnS{`!_!vH#qjBq;A%&KwQrvUUoZ(iGzp%z z5-!$%coIBq6Cc|1w=2@zLAb}?_AJifW`Oo~Bm%WL5?l(}49Us$( z*zJypd6!7XC;0G3Hxle_6|eCR6aN8&zsJhoiLN6MT5YfP@AA1=#B~w=7#-x7L*$?8 z)a3X0JS+mGPQ*Xam&hWC7vUe);)mx~RlX!&5{o2F%zuh+3X4p+7=JtI$8cz#e^ve| zz7!TonV5evh8{`22!A|&3H=!NJPWM)pG^MA^+Wsz4E{WStMX5#|H<{E>7Vj@>8$ff znvJCn-i3`d8}K>c&l!W%;ibj^pCeTMli4_~j?N;-m;Muls0}HZY+&T`O0uyYgWC9* z)dx%)8BvO`SLHK(K*7(Nxc}<${cG|~oCAy~u=H={m>Awx%P@1q3ZG$W?Mki}YffDc zP!(dq&=a3)W9w0E9SKTaV!8C7h7TL~`MEq-P9WaH;cIW^Rh+{8h>HuiQ7v%PuRck^NHKbIGnJ(@bih=OFy5meJaveBz5BU zX+GRAOtVkeKHZnjBIy&imvha8?S*|NZvS6U|2!A#xnNwbDNRXnmKCJ9lvql@HJj5| zml#nU!#i9m-pD${qinIuWvi=4?8nw4Wpyh{tUp`bQmpYl!JifPB@B$&FCru5Bfd_w z@}>TaANn=VC+Tm1Uu9a?*t!IICQ6@30<~8|5<13_VEn57Vm;D2ChnO3YCY1*=KbxF z);jNRkF@%Ee|zNXA-cw~`tD!NjJ0#mavHZ|%yARzq>MZ1kh_6;}oti_B5> zy3EWsS$ZYIpE|U=2e3eO3^RgzyYgYh1i9+3q2{&1X~a+#aG2@CYYI zh$d_wu&XqXC)UqEawO4^BO#s~DfT2OhkZ&Mye}of@gzq)d^pAKk}}v+&GuX$@${z2 z7IuoZq@zT$ZA|vChwG4Zg#F6#Ivnr_9g=ME@GOz*ko1FRyN&j$?-+LD$BAT%+qVyw z&Qfo2`wl0XvMGIQV>4~1$R_=0oQxC67FTZrljx1bWQ!|1#%9Q-$ZqP&YS)Qm!}|t? z&b4tHdW)-9MQ?Fs!`LR0%_J9#KQQ3Tx=neiFPquh+T}-yPdQof|S1*d* z6r08LuEav*z4kYZ`(5d60-L@kdQ)ur7wJv0>3gm>#ins>KAFEMHdSo?XXoCyeXBN& z8^@ExLVTN!U#pk$HJozI;v5(gE0jUn%a6Oy(}x+`I_sCFtap~xRbi#E7uQI}v&)!2 z`TEd&6AAa`%zJwYbnmV#rOZ{jHZzHSbZ-9EI9+ARkM5pviK4VdqKTc>O!65#zHGA3 zOLdK~GMwMedezE&lLI5mTnuNRB2>O(NMuN10Nd~+VlRZKzuOS z+8=eVwD%b86_J>IO=?LI^YRTGdoteDvD>ia8Y_e^u;9&?401pYyjRD^UO~Y_jdA!x zWwKPn#fMF;)x_nGEp1tP1>PcG%bO?&);l+LEzIZ`Z4{* zW8?bCvEa?|dj;OAUqd{+IKtao#vjvPrNCS48#dcF+SEHnHuy7O{hm`SQf|j!;J-^~ z=aE+Rwh`JXvMur8j+BJKuC&g>@0l^Sqvm4w%*Npdgj8^;aWYu03AEi*vD-V@ZmfnM zw?kv$ui@#ML&qcRm0eWYySNyhb{Z~*r+tBo;R`0g(>3?S+S3*C#qe}Jdog^)IJ|yM z#wNj!*B{&yIn(fQy6!rSzcD$+-}@#XpmQq7?mcu&OSwH9E?}Oq+rtGOeky2I3Wpi@ zukU_Q<;1lSbJoB~1p{aA-fMb!;A&RlYyR z+9f}u|4OqrA@6(L-=O5*E63|o`g1P#@GML^a!$pcbd@+dH?k9*8=2ghp`7n^G8(A! zawof!j(fc9?th&3(AWPlvxR+Y@gLu)cKp-ArFljFGUv&|X(^kUla_-fTg(|H4DTgg zm}!j654u|fRO??Zeo$~E(%$X5El&VpPvh`>IH+X91F9#tZ+Pu|0r z?_T^4L+H(6=3LwEI0gav3ns|A%*OxpG>2&xC3sNcDQgo9@8I}q3B#K>OtT*0eG;yb zaE*j%)+2ma!i^F>CSk9H2PE7s;Xw%(NSGEDByUK-4kxi*ybVlvkAy3W1f0%cCmUEH;2a4TT`pj+gpXYz-~tJ!UnAfm35Vft#Gmw& zWOK3$>jg~uAh@PNz%+h>2d@*bkS~i>ZWXYQFN+PrcZuIIPM^Up;M)KM_W^ccyfwGT z_Na5&Qdv8Ek?@8-1|@9hqo!BVbACx|P{N|WB-VaJ;0=ASqXIVc(JNs?AKp(3e1RO# zNeLVJ@O?($4Sk%Ju&EC$8A#p=*?v&MhCXUOEAT=-=tIInA4$xyCL4cnKA-i{M2i>S z2k`U4=9SP>JATLT8$_H|LJv-S!Ux%d{^vjE95|ya6zz2Sjy%3Ck8jCiF4|M^pe1!h zF;C?gy0%H)tm3tgVR+s2$ZX(JjNR3 z8BuN48*3%FAuk5K1X+%Sk5P{cH(spt2L8`I!X(fIg4a^k@_J+S6hnTL*Yk_!SS8NB zj{J$Q{AqOrJc)WcUQ#y3i?HD(A8Cr}4gL6JeF{Wz;7^-a%T>JI{Gx-2u10-EdQ6_v za!p8I56?5H;A8T9IqJQ~#>-iW`ejPJSJHP*z(-;I7oC!Mp@MP6Cu{031*t{4gISe~ zvv}>FlC}7s$L2O&WfNIELn`p-?MNYH&!JqQZi-fQ&=Hc_QeJs8O%G$mP z)psJ5nGNO1bFzKD1z~NUClu1|v(UXI+xMFJi1GYlZGQ+AU@@-$d*w${yppfsBSw3} zj)#GuI-jI74ku-M1E%(r-;8#K{STn=#oC7?18Prn*7lS~NCgNlt%T)8WjG+JsXgWO zumH_}htQaKt}|=oNmw4KQ(1U^R$!{8d`0bvzgEHms?jPfP6pgV;79V8S?CNwLrVNn z|Ca7qJMvbRu%C=L!6^Jmaa=P`9O)Z5G=7Z;@U>7J^^c;VPDMo}Ho&xE0U3RJ^I)9* zJzo{g4F9M8saWN2knMez{@2U)hW}D~N(t8XE1{peRQn@Y zS=ib_hijyRsO@;w1nrFYGLA;OeWlxXmhY=86=mI)vXbVmx)o*3yH{+#@ut$|9f4i- zUAwERitAc--MD|%4Q+wabz9c$ENiT5+1RS6`A zzq7rzWjoqcU0=F;*Nr!}Z!d4&c-@VA_uSBSQ)%PgJ>@%Fc31E0sNc-nmRFbT?5wM( zTGhOH8~X0tyL;EFuG$qh4J-b%jQoG_h>aTcr%f1aam2O>8R}#1}P}*Yl8z?PqTVJy8hCta?NwaZfbNRl_ zjqO$2n=49gysmuT)f=JgE~TU#cfDQnnM)wXVT0As9L+q|x^ zZT+r&m4ULdwhb$GHU&zzZz)^RxVOG!ZTFsvZIj5erzo(ixVf%%GP?Gz`d!6)>&w>$ zY20z;+jMg^>8ZAS<1KaFtD1IKZ*8tFsk)_hovOE;n`>KdxMd63u|mqaYh!oyiu(9G z%lF-UQz>s#vUPK;&AM{5DcRGpd)I{RR!V(V?5^It7yi=~fX~F!*_Ku*N9)$2x|W@d zwQ+U3Zp+5kUp^go&3 zZKiQouh?^7cge=4xIR$6uUqPUBd0BCm_)vMt{>=qBKsS0A^N_!&5gb*>RQ&`yl3ay zo9ZSTckAW@b>%xJwNKd!LASeg`y}Jp(_M>vR=uKp61rwdS9YMbq2w{nB@^T?ldh5a4b|`3*hTq2UM!g7ZN+$3lpUDJcTBn!L`QzH zYm%|-6n#(NizYv==gt+A=((Khxq9QJ(e1iaeq|H+ieX>bZ`CEy?WwpletXJ8m&UKW z@zUrjE=@kHIWCD`UCE`%S64Ere4BWzm2Yg_Q$3NdZdPNYxo)Dldp*`oMSI}$8;iEw zBG(_tOO1QPn&PH1n(uZN)mLxssxH~sRojAjr5bZa3FeCRn{V0OwDM|Wb?0wsZVLMM z*RHH>SQ{u=xq9!ay|pU?d)L&jXlSTkvo;V|xpH;g-j#kqSJ$*RP}ktE>*{O{baePz zR`^>&{(TKC2mCu4T07c;`&t@W+L~6cELLF+xF}Miv;f7c{VR$tFY&J~Ub)i0(qCNU z-x2I;5dUlpGU|J5+jDPCD7h+bJH!iG&h*= zb^&)3m#nGvukhD*cDFawaYaal)CM3%eMd78O|AQGY-;UX+9qlbQfi%I(Bgc(XpWQA^iSKr!2kBZQU!17M8RLL;0rA6NiOo&qV(6hB1;;hHl3wpXD)r+7XUr$uHGoup&67&<*E1Nj7rJ$!Pa=qLGFhkLgF|IYT z-iV`o+3bH|Tt=PB=_9BoK0aA##82u3^i5E7JU-bI)L%V8{q_m!pF=&#Gl>4F81XrK z0(pLg4H2TJ{Vgg+d|opFeLZO6$xn6h>Mwu~@$Z#LBYr;tdaAFa%^DP=Z3)h)zDCxs z=UB#eNeul3RDpTU8Tk4Bs{Y=ywZK>)b zvVIoo<&b$zk5Vs(%xn5&J@soo&4@Zvup8REWPJ8^Pq~RIGd&W&IorpLT_h zdKaiysmFe*S^AWEmHzt^)O%oGDkP^m-j%Z6D(4zmZyoQTG>~Q(xG zRq9p#DbksUk4m2|>q(z#_AZe1ODy%3(&?@G|A?%&>c2+TQ@>*6AXHQ#gM{M9*qbIy zv#clnQYE}WyR5gyW3Q|y{%Sl9DE0DI8t3zxQZG{tuQxtIPkdB)u9C{K)*EASf}TRb z=$qy#Dn`AJLyR@bqftL7{72- zh>^dkwW*Wwv-+k$b3m*$qoGz?A6VaTpsB7ws~4JWX!P$523i^}LBzf2uM;x*TZ#Vq>vwKhFI2mw zt-h-n-CyDrCAUg*TcFlWEtmA|e*>EeBJ;nyf4_ryM=rTEMm4Vr z@Dp;KV<6>pV1V?;MsVG!uN4UCqkm8u)x`y1V_xJ){Id{0T9WzjQ_Gt30$L;2YqU(K zDbiR27;CYw15f?SX-qW2zsByDCSZf4qrM*+Ow~qgqY){vzYO#*z wcHqfGK9vaP?XQT63$mVuK}%7iyM^y7Qm!zv1*t{a^plulqN>^iqGp#Cf?3pZ&to8=n9C(W2>dbGiHW_wW1l z{{9y}f4G0&i$Cnof$oO=FM_Tym%IP|qc`lo@8~_#AMd~D+5P<+w>;aw@4mn7f8p!D z?mvAa(agzhggmD&z5nQiNP8dX>X7EW_aE-Rr}oAEec)9vaZ&D`lDm#hDZT&b3!nc+ z|CGPo-=CZQqyE#6AMStQhc5yqnuWQ#O-SqU`;X>c{5sywrcgWt7%} zT>KmN9W7XK-_ebb`Sgun?XLs=^yh!nzwxDC_1|0n8_>+p-BW+x(Ti$7fAnuo?eD*L z-`@TUzwx*Ir!RiGzwW-5@O-g9UUyFJh2ot@FN)j+c^;R1_Wy1FMbH1Re`Edg{ZpR* z6==@Q-LnsAZMpyG-)#Oy|LJ;^&!x}yU$pY?`g3(J_TP*0iPtU8#RHYE#kbtM`h}Oi zo{o1_ekUHuu8tq7%uxNNULI_Wp^gGpcJ&&OT|FXT4;%}Q*he_~%wXfI=ZWWjWH*XE(-q>p8-aJq2dg1E2(CgO+bFVdrpod}TZ6vqk7q{&G59r^i!fZ00{gc&Q z$6uRRknLLiot^eTZAg@kNMu*fLHgG(3=D`1_J;P}We>bi)|*~GD=@HbN?>4Zbzq?N z-@DIky7Yz6Z-c>ouLrGt1Hl0DC-!Z7GCUxT^iJH64h-zQ-5%(;rtsi~N5XG*T!iNq zyt~aFX#HvTxh?y`Z*I5&@U8a1hV^*fjOVBEY{&B&Jh$R`6`nWZxf;*w@w^Pr>+rk? z&t^Q&$MchTF2(a|JQw46Ii7RzT#08ro)_ae3C|UHj>q#nJV(=0JSVPw;um|D04@i- z2+t&*^YAReb2gp@czzzwHGzS_+r@L+Q6??c)Lm**MtW;?kzaDVPW{q)3M%k>e5po%5%pp1qWO9b)TEM^mM4@tKH|K3_jPg zH2fyzuf>|YYi;+GUF()k*|px9vg`UM>zurK^F{gmd2lrAI^}QU^UDS{oL)9?LuC2D zEhiA3G%B~Df7!ssmzE7||NXLoTdm~-JH|uK&m#Y?0)Gphuj9EL&jP%kGBUTJ1Z7lR zp4*rNT#kFvh}?$Jcz#vo?aafg|NQpK*ESpty;*DZrh6|t6*?8oCR?^3571H4xt1S? zR6c4@N0W9vHbFeM{*g($uJ4|-YpJRu%EzFpADNE|Cm(ChfAZXxrB9x_#)|J>a~Z-H zQ8>1L%~c4mrf_usnolEq4TU57xy&Kgw(X&T4O>EQ#sW&NV#u}FBUf6>RRy^w>{|E8 zgk9^qC+xa@&4jm+YosezInqp0n&JIx#v?qM!lC_ZCLvr+;o$x?^$1U)uqEY!ZHl+` ztiBleK8$-qLFmm)GBh9pedlfzz3B}nigw+wFnF+IY4G5UvtkD~JRNv*V@YV>9-=J| z4Rq`d9^BX+JlJA&pW7T4H?<0J#S7>c~@5s}=aanT&{ z)jT(EZyNaQkFU<%@GX1b#?LklY=6nh?HDfxHr>)Ru<&CdH~kaA+~%*^1C4co+|-#? zZc%3__q$-hz8ijeRc=%LmAUCRMRQ{z77oa)w*zhJdk)T6&Cwj ztqb<2!T~&2;A!D`KAyq-UYT00<@@7Zntqw0KTpvwJ&T_5{C4?`2g+Xa^52SlZbAM? z<{QS_oX^J1l)p$W-c|P6j0IM1E6QclcbDd-*F|$PW(ITh$oC6Dv1>|`c&-_BabtDU zKtUk++Eny=#``IF-!;VhSwuhVZ2E;nKg9b2q91-XeF@PI@jglPBhIETC;B1Yk0$!^ zv+2hZ{Sfab5&g)s>FbGpi1%}ee$?6Yi-~@S_e+Vs;%xf!iGGOp7ZLsFv*|A*`XS!0 zCi*dF(_cmOL%hF+=*ON-|7oHh;{AG}A9ptW4Mac0`z=I2{%ra*(VzXk0D4Aas2kTb z4YXJ(jKjz73t${Y`bW0)3G`R5tUMp%mgN|~EX%!q{dpMIEX}=nYg6u2bj2GqmiCT+ z2G0vTH~70??uDuLz$uK&k3+Xll#1NRk%8QaU~J0sl~(SBF|t4T{RDde^t-$~27eI6 z*r@_(j7C~xkmgvVJq}~0@ww(KY~cSASJVpoU}Z(%rY_jS+K@dEYZbf5{#A|>H&y*6 z!0+P7L%|4pU|OKy^A8u>{Z)t;FS8FWDHb;c4j>#y820lxg+FF5D7eyIP&nQ0ToM*H z*$-WCOG~`zmZon4S8NYV0k7tO^>W}_;)*F@3OBrb2;sW8J+K7f(~uwUwEWW{{}Yh^IO0NXLTL&vm{XPt3!AfD?|E#D=f@?KIUZt9}2D)%Z1d&>Jb zU@GrIar^4#K;f^t`gEBWy}aj;l$~T*5?K9m)02n`+9iRams@`T9O{Ph)b$kd)Zod} zSTj$p$B`$z(|MYTJl%#o9YQ>vr#=50VagL^KlC)hDo=a<8ex^EyPrW=<*ACoDo=-q z26-yz>2=Dz6?kvC2mS^!+8!BeOc}fW4Km`LmT?AT9E6Nbh^J+&I*hQAvFV2hD;cYP zgfPhnT@U;%!X#stxM>f)S28wH7&6vTS(b^HiOwtE9>BG$Jn~I51myP7GWjd@xMb@>+BJPm3(`iLs-ezL}4Z0aiY=kp&b&PSH7mBkgvrf-yBoE zz>gsx-f8(}k$fd~e;49u`C9)TVI^PJKOn5+YyC%rRUKdOV}zA_$LYP2uZzN3K9pY* z=)CeB2VC3ck?$N+zNVv)5AU>m4Un%A@>Ts4^W`=l_zA*FzN&vlSjiXoDZ)y=QxsP6 zbx~N!S4Clue8&NMsiQ$cUr!gkZ%d(JA`<;{Pz4S!XzK`?9jg< zto83_2y6X2hOpK@daw15!jO;cYz6cWbamX8t(OlWOtMniOHlW%#{mP!ZTke|?(xWd zi79vBFyzKNEq98_6?%CH@f^8-j<71%Lnjeda_{*C!m3>FejZ^ZcNK+|+=qw;au;0k zKiswrytixvzl4mv9vQDNWvqG{GUA<Lr7s6S=ap{{;MxNo`95XJ*ZL&n!#gcs4di>6+BT)9+xCkHEBTK9 z8et{hp_dR=^6mW%!b-j-3M=`J6OEFu?m`>>$*7mRfcMH+bp|r_d1PE`%GmXN$cT4Z z#)**eiICmjig;Sark4>`GPd?3tYmEZ4}_JBwXYzoWIROgm5i+v)@Asy;LBB@^U8M! zaP1L~e49-94t*E$;hmOm66E_C$}sSs%$M8uaSAK>0>4FA$#BwrcY)oTcod`%b! zQCP_rpfKs7aP_ciAWufXwZ}X%c9=3A{|m^7cUs2Dka3V?e4Tkq87Zt~d;?)EBZXBN zKJzBRstj8xtYxIIF2my(3jy|)Ve9W9-x-g5J52ei{uJ`zotAF_uOs6jyz|P~ z8ifoAkBs-3GPZslc)Zgx&WDVZRE~(}lw%BGU5;^tm5f~l2$PI7?@)-ak}*JGU5-Sf z>p5`V%S7juFHnT^Qy%%gWXiYaYrx~3mhT+M_iz;U5b?Bpdx{ZO@*OHcSjo4i6k#P_ zR{~)rUloOwe1||2gM50-8z@714IX*EY|7L6DDZfvo|m!jCYSmSjkvLVO_68Lo$k<)%)i4rt5gT zQ1GiJ(0JuM4!E|(Bj=yx$@w7gc&Fu@OJzx8u}bDI+i(hDC1+I?!b;9SHNr~HQxsNm zc2QW#Sw&&UDdk)H@^Qdk`MM?`{Wgz$-!tVq{w3h?PRlo&%8|xmi071J4Z^w{CnBuN zaT3D1948~J%aOvm9EnED*MzZHttsCgz_nSAd{3G3RqX;E@3ee#Am77~?>ORV`3}_~ ztmHdB1z{!Mp{WQf`SwmjnB;@6(nMh;-*KX$dbU63w3ja6y=7Q69WwNIWc-mSW78La z$2%?KJjgf*8Cwxg%h*(pu#&NL2Et0lrUrzSjI}cnRx%!<_e#cA3bQV{{c!-aUK#fQ zuI=^6_z$Lxt@i+rcUs0Nkg=KO0Vqv9?wWvEifur9|&gh@v9cTE)5hl9e(e#DoG<3d`5EBxE?>?)l&-4t zE*oJb=bk2nm7HBm5ms_;I1gbZXMn;=&OJm!a()|fW*>IiPZemqavlO)d#vf;Z~xR& z_pg|;wswLB@3gGC4IQWSl&nq55mvGWRv@foZM^_tCF?U6BCKR>rLdAUKw+()fxyd6 z7nw5dxdiE-L0Mufa-9%YRAPKgG_U=tG&lbZdinGG39r^E5`SQ zl6`OfM+w#plYSh#QC;Y4mAShVl?2g-6McP4ZF=Z(mn_?n!X49ai+{QWeQ^EzSkx4ssvF9?Wi@>0w_9-AH6Mf0WhMS;dQep{4#V`tOBhl3+-#Qf(KFMQ>y z+|R#wN$wYad`0fJ*A?cD1uG!aXncIL<^s|j_BwIciUT9L*=F&ug2`pT!p+;Gnwh(r9c zaes7wjq>xu?;)T3Hb%v8nrDyU8_!$u9gj5Mi3V#{26TR=`=E#3zAK)i-X(p}yQIe| zpq|H27ZE(4#xrtd}u9`NGcllJhr_cP?-Ge*13fS?5WYS))%` z{&&ou%$on-yi0m5&y@xK(s|Y=|3C3jhW|gdm2dkjzZE{|biH5flYW&?x+{Fr)%d9A z_u5u&@F~mdu+M(%kS?>|`=k&0Og{>`x9S(P4J7%l4!yJOc&}wP&Zo@#eU^bPvv2yO z{|ldV-}OmX@Bx(B5udWO_@vWiw$>*-jW7J@?`F{X(X*RC_fCD(d)2eseabw{C!H>{ zkv{3i`lL(wq^kkl`(0*peaiAQY^xuA&}DY`UDE5m|0p1p^&-}Icq|^1X)+hZ&pLCmj(%l5Q_gj~5_bJOTpLDv+M*5^5>ys|!lkS5U z&&>5H%hTa^+5Uaf|D8{|zxPS^PoR6h_3QYw?fKk1Y1lux=B zLHB-_+3$VIa=%YHUGKYn(m&vn?vFv|$EM7NO?kh|?3+GiZt_W|%WSz%`ip$hUFMSx zn;G8sGF$6YmRB$yeK&3MHJ|j@x9E#+z^9xa^D*9ium0LdpR#EE(YE$dpLS5k`DT(C+u)Nx?g?(?R2_NS^moMXM=Q^9r8*4v`@OjKI#7F2T*3me9F@1 zlTMdehfn%WpLCz`Nk`wL-|un97k$by!6%(Av&lZ`r}?C7@JXlB*Y@pUw4?XC443+p z?cMsOKlbU%{)-TSR)dwt3> z-zS|evvYjXH~FMn?vqZZuiNUMp&h;7Wq7qu*{F|yH$KC!eA1usN%x9RI&AQK-^;AP zXPHqS|8C0cPkhpU#V6g@eA0a#bnmyG{gqEysE>a)Wkx=tANs3&(p}?|?$e-qzss!6 zr!4t)zB~ZcOAkyv$YG+4ue5Um?dGa?RzV|2YT|-+7(iG@oTu?UQblPr4GHbRnN~uluYI zzkS2=KKcEtPr9Q%>7MaP_kEvqf98`82e|mMdA&aA{>UfYeLm^BebV8;D_{B8;*)Nz zPddLgew9yt=li5P$0yw^pLDf8>1bV{pFR@KkT1FdpLF^C%O2Q4oPW1(+HVFQxc-&F zk@}1cZTs+q2txb=Zmdt|JBz@oUtjs%fJlgA3$bbpmn8xL=#JQ+V0w1}?iga+h^LJk zwpWyi1hF~mx^Z24%eqbN0`)6u2y4Xy!oq3^%d$kUI2f>k0TC)hx1z3KiJ+8kljj09 zX@n)d&oXMVRSTqiM`kHGZxj(*%okxuG@2;u+c&g#q&r&IAf%t2nbGpK@3?W}mi1d*__}Qh-d^tnuG_k~t+hkccdW~F)XS_jU$?DeQ)_#B^XA#j zo42lOR&TFZx21h+`qI|6_M11}a?7T54t%+U+h@+2zovP1^V*KiwsqGr1I1`fYx}z9 zwe6dL*tq5TD>rWGnBUy7xwU!y*7WAq4u{XO){fTm`PI@-T(P`)R`Z&zndY?{HxuLL z*|VBwHanzei_qMFe9C-%2(r<>p*6j3?T05L>2>Yv(%X?E)KYrgrVoX-As?+Dij1sX zw`SY*&Fj;xo7c(O$$!yWZ@fV>yRLQ9rt8zKZ5z1JwWYUqY^7Gu&~;n4w0E>_>1g;6 zsPA>HZJXDnulrC`GpddAl3$nnQkhP!K<|qWK^^Msm^#aEKQv8P*DYu0ttwb`DqdTx z}h6UB@=K1Z$hu zbT;#Q;blJ4AM9+j0aZ?QYOtq8$!vA>d|9yu|_ePg*Ubne^>-zQW>pB`lbMqFcv+@}( zzjXPf%@Tu_+PZ0DYx@ucy3Mw=raLxvY}~qqhcXTi;*P=GhiZWOw!O3vb-)KPdys!E z;P?1Sn0H;v?~C)^3h_;B9uJDrv)=zT-dp&^f-?&cz-9A)%!yz_I=ydLXuiKyy>Cq(D~7!g~%yBnt|r?eY_819x4AC#8snSQkX8}X9qn?i@LJeS`i!U%GSm>C67~>hju6??knc{sx89 z>Dmeo(Ii!zB6L$Ez9_vvbo!@MdilEYp40#Q5c#Kf`T18feEB*VdK8E_MR4B1N-o8#lF*T}pLq-MXoL zS_+1L9p=SSGiTS&nU=bW9`%h2#L{gWH?2*zceZz|+nn0Gt-T|)cHR1oTh^tnSh@0& z%bTxSzWVYDS640xRQjscGqSjFmzKSt~|;L)PjfGfm<20Tji81P7Ow*i-n ztO1V@83P_J+6;J@*kHg((PF@5;^PLK5UUKhR9s-dCBin~Vlm%Z&+k4aVB8t`hs0F;GoC`ym+(=0}hC+@Zwnsh~LVPAe{>| z!M^O$<}A|kGVRTL3HP5b_DpD_>6ousAr3MX4SJ-J$pVY=?}mMF^v!3i_hRrZG?S=9ZtdK_Tg@L3~_KD z!%YK}Gmis?{Z0HiXmB6F-E$B;o)h8#?mpZ{a38~c2DiWhj5gdoxO;IQz}<)Y2<~IJ zQ-2Q`akt>k;_k(L0Cyj5`gokeZR2jioyFaQyBGH{+~OaB$8F_puic=QQYF1P*uh*MNTmd2pY>-F622arfaqhFiQ0dHw_WeFb!H zfDX6#J>KEAVZYmO_lbZwg8Kmai`4Ld=uHJgTU9{xO%8~bsR3b6NAEBfPuvI2MK_GQ z1q(@W?wLqj8W5{+_hEsOxDxR`6%fa^gU>C1KZjoWLD1nAy#dq(=&>NI;c@W#7Wm+{ zzXRI80{%(Fdm4Ox2!80F8nDp$$S;69h42dj(Q_JU<357>4DQ6QK#x1|B4}_o;I?tE z!rg+q4R;Uj#2LiFjgyu|3+^oLUfg}Sd;SAY+(&R9!+i#~cm;8Ar*PZ2kNg(#2EjLE z3A@k|sU(Ieqb$*jzOWDXsHjz>Q-5uMB=OB})7Q;R+YcS9X;xR2mIGsQw#TH*li z1Qyn};68vmF&$yt2XLqAL5KSQ7SxNG;ElT%_Zi%Y+2Doy2=0bC2;)A2yP*+b+(+m+ z7qqyu^FTizad9`mpFV)wx+p6`c8fUkJB){rO|o$fs9POA(&&b>s9*iQ0d=gy$1ZWh zJy*D48+EQ}LJRCjkMa_;-3SB*Y2?gTpMv`UL+ymPo$&ZsD+R|Cd`!U!1)ovyCn~Q`#^Z&R}iQXcBO<#9{l!<$SRoP2LSi3bK?gP|A2xied4#O z*X2j}76p?$gg=l&Vz{!~C_jUuPr(Nid_=*=6x_Q{(kHe{`U471DY#F;4GKP@U|Ye* z6ue5oXB6C`;0D+@y4n=H3NV#N55tgO!4E39O~JhyhCL*D!c+dvzcAjWDQ`V{O3!IU1ASA&8nKEbQ1Bwq7N!Ojppl~)1ykzRWFCt!bEd;+@e#V14p zcFDz;2-06~{5}Os{s}=k?cx)nO~E>S(sLJIB2ut#F8_oe-FNYjN5MKiwFehpBKlwt z32x!?8!dV=5~lho2t>rOhb2tqMdjOr`2wO(F+5lFJtko)Z=z2;F5y)we&Smawi!-| zUIk0~gb=W?S|0rNl{of26Ye`?!o9G=Ui$10O}OWHKum^w2_SDlJZ5x9_Zi$zD+T^eI@jZ|J9jb^C5e$@sGU zMns!}WqXZ?UIlCYKBC}WC7-C0@io3d!5ZJ9U|rum2AmLm23&$^1<6nAU#i**mxxsc zoDf+9E)fS5OyxuN>zIP6d|4l`0gURaN6Bw1IK}W>(WYRnKfMZGrQ#n^u+4A^ldh7V z)}ICg#!N&W+>;0Q<-uq2;0E-SUj0d-&+_U=aQ-k68e1fKFkU-`KDGz_Yba4fGXjD% zh3?RJ=ozkH#ZQ8vf*8*zXM+gORqt=aGP=#Uua~+-{TK^YUDl zp2J&~_g>E}K1IxMX?i{9xlF!x-YbxXKYVF={XX>;?|T(NKE00DeJeVep8>*{Su~W0 zA9PXx*F@ZFO2p$ghXw_Fb7Uq)a7AYgVTGu|{YZ&u%me~`ocH^q;zXtB-VtXV0yZY_ zgJRto49I0s3W6WMxqgrrG!nlWN^c=?6XI^%CrZR)nSvORhs6D*;=~MbUq-~f2e_@} zOrbbjDy$tAm#ZiZ#Kg%`u`3-~h!n(NAJOj=6&W!Y23%8)0(p9+xT{?RtOmR&fc-iV z6v@oAno3C@rTB-_Q9yLTQTQqG4+(x#D&?p2xI`$uAI*^bMuA-MTe0nXvBx-mwosfV zei15aT;PZN_GB7DWIqOvVRi%X;WSA{m&Q*Ikv(zoQFP`Cf0W`!BFj*|jEB7UrA3$& zux-^Fg~Si#6Q-(fTlIROc$(xFg31Gm9Ha80{EHOe809}KR&2X3LiFf$S>B?wm?rsQ zbabLL^!UxOL9r2Nsvp!xQASK7er)j9NEf6r%xy1>mwTi)o}EgQKXvSBeLqFVsk6`RxQ8i)9_SR5U&q z0y**?fPSJpDjp1n3MoEi>SSrC0(j03!QkKcpoQ38`k-afTb7r;=a~p%CdA-z)bBH; z;-P1v5k=4X{}Awsp5cR?QhlM}fTIGiYH$+1`ypV^0S!|5pNa~CEp%WSe!3lU>T7L@ z_|~5fh9cCV1wg-MlK9qRfT=@cI3m9FKLC>gYWQ=2N#PVcs#7z=6Zy~Q|4lExW(M@9 zgY`}8&kU!2v#cKmzn_!_R@`%mN&tAYkM2yAN>^TJuiY71NvvVgFUzA}GoW8+8>B** z8Ld3*JxRR41^nT3jO&N(@5z;brRFaAK|j6*{SbHoqXqjAM1P9!4aA;dv>)`ZDh1%BK5GK9|J^$xh^Xj)GDFOO zIhPj_LHgfq7X@VM!qlz(q)dz*m1T6>@iW}M#%44Odyn#15r0ugeg7i2-+cLDZwrL2 z%Y)i?SfrRE{y{$KzXZnVY@dHpI+Q%I_aaZeem3d$LiK;T@+^HM{ph2*Ap9xhuTHjp zu0PnPcUylCWFiGzztq3$`n&tR)L)TXf2gmXOgsd3viwlLJ(*aMQ{F?@FBJF>S-;3% zx2|8QZz=HSi{*nnufJ{mz9)J9Th#9zD*w8Ed;YNX3wCDxW>tD0RQ*2n4)y!ShrE8D z&a2-C|FHE7c4qzdeZcjL5vUv=zeW4~{fE4M@nas^f8zeN_YYgYU}x4ZMr5+T8?yg> zPxkQa{ulQ7f_JE242a&Q|9wyL{I{^r7viAnVnb0(Hs0JOEAs9Hq36UP(&;HP1OV8(l&ymU@ z@Or)uJ|@MFIsP5<|6V(}_R7CTL9K|NH2kT?3>xoJ5RiPxFZ)kAKk%Ondyo0a4_dzb z`p);S^6QiI1=AbQ7x?2SU)P^(!$)w;7w$zlQhs^9!aE-by?{UQuIqbmUVT3b{;Tnw zTG#j9y!w9peXj4@-@d+Y@5FRt1M*Mv$I$2w)1{)T^X_}=i9yBqBC!T~zk_v8&Ig14?o2J>WBQ8b%h5iPnMpjq1{m}A zH2?CNJ3ssqwO7dN%)iily-J>%$%zu^Yl7Pw%Z&Js-<>U5tzJIbL zusahjr1S=h03*F!=_35>3}Bv5F7?bO!QP=gK9ngA)BF$S_hJI`WlwCwGUh&X7-fJl zKe%FBHio$rTs@c%^fBKMUrWTe$R5tH^5=(o(y;_;R`y4b&uITtKA2z81w``e`Jy5& zkAQd-^WW6IV1HHqA_dgc2HPZz`KBUFFG>EKzc|~&m@~f>i_vVE98XmK!RKpsrIkDv zkUV+z4f&z@dttlzH|LY&_mKaa^GPVb0@fe(J*OuV#!*79yqNEj`lays_Vd1sh|>3^ zwpI9^h4yz>IvRD_!?Vbr6)7Z`%OB+d`Y`opOb`8Av28~z4jB5w^Uu_t)E6w(UJ8x& z(Y>R@Z9m|*qVpU<-zqVG&GVHjwrvP|`8|{gvp%rCgC6bcF;33;UKI_AXViEEc;q)E zVlMnprh1UtQ>YyCrNCP|!ctw;_cWSMC45xjAId}+jwqPh1Mqsj6zvV^Upp%E;35;wP;7M}{qD>H0Nrwsz%cF1P#44V@O`4YmftWuM`NY^%ruIW z=q8m=@T@%iY;-T(8#KNrQ%55$CqEBmCK~liXC@o*yE9cDcnRry5p|EO@BPHz^Zn*D^&d6b z{KW+D(2p8kpyk-(Ij%fI*yDV8jPVBYvpW+R&+VD?>lE4lF_B8h!|mY=^%tXQJ*@06 zWcpYL=Isw^-c|ZfbPkT@{zSBJd&&R4lP@ogw^P)^(fl>OH>|kl99152d`;y?T{Fh( z{C&75GiEe(SFF#Ff5o;Q)FxfheqemTkoq1Ss=@a*`hF6o@e{rWA5Pa8?Y{=!d%1l} zJbjNYB74R4h4R~#E-jh@crYRL3FYg4Um<CcEr!Tb{1&xrD4c+}Xr1fxDBTt0k`0Vfl) zT=<1{F^1|3H^+DMh593IZ+i{>0DXf?kM#qcy(A1SSKn3uYFpK>^es|!gt`}1zGv%` z%#GH!?o6p$zbJ3W!{5*0KQO<1eZ%-SPoKiPrxYS6eJVEewI@?p^pr;)L*HOedL9Y$ z`UfrVLl}QJ`U81-9x04){wY5^UMk4zzdYk7&|`d4n1V2a4Oyn&ohbmjfGE}VpN_>G z{b2fNVHMHOEoJ%9p1A*STPoWx+Ed|F!mIYQJ0tb6Sm`5;pJO%%=puRZ^b`7y-Wl;7 zeIJ{re@PxdyZBLg`k3JFb1uHz)lZZM=_i$!)XxCvXDkB!M61X5YlerDeuhauC7c`u z{q(?za_A?+Cj!FMPrzRk(_Q^L>-+Rs{f+6L-e&rfD4%?L+LI|NVtcR3rzcZcX(KnB z-&c5i!|NS%{xRN@{R7dXeUrXA<2m#fccJ~0ytrcY{n}WML-k{Y$2s=C{^`DTIMy4W1gH}CmWat2EB^I;`=;>x&EXlmo~jZki`G*iy|OfaT^EBt z-O2MQ)IR%wXL|G(jT!iNnm_3AeLhb5hy3w;Kv722*aRoq*ne&8*8U*+8_XBfWa0%j z;>W3fLHwGGDCh&d2j=w~yq*I1iJ9Tt9}Q=Jeoh~QHM zj~ZT|QgAK#%PG*Wl6Z#44##<59=I0qz4!*i_rh~P?}e`bzg|!LRfz9}F9H6H2Y&(h zdGR*rz4&F|@5P@3dWEm1@ocb08f@8LV?L?IrUIAaUDSuDfu+_k@;e{=RQ#G!)OUIC zsw=O=B0yc5&%Yo(>hCUs z_59Z4OepB+7vfiBtU=<(^Kqc3^?NKn@n<-~<5l<*D!rKH#D_d(m>=Q(@-+03n zp7~3ECdTUlNuFq=0qH4yMS3CTUr2bQSC$S_hb;z)AH$(S9`7ms(C46P?>s+HgZ|y& z2YT=iIWW>EdTQ@&$qpf&V%VY{5d9^`2cG>mZXd`GU`&V-eLu@XcnP<0{u?1biW9G7 zeOF}STwZZiUX70ckNCQMH)bMX4?XlH7II*u2YI8<4K5M7z8iOg-V=XvChXb+@C$?yc!E+EF<(!cL-}B;2L2S+ z-^x%D{SUa)d>rd_+r2c4|%t%@j+1}tLR6<@O$Y?c)ZH|B)vd+*u^1D zKM~VGUiwnNia*K={KZJnmpJsFKyhgLoscI1*pP>o3frX){im>eOVgA5Wq?ikPk=t* z(6=T{`Xpdeo=<_k%%N`?X3|dvZ1QgfebS+ChWqcx6HaD^0XFGdKtI``|8#|i9`cU@ zY|7IN`e6?J+R-NcXuu}_PlJAxLx0^Elm3%{6+P=ySIUFu`dEwjqn-F`#(MF|Oa}Dc z_}MBGe+}rp@>}9M@cX30@A`2j{SL(U${*|^{mnS~c3ZVcUxD~uddpe^{?|DC*JBrz zQ~qd=w<5k*o?tKw`W+7aXD68S*U~O5**;jGE$e#FS2*to5M3mgtdQGIKNN4SwRYz~AEV>#Xze3vEw_taZRwI`OvwZ^r)&@Had0 zw@-26qy42aR%qgd$weSxd!Nj70B^?c1ag}bza8#@Q@+sGG+>i{JLo%YR$9Puu@6?|oy%_VeDTAv1 z=78Shw;B9yaQMw`^zcLdH3BxvcO&TMD0<=#>}^y=vi>$;0@0BN@#g|I(?f>rMjgMM zm&@}KckzgY`kyt=j6V;s8Gi%f&!zYn-)&Ei5vTxro+H1A(|D#pjTcdWQG)9U#(n^l ze*uk`xO@@6jLL`Ri|5P!t}rd+c;X2B1N65Qn13ey&Wh*!|l0gS9LU&A;t?-4EJT69Q67k?2n8vK2fU-6LJLg2#W7K?fd( z?v5^7;j6A;(1BBHOn7322~R+GLzk`MPq~&s2d+bRL6@!YQ!5#C;97JCblHSQdlOLC ztYhC6rzZz3UAvIpb1}SPUy8i*7?GxGu?2Ysfl>ZRR5Ztn`JufL z-mw=1w=4A|tw8%aB=!UR!EIU`yGz~heN+OT~L+M zF!L7^F#wQw*dN#zp6|rweYVf#@YjexwQ0$}XbL)L55K~xwH~-UlbB5O)r!76Q&veZ zD+$so&kTzaOl2VLPY4sz%DzGV@=QEPc&cNuZ_MA4|8f^# zZO_O*rMMyT?~oq)tGgWg#QX}2h+iQw1V7lbVy```$y~?&3$KScS(>QHtYQC!VVOQo z-O}k}?S8rn-348?(&sR1m2?Mk zxEwt`UADrH+RC5lvu-7Op-Cxc?e|ZI9)!(5!Qt7e|FMTSVXgA@C zHWMCuqY01MV#4EZFyV3N{^_!ne5pAMI&dYrd%A3euf}+QE?dEM=-=tG6&$~XL55ex zQt23mr*zp0UwAWv4qSlYCtbF}*KT0YfoIG!;fd((>9UC)_G|W)ZuyaacO_s|e$Y>h z$M}1}6^_3!=PHx_D!}IW0t#ne>5L~DuQutg25i!!!0f9W`nju2`c;4pc?1g3zS^PR zc8N)U6JSNp`J4AK6aO*5X8NcAdzHhl<5CYj?Arv_o-@CXulC}TnKsaS>`%*q1x}w?L6}`y>6i5wMs4ColBi z+5b%={$`2~{Yqvo1m5J24&A=M;lHWL6CeCH0`~I%)I}aV^GAb~<9jZji-0%zV?beF z=U9Ql_6Z}P{0%wFd3Uwf`6 zKKNe>*vr59d=H-auSfiqj{N5XZ}M*ee!0W{vc;bG;C~)qFaOpR9z643i};s1@~;5i zQ|W;C%mjG0KlFp3lJeHkp|M>4?AdFERc#{N=^zq&vQae6jXI$smCT#={t2 zO>yL-`CMnbE%%pV{659WAHI(zd32=a4?u5>#~H?0ko|QOD)P5DT~<~~^3r&j@aQHhSvaA`S3b;rjEcf8H`;OGjCe~RK4j2)w4#)tF9 z)A3ZLfsa&I8E~ww)_|ikCTbY@bNzWaA5xwvno8qS8k3X#F3%JXBeP4kZWi#$AKGNwi+*uZD7!W$6)+Jm(6(a>jp0sKZ^LHXA@TA zQ|Stff4uOhIVL<3;~g))9K#ncJZzN-CovrH;>$2z@xloVAG~lO`e!d(fbP=^$I-uf z;n*ilc;Ye>o&fVtm#xaD8vO%Zwt}l#2y)>R?7J7PY&PNXu+Lt65a!f@p}%unf067( zx^%URr}1wIhBFR5;9`t-yl@eQBVIU){$9g7K_5YPtl?6?Vf5EtIE3!hfl0sEALm!_ zHy5WHT>Ap~NWbdEfYdMecZ<_A-1ZE8g}-r=hab{QY%}4)%_dx+ES}BfyEt9v@pp+| z{6>>LhT#oeHqj$}Pk+Yc7rvg+s=gMdCyuAK#`c8qAq+3*vQ_-~V@3mV@xgV3b@B5m zMgem17KRUW*-9TO&STJlN1^+q%U1X?O$<8lXmpQs*(&{nTmLA(#p!z2UZVUl|F)6I zo%j(9zv!}6{6=?vg7Y^F<6F9Hg)cMeFO^QNXS9P~Fm5bfxcM80h;-R3Kjw2>`^o&r zE@iY6zrd}3##dteMVGD8pY6_%Fn;_JMmzYK?)(npOWRC*+*2N;4<#KYe(uO}yl~|& z-s-`}7N=`nd(ZrfFubSBR`SK%@et!jUdm_(A2s}+RJwev2M_(O#`uFSo68IH4H!Pq zWh;0B#us$i3Ql48K$or354rlq^&fQg@g(U(#I0Y3Ew{cH7M0_{jq?u}^G)2J#3>2J zm$>Es6!9x|%bV%rA#Z=QI30G&lj9e;<;QU0gp`rq*aZ|HgFw|^jhja>8*%18zLDTL zglD*NHo-F$T(%?hIqF|{Js9IhH4uJ=Ltjx(@Kl1IqWq1nBbfSZ$!`qLXHlO`aM_OF z9VWj5i|F}c{t51X-Tt|(nDAvp!2E}~{Vnseu5K-ws2qg6!=3q*MhtBnJjS22 z98dG7_;X?|ekc1E7{A$JXM8>zs_uM@=i-x@F_6a_zaFaQ;wktK%J2VZ zFUibEo@#g@a?XQi1G?0k= zHTK`E5v1?xdo8{{ixG~$1-$EzN&mv|zaTHZFV;|3F=&Gy`F9yGqZf3+9`X0yH&l9n z;ScclTkQ|vd*z)0kgD}Q=JylO^L|a7-#g^`A@(OJAzY*c)g?%;&T2t0^eTUUM0>Ag zI7l$~Vf+YPQP<&0^w)*32Xxu0|0^EOpaU0Gns6NLpDtU`hsQDKz#%jrx@?8#?MEIs zlILFp$vyPoEi@l5T*l_uE|fD|7LsG+e}GtMp6IoanL@oIvxS%U1C4at0l^ zg4|bJw!)8sy3=JVcr14>E?Saugs@(Ag+B4=W-0>Fl8{-joe8+Iu z9d9vQ?^FAG@56d{+A+0|sY)jn0GYBN*b4J;Qz? zavyP_8v{JEgFgp3c&rDSbP407{e%1#gG7(|#(qNys`w<&Dwm$tGfzVG;Ntk$pHyU| zNBmnf{wd;DoW~#O@FnFV{%bWouU{9FHhAE}>9Vu-=@I`;9{R)S5qbQJ7|R9}`)7&&9WFh}%bN5(4}3U1 z+~9u)od=u%0kNs({j-y@Eb1|B``^-ov1i-{C@}@P_!EC_(%RH$BduI*{3+ zhdxt*DtgNAqX2dLr}Xx^F!bfCu;L{gIR1^f~`^M*6$c@kKPi z)BAHN{im3pv>%lIGX`u;I_bhlzrkp4yVKD{l%VE!I6cixANsvJ9a)4L_u!}J$rD~g zeWQ*~^8Cb2kIUm<4LCUImG?+beExI|$oc%`fQY6;#`>ISI*0>J)rIoMevL^77?k??Ym5&_fHy$|tr64zxkXV?&*~PVA)ous_Rho*G(yau_r}wzn;KLFEB^jr^Cp{7M6F zpFfg6wur~)YCpfJA2NTiz)pIUKRDP9jQosr(tB!#xT6!)SrCvs(0}5WU;gBu0P7 zPx+w{mZnF3hB@g$pYG^{w)TL4@`LlQ&HNC*@FJe?cJssg^+64NrS|BxN2ISFdvr2i z-_FjDSszmV6j*C{N&YGWjxCztfsr5AKES@SJ?I6&kogI6eyWZ5UVFm%@z@W}k83|X z`H}Y1&5yL7R6o&0WUjsPPxQb#ztaB5@}1~c zzIocyGegR6*gvvI(+qw=*B+|+aQidae@*w$Yk6gUsQx318a#NEw`(6g^2quj`w(74 z6HIP?LZ0@#lJwovo@a3T0lBIVN-wsECct!jlE*8rDWBAzIUahFmnPVBe8}tCiy`Di zdAsc!QyL0-4L zpCxZ8$?LXHlrQ#c^8O&b|JIZz;LgWTBSn9aKOfVL^Nnae1Q*5|v>#f`{9?_8yBo;= z!+b97Z>}UP`Inf#E=$YxK7;*ozp}}%RL$=^31a+Ve&=hq({U3x|41a5Kc9z0zvn>b z-C(?g`JP2IT02Lb2Z8flVh0gbp2t&(^V^Dz^A_{d%ZFd;3~~P@aFQYMhyP!b1-G`A z*LeN`^YIqX|0w?#=dDyyf2Yg8F;hhAtss#czvlCwoJrJ7yl{7o-VaRWkLZ}6YKu_$ zb>E8S@eDFRPR1Jg{e=>_wnvS}8+X|JdnDk`=bJTVmeyoJ%IQ5VfA6G(3=WOAUyNeE zGUU;C@UKdT`1}rjh5oN9Eq^aY!h{bZE9zo;Yf#lM`b)w`2FX9-3oyR_GQUA2C-Koi ze9)2?jZ*l*H^Enoy{CK>r z@HLr|1f5S&qvksnqx^V$O?b$!<>NBN_z3$UTzQc`_CvVxqC9YZC=?yumEu>Pk@B+q z)P8n^uz?%pq0U2L`FfRnn2)t^9tsAmi+KSV>KF5iJXI|HwUyvsTtxEDB>v!s{c1&& zUNOP&U+_KLoo|6W`T8q=@4{`LcXV1oLFGeCYD+MFLHD3xoIio_Jn)zkB|Lu5<&I7q zKcwO6QW4KYQ5DB*$c*+?37E7;!OyIu{cq6;|MJgI@FWK01#B(a_njr`djfr5i+k~f zXrDs7@uWB=zmEYg9D3vrcwB_Xewb&$??GCBDI~uip^@qJh$JuCf9h_c_u?5QYsdac zYPVcpD&<3a>frU9;D-z4m+Iya{@Jj9SbxXzWj`yHzDo8aq+rlWd8;~zJ}cv+J;yRZ zR4BpOfH=8R$xrclJtoB`dc;S5kiXP?PJgjeJ`V<=I@yaE20xi!4_wRf9r{>%SWtoR zoYz+|RmJ&N^xz+BSJr{_dkqeS%+P*4mKWHA$!9e8C!rj=|zp8zSJhf3++&GJck zsC;n#AL}-i58ycVOLF~NcTo8Rs6TPevv_7F==FiLL_Y)eu|V2uWgn|LIKNK$>+>zR z|Kap3K95D02hMX5+@93=7LcF6C+YHwXW&PZK1zA0y%Zp2f%7!^dvy)Yqig`|p8qnL zzAg`H`(M>T%~jPW(#LrjY|!=jAY}ik|Iqe7h4UYzy+taZ51~G|d{QZtH-453=QEMK zNlG938)^@+{DSab#r$SB(g$9rkMF~G({5R{9@wJ(Yzev-b)GTg337glI6w4#15vY} zcg~-vpG@ajvAuvjL;j1YLsIjmG5AKg1z=~_ZQTp@D46PTB`$udrT6wjTlgMiTk< z3g@w=y5Ghgp*&ps!}i8we_%iG`!uOh9{WM|*4thI$DnV>ptKKbVFsP{pT+k=wEsMM zwAT>Rlf0CFZ~G!KkY*G(H_#U-F`M)Bkl{{5^J z8Dwt1biS75+FR)7`O06@`W2?%ccg1I;=|s-p7VgWZPm*`YM;^VA>Q8(3Y;4YeN^W) zi6HC^@brCCe5vu~wxyu|S6Yu(EAX?lxTOAeb=F$-1iq=y^GaEKo)^KVX#L$-t3IXh zwRE1>D2vbUBK!t|hvPshg=aWn)vxlxMOJ-_7miu=Z3?a?dm3STT1WodQ!4>S8LlN5 z_7VBz0$~3U`PcMQl)nc&cV}%^fGOi;{HZQp#&_^Cy%ghj68|CNR}CFMrqiz`JnVG^ z&U>Z)ATHs+Tk)GPG(RUlV_p1^@{DuwL*{3^iytz6rHdaj{wRsJ-b((8cZfgQ#SfYO z7#BZe`omrPknu;j_#xw$yZ9mFk96@v#!tBTA>)_1_#xvbUHp*ohq?G6;}^O3A>$Xj z_#xw$xcDLCm%8{Ns=38kvTU-?TwvkwD+3KB22%!>rE>%jpI!GVobk!@v}yo_=Quw zcgS{yEiNJmQDQ*;z>IIa$7W6TEoXi&8WGe0+{@<)irL zBK`s={#<-saPbwHseIswoB!FBUOdG=2l3}Q@#l>4;wv(h!_4$k6<$2WpO5&BPW;jM zoZ_axyfYwL=mVGPziKk$0J(UIkNH%4wiAB@K7T3vQ&e9S_?%_ZV?I}|f93KT1Mw6d z_O&K67M}-AdVC-BtQQ`S&wF0_Dtrkr>G6HX9^uGSjrj_NXL%-Iy2YeNfynj1EKgm` z!LvM5Fo9swqk!c4U#72xf2r_H-{hX(2RxMzDpa0dh4w-)6wIzB`9P2TuT9iI*q;JS zcqouPk?@Kh3U5zw;7AA!#GXnp+E*$QikRzx)qD<}M}_!UkB##Q6@P~1_e6I+a0vR6 zmHrU$7(axmJ@9#okRRv$U^wjEk4dl%3S7Wry>Eo*o%O!y$mAw~eBK87^H?T~^vPq) zGClmYdK8yTkMQm4_b5n7*TN5_^k{)dAI39~2jk_*NKl?{g#2Rt0iT!8{wUxI%-3N+ zDbMrZ@(fP~d(p(_6INn9aL8RxaB>FpYX*;>V&uPJeG>VvD-pr2scO{t)(qoom9ec_th^$Alw|CLEq`!im`?T&jL4-KO$Fdc)O%APtwR1xXq%S!Bj9 zUSh&U3r)Ciu?Z(rCS0Z#glPW5W|{aA<4kz;NE5E8FyXOdO?b>06RxZ@;qhw0l1{(6 z+Qe5?nefDkCOlz+2~U|~!gXpvkmf&is)?^v3sM-5`Pp#=G=s?NX`iC`-H|csaP|7y z#p!s2hW9QUOYnTHbAHd{5}Gh|@u?zAaCzWplGj5JJ@b#`tp^U5!=QTTL+<`#=;z{e zVcvS;0%JcG);GKRrW|Io#&q^ z*zwO4T&w*v1#g(4{4)&`p8PW}O#Yb{CjU&q)vmwwk$;9tq5kkK{Kb#_GpBsq@jb=^ z+F$<2KXc0GL*}0?&hyXC$@9+^=J{vy#+v@exq1HC>`^9uLB4;sn8ue29sg{8l{4OE zeXA-m<1ecA;%Pjc=U?UdXX^KSNQ=P_`#s@4nB#r&ujbQuoAHf#{@J+UX8N-#^W$Uv zt32 zYv9l3r-$`89)HZ^f3bWXe@pS74EoUy|4J;6^!h&@e@oGu{+P%AVtFQFxMb$v^v68@ z7t7=Ew-kTVADfEdl~hhV*!Uwf4{0e^cKJsnGDfh|7Uqk!AdY?Rh?SuBm%=1_E`Kl>uFOVPStGfOf=-Sp?0&F^oJzalf4?fYb!-^P5<`~UsT7CoFr@AtQC zo`0uc$G=msJg~`A3!sOq1Ve;=3TU zEaJqUV0=%j$W-L{XZ3mhon?`K=6)|5V|;JJ_qsfPb~vVg-TK&$^GD|({tPGma^w42 zMP`KjeOkw#&G+{#YZl^9aN>`4zqfIDjZNmKhxHD9e zzf~|4On!e;FbdeNC3zVRSrY-*5e$C?>*?Izj{$6){|b71UvtlM1&sX(dH!24&wu0f zaPs$2r2l68-U-$R^7=?#Ux@X*`RB93f0O=@I)5MjpyzuRl`qz-!gRU*o9FjU*nb1w z@ZUV=rLzAPoQ%-W-+a+}zH}@~ z-XMxt`6E%aVOYZv{yj|>AD(a0hi02_u+fAQ6HK^tmI)`vm~dIW2@fA`!o$X!@JMxF zgqF9w+{72nGvUHTCS0I4Z0q>(g(kjah6xwXG2zjpO}L`MgvX6D;j!v~37uXlW#TI< zO}M(+gsaqn5juWtt%;vJ!i4MUOn9O?@PXr_{fu$HpP_v(PLIg@eijV{1@zbXeiqLA zo)&Vym%*N3e?*@DWx40)F#id8-`mE!-}5+rB=7rNJn#EjEbqL-LihX~=3n4`Ut@m7 zwcKAh@)y;?o;dk||7Cm+!}qbg-?K(Q&f~%U3jTiS`JR^d zy)NSVcd(!E*ZBAOq`wx0KNxlWJLIQqhsEDZnIGCe=y^S?{GNvM0dfA{s6Oq#MbGlz zZto0Yku_xH{rUG}J-z&WYwRE4{Z{brrqO`2tp)3YArI}Be}nSI`ih#bKZJikSMB%0`8{M#aiJSku!_1ElQ}fVUlM))w=kbIljdfw9Uz{hO=R6eVS7gYC{Wnno|DE-L%hQ@v zWAO8w$Ab9SPaPGj6#XOE|K{uul>1YR-;ZYcqNr%m{AquovmY71uV(B==HFYx{zSe1 zkm1AWB$*GD|DCAs;3S^lRP&EGU&ezc{R>V?Y5B2!7UvNpsX+98N$kH&QiZv2I7#}e z;r#QAu)j6Q^TW>maOb=QmJjzLjfvNPb>#;5n~=Dp2nCETsLnNesXode44*$hQ*Z9n$-E0aJMg_5NC{ zPmudzNnW|%)jdB$%S-vA`5-NiDL?H;j!66Ml#jFjp329f{Lr_3j<3pxvO2WseLz#=Ed* zlyh9DHMSSL-A2JHfye$lz6ie)aNKQwM@s|H2acfi1@$k-2TDJ>cSJFk<%{bFzb9Y7 z?Mu}c^)J+(bpP`B&DgH|3%~^^pV|`f63#=B1`hjgVSfzykzZqf?jx6phwyuO@E1gE zc!1OE-VuTB@x^dC`U`uV`u!>7-`Wx6_RHzhczK7#`;YMZx%_+eq4o-Swq7bF#2MO; zGKxw{@cybZGXR(H);NaSN&pwjvw(@dn)>IG66*i?dY0%%v*~4e$lrY{w%;Fy0#G4r zxgYpah=PmapD026V(#DR74~-pLhVHY4xYrXi34BC^SSg&Ji8O|3k6pg#ec*GKFQ|; z5?aO|Df7$xNxmZfy)4F`EVVHIYq5PxWT70D;8!lopXup*z(O96Ncxo+Z-i;XAAbM3 zG$syH`XkUbadG9MmX_Lu1tj5he>^-_-n#UpAdZ>`Y4aj zNJQ~;CF9{m2ADtdi}UZXNxa-|RD!*jELvVViAM9_!??GEeehaV>(3}6bc+!Beab2c=N|sMlbQ>@>nj5eXOW#r+Ixmj; zj~bePKT#Pl`&vs$_MiOB?D)z@$n*4D7p(sUtfX!swd;G2ORo`Jr) z{pXLlJw*PJ_}eHS)X&(Als9$;{uR=@)TthlpY|t8{p0qB@;Cew(BGGK)_)fYKHv0} zfVX|o_c_vE^?fcx=ap6g9wZ{@JK-tF`?KM1SUA6t<5PJ7zZ~P)1gV$>d5GRAANT`D z;D6pHYN-Fgep+fzfyTROItgVFQ0oWo4$I%OC#Asphzj6|zuq6w$ooaT^g-@l7|-+; z3fQH;`yQ+v&u$Ci5D5HUO+XNA9nf%4!pAfmcF!*X4^dMpzUlaffU8RbE9#aJ2A9?s zZ~nL9JcW?R5}M1S68)bizp#H8Z=rNN@@Mko2i_|`uZK72^U4S7H@x)l-*LeIcAeh< z!J(7`lYcL)1``h1CTy)T;ZTbUJL6%>R_7nB*?EfKsI9&?+pp9qN(I--P z{XMuF_5+yz!`|Bm*L78SqDSZ6b3deO>FVmkk}X@;lI$qKv24c*O5!*-#<84`L?#3( zbOqYD0)%gIVkbvLkQ>EK?CA_v42@%;A-K&{#a}6wbx#I@3r?AD?|Fm_Nw-{Hwvj-{Be!+S`N0g3OEb@qdO*1JgnJY|28@wkwV$M6UL@BMmyCF7kMpJ<(MB)4z=37|r$Ndh>t?f&xAM4!~ z2K~6-C#Vz8`4i}WY-V90ham_`*%rKz6F7vuH!%9&Lh$YTa|oJ1d_->1`jaou_8{9H z9M32^WnacxJbwrJb}R5r;`xC-&%gQpU~sUE`UiyPQt+e5ACLBO{Kz~0oA9UXwagEv zBHDivAnSXm5BEcv`5Wi!nenV6^u{yDXYN647yJhlz)8gBcKUmTCjLZu82>wh6A9MS zATPx8iNA&MF%L98_;0YEkH%4?El7pq;kao(-fx8d&+?c-^c&@2{tyrjN&je(eDqcX~hm=6G?u9rR+k;kc%ENaEwx*MY`-mhGL--j9s^n)M9_ zkNdyZ?=djO%ee<}o#WHGyuB>1xV~xdrwEtUKWSi;UrSoopLF1U+y_L3j3mB=`%#g= zdGWV9f}=^R$a6vN-<{%kt=3~3YCV?i%lz<9ML)hXqunumA1eI^;w3&3@;i$6XV~7V zJl0ndyqq6Jdmuks5+5}BdEUR#7W_FNxgX@*VAaT{sOuX^TFWWEql))ly!+8dl5*tY zQND6`gg%yE%v(I_?~Kc3dVF#}hrAyjIrVo`kK(RuOH|2;ZMWg44NnSgR(zslXh8<(dnxX&n(YxXzKL1 z?`3ULDYL#5-;44fAeOg|^|cm#d=K(#{Eu3JSNF-J`Kwh+&yQGsLHb{cZ*TYT2W!$F zQ+%aj@e&`^+Umolk`EUO7Cy)N@_aBz@$gMWVr-Ua+A=!nc$X%9KIF#e`Tzr-JcSNW8``oGVJcc2~&!}!4cBredYc=MmN znEBZBi)z1PJ}KwvzKOlW_&E01If5r5y4FWSqW z4}f3#1K-o~8Tx6zbXC${fRhG_iw%h{)zT4*gVra zrlma|nZa%zx{u_4y4|0j3q3Bqq|c!j^1G7yYCh5bl6FW8e`Rj>-~+sf{z1qmQ1YSu zKCbScl>Wi-9{MBvO&~k@`*}+`|6-rf{5*s|4L{g#@Kl27=XbKGeCim|O*5C*H)W-X`%zC_?Z_Qqot`A_n zK<@8khqC)2r%LY;0_SBvB&%NsX7ZOw@$7Fy8{n^udj71ePjK@+U7zsm3GVAcel)1= zub4lLN_mt&=!g9nQT7A<9qo(zmyv#tN)LU-ekkC11!FurFjE73Qo$qQ-$nx{5buv* zag6y_1OKAJj}oo}uD3}20uKO&#*!X^R{%zXTDTAJy$T+pe-`A?(rUhf_2~mML%=_w z@qAy4!#k!oCzZbde^1^6l=%(T?}lS>SCmhW{W^L7%cO@s4WoaMf9e?JF&r1mMf@o? z4KT`+cyT>n0*v?vre`KeZ|>7mFz9vS;igbtdn=Q&Os0qbOlzy?RrQH~Bb{9E3@35% z%77XAQ2sJoKjfW#KX5n>R*1{HF3^xS(6UGQTPR zg}PHd^(jkVm!Chc?_&f}(*8l+$A_MNg1$k&{7#QP$lr@^r9Mz}M!w*Gji0}ip|3dT z*B{A)Fq}?m{e`)6`R9F=Y5ABh!W_8xavNob9xwS(zjy!5aNH*CNgYz<=Mewu<`3om z2=OT(A7x)gl|4Df_KMQ`a?l^p-w~df(sN*j_68i2$|3d_xYmC8!rwY(Lis4qi~m*j zt6}pG$F0jrtI;>qC!tQMd~>`Y#_AX3FXeO3PPcELTfefW)3$uro1rxSAoZu$tq=N* z_*+*#i|X%B^HUySH~R~J)AbMe`ByBbY>j@Se#d@|Evg^-H|&@1*AMw-?3=2ecSb1r z7TVMNQ)o|)FHdUzY0sm6pT007U&sETy@umbMm{;1CQm-XzjOjg-|z8@@nhIeU-0w? z`s&?>1bOyk+9P8>`<6TUj`ftBV{Z4?TO2_eMY;x`^(t7@ws+D21elex~*h{Gz{R z>`7=IT+->WpNy>OcwO!#J;om4z0Z1v|IJQ+fIk>z>|wYW*=_rQ9%p=&{<_K6k04|3 zRev-3sQUjdNB{E~`xk8_y^x6JC+FGk12eslS7P%I$1AU5QmtQ*e@V_)ru@>KO56Lu z4E?RAoczVjq`$~QM1LH6fcGpQuZ;Z-L<&_)YuqLq)&zpD6W@q`#D}lD@-21kPpK2bz}31O0m=Wk;V@ z@t=3>mGocHk5qpPnf{;K#H?EWuy6FIu?3_18}!w?AC2`l)c=oCH?nR+Kl-O*uf{N*c>K^_$a(aX zpF+ld73DlY+TIcFXGr;hzKOg{|H1fy=5X;5&K5jC`SQ8Gj&>^Ao0i zWxviU{?Y$U{V>|P|H?{%wioQb19E;|^9On5J$t3>F(Q!d_!wsGk?_YUw;I1VUvl|V z@-zCn*xvc{qyCJ&%JG4B^+8`CAIJW%zi@utQ1lIv--^i7$jh{!sUP~_+8f&A2^KC+ zj_70L*%LLMjB0%p{~%-EvhDBtTgv|2W%Hwc*Pg(i>6G?ORm&JVQxJ81ha=&SqsRF(H14gH|cv47Hkg+AT>?7wpVf}xl4qx zgYie?10~n`ocHDzYX0@T#h-`4VFe^K%={bLFFc=ks7qt+h4zKFci`Aj~w z*W!-`&iF-pQxo|Z{-Hk^|H*GJ@R$1Y(jRzTs?~4W1IPd2e3<^vE@!;&^YjJvW$m}8 z575`tpO*RpsV~(BXFLRdpf6+ZTBSb2ALZ-z7s&T|wx8+GpeN%`=dYKxCzGh(vv+tO z4D@;99r~kJAN&c|zCqrK{< zqmc*tn`d7kpG^6oWB)jRQ0)yRH}Y`oC;Q8?8yxzVn73X+*^k>hWW!oF_%lKo9@Tg`eEDVlhphav^FPI(u}7Fs zr1nhoH#@%yv-;=y1E8mz)tA(MKF|J8%*kmDI+6|2L_CpV0 zx!HrUd}!bydEd$$utNQOAFJ3dwpZvG16-~M?E+5$F4-Nn;M`%rIlJ8l`dilE{R`op z=})tMyw^TRa}9pLAKu?$2T>o^<3V51;)UOQ!NS6CRJO42i*N>0p74wJ-VH4Lq88x_IN?;E(4+SI7XGlGYVg8enBlL4_R#r-zbNnXhvqlHLa6$VT zzs@i8hvq>s?q{R@5PoRz=ay_gw3p5={7^q#nEGkxV}Hmw`Y7!u>;FOz_(A;TLazFO zT*P<-?C+vK%K2bL@08#Rd4taZADIK3p!cxg%Yng1z~?1CS>^9&Vfj(X;ETYAuD(kC ze8u3)!29~j{9(7ozaa6+h-WwHbLWY#IECt4^rwpD z;3*e|exK6tUa?mTnn2Mn@^AYK-kT&$doSPVDS^fJ;SZ;254CL<_^{w*?xFA@@z&m+ zmim-0K|JCw5KmJM&T4uPpO)P(_Tv%M{w_mDmW&TUe^K!*sr&%aY@Y`1&{L@aXH0fd}q7?PIf4Me=PBtdc0Hn-QzevMs&da zZ`fNBkD?lz^fKZ{I<_GPkkv$Okp@ zM;P`;mEY~<&%@i4{J0F z@is!SjZX?g>hpTWCxPCP1U&`m&G~%Bw{k5??vFfo81gNd{1V@T_(ai{R_Y7Jv%fOu zjrg7t$~zjD7;w{c!Hvj@EE4bi@Jvm`AN~sQG_Z%bUmGwI z5XN}>@Ju^kba=v}ysxbS7z`0cf8~2eq`!-Y_|Hv=eqchWe-Qoa2u=CRm|vxr_^j#r$MSOC3ir#fc>&9N zaJD?ch_5T4J>+YFe!+es-i5Q71#`Y>^3a4%pzIst>pv&IC^)Onf2GeWnfI?e4OqPa^4Q@&-|Cr{!K65e{yCQ zR}1kee(Fh$-T8(fXo&w^Ph@)Lz}+-Y4%|ie%7KI0bx%(;g0P|U=XxACkNbzZ9T@g) zWg4Ff`^ES3{sHE9I=&S!(|>Dy^~PYYieCiWqw=fwi@o^>tLO>=s3?%ppW#H}|4zz!@)?$|y?^G%`vt@M z^0}{Xef$50MHXqVlB8#OO*vnI_UW?ih5pa5!?u-SD`8D04X6@VHU*rGZKYjW? ziWZ^&VE)WlSQURnc&=C@EMpe<0sSy;F8v`dpvKn_dhp(8m)J86Q(qK4>U=>7=g;L6 zenQY!2wi?^No_sP`D#@81EGW(CHy>#HTC%AWxA`6j=HItNXY_-Vya@cpvUC1%0Aru%}ob zgM5N!H{}a?4e!I7@t{}mBHt1fz{YnCCpk8-p%>|cysZ!UYsvEUdzfCopE`Uz?4~LU z4}$Mr{})DHQeTiKJxHSUgXfc^y{vvX{U7zoas3-ew4yi4lU{;^u|7>vH@nCW_{I7U z$#3>3{;I>@&GpcK6dhpwnE#Mh+aCKOwuh|2u)H<#vQWk={K)YS@)0}>RrU3(|FXjm z-sg^JK;>ikb3qsNQFKhotEl^om;f-wt7WWvTHC8yTw0)j7p^kxA@1M}Os><*eIe6( zl|4d!xnE>@uh=Wx_gzr$bL;$T;x^?kN`BZQoR6jgN%4Ql3qZ`s6&h%ui7N zt~kt5UxEPXm3$<=aJr%5Q*pmbm-KBWk@SHmgp*#@FQh#1hx<^vrLBUdr2n4vH<2Io z1F!D`1N;N{ADR0xK%Zn6e*yg9e3-c(E>C~LD^KE?R~i2f=ec=4wr>^fgED|AUcxTX8V0Ngmv_I-s{zcFXDIeS?qwl*h_^3;#*Z7>o zf1?12??rtyh$tWS4fFwysGrk~Y2nA>cT9`F#Pe{{*N{fl7fE`8y8doeAIewlCHDym z{Ysxw`6+pefkSRXkMuvMew9A0Uo~G^pQId*DK^_SvE->1_QdkB7neBl@Mm*S3! z8eyBb215AnQ}P!1Qywy&IP|FeDLqykx_(J7^uS&&)(3yS;L)S>Nlk#cMY5zmRWtfj zu4eS9tn^9j0rjV>^hxeu=XeN^&F^+jN#zrd^Z zxy1Ui{WF)A{~OWAfs8)(4rKJPSM+h@#TM#wYQLc0!fzqHpG&oWujuQ@wAia!vY_mdcMFDU(x3#^>_E?_T1 zbK#Y&CxBme7uWvF0VAh8g}3^w(ld{0zt9{=5*X)CtCl{TKgD`o6#V*q>e3^uPkZ(d z?Tz>B2W)z*r(nIWWYgpP%qxz+$ZysI`nVn-?d6Qez<0ub=mV(>qlb%b%kbgW5`!FCh_5Fs-Z??0fe(JN>r<0F~ zK70J3zC-(KK{!a^!fSjggul?5v;7JBiSxlZD=(ohXT~=(erbCT`kwD7p?zDG{kP*m ztFr$x9+a@YU9|M8@~|Pt&+@DK6a5j$7vt$NJpiLV^!boeRxl{QyQ`^fl#`G`j^&ucQhd9M`om*Cu@IP%2ibLhT1>{E+k=X07wO7{H zwS2Ju!+4#hBsnt$eK7YUBR%B3=6D#PzbHKS7A|MdPfrbOr;I=$9Md0$82n-JM6;Tspg*O$g~KW86eIsV&LwZ_8! zVAij=9YZr)Y`!HyPu33hx*Z;;jG<{k@eY4`>vFxll4io z7tZU;`ty3O-_!n?{T|*Yi2V|~za#er;=Um6&wwGhKkJ3X`=QzWJ5N54uh*XXemUwB z`oG>EX{h}%&CfBt{*LnxC7y4V^QXLjhv!dE@IKCsr7J~$=E47zUq0%S^X=+jCHpu7{w@)dvfH1K-A?5lXs z>(8Ek`U-PD%CThmLJg%a=Y4a?&-HMvAhI6L`(L=gvrsxL_oJM^`AmUX9?G9j`XXB& z@}s?F{axe(Jj+w|4*V0J1pQ=xx`y-5(2w~!i)6klz(jX3GV0nk%9Vz@nb$2SZ?VPdU{iO2q);c4W#FX`-$e`;(&~& za{r0&!v%8H-bfF6^nClJHuCdmTDbt z|Elai=tcP_lCt3YwR};ZtWQnXP#^38?l+b39rw3LddO4GpHA1*dh(<3GD$D>BR$#= zp$m}jIkX?je>5pedi*5qaUv;2dx%f#TSj_B-l{$2eiDp-vR)1S%C@)6Pp519%F>?4 zfoFNTJyE}0P%Zamp#I7Icc$%y`;@5f(Mho<(*HqE=}E|^sOgz|5Yv}up)nklVZV-G zeVVpT^j+kk-iyP1Ga`>;Nt-&qs_*YQ6B~QZ{xb&oMiuH8-jCDz9Emg8s)7grk6*g<>m>Xm7PXN&ABSFuDKN>HQTy9YlM!%l#<&{yV@$1y7s)*ct^?c&)#4346Ex zP~Tj_mC+Z~k5YfR9oO5b`iY+{rTt&ze_or!k{$d8prL2%*Li+}Z^uDYT2dE777 z5eOXCeK_Cn;ldslE(E(w{v7k?gIi2~?1empo_edvUnE=#ZrAy7zpdY%V}5&{o?`j} z6hhU9{VcD2pE<1Ro7453i-1RaqkTj_5Fdv82tqRTw_D9|~2Ybr#lH-@zUz|_mrZ1Hq_GE6hSKD8dD)yz3 z=Ewch6+3@X`*B|X{vqyvL%dnA`$2X89OR#+SK^zRc;kUa66*{7Nn7f_o}R*eZYKOx z`JWiS)Oxt}r(X2@L;AZ2m^SZ0;6BlCp$=H>SC0k#=04wR@FVU6VtN{v^nOv^w+j!6 z{yF4}^(~p70pff#&-CnXvflxi{`gt;5bckr^1yr!?QiXe#K&>`f;Ca&7w%&XTWT1u z*CV^^-{5{7)JJo}4|*~ceC0j`1WoXNkGyvv{)o&cF+PoB_eJPsdt>~lffgC>DuB^m zCYD%4`_d0!zVLv1DCW==U!ku2W!X2cn*{Q zq{jaU^kDy7rvUyi_rnT8FZa)3zf^rQwmgAPY8d5}@3Z$41;M)Z;NYK-J~O{SSQ^>~ z8~&<2(SI=CE`)<52=pAU1|v@tqdGwuZ{Yt|)P3K|-^kp@40#;PAwKIQ{BiR4qn2Ru z@-57b&&-!f>3r!N?>h+PJ)`wiGgHuyf~*HS_k$H$ zjl5ycFkh11Ec8ErnDZs={(&F(!??d0QwQ+N{($+}+#g~)>09Va^nc*bj^KV!7SKw5 z!OvR2RIrlh3+@v?q~2pu_XS^gzH{(9@V_=bp!ze~`@pfV7?}IWMz;kAKLg%2>ifvf zbuQ}fOZNZZQ-Rn_^#3DC;Yvu0@f7+#MyLOy{o&7DY3DEC$FnzxR}5$0r~DE67mzpO z17*PGeL`rz$6!C$9%Aoge)+9H&YmtcA47d|U(}y#c}{=CzaL2EXFn*d0vdPlEIB`W zc;+_ezN8_v|I`TEL)~Y@^Kf^FX;Sx>i+?lHFnHXjQ%k0%Bt7*9_N10z@;%H1Fh|86zxSzp&3_*3F7eoT9ZC|C_oUJm`NT=+*h9KWt6Ua|zr)dKZ5~ z=6ujL(Cgx_1$`+!pXql z7T09`M&3h{_PjdNAFfFIo5=6>nzpxE|7u-LR5hjNYxl>6Xe;X5#QG$CSKl=7GTs3O zzHkNbS804Te#@OT;JeN)NP6POfWKPPi}wh{zX|kvgyRV$kMv@nba~@Qf1Rzbdb_M2 z&_14v8d|um<=VkpCF;@g~osO923j*o$cLCJ?hr+pCe}*3@1IKLt{1#vjp_ok;&?$`kgT`gb$n zTdjPtocChdKZw5t_%|7RT+H9W^0rd`W6<|}pR4ca{}JkIY-f7N7x8?zLBa6hSLh$g z3vK|ggLu(Lymtbu9)GxBr|v^!7PK$)L+yuX{kT6~#shtg8#O)K;yfNWLVmuNhw|!i zh0_6i4c1p;yrR`l!AENNxA=j)tMB!g@uU12LT@$REe=_c`YZVhFmL=Cy8Qe=s*kL{ zq!(lrz8o*V77cFgNj+}G@W8Jj`Rj3cSU?N6t?}XFY9Fqx^Wo~K3vY{C_4|Z%Esypg zA79q*B^o^MtKruWez(Puk|+C%@Y{2_pf!El;x_#rqRHP~_whyjKBK{Rt#$Cw=aR&; ziM?R?NwD12AM97O=I^II-~)bRT;cps^-sX)&+5GTv?)J-n_oVzPe$ef{Uhi<-iPeP z008|k=}T{=>0HFGes7Wb;Ow)CPtoL}y@`KA8$4s=*m zzrbbj-;F&2ztuaTpv*&hSwDeD`*$1T@&X1-%YQ9tOJD?XW%n$j(-}CG* zV9&n7z94?mwHH}?3b>eWqBL!9!B0d75frKRgZ(9?3Why_{zPtnM*1$GE&tfh#`0x4 zp2MCrYD33E@Qak z*8u)7@I|--{2DraVY7f1E^TsP=x1KmH&XpGgX*R-d~u9P?{P zeQ5u@)kmbS$6eP8+U74%ft>zOn{e=AFRlJc{_1ruep}p^(Whl_KlnAI{;dAO9$zD9 zTYqn+|8`(_=hx8kYQI`Qn?8{9qeeb>-{37Pa3eELvC8`bY;Q|1+7s_p6TDS}ZzlO$_-Xx%c z$*+1J%IZt&b|0VBzqT73JoK*%!#}@6X zW*_dr@XoKH>ucX3pn_4JqYt!Kct5~z|3a$&f?uQ`k<|}r|0tsm#f(1WJ$*oZ-uQz4 z7P zf5Lt_;8l5x$NoR?{lSNHe~W_2^4|nMfjP$dvyL}-Te?28m*a(6s#WV}Y|n6n1gr7& zXMojI$BehwuX`WR_%N!1U-Ty|NXdE_?K#Jna1YV+*Uw12qu2-jg!EYdpku=i_;Vc* z@1yQCFxFoXAE4GhxPFWB=W+jL6L_w|hd75hIk z9T1O!bAa2SFB-;r2G)c0`Yh!o@8g>IXQ2=A{T{uDAC1`G$p?sX*Li6hK zM#(Sz8&f{%&8xE?(td+n4-4w1e8h*KJ#nGF{)E3!pKld?Vf&@>!FuF{o@4yVNv(WSo<-wmLFX#i-&r*5aTL8@MZk_(#5@5dfrs2KVtF`g>;0gW@=$HLt z;qcsH(vJr~Ex&jV*S=RP?T_{6w<#?8+bN#^e4Ddh!}D%YipP31;|bUw7oLLtLOyu^ zPwkiM_gLAV4iK;RdymAeD*k|Y@OLC`TaMN=Gb$!|T z@m?%096tS)}@`u7>(4+iS`+p6vp`ZMG!@#0XWluhAe-4V0VHkJ1O~OR4fLz(Nn|M|-690_{)vW%c9FO?uH^r$1=> zvk`cyPwev|d+gaO*k3iD(BlQ|XT#(d`*R_C%=g!g{h&Rz?_tULEQ}|qy?LJYNu57R z?JsvEjXf&E9`SgOhOxg7{c^@j$~z}ZZ^GYc?vFY4QTj{h=p*0j&ZY8%eEso*_PrFm z$?z-VeH3uL%IH7FS8ser`#buY8Ly;#XS{?x@ZsDV1HR`O?7{eVSiy`R7Ff-v z=)b7|Jzop|4gMMT!zmAmANXG8y^?FP<9oVm|mW&Nk(|w0Ymh#8;0nzWT+|KT3Y?-)G`K zkbm;fbA0^u~)qd@Pb!q(YUhZ$}`!w`?a6B3K7$kvD`8!^GGs^$k z1FO^gC~q#oc0@xm(0(EQ>RMyp(Y}C_&7vBTLSQU6stQ#4WJsM4U;@Eo`+>U2Pv9C) zCaVwd(PZ7aCLm=G#wf2UEw(!UGRSx*4P$+j^BcsssOp-spMd_6&KH0`6Px4O&@br2 znOJr^P$J;}`kz}6uktY|f3u2LnJn*yDEU#ozt8zXI=&!2gZdJwcoKQ<{7gJ7sii!L zZ(k$&F>K@8hljQP$ov8JWewZk*t-+S%3(jg-Ho@V{;ZV!(Ums7y_CjVgFnV6i@#>$ z+e^y6>iFaNq`oNLndzl}AwT-h8p*6% zyn3`!!A%pd9;NZ>9l?pDH)6c{8p;FqoA>8v6CwJGcy+-KYrKh1$9tWENB!L2fP7T_ zh*y?)a*0QWd=T$l+ylJCpHrT&&sM&mALCEa@{jt7r+ia+1JC@1|0Iz2CenELqSOz0 z;{9~CcUkuPj6d~g(ktmvgv1vCFYj9#JnAcQK~Cb~CBEpO%pX?? zRo;Ap>yaHi;^6at7K2DYg0**(J7wo|ls>ydR`|MQ&hy7BX4z`s)byp-p} z&kO9t&u8Mx(H|}~ejffpk@7VCcRXotZvZy-3Fk4YY;l8!z3g;hYyTOKEBc7*tGx8M zF3P~q^1RNs(|GzH%Xl@y@!5&TUvBAv{&GCG`VV+=e<{t6czpO5T<}%zo6Go9)f>#p zKY%|c&smB;4EsBIsH7fn(*7;)J79l7Cr5vnLOiKj4pH&*;{PcBjP?ccMZ7_Q=ilF} z_gB!qGG2^KFf)^A&l&nCEfj zxdNnt{J_r~*0<~@FuenJ{JN%U&uf9dW65<=fRXo^c&pBi@?IVc=3kP*Sm`&$57^6f z6RbjyPg!|R{;67xH1ZO9PiT7KznsB(imJfSi~l#D&`ay@(p#EH$4}6clvDbd9_Ov} z26HKZzT^C!RFLvRZQ@6M#+Kx#L=uFb#qsWw$ClvdDb3IEEI(tA|3&aqH}>MS@-uY- zelqqC@|rxfp2EOKe@f4%zNYy(m(?eXr=C826mV9bCMN87j`HUdN59ZNUs=BnFG+u@ zU&YM&#^Uo${`u<#&R2DRgg=S?nD#$euWgKW?rY9)2QyR2`Zd~f^) z+>dAi+7CH}MSNT|`*Y%-jL)k12JDs02gbwxmFfBk_RIV7X1>7v?EKtHEQgx#`$H5KV4gDmG$Rt!dOr2?^EY#L>@0de{pyX;-R-A_U}UQ*E?`O zyz8G4cKtKLu75_jut)oISg-HTEkwqiVSX{vm&=iUIln{t0h9hdLw{eZtgox{tW!b% zXfAEfS400G=xJ#G3HiTHczZqdHwo`(`0!3ce=#_4d43@>_KWn7=H-FNoRL3Z(!bo$ zKiVq$_3a}6!$JQ<{Q-*%TD_m1rh>X2r|Cwk_^%v`P zB~HS}pLAWZ&x4Sy4@a=yGm_{SYB7t)_B{_8h@-jjirH_nF= z{;uIi!>1ZP{u#{=<{RGl343Dv)9OW@@5KEO>3sJdu3tW1`g`G@{?AyD`acI{z1FJ_ z{ZsA_G5!zEe@^}-SKs-m@nLSZBhVh2*qg`Wjmkc7{*3hAd2Ez7d1zE+x8n=i%h*4( z*O{2-BluyQqO=*q=0mM-=fUVD~#_Onc||ychGD4H|*nLe}&dx ze?mW_gFJs_W9qNK-$=?S{gR&QufL`J6^EbmjPE2t_=$Y|q5K#5U*N~|_j6f(rXcT& z;HPfFjwWx3zyuNPqcmHGp~A4{D17wQ`+Df=qx2ehw%?f4DEE!IBYlc~HGYGBz;^s5{e6hPwDb=KYx~rDG%}vQ8u~dtzv}TD z^k44uXVO2a=vU)6=m%`aZ_+=Sp?_4-FXQ>Eq5t(9ze)cqjnAYX_I(7aFhMaEasf<>~xf z#_Kmo0qJkH1uM>FGfNoHzm1V43e ze0vT2oM*hY{Jb9H^+oXGkJqn(pYx2@mY;tKv)}=U)Qtab$Z{~MUB_I?}8tUnv7q1xaCg*XXn>z z()p>3*TYgkI$n=laJ;U+uH*Fu=cDz^c%9NO{9%604H({`~ zdB$tY&+9Q>Uqt(Ce7*QN&v^<~EE zlzu&4tMwf;(W@P=?fmVZWq#e48LwXhKj#^*EkCcvczqG=w8N7>2)2iv-4~1Z@=pO>#j?jUw5YS zIT^3Jq=0n1mghFXF^uGS;?t^Z@7Hbr`a<#T*uBL3>J|4#Wk2jSjMp~4{q-2HFM^-? z>&4G`#%s&Z>oHzm1V8?Go$8la|G4P(InQ`)`FSzi+;C#QtPn&og#M7nm)YzYX zED6&1Ilzd&_u|{I`lK${lK3OMr-Sz{(*1D8%N5gjcf_a4YL$r}1Kf%8>F)kG_qXLd zF5}^WA7|f^fFX>>g!OrKiKoN*`C4qJaye7&R}WtLeq)CmM@r+d2z&8nfW3G#z=bqE zsDkrD??L^DFU)a&{y;y@znS3e~6D-i~Vi#^Xm7n zL7%sO4cLo61MJ100p6+k!F~4!E(iT49!2)Ab3%W*f34!@B>qhHuW_E-r62p-r2p0L zUxOYSe}?$1F~BzdOv5(*Ov9gY=)VT^oA@)?zs?K&X?z*N6-|7Z>|dijEd9`@33Xq~ ztKYwl!oZ6+1MJ0{0ruj}06&xJlNWyieKPT87dyY6EcXS^@qm`TuL|KG6R>`6;V#T=Ika*^k5v>ipoZn;q)?p%E<~bsl3R z3Eul8N|5th;Fs^uC2;~&cVi;Ryho&|b(56XXr_idHrJ!pwf{sH3! z?go~hjt_hg$Nj&K3aH~l?KmHzt{c(kLs}SLSY`eCdIK$GRu0nfjf{^9jE@Z-mOp*`{573nql zy8RG^g^%q8eVsh?Sp{o(FD}ndkM=FmK!?}z)$%yWB$O}cojf!`2%phsm;cqcUqHPN zEc_$h9q)q{>qxzDvT60R0a)~v{BwJapKd>+uMYo?ej5GK{F6V}SM^;SfASuIZ7Ff>HxN@Pq$VOO702{}=rfxY^7>_eM@bzrYXA z!-~b$_7>w!K6oxv^A*^$5xh@TM}A{JdH(~=r@VIt`*2y3>(b}b(S8fK-*39>N#M1; z!TH0Fb~U6+iTwfm;jX5L#K0fv5__uX&+|T8+S8nV|KPGDoKDN9{rXT>T}A+t{{vko zQyBhfMcu!1PQ8D7SrUBI&EK!z0}OTfJpX<2A(4+d?~#14YY!RJ^2B|+ANKR-gua0J zPthN43n@RGpBMeTEQvnirtgef1Cl3r$iE-&Ly|wa4;}3X`F+Su-w*$rcsY-O^oU2z z(^RD8had3MtM})OyvCDzyPlK=g?vCC`2V1vUddPZm-1Jt`(H@Ez$5Tg`@~CFy5nSPxKb~CPCHvvFy!Ur0g_ZJA zUNw*DwIiS0n47;psio~D^sVc%{v-HB`(Ey*A5S)Poi+3#eSgv?`9;2UYOEPw0V{6-(r_U=>fH(Pm!8{G7raj#rRD&>nj_4{}>eQwlE-=B1Q z{PR6ex4-w}y-Dhu=?^$>TG>N+%KVV+d0E#UvB8SJ{-h_B2jtfocLzKlZusqtmxpry zg{3FF%%unSi=uboJqF%~D)jf{dQ5)E8~3fYx%Fdw>|%51@e%ip=2~5Rf6|fC4|ODy!|NqqZw~IW zlH5W=VEiNU45B^w)3hh@e@N9Y=lSR__Gj=PQS{Hsee`(mtktc5{0Lx_XX-+-Q4)bHlUe5bYtSW`aaeXlA{ z$!GlgNYCs#f7*Vky#Q}e{8)YL*@M(hc~>yE(O=Zpo57E%`i%TBzw`MU_2l(UGyhZe z1@&L9%2)c7!+l6~{=iS^bH^V1Y3RXyi{AYv$bW^Z|KqCtj|0!VDZir%Z}sIeVjX?E zT;YFV^l5{FXGI_Taeoor8JS-zdb;-jHuRwW%T)T8r2POZ|Nai8&$xd9@4eXehJDS+ zd$rfo%QT=oL`$*3pYg%q>%iA10>`i~Wg9knNK98Pn0(Q!W z{BQvn1D?_s#eC_%D`xd|&!rkI?spqKhAopX_}S<=OU~L;Wrsf4S$&4qS_STYY(c zzR(2T&@&z%Kh7I=OnK0UvXj0tlKe4NjMDVAxVz-wp)X(Zb`ZOFY#CUzJmG;J$RqYYoA*3r7!vN^V?^!{JZ(szQpx?!*8~~T(~`s9m~iw zYN9lwpNsp;6L}w>ZI91X9X#ZDwD%~=GxUwepX>RY1EW1^KL52U<$Ia&XX8KW{v$7a zPt{3}{_!whpTiw~YCJ-F$OA>%o>HEEdo0qw9#6l=<73Bsc`o<*M=j~j*q0+6P1JAn zXR*8-!`}l8?)K;vca1hXntS#m^L(|AQ9)Nx^^8 z;*$dL@3;6*2>ufm|3F0iy%s+!_*sj;KTrIa#Xl(c2Q7Yoj`$50|8c>8+~W6z#9v|Y zydX3PW-R``fcVQTJ`sFk@zoCE*IE1nf`7o`=i6QV>rd_%{QVYxc$tH*_9y!Vzu)3N zjRzr9{q0Zo34Wi&|JybPkNo!u{yvL;tkuC|{u>KEw)j6R`}msRYZm`+KKydJQ8_qwj2=?Hb|DzdL-VvP5FU8!u78{mZ$GO%&u2^^dkMH zeac-l_f71`sN|#RpT~O=vL2`9kNdF#_D{JVOX%C4mIwHl-oFuf3VyGL zj}l4;9~lz*gWyR>Q2c$+yJ_F=Wzr9L={N1W!^4O1kSC8#`)={@XwO10t=kj#J9z8e zxDPN&>R$O#veARX_-e2GDB0rSqh!K^!+2XJ{Z_ z-5x$l?($&x`*J;@zRwH#^Y%>sJH7N#vfG2h`1VZtTRr@dq_vHzY4m+^|CY4BAYLd+ zc6sTu{pYXnUV(gfpcM4Sqj3ojI0j9#Kl>Q=i-OONKQP_EeF4BfX70;=6JWV7t{!|< z3efk(ne`vu*D3EOc6R;~#s^uihrU4`(@Z|C?mH_1{-}awK1ui^3YPUG!XH+!%;yPz zNWqxT!~WGuot?tJes2!=4=TLbpX}`XM^iucCz+lG!@N&~^lE*E^vZr_XXmdn>Dk|$ z^!H}V{~ww3Vn0)Q)cT84{sJpF345XGVTX3g-;gbTfeg6m*6<>kqgi_XB9p%1r&sF_PWgYENniKVuLG#=x6|_fStdQlJE5QLoh)<; ztoGOP4$}r;puY)z2k<*xysV$5_y+L1UA(NUAn0Q_Sv zUiP`0120J}a3`*M0Js2z#& zw*Y>_%Yd;z3V)F9M9Xydi!%PF{b^3?(=xC8RR4jeejX2QWBIqT{O1p&{AwE{7SzSf4QWR%AfUpKX?c0 zyGzxF_b?Yan}8Qi>iSOm_0_(gr}oCxYHf57@T@X()|Sl-G@Rw|26OVHT>gRX`{;zfOFRowPBh)Ya2IDMm zi&OvAZvMsfZ({n5Y5m&%rS)SpN$Fo)zqY5g{xvMG?$kfz=3iX@DyAPu>(}->tskS| zE7iY@69U;!Q~dX{ygsM?6>k2;_4hD+w@Hu1nO-;j69A0~IqRLj@W9%y?P(48>* z56~ZZFK3$VNBxciSuYp*F#nxAG@%Q?{PxEk!8;M3c3OT>@M8a&y$K)vXO#Dt(NEs5 z1%3gm`?KDn;79Qu1pI4$@JA$GkPjv)80q=`fyT>ydR}}a@bJ%>{}TN3ZQ~WUg4kLkzzk31v_iq$pP=)Z%^#O8tvyLy}{2AVqzV8|OpX}O&!eI|M z9uN*g?gkkc`)R_@<9dFK_v>#)?v`B^z8S+)i@F~Z^5{=)k@q=n(e-hD%>|I-Ob>g; z^0>e;PZ<4=?}hFPs-e8+i}+!{!7T{*fekDLOU5seczim3X%YXIFU|jLLeHSAcli9@ zdh5mT|B&VP*4t2f%dHl^4a1|$|Lwy6?HT^RlHvcb!~d<8e|b+8ubdzjgS( z)AC;w{@-r-Uwqyq8(+Fe|KG7R|L+ib5Q({n|GVzG82)E1zjwWZ{NH8acfkE|`TsrP z|MxQde<{QNV-Ej!S^njHVfa%<{}-Q6xgh`VUz-1SD*js*^MCi<7sLO9mfySIN&fG) z@H^q2x%|IN_y|m%S+4S z-RG6Zo;?>MkB?ja_q-d_wCu6)yWvi{^4Kf#*qf2Z7c=tsH;z2^WaP2;66Jvr2jiED z-@RCQ+;m=f?AUQJ@|dyw@3@Kb*kR$D;2yj3*eUYZnUTjAGV(a&$YV!F9y>2l9%?>y zY4W)7yz*!?E=C@S<-c(w<BY!`WK&&cER8F?IZc{ox zmB-et7bA}cEdN`tr#!Y=_aQ4^4OM<$FYn&9(ClgH6xF0mnaX;_%2l**PK@# z6B8FBkNYkE6W3546BfP((=k^b*NQx@&B)`4j66Q&$O8{bc>d(Imnsh!kV}=v_<7~A zWy{6LW54Bp%Q)q+#lqv5Ub^zQTI6wcMjl5q^7sQs9$PZFsz>xg$^Cilo34$(F9&64kk9z%L zKSV%r@>rdb$7eJ0m~-SYkdepgOO!|bRmh|Nyz=N{JRn7ASdo#(XEO45*pUY=`0@J7ic6G7^;O8D=e+XhW<2Bt;1?g(&#<18 zz7K)*AiR%FfA}O9s5pPXeHA$0E9Y~k@0^zPtnB*Ost1v_xg8?_{=<8_9v5gBu01V(L4QM6G*(H}7#CO|}*44*qDewb1b3YYTfE zcs^Mfx#iC%tD@(f`sS0t=mih1M?Z1kBgsmH`zvbPL6#d%6uZF!$>-&FZf|Y9hvRV$#;eGUEV8F%CCvS>e z`62zP8p;a-rsw{;Q3^GH0r=dlg)**4m_W%kJdZz(PU#m_J?$Q5cFY ze&`Fve}l(+kjc=SQkebeib6faFS`8sWHj31B@CvPUkl2w zV?Jx@n}Ym@qY0-xq+iu%)8~+W^_$c5hgsfG!{Q^tYcK)AM|tD@`eXp>$)@~{mZQl? zVUI%}-xuBL)ORH5Sns4~dG)PMd9*K0C;yS89q|*ke5N17gu~WP`}3rizC4;qzk0$; zUn!jQ(wBxa=?CDzJLMM&XT9`zFV&$3?T__!Lm%67`HmE(edyip!~M6W@EGuYyDSX* z4E|PbG4=mQ5=%!zrg(hGE(6O_)Fj)`$NIU0O|ds zJl>OGYJ9K?vOf03;0JIh%z1wQz6u#nRdOBwhnLo_(n}sJ24aj{Y>3Cc8##_9HedSq{m-cr&e^=G#ry)N%PyG#? zceM7Pd}YH)&-ZBZfJL5&@7uIb!Y$JH$=RE^ePj2J_?|b@7xa7H$MpWQoX19eoA&Kw z^E&x&a^8pLdCpyO{@ldR&nFv;uDwM(q?C{I$+Dlu_*3_MbY$LRWE2bP3(?+Ep^ooF zJZW*^A^fJ}F%h3xa`WT7TUlKw4Siw$*-9YepFV%icvNZ>;+JH64wJT^(FhulkJLAx zY`W5=H^ec;A{mzaKWnKa6{x@Dm-6S6@hh7^8~KNL-x3vAdCn!Ock(CFQ%fdcEzH1=t_O``yCaLg}!6Z@eV-LdpYwcz;sP?}_}7 zpZm2WUHT*PqkmotO>o|SM|!}L9{Blqt1LJz{GGb*1^KrUFZa(de;izU9pS6t4@-RH z^QwJx|DI2-=la%yX}`JIElS^LzaVe;zslWE{%;=TZISZz`@n#&;rM~`z;gc#_FLOY zugvdI|CzX+%Kv*)kbgnFe<$z1vprP2n)EO5&;CIEQ~i!2DUb2lY@hgQFaG;j^2TV8 z>GgZ^^Ko=N#|I-H>~CKq`pErl(1ZBjtAjf26|QX=YRCOJ(S*V`1%Cqc{SxB1j_r;1 zfj=$qwYvSmuUu6zT~qwhezJc}*I++E-=pz*!NZ=Z{q*DwdVEm&fc)zjKVa?$n%sY& z^%H>QzTck@`tuRgzsO$hzsqYs_RsaquKV}nalL*ez=csoKiZF2d}zk!$JifKUn7k7 zav#V2K+=EsekSHuUc9z?uec!Z@7VbFdrFqxAOF5TA4B|o_I;c)a(e^o|Frb~EI--# z;HAQ^IDbU@X6u`W|M-pldcFqvo(WC7zM_xs;Wl8ObgC`A`0YZ>O;ddnwYRo!XF>^|LIOq4d{R;8 zEiB%IhqMcQ@W+?=^_y@JgI@#jxb(GBz3?ggT(ZpL_e|)%hevx^t#DVsd>)TGJGtO&@?(Bb75&!yaQt-Xn@`rsb(N+(%tt!i^hcA`azR## zzf8re>GF;y19E?n!5>X}@c?~F-_fL71Yq#<$p|ibO7Zi_+D;cfnye6koAg=#fblrA zzxc#|z83{pe1a2mS1=9bGVPo)kmzSgkrC>eLtG?wbz~e zh~E_dLj2JmiGMy!e#9SszB4G|z69NyM1SBf4~ajb=huMOkRIhf=jh+dc^L!$cz@7Z z+FWhcwLcBGclG8`!0=DuUjyzQ*u2idHN+#agPHS$z}Hr691>G8F&3X#6XZ z0C`RJer0=ch^@ld!=9P->r{W14pg>i>@-&y(hhZ?~M@EBiYeG2z=?3D3R z+Xu!+ksh-ji}Kz>Snfjte_na8_q8Oa?xS(mV}+jz`V;UNKeW9P@pYNw=U ze)t#O`5@eP=8Y%7%lR8FnA!2xOI}=uFqk9$xqprP5YvS2K-qO z0{A@>_fVV~kNMTUTWH>uii4jp&Igl3YkNK(cf1D%4j=e|zrgcBRPa1uwC9<4C@l~9 zPvHFX9;7zwZD-=)R9;9A{poS-3Fa$3(q7tr%qPoxb|b&(zw=3dPs4*(_HaI%;#c+5 z9XOob;raLB>=mB>jrBTpUe?xkO^tJyIJ+&7@r(H- zeXBHp!DGFwhx(AhlRcvj9L{b{>qq`@_S&?5(8K!O_F?_5Jq>3k7T52{Yd(2H&w8gm ztQYoJ|C9Y~_UcUiThjV9|L*#t)bErZ&TdZY7ka_J(>_?g?4kdi@_%^`*NY4s&Th=q zzhQCxPI*$l1BbJtY5hVk>i6{F#-1Hcee=l;J-Zw@oL!gJ5BkH|p~dw(^*iNB{SF+?cBl2j9)+{DOnKF`Jlt1m?Ef;m|C3uL_J;Ng z<1^!7#a?Us?%DqrGWLFn`@QGmx_AB);{&vXAMEcE{{OLLO*%hQ^ND)UaP9xSCC>Ma z{rBc8=s#Ki-Gfmd{Ka(sBKBU~*H@sYGfPj^pWp1x(9>~~kVpQ|Uf%u_;4^WJ-5sAA ze?0nxo-Vqh7JsSpn{4_^o$rLW&vrSh@4&Kz(8Kjteu~}^dbdyzJ--RUQWVJG$ng;R z&|eB=y-MycBfnu0+R0Dkk*}4)P_I8Je<3Wpc${y{p#r_$^Fl`+^Fj1(n_uwS|17HW z+y4D&-u#jJ8Djm<e6;f`r+n}m!dQvFOnyK=&!@sVs`(T0!`~GDS^1kd z+LF1y4dZniD-?Sw^^3cYUzGeXz4H8tBbZ;=@=-tYOaIg3*?6eVSIc}%`eS~X^hf$n zNPh(LuWWuhUL!r|mHuq}W32zD>48W8N#!H$W#+e}cM$VgGzU_c^Z01bv^}J~)qR5@ z=bt!lfcaEJ(b@Hc%zU2w$oQ=1m!OXm4t#3=4E_V=7szk;$9#eOihm{gxQy{q_^3}~ zpueQH7nFVhF2B*@760v0pOl{iech3rFM|Gg+>c}1OXNS%ZRN-M?0lc~4N{*|{-r-x zIGo~B?d#2Ngxb3!F}~_L|@zBKmA_Eq(7JfO~x_M?;imEyvu z%Fo)@to`-uE%@{7t+dB0*jMyF+Sk;6alf~k+E>UM{II)fdc>X<9D549bAS30_SCbN zlux|usq&m&Yvrr(9ol?FI9~0DM zKc4-3X-eC}TOIqAwTF(qQ}(OUy7?9-y<_iQ8Wj6h+I+Re!~WH*{ma@jl;_*O7OrQc z_U#_t5A(y)dxU=4H}cbQqky_U{ZF-Tdl#>VKOPTU$i6i!{b-L1+qYcuZ#$jzOWC(8EdI6Ix387nEeu%uc7x>y_WrlezOA^BeN*9D z+P}E4ecQ3Zw{P25`2O4IivMNbs1fwv{#DqwHCF%6{kGV*6_;Y)xLl8q{+IY~bpf@% z0{e#eU4FnXZQl@YtMSUd4O;lT_HBo?M;B+`wma#UvTvgn|JvPFHx=$J z_Dt*>&JSsR)cjPI$6vv|^`+@$e#-Mm_*8#ddVbol_UPj5+cqctQub|~#lLp@c4z4w zLcg_d3=hNyets+MTknPJn+kW=?QvoMZELS@->&WT{kM_c*Ui4c-r{`@S>Lzo-TwZ_ zYg>;)JQ~-NZejE3_4i5~EaQ5;yw8I5itK#UTi+AvLFddd2+?!TalP z$Y0-R<8fp?%UiGe4Xn@cydA#Zko`lvx!Mmw73hD9*5};#B;XO>BmtbV9tT*h*XjKM z#6w~Ijo}&i*q>?dO4SsW`D&*Rw=#SgAJdC{llx&6zvo+j+iLBRXW!I%8rIvcb?{5s zvk{B;>>2X=>uWi}5%=%Re#2z{j^M0`&%*u+^nK?p;LUjm!~;q^lJqy6m*9Q6vVh7? z@hs2x-Q&Fn+Mg^--}6L&xM?OI?2!(I_##=KjZ`=g<@48LjK`J@Mv4FWKF>QG3{(J; zIxlfqQW%l=aPGIGe$ZEVFX>}EJK-c4`S*l(pkZ-8Smzf5wVzY{H1~t@b6*G-f~TMn zT%;ZSuPq0T!CIY!!qZ5d2!oL)f&$}{N7&y_j{(jHO`znxzSC2|+9$9g4p`pL``%#i z@Ua~3YsA-&ZH40j>``%~NtpD}A4GX9TI|PZd4H>zp+Cyf&jpO>9n<1Zc79R6he`T> zI2d$37O4Gr&CkKl1lXZ?NYVeojlqFK4nN1kVD1kceuq{EBL@SCw=Np~4rcf}nBnhG z=1!!0`MWc4 zuLp11ccWK7Vn5lxa=}?y5k&t&{k7x<_5PyX|BjL!DmnQVd|SW?s+=bf`Rw%Y zbF<6jeKh2#x<8Bjh#&U$gTI=zcYkK&<|7py@Bh`L(nc8XwaWhNSCeX) zaLK~8BH@CD8J|3e$~$y?GUcfSEcDFHwg=kaYIvEtfLqgx_YK!0AR$D@n<^B{n`l|S;bfy=DloEO1)i)C(k zu&>>Gk9NAYPvTXWpZYmnds)H2Gd>sRFN9v;i>zPH%SihofS#X?cWjmS-BP&F?&4YB z&+!A^Ka%poc#|j3#rM6lui?}BIe#4W5j6FE6ZN_95!C0xNwmzZ|3uPOLj3Pwo0V_1 zl`zitOZ$z_;z@VphrO2ZVSF}ROZjRzn4d#=fK2^}*M5)4ck{0)?>u2?kIgTN{6~^X zP-l8=|5-leXUZE%s@U&M@ysvs-TX_*&+{hYP-y+b{-S;#568qH{}IIZyZKqZ+EG^g z6ypy4UKq?C(zATxr#JsX%E$YU=6scuSB&$*AM9Cy>2pbE>MsF4f%90D7vh&hUL!ca z;nv^pzW0grc(1d=eJ_#cdzc^NjpRQ8`#^o{(QqfuZ-{=LQT=ly!Tv91glI2KUnuR1 zEJ7dN8_m1r_v1d{K_KP*3Dgh!Q)~byuiis7^o=B~ITs(|Je#y1+Z*Lo z;Lns%n)bKwK4l>10}VYP&L_~GA>K^-6Vl816Pb@-ykvgrEBd$4!}FQKzfKSNmK^z3 z=aUY3AJNDQ?`Pr?K`Z|%%U9*StoZ}G4u72gNPn^A<9uYAAMYJibi5MUQ}Gv4V1_@` zmv_s9|Kz>5H-0>l{%`y#ytkHP|28n_gM&kUAt=Znu-Ic$KKxZgqowJ`znRN1y~&UG zowWUw{M7xh`n|R@#~J^{PqpuOTwccZG5Jwnr_`_Smqok*hiiOD{|x4*N7&c-DSgKL zLEPoEyn>7u7XQsD?3ezz6ZfI0uPNt4ct6TKT&aNfqx=;9;pKNZ_`^AQkNbN_z)mvK z9z2Zmvm0pOS8~3v8tVtrlD0pyKZ$>%`xEVhv^ZhNW9~sTr&rz`l@09Rze0KUcB1?{ zD;p>wfm;CYs%)sc@b=1vQ4Q}!{EN%~sm1(%HEHdj{EdIc@k!Ye_%odU!Jk$AU!5OV zi}yvCUgk&eukqe{I$wf*JUlbmx&ip$9fNpJp@ru^B!EfMqdgv;*;%modrALIc^~Hc z0XqLZEsOkh|GdAt-*TGi**;kBnAEooK>wxv9-i5{Tgv+f$}flduD{)fx83T)H{N35 zdoJd_-7uJz6;fS)|7W#7t?KWb&0S4-sq@u?IA5L1x#z3c%K*@9k3do@Sgd2Y9G;F=pWMGjQzxY%oF;2 zww%|&eD94^ck`ZQf3ii+XG{Dt`rr5*%43DJcc_0nzD6#n3;uuh-Uq&__$Gt@Nq$C3R_ET{Tu`d`bGY_Ir!v(Zl^Kg37vm+q6%^#SfV-S0~1 z@Z=A9U)P;f3n)Gk8vMb(W{Bmp66e2{QElwb6HZTQo914^I2uSMla21I}F8UA!%c00MLJmE%qH<5qH zpKJe2`QD`EZ|&RqKRm8-!0Y^` zexOxQ`u}G2tFvF4aV^pq2t=U z0rhXme}THo&c_*dR&Da!H>!W-I>2#ZYp1%|FQc3Cn-)ie0`E!j`&p%22Dhq!-UV^>35&Z=t z5fGoB6|lX>d=2#JGPVAncEW=m)N;(1J^F`c{_~a};#l_cNj1Np<=LUXFTMT`w-`9# z(cT`}qm2D0dyM%s($7O00e{E`^49HJ>$m@NC+x-2DEtdFUugBwER^}z?kLg!HL!pB zvEOe%eK6yPwtr44+;1}DSXUS9YefJyc2L5B^l%`uk{Wfj>v~>Z~#f9Q+}D zkN=~nZ<61~&g$<2^8G{1~Ytt13c<48(AMJ=h>o3?J z*kfa#NGZboPJCaHB(CpMM15QTSWdfj{eZk={!kuEb^B}!zvq63apU`NgOK;0HZOnB z&$noON%wIeJ+u!Pbe<1i1~>Jd;iez7m&f*@`?`R-*w8}QPdrnX&%UA>Li>JxS!$-W zuPTS}0pd${()lWS^13{--Ky~?j-9weshoBLZw3&ANBc#!>aVE~dB_|7+`a>d z*`{ywYfjI1F9JU{LzcgU9)IcdPv1%WSa0BU{=@m{T5@MQ^?65l8vnY^WYJ_Gulpa6 zFLjt8Tq9rj+x(ILGLk>}hw_j2&qHRTh0j&+&pQ4rUz0w{PyIi~mY|6k{@;)LW)z-c zM_#x0Io-3MUu^#a?H~O`Js+U^BXEa!x`hw&Xg1-qsuhvIKfT*(PbweS8>&w_fkD(S zEl-}$fIsAs#r;JlL{5i?n+~|)fg5M%a8xbDk*#+&04{q3p$M)g+h6M%VzDVwwaA%FXDGfK~9~R`SyAJLKxbxOs3wI;j zea5|T2GXxL^8wUfhCX_1AC{vQ3>p7>6Y#Io{RPe6g*)L+oAXVC|Aw|N4b&mm{dM%W zao@ts$h|B;|5o46!t<#VpBCr{!lQme|Dbx~ybblg;U<2_KNo`v4%ovBH6{9Y{hz;xhD%Rg+Y@8&ZT|uGmGo~;w@~;a{z#vr zet|#JAFz0oKgN@&iFb?sfORh%zX*F6 z_NSo#U{4KL~6{xfm*9xRZ(f4-8vf3kwT-}i>t`=wd}rhJUO zxBS0h_TI`@+xu?M9{S?}w|>OidtIN{p2I$_e{6$S|H|8EGrmLlTl&MuvqV4r; zbj1M1|8T$c?DbXBUSCD__6pkT+R#43-oswc4&9LN`0aIC*lV1ZW_zveb1mEj_Uasf2bbP2kiB3Y}A$~C5v%c=Mx=0h8Mcj||f*wz;qV}%mAAzU& z%X#Fc{uA7o??KBrb7j6YWqh3~r}eUF|n9{6Lty&mIHx4!{=be|Q? z$3Aw-^Y`w%MSoyIoQ)X)3muM3zwdsLjK;LNa_1j^r8vybormIU$kEi?rYPjVrG+} zQTsZG#+6ddwtuYq&k428@{92|%D)1CsRl^aq_6p_sfqZ*_{8$3_YZ3RYV>@1dH!lN zf7+&4{%X_;pT7+g;jgyV;V+b5#9vBR{qp?9$*;xdZ{tMxn=!-TFGhZ;qHYhmKQ-3! zg$kuce$ke1+eG+Fryc%c<)`(@xcFP`^S5at{MFSt{Kd+T_AiXDeBbNyw|OG`)zbw* zcyf9_XleUM6=|w|eEh93{9UNL9=j6vYy0+ih1Sp7-k^SKlN#hq8x{v;x>>%aJ3xmxe|2a7u!QlUTQGZuHwO!)44 z6}nGhez!BdmiW>37xw&}owaMpbXt3yTG=_fj>=QZ7wIQDGiE-4`a9t7ot>$*T0fil z8{n^=fexaBpQG2;bpJ@>6KknI!}fk<=QVmiiHZMC#Bb)CHU4WffB!a2D7o=n3xkA* z(wFvFuNR)JSw;S3)(f#7?5&4N`-}ZxNgbT}>l8osZ;*Y%qxWmc^&IMt4d@Q0DbF74 z*D~?~9`eL}==I_LSh#!k)_C`y^Zr$Rzc>8pK4e}0Xn!i^lWBj6wk6ab`x}~HGxyC$ z>;I4?_qPxp>#6#_Db)X&G+w0jRpO^(E%tAs|F-f8^Zg&lThF(XebnuR_TN)~v98Or z|LAXc^1%K#XFoHQ&&+pf>6r4@>peMLK#&LQCC%rN{kP)}-M@_TgZhm1IqWYiM(e|g zL>_ovufqNcy}v}?SB(0D`|S04j_!Y=eqendsoH)uTpz&t8B(|XfhL+i!*WpGx^@4B zmRpt^ebZ0=xV|NC-HT`F^?FkMYv_K!zj>1MN#ZEnydJ-(Gimk@gGTp-&F*Yis`VZ1 zPX<4WIuj|X|5|A1KV!V5_vSu?0=Yk!)rbH8tn z`8vHG>g*r)*2}1U;rmO}9`$-2>Wl18PyYHDt>4mP>Oc15MC*T;PowchB+ol_e=L-z zu5Ztcx}LUwPm0tV)_)NT`kU|XOyDAa)F+CR^kFx3-=p#5wb*Y#{X;E3)PI@})qhhQ zaEJ5%6dv-z{wWd{9<+Cq2lkKY>S^30Z!Ew630R;)UeQSR8|(RR-M-M?;5O^&^u2iS zgZ)tYerw$ygZ#0+r<1?s8q61ehwd{zLZ3&%W7hY)@r2G#t*#HczQBDg^cCetKYIUA zJ(*mzryjkYxqi=ynibj;@qJ&-|Y+ zUsx|ChV_0U$VabFKmqFZtJAMZy+#U$>?8bXe+lJ>?_h;>tDopTV=YdqM3i1PhG%)=5A&apH-#tp)Z6<(Vc!yz9^Y?_?=R1w z{1(?h{#0IAulpd}Nxgnf;mveVEy)x7tb{%{^$+|5PwiEk3v0jT^b_}ZNuI!?{-kKd zF+u#ojqz~}xp{xkO61>ESQ8(W*ImR<2D1XdV?TD{XNey=Re<{b=yhNn&n?%X{#`8X zzxeY3ci{UiTKwVpAF^)+wr{$2N-!#_V8F}7$CF~vU z+xT&|ciEa-sQHlnobL7x?fJUy`pDj(y~o+VIh5TiVgHam>{-3BXRsfXzH1NR_Us?o zqbXzWV1GS(_oT6R_H$>b%BV?oG%()Z4Mdv_haw?zFFvtQP%*Y|n$&l?Y=gnje- zk8hBDGw1!P_Ko((lpD{zzdD{9<9^5k)ZN6xbSLO7;%z0sDpZe5B3-_8($=s^=qle2V^N@+k0neg^yl z&mXize0Kt0|IRG@_4r-KH*+RUUFhcb=y+Zw`@d1sO^hfFup77N6^m$68@6AtQ{SE7Bdj62VPl@>|_k29&FKNCj znosrY59Xs*+WX~s{u1+5HM*@^{Y3mwd*}Hxo_|y{|ETk)TYfm-it$~I$v@9`)jA$@WW&VytApHvK8gWvyKfj%Ej12}R&bqmRVv6g=u z=JSxgH=mCAz(*fbx<2XnpD(>12=drY^NU>G>-Tg=^%dd${W7zZTrV{1fy6Q7Jv>Xj z2kXb-1Acynr%P>r`mgus!K6|Lfx5SJe-81V;|LJ`z#dgho%lGdSEXtFBZ2czRNwXd zIp*WBUa0poV*VKS2Q2gX#eQDwKQGo$hl|25Y()7{ZNpnYd7Zvf+Y8;I?ELF>n19vw zgxJLR71uIgLfYc(c$w71|vEa9o zl%M|>^Z%xX;o1iH)B36|5t_e-|Eb+Pzl`{fCe*jFKhKsJ@Ms!1Ppb2qrS%@1=f?RF z&Ht@b|Cj0YO&tTp_q96_AJs1VG-Lg!4r`q#4>ZDhijUIQ2)n*Z{`@=yATEIg);r7MSS!p)cHl{d!$$L=(*ZOyH3{{7v86#CqlWhh6+W;QQ5O z2T$wwwu6|q-|tDOv<;8?y#C=!PI?Ov{uwVk(tEqgm*bD{gC4#I-zPKj2Y=i?*(WATG3>*FiLKcJd?e1-UXap0PsFu!kC*bh}+o=R=<@wE#3 zo6F-15}#7N7QY?)QM~q8Oj3Q@>&L&!e6QZ_=Y4pjE^6;oz8xy><7@rzZMpS_@V%w@ zwsQR61N=O7(2qasf4@=2|AczN$K!nWX!Y}`#7E!LDXITd--cDMAAf|8tv~bC;c?+N zs3YUTcd5Q{;rrB42QT-pME7&>{_RW7{t+s_HR>5BJdNMmwE2ZSBzv#>Pq4Apes0*! z`#-e(Oh|ipy=M<+CFK6_^?SBVAiX;(q<2S5dV5vD$=~d~4bk~&HKSX(D76}BoDX`?5S%A&tE^4@a2X6Jl3Bvy|mBnw`x?r{ECUkc4I4q zQhJXRZC>@|S4>Lz(fL~4MW*xDn1Axmb5nUa{2_gukG4JPyyiFVJa-r$;k%QGO6S9X z&ztk(CVta{Li`y2c;~xq`qbY;cRxkr8}Ga~+M|Eodx3x6d%L3e8c5Vi<%R7VExIdo8{sEkE$+Z|F#ePE zZ4%<6`x9Wn!~X4L@JHpPU$z;k5u01@4sQ^-7!8K z{k;1EIxjcsdG`d`vmQ^z>sOKw3BcUB5L3Ur^YKzYK_1?Dd0qcZbuVAOdN^u$ib-#I zm#G?|{5$LQ__KWZjvk+Pp5F2o)<4Qm=gbMkPyFifHB~|#->ikJ?+5|GEjq=iL`z z^H235ioan(@l*b3JRZe|IT5&DZ+TICH2#j_(;onR-SK(%BOw3l_Z*M<2QSk7E92>p z?5of}*ca13=n(yb%p~^@YBOQ~;Q6@zK?df?_79q*eJLhA`?6Ttm-zmHo_`s8|G-Yy z^Qn2$pLz-$*J66#YH`^!y?@PA$XgK?ZMp9Fj9^+zWmzH0wq zy#0eoJYPSy{=ul{>!*MJU~=sP*&niBk^PyN{VnYuOuWBA3NPXhbJlRbUVA$S_j}X$ zM9=rf_75i3|0?YtOuWD0_YdgesEOp?xsO5j4-R|%gP8daab8`{ceHu-e$@WO_oY3a zRiS?oH{Y>g?CF*0`vv~`6!PznFH6tg)BNdaXTHO;CkRjZZKv{}^%1ze`2y{x^|G?_ zJr&1ma=v2yo+fo>O7T2D48FrFh5vw ze+11x4>|IX>uGcz$mMT&S5i;kXnl+RU)!Fp#3?H>7~rqn=nt>#uEB~Wtteyt6aKjWUoVen zH~eY8!K=V`%t5SwG+h<%9V^2q?c#{nZaNe>6X6 z+_=A!+BfDiLw&5>PI)8!`Q1-5&wwJC!@{8-P#eg@JH`x2)ypGE7E~qLXew63@?tjtze4>W=@$cJU zeoFD?{;&0Qynb!?$9lQ8r}};q+!1ag9{}HYa6rT5o(EZ`4 zAJ~5ebD9s&`=)nOU_5qwg!7ujN?rx1pEN((MZ=x^9U6oC?Qs59ufOY1dVG}96Ed2A z?XFSCfTsG&7!Z{K0<){Op5Q-9=0TfM1NaB0SY+>}N3pH1H39Iv*IeAH

  • i=ssuZ!dTe>3zH5KFe@O3t*k!yOf13Ky z<*og@O8hS(J+2S5-vduxw^!Pa@z_oezo@g;^ zEzx{VDyQ94-^_VUyZ=MqH&xsh_U|HpJ-<)#q5f{3^#$R{$@eYNcw3j%AlehjyEBOo znn2%BA)vf)p4A-KpeP`J#IN;J$6e4T@CRP^r)%wgmZ<%s{h&PQK6a9y{$32?pJC6h zf~4W7rp6jK?TlbK#DXT*xPOHyMWQh&nFE%}YKMB_ljfec!>-%;o z4ijHmuYVPa`ci+4|0KU?ul77Dqze5|$}dS9kL8!_oxab(>Z4M7cOvmjXzyq}?{c)h zR}k^P1>8nB)qkrJs7jzJfvN+~_5O#7#jj&H_8YJK5sd=%bMo`&@%t6} zd*6ept*`0bt;OO?M8Nav`IBGysv4$0fAmLRdErlw9eZK_DRSxOY6{fT>B>r3fCP@?rv{9^T@jg*4F-V+y1ZZ@3@v?|63h&-?HKl z@W)p`Ir)(uO^W_G`Bk0%3|x+PGo2@Qo|E4yQiuEwnm0IP=CZV}?xjb553T$3GsU&R z{0o`ZQ@gt}fB&`LUK0HBKNj!%>Zg9CIO~S|yE*G(c-oRU;BD-c;C0b{m&+nmy0KhFW>U&$^QlCOBDC{f2Ds; zzWiG8A9Pkee}3QR|3Fi}`&#jb_YD0d%It6UeX98VV)1+D=byiLhk9zxhmIZl5EZC- zmK;0wI9@#c$DYE!j~_gEtm&yG{Y#ED5%H=1{{E&-J^1V3vHqv(kM$ipwzw|;+Kd1F zA6~vxyyNEnpZ}??f7MxhU-5GB-yivnhnJ^5{BOVfuP?t;{L5nT7hn41%Ji%M{Oq@8 zW>nuRul*5SGlf<4ylPs44c~WPgMU;4HH)eg+;<0gs$HA(FLv7Aea+pO00)@v4mQ=a zZNetoEdIj)o*?+g3sQrK^Qz6nsd9JEZA-Py&V4{#6RZP0uhR5STbP0eXBO}&f{)}( zpiL>H3|5gIaTp6{j=|{%0!nEUC5a95lo*`=DW5@#4_hU!!9gn>=mX=*90U|>;>-=5 zeD&3n_vOF;z{ORcDK7ZVtLHzz`+G$+qwoIdSO4zblmBJk+M~~0?mbpqboZ<4|Kj8e zrVf4Sd&U3r&JX?pO=Xon`_{aMdi2^p)U)$ww{id7`R(BOpa15+zial5q%@wp=ZD4b zqb8nzb>ENf408I5;|GuR^&DKdr2eUNQ+eKkp2MM0)-s6vF)5i|>r;jydvs8%cajK6C(>+TTF6miPkFRB* zfbvvl`_nyrj~{%z@9_j~_=5cV`i~u3vZN>5zwoJr{ZHldHp(T(o_cEG!lwvHRq$-_ zKNMffe^ED~fBv^0e6?7F{`lItV=uS7^pBrEdH$u(z5L%!zO?UO|GIecM?bt{!LuhX z0Z}YIv=9G$`_&)JzWm?FS^U;{{PUTYi;2Etw0?@}t&2Ck1bQ=%c6RL8U3dTPv16O= z{^GGEch`Uws^#ZiJO5J+f41+F&^?PsUj62e_Wi@RzV*`E5Yvs%zVzfj7Qgkc|M<#1 z*bDyasgrm7YVnUg`N{8|9C@uctL{_#F1>dC_wM-gb^r3(zF+$8%f(gHoDd!TfA)cI z)gno46YwDpMgOCKNTWMsi-GMRIn)1nMgM2@|DgHg_Lp9TK0EoqCx5j2FTeNYmro+A zU;f_r{`Pxc{#LEF888+m_LqP3>Zi^R4-fzF<%jnDVBZf;zV!0Bdw(=vEvLc{0zIv$ z@W2fxO@NvQ?u#IN)NzjxfZf5&RQ(7gK_9WfPcXB#{uS*G^vAFg3)8i)sJq~w402lT z91{6Y5}v~cbpg^|+pxAanAwOpXr^!f%-g*lNL>F*wbz>%0?086PEq361^HiH#-ENJk^jsx{&ZA`{BiRe zaaa;R9R)((-lI|@l$w27x2yVK{VrRp46$`vsvb4*v`S&;J-$ziT23{!VBMN)i#g%a zY#7>nZ=O>(rXygAI^DPW#b6!M5#iLxb2y$|zX>(Q8dhG?_Dan!h$rVqueQG3bVh`mS3; z*i!c=bIMjv8T4i}2o*MG3HLY`s*H3#d z?`ZA4Z)(VM!?cj+hYXja$+LfI$n)`OA$7?+uz>e_d%o!ff-3mA7jzXHA3FH-u*b`nhHsLxLijjKxf zfff77^rcYqP(N9xgZ{R(TDoVPNQ|o|M{&uz;rEW6QESfdFd6BV)*~sE7)b^sNl4T@ z7$4Gn0KJ4=+tg6YuZ&BMb}+J|k>F1wA}v|Rtw-o^cVM)PK_TtH((@kJ{p7h1!X2^# zJce(2c+K&Vn}X7alt!bdMjrmGaIePT#*L2#JFdG;xHl{N`2ib$2c}Sbetq{r_`C7Z z5R2RqzjQj<<=4lb72#J0zJE@*+k{*4I|%u2bkb+{bd|rCKcNBW!z($O7NCdbMRfi0 z%LltTz4DuCIup6rk;hQ#k2*FhiVP7ZPLIC-fBZvl_iE9ZSb-_1wp}|Hx*H47JwZ=` zu0m?eH9}|eQy!nF#YvhvmA|TR74T*SD5iX(_-pVXF2sLTZj5;P5DwyRC^U=`f3R)o zx>wY%>6TllEfgica98^d{upzjV^SXT>S`(F?UsjX(B(1J<*%0^QQu?pqw9}9Cot9L z2qt7lWO+_^cbr{})75c0y;9!C8h53596JHM4^?LOk5sl3NwTS--mi=&I}ycS*{&o# zyEO*97k@cB6y?{qbLH@(+pR>RM=xuS*>2S&W?*`8>~<@asHN%1vD&Rfs;7qHpP1bu zJv!6cwcN~Up+~3sn0Kn%Ay03Y&YVMwOmzu+Ignbf(kz0+PXYI|sxh=el33V?{at#e zWet`hxD&5`+tI1lt0)b*6tAf8Z>vM2zv&HbragHntTR5V@;BydGPTv*z& zr8&C-u@fd2aH&s;Us_FV%6K)B-2A#cN*ww5`*-u94Cs%=cy;(~zf`-{zH#xApN`wX zFfGJaz%brThiM!)|1zC_na;m%97JO}|CS3U{VU-ZFMrF0lYBAG{ndZdI<1vuuHNY?Yrpq0#&K)^U?%N!rY++8gBxMES~H#Xt=FOMH?*Y^ znJ(;lS)_yZ1a@!)mpZia6ZE4sD70y0H695Hb-9dg`xDfzHXl2Yj5en>Z-|#8sHI9R34840S{VD6`z1{&U)FT21=*k7?C)w>{yX%WM2$(jlh&*&qczQm4R7y~ zr#0zBMkHO7T#R%h|DeP_OI1nl#A<~$C@me*cv9mM&vFHj<>>6vO5{YJwja8Kw6oOA zM5BFnp~hs*Nm?)JBd6lxiwk44?z%9(7>$_c*%4$9r;19nb1w96hjuHYeM3wkHjVA8{<+Y9DAr$v{y7}Xkk-ygFF{8-dS({1N+gc(M;t$)|GlyPNX0&X z73)vYnrC4wzlG3RVG=AZ{m}pUSbvTkm5A~?&a*Y)xsFEyI*}&pudL(IDYr+oyn=V9 z`IOuUc6TW6my!5&ZGtA zL|EyWHGGVw#J?v>fj*}6O`KU1F*t}Jji57iBj&suM8;?83?->g*-;!LE*DN58GLf5 z#6J<%C;iEv5`QOrbA|A^IDaQRee%cT(&NzKfIfm$I!i?RZm9%T)hC^^f-VIc023^Y zQOm^PrbDMS+Bv>nogep{pI(v+i;X_9HClN3q!zdw!#Tic7q4k|&bdIkYSBHTM{ZqY z&bh#lhMo?cCF}*pJr_uB9cCQPPmP^hJ=C!x}VxI1^2%SSaY3Q6_ z#Wow-JZjSk6RG`oCzK0~R3iNpBvXJyCtvpmbn-Q_o26$1(}=_Iua#*uk$-f0j85Z) zs}?qkN}pv7G5W+_h%qGsKKaaRh|_20LR{W9P{Qdx#U!7f48*Q;I$MGyzX0)+oOPrc zUupyRWOT_{Kz)KSl3_BCG9)8__8zk%ooYdFPa4{8G59d&2_m;WO%VDAy7*}~IKW^Y zXD6Y;?e1`Lo3oS7Sxl9Hp*D*wfSL!Kh;xq-@1=EYK-Z7(>?!i*_t?o+RXVLr>8{PJ zn!7fW!`os<^<<}#Nf}==u!#zGs`Xx+v@BYIqaJ3j%nVIR^+x<_ilTmQ&D zBB2=no-j8kt{DHwXk}vkqx7M&zLbwiJN?q>s)bkloQ@N-_o_ei3U2v%ee|0eKHZ5HVXx3&?2 zcBL@9#=`3>^y}6Tk^&@JL2E@dsI&%<7Ij9;C2hA2jmy+o+xw@s>&zRX%E#1MsckCh zIO@sCFs2T!n(NfL8tPN%I)@d2NF$Zbq@DoRw9|s##Pv_%%vBEGMJ9Wjr!!Xt))@2= z%fzC$;|76Vy#l5Zxr929NYwNg$s!2)AmRBrI%^-DpQEgM{y6qBs{elK!;BWbg8%+x zZ1_t2PtgA(iNvDj7W5M`Mp4$|js;B4_sle7ICP&z@uxn(X#Od6U{wEF^=O=b!0C^q z;?hrq{;#BB)5rH14gZ%@ap@;Q|JPEK{zPj)iTpzTaHo+Me|!y);iLQ|(v-iDKivb4 zYFJiR&=~pSK6I1+R5~vIJz@T9({b{TQW*fc&jpw5GO1`_SxS`Ce8kP(!esJoEBMKpzKmEfGw|Ol>Jd9F8td z$bOK~#f`OrS&fH3>8$8sT^!ED7+tKinX)j~Iv7%EF2o%<>*rkSuQo*~(qE+JwEm9z z8A@{2K~};)+v?2^$Ms<`R-V>-Q>b*2ht1-&<#^pY5uUnX`jPQPD?LGNA(byV1qJ-5 z;bnYdz>hb+@#c53@zagp$y);_qbe5d{ zn{INMYWzmF1*7C3*0y+M@J91L9qay;meJZdXXE++UO9~gGnV!=mUza3nW#)&Pa0zh z^G4LKI6lTA`-#d-hL7VzV#e!!J$gRUIapDcf;wlsGPu(C*Hdng~YoYOg-iI&?K!dGgyqqa-L zKi=@u>imwGO{8_I)@$T!BF)>8jdtJCxxn!5=O9q|5_}lep19!smAp^Yb%*;><>>?B zfp+%eH^ei|?)Z1(3)8e1FAUM*UKx3`pzMQk6gn>vKG>XEiQbh1*}JSK&|!4sC|s z?Gx>=sl!TD9oBaBlQ%zRMUHkw=p7l@mBA}?(adtm?pj*OqqTpu;!x z_oTDGw({2h?-@_E1~l5Y9?~2)$HD$O4&tv=R{jbi?JTtM{Le5ggp)1JUxidNFm<2f z(;{V7A$w-nud8F=`Y>1?+LW&R*utpeOs16Tziz5 zN^ZXwl&_(^Si0+l%1ynXY8lhk3Jk^U19@E0o3|>J*JS$-`n&CL{f;`&*i#qnpq&mi zK;Hc|)mwR0YrchBRHYUwT3kZ!Z|$jb_RJ%Osa`{>EzZADf3k1fpXwS6(;s>#r0&*N zs_}2Of3?M-1gONQ7o!jQSU2@)^iD^(C%cz3)iszmibejbEp7t4_CyP-_9Cg*%sR5# z!XS#$UGUX5=g*`;9#SVw$W5oQRGC`@>A2Nor)SrMJf7_=kw=s15}#?!XX&Z2vG5r; z3nFLcWtwq(#nR$tH^Fbl@SB9uBpVt&ar{#L=9zLB6JGK-Ud>3o~SPqy1gBvVErbOKJMND?W7*C*b_f!8P8#(~!--Xy+~JFYcX3M2iEl;)L@ z5}mV?AxKJP@CiB>Hx7J?&dH4fUrXoaBwo7dT%Dvzccqm8e|0Bz?R~2E{pIY1v`--0Wg^>DTpvZOyp)`R*Ln88Gg$rZpg(Us-tTboh-^yCT8S}p zL#{y`XsE8dn0dwOm>FY=X`uC{REv^YSGq-uOW2i{1n*6;5}(>SSN2_|x?QaHGklx% zmfAfaTbOiLTO1@pIAYp|+~Bb(#bU|c_oC+ek5?rM0|#!ap~y|Hw4 zB_>7(c@m0S$*2^j(uzae3_#r8=}PI?edQj%GR%+P3c%QSv_uG->M zTi~x8cw^J*%zvdeRh{{F&I(M6z5ipM&8^P-`{#?^`ey)SYae@Esm}cSb)~xJUrni5 z=IYGOi_W^Ix46sm_YLv9lu8J^v){>dZfcfKfsRR-22( z;yzsm;qDghH@olmR+#y(wzz7G)9tI;;^e%{{6km)8tzY)jj_-!kyZ;8hHB@PzR^_h+OjP zx#Q0IRrmapaTTYCuk5_g(LLpA&)${CTk52!9F0r2=v7v29L-;_&aB!v zh@v|4FWZcCS7-jKGyh~vt26&^g?s;H>mK`hNOjM@zaHYx{QG;At26(4rQ}NYG`v~m z>Yjfs^EZ1ho>wL%+NR=dKYlx^?)lecn)@-j?I)GPHYi-rFHo4YTZE>V;s`HQ#Le<9kqiboi#_O)y zI7p1f-_`rINXOujHR!EB^FNlkJt>td?e@O|PzYptaXgP@1w!2EAK|u8~rB(~r-Us(8o0kFWftUmsujOTRvTQoi&%cBzE>{e3Pf z-|_F`D}U|R$5;N^uaAFSU;CByVC+%}XZ?wb9#yA$W~vcr(~5We`}oRV_x16WzwYbf zujJQ#r~1fEj)c7#*hD5Vm&yxYBuED%RCLCrN%$wRu&H(gtKdrQS2Wsb^)XWQM&d?pkt~xzu8b^QJ`%y`axC!|axC!|axC$O99^zW z@r{@3*HikVrAl?v$d9WN)jL;CaVfezJiS3Mtf^dMU(YXnIi13l`;~s>a7>r@;AWK? z!9k*6q_rlWEa;X+*ogi}O%8OQ)>@-Li!jgD6cSNbnFfXR!#`4!2i~W(*5p8o;tXn( z=SCWV>)%?NhnwT};{wmtFNzO5d*Oqcf^y?~wkGenTTwUt_+=XTRB0X*Xi=I-!^+O( zQ~!Pa3-w}6w-!pQz_X(5A5SXz?e|LUBAD5L)?b54o2)6$f;L4O4y~4Gj(Y~V1rGu7 zz#I5Fh=(Z7v?u>BH7Vde%{WjsgR=BSb?zzH!wQ$eT?)AGMGt*wF zwFfgB>tE3_+#lQsCD7W?kXoB*sa>1N&5it$3G!Q;Sv8mFF2>aYMtf8#BZ_NOtkp^e zAFM|iM8(+~qiv6izq$f}_O>|isYa;Yddj7dm z|D+qfy~KaN=bta}$Hy6+^!rNu_t&`og%bZ{t?NHj>Ys7_S+r69_Rn(tvsUzxf70{M zmH6*(bi=n>|FuhMYfIj3Eh{vQex27z%Zn?|aRD1)_Fv_u*H^+%((^Br`0sCU!w;4C zCp~{Q>@qJ7KlN^WS=$kC{Oer*ob?aoKys3D0WAuj^uo7`@Da|h$N922zdrYs$9eU( zP!_j8aPv3h;>y(lr|jK&;Qk$KTe|KQ^=o>aId^rTSgI9uuc5>$DzZp9Td1lx#wrHe zwnKZ&u<=gD>en+3vXd9r5sNP?!WB z$WlnYqcj5D6_x(d-P+KWk~Ha#;%rUjUkBfDe9AP&f{|gQJ5J^jBi*FWOK35>{Yo$~ zvX>a?b}aIYU<3xt_AISotXLqAlfb$h#*r7D*6H$|VQZ4CX*wKEqs(#(-l zUumS7=K6}BvVJ0e-II^}t-^Br{qUIMq^EV};(B@`oF36e@mR$Y`HOUD0?NkYV7#OQ$ zV*nYhlAHR95)+q}#EfNLVJz{)BtxN)q+ zRC34h5tqkG@x0Ogj;y>@2ivVJ>4or^ks4Z0fc||hrN_866~~~@V&`SPBtDs6j-S+Y zPco6T@%NOAKjJ^>@z3KC8Gq}?#;@;Oo)Srz5R>PgM43ECJn8XGSpRqoBzcl3$=uh)na%c3&w=BJ&j|+qj;D3>Rxb$1u|#Yu&~WPNrYU9mk&x6UU3h_!<}*TkJ8B z`0={O#DhB<9Q1^iY(_aP9`5hM>QzwLQjVABI4e*o4o8ljb*C_I%o?qu;6(mbH4S zyQ@}sV)Hpa238o0r{?s9tjp1w>08eVSGJ0iYlUZ*k#S`8CuW7qt&__#ty*EdH=;Tg zE}4A_#E#Ul)SS55B*RH}+;~l5-fHfu6|P!g8VAzjjia2gaBUe2@m99Nm966BTH#C? z8BdZGPE@UMVD}qoE87+%J+~ZNR;_T=3XkI)$3%MtRV!Sz!o~)dwZgb%m--`a^irLXR^7_qYdWQG%dXBy%UV5N_heYbN@TF`UAwBp2yF7Fmx)4!$;ibJmTcv2TzlV9%7yYq_y>xgAn6{6yW3j3%@u zT-hRy%>ujfaMys#?Ur)*_jU)C+nwdc@2mmi62bUhlay~I>+1UBE*#puHSJDO(S0{{ z<|Z(eQ@lIC^gFXSwL*G~Fu^F8)|pwHW{Q8KTLO~iZW?(POmyc62H$ymds!>S7i#WFtwh&D=ZJ$yZ4^3?N^Te=;faP5yjpsPxWF$>-EXE z>pz4W-&${`iIE^vsKdY5OGsQQmBA?#&b!>rCgk547d%igfCgYaHHL9mHPsv z*JEyC^J+cCr?0!C_%PH!n?9jZi9di#pLL>o4jSD|@Fk{2btq}>Gs<+1I|tda7e8`UiT1K<_^Yd7)! z$LwA^=f#^^=XBPuopTmlWl3x9Y&W-DSdQF zmdAMAmCxaiH^zzLs~qWCGio2PDx<25NClv+P4#}zsJHZM{{5AyskTxz)#dByiPBlD zX&_-Q2X@3?kv0u#zL$qVF^5O3HyV?-J9CCKzl7YK!2HvpFPz|; z3A9*aW_US380Xl^Zn2Q6BvdshP!`Iml?iB`ra{=jdC1sk>gk!xlZ zY?m&?)_W89E=euEvq8G~kzgqmoH#Qh;|XTcmyHPjtS;5y-Dw*>7Mf zfA8jcwcp;7H?{lSn@771ezg3hIe*n#0K?`7Jyc!a@ay}O+5zd;=e+m*r99d*%5PeY zLp8s?ysP7S|J+PYoTC}_T#cw7-d8L#(08Nx&nxkV>2tnha+vmyT>t!vS5)g4$$mg- zc>aAA{Kp?YPxh+7_G;ka>1U6Gyu7vNg*S*_32 z`aCV`GX`t0cQj(7r>dW|KFU@1KLy3&w5-on{S14GNfSM#rZ-K)U9Hbkd%QJDf1|3O ztM$2LkMy+K^Ih#fYkf@pW4)+at4GS!@Gy*Hc!D_3&}6k52>nSNCsB?s-+O{fY5FRX3v$?3=-@2;sh1lZH8k` zf$mw--{CVR#81B+aV$a@Y|~c-g&p`g{G}tKt|;P|CJ4T7i8)?q^Ms!QohvYTqKwgv zRgB8Q9jHer=QyarIj4P?KX!=9wP~8v1TMMqwYpkYvZ&#&x86@TOACIzo`SDgb7<^1 zd1RmJO<>iT*lAe5r&Z5py2%&zq=IeKIjZgjcjqpDtj^X|t|68CBySj^a$U}n3Yo>c$|7!|@#l^y|{R_<$#W z$mRsjx__2LMDnoi&k6VE;dbv5>2v()t`Tx;|Jb`~D21n-5aj-S$4xQ(0lVE8K*Q5E z16#U_i}l}1(cknWRj`+SXEi*M&o`-b@sWH}TK{P95em|O3kAHUI(Les| z+y~V0B|mt1bMlMtT6$yif4$-BPtI!i;I8^*z`U&iNTsrBFsX)b4BZRVY0rHF9d~s6 z-=R);?)eJ0lRN&`E8LdjxZke&Jok0#u;-qqdOi2s)IraEt;&1uYgD)Ao~t@M_tk2r z=boe5J@;(2*>h)9o9CXTHhAtvx}Lzv&sA!*=WbBVo_nTh^4#?*>$&UHJkOn08P7dK zr9C(5f#8K?C*g6?VWAt1^YW}f3tH8 zz0LmS?eBp79kRcfs|_w^e>?1NYIzH_WPP}K5Whoshw(PwfO1`gTrUn(hFgHWZ(&W{rDuK6x1fWlAru5siD>;A;K^A@G0^UKqZ7}77jrhgwYlBzg_geho z@|_Oo zyzL7Em4p5*EWt1IZ^!MhAb3083q6nv)X+otg<;6z&Eajwo5$OSH~TArYQx)sw-@gK z-a)*>cypbg6!U&+&d2?jn9+j@8VB%Fc#az7~fURwq>ZdlRa8 zGiVZP5Nd#C0*L`d)w^UiDpI z@wVg5zX(6P>U&9*#hb(1jyI3D5AP7(!5<*KOGxYA@QXM5a#FS7&ExGGLD(OHhBt?| z9d90QAKn69^$PrIQ)=LPw4}u;HM9(W|3pf)uTQB0Y~c{z{6@U!b9DTAN)6-9JeE?~ z-+&wM0ABS(O6BnO{U&V9Zvl&U7;o;$lkXd6mkwt32vJ z64ixt+)ySF{;g2HOfURg^mqOf(cigGM}MoYM1L3lXY_aK_FU}%vPBQwo1=oF8nkY@ z7mM7(aHH#>=pHN;KxWVb@H+HyEPv!bY~92U`M19d z0>wl7swsbY>!x$=6``m z4_P;zVeZgN90fW%l^J`lRvqs4pDf@ z?~DK=3+N4wPvc)RZc2~jm9uUNuielsgy;BFy>XMg>JcB+OE><^HLia~QC*GvGpg6R zIXuBv8$GTp8}@`HLF`8TRT>*nxO_alF#Z{1<~Xb+KpBkDK$ zsz|?G%kS+fgP{?*seI~#T9t!tBsa;6oD$| zc4t)qx|RG5KBETmu6vHk|EBM5`_H~R`z7B!{HMNq@LW>83-Mci)KzddTYNTc+{7>W56m+PqTRaN+ESW-Zr=meP5Of5GxQPTPx^w~!<&qo z+jslV8MkS_wQA71O?$0{4Kw~!|0#U0b@y5Ku<&pDd4p&F0qYLS`=iF6>&M_Px^C3x zcGsOz!`oapIz0EeZq&zLcHNDtW0!H0d`N!{S~tm;>O=Db#=n>IW8GQpUZwgTH2z$F z)UOzKn+>1oH13>sXI1uL7oSnh-L89%%6-yzw|&ZYw?E>$I}Z5np+{3*{mJ~EUq6zo zuU4t~4JvyR^y)^G5A16y)1dNjlBT6Mb%Ql3_?v7Qe$;FB_f`9Q8NXW)mYy_1(VP?6 zYI+(3z1h0gAU%3eGnJWH(oS{m^OR~7kLok-C(xIqN8;zdy9^zDPXXuQ24hls zm_BUXOVMwoN5;Pe{pC3NW!TC%`WZvk@u0?q&lwZFdY_dK&-iT=be@Uico@$!r0Z;q zOy>mpMLbNmCM!&r>HTHmVgK>|Vz_`jTcJ3CZ`X9pN{QzeH5~&G@%%Bm!%2M;sPFN0 z&=YF5Vmb6#@5NAoL}$4%{gal?a^{)UR$FPz{HYLwybSvn|+W>?i60^gSgK0oZDSgh`qPQqLg;P;|v@CjT z6o@fAPn)0cUoJl;P`#z_Il;y*y2KVwR{v`jc!+ly{)5K9!@|O!Cv5l)mclZY;qQX~ zqcH7ZK>NpYHvD4YA;aHj1k#ZcxDgw^#Wn#-pZW|k{6P>9MM?TgsRM#1tN)Ga2#4o# z#86YmnX@?lX|v(k&FOPEF8_}sLW(oTK>N$#`>mV86HbOF{X!x@^YWi2>s)K91%=o4 z%10@xUliw03cA{5s}pA!{v+n+kJrLY?KyU`(d;QDXO~!|2buV-8*dfoM3I~I}DQRFNG(SAj7vnK99$w-@eQsvj{_vOrOe(o(~~> z55ng}`gsm-GtB98IIh3fgAfjOKwc69D`;>vzv=Z*)n+qUl7b^n9+y*Jzbz^(T@@X(H?t;_G}+VSA-hdSSz-K2k; zeqs5F)l0LtHs7|w{QHYbvk%|@VAtxKmu7c=EDpZ1`S#meZf|LEfLm9@$GU2HOUu&i zZL3y0SVUTizHMd8?YFhuw!*>fdg#9UyLLxxQhFcT_0ZO?TUUU!d+*=s_#q3PZ}+V$ z{D8=i4%WT76F!f ze!K2Nt{u;o)h)z0r+eSSPSnd+uf7d(b*syL5ASwDueg2r?W?iG%|lUa`V{s-=JOJP;ZO0NdoyJFjeP95;-&hDBC`*xI#i=Z@{!E~t_$5g&RmyS4MahacGSV3&3;Y9^|AmS%To4FkNXO(6fft-Bs( zC4~64?z;cJ2Sfkt_N`r8vnDY9b;4^84W^Xdw|rIh-aTDATxA8k!=Fhnib}`n#P{I# z`yY7Ut(qGB-FU|)?-nSImA6xosYR$H}ifQ{K$G* zU;B3$`65sHUi`W~kzlonwc(xgZTW`&QJlA{njUj%Ni@~|^wy2%m zVYDG{^hXJQ%c%#4_69u1uxQ0f#2H1P{|@$vey0~{x*f~U3olr_rROR zh2c=*Z5fzGhw$o;VIlp~pG7>S zvs;K82~+%omL)Eq|8CRe^$wOl9VO8G+-c%<ho`%Fxf3iT`|=_zl-eT@8NqJh&xr zO}<9Zv(_J52G!LZ7{8nC-`9ga*P*j6q4=@=Iehx;UlhMfr}CwT;$ORJxo=poNbB{-|x^} zeA+feiO)H77vJa5U4G6wbQgcl(n+4K4!dOOv@Gw^57?6}QvQc5UCRH6r4#?ymKr`I z4xMZQKZmmBFT!IBdiWg9nm=y?eL+Ztjk0v+hk5O@bgoxSKkDFZk?Pp`9lT3_THu>C z8hfJzp6SOeeYN28gr!ry?4OFBGuA=$8#TrJ({3F^U+B<>%FweGL~@|^%l2!rrE~jb zdXvB_?9-%YslYS6S>Ur4-y-l#-ze~1cbNFw1fJ=e1U_%^n+2ZfTLgZc#kUJQ)3*!! zCX3%G@J#O#c;>TP;F-Qx;0JB|d4Xs80fA>e4+=cfpAdNFvsd7menjAz&pv@?`cZ*r zKKlip=}!wh^En{!Og}F0%;yP#XZk6DXFg90Jk!qzJo7my@JxSB;F-^}0?+hw0?&L7 z2|UwZ6!?yussF9njpAdMa9}@Upi$5&zOg|#e^KC>J}mI8 zhc5{{(?1)k~Y#YRuEK4dvGSUT%Zre_46`yq1$p6T-hp81?F@J!DNe5y62 z8C)#zOm7x==CeiMnZ8=!xg6IBJkxUmpSJw06L_X?5O~hlMuBH~o4|9vHVHh_w+KAv zt6kukzFpwi?(Y&-8Ty&w74? zz%zZLz%!q10?+hK0)NWNbF;uReY?OjpF0Jf>AM7;%dtb?ncgMvtmnH0p6Po9p7WI# zc%~l^c+S^BfoJ+5f#-Z37I>x~5qQ=!eFD$)qXN%-_6t1IpB8xLb3ovkeq7*rp5%nU zGyNHXAGZCOg1|HVjKH%T1_hq!&j~!s;jF+j{hYwF9EJp*=@$f^<#18pnSM#&na>e{ zXZmGkQ^mc(i zXyw0M;F-Ql;MLlct~eb6&-5;V=X`YwJk$3IJm)Jf@Jv4-@SLxM0?+hbfzMg~4+%We z4+}i=c|_ou-Y4+P=TU)YdcVMDEk92SJkyU0Jlp*f0?+hDDX@Tc_oe_AZ z4+=cz>p6jE`dNYJe4P_`re6^FgSLKN6nLf&3q1GhE(tu-M+Bbpby?trbYqEuHb4uLgl<`doqUxA~nX@JydC@T`Zk0?+it0?&H5N#L2@BJeDS)dJ7- zH3HB5x}3l>eVxEFpBn_8=^F)pi>=RmE{gl-Oy45#%xAlZpXu8Lp38Bkz%zZ9z^ASL z?+|#VcL_Y_t6SiizE|KmUwMIN`V#`r`RWyTrXLb`w)=+#p6N#fp84z(c%~l}c;>TT z;F*40;F-@80?+hk1fKaU2t3nI2|V+8THu*}M&OyxL4jxbIe}+BhXkJK7X+U9yeRNY ze^KC>&tZXQ`Xzy9K1T$e>3sf(^;O=suT+z--AeEa zd>#^brXLn~F2^GR&-6ZlXFY#Z;F;bp@SLxw1)k{x0?+w6F7Qk*2t4QOl)y9nw7|2T zIV12)9~5}z^ErWM`dNWzKFh^ZU#kV4 z>1za@^OX~LrmqutZbusgp6MF}{+#8fP2icnS>RdEZxMK=w+lS;xn1CyzEj|t&s_q~ z^bUbf{iGQ$cL_Yx_X<4g`Mkh0{eZx8xgQjGravL@T<*OB&-5b#&vNJ!c%~l}c-Hg% z0?+iP1)ljF5O}5^7x=kWo+kvJ>8Avq`8+M~Og|&=T#kbR&-CX6p7s1$foJ+Tf#-Y; z2|Uv;2t4QOqQEo#lE8DmMg*Semj#~nj9TLBH>Rg7o%I{@nHG4aHwZlQnGtxV&lh;+ zGb`{+Uo7y|`{iwh%w*2=CJktjRp7s23foJ*&f#-66M&OxV5O^;4 zQv%QQL4jvEJSXr>KP&L8=g$c|(}x6}`Me2gm z^&6LC%F?;N$Mm$ovz~7dc&29rp7S+V;F&&8;5lFO1)k|m0?+wcD)3Bi7I@Y(EdtN< z)dJ6at`T^q=LDYlTqp2MZxeXtbCbX`eY3zbpIZc;>Fol~d~O$brtcJZ=5v?8GrddT zna^&4XZl`&XFl@+&-4QV&wL&fc&0xg@XTkgz%%`bz;i#pPvDt;RNy&Z{Q}SQrv;wd z(SX1+{kXty=Kh|*Grb`2tmjV&Jkw7LJo9-*;F&%s@XY6P0?+ib0>9nze@@_;enH}S zy+h(H{Y8Q2avv6Wre6|xF82|EXL^bbPU2yG$Z|+qI_pEGHwZlI`Ha9beXhVWpYsHs z>GK7?!^$%&@JwGS@XTkkz%#u?;JF-E3p~@;2t4cgoWL`EoxpRxHV8b^HwrxGi|(GL zhvm=oEdtN^Y8QB>Zx?vhGdl&I>AM7;`RovQrgsTE^Vu!%Og|v-%;!OYXZjNY&wTa@ zJkt*eJo9;2;F*3z;F-@pfoJ;D0?&L72t3n|3q13ALg1PHjKDLW1%YS!DS>A`PYXQL z2L+z>{Br`&^s@rb`8p@?Odk?>Zbugcp6M3_zKi>N0?+hI0?&GWMBtfzS>TyZ^&Y={ zF+FAJtPh#bw7@gHLE!V2|BS#heV)K`KYzZ!Gd(NtT<(hnp6N{j&*i>U;F-Q!;8_l9 z1fJ=5xEiGkvGPb2;u3c&2vD=F7dRpL_&jx{K`doo8*z%eu@JydC z@SLx#z%zZZz;ioo5_qOB75L+ppJst)`f7piu>J5g0?+iEz%!rg1fJ;|1fKcaDDX^g z6L{uxlfW~5i@-CV?E=sA?E=qy?i6^Y?-F?CvqRvS-X-wQS^0MhJk#?6&vx*Dz%%^` zfgiTz)hqB!KP2#+ufqb*^dkb#`RWsRrXLk}&R4&{Gkrkd4_f|@3p~?L2t2pbX9S+< z1%c;wdP?A#J}B@khvx*I>1PF=?fyA|XZn!9GoKd(p6M3_{(`MPFA6-JVPYXQT{RV+&dPd+mUvmYX>GK4h^EF@KncgJuoUf$<&-7-2XS>iM z@JwGV@XY5LfoFP7;F-^L0?+g|foDE92|Uv`3q13=Mc|pBc|hQqeo)|<&nE<)>AeEad>#^brXLY_wa%DDX@O@x&j>uz2L+z_d`{q*epcY;S$Uol zc&1+vc(!X71)k|I3Otwlu)s6@lE8Dhj|e=|Q})6Z?jN!o(w5Hpkm(Hq&w8G|l|~Qa znLbzGna_Cw&-D2M-(=;P6?mpE6?o>eS>T!8BJf;}s|B9vYXqM4d`{q*zE0pdUmFCT z=^F)}^VKHsOy45#oUe9)XZm)5XFao1;F-Ql;F-@3foFP`z%!rS0?+gV0?&LN6nLgT zA@Iy+ufQ|?kiavahXtPLM+Bbv>=SsVKP~Xg=YYU7{kXt0pC<&K>CXr}^H~shrk@gc z=JT|`Gks9tbGCgwC-6)^EAX7Ja{|xwA%W+1bV1;meo^3;T7F&>c&1+xc-HeH0?+iz z0?&LZdqpMdE2gI`o%JE}nHG4aHwb*YY;F;be@LcXo z1)k}v1)k-wM&OyA6L{A1>ja+Z8w8&D+$iu&Zxi?~E6+^=&-8YQx93B*OT4A;6nHMj zT>{Va4uNMq-zD%&?-qE@*It2VdS2i;Uk3!9>Aezf^L0q#E&Z^-vz|F3@J#O$c;@q{ zz%#vH;F-^-1)k|AB;NA*jKo`dLExFsQv%QQ(*nw578?WO{?ZvmVF@Jk#e2 zJh!8H0?+jM0^e=<$qGEvn*^Tq{8E8udb7YYpDhB<^wk2-e6A6Arso8{-}1js;F-Qr z;91YN2|Uv`2|SnkW`Sq=7J=t-Zx?u`@0563Z+A(&rFRHC>-jE$XL`55GoO0}p6Pjk zKVjv0K;W5vNZ`2~4+}igj|e=MW1qk?{iwjRp6?fUravw4oUZ|aXZmr0=X{+Ic&48c zc+S^pfoJ*|foDB4DDX^wPT-l(vjWfba{|wN4hcNdUle%eb6DV+eo5e&&k=!V`elJ< zK9&8j3+p$gr!1ZOd(3BA;F&&G;F-^P0?+jM0?&MA1)k}P1)lkA5_qOB6?o>eS>Tzz zTHw_O%>2(9foFP7;5lFG1fJ;|1fJW`MuBH~o4{YN?R}HLGkuG|vz~7kc&2X`c;<7b zz%zZ9z%!p60?+g=fuCpP*)8x)&kH>3`2zya^n(J=<^F`gGrd>fx!ey4Jk$FGp5<^< z;F;bp@T}*b7I>x)2t4z7T;Q30Lg1UMJf9JGrk@sg=JSle|No`kF-)UrVux{GLJ}*n zbXGznlu&Sr3ktIG%y@S^7n1mHy|Y}-d$DePCn6!)%-D0F^$Zq#d-0wqNJK$FK|w(x zB2iGdprD|jAQA-y1%(R=3JRo9P*6}%xS)uQ-)Ay@c6CuMtK7}bvp<;s_kH+tSJpD?Z>$;Yw-^g=axAOFkJlFLgPv;Lw;C$np8S4GP1t0u;9X%d8 zo}CGKdQzU9!}9bod3L7c>ErV3oRFud<=M#}^5pYBE6>hJy?^?YJUa{W^rAdFOY-z- zdG^o9(<}1qtjg2pBlrMxm zTk`ZBc|MQq%F|o&?A(*5@5{6EK%Ram-{d@$r=Q64dE``{ekRYyRa>5ZF3-o+g*^RQ zp8IekPrsGt^T?e%{a&7(9eMhL{A#$LvHl;3^8VK^=!wud580WNrw_~X97p8oqw;(n z8Iz}{tPs!8A<=Hx*~acQ=W5XS)RTk&(2kO`kFjD*X8M3 z^6cD}r|-zKb61|;l4s|hJbhoDod@#tBYAcn%hONf*?B5YKa*!?Tb_O{&&~^Z`n7!W zW&h`wZsh5=@?6)QJpEpt*HK5F{vh88ulHE@&z=47fB(|sp>rN`&L`yQNqKe-%hN~X z**PjtACn&s_cJ9=pOELAPs`IY@;vvfJbhB0=bn?N7v#AQMR|Hjo^yU$o<1Yb&Wb#} zDxVGab55Qf$+NR5PhXbjIj+dlSLHe9*W~Hz@?6)3JbhE1>)Mj1@5*ysEqVH$Jm<{5 zJpDkPorm)DBYAcn%hS)~+1Zw-pUboJLY{sp&(14(`n5beZ{+Fs^6c!$(;wv784G_< zCSSkNheGFkV`p5Ro{(o}Ql35{UktCWQF;29JlBBm^7NAYT-ZM?Pp`;x&R6B>bMid*x;(uh&vTFD z=_~TwhgEs{nmp(Hx;%YDo}HWW^ey>jxS!kd^p-q3_vGpO@;t`_dHSI|=lqd8{aBvs zI+3TJ%5z<3^7IRNuIo~sekIR2b1hH5k!R#xI8;2`8*Q)VgH=uJfshW&iToGi_6m!^4zziJbhH2>l%}% zr{p=u#^vb~^6X5@(=+ny%*xaA^6Z?Frx)beS(K-jkbPoI%zXGNY~muF`~o*v1w zvnfwsmS^XRJbhK3oon*+b$MQQ8}jr``Qz~Ko86M9@5pn`@5<9#^4zyQdHTLQuhRp0 z`l0+`IIoW6=_m4>^QZFkGkKm%Tb_O{&vUtur(er+A8zF7xAL6xck=Xmd3JW>=@0TJ z;eN*cX7D_uCqm~uWM@*IJ}l339FeDw%5x5n$iBJbh1|b9i5#ejtApet!N?o_=f}UcV>yp`Xfg-_GReZFyd&=koLm z`P1;@Udq$2<@r2vBTv7TXXl+f{a&7(9eMhLe24RJa_~H)$3y2l#&Z0cMB+ql4mZ#6i z^LeBqPp`^zU32pEhCJ65$wV*qS_jU7npA^7Ku4c5cbjcjejHlBe&< zvvXgbejv}zLwWj1XonY|GQn<=J^5PrsCB=aoGDTArOZ^7MQ8vGDrp$kQL> zxvtpX4xVrHq0l+scpb&%=?VEn*prl}kH~Y*kIK`>FMS1$PJojNno?emXoUh8$=j7R0m!~)6r^5Y=|B$l zugmisH{|J?@|^Qq^7L(au4_l0zAMjlwdCms@?6)UJpD+XbLLo{ej?A#Q+fKCJUiR+ z^h|xn>>QD& zkIJ)iOrAb2UkR_T33+;2p6klU)3frtjwa>lIr&1^lb5F#Yy^7N8CJE!I8GxF@L z$kVIx%VGbVJiQ^$IUmW>oANyOWqJCFJkNbqp1vW^eb|(zZ^?7cZ_Cqn38y+GxzfJ zjyyXb_Pc>`cki$K}~MAy3cBvvX3Oo|9*1 zUY5=?Kc)d5}=_~S_^Q-do zHFFe_RI>LrLeN#RiK3{Cf(|6>L!|Qifp5Bt@ zzU|4=_vLw=9>~)V<@ds#BYFCXJfBBS<>_bg>}<=^&*j;9Ay2=QKj%D@r{Bo)dE{1} zekaez)xA8uBhSaxgFHR{quAqLoQK?pMCg2dNKeZ1d1P3gJ|fS~QF;29{B^jWDS3KY zo}C$adRCt2I4Muh$@6(6FHfJ6=ei2=^rAf1Rg$MyEz9-MleR=wkJUfr&=_m5+Je8-P$+NR9Pd}Gu=Y>4|TArOZ^7LDIcHYU; z@8#Lqk*7b%vorQ-|J>tzqsK$%>%B_&{YVLUdQzV28kVPz$n!cHm8Xx%kB8TLN}fI; z&pDr#r)T8ZnU$wc%Cj>kPtVI|!u~0FdQqNpz9dhdmgl+8$kQwGJol8J8s*O@%MEzdb~E>FLZXXm9n{Ysvl*Yfl`d3N5*(>wC)e2}Nd@`GRRv2!SN&O>@! zo}CGK`iML`N9E~b^6X5>)5qo6IU!F^%d;~hPtVHp^GK8O^qf4uPc<)3FUS|e>$fOR zFUfP?rse4~^1Mzf^7N{FKJ1y3r#IyJJQB&%oAT^jmZz`CvvXCRz9wJgJd~$z+Vk^) z_Mvaf^KrE!Pv4d29Lm#=&j!Cfq$fh>>qD+9DNi4k=Nuc6r;p0Bb4;F| zl4s|*JUt`N&a6CrQl6bTd3s)+om2Akf;>Bm^7I*bc2?x+Re5&K$}<%>BYAc< z<>@Q(E8+FEDoj`@_#%G2-UIp^=?=^c4?KFHH!F9*Lq zY=!$d6gpoY(v$M+9G0h#$nzXW<>_PcobxGp`nWvTH6c$=%X3{Bd3sKs>&na1r{ptvvls{xIynm#06-bI!*;AN+ccJ`_4%@A2H@^7Mo}&pjzmAC>1mjLFke z@|^SI^7ILLcBbX&8TpfNKeO`mygWOnzb3N zNAg@(Q=Yyo&pERqPhXX1=bAiyU7npA^7L(acJ9d2cjejHlBe&1XonY|GQn<=J^5PrsCB=aoGDM*b?izHa5|ck*1=_E3ulMK)dCvKyJbhT6og?z}QF(Tb$#Z@cpJmOS@uPoBOn&wV?Prytt?sQ_b13=f1V& z>F4sClNa*zOL^|Yl|21gp8IekPrsGtKHSOE@8yf(K6K>iLq8cj54jKV&^Zt333={A zQl366&wUt?r;o~WAI9YADS7V0xIBGAJ{f-AJ1tM2l;=L=?Z3X}#wkl6wlTU;_>+@v8+lT!J_Msoj^Bj-l>BsWL@P1C@>1XnMozs@5 zpUd+cFXZW$@;t{YdHS_{ChWhFr{Br*bo;@_an!7&I5V+p*%Z}^zaDpUThtxPNY+$3u>W439{cIwIm6DO(Ea+2%d;~fPfyCTb6B1}B0n6yP8gM^kIS=vLY|(M zXJbH+Bw%&iO`<%ST~@!_t#JQu<>?)Hc0S0{V?P@_uh=;hI_DKVF3-+{JUuDT|NrT*Jbg@_ohfb=XOH`jk973-a`$JUdJB^l5o^&dAd%^0~0TDo=07von&XH|5#6 zEKgsNXXmOseNCR8JAS zJd>xl<=J^IPrs1o*Y_{w={NH1yp^Zl$+Pobp5BpX=Yu>w_K$<-8$Ulk6guY{Juc7B z&nM*RN%>0n`R!qOdin$Z{qD4Qe_kx}f&Z{)L;s8J&-15({qURbb^Gsi1Min#I~WQb z8~#|i7(E^S!Cd&3-tN7#?q55P{~dep+kesz$Lh&MKXlaD6Z@*R_^wu8D!-b4H@o;v z?4!k6efFa_^Y7-%vx|$f-^9wVzpl;}X1{*-dhVx>^s{Cx)eJ7GHKSO$OQKqN{$14l z=d<*}+Q-(;U;o;-Q=Y&Mg5e3xB6AH_<){Nm-y zpM18sxUl$TE&BQF{8Fv@a&h79r(Z^k#o5JZDc1Yp%hmbWdhK;oo?Tk1Ek@;~uV25e zEiJ`9de!}}-}WwIpLUn!zbyQ`RPM9)Kgg%ucl-Fm{!a6wdUr|r?X&XRg<82(d$+W( z_^Y?Iw+m06W}l^>KkfbJMdsP_C)rHzZ=Vd1kJB$+WL{)4gY{|llP8(X^G~vw?puBQ z;%WEq2Eg-=GcTTJ^uE6JJ_g=o;Gc4v-u?RXTGYL{wb{4f(ayhG>^}Hz?LXewNADJ* z+DD)N>=%E%6wSW=s;^#lpMUq>{V0kS=U;X2X|MKu_YDVXbv9r7_5ADFK<|H4)#l1? z7H8kqVuivlzsUC==@$#tuj{qJr`-FeM!A4KkGitZyL4NkMAJ7qgS&_wQ_Z--u(yj?|$`* z?(vf?M{j4#Zx$Bc&PMOwPw&Qb?{qi)_@7Fjm7kVhEi}95$=lvLEQ{-DcxKmMPF@AJ({_;LN? z@qNDC?_PZG?Zc-p-aNJu-hMv({sI00Z^!rlv->x_P2akQ|JL7c&wk$i&%1x~{onHT z_WpkF{eRc}tNcs7*8l%i^}nkAzp7iezg)IrnXYL}6^*^jVl@S2EVDOu@8P!| zmI_KY>!}&A)%`5nZTsK!LfdDiGj@2$mfiO@UgvGuv;Pg(3wZq+_h>p%(bDzW&L4z+ zJC0rbaJ?S-pvo4yW@Cr{uZ{H`>tKgLd!#3W6@D+<)Bj-!d-In%J3KwLr{kDPvbu=2uz!R_xoJ-Fh`(}UmAJy${d3|@!xPQ4KB zJ9#4fnDaZ~69-O)N9LXi_su;W-oNTh_|pC};gS9Cg`b-HZusHW3*i%|-wmhCcqBBk z{lMT;Pd`1FI(Km>ujM=8i*rtd_r3R%@Keq=!p$cyh5J^#2fD&g=-|P@Z6^;5?wfle zTzTeXxVi02xbMIl;gN?gg)eTu1e!&m3g^MWi*xr5?puX^_J1dw*NX8^zY(sua4Ed) zJsR(iL!r4CZ`DEcb22=;^@Z>V@Rb)%hx1Nd0Pob#AB9Ht9~^vY?$d*z(-{x(7hVh` z@6P^ya$-+S@+;XvvZKew3LUVKruWp;!>1-`o~!3(ZuH-H{`i4S zo2`73{7?Db0DSLd_~t#|yR&_-=LWuaYz*H!X7N4pN#%b77XMTJdLtjaw)2OfNXg_k zhac1rU;LNit8P0>d*_$MS4Unf>;Kgn)}M;`ac(BNI{ZWZ>TB@pp=@^bT*;)vLk-&D zbAO{>8{W_QX&y0c;L+v4N1kQ(c{@X&Ro(wZrfHFa7E}Z?nV0f1MB-{x&=OP6O7|39NtQN0R|d3=KcX4*&W= z(Eg94(7E-yAEmhlo)2H!`HE>SiS=E`^A7kYS%$$YjrlIg1Q{S%WaP!95Yu|%%H?35t9ckdS!4aQZhTzz!w|uK zgP1jD>;q4vtENO;(f;GA(fTs>$8MHB*At@giGI4~>E~ml#2@2Y+O#VlS83eGT$!G$ zDRVrb)I~)h=NF1YWE-*x*@^ml8+ttt{kaf71IeEL?dbnoxUR;l68Am09>Db^UjK;u zGr0acjSqXh{cJ+W2D>W%vyEq0--iBOGxU%P_IU?>OLqDdY(O@@2%Dc<$ohw1KeB`J z>G#*L{%{HFXZpaSWK(7T%io5r;LGpAkKcpM{vGy<`2#zA`4H@Cuw&CSp%3!YkE@D8 z*OH2_kp1rI!>o^^a31(Ef6?2m)~nyG{dZH(E+Vab!l=45ug{au z1?n2S9i4R&+v#iTXb<>11I-H;Y;Et<1!8nGc>})2yGM|KufrF391_7w0bk2q;cbw^dshh3 z=&SGATDK+OZS@K3SOMO7CJ3fk11${!ufK^;I)9+Ov%Rz1kLHAHZ|msvwsq#;1@7D6 z^|$&04R?i^VHz%r)s`G6mria#Bj7Hup%TaJvmAQYOkrhPF3|N1EQ5-1EmrM1L>?hC zksDv`r6KpykUK3cjN|(E(vZ(t_HJ6X$U0WQ&8H!E{ys)6(rd86n%BN%ONXyBpVig1 zJzf{ZtSnnuCopyS*s3*qJ9zYt;;AL_m%Az_yHRAy1JZ`wvb8Mb@I)(^{TYXk9|hll z&AN^6*v+UXIUFa2mhwE{8Ln7AJvoA)9;{5qtw+*>jF+8&yN{(s;S$WgA?el7ivGKR z+i-qiEfJJl38Z@};CmNc(^7Z$^vq(8$nKt54t~oR!RJ^zk#)th{Tl#?w)V4h2trl7 zj^IW0RqHBBD~jAD*#F}kMHD(DA=1ToZ6T1qGYDTS>1y=4>2#sF#oO83-sbLXZ*S?C zkPbh;aTTbkQ^?zL4_E7sJl zE2~*kQMIbBY{S~JRi$O6Y;Ki%p?j`3(9q=WXvKlRBVIR#CSK`$1)qOdc-_wS0&bu0 zVB`t;be}Hnam(x(vHP1j&apM6GW!-g?1HwB*hSy>(dY~wh?Akq>C7(ccCz&4qG;Vl zqZ{fvbDQdPR%OIfpH#P@Tig8+rmw&;Q=Lt-=cRI+l3l6TXJLF8#gzK6LqlO)Hg7o$PSHZbu6i89126hUT-}pzXeMi7pn=4<@nK4zZcR1lw<> zyuO{+hqCR3LET1p#t0Ajf)gXzs3lB>wlUt!V3HoCKb6MP0}~S_QvZX8lGu5oXPiEX z=z-4-+9%Rkr|6&P`7{&s=pXv9kvRV94>H#Gs3$9&Pl{?>bXb~8%$$`Uq%0@b9rCq{W;T6^Vg$4^#9~O z79o4s9kAz66?uPIjxxBAuL*B_c`BKCI{h!sea!iB1%$8Leg%D?e?4GhPQsHC$iBfDeE7%! z_#aBMr;C2A(IX`dN^zy8ujMmjiWDmE0ChQw6pf6H& zvN!w#^EsE#SDMd1Ph-Vj*91BFFYx7G|EX*)>ZAKJ)5BkL$)4O_F&_EXW+IXEAtN|r z2E{+_k3XaSmz(z9rbm-}s_%WwVO}$bY6K(0J3h{~RKFPK)~2 zkW(uROihUTE7pS+l-)t(>r4R=HU7u ze><<~qxlow%GYDezvZx>C}RFu<00AiED^}{yD&I@JlUU*k9bj@oXcw}9`x?S(y!u| zpXAeH^5r-YFU4o2_i=vH^g@b1ej7|i>o?6GoySX^@c44WR~3IRfe!KWMOCL!2T<># z8(Dxq*v)j>^&}Qe7!Uc`X!#{=Y<-gX0rEG@pSi@3DyW~>0sB1x0ObSlABw*(kwa4Z z4@mxFvG@*sLLN(e*A4MKJ5D@D4ooJI`qcxOv3NZ+Ut1oqQ{*p@^7xMY2mXwY zj>(^Ul4yM`PgVj5dz$@^^lvl9XZwjA$TyUH^_1^8oWH#G;Y5+|ko{63>^uhK{!8|` zv46t9dA{h7`6bE2i^@L&HV0py|EJ2Il5KBcIaA|dJ#%8KAo6Lxo+aBT+Hn7XyoBfL zU51~LyoJ5M^8tkar0=pN^ZeMs^J9B%(2+drpCj2$@+Y*1{CqqR@Pq{FC}i!EjN$JtH^{bt9gpuMwYsx)Gm_Ev%6$G8PmV37}jgKY@Sjqsd7{`3L6bIaJBMus`zsOmasN56S*hV)lps zo9E|zPS5R0a{~G-3{KT)xn-okv;Upeuc?Xba)jqsL(|}2$(d3A8k&#&Ku#vaM*;mX z?Kd3Uf0Cp7ldPO5{RP@@IFh-&$bb0$#Lelmh#vUdV6r3XPsnf61MVz0dXEr2`O_{3 zUmr1E8qvd_GL8P08_aSW+ZXO~yJ^00dhU-l2dpFf5#v+)s6T#~#-0gtB#WGjq!E9G zy`UHElPKRoK5ZM=3tsv9mgYi!=8pCU`O^c_^eLc^xGIz_^AJ1(kP#Tf1yx(d>&syME=Fx zsFQxr&qp2U0(D68T%`S*BO#_=@{i;Mi+(d{sq)v0cmRH(4K+WjFuxN}hr*-tKQTRq zPXj&)btB$|QxDai@ad=<@fj1$_z9@T@Q|1LE7u3)rMe3o2>sYFpL55^$NA(E@#SKb zIi5X_{BajcCI5y$jbpD@kw2wU3xBEKF5<}-QKMp5FNRU&@n9O+FZC#@BA%1J6R{mJ z#`DBP>R-k4grJyBtOlbXf7sp4&;W(bN3j1o)QzQ96rJ|x*dL2SOxhp5&i$RV6Y&?z z^(=XjBp3Xp2lP}rF^Tk${SRr&qWOZr|0Kwrf${O-U%lW*JMrL0yc1qeV4NQUyoUL* z%YmZ;6dTry(-?o3u!q0?{c-FU6mMVU=X**&w=56#>wNtnf8*=(S(81j`I#vnf&C5y z7!nj|KkIqMW$@Q^!a?y6{Ed_QF;l)n`HY_gru>8Q_vB}ou#nK#?c_80pK3ni=iDp( z@_Z)hf3bXqpHG}mv)?d(L43Pz{_FS4*+V>uD=9{Mw_=UAUj>o@9ISg)h>Xpi+Mw%;1k%duPhE5p#=C(lQ+Xnp1J68%~75yVSVKH|W5 z)7{|r6XYXVV!@~QJSsoBbMq1MpZ_oOkvXRLHG2O|*#(N+e;)`kJ3)Bym*Fqk^UdN|2D-hLjv9p^~{2c>>sKkR2F z)8{dKeLj!R8+T5O?av?L`*VJN%Jc2l=YxK-VLyHdpEuYi^ZhvYk3XmVc;d1C7AP9AEdD_f*-M;MF%+lDewL$mY?AB15d?j_cU-%1NSs=PXnL22I%k{p>_b@a$r3_zxMkNe-!zP7yGJzyXkNL{%4ir z{;Kh#i+>$CRLExaJ=N=~b@$EAIo{XDSiimY*^&4EwJ!2Cj!Idb@Rxl#y?uQt?Cg(U z{{8PFZ`oN{?ds{r_0_crt81|vK}i3}EQ;#|^u|HL#i_-K^eRgA*tPOalLarJMhgX}F|M#;K#E3K0Ye9TgV-+kEmQ=gk)8c zVMr8%WcE)*GW%$cB1@P!lQdDzAl;2P8$AcjoZECO=W%te3(lgWq7$R-g|3Ur+fw4q zSfQt4w9%M#v}!;NdK`r(Zb%2Kb$#VSn0s#SLA|s&>(x9qU{x{lBVa}KdWxRoNcaVEVcdAOjd~{N;^CDQsGreN8%7?bO*AlS zJ24(-dTNdH3g7(NlC&$mjNxG?ywGqch!^;y4h;twUsiQ|qs3Hz5_QES(PL1Z>M)v! zKd93!U;2Ot8z%a5nJmOZ{d{X+N)KdGc z_7Shm&G3Gfuc%{XjWx}n<&u{2g@!G3NLQ`wZ z_NIBMCq?I_TQFd4A5Ys>n$LUPs53oI46^bck6z^gQpbLzrRKHETz?XHWfrk^*(_@p z|F*DU?K0E9Wof?zS+lV}kELF1sUusx{@P_+?S=1+72j?Y4ajk%mwcbu^=YW`fC{ai zMLpY4Cp^`;GPQB3$na?R#+2+UaN@fO<^P!1=1b*o-br(UO{ zPF<0%Y1AB5kL@#TI*n44y4GIgdOYRkMx$J0EZxM*RfyAD$pzeC*Pvq^p$w2%A&roZ zM`f{bay^sxTJk*V$7p>qf?;H49oY>M1O8sDhd$tArZxJrSR+4;F+aPg8ISwVjvK2z z$Eq7=*u5Ao){wDu_DS(Zo!^V$=2cw83Kbtl#~s>Z#y?53tnRvX^{04Y*vV!z8SIQq2E%1=Cc%YMj|4~=Rc-E;Cem=Ik-X!$|0(%)%ylYZi?d{%kl(W}iJt+=4S zqxYAbfAd-1=k09o^R8dhd@}CXM68cRy*;CLY#w+}EN1Uv#;B)e?VGy3suX)|#?QXB zEAT!#499L;l`%))pOtvTXIOM3YdJ?3K5oJfi0{XV?SHKQ(I?|v0z?10#{2J4ywUg0 zjQvZzLi0a!TdivqE}SpWd=llX9S1gXo?)cj@EJynRCCLIA{In(q1Y1)qurKMK1q|C>VpM&TB zR6}Yv{-`cL7xt(GPk8>%OUoGBT#mmj$Lo5}{~-mDE$ABW***mS1-+rza1s>#Y8~oI zyN?3S59T@5_22S%AN**@M6{H{NB1C})5YD|L@%Dxm0^NW%0uWmnM4dJP@!J1b zw)>Jrwbu6dV@S&xm*{sn{gEb$LpGrGj>b~#C=HQ4tbJ0uYB}MXsA4EqTJj6|nYdU| zCGGcR`xAz?bZgZ+`46@CSlDQkCU*lzg*=aVx3;e!B)*<0F0uEo$o9Dsq;za;Py4yO zaoQiZv{yY_+fNq?@p}}3yq=crKah1ApT28iZBKrWE=KV`Ag74i(4KBpdu7K%=$K$0 zLK@0B%M?@Nt9Hu%kE8L8+ABJ0PjuGyG|x!|7B^*#922D{M_j2r(e#LF^c{aP!22wT zLgdX7d8I*R;eY6As1x|R@H*m&`ZpEX-eUq|BX!`Msf3Z#;oHjKdMCgUbOdE#vhRF zmH(;nRXYX0AAor6--^?ori8UU{a6&Djvr<=>Ho59FWHY?tomPr_WwDK{HpzB*%EHb z?Qb2Qe1dekxn2@lE}3#bbye*Ne7o&<6qobUGF)`Ns(EW1k9bw{!rD&NJ<>sp?Knuz zCq<{OYJRD!YPYp$&AN`FvLaDRRu(U4>1tS5+_G`un%ZqeEo;5&8@o1cs3~Y@U0=Js zaCN)4sAOfyy5gpW)@7ZWHq^8<_}8p$so1fsVL=mMYibE!+Ol-@#toHC_3PK=Z`x3~ z!n?k3drM_i?V26s%UYH_R@>ySS<|w7LG5D|J1RCTsA<{Mx~!mnxoBJ5vb3dsLvg-u z6YqD$h6T&I>RXrQg9m^8g2lDlidSuG+)&f8al@)7R$(~iEbXT88rN607gy{ktzWn!9W)!yP&1shwpmTfDl>E5`$Fc5dliX9s^ zg75nBWluD87dAuQ7U=4U`jWzY--cp;eX9@lSyx%#w)%;cWP|1Fn!QD9Ru;Fd%5P|0 z*Hpi$roCjNx3<<_v$Un8sl9al)@pBY$;xHrs|q%imvwDyZRy%nj@QDKU7MCy`Wsu< zbwhs@s|(gW(YU;_W#jqRQ*@(j2JUUcY$p;{3v;TOL`sr2dhHE%o^YzD0%JMN1a>7JBm=8yD0Gx`yU0-Ui?G zeqv=U|7{ieRTcc&$?a4MZt!gKg^Ly~T9}`YA4|cns4Vlgbok_ZNiBX)#^-I#59ULt zojG^(kKc-u7wJ&kq5K>w<`U+z3M?wk5^c3CI7 zA1EBIr`5fMZYtpC6%N-@bzkaqJ*$7yPI&%@zY!&%POP*<*K!kds(b78d9<~&_tlYv2`wKSXMzbil(J(eVnGvMJrKY~aam(0IFCFW zpyC5n;yv$qE$z|+;B=xH_TRPYpE4&N;6X9Spi z8&{&w$2e3@i3@uZSAt)WaJz&n`6i%0R<6jq$?!c+Rp|FpJwSt>5KX?o)YyK`lsidsB+W* zH`1TD4*ZG%Zj?7y>fOrcED5*DJ0J~YmG?CXxAyZ>3Af7o3qwCf{y#Fnjq+y7`9tFx zV`Tny;725!_%W_#MP1+D&f{~e<9)4aZ}~IrE!h(#UeGnfpOlv28*xJE_6J-o(Ple{XpiW^5Qa1|d0EN~SEhAnWl^s4hu)xTQrXMsN| z3f?2_x7-3(>%HFsSL;0!X?I6meRG>P(9N(%X!Ne|cC|G$J;r}>vJ4Mo#;-f_-*}>* zYh*XO^PrCEn%kQ3GmrHh9khEV{dZg)`9=LV~ww~s1v_GS>M&^BdDN9Lld;oTk3nfxxr_G z3+?%uuW&Egh}M?YUt*G4~&w_?S*m8H$z7GFc>%J#;t77TxryFYG&mUeGrU1xh; zBWFwq1oi}*I*K7Zg1)=y>{iQ>At4#>EBA_VDS1au{*?yG#vi&ob23G!kz{j(n oL;BBUIXu~qTtT&0IrM%EY#p1Loa>->{P4JxpV*)$d0E^4KOznNKsL<8Gpz>*$$*4Fyf!nUB{fK(N1v8t zT85kmk!5Uy!Nlx_oQ<<&;_=clWSpJMIGW9w$P*vpaWI25*{n|{=Y+@%kpMHe<8>f0 z?*0CHbl-c`Ai$p3uoQIPTmS!8)&Hvc|Eg}?{_>LLOLa|Ss%Y$07ON>JW0}3Ndk??$ zuvAdGSx?P?t?p;pZrlH+7ur5Iow0*Mw(P!l@j7eEp7U?Gp2O?cxJT28ik7a|cKjgp z+i~pbhwJpv2UWJvH5)tle{HPqNC!Iz+Cx1Vtmu2$p8gL@+1tO=*}>_lJvC=+llz8j z6REF5J?X6I|LC6nVYJgV+uL;*d*tM^gO%s@4Q{*i?BMcK&klY|_gn?-b9fz`ed48X z-|=JN$DQ8^AKQ04JTmV@xNqLc@ZOcD!WZ|R3Xkl)6nbyrnvs=CsK0o(Z_~}bO2|wd}E8KkiVz_VlCD0XxLi_g*Zauzl@acKS z!j-3vhnw3@h5Pos6&`u`V)*>Fi=bH;s&MWfJU?&G;L|J7&))BZXSZVflW&D9&Rq;| zy+q^vaVRto2ouP>n8pM*vp-!oXT=cRD+InaXlWG(wj@~`*S%s%&W zAbD5z_mdNQYLZ{e4w4-`Hdf?-jWoTdrXD^uN%LGiW1G}RxJ^1MirODIVgE_+FX025 zG*qz(cRWt*jxbjA8eloRjp*x#@RJX!N<$y6D}mp6u3p>m`cb3*#`DJy zY}y>Q~=@Uk_!ot7l3l9UN-V z4xafN{o3$e)=%?@X#yP{^>%aJ_e)Zi_)_?XscJ<#1S^v2-cJ+J`yP5*p zV$8MS-3k4}|HAr*AJh(B_+R?f3-7Xn!+)I+8vZsrc%cDn>IBw5@}tRsC5DC{WCwr! zAZY(bQs~UOU60XR1J4Jq?Rd>Jm&E!mXpRBQdS~otf`eBG* zzd_6zGxmWe(p6I;u4w;p)o6Vg`(rmtpX&+H_(VTl^YrsEQsR$sEp6JBkE=B9W3Eik z)s(rOQ0l_skn;;AA+inGgzQB9y$ijbh5lTKpMhjg|2FjhEnHXORf+o^T=(I69It=G z{V81koyLc~-hDnHWP@Fm|JlZ~tM5Ypt{HmB1^Zlp-;$kv1sjmf&%@?t3R(XU>_>J` zKK=e`)*miq{Y)QtjBKjxfBD<66@2+U`0*v!?B8L(m_M+Cmk+?M20J!Q6Z#-O{kW<) zbSOkZqEECan-l6L=RiU62Q?EqMYB+)X^E}^w#4}UAtRFonPxZn%WwJY<^pN zr*D4S6Z1Q_HMebUkJTHxWc`Wx(VK?$R=>BC&F}ODJLe10>KeK_Tf7||b>5biE#A&1 zU!bnR+tFDkv7NrQj`o1RGtgXEu%*@4>TPM>(w6^7dw$)2ECvJAIZ!U0+<->FU0_2cj@f59^sbr0%C=mf>ls)E72{g0+I5ILLS`a2 zzT8Vg?xi7jT3Q&#_3x!2pR?@Uv}}=etbm(OL+<>2j9R2uV}munee>oHUuPbxt806r zE{a*PWJR68)a7BT*6i)z(L0K#mdIc3s+{abk!d!NHtd$IWif{*TEXnkIDGsl_k{)Ed>;T+-BrOV;VD=43uZC9i-v!); z^9yT)^~`ecTgC`J*V>7!E1vD&06?_0pQb|)s^WDB zFRHIxTTxa~>@LOrALl5d&>;ztK7!X~0{J_G@Ws-uMz5Pr7n)nVoz3lS?#}l1mX5h@ z%xs^x!{^Q~oWF3cdo5k&7cFKbUCk|x?vCz`PG76LwX37k-RRrg+~#wysj6DOx^C@~ zn$;CmE9;i5U$bOo*^)9gugYEMp63lTG`TxkaUk%h*NvfxS2|z8=N}ecx3N8d+ZQ+( zc|soDr;B^sGJ8hs{$`GIY)z@mzS$1DpzWh}(f55cI)ew|Wax4_v&*`jEPc5+TDQ^Y zhPuw&raGNf8S&I7)otk3c7KHF%W=$9XVdJnQ@Ksau2k%^Fg}dpN`|d6a9au9rlvAK zx_7}5tMRhCKm&u}KY!V$2`{r{tzq^Q2P5z9vElv^3EzgJtya9Y0D(f|(Dh?^y(X_8 z;aWxr6!MGFvyHLd9a@C_5=b)8P#VkX7W(3oB*)2)ZBLBw1Gp4W)H~UHI;2wt^r-=s zN`w2WFWcD9vsv$6dxTvEfS7J{vRzLn+WEK}vQybHH~Ueykbyk~Jo@VmVj@WpjlZN@ zXPbx@y~if_xepbT8F-y_vV#G;9W7X7V1F7LTEKFHw)=W!C9{aWKZ(6@fXxgh*nTtR z&279slx;5x>NdhNMtH~RGdYQzufI?3m+ z4%>*|q~Ii2k=r*x;RSt|5#AMWP9lC&&qR{gU>Ym>D*8vA%ljtp&zXjrzaIUe|EHd2 z5weHf0ecQru?Kb#EfvUjF^zqtdwhhOgYdad=tm#FwzjsG%ll?FC5Bkpy zPM%EuHlFKW)Y<8mb)}#3tZzv3!k*|K_Hc41CwqXOq5zvXiH|Spy*m;j>;nME2jG7h z0b7JTGEyk;OS(07nHbQ7e=G~=X;fv}$DAKmK=``tSI`Ig*8?`@Bs?jB>>HfHhmQ<^ z|DiN{D)}=OO8+M42l$L&?hINt;a|LcZV=`euGJ=!e1qAn{OgB+rGoL#6a7Oou0$eQ zjDEJ0^cRuxE$PNGK@(g0vxy)0S5e@hhq%2oFtxwLg`H`8hYj&Y(Tbr@U#X z&xx>$vR;2ln-QElo5q_<{J}oEx*c=*e0yM8D*H<(n-QEdlkjsmegWIl&E}A56Cc2$ zGh7UMJLs{t5&cmo$Jpzs~0iw+EkpXixgGqkqoNJ2t&M>oB8rH4@p)?VkzzMhobR zRh{e&|G<3C<@1&1^Uu>*$=5YOPW}sg`PY9cn~VDB{>=37*IcqE_g9QZ{UIN zx@b@SXa`>U8~hvkoe|ZqvNth6_I`)-P4_5L-!z_G`R7n6^wTNuaz2mNzeLOS(Ekil zxAG6tKmSCE@EEWB>x)eN_3p@s>ASwoqHmht27UASn!)v*A@v9U&xq=~J{#+W%c5_W zE2i%%(w8eXzxn(?d#P{qpAm$%AV6fGSFFD@Kez#zQ5T541ki(|Re|tN1n}=-~8@n7=u= zKFHtBYWisYgtzkb81ru#>?ewtf7Wa^T4(%n6(fvtQPL%!}?Kd3B++O5Ae1GER^jSm?d~Pt=5%nkJx9I_QmK(iCh@Sju zr-QGL7%z?J;ZK=Hf6EPKxsC0McDmg(-#9(@N1FrI5&nqrseRNRKTKoKg*lQ%&P39P zzoK5yi}p#B?;xMH4eSB0e0@uEAwP3R`-A-Hfob{_&_`UD&%;%z>49wheyUH#1j6_j zFDDCiK0nYO?3+pM;-~old5eOXnbCM=<%jt1#dz1}ht@9ybRrP`ZT9yAG@p#~0qc*+ z-+?#H2jETq4!mJLKtH)bv;V{2r;&Ymz5{+Uf`<73c?lo&|BD*sQ}8boijObiONhw7 zm>YG{@7V>YBVC{lNuKkxe{&?n^h^GcoM6#!CM{L|dJzx6FSMcNXBFmm0_sqBbp9u% z$M9*uC!ub{yKw5E+7mtx{`66mm4C}=(syrS{Bm1QuMpeXf(sv@Z zBgS~1m`MGrc%Bdxvx(JU6yy)Px)~av(D?}VUkAFe)QY0h{v7*bafnI#!#BCVlXfEh z61kowFO=khzx05fN+%|f9{D^nL>j{kWLx49h zUv@fhRDfc`dT|ot?-cg%*S|lG{et4{>->CA>F1W^!G4{uALMU*eLiinr!_w_w-sLz;QL z#(X7wbiE!kU*YrHlCRKaoQm*AzMoL`hdpurm^m5Phzs-m)TfxQkUzxDSEi4Vuehh* zIA6))`HDM!z5;o&+!WuV`AX1`Z;Z}Y5KrA%(R^hX`upVhNEWTHJYJ$dOFn{lY05_& z7;m~8{C(?T{q_QUzA)B)dnXr1txr5Y(~%l|UX(-e{=^=RBSvKT6+e=lK)%18M9H&nB?S~?nBEP%!cxKvf znBy15H|;l|Z&N(uc=P_lwBG>U6yJt<{{0E#*=3WzSmPN!Pm0+e{HEhPX<)z95A28i z%w+mJhOf_O@p%V1_h3Bi zpJL}%SpP@w$M2td7PqOYO%;$2ZgYPj2x3DVF#aJO8onZ!v#w zV!u7b7~hJ56Lf|Tu*Ck5=11m)iLj8^e`jO=oi%|zpEK;gr_z4W5bvC{A&Kq3oz5}# z-#AdAay#*^>NfV@=~yknpxCdXy$ki|eiZrI1mk%V@EOMQDDwYI$~VRSlh23D=>C)M zzsa9aqVaC>WadVl^3`(-pih@>&?oGlu3PkJ+K-H#zy1N|`^5xv{)+sz{OMbZZ^(}f z-w#M*E9kI^7ehWvOM=`_=XbIQ#Y6sv^#`99^YdPDo?$o-KLVngI1fLH^Kel1B)*Sh zm8LYy{yI&^exH2}1l*so|GXi8_&gl`AkNpb=<`$ld8AGsm?u&l{I0OA5N$K>EjmJpB6j+&$~H!&F2Sc?2F(>>}Syd&VS0gKZ@li`24_> zpL_;&NOiB=)4)9q+|$544cybfr>+4yJV&Sfdhs+rR%= z<+#6U{OJ5&M-CLRS$)s+x@z5hb8?RM^)c3OuYG>x{eP{Ce2t@0RwevpUrujdUkW?@ z<5z$GyU06swxo8|^rQN!+JsfL*o`2h|6~@$^?dr`sHVv;E%G|5u?Ob>e}w=_P;Hz& zKguL8N82kru+`uTgG;4ixJ4RdItJf(Q}-M8mj@org=o-9YuO@r;STQ zd&&SbZ7%OmjrUrD9rcpbl0xg3}mRSP3A-qsFMFo*yQj_!w7S z10&QIy+k|C+c+z4XvSjr-6>W)r91i}XTcS=37;q7d#Wi&5PP*mupBM%@KMFO;p~Jd%G=z(BnId9F_sp7?l?YXJ|%#A_ere^X5N$_DZ1qw@F}Tcv5qIy4=HaR}Ly zis;#((HlruP=g*v;fWj4!75!}@et;on|n|%ZO(c%j}2H=jQj{#alM|R=Qt97 z$5#q%ly}kCYDKZG7%`r%r$N0o1(rsud2n?1Wo#~K1NR(ia}b_JBf$zI*67u2cO z>8Mjzq-z>AN7ZBd44Y1)6sNAS7rUNFxw+9O7a2=8@p2X7^j2~KH`q1kSVt%WBvwcx zq~lRpY@A%r~+DG@Cd=4f=R}fl$P^|QKTHU0dI4hr3o_O?Xb4M#K z=womgIG4cCzpnBAdlYZ< zy)$F~60gww&)in&T7?Ve3pAfZIc>**O`K;KX*Ya^(L&YSa_w0r0QkrT0-eE$hkTGx zby<}1W}g2O_t`!h-Rzkca$fxI3lm$}_vZZOx9e?xHFNItynmYZ;{$1_6)hDRz{G$R5@{sa>_4@J&=P6e}(Hh5Srh zEUA+A`?CEpLtDDF>Ye+0PHh z#NXGE{iMf>c4z)rTo*ndu35N{{3asQ{}ezPAH{^4xIqA^V`GuDQP3K+-yLWCugLbf zl9I-!Vo@LS$@U)0_(x@X<-gRPT-w@xAtvuLamN3~Mi8U8`8#9Q@s%G{qJ1ygdo1G* z$o9(r)cC5Mg5M87y!P+JX-`wa+Ma$a3Q@-oGn@2(S+Bing^>R@JWFUcR(t>EpFc{+iV-%L-~Auh?F(zM!UMW9!oV`emYRNz3Av`t>Dw zzKy)!GPt%D1eMH02xG*0!!MT)XwA z=nC@3qFcJLHQ2PVunPKUEhm{bZzSF2x0e-cS>-KRnZKcR%aX0dHQgK56$RpsS+RZn z2Jl^9zVyk4?xJSM+X7uZSzlU|=UZRmuW$9iK5HxM+g3fff^4vCZL_y{^@@_Vm3a-V zYn$pf*0h&y@YdG)YZkYZHno?n+fwZY8P**|H%!_3m;Sd zrml6ZEzNQpkEP&ORF-;MI(+iIv=+Z7q>&*HXXfd_nX~^o^WAyzLx=p2j+4pnqN6FLNFl_f7o;yQ~x4 z4-^j9lj`0=Hx=-+3WsZ%x-WCOp4Y!=Cp`bdUkRsvc`@K~4Z&!Z1yA&427L5ak(N7O z8Ak)WAXZwUYq<$J)xGulBHG&7`|3!-gq9DEGr@!$O4%_Kv7m{s9tdEgxU9AloJSrG zQ1O8(@t*g*hIfrdaZ0@BObdD*5l6MK8OBveK_BDC43|401;ka*8+YCU*^WzwVdtjg zqA*Et<^MuojC~7oDVXR%ywjJ;iQXfD%J0KaXgvBA*Ma{Y?c>qY{w5y$(d)o}066h; zn%jXfHLtYm@P8PI7tvQs6TB|vpfTf(`=*rFElr^2S2O6TpUV=i+7?K-+N74r{&QqM z)zV~2&wnTR$J3>FQS<7nlFwHpT+OG~Wj_PbfC|12@N_jOdRm1#6+Fe4!#BzLDFJ5R z#+B&vFbmEcz-+%Dltz6t10`3|8bO8g`XJWs->0Iu2+=!+75Kj6kK<4yzI zm?|EXa2gk2e9T_Oo8Xr&;<@`l5b}r$jx5{^>a|rIhX{u2E5N53e!SB(Dmt;>KYMT*Ze03tYv4VGCR>z3Ti^^{>|ZS>TU~ zg7--KEwjMYdhfTu)q2lF+TBrC-`wU6bTjM`8okTCU2P3bkMp0LT!IHOb@ zHL{!Cc~D1n&27#2naBE$j$3Qi*3~K)YW8)ou{Sw#>%f0ra%}1+e1Q)9G=vnjxjq27 z%YB_fT>Nke{f=e%%G!DOJ-LQ$f@&%BvD(*J+=<_xtncdd5meBlp$XdPE%QCm+~708 zh4y?+^5^cxW^psdqo1#+YonjXTfThlin3;Ji?5+`MSEjc3x>bR-5<9>OS`wRuCu+a zkuxR)0(*i_Q5CWnC0yDbPz1L+7RznC37dHXTRQ4G{Jw_KzbhJOZ}fJ0*?j&k98y?m zIV36LC_ys6p`$bZk(<`Wov|0bdE3VQz0}*;znj{8;?2C6jUDQjTkp`{brs9&Cw||J zAR<#OQ(sqPf|@1s4~3M=w5l@N=Mzk(1yQ(1-kp|lOd(XfQuUd*p_I#`0P^bJE^ZFX ze2hL@qj)F7aWn_}I{;~XV{T0?T{8H3sA4E;e06*I*K7Zg1)=y>{iQ>At4#>EJfJAdS1avS*?yG#vi)+NN;K@DxB0ev9atnTg=gau~7T=LI(@aaxcklSl@LS1j z^38qj;Wu`AhTrwDG&zRkgPtW-3_?BmXGJf%Q&x{Y>|LXYZhhIgyHN(RXJaF!oGxwjnXT`(gTc15M z-rDi(`04w9GJfG7Umd@A_!XpCJ$%mt_ntew;{J2PsQcoNA0BT-ofo%1JH7>VonAp{ zt{cAa@V)2mZF}I{(x03ezxS^n8NVm>$?<};XU8wx_u6>DPpGbIhnIq0fKTg=htd9j z9ADb@?D#DYKRZ7B%J0VSeSp%eAHHzk{pU_2%{_%*9WSW-@_6DS&x|KkJu}{lHeWpa zGSZX|6R);D^4I6?$$xmf;C|Fq_{{j${2!11XzbPTdryyzCobJEeBs8s&lQ~c>vI=w zL|uhn9=|Z-r{i1WpG13J8NayfWu&=&cuV|Wp`3fpUAXV`ctOe|f|N z=5i10YJiN*b_FKSduE>|TAiD%vE%_XArG@5Z$c*OTx`u#kO`NIXtPhh;P%}}`A#yn z=KF|S$n%J@9)sMxv2)Y#oBPTkhk?m=@A%;yYhJkC%=30{;2T>rI=>Lc4~49+*7)PXi<0IUJZ= z>Sbwfc;Zk0m&Xg;VW)4OA3yoc>jIN?bK~D_&5D1wsTfgp_9R6b2NXxHW zlc)ZvY@~IEXJq)NfytIfJR?4K*RIwtxF!QT$GoRLg*1i8^CW0PNPDzwq%Py=^(|S5 zdkk@RBd!N&-gA$f`WEo*z>UB!1Ai5`1NcVZe*^v^@Im0~fS&;VEbzyG7Xg10_(9;k zz_WnA4E$HXJAggFL%??eR|3Bt7-_j1_%`5;z%K%K1OFqQ2l3pE_(H^g7I+u%jlhe6 z?*m>7ya_l9_>OY4+cSdpOt!oYybjMd@mz)HIG!uv;Yxp1;TQ0X%<)X9&;V;(05cyTOl7qaVxgycN$MqhEXRYy^EF=s`T^P%~V5<)IDJn&Q~w^mzfIp~oS}TalbG}U1H0w~ z26nYR68Elu`yQXA{s%Z@kw zV8!v~FRVCzFtp@|xTCqMf_UHszyD z1!z+N+Ejox6<9o@wh*tVEyO?QI=5xKJ$Iw75Y^>7Q~vc>_a7-tzOYrQYkA1Rk+$;HX4cfE>ZCZjhEkT=> zXncARzLd%*Yb^JozFkzG_e}lA@cjUN_nfIei0=*b-F>FM1K)3%HE0`-i-f%r$!b7Iw0`<%RU)hf5MiZn@Swa$7k*L*9|r2fUN_AZ^R1(U*{S zq^>-4eIW3dx31FtZZMJkk9x-TT!8PG2*0s4R55acj9ZF4UW%i7Q05jqlM)AZ z*<=?R$xUR!4&N2eeZmVK_s^DMPQ1YdId#{Z*}~YQKaQQLckevok8|O<15fnHy&X@_ z89n{BGh4i7`Wt2Xtup8}XO&GZgCBJPbG=311$w5WFOEPz2zs`p7lzYIUAbqeysbHG z(kgR)+2mF~@;K{Si1@q+@q1B6K9A>RS><#iez~lBZFt>>?*_d{(l0ggP ze}eMQ68V1>fqrY*WYS%rJLM*Xcvoh4`~#FeLj2!S`q|;>KTqi+#Gj`0S>fp)q4W{r zze(w@3QzxSN*^Kqdz3yqJpB(SeT4WYDE*x9^iNaz2=UKS`nloh&r$ja@&8Wg=Y^+# znbJpyzewqG!qdM-=_ACCQ~ImJ)4xgSBgDT=>F0;1e{a)B(p@}Wn-`KGAGF@9bLSn; zcjbP^h4miUqQ_v@-q=|QzjibH+lt}04{d_qTR!~m9c9B87CqUfuGMHiVd|&{zOfH! z5|K6ud6JPg1wQeN;S2En#^9s=_D(kP@^x(Tb@-rfB$ti+7ktt;j|L`x(+NM}Q*py@ z!B?I5LttbQ{_B4vCoO#&{_H!y-!c5|7t2P*mLx8{@YBHL82q>Qei4}bj~4@zFZ{S{ zYrkh|JhhJGJYTB zXOkD<>-}z4`N(^}V3V)F4}5h|`N+Fpbd9_QpYivb%17S$ed_RkByU~%;-@wbzx^ab zI|FEcU~=*(%Dgi$^7f&v!xMjCBky!>0biWfVLY?lBSJGT)hkN8=f@^a8K5C z4PUs*H97WEHhE#umZcXyg)}e1Cpzk(y&k?Vfr%|vs*uh zNH=!KJ>tvT%G);9DClD=C2b$kZnX1P$^1)@AF|H#ucI{EtTY=?{&iID5{lnGd=YXw z7jbhCSBmF4JbigPhO_W}4L-iSox_=UF5%y6hU4+PhJU|l*o)^P{(aZ58&Aj#((fK7 z`67&p;aSMP?;W0j=W_mi-*7UX#K)G!wbakarTCox@x0*|{%Y>< zOApQ&e)*y7;Wz&Ns^ODIKQ{4&g#M*pPW$-qmpmVt_=iowvtN1qBWK^6=N|dpKjsd< z`cLzRfB(-{5C7-mIm2)MJZtzZ==B8jbaJs&{OFB*b>Ye$#||J z{Vkgudx-S6d}M49Fy^DN2Z5`AX8~i4JN8$=`+z;bSObsU3ET+$I>Q*@`8zzZRvzm? z{B6L$0KNhEFoP`DoE`fygPh{|IG#sqCdPt@-*nU2v40|b)5O?5!n@9neUb34iLsr8 zcb~<)0=#=-tcviSvtvgI@0l380oZf%#2DuHQ;*+#7WIsr^6Z@$TLpX+&y_^qcXn(y z;y2-WBjwvSF}4Wtn`+M@A8_r&*gV4f&yFo4ynkYB7U2VD5f6M|Vl0`;`HP9M1Uw)A zi?d@Mu?}uIC*}p{I`$PeL-#9SgEnD4D95}|hWYskZBF`dzW8vy_;9}XaK4B&U%)@& zp$`AT58@yCr{>k0zwKh_%>Q#vW^U$k(E-Y-e-oJgt;WZP#E0<=;FEyQOqNc?goVS+ z-5t$sJ*)8T<9%_CC6AO- zCm%pH;?H0V&F<(j%VYm+RM;F_-l1p%ERBjf7n}KdgdZV2k?9{Fx*;FBA^+Xn!f?^g z3kKnX!j-RY3l;@|8vB~+gFWrVMMki<@lbO|(VF0)zWVN_?)uiYU@vT4u(7i(*w75` zHQ3Y)``6jn+!TbX9qexI?Zc9wDcEo%$m5w>c=6>G-iL;u$ogKCgNpjvJ9?azeds#< zKeyu0rdo1roe`JsI{o)wdzpr1cYAYtVduevJbrYeds{m@xL0D8L8Zel`?K;jr?a?ALG)l3z!}!__;_bD`zVi}5f=Y7 zHF9~*dlk{$_Ei};Lmr9E9D zNfw&E0AB+FS|zSTy#HjH6)(xGhNrjV{gTit%P|X}y%q}hAl`$0A7_|C;QKZp(02;x zPd`1=QtlcU$Y;K(y9V-o{HsW_+*ydkCeOSZ5FzqjaV_#`)@zOxMN|17bbZgu`kXRA(b)0A<#NWJ!)Dozy;~}_ zl=(Mxc6ULRM4-(x+DBW9&q0dn>NTM6n_y@CwCmK`R!_?9@9pet>sju{?AeSxB7gC! z!qv>l#ZLb!|BCwV#uk51d$71@UA-SwQ(I}@4QCLJwjO48Bi_5x$8diN+{X8r zJ_ga@fxerXN8c$81#gD^{c^Vdya&CIMdVxO74=<3En=*Mz2RY(W-_nD#q5%Xi>Sqy z7FhI^?7W5x+4CA+!JgCba`u#lm$6X|7qGJ$UdkTTa6UVw;U(;(h8MHXX!sg7sNpwQPQ&wAt%k2=H5$%g)f%410vevnih#>3{?1{34QI1- z4PQl5ijqEyU2>~)ghV*khAThLq<6{y-dpI66^JfT5{K1_$mOe{h;s;@WrohyHj%N!@{pC1I zVCo;oP&ncb^@rnZfvNo*&k>l~OSoo{z|>yc8Du9VW)jyd5pId`xLSVE0=RPLzF|;Qh{CTuaU=#_Q#Qbn5sQiFzZ=J-| z5}%Z~M&k1l*GlYf75VGT{1UgA`6cc$^Gj?6Y9iu#dXVo^WxOKjHfDV@cunDM4BS|OsXcnPgjR5QI9e=ikq1l|N{BD~@S6sbWyUbBvs+IOLy^1xCM+%T5@{Q^fL=ESHZxwUecEvm6?r zu9|TV%JZHu5|R_BzIgNi<);`dhj`jm10V9=(pc4;lRy*(JnUi`d-yir6obC8B&1)M z$~GT?98m^0=*Yj);KMJ{13zS>`v!n``(MgsBS$!=5aUICF3{ohqJf`V z(0#rElP>aOLZSfrYqsx=y&;1aFQUMYP=5xKGKKP2Vmx3>XzVDwMEvyu zrL(F30KcC~1OF0z$sn2hIRQHH)5K5oP(~Tnrn0a1O;z|v(Otx!Oo3?tIra)nim&2o z2d0TYrPnwxO$;i%)`4lFQ0a9JOcRGnZ*gFnNK|^41JlH!(hUb5aNuJOJScIVmpyi5 zDuWTGQNZ|QL!PVv`6)9o$=Ag@S=G{BNfzVlls7H^^N`2hb^5%5A;Vp4_B>DBt>GtsKkWb$J zJX}ig6P-PfK=OV6k*O&+{3Vc+*_exC5Bwh!A3K7ipXqW#h<)rO7ppe;FZJKa@8Vv; zD#0)4mxG_%Q2zkx=iz1IU!2Xqb7|~YBI?6N1RCvQFS{_m#7zyr-tqopzSPDO;{pAw zpyh-)AKq`UI2xZW|D|MhZY8Vg6Z(l-lWD$s(rQ0SMEILoi;Z2LWCenEfuHm^P3f*&W0$~1Mp z4)t&6`yt51`xok;=fHC`JZQvMBBjN@L1SAb8Df*4gT~xS%JKb*dj(`|X6{NN|)kCI=NpCJSMB>hqHtMWEvfS;s4GC#&Yh3gB)f*+V( zB|k*_=UZ6Fqu;_p9_{`Ld9?eR?C776f4e`Te7pUke0w|+^!Xg)rTb%Po{)jXdpR`- z;f1eazQ^_}+yhhpLk1S_7CngknL=Ns{D%6oG<>`}Ihot@d@k>>Pt~GqUOwtKw&2?( z-bI=}Lj6_R{1fU=GADelAA`n>3ZK%Rhm5V>0hD01uRoD(21bzSQ!BZ?S^j$p?3bE8 z%~4;r*8V{wy@Hk7{U0=DB||-x`gHvzenK26@cKi>9ED$LJiWo?#+`nRTA^?PW1usy8yKf8ie*zKYD z5*nh^zr@kM{0iEBHOn6~mhnr&CJyy4%IPK9qrv%NewXb4;mXFKWh7yYx)@KFLdA{2Tpb18?^rE z^Vd$yH~M@vXap+Q4z+)ql4;MyY+tCqj9t&WXUbo>)}Ir`tSOj#YkdlNS}rz%MS2?F z9E-mm@OMQ;zQ*6>n4j$ayZe1?E7g~V{R7O`G(LVSe;mncA>T-}^tG^(+jn#P`&VlH zr~31({0Y3gdC?zPZQP`ef_?hsda3S)*tY% zRQV(OQjLT>VE&kLg$y{~mVS^uqS2Ot+asSYPoz(pz2fRd8RUHAv|n2s_UOiBvhTJ$ zZPn!GAtTE>VA7xF_5#h-`B6blBzuAN8u*!2ZDQ=#aDG%vetanv{HRv=LGte4hqM3S z5BEQKe^!DY5}`_ApPl@e34&_BQ`t63%kz_e>9ns~H2-eUxG|ZG{}u5=u}>sVDbz%j zA0{^WQKRxB^NRTKbT{{|Pyv|C_p7)!1&qM=vz+`OZ(ikxZZDnogX}e}N9_LV_WB_s z)jMF)u|EfW@~QTT`VS9Q^dI{Lz^U*^1vdL%tM>mTACB250rVrbJUrb!pL7v{`cL|y z_WuTqZ>;|7_CM6G+wVbRQ8IDZ?!Ru&Ble#h9%X*OibU%_Rup>w>(u^#;|lw~nD6iW z@%`8R1E>8b|A72;yZ^dB5V8N{Kq>vtGy5yYKi|aW__wJ2|HKvczmVKXgg*>m5&KUDQ|bS5v%j+cMJ6`;-=+3{&lUE6E#J@kK?62&9sfo@08*2aaV)P&T@#Sa`1&q(r-5dFKPt^V!&iPLFw+4;m zWLodq{Y%vRC#`?d-Ug^{8ejMqnC~(yf8ZtIZ)`C8hy6^N?|ddk|F>KFjkJH4%gcG>Cq_)vqTKH|J63Ck{%^*0YX7J|eycyUo`Zhi1Xu8b{Hu9d|ET;tD?P36bJIw^MfwMUA>SV} z^TXfjpKrGx<9EWCo9nDU-ztyVi~SS7nSUknFAA%Fxs{&UyCjYHVYYYBfPB}P`KkWJ zcKw)tPZ$fAIP2#Q0_P|72lAa~=6?|Rmxa||WYtgYU72S4|0{tZ-wkGds=v^#pZC9D zrL+FkR(aH)gf!cq+zAZ%K4_Pxk3Y3{RhsR86aOLKO?LWFe~Dc`%=A0KZ+(8me5&a)^~aVsY9HinwVD4xl8-R`u;q>TwIt2< zcZgq*w>C4s=#Q$OT%I(cXhn*bL8 zem}(NUz{*zYWq>2!1-y#XJD`S`SyH_ce=g5f%9POAK=0k=zKqg_76_^#C|%h7gr*^ zzJKs^w{(w5OP-+heuVOvED!fphK!AIbVJEJAFJy7l3o6rxncg5^L!ZcvXS^F#*4=9 znvEn6CO%;-nxgs0mx=u{=qufiq5Wwl@5c-od0_X{RNRC(j~FtpxyHg+Usv7pFcMRM z|A+Hq+|S|VWzzi}ly}02=eR)b#xcntwNh`!9O>B)A9IFO=s^AtRajMCrW#hq~v-+xkK6Nrk(l zrpLjml0IbkdDHkmyuAzXHH16}ygykGe>*?U*WfPNnC|na{iFR>9WUlZndOCyB@nm} z7kN>BHms*w-W*4Mk<`w=l=)ex+QdV~+#Iix{~;ruZ(x}65i)RFT;YGnC~&lA87pF; z7Q4KhoOrD~-+YoUyS%Hl@_u`S}Kaa(E1|SbE20TA| zF~>MR=Qw_DzQnj60=h4!z@}g5VGC(JfDh!+rJv_ve4P5C-`^qqRP-0?tBCqa`)j2C zxKC5n_nfbc_lNZVa`e-`(YB|gp9?n9{AltE_a7YRX$jcxw&_EL-w&kd=K=_zjY*&V zrat35zUrRu@Yb37iuIS7J&WaLYX{d}a) z%qpCBC`e@yei?2Y8FnEW&C zr)jTE{Y;el2>Hi!V(b5qF&oQe8*BC*{#DgI-{7Oc=Y!X%KJ4!(>w7*wDEkLPMv}vx zW^4L=!bs8VJKg7{`QMx$kRQXX%Fhf<|L8u5uK#nBRqV9aAw!pcexF1ue<^BL?Qss+ ztJbgU1MEGP4=UZMZ{hl(+vE9I{i^w${!7MOo$j>%+gTi53$n*QPb z&0L+X>to0$l;e#DFMp-OerIRv`mN7rn1AQ$blpE7eW&9iQ(mF(nW}yR&rabjuv*#At^#3dRPUk0sw@360It8#jO- zCcn7-Rqk_G`eW&L7VM3!&qKyl*q*a7={Nno68GP^exK#sw)Hzz+fSqWLHhoj(BCAh zz8p>eP8g|NpHnS+KZ9f3-{<~5>7Q!P={|)oAe#VrgMOu|bli8Fq3Ub=OJU2 zqx`w>uhsH(eT?L9Oy{pV{R528d|lu5`Hb|vR@Zl20Hk2q_p2QKd|^g@ma6YX*~>Lt zGPgitq3=2K7u)pzEBfwN^*!6Sp7-Z+?R)k{KhoR!JZED-!?3T__Wa28kH$~lKf(P- z*e6?`hm1LMR25hUEc{KP3WuTzYBd%v2X^bXIT23&T+b>-)U*OzNV(? zI3q*H>FGKy!G#c8UWbg;IG<7RTAaVAcrngbRP5A$=lqg^?NK{D`M2u)6Eec>tL`7n z#de9BU)Q&Af1GwxWZ?F+*s(s(0r%DN)@b_c)VE0XS)UK)r>p!48S5P7>G~OIe$n+m zKV9W7?Jwx-i=`T-{nR>L-?LTyhJBwSE~i@dy=3n4Y*pV^=M-sp?fgQCg}yJ&FRMFYx%v)!s@(U&-y>|qhMA?$IDgz&pQUMzcR+1t7B zKUMvohc{_RjQdU6`knjh(0AqhXVAc(<7wwj*sl#h{;c_Bv1WgWZ>;z@9^iBYx+E76l(Jy-DlRz z*Zr4}vDPeIwtt<&pD)t%d&nr!=F^a&`zucWg6@~L==wfa>|j{-eV*7MvGhGBWA$8B z-}4Ku)$sDdQi+AW7nQ8F>CpELe{y}N`yiISSNm`S8sT#F9r~r+=Y@U^*!qR@p^fB` zn)-(O<{PUOY}(&6%ic=+YUzIt{8d%|ug3Ll8^>P1FVoI<2aTo5c`W(g1rGaNq}k^a z#xg!1EVJ}~DaT7K`(IF?>+32_Um;)1bbEj8QWb}cWm@`>L8~7bpdZLzqL)v*E2_Td z!{FNUk5m8PPe7qny3^hY>%?p1`Z&v>@5_sHd#&r=kg@h!o$j>1;roTU{Vpuh?YFMa zLq_SfI(@xE|HJnWb$eb^q}y{{zlV%#uhr?-Y2^A2qxdpoM?`}~dMQQG>RyK$R>={%A4zh+wczERj)OW(=<+4?s`fr) z1i%d$xV{%T^tlW~Ro@q5!%D4R_fIijz`&|>r@yYxFCpW4hrSmS>iVqPZ_Fn+?N-Zo z&L`pigRbw(3w3=Cn@>u0x^q4WpRaU%FD%scIcz>D)#>_tLi$eE9ZmU#|2|*rs95@* zD^@s`eP3J>n6K)4!RiefUcUBPiG{uwmST&=+|PgC`i}LSOOYS#e2(8=m^BLuB>tYn z)PM6louAM3`e=vfa_zmQfB1WMZND7nX}JHV-TxUhFhBThe&W6v*QW}pPa&g-kv#DL z`46|Zx<3wouuwbScKB;UM!9w#9>U+R(0qy@_v4m1?6GrwUZCk0$}89PX_KbE4|Nyf z_4W3A2mN2Br`P8Pr@n^_r+zPofUEkY>tDG48%v){GIV`)&IjT4Q`gtk8LB=$WRzll zu=^V_ocb@cg%~dO)0aB@@3lDaxBK^yaV_QpwLD$lp?`T800c|_u5svJ!SaB8z76|Z zSh7LGrK`>Jad94g?b>{s4*mP!&cjXp($2ph_ThpE!sXdl(l6_Ln(Nnhcy7L5ZR%Hp ztxsIP@@#yd$hRe|vmMUR8Ss zf6<{I>oj`ESSb682lq!8;frej3gB+p<-;Du&=;qC(r$zdy!>k%_OGx+)ff1m4*k*X zNyyOcN2L8_U7w0dRDFW~sp(hv`cwBuo#%VxPw4iqq(s%PA!CE4Z_f2B+5doYeopq! zc|I+62rd0vAXeCx{VQ0#&^~V_{VUDWaM88qd9|>AB^%81YTQ40-}X!9=bm+@Ua7Wdz62D%*ZZB~E&9UCSMw6dN`^o9r{j9a{25kLx z?$-_(iwktR?%zh*|I+oV)UWE7Q$OMV7wB~7ewDB;Tx9tCQ0&kzzhBj_kg-7PKlT#} zbh>W82aRjP&Vw=F2$p`$7BirwUnP0U`7ZQp?PBx%SKRk1&0npZ2m1?(wDaI=Kd|+r zsbAXpZ{`*2mvtV?^-DRA<@OEtL6rN}T)(XQ$oBn7`#x2M<2+W}r}SC&PTaq=^(!%6 zwO{nU3f(?sI__gF*7OVarxJDj(CNlO|>iRal|A+Ztv9526G*Ei>U5zAkUci684t$mm;7VG+^ z&lh-~LD+e$kB^6?Z&%@h5Q3#|MSk|H^8O-}t=6z-B{*|R~;(oK~e-0T5 z4*dzg|ET+y3*&XX9Dm+m%RknC397tey_KNK7w&&1sPX`N>d;r+K05D1hTngTrN2p- zuI=`Pug{V-d5z@1>H3kUM(fY)BKI*V{b5wnVel1T@&O6DU%*?mXKY@#~7Hj9D{)Krq9r~B?f$N`k9%}e- z0pSDGKf1pR`H|<5{Qjfz_c>f2t-p^U|1!&QUMcLIY2QqJO^m9KbYIGrSIo~2e>EXN zwU4l8nm*EfeAQk$1)W)^^MLub^AwuPN=pg8vpFaZolkzpZFr`2{uguF_ZqxrH`lp>w zp7M#mcfEZ3H~o4J`ltN;7}q!J@42|XInD>O9p{6(e`)@Gk8S^SeRJwhxP5f$o1c#s z|A)&5t!CBr+VuK{`>D7;rOLa09}f0U)4y>23bzNkel2v|@73r3aD8*?SGfM_`W5Sb zuT$Tg`W3Ffx;=|^zt^d6PW?&^J0EoD*VPXFn(Oo1=Y7zxg-LlDUYvs8A>ii&!oSSU zEU@X&uhb9RzLny@!#WQRAE17je@_Yd<@#m)JtpZ_hT}Xg({UabOTVVS-$weS z+p|=MJ%f6ff$JAp_bbt_EQkK+{uJ~J*TGc%bK0*Qvv#@Pd9`ExHrHXVocf!Ts@pTC zel2vAzliDj<#*UCeSHagma5w`r+$Uc54wI$uRnDES+{3tnti1En$Gh$ESM22`<3g^ zuPk4-eZB_$nwylPVSma(?L00sHP5C)zuwp1%V_WSi10pOl<&3lH_H1)yy#t51Ku|h zA;0#1xaS>zk7w@B9D_cmn}2`zhW);2{QaHt?|t;Y=S{`?TBvpk5L>*TDi80I_XF|w zPvgE_hQU}J;ZeMg8h`&c$8hskRWo{DJ^sF*=((WJm;fF6^cvpREK1Db@0)klpVH6V z^hUI|};F zyihRjZ}8^v@kRf_-e=u+NgHo^U$WZ&&>i!qSR2NQDn)+0e>n|E@E_xc@!;=IH^+m& zzxpr|(C-B_q>?;vetC#rNS}lEu~YC};=f10LU*{QSRG2FR^t6}pE!c)krG++=OmGl8p(4MF z%MbBK$PbNYoOpkO7%yONqFB$1@ws~ob0=Vi6ZwVwk$sW!2L7i`$@a+pq;Y*g5{cvJ zrw~Qp{PfOH$II6ppNtF42&nIex!|9;p1}DBecD)}^V`>*kTlm!&&NNWzh8+*QTlRy z{D^g1vrYJq9{}AH=et6j>K_Bz5 zyY~}+{4UNP!mY&b8iA2tyf42TB|s}tAI1lNuLJ(k`?JWNVK+_$IlmcR7BHib{uAo^ zB^nCxdk%5-?>XT2L+JM#s66w2wPr7#qx_z!coHCbKLzGDvKR7w5t#2+!d@{h&)i;N ze#Y-F@ZsW20nX_5cxZm$_T_23|ErsE+_eksNBi8c2Q=TnJ}?G*I0tlsJpZ$>M~NRH zh#>mM+k+{Mw+HuasC=TTBj#&fMjPIz(Jkd?73?eCU*YbT_8O`}_Unf|*j_aQ z=JOYiOfeaO`!ossef+#cN)P!YdrtN7-*bTfK=1G8@<;UNce48v&EKyWNMd6N9E;zt z07iLLWG{FS{dXH~HhKsV#BcDo;>c67{!wlptB#ocvD4FUwM9#>$qV?Yzwhlq{rv`} zKNWy|qV-V|#-~s9r=~nrM|#;Q@p~TO?@uxF(}6nW2mkc^_&t;(r()-?nJ)h?W9P4( zE`JJpI9B^<12<~>H^k0Q8&Fa6x5duiHC_I{i=Ceq5K-%YCVGCzf5i7&49N1DXgpPa zYbD0Bf}M@V&uHZ_SsvyGy}VqMSAFH>`KBu`hkfTt+m}9Fd0sYprR8N$SKcdEUY>ut z^8SVW@Jid4KV5m>VNYFYc}3He_a*jBwDK@N#lF9-${)$g{p`7D<(c!5CU5F|M)~{M zzr@O~$(J*K1$#bLeoda7`7_xsW98T6$C>|^vGZ&4;>>@FosU(&CLhlH274)1eoY>n z`K#Gjto)^_JUR2HvsaY-AHjV>VXv{u#r+EUeJnDs{P&(PAJrV$;2z}ZALsVVnV$R| z9^`tq=17s2zUs(3jD8yszh~uPqmK89{5Qa&b07ZuAqPI>z=s_8kOLob;6o04$bklfer#kXC*{MCl3hhBd+vE|5nxyJ9m>pAe3 z?|r=Qqdy<`^iQV_fAr^TPX1={&gZ`SR%*-SJ9qr{$mebH-`^P|@{gu=zV^9UHIsLK z?zcx$PrdVpi{F04^*^V6{_RH&2YxeE_SUbiJ30RAbw7PQr)Ki4f9?CIr(o)PMEdsI zKbrrw7378A_ST)rk4@e1i^*4Cd+oJf-SPYHzV)mBjc|i2*_$-MR{~{xL7-rd3tNb; z#e!T|pl zX;SAaX^v`X#3rgUjmNE(!#7$*8Y=5&F4w%ZXvJ=nk4;(k=+sRv_q>nAzs<_1VKb9( zsw1lw4=j_WqX%NpX_-4+{^}TXTINodza|EqmbugAuZ=;cW$twO>tfJpnLAznmKbze z=1!NtD+ZmGxzpu0V$cU-(2vES4|4h!O5SFHFOgt%BaUYfY-~4B#AbVm9HsSZxdQxMLJ(*>5QxDK$e&zkeipFW8O1+ z9&ph-;7avXye(BecABb^?K%8t z2IgLua}IaUQ{^89Ac+OUc@hMK^kdQ7#c$)h@g|b9nSq%sG}G;JixY2((&+Ahn}yt= z43(~xKS1S=n&tbP<$D4i7DB_-{A&H)fR}~5)0EHRsr_`GOCjA^ejK_+jEh!&JhF!3 zwerj6T}*QDD4FVzrjY-1qsD`TKmwz|R_Z8A&P&qL_yRs$LDTBd>%hB!!6&m0KF_Os zNDL&hP@-0cR(?_-iG`A;DId2r&5jXbVvCwzz&rC1Idkx&h8aE z5Fk0Se4(Rpm!U^);;%LFI_1#RqmbDHhaP1FGFT`h6sOYXMQ>mA`)pryJv1$0l=it5 z`ZVpU(c9;_QvOBP<0$--dhBs4^l941{VaQYueg2L&hfS80HuCa{-yq3Zu{!qC;y_) zA5q#T=a0*6U(5S!U-bDWO8eyebGhy7dY|o!K7U1NpPaugw|(5ty(0P3=ciJq{8{r~ zl=ewGt+vnAz~EilfqS71?j(C%c$BXT9d=5aCkHe>(^@ZNtr<-RTA4>R`haw>-HUR# zzrpX=NqcS0qiCK+7ybe&v4chz{(`n>)aY`ah?-yOaa6k0g1^!83;ssWFZdfhzu<55{IU8k z=b=Bk|FOm|mi)yUzgYZ>HGZ-B|0m$T@V8Z8htV7Z$6o|K$7r@(r!&k!d>z}^g!zo$ zOye2L3*zJX2Wz#mloYSIW>cTa+~mUCmB6F}ZjLXdwbStV%}Qgoue@<}34Pp+S($IM z^cwuFi8mAnJ!KdScOoyFICylCfSL1T5 z+ds)=f51mOY0iBN_cUB2k1!S&RnA5Iu5$yu#Twc}txXtBz`ghg+=vfxI&mWs-JKBf zn%sy;kK^3t+_32?H<$_(eGJD$tV zK=P5@{mM8#Z0zM(UMn3VHH>8g7_+50e+N zTWimv?xYa!P3mZBWO{NanYaa$7nzTcSGjQIS{2ndF?$*78P0_icOuOVy3e5W zVn&GQ+%845(y4Bmi5X^UcSO1`#OY+WY`V&Yfy=Nvx-3NXNn-Whq=#|Ao#OTp7ozzR z@HIFcTngE}u({-(!MR9Wib%(@opTYgpwKlg%!qAobS^~oMa*8tdWLf$IpmgHP<#Zt zm(U2lwjqg*Xq8h!MCbEDM7oc2k!zGq*SK&Q-n!0(s6I%n-kbDrF2uVz7szk4TWNE_ z?c-b`y^cr^`NW!sWGQ0)M8QRp6`TJ4xDeI%Fnby68O8;ibP^Z%IyRz}lw6?MF(N&Y zbAiu(k?2?%a#^~xbrX^9 zma+gF6Pb=fAL25&sL)j|L|(s&UfOGZ<{F(6{>OQ`-w^R!>lnD>a_*Bf?uvIc9>m^> z`FD3rrZG=daZ4ek!fW3z`ab=fJS$Hrk1t!FnM5D@US8_uAG*QATQE%9FqjJHWNZ`} ztl(al8b86xf$vHPFqMM8LGT0Be~k65HTw9y7smEXd+#NTj=oaJ#>d$PeQ!#i-vf}^>FCul=ru9uwK3>*G3YHZ=v^`B zMi?E;zQP?)oj-bhUySm}qns{3`C8M_$qSi|o*#owmP^mCk3Y@n)6rQBdU*f!^27VD z)5H6()5H6()5H6((=Vt0HSu&8G>_{K{?!7Z$PQfyVA$#)ddUj@y_(&B9RF6-Lww$ozhV7 zW+>kedKsOxde?c`-Sj-*XZLMo#qZ-a5Toz(DR_8ARd!v-~+$Bt=u5xw_7sed8EA0R4t0qfGRpS5T>m(^)_ zKC9L6)%0tFN_jb~TElp|mWJoDBH%KMzjNr<85EpNzb>p`ygNn1S?rQqrO#&PHJr(w z)9@_z6tF$s8SJb^pUF;XIGvr;(&G(l8arZ`e^Kg-W9b@(pP*q6gEMl;J?91IncN1|e=lfXN@Tqj*+ z7ywKZE?h|T%$s8FZEofXdR*S(6n6fzjQRf#zc-1GcWI6_FVdXj3|_p0uMUb`gd@T( zxVq=@xpV?=R{2}T20n*IZyjLX^U}`Vn=kRlxR35cA$(7CmYXxQ!KX;wLK8Yc^ zJbk*v>D2;fOI)-=V86sQdj!sxxa)wxMG_Cz30%rC+ILW3Q9tlufkpk`&;Jlu)Stop zcvlj&U(}EO-XXB4KZBhd5Lnco!7hDLU{OE#d7r>${ey!7A47ZM!JmOo35+U$lh~t> zb*e8Q+w1?lz||5DN?ao``+GsJmAFgdI*HFq+#+$!Ns+!w;!%kWi39&j&<8k9XQL7` zi348{be~C=Snw;I1->Zgf?volvER)9B|*WBPFEb52+N-XNfc%BmJMg5QmiOu>wrL&kfCy)8zXwWYMoX4jIGM$cb zEW)P_A3x@(L5!Ohuh7EmN&oYYm%k|s0DGS-u(n$3B-LqsQ_L?NZkB_;@=)e)}ko=bH{1?Q58JJ$wzUMYAUn zPcOAG`&%Q)d6@4h-sDd}#?wp5JPh*a2pf33^+r02T8sFMbh|x|4hvEin!dnF1sRjy zS0dhjlGf^as6qS+CEhQ|y-~{1b6+{4;ROrALO{ke`gD>M2P=|fH&Xs5WxkuGJewJu z`RQGK7@Dx)PZv=wG6SWdS8ABeF*#QYhC0inoX6x)R6B?c$Vl#F6SZbMCx`xFDohFH z_?b~8pOin~Xa;^pln1|faLJ~&JM(W5^e&mqdi!@^D=*) zgW$}+Kro~$;7Iq5%s(g@PGd~-i!=W*M4*VU`tOIILc!1fwPnQ2Z;s=GD45~`GM)K& zROUCaS-+Xj9RFL9dAj_DEP(P;I%j^GM~DSbFD(pQEurigHSL&!k;63V0YmpjtY&L{=^oPZS~jb9h$tt>Hq z=~hB~8F)yb=L^Jh4j$BhEfMeyU>wztppj~SAR@6rss#sR^lsRr5$ZqnxX5P8Kh;mc z!9Lb3^9LODza;aU@=N(?N^s_14gTC4q5fWJDE-Jo!CAj4M_Z9Ug!}jnv$IP*B}Dc2#b83|0iXBzszqwoc!O7{GW`_|52IW z96vV-pz(LsPkKW<9sF1P6Cs%XgZVV`Q#=0Hd=Ob?>8G>KE*T-8YJJpJQ!kv^Ok5`g zM9s&e%z-o$%BQJc=F`k~sBHJ%szZ%sB5c}TUeearxT?Hu|Ek?L-%{4Lr+!~k-~QTN z#f|OzZa%!`rq24ZP1`r^EpKUTuk1ZgyQ{6SYxkbEEw`07t=ZLCw&k{+wZ#hUaBo*b z`)=ghwXJObzMF6E+FjLFdE?Co58TvwOIgdo166z5_t)<2Y1+#3R@Ij5?QN{ywWe+B zPSo9daR0tFeGRL2wKP?=-P+K#d;8{+9mP#mMoZ(W?S1=K?JYX6dreVuZ8_>_M*0Ic zAK16%_J&nAQhmky+YfENrEJHl#*+G*>&x1$dh5%IJ1a^K-Be$`L#C-*-BxvIYfIOz z-EGw+H{V!wXh(h7`p(T&hxRn>+uB*a<+h6E>K*Md(v&wJ*wwjde?9uRYkk|Mmd=WO zhidA}%R4u(+S^)RwtIW|s+NOICF_qIsNNZ)JqL>F_Z7D_cEn28)z`GI_+V4j`floX z#P)5ut(N%IP*r(*J3`)hZ!)t2nKy`C|6NzWrJ7haLb`&+X?`>&_ z$lFcZE30-CAE?^gx4*ru??4qkkc$J=Tf3Ut_a1>vZn>#=@9j<1Tif>UD^X-(@8SA= zYYx^&EYlvZP5X9rHMFfS+P`mW&w*HJ+j-i~`hB|&AK14x#`uc*NnaxM-I5nke=ND% zO8u@~b>P<8SJPsC)V`H|wr$Y+rxdfxYW* zX^hqHj;*&gR_%>BKIN-Ky8RuyWAx|1kp|3XwX3RPq-&Gu%5QBbIT9mnmrT3kmi_yT zwkUkJ_$2!dz81&g>E_02=tdNI+iB`&Z8>zU&4^O3sb`f`R{g%d$bG43k@Mey+Zx-} zcQ;|4jyO*`(?y$?^mIK9B~j)tD_slKYc%bv?4$WVk}g>N?LdE5mERgocC2)(C>_bg zz8HPkE9#CSi&lA@&%LW+@VScfxwi81((Sw4^2(#hiaEZr-d&fK?m+eBm3N@ZxV-YJ zS}reL_2sn>KF4L1*I06S?Q1NF*}g4Y*QzQz4%9}I)vcea#3MeElFOM<1rJ>7lH z!OHr!9`oN~i4;OnQ2bYn^s>WuEIi!Y-O=0@sSlDOy0X2#tLL`X+i!1cw(xF&dx}d+ z8-lBXO}$6Dnj1MIBtshN(VC{7HV|4n4&B_^(OVkqZLbd=?Cfr@@3qRQsPC<>1kd1 zH%j_Ua}&2K`njMzhJLw}T_1DZMpO@9nTmAg zzsGpdCNs=f@r<-@Ez;9UH9UBfdQGZG&sV5s6n~0%WfMnIMLey@^?>6l>{{D=F#A<1 z<4rvh{9^21k<^S+OZo`nshog}H1!8Ro)s~DD;OQAoU5Y5UmqoYca-?&5l`(okNPQ? z`bj@d6RF;R$A$={r~NGorhaaWlD-LPB9%jNk>W3*94dcM5>5TZUJ8$|q0JfuGj9pL zQ~Ej?U%{!2?Um&`D&x&@IcN4)#+!QjFS1|ha(~v;&x^90fc!BUGyXor&op$awm( zB()qId-6c#sHZ@KN<8*ct#C?-SJQtdO1ux_OM%K)`@34kJKI?*>%N+6Bq|iJ0Un}FC{QroIr+Ue@ zo1t1N3F(o!H%+uQ8BgU)mhdcHGTy0=gEF4VSM_mJiI+!dyqq_bcsbSZ_!J4K9JM_g zB(t3HW?#HWPpV+nO?HZc86V&X_iyCWj6W~+=QECYbH0Dx5pT}Z>8Ou_S&lg$7dhh1 z`Pkh5Hq)P%{qJ(5H|OJ1j(BrEX2R+22{yEL)OR0Y@XMR(x7GJ`G`8Hx9reuy{Z?FI zcXL~PVQ;h1%Px1tLInj|J6d}g|JKx6-`3pN8^pS$xw|*m)7RM8+|$Df8}QE+71`0( zE*jg~+{0oooaCaSwc}vtWu(5fxx1&evqQ3>wV@k4scP;OtYC#5oxRP4RXc88(bHSs zcvz&W1g~~C_m=hccDFY4^)^$iNRNzKY*T$j^R2Co&04(RY;#NSV0V3c^JO&gAnNNy z8-pE`e%rRa+baZXw|6%6wW0dUTv75-Y3rAU_P*i;ZR z|NHy*Td+IQ<5fc`;fQJo85(=Ku%{#k+#xnu!+ z)bPAV)Kn_{HFDF*M4CQ;>5F|0bV@1Bm{r81vR-F>CN`gRkBahB*=iV-`OWpUNi*v= z@c_P1oRf!_#zhKedMEy=tUp@*ZRS3VC@zeDE6{ksDVab%HoT`}6FJk~E%Q73ACUQJ xovH?B`|koh()gwOM8-=pGj)SpQ8T|eevhGIXWf(~TlQ*HGKNZ{-`#ZP|37|@M3?{o literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb7a6af396879822c1534ec7d3163f8df.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb7a6af396879822c1534ec7d3163f8df.o new file mode 100644 index 0000000000000000000000000000000000000000..9e9a92ff1bdaccfe04e2e9e18eec04fa9a19fe69 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY5sR!clA0Y;51p5~zk!Y~uhwjLlP0}=x9+RTcU)F6Q#eOi)f z8FC^-)-eWy8M7PbY@8(%PnMP;;M0%`d?N5U)9aMUtYFyxvptU6^*^hVl@S2EVDOu@8P!| zmI_KY>!}&A)%_ydZTsK!LEGo1Gj@2$mfiO*KId%NbN?CF^Z5J=?$LCDqNVHgT|W%{ zZXCPz;RZeQL6t3Z-Np|8UmNQ?+QANk_DD|#EBb!6r~kuJ_V%xJc6fSfPt94|q`o2B z1nTQZPdY35Kf0%X80~b;_I4e{9y#^wVCDG(gF7xiJGk=nvxDE#J=Z|{96pEVocvC> z@5J%&n6t;Y<5Zhe!5b4nH&hz3?Ng=flTOy%$cI@n~pd z$AQ6To_%&Ob^hYeoR;r~FU&h0-go(D;b)w0g_}=Y3iqwN47#FF=-|P@?I#Wl?wfx+ zTzUFLxVi0gxbMJQ;gLr!g)i*51e!&m3g^MW3-g~E+_xJ2?Eh|fPAkSg^;Wpz{H5^r z%QW7fhC=f(-s*$s=R|mJ>vzH6`phQJb0&m{vsH z@O3dfZpLGw5yCGAuaBP$CnugqyFWv_{eu;N?Mplv9+?T4mVNc7iC?W5x%hG*c~ABa zk`sGsl3&XXk{vxZR^)(v=<7%DlMkv&LmzG^f!}$qUElTkF{A&+^T!Wt+Fa$6 z*WQ3%4`s7!XGou9c-BAi<4J%ehK3(zhyUea(Ehii z(Af=p9;3Mio)2H&^_poeiS=E`a}oTLEW_ZH#(a-tf{d3SqZ2ZbY{?VZwV%>lg?vW2 zSTeirfjk%fW?BCToPa=ZsQ zjPjEVBd;cgMj#J#(SKn)JA4T`8F@J=#I&Bcaygji+8m6Ztg(JtH$JZVQHWr_Ma&vA z_JJqTRZ}9aX#a85Xnh&`W4B75>j}~LL_b~g^z$)N;*W7XZQ9k3t2FLou1wFhlzE;| z>Z0P1^9v;*vJKgU>_q*&3%#C${#=Nkfn-nr4)p&mT-V@JiTfU058!$NpZ|sX)42XC zjSqXh`+P#k2D>W%vpvABy$k)jX6PXo>~j%*OLqDVY(O@@0GppJWc@?1AK5|q^!saB zf4G$OGkxGOvZ=EFm2bmV@a6a5$CqKVe}(;G{=g1jIRv{J?ASC-=!5+9T4 zqG3ms9lk)Ducd=k1CvWw9Wzf)#Z}js?E#yn+ZeNt!;&Y9&B^Sz{j$hwixom=|5L~$ z$+pyso3sC3T=g9+(ZiOo1aLH&C>L}zb#w+gz4f?L*X~wP=hwQ9rnbf)ThP|t>08kD z#DdNp&23xTWA%nES$|?d^roS`)$i?O3p#zl&ILlWx`wXK7H>yKowucBo42#c7pQCS zc68QBY^SfSqdnm73^W%OY-{zkdRv;ewdF5v&#(Kp#bAKC{Gx)og1o%C{JgxpLfqCB z6fDYnG=K4uy1G2bDdf5b#OP@927Hb8jvxVFhcEC1B!ZO!zLtB!+aQPco)DzbSKqa* zZfn5X>J!$n0=x@M5KOZMS{edge-oc{{y=+YduO*F%?a1u*3s#0>&&|c+_%B&Z}kNl z?g=x)G+Y*|Ejdswo!o#%z&&6?C63u=IrN^H!pgQ>pc@%j1{LF4tlD*mJVIt7H^1CZ zL++;`cUxK*$Mx^0A)mAC-L!0xb*z9}PebnheT-V9*J6XUpndDs4qs;;tE+2!qArSA zwQNpdL%u_c-aNG`)FDeF2U>@l3oq1=)VWJ4d)lu z5<$t8K)R;_zIW3#Ep<;%&ur$1?CF{9;J1tse4e!vSyw#UzX5<~Yd=GWAXLTY2tHI_ zy}qKXqS#%E{Xfo8M4>|xB3+EnRs#7ugYd=Du12q$P8XV6yq(SMZSKzY_Lh!$Zp>_- zx5MYoFI=!_o_jrA78EUIC0)%ejqZ-_j!s{zyS1yM)7|LX+T7-IudAwBxwdZovYNFO zRjcckZCtl(b=k5qHowYU=$`KlG&H$8T5%rmsMn35iB~#b!S^2)UU#sk0Jkr6F!F>v zx=$DPxMlW?*!`^>=h&K3nSHArc0t=m?V|4oX>k?y=#1v4roy(N-%y+kil!ap?N7yk3{rk8mv` z1Pb}Z=-I|t?=CIEehnlUXefxWd=Uyoa}JGZbu6i89126h8D8ipzXnd*)v&0KbXYcIK*ZJ6Kubo{N@f` zAIi2D1$7(Y86!O83r>h+qn0oU+QxXZf=POm{$v_U4@^jyK>ZIMN@C}To^kplq6a=V zXrDl5ouYrD=hIBkqkrhbM&kIZKj8f5LOugM_(_xY0zc@F=(+wfIyK@4b)Dq%SBGuH zZ&Gj~tjO(~pzwk|%n0uaI42Unsb?ceY%q-#{RR3*oy+?s@6VZrn!g_Xq5r4%u?X42 z?tndqs@Oxjh?WZEyOhSh()~b$n}hJVPUuH}V0~?EEtmJrY{)xFCtr#?f7rvxot*3eeu@HY!bCp4sQ2zlh_DX;ARmDLWdv*y z^2kV`z%T38*cD=wA=mn3M3N1hQ{%1|L2$0RD&4 z?5X6>SSbBlpda8fg1Iwj-GqPf_PIfrW4KnEMDh)0v+}PW0hS8Jzd-aeXIza$v>5#y zC+ROD8CZe7-$2EtUPHlg$XuokjS$9KVn~)y?LTY7-yAqBC3! zdOPT`wh{d?C&$}S7x+@(-N3s|csJ?-zYO@Pz~>44uxLLOb%FmX@R`7S1b%iJ?>`fD zflt=o3A_t+flttdJX<6_Tk^j};x*ujf4{)jFNA;1oJsO=|0n+n+HBmH zl)nY%%pq}N_*t`}_1F1);r8J35A8``cJ$BrdB>)gXT88^U5!Nca{Fh(zR?2uVpS)5 z!#^;ebNPIw`TWZ?R`PXCkdyxcU;g!<%jTj!x<4~L{56;C$^8}Mk$-I=5;-3-f-`1N z{Nw)k3+jJ`ssG+x(_;QKbP(%Xf{px*FAAp@!oRTINBxWZXIcV{H;wzxA+qPRsDBMP zwW7e(gs8t_J(v-kI+f@XWBxWZlj_{x_M*KD?Ulc+CjCxznf%QGyj$4w*ffrJpf1{z zKiYwp{s#YserH7WtL#k-kiFj_ebYUP)HjW1SN=Iv3jK5nyqwRY^)Jz~J@h|=)UEu3 z^v}PMB0R<`|N0VBf4#dhV*0Lcv*?@Vw?W^0zGiTJXGs0Q|1+ZcuFuAL;j-x4<%;RM ziuC1*&2K(G&|c~r{bvNBEeH@9=oRZP%@1xssjn=vywqQGJs2uxJ;wZ70sDy}=AShll6}t*fn2|ff)6}E_UGdxUX&;2@|ubVy}PjVtN7(7 z`Sh54IZnh&@m=YIoZmFPh~kgm29weHP4h?R@lq!|e%$aEioaJthxqxjs?(?gsCUtg zEWjV^W;*S95{t(_0QuQy`6X>^eUkYB@;A(%`NWSZs2|@6`#lK& zNdBX-_zrzS9!q@J4e>oYPCQ2rOeT@~)dQKacs;aGTM@8RTMs{*14V z$)9_YXnieDRsslnn*ERTZ!^Yc`|+K~H3%ZN`xJ%+ce zXMBCLyJS2W)4!`2)@$w`{w=gWfq(3y$w@`|2j=HlRLQ=uKl1%daz_yl$^MgL_J{wQ z=jTFB&+SQb0{ScpPSt6-Wu(6||CQFSsfp}Lgy%Oy)8JppnNj~5T8RBXPA0@h0sS!T zHyqr5lB4^RtehzQdD?F{lDWOefB62y&FQm<9{Ai~vLotG$ZyjF?kqQYj}Sfi({2Y} zA2D7U(ZioIjsBJ!%yJvs7wvYtX})oK?vFMHtRwsp<5T;nKYp0To(pp%i=2(55r0L! zpcn0vDBnRoZ5wzByz=!e&4v8T9qkYDrw69#lR+PGVLlI6rKSh6^(j=Jgb9T4FU_V6{$Srsau+|%2gq9#%*>3&Gb=yDe=o+nF+a3^A)ph1@NcufAENnWoDW!kO#TkM zX+8jN@^|13^8xzF4VwKQ{yvTD%kv%Zn-Mh32gpnKsQ+KmD4&9Vp-_B$2|q$a{>9v= zlYY-FL>=h@bx87Dp#7U8A*NsQkK_c4eluyQ^4E)a0DhqjH9xB`zY|c0!lUy)F+GM) z13n3LBi@Bm57nOV>8Km=8RO0P@uj0LqheSuhEe75U>eyk^#xQ#JSTl8Vmo4t z=ZT5bzl!GxK{1dclu&{NayyC%m4(I6nk<1M_9K z14jiYHmnz?F#c{~4}bmpK-mWMRF=aw6hA^k-Vnu|AvDZ`8A}UPtNC9_vwTzcr+p*K5pI z!bjKZG4mBZzb*L+ZN{kxf8_fKWq;Tc=Z~3_fQ`5?-%o#v`3m_%+nu|@eJ`I{oi>1ZHiyWr}O~-_96QH1L33b4E(1fp3#0Nk}2}L+mC0a{f0Sy zVSLkm1Nt__Gmba!KTP`#;7##unCIW0FrHm8`HMB4;rpbR{lRZK&XWcXO8vlo*w0L& z?_>D-d=B3??wSzWpU>p`bAEox^X)emf_{==KYj?`H`pif{W$lJ|3Lfk*m)1e!~Q9D zeueda^nQHG)SJh*$=u(k#E)-?ACsr>{g>L0kB)Dq^Pk+{l*yL(7CZm3?r$-FZ(+YZ z*%;r7g5!0DFR;Y^kmg6`_zAF(*nekZ|D83SzMnJfzo*iE(Gc&Pv>}P@zn#u8_TM;A zp>ikjuIdi<-|1K_!Jyc$qP+|C=zbLW+IZu66Yv?v^C)TEh?Xrk$n7o zk%ScMtDPK#w;xWY{^{!$*`F3W|IfQMdClhsY3z&ON97MZU&SDQgn`vM;B%uP=q2`N^w) z_M%OTI@y;(tkFa;(9*)a8%Rej~00y)!2h`fWJxrC8##eo^q8( z=wcqIV|5MxXBoVYbEfeYklZxos zpwlFe#UD5owwfj3T%GHJv*@Vk#Ath=>*DgZlz1~% z=&1y4G-e&G8c>5CN8yPZ(!m;CUo{hR&&@rkmo{g;n#TsLDn@<;thioJ(Q_P$zvnB3 zHp;tbY_+0TSBw}>H`1W$H~tC~Yrj{w%Iq?X8_%{;&jejOys`rIGg@lH$fLB021acs z#^X#+t#MxEn_pX!cD0u=JnV!Q8V&{V0{^K)!vV&ZRUN-*G1Z?&T`@`Y7*wY^j3(j_ z>U7JOKH$NIiT)i+eTinqdo1;8OP$7}{`nGt{x^yfZ36M}Z(1jhxZO6Nl)gB%%znFl z#A|aayr1DK>R6-QN>5|;Va3Pm^Rwf|%H3FX<4o+8F^GGaa4YijoBCC}_G*&Q)EcwB zX3|QO8)3%l7^IkXVOpgb{SWD;d^7nx0^)+avbR;KVWu!8mc^?LaS#` z&oSHc^>s+w7wX@Ffy}_>;{Pee?QhkAMi2L8vR+Uk)OtxpIy|9$Ngu=jn$rG z)r~Xkehe3D$XGi2qqYqHQEB`(p(vDjowxHMzxRbIr$t+h^`>C{GeFr@3y*0KXFz*t32`O)#i>?T+rXu z`%BKh`7H1AcDDC<*RN?l8Fy?V*2kjW-cdU?4?HLqv-dJ%)KjzeZCzhohP^i9XW!aY zcpn{xW4Eo!m?QA-N<88_EIN|4oTCdLH{l1w@8iVwKi2=~lW{JAp?_WD{r4!|=yzwv z{v}?a`G0d;qiYo|oG;LP66K5?2R3n@VWi#k8Agj#bIY}7g#h3y8whk3Cm!-aM%86e z%3FEd2T%Ng5Jj{jSZ&yAk{Lkc2W&^6w(eF*+5`h#M_Nl^5sb*L-t zJ_a~HnCDbC{xgU7!HP873H2p2n22G4Z4qBT_u4E7p(l zmwK>5hgGVeIWOWlU6~3Ty+)tYB_WDFN^plaG}HMpou$1;)*GTUMrm?4a8$_ih<9uI3PR%Nnc@=r`xV(fSAvv|t?g+) z_f(wr$1Lqt&(`+Sg+lxuMIf)IWcv?foyMo%npoSD-=m9B{13<};x@FWTh(6K@en#D zn1_&ta@I1%)cC5Mvi}oke6#k7j@lEQwLQ&qQh~)y86(F;>B$jSYELvhq8j~+|1!Y) zER90s%@TQ~L1p3p(bZ5V@b}<##1r*zDzd%D1hhCAaJ0S<|0NbWJi?!spBp-kXFoq2 z6aT-C>?b{5v^)E!;=1S|am~Voj4(fO+8t#Lf!Rm}@)J5~2c2Qjwe zAUU5Dow};|rLL;ow&JzxJBpVTi&DC(q@bm%p|GT7Q{md$?ZqwYyc-(3Hf^lQZ)n|6 zyQ64LySKP>Rq6VYriRw#otrn-v^4nFu4}2-xxAsE39z-bgfDAZx@ObH%BK1a>+?2m ztX%2cP_(0^vZ{9N&hq6g%O9_8^4F|wSy52?c*V|&jRiF=n_HLX*RK$5OInt;)Nd@w z^KIt+uH0C#ysN%-X&!j+*B30V-CnYKd*jBMj!hd^Ke@7Xb$(-cZCB%pmYo=*4SbZZ zFJIr)P_VR{#%tVA*weO^oXw#w+Z<=a+!i2Nb^fN-ZOgV7*K}{%P!xzeX2s5p zo4|K{`SK?lx{I12ZwqwwWPNE-o^NA`zrNK6`>e05Z(H-^Dzd?f_08VmwW~_nR_8Ue zu5YT}T+?2<$y;0NuUXns+SFdQVOzDgq;%Et^40m9%a?U+YHjJ-T#irSs;${=9iZ%J`pKM%F*|KRv0qk9qzo~6)ZPR99uk9t3ThVu3C_1YJY(R&Rsv zMz2^|$KS0&Z&ktDPVS_FSEJnU6)E|Jixw>^%*$KCX_k9iI(&8FZ)q)FDC6_C<^}T@ z+m&-yKR!9nU|_p_>Z$IfcWuOx>3`UC-;^v=g5H$6pDjet9wAa}B|0wgpf0Wd?lotw<}K zuZ*JsUKA@W(Y4$Ho$B6teF<&t?0t14VM5CX$C+S44yEiEidfLZPY(pJQCw>61m}^5 z15|vVO1$Siuj5^#QJfO*In#okN5oMrY=&_aQqafvF~j9fNC9yb^v0dHK(^zOVc5AT zxhPB$T=~Dy7h~UoTnZ+75P#`M&97$A6a0#VtF{Fat~RMip z05{SfzXANJ0dABxSL)r$=WGeL$~zzpWR>>~3Agt1a|yS~`zu2~M*crCz>V@|%K1a% z8e?SM4d6#4ocJ-WW-mziBNq68f!=7}*CpJ_KRrc<3h^^p-15JRBwUTFe6dNw)wl|N z#6WKx_XP>Jj{7SKw~m`C=PiwEoG&g3A2nYjocd9`kN{f@aCyYTnHx5`^4Lyj6ZPma6N05^^slyEh!qTemy)^Wct z;nr~}Z=piD3r#7dJBn*m6x_pWj5W!tf~&akf(5SP!+-^@;=r&4u9jYP{;B$}mUfyA z{-`K;kA$zVz}0&1x4_kU&qUhYQCHvG<_&Z+>=7EhE4^K94NZ^pS57X&1DWx5NB)i{ zdR-&C)tv`*RM*_rjMqHYcXZrdv$n2Q!BDfWgN?n(ky{7;ddab=pYR1b@M;JtYIA)6 za+mu$g}8We3BAX%e0A-7yicxShoD*xeXR9$7I)(Pll5JlK7tB*G&DgQy=A^9nj3s3 zxX_-jNxtrGY!Ixq@o1a_IdS*g7^fIom;h@#1kQKe0hi^0K!7e*@(f-T(jq literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb95baa3070802f80dcd903c53a3964f8.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kb95baa3070802f80dcd903c53a3964f8.o new file mode 100644 index 0000000000000000000000000000000000000000..f7d20fb18dc683147327d7db6dad747243ca594a GIT binary patch literal 28192 zcmeHv3vg6tmfqjDTS6}+fw4v}P)i6~U`ezj1jxp?^#;Ab;vrCG)3jbl)-$a}Zkd^} z42jFw7^94nomAEnHsef2JY3$?)NY)b88aEjV|$p%n5s##y+))g@Y%?pOBzVD9fy z#izgNGUh(s;-0(W7PEir7RQfGido1$JF`esf4{*0(r@a;n}1=5*}}A$&U5apX8?shd($GeYgD=(M#>GMqA5&9zFcRnP~En52LFtycZpAx)41x`e8I>#k#pS zK6v_UTG`sU;R{clZF%8T^aS)T4gWmadh$&4c-x23r`~@rn!2)j?)Z@-XNQsBlVvBP zt-m-GeQ4|#(G!P%9=&wpgXnw1??YzI+#463K6?UXd*iO}M(;U%D*EK(uSW0r;0)*a zv$6}2xo_^tjw5H^d+Ny9mNO@#C(fLTu6p&==1v{ybd-ARe(WhP|J2i8y9f!{zFFSnpXQkhbK3R4u`m+mXqL=T-nq0PF?p@eOUAcblb(H52Y`hNrloj{S9fJK!Xup*gPDE4CR)^3wEgff~ucJ-g zD249IjdQPGIDB^Q)XC^v*_r5}vJaxku7XeAJ@a%YbVA9G75wMq#F@_I9~VTZpZso7 zorFGc88e+d@R>VZ{!h=j?>tV}9z#@{{R>R?@15`;*~fcaqWU%TvCBo+oyX4`Nv))J zOo-|qgIB~l($>rHgO9h=&;6#W4u0kT^veEMzGwH(`TXr;w`-O5Ir1;SZ{qKm`4-Sf1`0;Gw3S%xA@o?#&c~^p%)MJLW)UOz zU+fjL7w=A-BO4d1FlHjwF5V|-415-4o4vCC$7W1?*~*(F{GYBwdnCKWOBC0iZ+Ur+ zcz=UAS1^uxEs zEM#ZjyT_ROpvpCO@$Q5<@*(o4U&A*)e82-<;vD%B`5gHYkL8rFF>^oHw@B8P{Oj{A zwR2Y%)O|wz^Z#3Y4KxP)kgplszbL)+gB}0HB{IcCJCs@9<)R7nlQt&#V-r4;kOM8r z{J0k2S|lJ5CkH3Pp>VJVPx?MRDrvq4CI`p*BBFe3 zJltPCHdP)T9vT}Mx9Hw|ik>PrA9}|}CxT&79`28X%VpI9z5Bu=!O6+MXjPzpw5Pu> zFg6|<4Ja_&KQ=iYnh1x6sw(%6_K%KNRs>95&p^dmVg^Ts_KsDo9k2BHd~4UNtysg? zbp%P|I&kF;O$2xg4P>spKV5qTA z_67z*!O?!%J5ENhJdT4f>kW>K>^LgV4_@bm=K2^$}q3Aub>DJ@uGPV6Qn8Q4T5nRWBjm#4 z2A{W=A=g=|++K$KUM>r-cKzZ&U}!9irF?KC;0u82-PackOpaDX0)0bMfzTKhy#wJ# z0s~c*fyt5ahp|Y;WOsY*@_V-Aj~nZ#cbtryU%PzyH$Qr0--aDn`S`%VWPjKv0)a8O zvs!;|Yj121NCaMwU0QIGr+gD9O)PqDsfFAP=ChT+-PmP2Qw{SIoow+3%wKpF|ED;Z zFvLP}E=NLMIz&Kv8k>JNp8!Z&fBum9DVj9ze-8fQOq=~e3H>38NRNIc2{BGDg6}<+ zVe%DQys6}M)>{4o_->qsI8zL%STX6D63YEGx@V+4HZxNql46g|lqB(^5X3Ka8d0=v z15Z8}QrELfH6`h7m?xi^12ZMgY?8i(Y_kZ-#G#5z|B`s3i1R1&lm_MPsc$j@$r$# zQZELX{@`T4x1y?iO{sS$-O8(LMBTojkv{L_^kley)H}LwGVJZ^9~c_z_io>^rEOba zXJhBK&09JGjk~rtb~H3Lh_Wr-DsNdZ)H~>%91T?X)&;%Dn#xM&L-_xQqpZW?aqvB> zlenz^l%5wQ>8|Bxl`jGiavqsSh^6Wah<~@aA51jWwfA<`5y60#HS}SP0RqIzNgExOh$)0XN~ZrBng-j zGj2SuRs3Nb@r~lz3jm4AMfcCu{ffGOhWjf7AYnM*zS6PKz+8*3NO9~Me2xFVx*yo@ ziiy8~HFB`a8R93?7zijmfvYH8q*Dg@n=E)GOccUgTQ&S!{|4=2X>fHVab{yoWe zleQ2maJ`c*W<#C?5|5oek|ACx5C`@fF>wwI8sg5ULq91d{u$FrqC8!c(IKo(uwQPc ze~|%wD^54Ci)e^JDBb; z@#Rb}H|Zj#i%fb0(>xZk{rj1w!+&XiEz>kEN&D-VUT5N4nBHR2bxhZpbS2Z3CSApJ zl}R@;-AFW--_0}+wC`*|eFW14%kpewdZS4{!1Mzq-Nkg5N&A`h6V3LUm~JxhW~Q4> zx`F8ildfmF-lXqk`d*W6VH#nr?6)%AYT|87x0!T1)BJA$ZtqP@Z!+=COm8;n^-QlP zn#*6qG_4C{du(MI!&7`+8y`5;bzk}%=CSJ>Qtx4a<^nHrX zi6q1d@a$Ove?~l{g@lLVVWQ{PZj{fpB9iE#AvtCs9$w23X%XSU@D!V!ri(8z)4i$r@sV@@Hu zgyhpeqk=>a5}gFvYooJ4mq@yY;?rA3;{hbl-sInfWtE@{iRSWc@DncM@flB6xKn&Sz}SJo4);tTRs^(4>n^$O*;rH*hbKG)U~y;kN2`E|7V}NBxFVW z3kdg{<=dD@IL9B`e;|=?EB@RGgv9uLv%UyR?ILH0O)U&%1CA_*CUvp+Ci zpH8@ncjT9tPPq2BtVqK`!qxZye8ED(E&pmtBfNVJ8jR$SXg!6d9_S=&Qx5kr}B*J+-V*9O0gj?fLZ64vZR6fS<%OhOJ zb5>+~F5$NEAeV4!Jm}0JT#Y9v-?kjWt?^`QHsNYK0{&n&;XEE;K3yJ4E~B|HMlFi? zJFvezCf)&q=GVK@2`{C-OeP%T!Sayd^~TeM6f1^_C+{ZTuxO)$e8bXDUQWKnIPxow zba!*O(s%?qCsG}wzR%Bcad}om%9k3T%W3|`c!Kg2kuO^Ij13fUv7>M16;rR zDOPoUk4AaKV&Gmmen|UUD3&d}1h^VsBwk0pZ!2#l zjSV)sipB=k=k`r*q*%0Q!$YxR$tSxhmMxq{S37QOq_Mz~Hy)s|z$V{CV~LIS`{U)C zC{`@}lm14z)KS;4+!#gR~Z1^^cH5#^am=rW9!)kx0qt0@NhwYjg`s7J>|qGKQ9$@IQQ)`X1Y@ww)>ObA zjNz{*9rIy!Stc5+loZ}60G;C{n#L44|K{!{x`SvQU-Om|T|_k6OWMbLK=$8pk-ZIs z%lVn@ttHx;Pjj~rT}S#nzvNXC&GWNqFG<=v?-Gj+S65T~#WIg%(D+msT2rxxbSd$E zBD<0FNk*05O*ALApSzK0Yd+5FBHB*^W`3Dw`#f<_RIec{2^aE1eVRyK&R1NY2BInE zRQuEu&3!8Cb1%_6f138Q&H7-yN8dyd6MpJHw%n79a%{H| z`+4O5roS4IrQ*c#RW)>eK=rE#=c2x+8@aw{kLBT9ERSa%$W{GU7&3f$I=|Oe3u79g z4tZw5|CfjJa>*W))cl%jkUTdc>^Dxk_zTs-z__skAwB)yxRHRb!2C(i+2;J1 zn@2l#2yywE45t2g`M+U0%&rETZgDBYFhaRhKoifNW@Y(-{Ay7i zPE?*twEy?S1lAKUl0@ePty9@4Tz27g!C#ExND7u0F1i1Pf3{BLW5Z90am@c-Z#=#% zIUX-8H1StRKPe*~r|HygzdSu2r|H#>`_kfZT0SV8?XOuFkJIwO(l3tCa>1etzhUHZ z>{AP1JZzoHO^w%2#n5KSw@#JbVdBWII6}({g`<6wCJovCOVC7U&*BJ8hn9YEgr-9~ zP1B8?rs>8`({!z9&X1M{cAS<6cABPVJ59?0i*5;}7!;~J;$gflj?i?faM*7Nr5j!o zr~1)!9H*Zz^&#{Q;rYws@w8)3MyN?6*#N7scZlm^v)H zCA8R}_>kqN{AoF1mA`drS!TRGEeGuSnYoK)=~Vd_WBF&t(+ydEcmiqp@IuYWy;c00b%CV6-yufnH7)I^2NeiLK_TP zPg~`q<(1uj_MP$iw7jzPTc@&?#N)KQvT(G2d3^iP^3t;35?X7}dR*t<5=u&v?FAOv zvvq38-FEw}Q{{KX(|4_irz?w1n%lqFC+jEe6=Oed<`4TVA>SIAzsCI-tF7|4POV*O z+Jk&?WF?kwR{2|}I!fa4l3Ekz@~^ok9&f}{ZrN`MwHOF^B;CLFVJf%VU$rV;zZQaa z{p|(@1g-BWj%>hk-;S3UB^HkId$1g^?6-s(jD|RzRuXpoIs*=kglx}ecs7PAut22s z5k4?}wS=}B2pA-d`&JZ#vhYp=4k>Z$cUwYj1_Dsh+&`NP3>cDTe7(VYbJ-q{Z!$0- z+3=fc6I}3PK+nsvkHXO9;KzGKQg{5kLGR*FDjDd2b1xYIoO@r zaIY6NlZo-XGE6u&eAzO;f~9`$atx?8JU?HyADFD4Q|OJuo3dnkYdo9RUsn0nFOmJD z@g-FhuNGcjF8fpCm3L75Tligf$^O^)9gE17EPU-+=`R}3!h+6@cXUX9)A*ft-(|zG zKj_AaN3Bn(e-hSFe6xM{!^V(j3nmCjqrW$XN}8#@j5EC$U){IzBYPgbpW*gev@lKM zET5T5^P8n#m7Jt$&hMV3`I?44*`xaM{)O#*rH0y*^uTx9=f&TBKAU)>>x{vltSKIC&qK6d(A z^7(P{X8F$Imm%C9GC$Tw{qrMHF5SZkm&N?$|BP{6e(aa?sXZrpu-~HgG2+{2`4c(R z9swTHk)QEIt~gEYOYMjC`PJ+1`1ds8~%AT!pBL! zjh%~S$n^ijJ;l9&52^z+We76|xX;Ww7X&>u6xFcV!GU1dspI9C) zfIH7TAonkHK9`eUfb7V}<#}Pgk%LFBAD?e6WB=z`NIHl1bC&$lr9@k}kwdnLNB))* z$+!F&`imp^+`jCuz^g+A`EtJ{{j)lfTY%~yd(&TYKZmJ_J+eQV56zHeBCFh1m$WDVZTfd7;sW{SgmPmvf9275gYxA^ zo*`{9@xTPr)PK}|xrD?10*p^YBYCKQ7e{h(h|d26<)QkZygVP$c*Omilh5T*{kb?~ zPym_rZJwsll#(287DsID&+S9>%%lAOE0n)^T8=hazBt11fH*~d*NZ>3|81W3a&D}D ziOP=v2a&T$m7nvs=eNX?$9Nbg&*h=ngpw?O65<8n@i~I`1R3AD|K&#DL zUb2Ox%3BfU;a#53pgp!uf0q5fDL`UB!T4JM`=1H=zCtW^&@E0o#-Ce|mW^}!^L7i| zZh_k^aJvO=x4;)`frSvq%zOHCd~CY>Tbeju{vCZHiT_ADeer*O_|cUgjpIv!-#Pw5 z?8I@Tnd8S-iLE(U(8E@_C*s)symxp!G;CA4j-S6H|UGX;&#` zc?)+fa50?@oHla@T{Td%@W-+|z>T8x#83E}0SAR&uBfRzW-^MB5`W9M+i>EvXRYlq zQd~U(WwMgZrpu{Q0x?%=8sw==?nLY`Z1h2O7?qUq}d*TqTadiv*8qa>*6Tj zb#X8Hxw$(_`YxWSJf`ot6a5a~gSdFm^Lyy7pOqG5 zxIRgQ+x<&w+Bs|x-~Fdix>yqCx69qab=_Y#tWomL+r&MRHPUW>57Mfe8JxaT=l*GZm)RqxuWO)=N{y4Zor{kOC5VKd!^hNH)H|BIR|jHaPf- z6yLiBU*pdEN-)0o@0xg8iZdmD*R)9J^65&*-!;(*e|(a^Yq~R@3F^-goQpFp`xDht zME|ZyE65l|pJB-G9-|pEKAFF3Qfw_BFZIPK8c{GDU6uTb(~D8cp3D(dF;j7$PyeS% z??2Hwn=Lu@?*NA7!&Pa9&WDk_N`Yi(G}gt@Ewl>_;j~G5$7qy7dQ0BSPp2&T3;8Wl zTrnc5%T7wn{B*W^4G{KwM=^?(i+GhL%>-0-r#}9A`P1?dt*`s>85ky> zU&-m)MWwH4oxj%8{XYfe>*;S&21uXeocezN9#zrdrea1PV-^BLB7N%T8C>>+ul%vi zdOMSz*DD5PsuHyy7eX^|{}Xr}m5J=fC8h6=V>+A+*z>SY<*#$d>HNQMGdy4aO~5f} zx`2A7-NfnpB=sY;grPHOs(+ji=m*enls{4&^o?4{JST7Fr;;`4?a)6sPyTyUg?vg< zo{3-8e4=0J`yKfo)%q$!(x;HNbq8BsgGxO)PyVM>hk98`2xtDPB`~ZQJos?a20ESKq$fyE-d+ zN4s_mS3fu&tgUaa-&r@T zk-p}g)7xOLy9F|hlXZjLo!dq>2Y2k4=&Tv3A8M~}su}L?>K^Rtiuj~%_ds`7#lwA@ zMv>RH_WFvN!Jg6TslMhWU$CoYAM)A0yK1Mer@85o-s$Rx`jCIySZ8%_GxC6Z-ALz1 z&sfJ4mub_^p+g6M>RPKH7p}Ib* z+vM(D9gnn)whZ=G?i}de)G--X*BvAK`WkloMq0M)P<3eN-c?6^Fkkt{AFAzZ?rE+W z>}zPBw&^tYR&`EycQu9Q)!Y2=4z&01?rqgW!LI6ITw`eW;oi}a%I;mQ@Q=ZUox3`G zy`wt^d%DrbX!E|7ZB4R1c$>jU%`|)jzEeH0yQ_L%PgUn&U-QUR&qU{V>*j}>wslqe z+8=2}Uw1yTduYx5Ef1UJ)Y2QwNPFg1UJ{FK4U%i1m<@Kyh z-dq=NNEBH8)m-%#~E?T{aWJlUC0U88j?6@NzY z8$eSkQg(i+_`03{RF$6>dyvy{^>0);eX5HuvlV>hRN3*W4%GZE@E2(#kbN37r7-Lh z|EQvWCE4OTxD%fivXowxH}NL!#9yymTHVaHxdZmJ?~sb#w%zIA`;@(0@O9e|>#K^N z2fo_!GUsbHzI`kDJ;kSd&C_4-biYx)c7Cy{C)w9!BNO!hE=o>2>lJ^Q!;ar($9~GR z$HvD#&P)*--!6an8vM6xe0zQA+tZYs2RZ9Y=LGZFc~3p>>;z#?ObiSJ3W|Oz)OZ#R@&V0YB_|AL}+w9om<*4F2&ex~%eOB>xzM6m0 zCU4L8eZ_a?`?2CX^QCnSC0e^sNa?oI+egj!lMvFNyqWTP-uRY-pQ-%mw1cnbfr}2l z9`E)3UFWOE`x2ColICNG<+RDc*W>+!gRjSXA(!2gfu5nUU}#$K8wJ~f`^I_)TltmY zjS>6?gWn-2-y3|>I}v_0;n3I+Uenz(Ir+uZ#`cZM0uA*~iW_fw)YgGt4}N3mss7L; zUX7)yI@A+Fy_@^PvbuPO9KDCUxnoBe-Ur+}ETx*z9^3lEwPCzKyk}pypSV&UigDb& zU_<}ZP;Y-6pWeV2EbkvwuPeTB5eJYTy}m0jMz1$+Yunl0Fcci=?+v$)_w5@&_BXj3 zZ^o5n+0;EU!Ziu-P&&w`=7lYz;J{@$zK3m+Qq z3x0ph9Y+vNMzQ>J5!rspj!csCws zDdpD;b)VQHAHJp5S@i!vn)lT7Xr5R6DVUVMeT_|;TUEY(B5X-==NHN6OxclHTkZU00|#NVB=4=H_T z{kJOpqYedU`TK#-*MCXMe=e)+ssE@+wZ2|UKZT5)d6UXHwShXS8itBNzbA9*|6fUn BX72z1 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kbb78dda30e4032356ec7be602bc3b424.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kbb78dda30e4032356ec7be602bc3b424.o new file mode 100644 index 0000000000000000000000000000000000000000..3db34da18b05564db600a56889929e811d40b0e9 GIT binary patch literal 67256 zcmeHw3wTu5b?%;rMxzIM0YW?sBM=~j5QD_i4?GeN32ZEwn&xXzn$(Kz&v6~cA$+~5Z`!XW*dc<%4pkgma*XHxYd_A} z=V&Bh$LVcTZ+kfN@3r^ZYp=c5+K+SQ=$;L?Zge^vOeF{V7Bi8`W$cX0%nk7D082$l zAB$8CxPm{K<8%EV`k+*AE@MY8xaORE4WHLta|(Wr>qUHif$yj~MU|!N)w{pb_f85M zd3&d`@2zdFzN;>F^tUc{@^~jZin4#kCbJdaoD=AOdma1nt4?+_H#Jgq-j#jwf@=!# z`ZJcrR{XOw&_9TJPKWEqLA1Rn^^tQYAAa!MVqAaq(1Yjxw=*z;vc32mZNB`}(8-rh z4efjLsi7xvU9{y#LwCOU!=c5vw&1$y$QwhOpSm=33$C}~nmm71-#@gfO)vj&=<+XL8tQxajiHlIzJYQp`uZMuByXcO8xxrEBx z(0A#{2hTm&^2oVGKRh+`;GaG<^g!D8hYHeP9=i02a3KvRIu?YLrG7)G?ZNa(oi#GzP#@u z%B=1qU2T5qPtQG&|Kw1?!)UAMrJ>FFKN$MQ!8e8;JUKX&wCKjZOSjyAuHe+4p1X7l z+A8|W(4|Q~8rtmnKIHk;(B;J!QRb$;O`gw!&jaT!J#=!YAoZ!CzL$Ooxn4&9-WV#_ z@@te?+qdb+pPuV`_#~%Mev)I(@TR98s9N-^uZ5ECopUD16RAr2-kdOvYrw@;crj)U zXQZkQHa67}7&-5ndXi{0PPW3Z2hfB)OohD(o2YTH6)(Uh91fyQJ$b?Dy@l!>XKcmy zkT;jt5xky;-MqDJUEkX~%V39rk*jw~}nyyfm-B_0qa!KfZdmqwl?TXW!Lz?C7PPi`?(- zb@dGg-LT^X#8FS*%|E~E*l%FlC!Q_s_Xd2kncJP-_qV0}XZ|+Q|DAn-kwtEnK8+o! zdduZG`CFG8zQax)ob4I;yBh){H8VU{o3lMvo1S;|pLjJe(%k0i@B8P#NYlSLMh5S> z_gM2c93y>SF70O(gYFYQ0)2<8zv*$u$cfLD_BR!w&i4^-Upqg)ChO2mO~;Ub2>IE_ ze;PdQ2kr!Z-`RiSTflb#*8^V!{yK0Q@GZbU0sb=Z9^e~*p9lU+;7hEsf7AWI2Z7fDzXluv{u-`(aNUmlBIN%ia24<^zzcvM z0$vHc4mcb5?lQ>j>W4fdP45D)!u4%jOK=^+bt$fI;<^~u%eXGY^&+nGaQ!K+b8!6$ zt}}5xhwC(4U&eJZt}o)6j_dQdrr`Pvu8Fw*1FkMyzm01Qay*6W?{NJJuJ7RbC0u`l z>tEpdYg`|}^_RFtas36Z2XNgEeS9AMSd8ldTz`Om?ZCAj^tqr1ah;88JFY7oBe6s5 z$9E$});NofHaM3ZZM@(=y5|}H(Y=rRkCq`nHk0-L40CyGGV71ISpV-*SU<&A)H_;x z@tK=yED3P*W~=c<@`Iy@C}k7^;GRY;QP1fd&;TW=kfg+ z`ks8M_GNs3k-jH!IR{fBSqFC~jD$axFw(p=;VQ|+&K{qCa0le9QZnU3rUJ-R0GSFP zQ-Pr~l7)0dvXK7Z?|eSv^4yQMqSTi6RP7`9{vdr%JXQO7d_P9tJ*R5FgztYr-xGv9 z(<8^a-*%5Qzv+f7@I|g8_}!mCXPc_riB(gy@vIat|wzG zcc8s0YR`SD_EY%2o4&hF)$YOfI{NNBRojN|chYx<7)#9ANmK5xx&^v_5ufG<-B){# zxks2|;HGAFJk)d{GSAI)ZJfG7QHDF_^!KV zqw{JoiT%%7#%{la^iHW)R zUJqXvUUE5qya;jeW(VxlS$S#`VJJ|EbHH# z!A2Hi-Lx;ebYwBsQkM|#4SE&mQzU&s9Qq#6b0ocJG<~&W&Kcsjc@`Tnc+M^z+3Z6d zds}mnpBE>82inNz`J9)*CxrY$+4joOZST7ubibspjYB^M`f^F%7>E89&{s?P);RRP z1N}xx-w}s?0Q3OpQU`|I&m(`mtoNxn^?m|+g{1F^L;pPJx5nXnv8+GYkul82vMo-% z_d&l`((j5x&t8Z6nWFyvap*;$-z(|I;?Q?c{VY-c@i_E0sxRr^h(mv#>Q5H+|1A#v zK z=xNMrZ*AL%we1G1ao6{~yLTPdzGZz^?=J1TH1GQ@`hG2T$c44B7xW~QOGfz=)Jw&h zcv9abtosJBM*YP-tpDN-Y~(jsf4-Gc+W%XuN8dga82NcS)(D?V=z9n2)ZyO+`bV&C z{aH%#qIa>Leec&>`>uYuw1046(xOX03XBY5?e_k^1V(=LT43bD4@&!A!`kch*Gl_; zjy2iO-(mRfKpq>p_?y!Hi@$O7U-*He{}))lU73mYV{GK#>r4BG9-{he-Iduj6Eiu&(w z`<`VlPv=cKKX$F<^8$U;K9$dP1o&FFDLH-{SMUBdY|jDP!`=+@=Wx&k`*Xwo68d7U zv>$uHE4i?=FLilYU)qi9INv{cF))$=-v8zb^q)R7d)QH0y2zUpz`p6a;rt}zCEmE` z#JV|W@=@l5Yu0cDzg{=IDYtY{-?OvMoOpKTnN29)MEPI6zOM;&{t=%O>t>ueOnHZ< z4Ezd!_nkO2nag@&-Q+V#zVf~TlwX80ee1H$B)t?EIe92+*yjj*HwpPv zx9?Eq@QHPq;8(%zB*=3sE1o&g-wDGPfFKZUp}ui0?wm-_my(b~*!j(~-9t*HyTB z^S1V7pcFws?UilYzF1G_mTY+$IkE_!^FNr`cj2=$`d&XWz3<`|a{Au- zt7(154}EI*@x-1*UrGOT-xID+4FC1I;F+&J`-wB}&vf?x@@sSY-uT-zrv z*;##W|8sWVJMima`02=YNB_HDWh1{^=jea$ui&Tf`@w~X8!5P!kpGsB41R(9x2%6~ z9x&q3;1S?*;K{()-wu8jcqgz682jJBdw}bKf5R|Fxc(AX?2`vOk$({QUx050-p63e zm1hQjz+k7iK8x$2%HhEv^4HyVX7Fzb-!?qBlW^6U!7mf88Xnw6c>5W|72xf|gXM&8 zKQnlU@a@BcHv_xw7#>7?Kk@7xXV6an3D=I{!4lv@xGp97&NGACk-rYtTd3a7;lX*x zUsrtw^?<8~2WJxAb!Koe;a$UnlL_xWgM8rK!-FZr=YI?jCgS?+|2Q+~5_{mLb0RLl z*Kw}60lvQ;bI>}(gEGX0QpD%yEiviic=2((_&8pC94{t{7g!(hRD*x94&opBr}B+E zzU^QctnwvJW=`gC&<&JVejqUYTY--kh4e}5ZFuw*no7x(}tf;NMtFfrS_#k6^X6|>-TrH)OL0TTT6ni;o#oJ)&oIFxxKNivpuw>wxjc4^Idnf zG#Yrjz@5vASJwqgf(>2W9gX#ziDXt++u0aw=xhO@xoz(q&23$)gI%q)!9DGv*4i$E z&-&V~+KoJ`Y}f4@g2lnQ_HeMFxs@0P>0^|%Nf7j-pQ2xX1l{Otstq+Z{PA=o)Y#b= zI)ENwltPUye-ze+e$@U^bflrNZvWoko=|OTqZl1~Ms1NL2X0nh+tRW(RNK+SovtI) z-qlVX&ryAQTW42oTbKWjz5(v#+G+ztO1sjiXMwqy-u-?kSme#FCQ;MK3<0WS6dd2HvPO{5Nl8@4?)1PwhMgF(Eir8PJ3e?FP;9=?Kmu_AMK>1O)|7SKHED+Zm^Ta=(t+P*-zT zb9)`kSu(Fl z5p82d*~I^&ZtoS@p4|p08z)|PxSX-GI4s+`W7GOgrM`9Tp$^!RNOX8c=V&YO*+W?! zU19k9I?S^^I(2GpsU_$3b+xy*bQby$JsWXGwgxv;-@ak%`VH&Zl5M^c-;&x;eUq=V zHMq>bs@8|5Nme>{!#xO_tozvg$agRGGCVuQZO)zPWiTC{=)3Yf`c7pixwFjg*K+*l z-s6TXqTVXEXm1)x#8@$V%f+r_GPlIb*cA)*lf+kY4f;}c-hzwRD;B(joweXX_JRd3 zW&;*nfL|@>{1>s)7M#ycSnxu2+=3UdM=f|h>$TuKcG!aFv4{oFWnl}>WgQmmV@(!3 zht*i{Y*uZ-*Re_qp2aFGcqR*2@C@b$E;aN!o%t*{hhC5HAmu8_D=;tYvvB(9XWL*g8X zBNA6j+$*t9;^Pw6NPJr2e2E7nZj$({#D0m-OWYyx6^U0%4D06dhb4yfa~zS_C-Gs4 z;XRz*tLT_+NnfO&WJe+MWeQCCAneZ)nEKD@lLe;!67I#^sq{@aaGk(jNsr7CI78yo zSU^zu9Es1*6PWZx^{>nqIA3CxFR)+Yi~@nF|5Uyb6TR7f7CXB{V9|aSt6VCuXdiUH zz@q&u7AO{&^hNxRmkWFt*hBrFc<}qWvs(MPkuD^wFxfpT&A5 zR{Av+`ICO~6wai{RsjP57b5N_;x-IS)VWfS%kLsr)7%tuyH3o(*S4 za!nt0xs$yt4c{X!d|xT^_u)2OD?WRvG~_~}>-+M0RbJnd*X5!ZgNyKaU#)?jqe@6B zU#*+hi)=GrVs#hMC@=Gmk5OJ`f8>|w-HsT06>PouTu5iryWzZ)a`7ojW6yRcbI1`{koP{SLZ#2x=i>wPvS+u86=!-@S@$%ZK~>HIrvm;#kf58H6Wh7a3tufz?EJ>89MAYncpOIZ%=$w(NV zQiaLBF5k>ngjt>_{zxK?U-_agSse6VjkZ4vzgJNHfm_Y;u;*wvQ5hWfU$U2g5&-;{ zbtT65c+)??&Ts-^9KO?*ANF!Il#opB%>5Pq{Mvpu-=AUhy(}*c{^=G5&;4@_E8jl_ zs-qN1A$h4iY4Du?C(&Q1h3GE!w%|wn$L9AdFU8nc>=5TSJ(Q5Z<)N_hdb)Rd$dhPb z$e#!M^??*)&>!S~xH}e0zZ`Az0{xA9jro)hl~eye|IW> zBmE%$*!6iU9ukRw#M4F6pWD>mxpa0o3GHEH4w=2|qJzhq2<9{1f5cDAc%nbO;Z!dh zFyrIHVaENl!*?Zxom&chd%cFfG2Y)dmPxB zWR5c)zw7Q+neli} z8uZ~u-~?UH7tx*)woI@P@vdis4VPQ+V`0BvpAVo<50+=FBLICqPx>zQM>PJu;q`tp zXw_bCc%wg}Fv_3V&+vExQUvN=s6h?3w^pthd+`&csw9|fq!p!Mh2t(fFbWpvplEI z%7A5P^y_T&*%{>AX8k!9ILnAKcPDqdh4W|BLpdsemHKKf&wORX{$^T$~=#J`7Zy5H62~}Z|Ki{bDfWB_B*c*>u#L(!^8ZT`} zME#}Uf4x6q;Sc*AF!@J&Zm{St+EdEbfSKZF=|AQllIKQE9<-m#s6lEUKSHDOrhY_u zI*(HE;OYG^*8s|AqI^Rd)}xi)UDO)uO<|YG9_;&3-Chtc!eT1nRp39-Fb?;$`NeKe z!}|E$JhVn$+}>qws|Up^r+bkJm{bzw+L} znstBR*^~_LHq|Nx^kDfS`7H(Zcrcx4{HI&wA^)ZDWb()Q z(5LgCoo(_Xf4R<9e~wMQlF4Kasz1G9pO^iU&TrmiGA2cj_KaT7S>?g{#IM(1kZtYH ze49KAZSveOSzlklUke!xu%S=P|3A_BFP>~&Z!WU&FRn}KaYR3KEip~zv=wbM)C6*^=Uni{auGDPw{%Dr9Y2_ zXISnZQ2TjCeOmv)zapys5t7f7Z_r7;d6Uic2FVBiIWNoOyymg+?0LF=h(F(dOa1kR z;V-?4|2EE_uYWk5+Fv}`oWG9%!~QSm{1+^?^IvX}5B7ds@n6dMuQuqUpOVSue7Ox6 z_WZief3*GIXw*+fdD!!5#b4+rV4>eI*_^*fKd{$}I{#7j?KbTpoy%+58^@Y`2)a*e z&pLjBZjP6zZ|YyrO??ZxsXswC^o8}Ib-u*;>>{;Z5%b@DI8XIHig-om&2i>0`+noG za1s_IjCrTv{?=K{-?ZMQ^(n2lA5UX1hR9nf;m+kxm*V_zmWR&o)%juh{wKX#I34S+ zx3_`rPKw=Q(xYLV;nRdbiLXEKtXzNv4BUhJ0UV$`&>st@a(Plo9zMTbH_HdQiWgDr zXT1SYf#_cdrMc;zs-hnYC#Mc*?L!`P5Brz*#CdNN>p{JJ+&@dB;I8U_A)K8?`cN4B zV4i0EW8p-)`ycZROMbTXf#v)g{`ZS+=5g16AyiEF;|QlUS#ef}6=#PnIFbAV_tcW8 z{-sVfiS&gJ_{sG!t)FSU_*LcKPQaaYlmD@B zTFQBC{Aj9+E2z92IBS?vY# z!Q4!K(J*dI1*m^WT%Va<+O8>m(RgP?IMo>c?86enpNXE+YsKzIEqD^yw>JKn=J>m; z=d(D^vX8Iz{$IOwJ&XHA+}~+GV(hP9@owVniT(A&_A|+Ho{06Sc3zmy?{6FSg7f*u z!jrgtOfu{xjpMYj?PZePUMQX_{lXs7bb2&wAOCEdz2w;J#XkNsZ1zIOZOUG-o(}Bj|K@!ZpVmJb%OO<2+ZHX&-}DJo}2o(q7!d7P`eg zEb|Ne(}JNt>wGhYJ>={5(DKpSLjd~H?Lj-QCwrjvwW9w=*n=K#j)ngd*aPed$+QRY zgoc4_NLfM)t7!KiD4r)5Qm$Wqds| z@~wEbUt$#>R$J&Fy*;ef?cu(U-W~>Q_F%a`Ie8YHFZ{QT4*|>gdS+Bu@$5>QJycuh zzn?u|e&pOJ24Ft1oWJw)tjUw-(D=yvrF4GI&r>L#iTw^I zC$sqdJib4`dTsw1?@gRe@jTsW-T%{hL&B`}pqu(hvF-o$@u2f1>weCzZ|F~7uRHAX zJJwg)di@CaCE3>N_Vs--jt?#CuW|Q#_VcN6^^fr&{qq|X+I*aISm|5pzgOamLjUeZ z6`kWG^L$9^-`LNS{#)$&*Y?v#xc)0F`cKyP)6m}}+x{ihvOgNFfBifL>%UR^as51o z#?!8U`+jVq{k8Rc?!)Lm^|;cfIFHFbp|H?@&S{BX7ubEqf;Igo3j0_4<2=&8PuIT# z_OIH1#^eOdL7^C;ckQQmsKO8XJ({=u$q`}x}Mt8eS`2$pyw z?LBos=~teIzF@^!XO(^h-TjIMYxZ^8)nfKN3HWI2K-D8UJW_ zp`|^{AL-`&>%#cuC+Ye<7A{EA^>g_imN+9djd@&nsvG(QE}Wgkaf-xCe7K_my23?s zlNF}>fqu_n2&d$a_AF=o`D8=hoFr2}zUo^bP(x2Y&GEc9C|4Ix0 zXwNE3eSY5TQ~tIM_Pp4_Px)_>;cvyBM>YPWKMWWWotMHNdRE)oTVrX@zMg>omM%!m zminUm(egao^iPZbqr5*qYq>0+K>A-+;9hCyo9-{@{*3cc&v8hn^xYet>p7t?)^Dg^ zoRhp-)=$Lv&%pgGGhX0%+FALkJUtJF`B*=HiuNqD_7Bf9n)XNY^8$-~!d@-*$j4WE z9s=i8xPNB0kNBnCFF6AJ=CcKm2T9tuFU_=%%e$bzr55?fJ_c0z3p8H&+ISrSwvLx| zJSg5PvFqFJUv~dE7S_ju><#`_ zX^sck->@Ms#Y3xq*!|HyKJmxH^bZ;joBzzy{RR5W)BVFf9yETYKcqnZxgIZ;hsqvl ze9ZG*>R%>>7fHVs{pV@^lRirS^t`VbAJd(DJS@)}<%{<(%lJnAl3}z@`Yf=}?c--1 zAF>~ePmMV~cKwrm=4kxw<3Zym<~I><3_AdpXa!L z%rnP}p7+W0_zZbRj|cfrzCkDd(EWk@BaT04&tr{_XZHDKy!a&jp<_IK&-eH>`Rx8h z_PAQ3#~TlQzOj!-9RIM-FL}DZ*!55TVU7>YFY`PVnmjZ<=J}?IUn>5I@d>E%&(Zk& z0p^qQrhdskDph-Y{q-T{lZeT0w14Dzsx|(j$K&^&PY}OMe}Micm`~<F?`zL?V{bzJMvd<^h_$2+M*Q{^X|7ic1 z=Ls9~^YgE9*JB^TKeTv7>tXABKH5L>Jcl)TXgtBSI^r+LCL!B6L~H}O6Rk%8yNQQxi~E5`dk9tex- z^1OC>w_IHvMmBHX<>L1Xsr}c}S$25g{M$XU3gmHy7O{l{X;FV6-Rlf3w&5iH)D@xBn(0v}kroM``rIfPxkzp0ZQ=zSu<+2I9TzC7g6d6Y9W-{gnq zc~E{q2GOWI-aoV=oX`22xL^V8Pfc97AfVyzewr;H{l-%9Ge-R0dPTTsL8V4t5%w>r zvEbqb9VTX|j~@)k`hyO_`E;Vp`3YP=@@cq`&eJtqlw-wy`n`olFZL-6`O3TV7}3>v zd|dnNmG+789`!zv{Z}mZniXcIJwU&BKSwgcfv`u|2ka9+P$m2}w@;#z{o;L397pe7 z9b+x1K}tY-c`(0TeC6W_eor1ujqyL&L+^L0OJlRbR1n3l;C)cwuW-&4{{ATV3r3m@ zfRy-n8kdKxf%^yUci-2Y5Bj@;eo^Nqejj&VcOmGhE?z&M%47F?e73+%g#CD6Th?#D zJ4eH8HScfTFn(V*hj~MJ;`b&jl0d&_^@ipr%qM|4#s1L?^s3k^755zsU^= zDS>_j^!$WH22N+Wp*-Wgwxa$bPl1I#KPK8kCiv%u7T^cmGI4xUp16N3=-yC4;$lJO z^5ljV#)LeAo*SA!U&^ESEs*kX>lhznd*5 z{;5;(;=R`wGn;%)vUl!J+}Fcs!_km;XN1-k;o1 zg85!%=);Nm)aV~@R(O@!KVXuN=pA+Maek2&*BY}we7yXA>v*lEFZ0YbnEcc?wyM951|Y?*H2DOP99@@)k}Y@5&h2vng+3;rQ|v7TV-3B>j#j z?@HVe6!w+C$}f#4Z(AIBJvM(3&lks)m)pB0AI8_&K|KsBW`7nRw z>GKDlPX|W)?U3<;&(8xYzLNhgNS5;}FYgU4PLcB) z$0V;lzYsq!=f~rP`~FQVe+{b<;|+OnP}2qh9P?@yKS8~Qy$_U5+wi+hr^ zXUQ)yW{u~|Q?>O5{Lkew<^#?@HC@^}uTSHj$HG!S321M=M<0*e&~leP|HB{Cr2Q-V z!tV=A`Ex_5iK!YUdrAUU_Qd`3tn{Y~q`#7M=|7TR@qFFCuwI*_txpht7N_a!C#=6F z$#{hN+rY0d-MZc>N!9(6Oa?7jRKh_Uq50 z`W&Nvc6enW`486-Fppp7O?%A_FJDY_KBkB_xuJrX9v{}0+5Oee;~8&)x4$@(##8wN z`QLJO#qh7}a0&Ncu5(`A#eYA|%cK9$2k)Qa$NlNNw1+zKzhxQHo?xFxfQvJQe==^b z-p~sEdwSG2?6(9j)Zp{gg+#8;NgyhF%MC3nkoLjHkNT_o6UB#>%7FfT*e6WKTR3+a=oz9>)qEyW=<*pJa0rM31_vlej7Wc>nDKd={! z&tg4Za_NnlwfD3%3{N+Nx9>6-fVrJ+7_O?49?2 zQlXGvw4WW$OeKHG1Qh2#S=_#}2y^>hGF8N5>R&qIQ&F~v_rR~GyUCxZKdQgvFPYdv zD|_>XyiVXE{`vL`p@zarTm zUY^E-bfw}gjaM>j(&Tqd5%wKHee(AyQ-%E!CjXd9e^Q4M$JpO^WqX`I=Id!R-y?DT zlmF@djrn=nQcf1*!{f~r-CpNs3x5#pL4NmCULOCT{k2shKC1EK@$`x%-do}c=1;fu zKPjJknw+1x{51XrEMoMZ>|gfJ(AN|+Eyn})g*|HM1LHM=`<9IFSWj?&G5N2nK%zp=*u;Bj;bTSLnl)Qz*F8 z_&VNuv_=kfULok{kA#M{peCE-Cfnb02D5550M#!t71S1-u+lMpX+ zxV@YD74eO8koB{9eKB8xzkJ>rnV>`7nZjR{J-aOOLccSGzw-WZ`^aJEjsBB-bpETx zBR7;O-*14-=hFFl*V=!w&N-t0b79|QNipiLDSu|Bv^Opv`HPIdGTzTjBLBdjYv6co zRc4HASJmh7BtzK?_0K&^#82{vrPld{_un&H_&d?5{PbDUpMlALrd%iE578-}^5?aw zc~L&$I$_VM{#3j$izm)6JDkn^i|2vAeSeW1o{>rY;qUzAbhmrDs4w}u`18uhf;}BU z{ZacBj~*Y%K4!3CV|);=XN!0$#t-_m?_aKy@sZmr+4oF55U=c;&ad?NM&nKLDSISL zva9$=@gtk%8}*4!V`bWh+qOUPSmfvPVTSZq_&eg)buxaDd|0og&XVzs%j*qYH(SOt zj_>P+HD*Zv!tV!Y{;k|htz*k9R6&MjCar%-7iRU$R-X(SB~o>(%WA^NEk*8xr)j4gE<< z(&smfSFZF|X^)fUa$lC~Bgiw4$0~`*AL#y9M;-1T$^Oy#rMccqPL}e^_?MC*`zNu> zYbaRgKS}rlm+3+L0Im`PeVK)nC`nb>fzAlIL?_;@TzcQv zT)N+mxh;kd-Cx1qznLCp;=GFfo(O59&u z*5xw)eh1E*r-M%7^ZGC2_iag^Ac!RT$K@dhkbl2}_!Cv){@4K$@w@| z@<=+ppUU$0MWB!8I{Bab;2h}pWUqC{n6Pj32y z{QVN5zp{-zoD_>WF+Ls^NoIqI9J`<&j=`^-uQ#bV(SK*y$^6ZL6m+5A^|+ra+8^M5 zKd79}FL^S{)8B?0tGvZtpik@bAxGjr?-9WM(0IR({=COMkoP!yg5D1uJH+LQdq19D zfMxt1&tCuR#Pus>{nzc!o5c9v8#ysi`}BgGG28dE3G2hZE`_l%y{z(2mo4CGjy!JELnTguZAFuv>6W8~TSATfo`j+=c==Mkb zZ(-jVOMdu&bi(%u=+Bc7zrhdAPuWKU{XSUd7ju=v-v>UZe1^Tm;}`kh{FMFVjpFAh z?|x})e(HI#h#Wt}PwVqy_ds7c>^ZG}pTO^LMSSx!`uk`!-%xntlMcqCvb)RK%gaB@ z_JW=H!2iXe$KX8<99{Lk`n!hCKYx49`>{WnaX$9eEj@{fno5`R%lUU*7W>Hga>{y)VX|@D#*8H{-;6PiN?#rmK@ZJ@XTL z{@T-1PSC-^U+W z`upEe)~m7DJ%?V6{nZ=q9XLqo*WW!DV~!Me@-ScNmGT9jiND7j*wQ;vJjw5J`n}FV zD|XO{fFmUhblTaw9M@yPSlYmnovF!b@A3yxU2exP{%nXmD2zx0DovZMl{sW7LrSM) zE#q=n_-JLQ+>afOnJXd1cGPn@m}4Mzo5MNtQ=WHODGBCG7Js^g*ouT@@OX6EW{*dw zZNhl;iV5hnEf}wUs~dICCaBgd;>GXb5pk>k~Gnt)E*$nolTOhBh?=S>Ope_4-~67M{!k>O7q2w>cb1PI#6yRg@}AE%T6fILce3 z#|dLBt#>EW$GP>s;$hZML;I%h#Rg(5-~dlipAHh^(j$US2YaRE2a<_@>3Vjf>9r0A zotrr}rJ~IMN{QqsXYiN5QIOeXg7oF&_odW5o*d=7vqhZXaS=7FnD@+x1&2~77C6$p z>)&N8LJXX#(G(p=ZP=M`b)LCulEfD^K!jaWKm< z98IyuLFY`}hA@=LY{0M=vaxwH7QBmRFmK}u3^-ZT8BK{AcU7%)R{$>|aXGVex`n^h zM!k0a4wzaLoz&~=?Rx?q7WItBKZ$%m9jj;9`MU#d7KKvA;-45uWYNTN_(zEUfa1?% zpe|oRAb~|O$Yb&M2D~il9fv=-Lw4Dq7L8zDA!pcKa2h+J}fnOvbGg zd;7-t8D4DlQNxEc`Z(=ZPSF0?KBn>4$H(w7jXqBM)uY?DjL(=ph$Wv&7X7OjrO_RA z3^S9N_zYI(Og7h*hxocui=|i#MvXNk84V?qe#kX#F4SLYO?8*$r+PGB z!`?0Gf;R~tljf|^6Lh+Yy2PWbz~gb$H^BGhQ=F(R1{&Cb7Vh#BjC(Stl*FelU zj*JwK`8w3jm)cVwv}=5ys699BB<<_?@g$&#{Qp-1ipd|P1Oo{lR0qDXB+xalMuV<# zM&W~Lov)>6XJsGvKUj;jvWlzaKUj;j*81pM9PRZHH?8oLr|MpB^L!6HJ7FBpPt@s( z-?e(Z)yJ$dk(V1DO!lMce3S*mv90lVlmGsqc#~&-bi7BeVB%Vb(vPO2xH6EQq)$5Q zr%ZX5WmMwu(-HO1`U%q|g1lO#Njg2umJ&?_J>nj4$E62SoL;Jrtz}$#Bqd7ph;zWH z(YYLu$82XHC7MKhNREVfa-=$wi4Wv(PKrYhq(+I(<%ma*q&lgeB!@H2sL$nyr#Fji z0XMJ9WlJ*ZnzBc{T!&n?Mh=wX=55!FPh#oy_oi<`NBtT zZxi|HM0y)zOg|F6X=D1K^rnsJN3J(*Or7z@d?LMRW2%k$2lj=yZR=wi*N-Rgh4^DS zdTlCZUTvR(HM3Y*ou_*QanH66V>K{(wNiyW(H`uH)U%|Rv-q0Uc=B4iUw3a3-H$6> zGbNedS&&!yZhTg|&F_>?c#IF-G2_zq(IyU)rRAij7t%WCrr{05WkgL2Fm(nN*5~q60k|gFvZ9G(@dg2456{wH?`&9qwkFlWt#xDqJ1U=tIH%7Hr(EV1rIqE0k@@MLF9xpzK*FH1)HxRdd zlYS*`{Q=__60ZolO+Hh9UPs*a)jkA2Z02-({|!3c#YOtF^Ec>~ars9K{WZnqA5nJ~ z@I%#j{N)q!7{W*6^2hA~nNG*$A28&5AwGX2j+~EMKVr~_ENxAT-A zmu~j2L(uK^Y19u0x@~+S2EAI)Gi>&$n#T`KxO_QQ`hY<{9=CndzR$#MKVbORSwXkk zhe4<4Zr&kvU~@6*9;^AJ#l0|Q#LvZa5>6xNfc6P{aR*jnJk?NI@8$Q18N0p4xJMj8 z;E9M{se8nu=mUa|kTgmL>t5Iz^8*y~$2o1Qn1Eh60lj(xI-Na?*FK#Cj7O)v{CITQ zkB>*EbFA^``4iCn6VT~I$I3r4>g;7a`sn^!>yPfgl|E7bBap8$0S^-y?+K*ulw8cw zL)Y}wdn1v2=8fGO$@lWFcs?9yxiGj<|6VyMvfOlb;6?~#f`Pk|FFH7p>fb19(sxSK zUL+dHw>RgFqWNs1WXOlj89PRK3_ngoqTlCH>JS&E+OL$DE>!NLZtuM0ANZR15|EfK z?xkKnOH!U2e_~rjRQj%>8+})i-kqg=-{WS~QTHl0yPvMReeA(6&VAx-4FcyH^rh^)1sAbb zEO-e!Yr%!=1q)ux1}wOMow49W?6d{vvlAA)kR7++1?*7^p3izMIFB8+;Cb}BEv-Lu z8GewU;at{X!9Lbx!E;!R1$T9+a6G2hpTs&W zIFD8&FEP&n}LjcaC%Id63`SJ35fFH2>S|IOF{K3DK@ zFPg<;2*r_j$(=uob-=OlbPMai_U1f3y?@1+?>IhRhR~ZM%)O?=)e8an4{Q*1nZwIY zOH8vc;j+$v_?vS`rVw&}M`x1vG?v?~*^>tig(S8>5VaY&r(S8;?F0p7IbgVUq zuJqL_aYXWW;cY%{yph9)iS{^NR8WH_yiYH}6?%-|a~hw4zk)qOPi}hCmxKRL7v`P~ zW0Z+}p1$HKYks{duk_nC%75>1`j>uo?cj4d`8=g3ZTV*<{#Ud~32PqnGy%8=@q>~* z|2qls^VOP$${V|G6upr;fOM0TN3Bs_64^$+S}U!$M3!Q z=BlAkJ5Q=`SPn&njc9|6>`sbk5#y{J`X`B*melyEEV56kAFzqhBVStL1C0_{HpyF^nmNxOeN*&w681#&1(`>*tiTFU;ZeM&acQ9_EkM)zcY~$uYUD2qN1{2YM(-qz5WVh zU^b#Oq~8-k@eO1Il+PK-`+m?4e4~?Mf?*dsr7@ps`@0y zhpk7k)bLMxo6=vIyz1?dtjaI!)fBFg2BO#FS<0VPL3vgFrLL;p-qP(m%J&C9#)m1I^ z9ouhj*>te1VMSGY>869*s+VcBeO(=Ot=mzr>ekX-JMXxoV|#ha##`>#v-`I8J4>7P z>@MHYx~qCeXTxS*x4gP|M^}AC)rywQ+t7B`o?Sav?5``SYHBENIZ)TJeanX8t;-t9 z!%g)iTlViN+2P;4eTBcVx(sbJqWtbVcJEwqS6#_1)ZVgPt$R1zS-Q2PzPR>|+R|2| z-P+P+?dyy8-d0<-RhHSfyrq2a=BAFS?JX6>cid9GcWZ6wn)VIldv9;pxw*Y;)4}zP z6;C{xzByQ+QNu3Gf5YE8?!ruOwa_g2=Hm9=ju+0k5Ex_wJoNzeE`vhLj2U0u=;pJ(~rgLjtlI>lQzn|0QeqfYVe&RsjlthZe1vtn2E z<~^{V{k5=}cskqCCgo_`>aTCz(Nq^#x9hfSEZ@3pcln0>yINcJ?=Hs&cCou+b4Nq# zj&9iGrrVb7xT~RJbIY!s#hOj**jKxA#h%)@Jk9Z1x3j9Fu4RpX*UrtIyC*8!%FDLb z?yTCkd*{jt##gjY{t~b6hP{aPC$hWE)bHw&-3N9RZ)}ch1Lb?WrQSF4vc-)P$k)L2 z1HF$m{>m>z+vAV9YP+JobiuASkQG4yWeN%kFjT{e+UH`G_aH^#8HZOT8Z%iwD* z;W64(ezuW#)$ZIMzc1^XWc=HGu)bwYr~z?0Zk)828!IkZ%XQWjj}gC&a!u53xMAnU z{S^P>`GV2kHuSfo?7&#IW0WhQa%2}fC+N!#(e@a&Xz=5D?kbr;&*fat)f=y^+|Fy| zS2mWdsPUEUR$WuM-4)lyZ+Ch4+W3_>U0b<|Ym*Oaj%(sqUwm!y)fZ1F-zM&BZG?qkqd?a{Yn0)U;cyDef$z`EG~5p?dTF>f(+2>sm3d zRAbI4#$2&}^If}|`JblZE45&2OLHi=udZ&@>V}5e5`W`ze@Su4ij|G^tLhq8`itx8 zOX`*vFAs`x_04-~>l=gh`@33dJ3E7|CBfEkaBpMlf#B_pZJq6*y{(O{?aeEfFVnFF zj7zkXw1JZ1V2OWKad73b<;#Q1gR6tLhxRv$e>T>(bT$T2-j?GJivO0ObctM--aU6; zW2mjMC0-vSMRa9LZAa(9=DY4{X*BS5fjgHKudWN01RJ`#I~wb`BBVm}stI?r9IT)^-_u*4K8`Zsb{IyKdhQEDqMShl35xt;9G;AETsAf}nry z^}f45{b}@NSRFChMTc!Kh@Fa0yEe2)BW6zfR6uEyY>{AlW9X@8ON zms$U7%9pagn3GU3BVm&`SRmQ{Nvr=IwK4ld+Ih!V^--Q`r>e5}zFzTIpO*1C=a<|> zFVyIUPBi+h8a>}upX6%M>26t|W}JSZ2-SPyL)4SpM_mTM%gz3spRL=8t~e)#qMvHe z%o-$;LpxWlc}dWoM!y`|sgK&X5!JA#iBe$l90JAjxsD7=M`O1%kz8L#I=$FdVIeh~8#3vv#mH(uTQN9_9 zj>l)(82LAik-vS6{8x}q@|;Kel$3v7H-+pO^V>$ovW!=VY@+|0aW< zvNftQMb?{Y%lFIt8OT??O* ziTEqo2$P*tvh!(@`Lk_&Iy64|U7%hqpTB!uu%)Ljc%pXo^4}RF-;433MCJAVE|>Xs zIakYkdwEUUy zEx%G4y4cQZ)y25)bRXN35bs_&y7-9_I%YBH_DSMsJ3aIqNMTz95L1;uPXn%^q)s<`6}MO zV#`-?Is@%dQhZcA_S^DRJXYu5s{DD`{|;Mu6^~EY@>M)$V$t0htZQzo4RtfD%NuHM zt=->N-*gLKsBZ}4ZhRA8V%fI8RS4VM*vTf|ILS>#bK9QwYbbr7G1S@I-X_)1To-~)${V|c zDp*lldskyo`PMs@bavI&?-QjqLRZ@xyGpydLd|viyBaB1lt)ENZbR+*#skgujh1|& z*~X^eo=|OT<28u52kmu1#$X$jzxCD~ThLA=uR(Y~X^4 z4naI2r+}g_dJnO8G7A`cr44QM?G25$H@7r4d~g|9Uc)tXy*9MBGuVkuA1mtY3U$@i z;o9BGuc7t^?CMw%|5q5X7`-jPc#3nTyg0)9rWa}-fd2MRJeXP`WNtw6*8{e#q`FD`)9yvT|8 zXCXecBn#l9C+~`up6bnB@mwEx2Klh*mDH7m%U%s zxA(t8)~9`{p6v493wpfq3tSbouE@&N4O)t-`fB{1M#J{DsY=Cf>0jK3mx>`V=vRRD G`u`77k)@vi literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kbe6ee4564892ecd24074538ed0d17f2d.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kbe6ee4564892ecd24074538ed0d17f2d.o new file mode 100644 index 0000000000000000000000000000000000000000..bf848e888c9b06285164c4f0ebec5e37a965e760 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAPh6{OzQzcG9V!k2sX2#B{fK(N1v8t zT85kmk#&s0V8-mmJ`-oi#FM3E$T&NhaWtE=kvBfXaWI25*^Eyn=Y+@%BLVi{j@Lnn zaqsunqx;^g1_AcOhNYnU-unN)s{U8i|5tVE_Lorp1U372^Gan8O z?>sp0)YDH7q|RR&n$!H9@Wpv2!u#L*Y4|DUo8hLDm&3iQ-veDiD0Jx1z>bp#2lmfD z5iUP-GThX9Cfs}Q&G7I;m%|r#UIxvQP?_`4z{UAb4D4Tvehz#mJf{WYpMEo3cHwe( z$9pv1ABRHoG2Yrk=;vg3Zp*jB!@!qcI31pI>H>JDe*P#leBjW)Q}dr52%SC|zKB=T zdvAv4o;npSbG{e8IR64@mWSrFJTcG|cUAJM*+H_S$HodAu#u*BSJlC%CTX6LbGAvngxjjKf~ft06ZW43{}Mj1RbvIO z!v{2taFcomb;qOB?igbQuL73C+lan?2tWCtqB!*7rXu*AXXM)M*Nz+gH=aL!VAJL* zpCtcNzSj@mdj-CE&-d=h6v5}!4>LahiuLrZ)$hqQ4M+WP)Bj^5B zzczG$_0c?HTK^-ffR8-O`Xc|r`YykskGxaN`p)0SM*ej%>${M~MlL3?krdDtVy+GC zOXwT=C)PLgfOh24|I|k=y~B=v(Z_rFyKX6IkEyk0t|_7#ezj9r?EhK>Od5 zLgzN^eT3#3cs_D%_p7G4B-VE!&n56rvJ8P&8uML}2{K-Wj84czvL#PqBR`?J3i*t3 zv1E4519>j~?TWr($X6!i8V1k9;CUE4lZ+(aCGa^8{d*yk^Y66KyO3k}m4whRe5m>j}~LL_b6G^zkuL;*W7HZTi)ZD>Uw7u1wEJ$~;df zbxC2!`Guko*@kRFcB1~?fnLu;e=fw&K(eQAC;I;;uIunB$9*@h2XQ@#*Z;=-8C?IJ z#)rM$c{U+rgI$&X*~YVxcc6dQOg-d+eJ;Uo$xgq94anvfVe@l~S>GV+M|My?{r-B^ z7cORfOz(e$Y^v;kyIsn-qg3X_`Mx$L5DBcu|SAcTi@By>}_wa^)@$e_jWY;0=4zt z_KsSK?eMj>w*~wifu_Zawzv3Nyvf72REvL$ zmgGI0zjRq`Z64$la@_@Dv^RPKzJ|L;kbtk<7kCU3!Ab#N^IhR>ki&ad2-4uI>)c+u zE#PhO3F}w^-UTKIrdd5L4FRvekxx2*psk~=qsx!xgllVU@9?&EP6{pMMZhy$v3|O91VJ5GnSoo6qz4%?zF6GZ;C0jKLQ}K1qp7Xc-O<+8+&<5ZneFqo z``r197c80Q-bj}P17yS=Nu!`I?&>1^+CH~6+Swffu}Dk@g5uidz!YJFM7 z+S(O08&<3>Sy95~SGX6u=X(S7jqdgqoCiGYbz^AamCjf2`G2B9w#HOu-)4th(Dq@w==(kzoxuZfGIS-K*=5~MmcCLLt=njHLtSTX zQ=QJLjCksk>Na$1n;K#IY8*4w+BEx|RBltUD;4`}j1Qx@l3}Y1+?Ip4X{pSQ?p<)i zO1$hY(7<5$&tLZG!pm$~8<;)C!N|M2ZMa`5;X84()q>Y{AW&!=x_&IL*W~phTuTUn zLVhuNwlUVTTZ^z?0Z9fLOk-JHLSKB6fn# z6&w3SHtRWHkFYBM5Yr7#ws(J`osYXYJC&VqvmbQ{8Q7D+qraXYCXxiv_$#_}wv~9% zyKRD>`%podiPw22I})(l(Sk+#52dleg)BE{yRZMBC$NZqD2cs(n9T|%*nTtRjh(zc zm~Afz>NdhNMtH~RGdXv7cdI?3m+4BLp` zq~Ii2k=r*x;RSt|5#AMWP9lC&&qb2hKpHFfbM%iom-h|cpEC_Le;xWm|4;5`5weHf z0ecQsu={rtEfvUjIgNd#YkY*8gYdad=tm#FvAVjN%lk$)C5BkpyPM%Eu zHlFKW)Y;jWbfurttglP+!k*|K_Hc41CwqXOf&iO1iH|SpJ-ZVk>;nME2jG7h0b7JT zGO}3US9EFY3NfGw|5z2!)2Pa{k2ycCfbezOub~h0uLo?*NqABM**7?o4V*I5AYem+?lj)!oPU?+#t*`RIN=W`3ACC>GMZ{rGoJ<68(cSuSOzTjDC)j z^cRuxt?0rsK@(g0vxy)0S3%&x2f4j8F9B)Tm;EREG1MfEB-KY!v3gD*!pC|A`qWv_~1^%nRX9DjL_}OW^|4h^c zK3Ut_VNB^9kw{3c9)^m*3)ktI?w|^$=8!ezORCTg9 z`~&kjm(N$4&%a1xMbB%3octH~(&v9Bn~VDB{>=37*IcqE_g9QZ{k(>dOOx@b@S zXa`>U8~hvkof*}yvNth6_I{i6P4_5L-!z_G`R8CU^wS~maz2mNzeLOS(Em(QxAG6t zKmSCE@EEW3`DLd5dUj{T^j+6#(KpR+gTDEE&EWdZkotrFXGHZ~myPwpWzo0G71MVG z>B|+H-+X?cz0^1Q&j>-^-vw{Cr8(Y1DqyJLyIi z;16{%opwEmg%id@el}WuNgG?AWPX7B4fAI{@uLdrCw9Spj{`vY0Q`sI@5|(n)cymK z|5z-(L!Xex65n-0e9w*(&yfR@Nu+-DKxQmn4=&VJ1?&{L4e?#q$-X?kBmaRvL>rC{!%bp6mQQ6jzisur|E0NC!lV`CnDW4;*(I1;VtVK zU*GI58BfOa?+S+Xn)`=;EA3CF?}c()u+mkzI-K{AO@E{3|&#>R*Eku^-6Eg!m|+AENz+ zgZoc%bbpeS6Q#dE`wd4jw-@;j-=DZSeHPIJpBqeeMEwc*ZF<0+xbCERSub>C? zqJ0wOJIJSP{ZD{bzP_coke|7u{Xzcpz;t~I=p!!7=b?(!^gy;gmFkl*fiOPC%gI8W z?>EsO?3+pM;-~olc?*J>nbCM=<%jt1!Fbo_ht@9ybRrP`ZT9!WG@p#~0qc*+-+?#H z2jETq4!mJLKtH)bv;V{2r;~koz5{+Uf`<73c?lo&|H~TXQ}8boijObiONhw7m>YG{ z@A-wOBVC{lNuG1Iv zmb^rg3;xmrdMcfmM7qiThqaZ_d_mxU8syHv`1tUz9`K``c;F-639lzG&JO`z$9&o2 zz)=B;4eP~ejK4?N!(aFQIQC15x3BT@J*A&pmIwQFzJ8Fu@%8zv$)48y%#@G7eg^{# z35v9z^*rMW`0G63pm+%W#>xGdDc_-d#!muM{z3VB@-s|WNa*W!@|nC(HJ|Zw?v;Ld zJ`?r7SU$tgC(ftYZJoQQZ2{h8Kttk0(P8}%%#*HL=3$9fdoZw+eZ^&0b) z@X_^p%zTBnapQa? zi{~ru`1uOt$#PSCkLD{uL%uOOUqL)|XGQasA?WXu=ObCPzVdj9{w(4+fc?AUyxT`F&~U%VWl? zW2SgzIA4eV+)liT?!U3VnfKcZ@%h47`|Ukk9JN02_)JG?^m$PZ#gqH^e4jiKQVyoM zOz{cpr)ht^jp9=x-=Fh%hWsdfkDcdF*bk(!*O9;ONl2u4o-{GdLHSTY&`#kxf#=MVDzIX^$;`Su$NK|k5BA3u!G8|;($ew_Qqf2RF-?7Ro#VgD36 zzry-IdOtpO+KuDe6z=a+J&?Si=F>i_qUk8H?iNI zVvKJE!3jFU2Uuc%Nb@6e!bDg|?7y?I|IV5~pU)Zg-_vNnXoz=C+K|Nd-%jTk`)?em zP`RCWS8*Hr?{ut|U{LH=(cXo6biavwZG!Q<3HS`-c@+77Cgq!A|HD29(vl$e)A=1NLh+ElVg149#r(WioM#x$!;gXJCeFi;<2)Rc-HGqxSfw$| zvcFE#vEOH30|EDE>_2bFA3hIzyB|_k*{%7%DRNV>doot?M-24fBeet ze;0Y%&Q?^fn{ixUSDmn~8oLpM^qwdeD|(JIQDapA*fftwyrW1D?zC}o4E8Q|#@B(VIP;eUKDk}n{aMT!;)bm5c6CdO1t6+rs zqL*mLc^hZt4b4~#zc0m#r*ubO=3I1@ZN=wF_?~JC62u-Y5iCbbJbYBKuK4XKm*>SD zACKf;6fh94f1c|Tg(p5<cQ^a=YT=qjI~2-0}K|$^}M9RwWsR zL@`Ka|5PNikM=0Cgo!gr6Xguj-H5Z%bI{DWO}BC$SLeFmEIKMWG1^|}y0El0CEkn` zdMZL2jaf&l1k|9%QF!8pbg)j>*F1>1=jI;NLz}Z6&0_;r6(c_aR#>N}=sAwW-|>|~ z8|7UzwpvlFD@Kf`>uFH+8-E3gwcn>(Wp)|Hjc41aXM!#sURi1nJ!toV3+es2YF^mH&A3Di4r4_9HDduU+Q)lfWyph_%aRS-beR zg$-+$nf`4{`(?ISN0kRuX!R`W z*@imdsm_(DjY~y_M>_zUHUn&|)ASq{VzXy2=~zQx#!8R3*sg$bIN7UP^@2L}Isx&T#BQxvBZjc!8_hLQt0UtB1(VxW{`Du*#*+tEG+<$i5SnWAh z-8jSU#c;8PjHRl1XNt$JK*R88R#S6nuPSdU;i%~l{ z8FAIPCJX&R;u4JMQet?suv|8GAUd>W%$xsBK7?Q>}D{SRF? z&D6xvr-fF2;?Z07L#BLaRQu?jlh46~=n6v14~mukPOF>r6KCbK$`g-XZSH8r1^pep zzvTRz&+JamOZNeJtwj8?|Hez=L8jdml4KJvD3J()G0^*lRO>_N`ro z_t9ZEcH63qIRgKz#3Me#q9a+$IlAz16TV-3KTd4_WBrdl8Rrrh`qwqyf4Aa|zISHq zU*Z*-|C!r5T`P0pe1YbZC}-_Bu!-{wBkhLIFj}ITTdv)!1OOk|K%jFt@sJNPsw|08 z-puoV;y&AFi<>?5Le7idePLn?``+B&{IwfYL$To`DEiep)RlH0 z0h}MqbE@mV&Eb9UqahQ~at9curS_2}YGCF{NxwJn6-V6wm33^`rcy z4y@2&l`3e?i+D~~rUFN=(dTqYh@y`Y+~y6Jrwbu6Z34D*7W$6o+g;G&GiCM`?)cVeOOJRmut9NEJh|(vn}u&%{k^ zd|$RdVQ5RYR=tz|PzU#bd;f}TpDRI1$JX|=pL-%s z`{S1Os%LBa8A2g`k0OxQ)3W^svQFdEcTKGA$?ws{DE|B96mc8c)2(W+?06U*6U;+M zLpf)eVrqQVPTBu4G`>-LMMv$4&f1>lIjO+nri_teqIBnoE43$@Zc&ZC<4*>7pT$v# zyjdo%G^i~64_ys)0)H1?M?6vgrXt&WOhAj10Y~c#@n2-2!z288`I({Pc=q$dG4c0x zWIyTgqTRVa7S|>Bi)$7xB)^FW^*;rW#z!%sCTeyH$Z4|Tt?f1nQ|0}Y6uB4># zsaVvduB$&ATv0EBGkgSh}&Z ze$nzS8n0nfd0SE0u9CXN8}ph=x0gr9E#1D>Te!S!Md|hplBRTX>&BLv#T$3r6y2iy zvFKKAX$dxNSzG~qw3L#}+qRJI^4m%lZC~dtTARPQW&4U9g;iafHx&fpj#;*=W;6J% zD_!|`eOEyfYai;CB*EM1$wrF2E-=9cEpEv0xZUemc{Rk^>RWn&lg zSGF#HO*HBa5X|;>D*#G$ErX`Oke^b}m zmgc5F?an&iQlD>0!O|tmmM`+vH_-2rEGbyL%-4|DkpJ+uMGduruD)rTx88TXpIF(z ze_MrqRRzCxayym08$6plfANwfOBUzlE#ovRz0K{uTJc_7jUSZpd0X;=d5rDOxuYMS zobyNS-Cz0=jFZ10pPNW6#@gdmOxrVEo0gi52f5()Bb5Ph%{AQ5!@Y5j`6Mm4R5qmE z?|D`~YX7f^|IP8W)NeRn5Iqxp11At~d&Z!rv5p$(UsLz1oX5s}Lw~_8>qPehg~Ro< zy0_3x1N^+g;aZ~ZOPsD}^>5e-&;RgO!l_?g4ES6_Fq&<_6McyRAN^IN)y|j4(Eu-q zm6qt5Z-P#BZ@s>Vws!WuI+8G<<%8o)Fd>Ihb__)~4tCJCkf zznA>u=~BF?dG)O1^JNKF^XWC&Pro#vf^PymT@8w!)}c-XPx0mOt+M`)0?fXJE79j+ z94e>8g}sFLco`g>UT(u?87bScu;KnWE9s^u%5KX?o)UE%`lsidsPdcvZlphP z9r#rP+$e9Z)Vr0>*%EG*w_h5_D(~wOZtdr15^k0Emxg|f{C{MC8|BTE^M}SY#>l+u zz>i8e@nc-go|Ev0Ebx8pRM~#~&$6alJ8^;YwxEfc{?~!opxZjg->$sG+ zP@&v~rj*hh#WgAl?%_4Y8s$~NRor;a0$1^&-vU>0V8{YjORqZrRQ;>Szga+?wZ)aj1Cs*Ks%=mRj{u@v9bB*j~ zcOKMHZBuI#e&(^Rz5UjjwRW}$hMIisZ0t>r+&b`|mmHh=F<+n^KMf&8ZK?}E?owZe z5EnmOLce2Ky0&^geowA`r=VI1eXRF&6n5bEC+j*pd;}HrXlQ~qcuRbbHP!n}aG^b4 zqx`wMv02=P@#yC(YFp{&@m8NQ&fd4MhRE81r))pj>U2tZo+2X!1ngqcE7KF^zVuW+8VqaUbcY03x^a|S`A4` zI7*N#sBiDcUwYHpxI^~BH*egyKbU$u8+cQjPrjQM^RZq1cIzGb%dTR9{e#_;W(Ou{XKv*zA?9^mM$55JybCiHNLt&E8A1w#!{(*cY_fc8(%>QO*~n3SlKEl z6u*kzTK@|<{`J~_WI-stRe$La-f9!Vtqv#(^VLebL$)6!zij`ErGb@yAMo+)=aBw$ lMGjB)BUe!ERStas16#+YCgf0 z?*0CHbl-c`Ai$p3Fcfs(TmS!8)&Hvc|Eg}?{_?Vw%XLj-s%Y#L7ON>JW0}3Ndk??$ zuvAdGSx?P?t?uX9ZrlH%7ur5EgR#Ryw(P!l@H%J9p8GGjp2zE7agU}G6)jz_?fOCJ zx8vB=4>#zc52|dTYc_WH|JhjI(GGSPv`2a}Skd>gJ^de+vbTPvv%@n|duq^Ly^)YDH7rp{j+n$z-~@P&ED!~5R*Y4|DUo8jgYm%@E3-veDyD0J}P;Pw*-2KUWB z9Ppu}NmxMwm z4h+sc`NUudJYSgq?eM<&Z-OuA0kZeul|1JQp$h2o!b6~e-cl039S*(wW;kW*$?%-> z7s1!X@VJ?eghmLz9K7y086J7?Jlg#U+U*~#0Bm2PglXAVev>%0EV`D`Q*htfRYU<%rlQhrOv$jcngxjLCqNx3W6ZW43{}Mj1MPo&;!v{2t zaFhCmbjPFA?kHnLuL73C+lan?2tWCtsxRTp6}h+zSnaD-#a>n?;W-H9{Hs5zX6N?DSy3@4_@2#!%(Dj@>|0X=!Y-- zQ^{4gouys;Rms(nmzMSadM)cu#r!xsi(MW5p?>vs`1MdWyLz^C(&3>7?eN*Z)vpcj zXZiDXNj#IF8?<|^bf%Egk| zH4o&u@VCqQM<8E?lxqY$kAUY9@Juq2d>6szIP~v@OwPa4LhnM3kyjEzBaq`=$YGS9 zWEgoRF*E{spo{(s6WHNP(8z!v4LN?e{`JZh(yZR3F@0zKHT(HkY_$}G#*RTQE`~qx#wvhD?!G2^1<p`0^px)nLb_=|Ugmryo}phpr`+Tp|12 z(}!6f&%t@%$NWWaw_2}$xAq@2=3rm$P}T{y0xt(T-+`+KuOz(EnTv)U zRd)CSZN8QcRt-!pVRg(rJr!47W47@&O}8;-ABQDR7MqjVar;&iNs?`; z7dL1Bv$*OzSfYn5VF}=93Q;cTXzJ(;bb9M?r>@Dd20lE4&SIc<%~98h!O$+v>Ik zysbWA9V@`Qzy!fGYoMhe;Pp50N#_r=ceZzS`_Y_m?QI>M-nP!XyTE-Ly#7{Spy94C zGfcx}vD%UY<hW}UK;W_%ic}P7Fov%xcM~X&fmwVMS3kZSPR;>Ztd`O=CQiEw#VwCm{rSG z)d@^p9=2-D-VPqUqj+kG{N=97$!-*x<^XBKZrNIva(JQ@%>I6cu$Bl) zt_0FO74W^Au4$=zdU|FvM`TaWYzMz(jNtREoyfZ4+5U9^L|gkAIs~CAUPthv`s(!+ zWfjHlQtbb6jv@*jk`U=)ytWd^-x-82mUcCI-E_Lp+~VzQZf|pUwzs!*%yVOA`@9`K zcYfi5Mf2S2>9U|`DJ$t}ZfSIPba!<6THUQ(9i8q*-`3_fpL<zCE6t*BaE zw`}9OWvk1Um9hC%?n3u`Z=j*c-O-BkfQP+q3{AY!`3gS&u<*KrJps6Vp@Wep&$Jc z(^-`fPkmC|hHh`S!saW`eBvg2;{qi!JsdlGo`*Bit{k{}v?S+~x%5HEU< zP4IIcDkwAYI_G4E19m%Fu*kr{G&Z!5qPK5Qh8zxo5te=g)R(1V{eX)o}D{)nFIFQZc?_^lBitN>&vimS`uO#=wY6N{H?kq`WSx8|;(&h8e{OK{Wb(K1 zT>qlZ&b*{6{gh{YU78p6ME|gdlRG)t1N;;P*u+VEd{OV+l@MVc06;zf|H}y2BIJ>g zLV;h_t+C6*fF}H7ML$;M`e+pUd$J*%RGtE~z&0ek?k}#h|x? z9%~!XA9HfN9d&^(1>Oz3+k|(cF7V5Mp9XxMzz>V|(@+=quL7S5yhq??r}6$XQ5X1R zZBMrgcr2ncev`nvP#5?FUC6UV;E&6^Fh} z%;#J_Uui!7B8`ZAKJ)5BkL$)4O_F&_EX79x@JAtN|*CdEJQ zk3XmWSD5z9rbm-}s_%dLjG^>wVO}$bY6M(0J3i{~RKFPLKN6kW(uP zOiPIRE7pUV!D-WoJ~8HR(=w^f{cSJWyUpMg05B{GK)pvb1)(e+K-!4~7-&Le9 zS8RUs`GNLQ-{?Oh2yH=t$Uv`He`$Vj14?~mndPPaqU*s>F)PoSZIW*+ef91_K5!OI zX#L9h6Uc8f_#4UZu=tyu$4@)Se~Q*KyT~7CeW<7PQnXj`ZD!EH=^ZhDb8vl-zn#PT@)NYp6t)ZN4zLc&gC@~4|;cD=~wZ~Px9$8 z`Es0ym*TV1`#8U8dJ)ARzYQj%^_%98&f}#{czn6x&lG#8(Dxq z*v)j>^&}Qg7!Uc`X!#{=Y<-gX0rEG@pZUa(DySdd3Hv<`0ObSlABw*(lS5Mb4@mx_ zvG@*sLLN(e*A4MKJ5D@D4ooJI`qcxOv3NbSP+JkOQ{*p@^7xMY2mXwYj>(^U zl4yM`PgVj5dz$@^^lvl9XZ!J;$TyUH^_1^8oWHvE;Y5+|ko{63>>LK<{!8|`v46t9 zdA{h7`6bE2i^@L&HV0py|A)$-l5KBeIaA|dJ#%8KAo6Lxo+aBT+Hn7XyoBfLU51~L zyhXjh^8tkar0=#R^ZeMs^J9B%(2+d*A0ydL@+Y*1{CqqR@Pq{FC}i!EjN$JtH^{bt9gpuMwYsx)Gm_E zv%6$G8PmV37}jgOL=Sv!Fxe6HC*-&30e6-gy+??i{Ass?ua6in zjp*S|nMQxh4Q9EG?TdE1-8A1gJ@-eO1J)7#i1Dd?)E_@gW6y*+l10u&(ulvJUeJs7 zNtEv(pSBG=0bcp~mgYi!=8pCU`O^c_^(ml_xG2i=`~=ivc*x8BmFolYQr!g(gnn$8&$(md<9u?7_;R7j9M4`v z{R-k4grJyBtOlbXf7sK_&;W(bN3j1o)QzQ96rJ|x*dL2SOxhp5!Tp`I6Y-bG^(=Xj zBp3Xp2lP}rF^Tk${SRr&qxpis|1`**f${O-U%lW*JO02&yc1qeV4NQUypH*@+kvA3 z6dTryQy71@u!q0?{c-G<6mMVS=X**&w=56#>wNtnf8*=(8IwJ&`I#vnf&C5y7!nj| zKkIqMW$@Q^+(GdW{Ed_QF;l)n`HY_gru>8Q_vB}ou#nK#?c_6gpK3ni=iDp(@_Z)h zf3bXqpHG}iv)?d(L43Pz{_>~rKg74sI)53RuQ-xl_9Fch`3UBl%R&Cg&sX4|=6q!# z@W%C-)+=kiVpx9>FS4*+V>uD=9{Mw_=UAUj>o@9ISg)h>Xpi+Mw%;1k%$mtN*2#o z-0|}j$dl!!_#Vwyf`)u!biRUk>duPhE5p#=C(lQ+Xnp1J68%~75yVSVKH|W5Gu+_! z6XYXVV!@~QJSsoBbMq1MpZ}2g$UIa08omFf>;gsZzYheNogh5_!1;Z7*UMwZtD~lP zWjJ4l|J+WzitfL$zM1#i3-S5FSo`hWTpYDN@%T(fYV>(g4#kuE_!)ddy_MoqBHy3$c!vBaeYc(GPuLHnvDcBm?oLRgc%C#d%|ZE4QP57|I)UfASpUR% z(a{YoUOYp*NdI@JOlsfh-b7Pie!rX?$+a(X}@8PUl`xC z-+;bN@r>im`w!E819($>8|L}GK%A zKA*$qjk_ks_U8}s{W(8B<@xp-3qe2Gupd8!&l~KM`F@=H$A6*yc(saOxCGe2bm`SogP>zc;bpo??t| zMZpO=!v|Pme@OEqbHYSeNbJ9}vH#ARK%dVU_TSTJzi5bePTG*f_TNtD82fJ=s8G3` zcvp2B`|otDmS9lqSJB>udUQXEd~Jg9yb1UW<9QVMe5f_D|O>`ZVoF#?D{=fb;!Qf;oRhep|lp*5VuTBg6Lt(%33G zY~sa`&(e}0_tW{EEJE>+zhV8s=f(WISDa@U&cly_=qAp?kKsHVls$>><5;CB&9c8v z)3M)YUjqU6XY4<3$R9orhd+q(^(^}Qlz$$n(+B2>R0qGSEcTPXqTfa8CpGH1MfwfDX?QY6tKw2iEh8Yrp^SN0G0-)K~r6&42d~Kd&73SB)QC z_?yU~A~w74sa{vDyKipJvA#aW`t7yPj=cZRb&;=eRLYu!zwFED?dwZnXMX(3?|&D0 z+s>BNu9Y1~9N@1IKnbdiv!`6)5xSTM z>R4UF-zqx^4)>0a3&9(`0EKVxe&Em?=Aqc9F3dr}cS8+4lF zvG@b0;>XuGh|IX%40_j-gx;fa(@1gfQMq05@lm2Ac?=s9TS+@@POkE?TCa26dEofvH|bX{EDmJ)Bq3O$ve zjmE5_RRe0!<0w3FLpoTa>#H8b+;eje>ZQ$Duja7))gbh)Acl{`i;K=#oF)Htung|0E)IVP$(EmnpqD>$^{tfHo5x3grlhPNbmf3H$k9civ zhW9gkMICFjo9St+KCJk7eSUV_Sh*XkZk&m|G6r!k6K+L*enY>C*IrE$np$JFH_b~u zDLOCRf&pv$c-pqoeBSFuo#}C6kd^;<^ePXKI`$(iHLqRf`jfybvxv3JW?8%Vw}lOB zmzn-;OZz3rnvMN=EcI$j9og#j*Dm8~FMMyT_;#acK#n85e+@m z;i=A*sf|lThDSR9n>GV%tkLuw7hWXwt zqvoi3Y@cD%X_VsBb@pP{V<|T`8s#Em=_X#TLY&@8F5m{c1|91NWq`yAX@qnzTaQlIKxBM(c|a3?nn^$Zn7r@b_Xp^Z_3;t+*s{7R^2$m z?!|DihK!}NPl`9{{9X(-) zxF!qzLE;jP9E(l1gd20eH{sqs^ba#v?o|DhPgy;%F%MXf`{!Xi#^b=?0v&s{XUTyAZ#RdHxy}#uA zo6qt-Z)ba-cm0~?lX1r;Vtp*??H#pa^T2~*F?%mFMm;ra-_rHfW!P&oe)g?hf%nm2 zICk5rj5z}Tti&Te!=fWu%Q?F6aT9((d_PWX|6~1+J{ji{82Z;W-hYqcjlOqg>|f#) zn*W*G8eOY!;e3JSlPG8GIIxNH3?uD^&oEk~np>_tD+B-^*+8JPIPs7VGO8|%Qr^t- zf8svdXS16<^+L{z-+f_XEBoHu-~4u??XPCdo00dE=|4V{mRix0vJgXl4xay04XNGu zqq@9Y*rO6W;rTx=FK28^IsUdBuj@VkhZID%pliHm`w;w>^oC-?Nl^5wb*L-tJ_0yD znCDd2f1AVm;73CyqNN-@x(D%`F7DPQdhwjD3=@nhPhv{hn0V5Q5ha+=n*Z#+{-Ipw? zwYJ9}Lt4(bM87NOk2FynvH`7kG?rpVX^8A$?UUM7%L(5^6+^Mol3&Qr#Kn>-X}>Sq zA2+n6TdUs5f2h63!bYPsxf?htnYj(16il>>ANP@_T=~IVif-aa*DVO?devvS9UyvjtS-=q@kR( zOffaSYNzb~7#iQGy`rP`L}zVJ^PE&*aZ|>~F;RMQ#Fg3;O^>KX-|;5{ywB1oMBXfs zR~l3n{)euHI)T3nuOptQe^Zg|Jtm;V$$+Evh4?SA(BTpOy!_12aXkC^;h6aQILsD&k|_sNSJj@tx7&_KaXCLN!$s$-nzzRBh*vc)tnF0YBOS!pj)UZUQgrI7 z=9jvvcH4^AuJ0&bRxC>Cs*-}1u7<*rmQ96gYquA-tn+SY?Ao-kCcmL|L+y^DHSONw z(p9DFOPU&5mv?U7SkuzrU%RfQV(0RPf+oP$))Ky~W$Btt8!MaYH>}Uwys>hncSF&R zmddKywL8m~w=93Ow#i?!wq->@?V}YtD>fF?v}|r&o?pL0v@K~_+ETxg|miYdSVbdU*46oq<+WXnD_VA9j5hF5zP@~YS3|+l zZW^y~LuGqO#m=(&!u5GA<=ZNwiwQ) z^&|E_PyBC=ucdy&`GV+~=o>kKc-uV&J&kq5K>wP$U*SAD?i>0Gc3CI7A1EBIr_{ZL zZW`d{6b{!ibzkOmJ*$7iPI&%@zYqY{w5y$;p@PE066h;hTDNLHLtYm z@c%p#FQTuOCU{NCL1V@n_YEnpTbe-4uV&CwKbIw3wJnfvwMi|L{pZMjs-?-4p8sC* zkEcuVqUP1JlFyeVT+OG~WIqGafC|0=@N_jOdRl`z6+Fe4!?(!#KMFAW7Oq5}hjFN! z6c_dut^~g#;dTjE@=ZX0%6AAgQQ{|A;CT{01#s1tKwp&bsel`|jJpkRW2$&e!f9Nb zCPvF~c}L?K;bl@z>R<6jq$?%d+RsJ_pKj?#>5KX?o)YyK`lsidsPdcvZlphc9rzUk z+$e9Z)Vr0>*%EG*cR(7*D(~wOZtdr15^k0Emxg|f{C{MC8|BTE^M}SY#>l+uz>i2c z@nc-go|Ev0EbsvXz0tm}Nw}4NdWsGe;%ADuvTM~#~&$6aZF8^;YwxEfc{@0M`uxZjg->$sG+P@&v~ zrj*hh#WgAl?%_4Yn&ef%Ror;a0$1^2zyeosVAujzORqZrRQ;>r+&b`|mmHh=F<+nqKMf&8ZLSYM?s8wJ5EnmO zLce2KzPffkeowAphoD*xeXR9$7I)(JC+oX9eFPQsXlQ~qddqx|H8=Q7aG^b4ll-~6 zv02=T@#yC(>e}e%@m8)}zpAX++v02JT-Dy#)q>$~a`(q=(9-U0tm|yAYvhaxfxw>N zQ&fd4MhTa<2Nc1rj>U2tZ^CBYz_yOM4!^Hq^zVuW+8e!{UbcY03x^a|S_w(YI7*N# zXz1w7UwqTrxI^~BH*egyKbU$u8+cQjPrjQM^RYwycIzGb%dTR9{e#_ z;W(Ou{XKv*zA?9^mM$55JybCiHNLt&E8A1w#!{_QO*~n3SluQl6u*kz zTL15I{Oh&<$bwLOtNzj<2lf10$|38aA5;Xt- literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc1775ee7a18143c76ca13d2b494c4727.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc1775ee7a18143c76ca13d2b494c4727.o new file mode 100644 index 0000000000000000000000000000000000000000..d528844405da5b5627719683ac94a90ce2915dc3 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+mFh-3m%)>k+Kn7;w+13MuWI#e7A=u1_mee4D9(`Jp zX&G`NM3%7)1{1RzayHJAiN{OJka1=*<7hT#B2RpX$H5HNWV1e*oD(85L;~!=9j^n4 zaqsunqx;^g1_AcOhNYnU-unN)s{U8i|5tVE_LrBhTA^ziQ$=I1uvkq&8O!XA-Fx`0 zhoyqj&3dW^Y;`}&cH90pz0mgA>5Ls3vSs(ZgV#A*_MCsi^*mm`#yy%&RJ3%xw(|#} z-;QHfKir^)KB%yTuG!e3|7&A?M?2Ud&>rr|UF~(D_rg!ldpG=W>-q5UQ}2dTW-JMf zY~Mfl^fS*4rp|jLG`r1~1IpJNVQZ^t10f;n}Si|J0k|vh$b1 z+uoz`{x}qxhw;`NKtCtKb6Q^vj{skO{#1DO$@Acy`uU^K$i4%EPtSX1Fm&of_yS(d z@4XqGbMj=k%=upU!o2gKSr(ezx_7V-x+pt;9DSb%hfbUdmpe~~L-Q_$pI$>eFAarG z>>r$Sa_?XWJYSgiV)&_fZ-OuA0kZeul|1_kp)%-lP9kWax0J*e!=ZQI3~yU;GCceI z#qfp4&WFd%cr-NfFz_qD>%%_|C;$2)+WkprQmkC-;_=t|%t&$0fcS$D5cnLB(Arr}#Jc(WX zCz`8}&nOp5X4gEB=fdAC?;nADWm2vY@H_&ZN5C`5Nb+3-pX1QK7cx2jRtvogIYwSd z2#r9FcOi#Sev)D2mBi2pmK1NFXF|MUeyYg{`#(m6{ z>A9LR*Aq%zTo`hGp(sSQA)An$sK0lh*K^RH3-L3M?CIZ*{=bFmTD;0}--GLZTu%Y_Zu-7}!C4_9StMWhFcy{$2=-)L%54m8Ui||{r)30CyviSwr{OlsuKLq=c z9h6VMzmD~Xi&;O@2OcGxD*IplHf#l7eiwfH9&GmSuwTp{*rCe@VON74o2ChUke_~B zQ5d?GRCI;xcTXQ?eH?-Fz>oQh-fp#C`EK>!Ys|$iH=q*~ooN~c_@Jy4Y$aX}biN%| z4_--lr85@|JF0B=1=@Tq9jp?VT*B&@d3q|Yy2fndZJKUl%sviFo-8&ev*Y&5BCjn} z2%Y_dkV%qlt`|3F|FgL2+gYNAEoBMdXbMrz?`Z1i40L+yai^}`t)kAabsbG@jX^fQ zt-aGXzwPn)o!gt+wzS9U4PCPS`26TiLwl>=+sWp4`huPFg=lpRU7ao7j*dESOUqVo zXOk~b*Wm5wtdrPIUt33ez~32YUbJv)tFP7D(!8}T|B?3my8l=V2B^y~SXj4gXrM*ykIgPcOHyFiSNCU3ykc=rer@OAhCk3%9@Dd20lE4&SIc<%~9 z8h!O$TkEz2ysbWA9V@^)-vq%lYoMhe;Pp50N#_r=ceZzS`_Y_m?QI>M-nP!XyTE-L zy#7{Spy94CGfcx}vD%UY<hW}UK;W_%ic}P7Fov%xcM~X&fmwVMS2}JSo7PrZ0YcI=CQiE zw#VzDnAOWy*9lBr9=2-D-VPqUqj+kG{N=97$!-*xW&>%%ZrNIvad@H?%>IBt-fMURwy{?+n5hi@O@VZaQ6PZt-?Dx3{@F+uK_@ z=DIPnecleAJAcvq#dF;?beUhUj1_e?w=}vtx;r|3t?t&Yj!t)@Z%cEV&%M5)V%563 zn&nmN$|}~>Ew5d_d`-#n5;m{Gy~sV!8)#^9ceLU_V2Rg_p@~;IU%}@e7GAfry@1;n zI2d_C9^I#ld)zX6M(qA(j&p2Hsm#8`4!fZ361(X8J{q0D1938RIi1;M-A-f>XYg=bZfgm!t_-*W~#Gk_Svc2res$t_E{JoMsX#>RvEZ02X9kT znIGM|;E0uY*Td$I(_RUR!}cp>gQ?vAkZB z*N<>5Ap{Ef#pv0_Snp0P!hQ)P8E7btWpxXE@kx^7WXE?TM)(0-3MlHGY(5>*sRH`c z086F8{gsz(?C066cb`4NE(1VJH#*twrxNXa+>P0(?6{l#s9VUuo&+BK^#(DKB#6de z-mSCE#Eafz6a3tV3d#(;&N8w-qPxO4633~Joeb`7Gf8__9|6Istpa(x`(q7;P{SiIaUq+`!{GhIr zeE!O?jrdIpPJ$J=eG?R3(1#h}T>qlZ&b*{6{gh^XLz)-%ME|gdlRG)t1N;;O*u+VEd{OV+nGj(g06;zf z|H}y2BIJ>gMFPLPTVt1r0ZsVF%7C6mRi=H+`Edn=uiJhFeV~6mU}H|glM=|j!5Mt` z$N=~sO0%buKVzZvZ-#z=&j{wupmh`e#oOlwVUFQyZ8FI>n9WMRei&FP82sDZY&ctv86wo_b+e;JU6$EC^=JHVc(`lHWb0h35+S7c> zn}+(F2)iWf_3vpjf^%oncyozA*k^aQV=kX>4@^sCf8k^^f^%jPeh$YkV0*jS98zuK z16Xv13qfxOJ=Qj&Kj!3kJL&>o47?k7w+Zh?UEr4kKNa{qfgcv_r=l+Ke-3;m@E(Dm zmB#zeL|x#MwcXt=;IW9(_)P-uLS5h!bRo}XiO-h&ZB%D(+l8VSns3$MgB7_fySH0{pTRr zb6V8DhMZbKU}{3tU$Gv{2u__!^ocQlo0>^=?r(e0-i7wc-`0?Rr@Bo3<^bL;?0IY& z$2(9L?a3eQz)OFFe?z}BqWV?#CI-mfZIt5eF1pC4#1^^N{Bg3uNOhz#_K^_S)cH=xv4mRVlvFS;HK6|&N-Stj|$ z(pT?J zI)Hi?-N*v`fo`VLt|zf@!g$EfM$0d0W9yU550JlM{>&qOR6+gt4%qJr04N`T|4{sW znH-YZe?amdjm3B96Y^N%yKac@*>U1Ia$quv)UO`MjK%Ar1=`Ahog%j(zUw;Km&bSH zKk#RKbWHx-lSJ!lX|fVP*wgHPq<@<+KHHD)K)#{mtEYU&;r!*b4=0LzhwPUUVdpR) z_g}Kljr|k;&GSWv%r8kEUR3@OusQhp{MRafO18a?)*%$UlzMo0%DB>a6 ze@e{$@PG6CT)^qMJ!wurp9R6GIxV-1^mpce(E2qskzJ1P{AOqx{3|&#>R&?(uph|D zg!m|+AEy0=gZoc%bbpeS6Qw^-`wd4jw-@;j-=DZSeHPIJpBqeeMEwc*ZF<0+j;0u_|!h?j~}M7XTu!H zB4;CM#9u)#=tcV^%6E`Y+XnW6SH8ZbxzL|G+8^Xk4@}difIi~Fd>*byO%G)2_fvf` zCJ@HQcsW_9^Z9}PVBbt~7eCDh$XgK1%#6k}D?h}4FUGq*KeT=!pc8@cZ?nH2r1@l= z4_JRp{tmopJ^*j>ci;{40s6@en*AUCK8@_l^BwS;5j4yP$V>RB|6kH5pMrm(P<(s= zUqVFw#oVZqe$OpH9q9sfNb+2u{hK2preE@pO_s2lMa6U_JtsK@Y-m-{Q%2jr!?3mgdj*f5`S$H>R| z}#q)%qm`$t-qac6S-ObPdh0aH?|2o)>rB)Q3_UG6ii$hG>AHKo; zowO727s>T3d9frH{G|u=zVoU*qR{N+>0tJ+1kf zDIbCT_6Hae6lp)}dB$b%*LB=M@eurtllw7KzC-zpp9H4-gYx&}XPK~&(AVwcGkKqC zKI7-yEB*3(ChC8&e1@M-oJ+IcFn>XOyKesSr|>_-x6e9%8J({aZ);GgDvWdZQU^_kWyYrbMwe-SUTuwG+15%C`SGp*-XpH1sG>RDK?qx5Kx^(eOA z8q&<`HRdbfqwDpU`3j%kmVAXa<5Yw{^8JLeKkSL~$IQvVMqHThr$5Dfh5R9IzA}A` ze8oNe#`#JX&sW^>^A*UG<)-)^%~y>1#^`(n@zk9a%~yt@zfYczWYPM{<0bmDYvM#54!xLj^%Q zh3f>K?_&KE=S4?1uz2wd@gn^{c>isRU&yEQ0RQwM`uPLlqwx&^AUIV2n&h*cQ*FlSrh2D7`gG|AeZv0fx<#L+{m9t)>mP8wUrI3NugGsppSrd9 zhWyCz{eU#Knhu+IG32wfB*^`AekY4iJmha!fAD!RKkpUi8HV%lqaeD8^YCLh4+mvW z;`=yOX-c!~uhVqw_u1D#!2KEf&l~cG&%@yl;(R@eK0oE3N9y!}c_P)p?`q3_oA1}6 z@;MpF$Ill@NU^@!$w7Gg;dJVsK5miyX|ePFyqlBPe14F|z6gHAeij|z{HL`0qgZ}| z&ks!b$!Ac9RQJj~4cybfJq_H`z&#Co>KdTKbA;Lfe9M9L{QTPQKm1YT&tK}R{OzW{ z{rjJlkNeBUk1qUGW+C+ApSA7lOY>gPt@|JS<6*ElL=ZNgvl<@EOT zrLZ$Ue&zSSi@a@T%d6K;Kc=s(PFP!w-3UVZPi9eE&!;brYMT7gBCn$wdvFf$R|ud4 z)yCOVuJ8z5$OCn(uHkQ%!TUI8%I~nfWo(bO41rv$XP}Q2J;#}-u_^#;nnxtwQKScV z+PFltrwl;T=JNj3c&{edQ7=j@N~Bj|s>iOCZkjB30X14EIE`_I6#-H>YK%(i`C;OT zk8$NyFhYINOSI#>jkEHGW-Nx^lVZhFx}z_3F1*4vL(D&-IDI6CW>fE#RS;cq@!kskpotk+ZY z97p2s_)4LT@-7-%tti$NBgWJ9G^qNGzXHYD@6oL?yA0#Tvu)HfK^G6NtU&#YmfA4# zC~cyFQQL{}IMY+BoLBhf*OsJR>17NLJK=?fLqWX2A9ZLr!1%JN;~Ooe`jeQsl(MEpUWZu!y&JlHVNzip{6)y#O0rCw>N(|FWBUn0mu|s;wS7EoTWLP;b)(MoI5Eh|e>{4X2S^?Jk(Qd*E_3}!;FVd# z+GVq>UHsd^hPBH~|F)(5Qe@4>{ydg?rKOH+_4;d)rS3o(O?A5J$ zL7jS?jyiQkx~5TcR6VxOu<0~PVd{E&q3iLKn;VUCk+F0WFIOQ>ZzUISgI$A;b%ZiN zVudt9Iv$n9#>w?e-fPM8s2`*C#R!IxnRR40NDTOUu^#$>kD1o!&ti@IG{*ewqGmkq zKRa%$_8hBjoMHE3xL8BR(%C1)8+Cp!hMQM$5i3-D7#(+Lj~V|Y&9b`d*43Zlg<&VB zX;+ZNsGXdQxN2OJh5jIMiAIjaCR@Udx!;p;Zy)-HnXB##_RL; zIkfivhpwAuYU1eALMuP<=q>vpQ$94ReRR*s=U_r~1)=2!#Y%st)lK?|v+`NxiAS$C zceLVy{*K;Xa{kR{d7rnlz0bRTP4mgPV-vAH7WMXw+Oc`yL9v*+&=rA0+ZB@n`fqz!w5uahvk*wt$UHG^OKOnvzC$|5w{zspTa|sOn>l*LB zNAX7AJ2Un#@e0lV%x$f%mAP=fK=VnIGj<%<#Ce91cEe{FEmqAf*PfLEfRAh-&{>># z$Ojo!mP9FU=J`KypY5~B&7OWf=Y{V+Ke3g4Z_aOiTWkBvnRBP-{nNA`A52RvYe`vv zAwLJt|EYx3Zv0VQUM}oW4xaG*pO;rKwz(94TaMTDp8rD%B3sZk-m`rO{tJ3TvEd{r z`qetrm3AKmoFB|{s_Vba=6&#^ArsLu4jBWc? z&*_Twqx?lZr%TbIIWOWlU6~3Ty+)tYB_WDFN^qMuG}HMpou$1;)*GT9C<{PrWe|1R z1jK9qW7+OY7S&qYG z z529m&c?fAJXDw4qjj!4%`#+AxH)^lws6EkH+tWNJ6*|lo*Z$d_C(Vos?m4+ z$pG)OI0}(BOXZaYm4*MItD#Qd@51YdC+go+WP6VZXmK*&Xni66i!5|_gg-AoHFO-$ zettM6{=SavCp})YJNw7ty7&Qc&BBG`HxZ%!p8%xsQB0_b8wHR$HWo=61+7Q>J#ohW zifo@NDQSEv7WFZoZ11s*e@wPl{!8u2rLFB3WAZ*7XZ&w$0x^o4zcXeXU-?lv+V`Tp z$1?taY_I%Jjj!4%`27IHYyWnf_B17|?div&5Ow@8vq}G#WqZkf^kUWjI<)^UapYI+ zFUyv2Q*M9j_~a9$)6Mmg&~nL?1FEZPPvF~a$D_EMpO)dG^Ht4T<9NiYnitk~s_u~v zVr<7jay}_Kbyf3AT~)iSh3jfM3YQm(QoOooVM|xTqN0|Ki`G?dD{NWs-O$*zv9>C| zp>;#`_JXzT-ooP5#Wh7u4XrCWH`P|PH2Bx8Zzjruq#v zd7Em>S9v!SY;P&As9v|DbVbXG$EutBRqI+-F06j6Y)4t`!m5@{tt;~DSBkbpEz4T! zYm4%Hn|QygY8S5Ps&8GE2Oj+O3m>W8Rl$z2vi9YrTh~jP(v593t+k74 zw%run!u+x5R%~hwHf>r|0e!TVlFVB+k?!)_OBQZj>n&Q7zp-`e@@<7x-5WO)1mcca zwxf0<_^vNq@kB#+K{MoSfv%pYFD}UQ)fV~dTYa!kO?iFW+9y_%4OZ4PdkfdCE^1qo z*U(ziRKKaJy?CRyy4qj0tfjcAy=24IN^eo|>J_DH@;8+(@7mbf(zU4+uSKi7Hmxl8 zH@4PvLw{v!^J|`HTv^_-al=B`yDEQU+q&weO~PK=ipsa3@4Uv^@-C}gyv6<}Ha0JQ zRQa2_*0r`Y2kN#rC*><>leWQt&G(E4(coKKWi;jo*{;d0X>> zc@S!6&K>>ucEd~PDO7;8^dF>Tj0ZE9*Z9^``GkK|bKmTS15i+kf9 z^GRB8sjN?Z!1J7b*#2J=|C{4$so!)yFM1~WMou8!c8x(#V;wfozozb2I**R~rvAKL z)`{*13Ww_{b#I}Y3ivsN!?i@+mpEO|>EE;yp8w&mgj2t~81T7==qz(8N~{1h7$D*4YWp zBM%3t_&}9-&wF0ayGElpCEjzU1wD_5qgvPu<0_<}kMU!M%bkz{;wtEkJ8yw($0ftC zb5n9rm?XIJf1xkNz6H4yO!Oe$=}YBA?~y>|_hBeB9{uX;z<-bS@#txP6A!-RI`AI= zPW+tVc3@1+EA2Y`A3@?p^p(;CuSq#*%y{FzA?0;T6R7#s40`J4vV^O)3ng4_QcGn2 zIkKNhX)>kfzmxpq=~BF?dG%Gv=gShV=F@Akp8;t=1>XR8x*8Nctwo&*p5n{ln`Qm9 z0JCr7O7wXchssHDVQ=9|@GBB-mvAND1oWqThfotGev$>AC*e~7S8WOOMG3zjaO0M7 zmjP}}6^}_cjf>O7XgMzLXj~(_M9NA1E8d87m4sXSsg>|)mVT7Js2}4gQIDa2dd`U| zM+|Ty{qgI-uNdG)d2^-St$fasaI3rn(m+;uUzc!eKR=altGvH3^kd}zBLmzhZ>F3- zG_Em5=3NJVSi*@P<7#$9!XLK42MqK^`@SaOR{rTJI#h_CDdLvDFP3mMuJXku30LDP z_+bORaoi&kZXNd*5^fzgRnA))*EnBX5&_-ZPPQchuE4w|N8I410t|?<#LsTSL=h{3j=u}Gc!)KOh?TQh#;`b-(yE=UY74&Fmf;M_fe2+Ib z_)Ku2Jztajxx2Ah+=B7w=PT;k=;!fPt*TjF((G;VHFU0SZ|rKp@He^p<2Gn%_cqpb zw%0Xs#)Lp%Pw*+KLKdThE7}8!;8w?Cxs5krGjCvPM_q^C*D(5bMFZ`P-cB!@&)xxWJvt<6EkaC$;RYv=Kg2}WX3irsnGct}Tgo;Ta(NU$Ui~}7 z%@LW8(PwKE?_@ZR=3svZAdPR#t*NC;244?V3`LEvuFuK#)VHxzs^HydgvQ2KP(l+= zRvlKh2@1upqPN!nyBz;|?LV?06yK`9bO>*?3E@@+6ovU}rQIgmkCI=uf7a5#%D)fz rc=mHh|G6xOC;O2rsP-y{-iLv$V^fo}9rTVL9+&bH8}uYEYy1BPSX&jK literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc1c162425e681df83e1b62b93d0679bf.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc1c162425e681df83e1b62b93d0679bf.o new file mode 100644 index 0000000000000000000000000000000000000000..f4cc32c2a1e2a876ccb7fe4586954c38dcef0b9c GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+mFh-3m%)>k+fiTR(v#l3GG9V$4cx+}xOKOlnk3KEQ zv-_1#?frfM4tE%$H6StWV1e*oD(85M8epEJ6;D8 zrc{V1?h!_6&Sj!ruI)&JNE^?X5X$o7z8Y zn@oKj=}l*a|3~)>jG&#a+1{+j*rTVO8mc^hU}(pCPYta){nXGmbG0_O_rgyucsKlD+xhVEQ}2dTW<3-d z-Em;($)}zgN?outG`IEJ;S2MRho5-wC*dcZZ-iS;TnzWGdJlAkq0qsDL)%Xr7CI4PcC?BD0J#X_yS%n z@4XS8ck*Po!uekK!h-XlSst3(wr{8(x~MpR9DSb%hfbUdS2|CJLklj3pIk#cFAIfE z92lB+a^Fx0JYQJwLimXVZ-6i80kZewl|1(gp$h17ULt6qx0J*e!l8HH2yb6;GCcSE zJK+nDoDWZ!^>AqPLEu+_*9U(T{xRUmbN?hX`pCYaihVDHlh1<|yeDhfmy>_Jzh>_F z7X!(AvcH#{*jtnQN_LR!=&`Xv2W+J2y)_N+sVSQ0%30f#e!^|hSz*-vzzO?Lfqw}f z*rKt**Wd%1Mz|^c!@A=UYIl^e!dC#x;cY};KY*WnP*oE8a6>Wt&U5AJu2+v4{WqRJ zeqht)DW4?&Q@%F{-+LLpdDr)DZ{O>^j_(~E$M=p}e2;um`QM<$|CGO8&j+vW`avjC zGWE@o`}M;Y{-yYe+s@M7`DO8y(HF}GezlGbq+))Yoz1R{{6N3*8vJ@Vn_W3uGUf1a zqjvc0-|1IJ_Ok(+M@$=hcqQKVB~)7@H_ugzw*x8?C{9nCWJ=5#SXvIh&6Q*8yNlJRKOBLBlokz zzrG)||2-*mcEg^BX|93i!&i5`Vwy{0eHZe)1O7>t5%5Z5zDqJe#*2{A37JT?q;^d`>|BUdZJ9doA=XP?nglK%CpQ(8U_!uei$GDm{-IY-htneoqh!ykj*c^=4ThPfnnH> z?4W%5{dH_0T*3yJKKL-%RN4R1w_q#y^1JZk_h7Sshy7yyzz$zJ1iKpS*gQk%gZ%X4 zs-n=*HBC5B!+F=mIiTi_CJfOzJn!t*fN#?j;0ai!p`Q-u0WT!0e9-!(%E*tMgjeQQUo-q5FF*xl9Y?d+`gwzh8b zb~XC~^^M-nu6l{>^0jw%1pHlrmc<3z+I(%^)|PGU`Aa+U>;Gdh7@$7Cu%LeV()#?o zyu8H)_4rq?B=4d8rOWE`>+>L|kn0W*qqEr?@HO2zf&_e>zQCi92v!RCTJH#NgB;#F zLXak3L-)4&tpRVFPguta@Gdk#FwGiiX$W}z&3w}N107u*T|Is@CtOE+XP39VEAI|) z-$t*$%@=6ABg_oba9OOjx*94Y~dIF=~-shYi-kj;&ieeO-C1 zzP|m@`Y2{~+3I?Msn5e!t;O5PqjwZfEs?+6Q90T5BGX(TZP+ba%W@7+w1U~6a`?nC z@Qv84+xU*%jC!2IaZ+e0&jX&}iuKc*BM2J6%1qpPBt6J@*#)@!Xj&95!R+gjUJb42 zzXP}p=NHxzLCKXsx~BrZchfa3bx&{a9Oj7Z>7C=?w~P^dzO@rsS3KLl4uEKDKSPHg zRK@EEUQ}OGS5aC~;ztF2!prf&5)T_+m+Slh;kB3oWhQu9l8=cUMP8 zYv+77X134U>2v2VUbtkwyN)gk3zxIv?v~ajcV|y$m#@v;*4^3VZt`tyY4^F;S5>WA zS6^3Fv#z3QO?}zM^<`^H%SzdTD)(ad0&k$P+1=TO1A&LUZVXMl()kKL|FH17gY5&{ zzR1DI6Y}UjUEJfA*)wAIH*%a~YffeMt#;T2Z6C6WzVD&Y89WdtLzmK-UDoYn=}Se? zx{XFR)OF@I)#+IAhdyj1j=%Z?&VMfCGth&dG-)sJgZ_w~>o22ABYsfV zNj`sd*hc&&1*gD@+`b74FX+RJ@UDP!3h|qIHj>1K(pceNpnuf4ysz{AoN1`}8_*y6 zfBXp+A$!;zu;*|UyKfiKQh|II)7V#fCPuh92%qbOe)NfTwY9Ze-q*7s?^K<9DdK>B z(0^`l>QwT#iCq7p&d$83EB%ybeO;Ot_C)`%hm$)w*#rC(2H4~&e0)*w+m#Su9{@l; z0RPJf*dpYSk;MXE)}ygY#DFIJV`V^3qbk!r=KQzF7UCVt>wg@Fei;P%qQc!h!4bGba!{&X7V=llpei}p01 z@@AkuFTyU$dc%9#tl<2)G~RsT5BAy9ks{FBL65bK=#M!$-j2G!mjLeu-fhCWQ5X0!;HLwhC-5Vp{dCj?{?CEW1l}X? zbJBSKnWzhVvbLwk1w0l}8oycKU8oCuf-dCQBJtUh|1A=)0Z;t<1-@Ys{Of@SNIvfW z(`DxWXh9(?|xJ?YDi{y9Hy+4SSS;D2j+7wpRY8Zf1bvQzorRt@?YS~zxGqvT-3+*XQqe0=8`?RzhXS{uPsC( z=R-zt)+~yD+#i2N{jW6j-?wW<%%6r2Vtq@nk-zap;q)T-7uNfzf06&pNTBg%aQ`_( z_M8#*uVJTF7?_?A^;fJ1vx3v76MbUL-==3$o%`Efw0EJs^0zgl-{~%szd3++3ws`$ z!SN2%MSJo`JMhxq;NQ^itf+pKy@>&`_gkcIx<`@vrt$2`KZi@8pDuxy^LecPC0e$J z{%4W8m4A@_`6p6@$9UymTW0F7Z&yZ4-wo{+ebf9l=$p^i46g4CsXzFCMpWMo*;p@J z7Ja*1F@0B&zFe{S&F2T&OMRpNj3BfH0V0EaV*RE0!3`+&m1UNf`irgy!$qt-YmQ02 z@$}WV3;DoVG@yQ|&+H<9p!K1F)=SY|#kW~O2d8($ z{LR7jLH>45)5r2Byp^xVn13r_KT*W|v&KWR?-?SH>vv&r;zY7PA0P3eJUN%wR6OY0 zg{5D`FF(nr$K=a#B3_EmO7G$Prs;(gfBZI>jMi_OKRS<>I^pr_v|?F962zVMCw-$WX9t4@FHzxz)q3d5Z`s3?91aj z@*ns!J~}3U?oFcgwLDn~Ana-OKhnR=7@zIOcOu_V@-W7m>zC-p)iLi4R zkozy$=lcE$|K|CkL*|zx4=*bJ2-qBaef}RRe@eE!h2>0*hxN>ft%As>`FfUYqiDnZ z1M(7{uXh=KM)DT+0nY~z{*%7jmdx{G2hWe~xj{$roc|Tcc9K7#J>=)(@%h5*E^zmyrUex5~@>6!DPk zKP_f|_`i96F5>juo-`+*&%)qzot9fh`aARAX#JX=$Sy^Aelt7+{*{~=^{?SY*bn4n zLVOg^kI;U@!Tl#Wxi>~Xmh|i!XGg{wU7GahiUBTFh{b; z*+?4kSJ(%7(LRat9puxt!F}MBuWxBC>tFhidP`iKkjd88^eJ&>*6OZBOk zKo}q6OyoJHc%xFBb@XWB4@SlTbI}UD$l9_JmJI-H6YaWX4ZIJ%)$8++VpqATQNj;6UidhWVU3PCm{j zmxwPHs?721dE}3~St|KA{AmJvy^8!Pm0I{q{dN&gzK9wX!+J4-Dvt*<$bPBMqAKD! z={phI5o0`0Or-u*JWmLU*~Drv3i5|NJq!&{=zIkGuR}dpYDLj$e~$gJIK-s=;p^Pr zNjnjLv0TrRmq>EKUwS}Kr4y4#FWLW)wj!D@2>efi+!+`jAO6(`ezfEFf5bcC^#sQG zA;4>xFS{K$DnPMey*P#OcME&?8{VJ5enIi}Rervw^mEhlV871S5Arv@KA$n!)0&@| z@)6kYK!71Zk@mBmXIuh*-Nzji55eCAxgRs-JCx7(Nnpx9D1T3Wnh6UDecehvllQ6S zGk(s!(l5_vqW%}lXZZQVxitHA^B2UoYvwP13jafV`>gYqvH6N4`6VyXPmzyczPTLa zkNkWE{%Ouv76ETupJ~0a<|~Hv7x5ws>ot}W5$~Zt(|V5e*|dJ6o`v-~N{{wfk7E0+ zVa>c=W4;nTx?Yc)ukiV8$yaDIPDS`5-%lv}!=5;Q%$y2r#D)2O`cuqT$RFb7D>KK* zSKKqNpRZ)`e8n9j5pH_em_Azk|h>=iqB*6quVzhA^-XRG9Q_5ieF>*-;`aT$o=<$AhQ#M=N~w~ zFYkV7+<0}=6t4{D>+qjjiC5A6H`X`vetQuiK4JYd?XR~|d`jf|a~{u-AEocM^ZW_>fi(6S^4Hx7i4@P1CZ{8AVI4?T7fyIkwh!^Sq!TWDh{6apZ2l%HC(a#?UAB|_=KOOOm_Ct|Ok>A~X zJTvV#%<&83oAw*fw<(@+ym|j&+HU}FifryBO-hwyoWeJbCNbN~1+v>%V1 z_h3BipJL}%SpUcF$M2nf{rEPG`}@7|;~V0~w0rsfOYO(U#y8XXPj2wuX_ojFJO8on zZ!v#wV81=h7~cwmlXQj;u*Ck5=11nF$*_>ve`jO=oi&L*pEK;gr_+AX5bvC{A&Kq3 zoz8Lg-#AdAax3w!>K69j=~yknpxCdXy$ki|eiZrIB;$D#@EOMQDDwYI$~VRSlh23D z=>C)Mzsa9aqVaC(ROUvV^40T;pih@>&?oGlu3PkJ+K-H%zy1N|`^5xv{)+sz{E3^3 zZ^(}f-w#M*tLdA4#YF>EjmJpB6j+&$}^s&F2Sc?2F(>>}Syd&VR~#K8och z`24_>pL_;&NOiZ|)xcd1+||Hc4cyhhr>+4yJV&S<#J3z+@6WIP{=*+e{`|%M>fdhu z`+xjd<%GX!`sl*nMh+FSIsH%exoX}0^Ky>$_cJzNuYG3p{eP{Ge3he8)+YRQe@_!mMe=>*SdOm$|RMX^_7I_`j*n@L`zf1rn zs5Zf#a+ycyA|9w?bq#;B4BjU=Q+|i-En|DNx;ajv_s{ z(h<#cjSdNx>_^4uC z@!L}_&x<=g9?8EbU?5)qJl7`*Pkg+{wSb3W;<#r3XrrG06K9ep${D1)9%p0cpqX=RE#znvyN5`s6mfo@Wc)2V6CpNegJdN%{{1(HfMdB#|Eq_Mt%gWs6kKB za~z4k<12+W#=B^2wW3&8j2KVX(xB=${t6UpzgM@)>@tiS&$dy|1YJD5vI6xpR%*k@ zW3-6|Mr|j?<4jMjabD(|Ut5xPxsNeC?1UE@4h8W7f7GGj0OQN5j&HP>>W`zYm?U}( zs#6_C6Y&Riy5&nB@LmU^|NPUBJke2GB+8^wt>f%y2>t&>OGY@1I? zUzA#Ezu7+GwYd@A&+rv>ywPr?r}6r*;^X!C*>Pj#ZoIm2Cick~#Jx(lw2mqw2AJhE1nYic;6xi(HSU+}LQ8i;Sn6c)1F3dNa9z8|)f%tRs{G z5-X$;((#xqHcqZ*@?J}xNBtPBFGeto%&a53L1Mt)jrGt6e9W{)e->-xr!nSd7d7K? z|JiY4wdZ(s;|#kS!^Ijhp3Xig-l+4tG2Fb0i&&xJ!`Qe(d(8MJX_nPpx32yaFAO_5 zO}mUNM(yNe#8u;(Ec6G7OEhvUHrW!c&;8znyZg{T%v_DPy5H>lzx8PFX^d{=HeR2% z&Y`vUKXlzRQxnIY7Fzjq^tbi?lJjpq%lo{Y?S0<$Yno5S9h->tv8cCq%#O_i4~oU?z04T()U16|*VmL{ zug&<`w{{ucM~C6qZL2cw2>i1WkN6CWj$|$8=)%WM_(AdgII;bY^*{DxoJ(NnU)Om5 zy^1&b-kGs~iC1X;XKrhCt-^)#1)5KyoU!A;CeAaAwCg^@Xo+fWx%RFU0DNQvfzIN@ zLq5o;x-?38BhUYd`)r@hZuaDJInRIRxyfzpyYqhY+l{urnLU4I-j8Se=ulc}MQh3; z4EZ^D{!cZecH@ug@^WF1O7Mi||GczB>~#=r#JBE(uZeQG#2%p_$H)=`8I%vfdc|Kv@7{D}$)h zCLmt>AIWxKvZ&VD9)ApJIpY%juB1QGL~+OlwBFHJiXEjPvWK-#YF8~Md^1%H#Y#(l zAwLrrORA*(o@{^I(3WnkdME#(_8toxP159U;HZ%25%1Ra6@YJAmB+5b^AzFvDpN9~Eu+Meb)slei6OB~ggHSthSEs4V;sT@7^te+OPiJW>CqBHMdRK#P+BN9zmmUu>bnBm8;!siEU| z_VdFr@%MFPKk4zJ-Pu1D*CqFfYZfjfzljL-{{$e7k77bi-Xwt3v9UWjpGroYF=2|sk%ox zh_M|9$@!${)K$$dbye-Q6|Jl5EGjD!rDSz+L2Gy8;^Nj#i`Ug|FKS)y-O$v%X=6=( zW7~$>9ffNMa_Y*pKu{HF5S?xvNkJ26H( z_$aR{uj_6sSl&b9HEpQuD6ZI9+OW7TueE$zWpv!~ZEL(m%R9=-x2=~n<(t~;+BPn( z+kQiI1^MIAt=QZaY~H-M3i@a(Cz-cyCf((Cloo7T>n&cBzo~6o+4iEEo=qDH198W! z*tu~N_--g)@mOO|VGHDKg{~fJC@IYIZ7lXTwE1A4y2^(3wU4bP8?3Br@fNLHUEIDV zud%JJxnXlnN698{ZLPm%d22~?N9l%b)!yQg)ho)^)zDX+P%3Puf?moH?OSp zH?`IEKz|i$^XndKT3Ok;X+r_*U6a45eO+zyW?`@G#g$vpcV5%R%5JM&ye0m}Hnl8y zSoxc}*0;5`1nPG*<~QaqEm%@e=v%rhziI2T#lHN8r3DSk7dPcCeQ0^Z)_OtL*s|5z z=)2ZWtgPq1twO)5f?qqil}g@qo=v`Z$&w|D^YZXxDfktY72eiPpL{Q=#qY`Zylr{G zJP5Tb=eB-)a?T&QcYWzgkRyL#J~xqCjJ3yVn6`U{Ha#^P4|2ipM{=xq%Qf83$Gvfn z`8X}ORMw~7=XpjyV*ks;zjAyv^&8ITM9)Ou#0kXP?s4d8tRn{cSJnMW=g|q@(4Vu* zI?;V!;cz{r?k#lF0Y9g3xR$E>Qm5+~{Tp_|^FRERaO#&A13uRfjOJMIL|9YYZdn)vF005*xsIy=F6 z6R7#s0(yd9l5o|wK*H4~wN&<> zqsEmcQ+oaf$v>Vh#fzF(uS)utBwWp>Ka>0nN&_nR2GFOgLDAD%)T!Vpz8t)KCawg(Ea7$uSMp6lf68|VHCf`PSm1dQJ`HfymOx*W@OuF_ZW(tQ z;Ko$(n1s`~I8BU}~75HS|xMpo?hW)X>=By0tGwOqjm?knpPVei1DWybj{G;C z=;s>QjqW_CqxzQi7W~X(LucpBHEZu~6AZQZI@$P}9JzJiKQB2x^`pK(Cw>}2irUf; zfZXN2E+H;{xP*SkvV2YL0{os_;|@W!0{U3z>niHP?@u;#clih^=+V#wZSt1-9&Ks# znczZuzGnGzcjL3T730y*SJb!D&*QCHRkymd#oOv@>{{K?)ZL2VZ*cd=ZP41`ZL06; zsBhwo34y?#;8RqEEJg`ebOaQ^&5p%#n{L2n-oUob`cA*EaqRDk20EI&U0$}3zYB*H zR$2u~N;yi9ENtxT%3pdD+KBrLqqnl-HMRM~+us{C=k2kX7qhWb{c`JV`n#@TdHuxi zdl5urs%7fyvP@93Wd5O$a+y|DM*Dn%$+RE}_sF|5GL9*PidU*W8#k14c@#ii{X50Y zvoar}&( zI;?IN6pCL(Z>|4VIsUcUe`G-@zEywe5Z-DN!mSD@3iH)UyIr;)Bfo6_w55TSe;@Gi q?B|gFb4d|5e?({pHe?%XLj-s%Y$07ON>JW0^g%dk??$ zuvAdGS$B26t@h{HZrlH%7ur5Mjj_Xnw(Q<_@j7SAp8YSlp2zE7agV0s6)jz_@A^UL zw`16~4>#zc4=Qb;>o#`y|JhjY(ROwiv`4x#Si$$QJ$)Y*v$ubxv%}L;yQ|OICiV{6 z##3KMy3<*~|J6NxLujXKwzq3B_VB4^1}e@U7}#<7nSqt3pBeb3?zslqXYo2b>*Tk? zy(f-`A9H>ueEh(P@bKJ|;oiBY!uwaB4qw`TIy}7pa`@@F?}Z;}IUhcL>b-Ev^d+I; z9R~)Ue&(5h)VYg8vzosXzA)!_c;DrphM#u66>d6lDcrmAGUy6Ip@Rnpwx2jKuy5}1 zaK-5p;ilHp;obvpg@+%x6uz+I5@;5M%AE%XF3f#$VBc!=v;RBcSuGg<)LY^5^OwTg zFVlE`916|Fc&iVhpA+HPE#D3g17C6eRCw0O^WdHO`J>SA{(}Qg&wXYfbm~O-0$xp* z-wMw@c`{t?yd1tT_dIBpg=V!pInWDTl%GG2zE6ZhCr*VcoTtN~xtGFEuO^ckfiLI*viIVZJnIXga_I8HBcOraQWC!%4!!qQIA!a}@T~I} z!Pmv`nCXv(h6%qMygml_SI?u}pP=3TfpWn1C7uiqKMa_bef1}aU#%X#_;Mh5Pxkkd z6T7REU&{`X9X&Qy;DC)Zy}P;&J~ct}Tsv!<&`Y>2IxC3UA2?zE3GgrB16wp!@CJN9 z(+D@AcTjgcM(vI=R`42NIlPVN>xb}@4=Rg8A8sgu-+8WG-}U-2qyNV9#}91UY~_>W zf6Djz;d`&bH}Cu2-R*nbH}SorqxjxYi|>(7D*x-Z_@DCEoB81NT|W#(iYLB3^pJk| z!ao&VbK6)@z5){;hs} zXg}+tdBn8-M^^wJd5-l({+ab%dR4#nZZYdS_W-;0FAG`U`80OzLK3@{0@^~%wV}NU zeMA4m`i36T4qyDA`n8Mivcp4vlMovE7CU^g9&742);Ij4iGU@Bh8|*v|Melz{&iC5 z?1nv$(p&@2hp+E?%`}(9`YzwZyA@E9LzDF`a#!HaV37JT?tAwKF6ScFJyB5ofdizatyzk5E_OY??Dcu z{3OHhtBIju$OB#UT^Pp>UxH4CUrq`!tvjw<4(7Qw3*#qitdG`>k1Kx|BG_*bvqp`5 z;E8nAn20Ode_T0IU&j8}t-IYF2ZlgPQQi?$mSPd^Ro+C-yrNqc2GY3{#w=- zE@pj9?|+nRs_cK|Td);;`91jYW!UWBV856@u)|jl!L9~7Hcl1#AV2-MvM_W#spu-% z@4i0F`gk7B13%_3db{0v)q6GnpfMM_Qjbnhbf#$(;DfSGuoZYY(D@ErJ$NPImCjr= z?1-|%7ijf0x3el>atW(r=IN=p>Ke0+wQ0JIG5Z)Sd9v7?%#PbHi@df_A$0b?giMlb zOP#nm`=7;C-@y_+Y$;0sN0W$hUVCGEN1(%7hdXucY7upQt!-~?Z3wb?t!*8?d99Dn z>)6rMy0tAf2iU-VQde!x!wBCq%2Q@9b#ywzt=Mo13?JI~sj~+Inw$ zN3Fzm_*&cB0{)Ib)4~PYT6`_u=B90}`HS1~YyW*Q7@#)4U_tGIWs7R_^YZc*;tZd5#x{-lpP%*B>s$ILtBV;CW^UM7- zpL=@;S@iP0JQp#|pUhH018z$EZbmEjC#5+O}?O_jTm4+S=B~YonM| zrK@TMrZx{-wI**nkKR!{wM71MPvvAci%heCv|+bwEz3AO(F$gN%Hd;2z&B#EZsR+4 zGwLZ0$4Q~3yaafLE7nhUjv%N5E7Nf6k@O(rWf$P?qiIpN1hcP8dNs77|1RJ*oL^W= z1SMAj>7EMs-c8rE)IHtZGnpf@r+cP@-!ew8@z5hU1)0db~LrMx;xt1n%n2NF|&Q% zcAqTGImaJP50clcV|EuHNh?gro1rdFSOU1jCUwYBR@tJjuS zuC6WJxUO_{Nofh2Tj^ftp6d&DQ;E1j?4^A8KJJJ^$e+vhtNc|soD zr;B^sGJ8hs{#K52Y>lbRzSR!9pzRX7==%X0oxuZfGIS-K*=5~MmcCLLt=njHLtSTX zQ=QJLjCksk>Na$1n;c>KN*puQ+BEyDRBltUD;4`pj1Qx@l3}Y1+*W|MDXGkl?p<)i zD!lA2(7<5$<1hPE;bpe0b+<>$t|f#( zA-@g5_{tin-NT~{btgeJ9vFC z+g=dVZG>lx@Q^P!K9Y@E!bE5r63^a_}rj< zJe_rl{)wJXGeM93p${90N3;Hl4yer_GK>VhjjU=&wG*<9u=pS`1@0+|oXBuk$I`oJBpW4SFWDmOo z_8hEa5AGsbDv<9|8v9Du*a$ZV;d7nPk3M#NO-&7#_swj`J5eWJia4Mj^q(7?IFbBq zEZ4uNvokO2N;Uzg^EJ<&hx;p9$E_5eQx0XBXDA79jab|pmE2LO-{!2dD=wg`D- zWTC*9c4_PiF`xfC@EO6}>9lUbzj*uHAj~mTqfI3F2C`Y%3rB#Zg7Gg9{ln9*Mj~2_ewLH; z7m@Onc43*Ii7oxv#1H(dAn@?R++La(uOKjE7MF+GpH9R4oD*SZ(Vpg0-c;0QN7yA< zue+>G56+oIdW|L|YAHH=R1{1o8x1b#@gpMtu;e--#l;5`CAGmZD3iMqfi zYkRs}z+(}m@f!u+g}T5e=t77SXW<=|+^ZCN zU_R&a`AYNo7ip~M1x=8X{{mn3!p~%LQ6Jf#nI8U{OZMdcit)(5wh)P&4;jJf(<%OO zfBZT1zrxgi&#tL4e;Pc9^)10h{>B%D)AQk9Sns3$MgB82fySH4{pS$bb86JT2Ax_# zU`j&NU$Guc4^Ejv^ocQlo03U&?r(e1-i7wc-&T`;r?^c1<^bL;?0IY|$2(9L?a3eQ zz)OFFe?!01qxx0$CI-mf?~uOf9!2V##sl@Prul8qH=nN=T;CZ|fAIf|sJ`p6v0k_=`gXZu`mQ8> zxnlF1&kwYh`bPg5L1+sCMEZNg`b+bJ8&K*i%PcST7hMkq3t3s#Op|=0>8ocK@`1Bx zLhDz~pFn<_!QV)JhsEFQJbv0q{!_G`*+u?9>q8x_m!iFjZ_|SgPVb2Mn}h3v{Oz2k zkK|8yD_@T>|5m_$qKNrtjfZ64GejWQ?}Fgiv1ETfKH^1LaxSl_c+j&8OTUU=ev(g* z$(Q3qycD06KEU}+(+ep6_-!y5t=}|%bRI8t!sE*gf2R0*1$2m?FRMC@+K+lC-N*v` z!7irLt|zf@+*ruZM#?W~W9yU550JlM{>&wQR6+gtPT21W04N`T|4{sWg&dOFe?amd zjm3B96Y^N%yKac@*>U1Ia$quv)UO`MjK%B0`Pzzrog%j(zUw;Km&bSHKk#RKbWHx- zokZ(vS+Wv9*wgHPq<@<+KHHD)M82WqtD}6!;r!M0566pqhwPUUVdpR)_g}Kl&HWSp z&GSWv%r8kEUR3@OusQhp{6AFwlx%wk%b6Mv>zNZ<1(8ql^(@&&(T4j6A64J9I%e?M~qMHqyG3|8hbX(kt}jHl1BU$^nhNp zPojJW`LwP7N$|?ow=@^>Gk3H<$e$jVs!sxa#D)1hRGFF{$kr!QeIh0h#>aR$S*Y{< zCi;VYGs#{2G#?;uK`=8j8qci!5dS?G@5cPl`h|c_1j4_~{(gw&lW{&^{W1AF@TU0y zyvg5zH_QjquIw z`&rL3u7JPJ;|_|4;BSoFkD2lv%4hr}Fy$YVzb8M-goT8@?j)bd`&9E8Kj&WQm*+E4 z|BK}_{CwhEn*FBv3*y@i^Orw`{~^A8*7?iGe8rLciWljp$VV{WTn_R_e!c?#H0LYx zfj6$tv|d^B6~p?Ac#(zm8q0}@_t2kdJ;(ZNTE9`x!g?L0M|-SCvHjMdW?ru`UkM*w zuSd;S`24oyE3_G>BK(o>CzSnRPn;&QY2hQ)yI$s$zUL7^X zE5rFZ{O3;MRdoN2_07EBo{!HLM%!=i=HjUJiN|L;Qlrm{awwiWz~}qK@sM&b&1H&D zSU*ks>#Y=@68Zj|$1~(d>AUSbf5LtsjlF^Vb$3D{#q*@`X%5PV3W9bD*9kn|#rh}C zi;iw!@!}ccMf$(<{@WD4kWc9W{^>*X^9RC5;~Dr*M?9nbP$X02cefwUO#2OU{KELA z{RZ@Hif0^e-hY_(8^D|5+YrycKVdw(V)7SjJj3TnG5dqxbet#kAC&rm{ji^zNT0{> z_4yn=Z`?ILwm*NE@6Y-9DbKgxoDcemhW+>YPy`$a>%bJB(+w*PiIN7;YlK!wVk z#JkEn*ng*EwFHA=zl!!Q)T8@R6yz2ZE>a2|dXM7MArehlZ~pzKb3AIB<Ncr~LCsojx#6q&oOrW!Z1@{aRE$CnNdz`63A^ z)>k_@2yZ`>PW{uzEwVo?cK)AtYx0`U57O8d!H?L_q63`&ly!X+%TMt6fhj-v4C;{T zez~uK`x>~ff%_V`uYpfp19W(fP}`4hIk4_uT>t%tKZ<17OoUBJqwQJ-E}xB%(cK0Gc+3 z_ov2tEy0d@QEE{ly$Vx3cCBplM8ONF(L%v#jH|2&kirpTR8h~55KnxJtFM6(>Wf~Y z9p`O~l{YkFG5p>XE1uFFeW`Q7Rkj76C*galDM%1|v_!BRE%ESC#k%6Rr(B*FcYHjO ze^J0dy#9HvPZXZ`c#&%X55>f5ALV~bO!vwL@#v%S_!(QHX~{Y?9fffS*^`Rs*`U)T zkHsH26}KhXWbX)DLS)A6X3)EyB=jDYn?{O*kILNM$SPq=QiESd0d_Ag0twT=)`Dyq3go3)|7ZNR_Lh+ zZ8T;btqM?s9!KDb8`8lVU0?Mu=AN5-P!Da+dNhv>SXGSt2v}jAo}%YC5`V{63T=dU z(b#H5v91^~o^GT;)o=V2DAsSv_XhLJ~T6Ag^oPK?Ky zo?7j^$~V8ZB<*SsV|dsJFEktq;sySxL&E{amsK6#Xff5FLR~RQ^cYm9I*cab59)Nw zmp+`eY#>(Ajb>mFzkuivSnQ$xe^PBory!L96(9{~Wy=h+R zNzr-f77SS1$J4fz=JQ@R>P(LlgRK0=qgQ!=)Uh9Fsd?=(*PjAjnMJH!Hp|+@zb$N7 zyUg_OSlTZ|)@d01aymlE^d*OSd#kZS919BYcCEsUueJZLvphBx>P|r5h z2~TydOl?dmGCbM=*tBV2V~wWgxDcB?dq~F`0y9>6yv23}l*7qh-KrPVsn=KQMM|Ok6fWIH>p%3_&X^s9Y*2qs|%+D@r#^e68is3> z-+Y$$c{|(tyzAFApNu;;5$j`7Z|{g5n+F~gi`jdbG3u#V`?jvHF2P=#@w0F3D!h*l z!?D{|Wz-S)XC)r-85SMMTF%jhkDKuQ;`?!8`ycCns!X6di3D5s|WjSM8%J9GCc-`pvKcpbC1zqDk+lS!4q&E~BPJ*IetwUXD_ff$4 z!91tB@!Krk2R|Az5iR5Jkv)j#baA&f(TnGFWtd=8c?wg?#>A6ej7agEu2?_HU(|ED z6fK(bBA(Ngsld@|^f_G;qUfUpcX&fHogdR#+IwWZKKg;O0K`@XQKwBny!Jnq?Y?AD zt+hS=8PambCHh@K|D=iHkPT?Pqp=h_N<(B1YoFAvN>2Djsu+rumi$6~CN7p#N&9`- z{$Ie=j)~HpBd*k*Xu3r;`i_4Y;C&WHA@XLa zywaeu@PBkQ)Cv4ucpdRX{hNwx?=b-_P6iyUFT{V5g$|GK=jCUHj^o+S568s+*OC3C z$BTAn|5#iXJt(eOxRCrNBGmsAfHXde2{nF`08+=sB59+bb!fjg&iG%E?QBsXe*0wf!PY-lyY?|MkrvMsf3Z#;oHjKdM0c9<=vZ#_yNymH(;n zRXYWr3_!g0@5E_OQ^MMwek=-6#}6}`^nXRRm+VI`R{gI<`~Mn8e%1bpYza5z_P35t zK0!L&QYQ&5mrOaJx~ldBzSDL*ip%+F87?|s)x0&1N4%c71zcX`v{^tBMvhch)Z~YTmSPZO!(==5^i;4V{}dR_E8ZY^d2$u%^vh zSiGuueNkh5%kqxR8>^e^{cG1Xm+xF&zn~GYwKaqD^GU zqq(B8X6??h<;}|-x~ z-B!HGTT|n&Ue;XP*jBP(Ta~w{c-8W<)%lytN;@~TGr*(M$`T=t~Uv=&vHJbiOi%26$1d zv_#i@3v{Y`>-8nHwX^rtk%S2?9~@_b2|1LqV<=)l6JI?Lzy@)tu@js}9u83PfhzHy z_q>jGjYe@wyyr{{dL9u+w6Gb*RY*Y}Gm&%FWBZ12A!%%2E`c*f8{~qn*(bN7W9(>6S;6DJI_&LMvz?hm> z+70-B0f`sUS4k7RF6E#x)b8@G(R4RB+scuc}+ zT%0CG%W-)};~L>5Qcmh$@kXR8CEVK2MhTy4=||~{`Z1mobsPGp=bWhWya8^cKYj!F zRRi28Z?4q4mCuQ#1Aj z1KcQAql7CxDEejzx61pHgj?k;ks(Kon&_-ZPPQx7XG+wR!_x410tI?@DiHYklKm{3j<%@jzz$x+DLMC;GWYcB?xN z>ZrDogF@c3VJj&K^wd!zQ>#DeI~fjp082< z+}-FbZpC=?^A)wN^z(QtSFT@G(&TOS)px9FYv^po@VB`8<2Gn+^ET9WwAD6n#)Lp% zPw*+KLKdTh%i98q;C9Dixed2qGjCv9du_YlS3mN1MFVXO-VQID$KQoR3M;LIBqbar zNaoeIcjPa=Wo_Igd*NF*ZrmSCy^{^Rsm&+f&5QZiu711qF8yU!vA}-9_q_-rGSxEm zbX6v(lV$#)kaC$;RYv=Kg2}WX3irsnGct}Tgo;XKSU7wTfsc&PcQo*~y2#t-epoAu#tU9b}6%>kJ zMQ^SDcRBuz+J9t0D85yH=@8y(6T+t=0o1qt|T-e1yrk zEj*52u?3kwi6@f`H}NDiF^-eTN>C;fE3vQbaU6%R@@Kq~bu+|v96@Y{kv+EL829~l zovPEPj#`L3)_?!;qU`3JulBB8yLRoW+EwRt>#hx3Hrj1ACXto5_^s-Rnh{OMjJg?(_(FbLR@)0Po+iS*+^YdA{LyYuV5K!_H3RCx;p@Ix^0@ z;z%c1f5*~T)xX<)!=or?w>f_1N8QEA4_`R*&;u8i;QGdc4_x@a?Y;@{4&ig6AzA;w+(oe=N|Nh0X@I!Booq7CC@KuGw4?le2_OlONxPRH>V+}8z z9qZ_RY3$5HKN-99^*6^ZA9@2k%ft6S{J@1X%O1KAM%|Zx{Pb$)5rLlU{b!Hjy z+#J62_yZRn=zRD>@lVc zW5t~>jotqEOJm_Ten0lW!^E>XeCff5E}Q|+{UzTRTfFhBV+l{ZIF?xV;#dcCzI^C) z@T>}xts0*A^o9G29v@r$5b7#owC`{&T`_Ye6dik&R=9f#}8 ze>t2OJM7HSd9I1?-0YiZ%647t$Z%b4f7UU4`agUV9pAD|guhZf-2Q9Z#OZ&o9`4xY z7!Lo`H_`rt1F{DjJHBk2@NFM;p8gDYN>JuG&`yH)c=d2o+VQpR8OVDIdG{i(4?G{* zhfjYO_$Y7-@aw?e0PY6f4Ezh=uK*taz8Uyg;4c7w0=NM9JHU?s?*^U){8ivj0dE6# z0G|ZD3wR^&+rHuUdx7r&UI+Xta1i+GxE{cDC-O^>{{`Si;LX4bfgc230bC250eoi- z^mYtGpNaPOfmh=CF0Pfhj^SF0>sz=k!Syn(i*S7%*8*IBiEAFN7jVtN^(9>A;Q9it zvv7SD*Hm18gliJ6KfpB}*YDx#!1X)0Mxe(Nxc(8>Gq}Eo>z8nS8`rRCeOvm&;DnAU~4BhJS-~JTi+7M;vVUk4bEp=2nzD5qSMaYnxvA(b|CB zb2ji6?l04Q>e;}%xF4hYl(T^kasNKu&p1ov+(nr4y`jb|U#PL;iMXrYt)tH8^FnC@ z--(-O`t{&iw&}FcwCqINmzSM5Fu3f*!RBQr z@}Zy9aV61?dblv3GEYc6CK;)u99Bt{HX;;cSFxcsZ$YjS`3{QL#M^iX|Z85(uHhAx{&=a z*7;h-^|=>y4N_h1vw??k{{Y>`pACE#_b2JzbvE!N-2Xk@$8mky=Y>uVyz89kc*_Z0 zF!s2OE-q;L8`$jQPyKDvm{Ztk0d!gfofbi-MbK%HVy9OTOPTFt^yO~U*GTm_&jvn$ z`~7t9I2$;C`)0bgpAB^5{&u>z>3xayI>G(L#?7$(3;1+A;JkY5q;rDVM%K2!;yKar zuyf*mf$Q_iN1_N)E`_J0K!+jR+XV*+Bv zj=}ZAw}`xA$Z%2~)d5-cxF#lq8jJGJd?CrkhVv4bz6Rgz;_=`mhy7>8m=ABU!AI>o z&ek(F;f-Tw1NQA_y>T{Nx8aKR*tg>9IIHq+Ia}|P{F?=TgW#`^!N2Kjy<76v3I2_O ze?tuZ^=IpoC4a5puMzy!f}fpOT+M#whyI6F)C@Ox*u)aVONTP5Czc?Fx`cV&pf`e^ zF6awm&<}u~Dd;7W>8osc=SW^dE}Jl9&a0ki@S=>huKCC>jFG<^brkV@F3XS;ME+7y z_ln7NAG#OxGC^M#gMJe9<$}I32K}p`uM+faG3eg`eT|^+jzK>Hx({?=14Hj;k-uJ) z`$UX#zW{xcpdW}q|19WRV#s~HW;oI28Rvc39i!Zbpx-U%M`O@4YEgcsUjE(~^b*kT z7W9)b=-a4#nqL0PG3ZZFc|reH4EnQFewJST*D>fvswWcf2Hom6gUGj~PtJds_+#Y% zJ@L<;%>O0gkCA_d_%kN+KSBI4^1nm;b0+hDkN9Kc|A6>2C-eV^_+#WhOZ-`r`ClOZ z82K*|fA(bl3&bBI|Chv{GnxN&;*XJknfP-j^S?#>G4jWVf9_=dcZokn{` zO;SVgtDo5re*ZZJJ$=yMH!*P>GVk&YzkjeHJpM;E{6SAWY_Tak`ZY&*w6bRSlA~t$ zy{Fj3xPNo_-Q#THzxr&$@9x!!hW}7&M?E&wX&-(g)doH6!IWD9a4e+1^8b;O9lQ0@FLiG3Go7d z26!cKrG5C)S8BtT=GZ1if668<71S4B`V4qpMGSP@L1#Ps9LUYE55fAKfOVVt^TX{= zg((kvmow9zCE0X8X?Nj@@*i&hp|W?U^Ch!9cD?+CzA)8K{P{K?k8j%(W7l!@&hNqZ zZ16qo#jtLU`yKECDk<;D0d^*?Z%r};YPamJfbv<1>>s*4jF1#50#o!6orkzW8 z(Km7Cc-pwv=KI$K?yPf$KKfqSHr91#J(}Hb=|XiSmn39=^`YuOyx=2G1JEznSDNqWrDl z%ka}|LWR_M(fF+y9xVXId^Gw9a2@b0VC-o}KLxx8*a3_^@915?Ex>Ox^bxMV#}#|y(LUtg z0sJfATYwKS`0|c(qd#WwQ(T|M_4tnQQ9tr)Z#_5q&xCIsAKgQ^@!aTF2se(8ZYR9+ z9Of0^o#Uf*gm;}AJx+Mn_~)oy2w84jD>VPnoDgy$kTQW(x&*j@T_!)i&evmfX>3bv*%>NgiZ zt6%Mhn9Ijju{dayO`Iit?S1{h{y;PC2K5eOZ2AsTMqYk2KxH^U6uYW&3~}1>xf@acC~f)^#r#D zdi(C^IC`|R&A>Z#+*e+)s@Y%ZZ|xuGZEN8|gs|p7Uz@+RuM>oh?t`~=boa0F_jd*S z2YP~Cfqp~I`apkRBhRYYw`+sH!r$DZ`CB`>NU)zi2B$>>e;L}Tx9cO&jK20ju&wo@ z(~w|WUt919T7+H-wsn3aybbLLd?Xsu+SYvdp#MNH(AB2*jx{4tqU2zhwFEjl4+aCh z?R?Pn27CH@XvA~W($n47AL#Bc`v|yiOQ5%_E!gsr=w@^qZ;Q1rd8U{;c?X&^J_3Cx zo1^+H&;7^@VcEC5K~r_GSX9h&u`Ji8#}OhDx%TBo7;+;F`R^7MPImo5zaKFuLis?a zzswKRa=10%@9Qcr)BOD{2iv;Ks{98J2ZF7^Ku4#)A8VbzrKi*1+=l4Y-`a-tucxK0 z)sIlyA8hMCj4eT{zj?sV^O;$A`Sl4u3`HO^&<{DN=x|qepS7|Zq0|3yI}T;4rNGvj zas8pwfB(IgTv&E?wRM&B95~R|)?dc_{_Z0fA$s_~bL)n!ew~6r7wGH=^u=J1!>>0G z?CMJte9qG#DzfR^uK-wHZY5!0LKBNqKBO z*D+V*wTr8)SN0WUll;GSeJ_jptUf^5IPt>EPc!+N%CySLK2 zED&sI_x5%9%ga^;AnrRv2zN2COfTQ4?Cssr?Mf1p29*3p22z*oXnaOoWyo0IFW4vj_Q-Z ze85qEbF)6%pRJI!9m+R^m~-_N+eJ)f5mIIZADbI0>ewSNTFL2dfnf!XGX;i~IQ9w*D{@>U zuutGJfqOC0P<fIFj;TE^$n(@hu?_Yq6PVgZ^xpMC9&{iaf^A8jY~P5$vVAlW zo9RA*B|RjtY~P5$vV9)-2bI_9>8wd$ou1B4DfDD^QK2U@$EtYd%=I!KeBu;Bmn--T z6~Z_0IZq$>NE38+dSF}l0{-V8=R7CX${bna4rij9rQkl~!2Jr5e+Z9?y6`zjC7>e` zT|X4ptK#|ruFG{6axK2@$Tj#GDu5*OWh}5>&o=S}G@fEfewlx23VuodYhMTNv_;r| zKq@W7ucWf9fdzUM`M8v%u%`#a>Utk=a|#>Oz^I#zv6oUo7cnFi`*I395p*YzsL05} zsqFbuHl)pTlRrf+Mp8-skXGa-|C2aNb4F;0FUWvlU&&*SYO^9_;u#rm*yp#iRe^hGm~px?W?Jd!`Pyf@r5e?%h3xxB2PokC9PU((N=6?BmM>lpG2@etfg zm^~a%`ad)fiP+%*U{9j3lduExyu|knfOXXfb5na>bFfW@J;{D~%xdrQe#T9u+pCnf z&xiVWLj7A%e@H;~I@!+|wcmwQc3%SO!^Q_~cC*(V7+=mvs0#(CevE&mKOt{O^SIfQ zXpb2G4`IAg|MYsVB(V$IVQ;sa_ZQh4v*<;C@d#B`w7U*v)uFLX55GB2S#h$-RWn1r;<#l^Mrp-*Cu8r6+|3Ux9?|0KAL|*Xe!6xKX;fFaSMmg%+Z+~qkG8-_$EcDDoT8yLRvn=%N#E?SIiPAA1$sT;d<@Sl=7|j#* zi07D3*qolgF`uvrzh>hY!&m3WcqMt+62B6sj-O{pfAjkE2*xXnRYL`k(7wmC#00XZ z?r+;+KPI0c_@nlpDfjP!Kie6i>C%W7GVvO|9_%-pWuZPKE`RV?2EJ+^#gA$;BCx2j-eUcl#{*Qs|84Y-n@lpKj|bqz z9s@gQz4P(_-o&3N>95Bz;!l_Om3a2lVVIlN-y-Ck_h5P}V0HuEt)&4~PiuFshKVPx;;Ia9PkAJRz5A0Q7vG1Z;cGMvIWAIp^ zJiQb07FgshjN144RI(3-%LwVq`N^JMY697F)W+>oDDf+>C)tPM53vWS{Mp z@nE$NVM~0cp*v;b{%f%hjVHx^*|fY-67944+niZ6ed+eN<$CDluas5!#RtYp1xPX3FJ=Lb0*OK@lqd~3G`Bxl0O>rZDF)OE^nW-e^B(}_C>tmlkLrey>Y-2U60uY`0tbCuy*(t;-^Qo1l~Uh zqJJJmJV^FRB>W=$;eK6t?$3zFVgCeke++4H#S~x3{yeFrBwjT2*Pu2_>7SEYn$mwK zwRD^pMEhe!Y>;93X$G5S@J6|u;`T9fSov)W$O#M+h?Rcm0JS4|6&Oem#%J+Zf@k;YS zTrtH{vOhkO|ik;^aY#Of)nx9SM)iKFG6n}yLOd7A|@u`nL z1PCLh@%Ng0HLw1!HU2!7@fTn0g?!oHgBl#o$19v-y`66Sc|7X)n{K_0$MJ}1{F&`x z9dB}d9pU;#$J_G~>;0E@Ug9{te_xh3o^U)AM56H~_IGkUEu{G7^Co+z+OYmlTF(km zTDKR?hiRtuY&+)PBwHM-t+{`#{yBv6ZJ%tPwD(11nBtklQ5C0LQL$%S!3pO1O{ow2O=Z59{_h<@ z_sIF~o#=Ti);Hxm&*~q*G~ex@Edi~Ee7<`gf+*?nGCyzmcC`QKc?SzgOz|hqC%$9G zv)$~zC^q6DdftKh==lc%7B*AhAuX-A2w0A%N40cLPdE5yb3EJN&#=Tts=uAoa%Y+2 zsgv4Fm@_(FPHOYg&GFPpZQ*Q{KMlJvGk?0`j}%Xt{ej|lBcXl#`3k#7Hz6?mbFnHf z1IvZRYbV$DcprEv47`S6@@JesdO=5m zzfyk`DD`pw`kw4BnosAs(0ztKVSIbc{dH2y_L%L*`}^}zemcMOQf<7xQq;FVsSo;> zFLbUj>}B=OAuZ1ZaYp^*kJ0tW0e=?f52F7eFE>5KE9Aw|`kP>$UqPRnkvw0}`S_Zb znOG=rJk5_ve-9p8XwskTwMfyQ){AJoP3>D|q360z8TxOBe6>G^G_UKlL8tk7p2%J=gazdrT!&(YX4jNZ?e7fT~C_yq5d<=BmctmGi86! z`DAoHr1&a^f1HYzC;OV^E8`dKwe~OB+bipTt(5G6{+?&p1K1nwUkB<}`xEs1Xn)61 z`MFVh#>3xpT_aLH#>Yu*zNtS-fld9v?W^|Zwe7v#Vvk2qAI@`r5VZ#%KX3>+f7AGT z-p~j0FZyFXFp`L0#QBa9kL0<|gDA_Bea-yp__4*Xuh;dmQhupYKE`;0{pEOyY43R% z$#Q&gd*@{)O8+H$rxh0(_0j%yu4~l5H2>tgu1Ku=zc<^t)adW&`iJ`SqAVYyzf^lt z|ES|dwcjJ4tMecA&$vnc;4!nkCi}l?e;oO6{}uMb^GWhY!(KXuzY2Td`9J9MqV50F z_p3_$J?VLRD(#2y03H5A{)O`m>-jaEhf{>r+pM35Tc5{QX-gM`@Hmmt`xJP82>c5| z#NPyd*gtE4dK>9JeCYk4I~PV+tBu|l;_ox@_lFMQp57mNC52^ZY-tF*q!W9kR6TFU z{swgBxkJ#gAIZ{Kgr3(kdcO%c4X@Vn_KZLu;F&?@SPF9F;v@9FNAk>|L&sM*hCDkn z>U%BKnW4owOc?U*JpQ+*L;hOtbJ{(R3VonoRghO7;`RW4q2@re zMs#i;W)C{`=Q-m28PpeT5BTvu)>3-@)w=_XWIxTpUuAJp{lHbIkE|{1P4z9M=OY|* zeOwMhAIKZL(-C2vDC-8xffVQ?_KTvwsu1tf<6x<8)k(;g@=#wI>?58hQ2%oM$lkp> zdY$w>L7Eol90BI`Kz$k7V!PRXGlOw)u%*AW?sS zo}m>x4f|2~`_NrfA;|~5P@B!4cSyWIOP@{cTS)X5wzInjJn5tln>#O=oqibDneHM7 zAUv7^>`r%i1ujYjPEL1~2|SkTMT+F+N;v>Ji+HP=qkiXj|e*e?A{QL-uMA@9%NC zbAh&y&wrdw_8e?*8}ltN=Bq6I}-^p`tm7waFmFu!_(3zvb6 z3Dz2r|GVz_R!Jp84!n6QlJXuX~byqMxMjJI{c z^5s}x^!|i@tqU&q(eqf@zg61WQg9;i`3LLmRy%#~fsbeKV|^}-2X(Q={S))W+O-CT ze-~=@#aO`ERxux~3s$YddSb{g)D|!H0`qtX{Of|N*URHPIoDG0c}bowpC)Sp+U8|K+%>4C^~vXyA&K6~I-5dHt0)WBp>Q z1%2kRQhW;$@;rE7gO^`^3)UmH+$djFi}i(76NbL`^|8`27Sg}mi1WgA!4-8ipl>kv zSJq?xM}Vr=H}lxqwPyLApq=;Ml8e|jV7x=#+T~In-haaT-k2c1P2n=%4=K+-P`~;- zkot>7<1x_bc?K2c^Br)OwutW^`MSMByg)2YO#W z>orF9XX1HCp|%{y$;Nu`Lc9Si{0H?FY8ChvlR<}n-9+DNGR0GNJYO*ASpSQclRh^a z@!Se5KT&#`Hk&_>kn2sMmY!**qkJ*?*LRaxKMS>tEHfSP6y%xfFT!w%%BS)ePrlhk z`Ni|h^fYY_>NA(m%r?`hJjFxNcxtKFOi$C6Kp%7YlKEyjm4^X+bBy}eEHaPpG%XAD znagMAsO1+H!$7_zR384FrWJ!O_|d;df;kJ(0AH3Ne<6+!jrOB|mH{jLIjGQAO#CR1 z`2hA(=~yp(%Z&QUVK1|O^YTpcss5ZI@Ru6$@<3PREe5^ZpjX0vW_jLxt2|Wb%QNb) z0$r7d{k9&TbNyGs-e!683#{@`p|8rQzYKI$9>x1BMSWRXesq8ITB_$rFgFJk82gDT zZQUAUentI-T7Evwmwdkl?A7Mx8u6N-qkdyPPt(?|qxp6L&A-o({UdXYcn$nnS`LrT z<{IY>IXN_b`1nA)P^HbyG1c#^8l21NGG0YIl1uzd#3N~1E(QF0eVC6nX*qL0Bs!08 zNdLL{H2&rOwXnYc{$-9|a&xJDbM<(}1O9ae#(dxo78LOMz@x`AD^|e3NIntItX-wz z)in9DCHSfk*KIhaq`5@S72zj--ux`F3pVLoNq2KMs!*-Q8t z(m%f(^BJ2*82*p-JU1QdyPLk|!Q+Mbm2@6B-zZ})A5c6nLIa%kKclhMb2>hf>p>pYb2@G{&zE*=?2cUj8Fb9|>9c9PJB{_6o@YeqY54x5T7IE&esd(4k&gZ~t%vE*-YAdy_4j}Hd9uZei5x7<#VBc zFUin<2IxW`E`MGwt><$M`MEgXh?Y;&@_^0yrzqz+M}qTkaL9Z!4E<9r=N;a82q5(J znCmwW)4y3hEq`YD9-Qw*=|_TIFBI^l8u}N2Zr0!HrS(4FsJ|Hgq|#A=ufQn3Bzk_w z+wb+#`pn}Q=vSo`EeP?!gY^~Ti|VI%hR08|-`3K+#`+A5^*2pje`oUbH_bTDNsGp7 zke8;-)Zf3R^G(!`_$tl3zE)|oW~%G+EY5FQpS|YwIZK<#x;L1MNpVS7hiz{ng(9dx`b@8CvhX-VieQdJp{zwX~TG12Rp_ zukK(&gR!3T{$|TazSmg)y>!4UG0rd2(;I+cVzJ)G$E(#i^+(_IOlWP@v+43st;iiHhUmRP1u8c=l`+JDn=d$WU=smWneo zRh*M!!dQ>u<1xSK>mT~di}}bz$NA3Oxhl@hRdHUPis#K!aelsv=g(Jhaj^+QANw4* zlW9E`zb9aN*p07f_b-#ZS_3;$Kzc^o1eh&cO&$fPl zNB_R_Aeb?0ys@2q?JE&C^Ns-Hd)xH>zxn-H`95V4==45iA-&IUexEXLQoM1G`TIn8 zzM7>iH2vNKJx@<|mqDC-KLF2@P&Xy?BA%Dw`Pf1)5T9@Oe9Kr9VVXamk3Ns$&sX6O z^!zgq^abOf!#;1}`{TOALi#?E-5L)iAHy%jgCF0g;_tVq^gLFD_iM;MF#po`nX%r^ zX14(G`hSji>HbZe%Eb8x{hk2UbNzfC_2YS5p$~Ajm>=!O9QyYo==>h*0qC=L5Wkp@ zGc<>NmrFl$A$k?)z7b;3*9&`a4nL0(=gDq+Fe{4jdjfb~%*&GxQ~T|~*-<*|Ri&ld zX*`Ja#95_f*u4tQw3kIO``1r0JJ~Oi47Cwo0^@~N1!wT_7o}(N@kSW&Y~27k3Z)QU z7U16pA$Hw=;183&pKtLet33zrER3)w_$o~TD39;IB47L_gna#&3gw zpH%5>e%~mM01x;B2aWH01}OUZxz2w&1sJo(4pAP@Tkjdb?7KtI=hJ?8zKiAqB+MVB zz+R`9L&8MQkIPqZL41LN?QuAeK?;F?JNV;VG~ehrmE{NLM}&OtAMRkWljf@^-5c?8 zjDJvHelR~G>JvC_zCIpwI`oaF@fIy#>@DUP|M2qme6lZ3zzZ}lkB18Rc*tYxY6|&$ z5Fw`lo!hHP*aP!Rey|`S+Q;eZD(n%G7v+a{HjNnl;SOde(tM1B_CJF5=OodAgpTR? z`2xG=2<8XM;rh)lQ0> z?2pX-bMI>AU4S>7&GnJL@bQK5u&&11f3Rk+?C*HAU-wVmevG#at$_CzuLJFYy=NHy z{aPMx-+VTt)JO5FY#;hB+2r5ObrpL1%=Rr{SB&!Rpfg!tFR1@ZQD58);qTPnz^)Xs zKN9y3YHubRG0Ic_kp0c`37?Ndf4PG*@OB`j5SN!9bn^3ep2ho*zfZy;)t4W1@%UcS z*Ui`cN9vFAPPPB+accj#T-fj#^4!7rRI(2eub=!Qn|W>K{-Oz5+HYNj?w`Cq-haBj z(fZ@{_nSx}^j+7e_Al8dhwU)ruWQ!ZBgY{9D*R zF^%S1B!QD=(R?fMz5B4Wz&B3y_F%0=e^C3){Xy}e7(aZz$A%SHpTFUG{F47*F}LgXjzj#e~f3xZ@MT69z9>vc=TWKewtn)g};yT zOrPQ1jCH24lVW}Ye=2>y+kVW+xW8d4rTG1sKK?q0UMJR1q^w06mRjQx8IKQVefWi z&?Er+t{?cmPG%!KUah0`kEbaAj%)BM{sMcd@2fr%dq3qF5#P^5|0?ls#8x+;v2!@i z!taN1ePZs%GDUgVPc45J+WX}6)&3W!E)V}xzK^Z%Z{Z+jeXf|kFNoiV8`!}{rfSbr@}z#;K4y7jpHoxIllyH& zA9cT7H;~MpnMxm7Ue;e2%fI$2v?zGML+Exsb<4E{_qW4?}{+*@%IO*p}Zt+Fl2mH56+w=Y$(1K;cbJq2{y z!5+8Wgh0DGWnQ8^IWgsZ{y@VKXB!{1$se2ziB5>pk1KTjpa74KjPl3rN_l>ez}U}h zwwx8HbtkWt*+wE%tNh$*v*&!m^**a6rR<3e^_$ZOGOt0Wt>d-mwDq_aowknGqI;*I z7fnMi0{q^PsKQEZ6f|3 zMn)nynoMLNorg{mV$w%+IvvYY*BwbD`PJ*$ny7EsY;@{jt4~IqKAjo=+Ugj3>6<33 z>U68kR!80?zJS0>?9b4>5v5~b-ZI1E9Es#{wiNgJ_nF=XyUm`c;CK}eIxN`2KWJC! zb_+kwq80gr7HoG}=)fj#PPC1~A;ocLNZ6%1=W?PwN9oac>QUnubOgOpkiGDIJQcW= z%GiB&Mv4x~lN<}(>2tC{=b%DY5=+~Qrc&y_L2AUG-A=;4b=wSsGkO~o|N_{;2U*=38>#h&zZV@8j&$< ztmrVc4UAq;`@pb)iGB?o{$v}(`{LOkoY&k&b06t8`jc#+`k2~BQ@7vfCAIyAk45RW zW`@xpP1VizMBVOF_YEBcRVODqYl@Oxrg7rLtnZNHMAi_c^Hs&5V+!DjWYjFiUz=!Y zg&xs zOH&L>#nC7-e4Mlpma(+z9=bu6Xm_`k?KUu8w=lRriTIpIk zkyQ-eu=vaM_^LhUBCYl@=T?d)V)`ceqx4C6m35o1@Kt+Ddiwj!vjah^Z7?o`KaS-X zaoC+qj%7i|L&K3oy9;#Y@WrOHq(P#G>?8J=<$X!^cq(u6;csmj^0{qj%s{VbU7;kq zn`A+kL6tYkGm*jOE(k)uv7h4~b-4%bI%5I1+Ci_?@YB#*cHkm#WwcB7UI@-p4lzc47C~u4fQ-4dp zi?+*bw^+KN-PGsQr!jRi*)67SXj=^3Vyp!eznfMlmN*irNv?54_hz;P+Kbuer{K47 z8sUt`6MRh0{D4UKQ8Hz3wwfYk>k6IhbCTR@Tytw7sZUh)Mzr4B>Wtlv9a9tbjiaua z%u`G|GupYDlfG}9dL<|tLH}AKl$J(BLkCc=jP)gt%Me|iyxv*8Iz5qFS6rpFYgRd3 z<`3P6NRl=TlzPBOQ}Gk{s0Tv8U@}+UB;6)9{odoJxbfDy729y!b{0?%lA! zpiTbu2F99nx<{q^%yy#knHIY70tFrb65UchWY}gXX8j`u{fbVv$Y+KvM`Fu2=wAFq zzRCKH81x-F-P(TGSx+IIUZl1!B+2x%^d!H`LO04^)ajP?8^vkLh^3#w;M3_A{i5~v z>U4{KKBN9qI^CilGwA2-I599rgpEdo^%NsI1Ktf|!v~kpo`Ygq+c?RwEpg*KL0}uR zRkyhLi3el5nyz)?F^N6`VMv+ySarBotf%kurrz)9r$bZI|I+eo1ZJ6LFJqxA5qymD z3Y}tEOL=t;n1)W}&pYWjvXJSC?kPE#?HFCB`rFvc1;6!DHmK+DAN_KYvxz^DeJW(I zlH4@{(LE)qFZ37ujqF|_Pu9P~s15HHApN!LyC~#`M0u?D?RCNM(_8s2fjG} zv2T7cy^DQ&@!Rk1b^Oz!rTJz5Ht)GdQj_aD6H9SSeBBzq@?n?@EX8 zV};SbJI$TsI{&Wpl{*g+(pb`V;DBg|LI@mqoxdH|ql}%xXXuldsT)H4 z?@I5$O37b(8UYl`DFt4;vpDe zl23mJ`bvt9Ngn^Z&>oe3k-h;7eo6P@L4?$o_%CMYnDiq&gfARK>FLaa7w8G=bYQQU zj=u{%-%Q7!t6QLBY7hOL=!>u$l_yMpSK4z($J9Qek96x8EINj5iC?y_>~C}|^*<%B zq(}W27fJ@<;zT5DHf+xihS>>$$IWU z{xVa(mvb}LKZP7Y{6^>rH6O1%0DN{-4=cDz{^<6V<^(b zWm58DPNdW>3J?#yaw^r4mfH)1T4hqX`$Si4vJfqhk>3dw6*bP|R|Y6Ou3wCQW&gA zpRXIF*Jt7TK$O>iOjhq>nqREt??VPeOse05HHng!{~PrYSzh+zBXAzd^9eqADHG)- zCLg2vWjWdZM^N}$%9DI0iSkWKrp3uXdkpeO{~8OQ>=FEVbtA>cD|6jNOmAhlm%0=@-?qxz8)(qj+lnL7kgM+UEijJ^wfJVyQRd|GFb{!jH&vY3yx ziSj;6{bdhxb|(Fo%F~o!Ex#Q0c_2pp>;FzK@6|8zZ+;r(r5`n*{29XXZ1=B^(x1gyvlM1t9S0MJJ?dKr`oMG6`hA$Dr-9TRqnj)_Ug`EfjzB<_w8*g zZ|T}|+o7skdji$9TWfdMw6}C^?BBn)v9qOj=dRBBJ8D|18hfhi@7TV#+(bLn-`m`^ z6XhDWRPWn!+ikr&>pC}XzU{#NTYGM=Za=WUZgSzQ1{@eEN zsXE$RxtZ!K-`90;!|m1EDqAW7w*{)ZjCupr6*wEhFxU+Lp z#ciAG4sHunukP7UcW_tho`#;9`a9OQZQ9l~4Npzm{>Gl#eF3zwadl^Hd(Zkk2X_Q& zYI-(Q?(PUw@7!8b*?yq4V)elOP1~o@XMb5>PkCoc_jG){hg0)UW=I zy=156x{XI$2C6#t?cLV7x1#ZAbFJBKyBnIjZ#}w|`f-!cbuH~^oCWwWi_LXYlkWi4I1+nZzBwsz~r zx^3nA>oy$T*VTD=e;q#Xi~XA#dRx1855OnuZ!O<_v~^QM=e|7^CZE`SD6psMKp>{f zXn)o2Y3yz8TwS(rPeb4S>3F+1Z%<%PO1!DUVVGZbnWhL zIMPzLd)oe~snq%Qb?=;}J^KfmF`wudlgciuudnYo~g();${!)BGQ6EEw(WMtdu3j!fk{245xdkzeeYrY*bm zx~K3(Lmsztf8{iGuH$yzyYc$?_FS*LnyGw6_OGb7@w)i-Z@NBt`|GsplULV%eSDj) zPd~&Q*Cnr|;`;P!shC#3dOp_bHg@mdJC(0Cn8!$G%T#Okdc;j-`{DB&%eEdB@dxIm z_WgQHaeED|ce~44_ck2fTe0zQa~IZ?y;w6UuvV;ZIJ&Q6`C1vR`MWwhg8oA-m1R|x zfr=Fs<$(h$R+Y6@t!P_bwz{RIY}N9>s@CR}ex0wS<3OOL&EImkzcbL+=kKcYcWM5E zZCywFyV|<@dV&YL+PZo=RxB?!V+CB3sBl`qYX6Gz<;(r$Rh29J75Hyg@Nk>{&&EJ! zUz;E7Mh?F#^XvbW`&aqPSNKJP@0@?AE!f@G8LJI~qBmu0pttXij-y9A+YG!@$9?4$ ztD60l{?`71-nJHQ2w{-s02FEM>ja^r``~RI-TkZl{ape7fu3Mjpx=ETRDUN&qLGj)4o(} z(Rcw59fVQpO`h9i$WQ!y@}nt#@A*R7Kcs&t&0=C;`xmIOZ>NO-%GHCVG*jJn2P$u1TgMr?_jnM!Ud5 z)cxB=Q~xE87S6yw$9IYs0zQHZ#u*gR6UK#DNI$)A<%9IX>x+f5g(Wm3TO(`VZyB%BN?3BtImGa{S^fMYkIr)*#8U6}Tt< z9U^}{r!ux%$T=nQWxrey?Um=ya-9BLw2O)JSvfv03pr)_j}bAM{~+>b$`fwXdB1R( zA;_ovA%QOoeV)RV@-O_9Gz%yUkWFWWZ@@}#d3DP7QKTk^|9em3%DS<3pD z$fqB#7pCAj$4vR6$vo$@$ftURsys)Y*HgXb{9UjEC6X_7BTRly$ttH^QiNjSaL{<}oJ)&6@$KIyMV4!Rzz z#2}$@B+tf)@-dN5@`XuwfhLh}9giWAPx8&<@kvv@c$CKFyk^Q5Qw`7mKsW)(G3&Eb z7|WV3+u{U24Fy>@ty7d_zK>V4T5^Qo;uXqVs0V!PEiJ9ue71N zrKh!RS4U@C>z}RzD{Q@vrU!xt`}}>_^s$n@{$PKg8P|a>ehv1tVpqpX_`kXn>%F%H zEu;5`Vd{_+*Z>?KBdjDrzDG2GO$rSJVe;7~!I`Tw$i zzl(K8gj|ZCk@GtITM_3-qF4?jWOU9zdFEJwh(7uUX|FlB0G8__C*q$)_)tjZ!^fOH z|6?7k67d>^>9j=37(m8i-vpiN7t6R_2>+fE^;+wbSYGKpDk@L1&FM+0x7c4xnyg>q z5N;^XYQrbLr*l~Ot@x*+{;A4u5&IKe+GP7JgYu-mHGS~^SaWo2<-J#wx3-^&jS}rs w&B?0&-Jr+nzwIUD-&*=q6gkeaHlo;jzKTAA?-T(jq literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc3cc186b225f1f23b6d1efb44ec7dbd2.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc3cc186b225f1f23b6d1efb44ec7dbd2.o new file mode 100644 index 0000000000000000000000000000000000000000..e310fd72d380632f976ba1a867b285f377103721 GIT binary patch literal 52768 zcmeHw4|r5nmglXiS4mY;`3I;8L{ng}0TIHVC@Oy{5Cs(lG1`9J_7our63IVb3c6@B zYluuQCw@1e|fOkc)S z9+1FpAFtGoV}Y_gZ{c-3P&VbC@cuSl|BP!u$CO0_|BGLrJn%yIuXEMlhhO%0pI#Q| z{v@DwoDTS={3@XK>}*#%#z#6=91D!0a!>RsHTx%&zid(WhhJWZa=T5^yr?$j~;#J2V0MBefH6#&-i15kg*-F9g|;qcHrS@dk1n0 zemHRIksl76IQ6rEho60A;0H@z9XNICs{=3J_V&QZM^6o0`Ruy`O_6s8awcBW{c`hT zNB1m!>}cewtGiF!`on=k&+Z+#>9JP_9)9;{11BDNci`b4o&v72yZf=nj^6j|)}vdl z+Bbb==1R2y}GW4n_nI1ZhjkB zvTY9RdF<%JySE^f0i82f8Eg58QX(se$n1xen&tM~-$kbq}1pb?-no ze0B4)s0;esxAi?>=5?RAa_iA4(>EX8^4<>y9)9mx*nCjyd?N3?froS61!jKtP1Cj< z-7@Jf2f}avY+%cwlLKL2*_kbqx2~A{{txW%#OX zmFiq^kuNsb8@Ol><@LOus{C+~`uQ*Yh!N!V1}@%n40&&vSm}8$Re8`K+wb#Hp2I)p z58Xm#d*4k}9sqxWE|)2Dw%FoWrT5e#-_KLO3BcCaU?1Z2O~l->0@Z!|5?}YRKxoR_ zW&Z9H;}pf(6TiJYHVFI;C9ozmE4a@c`o4Li`827yQ#V z9*>Vv_ivn~Y9^`J{<0F^{Tmgot=M3wR_(B;|6Ho>AD?h|;iB071$hoW9L9B6-CsLf z)r4=4U7K@f?AqL`VjC7rR`G@G0-yhb+G~I-iVeP!r9PMj?4i8aP#J&{?37U#t!Jdael^*XBKGLgU6ubmB8L-?C`}@&2?mVL2O`rRmP60g{sB^E=V>+ zT{evI+VISMumQX6+RL`~!r6eX}9|>A#u%&PL>20eyejbHzK)ELJrkv`qkczqHf`orsSR^2%W2 zo5-6~;nKU*tM~ZT$On)0{u1;~LGSTPbbY!#d^h3R#MStlac%V8cnhvcZ;G*{xHfwG z=-TN03}K-66zE1V^(OLkeEBYZ$9J2msYHDJa z^G@xZs^)3L{Z8^L{2T(Ge0u==-dQ=_F^n50?+tbj26Fej85lj~OgyXmOdz=D*8ywK zugbE!-_8wnzjb}M``rb(-82paFow~5hHE|l@Eqv}y%6(JozipD6Du*lDb+dZTj=Ax zfxr$b+gYP3`+kbKZJF9%hQ87XoqACxI?~4+~zrQh1`p8G`-`w3-<{Rwu7eDgrWwF6`{vmt+GHdIKlLehCwyB-=J7)Mt?Ku(f4W20IJQ|Mn4;)|S z8?4w?{r6ih4L)-7#&LV9fj#|$?EN49#0`RziN^*X2Jd7xbm)Em=*|ze&5hl9!aDhuwc|t}xZ~tkeS;_e zVsZD0FT=K5;G64x``@|TxBq0T8a#PD=7s{_{=Pb3A5nuR0@jYcIQ&=_*m0r`_{#%3 z`tJ1&9?XAj?1{CYM+bKF#yjVoX!Q+#cxCD2lV7^A`@|3OyH9@UhVH(u3N-yRf&DYnXr>wbIRfa5eds&{E(TG0m%xyPdo~iCW z6JLb6XJPlptySHhJn)iUKcxbC&eHM8aY=Dohxn{Qj9?wNhu3j){Avv98WxNBY5OLW zA$hm>MuvM~gt#{J3tA6^e2V(Uu=QUy0u0uPgRJLOBfw)ld)|88GK((|AGtt$5Vn60 zKF-4XOL+hG2>r+976E^*_GaJw1N0gRZs`c+as90K_fYPO_EbBhUoIbEdmb2J`>_1K zy+Ay-)Ab{4-$RgpwtmSmMe+11aOZ1xcYnHQ1o-d>Fm9Rb6SfikXhgAIKSCVKUvq(Y z*5@BbSgv`5IM#>dlf4(x2LIk;+Nw`!7bb-fwWIV$HpbF_##nj)a~<|upGwC4pRye0 z*iTj8ohWy{HvFvpg8lL){Bk|&E3?#$H?vjEsZUaujHItm8li18Z;xcI!2mNN9M(!B z!pS&z26p0{YERb3sc+L-{Kv49)>c^m&*=THsT%S(_WV;qK01pC?ST)TAlz=?Xphf6 z!8?tGN?|dD^}gDP7rifk_s+lesX`Un00POcd_Fqt*iQdUT7vfw&};D?!wYSoic}#L zAClHK*ewlB?P@u4$|=_$_9;JB6n?*71@Zzuf54}#T&QhQAYWlDRu4Mod{vy`SN}un z$FStrovSlgbW!rJRar4LM+KqLSVB%~U(=qj6Sb>w#qS%M9a_J~+t;+zC)Ko;)4x$mV!sR&zf$U6|We z8+5h?Hj|2bEar0U&N{+bc%0||7t4?@mLXUG$A6!eg~NpY7t4^}vDLe0*rL{vjFB%x ze*gC|JV@Ws+|WF&b@l4@hD3#mV-pom=iGY3t#KzOUJ73?v(DBWFr2d18XRd*qBk74s3SNOx$Ul-s&A@_E^4*g=$J((IwYapEEzv}4Yg7BwD0+57ZF@s>#;j?xr$+Chw`rBvs)g$s zo9d(O8`={M&C%v{?TKi8!|KMChG@;QWlQgf-*v-^J8oKbTl|K*Yi_u$`i5$C)w1ZU z=vB2ijEFi%3bj#GO}f(g3eKH{u4~md!MCP|6h*>fx-N3AajC3Pnd^}p=UA&Pvx zV>xm!ahw%W&-+w=k+KY(t#*U1O4C`Y6ZA7_I;hrwu1(XHS^|1wnhvNK=!`tS>QSoS z=V$po)d@P250!L&0Ebh7aVh0riX*9bz-LXa(Ctq#Fw$gAs!*{j(VTd7;cPu@GkkLGO5b(_MT&IUh?g&4?KE_uq4ya&Wau747aO{;z$s7o z9F&iac4*2cdO1$|NWPPAseQO%L9~-^ss4-I`KZ6dO(O!vIW*~)kj(|otHT9=dtiquk{ZeY5p&k8FDmufPza2F7cgCxkASh}5 zQmPHK9uEv1m}4ocv{3EKSE>Rx3U-dgJ9MxLBWfp#`OM3@OqWuDk{5hdPDs({)fvF` z)rP+oN0+#v;lCFdNE93N{#*0@iFyAR@70unM0Qgjy3o}1fG?$fiBi--@c>teIgizY z7ir;-?$T7(5a5v3%w4 zDAD7ADnO0LrmK>qN>LlBi{YJPWon0=pHJdaJ&zTrmzJqRHiAwk$UTnw9zaO_A23MY ztMk=XTNRLV+Lrg{s-p#J@w#g5XLvMMoh(q(liGDaK;Qq3{4F)OrUV-KUCE%qgJtaL zq-;tr72rHN3+ffB)SIaSb+ABvCz+jy=@=}&Z47PTwW+mja5cjJSW7jIc^{7dar zkxBREs~@jR1fJCpdoP6q_L{D&Nivrlt#oT@*68bIPfI!sAv3F}y z75G4FfBm9B?fs8gDb)x@4&>G6t8Z_%vU)(KdSNfxXa5!sD=Vb>14NtF27N4`JoUX; z;H6Q273)Lsa`Jt3=>rh16SO#1<|BPbpw{PAYTq#_^$RelQG}(vvJ!D(X{RoA4LcDc7;7_UBpbHh6K>0z8T9hci zKz%Qnl}GwzDa7O9>4;x5Cl&ZY6z>~1WaIG?q>{@bq;I7i%trUC5xT- zJB0GkH)5-7$DacMh4ug)r0xp;>i$agQ+-*c{*t7n=>XAGKB=;)y0(P_4vq4 zbWt)8p!^iohxn;{yciCoMD~L23Rr3SP=Q)v7Z=lDA3TSgd(zS74)J07>;?` zE32KfMcN+K?NMoGmuT9^-}`lle;ORqo_>3r(f}GJ&|k@ZEq|qB?-Dy$O!i{*%qIV5 zqdj1$lN^8PIg0%gY7aSUs6s%lxJ(t=!63?aX7#~Mz<nu+Q1(N%T%yE?AX^oUdMB#9Y`UbJ2^i5c1YKc@leM{xgF5%O8KdLf%=ji z2oobj@(^#J13Es4hCeH9e?a@&=g{SLQ0pJ0@{Yd=uG&_f*8Z-X5G1-Y-CmOdxG_cv zc+^MnLR8yI#1Go1ve=am`$ly+)um=}{aIOBVb&*^r{ht}L;a(Yd3ltd>gmjO#@_^q zM>6B@134*m9WW zi+B8>`2({qRY(c_U5+1>us`kJliI&pUmb7n>G2~+7pL*#xI%m86jJ@rH(R|;<3~Ye z{5ZBvO|-3S8mjzcUtepe+>R6wt>XpuAF{)`{q*<+dk@*p@WkU8@Yyt;Mo2#Nsk9?O zXNGrZls6MT#=j3S{|%t8lYE*FAn%2CC!Aq5#(y9B@AHb*A*s|udc4K>q2r;A`k$U} zml*eOd%OIP@wmvojFx>W6+-=J&+=rcqIel++P5Tmsg7sNue$v!?Xd($0v_?&^}xlM z_`!Ok>w!q7J&#`o`*U35eSy`L9OY`S2YrX^+{}27@fH5c;58i<`B@@ATjaau7wv!V zeB!sGxNnT)#B<1RkN3p4-=3iT!TE(4-?YKKdi*K$+m~nXkYCvmjk@F8Z-m}D9sNz(Ut|}$>$5g)$j#CHf$$S3UN5^8&qyG#zD1&b zuRkj7(o1#ygnygt8&&M<^w(VKP)>jS&*-lcClYP?Yv_`135;)H_-{gZg3f3BB~Ed|_tE%zk&|ueM|r*A*5&(sB#*j}(lic0K5(`aw+`uRcM`PQ@3l3H2kAIi)5Xy|%&th2zU_3Wz@-(#JCr2e)m zAMsqJ<6|tv$H8UjA5~T4AiX}t_&X{YsB*1GEnZ(%5gz_5N>)|jO6m0?^g(|bOXItq z{|irwa|70B+vT!3kXr%WGyX@RNy?@i|yE5!YRA{^w zw>?*VI9&~0ujUM0uQs9ptykCNlRY_AAocustRS+)uF|w#pF%&<7f2_8-kuES({|9= zk0HM!;*9S)K50E0Jm-40t0TvQ@525CkxNPMUqkI#dVbYMnG0bAYdK1KeceA@r=`uiC5hoJ87&Uwak#9v`PmFM`|o-E3z;|Hd@I!1ZoOUA3G zf1$o_CojIZ2h}^}m#OlO+#rqDm=6ooFDO1DV*Q_&i5Hajtmlr?dfT;rL;l2$LW|@@ zXn&^TC6ZeNn%f`in?gS*XMMj7<5@u=wO@qRGr(`$ke{DkZ@~Y0JfQOx8V}Gum@iEC z!P~KA7!Qg=IHp9xc<_d!zb+4Pg1%0W%()fEG z`?``^Tf&=oq0z1%)03e)M$_CyNskv@9acvFBYHfzSZ`SLc;F*{S?=-kkR1{J>I*po zB>4;HDQIs!fuGf1Ihp$Cye0|#Vp<=qFWMus9+&N}bF zTYMh>mRz`hIX=(gU+vMfe=jUP3+)nT{Gk2P@bT%H--pI$(&Imj=lVR2$B(X#F(Mve ze#~ipXS_XkJYc@j<1e2#$@%zv@!_#g@0Y`RJ_*zO_o6dD(EJo4+8GbfewlbKM}MdM z94DXV^NZc_wK)8gXFUpiae!*pYuL|w*I#sgD(A04&-_K_Te`oe<1g$P&nt2M>6wqj zdX3KKGya{9^^aK3xaTkQhi@kf3wxk1&o2`@LV7;H`5DgZ$}rzpdcJ|b^m$!ZlD@6M zBpr|bP-cFZj{YAA(fnYg&wt$ef0}<>`+xXn;|9y0-cMrw*kNZC&;Xb!cIw0ZgS;}L z&HV#fKRNZIy`1$+#@{dL^=MXlJk|40#94k&6zch>FXVJUXa1q_*=%_9c#Zj3-%d!+ z*K}URNsm{wUCWGDJM567Z=3N?N3LF)0HozTj`KEZ%G6vZe`3zreemq4l$Ce0A3Qj=nU%@?`cU=DbC(=lmVcQ47vn%zW+bFMGd^)%*9M5bb}C<9_;r8Q=l!T(@4t7V z0PW`{(fuChJYM?;Ak9jFA%c%7aT{OZN&YQ8<5=>Yk2 zzC9&H3h4af1>yPjB&JRNm2`iC^F#CP$xQ3}M;K3w&3O&{wG8qmP=|N$uz$WCWm@CQ zlh!;k)#*P4R#8WAo|fOgO^?6j9VI0skH_Ppj)i)7!1^5dWz;|Pc@>YBXdg?rZ+bs8 zS4B}?-|s;C%(t&gk;9CAIrHr~OzZn$^X=!0}u zlV8cSkv}s<_BiFYL4F0(ru?jw-tSQR;QVyHJ%eeJKaDoruKK4kZRAgMrkR+LXTp3oCbifN7myXfLo9V2B$_>o3Gx zQOBYX;Y-HgHihoLbLTn%jsE*;0nLxO?)h=?x=3dH#`urI^#9eL%bLHb^lnr zZe`fn-(mj4{7{|@>+yOXt#44D9A9?W*||FY%>3r>h$-@i9xtKqsAM?29SZY!P4|~$ zdO*Xdjq{JnjxY{nD4{=NzAH%<>+vP*jE9Bg$yiLqbUbHMe;<`B4%2v3Y{s+7WH=TD zt;Y*J9$3!(Nx#$IOOn+Yc^H33C98G$YATQVFn(0){Sx6ZzRb5H+}{vSWuWu8zf*jH z&gb?sbOE=Qp)D?N+BeAMO*~|AdEKAq+J%wppnZy#xo;O(^X+VwXUdoIcmny;QU9ep zewg|$<9PH_`^>c?ktr-s_un#(M~q*je~9H7`C*o4=p2@3=v*$(`s&B$3Q6D71s3YF zCgCzdH0r~64gX_AB!AA#hdza+S0gKfFD<=-@ux}O!q9AQeyALDspAjmTg?81{?r~N z>`!C=SoWu($FctmeUA3f?J3(MRP1UG)aPms#LHaVe^2{Y-p|wi4{87CJwQMCr&#-^ zIO8A0oAD3kBTu|(`~DT}n?N9wV z8AOMr^Jji)TVx9iPMcqquki z@^w58>yJXKg3Fusj)FdWdzXrMzEs5XWg?!-K5IOel6_}y@6uATPq*K&_Jy>4nf5NV z$Ug3$)ZQyWXWBca$Jwg(&KF;mL6|)ds{r8qrX!B4)XkM`fnD`*M`pK`PtAB zE^o%yJT7m>uY4|_9>1nRTixD>*ZFn<%hS)(%(X3xZZKuquNeB8{yIUg51suZtq+~? zD>9Yk>GroGe$W>6cla}`*PET#lH>UYj1S>t5vEr|W4;dS^{qRfo+ruBG(M0h1cmWW zUaBNblm7WBegDW=e`tI3{r@yw%;k-LO1Ql7&sZ*R{1b|he{_3m|BOo+dB#7b5%N!_ z{qyJ2{WjhHSnpRRBXj#RG{%Fx1wP<;e#ZI70-EpJR((YESwz$Nu5A_W7vTOW=sbtk z&l_oZ@II2W{-FH18h=LPL&;HCUTS}t`7cyV?P22G3gJA!olo^S<#qjZzar1k=VR8# z)Q{(T9DR&_9}k>(ldHDAT^0o@*5i2; zj~EZv+o2*_e}{<1_#8@xaeLn_KZ4_UxBN0JAB{YUXFDf{mUG}4dCq%CR>AlXwyiMjhqQcMUw#?I2h&>~#CZ(k zaX)a&gE(GfdNyRaS1q5Yv#z6^9YOOHp|{^`Jv3aT==hw4Xv4`Y2p`fB`3 z%aC6fPUmC27`6-YY5(BlZ$myFPvG*p{!l@h*7fHF$^V)9bHdL0o%~PrXXOEp%ANYL z{-)7K@%lS#=SKAUTKk*y$%elS9{KGnCeOl^Khp+P#G^A$YL1b>Gz{@#Z5 zcPMjygLosn6=i&2$o?YIq4WkH&o}1k^E~ti@^@P`b^&s}LOd@H(`-x$`wPtHMIE7e z@Fea}!9L_4vfby6M7z!#5nr3|{3*&(hrsg)WuUV{D#kSB!#|ydo-_?Kp4;s)H2l9F z&#RKZg2rETe|8~g?1yy!_S@NN5z|<2!GGrdC;WS8y87k@m1QA6s{{|Z9}k}A`CmHE z%FaB$J>d`3c%CsF?U>y$r$^BEj6=>6)~wxG5C_(=e^ z>+{U-|B=bZSE1x#dU)*r~AXX>N~f6!3gD_wZ4xpyuQz5-yC&Z*mqukekS{xM14V}$D_1A&3Qf1PP|w_ zU7vHl@wM-Wc=?uc@(a!R$9H7Bd@SlSpT0}ir$=gcd}5+E@lv0!yzx0*AwHdaYCq&J z_QdCO#OG|)=kezV?Bn_{UwZ2+qx!5M<)1ZPHmOst`WA2aIMZKx#ODuw4`>=HzDQqq z;0q6Y;ejta@P!Ay@W2-y_-&GZ?H{ZE<%5ski!Vt1>VH1?$*I(J z*Qffv{Ki+_@BW9UuZcd0rSHpc(CHx(9V$jtQnmf&H+uQ^lTQ}(J^22A|MfqPQ#1T3 zim#8PkD=%U8l;c5l}w)@2GhP=E9Z=&U2B#8EKr8V@3v6c+^n3BbiOY@AD+z68v|24 zbXCC8G|TfT+8kHatDK+veB-W$lsiz%L%x`=C*|hgBX{83?(>iPgRGBK6*gF%fgIJ9 z)=+D@5jUkg`F>A6HVGo%Tb>@WJ6CyZqLlBZ2s{zf)zLp+ZhnrlUwFuO9W3!ja`FTE z(UXABcLwsEqa4*^N>t@vmP4=GeZH_Ck}?YV7Jubp$H$PWB~wkblMDo}H9qoB2*LUl z;K4&UE_SEgDCD1zpLIr`Mr?$R-h6zcgR06c*UEcYuTGcZJE5@i-f7G}BeMAOT6 zCBAaNY4bA&pa@}f-V}e9qlebhFZEbMdT1SLEEjHwzVrxBCO+kvs>i24WFmzp?SziyoX7{a9 zSm$RFuhE@y3z2caFINeAEkEvLj(^;-nb-||k4ob^)M(?;gKecwwA-O*rya^i&OXA~ zp^W=;l!@)4wabB&w+wD0DF5YSulNW*kwmL^<&XK*CM=kH=bP(_ys<%c==Sv3fxv>Tv`f`#BzEXMDn-}u`^`)|1gw$9LnaOamhX(; zl5wzb;=(Z$FOL46KI3U?Z6mP0$%Pi3XZX%k1!94ofVb_u?W_CD@%hjN z(+%tF<%elozvVi7$egEC;f6S3a!B7Erq1+U|3E>^T_Lof@xd8CR)NYM((mYd4KsZ# zkR{9SOqX9sHu+_~9)BX$&VHZze&*}~UBAGiqZe zg;utOuMFzS$Eci{Ib>N6K0x@KbKm6nQaoDfw)D1xj_nJJN%$(_Y3m`HYJY}2 z?IkqTQRkK7(Nz006Cd`>J-0u7 zsHZD9#2bG1xak`@ZhCa3vBdAK$CK})2vn%(Q}r>F%{`yDfeCtILi-By3cq{wp?(W} z-95sSpN)Lz>mC#2*o)SjlRp#Z`1*rTUF0Fm#+a1SY3MpWGe;lr(LD>lYtG;3>%lmb zF03VG)({*0o&Fv_&38Y?{foO6o>5hkskTmq0U5J|!f__eX7rSs7eRMHfKYVcNo%}w zBmT`~;3vhj9(2}y7EN(4D{TwSc0)CF%V5OvEV?sQZXBoYQsG}f`8w$OY~TJ(pH3Mr z_22Grtr*^|e*80E0p7XX$cOiYujozS=OKsdAzTPv4c@i7?@OqARX&=i@6> zCb{!1)jQdpkNU52(^;wqZN~c1cV~AZthhexqd)Quoz6#45lqWr_X@6rl_>^B?Zo9$P^$9-Lz(P*hADFQYMvL zUP%8HZV`Bs#8O;1pDXaH@w_R&*IV3)brHCXCXq`Pc43{jNYoVU#X zO{RRgVX=>8c{;K9<}l@Vi}Ea4mLKmpq%Fn(^M2Trf6X)w)lbie$nq5T5HUpmonst9 zQRgikV_crwaT^2^FJ^H3)?>;u&GmCRZvXWte6I40qw<84<-Z0Vxj=-e;;ziU-9hCE z(}|Zm^`S7yT;yib^-{y2N=+jFX(gC}_aDK(RG#E}k|`haFh!6dkH#0$f1$u}{d3cY zV8gdx8}jkFI%+=?@s6O_?aq6?{?b}}?jI!C?*V~c04CLs6{cME$Ru@*IC$KbK?vI537Q-#<)wnssFPN-!|%v>|>x{trpV8#|=`??CxK z8peJu-*3txOm+Lq`Y9&JC;a=m2A+N^HWl(amnZh8EvG|z=v1rg4ft)}x!LOKw}K~qwWVTebmr9PjG1%3HZ{7sskJsSC*Cl- zdDiNd$y1~E-hv`^YZ_ee85(=G0;`%@>(<8cOUF&_GLYXg3*sAF60>GPtp5Grj7-Xg z7A><`0|(S_f4m7wW^gSt(=sNngrT=Kwl~)%pk95He$zBcw&UlEYVCX1(GMPL`tl0m zI%#T@ehs-U0lZ^9*mu=7tz+jx9)6wc-j;NJw7xb`8+8ifzpFl{d?rg{T=HA$8=L3N zN|$slT>144>l^DD;-vPxI{f~UEMKp4Jk6ohvUQ1Q>+0yL)^#oQQIxp%cKpI)Eq=@u z>8llABbFz*>PxgnS2e)@P7_4!+Ln78qF=pg=8W0b%$_sr>e+J;1blrB&0{yUtcNdj zHEz4rS7&7E86tls^q6^$d;}RONO~vi#`*?NwBo0VGVSB&Ti>|8vAq$o6kW9;I)lFQ z!IOsIXW%=>@p7CCQ}F}UQCA5sUWZN;x;SwB&^F+TBR=EVjDSpDM*GWQGbfjP9XL9e zC=%z*h5-p_uG?N6FD_A9q+6bj2ivLEy;_Kt_NL_zv5_h=DI+V>9-K$-Jc;?fu>$8Um-$K~ zW%`j}o-YYgPXA1rig&jOyJ(`KR1>*0_MiMuSl`O}NUXmr-Pd= zm*g-1GYs@3Ym<|u|G#1OA*w!d>B_XcoF^mS$^VnWKOObdm_NJtyQCY*eZf^O;<`T9 zrHzTuQXlC3uf%z7*qKaPXSxsd*cX*LpN-E*qIniiGp8B+z2u)DgXHi~%YOSYL=eB% zh)Vkh5{CQTKIe8b94FFW82M_nli~KqaQkpaPDW1_=kWE+&8+l$AiWw&vy_A9SvbRw zVSz+Ht;C719DEM5)9>Gf{Tz#FCN6G)UBmGQkvAN_>@4`ov+!%rQtyLjkv|4@jY4}x z@lJ{3^=(`ar_W!Wg}>=6`m8vMK8w!6AAc74KRpZojkDAXxK5*yZyr|HG)LM+f**C@ zYApDp%r(8;ZLY_GZ$eRXo-%xruRrH)=$+=Bc#($f>uVkKgN{7vz}v&9L6)hSjcBRO9h_!QGsVY%LShKlLVgi zoGS3lpCjscf4%wH++%O9ZD!&*$>nZHKh z*{&vmXMUT&M~(c1z%!q}E}!G4!r<|>)H-oIFn^Pfzt7+|3q13;3Owt%UErDjgup*( zl*^k{9b`)J&y}K^ZNvTtMSh%foFcd!1o#aX@O_{#{xgs;CV5` z@yYy|#>|gx0?&MW9l1^%KkNs5ZIezi9y$e{{jf>knZH@!mzeh2D)7wTF7RyE69Uit zrzGCU-zo8izgOVd-hBek{O1Io=gsE@p7}2bJnPvb@XUWv;*FjMCEoB~6L{A1b%AI8 z8v@UI_6j`nj|)8O*(dPKKPB--&whzF{Er2m^>n^8KeK*gzGV&&I6hg=kl}NDGT(f= zdU}1xdg3tzPdqb!jF4Y#^weLHjuI@N`K1ES>*T1A&-`+MSEj#D5_slU2>fz`pDpmr zpCj<>x48n({FuP=x~N*?2>cPF|J?%5{CfnR^;{|N%&!-C9%t4F zJoB3bp7m@Kc;+Vsp64BWZJQ^4n7>Kj*{;n3&-|?d&vtDWc;-JL@NCzU0?+)X1fK2M zDe%nSE%3Zf*emeN-zV^_=W_zj{O1Lp*9k8OJo9@5ey54I7X_aAhXtP3MMngl`L7E+ z+x3RPGrw2h*{|jcbfHBjleViZh>b#?-6+BuN3&_jGpxZ&-{eIv!3`FCC~WA z{0@QVcI*^*=5G>sUe|3Fc;;^vc(!Z1z%%~|foHp(6nN(E5_q<2x4<)hufX%VVxPb> z|2ctYJ)akN=D#5DtY?qFGyg?_=XJ$FfoJ|`y z%s(aY?1z4VXZ~q{XFr?~c;@qAJ+BX0Ps<$Q^Z3U6kl{0)_2geUWIXfn>zO(+p7rEk zkz_pciv^zbEEW1Nf0Dp2H{;t>foFb&z_VSm1)lkH1fJJ*a|NFHF@dl6L&yGVfoJ|w zf#-R4xxh33Zh>dJ?h$zAuM~K;3twfb6Z@a}YXqL{Y7%(nw+TGkl@NI5cSyYHU!4+f z_?rZt_1rA*%-<^TJg#mRc;-JL@XJkmJt^?a-zo6CPS_>z%-=2W9Orulp85L(p5y#E zfoFb?z_TA-6nN$z6!=avpB@%?<{uGw*7G%iXa4H~UvK>RhQKqwPvBY4Qv%QYeu3w9 zJT36dKO^uwZ+UOiv^zTDiwI9J9?-h9F9~XGGt54vWe@fulu6}`M{%L_{yUqwa z^Z8*I9^ZIgv&=&>JRUMXV)%^bb%^;D$Mkt1^T!B0+f^*&Grv^ec^wiJc;=T2Jli!% z;F(_`@GqM9oGtLopCj1)kRlPYFEpcM3fFZI{3^f49K1-}VYT^Y;ln`|UY_Xa4g7ztZ^g1%YS&VS#7A z9T9luzb5cJPrWYi%zs1R*$=$}&-~*8&wl6=c;=rHc=kiTz%&1}z#lR3d`94zZ_$Tg zkYs-h89tAP?4O9hvmf}Qjf`jh7=dR$6bt#xFBN$9Lsa0IUoPHxkD)8)wjLZncpe!JdSM=c;;^w_~S=$_EpBDHE(|^whJo7&mc(#il4(9QS`IdQTn8(8!BR^#L zjAwqaz_XsE0?+)Yz_Xs^0?+(O0?&F*6?o=XNW7UJXG^@{#{{1BtQL6YFA;dwbE&{H zf4RW3o;3o`{JRC7&xP(0c;>GWc-FH?;F;eh@T_M-;F+Hkc-FH+;F;ek@VxHbB=F4N zF7T}969UitCk3AMd`jS%zf<5@&s_q~{M`c2{d%v!Gk>4JbH9F0;Fji;l zey`!vbLsBq*!v8ho&$ID`wgF-uZ}uXk@Md?^iTboj>`DiZp-kgU%UAs!>7*?jCad1 z*Z3kieA)uZ;7SeOV!om02!4g3*9bm;&StaV^XEBw1fT0wYa8sAhNkxTs>YUDdxM&0 zH#F5wTiw{w7_YVM+6^jBA5mOdyRM~f%`F;wLlQrdXK;yzWJ2MirRx$6aeP1x`L#{) zwRN-V>SoNjdezLCm8)m0o;hpP)%7zPRTk4Ln-|@Yu$R}`iFP$@ReSq6VQLX-ufAjHiki41EiGsfG()A`8(}2@*YW?9(58rWV!~H7N3Myxw7Hx5S$qz2dDc zb*Sxv{Tq+hCt7XC%`A6a3qIdl|5+t#Rkff*Yux$pq~jy_$1&X@;Q%-3c+2g>8(VDy zD6d!akx+bY81EaJ^}F3lAEIdc6xvs7=BIE+VtsRb_3FlWyuBe2Z=jD;qcK{W zD+>i|awZ4HqpB-O6Q(v|>qf<23CgaVsGFLO=E$dge&4%vH@fj7@K)JS= zj`;TGSqK+=ay70$v3V0+RyWi^=pA*fc7x;ec|bf@Pnz;%KRpu9<=Os6QL(I> Uifn$I{wq~s978%%jL7o;7rh>FB>(^b literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc415ff8fa08210a6a77393bf85ddf0be.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc415ff8fa08210a6a77393bf85ddf0be.o new file mode 100644 index 0000000000000000000000000000000000000000..0239a84cc90c75d5f8f9aabec2bd498d2106a3f9 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAR99AZ0i9+G9V$45Nu{eOKOlnk3KEQ zv|5e?({gvgbR_L0>RMFV0ELKxc#xi?j_a1)h zVX2^Wv!1E}Tiq|R-M0TtFSLDj24jbYY}tM9;&s-RJ@;R5J%`tCagU}G6)jz_?fhZr z_v6^r4>#zc4=QY-Yc_W1|JqpJkq&kUw1;~#Si$$RJ^ddRv$ub(vqLjdd#cXZCie~5 zCQ@IAd(v6K|It1D!)T{#wzumr_Q=U+2FuUwAKZTFnZZ@3o*Dd>?zsxuXYo2T=froy zeaDZ5A9H>;d~E;m@W}iV;lBAN!~52p3SZoJDm=39Quyil?}Z;~Jr_Q9^1X1%%tt~a z+xHJX{me6isq-HW&1v~=`24(M;k}oB7Jl0KR=D~2#c_i=bH&DsvtfJU{=*!M$tH&%W=5=d@z{lW&E~&Rq;| zyF}ytStv9gevXIdwtgo(0(|+oli@ih&VhI8=L?~ceFp}gp8w2X=;ZP6dAyo0 zy%nB&;zYR2c`1B;{yESr3(aYLaKWUlKEiF*SwYnPzzO?Lf`17g*sQUF zH{b)BMz~3RL%QQJYIlUOg4Y1c;cY};KZ2ipP*EKEa6=LN&U5wJ&ex9`{WqRJeqht) zDxW0(Q@%F<-+L9ldC&LmY~Sm-f$tp|!}pF@e2;um`QL!W|CGPp$Oo_O{81=UJo)Y6 z2lPYd|EcJz+s@K1{JQAs$ji(7f3uGDr(%AbnZ>RS|46_32K;&`n_WFqJn7I-gLdf5 z-|E+f_pyGOM@$=dbS3bS=U9K_pIQIKSM{s!7PJ1d_pz)0vY7RsOJi5hC$Xz3pe@8) z8{U)9Km1RufA|6I(1rh@U%l`yJ2d<^38CR{vqKjeu%=F6{Ubk~3|L}l_yKn4w-12! zza@pvY}oxM%{B0R=-STLOmj)B??Rpn;Gbj}2Cp>cdn6NNya*YckcnhVp2V*Hl;$er zGs?x1*)hE3X^(^$~Li`LQd-}Jd|8L>C7O!&L_u#r8*W-BoFWjHP z^=*L~cIfg!*wtXirs+Z-(oPZpb#*>U?7k=GU~gwFn_ zkV%qlt`|3F{=K;B+gYNAEoBMdXbMp-=xFNb40L+yai^}`t)kAabsbG@jX}1ct-aH? zpzZMmo!gt+wzS9U4PCPS_=4z7Lwl>=+sPJm`huMcglKgQU7ao7j*dESOUqVoXOk~b z*Wm5wtdrPIUt33ez~32YUc6{)tFP7D(!8}T|Kaxhx_?^?2B^y~SX8$-Z)shAUS8hf zMRoYMXi459`42CxtILC&Law_&jE*L6z}I;92omsh_yUhZB3LQlYq=}D4RUzz3PBou z^<7)*wgkMbK4Bdzz`MW%!8B{2r6J(;H}Og5543l-cXs>HoN(=J9i85`&b+(8eH*;~ zR$rjut}ru9!)3ABk^|+^$qi@(+yypN;+TDwL+_d?tZd5#x}Je$P%*B>s$GZ3BV;CW zoAg@;S@iP0JQp#|pUlG~~|T$EZbm9X41C+P7@!@O9>~y1KT< z>!O&|%U9P4OkEzfYR%pb9=)S@YKi>iuFA=76q)7#X~S;WT9$Emq7}^kg2Ts;g4bfR zZsR+4GwLZ0$4Q~3yaafLE7ng>jv%N9D>HEGk@O(rWhdb7BWY2%1ha2QdNs77{~q8r zoL^W=1SMAj>7EMs-bL56)ZINjvza5ZyJxn8-!ew#O<74Kw2;}b!!WWCX8oh2hU1)Ccb~d-SxjWn2TRP^s zF|&Q%4xc-J@q#7u+%@bv3s%x;wf%I(@C~)~=3DccX7hbDPh-zM^8)y1JU> zRqM(s*3>PpUB7%y$?_65zrwxPJ>MH>XmWS7;ymCHuNy-XuXMhG&p#}@Zf8#dZeQqN zD;Ik$4o2SHW5fN!622WrTdjC)1p6j4`tg6g1U|Hj1eC41t&(bQA?N%ZDYJy!6ZFOeg2PP&=r2YpFCb6?b&p3S& z(F30wv`?h7PSHQn^JymN(LeNIBXRtdA8`J2A)kRB{G>^Hfgkio^jv=#of`3jx=!-> zE5kP8Hz_y?R^;|gP z!O4@!-^O$Oi#j{~vaa+~n)MB7Uf2`;!yZoVU+0|0)PP_#n5JCdMlW%$mdHq4uZJFhA!-*cr5^`II*u z^|=vtQP%4(X)}ZK=FoWah(Fk8cei66pKte1Pi24QWHW@ z=nNNv-VS=KZA5?6$?)UEse4d?xT7fuEhm z`_DvO;FGo8-7es5?0`Edy;1hHq&t{3wmi%v)cnx^s-!Jg>3*lc6K1lL$|0n+n z+HBmHl)nY%%pq}N_*t`}_1F1);r8J35A8``cJ$BrdB>)gWc4mA^q4;l9l-jQU?YFyi^8de@Gq?QQU4R&@ntspQhA?mMK4`v3ZO(Ximn7>WSq&oMvJ!tPjd*yFyNWaruCVz7P?-uqvI-TPk zsEhXGk9Odtzrnwu-f(e@0Z__1Rc2To!%1 zTrquDkiJ~8`OW7C+Dm<-|BN8C1py)hy<+{P`N0h+^_69om->sY2SbIdG;6j=zOnSx zyA%1q88o5wE9cK4zs=xpB)`MrZ+0F(?Iiz6TF>kvf1vfDp4LmzUd6YWK?kRI#Qe>{ z^+Eo2R?|oGC%l!f$C!UBVLwsC{IkYGvhQgkkn49raQt|(KOZ0QqBJ>|*Hk>{-HD}N z#VWaUx!d&r0v({HEyz6o33Sn2gqMnm;;^mpb9`<%Yjh{JjD?#Lt&iokkr% zy^C&S0scTY(`na}SU6!kZ^a9TZ5dM?C%a+XZV+YTV?YTim^6Y<%WIM^9&>r&h@%VhMsSuMe+8G;5gKcc$&UOd;;o5d?L~bH`NrwFKiV9yj_^l}Pwk`r_+c7*Hq4PMawd{S z{1x%Bkmi$dK4ASZ z`8)8Y`2f7h-+?#G2k0j^X!d{j`*gA|&v(FYM$j-HATQyg{(n)Udd14mu=TQ~$ob;WD z?T9g+Cni$=DxN0<#cX0#7zO#m?rw$#D0Du8{nx>6EVZKOv_HrGSR7)~{_sui@1&iG zzeuiU$x9@;;4eL(r_zZ@q=)Q(P+Jkr7Xj(K8U!PB#>}k!8Hp?FyCAb@<)EY z0{=AUD+_@)uFteyS@RXc`ippxh4mWCiHP^mpJ_eE`fOUiQP0A99i>NmtVgl^){tgi zuQ6W7<%vbpQw&W|c8K)xrk?$vz{b5g>KW0t_HsZp3KlLf*E94Jx^OYH6&6~t3_Ry1E3hW2EU&mAITC6KE>xz`O%%5kC6ZTf0>WWGsUmb`)|rFP~`r5e~{S;!t)QD-T=QR@?r&vc|lpBLp&Jh_k0_sJ6> zirpWJZJ)W8N z8|L_h@lE>;=-U*}INrSfFzq*hH^sMMo_~MBcy`(3FV=X5&y!;I2fyh!PZ~HN^#l81 zKQoy=kKya{S$y8Ob7E|N{vh9<^Yc@lZ@;+^^pg$y@q_rh!9JPq$GLy}2ilLv&U-K( z_D`|%E3E&c_v2Hi-8jBY;r>1~etbjxm@<{`ztnzwbbK?N|KtXzPO-$d*!hoje~bBh z6Z`Ec#`sncoS-v&fF<^aG(R#YOoWBR{yQ7{@2m;*`J7?@J&pE@hIr?s4M}YO?R1W@ z|Hgp|mD`DT6}PefPRD8q2E~3A?Omux_oK+yCK%6~fX^_VN0I+$QobqnpL{-KM)#k5 z|4sgc5{-A0Co?zdl&_v!2z|PAgFa#Zblsv)(|%;^{Pj;b-!CSZ^H=1zrF(BJz9By{ zd_N$Kt){~!UJUsxEeUczo!`kK6c70u)*pOc%+Gtpd4}OU{0N9{;ynB)&ci|3llVT4 zRhrT)`|C6v`+fFx5O9CS{_}?X;q!3#gE(K$qR&tH=aD*nV4g^I@VnZw-{$+ZsC-UF z^6~RU5>l+Mc5)EjemI@_r;l4?e_HJPKkw$`HJ=}(u`huiv7bc;IR7c_{wS88;PV4h ze)1XAA=SNdPXqTfa8CpGG;mJ?pSlL<@EoCb0N-+8J-@v6#}9uT`P$2UmA~KgcmME< z@^OFN_|f^li5x6ov-_Uzbyd6j=H?vj>tn3nUj5w2`~O@Q`8r3XtWEfu^T~1|JiJc>-qG>QB9LyTI6+9V-L;&{t5w+=ea&nc;e$lt_3_46R&-g|IIPoD;vb4kILg`Y^|mx>(F!*#vx=+ zDxznDPLn(qf8bQymSmGXqihM08Mm83?|PEZdsJ>3DGoj=w<|tADz{t69j}k5TwsJ` zRgz&y6oX{;Pen5OXpbUGm^hO(QO+RUjW`=U2hE(@bSvj^b*>A}qNAb{qwR&R3rpKl z;>}p0ry{h`n02&DKn;2vg(q%E2WxeG^@EsuZtg+7v^ne5JT_oeG4dl|h4p%hp5sXT z179h$QQk#ks};q%V#Ij5o(5IF@mHW&`#ri9 znutHB(=A{6fCn2U`gbh#rJ5Pe z{Z{*k*XCw;Kh0Ovu|~U@p2q6KijUXlXUC0|yRquVnb<315ce|SR^;b5^s9L7)g+;* zHD-I$ywsDT^U^ICu(pq^y>8T*9w!D_`Hx4h@&Ku0Khje3+GVak1-vqgSi5YN zwTpjS*syk)>EE%mUy7{R*q_Hzue8*WtzLiaGOqT*_r{8EH;M-2IMPdg!0h^TRCz#! zR!^gzZKxBT>Rg%HxKw0#v;(kdGr-1LP0w*5HhXrHjx_{ktn_$`?FuM|lfAlCFQ`+m zGf=0lNY^xKj;hD@88)3pDNJ2&FLXVga&x0mE;5#G;^ivD>8<1fZm?_6v5rs%NUV@X zNXMhH*f_bK$$Kq%9`$3iz8JwUGP92C28jWGFV;gJ@G;XG{aLJ$pT?MRz2SUUTpc%#nm#c=Z~E@Fj>52ND_?J?t@q*+#X-MacyyfEzKH0=tq z7`2m=5m$|Cve2I-F44%b*knt%G532C?(IYWG;=lH>VB*9|MsK7r!l&f+jxE6K8M!c z|I~HUOidhpT4?1b9=&BhWXgv|wU6#O`5a7$t{}Ahpjheew7N+@aaKO7Jn`t&=8jfe z(BIMfOU}RfEbsGnw)c70uW3FRcWffo$D-bzQ9CvdJSY~k_b_ABQ?vGMU0+jzy*A@# z-`W*;A038cx2?*UBk<2kJmNDfI+C@VqYEE5;RnR`f!f(r)++qa~`j<=V4S0Pv9w1UiEg5BVUY z%91GM%{>1n?z4S1x!Kb%=DhU17bmu|@6Y|+?`v&;J!{^Kyq`}0$-%VLvX+#E81i%Q z{GUol?ZzL~<>kU2<=_d=|9NEvW1CCyx8-0b6dO*0qF=2; zU1|4G!1=*Er@H>z9Nq^%8Zr?r1w>Hs>=X7P5U{rYuQ_9A~lU|HS@tm$$ zKgwU!bGj5Qn)4!_)0L^f(QEWMT@s?`qXf5kLo=Np(^=YkWW6E!fwBO^Rt8b0O+dW% zKauUeY*DSXJ^mQda>gb4T}gkWiQO3Bi^m;%Ls|DXNpVg{j0Klt^_F^Tierq z?#Vdqk6PNRo~`X?2!;4Pia=gZ%Jv_~I*m`?HLc4z)fT$kK0u35N{{3asQ{~drdK8gu7aiaiI$HpRQqoDO@zbDT4UzP21 zB_)kd#iBmulkGj0@sG;(%73XnxwN(Y5=`Ewrx=leek1 ze3f@Y!S{Y3c&xg~U$w4f<)Z4x%6631E~;wT)Vd(euKLzxdEmiczv$uWZAELgHP%*jY^+`L#H!Xc`HiL3U5zVSc3_M)@KIV* zTGQ3AXjwOn*SMj)y{K$QN&VuQyq40f<70^d(DapKL6X`C$y=2kWwcesN`5RleF5gyI)xB{;K_Kp!Wjks& zg75m$6;CvD7c@iO7U=4U`r?8-Uu}`UzSRf&)Rfn^t$kuO*Y}zac@3>K zP4%0q+KV@OtE>H0%UX(?+DkTUt@IWZuU=8QCVx}u@~(}oEnS;R@mjpPYtzbde`9M+ zH}qGwHoxYH#+BtQ8#gS1y{qy!wympf+9d3?t*Cqp`p#>tE$_10#arxuVq^1?N0q;+ zYh7zgbD(Z}!;<`hEnAju@#ZaEl%MB)*!##Mi!L+s^Hg7Zl{uWgJ+ZHFJ7`_$>Kcv`4s$&$_j5whfls2SK|j|eBRc)U>;*T zbMENJC+B>@z4OaohH>&2Ka;rSo_N;vh)ivgc&2u8Clc%m;c;G@5aw95I) zI2zzZvCiw0v-!2`1!F%8sFk1xJF8X#zFBnn6$fT$XUvc9DduO=^kkKS%ac zDNUyI{P&W7JY9+xHLsqNe7+*#YCgR#`x%f1RPYUer>jBH(^}N2;3>WwzFF4)QGnUE zaV7dZj6>ywxUjczCHNHyw@bK^Zvy&LzC);q5Cr0++~0p zQ^lhaPUGSZ;CCw?u{n~~Qc8Ce*QhAChu0Wul2-*+apQRlT*Ze03tYv4VGCR>z3Ti^^{>|Z+2D_g zg7--Kt+c?^dhfTu)q2lF+TBrC-`wU6bTjM`8ojH$U2P3bkMWb@ zHL{!Cc~D1n&27#2naBE$j$3Qi*3~K)YW8)ou{Sw#>%f0ra%}3yeSr@AG=vnjxjq27 zOMRU}T>Nke{f=enn(F!ZJ-LSMf@%fyvCh|7*oohttncdd5meBlp$XdPE%80x+~708 zh4y?+^5^cxW^oI~qo1#+YonjXTeYfYbxE_g#n;fey1lWh1;gLu?vLA`rQO?D*V$gz z$Qcs?fjz;es0vw(60T?uD1uuZi{&=ngw4ExtsQk8eqY1r-xUqCH+nn0Yyp244k@g( z3X+s?lptBq(9xOy@J(yu4%rLeym90HVCwB`;7x5l`EFj!#}4(|t#{}zyNU(&6Ta_3 z5Rs{tsi!M4L7gh|4~3M=w5l@N=Mzk(1yQ(1-kp|lOd(XfQuSH5p_I#`0P^bJA#R?R z`51k+M)6LD<7f`{_W;uP#@w1(x@7S6P{mNx`0DzcY)^d~OQj0ljYeo}d<7*m@nqFu zWt*T-{3?2D{lCcZuh;$~3qtX&`b&rKR+|uRRX|aguU6V^vi&IeW&39>4Xph8fRATC ohxDJza(J>Ixq@o1a_D^+*g7^fInzP!_~CIWKe0hi^0K!7e`(4d^Z)<= literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc4b328c6e8f71c2b2fd152f8077055f8.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc4b328c6e8f71c2b2fd152f8077055f8.o new file mode 100644 index 0000000000000000000000000000000000000000..99fc544baee9793eb43f11b3dbe03f71835622f6 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^KsL<8v#l3GG9V!kug$DzNevR{(WfPu zmLVrXWF2ELm@zxC&%{|W@nmTkGR{tB9L?rz7*8l%i^}nkAzp7iezg)IrnXYL}6^*^hVl@S2EVDOu@8P!| zmI_KY>!}&A)%`r%ZTsK!LfdC&Fm`CjmfiO*UT1CDbN?0Bb9ns*_h>p%(bDzW&L4z+ zH;!HXaJ?S-pvo4yW@Cr`uZ{H`>0pOId$=cq6@D+<)Bj-!d;3>9J2WG;r{;`pa^H|` zBK38+C!H1kAKlYGjCQ(ad%F%}kDPpFu=3pg!R?ox8C-GdnZa-Bo~xjJ7Oz8dPJBDu zcl=oRG3R%}$Mzo&kIX+2?wfxyyl>U1@Wp+n!Xx`Gg`b}PUihKbbKzqr-wUVAd?Ylo zegELo&pb1jI{)F&oR;r|&(Av+-h1h%;isK%g`1CG4EL?L1iHdd=)i%&ZO8Wy?wx-u zTzTqvxVi0AxNrYk;gN?fhR<)m2%5#A3g>~r^Yfn^+`9_>?E6l5PAkSg`Bu2%+{N&= zOElh}hC=f(-l_xW=XiK->$k%rz*n9-8J=_E9C)XG{vG^Pd?Eoje{sk5}`h zx59HzoCsGqFNM#~KL?tnp*gKj4)#G873Yqj@8jXn@sr_7=c#aL{>AXqtBB_%q0sUD zgL6+jIT!-Z=jVSrym$Uv;0t2hW9*wd^ZDN&ITf$c2{!$-A?^ zpPblJll)qCknHHOu|fxIr0G31_3)`ln&;{n+oV3iZPr;~)c(K;`%i*@2_M+3vBEdt z1DZy-Nqs}Q<1uP?gt5Zc0L$TRL|;FIpL|eN68dm`G5pSR_1ezYj~e|qoz*sHvr#z6~1}T_wH=p>$!pN9T~&-j#zw;d{X(}fW`lmzuw3PukHL{C{i-{?coRX zL+Agw_^R8^(k}d}`0B{ZW&OWi!}?P(KhDfzSBHP7Uws39J(SI^o++7hXsAIubms5$ zYs33kKg}bi4LrIW_{ejtKk_fE|Kh9q)ptu+|JnQ4)qh>Y`p>1YtLKy0)fCVcVXh7D zN$4N`XVyRbfOhD@|In{qc$Xa-{@aAm@VD5Z3k_IPC$RpJA58`Od4 zLTA?Rew5}Kcs_J(=WC|9B-VE!&js*LvJ8V)8uLAp2{K-Uj84czvL#PqSARlt74jM7 zV#(~92lAZ%yR!Zf$X6ld8UfEE;CTc*lZ+(a1@Jiz{d*yk^Y69Ldyr$~)r8OpGV*d#h-p1><#I63)j1eHS!4aQZhTzz!w|uKi;q4vtENO;(f;GA(fTs>$8MHB*At@giGGIW>E~ml#2@2Y+Vm?QS83eGT$!G$Df2v` z)Wt<1=NF1YWE-*x*@^ml7kWJl{kaf71IeEL?dboTxUR;l68Am0?#J~wUjGaCr*Qo@ z8Xxw0_ql|S4R%%jXB*G1z6<@kX6hjq>~jHrOLqD-Y(O?Y51XG^#QKL|KeB`J>G#*L z{%{HFXZpaSWK(7T%in^n;LGp9k1xSy{|5WT{DB?1d=Pdu*s*E4&ZH(E+Vab!l=45u-t^J`s4Q(I$@Eof`+^et$6 zd_m{-=C&>Ev3f(7tUtaWdehL}>i2fC1)aWN=K>*GT|-xAi?^er&fC(m)!W(R3)D4u zJ38wmw$st<1!8nGc>})2yGM|KufrF391_7w0bk2q;cbw^dshh3=&SGA zTDK+OZS@K3SOMMzCJ3fk11${!ufK^;I)9+Ov%Rz1kLHAHZ|msvwsq#+1@7D6^|$&0 z4R?i^VHz%r)s`G6mria#Bj7Hup%TaJvmAQYOkrhPF3|N1EQ5-1EmrM1L>?hCksDv` zr6KpykUK3cjN|(E(vZ(t_HJ6X$U0WQ&8H!E{ys)6(rd86TF|~_ONXyBkJZ(+Jzf{Z ztSnnuCopw+*s3*qJ9zYt;;AL_m%Az_yHRAC1EdYRWouc=;fYo-`*RK-KMKA9n{^xC zv71p(aX3y2E#)P^GhDHLdU6CoJy@B6TaTm%8815lcOOZM!X=n}UDB(e75#Ssx8eN4 zS|TX95=i${!1pe?rls!g>6y(Ok=;GB9sHItg3q&dBI}B0`_};wZSAM&5QM6D9mb35 ztJYPNRus8Qu>Z$7iYRnQLZlDlwS_?b&LDiTq^r^ErqhMy7H?;Bdz-tny}hMlo*OgU z=k4&h^A{~xJkPz3E(;2mvf{4hmPU6+cSonM)!o|F(dlmVZE0@vxz|=ztyojHuB>KF zMb)ahvJGp?R+W~OviVi+Meh0DKtq$eqZQ`?k9ge}ns}x26@30-;dMKE5^(!M2P03& zqx*Dmk6UKXh~3}JagMDimD#u0VHdQ0#4h^2k49(kK%5L+PG@#mx09tW7e(tf8r@LW zncGySvnnH=`lPxI-P)!`n7#tXOm#NRJ|~sil};PjN8v?j9TNAC~a#INEB(Yby{aG!9)qme*_Y`Vp?B zgg_y`7(Lq<>)okE*sp*j0}Z9ItZtz%K1p(%?AVUP2tR;J0Y$x&Eucd>RY0E_V5u~? zzxuL`{UV$7?z2bOWdMljMkm|7H_^_=-I$%qj=9;7x`hnvDd5pxZx9nnf@u7*Zk=r= zUi2QD;O9P6P-fzF*2xY9>~^$Zk%0qgY-l0N4chKI>R!Pj`hg_&#z8hKm|*+ulsC8Y z`cSsLFsR!I&luq$UvOe18?}VV&^E@K6-?5j^e56-dSGJ0MCyO=U=lk^^o-Ld5k2s^ zLHk5H>lFPHJ)dTR9{ocfHWJ5Q{Q>7c7xEeC!B3jB7x+PcM9=k?(WwzXsOu!3zdCFq zev^WeU`1}<1cev$VMcgYz&VNdO+6DyVuNX{@GsCm>RjG8d4J9{)cp165B)#2mqo}P zb_eV^RK@P!NwicT-^DccmG1EoZVtldI-wtZ{JPrOS}yOK*^qa#PQDazKtJd|H#m7R z`P+D|e^F?f7rvxot*3eehLF@;v_!4sQ2znh_DX;ARmDLWdv*y z^2o>{fiLUU*kxis6aKM0pr=ulX&-ZbTmj+hwqHXZ=wA=mn3M3N1hQ{%CLcaB0RD&4 z?5X6>SSbCQp&#Hgg1Iwk-GqPf_PIfrW4KnEO!5t8v+@@X14{+tpC|eUXI_a!v>5#y zC+RODnt`m>22_*Y@z!3Vj$G%;RbVAdQi54As)hWR-!!p@*Q&8NKSsLzeC zi?UvSNt+p*H;2ZXNBqG)ySp9p_44uxLLGb%FmX@R`7S1b%iJ?>`fD zflto3A_t+flttdJewsxTk^kI;x*ujf4{)jFNA+R_#ny0{h$0RXtQx& zQvMd4Gl#^9;b+Z?)?erIh1-MAKeQ)(+0j4e=N+3~p7lJVbu|*%!|k65`$h}si&UNL z4gbJ=&gJu!=JPMoSn&&*ASeF?zWjxs$>yRyx<4~L{56;C$^8}Mk$-I_5;-3-f-`4Q z{Nw)kbLxM&ssG-c(_{WLbO7sHf{px*FAAp?!oRTINBxWZXLL9*xcsDBMP zwZg!(gs8t_J(wAsHjU^LWBxWRlj_{x_Mp8B?Ulc+BK=Ntnf%QGyj$4w=yZ;Epf1{z zKiYwp{s#YserHDYtL#k-kiFj_ebYUP)HjW1SN=Iv0{wIfyqwRY^)Jz~J@h}5)UEu3 z^v^$$B0R<`e_@HKzuui0F@4v!S@ccw+n{egUo*JAGo=3D{~1wz*JoqBa9Q;2a>evr zMf!5Z<~N@oXfO4R{xgEm76gb4^osSD<_9;R)K`{SUg|Hp9t;(+@~qh=`Nq;$?@r_c zXV8S!ube-F{5FHXk^Byezu9^Gw3GZNX+5)x{DIbodRi| zpB|Gh$BB3;J}bSC^P8p@QvC7TU@}_2Y5wRuUh0I$mmB^<@%IYo5IV$p>0ke`i~U(&|bCz&50f5ZHlPyDEY`mr6b-xB~(J^=rr`1=YuB(?v5 zFD1gxVnFV{WS<-R zC;Xe|iw>Dzl03Yq{3Bp<@b&q>tNbb1_70XaH6GS8C$CK@rg*cjQAwfV|dGY z#@9EyOU9Ei{kw``z2^Sm-%R@x_{UzFoK%#5V1AxKmFx@qBj3*?cNFoE>^~)DfB3(7 zelFzn+@3TipwGhKG@X`PM*2JbUupfCmdGwgcz!c99sZS^8TGHBh1d_|WI}ut&=1po z!@>P0Il4c|%8AmSqy2^>ncIu}hwo3^oIZ=_fzJ&lJEH!C{5Cz{&T^yo2+@;2?Q-z- z5#yy1J^U%t=x@2fEVr?J;V!qE<{PKy{%CW+I>H|@KDCefYI-1BpGx(~m_QgG5BAL@ck$DFfV_pl%*<#!v+_gy_hP*3^F!+w0y+^0|2F&kL7Gp-`GEDuG$kH)R8Vwha}H=+P^sxV)`ZjNKUZmHiL+_;8|E*FZ`aLV{v7^?`1V=nFQfAnNAfFPq@N-m!F+Q$$RGLn3jEWY zuPg-KxIWW*WzAO%>o4L(7S?MlCnDZMf2Q>u>$7S7Mm-Dbb(9|Mu^z?tTSJ<8y~cbc ze004YGhgBJ+mf%)W}J%fN4}p>_J=)j{+Kx#*oX`B{nV$JuaG~)%~xiOk*~OC+&Ev! z;`xd@e!c>EvfLEkqxnkEkZ+96R}fF#ScxlQ<92jqg z8~lEPd?ZUO_!OT<5I!2uz<)a88SRH6nIgZt^>}95Zi#-V&^~B{VnG2P3*U) z7~@-EaDvY80hZVw()`GrFcB6K`|oV*zq2OL=W~Yr_cYos8seRkHYBnAx6?Vs{u>7> zRBk8URo%w^I~}Vf7!>z=mhZi__=fz*@cn=^ zwvrB;croO&v?R#=bbcp`P(0*sSby+&F+cAW=NX3c@FO6)iSzKII1dM9PvZMHR%uGJ z?61>w?DyH%K*0SO`_CKlhtI>|58`}1i#|W)pGWHSfq5d;!S71Tew**tqVhQz$;ZzZ zNl3B2+Q~t9`{8uzpFVDp{b{lD|Gb-%*L;4E#=Z!C#C{eX;QXh&`=eNXg3k|3`N?Nc zhgA20@Bd3(KW)`MbB|2YODqTo8}RTcNFQtoi;8J?I{D$ zw0XQgHQs9pcGQbgixcTpl+=ea&nc;e$lt_3_46R&-g|IIPoD;vb4kILg`Y_+B(>(F!*#vx=+DxznD zPLn(qf8bQymSmGXqihM08Mm83?|PEZdsJ>3DGoj=w<|tADz{t69j}k5TwsJ`Rgz&y z6oX{;Pen5OXpbUGm^hO(QO+RUjW`=U2hE(@bSvj^b*>A}qNAb{qwR&Ri^|(l;>}p0 zr((3xn02&jKn;2vg(q%E2di~`<%5`eZtg+7v^ne5JT_oeG4dl|MfG}$p5sXTJzpub zQQk#ks};q%V#Ij5o(5IF@mHW&`#ri9nutHB z(=A{6fCn2U`gbh#C7K!UvDB+AbsCTQ=Su|o-ylx33Bpm7 zug%TyewweSV~ut*J&o0e6(6t9&yE`_cVpF!GqG33Ans+tt;o-B=vVRDt4TsrYs~hh zd8sEw=cQXPU~L~y+g6&-d)=rrJx&a=@*j_0=nx5lAZ1(IX9cu{8Sn2T=+Z9j_Cwq0PUQnlAXP{19 zk*;af9956)Gi*AIQk1&ZUgUZ_<>p4CTx2ZW#LHEP(_6^}++f$BV;!LkkXRv&kd8-X zv2k)ellNNkJnF}2eKCSzWM&=N4H5(XUaW^c;A5sW`mEUEZ13}~U(FVK7v<+L3KHgTR|q}}itMvGN*%e7~@0N^7V2y_M~9`Zp()umC& zn|c0E+-Li2aGruOeq@^PkJ#T#dEr1{V0D? z&*@UMXwHjxPFJP^N3YT6bV-P!j}qMG4b60ZOlN8Dk@be?2g(8vTNy;1HUaV4|5&#B zl0~)F_V{B+%NdvGcRBr$CW=Egp!JT%QtT)Vkv*(^QoCw7;hU&pC{|kX3;CJ2SW+eJ z_htKIhPHHT)jRnQwf9)qXp|;*14o5Ck9fDXuOKA8o+&P|_piwIxe}ywY;8~bxhLba zKWb^OdbYNoAr#{GC<1vsDcgS_>oh)n*TmYM{2pD5;(tI+5x1c|-KzG=jt9{(!90XC zlrxqorp8z8l>Hw^;~TYCbkv^ctnF!@lL{7fsQhTE55!L8B{$zmnSrUcF znm6S9- z6^r_qPqz11#y=|CEB~eT2!0wB(z*I<$&s{+7tM8+wmwa=ci@3=zLZ4);J#Vs^*2YovM4JgBaU!kepA7 zPF>afQdiY(Ytfo@9YtkDqLi#GE@z3UshHg2fNZ)ja#yS;FA zySJ!hWy!kYriRvKotrk)v^4nFtZk{-v839vP_gfDGbx_aY=%BK4D>+&{js9fP) zU%0)cvZ{8?j`C$K%O0z3^4F|sSzb{4SjCQt4FxqVn_8FU*Dn`si(8hq)Nd%x^KIh& zuGmnptgF6tX&!j+*B3loyRCTDw#E%L9UC{SdSXTEs{F?C+OEdsEjutq8~7++SH7;R zpq;1mQAF){PxmHX7wN0CZy|xusZb9F9jTq&KQp_>NyS%t&3RNa?4UC-&?uoIsD;je^Kzq}amxrSgg+kz+hQUgBvt4J%HuZ*Js zUKA@W(Y4$Jo$B6teF<&t?0t14VM5CX$C+S44yEiEidfLZR}TcRQCw>61m}^515|vV zO1$SiujO5%QJfO*In#okN5oMrY=&_aQqafvF~j9fNC9yb^v0dHK(^zOVc5ATxhPB$ zT=~Dy7h~UqTnZ+75byM*a-#Q0pz`}L6dI3y<#phHK>K*~w7-c5f8;vw9{^7LoaT05 zOwB9pI{d$Y#Ea;wr3qe_a?qIZ#(h)D>y{=^^Q#&3)X!xJS8WRpYk0-O_caa7I>b7PXS!DCD0cod@A6^E#oc&+?XmJ zm2estr-{*WOy1GBMtG@|lloV@5$OsExAwC^!lzsMQTn2OjHg6BhW_a}C#pPefE($L zT?c-}05{5;EA?*WbGC$AjWIIs zI`G31PW%{Gv*#uJAq#xKKyS3~>k@9|pPr&ah4`5wZu$FS30LDPUu=?aHLijmHqaZ# zeO|(?dQ3c}wFO=Zj0iN6i-rr+ySKB*10^TpsaoxegoP(Le41X0aSh$?N9S zjD5iXH_Fu{;YtsRzD2^V^1dYDR(VTh$Wi0w$#GX0;Kp%-60XKo^t&Y7I_~!*+&V7h zEmSCXp(&+wM{$jcf_r$4u_k#{a1}S6x4>0=7_h)q92mC1)zYiZKUM!~y`K&Ks3>@k zwBK?IT&?$h3tX-DOr+f%b@k0{-at3Q9-+~@!rRr>(DWGp$;mQ2kQu-3$baLBey)+- z?9PKas%vg*#?L&~cXZrZv$n2Q!BDfWgN?n(ky{7;^O9pzKkf^3;HM#^sLk~O$X)L1 z6yoBCOXznj%U9LT$M4BCY!_6^ppP}a&Z18I{$zbur;nh59t};HJl={G>sFRFds}=Boh#cLyIL^(P451<4O-g0jdh*vb&Z@c zArROTe2S`&#VFyj_JAU|)v;J^<4xGi8`#=W*WvdyjQ(BGKzpON)5{j{cj1u2N-H2q zDMtyC1q~gY`48W;HtvwU@XZ@H?hmHk&IaDp=9BN{#eD2gzukI={<5oBU_ase9t07Y zYMFYvA`{f9GXGFWxlF4nqkTTXWLgk~d*t0|8OIbt#Vb{xg&Rt_JPIJM{vG1xd6|#V zXKNJiWH^rIVE+Ihjc?4YsijK>Uk_CbMUAhn&&l@Gx3N^K;N571#>Q7rLK9C`9agsq z3dOIYx7Pnkj(@%OA6XEJZ`EHqgtyv+a4Q0e!hE&TZj~U3K-J> literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc5932f15ac9d608adbc379cb72c7640c.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc5932f15ac9d608adbc379cb72c7640c.o new file mode 100644 index 0000000000000000000000000000000000000000..27110dfeaa260d9b1a369b26d70bffa638c138f8 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Ks;vR+13MuWI#e7UYl9bk{Tq?qfbjR zEkjO($U4SgFk@z8pN%tQ;>pl5WSpJMIGWAb$QvKxIGDkjY}O}}b3$Z>kpMHe<8>f0 z?*0CHbl-c`Ai$p3Fcfs(TmS!8)&Hvc|Eg}?{_?Vw%XLj-s%Y$07ON>JW0^g%dk??$ zuvAdGS$B26t@h{HZrlH%7ur5Mjj_Xnw(Q<_@j7SAp8YSlp2zE7agV0s6)jz_@A^UL zw`16~4>#zc4=Qb;>o#`y|JhjY(ROwiv`4x#SmF1wJ$)aRu(yAuv%}L;yQ|OICiV{6 z##3KMy3<+V|J6NxLujXKwzq3B_VB4^1}e@U7}#<7nSqt3pBeb3?zslqXYo2b>*Tk? zy(f-`A9H>ueEh(P@bKJ|;oiBY!uwaB4qw`TIy}7pa`@@F?}Z;}IUhcL>b-Ev^u?j! z9R~)Ue&(5h)VYg7vzosXzA)!_c;DrphM#u66>d6lDcrmAGUy6Jp@Rnpwx2jKuy5}1 zaK-5p;ilHp;obvpg@+%x6uz+I5@;5N%AE%XF3f#$VBc!=v;RBcSuGg<)LY^5^OwTg zFVlE`916|Fc&iVhpA+HPE#D3g17C6eRCw0O^WdHO`J>SA{(}Qg&wXYfbm~O-0$xp* z-wMw@c`{t?yd1tT_dIBphGw-qInWDTl%GG2zE6ZhCr*VcoTtN~xtGFEuO^ckfiLI*viIVZJnIXga_I8HBcOraQWC!%4!!qQIA!a}@T~I} z!Pmv`nCXv(h6%qMyzU1N=g*_vpP=3TfpWn1C7uiqKMa_bef1}aU#%X#_;Mh5Pxkkd z6T7REU&{`X9X&Qy=zxthy}P;&J~ct}Tsv!<&`Y>2IxCFYA2?zE3GgrB16wp!_y&AH z(+D@AcTjgcM(vI=R`?oVIlPVN>xb}@4=PJSA8sgy-+8WG-}U-2qyNV9#}91UY~_>W zf6Djz;d`&bH}Cu2-R*nbH}SorqxjxYi|>(7D*x-Z_@DCEoB81NT|W#(N+!NN^pJk| z!ao&XbK6=uQzEsSQvoqMWp&#nk-hf{ZX0vN&OC}s1tk({o{agL| z(0w^J(nbg(P+@1++z&YeRbz z`iB0A^$k6w9lrQK^=lX3Wrv6UCLuKREq3@~J=WB5tZ(>769G#M4L!sT|La4b{p+OA z*$sOhrMU*44`1K)nrSYH^G_^E0=N&gXdxJJPe*mMw0I$_#A`&y^zWIcUtH@$T9qCLTDIryazdq z@{h+w}#%o;WJ zfhW>cVw@RPu3DNjOKTY%W@i9{3k8wS1>eY`cHSS}sOwYBHIi67J z!lID#3&kO_4cUb3ME$)By`F>qT!^26WKZ7?^#4s<*Wguw`)*th;CceD|Bd_8xc(cB z4|~1)TtdhOyDI;)jb+!~h5lXB^^gnpxd^`{JN+6qAe&!+&Cf1ieS@$c*+KdA`)gTW zxP?_gZ%X4%A(Nqq~fb& zzx(*IMi5B!+F=(;hdy}nb{AD?4qvcio)E3JzO$p*+umO5ZEoJ??P&A`YU{o2 z9kmkM;cIPg3-~(%O$!RPwfI`R%}v`{^B1+{*Z%urFhFg7VL|P}g2lD@d3kvY3TpAM zU}4_k{6$M@Yx5wdkn0{0qrK4^@HN~!f&_f+zQE&<2v!RCn(qm3gB;#_LXZYuUFWvi ztpRU~Pguta@Xj+qFwN>|X$W}zjeOGi18p5`9bJAjCtO==dxy8RBkvw?-+HgV#TTf* zC(I1fa9OOjx*A4Y~XGF=~-siw)Miwyj&+eI0qMwzl>0+9+n# zvQ@PLQ=5maT9db(NAD<}S|WeBr*g8JMW$Im+OS);mZcn?Xa%!B7>YncInamN{(>>F{Zy6)_9BU`Cu6VY89RShReufS~sEXGSyr{l< zeR*klk-G%@f1IO;LWd+ox(KhW1oC$T;fp1m4PG~$E;KcJJDS>B-5qUh&FypCnAtvW zyU(4!VBW$x?)7w;SGbfFcQ!RQxZAthJA5tfmd^GLcY|+hQ>)LtuCj9F+S>KYs@IlR zuC86Saow`jrOQg$+)DQX_grtFzR}&@g7bjIUN?p&Ug>-VpMO|*-NBv&+&D;I;z1O-W^bbnk*A zR^erLfd&S{fBv#h6<%h`TF2}u4o2SHZNvQ{3EzREtront0f9o}(Dh?^y)LgG;aW-v z6!MFavyHKyU0Q_w3P>{0U>eKn68hqkB*)2)?@WyF1Gp4W)H&HaI;2wt^r-=sN`w2W zuGrWwvRTi5dxTv9fS7J@vOW6}?R?x#*{STfoBgOu$iSWg9{u$MF_9#Q#$VQ@vn|An z-fa{7Jb((ybiB?v+2MfQjutG^e=v;=&S$wn+XE|tYgj}-n8e;V#AXB&Y`>ZG<_=yT z%(fQ>bsOOsBRu2_j*n!cmM{_8#&|P=NqUt2WEx8kj87O({SO>UV&{mSarz{p2R=7w zA5UkUqJN_2(@fB#f9S(T;`pmR;QZ%8J_9}YNt5;hKj@F>x&AUbG~x$!o#gXZg>A%d zQg8yS$nBe;@PaXCp~$AdMCN8Tv<^%ljtp&zXjrzYhJO|EKn`2-(B# zfISB**@L@?mI~y%l*Yc&H8#S{LHJxJ^rMemUsF@V<$W_7@=nyrmm&`62mR*;Cr%`P z8_V@C>g>$Ry3$Ws*4L$ZVNdiAdpNn1lRdysVStUFz{eN$o?QtM_5lFo1Mt6$fGt8E z8Cf9k%epjng&5F;f2;`TX;fv}$DAKmK=``t*U$(0*8?`@Bs?jB>>HfUhmZ7w|G_kS zD)}=OO8*w<2l$L&?sQr=;a|LcZV=`es?jErd;{66?1dx1Qo;Bai2mW}S0fQEMnB6* z`in^UmUUs7pouO0*~Ab0t1$5J!`xn)7_TreV-}Z(+MiCt{G1bEXVISKQ{Gh6XGhp2 zS+BdSO%KkQMdQsO{$QUyU5+_?zCAcKmHnlYO%KkVLHOAmKc79>#b%Rg6CcE)GgJh6 zJLs{t5&bbI$JvYf&VJ-nZSDler6i)KNEF< zPuBKyxq!zaO5-;QybE=KPtb)tTO>YP^1nsmHQH4@p&?VkzzMhobRRGsV% z|G<3C<@1&1^Doj^@e7(DC;tV$?1i7n=Au5bKQlf2HJ9wk{T1Vpe{CTWIUh2D)2CDX zVJi)|DIh_WBxRF5bIlljr@%-3a978zp&m%{fqo(Y66WnmHW>jvgg#Oe+@dd z!oZY-sJ~)8m>!%mh3FGw{x&6(>fGP`e@iz26~y(>;pRH;rdk{yA6z{d5StoX;cmFVV6+^go@{t^9-Z z&p(kOJjN?~VTq}~o?RI+eb=>G^iA{Ipl?23Gq}Dpr2gRl8Bu-LWn;Z?S@i94#q?cC z`f|nQH=iG9FZGT7GlI|-1c>zai1nA|2RES9SC&~`>Myz;3>LAnteGbHM$=c%F60Ae z(S+8ooIip5HiN&B{0@u1*?Ii5ll-S>J+q7af!2pQS}#R=72l=@9h}|~^EU_A2l?AM zO&`gh@K(MaWB#px{X`M-&l(TOzGsL)uHS{hv17^pe0;=V(Ia8~#l3_X_9`KVMdL8nqwwPP&l= z_=8Gw@>mQC6`3~7HCBn{OK<>X}pPTz9 z{F~>C4w+w)JiMsUex5~@>6!DPkKPhH^_`i96 z&gb;po-`+*&%)poot9fh`aAO%w0=!VWLF|QzZskg|4Pn``q$um><4l(AwCM|hiJdy z;Qo^w-JfLTMCs4de#4Q>?M42>_a|;npGEY*=LVA>tFjb!f`iKkjd8jfqJ&>(WrusxoAdHXkaI) z0eF+Y18lIH^L-y8`s{gQtqCs_2GNlTT#PQ(N73vHefj+!+`jAO6(?ezfBceZ)KA^#sQGA;24$FS{K$ zDnPMey*P#OcME&?>)s#3eo686b$-65^mE(tV871S5Arv@KA$n!)0&@|@)6kYK!71Z zk@mBmXIue)oyQ#%55eCUxgRs-JCx7(Nnpx9D1T3WmI(_9eceeullQ6SGk(s!(l5_v zqW%}lXZZQVxitGt^B2Uo8|E*63jafV`>gYqk@<=v`4unHPmzyczPTLakNkWE{%Ouv z<^ykBpJ~0a<|~Hv7x5ws>ot}W5$~Zt(|V5e*|dJ6o`v-~N{{wfk7E0+LCw5gW4;nT zx?Yc(ukiV8$yaDIPDS`5-%lv}!=5;Q%$x{p#D)2O`cuqT$RFb7E7L~FSKQNXp08x_ ze8n9j5o~< zem_Azk|h>=iq9kRqq{dBA^-UgnUBmd#jlb3Z^|xEpX zidTm7b@l2U9bfiX~7v)eqd4SLNiQ^&VV4BMm zpRj(K_Sah}J|*(~Ige+^kJ5MBdH#g`KpJ}k`Rne4M2hE0OUy;1N&h=Gm$=z z;p_7`eBQWgd~ARIFyEi^^HZL0zd0ZD6AkzA5&fd_H7G_n&ogtvefBjFaDT@B^QQda^KkfsIA71A&rkX1kve@~o=A1@yUMcP=KHm%d`?F4@$*Fz zQmn6bauD8rD4qJJk6UDaTI~Ek@7ClspC6>LFM=PjpG5~a|0(PGD3+h#^8-_U@)^`2 z)%|i`1NSv>Ujz3wa9;zTx(4X*9HF)!-*RBxzqtPU4}TQ->dU=Vzuo+I|M2sQF@M$Y z(S^T>94cfpd!O!c)wp|S=N#+pWvtI$^W5 zc%Y8eHT=yocpu|T`5m^mjP2EyB9Lo!4D_+0=QtBJRtcy$WiS#N;_1Lwt%@YMLphgP?r!lUwVnB*Uj8R2BKSDh5F|NJ_MyM}( ziFTZ~F;?EtjK%PKQ>=JOcl0IBf~#x`K2O5;R8x>3_GpP$p-fq4D%T%RaB@$n+p0v?Kq*FMVsmYD994dT&9(yn6JBUI6vPYsQHO>Dj4!J?zR_Z;KZUwtlISt0PIVYf#2?h@ zmM?w4gAEh?JC^zq&5ZY0>Q$CHjYs|SB?A3#6erpQ;^W`6P9AZ)Z9XY|QEI9EcKe9e z=2mz=!&lVNM!S`sM(e|hkJsmC$BmV{(dx#T*dt>Q_cGyDk*jHl0Q(N?m6!ay_1MYok#vGMaAUti&(a5pbWJ|a?_j?oW??eAEb2Z-Te!KJk&ZEJnF}ju8czxbEht}Tz&~?*H zO&obzXyqp!y=6aS%7;d@kM247988F=Ahi6TSn2P!x=BBARz9md@#xj&j#gaI-_`p| z&cFFA@AG!H_j%W^X+9ZuY$DdjqTb#SJ2nqIC>FE#GGo+Jv-WLWUtNm5HsfdC+EsWT z9fo7Kt;(n)@Xty-;xjBdlC_+p3m-S(`^ER;#P&bd|HzYZE`gzcUE}?CE8ggPXU6^| zUZMG)xvkN)au?1QXg-N@#*PDdJ(kSJv zJpU)|vwb$Z+0!rPy!71{$G5QW&Hl}AH`@Mc#++$+KbiXDLusky%_;LSx z^MiR#b>p{LybpdfWFlJ1;Ujwx&*|cBZK4;?>B=y{sPYu1l#Piey%>?=IbE@Sl)tFw zbSYXi=S4iHD^r1^*XVP)Bt+3i3GVQQW;#Env$XffdVTZ*WdVq-45Ch(fOzeHEZcp_ zqFQTv{4u2Ej7#*pg8oPo#UUHedPid^c9e$59@ajoU6q{hjZ`rdD=qnj{7hUdsgm~l zvi)>e`LCo&1N|dn{};NRzvPqe7lXyj$Cs6B1v~6qnfhS7iHK2~s+?wx|8vlX2P~ zv$R(|TiZ_)3h{dsfxMoQ?LUxp8lS#vVr@@;k1j^=-!G?#+t8kFReNQ}L+F@b9zq(* zS<4htViTXDc+1_IUTAU0xT3?9&VhbG};m^y@3?0X_pC68izpo?v zNskxp&i=8uE__g2vv48#O+={wCje=D6ccLvCIO_5jYZN%LF>?dZ=CVJBHQOmN*bSv zMSaXC+j}hIACv8s|5AH$X>0q1n7mKN8UO2>L5$+&?~GZ;SAJB1_C09tv5em@+bjQ5 z`?ca&ho~DGgJ^fe|qK+SCHtGM0Y%kf5Uab0Gi}wFDj{K_q715u zu4;a%t7^BcXzlv;qGd&*l&mT)Xzr|EP~5y}!P=VbMa}EH8yY${ZLH3(Z`n|@qi{`| zx2R-Q$@=2P`j+J#n>SWB*ZbG5YcAipyuP3ju(dUWFKu4BX4A%s#<~sb^EPj+Sn1tR zxTCqEvS#hhvgOUoAFFBfSFdeeQBd<(`Ofl<1=Y=)TbAe7tq^UCo0m4%Z7k07ZRY*1 z+*q)@v#w=n9(eHA6)dXRUc7pH!^Z0NO&eD~v9e`#enVMJXTyr-ofxAPe3Y#(Ti;n< zu(XTDYuHfHR$RWbv~I!ryymiP7143awypLSEp1y?wr!oHDcjV#zGdTr_1kZWt{{Ij zy5*Z&f{mLOR6-vuWhC>~&7`~hw$g%aYrMs)^Eb6@TeiKZx@*&h!a&?H%XeG>Piapd>f1XbuB*FXMIIo>zXH4kquU?Z}JwcT~*w=Ib8fF4d!r#!cz6<&* zUz5N7iG~#w&6_q9z~0sQn_Ab_G;S95+Fo3-6@BM5Y^>^zHV_r{o+Ln^Xh8_U47G5Z@uqEKe4io z|F#PKstSJX`20tj{^S0y#^BCKeb5}n; zIp>euyT0@#7$<*TJ~xqCjI}4KnYMeXHYGJ14|2ipM=As2nrpb9gL~s1^C?qPehg~Rof zy0_3x0sNf8;aaNhOP#Lg^l#V+&;RgO!l_?g4ES6_Fq&z>6Md-xAN^INmCjej&;T!r zm6qt5Z-GvAZ@s>Rws!WuI+8G<<%8o)Fd>Ihb__)~4tCJC*0zLI}MZ#6v0tr`})Kb}hj_jvOnoQ~W z?(=QFE;2Qu>SA(LbHK&{n0JCr5O7wXc zhssHDVQ=F~@T(GTmvANDIP|A{hfw1seu4#_C*hL-S8WOOMG2n_xN*z4+WHu3v>carG_Da|D&?g96>mhkQo^nMY?Sb+mVT7Js2}4gQMaLgdd`U|&l})I`r|i% zUp2su^5#mtTlt(R;Z}M3rGc#Sz9He(etst5R(XGE=*P(aM+Uf2-b^`vXk25A%)0^n zh=dbA#?|b334g=_?>EpJ?fbffTluG_=ujbkCW%}AzEHx|xXKqBC0vcG;71Jf#&MsQ zaO=3glyK{~sdC=ZxW@V7lJF7pMZ&2c#S00r#Q>K_JY23L26*(3dw^Le2UGI8IW=Qn zFu;v+HA=YBgQ9PiaI3s8Nw`(sQWJqfptOL+?w z%3WwmDcw-YxC7 z!U9+8z25>?>pc@`cYAGJQ>!=7#jr*1ePcWGlMByHJcSgoBg;4QI)o0*_QZA1I$g6*+xOraYWAxb? z#XA{}qdC~$14!c=b8BkplEK$Q6+=)6!fY&*Tk+fozzGXIl>tk^u>UK(Lt=EvZ2QJ^HjH z(=y~lh^%7_1~XMlx@KdC|F4bp9qnL;L3^YpgB5;1+tdGH348n3Iy*cwwWs>5ZA#ye zZ4&i$q$iyf{vX}bKa6&|W_!C1V~?DAcCh07fx#V@pB-Fv`q{y6>7G&0K8M%gxhKC9 z?mKZj{J8VG;o}ERghv*f4EHTK72dz*bokQ#)8Uc*m&4C2crW~L>-q5UQ}2aSW<44j z*>Pa-nP;CJOkMCuXl~1Q!x!cs5AVDDv+y&{x5CXQE`|G6T?SoYD0J}P;Pw*-2KOyE z9}6~Cr^gUotMKG7Musove4Ytrw03=i}Lfw(f5gP=)|dTh4XYcwBSIo zBKW!(o-pgN&l3g3Va zXd2-r_YLWe$En>>#tL5pEQhxdef3* zFZ^TisN2rcF8;cBbmZmb{l8hq`cpAK&dz3|!#~nT-+*5aWwX(dBn7V$5sL#d7kx0{)zQpdQ~5Nw}kbdyPu8z^J3P2K8=lDNMfTYpe@2&8{V7H zKm3oZfA~S|@Wub3k6wJ29UlJcgwXJ}+2M-~SW_pm{*fO~0W2{z{2)90FAswDza@pv zZrt-2%{B0R_}Z@5Omj)B??Rr7;Gbj}2Cp>cdn6NNyaXAYkcnhVp3FvnN^=$R8RcTh z?3xGiT=<*i{UeaCT*@^9o=3p*2zVwLNxqBVa{~JJLMG?mYN7Wa$H=P*p%KXO9^^2} zPcn?Wniv{^JkUk|g^BF&CFo@2<)jePdg99EV4l&r7(ZEK{j_d;T=}CA!G4RFHE!$! zPo%4+L|oDS1zN zqLA|o#UZi{*@Wyw{k;plo`e2eh@XLEPyY_||1DhC;#GnB9$XLLdIGQih5OUE{ws|S zd%gR7LdXWYD*v5kPG&?2)`ve{RTE5n_qy<&n{;DL$Dv&LHYFi>sWud zg!MCh;4!kPvj3HD!&dO+_u$8uVY7dQ{bK&W4qrJ0yBh4+G(+fv{Pg3>qR_RZ;;Ur8 zd-^cz;{`Ym{FuM!?N;km@74Uh#$4=513E#`nWj;I56U{hR^sJA=R0uq;FW||I&;ym zW6BO+pv~9P!K#4CC9IB_r>EkoYs_|^P19|R*(YGhlf~v_cHDkhc!33e=n~34wmR)OIZRqno5)lJDNH=1D)P_+^K7KtEls9T}M+}V~{OuYwz?eYSu`qhm(BA6zcCv+?zF_Am;_**VfS<@OK897Z+@6^|g9inzyy(KhmCG_iu~A0Co9=1$FrakJRPo<>f6dsKdX4 zC3%nLKeDv0E)Q}Fx$XinI-0xzU*p{)NWjxoZSeYAeSwC% z!ptxYm&IyJ4wOqLH=q%47uZmVWA<4Ny=$hhvMm?rdIpw3#kdx$b{!&*keSGhFZa@r zduhm>mKMfw{d;N1=PY|SEn8$AE8ynSkUM`LqZa9P*kCPe-@3KK*O|xa>e`;Di(*zU zUtK3Kb$QsTHG4aF^p4`GCGwZMDkr;9WSR@44ZCG)S;pasRxtZ>4!>^p4ke6ggf(d(wuh2|D-XLEa-yR*H$rDMJuGu!9w z@VWCBFI+O;y@4(Z3zxCtuI83TcSm&DQ;E1j?4^A8KJJJ?fz+ZQ<)c|soD zr;B^sGJ8hs{$`GIY)z@mzSR!9pzWh}(f9o{I)ew|Wavsdv&*`jEPbUYTDQ^YhPuw& zraGNf8S&I7)otk3HZ8*RRXApYZ#M9nz@+`qThRrNRAG zS8VK;*{pZJJ;JU4Kuk9}*`9rgc0TT=>{NE#&3@c1WMEGNkN$dtm`D;t<1g>l*%snO z@39Gf?neb>7GCF^>~O$tM++7iIGDzU7O~u*?f$BiOcv1(Cb2gTvDv`{+i$17xr5h- zvh9UI-9~uE2oL#!lOoxuB}{>~G2ZN8k{+c$na0uslM*IT|AU8;*g2wSoIZ)@fzJ)v zC(&7_=%48MG!yjbANsJ7IR2^+IRCkj&p;1;(xkn>5Bei|uD^^_$;5B!*+>!_Ok;(Af&NkF^1jLYbEcu@uSb9A|LJ`!LiVsb zV9%jS_P{Qpr2_dbrLnJc-xuNLAbhS9`qA&(P*YRG<$W_7@=npomm&`62mR*;r%WM# zyN~N%)Y+Mrb)}!OtZzv3!k*|K_Hc41CwqXO!T_5znU62(y}J@3>;nME2jG7h0b7JT zGO}3Umv?LI3NfGw|5zE&)2Pa{k2ycCfbezOZ=etKuLo?*NqABM**7?g4<8u-|3hi^ zRPtvml>RNy5AYem+*!14!oPU?+#t*`T%%1P`3AFD+1HN%O9kU!Ao_=9U5!Mv82wx) z=`SMXTi%Ujf+n`~XA?j0ufo7X4{>{GV!Xn@?73VXYJWNn^K*WLoke?^PkA#?pBG`5 zWWD~fHY+%PE{!*z_=A1+bUWtr`S!q!RQ8unHY+%9HsR-S{37;LH=9SQO?&{0&TtXv z?V!inM)b#=9B)Tm;7fpa1MfEB-KY!va^R-}pC|CcqWyH#1^%nRX9DjL_&I63|4h^c zK3Ut-?E)T)D2?AF@GjH^K0z1qY?1hE$^RCK*MKMf{Q_UV2>$iZLnI&ffAX)O&BlF6 z`CD-AToNaSpFKNTf1S@4ZVx{H(4O>VNB^9kcWioD)(ec*)ktJ7w|^$=8!ezOQgyO7 z`~&kjm(N$4&%aD##b4J1Ir%T}WncffY%c0!`!mzSUvtTx++Q&s`PUXAk@F!VIBOQg zKkkpep#E2y`tRK}Bj!&-2eG~-*vQ}bqHuZ<{0r-S)W67oW+c#fGr0d8B74q=`qz+C zD-29ei25tmgIU4p(}_MY=5Nz!^%eHqi}o(GSN^t!^gG>U@;3+YZeh=3GdSLXx@b@S zXa`>U8~hvkofXxuvNth6_I`)-P4_5L-!z_G`R7mx^wTNuaz2mMzeLOS(Elt_xAG6t zKmSCE@EEV`>q|}j_3p}u>ASwoqHmht27UASn!)v*A@v9U&xq=~J{#+W%c5_WE2i&C z(w8eXzxn(?d#P{qpAm$%AV6fGSFFD@Kez#zQ5T541ki(|Re|tN1o6=-~8@n7=u=K4^VA zr|Dz)6W+?#W6Zymu%9Sm{#oN8+4l?)$o0E0c;9_we?C6qMOkt#uc>&@y9-OdieG+` zPmjr$<3zj^pOxOv`AyRcDgO9vFd41iG=Fp+FLlD>%ME{__lrqiw`v1sCbke`i}U(&|bCz&50f5ZG)K>VnJ`thBx-;)4PJ^=rr`1=YuB(?v5 zs|NZbN+6b+Rvy@5q1P&-mz= z{JAHI*4MIRC4jJ}+5brYHe-CYAK!_5L&;Z9`HsW+>uVoQ68R3Dzl03Yq{3Bp<@b&q>tNbb1_70XaH6GS8C$CK@rg*cjQAwfV|dGY z#@9EyOU9FN{kxK3z2^Sm-$MHn_{Tn)oK%#5V1AxOmFx@qBj3*?cNFoE>_0VTfB3(7 zelFtl+@3TipwGhKbe)!4M*2JRUupfCp2)65cz!c91OAnq8TGHBMc5DIWI}ut&=1po z!@>P0Il4c|%8AmSr~QT_ncIu}hwo3^oIZ=_fzJ&lJEH!C{5Cz{&T^yo2+@;2?RN0> z5#yy1J^U%t=x@2fEVr?J;cmB^<{PKy{%CW+I>H|@KDCefD_`HzT*%Md(f%NRdSHe=74#7o=JRl6YI-1BpGNg5m_QgG5BA08FB0+7e1N=#!OYBPJhSpc{P$wK>+?hF7XmsF2>&+w`yrZ7#`%Et$K>z8 zo8|-XCVvOsFdv|w+@RV2;qNoZzC7OnzZpTpe1N=!kNW>5jq)k@7YfD4m+&P-98lcem2=-rxy0O%XqSO8y`(ts4N&CY$xxbTkBK~5z zo+U4lKfG+z+-p9Q%yFg`x~s~7xe#~=KNcf#uljPpZ)H!xpz zJ8)EhV#9iI3ghn<_VCxgKY{&<;_d7Fd{61;mgT{Iov$C{Z+v||W3s0;KQrYcu-}0I zLxLjhXFboj0{*&=J18E4zX@_bX3BRcpYfBxlz&kEp8OmW783fpoqQ(mQ_W}moO`8T zp3g-6FP6{n^NDk5_8aCeh;P@;U;Z5ahxqnc=PzUP6-V+bUZkHQAHjTcImjRR`3n5g zoUbeb-nc%~dS%U54C^oAMHbd;EGHt~Lw~0A9P6`b{YE_t>vfbK?Xe!k_FF@mdA-Jb zC46+f9yed%^V^cI&}N*9@JGI%Q1*vCasHS&1=xrS^ZoRvn6HpO#LZV`j+3vrXWlqp z$>RBnJAS?bd9vIT-=q0T(2#G8%~udl-C5CmWf=PVZmDR8P3GZ|42>B7D9u-hO*G7e}p6JU-Kr8hu`rL-FK(KHsNIf|NsP zE>nEM`f1u9K@$oJ<#3vyAu*Ao+nL8b5K517_?KkPT=`2 z)<1DxbaW$&7tatc(*KS3-=_G5d`b`SPamS6KM+0|&%l2=;u-CSBAFt;yY+Zx+HaWS z7sfa3H=u7*JmYxt{=>B20NxbehI#(|3FFxnlfPKw89q;n*&qC-<2-5LpwtiShyBbH z`aFiO&*$)Y3Uz=z=ZvsBUcpgRmpGo7^8GjY z6G}ARO_{>ns8harei8KP(hd5A{nK@eK27_P@$=U|;(Wi9V9sBW-k|Z&-iuc`-ll73Ud-^YEh}x{34fV>k~7Wl!S!I96#& zv+S?abnN%p*FeDi8T-#0@`umE;Sb_`J&Qg+<)26D^nrOI)xqy-%YK{h*P`+{8Og`b z7fDF5zS_w_c>CdW>YqMtk^O10^Z&e?lh=HHkjB0Ue#Cwj9pL<@tox%_euB>rO!>)Y zP={3a$~_I-)4)9q+|$544Sebvpu=;7+5vpaf%W|I+8;jrapbEn_f`FF^WXmcFDfSd zRpUn&{yK7~kj?3Prq@;D?wgl$tgnx;etXUHBk%uHUF2&Vm9jSBFZ*(O`}$JYnV-D+ zhu=ruv9skhYiAzQ*VZJgt-)>tA^m4_D6Z$z7e_Tserb`{QH?z~2l%T5P=aa`>?v1y zgf8NNI#$>4H_PCCf-~j!*xoX>S6hZauGKTp$BLfgOw?F405;7d67ML|gF9_PBHB|1 zplS1Ye`>tf66~lKrxqvDt0>iD*UC0e5xjsJEfk!_xXOwFDH=0I74`fu@x;ft`WhIa zzUU>|ao#3ac|$W6!|zS8;wjzHmpTisvMu;L3ExvqL4w$;C4%K>iHDCW))l`!q)cM<12P&)8Z`OV*+3D2zkMo>WB7 z2Aw8(EdIc$xGl*hd&k%kA~S9`gWmNdq4%iVG*TRVRBl&%d{l0?kUL%4bi6g|h0_dxJ>WWFC$DlgZVKfnc zP^VkI^Z^exO!V(q>Ps~<-eaj(S?V+%_0N|G^j|AZv-NegPh|^oi1>9iQpkp1O43JnMjgXGV zWU+B_J(KrZ@;vItXniq)VPs|<*$ol{{$8wyKHy`fHTttyBR`EXKf9>CX zs~cz7y%;Xmknwc(N%2OV-;3eqRb0dh6(7dN9ol2YKS{H!?z(mLr+8u5$!XeEWHD+d zCnK&J*JPnTN?f9mW3kDWaAWTGCfwVH{&D7Nyw&|y=l|_TgHL00E4T6bynPO>z5lW6 zrkR>J_O#H-Pds|de#n#$jcFgK49(YhJX76RjsHbM_+q%A{6nkyP&%U** z@IE>W$8KAdaYx{vm3YKwSac+7IY$>hZo&_U@5hPlf2{wpC*xcKL;t$Q`|nY_(f7`b z{Y$(;^FMQ2t83*hoG;LP66K5?2R3n@VWi#g8AeM~bIY}7r2ya~8whk3Cm!-aMpdOz z%A0xqPuyqwY<9C}Ud(yvdoNCEW#6Co+uzmN{%ZF8nR!2*@smSospTyxi!kKp;Q2pQ zklKwus>{oTJu1Kxp8xa83dXjS;cv_Fy594DNI_%^y2g9955a#$ZzwjL1Vz7Ehq}`4 zV}SF6c}{ixx4FCzel%nvTE^jHdl1j*;%;rC7tiU+Fu|zuG^Uh|i6^}nk>WXBv3``l zsONMkS~TZHJf|yDfuq;xbGjr%(MJhx^M+JA6jS4?cFO)wpz)2`D>`aVbk_DX&q)OqH)V_*6Qw6dT&X?L^oVNo9e*;w`z(n< z9UkG&%g+rR$FrXwj)}jo zBl}5@7wyjeskknAKwPtMA^A;2sQ)_vX?zqDYSJbFq>hb6(ndk+(SC27@xLP5=SoT% zpNd6&%qQD>EaM-O?Unyhdva-O`z4sX&%_!38=FCl;^yy+S;tp?RDt%rXz#I%KOoyH z|5M|ub_zZXfOzfSiPN5@gta~WSQMg;A7(b`|B7rc*^geV`d^3k|1pmIs{Iw&5^l=v zZyld}f^@p2UJ_a^nQ}mNRqY9UyX|-sm-EvyTy(ywd21YxcvbVl+D_Fy(m{;vI7rSX zMW?Q6eyOW!x2%AKryEfHU=Qp%&tl3ex zw%uD)vbtnLaZ^Lv$THkhP;-tZ57dR%eJlY7At@njetT)bwzb~kHTj!bw=LgZRNcL4V__ifnB_ZbH-Yc^ zvK3D@bQd;5-WKTU$@-GQJYQ|GzrNK6`)sJFZ(IB1YO=w~4b9%7b*qcp*5oy`ZfL6C zT-{!>$y-z7uU^(t($rqMaa)zQxMcN;vNide%a(U-YHjJ-T!z=;)m@ucR`?rRH*`aP zh9G20tj{^S0&%^BCKeb4Ncu zIp)*5!p8w&mgj2t~81T7==qz(8N~{1h7$DYU~8(k%t3Re4t9a z=RL3IU87N)67M^rdp5_eh}f`!Ez5kAC%a;D12-c=WWti3fl5I`AI?PW+tVc3@1+ zEA2Y`zbW~zk|uav%0Xks8}|SbHR|6jO`zsiGw2C^MZ#6v0tr`})Kb}hjv7~*OzHXW zB>#B26fbICy)Nlrk#IGio|60wNCPVPM$o6LLDAD%)T!Vpz8t|h5*9b3_a#H__HzHjn;nsd?C47dZAEhtq$9PKAW9XlrbE3)%2Dp*__;uh{ z4RE8pxl->|KIcfdRo($1=vyS*D(_1YZk4xGh8#6+o*Z|T0d5>ODB)^cMZa6Zt>b=Q!mZ;{-a>_P z7n)K^cNEvCD7c5$7;BPO1y^z71q)oohXD&*#erc9TrIuo{8RN`CG9i^{83Tx9tmG* zfvff2Z-J}zo{6-(qprTW%^T=u*dsK0S9!bI8k!#GKRLM^4`jx#JM!OnqMvJIH@owo zj_R7*n(;G_^&K6z)~v0oRWQ`->tN$=a^%*5|Geb*)KB;V9r$SoDQa_l0CJc4I)%9S z;S%~C%d#~!3-Eh#4LbzY3g}~u)#)RsphrU!w9#AYd!o6)XMzju`I_X< z-Hp%UR*XkKUs2aaKaaO+)rQrj&E6JYL+9%D#;z6&f0MgEZiAL~Z)06&dtD=EOb7(_ z1fQZRWHCy(qCKDpZgnh{+jtW;^9Htc)OGlM4P$>-G|=AY?ewyR{9QPtu+l0>Qp!<+ zWMM-`XZ|C%ppCe{FnT*XUQ?S-zWu#fd)^_tc`+Y5)Ni-mp}*`Z7T8btz8676rdp<+ zuF3>;n#?~GQZCb~%4nZYFqsxa;U0N+M#eFPQ1MFDXXA!aE{_7ptAD4sc|qo5^w}E4 zI~k6nIoRI;NaGuGYij9|!Pi3-Ls8?a>+`Zb^=&LwDtI>k j$$sPts=dmg_hVq|*wo}~2fgEm$EEzl20h8k+W!9mWTOyw literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc9754f51e923c13afe1180ef003e4fa4.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kc9754f51e923c13afe1180ef003e4fa4.o new file mode 100644 index 0000000000000000000000000000000000000000..e73700926e7424a3ebd17adf2de147e46db0f0b9 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m085Q5%)>k+KsL<8v#kdR$$*4Fyf(9 z-;QG=A8yb?A5_{x*KF+Y|Ff~)qwVZ4XpeMfu)^=FRXNRY!c2}RZP3|4E zO{Bh#bf>ez|EqiYhR{yeY;V_M?BP?-3{;#yFtFqDGXpD6KQr)6-7^B(XYo2b`{cL7 zy(f-`A9H>ueEh(P@bJ8o;ofb-Evj7LJl zI}Qvy{me51sq-EV&2Iiq_`=-d;eD5X8h+aOR=DZJrEu@c%b+U^g$^DZ*nZ-`z`l9M z!xg7bgqvDVhkFma6&`-*QuxA-OQ2aCDt8_nxG?X@fqkpd&;IX(XSZPdQ*VXK&tD2} zzf9x(aVRto!?Vv{ z1YZ}!<7PY>8YcX5@cP)vaMSYhX!j>*w|}4g3mQf@DXJjTJgzBTetFu7gia(mW$)ZIgNlw?$`#QTqcY>^}+qC469u#tPqn z4`>?UCiM>Lj>o9oQN{{i11y)f5qwH*L?%uJ35B%9kuu#`K0o{evAJpf4z|pUfcDF1qj|)%{zq2;A9;@TMgE!fU3yg?dAEf1ox6{X{L3QNcRr1cTu5RgDWEOFTpQY( z&^Po?tZ(Q6?eN9_sgGQImmMDZn}pENx7gu}^;lCUu)g6RO$ID6H1q&F{I3sy_OFvd zXE*G5l;#?EK74K0Yo@s*)^{P#Met9u41re~^F5LYGG2m=PRK;EB~M}_KcTq_`HXV0 zWOmI1c`p3zvc6%+S1#om2G7Ibc^EvCj3nPh@Hr0sdm)qa@3hc+kYo7OgwQbLcn@+I zc0*u|8TiKCb*>h+w}#%o;QH zfhW>cVH%p)A3DNjOKV9?m@i9{3k8v$6`|8J)8uu|*re`E&t|yeb zxG3cOLUD*}LpC8hQGf43ujimY7vg6i+0(ZJ{eKhJHF#Cvz8lvAxSqi4f8+i%uKz~k z!(Q(`mk_eSuFC&xoQh-fp#C^nJe(U4& zJ9ad+Zf%Rz>pNxr@%ho4`nDFow}Z{^@C7^O3(;!pJ3E@a?d`SR=H_kQjz(Xgw%*&` zQ7f?>zSj1(fWITqv}obB7GI0ExoKN#!NY9@wg0{t3{YE8xUhENqJ^~u`T6;{;NQZ< z`HvJlyri}^A94!0?gBB|8@&Nv!`&lDz}M~zJPwIqrGT&buJAU<;k_#aY4Fu`ZmZoI z@V5Abb*upId=muIte%#JfY;y1C!Igg*3s6{Bv*ieaM_E`?SYo@TWEf?r|29`m^xE8B+?IMqmnaGVV_tKDi zX~>>F%Dz9FaZUvmE@EF@n#vb|UMFXZzOy5N+*e=n#aecpbru>Z{k6 zmzEc~OR)dPIf^KBNJ69!!#C%re<$PQ(LRMqphvEeXbib+vjcf zxeFG}Up&{no-XqXm$Kr{rsf8Bdslmhuf^Tc+1}x9@NI2s^|{woR<2xIyM9^q+VaZP zwaYfHTeiA%St*-W>0acX=MB_1y4zcD9`K0QjiHHGI$y!(9~NGBuqOexFK{sOgnYVB z7x%bj_Kev5%^c_08dI5ls~vVh+ehr8@B3(U1`ou^(3Ny%mvuW?`btr>Zllo+b)C6Q zbvmmu;;B!n+t96TYJ}-4am-X})9kZTxlPHgRP3`bK8)f@hOIJiTLIptr7}Odcfk>> z@Upu=1B2l|f7!Ezm)Ww`F?))Gk#~37aR0D`@4(Sk3troRK%sHy`mwxTlh==MEhPjB z`Ninj##qlTEy8{UBpGNhjb(KSeep?>>tx4wCPw%HTnZ@aoNPWF(y0Ra)BsDR!TnWN zZ0r{~tY^PH!ma>7OgA{$o_&dSKJKQRRCe6We$*voU{3*${(6F#ND@TjFYD6T7UD(k zwh4ajLj`39UgwS9`XezMsiR~m<(-WyqUoyJxYHvjim=BCQPLM2M#5%b41TLeG<_FpBJ=G zq_a-ZKhg7PCg{;W^kE}${8b-t{_`N8fgb#%Nqd1G^hfkue;FMb@q@Zf^7*U6HsUua zI0;td_DxWDK_6y>cLki2h~Lz+kt8;d#tQ!o{iDw1eUta+Ohe6IhyKw2Q~Owi>|uAn zo`aR_{#`^%1@c`=V_)eSAK~U8e4Z2f(Z{c^sj1=ezL^7gC+p-(5eM{x{_}#9CzHR8 z=lU0QcIIVW>8C8~>(aciC;EpyoZQLD9^j`iz$Q-Ojd`2*D2CbX$FWx>c2y+b8Xp>34fgD!$!VzGpVEhY2|KNp#FqYS;s^d!7n>|Ef^%oncyozA*k@0dV=kX>_h+ZFzjU%0!8tPtKZoNNuqV6N98zuK{aAE{ia>7% zJ=Qj&Kj!3kJL&>o0=ye|w+Zh?UEr4iKMnYNfgcj>r=c$JUj;rBc#pu(O5^=!qAu{s z+MX^K@K{7?{6>Lyp)T+Vx{zm!#OFx|e74M42pl; zAAe5$uQ2uBvnxC1PlE@sz9rbm-}s_%dI9_k>wVO}$bYgEXuNFhKZnSk*-`%*bZUix zX$et(#d_CD|=yyslT3G88Lm=wOaH|^V^_rK3_AqzB8o$;Qtv>eb?n+y>MCd?Q+HRT}k?K z#pXAkA80T2js7!&&=v%U^!JGMm*xjIpww5ESzhWdx*iM`v9hdLCi%wFSI;iw1832M z*00<@f&4auzmfb7i@(`<{Irw&r)WL1i~NDshdNp>MSB(BW&|Ca-VyUR2iFJr+c`}i z&7bgAz8+)#t$_VR5%bR)56Qk~h(NC2g~9RT$^LwN#EY`zJYG}rpl27Beigs`B%dCW zFV~59DLyN`kMoIV83JfaE_K zi|^1U2_0z+@7sUp&Gr`i8V|2AWMwjbY#d_&1sNBNG!`KxOmP89hL*)Ju+&S60Azhs{q`zQRH z=Zg-RUy?k$sQe>fbMW=~f2jN^+4c^WGc_L8GbgqRBA@2#S+b3y4fhYoOL)HCW%wD% zTi63UA3*p|`fgh?&yO8EKep!u9m%u)F_PmXe?oi6&&T8Qh1Xr=d?H_YqZ0Co^&~~m zYiJL3Ybu{f%m^l<4u<;4KdHYI3>U@QGlJt#H{xmf8u1CJ8}W%qw~Y8C)MI$dddAl` zyGzECG5x!eVZG-5;on006ZpqInw(UWe_(!|MV0Ie`y=1aBzF|?knBGtW`Fpb}cdf@Yd$&RQ$A-_!zxU<~oJwo*4PrDs_eZ+Wa zL=S(;H2Pa!Fw1RhU%1=truoL{xj)(*u#WIYj8E;O{`g@Udp69GEOIuIM*J1_fL^pu zqI?JWw5|V1@XFV>G#BzSceFpqpB~88r+_}DLO>@1;ooL|KScA%I3KY7nEV}h(|iEl z6<^%MT7c~1n{5_lO%kv%Zn-Mh32gpnKsQ+KmD4&9Vp-_B$3131){>9v=lYY-F zKpp78Km=857L-38=^LkeB-_*9YXKx(gf#{n#*{^Tx=>`Q#Guj0LqheSuhEV14Ae-!$`aG&4o|C>4u^lnS^Tb5z zU&Zr;pqNdp8lxb8*we+(0ENy+u>U&Lg{4*$o%ZM0AB#gw+8@5j{hhQE@fXYWEP1gc z7yP9M^i(=AiFA|w4{6Jz`GUazG{~KS@$unbJ>W+>{=i4P6JAeXoF4+bf%&rAfujNx z8`g_c7=O30hrjOqaqO2AZ(rxv_f%@Yi|VLGcj$jg$K^Q@%s_jGqLi{DboMow*p;iK#I znE48y-ojC{pC{l@u97SC7Q z@$(hPljWxP9?e&RhJ0gmzJhq_&Wh$ML(tzR&quOoedX~I{aNx6#7k2?;=p**-Qf2V z4+NQ=AUyxT`F&aED`Up1qo#Oe zIA4eV+)liT?!U3VnfKcZ@cF`6`|aIa9JN02_)JG?^m$P(#gqH^e4jiKQVyoMOz{cp zr)ht^mEuz(-=Fh%hWsdfx1Hxt*bk(!H;}*XPDrG9o-{GdLHST&&`#kxf#J&?Si=F>i_qUk8H?iNIVvKKv z!3jFU2Uuc%Nb@6e!bDg|?7wrc|IV5~pU)Zg-_vNnXoz=C+K|Nd-%jTk`)?emP`RCW zS9u%z?{ut|U{LH=(cXo6bU%uGZG!Q<3HS`-c@+77Cgq!A|HD29(vl$e(*+$YLh+ElVg149#r(WioM#x$!;gaKCeFi;;XE9a-HGqxSfw$|vcFE# zvEOH30|EDE>_2bFA3hI^08~zyHs*k*{%7%9@0~?9J`z?M-24e*Eh1e;0Yj z&X(1znSM-PQwdeD|)UoQDfBr*fftwyrW1D?zC}0)bm5c6CdO1YhZ->qL*mL zc^hZt4b4~#zcSDACKf; z6fh94f1c|Tg(p5<cQ^a=YT=qjI~2-0}K|$^}M9RwWsRL@`Ka z|5PNikM=0Cgo!gr6Xguj-H5Z%bI{DWO}BC$SLeFmEIKMWG1^|}x~Qx*CEkn`dMZX6 zjaf&l0@R?#QF!8pbg)L(S3QWi=jI;NLz}Z6&0_;r6(c_aR#c~_=(&!>-|>|~8|7Uz zwpvlFD@Kf`>uFH+8-E3gwco2-Wp)|Hjc41aXM!#sURiVy_zI6wZ?33nwNS~ zbY8jz1J?HOv~8vNyw{C7)8oV-EC2E6RURO9>_=K^Uc1cor+`;x5o?#tvUc%r3meuh zGyOZ3_Dhg88~gKE>Q$CHveoOaUB=a3_}*CY?MBgn97lS|_nBSKMwJItX!Q*0Ifgpn zsm_(DjY~y_M>_zUHXUrN(ezvwVzXxt=~zQx#!8R3*sg$bIN7UP^@2L}IvsWDigZn* z=Bj#ZpJCH!l%mvi_9EBgDK|G7x&T#BQxvBZjc!8_hLQt0UtB1(VxW{`Du*#*+tEG+<$i5SnWAh-8jSU z#c;8PjHRl1XNt$JK*R88R#S6nuPSdU;i%~l{8FAIP zCJX&R;u4JMQet?suv|8GAUd>W%$xsBK7?Q>}D{SRF?&D6xv zr-fF2;?Z07L#BLaRQu?jlh46~=n6v14~mukPOF>r6KCbK$`g-XZSH8r1^pepzvTRz z&+JamOZNeJtwj9kpZgz=L8jdoMFaJvD3J*7en;*lRO>_N`ro_t9ZE zcH63qIRgKz#3Me#q9a+$IlAz16TV-3KTd4_WBrdl8Rrrh`qwqyf4Aa|zISHqU*Z*- z|C!qwT`PCte1YbZC}-?Au!-{wBkhLIFj}mdTdv(J1OOk|K%lcY@sJNPsw$0A-puoV z;y&AFvztBrV(v@deQ{z7``(=2{C1=5uV&7jp8u2VA0J9fEpJX)fFVBz&;O}{)NcGy zU49wfYL$To`DEiep)RlH01)Lwu zbE@mV&E|dZqahQ~QVt*8gLqCCcWV>9curS_2}YHtFr{ovJn6-V6wm33^`rcy4y@2& zl`3e?i+D~~rUFN=(dTqYh@y`Y+~y6Jr zwbu6dV@S&xm*{r|{gEb$LpGrGj>b~#C=HQ4tbJ0uDmmdBsbVNrTJj6|nYdU|CGGcR z`{RbTbZgZ+`46@CSlDQgCU*lzg*=aVx3(`QB)*<0F0uEo$o6>>q;za;Py4wiE*I*w;QKO7T(Uq|+n z9xvLR{bO-me80G6;X?A8h*1Ae0MhsT$)}j60IOBgsw$GE4G(HuJ z`j}6)_gKb1Cfh6jrS{~~*7l1rd7q9m{?|8y7{$%s8MBVB{HOx$d(hrv8NXk)SN^BQ zSM3yhDgg1?zZ0iDO$lpz`mrcP9Y4%$(*G6NUa}v(SoOaa?f+{W`BnQXvL)P<+uu4q z`2^{7OPwUNTr%Z=>Z;lk_;%aTTB_!!Ww_{kRrA(39`UN?g|(fkd!&OH+i{SbPl`@m z)%;Rd)oxqS+V$;4%ZfxPSyjBSxwC#zar34{YiqU_HLvq-Xz1LuvAUqXWkb!5!ZmH) zqLNi5>x&!fTb6fh-dNpS?_ayFxqRpH`h|^vt*s$^Y4g%Gn>JQ7)@@jyzjl8Uc7pH!^Z0NO&eD~v9e`#K|@(hXTyr-ofxAPe3Y#(Ti;o~ zaA_Bf*RY|Yt+;$=Y2Bjr`ORh9Dx%|-ZCmXvTH3a(Y}-0XQ?{vfeaps0>$l$&-NJ&g z=$3D82{vwCR0(~wl#$F^Hl^Df zSGSdH^48S&tCu#HG`5v)*jD8&E?Kp_Y<0orvSpo{TADjIm*KT&Rp;gv75;{n^_DPc*EkXx_A8A?#gUu&H%zP2*-^ukFPZThVuZ!^VnEt6jV${wFpyEq+w_o4VGv zG&comchoO^q;T=p!UEsYg^TJ77J0Y&3JR9w`?lhTN_>mAdKcFUy85QA-g@8leqv=E z|7{ieRTcc&$?a6~Z}4pLf<=oLFJ6?Nzl75)_cpitYQ=j=4SrC@=WWRk<}qPehg~Rof zy0_3x1N@xA;aaNhOP#Lg^l#V+&;RgO!l_?g4EQ`lFq&n-6Md-xAN^INmCjej(Eu-s zm6qt5Z-P#BZ@s>Rws!WuI+8G<<%8o)Fd>Ihb__)~4tCJC-%y{FzDdlxb6R7#s1bXV{iiE4S3ng4_QcGq3xw4-sX)>kf zznA>u=~BF?dG(y+^A!nK^XYZjPro#vf^PskT@8w!)}T%WPx0mOEwcWP0?fXJE79j; z94aTpg}sd{!LLfVUBZ=o6VRXX9YRf%_(>LczJyN!T(u?87bScu;KnWEZUbCy5KX?o)UE%`lsidsPeo4Zlphc z9r#rP+$e9J)Vr0>SrTrQw_h5_D(@Q-Ztdr15^k0Emxg|f{C{MC8|BTE^M}SY#>o8Z zz>i2c@nc-go|o{4Ebx8#VM~#~=$6aZF8^;YwxEfc{@0M`uxZjg->$sG+ zP@&v~rj*hh#WgAl?%_4Y8s$~NRor;q0$1^&-vU>0V8{YjORqZrRQ;>Szga+?QZ)ar+&b`|mmHh=abKVvKMf&8ZK?}E?lND8 z5EnmOLce2Kwz_5>eowA`hoD*xeXR9$6m{VDC+j*pd;}HrXlQ~qcuRecH`V)0aG^b4 zqx`wMv02=T@#yC(YFp{&@m8)}zpAv!+w80FSk>0h*^J?Da`(q=(A?&2sO@O0ZQzUv zfxw>NQ&fd4MhTa<1r))pj>YmCZo+2Xz_#|N1v8t zT85kmk#&s0V8-mkJ`-oj#FM3E$T&NhIGWAb$QvKxIGDkjY}O}}b3$Z>kpMHe<8>f0 z?*0CHbl-c`Ai$p3uoQIPTmS!8)&Hvc|Eg}?{_@fl%XCd+s%Y$07OyENW7$3Ndk??$ zuyjzmS$9ppt?n1OZrlH+7ur5Mjj=<6w%p!#@j7eEo&7Jkp2O>3agV0s6)jz_?fhZ* zcVpPq57+DA52|e8Yc_W1|JqpZk#=?nw1>O1Skd=#J$)aRvbTS&vqRI;yKBzaCiV{6 z##3L1yE9qQ|Is~tLujXKwzumr_VCGP1}e|(AJ~5BnSm9jo*DS2?zsxuXYo2T>%_Ms zy~mG59&>&ta%}(c$ne|~k>0r{Bl}jJid@`xDl)w9Qsn8m??oPJITty0^1Vpf^hd(O z+xHJV{me51>2n_r&uaco4alW#>T&RvXb zyF}ytX*fI=Z~t>E1=8s4}k`HOH2NCB>di6k+dx*BD2n2 z0ACj(W2QeE9wz)U@cPh+$lm00X!mDmw{M^Vu)WDABEt^?rsZDwY4TTVhA+GvOx>OP zgVf~in$*{FLu5yfjTJdyBTetF@xiAiXr8NQY!iA3w^?UJG5Z52>^}kiC46AB#){s6 z4`>?UCiD*Kj>o9o5ypyM11yiX5q}JO`ENJ zlKfBkUO#;ARruyT-@CJYulokRcVrabJ7V!Y@=4`?{TBaI{(2)Hytea4;b`f^w}&3k z51s#~lB;ez%ee6ClB>fnFYWuy8rGMN`Eh0jyE^nE{puU=>%m-h^-Sr6Lxc6&p)-G{ zUmM!T`e+_8t^d*Gz(=2BebIkreHUNVufAK#`p(|RuKvqH)^{$0T|J+|uBL&u7;|lC zPg38|Ke4`{2ed;M{)c|`!n^Fy(BCG7hrY!QU8u*JI*#=X|9B!`$>E^~*r9)Y0JQ%t zC46T6?nh~^f#*ZlcD`nsOJaQ&@>~G_B+C$Zr7_%w;la|6W7&vRpP!I*ZsI2$LoLL{uHkN zM&rX??>?6lw!yB-|7>H~)pwzP*K|GXf_*N)Z^=%-fepy!=V9|R3t8VF>_>J`KK=e0 z))y&deN69vlx(W(fB9Rm6@2+U`0*v!?B8I&xIeH%mk+|O20J!R75X4Q{kW<)d@ZHq z3fb?TKFs=f9?kA30|vyHWBx{WdW7%X|R*qqFc+b@f}wpbx__CJM8 zl5De2+?@H3;;L_F$sV?tC4r+!L^-d$vArYM;q~E8UAtODonPzP8(SMfY+h?yhkst{ zgqc?n!WAqb>8OYt=^7Cf3U9J z+ul(pu^s-__O@W4BiOWX!PXXki?_LHYir@dZH0CJz8DNpS6H;5u5eL7U132%!NLV~ z__ttD!6StaFRrUAfSf|EyFiTgMsLvHaQ6rj^tbzik3%9@Dd=y$E4&SIc<%~98vMS_ zt#w<1-WI>GjuqgYXM$jw)zi`t^adLFqzeSwI@&tA0%%USw$}CzZ)-=vUEsd;-av~# zSbtZT8K&W~SZ&FHa_Qs-Gy?7d8!BzY>+*IAX z%*v%J>jb8*09&;tZ#$3PF+8PPj{Xm@PUBt-f!URwwh=m^0VOFJ99ZaQ6PYW8+CwY9oC+S;1i=eRMm z{oZ!JyKv#WMRVNi=rXTp2`lMrYHo11ceQu;Tih+3?H%q0|CXj!zk6*})rvKB>z3B6 zsi<02w{*kWrK`%8ma(~2?uG8T-e7&ByS)YH0grgy7@Byc^A&vlVc~T> zD4_dHagSSO&x+sQ%yEvbF`e1B*kKp6eZ(&MzK=#{@IaglUCv~7S+|p=FBiw^HX7Yf z*O}W?r?V;}p8BM^4c*!%N146?$4qrL%|0ug+m!4|#Xb|`!ziv)*eVORmEdhkIt!qC z7aXw~FS`pgFc|*xmwl@6GF#4CW>0f4^6qXM?jM%$?Ks+M!D}lJC^QaTKbF^P^7;|3 zWrRQ>zZf~&80*=oMcJ=`Bm)g*u$(TTFFr}~ob1?+z}^ohqPD53+O` zJWzev#(tU0diL3)>@onvbc2)a-kWUa<8I7NXUE*^$6Z1O_7w2wuP20wBtbO((k`8C zCSLSzo8ad@R8Xelb=Jua1?_gUVA1{q8EkMq%MaP^bANvli|Pkb*c%7gj8Ky8x0Bx7 z&g+A@_M(t(BRpe-hkT*&(OlG$CPLd7Z$>CZkI|pVV41=3N#m*ifrBaREYUMgpF;G& z=ZEa$>8w-qPxO4633~Joeb`8xK=lWl|9r@2pa(x0(q7;P{SiIaUsi`k{GhIre1YnS zjrdIoO@I}-eUlVk&_@{IT|wsr;y3+FG=&Xhu%f>}|EP0$-{k!{Gf)fo&>#ALYA=hD zJ?swHbFhltzmsUGK)#C^>?>Vkqud;X&v!yU`q*`~wY6N{H*+EHM4fyo>VSUGe|~7< zMDn+>T>qlZPQR=x{gmf?U78p6ME|gdlRG)t1N;;P+4u>3d{OV&nG|Io06;zf|H}&6 zqU4d$g#y2{OJkRb0ZsVF@}QnURi=H+`Edn>uiJhDeV~6mXk$*olM=|jq3L}1Xg~NL z%&@1EKVzW`Y=(Y-&kE&Fr*#wl#oOnHV2+_$Z6e7xkju(nI1DTmjDMc!ADn(A8r98KNg*# zV$j<`kF|~Hk2*Qtj=I2?0`CUiZNj@z7x<;XPXWF_;DpnMmY($O=uL zPVtZX<1eWH<);37c214^)8GNDZ%H=tH@+yGnh*cNdLQ#I@}H?mG~QJ1KL^R4Q)B)$ z=+ugWQ<7r-iuGW6Xv!3#PmcTBlx(VVf7^rhF0@zvwurT&!ba0-hsMk zPyT2JUiusS8~U9d)335OF+lcyhxASNC{o`vo?ZFpU@7#|A@FiOkJP_J%l6R!bW*qS z57IyXM2hejul$9@rv7?%X2tdGYqjW`=C?uLe7Di!T+;j`u62wy>MCd?Q+HS zT}AqG#pgGlA80T2jsCMj&=v%U_Vc5(?bqU?}+=GgX@F* z?X0Gcyyk6kiTL6%q4zQLH*bc*zXAdC?A0TQ2c#`9Fp39 zK=L1n$9L!x@>t@#Ziw%>3F0|&U^0o+uO7^f$LqoQ+VY^CBDW#F>pIz&$9LpE@MnB< zO#a-RLhEaJsuDoh)9inwf15Er+mG!)zM~xyzQy^J53kkL~#(N9xRfjOIGYpU@ui^YQq6;dK`|pU79b;U$sI*JB>PW_+aLaK zo}cqMJ+~*#3FxyZG)1T7mXZEW|5sYSrX;h=QJ&unPKAG^X2<+%a6a|}dD##j1@uF- z-*9mMNsaAKa`IyI=V-s-Nagk-|Ka-+H>b}bdf@Xzsg9UGA-~NGx^vv3>WeXWHH5fcdGW4yc^ z)cJlB{lUK3DLO>@1;ooL|KS=Y*I3KY7nEV}h z(|iEl6<^%MTA2Rzt{Cz6fm*+d+H!EbA50IDeG5^1)Q9cF#LZSHh6263p{EN9! zC;gtCk2=x?>X76)Py07VQe3~}AE`+e{btir74V680DhqjH9xB`zmrgh!ejG4IWvyW z06qnEBi@Bm57nOVnW!7_S>w$3aj3`fkeB-_*9YXKx(gf#{n#*{^GC_Y`Q#Gu<$RSn zp1p+paTiM`|As$}VQ*HEKc!QPKv}>p;>j0LqheSuhEV14U@F-!{drVHJSTl8V>@Du z=gG;`zl!HcAu*d+4Msu!u)B+)0ScXuVE=Wn3rnpiI_=M~KNg3Wv_E{4`#WhT8Yq$L zS?VH5F8Ip~>gjZ16744YAJmq`@&$qaS%^CWdcco%?17JXC%m4-I6nk<1M_8< z14jiYHmnyXG5#)LkAUy}G3-|qZ(rxv_gy@Yi|FLGcj$jgk8?Q@%s_On?NY{DboM)MuHnkkHrdow*p z;bZIdsQC(?-}Mv@ z=P`VJK8w#AcaD$m&mZLbbAEox^X)h1gMOl6KYkFOH`pig{W$lJ|3v%o_<0Y;!~Q9L zeuedaL0kBo1o^Pl|C}P_zn#ue_TM;A zp>jL%uIe`S-`#lI{}{vwAD#}-J93b+ndHt|K!y_ z{66}Qoh_|hJ?*Hzx;ANbEp{Ua=|7uEalMefII3y#ON+dYYW%@Dz+WMN5>y*wPrJe+ zbTJRq@w$e;SqAT8oN2$u_Li|d+7bkE&Bs6=FM6IcSz|Q-*fftwykkfY?zAz zqL*mLc^hNp4b4~_zbDO#r*ubO>|Ah#ZN}$G_?~JS62u-Y87#+2B797-uEgyrm*>Tu z7?0#%6flsef1c|Tg(opyrjBq#}AY z=rqM+@dr-DZAmuSGs2b-nQ^-r^sc7}y~pIHkrLoza=Q}aV{*HN+==>#$puD8RwWsR zL@`Ka|5PNikMt zdMZI1jaf&l2GpR(5qRQ;bg)|2S3Zcj=jI;NLz}Z6&0_;r6(c_aR_xQ$^gKuM@A*oh zjqolSTdgS86(i2m^)#sZO}qld+wak>GP?}pCbDhJGeMUKudG1*jFj3i@(69Bfl=GZ zi8#~KYn)g3=GT^@UFl&A4?E$7hC@NTz#nyJIKcR_s^c3irutK;D<+8^gX&a=(M0?~ zoo@Nk2Rzs?(Z6GkELF1sndAWKVKry{|0fQO&~u04eR6)x7y~D(if+f*>AOv zL~U+{_tShu9c{Fm>1nh+toTHIes_zUHVtg7*7Q6VVzXyA=~zQx#!8R3_^yC*IN7UP^@2L}It_K| zigZn*=BavopJCH!l;ZTY_F~uLX*V|-x&T#BQxvBZjc!8_hLQt0UtB1(VxW{`Du*#*+tDn+<$i5SnWAl z-8jSU#c;8PjHa_siZ|-~UJN&{;v!b4_%Jf=&>l1KNt$JK*R88R#S6nuPSdU+i%~l{ z8FAIPCJX&h;u4JE1r{k26;jt?suv|8GAUd>W%$xlPpP?Q>}D{f}KY z&D7+Pr-fF2646`sL#BLaMElsDlh47V*a||+4~mukPOF>rlVIhu%9Ds*ZSH8r1^pep zzvTRz&+JamOZNeLU*z8L?yYz=Ps3dk-^4JvD3J*7a3o*lRO>_N`rk z_t9ZEcH63qIs*T!#3Me#q9a+$IlAz16TV-3KTd4_uu1R?BkhLIFj}OVTdv*91ppt}K%g@?@sJNPsxFIB z-puoV5plO66hyY5YocfS5d2s4hGN4>Q0%L9s4MM0 z3OGNQ=Tz5!o5lO!M?)r}B^*Ao2l1RP?$#!H@tm$K6O1ZPVM^JUc+!gzDW206???HI zdQO+3MRQ)nbGot>IChOar%OT(NFkF3|nK2R2b_{t#av5G*KL~0j+m5mSRU~i0onQliF3w3ExN+L$T75U&znI#gZy% ze<0f*Gqj~!tKP|fsJ+L+MuRlD8#pTDdBnT5eFY)$^-OVzzkfxx&zB&jV{3cb&pnx- z{ZUJM)w8wzG@+1yM-j;DN!k7bS*P*oyC&B5PlHWvx`hNhB#z!%s#%~lr>eyH$Z49&)?e`=Y|0}Y6zNDn_ zsaVv<{Ib2rGX7E7UimMzCzrOiUxdl~bb|4}z6r!AZvM`gb$sPVm1y6C_8!al{j$CC zKQ+E;r{I$TNYwtF1np@`SliQ&MIq`0U}ls4FU$6l{n*8-|21g;pA*Qh+FzC};ilaF z*73Kq*0E_rO>=!<&D!RQ9n0z$Gy=A!mhffGOIB~(P}%5Pzph}@hRPM* z^+ns8E30bP>?mK>yzH^s#z4)Q=H&}&AFJ3=v0*_?^QM+%g}&vYZAtTzX5WUA0{-_i z>lZBPqVXEmSGJW@>?rdsTvyOszO^zoZu!<#-r^-~OUt*el{DoWTi3O0Sh#N6P0=kV z9F1<-rj}6SriE3|M@u=$yk!&VuCT3a!PeE@l2wHpTedFUR$SAyaeYxR;g}UWHf#jn zzVc;H)OQs%LEdKQ>Iq+IQGtI$Nx;|Qhke#n`dU{%v65`Cd|i{bc+JX^)>Q@dE$bS6 zn`+uhH+pMp12s#UOB>tD)^Dx$mXxksR=%omQ~A=)jV;Zco67N8xUzH8^2$I%%epS; zuVQuKx+fZzS2k~4zX0~GDcsn)rnYgDu-CSd$}Q--pkYI0r`0ar(!diNn-)E){7qf! zTAG`Jb=&Lxg-Z$^ZrD=)NPWF;;o^dY1%9){yOnqT8kf)@q1eeLIsTN%)6r> zpPct6?ww!y5{y$guaKKaEymgtHB8$zRhyEYiwC*j_al`7am_c}&%wQMkNFfWxK!4r z-|u-&KWzUm$$#niTKYGfFN&UtzJU{nw_T&q(^!WM^slS?<<28xzM;Qpmvy51p~B&M zQr%nVrT~6c;czWe_hnAkbNV;zgy(fRFtu(hBD*V`zXE z#Y#(b%{M`(y0>0mLR&j~UmZ!9(DK1?CYX>zDLalL7BunI0|9Ihms&f)dF0^$6(6XQ z=y}g;dDm!^pd@Dxeok?fD=Ebxg8i& z^Gdr8|1Ti%BKm4+g4d-SG-jf4-<0yYr3uvhY63m=b6LVw+XWJ?HmPN@|2)}GwKSR1 z^YCLUaba)cO7JTZZkKQ+-#GNAe1}luC4PbhULfI<09S1Z^hF7u47hR2xXS=Hriw=; zoW{jzVyqmKcQmdMUMA(F{uOUTxM*3sd zfnPDejq>J8y<7R5DdARm`=x=b^1dPA)_#62;Z}KnW$4Gq|3?P6QQmAhe`s7|j4Zeg z{IG-*KgQMUc?o~W0`E7_8}0kLgj@Nir|3{2ekO@q{=P`U)ws$R8zo$gtKf$X^u}?Y zmvHO2zmjn4xao4<(zwR?;*#(Y^F_j`AH@p^u-O2YM?74v!v=Wlk9&YwBnMOSx;Zst zUogOpay3f0(u1OJmT;@QFG;vn-ZB|-)VKw5+!Y46aomuEt8o?mE(y1e`#lM_j!St9 z70O*`N-5n@T%)4k9$sUtQC<~X#f|4Ja1|f=EpQbFhAeQk^s4hu)xTQrXM#T}3f?X4 zx7-3(>wUliSL;0!X?J^_uc_4=>|)p>G*Yh*XO z^PrCEnp&IiGmpOZ_FHS#+Swu)YVx%f0ra&+p){lRwpG=vnj$rps&<^B#K zE`GR#e#f$WRqb5-o?QKQLA4C}SmW;~?!fO)`Z_!O1QqmXXo5C)%lwZw)%#6wp*?@2 z{JFc)S=@s0=;tfyTIuKUR;*aJvaHG5?62=w+1Ak6jNxx`_s4C}+~#el>u9TM;EV}@ zz@Fe!RD~=?375466~V2J#qt|&!e-v!*7mygfWLm^?}`T78oV7|HjlpxhZI&?0ZGa@ zN|4N}Z|^94_@=dShwO!K-na>WF!gpe@TNAOd^a!VW4rq8);si|0C^4U5I4`ue2hL@ zqj)F7aV!V>K7cg7F}J3cE*X41R527azPdgq+f(1hQmuk_gAp1ZUqJ~?JXv*E-6|*) zzlz>k|0_BE_1b@AK`6def0+>8Y7@e(2r3Hm)k?cfwjUwCZ2zpKft7zh@QLi_kp6R7 k4o~(YS5WO$4!sWpTgRp*XWHoTY*`x1vMtH7EXxms@niI`<+p_&*nn+Ge#-czjBPju zB@AOqAfX^Dy#drL`&dlTXiB0B`Q3I&N^=KHU` z&tvaBBaK}<^fn&{6j z%R$O8i>*87Y5P%W*z>>W52<=4F?Qm-r}XqM@b{XhwCa~QzmC6O;WM(1v(nP}(mh}A zdoP;}zq`rZ_s&{R-z5(_@$VjX`s87D0%`w*jb(GcS{mtpw~77mXKr?4Qci5$tDb_> z=RM=7tUqBPHuuNwNdExxxm}(gwxRCIoJU?c{qSdAnU3=tpZ&}$|HB;_M%o_yo!EQv zsln4Po*CTt=2L@D;#|4<2ZMLK`TfD^IPb@K`J=xXT=CR}!Ie0#!Z|SIhQ5D#^Rutq z@%m?9$(b>)Z||8W2T$jIWpMdR-ygjA>lX+69{$bX=_h}KbaVUq9(m-IJI*}(%0n}r z99;3znZdmWUm869@b?EVeEBzn7x%q^G;{hMdgL>&oSyOUD}AW@;`g2$+>1Idu6}87 zIqEt+gVNm8cj3v;yz-g-kGxX({WF7~`QxVsAIkggU{(H0gBL#g=3v$Lsjm5bm5^8A zZ||BX(f%(FR_=dk@Qx>68ti-H*MpyVgwovDcj2=Szj7LB9;*4uVAayU9?W{`#lb+! zi-UX7=8OAYN1FOR($$Km{`i%LDxMsydKh)pyg0a`;(LStH1M0j&zv3@%&J__cVXoN zuT-7+<5w=ML|rw1J$NDa2ZJko-$i@g7`!l=ILl7e0G>uqx-N!M+#2 zk9NI;{{3dKYV})4v#4+RqksHL-@~W5jIL+7N{5y|_0YP?H@+Ory07$XmM^w0>szG; z-PeeR&GqA+x!kdJ+cCzjb47+<^;~zFWbJM?*B%d$iSck9#+w)u?JhR=d5j5{i)7cG zKJWIgqAMJOZPl?N*>Rt#^*a8SCutJX*^nE zF@mx94;ZT-kHtT%u^1a=ES^j=7Efwpk;bSs9?xmx(Hg&_jmZm>`wv`bn)Sm=_qh5# zIOOiT)Wl9)*i`9#e~+hcsLhLUoB=!P>s$QGyL*3&v3=^5 z>;L+`$nf9a6dA7cvix^EzSIBi@q#<-^xY-C;r8*qOOdtjd-oRjE_Hs#(|_t`k>Spt zx`t2vT|<9+==j3UgShr%T>D#G`(i_X=NjN20RIK>=YVT~zYE+8d>ps{_!-~_fDZ$I z=6S@yMb>4{tob;0dE7I4E*=Nj{@HUJQnz`fjE9*gq}IOpU19h|dq{x;5;IDZpo56)l1Ie~UOh4b%lK8^ExIDY}>-{SnIIKPGS zBRKyW=Qz&4!ud{|*FztlM?a?Hd?(J|L%(jpc?aZWkhkGng7YDq=edRx$Jr0>fo*Je z*PPhto^@i^`Pvh^zg>G`&lhS>G~#+<0_*=de0X9k>rZ%C|L?L{KiO5}JJIp_w->gb z|MtQTckoQdoA`W@KIfn5co&}s>2uzhjt}wqefpexhVuEt@pW0@*t()fY~9|cGA@Nz z4|tz1jfIZ=ea3M6PvZ;OvQyqu@cZFcPflT5pZR6`-)3B*HnzK`pV;~B87FpqVaAEw z@fj!fY@cyr6540A@dj!uwGr*3cK*Wj8@!!=k2d@*wISzB$2al$YxFt$OviWd`P=k4 zaHiuWe13sGXYqCp39U6KSQ50&vbkqpL^-E?@Y&E;Pao-=M2%F zqFC>-cfG@V-}Is_1E!8%ozng(=xpmVf7CwcHT5(FZJLHQO+%Zep-s~iJ^cc4l&mNF zUfzQG)=_=lGacLT`8N9OIn%KlpSROz_nD4^_%r`N57?q9&)-p_b1 zb@zIQnd{ubz3gPP^L+5czPhabJLY@)?`p(fueX2iBi`YMkhb&l=u6z&-`*Hs7>PXN zZC~oX)Rx6Q(ZSe;3y2%D5Hs$LH}@|#uT>(Cm#$GgD04Z^fvniNib<#cEZfEUOS72p z!T0%iJb1z5{$VBT;bIrYsk`;ea>j*%3D#)hV3#-8irSd zkw>qq4A;w3T)zc%RPgJ(EW4a2u2-9N&r7a*-vf}>n({>{{X&^1D;W3!0FBjL82$3V99W_nGqE6!JBcKP2-1MGE;r%5Td5Hii5en!#wA~Wgwm(D{->1wy5#g^eL`?jnZGA zoc^1XKE?HKQ~JW>^xvWMDXzaj>5G!nzeMR%Tz`eqk55kjBTAp*`oB>63CZbSr}QbV zU!?TK$?4yu^eL_nQu>L>>EEUFDXzay=}VH+f7sL?xQ|~~>_PzJgXVhe?(!2AuF|i& zFxMkr^bCCeJ8PFBrd@*Awz=>9Jxz#t8~ZNZ)6jQe@^|;sJS}nDgIL&)>sgQokS-hP za}WdP_FX`{H-MP*SNF31*KcCOzeU{nPIg28zatKP_jqLZmxmA=d@iH!J;bL&zl-z_ zBVPS^cA)Zo#IGN`wWjaV7aRHqre#%L_(5cN05RK#{~8(o`7a{F=fBs`{|m%iul=H- z|Cflxe)%55XBV!q;n#oL(Es{xUH#|3=j#6z;~yZYbV zw5so&Y*+tp|A7s^vvzggTTO1%<3gS8{x|YnXotK1Vy0_&@UzrLSO35b@SUHF48sTZ zzdZr?dNw?O_;Fwwu73r3$OryM;D3ZHANVW4)1c?+lot4Nz&8NTcK2WSVpHFR>s`YG zKVZWbCNHnN@HwRU1>&IN9$MSs>p*^nc?kS>7W_8d&kxr=(?{1ZcR8JQEy*RuNv98I zhYN+gcw)pI+XD6Ipj`W>${cq;?bt2C{;qO$___Ghv zwc|xY4{}^|_7um3L#H?{Jo`My*AES#U#F(kRQk)C`vSODLD#3x>+_c{>GMN2o$#~Q zo%NS5<#IpIKaRKQ^jGHeojN|2xAj!h*t1#TmcA;auSA-@rqJ1}7bC-`kB5fBuE;mC zaGmn@9Un7vs%Z?$TgJ;feLOf6z;)D%`AHb!daF?5|4_{o2v2%lKxB}@0_PG1~ zVU&hy_uX;#^37NGo_sN89EeRbbOjMMSBR)lNy zINyM?zkE$!0Y1;gpTB%<-#DD7@z1S&KAflU&o}pZah}XSuj_N;j4^}s>-%W@)Zwpl z>W02CIM?#e8~gHcuHm0=>C45rnt$HZmyI*&v2*HX>gRAJ{$Bmwgue5i9^d!cqeXqM zKUUcH&R<^Nck=kQp)X{1SN?VWC;PtS*)sH3O>JlY=GiT0Kb+w1|Mizk`+oCxC4F!G zlH?nb_P5f;b9(auS z+t@!a85s6x;8EZf;IY7%*A9FdcoVP(81vqNdx3WV|CZq%;rwfyF+UzSjO%v+|10of z;C&2Zx%KS8_ZY@0&d=g}yme@x4cD7)K0EMtgl`@i*hF~U*?}(-UNvqhH--hB1i;p@Z&&o-uKkBu zD=w7NMK>ZbcMi^OH}m9rTyBqxd9x9on};WlVeN(8E1nxRSOPcurf8#??CC~PUF`z1 zo%@(C!s=NDS~Q+g)*S9Ud?b3LV>>=s=VJ#1=I6G@?pQ4`-;V}ICj=s;J;5ms|#mvN*jgeMtFC@~+w8WTM~-#v+QADk3)|jtcvsub!}}rFdvMRj zy$6rfw;egq(YE_g^gzcEyPW2ZBOOcmrN&z~ENQE2+kVJs+qw4u72HODc1o=UZMEp9 z=+_@WHx74pM0f4{!|6zL*Wq2!JJBQDrRc8xe-NV${pk3E=*Z4p+mG&P+a2vVuuI$> z{X$2Las@Qo(XoI3o@hr`CnsH3^w5z*#CVQ&96EUTNXNk=wSNG^cSlFpfnCuZe-O7B zw~hBjznA=?X*&50lrQ`N+(WB7>X_wM|6nL=-CN$Fk#4Z*RAgIh)Li4+TXT)7oyj;ox9-w4(-^rvkjqkTXffvqnHxxY}@ZiNG)q? zJ9sA;BEtXmtCy^96B3ZFWB=Zc!zmJ2^y}(~9@%?j@1cV{DzVF;(h-;aLB*P*IowJJ zJy-?Mj~n@QpMIHb<=1nQuK!Orm6M-z{kypCX1OfTy;kpOwKKhZs!PjjdpwD!mKizcB&$%xat6=`beX$BZ|0vQ_>xG!u(wVm&7erp~ zvN|E8@)nrs=rbKuPb?m*(5vysQEdT9g3!7+dg&S~<83T9uXUPdt6u9~(Pb+CTG#gz zvp#(c&}FQ6VKX^nFJrN6%`MBDmp6o)4n?~#mc&8Bh1SvL;cqux>N;Y8?@jQtVOn+C zyT60T9X@jC(Eh{KVOY;ySR)G0nq4!eI(!SAYUbX^8jtSXzcYOJ*x@6)4ulUJJ$xj* zbJy;@2X}=xtX;cmecLTd)~#Q@c1_!o%^Q}iXd*4$tU_?&u63KF~I+_J)oy zs;0Kmx*MKBXl>oc9>8_)Oh3b(DRwV?9^+>iI{cu|7W3#crJ=(caz0$7)u(!uBuZd@|w-sDVEq;5FEuYC=Rd5aaiGpXamla&io>%a6c22=n z?5u(-+0zQHV5by3jh$5RRQ9-nr?4Icm$L^IJekE5T*eFqPhwpP4zo@Lm$G&Rm$1zW zp2%7iT+Eg!cmj(kcs#2GZm{)R#KHs)EO{mlZsgJrC^M z?+`ny$j7i#3J$W9O8R`(qsa4EOu@OVOTjs;UBTI`Rlxza4A^N;7K;EoaikBrrR z9c6tJ9as@#-W$7o=m|?ugUB(unyIH8x0)E5;Mi|sXo};_CXSdmXyR5Ax0|@##DymA zGI6JgVICFO*_EoU(7J)_kDp*gOz@mK>jO`Fuw6B77?G{+HuY$e0S76b;3f6i+V9`Fbr%Pbb zz7Pu^5m>a(&wB0>ShNq(z&!$s_W4;Ycu)N~iT=94?^Y9^0`}4U{|6yQ5{`Rd8z_A% z*AK?0i8q@#_>7RZo4D1)ohI%vahH|f#D)ShTN#oiwp%AJYGmNH5wKVyz|??E~LUEZXO1Z<|=O z&(A#d*E4T%n8keXXHyt^6@TrYg1y4uN&LMEAJv61=M5s3bn*Y`#JnY5{0skBH01FH z{45WjV;+2-XI|fj$JGb$w}(=rjX3E1p?SV!o+yECD_kUnje4X^I6d`Y6*@>_^Zid&mLRKA$t&~?RhM2 zpeI6QjJ=c(`4}O`ePCb6V<)1`OR2AkbC2Y+=V!7W!`s}7v6*-^kx%9K7#Yo{BwS>K z5*bMEE`UPMm$D~}`~<0XOaYAl^J`g7l!fRD&gw&WPb31x6Yj{GwXew#&l;Kz)BF9yWh z`&udMKgKZ9=(A17R8(G zjT-?zUiroIGg-Xb$IP*V@=KwA^q%e&9j}o-1E@d95vIE%=S^9Yba?+=T zzjK%}n#a66_P0k9%>g2#i}W*2V7g%(dj%$P%6PMeiLA1`UBg6PS>CDPu!g%dT%ln@ z!?hZYX}Dg)4{A7~;T{by)9~XOKB?hT8h+Zuu)Z{o zueoHi0T;(!#K>R~#}6Ju(zE5BW~%=+7h7iO-{hmN?^#ELs)W9XkE%Xbq5c@^=f~@$ zKcAz&SMu3|S*Q;&Hk$8eue-Q?iXmnL@{@g4?kD=wW8^f`^x3kX4;u)xki0AWb~bxu zCTltB_w)Nj`i8yxuHAl=hx{$aEbWW&qnF=uG)K+%%!+JEa8U#K>bhN<#p`$?mXzjpGgMS(nq{I z+gg9({f&38*6=h1KVkU&vj2#8Pf_IY?#UXSq+r-ve@v>s$5=&M z(qnv&X+v|nA{gIjj~o7PDz%^67s8ohf1!=xwiJ053rm>l=N8ed-`yQxC6e5Y@x#Zb zU0x3M72>dQE~D}2;5>#YxPO~$`1vZ&A(DU9-RB@f_z-i;pa>) zcg6$dF9?$J5%)vncg7FpFAO?-6#3=x2Kx~t=PBox$7hdG9E{m=kw0wfhw3j0!V)R@ zo$)X97qjIee_DND0w%X7NguX+1bx6j$@!D?VarF*2Lf&@e~%Fi!V{Y~KP7)1eF*+L z{S*9m?ziBp(;ts|KM}9ccyZb*%5%nJrW#LRysGmFU;}Jyq#K0e{G+%(cwB_=RP=}B zVdSl38 z)&C_UKfe|@(~Jk=-Fa3(V%DF_mOzd}U}L(}Uew=RrL+(6H`HUr-!$IFH2baoLO)-H zKAR&btw)rH``OHZt@iX7Ma@AgJ>qxpt(l#-<1<%}aeY3wm*#v|w`cM00>1X(w6{`e zFY0g1k?Kc#LwqsO>>v2=kNpneuwB0DPvhMcN`J}UGg`FZ>DQvYh0X9ic7K$3f!n9z zX7-xhAEYnMccZaReG^#`Fb-4yB}#utf58Z(Tz_OwCN&q@`l9gw<6w_B(m&!MyM2Vi z&Fq|9pBj(#80F3Eb-8`h@@Ya~wU6{S&aU4>^Q|gIH_UGD6uz|K;HgUcdyFbSqX~zt z-|5Qu>@j9ES4jQoF_ty60jE9G|3bTd>d(r2xC6U=@$MB`eKr1=R9}x#jq&D^($7@t zSNTZ(Qsp1@zu2zdMg6bfi{bY8sca4_?XPI2RWhr;J;to&T18&p%q}|pi+3+)7_oGi ze8U3>`+kyrDA2~=P5CrIv&xTm-^gb0?AG|1ujqsH7mgqSA3x+T>im2W&b%KzMlHs# z)82UZY(+mk#++vMhST02s7EHB%h$ZN;Z1_E_;J$JGCd2nw+6&%$O}ez4jP|JqECR|O{f zQ<_ily(M>ddj&u3`ZBEem6y-?^`@17Ch|`f>uIKbY5Xjc>gV={2E28@yUP@P^%#r% zG0?@TzsFdB@oSY&_II&%e_($G9eIy2As_D9Za?+ETB@JoxrLD9FzwBFrGC`uRenGrhP*H3K$J|vwW(*)|PXB&^X7&+@AOtttXl3pD;2Lr2y9Y zT90vKbHu)1bbsdi`C_s?e!(XMtd_pw-8U)vq50j7C=Um3e~&S@nW_`%>3+@g$7oU7 zE)V1VoXm$CwDKmgxmI~E3;*WyNA%C(8@Cq)j5;9c!_k*W@7Ncvj}W5)YNmf7I!V;V z@L&x7k@UGBf=lLjejyrmiv%PWd?GIunN57=cW@<9&6YEF7 z<0BTv`ZL$xW>bG(%7Z?>>geNQ3VpzTPG9BGMu|6OrN{a@_=%N!wf{kC67?VPFZy3H&casz+vWb>KYIU@ z`YS?J)WF9tA1_M(wElOGXXc74^fwdZOq zzmS^f^uM6k!dCw~<^FHDqW-@SEhBe{BaiblZpivrjQ(l;Cq60q(c`sg`LxT?vRAa; zW?Bf7KWhAyvi}r7O7X9Sp7Bozs$l`{|&kS@A&@*?+@%@D*kKZ|AlCcY2HNtvEHi1fBOBW%FW|E z#z)e4(BjwbN-h4Y0Fn8@$3u@XJ)b|1F#Q|N-$-9LxLnZ>o_BHkAjK!dk7`RFSnmfe zu=Qo>CnoFX>sP3sIto{CNcxfD&18P0(~laz#-SfIK26pS&F>vPU50-2_$Hlx9+dTS z{}t+Ip6N~`{n+s){7I7iNT(k?o~d-=8Q52v@9N`+))%ixKNMg|_GL~w{q)HCS#^c_ znNRKzhom1np8t>NM~@#XmH080eyDp_q@ReYpZawAd0f`dI7L4_hEIwA<4N{s5xGko zLO--#;1U;vKT_5=$lvPmojjfpZzS0xbv}|b9>G6(zCregJltjJ3*!;aU1rg2fiL`rj${Ka!^Z>iQz$ zS#p;+EIyd-i7wIoC;Ow^Z#{m^&!-uo)4yCPeqD_HK2rbqc*+saC&{0P_{3+gzr7~r z-(hM3pTBV&%&{=~zuJC&kEc!-OZomLvVUEzZ+jS?m$zh*a_As=KC3YV;{T%l85&m0 zL;lKgwwWgu`R~W`{pPdKFYP}lD}nvWQr0tGh??Or>F=jBUL<>q{>8iP_d@&(`x`L- z#3nwJ7sMI+Lt2i$?d;z`f0X=aZ?93GNjvSV=Y=WDBYQ#X0ld7I{ZH`rlRYd=+P|gm zZ@~I|y&Ny~7}wR;BfoV&pD=O~w4jhP4(oSl5B3Le+-%}rV;sy|B9S>H_9OHf*MaU1 zru}5JC!F#9I?(S!PxIFC`|}2^|7JI{cbxSe?*H&W(yG4~`w0^Fn*FEoW0YWZ$nZ^7zhFtNgezQ7P|N#~2=v(-Fk_ zt&0I)Cp@~q7eC3u1=zQj>7nQIE(TmuSk5u_PjDOi)E3J}diius=ZYFTy^E^&FeeZ~Oln@89L}IUql86a0!e_Xp3n>*aU^ z{<(eu_0OD7Vm~D3Pr&ATw&0h|_X4;xhu^)%^)UYqCcZP8EfC)ai0@wopab|#;(Km0 zd)MJN=llH*?ltlPuS)#Hc*vId_kOi0F#foxoY_;zjglV z;}!Np<1;<&k@;GN?ErH9la=w>YlJkuj#cDwqsnY7P8eTtJg-&xT3)R3b&8U|*GR|b zYOVYk3ZHw8sl_UPr)lM<<9Cf#{!E?U#WKHjKKB|W3jcNcP0wd2+_lCl_+Ee;j>G2v z^|CzxpUXlO1;l@g@8yM63Z6Q?+Qf*jk-oU3#*u^n;Su?d=UG?Ie|qr54?pb>qUZG( z5Oi4aK^}h|YxCcKh5QGxr2WtRT%yuLmnVVhqz|MNKi^KAYXBD=%) zUZbc;#_9Q=-^_*_zH|OR=wRZP%=ca+oF`-2|4#8p{)qfn{j<*JUhJ2U%hUaPvVT_n zC-{r`g3RwJXpPMOny@OLnZzGeuKUMy{I3nG{GX+jU#$6a)nDrNjQp?c&wGu!u*(10 zTKOfKe^=wHxRD$$sQx`1R{5{TGi4h8Rr^WL|7p5sjbHE|8{Fuy`Cp{@|C;gTg);wZ zi>nk|S5jeO!T)gCR7Vc}S6^HHkJ0!q?H3aMKgj1pHvhx^6TE+y>;J(oW&a2E=bV%I zKE9szt6BQP^JRMAYGa#UHvfa1Z$X>?l_w{D0WN#4pu9 zmj`9OK4FaE?cwjgz&}1=sPR7XNN^9At;mQUsKb^l)r zgCWaxewM)8%W|DR$^KLI|8?am|8;(bHU6voiW}+t|D1A_|8q6|mudW0`I#zyQTbn9 zuJT`xU#4pOSK|*n-*2Yn4r}~^|Knx815e=lF->~_e_mTsF<$0>UARWUbIPhsEcjnO zwaSr$|MRXb|E1@@-2Y4alLY_q5~@6(ulN7)74bi!?6>MM7K7dxZ}xtO;`*?}_a0+n zJx%5;`4dKj^DAQWe=*04ZT=TSy$;`djfoRwd|CYez~MXRe~*K~--yh2#Dj}vd@cAd zmzV7SCu#gw{W0PZjsMf3J=y=)VL3qNx6V&`ek{v%{?wv8^WgFSEG_<@gZZ|cU*~7J z#($MxaUvz*F0EfB;U|nH&i5vp|BW0s+WarU_?G!!S}Nnq;{S&Z-#P#P z#KD+4q9dj3zHs`6`^!sqnX{+Hl}*KXs&AF>6GjW? zdyCEgr5rD{`5z9;eD5_TO_K3t@gEC>PW`@S_PEUdvKE=|@CQp}oSy$>SWuVwH(B8` z@n0@4+5b;h(#MSjT>JbV&i^Wn|MWCm=6?w4mFpj?jc*KpMXrv2-QL8Fn>7BHSE&5g z?Mscue|7yKZp>5q6E|+q_&>Em<-h9R$$qJF-G0r{%Ac#P|5a3|{7<%Ds$8{S#Q&Je z|FC$1V)K8Jh}dlYhpHEZZ9Zc@P+l`%!BcDJnON|@qHc~O|4-q6z0CiZ|30qK%jG}b zPm=Z%^%xDx{%5>TQcp8LhkujnH%RyiV+rT`5}W@G95>kfCw|HPe=;6eIry^p@7DPL z=Q97xm%Mr}N~HHA1>YKEzNY7YxigJnR@V{!#JV*YY!vBcO|Hu8G zuomwiC+%idyOer z0C#YD{*P^DUXA}Jo%SLAUnKLr*Qi$f^%KTe-oCLTuK%Zt|7*so<@>btKRy0m!L`r- z5%E95OPSv*AyW9S#s9khPZ$5I_9kvD*Z6<6@qe;?QTczh@qe;?QTc!M@qdrX|8gz< zpQ6S8p<4WSvK7z6|5wy4Q*hOs1tzxQ|M`ymKZXCxWd1+xr$_485OtaRc|ywm(H;YQ zi#Ytk_ixlwgl6%*$Cy^XMZ!o0(7MKi;qW zlGDCkqbj8O|5}BwPZ*WFeU&!<=W|?PV&vESuNu!|eq5=RKYzY#58}pb&Ht~0Hq3+b zzphT@|7r+j{#U@f%JxC`pL0MES+4rGxS`scxUo{>e|4?u-_`j;uQ6}FDp%*vdOSn+ zMdg1@t?J*^_yqgW=Bx4>mHCZ6U!wR$<$o;!rr{ad= zu;c$}xIs21zRy`TRpx*7ycG(rsb6Md!T;L%3miH4k1t5W2v4>D=W?F^OznT!{rV`~67e{dDm>0P{)N9z5qi%FDYvdq8}A)r}0a9|zwj)$-7jJU8Eu z!@s{|fwCX2#{eJwGCwgtKjrX~?x#Ou7Qy+j01m~upHCPS3Llf7&sA&t`DzfIm>MGe z80vV7chAuF=T%h5e8qTF{5SZo_}|B()z#z=EPIFft1>^w^P+LWBZ~18H)bjQqy0CL z2opW#{jI5&`x`e@`w%x)YkaN4mmxd)A^Reim&|wBKV$zG^ex*LvIjC>A2a4)Jy7o7 zyn2G`knv%X&D-($vntOq*f#|@S5(BF09iG>|+SJfO~jWOXq)4Ylli6c2lk*& zo>K$S04^hf?~v+D1zr})j%KloKO&)UBx_OsU1 zuX6UUL0&t5g^9)f*184D9QjrIx1{~l=*hV zf8g5kP1&Et`Ie{g?W@<6Z_56!RD8R}{%!oneEVqm)~WJsLNec+{aAu;3I^XyZ1c^L zeSFNhhJD<(Doq|OJ)9viAZE78^`OSSdM;yJ6F{YKy$)+ZH=^+{!a z64oajIrvsF;(E#DtWRFf{7u=9RDA9EHcsQ)Ihk)I+J2xKZ9h;tzR`X~XZ?cSCsWri zYbUDyOy`@v9;l8_tRHB6()R-tYtIXHzF~PlX`jA6*vPFL|A()SHfigNRW<7RNYeU^ z#y6eM$$U`xR$Zg6etL0lw8vmw0e^&v1eN#Y!`l8^_5GUk z`)`x^pz`f%pTFbrgmrLxaijM9?P{ODS0wGf(fBr5d;VTr+vx140pG%P3lv;6N8Nu@ zJ+I!8eJ)Ts@ zKgP56d_a93)oYY#{!e|MC4D?uY>ig)efNo)e@pwkTk~(a{}H2;SHk&Fsysit`gk&| zwSSVvx3tf@HUE|zPwM+!w0Lrg7EhMh`&q>Efa-b0&VCj?pDR)JyVTAPJMxc&Z_56b z=lr*F4PGn0VZYmWjc+f?d@IrRmlSLJOVYhxnl7G9_kOAVewkSPu*PTd`(^6$Ebs}0 zk^P$*&w-CxJgM?AZivxh)kn{xRR30Atj3dSywz)@d%sLXTTc12G``K#zF$J+E4}}v z#+&N?nN;tusn6Sr(|o@~mBu$UUdH!PC+#oM{M)oJ@0|U-8}E-g`$fR7`tmvj&#S0a z_Lme_)i`qSZT>Zzzma{De?MrX_e+`V?}q#vd4DdSf1~Xu;rmVTmDJAqB-T$}miZUb z_J0&B`#;j{?@s={kg3Z473x3f6NmYIPNTo?0X~IfK4JgqR2e@OEf0~0vicJ@^!Ld^ z+J12Lee+b~RgPEn^%eU5Bf0+cd@a%ZpZY#!+!&pYj~Nx^ZmEB;KN|nk_*Q-1PU|N# zG=Half5T%&Rk^x{{{-jVb9m}?{At@T;uEy0l&&i>J(fNt~Ie(kE|-IcI4of zcSL@HFIT>wB*pi+Df>wlTrr3{-tn3HT_g_6>czHbO zwf9Swm8s)Re_zVyQ`aNCUOApj{=O7!ud)v47w>axFk`^ja0roYcH4#P*5 zU#G1{s{WJ4qw3G9Fuvvds=Wu_ysBKCFQ(%U;zx{cxqN;7qQtApb$%xMW7VIf`@V#* zS6x3*`AXkUt?eHvzzxS?^X+;VKpTT!Q^%>_msC}(?Jp^*HTRE*^^x*$owA?AKO*1$ zL+g<#_@?Y9IpV*WxBv3}S6aS>HNO2s=39xf-z53_iPZJ}WWJ?~AAQ<>MxAfj%KD}L z{eHdpeO{F>`u7u+g;lv~kHJ@sZ|eIC`uCyG_Xnwbn;KU6l+0&Uo}SOz`iS=ZL^_`f zG`?N0*$c5c!G{c=59;4fmhStBiZ#Be_Ja7<3Bj-{|`aW&3=+!XJA7cXa=jrKG3tAGHte-~1ZiN+Fj0TXCUUzdByPSB-Dk z+ISV#n%Bw4>tu~@;X;*9s{QXZay0*ztCc_1?vD7rLG}AR5lHCAgx4f(IZ_4i_r}%wel%HXjKf?DNW5BSk?c)26_1_1cYk1r-{QME4-*?9E!RHv* z`A_NFk)FoSJ@`G+oi6&lcm8|zUoyPf?+L@4?1PMc4<5fSRA4Yq>_NOc;Y|8{u>!-F5q!dI585|3%H!P_*9X#< z{XP~iuK?xQ{XzPiC~q#lUm6WZdg@;WlKQBA;JK)ev~B8} z>hsY3;h484liBS-dGUKt^?u}~Moi5^d;FPUd^FqRoofW}eb46oL;AVBC?9gG{nQ@H z+SS^{+Y>T8UP^8CKV)S1&Oy#G@)sD{&iw-(6V1qg!D5M*Eq!|xeItLy2UdAhe;n;W zdR{-*X9o2@X7v~Cp)Uv*;}PRyZnuXF8VRTl%a4qgLqjP!E;ZQ~sIqm+Lv-Y<%5@ml%Y zuXE(Yx2-5o&c9vwSJ6J6{|NFQw)5}A_XETKn(~m*hWh0EyKy&dxr^$DNr-H<^LOS- za@5~}_Q?6~$dl!ie;e|*+4=X2@9VVd-+}hZ`48gu+V%7Hw&l^=xs6Sn*w^j9ta5af5-a{K_f zB{$pO1^GQB$9SA;c-=9KPm}+!cNrAENZ99&DLB*3h;a7!x_j&~l-F(7m&@tO?+5fb zrrVEr1TFZ8IDMn~4wU66B^k;5#VipDr4pHCSXl@n=C@}8N z6eExS{;Hkc@L$K(Z>5j>uM_r)=ZAgV9J1}{b{Fzj&#>!*y}s|*hLCMvxx9FyDsK(h z_Lj?Ya#i`NkZpfET_|r^o?Rcu&3ShD97po(@;F|Qr{=G(R`buPR`b_ZtNAO6)%4}X zYWi@on!YecO+PM2O&`or(^pqJ?IV2^7ppiY$Hv>aKau=vKBM{6Nm3ZS&j~&)!}$D= z-_H*@b{knhu>_B2{u%Fk;%-xNT;sooksW1s^ZWU-hzHA$+)d5DH;emAd{=t(ZXu^E z@K@pJ(W7_MXTcwTIJ#wvitoBh#nGsWw{BH&d%KJmMUQTQzhQ~4825)L-;m|uXh(;N z_wQHnfdeYuxl_fvc1f5mif*}U3+g*+*SBq(DnGgv{zq!h)+4ICy`x=`M-Qv=jsvi# zcK&i>zlY41<^L8%ckbFL$-~jMHYvZq$k>&b2zj;{UxuUGx6ATH(XCP1pWw8A#|~LO z*N9Sdi9_%;WE|N=KOb$`hj4WFZaIC(*uS6Rmu!Ql4~C*6H|`(Y*#jg zPe*9H!2X1zot;v8f4PwzpaGRla@dz}bnjkS9**8|hl=;@Q}MxrD*nVLRD9@=itoHr z#a&%0zULklA3G*t^d~zzjP~;T!~1i0w=7>2?cCc*a&x?t8-W1%qg}#ZU_2eV6aG(< z@83^+qG;QW?Z7)tJf?g9{$|P_MEr^N zEsD19K|H`Z?erb{;NMw?*}ex4v;A9G%*}t43cnAuD7yU<@Zao^oqoqX@WgZ%buUM z4s!cdgx{B;`JJ2m-)l32>imM{UmiaH;_{!6eS3$IKACCXaDO&S@s0hk#}I=G!DW;;$_MvX@)!czW0Ek{tPOMS3~^Z5t#x<=;Z_(;+kdDL1ya%JPu0 z8|BIQcdnD=lph_4+-m3VTqVmx#_gz2&c9)!B==JOUC6)N&VR=mNsjxo5$Wap2Y7sD z*{eIF+qR&Ck=yP3TW*o^qyKHda{gO3Npkdm+cq>XvXSy*{33pb+)R)0MsnC=5w8P> zj2+OATwW`d_wD!@{n@q+6^i(rr@t9;wLI`g#NWKWJa(QdkO6BqXqQb~(yM6m1 zm&;qb&5rNU-y)+uLF5$tA^omhiwYw9DL?shW7}Fg9tK9dx=qBZ_w)W28Jl?gxy@d` z*|f=yFOhz(v2By>UtY`i&i%qRE_eLRrgk6dxBLy_jkO})c!%>lWV9bZBqHR{_cCMK zHaUI?83%az`28Ph560J~P2`{4!au=Yw|j{2?WX-j`U8sm$XYU|BA$Uh%8@^a3q1X6 z`9b7AU}MBje&~m{kCe{ioedkHU>p&OzY*_j+^FIM2UUFVkcvOCUd4x6RlNQV6}N6z z@f|b(r24n7lkuYHnvLL-?N7qdb$cW^`n$nR{d4?TARx)npRHR}yl0h)yKa^+{Ku+; z2bB75-YLmZ-mYCL-o0DJx7?!Qjt&*?+^6E`Y8CIJ0gofX??=eku%Qdc8eie)=FPGk z?cExHe3j`B!_ixBmE}eFeIwhy;QoiBx7{Yo!_nJsSMioDDsF32@s1rT?(9_Y-n|lr zK3DICJ+|*(xv|m1K-dj-Jh8R|_D#q!e#w7tw=npEc%$o(g7-xCDEJN@5Ab*e=|hND zsJ~oaJpbxj;ZN9V(ho0x&F!%7Y>gQYEQ+peqxg8Ojc;BD1IxDCxRngFh$o=G2amB$ zs}LVF=g@u|iZ|BrcxjKFe*I1uRCWhp)EDgDxRK(EfE8aHW9_Y0ei2`6+;#==1^<2< zZUA6a1}GkQgUkaC;ETlnRm&WEIur50YGB#kE?e%%$=<95mgRvJk{tGQ%__1NtL=DT z%OY7GGL|E~oPWhiNe+9yW(^t;S!>(#n<1CeuY`JJf3ju`#&cx3?N6E^H`C*O-WhFO zg9?TJ!u=>W)+~|oqd)6`WJi*J~&?x7J zzW|oYZ)}p}Xx}ZjpaGEvTc1sk%jIvrh2oV>cKdHxAmvAYZv~diUw{qK_Wi>B-MkqM zh&0*lUkJHU{$`3_csvJvAwI?Uk;_}OST7G1Mi$!TErwhz4}2Btr5F!I#x~9W-?kYQ zL>8O_Y=aysG~27(xoci2zxK9KUgPp!OE$yb!>Re%YZ+O= zdB~uz_muOmu65)ysr_C!yvR*<{g@9A``j#fDK8)K zaAdZfe+K$*$wmDeA+NUO^uD7i-vs##yZo8xzbd~K@*0wZA9Ibit+v0#`Wxbnt+qeK zeE81j9PjN&Khv(SP4Ty3BQVE~2f$yf7X(!Q@8$kKV6PW=#eVB^@c-Q32fTLu+}{Vd z-17HUJd#2A(O&rfIov;!|L6Wb5crVjZ2S8;n!nE=b1V7#IdfvT;Q0GR=KcxG-)C$M zLO#d#_jB4L41d3D)*{HE63gGuR`KG+DvsQ!;spy-yl|n4o0?SIc$1178dN-gzKUB~ zRNOpQ#Y>i`xV~P+D_5#`=?yAgzFfug=Baq?S{1Ket>W5R6|Y~f;#oJVxUNpcn>ML< z%?uT9+^FJJ)heEStBPA|RDApGDqb{G!th_!5!h$pf4Tph%l)U{_J=cQLLvFx}vE z2W*CoYwF4W*VuUGJlIP%)5f#vVDQ)q8`svt-mztb(O>xEjB3k2pU*^nb*o^H#QFfQ ze|9Sjo`@HOKc7=;r60(o{Br}gKhL}@fBv-P&-wga_ut+M$-cvXX8@b_?);;O?`Kpx za@vnm32ezJ{-0Up$jQD6`;%e&?^)Ajd5C|1w48r-xhzNi01S4d(zLJT#@s2gJY-Bq zd2;?6rpj{44}%@4vh&|KO_qlY@%{SN`c}DdQ&^T$e%!yvbUXjTNsfFi?6dejekcFp zGDpt)AHe;Kl-v0mCd={=fB#8te^aq6r}_~fM5fsJmrRu9A@lpdo%&l!WI5%BgO5zL z^Di%zt^Q)ckiKfB-Erjq+nWhRoj|;`PB_Q~3y>B4LsPhYWhZ#VpU1V?rL8 zWXmU`|EiqgvogDU@q34I`QTe*vYlW2{-G>~eTWp>a@y~&mQV51M7#V_^k0?3oEA0BXKdd-=fCkr71vBt@eMbq zcNmEa2WQE)tdR?e-{*6{`hX~$bZj|*#0}~TKMlm$-a~S1~%;-_umtX9C>L*$oxHd$A6TLcjRQ>eSH4l`0q&* zWO>LCzfbMtpFB>MBfk$0D^g_I&vIkRb+SBUh~Kkz@=q&}<&+-=J2KwRUwOSO4`Dx_ z(?5zoro&u0>jRX367o;5^UoL~%dwvi?XmR39-p0_`)8jI0aPSNa^MjEJ#Vu-Q%>>M7+XFT{a57_Uxn=QbI^ZP4to$8Yv<2J z|5Z8o7|F5adFa0?hy94;+U4h?|Ee5(jpUj6i;StpPKsvv3_}z9WS80h({fJhW0HpDhg$P9x@6Ss_F4QJ?z;{WY6IL z%a3FsewY1wb|LxmOj8bjZtagJe}ek50=69SK{z^5^C$iy$^PSc^d!yS1tzHSDVjen zyiS!@YVktx^{QNt55i+rxgLL%j8Wx!d{CCF%60!=p0CQ&#V@6@KZHKUk5}_g*8KUn zajHC1i#G}iRQYr*9ts9kxgH;d=ml^b5i>r}<-({g(W1AOL$R z`Cnfl?3;v(i(zjh91es35-ur$J&|x(8Tc(Q__iooUarOmUhd!hwm;1n4}ZY8zXu-> z4~&C-7XFpv%mU&=v7LTQ2={{T1tZx(nrna$t{GnVo-Z9_$M% zCoKFo_-LjdaFPF>T4DQdS6ctwdklrq5yRgb*z$LnfO+i9V-FY!6U1=;X5#m1G85F_ zgpZ#0=d+3TkT{M%Se3zi_RK?xW_A)Uz_K9L^MB@2e(y&7{)Ug8f;@=)<$0{-=sW&P zFrSIUasK;O`u8#1f6SX`K8y5z{8vWrOXYaNsEymd^Z9OifAndH`S)}1`Alw3JIQIj z0$gO|psCp3Z`$9G;m)P@yN855%IRh~OM$yu-_EA~ynyvzKpZgp|5D1{JmzKM0=9*t ze!S1k`E2!HZ?A%%FJ!|y-(^XM_$N^!WUKl*rZcjqeoeb``2$$we*D}GO5g0q|fru*DcGL$s@w;pL`#e8kDxqup2Oi@0{l~^3eXU4;Di1iv&0aS1lAu9#KHoJdTFUJU&w}~18NWXwKW;PR#D^{#FI>OwOp3=CzaOOUO#XU^iQNy} z$Z)^f&H6=t(w{Y6{dk`c0kV`oD}V`sgnij#ZCr}}NT!eg4nVau7;rb_kR+q`&|K*tcU!&l+b^l0TZWeY~ z`dYLuD?#Pk>9_m1y|&XsKX?OJq{sd+x?haF>fl?MHw!;-$jc-7Ub>$r1%AzA`lm+J z02ZUYxIg^;T-Xl}qxY}v&I$VzH@qx@)cipD`(dAK|FgY@%{r37^_d`}@LDN55%n$Q z_RsviV#WTyO!++t9}Pa!fAROBVey22nhF0T-X~^eyj~P_bNX;4z}|+U9`XJNUwslw?-*ForZlAc3hJPtC-&f}OU!v#XTX_KRZ)8W42AA7&yw zg$g(*J?tm^0as{OnEX5WV>h?gT>g`_?BT3L!j1dmW3d3k^9#Zr=!avJ*K$l)Q>*`O z!_8{H$;)R9`fWb;fvAt2>tu;_XQpf6*5LQGV!8^|$w;zYEgmr!P1eG5`Mb`8!9;|L5uRca4_+#q{}& z(ei(gK7VYq{BzirbozgAwEX+i=kFOU|CiF|e|)t3|CT<#HU3hLpJLXYuK%Nszt;5m zM;(8$^!Y~}e@~B`AND8So#Okbn5j?L)0U%eEAttOe|L_g5AZ$Soh`mFfb19Imw0!; z43L2Z-|Ar#KNPM|1|5m+VU$#EB|qJW9j*KrGp@G$`q9eoWZhR=eq^-rvAh3j%U?EH z`StAnk;>PqR}awRfcQM&*map1jiXM&(O{JuQ{D1@XF=e_+)O z>szyqc=&o2pC6*Vt;hDe=kWB;@_1ZNPxCi^nCok;$J&+jEtp^M_xBUWJ?xzJeYO8T zz(P46|NF59KGwj;8u(ZPA8X)a4ScMDk4^*hFa%5e=kCWtrGaO#YRn>^dh2uF-}oEpO?@2iP# z(SP@T;>~S;ky!iV|33ERU;gT6n|^#WcVO_hZ+-dKU-KOq9)9nC9o_2r+6N!}?&8;; za&7(8ho3yU^~bTA#ffiz?Wui{$Nuu(wyi&&d-8ifc;+il{d&BWz4Mpv<^SuiytjSo znX}(~>b}qaZ01v6*?01n?|t^yzaHN?eChoAU-R82%K5y={!3nIr>M&m)g(&4fd&oo zc6?P-KSR{|wTt)u<>23Z_#TgA6K^MW57MVgr=LnBHZA|DxSIG?BJtkues*s|&+o*A z#r#wm{-+Q5hqrz}r+c$vZ*Z8n#G_izGO;+8cnjHSA5+-#?=#;T^S`N#|9vUpdnU2? zUx$D5CO>}q-Y0)X|Nr!!x4!Y-Pv86L@94ReE880w;t_ZQ55ygOA9LYRge%(@c%RGf z8-pKilW>EVo@BVP^B|`u631Qh5y&^}#8%^+K;HZOf%0*;Ybfp#-*Ad207ykgLtcrL z#^aIGh=(zHn&U1xO^lu$ndMMfKXkb!%tI^I zw?nQxfAjmM$#J{gE*Zx)>>jJhvsCQXaNMuSH5|{>$iIeQvG(r2cC4C+SP8KS|%e@A}an$;SmX+m+Q}nWUzR~Xypf}eA*&LM^G={7x6K=Nh6h?8Odbv%u&im>Clf=KKE~O`}`3< zeiCBj^0OjYES@z=`7y;lhSC0p?D9avQdlhWPtkV(O^FAT`ckF~MM5kd8m(UCzEtRX z%Z_BTc(ziH(vG}H9*gIVR(=NbcFx>E!fN&Mb0Rq`o-=a!xshBJ&mFCNMQ`MH8zx)~ zC~XvFj9iDfccayz^nv?k`Q8g1j9h-&v88f@e0Rfc#Hex|`4K!g&R6P4N+ZU&){pb> z!3}>!oN!@}72ONV-?p0dP&vFTCnol#w-;)gN3<7l>&|QTA{fC>3I*dnSw3Opwr-|z z$=MJk-b#zpJg9MYM*HrFSmWhRkm+LyE;<)MEm2H+rHRU zwr@nce2v=o;8nJd?7DJ)Et`Ig+ShZH?ThH`vwXreYTx6R(LV0?)R;@Lfy?wZT0Z3( zwXyXw+Nj2yF{OP>Z=V%&UZeJHR@-+uK2B+$UvHlsvzpI$;YO4(YWfhLTau4rquZ7C zdA!%mhsgEIHd^=)w^x>r(!S0sZ=Vux6zcl3VrFUSOVOX0Q@Fj7+_jxyrtTaK+uZ>% z?q8*~1~bmV>mZ2LE#H5V>Q(xs%4ys+(ENPIPLyr$lxL0j8EO7kk>hK~$?-PgEdNRI z2(2MzT5E__y;fQ&9jznU=`7FQxMNNo{keA(jC-F2TQQ3K@eG7%G5ockrRrV4)(rMAs z$f##k8`N8W+-tTL(PFYTu=c@w8#ot8DaW*nYYS7ElyY}woU{?kJ}1i+ZTK%k8y<=` zEv5&5f7-|vy`g*WOQnq*es5#YMoM`+hxeA>`;__palbWo9J!*6?DX1T%4z~J|MxRC z+)i&0sV9#OOf7kD=-#H1yK{MO8I6sU@~n6kX@ka=BbT)i%T2Ehb*>cty=epYmiLzK zeP#-6xO3gq8-_`DN;xLlytmL+YI&wRK>499NiJ(6l6R%rsQA5U19l;Y?p@xwym(4& z1maxoJ?Bj+$MQ-p^%ia_rQDqzCplSZNq%+Opg8>Z!xSv8Wm9hu^tt`1#zt10@8_4rQGZ-A6qH&i?IXa00U!d z`|7k2`8{ewFeP*@%<$@y@j??$}OhQ*h-n-WD1QfM}BqMSoV9Wj zCpq_N^?A*Ec>LQ1)$1xbE&S{N~2y%t)Cr_n-6xuk`Z za!Cu4{OYtYVw`#{wBSw^ky)0L>_RHJq=l4nNehzv>a;Lod@5^Uw9iTNSv;%eaV7CS zWiKWFl1grAiTq0{xv3={yEt-L3oMX6N>{Dph;iv?J(p!Ev>~+59qn@pEr`eqy^N>Qg4-{&G*TRDMqVSvp_Z1g94(c9 zby^rP4jrxMvMfmpalc?hn)s1n!P4zSOww2pJ|vcTF16fZ1ZHwl{;Si%i1DY~)6qUB zX~8Y6MWpJb8^JTh13Bk%q_|PiLavydV|JfXj&&}Yy~k)R#3^6a0)|ogRX=6Fz-7mr zrY59#Gk`cU>oVd^C7m2^N^7Qwwy?Hgt>Q^Bp`#slmRRM3_EO6Oe5Hu5jX2SilONF> zt*!924@WL*@3LY}Wo7!ZV$LfXLzuVH$}z*rZt9p33qxivQp>Z&>L86Frp zd<;5rMGJgy>*y=T>iYX-#hh2B1#`uCq%~4WOYW4>7?fw->r`^L-x+I;{OYuDSuy7o zX#sq9qZct)MJXnf?xo-DAvvvIIlYkNS$y4{o+6}{`<$`nG9qUBYe<5l<;!sKeJL_R+}yRs!F@V{JDCK8f`%_XPPVXH0pzdJ6KSCzw0d z`CL18qi*XPi@*o|l+)Su)g3h_n|Ymlhf&0I1C|k4N*C+>3-}anjqMt{tKI+J3^(sUNpfgMTYp#Euga74r^rbseS%0cLS%0cLS%0cLS%2#NPac2j z_)MWcW&EblpCTvywfgBD#ByQa=sVr{0nf?5ASplR|EuoWVjDZo^KeCPg^eoHies;L zy=(4zou!>!nHOn81#wDI7gDsO)#}01fku=*d_%mBu=tvZ9_m^ zKo2P(7X{*n1lCiWq6WGsQotzspocyfL0YItfxZ|>|9}2}s5wWZCA)oSW|!xj@4wAI z|6I<@kUHn(R?8)NfA@0B)hjRIsfmz=Sl)U%7@_go4(m^04*(zo_OzcEKt}zqvLaq% z!u(pVv|QHgdS8L;A485q{g6a_4{74Ew4UIEWvJ$R&zJp!JWsbRN(hlL(oYV)5L z`zAanJ~ZL$;(-ZY6TdX!FX7L7G<(m9cTM0S}hZ@Fh_*VW%jV@KYjf!WTu{gfECW6Mj;JP58VBnD9B_20Yr1?-THs zy*fTC93~9U$Ti`|#K9?D{tIH?gdY_TO!)KSmw*rJdqBKv;y)sGOxOX>h}GpE7F#BM zKRi=i&;Jia&4l|z+JrwR=1lmE2m?O!2cJ=^*GI4LoA4h9w@LoAZ~{Jz-$TM-!Z4FF z;U1iGAJ|T*_K#CQ9pju)$IzV!-?Yo;>Dfl10$`}H!5?(>od2CTHQgh6Qav`i$07Ef zu!-6;Hn9tzefZd)b;^+hN0V;J?sCdM?Ad}z_ybsA1fYZOgPqpXHh5g0O*pQ?oH)`F z_L17@7C0*(ngnC8u)*Ww8P74^V~k@w^7k3X8AHpF`Z&fTA6lBkX~qG@IL0GC&A7n0 z#Tf6AGJnQ3#(RuijQ1I@GsbB(`tN4!WZYsLVC-j{W_*uvi*bPQHsdYEdyK=3_Zi=3 zEMOqT{I?l98SgL-Fy3W6cZuZZ0H3kJYVT>namKYPgwu@opCMdeJa?6Fjq#p~FvbJz zrJo~=<(Kxa5ytW&-o8#4%ZE4s<2L$_<&p9p!pgrEtU{2l{5$XwR{o{og$>A8{_VjF z5>z}++j`+;wSe{<*~7w+NGeF0ucogvmb${&xtIf3V7Y zjWGG=5`otVlYcIe{xM@toscDjh4X1v9?z*sy> z{2JpL<8{XSj9ZN7o+J5tjQ1FCF?N56`1cuaGgkgNt`T4P*J7;vJ7BE*t6e8~iYK(k zLBhS#{u!~w*dcMha6V7`vl2VRE@P*}1Msq9;=33>VC-g`_7LCC_<(VMam!2mFk|5( zO#VUpU`+l&dmAD?`3HQ)diBJ9kpE3Cd{2Rn4|A5b!`~&`%iBJ9kpE3F85)PO^ zWBVZg2E-O)@~>AoMu<=T^@=UVl%T#sBhSzX$^_C%o_BB|h8)&-e}#JmXVM@cf%Eq82{X*Cyw-c)!E+hK0Vz z_yfoj6U_eA*Ip9;D{o?1>HN~1Bl2o?kMf5P@)z(}18GjY!2YP&`UgaA)77d0hD4c^ zJVSBTVn@+;_`)i$_@DOtEGYEgjm?rqB>ixQQnCM7(&hG`BFXbRFjwdixLajT?*si5 zY@(|2s#Own0ZCW!39wopxY8pA%tioxRnqAV8#*%%^a01AKP_Gz;medRX@aUn;Y%f* z+eiJJ1NwE1?quCY7dx8i&vc2^3n4g?0!+#FnpxZ&+QiSnRVV%TS#L?-lm%Ixxi|#WY7;f@aaDvZObZm)?$||>8xOeZe}cxer0P$RuzjL_ zz~aU?uwam!@(AyDgfseX)t@4Mjm6rhH(39$e-q29KM&+r;q?t|MZWlu^{-eMR{cv9 zkXpbZ_c80=hpl%^SjRZNSoQA#0Zbgp|MiQgby$AYMwGs)$Jc;?v;Ycg>0Q=WB$dC? zQ}w?F%Acga&IV8)<*fQRj$jC&dzn(}DT1wkpfl>j?L)K(wvQiyGTIz#Q~2UHe8x;U z$?;jkNXGFB_0hg5v3}Y_v@|l{y$$xz|DZ)qu>QK%|2oxQ4H_oA4+!hep_^#^me7A- z1Bdhke6ai`A;5nGfy4YEnOGH=^I)1|rUY=XK!NX~`jZa%@4=i1lWPB%KPF4~M4t5n zmi#|vebs(ZABP01{t(0mrqf5t|2oVeFgfLs%8!>yU$vtM=)VH`0Zabxvi_E)kNGP- z)qc+dp_6`??;ooDqW?G~S@nm30CywU>@mlGkM*6PqbFDY`l|k*nacXN z=8ydbW3n5xTBcmp{6U?SK1zR9Jse9_|D;?FkE*|_vz{ONs`|yDhWRSZ6|Cz?lIqW9 zaiRK`IxD@E(S>MeB|A!KY$oU}HM728DdAgKx;s5GqZ+xU`(S zRlPl0SWbqb;E`5L31>bQ61 zmJys~naQD2XeCmp%`cR~-lZ2qE3?MvNOdB#vY3lSs==wd|Vz*^G)5O@g1*9X&O5S^4BtG)pQPJ}c}m&JadNe0F6LRzg+^Af z+#UU!x*NwhWkQpi*^S|1B0gJ+d*?SZV|v`8kxb>*<_y+jn0<{+Zp3}L&Nf3UcW;kM z9q(-9P-iRzI^JYG5$mcq#PJL#;*n)&KTQMLOsAO5RM?NotUFtd7BU@UJ2o>Jn)M_@ z6U{`q)J%rp1MMOij?{AH=mxaOsau}tW-c5lC1PH!O+@b)vEgN-qs_zm8jH=>GNloB zA{MD9kCQD+*{TtnzmtsJIHrCnf9x-v%HD1-l>c$oW+hK3)Z#Px{j0d5w zmDaoDtNPg_+A?Cz&Se=d@c5U!n=Os3=3t!eI8IvSx{XU_xq8OiW&CQ(6)@lRTx_z5 zB70fy$^6TDkM(a#_O;MtB^mG5RwH^J zDP_CeyT@VP=7zmD+}=#)#z>0fvc+X1n@?q% zjgnEXr^>!mc|Em~FR!H*^Obsab)}pySBp1>JUTXkPh^@TEx?oVc)fmq>V{`%DCJFg z+^NOYW}Xg{MyZ}pc~W#fq`On}&y(_}xWHe(d?&wJ$(K5nfjLx^8KYLeTio0%<=c3H zaNXneXHvdYuCYF3}b^&sWG{1xZdzxholi)YXyOQaT;w#|9?y~IV^lYLLgT|Tvr zLZK^Ck`CA4r2kUmJ}zJ|E%sdqeE-zz_8*-7VegOn-s*poETeo*3ee%tkCDf9DSXY+ zP7d4%!&K|LkZ#*QbHDG${eR*3%Yio@ee1C|&ps4B_t4anQVf0jmAsLA?LH+7*W;uP7yXX!pL$Ean-;!)Xs32eH2$>4cPUBcqhBSRZ)bt@CzK0BQ19)} zpvTsaoc6$$UHqrqTemXdmuFTI*ZomCx@6P#mFLFE$@idKHf`5Ot=rIWk6f9Wv$!Ws zt6-RUo%XFs9#^VI6MPGHgh=w;=-MU@Ymz+f$e7c20S;*KQRVV8UG*c?F8&$HrD(dP zmw=9T0!&o>r@xDQ5yINZ&f{I^FLa?VbfJF;bo6H*^2en5=Y=l*{0nS|pgitxVN(6` zYhC1XC?h4!2c&hPAAlXS-(sffzp$4=al_3TNJ`fW*C;>7^l`}*BFc8QnXc;PpSZkg zA6oU(Uvaqv@6WpIWa!sy#|=NM!KCQl1^Oduha2dB#!Y4m>?3`P@vo(p5bwbm=`G&( z{RPwio#|m7=Q!JT`92T!R9i!pM_KO+7P_11=YXzagQS<3ejex?3Q75fM&}|+$_~>p zUv^bee!_IjSEnyR95A7M{06ZKoZLuFLo##0}?olJ+H z4w&d+ZuHjpFEZU4|2WezUz#~kF#Oy}upg1bc?kGmS3 zze#E2zBjK7~oq0}{b+efX^C``mx-Tj*-M z|Ik8jaRv^^50kQ^#$&gIuEt}v|E=WJcwDo{tIoP(p{wy&(4@Pb$`mWc>V|-MdCr(N znw4zfMLAKQSQl5;#o%hbWDGX)>kVuV z>tZkihfig)(kzp+#e7{Hzi{G3MX|D6J%RLEezjh#RyYjB%qqkrly6Wd#9*b`$Pb2Q zm#)_vM)nR#O+u^|@{Q3(W3`xRHu6X%c~CS_bH;dnt(eW5=oHy}A+@|}l=CNWaT)S! zfRCvP%1=*6XT~YCGu2$P1lgZ(MJa2gR5fy`Mm3d_j!}WYJ@S(vl|?Thtep-8+N-LG zO17HIFBVJr+^6?JT*;lF=*H?wJynNApBSt+RvShJ&KqTUUajU}RVN1Jks2}8-ZYdD z7knr7>Gyx7eexFYd$^4-NHsl}tv5V3+M^fu@(vp#%xO2%|A(6jH1hxNjz5L_4o|sE zh8$}&@PC`WsgmLVdKYr(M=p>ubeR`77Ll3#1*1VVWXk zmAB%5d)CE h#RFExV&GI%>8tvC4>GpqjVi7(>F;r15cPM^{~!5^F*N`H literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kcf0367abcf989ba1c42a0731b61881c8.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kcf0367abcf989ba1c42a0731b61881c8.o new file mode 100644 index 0000000000000000000000000000000000000000..143ddefbfb2b05d77df62fd93e0a0f3d3fc5aabd GIT binary patch literal 349872 zcmeFa3w&I~c|JUQ-_>=smSxG7rIjo|l6AE#WAJU|3pNg5V{QpIU3|d?U&Kl_N1zl* zaS}sZkllO*+WZrlqzP+q>ZEBCr)iKAN{N$D=RRl7*?X5=ahccSVe0U(FEaT6R<&oyAmuHn$R7t zU~JFxzViJ)#d*?KUi&j#U%>eva6`4QJns8vb-s1a*HXXCX5+8l;7z@@%9nb}$M(GD z^VI&gkL@4nVS6fa2G%_9D0R!+h6(PV>5BxjO#vcVjN)%C57h{`Wd1Wl?f{Rm^Kl{hWZhzrp$8zfDq*A*NjP3u$ zfw7l9c4Tb-%ikVLLGShhFGH_2m3rWTW49mJeeAyaC&pGjdthwS)@R4|@BZ%COOO9z z?9?5kGdr~jd`?~Yz_H5__kQTrAkOvY*Pbk@(Ma>oeZOA&-Y-;{NmA@cYPb$1Ca9!?<6`{W*T#t}LmtN!J(iOWu5G zNyBq*-Q!8Uxy_q;YYE%)(hW8KU)|s`3sifa$9H{!Yt*Xwb;1J^aU-iGT{xUR)@ zC9dtbUWV&UxGu)^23!~7dJV2~a9xFK1FrAG^+H^i<64I65?rU!l|9F9dGcqEEdnkA zo`-7;*Nbt@!?gw1EL=Z^YnNv{{UP?;4%no_J9E#vZ`AKuKUBYG!{Yiqw|VRLQ2wW< zu|p@&W~3*vL$7DELuns7MEMDSdp3408{f2H+4#nwXME%BxPG=gwP`D^NphE^HtoRm zZgNjgZTcfzhsa%;+Vl}z?1>z|RkB(?FAxE^8DUVJ<}GPEYr zF|a0UV|ImW?m8bk2pB-8N8+^E9 zao{aVUx)XiJ?n;Q_pD!ByJv&9cF%3!sIk+gr;F10Q{Plx)+v3PURXN5@zm1s+k?x- z?|vTc3yV`5$Ci$7dS&VOj^8XDzsI|5yuS=Q{|M=S9rC+z{S~e|aLvN~+9|1xd9YDs zQEF2RxCrNklT#a~;`&*cw$qto#FZy2$4a7ROA`K{`ROULq z@0aPQLLObX=Pzck=Qccg;hx)uF5I(N<`JbME%QgDqr^@}*Ja;W(6RU%3%b0K16^0a zy^`GF16|j{y@uSO16{YmeIvPp2e{3_Yx@rW_{OdNx56IDD;K;LYP^z)R|R;@*t7o8 z8GAMi&De8W*Nn69n&R*(LYy&*GjO1*4DPAq_8;iF5bjEH`wnz9z+Fpjui%BYDRO$( zn)e~yM{sV;^1o%p{Nv0sx?mF&Q8XzeF^_tQOGHzAzMam|Vhtf{Wp z|L55rcBnkUcwZN3^FO!9gZ9YheJ%^`8ni)#yMk@w*1xx>w!Uy{s{8A=q%QZc@#NDt zr#gpjO09d~#?*#C{XnYn7MAM$n!)nCTOH{abhe(BXLRsR8BYRhL6}-;)%Wy53mEB z-W3Pp0S~UrarNSQ8LqwqI;T$WvICJprN30_FOm9-&(NndpDn!+Pr(~{_}h@qtwf%&=O(@ma>`OHu-7oprp4v9{Ts!jOj>@+2 zEKlr>ndtY7`?a_~IKll{q(A9Q{Yy!Ig8NydUwEc|9_deTKSuhK&(tp>{R!?*CHL*G6%==j=XEcVoqpNMa z!yCsqymY?@<0vYB)V4l}{_3?=mtowp4C9xjsbAf;1ml{;skiQFOT8Fc{w9s3_3=-7 ziRZcWuYIYPW+uj8#JKzf%J%d5EOl~C<~H}%pq(VzTgMq(WLd-O2U-wa{w zRDw9BBCctOvlMYp$C#-s)jq)PtoTcIZ8b|ADlhTeIf(YK+MgH?cd|Xy{*_N>cb<5~ z!|z3qhOEhn@j6e|$3B&t7(4NKc5S2}acEI4yYn!@h(zE<`?>ER+*c=De%WV#lIf>p9KGX@Cz=77I`M$G5BR3 z-y!0**lwkXid8(`{!+=%H}z*b{-boiOV9-npZvFfM4h?^5nc)@lD8(HeD-Ikx>9IO3k-*a(hkaP)kOht12x335x0o^Qh~Ii4Ul zaYR`^{2jQ7;~>To~jRTk`9C4sOZMLvG1$AL%H5sE4Gd^Ya`7zYdMxY?I&Ne+EC? zQ~YKTzr4g)8~iDL75@acRjHGy9__?@8mWSu|!FK|nKCw>68WnNx5p+mc(^mfR}uztHyT zVPKtK=g-0KpvLcJlV8O*zz_Enzbf$i6xD6`Q#KrY8E(mM-!I^n{03ivTk_lZpKwcl z9&$^5`$z};ve=*7Hf)1j=Xe6RdQ{`M&g9tkb#TNz#qk1geA1s7t2oWW<~H=a3b*7~ zF$TBf==m?WCC3-ZEjbR7TXL)*x3b~YzE@8G>-+{^1HYphzs)AU!LNcJ?kRp3g5OVI z!^7~WY`E{g;g+mkU&Af=9exvTX}`zbf?L|Jjogyo3DQyg z4zpMHL8jZU?Kj|eTI1Jm^7H%^_~D-7cM13{%11qhKgF*z4Y%Ys$UHW`P7mCY-wH3> z#Lwfo^90?O{07NQWpLypb{VXIuFml=a5Zal#+^G&jun3iIqoTrmx5zEIC^|MY*CNN zEjfAua7!DWAh+cBOb~8q!%lKbjvjIoM|PjhaUbsK9NR+R5Y;&DHaWIE4ms{Aj&s4W zoa_jHwjIN8D?3KumK-~?;3ke}-`ld`mK;yeePu^-t9d6o2ZekXvd;~Kw@oBRep z2RZI3e)GZaQz6vj93C#W;b1P@lHa~OxFx^Ae7GgQ&M4fHpNHI%-#+Ms!B35MPe4}P zr1ANL$*1izkmH`>vjBXaguN;X5T@*xJu$c?pNdIvOFo`LxMiNcNN!o~2FWe?RFIqU zbYQogr)`rnYzADN&^Ug^lQRaZk_J&Z*$ATI2YH$#L)#kmH`>I1e1}1;>5xr|dX54Q`dMrEsf! zoesC;h_z#zqleroUrC2Jvf(xRFCIzO@UcSH&nuv#^E?b(-J$XPQU=e5A;&$%a}L=O z;~Y;p4`0+_a!a0`3b-ZD6XcdWpQ(gf+Om_}lBb8<;3@d6d-X7|&aZO@;_ue@ea+-| z_~Ve{p5oU+b_BnD@Mqhx3T|b`3*c6Eybx|>$BW>W{5<4Vb|f9euZ_KW0y3T7U^U_& z(D*%N^7HJ09QPEz+2Hpn@H-5Dir>B(xFx^CwQx&*`)0x|`R%HMoA_a#QbBIX?=b05 zJ`e1)>q{r(I!8}EI1FnXzhiQ&_+!X%PjS2$9Mj;~27iiUMFZTDV_PHKl4C^^+>&Fs z8E(mOAKjN6+sLhKNPRK%bbkAQt4B0`-!u7j-Um7EDSox!*N(Z~35rk61!lo5`JHHi zTk<8wQ~jBlIt^h}YpA&16 zKJm<-Klk^*U4iHSUR)RBx|SvQGrkV&EAX&^*p*mwJl+!AL+hl8m7dl&f0>hd^TD=5 zpYlz<1M8mGzVzwqQ$PKaD^frE^J`PTT$`Ob?kfSOsdx@OZU5`5La7%o%1XWT=^Jp3 zr2ga6*QH*Y5eB{zIEd?2xCU@tg)5#Be0nvmSXce@)wp5}_0wx`_27C9t}NB{M*1Pd zS$hF-Lb{35$TCmsd&!{<9PmuL_PDQ9#NXy=J>i>Fi|5%_m*=wP`%rFQTXieeb8o@A z?#)==y$Lq))V{gxMyw})AceJGz7~w{ok4TC-Up(#rEI zQx|K(!#m-J!W)RV)B6Y5&rNubxD0Poh!xWMdKk}k-Y(oS#Ca|q>{S_8>8W?2k8*on zVHTgKe$IL7WB1SSUm!1nxIT^R5bSrhd{$|VxztnP7rD%rX)eP>JL8Jye+JKOZ+dQf zKKZQDq-@selFQ$R-Emgof9pK;6`$*}+{N>(%kV$x!iN7B>&lm1+Hbi_J(ch8bE&`D zrQWqJ^{QOt^E<68x4Y!>E40sU+974L-?-HGxr{#qy|?QZR2xYAZt$PG?Rcj)o9>eH z0hcyVHv580{l9Uk_f?mAS?__(j=JR1;Zjf8Y@JJe8eh06zgwW^rkvdgy>s-?qOFdb)1^nK>@GJRLaC`tMTzA6@GGlS{pSf!@1a&R#+sZu-1qp7XQW2VJId$fe## zT^nCj2pSaXN z=~C}SmwGQl@7=c9Z(MSDz@?tb_q{IlA9AVp=g@Q0rnI0f?PySK)eG^1Rx`c=w&o*QU7SqRNkIYp-2|57Y}V^izt5%K2VLsXGwFAG-0>$axy*2>r)+kSOZ_^RdQC3%RQ#%a`xNTY zyKTe8F1elW-1Pe{=d%CmQtv0wbJM>5^gS5w{6l~y3xPhrQVG$^=^gU zyKS>>ms~R2jnBF)pPBwb*uR8&zUMIZp*{3lhVjwg(qG?i>F*BsJ6AqT`7P~ut&6bF zHdj{mJOO(yL0GhhvE0v^N5B4?YsI)j?795Rf2Tq5-243MT-vD8rCza1y*!tCewTW` za+x1)`-T@>hWBHaddFPqJ>ydE>n`=aNBauh^pThhxzfvWsh8Qm>_a<<@80ci_|No1 zx4q`Hd1(6(HDlkyiGAt(9Drl>FRuHlheg@k5O#~<@G_4FWrmFg1*Ur|aE9UI4qOvB z(dM!O7Nua$IPO@V+`4{q4@17PDw2959_Ga^39r}7e7Qc4*XLpWY;-5`@Z~Xz>0a?2 zz&0iDvafTFO6{?L;8P7F)8V~?1ruy83xLs7(roD2*wdTr?d*a}UHi5O%&+Y|8@H~r zSi{zBz3Usc?r7+}W7F0R+Z?=hyTm&hY{%MdTe>@YSwru7tG7WUt$pqG-p!poJ?&ds z+P7?5-!AW7yMAlWw&aza-92}0y8G_U>ur3Ez&*{g=61EWw6E*!>t4T>2OtCL>g-wH zzOH981e>r`pw{*5|*p}SV*=vWhw6nMKGJb3E2d-V#KC8WJo7KK<(-sQ2 zy=7K=bGxm2CWZDUq*J8p_aGTP8#|Nh*Zuw^B)Pt4eR2m<)w)PFl&MoUj?quHRY%uPinyu~Jy!p0dXZJ>4>AI8Kdbd%n=V0cCC4nY{O<2f6;^r66ZCc&EISpw zEmr<|#5h6?Mb3Kt!!YCz!;t@N!@@I*epP!r#-SL>cW!QPY6n`oeO+gJ&z9yUtG##a zZR@u-wYJ~3y)(Hk*|}+RdoTLB_O;tKw|A|_2)2FQdh~zW)~;XIj-hsYa((Z1ObOPt zclEXN`%DkK*(H&6f3B8n$iN(6hd`iM6+HMRAsM zhHI`|c4fPepr&?i-qhJMfr6^D-JQwaO}(48ZRJB5I}8dPgSp?U0qR+MX~t?G_hI!Q z^SFfH$J4=#;}(8DH{&j2U%*S_K9+yR{eQrHFMgq5bqo$1O;V!@%NKNiAmjc$^8P`2 ze>E+#;AjIyA73F6SpC0~-^VIZ#_`9vAB$={r;7OBha^=%HzD=KY6@d`PlUxs^6mR7 zxDRuE=YCuYQZgJxNrm%u-1lJ}Df39_*uY2YIC;ItS+ly4Brlq3KzIaYCF{%{FcBq;@9WyKUR%p1L?1^7UBfiZ{13 z%&v=HPnU+)OW5M=n>MeD_w@DjuHO>hvc0D_zHa@7O!vZbtkRlFr$-I>wIr*y#%9m$;lg zW8gA2YT)VYD+VrQ*o~_BpT-_D@KiQp;1c$*fs5I&fv2!t1}Arm+UU!Nbz%T8_%=ad|y0uk1D~IMkm(|!mH7xH03cCrQsG{Y91(ptQgm6 zxQD^ZFwPO2@k(4VR_eyX*rPb(*ie&r0C+Djop>mH9B21K@bd`bLI*keDd>F;;p04x zvjZEKPCtooM{yr#6V3$A)i?)mj^G@{c^oG@3_Y9)oEWH<+YgzJwH z56VJU0 zZy~HBkbfI`-$9%>6F56?HhmXYoZUDFa1P@f!FdqpD9+K6G&oM7=Cp;AJB?qnJ~9R}L_LqJtfM5&bP_QyX;< zd8*v2k*CVdkgv*p5c#Xz9mrqhjau~$;P9krWW@2P!~?)6 zUW_d#c^-|-I1WoZB5_pWgAx}@+)*U-DZDIC7CTzvgqMZzPNN;5@F~13PZjG%TR@oj z;I~QGL5az~2Y485gW}_X{}G8P{w$B54Wq44;c>ad#E0bZKNsmcK9ACqW=#@D=L?*W zI3aQL6GFdR;<&^E5;sXato%znBJpa84@%r2@ukK|xnCF3J|9h8{DCww$l z$O%(;Ba;M<^7x{R%B3FuaEUH2WK>oic_AB?Sox>&>&T-lA+eAbGAh%KypWxiSox=N z?#QETL}KNi%Dy89ABly$2z`z&FJg&Kfhm9R+chk@R^X%3zN^tDQh28&j-!1fOywcV zGn=ixUtqE~*=rc>Cdm&<{ex&j2~+vV@>H?}+Ecu2}%AAhChD#%czGzJ6G}ppFbqP&3KHkPe0yE^5iTFd zW-mVoeZQlR^bA`KdDOn7k7I=b_((E%6<4S5xV%1(+8ak6T4l?RA-zck10ubv-$Z(O z`u$WCx>x^-<6$chT#B;E{eOq}tzbO8Hox6LQFMr3KFdk6l_I>;K7{9g;;vwtZ9;%# z@5(%uW3iRwzdJETa@O4I!zdN@qJO|%%gZX@ zR}75$XJC|I10x9Ff=~EY^8701`NbZx!a=GW!d~UT!Qg<6^I7Y|K@3jNN_ZX!u_MU; zheN?^1eK;jaWdau@^FybgJXo>`Y;CEhQ7~h>U+IB{p26|!_W9ZJj&81ke|=wvqzo@ z_@zEC505}D^*KJ&CyEOSIWT@CvQHFGg?Bx|{6Ta`X%VS^s z>vW!8U7b^`ML(Se0y(z z(t}Y(HanHi8tweW=p6oy@P5$9mfwd(IetJ6eJUT+X7dB}uesN2l&2w!cBaY$;d!x~ z%3EN1RD}AwCmFbu$|J9@DBnNb?icknjPzIKp}b4X%X>Na79y#1i1Kd6pU-YV`ql}# zkG<|DyqCk&2c**<;^{9*`g|w=syu~Dl3uTk!N1D#`{N{MD1T@_cK6}=N+A|d7|W)G zcfww*K`6YdkEQ*^3w(K$o|6H#!nQxzulx*qQ~l%VBm1EJqx2=<z*%e`FZz*qb!|a8SnSTh+$j{9D1)hG73LGrZo@DCp?DMAckRGH4`SVF? z4^hA9p!TyI`E`-VKf8T;I4GXNiTuiJAD_4@C(Ta4yK0~8_6uJQ0iPpDt-SS-87!ly@(`S19L6*36%z5g7Q2i$;l z&_4LkKL*KCbny0r`b+5@{K_F0>*8GggKexatlHZlZjZ0_urTRk86eDlP{2xy2Y|rm z#A#~(z)xYc@88d7%Ofu{QNI6#*Vjz`XkWurz!YCfPaNU#1NC_r<(<*EpZd2SpnW)# z5AUCUTRxfPt4-Ap%3suvOgZKMsLKBrk-nOBk}T=J!~BE&cz*42nP0HaLl)Y7Dj#GY zv}Z&8*{TEJ`8#Zd@(`RYAGXk4C8Yku`HKSlKbgPaH>C0x<;%!lDzD)4#k1z`JK~d> zf4>v?JA9t`yYuY%i{Q-s9e9uP_o;K_@0b3MlM_&+LEEKGQlOpFh4${(eRAnXvzT zM|^%K+vms5Gk;f}J%16LnZF5T&-W;QY2Eg0`TJGH=e^HgS`Watp0OSv#A-y7+b?$k()<_AYtVZ2NOea4byV|@I4#7pykI*{n2y-phA?^X-b$;I$a{E&VleHc#~?E}`MCg4Z8K>@rSKQaf7-R@A6JN0QmAbomo^#hm%_^H^`Sa@Bl;r+277oU#6zgWgOY&rNi$ zQuzlzns1u((SQmM^U+4g^PTxA=6`!FKc3-{HNvmow5E1`#xf2`)U4- z^|&ytPdvFDGyPFC$OXVy?^?b+5kA2Q4rBf0%=L*g=6jKKGBMQt{KVVbi{(?)CpAAG zP6qj|U(w!D{juvOpFa;-5o5ki^F2T2=a|Lw^}Ekty^)6}?Su7|a6a+x7U{+OJ`3ui zJ?H5Q@%ANXuP+3Hv>GAC4_M!MoB1*x>Hk*qcbm@&;*-%{AU|onlqF>Rl)p%C9DMiz z>~=o?+wwbC{lt6|qcMjc_~0q9jsv26@&5A6^(T5hKsBg)HJ%TkzI`+q!~!=z!t7b3 z&l|vkk`o^E`8H{;kMg&Cdw&=u0Ebxbr1~Ojq+E~55$o&r`b@4<{$ao6ee)STE5drA z=nuDV4CvuKV&(Jlz{@-I(H=SN3%7qDkM(=yc!1|e9{ZXaA5eX%Oj84zSA_Nl`riJC zsGo8@jn=bB9+2`!EI-G7iJNeM9xc`Y_cuv^RUL%F6V;u#b2?0vrg88rbjG zajO;bVL20rXwO=$NQBmd6dtzfuz=*)d)S&8e8a%C!MPe9>Mt|fr~dK`oMmE*e9Hs& z7_u$_7!sA5(N8i{3e&r%8<99J`AmJ?v{A{2t(qXMZ>#p? z5$ggYf61QoBr^S`Xdm`kaf}aid?B*S!9&&zmPk+y#Q}cHx95cajm*i={=I%?`8ZRw zVI91_ob~wy_)&jCImFa>_P+VDev9!e*^jzrjECj(w`o(UyW;r|{>!%yBRA;~?MFuX z{Z)8gM?BvZ%e{`2~J^lDS#6!0D(cPw4;UcvIMy z){n`aICy%GBy*@az@g-s&tGGV>N`L1ya4Op!3zOYc^+{U{vZ;12^Cv$SBaw`8AAMo}oTee>h zCH?+NB?sTVv_1)o$6glxAF_)10|({*A*+ZF*c}|>1(IVtzxagMJ!ynh7yaa>$Fn=tZI;G)7=298CW9Q;xbn@0JCll!;J3-w35zCC7?59l{I`abj@ zBk)a!Q{I*Xb*~z7!UgP0qmbW8G zHGfg%C7Ztk1~0NafAsfu`QZA2NCoN7$>;o0pCZJkd$Fj$ zs85lZB$xGRuO-T3zATS4KGMs7X8A;U$4*T6m+3=&$S8j?UY{L#aYlKJ@(gq2MNaud z`lx&&fjFpq2B>@n{V1Q=R6g;%5V(-aXEv2jfn&uepBj!9p?q=-|7Q6F{u5U3l)u0k z&r>qnb8|j*FU^mP_B~U7*viV{?Y*?muvK23Kx%k;U!(C&Ks+zE<$lpW5Jvq*c}0UM z+duRdA4T~jzBt14{Lzc{jyHgE{4?zJm+9q=>fZ_|^Y#Ua&)hV_dK_jkjBU*G zUStKj9^b#eHf^sr`1$w%<1@ydS;Y#He*VR#^p76o>k+8W{!w`4`sg28t$BW8$HT{X z7W1brN-xsmqxFg$i?IaZXgBBEIs|=St`B*Y)io~x|6LS6{G-0l8--NEem;Jo@f+k9 zSaW&#oXhz(VtzY&E}!2{$NcuOJm99ebI1c>?9UnOtD8F~E^#%@hpXq#p#cWTIgZ2s zY7LjdzmALHU&n>;uj45Ek7)jL;9tjK_}8%?{w1!&dX6=png<-BziqWHcQJ4|yxFF25P(RQ~hXj^u(4%CFhM0lpqkW!-qOE{|Drps)LXf0HABko+&8*A-g* z%K!Dvj-1vTE=G8||7$VeHR4}zGyKo7{jX`!!iHmLjm(^=> z#Qy>C(fu!N(B(Ul%?sWS|4p|4#TZ^W{9@J(;H&#jRO|8`$yv?U!~X}Q|BI~VOYQta z`yaEeLwcnDr^wzdv#y2y4R(ABYxMA9)-_0<9$#H^0`OAGe}o7BEoTDY zvm;s4kbwUsHvc-#zmE9B9wpYLyuGWZ_6|5^Ekb(q_-kqw!hak2hrA$JT~9my(x=cK zf-m^jnfw=^yO#c+BKy_UpoS+F+59hs+~HSZ)jIKWd(NyzfmZD+_^d6l!$bTO9^y|d zu=&*1gRsuWY0nWJ+T&R?e&Fpf+K;%^SaZSUv3UrB=U>cPKg|wb#9yn&Ut%>l{1E@b zWMd=EFVY={{lXp%^(a6he#m21W23%)Nbytq(H%?j_<-wdK0Lk_d@c?CW%-R+GaE@i zJ-U<6KdP)-5Z!=mkuuZDG@BFa03C&J&M!tX+b66r75RLKE#D@*`%o+=B4# z^BTsEeJkA3LeV;R%>Rh&Zo+%X4GD*`V+Hm zn{3CA_`=pLEzrN0(!~9L5&kJOnVu@Ew*I;FA~s9fqt$9@(89<3to|nWw8L*5f}Ffd z|KXln7!t_04AOcD!mFhLg~O-1n#LNIqZ5XLgZ%o|0QP-GrLPs>>i3`7tj<`lCv}zCKD= zrl-|vsHOEKmA)#g<|bVpv%0Fu|2zx+Vc(cl*J6hcTwu+d$=9dmitxk9mRp3q;0gQ5 zkRDNfDLupo^^2EhJAL(C6yHqZe=P5glm znE3kZO?<-*CcgGs6JJLi7>)#$=Wsl^hB{~sUvrI#SFbkl)mNMNvdc`oe7T92E;aG8 zWhP#{*u+bgm^hIzaa)^-7cMmMqD3a2Ki|X)7MS?bOErx8H#4ra_ZV+A0L%6W`B`Pf zD@=I{ur9BtH04dey1a6RDQ^YVo` z#iqOiSeF;0e{=dn)aR+19PM2RurA+Fq{(5wsp!8P`Ho}{Z?7Bd@#-|R|BgInZJJ{G zFGTyS%QvDr>-;7oK1Yu5dys*N7P~($LU@imX5CR>`k#XE969+%25aNVDD)k9%(^|M z`A7N-fF1dcWcL(mzc$+Gi%l}+Jv}HOV!Vy^4Eo(5khnvR7h={H=5>IhW{`KC6ac0kseftM3)_2PnH%Q+K(N; zrvGl}Z+HAhP+=15?fCk#Gxd8Q-=XQF0wvZ-9JB7G(SjH+iS&hgA@8&G{Sj2C#BFx? z_!V43KM=kP^1B^c_kNH@@9`7Q&g~`D3rAnXUrDFYz#>>2Xgs!2npBmpROa_-EH2s*h1h2~= zMKm5l{4pzh4b}JbsF;7mtPs-cj9(WfFI{k{^3TIxu*3|%t!<$u$9fEZqT9V%>`xC{ z3lUxd8aUt|@hvgqTd=^6kJ}@(Xg*X`euS<03t|0xaf^?K&kUdZCm_aw@G-s=>pw@u zcpCV6ijT?h=ECHSx7-M5hu<@wQGEO*4UA{Q)`A6kd<&CVH@#o!b9-msaIM0e@0#n@ zC=7YHX8{8t5#FPD$giufzD$!ZOfFuwSi=R@!ddqC5&ny;aAy<|+)80^9ElJn&Xm4z za{hdt9^^5NFN?6gNb(Pm9DItb>}!!uLOx}RCdd3`R&_H(YW@{Z-f@Q}50_XA z`TEB~xqcH*Mx(mC2+uz#{5CuM+iy4Jp%BTb%@Oez6zKAp)r_~$<@}EG-?BxQ$CI&` zi8pUH@uW#6-nPxeIXNcYy4A$FxhB5zP7~+lnK+p=aelsuw{JIbcejam>@aZ;S{OPK zWFO3j`ua@V+iT*x?>6yWcbWKuA2hLLnYh2-#P{4|;`VkE-+Z%%(f-YBp!FEl-r{|) zdYZ52<L8@PBa#$TGe zqzmIK4NqM^Rbd|fv~?H{Y4XC`3JpAY1I9C&yl4}~D;f@O#DqZNM)14i4vfDA=HW+f z$M{;)_iu^Wcop=sHe>v&$+vC8cvi#NTQNS>aPUspN?;y-GKuk|Cf~jt<3kMxc3}Lc z;oiPp1K)Kw#&4S3`XI(<0`u@g{TNSa@|&IU*-7f(qBE&|h(ky8j|CNksSF4ln?acJ zRNzU|2~!@5{xPSVFy)z$=avwrJfijn<+GOTsmf=Ol{by#g(TaVwv9O+g4HLaOReRd5yAj@Zj74*-V#qCeeXL54_d!O_%W-x98AS}CG1~im6ee_%O$@8V0(Os z@JpehB}x;g)fX_4tDR@^U+W5nn-nS(%M_e1%bMz7vdDlYwDS zd%e_HKfruw3e2y|GuIEA3&|dnY5jnwuL#YLuAjM{(A}VPlfsP z@H5v3nwx2TV4BTmI*O0fhdn4fn*UF=!>d4g_3&u>rJa7Rj|dafZT%U}{GQt@qR-#C zKH9!Sg&lv~na^|mEX>!<@IfFk!w$a&a-B~$ij&fRklOhf0Bd!y%Z+xuMhV$@~!ZEev18v;ba4Uenno` zk#daRCr@tRAVJ|_JXlz0;we*19F3Z|sK~?x1tu;jF>x$r;^JZxPnu-nY12%clVjqk zQ%#(kYvSqCO`Ml!;))6rmzJ7%#tajO!zPZ$O&p1sSUj&mk&*Vz$};7Zl_t*4Hu21v zCieSHTw816K)}SIkcopq4WqmkPA)|L^YQglRDP$>g0MDzDkw6?gRv54JV^SJip}w0 z&NOp8m^;-T5AyKyrso;qmsZ&0XS8n^Z_l9qi}T?)9!Gnj;hGw>2O6%dM12>Shd*;B z>a!-VtwsISa41A$B^5r_OA2XxLt|nwzKIkNrZftiHHk2#NshO22~!$`++RwV<_H1@ z;)H7mqkdxiQ%>m-<0%h~rz%N4lkOpXMOLVmFd7{`9)Z62;u1wD$4mJGe<{XGCp~N@ zHd~P@A2cES^Tm5H{^G{~{$szE`TdeJ-UslC?^WQJ%F)J7w7(1QtN8hTeffUQZtV9W z`}6%`e7{#WyHZ5RzyE>tJDyfCJ}CDz_hLKY6WFg$gBYZz)e8FNa(#?P`2MYcZ@z*3 zzJ!fe`Ds0m@)O@PqV^K)X(T)fEcXW^ys&Sntxx;Y_;xtlK@P(H*sn&#LtvDjBgqKQ zE-vSM!oK%&x!{BMn<#lW;9v5+f@6p8Y6JUy$X*=Ezub;!`w#mzJ95OIP07Kb_~m5q z%gu;C?$& z5nqXwh41k*wvxu%;8S9S`FJ-X_RmC*B%=<0#9w7i&KiZjDlbQpMNWFSJk0m|hiQBa zd?Yy~BYnk2`gZnBo(#)T{p9;)>HUsw!Z!b)ZKu;2GJ2ZPT{r@W#(VLxw>x1VzV?vdn- z41O0m{Gh)lIj_7L;iVtLe4dxbnhd_R2H(BO@H|pb@6S;FKIQO1{R;Y?F>t{59RvG) zM-3eIeb2#2Po2XTeD)@T^N4_o??|%V;J-H+5c_H5`#eXI4Gw?q-#?G{_qMz-BR!#c zy#JSSO3x1*K0G}?HgLdq+`xX{Ne6>}lan7PpF8`)g)9yQst=FS`=iZH`gr-7<)nw> z7KcB_vkkt)??op*&hKRd2Yjau?DxIqU=~W26o?KVH}O8=kz}jG2Yint`F1G?Rr#Kq z!RHc(5BTlu^W(#rP^a`&<{|v4c;QtZuzx5yZIX#6Pu4MSUlP#g{z<=xzYoYm!1JU~ za!RhQA4(SI>G&5If6)45n19bLzzCNB2RZg}OlxTYjzhdW@$sF^Z_=mrPkgnEufI}x zA}Xr9?M?dU@$shA4<$?ad=_~04#=Jjy^9w>ciuPDFBzAAodkBP6sWPc}r zM0%b4i176A=NDRj@aGqHe(3d`%HP@R``POsl{eJCVUs`2rF6bd{zNkJhxi8P#WVDs z{Gt9RFt0+BQ~p$H*sMQPUrMz4Q4Ty+t53ZCIQ8kNd=J@^@=KL}-hYhXhF#u*^QyG? zO#A2a^f>kxd@s=SiSLCPMtiH(KjELspC~WnKQxaT3}rv$&spma@uB)6_>2CV=B~;= z$;tkrJrnU!eGu)J;9INd6JMI!D*quozjez0D%i6?>(8kC#4`Gy$y$9T`%g0ZBi?@< zLfXGTKnF}^(pC!AMu%`=~H;r|0%wN(Q_F5S~Pv)H(SG@ zWJD|PWX~+8yraLN@Th+j{WrBw*;;)OavD&H^oahQ#&d!%wU?oJty+8(9*yS&Us0c{ zGvw6&3;R-k5|}qvlN0|-G#tYFt+sv9KZTt4-DH1Zf6qYTkXGIVe;P0e{-Qsp0Tt(e zQnZ&e091U5KMlwfKYKlernu|!GTsMlr2YJSKRtin0qYfySmOOm^*w{k?}c1s<=5Kp zZ=k#k^#^!-2wi@kIfuXhkT0KSwI=iE1xy@bzMAa`VWZ>ir75-&{`iR`$r`kM)`r5aSTzp`m;5dc`v!M1=_B6|ygt|L)b@L9srQ zn}^{VQXuTn+F!%J&kW4xPX);wZm%@GF9ZAHd7772&(-rQZ~xWEYaBG*!uYw>YRu8Y zgMKw72=$e(58!1{h8bzN?v6}3y44e z{t{1LnA?xb!M~t?(j=Zg&hv84huh?2Irs|x+=d!|ZgZaA2!7QMwaHGeNDt2=JfA7( zk3{V7q0jjqRD91^f0f@0cIxZSzQRe2(nkSS=h1jQV&lgM2Z0X>c~uDZkK{B#Rn)Hs zS;Q*DaQt`zwGj2M9GKq!miQU$-?Op`XZ-6ApOnulAgAr2kb8Ghef7lgJyu(e^kYA3 z*1Y{+S;X~$clKq0pNt>xi@-Fp}UhpOVf5aBQ&exwMM%?20T}3bXcfyC9 z{6nAbZ}m$Ieeph1*rN4i(jV}UJ;K;8Ed6`<^TxTH??RP7@rU_-+xIE^$E!I%P0sa& z9O;SgB6|)v`n$NkEvNRA$}(?1kv_67w;#11_%SfDFSj4z&tkoB2<;)9s(qvOov_;f zQDrqW()Sj4dqL$T87dO(R|5Mh==o!)(8k0s8OY;p1B3i&q#xlyO!bl1SE`SM zMSZ03(0+;daSz`kj2CnNPW|HTk*rU=ebVY9fBr7ZCuIWhN&4jjqy*xi@)4vB5~BQ2 z`Z4|{EckLPzo(Vp&-48~5ytx(Dz69+{Vm_$&C3t+8|8(6FN2pCe9yw0oXzVaexHM{ zf7{>Z;q56l}`BCqCF0ubV^gE%kkb(mMWq z1i~-U_YsQg=ESAEn!b-vST~0%2+21Rj@Hdd=s2ft&T55e|5&(gPKUyz@2{KFEpZn; zA8O(Cqk=y#S_Rz9ahx#n6X|Us+sg5}tzSuUm2&xm zZ|i^hIsBJ7@(JVT``ZM+3F9vo{ylHUzr+cDLjO}8`GoOLbL12HFBEd`+r>Zm9R7&{yK_7A9o_`Zu)|1#=N@Era_j@+>y+IyqCq5oJO8D(#vKjhyV zJeAMv@AJs^smhASwf$wN-=(EdvnW zb3U^h0PXSN4(vw+;Y72o-@v?Ne=3jY-$;%DRw6=j*o%LUhaO0eit!)GK{%0Z%NrUH zp;#Y*KFL8i5ge+WD>HFPiHVDgO+0OyiKk9A@#M)SE-WKZnAq<( zaVTWsU{J$Y--=hnk^9d6%^5Q)6-If&`$^OJ^FR7N2gl{*XrJxj92DiV3lfCryeOI476ZqaA(uzKG2ymv6Xp_~(iFs4Xwx^ApE^EOY-R z-)~9J3@Lp@R)p`zM5iZlRxzzd(la)ye|TReW51_AN^*JzBm4()2-7nds_$5zYeM@d zo;UINdN5=EXUN$viuMlrV#5k0TJ7It2fe7%e80i+NC#h*LKi=dlv~4)F`B{`N zU_kz2{yFAjXb;r)i|qN>ELnb-nve1K;n>^G$C}tqYS4`NSR!LSCb2yqlek*V$0XjU z=3_dh`IwGrKBi-uk4fC+%%7b3SYzgVOpKT5`+dkCHQ&$y&psd1=3A1lIUm#JU+AClJP=*28Ga$`H7!tG@Q@xfH5C$ru{v9Jp=Qdm^F*PPd4HAT#z2jKg9PE z<@;%P-#(w7@2Kagcz%WSClDeI@WFhmk&kcW_fulY#s*}V{rv}eUmxZY^E=o#Y|ReY z^DV>&KC}1_LeP8^;nDj3Yzmta$>&$I@(19Q-y;P7d>ToK`5DsJO8(XS4(*NlKIkkp zUrTEHKk&S!0QGGa{Ns6tvNz3dMfl)H^DQJ0N5YJ@Rk3`@&y8g3-#x3m_JULV)`#~p4ajDWkG={kIEM+ zRCtpnnetf7#JRa9&dD)xON)u~@=P3wm^d6Zadx(ev-IbONM9ge%Kd&5H&F$~q0$qg z7e*ES9hh%r&Zk;h_vbLfjr|{)Ok2?`XbdtZ$$`;rZ2s z^Do+?M)jTMQ_lR$2YvDV7ygWo`w!*Oe2wSPQ?!1<*So}eikOe_`pYL>`2A@5{y_79 zoR6LK@%5BXAu7I}!p9%x{H)4qnQ6DT;B(gJY2c%tC$-S}^T|c$n7{G$=y&)$?Q5*} zjQQI@#{5lUd;TVIwVJ<4yiv{HbWHO%9n<_x$25PFxQo&+o}c|;{?^UQZ;2)5pMRLY zp^&0KHlDwrKB@BZhxwarA7?(q?IX(1ALeh2*QYA=drfYh}t*-8!&EM+k z-r4-EuI`=9-|FhlZ~j(ScYgD?y1GIPmfX+Z>gvq-Tm3oaZ~Xh!=P`e)t2@v6Tb((7 zqxp+IpPbo3^BcZj3iXlZZ#18)r}-G>C-{EyOnd$!<@WP5w0CDdKf~wlGoG*6^F0w> z3wl#EpMn26=Xn~!!~Wu#_Iw8M;rUvfJ^zEe#G1*UN6n=94aOtq`W{J%)s%0}{~Dd= zZ}2aEzd}A=6Z1blpX1*rfjrEA-y#9VIKT(T*%$YW(@G+m` ze8>;>N74MPk=*w9J{~vmj2R}btTgfT=_W2MHF0^liOb4NTvB4<;$jm|n`YvvQ%yX1 zvWW`|O+00aiHnL%Tu@-*Xw<}$CYd-EGjVRNiF0yHTwibEygU;}A|?)pO`M%=;w=7q zaZY{(0;b&WH*qLr;$Tq2zXS8PbIs2(=Xdq>=KSt=^Lbf4RVmar%-8Ci`C5SHYv+1i z#=qAh=4T;3Kg(xd5YNj_;{CvP^Lr;ZATb?l%CQ9;)7tA^!XaU*&iS-!H*>Cmjg& zDgM0^EI&$&_8i~$aQt%|TW!ky1t#V{--%lYkEb{4i#q&~z5-vVgAslXf7I&4pO5aE z4*2Ks@$E%6#{1X&gCgpEIK2NCW|+V8@`Cms?}Npte^7F=Z(NHH?}x?seiixtD&98@ zvtjAKE6;=PfjH>_f4rZ@*Q=cH3Vc;sczE9;%toaDN9p?j{PDNo%l!vqWI*K~^BwuV zm)NgzB$;c(r@gO+@2?o|tMTu#gq+{smDpad8{65~k z^7>qk`sMSHz103C? zJ9vIWj{FzD4@u<>7{8B68}jIR27jN?+dl;k9K3zcXHVRP*J*zSoQ2yP^4TkW>i4Me zeqz@vo=m>R`-@!-^!u&+eM|m(wjkz^z%U-hYXMOb6JO`kNv4C{O?fsllf3x4}TIH)%5f89eoPFfWL38 zhmTo~$Kf-Xe?M37q42YXJ&6y#cSL+9;CerEow@9T2^ z$PYh${etHwkB|E=&1POLJ>q-1BEClGYwst?_yVj;)5m^c!(OtyI^j<(Vc*d7wfBp@ zL*Z-hgUIkhtkO@$T8X>#$sVUfN@E~g*zRrbrYhw=NK6V!jvzEA=2 zF!s9m{ZF{G@4s_H)xJEE41{cq`WEZ=`S^H);)lI+`WeUFYw-K0{Q1KBWn^Oi7mp7MG$Ot{ z?w^OJ`6ul7pKtgt!TZ(rQF#^GL;1dL3*v{pV*Nq!ey_wIp`RzD!}9=t-}0#*zMW3o zdo&N@VV@}9vc3Gfq^vJ2kGCg3?Mn+eF~;}LpXv$A_8dW={2{&kgL>U@`hDPXgeTiK z9^NWG-VRf2guma&c1P0b0PO8&16l0F5XXMlgJXohypJ-24!K_r<)PydKx%}ZUB#C6 z&1V)b0HfGX*XUc`$3r*thtHy)S>7;S-W#G%HofmJsCz9|2HiyL^pluhqLmE!w{bc9c1{~E%-QjWGo`qCJWX8eAx z{5=x>`_(AF`2Bc~UVmsmFfShoz8}ut-0MqMAbj;bMUs;~ey^IppX#-|qP~#*P(J8= zu&5Vq`n`5wC;b?&;(fP*-G1Ib;Qc^eo?3SWk$j9eJY)~7H|+|E_XVlYWB)+Ou3(Vj zQ}0K#?(!mZUEk;B<4s%N%k8W5ciqSQ!Q=CMcu76X_fG9sP_ zYh|Pta$O(KBmKc{m7e=({_FP+C>-H9IBa6?h>3#-P3#?Yu>Cy>N+ZT|hJC8a%=Qua zZ@>RS<>3_GKebfjIKaybl_x6y{CO$#seOh-92^Io_87Ri*PA9eFOSb|^9=L%C8I2T z0-r^v_H3BR>$H zkss^mdyRViZ|A?CP2)r7d#e4?zM}tZbn=JSU!y(yK_gp!pB%2s_Vb&nzmWYc^?$IA zIC%aAjQ*DLkLL&dgWPNRM1M*3h4;4s-ri$BQ9g|)f4bc-`Ul>BQhy@9|H0e;a`vnq5QhXn`U1{=J5Gb z8RT03!_#{Wx1V}H3Gxd?bYgx+^_en+uXlkDzSqTn&kJ|`w4S~j@9z!(K#u-ApM8ej zhlYKy|BTn)k2>Gm;p@fhW!mmPqdjVm!{@5Sl^I;wzFAtF{^3T6_i|}Ai{2raMC&I&c$KD?Z z{fDfqAP=8^pL7*Fg!jwg9S4$u_sdIm`S|x!e6-(%haU=4_#yee(*;(T@7G#M`Hk`f z|5Q?Oz(2-!twUb=UhjZfzbNQ03RC$=M~5)pD5v!D{ah<4JotxwywSTLry#(;pnnSg z9z>+@LH<1?gcm@AkAs&7*e?*Y?T7v*2LE4&RDItE>$O4R%hziW9^S{N2Y`(8MS2Q` zyk6?>(r<|0({AneIr#@Z6kfjykMk)M>k-k9@bOZW<>UF~llf7y%gdk7d8NHBu>5>K znjK&NWIn$L6+RfD^y2#wK7&tx3>A>)D z>nJ~i!2udSsv{wwjm8uFj)q4*Z`M|14_AU?ai8GH)(2dZ<){s;x6vB8j-;1`F`DA#5eZitR1GrIw_x(}dus*`Sm*D5+ zp&ak?=M>L@n9Hka|5BiE4jF*-Hxdp_wdH*OQ&zDZ9>)>*m--xsA=j}#W@2A-4q7iB zKIz}*&71T8v-kGFbsg88;KhAk_W=)hzypX65(FNAM;ZJO!7tF1M1xNxB`AtxDjp|g zXC()sX<1I37``IyOAE1{U`7&c#RO^DB(tfCRx590C*^gy@`t@OJK?%mDw%P4wXW=x zC*_)GSRAj$K4_>V}RyI^4C|0GOD^MJx%Zd>7>;J$@nk=J_2?{EhH;uPfB2Rue> z;vdcd?gm_U;0oYv8a@Dg1h}c;&+@$zPHJK5%txKR~~)HKU)T`@+p7X`d?G{5&GOn!h5-Zr|k8pogdD> zruGY#l5!W_OX<(hr^QEymxHf?9vSbqV*Rlw{w_+B{s(7>}iY4zoXEy&`ny|ELf4>s~ge*n@9ZgDDJ`>H|NRzg`N9!wUWk?C~MMC0-wj zk1+9OeL%s(QA@D%{{a(i;D-|fWB!Wy?f>>jAodzwyNoZhL$3tNzx%ArM?CqWef{|g z?HT-`ZyEW}A1b~>Dx{Q1Ph9fG>&K6Wq3HWM==1YTAEo+2e&Vv09`bWOADj0{;SZs_ zaN)N=D86c&+z$u5xMc^ss4`s34URJfS*(N7kK~LCcsMyzS1TB@*uKy(O-rE z{({0!5^ex~&4I@OL!+dBGoEK`q#OBVK67yt^5ppn13SPItT&{e~@1t^4g>2 zh4s_qR;ZUQ|03yGFZ*MvJj$E;@UTjMxC`ZNA7Y0$_1BV3n{0lmFC3KkC0&0l!B}j{ zSM;oNRL6wXVYc}kNR(2N8ZGUihOXNq*wlqL6)uC6ZBKx zUQ+aySn+nf5Z zb}e}~^-aXnYsLYy4yA91m-FPga~%p)`(GmODNL7KeQDHFdPhl)I=Iu!jQY5%=)2>L>MG_TS_ z--eR9V^2;2-ZbXRV`m@xy4J@$^eJnf`s%<_rmB6oUh200rhdxW@VnApP4-j2Qh9EN zd2sXZT!R8l{gCJ8j6J(`4f|Rhe~RDGJs;Yp?8CE)U-S>~o3VG>2S{7055t~)fxZlo zH`2)RC-9yc=`s3@{@Py$tm%h6o=WYNq@V03eWpL7eLZ`l?A^;+UuaLaqPw{KPYtlH zl)jzbeMGAN2W{#2GPpKvzfst0^&Xt;x`3Wq(udJ8EnnFmRB82<_XDN)iMYR?w6DRR zlJ6SF9a6hzaB15RppDmPdfIm+6MdD&AN>KLSO4Hze7nQwFkn_WH)7R`X}k7 zE*g5F&rokIuPd~tj=jNp5A0<(+sEj~P|}^TA5#*)YT9?DKQU6F{h>^ier@#n|B2(Y zx8Jq;3;#mmjkJ8A|C>A6Us;CMrylZR#uH^95b-Da&-R}1>^bahFKIX9IppKnN4(d7 z`s=!W@?X!`kBMH|Z?-Y(A4Xee-26UAauloOeihk)Iea?Mm^!GMbNnhRO=lx}A{)u?37Y5srPn{3O z+WkW4b0_Pr+59|@?&crHgAw|@J<+%Iz4X6QdG$H={1o)N26?v>z0!Y_d<=iAkA2Uy z7upl%#jS6n)R*dG|A0%cvKL=b<)i=L{aAEGH~&PH?ODg4lIJ+Og27YY>s_qh&;$Db z^J4I{cf+0R+opbH4_?vwL;K{|Z}G=AdiEatRatf&e@dSm`+)nLVUH^;pKYS(8}sZv z>hJRPdC0r(0QRB6zRfmZe&~NE+k#~(c&zHT_i)CZS0dWqR9~I>4a!sY2@$2Xzp*=b z^66sR8Tr6or2dQeUmH8wmyP{_eN^`qaS=r5-CQQ^75pjw*QfSe>aRe5U~O2x==-{i zJ?fVG7ud!s{|LH|OW*n`%QyOk{;J{|rT#;ZS63Hpm7yQ*rtJ!Lp70QTFatn;-WtruIt4`;m-&+R(wWjemjr9vyqc z{?>;J(%t^%+Y4o{5CN|B9rjY%)94TUm5BDj=o{WQOz#iD{mt-~Gxn=oW?y6;%3~;5 z{zsP{MXzc9p`_C3+T#hl-$Q#M{KCJOh|2}`C#U|dG(GL_KnMLf z!yo+JPQSiNk@S;CMSo|;e)W6u!~2ttePn;Clvp;)(EL00iS+dqQK7?sc0Saerz)iS z%Hmyl3`qUtLA4*B7tnYa54tk;E|LXtogVWu*B)`bqN70huues9yLVqG<;k`(`czAJ zzl%#x$bbuZ6HLo1V?Wx9EL*oX=0A@AEcUouVBa(ScPJ@j`hQoEwx$DrC?ENG|BPFI zB=Xel4LqOYcJb~ z`nypk$G>sfpNEicZ}msRLVu_9AE9`4`J=t+4*jdiU*m=T)#R`7LjP*=*Lb17n$aK1 zU*oO(HLTv0QtwWMSaL z_TU8#i~QAlJ3=J%eJqaTz8M8qL&#g-*MnbPIs|+%4M2WL-apdscMyMy@|AcZoxg1? zcJ$>X+KXJy>bKf&Z_i!O^jujfPk+A<`@ad!N}%vn_iwZEM?BhW`KXYBIi>7JLO#n~ ztgixn!uwW>kG3xd{|0SEqXhlV#xF_y(BA>a_#x;w-va5sM#%kv<%yf7}R^DR0( z&L2dWy#%}eiSM7V{8E}9^a77~lf1&Ce-NfRw}o*Sh&{LT z@jY@s*(u@A%J=1J+nFR^BwpezW&by4+DGE^ zv0tB~zDT?j=tX=%*g|^QzX!Y_>5_P%E**dK*NFd=kMDZ`e>RTffhLJJQ2a*neh2v_ zZ1%Sg82--%w<0V01NL7JG2WwVH^G0xemeXY#ODA;0>bcL9zWg=7%fJ4lKay|z#y3L zVa`u_g#WVePyOsxzM^i=p(H2xPAi{Koo^x@{U7lh63;Nt`*6`;s1M>RYW*XD^pw){s{Hy#6Vwx zd4k!KdlX*V55~KGUVdq<8sG5O`n-U8e?{bv07TGxpKXs4;Xub5ARZF|VDd@-dNMBR z`_j;9(0+^$(Rk_Kj0YkA_#)l@%-_24_HTLdw>+=0Md^d^gZnAye(ei_rr4Ky!uh2O zQ{7yc_cOY1u6oCp_B4aADgJ&P^IZqV{TGFz50~&BnN5%T9%`Hb;8XddN7U13;8y(c z?MD&+hWCX`d??=kss8c#p?$$AI$LHIAXbl({q{*lC2M>uaM_J#Qm^F5y)VaSw^ z^c9s~y>AG(UBT-7MTGNOqF>C9@qu_%zPulX`TK?q|CtKI{e9SvyN-v!voaxqegKZl zdsWPj@dNhe7W^MR=o|23N9iuLAa8u|e;9wf_*!ePvG4Ub5&@g`7(3bvc1-#y*#BG_ zzjl%Jb*TQsN#Qu=FGo9TU*7wh|GQ}U>yvnqJJ8k)L>0NBKuMpUCwRSsooO#f*B^*i|yp9;(c=|h)C zyC?9zEt5j2q&-Lv>(}L>eQWnheC=Lo-#o?>JlKc+ydV7;?N>{>@BJ%emitlw+xBLm zl)v_$Z23tS#iQ!KbO`sU$a`*(XG_j2L%)iy{!YkyTsr-W)vzY%gI4uxLQnU^Vpo;2 zls<#{sn06E^atpBr*v4OKarHj_HPvw{ndN_)S-U@@0D5lppTKrOY}wPn>ZSFy6r`J zo&E=TL=;b}-O%@3B=a9|qxa#-1mv3Rh=^*#Bpo)7$x z!i&8@c@mzx&+u1^`K*~2>vrPI-`dZw#%=!dY z^bDx_lzxoukn%5Z$XEG$msy^qM|or8Ql7$NJQ}M>c{cq$cpp&K+Jc|U`LeN4@;mrY z*3$ywQ6A`D!?h^mzXO*0Oy`@h*BIaLLHwuK51qagm$)FVK9#?hZ7unc{#%%D7&Obr z{S5kk)0T|qfYp7HOoVty`ft+q=&Hv5U;!qcw1)dY>VPHw4g4ZM2ONHRrbm5x_{A@1 z561I1@xPKD@rL1oe$P^uKaKZtX+O;QJMi0iFF?7E3HnG_)&Gw&ei3E`^Y{Hs96z6 zdEwXM56)|OmL)w%)byD9S`Az>?}tl$*zdh}4k!~}i1Nfhh`g0QC;d&7aaE<)I2gPrAM&`cN|WONcy?U)k@J-?To(zrv^G3!Ii;O3R0SKp$xD z)Q9;;_WjCieb8s+|E2mQLyZ(5`P)7H#Qinx8U5){`lH@2Mfo(C&==*e5LWu5`V(NK zZ;hYO-lzJ8gA@jq^o<|O`61lDVf0PL2hgka4Q{B&vuyNDMs#C{fqP|eFJ^ce>QCRC*wb<_E+*x^-twzdqCbbluvq1`IM*FUi_x$v-)o6%d}tF z=%?T*AEBr5BbiSCmh}#ySLs8Uw%q9t3UBqp+-Gd+FKK;M{fqaP$XH`- zA6NfY;~)5y{FYPxls;zkNzWHjdnflJ8~)J0)B*+d5$zkrH{;yGg*gsVLyf0nsf3LAWI=_Ek4brRe zEVU;xUOD=r@K#^e2uyWQ`r4C@Z>SID^>I8k?E_dYJaWoYcw1hdz-f8i=7J>YFUa4^ zy%DE8*&lMsQ+QimufS<}xbH&%{0I7g{b!laoB25GpBk>22K~1u?(U`owLFRQS27;r zW**{F{fn4E=#lcGbbUbN?aQ|Z_W{wrmGYE6Dt)57l|HKJBI^))Qi)zM_5|spI2u$2 z+Um2mpEBM;f6KBSEA%1%bS(F|waj_6DDD~1_C@@cD6X0D)ASE*e^q%6_~*PXQqJpL zWdCsZ2R#j(H?;W6)o$vy*s}{yOZg3)_j2(kk5zgqqhwCa7pE~lxBUzHiuK0GNiY3Z z`Adpl%wM`)dEmY8_TXnzC7ItTc~;f0ZSz?2WQN`aFYakDdXUyYR{?mLCob z>iLI^Unh^1%Qd4Pm~XN?y7JZ^0L=9d>2KiwwF z`(v{I2o~`V=R>gnknW!$Kh}$=o=y3O`c6L9i~T;mA1dj4#hxGy=p6;V*lYLq1Yhj6 z1D4=Hj@i!>d{2?{N%;gH_SpR@!G}F;ANjc7aWo90|F~De4LLq=eMsG;>sK>z zxXikj$shd{qfZ+J$N=ew{z@6FGEe;WQe zWdr@~VEtSlCn>T%4u1&eCBy1#`_y_B_8SQ2mo$7ihxPutOl?6A*1LJ17>fH_6rj(8 z`|J7pu>M@cSz>&YC-~cbRJrT2-iY?Wd1G0RFLFH|^zv7^=mGpgH z*#8LgAXx67d9|?OJmj5_w|u%iJb9sh?|c;Gi}fd&fQme1KjWZUZ`b@?^L!T4%Xuu` zcfs>mS9l&PiiSjAmLQ)~etoE4_Pgff{3*_FeFE*bPAd@l>)(#zzNNLI8teMnWPSP! z;zKw-$$kgqvy|l5ayXXtX_WVTGPv%~zXUw@zkvTjQq2GPm%g+~VBpUugUcMh@V=Uy z59fNfc4%e2`!w!zVt-zaPRahx^YAwXW_ifJlnlZD)b*i!=#SWckr&zz@>@%G=W96M z2mfSg5$Rc;oR?SUEu}AVeVz2j!(wmEeI_Um2Rf1eF!Yh-VSR`3C-b}KD{6n~WwqaC z^$qUs2a!Pb3%fwy`aw${=v`dsRehiJ5l&!@nn=$N?Y$J2*3(}y^r1fSM};4x2Q2SF z%6&ti_sO`V-_Tc??%3o6SJfD;WkNfRJ zzNl}R{@8r&M{=I>e4_j}yhkSVL%wRhRD*m#-;;5tq(}e5{Y!FRhQvQDUq<^8kM&4x zpGDr!CsBTm{HFDFk{*$_YEM01`mvP1g!ERP8ZYfrJ6e|ZeGYh*r}qPKUq{68RL*;& z{@G*iO4|$j#nkuwwx&9t4EqgwqGwS7?ne-M7axVXd>a~sL5}+YXYhU!$5+vJ@W=ZG z*+K9=oXF#R(jns+f*1tGeX36XnL_z_iRZ=nc&*R*Jmt};VU#DZ0BK&9D{{>$yac{wtx0KZKEvJ0m4=nnGp_%+(|6Jmoiuyi>C3F9irsuDY!+fzq z_CK^AmPHCJ03t8s|I1@Re?*eI*`95Y!Ne!1uIsRFF0z7-HndW~@)3=%s$+3R5(x1BG`HL0!yWBr9_lwu!!ZQ6swLkKwA>(`M zgvk{4hnSxFo!;M967#40$wOGbi-aE};rues^yEkOKLE>p791h8|6Q^7U0^;4`_zMY zAv=CyJdpXW8n0yjR7&ID@SgB!@IO70nQl+6xzdVZI6aWPFzOFU7yes~h_VkZ%$9kIARwD{65u z)Z-=gv+#c3LYVGfL*L3~zn+;UyTqR_Ey;Z$kZ%rndmly+{4W~cJ)`}9+(!@pzqF*` z%jKYPu8?Cqd?@{GRN=dw`NIFq^yz$o?ZN#o-F`t&r9+Kpe6Qm2Az#0suTSNIj;OBb zgMWYWSiV>uCGn8=Ew~SJBaEwjW@4!;BU2dVXQ86UyV zhNs$P{Z!)3F#pH>nR1_%xI0Pklw5Rk@M*6zY@vOa8T%=eD1>dG}qQf6N}c5Xsy>;QVUJ*B|76{P=oxpNZU; zG=lb}AqEGuuUD!?z5lK33-qOt352hxZ67P9N{0WjPzmZ$guy(4n}n7qG@`f}%SU~+c=HJ7 zS316;(!hNhEx>Z$Hr7wsf5@-AR}A^U9*&7W#!ZaNl;>u3e}f*cYViYdUO<`(_#RGktHs}r`fYhyKkG(6nLgTy^xJIu%?M|( z<+Tw%4g4;P--6-9;*q}`VJL0xw)pL6T#HA1&kigASQzqoE8KS%-UD|^!_c2{+_;_N zFZ_iP*S8RF*ZA`+eX6ev=sl*_y7xRj#+)?_MKZ_4ejHlKhQw>HEeIGuQ$`b z?sWCPt%37#+5DT4zu)HX%=EwguKt7G8LxkU-UC~Zf57JN%Jk2u_YUAImY;o2|73eJ z9MhFoGt*^jEn+qSv+XX3k;S-$j7@KeY8X)eA$sn_!#1s-Y0e?8u`$;IRR{N1)c z3;y=oUHnYEZF-wm|M<9%-!!$!!|&Yb;`@`SZGQb*w>o(6_rUlAPX0zR&EoX^WwdYO z9b%to-#{PY3n$2n*cYMq)?1zOqM3N>&aF;)^>}j9$44@L!bz6+f_i-0Z7v@7vuyRt z-?hud&%~3rPI~3vwcEjiz6T!g@+Z5;UHzGfcipzj$-gbWmGmfkC;ZhPU}M$kuL*y< z*Y8GpqdzmTY9G-D=y!jzYuCHx-S*u!jPcG%zj2xQjeaze+iwSG<%{usE9;xG{^7=r zu0BMMY~}KhwTEcWNn0Q4M|*6v{0dyJlNwXrOzg;;_;|8t{W(d0E680Ac4>aM#Z##~ zfQP>^*^TP8KMXj^H60lG?!M=T{=8G#4@sc!Q=o5h65?^jBdn(w{|e>veL{@?mcKgQ zGsO6guPObdJD!yIvPtdF)#E*TTzoAq-%fd(c%*uKtK7e(^GCHfny!2J!2u^f-*4~s z;qEm)ym_+=<9=(rC-arIF778>F7W-vamwF}@32QbZhFMK%l$rvo|(8?`NP(p^!2&v zd7u60D?9XknOG`}8x0rVpWrzcPv6GI96b7SU!TasmcMYMbBoizuK{xCM?+M9-)H%|2({>OHShb4Vz|CxAT2OI`PkLV-(15_|( z`@`U%tq*v<&tl7${Pnliv-wfL8l8V8-YNQG`fDv27-aba8c+M|$`kP=Q>;%uwVqUv z^&zgW2z_$CUfBo0^|-qi4hX&`^%eTmTk+vL?(pHgdwuwAZ}Z_wrw?~9-oca)d3E0* z{n55>d%J^&{q5b`i`40ORjs;s#LG+kRcgPxx?FsJ(%Wn0EA4yWfP+W~E+ zP6rSDz+?#m?A7+99?#6Uc)X8Paphf)_wRS`D8Hiv4u@0znx>DRxf9N{gYVzx;&ER& z#Iqc9>iXe7Qa!D`or%}%heNCKOMA}D`1-YGjpbkTtN%d1lYS=d;(NAf{VuObiH zL#z*&^hKO!2zR6~+RN3Cs?g8!l>GVrX;z-@d#5YyVb*@KyI?hZUpVD)Ji_`AK8{DDdCwlhUU9s*&*H10XK!KOvij%i{}k|Ud#nCHdxu?D^xquH z&xgNm%cni`?LXzy3A~aw!u(8ph_?Tf_Y1(A_TqT;PFr3T`W%M;Duq|*n`i&U-n#wq z6w>?S70$1@{t@W$#~W`vK>wgU{E9(q0&OF#k&ie?{pZ`-AFl>b+{(XT=Z8iz*jPdh{pq zchP}qzuoeE{Xc~Ief!M*=Z`1(OnbTgLydQ?z4PqjOH~=qY3~gE8T}OhfcDAoEyi+(0=Rj2J3T;^!=)fc<(Q{U*d^Fp7%icL&c>baCN!Fa2V&BWP(f^DicwMD+5Lz~gvL*~yQ10#R80 zB=}m;qOS4-d=~57E&Q(<4_+5Tp{dp;Q%^W)FmcxcA&j!67D;uCcId6dRO zgMY>klioD`JW}>e#|JMZqbuT(ncl?XJe|nvL{9uU(z86ypHco&JTWp#SiiUVbR5Wj zhZApJA7gno-dx|!sN+?ZlE#S3@8ZI`RDUF%Jl~M@cRgQ6`RiD}&O(wsX%2Xa4+o6&;;$L{F&>pH|Hx0g@Mq)wfoFche-aFc zKH7NolGG1<)bZ-&qO`a6pPt0}FVo`_{WuN(-St0EUy19<(q0lz^i&#O{)1DZ&t(-a zChuRF_;QK=X8&~K%VAG&f0V>0VLeOcljjk?OZiAVIMSa_%2$s!NB>(&)>LT^48KcB zUzMghg?pGzD!Cq4b6hJVNx_Pjgj zcHb8|5&cKt@urX1bM#l0e_qLNabZ}t_K`sJZ7~^?)bbz7AMsV!jz@2oP)?iQobS}- zEhRm9R$%1mtyd!64CD2%8*h&I=UXC@pySK=o)`UhR3h{-9;*&e;*TI-#`CuTOME)o z7xC?Ic;W9$d?V+Bd9_}{cyQpSlWhEWHeMauNuXEay(GSTI0HNJ>=#+y>xgHs+wvuz zT&+)%KNSzo{Q`W#ugI^S%1`2pw`us>5-*<0OUIXI%j=WkP#*Ndldn#%^ee5O>(lsj z{dxcV1L6b4KIr?qyz=1xJsEe3eKGm7{@nH8FFz*c%ZWz^?8UDG_Ttw8d-3ZyUFOBF zmP|`1X8X8sAR*_2S#}wKTq6%I{9&+XZ&w+XZ&w+cWXsp!a6u z+u<*iC=b(r8UNS>Z0r-xJHV)=_$BODxv;hW)EC$j*xNoA4|}}E!qDg0V_!?--LXG{ z{1Y6X&HYl)|8x+Asz27ejl!=JmKw4XhvW{P_cHZ`e1iC+Yh*6nqcjCxza&pUeF+ zm*P>!f8%&DI?DO68P86~eHnYdCbjoJ5_{j5+WQgk-xvJ{o3ql z)$#Syaj~KN6VUr(`unf2Fnk<;DDM$zRxLtI^N@rFT{)d59|9_0JA!M zUlaXlbw35?BY!XB!H7E^%zk}RYB%+Z|I7KDelPPF;C0A&#onnufys?~p6a$f zd!7pXolow#gY#h{?@RICv^{<@8Z7-t24kiF7(Za|w(emSdVI>tbM}ic&hjaF3B7qu zFZ`EFxF1^K4ZR#}^e6Pvdb{*Sd(!a}^d#k!ex}EHCI+zJLmUVB>5%&uQhuECbw8O} zm7j^I8K>YIOFdYSf2_eb<6 zb)H7|KYgEq?SE;1q&+hCv4sm&&)@RSKcar@-`~p?mihem#lFgY)n>kd_!4?*K{xFm z@F&V0I4_^}*ToJT7k(KQmzy}~54ET!?~CgAhy0UCH64FZ-%zC9o0j-)#JA)8uGv2m z|D-n9=jA^WSNjn!sm5pIpAOgP_zloQc%jzA`HA@NfEU-{dJ97j^m%A7C-VnWAN*~# zAFBQB=~{26@~5P{d~MRNAO3jN`s1X3Qk{R*@#)<|dkSv(!Sr~KTV7DxivC_h|6sq+FY%o}ICS&$kJkEwrb9pB z{q>X|!Uvl^eD|D&bHRy?&`-ynE#86i>4yHpTZ6`=l0W97koO_LivAu$|72b0A0*8A z;f`wBo@k$}ysrm;d8$^o{wDGZyy);lyzws${t1Vl+t41yo?$(&u}8^Y-iN3D>=_mP zsT%tC)MY)RpYZF^pM9X`LZIb|^Oc0ZVd&NH#ioycQPYe0f@jZQFN}ZMcZ26Iz4`7D zu3x?q{ol|x<^Ld-G2!Y4Kb%ZMfAs1DO#cTXf%1QF-hK9)erd1#VsB2z z+mt=v{2A%J^U^49_T;3OAMp*T{e!)`6z|N8|L*t#d$eMF*`A*N7Jp^uPTybIr3-*n zyUO^!jVW$;dEbABE7v$U9(-|GH~!_QU5 zcak9d6jrq7O8*P|nErh^%g>=3f?QSo_^(cKl^oBPkzQ<0Y9dHk8?fQ z^(X(&8{-E>s`=a-<)_-Vntz<~Q(nF_{$J4X`bO_dz<8amSIBrBME@7nOP>dRPS9k2WB`U3j*^~dYY zH#uH!Oy}n^UT@y)kJnqA@eTHOmGyl@%_+ndGDRmN+}&p(Ip+L^D*cHh^CuV4TCx*mO#-Q!Igue0;(dUm`{?<2dR@p|GW z$Lq1o{CZ--AFmtN9k1(e>Ue$K`E@-rUZ?cy@!G~az25Qqzwi9I9@M?@?G5mAmGRp0 z^Ja|KH_$%*c>M+Jk`Fgsq~*0fM z>Ue$K`P*P-yiVyi^J^V(@Jlkk9?Xo_Z-AexjMsKP`euyRH_$%*c>M1_>w(OA zxLzMi_pkeJZav&QAA<2!#xH3MHRKAM9j|-({dl|HUVpsCe%W=#YedlgH`>3Rf`7qi zXD{9k@$+UrTIK$AO23(3ztQp9&fnhb{p+jnQ-8DgxypEL`FS(OYj=Kq6@Kh^{YKm8 zD&w`~=gkW?>%FGJ>8#1yt&KI;M!jD^ZNIvN7ttN z)1ZG8@cw$rAK`-zKZNc6H1>DhcrwyInV0rZ@ndL@Nt}lP{ptR6Pmf3cHp{xstMUHR`( z@q8+t>~+w;Z?r$S5Tx=Z{0&F`gfBYsC;Z}^N`H-bvi=*4CwnsKPvcL~-_FGSRk?49 z=RIKmXUl`Z%RHdP`^zZ6IBU~w|v7Px%K z!Z;uCG|uzx04SJ3> z0+xsPB{^?_`wAauV*FI^iJS`gK0I#5SMhg{H~5K`r@_yD*aJBaTT%S$_@{f4Li7so zL7RF%`L5$_!7CJa8{(sozZ>UM7@rfpiu<1-{~nwdllJ&7-d|Ml(1llkm-h=^vzVI*xcG@BJ&Z590^BS%E%3f%~@dKV4p~N<8zo z5pS6kBeJK@KjS_L=R64V%X=2gPkW5^*pO6;OyAD_e`T5PVe9t-ArH`t_p>UL|L*nO zl>qNqC^*z`LBn|smo!||a9P8VhI1Nj*RUzCqG3~Bmx6KrX}0`Q!1FEUKC)}abD=%G z_Z6za{*%J2YA@*L?8#@yJU%7wmF4;Ap&#;ojedVZLcmz+1eR1NvQS*Cpe zoAwd;0ygbqVBJ2jU$ZCg(-mAz9)A6hN1hUr{;&1%0+Uc4pm+9UgAhJd-ZlBZGK%}4 zmkV{IUcS&$@$z_Yr3u*bPxqc5%SZd&6#wK8_EUW~#-F?=VA>1uX-f{ z@$r{Nsn4nWalUb-J~1~w_E*pk`N@3*>G`i_uuBpsJnZv%0rmZOiwhm;`Ek*=4I6&W z?m+ticrh6Ww7>$|6P{nCeyI3Go_EQ21)9#%3r z{Ao9RPh3&=9}*k+Mmd*uEd-lY+ zkm_RiH}BONe%_MQKIP_T`LvhwwU3Ma9m4q^%2&?IhcgAL9H#+W$G;KiN%lq1!7D{y)e2AoY8|!Sr{42W;9C=dH?ffKvKPeR3a= zroVyv3tW6n?)Sxg`@CNi?G1W6Ts-Sz`H;8V_YL?o`?IMZ_m$}5<7#~e=egQ5`9JUI z1IPEXoGz>TQCMGZ#Z5n*oTxhdg8#?8`wfu)^NOCMX8b*?;GY@z3<)yh8|W#zE^CRlD+Yz?=51B^x&US5~0M6WmvubMc_> zjH1uV>$HL~(4f7^zl@Kz{hv5N{?9u6tMoq=`A&Zo{GonBKi*piGxeR~dV^_S(l6y- z_>|VKPd?7_KX2_53@ScT9_aleOc#+~w-5Mp$IC|2`ULZTE~Rft(f5y{pVP_H=irIp zQ~F$cINQE|@F?>?W9t6~>SuDD9`&D65~=QD~P)t=MIhvmIihyG8~o*Vt2 zPCg{}=@|U<8I%vafoX54U-JHu^#6}1l+S0ieBeKceOhDff5yn~JE))ewfv^P zf%;rHiOW7ee8?00n)ZqFCqzH?8vFB+53~FwQ~pLUzjIs^Z((A7GUw` zao%(~nUni-tUQtrrtiaXJ{bp3(_fjt?`LYh@Eq`fZTd6F@51pP=Ks)vYjNpOM_zc( zY2ncp@TUG+oW~8$pijvY_~VI-XZ?!5qk6uQ!~0)W-}2LofRW$OgZKK5IpsyTub|}0 zb2@onW6$k=%@GPTf_RV&re3aDBs6_;31D5w3m<1%0H{mzw<#i|8)FR`6-fX z<@H$fm;={V$b0ee7Vt)1)A792pXUES{sV6Qxcs=oKlJO7*va2li`!!#|9c;D@OiY? zhkU%RA84=l`S|wsZq#qu1N~*r$NTM7i^H6QM|=A1Gadg<;ddN4=BF zJ4gJdEdIg#gC73LC)vIS6~22qzAL=T!=E_e;)jxxGJig)??-GTk3a6<;h#St_$Ms> z6Q6MKct7uxg8!t&Kl-SHNB)-JTNaKgAN|~e_!ywZ}AU3Wn&;^yaj3GDB(K7jJiBt={ph5pR-2;|3m_Hv)2 zx!-8^WF1(3SpUKLewH5NU&(rb)GyZyAw9fgtXHFawfnU78GwUgfkHS5(>tj4G>+p~C+mG&- z;Z&SLz=foV-1zu|;BWWvVJy#KQ2v5%B!7%YW`jWHPgu{`eso8g9_`2b*X(^qg5Q+l z0VDq(=UJT|Z}2{;l8;^wT8fKZg4g%$-GlN7V|`!zne9hM((*tr?l&!svON8M{Pv@` zuYoYwAwPv=FG0&6>1#p(#NPwG+mEX0S(<*2mp+VlgI0W+|3b3M!xxfm9*p)Z1~2RO z#Qwkr&9ChD;Xa5kmgl6L^0s*S3(2Ghhw*r({5t7falaVwPn-AS4gf}fn?3d? z!D%T#-}i<2vyzvr@8f+Oe?AI%o>h3cA14C*c?HY-mGI{jEb9e?pHc8XQeO#wR>42Z z(Vo>Jf4&O*X@yt&Io^C9>%~kj{)wKyBmGh~{W2%e7vMkW^Z`U;);o~?R5tx%NbaU@ zK^{)}&t%j8RVIDYPya{R^naI0U-#2LolXA_ne@~+hn`Po)Bop8da>WBd_I*;|4*6p zVqcgZ?eX~e{2%W|eZhYnsRS={1&@C*0*o0p;a35Fy#lyT!IMb;&2GTB0haiO0sp%` z!059E{!fE|cPRKU+V4zQm__mVPoTb+4iT33bY_17_)4|A_FS$^d_??}Q&H`!rB1>X z!k4RMj&A_l=KEd~dxrF~o@?`eRlU!m;h^@7jvnUkR{29*W>0an&3DgB{v5_X-NR(O zMS6L^%H*$nO~O-bdaj3vK3zB=>*G1BCy*fdV9#+GI)_)AK8NKNZ87A2<8(dcDOtbI z>Gz0D{&f8a>n*rnlW+^jJ^5~;HVDD>}dxr=Qio5}Trz-pTR-Tu8d)ERlF!oz2-xbU6@K;@~11}%=C%sJo z%Uu7WJYEgB{HW6ZTk6L)&~qfE=NrB2S^qHWr#^h6x0m+`OMTeCf;`jw7{6G49r)!7 zkRR*&YVR2H*Cqcc!oA~e{;#df&-Pfppw_b<|K={x^ZSaP=WyTFbG;i`-lUXQZ3Eo9 z$t~}NmE}o)qdtJX$G^S@6}-qP#^_ z-XPnv3Ha$$e&6Zc&hodZ^3i_1z36SJ{J!g#5Bn$Ut@G~`dklG3r+Ho86?~(kJe@DmdZvL!9y=VTNdcE_x(%YDSuc9CF zE|q57{4eJc)u z!9S(G!M@Nu%6cN>D^695UwcpQ(>eox1NtQVp@Gybae zu3uL$?#GVw2wq`&#E+CqQa+-HFkbUMY>k)umArTv;4$7a|4oedCnAYgS^lKNo6H{n zli(?bAI59EMwt7*uY&)fZHgMrKjSwb9`qmDXYS|Z`2J>K_OHT)yb5<>d<*c6BrE{k zX5K?bJPFnZ+5c32tRLjb-$~A2VPD{n0&G+9Y>)@{o7i9VeRIGg{01)7Q%n#0$n`ZY zus*5t^L@Awx9iD!mx%uWjQ0?q4bi%wXSMxq6wXiYcdy`oqtCzLw=CVnE49|vDMn2`3M{XYNw!he6J{TDL)Ke{UaW#PZy^1m`(`MUgnaCQENgr4QLRr>$n zb@-R||9Hy(P=*|_&=WEKUtN3v45qtmjCP9Kl)b~xuCx)zR9zPjF&2} zE|2wA9xLp@@O9)N`tY$-9(d5i(+9YNt~^FW9(Zp<##fFXvnrG)UP8C(T}64JjOEefP3r8V_f7ho{`5R8F|FlkjMB<$^+Hkv^+Lid92Wn z$?M2N^y4F`JT_+J0r%XM$0m`-ri?u1GxGTGHRQ4BCgst39r|&rmB$MG*m50th<^Nj zDvw(;^1$@MmB&_*$JUHIelH`B4_!kZTW?Yx&DSB1?N%Nu^kds~t!X@o*{+i3S({Tltrm-nsJF zE%Ml%k;exz@_6VP^4NWo^5}jY^0>{)V}*X~xsE(UKi;3pmfB&jf5=vwnj8$=?uA$B%y@DYkRPra!>`Am)=x z$*&E61=~lmp8A{uaoCkLoTOJ%0pY`Cu;sp<0TYSZXn? z>U%!POS~ucuY|s{$xN~C;Lj#|i%k!{y*THE3vQz$2GLrur53c9G>%n9B?>X=p zz9$nbFBg8Q>OYf=z&}o5z9;vJlYS{#pTFY4h;KQ^R>Z@ZyjgR#4J=I^yM?QWG(+7<&0RQ%U#mA3wyi)l~(7!#! zmV;kP?#c6flT+V%+H;k@jP$#T&pP;}&FH}xM1s2|*KFzp3>yEnz3BEHeI z_yWrBIgsMH|G)lM9JmeXC%B%h$}2z~w-o0bdih@17LT6p{Z4+?H;&yFCx2hD<>Uu` zjO*QL`L*J+4*pD1#{9(aS7mwiJzn}s@q(8=+K@>ZRJ@!_6HAMU=zhpTD- zFR=V_+CK%3-2Mo>_-}^3|MW=kYm7g@AJ)THuIG8<4aN_DediF?tK9gRzry>nxe(2d z59_I1|HuBqdjaWq;ymt;&k362ldOLX2j7E3Vb1rhc0QHs_p;tE`|IxtI3dA!C+qF6 z>-|~2*K6ZpE=I?syd{*k^8MK?|Fb7|@%bIrFXy>%Uqj(#K*k>6zW6RImpU+>Q~e6g zN9cGHzPAYl3$Xqw`w!Xidizlo9{vpTn99+;g>*0KFlKIVho#dY|*7K!2!~;dk zkAWT??-DL_QPMX5nItN^=c5oGQ}SVkJIHws8IS$<$9SG3@4ctH{iq7}RrE4`NyTRy zdPBU=DElw^KK(HE<2Ay#hT=X0IUl9H0dt>A$mg@w_kJkTfTyDoleR>S0X+@=aZc?*M5I@r@X(rlllbza1re% z`bYbK{jt+=QJ8n$-(`a^-bwqHx663~kstD7zfRtxv-@>Pa0l(fynPQA>6xGOq5P$I z-%i?p=l$J%#LM|~;8ETk9DmaHcVAJ*!R-6H&7Cg4iwoP8evy9I8^pI^KZqaXhj@CF zw_VED?*#(BeJ8LueFmd(_Xk*rdHP^1*)Q?b09gd%%bX+ZD8E&k$xf-o^b+ z{oqSW_B)>keS8miMz_!N=s$$-AgtO42iMsiieJPxNqf%oJzn6SjJFCN`lQ7t5SHTnan|5Dhi`;RIQ`cvA9;v}99^sxP=058{7dryaziugBrJz*N>-xarN ze7F=w%H2`@8U1Yox<~52T?)&Lcfr>s^q!7uHICn={7YfGa*s3~^GUf6!Su&VVF&k9 z@tOQ89>nB_|6b$xYtm!>=WxmiuB%`lV!JuaCc_MuJlOTY6piY*JU} z(RF!elM!{_gN27B{vplZsm?1|{9rp9z~Ij&{p~LOSPy6yd#=-CJ_G(!c`hcMUi%yW zqoVzfCEO?B(#!iN-1fqHM=16}+gI2_&;MY22JJCE?ax$ETv|WkNz8jY98dFSP@n6s zoQ=DS-N);6?$BiM_@HkkoaffPN8q!`+U`0~CVwe?uP5MrLGYh9kU!-= z_XIV(KQ8?8ycF~w_=d92^#9ukPmms!K1aCf&c}XmENGAR)LP@hFI<3Q!C-yQCcyA- zF~0`fKeA`ih5Ls0++tz)?^|U2)cI$J{xZOQAoy@!4B}}p{_mPM_5<<3=x_Tto}};F z7Ppgs!!OjK5Ri@>A;X zoP2lCAgujoo=+70SM~w%De63-iGM+P>3g|ouP}DXgS=~ae^m18_!o@VVjq?(sy#4X zX#0fzi}V;TwEY2mag^~T>b@4{$NYlv;a2~V-WxxFm-uTgpxN=`d(j7lpBtS&40`;7 zz8=Z0?aTI~oy$%5r^cVY6c42IUY-iVqy04ZN`*lmj%W%YI}n9w1K*V4;Q-q`2pSo)%-~RXZ^JQY5Bzgu7{-X*nr#Lmy$aMT>Eo2i5eH2 z^k`NM_IO#K~R{b!S6qlFG)(k~^`1H(>v;lg1(p2AMQEI=%f!9hST~WFRULA z9Q5*!r1c3sXg?=E)>jAa@zV3+GkjX#;ol6r-N7#5S-q~D#dQ9r}zAzMH6>2%!b z&4;}5K|lQ8)IUdh_?H#wf3p7EgY`4NK8)X;cD#`K+SB?#56{a%8|;2qn19IgH<(}g z8^?3-Hyu87L<-RJGpvVu`I+7wj~mgEbiN7tL2pUMN4=kPCh7L_4@K{n`y-BUJ}K?v z$ma^?|89GM{^{_@5&8=O>B;t2;OT#-@@d0(T#4#PZtSJ!A6}V)ezmLZ2k2~&Q z@nR1kKa_`y@%X{t|3MD^Rk1|>#O!BczOMWGXQ+R2A0PXh_}9|@5%h;2%NKr>xzP1_ z^PRu1=1<4{CGn^9{MehHu)MfiZO=&lDcJMDe%s!HujmC!)CYJexxL?QpG$Ek;};Y? zz+*qyn{Q%15;gYrw~9fl%J%~R*Shx(67FC;IQpZuukS$lynn3LZ}mm$?`2PP%J25( zucf4x&c`r+;rxmP%6$}Kzn%3Sv0e6P`_lE6QS`sQ8d3(!Tu;LOJLW%f|BB3izTd|AKx$u)@O)v|E>9|sx0>Nf|1M)g-=?A^I?>DutkpG#4 z-8;1xOG$s-)wiW&SKZaGtUVrzLb1Q*J~Z0*0aj@3`#`$>g8E>O!)w^%JhT=4N#94s z{_5HXft~rElozJ<8T@(U1L8AiKcxff^~iMn4O)@4pFiUH1wTFJFBK~fW$!vudpG9W zd(;p6hw4&!uC%`;vF~E9K!4W$diGV;N2*?Vq{nR!Z+!;kdG;3c(7vViG6(v4((=HM zXKw*xJxzv#)Smg{CB`?j4?oBc=Xq(5Q+dFCgyq!!p?>t=_Bv9h_PLwxG(O}9-m{NC zsOCL;2!2q05BW9gFC}wd5q|=n_N=zIuKNG?3u15R@LkQ`U_9XT0Uzq8y^cE=52#>_ zzghbO|B3N&3NPbHg~!1(Oncq=RtpQij=lZSA#Gp9-fH{Wo33Xl`&xJH>)FJ&r&vEo z?J@M(vzM^fH)bDqy@~emLi8SCnEvn-`0stt@{jiT#j}qi*RzjvmVUI~_3h)^N31yhi(!?b^kJ@nJxfIUp}Lq03*;gO6#nze^VYEppqM;j5wdkEa@Z@@oI?U%^^diJna z{Lh-+&$zxltktalsqA6OkN9^T8F@SV7pvRDbUp!kSFwjXT>X1CWP#Jr#S%AzpL59w_AJU+q+f$JNSq9I_VvM5c&yw2>)-3 z#e4oA?5pSh!C&ff7V--d6-zD?Ttm3`Z2;nnRMw>R<8 zp3%RlY)WCVZ<{TARr_|2wMRE*-|lwOuV&viS^OKfZx2VmE(}=vcEIxU3u)ihUeCVm zxAfnTeLJ|;w{QE``u^MG+JCWcl<2?MH|~#PK6^d?ZQRwrH^;uMy&3y9V(VA-jmu~F z(0;4iw+V|^_N|`6V&BFrd{z5)(AuLLvv2#I^sCvoNsE8u_HA$UYr=rFZ+BXLej)AK z;PvcV)6#!K_N_VS+qW5czBcvWZW(;j>>KDoytwQ?rR(9|{OFC%SGnFE)@rx_LG1^i z{VIq@C;f82IQ;o=!JnUc^Ix^Tcf`u?zrlQ#^I^pTPMBtlx2aM$xP6oopv6e~tF52j{H} zUi>!|t|k7Q*tc~SFZ`@FKW$ojeh-^6@$#=);<&u+1J&z>Q_KmROb;;{<2UuakG ziWC0@eaD3k)OU%;!v0f1;+>?w$@zW&&^M}Xg1GQnK&ST(xl|b&Z=YIU9QR#2o50Uq75TAth4Neh07|Z>oJ%3WlL34s- z#=Bpx78n0VT*v(acF4FWJpP%WaW0=@yt3@)y@>th^Mz7Se_Kw146fk4xN~8?g{1Oc z+)Jl|4d>eBeWZx%KdArl^Tl>qa2>+3+T||Tn=oK{V{aPgauPp){W^({!h0p5r62tF z{7EjQ?kxxeYD&jz_->wx0t)l^^{vy+Nd-y`4r#GyC@_IT9o%TffC{6#;Xz(GTR$?uW_w*MKfM%Ex~0?8)zC;^#VVA%Di-Tzquza&QT0sXu6M?1#Qf(AqxT zbvzd|B?t5w^<9dqYPm-6yw6o{FKB+IlU)x`-;KW1lG{b~~Ur|Z--&sE5Gi8HJ?@#f( zubcif>>&e2j2FAP>Z;asz5C9(3jKdK2%#C z_PfhPAkF!Y({YIh=oB95pU3-oE*|lx`hC$K%lQhN|CM-NlOOQ{^4^2-H;}%a^_%k; z%uj`$uN{^655Tg&Cg)4Q-xBJhKH2<9(8>FPalS#v6XxZ81sBhFSvik_^EOM*p?=cO z_y*XUr86kcz$pLuq=V-tMmx=UmqnE4;HMY5bo{T1Uq`(2{p5ds$3JR*%k2N7?K=PT zLPtZ%H%%}6?)a&s$9pYZb(U}9ITy+j?+1MpePH=w-*@~(r|;-+(?6d?dDd6a^_Aj^ zelG??%;joZ1NY0j=@Eb5+0FW1Q0156P`_^yApI!#&rAE^Jx-|~?=u;F68$a3k$!KZ zRiM4#dC(@y)A66yzS4d{p1l8U6nMFh4)VtPV3F2O!Q*`=b-v~j`UmvMyk8~p!GIC3 zp}MCYpNI5&Nb(*S@OYo9KvS@+^Y=hMsqX@VKHMkwujs&-{*3oJ5eNF|Bq z)b#em#d}%d@~fu5c6$26``e^H>Hb*3{fO+J;%|aqz>(5t9UqPR?qowxr{{gz65yp_ z(35xTXL;1O`Pz@gzOXz6V?0IqJU>JGkNtDqpK+fZJRZdRQl8yN&wwH05Agm!Tw<5v z;g5UokKjGY(7|*3%)U<}2NuzPr2gr+(53ZR+HX3}cThi#{k#+x()I$q)6cbg@7YYB zDXIG$wZ3EhjP4vh`j<$L`yAPyq&&3WrROSaUVOk)AI_k?9QcWZ(!oa~i~OayP*A^! z{sZ2Z*D2T^0O$03G|*q@ryJj{69E3fa&D2&Gkr^spEz&$`0=o^ zn+9PW{HMPd3FJLT55J?hn-%^O?`mM#?&?g1A0p3#FU3cNh;_gWeA7;Fegcsol z@O$wP;IgM5><^r;8GWXH+x`W9zmTZ$68@0LJ6vej`j2>C_?tX`Dhi_GQ8z%}*?zFU z;_sRFBn5_iA3wfZ`BU#11wW+!t$7*YExaeR@R94^|0eP#{ggN6Z?mEu{DJajecPH+ z-hZL|a)39_`0%E^7Cv$#_f3XDyyvrw3Zq?Nx7|McA=#rfYk4zJSx(BB%#fd}<@Qo-ZB#JsiNzz-z{<#XdonAzG;ScQz z{B*qkz!Ac23P%5s2MO%_BM#;s1RCc@^Xk3dEuMU~A3ZF^8Gjk}@POs_+f(4DASiWz z(iG^w`!=e-8lV3z$o4nBZ^Q0p#uw^CH-BLJL*Hu2-FHiV#wUxs)O*5+UlV%;`h_6f z-@#ugcbM5}A1_veKRXuOUBCgKE`9zJ>5m)$R>m9b_m|>>2)L9lPx$~oa8SUOz;8)P z2W@)+&-m(5;N?9+$RF(`0Ri&fAn){-1cZB4^7h^$bCbi&v)JREb2R`?E~$p9y!k~I-$)J$e{$aw>{-u3K4g97yz4y~`(0a5_r2o&da-x7 z?=}n|0G#I)dGLMeIWk0ff*;f`kg~5V-o#eV_8I(?7!A2qyVxKHdqrub6+ zj`eF~zGLB$e%dp98b3O~`FIM~2RVGD@Ys+(a3$%VdL#&kgF$8gMBh5Lx1bZb+4n_Y6)PF95G$%(utj?`V62i@m1+ ztM^Jl0^mb{!=lnRz?V-2v;Xd4K?T%T)c?d6d3y%l8~-l|tG=8IhF_p{gnV9`7x~P7 zk*7iUAwB%1$B&n~?fCdym%PvYioq-UApWe_qrW|b@pp@0CI1gNKhNv=IokNeeJkvD z!F>;e<=OYT|GymXXDEF%FXx3(XrAZm;GatS1v#`I@Mzznyce<`>AweiQCY@hP)B5tG^eg0=!q) z+EWnu>H8D>{_g587#9 z2fPo7`^91Jm193`Ncw6}*YiQoxS>7XZzOp4wG9)p9 z`bod&2=(PC??Y>Wetfv!g6X9^8qp(Nptmb6AN?EjcPag9O8Z6V?`#j0fyH+6SLOYP zK;j?Za$Lp23cr^4Ymk2x@`s4<59OERu&U(+{8ZqzkKlX548AuQZJ~8c{#tJaUkgT? z4!$Ju{HlH_zZ8tt9ef@a_!#=YAN2LlzqAQwDw;z7Osw8ZU!HGCf9HFi);~u7hkqgC zq4;}%XJYl93G_Q8|J&krzHd?w?)?ARdmHd5uj^cN&yU0)2{B;V!7&MAkX;#*SYUzW zUUEDlB#@LgvV(KE4XI_29ef^at3poXlvBn)66`~d)t0ujC8?5!o};?7S3UJT9F1^0Zov(7yi?s;(MoEzUE*n|5emJECDr7@&`%{5X#Z2HhgPwr`f|G4MB zEe?OwclAH_0l4eE`(dPi{N<;iKk8ZxRFLg&WBnQD{}u*rtZxm@o9DYxpV9xeAArLB zezt!k`bQ=MbUqE|1MLRdg}%Skf8Rg0a1hNG49)ZXW&6DEkNENZ005HWe?rm&KXA_% zH`?1m?|hbNQg{FQg&QP)@_nFl^HU4^-_zu~!SCuXymy1A@8FN}`P5&V)Oz68ybbp7 zTwRg=jr5nHpAX{vy=Z8AzTd9ToOAs+q?BSL62je z*l+Z&t^M8@;b+0_2ibm${qXEJ^b_jmgTDQ4v-S7E^7i{_Yrm~LJbN6=V1A$Mx7r)E zzn=U&d{eRgZbXBTCtJ>bOZbre*7lI|?RO-+win-i+xrP3_FLLhV80s?JZ!%kz4_&k z{r2OZ(tbm2K>u8Q`;BjjfM4*a_B|c7-;HVVZzTKONdB-NGueJO7TNDW{#V0(r^$Yc z{q2E#Ath_CL-rf(!L!#H&woVhb2IQ+&)uf>Iv4WC{ZYRC#y*h2GzP@5Ut*6-+V6RG zea`xydm8w#{g(A6w%@R~zCHf{>^JPIx~sC^^X-DPwYR{_dRbtfW&RS&9bnknU+XXJ zV=DVC-P?j`R(mb{z}=|s{`Jkiy)I?H|0~(=|6A<$B4@wn6xna+yLHSz_`L_L{>udhU63j<6uNHG*_kTd&tR8G?mi1Vd|0MjSK)v}e5cK|7?lO_hSA9@rg`; zM|m~*`#&(A#r$i!4jZtV7vQtOtvLVt+lO!=pFEJ)#9ffLb$IS?OMh_QO;Vn=d=Ot` zypHpE8y-Itv3G2LW&OjJhsY1%*`MbNF+Vs@$m@E3&UIALVqxrFT}BqrN!roqx{(U!gK-mVd#9}hPha`w2xPxV6SnBwA;^P`>jUl@NR%#kbMZ;rb! zA}jof{EGOKzEHqlJ@JR{hepd66Pnm!5aw_AO885qg8B!2Rf0dOpQJx#>)#yW58vmD z@`wE~sVIL(uY|w3b1U&@_Y3*_rHH>d%pc}^8$ABbRbHQ5jr;a}dj|cr8s9_q-KZaE zPxEC&XXigl=${R9bCLV)qV{hI#_MtK`|YCNVK1!x%d7p1m9oE)_4L()t5^H{i2a+p zD6*fj3;V~s$J1X62iL6e_4&E^@r8qT+!1m^-*0SF&z{NmwXN=wK>_O5P5o_codF*6 z(+d((wA#Or{d~4ZAowrI{Po;hq<#keDbprS@n?Hw*T0f=tHi&|{BhJCJ&N_HhMO&1 zdI5+pvcC`Gf9ZepY5xOm?0*aPvmyV*_Yv0x_Yt3x^?vNhlLW**qCWxqXTPW5tyeC= z`ttnw(tEP`Lws1@i%WYzeQOlFtoK>>y1tamKde4D->&}w591iBpK_nG^nd2=^yKaO zQ{4X%d)sXVjQWE7UhlqN+%EkUluu;;5#*2h-*}J5kHusEQ?TC%>5Ki3=I^@svOcQ& zJ<)%T=Mdi7yD!a;VZGn3kJ8*dV}7U3jNRqUZ=V>TJfo<{#uYPUR)FY3$QU&8dM z?*BypMD9nlyolD3_AuCB=cnv`5A1*7{$Yvr9&dkBx32ei_nRU?UGI?#NRa{Xhx&*9 zG_ChEMfMk9zlXOUS>~rOza#tAq&?dD@HZcj@}J|EzpY=Vr(FNB`TeiRn+Oc~_2-)< zQqebq?d{?54))tb_7`0z>BHXGDhl~v|4p#p2=d1KUAvuIKy*_7IsSvX|19_(ueW|H z-{VF4B5#RX<&E_xd;jd3YcZdR`*Sutc0@iOg2$VmjP!>Q{<^{W0e?0>30og*eSsVL z-kNUjd)`aXf4*3^T;oH1aO(@+{7^6EN&1rsTOV*syYTn#`sd4l&-EAXLV3K!?ti$> z-8Y5yr~6smd~O}|2k>tEu)2TYbyojM{e=AD{q=f(wUs~S+Y_&e!f|-ik2Jj@38fZ{=Nx3%`Z6_?|ZO2im9o zez4mQg8dmPUk@+&ld3ph^d0b`|CYdQKiXpR(}0E~0K9NRJ_%1gnEwO*Cd?lgXq*N~ zAN7yR+O$KY;ol zu}=|ug#Li+N0a`MXJ2lU{g8G)nUyE>3HGmg=Le+xu$~8dV4Iemk3oJG^e2P*1b#%n ziT-f%6Z<6JgZ9b?c(hkf{9-Sm576H1{(ft(;D2z?%=Pv+!(KuE==^<3_8W-(+Kc#v zUnf7*2d{tS+piu^{?2~24ukTcvy~zbL=g+diQ@-Y*BB?0#Or2hnKH2dP_*sVaR4FfOpT6_~lxLk? zuf%>G+dh`L_4e6gpL(Qu;X!$QTlOQ|TGBq*^8cF;T!wwZ^4*m7>DF7Pwomau|3m&I z?9;8TfnBD3s<#c)?l(aDg?+LYeg^jG@Rarm>xuI-ke;m{sLv651%4y;>6F-~!8%XD zz}|hYlzpmm?X_C_l!O9C{gwTcHof{l|AC*?{pssO|Jw6Y@Q?R5M(oGJ!GnV95B}L60iX8mn`aLi!}e+QUZf}dID2*dU^1{jRz7OKU@s6q+b=yoQ198}H0)8* zw?_yMd(pK5_QQvyadZ&i*{@_4mW}3E@9A*Cc&=gz+=)uMztajK}1D zn~;5i{*T(5+p4fPiNM|qjNWJIEjSndkSf;a<@H{`Y?n z{965w@`b%g`r{#yM}MN#6A<(}>Oa=IMO}L1pOWLFM)*%PKJxbWSQ%k_1pSeg@sAs? z`S!=!K-eGDC)>ZU_67Vc>u+rI?a#>%!0nBP?3&UyN{xr0e{SnoF z#p5B#E}naU_vB~yr}X*bp$0vG;l@Kv-Ul^;@laEA|64v%{!cuVZ=?9Flv4}$fd;_(pDpL#sB+;zw<*WTFuZYu9$do%6vP$GlkvGoV#6R|g6 z7JFmrmP+}_`0BF8LknzwJze#9DCON3dG*FacE{!wjE7Rby+D0O`?vZ@?GMf~@OUV- z0@A8#JOus5_G8NN57J*a7_vWbUytv(EkSz0cqrwy4`)xyjfYYd$3w6`&{yThL$F8S zAND8i+aHX_yRaYJd!#?ON91MaPjvhl8K1*GeCkQ_q5lH`?0D)M#phwb|9vulpy4+> zo~!dN96^0V_~?4kd=s57)cHZ-2>J*2HN0a`{&o-Q`_J0Jn7FlYt)t@8j#+ltqx z_6{aZR^aj=Jl3BI&5P`Oddhqr^F?-lDf~nzn9u!Hx1R~)N$j_${nOZ=vj6c!0V~>a zIyNQg$^5fF{)B#({*Z8F$LFxu*TbGlf7M&BMfvn6ta-8HRqXGBysf+M9&bH(<$yOo zU^`(buj_&L!+%-kA4B_}zU}3w4)(?BU6$&52$1IJy_+kDJY}_1AmrYqyKMP(L7V`^5f}9nb#-;`b=c zcKtHGTiO~PnXiR>PVDk}J2uR@X7f*v7d(H-5$c!B*Gl-ljLa|HX6Hlg{FGe}{eT^x z+Ze?CKb|%Rahf=R^1-A_gE=3wZdo9=Zp63YaVb1`y!^+(@6ZoW*QETv{IpsB_=K2e zJbAQ-C9w^UD}Mn#KLwua_s`1s*gg_|iP;LbCl5^1q5Ni+`!>1Xu)*Zt>%MpMcFfmJ zrY^|@MPVQAZ+$Vo;qhn9VmpA?XUC7gqa(S&!=wI1@p~Ze^ClC*@5OxmO`bqfa^H{* zpK{?_CH#d@dUk#<5+3-snw;{-=?OkA`wnO?QF#@tyw6n%uj$+J3a5V?!nd0X zmBMTKCCe*iTD0#n&6` z2P%(mH9JChxgVmay-b9|hwF>f_Z8+qC_LJi`#z=HAM@Dc+(3T_e*@-V6ki|xJ|pMn zP%(Zq%ug6E<-N!pipIY*`hAu#Uidj=4oC69{r-BlsIyGo=tYEFjohy81j{mSxvXX!!M{v7fv z`5y}BzaKIXxAZ@(Jmz70bKu_pIF7gI`J`_oqw=!z6G?nO`Q6@nZrd`FUm@}2u4b9Q zuy;V1quWeWeuV_fPtG^l3&Onp&dKQcC+xQxoEPku2EV)?TJkINkD6bchtl&ol0Tex znrF`+8R?ZHyqs?erI(7-2aK0WoreNG7dj8+J}{c|_fy-sc!^)a_g~}r`}?WqN&hkj z>uJo*-gziV-}WiN&y52(PvY;VUV`&F^St|ZBva4_avsW$AM-hJd>_*osUBQ zp6b7elwU8dWr)=OyQ1f(z<;E^p?@6@*RRWpKXm@e#V`F=TfQ!STYk3xd^!Aw&TqN+ z?RW=Wav2{8{|oVF8U0X&d_w2HT>KJsn145(K>z=8_z#~SllF17&yUS1IzMLlmHN2U zo*$cI=N3zzA6tU@7&)W$deisSR!tBtSuisFl$*K_$7*$48ScSZ3_{$+k3 zh|le(xyEYIZ$!^WMf5-T z6Mk~YKmK+7t&qKv`Vj7a!5uwMH3#lwaGt97X0lJzzi4IqRJwm5?ZuskF44aToyP({ z)W5Ly&DT%xNB<)3jaPEEy=3}B=ds|A{zc^cRz&~7pYva$zj5`>Z@K=4ROO(*5k9{q z{f*H1E!W=&&JQh_qJM!YF6h@O&u^8~pP~L?3H=#4pAC6M>ep52Uxdzwx%hp3<>Hs} zzI^*1I$!4E_w}`l-^#bFeyJp%Il=if7ylc(e-S?4CiSu2eQ-N#`)iT&=~fzs5*i*lKg5P-mmDziokD4d_SG!)}@*nL_T)F*^_9q&z zsJveP{={gcKT%@7h0dSxe9P8I|6t1g$oGoII~DpPrRG~USD#*y{n-=EjZ$xP z`htAFRQF^1`-kQJN7P5$cVzwb{@Td+!R`-8dG`&={$-4Ru|J_+#HIUTaX){`cf()y zr(HsNeOas*qrV6D9P4)Xk;nJWb>|-dLi!t?L?yNl_`&!D^LfZnZjbrG_bq>LqdO#S zoOb|T_6yn^OMb1pDVU!``ilpjwz2gsnXvnlH_Y8^#2(o8hV@L@?*M-H)SbkBJ6Ugj za(%=f>7l!Y^!~d|Px}9uo)ovwkMsXG?>zEvcewEQ@#3$O_zQW&kMdYN_^&qgkJl+b z8HY!HL;Muuz5A^;%ys)O+;eHa?4<3)oH-2EQ$xC@Tc`ZAD z`_WA9K07Ca@XHRzC4Zh<%J=Zxxmn9G${YG;f&Kn=E?+nfH}KLv`ov(n`m-AMV?819 z%l;1u-{0hYuu<;2ME$}2jF`#H73}_h%&(?_--F=w=P~R>d@`N@1Go?lFXoGIzM~)v znHV}4HojEK+fw1BzidFRv&h|L$7r~}eVMz@7erhCu^-KD0I>Ok{Fh%ma{sTl0uTPgE&eJGxKaP-z6QKK z&u;5m4la4T@`r!D^_TPZcK^dp&pYHVT`gPQ)_x+ zU)Dd@%U>SkZ_HnW@~h8Sgp)7Ery|?B9RB&n{KeSM=ee`iE%nXj-|YuULLS)Bu*dHo z1^#mXn3a#rmmoH;J_uf%_P!zbWBl7_%W@3u3FUn~?&}tL>iPCL=)c>tlqgVsh~K+k z`kp@9{;%(!YoT-9d5)m{qy3=%$$ndrpY=!lbKHHYAldsyu|LM%Ph{^)Lj3q%2U<|L ze3MDP{-Hb<>V4gEUlRD2^?dKq_8gMG#AnMRw@354A+n!I_IrBw6WaC#e!0Kl+jD7e z-xmGn-{)=fzv1x=rEjmZ&DCuLe>Qz7zHs`{{Ux|BO2qHsZGLQP6p~FZW%}W;9cKt3kCdjKR(iM&bM!}9(OTVpUYExGr;X-ruJ`5 z0yPQLBv6w;O#(Fu)Fe=oKurQ~3<=0S3G~?Wcdo)8e|o9#F6`pH+;|KLE-OZIA(hm!TkK!-n|M}nhP~pYz6bk=vsqp>h&2HS~ zbExoK;U5am7k>AjZo4!m+j6w+J-`3VJ@?A<`zW@B_ina-d;eT}_qBK5{hrO6H@Dn- zt))M+@!AFVcD)CGFSu9yy6;_nQ>?u(@%68L__a?I=6${MTT#*91__W}(s&7~}>NDtm6;XcP^czHt`QbM4o%u?h8iOE0Z zGic*8xo6sjy+F&SmAR*$ddfWGQj43yYnS%?z4_W##~xl)`1Iq2SHJn)AMUwyS7Be_ z>tFrDf4;kL&0GKNSN>+td+)texcA>|?RdEG58GD#k<1^4p1=Ahw78cGKP45{er$$oCm$#t+5Fqz?5SQ;gkQzDg)bQu!yFRI$85UjhRAy;>`R5TsE@xFx8)!lB zOva1znWvr^S-y1X@})pFn7H}Di$8pI&hLEY2ZcS~d06sr-}_$u)<+6I`pXyZc=@-U ze{s)uzgT$rwI6+R+2n719nGmw`0mS4Mi*cEQR=mS7H8oHpx(8saD%z`J+{JSQ9mFN ze0?Biy8qyr_uqT(GiX=8@ys)uuSH@8)V^0=JaFSBq<2^A#9 z!Rx!vgW&D~l5dTeNdc&TtTEXziG+;llKNu;H=4J^)mY!~iuHdYCbL=brK$Q?%ss#- zAQ=!sPc|X=X5eLpci+4_ZG3+C<`sZLT=xI%jyHROjoaIh;7*EO5F44+MaActvHEmEo5@|Ra8K! zC3)wzTszG1FwKRr6S|==;2 zBEhW!w~Cssrlw@jl29zDp|Ny5(v%0{nlzK~8#2hB`5YR9mp>`gocN=BQQz)HqisUl z>VkV;ESE5`E|a!7iZ5%s`4x!6*05MM2KBB@z=oETP_#$}|EJ8POIh4-`EFoA5>EVf zC+!NV-Y)3CJ8Pq}wWuh~y|HPfiQLSMBmYRzJw-D2HP7_rb)clYf_x>S`C8}lRYIP* zSRV46UfzmI)|V%xyVs?QS{at-<)r!g%5!LX^4$LulIMxpBG1FqljoOyLh`KlTBm5u znO#4JrzcN$%Tp=+c{y#b>h`{Iw#aj2dh&d~%h!)ve?IkNl_%NjuBOdNXx;nj(Zgzm zC53qgJ3@(r@~fb9i7mEv-fO>bz}|K>CmZ5T_cTx1p6xyZ3mf!ooh9NL6-_6H=%VT5 zL6I={k-ti$V=YgRj?>Unr&E1?zztjTZN~L(Jf6HhzdshRJ)pQhme`jp)oVhmevc=C z9#78pc-GytuBGdSZ=-+yjaBFk{l1rDbUx4zdZp%PUe}*Va_8Z;^y;FFNQIV+0^0hW8>KTQfs$+oOcL4}B{P9%R>&9h+iD zTEu+-Zm&o@e{}P0FIVFew*-}ui;P-OjUxCn~8X zZ`6GN{(b=9F-#_Y?Ts!Wexvv^6uvVS_0N*Ko7~Lr805bJT&TR%JzM1;$zN0h;HuUJ zllAr!p+cbE&cgDVRqcV>|7?>tkK!%=QT;R1^--iffd7bF^flAO5-}~>j&k*)|8*`z zETNT4RL|k&=7=7gX?kzAx1&fomS|@Y|M(3jF(;e8Mdpoi+6fech$rurUj#2b3$6nt z+Cn5utq#2IeT$mBf4qG;miHC8OX*c^V^`{yS>)-)5sWetbHa1Ew$}{(!10Cfbzywc zYLhAP$uM5+2glFub<6APnO4bNN)Dys3FW0eJj$+|_kEIge?(R}@B1Y0KE7Pu*KO}u z#d(Wwzpe$k8_SFF7^*+Iug%$JduyX7@G0>1_}Dw}^Rn}R*HxbK_`P-bIP_eTYsxoO zhfl_n5`R;6Dtv7FiuJFUPuiB7ug@1aeYj`am}#c>9dyiVt0l`jTpqbbTOKoA{xU|E z%3d-*w*I_<`s7?*Z*+GlyI4wBm(uB#WK(V2mEsB63F!U3z8!-2)=5&g^!j3U$xzeC->G#{8whT zM2|K)yOy0DdUU4St(k6zBK`m3nR957nJ(c-4p{3|nnhsz3~gIHqUe2F!VYi0iU&e@kZ2LCUI}E4Y!#fUTu3lRH6!mdV%;d1=^E~RM zo9o2}!N_Cxgoanev7C)HFK$hP<}UNnoQ%?+2R2m-$sc4X3}=Xe;HbNSg$ErG1Sxw~WT}cDa|= zSh9<~D1Opbfo9rQ&zfljZB}Iw>^4foOW{gsJ)w0qtw-zL>}#qbO(QFA&{n9K{!<2% zwJJnVJ>ZMAo{L9E`6gc;awazUL@XDJ-Kr;T#GQ?GZBI4GnFV7u%h15v^KW9lz1A}gMgGQ< zikxX+9;8i*9ME%JV2fyOaovok<1KpbAco!kvgJSge8rxci4~oiaTN4H@&!5jf?6fw zFFG4D{=dn|6!s5NkPpIq1!rT*BeE7DXLlxxxNsDoi^(!v^dSyOBkCV;kt^x%*Ecg% z(mx14T+%-XFP|y#a#@Z8`H)NTEQ6eZk`h?ckO-Yg3R7_Q12PYn681($$|MgL$~_)% zGiKA8>b%f|%}@9|5t>x!k6#BI3wv z=mE%?rB4Hc9eMtlC2`v@)9??cSvGvh^F?{2hDY0K(fOpD_4dyn1$p;$itNOY^GG2Z zX4p3EfN~HU6tZ72#)N0!i_Rz6Q&pkzn8W2E z3jc1HiPGn}V)vz>`C3tRo`{(_C^_H#LM$6`$KrXQ+|F+>kMmqm=WeI5xV`gS;qzf} z7)3N~`1lVvq21d7@B51C<0yDvmtNB%JG$4Uvn}h=9rcxLiaqtY%{$>Kc(GF@_HfH+ zHLskNlsWBQ4)v9Bc+j=9YzlDUQ9 zB<04-uz!sFpC|tp64CI#NB&(D{zdr5Zk6*1F8#V(ec0VqmyNh%btd9|A(^j>tx6;O zX5k-wIs6-POXHFn9;TtyMKDs$86&iCm|Q6gSPoF23xoI72&R=k84$rL{k9H_QNTpWwt! zA`#x_Bo_n76Zne-#da8-{wemCHQg!v?V9ft{&vlF3jai8-&P6!BKx$8{Sy`Z_eJxc zvDr5jNf5##p#a6f=Bw^n#``g}=3UQ}`#%{wc%Pn@^SU zPiXp+$x`X>_4#`x30+;3{$5-!;PHPkSu%a!|FvYP^!L_jc#|rXe%$wur%I-e%U8Yh zlPO8xKa)l^+%I94zaPFnCH(o}sRd|w-(UKPp%$S2T)uk)`8JizpYQ)lQ_1}K^1s+r zQog?bYfa#P-LkbsGvii6m+x~VV2)o zV|X_k9yX9w=vq#(Ys13ptp4?T+r>S*KGE{gQ*{-4WEf@5>Q!+mE#zSfo%5Q~T`Ei| zT8VgfART{ErG$SZ-C!>Ig@)Xm{IvN*{Fj+t*IV+j%Stc8|8&!@B#+DRpH}`aBYw_r z%?+KXPYHRx*~??f{L30t37K6{o$6ZHUv$>zYNt_)Z+iBlCa;+-uXPL06rGXeGJ8GR zYH=FX@KG7V@v^&`{BoE|?rP$^8oH^Pbgl-kuSXm|jyhC{?^0zD^gCk>xvSF;ydoP` zDG#&2U+jMU6#b%U$*0OXFKOx&SJ64Y>B_8{G_MwBy1X&I6d$E%9IlePnmDS#R7z*2 z`B#&tYA{!tubGm@)!^fD;wTlS6zxjoTt)aR)o0VyO_kE)_-56~0iLT1t!iS8IA_<1 zWPT~~%DRT5* zY8r~wU!{kCdADNmo=e{Q6}d5kc2RpfVL~@*kiV=_Ra{F8*4k2~_E<@Yy92MbQy?Xt zva6A@YbR$<9bC?8S!k84(XNuA6hpMRLJRH3XvAqm_sMaCdgeWL#Z1=d?W!$SZgmAM zm^m)F(k6RCpgE3XPtHliWT$~bZo5@4E&Ke|eY&6HS{@>h9K#iOfm0(tYQp_jhzU$koIh4<@A(W;>r-uIt! z)zE31Q>gb=6Iw0mX*bc3V@d~3-LvveKFJsI5E;LUHfCCMT@hzGpjoXthajAGo>hG| zSH52fm7`cCanH(fEO{pwty*)LvRn2afi~O9aaONia~YIskBq70j`o6b(hJImdqK4_ zHrTX*p_zLik1M@r7Tt2$nFhm*r&9U6-u`3J-CVfX!|ZR~+vss93<`%cy_NH=LNmRU z&vbc4EW9oMnSCqMjA~JpS}e7=xVQhb5f`xOQ)_WoUFX(X+#A@2)mj`RFw<_YuC=%~ zQHvv+R_jGduUUH~b`LPz*{eK`m{lz-d`8LZ=RhJey`QcnI3q@CkQevMr*zA?17pt} zB#O?Y%1N^6#y4GjF#C)%dv*B)hj%kbkvzC{r3g*(k@C(@)T0S$C)xg1laErfB&E_W z;;+QaCS%xLO}reYlG|J7Fzz%|E-C9A{=B_i=inG*p1%y<~){|m7EX8dy?fMequhi{HU{M(f$chuE zAQC8pkIT8WY2cG`j%^zFdO6p|cy`OVH>R<#ie)B7b6 zelK%1XIAwdXIq(Yn<>=?iLSh0HGod3+dN0Vyj1%VQ*=x{-7oyjsQx^SnQ?`=+22%K zdGS`BYGY=MZ)Snkn^7%_wJy7<#l>~yMfaxE);b|E3@@(gYIm)ladqWcYjM(hm;D&n zRc%|W_T~ed5&e$tm40Ve@Kwm!5DUjS39sO5oiBWGXSObPIEYRyHhif5@k$@6@$IO%&nY^;1ESC1_8dh z6@vg@-HJhguWH4B`DJ(YQi;s+#{;QUwJH$ct6LQa@YSsf1o)~}1y~B~u3jpUS^K?# zROGHXjeirAMP7p-8FC?J@c&XOdm~0_^r+SzaeJ+YkU4*|5MU4Y79s7kg06S z>`&EZ{%bS;-T)2%+Ii^OZA)Q!L@XcoF)O*L#rZS;wf&F~acv&5*5ZmrZIxP7CI4EB zgM_5l@#EZsQk(g&&HNXwwbg7~&Bj3jcxp5Mm8@26=D#-cFD*sptjnAcsLlKbbJn$) ze^*z^oDq1V-u4sOt6aP7$DKcZedn(t^B=XD|2M?Uf21AN_Wav2&C)rJH}ZED>a+D` ze|>G{|Mk_JwONtctVq$UNNqnPx~{dEe+U6{0yBNrhs+~`)0y$Dph|Nnphi5Y?b`W&5CI6#mz(2?m+fM4vAjrYHMw+t+v)ywANN@ zanNY5*XW?u;%Y6f*5YcXY|8YS-h4Cv)!gh|sqc6>s{tHmHGD+x=+DoQulmmMzph&8 zj();dys3xq74PUL{8hQ5|MI1z`|&MTdxLKuSGu#G@D=y(6Tae|{e=Gs-Pv!s2z?Q{ z(kwpVE8f{p_=-385dJ50XTRkl^wsA|v-pIsxPPDU6>sh#{EzKxz1-@8@02`&Z?fm= z;6Ky*kyGB>6HmumCandGrDBr?e|j!w6iruKIw@bzp7g$+9miCodfDI3gfXJ8<&AeX zubMPH3&bOyz~{yG&dV1<=qddX?Uer51U039eFcAvBBl%vcBk}js^H&T>Xe}LA(9QA zL_N8MWEsnrk!6^Vz4Dp1Ddbq>PjW2sCpi}RLylptz2f_Ea6M(Jrj)EI$tm1}^Y+kb zO`YzqeBYhJ2rc}ozhd<_Ojo?nVz-#&BsM0-Ch@(<8*?U%SDgF?xEEsix93dYMmV@7 zt|+c6mVKK?53pVFyh}sl4*X*A+>&s-Nau#II~L1b@9`IeK^kxMXc`~!5YJ}9aXb;v zEe^Y5aT7@cX)cQR$4tcigvp!Y^ulS&*P!Uz>T}R|vU8jU%CAir{x*A1uM;I(W2~_W ztygAY66wNxV2dK=sD^VhjDsEe85;(FcG(EN&64wI;hP2y_F zMlZUI_haI||Gs3{KU?IVNQC`+iv1(ubH)DgaQNY3|N5|hzQ})H#DBcVKamcH$3_eL zgz`5(?4K#}PelB)MgIGm!{K|J|GMS%^+oUYj#ZWx{*}M+2K*!<@#QqWC}v(b&EaC6 zn!^71BL76he_YdWJO{k&YYfMuoe|AffJ;Ph8MI@0LjDcmc(Tr4@~v_Ce-kZ;W8d6x z96cI`@Eny*q(0{q7r{j9^sr)z%VU3b$L>p^rM9B&yb#NIEhrLSdHfSG(fRCu zrS5>ACz92~AK+sN(Vf#xKM4OqvYPytY<*4LNlsTZ9n;t?c8O{1E|nf*D!HqPqZ(c+ zrBlg2u(Juw^1$r4{-85uMP(P@_st38wb#$-1^7e+kI4l4FrM9^@Y3I#4u8@g@bOjf z7r~d~?`o7uHT96?!0u}J5DU-YWPHW6QaHx2J17_FhsTpzZ!%SutMr37&d?k}JR3I& z_AeEtT3USj%{RYdOjg+28%I6QrA%|Sd2iWz5yHpo<0-{AxLHu5*uLd8l ziz|Nb+%oZp`-P@4UqJdZeRerwv!u?4^N^3_3q`$6S)2>?Vn?g!ZLVuv+AMQDL48bM zTp6FM^)W96jwkTn7x70uV}JDNgYe+UK3$Ei4M7~yGz@0C<%wdV>8L&>Oe8IQ-MmCR z(X_O_Qaq77eZh-{`H7^Z`YjqyG_4b*>3AY(t#d+* z@*GX;QzhexrnSCwJdw2GCCeh3)_Y6F6HO~wI-W>cUnq@<$gsX7CYsiX5N0YFeTi^Z zm&~GGI_usmb&nPGayZ5@JHN|R8qK9dciAmzsPDtjUBJ&O9c}rY#r1&aLb<#ujl-(x zlUGv{FzB+KOq z>l{UK_$!pM0vX6%M>LFQcZ7$;Ud#k|<sd zGp1DBMVM;VC`!zomWsO?o;ZDWS0l5K6_IuGpGYqwqSfPjeQTL2qt(fB6vONk;kKIA~)%6BS^*AEsP=$pLrFk{2d3;`(a_2mk(!Vup zZU#+s9%x424N|k_T&v$`?rN-1DGOM$+|ivVTrZ;9cSc!mETMX*Bbetmu;q?lQzp-$ za=NT>dO54SuyT4;S>;Il!I(PUSSEfh)0!2w^I~tfIrlgBn3`+(Y%G)WOPOA$nh#HVpSG6B9G9T z@ch@;;_fYXmWt(D$-0L9<=hg&&#Ha>(mv%ro4NL;u0%X*bdRDP?3oPCgD5681*WSp z)4*?VC(S;42aWjitZ(Q}5?oA_Goib6<})uN;3bGNVeicm-U8Y_zB|XJ>)jtCcgu9( z&ZG6Xf605r{z-UwBX$cy6^LG#8 zT)Cc5(f)+aCLv!ptU!9%Hc8=yf=kb)X6(%mLB117&Ud?%Te=?c%YztjV_Xqm!QC`o zJ;1GS3rzNNl3qM%Z!^nDXniC;du5!&2W5nIcibf7hv2e@E2OT0CN~cJhSEay#Jh7S z-QUo4eH!(lI~h+8G*6mkh(2ca#@u9@Bz9=^$K}VnH7vr9leYLG6rC7|#1P2$zre-v9Z}lrCST#Ff_zKK=lXK;S>)+7X(Lpx zQJLimLhq4#wFHB*nboUr;D5>;uhaGKf)1(_CLY#LDO~&&(oaLR^Ka7Lp(df~+yi&* zTM)BoUm414R`v5Nzft+lw0(TDN}g)Vd5=6==1d=1&2;%jT4nJZ(kzg0BnNK1U6E$VqeGeP3DK;f4Y|G9m?*|}j)0LFq`%R4LT-&i{cRi}e;b~AxY7gU z*;0$BJirS>_~nnUO9jcPUHbWFGh8y_zyF%2P<@=T)8|nF}o}^KU7}$z0qd^F+el zj$dY`I-V#oQytWaBL05dGS}~fgzLGKNle?pwYIFY*ZH*qZN{!p1mn93ddMotu5EFA zVWA#h6p(h<6i8abMRs{cqm3r(}7Nr{V=(DlG>aX*`ym&XTf%Gw05<>L1b_7*A^yHB zH@I8GX7;SkmZ|Q_ru;|pm(u*rA}fz}Tx=U4(y~K ziZiXNdb7wLMdcN}d&$pFt-Zc!+v_Z_=Tq39GVQZgpKJBGu*Vu;)bE+L3OrSPj@+GF z)6cKVTHdUh@2=@*G-F)!j(_d0HONEPTY|e2qvNkLChC5pkI!oL`ReX6tLf*cJTd7< z?a$i_5wyQ3{!KOBo!$C5C;g4QT^xg2t@Sr*{SBHgDEd-#)IS%Q-x;egKQqDpsp9Wz z&hqvYkw@*@7q#{@ReKqo_4A4Nj2!HTo~f$sJ+0OE*{#2$^W!!BU8~Qv`dpiTiABe^ z-Q=##zeMG8rQ`pq=U;02xu&0M`njf`Ngvnr^Xof5Thq@`dsmwukFK{x>PJmK*Yxun zNk2z#{V2Yb=qs`KhTS+VlSZ+d$(*N&YgTbsFx_T;bX%$04AwVlXUb0=~{U%;MJ zu#L46FxfZr4Q!QbNaa4sRYFv*sTmWCy+xG0JgB*FMot-?2jS;EU+X_9fROtq8eUKK zdv|bVz!q4;wgBfw$GwCbLnIBSr~TzOOpC+^y!b;l$8n>=CqzV&f9L)sa_@#aeCxx1 z2>Hvc5aPD}C2tgw6n;4rLfm`Ejl;}72NL5xb%$dF4NrGn0Z*&_+Zg#dYBG}=bxb-??hU%8$QWCNWqIBIs6NHRL!QbwnTcly}33{F0~)Y zUnf3k8DiPz*Mw%4+ISW_TA%yX^oN{Vek31PJH2rif98HKTRG>JpJ}D{REhivjq=xR z>yt6VF1_*4)dR`fWUGrYpT(P8kTP?G2|V(*o`03UbuXp-Z8xk7W?|^>2a`sGoV+7x zJ|(|DpD|B;X6eyC`Ao~h=Go=n|KXOz@Bh?_8{7Z-?SJ+3{HFKqY`6`Ww=@B1Ova2Q z%*2hpdx<$2alc)T&WHTpW{yYPiw*8?4*B0;aGOKO{Z=y^abIr^N8F3dP{jQfb0Ffr z4qf(eeAk-6h`Yt~Mcmhz9TE3J(-UzoFk2$-wCRqx=gSQnA^w|9HsYRVIwS5T(;jg* znzo3$LGA(%#W&Y1invoI9dXYwsfZi(AmXkwP!SWcxRxJkal|cCAYp&2KSFq!{0RHo zN+A9pNZL9V>q3Ii0kOOBd&7u{Z%UcOeTi6dJ;MD8f)t^bkY}=G*hd~;KvC5)1){S=*Hm$uG zziIqlgWqdG$HgtfZ^Pdi&~C!LEVmewdk1KEyWb7n0YjA^MjQT|F+=|ue_sS0M?%Ks zSjgO%N#o7q9c}_`LCo~Dpl#oXc8<4wDc+v~2FA4;Zy(+vyrX!>@TTtu4sRdcVZ38_ zC-Ao4gLv=`;mzZnz?=Rc;=r53JBoK4Z+bJZc>C}U;~m3m?gb5RH{PKSA^aBj-v|0m zgvFbA6mjDn>PNhI#~uUyV~F$Pzn`)yYRa{ zVbb>`OwWDr+X+ScIB=f??Kk0m2ETs{%%9-*Px1a=36sU!hj$q77~YBhmN4Ca1{&TW zyrX!>@J`@OABP{_zQ4rp^LSr?<-$9PcN}l}3rORO2#0qR?-yl;^@7TMOX8b+ye=qCy!nmzeIGD* zGw)BDKD>Fn?LUJFL%d^ndp3dovv9*sj^oXCBM!Xg?xg9yCyBZQ+6NH_Ub7iAym`Fo zdl43I`-kE85&VtrboxH{;mxAY9e;n$B(puH2W6i?k&2EQ|8DTNLAfeD-x2(sL-{H_ zi!#=~4a!;n?n7DY-^n|(_3dB{{~Nhe#l#9m*cZ3lnI!H$IAGo(7jZY6A?KDmg~UDT z+;ZoJxW}AZ?${7_58Atgm-rfR#nbT8Aif6pXWBfs#McmOHRHE>ZsDgP_EwXJZqV>C zxW|Ws@G*oppYz<3enSl90bL<@iBJ58oLl%2|6Fs-=1=x?N&beNTh1hjJMY|C=kA&B z;rpCB=iE8x9(L{_4e#8tM_u9@cW&8(FK(0e;>$bt1l;g3W6qsk;PF3U-OZ-mx#dm~ zk!QDa%N-=*p19e=i+@85JwxaW$-nrETk4RwCBJiGX)^?UA#TOD+gyS>c^xSEiUfM{#49B1T*RVfs@Zs?Mpqk)Q5%`M$$KVZjrahzkRjm9(DMMyF9nlkA~P%)3Yv&Z!zuY#40}xG35Uv zQFr&hkGeA-kGd!JMcu>yC1KtRe*0{C&8F`UJ-6^H<(vB}++_oQ$)C8>e+zvI_mIO6 z{ddnT{EGkZKYIAQbLXK;t-M_O&NO;%(HGLbyPG_>*S?_-=6h~!-(w3rx7U8_O?IK@ z_S$Q`8Fg;0ubFE+f34p`&aM8YCFe3iy3}<*qt_` zH-_CUCf^!%UuQBu<+(*ZQh##JE%KH6FyY)2&fVSS;iWvJ{`B19xwZb}Z}r@x4nMZS zb4z(h{TaV4jBha$tDVf`VeeYsD30muQAERO{QmF z%;aES)2I*0bd!uY4gR-p5;D|BY?=K1n)|)velNP;vvWOc3L)*h2I7H|@0amw7nY{)Mqe`=kMup|SwZw}=RS!3sXQG2 zSvQfx^dsmum%<sqwUml zd3|U7TYjb9O86|Xu~sfQ#gnoBdP@YtyOzHroA-Ht^r>5ax4Q5f9YJNx;qQU}r(pgy zocCeN<;SYd-D2meGdXpDn#{2Q>h=$j4d>Ro8 zE$eXllXKzKt?6qxE&q=p@KoVPTm%wca2#Iri--WtEAr%ScFwnY)k4CHy~;VV{>Vke z66d>>pug)~b&^EgBl5FMC;`8s$0WSOA9-B(o(PdD8T^hSe&K%|;fxEvCDi^(m0ut9 z7VZ?5@^dBaMbz^%9{d5;%MZ=7k-E1UuuJtKc^4zTbYj3v1Gj@v|pjW8eXE65SJ4URo3|J(@Q4!*R@yU ze#7Yrui7qx0MWmCTsFetEXlQ}DB2ipU;3-zHb36?zbd;1-$ynjEkBJH`B3>(RU#+~Y=|JegZDGNz?CiKboOb`mgGlc{%GNU|ja?7! z+8zK_BGJHa=SPw2z;o5=H7Yin?nfUBMZL1K^NtQrW?Bb7`q-{;=sQ>5xn@;oxIXnh z5{lr?j@$15?_fZ~_dXI1zf#1lUONT@w(eYIy;N4ckA(cdkd?K1-rwIB1auO%dD*ox z#LDe=+|ki_$L-q{xKm5~~ z58B^bdd$y0wCmyf2e$8cpf@uB6_OFv@aQ9%`};ro*uxJzGGN`y+J)*pD>4sQjRL$^ zdqDos{W~91<%9U{-}%r-AMyP&z4s5?pYa0Y-%xn#VZoBpyH>8rY8K9Z(dj@021l%TT#OW{EiSQYlW~nJmXd zB)sM~lz~uquRb~rDZxEc7fm*V_vC@N6sH+nw%Qmmgc{QOTYZtQHI6y_gyQqkw@Swt zm`V-q{jI#;0Y9;p&e#3zN4_i~{Yd<_K8awhNp;~v>AUjv{ew8~LP?W;dqMD_pGy@LJ%e#*Gl8r>~Kt8kfQjkF*iP5V_CM+R3!YQjLD^y_jgu?svQ_`1S3ngVkgzz4}wnL+MKMAC<#z^0W<0uXHjtQCgGdb5In1mFsHR@PhYQ01vASx?O%0_HRG_+u>J5 z{3sU>7ml~X>=^W;^?8#&iujWr8|Z@97?!PCjW~k{?B70LYYaMWe&j{*p}JasAAg5I zf37@U`*+~M;{0$Z^Ho7O`?v3Rr?GuJKUT?^_Z0DCdw)jpE(Wnx?Z$QlDUPD>tJf$l z6gIH(A^eW}AL-rsK<3eY87e40`m2)xu}|EC-NA2MY7#fuT=SkbHqaH9UJE$lk#ACM&i+~yviF}Mu4|7 z6&9vq^@6yt*2nwjGj^J%n2v1`HvTckl9tbpT)Ja!PqNDYKLaQHWIo`<8!S@qD^WzV0opg6 z%2ytV=UVSi`)}6KIi5|9{&tFIb0{9npZm{1i0&W5f$va=9@fDlW#~E&DSYbOvm|oL z2~Oy5^;0z_>zX3tM?&;4KINJshUbywZ0QUir)sUg`S@uY4XLywVR5 zUimysc%>g9yz)6rc%>gDyz)6hc%>gByz)6pc%?r_c;z!sc%`2pyz+UH@Jc^Lc;$19 z@Jc^Tc;)j9;gx=l@Of7s&J$kgFA-kzH9>f#Um*Np7yl&Tm41=%M;+cQv(?n7eo%VK z(N&*}JA4!2m7XTN^4UUor7t49^0}DsO3x5p`D`V;(pM1vjN`wZ@JjC_yyj~y;gy~x zyyj~?;g!CT@S3kpgjaeu;Wb~I39t06gwM?L^iL1rmEKEu<#Pw&mA;ek%4Z+pl|Ddt z5B-z*_H2N z!YjRv@XF^3!YjR<@LG-?gjaee;kP(`))HRnS;A|+))QXo8ws!Z+C+GzZzjCvYYX9( zzLoG9SKoRFuk>ERE1x?Euk@XSS3dg)uk;+@mCyZzSNZ|ME1w4muk<0pE1!o5uk^!& zS3ZvrUg;x*S3ZvsUg@KRS3aL5ywZ;oUio~E@Ji1UUimyhc%`2r{P?`2ZRlf!SNdtf zYrf79Ug>8EukC1@@Jc^N_`KuiJmHl-LHMl0Um(2FCkd~7UL?HI%}t)(Q~joVCLLY- zhe}To-ZUp|1WkljdJEyzjxQp-(ian6%RNJQrMD7Z%e{^8O79@N%Au3+N?%KO)$>`x zD}6oTmCucYSNbNxw>Wur6JF`MZ&%Ap`RpOQ(t9a>Eyo>%SNcxEtDf&8ywV2gAyy}_5gjf0z!YiM{gjf1e!YiL6gjf18!s|TADB+d< zEa9`xejO*g(oYax<#3YlN?(m(2SNdAQE1y}yD}6oTmCucYSNbNxE1%tjSNayhpKzLW5puRg*neSq+quR+2qeK+AXU%G!fQ!;1*7^u>f%J)9xD(pL~(<02OnBvU3*nW%mGD}QJ%m?!FX6Rcw}bFX-${7QS0CY(K0tWQ*C64Q zzMt@#uLFcv`a#00-5(;n(hm_{`8-T`r5_=@@;OX+r5_`_@;OR)r9Vq}mis2cE4`cWTJD<(uk;?ms~ma>uk;;+ zS3SRz@JjC^yz)6fc%=^#K8gD{&ql&4y_@i= z=Qk5x>01b|d~PMY(t8N6eD)Gv={pF&+3~-V@Jb&byz2Qu!Yh3@;kDdzgjf20!fUx7 zAiUBK5nknRnD9zJLU`5l!-QA*QNk;qBZOD_F~avcd5#iZ>CX{f`OFhu=_d%U<#>|t zN8EulX7$ywYDHyyk0y@Jhcxc-1qLgjf1S!YiLfga}nW{zL@aJXNK@fZza6)*+zJ!uOPhg*-m(+uO+w(Zz8<%*-dz*ZzjC*xrOjb?;(8N%|G-KUgV5SAD4T6yY^rO@vo^n(&&h7Q!n%LwLBk5^qWwL>EB!d(RnI?1c%|nFuY8^$ywXn+Uimylc%_dK{#nQWX~HZ0 zEaP>(gYk}jj__LU=LxU$mk6)rK0$b;UnIQB!N`G1JgVQ6o^*88hpOjOgjaeK;g!!c z;g#M(_`H+nBEl=ZmGH`E8{w6{g78|7?Sxl)2jNxEcM@LdYYDIU$`W4b>j|&<+DLe% zZzjCvYYX9(zLoH*XL<;)^j^X%pF0Sz^qquPKKlr-^xcG4K68Xu`hLPIp9cu9^n-*~ zK8Fae^h1PKJ`WRK=|>5#e2x%a>Bk7Ke2x-c>CX~g`8-Z|r9Ve_O-X`9o^T5j{g+l#~eLPc01b|d~PMY(t8N6<=9JjrSBlT>iM07S9%}eHD3dS zSNb5~HD9|4uk-_i*L)o$ywZmVuX^SX;gx=v@XF^A!Yh54@XF^=!Yh4*@Vc+>7~z#Z zO85y^AD$(=(({B@Ih-K8(oYgz<#3AdN*^P<%HcHOm41fsDu=U#SNeIvE1xeBUg;Br zS3WNgUg?vBS3WNiUg_qYp59aarhFzHUHgYhZz6m;?XBOX39s}P!fU=35nkzw39s!m zLwKdP5tUv5nkzogjYMboA65CPk6Qa2MDk9gM`<74G~`H zhX}9vI!t(_A0fQvYnbp#A0hl1SKp2iUg@KR*LM0W;gx=z@Y+tFBfQd25?oDP!euVIvuVKO~{V3tJ9gPrP>Bk7)>G&BXywZ;oUiJKQ zgjaf=@XF^2!YlnG;g!!*gjf0);deOxPZM70X9=%%ZJh8*KSy{i_w$5T`b&h@a-Sf) z(k~KT<)9aRseV&>(p~VP{X^CBDMwel(whjce5MJn^cKPoI(aT4ywY0^5MJq939ovlhww`8CA{*v zgYZh;u2;r4} zjPT0mDB+dUbM0nNn6NFd#1;Q(zlZ03LMZzne#$9Qt`c3IcM^}9~>iAC)Ug>GVtDbKm zywVpDUdw$k;gy~tyq0?_;g#M_c$Gs3;g#M=c-8Z339s}l;g!$zgjf1T!k=*R+(dY# zZ(+Qy`VPXYp5IA$rS}nD z`5Yj;(gz8jTHwxu5MJpA39sciM0lkiBD|L4VZtl@2;o)F4-;PLM+vX_8X>&Wj}cz; zHA;A;KSy}YSDx@nKS6laGbahJ^izaaKF0{J^wWe_KF<(d>E{Tqe4Zz~(qAIH@;O0x zrC%Vt@;OO(rC%hx@@d?MV6?xd^d?7VeV8V^^4~&u<#Q3?mA;tp%4de~N^d2+^4Ugs zrMDBl+qJI_!YjR#@S3l+gjaf&@Y;^n6JF^X3E$!P*+h7yZzjCz`7MN3`c}d#pFM-frhKLxUG;;~n+UJvm?pf^TL`aqWD((&zL@ZuuMFXp-b#4QR~zA# z-a&ZHS0~|>zLxN+$FhW1`g+1EpBo9U^i70UKD!C8^sR(fK6?nS^j^X%pF0Sz^qquP zKKlr-^Z~*vpM!)~`hLPIp9cu9^n-*~K8Fae^h1PKJ`WRK=|>2!d=3*{=_7=naP8|D z;gvp0c+J1PSAe2x=d z>E{TaUg*t-oF}}}CkU^4{sQ5ZK1p~j_ltyAx^drnRDG!Bo^*88he}TqUggk2c%?5Q zyz2SIgjafo@XBW^;g#M-_*N&+6@*uMC*hUPwS-rCmhf7R>j|&)jf7V{zlrcl?6budRev`VPWtzIGB`>3xJ(Ju^Ufr4JHb`P@x-rRNB*eC{W_(uWAId>$ga z(hn0}`8+~+r4JKc`8-N^rH>F^`8-B=r9Vq}V5 zSN*2+6ycT6Cc-N{O?c(Ah44yWMEF6+|6;-`y_N8)=i3Oc^c95Ha&ITR(mM#R<=#nn zrLQNv%3&knmA;Aas^_~2uk_7?S3b88Ug=v2f56GJhww_@NqFV6kMK$#AiS32AmNq1 zoA9dVbA(s=e!^?M4iH}H2MMqF8X~;Xj}Tt-HB5M=A0@o%nGwP({TShu&r!lF{aL~* zpT`NW^b>?vK2H)}>8A*8A;=e4Zh^($5lJ`5Y&_(qAIH@;O0xrC%Vt@;OO( zrC%hx@@d?MHC4YUJ?ZGG50%dp;gy~y{IF|ZEreJ4BEoCF7873S8NzEjY9+kV+X#Qq z@w0;PO4rXxYkpPFcM@LdYbk!^GfQ}-uP40nxsmWn-$eN1j{k1LD}4*$RnKoFywZCJ zujSrLc%|cK1_J! z^C;n!K0C@3f> zC|pobP`IEV5|Jn`I$%$bm<56Uy=kUV`@o;gS4>7(+@nUtreXg)2s5#S(B&N<(YFup1vyo5awT#r*Fu!&u_}px8ynR z+w$}sdCq%7p1v>7IyB|!2lDLmhw}6zdFDKpr=Q5jf7_t z8!zPP*Yd2xjXeEUp6AJ(JpEpt*O3Q#`lCGS7W@0oKFNMakB833ORQUuJUt=LarMj7 z2jtnu2Ic8P^2|9bPalzI&QW>#m^^ci%hS{H%$bp=XXTl5LY|(JXU<7^dQqM^%kuP! zJabm%={0%gtjp6^K^eOI1+cu$_bFFzVSzHQ3W5ADPCdt@K_u{`T`B2Pb+=Q=%; zr?=$$!{eUI(=X+D9l4UHU&}M+jXeEUo;mO2>G$#(_CtAkY^-}f)wkl6wlV{F#dHRMtb8gDhcjTF~Ay40x zXU;u&`o27KHs$FD^2~WCPd|}o&Qp2%nLKm0EQ9k379ko_#(cPw$s!&H;J)pgeO9$yVSDPs+2;Ps!7#<(ac6PcO@FhxM$;(^uq~ zb5)+cCeL|Xm#1&Yv(InJ)3@X~u5Eexjy%WJkf-m@E# z%y}wLKbL3D3wipbJab;j)34>3^G2S2E6<#F^7Kb}=8XMQ_x?tYhtB@SoIUdNUU}y1 zlcy)-nX_M>J}BP|*Vm9deOR938j+`u%5xnh<>@K;oiNXsJUuPXKA(}NXXTl5LY|(J zXU<7^`jq@xn15QHUY2K{ugKG@@|^dYJiRW@d0&yIugkLz8}jr`dG`4&dHS|IbMDB~ z8}gT7J$L2lO?l=#kf$HYa~_Z6>BsWy^C$B3Q+bZ-OrGA7=eW-0=~wa`*R?$TMxK4< zR-S$*&z$%2^apw7e3Ym6{B!s3f0?s4bbkL!?~`ZFggm`po;e5P>4WmjIV4XXm1oYR zJUu1PoMZCzae3xU%hNOR%$b#^=i~?fqVwl>lk)T_d5&vZo?evaIx5T4EAm(2daugU z>+xKk!PQ8$kTV_Iq!S&^nH2GdsCi%B+oh= z%hONf+2>E?>1XoH*^;N9%RhwmypX3~%QNSVJpERl^LQsuzn5p9e~_m?%5z+?@$Qcg z>G9C{_>kl3k*6o*Ij(+r`hYz9%%D7dNS--|<>@2x%sDDgACqU!ad~=Lo;fq}^sGE{ zPRP@9^2|9YPcO_Mvaf z_a!>7FFW$|hCJ)GD^K5(=Q`b&r#Iyv!+vrgPd}39b>vu{ej?ADr}FeOdFE`%)6eCH z!g^lF)34-t9l4gL-^la4x|OHj$@9Frm#4>m9((%1{>D1QL-)sfJ@UMc^vcuwkfrw_|B|A;(&RG#ygl&7cUc^w&(r;p2XTxofFMxNu!%F`$1Ij$*r`m{Xz zSW%u{mS@h2JiRK6nR7#)zA4X~Tk`a6dFI@ar|-%$=bk)$U!FOe z^7I3F<~)?AAIUT4u{`}uej;37EqVI6JjZn*PrsDsI=YglU(2V$JU8<6J9+l`dwKeU zJaazE(__Er-Vd2G9y^Iv&SP1gUXf>?ugcSF@*G!Pp1vZ_ajnYJ zH{?04O?mp3Jp0VHJbg!=IUDlyU3uo*lcyiZGv}c^{YaiUkLBqn^2~WEPd}4q&Xzp= zQl2@lnI^l z@0YKK>wQ3;J|xdRKP*olk!Q|Pd3sWwIaBiVG5NJH|F}FoBhNmcm8Vb0bKZ0E^htTn z`;dFI@br*F%19(UyF4SDwYU3vPR zJjb;!PjAX|TnF;>V|kA2M4o;s&pvY|PjAUH=ea!nLY_G<<>@!_%y}zMzmsRqdwKeU zJaazE(_`uG{g651p|c;-`{bE3Ay4m@XU+k6`k*{>4$0Gp<(YFto<1ti=aG`~^preb zr#dE2Ps=yL^_!8WXXRP933+->p6hf{o<1eN73P_irdHRt&a~{jnPvnoo zdY;PD&*ho(LY{sp&w0F(r(etSI&vdVzm?~>?&RtB@*LNLJU#wd_kPH6^@PrTNbi+r zAM2B+C*+y4U!Fc7&zytu^bvXH9F?ah<(V@jPal(K&T)BqTAn#G^7NcMb56?Br{tM) zTAp5%XU?)by&}(?Re5?{z7?*o6?yuqJjb;rPhXekI@*w@Z_1y9dA8)~JM!%F4SD*m zJag{J)A!|>vnfwMkiQM{AIj5@<@tTYi9G#Op7VYtPjAU{-p}RfSMsdGwLJYso_+pS zo_;6KocHqd2l{Dy)Mt3EAsSp zdFI@Zr*FzL=axKuTb?<0L1@5}T5DKzEj2l5=(p*;Oap6lpXo_-?#7_RqI zd3sBpeg0gYej(4Cm-6&0dFH&9r{Bo;{4D;oA>GQ;@8#L&ALQwe@|^eB=be3z{f!zdU_Fo_&5$o<1bcoWt_;5&1+|&rx~$m^^ci%hS{HoX3njJuA;X zKOs-g$#Yzj^7JWrj%!+;UXkaxs`B)jJo`*tp1vZ_oU8KmHF@S-m#1&ZGv~HEeMg=- z8}jsBdFI@cr|-)%XH%YjB+s13^7IpV<~)_BpUE?4OP+o%&zu+X^eg$%aD83N({JQC zu3LHfojljky*&LvzCXBWAgNjJnN8^r%%YU&*$Xnlk&_tB~PE0pA73+l&6>F`MhXF zo?ex|3xBU%O`g6g&pNEh)7RyBo@~g|H|2R9*^;Mk%d>7f^7Mv0>$WRT-;-zE_T}kK z`D*xj_yc+Ri9G9eDo;O?XP<1z)6eBuhYNZ7r9A6!B~QPWXB}?j>9_KYus(P4^hbHt zA@D(D`_eb?A|&_sX*lee(2#JnPUePalwH9R}s;L-Nz%^WMYq^rSrNkdmj5 z$+HgQ^7OPk>yVMBXXROk33+->o^_a%r%%amhVLIWEl)4YSHu2Uk*8PXS+|-zy)Mr_ zwjxhom7faptjW_ikE^zRmM_G;|)9 zo|NbLoRX)H$+JG=^7OR)Zn%yz^7Ki0=A4qJPs=lBQJ!9wXU>W|y(*szud_9I`l>v0 zuF2Ea<(YFsp1vv1oLlnrZTZAscJ{p;dHS9_bMDL2oAS(gAWuJ(XU-#e`my|enEync z-jZj|b9wrOJab;k)34;2^ID#MBOed1v$yi}2YKdvl&8mjrF|w`U-8hn-kHBgo;iEv z>3#CWFn>axJ}A$eL-O=tdFC9Er;o}rXHuS?l3xq+kIB<1l|wo?ew_&YC>EF3+4R^7K`C=3JAfughPB`8VY0+w#o0BTsM0Gv}^6eNUb_ z_vPtL`D9;b|2&YVAImf6i9G#Oo;lCt=`DHYJeQ|m$k)UCm-6%*dFH&8r{Bpl=e<1r zL7q7u<>|3(_x^Sg=8uQY{zmVUXU>E?yEqZb4;E- zF3+54d3r{kIkWQg33>kh)0{kgTAn$J^7OJib5`W(Re9#D$5{ zZ^|?0mOOo1o;i2q=?!`Q{;6GgdQ+Y`59H~G^2~W8Pd}Dt&J%h1sr>p!o&EDno_-5{Z5|m|9>w}kG<{Qub49)I{Ov9N1i!*<>`I$%$bm<_sjGB zs|V!i!}82IB2OQcXU?QNJtfbaWAgNIdA>htTAn^3&zw1V`lLK_PRY}!<(ac6PcO?) zhX2pKB2TZ&Gv|steN~=0*W~Hz^31s*Pv4Z^=KH9~(;M>4xhqfKlV{F-d3sZxIS=IN zhw|^gzluElRGvA{!^htTXK7UG{UY2Lhiafn4&zv=RdR?A5SLErd@_c>inmm0|o;kPV>D%(m zxg$?+$TR1zJbh1|uaDZ7ryt5Q=aD@9Se`jg5{Z5`a@8#(a@_c^&QJxxL=V-|dAQin z2mZr89Qt2HpXVP6_QN+niuxZ#f&1mp4&tF>!(S5@qmPDPD2KQ7R{PGPAC0H~#@_qZ zpLfEsdS7oRJg75I?5on!LaDNxpP5@IEPWe$wp6MVp1qq}n9CQImI~j-^0QwT3)#Zg z3$x{4J<-qVv7vf!wNkwn%SWVE%Fiv-qF*mm@?Xuq8-4l7tHR9eyVt4LGll1~$rpvu zS1+H>eDXY%dOn+ay35GV)nd8dzJ2rNm!B;yEiQdos{Oh!w_GZ|Sz4U`^vl}PL}976 z9BV&(zBpH?lxA!B!t!!ysg_^0y7+__azjNq+uietxl(&y^OI7ngoFUz%Th{v!2q^wo>@zps-oUp-GH+dqBYMLr&V z{W|$Nne47FQlC6eCSQG$N=CQ(`1OnEXIO9kU_I$liSu7O)zxVv~gnYWA+G4)=bgZ22ILrBsX!6=TnqYfH7lO!W5K`KPx_i^W2%(0Pb#XN|Ja zQKg?^zV^+0{@uH||NR>LK}*WL=a);he5rlWMkjP}K3{ELsrgj2DLf7LY<9W!{FC=P z%U4g^M*9qYc-4gZ{<>-={$-j)w=C{?-?9&~DJDMphm-5Bs zO7shJ3%~nKwEd*=wfRE+-Qv=Gq4xfM+6xn{bd-MaZ@JI%FY+^s^=LntZ{KnL<%>L* zOtkdh9U;#)9Cpti{-*KAu~_uacYb8P_>-shH=Q@{e?5A6==#zA_SxKC)B3MYX}?^wUt)OO>ksg4#M+PlQFxy>)$q7} zJl^MRGdlS8?Ze9#H;=7`+y6)S`T>4{+wuN?Mn7rK_Eces2G>=qEq<1Gl&L z`|bPxzV&Oo?VLE)sl9$bj{fEL^pBzg_c@JNEEDE$|NO%rwqJ$Yd;Pe5pKie8bD!^` s8-CyXhhhHB0NeBbS@bWrXa2ot;r-a{n@qmztbaY6nD!Vy{hr(ZH+ZI2qW}N^ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kcf09707be55af5195f4589640abe27f2.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kcf09707be55af5195f4589640abe27f2.o new file mode 100644 index 0000000000000000000000000000000000000000..1d9f6c5552c6bf5b304ad0b58f3c9479a9cde53e GIT binary patch literal 28240 zcmeHv3vg8DmFC~KTcY(q5*QP`kXl060!yOx1lbr`Z_pD<2!STqc1vm@S(~y-#8l0$XKSm>V1~umIJn0fM|uu_bYo|i+q;C zKL1Uh7WuTx9=T#?^Z&=rjvSt0^N@Wmyo6Q%FyHg)ZyMM;?`dqlAT`{5+P?hA8T&G_ z^;|fERsRpo^Xhr%X*T;iUgUk?<%6d#9DDlIhaJzHI(huysXx;^pF{Q#uKBknUXC6) zc08K$%*)Y(7mr3i==f#y!qaa=+sb|sJ^1p;XwowuN7tPDF#7ZT=b|SkK8_}@S{Hfy z;?t*6OKT!y=bk#%`ttGUQRrV7`$e?v*vaUT_K%}aee_{8Wp#Dr$TQEJ8bf|hmL7|? z{qlJ9p~+uHj~@I*^up1L(GSNyf=pH9?Q>6`I*PKreb*17_Z&PPeRBUB(R(hQ6nU1O zgUo%ACp(`x_2E;`oN7IJEPC|h@#va2-iVHU3_HgzM%&UphRnT@HT(CYe)~^7`Nj{T zDTzObu7RBt$US-2x#&}Gke%8{Tj#-3M@kQ#`gzF@qED6{kN*7J$>@cXzla_^{t;}R zizbzBh|a3ptv+`&nvAwOfVOGvJQ;lpZSr;r zbXRYTymjv2smSqT(MajZ=z-FU(Ii{`Y4d!3+J5H|!XDIEwcfwLME~9i{}Fxspp8|(i9WX32)pyh8O_;7dWRXS z{t0-6Tu0dYG5p}uu7=2O`s(3Vp3krBdHp4`e=g*2pW1C}l+TfWDc?E)-})=~-fiFd z{r0W!b$sja4Segc#ka`klz*MD_?Pm>>-pG~JwJ}b?91OdZ@+W?gZFLo1rCrcMY#uRk|M@{S zfBx>o2-!Gai7^wgcK$v_W8iZr+x(S1Khb02>sDST^L)M|?XjDcFY-hvf09RAzsmDSUO0~{>3NsG?6PqmaY8R` zLqjCFq%M+LRUaW=xdgxX1b##I$VbQ@oUpqZ*9n{FM<3(iIr?qj$P;yRAYmdvh@ zZ~Pkm&;|V0@Rwi1U((?Ne-9r@ho610(jGbgefY{gHlNn4Mam(Up2j}^_fFJTJjQdxmw_fc1FE*CJp6`G%lE!Q{nm}EWCf;UTPUJvKG7qjrxR$UC z%J!-><_}Ez$7k49AVq{~h#V~ycTHpV#de!!M%uw(XzZ_fos^if zT|7O}B<|W6OZ2eyECD6TCdsmyk(pp1=o`S3y3b7rn%})MBa=fRRyH{m^p{P}mIcQ~ zCx@pDdT_U-XUp`5!KsO9Uyzjr{h?r)sG4_hcW~S{Gvl48^!g_T{6pT!slbF+fb!yB@)~0L#z%Keme)*GxZUoWs+#gDeqBe9M6Lth*o4=O zdWw480@auq@df-tw@yU@{uzH@7BxaI1^nZ;gtws{zFVRqL;iu?yS&2z--KWEj+Nmn zi{l{727Tk>y8^!H5gv5YfvMnBaBdpvgiTG(1bvf1_buSQgTCnre_-&I=w@^cuZy)W zIa3auyaUaQTc8hBb>e-Nb8ndtR(;DWbgd4SlZtsPR(dmH9if&Y*S+6fhFoWyXW zlilsL%kSBeKW?m}-mx-ne(m!0-~6bNy$3t6vZ>+W8Gq2tyxvK;vs{1obToB%1p=?f zF3mT?r+ghJO)PqDsfFBiXEVlD1Gi(BZB4bDpJ-*XZ*l(OtN1_D=xkvzyObrM3_3(W zdKw27c76gNX!ZFc&QDgPe*Zi0mu8sk7YOK2P(*t4NJ)tC^a}W{!|6I-ve`RQUS+N1 z&w+2pd5AT|kcuReo++W+kJ3Fo_3?1Hm^ovQhl`#3kulm z+ps`B;l1HvYc|g3k?nFqGH|FO(!VAiE9Ci;c}gSVc2(OG8z(dDIb6R&e{^o&(zvD0 z)qn#G^p!|-$U|LKgKL;r)4>qJq5<=Pi%v&I$9=)ksYzFGYHEC@#Dzh|@0;UWQh54mRMW`h0+*Tn9bplirKJUZ!jZR_f4@9}PL>h9Un)#+{8v8}1I zv8j=jcDX8DrM|%6h-+rTTkc-xb0KRgE1eJF`w>f7$Jl=G9c!JutpAFhmpE0|@@MI8 z%&}C+K;)*5rX0f#_yg>%bBH{?L8V|Uk+~VWw1hb%oxqAg7wEKux#DO$%Z#Hn@-dUV zjcR71c_yIkI7qYS#F*A@V|K6I=2%_K`&B(EqK>%^b{g^N$Vg>j6oC4kE=Qh-4trJ=b0j+%OtG*X&oz=ih9kZST)O}u zQMu^;rMzE}_b+gNod6{AFX|H&3ys7?QhY%%c33`G`3nF4b>F+k7Gv+h8adddboSFZ z3}%xXz}r`G6yV z#A7F(NoTL;v%Pz?7&{FH4RL4Epzn;aKjgHNm8G##I)qgT_RGxlFVmrq{4PL%^b6D3 zs*u)CMc3Ffi`dC$+2eDX#)p)gyi``$#fm~JatV(Ser7FO9x9Dx%raIA9+f{K1plNI zLyv=A9xCB@3Gq0dO;0A=#c>z$&XRlqr>S`ay_3^BiGH2%PEL2~_)1Q%)agP_7wYr| zPV=#l+rOXFboejq*KnG~C1HOZr`PFt7pJ>)x}MYZI$go(3Z1Uxbfr!=ak`0UUVg@D zB1QfiIlWP*H*tECPWN%TPp3Vc_ULpor<;l9_FFjJqT`L6=5fL6-@xex9lw{;_v&;j zrxC`od~KX=BbwW9=XAS{cW{~?^K<)~IlWoOw{UukPOsRjoW~5=3o#ADRcyNc4;}1{bXFZmNWk}- zNU-;II$JdBaPUEnSfG~%?2Z+9jC}!|x2J>K<3j}KC68v3#Of%twY4YUU|av=IV(GJjYsIWByYrb-y*_gd?hRoc@`1w z5%MVS#zew-d~*An5(zir)1E-MY;V|S354^+%iH82oX0<>SFR;mj#rQ`TubtNyyEtE zmJ!a!J6_(-GQy4VuB3!$KAv)W_4Py>TFQL9nBK%E~*Icf6x`1Lu)A6L8Xac zibDPUJY2>2#j--xG3xu=xyMj8uD^d=e$O!9p+mY8UdCtkjp zV#Ux;YN1#$O5~#lod99UV#e}y}teEh2 ziZv77L9u4SH&d*d@GTUpCVV}`x(TnPSU2HYDOODQ0~9MJyqjXpg!fRanec5Cs|K$1 zQmmTry1ID!K8kgYV}3pGSdzx_U8#&=(^8bi3IchW=wS%Golkfk%lDY@{5%~$laKW{ zMT<>-pA1hD_yyC*Zo&VwX>E6ihmbRdu1Nz@5q z_`8gUp)$08J{U5+y%u;G{9DGuP$%%R{IWQFZJF->LljSI%htx>?zOsqbKJdFj|Xl) zhx}jnS1q)H9X+z9mZif8>Q^4jMtx5-@%o}YRtB@NJPvQlmi<={(A+sHzc<%1Z4RLh zdCr3WuMFm7lRYTO`88W3dESVy-!y0A|6t8DtcTCRekd0k)D!=WHxlqwm_O+`Q=cER zb7%(-Azr?g0|e6$ zN%43=fsVgU`p)!toTgK={j#)poTgVZ?oN%zY55>=Zog`AJWk68L%%3Q%LRij_>Pv% zW1m_8<6+xuc1pZ{3WhdAzHPST4jo5+MIl;VNF43!oY6%4FGCZdJ&Qs#9UA&YA({@& zG)*^Vnx-2wP1Cicd49A!Fypj5Fw-_kEiA#*hJ#^ib5TS{kGY{rF#2uJPpfD!+zVWYe_twj;X`I zTLViqiVsnK%Ab}KM)})j^D^S~X*pol&&XaXN+-*|6w5y|o~DWN!z+0C?x@#sl)p7l zq0#!!$iE)Tr8xb}`1&_sXgA|aHIcuvPssRaw+5U}(OzJo zJ=IUWpjAV_BdPwq4^z3>{+c!M`gIUA z>u=LAASiuDQD_5}`)0gYD>iVH-+|?TVZSxds5Qpnw30CE*K2TSBxH;8rj3DeED&jZ zgb$2gt%0o?0tQLp?o~yg47^)|Lkb+{2d#m24FM=<-anf)3>bpu_k}zY@mV3-A12UWt%2ey(O!yPvqrS9 zqO-F_dw@p$HwG5rn+tP&Gcy}O^ThGQd)9z3+M}dI_eYMe$)UO1zzYjS`N8D)>MZO| zO}NX2nu)~moKj3UCOj|CBVnPRy%Gbe3D3`}D;aoAjqn$RFUNw;jCXble^dCKci&~g zu|Md~#D1Ue4|x3&)=_+O`|yX20Y?`m2tlL2HwKDZsJT8tm8@1G)T>lM$yRozo`zJvcW3^|dmoc^e zPG=qZdQGg)vO-z0Z@``=&r_O!U!stv_1k$`Uux_X8)zG7lxJxuCyT}xe1Ajd$xI7m z8S6ouPgI9;bLhNSoeyI_k$VO5(~XxFvHuDAz1d>zD%Zy?2UPFEKF)Jfu>W8z{22SM zIKPH`J+`|%QT|v@wj^z)NJSE~b$z`w6w5~Tk7d0X=K0khu;?Ff$LT{pi{xV`t|gxv zC$E?96n-hf+e75X^-=%aP?VSM;e<or9{F*c`;SId3G?#3OW$vd#L2G=WQ+Qp0Gsz;ZoZ34@c47U zfAXpArn{r$k0$z4`u+#?&qh2vCgPR12TdPg@%;?!>Gs1gawa_&{Uh@8a2oBkg4<^{ z5${=nT)uydB?3o(rvz9w^&k5c;uZE51+rK0zNAF~Xx+cN5f{imE07(d`74LU8-E( z!Sa$VBw60_ARpev`3%})>)hqc|3v{3`zglXeAxej(f=>RV*Bjugk}7>1!&P2w?A*U z!0i^e-2%5;;C2gqwH8y%xAIBK8*cM>^txM?#en_iX-_lqxDA}KYp;KBqsk>(6$oH>IQCG zWaD%iaN4}>v(-Y)z#m_Z|MFp4VOrv4{+|H*m`ALj<0_{WIun1(al2;4X^&brpe5S| z7|LWMlTDjdrx;?klvK!5ne2(!QJCm`oUY5jfpNxzX_whbDlxNN-plwuRzgFxS}azV z>T%R%WjN{=v%5_BrC9T$l^w}^T`yCAG3p*uT^CvFnu)q<>(eSOvj>;pk!g!?s<79_ zze^{Y{>9+BIO$wZA02*O9F2Wl90j~C?m}<3^e-3wO+62J-9PP#9*ciMoIPN(C28;` zE(@A8$YbmAWV_1y+~C%w4kzF6N>59!Gha%Fc=T+>)U#E_DyMLDx9TZ*byxU5^WEG- zdzV{0$kM|Kr`G4m;$aG>@k*tI?5q6?&IjT=MCsG%7*7@tQaFtzJh>3Gqz_AjAn=5T{->mnMJiUgsLPB z1w@sUx8B0{2#$k>F2{rTkUxW95&(znZor)qJyW9m!vBO{LQD&DOy$qNU6(u*jW-04oqLcbqNPaE*wm4#TCZ&3NHF`hJHUVjSjrl?*~**(tV-d1`xl5A83(7+xY)- zY?br|0+HYWah-Nju(j%uMD zXb7uK(mPBe71F!nrhi&x$zS;YAX$nLl3jL8*Ax%dcCVHC9!oKbq>H$O8edsb*{%Bc z(wUMP=mdq&Mycc(Cdhhz6PVSmd5p`O5~Q{Atf%)qesE zh+I|vH>qQhT>R7U5v8yC@mUxqo=3{5+j*(4XqCUxQ~f^+;yyv0k;@o5tET$L34z{(hNJwE;-EjT z6U@`%rhiIVo!$oheGBCON3ue0DJjm%uWCN#m--${{y$XuGDFg*kT!J(w_JrvJ-I;s zXJv=FxD;d7{FNWILVqvxJ(m2PjY82}e#&3zDgVs@Vv>ysK3&8N(BuTsQ<0`Q`}ISs`q{2F)n3FeRWs*#QuNPBbU_U zGcGp}1(G4}=KQFvDlVkdRW}b63dyX;St_0dtbeNbQg@}N1a{T+Y_EKvsZOMZj{1u6 z-Gi0&<2x&RdLODA-{$KZ+P!l}clqE%U+-A;15>`bhK`2q^&^85&B6X1-Q$DPJ=?~& zJlyE(>waXsWmnxpb=AYYEzNT~`^v|MTDH&iz+QhVWSVB`NBX;a#<%!-d#Af=#~VgF z8k%dz`uqAvhWbKoq1!*)-&g+d(B=u`)zi^XUOO@{Q9V1<((Ly2)$T?<+jdrNcMr5Q zKQcI1{m>BdZ=dY09&AA#kgp%_9v_(OoaJTOynWPH2i}hEk)0EvaoBGinMVGbD|*{n zc6IvdYNwi7c5NH#Yn`g!@^H&g#rEm`iV-Tu_2e2+@5&p=44{lRkZB&8sC{T>Ut6!Q zZlYzpxvh7&pW3#3sXmh9?B7vOeXvmZryi;sY8hy$9T{rum^10L3|4l}_4hRg7u4JGa4*_>Y-dmPsIRYj z4A&&ueQa=IyrO?c8~kIWar=%=_u$0#k%4~nG1`2nwWnFs2X8+ZubqRBz;~*LclK59 z9;oad8EP4y9hmN(YTNQ~b5CEjyW^2I^mX?mJ4dVTC!h3AjE@GqV}rx)+I8-A1ODo2 z-*9z#ZS`Z$mF$8|e2b6I^v6s%X<)-G-uX^~!1@KX%9&j(gW!}Y)(7;V;Tj~nWceUpn|0eO@I{zf~ zducBS%OpR<1;ocUZy-;3J!g{F*Tbaii8ksFX&K+u zzQpTKQxR-@w;*^gb%?vqL3}=gafW%GS?GNn+qh{cEsz%49}9W8*t6)^ zP5CNF$Q$K}=hI3dJ_-4xcqXVRTuV*uqv}P|E%BBAi}qsd&rmPLGaD-gpX5D~sr){Q z23<(L;~M6Tr5Ntj`;OFJ&v zfr{S;{t{&bvQLAi6oh@^ACmOn3O4&b?!>1BEu}ZajlF|A@z+b2mN&g^?tnezJEWqv zZ+BYwZfP$YeAPC@`ljUPfG@Ycob#xOZ{Cu=B>9xDei96x>Nm>Q%rBDlB>SptWP-jQ zqhz(SUh?xSc049K=F_GJO?>R<^b|7j&GHAY!GG7pH`kZ`yPA@@cLx7j>LCL?z!v9l~yt#j0 zlzeOXk4ZkY1N}plN=})X%2)X!e$>yC%2)B*P4eb^Y5z>gny-34ku~3gCOhVMIVAbk zczH$g$&M-u<#*b|r+uwT%jCE~^7Q|@Djk-5RbQ6`UX*-QU&X&{k~i1)3(2?E*C8uo zt?wbpSNXbSzRyX%%2)Bvo8-;;ekA$Ud_R?ZYreFup+svJ3MtigY8$Ee9uh(t5jS04 z%^S~K_!-ilPFVPA9yo8|tMOj#<5j+Dye~%iC@DUMSe`ap_-eeLw(!+>&&0BO#yc=N z=?ly;{szK!-|or5kv9H{@um=dqru-fDBd%C(>oFVD#OvqQM}fBU}om4sZH*l5Ct0b z&#)VBdgRuDzfSzd(zE`+3|^%rt2#OmK)qZ1K~Y`2bB^9y-qP7yiuVT(jtQw|v`3FW zSQo?##|L%?{lpdWP>kaa`5OJRql5l9KE1KgSLPp)uQ$GN5r>f$4n)l@NsGnE-C76`Id5uk)TV=i; zB1}o;ukPQK`efUj4oQ8rUsjmPU(rK&AfB}i*Y^nttGt!|_cDL0zDMeBko8yjsd6iN z2`X>x6T%+-10@lEx5_>w^{w>}OZ`I@1#9{Jz!&Pj^?s4>C7C_-A2q4cS8M5~kg+vy VQpun%(vS|xhM{86@6D|G{}&eXagYE2 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd008badb8f4e2cb34ba95f2b40b06014.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd008badb8f4e2cb34ba95f2b40b06014.o new file mode 100644 index 0000000000000000000000000000000000000000..e2d5736c346a9f213b0cb975dbbfa3039a72eda4 GIT binary patch literal 56472 zcmeIb3v^V+l`dTURIAleH$n)+ODzx}@eq0-3^v9s36Q`%WPy2&lUC~ilF{pLNpOUf zOS^FhevA<~e{5%FJY2_<&;)EJGdDq*Oi9 zU8l}dEku%8SsDKnt2yVZy=&L5UAwAw)j8cduzLL(tHr{UWMSWB8d7o@d+gt|`vAWU zFb_EWEK)OI3;s*C-}Zm#gR*@&j2*pT%Rc!gK5yBw=l>elxAFN6?ol*D;ic=tS-?B{P-+0h(Nq~^RW^W+6v z2Fdy|OJ}A3Y7O)cqMX%Y`*{#`=X)MLck-bJ&Mm<8-3K2y_dl(H5%BimbF}%=(?cg; zK0UPO{ilac;hMkU)uFrJ|Jl$2Tw8Emd*r>Lbx&U$x*gZ`xVq=u6#L2h51zaG?FY|! z7A=i6pFTBoGWD6EwXgha=+Zkc55*pOZ|LNy_rO;gi#`1Cxw}t4bngB|r-s(Oa(bw_ z?UkXE5B+TD;y2$Ly0qtA@GObl|L_CnPA+=rTnu$z`st~mX4H9U!z)8;QP;^u#B)pR z;;9GDJ<#&-x%{7<9(v#}pB}nD?S-NF-dBb$KKTC7{GU->H^=fppN~)T##7M$n?w06 zuMFLN>Xo6`yYCD=@G$W#k6nE5p>rp}bAQ1zL-W^seJJ_qmxtVyFAp_C=SzFu2G6n> z*=pU>e|hfyyi-H-A3|LPFAuHD`{~e62HzWc;N;*?a{h|g#oG^^n}7N*&t1G7brpPl z=wj-tL+hL`K%aMqE-iQ)Jh#TyI==)t_n*7?;K`x+o~MUmFaHdBy@K|=H#C34Z^5%N zw)V(ho{K$nlG7+Z*^)iH_UZd;^56YtIQib}Gs(_KP4W-3CF<9Jjg`94XBKOurVc(f z$r2bjZ<}Ws`>+HvOi3MzsWvPYhk4?!Y3>iqD?w^!Ror5${lB{^oPis#mmUD z&cbg#+*A?!Xj?h_FfelUo*$pk`*R}y{Lp5ZulOkWqvDGJ_~Lirt2cb{v*nACYxv^v zaeVQ($rs5-6@MHs`J>{$*Ye4Wv-?{wRuujG>OGd&C+*hQ)e3g>;WL#%}xd{$n4&w@*A**6#}VXEM9pJ9QvZ6Z_Y|$eF*7^nZU(U?ktpydT<}C;!uC z$Jk*f_s?{WeCL+HNbOYT)#gm+)u!id{U_cCj5L4SG7|e*S%1?nEh8uXzO27_qpd&o zYG9=4X&YomYnq?5j083f+D|+Jo&uD40kmV_Jy_OXn||=trcC6WMcyIgb%N(pYyXMw z0`CW|2Yws)8Q?bH+kyWD_-nwsfNueQ9{8_-KMy<`_&dNyfVToq1pYekmw-0{+klS& z9{^qh{6V0<=@9Tf;FZ8{0*8UWiR&(0HzU6Q`F{mm1AII1T;K599g{u2Ecn zgX>;gH^UxZK|2=Ux);}+Ac1xFjKMMoPi z6dv96qr#)RpDa9Dj{NvE*8eN4lPi&fqsgP zH<7NSBlIIZ-xO;H*Yi8j;oGEx=XB@?xc?sAr<@KwkNY3dz58_N72Ll>_sLw(!L&&F z{+*5y>GO_}=8cZ4q!)Yb_?-P)p=XVvQyz4h51r;ir}@xnzG^elg=|H-ko_>$`C7*H zIfS~RRF~^?=waMHK=((ewda;)bg`$+TqcIbk!$8B_e zPVN7I%|3Vbi?u^`VW&CJX&!W%2c70Yr+EfDy@^;#Zzr`cx1zops?UBp^m*Lxq?8F6EC zbXEUtA}=3h?373KK;~Lp-N})fyquGNm11H2*~v^^hwpXrnDC;_`ty9uiMLtcr`GDz zYZ)8yJJ{)vb<=6T!-DHZT+t@$23&2Yjr{9RueB@uw+sGtf`4rS{;Jb!T?&7t;9n#7 zS0~_Kb$YEw;ja+<<$}LV@Ux@y%h=C@(0|X;^8R%`HnIRQ)1J(-kp+mOE@G}%={2Bd z2>RRv^j)A&5%hx5^ktUpGbC@_bT*>OoLM%q&W|$Yx@I9iH$nbZ)RD*Yxhz#q82JlD z-AhN;z2^|0O9XvQ0{YiMUnb}q6VSf{`U*kcnt;9+^Z@9>2CClABY%}B z_xS|n{sr_ZLEn{t{uR*ICy@JgdB5A@8|Ho4mZ03Hpx-O#`xDSJD^PxdT>el3dI9M7 z3i`1G^o>+LT`vD*0{YWbUeLdtfc`v{pD36AWdi!%vJv;apqu?BihN7P==_I?KSBOi ziGR{){wIh(LHtAUY0 z#BZPedtl^OZw5v#{Isn9O~hYsy;;`(Ys6*0{+Qw3f;=|z_6KGCZ+~FvzwlE_|8Ee( zU7m*e<80(V>dX3v9;EVY!4BmdcH>Hlck`q+mlmi`aE%SJxjv?2D}3M=ZdpiXQ5yIu?Qu=Za{ zvWyHpNIF{j2XDeU`bb~|>tX-pX~2`&$RJ|L!FkAk26pfQ|0D1}g60K&26!IqynuLt z9|67zxWw9j@oN>ai<2!QgRioYi?i3}Uwi~SZz3i-Xrr?pelFx<*q2}}PsSQfef#OA zvoXrUUgqSuXG<2@XPTV2qWq_uerVXslYi*tvFqi}4aBH^;?J=Jc)Z)>PF%;;d%g$X zv%vSTH^Uk_9JImz?C?KFEN)BtxeKz~3(I1jCFQZS6%}0W|9B}d;)Cp8+5-L09-KLB zDJ#o&B?qv7nlYT0jJ%{3`6nu}&*XvUgl+n86~E3HUYk>vAA4^4nG?@VJF^!2O_cwQ zn`2EV^AmhdR7^c{nDP#04Ikz>>&yv`rwpIqc*>ac9kxyk~2PX`lsF(nGRb1Z5gT7%m@=-7LDt_p(R;_1npOx}QF5L#- zxydq;2fo4G)_zy+IvW0|-w#mAMqDVBljJbqstbK*LO-`^3l<2sw)*Tk&2!e_w0 zIY$0dj8D_tEwKr>7V`T$V_sYf`2E&cDy|Fp{kB*Nu4Koix!b9oBl-B8|LL^Yg)dEw zy>%oj_V%MwVjuqP+SvfVZOcNM`i-+1n>GoMbg_P_JZ?AUvM zKQs2*f1DBfkLRYxKKj?p*vA;x!x*O{n=SoUzQIO*S7GV@lR z*~s9dG=9tb2WJCgJ{mj%TnRi87<=5oF9B}@wgF=gJa_=O9{2->KEm}KT(MUk>_q-P z;C~0c4R{ZOFIS%#{3(N<;`$t}2djq%gUGMAV3n2PYEVc?S8wJBJ5TNX}mj4<_OI++Um- zw8{J6rgQSVfU%Bq#nl-5tFQ)DU_L0vyikVu`FX>fbYs4_F<;!6FK)~iS5hyJU6@6PX8n2!}c$;r&hEEalza^`=^nEtQA#|7p+xCZb^!e;{Wk(g0wPh+^P zv89t$gOW>BD|MLbHp-*U}Zm$brfvJkIOgy zugX{J9_9?NWy}GMrV?jCXH#caxGPkLJLTHbDr0^Pb~d#&NUWf(y{oaHZEr!>p60e) z?HaD{7I<%gdQ;!t+7aqv1zn9&SApELV10L2OQ^Fm*jf^7m4dq)TlWS9WlLjQXM1=< zsH1aV^ZxxUjVj(Oq8x_jp6#wL^q?`cw5YU$uq^&$vaS=@fqkt zr8!2Q<+-1kA*}Q*Z_rpBEEX00T&$Gql;a2yiCp_~BMi9_hWvL63rD+tZZL=#6rp^m zC0G~)s_$+H1v^`d3Z-CI{qDxL!qVXG?ohZP9BOU}c44gx*0;9=>lzWg1{)f&{7hRw5ZYTngp)SZlMcu7!o#x7Jgie2UI}XECOM$I9 zmvn9dX%bD%QN#iklCj4W|I+w4CQV&Drd8JU?|*{{>< zDvR79lL`e#u1(H=R^-1X@;y++#!obiqE_&Hw=olFJMzGy5`R$Yt&R44FvsRtP@2r$qHcqLDmvHLfVC%_NEsR+rN;%44&* zj?+b6lejANDt$%SB>%sic6VcEv5Yger;^H;QoJK#%lqQf&f zM_Y=|F3RfYk}%#Yu+IAF)Ty~8M1$Mk)!yFHxzLZ0=Wdpf%sTm7xwon8Kh#$C;Ajs7j0Hm%Q!vfCVz>4 zQ7ByBRn@k!pR?I%MvC9*fUEm^i*?I1`K#UJ7d84>{$cOVtv<0Rx`O3W3X1`(`n}0Z(B*1D;G%ijF^%UAF4+ zBzE3_GuUeeJdwQ!to3&~J7b_vU?&XN$BrBLy{ykbPh$}SPGubi>|wPAoWiON*v+bd zHGPs<09f-k7h7h)NvzPo?__>p&3+E%Ghocc25e&y3%hKwD*leS!hkI-!t``LNQi%v zosUUN89JaEg|H^srp1|cy^Yy|HjBLo<7pFP0em{}vF8WWxt-?4KP1`n0@S-KfJWdw zNEX4zk3}~E9efpx4Xk3Ua1CR}D}nK{WMmz*U61-V0B=M&=xJRSVfN*hEw5oRj8kKt z#f`9@MVEr0>f&Yf0@9rD#t#7{Q?IBMz3)C6oCT*R|#A#uwUR>fvW}X5I9fZ zh``$g?i09H;Nt=}3H+?U9STNwDD8Elz13+lruK1+Wtr&I9*)yB{W-=ON_5hf<4hTo zeuQhM$e8qT0vDoRh@Y^9>Jymsb6T9N;})3?7GOK8Mc?T4>l0YfZ$My0zsmxXeW`vw z{Li4Dz>0phWnN~V?q|=!X9{794t#vMxW}g!p8@ztHDua-AdpKyFZ$SL+DS!b%Nn-X z-7W^RVaCM$Qjxz0kD6NX*-hNg0g0}kitANz{RGz~G7Cf3@O4XB!+%T#kd%BSmRKcc ztN8+|Pr(#^CI9#s{0iNCjoxF4vp0}S3-JXn%jyxKP7ZKE8hfrMPcCv8m%20-mB1)B zo3U5CpbIbu^e5BU(XcCzM8yXl_Ocfjvp#9Qi~2i$KJMl6B^HPInJp8BxRA{rm-6Fe z5_={(@cB*brLeM^!^+U-aVaIA;+41)8ousj4~1DW8MmW)D23<`Ni3Oc&{2Ikg}vrw zmEFDKTY-@jcFD_5?Q_OiGZ@?`Ka2Et@IFNjmmif7dy~SHATQdRra%|r{vIXlumewE zk4ipg1guJX9+lFaBv8X1XGDklBqy5dWAR_1{V!5`lbk73p4!9P=X6s06^#0PfJjsy zUIy#IHwBFt&Phq8Ki?@M{(nar`*wHS)Q!XW*He&j%UdWXObjGgU?r!m6Rix(E7|GU6SN$&r} z5+90$ZDj8$J)9{22Y;%(f4#^69sUxPk|0PV_$T>GK*e5G)aBrVn>#J!S;5cigB!88 zyzqZ#9{D%-e~gbeyB$24EgP#+?Me2_W@dXAbun%#*gSWkDz{@-X{Aw zH2acFV)Ta}(|E9@#r?mF*wx&4pq-asP};ymfL8pPy(rUzXPh?*EfqG`(~ASbzEaNEM3yIvV!o z=K;iBiu^RiVP*Nfsmgv@)HlzhpVvzoX!^}H=!f;t7a_X5-dp;l47>)dV%TRgCKjH! z{5~nu9RXDA(I-uDv9Idv)0@R6>G-3)Q+fP_EKz=%m;0}XFZ!ezE?T}cdDC4q!6-P| zn{BFprd#z_^yfcxGanXEywNYqk4g&+?T_}(H?+4;^1IkyYxPBYbJz@nzOzjF&i2B6 zHGQNagM8GV?$Onc{(}Q({nsbWanbUtwWrXePp*p%=;fCfYq)sQnm$ zqWu>5>q7LOMvwL`HrcPhtsjr1KOcbHem3&IxqL$*`YYNy&s2V{JEEH}sQ&3{{SMR* z|9L{GAM0`NB!&?~N%(7~$-a}lG+pW1lkQjLSxBGxj0~vyPnL_q6`7ynRrntqLMb2Z zoo3Q^s#`xlk-oXAe3}p8Z^xDLG#|`hS){G1?@W`v*K5L?=1`4psts+ z{_yn`FWQNCoUcDFL;PKU^<7PvpN!>cy?R%S?+B0A@5B1(LVSVNAYLy(yqp}zawNyc zkbkxt`81zRz!m$Em%^khCE71gJayhx#&KT?`@f4(-VrDJdT4)v`X86<9J>ULN*;74 zwEzk9sFcbA0h+O7d`xmX1`eQpu^t|i?Dn`0M|&NH`4j7>lLADI-;Kj}^w%e1zLip# zOP0>-do=8^Q$VluSD&;XiPk3tQ+>wrh(~q$Ju0Pn^z#Y!Cmy{&9h1^AfyLv_VYz-o zJoA#5{RaCRyQ~lFW8v6S1qHZ2$T$sqVm=Y4NewoyC=3g(${h2144E<^2{poAsCRA~Hhroz$c>l{|RH1vc z{K$mD>Lc z|Lgmo=0i2!$NYa>k5iKTXpTbvQ_TN=cH@7*(4S7Nnq}u;y zT+6xm>l^_2H z*rlFNUgvsHjTSRC9()mkDCy$Cr}X`g^OcAmXS&#ZdW`+#*EKvk9!%!`kSzS6Pnv9q zCm)wGIXzRQPfW^#Hj2NR6%<8ug{CFj*?0E{^po870+k%^6<1@7r*vNi=5S9RqRLmF*R{{sm#x!RHcwNJy|~< zqS6eLyaMNTuxj$;XjUmt@pvJNHYxa+G&9*aezFbnU?0EH9ytbk_eqPLwGgJ(hw}+^ z7$uB18sD>E3w`@%8{}hrPUPo{eE&rD#r$DK6Gf2aEWTwNO9RDYq9jH9(@3FGIXqCL^xQbYTqy-N-BV^VIO zzQ3X0;#s~7(jWR-a9**@z#ql=r`{iEzfb#DW&D}PQ=e4i>@etelR=+j(%d|wf6Oz; z?~{t1lCC_?my)Kan9fs{o8;x?8T}*QAP@Zu^XSL#sPSzYuagY^1N#)_rY%wJ(}(AY z5mg_YNA%7%>62+1pOX#xaQ``}^?#o<(|K4YkH)vYe`tK_?R89=ZX6Fdf1ZW%NbqN} z&m4n3+}|G3>O*^q=6aT@?J@fs=*4;NWrA*lzhyi7AYAb`oToY;(_!*Y+5gBt9sE4V z>>v913-Z%5eY1r8BpNR>r=;ZyoDBWvn%XnZ(4J^7z1_zX+PfJ1`35@qi?+Ymgz;_c z?>;HVc|xauw0DuIe7f_vNG2YGnXYLQOC&gDt+&|3zME2C^i=m&< zzqtSOY5K|ir}UQv{KoSZ@|PD;nc^?z`GWe#uhKVB|L06eQ|yN$iUVxcRF7gW2jm<3 z5A>PS-3wL!A%A>Ll}GkUcMhmH8|97tVYENBS3f_HzZm=1Y%imIj==x)@UlRzJ3 z?;Op4les?*>hz`lB>$27o6h^vo#$0d_Pv(<$=>wluQER$0XEv7&d+u0HJ(?^a$Z*D zlf7~gfAGZDbF#l4oBNOaa~SN3f0Dh8{6_yb+grA;U-m!5=j4y7ePs;)75^H`X&kfJ(6FgtO*vaPOIGfJ6@q9G#`9ICa9d(ye*vnovo!$>Za$UY`V+rJy zN-kRjZznK%-{6AR?va=+0{)I#@Kd~Z&$2jcu+aNDjNaEdwa*>rKL9}Q@8EgA`FTR7 z#9WcXTt7zd8v!%lJ_Y0b2Z_b`3lH?Z5za@`@qPtw&j9KNo)BgTxWNDV5qkf@Ga>Ad z@gT>LXJ@MXx4iaDyw9Vf}Q zm){PP-?u0Yb9Exz9w?tHIjmeC**?q~w$i&7jEVP4GV#8N+CK1m!pwyQ1MfeD?PR|! z$;#iavXVaBo{-;ydlI@U4f?pPsy@&!tH(;x8G{Yk+i6t4+mBoRJ_z=YrN^Kzu;?#V z8n(N*J(d1b>+i7B`vK{aGkE~qy#3G*_U7$X@Px3_i2#PhUsLSsFxWF4&rd#4><9fz zC8wL-W>Ng5GzyIJ9FuQZ{BJSZ zZ?}@YomTYE^G6^**-iQ?@SHy$j{TM|M0ve zSDNkc;g-|EKS!E1o5s&F;(sZd-P_}t#dj<-W_s9(jljNHZYn@{Fbz0k7VjX!d0yZt zv)qM(KFh*rJe>u^`ye!)?CDa8jT*@9kN2GpNwW2@CE@l#`|*I7k6#=0-@ zJTyL@k{pPC@&1i?9|7&h1`X>A@BdAZp8|{yB7aAFbEV`|jZXDX2iDWw*f1*Py=*U@ zf00Hh&WI>K1D4Y0=~6cM_2p;68VdagmG_{30@LkE`J4$Fo&3)adHV9R@UDbXKAV?^ zy#v{5`8ik)6grikjr#QE=VCok=@zOV&-(*@wR}EaS5WCFp9_8T<>zC&r_rhWOq8Fk zmS2SP0F9n5&4a%B@`Y3MbSj^N^0{jHqAWcq9vVRDAKIjWodI3NG(953>`eKz{ ziuM}I<9Ty+%5QL!eZp--PxGy3ikQTm*_$`psyszWgnoC7|C#bned{8_kfi zzhgY({TP~WF&|?+r1su&%Tn;)tjhCvXgKorU_AMyv|Hw^O~;$g0lq%uN}g2QF%O?# z$iL!gaXe6t4@Bkq4tq*y{@`@fpCzTnVIY=9>lftBkyh~MyL`R?JzdI3rSZe-!+6E~ zu|l0c>hL4((sEh}kT5>6ew7|uo*~Zw@_5DkvO=9-@U;%iFDumfrO83_%L;XVsY}w$ zmqUELrt{h3G=b6i9Q3J@mM{O5*Y_#MR%sbfB+l2bf5!Q3Rt6(>ot`4)6;pFLj{pyJD^)^?Uj_0?Uf2<78$;0?yaY6ToXUx!*cV$U= zae6-}NnW3Q{;=P#qeFg9F2=L2e15);?kbgXr>p%5d;7x+7wYL4?|Jl|Irj%nx7&^M z>CJ>4I6yNx(I+lXh*G~(H_jd<29Bc3a7NUcA_d z7cDa4f&wEhEHvWMQX?)ZGUDQ5BVMw^h;O>dh?g$aVf5DwKlP{9-?!YXr(^!lRqc)W zXJy#G5c8kPpDX3__dW7Nys$F7r~vbuj-E7~#>=fT-M2DaRE+scN6*dG(II~+%||-A zlfO@yugag1P4mMH@=uJfmEq|rR6dz7*Kg)*tVb+IrDxBZ1w2RK3B5^4G#=wVyidZ* z`xjuoXMUAGXEEk)wumtN?{Fu>c}WC;2{JDvqrO?CG(Q#*o!2+}Cd{{NsVYA=S1;e! z>*V9#LC^az|3O}^Uy(PMOxU@cJtYqbyuW$v{SM71Ze9H1b*4)*^zn-&SDI;xU$RZ{ zi_;Xp*m*oULyccj5HIN0?^IL#l4gi+_J-|tTK}_Y{eZo&KA7ScuPJ^>*2OQh{@Cq^ z&jP7x`RS(k#Rq+~`5;%K#~GS_R2~WjylQ#BDSnw?ieF}#;t#t$1?7Ee`CLPMkuIeh z;ul&kIw+tN>jl-14hT%3^6-arNq*kJ=P&pd(J8)5Cps|JJ1rg&bezWpCaUyFrg+3b zFT4jbRQ>Gw_zUrAjuxMA`)3;Buf5n0`cZz8s^4TSUg7mS@V*WU zRUJH@*c+bfKm~zmM92QbFU^~)#w);xPv)ud3EG>5^CF5z=4tzB2d&4vzrbHA&ExOe zYw^ZBPFMC%Snubm>pkL?D#0Yl9PoHkY6DJz+%$Y`@vK=!JbShg=jIymoH;s-^(EPh`ApUi<7Z`f;zWL4udYusvuSRi6PT~BoSrg) zmcofD-8F;OyO}EXWKg^?i7?kcH3t(o%T(!UvuQm}Q?Yjr#y9f{oR0GeI*(G~zslrg zK$H+~_yOlTPSVdN$B&R#lI~aN zXdn2KBEX9Lbjjy4^4m+OJ$89NLi#uf^Z5QPuP+z;DyH=w?=R2;eU}IOEn4sK{Wy;i zdovRVt3^6*D0&l^nTpUTE00>k`y1``^s>0z|MK@^vEK8xb<};t_v5fn zD%r;>%G3D*`3K&Q=jRjJ`Teks;;s97rdSynj3zDUJ0U#DL-vWkg!5b2i{$hA>+qWy zKPLX^;QzkNUzf(|q<=@<6;c1>NemN|&>!dFnUmliSkmS8+}DHG+51F&Uq|^cqZeeT zeN^6lUyl!TAwQ3wNAUBE4tl8>&l^gkkOzK69_dH(p?tmzBZjBY*BOx_*q>wrXE_jn z2~6~9NeG~Hc&d};dzqi;Id)o4keqa0?85geY5ynVBP=I8IWEgX4(t=X2gjSWun;W@ zu%9bD(@E?E?kf>C}l_eIJ9JtyqMpDq)L<0)}jf0+*bF*Q=6_H434f0yj9 zT)t&xaT1T$;*=?upOVA*W$X&O=)e|<%6n0MLNWq=N?gC3@T9ohe&ld{lj#5#iSx6S zv#eYnO}^79+9UX9;O!Hh6nT?G`#C*RO1Jv<;utiOSRSB0iov z>foQ?Uum_*5%6O_ivM_d7ba+tfahSmkUiz|cTTwn2AJ!ZL*MPIQ`oI>`Td{Ym`!i&7r)?7}~|eutJbj~8zs z$>nXlJk6KN_{_oii?P35F3}&n{K|6q{TSX}=xelJHuSgh^8}^-mFe>Opx7sRPgD{40wc-2R$=S;{2nG&oaY|{R_@hmsf#GNnUSOhO;10>7Q3I|4fLp zFeK8Tn6=V!Y9HpK@8X`Gr{eoLh!3)8_Eq+8s2|}K>Z_L5i!8~*zq*}4=aq;r)b)Ze zes3ZDjB)>p_IgSG_`o4rAFSw)3AA>|=h@jP@8zzMCC;;xx;$~X4^qTI_CWn!wSM4I zJWt^B8ShU#uk97zZ}J`Hzb`SNcLKdI%ThDEZhk%hee~@)gm{sB&z*(wis#qH_{GbN z@2gUOcqXg;apA}O`$%ek>E{Do|M2lo51vJT@$sCwU&XZkcVzYwW|jH6{=)OP=sjLt z|D;)0h9^S-t^W-D^(6IIJYBZ$ed0XCH(71(^S4LorWFQ(}gY>*XXYWgVzvt1`i)T#ZV`if9f%qiR_>iAh zj5|IaGoDAG|0c@)$Dj9%IX)(h9v}R?RBB)ANV8e;gtOw_8@^_TrS zZ33-l*KcpPYVSn;z1HmC-c)Tpzs+i#|I;kjVQ;Uh?{(Q*cRnrK8-K=>#*6$s6Y-a6 z{+I3TRqc)A^8pl~c=D|y#{T5?cFW&KQRf#Az7Ik3qhjw=H_g8(8D3#;ucp6jZ^T=X z-^1Q1F4f)_>{eQ@4fDwkv%OPMK$)-4n&)%)t6Q6I6Z?0n(cUS1K6cZ5oY22h5w2_J zyYO%R{cFwMZtmZ1vKRI@S@``9T0c`&d*k?Y02$ogrt!n=?ZOvokW_oST}FGS@b`_9 zRePsu`pfoqr{b5?uHV1$`mr*fJZT+0pER58oq__2z2AVn@%USfZwk;}*EsgJr)c)3 z^-TVLxl6M*UihN;N10Dj@OZ;u?P&i zWYyj-en6x7cQXBe6rL9={+*)fFWcLNZ~R!V*WQ>fc_`SC*}>zH`wt{pRr}X5qy6do z*U4u4C!>I3{{zPP8vEPmJxN-BVE)dMGPM3cd+~l;veEvNxWDuGQucS4itLa5KF#Nc zld&B$*q=XNP~z!1ct4cvpTX_F81>0p6MTPwkDsJ2#JdAfn6`1KpW3g==k~W}kUwem zciIWZeNp~#)=r^_&k-ceup#Qxo;PkLfet(9}`>-Fwc@D=N zHT*mWzn^8}`yIXkhdr}ECvkBa6t9r~-9->dZXefY3Lxo&@0XK&qRRUhSw3MdAO2IB z93Y4!mwzeDkH3Mj--104;eAk^ED!VjyZSyPWx}7+<@YmLyaf8gU;YX2^T{P}><9eg zPCi@8GFot6i}M3+&l2Ul*3Jl{eK=oV4E`hzc>(Od-t39XbZQaK6Rf>9+z5W`pKzYX zf55B5kN0aapVN6DKTjliuPx#JPJd&^t_Ce6%md_)p^C-#fw@Oy#s|KV4 z57>8Ak7C~eypQu+OJ$GJ9*v)VGjl9{gTKI@=HK@){9Xt8tFqhp{lChd4EB8tZ~R^v ze;;V{??>qQZ^QX&%V>VX?+xhr@p~XW=STA!zMpR9-@*pW`uLyKeV^6L|KezV!}nj! z{43axb@J~*d?7nTA*0_LE9$agw*h~qMF!)iy64N*Q=I=fZXYv0jUS$W{_c12dw=)A)i?h3jeq;#iG#PqzscT;e=#r=FZYie$m#p!tgq_R1Act^ ze|gX6?s)aLci!^!&kmgb<`d3OYrb?--_}q5_L)zHKYfEo&GGoF@pw0WapW&19s5|$ z`2d$sZPxF{4_ZyJZ*!Rb#HUtxi#;C}3faqR2e44Zv9 z5xuMde?FnIMfkOxUsi8-)5p5;zT#olP)qzjj1R==D4j)Q9y(b_NEePt$24V?d)*|z zY!zFfjg2x3oq|}_dQfLT7KZ;Vl?-0`Egd%Mbce-KNu#Tb8p4yMa%ZMI8c`ZKX3Sfr zjyRi>Mw}(hwdx9vLr6f>Vs#p@&4{h|GN^%W!d8cgPFSnYPW4%*jct7f>zB_d&oGW~ zt9{fAFnj9UD|l<-4pQ10u(GJtYS#@^tDbHP*jUsSHP8(@;Dt{&3n*{7@DZvn?+7?p z)G@YvXTZs#&TGhzko*BfJ|Dq)`ALB!7EKykzANCu4;EcZzMJIpw>n@BKEn0#lhL$j z^4Rj__FY3h&XZvu{*({$`3#_!FSmbe`EvWNAwNR$2Nd~yhS1AT38b)S%GmM~*vC@G zFp{)XHtmwzI_&~Rec9hpF2;tH+N2H~^1D2bv&`eM(^cnxSV62d#s3t3jZQ0+Mu&Uz zR6ZYn#iG6eI*4~2ky~EBq}cqqmQLh{e7u_FKsM<(>XSe|W{6}fw9uBYYt=}e`KPUs zeDzOTBYE$iwnp;jgf*J`Q0dp}?8o!8h9&f7S|bzINHr+6C|9!suQQ~K$r^c_*BOWXLGCS#MP1EFQ zwt_7p_NaqokgZ%Qk4{DswYM)Lw zt0q&cGvHEnizcE+T&hmG`n2+qBu$?bqdugI+24LIyA33?n^rf|bW5l^+BTN0ejnZN zOC*Y&wYt%6#s3r9RcE(^x}j|ebQ?W$mr+z&zYww77p=4oMce~%tIwJtYh+=TVNy=f zk1s>4+*9tf&Pu~C@YEo~mKNaezRNWFAMcLqcVO1#6c3u#PLfgP!Jy-E%2CR~%yJ63 z9AzhXFQc6vm7$MPUcqJSt!^c+JpJ9d>Ie0x_xk*C?nhfHq4aU2B&-=4jh2p_wQfV4 zPSJ%f(zarfK$JU(=it74+9l8~QlDskwocQ}-Tb?Ofc6kQBmM^nA=|ekOUR#XVPz%)-P->$*ogewf@*~X$xG!1EK0vNz zU6xktV}AZ}wAF@IN39VhCgSygKE+5^cV2i2oN2y#&}NoT!5YbD#`2o{+JyBFsO=n( z=`LgWfT~}gOgHJL+3B)O_nXSA_V=S#pDE}TRp zwm1IfC9`Iwrw#MJ|6#KY#PMPdn?h!0Y=HldF<5OUov^Ev`^3)sB%ta9w$UE0Y4yWV zz3_)=g{6_w0RMk1tp@18g{RtUYxQ4cD>0G=#GzQ19m8{hO zhOeA3u_fxkaYnwn8vk(AOw7lqHZusVL~R4|)$Hw29AlBKRIQFD(n`0=;jh|6SXrGF zKcA+?Ag)AfhxsH;J0ZRo6Brhw1+-VES=cg6bG2oG3ujsaW6CV6ckxp<#(p%& zU9^;DM6(Xp+66PdI*)x;K>PA*mFMR>*EnmCPhiKEPdn6W$*28fHU8*gqQ4Js>81R! z{yzK?SRzq6{zW<0PSdo#5S`$;m;S1|-G86TPI6ZWME8_z%<@P5o%jwRPpQ9Jt&QHP z`>(I>ypVrcv^Suu?~nQ0>ZAT{`c6N4;BRI<^{v0jXl4H}|AUXW+x~9e!kogN%zWX9 z*R!_8y%=V`@we5v^>6%b^)rUQJ$~bFt4IFezpZ{83(9$Xd{Kkb54VXFrGgyBQ0#;)4lnAdG7_uKRGsSKQ9yYi+d$@Er*>BXAH4+`0(= z+v>DK<}Wg0_NKqBUM+B;z#Rh9>O%BBftLyVtiZIo5dAfQ0|H+bm{u2}`><3}eN_S% z3QVgDuV3J5fjb1I)rHqD@OFWp6}VR5*92}7__Dwq0{ieKWu?9Jx5!_^m&^!L`v}vQ z5D8Oz2v<*+G3igZFI&c>FE2k!#-tzN!nrageduqi51?O29%1_1>X!v3{pfG2SBGRe zSb*)U_MpH*-hjZ0ezf%I^@|9s=yzUVMZW;n6yjI(Q~tKPAHiUCI{XHoz|GJPpJ(wo ze*pQYbF9CuPI{8x(lycFO`ndklw8ch=*bfK8l{(`@>R*dEbu3&j}q2wsOib1!Pl9a+LfNWAH2V-}j1x z8MIB^+o{Eu^L$|xHJ35C6ZrWiT9?b%w=nMTSK$nK1YSVC75qwjmDncNCg&^m35b&P zB{`e&r_DtkT~_dX`i7=4ZAX4OzDlg5!b9?1CREQ+GI15%$QC2te|)r_)yQ9@%lC`2 zU1P}6iT;Cbse569Q7IsD>c!?sVG!=1K<>MV%D*7W-Kjlr<7Ld{>9-TmHKUT>CuiX+ z@jMX^y+W?2qgZY`3~H811rCd@s4@{vkdfaB6%A^q6{_DWJl__iMQyFm+LHtB1PZ%2E z0dK#733<1{a8y5%fpp@CoLMc1GE#XJ$jHAACa8b7Pi9g4pX#S%G9POcG9+EFF{h`2j!}iCq4eqa&TG3)E}WSR;r9w<*KibbXCTMv6O~tg@fqJ@vIJwM}Mb` zFXgI~+g-MKYvu0xGC5UjC@*g5t}iKX*-^6j&b!N6wuH7dbnn<+Q&iu&?an==ceIDf zDmGMXEpMuCUDLI5dreDy$L1|9Yxk8ml-9JDt=+e2dy$T|r>mo`bu-G8&(%@ENZBfn(9k7bnhtHTDWs_X<_5`a@5fX{+)O3+*Z23uH<&AuV_c> z?$vjfZ7iuT4&50lYgOwFl@+zGD&BoZsC=W~S+k_2a`(EXj+)IaRmFGSUb%Z?sBC%r z>dM_)8n&%#FJHTFRb$o0)^T{s8+X>USL_I(jWx?#Dw^6?ZQETPDlczeU9z<~RJM6T zc}df*hT`QtJF7O0qtDL5(6*wM`nK`-I=UOS742%MTpp%&C)984zU^eEy2>^C>w8L@ zcWmF-vc0%we_e&%Zd=#YwcW9Q1NCE-&~@9Ip6w+KiDg#q-gkEyFH^j6omQrz5@m{a zcJA0VX1OK8o>e=xuiFLx=?=kX64`7+o6w_eV_|*k)~33IwyoH(rgCG^&dSx@J6c=1 zcUIy9zt~x|uA`xKYY%*K?Hxs1_cv6nYuT}_SmzU4_k^~U?g}N8sr6UIwwjK*mgR*z zwyo>jIUa8-=WP#dtJ$-2+tP9RSFWGNOQN=`{vy{up5Lvbc5g4)xpznLn&yN)P`SHD z*nJJ>Ep8k~zXom}*nO=2SH^{0_qF?6sk^Gab@{%XTbJKmKVG}r*6ppY+&XUml$XeS zJK8pn)1I9@b(qh#msF0!*CP1J_tq8njKkX@csJg?W1G}6hTT;=iMGS8MdR6Yb$u1a z#u)y#Ng1Ep%Q4nkq%rDM#@QN@724LFxGk%i#Qe8&UwzB+a0BM)g!81CZ>)LA$k$m{ zJjVQ`@-=a;eB>9~#%aq|x$ZH1QI*H-+*LA; zoh!MWx39TAzHQeluY4?DQTkWZTXS7}JFBiw-p)$t`s7tMT_0c7_34M0-1x!rGBindlllQ!kzH>HH91Yi}(ZcQqxX3rntMD*1N5R z4cpgsZ!cccUDt|rWjof4VyqRb*6rWXyyR9TRtvVaG>3zG8VU=S)rA`BmhD>7SX^IM zvZOAwymVJ_-IBt(!li{pOM)_Aee_Yu!)=W%iP|72a#J>hIy(0?@893jsN&5s?kp-^Ru?P@HgxrLG}d!N z2!qsxpi)C;3kc0^yYFmn>sl7jhM%WCLX*DeJcnp;V5 zkUlD>NrPbFtegG!eBlddOHn})Z;`UOvFxwGQ_h)|>G;*)DWK3tNx8}mt_$(eKVN-} zHZYX7cxD71v3}Y9|2n_oderko+TX~MiNAp}$p1Yw4nOTnzpUfWpcf4&b$Djfs`A~B zc^*&us_(DT|2pG|%)go9sG90nJDszUZvV{hp~q}wTX_tUPsR}l_$OE&#@^9f2>wcwby7DIH;WXD;J#qddtbeNB=(n z9qjBs_<6gTYD;ipD7RDTG_3?R#iBb`4m>H-&Z1p-5}`6G`!>q3@tvtk;x~u6XvG*t zTB37nnV;`ejm$tcUA+oX=GXKw=Iah|1VQGf6DebQ5HMY5AEjN(M7}bPWP368f6y)^ zPcP{%ZhcP&lzLr^9vk^5I>!7 zQBua|$}#vGz>`RS%1e|_&-X}vpCBsZ_lw}Ce0s)2Nhw>5d&;jB`KvgUv8_T5J&&TK z^vgM=y&_*3r~fS4MbGmnDdX&tkQ10}%zqI16Onx_|2>gkCFVK1$XD7&KN_j{8j&&teUd4^P~=ZVzEYO5?h^UakT06db9#08qRBkx zgvh6Qg{nM9Ij^UB_4!+12TCMg(Ty|8mM&kx;U(f%VE?+O-C7g-m==tRz zf0X@4KYQm2{srLI=dTw|Z?^w-k#DyDT_T_Al_Lk)KdQtap>d>~O%tU>JlFQ-u|GFt(nWufIkCGxsnU4!i`O18(oPR6)=SBNFO#I4xe8QBk z%*RX)x;uk)&26D@4?|qu5LzGVZmVy)od@cxCH$=sRuFD%2^DlTN?q)FcPu0**xc6K z#rR!AbEu`UzAK1*OJlez*x6lQ-`Ls73hMCR?Zg366|F6!(G)$ftH*1%Sq8@tN7y28zM-Cd28 zEAykEA-5s4s&Q{~eWM{?wrpcla923g+ISr*?m~TC&@tFX{Oi|m-LOhF?S}S-?iN&k zojXe2D=qDzhG18Fuz@QkHwfy0(=SN|4#e*J>b9L zF+zdd(*^aNT}4aP*-MP_1P3F6qPnSROW*NlaHt^9{D0j4-^IEkLN0?K-#^Rniz3cZ zh+;WVAU~I&Jbg?+!~p$=)T9qCfR%NT6Y<|Xd?+Lf;G<82ugGYbh}S4grzKK}0hCzm zTcA_@Vi}VQ;s0kvz2^E9tX%0mDk@L1^=UwqSN7KmO{rhO5!_JTC>#Dj<}mY{@vEZ# zvC6Mk&S7M6qwKc`%9H-)wEidN92uK=4~g>T_E(GYv`^J1v;Oyjo~Zx)|649~Srn#b hP$;UDSNiWPDmK?mMe=@1|17aRZp literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd3364a16830f1a3bead73b2be4199b92.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd3364a16830f1a3bead73b2be4199b92.o new file mode 100644 index 0000000000000000000000000000000000000000..13fd68ccf230612667a974e1f9469cc3f0ca26c7 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10HeenSLN?6Av#l3GG9V!kug$FJmKr3`qfbjR zEkjO($U4SgFk|n;J`-n2;>o&Y$T&NhaWtE=kvBfXaWI25*{n|{=Y+@%BLVi{j@N<2 zxcB?((S7e#g8+MC!&1YmG>eG0F`bB})~ z(tqq|z=$;gHMCnKZ#--|r4;N8gmZD%7#PrMsRoBd#T zbjN|AC!TzAD1E^L;km8fjhvf*G_vo#pGBTgP|wqx%mIJ+a`)q40@gk#l&p zy!U2g-tps+D(8EVa|_ObW?6V{+harh&_&hRqv-osBz){dq}q8h5?*j2^2BQ5d1*L& z?7+~xYm&m zq$c;)rM{XQB0G9)tk?k?X?ky+4?ZNo-*B$5R1I4v*Z!4*%;tp#5(t z;nN%TJVbL1JRiQY>s8ZS66?E==REi)Sw_GsjrlIg1Q{X?fa=Z&U zjPjEVqpu`~Mr1YJA45;8GR`w%(UKwaygji@?4Ccs<8oDH$JZUQJ7%AMa&vE z_JJqTRdX_~X#a7|SbZ7$W7kWc?+MfRL_bUO4Dd115|42uW9FrgYc%d-u58ccwE3QJ z`jV2c^9!Y6vJKgU>_q*&1HGPs{#=Nk!Bo${4)p&`T-V@Ljr(3)58!$Xum6SnleqpH zjSqXh^K??!2D>W%vrS}|-+}&Jv-PkG_Bju~B|H5FHXxgygUwGDv4LUOkL;j)`u(+R zAX3H#m_GOr*;Lv8;E2m6Xy; zWWPK5Fze%4I1l`ozv%5o>$UIJ|AWR{>|!H2LD89}QGgH0I>A=p$PU z(XeC64u7!S-`dG)fypPVj+v*Y)h3y?({)O$2 zEbQ9R(!RALUT^G{^+y)QZW=q<0^TmRu*)CnS|~(oXzcE4^>%hPcw1YydApkZ!G=a} zXIF#7cKO>oJA#3(U`x^BZEgNGZ)?l8_QD4`3LE}?F&Lnsuy}Dp;WE4m3JQuAH{jpm zB?S)_KCrZ*p#X9Uxo!h7I-9*gf79(FNYLNu4?Y5kV5OkH^|tUf$l<*$1Znd7y0ItAZ;X2woyS(jP1-F6wHhKeX{$S&6 zVP=?y%VM=92g;?B8_)>24Q!~yG5ais-ZoQM*_I1*H3Q3_VqA+=yH1fu$V}wgmpf_5 zoiyZDOA8aY{+%@BbC$iEmMyZ56>$A&$gRJRQH%6iY_Jw~Y~9-F?Q4iTvfZ%E_)3ndSm%!*1DHmT`Ep70mve!zYe`Z^UNZ#&_&y z)Z-kElR`^*0q`tWyr14YLEr-`vvBK?^dRG97vSzA88Ns7v#(2fHMFAt9^f{dUsy{N zC0_#Ro(lNhP1lU{J-xkim?OHUcaDSKGDh(E)=p$y@ofJ(0HUq^6di(46|ZOTqWbFf zRpnJB?lSEEagHJi9g+~~19)vEP@pRWUo7ix^1A7Cp{3Q^)zZ=K?&|1h?VRt%%=UXb z{qDk|g-hnU*VAQT@iJE0-O}3R?(FI8^0&F$x;wkvP5!Md?SA*Vnwphs8`f9Ut*xqA z-B7V{UB&A1igLD~#$DuI;0-o5yF1%(9`K;ojiHHGI$y!(9~NGBu*U$mFLE&QgaW$H z6!*Ag_N@5*^&IEen$wwms~vVh+XwBU@4IMp1`ou^(8Ww_U)9iE8xlPHgRP1vwK8)f@g{`u1TMgc3q_Y6Jcfk>B z@v^%>1B2l|f7xdWFSF&WWA-!$Bk%6D;r;;$-+`m8HoUe0fkNZZ^<#OxBCj9eT22TQ z@{6&vjj_I6T9o}7NHWlH2FvLY`r?x$&&iJNOpfvcxHM4soNOT-(y0Ra^dL*8!2`7y zZS0r1tZ%PPngV%?1 z?ZqJ+`~lAx;UQmWax@pUq^Zz0#=9?+qQ~fuXRyrR+IAhdyj1PN4P!&VN4SGth&d3~4X$gZ_w~>o2QIBYsfVNxndB#76w4 zgr>lX+`dT)FX$tT@UEb93h|qMI-0_UGFb6npnuf4yl?RSoEfMEeCQATKfaGe$sTqG z>^WS+?%qYTR3P7l4EB|tiBWD2!sk1oAAREb`uchv9=s!O+bt?JW zM6Q2PXQy7$m3}I7zAnuRd!m2X!^xeT>;Zm?gKY8?KE9~;?MjNW4*(z^fd6F$ZBg>b zXpz8I^l0oNF`x=BuT3TShH_cubI$-v1>>J1`g>SS;D z2j+7=pRY8Zf0@BbpVI_6`7iL5&;49B7xl6Ind#xL`D9P-uNaT~YYUOc`H&TwJ)7bm z_s3sQ|0_)W_wAY)_ov~5Sl^OtU*tbClW4q|+U`8-zt5-r<9|FcQm%0Ec| z{1Yj{W4y}emYVwO+m#j9x3ArzZ<^l*ee?O6#r2&f^#}jYis{>zi}k`~(YMPL*LMx+ z%N3vBe14$4)HnLi3PD>CAUfD5)?b<*+<;PFIc9mOzu0;(T*4}I=9uIgPhWkzkPnt$aPk{96J0i6Z8oH6D_EPZ5D!zl%c?CzAd7_=p#ksrkI7;z8doEd45e1xP+U zE?=G#@lt$NdKc$6LocTI6R^Q#w0_h4(RsYo36C!~{DtE0WzZphzNG3j>LBXfbR!G! z2YZ-KyPo8dNfRMI8!Nx0jjvBKKS2J5`LlrdQ3ds*J7K>^0ib*U{zLKiWpYSr{{hK= zBp%B8>shLeq7C;C$V+&>-emCB!5DC$j`^)^M%)4&TGz=HR+p|IwP&eXf`Wo>`s2lOgNVkmm6x8E*%X-Gw zH@i#5lX3mKhGD(t{t?(h`xE%bKAN0Vlz(7;o<^1I3;QGA&n9;i@sR94EpC7Kzj=Nx z;`H2}G$)|X;?NA8mRm;pJM~{_{hE=?E=GBNGdvUim6{#%ui-`559DP-d=$`+(0;?g z{UDwx0~i0r|14?bHF;nA2B|)kNM+=8SJSDN3!VYXa@0D+y{Em zK85lfW^{JRZ7$4*1<)F^@ zo9GYr%_esV(0qWr#i8u%SUj`xL;Uw)ysPs=>lXq#5eWY_`}-l9PsaIx^~dDzz?MRg&pFz^Ig;Y~CI3iGvgkLPma2eH!~^gPZK(NKgZZ6=Iust8|H+wgd8P36O(8!+5eEXJeDsA{Lez%85kcQ{?!M5w4?WY#5>{jB*yt6!0VVVyB#BK(o>CzSnRPnxm@?pfE)S8{m1 z;!d2eK%N{o#rIgg5;Ei)WAhcnQ+G}*Um1b^K6yTpL+dM#m*~%uk04%}@(~Bdo8<<- zpCBK}5eq)W=P~)wt(%XK|NMWMkIXm4ud(}Y$}Ujk{`)|P*$Kk)51ijucE3DsygFiv zSBCR-_|MJ6tJwY<>zjGMy$GK#jJMz3&Bamc6OYeyq(+|?+YmvisvblGaQr;6^HB;u9JAai}g>O z7aiHa62&vbi_CxL{kJK8A)nHN{L_cn=MRLB#WV1qiFii)p=h?q?`}Mvnf4pz_=WLJ z`wi&Z6wf%`y#FxmH-InK(}xJSg=8`(Zybl|GN* z>+>0W-neUWe1Co~-=FjIQ=V_Xu?X~24g2v!_`JbBmG8&7fBYxfkH^n@Fdp_#@$)OJ z|6}*#(`Q^ezD?u)K0R@KL;RRFo$tTYetc|vGoAnBho(=n#JBkQk9B{G`FkDv?PW~k2v2iB$@M9v z89Mg+>}w$4{*3+SHTlEm;qV7>zMey$pYqQmb^5?Ond;zom1V!p_iHivoQ&k-=Zhqy zSYPerAiVuZCiPDrx5)mq`1yaq^~q~KKgeKT1V3UwiwdF2lu{y6&8m-=gexB2h>;TP2t{<`U- zbAJ;(RLth|Khfu^clXcBd$GTtu>pJi)1&YIb3^oN9F?{v>0AAIef|Au?9@+Q`NQv{ zZ`)Z#{hC=X>TBwg*3@G+f{^~RITY6m>5HS9Ccm`E>!`*boCEwN0w_VX3HG#0JVKZ7 zKpn4Z_?u<$KEavxdu(qR+p8@@AlG~h^zowSIg>S32Y^lUh{QXF^x#gLkc{?}0chHM z-k%!p)g(LWrRk-~^eRdB*tN>dQw1-eMhgX}F)p!EKuX4pQA<7FPdxE4F1-pys4sen zcAU2fR^HHz#qoR7tawUy^rg(~`k*tR%w66zfXdo^p9!+==l> z{zU-;iTdZcK2dlQ<3+9oJQS0teT@GtaosB$B%+VW6JTtOrlso8bPUEJWKSxhXM;{t zJQjc8RNR(ilf7eX36UAMn?dh-iqLyZZW<{8J|?#-F+L`@TgaWLkC(e|oU{x{lBVZ*yJx$MZB>$eT6xtZ? zqOsMAVqG!fJY7wLs^7#bP`v$K-72%oFm58-#yk^riSWt_)X!L{4I_`yCK?#Eot%g> zJ-yC(iEn;wDcYqz#_+HcUT8QJ#0&gUhlT@;FRMDf(PFAUj=Ex!=rO2Hbr?;=AJplV zFMYs+4HNy_miki7jQ3dTwU#=KNB#390{w3kC)xz!<6pB*9&w{>J}G@kdb#~Z`$*L0 zdU!v@SJd%FyPlrL>%)pq)aPf%jg`Cc>c*MaCu0!zGT~O_=hyVBMD5iip{X@)d(*ts zQ)2VdEf}!2Po!-t&F8&t)R~?j23h$}M6dDyspA0BQuEqnu0IaEGK*NdY?if)e_PnF zcA4qlwzOZ0tl8L~$5OAg)RC=TeeE)?_QLnZi*MJ82IM%>OMbxY`b<=LK!sLMp`L4~ z6Q1f^nc9SOWO%d#uxYcv#u`n}b0Ic+_K=P>1ZJ%Cc#H1}D2J21x>YZzQ?Ij7r>;oX zG-{rz$M+dFokl51UuQ3IJ(6~Pqfst0o^BH5D#YoHgfLm%)l(;EF*tdXC_n4ewLOvL?X$BosV`n|9Ysh#y`=ofI&hNx<^C~W4g^CYj;|}dH6Q876R(IXH`cu3x?Bq1<60#Vzlamow zjcc;dA0;l)$nn@@OS(4qdz0?$L;pB)HPPyRqx1jfqrs;!x|Q2Recn8W*53cvb<<2u z9(!772I~VNk0izKC3*5=+)+qR$S2E()&x! zzxgch^LDoPdDpLLJ{fmxBG$*F-rg}gHV-@~9<%o{W7Jc#_AOmsU5>pr<7eO6C3qhl zhGVy_%D5x&&q_SvGb}oiwVb01A2;C##rNaH_CMbL*pqQCfuVn0<>icRsl?xw<8`&?|B!;n7IaPYY#)OEir!FcI0=e0@ z2lJfj>Th#-AN**@M6`^<$Mzte)5YD|L@%Dxm1Tla<#9|Y8xv1@F(Soty5jvPe^Jlr zQnYB!i+D~~wgShlvFCJ2h@y`X+~f_-bbd@{Y44Hs#@GkS0uWyrM4dJPiQ4}}w)>Jr zwbu6dV@S&xm*{r|{gEb$LpGrGj>b~#C=HQ4tbJ0uS~=mHsbVNrTJj6|nYdU|CG8Jn z`=f@obZgZ+`46@CSlDQiCU*lzg*=aVx3;e$B)*<0F7fxT$oBaXq;za;Py4yY610EO z(q8p!Z9hvWB;Zj5@_ItH|3KDheEP15wLSSgx){U%pqwIZLwmYa?Ufx5p<{x12x%y% zEmKU5ui7d5KZ3^BYOm<1J<(a)(>x~?SlpB`a!i!oJaMJ=MAIv(v3LB*0PnLb29Y;Q z<&_4Nh5wfcmkdyffdaWdd&eIfo!Ep&KQ~Z|8|1*G$pL<>BpiFbpkN6N&gpRd&z$6V%7gzwExcuD^Gg zqqVxGe(lc6<*myfu5S+1t!-Vgxc=d)omCqb*R^hLTVCi}A=;L!_&QwocMiZfalOwy|ja_UocsTsR)x z^383b=FLSl&_`P($-H$l>8`M&eDSt5-qO{Do7%QjY%i(n*|ecJm~hOhof|iSZ(rr| zM;m*JTOe;MboHpOthm6xu{7Xo^TR&tt9|Wj9$iH?Sh2pvTe5alY5VGe#og(it5%)8y3Ufb%mSS*VZ?07WUd+TD=v07c_0G?zY;+TNZe9Q_GTvl)tHKLtAT0 zuwh42QPBfSy!a`SqJpi3-Xfph+w@?OZ?Vt6q;T0XAAYk$&^5Mf^)~vi_7f}X_;0Jw zud3kJPHv`BaE)h^7Zxp9vZSb>U@50r?rrV#H;DJLdic}Z+dU3FjrELy{xx;K!g*xEH}vQ2vQBhAR5)BusCx_D z48YGQ9IoZ+zTD}0TK|Tf@ca*dC7k-@#emN@1fw|?JkggM@UdS-TIqaw0uAuISZRr_ z^*ZQO_txtRXlrNht0M^$T0S_=1QT*7Wyev(f+oItAb?HcQg0_Xk31Zp;saF@J@0uP z?;4E~ltj;&7W6zKj%i^tjH{4>KF*IBE_Xr-h^wGC?z{!E1D7nr&P~ZhVUpm=|AoF7 z`zGX4Fwui}r!SQgy+;C--$$U(MD(k!0{;WrC!(kQO(OV%SAqW!aN_3_w*zBpUTIh1 z|2ZUHL|-dS@S2o^#!NKs8&Y1kG=Z95Eug1*myqeZc@X z%GE64N)L*@Rl=?Ez98XNdCO(UQR5cKaaS7P#&JUuuEtgLyCvK@?)N3!IxgicR48|$ zDW!BragBz*T%0w7^vy7_q?B(yPutRsU+ep9B7=D0r{5-wF#{ zt@i;7T&?#^q}`njzLs`xu!muf(Bxg|?QU;uewhE{WCb3`j9+);zwtyr*T}AS=RqAc zw6wS2XC8f>oj2C3y}L~?)Z*`C<8N~0)`9=LNke z{f=ej>iPxvJ-Nmmf@(SRvDV*J(uLoj^mTXn2`cE(&;)Jrmir%RY4n@mLVNyZ`Ez&U zv$z%G(a%>jwA0Vytz5Z&Re6iI)!*2)s-vm96~kZW?vLA`wZq%g(ACk<#2FI;fjz;e zs0vw(5-#ruDuNpwi{&?6ht0gfZJiCB0e|Dz-xUpZGRzt+(heyNU(&6Ta_75Rs{tsi#Xa zL7gu14~3M=w5l@N=Mzk(1u?is-kp+hOd(XfQuX_ALn)U>0pvBXQ`|f&^D+8tjpCgQ z$FUsj?*XLojkz_objjfBp^BlX@zwQd*`E3~mRc3On~c!-_zFsB;>oJR+IB&q_*L}Q z`d`WMuh#w}3qtX&`pbmyR+|uRWl&L=uU6Xavi%tOW&5Wr4Xph8flp*VhxDI|a(J>I gxq@o1a_IdS*g7^fIo(O`_~CIWKe0hi^0K!7e~Bv?f&c&j literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd3bd7786587938e328844c978cd6b2e3.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd3bd7786587938e328844c978cd6b2e3.o new file mode 100644 index 0000000000000000000000000000000000000000..d0e44fbd91a57ff826872e972c8097e6169a6122 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZLOf>T+13MuWI#e7UYl9bk{Tq?qfbjR zEkjO($U4SgFk@z8pN%tQ;>pl5WSpJMIGWAb$QvKxIGDkjY}O}}b3$Z>kpMHe<8>f0 z?*0CHbl-c`Ai$p3Fcfs(TmS!8)&Hvc|Eg}?{_?Vw%XLj-s%Y$07ON>JW0^g%dk??$ zuvAdGS$B26t@h{HZrlH%7ur5Mjj_Xnw(Q<_@j7SAp8YSlp2zE7agV0s6)jz_@A^UL zw`16~4>#zc4=Qb;>o#`y|JhjY(ROwiv`4x#SmF1wJ$)aRu(yAuv%}L;yQ|OICiV{6 z##3KMy3<+V|J6NxLujXKwzq3B_VB4^1}e@U7}#<7nSqt3pBeb3?zslqXYo2b>*Tk? zy(f-`A9H>ueEh(P@bKJ|;oiBY!uwaB4qw`TIy}7pa`@@F?}Z;}IUhcL>b-Ev^u?j! z9R~)Ue&(5h)VYg7vzosXzA)!_c;DrphM#u66>d6lDcrmAGUy6Jp@Rnpwx2jKuy5}1 zaK-5p;ilHp;obvpg@+%x6uz+I5@;5N%AE%XF3f#$VBc!=v;RBcSuGg<)LY^5^OwTg zFVlE`916|Fc&iVhpA+HPE#D3g17C6eRCw0O^WdHO`J>SA{(}Qg&wXYfbm~O-0$xp* z-wMw@c`{t?yd1tT_dIBphGw-qInWDTl%GG2zE6ZhCr*VcoTtN~xtGFEuO^ckfiLI*viIVZJnIXga_I8HBcOraQWC!%4!!qQIA!a}@T~I} z!Pmv`nCXv(h6%qMygmZ>kI$pspP=3TfpWn1C7uiqKMa_bef1}aU#%X#_;Mh5Pxkkd z6T7REU&{`X9X&Qy=zxthy}P;&J~ct}Tsv!<&`Y>2IxCFYA2?zE3GgrB16wp!_y&AH z(+D@AcTjgcM(vI=R`?oVIlPVN>xb}@4=PJSA8sgy-+8WG-}U-2qyNV9#}91UY~_>W zf6Djz;d`&bH}Cu2-R*nbH}SorqxjxYi|>(7D*x-Z_@DCEoB81NT|W#(N+!NN^pJk| z!ao&XbK6=uQzEsSQvoqMWp&#nk-hf{ZX0vN&OC}s1tk({o{agL| z(0wZyA@E9LzDF`a#!HaV37JT?p*6 zhrQl?E+J%tU6udY#atW(r=IN=p>Ke0+wQ0JIG5Z)Sd9v7?%#PbHi@df-A$0b?giMlb zOP#nm`=7;C-@y_+Yza#MN0W$hUVCGEN1(%7hdXucY7upQt!-~?Z3wb?t!*8?d99Dn z>)6rMy0tAf2iU-VQde!x!wBCq%2Q@9b#ywzt=Mo13?JI~sj~+Inw$ zN3Fzm_*&cB0{)IbQ^A65Exs0SbJMog{6%f~wg0{t3{ab2xS%$F>7v^Fyu7@E1-1CM zU}4_k{6$M@Yx5wdkn0{0qrK4^@HN~!f&_f+zQE&<2v!RCn(qm3gB;#_LXZYuUFWvi ztpRU~Pguta@Xj+qFwN>|X$W}zjeOGi18p5`9bJAjCtO==dxy8RBkvw?-+HgV#TTf* zC(I1fa9OOjx*A4Y~XGF=~-siw)Miwyj&+eI0qMwzl>0+9+n# zvQ@PLQ=5maT9db(NAD<}S|WeBr*g8JMW$Im+OS);mZcn?Xa%!B7>YncInamN{(>>F{Zy6)_9BU`Cu6VY89RShReufS~sEXGSyr{l< zeR*klk-G%@f1IO;LWd+ox(KhW1oC$T;fp1m4PG~$E;KcJJDS>B-5qUh&FypCnAtvW zyU(3pFmK@;_jn9vY}T{i9${AiAf_9fY|p+#J0Evbb}BpWW5Bei|uD^^9jrc)bC;9wUVH@$A z6r2Dna{DGIyr2&=!n*>_3B+&e*+>!_NMnV6hW=6K^1jLYbEcu@uS0+6|EYZ}LiVsb zV9&uy_TVm}r2_dbrLnJcjg4?~5I)xl{pe%Y*VNQ-dEd;2yc2curHBLiLI1hIi4)1+ z#&Z3OIy>{SuJluu^>t}p*c1K39!~D$WDoFD7+~Wk@bN{xXIDameEE8nV0G|=eolff}{EN5G4Z<8lHQGdyZy=kMy>J9rDj5F)(LX%>Y9yk?=w~@e ze-SC)vMwwWG_j>WoA`l$6$T!DnA=Me;}r&G%;NG;`_pNdpK~JYEZWn2%A1P%>vfm4>A^X(XuLVZAMCTI%Q1(~w+E-DvcGh)>A~4E2tS+S=d&lf*lbd5;)7UphKfLM z2R+s{qCe*3csuF>Ujn=vc()1fMqS{S0Y3%!Jb@n)?WdqF@LvT!6L^on&rIX}XQD3f z$=aSS7w}j_Y5YckccCuu3A&JHi^OM3{wLa&d+_;(_M|U6`se(-W7EsBo@cbKMk0H;{WD?TXaRkZs*}Cp zADGX%e7@3r{zV!qenAuD&`_dBF-x<`@vrt$2`KL<;opALbS^LeEHC0e$J{-=|=m4A@_ z`6p6@$9QEgEHU-hvnwN}@48ltzG;3N^v&mM2G@6n)F1pmBdYJZY^)b9i@sg1n7%7X zU#{5v=JNyXrM}UBMiAP90FnM4vHsHh;0BcX$}-DK{YBS!oO~;@k9~gVQ@={^sENAb&fj z=_C0Q-pbcw%)b?|pD1GfS>qwu_Y4up^}8@Qb}ZSSkB@j!mYmCLDjxLg!qTtem!IU* zWAf!V5iiAOr4Mj^)AT}$KYklbM(a1tADzcbo$&Z_!=EYsUI88A=gX>2qxPfTNjI_p zf3S<`wChPM8aEd5vyt*k+SvLe^8@5>m_KugA5~C4z7zI)0szVf;6D_9Um=I2_8*Y^ zM`Q6F`h+}|_^un`dv=_7jvSawBK4~WGGp<2aK5%8V5i7!i0`^i_T}*%`49XVA03lF zcPG*MT9&K?5cV|tAL-v_jL-JtJCScF`RXX&aX5c<{loDh-y!>@MA$hD$o-e>b94WM zfAf6NA@fU;hZmK91Z)nzKK~DuKPB7V!E&a?!+Pe#Rzc*`d_7CHQMBRy0eK0}*Sicq zBY6vZfae1U|4H9%OXm5pgXhQg+@K?Q=08TVo#anw5Bd3ce7^9yi=0p7D{oXnKCzyp z2zm|ep>9p(Gl?0&WYob>Klvy1mxAGE+&}zVXnz9#*hiC-it-Q4&$FnKePMs(``H{^H-l5*U&)zK{~DZ+{XkA8#76=B5bZY{ z+<%gz`;)AkDE)cbZ#a^07cvxpw}++eaJ>QBgT(*y1-H+qi{J^9ma2VWmC zUK-KEpE8a9mK)4+8`~G|cDrf5aeD5LHV3RD{1M|*`=~#Dn8uzBb0mwLjieEOg*~7b z?UN|qK|XEke-ga%^)1bX{LCHg5Avr6rs|VGA8}zm4^^h72eS3aRG)|mgz+(6P8RBX zzlr`}-%N5BKg|coTNupDjK(u7Kg53z#=9{;w0+^Ca& z&&@|2=>l~~@?4<(n1(lP=~^!^FJ{?hED@N z33Vghg;NjJp780Y8}S+A%=mGr$MBGs`zzN6i`FrRZr$;bKR67l6ir8%Cx zg#2+gOC|q?KaF8;R+2xZQVV~n-!9_G7g3{PSTBZ9PG7pE}(Zeb69-TPzMFDc%>&d>Lher{VH?AQ7FLH@?q=QAdITJtkgJ_7q42rwil z(tg(Sj4R-;^SFcJA^003_hY7fhw>Re2~7D1;eUv4pLPB+GGB2dzv4yuDe@7_HCqnRQEb08sF~Mm%vZuk z*XvR96+XW$`3h~usR)1M`w3-#*c0cEnG=DHxG>*Oe~S4E`9s`%W!fnDihJ74^OY=~ zuejsqE08D4P4PXNuLKSG#>jjH@zk9a%~yt?zfYczWYPM{<0bmDP1yyC+&!4a#NMmmxf8CvsNbx*re42yup~9e@!gT`Acd`D7 z^P;00SiE?Kc#;0^y#F@EFXU5tfPeZB{rrLO(Rc>_(-F^TKNQIn`Q7cuGt+*<9KSHW zX}IB~+~DL%miQJs|FQ0GF@JAizdgwq z-wK1{bcPSG#Qu=xN9MTku#nh)XJh}JHI6=?Gwi>o(0Siu|^0-|fXWhabavI4HXl-^Z~^W13}u zou*^I&%Oo%?$6kN-jqLl9u9vH=j&PY`6>TAQl}5h6R8e zx?k>V;Jya#Yv8^H?rY#v*8m-!Bh>ceTMn%I7uSFP;g2F;eYv;lx10a&AAVjj=C2w) zy6`uVLxpT+@6$c58h7vPoMXMcjP==To*RDupKBvuV-nGxG5}4R z!~0X?y_R4{y*RZvkzPfq9=leyd7|J2)M%mLG{#j{3`o(4F{-HNM~Ejr#?{xr2=zrT z(T?*r#>yL-u^4`DiWN`kj=sdX;40gK&y(;y)f6O%Jz64Ij+S`%sA665+fy#ji#t9Z z$-gLIAYT7G*Cz^3e7wlDfQMq@wU6?@C8m32gLw2&dHjs6(X?b8nvTLagzQO0^lZ>+ zlE>l?oQm6$Y_fNREg>@Fb~ET*PZD~M%1tB1!AIqG#m7hGb_==V^%0c|jF7BKG7O1g zkj(z6NM;}DQDg}dXObq$8Kk=zXCvpJnRAt#lv--P-9F;A zxfR~e@D+8m(Qc)u(fY9BV&5{SEe>56&W7w0BqVcu(3wdb6kkco;{>v4S^XeJ>Fuw0?OfJuWr=~>eTBr)Tt}d zHI161>al%>O{Y>6~eBa{IWE2I(9@rW!o zPOfM2UQ3=w{TQt;Mlg)btRuTYV!+>z_0R`=%(O;-7Hj0EG3I9%HREyr*>PjF=V*1~ z47(r0#Tqi2&ORyLsPp?V+`Nj5SfS#>$hbp$%=jm1mepOiuKpA+3_CeZyNWDE?c`*{ zRpXj0^aqJcG;%C9*%EHf{oaK8`_MnkT#dK7-|qas^JwsCjBe#NUY~c)p|$rvblo&l z6GxsFTKS1bZ`lu-@}Uv!qkB$12NR+z2rWM-R{FcGZqiSjmCq_qJbJacqZJqQclG{~ z^KU-O`@Eg)ecttJnoq_Zn~3$XsJC~-j?DuPipA``%oz35tbJS8SC?Y1&G^~3b`{=7 zhvC?5t1{{c{Ie2|_za7VWG(0D!pBYce)0V{vHg$rKk{UpOJL|<*LeTkiZ}Y+nX!L~ zS7`ocZfkU{+=cT6nopvfvE#rd&NGa(n?A#6p=xfqcCQcsd}ITG&f>&FKFFx5G)j3Z z&;N=0Y@f|;_VkN6FMapL@h$9ovw!p3jkdpf;+sSna+>tEbTqAULXBHSpZ@ygQ(LcAYS_)%XVL~ zsMgvZe++3k;}ZR@pg+@Ag`xn`wwKD#;5O^Slg4|ql*#z_sc2bHngW()n3{05IQE9hmeMH z)-uJ^_^O?<|Kn(Uv-XOP+7q3%JSa{F7y zC!ZjlZmE-mmP@7_P+e7f0^eyn9>wMSv=F#^GVUE ztD0Zxs@iQUTD!izXjzdcC98@TG$l$$-Gcnl z=$3D82{vvnsDwUR%1Gv|n@M;1ZKVsgt??GG&fnCsZQ1ss>aI;23IlP+EZ@0t6Zozx zTmD3SS78(6ZHBI%s4FSV^KC5l*R}XypY;`Wt!tiGMK)NmzR6p(c2#lf>b&}v^^J9# ztJ_L8d24F?)k~X88rw=YY^(AXm#kV|wmN@v*|N?}EzO;q%kU~#)wy{^g};+;a_;KK zC+GZ;d)Jq~1mon-%jYIii?Q}ZHPd!a)uyCo<3TR?{YYg%TyqWgb8v6mV?IR-E|qnu z4|<-{kJ$e_@xM8~mii6ni=tiw0v-!2`1!F%8sFk1xZ;< zCEPmhFD2YMZmOKOG_G;JxFmeUe35YKNAW@eY%##)5f7K^hyfn`;~roZ%E6SpZcfeE z7YuNtT#XX0^q}aQCEP0SOA>CCw^W84HEy09cclSt95*Q8YFtIXTf(j5eow-!<5J#2 zg>n~~Qc8Ce*QhAChu0Wulvf2;apQRlT*Zfe3tYv4Aq!kBz3Ti^^{>|Znc$C#f_F>% zt+2q=dhfTu)q2lF+TC7T*VO6_bTRA^8oVpLovrnakMWb@HL_dX zc~D2SO|4D%na8^J_S%f0ra&+p)eSvoTG=vnjsV)Gy%X}R| zT>Nke{f=eX>YBOuJ-PZFf@(SRvDVj7)Pdihtn2LX5meBlp$XdHE%iO#RPQsvh4y@n z^5^bGXK^dWqo1#+ZKa>bTe))ms?sKJv#-8mRa--6Glsv#-5<9>bDOuJwxg}Kfiorq z0(*i_Q5CWnC0yPXPz1L-7Rzn81)F&T+uCc}{l5B**yL(98y?mB_t{3 zC_yr>zP%%V(JgD^F4+s;x^d(FVCtQ0;7x5l`EFj!$9DDGt#|1!yNU(&6Ta_75Rs{t zsi&(lL7gn~4~3M=w5l@N=Mzk(1yQ(1-kp(gOd(XfQuP_Qp_I#`0P^bJDQ=#Z`51k+ zM)6LD<7f`{_W;uP#@w1(x@7S6P{mNx`0DzcY)^d~OO*=V4Mu2ed<7*m@nqFuRjZ&- z{3?2D{lClcZ`A%H3qtX&`b&rKR+|uRWk6AwuU6Xavi%78W&39>4Xph8fRATChxDH- ka(J>Ixq@o1a_IdS*g7^fIonR}_~CIWKe0hi^0K!7|9UbN{r~^~ literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd4187341a82c234fffdcd7d661616fb9.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd4187341a82c234fffdcd7d661616fb9.o new file mode 100644 index 0000000000000000000000000000000000000000..b9a9adf3fed1e47b3c8b705ce24284a78848dcf4 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAPh6{Z0i9+G9V$45Nu{eOKOlnk3KEQ zvYl5heFm??b54Fc z+;`%5_)+I~!p9Gs2#?G^8Sa~ZD!hN~>F}lfr^6%r-wQuA|K0FIt>?qXPrVyXnfY*N zWXFNQr=EU#Fm?XY(43a!;}U3=gvy);2QSQjVsPJD^t1mv;W@1s|J0k|vh$b1 z+ux({{x}qxkMY(XL_a6Ob6dY19s$1m{HgGqljp%Z_47xek^Kh;pPK*lVCd9|@CCe@ z-+MDW_vFcNne)Byh56?}vph7X^@+hg=%VcWarAv696E6-T<$y_4$Z$5erhf8yet$t zabR%n$tMOw;Q7M*Z-@8Ie-nH`50JeNujDyj2$eyX7ajr)^p=wN?QrPbH^V7gPlo55 zzX-lAhR4l(Bs4tR1}9k+*ky^^IW~Q>$PJ>|BdI5AK0|H z$|uSHl?P^PdYr*pdCK@ zxB9i={j8tn5z_`9Sp|ILS=Jx=N|HzLf1C|&Xet;eR*9SoR z*GZwX8}~dya}7KnzP9UC(_9kkyO8H1_$OJ0!7GjVF3AKLFF{5pWFpy;C$XzPp}7kA zjB>GLcFhBMF8u9^{t?JmCgmCd&m-V@1U!?BB;Q5wIS&1MA(Qj(w9vbdW8{^D&GsPo%4+L|oDSb(kJ-8mg^#or38~3Mi z{WlsP_Il^pgpdt(RsLri&#t}${kvxBAs6g(5q?W{`Za7oHopLypIyxQhhRUlgYxP3 z*R%d`G3#ggz$0W+W&g|Hg00}o@4}DYgU$X8_KW!gJAC;N>}s%M({!N^^3#ti3PabD zims6T?&-sw_B}OzFYkd8gsGB4d?_#XPQO z>C8pLjw(BRfi_=D2de}om#{i!o}P-Ut})wqo2J_svya1)CyUL=?7021$ZHD~LTCR= z$Rx?O)Qg+5|5;r19W2qqmaznIG=(S^bToBz20FdIZ!U0+<->FU0_2cj@f59^sbr0%C=mf>ls)E72{g0+I5ILLS`a2 zzT8Vg?xi7jT3Q&#_3x!2pR?@Uv}}=etbm(OL+<>2j9R4EV}rGzee2c^UuPbxt806# zE{a*RVojaE)a7BT*6i)z(L0K#mdIc3s+{abk!cQ)Htd$IWjTi@TEXm3Ieh#mcr7;T zHojvwqn_k&oD^Ehi-2diV*T{w2!eXBG6T0BNe?n!b^-1_nihpiF#EcsS3@iM?*eYa z`GvJaP;w=Z?x}$9-E>V$-P6-En>iwTdS*NLEn@_qXYEAR70>ps10dSk&(I+VRq;B4 z7uDC+l$Defx{I;@$2p29bVx#^OYz!DAb)2NzF6GV=ylWSLUW6^v$?&^-PzvW(lO7C zneFp-_}ux67c80QuA$31%bjc6D^R8+}`w+kEZ~6&0)3*VU}3 zT3=SNwr)l3h81f|R+O;$74F6E`QAW7le?o8=K&9U-58p9rSlbh{$b&D2YUi=`$7jJ zPspSDba9VcX3vP--^_82ttpk+x7uMBw0+nv`o51wXYfFr3|&rVc3HQRr7sso>oywQ zP}iB;RHw5lBcA%Cx((ghrbd{)8plj^HqAaKmD`l;O2s}Kk%S}OCS zdlwwB5-+<8G%y(c^Ot?P@G@K024+ujF!Jsm8}64%_zoOxwc@o62oxHJt{=nq-pLlwA)P9qPYtkC z8r)xb*~Wg6&3gCSBkVE&#B`&R?b(-T=i_e5PG!g4>_^=~2KFTI=&v`3i6lWZ{)%p$ zZ6RLt9-H9jK2%U<;&sl+4hQUZv|y2egK2DNAx&AUbHR1<#o#gXZ zhHb=eQg9Ni$nBe;@Pa+90IuqXP5J)GRh$sXXRAiyS0;^T{Y@2-Rh`v3s)0r+1= zz!o8oj4T%T72O)UOblqkKUM|wG^#S~W6qB&Abj2SYv=?0>j4{c5}uSm_6^SD!$$_d z|4^DemHZhCrGE?b1AIm>cP6cy@GsszHwbeKS8J0=zQJr(`uq`KsbKsIME~H-E0Ks6 zqo3m>{Y9jFE4r~v(8QMhY~lz0RS{gf&VJ-nZSDles&t~ zKNEFYP^1nsmHQVK7~|K45GWBxRB5bIlljr@%-3a1yszp&m%{fqo(dIF6%o%_!rvgh=u ze+@acg21$dsJ~)8m>HZljp!3&{x&U>>fGP`e@iz27E%(>;pRH;rdk{y9_({d5YvoX?~6FVV6+^gol- zt^9-Z&p(kOJjN@1ewnGi-d!0neb={H^iA{Ipl?23Gq}Dpr2gRl8Bu-LXJfr^S@i94 z#q?c4`f|nQH=iG9FZGT7GlI|-1c(guiuIS~2RES9SC&~`>Myz;3>C7{tl1{{#?n{s zF60Ae(S+8ooIip5HiN&B{0@u1*?Ii5ll-S>J+q7af!2q5S}#R=72jqC9h}|~^EU_A z2l?AMO&`sl@K(MaWB#pz{X`M-&l(TOzGsL)uHOa0@#D$V(Ia8~#l3_cG`ZKVMRH8g&5m zF1nEg_=DX{r(I8C;e_#!pN*DZ(#F;&nI9m3!~B^~{HTKZ@tv^W;{Z@T0RN%*`!YEs zwf}(RKN^eg&?n@v#CP2g-?QVybL7Be5~*K3kQs~DLkqQ40Xs!*LwwhDvM-PC$baC^ z_~@AYxhIL%*V1GqfUu|8|49EfV|=zB--&!f$yZPLj>Gw@YadP&`3~7HCBn{OK<>X} zpBwup{F~>C4w+w)JiMsUex5~@>6!DPkKP6^= z_`i96F68vwo-`+*&w}7Iot9fh`aAO%w0=!XWS1j6zZsei|4Pn``q$7x><4l(AwCM| zhiSjz;Qo^w-JfLTMCs4de#4Q>?M42>_a|;npGEY*=LVA>tFkPPl`iKkjdAK4qJ&>(WrTSz{AdHXk zaI)0eF+Y1811D??||QopkY2hUcyKH|B^=e6#NT?;^T|>5+d?1 z=0=_Ldu}1>NEfI>lIH^L-y8`s{gQtqCs_2GNlTT#Uc>|N3vHEKUwS}Kr4y4#583~awlbP82>efj+!+`jAO6(~ezfBce8fB9^#sQGA;9aH zFS{K$DnPMey*P#OcME&?>)#*8eo686HGaOQ^mEJdV871S5Arv@KA$n!)0&@|@)6kY zK!71Zk@mBmXIut*ggYq(fNua`DHKCPmzyczPTLakNkWE z{%Ouv76NZvpJ~0a<|~Hv7x5ws>ot}W5$~Zt(|V5e*|dJ6o`v-~N{{wfk7E0+AKH(SKKph zoUdf@e8n9 zj5os#em_Azk|h>=iqE6+qdPYrA^-UgnUBmf#jnx(Z^|xEl2U9bfiX~7v)eqxsT8H$rB;v zP@2mWpRj(K_Sah}J|*(~Ige+^kJ5MBdH#g`KpJ};`Rne4M2hE06Vn`&4;2LM6s{9^ zzKiuwoEIJ4$l}E_#EbNQ=l!=Sej%UI1N_s6=;sfFkH$0bpN@D&`=Lmt$nS1Fo|*O= z=Jwxqti@+KIIKEBc{ysH+d_(-0GL`SY)P8()d^4T@zA5&fd_H7G_n&ogtvefBjFaDT@B^M?H4^KkfsIA71A&rkX1kve@~o=A1@yT-EL=KHm%d`?F4 z@$*FzQmn6bauD8rIGy^Zk6UDaTI~Ek@8;w+pC6>LFM=PjpG5~a|0(VMD3+h#^8-_U z@)^`2)xC001NSs=PXqTfa8CoDx(4X*9HDjq-*R9*zqt1M4}TQ->Pvl94cV5`=07`RlED<<{azmW31m^{p`s5|6CXO8b_t9OZdyaoZh~^6n5sv zul)XZk+1>MY`Sit6O_N_*xg@6`l5>P4wViS#N=_1Lx2&65Q$phgP?r!lUuB0vg9jZsNGKSVt7F|ND{ zMyM}(iFTZ~aaP{YjK%PKQ>=JOcl2e>MOWAse4d2wsiq)7?9~#%aN;^RfG1w0fJuYHvNEiv6I8^oiJ%HwBjou(!0&~y~WA!JV~ zqGy9nlROrG;8fg}WRtz4YzdJWx0^xldXmt4RBjq64n8WkD?UCdw_C^^uaBr)V1#5< zl3_>`gJkwkMKb$nk0MK$IFmF{&LG{5I2%0&&79kGE9Y@_t_#khqoNa|?S-xjOWRW7 z%~+wQBDB$%b+k%A4SF1fCvHdw>vVn1gP416?m@k@IqTItHegjT@*`k{^?Hh)<4F7+ zUn#Uv-bG`p6~($@#CW=%235cDSD;w?y}DIqmtov^wvBow=;Gm(6{w%lQX57drA;(2 zYCAC=XL@Rt^9tYm+LE*@y^P^uC%n*bD2NyMqYe!R7++R(e51uwe-d@YB++9~o$4@} zh(D;)EnoV82OB2(w=MN$ni=o0)GIA@8jt$tO9c9_6(`yR;^W`2P9AZqZ9XY|VQPu} zR{Mz8=4N<5!&lU?M!T7w#_Ge0kJsmC$BmV{vFgT|*ehcY_cGyD^l8P5)Y`* z>KW9t4Ryj(ohwrtmx>IJb^tbQ2H04q={YXMX3rkdv4+5ml^$=gT><5AvRAk21$F9m z2I|xm>6%8(QT5n9!=}?Hg{d3tg|5d^Zf-QnMaI%iyj+Diy_H3CEY8zf6 zqjqvK;;L~?7W#w4B^o&vn`{X;=6-L&y?y8(X0FCt-EVdN-+nasG)A{_8?VpX=g`{w zAG&UusfnXc3$6UbqqppbO!?5L_R&2jpMwd}6@-=_6f6CmRyXM<&dO(%Cmy}p+|h~) z`a61m$@w>*<$d1H_CD|WHO(jEj!ne+Sk&7)YRBe*2gPFcUS^DXYSzA`>uXD}*Jk|e zTe|}9qr-6QwpAH(1pZlxM|_4wN3xc4bm8MB{DAm=oY?-y`X7BV&LuGPuWP*j9>p7d z@66c0#49xaGq-iRR_4O_0?j8;&e(Ba6XzL5+6|v!v_v(xTzggt06wyTKxc8{As=K^ zSrVnZndkq+eYVeLH+$-ZoEN|Q!o*hgy}7^nZLRIEX3d+C_mk;AK9rVP){?RiLw*jP z|5FL6-T0%ryj<9$96aIqKQFIjY)dKrwj8hPJ^zOkM7E%7yl49m{Fn5GV#7&L^s9BK zEA2i4I6s)@RM&r-!~5VzLnfl-96q`S@tiL1)+T!KoURNLj4DrJO4*or(u)x(p3@cU zNBN6-PM4xZb6&)Ax-u0wdW}A(OF|TVl;AdRXr}XHI!k+xtT#kIP!@pL${^~r35eJJ z$Fkj*EULA(#~(vl&bUOstLTq3Q5>=Xt#>q*Vn=C+>|yPb+EvO4-$WHdvC@)X$j`*Z zk}7GxFWVnCw540C-pPNcy~n~vqcpi2I4b0M#Jjb986olYOmT_5e?_*>l^~^KYkS(y zJrSqZv$g#Up%A}E5yAK!qxM8+ZBO%@RA6ya#>g>IdUC{-+7nHWs7BxMCj-3C z;wVJkER$CnR2Ke+u7)~+zYDJ;o~VCQk?lPupvB36qxFUOFS5|#5&pdV%+PT>`}yIR z`1?AtpY(Xq?(83n>yrD$H47J#-$aD^e*%!kM=_x$ZW2K1*jOZO6tn^D_r@9jE3$pA zq@?kwSk%XSvc1PL{xR8J`7gC6m$tTFg30?-obkWD8N?`V{?3?neC0>wXy1$W9?SRx zvc2*@HNI-6;8OvJ*Z%D|?P*F_+tZIlA?o;HW|RIe%l4A}=*6o4^=SWJZ<0Kx~g{D3fI?k6s{-~rFc!zqL!|P#YHWf7O$_~Uf8n1yRorrQ*Bj#L+i%s z9R=&!y@kbViff9R8d_I&Zmz9rY4ES#&{DQ@Wy7K-z}8n2zNBUOx=pp^P4ydV@;29& zul8;%*wIp6QN4a=>B^Rsk5)JNtJb%yT2%dL+0L@sMO7`ETUX}SuM%yGT9&ue*B0gZ zHuHX0*DhMwRo}Wi4?Otm7cH&cUbJ?5V{KK(rrNcSuWntN-&k7R)wrr&90e!TVlFVB-lkW1{OBQWg=Pg>Bzo~WGitU9}-J3QR1mccawzGB< z_^vNq`FKNjK{MoSfvz5}FD}UQ)fV~dTYa!kO?iFWy2sa$4OZ1OdkfdEDQa7r*U(zi zRKK~Zy?B$iy4qj0yrsCQy=3FIN^eo|nw6z%^Ea2S=-SlU(zUr1uf=P+Hm@r8H@4Pv zLw{xK@@pP%Tvgt(Y2zZ;yDEQE+xqIJ&B9*Wi^{j6@4Uv^@-C}gyv6>jYgx^Hy(z?|MJ6 zvVs4$3jL}Ie(mITD)?!X>;6Pa{^BJ|mMqT8TgGWtdRsbtb>h9a8b2uG^S0&%^BCKe zb4NcuIp>euyT0@#7$<*0J~xqCjJ3zBn6`VmHZ3(94|2ipM=As2nrpb9hkN55^GRB8 zsccBS-}9_~#Qx`r|IP8W)NeRn5IqxpBPS4VyT_oXv5pw%UsLz1oJYrfLw~_8>qPeh zg~Rofy0_3x1N@xA;aZ~ZOPsD}^>5e-&;RgO!l_?g4ES6_Fq&<_6McyRAN^IN)y|j4 z(Eu-qm6qsQZh}sAZ@s>Vws!WuI+8G<<%8o)Fd>Ihb__)~4t zCJCkfznA>u=~BF?dG)O1^JNKF^XWC&&wwLco`g>UT(u?87bScu;KnWEZUfwy zDjt(?8W*RD(Q;hg(YQuy4^qdn_o-@FW z^vACQzhZzJ<;|6PxAHk#!maWSNCR2reO{rpVAt@8fT(2tS-j|_05yqR+T(747J znRgxd5eX-LjH}sm68?|{K473X+V?dHxAIR<(V;^8OcA&IeTjstag{GNNw^wU!H*c| zjpIHi;ns0~DdE;}Q{}v+agFoECE=syi-c1@iWd@Kivcc=c(`0g4Dje5_W-j*4yNRF zb85!EV1OItYLalJ2Swi^;Z}KHlyIxOB{Jlwar5N3s||4DxIqb5<0|^y5^f#$dlGIP zm+}@Wl)KQBQo5tKMn%CryvA6Qyehbg8_!wbDn1NY;3^IbTi|NxRp+0of3@Dv27go( zyhqw^l?ATWd%p#))_W$>?vA?p<~DDjn_-X8=w0pYYHMhEl>g-93OtY*zwXF?gyEZ;)hGHo zwC8J*KX*4ai(4@s{d`4T8~r@q>eV%CN}9bbzJ|^!|DS`x-|7u4tgW(c9@|3;4TmNMWVb zkfem81j&Mij?Vn0H?563WG{U4#*O=fskgI%H?{fXyLmAmJJfHt-l4zjDi+vJ_`VlG zM5bD%p03CQb*jui6jCnJs>*1ePcWGlMByHJcSgoBg;4QI)o0;`QZA1I$g6*+xOq)6!fYzMvLhsUM-#0EXd%i8|`1I7s%W&i*H literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd579851969c62870ca4274ebc92d651a.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kd579851969c62870ca4274ebc92d651a.o new file mode 100644 index 0000000000000000000000000000000000000000..a4bf555cc03b61451c3530bd367c866eff01a0a8 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Kp1A?nbr#-8ITZ&*Jehvqy`D}=+lx+ z%a9WxvW#son3&y=vvHP8JYHIcjI)y&N3%H-dE!Gn4rZ_>oAt@$oDi8I62>0f@j8$g z_kMppy6?Se5MWPiSPHuDt^fb4>VH-Je^s|`f4OYsa$VDyDjIv4#cB%5SY~hR-otM_ zEESY))>AWJtNU5D+xEZdg|^SkVC?XaExYe+yw2OQ=l&b67x4Nu?$LCjqNVG#T|Wr@ zb{xC<;RZeQL6t3Z&BhM@UmNQ?*1-;g_DD|#EBs!zr~kte_U12jc6df=Pt7^o zMC$8EPdY37Kf0%X80~b;_GTT%9y$H=VC97agFD`PdT`~Lrw6~Kd#-}^8N3e9IrT!g z@8pT_qt5SyPaHTI9+`hC+&BMpc>n4%;Y<6^gh%$j7k+B~yWxjgFN9B=em9&l^Wo6Q zjst^FJ^l1x>ii|4IW6A_Uz~R${N#H-2|wk0BiwxQQn+vBd!Q=}g$^DZ+PRN*{0cya!|!6#RvpZ(tn&uPW@r{4%yT(}h8 z{vM6@$Dz=CjJNtA`Z*b%+xkLy1o+Aer^9njT>$UY&mVQ>Vfe&iBF>=U)KLve2B?eS>|_Ma6{^==)?ibn6F~#Lr6j%(4!!$Ec>D5G;W-!H z3158lLU`QFM?xbH0lyr)?f?&fC(rql(8#0v1}pZx5Kg`TTJWB%WnW4D_5PYU7hVh` z@5%mta$-+S@+;XvvZKew3LUVKruWp;!>1-`o~!3l z3SWZ{Xd2-r^$qEcN2%R0#tL5nEQhxdefd*_$MS4Unf>;KhS)}M;`ac&m7I{ZWZ>TB@pp=@^bT*;)vLk-&D zbAO{>8{W_QX&y0c;E@%;N1kQ(c{@Wtfe<6)sy_m$Vrhv8x zb8UEULjUl;u>Ro(w8QWGFa7E}Z?nV0f1MB-{x&=OP6O7|39NtQN0R|d3=KcP4*&WA z(Eg94(76qJ9-+Ago)2H!^@?dOiS=E`^A7kYS%$$YjrlIg1Q{S%WaP!95Yu|%%H?35t8*}Zvc~#p-T1ibharOf z1~F^Q*ax0SS51kyqW#BJqxEI%kKHVNt|vs}6a5U$)6d69i9g1*wCPtquF|-VxiURh zQ|5U>sf&w3&My>)$TnmXvJ>_9HuQQP`g0+E29iDfJJA2Pa9x8}CGLB0J%H;;y#5jQ zXK?*@8Xxw0``Lt$4R%%jXB*G1z774mX6hjq?DG!%mhAK^*nn()5jH=!i1iP_eq;yb z)9AtyjNW`}Z1ivC9qU1Vv|>Mgcx3>jYbYmjj*ez}16S z5?<-dMZ=CNJA8pQUrPt81}2xVI%b}pimR?M+jyI%+ZeNt!;&Y9&B^Sz{j$hwixfg< z{~%-t^J`s4Q(I$@Eof`+ z^et$6Y(eLa=C-Zvv3f(7tUtCOdehL}>i2fC1)aWN=K>*GT|-xAi?^er&fC(m&D+`J z3)D4uJ38wmw$sLyTAm&G;5%xA>j2l@k!?ow0E|5cKgwsaP4g!o!++2yt}}C8@&Ej zU!dWxFf&ZUWwF|l1Le}m4QK@11vXUTn0=N*@0uyBY|90@o`GdhF|Ng`U5CgcWF~Us z%e^$@UK(^|Uy0*va zqL@`>tLg-%E)QF^W^V_N-cdZYME-JDIn|VNui}Y4|s+v)=y84AgBi`GjQvX^dRG97vSz=X;HWYvu{XxHMFAtF5ott zUsy{7C07FJo(lNhP1m&4Jv}|MnIp2NXSRdiGDh%u)=p$y@ofJF0HUq^EFFSS6|W2>PXm0U#Hn+FAJKNh^I_9}C zvwhwUpF4lig2nUP>*=zfa2YG^YHn$CcXW4j`dZzsT^*h7M&H)vHlKT4Rn^M1b?eJ& z)>c%lt}ENPu55K_St*-er)jiHHGI$y!(9~NGBuzi5r7djYu zLLS|xi+kKMdq(X3W{z`gO{vVj)egI$?ZbA__kA=vg9qYd=yE!<%etK`eYq%Fx6$Z^ zy3X9DI-OM+@zf{PZRplEHNy0jIA*G|Y4$m(+@@q#D)!kJA4YK{!&Vu%tpsn=Qkfs! zyWoh`c-dW`fx+;fzwFb6m)Ww`F?))Gk$3mlaKA*tci?EN6|ZeTpwKvU{a9YF$?HeB zmJ$Mm{9^QMW2|?V7Gb{xk_7X&y$IEKJKRMRCdD6e$*{wU{3&#{(6I$ND@Tjmv!rG z3-O}&*aScKp@K3Kuk%iJIAFJ<1&a(EOk+a}S#HpFADf!UBKpB3_SzvfE0|#W&6L-7 z@cK};y)cM_Kj0Z7Jmd>bjAWyhFd5p$c(Z~@dX)ZD8cPpMOqfXh4<1Tl=ZT(i`Xr(U zJ~wEeNN1g*f1>BpOwgl$=)*?h_^UtQ{O3YG13majllB5X=#S{R{xUi>;sk3go+##=g=$KElmG_*^ISqmN%-TU*QJeLWlUPS(knA`a*W{pSWJ zPbPmG&-E|r?CguW(ocETH>7!CPxKFaIJuLPJ-|<4fK8ml#~1b9T?rBP0RZF!@V|_J zEkYg{StRgf-5R@03~0hXRs{4ksxs|k&W|e~eBJgd=mY)h0UL7?o|HiL4bJ4lM+U(E zP?|lJ{22?Se+%>jd`2*LCas(BFWx>k2y+bAYLiL6!E9Fk^&`Mi!T1-6{=u17A`vY{ zKgUV>i%9v(y0J{q#FqYS;s^d!77T?W<~3-^7+E;!RH^^lfLZepY!vUO)t+n%4l7UMD}v~XTrYG0{S9V zCws#`FrRbze5LvP^E6icbxn|y{{mnB^`FY-qCUDmGd=t@m+Z;?72}bAZ6OjlA2Nb7 zXHxv*{`fQMe}$?4-d)pU{xozD>sx}2{EaUPXBNW0u--@gi~MJL0*yDF`_Cb==k%z5 z4LP;Kz_f&@zhXU@8Jsqa=o4f9HZ7Ct+~4-1y$kJ?zpWley*z@>wj(4Cg z+LJ%pftUUU|Au~NM)j-gO$?B|-y(g}J&M#fjb~T>IaC7ubPBwj&!hD((Xu`CKau5YvGo94Gc-+aDiaD8V;{lWh;qWZ4S#(Lqh=-cIr z>AQ;b<%-R3K0nZ2>KpxM1feYm5Eo3g@Za}H8EVI1SUvxbfDq`hXvrY1irLW#y z$Oq1$39Vl_e**b!27e>@9TtDH^Z02e`A^e&W*7Mbtq=9IUW)cAzRe6eIK3n0Zw{^x z^0)JvKAJz_t$aPk{96J0i6Z8oH6D_E&k})LzYBxo$CLf}_=p$f$+^6y;z92&Ed45e z`AI%KCSQ&d@lt$NdLQREO)sSQ=UsQD(bpZ7) zx{(F=gWXK0T~A`sgz=D{jh0{1#?~j9A0U6j{FzVusDk>5ov`2I08l;v|DpK%5;-Kb z|A6E_7K`uDC*-ljcij-*v*W~btr?!RQ8 z8~Z2xo9BxTnO~ATyr}#mU~};G`G2bXDcSZGmNPXT)-xxz3L>B8>shjmq7C;C$V+&> z-eve1$y?Y9JRd;#Px@|KGS814JU_PQ1|7+>|2dNFB!5DC$j`^)^M%)4Gs2lN#NVkmmB-CSg z%X-GwH@i#5lQI3fiebIx{^8$3`xE%blQcQ0DF49xJclaT7xqWKpGod0;vw08O3eQ7 zfAjoY$mzK~X-+_&g~4e$Ew_yHclN*2`ZX<)U5@blW@tM6D>*ajUqcJAAIQms_$Z(s zru~M4`%iLof0C6GrN2P?4M#G!7x@q0pSU@F7SRKr8%%aY{R#PPdcd9KM(+`#Cx6=Q z;OisCOCx&tQ>M}1a)ViJWBbD0Za2+0PS5?(=74pCKVp1pAN9u%)7Ud%j%1N@ku>73 zuov{AeG=t6$fs=s`@kz--_l&j&)m`eAb)ybx;_Q;5f|q3a8+u0AX}eG^~sn(7$4*1 zWTDRYo9GYr%_Mj6(|mxug~80sXgss>L;Uw*yzBEr>lXq#5eWY_`}-l9PsaIx^~dDz zz?MRg&qdn5ITB*}CI3iHu;@3FmMVX}hzH;o+EDYe3iCSwbtpVK{}a<= z_%z^?P&eXTIQ3BN37?L-5uY)^jGusd3=es^zjA#*UaGsmfzXc)^Er2ne4I}%5nnD= znd8~>$RBsJRPt~5(>V5e75P&tweXkv?INCh5j84?^)hW-I}v}e zT+fmhOLD{j1jhLxz-yQ< zyB#7H2V$n7sR*g<}ZH+|3iHHtn-)A`HCa?B`?xXk&j@$xg6w={Coxe zY0g&`0&iTOX}z-MD~9zK@gfWBHI@?*@1Z}_dXDwkw0@(Wh4ng0kM>xPV*9Nj&AeV? zz7jsVUXPit@cC`YS7&&!h6AJ2xL8|M~wiADL&0U!(WmlwF|6{r7<&vlE2pA2`1+?|Ny> zcy-JauMFqw@Sod>SJC}9);IHhdm%nw7;C@1n~S5?Cmx^aNR2)(%At62AD{1&Cql}h zG?ytpVf{4iueVZsO62=<9?y^;rSG=${0aMkH1-dH-SBZvbzKZ^JzQ{)F-DvdLep@eH3Q#q1A$({Y|Oa8T+8_QQT= zGJPJy*XQ&2ym8mW*#7)MzCY*Zr##<&eIe*48}{Re@OguMGT)DL|M+v-kH^k?Fdp_# zvGXge|D*TgQ>WcHzD?o&J~e)PL;RRBmG8gQetdL%GoAnB2B%K3#JAY_k9B{G`Fj)l z?J36iRv4V1Gkkz0_J=e-GAB%gg~a|l8~g993H14#VgEgi_KSvi=cEluZ2#?aj;d1C7AP9yCEV@;9tM_`H~(_lolj!+H2I5Z%Oi_;H+vgR&>_eH^Pa zrCIjZX*%}%>}w$4{*3+S4f(_8;qV7>zMe&&pYqQmb^5?Ok?P=gm1V!p_iIu4oQ&k- z=ZhqySYPerAiVu>I`vNf7ymkPsF2O>d#cw}>+YMIbG)yQv3`5)vm@{SYhC1P9F?*r;V=7gdi(lP*x4Vy z{QKWU-muYKg*3@D*f{^}`*%a6F>5HS9Ccm`E>!`*aoCEw70w_VXarTre zJVF=oKpm@V_?u<$KF*o)J8W+m+p8@@AlK>{=wn6CaVBc41^}Dp5s7yc>A{^gE)nf1 z1JJa2ygxPGD+zYgi&Kje=~a~Kv1{dK0Ye9TgV-+kEmQ=gk)8c zVMr8%WcE)*GW%$cB1@P!lQdDzAl;2P8$AcjoZECO=W%te3(lgWq7$R-g|3Ur+fw4q zSfQt4w9%M#v}!;NdK`r(Zb%1fbbZx>n0s#SLA|s&>(x9qU{x{lBVa}KdWxRoNcaVEVcdAOjd~{N;^CDQsGreN8%7?bO*AlS zJ24(-dTNdH3g7(NlC&$mjNxG?ywGqch!^;y4h;twUsiQ|qs3Hz0(HeC(PL1Z>M)v! zKd93!U;2Ot8z%a5nJmOZ{d{X+N)KdGc z_7Shm&G3Gfuc%{XjWx}n<&u{2g@!G3NLQ`wZ z_NIBMCq?I_TQFd4A5Ys>n$LUPs53oI46^bck6z^gQpbLzrRKHETz>+1Wfrk^*(_@p z|F*DU?K0E9Wof?@S+lV}kELF1sUusx{@P_+?S=1+72j?Y4ajk%mwcbu_35bcfC{ai zMLpY4Cp^`;GPQB3$na>(X%2+UaN@fO<^P!1=1b*o-br(S2E zPF<0%Y1AB5kL@#TI*n44y3StYdMxGUMx$J0EZxM*RfyAD$pzeC*Pvq^p$w2%A&roZ zM`f{bay^sxTJk*V$7p>qf?;H49oY>M1O8sDhd$tArZxJrSR+4;F+aPg8ISwVjvK2z z$Eq7=*u5Ao){wDu_DS(Zo!^V$=2cw83Kbtl#~s>Z#y?53tnRvX^{04Y*vV!z8SIQq2E%1=Cc%YMj|4~=Rc-E;Cem=Ik-X!$|0(%)%ylYZi?d{%kl(W}iJt+=4S zqxYAbfAd-1=k09o^R8dhd@}CXM68cRy}hG$Y#w+}EN1Uz#;B)e?VGy3x)ghD#?QXB zEAT!#499L;l`%))pOtvTXIOM3YdJ?3K5oJfi0{XV?SHKQ(I?|v0z?10#{2J4ywUg0 zjQvZzLi0a!Tcc|gE}SpWd=llX9S1gXo?)cj@EJynRddU=XN3UZBO3^G4ksS+K}OZ3 zQOcWn{!iRz`)qczr=H7s{=3gjY-Qh@`3rKMK1q%6dcpM&TB zR6}Yv{-`c57xt(GPk8>%OUoJCQjWhZ$Lo5}{~-mDE$ABW***mS1-+rza1s>#Y8~oI zyN>|Q59T@5_21_3KKRj)iD(&zkM2P{r;EF_iC#RXE5ih%$`hDUHYT3*VnmAPbjA8n z{-U1KrD)Nd7xA2~Oa+c!qtEG*5Jev)xXl}y>HL_^(%vKM4bcyj1t7LEh&pWo;pYF46A_`XfyghipLW9gU^fQ5qt9So@@Q)pEi&QN>WKwB#4^GjXw` zO4{$s_9qN&>DH=u@*isNv9Qr7P3{Jc3V9y!Zf##dNPIn0Tw?EEk?nIONa@(xp7wM5 z;E+I*w;QKO7T( zUq|+n9xvLR`(trke80G6;X?A8h*1Ai0BL*_6Kdin0i=$NMbbt=>(G90obkUR+viG3 z8lQ?qeat7@do1H0m+h7RQhRb~Yx~8Ryidg${~Mb@jN<0+j9JH5epHF}y=d>Tj6WdT zEB{mDt9A-L6@Yl{--^?ori8UU{a6&Djvr<=>Ho59FWHY?tomPz_WwDK{HpzB*%EHb z?Qb2Qe1dekrCt(RE}3#bbye*Ne7o&<6qobUGF)`Ns(EW1k9bw{!rD&NJ<>sp?Knuz zCq<{OYJRD!YPYRu?fQ+;Uc8*5q`{A<^>RP0>dP|yU}+FHVwwk%t-X=7zm{f6~SXj z^lm8J(NbAeyLM;!@|NX~);9TT*0!uDsC~3zXT`>XnwHJ2%k%43h_=No%UbF;7U%gk z^L|%uELh%E-?}UhJoxJimeg)9UcJ3>V@=1VjjJDD*}6KvvAnjcaYf5cjL`-@%GZ~# z?`kMm)=lFzZm4W8uGm>xzi544OZm3S=(y$ER(p$6{cUbufe;x;Niu-`i5l%jY|siy>)`Fp?Ryf!FRo% zSXsw^TZMjA1;2K3JC(c}Jez#c;>C*><>leWQt&G(%e^ffKKWi!i{F#+d0X>>c@Sz> z&K>>ua!d~PDO7;BH$Fm3mAZCYwJ9^``GkK|bKmTS15hkN55^9fpT zsjN%A-}9_~#Qt9r|EuF`so!)yCweCOMou8!c8@_%V;wQjzpCz6IFF6{rv98=)`{*1 z3Ww`yb#I}Y2Kae}!?jf1mpWa~>ff{zp8w&mgj2t~81T7==qz(8N~{1h7$D*4hcqBM%3t z_&}9-&wF0SyGElpCEjzU1wD_5qgvPu<0_<}kMU!M%bkz{;wtEkJ8yw($0ftCb5n9r zm?XIJf1xkNz6H4yO!Oe$=}YBA?~y>|_hBeB9{sB8z<-bS@#txP6A%9Ib>Ke$ocKA* z?ZB9tSK4*>KZ?YQ=&Pj(UX^mtnDNGaUCQg0CQ$RM8T8c8WeHbp3nW}^QcGq3IkKN> zX)>kfzmxpq=~BF?dG%Gv=Svc<=F_XPp8;t=1>XR8x*8NctwEg%p5n{lTV(x=0JCr7 zO7wXchsr5&VQ=C}@GBB-mvAND1oWqThfotGev$>AC*e~7S8WOOMG2n@xN*z4+WHu3w49K4G_Da|D&?g96>mhkQo^nMY?ScnmVT7Js2}4gQIDa2dd`U|M-6Zz z{fXF3-G_Em5 z=3NJVM8b(5<7#$P!XL802MqK^`@SmSR{rTJI#h_CDdLvDFP3mMuJXku30LDP_z?rW zaonR4ZXNd*5^fzgRnA))*EnBX5&_-ZPPQchuE4w|N8I410t|?@Di1TSL>M{3j>N@IYq#x+DLMC;GWY zcC$MV>Zq={trvyd?vWi zp07#%+}+qLZpC=?^A&Y%^z(QtSFT@G+U#xdHFU0OZ|rKp@He^p<2Gn%_cqpbw%0Xs z#)Lp%Pw*+KLKdTh%i9Br;8w?Cxs5krGjCv9M_q^C*D(5bMFZ`P-cB!Dz~6;K3M;LI zB&8fBNES47bmlL)X>HsYd*Pe6ZQS2Wy`BBLsm&+e%!}FBp?f2bVRq$>!LSy4AD4~fbs}8H% z1cl;P(Oc{PO^$!P_8(agif`3lI)t~{gm5bZio$%g(r%aSN69bSKVxZN<=+QCll{mQRC|>}@5jK_v8lfbft3Cot2@l^bYH0=QfL`DPf4Gw>PWPwMrd^+CUhh=BChw z=zjlxNaq|23?$Pe3>&0#_W%ET@Bg*;|L-mN=!?r%F4r}UsiLu$S)!((jOF$w?!Elh z%Q8XfW<9k7w)&styKVo6UTFLD48{%(+4B3|#_NnNf6l+*dKRx=;~q^XDq6Z;-TD2< zZ^yAK?{Cl}?^W9(S8eRj|Ff~aBOUAzXb<<~u;TCLd;8xnV{iUaXNP8F_SBxXP3{}A zO{BgK_hhr;|EqiZhtW>gY;V?M?2!{s4OX4qKe+wkQ-do{J~j9a-FpSJPvdoH_VI5; z`;HxrKJ5H<^yvO$(UEz_qkZ#EME9*e8NIOYWOQWT#psjs-ihAddNz9W#5>W9nGZ%r zw(lQ&@~Ni=Gv_@Jnced3=()K^qkAv@B>JTDjcD_+3(>xn7eQAXi5xgE_{g#SgL~&4 zjaHpJ7Hw`j8SUHuMs(!<3(<4iFMwuoq|$j{@Z7w|2luW_;@$$3u=Mot`yp2Ms8 z;v3O9$B#!Vofo6$=A8x2(#Y)A#|QhMi^{V{(f6@v=X3ME72P}U4e$j$K=wYo(r14zQVCt2yB{>rTSn@)qLFvrh-Pd(9-V#m zJoq{v9XIo#$Oz$=gV+0yM}M69Yqa|lwA(jW3E1A$a! zyYs)7p4wBJ{z`tB?C7c6_|zoLd*!rkQXk>A=&U$yf8d1uC&9mj4{Xs` z@oVq_O(Wc-z9HT5Ftt0vSn(@>74SBquOGlq-m5N)yuYCoe&@Y%b?2)u82vYyKfY(v z<|v;e|5Ls<0N;BVzIn&@KHI+6a~h_5W%u>(9jeIDH?xGW-Mm%4_iJp?r4bblIdsLk-%Y z(|@O59p1RIRqDv_t3rmwx5^+w9Qr-=;){zsU}rZ@`*5f%T94a57-2k>PvTpZhb}-TBQK^!nAVe2E(h~onT_$&HP%n-#)s8Eh!E^I zh*@LCKJY}kYD&cw?LVv@tuJGL>_+Jey%8Fp=x1o&em+J<@-ePvO~3qMwZ?tSmFvBd zG1nW(TwD@yey%h^wjrC4ov6RJq1Q9ep9}Fbll(bOaNmRLeq4{?^}lg{ z64!sH@nNsGpGk?>U{~dTw(;!B+t9yjrXF#@KIh@LWT#)j24wSdu=(jltbYjhBReRc zes?YFkCw51rVl(sHdXe&^i9|bzWfgS_#$lf@33FOAK0Nw2Vqx(9h;^LeUP7iSX~mi znpS$5>~}{WW_>&h=Yb#d7rotVz2=>|f7F4v#@jUA#+ZE^mONQ(PG-mL7e!uMq7XX! zCn1w0+u|2Dr~g!3_3bRx%a*Vda5RM|=XW%9bcQ;8e%z^RcdMxLYkfykTVt5bZ)@)i z%x`;ie&_b)wyo`ndPA42KRQ2t)6m`;^mVfNoq=%Yd?8wWLsw^uucM>h*V3}h*Vz;Z z)i?M$I_o92Gtk!29tw7bninnH)*5K_wKQ*QD|(>4sQy0|g8}M`iWk-wJ+P#{$m8)W zT3C;N3m1DHBuKpnatgU_12H<9e4#+&?ITDi&=Ckd3W;E)P@v_u@HWWdyDbE14EVdY z)o%^?S_8s5R)BB534&?XKubf&7i{8_E*NU>Z13z2qB-H(+d4XZZJnOmzi z;kGa{Ov7cd+L8n1(#Z{I1l$HTRN|O@mP2owDXeVE1-h1jWl%A$#j0J0$RlJXa{bGl zG~`Yi@>xp@leqq!G~`p3y_=RTvW^vS<7voee;=b3>9yEk&2QhjwIk5!VfFQGkJiUA ztCp>*7nphvwrb734j#SZcxs9K<+jSnt{0hR18Ku<*;}J#x9FCJhOL-pf99N>Bo&rJO2P-ph>y`8%<7Fq{?ju=oxCFDWNqRN3qW=!yHk@Br zOAMt@0_mO#_})d=tjyg#J+qi2w!3GRgWobn@VVAbWL@!W{~7?It^E`of>0H&!+24B z_4>;4$`W@O_Ww9X5rqy(i1Y!xwh}1V8HO*Gbv63jbh^;o;_GZ~Z*zCHx3_f6bz^1+ zd>sLI(W3c_=epO^Wq$EeR@&9v(&+B!?&u7(x?8(CI^B(dt<7x#_qyupm22zQFRNW! zS-rY`*~WFtR+ld;XY;Dvi`?^kp@t@RM=Q<)9`v~}H1SI3EBO4w!s~YSIN)olv*e`)30}W-dyl$Z{K1m9k?C6fv7(al^0EOSl=F=gaDxl8{u}m5~ zSaZq7exA>I_t|6Y5&*<>qm%94n`-CdZpzPON8Rj)-9iTT1n}suH;joSK{WodZk=r* zUi2QD;O8z>P-fzF#>oza>~^$Zv4I0wY-j;14BPG+81H5={XiOf?I61^oMQXUl-IZO z`cS^TIIP5pfz?9jxNiPZn#!8CS;=ozO^BYNNq z!}f`E)+zcYdOpnrJ^F_}Y$Q&w<~`1TA>=dAgP$yEFYtr@h@R^&r&A+-P}fPmU`^CU z{HBE`!HV3zDGD#>qm1ybkaH68n|V5x#s;%k@t>oA)VaK`^ZuM!s0ID#5B)!}m&M2) zb_eV^RL$<*NwicT--RsprS9=DZVtj1I-wtZ{QA1OIxg?)`H*+APQDa#KtJfeFg$rO z`P+D|e^F?f7rvxot*3eeu_hE;v_!4sQ2zniLv(pARmDL<%Dc8 z^2pdCfnV0Gu}j2&Cj4VXNYA1w(>~<6z@Woor@!&V7WR!|@B)H=Q|yc>A83GYT-;FkeE4S0{h4~zEGP#5?w1D^}LSKw!5@&0pB z7x;8-cee|8ETS}ilfb)B7x)xi$g@S_^CkaVBwhoa_zw!ae*ygK-g`+t?*HUpVVjNn zlJd9k?Aatv0)O9q@%pQLzHoc+`G@wTFFX3@{Jdq;EApOYw64Ztd$|2`Vc&QGeTk}* zz2P62&xL%x(tQ4T7AyU#CdkQufv@=LPi1paAKjms9{yTL_T>JG@yNfn5Q&@*IpLWz zDgJSP{2BGX!qk87&glt%8ajaWEyYIu#utT?3*cW^@8kYO{xdy=#+%Ol=OEd0dfdN; zoLX^cT1wnsu^!9}Pn$;csR@6ZmP>W+Z+p<*h4#wdR+E0GxlI1%0NyR^`NDLLcc3oX zlRw&lm;MI-hJI(p^{ebn43NFwB7M_6iqtoaXIK6?R0jQY3cQ@pqxCP*vOV-alhm#J zgY?foks>_CtN7{?Q-8fXa}xUYw^{T}^V^_rK3{XVzH_Ah;Qu*sef#sVUbrm!cDWMz zt|ooC67!qS544y1M*lfsXbS?w271N%OY?&pQ0gnsEHCvJUk`>#SVi6}lYC?8t9K{z zfzxP0>sP^_Kz^IS-$;Ik#oz2ae%eX?6SSV$MgBnRgP+z*(O$*3nPCT~cO?AH!SzA@ zc1F`j^C!HOug92wD_}oS#Qd|yL$dEFB9QBMad`ZAvOgam@uDKVkk?c^=-r8>U&XH= z$)_jeD{vxSiqA^#;{0am#T0*nHkgdoZ<;?kkC!^(@#Tg;SNy#MI>gTxRh>p1K)s7@ zWC8v_H`8g?lUg!iJmhDi<(IUH^-1Oj$lox3<`F-tpnh}*?DrS|ln=mvDE_`g4oU6b zBl(Xc;yd&Sc`Wf=H^le+B=HK>f?H^R0{_@claq?_56sWgsFHnQf8_hQ<|Ap z&(8&%p4*e=1oT-Po~F}s%SeBx{yVK-(^A=`7|(Bpro+F|bL0Lsv;g~of?SA?0{UUv zZ#cOBq{sIsc?EI$v$WrEq;q?b|M2~Zo73kJJ@AF$bVuBukl$v9+<9*F9wU14r(F)d zK4QEqqK7}_8vU&>oaZ*SFW%*L(|qIf+#hWYSV#CH#;5jie|$fSJsssp7CRlwBL0ed zK`+{;QNDwG+BWbwc;)L`mJ9irJKi7U&kjx3r+_}@!h9aC&dd(w>r<&d850QOW4wYq z)cJlB{lUJu@1;ooL|KS=Y*I3KY7nEV}h z(|iEl6<^%Lo7&iMq{Czsvm*+d+Hz#bE50IDeasR)dQ9cF#LZSHhJidg8{EN9! zC;gsTfI89z>X76)NBcKNNqVxqheSuhEe75U^>|^^I23yJSTmpVmo4t z=c%dGzl!H6VKJLnEk;59u)CX~0ScXuVE=Wn8%wPyI_=M~KNg3Wv_E{E`#WhT7A%$P zS^8p0F8Iq1>6vt566+!RAJmq|^96zbNtinWdclu&^qvoRC%m4*I6nk<4fAD} z14jiYHmnyXF#axKkD&kEaqJfqZ(rr-drChyEf4nVeElGQv_f{@Yi+JLGcj$jg$K^Q@%s_OppYo{DboM^rxAykkHqynas7NH zkLN4y_lKFCAUyxT`F%y#OJl~X zBc^y|IA4eV+)BKP@4vCWnfKcZ@cF`6`|Vv^9JN02_)JG?^m$PM#gn`Ee4jiKQVwOg zOz{cpr)ht^mEuz>-=Fh%hWsddm!0QN*bii}*O0&NN=c=7o;ESdLHST|*iPX(h3C6i z|HOIGkqsJ&?SOPv2$_qUk8H?ZHH zVvKLa;R!m!2Uuc%Nb@6i!bDg|?7#D||IV90pU)Zg-_vNnXoz=C+K?pn-%jTk`)?em zP`QD29(vl$e(?y*uM)8opVg149#r(WioM#x$!;gUI2F}A@z`zOa|9fssUi0}u7W+K-5&Kzmfb*Y@Ud%v4$l#42kaSM%G=fB%m^s~Yz= zjUSx*+t|TkHmmQ+URRyFZ%)AreSM7e+v}bgdG}xIV_)H@j5R5L-B-}t*O$Rg{pjW2 z|1S2Hoh_?dGvft)OW&ouxVbAc*l`m+-c)d(Vj8@ zO`FU6Q{%moVn@9+vow`nC7E8kRpbdu4-U^l^EDjIGhMbRC+G!#ITONk#N* z&}o|2;t!mP+mdXuXOt}=GUIkL=v_|}dXLLZBPGGd<#r{<$K`elxs&w~mkW%LtV%Kr ziDHn<{;^1AAMH_O2@_|MCe9h8yB=qw=b)K$n{MSism^u5S$tG0phnueuj=&&@rkmo{g;n%4%bDn@<;ti-Qp=mn0{-|>|~ z8|7UzwpvlFD@KB+YiUsRn|uXIwBMs!Wp)|HO=jD;XM!#nURi(NS=+vej#^UB=a3_}*CY?RwFG97lS|_n2Lujw%nR(CR7F z^9^;vQ=KbQ8<&X;k9GhyZ3fs_qv-`M#Afeq(y@lXjFld5iCqEZaI#mo>IHS`bq4Cx z73rErEl~BuKEtNdC?%Qe>?N*8Gj41&%0rq}ofvLj#YL=8@nLk_p*?2ulQhffu3J}siWi2RoTgny7Nd4@ zGUBRnO&0ot#3dRz5u0o&*XMpu%AI}aA7-v5TitJV{@;2u_&7$la+|ErTj$W)`yaY) znyIOyPYbR5B%`(?}&j5{_F>l0CL&!`=n7ao*|*?X8V>Zw`#rmn9p$6lN9vv2J( zypImUvD;Q<%n|r!C0_9v79Gi2&e4UBoA3kT`*C9XpXh(|$vBt5(7&$n{(BT}^u04< z{}Qjz{LkFh=vt)<=Lo89cm=L(+x&T|u6*>~ss=C>Pde{N`>Z8qjCHxLcd(#dErHOfafEfhlEU;z=(?q zm9*cJ?T;GT(ydkR74kgd-P*pAkobD0xFp`cB-i*7 zC8VL8woEZKzG|oJ|0o(?uf3w9_C#lGPxG8qU~yB%$T3lR3dEJ#6HSk(#^3QL1H8|& zI7Hqokyjd27XF8>hB|@21Fs{VsDD$D?Y$ueAQ0Drvi|y{aZ=e)0D8bryq+#)Ct1OCjDQM?Irv1i&g(?(f+?BkzcjHBwNBw zx&5u1V;EyKmNy|FlhQ_W<8*7UiS~t{f zFJ9B`D=Ax5w!XBfp>=uZ=8d&24Z*eRS}J!eZ&=s_*xEY6m$xikvuR^hlYhf{&*qI) zD}5V^x3^SP*R9=AvAkvZ!*xx;+O;h!7S=snxubI9!rGS2t;>u2D@5DUmZdHJjisKz zX5R10jSH7|`CFHIz(df#@PWEVN>@M9xUsfl)5g`0t!!Og)L2p1)wrT%2gYav9~J8> z)^{~5T-r_JHEyVCFRk2B?q9Ut(^9dmDn4$-w$;9prR~cqwyl#i6`R`Dw{Bdt{*fD^ zTUay}-SW+?;ik=ts-ch83X*y2X3||zd-=j`YkZ}vi#D}xTlPpvZTF@P#i68QR_@ri z34Hr2mOs|eUEBgs~EOIymC+RHa=tMQeVty*5Ox@dF7vaU_7EnS-{@LII0Yx9b#U}Nk0 zZs@OaP0{+t8dp@cY}&98_O313)V8*+X|u4`Bc)Ya(YL2@V^x>cF21thW1E^6KcxIk zUF%z0nnU&58<%)I#U5YLBLC9Hz=KN`FZ68nd;D8{ivmkLOBWVxtrv6+&0Bp9fouK5 z$~ykrD)g%=__dQ;sd%pQZ1SQ-ix)3ms7;@-H&e1aBS zD(f=u_CBK@w*N)yUpl^$`E}=WqGzIS(AL`o#?)= zaJZgO_ZGTofS*w~T+7vcxzqKG{&hRy`5*pDIQ7em0bghcMzbtr002{@n&Q5S1c{o7D z2dX4{-t#)%H5w%;$(}PU=y^mO)xu^NS0M#`f*&(n?t~N&S3z&wc?)DaE;)vso05yd zB*B&c3w<&64alWnq6hI#Un(bhuLLT;k3ylz=vQ3>{(H1fMo;^jWbg;C0slVW#Lp>i z2gcOA(yqb(>yrN(X@Xa!95iOKarYxpqyF8}1ZsXYgP!1*BwV#!DB)_8S}yx9P~%FI zDLwy#6IX&?mTc@CW)MMzMo^ztgvj(`4 z{^&K}mkn^EyoFNlRz7D*xK-W(X&|e-uSvMIpPx#&Ro-71`Z4nVfdOumH&@Od8rK*j zJ=cIAmT=<7xSBmH;rCnM0|t7deP5MuEC2Kq9V*1n6miSn7fZMrSNUR-gsX8C{IG%E zIPS9&ZXNd*5^fzgQ_fo&*EnBX5s{3j=u;epKfbw~aiPxNz* z>_&GU)KPtNTQh#<(cjT=bIsbiS_MPRfetqICP!`^_|Hp@P5o#f)PbLdkfJvGLy)^7 z&?&^l50}vISXQjAn}^?%YuGNRmO~$F1Dz$E`29(LS7(5rf*uV`&_-W*;L+xWfC(raT(NKG%uhYlo^LOEp!b&S4 zNjXOelKBlCokb7agf^1?!sxB+cuj3S`u6um?fDtm&5QZip?9R~vr^@_8A>}fys*Lve1e0k&9PX8Or(_&c2or6c@6FmH_ys^ zj6PeVcqhYgJO}#+0BL+*>nE|*Yo)NEAG*BqN1hiwOv05 z{dOF?`r$@B^g)F!bj`*N|34e+JKDhxgZ4;I1}pepwx|EYV)oXrbar?~YERW!+vL6> z+eGT?NKZN|_`kZRe;Dm_&GuFu#vVEK^kDh<1A{x>dwOv7>8A(3se7)1_8EK*&pG++ zaNmjJ;YXd{2_HXjB0MtxWVmnssqp@_r^A=_pAL`ge=q#h{CC3-wVn?jKlN@nW#+@7 zksSvHpL+V~!PNOnLvvcb6TUF-czEA?KMgPKWyrycr&O=u-H?j!U3f5-M{Z9K109iNSqq(a-+xgy*zk{8Mj+%g$d4 zZ-0-*`{Ph(KE_*n5dEA8&u#s7cm(+J^QXddPM!zv)XyJtR1}9k+*ky^^IW~Q>$PJ>|BdI5AK0|H z$|uSHl?P^PdYr*pdCK@ zxB9i={j8tn5z_`9Sp|ILS=Jx=N|HzLf1C|&Xet;eR*9SoR z*GZwX8}~dya}7KnzP9UC(_9kkyO8H1_$OJ0!7GjVF3AKLFF{5pWFpy;C$XzPp}7kA zjB>GLcFhBMF8u9^{t?JmCgmCd&m-V@1U!?BB;Q5wIS&1MA(Qj(w9vbdW8{^D&GsPo%4+L|oDS1hgJ-8mg^#nfu8~3Mi z{WlsP_Il^pgpdt(RsLri&#t}${kvxBAs6g(5q?W{`Za7oHopLypIyxQhhRUlgYxP3 z*R%d`G3#ggz$0W+W&g|Hg00}o@4}DYgU$X8_KW!gJAC;N>}s%M({!N^^3#ti3PabD zims6T?&-s(oPZpb#*>U@2k=GU~gwFn# zkV%qlsTVhA|FgL2J6NKJEn^AbXbMp-=xFNb40L+yai^}`t)kAabsbG@jX}1ct-aH? zpzW~*ojaP_wzkLW4PCPS*n;RyLwl>=+sPJm`huMcglKgQU7ao7j*dESOUpKIXOk~b z*Wm5wtdrPIUt33ez~32YUc6{qtFP7D(!8xLe`$Mu-M=pe1JvagEUH_yB(E+%FE4K~ zZtE7I_Hh2vWp#CVkW5?;b${z7AjDF-Qa}1$-@cg||Tt?_D8Cqp!Ye zTiw=xx78=CV+D8@m>`&D4YV`_y#6LW>HLBA&i2l3KbjM+y{)6u+t!(P7r1YO*Wc<3 zG~5+thH1DgR$FqQTspY{jextrhDsc>&vNKpGli9Hxj@%5una23wOF<55P5{mL~eY! zmxkO+L+-S+FplfrOG7?q*}G}kBI{TIH=l;w`TH2PNUz5RYeDBIDw3HVC&v3>1>B$iU^Td1o=yW&wwl=r<+#4z?R28i{10Rfrch`M=Q<)9`?F1H1SI3EBOAy!s`z91mO0C4o04k zNB8OC9=FV%5xc*c;~ZO4Dzk63!!Bt1uwC?hAC1o7fjAkuoX+gBZYN7$E{xV~G`gX# zGqFnu+Snd)qseNHO3DcO~ZeKy92QC!KeRR(U$!P~S{=12D~ zIASF}b{A-1F#O{m`*h)DwyX`zp5kES-90wkFO~2eINEB(XB!YGG!9)qme*_Y`Vp=r zgg_y`7(Lq<>)oYA*sp*j0}Z9ItZtz%K1p(%?D)>a2tR;J0Y$x&Eucd>RY0E_V5u~? zzw)w;{UV$7?zczSWdMljMkm{|FVW7&-ISfmj=R~9x`hnvN#N06Zx9nnf@u5|-8$Ps zyy!hP!OwlDpv=VQoRb|6*zIV+A_E80*w8|j8?@cm^TvD@(GMoE*AKB-!35iHro6F( z*N3w01wq|Lc*Y11`GON8*{CHL{FPxF z@tYK!1S@j;CMdk14>Q8M0?tXqZ|d1d5*tio1%HPAQRni$!TWQjq2{kgf9U_oeJn!u zusdMSp$c~YE~2FZ`7WiguXK-(aB~no*9raT<7=v`tGT>yWJBJ`I{8w>0sWx=+~DNN zbXO#7Jg;|d60xBVLWK>vEc#+-yFC6IlCGx_k50q{SR zW=|!5#zN`e0{sA=5zL)Q>n8k*x6cj29K+SxWRh<%o0UF)1XwB<{{qoJIP*#*qQ&Uv zI7xpIDc_22EE6=br9YeafqxYQ9(<77OB3T21ZK_Q@=*KJX_%k$BJ3>M(|pRCj{4jP zyCm!N?`boG^XAZa^N2s#XHU0d9-nXbPfulk>0~p5b7v8LF2^rqPjs`nq}s&$vFHpJ zg5C~#tZhVp%*pX~)CIm6csKBF6W)!wz^?#)8t{1nKP=i$LtWs%3VbH;9)X{o#{17i zUEq_oJ>4$gv53<6O#<&iUEmXRAau^$X!&4?alpasMa(3fgSk zmz2K+=gc8-V)$9JqV?DKeBt)s^AGJwUv~7*`FY!>mu5Z3XkCp&_Hz4Y!oJZ0`a)GF zd&564pL6+qrTP4eG*W$M3o*Yub_4IRY#mS7`)FN#AsjBK1w<*_D3|6+=Ir0x#$DX#GpHY!CgaTZKMoi!JZ5Dmg{5I&D&({pD?+mFw_J% zz*#h*^(*I3AivGvZzR9N;%{~yKkX#{DO%6$B7dOup`O-D(O$*3nL!7qcf|b7!SzA@ zc23hr^C!HOug92wt6)D-#Qd|yL$dD~B9QBML2&$dvOgam@uD<2m)BH0=-q{-U&Sv! z$*0HU%W)!JitkGAgVHRGmg0K)s7@ zWC8wQH`8g?lUO)mJmhDi<(IUv^-1Oj$lox3<`X}vpniNO?DseTln=mvDE_`o4oU4l zAo-8R;yd&Sc`Wf=H^lesIPn}gFquT^R}W;y;`Pu%ZB@Wdk=qd8b)D?X<2&*n_%psb zCV%cpqV=^jSqUKQY4$(Tzs(q*?Z~zT1|}^J53kkL|fZNAm1{jAT2>pU@ui^YQq6;dK`|pU78UuYi1F zJxLMt8rnnMn#yMqGlI#egP{TPPwFoP!$tA-jNmxbjd+^AMtlP5MtmaDEh9b&^%&l= zp7Hg~?vn9jO#iN6Sg*N%__xsh1pcv)CMOl;ADEwKQ6>Ar{>b+;$sI*JB>PW^*&qII zo}UXjJ+~*#3FxyRI8CSJmXZF>`~|IF(-PU`2+wbZro+FIGo$`Bv=IA&oJ@$10{UUv zZ#cOBBuDorSvgVq^R(Y^By)R_|M2~Zo6~0zJ@C1~WJlDWkl&^U+*xk)9wB=2r`-;| zK4QEyqK7|a8vQLdnB_LMFWBvN(|qIf+#hWYSV#CH#;5jCfBZ0wJrm|g7C9S9BmN3{ zK`+`TQNDwG+BWb6c;)L`nhW`vJK7)QPY+Djr+_}r<&d850QOW4xR! z)cJlB{lUJODLO>@1;ooL|KScA%I3KY7nEV}h z(|iEl6<^%MT8#Mbr{Czsvm*+d+HzR1650IDeQUAZBQ9cF#LZSHhB7TI3{EN9! zC;gsVh&s{*>X78QK>IgGLQKEpAIS+8{btfq<*yg<0Q^E5YJOH=ekY&~g-7RqVtNdp z27D6gM!XBB9;!Xz(@{6#GbWhv6Ht%gAusn=t`Epdbr(1g`mtd?=Z=w&^T{RR%Y_Pa zJbMxO<8GEp{tbT`$KI$Qe@dkm{t~}k#FHT{@ycux9G#CF6O z&l3}=e-+OYf?_tYDvW~sVNW+h0~9(R!T#$|Ha<$^@1Pm_yZsDPIx_maefH!I_Ar6 z2aXC*Y*;T&Vf@{~9{&3G$FW~hynT(I?P2LPB4+lh5RRs`-qcbFcKv z^O>ms#qt?`K5;J1e#86)@$I_#%b&vk5Z^xQ{AF~$;z)kki}X|EBbaY42l*pEUx9y` z^Oc3b8`o!AudMlsVf{tC$ijM!7F{y^hkOJ=UYxerrfGuh*Eb zgpaP*W9BP-ep~Vt+Kf{X{>b+e%Koq?&L1-;0~>K+zMuXS^A+-kxcSPAG4d7nj2q`G zSv+5H$In+FPnMhFdo*7O8uE?N`3mBxJ1d&63`2jPJRixT^_9m<^k>OO5HC&nhy&xz zaD(4ZkdI`E1)t*csQl>8%}2<8{zK*?^Gxw;^!}T&3lzEkJ`iMfg7EwU=l7*uFOM0o zj+)|?;d~wbb35@Wy8p)dX5Mcv#Pn!~@eJ`I{oi^2ZHiyWr}O~-_96QH1L33b4E(1fp3#0Nk}2}LTaRa^{f0Sy zVSLkm1Nt__Gmba!KTP`#;7##unCIW0FrHmD`HMB4;rpbR{lRZK&XWcXO8vlo*w0L+ z?_>D-d=B3??wT0epFha==luMX=i6^A1pQ>ge*6%=Z?I41`*H3c|AqGBvGX2`hy7FR z{0i&;=>7QAX*Z52@_!|eLrW|e@~>Pv>{C2*pGG2kQ^MFXrdH;ylA}9)1)=H*p?*4CmpX>`8nd$0|)} zmi={_j{QFS8VI;QWB++W{_uS`{6UCgxT2wwKBl-CG zA_*zhS35ZfZ$F$){nOVivOg_${-1Yq@|w>N(%2WlkJ!(m1DyYqc7GJhPw@SLDL?rP z>X7PQxu=198n~x{dm6Z>flplnba;+XJAhv~u%2IB`~8PMihT8@zRKTj{=0wpdHJ}% zYW(QJ-$V`-u-Sc2^}4FveRFe;_4P5WnWHjUtbD4^W#^3 z|GUWBcDAB=-Hc=Uy6S{=)!2<7r2lj_#r1sp;i#s`A1(4as<8*>0Dpx5N>FW_J>?3I z(1kou$LbpX&oX!)=S=w>wzrJ!)s`cWYxNBDv7+ZV6E#)^fKBs=#5;=g;7%Kti1w5L zXxcp9pBnGg1Uu?QsYQwODNOa)wbIR#1uvjR3k9b!uCO9N3P+7mNj*PAJn=EEyb4CB zFZzgfoVRgS-q4K2@Ox9NcuIHlWzI!c*cN=Bgx{&AAVKWa62Wq`#KT7w>x$o=a(Q0d z@$pFhMF9ix`scYmQF!9xMXm)r6cev~l>aR;-76c!qmRntXKbCOCF{_16viQBPb#8k zgHDq?7JuMW+?HgMy`yXikr}s}LGOB!(0f#F8YvDwDz_^>J}S3c$Q`ebs9a!#WL1)3 zNECx)_D@AJ`)H3MOPDy5G*QkV-HkXKJqOL4+jJ}EadoZ>&Z48D6Qk{gt_w@sQsT{6 zp{F9W(U^6#Ndx3>WWFC$DlgZVKfnc zP^VkI^Z^exO!RMC>dQ1U-eaj(TIw_&_0N|G^j|AZvNBU`=x+GSkrh3}0O-)8SF63ay?& zJ=;(xJk_}}wQ;G)@Ms5M(`JB;b()^zLTvWzAsuT7%vkC1C$=k~98UJ?R=uE3z0N?L zx*}cEs5z=0+h^Ex8l^CGgT2u8Sjx?fM!CpXx`~&o5U0123%J3qLB~2m86dGj8X+Bz z%3|Z>dM59+|YxHNaMt&M&es)na9`~OeH&%O& zRX5JCdof(BA!F(6lj4m!zZb*JtGI|2Dn5*kJG94)f0AZd-F55gPw~RAlhd>-$YRt^ zPDWfcuE|1wkhnx6$6}K$;l|wWO}MuY{lm=Fc&q!Z&i~tw2A{_0R&L|Sp_ur#R>=X5{^3`i~E#rIxj%EX0tXgXjNL zLTWeuQ(ayz>`@M$@cf^bS2DJx6#uszpX)vUhZID%pliHm`w;w>^asU;lc4BN>rhwP zeFSiRFwd#3|7Q;GgC7lYVLY_ywTicfr5G7i7**_N7CHISK7A_>ei3s)o1R#x%VnR*aB!JYhu}In|Xam~sjWhmNWcyr6 zN#j$osE_$%dyi%OW3s*SUusV-ZEe2O(eJ|R3EaMNz z_R9a%_^O?PPX!=e`?uq?rzv4=PcMr?)bYd2CjDQQ?Irusi&g*Y(f+^2kzcjHEL*}& zx&5uNnQpZLTd} z?cG?gqour}di~DQl`ShDt#0yHt#4VisQS^eon^I)s#-R;uFS7rCE6CXEN`i=Ez0w4 z=KZd&U9_^RzIAyXc<|RRT3WrmXzljK+NzFCwQC<=-MTivv9!9YaaGGsjL`-@N^44M zx*8TO@22q@Hs+n*F4_1s=Q^>#znAqRsN>7_0>(Ag}t^Hm2XAgd5yK@T~@nzi~WypYF_e)@;7y@ zYi(%`)a__ombWN>N#3&hdT+jW>(WK_TN{=vUb=LdFTcU(eYg=XFX40z&0D<O}cWt?WEx23~ZC;k>!q}pPaq<`Ba}%k>SbMyRX}hOu(^9kXAQ!wpQW+4}T*Li5+#C0pPtt-* zWkc%yo@ezV_CHVjZ;r2}e#7~K=$YsnIf3}IdklIS>xhB=HFdwrd34-2^cU>1PINy| zI9yMudkft(z|Scht|jWe#OZog|Aw9L{6GFmIQ7em0iSCKMzbw=qAxMvqi;o8?R1OEZw#LpRS z2gcOA(yqh*^GLjizEYatH7N&;8E@P-q`Yov0yV#yK~Mc$mT=W}k%X&FYKiPWNA^=G zO{Vnx_mY1+U5Xbqub!2BzAWKtKD{RU8IT54@Qr|{t3lDzI@GD)DZU)OMb`gOfZ4Zj zCHg#!L*=Bnu(xm}_!SAaOSqD60{T!%E#cOFY9)NSr5~j)>c@CW)MMzMo^ztga|XDP z{`hs^R}65Yytz{ERz7D-xK-W(X&|e-uS>YKpPxy%Ro-73`Z4nVkpXU$H&f0Z8rK*j z^R5FwBH_f3aW#8R!XL802MqK^`@SaOR{rTJI#h_CDdLv@T_WLXT;+>R60XKo@FNC# zHT;qIkN%*MwBH`4J;)Mj*Vt~sd9xm4r13dbldw^LY2UGI8 zIW=QnFu;v+HA%SAgQ9PdaI3s8O1M?t5*c#TxOsBi)dsk6+@OT3aTWb;3Ac{>Jqfpt zOL+?w%3WwmDcw?>pc@`cSl`)bDKBN&9Fyk^se@HwKX(7%3nFT0uN-y+a38kp6GRr z>}Gc!)KOh?TQgqsSl`icYt7oaS_MPRz796_CP!`^`0FLdrhd#9=)kKXq^QmH0mxnI z>lEVR#U=C}%hI*g^YK2ph8==xCG@f0*IC$!_fOV$b@~V@=+V#wZS17M}U*V9#N~7VM*-y3zf;^iC-X7- zZjIud49C$N?C$}j@r}7PwRFkg>!FIFsPWbHS=pZYHkL{iyc>6 zq4-tw*7|>!<6p1+M;3(QTlJR?;jK0y-0Fa$Fkh{-+hzMv^2_$mSQ=RQ_W>Wzeh%qB mm*wzeKXL`tUggmHF|c)PYI3%N{^G^sQhs8Cp5$e1|NjA<+a1CH literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kda850ab5991c75d665f25508ff6d34b9.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kda850ab5991c75d665f25508ff6d34b9.o new file mode 100644 index 0000000000000000000000000000000000000000..619df6037f23d20db8ef7c280e594a96dd3b78b0 GIT binary patch literal 28240 zcmeHv3vgUzmfqjDTXO4RNjAo#m!+00gKQ+Z^|S@XwBEL!MzSnhnMu=9w=ALOw%VeF znW3>a6b1uUhGa=)nRs`|6zvDarlxj7iph8~gaO9P4zAjSO{EIfu#Pd9F}-9kGnmeI z{>QzyujL2X)Yjnmw{PEj&UeoFU+4U2{!H_H0og;i7T%bAF?QtG z@mShZFUAg@KN|b6>sPUJZ@m`lsCX-O@Wqp{l&3z4tvmBk?3a(8iJhGMB$m2nL-ft_ zPo7FIuZxbKdE!+2i^pR}p?_}ttyssgld&V6pTwT{_@h|b+S=%mr=B`Bj{F`kKNjow z)$!Qd@+?0C znY*HocRzLNqbHs^)qe6=?C8nkv30M#790Nrc8;Bob!2=3nLDHF9(@$`d-T-ful*>N zmi(jGI@n2r+~c>Mi9PWe*{P3qbRRr*r2OEiUzYtS_IUa6*e}nVjGa6AR_y5Uk74sn zETw!?^uxCvJ(ae0WAqJ_=KyTH0sYi9cSjGv{yDVY+A~LEsc5SMXq)!#ld(6@CU2HO zckSlr8)puliXJ}}i zuxXLrA^6O#HqRID*>62U*nJwS)%zEi=-*r6KcbKCv$5LO(Z@C$VYePRtvNeL?=WMv zKLf9r>j+yvg&%y@(-i&fKqLIh^TnmTufAaR&&B-hGrMh_@;UM^+1^mF!d{Fm(Ik2fSm|Na(R^u0{>lMnZ@mp^=;Eqr)~Bl=+zTR@E5eRh~F zoV`6cN;b|`W6VUXoxO|E82Ajzws2|h&-9r1rj^&pJYTFud!*Rd%M{mN_53tSynn!) zam~30?LciXnv6U2zv?mCLCr~Kd+d7UOFU7^pXAZjFY`Q77tiBTX2Hd;dTiWBoY2eI z)D%rEYlx=TG)Bo+F2FB7h2M}p@)7a}C+x1pb;9QP$tSq~HR|7v`#7$HxQ^qprLs%p z8^3`+^Z@@2{N*?BmrVG;-@%76;b$MNwnxwY5WcdXEo8K6(Mrf=X0R_ld7mvncHyHt zwCMS2TlDPhNm244@~2PXo1fg{053U8zC=DpzQo6J%2$ir&Ffn%>P!ChRZm0o(vrrD z)Ia~f)i)W90YBtd1oy8?Z#&=rpKL6PJ=+CkB#qf@G=aJ#O}s6*oXCNeWFA~ga4ln5 zlr!B1w+0eJgNKqq@elTJ3Bh%53`D?=}@3zYOW$Q zJ~lNnZP3GeBt2K5KMYS#&iF#CA`}RRDn!-1!+Sy#zS&vtWVJUiITY}Fr>29GUI~T* zQ?t{-nNV=7u4eaSU~;;y(yQ}^Mk?!w>6;kaJylsZUFCMW>uTyMYxs2)K@zzdd`%Nx zH|i#ZU}EfJ$yGrMf`!GJ-fXlLEmIR^p2I` zt4QD=%!Yjv6T5@HnNc2eGr{T5bZCAC>V!>C&4zqaA@>d7zQewm$v|-UhUjK=4X=x} zFF8{VoxB6hj2oa2Rdo`5mUC~I5LSK5D|Dp}mXnHkELM86VjZEDB3HfNT!vg_sd94} z@&~yryxjFmyxy^?5SH@339s7=YIu*|=bfFb4txD$bKc+-7QG{(2fZWJRo>Z&=?Ab# z#$tDNk^EW?gWbeZctYUg(WHu0TGp~0F?kv~eeO)bGUV*^t zu}kyK@+n`(NfV2n8)_kU)!B@(wZQGzWm{7{=hJCJV)_n9J6oE_1oaDz&K8!k3poPH zqC*6vC$agr^Ai9;tIwZseySq%``?1UJj-OiNI-vrBGRKrNT*X*RX(!IgwLe^APIv>LKBbKs`vq!;qtatLV{&RX>=2TtF zpQXDo$8sS9k?T5|a*R0O53skvA@cYZm4eRM+>BjV#vGDPVx^#qblSmO3ACMMC(s)C zm`UD7H8asX6VP@Xq}lW0OzX5UyVq`WtS#kzOMY^#1< z8`lL~SvatDtu$9xBS5Ao^9N@o!ifcl;;M}deAdrl2=q&gW)v4|bdb&@}hBfd#o zy8$3kx#<3tykC;{FL8gB03`A+>Jt?U&BR1fd_ggGSUyeGf=7xICUU$5=UdRQ{we z{F71~Jq~(hxQydv#N&7_J(+M9$6dtxfaHrfP0b_dU7X%U^s9t-bGlo{S95x`P8V~! zSf@8}nvaFt{@t9W!+&AFj?*+Q3Huv3y+Ox&INhVujht@O=_*cF>2x)xt981C(=9~v z@-t2oDe~XU>CHNQ52x?Z=>bj;=(LB^9-VIGbSu%^ejBITbiA3gL1<9BlU zPMvP&G{RVxuY=PaM05L{obJ@|E>81fer|sYr?=?%R!(o#>5ZJ;sMGbFrgea@zm3xv zu4MlAa{6AK?&Wl^PWN%TPp7wYdb>{dbGn~sUj7D7H|Y3XoW6@__-A^Maeto~`qYm8 zNh%4a@F6#@k-uj;R)&{wc^Y#_eofe+(U2_Sf$(IG^OzxfVWwfYiqACuk%PUK$!dd1 zN%(#f3HIL1WJ~584nD{c3-t1!-LVRf@h^e%_H=N2e24(O?4c|>k3Z+WO!j&q=w)`Y zmz4grQ^?!7Jk2$NPU18lg;1Wh1Md)m6tT09GZEiqS-h;P5Z^CkfcAh!4n&_N+5sAK z4AG?|pAH%oCVG@;Cuo<6&IVm7=pl+%S2>LjkU;zQP<)qHfi~iMlZS8-pHHLx?)DIF z#AkV#KECkwsA?iw#vA0Tn@FC=8}HwqM#7DFt!p5wecS5`(eA5Xcx#zvxz@yOvITKUV$@ST}tkH`0`ncB*5QYPWt zAGm%~2H|o%KzWig2+#jjq#@4Nw^%Zpx@ynoR3%BenTGNGTwpT zl}9*_caCq*C7h2J+5Mc(W~=a5-K9zc-t3K3-vdT^&p* zr@1jsEsFTtx3?nBJ^+K}+uJe-FQdLpA^dfc*Icedx`<*$)A5vDXacO2UJKJY2>2#d5;6aq9d0l{PtkRjkrLSI~Hb@dV{5CSNq{X`3iEISqS={|bG4 z7~%E1n_^Yv_sDz}a5-LZT&t@~&|d*ujvoTwK(T1(mjHK(@k7|}p;$KXQs8oY5qKl{ zzNx%bG&Y#%Y8o53K5yTY7K%lK)*KWohJ1>hV%fmOXl>xyW*Q4j^gT2dnB)g&EHTj@ zPojJ)#fqVy(nhgj$ZO3MYX+_@p;$BUlqQN*6P*mtG0Wdcv2N(6BvF4G@>)B^iV5$a zSTW(96l*5Di(<`$Z=qN<;ae$IP54HNbrW7sv2Mb*QLLEodnr~-crV483Gbs=GvV7Q zRt;S1r&u-N4GoF(T@>pa$NYNW;S`M(y3!fLrlll<6$J}4(ZdjWvykutR_HO|g#|i( zx)AGeiWZqlGa&CSWQDXFg1u3iFP-pDB$^+Gl3?Fm=vMNtu1EYA#Mv1zDE>!CztYVL z>7WDpSijTygVqBx+b-A{@>>(mAEX=$czmvhy-If_Xr4|#%^sa+`PqnvPJz zE+xLd$ZjEhl9A=N6U`I1pSzi8V?NFsAlgF$dVX1Y`#iRv)z%Q^#D)A&pH`9=^A)d8 zGtm@tvVEF}=6x#ab0^V!{?zSf>-C}a9*rL}9_l}Czn$d8e8lZ{5^c<1xm$>)7?SO^ zk!U_Y>h@i_{mKyh2Z^>nn8s_ftHruUMaeyVvXf&2jg7 zJs!CIJo10tU$yWmcJ#=)dX@t%FE7NHu#HH5>CbP!Y1|~@XwC9d>jB!;yC7i zmn#u3NJ+$tigf%{(syPi;xwI_?N?+Z;xxUQad&zmPRj?0bNe+*6LDHT82Tk)S}quL z(f71m9{bb+7!NzCn(G z3Db0FrfIq{(=^?fX_~Gj&GVz>ff=XeftjZ1*-X=Nz@XcMsTzeUAMr3=mxO6Ll{oCT z2QxI6j#K?;I!@5f7y1zTfbjg)iFkSef=wijuPWAY*zcGtUaq$v$1||pH0*cGxt1m3 znV32Zygj&FqxcZzr~GL-VU)jPt{^K>pOyn={jA*OqI9zS%dz}3;~APLKfHpM@0La# zNBP@>RT{1TjQkt1TuRW-PON_uhITW)Tod^#`?NffIB%c)ZzYzWmMexn{Bva@PRkbq zZx3$LXgzI|kCseKSd(C?UAxgrs#<&}Y>{VNjNkCvB){q|s;M(c5ve|ymB z6zv5T+OuPB#qDPM9di}8CDOO8Nu;Yvbegw+iCff9*ek()Ue6!)+k@^Jk-x${7^{u) zcg)qT)$KvPB)k^OH>3O=bKRwhcxi)<^YYi+k%+foDmU!62ir9SJd*0)yD*iT?XO#x zsNVoVv;KAs1A@|bl!P~7xo^fxwNe8|`5jme81~zP&02E;PAdtsexnA5MnbkYZ`vHJ z!~&7lNBF?_)gIiYAz+Xc?p{*@%D{UyIHbUFe$XE5)DVDz=KZrp!+;@Zj&IcX-dwZ? zKD=I{PD!l3zihl#Y?KaW>3cqC;xsrj`)d_!5_)0A3%y@UV@Hd6u zdi!lA9Q%X8Z0z^>{(#pnX#>SKw-0~V9CY+xf)F(Ndvma~jq1yBPA|uw@*DY)J%`%Q zaC^&^rYoGwXQk2nX6RR^I2FzFyJJ!S^?G zp3Jmhjic};)Ti4gi!|_~X|8UOh5uRV;0gL_tcY;3Tb4WgZ z;!5)Q3G#aRPT`j#ygfvITp#t%565`v9!R>t7BByo99QMXemS4ob7lzpEovVvv3*uQ zmP75~fc;4Ue&qHB; zob%{R^GX<>VJ4JMwXHUYM`t;E~slpKld#|L3(3bPnz34Ea^7h&FI7hins% z_RBfpe8ZoizdW4J+n4(*@Y-NuzSwVx?=Nb@xrL|>vKKV`HTNr+nz>K(N85o2Sw<56 z2E35tjS^3p5U>fSa8do)mJ+%QKa2c0&izNDsziAC-l6X|Mib;$2XjUJPJqq(FTc>m zC3yTf;6H`bb~C*(@<$W>8GZkQ`sX4Z9v1P++k>W$i1>a6_H_FZ7&)DpkNy$)c{q*s zTE*=%n~3+EU_ReJ#*=}gzte&&m->(W8u1EyOMI5#^!!puBuOr16ONZ%#fh zkL=IoL5%`PuW#Etji!{uc(XigYJc87L`NRx-(R8pZS!KZQSv2W9uJ69tp;vepK8@&K6mIp1(Q26^1;JJLM~)o%{PgGX&p(R)b^Lqp{{GSiTbd*FBBS+313!MSr6s5SPSCb8 z%<2YiTVmsM25{QE?YGrK&A=a7iNEq;T5(45MgErn`Ri_kUwzPD}Q=Wd{Zw|e=+JFQ(c!>>za+aY8x}EF0%WU;gM-e zaH_D+#=lD^n*PM#syOLfO&=Y8RUD0dRU8GpD(*sWw+*fo{!Kj(dEGzl$sUV;LY&=e zv!!V8CN2w_G{|Ec@npNq``q9*q>rRt^UBOfZ7^R-hj{dC#?-S_#ww?9b+_s%d39I# zKk?n%L;F@*Jjl|+3a8fR$>L!Or}0Xq#q6v73(f};JVfc!=@?HI4^lXdB|Nzh?dgrl zwyb3MgbQaFC>d;$m(aLLNtA`8XbhpddX`#{;q`G6ZuT#sY36W)Y}*nVl}ZGmk5G1I zdx-c~`UOOdl(*g@cLez?U=jd_>~6xH56QY%E%@1K-{h-35(h7H$Ynl%%X{`)7%R`Q8b zU!S5;1w+zh$s;+#7^%!jA5|qY4fn4-y#i=KFL}2 zzXKjs(c-2oqK-F<03wk-`AGzqIpI$Mxy>dslb$z924yM}wI45pV&MK4aBM0Q*-uDP z-;=2BoIxUwmbXHCEPY?oq4;qg0 zM@oSHyg@ME6F2=+%IfrX=3m7en7JTMm1-zfFBnDVEPwCdAK zGcoJTpqmrgKO*&|x_+~^e;@S!#Uk}r`cA2!!2VJlPr=+}S%Z zJkz&*V(SCVzJcBcC)##5+}}_;(%;rPziXg!!r!)Iz7O^W+ac33+c-Md+c&Y**WW+W zTR+h>*45NnKR!4xIO-n=yM^xH$lyTb1O6?O$g8ibsj_}_XtH+B-`48(4b<;JKHGOy z?{E*bwLUmJUwgkF`FBqB)(*EJ56Cx8^iB*-bRo*;nIV+%8Zxc^$@=?u z4RrMT8YbH&T08nj2B~c;r<<#G-+OHdxhNa*obd9aD`XsZRhXKQ?Jl4QP%h5nIQ3I zJTt?17vADrVa!_@aMyY94{z4RUliZz+WXz_ zq9&CUmApP`P`5qUi@kJ#EkC`Ge$~S-E`W~`^?-XBF7qyalm>1}+tXKhzOOyw_*cpQ z+W9BxKgf7aSSEQt7Z4xczJ@&I^^8egUmtJDIF$AS?KwVH0w=jwu0sx6QT{V6CEBb% zWMq9``vHIDfyEnUP(ibgLvkBKggrDHTgQzm|N}&VMP(uQtajuKtZIr(1UM1!0E0 zD($#r2P*ym_{)?L$UX_0QV8~me@N1QBiQVRxD%fiw3J>GH}*E}#NQ}gTHf@wxdrx= z?~sbxzTIl!yQRHc@KxIo>)Vo_2fp0;a?V30zIjXfg5*=a`bjW&s^2JIGrvUElkBUq zkqP>KjFQ#PM#(R**zuU`m`|JTGx4#X(^J^QH_IQq0{uO5Yaylj7THlE5 zAZvZ;ykIdq&q%(tzJF@6V=n(ICce48@0j>%k45=ixB`Etj326(3;|xg`y~Gk3;$0| z^5*`1Uh=KwKPLIq4)h0ADmi6lDqrP`_)$MkDqqF#G|8LurTsG{Yrg9JMAm!{n(Ub4 z<&flCk9P1^jJJgG8x8)>LGhm9>)wg*R~e2?jp4Q4L$kAAPi<V3u8b(<8SI{B`2jmYxd)XYnd6S=F(jAnM%~2#M(_TL>n5YnnC(0>EQ5SO!Tc?9c;Opka zO8nQ+XTIR>S?}yjVEFR)$OotWzL1Yq@K528qL(^RlV%PHk&5BjP-We9x5hVWZ+!h- zyx42<&3p1nuZj96-{XteoK-LG|AxNRUktAQh|m3)OyrU)!Aj|fAa9rRg97Dntw?@8 zKs-5p>wJ%V%CcYl5heFm??b54Fc z+;`%5_)+I~!p9Gs2#?G^8Sa~ZD!hN~>F}lfr^6%r-wQuA|K0FIt>?qXPrVyXnfY*N zWXFNQr=EU#Fm?XY(43a!;}U3=gvy);2QSQjVsPJD^t1mv;W@1s|J0k|vh$b1 z+ux({{x}qxkMY(XL_a6Ob6dY19s$1m{HgGqljp%Z_47xek^Kh;pPK*lVCd9|@CCe@ z-+MDW_vFcNne)Byh56?}vph7X^@+hg=%VcWarAv696E6-T<$y_4$Z$5erhf8yet$t zabR%n$tMOw;Q7M*Z-@8Ie-nH`50JeNujDyj2$eyX7ajr)^p=wN?QrPbH^V7gPlo55 zzX-lAhR4l(Bs4|PgU}(*+H_S$HodAu#u+sRMo?$CTX6lXKj=E2)9LN1yTD0C+t57{v~{1i^d9G zhYx5P;U@JB>5fOK-BHF0UIi?Nw-J5)5PtGOMRDlEjYaS~&(&+YUOQ&=-+2D`flZsM ze3JZ6`Q8A0?-lswJ>R>teXr*RzISvC-#cpYJ@QHAe*+f(Q~r7*AH25fhoMOEtTFRkeR^?KHyiurMN7P~t9L;dRO@av&$cJ*xWq{BlE+TpW* zt6v-5&-!T|F>TKj+*F2Et!r!jwAAx*jQmzs3JOZ9az%$86@?8X<jX;if zA%{_Zl40bP#Lx)jfiC(lOkjsEK_??GC54#Q6IU(=^IV;S@sl;yPwU3V6+a9S>^F#6 zW5z!4M7nB9#1-v7t{AN^V}I;s>2p0H8lUKAXr6vPMoRoKuBA=C@^OX6eaw~VxtcQ1 z6G~lD7;=80C`7g)n~ED6=zlrNQyvlLkgX;lYPvG^xaeo@u zf1~kXuXmnJ2-#p)<$t#E?CLwvziXx*a=|_q;kRU`U&97u^9!)~*~P4X2=*g8D4%|R zJ?jq_vwo%zJVG{A_P_is*b2V8ZHt8ncbJX}XOu`#3Clve=x=j@vJbytYswboRf5 zOpu73g46+4n?VY{_ zZI3PJ+|k^&wLMmE=#urv7DR6v+FSkJPPU-a7wlXhM5}A)>TL0LbkuoUTDEyRn|y(~ z25(1aoy2zf+B(_;{?0)2;ziq9eXZV>=51~HOWX77{(UhRpf0~)QQeZfyt@3nyu8JW z>hN#TlDvoWmoBTT%Y&RkuDd{tjwWxw*Le2`67Y5S0*^r=SSjFZxhuR4a(M3wK^lGa zUEAum2E464VI3>LyTAm&G;5%xA>j2l@k!?ow0E|5cKgwsaP4g!o!++2yt}}C8@&Ej zU!dWxFf&ZUWwF|l1Le}m4QK@11vXUTn0=N*@0uyBY|90@o`GdhF|Ng`U5CgcWF~Us z%e^$@UK(^|Uy0*va zqL?)+*3=11T^_b-&E5_ky`y+)iTvfR%E@jNndSg#!*1DHmUDQb70mvW!^e+;*J877 z<2!aU>PZgANui~@2zZ7o)=y84AgBi`GjQvX^dRG97vS!rX;HWYv#(2fHMFAtF5ott zUsy{7C07FJo(lNhP1m&4Jv}|MnIp2NXSRdiGDh%u)=p$y@ofJ(0HUq^3>|_{6|Wymo$c){9rN6n z**sHimSh2QbMG2c<;a=>X?+r9GxjR~M9`LZ&jiHHGI$y!(9~NGBuqObwFLW^S zggm-W7x%bj_Kev5%^c_0no^m4s~vVh+lTF<@B3(U1`ou^(B*VymvuW?`f_2kZllo+ zb)C6Qbvmmu;;B!n+t96TYJ};lam-X_)9iCnxlPHgRP3`cK8)f@hOIJiTMpi)r7}Od zcfk=W@v^%>1B2l|f7z!CFSBKBVD=OTBk%69;eM%v@4(SkD_+}xK%sHy`mwxTlh==M zEg=L7`Ninj##rwzEy8{UBpGNZjb(KUeep?><7CHoCPw%HTnZ@aoooRe(y0Ra)BsDR z!Tpt&ZR{7>taraX!Y%_qOgB2&o_&dSKJKRMRCe6We$*{wU{3;%{(6I$ND@Tjujtm< z7UD(ku?c?eLj`3fUgwbjAWyhFd5p$c(Z~@dX)ZT8cPpMOqfXh4<1Tl=ZKzh`Xr(U zJ~wEeNN1g*f1>BpOwgl$=)*?h_$xo){O3YG13majllB5X=#S{R{xUi>;sk3go+##=g=$KElmG_*^ISqmQqtuCC_tzL5=iC+p-(5eM{x{&Rzq zCzHR8=lU0QcIG8r>8CX7>(aciC;EpyoZQLD9^j`Sz$Q-Ojd`2*LCas(BFWx>k2y+ZqYm-U7!E9Fg{1ITOVEhY2|KQ9kk%$(f zpW`I`MWlQyy0J{q#FqYS;s^d!5P0xGZZA!YR}h#rhs#6lPp4si&Wo_KXixJgZ#wF8 zBkYo_*T1LD49=TFH@z4_-VlB3H-2VKMi$(|0?jAz&-Cwsx}2{EaUPrx(J%u--@gi~MJL0*yDF`_Cb==k%z5 z4LP-fz_f&@zhXU@8Jsqa=o4f9HZ7Ct+~4-1y$kJ?zpW+xPIH<3%>ley*z?$Qj(4Cg z+LJ%pftUUU|Au~NM)j-gO$?B|-zI(2J&M#fjb~T>IaCb&bPBwj&!hD((Xu`CKaeF1pC4#1^^N{Bg3uNOhz#_K^_S)cH=xv4mRVlvFS;HK6|&N-*(UkM(pT>; znp?!RQ8 z8~Z2xo9BxTnO~ATyr}#mU~};G`G2VVDcSZmmNPXT)-xxz3L>B8>shjmq7C;C$V+&> z-eve1$y?A1JRd;#Px@|KGS814JU_PQ1|7+>|1py7B!5DC$j`^)^M%)4Gs2lN#NVkmmB-CSg z%X-GwH@i#5lQI3ff?>Vp{^8$3`xE%bKAN0Vlz(7;o<)`H3;QGA&m?yg@sR94C1!v4 zzj=Nx*U&=j2XZnYJ__iE zX}{s%{*xTtpJe4k>Ce-C!;#GGMgGJ0CvHxkMfAYu29q68e?oqn9&l&5(R+mG$)9#R z`1*+P(uf}Zlxg(0++dd5*uG%5+fDP0({q2cIba>(j~JiYNB!}`H1zU7rH_hzs+1xFR(@kgZRp`eaNXjF0hh zvQX#yP4ox*W|F)3X+A*Sf?#H5G@e=cA^v+Y-u3yR^$P)=2!wx|{rwQlC*yp;`eX8U z;7#)Zc$2>aZZgcs2lMvoO-DCgilA^h|ic{#!o;!hKIb|U%5UYFV$V(K{j1jhLx!0VVV zyB#xPV*9Nj&AeV? zz7jsVUXPit@cC`YS7&&!h6AJ2xL8|M?G@kIXa0uhIK&$}Ujk{`)|X*$Kk)51ikZcD+1i zygF)%SBCR-_|NUctLXk4>zjGMy%3)-jJ4n1&Bamc6OYeyq(+|?+Xa^iswla(;So!6$I@Rt`m5^ zi}g>O7aiTm;>9z>i}Ziz{kJK8A)nF%{L_c%=MRLB#xwArj(A4+?B$-neUGY=8bB-=FjIQ=V_Xu@Ll=4g2v!_`JbBneWHBfBYBPkH^k?Fdp_# zvGXge|D*TgQ>WcHzD?o&J~e)PL;RRBmG8gQetdL%GoAnB2B%K3#JAY_k9B{G`Fj)l z?J36iRuG(^Gkkz0_J=e-GAB%gg~a|l8~g993H14#VgEgi_KSvi=cEluZ2#?aj;d1C7AP9AEdD_f*-M;MF%+lDee9!mY?AB15d?j_cU-%1NSs=PXnL22I%k{p>_b@a$r5bxc2)Ge-!!ZOMR8U-TZg|@bmI< zf7STWg};d$Dqyqwp6Yc~yZh$m9P8_2tlwV!?8y87To?HoN2RPw_{+YW-oCyRcIL;g z{Qh^5x9x02^|~3y^mWw<>#DIEK}i4UY>Mmo^uK0Ye9TgV-+kEmQ=gk)8c zVMr8%WcE)*GW%$cB1@P!lQdDzAl;2P8$AcjoZECO=W%te3(lgWq7$R-g{})r+fw4q zSfQsPw9%M#v`Rn?dK`r(Zb%2~bbZZ(n0s#SLA|s&>(x9qU{x{lBVdK~dWxRoNcaVEVcdAOjd~{N;^CDQsGreN8%7?bO*AlS zJ24(-dTN#P3g7(NlC&$mjNxG?ywGqch!^;y4h;twUsiQ|qs3Hz5_QES(PL1Z>M)v! zKd93!U;2Ot8z%azkNW3J1p2QPC)xz!cfhU*XL))jg`Bx>c*MaD`OD%GT~O_=Qs4LcOTN$S`gBxzK!sM% zpq_206Q1f^ncBEiWO%d#uxT^E#yU;UaUnK)_K=P>1ZJ%Cc#G`{D2J21x>YZzQ?D~n zr>;oXG-{5j$MzXEokl54-C!?tJ(hBFqfst0mTuzZD#YonkVZ(y zqq5jIxt_^;EqNaGW3;{)!7wtjj_d}B0e>&nLm%)l(;EF*tdXC_n4ewLjK}?F$BosV zW7Ul_>|P8PYsgqS`=ofI&hN!=^C~W4g^CZO;|}dHF>0A6ej7agEu2?_H zU(|ED6fK(bBA(Ngsld@|^f_G;qUfUpw|PS|ogdR#+IwWZA^L%`0K`@XQKwBny!Jnq z?Y?ADt+hS=7}9daCHh@Of24`xkPT?Pqp=h_N<(B1YoFAvQcn0Lsu+rumi$6~CN7p# zN&9`-{$I$pj?G+uhCpv3;n&+efi<>e=j)~HfBd*k*XnI67`i?&t;C&WH zA@XLKywaeu@IQ1l)Cv4ucpdRX{hNwx?=b-_P6iyUFT{V5g$|GK=jCUHj^o+S568sc z*OC3C$BTAn|5#j?+%K+KxRCrNBGmsAfHXde2{mz(08+=sB59+b4QRhN&iG%E?QBsXe*0wfz!I-lyV>|MkrvMsf3Z#;oHjKPpH2UbOdE#vhRF zmH(;nRXYWr3P8N}Z^vm*Q^MMwek=-6#}6}`^nY2lm+VI`R{gI>`~Mn8e%1c6Yza5z z_P35tK0!L&QZETDmrOaJx~ldBzTI{_ip%+F87?|s)x0&1N4%zNVvaMWHCgYl;@NbTuq4YT2}Sef9RjmJQyGja{2+tMVIKH&*W` zSl8|?EM8MwQ`FSZy0UX~ZBPO3Vmenq*YT4YnGQWP6Xj{~>yrsUjD9^W< z_q)1w(aNs+*5!HN!C${)QOr((10pRV_O)MjQAjttqYP zYFM9+FdxTV|HdJC7guPEKNLDH0NYO86jU0k#Mrsx*s zk43j~b8E0^^WqBVqqUS|-nyA|m)~BpXxln((c1h?t=m>?FRbd`w6P!%cg(V#wVS|q zed)@_8@dacA#V$G^>}@8L7uO+$Y0;;gMDhs>)X~nzJ_eDs;1dnxPDDh+uFQ_)|#gJ z%~kEio4nQ4{;K6I#ZB!c8@E+@i;CB*EM1$wxpYO>rq-6O&82uPUemRCRk^>hwWb^T zD_fUe^LXQ`@|H~-7s1|D`J39-S2t}I_S#-lz7>7vHP)7QS?%I2_CLOwx1+JXaYjP1&~ zqaUA~^GEJoU-}Y^lfNLJn@BCj+T&GB+dW;ImYR(Rx#0IBl>u?hHQdj`y>XBEBrUj9 zHl*I~c~(DS|MSHE=J;CbH=Hkso{7GZ6NtCnW6;xBM-23@sryyVqvO7zzhIYjqWgiu z;d)BlTj-_%eoo;gH|&JxfA}lm)Gsdve6ArF&9>l)zQll!{wmUH=gZ@0 zfEUC{OLQ$aL8rR6USC98J9}RpNtn>`!Eq*-kV7duh9VX;@znzXY!sJjJHdJ6;Q$pM zs1oma&l`BxXcVW!d(O0==MiyK3!7nFg%tELe#~&W6H-821-)_SEs*WFWEggCN-heM z1Xunq^u^dWA(w)Q9>hC+shsFN5~%z>428y{UvnM!@6kRUJ?(Ge!5_X3{0D#&KWDfd z7*q2~yAJ=)Bk>~oN@;@Eq#QJ6ym8-<^17u7)ck4&J@s>0!d2Tv60SC>C9?k<*-xc3 znbPy$OaAe6DPGjPdRFrJvV^Po^qTBvKpIfNHv*on21QTnP^W^Y_;UCbS^q}?X5Ye< z=<_fRm6PJa-olmOS0vmn;Yz*<=ui0$p(aZFBnv!G!lwYP+7jrC5$H@Oj2Dnk)OgVpOTw{#P zyAJ$_gcCo;)$BP5f5-wKFwh(A`$ty^aO=3Ka^BLo#`)rs@KN(c!l@s{3kk5r0GCHRT&^Pqc=V5ZfLS63Q}Vhw zHDg~ez>RV>Nx0I3qHmFKtGq8txK-W~8FJLPd2-y<2DowDpoFV&75#1rw~qTg3Ac_* zc?%WFU1&-v-BDblqTn80W2{MD6WzSut#Y0uJ(4dH8efSe{ymK9>|PecjUkEL_gQa zZg%HE9o03rHRER<>pMDbtyx=Ft6-?v*TKf#Y(mC(m}UuR(_et)vQtJ6nNL63$eXrs5p_gHg-&jc6R z^EJtzyBnLutr(AfzM`&;ejac2>Y6nr&E6JYL+6_I#;z6&f0MgEZiAL~Z)06&dtD=E zOb7(_1fQZRWHCy(vOS;(Zgnh{+jtW;^9Htc)OGlM4WoZoG|=AY?eww*{9QPtu+nNs zQo>P!WI;nmXa3Tg*2W#O7ruGp#{I$6+u6XI+I;feyqJ$2>bG0(&|h{H3+yL+--{q3 zQ!P_ZS7d@ZRpuWGDVJ$gWwg&Hm`n?zaF4t@BjcDtsCcF7vv5Nxmq!8Q)xT5RJSX!p z`fQEjoeam(9PIA_r16cpHMMlf;On7^p{ViI^;y}T`ZktI6}%gb(AfA2N@(KAs>8}Q zL816n^w#=+m*Zcr{YMsr;#>8X4&kjfA>8VKqA*{rwA*F-QS!_7&sZ8*`S$@I&wdW+ nKbPh3WIu8R)n4V$`!TR}Y-)10gWmDO<5GTNgP!DNZU6rP^C}jX literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kde44e6074303c68d4b290ad3929fb761.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kde44e6074303c68d4b290ad3929fb761.o new file mode 100644 index 0000000000000000000000000000000000000000..9d36fb444e4c284067ba41ba01b48c29d58b26a3 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+mFh-3m%)>k+fiTR(Gpz>*$$*4FAlS@^mee4D9(`Jp zX&G`NM3%7)1{1RzayHJAiN{OJka2c0<7hT#B2RpX$H5HNWV1e*oD(85M8cTC9j^n4 zaqsunqx;^g1_AcOhNYnU-unN)s{U8i|5tUZ@0XXYT&`;xQ$=I1uvkq&8O!XA-Fx`0 zhoyqj&3dW_Y;`}&cH90pz0mgA>5Ls3vSs(ZgV#A*_MCsi^*mm`#yy%&RJ3%xw(|#} z-;QHfKir^)KB%;XuG!e3|7&A?M?2Ud&>rr|V1?hy_Vj;P!ruC&&JIma?WsO%o7^{K zn@D{f?n!5b|3~-q52Kx~+1{$d*dwQ&8LT+Je{lPyX9icEerE7ny5}lrpT+CY?2|8s z`%WAWKj!>S`1t-4;gNYK!+rBkh4-yK9lp5lba-UnrSQ}9-U~n6dOm#o)O+ES8IOcU zw(lQ&`k7}2Q|B!X&2ITl_`=-d;ioSBB>c4V&2aOHi{ZYNmq1q-3LQ8wxb4LL!KdaO z4_BN%5pHff9q!xzW_aY`i{T5~FM?)KsN8vA@WQ;kgHNqSKl{EDp52P^PrVr~KYua2 z?Glam$Dzcr^2&Oo(J#L&mVveR`i6AJW7O^_V}-8*mc!eKzJ3Tl`Jl2S^x=kL_?_qKwVkgWGx~2lfBe9v z%~3u{{-=Cz0KWGMeDj{~-Pyj^a|7QyI)?8ZwfG+Sr1HN3i~lKqy^#-I+xf##q-642 z!w>3*F8oXJRkxj`z5C1Jt0ONh>;KhS)}M;`adsxVI{ZWZ>g(|9p=@^bY{{fULk-%Y zvwx#s8{WtIX&y0c;L#PpN1kK-k$+|V7hln@zEi^b&)vta{@X&Ro(wL|azFa7Gf@32F|f1MB-{x&=GZUffT39NtQN0R|d3=KcX4*mK; z(Eg94(Af>UAEmhlo)2By`KoCyiS=E`^Dg)&S%$$YjrktQ1Q{N>p zPviRUG(PP0&T|PN8|AtyjHQ^YMgcx3>jYbYmjj(|$JK*Z z5?<-dMZ=CN+kJsHUrPt80w$NRI%b}pimR?M+jyI%+ZeNt!;&Y9&B^Sz{j$hwixfg< z{~%-t^J`s4Q(I$@&2MY( z^v!R3e17Nl=C&>Ev3f(7tUo?KdehL}>i2fC`JKLC=X@brT|-xAi?^er&fC(m)!W(R z3)D4uJ38wmw$sEs490`3AEDsjv{%b|D86jrw70$tC*GN>5WV%4rgV4+?_<;=y%rm+`R!Y_boe^+SY2J) z<8@KYs%5L{1g0(zTeW6y2an!SJhepra#!VKH;PQNfwW<_Y%NPUJkbhff5zeCN5N~c zS-0^WyBYN)hvTHsQeFT&!xihNCr1#}gO%yH^+|Nh+1)eC!EYHO_*`ozvaWcxe**x~)_#T#L8yw? zVZ5lmdVP6md6ByW`+uCHh(d=XM7kKSEd=s+2H}e(U5#Ehoh~%DcsrZh+uWV)?JXU1 z-I&=vZ->vFzi|Gdx$gCJnP0e+6?ZkaG`c&wJ34)>?$)l3PIsekOLLpgy{@uy<=VRS z%c|FwSFWyGR=aN5>e6MUY+j{%p?jV;(9q=WXvKlRBVIR#CSK`$1)qOdc-_wS0&ZX6 zVB`sTbe}Hnam(x(vHP1j&apM6GW!-g?1HwB*hSy>(dY~wh?Akq>C7(ccCz&4qG;Vl zqZ{fvbDQdPR%OIfpH#P@Tig8+rmw^?Q=Lt-&ranwCA(6w&%*dHiYpnm%D`;}c$=Ea z{OH~VN36oj?g9-AhX4F!pC-J_mbH%AQyh%EyT^w6#S*?9M_a9UZ3O~_#-Z!S@_J2P zKf<+?5Gdpqqh}jqy*srC`z4TMprJID)h+bJCrOTz9p8}{;RkRjps07U`E*F93g}Y< zER_cLS6#NTpJ%h)ef9{u3;;3R=w!Q}O0@HFH)f}@<8Jn&ZXpAE5_t628^lDCAR2#J zx6U>bFM5wn@N*w3C^PUn=VXThb~{?I$iRU#Hnf1{25tAf{>5|_(GMiC*AKFp!35iH zro6G8*N3w0g+bj$c*Y11`GON8*{CHL z{8eEa@tYK!1S@j;CMdk14>Q8M0?tXqZ|d1d5*tiog zu;>gIf!+>!tZhVp%*pX~)CIl-csKBF6W)!wz%K)SD)4y%KP=i$MP1{Eb$uf#J^wQ>leVk9(su6<^&q1>1 zw5Wd#Ikm#T)P$(NVm+7_CEBpEqQ-8fXGh+I#Z?ou|=C?uLe7Ah!T&R&`mWE$df~F@ z+vSSsyOQ+fip_66KhR$48~tYlp)Cjy8R!-3FU=2bK&h`Rv%J(_bUhdNj^O$Uyc*;QhZi=ALlnsFQoY6x4~qze$)KXdA!sKk1scTRq^*S=ny|&Qgs@2 z0QD}qkp=hz-At!lPh!!8@sOX5mS57w)+d=CAb-RBnMeGng8K0tu-_8^P(A?vq4@hU zIV83JfaE_Ki|^1U&Gr`i8V|2AWMwjbYtd_&1sPx+3+`O9k`P89hL*)Ju+&S60A zzhs{q`zQRH=Zg-RUy?k$sQe>fbMW=~f2#Z`+4eS;Gc_L8GbgqRBA@2#S+b3y4fhYo zOL)HCW%wD%Ti6RcA3*p|`Yu~C&yO8EKep!v9m%u)Ig;%pe?oi6&&T8Qh1Xr=d?H_Y zy%O??^&~~mYiJL3Ybu{f%m^l<4u%HEKdHYI3>U@QGlJt#H{xmf8u1CJ8}W%qw~Y8C z)MI$dddAl`yGzECG5x!eVZG-5;onUA6ZpqdG&!j#|G@k_iz?X{_D8;-N$x1(A=!UQ z%>M9y^ZZ=E>A5{=PC%c9!Kpeew~X|6=D*YWH8qi4j_~|uXd3)0IWy{CLkqAU$jOBG zD4-vv{f2}4PjYmBl9dysKTrD&M>4k;`48WpxH)|m(F30wOm;;53Hfb$z@6nr?-8OW zf7<2X>m$ZXBYOB#rqSPWgIR84`@&srH_bOr&;8NnfOUjFVti^J^~Vp>*t21dWRbIx zG~%zY7xbci66HI{r)>j!!7E?i(p<>T+|m9Ze|lh=J_Ym<7v}SDWomjLTfd*`lQDrX zKE}(*LY?n7(I4!aN$%pO`2cwfgPEDpcxL5?`0vGd*XM`UF9dWV5dLlU_k%Q_jPn8O zkICPGH_Zp&P5utNVLm`Vxk0o4!{4WoeR;kEelvoG`2cwdANBu>8s$^)FBFQ8FW^gv z$iJ8yb<*#-1*jukpbkl%3$%Z8B*gSf{*jzu(QhU#RsMPr55O<9q2^~L=63?>PD*xaOz>ir=xDfXG}2TC!ij~LtgH$Tpy5^>Mn2~^kc((&K)Bk=aWms zmkX8Vc=iJF$6YLy{2Tr>j=fPy{*+2B{H1=oh$mk}jf!Et7)F)HgK1>H)FY^hcux9G z#CF6O&l3}=e-+OYf?_tYYK(&XVRtt}0~9(R!T#%DHa<$^@1Pm_=6wuPIx_maefH! zI_AqR2aXC*Y*;T&Vfb6LPB4+lh5RRs`-qc zbFcKv^O>ms#qt?`K5;J1e#86)@$I_#%b&sj5Z^xQ{AF~$;z)kki}X|EBbaY42l*pE zUx9y`^OXg_8`o!AudMlsVf{tC$ijM!7F{y^hkOJ=UYxerrfG zuh*EbgpaP*W9BP-ep~Vt+Kf{X{>b+e%Koq?&L1-;0~>K+zMuXS^A+-kxcSQTG4d7n z^c&|ZSv+5H$In+FPnMhFdo*7O8uE?N`3mBxJ1d&63`2jPJRixT^_9m<^k>OO5HC&n zhy&wIcZ1(gkdI`E1)t*csQl>8%}2<8{=dve=9=Qy=>0ck7btT7y+6q81mXDy&hN{* zULG@E9W})(!}&V==XT;%bpMU@&Ai`UfX^4k+HddT;;8kB$7ecHqtA{rEwA-e8~1_v73@{+#yXvGX2` zhy7FR{0i&;=>7QpQ*RvKrf`41KYn~e{Frh--+!t7_~`g%I{(QH-ao|>-(u%K*8MH! z?@jEtrx@c~VQ_-Z@Bx3Uz=b&ZvsBUcpgRmpGo7 z^8GjY6G}ARO`go$s8haregX99(hd5A{nK@eK27_PvGdnI;C#QBV9sBW-F zk>UFRX>1i8Ht}M}XK6{0`|12n7NK~^-?0AR^J0G9E6y_v=ix^|bQ9;{$8a7F%AUmc zajepmX4zk->DceHuYrL3Gxnc1R!ym-?dKP_t%0G|P=>zjbs)OHEmi;#0uSMl^ zGLny7l)S@%b=`~;sL znDUd)pbn|-m3tbvr-6GKxTk@88u-*TK!@iDwFCH;1MB(uwcmgEqsU*p)K~S}O@I6M zKdTt`SB)QC`0L2QLN=@K>0VcjyKhd;vA#aW`t3E(jlBP_b&;=eRLYu!zwFED?dwZn zXMX(3?|&D0+s>BNteJjHUsIE?rUttag!G@xqPU(Y1tIlx~bfD%+2 zXHU7pBXkiD)UmpTzgY(FVovfu^OXrbUV#uZi!NYSV9xUZ?fJLB#6CQB3O==c=)JdUGdvf zF3*cQJ|4-xC}1F7|2)?x3Qv5z$hClnV&b)r^1nHzdu4-o^ig^IjIGhMWF4B0!Z?KN zNk#N*&}ov#;t!mP+mdXuXOt}=GUIkL=v_||dXLIYBgMf-<#xr#N9A@4x#RT_l?#lJ ztV%KriDHn<{;5c2AMH_O2@_|MCdwJ4yAfxj=b)K$n{MSiuFiGAS#(r%Vzj-`bx~Pc zO1v2>^i+&C8ncd81*k!fqwvHH>0phnuX+e`&&@rkmo{g;n#TsLDn@<;tf*d3(Q_P$ zzvC-~Hp;tbY_+0TSBw}>*VCZtH~tC~YrjXg%Iq?X8_%{;&jejOys`rIGg@lH$fLB0 z21acs#^X#+t#)4Fn_pX!cBPjw{MZREG#m=z1^%c*!vV&ZRUO}GG1Z?$T`@`Y7*wY^ zj3(j_>U7JOKH$NIiT-U%eTinqdo1-TOP$7}{`nGt{%ggFHi7u~H>{IK+-jRoN?(*( zYQNP!;Nkd9NFFrpJjvR{rDBt2{vJ*pIZ-ymp!EPXe#ZBGxXO zW$ohM7B;M1X8N}+?Ux{HHumSS)T=CYWUJR-yNs*7@V&9(+l`_DIga#_?=!nT4OJdc zq17{}XB+B-r#e@rHZBzz9_;{Z+H|n7M$>a#h|Qkeq+<<%87n>BV!Hy$;bgCF)eGv> z>vYtqE7CQMnxpEmeTGe^QHoO6*^69{r`+6Vl#7g|n|Qejae6DcfE(-@bgUzk0TL^u z5z_IfEH+NAXYyW4o=5!{tuIC}jLfVfyFp^W-;4Fo2Yk%5Mt>G-j@X7sJIGGM3IhDc-2_dokR+ii=pG;=|~;Lwn5lCux?|UAM0O6fX=rIZeBQ zEJp3*WW-hDnk@7OiAywcEH>E^Zp{6jgnRqYKg?W>x4Pfz{J;HZ@M(;0#vPl8^|7e8XVi|(0}qPD>^;mF_0+6=OV?MIVz15k z*|&BD-baVw*lnva<_P?=5|8){i;iS1=jg)6P51%v{W!7xkM%$LWSmQ2=wH`(|2>K~ z`retbe~DLU{%3A$bgkTl^97nuqMWhgz$VUb7-={B4WmV>x#ilkLICiQ4Fo!i6A$@A zMpdOz%A5K9pSZv6v&qe#em>`g?>;}Vm3?o{Z+=^A`>UCAr|13Cv>zW#OD%6nS%4uw z2fzPQ1*zTmqq@9Y*rNhG;rD-DUe4I&GW=~hUf28mA5sw6g0As?+lS!4pf?m7PJ*Ie ztwUXD_ff$4!91tB{@ZNc2R|Az5iRBL(LIRY>Edo}q8Go@m0^NW5BEE{6+mvm!d^;Uc~QoWh!v=8vUIv2~qS>g4?{Ina+>tEbTqA-VpsjSpZ@ygQ(Lc zAYS_)%XVL~sMgvZe++3k;}ZR@pg+8njlSbg z26&$(QHZ=*BCj;4Ec_2$4Rr#47hXp^QU9hQ+j~qvi<1FI>kIK;Y@x$X`1A5pL&x## z=Z9nB@9W5Z(&I(DvwtkEiyjcyEL=!_6A|iv3LuS-VnR*aD1g+lu}In|XdT+`i8KCJ zWcyr6N#j$osE_$%dyi%OW3s*SUusV-ZEe2@llSR3<9}llh*8}9oiXe9%8x40z8CF1 zmhlHeAQ0D?*|}W`?uq?rzv4=PY;Vi)bYd2CjDQQ?Irusi&g(?(f&WjkzcjH zEL*}&x&5uTwAlPsAZjZLu1#*+Uop< z)(th=3)i%Ji%M3NtS@eAXkFg9skXYM!M}E0OZkrF4Fyert*s$^Y0J_z8*3|?>Nl*< z+f-Yz(z~H>drL)S&DtGh%UhN|R@3CKUfZ&wpysji9p$wJ)h(M^m*>~75N(TFmbTQ_ z7U%gk@qSm<7A)_oZ(W)P9{lwMi)*$Ouin;JTivm-cJ&i0TUX~dmeq7Mu4vhTG1|aK z+4{2eT@3|GyJ@_}4HfOhFBGvsZ7uAZnbDa`ZL7W?a4eX!5^iu$%SPpl#vtXSXdEn2&(xNUV_ zL+kpc`c2jCB^$jpHU8?QEhSCur5mFNCQ^&B_Cz()c1_c!re@=ZT=4uzjumgYhWoj=H|{Z? zqy?ACy3_|e&*_Kl|0VIiI=+_rP3QBXXQFT91mbPi81yvOVFUeZ>VAdu=(um{&)a34 z=zgGZxSmq?7P_f`pHnzoOVxd;)AgMGO*`TFAO1=>^~;L^pKAz4vn+U`FE!w!Peoek ze0dxV@Vr=QiLT`)=v4RC>kDXWXYZ>c2@_iWaGVJyy$K`sRoJ&1SuQaRCkBvAQ%7z&L?zv?>h-=lpzdfMN_gFkW|_zwUl ze$H?^Fs9~}b{+nYAn_vlDrth(q#QJ6ym8-<^17u7)ck4&J@s>0!d2S>30IrcQrUlw z?59eaOzHXWB>(tzDPGjP`l{sfWeHdF={4EUfHa_jZvZ@94T_%DpiTw9;>+QiW&N}O zvv1={^m!PE%1LoyZ{bStD-v#(a3$Xa^rw7>P!lD7k_Db8;Zp!tZ3*;63BMn3y80d}meNDox{L`=KP$7P%h+F=?NW#^)$`_j?T#c*X zhYj?`agRv2b=+S_xOLoAId5rP<9u;R_^A0J;na`fg#_4afXgEuF4tiLJo?8yz$}u3 zDS6$Tnz1h!;6}NcBwXo1(YHvrRo)jQ+$wLW3^{7tJUQ-41Kc=nP{P%?ihh@bTgUyL zgj>g@yoCzoE;OZ-?kKKNQE(5hG1ern3a;YD5er*85rD zkBWl#Nc*j@z}0&1x4_kU&qUhYQCHvG<_&Z+>=7EhE4^K94NZ^nM@}xo4>IHFj{F%< z^teWLvpWy!sIIxK8IO6a@94O-W^G-qf}v(#2OE2nBexFx@seXxKkf^3;L#9L)aLpC z?`K@{$hcV}cAQwSBWRDC9HDCP1ffV}#5h?^ra zAEVFKDBj6%9L>T04nP{;m|Igzmkhohsu+qIUtOP*?Wu2LsZzna(Fl!=ub_k`o~$~o zY7-QSUqx@N|2H}Q_1b@AK`6def9Vk3Y7@e(3@8fo)k?cfwjU+GZ2zpKft7zB@bT>D okp6R74o~(YS5WO$4!sWpTgRp*XFKQ}4<48D6C3nPUe@;i57`kE?f?J) literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke0c7c73e4b05f100d7e599c08027ab7a.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke0c7c73e4b05f100d7e599c08027ab7a.o new file mode 100644 index 0000000000000000000000000000000000000000..38a3fefd06385ef01aae2daa114e4e8963bae07c GIT binary patch literal 47032 zcmeHw33yb;m2UN-R%<~LAjE3afB=a_Xkksfv?M?an=BRyjgwYOYLJY!*J|MwR^oP( zI9`Y?WWU%>CK;aNBs2lyEMKC|c!FZ*vmM)UFkfEAFPWDR><~e`FfzdsG4A)Dx>dLD zy=n<2$@3)sD1BFTPo1SsopWlvS66qf+qm9rGBKG<><26g$x_DpzZboC@>?fM2Bm`q zss=2sUuHTi|C=uG_ReAK;2BHik=Jp(VaZ(dd*o+v{Q>u2O_#KkU%B(?(0fU2j3Q9O`5TLHh`s%vO9a(;2>0!e0HYnH`*y9H=^N$vASxl1{Q7VUyU3 ze>XeBgWxlpEU&sycTVzyr;Z%B|I}jSZ$5DUssCkmj)1lo*TLrV$A^x*d~9gX+sB8F zBG1|M^P$_{{@KuC*P{A}pl zH{TjMzvoTREDzoH;QgnLEIn{4gu2hababd0b)Mh!%1}A#I)?% z=V!-;?*IJpq5D#vA6k_9%Fww7-X2=?Gpg%jp&Z~B;cDJ|6#9R2D5vF>q1%tXG8B6A zk3;uANHl9g=N>q4>Ii7=%l+2SqV-=NN;v-VP-5xJL(S0n{GPL*Ssfxx^iD0s>pt6=%<5k4c&iaa3~??#?ZM9_ncaE?DMD2Z9rYQUmrR*;pamYw&$VG zn?vUpp9RfLp>o^jAm_eQ=N>pRv?%%bQ0V2KL9bWPzPE-JZF&bZH;2j}`uwTTfg>D8 ze^8nf(d2KvlwzGyUXOr-iMs!)GRQpsF4*HpS!|Ic=G8gm86c zwnB{u;FNfnf_PJ6qRzxtynvW6nFu%K$QiSJ1MwYVY{idJHkb1#vQ8jwE^a9aU8*id z96Cp?-1(Ek+IWr?pBF8rMKVSy9%U>#5sN=StX_}Bk5()O#)-v4SBb?#hFGK+mGL-W zh({T}{+pUIP<8@k_n@p3G#{A5hkpQgAK(VSX90f;a2wza zfPV@2tAKX_{y5<00DlSaCjrj`{C&U=0p0<4GT^TR{v6=VfGvRg0N(|8J>Ykp;ih{4 z-vRh$z^?=L0{$lQUC6hhJQw9(0$c@n1Kq$Qyv43%m>YY~=08SDHp5``N2^VtlMK=N@b{=O6T($ve2~nY@F$pU68{jPl4# z7XBUP@yKKrj#yau&q*vyV=MR$y3amyQ{9mrt^q}X7r3ZHfmmb_*zx3c7=qGhtMY@uX(2w+dUCbR^&+kKr z?~o43$J{@{{SWCr>6rUD+&@G2iO1Zp;QmFrPvCkErUWMKtBo7+e==^Qd2`$q(uGyGO7nM%aJE0 z1gf&<9Qjg`iG?#0m@*IFZR2aga~AWfIT#afF(FRPmB-2%8*#+3V{Y@7V~#iz^3BN6 zCi5obmScMQjmOHZl754ruMqU*G3d*VmD?qKsi0pk=-0)dUwf=PS<;sX`eH#}BRvgz?mhPapC|A)$H4aizg*zg z$H0Fb_|*cxIR^gwz~3nFJ7VDX0`CM~*g)0$Ih3yze4mWL_er^elzh;Quv>Ufj>_C0{@*D z_~(d!vcmtH82G(KBZ+qdZ;YED%1!B`%O52A80B9i`YEI7ze4me%8w9z#%TKEL?5I4 z`$Ru=H2n{WK1TVEh<@5=`e%qfM)`9@KYcX)i$ouz{1u{~F`E7q(Z?wN717TeO@Ef? zW0apK`dOpt-zNGP?t{WE;5g)YQt21XE%r<2{ zZNh$!=Asjr*Dh{Zk9F-jtZ&zbF7GbEy0Q$ygIl2%W>)ZxCzLKitK_XFtwH-o+a9Vp37~Pgs*)+V33seLL0(566Yx!&-It z&(81&)~~-yO3b;8_3ZoaYz|%dYEgJ_VM5NipF2kev3~pD*UpjOz3v=2^HNdxb*#VM zc)ck6d#uZT{~p7=31w{L?7Kzbv+tV1XI?Ue|9~~zg_)>7!bbkHp(s4`0P(Yt^H}Ts zadL6^gI}|eH?acb(+GLsyBu% zCYi$T{(y~K+_EY3PKg=ym{6xV{AQ{NdYHrK<4q$&50H+g@Zc)UqYpbrFdv35%mh4@ zjSONfIk*tz-+~=d0skxDe+4cT@V5Xjgq;@?E#QX%uL4|X4xjsKN$A{E)5ze@*~qzh zeO6YaOqSr0!KvNP;UFqbD_4kzD!u;oOE%CMI?a@DgXlM*vcHss*{V9Sqn zdwGfvtvq(E^jXdj)lc+uOisSuZAy$?$CW#Oh}bhB_OLg@96Ic>ApWd~zqnAulJcq@ zvJ#gRg_4&Shf;1V;c`FyqH`n-vVUW7hM(F$d)QP|lw(hDV*hl*aCQR9;&043T$0(J z4VuH2S;J*KzhSt1PEk(i*;)OEpPkuX4*Di4|Hj8cP2l+{uEQlW`tPN({nLl<+r(d9DCNvM^`zLc<50^~tPjHlm7J)toG@+77{RuBSM~>{DH0&@r|1AOK#2eZ_arkh_ zM93@S@{a6J8%{(y>cw8g0X@pqdIs+>Q~AjGTM#>|Oe5K#8{BOU+p{XTZi5X9esGz< zRTFMQl)qH)F9bhgo%63EnvE*WjgbFwlDm+~H-*k4PG_KOI?7ffUxnPBwKzpoA@AtyUFEvTV(j^yAv{nE_Pna|A#z46fW(Ah_)g)aWf)X<^*pB#Q7z9;AF zshxp7e?dP*zaLzPaU%(NA@$#)k-11K@WVe1!au$gx)*>_qt;fPW467QlNLV!5(^@Fj*gMgA=E{guOm zE|izt+CTVRf^QujtR}dsfAFgWR}BwtA$V&)#udO@hX+du-qt_3pWto7gSP;-Y#$!P z_q3^H798`kwpcvyq z5yt1|bYs%>@#6Y;aechFK3-gHyukX1r#k$`I*4ENtMaYwKQys4R{1H8W@ctG(F2s% zdK8#`m*KL5cn@+Xu6SG%SsIBMmG*eNZJw4+RtZcNVa-^HnUj&5&CD{vVlrDy%$kIy zx=5B3hI1Hpm+~C3NFvPaM~aSuY~W7i#@VIh<~_{jWUE;mG@3z_xt&d&KCjPRk2{%n zw<<8tU7byBjeeHf*6#D0b2W7N zTHKwTuGT_VtKYTT)4JCsFxxzBo$cOD?vBnon)mH%@u=`t1@6o*SY7WbbT#_AJ3I|s zh!9rq?)11CJ6nKgZri=Rxy`rQuLPxG{o!a^mzB8Mevf>)AEssHnhY2k!VPxr@m{q zYnRvE>QTI7EO6)QO3=+3+$}A;z3z@C-sw8L?Y?&E@f>PsZ|n5A+kAN+f$(i`ceHxE z4Ic?N!)?4RMqlzmF?4bVY70IBK9tST$1E@X$aG=pTi&3t94sak?N}`NI+b;VSc#0E zU0;S=UxxgnEel77ewNFHH7J(y?iN>`3s6H>qubTlnxE%)`5Jb6+VWPoc6YhGjb3+i zi_3?(&ehP~;;Q#x_3CQ$VE$`w@HD!x)OLA2zAkJD8eR3>E?&;G!t<|9co!6bNVgAi zP*GQFTc@$I>r1DG9<;m#b|rdWf?8 z-@0ktCYORir*pS7yE|hL$nMwS_WGKA&FyV`Rieru=~$P2q}H0_&A2&0T5t+rOlx?# z%~;Gn!OJI%D*v{b{S1zRD3QDym0Mjs zfnXrpSdBf1YBLCv2CFNnkIL9w&O1w#HHlo-D}6=9|3B(`P1I+M0V<|uK7#8c4$C(0 zC|_G%A@M1 zBfl_r`4Yzt%5qn%VZ~j|Esc)O?oOYl)zRA3>2oxCb~U$o9NV^R*|^oUV_ntO@-3TP z>uR>G+q`z&TDEkHqtLO`?QLjsbhf(k^H#YXsG4-8b2mJLFzC96-Gg%LGCRZjQ?NGf zC)yc8hbOw1J&*2*hLUws^!{3o|E#;Lh(++NvMTjWRf42|U9_+Z6PZ=O`RsxY&LfR4 z%u(^n*l8V{%U;vLOW8>syo9}=gBP;_9lVJ3>);&rln%~jhjs8mc1Q;=V2|nG`K(t5 zXR&*A@H`gK!E>2k2hU+0I@rOQbZ{oC)4{V@jSjwnRqEhbtV{>bWKJDCgXIA(QtdmP zIdt$emZpQJ(vYH|&tMnKT6hXOt%K9qNgX_yy#P4s?@6p*ho8s}>)HSceWyW_3C^iB;<0L{?y=U8iwdzT!&zjH0YU!s}h$3G4d4juo3}<1$3p$Ivc@+ zWSKr}u_oGC3ho0I+^-bnd+?a371wT}hTce&e<1QJB7a}x%M~&P7x8&V&cQz+CL~!d z*8*#mVzpd=>QgL9FUt>&K`-%t>pXa;DZ+jWsdijvQrYxwu_l}YI5&ko+YRGVx))b{ z3Jdx{sF;qiS5kqel`ztLTu-F1gI;?!iHZz7n95#Q#(Mqnc8a6O=}0Qc@AW6zDXt|v z-9IZr-F!j@40|S%J?>u=Art3hAh2H8!jipg7CEG&a>xRH>*M~UMYMj2*q|}wCHvVd zvSCLh=ImdlveN1D0b;Rj=+M5)n1bM-p6bW+q z;cnSlE8vOjQGc2>08**{qy9-&5*US(tpN?(>$hUrlExyxL;GK#_QhF~h+pXcxZi4} z_DUG_A+jk^eRvtH8-FRNhY{eE6#R3Y3dDbRq_FRFMeJ4v1Dnu{dWQn%Nth`xbySWQ zFb#}am^!W&b{b&n$Xa}v0j7?v#a9|&>gZZL4UAftI=&XB0Mfz~Km<23cA`6y!U!fW zo>|69b9}-eivO4Sd3Qz`VEA`hw?I?&8sg=?jiNxIr}&WJz~jMhWlljs@tN{(@tYnB(k5&o9;F9y$j-t^s`U*TBy5GyzzL?!b5il z!R*0!(*HnrBx1&V0-_{Z<`Qhd{2}3cyFu!I%4Dbd->|SU)t+R(OlGuqzK?NJDfU{% z+vh}mJi&e&QGY;y>@3;O8nxf4RCaFy>cjkmHrv@*3-8Z?RxnWgb7$-Q33C92b(bowD`E0R*b&Yr(YWt;9(giAcymxHN~bvUVgq+d-mp7{8e zp&S1W`mH&%@rvYOe1^V#{zwklBQg+3WzP{DZw+J;tl&TAw?!C_AaDU6?1|(1N3d{v z(fAf1esz3L<^!yRgFO=r^eMIgDvaV&4fr%0O_ovoL<4@3Euh0sj^Yu&{auWXFkl|i zZz6}eJvki5VLpIzIG)3_CgAjMBygAyz=VGziNkyV=JXbQy#0jc7uZLgUuGk|F@F}5 zK%_Gd!CuJ_#1psILGRQ>*#IMUvTwVk*Pmdg;aY7k%||MKDtpqOm`(PNm=%8*=O-F3 z)C7M9F&<%YuFM$(_hiU`6#klMr{Q1L_oRR7RI*o8{uH^N=JXg}VJtNPBHY6c+X2mM&IYWyAS$P{U8{PgCnR!QKTrdA05N$8%H%sM@!+4GO_===QJoQZEOGzm&*fS|=U&M2-fAUlsj--6j z$DztIDeD1JK-Fiuot7gCJ*`I(UvRMG5BAJ7w0B0LcKjiIvsC$1KjP<*d5jcP^+mirA?1@j=CGwCUB%CvNb751x)Mj zgmgyYF`)AGcP{34HNk(n`3&$F* z^*_<47{=37-S|lHCMIw>zD(BjSH!2-AqhOz8=ukIBO@h%>ZN~zJyUe_!Jet|v|sSg zo2-rJNBuUg7CeQON$eN1j3M142f*}Yo+Ws}KYQU03@ z{I}@zecXTZOzr-Q_ou^Ac{E;Y?L*_W*uaST(SHs^>3jXFCTruF`jehN(@_8HdHVX% zU+@<;QT^u__~+{CM}JG?2OVmE!uZsy<)3fhUudX*!7^?A)Ssg33F;3RPmgNlEj7r? z)#=mg&$F{fwDkD~`a(nf1*;78Th;XnA0L<7*>SDB6$W`Lb@h|}Sni{{qT7HUFn?^K zfo(S$@YoJ$>1|k^YhkNH58JZzuyu(Zw&m$z>v9Rl7>`PGPBX5r7+*ev^|l@R5onM7 zKrZ&z2@xz`mHjlPV1M6$Z>x0qL;4@L4j2t-+wK!&!PHQZxZ{DW#G3)s6W{0 z`~miO+@H?jbO8tb*u!D|;0bs-KX5vkQ^I}duaUb@zgR!hdRaT3^!X>)Xagzneg5?H z0WG~hiP;rte1G<+H^oZpQyJgA{)rRF-x8+!_A3(}%~q)#ikFMgL5;4)YhJ;UEt4&qd5f z_tgQQq|8Tj{zTqV=xIF=r(19K`jaM5d`Wu*{gZX}eB7VN;~`Ovk2ns;sqryIM<4W0 z)xmxK#6+#X`}}cnr*;0v!2tB*3G2x||0I|z>OaIwBE_HdXP-X>!>1O`NTdlQijTvD z8TBXCPa8Ce4wc)|XT{5z?R*RJ|Ey-C36kA%&1WB`|Nrj#{~?_}ZRv*vEdBqKfTjQYb@YF||7ky@`rr4r_dkt) z>UkB$|3g|hX@UdIk@yc|{QtY_{{uRI+R|STu=M{)0ZaeCrlbGs{ZIQ<)&J~o?|-M( z-|)Y>-&Xp67Mk;6`v32)|DV?R)0RFcVCnx00+#+C*3tj<{-+JQ>i+}wDO~)ukN*xj zV5Ic~pZ}l6L`F$j|Ka%?U$3Q!^;)lgrf&aE`v+PNC#(4By8Sk-*R=ihasNzCKU1Zj zG9erMGJg8}2?<*IKK~5edhK!lbk09r@I(GgJ^d6kKe~Pk`e#Ph2Ymk_cGuKI4o}8S zP-!w?tQU3b`9A-w1pRt3Yq~!EZiuc2!H@l{txVINpkJK6s`X991Ir*anb(i=0Nwhz z&p$gsyIu|YGY#^W+A6^+?G^N6xhoTIPZD#mAgRFR_xT+O+VySFKgS?1*H#14s61sg zD*0)BmB)fj67KWQP0+80=jr72`Li$_XxGd0b@uP|FSFHY>g)4Qo2K&@&O?;=rS%l~ zV}Z{9LH|NseOO;l!Rr}F%KEv_pB<;QU!Q+boVNYv@4~AQlhT>Z65!8s056%H!QmvT z59bRuYOHFHp5+V*QsVjqdsamC!+rz&OC8uULLTR*c#Ya8Q`s+R&wKniI{o?c4Bh!s zzO4h5MeAQ_sDD+o{x^92xpPxw{W!n&FV@MI{a5a{di@19zeYaUkH&Ase)POywV}Q> z(fW9N;q5cc{vGXKwqVjU(S8d*PoVK2A=*E6{U7vs*%`|PeH_^@e^KH}js5ifAI~Rk z0mxJALH3+$yH~=B{R?KKtQPzx#LFyQ|Dpax{!{pu!9LS<@kq}LwDA_~$YU2&@pcy~td^(#gfIdhfz8$*$#N$D8-Z$ER^K89fj>g9#oxEVr zVvYV3&sny|H1yQ}rM`-Pvm7Z)RR7ZR7OlT1J`PEGyw8CCukjc8Pwy|iKWKc> z$4jth5zB@=B>3kc=!5l+=6tf(zrgkskWu?D(a>i?p3(m?{M#E{PmT8PTwA|J{%C*Y zYU-o&Hf8>m{!#oP?QelT`uP|5`Li;l{dxaeFgsD&Hy-rH{x{F|f<_pZ38My{?Yaaqy5QW+Wz(sV59%^_T~Q3o!{#F7x{DS{zmbJ=gFG$ zWQxDnRDEeY)b|ggzl{Eg?GIi5n5XX_`tvN-d)#==J@_ zXm8j9<4=|ne^_r*JgW9pV8pkGcdWmGpBJ^C;k*swjq!X;f8J&|Pl@q<$+=DzPv_UT z@IFNB=YupqcGTnfU<`hWuT0{n*Q#E_eAT1A-(;uvRdD`mWldP`rvcB;`z}V`U&48h zQGeZeKj>fdQtgzK^ZYEl59k2I-)DL;l}++9R!8sv-fzPB^>n|Pzxu@JeM77_2v7P= z7zQ5efw%Ggk0KGzsqns_vHs*9W~T8J&qMk8NRQ8CEAaeEdB1_5pIb@(EwcV!(0Sx? zK$MjI$nJ1VMYXqKF;OtbuK=cZH6T#B}3w*tWo|n%ygSDgn5}uc5S_vjw zchnE__g4w#`q$%qRlLtiFg?%fsK2bjq+v(>`vkN3e)A0^&&>V3h1tDR;>3EvO7pqh zn_&|>GQbD|Z@QJ9Umy`|wR@+=ao8XtaTxeGo3f)+;5ptY z5m6uCf6eh`MARK1>YL+DkBIs>-tJ}bv;$S-Ed$;~Nyxd;U*mp3&c+Q;FWi{tqF#8G@Qy}+a-(BC{)(Km{p z5E1e@e+Hhvr|rdi?!>|6GgUmn8UFYypem1uvchwtG_&#g3EfYxkxm zQP(08!jr$r9v#B}_pD)N8+x?Dk1zwC?4!gF$&0fi!jJ+&o;5)^4_DxO%9%MHBb>so z#2@J{;tlpr!3(k?q5e7ERJ@3+h2!W?2bIL})(KjF*}b-;nH-Dyev%3MDUUO`{m`Fo zF3{WC?u}1W_OGNL`QMhxZAmhSzF6-+vLC&yCH0@|tdW6g=Kvy=N!c$6?~~32r8PS1K_0mG>`;!Hiz1_=mqf z!{2wI`3$=qC5iD3@7p{FWHBoImA$9zocAklmb zf8cpEPo@X?cs9LTps5h!`6RE1r$S|X{x@%&m_Jgm-biIX=~TvRE~6!d^?6176k>jT zndUz+U!$wg{PmO0cr|=!Jb%4A!bAbqZ}Fhl&M(Vo{@@1X?+4?30CSI7>DwglwJq#G zg7QAJjRg`JKAK1{y${V{$SbA!f+wZ@X1|$r&2d;Rxy z3tYFrbqid#z;z2;x4?A^e0Ub1y*KtJ152@N8a#ntZ@hQ&|GDjJe~#RJ>6XZqkxyNE z`VXhS8u^z<7421ZVp9g$novLc!GzDTO|r;+&J8xc#<(1|bq_ReFc-+%k;jpyF` z-tWS{x4l>OxmCT#KmDE3mX~O$^?$GY_L+AgKi~W5)?ee50A@$)+AWRv> z@7Lj#%^A*jqWl&eKi|AD_NvJ=b0uos%4-dn1|n3eylyp_XMWOlnH7;zX8Zw*;1ZQ% z`8ysy?<#nDJ$F3*)mOpO_F+7J=T-2uofwb5>?(NLUW~_Ic@;cuH^$?yxeA`PALH>i zT?OC4@lO<7X3i&R_x$4s_Kx@rgV<^CKYuQ2#HSTXTSQzdelKdUCemfz{F#Db)>udM zKaLDUm=hmy2q-kPdyR=7Q1EmVP*l1%k>nSxWjAX2hKY9Crt)Oe=~RgE+f>RBrQg=T zdYx`HnM$d56;VTYGL_b5D7_KfC^5pkWonPxA4fgTlwx0dnJH}unu2j!*ldI?dVJ6T zo2>>s!O{A74=N4{Qv-ov%>6Hs+9VQl$I`^J%v^LywoPk#9bpp~z*e{A_m`^J$UAo&AQ zJ|7{p@{^oNESNO5{22Bz)id;@XyQGt-kcbXMP;^$#@T+8cBY6ilj!gPF(>HbIAErB z$j-<|s(G_`oK7TFEC_LQ80~ao*5ZX9#>6XeJT`xd6QATr(L_1AhE{&66OUk1$L1Gt zJZAkePNQbk%NKqalV8T+Sp15g$JRd?As9>^TmK2p2`o54$6qw_e4=^g8I1NtU!!WF z8~F$WhDaD|ijtG9#o5U=x-{{Cl||4ZS00i+inqjR@fb#VDqdu>nHm}qd+ND6D{fd` zV7$iDiN+s3P8;P29i^^R7r&1oA2UOO8CpbrF_JoG?^vf)*5@BsJ$78$;SO=^)-It@PS;5njqNE_l#|^~A#5SkbW4nZAFwMnB;8`+S%P9i(k)5l zAJd;>=@zqYXM(C*%(?>!s&1OPb#0Tr3u<(eF=n*gSh~fu+rU-q#-h5#)E#Y;x@mMD zHG)cA$I{LGm-tj;x0t%2Z83BkJ#s5OdI{F(pEc7wG(h`RKChjoU5uR9@Hsf_Mffbl z_)9>e=bbdw27CA@YD2oba;{Qh>GwIJ$^vN*Tt4E*|8bn{08|> zm46_ne5T?rDEREL>UUsQ1pV^#_<+h^iO+aNcYh}M1F(L-81?Mks0{XBOsiyCyO@Sa zj%l$SyGa2&wdpy)iposg&^7jI0z) z;uFl3TU1h#%fBjs61t~E_0hMM|I)88cL?dSexU|0CHa5r`c8}b^7w|Hp?7Q3U-Rwa zQQsx5b+G%tJom|Oe>uICeQ(jb@6}kows6Uuyr0f~{-M<5@|MJ9IL5sG?P6~J>)$TE z{_SGzw~A>_(ZW1P=v&9v@a^KjMyyc9rI|WSh2CrYhl>!b3nC1;$zkq85JRo42H+cO5ZW1s}#Z-TXfKTFOMcH2Z_HQr# zxSn8YAOCi6rUFxY2+o_Qz@$IH9Sang^d-so6gZ{%HSJP)7bL6yJj(O=T1}uL=Gv1H#BZM=?l;(?IvW z;K$csc_JUv_+sS07X=V8s{UlNoI+*knSr*0aYpzAD=Za71MY{ z{3KtO1b>~b&=6$6JqdZFf3bm1*6-K!zgYh3z_^9KCn5i(X+;F873dxj2beN>`*lPp zy9I`$`jK=f8~qIsQl&sDkV-2-Mt>W%KSupEEeeT@f2yC7!FFhhq{m;%$5#ome;O;4$08!v)Z2lTbuYpYl6t1sL@_2utu-M#MiqRoX31@7(c zqE@wDcTs-(+JfD;x{Egpn)S+3VchQ>mb)~zvHC9)&7nk3$ z)>F2*^(r*Qp4zJRlAUg}v1(0ANmKjU>fM#@;^Ovog*%$vMO!x&7dGu`ELhWBTejsY z`qbvRtMgkL+O9^|(bZU;zpJryjhEUTQ@`>%YRFFYrR(=KbgyXMS+lvNrl4wHeTmj? zJ1XkiZr!(u{8%P*tzO?}@FBaaI2m7b#Q#rWBgL-ny6lXWA*wj8vkSU z1+~3xXm4Th-mzjwr7I*lii_&2v}K1<_ZYFL%HwwS6<)>8rQFUn>#vQj`dZ}`j}y~tD^@GC zeWayfthswF)=hb}i23z-oA!zI2gaqQT4hafdoj&-JMtQ9D!OV4)_2vnVqU4ioKb+e zVr|8~oz2T{lB+dWYfH1&wa1g!u&QBIp=Wu0-ilrMd3lYiJS*0$X~H~!sjd-KZW`7RBt1I9(_C=EcqYi0iO z<*xh{g)3bJu9dEB-Y$>wYrVUr)8hiUp(GFgHJE<#U90gU8hq#6f9QXmJojUcJ3sX) zv?V_`pSMVEZcO{C@SJkCX;w16O#+PEM-fpLr&N9|K_<5XRPq>YU?^=%zQOsh`4Q{? zw0*(;X!2K5zN|0l-w|?H?%N26^?#i2<)Xn9BgHzmR_OUobrgq1Q?EoS^ih5E<(fkr zfl%n_L`t6?060lwAK9+eqFnYP85jQ>?UH3$Nw=Y#_7DJ`dIYMFm$YP`sYBg z%5NH@eCrtHuc4guIgR=$$^JU&hW6(E(CYBs3i&cIBa!7u! zK+67$vlL!lNryE^lD7c&L|-S$*K#alJA|C4M7i|KDcN38F8k>(M7z?&xt#2u=Y<@n z_*#lAe*op`fAB(`_lc0{g?uXS74UhX&k5vI9!OLQo)hJ7iSjZr&RIpdY#;p-6d7xT zN*DMkhVnd7J_F^Fmx^u^%IVO^(LM$0)s(B>=YibAnsP1u(_@s|;a^Imr?$PzMY&PW)uPMs+A^?)3)#pZGEMvKBixu?L6=dBsPf?QPP7dMiRFTW_)1v=8W+<2A{cDDDIZmgc zK1x!K9FOx1<#Ifh=iie4v}k{afnJWshYjU&JZ8#rywg?R+~)RnGwdxI-5cFqZ4FHu zJl-}>%Q`=OD=pXSX>sTJJboX$)*TB8ay7R#`xw7#Y<9PJ8hkG7TRdK$tFx=2!PD8v za_jNyza+A)t5s39+0zN_ueNa#+l}V7UG3LEz1QRIY;JE8W@xVW!X~92pJED@+t%*$ zWlZrGzxt%t3)dVEDbpSQWb%jcm|g&vH$(nj}M&)((+kFH#?tf$Gf%j<6S zT!V_cP@fMvy4r|-)IOH8$H{aTRe^bbsbn%<25wh?cLq!>g@0|u-s0c*XOQB-rdS` zZ+oNL=VrP5w-UsP_co%1YYBC&c2Vg5k$2kH+R}f>V}xAA)42_uzWkMl85*NHaezX3 ztb-AOQNy(Jj^=&)T0APeRe?M63s%>=3SEu9Zp0TqT@mJ~cjI)Uv9kq;=Ci8tps0A~wsklnsc-kY8k<{5u!}C0(x8DWPZ|0Dy5B#*yd#!e zIzf5gQozrOb&f=e=|F;v&KamoyH+5ilYSvJX%`oO<-Ev|__YuhEyrxfc62@KnE;#uO&}eoEA9tWUx+r}wCcpJZ#(fZ&(=Yl)NfOE`cVDl^*9 z@wh@^q&LDp7xjVIk^lbzDdN@l literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke2a030185a3f56dc9281490537f5c6ab.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke2a030185a3f56dc9281490537f5c6ab.o new file mode 100644 index 0000000000000000000000000000000000000000..b9f409d06008fa16bc0ac511553cb76aaf4c5688 GIT binary patch literal 28192 zcmeHv3vg8DmFC~KTcY(q5*RCbA+?0C1(rlhLIT+sx89&9mJkAMW}BAOLb9IQYUGwl z;x_EMjEynccz3cj;|#O$X45=eo~_i(cq)m>#_QOGWMitb>)G1cVKBo0Ha70@WN>2K z`*-`=lrj8{+GJ%lTGc-n$5;kvavs72Bo-+xyFoV4}bKq zR7kp5xch|N`#%a?_FvHj-9v?p&7ZXw9C;7dIeWqCzrp=HuD``IRF^4Px?kJ#gUCOm zuup&8r$s*LvPZ7j+5F$z*^$FDY#y>NhL^DF?-zJp`*j0*=dUz2Uzi&1K4Z^3a@M|# zY`qxHVAcOq^SpKedYaAtju(0Ff91gGOUIr+{eH*6)2EIfIQ_?(=TpcY!ZrW)#4FJw z$Bsu+4!#mSaQSHTV#hC{mnPnfww3=pdf=5)(WHYPMpvJIKRVWYK6+~6!)Wr#b&->o zpFf>iRudUJ|J>=;SB^)ILjTg(&!cU}PDPKje;9r4gZHB;tEwYM4jw!`hWwr_I~HyG z#qsE4lfQ@_J@E7BrK6Xl?~i=|nYEFV=bt})6lFVk-*=-A95^0*_USjH4_rRQ^ZZ%a zdB{8zdA9T5>Gz*Ic)In}vFOoL$D^y?d^0-sA?zHx9BoVc5Hb%&RzLkT>i6{NXW#s8 zG$rx7(bce%0=Z}JJ0E@SO|nxPY3n?2`bgP<(?2WyZuHr*gUm;$3KA0 z^UYx8`1#(wV+!ggw`TJK+AqJQs&|A;=`X=BxIp^t4g!tOnCR&%zI-eJb7 ze+*s`*Acc}haY^>)e!l0Up@TF^XaucZ+y?}p9}fhCwALv<#Xg;%C}CyxBd*ich|Rm zw|y&o6W=;~3*S0y@h$Q>oM_VE3cDzK3#?ONV2ilD6T*6dObqCzr&ny z%ee>bKy5IRh&%K@?=sp!%}Hi^>~`ggJrT;E0vx_V2kqh5}uk2&0|ijhmSbGON@{&k~5II^Z?wZEzi|sZIC3K{qJ7uz`F$W%>6zjq|1!(L?yiQ8& z49&nuJDbE^8)Jzcww@)RL^&i`J~J{C3pry>FWj6X1o8ljg0{_#7)+fWbR9Z``X|G@5D-r<06!Y_Kq z%J7xPaS&#MzVY#00pIip54!2VRB$RdHw|^drY2{CzR95b4shQ=-}Hn(FnC9FGrES? z#oCvgDThwpfo8@X(1)rz@jlDBcT5PYzU39VQ3uOO#XJ@(y&18N5DSl6eBND#++?Y8 zcNy|Kxh%Zi^^3jU(a9i|^1gAe+Y4%N_mIy!Gf@@t4vo%w1CvtoEEPC#!h1@OXvsJ+D*kxN&E$1g%+3fc?fAMwvpK5fru$Z067Lbb$ z5s;q8=HJdw00gZ*f5`dCiq!9a3I5Uyll?*g{XU9Fj~*!rF`ixp-*q@$=SwzwN6M?L zmHc_|?KlsyrWjJOWYRMwl>2LRPfvX+94=wb*i+#WCx2v&_@!1Ol2&ct$OA*@I=0KE zB)ts_o!6KgscLRd6lK5)_L$mqB)I65`y3QkRp&y>0_$oPFTepf|R z`Px#~HoBEp*RuNEqvJ!anYo#uf5J7fdnV`_@(+(r`dwSQy4riZ+nTz2Hg|P;o3?Ll z>TGOkWMy5hDp#2=FgW6xnebM)*ZEw?n#xM&L-_xQrL1G@Y49CuoV=|6lAf11RoC)o z>2A!iRLDT&wvMJ8!w&cZ?5%T%JibDuU@Vcj8N0HCIV7FHN;W}EC9Uz7LGaDRgUB=Rrn6BP@M#6(hjL7}Yj!OBJ8D_A23yOPd+ zGKYbH(lfY<(pdJKjHW`+18Hn;2z3|LXY6b`JN9^Dj31CD!@_?g`F_%7Y$dL9X>2~= zNFedpiG%6vjRLlJj}~KRz@Q=SY#Q{PG4`)G?PTR?tc(s}Rf7F;GyRKn=p(;N5Fq`c zbha|2^^<*#9bCjtzQCTEb7*`>$<9w@XB$~@Xn7u0Ek^j+H7qk!7R#7rtPDIVe?kcU zNhyXN2b~!z<#;LaIG#gKCfvnw7xB)Kd?BZ)c?7+K(>sWMgYZsHck1{GPOs4EB2E|S z^af7zv5?z;nA3FlFYMQFn#Ltze;udS>3A2XyL7sq)Ac%C$>~a+uHtl+PB(G7iD+Ja zJE!?Td#(%hVVovdlxHKSH|q2woPI>7`#9aF(;iNHi01a2Io+(|Eu3!A=|)aB>U0CA z8+7_XPCuyAt(-xt&&ujMqY z3q*Tt;WUOXNk7WzM|HZJ)7?7V!|5KK-pc8%M05MSobJ`}I!@Q=^h2C}NYdG%gjfNd z9n0X)h=pp)7e`EpqC`z|D$YC>I+WDXJm#J zad{d8g?<93`KW_a!9=kdqwKaxne5r6gs!bSY1!#+zOoF`u1CI{g>-Z{Nu4bgIZf_%{$lIPxnkTABTfz?q3{#FrDo2_BoS_mC#_Dx`FOb$n#-qAi!t1Df9DgX6a23y) zp{+TDo5q72!j18uJDYGho}hd^*@PS8$(Ag_<#+`A(JaFGc!c?MMIfn+=E4}YDB^GL zp7Izw2L{cr_oWeDN`09`IL3n&0nO!#rwb`oG#yXcLB3(oS||C2p`WyZe2e4AuQ=4- z&%>3*BhcBQ>KOHXUZ#zgXJx2-xdytN=5LHAC{GdjqG3et*Y9D9 zRh8dUa~Z(pc)@Y4rY25*8E`p%2z(vIqM=_5+$F{jVZV!F*}zMH%kf3v_2m1e@>bH= zV4|yNY~cF5eUq9f77bc+P^=j8Np^~51Ef-5#DAqZS`E~!3Ng6Al9Sb%o#c8ZCkgtgzhS13Z!t+^y$AlN;>-gCMtiLH* zWGYF6yt{xEgoXSF&6iI2Clbw%0|~J2E^sUPH`XBj^JDBh7!?1*q+j7?1vtQ<@?gD= z^#m61SWj%ZVrR&2Whk$oa?I!Pxd!$s+!dgCI{N~9dXD8~As!N!V{>)E277BOU=POd zH;{(;Fsm#B4OU7D=L$e)yNISSMa;iBJBaQinvbu!D~K*48toSsi@C`MDzJmx1Xig2kSlhCJLMOQ2%lJtt2nzBW}N)Xk-4$*+ewOkZiB@MDzJk zx9`&JR|MfdNVJ_I`%4aJnz{X!?;(^D1hC)SyC*BgE`TI|E6Qj8dVfB-lc+!Zb^ntE z@OK#xL*;1y0x)DcxdwPS{9DGuP$%&6g7P?gO}Xy>LljSI%Gbob|zdDqM zCo0b++W$LZ4C@IPangA~+iX@cFFSMD;4fNnC>hHOo7jKDKig*Wu;Hh~am@cNS3I7d z6pt4c>i8R^?@W)!X*xCAFHeidX?iu|?$mgkmJbr=_SY_s$7%Us=og1*xnR(R-_UY+ z>{AP1JZziINr~4_!O&*Nx6PK`qvOc0I7G_}iKBg;Gn#1sWoRO_XK{$8LqoqfMAM;} zrs>8^({y8|X}Xp)&ySV|W}KD>W}2pFGfm3@gKiBZYZR({#KU-99HQw|;;`QuNYh+8 zPW7YdI8HxL=tJln;dv|K@zi_-n@AjAUZmr&-!@ycRBu0yr(wBi*l(M4Es4j|F?ASt zYhbBH@gd4j`O|X3D1X~*enz}LEeFi{897Tu>16qrV)i7!7bR}6jlXJ$N3%NGN04Q$Y8 zJ#CbamRDx`S@*{4)AGvDZ=20r7LU{N%D~b7^|;EvHQ;oL_5us- z**3fEezX0y+4B41>HAj3)0M?K&D+1&E$S!i6=OfI=MVd>0ry&wzrsBjtBvxv&DN~a z?LodcvW#cUuGP8Uj$zyni-n7%&9Q@%0+tn~U~& zYQkME)J!Cf=ayl@G2!|79tjKmoD~>QO?Y0OXg@Hyes-ZN4sXsB?XB=EzWy@Gw|<%E zAB8WgqIfm%@^aCi3a`9};@`mUyHE7L!tYr^u4LdfHNsyMo{0sW8Sm^A{-*GI@4wH4 zV}H<(6^~q>QvW2Zqxk0b;SU=FjxJ0Pf<}LD43xA`eL2qQrTFT;kssM}sQnCYuO*99 z70%@|QfPiN^sADbistz}usl!E&?kFTU%r3g_P$a}?aB4u6u28p3$cF^bO}~_W_lS@ z>+f{dp|97(`Ybz?9s3IGY4SX!3HTKXXBzTp2g{5(br zWE<;2oKIAT@^b0Cn9+I@>s{<8@~%OCy79^)_P^4Zwl_ztUFG_?WxwiO*vENp3icn2 zg`Z^q73a~Aug7+mC(0k|$(E!|6sbspwyv+2hGIF${>kjO!aTqF{TBWG?l^tOXOn#F z#EsQHX(Q>uSJQ~z$k2+#W){&^_G zkCT2AI}=M6>l4~;gk#zA{4FDtoAU|ubI-&UG5BBMH$1Hv;FLI@ zSP?9MJBJ?;`xiQ&%g!r6cI4yYyf9D8#v`vEKi|sd{?BV6=xo~08S=}Q6K&vHHrXZ~ z`CE1<&+upHFAe4K_T~NxygE>jC-z&yKdVDI1*i_P*Zno;bC{amDf**jf0!&I34en> z7jV2@;z{EIHsKU5sy|zjg4f{}kRQjn|7cW|FfZS`^#6^KIQbQU98td$VDtXVD{yfM z9)Aw_PXV>vba#~e(L{ej|Ib1Fa}W_AS8=C52D zZ&1Fx&50zX;H(&CcWH zk^Q+epiuzn^=+A>(Ug)HZs1 z&c$=%`j@Eu2yhVDn`HTU{^tCa8S)qpBZVTLPfx9j6#adu7#4+=p_#7XbuKb23&ewlSpGfRK(@vlLpC5j7?MGAi zQs8%vyb?Ql1Sx#v$ZEEwxA#|jKK^a&w|^h|+b3WD=-TJGwiHM5RYvQPI({5sOG!-r zji7C%nB@)Jw#dfmG~l$E+h?nVnt?x+=>V=3r6peF-wfEtJYq#n<JIYI;*$7g_6?g}Q3%(<-mBolEe@v_&{U*lXkeHz%6DPH-}~=(v1eZ@dihUYS~kJH zzxwa~VY~f5-CtVhemnoCFQlh#9#39_JicVV4e(^*JObCNxLi0KJ&fxN4vjK!2<5>w zjq5Nj`bP}?Hee@u_C~)ApoXT7r29hO`_JPCSLDn#M&H@+NO}V)Affl;I_#8SmM~R5RSEICCOYAdPvUn?_r^0p{Uw5P zX@+5cx>|^+-!&-(5yR*+3>mJ&G-Jjm{dY~0t>ojSzBolA3WlTWl1Fj|F-n<}KB7uy z3hoQ(|5WPz7dmG%C9D2Dz_5I{E`{lQ7|A6SNQOpZRUFkq+tCnKo1}M`Mk%Cs#ZCXT z%96j3-y+2oBP6@*n85T;YrEG1;eKx^MzM4em(--0fXZ&w$3K@TsXML61uk=*^&qDcT`kSNy(kD5q{_lZDRkXM%3#((yLV!r5PyHOm zWls3YAGg_HX43O|$)HSSqW0s3Pz>Dv1YSpFBKvVk>U-jt7AFJtGVD|N>n(CB|L>a& zFVuh2a7>ympq^niapECC{YWlh=uDdGA14I*5i}g-j}!-eqfRi-h@1W?Wp#Qh^!F{0 z|4vyUx0DoT;@34F^Gkh?CI3T8UuH=96w;>d;Fi~-QqL}s{|VWlE-uBGHGkzttQWc-pA_3xBB{q zcJJ8UT`@S(*E?4I=#;Onp`&42{m9@%bFhDV_xRv+&(`tHk2m`Ix}O+t*;V&gUG;Ep zOY_{0zKZdomThxAu-D%TnWmZgk^b(U@y)*8-s$ez@rKckhUVI_{=WW^p}vq?==Kly z_fj2;)Q${HRL>5zG`oF$wY!ne)*V&b+ygDmPYljgKQ@H?+b6rL2V0N_ zg%f>!!?O^9~+z)uk7F62LBjo z+_t^bJvgy#WS}2?j5Z%??P(VE!P^YRYv?s%dNeck=Uj?uLble2m!#zzBqn_Z>PUFEJ=U+t?JuC5sxtgT#Mv9{J-UA1nw zda%Yf;1zO%qr<*I{|#P=x0SyOPrVYuL|NZ^XM)6=?o15RyX_X{3S-_1zq`g;twc^re22MbU$*7smhf=<+y~M{z;3W5h+mOTdlm84$i8ksFX&K+tzRh2u;E;0%L1dSwMguQ4 z;UyNF8cGYKh4#flUM})1I(AdO3KH^0dE)uBLWoa7 zJ}I6FY6{m;e^6B)gFONm<_$7Wq-glbvbVHA=Tq z^23tf2%1unwDU{JSMB_#viy9p2RRj2|3Q}1ExY&%w}P*nN;@vufr{S;{t{&bvd@F2 z6oh@^ACmO11e<*acjD7RmeQNz#@@l5`0J%h%bVUd_rRX=9a2%-wtFpnx3rf7zG@p{ zeO2;v!IxWJ&iR^&Z{CW2Px2{W{qz?+)o+xqnO`jHN%mFQ$OQeri;~sOddbhX*zuU` zm`|B@n)ukq=_zF5o8=GOfd8(EZ>}$WdzzB9oKDHN);BCW$XZ`ICs@eNi;{1x?;o1% zn9KhI6W?6lcTIe?x1#*6+<*WGZCV6wdv~Q+l&G)O4Z_W3B$&NW*4oSW>US2iXQDve0&Y1YL zpH*p@92Y2G`aZ5ohb3Rt*Cm0MC12H7@voZX&Gr3E@~!oC$jVsjdr0zCzHXWCi;}PM zRs0Jkd2_xWNWL}SPbA-(FRg1R(b|PVO0}KZJ}SP4gpfwWO_x{m#kaa`5OJRql5l9KD~j_SMDE?uPeTF5r>fh;)Py&Z{Cwvcumwl`yOA! z=8Sp~|CjVF{bF$aCw%V1WFnVbaaOMS1$n=m9~3BuYen+&0piK&TjzV^Q-)=Wrsgdr zct0L!DdkZNd7s`X9=;{lS@i!vn)l@NsGnE-DVUVMd5uk)TV%c-B1}o;ukPQJ`efUj z4oQ8rUsjmPU(rK&AfB}iGx~&tRo+Vffz1C#`Ws~ZReq}6ie7@sTl<8t_J5%y;_p`3 z$E3cs{#&H}A&Y{w{C?mI^`BGv&lQUcx`4yOKOlnkKUGK zT85kmk#&s0V8-mmJ`-oi#FM3E$T&NhaWtE=kvBfXaWI25*{n|{=Y+@%BLQY`$Lm01 z-246Y=)U)=L4ZB6VJYaoxBmaHs{d8>|5e?({pF=Amg$gY;V_L?2(ht3|5}oKe+wUGlMHmJu~=C-E$SR&*F7x)`@S2 z`;H$AKj!>S_}KpA;gPu~!hLg3hWD*H6~4IdRCr|HrSQ{p-wQw7dM5qg) zw(lQ&`k7}2Q|B%U&1(5h`23t>;k}oB8h+aOR=D~2#cOQ0(Xg$^7T+;)8b;NH2% z!j-3vhnw3@h5Pou6&`u`V)*>_i=bH;s&F0{JU{oz!M&@{&%W=3XSHJdlW&D9&Rq;| zyF}ytaVRtw&VhI8=Z``o`wk2~J@=Wx(8=TB^LRC1 zdMiBp#EEc)^HTWy+;gB=5}MWe*w{Nflu)T>V!Xpm>rsZ7uN#a*)MlQS@NZy_E z{p7@+n&j7Vf@DXJjTJdyBTetAsfSNZ)I3+u*e3Q7ZnMscqV@+)*ncAYOZdQMjTOBC zAJ8Jhd zB>A86y#e^%tMJWxzISK)Ue67D@5mUwcf{g*Y37shlU!oLudY0 zzc##&_0v3J+Q6gBfsZ`L`Xm3$`Y*n!UwyZf^`E_;UHz9r)_*RIT|J+~uBL#t7;|lS zPeT9jKe7Jd2em^N{)c|`!n^Fy@ZThahQGxQU1-3XI-d29{AdzjiJ{>K*`a@Z5VZd- zDRgH2?nh~^f#*ZlcD`nsOJaQ&@>~G_B+D>(r7_%$4c6nli@| zN?lkSa(wa91kpT*ex?sRN;XyYzx*xO3cmav{P+@V_HVFX%pcgH%LidsgB_c)g+9nnKdvecT}vvt zLiW3-53@d=hx5RX`HS9ewO;*R?LTPD#V$9X6BM0k8U^^EtP^ZGUJi7=9aj%tNqD6* z7Y#eAZ1)A)d@UWU8kju7>X>qT-Nu-G9F{y;Y))p!?UzMfTdWW|`=3H4 zNw&FO+?@Fj;;L_Fi5|9?C4i&JL^-dcsiQN{>8;0|x^}mUI=|L+G_^Gb*}S&)PT#z? z$LDo!Z*JSt9;-KW$@=5-qBjlgt$uGOo7d?JcFq%`)irc=ws<=_>bxy2TfLo4zCc}r zx1+O8Vmp0p9qj>sXP~)o!PZt^tGA_jYg@sh_JX>9UknDQD=1n}SGcgSt{^`@zi>ev z{w-LT|46~2#dUT0kW5?;b${z7AjDaYzI!1$-@cg||Tt?_D8Cqp!Ye zYu%QBx78=CV+DBUnIM>E4YV`_y#6LW>HLBA&i2l3KbjM+y{)6u+t!(X7r1YO*Wc<3 zG~5+thH1DgR$FqQTspY{jextrhDsc>&vNKpGli9Hxj@%5una23wOF<55P5{mL~eY! zmxkO+L+-S+FplfrOG7?q*}G}kBI{TIH=l;w`TH2PNUy;LYhL@7Eginjd{$T2_IOBt*IhuPp@fcLw2$rCp6)H=Qmtw|G07+uPio?d>fcbKIEO zK5vK5T~Ii0;T-omy38wD!b-ZDTN>RR-5s62R(ESxN2j~dx23tw=U!V?wPH=(x}`O1 zDyml1E#0to>8i4&Wo&MhyU;z?8)#^9ceLU>;1RDILldubzJkv`EWB=KPXcbA?_lH! z`E;Ky?s3cP8L|7DInJ>)r84^#JM4nCkJv@u_tWSM9*C2n%jwK6>vppA<>F}FMxz_* zI&+)qbXH}=Q=e3~pKn5oXD*=MD4o046r*k@vV7{!$gTV>$361+`KWqx$; zf+JSrWp{xF2E%{;vS$k~vt_Mi_7n#r@9weTevyQ4$I(_RUR!}cp>gQ?vAkZB*N<>5 zBLoWh#pv0_Snp0P!hQuL8E7btWpxXE@kx^FWXE=!w#cb`4NE(1VJH#*twy@_@{?#7%{cFfIw)GcISPXUkqdV`oq5=7%K?bg|5 z;zjSV34ZQJ1!X#3XPxX&z-~ti78y8@#)jszyrAv=rvFsTBKmClNjH zc|rRGI_nhu6Fr}1f*$=tA2t%lU;P2+KM(R5=)q5#v={h6e?-sqm(i&aKd9>@pT9b6 zBYu;D6JbSe-vosh^kGJLSHL-u_)R?%Nn(R(tmx0tKk8iGH+g@~G}Qd{=nwrrwUe*^i!Vob!lGM6aB*;PVVGn5AahIU=t?t@kPCNXF`O1008*_{4XP5 zi;zb~3I%>?x5h3L1Df!U!cKQ#SHB%;OW zXE{lK5h>r&ZY&c}t@39RKk%=jz(Wsldud|4qQHzlWG$mz@jr; z40=20v9=NYQ76aSQ5X19;N8HxO?Wrz0>2daslew8{IF<06?K9CD)5=Wdjx)F8t*?7 zb%9UTc6Ymg$0ADOHwnB8b%9UNg*=-jK1cGuS>iR|iGRPq*UyK4J@gRC$NiuDD`>NE zUsC=SoHdKYiQ#9=h}K`{^M%`k&p)&$ec91J=jR=pUY_+lqjfbB*~9Ih3HwG1=!;dI z)-U)6=5rpOuQZ>3k;Y11&;&X8FYx6r{7g0%_0j#A>EW+=WKZs|7?1pGGm*&okP)0d zo#G$&$DdRG%T4|F?#zz))6fB|ZwWT?H@+yGnh*cNdLQ*K@}KMk8ZVps&q1aTZaMoi!JZ5Dmg{5I&D&({pD?+mFw_h< zz!@~5^(*&JAivGvZzR9N;%{~yKkX#{Nm|eBB7dOup`O-D(O$*3=|Kmlcf|b7!SzA@ zc2?6z^C!HOug92w%V9rJ#Qd|yL$dE_B9Qq=QSiR|$o_nM#EbIeJYG}rpm!&heigs` zB%dCWFV~59DLyN`pYxlh7g7B2+h8(UziIyHJYMRA$Cn%aT=Dk`=ny|&R&^S60QD}q zkp=hz-At!lPh#=-`yf9XEx)9VtxqyPK>mjLGne>L1@&V)V815-pnL%SL-F?&a!6|b z0m*+P7T=*y$YY7`x*@*j#EIv~fypFNzj`1u7O#iqYs&+6irj|yuIpr99^aAwz@PEa zG5K>(60NW0$w~lWPqY7#{%ywiY(KUG`G%6Op7I@s^HK z_D}dX&lepsza)8hQTa!}=HTn|e^>cavh5u#XKFmGXHIMtL_W>evt%1Z8}1*Fm+*YO z%kVRjx2P9*K7jC_^j)@Oo*z4Ser(SRI+ADpVq&~B*U%p7)>J-|m=R1y9SjYSe^P%b7%qypX9UNgZp72{HR2OcH{uhKZW-}OsK@Y@ z^^C7?c9)DNWBPX$!+OpA!@rsKC-9HGG&!j#|G@k_gDTk<_D8;-N$x1(A=!U&%>M9y z^ZcC8>A5{=PC%bU!Kpeew~X|6`Y&nynwrQiM|gfSlnwt%&W!rk(0uF%ax)=53h0Mv zzv1BilN{ZjWaUQb&(VIvk<9Hy{=@etZcd*?^uXr@lO0iiLVlYbaA&#EdxYr8pLRL; z`iSw;h#vlwY4o?eV3yn1zG#=*P4kV@bAPlsU>)I)7@yil{qe&z_H39VS>#M4jrc3- z1-)pWMEMT#Y1_b);FYg$X)fev?r49IKRuAGPX>L&h50;Om6{&N(Wg*-5+)GF$9TC} zsPp|M`h$Hl$zA+3A0Tg0Ff%h6&#e3q|GgOR`ux!Pg@8^3!oSV_evsyqaXw)EG5I_2 zruhK8$=`uD%m?TvFKG6E_X76)Py07VLQKEpAIS+8{btfq<*yg<0Q^E5YJOH>ekY&~g-7RqVtNdp z27D6gM!XBB9;!Xz(@{6#Gsc_o<57>{Ausn=t`Epdbr(1g`mtd?=Z%q%^T{RR%lRsE zJbMZG<1Us;{tbT`$KI?We@dkm{xZK^#FHdclu&?7@$CC%m4(I6nk<1M_8< z14jiYHmnyXG5#)L4}bmpK-mWMRF=aw6hA^k-Vnu|AvDZ`8A}UPtNC9_vwTzcr+p*K5pI z!bjKZG4mBZzb*L+ZN{kxf8_fKWq;Tc=Z~3_fQ`5?-%ovt`3m_%+AT^!}T&3lzEk-XCOkg7EwU=lA7ZuZ$V5 zj+o+=;d~wbb35@Wy8p)dX5Mel$L9-U?YDPvan$<6<1-zp(dR|E6i@Ew^L^3;NI8_| zGQ}sXpQiow7K%@ae1Fd48S7PVsW*;qlexc7i67q(KPFG%`!BU0A06LJ=RbMDDU&VnEq4B6-QQyV-o$=; zvN66D1;^_QA7F|7A@e^PnvH#A&{yS?teLiQ{e@~_Tq9NWnX+sj*e>>PZxBu2*pGGhV=)Z7xVL8ah_o~4?hB;n>Y_Yit}(#_9VWKW0j^f z%l)TEh?Xrk$n7o zk%ScMtDPK#w;xWY{^{cu*`F3W|IfcUdClhsY3z&ON9fZz2ba*v!7CdtJ5ezS+4)`}!E`x7R*5^8P>9MZU&SDXSCysxP;vyt$foY!3(I-LcwW_E35>N;!$H%Q_l|*Pkf9kuYnQj zi(aA~=WU#oH#B20{GJpmp3)tCv2(!{wi%x%;d`nnNDzCqM6et!@$gZ_y5hH|T%H$q zd_0nWQNTdF{&}uX6rT8ak!t}D#l&kL<$rTb_sRzG=%e!Z8C$Js$vQM0g>eYklZxos zpwlFe#UD5owwfj3T%GHJv*@Vk#Ath=>*DgZlz1~% z=&1y4G-e&G8c>5CN8yPZ(!pw7U-=N`o|}76FKy0xHIEHgRgC-ySaH3cqUSmif5%q} zZIpM>*lIyo(Z~mcx46ZXSCFYkwU7JOKH$NIiT)i+eX(Z7do1;8OP$7}{`nGt{x^sdZ36M}Z&)XfxYahFl)gB%%zmqV z#A|aiyr1SP>R6-QOiyF=Va3Pm^Rwf|%H3FX<4o+8F^GGaa4Yij8~Rne_G*&Q)EcwB zX3|QO8)3%l7^IkXVOpgBV!Hy$;bgCF)eGv>>onA< zE7CQMnyc!ueTGe^QHoR7+KXL}r`+6Vl#7g|n|Qejae6DcfE(-@bgUzk0TL^u5z_If zEH+NAXYyW4o=5!{tuIC}jLfVfyFp^W-;4Fo2Yk%5Mt>G-j@X7sJIGGM3IhDc-2_dokR+ii=pG;=|~;Lwn5lCux?|UAM0O6fX=rIZeBQEJp3* zWW-hDnk@7OiAywcEH>E^Zp{6jgnRqYKg?W>x4Pfz{J;HZ@M(;0#vPl8^|7e8XVi|(0}qPD>^;mF_0+6=Th~{WVXw{j*|&BD z-baVw*lnva<_P?=5|8){i;iS1=jg)6P51%v{W!7xkM%$LWSmQ2=wH`(|2>K~`retb ze~DLU{%3Bhb*;jM^97nuqMWwlz$VT!jI7XW-@1A)%q#6v#FsJbjl zc{9)diTiAyO>Xw|i@7g-_r(dV?0d6+^VW0 z6mWhp&#A8eHjDSckA_S{OE`RV58^pp+^tRY;yGOzCKy$o!j!Tx@uU|cQaqv6Vs8X%i5y{f}k4 zFIiM;ZI3^Ow48B?ewWi9X`(n}16uEBEX9t}5ZS}pC$+1V6TXQmhGL~9zmT7aizQXk zeqXjfW@t;dR=tz|PzU#bd;f}TpC>^|$JX|=pL;S+ z`=gfjs%LBaX+j}>k0OxQld}B>vQFdEcTKGA$?ws{DE-LMMv$4&f1>lIjO+nri_teqV(j7E43$@9#M_H<4*>7pQTZV zyjd)-G^i~64_ys)0)H1?M?6vgrXt&WOhAj10Y~c#@n2%0!z288`I({Pc=q$dG4c0x zWIyTgqTQK47T1Ljh-(%uB)^FW^*;rW#z!%sCTtWy>eyH$Z4|T??f1kP|0}Y6o}{Gl zsaVv;3be!?Oz6r!AZvM`gb$sPVm1y6K_8!al1G2sH zKQ+E;r{Gfnh}Zs|IPGalSliQ&MIq|=VP=#5FU$6l{piK2|21g;pX11{+FzC};ilaF z*73nL7YEK2Fhk_9bY4TU8w8w=OeZYyqC>s{a2wQ)mDK||~M+U-TF z+r7o5D@)gvG&Qs?>)f=VrlrBZW^GHwj%5uCngCl?mK>vh1?Us}F(t)wa6*tV{9L*cq@H$}Ig zU@W?2n_7cSn+mI-kJfUMdCMl!T|s-&urt(a>Gg40&6it0(G9i}HONO8oV$KGYh!Cm*QRp33RiY*T3+dIY+ct4 z{Z*_kSocKZ^2(Nt>leV@H3b{n*3>p_6873wQn>|v=QnPs?6TU$Tk3yeWAnmCmA|QL zU298o06&gWnBTDA5pVvIEsGcCH!fPTh%Q?eKeEWX$hXMnT~a6L8k)Cw8+_OMiIuhd zw^is@Rq$&kw^PZ#!L!K=3KuS1SeT!`nA0rtwsiRF#CvHieo)5eZOsqnGqyALj(&V{ z?jN~#e(6gvPQknaZX&f9YfscLZCAE7H8lqha>4IMDg)x0XSkn(d*dGSDOzx;tWABu z^PGOz{$CRRtK(~_-*CPtdM5ftP9WZPjX_Ui9X8OvuI`sRkBs|<{-RyhiS7pqhwDjo zZ=stC_*sR+wM^ZYIbF}`->?&&|KYEMQ@^|z@Og${G}D47`Z5DP`m0DQoUe?d0bUd< zEzz~y1fA;MdVL9P?d*MZBw<3!2gjLULJpZcX#8(dluu)uU?F8qMhXYi6ph~>w zJ+I|mqfwj^?>WE>N&~hD-y2e)9bRI0ck)5Uk`Y?8WcUPMx6?t;>+QiW&Ixon0*UZqR+=T zR8EKsdmC4RUy*RTge&>Rqd(<4gqk4n6D{z337-tOYD=ImO86AOja$ZD2DmX*JSyQd zE>07p<(RyqagFdYDJS)>cq7sk5^n8hgM?>W`ce9#evGF?J%;}2IVY+-Z-5)=k6j0T z#Q-d<#VQlTjd>)2C~ZghJ;)D`I&@U<^83hA0z)C8Q?~FGv)lDag8xD|2pu) z5>EUWSF`6O{9y}xz(8-b@9Pq7<)5CSLxuR6EN=PxLJ3#nDqn1pa5b)iA2!e%$9-PH zt>gYu!mZ<`%6UuU8t02k!bi;)38#J(FC@Tb16&^QaJdc};L$(s0cN2bOv&r!)Qo+> z05{6jB;iUAioQj{t@6Gk;Z}LeWXMtD=F4$c7~sZngA%UBRrI?g+&b>}B-}bKF96RU8<$z}3>L&OcTEYQ3Kc{-`K;kF?)% z3tX-DehXZ!_e`YS9d-51ZQejP!ycj0yTaSm*3k4A|H;Xvcpx)=-I4#s6a8ExyV;!w zbyU~f){LKdtncW!wPtNyt%9LuUk5lGYm+0l4*cgO$EJSV7wEuGLr77Z>jRLx+}A0@ z#SfRz?^u?vs-279lWW*6sFpz=YkZx>o%sF9`mRnNK?OY;nxKu|GT-CP4L%cGXwTOq zf9`H<7PnwL`uU2wHu`zI6)V=QENk|*_!>G_wl{XQVECKd{c#(#w0j%tI@{|SIb%W~ zuqXHwRUwN}!e#9NMR2QQvAo8cu$ecowWF@X?`s(SyP|>iMsKH=&ExOFA%&GzK$0?! z5+w5)Iyws$-Ly9DkiGED8#nF`rryp5-qhxk@8-pP>`=emdWZh9t5{$^;rkv05t(Y4 zdb%PL)G0FmP)NB?KEY&K5QTf>-Dw%e6hg%-RiA+yO1V4=Ag}%%;^ujokI`pq z6z^m>j^<#0417fNfxBvhE literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke48338ff5e721e5e5119deb5226df72d.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke48338ff5e721e5e5119deb5226df72d.o new file mode 100644 index 0000000000000000000000000000000000000000..1a36ba3763606cb414879af77f15691ad39003bb GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^KsL<8v#l3GG9V!kug$DzNevR{(WfPu zmLVrXWF2ELm@%`l&&C-t@nmQjGR{tB9L?rz|5dlS{pGS1%XCd+s%Y$07ON>JW0}3Ndk??$ zuvAdGSx?P?t?uX9ZrlH%586IEgRw(Hw(P!l@i}YDp8GGjp2O!~agU}G6)jz_?fgOL zx8vB=57+CV52|dTYc_W1|JhjIkq&kUw1;~#SmF1wJ^de+u(yAuvqLjdduqud~E;m@W}iV;lBAN!~0g93SZoJDm=39Quyil?}Z;~Jr_Q9^1X1%%tt~a z+xHJX{me6isq-HW&1v~g`24(M;k}oB8h+aOR=D~2#cOQ0(Zg$^7T+;)8b;NJPi z!j-3vhnw3@h5Pou6&`u$V)*>_i=bH?s&F0{JU{=*!M&@{&%W=3=d@z{lW&D9&Rq;| zyF}ytaVRt&*w{Nflu)T?ZJqVbVedQ;KU#%Ir@Nyt|clP&_ z6MJfsU&{`X9X&Qy=zxthy{D!gJ~c`6Ts>o()JM3@IxCFYA2?zEN$@Y>1DiEg_y&AH z(+D@IZ%B7MM(vI;R`?oVIlPVN>xb}@52{K+AFeNk-+8WH+xhxYqyNV9#}91UT;-GG zf6Dg;;CrvaH}Co0o$Y%)H}Jh9WBA??i|>(7D*qd>_@DCE8~Nb1oj(jkN+!QO{D6Mw z{67_6b=z6mg{FN;|Jxioh5d=k5w0@@h96*u{`CRS{&iC5 z%=+Dr(p&@2hpz2>%`}(9`YzAk>tAoKF6VdFJyB5ofdiza*Vv15E_9T??Dcu z{3OH3tBIiz$OB#UpP#@EU4%|XUQP-zttYNr4(7Q!2jeGete@76kE?zdBG_*bv&M{l z;E8nAl!z-IYF2HZePQQi?$mZu^^D~QB{}Aj)c2GY3{ui25@L1QjFVN;|>0s5s z=6Z2+=0A(8zMUm{*bZb~gC}bq(H* z&N_+h^tE-g2mGCZ=0yctTYatGmgcQ(`46|}*ZuoqFhE^?VL@HNBL#K&d3kw@a9dYU zusH9L{D+s+)#X7>A=h0XMn{u3;A^~l1PS;$e1XRy5v&yOwcHin206TUg&>W-`mU{Y zTLRuzpRkS<;9X#XV45}1(h%_aoA{*j2iiN^JG=d8PPq2Aj!th|XWm`lz71Y~t1r-S zSC|>5;j&n5$$@g|h7MN*~}5y-80+4Zy6)_JZmSiu6VY89RShRewq$JsEW^Fe5k%^ zT}5d{k-G%@f1IO;LWd+o`Y=9Q2;}b!!WT=r8oh2hU1)Ccb~d-SxjWn2TRP^sF|&Q% z4xc-J(SpVE-0SGFpl~TG?rLsnba!-jboyG|tz8|R?nd91<~E;uZB^BZHFfLCYSvU# zt*R^Au(oVfX;~?oU*%rpp6?AbG`TxkaUSr9*NvfxS2|z8^A8KJ+u4(V+ZQ?*c|soD zr;B^sGJ8hs{$`GIY)z@mzQqo^pzR}e(f55cI)ew|Wax4_v&*`jEPc5sTDQ^YhPuw& zraGNf8S&I7)otk3HZ{WZ6*y+9vuXA@sobVyS1R_|7#~J)CBs%3xUB?l(^8oq-Miq3 z)%e(5pn<{gkALjbg_qf~)-rpFgOPXl*l_=_gm1^uRx3VRfk2^g==!m|UX$04a4jVS z3i-w8*~VD!PA$TI1tb}0D2-)x3w`lPlH+8@b|gml0bB|w>YZ!>9nz@+`qThRrNRBx zmu>79*{pY;J;E*nKuk9}+3vlGc0TUL>{NEl&3@D^WMEGLkN$dtm`D;t+IAhdyj1j=%Z?&VMfCGth&dG-)sJgZ_w~>o22IBYsfVNj`sd*hc&& z1t-CZ+`b74FX+RJ@UDP!67idQCX&Pk(^%o3p?}o5yl?XUoN1`}>(L+je`+s_kUi`U z*mJ0g-M^D)sX)GqY3wWA<0IT0gwJ(CKl=D}wY9Ze-Z!%$?_`~PDdK>B(0^`l@?`S2 z@m&9+&Q8CqEB%ybeO;Ot_C)`%hm$)w*#rC(2H3<&e0)*w-I)+!9{@l;0RPJf*dpYS zkwpSu)~&J2#DFIJV|hSNqbk!r=KQzd}ILp52e{t z$)B-M`Zq&Az-I(=XVSU}|Kja)gD}T%tu~qD8_Z_qFB}Gz3dTQA^bgLw5{YOr`Z-S0 zUqs4R){SL?Cbslv6F=~;!oY(Ma(iiFyu!e&Ib0rUe<}_0b6$j_@gMD^)JLd8EcK`HL_LojJGdOn^;pcMvLiS`gn@g%qydR6sa1rS3 zpvT%q^hcc>Z%19=OMrI+?>6Dxs0(}<@Y8_L6Zm1#ej4fm|5e~Kf%gdf>@?ngCh7v8 ztnKc00gpwL#%~gM7wQ6^pbL36OMJHEf3w7Ez!U#|fv;Z(|9bF2l8^g8`B%_pf6mW4HoZLSc}DAMB(jIwKNI$i7SI=|I@ufk zf%%-v=PS+UU!<|(7c@am{tJBh3qO<1MSXOCW_tK*F4>d&E5;-L+Ds&JK4b)E&ZPLq z{qg72|8i6Ry*sDJ{AuU_*0%&3`5RvpPA!CgVZD#~7x~Zh1R8HT_n(7g&*@SB8ggod zfoTa*f5mz*GdOJ;(I>|IZCWPPxxei}dl%X(e_KWRo#rz6n*(^au;eF1pC4#1^^N{Bg3uNOhz#_K^_S)cH=xv4mRVlvFS;HK6|wTH*(UkM(pT?J&$SR6+gN4%qJr04N`T|4{sWg&dOFe?amd ziN$y56Y^N%yKac@*>U1Ia$quv)UO`MjK%Arh1&9fog%j(zUw;Km&bSHKk#QfIwpVa zNuu?&JXr}K>}mEt(!b3ZpY6wXAm32()lPCDb(k&xC3H2D>vYzqv z&F+%%WK93AVpy-afA}}k{sjK9mnJ6_6k~5?JHM9`>ft*Z;j{^E(+HW|x z|0GBECs{dB`g643a3phkk^k`hiJQ}B5k2s^!DL6&pOD|C2i#e1^d2F4@~2%6zCL2S zG@^$;Wg7i0H<;x%wlCb}cGG<0^xPk94p>L{BgUupQGfg}jXfLYNESI0NhAIWdqFSS zCsDqGeA+hfBzWcPTbc{`nLFAaEY$ga z6aB%yndB~hnh%h-FqoMcjb~PVi2q)UcYS_n{X#$|0^#3ge?Lg`$v7Xd{+Rq7c+-3U z-sJDV8|DM_lN&VqKm2_<*_Y=#;5Q>^m=BPb@KOK2s8K!z|3abo_!7Q^i2RGWQ78SL zU5Gl;1?rIGIZyjHM?y@$PEZ^ryi<3;nPt!;xi_g@e@#w;UO>gSFR7pOLZ4G5c;uUKIe{+kMqeT;>-Ceb3A(q z`Qt8@O8yOh8pqzOB7aJy7XDJdUBr_wqDIBAUJRqkDJCii#JPQ+g<*R$ls zl3ehY9?(fpLBa@CN3~E(eYZ zP;6K)PGbCB!XEzm_s6kcQoMbgpYJLC+_F5_uk-bT{Ee^Ar%m>>=4YmS1oqn>U`SA; z{jBF1m%(4xF$cv%@HbBG$4vPS-=SOzT!xJ#f$V)36;VZDyhqdnH6*nVqBGq2Z}uY`}T z*JI`@e12Q<721qb5&p>c6UzRuC(a);Cj%RCVZNXG6!R7Ghq(F5j4|>R_lz6oD_J~W zamUYBAWxQ?;(Ih-2^#W^(fJDEsXHs0uM9(fpFAJQqV<)>OY~>SM-VSf`G^DK&2WR? zPmqsfi3OkH^Qipj&do>2fBr+}BlAr0YxMq`vI`Ws|K1;Dc7pKy1LybUU9XH8ua21F zmEn9H{&PF=D!Tv1`exp5FU0eOvG&`$xHxKk;_;b|)aZFp4#kuE_!)ddy@ldaBHy3$c!vBaeV3i*PuLHnu{V&v?n+3cc%C#d%|ZE4VbD(DI)UfASpUR% z(UJ8mUOYp*NdI@B20NxbehI#(|3FFyilfPKw8J;J_><@m^ah^1AKQu}{rEvVZ?I41`*H3c|AqGBvGX2`hy7FR{0i&; z=>7QAX*Z5D29(vl$e)A^k&Lh+FQ!TN*e#r(WioM#x$!;gUICeFi;;yfIbJ&Et*Sfwe=vcFE# zvEOH30|EDE>_2bFAD)N9AH?~37Ck@Z&m(nuV4g^I@VnBo-{$+ZsC-UF^6~RU5>l+M zc5)EjemI@_r^hX_KP`6tpLcWen$Hi?*cZW%*w3N^od1+}e-z74@ch7(pL_;&NOiB= z)4)9q+|$544cybfr>+4yJV&SP!FNK}{@vFc8UF01* zE2~{Sx;ajv_s{)5ayDJ!JrzHjnqG z#(OQnj(TxwaUy++QayI9eA8sX3#idT!D);utQe4@QDan7&kqq#e2govff4G9KB67x zZJd=iG-EORo)jyd(j9$?v)~HbjOR)Co@xpb#9l2CEJsT`d{nWn`0Xi|=fxc#kK|tz zFc7bQp6e5ZCq7=}TEIgw@!CiE-yGAuvOzrhs62kgR%=?a4oyd4976V_B6>FHG|6M} z2TsLpNjBLt%9aqBal0Axt|tk-N9CrG;^3olyW-=ca=V4x@%o6$1x83#B^ic9F-T_r zR3x*H_9(K1i8DzPLGRIjJ#IgZ5N@s&aw>QAArm?U}(s#6_C6Y&Riy5&nB z@LPj#Zmha-Cico0#Jx0k6y=)-Ic6?c(1SHmqG{ z`gbhtmmq64_UEzGt1We8tJhzA zN7ZBd44Y1)6s4}U7r7o!xw+9O7a2=8@p2X7^j2~KH`q1kSVt%WBvwcxq~lRpY@A%r z~+DG@Cd=4f=R}fl$P^|QKTHU0dI4hr3o_O?Xb4M#K=p7d@66c0#49xa zZ*Hq~t-^)#1)5KyoVMe@CeAyIv>U#|Xt8Q;x%Mm<06emRKxc5`Azx%vT^gmlnfL$1 zeYejhH+%ZUoR_});>1?=y}7^n?FQRl&6+nO?6lC^nn~MZa2yy3+2Wfb)ZS zPIdi1b9f*8Xvjpgl*32&Al}o(-P%Mi-qV#~f>GruOeq@^Z+bBz#e2G9{V0D?@99#s zXwHjxPgkY_N3YTMbV-P!j}qMG4b60ZOlN8Dk@be?17!h-tqh`0n}B%je=OU5$)Z|o zd;Die%NdvGcRBqhO%#W0KfC!9S@>of_VsOC}%8F zOpUMFDf>T;#y4uO=%_u>S=-Y*Cly%SlreHll%5=MrS?SABdXEA_%8#z&ypxa-Yk(< z8dMhkA6*S~0)H1?M?6vgrXt&WOhAj10Y~c#@n3AA!z=uG`I({Pc=q$dG4cQF$bQn} zMY}V9EUt_17uPIYNPZI$>i-Eq8Xv`knz&H_sbgc2v{BGnwBHkF{IAINxssB`r(#hb z^U3xe%lJoSd*#2>o?P16elaHR({aZC`X&&gxcOhktm7*`szm!p|Ecj+ zI|ZK#K)m+v#A#1c!rGpGEDBM_4>Ozee_6Jd>_;zF{jWj${~AYr)&8<<2{+~Tw~kLf zK|0-BF9|J|OgW&ss`doF-F7^R%lT;;E;?V;yfuzTysCL&ZKvuU=^(~-934I9CCefhE{ z8oCRcA#V$G^+bJ1VV-Y8vA@362m7q6tZ!TW#7eTk@^#JLqBSdv+g9Z@w61Ha-&E6H zve8>x>#tebQqt63x_)c5x42~Gvhr2=o65_&Hnz5OZ7RoS(aNq(%Pakjt?Rm>zlzoQ z>z-&_UfHs7eF5xUlfSWTO>NUAVXtk)m0QqvUgL(!E~{OSeWXqO9-y;S2zCvGNe*RLt!ML!X;NiwCj}$c43A%>nE#3y-^?qVyE&pv5 z`c)PD+R5!y@^0{M^87`M7cX9vm$!t|Ec3Q>`0B*pl3M(rjL+Me7tCX9XU-k{`sAEH za_{`omtdUy1^L`WYBAQHsA1Z!>Dsi^Y`n+?zaOa#h-xbBU+z3I?i>1xc3CI7A1EBIC)K@$ zZW`cc6%N-@bzkaqJ*R)ePI&$weJF`)V$KJ z!~Y9NyokP9n&5RQ2aOqT+&87XZfOEFznVc${aluC)wV#w)h4x6_MapBsg@>Fdj5OK zKi)3Ii<(!@Nj_hZa5bM^m;DS#11k7>z|+;B=xH_TRPYvG4&N;6|0uxhTeuQ^9>$?^ zLR{F}xDxz|gxe)t$u|N0Dc>Q~M2Vkdf#*s16u?zm0)0`!rvh%=GVU_Kjj7^M38!&! zniws|Ak*<(%Yd;$#e7dC{r7!BocuLe`=%3zmqRR6IxRL(Yb>LSF zaHG7rQtwtiXG^$M-T`SKtGsVWxV4|3Nw`(sUmE%`^8b+mZj?7u&L0}r7$ftp13xU` z#E)?`dtSmHvcLxn^hW!>F5y=G=`A``h@UCqmj7KW;c8svi%k-)##Qjc272ST&r7&< z++Rw#b=*`rZ)sfPd~r$msQDt{)Q{qY1lVkV%Of5x*I@%Z`k#A%Su6)r^13-SV_z`9 zjdC?fxYC27Z;^1Tye~<(Ro+q=a@4qaa@-XLxN+Q|gsX8C{VoZ&j{7|cw~kAB3l+*; zXi6#FQCy><;2vIMtVv!KT*Zy&EpQbd1}tzD2Zk+hwe+gYCe{@iUM09UZsUtgWk6Fx2ep0Ec63a^%*5|Gebb)Q|fD9r$SoDQa_l0CJc6I)%9S z;S%~C%kovV^YMFf4ci6PGU#KCud}EVzdu>u)#)RsphrU!w9#AYd%U^9XMzju`I_X< z-HpxS7K}$fUs2aaKaaO!#k!TH&E6JYL+8r&#;z6&f0MgEZiAL~Z)06&dtD=EOb7(_ z1fQZRWHCy(tUaIzZgnh{+jtW;^9Htd)OGlM4WoZoG|=AY?eww*{I76GVWkz2q?Drs z$%2NC&isdOS{rxBUijvX8}|oOZ)XEf2bVRq$>!LSy4AD4~fbs}8H%1cl;P z(Oc{PU5C jll{mQRC|>}@58{>v8lk+Kp1A?nbre@WI#e7A=u1_mee4D9(`Jp zX&G`NM3ylIgNfM5Ls7vSs(Zjn{cw_MCsi^#WeM#yy%&RJ3%xw(|#} z-;QHfKir^)KB%yTuG!e(|7&A?$2!rc@UxJ-%)9;2;W;_xa z*?wT~si&VFOr5tRG`rNt z1A}u;?Hvq(=Zo`R2=ANs2Ka&=AbTHP$+N!@DuXWPB!UKdOG$ho9D4VS@U|7F!m}^D z6TbM^h48o;kA_Ab27U#2{mH5DKERV_|4C@%vAu(3dtV4AUjQw5Pu8-pB>#GU)$9u| z29kGYe?K{~rz-iC>>%0EV`Bvl*htfRs_NlWlQhrObGAu+gxjpMf~ft06ZW43{}Mj1 zSz`sS!3Q*raFhCmbjM@V?igbQuKW44>OVL%gou$3=%c83zFD~!@)jHOniurMFCc8TPL;dP&@av&$cJ*BGq{BlE z+TnA5qhA}|&-!T|F>T<{mB2@yW&M$VW&M|4)~~)@%=*vY$FBa{BG!K)ja|K%#IB}* zwh(h|cuzwA@V~JB;Rm(D@BAN{_P8m{UeaCOv*I^o=3p*2zVwLNxpZ$=Q#B5g-p)B)k5z=j**uW zLL-plUC3dSpJW($IWaT>d7z8_ixb%4OVG*4i%B7-^~9CS!8})IWBg=|_0ziXam5cq z1p5tQ)|jymJdv)N5^+WQk1Iy&%h(^gS^8X0h{h-S>6)jXkC768jB9Dru6$gfaUXMK zdakC-^@LIv7lxc)C<>8n$R=bb>hEpn^*r?FLi`LQd-}Jd|8L>C7O!&L_uzT}*OPet zBks@O`tLM8?Dh7u2_YNos{GG3o?U$#`ghIHLoV3o9r!KT=~u7;+594Ger^%#AAtRE*e~V}?C|A7u&cq2P1A%v$WK46 zC=6XoD!M}UyQdGcK90h9;K%$$Z?{^ne7E}VHRfWM8_)@g&NPhzd{EX2wh}J~I^T|~ z2d^Z&(wU2f9aXmb0&TvQ4ps?FE@5@dJUtaxU1PTKHchuNW*>(oPZpb#*>U@2k=GU~ zgwFm!$Rx=&*NdBT|4Cf+?JUv5ma+tJG=(VVcQkc$20Fd@)z|86Y2MnFzob3C?mrfT0qXJ#7S=6UT$i7h zm$ztP9sVs`ocBonlBIR|b$O6e$aNQp(b41$_!{pXK?1%GU*K^_1Sf%eY!&Tc=N6Ry3jqtn~gnRges zZ-dw0>I*d76=sHMxGYv%a-du~xdDxUyTFD@9J9}I=v^~~m2J5|*E6sTD#o=~wd)Xh zgv>;4e7To~+)G35w6rje>)%U5K4;mxY1ty{SOGVmhTQr47_~^R!v<@9`<5*ozRo;W zSJ(DsmsGwt=ZebqjwZfEs?+6RXN#>BGYUjZP+ba%Q6m6w1U~6arpRA z@LFuvZG6XWMm@>lI4QK0=K;@f#ro;V5d`&MWjby>k{)Ed>;&9>EG-I`VD=43uZC9i z-v!);^9yT)_snweTgC`J*V>7!E1vD&06?_0pQS?( zs^WD7FRHJpDJv-}bQfd)k8>1J=#Yd+m*BO9K>p4ke6hHz(d(wuh2|D-XLEa-yR*H$ zrDLueGu!9w@VWCB&0jp%T|<}o15?ds@sH~O|TxB1-bD=Jp4 ztE*XFwXUpUP2KX^^~=|kEH7d6D%^|Q^SptECU-|G4g?oywQP}iB;RHw5lBcA%Cx((gh?vF5i6^@zeY?^&`Dz_=wm5O~9#)nZ{$*@%hZp*>j z)Kun2_bxbMC0=$HXkak>=P&y-;bpe0^~|2)VC3CBHry|f@a;I-YQ<|S5GXVbT|buB zYx4RLt|f#(A-@HH*fJ*^Iy_3zSLpoJJ zpBiAPG`PR=vW@*boAvIuN7!Wmi0MWr+r2N*&d1%DoytzQ*^jz~4D3nZ(O+*66G?(- z{N>#`+f2OZJvPD5eW;+!!0Wt|9S+#-Xu%=_2h-Tl0+t)J-N%+d=la1U_SzvfGnioe z&6L--^ZHP>y&#B#Kj0Z7Jmd>bjAWyhFd5p$cr$}ZdX)ZD8cPpMOqfXh4<1Tl=ZT(i z`Xr(UJ~wEeNN1g*f1>BpOwgl$=)*?h_$xo){O3YG13majllB5X=#S{R{xUi>;s|uAnot4Cs!aQs^WzE#U$^}V`au7Bz{Z?}Cnb=5gERQ> zkpb{Olx9yQf5t-T-wgc#pApQRLF*>`i?`1W!W_fZ+GLV%Fq@Tr{RpsBF#biNe`v;) zNJNX#&vugjB2vEP-B>1QVoQHE@dN)V2t4!)+F61n17C@#YeLu+Q#p$6P+&9+;NO{?f^21n0~o{2Y#7!1i{tIi%Xe z2e9Z27lPgndaP|kf85FOcGLyF7(y1*|7ek$;J0zWL;Peone{{r|-;5`CA zD~&5}z&k-z@PO@Wj7g;OiH_zaDys7SXXGZI<^7+E;!RH^^lfLZepY!vUO)t$l%4l7UMD}p|XTrYG z0{TK#Cws#`FrRbze5LvP^E6iUbxn|y{{mn7^`FY-qCUDmGd=t@m+Z;?72}bAZ6*>q zA2Nb7W>Ea&{`fQMf2FDa-ksB8{xozD>sx}2{EaUPXBNP}u--@gi~MI=0*yC~`_Cb= z=d`GQ4LP-fz|@4OzhXU@5u7@e=o4f9HZ_y#+~4+~y$kJ?zpWwtPIa05%>ley*z@=_ zj(4Cg+LJ%pftUUU|Au~NMD?reO$?B|-y(g}J&M#fjb~T>IaCb&bPBwj&!hD((Xu`C zKZDe*{Dbt*KanCl#w-2$Qd57uJ2PVXu5YvGo94Gc-+aDiaD8V;{lWh;qWZ4S#(Lqh z=-cIr>AQmT<%-R3K0nZ2>KpxM1feYm5Eo3g@Za}H8EVI1SUvxbfDrBWuvrO`h zrLW$d$Oq1$39Vl_e**b!27e>@9TtDH^Z02e`A^e&W*7Mbtq=9IUW)cAzRd_aIK3n0 zZw{^x^0)JvKAJz_t$aPk{96h8i6Z8oH6D_E&k})LzYBun$CLf}_=p#!$+^6y;z93D zEd45e`AI%KCSQ&d@lt$NdLQREO)sGM=UsQD( zbpZ7)x{(F=gWXK0T~A`+gz=D{jh0{1#?~j9A0U6j{Fz7msDk>59kAaM08l;v|DpK% z5;-Kb|A6E_7K`uDC*-ljcij-*v*W~btr z?!RQ88~Z2xo9BxTnO~ATyr}#mU~};G`G2bXDcSZGmNPXT)-xxz3L>B8>shjmq7C;C z$V+&>-eve1$y?A1JRd;#Px>xfGS814JU_PQ1|7+>{yCEEB!5DC$j`^)^M%)4Gs2lN#NVkmm zB-CSg%X-GwH@i#5lQI3ff?>Vp{^8$D`xE%bKAN0Vlz(7;obrAqxT5W zlRxcp@bwYnr4c>+Dbwh0xxp;Av3Y3!LWN3zJd zNE-21&zO`ih#hzs+1xFR(@kgeZO^~sn( z7$4*1WTDRYo9GYr%_Mj6(|mxu1;NbBXgss>L;Uw*yzBEr>lXq#5eWY_`}-l9PsaIx z^~dDzz?MRg&qdn5ITB*}CI3iHu;@3FmMVX}hzH;o+EDYe0`ofobtpVK z{}a<=_%z^?P&eXTIQ3BN37?L-5uY)^jGusd3=es^zjA#*UaGsmfzXc)^Er2ne4I}% z5nnD=nB&>=$RBsHRPt~5(>V5e1^H7dweXkt?INCh5j84?^)hW- zI}v}8T+fmhOLD{j1jhLx zz-yQ7H2V$n7sR*g<}ZH+|3iHHtn-)A`HCa?B`?xXk&j@$xg6w= z{CoxeY0g&`0B>BMX}z-MD~9zK@gfWBHI@?*@1Z}_dXDwkw0@(Wh4ng0kM>xPV*9Nj z&AeV?z7jsVUXPit@cC`YS7KS4f{B^G>&&!h6AJ2xL8|M~wiADL^4U!(WmlwF|6{r7<&vlE2pA2`1+ z?RsgSJC}9);IHhdjURQ7;C@1i;JVyCmx^aNR2)(%At62AD{1& zCql}hG?ytpVf{4iueVTqO62=<9?y^;rSG!y{0aMkH1-dH-SBZvbzKZ^JzQ{)F-DvdLep@eH3Q#q1A$({Y|Oa8T+8 z_QQT=GJPJy*XQ&2ym9Bm*#7(>zCY*Zr##<&eF5kv8}{Re@OguMGT)DL|M+v-kH^k? zFdp_#vGXge|D*Tg_fNfXe4E1k{r>pz4e?{j{e1tW_T!`Do9X-~H+cUPOMHu+|5*38 zn7=o%-=1QOZw0{#I>QH8Vt+{UBXhz;SV-)@v$6lqnn0h=8TQ{(X}@TQcTU=n#P;7# z=NS8M9H>yaop@Jq8~g8ctd?L<>{rp=g?e;9ihON?@w^H64C8qe`F|$mn_~aT=R;<6 z|H=2?V*Z+r%N~J6ZTKnE&4R=N5;-y|A6!TQi3^uMSfel@7CfQ z@*~6d1Jc-PI&9*_kk8VRAotVxoh(A}kiTL5!RN*NyjPrO7|z3wf#@dA!;j-U9F#qY z@8ejdDb2FKPSdg9XI}#W_h;-sZ^$1$4~IX9^YtwH{FHwlsnZAMiBt!_t1bI&zF&*V z=VT-wKVKvv#rkR|2jT69)2V;@xJCA-#m@ipZcbkF`9T`{BKQ&eS#*H&pVID+V)+R^ zKQQGdpFtf`-7EJra8CpGG;mJ?_cZXSYk&^V5o!nUEeF=~^J~BV@JErqc(JeYx10X< z?|)W4?ynj@y7bBM9j~nMH9upT0P%Y4S^pypC$@!8yQRA%GH8 z8)r|s!XtDc57e=`hQC<`@8g^)zr*&Hu|3){1ahsOfj(CB9A~1&ssONQ9+7xQksjP> z;}X%HG5}4R%llK~y^>%@y(qOPkzR$V9=le$X|mu2)M%mLG{zNH1W4hiF)FF&hlwXX z#+6sV2=zrT(T?*r&dM8_u^4_&iWN`kj=t2n@Cw_E&y(;y)f6O%y;>qzj+S`%sA665 z+fy#ji#t9Z$-gLIAYT7G*Cz^3e7wlDfQMq@wU6?@Ii`DMgLw2&dHjs6)wE+lE>l?oQm6$Y_eyREg>@Fb~ET*PZD~M%1tB1!AIqG#m7hGb_==V^%0c| zjF7BKG7O1gkj(z6NM;}HQDg}dXObq$8Kk=rXQSt!nRA#tqL)n54CSn=&f(SRIBddc^hU7v<3 z52(=US=6%)b;47fD^nYniVTl-05)wp*jTISIWELz&u-GOhQN%K9&fQ-0p)PASGVc~ zb?S9G>eLnKnnul0_1Hecrqd{esq5{9uE$euZZyh8#?no^T!lEjm0Z9Lb`3h#5y}9G z719XlcvKb}C)YE1uO-i;evH-^BN#?z){)&HG2rjTdguc_W?G{^i#77o81u7>n(?^* z?6|SobF8{?hTV(dVhtHfXP*>r)cL&_ZeGPjtWfb`bljmmX8e;h%j&LMSAU8ZhMkZ`Ov8L(LE=hg9*_Ugq9x^EB&2TH|ZzN%4d})9=+P! z(TWTDJ9>Y~`8S{CecsOYKJWT9%_rlIO~m?G)Y~&^$L4_t#bWjzW{i4j*1oChYf7-! zX8i11y8`c{!*J}jRT*;x{#l7fe1=6wvX*mn;o~O!fcSo#*#5`*AAK^;B{1}_YrOv+ z#T$L^%-FxgD>VNzx3#)f=EC^`%_mXL+HqhL=NU%Y4WD7OST(m?dsYenKC*#8=Wya7 zA7oTn5~aME=l{fgw$COvd+NEI=fC^h#8&pbIluXBt?jR7&Yhn3Pt$&UC@r4@WxFp~RBLUIKZdlNafyCc(jRG}IAjA_?`SN=j?xg>!`dgctCSPIi7JL-r6s?R zpNWelRnmT6wm)HLOSe|NlmAeAkA;m!X>vDkRLJv)cWe7HLgMS0;u3rRifo@NK}yHg z_Ozed8>jtoOMBI`wf%IV5Whze$m?m@{sUR3@#(uJ*7oH0=wcNA19FPE4ejYxwO4jL zgpLX3A*7+4vrI8HzG|oJ|2P`osJ)`2_C#lGPxG8qU~yB%$T3lRa>SL|6HSk(M&I!# z1H8}TC`8^Yl~)>67XF8>hB|@23$G)dsDD$D?L8)-#mRu9^@aE^ve4lX{=EFu&~ZHb z`Qe!O`#Q3p^mx(k+#ieU;s?Yv3m1~#M1=aE0!ZVdm{1cp3LtfCERr?~T95X7;*9?l z**;fN()d&?>SI3H-eVd6xNNWdm)es{TiY+jiA)1lm0Ku_LBYR#j5{xX#bz%$gkR8 zmM!6?-2T?_$tOsso9iW^<&r4}R9DrWz_;6uM{zknEyG3UtD3jQ@rYM7FRblU-6I{u z*p7qbd{T7ks^*uvs&-oo*VS|sE-w_Ncy-ajmac|HMJ*c_t*hQv*s|Wcp|NXYZB>3l z>xSy>1#8>Ag~h9jYl@m0T32*#s;z2i@UL6nQnq77!@?%O)>RX}q-ELKjkV=X^&4vP zHr1A|@@^>D-cnvsy>3V8ik20RRX6#o*0ro$Sp8VpjE$GvsZ7uAZnbF39uM7WwO2eXvhWd41d3CsvaUR@O9o3)ihKYFm@n z&|1?}zp1Ldc%!$v+F!M-rMRiRWW&}Cd7!$R1*Dt}|!y6UD)!d}~o%D15JyvEw{E~{O<#r`KYHZOiu z`J1}dwYD?|@Z%_pw=^tS=q*^iw0v-!2`1!F%8sFk1x|2ma!9)+@oxW5~^d1RRejkQHQQ34TSw?Gmo!n}Gh5?+|LD#80xo^CWx<;HoWwz9`}M18&?h?lQoQ zsp4@7r*Uzb7%eB{9gS;*mqQ+{!;aMTZLUGez9;_r(&f##O%9B;jgY1wUe- zH;#K$!mZ=}Lc*=%rpkFs;~M9SOTtIZ7YV0+6fY#eW&>Ov@o>407~s)A?g3`898AgU z=G2UR!2mbP)g<9c4~o7;!maW?FX2{sOJvAV?vA?p<~DDjn_-X8=w0RQYHMhEjQ`~1ay*b3zwXF?b)H#G83B8#~l5x89+@>nfJlPyD_I zK}4onroOJo1T{0p!)cL);vd z`51k+M)6LD<7f`{cL37(#@w1(x@7S6P{mNx`0DzsY)^d~OQj0ljYeo}d<7*m@nqFu zWt*T-{3?2D{lCfauh;$~3qtX&`b&rKR+|uRRX|aguU6V^vi&IeW&39=4Xph8fRATC ohxDJza(J>Ixq@o1a_IdS*g7^fIoCn&_~CIWKe0hi^0K!7f4Q6;SpWb4 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke557e36f3e1fad7aebc9847d4b6b62bb.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke557e36f3e1fad7aebc9847d4b6b62bb.o new file mode 100644 index 0000000000000000000000000000000000000000..f5db1192a58b013b728b7abf0351d07ff24bce75 GIT binary patch literal 59256 zcmeIb34Bz?l`mYqs?}<=BP4`TO9+rygb<4%@zRn23Cw1|Y|lxnwSZ)_y_N(mTM6yP zvB74HAo*fD$z*trlc5PXB$;1=l1!|`9)~!Njrrwe{4$9{FmVL&!YCd`a*XxQvpj-Fug>USYLZm`WD*Z5D%6E@PpKv3noC^)U}9 zy)3+;-{${$p4axj>4UPp`HUUDV9PuHDn1u&d9!|t>udP@4)-XUsc7kX^`7rV|Cq#v z-`Qe~zP-*Cy=r5J|HH<9YFypZZ;bx<&4JU-!TgDYS7>EPwxKR*~f_{QMz6K{ZSQ8fDS!{_cgdGOqUxhDo!y>N1{sr7}y z;|G5_c#Ni za}PE@e6IMXCkG$=yC(+^B>!k|R>}*5mmYd^aMn+$u8%~EfuDs>)7le||Eq(=%`Xhz zb>fA==#^%RnHGLLFUW5UIWdNDCug| zlYe*aK+%c8SqD*9$@7D&iheTqS&TKVLG4aKj2HJEsB-swbVcthe3@XUI8y7cs&KX zd3#-D^qnmgutVSQ)q9>frnl#K_W8EWGE3Pg*`u;WA8hg4u+?ivzf|7G zst4@HUIc!#t*`NM%kZ&Jm-jW6pv;fpZ(lt>y(WF%&5cKpzYqCY$bSkv?g!if_30RJ5DmjLeo{87Nq0{%SUj{}|t_*;Mv0p1Ka1MpV>KLB_w zU>o2gfcFAk0r(HTzQ+3j?*V)Z;8y_$0e=H3Qe@a81GWSzMEF{UNRixPBj38?N8Mbp&!eiR+(m{R*ys#Py4~{sGs|;rbS? z599iKTqC&t4%gkdZiGHQi+0S$bvLd*LAy5NS_gaq@P1sU;@XDmV$1NzKK9Z*@R2pv zlEd}Z`G*@Wlpfyk!_vb$A1^&zf&7tN*7qCC{*=V}$iJf8;lOJ@yt(GW z4{r`wQ%?ro#QkNuPdOQQ2ls=()w~U>f1o|*-r*O zj{EI&Z#x;-f%{szx1J2N;{GnWw}`&PoSk_4V;gRR?w`Y_=|TI|o+I{QX6e7Vm>mr^ zUPwK>t1Pkau8-LJ?ykV+h`q1rVf*j_&^CS+ZHd_XYAPUqd7r&zh4rdGk^RpAW1B7^ zeoRE{*c4gTcdN`RMj1QhQ9a{zVXuRo3|QBl^g1lKuEiB?vaZ3^cG5_{`s7NxqQ6bjuafjD1T)v%g@TC%eOC00gG)e^rW4*s*iua3j_wTeEsC3T4RWow*r?*o6Y#NQnUpH+$SX`=l7 zaquO;-z)J);@~$^`E*hK@i_QaDlhThh=YHY%4dl3uf)OcE+2N^3%uEGBFML7j?RCW z=;P#nj_40coFIQi#@e)4GgpA&tY{9h1#?r8egh(1pKWul)ln*L3qkCQ)0^f!#Ae~0Mfc|Vq+KZiJ)_XJ_+SKh>0geFCp$5K#cmky{zxGkFw!EApU$i zsl4w$5RbmI&o}(rHpB>@c0~V(ICbbxzP@3^t-ncf7r%>m_Pw{(Mz4OUyl-G;V)3OH zeZvEY-QNE<-|%l<^$lP6NqOI^h`lbpTHg0t#ALtyBg4G~d2IN#Ka}^q_6JMfg`ZgZ zeuwz&N-pXjVZ;AkSKc@H5S3@cml5avKBJ=V{eNS_uOkM0V_HSu)h}84-bB3kR%J!s zdv7I2f0MMT_|;D@kG}gO2049@-#0wG4?Oq!`rh5SDmwHh*7si9O6a6II`EIS=)i)C zzDu@>zCS+2hKKyOMc>)ShX13(()Z4m)zP<;EPa3YHXD9>-J0lIl~&YaL7mpV*HbKz z!`gQ_!7@Df5Xori8@Lg3=%;z;~I9@a9)$K6}92zzOC;)?R`uls>vEl>MRdVcJB=?i^Ps-NidEj}LWHoD{2 zarK_>!1gS#J*>?ze-8O=us=KO&k-H5CBNhXFZZ1CsApkCH2J1V&i9kg`G!-$`xTq7 z@6^7jLzeRLVppOM>!urqiV~5Ra8vQI%DmG>pgCrnGE~j4Hw>-JFE5ThJ>~SVr*luQ z1brjr|KmrZjVSYDe2!I4KE0pv_T>!i=Wx#HV;s&NI>zDb(`Pt5X=niLIySSU*j2bJ z>PB7><pNaCYbzc5PqFJrc+ywp~ zCB8E$e@*l<>~u2na*($K*Bfzl6|RkD;eHW5uEKTEOk8L3`}I*LuG9Jb?NK|f)A;>{ zs1;Y(4CpsT$$rZ4X`HbsnucpBzrQ1zf@=xC-yEHQ>l}W+C7OgQ>9KLfR%+*PF+S&i zk{iA7z~tz~Lpjmc9?6cr{ST9(NB4bv=<$S};;*E9BKm~w&Y>??`cMDk(|4YJKiAs# z`>*Cj-}vU#=v&{tA^Pu6Pl>+suUXMQVq6bloDOfa^u7C!Z1_)=mcIA?0pk?oeqbj2 zMiQN)_>o_E`{{vi5`6p6z!rixoF4cR z!5fAK))Bn%H2ezSjY9)f1aCS$u#ezPLj$)0w%su_0RMjM={ruNp1xzY%|inV0Pn+f z9^to~9@vQdN?dQFa$AN5rXjy_>uHn&yme?Gm*8!u2WAtzZD=5a;O(c84|w~~Koar! zA43BPxIX_|RW`=r2C> z7vuR0#78_WK@f!h{6l}%zj4QREG(6|X!ixl$}ARofYN({gZxi5{<=WC3s)aL3HYS3 zRAMtK?P>_NHZ*sz^}rMo){02X>cQ1&Wwr@6i`8af_9TSqGTD+D_F&jv@f@*SA*}5C zLPkmE`>MF<|B|>`cQL1rEn!YbG?gezIvP7VgPnm|+^K7Ki-7sn-_h7wA7Uk~ZJiAz zt-DJ)cQv){Xp6yhT@v11qTSTBwX_F1SxIL@sIx>g&0p8m*&OKT@V6}Rw}kvV8(Mbz zC1z7YYe!pfO`yGFPt)CZH#cbTMge!sFI!UUU*NCr>~3$U<4h#8+CWExzrLdxh^E$^ zcQmzjF7bD^1pGVNf-QkgjnA?`XJ7@-s@S$^xxdU`+ZOWIH?%^!#%gmwfzI1PbA z_~4mZeM4>6PXCTzprt|BojD^=V#pyS1e%+71_SMl+()(t+dA9GL32ncsPu1qERy%H z3$(X11nWK!28`zLwwU{pX9B6s;QEXYKp(2+7;Tp4eqfxF>Ra9*vRgSHmbA5ZHnlX} z-4OhsbR=)#SSI|zZ8rK;hH$3Hhud+7h}yO2;aWEz2K^rf{eNqVu=u9`fgA0Gem|D6 z*bN9Y`%C?R>bmL!{*IRUr6GT3-Oh&A(nbEAU4dYIFwoTO??n9NuWM`e*EV1Q?5}S? zOx9M{Q18cT-5+e|?82@>y}!2G&-0m{`TXl+-UUIxGSCS=sHm%@wZmN5hYOYecAFL- z+!(+x?W38-Wl7w~z!>z!7Ak-H8&fI{*x1t0Qqs0#M@K_vDf9bVcVp~}y^)P;map*( z4BSbexhc>Qhd?=&_CT<+sk5oAl`rBn9>g81Aj8V;Z+(}Auk2)m{I|YJy;i9A0ml`i zeKzJ8)ykzj-#IG(V>~})RQ}hk#3yZ3{t|4g+gS3b{4dyO&p(Bo+8|J+l44lqpO^Vw zx!eC;h+^YA_C{8}=lO19CZGg7EJ#n)$+=08{0s8)aqAPH#_<*8dyl4R`4Y`WIDIVN zD-qMsF5;sY(6awhQme7HF9sb$i;63%^Wim z2`io1mmouwwKr!9B=NgR($PI7swW%?7n!SZMNn-PVN#)Wk&aLvE8sFtk$H{ss_IpJ zMcKsvFJ0fuvOcp7P&S_E;N>aC&Z1w|ZeF=;Wx2PqE!YlQ5{b+OD7F}%9hBAH8G_5M zL=fnurz%a&0rJn@&bGGZjyYb;cMW(3;+?;sWZ@j|X1bItTFNTAnwsmq9o-$B4K3c5 zu8vM`eZ!8X)&}pUb?a7d^lx6iVdKhmYyHc&Zd$%}+45y1k;-6o7 zW5A25NmhEUglFs~S$DDfk#C>pVr01s=swNG<;E?)hgtJP`O3QO>9PCkdAw!cVTUb( z-;H)r-z1WVu`>3yja^A&b_vgCR}63|Nqi+=!_Q;q4R8s2*#OUFXASTicE$kDX8i_u z7CUW#i`gjyT*QtU;F;{G0iMAgHNewZuK_M(`wj3k7B;{IEM$Q5S-S!DvPJ`($7&4l zRJPRs-@w)z;3=%y0OvBF0iMiC0hep~&0$^xoXt`V@Fa36I{GYj#j1xVvhxNwlbtod z8SD(;SbwLp(*}GRJ7$1W*--<13hOoClUdjRPhjl^*u!cJa1vW@fZePba7>;=<^vqF zHy2xCfD>4$f!@iyfMfb|uv7yKUu=MFENo#{ELLUj@D&Ew!op0C=Yxdk*V*x$7sTc0 z06bW>+>UGR2(zxnBW1tMVxR9}z3bqM@Sz9K_Tp;Ix07G|bCSKNnnn#v#ngNSpY;$V z{1H4Khb(9DDXjpE&nfy;GS-g|TZTCdAGV4yA3jIdf);XFSB05E4RKlqeJ|$tFg!?6&E#b2gu9on5340}cMZ)VP3@hjI6-gMQ zfWuoQj8Vej8VO?*akx>!J_)x=xJJU^=R|vHT2}4Hxp8(XL%`Htg3nGAFtv~1^B8zU zPwgT2N{)a@eu5b$NWzo61gB0FFv-W^Spp{coEBImCUl}F`3O$ES-{8woW&UIOD~^K z!b-ldgq3{#5?1mt#7tCP$>)=>k}oV_C7*3cDzi^1WMSA=>IB#pKBv-fMNoJOpR;tu z?W~44)5GZ14;fKvMZCMp-7ZwR{O`r$-dM%0D^= zy=t4jovO@V;u^WfGQxfhP829yNMSkMl50M$CCTjRZgdW%{rJ=-vq%VpLVb+AkODk9 zjM8*`9#3Y6gDxkr8tH#Hg`JtldP4~=vcHk@BPqncH<4$jc`UrSy$N^E2$XQ-Y!FYB!X}I(8jPd?z zmz|M)fGOIO%&a|DF+fs8d#buJMR^uZVy}~Z+70!84DG9@@+99of* zp;HzYwdbORRV)3Eq3`*fR<0_cFW58M=RQJJdyVvGkLmAR3frHE`Y>;! z%`Wzuh5M&)3kp#E@UMpc1i#);ri(oq^N$BZjE~=TFUIG&d913-)#Wjk|AZ1=ls^Rh$Q)}temC5s=phCA+YDFboS?QM#j)RCq_8yy zRC$3%`_ev-tacx@5At{28M7y}5B@Ff3&5-S;>4cGG5@|JnH_<@qxCnpSG1q}9o5hM z-?J&~iICgDC@!;NegghTD9Ikqlk~uYzdNksAL((xaCqYS>A8XBfIkv%U!-R$~ahH{^ApuXS*~ zi>A*QzQAO^(_;0bzyGO=c`(5v2v_=9WYABfXQ8Qmg)a7m82?Dmbg?g~$veX&@61Us zZ%v*^=tcv7)SsNGs~`PQ#Hd5H_7=Nnx>xOq^ei^XQ{-a(G5)=wIWG1^J^fq;i=!m< zSz?fn+Mlh}Z$W>|n?&)aR$pkTLEcEu5>x%9F7~*-zA^*<-q3($CTAa9O~!X3e%#vAMp4olTX_NVthRDYpXK85OE$QPVi z{jkTQsyvNvqdk)SQE|nekZ|*D7+H*b&v=4=~FP-dx62w32)ra|ijW&PMe3Zv%eLbpO$KSCSzerDx zsr+O^e{lbQFh&o5k)ih&H2(DZj`ZZ2>YqBzSU<)a-T;otpKmH(V5%SfyjM>@-9$gr zRNsu*`uZ@Q+_CtB#y9-&2|d4J6F>8O34i?=J^fr0{XA3sB@0aT+f&hBEUfGi{`g5f zzxgJ9Wrp^X{8%n%!S1BLyFEZih7-4&2PH&} zSie?vopNED5UDqb{c;_}(<3wgr%ziyR_B&jH8|VK>FxgJ=t+H?UyF}`L1yg-| z{(Dc>hxpMb4|O-}L*gF^WhU$W+avH_)ZQ4sGz`Cy5$BMSU#CyB#}GebJ@LD4=CliW zp)VVUldHjy`x^&`GuIp8tgQw(O>8iwCbA0-(O(n~QGZ=bM*nP$_0Q!)qvjWbJdh9T z8>bKKMSJG){V;G7`(bZm{yIcCCN28YPVJO8uYRF5%Y%>=tvr zqIdB18KeF+t-s9n1lYJg)Mkta)Oe=&k?fN|I6Eh zS8oqPF0Am8{(}C2{H6B12lK^IJ)D&2MQsZI*O)K<+xdrnC_wKY&PZ6AIgoz z7l_9qA+Z(Bo5*1t>R=JkWWb0=4eNy?p(%J^9*ZZBgbH(v_IN`qzC?NKUplLG?TLis z?k`oy>zjn9y%B0Mhq2#ah{unFrl#rR-$*FW#DA`HJ&G!QMM4OlRpR`Un3qLJ1kV3R z$eTtRG%8+>gz`=NN}O9k8sjITTUDOovr-mmRPd2dL7FkXo@U?&{lGow>zi)SKlW#x zHM;tagtD^@{e}G&bRZ?fW7Hor4E-Mo%{0`9czxnzPA=l{BcUQZUyRj%Bs9yRZ~x`J zIKq;i$(*(@U(W)bGc}9Dcz#a(SL&q3YWl$bIRvD{<%#qxiphud3(C*+Vo3^qygb=! zOh5C)`b)o`gy$#PdW6@91Dl5ZtNG4$R2Hj$v8nzWWA$S{6!n)BB&+&uuVK%o)=0C;gE>#~B2yXJ~!D#8ltXSbc=Yc+$&{`sd9^&z9|XV1GgQ1AhJk^ycvo z`qHATg_1sjB*uI`(9ejxF&tRXhSA#wB_*u7KN8?v-Z?FfQ|Ck5;7O@$S2T9tu z*D#)XL$C+ke&Xoir-3FLQGG46XkG7fi`2()!oz&w-yd)jdn%6Cq!=ZoNVMJKcFq$t(J|(3?C*>)%Mv zJcE2>A4e7ag>~q!B3*w`{~7zs*dJt{Mth;>fkohlMEzso{j1v#?G4Rvo&qwa|2aDP zdEjU6|2X~I3;!hJ?a}>P;5@D4Kf1q4boJ4Go^ia1{!sdNKpx}#3;gscSxWzW{LPr^ zR{Bi@y?OjibDq)3L;dB|;CVED3%toCTK|x}oz?JUf0@pH4W|Clj|a2<4*_o&ZMU-Z0*D``2WDFKhCWKQxXHbAOrpCw_kz#>X_{_%QB|L7#>C z@nP-{>c7}{NP_$Y&HR1fEMc|1$>)OHRFeR6%-NM9&F zCB(JQTvx>NcA}@}y_`Sj@9Xwj87Y($J|kI2kMqAMe_!_$;J+#GcwSEB5n$=crvm>i zf&YrW{E#f~0lrV*FNpH=d@)vkHt^q*_3Pz*PnItN{waZPBzXz}|4;{K0{#~P*TSfR zUb`3kVV`!47rSbI_7#4mNCTEa9zE%jrM=?C*A+V9YI%!nWK!~MI|5BfvaFUD(raN3CEC&!yZ?suXR z)NhuL_>1uXJlCJYA;*ITi}4V{YddOOe-8fsNlbqZu0P&qTz^iE!v2S*KWD$vANMzQ zz5ZN5dqSR09|n8o`pZb59Yeh2B`af%(cM;y3c2>p(v%QThWu-?)M$ZfXA# z&P$U0$Km{7&WJYtQJ>I$hJM?x?s zNAm|B`Dr9d~I5WdXG*RzT~%ay?A7v2Wo)qKtS zfZ!tC{6p{(7pLbw?itMIE-Po1Ambs!{0{n@BHet2_Tv2`yg&7084uZ54)KHhIM2uD zQyvd^C4Ruc_7`J4?bE?anax!r;fsz@?@OK7gYAlM5krc%l4(85!gza&uYWJx%FmmX zQse0PAmkD63soQk^#jIu7W*0E{Z2Y>m_YFd-@kbN5XFO5G5^W+ePTPXl*IEz+Fu9* z;`I3s5c@%@ObklhvrEzMcL+tWi1+xvv4 z{R+}2&Oeeo_DrF#MM2)bVbMPH56&B=J|@c}o+w1T%lXUoKjt4Rk9XvG@~n`}k-A6H zWBx|_8JMy5RJ05FSQNB!`q=(KqE(ZR@FA<6-j3sVlrIdqTvT78+@Dw!0Uj{NLw>@O z{`h(Z^D*SJYV_cj6-u(weg-|l<@)Jy;ng_Mr;78Apto_D+JpKQ;eDR)k4Yn9{hrn% z@h;Ek!U}^)Kds>}TNo9r)A!#>KQz`%A|CGMtasCP&fV2l2kD zn15go*uSdks==dcN_c*DIfa?e-=El%Il{ltpWX+q|2XzXCNb&XTvQ(CVTHRKb3eqm zAI9H@dIi;azz_BnxhH3Y{R;XZC1Jk)GR|94`;r;yFOSyqY1AL>zIW7qNLmWW+kJ0q z@H`HqJ7%it(5N`UL(y95m!~59xcH71JJs0gSAp1__{({@PXnz9PyQ*(`2-{0&Ki&t#{^KF6Po4aEX(H`M zqCJaxQhEO|dVlQsdEoC#)$}`%4A`B@-&Z1ibJ&xLHy{|}`cFpvwDAi5Bp=4xz1@ka znH2D7^b=Au!+JO&HS<0_oRpfmUk@jzW*(F<^pYM*7yI4);E(p1-?tU-%YmK}>Zkss z{zHO&(0EKw=L1;U(*syP34bB_=Ml_zBrbox1pQijGI{^A6w*Jn$C;_=qt?dzllZr% ziv1(CzvH?h_UF*w(%xY2@JEm*LC6#4`}-Kre7{@m57PUa)L&x%n%?J61bJA@FBo4r zcwR;GS31pK7+-~<6ol7!9!C1+{w@{MwS>{$9Gr(I`V3A__K^m65u^78vrEr&d zFq9rE|4a(=1~XErJY&Fv-?U&BF%V7z`kfZcNT>Lo#|XgZ2eU@>@fV$k&I3L%6Z(60 z9q<`hT08-JJFqu1gXNKl<9}fZ@R=F0_>S5~ucK(~vCw=(^6>RM>OZh|VkXKX1fud} zfAai0=>y{}zmwqbtti6f&ks)2#eaFwPbTNb*SCG73{$CeM$<>OxKrB z$NnSPi}Ii7U+7ErFXm6^&!NG!$rL~E{oeK(zJFpTnEX$Bjo8oeV!aiU=Md?W>`mAU z#*;TV1usZb@^OE0U@w*z3V(s{9*GJtykQWcz+U%?*o|@ z$(lg&@HH9i%N5L~4w3ZhfOjO&2_(Me;`W!o`1mLL<@7n!L4y9`A<$SvHxW=Sk@Z79u3M51@|)k+A$bYU&x2uq5B$>L8JL6G z2Y&J7=j|o=(f&nf;>L>QT=2eNE}WNxe{&xz{khATiMp` zYWryli0S(x(0(3I{0aoJeCyqo4rzW;#!O2hm~@{KGJ>xuMGf`isSTt4(4(erq9eUR%XCnVO74&n#> zrorD5`N(r9|1;?4h$DgctN0(~9i;E}9qo4Zi+On8^a)`QetcDz4?j#rNyNuoKfeGd^+VqWLRh}YfxdG>jL%nGKENmTpt-L}dTu`x4EDpv z!*3-X&)04VPT=-1LE6Kj9xJzR(cg&QdaQiBTWNg2KNKQZc9iG(;N@w2U_Hmz(&R4y!=Ctl0JotgcpY(Y9uv~gw z%3nb8CyDmaBnv#fA4A_8rq-`mzwz}rWdWax?>ESJh5ApP$H@v|eZCvy0?&F~i5ce& zc*HN8wDnf%?x2TvfYll1{_Zh6|H(sH9xtH&EQ(*ikFU?3q30blqu-5~7lRWzQG%+~A!+A3*p9lU)Zoyyr-wqGsdk>_K z9n5-QZ}J3f{(hXEr@NE5@22<}@)m}Y5@@LLcntlKj`;697$!-)6<%Kj3A$rHH0Ao-qK$GoAGX-gD zJ(Q9{^{M%^DiP&)!g><^osWlKB7?>Q+G7n$Z$nJbp81_gTwFCi_;`3j(qF)Oh1O@k zmv8~TPbz2^&+oZCtYc3E6H>W_7`|ok{^|TmYDHvaF{=5YG6!>6LlGq>9#)q5AXQt8o;TU)Q z&Gj>Y0;C_bH*ydB!8v#oYQ*Dfp>HPi?S#ogey*Q?lk^3M$ASzBLOP511ol@N#P?k> z=4Ivh3;7XWB+~fj{fYfRnm;(b)c*rmk8qf;Uk<^Zb?X)IbHP4&d9u%1>_1_C;XNq) zg@wy=88|6F^y?;4@qIAsIK4JMfnQ-yQWB3HWPkAX|4x?2d@#MoU@AT#J2;)_g}#tbATt|`5~4UFn=cADcU0ETP_a;pd?Qr{2%ypdL4g) zIe$8jr1v+MF(eJ=r7&MXA3Rwe=KG^L-Btn9-vZ(<)HSjI^26W%bI^gnl~OW~&z|WZ zZ!1$Xz7HuVp9e31{b0RL`zPGKY5x=Yc&0-xzo`i9`PJ?bfu|N>zO?q}&-2XV`>#|V z|Nbe=548Wp=Ud|U@;Y`fab$!xpqwo1X84g`f^E-B=b_a5kqplxwf9}#)}42{74d?P z_0xQfA6ut*GH!Xp``G&O`2M!;N5?O3cwbjvp6X9!&x}{U;r&i?`TNE%@4Z(0hsG~& zcz?`Xe=|EXp8Pe}s{g#9{GEuuxc%ZS66y~_yb60>-(77z!RvdP_m`QT#y3x7Z=C)n z1ASHZdyKyCllOPO$^U!+VWEr<|9+@}4>jeMqcFGhD37fV@?15_FK6<@8Ev%CoOxA?>+g)k)0oX>Q}$|)m4gKElGCw5RZbf zX@X!l2IFg{v0StyIo_xRFuk-r|M{IHa!g~Fb<7HPZQSBa9#y2=ew+eI9q+#D$*x9AM}#NlDimE zZUn-HX7!KUZn5Tm9A(N$a4WvVQgW0fv4pr5zH}TsE#asw1m4>`NnbZ?HvDj*}Kg51X;rmM#5H@W52Qtfgij7 z=_}Q(xsR9GwDQ=uanldqh(!3?EtW(pMo!Fd$nq^!&2lse%JMq9n?BaH_mvH?`WoW! zgOUCb=CeQ^K|^~Oaq*HjZ3vZD?RFFY@@4F%*a);(Xurj>(t|pEf*Jl=su*nj>pIvd z6C|&qF-|cg=hKrKvxr@1#U{M8<1x!mfC)q)FNFYSEH`ei?jSzg-lnB}FNj#*yX>6qn(9T?hQo_pDC&Ak9$Q~qVL1`EAU z16zh05(&Ph+~cuV6nUKV(e<~(G8EDLj;b?;-)F!peigHvPD@=qY)?FGrb;U67M7Vm z*OH0+5BJk%K2k;{E7isC()qK>#tiQ#ZG_S8p-2~StD7C zvY2aGk{SthnVIjhRC=aju_lCR;Pl(^&`fz%tH7rMpA_MEN53O3-j`%`QF&{>Rfp%J zP2&+xif|fRzfDgA%Z}C4pA<<{ayaA3;jz+p;)h9&332c~PlV$kMTRc1t?rZu<(#nt7pJT}uw{Bme zCR^OP!-<+~)>z$!wkg>nG5jbmrEG32+2ZQ0e;mEBm~3%nN86NaI@$lzvRVHteX7%2 zT-ngJII@lQ++sw}L5%(>E6qdwSoMl|?L5WMa$aNjaX+j@fTu+3s?qb%2Fy)6FgL}X z1q&J#v0$jj_7H5eM^L^r(_P`IaH?y?61$T=wCd)yN2xZ{YN(yB%2KVF!dF0&Au?Iz zIj968ix@t29K3fNeCarNm%_)|ZKywc9DGq+d_gi!*jJz>7N?J?_<%Q!eisz$yjHhW z+s*iaB_0iFR<~lqiPEsH2P{_J10J8f93S2bbg%Cb-V^54tHl)i>SA8UQUogIz6;aQ z$$r{CQAx;M3oyZEmb)DbU#r&k@=-$EdoFNgB~Vs86S>2#giFB9&@M z$ID`_@BHZcM$y!OCdOmmD47S0_)+q{3_Q0Z-VaJ%@xoa|_AN-XG}_01>HCd?C5&NADDJuL}i@+hx!@m)8@Hc>~3UAHiH-NFEn)UMko zd_PC?_c+WlTx&JNFn3-H&-aI}v8LqvJJ-OAJ>9X_q_k4GR{gZDxE7vP0cR8Fi9w-I z08YY*gfnbCbeH0}Hykcvu917gMJ|3zr{(Z#wvyg_xw2Q}@K;Fe0-pCxwyQE^(@UT8yYf{QuJRotMmgVR7S= zu(;UgiQlpet-K%{*d%Es^hn-PVwcoto?qv`DeZdo7|NG1q_D{3^ z@W-vTFVCElU;5*zKRT4+S=sEK2ii&E#A_MDoT~d>^7-tF0WM|mPrBbJpU2J{;1UM^ zq{Gi;80k8A4uc2L!L#X%oerMGFzR)1F*{{|i`X#(Jd+(Yz%$sR26#H_HNb^zzX6^` zXZm#Y6|j&2&S&ig*vlFXa2~5Mz*E^)1AGHpZ-A$;Y6FaKo;AReSt;Py@4DwOuK~_x zsRkJDH5uS6dNV**ej+<>fHT=y1DwIm0FM0*eL6dBz^Ace1~`>|xmZV!Z^Ja;ld<&H z*FS-^8(CeGZ4KO^t0mh5q z7X9zu(-fhHDN5GEd~{y1SnccVd>mTJ(E)H37CaoV<&H4xYAdt(Z5I1{5A)u`SPeeC z_}Gi9HP1-iT#FM8fS7rum5Q zy%P3G_>_ccJ|g^C30F(_iiBxCB77=3isV@@;Zh0He8lUQ@Ky=qO(afF^AWFK!i^F> zCE<1nUy(5X8D!Mw%N6CR{rcarr}h#Y&KG!UAHin}1x)QB*jprElAqxA*#ahc3GOcu zFv&-7YNdclKBvXbQdbC=;Q1gzvcD`6#{50fR)EBQ`I zSjp$zEAUFbUI{DtY)ew{rQ(IGG!gcafqp>Xa~3a5hAUu4_?*RuLB_L?%f8Udj<%4b z^a3Mj$hH8}N8Q_T6dxmt{_~H0Dmp+W8VhtA?>Fx${|l&tk_#niz9YM(wAz&4F5!cy zpAsSh@g9Z%*+Dspn)AO$LL-^;ZxY!&Up+6PEZT;oq{$=OE@seJKDAxYsjHM<(kl&^ z`3UOKd~PD;A>AzX;0tq*Jufa=zKUR$nertlzHASG;-m79jX|&Qf2nIR7}>a`7&$KG z`6{}J;b3NFP~hv5_e#Vx`1yEz zRJs4^mCOn%7?pf7=OuX#M-fOfR3Lv`9Wu~$bRbKUDJu)cIC+XDXm@KbgRlh2y`hPAId%fk! zkDJR+hy2$nACdx4{X}OjPyU8fSRv`u#akfmXwN{Uc={v%M&+fN6)6kL@@$jHqWWq6 z3ky*Di?3wnGF56h@CM)LB%jG61F zKBF{@n@F7cj~?Y{oLmw~)vwA)wX)~npW~G;{a>P->_%Qc&8p_|C(tSJ${&kUp2mW? z{Ie*3ALOqv>CgMPs9!&R;_2^8n4j={AMLPTL;o+7<-LN9_Sa+%><1|SsW|%ID$6&@ z0#Us<)VyjP>h9dJZOfvr+65aL>#Le~*S2q5v%GBW{Q9a;W8H!^UE3CHF5SLyQE9{0 z3e?d6`t5gY-?HfL+6A{!ee<`q>|B0V`Pv0_Wq~^aZ_IpsoinqTe)W|>8ZAA#ocw?i<-7=UE931Y{T8Pm3qBxURB$A``v4(AFHLTTUK;$ zT~HsNXVuO1+WmU)a@n~B(ZChZ=hF#mYEFPzSMg25h;-F%-P_7mG{v=ns-4|Z?<+WMS;IK;)pPwo?_>498W*DOYxlXTySlDr z>7MPIm)=!3Ub|aY?XIiZJZ}F~ED&_tS~rfHJ%%l6{J5Sw7mTClDz4|PE3S`j%k}cB7|T{v|H^tdTo>K;>g(gT zy()Bl{HhwSkFNUq_yc~aal42q?y8{qZgXk<)>U0w%T{#NwqRb_iaDbUbH%b%cW-N2c(aPf z{4LE*LI19XMT>51Sg?4Aeb zF7z+)FZOQ=b~T7UD+0|O4StZDa!UQ;-+a(7@yiC^Q?RQc*xJw>uMH9-nzAO)-m$0Y z?z@{CGxv;|uN zof@BIfzH4Ro>j4J({g{AzqT#puWxE0#(w%}lqL!Mr3D}H-t&o1pe^%D=JOV*#gXOi z4R}^Q)iT9{-_inxJ}MQe8(in$vrq>=N(&%Ln>;u8K5hMs{r_=(*7bd)xtv(-LhyXS@)jlcU*&tu7-OZ|NM7c#$?_4nD1>dB6kV$YOh`6BH;$42GV)2D^QfPa z8lN|eA5>C{43x?{Cg!*jbH4=@ci}ZxIt27%WzNh zH8Ou0$1=8A@;N2*Rll55?Ungzoc>I4viI zl=->Hmrdq5J-U4TQ?z3;pX!yO@*MTNgX-1iZ-O2u5q~8c!DQ!@%zPSU{!|m6b{!w> zcRx_RSC=mz{cyg=boqMv?~Re~LjO`CIrZ&bDD%y7E|K}>_MVp=WNz<(%s2CSTjrbF z`<{-EUj8X^ToHeLdza|)^>g$3G4lD3Bgpp3M>f3uK50zEU;i}plrCRSKcLIk)8ma$ zo`{d0KBUXn)BjADuc!A&V=~Lbzl;(!kUsU^yGYW{Hs!CEMsL>toig97{~DQ3^@_+r zw5M7Q5*kN2B8&fRmifeAs)Xma%Y5^A?3MY%Uq2rEb@}|js;uK}UB3P)vPS~MM=#G! zQn}`Q)fPMGX(*_=X`Z5_@_ih_^H6zJ`RC>MdDN7z{Qb+OeC4N8Q6DA6NBQGYQ@--Y zYTsMYpO@`#H_%)H?=l(GJaRz6liX!>-1yY(h%(QcXZX&HFR{al3M)vPqA$6Y7xRV zHFQAw@fJ>UQPI@8qwPAVcQ*t(n%Y{W8k%Z@&`DK8r%(keX>IFlD5+X|$J~z2K;14u zwF0`@*w9(t*%@rA?doixTtSb5hTQtVvWDGFbq$7mq1lE;|BhgwrQte6+=2Q!A)~*Q z=vS}ayk?nD?V7gwu4Yt!ohwS-E6r_zdVgn|zn%*w8U*o#oB|YW(YFw5rx<~@R$AU# z*H+)Ksj0c4{y#4RE3ChcrU!yMJNzA3^s$nT&R}Pt7T4|;ehs$OV^zmW_+MefqW4y# zg|y(iZkhh}SK8Na0e^$r2qmJYOX@m0=P!oM&}KphHcGlIfQ zZ-!q)#U#JE{CQb^wc3Xf>_+KtE(%k5bDHxrbB=(`wD-&M=JuD$^0ZFXC$s$b0w1sc nn6#fOvM@D+LQz#-_1{yd*jzUias8bBGI}F{ClVvxd7=LccmW@c literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke6b56a141d6efee652214e6d50494ac2.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke6b56a141d6efee652214e6d50494ac2.o new file mode 100644 index 0000000000000000000000000000000000000000..c9b485e74603daabd15248eb82444d49fcd7c9d0 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^Kp1A?nbr#-8ITZ&*Jf69PXNPB`_SBrUP3{}A zO{Bh#^rW-G|D${QhtW>gY;VF~(@_rg!je>eP4>-q5UQ}2dTWPa-si&VFOr5_ZG^gb|;S2MQhxfhr)9_QyH^a>*E`|G6z6ZL(Q0U;n!R;pw4DOqM zJY0GDM7X)_bhz)po8gg%E`=}bxCENTp$g~0!3*=B7~HoS{p|lvcup(EKlNs~;{2uX z_V;MKKMsZFW4zS|(a(wS+}3Z0M}V(9e=0oZ(#d;zcK z_udT8J$W)*;e0QAVg7m0EDOzPePXZ=x~MpR9DSb%hfbUdS2|CJL-Q|%pIS{kFAarG z92lH?@`=F^c)l?I+u?ol-vnRK17z>RD|yZrLKV>Eg@-@`y`?06I~;oV&2Y-rli@k% zFM_X&;c+t`35^hbIe2{(JUnq8?fwMq_77G7wlDEyc;rFAwCpQCN&ITf$iz*sHvr#z1-^OD_wH=p>$!pN9Ua5>j#_+=d{X(}fW`lmzuw3PukHF_C{i-{t>Fjs z!x#Rk_^R8^(k}k0`0B_@W&OWi%lcC>KhDl#SBHP7Uws{ZJ(SI^o-LVlc&I@;eD-hk zYs33lKg}bi4Lq^}_{g)YKl0D4|I#b^)ptr*|GE3v)qh#U`p>7as~3{k)fCVcVXh7D zP3Rx~C)PjwfOh!ef7h>Ge1{z#{+oo*@VD6Eiw#&)C$RpJA58`S%WaOo!5Yu|%%H?35t8*}Zvc~#p-T1ibharOf1~F^Q z*ax0SS51kyqW#BJqxEI%kKHVNt|vs}6a5U$)6d69i9g1*wCPtquF|-VxiURhQ|5U> zsf&w3&My>)$TnmXvJ>_94)l5s`g0+E29iDfJJA0(ab1H~CGLB0J%H;8y#5#NPviP; zG(PP0&a(+28|gWt~dh2nguHCJo&aZVHO>K=qwxF%O)3>1Q zu?3wwn%lOv$LbATvi{hD=uJa=tKZwn7IgZ8oeP9$bq!seE#8ifI&VwMHg9K>FHqOu z?dYtN*iK(tM|;5E8E9Tqu&veC>TPM>)|S7dJ-_bX7lQ%n@(T;<3JMn2<>%$)Ey68T z@*d7#vb3%)4{{2*?gBA7n!EvDfY;x|C!Igg-r3&S?MHLMwYPP2dfPhl?gIC1@cLVQfrh)n z%rFg?#cE3qluIW!pb>Bv*ieaM_E`?SYo@TWEf?r|29`m^xE8B+9U_mAnaGVV_tKDi zX~>mMe-LkbTTMd1?6zAow2(2D-MfZK3>VJ#7q zTnVImD&TuJUDHzc^z_VTj>w*#*$#fo7{TXRJCSw8v;FG;h_?1KbO=IKypG^S_0{Vu zN-K)oCD{Mt97Pm5Bq7ozcx@$+zcUD5Ea__Wy6JSGxy9Sr+}`HyY;SMrnCHgK_IW#e z?)*gy7SD69r^|xEWvsZXxuwzF(cRJMYjwAFb#%HLeOsH`eC~BsRV&xltuL!tTT!*T zu59DFvel(!rEGqcdy#v-H_*`J?r6n%z{6fQh9+L=dSa{vRo&emw(80(P^5{NY z+~bznGh+8QbDU#qN@ez~cGv}NAGV9W@1xNfJP;>Cm(!VD*6n2J%SF+;jYc=rb>=qJ z>8#3#r#`7}L$|i65vH%iF;ksQv(HK8HYK}KvCqc%Fp4V~w#vY5C3u^b%KYfw1xKvL z%kBaV42D1cvQHOYX3JW~>?saL-rZxv{Spb^fupTfytV;>LgUc&V|l$MuOH!BN(dD4 zi_x=T;*%uD$&T+#jPL`v6j0PV*#bJGQw8*?0hUUG`>QY8 z*e|kK?|yrPT?T-dZgjFe`x5PZ+)df3?6{l#s9VUuo&+BK^#(DKB#6c@>(<#8;zjSV z34ZQF1!X2)=bY?tz-~ti78y90#)cNM+@S3~_CMybh<-4My?%(z3MSZoGv$pPygrm| zFAU<~4|v815BY)q`klo`XhR-zl=_e_(5GK`TW&k8}XYI zoCGU!`z9#7pbs;`y8_Nh#Bb`^ND>=NV}*Z?{!!=hzQOx*rlIDqM}O%5$$czB_OLr( z&!H-I|1P4X0{Jebv9ENGk8pDkKGzBT=;PPd*4A=)-^hl%lXdc?hy(gT|GB}*lgZ!4 zbN!1tJM)sR^i!Vob!lGM6aB*;PVVGn5AahMU=t_t@kPCNS3-n+008*_{4XP5i;zb~ z772V=x5h3L1Df!U6#+ess!aQs^WzE#U$^}l`au7Bz{Z?}Cnb=5gERT?kpb{Olx9yQ zf5t-T-va#rpApQRN$V#3i?`1W!W_f3+GLV%Fq@S>e*{=682tKbPYdvM0LPTvBb~{aAE{i$HG& zJ=Qj&Kj!3kJL&>o0=ye|w+Zh?UEs@rp9XxMzz>V|(@+=quL7S5yhq??r}6$XQ5X1R zZBMrgcr2ncev`nvP#5?FUC6UV;E&6^Fh} z%;#J_Uui!7B8?S4uL*MUU*OB1|Cwwq>ZAKJ)5BkL$)4O_F&_EX79x@JAtN|*CdEJQ zk3XmWSD5z9rbm-}s_%dLjG^>wVO}$bY6M(0J3i{~RKFPLKN6kW(uR zOiPIRE7pUV!D-WoJ~8HR(=w^f{cSJWyUVAl@Zf-eVaw!G`|h{=JPd!>pMg05B{GK)pvb1)(e+K-!4~7-&Le9 zS8RUs`GNLQ-{?Oh2yH=t$Uv`He`$Vj14?~mndPPaqU*s>5i8G{ZIW*+ef91_K5!OI zX#L9h6Uc8f_#4UZu=tyu$4@)Se~Q*KyT~7CeW<7PQnXj`ZD!EH=^ZhDb8vl-zn#PT^Jlcp6t)ZN4zLc&gC@~4|;cD=~wZ~Px9$8 z`Es0ym*TV1`#8U8dLhLhzYQj%^_%98&f}#{czn6x&lP_!gAVcYB~_#8(Dxq z*v)j>^&}Qe7!Uc`X!#{=Y<-gX0rEG@pZUa(DySdd3Hv<`0ObSlABw*(lS5Mb4@mx_ zvG@*sLLN(e*A4MKJ5D@D4ooJI`qcxOv3NbSP+JkOQ{*p@^7xMY2mXwYj>(^U zl4yM`PgVj5dz$@^^lvl9XZ!J;$TyUH^_1^8oWHvE;Y5+|ko{63>>LK<{!8|`v46t9 zdA{h7`6bE2i^@L&HV0py|C`F6l5KBeIaA|dJ#%8KAo6Lxo+aBT+Hn7XyoBfLU51~L zyoJ5M^8tkar0=#R^ZeMs^J9B%(2+d*A0ydL@+Y*1{CqqR@Pq{FC}i!EjN$JtH^{bt9gpuMwYsx)Gm_E zv%6$G8PmV37}jgOL=Sv!Fxe6HC*-&30e6-gy+??i{Ass?ua6in zjp*S|nMQxh4Q9EG?F)Ci-8A1gJ@-eO1J)7#i1Dd?)E_@gW6y*+l10u&(ulvpUeJs7 zNtEv(pSBG=0bcp~mgYi!=8pCU`O^c_^(ml_xG2i=`~=ivc*x8BmFolYQr!g(gnn$8&$(md<9u?7_;R7j9M4`v z{Bq7xCnas8KPj7sIIXcrcypm--y4BA%1J6R{mJ#`DBP z>R-k4grJyBtOlbXf7sK_&;W(bN3j1o)QzQ96rJ|x*dL2SOxhp5!Tp`I6Y&?z^(=X@ zBp3Xp2lP}rF^Tk${SRr&qxpis|1`**f${O-U%lW*JO02&yc1qeV4NQUypH*@+kvA3 z6dTryQy71@u!q0?{c-G<6mMVS=X**&w=56#>wNtnf8*=(8IwJ&`I#vnf&C5y7!nj| zKkIqMW$@Q^+(GdW{Ed_QF;l)n`HY_gru>8Q_vB}ou#nK#?c_6gpK3ni=iDp(@_Z)h zf3bXqpHG}iv)?d(L43Pz{_FS4*+V>uD=9{Mw_=UAUj>o@9ISg)h>Xpi+Mw%;1k%$mtN*2#o z-0|}j$dl!!_#Vwyf`)u!biRUk>duPhE5p#=C(lQ+Xnp1J68%~75yVSVKH|W5Gu+_! z6XYXVV!@~QJSsoBbMq1MpZ_oOk$I;0HG2O|*#(N+e;)`kJ3)BLjv9p^~{2c>>sKkR2F)8{dK zeLjcJ8+T2N?av?N`*VJN%Jc0v7J`1VVLyHdpEuYi^ZhvYkN-&f@z{9}#>4(8c7BER zfAoHQ>a-ijw<+A;r^b(Oh#ym?^8J_EkB^RTrt_cN;M6IW_!c|=vF>j%e{W*HJ;fN` z3WF1Lh7YjB{*dNJ=7fo`kl24`WB;8sfj*xz?7yece$f!`oU|c{?Z2JQG4|g$P@!@= z@viDN_TTAPEy1AJucEyR_2_;S`Pu~Ic@yv%#`7rh|4hm^#r~7ehs@~ylkdODpHQOl zZt`U2MxFB2^9!L*mu}D}?4Pb%^l93UjGe#!0q6Ur1atn1{I-1Gt;ILwM~3eQq_I_W z*u;w=pQR;1?x*uRS%l&tf5ZBN&x`qauQ<;zoQEF;(M_C(AH#V#D0>p$$FWLNnq_~T zrenX)z6JvB&)9$7kUxAL4u25m>sj>qDgQiDrw_~%sSbWuS@zp}zZR9x$w)qazDPof z_0>)e!rKq0Q~&gFi|kK}o&V?EoV@1qgEaO<@FVuK=m6(G<=r2}@)LZ1V9HNEgF2+T zSMF)xo(Aq|;GPEVY2Z`W03Dtq)DGZV4y@-F*M9%uk0M`vsjvFCoB!?~eqK55uNyzQ z@Hde|g=}`;Q@yTQci-HcV|{&$_1kNo9eMws>mpy{sFXDcf7O@M+t-)E&iwe5-~TT1 zww;yLu9Y1~9N@1IKnbdiv!`6)5xR&6 z>R4UF|15*|an6+AVSCHiUTqlyxmM3WA1iu}Gf`tT0N6B-NW7y+5AL*aiD*w5fTqpk z{i*R@O|YY0oLZbnucB0sT`S)_S?~gCv`}yw;|ePVq-fL_)ztGt#1kLm%Bx_6`l6R; z$9WrPqx^4)>0a3&9(`0EKVxe&Em?=Aqc9F3dr}cS8+4lF zvG@b0;>XuGh|IX%40_j-gx;fa(@1gfQMq05@lm2Ac?=s9TS+@@POkE?TCa26dEofvH|bX`>5mJ)Bq3OyC0 zjmE5_RRe0!<0w3FLpoTa>#H8b+;eje>ZQ$Duja7))gbh)Acl{`i;K=#oF)Htung|0E)IVP$(EmnpqD>$^{tfHo5x3grlhPNZmfCN%k9civ zhW9gkMICFjo9St+KCJk7eSUV_Sh*XkZk&m|G6r!k6K+L*enY>C*IrE$np$JFH_b~u zDLOCRf&pv$c-pqoeBSFuo#}C6kd^;<^ePXKI`$(iHLqRf`jfybvxv3JW?8%Vw}lOB zmzn-;OZ%nBnvMN=EcI$j9og#j*Dm8~FMMyT_;#acK#n85e+@m z;i=A*sf|lThDSR9n>GV%tkLuw7hWXwt zqvoi3Y@cD%X_TVWb@n3HV<|T`8s#Em=_X#TLY&@8F5m{c1|91NWq`yAX@qnzTaQlIKxBM(c|a3?nn^$Zn7r@b_Xp^Z_3;t+*s{7R^2$m z?!|DihK!}NPl`9{{9X(-) zxF!qzLE;jP9E(l1gd20eH{sqs^ba#v?o|DhPgy;%F%MXf`{!Xi#^b=?0v&s{XUTyAZ#RdHxy}#uA zo6qt-Z)ba-cm0~?lX1r;Vtp*??H#pa^T2~*F?%mFMm;ra-_rHfrPymTe)g?hf%nm2 zICk5rj5z}Tti&Te!=fWu%Q?F6aT9((d_PWX|6~1+J{ji{82Z;W-hYqcjlOqg>|f#) zn*W>I8eOY!;e3JSlPG8GIIxNH3?uD^&oEl7np>_tD+B-^*+8JPIPs7VGO8|(Qr^t- zf8svdXS16<^+L{z-+f_XEBoHu-~4u??XPFeo00dE=|4V{mRix0vJgXl4xay04XNGu zr@Fjc*rO6W;rTx=FK28^IsUgCuj@VkhZID%pliHm`w;w>^oC-?Nl^5wb*L-tJ_0yD znCDd2f1AVm;73CyqGcRDx(D%`F7DPQdhwjD3=@nhPhv{hn0V5Q5ha+=n*Z#+{-Ipw? zwYJAULt4(bM87NOpEOY%vH`7kG?rpVX^8A$?UUM7%L(5^6+^Mol3&Qr#Kn>-X}>Sq zA2+n6TdUs5f2h63!bYPsxf?htnYj(16il>>ANP@_T=~IVif-aa*DVO?devvS9UyvjtS-=q@kR( zOffaSYNzb~7#iQGy`rP`L}zVJ^PE&*aZ|>~F;RMQ#Fg3;O^>KX-|;U4yw8#-MBXfw zR~l3n{*SJPI)T3nuOptQe^Zg|Jtm;V$$+Evh4?SF(BTpOy!_12aXkC^;h6aUI@Ct7=c++il0AxSXGs;iB_Z&0FJm#H*SY)^@7ykq%;P$3b#FDLQpk z^GjV-yKO~l*LM_^6^T-^s<@z~t6@=b%cezZYquA*tn+SY?Ao-kCcmL|L+y^jHSOM_ zl2s+^i<=r+mv?U7SkuzrU%RfQV(0RPf+oP$))Kz7W!aid8!MaYH>}Uwys>hncSGTh zmddKywL8m~w=93Ow#i?!wq->@?V}YtD>fF?v}|r&o?pL0v@LE~)>6N*IM274_q%dq z!Sb&9)@6C%!Czmnq;`Ap>g|miYdSVbdU*46oq<+WXnD_VA9j5hF5zP@~YS3|+F zZW^y~LuGq$#m>_DMeFlg%C}WU$1UHs+FP`&y{vrOI!RN$scn7h#zpJ5-xOUz{#bO& zH@60xHZQ7zK3dC3=B=Aaclqt51>4qmi&y7wYTZ`0y{M*p(}u!8+%YS5ZrlXE>&urv z-q2mx40&6itH=&z{pOnX zl1<*)T7S*5mXfCS(hb|Hy~QP~mY1*2-&|hSwW+nGYjZhXi&k}QUQy|9Y+c_C{Z*{V zU;lXHiprKv8wz0Wn*2>|YipY}3wv!ZuH1^g^BOl+c3JJ>E%86Tsd@1u%HPzruC=8( zfFDO$QeU{lo4+`}afxrM&$pzopdf#-Z%Jce-r{A8y$uC*g07)?tGB^-y`NZF$A4Re zepLm(c5*wFyc;~5Jb%&R#ful^$I$>Uh?SP; zT5f_)b#J}Ch_-h2zB-aHq2+_)OfVsbQg#eQENJ4Z2LjkAF12=o^T@*iDn3vp-t(T< z@vhM*PKo!NX+h5;;;0rj!?+45=wtkt;c_RWfVc{J!}?04ILVa62%j=9P9G z{+~zUMfBCu1g}XsXv}!yz9HpxOB1O1)eL&-=dy&WwgnQdHmRku{~Xy*wKSR1^WRJU z@pLI()Vz9D^7*oatNHYr>}Nn4P{B60Iu2+=!+6Q6>#I0akl|(OcjqwIE{eG znfUzBjGyrnYasB!b;xGN2C zlv28*xJE_6J-o(Ple{XpiW|>a;3_^0Sl}uS3|ru8=~d^Ss(-cK&jx=~6ud{;Z-oV} z)_cDNuGV`d((aDB`sOxopqpWj(CA(1?P_aidX)d|CwqOOg89&hE!^{Yyoy)C|m&QR2qd@g{8M4Q%VE>+t&;M*ps8puN%C>17M}yKqQhrInDRl%oX6 zf`*RH{3SQ7jXPv7eDlVQ`-7>svw=6Y`Q*EKF&{hBZ@1o|zw9a&*iZPr7ePd(TBe?^ z$OLt&%s&)TF4L;YXrE6onHEIh9(i|0#xaFZ@k-Ta;f7K!j{?Z6f2X*4PUd6u*&4+= z8IGel*xv(4;~R5pYUz@}*FzOUQRA!Yv$8$)Z7kI)csClMvGEm@(8QBfht+L@Lh-BU zt@Xc<<6p1+M;3(QTlJR?;jK0y+{%EWFkh{-+hzMv^2_$mSQ=RQ_W>Wzeh%qBm*wze iKXL`tUggmHF|c)PYI3%N-toiZQhs8Cp5$e1|Nj70#Tl>w literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke8996f833d1f91e81cf2644d8fc15fb7.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/ke8996f833d1f91e81cf2644d8fc15fb7.o new file mode 100644 index 0000000000000000000000000000000000000000..6c3100e1bead2c2694ac2c75869a3e3e321c1bb4 GIT binary patch literal 51032 zcmeIb3v^V+l`dS}ebj2T8mSRNyaZ|qfg~P655&X9xFrD+n1?Jd5?m*()(gq#^|vHA z!pe2KaU4I05pw=KXFOcTNz?=!PbN1(nIu-?EZgHa4q?rm@y*;dA=q&Q@q>{aTO!8& zf4fed)8`zu5V@H@YvM(*nsdI|yLRo`wX14Zozv?3HgDfzwOE*%EbMzMiqu@jYJVEN z_wrjWbAi*tg4H87-@jydZ2yZsC_9kJ*olj_j8m`S^SUi#*>7=u1E1gF9z|1CUbI=5C zQx|QiBRhrm{7-K`dj6g_ z9zE|`xhC9t=JeR98Q&V)`pVD8F8}`JvGB3C#!j7n3w$Nv@MDjizvs-c^AE2)J+|$Y zGh?kCuZ*2K_Vckz-+XKA^1(O3vpW3nV~?Cawer~cFzUYiv(sa(sPpoUSH`xYu2Un44>AWr=kBh z$MV`<8N287D`Vj|e?RufW5lyQeCg3+=TCv>;rwrnE!*<-v4p2z9&=W_Jk|=GFCTmZ zJnO<_t8Gtz_WZ-Sr^l8ZLtXhVk8R8S+1NjgzBTs9snM~7yba+?cRYB0*_qFtzjO!c z%K!S4&TH-uz}D;r@)X3HD%h!jCg#>Q}Fgl{nC67HhD& z9zHhP;+?o)n|+FCbyilQ`2%Q*Kg@=|DLzqWVI?oZCoC4C%|3O}>bQf-on)-!$H>d# zWfWP@!f)Q&RUUr7rVM`Qow#=IPlxpW9Lqo7vssp@K1%+m`l1)U_&xaQO<(+Y`C{-o zzIbvPUp#5@Me6hw^jzt%!|PeZBq@3RNU%EmuilBX{}3Gh!9njtp2U*hv)NDm z!6sqsuv14C*(bhxt9PPqzWrKjn*CbK^S0rkUwbE7zipWaf3D8m@N2*U;BVsEjO%XX=Oh0sz}3Ka051i86nG7AIdB^Ay=Bnb zHVl0xTCM`G#r1t$i*X&pbrr5}+`rK@v=*Ld zx>#_c`6mS@4m??Kq73}+r&9h@vdNCuI()ZoC ziMoFe-Nq`1q#>;Ro(m_JuunYu+q!SZT_YXqtjkX{{$%BerYBdPXb!DBaiD(XiA?CH z>bRD4B^{w3>G_&iJGh?Tg$~~)9b9MpKf?X@=|1s{|9Ra1gzlYZ{IB5tCAv@GdX6Rq zQ;zJ9n~*;lH_^H??i%UE&YfIxWH0orR&~mSPRpRvGU&7nIxW*|M!Jx#NEfmn#yVfi zxIPb}t`ODbIOBf|_m9wh{2Bk}aX(1+_A~w`aQ|hxkK_8ZED8?xy)R9)zAZr)j6H6n z3rp($J8breXFpvxCJ8$&flfKlDF-^`K&KpoonAvMrMHvTmwQoPHPt7b@qZHc`{~|x z#@~$ldb+os@ps_<9=f+EeTnrt!TqJ`J7D{l@M(QSy4F7^O)$&IZF%fupyi_b#KEG3 z;d^e8h7XtFGbjzWJ|;~(4BnQ{qb(t6xUMX8o7elSRJX-?&6mLbA3tMzE+KYIK-}0G z+BAH-$jd_+iSno($lQvnGa*==n|bOhi551Tk-(I7_+I+#{Eu^e&KCCv32y&CjX zL0=k!-VFL&LC>E|UuVfUOY*j*vk6V+qSA?N9+WZHm4*E582NiqM=sCjvNSmXM!AnbzhBUg#Gt2@qx?*z{DU#*`Jmq~ z=z}rnJE?q%QvS&p^rxx3pnp3C{dp=sODX^FG3bX%C!F_#ZuXlH@-3;8^B*Jr82Mi& z{@IiHpCJAi`KO3KZ8HDU#2+L7yTm_dGXM99KSutKh=1;6{+|$kjQr<`f8J#Nmxw<` z{wu^ke=`4h;*XL4E8<@;ng0#qkCA_w_|qrzzfJrx^2dmO;bi{zi9bgERpMVXng65m zVdwok-!LyY;UBc$tFvaG$hBnrz=HiAtwqmbU3+iW7R0ri5#MeKUp-KcxVJ2P?cUPx zrNu9_>Gx`Mo)9@|Lu~8-PXc(ID3ge?F2uw$!j};HjUq<<-F%lcQ7X?SE+f|a{j9R#kN%BKyoosQt;J=-*S=~Qej734JLP4=AHI_m z{!QYxyw@Jz9KQMjgPvaK@12-93YqtLhp!&k79Rg28~(6!E9_Dk9{q+bJX%~fe92Ze z{K2zqV%&E}`2C}7;t$=H;rDB{hu=%I48Qw5Hu2uB9pQJ%t*FO>I<3QRCR?C~b@+0; zWn%16($O+Jx)$r`cbgj}3cy@(;Z{cBA~+-Z0fq{FxRnk9S*~vFo^Y@Au(* z7Wf|aW>`bVeKz=?1pkW*M{G&IbU>DKMQPZzx-6Wup`6S8?=N{L+>rh6Ht+B=M;DD- zN=x$`30~}<7LMm8ATNGH-cWhQ*DE}L` zgj-PNpYRzfpMUlt$~!u5{2`9#ogLzM?)VVLbI-oW@tpBdv}-6QKhKf9DeOdEF6A#@ z6Lw^84m&_wPWZXmXC2vFINia^r*fT6eQR}i=;$o2>rnZuvk9Jx@G|h{fhSy^ayH>* z@5HI2DdQfC_lF6{r?TOrGslO@XF^^jmv`!@d)$e9)Qi1}2YPJP>KQ$3rTmG@x5IbV zS|)PAH+sN2?8x56bsKF^=%cFyt%hiKM9Wu*@;N9EU+3l563=#xX9MKlN^)~3e@FN- z{B%C@<{@t#u4{31WbX{8;l2bPNA|98Dy}*FzA9|TbqT+}Gc4h{nBP~2t+>Kxz`r|8 z{!@fc%hEmJnYb44`@6!)xaRZwz2O+f;JUU{?|`GX#qyIqo&hgP2!qsO-ze>1zd~_G#-Dfec z0Ph|jtsuPT?C4R#d&Wm^2e#ccK8pE$=()SjqMqR)+urffV&J2=t|EHP+0os|FUR!` zDpxZ;x)}N8wP#TdxORMW0pWdTN0$@cH$FOx@cy&N2i`wEnn-f~+xTcauFw6qv!ga; zAKY?YnHMnDajv)-V}BFYpmNLyWtbOAF+V?Vn3HbK7dPjNoAbrZ`C_{H0`U<~b@(4~ z5dYABRd3z(eG7B5E5|sQS((K`4^U)*DYp_I2bd4y>cuA>pP9@}VkV`7O@WT4wr*Ah zN;Xlgh{UWeT&-4Sn_;t9Z5AdaB2*X2mc(!l!|qd_BbKTJEBle6qhPChRJpnEfO54S zWOgrG$Ks&TeB#XSZt3m`^!V#>r(XNo70j=`?v{>5ndNtM_B7>p9Ln!G*xJ$D8O05~ z0w2oPZW=n}l%g?hNekcXc0b zJ#wV2NyEDp++A3-uHIMdYwYRkYHHv@gs^&lcayKNyA6cajsthKcJ!?C^|bqa&7FaE ze~%_-lfTEmg=dxR+q2nMuxVBkbON32bwwxN_+=;{ei}SzqQTRgSF1r(AnmzZ$k9yYiz>$*V)k2=tHRO z3pDlgVoT8ItMBvie5My(cw@qQp$J6!dmslD^|p6(n=88+I{la1aTumr3T(|8Hy%3u z_3ypZg=KepQ+s}Ab8~l7PXY7!Iu2okDB=I^9h-Of6bc5Nzpd5Z9fLs~eqH`RPis$W zX9tf;G#MlvaoNWz*1X<|n+Ku|rvT=(p6A=m+3XWMf5xQzzqgW{WBs0}>dY0LzwMShA~r_W}EB@L=);?}D`P2wu#drl^6 z`GU-hVv%qfDDEi&nz657*G*JRiI!C)?PMD7pfI`}PP zl)u6(L}1ls(mG@)Wu@tm&+MDhr_a+d$(@dTIwn940OSlltk`A=V)v2X{M~M9vS1k z9P6xyPMuoY{4}^dJ)NCx-77qpJ)3YwseWEYg?nIyRW;asom4w z+uh@7Y-(=pX!7jYwQKus-`>sDySMJz>DyepXYVqQyFA67mHt3Oi>JHYS6Hyt z??Kh1E1kRH8H7pKgX}@%ORF3V?@w``lV&;?T!$yRhx=;x#6w9+QTXZpMvniaW(mHC za%&}}zB!~3V@2#e8@n=-NdgzLD+XLZ8ehrO=&RTT1I}l!8t_VX&VX027Y%qh8!_Nz z?5qLjv1bf8mkk+k4m)YUOW79p24~d*v0A$IFVHuu#;5+NA*cyUf`&|IoLV_ zj%Ni1emnC3N9`BK+y;!f*nn*;Xkk|@R@L7zR~WE`1(}}C2MO`-lK7akl%fNwQ0Puv z7ettKyN%g=Hj7l~VpX6G;4^}cl;`2|Ai2t&lT7! zaHYUi0(%6m6Szv?E`f6e4hmc=@PNQ|0-qGPMc`)y?h<$e)05g>C)!(=q+n_v$5`Wt zPVM11C8|HisR|~2IZjhB=|{M0u7XJ)JFpjaCwYV|RG+}4pWR|-b+;;X>H@n(V$YNb zEaY7fSk=#6uFzHeyaJPbdHn*b`kfS5)z7vrnMvs$_6+$1Y!SrAgP_5SPZvIJFJmWB zr{o5KTmpL0M_L4DLbA;pw@FS1gV`|S;eL(CKZr*~?f4uZZs>?a*N?>Ynz(+5>uQCC zp=Vg~z?ydpv-os|X$dwv&tDWIYT2su|%RDlUV}Uu&e4yB0HDNDti0Hw*rHS>~b!ip z|BKY#I4P0JQ+s&(B#GLuV$_FGOo{5l%V2#Bqt<}o+)?~|w}SZpT}kZQy%7gSA_gqv z*LxIPpkk(A8jzeWFinVhOaoMpy(UZpR!^@qVH&`CdX)*&fY#G#Le%3r6D9}J)4K$2 zWbE0#ND?DVy?AjI`o9OPl$8EoCG(+7*hcoA+sBE@fAFV@hqsFi(BUuSAF4ZWe~=~S zRg1u13wz@DaQXg&MUQdSq$EDEl;Qo*C@mjC4MluKbGdqb{`^E4{CVexY zZ>m+(_f4u_GSvSG=!<3|LBIDEd3^XyDep}u%~z4g(L@G$^8!{HZscF2pJQG?qT!@q z=%0;;?jFMIv3Sz|SYIS!g~y>#B89nxC9Df7ey9(u@-r5P18Tl*W0jgc$$lBkZ12Jz z#!aQzYZY&w7xnRk`nRM0pn&WRvY!;S-}z+rPy*`1Mhk6rus3XcJO|rRfa=d$Wb9AK z8<1TN_QmLYgZY7vuP)DfiR?Vt+u_vgjsAW?(;xCszM_v?7ZoV}VDJ~B_qvSbKcy6)C{{jP?x5ky&Jq$Vf1mJx@4Z3T6;i=%1185vD8&E~1C}_G#4PLvM+J=@FX z@_k~E^qa{s?@^B9IHvU{vUkL)*gw>t4aIok_NMtkOBZ*ce)5(p z$|BAEf3}0BcP=06FP|T&LPqVS@s+%Sm?I8Vev<02iv0c=N?arOLvoHuzvN`HZ&bgf z2K}%;>ixkoAg3ab)c5BcOe{R{`Ud1QXAn@e$ACQ7!M?1sPyakNTgM;jpU>krWQp<% zlDYqC{~XNaiPjhD&twY?`evE*U7TzMUDYon7aHWFez!|kKl%?25beJKd5MFTU#)z9fk~fi z2OH7LFE+>z^%t4=^BlAstMaJ*h-gInE%4V|^k0-7>R)cM-!i9uJd*x%wemLdzomRb zqWV*Rj;Z`or+z-5`qQ=Yai|~u^MqP|2JAnZhnmn(_-mTUzH^dky3)1B?a|~}NS}O0 z2Gsm#o`b>_g`eV8_#YfXEg$M%VA6NKQ$IhEzS)|5T7TegC)M&8pZyEjJXEIE7wTVR z(l;ZSimUaJ{i60yCVdMSDX8fSe|u7uPxi=Uc_dw<~C4yU@MVA|%j5@(kwn(hR2HLD?BMav$m!>*Js-Nf8~U z^)bSkp%2Or%FFB&Fh=>EIE+XCeJJK%IgvTMN&({kLck?afUox7fV_MetxqbZ`i$lA zCZbNiFUUzQ{rrOYm`^9s@+r$9En**6>NmtUFD0|z^)b7o=mY&N9J?yHei}}yGGceF z0jDTW8{7$0p7#&UU(`R1N$8Kc(f+u+YEryt&3xkQ0~Ddakyg)@!}JRDXyw`GchS!%PEzNS-OzXeCJ@C({f4!*mXOdzv z^q-CQr@MulP{S!*0wbR1{jZEsmF|l6r_C7uAl@?fSE)KbP=8&%UFm;Xkg5N%|EcJG z%7GOR3F9%we!(p7hWbAk?O)#iy7id)e=chKOZLBUe)uoY|F92|Hh$d?8F9*hz?k28 z|9{ay*Y&?C{++h}GnD?P^;YZug8%&fABpxa?|&^`p!GO$89k@_OZNYNx$%F}(4Thq zkP)XmBe2^4XASf}zyB$I(fa=fj+I>ef8zM3`B01hG5?>`^93)|QTxxZ9=sPT9>4*=SKt4L543n$ znGdpc@+SL(lgEQjEgn$fMKwMM$GLF^aTBQ2+0dg^78GqV2#RCqFJ>(#p4Am)S}`+c~OFK z{A3v9!9E_NJu(gU9*|er>mW?459bx)2^q#4jrT0rLf`(y2KgADv-o)MB56DG!SyvwCO}hO8ov*An$;-_(`bVBY9{LyC zP5tyab+*BOV4s5Qq}7^z2ILfbP^Vw0f3ZoQG}HK;W6+2D5B>s$+J6J`BKt!+ zc{IND{X^qZZ?8c)-8dd_9-W2qNa%mE&k}<^+~1Bx>qC1Am%7$y?J@fs=ta5Cb%Jh# zzh~G7AUvwC$NohfCjaE;^V~n<_<4}oKlJk#e zK>v*SuBDni2W4abgYHRp=4$pd_b>N{)6xDJkhAR1X!X;4v&eo{heQ1<7{yCSe7u|c z&)gqm&nSNe+GF%5vS*aO3Us4?asL^J>ZkOd+Fus%8_!$FUtWZAs=t`$3+f+_M&CvK zpE);4wI7Zs?qgZ=U8=p}Am7-3pf5^yuF(7k=l%9`nmn>sihV@G)LvtM80}B()z1&) zFUI~g+skO5BrZ1J(`!}738slAK|5_%EH>H2n z@x|?#5glJ-PhQiv4bSgmJa4|#&En~N8y`F`jr}}80==vLN+NqX znWfYFLP(A)S8ObXyb{@A3*v1BM(-b7OqN_SvjxH5RR?~G_wHR6VT~4ge}~cgJEsr3 zBK!{m(EB`i-^l!YAx&nE;6q$LM(-a1GxuQ?<9!I3MfeL4^!^dfM^kWq%G)!7`hjN# z7y>Tvzj~bBhj7gd#3^``W5|=3Cja$hDNVM==`iF=cyAr=E2Kg`&c{+v|2QhZ`F}j^ z_tIpmaqf>!a9*Sp!z(J!tX1NgCci(b@-o zSAaRNVBmR7KqC9ildb%HD=X>4?Fsp9xF?~_Nzlh>)%1aW^ZKkHAi;+0jU=ky>A@|3 zKLq>7l0oPTEc%O;1SAKyr`ms7{aq5hFOVYJ6Gp(z+YkL9N{{h)t|YNc|gB^|J=z>D?8^$X+O)hWfSs0zcZH zsM!y6*pD{{{tN$Gh4xEUdZJ{vqJN%04*3aA(pQxaI)7Zk?F0Ww!+fUM6FB=JDUQJ* zBq|U2)gJt!;ILx{}ngZATr zF(1D+>c7tfB>uh~vRpJio|2^$vIq3#`wz4q>S2B1{l5$H6M@k|Wl}`TW zfjoWrEW8e((ldB@Of}vNt^5)!2P&P)FGhX(@=LLvXmktJkLUkhk5)bpuPiwqiSpT6`NDa6dWxKj_UOy6p0B4< zdBnTk9IgDC1$uglyqvF}yniuY5dRYJ=Mq03-z(5wAs?9NOF>_*(O1!1$BKUldLHO2 zGDR^_* z%h!i&*);=q%*E#y@~=oz1P_!WBO#@}hg`)pe{eeLpC`|Vz(6dC)-T9gB5&Z&clmq) zdWxJngT@c9591Z{#|CZwsK+05m)FxufQ0dh^{ZrXeX248DB~6L%LZ+JX|d7#vO$|) zTH)anWU6PFS^mJMe;$`0ds6R1LPxl0zP9x69FycjvjCk>4BhJb);+z~KUb@tXmoGQs zWy_3s)hZ)ixzdR9^NqNmz=%sqjJUASh>MDhc=c){Uc1(a*R0WD^w&ZU^=GudZ@ooN z$NZnI*&FlE#(-x9=0A-;Th5Ee0>g4eys$B_G9UAsjvmin-?>epyEg_3i!gub=-K>r zBteJ#H8daT=yv{oWu7K~VFrcP3&}q*zBUHZ6KVJ+5a#+VT8#CGWoq<{oGjoa0?+J^ zkEih%aU&k!{eZo& zKA7T{WK;Z-po?E<{gLPk2i_T4`E*nK;)XtY{p0oV3oj1^y~$d6k12kcX^LO6O!0?A z-*51`wes19_##D4F~l#lUUX4FDb@?pA06PGN#)@WDYEjsgU?^^FQQX?mqK)4tas6P zM9^^_=bfd|XPe@YIC=rzo2uz2>Ekc-Z)P+;;r356#9xQ7AM~L7Y)!v8(RhW|ABXpK zm^V|ae{nQk;qvApK8gAR$wP(SIU4^w&_(|Uc^Gd>Jor;;?~-Wz!sX3hV3vmpz4J7A z3qUu@%LaW3(NSL8A8hL4@x-CP(l}J$T|ji~PdswYQY~HqMtqW^#V62j9?pv>9?6OB zr{icn=KTf!5;-SMTc3Emk;Ccg{t4^-Qf<9QyizI0nv#zCC2{sTJ#OBH?= zoai=UyWNOq%rIhCf)US5(c_JQ zoZK9kGrHa_T&Sme0`c)ioSJIHZpnxa)1nMo-?;wp=Z%5cvpoi!HV5NVM^BoE@ukDb3ou?Z`Re%(|9%^n=S-ya zHAky2VTK);udke*IFpvbSsLB3kk-3J8g`{pyfB+E*MCMPCUBOf(UTU_dYq);oBAnA^cFM=YU!?fYX6`aTZcf3>p~{re8@8Qym=$0rZ;NxJu) zlTg2uKocPUejxgT-v6y5Oz$7#eAU|Tg6)uK|HHmt`;{JaM1 zo6UARkh-3}6rkmb5=P0dGT7Vr{u9;57~W&Yc`H2+Ab(Zghc)YK#LD}(^mRP-dCW1K zr((fZzMsk84`1z2{7HGgJ6pC|y`bBKKUw>2Xb2Mjeq1v9ye!4-fdWvH@2_dU&Gi%e zZ7N@&mWRJldCea1mlDu})IH>1um{dxXOKMdd0PLhff=;Druz7KaEa`)lE10z(+@wz ztRyeuLc}_1VZiu;nZOq<_`X=SQlHqr8onQ;^uJ&$-YW)wu=b}L+K=yd zrI7ag`>kl7tDi+QdqRJ#-@Gkd_3!iZJ;KzUuKIC~u^+XOebR+}iO%KK<8NHxJTP>x zHNxtULz5Kri=NkGJw<&F^ra$PhXHAls!!te(fIAEzpB+o_UNkr&_?sq1AT7r|3Kj% zN&@?pEIPliE~LJ&a{V%}zR!+Bz(xuCg;l_5cA9?`j7|@vN(lH= zEI9&m;wk<>;`<9nAVX66F9^H~{8AhORwTkyKF&_|Q*mZsc0|<2{UI}u77_IcoEj1J zaqI}VF!k^RdB<61K$@*|F!CTjGZ05Fb)KnUeJn{Nx(19o=p-&PWfNFSSk@9_n z2sMJ2pTpxT&H=o!ERLV&N9iuOrbtB3QuK||XGBDOTppfRNbW;8&!!?=eiFVw8!exP z_s@yWZ3y~p|D$%!;nkhuQjPt^V{;ZLx4JT@FiK~8^gJ;V8Vfhf<< zqxtw1{!0Dfm>cD{I0ErbT2Hm|TN%bko{V%z&MLKaLkc`Hf{* z`jFKAAbU#qa+zXJ8{1eN&!0zeI?8XXxAXZ{|Lc`yw86O-;d}0L*I{wKFa!428}T0FUNR3M~7KFAwB?|zHjIiIaK}u{r7R| z^Nrt!emw)U5|r!1AnBR&7^XwUzd%y0Oza;t*K%F@6hsY|8vYg983i)yJ$q;PRj2_A&F*_~A*!Lsfn6 zSu(;q`zw)=n%ic4__d@?&zbSteG~t2?O7hRL?V9|iTwU*B=Wsw+3zSh zzr*EPyd+2B|FH7c4_|v1@SV3KGbTQYwEga1zx!4Ovm{FR$`eIVr97C5;vX}M75XOH zomV;ierbgP|5&2kfF&^rbc*V2mRlgEG-;93>T)Js<%cXb$ue%WsDD%<2rf!GdYqoF zY-S8}o1Pxzn;ph}X|XI=gIae(mdz5hj6|qb^>e4iy5N)ctE`livf>X|h*DHD62j5z z(F>-b)7Sg2SAN|zbPBhxSKd1fox<8!#miy|fX3VzHu4__dN>+8{aUW8L{c(Ku_YBmN&pMk35> zak8MoL%Y40bm5q^buO(q#$L$J1v$9 z8eOH-5S}a*`_q)sh|P9$bfY9U_VDbE=S8+0QC$-w@wX%>Esz(Q^RZq8hZ48P_ zdb&XeJj8J_uX>onN2tDhoHvez;-;2w_u5&=ejWKil0Tx#=Ob7zKi(V9Lh)0}cX%E6 z65{pbCz5>S$pzzou*cB;1T-y_FtvQ8ebMUJw*=b#H5hK6!Yea#!SgBpw;3413b=gZ@ zE{U#DA3RE{h)rFERQ@QPuTmNvLyD)eWp=xzp%Hp$XQr&IVRe!Db3L8N7x|bq%Ykgt zandJ-e9RIFR%oFc9oMk}dFNlYMp}{nvNe(i|7B|=pN?6hxewKTy}^FGKDuJ zW{p&XT8nx$Ye5(qSQ}DPuQGQP4Hm0C2nUk9Qb=Oz$P#pV{E-+UdeAmvi%It;TH~oa zrsA0NU}A{qLFx{|#qTNw` zuPa3SjP#81N9jSAmHq-7>1j=h%8S|xwg^h0IFdoOa%em{8S&hfT&Dz$uJ%ckZzP`E zlR_z}xAB>Oclc>>E zJ=8X8_)oIiNKCs$>t<2iV(N~zsXnIDUDL%p7N*h--!O{VZq@%|+Er(_n7W~DF?5?e zbC*(N+OPug+NZ6w4vk>%Vs%?n6^-zRf5%BVRX@HAu`*Yg-I|qzZ`M^K!J++JzE*xfp_}AK+cy$hKDC>tD+(RoIo0bwqRIE5$Dm(`@_#2Y3 zejmxwPx({*KGJH%Ad~*@YFD!Oi!xFHI>B{6{i>04<)Fq(ayJM>_mphR(jWXYekZ9* z$W!aD(rTl30RHpqyCCF`i1vDQ_5C@&Ei~zOgZ6vaBY&Iq)ZhPYYCHSRvUfkIwS6sT zMP|W2EqdX2vTJLba}^?|o4+l@t$*{kg>L?~kp8!XXhqRu?j$$(+d^HfIJ(B?0zOin zhtDB2k4%;5w}tAk&=er7=)wWU1$;(6!C3GBW1beg^M>iqx-H0mTZke*{^A5?U;1sK zDuD|G?h=?*2ci!MyiVX}1g6!2=;s9X3VcOiS{;b)#^Ec~S1E9Tz_dE>`US2MxJzJK z9eDi$*9!cMz;yzj6Szg-D*|^3?7>TWYJ2IocrMVlW}pwXkFW!KdmL_z6BE_&h^Cf;y)9 zZ6Q*P{En`%emCgNdx%0WWm#APuHGF)BJj(oj}q2g^^lHZyk5?eIsY^TP)O!{ zbq%L{de)|<4R9Muw+VgJHS%R8TgzA1>P@D60oh!W95sJv3VxOTr(W|=z!-Nc64vwl z=srP}3-%sepP;NlVU_;Qhsl9CR5w1#wOVqCsxQ&?c){2l;7J_-XzvbQhsk! zd!tw*)#Axpid&~oheA?+Z&PIy-N;s;!2{a@Wu z_re5|l2_z3V7%y)a0e|b5qGi5zaYxpg*jNC%;o7_Voa5jlEZq2h zg(1u`spLbVD=JMy6XbteU#MtMnX^=eyd6ZWc%~I~_lnefRkd|WgUl-;rd>?}P&#f`yhGHg4^w&fvAv{plHIAbH z-44SEoebrNjwn1;CWTFq(RX!@#;AY%UZuS1|5QIEllfSaDDO4tzq+5ZGu3~oJWUDa z@~dH=M`F}p8B)r7l#BWsA6j|UkG7#aSZVwi+TS9|59rEM{c1VYe-|JlR{3#JUiDwn zpQa>p`4VJcHKK6QP=6PUO3B0jkN#)2|8A83bd2^7i1KRxSs{Y_-&{Y98?tHLqxdgI zQsW2ps+K1`{%^~{Wf@b)r@2lsUe$4{ua9(Pb>;#_TqhhvUyf%@QBECS>Qyawpmg`% ziUSR$N-Ez`R@BzpP+ZowuXy)e_msBn@z*r=?yIdXY-q2!>tM;9o&M7D9p!t=S{mB7 z^z5&#Zfoe;y{B#K;j+e(>dw-whj-N$>Szahy6W3^qg?g&(tR~|-PN_bqHW6^cQx<7 zv-6(PmgfBxd)xQb?(J^e#>-aJ7VYh6sH`q&+qMgJ_cZUTDe0{*u5M|pXggHjg`bBn z+F96GA-6OX@95oEytiQg?vjG0+A`G91pfVZ?XM|0QeS)r)mON${lMmXN_Q4F6#4J+ zm$qy5`b!HtHx(VY(_gkz@N8M#R&iikOIP*ow#uTr?x;Ah(_gy2b92RkJ&iTnI?J{m z-qcjNvwa$#vZnpjo#p%dXk+#Iw(^$FO*IFq{AFdGn~V3h`b&53C@XGhZY)~gx4&}N zH2Ul>@YfW!HFQkJ*VWrtQ`p>Cu|7cUj;Y_)!?k3m`id<_8v06F_toxft1YTNQeUpO z+um*U9d{ntLH$@MbgkLaS6kc|TV}<9!}pZ(GDSPLMaz^|piI&J?tL{=mRl|CS-G!v zTQmHp*AJhGWwRX}LXVD}1r6nD;Yj5k_Ux5$&Vt?hfuEzGg zeelVxcNXqF(pb5zZC_21&L{RB^w*R$`(w(C_E&jLbyt1c`htBm+q(Bp$J@?%JN-4) z2lv;knWlf0`f0qxYP;qyO8wLM-8O1>ZSnp?`----#`J-T1AW5oTR3k~(=_@ua{IvU zQ}w?(E|j{j-{)%Gl@0Cd5AWZ*{+@>E+TF43P(#JuY5S+FSmE2(v3r{K?C-0`d{$dr zF%4gv;43>+U(`1ZZQ%?t7Lw(!>5bi%O)X;n+kd#BZGE5-^K{I4(#$v2ykz9-t}mKm{?hnbs9w3TW=k*4 z|FOn`*4_@Zx47)kRKBC}6%!x%Ma?vA*{jq&g)eIIxSe~7r?GPdw{z{58{@0FQF&!k z`HI@VqTcEo;@e+&WAgS_$TudhqUFZ;DsN0b#2hyyuc7G1^lK=ZR==%$tW|93*k3!9 zuWr+ik+z1Z*6vM+n+o>B=eHE>I3nT?%u6l%m6+n5GFtEU7Btpw>#Z%?(p%q-b)^<- zMiJJEP1}y_Yh8Vt8r}KY+gby@gH7w!uV2%=uDH0duz7u9)4IZj=At#LS2wO}ZYV5i zu3zg@_!?T9{S8gNhTfhwe|NX9z1Y_-`wldgeta9B6N9?`&POx=@b|;JQQu zrwJ7L))cN@?JF!PUgInBt?}&%^foE~ZSl8tH~GM>)NWCA5Gk--nfMzygYw_8Y4~Yh`Z*ncDjkGT>T)ft)8sq9==xI9m)&1U`J2=y z(*AaCT;=?@t?8VNbo*z1r~HS%n4qM)lP*}MDi01S?NWJhzey>DnW#jxeT22u@`Tr5yEyjtV z(oVJ0q9v#)7RGmsO6F&k%d=<~ob{LK#9tWEXms-s7p)q@Kmxx`z91?5lk#oHzb;=indc0Ne5zNd%5&87 zda75SzXx`pMDkVL2$P>vGRtWZ`HM_)x^!~%Pk{z>`P$!?f!rZozMlUFQ{+3)zm!N% zeS24ne6ybGM83Jb7eoh{+dC@q&2ruo`R4Y1sFS1DKV6I~lCN*?I$gegZLXRkUlLA3 za`Z?38+7^lqrTI+d_Di`x_mwV2I1dk`8~RPJ^wjfzFxjVI1|ay^NSzPkZS2h2F7(ZV$ z<*W1ktEPN)o_3=?N~#=nJ}xljtMjpX{;l#~5bf_W@vHOkkSSlCkC_s5cl+vFJN$t@ zhPb@Zzun*4(a>@S57aly_}h3aKhV_X&+loHd)STcSV)kswWGC%@w>)We_K;Sj}QBn zra+IcySJgCsk@uy*W(AL>NVG=V)aVz zQBir4txqGW-eP~P($xA@9K;RfnQb^Sq;Qz|&G;9h{;A4uSI=P-ag*$~63UbQ=H&X0 zIY+@}-Umf_bNk(*Jnd8U$*lkVpvUUJ3qMp!T@i(;85D}D<<|5e@G`{iXTm+P9wRMFTgELKxc#xi?j_a1)h zVX2^Wv!0p(TiwsI-M0TtAGCdD24jbZY}tM9;B(HFJ@;R6J&(_CaF3=F6)jz_?fOCJ zcjMUA4>#zc52|dTYc_WH|JqpJ(GGSPv`2a}Skd>gJ^de+vbTPvv%@n|duq^Ly^)YDH7rp{j+n$z-~@P&ED!~5R*Y4|DUo8jgYm%@E3-veDyD0J}P;Pw*-2KUWB z9Ppu}NmxMwm z4h+sc`NUudJYSgq?eM<&Z-OuA0kZeulRW1Op$h2o!b6~e-cl039S*(wW;kW*$?%-> z7s1!X@VJ?eghmLz9K1eyGTfZ_8?^f~wA(*e0ocC8li`sE&xez>>?=P>{A$g}#g_uf zd$PZuoY+&7{AzZP?C7zvA_r`w={+^|@Tp0f=jvJ8q&~uJ(OFT{{=f~g|5Ls<0N;BBzIo60?rh)dxqc~sW`hUHa^`~NfoSnt44*yWU`a1l2D4Sh9TRQ3RP=j{( z?BD6vhWE36nnz33-rC0Q;@07CsbN8{U|60iU&!@4g7n0c36wnr9 zt_|-^=pX)P)<67!cKG7|(63&6haDdN+l0{Yx7gu}4Omkru>O%BO$ID6H2eTN{F?_r z``?m6XE*G5gytG}K74K0tERam)^{P#Met9u41-r1^Ieh&GG2m=PRK;EB~M~ke?oH= z@)_k~$?Td3@?7}4W&Ir%1<(kypk9ifjrPf|Ah(c@FnPEB1&2cAe*O^LXo{l`_K^=0gj-7I~sCq&~D{S3|1&&NoKKgPAR=~q6k(zuVgGCfyQ z=6OP?i;6?eFO-DHHe?gB6ZQ8F^m-2Zb0K~Pl0E%9(Em4aU4u_0?t5@Ofa?i-{uk~~ zJe zr;tgKZK)SGXaA$P>N{AXhb>_V;Ajd_F6e0L=nQmv>v5;9-L0a|uXPJ448{@8-(O+$OD-`mL+bozpw3xsHO4PBir-j0qsZ%fNIZ)cM) zP}kt?=&Y02PG4I`d%)irXf7<+*6M5Zwlr^R%U|4{U-$2e!2osnMFn*Qg-h!4^YZcv zaa&hVuqf~0{KZS^>hd6`kn1iGqoc_i@HO5&f&_dWzQALU2v!RCTJ8#OgB;$wLXbvZ zeb=_StpRVVPguta@GdYxFwGihX$W}zO?=Y%1MQvdo!x#kCtQ15N2j-~Gw&{N-v+P0 z)fZ^EE6fbja9OOj|E8B8`u4iBwRE%q}YS$t12$_l8 z_;N1|xtE6AX=z~`*T0vBe9p3W)3Qa@u>x*B4Y~98F=~-siw)L-_N`kxe4Tl$uCDE| zx+rGVvQ>2gQ7>YkpS*~}5y(=*$_Zy6)_JZmSiu6VY89RShReufS~sEW@K ze5k&9eMMPCvAY!ef1IO;LWd+ox)`6W1oC$V;ftkRjb1mME;P4zJDc0v+@0<1EgkdR znAtvWhtHi~xM0yd_joxWCgYgb37yV19`xy|QZS5>ugZQc51 zHES!XR@W`txNh0%vSnp#ewDk>J>MH>XmWS7;ymDCuNy-XuXMhG?>{WO?qE*IaMc?<)=nNi+lcCG$%r5J8vh?NRXx&Dm z8|pf9o9c8{WyDjTRJWm9+tditSK^qd&ZgPtq;i{*U8&e-V|*CJl?+>D;I{0P#VkX7W(3oB*)2)?@WyF1Gp4W)H~S%I;2wt^r-=s zN`w2WFWcBJvRUtbdxTvEfS7J{vOW6}?R?x#*{STfoBgO;$iSWi9{u$OF_9#Q#$VR0 zvn|An-eVK|+=mLvOnlBc+2MfQjutF3a4?MxEo8Yt+kI!6=Cg=?Fp0f>h|LNn*nT_Z zjUBu`lx;5x>NdhNMtH~RGdYQzufI?3m+ z4%>*|q~Ii2k=r*x;RSt|5#AMWP9lC&&qk8iU>Ym>3-pgVm-h|cpEC_Le?9s`|4;5? z5weHf0ecQrvHN!sEfvUjDUE%ldwhhOgYdad=tm#FzP7fO%lk$)C5Bkpy zPM%EuHlFKW)Y+Mrbfus2tglP+!k*|K_Hc41CwqXOq5zvXiH|Spy}J@3>;nME2jG7h z0b7JTGEyk;%epmonHbQ7f2;`TX;fv}$DAKmK=``t*U$(0*8?`@Bs?jB>>HfPhmQ<^ z|DiN{D)}=OO8*w<2l$L&?o3)Y;a|LcZV=`euGJ=!e1qAn{P`omQo;Bai2lKuS0WKD zMnA_%`in^UmUUy9pouO0*~Ab0t0?f`gWO)47_TTWYYvx(+MiCt{G1nIXVISKQ{Hsc z=SJ8iS+9Rjn;D!phsK*n{J}nZx*hZQe7k>oD*J0En;D!ti|}(fej$6Jo6RNFCf<)l zXSf*jcF<#OBl=@bj<=&O@TI`Jfp?qmZqx;S8SvA9&lC7z(S9200{>OuGlBOA{OmN| ze*KUL6VRAKlxYC zX5+r3{4F?V4v7=P&zcpjzsBbaw+EkpXixgGqkqoN+cv#C>p4d2Y9z9k+dmWbjTX=s zt2)^m{(ZglGCx56hWRs}_)!J*<2zx$#{r;x0RBVq_hoWO zYX1Sre>4`~p-;$TiSN20zGugY=g5J{BvQY6ATt)PhZbrp0(OerhWM`QWM3ZNk^jJ- z@zpW;b59bjujR=~0AWwF|B?P}#`tVMz7zR|lCPff9f$K**FKym@*T2YN`#%mfZTt{ zJ~#GH_&3iN9WuWpd3aIzN5JOb>+^qC`BSp(Z7gSMJgjFI)innJ3$DwY-)ATjs6Hqte6OnEi@kyx1 z@Rs$AuWxpjj3;CIcNN2W&Hcl_h4v@#k9{;bsVM)z{5*>)*%$UlzMo0%DB>a6e@e{$ z@PG6CT*&FUJ!wurpGCoGIxV-1^mpdJ()u+mkzJ1P{AOr6{3|&#>R&?(u^-6Eg!m|+ zAEy0=gZoc%bbpeS6Qw^-`wd4jw-@;j-=DZSeHPIJpBqeeMEwc*ZF<0+xvyn97 zuc#OFqJ0wOJIJSP15bcgzP_coke|7u{Xzcpz;t~I=p!!7=i#c<^gy;gmFkl*fiOPC z%gI8W?>EsO?3+pM;-~old5eOXnbCM=<%jt1#dz1}ht@9ybRrP`ZT9y=G@p#~0qc*+ z-+?#H2jETq4!mJLKtH)bv;V{2r;~koz5{+Uf`<73c?lo&|4SO>Q}8boijObiM~KM3 zm>YG{@41DjBVC{lNuCR|e{&?n^h^GcoM6#!CM{L|dJzx6FSMcNXBFmm0_sqBbp9u% z$M9*uC!ub{yKw5E+7mtx{`66mm4C}=(syrS{C;O#7hpLF@r0+y* zM~v}2F_HRL@jM|YW)rKyD99i7bTc$Sq4N>!zYcX{sTD=1{W;t-Sehi`CyC+$T1 zC2~DWUL?r{f9U}|l}=0|J!Jnw+VW_=An-p8a%W(CeE3%{_|c9(@DcBX*Ap1$hXAi* zzU+43r~t)=_2Lx9-!1InuYZ3W`z6KO*ZBFK($6i+gZ(;RKgi$s`h3P@PiuZ=%12^Ao5f|qB=}$3VA%BRQugn-DUvba4 zalVqp^A&ged-x!^*AfCFjqWQ`&^!Lg0kt|wYdAvk_mV5;9(v*)l zFy0I|`27U=NS0XeDL#+NkM7)jg#73K%Y0;>DSnOKe^YjWBKO}1g3L}3o`2x{zP#(@ zG2_)yQ@k>qufulS^Q_9J8GuYbh(eksA6zaqaa-*;>A4f&Db z_W@~a6&*J5V#sG{Ns#;L{7x33c*y@?{lWLe{JdA3XBf`IkAmnX&cl!4JRFogiSOfB zr76v_zfRM!-)CO~0rzL@KX1q%z7K~#i1YO<`u>!EAF0z9=804XzpE_!ZN6WN%I9Px zA3t9tA;tP?CkNr}htsKl`npB-r^U|y^KMRF^Z7v<`y%)e`&o2=^PlqWk7D@=zCSSK zC!awbQr#=}G;mJ?_cU-%1NSuWscV1^&kc|1WisuW?k$nuNdZ%jxawOJQez z{K_AGA9>r(mesDAaZF!Ro3N%9yAg!+pU$SZo=-m<)in8|MP5fW_TU`guMj{9s*SU! zT;UPAm5jg{S#X7I!S_k{ooWga#9l2CEJsT`d{nWn`0Xi| z=fxc#kK|tzFc7bQp6e5ZCq7=}TEIgw@!CiE-xAZkvOzrhs62kg)@WL?4oyd4976V_ zB6>FHG|6M}2TsLpNjBL#%9aqBal0Axt|tk-N9CrG;^3olyW-=ca=V4x@%o6$1x83# zB^ic9F-T_rR3x*H_9(K1i8DzPLGT(76-IgZ5N z^OZsyX|?b{WQvXWOV}f-W9jS%LZ)Ewy3fQQAZU zqqY;{ai*u%IIr-{uPsTt(#segcESq{hk|&4|J0%30OQN5j$gEx>QADsm?U}(s#6_C z6Y&Riy5&nB@Lf%y10tdmFFYMW0=Uz}QI zztukCwYeGI&+rv>tkG_!r?L95;^X!C*>Pj#Zmha-Cico0#Jx0)-Ic6 z?c%={HmqG{`nN6Zmmq64_UEzGt1We8tJhzAN7ZBd44Y1)6sNAU7rP!yxw+9O7a2=8@p2X7^j2~KH`q1kSVt%WBvwcx zq~lRpY@A%r~+DG@Cd=4f=R}fl$P^|QKTHU0dI4hr3o_O?Xb4M#K z=womgIG4cCzpnBAdlYZ< zyE9||60gwwzqzf^wF(!`7id0-a>kAWn>f!f(r)++qeZH@<=V4C0PvL!1Uico5BVUY z>ar;1%{>1n?z4S1yV+AO5^0X4bqJc|V!{<3nkw6)hUY1pg)dL9yW^DEiYn z)RlH00h}MqbE@nAnZx_wM?)r}r5rxG2l1RP?$#!H@tm#<6O1ZPVoKSVc+!gzDW206 z>qq&EdQO+3MRQ)nbGkAWIC_mfr%OT(NFkE}ODzfcx{*vcU4vQT2mn^Eaw#R>lw48B?epk?c(nN8{2DIMMSc)B`A+m?HPij{!CwvoC48=-Iej&dT z7fY(7{l08}+|ZV8t$HW_q4pjN8;#QBZs4ep=MnGL_7#N0&ojj(_V+8YeXay49b4Pe ze(s4l?T=a7tDddxX9$J(J&HhHPs#Ql$U2QrzcsP8C%;D*qxc_?Q^aj(Pq(VQvg09i zOfU~24dtw5imCBcJ7xdJ(D+8}6&xd`n-&ACKj|pgTGT>-^A^uA&ba;e6FF!MM9M67! zI41so9obKMyl8j!PsMf7{o@Ct7=c++il0AxSXGs;iB_Z&0FJm#H*SY)^@7ykq%;P z$3b#FDLQpk^GjV-yKTj5*LM^zD;A}6RY^fhS3_Y*%cjD$wcCqZ)_FHHc5T{Nli$#~ zp>{{nns#q->8jH8B~1;j%R4u3tZ8ZRuU*$tv2%GtK@(tWYYAW0vUJU+jg?LH8`kG- z-dMTPyP;@DOJ!B<+MVUgTb4gs+vKlV+p?md_R)%+6&nj`S~j;X&#zx0+Lp8|ZK>Z_ zlIPpZ`(3%QV0l-4>(V^%;IA)OT)Vww_4dY%H65Eau6}%F>+1Z*^4hM(6)ih4MjQAj zUthkytD#_NH;vc0p|ZWCVrN->;rhIm@@)V98LW8wPk zH$_*FKNj8c&8@+v&4pFaM{7CBymd3_F2B93VA~pR$?E(~t=pDuFRtm{w4o>vcg%{N z8#jUP`ts$EH*^;@L*5qX>hb#0qCDTm5`TTG5B6DKS>Lwi@l|Al73-V5#cNlUw5`r- zXkFh_zqzKpbd$HX)?c%jmMHVwy^D+TmiX$I=NB$ow5TvI53f(bYgCqdTRMF5Z)q)FDC6_C<^}T@ z+m&-iKR!9pr^5p80cS9_bZ%7$9+S8!7l4W z_XCB)^_04|&`ksUoWkK+rtZs}u4nac*a^@7K*~w7-c5fA~7^9{^7L zoZ)t0OwB9pI{ZJ6#Ea;wr3qe>a?qIZ#(hJ|>y{=^^Q#&3)X!xJS8WRP6bc#pYk0-O_caa7I>b7PXS!DCD0cod@A6^E#qzj z+?XmJlW-arr-{*WT;9>RMtGT&lloV@5$Q?^xAwD9!lzsMQTn2OjHg6BhW_a}C#pPW zfE($LUk85005{5;EA?*WbGC$A zjWIIsI`AVBPW%{Gv*#rIAq#xKKyS3~YZ7kdpPr&ah4`5wZu#Fu60XKozStz;YFq_B zVxTvU`<#SZ$Ni;*TgOe6^OnXn&KH-2kD4zMPW>oeNPsN{xIE(Favd?iqyM=Fm_>3h zC9j)PGxh}o+$dL*geyHL`W6Yd%KM^(Tjed2AxDjyC&yiBfE&jRO1K(V(eIXU>$u;O zaO=2~w@{(ng{G9!9mO>&3hv=G#+u|+!ByON&H`8QVZZ`cabVa2S4*!t|5W{}^?o+^ zqoUwF(tax}aJAn1EpWBoGm&<8)YUh)c>~=HdxS>sN^e(NL(`-Dm6OZxKxVw%k-y`K zUf0NOcIQDI)it*@<28@<9UZsUtgWk6Fx2ep0Ec63a^%*5zg}`|>c@P64!jydirQQs zfZXN2P9ZK{Tte@$EMHwaAMcZE*deHvLmz8>oyDDa|73kvr;nh59t};{0GQP)PV$6L8_{i?ENZ;P*?b5(m|R||%}$=x5fK});0v97bd zu8}h)1Oj`4Pf-=J7$scZ9#90gIu^@qya}6m1KT?4I{dzd(eD)vv^RP?y=(#hD;!c- zX(c2n<0wJ0prNBPfALLg;||#i-@I|-zA*K6Ht?o4pL{nj=3|F?ck3PcW>>Mme!{#_;W(Ou{R4nBzA?9^mM$55JybCiHNLt&E8A1w#!{_QO*~n3 zSluQl6u*kzTK_9K{`J~_WI-stRe$La-f9!Vtqdp%^VLebUA7-3zij`ErGb@yAMo+) p=aBw$Sq@M3BUe!ERSvx$16#+YCTBb7FJ3$@oAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi7z(=Yt^fb4>VH-Je^s|`f4OYsa$VDyDjIu*#cB%5SY~hR-otM_ zEESY))>AWJtNVGj+xCCxg|^SkVC?XaExYd>yw2IO=l%<>=kfYi+@tA4MN8LfyM7S* z?KpPz!wq`qgDP9-nvEU)e>T>4w1XW6?U9}gR`|VaPydG{?5$tv?C^}#o|?0^$$dk% ziPYDTo^)3De|1m)Fxu&w?X5bDJ#y;l!OHUo26w#o^x(?VPY-@m_gn?-Gk6`IbMo8a zz7xm8k2=2-K7Qatcx3*`aNqn>;r**mhcE3v9Uj^LUihi`?}i_0Js&=P>fLb4%!flG zI}Qv!_4Lz&sq>eF=Cph#d|}@4@V@tc8h*<8X1MvprEuTM_dr(|3LQK+xc$U|!F}_O zhbvE?2sgK#4)+~+Gd%LprSOFvmq4>PRN*{0cwzn%gZoyapZ(tn&uPW@r``-#oWB&_ z{vM6@$Dz=CjJNtA`Z*Dv+xqSB2=JBXPle~4JP+QfpFawX>_0g8)cmIhL#IxJFW}Yu z-kagMCr^eeobQD%%s&sBWuZB(PYm`!7ZvA^qwf>p(1}ywO6TctX#S<}Q>%&RrJ>M? z1A}uq&ll!@JG^iHo8Sw2fb4yECC~Xnr~?=P>{A$g}#g_ufd$PZu zoY+&7{AzZP?C7zvLI-T5={+^|@Tp0f=jvJ8q&~uJ(OF^C{=f~g z|5Ls<0N;BBzIo60?rh)dxqc~rF{l8wz`cpAK&dy?2hkvMFeI0&1l+CW5Etzz9s6jh?_HXrT z!~0o3%_F7_JhB4#$g`|J^3SaQ(kuGacS>0Qx%=4Fe_6!(&!@4g7n0c36wnr7t_|-^ z=pX(k)<67!cKG7|)URHAhaDdNn}pEtx7gu}4Omkru>O%BO$ID6H2eTN{I3sy_OFvd zXE*G5gytG}K74K0tERam)^{P#Met9u41-r1^Ieh&GG2m=PRK;EB~M~ke?oH=@)_k~ z$?Td3@?7}avi=dsS0UvZ0na1gc?3L@j3nPh@Hr0sdm)qa@3hdnkYnVPgwP1&co%XQ zMCzuEDDk_dU2C!1V-P{~PzGas4+M zANG3Z*@Tb{c2)jo8_%x31O2;Z>LC~Ga}j<^cKS7JKsLVso1b08`iEdYvV-#J_t&!i za0%;Y`oJS(Q)U0l--4~+%kRRE--FHm4fc!q13P^A5bSEOW7Bk@5AxHGtBOL`l8Ud8 z{qE_*tdHm5Jn&=wqPJVESHD~P4;pi^%MIuRMQ55u0X`_}1Y3cZ1D)@{)q__OUg^w5 z!;UIDe1SG!O9!h4CYP`}W}cpktFAHIc$=o%7_*PVk|&GJ$?UlOvdC+T6hdeJOUNY2 zw$zK8v;SFK^&KqH!c&QG%qUH*6M5Zwlr^R%U{x-U-$2e!2osng#~qs^Ow}+=jG)sDyYN1 zg2j0c=Py}WSCbthp zZ4G!^eZo3cfOmljf@#)3OGCixZ{m~AA87Av@9g%YIpNydIy${=oq2bG`!;y}t-e6R zU14UJhRb5LB?ro-lN-iuFA=76q)7#X~S;WT9$Emq7}^kl*7l5f^WoT-Ntw9 zX4I1$j*~)5c@gjoSFE3&96?YIR%YPVBk4iL%Pzp(N7JHk31(lH^lE5D|6RasIKQx# z2uiL5(mfUMy_>FSse5{QW-~`*PtR-zzh#Wz^Q@i7y5iaXbpS+L`x!a}p(?QCvub9c74w{*;NV`lrj z9X@ydq6LfRx!2QWLE$o1+|}ID=MH>XmWS7;ymDCuNy-XuXMhG&p#}@?qE*IaMc?<)=nNi+lcCG$%r5J8vh?MmXx&Dm8|pf9 zo9c8{WyDjTRJWm9+tditSK^qd&ZgPtq;i{*U8&e-V|*CJl?+>D;I{0P#VkX7W(3oB*)2)?@WyF1Gp4W)H~S%I;2wt^r-=sN`w2W zFWcBJvRUtbdxTvEfS7J{vOW6}?R?x#*{STfoBgO;$iSWi9{u$OF_9#Q#xLvE*%snO z@39Gf?n4D-CSK>9>~O$tM++7iIGDzU7P8!+?LJml#UlE_B=-6tHY=E5`^}U$cJTU8 zw!JWjgFoOIBRu2_PK;!umM|IG#(1-WNqUt2WEx8kOiY+a{SO{WV&{mSarz{p2R=7w zpGaq&qJN_2(@fB#f9S(T;`pmS;QZ%8J_9}YNt5;hKj@F>x&AUbHR1<#o#gXZhi$}f zQg9Ni$nBe;@Pa7S z8_)GG>g>!*y3$X1*4L$ZVNdiAdpNn1lRdysVSr7X#K#x)-dzb1_5lFo1Mt6$fGt8E z8CfLoW!)OPOblqkKUM_vG^#S~W6qB&Abj2SYv=?0>j4{c5}uSm_6^SD!$$_d|4^De zmHZhCrGE?b1AIm>cP6cy@GsszHwbeK*J_hVzQJr({`?VOsbKsIME~H-E0Ks6qo3m> z{Y9jFW!+dNXktr$Ht_@hDhxdMAh(w$#w!fWn#1Lx_NUV@Kj%f*S+uA5ls6sqxe<0r z*6ZKXW(McYq4DMsf3VM07 z9rRe+i2j(9+Hy+D}7W;J*rdCh#7CpPk10&qQ6| zleInFF5t0<()djR??PSR6LcZZ7KzW6{BMzX4S3?;FYxsX;a?9vNb+(2C;tlCY}}WW zzXj*aA#q~(S+k<`*Z6$l_Tcjm?MYvD^w0Tu+oqRiJ;!KWjYRfx`)9(w(E|D+RVRDH zKQNzj`Fy4M{EIYJ{JbW}$$x<_fBt8(xu}os&rA=0%_Vzsf5mv@Ut5So&WDWP%$XGb zxIg}!`d?w{zjxR4m_H32#QK(CBY)$I!s&(ZFRb@b|04gHo^VK^Uqeo< zFfc74>aSQ2W(KECBl^UczfH@eI`_A|XzxOM@3%?cbdMtSP2<^>e-4#EKb-FcUMMC-}P-4ebf9l=$p^i46g4CsXzFCMpWPR*;p@J7Ja*1F@0B& zzFe{S&F2T&OMRpNj3BfH0U`svV*RE0!3`+&m1UNf`irgyLq)7SYqm+gvGmou3;DoV zG@yQ|&+H<9p!K1i)=SY|#kZM32d8(${LR7jLH>45 z(?|0syp^xVn13r^KT*W|v&KWR?-?SH>vv&r{CKiIA0P3eJUN%wR6OY2g{5D`FF(nr z$K=a#B3_EmO7G+Rrs;(gfBZI>jMi_OKRS<>I^pr_v|?F962zVMCw-$WX9t4&_ZoRz)q3d5Z`s3?91aj@*ns!J~}3U z?n$EcwLDn~Ana-OKhnR=7@zIOcOu_V^3_wm<8c1!+J_TGzC-p)iLi4Rkozy$=f?gC z|K|CkL*|zx4=*bJ2-qBaef}RRe@eE!jpa;@hxN>ft%As>`FfUYqiDnZ1M(7{uXh=K zM)DT+0?!8!{*%7jmdx{G2hWe~xj{$r?0<}8JISBW9`f_?_rsLVcKsv zxc?+a_a|97QTp?=-*6;zdy)U}{fV2?XAwQ{xxr*d)Sr;wrU%?vZuA}@dh(~;4!%BO zyfmVRKV=&IEjO6uHnuO^?RL|A z+9y%IgM8XH@C10}>sy)&`I$S~ALLIDOxLG?KH|cB9P3*9eC4x z0N&*9z#HZR^phJj`#=1BI@y=!JK#4XXqXR>m+(>lzobz<1^+^!`1m5egoyl$xlt$m zo?D1I(go^}k<)H{vrUnDG-(kKrLN_gAhD$V+t>I1u`=VLs=Mk&pAqCF0A4Dsw!0 z5&7e8mP-B&e;UW$s3Lz#r565Dzg@(WFQP`puwD$K%HzRwvR~?RsET+_`cA}l#2C*L z6RCd{&l7@THnAFvg8X4mH$wvyIv>IQ>rgkAT2XY`pJRV44l!we_y+fP(oV!*EZ4K- z#gbg`mmbhl>BJ<`L-s$UEsy340{_z>cLv7Chkx~gAMN-9AMs9jJ%Mq42=F@Q%Wemb z3Q%lVFHT|n-NGLJ`uE4NUsAk%ji2u+{oJxV*st^TgZz!J&u2{bwB~1~d<6D85MW49 zr2VYt8JEFd*Kr5ML-03F?#E2|4&^g`5}5K2%HNZpVZuT}U$>LbNXJNgL(xW}rqu73HNHee3n6HG7 zuGeGcD|~)i@)g>QQxX2i_Y=zguqVzRGbaNZabdon{uJ{S@`t$j%8W7c759uA=POw} zUvbCJS0GQ8o8o&kUkMuWjnVlE;;B0;ny(B)f1f-b$)fd@$4m5Q$wv?`P5Foe3uEoKcXM&n`o!Zi9jVdhML85t?&I@)@YvM#54!xLxn**h3f>K?_&KE z=S4?1uz2wd@gn`-dH-#SU&yEQ0RQwM`uPLlqwx&`{FLY0Z!84;WW#>^5I%3PPv-k^?jQe!_T#bh9*l?mQ|$Z- z>;LHe_|$1Pj&D=AzfX-H-w;2hOy&D8wI3fH-%RH}xxuMZEb%RN{$t(WV*cL5etU{B zz7+;1=nNlViTxqXkIV@ZVIi^q&c^;bYXW^fXV`yFqy3^G-Z^PQ65D?}on!33aiBuw zcH&*tZS23(v08#bv0p`d7wXadDDt%l#`7lNGmPg^&-rZmg` zI!(uZpM4Dk+@G=kydi)1JRJTY&eyZ(^HctLq)s20CsG~!uCna6`F<@bpOcY%{Ctsw z6zi*<9E7(YPN)9q;}+SU7CZmXyE%Ey=Lc!*i{MA>XVC%9f6BW*isdKx{J@l-d;}X%HG5}4R z$NN*`y_#T0y*RZvkzPfq9=le)d9vUI)M%mLG{zNH3`o(aF{-KOhlnRW#+6sW2=zrT z(T?*r&dM8_u^4`DiWN`kj=t1caD{Eb=SldUY6=p>UM&$UM@u|>RI#r3?J1Y%#T_4y zl1}1K3?Qnz(XO;bl|mck zT{N~@QLHOQjHl~qQ1u&s1&X!bt6ODu8ODug+o)%PE*@T4f%+LOwPEB@+C&4RwiDxV zrl;08ukg*UElIo5%NQPZ!V3+Df_Q;H>d#tqL)n54CSn=&f(SRIBddc^hU7wCB52(=U8Pu~4 zb;47fD^nYniVTl-05)v~*jS_KIWELz&mPjThQN%K9&fQ-0p)PASGVc~b?S8n>eLnK znnul0_1Hecrqd`zsq5@TuE$btZZyh8#?no^T!lEjm0Z9Lb`3h#5y}9G719XlcvKb} zC)YE1uO-i;evH-^BN#?z){)&HG2rjTdguc_W?G{^i#77o81u7>n(?^*?6|SobF8{? zhTV(dVhtHfXP*>r)cL&_ZeGPjtWfb`bljmmX8e;h%j&LMSAU8ZhMkZ`Ov8L(LE=hg9*_Ugq9x^EB&2TH|ZzN%4d})9=+P!(TWTDJ9>Y~ z`8S{CecsOYKJWT9%_rlIO~m?G)Z061$L4_t#bWkeW{i4j*1o0dt4p!hX8i11y8`c{ z!*J}jRT*;x{#l7fe1=6wvX*mn;o~O!fcSo#*#5`*AAK^;B{1}_YrOv+#T$L^%-Fxg zD>VNzw>7#};llX>%_mXL*l}PJ=NU%Y4WD7OST(m?dsYYlKC*#8XK~^oA7oTr8l}9M z=l{fgw$Elad+LRp7r*<$#8&pbxxe}CM%!P_nl~fwC)0m?C@r<3C1oLo{2V<0ry5eb z@ke!exv)nic*664US7`FmU8@UIbPR${tqdLY(dv}&-NksFX;`%hLfP^SL;w$+I<9Y zelX9euKzZN_rZ^bOhn5#d~^@uIbGbXP4wb9T^S}ARi4C@vN7?b7b8+Urz_Tv@)z}- zE=7yxyol#?Wh!v=8huWegedwb!EN5qOy|dRmi8W5Z-{=NEC8{ULDXpz5U>4@WxFp~ zRBLUIKZdlNafyCc&>v}{IAjA_?`SN=j?xg>!`dgctCkbKi7JL-r6s?RpNWelRnmT6 zwm)uYOSe|NlmAeAkA;m!X>vDkRLJv)cWe6!LgMS0;u3rRifo@NK}yHg_OzdSB2N2b zmiDS=Yx@~OA%2e{kk?bP{Rgs68njlSbg26&$(QHZ=* zDz7xCEc_2$4Rr#47hXp^QU9hQ+j~qvi<1FI>kIK;Y@x#={CWAAq2qY=^TRRm_jP1H z>G7i7**_N7#rKPA7A_>ei3s)o1R#x%VnR*aB!JYhu}In|XdT+`jWhmNWcyr6N#j$o zsE_$%dyi%OW3s*SUusV-ZEe38llQ4O<9~fKh*8}9oiXe9%8x41z8CF1mhlH zeAQ0DrvebK{o8Tc)0D8bryq+#)bYd2CjDQQ?Irusi&g(?(f+^2kzcjHEL*}&x&5u< zlTVONx715Q%Oz6|sIIC#fp51RkK%HET84|xS2b^q;}NfFURc|yx<@*Qu^k7=`K0L7 zRn0GTRqeJFtzF+yR8}NP$*SUlmac|H#Vwl_t*zZ&)UwXIp|NYz#+v+w)(y2g3fHuI zi%M3NtS@eAXkFg9d1FmWgMaP1mWrLr8w#2LTU$%`(w1dwHf^kIs^73aZ}Y~=mEH}7 zJ6bBMYS->8U*59((b^_|&Dxd~1+|Y>?5x;WP}8!xb$NdM3emQ>Wm!x8#^OBRX5R10 zjRniQ>RXrPfd_wm!IIkT#jCeBZmj9pv~l(0D_d9RHldxhYboDW86CHL+iGvovi7p_ZR;dW`KGq@ts57u-+ohc1^Hvq zE#KT4Y}&l23i@a*Cz-cyCf((?mlkYW<1JpDzo~Uw+4iEE?oAsC198W!*tu~N_^vNs z{&+)oVKd}yfvz5}FDcCPZ7lZJxB6h8^_BH)YaU-kHdwK~*;}-BRdL(uyoT2GP4$~= z+DkThYis>A%UVjB+DkWVtM(R`tXf{aI)8I{S=Xl4mafg^cr9AhwRuIQzp-_FH}qGr zCV&0ojVmf!Hf<Eechf@NC^>zClSN$MN678EQk^gg`w;W|Or(7e^#;Je;WtgPd| ztwO)5f?qqiol4#fo=u*=Xz}93i}Lc8a+>AdmJVN?crU5N56bwwt$D#b#&+f0(T`8g z`6KtPFMSEd$zPDqO{5lM?eQ9>?VhepOU=fET=4sm%7D1$8t&)e-nhqnk``Pl>r(Ic zJgXnE|9Rqnb9^oJ8_pL*&qUwI3B=p(G3aTmBL@1{)cp$Q(Q)6X#P-KGzV8W?S$?UuwWde-&w^^W||gzzbrf zCAyZIpi|vjuP>sloxQJ)Bur@e;5ZXZ$f1-SLlFy_`09ZGHi}EFo!~t3aDa*rREhVz z=XJbmG>TK=J!e|b^N2XAh0QRoLJImAKW4bx2`M11g5J3E7RYv7G7LL6B^QNBf-CU`hC<`fueuKW_h=uFp7uBK;16F1{sX{?pEKMJjH!90 zU5EeYk$4e(wKTzNQVtq3-nef_dEL?kYJN3?p8B~g;i_$cgsV+zsq8;T_ERlQru6*x zl7Bp1iWfDno|SyQEa7TCy(aq^kOoxn4S=VsLDAD1)T!Vpz8t|h5*9b3_a#H__HzHjr;nsdOO89h3KT2QJkMWeK$Iw4L=R}p~3~(d;@$0~^ z7~n>EbEV#`e9o3|tGol!KvsEQmvC!8Ka+5)yuUQ`W90uM1KcQYrkp=Ct}#aDT?c+d z!igW_YWAFjKV*Rq80d}meNDox{L@o(s1QF>#4UecEa7Tg<%>-cuEtgHBL;foxX($r zb=+S{xOLoAId5rP<9u;R_^A0J;na`fg#_4QfXgEuF4qwQJo?8yz$})7DS6$Tnz1h! z;6}NcBwXo1(YHvrRo)jR+$wLW3^{7tJUQ-41Kc=nP{P%?ihj3*TgUyLgj>g@yoCzo zE;OZ-?kKKNQE(5hG1ern3a;YDa~8OY4+9psiUY$IxLSJE`KRh%t@pFR9~A}fk@j0* zfvff2Z-J}zo{6-(qprTW%^T=u*dsK0S9-hJ8k!#EKRH>32QuT=9r~n|TAQdnsvBq`-6 zL9(EsqceZWO>5&0*$dyiapV4A>g{acO>I8;ZeGmC4)xoucjzy>iUsx)zVAg4k*Suc zrziVo~PkkFpwF=&iMrdq&1tm1`WYuAHo1jqq zDtc@EzsvEj*Zw05Lh-HoONa1Qn-FegKv9^lR@&{d{V4fm`)4c-to-|ck7qxJ^qSm9x~Xxz9f1BG7_jq!W?Yn z^YI~Z{i-cU8^?X5+$2pAF%E5WQOj*>m>3XU)d7ZkLOx7&~^+Q*i1f{JrccDES4hui)=La1YX{sw`cv-2JVA zx6;|zn>*YCZ*1`lT=B4D|K(w)PV}&2DEr533ak8DL16IBRqV$tFcc$g976eq`X`Uyq!6;@2owIWX|ZBj@ip{qXq*7C$kv z;f2#99S2?*IrZ=lM=pKk*CVg){}swq3_S42gXd2ze)#+V+J5y1PmFY+%~vf*+n9dGL>(9C;w~dm|-TFN|Dz=(UlOA5vR43@iY?1b-cy zp8)?~8ClTz!pI#@yf8BGt6z>h_z0C*IdJKrhtHovnFmV0I#ROs%Oj~zK0lIH|NKY? zcz$*ND=1SnK(gBKGLBSihnTj{o!AaJa}q&By~acz@=O6J7048 zkIrAZ1#OjndF0ZhAB}8C{vP=G)yS(0UqP9h2G%G4A?kVH{H2FZjg(|OIWqA455d<9 z=-;nLN;bWYGBpG1AN`~A0}r3#ILc3T6^yQb@`0@je)W}L>b(VLQphqHzs) zSfvkR=5mL(wm`?GxdLMsJkw4QuF1_REqwq^=)*MVo6w0S7pr_0I^l8=ZrZ7fZr?3L zcY?9XZzHde(}=pBg5JEbW!1o&JL;f^fw3!hfA^%W&++v04Uelt=_u)=(!~ID@$aFl z*L3msOBcfv=;Db4x_H8&i=?AUABP3{ZS+Eqli){_2X6kwUB`a|-9Gtr?VvBJs~v9o^{9{kq+ zz}NyW%X-6;eCoeEUf2#hb=U0Vv9I447;BoDe5E5d`O3a$JcB2H78vXJTi4jY7i$Oi z{kv=IUd%$-AZUy`b;I9Hc0QeTb{{r}nfcF5t5%4pB{|xXa0M7;d zb-<4T-VS&Q;4cIIL%^E>djKB?d=KEYfPWJh+;<<~!+>i5zXUi4_$#>X!F3z*OOgL) zfVTp^1@Jt;4*^~dcopDWz<1YyZ_gn38Qb?Z;1#&OiR&_4M{r$&>ub0!#PwBN=i~Yc zu5)qyS6mBlJ&)@wTwlO-IPZi75Ni+(J`^-f%WfPQVqwH5e6;2UwB zjq5>Nm%GNIN7;|>hJS2wmmX_#FFn?NvFzBM@01F=)BU8=MCUsJ zpYIQEoe>Cc?RYZjiht9v_t}DQ&XKPtjWzvyN#ze-*|exSCzj-mfs-Dua8rH-Rbo{RlZ)9Un|S6iBo>{>Gc_^{3=<#PL{8g<=L^4 zTK3~c@V|d~-Qb37HntEk)BfDrv4x1EF2UDZ_^rTCmH2sa@OyyIllanDe3h%ff`PsE}77vR@P{GK@Y&jPLI?!9Pj#692bx z@Xrwa6hZ&*aqxH6j-}lTyi;!?$ahVR&3}Z-$I1UwDnBi@{O75BocvQ%J~y`flTpz?9@|B=e)#g_jLm5-DE43(b|TmCsJA1D6>DnB!}{CO%LC;wlm{H)mW zuTc3o`L9y>{MhoZQTaIeBUJvn*z#{u`8fG+Q~BAk<=i@0_T;@i~&Z|_}&xVLWL%H6dCm*#%2Q(vpmenRx92eGja zWl~W#4K(SX%|J{%Y2XrKzhT6v|8Wl+eC0+q_8Y{YZ=}}_{ug4>H;)F!esK_S!e^2O z-a@Q8`c7bQ4DsvF)6*8bjd=Fm*EbJb`C{$h@ch&Tmwpr&8%F&0-oFLLe*RKm?BWk< z2VX+`_3}%#gTFvr_KUX|?p?@ZW3T+CcJP(oxCSr&z%}?Eh~X~JLi#zq@&8F=$3 z8~d*w*WjBwHV(Xz?i&2f-?Ondwrm=BeU%&SxX`A1@K;$b@ZlbOHN`bH@(}Up8XR7M zdGs@ZG0ca9muCT<&c=ojOAgOR{#PM~EWqCc{7vAp0Dl$me8_nrl?D76z$*YRa}Qqn z;;Mm5(_Lf3KVoB-=B{6G=`$$v5@MpG9@^{S`$9g3bqVJ3RLtQtw(o6uYJl>vmN}Jh zZ^4#n)yV~+afdOit%Fl5Hc)YtWEq)tU?*0aJ&jsDX+6;5( zXrl-E=Y{?y4MaVeKlY)nv_-W885MN{nboU!z5nyMz*siw{&!Dc@XXQKqpsT81-{e( z)=$@s7N;UFrFy~1RRw2@QRbv4e{>zcUN^daPVIt$r}NLAe0tW|^(eoO^8fmVfqkI) zKK@RwntAqq$~!t^^nMP{ID3-Ad7~#eoOkwF4o@E)M!!zZFJ0g(T0M}4ykg2C{&%22LKG!g)QpYRcJEfBirS$}d2ffmJzYQ=bov zojRH`>URbHAr<*VJ8*RJ=*d-+QQtaV->IY7qiM)TyI8CE!N+>5o#Df7${&06X6Vie z*H|&i4exai`ieGi-iBKRet3z*?I7GOHhsOMpAUNII;USjWj0!6s!{)qRPTJs-!$+l z^mHcjW+1N$*A=+>iZ&1A;=U4pzM?GyQ*oWo?;8e^alM}3-#XyMbuPc(I^f0?I)n1t z21tL(@wac@Z3B~WE#vpM4`kt5%I~)iOu}^$zuz&Cjw{J=-@KjF&#?vgyYPcq0~h~r z=D^F3&KP**vAlsd{&M=jiKCww{X$CLf-h%%a^OpzkB$D-s>ZW_{q)DqzBkJ~_{*;p z4E*}{Y%&>?0$GY{6;#i z%gBCf$A%vx`>h)so(mZMX!udU^?;`U#u|6{4*~B0>;a55@bEo=TLJ%uVT^G7C9YU2 z5BDJdFyMa!d^6zv47%KKcK8PjdW!4QxE^g79d1Pas$0(v{~f`%jt=i2co&44|(j}F7XpM3iEvuJ1Vq-Xo+@G`(haa}_A z9cPEPA%7LFw-DWq(c!tsU$yfr=m75=9iBz-uCv1n3Enk2JcZ!hXORzh_vmms)$>0_ zhf{EU`hT1q_K0=xzVpH_z}B&^xCXYr8gtMp_=7t5gsR{og4&s0GPs6Wo|AvcYvmuI>k=)GXq6aAJBushh@aIF}{kR73mx8~^ESsu{ zN&DM_2iiM(SOYLcgmohlb7$b{b~Dc;kIU_GF>g9Tb(uVw4Er$b9`PKpRw3N%9|e!H zSlOt!IrE?5>fXpjxl-pVVHm9;eYv^Tc(bOO@;s_auOnkW(hFl9n ze%HdnSk*6TY(xx-P`!G&h#-6UFWueC2*1hcq$|@W89%>G@1)DoM8+$R= zHMSn?Y;0*q^xD|gj`{CkYkONGLhZ(2d+#AE3ECQ4jx_RordPi7>MB13Mo?vQFX}-< zhq?~*I2*ebI{hEF;xJq-1-8zNs}G%i_t#!(V%gT!-c@>V&z_$4-ZIwMc;HT0hzS3; zZCbObQD9(n&7B?1J#h*s^XqO7_IC7k96Z3I5~~iX9dX(3RjfJDjJp6x4|V~ZX(i83 zc4o7W^ZZFM`G4c4da`2jzlVG`o5TVS2~?RZQRET%-7;V8hGc*h58u%+tE%StX=Wy% zoygD0w)qJ^EQ;iU^*Oi=2+*o=3G)3XvaEcGW(~Z&o$r^3URjT-fc9D_J%D@<_I;cw zip2MAfS`Ly=uaPAvoh`thl`mndT+Sc$8SZMMb1hjtYqe`LWZFA_7@5y)mJUc(LE(< zCmabEJDc%E&}=SYvLSVmj#C~h#lxSX+b z*eu(;ef{e7wf{Y1gTvvzd(B-+S<2XU`%( zyk|T1i2O^Jl~yeBZ>LLX~jWuJ?l5%B6hz4&t+i)E@UACp2NBg*w6MEZ~<#F;Mr`a0bj=&3^<>yGvHY)V8An3 z8Q@w=zB8ELfb&?k0Z%8FqLt5Om)$x%ja@L{sqCBqPhrmjw#Pe%oi*^2*+~P=W+#mD zS*+i{XR@#XPh#B$oWYt5IGr^Za2i_&*ybmd1pwRn=3`X`oWjbC^2y8(*p^=s%Qj&6 zVgvTDu!~)Gxs|@dR~WF1g_(}$MndJcc)3lwYB2!Vcz4~3>#QhqZ^V{+qsQf4n!(B< za54Bhfj{qpe9O0!UvwsKaXyU{D+7bsFz{~tokP=sdvSjhOm7G?@5*lP1vs%NG4U!2 zyQ;yOfQu!J-f_H7!stDR%OvcVaJGaYJ&v!Ea6rO&5^j)iK*CKD_Di^1!s{d)mT-fF z`z5?n!Y3r$B;m`MHvc%_qp#BlR1Yf{(Nigw8@VUtXCVmJW$`LTN&&y8{Fv%m$ zRmd(+6|kre@H7F7`U)9?0a5*;KEQbb7WJXLghhQhEIe1>seinE2{Q>d%@=r|gaZo& zoGoGh5&@HZiJmPJuwTO2D+Da+%V*s;2w2ny9l1%sqP~3A0R1EWMSb~fXtjVvefiA4 zPQZ%4?pp+`_&c{*z>2@JtpZm3_1`XF#UI-xU{&AH#{|3+{3pA->_n4*n-mP)CH|EB zE=agc_SgR@fv=KqlY|2jJ|W?C622^9QC}eod|J?p`U+XUghhRY?1F?veTA$HlPmEf z>MLa35?1`3k#K`*Pr|CdWq&B>eG=a-VI{va5?1oNEMdPaAGlx8i~90ezl2477+(pC z`tn)X1ERdBFQ0`aEb7B}OIY#89yIw2NLcX~mayV)NWzN0Y{b?iA61{HY7+D27qJuY zM?*7UCo^H2(D?wyy$OF|{9O*?igER3BZh!K!vFlwJKIY&G0%)qk2lT7VCtA{alc&V z@5f`xF8u8!3bczv*Z1W0ioCunuN9&ggA3ttkBR}mOGHR2U&W%UMYfeMLF*~1DzEZS zj8k6WKkRGdZda8347K|3cQK32I1&&|%)zBJlRbR|&WsXtg0*C_NT?b@<~fYLkOh3M z0ItX73z_U#uzCtgu;|buS?t**tUolXn)Et)A(}F4BLs+2 z*u!4UVox0QMp*~IG?XtS{^^OfOrVeS7A`g8RES?XVsH=XDu|Br{fUgAf=kv<=On#vRB2koZ` zklM%L#3MXxN0tRA>-cAS1jPS6kjeh`P_)`hhU7y3TKod00pTzcFd4oMmpNcETpeHK zfXR?`e82&dVe9yH4!FSqlY;2wn<%S?H6+S>IScxpG=pSe>r9&bngS36DD3iJSq`72~eig9#LsJDk3#YSRLI2R6GAFA4gyBOeuw9_97T2zoq34Sgd1d^3VxGA{w6{vw!%pJ4V#3i1E&5v;!vpP*1WEvpFj zBHmE&{YOxA->dFwYX4;qTW86W)pK?Oz@aF>X)Y{>$m?JjvVVS= z-;n3OX0c6}TSXH1Ck}5g{MqB7nkl+JiS*U8T0y}37wKE$fL9rCe<)BBhBYbw)gM}0 z!=OPnyt)Q0Yj9l+DXhXr`jQ;=d27N3KE;7gt|8^J=~EqeUrpG+r#tXzHMHol=_fhx z88u-8pXI=3*3brwqCb0xZ3vLO!Ze1@tUNs9SnB+6d;0N)OUu@i{`tz&VcU1jU-n7R^-)tr$eB#H|A1as*^R@c( zHRyLWiKL(EbD{k?)f6sT{SDn{@Ehs7!H_4d2UysyuRpY)n!-s%-ybTfX5Y2pNZUg*DtC$@NF1Z>}MaNZ&lYf8c-qbRt&mM?#w%{BLyd zUsA*UnYF$tMm(a?_boKq>klohrf^u*-yd33&7QI0NZ(?%0PQ33{zdvq9sDhsP63{x zC;8`D{Z9h_W+Fv2`Rn|}P&$rr7 zA^GPpQUJ;Sg0mtSPTe7rX@5~xmublSH^`BUh{)g+M1hT_xHOA`m2vO`CEOw zmt4ho7qZnxe@%Zw{)O*fSpLZLXNEr#{hg`X6Sp60JjM7HS+MMHK=1G8-ba6FeD(ff zJQC`6iJFe&_&WXFlIaxB+Wut%|EBs&6M$yV#a92M{cAAS=XJ8bZKThgT7NNKPJQkx zr*Mvv;1}z8)d3#QV*aQ3jVC^zdrf-|@o2`zhoCQp{yY8ais{%<()vHKe|gbSou(Y>O}Y0#e)ugAbUvc@5r8! zq&_(PUCMM?&)E9l)%+dw59>)i{)zM@+y0J^hvr|NyE^~pwe*GC2R7h!`|xV=*b+w` zx_&G{|7Gu}hur^>HS7M*p)c@%d_8B$!}Nd7c#HfYU!QUR0C}777htXb!vA~qc!&E# zVo~>p7A*U3_`?@{w7mj?as0pDe;OYj^Yij*d{d2hEYj!bzacO3Cr){p>w9N>NB)x5 zySBcV@tPqIp$}fCzx1j8%lWWK!D2ov7Q=g(l!CE0(|Uq<>&|JWZ|F`f1| z?EY)bo;MSr>o-nzet~R zeK&nN-(Mzs7vqPuq&fd#17G_0EbiZ@TCj_4vi38vtAYJkdLBaaS9il@zW?1H%D{Ti zj{o~ZzH0WIjQ8nzN$N)FpNKw-m4>X|asOuQN20%rbv*XdWA`VMjs3}3c|u+%wpVP(uv3#Bq=}R;4k-k)%;^T>r2j4H=smmibGYqt9 zpT+9!je(`(@uZ_~_XBhN|SG|lnDCr*yd6Ox?5bYJ4zgS!jq<(Ygx(u(IR4SKYn%;*2G@^>q2{xJs(%m0Q< zzQ1j?e`8GhwmuU7(D#s{C;sa)_2)gLZ??V?zj!`)L6s+dr>EQTweSyqtAE5F^s`Bo zC;86It7biL;r9bQ?yU~muB+k z1y=i-|Al{}_HF$mdg$MdpRErZ*2Y`lZG9DZdwd1n z9zTJ%$4B68c?-NHKdc{|@mBqUb$A;AS0ElgIpZDY{^zIgytsNF;sJV|T~;dg<0w9$ z^#_kfhkVVvK3b2(#-q;#=Wr~eDEeH1pr?3|=u0y3JSTGwonYeU5g^a{L*KINdo%WV zz4Li@e<(SV`m6j|e<&q$wFW;Pnu>r1FWs7o^Es&h@lYX$t0jCK@eHV=b4DNG&j*i( zQsAC#cn#uJgadHFBHpZj;3NP_fPclGPxAUQ===|_Kb6Bi2_Fx63#0eg{iFRzIxNBy z${!CEB@G$)`797%0UMt^)xcxF0pg;>+dCdgO}${yL$Cpk=kf6IP3(xc5jxoPUd?Ym8ppS$~WM#O3-$-a7ZXWjWOwNBX#B0;zc^uT$h9jW`Mt>rq z84j3huV{nUzZmP0hq;PVdw5=0MDnuvyUsyhDro6nPQQft*~9C2Ca0fWLuDU zG@Ppcv_GWFI}+m7!^uT`OL4sF;eB>}MOcpO^`*@t{j%%3-l*?Cj<94508(};J#PY> zBKE@t{OTuJab8I>o8t*X|6C04y!=HR#`Ag(duNwQ7;y;jX+>o={!$Oa32L;@^}+GH z5&d`CORZXeaeb!!8tpt8&a?3Krcc^u5$(SX`##S3C;Pq{d(1YT&-I5U8_)N!zRM*4 ztmK9DR3;6Uf*%iMbNk7*>~%7SCtLQKf$7S&&*Pyp9CY~7xqf0K3ldpzVb>>KA3IGkkLyI~J>KF_pApHHt3&$}J= zkK+sh&al*H)BcW!d_L2@oc36RaKhHlNXXR3Hj)@wc+?Y~8-u z`?T+A#`Ep|P_E;7xG$693nicaP-^Bz4W{$5B(G_fedlsG*RpTYZ`+<}e$(N^_MPU~ zKY)E}`w_73X{P-Z^l_$gk;U%WqHLu-M-87OAWYU z_96)j`!1SSV&gv&`=6U%xaX8PiZ_;nwzG-=D!-?%X%dtNP`_}gRVBgbC`zy{f?K_L}lXcbheVxO; zv$FL1j)zJT*ta>KBA&qVQP=Oqke+Ve6>tbTKE^%}5a@WPeUzgNKg5fVP5Z9Gf*eK+_}ppV^Bnf=v_D#%$;ACfrDHu^<%rLn z_Fe3-Z__^G`47{+6UFC=?0arCJswv2i}{-r63L3sv*>t*f)SsW7vT|&7N0Mlx5$92 ziYp{6kqjbB-VTSdK~B19}iu3T>-eYe{Yp_9l`lweLo5Ndk%l#wC7lRGvo1jIp%!qv{$j4%bDW*tTgQP zu~4a3-|speD$X(IU#ES>#t)`_mgMO3>tmrZFWHwZPjkM7{fXxRw*93z?C*x6<@SCS z?9n?92Ou^#FhBO#}LoAYZVw8Rn5Ryp>YV&mFi?0j`F!<}?|)pIuYR}ZtKZrE4>eyI`yJb4d6Fo0abRH%r z;dvMg5R$dOSe~t&Z$Ug*F}uv(&j7r_{urKp`8=dDg?W^2jKTiA2aP05n8wUXv;3~(T zcO|l~3WxnT?Kj)8zwh)9iR^1)`$>GBH1<4-zP`fqGyZ!&&A!TfCH8&=?5RAX*nlgt z)%g=)U-R_yD<6q{8T%OlUnL)$4=X-|eZ6AOSKPk(Z5Ypg9s3!%j{S^8_Lbv!K0C!R ze}xiD4xrngIe*~1ceaUtzxFjR*L?nEK3~Im)oc@= z_&m5df0g7;^!)31C?ng%C$?X6{>sHCMs)d@=gH~!EhgTy-~Q0FnDZbQa3uRYh*yJQ zPqTeEH>o{;o|jf^z$F>#e2B2G+-&Xq2<&UxhxUAo{9E4?b{*#r=Q&lpXzXwF`o#OM zuHN3T+|tf_JRxeA@AJfYPVM(@G@s!-^-0~{oaa4!jP!>m@cVcuozGwC)_D)}JT?6u zLigXW-vsA9($n?%48IpkGhrM@()5GQgXr*M!EAb=vGu%x-bZJi_n_As>-uHRf9Csz z==V=_8ppQp*xwKMGR^v8{ewCGrDaZRfAniDnfQD+-*LVp7xruS5B7%hPI`Zw_7*!I zoA#EOsoT?Ip*+~HU4JBmZ}({PFP-mj*i(wbp7MS8MTKTh*=hI%l?LZ#;6yfu#qXQ) zGI7FF;bBi7=zNFqK8QH)r$zmK%U%}GJ@2U9n zZS8#!!hXGMC(dhStMl2H?e`($_f^jQ8RPet@Jr7Hg>?DvDPJyNIr4qt`2A!GLwk6g zIDeHfqxX~J{W2MSjM4if@qUeTdVlh3cz>{{0YNwP>9r$8%a>o?rh(H@%zpC z2fjsxk#JrN=T~OX@2x1^k3al0JtyP|aCs0vsNdaJ!Kw*Q?~?|9MIn#d576Tl_{vaa zMH9rq=>5*XgWhv9Al@JFkMb;FmB+hPW&hmNKb(&P-g6H43RxfI!M%}^MCy z6@CN0p+cOP5qNj-CYv7hSB9!90$TZhz@jh(8RkR-C^utL6-$48cIECQqsS3WBg8GxR`U}91x1tGn)gN~-MZ*)n$rarO zK2^gn0p3>;Ht^{hp6Jsm`ZYZGsmTK@`b+#w(#mf^e={mhXyq$I6%}V}nEKh~UxWF0DSqmYOohLtB=p_*y(IL~pVJM9JUYn}ql7WT^Rr_#0`z_912WzT@0IE?q#><5ATUVvd=u=jj?KY)_&GQLLu zdx3q}_62<8ZcmhTqD3mUCzHQ#{Tyy9$(~?;pE?5TB?BU*I++O==MB4OdpLYCliuIs z;rzDXdj=PmcnKzJ?rwRLfB%4BvaRlxQT~1(U$7brWS+e2;uhu$-jpQYp9VO~>kHQ4 zh1&`brxC0YzsF}`;2#CNGD+-@5lr~XI11{G6aSD(mO=JtvqPmGv_Q z`rS_uP_=k@N{2#&OFEJkEFS&n4*9guDRaTNe4U7WM{bOZi7;n$7 zAQZ2UpKc0($AYyXg))d`t41* zk%M6`d-iqc-_22pXEeTmZ|37qWuPC1zQv3m>05FFeG~QD@<>T;Le;ju$?r*6`j(Pn z>RU<*>07d`Zz)cFyP5S{`o{f5So(*SQZ5f{Flq9r@W~U;Bo8<2H&xmfw`cMfWG~!* zp#O7%6;WAU*<)0A1NO?zsc;fWzV0Y?^X0G`#8Y$f!=>) z+sBG@Izf*l?PGbGJdw*`KHv09efmNs#@ClF?1jdg>?;)?SfC`}Il&cCS)bIG7oRIqSg%&=3G(NWeBu$d`E%hM->sOu#_HzGxLd7e1 zjZH8mi}VjkmR~tZk0&TTF#SamLP43hzTJ?i#~Z%jjaimI=lYWomGw*gOX2$=yatX> zPS)cK(npe~9xq`-0x*x4csxbzcelLF<0r&7NuqsZ^7_0PqJ0IYr0DUC;on9`KQlD_ zr1eu~0{u+k{jtY8E3*kr+4{*}N2cj#R+g!sSy`l?nU;QLWRZSmi1^ZtcqJo+h2?n9 ziW9FG{JM)mAJo4&`Hk<7HGcVgr}LZ4&PaYI$B9=EkE?jFa5nhONFd)Nu5Y${CuNxO zb=v==Nv3=!O)~92c@oJtnaj6;;=$x3c0uy%V)g%DF|d^6dgg8?zex%B70;(^ev^`# zP@Am}PJWY;On#G+9Q-CZ`Na!y-NIkFVE^!bXVm;L8Sz5uWa*#D|B^qREaCz3$B)9k zC+CnoAxZm(^_)@mk5>-|b8^IdK;>!vsO0bORONla6+V4FuBj99QFy~Y@%T%dj|wTi&2h+^o(Di)^8Kkb za=y3qaY{}T;jQ&lj(8upHGfXA<;B-iQ>IY=b1eVln_|zO+FP|}0%s(XmN4Y-n^}m=8y!WkS6)W()U~Vti&mnyy{gd+v>YF-En;&5>lcteA zG8r%3x1Qm16DK6S&{xtwrEk9Av|Q3xRi4(@T>mJGkFSU)2^8?u=u{2}81pUvegGh; ze@RLDe2V!td74&!PH@sx!Jne{^7jqf?Th({lc4>Ya?O8|{%zIbTh8B9xgRa+tErn5 zrTV$PApXk@O-rKj;XH77MY1)XFFz{$&pA1s7im0mClNlE7(o3b@9DVhZh6;3`w5>q zg5I4|<0KF3C`hOScLzAQz?Gvvme((jeX9)ikmi}+@OyO??9BbdEKS8|A<3HXc`cL`?eq?(d z#M3?>$wT4i1XEK5f5_qEk(Q?WM_({^7U{F#pT7^9KJN#9`Th~)3F-^BJ=N4r=l+L} zA0O{L+uvaPYFcL4@hRGa{?4L!3JLusd!0F*#*aQZK>Tofy`aV`lhqWO@pw(S&}ZeZ zXnbbk`|DzSgnX$k+g_5cv+eCsw4aQpA3Sk+)A;1!`|vtFzF?q@&-8rT-m{Y7FX#am zB|#5)PsR7ymA(6d8Q1CdO!_zjg;hKWd&diyDTM|70#>mcZ#2}~&*SB3|Blu}!k#z@ zhn@B<=6@SMEo$0ttbObDjQu#TsUOp(aa*Da`FN!i=<#_?oo3&_-`Bw^@O^i$;@_}$ z(uZ;_KH>aN)#C9-F}{Ld+x~O8{c|3mKe?gl-2OR_)E?wDjWx*jJQ&Y{0-BGNJfTls zOP@f`?fL1~8z((m8{CBor+7r=x zo$&_g*K~Hlpr`o2w*P5<;eSXUmVh7M9N~Y2e(-qfva|=xU+_6ius1e3%n1&!+wk8e292}<{Og6FU5X)3(0@d z_0oPUeet7&(idMa6ED=|iQ~DyWChm#{G4Ehv3|w;%I7~$f%a*97V`Qqe?gywKMzQL z(>Z<)$BX^^Il;`RY){(jbiB`t63H9!(t4V|x+T4f(-(4jwg2zEUie2bKG?s*(_cjo zd2#=ELeeKe-WeIXzaW0+^7_0rSIrVb6@@2Y;J8 zU-%#Jhv%!#`GUrCDm!QOucl7C-%8ZC1^k-+ux9HFdwoax!0m$}=)wN{#Uw7DBHkbF zfBE|0dJZdlG5sUSH;@&#)R>L;jbY-XZ!*F7TWOx zjjz+*)1$I|VefUCz4Q8}YVimk?`gKZ^YKiV{x;U$LC@{YujE1Vhi3oJP<)Tk@Jo5) z91-dF4AvyaPuO1x#LxXR{EzS#{RVzM$3s5|5BVoPWdBVdTs_>x4_$RS5lAhxG^kuqzko*^+xbp7=FXpiDzZCx@ zvkQ`5_>WSKxAzN|aG3i??oY}%%>6ssV@tI7#pMfn%4~Zleef=|?R^Qx(_C*sf68t9 zEdYLE`;WEPIVVVOhnI=#gU9DFU~j6!->2#R9_vZ|eFshg`kF%Vx@_*B zx&QY>sXjHo8~P#j2M*FF^v7lD3&y|3p&w+AIH4lS9g@#?d+hB7`hMd}N1_5xJ)-kAeefAn z7nR5SjPo2^p#1yqh@aejZeaj~e{lZw0+j@^&_wrKQd?Dq#0eVL?({7m|L&|jRsOhivF=%DuF(l1I(ADD=KXJUH# z0X4PX=H&lo@c*gt>7hTy_tAtr9N$CK&kIIT-`V$HpCMnpK9cY7`>)TCx3fO%R*W}Z zvS5pVCLA&vD1|&lA%(jHeGvAn!8v=i|vowa+AdkyD=v zKwpuV-Zv4wkDW-=e)dH4mlM+)zrWJ`0rCGL`%S!&-E+L``iq9>iR(Z zoMBJA-}(&sI_v9YXWnmphP<8iRkN?X-}(&sJL~hZv+uV)V?3PoJwJYZSEbK?%)T`q zKZ@Rvw^N^XvY|xuhJ2m$e)io&^oBg0^uvki4f#3g&#-65YajD(;@^Wf$0Nj^*XsKi z=1&nnV^x6jWJ`N-{u7BG8i&0!94T`5^72pf`Po^X;yIoup5x{7jPmtI-evEq_J$ns z=I;S4I)CloYZAC7fol@DCV^`bxF&&X61XOT|1Js8vj^-a4t*J$Xv0ro<7(vYzwCT+ zYc%>g>%09=+${XgJ8!-BZuC8NIePh?(VO2bc&~9Ziu>r#@!#cwf-BLNqS1HW`Ll0Q z>iXt4Zw&wPm9cyN{P5<$FaG0aJAQI#(#dz&E&1pT2$WlliHqyl21qUi9_G=-Y3-^?LNmFMk&O*;DUbiQW@^`zN@* z6OCSZ=k@oZ(VxB^J$yL&);mA_S@fysPp=$)D;mA$|Bn9bPFyGb&*(j0|INSs^rv^f z{txI2{ylv7@ayz!-IeZ58|6`cEf4l>*p=Ud1?Zj< z;cMGinybE3>ZQo9ZS|(npL_GC%12pS6Rj-29UY3YfGdp+i88di9TzVZiyjr!*58>% z_1CUu)wXWCT(lGKTAzV71EMng>#AqavcJ(_lc!r_ zi#eLZzdw=bTm3d;VQR*m9m#QNaAdLxyB%=EQQqw`%SRlr+wH(RV7JGCcff9MB&V)8 zBYC_(op*P#T~^PJBstoU<@7S{6hW)!MdIYSeR0akJmOKOcc(hqb-?a42R;s;Mt$H% zhlxKOCf*E~`gwibRL%S4xJDy6G2U^OVI4lQ4(quRS=^WMg<=x!y2$qHbbCo7WU zw$9m@_@qD*izK-XyitFc>K{_|bML0tpBzYL5imYp{V9PI7D<_?elOK8c9HO`l&pK! zt7%3beCS>T!WyrQ)IchWq)yaECbgj?z&)hihqOQ%i(qiat3N%E&LZg()Q^`UU<}kr zFz!M1`ZEF}E29^$L_zC(StAl3yF*@i*029Gv zI$ryh4qjdRu{vnBKfVrX_3Qn&bWp=j(0|fF(+14;$Jalte!YE5|1`YWej*=e;2H6! z*ROma78@!VcxxWC@vatzXwObODr|A5*(;i(d_^;n92$rZYqjISY6S|Qju6VGY&@-I zYX@zU3hiZ-o zg?z`S7xEpSUdVTRdLiHO>1C`ju6#dg`6U{^{{!_`%=P2PW8B9~;48@&@^xiv8~m9a z1J;!!C-Xn7gK9G}yz$m-wr}FTD^>X}wV)*$yA18-Ac(V$pb8tRYpp1!KX+^H)VEo7 z13XfiI~g&3IBh5mnxcHH#Nz-dg83BS8I&js0P!bru@jSQ|EH1=`v?n$Dz(4F}B)ClPY>7IuF zVA7yy>m!ne-J3{UX-wjp9?lrbh>Le;^4@Ywh)0h-jWm)5S(1&{r4jbNU)`YC@1v1M zBt!I;xQ%NmlSFUHZ^Xm9Gu_l%hDjr!AT@$|@?e@rQi>v5gE9S@#R3byUry^z*2uBX+d5RV?)ewGw${0Ec5 zxSn>rUOV$Al7i*|<4PeeUXwz6ye$O{|G}g%u7`E@c071ODa6xwO&8*NPF)J|=rvu? z=-;mt%ze;t<67D4YhKm#C#LlN|LRpuDSY@|6)}o8HonySiHzuI?yKCPs?H5aq|kyC(@LnsKQ{pK3+<8vO^8!iVow z;q^!g44X2V*E4#-@g#+~UNcQdX}owA5AL|GnenhLg+Nl`IK#Av58zdk5QBQv>VJY* zHCZn=eyr*i(hH{yrD(CFEj_n4VoBGIHElekLpl~t9ZHQ)pB%At(5Ba=7k`&PtZnSQ z1co(%SOHw1Qw#E$9kvy)0#JJkJFx$w49yn--_Pm|FS{v4midq zM*eOt|FG4*ABU~@`10|)YI;ix8Vq_*!uIKHml4;HC4clpYyWEL?f#$9>5nGVzlhMk zfYttm--@1D^!WOQ&HvH(@_YaC9kRwJfcjxSdCGn!3He$0Zh`lkc-y{C#Kni{ARSL- z9$tU3NpH80Q(1A^#}O2j1m4j;jyK3O6t{k6;qf!>IQ8?>hrEVPw?i@%`+?O%>t zzuiASesdpdKai?S4Fd0wFSG1BEbz|$S@<&oUuNoG*uq~Bct`(L>-4ILc>E1n1rQF! zlfPAex4@6f-wA;qm%np9{e+_nCzk-5XCm!r=3Y!T2q+fPQ+A31U8BT@YoQM(CME1* zuG&@~Khw$BZSWrQOlKI5Hw*wto#~9h4{ue|(bHOY zBD_BaZ|=j!@@L{>`7`mc{F(Sz{!DxVc& ztfE9OzZ7G%7mkFBoz3|oXg-%P+2n9|I!<{Eum3@!H!M=>78j=4Z;&uuK8p4($okL8 z{sts0F5V?RM&~P6;m?b=AgVc$?$!K6_v#F$H%GhQ<7L!U?+P!wkFLA@?7_b%{L

    J8QrT*ck&ZW+x4JK09H+^VsJM_0LKk`-KcYV%Oj~tlNP7Y@Y!a;C;G!`Ppoz0bj=&3^<>yGvHY)V8Ao+ zeqVOE){-yYXKKKCEZczb{zC)KWtZJL{WNyLfbo7p1D?X31#FLZ4m)e$C({QWHGZ-g z77bc?d0tR_#z}=Ti<-F%79Z?nNdEO`2pMV zOTzo&br_L|0ekR$0e0EtR{BpNj1E)Gqr)&K0dMhgn{?G;0N^TIIG*pp*9hGk-OSVI zae0?!uyao%g!?XI0sMIvdR;076FU;^4W>K0v7eMLYT&+7GIJS8mZ1OEF0FX{tc!lJ%H*6^IbEB=Ng+#uU4`+>l#{+^Jq zPvWzm7kDMVu!NQTE=$-i%Xj}!uP>inmawQVpLM^W*9W|WMSZ~kNZ>_%`Rsy(6@Lu~ zl63tWlCa{h?8gGH_&XtC#h)M1h^>D)%u_W9A5koVLc=HF622avY(nUE=0Uv59)Aqu zpZy)=W1PKsZ7sYf{m=ir_0 zkmWh*QTZpwDX;MF_qqTC9xE!esaNuRDHJ=K!G)mwWGzSB{*9aR;cy-4d&sAk)~Wug z$hZi;obr`?0+KExiRyu`bEImX&);rsV0I!uC)?&H`~MZ_T(CYzE2*?;3G)3XV)<-9 z{$efPFKK(nsYk2&x4Na_g$QC&K<2cFF107@C-* zUIM>Q5>Od>CZX6-lG_P^I_sp$?w3Qc&Vh7*jPy>jXs~fk4*e5UnU>V}sVve@q7OLO zz-P^Q@{0@CY~tNXzh2^m)cY=5# z`jD)E=&78Op8N=j0On<+Ful|tDg3aw5`8Mc0<``e1~JvS%F2;9N^z)BnYe$}V9F+c zMf6m^ktBVSk?Bw};J$?Vi2piAIVR~tn*GHaKWwklw_a%cvh^y$e~ApbV?(KzjId_@#d`cLgsa)^($qkWF+FL#G}WjQ+ZPQ zOYM_Oa?)2K1G5n=VGQ{PE(?0UxLE%>hJ$Sb{ioyfze&=o@q?Q3@prZl@pGA$jap3@ zH!6NmS4B^J&{+I7bWmBQ?9*tB@}<41tKJ^*s_eo^r{E^3AUYk-QueG0%B!+3byakG zYqxE$-`iR%(yC2$<(-FGm(_LdTDI->J8C;`Yu?dzXxGlIOIy2k+`hl^)`QKpt2V9L zUbnBcYi;lDom)FwySLrex&ClnTjkb+wd)UW*|}81?eFbw>DmUmts851?YRB+?rrs* zYj3%I&+c0f-ch@6&+hu|UAuN}?`hk>Y3p~EZ|`kgx3#i!!xpsNyJy#q%0n&7w(e`I z@4U06d)uZp<(rqb)ra=AF57fy*Rt(pySG)AwePG$8|^5+`}W;CD(`Apb_=z)bXV8j zHFwl*Ue;RPe0y_km(^}_?b3s*%lF>eT(?=4SzFOrzjwpF?ycK8*OlLXOa0!>&9y5J zuBqR9TicEe2kX`!UfsTKb60{gb?v*i9$dAn8GYQkvUAnGgR6J!ZD_8mJGf@q_KxP- zZJX+r?c38OoV!yJy#qap@|gJlE~oxnU3V=TI|rCZ5bT9gut+*j(1ywS8YpT-mPLw6=cp(%toI z4(;mdJhZzWf6$BF>o#<^b!|Taom_wG((QM(t=rJKYe%`J6WjMU@2K3<9Jfw;yjJbl z+TGH*vTWCm4L!RPmF?nX4>s@Ex_|eM!>rzLu9QZ%@Eq8uD(;$A4Mn_EEc`wjFB^k^hfp3s!#*pufxN?i^2dta8h!9O=c51bx{q+8#$2 zt@^l}dzU4Ub3K>y&b3!pZpYQuS2v!nsPUEUZoR5Mmgq+)f?{G)lqSi3fCIDIy-`m``cSu+xIMA zSy5hARk6HcY1Puo_R7i?D=U^(R4p%STe+-JlxywS)7;wL*m|h9v$?0Iv1?glSEzAs zd)J+fx3wSWIT+mA)!ucm0~P470ZfQAN;-gws>bC@D=Hc*%2qCKEN`rAye)XBUHr4Q zxwEIeacQHt@_cby)+qj4+E~>nJA8NH{`TO3_Re^HkQmXGP0igshdb`NtFzsLw+Xms zX?aykin$Z|^wJTh-Xx)!ew}V6dyX*Q#fAb8qun zo>jN&wl(NX%fV1%TSpgF+(>^`Ne2gwWra8R@BZW`(U+yAVgafaH?F(3;+gzxSAIqT z-{n*5DC%~R>^of+;ZJ`$|2bN~P`WMSy1-}Lf9(B}bzy{&_e- zd0LnLu~vSn+QjY7xUR{nKkaiFk7xdA_Mhea`P9$n{zYEWx|vDq#l{Bl_D>r9x75Py zW0@CR;KxL>X6Se=>u8TaSCgkPlLEtzQejT*k#K~H>j>Ugg#&@@56P=x5d z^jpx$+G8$5sLRX#lkc-TkvK8w%#vdtn@=pcw0-4sV}kT4^vk8~`lxjqVLfQsneOLR z$L5Rjv?4W=C?A_or94`@ib|B{D^!)muOeUD#F1E$Pdid(!lMlA6WDT4{i>4r%8rU8 z**~MG%F}E581ku}fXr0(lQ~ZL4hTA4J=4d@ziFKOZR6y>hp3Iy)wrBj{gwI3PX9^vD_ic*7JEfScr?fB56F*6s{DtLKUwW?qv!)tW%^M+<@ZbY zRmsm&xKe&NO=LVH^M5V#*GWGoo3;8k1@$RiBa*3-Zki*%Oy+J}+NPm)_59l)2TD}G;*DU^b4t#7_R0L&j(WPadi1A2{aQZ% z)~=|QPM+{Y?ds*fHBP<{<4cLk>;1(?*m!dCStawG{kAFa|8TE1TXWi4MXUoG|9S%0sV zua`fkz4LlZ)4A)*4Fl(9#-0df8JM>2M%=!W;@z@zo?!NxTm+db-yUJ7P8vb z-do$-8|-K~)Z0$EqC5zV+_vV`?RR#xwj23EvhDjC_XL}}+OLAeJ!r2NJT@Mn@*6j9 z-?UnYcGJPOL!D^;Dp!9=2Y;5Ct40 zzpbOQz3u(;z>3dp59<@a|^CVy7)DCunns^R?7bsN-RciBYH>+ zzN_ZxcYmdQ^&0RyxQ$RMM!K}Mr+4Xc=nQQp^k4(!cX(4lc>WLe@84tIks+5+(FvO9 zkv=Ws9EFtAfdcuy1m)?m0wDwR57It8xB#r?MUKQj^YKR^Spa`}3e6KORmpgb!gQJ< zRSckFvA+SHO3G=>s^W}n*V&$e)s@as5k1wdry)tN*4GNB+E;KGHCLsB)#HW)wvkWh?Z22xc)DHyxiEb_epwZ|1U^-TBqvC$^X5;#~Z&h mC4%s>B&KdqD5~hy_&tS&ooy4z*+M}-Bn3kd{f690|GxnbwbHr( literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kec7fea04a276529bce9626a6fdcab07a.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kec7fea04a276529bce9626a6fdcab07a.o new file mode 100644 index 0000000000000000000000000000000000000000..52938977af5f36902511ffaf187b4a4ec4bd7880 GIT binary patch literal 89896 zcmeFa3tU~*nLfPEC2)WfAPFJl%7Gvugb+e5k{IKGTp-aTG|`0Q-xdgvfFTJV5ZzHa z9#2|Ft)rOstJ;~i`L|9RGNyHA`fEd{Gt$d9wbt6yU#In(ODAKk600^t9R;iMJ?~m; z@4ePK=Ws}4`_1(4R_mxq0wxyTM>tmK1kXqX3w zSr9QpEV6UR*Z9-?knf-953%|S7<=$}U;de2;O_-re(5i9e-VE#;WLuv*m3Fp&h6hE zcr$~Iy|KqT@cItlz&k$n;D7npnNvONLB#zeo55CnH9tJ~#s>D|fA_Kn3xbiI7ks&A zp7-TYUY}&s*{Yv;!-K;}=k@r0+=#Ll2k(FO%zgJhy9D=FK7Q}B|JxfLL)?D+J=pQ` zmq*S#eRkx~YhNCD4EM#`el&9PYd;)Wf_o?KTON3IWb2nNj$DuX4Y;RYb@jmaU;Fs8 zH^2DtXM;;u4|JS;Y~;+;uZ(PY=7%FM|LW5=r> zr$;)V^UH@`M4YMtveniv|Jk$mlsq<4dLPOve|ltV$qz=pKm6**y=R6;(iX29xOn|t z&z7G3vu7_}kFv_YIC637k4Cnpd=L7(GV=107ZK;$fh{S2ihS;Q_TtCSjFbkyJTma~ z524pHsNbt2rQ3duIO_+tJn(1F4%~N!%P2g}lRvuU%lGVD{K{W-r`?%3=YIIuOiy_1f^X&-k~MkRD#srn6aFw0{w92)$-`DX37_zINH+7# z^WMPqlLY_wC^#uIp^&J}q-q=$EKMaq(bNhEstNt9%KVSEGN^Kt{f3$ru z3}5^@eD!@_{NwV)$OOK4Dv2+iGJKJI)b__A!yj$`oyaFIE*w03al`T-zjM21;O(Q{ zfp<2r2QThf?0@T^Z(y|14?muQel%s^x?kRQ@;C79)8DBc41_}q2LCoP$bK<;)i)1? z$G&z=cxPWu;qgQx#JJk~S^ zpP%uJo&MYE!S?$R_OtL<`?rC=0DKGZkAS}h{0ZO>1Ahfqwyf1o*FT|1j>m5MGM#KL_3ld_8a>@W+A6fj0n82fn># zu>D=%AoLk)e+&3(+~2@`1@0rbFT?#c+?U|~GVY6Te-ZbExId43KJGuqeJ<|*jQcFy ze~SAI+@Hcd3-^D(Jp=c@!95lCFXHaQ{qwkwLyu44{#)Gd#r;j({{;8n;C>tKzsCIt z?!Ur)KkhH#{$bpA!5+7u9!qfV#{CDV*LK`nAYTA^Bkpr>KZ<*$XKefg`>}<-vB_Kh zV5@iegKf`OJb2*S6%QW#e8qz`2p^xz2LA(N`1lMqIPPPEqZw?F`c0eq{P>Uz;-4^z-PotnRe`G{*hVg;Q6t4?XeAroWl;4(ZtBUGiY-x0gQH_W7j` z9*8b|@ZkQX4;DZ_TgPiiSJDysk)FSh;|ACBYtZ3uNQdCr=6}HFZ_wwAv(5jA&)=rc z>1Ugt!RM#wa~jukcxq(&t^1~o9slr@v5xIi-XXo%`BPWjx*K}#v~?#d_GB^Q_ePj7N0*&pQi|Y zW=Brm_J)6~<265Y8J0G>a8=Wvz-AwM;$uxCerczxpwlAgvc_Q6#0c1zHXzd^>Yk@2f#eD+{zHT!WR^gmQy zGq^RIjV-}^>Cp7*u_c&8T|__c$hShCBjts0_c}!ULob{hlz%&p{EL)+hDiTR z9QlW<$I|bF-1M9M2>0Z~h99B$al&t-_%mbU{|Uv96Miqn&y9`$DT*H_{PPrlR&4w) zQv5jKe?#%}V&ne<#g7yI6vdw%8~>*iKTh~RQ~Wux@qbS7QVpi{dYcjsNb3!Sp+MxYjSE!#`-P*W@jFu*8%9O%K+3 zG!{L9vH$fQn=z-|gt_g;fwvBBz?`>c;GNs62QDuBUMH>7#!vV#7Y-mi4f1rv%Ru}f z=D<@2E@Hknj5+B`cd)@1uVG`q!MyYJjOxMv!aVei6XCI69>v_?6H^A>#Qb#hx8cDt z%vb*-BYp8(n7_XL>+J*Ye4%=9cv0Hoi$4mF4P(ys?!Sb`{^J+nvFCqKJ@^aExnB51 z_24ft7yIR#44*v+V`DG=rh4$j-*^U}|AA-lCCu9{%|-d+Z0tWK?|i{C_!{Pkzur(Y`1Y?e2mT{t>*8O0V$;A|-(%1-4E@7n zV<(X39pS;Z4sIP7{Vf}O`{)+fqHbXLOTK~O6*YqweKmt`KEcLD8?PUD;{+S~uO83f z8+&dTcs;{2_?xe@vDbHO8~F7GFUs+tOz+?;Ssv)&9eg>}GdA*Z($Oc?rb@{si#Vz$?6i7r(G!;NmRL z*zk|o*u{lg7GL}X;`{>hpc6jY+u{2_0fw~*#_u$Y+ti-#?s#H=!mxHZlXNf1BmAU2 z1$U%>cgMH1wL9%Esrj+V#xDvFQ27+Uz!TEiZUfKvPo4^oWh3vO`@(~dpI9*JsjgleNDE`VG=H=t z4PmM47N6dbf35^^PW$GK*75uN(JckliwC|l@7(F{%ssaS@!Ki>mOJH27f zxw|Rs#O%?#Ii7v)G{F3g(4v(EVF?}@T z3IBZ>!YS>*iD{#!H%vo*bv(Z_C$dM=5sq@PJ_$jOElxSZw|Xgj?B(m=J6C(gN)T`O zpm#7(w3X{N+#=+|%cN`%$*yVy6 zjj$@*uf{!4w0$5KpI6~8P_$zp2lqw%bNxUH?pN{89~kiCzL0<3IpD<|K7;tX2FQOZ z@z-8_hRLpHJ;u{Kc$~3_R-l;OJj&Xgv3&?|kswyK}vRzxu2Efmi=_ z!N9NoZvMc3erMjm8$Zh(coV!H1y9Fzc?RG55*z#N2G8Kzp94?9`{70CH!^TvLHw;A z8-9rRTQfMk5E%W@@B_fLz%zict{whU;61=TV61zG?*MKA{tZJL;r=V!u|6K|LHMn} z{{nm+@F50Yu0J>Y0|r0E{X4jys2?3}MEHgeoE!dI!XFqN-a~ljx#2Gm-Z?tFgYd3% z=vRPujSklmzVY1f3Bosy4qpfCyJ>V7{rlNUjWy!kGKik--t121Nws+^b6JKpZ`(oliu$y-tRBo?=Rl(FOu~am>==0 z3IAdq#Q*4@`d4rIhKFUd^F@z;)$34O z_mQ^F9#)UQA`*Bp6Y~af_j;Lcs?X!~d6+)~({p+GG8y(>*d5}zVYMyrvVRad%4p$* zqPU@XQqMyyCCsYW6lgSuVwU%`_w;u6Ht)wL`+oee!2I6W(|)AYV&zAU_O_KDxuv}K zP{)x2M_t@Hqo>|o=8?&ia7qIJxG z=5j3rG;3+@>^#`r+||xW*VTQr_b4%*qn4vbdU~6W^j7>4xNl2y*WtGAmOqMSMzis{ zm~F`eW!K3YP!0GaXhXX?y3g{^KN3 zpEfOwb^W5oM$ADmm2d8BtY`#kIo8_T*mHP!h1J;GaY8+$R< zHMSh>Y~0_5*=u8K8^*t*Ep4rhm})n6xAh*wlAyJ5|M5m1&eY78PcHLgPy|^v_aYw@ zbnNhv9<#9br%wOvRvcPaOA}i&VDeL^|M_b#dtljhxb1NH(E|s1+IlNkW8;xqzz{M0 z-?eSiwniZV>6$w`ntS3zu<6&;+}+#J+i~;=pOrXyQ0|zQ{ZY*|CmL`UL+HaUfO)Oq z;VEV?`w$PG8WaB4UdkscCj5H{_p+(1;)_C5A!8KzM0k@756Rt-AXM@39SuFJbv!&> z4+LsJ`1EX7pU9U)lw7D@fRAATog9}TJaj6{372A4&*Qt{At~sU`PdoIUJG6KAl!$2 zAM+Z=#rJI>pwD!nK7Dl03f>usl(4|~osp6N|0v>=nVCpz+swZX0V1uxjv5RXI!R%ykf)s|!`kKDF4t}=3a0$u>++tH<(WP}!PxP_ zVRFXKW3z1g?kyX)REIVk?e2mviHqEY_R&`3?*IjL^;+Qj28^>I+I8yaY$kGtdXFCM z>?sSO_iV!+QE2&!@|9(w-E=EowT9Ik>*#C^^&Ic%Z95z~e5|K8)Y^8S<49ZR#vMCu z*wwgu)6QL6c5H9l)Ntdb?Hf03WJ`C1R)m%|cek{MdJZ=(ueiE7grZ4T+IPb<2t(IH z>@I}+mjxKkOffnj@1GW6a2mH%Qziah}8xiJK(umpCNxDT&)9eq7=biH9Wa zlK8yD6%t>N*pm2?#8nc*yLo*g62tpB4oeJnaD2DKU=PRr5{D&zSmJt#Pf0v9Q^cmFPwBJg3QYCo7y~8cNA)9IF<)S+58?U+0+arP z`|}01^*I$1*w*J1CMcAjtx3Z@D72iY`jb0u*CIy1Qz-f zvh(`{7Wx#j`VR{%^eJRR%>oO33Ry*)z(Suwb_x>|DqrYR$g(>H7W%+=T>=Yz3fZL| zfrUOP?9{CS3w_YL+%B-t2l77=Sm=ZHj|wdGNnw$X2`uzUVVCX}Sm=|&hVB#iVbnhb z?av+%7%Bt9{@_08U&rl({3NcIcu3*~iP@iv_)QXrC2qIVOWb9rm)Nq?|Aj~&vC~U@ zx1C;MTc06`ZGG5hM0#7Fu*9}L5s7Vmh9tK2$^NWJ-_OenuqKI_#E(lHkU0BuB7Qc< znXF6VJc-Xs9O8I7t2ia%mvFp*^-EmAafn@#xJu&s|1IK&B|asw&|0Nds6td8xsy->KOJbo9{7GV=5B%{jRee%e zlf*)w6!y5pLLd0w7gc>yn6GLk^Uo_{T@dwOg|Vm@V^SshNBC_Q{`%?fA*_$!hyLs! zf5`2DztW$7fgk^3T%0}X^QQ+`CO$`e_`F(%AHrkm!}vQ$u~9BAy1y&$@5uYx^1f0; zV_w9ed9S?&_%0>FWry2qk&PnQ3761$if+fZ!%roMZ_D57d-QhCIQw_xN(+JKv)Jt8 z7>wvDz+ZVL`_6H)7%ub~Y=0(;TClG$9b?aAK`!H?UD)R{*@NA|OvGU0L-%K~Czr8) zYih6#emj0)Jd5)0xBS6}H5>?@Qt)i6czg=d&&-9PpU-EHSTn}SX2sM% zUG*bAd4Qr&+9a_GuuxB0lzV7JKa0lyTMpn2z{` znQYT>NGO39a-?5o@#YutfgiHcQzAfI{}=Mv;Bk8yIhVMyMap#++|?uxYYEPfvXH0F>u(xcN@6hzz-Yvl*C1T z_QY{mny_rIJouY)!N00)O#b)s9jtb--u|XPYU!VYf8TS1%+B_gREM7A&ryp`%E$|N zd9M$Sf%YdS0QN#oANdv-{O7^$luY6u=P&sEi(^yxeg(WCz=|@Nw~sFmC_eYU+G9B) zJ&R~1gFj{KFuzXP+oeLOkG z$iE1OmO>9w*Z6`|pBFr=&bGhgn`!Um zy~0$&Uc@)uo;RTU2+HTzi)6nkuKk|PVt1#Zym{Pa0rsK?{m+!~$YI2$^3k7a?TPyI zTQq>V{pEcYn?mi=6}ptco?XUjj|Bp}y~y6^&%WpAkNl9n_PA|*;SXm1wa0>b`i}_J zxkgC;0NF=e-gx=Vzuk@xEzsU3uAP0Js*6jMui5jTv)HzK?DRtZsFj+D=3(Prj29^X zv0MFa|9l`5_6VfXIBSm|(Y_3Ozlqm3+PBTXOEvt66$q%~L$q&+CXe(g$_!%5?{kh!3ambPTIOh6u zxyYaMRmq?0tK`r1Wa6Uh1&74USmhlSk`yaE| zpISNNasm^i_B;uBB%k67dB2sL6QT5czKH%7{vl_WobUZs9v^U|e|X8iborbdg^!+S z-&{jJFURHQ1IT~AAzzT=@^uH~`G!1{0SS! z^z(9DK8y4rr+$=vL5|Dk{22d4|CAGP_%71B{!HmH2}Osdmhbw%uwTTHi}XqDfdNI$ zFUB5@Jb^vX!KvwE?BU20*aH)CJAJ>ElY^05;;b0^P^~Wjtd<6cguQz}#cykRkFs|o!pnawW8K$?QJ|thL$cew)eJHjZuM5eY zTx@L=i2g-?5Vbbp!=>pw5cp9mGn4lpa(?irHDg8vgsBoo`?9q0Bic9Jp1|<(9mwDJpg%5kcHLq z%<*k@FxyS<>9^{F%R+JfzYO*cY5Ie|frz6Y z>7N&55mi5PJf9b2epR3R8D13U+HXFat(M=nK-1@|@Yf(UX+O0$^$&}KA-BH3i)PS- z-p#LAOW$vm1|qZoaP|vIFnV~D`1<^<-&!7|<)9rO$lbfD-?UvdXV=oL__eE>YwM7Pwl@p5TVU9r@mG2KefKs81=n2NXv6O zzPHcIHj_44*#Dva$eN|ppZq;HrvCGs@_p3)A(p4szaU7^&L87XkPW%@r}DUe%ldfx zeD-{e*N^7wU>se%{rj!;0lsPI)NdW~Q|ot~QNQY7SSjDz=jY$E!FV|MjWv+0_TM0$LqQH?2gwaAE|$$ z_QoaqOMSfVw`ynT>-&xNcBYKKNguEKHf#R;khQ=cLFU}Qp5px-rbn(lsl9@vGL1*r ze*s>w(Zsz}%7A9{_e@ym~`{k4c?a%h5 zYyA`NzcqWA^Ml$Ml)vE1GNj)q{M#MjPAq zXi)8OYGQj#SpVx_587>1{ll;a`llNZ1D9-Xb3T$kgYREE{oBOzkB6*We*~Fv`(ZyG zd`wrzKk|n>yS~`}1J0diW4r!MYW=$|tNu@Q7t(Nw3z>3$Q0DS_9`;D;59WMi(F}Ch zN`I-3KhZvY{uHF-z}R#^L2at zi1tl9|B73GW&EQ0m)P}{^)Iur-Tqx_{j)E#{*|=di0gN)zd8TAdIqWI)_>ypS={=U zpgxjcRR0RQzOw!+ZEV-yQtN*qAhw7T^oIujpXy#KyL(Z8?B6Q$1@wU^}{x@N{SWasGHH-WvNALfgx&U_Ki&$xeuVGo6WVcULKACrH@*so5tUxjAB*!t@8 zkEgmbX}H8i_9H%FI%Hqme#ZDq>nUw~*5_wrZ~1sxj?Zhf`54)U_9GJ8C(e9?>GlYc zipqSX%Er&fo1eWMIL_-c+5RY(hEH6!JyidQ$2Zeo^!b^&A4BVz8OYtOzrH_%^8x;d zEw|?*0logtd`$Gm*>#S8$ogyZv8OMy{&VDZ7u6s2wbc4gWSIY5hn0 zhi?7#{h+w@&(`aoXV+KOf8zPrUtDJWama&Tzia*V`B?1wf6)xu-*@Y8&d1`k|HSjL z?1|@NotIhvQgUcqzia*V`ItHXqWv+Vt6P74KP+zhPdp#Xo_IdCE=m3M{kEsND`od( z*Iy1VWD?E?>Mqswi8=q8F@t(Yw|>)<{jp^@-;?Q&IP!dqN7-y8FfN=|;{AV$v;Tng zH=loz3cSDLI5XSE@c(V1b3+6?MS3}(pJ$@+udDtNKcB?+Z^Q2X4W7RQ*>pLc(s`$} zjq>ptrO#sJ782VR>Z?7E2VSbmah}9{KL6OQ0rz=4&MU+ne=f-akYB$wJJ=6;+xB|Y znxpOC$Bwsiwf*`>t$gh~KotAS@<4(df6V>|=Ou7l-!8A;nm#82b!;5Pev#T=Mf+wj z+OfdJ^W*#VIj+7BS$KR7gNyXkzA47@{aN|c9&Y|Kjr?=P^I@4kAOFc9PI;IQ*Q@%b z^Vit8`K1~8Wf=NmzcSa=x8GWrPa8gN`T`?;p{5U>HwIaQT7K~yY{x3;uQJjvGRnt( zJhqhV^7^eM`BXU@(|HQr-lbakPQ!l+VHTq>|rKBfn*aKG;8>qw2FF zpEkH$eU@wKQGN>FVT*|NkM>n+`tkktc2%F%b0V5PtF-*0eODX$V1Ijos?Rm~>i#Iz zpLa;Sd^#Vo#>j82p%3kL-`T1c>b>cQTdDV-Tg0WfB0XSntzPH`CWe_{g&io=A-y8{I6WKpVB{b`>kJM=$qpD zBk2qOJLl>f?bG{XYCqTCNdFc2gl)cY|EpBX*ZW_xuj_xLKm6@UyFBt&y}zaMT>m2d zX5_p3Kjg3IuPW5?l>UkPYq@rQoXU6oh4hEN4cXv}FzvuQ}eT|Wy;%`*HjC^;!N%e!jRk`_*JzV~f{tNTn^*Q+u{B5-xKia3v z54b&Q7MkU|{z~^m#pUH_4N z;V>JxZhf#kqo@OEb)%ouFi|h%$tXIod z`e*K6vHf#M%MboJYL`#`uJ@1V&#p84Lm8h%d%67KSm_Uh-1R3Rcl|@i-S!i5x4nei zwZD+N_7if)|8d?)Ur*uu^x`JgK=qa9lkxo#?fI^@zJa|yiu2IH)0n^0dF{dlut%!< zyd3Ad`SU}X-_!HgR(gK;XrP1VPy173KVyGVO%MNms#`uTqV@Ute32jRk5PW5nK&Pr zI)%QG#Lq_}e(kYSZvJoN$E@ef^WN~sRJZ>nf0>y|_Os^?k678`e&FmJoUcIs^nDnb zk1_dun3Fi4hS=j%M#cGTI}S>9^B5x274gw|aD2an=NFXu;runn^XU6K$o~;5 zh2uPlqgFA)yGKP9kVmaWEF5NG9sqpOnw~v$hg<)X`2Nbcg6TZCx;{LK^WGN}`AK~L zMo&K--676rye;YzwPvt@D2?w2Jk(vBUW~$R`<$fnU0VEETKuTRy_u^k_7fK4^G7)a z2%oeTVEXTtchZ`J>6DA37QTJ2)HiD78@N>DP5*NHEWv*4M@6AL|4<6q&#g~^k)E?l z%^n>sK!FqJMdQ%UA{bXuF$VQ3`z_%GL+_ zBgN!jZhlv3`Mq?U<%sia**O2=VZbw~KZ8ed{PIWGoCPH~!s?^*#2yA*94h4)=fOEn zFD#Q7&ubvhEGl>97kk)L@(=vM|2+EnM(A(yMgKky?dK@`#ePn-&v_3JiXOMiw z#$D~Prvq(Vf8zgS_?fPq7sPrfmHI#1UKn3esef}Z#(S^ZzuSCEmwbc2!h9-UXJNQ? zG4UIZUj)C0iNEK#ww%AOQU8GRoXUO(<{R4nip}4EQ{IAfm0yooQ#rq_2|2Z#1m(Mr!PsZm3=)lza>;2P7E0m$j_5P>d znii}FJ#7C4zjJ9lV)Gk#mdbzdw=l0Hm-ue;yJ$|ShKuKwNsRM6h`%6I?#jXUX$kp` z`PXIfo%%QRd@f`2z8u9$(u23cmBleG30)8~g{~^HlzW?;0*yP%g3Hf2gp+mH+?3e|nzh z@P9P$A#Tmd@*iK9*Un>uZ$mD>aehB_9`dv8hx5LvVFf>8WpaLHI{cS>bND|8ryX6s zpS0#;d)39s`9ICr?+5=IRQ^xT)cIMIrHi3R_Qic4KN_+R$v=%pSn@O4!<1!Pv?A_?(lye%;50LXmutpK7Ot-*5~EM_`l5HXR$H<>->pYtL@z7di!c){a$8_|4WVh zON{YfpC3dm+6|Hy*MF74|77ETslopxTKV|?Pq1C*|2(lD>a6$Y7xLnr^?pTBve z8-~9+{9llY0ZPI7skC`x%N>3>{10=!g&qDAzf}I`=c_n5{|l7oy|h07{VN-X%!p% zH~G01`N@m(=Ng0m6$by!{-wm=zsb+o{zaevmmB;y`NA^Zh%?$1azT z_$l>GSod zrS}g}E7|;iHTKip{_UhyW$;~}uScylTK^NZlI{PkEmr6MC#`h`->)(9yVk(5`)jy@Y;1{5)1wpmz274 z@PGXuoc~p*pu_)$zy~;k-$VX~-2D^&{9XCJ2p^9vga1SB{NcUeKY5_af1STYMQVRQ z{IA#Izt8{w)A=9yW9NTZ<$qUT+hy>d##b7lY<>}cZd5SM51Q=x0nQ`Dj+e^#$DhZ4 zmzU1x4|pEW=MRNO|Dk;UiuWH8qyJzozS909*?hsA&&SRWE^j{Y{`mjjZ~VXee-i(( zpX8AY;^!OG=l_`hgAWU_fOg&oLFeZ%fuzem4}j^6+8=y0u!V~!KObQeckj?yuV4`|BGEO>+}5* zG?$uw;`wX5^|8LcUuMkb_4RMO{WW9#uFcmUveuP)mHI|4eLZjPZ_;{1pRcdg=I0Mt z>r4Gwe)@W#->TN$&y4waj#x1{`*SnZ`8ww3#U)kl`5NHTveg=1SH4nWF+X3ww$hbj zeqQ~a&CmIHoH+c`&f}a5Y~s%R9`a8)|IYc>ukvpRUq1&V|G*#Pe2l)nHu-nbTCzl) zPgDD<{*Co9ZKP&NoTlMd&jr6Mx{;cQL;s5actLyMl-5)E}_!`?ktTgy!u9t=P zxcOgY?2naIsPp5frSm6ty=<iXE^(@ATs#=oeg^TT|e7_}A~e5u=(pc|Jxw!EpFDOFRK^_Qy+Cu6NJ7fPZCGMH((&TdbXrsaU6+ zr}=~MFIye|UI|q57MeW&m_6U+h zD3$9!zLlta`($9vdp)02 zz7NaCx5qWU8Rr`+jq?px&bRl+H#x=U>rZq2_TQgx=J^JLZ~3u&bI&UXzG)bI)6O@5 zZ?60g!nX>QZ%+s4#0*|oGpX-0B|p!gpHHCogX`xR@O@t8``Vn3=UhJG{RZ0kghwo~ z-?G>_&mj0%;$Z!J1HQj*oM%|FL>=$?t)<5Kg^AaD*gq&$)1S0T(ERFos|sU3M&I8& zX)Ru=$1gY5hs%uhWQ`;XZk#V0@Il+J(bk7p595H6x}MbcSIqq!bG;h1t~2&)D%Yy( zN33U!{ZD;87JGgoc73Yv*Q{Qvt{-VVs^X|+u6L8~2O0aLYt{8(?0V7M|A}2c8s`~G zjQ!HZ;zWqEek?1kDN)ZeRIXgF;nh_|5{v!Pwd;yq`5%Pu+WCbS0?WCFy@z}sg8w<^ z8J8&UFQ@fnX^<)JgW!B0Qu$tH@O>%AOB{X+zLz@KE4^~ZMkC#|JR)$tbZFEh?7 zB->wISElp56wR&jy%Ky@`DX6lE?K6>uQ2$o_YYAk8Q;qczL(?cVQTsM{*Jl78nt5i zs6XFWxlZT%YJ>0Qey_>TWc#yK>r{Tn?#G(^Hu*`X)8xhbLvw$2-8!A`>nGeFqW&(d zJpZEo*$9S1+kfzUq}14-U83@z#`Ch8QkCy3SK|qma-OMbZL!3H@9Wl=xbpXu?_>|W zZvY4C=(3+rrttU8JLi?&4!~h?O`h*zZGZN=Jhl6LGBv3Fz5qXepuF#dpI8;d3hIyL*07oM%jQevzIJ>ikSi)$ziGIxaT8PoSUY!}=GmyKwnro==QDZ;)z? z*Py#LK39SeYJL-c|0wpnp_pBAqjP^>YK+hN_l-_k`uW4y@z~@mJ#Cd2&wq*WJf%we zz7hCZtnxSZ{Nm;ERsX&b&Vv-I{H5o;DnHHdJJWe%^Sl8%SX|EjZUH(>2ZOH_mHPR@ zm8;9#^MsIBt+me=3ceNDpIKZ*FQkGCad>Uc}f z8};x+vOYQRpf`1id_W1(5&jb8>kH7B|hkpTeJ=_qW zZ;)QjcsViu@ct>~{ayVQ`1qX5$4BsciDYm0JVA3th<_@|vGxG!)O*MIW!)rrp&fN#qCvMj(__46X}a|f7Z{d z;r(8zDnH?$#(Xq(eWUY9pP$n2CFuODF!*FXkIyO8^_!{rTk`L>R2qCT`J7v*^KF)q ze{6rD^KGTUCzH>iLUp{0?Jtt?&75EAe4_JK>iB1l=k)$X^E?3>9GAnl*~WNRk)xas z!1z{~YroGz@-Jk+??UjcxKMc?2Ka^-a{r$BrkyAFZU7S`T<;;@l=mZYz74B<3mNAL z3XSsw$@mru>Hb_`@WEUU<{5kgUF?gGFN=WG{nccAi#?yJ&foCS?61s!6$uwhxd!+>3lQ!96LVi_dDr&l7}r zdpUfYYw)co-99fM_*Iglye|U$E6cUtFCq9=m{*~lC;0vIO*>EUb^tGE!1W&TO?e+E z=i4QfZ+XUff;q-{f@FN7_e-hswb?Oz)5o`5gKrpq?2Ge_cB?LjZy~(jMlD~TFM@Bm zy4>VjEZ=p$70uE4q|fK@{;OPFo~8L~y!WfTZy`! zTlwDe4I8BP{sQOQsLHn-<2*sGah@O<-{^g;s=u4-F>`#&Huwg**cay;y>0(;_?Bnz zN$;QW{#9c=n5Ow#)C$?9%jcH`#`{HsM2MJz}r2`z_VQpd#$p(v|)sc=kH%-<>rJmoF~73B+CE$kFXrF=QvJ@ zuwfqq_L9B|=0EVJ;5^Eo_IOCtPrmOv+Ly-gK!+~-c^I!g_aE5r0le>9h_TI$ z`qdu0r1_I+-{|c{ z!MMzyj~{{VDJkN8QntKs+?i3Lyn^n~xGbN?FX;9c*o+tY_qzC+p|K+n(ix_;+n#MFp%$OHcis^arF5%oHAqK*SGp=h6h5 zE^066H!FjF-@(Oc_<<|BguMdY=^5hpeFR=#DB8!hPcD8yU0%q)pnKN1%#W8x_9lJZ z_zTCS|8V(Ti&;?!a=ZQo-FdUfU-^an*4G5aNq;we;X>74a;-VMJ-E)id|$5UkC6yC z*P0uk_TqH_=6t&#+Y6ZNP5#wIe8KmHKYBdiWf!DBV4s89XAT>3^b2&)3krWnGK@cg z?)g)NzY`{VF31%A%`tDEx$KgYp30;8$oBQGFBJao)^Fap%%AH|@u@y`dj`5^2L&G@ zC~pVK_fI8%VY0o*o`tMmE06k5UO&_q{G%6gJMjy;=Z{PMd3goh3&y4X63?C|{1r)f z`ur@xCs9A}bH2(?@Nt3AN5o%WnKzz?kGy{0rS_aXPWid@&F8=8fFPd#`kj7_pJe|$ zR`0R>V|}6GPw0Qw@1LUamH055MV$C#zp2RDwl`tkzsvqI(Cy6<;|tky86*4oSV*S# zaQ~pL*|lFFTk^@zzWV^XYcIp|wZ~o*aHr}yK1=5GA; zh2C+>&-MrSOSpwmg7V+_W1LPmDri;;j zQ}6>*JU^17J^XmU&N1e{^KU2JYw(9r74?DtV1FAq(C<6a{MRvVgn!>LYWZ0ha(*HH zM~&YjS>8KkoZ5Gs2I&`;u?M?TGATd%`y1N)_dKQdjq@K^qTe&Ye1PVc^7lpf{BX9# zxc~F{E#`Z(yS?0;e9{X%z1zp<4}3oQ)G`+AW*m3zg#OUai}@p|g83w)`SWbZ3G(#M zU_O=hL4vqM{kT3vXwnDsamt^hB7e+JWO|-I>{FW-CNVCN9`kqUPBeeV_+`n~_uS9* znSES}DGA2o>D{vZS73gL_WgVO{)$ME86f=K9%po8tP{V#(k|5RRAi01$KkbA{^l;qFtVE3hs zk9$GD6c$Nm!>JtmU>}Z=U+r;w_8{u-#r*NYBRqe`VBd|9^Z7p;;{H>6+^&xsKl{^( z<7@r`d+NWR^8n71DZhtdufOPlzjK~UUw_peA7++Vj|qQ{m%d)6hy7}gsp;=Pedn>h zWa;Sv43!@@eSPxuw4t0ZeI$8$+8|4q{_*7LX~CZ`{e|S|t%=fSvq&=gMJ7sLmpuL5 z6Qz$NPjCBA-1Zwvo_<3Ac|Cdh3H_&teKeWh&`N0KlEpp|9c*&5OAGw+TbL`ZF`d{w*j?Mgs*rOBXf4T2dHS<5sPG4#MRTJ62 zpFMV^`G+UU-(rs^%AfNm&iia8M>gDD)A%9k~bC;W6wpw4czaV}y_RY)9PveJ~ z-;Q^#Rehuuqmz!V3MEQl-lb4&H=HI6M0_^ET`SJcS+4lv^{4OQU zPcCWf{Z?!jOjO?|+4&^;#pE}kKYx<_z~g_1&-cyvv_9k)tq*zp4{Gsi zkH5{{w#yqbo{#(yVquK;|9#&A?_1z~3%qZE_bu?g1>U#7AI<{wEDQVfLpR_d&hQ-` zJgN&l@#gnmEPZu+{M~oo&BWH>?Pt#1IQ7!_zail5cl&<+^h-bbHDjZ9T*3)A`@a+8 zS?}I)Vf%X&iHSC``dr{w^wnU{ngKp{QBoV|F>^^{Z@9%(a(Fl`E}#}@Qp7Y3P1F@I~qUqvsI^l@S`Wb^5tL6IseP? z{Lj2}Jac$t{DEhFvo5uF&G_eDI{sPTzOnJ||9wx&SKof(r^i0z+b8n)lt}u^f8X=7 zV^iD5-ux#n8vp+B&%PA-)c76UpM7b~sb7x&NWjni>oo_%KNvqmg&8ThLL#%!hu&#@ z!1uLZVoG|)n|F-=+pou8y=8oS?%VHkMfyvHJ>mmc!~#)Bs>`@>2GoQoj!cwPr=nw=!u3WBNK9ZQgFh97hPn5{AwKWPzL#EaZY&DI1&EfM#Q0bKlXU$u7(!7 zNDFVoJ2d_Qk9Y2eQ{H0Lq?j+AMU{su2*I*$B6(dBIW6NRN?)HuPRq85(l;cL(=u(M z^i4_Rv@Dw_eR~o)EyE^C-<3pOkwk7Kkyj;=N0P|HN#u7Yk@qK&Kb%Bn(8O;=RCFhaz!75+>p-1RI=NT_;^3!$qa0Ki-|gTy@{#S&5#>78ZhJrj;0!N14mN~xq+io47q`0^1c_u-B}=i4Fa{M)@)I zHOh~%mr;IqE3m^lBb z;i)V-b)x*WF@VO%>bWoL_Rb7vvS?=FGLp7OM5_ZIN7XuHh4J7qD{=n9Cnl(oyBrVGu->^?^=U)4|cIg{0|{R;I9nflp%RwDgmpEX(i zx~@>agneEj{bZjvS^cal)Q|d1t$pqOGLe3gzmwH3a)tUO>~9n4C-jd_R=>L^p&uVp z^tp!C$Al>ll>>Hv{T}I9r|WpxeIPUSv*%h-dt^5HKxxAiRX#x4kk0>u=B15~Y`OwS?u{JWZ5d&ZQEjw|SH(y`1YLOn;^RXU}C4 zmH+#;U$XYQQvbbD|GQHEPv*b&JSdU^T&ew&^?z6D|H=GU_?O0iF;`8z zp5|l31al>tEAqYNmR5{s;@fW2d^M#hDg3V_m{J`K`fG4E|CF(do?)fgW1GDer}#=7 zjMHu$42}G%X<1oGf8LhdoVQqZJ;R1tG&PJa0*!@l*VtpMl;i7e*oh869!VccAooT| z&d^!KPw(@lQTmJ_Rqkmi!(RHwFaR?a_R-TZ^Vxz|vm&DzErvUB(Tp~k6;12K#kw`9 zwgGR7v<7D9vDWaWMM+N9h%1M+xHVyo`0`X?O;|&bYu4~5w}!s2Nb~34k2RFqdQ;+9 zL#b_Cxi>vZ*5GX)FTFP)dp)ot5&a%{EQOo^X9 zEy|ffzU8J@tq}=M!Z-Z1KWFou)&{>XYY1Kkhl1YJIM%?%HPwcplZ`9K6R~vijj2OZ zx{QQIhkX zHE#N>C}$`2?5hQg8p?{gTDtRAj4xCRhDCT> zIeJ`P%e0~Ra`f1|52DfD&Hu`*kZ`7!toF7%juoP|6_gp2YlWz5g}8EW(6K^XIUXO| zR#4du_vS|P4nu|izAVg*HhWmZTyGfP%`TRxE$yz%BBsxQQo zD^`e~UhxGt{gqiE;f&0#WwQH3R#0Y2@qNLqg)3LA5I?kR!BM%o9u`iV-2sE zS4Ogjvc2i?W@1677mY6$Gd1c(>lX*V2a$s*VD#6 zn`#Ai*;aV(XJ42jP9ZChr^lUlpz@F>-v1IiMr6hK>C?Cs`8wClU$X-LX2HaJr#h3~ z``H(E3B6QHnn}A_x^3yj#xcqDeJw?ftx##Dxap&EPnLYi&Hu`*@ZQh9&@XT+1cn0M zxbup%s83oczTCA^TsijXxv!D0xcOh172f;V7tURCE2IqJu|k~rhBuX4iQ&;yTzNFW zt;9#*cRsaFwfAqSn*$=?>xjk$N4b!Ks@GinwFg@o|Xf6 zcH`^|$T;qK%j)1pzx$n~Z!!9gQcbWXrFu=G^JaT<#9p;0;2EVRv3IUTLXlq$fQOm+RUkia zRc2)#3-JRF-ZXsG&FdMmPif<2sazh? z`TcS8XHNQhPh9;Sc|^z!{h^h-LPB0*)DP_@uE&MELYIdf`GvUoN9^Qyxnf-VJ90b` zqxK2u`8#5~T#?HS{oL|T3Av#kbM!lp_r<&4l<}YwkNIwm>*;sG*zlqybW(@rPo7cA z$Fn4WZ{A4kVV>%i0RL_fV>dQU^xdEs`H-kq9_49#=SY867}4fFbzt#C^7t=8vELM{<=JleYEs6tDY!*9Q$bLLrKz?B#~Dn zk;mFc?;qmWN9#Xg{YOt9$39y6IQG%xaqOeX3HhfhpAYyMjcop_{p>Ef?+dYeKU4VVUw^*ahIa!ixPpGj;ZlJkU&bzIxSTzw z;ic@nhRfKK8eT$QPgC+MW#=@!m_4rH5_Vd{i`Xd*7qf>od==~0a1py(!wczM#7g;v z3@^D-Z~^PmaEP^QIG;6XcmZqB@O)OU;d!i1!}vuf4bNc}z}1d@XS0xo^H{cq@qT>` z=dw#)HT_I>LBl!pgV9RBFKu6)hmt1{w$C>vAFY`6}JpScD*8d`7=kdp00>2i|bNWT< zOU%n(GLM}CvAb|oWPC-Hb(l+$E%oF5agd|%kb{**qlNMNcj;Y%d~Q~e0{qeCZss6K=%$^<6; zdHkgU+xm2&Z>IRRJ{Kwkw)LsUPm+_|)@Nviz_vaWs|2?78QLarzg_-zftkcZcwq?T z7vMO+F6|XKo8wFtY7#h4;`##uhd7?jBAo)4aJ+y$-Yswi$02sX61Ym@>^lSwOI-gk zfrUPWEOM{FLZ3o5^nk!ZpF(!wlL8BU3K_%u(Mf-yPaz9^PGF%=A>J+_u+XQFHJuSy z=!5be7g*?n-sEcn3w=^p{oe^J^g(+L2`u!1|NK7!3w=`9`KJUH`oJH4B(Ts2{_;-( zKaBdPz#p>zUx85t-~ej^_o+U0+&;)p;(Ce07lph*;>RU!k~s9Tkhe?RFL9T|>{mi= zN!%rIMB)n)-z{J zvtQzDE{A{qr;rQ#XR^m74srQ(79J6DVgCi}yu=k;9%4_ z#)Mqx1G&UPAILFijd%3%U3sV9c_U2k3k@PIT>+#*F?_ru{7sCT zblKq-Bz^_u(M7*jks&_vf4_;blP)v-74-S>!tM0~#h)so@PF4)1#n#}^|9yYPm5qD z++HVaG{Plhc}6~V_~``kZTaQi7bAg>20G(f!^2aIP{uyQ!}*4)Ci`p59en)VH(D_J z9>Tr&kpsKF^-_!<$Km0&eZn$42vvM+x}FKpIvzeVCcFXR)3aTDn&f&2KW@qjl-Ra6 zEkk(d6z#+5S3Sa)D&Zk1?oE)7p8E0;H7`sMbA@F{OF$P%cW?s9cjKl1o=kU>^k+MP znSKTYXquQSlr4hVWde$mg+E&z+j0#ssF^3FyjwO!ogp+JBfpa>YHFO9P5&5Wrd)RW z*+Jx=ls;^z(MJ`-72^Mj-#{bX&GcJ@yh}!NZhs-uyZja+GyP5k--W5ElTdzmPNpw1 z1ZMgIVUVt{5$|o8zCT?EsEw(AG1K3T0AvwU{(bUCw?ZPh<0E!@yB!}u#uOHo@$B2< z&X0D95W9Rko!$PoAn`=$Etw&ur+8+1>PN@~pqF+<@&$~1N!%$t_3IGN%Ku=NNND>% zl~0#pKGr7FhmGcOor5biy*^uo*`j5q_ zf0s;ew;$Y;x4&6F@rG;~UT8G5EWw^X*mpZU#r|E>!DX4vPg#s}v-xV@)$&MJn-@+3 z<3p2l5H%eSdeVp}q0KM*Zl^m~y=!;v!Io-qZP->**?FvGMNQ}46}xV_xw`Yl<~^;) z_BQNX-g0=)O@~%};AnI8hHV>m*R;1B-rT#dVP|Jc*RC5ox7=FOx@zaq>MghKXjrbu z4)u2JKfDX+cHU6Ech5~Xb?vI{+83a(8b_ z-Og2=TX&%B-UEC0tU9)T#m@HD+Rj_{ckSA?sdD@B)>^B*WyQ8*dspnP*tcs{MO#A+ z%4kFUeK+mfv+B0}E3T*VmhU}$aMR7z+gG$yHs91-eb_0txqA80jg<#K&|I@!#@W2G zv-aTD_O6|~I_oNLy1w?{_U7s}M>o|Tys>r9)}u9BZr#{cxBYOEI5lnib{^fZw;6TZ zxu$bN`_YYi4%Ro<)EwQkVs}S#^{#C-E7}jVR<1d|uWm;YefCu}?^)j2awJ*2u4Aox zmLF)XUDHkVj;r66TN}tu`)fDf)^dDR$KHnRoeh;cZ`;2?wcGBk`;UC!wr$jobyC+o zn~yiFXpNs|?ZI1buI6bfw{LaRY^X(=%6&b1_ascWQrfd_Z^PCD@SkJN@R@iv+jd0i zab$Z%%i-Pa`{UYn!?w+}+n4XF-E?g4;m%|GYVilZ*jKl;tM%~i zoqP9GDn7CMQ1hNu2b$yN>9*H~Jv+Phcdn_}yJu_9zGQI^^SDQw_v}2hZ_nx^?JLSB zzQn7$<1eE8WPZ1m>fNwn-z|G9H+RJKf!c$|rQJ94xRq^5^lRnzf!!0ezs(C#_QY*& zmtEI#c+IW*cCWd)C0V_XY`vwWc6ZYDsaYZ7?LD$9NqzPm-;e&RVMT3{c%3p{%`N*Y zk0*)SCF5?tdG8*pGlAV5JIT7kuFI3zbW=+mxRJo$cG&!Er~%hHtpw%TJljlpHSakV zzb+fwW&gMD)|SpS-L2@S}F#cp3&?#@ZY z+gCTa{PxvalgqEReRA>YCZ`|f9FxkgrE+rmwNxh6Zwu#I?dBu<8WQ>HR+S^2Es4hN zjhHu8?1Rs5uGn^)oPVHSYTqa36gSt+S0tg;_Bu`5wE4= zKyyo5W6QDL&gPz;#=|Qb4_l1~+YaB-cw^gP|e;OH#Jr^?mucZwsst*j2r3CiD_uiSW&n(bo)m> zg1Rg(U(Rb}FK#@y?ZgR*1)h1qe7?(PucIhZ5sxYUDZ@=YoAEF$VCcFrI6wRe?%Q&$v^$!;3Juz&i?c1f06Uq z+|T4qshcxp%RHWjbo+^u8vDfd^67(_7d(m5BR-`L+HvuDqn*db%6q7eSBrA7jZYw+BSwRi??qr;?qhs_QF%Lm_QLfQMj^+BU6g_ zv?HTm9|xSS*vGC{l?=CeWc$TGp2m?)(`EB>egb{|85<%LpZ2%tviZ3_LHt&V!(;0IP*}Y1OUQ@vk4T}-U+kp_ zyU}J1E<0@{K2!X98NQKA8QU%MIVHpGc6nCT*WQP=dHNGsFDCbAOZ+0Emt{T`I7~yA z9sY5IPqTNp5q^(!nSSI?;r$Zhkt)BQz`qn8k^8>?D8pZs;dQc~lf^pqn}PgnU!x>B zGTlrgyh4W4_LglM3i_Z7r)?)`3Les@gv%=Pkkc}p%9X0}5PQF#%2mT}gdOOj{B7L` zlb_RN=F=|27Z~|;Dfy^Rf%=tj?58@{X(e2Z|IGy90kkh&q^DZnl``Davr2}W^}Qe) z$gJ{?kghn!jHK*8As*R#(w(j1>o?|U7Z9P4#d_VrV ztSpZlJ1ms#XzPLY$res>QPFYaz|l#>zNM|Zr{m}mX@-vd-LOe*Tdyz$D?f6yx2?Q( z`%O!GdYfAgiCCLqt6goq)xEvl9s7^-ii z!h;HslAkhiJFwn?Hn~s?S`Y>)_*D z)r(+BAfXP#peC6E5F>+%K!{Bi(DoXrnGUKVVQm0dTWljx9;6aljl+#t0JRt0J{TWH zgVs@j^n=Vs1{dw@dH#d$Qz)hC`>;r a{`LbcMzB*y}RCB?`O=g9_-)*YnTKl z*uB3~Rj2R0M-m9f8D~7Xw&+`@es${9sdG-%J@@wQT`N}K5ex(r9|84U<(8PLRQ-GI zd0d~yRRNH|?ypqpiF2WfBY%g#^P!3v{|BFM;_vTq4rp0LEc9Rb@BIg#?fpf*dhh4I zAMCxnCe-_WNIh{m6qxakA$8KnN0rDq1(I$s(%vhT+OZ++w9z}wph)7U%x#jl+?0?b?ae>HFfxJT}JY2fR# zemwAl_BRLKKJ^YTpYDAt|FJW7y$L?|0CVb@fgjYpIB;b3O9MZ6^Wwma4&w8v-rl}P z&wTx!$26ZA9e=6$-1X*<2Bvp_PapBQrT39pkDmEq-Pg{%1s*?qvu|Mfi{SIjDUA82 z=$rT~>b(>FynX7?GriCB4a|7)F9&*Oof>!>I(cN)I|E;bE~3*H_x_;$YiHj6(cUw6 zg2$23&kXc}$Bca^2YTQ6$-t5Ji@@C4`_`l2Qvg2ghX;<-^$ooB;~x)vecvg_{?5RR zXWjwkw%+M?eCAkuppzSmhDtq}KU~?yn z(xts~miE?x-{|=6buZ!bF8uv@Xwq@D>Tq}~WbFIfNyj^b>clgl$ye0rgHyiq=ePBK z1DI`hsT27aUwfW9apd8OhOhI|!Kbz)T@@Nz{!J@FHi+n1=9 z{}ks<&^?t`2gK)PH_YCOY=kXaCOa+-uEB;p7w38{xi!0FK2!g>V0cw{Jjf^ z6K?=}u1NKsA0OxiuNiMv1bg4QS!q5WRL9=~{t1c?%p>!+ftNo(yt&#H?EU3^q24Qr zFk(p#Voz>wqDI~S{4BM;R>e3R&~U#^0M4 z$os~w!uY^$w2PL+PuyODxtfjhD9$lQOXlGGv+-N8qQ#?YztR+4@|CBl zZgl*;M)3O6?=}9Uxn8ZGkrRI}QKH_hY`}RG=S$SPOXjNe(}BA^5PY|Jo?4$@5`S;( z==g~*{YAWYE9ifktKLmR)@|UkD;n3BL?HI=(~W99wRvN{T3?X^o(my+v1e=r{;}Oq5yC-SDpPfQQw z<=>tYA9#cMv@$-w80S{TH*UqbmGSx8aBgKhd^^sqj7#X;%J>E_5y(jLXr0B;U%f}3 za=$#y527A4*4ZNHtfB}yLpzpd$r7BCJY%7Qr8sAKmf@V_flWNU9Oop@64(r#vpj@> zJTo9qB2e~|$`#f|FrRPGIoZhWz`KE!I0r5f2)tXl3iblM(%dh()7ZX0)9nRTSh|ab`=iYGxr+tIQP3JuB_3RrM_wT9oc^KoJl}mbQoCo0(cR_FY z>fv>XK>kmbp#FB~XZPpL7@z-rjA5;3jCKAo2D(rW8jo=?#`veBFvbm9Hti43{{hae zt?j-S=XQ*5Y{a=8#@#sg+S>EL)i%XXL~_(;D@LpPzt@g(w_ARw z1LZX2ofNpgIf3)#kQd`U*`)rTIO6^rp|N_O6hbWhCH7B;pxBCiPY|C=@wr)< zINOB%R6wOmYZ2SdVy^}KfmoB8|10dJc7?`ndpe-jU)`p9&xXbvc?0X}IjpgN7aBj~ zd}zXv|AX?Ip^4bjmCg9{2bn)Y9G)?T_*faveo@UIN!vQK#Tq)}Z0Ke!du*ec|3+xc zjM}i)K|>UJyu9WP#OM`>)yomHmqDJujH_Ksdq2Sb5Aj^b_e@B~d_7(sC+T4`#=8W% z!9El+n|)|)NR_c}Bo=Xg{~oj z{!_1!qp()~J=V%Mu&4U(A98NL{~_CP-_N7(;~$d8;Sbr4?B_<}Gu!b#w7XtB^vY)cA#U;raR>$@ zny0)G;^cggzl~)4AC~{qux;eYkKFOA%pZ}ypTC+J&$%o5VH<@-v~1E+wd93AN)v_sA60e4nDTEI-RZUJ!&nguwMtmXg~$A zvknFWDl|G22;!N2pLC&Z=GJzQGc%2&%uKdjZ)a4M)(&*UU$7o63!=MJfjp z-9*UQJzIN{PO^Cu&iHdjhbimNM9Cz3slvh6M>w+UCto5Xqi{RjbpsnM#APTjtFbtt};y zxT9lh^KvcCvaZf8ZTD?$ZEk64>u8v}D3R<+Hn%6XCF(IwGp!$7ZCAK)t9j#_|B%yh>j{-N_j9_q|)7-sPJ6*Ta z1uxvujpkI>)!CD5?o8Hy0?c=Fb9YCpv-uM-&6pZJ7CD!iXhWx-Kri7FFo!%Gzs+ju zC&q<&ZuNkMn_#=Bcw;fQ>oNNXy8*fO$M3cw*cAP48}iR;TR7PCYZ8gJ&Lp<-*gV!J zP};n`r8&{lF}FV9Z0=4toe69{6Fu!+U&i(r%iQmFEx%=(eP5KL*^z{i?^^!(U-ocJ z{~Rv0W_NAb($ku(SBXUD1Gr2xJMYikz2feKse!%Y3az+vKg980 zKrbzJ`KP~N08c`td^RV1@JT^rG-N)i1v$;6gf2jNx7SD3cpvV7!I_JB>8BpLN@=!~QowLeRH zg3k7W7I`FFyDwY&D4kJ{(Z_M7OO)6#*egW;7U-JCd2}mKROjJ+X&642eSGRLr!>yR zpwdC-bdX-s52b6RS|T*Qaa2=`6}1jt7R>q+tCfQAf`I)J~^NIu>3 z`8${2xg@r%%jt$~nS+{AUJCby)D|LjCsT0ZWmvajl*MUlZ>DriEZNo7-cuJtNNjEH zX^l0^o84F!`#gQjo_~v4y1lKvCDyZ}C)wH&>)76tjJ33GY3po_tzWZd_2&|wU$O3U zcdof7v0}se756M(v0Tks6Pp*C)$DBE8tdsuG}JF_j-hLkmGW4~%L-YyDN0|3=R~xu zx6*k@Bs)wxr!mkGE_KgGl0pk_3B!KTZeiH;F`lGQ@YH&xE|sXTE$66O6}()TEr(T1 zg+1kvn&2x3;S<4sbnyX<%-f#&p!T7)@SmGAD!3YHzCr#zz8;=CJw;o<_7288C7{=$=uF@z>Is&HOJ!EIDJg!2WKz71){ z4)D1T6*#Cj`uxCtzHdLT+Rr30#X6HB^4Y^nhUDOMyXR3YWI#{P-}AH^Z$L7p{epPNbJrhsmb z{({azr4?L6^&~H$aa7_D{gtM`?Fy9L5B{iMu>(3rj8(yONaIsEnF_*}L9x2ME-6KU zZc~v8rl1HCD9e8Vp_Q77KlESXs8Lj%iGlygH7Ms0ca@1hK1!X3JgLSC@&_&d`J!NH zDw?B>g5)`06rPw0Wk}UI)c^SkRg#M3K-x^^duR`N3LL~r)OWAFq~)ETDsD6;pvM!Q zpUP__8&Rac7Y&~D(OufD)c-6}^B;S1&#e zeT8yht3_I0z^_V$LWchbA!=WYGTCzQY=-y+RyiTWTO5H8G=G#0zQo|?r-GqfRB!t) zNri&Xp_Dm^@qyno@cYh@BQPo>ANmLXBIWj<&;3KcMJl%^SNojWAJg_X-1nUrSqnNNi7wU+^jLZPa0X3f&`8-)BPE7UY+R%jNQYkNk27*FgHbmrh;14L2b|49<6)>)_%i{+Foe<=ZeAwDb?5n_OREm`fuPr!Oy?+&-DDN%skUS z;S(A^ki+f&l`db-mT3>zUHbDJ@;~^`tVSv?g+EY#C>QJUK(A%12hCL zabl`c<6{;-B~`G1@Qp?v>W@>ox_(qyAVT&&-%+E<|9L*A>?|xGe~r=nqrPTm)Wk=c zPg5?S`HR~5E!jD@km5_GcrWG~_LC|Il6{qud@ojjf3(QUzr-mhq!l|8|GM_KlAZGk zsXkg(lm~s5l{ljdX+?-Ge>_O`w|mFjx!OJ`UT+PcTu`9PlTdCcQWFqwvj_CEgv%6v zp-+m(O=M71z5spyvD9~&t8c{HiK!B;pW>0|8}WazzKgZKi-*^Dv_uPR=5NZ*(&Fpr zyV$SqdhD23-)o@nGOh12t?xOc?=gOT7nhb1KCADBQop`SO2?w!)pvcptM3r>U0nQY z>AP3zdz`Cpt-pM&@4S)d``l3a&eQtNyN13ge!2EtmUkU}=lS(Lo1ROs_Wd~YJx=R; zoYwam()U=uzQ>h~BYalhbISbs9$!We2Dtj3J=@iH4)k3%?$^@y390W1uD-SY%zh+1 z5`9-uyp{1Jto0qfhQ9R%#nw;RnHTo!J4eKmoG|G-CmT;_*US1Y3QS2&(E6UB_5C>M zJKxK{#F>yc!Kd$>Jiopt=1uhJyG~z$DjiQoLEjT5+?c)t>L4vJc7NB9oS~OjUjKus znW#)>ct4kuszn(G0T7>pPNi=@nUkvW)#s#YeB}}+Qla(HZ{o$uNBT`Ytrr)IR5X=? zM@Z10+Ku(#@!SZ zKNUqW4e2mG?ANe=#q_0j@qPyDaTL3oKzAHZ>OwIJYe5=Hu}c1V8sH~jaWxR&kooNWGJn!Frj+$DC-Tryl%rsov8*Cu1y5w?}3TSB{fy`gt{+mr0o3BiuKyl?)$2dykmxY>5p`5Od93lDqQU;-gzE`R z2-tgZr884IxG8J@!71$TH1?m`zNab8Z&?4CHqFd;n)qS*aO{*!C`2mY=7C&%cT`uL&t9|{Mq|J>=f|DpWnEcs5>e-5kV z#Oq`CpET(U_PZ#95oGPN-kF@iek)s!In$IzOFE3Nd{NX<_Cnsv=PuGXZ|4{E~KRJqL^l)%}ID_v4dF0F2{^r&zs`s?d)BeQu zF=vwYC$}8d{$$IDM^mOuLvLDU#0!kq4A=CJ`kET5xBeBWcB-qXtYriq{#r%leEdPb zC|6cenS9!mYip@IDO;XCoyz3Xrhdi@D(g-^jKB5{^4Hnd@z>etE{#9*|IgW9kv|xT zzfQltzwUOJiT`z!YE9{C3QE9yP%|AqPMi9!BacO8GNgE7JwaE!=br(S1$ojTROzD|?t z>r{$I`Y`J&l|O>@^`rCGI9(uS*Vof(u@?5X#a};_!SztL9{&?44_#b)!MtwMQGq=b7Ip#?J zRaEb3@Au!Bb3^_b9=x7TnL_KomRYZtH8oWBuCG+~uCG+~uCG+~uCG+~uCG+~uCG)c z>H13dH?sFfs*wLnl{ac$=6YS{`?ajua=0-U{7wGkah*>q4x&i=Ys7EL&r-e4?_>W; z`O}S5F1?h^m)@((Za(!3y6omt|3H_KPsRMsPc=40UaJ zEtC8@zl!@EZaxe%zpo&2MfIUkl@7BB%OaMA8|gUNaNl7YPrsblpB2^ zzdB0S=Td$Z`UyIPFn%25&$vE7zBQ`nlJMt?kZ&(6C;b(A@~u?X`Ey)PBL7-W`WQOj zO8F6!Z!K{1BR7z5t+-CUmGUDuo^S0AKwz>DeZ3|1U+n6?stEe`<^PJWncuFW`+22W z-)_FouW#nZ^AmLPT>V4eq<`8DY5T%_AU{<~`UXGFZ>y~SF@GgH>)m`5^4XY=((HWD z{H2O@eyUjOzbHWYDYL;g`I-`^IFEK{Za%QVlb@Oc{g>oXKG@9%dh@T*50!uQ>AzR% zKhL9o%BTABf!EgmCenYI*1ygF1*N?+KUe>B=N$*xztunXXE>lAt^aTt=^Olp(f{l) zT}tcw3op(g{g?UmpQrVor}a{YYis^`q7mfo{lOhsw_m;jcsFXLWxx z>DFJlG!$KK}JJo%JW|*R%N&@>g$ull;Mxe;vYKCs94; ze}?eav2MOs`|B95KWKmTqqCWXGvjMf8B~F&FT2K_rq!Huf@K6EZu+f=1c3H3EuoI{Bg8* z{nXDBc=ECEx2Sjh#eTZb&F5-=oiG~tUmS7kZ#TL5TkVfIqv=w>t(W=P!qJ7k`a${G z(Ry#0-T&S+8so|Gi#bL5c~$0*{keYL+bxf_<`mcZ;q_I^>{(y+n(XmcEt4*r{jj^= zHU6q)(Di0Ntk-y5Hv3_BKW+BIT8I(shkehlWB*9cFZR=LDW8V@U5PVp6uFXq-u8S^ zVTn^_$}y8KpnNRlN4huRdD;J2p;kuC?7)Te^DwvJelKkg^_~{|_V%c8Gf~F#?$A#_ z-R4ZGs^)sk-)+v++Dcz}+6-E*bbDNXqJ3G7LPD7KyZGYG;OAXHr-S6j^Dg@N`VN%o z`TR{qYSM$c7XKiiK%V)j3Hag#(ad=ErBp73n(i3JbEYV`a(h{X${0^R@%tw9qmP65 z7tcd|hs#9lM|nG)*yB=>`o3GQ^Rp{==$Qh*$S37rxxK*G-}k@D%pgDHeSC*e1*mu) z3i$@X&o@5ESDMP0=k@#3I={1GM?jq>2O<3+Uy+@^E!`1%zJ4X1t}*^&!}KhJvP#oaJ#RfoLzlol|^*oOTd^AOmCm?@9&#V1#4dp*5AC%TI zq8`tyQ8D|x8uBfsd>Zi^@?@W%rvdBhH{H@G0P;ti@wmaE4?J&8>r3`|wSehQRhji0 z7_+`m*|WY_eUUvwzw=WR8PYI0&bQF_FMRnH`aT8nE#~=iJYQ#@zd$qSHr z88ZH8XUqEM#2q9Ejyd?#JRX4(C&eBKW0;dre} z71#3AJH_txs9q1N-RJ9Iul#%*#Z>f<=iR1}rzy8QRX;Ce%PQ*RlfmG~`o~zTJ!wsX zym-En@}*p+=QpWnop$oM{v)1W zLVm5(y*@^N5oauJD9}OpHNfZjuD`3DiTe2@cf2=8@jMidAJ3!a++_CaAR2$9vhhdo zr{@o8{de2zbQPB&f4RvuW&L#w`KQNU$6zb&^Vc!PUwJ<`KA-L{>O=eMcnSw*NaU|W zT)$C7(Dn!U$bT;_ux0JP=au27{db7`sLhSJ{!8*cr~AkEdn%o%^k3t<1Q?P3nrwpC ze~pKKc>mQLhxA`X>(6!kmj;R>>%S!`%WsJF!u8iQ*#qopvYVf^{u&Q~U)Eo1-TW-< zx!x)8<%46+So?f_ocQ7Sx0`PA`)|3MFV_AW9cu!L#^d@iPvm!N$JU~IW&9uYg(AP3 zGuC~+uLAj=^0CGq3_mCa~93sA8|83(}T;K1D^yHgdF~yfd6o0C#Nggc&#w*uriZ|eIuGe*amHs{*ngrCz zHPBD)Se-wy@fY%ruR^6hO#Gm-i65{BdR~$G(`|@f&uAl0HeS%>zc*gcdg6;01^)GY zG`{6;53|0b=!q8-t-bU5?$}&8+dtw(?9*sZWgRbG#W#`Z7;e3{?cI1m@_vNzf&{@a z2Y+_GC->t04*NGv-$?hDnj`%)XuTgvyfEJ9#*6E%_cPsm@wuW>H-5lgFyFp>G0L>w z*Zbm!FCUHTUElNdXg}Ru&vbk#ao0Z`FJ@NFB;91=fj6IB0e+LJ{P`+h{u%aMT6OLG z@=aBkaKj(|hImqCB79uONBHxg{Q|&g{8@j_BUejPk@VsEv)Ql0U)}wT^k>)u#UEmz z$7tgXmGzixyrD9!=@eh(egf9$D!jiaG56YtpNw;5_2%1J}ROT0YAEA^@880P6?k6SO^3z(78M z@~^1R=3fWjk1a0ttjEYdO>)b3GR)d33%QYaj35$^VM| z{wMFphKs%SMEPW>IxBC?8B^@>H`rT2an>KvAMDY-t;O_T?S%EM8nPj>cdX~dG<6&w zVoq_f-(Dw8rCQgYa;H$)u2-3r7dIqIK#{s+$|R*){c*LQ3Gq5RK5 z3ZA5|S1I4~)bRNp=m+a@=0>l-wSSQjBELRFzDM&L{`y;6j&J{-<%jnb{O0mKSf8$! z?}0ud&N$!tf&HYhU6t13#rN~ZMb{C4*E2rE24DSbN z|F;6-WaIe;JKwbbd;DY1_cSE)_4_;Y_gUfZzt;Xw+eZ$C8!Rxeqd)tI_kWw{`hmYc znb!V9*C%i%yB<{pX}@->_f$Ms%bnOSeL`J^($_cWT2ymY;n<0;erkc#GdZ%F_1T+eC$MK#_8{W>kg35qI?vc>;7qcsc{qzTz?(n`c3PP{MGKq&aT3F)_=E}wO8-Q9v@;q z_D`Gd38wPRy#O8WKb-%%`5B7uH|oEG*9({5;QIrv|C0TXzf5!Ye?#Vb{LlNvoHG4> z1=s(ki2detUw#Pt!7|T&5dC?c--AEbJA?LzWz)%Ew0~1Rr+{8%XUh1#0rr1o*x#Bm z_@>r(xW+<`yI&VVt;AW|HS>#GBO3$57tX%j6Dr|u0Xwie~o-uncl$Kc;@D32tNn-+_EXej_RSWYG*8k zAEk4+djDCb_nSKC>+((d(vSDIxE>p)eUBO$f2XqNdaSaNs@(l=MFo{Dh&dC+oBUi{+pD}Eg7rK``%As3KtFUpD7*e?{w3qJ?Hc~LKK1Vp;7{YV zt=oEBAC5N>)bv;Fl-xx8^))s4Bi@YHxfUxg**n#{{!&~?Wz7TfV?R+rW$Qn5f63-c zct4-$Tk`u#UL2I_jy z>7aVOUx?s^Rr=8Tg@`lC#v_XVbD%%Zehce!r86m_Z)j5!6ECQojTeYtHk`z*Jd@3} zG#Paj-M<}!(33X%1?Z3RQKS!jfIr^f2lKM~2VFmAEYOtkD=`r z#zk}D~af8<5OF{Ha*Q*#0Jx>HxXY(bv-lFv=P2y2~Wim`f@BMsJruSv&_WFKJ zwtdVQHOjO9M*buc@s$1RALf_*F`Ms!y~CeJ>DX`miToMsGv}w!-t#^i*yG^)H{(Vb z`-p3MgMW_l`)A%LuRm8KpF^h2{*U!|lwMo1{yAn0)oY%5{VBoQPhEeEICy_3%35Xm zeye`JN;dvfJB9lFKQ^C(>zy3;`^?&3^!*r*{><|qyxvzkMK@D_Ha@_g#@hV?h|oX$ zS;uD=KYAkJ^?`iw{8T>Oj`jFYseb>M_HXS!`u>S|--zayuisax`Jo>EWAC4s_Xk!x z1u^1}prTB^!$dpuq4P!bz7f5b@c7Hx^}uHRWgoS~F}%O%_O8E>{2!UWJoF&ce-_=7 z5y1Xje@P=AX!~>bqZt2PPN}}0(|5eEKfB8**4JllxiD(%1yKO{yUmG=q4=(C0_D4$ zf~c`)u8)?`&~!cG>upY65w&;A`TF|_+4W@T=kvmDKCh*S@^xWCxX))r{Lf3d{C*?PTiu=IZ{(BD zTgiHS|BZ%k-}h?X4_M6mi8syrODXo}(ECfFALQ$DNSFHksQP)VT9oK8{rUKL@#|pk zz*pl(s)0@Ld_*2L(M&JUcpfc+w_$2uVtWni>)YqFzfyb;=F|1Fo39V%=aTQ3dg33Y z^*PIb<&M2191h5b{ObHvZcYdb7+){LAIr{z@d`F>YWuE|&0_p!phbh&$zdA}LW7i~Crz9u?REnk%6 z!+v4@t5N*W933YAF>$XcYmlBV{eGXcx`KN2?|U8*_CWKe53;uw>~pgIgEH{){h2}i z{2S^h{(}G79s9zEw1)o{@!)pwzpCh6)0xb}A$s1@$M3hWj8XjV_?-rRr-9#T;6rJE za(#%u{hR12<|n`WFRy?3t6e9q{;#jT{=n{tKqFTVP}e)Tu+{Oi{?cTe+?<-XxR1Sdl zv&^#d`0#w9U8d;wzkM?Et6g8G==*-A>^*$!%9zj02ly`oG3Nbp!#|?g>xutNYepbH zoOcDwlA@PM;=@x8yW*QE`6wH^w=X`_alA$S;6)Xtx{6qlYZq^1~5-ITWQbtTd12%d1zKHKn{1 z7y^!u3+LMS?jdd5Yj_Q27Oo8( z-r`0}O0fi1eqo%6<~U~$2nJRv#MIrU3H}YNY_~Drkid5(@mXTrMkD-u+$g&wI18vH zx0L0fmC6Gjc%%xUvp8br;<;D!Pqw3vKoHgV85Gln{&!F-NX)Iq(PE-Th4DiyqSesa zb0vB{L=q8d5PwM3AG%?wLT)Q7RR}+Caida&uPIgdhNXh8`faOEhOWHUYz)t}=H~-L zS+naOL&w$6{)@2m>s$Gfk@((6Xyrpny%9IhFiXL^ATxfqt{dn3@i*M> z){zcrhUk;7LFv=`n{=n_^D7bTS#Td$U9#n!U4*3D#D;#*mJ7b*r`OwZUyGl9+Lrqg zf^N%;H~8sIwtR{%>u_+tmuz_>J+tAVr)_!TDnYm9y11Xd)|T5p?Wgaz<-P@e`e|Ff z*x;uZ&o$*|EB*AdwtUsxjL*s+(>IHiT1=OhICxx5wtNwHyS?;c-2J6vANEgl(9I#l zpX_nUp54mxg1M)IOoR@mo03w8?TMswz5JXG!j1p_st%MsR_6CtOO*8Je;X$Lr9Lbb zzN_i_PNA>P{Qj!T|I&k|nty+l!Dy}z^ZTn-xm=0QA0eXHDbiHNdmta`l%@&G`Jf|i zDVq^Py@#FFEQ#)|4AuVowp}0H{O}yIJ(-VpWnyRw*Xwk|Yy-m3eM=UHHM9YWkUXi~ zlXO1<$6iq>ii+%o;sy?%7%@A=eWot^uhjc3sG;_8A;u}&K889vXh9&^W&1m+5@FGN zxc^?X?J2KC2kTh2Ux&J{4bi^b`elu7?fxU%R~m(M$6bg%i`(`ER$O#qncCAlr}5`KE}${-T;?UxIctPEwFl`>3&8|@{xz(Q+7o}@VcW-jL}6s8I|TkD z|5AbD{*QSW4>fn<2)}?d`)CIyln&%l@H$t&{Y=EKA=B|28#$a(j}4bopW0 zQ7knN_0Wm`V1MOLZ+|2!`-Q)iTZ-EW_O{ccZWrS{WB=k$Znu6%ck2qr>2fT1MXI}X zbFwFPORT%uY3^uEwmR0s3?TO9t*xE0&CTuYZJqahD%Rw*wrt-_zr3ADws!V(If>?$ z=592q?b@FFRBXMb)!y4#Ul+sga5pT#-`quCtcz`F?`lpqESO#w>uJ3|(GHjg+yFpZ zXL8sog5!+An7XxwbAYx0K11%@+Z%DLtc5l~0Z0~Mw zz4wdLo4k{T^Sj%6I+~N4x3;##l9;L(Rd;p9nw|T$cR(gxuIAu_y1H1aocK>FJ23Js>7Zf#f)+q5Iu>azx)e5OB_mE_mi($;aS zydw6<+tT_#+ve5;4fEE`_+59|zD3tq%c06O+mo@bEwN2q+dErgD8uAt_iSzMZpHb% z^gO(^c3SqN)7Bys?9q>re^J9NbE$zn2G!54Z`2HjtH*U#HU9YU zv={T;n4NAjF;H7eYbwSK?WduLf4o2sJ>QfJYGk>?(3tkHx zRKK86-P`Eei$@lBfSyhUwO_bsLF2r+s@gi5r{3c~0N8{1>sUC%ct!mn89e;W_!cxY zQrFd_NE{+`W_zf%YVfqzo=t747vj$vvA2Fp+w3Kd)4U_LrOSyWDPa11R8wL-ZC^1_ zk~sPOpIYepTit-ko1ws0)*-hxIZ$3up`U^0eGl=dp?~aAhfi+|I~24P>kB5uzZEO7Xjh5VPG zx_TM@4SMJ_ZaRz^u#yao{v|cwX^AW$_Lw<+9~H!hXNdJnFVhIVkEzhJ4UO6{@qg`nSTY4owr z(pesk8~ZJt_004G9=si*=Juoq@1;K_@Eo@e3Ov&fS^6R&=V42yaZ$|X<5>^grnq&L z{&5+c5nex+ZW}DNNo+~qY3ZZ~vS*IBX-j8&X8In1ueb4eufQ{XpTMuR`27OU^uq!l z-fsrbC-6)^D)7Y?e@x(+eq7*RdED@SR^XZ5FYxCr{sn<&`e}h@IbRWYroSriO;-Nb z1fJ<<1)k+RFYrviAn+{bMS*Af6@h0tuL?ZVIS8`-u$=lwjG$V!AErla&}2NzX&-C*G z&*Qov@JzoX@cmZ)%L32zD+14QUKMzz+n1kY;}y#pwigs^&rFY4I>)!eRv!Bn=WKjq zI$k!W55`}y{y0JKXL?NF&szR90?+hXf#-462|Uy51)j&%DDX^QB=9`0#RAXtn&^OV3d{j|WdoUaHx(_a<%GCRMo2|Uv;2t3PqQQ(<= zN#J=NFAF@=uL%5dE6-JdXF6Z3a=hYkh3!Qw$3v#`kKHhy$7NqCnT>}`pJ4egp2t-# z_%l5w@Vu|95qPH83OvhMC-6+K7kHMlQQ(=rT;N&GRRYiS)dJ6Qt`&HuuNQchbA!M$ zeWSp$oJ|7H^sNHVa<&UR)4K(pE^|<$i3P8 z3CH5OMN3k06&#R7lX>a$GXnI02(ma|6SnU0@~@$Dyh9_s|2>GcB7 zalTRDnZ8Kid0dMHp6PLc=W#6;c&4uvcple!foJ*#f#*20QQ(>0B=9U}i@-B|tH86I z?E=sAl)$r`I|ZKUX@O@s_Xs@G_X<4AxliDkzF**3&I1C^^n(J=avlk6M2`CGbptMc_FOzbf!de@)<7 zZ)XLb>E{KW{q%ysGyS5#t8ba{UJ`hwUlDkY!&e2K>B>I$D&t$&(m5XTd_@GF=cPd4 znLa__Sr6p`&-9qUa~!S_c&671Jj+=p@Jz24_%f@{MuBH~T;N&GJXL^&s^SD|Bp6T5J&*Mr8Jk$BPX|^AZ13Lwt>1n~A<=i9i zOy4W;EayIfXZn7D=Qwac;FR9hdzO4`cZ*rJscBwrXLr0*2A*` z&-8wQXFa?i@Jv4~@GR#m0?+hU1)k-6P2ib+R^VCA^8(NG3j)t_UKDtyUlMqp$IAlG z^eY12Z}V?g1)k~QV*mbXt+j`UrE@%FdV#>R-t4akX0LylUMBDy2PO#qOfMJsMk`ND z;F(@4@Vrjd2|Uy51)k+>6nLgD5_pz#vA{DuF7Pbpa)D?1YJp#6ukY3hJkvJ_eA?<~ zqrfx0N#J>0EdtNcMCj^D=F|y-zo6fJfXldeXqbDwBy|;@J!z?@I0;q z0?+g(1)j(Cl)y9npuqFE4hcNd`vjizghvIQ>Bj}0^T5vvJk$FHp2ziqz%%`%!1K6H z2|UwJ3p|hO6@h2^YXX1P&evIiXZm@8XE`qjJku`B}; zuk1JFu%&Z6WO{+X_gnuh7I>zY2|SN$g1|GqT;O?JF@a}#jllD`Y6YI@^#b4Wgc)z6 zz%zZZz_T631)k~41)j&XO5mBkTHtwHYXzR^>jj?2wL##S-X!pQo-pHW5qPGz3p}qQ z-2%__q`>pIQUcHPodVC}N((&G_Xs?XYp=jFeZRmTv-W>L;FDDL4hx` z{0|8{(~k-~>*1KdGyS;07h8XPR^XZ5FYqkq3j)vdlLCLm%704WnSNT}dEfPlz%%_- zfv>mnzb5cZzaa3ehl>Kw^h*NIc5+$ZnSMp!`8w;Wz%yNy`r{$%Eo|u=51Aejc-C8i zz%zY6* zc&4uvc-F&ufoJ*#foDBz6nLgL2|VkeMc|pfRp40DDPJw4R(*n=* zJp#{i?iF~Z?-O{IbHBhd{V9Rx>!5=I&-6nA&*M5Q@J#O$c=pqy0?+hg0v|hI{PVcL zGreEn)i(|P1%YS!Nr7iMPYFEJPYXQD`HH|Z{Z)ZyIbRcark@x1RZkfCF9vJx(^m^T%ehwInZ91& zSH7qp z$F*PJnf|1}vmHJq@Jv4>@Em6j3p~^N1fIurRN$F@OyGH3#|570&k8(`t6$)mep2A~ zSbsbv@Jv4~@VxH4BJfOqRp76D%lPkW0?+ib0?+%3^8(NG%L31OyCU#Rzbf#&ueLv8 zo6YYrJ#4>>!SRj96|r=VhfFUJ_>}c8eyn*^Te{9X~3hxL{ec&4W$f4d*vDe%1COA9>fVUNHweXqc?9`*@5)AtKJ>*0XF zGyO?{-*5Hvl)y9nu)wn(`UIZoM+Khsa7^Hteq7*L56=ob)B6RU_3(ngGySB%vmQ{z{er-=9xe(z(=Q1;>*2D%GyRIdvmUMrJk$BT8EpTo zhp_!}8pk)LM=YK3>VSz?1p?3XGKshLHbLSoyC4+R9%q z@JwGM@VqZtEbvT^3p~%ua)D?1DuL&DSuOBP-yra@$Bh0r3Ov)B1it@qgKrUdrf(H^ zma|>pncgk%$E`k-0?+i6z_VTL6nLhm1-{?%-y`r$-!JfNCkF(c=}!v$Sh63Ov(KTRMGz)BAnTS1g^rFX^Sf zYU%Vn#|pz*|9jrjIS#PBU9fbH8%)0_@N93F1fJ=a1)lYJMc|o!Rp8HCJ2yX!=6-LE z>BaU-wo-om5BxO^}JZ% znI0E-mUFqlGkul7vz)61p6P1^{<4*Sy}&cQN#I$|7J+B_R)J?Z+XbHK-2%^YCIz19 zDSKP~Va4_^^@roSrimu!4^P2ib+Uf@{| z7X+T^7X_a6a7o~qep%pI4_5@9=~o5*Nvj|JGC#*3rWe~U`E&eXIm;}a@l2l}@GNJ! zz%xB2@GNJIz%#v8;E!4P>ja+Z^#afH*eLK!UnKAcE&s&=&vbq-3F(I(Md$Ytw0Q7d z`c_M)_fEu2Xz=}uA27gMIFw?({B8mI@j~x=3>Mi|Os}`)t%82mHnNu=`bQ8ofBU|M zqXN(G|F|INERWjO>U6fY_arv8bv8RY)NH4GZ~>FQYaxumleFP7_3vp4nh zTob02q4(vVTfJ_5!f?wnS_a9@yY*~0oe!*RO@ec4bB7&Q+a?F22GxvB&F<_Uk+U7efJ+YReCk!VTcRe(Oe+dJ{fzm{K@vu3Fo zCA$*l9b-mE(2p6qDMA6>qS2ODhj(;247lOp!Q9%LdwLSXz56h6NAtE;^ESLY%zF-7 z+8$_VY*>I75z_11=oM`^ZSdgkq}Lr1Y7o&!@vv*SWx;|*a53-uGXD~D!C1EdYgN;N ziA+Awk=U}O?fNdFX7?nWWb-C`?P8Yo&&S-bFF^|=l(B+zV!zy$|!RsB9iB@_8 zCG4}SBS9}5>q#ui#)WO#>8S&)^On}lT`jGN`&v5*iPuoJwu1wmA_TY-Vq0}5oh=Ou z?(1mn(C;y9SdajvZQk4jk?c8GwK{E^sWce7W!_xT+ENmUJ36+)+en&aU7cIn?%Ueh z+=4e8LjQ?mSCZmTf`h1yeIMrINxVl1{|`FWUN)rHZQNx5!n*Ic6(yp{H&4Yas@<+1 zwy28v#D3DcKwZRbnLjSrK;OUAQ0Qy&v<~ok#n)8Es?RUZ-dZ~loJ@quI ziKo=T$v)Fg%FF#TSGxS7M|Ov6FY&TW{D&I<;eRka@HlDwq+@Q+^dbzN#xIZ4R(?sB z*sZp`)L*u}Xu$n5-_58WO8$N;|9%T3`RP?d+@9q>2*t_1smbYQ=wDouTg8$X^yrIh G|9=6OeoLeP literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kee904649a81dda6daccd4f9b1b52a773.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kee904649a81dda6daccd4f9b1b52a773.o new file mode 100644 index 0000000000000000000000000000000000000000..f6bfb3864472df4fb8483906ea82e15b943accc4 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAPh6{Z0i9+G9V$45Nu{eOKOlnk3KEQ zvXo{tF;z773X9bgl(Ed-*u96} zdRQtb-K?i-z*hJ3Y`5+I&W3Tk&<7Q^&@~%7{QqpM?`Q`*4B8_-8LZ%Y*`EFni`iSh(%In|sXbL^ZIkj@9DwSr=K4DrtY~4+Gp@OJm=)M z!+j@?haYu*Cw%yy}|Gn^2^WP0W)OtRA{M5VQl$j5Q zMs^$+eCp|^2UF)S4b5r!PWZySAQU=yaB%yH1B3hK z9}kzGJ`rwiJ00#j@Md`Ap-bTlJ1&7{NvOA4lIO!l4tV!sX7>;n4g`;iuLT&&xug z69)$8o_u031fDO<|8{ua{5QcD^Z?oW@JgQZg-{uEdEp_@KyN9D-wub~eKVZ0^<;R? z`HSG|VtCxlM?xcnUkP3xCHQ%?`xCU=KUfCXzQmK^kp}_OvakFk@vBuM7heh_@5%mt za$-+a@~hcFvZKew3LLPJruS6U!>1-`o~vhVlllm^MP~(3`vWKJKMDRNd|->l3SNf~ zXd2-r^$qEcN2%RW#tL2qEQhxdefyZEc3t0OP1=>PS4)}M;`adsBFI{ZWZ>g(|9p=@^bZ1JSSLk-&Dvwy2! z8{W_QX&y0c;E`3pN1kQx(4e?E;}y^zGNrhv8(b8UEU zLjUkTvHsx)w8Izwr+)R~JM8fA-z0>Fzr_w;Y`~g2f%T94Xfj}lq2UME;eUMqw11rx zI=gYtBQ)2*^Wkf|UNy}nvAzp=E`ooOWf;8DnD3HIkns{^bV4SQEqM~V`V*R~kk2R= zOJ>(RkmthRuIL|ud}UIu5%4?$o=3nl$w=~D1fS#3zZWt&|4s|N3pqw!NeGQVj&~u4 zQGSwPW*u3=!-%h*@LC zKJY}kYD&Zv?LV#}Kh6Js}#O=x1o2em+J@{4uVjO~3MSg~ol%mFc;fGS3rA zT~ZiwexWEtwjrC4ov6Qepx1NIp9}FbknHK-f&RaV>pHy3ao>aM0bEbu^}lg{8rOfL z@nNrbo=phZU{~dTw(;!hJJ7#trXF&^J{RG)WT#)l24wRKu=&}=tbYjhBReRcet$jd z4;Qn3rVl(qHdXe&{4LlDzWgry_&wO{-(bI(Kd{4>55cYmJ2p)h`XE33xS}w0Eve`V z+3%h{%=&l^&I3Q@FM7MxdgZ&-|DZ7!yWD_IP;{ng6ySrhPOw#YInenITs?Rt;g!x@ zH0-Fd!xw1twREsbU~&noW9I3pxau0SjkjsKjWPQ;EP1lnoXn2fFN?gkP$6{ozl2PZ zY)iekIs2c*Ro}r9J!~0E07p}ZazRH^M`xhZTaP<+?QRuyey!_hYHJL#1#Rt}z6EWM zE$H0Q+_tqnR&VH%^~V-OZyMTL{oYQtpwk!ZTp&cNYv}51@pg37d0Sexc{`hYfw~57 zM`xYHcKX^n+5`U1K=a~7+gg3C-j?QVZTU;v^XvY7F&LmOzhF__qWp*J^7HcY7UQ;V z(V``J59cpkR#%q?IfYzzffyZ4-hi+1?hz#5>+l60gG8`Wz}IqDcpK#K-W7s0`s%y3 z)ol%UTYbVhR)BYb34&?XKubfw>u=(d&L3#+Z13#$qdDQ)+d4YEZJl{{f%`Ui{jI)0 z!(Cx!n1;(@wIv72rIQ=b2)GMusKhb*EQj7TQ&`!S3v@jL%b;Rhi&eW0kw?f(}J%H9FCJhOL-CS3|Fk5o*Y3?4_0R2)+6ab#>+0i-AB`+a0zB#m-K3AMgLvEZ8*QM zmIz9&1kybf@V%R^X{md9dS)|6WKYj*2ft;E;Pb4V$hzX${&fIETl*P01feQkNARNh z+M2SGvO;$;_Ww9X5rqy(h;%7lTM6Xv48j+SyBfW2I$daP@pd-1x4App+gm#3xiPbS z-VUETfANAP^V~IbSx~T?6?HYYG`c&wJ34)>?$)l3PIsekYjc~=y`iFF_4>M+6;g3muJyzXF60B&FCVB`sT zbe}Hnam(x(vHP1j&apM6GW%9L?1Hur+eP2^(dY~wh?Akq>C7(ccCz&4!f4$_qZ{fv zbDQdPR%OIfpH#P@TietK(^un|sm`X^=cIC*l3l6TXJdRA#gzk?y=#1sf6#q(N-&7+kil!ap?N7yk3*nk8mv^ z1Pb}Z=-I|t?=CIEegz~MXeft=qn0ok+QxXZf=POm{$v_U4@^v$Nc|5UN@C}To^kplq6a=V zXrD-DouYrD=hIBkqkrhbM&kG@Kj8f5LOugM_(_xY0zc@F=(+wfIyK@4b)Dq%SB7oG zZ&Gj)tjO(~pzwk|%n0uaI42Rmsb?ceY%q-#{2BU3oy+?M@6VZrn!g_Xq5miMu?X42 z?tndqD%kzIh?WZEyOhSh(mg)H%|ZBFC-kF_uc@xC=JLLg4S6T)dhf1;2>Sp4@&Wi?M!*&! zkBlr9_!Zq6yG#sd!ar68^fan6?PJc5D!2eL1 zJ(c_!3#ESx^aFfGFn1=coA58*J~s$+3|DKDNxs2sR{H!AV5wmI3q=3m%qx+I7Neiz zB>hFCd@H)KOwh!Z{%qn0{#6io@Ih`bO^jC%m^FvXL+wweVSdhwu(N1S^C@pS>T@IP zlC0Oir_Bt`n?vKxBmQ8YJ>8CZe7@a3J(c~Xlg$jyokjS$9KVn~(aq+PY7_6rqBC3w zdOPT`wh{d?C&$}S7x-e}-N3s|csJ?-zXJGaz~>44uxLLGb%FmX@R`7S1b%iJ?>`fD zflt=o3A_t+flttdJX<6_Tk^j};x*ujf4{)jFNA+R_#ny0{h$0RXtQx& zQvMd4Gl#^9;b+Z?)?efEh1-MAKeQ)(+0j4e=WUx_n)Mu`bu|*%%k7^D`$h}s3ss%$ z4gbJ=&gJu!=JPMoSkd#EASeF?zV!K@$>yRyx<4~L{56;C$^8}Mk$-I=5;-3-f-`4Q z{Nw)kbLxMUssG+x(_{WLbP(%Xf{px*FAAp@!oRTINBxWZXLA+qQ6sDBMP zwSvI3gs8t_J(wAsHjU^LWBxWRlj_{x_M*KD?Ulc+CH+ownf%QGyj$4w*mRC}pf1{z zKiYwp{s#YserHDYtL#k-kiFj~ebYUP)HjW1SN=Iv4E=NpyqwRY^)Jz~J@h}5)UEu3 z^v^$$B0R<`eSVp#zusLLF@4v!S@ccw+n{egUo*JAGo=3D{~1wz*JoqBa9Q;2a>evr zLHcsV<~N@oXfO4R{xgEm76gb4^osSD<_9;R)K`{SUg|Hp9t;(-(yZAg`Nq;$?=Iv6 zXVHY#ube-D{5FHXk^Byezu9^Gw3Ga&Xg#xw{DIbodRi| zpB|Gh$BB3;J}bSC^P8p@Q2g=RU@}_2Y5wRuUh0I$mmB^}@%J+55IV&R1Gke`i~U(&|bCz&50f5ZHlPyDEY`thBx-{SyKJ^=rr`1>+BB(?v5 zFD1gxVLDzl03Yq{3Bp<@b&qBsQf9}_BNI?H6GS8C$CK@rg*cjQAwfV|dGY z#@9EyOU9Ei{kwu;z2^Sm-$MHn_{Tn)oK%#5V1AxOmFx@qBj3*?cNFoE>^~)DfB3(7 zelFzn+@3TipwEKfG@X`PM*2JR7qos&OJtWLJii&54*yEdjQZEmLhJ`}G9f++=!a>) z;o$z09NnK}%^NrJUf3!JZ9pR4{pV~+L@xwIsOqe5C=S@|LUdokYi`Jwd-0i6hhf1CaN5X~p!e8Bo+@^|1( z^8t91zXNZW5719;(Cq*4_vvI`p6`I)jG$pYKwiQ}{r{3i`4s#Mh2rCj_!1)WFXl#_ z^m}e0>PQ!;Lz3qL?cW>;G5wN%Bqvz(n@LNRzh1-x@C$9I`B{Paoq#$N9-aS*=`nm7 z@JXl}@h+TtsP=?UN8O0em|(_FKs|8J|HjEUEo0I$A_y~{yICsvH~eWFd!vH3iqQ0ROF`>#XYSZYPlX@8FWu{gw}{oxzj-$^?Wf010z zl9xzw!C!hnPo)!+NDtZnkhU_KF9`fkgWMSyA0PhJ3x2fY4}8Qs;q?T@`60mTm@m5> zI4VG~VZAto@plV*`0L*v$9_ri_BDRKr}T5n@?gKt*AMbHzCNEZ+0&Yzneq|X??8Yd zL6P>eo@ZPJe_h8N6c54QIJqA)OKPZ1seufDP34Ps8K9l#U<}-fIz0xnw zXQKWW%V+rc#JM#44f7Yox9jFFe+vIYeEY2Pm(lr(Bl%@7(od0(V7|E=^%wCX3+pwO6A|yBKht`S_1Uz3qn?HJI!cfBSdU`+ts%|4USqxz zKDu6ynXmBqZOKr|QR@?r&vc|lpBLp&Jh_k0_sJ6>irpWJZJ)W8N8|L_h z@lE>;=-U*}INrSfFzq*hH^sMMo_~MBcy`(3FV=X5&y!;I2fyh!PZ~HV^#l81KQoy= zkKya{Iegx@YhrAF{vh9<^Yc@lZ@;k+^pg$y@k98$!9JPq$GLy}7ut`<&U-K(_D`|% zE3E&c_v2Hi-8jBY;r>1~etbjxm@<{`ztnzwbbK?N|KtXzPO-$d*!hoje~bBh6Z`Ec z#`sncoS-v&fF<^aG(R#YOoWBR{yQ7{@2m;*`J7?@J&pE@hIr?s4M}YO?R1W@|Hgp| zmD`DT6}PefPRD8q2E~3A?Omux_oK+yCK%6~fX^_VN0I+$QobqnpL{-KM)#k5|4sgc z5{-A0Co?zdl&_v&2z|PAgFa#Zblsv)(|%;^{Phnw-!CPY^H=1zrTcCzz9By{d_N$K zt)asvUJUsxEeUczo!`kK6c70u)*pOc%+Gtpd4}OU{3wWS;ynBq&ci|3llVT4RhrT) z`|C6v`+fE`5O9CS{_}?X;q!3#gE(K$qR&tH=aD*nV4g^I@Vmyc-{$+ZsC-UF^6~RU z5>l+Mc5)EjemI@_r;l4?e_HJPKkw$`HJ=}(u`hxjv7bc;IR7c_{wS88;PV4he)1XA zA=SNdPXqTfa8CpGG;mJ?pSlL<@EoCb0N-+8J-@j2`wxE<`RYr3mA~EmcmMG7@^OFF z_|b*Gi5x0mv-_Uvbyd6j=H?vh>tn3nUj6LI`~O@Q`5H&1tV{UIzMS5^z7%%m$FKbU zcagX4Y(@3D8OQW>)d}mWu^T~1|LJUs>-qG>QB9LyTI6+9V-L;&{t5wHvJL*NLMTzt(O!e5c(#?|vFQ7&X1*b8tup&STM~zWQJwHS|@iDHv3Pz|e zdWm+Nw{ce9(2T|KdsD1+=ea&nc;e$lt_3_46R&-g|1B}yD;vb4kILg`Y@Mbh>(F!*#vx=+DxznD zPLn(qf8bQymSmH?qihM08Mm83?|PEZdsJ>3DGoj=w<|tADz{t69j}k5TwsJ`Rgz&y z6oX{;Pen5OXpbUGm^hO(QO+RUjW`=U2hE(@bSvj^b*>A}qNAb{qwR&R3rpKl;>}p0 zry{h`n02&DKn;2vg(q%E2kUfw&4ZYGZtg+7v^ne5JT_oeG4dl|h4p%hp5sXT9bYN5 zQQk#ks};q%V#Ij5o(5IF@mHW&`@OnVW|v{yc(#ptCg|efl@+L;(NY^m9;Hn*Flsw7 z9%p)LmGcVU{MwSVE4_^2VJE!Ma43iu_@@pH2N+*gb$p}6RDTk6#U#;VP@U>9nutHB z(=A{6fCn2U`nN6hWtti9vD7OqbsCTQ=Su|ouN5cS1mff0uudLvt8G3hePL>e{Z{*k z*XCw;Kf_nlu|~U@p2q6KijUXlXUC0|yRquVnb<315ce|SR^;b5^s9L7)g+;*HD-I$ zywsDT^U^ICu(pq^y>8T*9w!D_`Hx4h@&Ku0Khje3+GVak3A{3kSi5YNwTpjS z*syk)>EE`rUxuvN*q_Hzue8*WtzLiaGOqT*_r{8EH;M-2IMPeL&+PhiRCz#!R?ncG zZKxBT>Rg%HxKw0#v;(kdGr-0=P0w*5HhcDvjx_{ktn_$`?FuM|lfAlCFQ`+mGf=0l zNY^xKj;hD@88)3pDNNm9FLXVYa&x0mE;5#G;^ivD>8<1fZm?_6v5rs%NUV@XNXMhH z*f_bK$$Kq%9`$3iz8JwUGP92C28jWGFV;gJ@G;XG{aLJ$pT?MRz2SUUTpc%#nm#c=Z~E@Fj>52ND_?J?t@q*+#X-MacyyfEzKH0=tq7`2m= z5m$|Cvd|wSF44%b*knt%G532D?(IYWFmpBD>VB*9|MsK7r!l&f+jxE6K8M!c|Il^Q zOidhpT4?1b9=&BhWXgv|wU6#O`5a7$t{}Ahpjheew7N+@aaKO7Jn`t&=8jfe(BIMf zOU}RfEbsGnw)c70uW3FRcWffo$D-ceQ9CvdJSY~k_cCMDQ?vFhU0+**y*A@#-`W*; zA038cx2?*UBk<2kJmNDfI+C@VqYEE5;RnR`kAWn>f!f(r)++qa~`j<=V4K0Pv9w1Uico5BVUY%91GM z%{>1n?z4S1yV+AOkU2<=_d=|9N>OV_Qn`zvXyc@A*HZAhHEr<2~Dl;J>6d6dO*0qF=2;U1|3b z!1=*Er@H>z9Nq^%8Zr?r=kU=zi05>1w>Hs>=X7P5U{rY$Q_9A~lU|HS@tm$$KgwU! zbGj5Qn)4!_)0L^f(QEWMT@s?`qXf5kLo=Np(^=YkWW6E!fwBO^Rt8b0O+dW%KbGyj zWKpfPJ^mTea>gb4T}A(-iQO3Bi^m;%Ls|DXNpVg{VTG4t^_F^Tierq?uj_< zk6GHQo~`X?2!;4Pia=gZ$@U+}I*m`?HLc4z-sT$kK0u35N{{3asQ{}X^TK8gu7agzX2$HpRQqo56FzcVqi{u`D8*}v7PWLWEG}x_QIA8-i?i2n`*1_8(KG3?rx|lef9He6@FD z!H$;lit6<{OINn6e6+gBU$wqv)uQT0%XXI4E~;wT+`2NqewAoj)Uv##zP2dOx0(05 zx^~gZuKL#HdEmiczi4Uo_M)}h8*8gNHr1|ue0A&E{KnGiuEtd@J26HZ_$aL@t?6o5 zw7i?fYus4gUR1WTq<(QtUQ6k=^60pw+tzvum$$Dd-L^r}lx}LPX{}vcv;C&%7Uhpc zw{ml9uxaz+3h1M?lw{tznRJ)mUb1N0I&ab1{7tRfR%|b<>fW@mAP{%VvYoY?z;}J= z%EueJ3z{Ks3v~5(eQ`maueQiv-|B;XYRc=|);+$4Y_O`P*;}}NO;Ow0yoT1AruxlQ z?ZunC)z$v0hqMc1a*mafgEcr9MjwRu&!zp=HZ8~Q6- zmtXUEHM(@%_Z$m@llC8_@^Xm&1c^`gwah;%RXx{2=@LlgGRyOe8 zR-s>2!LOa%P9^UK&nC}bykyCe#d&$lIL%6LONXycycbvF2W5QT*1TXIW4m(h=*K7L z{E>Utm%aq!VB2;=(um_FW6FNV z^9J5E8pSE`o--}zc|;u5!e$s(Aq9PmA2VF;gcJ~0L2ul73uHSk8HSykl8eG5!Il3D zeKGb;$faPS2k}l{Dkpl61S-D|L!t5L*IWnwd$f;7Py3sA@Q1Gh{{i5{&lzq9#?-vh zuEYQHNW6%?Qkvj3DF=-iZ`?Peyl!a%HNTocPyJk$aMgB^gsV+ziR?c|_ERZMru6*x zl7Bp1iWfDno|SyQEa7TCy(aq^kOoxnjew`CLDADX)T!Vpz8t|h5*9b3>a#H__HzHjv;nsd?C49Q2AEhtq$9PKAW9XlrbE3*~2Dp*__;uh{ z3~-~oxl->|K4(j~Ro($1=vyS*D({ODZk4x0h8#6+o*Z|z0d5>ODB)^cMZa6Zt>b=A!mZ;{-a>_P z7n)K^cNEvCD7c5$7;BPO1y^z7ISX9HhXD&*#erc9TrIuo{8RO>*8ADukBWl#Nc*j_ zz}0&1x4_kU&qUhYQCHvG<_&Z+>=7EhtG!)q4NZ^opPXEQ2QuT=9riPIRxrQBrY9;is-q%^!iQk{B@9OjsRM4ZL3EJo_@jceu;4{I6_Iyq9 z=kCU4aVy57pRcHEqo2oHy}D*iNwc@b*U-79y|JqW!{6lYkK3T7-P>5#*_l9X_i zAX(7R(V4&WrnPa0?1gXMxN(0l^>#M!rZ%5^H!tR6hx+Z-JM@=b#RB^Y-}fSj$W+VJ z(-oPZPL=tGLds=YRT=H`2`1BmDBL6O&d4~X5Gr1&`YhZ~%H>f2dG+rUH_yp@j6PeV zcqhYgGza^80BL+f0 z?*0CHbl-c`Ai$p3Fcfs(TmS!8)&Hvc|Eg}?{_?Vw%XLj-s%Y#L7ON>JW0}3Ndk??$ zuvAdGSx?P?t?uX9ZrlH%7ur5EgR#Ryw(P!l@H%J9p8GGjp2zE7agU}G6)jz_?fOCJ zx8vB=4>#zc52|dTYc_WH|JhjI(GGSPv`2a}Skd>gJ^de+vbTPvv%@n|duq^Ly^)YDH7rp{j+n$z-~@P&ED!~5R*Y4|DUo8jgYm%@E3-veDyD0J}P;Pw*-2KUWB z9Ppu}NmxMwm z4h+sc`NUudJYSgq?eM<&Z-OuA0kZeul|1JQp$h2o!b6~e-cl039S*(wW;kW*$?%-> z7s1!X@VJ?eghmLz9K7y08BTuiJlg#U+U*~#0Bm34$?yolwCpQCN&ITf$iz*sHvr#z1-^OD_wH=p>$!pN9Ua5>j#_+=d{X(}fW`lmzuw3PukHF_C{jB4t>Fjs z!x#Rk}ft`7fDzxq1-dMKM+JzF~I@KA$x`0U^6 z*M|4Aews&28+c>|@R4U(f8?K8|D{*-tM8Pu{&V-StN&8S`p>7as~3{k)fCVcW3CPF zP3Rx~C)PjwfOh!e|J1Kue1{z#{+oo*@VD6Eiw#&)C$RpJA58`*Q=(vB-VE!&qeT0vJ8V)8uML}2{K-Sj84czvL#PqSARlt74jM7 zV#(~92l8C_+hzSDkgr0@H3FVT!1D-rCK*Y-i{Nt{`u9R6=ih0ecOl2fD+!?y$nh@Z zFv?FdjJ%Q<8i72}MgN5f?C>S%WaOo!5Yu|%%H?35t8*}Zvc~#p-T1ibharOf1~F^Q z*ax0SS51kyqW#BJqxEI%kKHVNt|vs}6a5U$)6d69i9g1*wCPtquF|-VxiURhQ|5U> zsf&t3&M%aN$TnmXvJ>_94)l5s`g0+E29iDfJJA0(ab1H~CGLB0J%H;8y#6=tPviP; zG(PP0&a(+28|G#*N z{%|SlXZpY+WK(7T%in^n;LGpAkKcpM{tfnv`2#zA`4H@Cuw&D7p%3!YkE@D9*OE%E zkp1rI!>o_z;5_hS{-U>AtyjNW`wtp(vC9qU1Vv|>Mgcx3>jYbYmjj*ez}16S5?<-d zMZ=CNJA8pQUrPt81}2xVI%b}pimR?M+jyI%+ZeNt!;&Y9&B^Sz{j$hwixom=|4Yau z$+pyso3sB}T=g9+(ZiOo1aLHkC>L}zb#w+gz4f?L*X~wP=hwQ9rnbf)ThP|t>08kD z*n-X-&23xTWA%nES$}Ln^roS`)$i?O3p#zl&ILlWx`wXK7H>yKowucBo42#c7pQCS zc68QBY^SfSqdnm73^W%OY-{zkdRv;ewdF5v&#(LU#bAKC{Gx)o!o2*t{Jgxp!h$;d zD_E5GaQ@;Yb#-}=Q^<7}h|$sH4fq=G9zg=W4qxCgNCYbdd@Xl{w?Pi?T_H%LufA(r z-PVA&)hDcD1$Y;jAed$iv@`^~{w6-@{DJn)_RelUniHsSFdpN8D|`xv!Iuf+yyLHpLN9lp*yR#(^dSX~se zYT2qffvL;GR;}6F!J~H+Pc4zZ+*LW*jUv+=AZ^$!Tgy@oPqc#BpK|#4QSgn}tlRjG z-HdvY!*Nn*DK7$^;fnRslOqV~!O9HWdL%u_c-aNG`)FDeF2U^Ul3oq1=)ViN4d)lu z5<$t8K)R;_zIW3#Ep<;%&ur$1?CF{9;J1tse4e!vSyw#UzYc(CYd=GWAXLTc2wqfQ zy}qKXqS#%E{Xfo8M4>|xB3+EvRs#7ugYd=Du12q$P8XV6yq(SMZSKzY_Lh!$Zp>_- zx5MYoFI=!_o_jrA78EUIC0)%ejqZ-_j!s{zyS1yM)7|LX+T7-IudAwBxwdZovYNFO zRjcckZCtl(b=k5qHowYU=$`KlG&H$8T5%rmu-A>DiB~#b!RH?qUU#r30Jkr6F!F>v zx=$DPxMlW?*!|5M=h&K3nSHArc0t>R?V|7dXmkb-#L3X*bY_=zJ6Za2akOru(G7K- zxlMICt1{xLPpaF{t!-+A=__%}RAk?y=#1v4roy(N-&7+kil!ap?N7yk3*nk8mv` z1Pb}Z=-I|t?=CIEegz~MXeft=qn0ok+QxXZf=POm{$v_U4@^v$Nc|5UN@C}To^kplq6a=V zXrD-DouYrD=hIBkqkrhbM&kIZKj8f5LOugM_(_xY0zc@F=(+wfIyK@4b)Dq%SBGuH zZ&Gj)tjO(~pzwk|%n0uaI42Rmsb?ceY%q-#{Tcd4oy+?M@6VZrn!g_Xq5miMu?X42 z?tndqs@VOzh?WZEyOhSh(mg)H%|ZBFC-kF_Ute2W%jJC|8}d%p$(JGy=m-7h1}9G@ ze;d#BFY4^fOS;ledDhpZd0|iV4|_PdlaoEbPf>tPoW#c$_1;|x5%vK9!2eL1 zJ(c_!3#ESx^aFfGFn1=coA58*J~s$+4A*LtNxs2sR{s1EV5wmI3q=3m%qx+I7Neiz zB>hFCe9OAAOwh!Z{%qn0{#6us@Ih`bO^jC*m^FvXL+wweVSdhwu(N1S^C@pS>T@IP zlC0Oir_Bt`n?vKxBmQ8YJ>8CZe7@a3J(c~Xlg$jyokjS$9KVn~(aq+PY7_6rqBC3! zdOPT`wh{d?C&$}S7x+@(-N3s|csJ?-zYO?kz~>44uxLLGb%FmX@R`7S1b%iJ?>`fD zflt=o3A_t+flttdJX<6_Tk^j};x*ujf4{)jFNA+R_#ny0{h$0RXtQx& zQvMd4Gl#^9;b+Z?)?efEh1-MAKeQ)(+0j4e=WUx_p7k7~bu|*%%k7^D`$h}si&dTM z4gbJ=&gJu!=JPMoSjqF6ASeF?zWn*0$>yRyx<4~L{56;C$^8}Mk$-I=5;-3-f-`4Q z{Nw)kbLxMEssG+x(_{WLbP(%Xf{px*FAAp@!oRTINBxWZXLA+qQ6sDBMP zwW7eZgs8t_J(wAsHjU^LWBxWRlj_{x_M*KD?Ulc+CjCxxnf%QGyj$4w*mRC}pf1{z zKiYwp{s#YserHDYtL#k-kiFj~ebYUP)HjW1SN=Iv3jK5nyqwRY^)Jz~J@h}5)UEu3 z^v^$$B0R<`e}0LnzusLLF@4v!S@ccw+n{egUo*JAGo=3D{~1wz*JoqBa9Q;2a>evr zMf!5Z<~N@oXfO4R{xgEm76gb4^osSD<_9;R)K`{SUg|Hp9t;(;@~qh=`Nq;$?=Iv6 zXVHY#ube-D{5FHXk^Byezu9^Gw3Ga&Xg#xw{DIbodRi| zpB|Gh$BB3;J}bSC^P8p@QT*}SU@}_2Y5wRuUh0I$mmB^}@%J+55IV)2CWke`i~U(&|bCz&50f5ZHlPyDEY`thBx-{SyKJ^=rr`1>+BB(?v5 zFD1gxVLDzl03Yq{3Bp<@b&qBsQf9}_BNI?H6GS8C$CK@rg*cjQAwfV|dGY z#@9EyOU9Ei{kw``z2^Sm-$MHn_{Tn)oK%#5V1AxOmFx@qBj3*?cNFoE>^~)DfB3(7 zelFzn+@3TipwFV) z;o$z09NnK}=S@|LUdokYi`Jwd-0i6hhf1CaN5X~p!e8Bo+@^|1( z^8t91zXNZW5719;(Cq*4_vvI`p6`I)jG$pYKwiQ}{r{3i`4s#Mh2rCj_!1)WFXl#_ z^m}e0>PQ!;Lz3qL?cW>;G5wN%Bqvz(n@LNRzh1-x@C$9I`B{beoq#$N9-aS*=`nm7 z@JXl}@h+TtsP=?UN8O0em|(_FKs|8J|HjEUEo0I$A_y~{yICsvH~eWFd!vf{DV19I%lvi`Pris66~lTlj4F=@)5(6R&!H;fIq5qQ+Yw_t zPfVo#RXk4!irK_!FbeXAJ>3iqQ0ROF`>#XYSZYPlX@8FWu{gw}{oxzj-$^?We~Dbr zk{3yG!C!hnPo)!+NDtZnkhVOUF9`fkgWMSyA0PhJ3x2fY4}8Qs;q?T@`60mTm@m5> zI4VG~VZAto@plV*`0L*v$9_ri_BDRKr}T5n@?gKt*AMbHzCNEZ+0&Yzneq|X??8Yd zL6P>eo@ZPJe_h8N6c54QIJqA)OKPZ1seufDP34Ps8K9l#U<}-fIz0xnw zXQKWW%V+rc#JM#44f7Yox9jFFe+vIYeEY2Pm(lr(Bl%@7(od0(V7|E=^%wCX3+pwO6A|yBKht`S_1Uz3qn?HJI!cfBSdU`+ts%|4USqxz zKDu6ynXmBqZOKr|QR@?r&vc|lpBLp&Jh_k0_sJ6>irpWJZJ)W8N8|L_h z@lE>;=-U*}INrSfFzq*hH^sMMo_~MBcy`(3FV=X5&y!;I2fyh!PZ~HV^#l81KQoy= zkKya{Iegx@YhrAF{vh9<^Yc@lZ@;k+^pg$y@k98$!9JPq$GLy}7ut`<&U-K(_D`|% zE3E&c_v2Hi-8jBY;r>1~etbjxm@<{`ztnzwbbK?N|KtXzPO-$d*!hoje~bBh6Z`Ec z#`sngoS-v&fF<^aG(R#YOoWBR{yQ7{@2m;*`J7?@J&pE@hIr?s4M}YO?R1W@|Hgp| zmD`DTRkyMKPRD8q2E~3A?Omux_oK+yCK%6~fX^_VN0I+$QobqnpL{-KM)#k5|4sgc z5{-A0Co?zdl&_v&2z|PAgFa#Zblsv)(|%;^{Phnw-!CPY^H=1z<@;_ez9By{d_N$K zt)jywUJUsxEeUczo!`kK6c70u)*pOc%+Gtpd4}OU{3wWS;ynBq&ci|3llVT4RhrT) z`|C6v`+fE`5O9CS{_}?X;q!3#gE(K$qR&tH=aD*nV4g^I@Vm;g-{$+ZsC-UF^6~RU z5>l+Mc5)EjemI@_r;l4?e_HJPKkw$`HJ=}(u`hxjv7bc;IR7c{{wS88;PV4he)1XA zA=SNdPXqTfa8CpGG;mJ?pSlL<@EoCb0N-+8J-@j2`wxE<`RYr3)xX{RcmMG7%5i_y z_|b*Gi5x0ov-_Uvb=A82=H?vh>tn3nUi<9G`~O@Q`5H&1tV#IGzMS5^z7%%m$FKbU zcagX4Y+3D^8OQWBwFzr#u^T~1|LJUs>-qG>QB9LyTI6+9V-L;&{t5wHvJL)B=C5iMZPW9Ne^39V4FQ7&X1*b8tuo6IuM~zWUJwHS|@iDHv3Pz|e zdWm+Nw{ce9(2T|KdsD1|I@bkf(NWQf(e^^u#pP`&@n)>h zQwiE=%sN^%pawmT!V@>7gEhLo>OsssH}{}k+MM-j9viT#82J&f;(9$r&v7LFj;|Ej zDDR@N)rw+WF=9MjPlKx8_$yGX{a)QFv&%4UJljS+6Lj(L$_muaXsHb&kJ2U@7`2@k zk25{B#(9Nrer-wGm0rg1uoGTrI26PS{85L71B@@LI=<0jsy~UkVv^`Fs7`elO~fD6 z>6R~jz=I7F{o9uM63vYFSnAc5I*mvD^CbfPZxkom1mff0uudLvt8G3heQ|1;{Z{*k z*XCw;Kf_nlu|~U@p2q6KijUXlXUC0|yRquVnb<315ce|SR^;b5^s9L7)g+;*HD-I$ zywsDT^U^ICu(pq^y>8T*9w!D_`Hx4h@&Ku0Khje3+GVak3A{3kSi5YNwTpjS z*syk)>EE`rUxKXJ*q_HzueQ{YtzLiaGOqT*_r{8EH;M-2IMPeL&+PhiRCz#!R?ncG zZKxBT>Rg%HxKw0#v;(kdGr-0gP0w*5HhcDvjx_{ktn_$`?FuM|lfAlCFQ`+mGf=0l zNY^xKj;hD@88)3pDNbEyFLphaa&x0mE;5#G;^ivD>8<1fZm?_6v5rs%NUV@XNXMhH z*f_bK$$Kq%9`$3iz8JwUGP92C28jWGFV;gJ@G;XG{aLJ$pT?MRz2SUUTpc%#nm#c=Z~E@Fj>52ND_?J?t@q*+#X-MacyyfEzKH0=tq7`2m= z5m$|Cvd|wSF44%b*knt%G532D?(IYWFmpBD>VB*9|MsK7r!l&f+jxE6K8M!c|Il^Q zOidhpT4?1b9=&BhWXgv|wU6#O`5a7$t{}Ahpjheew7N+@aaKO7Jn`t&=8jfe(BIMf zOU}RfEbsGnw)c70uW3FRcWffo$D-ceQ9CvdJSY~k_cCMDQ?vFhU0+>>y*A@#-`W*; zA038cx2?*UBk<2kJmNDfI+C@VqYEE5;RnR`kAWn>f!f(r)++qeZH@<=V4C0Pv9w1Uico5BVUY>ar;1 z%{>1n?z4S1yV+AOZ_Dwz-t&J*L1YWM#(TC8!GB3_C^nn~MZa2yy3+0= zfb)ZSPIdjaIlK>kG-M)L%HgAX5YOr2Zf&9$&*{oA!Km^irj(6|C%qVv;yGQhew4qc z=X5DrH0MP;rz=x|qu1zjx+Fx=M+t88hGsfHrn9v7$a+Kc17!h-tqh`0n}B%je=OU5 z$)Z|od;Bq^<%~=8yMq2m6U8AL(0WH>DRz{G$R5@{sa>_4@J&=P6e}(Hh5SrhEUA+A z`?CFULtDDF>Ye-sm z%2~@4Q{$_4%Knd`@r~LmI%-dJ*7h{dNd*=+WsDpXr6)&RsXfv3h-&m5e=@-PER90s z%@TQ~L1p28=xV4F_`C2r;)(h<71`cn0$Q95I9gwb{}Kxw9^uc+&kP;Mv!5T1iNCKS z`$>-%?auzOxGuV1T(fW?`AtNq|0e)xd=wLE;wAy4j*UgqMnUV)es7%dzarb`N=h1^ zibZ|QC);~0;~$glmH$$Ea%pS(MVP!##TozWn?a1?=I@MI$5(z-iT1r{@3D+OAlobd zQ{$_43O*Hpc<&#sjF(Yt$6MFj^bs-qLi*GDQM|xC@g8&RJgWwdvVJ;?}o;%O&e?S8(KHi?kHN* z?kz4|Rl2^UsiAdw=jM$yEe-y)>sl&yE^jDk0&Hz9;mcZD_43q z6zyoKtg2nRvwV5W@<(f%{55M^Rut4eTCuZYV?j;J=GNu;^(#c%l9r_{^&3m_e4BZ{ zD>oJ_@2YQIng<^I^#zM-x0kHm-ng-*W7EdfkFRW9o!?kq+ts+DWhcgH10Ut<%hz`` z6fEtg@ftT&wwF}wEUPbEpVv~ptui`p`L@;G;-&4&%D1hPH07Jx*0*jfT)+LM=nC@3 zqFcVXHQ2PdunPKUEhm|`ZYJI3x0e-cTjMQRoxiDd+p_J&HQk#w6b0grS+R5DCh%Qf zzWniq?xJSM+X7uZUSC?2=i6A~uW$9iKI<#%+txh3ifpiAeY3ZC?W&Tt)p-rA>znE~ z*R+>z^48Y+YnHZ@Hno>+*jDW=DP6U^e0Bcj@?~9{T3fm{m*Z8qs%!I#N`GVP`flj2 zVom<~#~W8vwrtu^0DITuZ)#gx+q7BOYkNuMR`i|MxUsU!Y8P**|M5-Diyl$_rml6Z zEzJS^IEt^{>s^vp*jP|lv~+P}!{UdxE@^03+PJu2af7#@&{rqu8k)Cy8+_OMiIsKy zw^is@Rq$&kw^PZx!L!Np3l}X~RG62ygwrhdwsiRF#CvHieo)5eZOsejF}5q`j(&V{ z&L6pVed$XuPX2;?ZX&f9Yme73ZTEC-T52{P{(@cBiS7pqhwCYI zZ=st8_&J5cwM^ZYIbF}{->?&&|KYEMQ@^|z@VSOyG~0qF`Z5DP`m0DQoiC500bUR* zEzz~y1fA;MdVLXX?d*MZBw<3!2gjLULJpZcX#8(dluu)uU?F8qMhXYi6ph~>w zJ+I?kqfwj^?>WiF@E-t9{G8!-U`)*` z?K=EFkHm}UtECBElXB3Q@y2~a%IlUUQ1h!9^wiH~30G|kBwTG$%VhsKvY%>cGNtFg zm;B@DQoN{n^{nLcWeHdF={4EUfHa_jZvZ@94T_%DpiTu(@#XL>vi^?(%)W&y(dS_t zDksH-y@e~muSmFE!j*gz(4X=hLQRzTNfvmXgiirnwI$FOC44I2#x3J+1KgM@9+Pky z7pIBQa$MfgxJG!Hl#}{byb)2C~Zgx`bQ%`I&@U<^83hA0z)C8Q?~FGv)lDag8xD?>g`! z5>EUWSF`6N{2>c`z(8-b?`sln<)5CSLxuR6B5wKnA_-UHDqn1pa5b)iA2HAy$9+!1 zt>gYu!mZ<`%6UuU8t02k!bi;)38#J(FC@Sg16&^QaJh~c;L$(s0cMdLOv&r!)Qo+> z05{6jB;iUAioQj{t@6Go;Z}LeWXMtD=E-qa8sNrpgA%UBRrI?h+&b>}B-}bKF96RU8<$z}3>L&OcTEYQ3Kg{-`K;kF?(k z3tX-DehXZ!_e`YS9d-51ZQejP!ycj0yVBd$*3k4Q|H;W^cpx)=-I4#s6a8ExyV;!w zbyU~f){LKdtncW!wPtNyt%9LuUk5lGYm+0l4*cgO$EJSF7wEuGLr77Z>jRLx+}A0@ z#SfRz?^u?vuAPtHlWW)^sFp(?Yki%?o%sF9`mRnNK?OY;nxKu|GT&p(4L%cGXwTOq zf9`H<7Pn$N`uU2wHu`zIl`Ge;Dr@$(_!>G_wKsOPVECKd{c#(#w0j%tI@{|SIb%W~ zuqXHwRUwN}!sYD&MR2QQvE0U+u$ecot)s5P?`s(SyP|>iMsKH=E#U9MA%&GzLXt9$ z5+n;6Iy&`=emdWZh9t5{$^;rm_$5t(Y4 zdb%PL)TuK6P)NB?KEY&K5QTf>-5D9j6hg%-RiA|$O1V4=Ag}(N;^sM-kI`pq z6z^m>j^<#04P6y+KbbAq2{7nwHc;vYu%*a?8w& z+mN`7jWNnN*-2$RVKdHTnup7qn%WJinK6@bJhlf<##Bw>O{KCd_OckiaIbe3&lq>U z^FQvreGw1urnbg}A8y}!&UeoFU+4TUb>GLDJ6bfGjj3c~f5r?-aT#-s8_yp8=wYdl zbg^*HX}kA-6u9iarVF}93K?5CZ!b9Z4z73Y1?&C__Y1iG8qZK&p=9ZPb>EL7f0x2O z`%S+V`Lx>}xoT$%e`{yQj?S_L$UYZd#%g|8;5qf1M)uZUXl$V{HQaN~o_XxNeFfQi zE}X$?{-@?SbrE`+&Hk1bc^`c7(Ai5To;mw|=i#$wP98e@$C~Fe$R5G9@cQJ7(PJl0 zMpF*I7(Mji@#uSgb^VMj3`J2&0FP@1e9sVe~?!x=g@s?$DUmAZi+J53p^jOD7(WgInKbo?(CUWfX;j`n&@2RpA(e_`S zj6OW|%joe#Z$>X2|1kRg_y>@wj=XWN1E=(<;5jgEf=J10Jjwx@jrnfoH^o_GTFd*bXU4`A~` zG^uP;lY54jhs9Yjg*~<9xVGXnq(`u^4^(edY}_Xdc5F2Cnbh^l73ncqJHw&S&b8Y zVAH}qgYcP^HqU40>?@BEwp(L0djA3w{kszWBl>u^jn%w{KDOBiTY2of=4>auqm0%3 z6ucs?BW%44Klrq}G4h-K2Kbfdv#a}F`GMI#7xTAI?Y4Ew=g7a5Z=Hs3{TY1kj&J>b z`&Re{zIF5_zID{%TjX=fzfN2HOZnrCeC+DJmm@KI=35u-D;M5-$F@-DVCn1NpMPn$ zN7`}!;CpQ0!vkyqe!P&lMvEjN9xfeVr!M`BExePWMb6u83rS1aXGs(*N$hH&?eqh$ zL+}0Bgj4VTjGc=8f}Q$cLqg=Q?y^PBrL!Nuw~w8A?;W=A-o1{wifM?WMii&uD|GhIYPX@!JKi^ zxd-h)Z7`aMJM_QkHrheWNoITOR^^L55z3$B(blJU9?6U6aWy^v$`{=>?jugPtK=IW z!ymeVe++;582*wDANWi7P&)kVz18-}#qYsa4zPu^7A;Z%x%4#l*+=iN1;{SEf3Ft# zu*w#>cuzuve2Dz%6Zqyw4?4h0jF2yp&yg?jv7GYN!VmEJ7K!?je|^ziAGx}u;R^N7 z|8Mn8L}S1Y`4z_f%hKCE?E5b^mceE^p^T(4n~f&WD;=EVA6xM0L=Lnh^Wa*7YZ=R+ zY_Cb<{=k%fVwP87c)SYOXvu zJ~lNnZO}t|B|TTJKMYMz&iI0?Jm?Pv%SF|^Lwkc0zS&vtWR=%HIp`nuPE7|Uy%G%i zr)H-EGr_=ERpp*Z|KxOSg;(bdj#ShV(>F1;XR4xhy3*}-*H+h7RP*Zwf+TVy_@*Yj zZq!rM>o%yy?5Ho`AHIDm67bLZ19PYmdMV(axGlU5_3+&m6&dyq?%m@Z3HT=cqIawe zUwIq{VK(HOnAj8W&5ZJ(n+Z$@r-Sn|P$z79YBuPb3c7Ct_Z{-hO!@;uw?#LjYj|C( zeaV?}=;R$}X50pSsHzk1vz&X|gs|#cUZLxCu$)xPW3kek73&DG@VLq6on^=kmMV9a zA-|W)!fRc>*y|me3SuemoAA25poaDi`@FN0RUz;2*qk>og+=d3@KNtbRi$@!V)_v* zk}=ucS-bq6E&1cdI_e!O@6v; zvR3jJz_;T(#F}DA#ga+Slu+((&^0=Ib)B9OPu_XG2)k6jYwLxfg=wLq3hTw zo09Z4Es{@of4Ib&jq^LmHj|JH9IA-)FNw#Bc>ZLb(x|vy)Aq#1$qaiA*RRkYT|2fm zZLN1T;s67EB@!L-P*>IB8X?w9FodvZ#C+hQ(~+?WUvO-C$`zcRo|r9lVUY3rX8o>; zs`Bbm*ABXs*VM6wy<-!@uG#t7pnuXexpy|`8upKjP5E8hySqDjy*rwFdbf6Wd7F1` zZ|-VpZenHKt}0iVFEBLfnw|7kxHtG*$ePMZ=R^4Wh^4IK>z%x;|CFAWIaSy4 zXX$Rtv0TVNZh&;YVrJ!>*H)EHVF^8lRSPAGtopvx+9BpS=akNH0 zW|Fs2%}g}U1hgFoY4+S0(>iR-?zP(-YfE_FlAqixIZ8^{krc*W!gUUp3q_y7bu^8! z)40yzVmPpM;nLbr$2vvXj9S3)}HrEBWI%;+w>^ z2LKY4i|${@`&D`W9QRiUKqCL5K2fpIL`)>b7Zl1WAFOP} zXY&{cC_Ra*D2-*$%V;VDJ($M!hfsG>ea6nGvlEXb#`pngGA#Uik{=*##@66^H;pX> z90?>IJAF8vy;8vT@6%%J92hjjolk?lGsgZIr=6@kjg`?MtV*z7Zl-^k4t?Zz2?C^F zl+Mcm+(o>1NxqQN)I5UT#pzu{ze0Eyr@M4~HK$kWbP=bE zbb1q~`B=#9Kfq}^{1^6XIZfk|u)l%R8+5#z)7?7V!085^uH4$W>htoYe-OK4-o!-vr?L>3?eVp#o@p?|z>-7DczF*SWp@dig zo*gUT&xnWA0CP}0%nW{FNBL}PLWvFcm*+7Ddf7KJ5_r5h_ouVh3P3MQz~4vNlGJCNkk7~r zE#dMs1`7QIPV-R*@z#3qZ97CvJaN zIpKVKdzb8Wl@o4^Z>6O~^YN40YiJ>t-Yf^=1=pj3?W&2$$m#@Q1Pp=i?FP)762bGMWox)S`&L z{rk#e>|HQue!V-5@KWl_B*HNstPW@{S3F%vv7+gC(k}81gVwspHw^ux)#O_oM}Ebj zfdL+_G#-J@4%Nh{@AEQkygX|{<*PK%9O89v1Z`f5{fkgPimxCHPMOi9JBm=6zhh5QUdk2A+NPjteEh2iWL*y zL9u4SJ1N#o_!f#)6TX#V)r4=PSU2Hy6ze8@8^wwVe~4nmg!fRanebkUH50y_V%5O4 zK8jTnUSA(i-%qj5am=p=A4}3$0qt0@NhwZag@Jrc^e}|pC?Gta6?jZ|LB5WkFTnbn zqD7{XG|0OPSV36GkJ5bUgnuH@{5Y5Z`|bj_l7D4A;y*veE`UMtKSKHyZdO1C8z>Lf z>sU`<0gv^>w##;g{MLl>1}MjT9-r%Bufkmcny2&6vM1(QUKZjZaTPXK7j3XtT>*PA zhW~?Um=CkcGSFb9r0{M5=xi6!G^U98H)j{oT}1QoHFq`9MMR^$gni5hWdB_o+1o_8 zn4h`5TB42lG^d;B2GZyAOKv66e16vLIfcFRHn!|gO%26gEaPxGjZcMv>WXU8rNs9W zSb2bxg%*VO?M0-d;&o4u7pC=EnnrgzFxR4*}(?arMzT)+1BAQ}O zwofC`yiY}a?jxGdpSt}ly*^m)(Kk`pjEDM<+ixRzF&}aJ9Yh=RSI!oqDTZWwZ6un{ zkGg%AZoeW3|3RYd6xm;PNYl*iw`w1ulpuip=Kg(IF?JCo`CCyw`{xJq$(=;~;jafD zD}cYtco;56`xk&A(;Mr7m&3nhJPdaMFE1#M!`GMV{y$9dw7z_O9PVDP`!~nk>-Bix z_H)Vqb$``DtJv{l>+0zIfa+Hf%t3umH}m?UJyr*EusjYwm?QhIFrc|}RerCnXWBeM z9rBz7|6d)<%^`bGlJjehM)JH7VZV9a#=lU_G^~d&z0N4k2E?)`JLro@hOd{ITE7!1Twac;kQX*^EL2SdL&M9T$(F8r33 z!(*RX0OMi%Tuw^7ehP**L%w~k^e!Doe#IeLUPv76>zvg@`>#L~p*@R3G#wiH#UYvw z%`{CnW}2oOGfmUAqAX%eO*5eirxJ(#wm_QZ z(s8OEO~-Nic|spT?-HK3Iv!8WN3e;+@l{1S4*TtMMa%W}<9HgDn}+@NIoGmyJRMVq zfwu*gYZM=%{FFZ}Cyes9&*f*t>(g?;te=syT$E0he>s+aW;{(3<%d`B^4-;-<0yYy zpi-mtpOJq9mP>K^S@HF6#L#ZWmun(_WuKNO66fuc_qF))({jbohks_qmf`Dl4%wx6{!UZ0j%hJO28=8AZnmRAOj_AifbKU!WI_S*ur8m-4w{%rxLQ?wUY zXwUY!75A9!x6hT|9Z%oACZ4V=)@k1U#cokQVXqkbc|CvFZwt7qMg9u+V5~OE-#%Bn zR<{TF;?P?Nsob#N7HHED@JOnE@5fYbw!dy&yna0d z&HCFl3mz(%{Avqq(-1I73U{w524&zq8XQvK*zdLlIy3~Jpn3mn(J){Ln&TTazBd=`0r?gU z1Cj~9Pow>WY%ky*%>&AyH*5TCg!H-nIRA#cgRf7dJjG|FXn&YMf3*clszrM#dfhtF zzKYJt5$ypQ_1_#=g1=Ol>zkF;1ezy~C*HdbgwYEWZ9S%C~Wa z=pTizsG@i^@bYrep9-(Mi{jtF@4j2~zrycYMy_PwwY9=u6rPC%of+@y68@&}mG|6j z!m&RXz=}t%PpN+rHc)(X`|yX&0Y^6`2tlL2HwQ{uslFWN^m2T4-^h>bIn;iJx7V_z zsS4-v87VZs8TwU8PDS(l?p>9qXy}tYsxRNaaC={?qxR(buL|6arG?l(3AzNUJu|(6 zsr7d{>(tk4Vttk!%8q>v_B45((hU4Eg*2_-F4Fo^V=vi2+dw1#<)Pec8ej1D8h#$5 z1+tCxAkHUhLV3A#Ud(8{iS;h_6M0u5Kht!13Hx8^OxvF$)~<4W+I=Nwn%$9_4F+H+l8ExvtL zKbcMK;pJmG^3$HoVW+8msr|4%zjpl{Utj+BIJKXEeoZJh_i@!fps9biVT9-X4gWk6 z;>St9jh&07i}eX@H^Q-OdH$9W%FX!{`nl&~OBnpG@LQ{5=#Mad9^oX~`Aykg8E{IR zPpl3Wz@5Vniv0_n&t>NoAUpDLabB3GW#f_8kDqVlbN}bH5Og-}=M4E(tB5vmEt_l; zkNhn=lxO%e^p}V7c>8jH1zr;<$P@c5;h#03oB~t_+3WtA^94-J>=ymedN53uk%Yg& zp9?tNAn~LL0h@3N7uBDwNx^IIi^z}T+* zt>X5XjpxU*19^P^7)u0>{!R(79O^&zE5s}8EeYhT;(bYr0?@jD_aH8ie|8`zM)OxL zjW;M?Ug%lUW-|}YaGLs$+AoK2*k6M2iD)DT_3!dfb~e#@S5O|R56a8uLmH2G|7Pd$ z^2q*N9?&R&^!m2W(`ZUbj5o_eruOIULv;8t{`V`Czja=WHcGxY#Nz>Riu`UAe{%oZ zI`87Sas5kFegrs(>@BkVJb!b3D-3yzhjH?}JT#k766JRyez7-?F;c^RFs{7FuP{jU zq4NZ^+M?wpTS&6J6+u3{i}M+@$F}*;v;H>)NbF}Ae+yv$b4K4+h{X=r*=fu8a~sm4 zaqfKHX@NT}aHj?Cw7{Jf_;M|<6ylh9Pk(`rEth{w6X(mnqfaFEA8Du0{?Cs;zWS4C zd@1mI$6kybKZX=Oc5EHn*4OvzeV_a;_Pf80{q z|5nhpQq1xOZd+pGbQ*Bl%pI`RLCwG)&vXFSiqaCV@NWhjU>>oert;`XD{?0Omg9EK ziqoF8eo#xc4KkF;NG6*$t4;~TY$>Ubr!v_SvBNOY2RL1y5lF7j&Tup=Wp|rwrC4pz zDvo5ns+XxR7&X1Au1leb1igvG^Xu*+Vv4k_J!VvY<(Wys-gKwrf1g4Q_qv zNb*gu^t9x9^QCl%N6%(VJzHh0atc>>tDcfqcZL5W-_1R=KhxqtmL66(wLVW44^z0> z8Z(9q@BjSu+~ zhs%5x;2`Cf5-t}GM~~t@zs6R(= zF3&LR&(sJJ^}8meAYvGOh9SdslxEEMr2no-vXy+i)R(7dM8R-$P4Y<25Jo9;(nnOu zOu>CI{hv#{|3v3(rexK>3mBFU*Q79=4`_1VuWOuoe-G*X>Ip9Al&aQ#VD38;*y#)6HwW$`uO8AB{j$i3ZG3<-z_<+ zjjj4Uz@Jz|KPdH!b&KM`s$VD?WX5BW`;N+AHW=kkdjYHdlVCvPn)1I!9gF1RpN5Yp zebtZ8!Z7hXQcm42N_|DE{FR>S|2Zh%NPmkoK>8$S)&B$VsEQUhWnp!USqKn`^r@f2 zxXcM(`QtVl%}jdUC>fNgOw@k75Q>5OpTO&=Ok_VUNqtWo)8b^nUVwcne}hF%<^MyI z;l=uI298P71=KUlCQjcksGrCs44p|+{o{l{KZu5-{E_0IZ`BLtIdRiJrL0bGhyH;@ z^4~2hB|gBpF-Ny9o%v?D)rPN`Ja{@>f%z2S@TzZ)CT?i z(DzvKKPUCg<){3Wp7P&ZFc#C#Xc8H3G38GoY1OBfVPe*qK{v;>|4ykd)%BaT{d=MR zFBYl4(sxP)4Hl^Xt@%^jP@Agvc;PWFWl4Q?SNg>M`|6QP>hT$u8;C;5kau%_R8|!i zQtFzUhYE#c*5fP{&jQvzReY(t(%VztyQAu%=6aDDI~yt|_6}7wOzf)a?R&U>V!N+@ zc<-*AJrzTf{e9y#4^8{(8#^0!G>i^Swgd-u_Dl@T^lqQn`bd+nzvt13);;wP*Vl~n zwYJRf>aUm>Zrw583wr}?kZGQ67#-;8o!IK@>znDRn`j*CY;36;ALt(#9qtdgh3>$} zK!3#}!&@eiS8r!yMcwG&WX;@gYm3|0U$+m#khl{;nzDo3dtH88p( z4?Wy4Om&;xwX^Hdj>)#sp~@X216#UgmwTeEyHD1kX<%mq^}%B0pMJQ0 zxOK3#ZgjY*bKa!WI#ks&KhWP2TvTuCBYkM^@m;+&W4`{Haa>bq_wk|0iOPYU?eLG$ zrX4%G+(VN)Mh6Gb$7u86w%!&|AH2- zExr9U?#@Ts(bqkX?i#CpfSlDkIWZQ%+wA-!8+{vVM*P)v)&8O3n&F}9O1FQowq|6o zVyL#l=M{28Vz^rAg9!=_ZOh+V4ujZ48)`M_$>toMLkqw8}d{H z8$VYTyl|(u`y9mQGZ$y5`GttZ-mkHpn}*UNX|a8=ke7=*i;mrtuY!cUQJ#1{tq|gq zkWY$df||y)+|)j*UNp@TU-`diFUI~D^-?^uu@dk}-XodH@1tnY#pFA$!~Y%h7n7&` z(PI1!*Wv#Nd@9dNydM}-aYcX6Sj^79!@`Z%S9XDlE0SGI{ta2*(-!$L$djEJ*)>YH zQu4!+-vpXck+kzG$ye?Cr?UKfu?IO7SN}$q(=EIBGPi=SoJu<`*@24R5B@S`1hUV7 zrWAyI;vbRpuLYZZ4|n3zLYC62;>O;>o%kE2OUs+yHh00E@*Pr9+qRV!zFXSM0bjKZ zvA!<(x!}t!FXw#2#5Zq6e<1mkuYURqp6WNs*UT@L^(6bMY-ED|-bKl3XQSljTkLpD zcFd zrIJ%-rt($3h#&3qr1Dk#4wJk&U)ncQvgZ4B$+zZv$YjSHFGnQb8ZR%I?5MI(e&g$rg3zDzutN2$;^5*(}F8S8_I%H+6^*ti_DqpwE_c_T| z`6~WJle{_K42{&l2siW44~ev{-CHX-XTZtA#d&KE5rMMhsK3e3)-XCAFL1J1>%EygMQ)) zc__wlhkZ@{xv?RC9G~96=qvY+%GVX&yoe*nk6z#9oubzpcXaINY#Q@T_=kd>)5Ciw zko_(026-P$O#6nt!D;U>FPNwhiYLk`NKqGa4O^xIO5n@p#fpb-q0fAQJ+t1~8UN6= z?}ZOc5Bq{XR?a_#LyBJNKuwxBBt*)GW`h;Ax7-?EslD;7ckyDcy*Ka4E4(J^AAOH6 zVslo#i2p14mVPm~{sTS_U^0n!?vAkBMndeqM={uE5g-@L{q&22JY4-uxM@>ln7OMS9! zPDiA^+Ak|i<*(>rJP^;?h8g`r!YXg2|E8FvgHE&WmCpS<>WW!J~==Wq+ G{r?O5Cw77W literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf0ecb78ca25035a27ab95b3b3d67f2b4.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf0ecb78ca25035a27ab95b3b3d67f2b4.o new file mode 100644 index 0000000000000000000000000000000000000000..fe89704789e92994b8001700e8d57498c8bca17b GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY5sR!clAVT>ACn1^}jfozzGv#kdR$$*4F;<1?(-BN>O^yt%) zOv{iHS!5mCU@&8LW1o$)Wa7!vGGv^c%s86O*~l9o;y9SWnrzl5lXF63hDd-txZ`zD zV%+=v_2|C$szHD~@nb3IzPJAWud4r5_5W4fy8V@vYgXx+##GVRODtYfP{y+RtX4jbhEy?AzR}wa^1H7O)s>4Y7S$EMr^qQZ{c;;mb>s@a6O0DZ*h;N(-bXTuk8L| z`1e!T`1_mm@Ow43@D&?7^nYz^;7Auc1lq%WS*+yyxt_uI%h?;h*4d#s>3wx)Y%>N% zY}2T(!+n{olaYPvPDL*6I~5t*_fF)A#cxOMX+IY^cJl2=+T8oX zV>|Z`Kk?*~!|99f4KHZ>Zsh!;W0A+-`B~%%=j)Ny;};_XYu*7}NjQAq!0>~|_YXh5 z_*kU+)bU7b$EnD`{?{X8_gsve-+2)<%feO81H$>H$HQPw_lH>Z9frN zaP9*5x)7N%_x|u0;a7pzo!}w)w`li8wA(ja1=!=sCn95apNphwxtD*M{I$BV3oisy zAIbeeYI0v)>dUzyvZKewN*u6}ruWtP;8W8z&-fYJ^Z~+c(^*N({=f?o{;r@wN!mOjk-cm~b`KjtrbyU}{>+x7pTF&DekgicU&rfC%5L$Xe=)p$A3`A%Ftc%|T# z$y_w-gtF5g?C`gBv07mA39Dn~>FK!Y8nfMI({vkS_9rP*7ad zh<`=P3hpbscX?xD0pt{NeGbIvYVijB&7U7Zg8nXl@F7S9D+T>+p9^n;9Ny1`AkBVX z&yL3JL2tWXSjP(RE-^tc&6;Rw2zmo8e9{Gio!y>4s58zG5aiser~3)vMm?rY6g};#kdx$c3mQmkeSG} zFSpW=TWQE=EiFvo`nS@MPg(YETDHhKR>1YAA)ozyj9R2OV1u=!bNlu#e|G_EZ0vZb zF@{;Ya&4o)G!|g1*6Quz(L08xmdIZ|S2@|WBGUpOZP+ba%L)!pwu0GTaQJN#;G3~o zxA7gj8TA;4yx~BrZ_s}&X{gJ-D`OFc0q;I~1-!ew_Lsdmpsk-Cez1-TroWdrw!lyV<|JwZre;SW~lRLu13r zx(!t|>l#;X-nepI#mWk{xW--VUhEAvwYa<5aUO7=*NvfxS2|z8=N}eccd|zTw=Z=t z@`M7q&lLB#W%jK2{q-E@*jmz=eY+iYLEHQ6qVL;jbOsN^$T4 z&bARRdY?`3b2}<1bMZRsWQT%wJ6f>l(18p#vXteAY_|^`{vwO&2U6Ip2id$(lI?dh zU)#y+Bf0jHkZvP9V}yr%p=r@v)RJaE+Zbte3}V*^bdX5NSr|Jdz}A#$Y-DjKN-?q;0OH?J=b4Ww?_P+u9JL$ z+K7$#O$kkh6}f$r6kgCr7~x$(=XByX{Y*544QH^DzeNA2b9rCm{W&vG3;56<`hVe8@&Wi? zR?rqDkBk-z{K{U9T_OfF;UBAmdInXQ_95rT6%@X1`wjGg{`H`ZISEfnAp3^q^5LUH z;D02;o=*OZg)*=W`T;&Gls}i&P52jYpC5ubM(ed1B;Rl@t9U+0|0)UIeK)t4CdMlX&Rf9cq4uXTFh3VX*%`E_`BX3) z^@UM(QPzF$Xmdl07SMQ$h(FlpkzU6lKHu(~ozDKs$>xR@&LjLnj$g_i?PUu|waIs4 z(HSiTy&d#e+lc0aUkUsy;0pwPRJ5Oky1;)8_-x=k0zW^4 z_n(crz^7`D^tyn@B1+@82)qk*fltzfJliBbSMtA2;x*uj|A4^zmcqa8zMJIZ{!jiD zve~#VDSrzsSU}>$@$=@z>aXzm!tKH5AKH_??C78K^QKL&%z1{QgwIe_&o$wvOh7ll(x;a^zqWBx_{GdqdKo6Y^_AlY+v z%)dsQT1jwLQp{hm9?T8Rnnm=Xd(qy7_R8PZk$z{nO#bEo-Yx8TbT-F3 zP#5jVAMLYK*1EB_oRhkm*RUe4!<`j=?g9{Qh4 z>Q??i`sbfW5gy}JKD*r1U;pl`xW0WI7JbwFHt3tr*DS8@EU7>Ee^yN2zFe#qE{nch zuDHHyNMEk_{O0om?WMlae^vezE@2{NM(Z`pPlOOZ~;xgOO5JnKR!c-(>pg z-;I3W44TmTmG?!+Z!`EC$?vfEo1Mo`JIQ~N)-$`vA838>(RwM`tN1oI?ewtf7WHLL{$2zf;^zyhPNNQ? z-a|LC0DqvD>9p%fE}eQCpI55RvY{=P^K zN$uYw`H#foJM;;8Eb(18#P{3;@fX^~h3fb~n4Zgn#pV(IN9ol7|0B%#8yG%(|kQkwNbR;{sDOj z&)2)G03&%z`hn*I2>;34V@u`vv4iKw_WY0|b^bp_bDiW*Xb<`McznL_x{I7oVt9L2XbS2^JWXFCJ_&UrJ{jqj5ubv3 z9B*0A`1)pd$#^oUf7dXq*W5n>+h~6R|9G4xCl%!%n4f1*CHunu$oI3!9Ys7O`_GKq zAO3HipG!GCwPQ!;Lz3q_?cW?pas85iq$XMPn@vkqz$fAX_=Psq{H($JPC^|DkInz& z%s4&+_!QKQco$ARRC~f_qHe@zO*P}Eq8`UXUhc14ACQ;oE^r|9W5ayTpCljWlS{;x z^EKvp_B`^(JuIF48~!wfy;ej1luj)I6#=`5CtpI1iebGNMU}^c*<`=;XHXUKob;WH z?T9g+Cnr%t&SZYPlX@8FWu{gw}{o!le-$^^s zK$%?6QkO|`!Cz)jPp1=;Xdl`CptdTOF9`h4Lfjb`A0PhJ4}P>`cYVM+;q@fO`60lo zm@j)AI4VG~VZAts@%IRO1bpvKVZWky`wBnbQ~J4Kd9YvS>j(K8U!PB#>}k!8Hp?FyCAb@<)EY z0{=AUD@%bluFteyS@RXc`ippxgY_EAiHP^mpJ_eE`fOUiQP0789ivBktVi+v)`(_a zuQ6WKW5JWHtNEBKlL%@E94Ie^OZT1w=`O#-LA0hww|1uw0WQt!C_urIVpve9A{t&Yhgy$bPzpw0h zang8o#1yX#=j-sFn~7Jk{WsP(^L~3NK3|w@zrBZxqt+)LpXo@AJ}=6ncyc?R?=z-B z%8?A0DL!HSH0`gqQ+!J1`*R-8kRN64vGe>1`+*GhD)QGoNy!w?Q>JA&C?6^b*(qEn z@q8ESpExf%vWX>%XNVV>|Hk`oQ~W|cr3d+^53$c52p@}Q;6D@bjP^s(Y?0sHcsw)h zH_Y)1Ie42 zer5)J9>dq?v-rGm_q6!_{BFKK=jW$9-+pZ==w}%A;|KA1gM9|yk8}U{540bTpZ8!q z?4RQ2S6Kfi?#J(#b?x{zll%J}iQ^mM$ILtU{!8u0C&o9^`A>f6j+vJD7C-;7?r$-F zuVcSG(-_}MLQ{2y53t1kkmg7B)M>Dg*nj6@|D7|HKA$t}zh}{Y(Gc&Pv>}P_zn#uW z_TM;Ap>i|vuI488->PZxHxD8)nmhV=)Z7xVL8ah_o~4?hB;>o^ZTit}(#_9efI zW0jT+%l)TEhe9n zk$n7ok%ScMtDPK#w;#=<{^{cu*`F3a|1Y>cdClhs8SG2oN91$wAVj9_U=D7M!(KcY3q~zY9OzFU?7d1 z{>e*!{6q9jJ6l=5e$G*SeSOmUdhA9J(tkFe;(8%{aa7afmlk;))%b&RfWJ%tC8##V zo_3i>=u#f2<8=*xvkcy+IMe=s?JZ+_wG{~Dnva1#Ui3U?vc~EFuxTEVc*l?)+-Xyi z(Vj8@O4Y(Aspor$CqBmI zm%#}2MK95g^ESoG8=A2=es7u;Pw9@n+*x#)ZNukD_?~JS62yKj87#+2B797-uEgyr zm*>Tu7?0#%6flsef1c|Tg(opy=Sy|O_f`j|Wc#@1_Est!%ZU>rjB zq#}AY=rqM+@dr-DZAmuSJHeI^nQ^-r^sc7}y~pIHkrLoza=Q}aV{*HN+==>#$puD8 zRwWsRL@`Ka|5zlmPxL6Vgo!gr6XOihU5m4cbI{DWO}BEMQ0KbfEH)}SG1^|}y0o$* zEzyh>dMZO3jaf&l1=OI&33%d$bg*96*WQh}=jI;NPn)xT&0_;r6(c_aR_fE!^gKuM zANWe4P4F%nTdgS86(i2m)ikL3O}qld+waw_GP?}pCbDhJGeMUKudG1*OqAL%@&s+7 zfl=GZi8#~K>ztSQ=GT^@UG8TL4?E$7hC@NTz#nyJIKcRds^c3irut*3D<+8^gX&a= z(M0?~oo@Nk2Rzs?(Z6Y_FW1a?kELE~sndAWKVKry|7LNbO&~u0HS6RNH`?Z-(wC-J z*l)CtL~X8z_tShuooux0>1nb)toTHIesbJcxtpwRoQeH125~PFZbg26O}|RiUQH63 zT9dXn%}YHcHZR?R0c-n2+P2bs+UrK0=?P+xmH$NaDi4r44j?TxuU+Q)W56r3h_%aR zS-beRg$-+$nf^^n`{l@*js1Bn^;%0E+3MBTF5_x1d~dS&cCBbYjw8L~2h6U|MwJIt zX!SJexrRF7sm_(DO-V_zUHV161*YrFWVzcKF(y@lXjFld5@m&GsaI#mo>IHS` zbq?y(73rEr%~SRGKEtNdD5dEe?WL}V(ynhb%0(vAO`=?dIK7cvzzuc{I@S@&0ErjU z2(xc}t1 zvD$O8x^afxis51nnM`LN6>rq}tr%`z#YL=8@nK@zp*?2elQhffu3J}siWi2RoTgny z7Nd4@GUBRnO&0o-#3dRz9-C}Q*XDk2(ye{ypJuKmTHSAS{@;8w_&7$la+|2no9EEl z`=7dQnyJYXPYbR5B%-(MhfMj(?}&j5{_F>*G;x?}Qzj2OboU*?XBW>Zw`#hOV!xz+Rj2 zvv2J(ypImUvD;Q<(h>M)B_8n^79Gi2&e4UBoA5*8`*C9XAMbzS$vBt5(7&$n{`(Yf z^u04<{}Qjz{LkFh>sply=Lj0N^s2VBr5N&4 z@cf@zNbSZS)fME!9@XFp&;NOG6=T~f@wer8UG4clq#&{dT@yXqhv2`WHxwIAf?{8- zLtSb2e!%&`Jg2()+XCJPKN>O-t>Ey9J&5OYakn5nu~9I^qecQlq_M`?)cVeOOJ)yfIqLKQ=?(vn}u z&&0)&DrtWp+aEKurCY1s$$zN5$HGRlG`Sl%D&%>@yS054A@TK0af!cwRkqKUAf;n# zd)m)EnxOqrOMBI`wf!8Skbp-K$m>bj{ykZz@#(uJ*7oH0=wbr@Lvo6^4ejYxwO4jL zh>i*7A*7+4u}m>FzG|oJ{}38qtG%M5_C#lGPxG8qU~yB%$T3m+^2C+e6HT9}#@_KK z1H8}j7)0JImsc887XF8>hB|@22d^WZsDD$D?L8)-#mRu9^@aE^v(VuY{=EF$&~YOB z1>l(Y`#Q3p^mx(k%%6+vvOC2!2N#mxM1=ak1CYi?F`=ey5kTtLSR`!>v=Qz1CK&&# zvVFd!r17a()W`g?y~i^CQQ2PkFSRF^wzgk}$@@ft@xQSZ#3*k5&X{$41V;EyKm;tD3jQ@rYM7FRblU-6I{u z*p7qbd{T7ks^*uvs&+d{H#BsWt}GR$d~I1#TTfGQS=*N44fPL}wr%uoYVO&xxvsFO zeN+9;lJ%Y5((<+C4P`A&?W?-CZmw%<3T)WeR<&zYQ&9_G8|n#P(Y9j!md({IzD*4U zTQ^s)@op;F*;ZXszhPJ9sYxF zWd;7Nyx%pOi&pjc+E)~Shk&o>-ueg2);-v~xvp!==5-IRX0q<5uoi=Pg~)xw3M{MoCk-rK6#Jb8*9i z*F{%UI2qllt?i+ft;IFaM|&m7ynQR_uCTMBXvcbQ*}B3l?K@UJSX$S+Wm8Eo;h0ss zHg5snzRFb(H}#gZLf$s$>S14bNr8WJS-{ushkY8VeI4r`UQ0Gu-O%bS-LST-V_iX0 zdqazFYh7pg7H@rhpl(H5c}r)-rX983vhua7D%TZmtz6l&rM<0ZYb9RAYkRh?t`0P} zH}pb(RqG2I9&TP;-L_>@5$s)8xTRx5ealv1uLsMjx1;ZZ=FQbTR=ar10}pR$U3S0n zH+5}nZ)*)U?%ZDBZ}Q!@yvbWsQczstExOO^TT$XG_7ykZd*AjV-?BzQ*VMY*+vLC6 zPpoX@zpX;Qs)AoTxtU7AHJ(jgSiEf6vf_dQ{Co<2MrD<^t;;Xp%j@xjGJbD+L8ySS z-Fcts$0z6gnS1w_zYOCPE-B34db)(_kNQ}Tayd_Db}&gVqWMBmH_#M_=p=xMCO2Kraj{c7itDc{tev&%Zs z{ZQd>J*n<3bh7|It8lnhsQU`1>uLR)cEa;N{FQL(mlp#*-w=%ETku3*VZg_J6={w0 z#VItvb7G|>y0+_}Q{7vy&!erKy{nESOlbMwI1^0Bp_Cm*5eu65>VW_@i%Y$o;5_nh zfQk=PN%Xwujl63#N>CC#XIjwnh&Z8z%`mP)3i>!dX1Lr5DIl(b-njD?$WC0c3_CX^ z7llcJEB_bzV(eRxOTk1B;+?)!PV^oLRDK_ULKD%iy$bw~XrG9l_BV;(_gw}4L%@lj z)7%b>sd=Sch5u)fcoBWAG{GxU4jMDjxUWfh-O>bVezk(0`ne?Gs%?>kt4(T!>_1QT zQ!7oT^!)dde>`1^7d5Y*mVCY_;c7m;BKsMV22}7(fM=>f(bIa=so*KT9KKD~|51S1 zw{a!<0*piDgt)Lba3%O<3AanQl5Z;dQ@%r}X%au(0xyv8nSiUd1p1PM-vPLB%ecn? zms>dwIV#~aE>06;<(RyqagFc_DJS)>cq7s^5^n8hvxLvK^rQ4e{TNS)`V9Tkb52xw z#sD|cAG-?tvH@f=m-Cj!HO?28gin|+5>EXnUPyp#2Dm)p;c^`|z+->h1I#ix zn3C7csTuo%0dAD5MZ%RH6n&e7TjhOT!maXF$dIGPEs*1`F~E)Eh9q2#tLXPgxOLp` zOSpAh%3G*V?m|;a>5k$W6$SV38e=W;s^BVaJY#{Y_%LLFt2i)ffvcrgoqwwS)p|c4 z{83TxK54(z7PwmP0~WYi@0m!uyBdA19o}Fs!ycj8yT;qo(bVz)|H;Xfcpx)=-I4#s z6a8ExyWX7#b=26}(TbmW^mTRJShJ3vcEM1qzl%-2$&p(J{_~QPQ$OSncHyTXq^PaF zAmpy}cMEax!zJ`PmX+)37vuNjnsy4RRnW%KaaO&O~cxXR&SfXse5f_b59$Fzs}tsw?SK{x4E&q zv$2^oCIkX|f=^KuvKS>?)frR-H#!!}Z@vzjd4oH;8oL7iris5R8tiQLc6-?p{w^F+ zSZNI;so*FgSB&s0C%PqE_0^aXOCQ2v{qvwbD$dlG=K||5|(RbI-m6 zIOw$B_c@+lPVQc3t-bczYp=cb`<(k1U45C`NN^}2ia-h4 zky;_PAM@1g{u%D)JvA*q$L|Zce}OPaOKakuf7bU~_kAt%n*uTR`c3Z4YfC(tw>)C| zYaUn2e|p63o&92aZDD%pF;6MQ`{)rN&VQ2dOBQ5aziB?=CC1*m@4#dGJl#L~a!q3I zap2r8&okXfzxkSN&))IMpFcYtziaTj%bgelj}hG4uh_O{^vcg?Mz6x})%eY2S zpZnG5OLr2@`I*&_=cQ{NdiDyWy&H7RNb|vm4vlVUdU8e!_T%v9(wl0 zk9>Z#<@}tsT(w(0fZ3h0Ok9==*^((&|eX#Y{pqZQ5(z^B8E1N#@?01gu z8GUf~W20Am{`;dZU3GA@dFw0qeR(w0d`afTqHWJ!>Hjd~d0g_@^Zn5)U-<6m>ed%V zTVD7jXfDlc*^RW;J@o8%)_#8UrB>+Ynx{vvT=Ij_O!Ld54?;ho=8jCrRdX`5?!l!m zzVf+bXsG5ZA%A*lXn#$L$~XRMwmpb4a=Ft>my7h$GFM{kh^K5f;SwjZ?XNEo&-~O4 zn*i>Jr+oJ@;C`lTBk}8O`%~`3UYCn-^7%1$@G6Si^GddTKc3fdyj+~q*^;(R`aK@0 zeDlQxtA#f5!{bu~7OG zONWl0oElCKE&a;2#8{J8M9QM+rRO94+b?#FiOV1JKK9|n*o!4QleeAc8e7@o8e384 z8teV{;Y&NOdC~hDk7xHA9{27sj|=$|yElB(Hzp46oVqIM8hh-4#Mr>Q3-+#h)c4lF zmH1tUXZI(@dVf58Y0qxoTdVE>d|zU0)ou8_2fy#bZ$EzDjoAlXvG?@?u{Y}xd#OATZ~Lm{i^f*pwrFgX-!-=C4*Y(mCbN1S zev=fg&aA#0zxPsjN@n#3@Ho4elIX8tO?e4e$Kq(hcwSjSV~)-n;5z-&-#QcP8W49QRV52ks5;?b$tiY5ba( zygi>8z7#t6V9$KtTa>>Z_q6RRhFi9;oZqtjHh0VR+rQat<;|Wi%ID8Kli1cNf2&_u zIJWAgg=2U47meL}4B^I#%&O6aW2;|TICl5%7LMKLUNp9`8ghOB`F|bwd-3}Qe(%O_ z7|&ZKW>ytLM|I_y)iJ>3xEsqdt0v+1Gb(QRInwP41n^o!1`s9uK5r zJ?oGM*eKat&yTz+9}Os@#_gZ37th@GXyf+Vha0!gS7k)`$g1*@`KYw=vHY@cUfMJN zo0l$khxRPL7U3%?9Ne?~MueAAIIw5=`w)INh5dV2=a6f|-QKZP>%4CTT}rMZ$kkz! zE2-tGgJ?`O4yF=oxULlsfh;}iAHgWavg1z(9vu4nvbuC?u z-__uA1%AV!^wOr<-G5Wy5_@Yxg4?=Kr}vq8F8Cvl`HqzJM*qVZ|1k2@a{W)d}C%!>kXNicLg$4knw_MAMpOnmVg)|{`5?z z(d)iqPskNL8TW}jz3$8R#C5}0o#1@7uciF?Gohi9ld~>zXVyb6YrZ-^GqX96nKj#!X+^$Y z^oZ>(o#L5pl*OHOonv8F?BsN`d&ct?JRch8`FTV?;Z*vIiGG~tVWKZRmA;th$9Wzj z`m$5$%ZYxR=aY!O{8ajCq95mZBhgPhmA;ke$9aAM(N~;G-$C@_JfBbWm8a5QM)c!6 zzmn)Dol1W#(U0?dDbY_pmHtMeALseIiN5Mo`u7q2IL~h*`YET<-$C@_JYPrj)u+-Y ziT>2*Vb~e^S=6M`7E?A|mtrL|5jRC)n~rjXU$=WZ9nl zu0Ama`t5ce*>45VcdA4hlaSVAq*;Zur=ZVNo#{@CyN-TdENc>py)~7tyN2Koo4kp! zV6WIt{;y_=xNFxdE`An59>Qgbu^F!LM?P7U7~O?b!S9;ffMtE#TGda=$wC1+lEfN8z?tJHIHFHHQ*o z^APU+pUhj)t$5Y-r5s)D-zz$wc(wB%#ImxY#NMW;P5wet{-ZBKemv9i&xHI>LjGRF zg)DpLxytVB{4%Gvmt^hz3c@NcoqG{JO~SQacn`haU7 z-%-F#ahq(Vrfi)rKsG$nvb93CpF+0Ue`MY)o9k-`D?QhK9bqNg&^Hi%8#Q+Wk^?l4YK2>8twzhd?{eRru=B zZvuz1p**<`AWvB!Ul$WtfcX?@rJ6T*}y$liGnVU?%aZzHVo z6nqL{m8V_wT;-{g!ld_+otEB@;+b9VLx|gyu*uk9%INyvkP*+cjI$tP7BcRqG?h*4 z`VPWM#{Gv7RxD;f6_4PUaTU+)hP);4<>VI^PfGYBjB_S17EUoVBVe5i*+ zXP0k3;HDm%eCM0;4gE9Z!!s@4c_d$PV$}64^JTj|N?|3R>xT#{`HoUpmGM*mjIffg zm%>Uu7lpNa(BFQ*cKLdL1o`@G@?B!exBnT)hi6*8Hpo{4`F0_m*5A;-Agtuu^&G-V zzM+3bSjo5Q#|SI=TohLF?IIeRe7(Tg<#YWT%`YLWbxUEbTcXirzd!Nn5O8++YJUm&p0LUH9#g)f`yn5mY55u;-zTYV zBc7J8^JRpUe7(OySjpG<3c^ahre7nhh=ldE9*9em3*#Igq3_pDXips>OT-x^7T?!$>*Z5mhU>xtNQ`l zx?h*7o;X zgth&>iLkc6w-DC$M`3M$M5E;!Lfr<=E?@2MAm0g_d>c*qj(!pH;hC22BFHx{0{cTe zEnjCAVQqiHWy#m+LRj0M8)1^q<+^J>J=gX}VaONW`5~(gyFhD~aTnkw(V6RWZZc(b z{T=Xlre(YsGIm48qm-tq!}~o5D;bY^5mq|f??YJ0_?RDIrNd4ND;bXxjn-jN;?-Ti z+2!jDApNLKzOANwou309&$N7VAzuyT8$vuSUvCg$CErj8VI^O07-5o+{CWYxO1`7? zT*)^?VJ)9)!K*}Pm+vUxrnpVMkDBrgeHM5;)AC&c`92vyzp{||avdHjLRiVSs~BNj zMrL87mRiGE$giL|=S}!dgZOlZ*|ye(QcbvzKe{B*@TZlkss= z#-Wb`k7ruOcR|McA>%H@(|R15jIb`(Dui{pPC;17h&f|RMi+&3xe^V@C`Ojydt+O}?+0^6md9@OY->n?rh}{vqO7dTc;g>v1Z= zT91tgYduawSjp$2u+}5dX!$xZ4g}6F-%u0MPut`>V9Mv(4m_S|`Ob%YpM-q-5l_pv zs~KS>-~JYam3+IVBdp}xJOg2pkH#kyR`Ts98ZPGttoqUmyj@1uOvo@|lkt0|jJ1Cb zJf3M8FNBO)>cb+Qma(=KVI^bdEQFPewQUG18H2MCRx<9Q=Ss#-3Tqvr9d?1vF5fP| zO*?J!{m_)JcMI@%rsZpaeBBu19i{Yi8*(1PO1`6W5LWW-KObQw-(&3vlYD6BIw`E= zJ4!TKzFP5W?FFWM`vEsS(Yg1fZ;3H$ZsjMYj6)v;4W6AMBjQ>zUWl-kaW2AI#)}Zv zGG2_Z(qSiswTwgq8DDx*jD;`H`JG~Io}UFp5@EAi_%qP zK9oRM$+@c&VI}9#e1w&ps}>-v4NuYTqn@&s-~P4;>j=fW-};YY>_cOhJh^?o;g=i~QQk>EAG z9_%A5rjW*^4Bw;b_Qu?#yqUF}Cb6I!s2eO*nFn7svd``VKCVIKFrn9qF==5_xJ`f#

    %dq7?b%a?7X1T=2*y#36p^ zkTbe(Lx)cIJ>rnx>VPPvdG#RHbWX>&8fl)5276S-bbe+!pojgQ6VHlsq%S;2dLMM~ zX_SQ@zX$O<4E>#{oONDf4(T-ia);$I*&$!}8%J{f+JAO^(|YZ>l(Wv0)>*qlS^g>L zOlHmhJLgER<+&m3ES;wv^8bVb9sYN$D_?e~zr_yebh%&UkiN?y-7<%C4Gzlr53MVA zIF#iL_-7}6NbBr(4(UA((+_~|bnSxnfh6Be-n09TKU8N^9Ll`Mp$@dpzTlAl9~{zs z)gfK@9njffhqClIq|-WE;gFvC7f$SVE$E!s*vqR@duiO3`9r}tNIOJdBIlB&H4*E}J4ruIs=|8j| zd)Q(AdK}W}a#`V!ezilowGQd-0^Qqf%MUn|Wr9OGt+R;^>8l*l#U0YU6aASB9LjRg zcaHVnA^i^=()}-obpHanx7*HML>f-oyk}kKrnC1u%;T^_x(_*|dk}PQx6VH9P?i}E z>2%#}b4cIrknSRfbUJ5e<3dl__Zx6Xd& zP?m=r(&=*F;gJ4ehjf1fIwwA54t&bnt+OvUl)2L(oz~eRhxAuEq`TH39S$$N?K)fG zP?p!wA3c}4dD0<0&M7+Li**D?bk{lP@BX3VwTTX8(e|T#?KKYlppN%hhq`&(A>9)W z>2&&9ZtN*Ms~y_M5c%=H>oDCvb4aK8cRQqC;gD{XLpq(l*3ZAZ1NHP?hq8r1=fofD zJQO>mpWu*gqC>i0yaV-grbAi2?sn!|w9fWBq(A78?vO*ezkUaFcEq79eGcig&ITOP z4?3iKzeBof-T|Hcg+p2D9nxu?O>;;;!y#RpLpq(l_HUm=J$k$QZu1?=cCKU7A2}S$ z{+mO(pMuVbfBo4z(BJ(}hcbWAA)PMwEe`1)a!9wsA>Chr?(Md-#~jKs*CCzO*(DC? zI~~$3a!9As*LC$vs7G(N4sUWO8@2K0GG_RtL;4dA>0Wb4hXb8&yUxN6bw+Lcx#;W@ z4(b2aA>C&k(tQqeZ?~O&-JvYh#-EGMXpHED{ziv%?{-M{KG410I_q;NOTORuj6?g( zj~BrHCDiln`>_x0;opE3Wy9!}ea7aa2Yi9@<) z9nwAJknS4}>AvKU4litR_q)e4ek5z3lW6) zA#Utb=eq}iu3z2oRhNj06AxmiG%mMrxnMgYrC@q?Iqo20+=<@=ZunkNBBI3Rl!c*jNxjio7EkLuPE>E$*el>Bwd=Fr!M!3Z{SVm3O zvqj2xWLU|0r|>7lT;YR6lZdjle^viLa-erPLi%@bt%UivyMNWX6)DlWZvDW@)^&Hc z4&1qV-EHf0@U0sZe0QrAxOM&7zTN@RIXmed~sSHNE})-D~G`uU)^gTRmO2 za$W!W3W?(*(A-75wL`&Qn{3>2f~z5Oe@SM;v|V)eS) zZ&O{1KF)&W>rb{Pd5e3hT1v44tlkRik3}#X4TU>__OX z$QghC)D8JlH{}1fZsDn>-_YHSeki)~y=%JLx&hs~VMT9u|JvDYsqTSWZ(q5tt-bs9 z4ZX<~$==m#x(CqKb>F&vP51JZ=)rccSc&#;{jDokbfc@?om@Gv0Yicn-OC5N`MIz& zpY4qK1_%O{y#wHbgf^^Q*Pox5!S*~G+d0%tzPmfQ4&vN4aBuf*=q&fIS$_{Y_GqC0 z)ZdcBmj7plDTaTh(#e0|yubDT|A0-mM%S-hxwduvZMXHW9B32W-Roe^YRqu`HH)t4 zmKfC3-ZiUx`^O>Bb+)fJIk0+Q_4;+(m9cmbcXa0ds5+=;Y^8;02Hu0&gZ%3vevVbb zyz9OEyeRLf5MRLIagT_c^8DL)?#34i4j+`$Bnd^)fawc*9{slY{66)(OFe&rCRuQG zLPQVGkVshnzl)z^wkYrVb3BhlZF2rVrgIgtR14aKqNh0*x^P}Z#CJxl=klo#A7uL6 z^SDA}6dx^;=JO3a_h23=|H|TOP!B0g7xJ@_ej|lL>2#9_W{1*EK@JK*&u8Q_QLxsf zH-QH-u6Na45=rqe2gnzN>7sPf!|A5{WP)iVTTYm$oJ>Nj9rR4p8R@T5{B*hr1$$|d zDnt=BQ6#=7z5nX;kE`_ZZRHP6|G{zcPtWr6FHJ|}+oTtp14M`-4ZE*^(t!U*CP> zqNUegvE>7bnU`L3&qSO@j3CCy~$fw#rxNG&u+W0H;$x9Rw_cQFXhR4r}z+_ zduIg&)vS9dToPot5#%t;oG*H=y$6MLP4vccVJB0c-rKyWH{gAtSEfCYq(hw)hdts% zt?(+iMm%M})ncCkPZ3`+;3|P#thV@*#bX9MN$fP>O7SrRt`H*zJW*^m;Bt{R;4+ai z;8M|Nz!StO1CEIv11=Fa8*o%~8E`~gZotJNVZcRVt^pT{HUlmYO$HnmaRUyC3Ih&` zr~wB=(12kF2J91Qw>aT)>+*tMG+>WNyX<(>3j=nEw6NpZ2#DVjuOORqbs|Abs=Kf# z@0t9Va94?sU;d`m?+NuA$3iLLd(>}Q{q`-C&)cw> zhVXsrH-4FVju-mVbMNegIE;#X1osKtRJo(L6Xk$E0Qi1{amOD358P?oCvdlY*(JJg zr*V6mH@eAJ{^|OY2L4bDz=!+J_4YPxbno-PZsELl;(h{m0!-TAz~gYe(PG#!x-ln8 za8CdpFSyw^3A9r{gT=llaPPw%N2ex=I=OEN!l;*B*Td#FfNl`a(QqW*k7u}#;O45m z2{0O$ogV=Y+&xx;ZA=Caes{G$8o<1zQ4w^|G@p9!2b?7 z+(-N_5f5OsF#>;pJ24SH0e4@GOFTjMG?(bX2En#@cn+Vl^JbUWhdbU2+->+>jc04% zHj|*~2hBZrexFMmxgR_?f!}7_*lhX4U*dik&;A0wFVrQi_-_b7O$g8LNwgo4uw z-lyOZ1s_)MP6a1W=aG_lLcu+NN&dqOSBpLcKd0cdf{!S;ZJVSgeAwj`J*bCVUK&0i zVajj7RV;c?2MMp~Py9l{lpg6zpl<5;1h=7{YMA&ZP-h7yeUU!NE(wLFqIF%`+6jv^hNp*w@J8%;e}%7?Kys7SBpr)7cqav`^6LR zO9ZDGZW2-WD4jmO$s>C1vBT(L!*}KI6=L80CVb)n6OO~b+3DLp>JrnDUKgiVEz;N+ zPB6(!W4_`<0I~9BcK917U^*g#(!biBiRik~7 zFx5w@?`Z|=`n^xVy8a66G$DSnz6M1?!P;Ka3O>T}dBwgwy!c0zzQVUDSeCb6^cZkd zj2JNVYrxp^V!#z5{x!*8+gFzXSD+n~c+wBq%M%JF{Stga(<}MnUzhYf3@;R23LfG5 z1^dx3<6%DvPBYvjj%Ym7$3*lSlAnwZd(MOV^5C6$@Zmf-`lM+;ZD_CT_Tia3L3pbQ z#Yh@9gm$k7_TY^c3RDw;Kfb-yWHIn_mCt#+;ULe1r9B8=pq^tBse3JM)E&C0y`$gX zso%HM@0;qEnoPRfxM@tS=N=a3J=b$02Z$J#rq^>d3*}?$xdH{M1Yeq7KaZd0`A$WU zPp{+sSN#s(hX(7X;1aZPeMBE|xi?-clUMYI zl4YbqTqW+q@NUJWEg#2ZzBDOIF$4z7_I0bUl|oqU)LK7@9VVi z&qqWo{4qV=H*>AfBig-PL&>7DQZ?|u9pe^#sp zn)FjwEDBS?L;3AXj1peV9&l$VKK}#kP6<&9m<=BAaLUW|fgXv&5%HlEaiOg$g*R({OS`d}FR+doEzXQvMZP5NMv^H1?XKk}3pf=5O6DD?YOL_G48&#UO! z{vH8d(KEbvkQ6}|W-pK7qk@CdoHV@o5#jZt;mERmx&*;NG&~wU*e_%A{}sjJi+`W3 z_EJOS0^j9v@x@OA=8hr3LVWQr0aL@nc)%Y7Obw5MD|G!LhR9Do|1a3_D_UW%13tZeb_5y0TTctfeON7(zz+w#(ni1#9YD>ymG zhx%}KelG{fS7);i6i|LDlWsT5Bkd_znZ#h6gU$MY}Q@MGzB#A zO0dBA0dW}KJSeihlec)PDL>Eq#O0R$Nq?U_Mc-8axI9Qd@K>bY1cH2_Jg){mtw8Yt z>CZbC1}pUk|FAg((J)@Yv!HkZvwv24AOb%Vj+_Edxlc2P6fd8O--P^BBR}9y<>P`s zQxV|*eq4j{nrkaB@`rQ%m~y~H&WfXWqWzQQzwpKh0Q^YGS55sy?VmqO_J9ZQ zaj;(x+B3f(e}{|x2eof1I^?SXUe1d${<&X^SwZb@u|XL8zuGVG8b?HSBmQ$GqN*Yd z2>$%&3G#oCPs8x4FAHhkf5G)NUw-)4Fd3MZm+}(_KfX}kMpE#HC(%En_U1YG zhg0R@_Vo9aC*K}BwY^gLqJHG#sr+fKmdpP*^4Gjd2}|^UVEI9RTwbdjmKXH%aLOBI z`yllQ-g_gdbTR)yYVZi^D@UAO&GN4Kzf<0Q=UCpGls->W z-e6~zcUsZEQ{{c&Eam;?fAjKAonv_)Jac)2omt-d-r@2_k5{*6;IPieDgV$_myWZZ?H4VJMj*e_k(9C zZyZ7UZ(ZIa=UCob&s^SMXO?&R9WL+hou$0J|L>GH%_o(iKiHR$^G*5vJEOmj@e9V! z9$Gufx-owK;S!8bRsf<)_TSwb1$GhQ;_<$mpE~pS3;kK@kkWHA%MbRFn_omD zi}($n^7_P5cdk4iv&Q$bJRkS+`2Gy#skD3A@}%h=T&F3|2hLoc!}nqS8ugba`Hx*I zS}|W^UC7VSpx4_$4<=U^icdX6^RY( zW#M5D?u`vnAexWH{Mpc;yMg&vg8y>Z+lJS8K2*^B3y+U!bcu`19P{mmlHmk_dj5Ji zRm}S5N1R?kCx@;1nrLpmhUVu}g?hZm^CQC>gL(7yArzdto~HSk;&A-C*=#Y@H}I$V z4s$-lmm5Dnh54_IMKXU>f1c|WJ5t^d@yiy<_}i1=AjQwB`D0`L8~)&tRDqY~e=vU+ zr1|qFH=ty8!oimS#{A;q4TV80AL1H;e#Rev?jttO>95@R^Ft#^e=)5!aeIvX z8TFU)o5F0eRln7I1LlXYCWP^}OMFWC1NcYQ59arRo3w=YDnE`QK_$)7Pj z>~Hago?wW{WqqXjqH3g?4-D~qGe{I%sLh4p#e_4YzM> z@6b2qTjz3es{Ydaq{alNR6|4de&k=Smr;GI=8ZU1-{5a{VE%lc4UhWlwZp?3OHKc=ktVeHVtIC?>ix8S zp!pAPtTXWW{%1$35$c7ylsv;J+NrHBSs1DZkc~=vdpPO!&LCL%$A{tnV2~D!^=@7vV0%FP z9jP$HbBThNW&G{QqF~&{Z%3*G&1E(lKPmClUIY^$pbPOwl7)qh2X&MfrIK94l^9Fyg=1FKWoY;l_HM2j%}Mqx^xNW#Nzc z#Nxq^vox>|UhlyALhvv(2dX>{B_nzE;>)uauVF8cKS1l-*bkuWu^@QB#&3JlZ+*ZfU;bdkMz{TE(X`^9*Chw)E4_BZSUH$nRW4$TtNQeNNZ zcJFTC`3;_L@W}ON+E0Z3VoB2D<@t{vZ{zuhhLqQ{52-Od_Up8#T0JBk^8-D`Gd}WP z$d8BSD+*JB8}q(sAJ*G3e=!Vt;AQ$W|KVv(Aig#Kk@B|g1J$p z>>nF>eq;&8kMTwxKTg5;@v&mSRgE1Kfne;f8X7EZ?4Smp;3gVxMnMmaTX02VN8FAt zM0`6w2)rHkf}euxke^hGID+V$AC!M(YDSj)xreL<@=i-Nx7>Kc4cJsB<)4;1Z-%9B z;2Tmc&6I!C=K%TR%G7k$|9MLPm8mA?*FyX-ekw^$KW{qtXJ0~jz+R@M+L}y$GpAE_ zRr!FQEYCK@59KpsCh%N7NDt-P#^p==AWup1rWS_7fPp7^DzCm+lH0fT)GgDk@<#d% zshe+lCOc0wk^Hd7_8q{3K3BfC+(P&ngopeMshgV9!0P=dh<`J6I2eY##F97Pd=uf{ zl=QLGO|4YF&>!afwi7-2|61P-shM%^KazfrE%C-uZGa&m(L?@*RJ=Baj{rXlu);&0 zyOYXKXVRYs*r2~22_$BDHGN&ZNk1E~qCdd=uSbH3^DO%MsvP|`$g>dB zk(Ru^hSa5i(I4jg(0;n}L2u!apE8{Cvhaw18DKlU9Pin*@RYynk%7c)N+0=+r4|5h zr=N)0V$p-&<$&$@D!3Orehy%TSNTJR6X#p`i(>nXoqiEuJHILD{^sy^CvTWY`oEm? z&+=UWyrKt8@;9G>Jy1`}6X@^=7l@<)ae7h3#_FwvKb zA4^>e*v@~_WE-C4M+OsDSn^*4yvZLKN_1HKi_zV&^S=tPo&RJ^-{$ZnKQfrO){=id z@Fss`C~=X+KN7UZkEQT+WK9qIAb!YD;wr_D><JRB30wu1q;?wk%6(99?-aIv*qUv)1cr$$nl(>=N zgCF)I&&yJJ`y@>KkzgV~c(h;WpQ!!ESPuYwJmtB;%x@{gO-p}__OT=Bk?TPuKVa}j z|A+Xaz2WqlV1McR8RqtJhR2GJ{l39eQxlShM7YpC#ZoPtzLqbRnlZzQ4}L_C+}#gq z7q`bL(L(v@>yq*YnLdFhxX^wQY^M+QHwh%jTZ(But07e?+yJ?Kc^dTnGXxTpc=Qir zsnSyDN9G6e6(q&=l%6<@{dvewEO~utGvP1c{G$A)H=}1FN~>ji3;G)*J^Fu;zaiCp zJ^Zh&J$rX43JmKEOdoAXU4IMgU9_tHPcYfq#POkX+W#3$B0s?M_0$p(PqwwSGLRts z1>?z?GfjBbEEA5$P58X?Ot`kzglEq-;kr5#KL30ZuCF)YIde?7s>*~fxWI&`Oflj1 zb`!3yHsMPyG2xmT6Ta|56RxN*;f@XyuBG80~~z=X@oZ7}R<`gF^l4xqj!FTb4aSNbQ|!^DXuyl9aLN24Zu<&`E}Qewi3 z7n^V_X2MrrZNd{KnD7->m~dgC3154y2^STa@KsltaB;B-cXgR?Bx1tXTw}t)pb0Ns zYQmwA2`^bDjwPF>-%K!#r+7Rx{oPc5u2=rHBRTPM3?VU~01!zZNQA=6Bpj{KdR|zi}h~IL{U!xS7gJ;u&rt{YyB+^qwOp zB|YZvXnu&=cixYK`Ju(3%fW~)IbSU2Yev+3O)KVaXnsb)k6jCx=4TXq{2{=NTYu0C`WA~G7zkmg}eFKf#~!PwL_P zN&d}LUQGXd1pTwpQVT}^uViC27$7}bUqOCKHkOuJFsJ8<=KAY3=pU8?R{e9v=g$w6 zLH%~TIljL;Dd!8zt@(k9yz#y{UI%?E5=@j?aL0E3#`x1k^bjCXXVEvp-0bwmc$4L!@oB9kPknCu$?_QEO{RwciAIZm z6I`jCzcIe_k~|O~QE$=bk1tzWX?)p4c+?-F2g5{wVAP*@D$;6=9|3nHBN1-Ttnp$b zV)+LiKhBivK{P(ae0oD_2FTNLyaYV_BhOFvb;GJXx)Nf=-jjuc&6D}(=;b_!^D=JL5q{M_LPBh_I z%!DURGT{jmOt`YrgbNEzxT?y8i;7Hm$`lhWE;iwK+=M4jHsP8Y6AlJVxUSBGLm?Bc ztu^6r*n}G!O}L=IgzM`~*y}anrX~~i`Aj$vFk!#n2GjgTIVLP@{ikRd{Hd+KR8m3h zrS30vBx4iJ{@{d3W`D4-((DfwRhj+4;we^t5dN$qIXQ0jcgt(wPi*Pc)tUXj+S;&z zZ)`+;x5-~$PwiJu-lnGf{vY+NG`>6)^iY3?@~rw>B?MC*B^;hWFy%@1mkNpqraVZz zm-~0RKUbde)e*jy@Ti}g`vteh%mDqL@|3@k@HEDd@dHf+bAJo<8}!`%GW~I!$3}lE zBHobwC#;ul!r~xB{BJCZ?O-{t*FOdmLfexZ#%hV@k{J2@Xn{U6)=5SU-VDe!&^N-_#xq zAipQw0_THy{8owmY*arz)PICLp!ZPy!1}6}+MfWQf2AHBF2I$kDEIHM-;L}K{kam~ zKA@N%%Bvw2^|b+x5`WacO@k#R)SrVr1w~^q`gc)!eU$9)f#0UVLarKYkE^Z)9E@^h zp$dWZ*G+?1u8kv1@UdbMNKPn`1_~JGg@1+eC;t#q?;jXRg?un~YrjAsS;Fi2c!c<~ z5I=?k*Xn})0?ATtFZl`UGvsA|pvsT``2{rs9(wjbcJzre@iJGGwx`U&Uq zSA+bOSSOZM{&{~Ur+*;gVtpq7k^FU}-%{RR!uVs`SU$>sc-;K^EdBeH{FMLqfIgRh zzpuvzlRjyFL6%pD>kIp1y`RrsKgP*F)u$DjAMB0vm0#c1XguO0|9ZcKE`MK6KTCk0 zp!cuBU*_^F<;#_~EPtQ0M=DUOyhFY|BfT=aJ-M`fYI`C5tg_LQzsR>YvIo{T{ITIL zPNR*M`14iTpx1v-nYotFT z15x!WKY#ECr_?v{XVP9MKLOuX8-L2LWgmFIg%kVusE*J2IJLd_xW=EYzBts!P8&b+ z$DgvnMtwWDjqAVN{@LHB5kb|jJpVjyefpS^KRXeD^~3E`&R?tcDOX>7 zTwlJX`NN*IJ#hKFkk|hLen--5Ur8SaZ1K@w$ZJ1JZ1t7OtI%d25m$bDMCnt1LFlWhvsY8AH5V} zd8vKV?G@pPUdk`=H2$>9e;lwbFQTXMsPqRRw)dB9c(Sh(2JG{_W`n8xD1F_3B>U&_ zh1LJ^8~v#rNgoZ6EcyWUC+F;u`VSZ{0nZngr*$3({Vl2=0i(XOuvd z&==OzvNT`AI%C+!Fx3f(_cP3SVtjyM&J*B{WSG~3bovdjAIcxjoAP+SBPsKv@t6-P z@b3Gqn~ezbM;*x^>qEN*MHbx@W_?^?2^0E%lUSocW{zKE^xk8v{ksUZ4-mALBfGF)daGl>GViL%y#~)koMj#Vw| zwpbCoyjX;=A76+g30N)WLuvri^b%nACd_Zo!hCxuSpRST`XtX+qkjT8^se1snWyHj zMeP95rvc*vz92>J`^|5PBbZ;odSiTZh_Wu}fe-e3*uQi7m>-W1%l%%GKkRkD4^=3B zun#xpcK~;Aqapbt_vsIa`8;3mMS29oHuZeGPsPVGoS%wU(D4bx7YVlC_+}qWCw&#e z2T4B0OBm@<{mmjpUN1#?1pEDzfSj--d6Par+aB|)!G3vvfXavWm!tm+d@zTn`bnFq z)%w`Mh#PorKRA7qe~k5yCV=)2Ab;L|pFE!skK@DA*e{m!(!2lV{sNT$hwwg0O0Q!K zFQ|omE)H!1M3-F8t;GB9kf3`1Wew)10_^|z{iU3q7cck%-dg_*vHaO4=}$Hd`a{*y zUV_qIe1O?r_`GI_&hr}fLi#*nvzM^nw3l$$v=`RD-Cj^t?Dj(E!|e9rW!Ijw7cb9$ zvi>mNW!N9e3-$o@81_N1%|2**A=ww6v3;=p+U$ezHv3?_guh4jp!_%aL$(KbenZO> z$N4w1SP8?vk{((xlQ8Uy<}V4)`Ui~nO|bsa{`35<7xjbVQ~6{5iQX?s2I!~y4*Qsj z`B`L0+8gmp26=vx?MK=N_h;BXaNaE6KbYrpB4`is+6A6mhy5JK{($DkW&5e@sXXN= zwAR;=o;x{%>xYP-k%B!@dm85qB0gX`zw)^jkszOK8*Nyl-tix=%Mp;+`fVzy|0AQx52(RVBYTu{8XI3waN$ZWz^r`3;KB} zB7F3IO1hRpf6%9O{)6840Q(BuJ%i6vf`a$IwkN%GfENmaK7mJmyyBzn_w>yNJ>Dn0 zIP??o94+na^pagJ3s_BS#ADcDY`_bZWqiN(zLwEZfFM||3wDEF&y{F)j& z9^=bmGk!&d9Z&JA5TEx0ar{Zxs+i+niSzJg{K;6J&*3RP1d{uGIDRRnuX6ZG^vBHj zWmrzi;VC`@lKY!D{zPm?Rrmv>?@d_errJxWX?Jn9$05HJz4!RW9i3M{x04YWx32!_6l^TOya-kof0 zM0Y6>AUx_1_A@n}+kU1dp8w?YEWDp-I`o&e`XA^Y@cEA1ex~WuftTlHkX{h``;k0q z1TNP94DR3X`^GT-!~Uj5?*GvKCdRi2H%PgEgZxl@RPOsJO~RMZ`52nON}zuNJobmT zPmInQ$;@!UY8;91feX&u7A3uL%bN zChYgyV9f8tYvU+?-k$_}>`2zv!{1x`l{%7DQ{caCa7_*Tu??=MgnzZclPA+&biLoH zBUxGo|0rSRKXIb{{6I%CTEhO@Iu9^m0{oXPeqj-h*Dd_CX*68R@h=JmHO%~r3upkH z!f@#l~+K*LAFr8_j`d+fJv61zm_A6t& z79~8bK@lGFUxfs7rvm;0^IMH1ubf}=(f+Kw{au0FelX;xWTS`Pm2d5jL42;i9AE8^ zkEHc}Eg$+DIBl|+u$#vPCS zX*_D_hntT4`Jn%M z4CJNp%bCZcL2Nj+r+>Ebs5#!m`U5to+T)j_yJL>G&O9EqjW?CN=6E!Q?vZ^wcee4U zZTyM;2-Zu{`kV5cc|2+xZz_3^KihazzRymNj~Y^E8;{z?pDYiprCT1$heR{j7R1Bj;!-GEsgf^C&qWV{u0((8&XZV@hbQS`&$H- zBdqZ!jR&D_OMfxE|A9o!tp{LyiScRUNgjXF__T@FBdqg(ay;6k-p62!MhtKN`pvU+12>eg!9j<(Hi}KGim6n?P^Uo{gpO4Bv-;{qID*yaa{&}bT z^GW&VnWjxM<;g#v(ifjz{_Y^s&I1$DUOMfm6nx=Xzc!pZ<*Jhw8|m+RwlI z0Q;ZR`_tiC#IbpK-xrLyjQt_-Kk&btAAAp0o`=EuZ5!-OpNGNw=jQ^%g;qw=)65^% z@8tVx0vn@V8lU4Tr1#U|yby+)Jin*T4`F{CCgk8>)%hV`84Z`Sx${HblLV{t8rYBM zebazL{J~L;m*->D{usfBkpdqAZ?KcJgS&%=Z_me1n1oQL}F!E>H zpQ_}w_NQ9%()zvL?@juksaP$aZU3l>Z|#=^Ecc7ke2|n^o}UQ;w&y<#*q;9i8-MJN zHRL6IRN3$(Puze*-Ub^?<;nABmVKOAAK1@o*o$qyr^=tT-_z1h0pyqGnW#Lh{hy3~ zBo#8=?@i@L^EFzZ1XF#H`jGme{)*Ix+#ebMtoMT&`#)El^1f>Kc;A~OedFT&lKJ~Z z41e)5_ILgT_IDn~eop)TPI@6N?e9c-i#Ld1C-f`#cShW#f7Y*6zt#9a!A5%M?~s1E zKW3GOPtIQ|*q}#!I+P41C^hY$uwSx_mLssAiS7GPGSq3|QGZ_<)Zb&o{>w+d;mYSn z`!jhukM?J_T#Nme{-9V0ehKNn9=`|Mw|@?f^nFtJ7yUg^d_RWu=j|_)^NI1e5W8tV z>IBJ;{W0D@dmrK-f`Z7X>#9X;V~oGQto)CczR#uBFPL9!V*&eT?Vqs!1&bfK@q)L% zNPZuO(t82#pWRqcK>FkTM9@$B##oH(iNAk>{wU2i0+Rd9fcN$nv%e4&zmDcxqcJY8 zXd3%3XQ4k@$op%MKGtKF1WPuS#K>MIl;S;qD6bg&8~2Y1e{5UO+aE!Dh>Q3^KT}gd zdLaQ`DMWbc-xe2J^xuoPym~(s(L)~WZwr$Du;;|{c!#MV)Oms z-x`$GEjA+x^p(#q|NFxE--{T=_mq&5yioo)4=n2^weR`g6PEn}^ZN|=-YE44_=5bK z@cm$KDIorS@F%!Fk%as{an#pGu%C`#jHkFhJd*SUEExT<*hcbwz*G9z|5m8rzNN@N z??<>9Si0nVG^d9N2$>$r8}!W27N6i;{7Wb@E*ZaakV;owb=1F5`Rwvc=pd&GVQ0@sjfQo z-!=*l%NwWn&C~amJmhWEB^|KZ>}vhO(@0uSkdKeL@6#pbv)8|6M#tg+Z6h zFMWUd@cjU(Qa>FL3kUa!g=_%(u)kx8XYrtxCr2;RcKQPbeZIXJ^9>E3p#3E5|9C$i zz8~Gb+2fY>2LEv=;{MPU(Lg28QHlMp*dLAlX#yZF^m<^AI8V@N!T8=W>Vx+m@qoXt zfc}50G!J&-ZzQ;hb3EO|6P5*$ko1?+zn;`u`y~j=c55{9YrV@BN{pKxrU;N0;*7xTLb|2QTM=%_{ zMd7go4}WM~4H02{KfW&FT3oqMp|w5Gr|5BU{{N1@_nx+3YHw`$#ry}pe`e-4m6u=a zcd^Ugp32KF+D9Y5z}x9De)Iafbbeeeg55n9OyfJRzt4o-X%qI3n6P_i4mR|&qS(~W ziq=#7t+hXj?BS)n@0*c*z@Et7_4+0GXVxF-5P15Y8tYfVvj^N+!b^L6()PVHFTdY3 z_x&`e1^#7ZGwqLwitJHr!zF*a+2>XIll~TXr9Xy8c)z@z-YOqOZC5vGTQFXILhIY(GW3o8HDCO6w%U6b-=lZI-_^y$ z7ymL_?SD?gZt=wr0)He2=li34eujNNU(MeKp#A{-lkyK6mHwrFIOF$Yp4*1+zjRSa z;gaQTeP5p2<5n(DXhU9HzCMw#!BoCn9`p}*Iv>I9Db+8`e{R?3)2KZ||K(>J)c51L zeFZ)08wy;u2ggufH`u<<^D@@EVPE$2-wS`Wg6K)!*U=t z9^X@dJy+s=zC^G5Pp~rSc3Uvy$9{2dobWWC4gbA$5bY6~Ahc-(+&%$zSL9&S&#d?4 zEgrn&1^rJ~-u%Wak>9_`FAvinZ{z$XkUn3i4_BkUr4>l}HTpNt&JvsP`6+i5kJ0%1 zEJpi-{?G7z?yUGK3P<(tss56I+gqi=Mix}gz3vj-)ygl-;Vdo5zONQz*F(QI)dH2 zol^yp z?5RrR(no&f`5CmIe4Y&FsodOSAsG58-|X>N`#qpT;2ofo41S{3OV{SnZUrQjlZm#EzA z;q!I8KZNOvtEha!Ts}+CUKaBCpKNp(`N8{d^7xbd%%8_w)V>yR|2a^)Ey(-lwz~Pe z27g0A};!VUmEIV{J~kQ56VA(KNao8RP;}*{D2?wqdJ7h z59(*h)^JW<*n^=TypNz{Ye7yv#2*^;6n3PM0DZ5L&ntMN9RL}AjJ`J#Dz@Mzq7M{X z^s5N=0}p{IKEpoXZSWR%*upO-{RFFM{*vFfi28v2ML|#cJ@S0q@gau2Lc z_qm8Zkk)X(O|V~Fr|==B_oX$A_e(uMZ`XaV#KS)F+n=GqifYaueLrP+-uHG6ZbSY` zfrp1C{Hg%pDB!paE(F}B;KjfP0VfoEp$qTt1l%Rz<$+Vu&zHv-U&0^c<2Mbuv(@m& zD0Qy?`QJA^lx#=)P8ZD|lYbfD{haW!1$2G|{og9t-{$#e?jJva;^O`>_Gh^_BlMXG(xB*eB)}tEoG|^#Sq>-&dXc{!iUQ4!&Qi+Dn$7{V%tN)l|PRex>mn`)3-j z!5%T+Q$@we=Z~L`h*=fvJ|PV9gYhlCM@;66>mjuNvjF>Qaxmf5aWuwb{1Z$wy6-3k zTJUGN@9#qY7_Wa7-zO#w!AMbm@%^#NBgXfM4gEqNX8)DizsqvpyG8$ZXz;Sj*q-_O zwyjj&AfWaL=a&vHq4C?rED+fT#%nXFAEYnvNBj6L2?CG&x;M^O`oVk~{O8c%!uiaf z>O0yOYHyjp{QflW_Zy)GOpUiH2bDQa#WU<&1ikxy1p5~Dv3)kA<}Cy$zbC}sySZqg zbzThqt39=7A(0_NV6y^?bvlSH z_m@jHw&eQ97!TrmnO;E#XZv0ue-Pj_-={@=Y$55mRXGNIBz_g0CvE^dB4POFO@s7$M${RK-$eBE zy2rVSAFYqqxTBOF8QFnt)L%e>@Olxqua&8pa=o1JDF6Cobp;tTl__9%WomYl6`$Ha z{@#O%&-Ak@se~y{6d(MN-5h??AT4jNlJa1@P@kMJX(mX?9T0tGYF4Ag|G+H7r~L}K z`1Q$|lZpQfMUVIuvk_MEA^z+JEB>)1$Ul|G$7Fi&H}%O`6_l2yuTRe5{ojfo<)^_+ zuifBZpRA~aKsI@2PqWfH!21uHEtvJu#QfWpyvW~F6F*}T$zNgN8|rL$ypM-oH;N}= z5TJg=X#c&TkH$$P&m0-wTc2#2MEPryF!DR86d4kvW2xVnaT|Z&XHN!0K|Ex6%|gvo z`bK_YscF+J82OR*mXPwmzGL_voE=}EY%Ha;C{L1KJZ|EfC)?zMywfHl0b6;_Dg#6L zK5@2(nYDI#Hx1J3Q~PB4b)@g6>KwiX{2I%Upe;Xfm|`wJm8rV=96i~06Ytk28Ax8( zdpXIk^x?(#&T8!Rvi|K<>61Opi@A_Q*!*~VuTTZs8%g6YB z7xXXdFRX*gANFh1SLR<@iUj4MPQa(u-)264Kr+bkpz~Aq`ZSxrw?{6T@RcdIZGRor zAKEUGlIgSk(fWlwemUzaSKf5~%8qyQbQ{^2)bFf`qzmOAP+m0?A&@SAz|9kpfP7y( z%QMk9e+GS*P+f83CVtZ2U2e%&nX1m!fA(+Xm8LuuG1?!c^<9~oILTJNsBh#i9+Ujo zz8muVK{dZ0LzicLvT_2QCm@-me(8LfJ^vL^N{{lW@?S#rQRjctAia(_F4L>?=gOm6zCYQIhg@DYMN~HnaT6TF_pFZ1CI9gimA_)xTbw>XXsr*_WBruULT_5>t2{(_N`vw*{q1He zKHldK_hC;j9*%=Fm!D|PUvqyc-~Uc5Apb*gq<)KS=WVI}M0vll)_)A=ajo>8r}|Pa z&qrD1TO1_WBNo4)&3|qKzuFMhCCU?}7faQYSo{Fj<9%yXmvsIsQzd!+UA_;DWR~(q zCYbsw4^w$-ec}5Bs48~-M8il>+Y{`8_Fvoa;V{V_!cF|)?+Qqlnjgx?M|Cf#@Rg~c zeLpwszp{Y*kHMudJ((X}Ua=JIm$u`hev(D&GnOhg{4L2tb;%}=%^y(v zFrfep*&RrI;eFkJxqr|3E5-|p40&z-lkz_S6GiI07?&5FZ&vcLKemkzm|vLRC#U^; zebVcv^$I0F`b(jlKY;zzV}B6UKXO-)5B1IMqq;;gOL={{`VRUS-tT3l&*_C&Ka>W= z$NPHn>c4y+P_BN4^XjiR0ETkRiu}|kg8|Afmp|6y>aib)Wb@%BdA8giDn{jk%JA1z z3>edK8pe2Q%l(+X(=f(cTkdxUEExV6@AoQZ-pr5R|E1nHga>$^)0X?OJ*0-VjDUX~ z(R+drSY8L00AJ{{@XHAw3UJGerNPHd)B_!!V~GL4MLO zDV=`U!)3$#DSaGw)Ob!m5Hi!p@kWie@~`-D`aUn)0`cSYi|B(Q^5XQ1k-vm3{Q~;H zhP*ic*lwcnoW3V7|1dX=m+3^+;ryu0^jqw9czXbV@+tT-|{W_rF^s%o- z<7NJ7!;idJ{~?s0if`p#!JK}Hm;4W3oPH!|=>zFwJs02OS?WQhD7=2(%^kl)o6c$)2pKj zM*cPp2BufX6?~jv|McoMO;7EWZ+dk?!K(=NPOt8=!=CBYJ$BeVy}D1q%Uj95bMLdG z_emv#e4du|@zfH)fnb0E{CjE}V1Iy`BEFuAxbS{Gp1)SG8}Hxa`Evz>UcVm-PcUCt zn{?;O6W<5Nde(GzdZ}8^@|`ljYR<38^vfAm?_p7RzVId1PqrHA z3I1LceZL(0(Y}GQp-PDTXxQ(QkP%?T@JGD9^HV(G^&Ogz3v}A?m_HS97^qwyS&#J{ zPnPh!zVky`Zz^=NzG!{wd4k;%7JxB-R8?cXCMG2WL)9~3N)h4<_IGKxN6&}09Wdm&hL z{1*F7j)!PJSD0pS?J$|H9ri=H3_l+lr#zu+y3~6A*}v~_z8uS_{U_#nBh5b><>$fs zrE}kNK>qXB7wP*3)E8K!-j_i7vh<1aac^|Xwu$GT*9*w!B~|hMD$KWzGe7O2@~@)x zCfGOZ!JP)Z{=Pi+|K-XP{^+~*_g{hjVZ0v}<)7B`>09Z2e)N8;1lBi!-^8Lqmv7dx5^?owwpWlbZ>&LP@d|V#a$@2MdGRW(Dh{Nm6Sbx?V z+zP4uXg@3E7Z=xGn(yYmB*DO={3-r9lz-aTU$A)aqk{I|g9YaY=hdkBohL+s>vKHC zudlVibl%R6x4mx=^#$K2j=XF4+=Os*d~Y26ZyOA}&u_v3yx-8G$9b(7e~(I~?+y0L zS8sBr>uCL()R~^A*T3<;!pL7gQaMj7=k+gopJn7rGoqbh`6%xXqxWTU|BRG|{y-e> zU#I@K<`4Qp8UG~H0}e|V`^nH>4fbQtIgwNRM~mtGh+)b9b*7J{Dyn*bQtS1({Vnt5 z{(_)>A0@>{|00&U|d=l`Kn&ZC%ONw&FPcN|YFIY$YVf2~Ok$=K~tRW70NIO7%Oj=}}SZ z8;wislHXWynF*fh`!&roGr?ql$76mkeWepWn&G)FFJySUHq(J2KR*w3nLJ2=lF)?Q z_y4c`arQovzY=I^;Sgjkt^Kj~+H0@9_S$Rjy@%&BP5w#XE8Px0+CM#j^7Z~K_{aR$ zzCGF^Hlffj=SMXChk!3ePJYziqt*(nAN+&=qct%5z2FDS`~N<1>_3|FSU=0x>$4oo zBP@M@`u0!Ly{PwtN4Os7s4_2vLgbbCS)R_1`b$b5^AEe3AJ0db{bc5s^QOUZ8Dro- z>tlXX-}1Oyt)F%I@EEyyjoQdNyZa zg$IAb4)&*(Kk%a+=1Izr4SPgAs_{+!ej!(;JuYLG^ddj|$Kvp$GI2$BQHJg|Gm`i(&p5VBJ%@mHq%cs9@1IC||-)wwm@<<3&x< zgLr7s7f8QYrH6jQd^sHV<;C}*%13;CZh#4pT1Pz(bald@0 zjAbTzTG?kJ3pzdk^@l7#=6mFaKQ)cN!u;yoPcQLfs86!9IPxFzy<{A}ivGv`dpJ`L z-15hgjxy!b#2=%+jwJx~{bD)7(*7NPC-V(q=AYW}ip0C)e3ZmfLqw%Na2~Cf#Q_dg zdkgygDuXXoe0;G-!I%coz9?=N`9ppg->6RnTH3GdF)WbDU-e#>#dCijc%Fx|3>SWv zBt0q8^vL~cfTTXQhyFxq5BLRG{5Mh``VZ}^TKW`T*3(Y?Dt%hNz^qQy=d8blo`QMr zLimUJUr=en&&bG^#^<$s0Z;kw-@KU&G4>DY=YyqoJdr-t|4Luc{(0HhKdQeK zzfyTq`Pm=fH_IozrhM|#j%T6I%D15}(|?hXN5PXnLQnm>vVH(8{s>1t)ObkgQFtpK z#ngYHXC7IpIYhKPLIr=Rx@>+jE;^XoDkd`lG6Ml7@ z_x6nb5r6VjeMi-w0 zKR7;A{XCB?`o3!Pks{6rFejqS zl+W|mVteurXipUPELIB9%8%ApkROb1$d8)NnI16G_bGb-%PPs=W#%Ize_FrEe2DqA zsmA~D2a7!46VBk|F3ZyL2l~Zc5dNV&;$<(8c32e_TO+C9mvHi^&Z6aBC3 z%_t5BRsX5}>c)Id1HMUl5c+U`tDz5Y6r280A1{=E33WPGXd zzC*3=;on)J_@U;bZk(r}J!9HG4Ect=iTd#&z9%EEIndLmxa;`?+4%&@?*VT94U1)Nlwtn%lr|x%lUo4 z!vEn%!yc^l8j=M3A3hv*VZPDux?p%V?8AJm;FH{+!gh$+e?$D>aNL&ek05_75pZ1A zuNc1v_n~;RUnc2$+O+FojUuLy$U z3;H9kHSxOIUvA>F<@_ef=XuIq%&+}PTjEgOJCR2RIDZ&<<9vYh<~<>yPvXPJSvfw? z_buwLP~Kw|{e|?Nza8?H^{3O{O;(D(8}8?K<>9dem#FY?sEy_0`G5A9Rm6`>{0G<@h74`aK1$R?FEicw7+{M z)aNf6I3F}g|C&B;f&CQF%l!>r*e2&`&zyh%=|P+?r~F{PlJiHN{}|hLT@+cuvp73M4@^@)|sQoPDXL_{Pi=Re)Ui>ujOT0AC_t|*q zndnC-U)@JBG|%5H=NHE1JVq4MKY;T_gCZKDzi|HeNHQRzG^p=)IE;7?&R?=00{+e= zxdBd}=KRdjq<8R>$AIU47w{j){TH7+c5Jo4LVs_Q1_j=0kol4Kap?7c%=b9IC;qQy zbV$x~9mV@~9DgSNT+)yCxOIIfpYo*bW3^wB1OwnyFjbx6dGBCu7U`K^-?!o2FG2d_ zOL{rqne(40?-Hh;+O7QifCs4mr>>fk`+X+%KBHn4th~Y<^Ijy-=fCEXCB2qDw0Cx< zTj>hp|4*Pj#*v;M&O^?{g(cKyhCb9M_O+2O!2Qa;{BAq>`9NIgXM6F<{sQ1$1vdo# zIL;2gI?4p@1adqQ6I+p5xhS(#Ki0` z;_<$f@Vlzy|AE-NpN9I8{vp^S93Rb7=s%=?0RBr&uk<(0>qWbWPwOj^9^tp@PZ_V6 zFJ*i|zF;rmWB+Nq^iOq9N&5Fu;8`B=82_kW4ur^gCe%N%cW2sP*bk<>M{}Z&Uj+t#1(x#=+5JV(kNaHY!inZKMc*9aWxy}BUj%;Oe2koD zll@2N1Kt-+eoOnGemDC|?l+pc4*iMpaDGRoD#I`IwO>%3#@N8;cWkc`lHle>cnw-EANy?D<6XIC@{Fo zjEA|Tl505ShuBXO{}rZkmXG~aIbSR9LrMO8p7fjiPfj;+i4`iOyrchkKCl5m_=WVJ z+Z*&nq^`pL#Cvl|Ig;Qq?&m%QdoO8AVZt#4$+Vb&Eq=BV; z+%$ocHSA+dJzmXsC zU-}p4MNwWQ_^>Wd@Q^2iKRpR~!h0VUmiBc9C%pQ~!AY;aw%|*;zB5vP6g-`l@7W`e zpU>dDIOV@{P>mN45Bbl+ct1?pZhCzK{VNaZnejBS_e}6Zj>jg>w?m&oe_X=(YfA5z zB7JuYU`Y>s!uik2ljxW6G(YU2vR&V(^*il7JHBC`N_^D|zo6oS7C0}e?4xT#;i0yt zSYK*Sebuus%dm$z+{}6OD(q$2lWPBDU(1F)?adltHUj6Z!lWeXO%?#)7xUBpm-k7r zelNzYvOdF~xSq-pFZJvDPiUXRGiL4cw7po*K_7Yd`$PZlp9cTc{3P$|!M|(brSKlw zQ1A!$i8_aRhCc!89h4{b4cd$Ikhni23dKH>^&#eiYFH5eShEa$hWhOJzc+v;2)pee*}8|m-cz`z z{=oe|avzzw|0g?sv+;5V{xE3f$KRpG&l$+q5x(!E&vV}ke*JC`7N0Tevz9PhUKW<| z*7)tf9dG218UMKda?Gv|(Eh_oZVaP|@e)`(a(#JYI=+DqXSON*py%_6y-x&7Q62=m zo$DiSy@T}LdWZ0bYR%yDntxc&d*?+V&l7vMM}H&b&0&4H&X*VD$9sXbEyT-=uz#<0ooE{?X2sRq)qfy>Lw0gYs74{viv49=@MS`sKcP=nurF zt`OSTh||k}F9=@VPeXe61B#7T;hyDq_!iUmu8_ts{Rr`Qg*E(Y_y_y7!1Urj0DaZu zuDj#`HO()acS3#8*v!A1_?7rgdx_$yZfHh9*@=VDy^8o)08GN^MG zzaFrQUyk}L9`$boI}H91%D)05*Tr82^W5NB|J6{ZDgJ|TdHG7L0PFTWw#!joimxVH zP@bEA1IkPB$j<{j>OC&>M>V+#!C-|#?zry0jU`@OD65|ucueSJWuprQQ^mi#9 zy@~5T_^(9%S0yF-3#>fi{_|0;?}BHPKe{^}DT#hPuKeA#WO($G$N1#uxcEnK{?j|3 zpyait+A~tbe0Avv&xT3-kNDq5MjC*z{uO@VzVX^HU?~s$#C`MXJh2`>TjF7L-40xK&zSHNNu}pWnCK(m?6(M@WBvFusEYj<3+;A-eDO z;C5DzQq+0d+nn*%)lyIHypt%cpCNyD+%by$izr`G|D-!!K;O>Hc;xn-rC;)2iu`>x ze>dbe#aENJWyT}7(^5R^UyJ;UZT_B&zPR&FD<481k7M}x*CBt!=I3@(im$}C+{ym9 z>p8e(B>xUiUtoOpycPM&R-bgb`b^q)CGg$0{9DnUw0w^5?Z9_fJdcBhN}{fmdZElyAAkBU0*3KU2_N8XU2aN-+sG{cm56c zc`*G>q`y^{S4-Yrvg2LT^ETkyba^$rUncUS^a0+pTe;GK5kGoI?d>dE>x){vaifDr z`pYkOV7|Xla`X@0M_ajaIdV6rYO;S)i|?#E>7y<29Zj}J?yEwx}ht(gFfBkv~kNm4w-|6MQ>@o+B{3};l{UZ5qy2-`&C3mjQeURe^pWv^) z+Qno3yxh-!)m082^HA8!zp-J1Z6D;L5BDeVDP?IdV7|{tV1uTwo@{8Xu{`Q`@B{ZP zV0vx_tB6PXE%6mwt}yjU{?Z25KIu!YxWbl?{A2N&Wy&9-$}7e7b_`f6--x$dZux8I zL48-)`XaHX;185|idu4e-O4-Zi&w7k_34^54HPf>6y-sG*Ia(BkEJ{{9{|tyKPpIX z?5{2H?M&)=$xW>*;QC^mmlX6U&QB zSFdJy`o5Sgiawr~3~4WV^a8HM8&Z1E{(2Ho{A_XIh=*hw@mnoEnc_IsXyvV zR;;l4LGV{z>ENL+7F9Csm-{6&eW1U8o!DoV{+_Z^Kj>e4Y=56_Ki=zt zJ?4(j{(jpY$S?H8EoJDVW!!%}RYHAjT`=&{`dA(c?APVNpQq!0pdUdW_096i+qAyH zeT{K@r|_5h68abTm2IZ{=dquU^q0?>^m&}G{A%z#%scbmIp1Tk{xZ>$PKDWKyhyC|M5HBq86UtvR34aOB-$Vb46KbRr{++ViUt;z%ao@TG%tL;} zzX*S0dzPN-9eQxSw9UsC+sRNvFZLfoP5{PVg!O;f){pi782m$=uM9r>9t!TO*5?D1 zy|aHB@4s34RlK?1e*gWC?wOKW&uf0}pGJJ0$-kTXZ~3s6)(?I}k=q`;Uy|WRvFPG) zzPjX>mkWn&d7q|#EF7^g>2>10A;0h+L}`A+Q{jCCzdi1IF3gYnPNh7!H6=g%t)@Mv z*q;T@KaBTiu)!-=4$!5?CKOc;~%fsUQ1)PW@v^XSiJG zQ~l+=&x7%U^W=G(9{U5{dp#I`IA5N#c*v(G?_Z?+a{ofhFXX3trc>^ZG4h)k4=sG( z1q)*AK_D(4`V0G&?O2c6_7KncLcIsW_hY>A6B17asou|N$9n`twV!U^Pw~ggVcZW* z`__z~tUT|Z#(6D=e)9JohaSd<+y2G=9_y#RRqxTD=}EEuE-$?+?}sEk91^;{MS)2V zqQ24oBIs}Pfknqwk)gJUaSvH}zF1 z9A|P_&mF?}jzYS^bbGV%gZz%Z!}!dO|Ezp?{SSU*^(*;9ej9o8^#%9Gvii=|mrCDq zcGUE?c;}a-K41T_KJJ&A@#E{CNwmk6Kh=NSk5clMNBbRpi}6f(dq40--%#GB9JqVN z*RQOf^QC#8%G0+f5ATn8^2hP9$0?8ULi&w-dioamJ^h3BQ(j$p;r#PvF zyCvSGeTL;ze|~Ykekc8~EHK@2{4xF`A5!1y@@Ow~*zu0_CG|lASi_&OBp2Sw%r?_`aMs9{?^Xf1TQoNBy`z$>?*DCr5vSKK2ji7d3ubME>_X^bpVaMd`a1 zl<$ovnSZGN)p&0~`_4W8u)Pmv%IADw!f|uSE_%AcjWifiKk5Ci4p(d#Sf!@QyAwR(0>oQ@yEDdrk}!NKBQkt z=V0Tv@dK+K?4QFPYc6|E$2*R`i}!0bmmwYiLWB6nwOn73efXdSVgH?s_3>N6f4(j_ zob;vdSE7E#gI16lj~t>sl&8FUkBs{DB=_@g0`KwZ{rn@zRkDAM^Bj_XV(;qh zKc4?+vN*^1@*4Sp_A|bo^94Q`4~Rb({IJF!LA(gtEAQKZUeuq) zn%@QONK|ia+2;(xKj4HShl+9)|-0@ixL=#GlJ~7KtAR{c}lo2la(% zFXE>=s0vfKv!j9h2KM6B5nqS<$t0c{_H0)G`9c3l{JM!(2Q25O^EQ56!D;+@QT4y* zf2D7fK4N?#{5cWke|*}$hW~I$pyaKA^GA#~FZ75$=r`>z_EKJ?d^H~w0=&)RmOqTIru~jjl^yYuehnWBCZu~jop;tyELC2djo}K=BGhZT}uvi0V-k0&>)zQBz@ZJvjCGXQ< zykNbN#)ktB|Bea>SMe-ZKa?{tJjI;AEq*Wq_owhvEUzzxB|g1316MO}1qa-0`OpWE zqYqSkI5asw;ivGgmdc~V8?V>!(-J?P@=M2?r}`xm{|@=^_^Z=veTMo}dgbpl_2+sh zpVEKSFLpju=YPENus(bM_q9@=;1l}KXD?>)pW0X8a>0pD2kgbO1NP$C0ekW6g-$=7 zy-@b!+2LOF;@Jy>emr}jKWLcx89!d=^W)hIy@40c4tc79V#jAt)Y{CIZI@5Qr&elMOK?Y)3_cCaxs_kQ6MLgHKjP;huS+=}$a`pF-#!~D|DLiBCiZ@{!-a7k zX(`W}cWL|Op?I;fzwG&v&eVR8`9!bpw7tUdL3*b2{7IPfPVIh4>`lT#&u+XQ3Vsu& zfjG5Wo#%QWx!hd9BmHN~+pX-6Bgy5Lb3HEl{#nikmCSt5oz4f}mGh1je?FLr{uA=6 z`Cx01egDquf6zAo72`(ur$Py9jXe<c}I3@PqUwZ>YHQ z)5&-<*^Lja_`y&x_g$G>ReR8X&|hmeu?jukWZOTncTKw*KT2MNzSEjM*dsoD(4X_@ zYtPUZZ6bpVKXD(YM=$5oHwwLqf1dsEgLMniF9!;R{`vRog8sI7^qxI!@ zd#&_Oo`?SV=Zh!yj+W-p|GcLERF?jI3(%jHk4>BGd9U-z%b^A7P37hM@1yzW4Vu

    N!>i~erczCF27j^}^^j9>Vd z!#FA_f)Jif}Qe=XJ*+wyssPwKoW{68V=FX|6tUmn?% zPunm2;YG;Aqr7hp1@%q1o(l?C&a(hNwkB#-_LiDY5BHZ+dceOGo7&|5gbMiq{g}0P z&If;LE9^(t9wO}8Lxf#>h_JPXaGvl|^q0|BIPYOa@ z^wE*gLdqZXSA7)w3E+p=+kjV$s`hqD|6{$hq^R$2SMu@rVuTkMd02Vz{>D$OL;J>! zeFpo9@O3z&V$MGjzJAp3_-%-2sx(LVA+&OZx3?0wKO zza8mU>rELig#SS&xqp`koyu@^uF5VHiII zMd%=3=Yu~rRP7C{y$XI1o^|*^*x0L}Pw>V*#rd!OCGf-OQ_4q(m;K};KNgUW{wN|p zWW6i==+Wat$;Sh@pGWCyz~o1S_awj%bN-n2C&v@c7bt#2y*-?M+qAskK84hNLH_f# zKa2lV7utI~(ESbjo$wb7KQ#Q*xR3whE(K%#;?4i)FLOS$a)IYValWeG+FRf!=HKX# zg`ettRcdeHBp&+Ls}C^kt8rsr!Csp9)2AeNYF`bgx1p?kwO;8ksF#W|Iq`zh>;K zi>3e2g7!*E|9pD|@_BaqYJ~CtV$65y%NIsJ6oj&GFP8qUmWAx|l>YhmKG)&p37QeO$Z)K@&O19>&}i)a4;5BsH;_DO2L^knQ8e?4|`<9YPy zHlUr8bs@j{{4f1-7?8)876*X|LD|ucf_1 zRsVhl!qz@O`v_b6K*K*R_5sHm?FZBU(AP&2$G-vm3fF#kE&TZj*$e#_ZZD+%8Er58 z2hlG-E_5D*`1>yl(qHrD+l!$8EcSw>|Fzf)F8#C@E`WaD zUbqPQ&tflF`hPt3Lh2t8dm;6oye50$eEyE}%qOqKUf=?Y{(9)gp9y=xZST*5z2LU@ z=k`L6+7MFy$E-i$r`29~J^qD%|6{cmvi^nIPuRb(JTo5;d+YU+as9mZLhW_g3!}n4 z`62#=OY!SD{R_2>z2MO+{)LOR7f8Zu_Aj{f(_Xj$`fHxOa1r#M#a^)VzZQGJrJwe~ z1<>!?3l~BES?mQ%|BuIB_($(wkbthAA$!4X@6Uq0;I{YY_Ck-^;4=2YKYjng$7Qnn zN%BIy6Auoo&Bd*LGJ|NF5Q7!Pma7f!~dn^fm8KC%n;O-~vh zFK}lk?E#5zKs@3C@eLV!BJamHB)u7X!oEMK;zRMiUJ2!!_)x@mEs^`2^!bAG$3NsG zo-B=js4Dj@<1?slH`~&W*k$^elpQy*IX0q+mk_y@dyuoZA<;>9#<;~z9^ z;~!f6_{uutDT4rhE?WD<{6Pl*XU3y`y-p4)FkK`~$8#ciTJKDeZmj z@eldVz{WqIJ?jA5_y-N!_y-N!_y@GtjelT!tLJI&XhGVW#y{xyOicgwt9V+8m*jlf zk9bnIz5Qj0pMCA|5BV|zzI1#G+A{>$#y@D-#y@D-#y_CFZhR!$8=j}V{Y7bS8XxJL z_hbJ?cplZnL$bXQjtjc&#q;9a&XIUX;LjJ&lEr_j3-5OvciKz%3ywS!eri1B2jMU7 z((#X(^8-%&G}lv2Id9v1rO_40`Jz!Zp7nW_aHbWi z8lS|QF}^e8NpW?4M)r@R)&>}&-zN$|f5ZpPao$PMU()Yw>+?0+l6>?W57f5sz43J@ zuO)bn3~gz>yd=^IAL)8lAzhTJW(mZg<)aHdtm zVW8l=hI1M&YPg``l7=G2j3e+Yj8oBlDd=^q2@{z3fp#K9f9g0u0%uOIwS=j%A$wLBhY67mD|P8_W33=6b> zx4dVXw*T}H?l)}aYsk?&-jMSvZ2!2FpxaMlhabyF|M~6j3eVYowohIQX8{-1zS$3f zeYU^&ov;4>T-ud#R7Y`h-I8Xjy63BfWay}pY3ynUezC!uk@zSN@ zFB(PrG5#j@e$G2TIhMS4X@kwNHRUFr#sHc`{kzmRY}LKe*UWDH}vy9 z8nQE|??dR0dkXY7runa5%+H`4dlR-;a~opZhpRZ z!uiLvkM~)yy9)X~l7XaC@Ljh3gM3eg?_UZ(yW;{LXiMokoJ6_f`#2tPz5(qw?+@dC z4B;Q&TiC_)Hov}KR?}Zk`XU!UkaS4>gEs#)5Om~M@X48$^20L0o#M|apzKuOneGImUJ&| zIQ(XLDt)Zwr_J2gBK!mXJ@Ou>)`w%s`{cb%T^`#f8c4sljPbMhPM7}!Ntf`aP4j1V z2Jd^39DUw|<*WLqbp4CE-Td8gR~zje!w>I0QS{%~XF-n?r}`W3k@xIkdfk8KefPz- zfBL&zJn2{U?`cu;P!@U@+4OzYOnSMGcuK#Ijru|tUyU|zh9vPmHQ^uD#~m&n@5wK& z5O4ec{qMv5BrQudy`V?#ubry?hTKQb_uQqt;~zHt@lKjfQ#)P}dE$GCR6n+V4+93? z=^y=`w~{}|cVQQ>hCkhLc@dTm4n2bv7eAKV)8mxS`McAFLGP_fzA;}uh5q;o(xW`5 z{&y<={)VFWv&iq`i%OnVdr)30*Zb!Eb?~pe3$USgEP0oEpGj`)QNWq-v%Z&K&V~n)bzkn$Hz|z~34Yf2+lp-sP8n zN3MbPX!=op@h%^KJD>Yb`4{JIbMP2XJ3=@ASo|yHUqO1KKkCV(+}CaSABXC`eMKMg zm)lvsmAClbj(ffQ9nyaWKajj%?3edj{oXG8HTe1ci}k2eemL{1-uY$d|B{QZCZ$aO zMMWPE-Ny3l_{zgUn9|St(0zPP_?OaGRQJQ${H?8i{wVVEtM}s5{Q10(|HXS6sNar1 ztoMBUyWi#GTU&kn-JL!@41N6D?{e{D@h_Ht5%s10`?lMB{O;0j4}bG^AOG&QcYFA2 z3JxCc!Q5R`9tWcz-q}g{eW#Y+vH0!L+r9i-ct2Q6x5Zzh?@KiK-}k->(rfvvjKz24 z@9^^9bB~XITkdTh{$20#@jF^~c=%uF^zk=`H+%TI?sDM(} z-S`*&THFT|-4DFE|17&6#`+{rbKBe>LwdMgG4$B?V2ANXl*jkZX+N3uh^0@{I~Pat zzLB}#WkLErqv^Yc^gSQ_CdxDWtGItC@7_Oz_v^j&7yNahv{&@W5d3jh$@C{o1pl#K z_3j_ydiNWgP__Qqum|^d;xp+fpImPl`FsHUmgTp$_c!dpeVtDFd~zj0RRHSSuxFWv z2mLv&*J1xjf7O#9s3Ez`r;wiwd(`t1w*GjdT3>L{Am#OW<>iz0K;UBr;PGB;K>Cco zBA=`gw84k5JYU83Ncwy->f!TA-Gjq;#H&A_)I5AXS>nNAtnYua<>4uPFVMPiRf^530RkT+QU?=uh!kdHy|pUsC0axxA`sCU`So@n??*aPi_(_a6;R%i=T$hVdti2gqaV z5PlW;dYbQx`2LmIk4!oKt4`eaV&oau2BJU1nWAzhX#cn;|L|w?y=zxKJ^2P6@)g!l zzp>Xm`TZvF2k;&=3()gLRz6{G1`VL~{D$@SW%SHhxPG8r-h(mo@mR7-31A@oh4fb_ zf4+h#pKAQ^zAm(H#epfg?_)~p$9*X5A9O`p-#RE84TIkM4_2nO5d0CI!F?2$jN0O zqYtL_rbmd{0ll~Rqm$c9DgZ%qgEc}bfKfF&o`)7O7`q19` zV?Mq2AL)&^a0Vj(cH@2QCmeb)AFg=J=?~nuQhy?o{}cV-KgO8kZwcxrGWnnQpwRO{ z>CZ#K?0A4?=`$n?oceDPK*^fB)^*G)!s)#w(kntKGL&kvs@r&^z-Z=`TWhuSB33)*=bMxiObUZ zFs`+P_Ee zB7fiNYPtX58pzW&l^^xP-=X~(YCom=XwLxqqo4hO_xoo5$m!3KwZb2aHrpPLAD|cc zd+5)R{`)5UA5UD7(nEgCc6GD-it6tOa2Ng0l3v;yOMj^TVtd!4zjmbbCNk5vO!a5F`618j|3`Z+XZ|%R ze~I$B*3F;kAJnhz1Gn{eZD9WOru~rDjcNX%d_c>i?%#AiTv%Ub?IUme)4n;-m5vt~ zzf)Z@zRmbQcqPXFU1~fWg*`UgHAecbRQx5}b=BGETR?vo;186qs)P%zkRtK7A*Q%M+3VRu|+YYtQ}}X?Rs}y0M{U#x_%PowexuZ`w{Q_ZAoeiv${uEZ*B(~o$@ms~82718 z?7cqvTQWe$w@geY!TY6nML+u6Tc6aiKVAdg*r$w_s8Jzt{}tu)JsoTR$^K6+wEI0@ zLVYE@-qQT^*2iei0Nx*9`?~dd9Pt;(vqQGAfd!{Q_Pw*NH;P(;6`W){eB7S2^ zy}yY2<^Z=>u^|f`gZdZRp9VGjWPe)4mw+70uZe*R}AotF0wx&L(zeh9g0{j2!F z=N(;s!2EFe(I@=qOZg$o3)x?A_;GKBAHTdHKU#$!eU=}Y_$&0Uf1lL?{J8YO_%R^z zu@&=Q*ne(*%#)9S3_mcPyZjgwex&=UNh2`?6P55K>Q!#!-UfyH* zgT)TU%WlgbHJ<)i${#2QGk%IPp4u#b=AWOs0RE)wQ}XA=3*(RM=bvtpKirQx2Y>Qx zjys;XJ?`=c=7`ImPT@~yI-cZxi(QsKnC(#DyDfhdzuuMd2LYLeKPB=90j);A%s-#C z5Pv{Us$bsj@B{PF+2m!Z$V;;Y{NVc{=itXY{jxN}52!+yAIpRv%Tj)byof(s#*^a5 z-5GxT!h-w|eT(-G3_o6jzJ2e7@nbmUNA2AFm?s~@8GgW=bontN{1{33A@cDKw%6gu z+cW&Q>m2+Txd?uw_OyxzznW?dRan%8TTWT~A@YdaeAqH096ex%o3selE@M2g?DMKdXg5t5g1n{On}= z9say6!=KyE!JpL^$se`8dwueAdCH$P=jP8m`MEs9A1rrV{;U=LtWEhN@+14l4u5uJ z_%nG9{;a(S{&@0o2IJ`h?U@ZJKh~d{AM@m8Lxvw%zPbF^DE!!%@blz_z(VV4}X&L!!xoT#1d21w|Fn_MzUYsXFuP*I2pHnz{O|xC-J`f zueo@>*EG($`9c5kzP)H%pdfg<3-43qB>qv(S3!P1j`-mK{`j^)?av*?`&hKs%zoxv zQq4U=Sf78u{kXZK4tzN2D6xF_zhr;!a8gG076)#NY7UI|U~-Q+`5(sqExg@M`Gwr$ z9$d+N#)ErvpY!0wxhFh$F!y;6z9e_tgO}u<^5CVp6COO2`;rIOa?g11aPF%fT+co0 z!7FlK^We?7=REjLxzir}=G++%z9IL#2j7@`!Gmwkecyvu=3esPk=!dT9Isl26}aUu z_NTF)Z#KtXGW~TW*83?uJJasH-*uSp_c(a&cf0VzNfrHzzi9b5oV*e4&9uCSlYSAu z6hGeU;%9OHG7LO^5>NbaQW5z~(_ah!t6Sbk)Ij1C{}!x2UHt0k5f8r|$0J?*rs%kb zf9nz-e^vCHga51PAXf$qk8~+Gh4<2vn_#cIJFNUD zAK}Bvl8EOC9Dcu*{!G=rJkF1`2lOA~L;9iM;bdj>h*SRI1F;g`TzPpb^Zu0OYnQX|HqqO=lRDb zalSA2t5UqY7sK<=&iiFpufad8{6q8?OMJsugZ~19$@p{O|6RoCU;N2}kM#Y?D*oYu z-s9za#@%5QJj(atbLsgF%)jP5guD+781+fKu*9>#|LDg%JlQg__l45ir9pG;!3&eV zeS!98`7yEgYtf~WUeSy5;gAM?Q0qvK^CZjQF0uWewMW3OWdQ`*$j8Yzy2;H?`g8b$ zd0&(7Rnh;BkK+~gM==;5560y$fR7|^l=Sdt2%hrlp64JxS)PfvnoBmrAClIO`Q?qx zeHbCy-o^br@?L?VPszV)f8_IS`Qc2L99NhbobH9gh;u zERpq)#Irp`{#`dV4z#I%c4o+L|ID@wyf_2rd^ouY%mP9Dgnw^M?KgqP1tmWOz83K& z@+7gOCx3kV>na%FDL)`z5^rbTmzz!AubFRASei5L2GOEx{?=SEq7 zY`5c*dUr4|}(u&!>pJdL((Xyl4Do$|LZ6 ze^dIK?ZbEkeTNYC;}AMkVW&2N4S z;q*P@n~4|xF@GG~xSjA-SZ|2_IqL9}?aja0ZSU;NHK{ydy}|Rw5CA2=Pm8|UMuw{R zM)ET+`{CR4c*ObUl;4mq)PIf8qu=}D_!55T@psthQyq_q^l!lg3w^2LoiN{RWBb*3 z{dCJ*e0?S!@`n77p$Ge0b8)_S4Po*J{ek$+>w{g?pQv_(@nZS6NPFyi%lS7exFPQ? z^ZjGKuZ;D#^zTv7NBBnF|DaE;->Ql}j0YvJe4i2ZJrG|hc&yLV{$O&s;6ake`^4Ty z+HVIS@tL^K0p(puhl+_m%<5kc2LC7a_Iv&J5Z*uHg1cE$^v=$dgDY7)>%;haC|->N zi@@9WeJ@*V`Gx+$`@cCDU_tXP=AWdz=4d~e@{S~JInJ-Dzq?uxFI%R))ZEn2^6%?! zoKM7s2*B&{^H5yPy}>E(p}41fodctPR?B+sag()G_`f0FWtPx${(Tq<7;pt-qA&mR@2n^eCYP6i7#56(k8O#K`$ z>#Lk!O#9}N4f38w3a_oY_0J{iFe>g4_ zIH8*IAx~xYzr@*FWPGE>7lAL!d!~{f_8jz$Ox9vgezPlpeJkgyAmdU$-jC0@^8SF<&mb9i z(Klt$H`K?-e>h&enEFrd?}NVy_F?+oEnv)t*YB|Q{Ur6(&D5{y`?>jRL_cf#=aOwo zUlWEte|f{f+bT+3}EtC-$Ch;|3q*6S@Bc_On_p(Vl@kz#dm<3yV5Hdf@+@_R8u!{d1A_ z&v8E61?-rf-{#_>0{Ls?>*1uZx?y0n=iww;O>@%7FV>H%8xA~|tja%P(;tGo+?;>h z!jzvI@}GC$aAwf6e_0>Qb6b8m(+^tl$$Ois{)UI|)Al#)EwrEXxZ{5=S&{#Yt#2Ld zm)^8Kr04u})Zz&j@{c)iI8#mQe@fNw>W^^dwt4l>dih&}BH+~C@6`39Sn!keKjEZj z{hxE-aArqZzqBt}z2C!cPwN-@(0->s!kLat{pES}pK!`!{ZBb?IJ0?P{X8&Z`ahhx zGOb_eYg;|*;oCFy(~6B_3+O9OsJ3V;bz+;V#a5!^| zuFs6m!s-t@__^eg{Bs_BdHxj-UYdWwz|gnh%vJNsD;Rlzy&ul3pNIdjhhIGp&lRb( zU-H-I;Xmx*m!){5TBYaXixk0e2-_(XXg(O{fc>26utEb{fBP-p8pHu`EXM4@@LmC z+4YgvUy!%M3C&q|Jrfn&^~}R@IbAP69%%2jNqgvj?ItYmb1QpS@VFR@AMvf6Z_716 zN`FxPJ3aoLj4L~sTJdKc5)#EW8oRUHQ=oQ*d|`N&^x8y{5mKH^uwU%9W$@b`hZ z?Ah~>zltjVmLNd?A%9zCJaY0!++ef)^8oy-6`XihEeEP&259lxYk6nE- zU%%ALzJH>l?D{_*igV@7Xpi!bET|9E@X+#u@!+l3p?~VW{<*&Hj(^CLx_?RJljFs? zUrEMuk2k)U-j!#b7k2&ASZ{XN@{#?P;2N<8*Q#{9Wq3-gN?NbAc^tN#P-fYuL`9<_E zKhT5qF~(1>I$qb~rxocdYCDm58UNjMH>vcHN9^y>oK*bz?mFhLX2!q7)0*+`jd!JQ z;BP3n<6YesDe}Yj1oQn?KZ(97r2Yf+2lS`-Uyc8OBpNM?x`*}Tm0eLy6zQFmCiM>aoYlT157xVX{vg7|^;dAN>?5ACvz8Hc& z9C^L^LJfzr>x&UzUu+z)<5~LiAG5x=miplJ>Wcvp7vbMJ+8>_2g}zYR?V7&}=!+|C zex)z^ExfS4xH83yz8E==zMy%5kK>>EsL1dc1zUYF;_8d_7O(UL;|cL8efQI;FN}Y( zCHRNX7fRv(W6>A2*Q+lmZdM+$@(%l==Ie{~wV&$?&%QwXm*L-n`eIFr=X^EKzPQ}t zmA>e+@WT3HZHgCtQ9F;mSZC8KeWACHG5)Q-sJZ%Lb()^`Q}<;S#{BTpsV~fa$Uls} zpnb3WC;u4ii@|g13;H9(;opTY1N8UFc&ReS^?WgxEE#n5!F+wOcJN&K!CUWK!1}{m zZ-5?e{Q`MDocQYxZ+(IE-ueah>G{?t-g*S-5$~C&or7z$w;sih0kZzg z`Y-f)L>X}ZF!sf{{S{n)Ea`Op6<9xGea}zXix0)+&W!(})Ae6u*E@5`nkBA2Ks-J` z*MFHWr}|`_KfWP9bIFAA*Kz$R1+pfB}DSbf2G z3G8Q!zcb4EFGi`TGr@nCfI0OZ3HVFq;G=wy|AjRE;4k0Jc!@NgPsU50 z+gD2dW&f!wjju!giM^i*{vW4&@4mp1|JVZm%_43{4>SH^v3?xSvp;2E*t(LTUc&@1};xTMGVu+~uI zU*AX8{h$*MiuBljko$Ambi9~%A0FrjKjr>x6MuK$pCg`o*E>~vPTfDxKcvrRp#k6t z?DL7eCn9AqsQ3)X`y!!7{Bf*5ukI7q^~0Zd6Ps`J_3Ve1G=r0xzq>m{zp{PCzsday zmfx%#lKq8hl5br?SlZ9~8V8#I<$6Ez2lks9PdfeKq$Kx0s{Kj1?_cDtc}BfAkjHs1 z`8X<&|8P>w$$KO^J`?i0hzk-EkIDTQ`Y-dkJr7|&OT_iNioUG8Gd~HOs(x4ehg@GV zzr?GPUih!Q{E%Ur!M(Z?iD);mqRn{hkxh7l-q5UyJge@;;Hi z#WiM7=PQG`*__;eY4Ar8kHGpC>3A5>o2SEOiudhdJR%-c_#y92p+4Yq3jc!O!x?|S zeJ(EcVFFX*8|jzm`x5(KQ1`E+Kh=HfkIDD~{pvcwK*xt8zNpWYAG~*iLkr*+>BoKF z+4@ZR(F^K+E!>}iOB`%^@VD58o(-D1e?pvxkp+XU59cfMHQ?oZCFn&w=ps6}C7zn? zO@emj7k+^Mb@Uh0oBO1T@sQ|O=$jnn2l9{oM%{jluOi|}ML>Ct4e%?HWkYPAyiY{^ zkSvk?R)L?Mq<-qBM%D6`d&@Kc`7r)5fAKu!F2X!diSn49sz=5z^y$O^A%0BX4DX*_ zv`qBP520`FZ3*t5?pe9%9xKRpP1jUI1v3mBK=!c`hRQ| z>7{*~FL0kwTpN}JN{fP#zb4~(x6U62BQ#hLU!G(9{zPes$Dgs}l409E^j8?yRXi>y zA>8jdp!XZ(J{g>cswbC8532hyhk|VXS0R5Q&r1J`Jji`5>~EA`O)k5P5+M2!{6zV= zA;J`T%1a#7g?{s%MV!B+4wUd8^8N4j;yywIJTB{onSuEZ=~oT|EAMyUJaRE!y^{RH zmm_}w4=aaELtwn$vf9310sMg0|MLC{_}?Aph5z#Y1MnEH`CTXs=K(mr(O)Y>p9(#y zzgDa9VEMgT=x2YSJrkeUucm*t|Jiq^_xtGc!Lzs@RPi5`ApO`Z%Ezzf{!YyQZxwrD zhuk-pBTW6Q-}4Yy)& z#Up@M>+}=T_vAGHB)-vGpCEnqJduZ&`d3T&BPgkxsOrDujjU9^hb3jmlOT0 z&l}g1RjU|(*r(~?`)BA`eBh_5Um%%(Am87Hy`}zuZ9~XMmL4A-3jW=6^Rgz+A1n*- z>MQmWqiW>o!>C%){efc9o~YVp*0&mNm-z|=3C#9ceZ%w{E3%#fJ}>zO&&9$n$#|Y!S%Z7$wZw}+WV445O0n9&uWjqoLPD}zW6*T`aUJeBlZ%70Mc@TR~ z-miNc9HBi7dky+_bQgX#^1tLUHUCV#FxA5IInd8}I*gl}l>S_S^AwaHd5`Yoy5Pf~ z2-;ZRmLDR_2J5kjPuvZNALQ>_xSw^p&}QeWM>|E|J!kOffAohs??-+1my^)HYXocZ zhp|2%)&-isxOJXB%_o{0ybq7%+4tK1rZvEPTKr&6;!#j&j_n!4c@fFciuM4{`sKdo z>yZ9y)c;Qfv`S4dF|#yLLM1vT8i@TOx=Y{efS&xeA@-dETV{vmxSP2W|n!9So@ zut<;jr-tj1m3{yW{TJbWhh`J?@dLkf{)Va___!qU3;sa9x`GP;j_)DF^a@0@x7aCCh@Sb*LZ-9P$Us@;Wr965avA(R~z6V=A#xLlPEd7{Iqx5~T zJm=?HT*8B-WCs2NT#8FgHNG2S-yGZrbvZ=l<3E&NiaSeU?ghlJ3xMym{Ui7yH=OZV zd@&empm$9Ef({U|_(Cu=?%;EUO#WOjR8#n)A?h#WEFSwy+4-vm`}&i|RzoGAJ>XaO ztA=)co{}vQHonW_|;|**|SsKE@?~E#?EU`~ai; zgZpw_;4l89_=o!O2aKPo8sudi?)S!XQRclu@E`BvNcm#lsr((m&~wOv{1`t6_vHZ} zSMV9a?SN|v9%X$I;D&~Q-&xH12|Opop#+NJOUw*ekku;_cJZ>R{dPro>c^yu!P zM-(3V8S8VI&(-?;5atKSvrXTH``uuEU^L}k0>9CJm_INa5RXOw1;z*FV|5<{;V<&O zNlq8dAkXx`^L-i)MrGfw16)$;UG|^g+siD!-Im`5-s$N#t+C@M{KbI{%8tocE(ezIfcV859k@@#eOzET8WY@c!ZWn#jWF zKg@rfn}L`2Jx}L3|J3S$<-Jpn==`=c2D+@AuQ5 zl|cY`CjYP;R{Vne!XC>tKcnrjZ?;0;hb$bQ((jm``uUgiJ=B(f9%;ej{1NY?Rs2!u zC-$x>sr7*J|8&o^_1p0XdN=Hm<5|i-gZlAaWgFhJMGr`Oa!B81=L6uoXG-k?^E~I% z`(*vV_NVt@meu}{ogr0*x7JfMENUS|6MQ-0L^B>lniP5im&6WI6d)<1~;#CoWl zheLK5&u7*VuXW@zL5=KSc`s}|?|K9LA0+)EuRMP|{h8c|%D>=z4Et8CU-fy`tUZ|3 zhgo}`?Zxw)%0KW;SwE=m;`sLL_udP&-^)$J;~V>ZqV!Rr-pI@T>2fL9q5Yqj+4oOl zI^J`)vOl3OAkQ*h;h%tghW7!Yt`2Bc>yN61=J5Tv_y_7yxm*KU-Wvk{w}e_BQ{Q5I zcbCg59+&=EzQSbA7ZRJ)P7IP70<(;Vo}Dk?XO&x9&FHDrRYICWmev;J*@99#(MZB zc8AP&q7OSdeEm2mv^ZXYvIpVpVFV6JWA$3XS~4>3S4CU>unx?y!9jG zf&OvpPe=RUANTy-T>sa^zyW>G9}impx3mxbRM@BRf7wkjC8_(Sy6R&LCV1oye}{Q z>oWBt_P4VB$_3Wf6_Jk}W_?|e_t#LNSzq%)0$lKp@)lfQpOK}Y(DyC+2NspLFg^X- zNBRCb^c_Hvhm(`k_XXR(B2QmE=X!hgLs$-c7Hy*tqP=!Kz(Pen7D4<{dlKcp#kL3h z?ZbF*=cC~`C;MT@E%Q^hycO8rpt#EZ1NevayZ9TnETTW8g#LJrIq@x^FF;Qp_M>cl zLZ9kykq5xAKlFMW?`1sQ)iN=CBBu@rK);}W_4*G>F4zb61n6(ce{gMXD>A51^074g*GZfsr2h<^DLeUrTvb`!UKtH3@q2YQN;|Lufzcp(OrA5wg=; zsUOsNBkn(7JXDin$*wN}oA;kPW&eZt_Q-uN9rkuHXi)#ae*t|=`nkTJ+JXH9(9ieN z!I=Q(zeIkpUn2XpLO=Ll{a6V@7Y(Gni~M7=z$3ljC4W&!mHXI%AJF%QV1HHk9me*2 zvcM@l%Abk#M3XZX-aD7|EBrfZ|6w!a3I3hxV_1KoLbV=a`yBiG{4-kyFdk30 z%$xtYADPZaj306Kx7i=4Uu;v-gYsa%sOmW0u%Ci>l@!K$&dlfLJ~PhO)XlR0jq-55 zF`cggKdAj#W%)NRf#_RJt%H|N`?MZ z&%ZK1%X=l@_j2v8<@6Gqo}@gMnZ7v+`GCGy4u9sk=VJ_SZ$8%d#o77Tk2kr1`B)y1 zoU-$ATR~_`=VQ4a=Irw^bQ9a6`yczuD!wH%A4~tA{Bg|3%KQ+aLVx0Ybi|{cYd+4) z{3s-1{0+xJuO4uEKK9GQd@S)@?tIL=YCgt&EeD^IqD*@|`yKR{`B)MOeK}drn)hlL zzoXU{oR4w0%XE}e@_4rS81Hxb^KqX)ADj5ChMteBeg1r0$l3i^tWPjr#D6I3anOhU z>f2HSwR*nw_G<-S>GS7fy?@L382m*1k=(Ch^f}At_?y~c=HJ$IKUd~oz``7*50PKy zU*Qk**@~pUmHI^HUyKjfAANSdhdq4%bZfuZdz_DV)8Ct&j|;ND)U@+4;$h7E^wdK0 zaj}q@kMo)NSoR0f`BqJp9HjFxq#ycYGv>1e=i}B^!50ExH6NF<^Rb$J%y@+Sg_JL~o*VM_%OHOrJ{Vkg1Mo7R zV?R3UZ$SRko`*i3n7*DDKI6mqME-30?0Q_Sx0zn`=Y#J3r%!hUmn&t-1k&Hx_3+Iy z8561I(?6Nk+UfxQg!M?ae(IlbCg+Fox_>%sE_(?9q!IqN8|yXPA5ZzmdYSe{i1;h9 zM=;+ao-T9$668njk7R^D^P@hz#}eA{1Ap>B5;jRusO4_}@nT%RD1S5LKSuu&7I{Ow zWh~P%-kVeQ|AsxSV$aGC$jboI+x+0qMCop3$EVuQ_J{#Vdg`Dj4nvW@w0wc3e6dHy zCAY~BdrQNRQt-#dlk%@@ANG5syenipv^;g6i5CX`1GtY5`$;UJc?J0e`2oM>Yxx&q zum3U1SAT&T1KxZ88T#z0`a_|Fdo4?>%@qJN+i+JZABfkUj91m=2inw`f_@h?n*p!Ni0k^K+;UO)Y~UOykl ze!-Wq9yIi&>34YPKjWp(%HPej*W{D@Eq4P}{8%sX)<0<&_lva!kG$KtU+kqw-anQ1 z9#7MyYG`+larHC}$y zR|$5x>D~HB-)1j8@5iQni_g?o2wu?nkCI-uz9#zbK=4tI-YA62iSpT6IwUr_!|1mk}CwEVNA z-{z-3Z}~g?^ye-AEhnQ zalic0pm8pKZ4N%?r_bDH<(5zSw`J33?yGUrZ$3gp)qI{!Qas&3^9p9e*(S_5R?iPJa3$c8EI0{Kokm@$5vK`E=WTa=kTDa6at{ zv-9I6{`?r){VY8{PfYKB9rbNJPkmb#sIL(`>$DI46fYiY!=AFde=7DF+ROGgy!N*v zTm_%3Phg+8{6PBs)9{~|_^ro6zzF9t@muv`;Pb25dhouE5|#(zQwmN$8uirm1KlW?gcrVnSiRVK6L0Q&Y(jM6Je!N&$&W|T!ekcA5>$!8rkHNnCiN=SmJYRg+ zkRKlgedNW5(VoQpa<;$F9xq-jE6;4t#l(-np7i?@`w4D;-UGLSC;wah_%hV*$s6hq z;}Sii)?U5%@}2lJT|Vtm>3?0mwy(t=zWDN;_%>a>tY5gu*5!+RukG!NFW-re)8)(h z-6>z}>(^NR#B|*o{|o5HOuS20KaK}qbM(hgUO#Sn9rZQOqaT|K=*N2SoYOu}KG7d+ z?=Ez;tgirj<4fRk&JX7uA6b3J^~mX;Dtev`&N%d(r~gA={+li5o*$bn^V@&ai=W7@ zCmG)@pUg+pC(6BO#`})x&ACmoKKT=gkGuH%YrOpQkMcOC?1zawv;4DCLz90LxF+zkE*)Bd4*Ib66nzSGg|ohr@~U+48tc>X-HjTnvdr zBQ;*d&!_3bkn?MbSNkb4=hEM&$B#J=hIm{Vu*zS^_U2RikRJR<#|L1Hhv-$5FY&1& zZw)~FbbrEMF8J#(UPM0Luh0&jQ>ltaguIpDe~>TDPyJ^0M;I>-bVKN)d>*{lOOO3L z*)J*{$DdDiEg`)%Ng@2rEyw+(=pGvk3KsSomz;vtWK z!5{ixQJRtmrY9)z#lXY9m3W@u8H^|N_dpy$;-PQY3gC+|^bLNMBG8{GU-Z$`ZBv3D z!1(~uXZ9!3@sIHX`C~i)`6qakUqpWr#wX|h;4jV-miTUzU&VV_l()2hOR}FW^rJtk zaY4?n82TrsDY*Ppe|Y?7dD0(^eQA11D?gRK+my%g1$wjdjh$a~JYg7z${$Ms27e}| zYb*?(sb98}8pUlWPwJ<;)~(<7x1#+dNTrwZWHj3HB8j&?-VB%x<8y-d7s#jT!+0&r z*x~*G^u>JoHMcGOISExV`e^?AHFMs37X7fMY`>&$k+IlMZgFpKHzqxk& z#i3toey8~tFTx1d-@YebeAR+mgX2L944NmANWLemk zOaJ%PR~&rm+fUy4@Go!u_s!;Sp84b#TR!pa4}3i>XW`$z*L?Ev?`=(*f3ol29Qr?1 zl9!)szWAM+n=k%XfPcg`zWBHJ@r~x!Uv9qo6e>-wXfx+ z58x)O<{viyqWSWZZ}_dh-WOc|$W0%=>2`h}KXz>Ot+y_^{q~(Z1(qM(x88dDt=+fY z-n;158*abh_S-)(a>LHsZ@B*U8+P*F{O#B7{Fkd&w5)Hw{DW`(%B#QCZ2Pq(U;WqL z_|;E;^YI`4+K+zcTUi#W869VF2b^59J&({Aiw2irKy#7Zb6Xrncf7^&LE346L4 zVTbXM39)fPW^hchWR}U%?vk^dJ?t4Wu1?qq*<~hpM`yx0Bil@d-C=gg3A@o*lQwSc z@AuyQR=xMF2L+P48|-@BrQ&|?=ic|b|9#J?%NPPMaO9#V8e*fvn z(^DV)(ntSb^pi)WwOhaR(XW?&`s;t__7Le2Z)^L*HdzFH%CS~XDpvpY-VdDrzAr;f z|NGv(Z^_=|zma>t@!GwY!dfbM3rJ}FFaOiMzxlymymaq-KYZzjKd;Y=Hhnt&=}@_3 zB&VLK;olFNZ{7PZU#nl<)UEz6LQ;t)CW&Kaa`-DH?+YyXD~r+eS}&NpRz{ml2j|3Cc9uW7jByTALx|M%W&Kl#}9 z|M%V3zIpHe{?#vh=Udy~y7!^M?|t{*-}}3l{_DT_?@xU7`}h9s-}}}}|KdB}`OX#N zc2uX=Nyk&e<^Q-xp8fYy8^%oLL1Kl$q$a#wO#YA_UX-VP_VsbL9fExMb8*G2e6Pk2iKghR(7rCYjZ)94zI3CqCK zyp9!@zUsuk9MX#x;ipTAUp1c;z9IHvZe8&=U!H^8Qn5 zO(-Qk-K*#=XRdOl^n}C6O_HJXy~g8Dmg*<`#6@`vN~0D-VUVk^7OV?n47=e?jxmr zpIn52QS#l`WlBTO_u3YBBN{J7N~M=9%Hv^rkw|*mJ?%wl8+NR=%$!%VX`Bl?o-`VE z((EJNyU!idhd(}aYRhybq?4If?#b1Ycu1Uv1Z2|-DS`1$tkH+#Oyl#PFV*rlHkpRd zN6MrQ;x<+czSJe*;%u1eDD?a ztAwJ(vQOQly0*+#_=%X5&-|W_MW=k)sCLu-CJs@4Ch1ijr`nqzF`-5ibxIUIQT^^$ zr8=A*7K(?A{1{)hFS>h@rj^~u|8ggo-Lr-zlnG0|BB>>rap#PyaAw@aU30V+bN3nA z*FL2*N5fKn*`wXKUp`2E^#s;ewp>N(FS6x2Z_5=>pQ%t<3bwzp<+N<3J{>=n-S42? zZ%}=P!>MfDNXtlX*{uy<-}>y^Uwsa}N%gsKSn9KXfA#sV-lY0;-qP#N8ad(W`B`nHsiqfcau`?68{!jvP{L*+lQW`G%I9 zVC>A!GxGiJCt#L!OQCk@C5kQQ944nE8tE@dTvJHeoRU0!ym)Je^Q##>zq%-|opKX7 z|29Uimo{FMb0_DVDtoS`N0`~8(jvJ$A%bVqqf)Ubb3|DrN0ce&i1PeToiA^E;{THK zpT9aKN1K13i9&KAN2)iCJoNk~=Xj0UGxvBl>^q)y_9Mr|fxT@ol-ZZFf605dqaV$S z+cj}|k^qj{yzmn^w*MFT}MrGH1XL^qGp7ZZ1E_O8e_Oy6$XiE;-ozluUuvuo~ ziyrY_FUXC*dBmF@W%8t#{0B4n=f>YG@}|dmWj6iIh&Mgr&tIxL6~6TLa;XowY&ods zbMm0WThecFo^Epsk&O>OGFYtF*4`4=c=!ZQeDUBBO<$HS3ydF(3_6dMmBqdBp-VMQ zubMA~wNBEHr0@B{$I|z__3zZZuilFS>s`}i%U)FCE&F}cy|3*8>R#$L+xDp2?EY{1 z+79vlx(|n>b&H(3cPq)Md~e;#wG#25Quh*{Tla%UpAd)X_EW&H9MlS(_S5&5`s`*!rM0eWEDzky4pwPfA%{kg|x&x0{aI#%BYfvtnn!!iqs4lx{KFozJ2Oq6;t`BNF7Au7hu}2~o z1r)oTBNaXJxP)-%m|l0)OX@|TrzBhF==B{kO3GRz;b>Bh;ZI1MUU0Khw}kY~+WM6< z!gBo5%SY2D<=FC|<3HOXxvL!7B3fo!*3XG)-m>bgrcU_vE~_a`yZ-wD{co1D-0h>oE z!=wZANN3_>a#X=S$lB-Iw;sn2KAQ1?dI0W=cWsYn{EzsHfcWeg;GYD?Ynh$xX|-#g zD>#{T$;ObbOLF^iKgfEYSHu|M-&N3>UFi=WQEyz)ni!SYA^Nf=HUJtj)u$K<5Ni^ zYdgBYStZ0=ybRYis>^>5K@tOW-`U2y#>Gfgk-2AiYM)=djk!K~^tH_FT2~@gwcC6uv)EQ zC(S)o(>jiw`IP*PUK`Ps=nM?9;VP+1xza5c_d=PlT<7+MUk;^}a<3MP@v>?!Zroa;VRzNua~FX4ikwwzv4mFK zJ+{ANtD8BDdg}P_Cu{>(|nd}F>R4~==_Un%m(e~aQ9DZRLAU;39R{Wm2(^j=-xaQEMMBwp+d zhmNFt;a4K*Bi{IzBdH^yQ&CCZruNHGnEdMrOYd~fn0%U)&l^q!FsgYO%~y}Tyjn_3 z_Jy5ZJs$D4u$EG4B+n7rZhf~^`?h;T?{w2ax6f)polT#k^X}1Ac{Js- zpz-;~-tIXclz0E8#7}0ff5~ck+*!l$Uk4E*Jib7W5_rFEKRr|K>6b_JUO=Z`>NVrc zdD6{L+}jbQ7;{o`^(FcVKk@srQWc-luKLP7Ih~Q~FB0u+&hhDqcgp=yqt2?D3CHDy zB;mm$gKq6BlwCcP_att~rEgq(u#X{T<8>a~HD0fO?~>lRKE6x5v+l4=Li* z3cJM@<%xIJEwcH?w05L^^3~r1#(yFlK2cjIb754Ssod5lbC*0YDrdLuxr*G)wsX8h z?iw$5W$zkaEU#S;$iE2t--<}j%h~^6#=ci0LHi$+ix95;+mXQiXX0Or1nqzD2-3^- zzJUFQGx6b)!2Qbuw(kBTCA$C2`Z)FQnA5?y7wedn)+o%vpMob6k;jYu^1gzmMiUs-@2HSH?#cmGGYXbawEq z;`@Da(D9j^JN21xNKUI>a!ix3ynVtK9+ogB80UU-8G#y>V!8L9>xh)ixlk?X#VjAc zLg9K@==A2NH?n%etrTuI&fDj87ta2CVQnqhr%&T>5EGUA$FA$j&w;9EJ+F9B zo3d^3%G*+pKX_X_s6B=B2dj@A_Vzgc*h+c6GVH~F>?NQ4`^j%hzC0c@{*5ZfUiPyW z=0VHL>F;Ns55v#yW&X(e;+Kz_V-MdK@1>t}eLd{(Uh3gt@ADz#r^hq6Wl+z39_Tn= zk9|=0nYmb%Z=ZX~b1zK3{k-1sd$I9hF^8or4{GOOsjqzH+)Fv`Cy$4b@9VYSH^R06 z+aIdGll=GP%PhZ}zvcY;8Dm7ias^z|Wse4N6q z6!y&*d`-`@rXohB-1QB)o~8Go>Gdh^$E#)j_S{Q{`_`N|(lsBX!MAfip1oJuyC3Hs zKHWFvf&3j-<6xiXZ!pe20@7oj$lJ?v8ZaTcpEvjJMZOcv`-BPKz3blll6?Y&<8{?4 z-|CnrUY=~oy~9dd*?F~wd%@CKIV*8iIORI1NrdvPq6e*Wo>d+(wtReT@jWKR@Smtm&1H@Sy$k6&0I$5oUE}dNmkCt@`Uw{ zF+EB`yfcz*_+Nw9o+FyxC!EndVdQ~$Cl9qRx872|yE*mFGn>!9$?Qwg@u>5!9?Gra zvTAu$4VP`F)x)FVj8pQijmYZZV0QH^XZ29mUUrMWsZX}-lHPf8WtVteUEDRjtRC(b zud9o@#XCEbE;6*W^z|wef9T%@&47rl&&6TSLeh8|LPFMOXRNcIje`1 zK4L+<*XjM@jI^`y7_0~b5;-C^gFK}y6H3Vw0fxP z!$B>aUA=Yl-)Z%bMgV$9X7x~47x$2Vc6D-#R}bC&XX}?&58d)*;&WCH-SnB&#q8?g zGtr2?2TAUVIbx2;{l2sU=<#C9(&-#es9s7u(4ksSvo`yVQ`FN+IeZ$eEBc=Ow4V1+ z9g7{dcG5ctxlOoFa=e6fCjNKmp8@fOHuJANzrJhLu799$<-M0lj??651h*sU(TKF! zVQV8ja{Eh9pF8OOP!1WK`xl&RZUJM&*Io}he%XJ#bUc3C3>_al=A5KT9m;kHdVro) ztEs5LgWlqYZA2aP`g~JIP`(jKBP<+u?guT5FlWTRUiWMjM%cqWt|*K!%rV&>>%S8F zpjT>l*`}b(RXOv&5vh*!Z|lpPPExxhFsO=^=eQD2LBJhZRNs z`t&X~<~i)KFS8D0mjIkBn9mpXyu>6=LAZdL z#rm`!^BmTv`B>v{zO@iqzbe-2>{!!3BWy{ZO?DkEx&| zGrD`mH;xYJdeuR1qg~IExXm7Rn65;~3vA6$nXFG`kNstj2Ypns=RD{we$eaF=gFP+ zKg8D=9`=aYZ+r)3+a{O zH|nh&g%MU5Va^Xsy4Q#cbJ&mf9QLEPK9z6Q*>C(_uB!%}k@J4`e-+Z+%g#sjzHrcr zME0oznOt(N;^nSLWcWRum>qZz~Gf{P(t^ zkmcW?6@|UmoA2sBy(5qNea>n@mfzcIL6+a!YC)DipsNKz^>fz%^M|LC|9st`%BA zIE+ug@O3bshnMmm^y^|idkZTFP9HtkZ)eQ^#EWkn6;=?OHhtJvyWZ5RGj#Q(G+t_k zN((Cp?g~obUPgDOJ?Jyfn>vE{NQA>Q!opZV!2J!5PdVt<UZ8Y!$GxO0(leo=^Phr@+u5DF^@(kBkoJ7>k7 zgMKB2uT=QLL3cb9o< z!!*LeSV3@Z7ItdcsbT-S)%)q)>u>hFhwG<)$eD_Wl-eTn3Gl1(X*}JG7?pdbq>U7~51gFjm&micL z_nY|3g5pOG`qeIsOI)4vjU)Dh4?9ffhQ;#gx1r*~3PNE80TXdy1%ZC=Yri9qMpzgt z2!*#8Nf&#)jztdD)tAB?_4?c)T^M175#~-~g%MU5VbT){zs2VKB8H9$84(ZlcWeqP z2>Ff+_p*Y(d3fG>{)jnZ9^v?)djR)x4YBYhBvW`3l6${@;Y~>P`HvDg=jZXY%X1G0 z^i4=X&n0lF3U4v8GobHE8@F03F(jl9u5wXlNlaPJN&j4)h(%&vJA z-eOdEi&1ute9&KRDU7hf2=n{wLp6sLRuK5OZ@=ODSz+LPcvc)_y}=yecn-f2D+%$C z{K|7Ua@5#2)QF=Cyi$u_c_u#ZZz>R<_l-!zAJ~-y$Iii`Ougm%O$Flfz7dJ|yekRB zAK2eiaO@0c>n-0Kk%-UxJOlB0R}zSSgML$C|0@ZOorB>_f6Moa%EagWMP=gi{-QGR zujel+AM}|PM@E777>A?b@-62$G*lAWGV-SzvWKRou{s(FKYhG-Ylq*9uP>6fK$V>@ z)=#+p_^B&<-hPt)yEk2&#Fk2&#Fk2&#Dk6x=?``!qv zcipQAs;hkU}lj~UJ5Sec|bB58`=a_>>&4VO{)7==qIe3Zf^6n=!lMHCKGxR}C6C>(LZ&-dCg zyJJv}zUNFNd{S=Xa$b)Sjz-E~H6|t{i^6kf?|OCCY?p;U1wM4_c!#;Lh+uoy{oJWq`?{MvY*ZuUl9qi88$6Ftu zU+=Jwyybez-RFIkOX_xCeT3U0ApD^D<<#jj576HN>mj&&9DdOH zdf55pl;3*(gW4Y-cKvaE?ivmp(YvNN03)~;X!!#CpOE}7T=2>+punLUkx zgMMYvHzu4C9k#Q^IXOc}V$Mg;l(X+YFV~M)%NN#bPt54Gs9ep-`%v`)J*U=>_v*7d zZT^TTMWK(B%9Y8uTPXQg-}jVr#rFAMeEXJj#nv2=>&-b=bkA3p731SviF{9S@p;cD zTvmGgRp&zI%}8iVZm2%y?k;Bcs%(}iD)ut*V8)kEB9OT$Yhc-ZrGlAKFEbg)6dm_6 z=|JY@TfEFFm`io1{AIgY=H-k%2s@v{75TW>j@->--u&V_az#Eau`}nZj}&nMb{_Ta zKTW)+O?_>#>ft=hBi_8QoTmHCGBe=5Pv22mc6%~1H zOCUepuiw6~mrH@m*{{4v%hNX}y_~XnKA}rBO|P0Sxlbm9()V(GQO=r5i3#btjIA)U z9JDTTsY^V;Bew5KZON!^)O8+=Zdvbf*BH*^uKQG&w`%^*aw@z1*DwXz1JBv@!UiMe%ePzwrkMwrd9(Gv! z8s{0Py#co5*b%T#_3q3C9S;R*oKC`r>e$G!_Yl9$osV+3JwB0hY{W!}JSN9;*fT%L zJT@5O;~?gB9$+n-N0D!?wo^#%{gjzwuOi)N=GZGg?>~NW@43i5ulCp@b8a^OJ@CHy z9?QGClvj=+via}C2i4hLweY2Pzgt$}<2nTnhIM*9VxsZr;ErYS@ZqESLl+JoF0=Pt zaalO*-CwwPP*zpk{T;iX^d(`Ly5019-9OvoJ+6osrSEb4e#6;&kY72HJ`xW}`h)J_ zcD=vOTR(Ew&AptXEV&m)XH^_!jqc$nC_8>`hNOR&!Y8D(6Uvs2*JH+Pyd2L_{7qlH zSmJDY1eeXjzYk-bfS>X22gk5}#=kH45@r0HKQD|UKjY79&^6om8Gk-Qfs8*N=U1kM z+bthl1ot!ke8fvTo1YP>XpEuTmr&wP0gOz|G<{<4R0{K>&S0@?M5dcXdZ`Jgio-%-BaT_$H`ILc3nKRN_H znCt9E@(eNG(PzG6OMg43^ga1$_*Z*2chKkBL*35#+Asgv1CM472c5&;55s9V9CQxL zOyh@5DQg@KvUe|eb3PotpFHx7p1^Ag0oNXa&fE8THW9>mj>Pl|U=fb?Vs@5&7-i=? z9yA=Zi zP`F5szT9)H;vI6#mdH77%berGH}f7i-1tK7Nb}kt=cSUBd!8B2YlCx~ny~@Le7sK% z`_J1_dfw*q)n0q#G+DBB?Rk6$d%WjOaXD||{DOMQ9*%8%Z+0updCs*Nxj8vxkF=fe zgT9Nv*X|EK=jHRu+%q^F)Km7V?P(4RxSq>4>^FQr^B39k%F4`s!$D=vm&g6s?(>Wxy>RGy#`~Rrvi0q(RAxN+PV=L8 z|AUeJ&Q@~G#J1;7v(+AEr_2l)+F>@xmPu!)>^}E8JLPLPZym_&lxc&v?qE;8{PN9S z{@K#x%$7m^sW7`av(th4RjtD8rpLJXMgzw`3~wEjM`3o$9LD8slQ(&GlZ7kysd(-i z`NG+mpfJ1X*wZa=rJ}ZrQfV&e(74xm#g&6Yu5z>Xq~CwfLO4DWWTV{&0QBAC6PFmBPL&eUg7+ zcHP^nyHfo2@sO z!&ZRNTA0^&$2<_59ezyBi<7 zUV8qqm2w?D?pqdb7IFFYoy|_4amwtz`+CrQ_jR$nbT+f0zxmrn%yRmci|Jcy(GPun zuteTZqg-5T)R?|KJR@{}eZ}s_HOJ_^|2vPI-AbbG@YRxVO{-PxOVl^y8}s_>7W&ba zaNk5;l+zOGcWlp^k$Q5f%JwWBg{omxYESzkFt+Du*d2wL_Xgib&&$|%GsHcmS$m4~ zVz+n3oB=Z!az=-*cbZ#9N$ zqzpVO@qWC=9$MzmdyULCPT$-Lh0+o(vH69zWP%PgUMeXKe|+eaF>MLy+B#$16Hh14 zrbs}xa`^}>3i+@N^rVZvUp$TKpX?}-8S<}Ay=of&K+~%W)_AD(@?^XP-#1%9Z`m!!Oh>gpc#XWvr|h5qau2g(#Jr>cei?9_nt z?|!KduYWsG_t8>+F8F6X7|_df#y!oK_AB&ffB!v9{kheH|6J(LzWz+dqrU4idTo2Rc)g16iw}5@sxRK;z6bSC&kw=x2`yX?_%S{o z_4jxG^~6ybFNeCn7v7kq?N>No{gFAoPwDY}8tWN_x1^r_M^yfl$|AYT)4gp%cPn`u62f1fxp({TsW$T0=)rg!ds*C!X8XF_6PmY61w+Sst%pd1jEZj5x^JMF!i zFF)axle(FcQSu?rDA8~eg`bnK^PG$wMIZLW>nPK(6QBF6hTX&So)o%=W(t4A6V@DC z5ccLMnrzFD$W3JTs+_-#(Z6*oGNA$eezff6RI1vP+HgdD4 zyJ7#6HXKRFA4fI8kS20A-QUCB|E8O@G5^XnbJB(%c>iAsD8Fi4emmYK?fy6M4%Zp^ z^B}Q`$%u;&j|AqWt!B)%yN<0 zd@2+ENweq+KWWmw@Vm^MFI;Kl2@OyBCyYFq;t9XgO!~s_Fyp>(g&FmQf5Hs=!f!Vz zU-)gN-xn@7eZKJHrpFh4%p`o_x0-feIBHsb;YUrgFMQm@ec`v5dSAHA)cV55Otmju zYGS_dQB&aymzbz8{D>*>g{2?(!beOhY&Jt-w4bac_`>oyTgV&l_!|$e&zO4Solcnp0|mOhRr>B1aI|g?)S`?`yKl& z_d9LBCx6=wN5?JqPi*)b?)S3&-u$K;9-Vf-BPZj|LnX^%{HlC5)j>)J?|YXqlmDwR zGxFJ#&*WdoLy^m3k9=C?(>GjXM&*;1Pvn__u=;-d1E1G4U<$Ei$pQ5`Ip6%_oXXUt5uBl_@1A_gch0BY(>$C7&7j ztZMj|ip;WnHsurjFnRjtJ%M!QfJ^>rQY!OXk+&8&CcxKk#q54F~>xt3+eNDCUnoN+OR$ks9}wie~mu> zXF^Y!m<{Xmej1M3us*-1;e-wA^LkEv%uTQPmCECbqd%3+uT=PSwHwy_N<-DAPsWYz zPwgoUMa`-WYx^yz0e60-{_$-)F;mznq3gW$u%| z%#01sNLb6a?1Y~*%QmdfCu)6e+VCpE|H0)ozBDA3$T)QB$A+~JX;{nqNa(oPjJsj* zt6y@%x<9QiBV!ZkHM}YBaX?t}kABtilAqS6_RDPk#isQe-uS4|zM74XnwX4p4QqXu zhANEq-)ww^Nyzxm#z&3z>1=$|OxiHgYd_D%SC~~BcH?FLjox_Kzxa`C{uM^YgExIj ze2a#)eM&SldU#iGOF)qkg~Zh7(S>$t3>D4QqXuhTdZm;)|U8o%muCd({o6 zoN%>i6+eajmxg3wDJk-Vr9K|_htp5`!?E}H!>MQe;q=Fg%zLFgtDfh}<@>tbu-dEfiT}#wCv7Wvf>WsH1P3u2$c=Xpk8(wwniJ3Z(`MNh8HJjt!uuS+9-mvt~Z+gQOrtkOMu-1?Emt`A9|B3&;<*ogbHk@$6O=jl5 zapTc{qW{c>op|X#|E(KNIpJzk`-fisF%$pqyx|JdI^z!~{zref?|<@#Q~$F+y!kKu z{U;{#mA8Ks)jw_`m1SmCk6*qlK1$j@5-l?cnYZMB=M#BSdzicFT&91$X1{-Ezqjr8 zlFXU9xwi+)af}TQkUnX{w{)MfN$+&*asu+tg!7)4Y(8llZn*6d%rD8Ymu{?Y{BN!f zd-OO>w+YhIHoPszbGmUpy>bj0q}SLB1kB%ZaVMYS_HX@K(n^)>gHm3-o`7o=Eu`Zb ziZwUGlS?)q8#L3LkBzX^m&qr??~o7T_xml=y_6YOO|q%aIXbSaWc6Qibnjaq z+_*oIL&hTKEeKgw|B^#R6P8=cQ4fZmv9ziM^VGp-3Wiit%={55U=z|rO8 zfVN*3{jq4)&e8wf?YBQCMS9A^Vx)A59-Gv+IQX|OiT*y19<$N^e20AesrMh!?%-PL zPaC(vOX8xvo9#Xs>6rW#l>X0by06%AfdriXafy&7%h}qau9UI~lu-%hpyt}8LOD-O zxni4Qogyiyr*p=d=rbd&hRKF;>W187LHw=X+HW;|oXqIe7kJ1PbN+QNRjci2e=D1h zxPSB^jPDVfzS#<>V^04`iT|>=W~9USZJWLdg8k<7Pq`J+7bkAZrr)s6N%t?O$2s;7 zONNp~&i>at%>(|}ycSwwu-v>$fXQkq7Nqtex zSKShA+DK$fLDMGbZwBnYzRykHY!mAKb<;NVZrwJ0oJ>pE^yt5uUR?sGuao-uO2Gbm zUvl>!lQg>V{?U&ot@9zF}HsbWpiXgL* zmeyDQ?JxW7x7nbr5Hz2X^j``nf4@!NW9^r2>a;)aU&oEsX~G__WiqyZRM_+kEnl)Mo;eJa-1dA;*Wce4BX4NqRNFZ6U@ja};O?Y(^Uvmb~huAJ_;a;>|%^JGV| z`|6Er*FO+zbH1xTbMjPuP3&~-*;DSnzf==@@$%JV{WCSO8!rUm&(@whS9h+i&I5Ly z3eNS+$-26l*x57n9#%5VrO!TFckXQ6*;5{__u6MKCvRlU)cw8Cd#y8h`jnV<@p7jp zPKw}*?LU3WpHK?qB9$%sm|2IIM=@3&Z}AAeLc++p0Xn2%U3?@Z3Eu##Z#BAdU`;exAWadUhnR_@`2c=obP86#(eDZ zjVqnWOFi9Pv81?;n7W2*S7V*mKl|d9?yE^Be4du>*`E)#@xj6uP40|;lR4551u>|yZFUqx7Rrd@3GI_Up6ZzPfvbVyDne( z;AuzA{^N=7>i+!YOWhq>%nx3Av8$8QcR4Xmv-|dIGT!a|H+nkzy3J>5e<_ANc=AtP zi(TwiZ=i9XIq5Pv@#;Y}{*)W<$vb)dau*lhTVHPew(st|e`8LZG{!BgjGwOV{up5S z_S@q$eJpE(o9g{eDmRgbcj{NN_H7>DZ)?XFP5aaO!MxydzVx;RPDll1oImC2vB#|s z9g?11=ETda;)(X=ySMtT6HoE%Ic+j?LOPdcS|ZEbz2xrSPFdCnUwV|+QvgqTw?A4B zsma}`&c9m^@79Op1r7x~@vJdA5hA7gH)5q+XRLAZ<;l<8zvFd`k*N~N-M?Y`S&7rx z(#G1qGnt?8)xIx(r$1@MI#a4m@7cd?-%NZq&vSC%D3ff)pse{```z@;{0lylg^ZC} zgQ?h?PtN{jCXv}WvxjUNZ(b*jJ7;@ya+Bob=eTQ=;gQRUlTH`&+E4zmF5@2E`9F7G zy2sr8xfvr(c7Lv?^zd$b<48gBk^_azPMp7O;u9z9Oy(#eL+nuA({8!kSmR9xQ=~1V zgW4^}9_I+a7(FVXT{$T{ylrniU;&@mh3w;u(PJ0RBR9Ps(nz0nSa15X=K2>nbwGW) zhdmC9)F)2Xi{H3$38_z<`!i#Y0ersAj_o5DL)R-Brk`{q?0y1l}+dTorybV zoP#u*tNq)?>O4|woTIeNUL|w8x(2x*jznKYZgPJ@hM| zl)E{f3{{r>q;nO!0)Tv1dH$}Ezdh5Ek=wc4t*z``@vnu)B4025t&-m^``zQ;aC26+ z%Od)97bm2BW$#L4_|oyR-#PwIqW|=q%GU`FpoNUzBo2%>U!8B{3&=P_X1&f0Mi5Gdk+f^=Mfg zfL?zHdDkIi-O}0QOY^~@HtPE`QU@W=RYFv&rg2wH-Q@W9&z1A(7(4s{&LGyVE(Sv#yl?eI@hJGfFAoc z(qlGWZXq*|Be8t9+rOU>z1*Y2muUWvdh|*Ax90EFwS9HdeBS2%>HHhFI_IM|>+AL; z<3+RFSZ;R>)Ny@Kd0Z1#aLHDj>T=6qW{caQ$o#ZN z_wprnC^8@S=w5!(qkHXH@#tRuj@7k3y*})&)#X+{pFU*$5!e5i)w%vBtgiMyk*NiF z+{vTsoPcd0=KiI;+?|x!QZe_>v!efGMm1*0>aYi87`8h474#7gZ=2M~ZPdei^)cdW z9om?2;-ODiy`Joxw7Qnd{%LfZwE@+iauoNU78_9gX^+0VgC4bz)`K&C?fg}4_00IS zdIj-Ot5*^aJw|-3<*SH?-avd(7xrX}6A!(K_>|?FiHF`oe3Ru{iHF`se4FLliHF`r zJnZZt9(phF%QpW$;-M#rhn@YzLmwa>cBY7jK0-X~93>w581b-koOtLH#KX=>;-OCw z4?CxchdxU@?3^PW`aJQlGfh171>#}nBJt3dh=-la#6!PBeBY?s|5u5JzD7LCwN5?|i9dIj;Yvyyn| zRm2Zl`>TnEUQ0a6RYyGZdg4*82I8T|iATAbh=<-xJj&HVJoGl=mu-35iHClHc-WaB z9(ot?u(OAF=)J_xSbO@2hu%+om*oeDhdxC7sO5)=hdx3)$~8(n^fBU5u5sd_PY{oC zO%e}%n)qRB|1IL7&l10B`8ndD&l8VwrHO~WKs?H|NIdi<;!&<;;-Rk)KW**5Lp=0V z;^Bwah=;yTe4CwrH;9M6MLgnGt4q2qof@UXLtc<529gOB`X zk&|&b@z7($!_F$=p;r@+cB~;DdM)uK)}A`zq1O|Say1YSJx)Bz)kHk>7UEH^R^p+z z5s!Jlop|UMh=-jC;-PmD4?BB^hu%*->>MB-dWv}1IYd14Vd7!u2=UNIiHDtI#6zDX z9(GO<4}F?=*m;Y1=rhE_&ROE2&k+wh=ZS~DK>Vof&x^!EUm_mmS|%R)ZQ?PGR)~jw zhxk5g&nofI*NKOp-yk0PCh@TIF7eQ}h=-lq#6!pZR`6Fd*8a#(`TZgEGOL5fJRT(; zdO7iE_X^^nR}znQj}Z^OhIrINE%DInh=-rACmwnO@vt*aJoF~w7i@ht6A!(Oc-Yxa zJoF31qa72(L+>IUe!hoz=)J_FTz$ktPZE!E^%D<$hS399=(mYS zJ**H9eT{h7xlTOv4dP+vCh^ek5)V7Kh=;ySJnS^@_WL*JC02)j>$AtVWyC{|5|46~ z6A!(Dc#P9Z;-SZgU$p+Dig@TX#IIVumU!rO#KX>d;-NPX4?E+;LvJD;b~Y0ay_NW3 zTmNmuL%%@0k>>(+OArsei+Gf)hj{3{#G_nwRKk8<@B4?RVE+}b}xJoI7WVdn_( z&_{`fonyp9A16L$?U^7R`V{e)_os=6K0~~*`Ogv$eU5mPYo2)MY2s0?1>&JE5|45% z5fA+~@x!)%t`HA>m3SQ2tq~7>op_XMgLvqh#G_nyiHE*LJj%6AJoLyaN3r) z#OgR-hF(TI{BV?b=;g%24_6Qmy^46$LpAZxYlz2jT`lp@>xhS)^~6JOApU~w&vD|R zw-66ITZxC>Mm*ZFop|UMh%d4JKS4b7F5*$H9^#?*5|48A5f6QUc$6zeJoF*rG4Br( z4}FAq*f~l(^fBUL=Q#1ur-+B0)5Jr+MLg`BAs+fH@vw7_c79&aEXdYpK)dlT`{ zn~6uew-67#op{v41>&J6h=-r=A|84V@vyU(c<6n^&)WJ-5)VB^JnS4I9{MoxXvY!a zp^p*|KR-r1^l{=*t_k9yPZE!EO%V@$hIo`~mU!rM#KX_b6AwL2JnUQ`9{M8juycuc z=qtp-&O5|IUnL%Pt`QG?op{)}K|J(L;$i1q;-PO74?CR)n`YiFVbB=iE^TeYa)5Jqx zARc~xk$C7!#G_ox#6!PLJj%5~JoGiGfmKeI_Z^t;5v&Mo4hZxatY&3pX) z8G4!3;SXVFlz8an#KX=C;-Oa(4?APTL$4wpc2*M)y^eU;Sx-Fl2I667oOtL>#KX>J z;-R+?4?A0lhu%(npS}KYfq3W%;!&U3Lmwd?c8(GceT?`jYyUX$&?kw9pPwQg`ZV!q_gln6pCKOYK1)3GH1Vj11>&JE z5)VJWL_G9m;$i1);-Rk)UuWy{4)M^}iHDsV#6#aC9_@ISc<5Wi!_RLM58ZjmptwHx zH zj}VXJ{88eej}eb@jS~-jf_RLhN#dbT5#NsEJ>sFy5Dz~;OFZ;B;$i1J@zB%6!_Ecb zp)V5OW$j-g9{O$OvEISF)$b6Gc3&kP`Wo?Q_jTf--z6UPuthxdZQ|kQjXr=W8_r*# zN30J220KfLhh9c}pRLa*@z5)Yhn+Fvp;r-)cC02IdJXaL^R>i7uOlAiswW@vw84 zc<3XUK1Dq2oF*RnE#hJ44DrxsiHDtY#6wRL4?n*^JoH84QLZK8 zp)V7Uadew_=qto0alA)7^flt)=humczCk?f+$0|QUE*Qq7V*%xiBDPk&HKn7TD`>T z@P|0gFC!j$lz6mzIq}ddh)26u5)Zwac+^7;@z86Dho7$_9(q0Tu(N@9=yBplZM`)S z551Lm*x5!r^mgLWju(iBo**86zKeM1J;bA2y~IQBBOc{S5)VB^JjyjhJoI7W;b%sO zhdxR?>>MK=`Z)2hbAov2lf>h`zA56NPZK|B`@=2bq0bSIdYC63dYX9D!vgWp7l}tb zED;ZVnRwL0ZQ`M?5)V7qh=;yTJnY;c9{MKnu=6hQ(6@+(o!i7ikG$XYd+={*_&uxR z_z-#-@hDf6cc<9x{_u1p{8seeX5)V7;h=*QJJnU>B9(tU3 z*x5up^cLb_XDjj0+lYsq?ZiXBKs@YB5D&eJ_}jMrdx(eLM?B`iB=OJ(h~KpBl_DPc z5b-G2F!9hwh)20biHAN$JjyjrJoHK8hpqin#6zDZ9^>>D@z7_8$2grO9(tO1)WZVt z&=-lvyuU;|^kw2<=WXJluModx`_CQXp|2AUJ2!}jzDYdV@hrvvNj}eb@jS~-jf_RLhN#dbT5x;K7`!w;;XNZTNpCump9PzMoo_OeK;$i0k z@z58EudwyGL_GA{#A9AtAs+f2;?eG_#6w>r9__wPJoLN7qaL=1hrUfb{5)O^2LA>< zVqXY@<3re4Vs-G)%ZRVG^%*4|dL{9&Ge$i0D&oa><-a z4-pSLhlz(iLOkpoB_8?&@vw7}c<58a!_H~qq2D4NcFqtFeU^CGIY&J7H1Tmez7~jw zzDPXEwM0DhW#Tc8ZW9lEh4>n4&mH2SuMrPFzfL^#4dP+vCh^ek5)V7Kh=;ySe8Spq z>?<(g-=LRR9sU7+zKnS2QR30=<-|j;ARg^rNj&sw;!zJZ#6zzo9)7-#cs2yKh#6#aE9)7^s*Ji`N zL62A+{te@(gm~y>#P?f!qQpb5ARd0cl6dGb;$deM@zAS@hn+RVL$4)%+S*@7JoE8T1dx(dfy~IQBBYw`-XOej6 zL&T#Uhlz(iLOj}Wlz8Z4#KX^z6AyiYc$90Bc<58aqg>O(L!Tuc<(eZN`aJRQGilxhS5Pdvs^1M$$~ z#IM=$-b6g~7UJROTZxC>Mm+3nCm#9*;$dfkc<5ckSJ?XOAs%`k@$mCW;-U8wk9Hp* z9(sy+wEGb8&_{_!J&X|#eVlms`3d5oPZAG1r-+9>O?fm8#Iq}ddh=-k(#6yn}4?C-fhh9xQ?5rUkdOh*5vw?W%apGZT z6Y z<1|G)^daI~tewNeLmwp`^T-(S(8q~~ofE`EpClf3P7x1%n)r73L*k*&5|4Rgj(F(v z#G_xOiHE*GJo?ol@z8G*k9t@k9{L^PF^{Yg4}Fbz*tt$T^bO*>Y<+GL4}F_>*lFA! zP|5f==n<>Kzo8vVh=*QAJm!%o@zBeON4YA9hh9lM$`vCXdJXX?S1s|->xhRRt0x|M z1M#pkPCWD`;$deq@zC3dhn?-jL%%>g>`V|3y^DC**+V?^UgBYAAMwx!h=-ji;-L=_ z4?Bm6hdx3)>>MQ?`WW%BbDVhSlfC~Lp=OUFY(a(h=-j?;-U8w4?72l zhdxX^>>ME;`Y7?RbBuWC8iq zjs081>g?Z2h=-kJ#6yn~U-Db-^}2H6p;r6A!(Kd24?&^Hy&q9)7-!cxBDdJ)05b@B5iEp>{HbOk~apGa;1o6-(iAOt55f6Quc=-8S#6zDU9_5-P9{L>d zDAzpk&=-kExt55BzDzv)%x&VKuMiJA?+_1tm3Y{>Mm+RQ;$i1q;-PO54?DMshi=Zf z;~M@Ac1Em@@eaL&c-UD+JoF0UVP_@r&|}2I&MM-eR}&9AYlw$lOFZnXBOZDK@qKoD z#fgXBL_EsXOg!`!;xUd|iHF`s`~_=IJMqvH#KX^b5f8nGc-YxXJoG-|VP}$f=>5bG zTl)uyhdx9+{QNNS&_{?zyN?nNeT;at`#AB?r-(;AOcM|N7V+@&GsHumB_4Lp5f6Qy z_%U0bY2u+T5RY}SMdG0^5udjAGb|Gi{SNV{hgIUCuMv-avQ9ko4dOA6Y!VOsF7c?h zE#jeX6OVc`A8`F7{2}x*tHXbyTv6homlF>^RzW=UO5$N>jCkl(#KX>N;-S|O4?F9L zhu%Ou?2Ho+y@`0(*-Sk27UE%NEAh}T5Dz;O#6#~Q9(MK+551Ro*x5%s^d#}Hv!8hA z1H@z8rHF?6ogyCkH1XrsU)>@e`YiGA z^K-;QpC=yek|rMd0`X{M_wZucOBOZRB zo_OdD#KX=w@z9%yhn>yDLvJG>MT@`Uvr`bCh`KW5mPGapIv*5Dz~*Nj&r^;?wr}{50{J$ zn&JE5U3Lmwj^c8(JdeS&z{IY~V9DdJ)0H1W`H5f3|Oh=)E;JnT#p z4}F1n*ttkN^d;h9=Q8ooZxatYSBQtcM!flLcRh5Sc<39%qg(6@=- zu;bl)(0_agy~OG`K7^kyBOZE`c-UD^JoF0UVP_@r&|}0`+WM>_9(oP&@bk6AL$4zq z?Osnj^akS5?s4LwHxrL~XdxbYEAjC2ZNx)wCmwcQARc;x_-b39UBp9A5)V83iHANw zJlZiuJoF*r;pc~mhdx3)$~8(n^fBU5u5sd_PZ5uDO%o6O7V+>iGsHumB_4Lp5f6Qy zc-WaH9{LjTuydJs=(mZ7oh!scze7CiTqPd*8u74mop|VXiHDtA#6#aE9(J1l!tdXp zN30J220KfLhh9cJ?2Hl*y@L3-9bc8iLyr-Ua#axzy_$H8qZ;C&*Aic2?WrRkdIRzB z^Ks&#HxUmzn~8_sLOkqjB_4Vk@fWQ9?ZiV*5D!1!MLhH#;?eHC#6#~R9_^ka9(sy+ z)WZ<*(1(eKpC2I}`Y7?RbBuWCYXNX5T&JquOj(GU_dE%j` ziAT8>h=;yNJj%61JoFXfQLa10LtiBxerAn$=xqY*4a7r_6AwF^ zh=<-n{IY$2W-IZ~+lWWG+KGpLfq0Cg1o6#_1IC(5HzXvG&{|9{Mctm`CP_hdxg{>`W66 zeSvt`xkxMm+4SA|84*@o2{y;-S|Pk9nkyc;-OCw4?CxchklEA*f~Qy^jYFz=N$3S7l>c|O?Q1^ zk$C7!#G_ox#6!PLJjT%q@zC!Ozi7w%D)G?QiHD!xARhWA@v!qQ@zA%3hn?HRLpSIB z{_wW7KVo(GL+EA1!_P;Fhh9!R+P#8!=#|8y-DAWxqZnKs@Y> z6A!(K_*Gk<&BQ}*BOZ3P6A%3Y@o2{c@zA@7hoA2u9(phFC|4ix(38ZYT>ZpDA0i&* z8YUk42=VYUqr^iWBOZ2+6AyiYc-T2fJoH<{!_FDvq0bTzJLiapK2JRCOcM`%fq2-t zNIdl0#KX=N;-TLm9(Jx04}Fbz*tt$T^bO)+=O*#cw}{7kinfV|Za(DqZzxy9>hN#S zONhrfDkC0xl=w9q?-38Zl6d&}81c}nh=-lk#6zzk9(L9e5511~O>2KW@zCSM!_PMn z551Xqw0jHj&|8T|ySEV!JwZI`p^JFvJ;cM$_Yx1ik9gRbBp!M{@!PgO2Z)D0Og!uy zAs+fD@o2{};-QZd4?jOaJoHK8QLZWCp-&T!a@`^x`W*2n*F5pi)5OEiED#TUk$Bj- zL_G9m;$i1);-Rk+4?EY0hrUic?A#z8`X=$P^Dgnww}^+G+r&dJX$tm-Wmac@7$qKd zmJ<)Xf_T_jNj&rz@vyUsc<9x{<2q6e@z86D$9<}G#6xc&K003HR9KvN=uO0<-J6Mr z-a#KX=g@zBeOhn*F~L$4wp_E!@Ry@q($SxY?hI^tnxJ@L>Rh=-kV;-R+?4?A0l zhu%g!>})3<`UT=)XM%X>UBtuA9^#?*5g)hXD@i=`e&SKC0pg*jh{re@A|CoM@paao z5#phb5f48^JL%&7*1#ABd@zCdpho7G(9(tO1wEF__&=-kE zyDt$BeT8_`!yV$GuM!VGzeYUtb>d;?2Jz50iSM!Xd6#(T=EK3qdl9R%KP(|0?N~-U z^eFN0^X0@tuOJ@fsw5tIjCho*ig@U?#G_s6h=*QJJp4=p@zCSM!_Fq+p*Ir`J6ni{ z-cCI1yg)qk1o5!5i+Jcg#KX>B;-U8u4?B~@Lr)P8JBNseK1@9993dY1DDkj!jCknd z#KX=B;-OCwZ|wQ|H1W`H5sz}s5D$Hpc#NYt;-SwI-)qNvnt14o#KX@o5f6Qtc-VQH zc<3v{!_GUzLtiDn-`c-MJoF9X;paDrhkloMwEGtE(6@<4yPN0z$A{3%td8SD)I*eb z=;g%2&sPu+y^?s?86zHg74bv1KC6j`UPnCatS26N1Mz6bIPuV%h=-qVCLVeV@hDd- z@zC3dN4eUGhu%dz%GEUK1Dq2oF*RnEb*{&j(F(v#KX=s@z58DhnYl%m@*AWjrPCV+NiFoME#KX_G5D&eTc-YxSJoI+r zr)+&*ARc;xcw86lA|84V@ssv<<$8&S-cLO0VSsq(DdN#jhKPqgOg!e15#phb5|4Tt zBOdxV@u;^6;-OCxk9wOT9{M!#F?+wtE#jfi6OVdJ6AyiXc=*Xh;-N1Qk9t@p9{O$K zQ4cG`L%%~j>S2|5=xfBc+WJ{19{OG4Q4d?hL*FJI^FT_0~%~^giO@$CAWD?(_;i^M};A|CCyOg!}4#NW2|uMiJ?m3SQItPu}=op`j%2Jz50iO0CROFZ-~ z;?XYK#6vItsOwkHhK#9n|8)MXuzEe|l~#X3^b;Pw%IfgL=+D(w$8}5SHN>Mo*AfrC zj(F5hJ@L>Rh;OswC{8@|R^nl28}ZQFiHDsRh=-ma9(HyS550%@h@EG9iHF`#JnS4G z9(sy+*f~Tz^kL#*=Lqr8M~Pps_Ky({eUfcD4}@y`A_zYySn}q4y9EJ9~+T-bXy_OcD>hpLp0gKs@vm z@uSxMA>yHr5)V7ah=)E-JnWnx9{MEluycxd=+nfnS^IAh4}Fe!*f~!;^fd9XbAfp1 zi^RjuCE}qk6JKrL+kBgN=&QuT&NbqpuM-bDH;9M6Nj&VlOFZ-~;*-|?ZQ`MqwD|oS z>@2f7{2TNr@vyU;c<2?x!_G?Lp~r~F?^9P1551On*jYzB^m^i9X9MxjM274#C~VP_ff z(4)lT{nh2fLyr*;JFAF?UQImgtRWtHE%C6kj(F(x#N+*_4a7rlCLVUS5D&eTc-YxS zJoI+rVdn+np(lvP`~A9zhu%j#>`W35y`Om4IY2!06!EZgh8#6zDUKJ$JR;-ROBhn)+=Lti8wb}kVQeVKUJd7F6XE5r}m z`oBXw^mXE4=LYf6H;IRxcZr9-MLg`>CLX%^xZl6w`O}Ei;oqP~iHDu##6zzj9(Gm| z4?RXa?5rXldNuKQeyWCe==H?I&IaP4$BBoXO~gZQCLVUS5D&eTc-)`gMm+Qc@vyUt zc<4RE!_Hpfq4yCFJCnpi?duydYx=xO3%=K}H27l|LT_Ae0+eT8_~d53uDtHi_3HR7SK6AwE# zh=;yOJg(2*B_8?~@wh&}O+0k-^M3!9u-CUER)>Fs9`o>zO41THxvBE-UcK7t@09EO zRhIRBg+vdFfPArb5V1N~`?GHbdaaG@lC~ zzJ9g4_eRIX%U3(Ef6=`EdUtQ<`(LRn&_3NEqG#!^-?CNakeDUg~o}Y8j=lV@m zzvYtM{YldyNV2=*^3|loclLIC?uC({P-syq3~&dWEtyFPsV+LaG|DtW!N^Lp}z(cO1+ zUGD7dzLf0fym6!Zda~oji- zlf7a|$Cc9^SFUw;v~^#-aqaqNuXJCzcJfsH>DsfW^mDH6^x2d3b^7N?AM$MNxpQ^r z>gxRJsroY~>*~&)sjrh%&z?IafBAs3&(@tgTSs~2LuCRzPngQ zM_2ONb+@OX_q=$u_gZJygXZjXNGGG@wT>>QtZp4i{kTPUgHi$Bpj%tsI{fgp>si3< zL?7$yd+zF`YhB%KmwUUrq}a*NU+H+^h0FQJl&R`6@4u0}p6t9R-(S4qd|$uT)tT&c z&1rG#vqfs4$r10PN?}dMjqYSew+>oq&}&yZ`gEjr)QeMacKH5FHh{9<4CB_{)$(Z$Xi-5p&wdL`oW)z5xf z+)sT+@=9mN3)ik+=}czw(ZWhii*)|)w>{T!s^j9de(@()G~06oYxM!h#=@#Jc~-Y#3jCcE7wogzZDHGw)lIO#f%M{`+mi Ls2Pt3r2qc`6PJDO literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf3232ddee2a0b12f6c32292a49d23f44.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf3232ddee2a0b12f6c32292a49d23f44.o new file mode 100644 index 0000000000000000000000000000000000000000..92b3f2cb7bdaf3d236e76f4c4d890eb6c1c50c32 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+mFh-3m%)>nNK(?8QXId|WWI#e7UYi-wk{Tq?qfbjR zEkjO($TGIUU}APd&c<0X@px$&GR{tB9L?rTNPs=K<8>f0 z?*0CHbl-c`Ai$p3uoQIPTmS!8)&Hvc|Eg}?{&M;9rMjjuRW$Yri`5jAvCO{My@%g= zSSl#pthaX1R{yhXx9xw^3vHjB$=JbRTXz3Dc%8Ln&-*uA&*AlJ+@tAaMN8MKJAM%Q z?F4q^!*zP-gKAsos*N4|zc$u?q>~*4?V;WbR`k7W&%lSJ?5$tw?BLAQ-r6&^sr|#Y z$<)`O-gH*)CCWP=C*z(e1871@Kf*oB>c4V&2Y={i{bv|?}4r;6xzRkXzTHPLr*O@ z7OpyVJlxWLD%`*C&G6`h7sKbbT?EbIP^EMK(D?;>hMro9e)fJRJhu(wpL{c1dG2C( z>w7fbABRE^XWg-#w1pU11^ zy*I=2PMio=I^PSQUvLgIOG0zo_6+qy7nSFZq3`42(D9SuD(9(iXu-ws(<_PRM?#_F z`-bM7*fSIY&*vAs7=CKOo8Sw2fb9KvCC~jrs1mxImk1i@EhX{AaOmAP!&{f02+uut zA$ep;df`NP6!7G^e-avfY|l{To)^Q(=RgbIleO&2$-mxPJNMj6 zf#hA;-%n2LtxbM4J4kl)*jSMRHq!Lo+6MU46wPzxjBQFk;Wp{4C~AM;g#D+$zl0BL z(pb^!@BvLD+?4)d-SHT;JHlAetAOS3HlnW|!cRV^E)9LSt^|JPxpH;KYe$X#8_yp< zuxay@Pm=#B-y4MQy#n97>wCAi@AY2C_l}I?dq*t3M?R_iZ_wg@%3rVNgI9O_Fcc}B z`qs$(`oZ)6QgX#@XK5FHS#o9crSgGat!4wMm>*|mvnwM%)UUh_zaGwJSI(49IXK*? z9X#_l`qh!WY=Gtw(*_@127Kf>HW2w&HgNG3{mMI~Y~buY?8?6tvVn7H?8^Bhb|nS0 z#h7a&yAuXR{)G*U+^-$H@W1pc7v5n9NB%k?H1chB@IoWj)Jbe$^hZ+xOAL+N&kp|j ze$f7pq|ljlyB?*v2A&UI-SMhvE{XMB$a4YwlPn|PmBxIRWP*$rA)^yAk!;CR*p+{x zxeED=a1^Zlp-;$kv1sjmf&%@?t3faIg>_>J` zKK=e`HV`gl156)$lx(W(f9cz>6@2+!`0;zN*}ubnF@InOFCBng4R&mvA@o6h`f+t} z=xS2QWwPI0eVFxe7|sJf<}Z4?*?P^pb$_oh7rWGmPEd5FX%ygtvQDsNcsbDdHe5Y; zCE=CMTr}*MvdtH0_qBGi8enn>t7GQrskrJIvrV*Vx{WdW1T1;7*qqFc+b@f}wpbx_ z_76fPNw%p$+?@GO;;L_Bi5~U{O8`gHh;m_Pb7xne%iDlEb?s>rb$+eyY;JD~vW4v( zUA~3wk1y=n*3!PYBUW$hmi5OMMsFHB+Wg)wwy?_=>{=*9t8eV?YV~$@)_Yr9w|Kjn zeS!K$Z)aD%#CG}GJ39jYu0Ts+!Im~(o42)POMCu99r^YDu^0?cpI=l^|M0{0`FVMH zg$4EaSFkwm;rxdlsn4&^gPcOHJ3x%iW^cgPbms^X@OAnEk3%9@Dd20pBfJfAc<%^7 zntTo2Tk1Clylp;V9V@`Q&;-FWYow(i;Pp52N#_rAbaiy~_|cqj9qpZ6-uAA%JHUM# zz5X^|pz)3{Gfcx}vD%UY<hW}ZW{7A%ic}P7Fov%xbZaP_TR^-MS3+hSPMHgZ|?MU<+1wu z_Q&g^m=)zK>IJ4g4_mbsZzqr5Q9QLo{&Gj#KbAhyBw`?s-I6Tn`W`D-v6UV^U zW3z7KJ9abbNe;(Jp{2Y4c!n$1Pj8MOXaFlSaqE%vAme2R;O--7QMd%NZ%BGIw4(nm z;5M9JSW5&YR|4st3i#eh*R<4Ky}fgoBeJV^j)UJaM)3L8PGnv2Z2txTqOJWj9fD95 zuS0lIedXH9vdUt2Dfa(3M-hb%Nr?0zyfzca-xY)}mUcIJ-E_Lp((3JM>1cO%b#%0L z&Ua&G`@Ee#cYfi*#q-^3>9VkB2`lMtX>D?M_H=gn+T3m3on7uG-{zKfpLDF3 z9E?06kM7gOJ#LvjBX)ly$2qp-f>XYg=bZfge!t~`hW~#So_PMFtres$t_Bj|IMsX#>RvEah0&mk( znIGM|;D|MN*@$Ry*|OF!dy0dRclX+G|B!@l!_ih7UR!`bp>gQ?vAkZD z*N<>5BLoWh#n{=#SlS5-OAky=m`wc-9Y|tliJo!# zB%%jCH)x+sXPu&dqUY00(4&9o!$#uxYd+xo=R!UMJ@`qJ_5wfXkLbDnGP*S42X&p~ z^VfuJ#BWk?3arTOo1pN5KFkR33OJ__zo};;No**M6@3-`qt4}hgZJl5L(SiS{?PxE zPq7Hu!|s4RhpX9rJBXGF!p%YWTqpFSPh4A9SI6ajBOCHg)ybD44(JE{ z=LV-vC4Za9^)Kq|^h>(ZPes-@qkDS_-8oW+Na z41)jRGKnODCHZoHv{B^EiGH+tb75k!lm~ z!=f`%40=20v9=NYQ76aSQ5X19;N8HxO?Wrz0$&dNbl~#@enhmNj=I4A1@M``djx(? z8t*?7b%9UTcJ;V`$0ADOHw(NAb%9UNg*=-iK3np?N#Zr&iGRPqH!OmGJ@5d@$NiuD zD`>NEUsC=SoI97qiQ#9@j@Dn}^M%`k&p)&$ec91J=jUykUXgW}(YhLm?B@2*gngq0 z^u?-9_J)68KIiiJO7r>WX{_Yynjk0t1-|0zKb6fzeQbYbdiZND*^~P##v}jQL?m)P zWCUl;qWH)C@n_WkGE@J3J7&cEX?Q=@w*(vc8($PoErNexy^s1A`Ol058gB;op95si z8BzZlc4|d|=?PJP#dQ~vD7$AGUP5P#L6sd0-&#wG)xD@*75_mbE$Le3AWqas< z7O7kL2kD=GB1L$NSMl{nO#Suk$cX8?q1~cyn%@R}^ZA;=^_?O02mjBA>boHu>xIjr zZlFR{6KrDZ}guLgtj0+WUx=HzcfF%0j0jO%<@uy(e+@sm{nxWG08Wc zzWR0`A2@?1w0`CM3FNmK{Eg&ySp3b-`c3mk=kZb}JigrURmI=SphNt8N!4l8 zLDakHMi$`r_b{DyJ&DDWCPIETR(?qvTc2cpfcy>fX94k}3hKwU!+uWyK=}athvM(c zFkBRG&j?OH-H4~@Ys4p@Zp0@d-7?~n zP>lt6)>@FEk#`W)NhV`2Jhkq08Pv9R<(d49}`~&mz460;b*dO_RCb^@Chh+b0 zG5f>+>~+r|0&hIRSkZ1*hw@+%nSN>Hkja*YreoDZ=xc;TiC+OL=Sv!Fxe6HC*-&30e6-gy+??i z{As6yua6injp*S|nMQxh4Q9EG?TdE0-8A1gJ@-eO1J)7#i1Dd?)E_@gW6y>;l10u$ z(ulvJKG2KyNtEv(pSBI|0k3?0OLHMVb4UAw{ON%i`ZUl-T$s-z)v4)$Z2exUPsIen z_!uuI3w6HVM1QbvCb^5B<^$v{3T9?T4(r{3O(4c*x8BmFolYQr!g(gnn$8&$;8|<9u?7 z_;SA59M4`r{mR*O-PKkVvZXn;cJBiMf(=)qDeicb4;?2pAEChZU3;Qmh9 ziTF$8dX~Ick_-OQ19~c*m_&NX{s*+B(R@MRe-h-*!1(y^uRidj9lQS{-U+WKFwPGF zUdMdd>A+C|iVf?ye;gsZzxM^1ogh5_!1;Ye z_siqPt0Sg(WjJ4l|J+KvitfL$zM1#ii}3lvc>C?0TpYDN@%T(fYV>(g4#ksu_!)ddy_w=uBHy3$c!vBaeW#u0PuLHnvDcBm?o3Fec%C#l%|ZE4QP57| zI)UfASpUR%(UEm5UOYp*NdFJsf1BbL@+m#QKYfUP{y_L>JOlsfh-b7Pie!rX?&jl} zX}@8PUl`xC-+;bN@r>im`w!E819($>8{zr)CyZy8O#WhxXZSoRW`FRTj`O6!{Zc=$ zANDg->GK%AKA*+sjXNgC_U8}q{W(8B<@xp-i$Fisupd8w&l~Jh`F@=H$Dh-FJa*oL z@vwi2onK-7AG;sFcl!0?+cfU)_r{NJh#%AL<@+zSA0HdvOy@tj!F#7!;#=(e$GX48 z{Jnww_B3OBD+*5189u-g`$L)^nUf~NLSp}&js17lB>H^Lu>YP;`$a>%!-ucN{kPLO z&i)$*DpYPI-c{ei{yQD3B^VU@RkU}Z9^H>3Uz=n+ZvsBUcpgRmpGo7 z^8GjY6G}ARO`Xczs8harZV~k9(hd5A{nK@eK27_P@$=U|;C#QBV9sBW-&QUFRX>0`@Ht}M}XK6{0`|12H7NK~^-?0AR^J0G9E6y_v=ix^{bOYz%M{ynw%HG8H zajepuX4zk->DceHuYrL3Gxnd?zjbs)OGZmi;#0uSMl^ zGLny7l)MbAgE`~;sL znDUd)pbn|-mb)6btAV>3xT}G?8u-*TK!@iDwS)MU1MB_y)!%>kqsU*p)L--4jeq<1 zKdYMXS4|(C|Le$sA~vW0={{GTyMJEJ(f)qM2JCgujlTb{^^vb}RLZJ^zwFQH>+ert zr+@s)?|&D0+s?}CR?R%Buc}K}RfpXOLi$hUP+ZTaFOF)O{L&(?qZ)f~4)B)=paj(> z*i$a^2wltrb*!%8Z){?vG{CfHFgNi9jFS8=Mxu2pQDDtG}kS|~V;aha6>Qaomi8tVB$;)#!O z`BgANebGy_qx^4*>0a3&9(`0EKVz#jEm?=Aqc9F3 zdr}cS8+4lFvG@b0;>guFh|IX%40_j-gx;fa(@1gfQMq05@lm8{7w*g0tC+@@POkE?TCa26dEofvH|bX{E0 zo)T}y3O$vejmE5_)c|VH;}|?~LpoTc>nk3>+;eje>Z8qBpXRXvtBR2y0V{6MQ}i51 z;_vuMp^fn_8e6R>))gbh)3r3H`i;K=#oF)Itung|uf3Wi zG_}TUZW)W+b z&9Zj!ZwnjNE;IexmiCVzYc}@hvD9lUb!4m8Ub~E|z3{#9;@kD20XdHJlJ7ISJ_A)A zP@&b+sAn7Mgr_=JrZyoJ86NEbY}!n)u}af(T!_t{U8G|Tff*}3-eS7~%Hd?MZq*Cw z)ay*tsVmYojhdtCv3-V3r%{Si*Vv0)kEh(&Xq1bLr<-`W3UPWfxqut&8g#59lmQYe zq!H5bm@GC8Gax6C460Xnv?u5Jh&_B#vjkmhr?EJs=Xz*!_Zsj&!pSRAT zwf8@C-854Z$DS5i`H4qw*$xgL#q8b881>YweM{F@mSL~W z_}RC18Qw>S;n;1fGVTccvl5T^42zCrE$8UM$4&S_@%=cl{g3rO_GFw(VCY}hc>leM zH~QY0v44qIX#Qtzt8}f>h4TfPPokW*A6ej7agE zu2?_HU(|ED6fK(bBA(Ngsld@|^f_G;qUfUpw|GM{ogdR#+IwWZG5UeB0K`@XQKwBn zy!Jnq?Y?ADt+hS=7}9daCHh@Pf24`xkPT?Pqp=h_N<(B1YoFAvMo#!SDw5MCuUfJ;g zIwqKhkcM)`GR4&Rs-3d`<7j-n_KJ?$6P>j^&2v(L#Z4I_$3*GP5m#zYG`*r4eaD{+ z@IFhU5P9>6ywaeu@IQ1l)Cv4ucpdRX{hNwx?=b-_P6iyUFT{U|g$|GK=jEq{j^o+S z568sc*OC3C$BTAn{#aZW-zTnFxRCrNBGmsBKpG##gqplT0I6eRk+f0J8noXXXZ)|o z_PLUh#;0OYAM?rf9?SSgWqakn)Sg`0+I}%6@6&O{|HeiTqqzAyW7hGNA621!AKH5? z;}6RA%Ky~(s-1%03qZW~Z^vm*Q^MMwek=-6#}6}`^nXdVm+VI`R{gI=`~MtAe%1bx zYza5z_P35tK0!L&)F25hmrOaJx~ldBzSVX-ip%+F87?|s)x0&1N4%dTnQMd9f&^D@qDlyBiBjS~nD~uG?DNy2iV%se8lv+Wf}0 zb#>c{R&{ubOIMVxEop9STiUg8eQj%_fAyNy%I!-V3z`92T}Sw`)+MVptgmWrShqHB zegiib&pkUuUubH+q$uBX@0{p(YB;@No&LU zl04r=-tY4D1xvdd+Lq*j2Y*AsLv>qAR&H%tU)#B1{mLhnx2?=?s;KL3TGqN9W3+>h zinSGMyBiCZ^w4-s>#90RDz}$46t2x{t=Li(9k*i3N^kL!j`E5vYa~s@hW52>>kHRz zy&<}S{PF0PZfpxSZ!D~aKH4fs=FJ;PcljM<1zT2mOIGG@Xxmc0wYaut!@8nC+%YS+ zuipT^8!DDQ(b!Ye0(o1Zt0x*ti}HNyOZ*LOKGK`;m1<&D=JI9t(`vkURsCWlks`m@`8B~ zYDdm({rKdZKXUK*(w87d{=$52BDEN6Pt-DP=L~InYBnC^g5QtiSn-x?xSx-E;~w)# zT5zeXNxjeWoPNmuUlRYT<7=tkbUrV7Ci*5$Al`P4Lr-HJGSI)K?w2`_O!%h$yj|9b z?gt8o>q&KQp_>l)S%t&3Ox>3`UC-&?v=g5H;je^Kzq}amxrSgg$ATyNG6O#Pt4Pb8 zFHfKWo);@E(Y4+Ho$B6teF1Il?0t14VM5CX$C+S44yEiEidfLZR}TcRNnBRj3C<%A z2dMZ!m3Ys4Ucy{=^^Q#5))XyaeS8WR;Y#q!5^k4pCEq0Ur+kM{lO=wN1)eA2(*Rd(3G_t?zZY=hmT{*6 zZcG)AN;r*+)5K^wChurmBfLz?N&PF{h;+GxTl-lr;WI4#D1A{s##5qRL;v)g6IBiy z;70mm*MMI(z>V_eO1)e8oFm~@c?YF|tn$7t;nseBD&bane_`my$p1$MxKZ9rIe%ze zV~otZ2K>PPWH0&FtCyQB+{o@{B7R$ku zylzg-*cS|Nqg>4ruJoYjTP55o?+X%cmA6cW95rs99Cx_^ZX7o#;c8q(zf;1k<9<)V zt>aSOLWOb{no>%46xXOIxQEvmYnE39S8?O81+L=5parhtz=#E|mR@!Ksrpyz{T%Q| zMZtTe{gzqaYQ6Vc;A*{RBJJ+1Z)j=v26`Cw2ueXRC%6?ftHCmXuEd;}HrXlQ~qdCPo{w>0`p zaG^b4v;4Wc@mbuA@#yC(>f7n(@s=-NyP~Yc+v;oVTG7$e-HPFFaQDY;(Awc`s_*Kk zZ{myzfxw>NQ&fd4MhTa81Qfx|j>U4DZop>Vz?RPXPQR~l?C**OI-0y)Ubc|G3x^a| zS`JCdI7*N#Z0zjHf9Qs_aeM5AZ``(Ve=qe`_V1=PpLjDbW@D%N<<{HucU{Hu`ib9n zBZ$aU%hcCpnV@FL{6it-GOenN_W1;pX+ae3k$0zM98(AtuT*_DZYbsQD1f~Bw~L#@ zG9RPQ)+pY|a2(CS{tiGI-nNKsK~-@8|(SG9V!kuclXwqyY)^m^G5j z$dDBwavftZn3^`uYMe_NHdk888!zHGm|`_I*DK9hAu=TrV2UTc4kX5N zzkfew&N&(oU^h0D4KwHL|Nr;i|7-96-+M-9zPNP7GF{V{DjIu*#cB%5SY~hR-otM_ zEESY))>AWJtNU5D+xCCxg|^SmVC>M4ExYd>yw2LP=l&b6=kWS9?$LCjqNVG#o!<}r zb{xCKO5^i(!maa_Ha)IEBbD>r~iXe_SP?Tc4$UwPt6(I zMC$8sPdY35zq+S?80~b;_EsIn9y$5UVCA{}gWE4XGq~c^GlSpIJy${dEMAA^ocLC_ z@A$FsW6p1fkL^Dm9+`h4+&BMZc;Bj1;fwoDg-7;X3O_yn-S9)L=fcNMz8g-N`ABGF z`~JbFpLu35b^gPlIW6A~pPzRuy!X;i!cRNj3^yOY817qf33NrF(18Pk+m7!a+&lkR zxboETaC6(KaNqtn!y^w}44>bA5j2ZJ70v^L=jT5;xOWx$+4t@6oK}o~^38C?xr^a# zmuS2{4u$4pyj2I#&++iw)^CMJfUi7vGCb$RIq**X{84CR-+{rW=RY$TI(a;N9c<=l-!58!Z+57NHp7Xg-1$24-A<#f?DT&_-hu(cNoU-Lac+R;C z;Ojzo+{{NqBZOZDUiX~{Cnx?I?fwMq_6=45wm0!ac;vxz;bblQ%8wJjTr+avr9krT z?C&Kf_S7W5njIuNdTgx70UK$0Pfb01YLe!;dd4=Xk8qoHRur{AaKiqR;9tTAHfyZt zb@+g$5pGi7knVVl+8tr6=vBaScpK5z58x;7SCxi7SYHCa^IW~Q^R=T!|BdI5@7uJw z$|uSHl?RYPC7KypdC8% zclx#AeXO765z_`9T@HNYIo2QfSJr>=75(ZvrL6z#eeCML6|(+wY3%CxBz83gw8fZf z!+R3?hyR834?mzCy70gBs~6s3hlc+)AvF9=cIZL_*3=2Cf8>Xg0ZR-GKfn(C`T@}X zkEGC<^}8RXxdxsOUEBGpX)cNNUC46*{F5xh;FZRFmt=yB7a^k)GLdY_li1ZC(_Dpo zM!8rryXJvB=l^bL{|Mx(kaCTH=MnHc0-i}mlJ5fe9EbkBkjeS?TIgNKG4e`6XasV+ z3ptGPlMExTB!)&H4|LIgegZpm5jq)pDJjIXp15*3nCI#ojGwHrep)v^s`^2QV821k z8Z-8RC(>0@BCcrvQPpUD8T(^5OP}ir(fC9^L-X|WF;e1>aV>57m5-`4?qjY@&()N9 zo>1zd;*j%mB_Xm6*@Wyw{k;Rdo`wEgh@XLEPycrG{|#JM<5h|K9$fe1dK|C+jr&u$ z{yU8id%g2qLdXWYD*v;MXII~W{#`TmkPG&?0KX+W{R%c9o1cfx&lIx$A=r=XpnUqh zHLO2e%KDi;@F>|-+5hr4VJrCZyYS;nu-U)EeldSwhb|w4T@7|@nlAJ~e)>^Wap+o7 z$rZBSJ$;z<@jRRde#~F=cB}R3cWeJqV=i{N0iB@eOw%a92W6dL%kgrc^X<5L@Jhlf zow;b(QDwU?(B^CDVAa6n5?05|(^GNPHD()c({vkS_HkJ9WU)D!9k*W;d2O*m=)w&yTaQbhxe`!q|sO3 zwY6?bz}xB**0BP-3rrA9vj$ok0$zU;pLG5}duMxRw;#<3*WT9A>22%Gy9?a6!Rv4J z1sd)OGs84o7OO2eP%fR^fJVSwU_&L2*=ITQu9?Ehwp^g=8CV7t<65lRb%;DdW+FGf z+)G35r6G4(S{TRm@1-H1vFzQnY>{=WfSXT4?)-g>TBO%tgSDW2%a#sbXCAAoYkRyd zidngIWu3s(w8Zr#$F zH5FB>>XvR;yL46A(lR!`%3bK5?+r9GxjR~M9`K0QjiHHGI$y!(9~NG>vnK(!FLW^S zggm-W7x%bj_Kev5%^c_0no^m4iyd}B+ehr8@B3(U1`ou^(B*VymvuW?`f_o!Zllo+ zb)C6Qbvmmu;;B!n+t96TYJ}-4aLiO^)9iCnxlPHgRP3`cK8)f@hOIJiTM6E#r7}Od zcfk>>@v^%>1B2l|f7z!CFSBK>W%d*YBk%69;r?L>-;SfLR=l-)NHWk+8q4Yy`r?x$$H|WENR03UxD-&-JJ|v{q*Dd-sR5Qs zgZry5+t|;uS?@l3gk1)Jm~M2k-Fp-5eB6!MsqC1W{jgiez@7pg{q+VhktB%5U)rs+ z&BTk|V-x(`hYHF}yv{n=p@7|v7A!JwAdL+zWVu1xeV(^hvWR{liM@W1%?c*ielz8b z?YuseZ7&MyHo`MTc*qx=7|BL0VKTIh@n!{+^eFv_G?pHim@tw0A3T`E&JsQ2^hrby zd~VP_kzur{bh7HB^$>*;Q z+lb$!;3Qa)+c!bs1$~$i-W708B7RfPM3UHG8Y}v9^p851_YK~kGYvI=J^DlcPwiz9 zvWMLPdk$5x`*#v8707onjeV(me1w~W@VQRtM<2hgwzihb`$jh8ovf2DMI6u%`p*qc zo=pBWp6g%K+3A;brJwSwuSxU5p6DO;aB?Rndw`#!0Gl|8k1y)II};-8eE`S@;C~qb zTZB9^QYi3CyES&17|?`&EDz{uRAt&noF7*}_`2;^&5rgOjXV zjj)TdUVllO8JstV#+yg{!9KgY9rO5nyMKBr`)eng8Js(d@N+qSA$zi$%_Y?)-j79R zxES@?ng zCh7v8tnKc00gpwL#%~gM7wQ6^pbL36OMJHEf3w7Ez!U#|fv;Z(|9bF2l8^g8`B%_p zf6mX_HoZLSc}DAMB(jIwKNI$i7SI=~ zI@ufkf%%-v=PS+UpQo{suWEvv{1^E0ul`gv7xmHond#xLxnxi7uNaT~Ycr9^`H&Hu zIg{ca_s5@6|I1DN_wJk?^QWN$Sl<$C*ZTiI(l5|Cyw2 z7cYT{h-!#7s`sVXBgX=p(>JR>(5!H8nHr5N5Mc*!0 zOy5wgfVLfwVt03}ezMdu9DB5uUfV_m~ z>s^MQk-SB{!1Dow|D^A-CG-5)!SiE#ZqSiD`=2A(PVy(Thx~jzK3{m)1igm#P`9S?nZ%4>GU{Mxfc%sCOTlnaygeg04s|1*rmqp7fVvT%h;++{PeMJ0 zx2$J;eY3k{JQ>r!s~Fa6?jQcmv_FA=?4`*`MfnHj=NVMVzOX;?{Y-L45f91!Q)2dq z|C{IMLQc=^Npk}FEDBE3X}M*jztjJn)~{)a>~e(XH$&6mU&)zK{~B6|{XkA8#76=B zFzq)S+<%gz`;)AkDE&FwZ#a^07cvxpw}++eaJ>QBgT(*y1-H+qi{J^9lv z2VWmCUK-KEpE8a9mK)4+8`~G{a=U51aeD5LHV3RD{1M|*`=~#Dkj9=3b0mwLiKG#K zMZKUG?UN|qK|XC8coMww^)1bX{LCHg5Avr6rt4EcA8}zm4_Bq82eS34RG*9qgz+(6 zP8RBXzlr`}-%N5BKg|coTNKR9jK(u7Kg548#=Aa0w0 z+^Ca&&n`qA=>l~~@|>srni`FrRbB$jABQ67l7H zl{uchi2QLEOC|q?KaFE=RFOZWQVV~X-!9_G=TW0#STBZAV4|ylNp1?Ri1b7|u zWtRg-1t>PG7bh|PE@2OU{d?osFDTx=#?SYZer{PF?AQ7FLH@?q=hG&8TJtkgJ_7sg z4=^Ms(tg(SjLYD!>zIS$A^007_hY7fhw>Re2~7D1CqnRQEb08q?y-i z%vZuk*XuF!6+XW$`3h~usR)1M`w3-#*c0cEnUjHyxG>*OeS-N4`9s`%WyTo!ihIV5 z^OY=~uejsqE08D4P4PXNuLKSG#^`(n@zk9a%~yt@zmK1fWYPM{<0bmD_(-F^TKNQIn`Q5F@Gt+*< z9KSHWX}1fpZ-^gLrtIB~+~CwHmiQJs|FQ0GF@JAj zzdgkm--?10bcPSG#Qu=xN9Kfyu#nh)XJh}JHGw{#Gwi>o(SFeo@0_$DiS56g&N24i zI8dQ-JMpgSHum4?SS`Vz*sr3!3-#!J6#3c&<9QSC8OHM{^8ZZAH^u&w&xg$D{*&*& z$)8Z7@ow^D=0=_J)pHA>PnT}cC+wfDTl8t#kBpta{sHIv#RPNyiu|^G@2$l*TAQl}5h6R8exRL;GPEVY2cm)?rGo?*8m-!Bh(JyTMn$}=huG!!4D%}eyOkex10X{AAeRk z?r$1DJpZ?mgGFq1-_yOWT6f>voTGhxjP=`VpBs7aU+W@Y;i#0=34h&})7#gV!cPC_ zmEZp^^0u8VtzA9isJ^;3VRbEbBM9j~nN4v$pT0P%Y4S^pypC$@!8yQRA%GH88)r|s z!XtDs57e=`hQC<`@8g^)zr*&Hu|3)n1ahsOfj(CB9A~1&Y5=fl9+7xQksjP>;}X%H zG5}4R$NN*`y_#T0y(G0HkzU2A9=le)X|mu2)M%mLG{zNH0!ZUM&$UM@u|>RI#r3?J1Y% z#T_4yl1}1K3?Qnz(X;B(sn9D6)i!Gf5NW4AR|*v(aO;b zl|mckT{N~@QLHOQjHl~qQ1u&s1&X!bqg!Qm8ODug+o)%PE*@T4f%+LOwPEB@+C&4R zwiDxVrl;08ukg*UElIo5%NQPZ!V3+Df_Q;H>d$S7s4wm(8+v z@ox(o)-E&s+m`l=ku@9p^H}QDmO8T4>#tqL)n54CSn=&f(SRIBddc^gU7wCB52(=U zY1Fd~b;47fD^nYniVTl-05)v~*jTOUIWELz&u-GOhQN%K9&fQ-0p)PASGVc~b?S8n z>eLnKnnul0_1Hecrqd|JscY@UuE$euZZyh8#?no^T!lEjm0Z9Lb`3h#5y}9G719Xl zcvKb}C)YE1uO-i;evH-^BN#?z){)&HG2rjTdguc_W?G{^jWzO<81vJMn(?^*^tiFw zbF8{?hTV(dVhtHfXCD`D)cL&_ZeGPjtWfb`bljmmX8e;h%j&LMSAU8ZhMkZ`Ov8L(LE=hg9*_Ugq9x^EB&2TH|ZzN%4d})9=+P!(TWTD zJ9>Y~`8S{Ceb&zQKI{55%_rlIO~m?G)Y~&^$L4_t#bWjzW{i4j*1o0dtIDv~X8i11 zy8`c{!*J}jRT*;x{#l7fe1=6wvX*mn;o~O!fcSo#*#5`*AAK^;B{1}_YrOv+#T$L^ z%-FxgD>VNzx7E5<;llX>%_mV#+i_qM=NU%Y4WD7ONHw=ydzK3TKC*#8XK>;nA7oTr z7NxwI=l{fgw$COvd-{c(7r*nu#8&p*xxe}C2HW4vnl~fw$J2jwFfFyBC1oLo{0uz* zry5eb@ke!exv)nic*664US7u7=5qXPIbPR${tqdLY(dv}&-NksFX#=$hLfP^SL;w$ z+IvN7?b7b8+Urz_Tv z@)z}-E=7yxyol#?Wh!v=8huWegedwb!EN5qOy|dRmi8W5Z-{=NEC8{ULDXpz5U>4@ zWV!`dgctCkbKi7JL-r6s?RpNWel zRnmS>wm)WQOSe|NlmAeAkA;m!X>vDkRLJv)cWe6!LgMS0;u3rRl5C$VK}yHg_OzdS zGEV!WmiDS=Yx@~OA%2e{kk^y4{rj>`8njlSbg26&&P zQHZ=*EUz@EEc_2$4Rr#42VO@!QU9hQ+j~qvi<1FI>kIK;Vxhw${CWAQq2qY=^TRRm z_jP1H>G7i7nLifSMfZzq7A_>ei3s)o0w9f#VnR*aD1g+lu}In|Xf4|Bi8KC}Wcyr6 zN#j$osE_$%dyi%Oqq4p7UusV-ZEe2@llSR3<9}@vh*8}9oiXe9%8x41z8CF1mhlH< zd*y#>eAQ0DrvebK{o8Tc)0D8bryq+#)bYd2CjDQQ?Irusi&g(?(Eh*0kzcjHEL*}& zx&5u9zEnN+TB`q5Z*VJw+ZdvPH-`KTrLrs1|>-yU5 zMXTGr#ic7t*OfFiv@Yx1w4tV@!M|p0OT~_54Fyert*Iq^S<8~u8#h!o)vsTdw`oJ= z3h(-&?Jbp6wQF{iFKb!$SZ$NPW=+fTg4)L_c2sOAsA<{Mx-7qbxoBI`vZSScLrI=* z6YqD$hJs~X^{q?tz=OZO;NjYBC9AeIZm8+lxM9^3D_U3OH^NBO$) zbzKbwOS);i#`Tr$B^5i$>I>K9wUlqIjE-Brb(ObxN&C|Bt!pJs`Np<&ts4s0ZM!MD zg8Z@QmThVcHf<`bf<9WyN#-q^NO$?|Wd&PTdrMa3Z*1MVbX##v_r~=_fw*Ht)laM>8!TVf>@8livZQTQUPJ4; zrut1a?WG&NwYC15B`u{*?Pcq?R(nfIS1v1GmA|QcY1hWqmaa|ZconYf+O)jV-`KjY z8~UqQoxkph#^seQ8`l@W-Zl9f+t$=JZ4&m{R#LeIedjf9sO+-Z#arrsVq^27N0q;+ zYh7zgbD(bfmcoXD#zzVZi@f=Zyp2op9$8Yjcu{^qVS%r4Nke|#mO4S#(7eUl;Je;W ztgPj~twO)5f?qqiol4#fo=u)#xMM=O$8%vGzm_({@eQrln@%K`!|HNM%4=a}D?NaBtjWK1B;I zm9?q&d!Ew|+y5f*FCAY={kro7(KFFEasu(TYYciC>#%|THFdw-d1Tzz^%v~2PITW_ zI9yMvdkft(z|Sfiu4U@J%;|bg|GJ&<{11O6ociU(fX_7qquCZb(U%$U(O*Sc;e2@< z4e)|kX^F1oCg@c6*6WLCYiIANBMB2)J~++<6LKhJ$56zACcb(gfQ{l(YbQ95JRG3n z16ATZ?|Cin8ja$Vc+Z&@^gJStYGE^stB`^|#*Y~;cR~t?tDraTyaloymkh(sP02-J zlHkh!g}xa32INvO(SvxWFO?I$M*@}KhoR7T^ee9e|2^8rqo@5%JoqEmfqx%x;^#EC z17m7lY1iTZ4atACG{I|94jMDwxciZ)QU7jf0yV#yK~M0@60X`7NVwXhmdXBe)VR`Q zO3(iw`Nz|xcv17}HA(-ngsb`Vq~vEn8c@O4gFamiik?=ZP6bc#pYk0-O_caa7I>b7PXS!DCD7+3d@A6^E#oc&+?XmJ zm2estr-{*WOy1GBMtGT&lloV@5$OsExAwC^!lzsMQTn2OjHg6BhW_a}C#pPefE($L zT?c-}05{5;EA?*WbGC$AjWIIs zI`G31PW%{Gv*#uJAq#xKKyS3~YZ7kdpPr&ah4`5wZu$En30LDPUu=?aHLijmHqaZ# zeO|(?dQ3c}wFO=Zj0iN6i-rr+ySKB*10^TpsaoxegoP(Le41W|16B$?N9S zjD5}kH_Fu{;YtsRzD2^V^1dkHR(Z>0$Wi0w$#GX0;Kp%-60XKo^t&Y7I_`HR+&V7h zEmSCXp(&+wM{$jcf_r$4u_k#{a1}S6x4>0=7_h)q92mC1)zYiZKUM$L(oVC%9~A}f zk?`dfxLWW17PwmPnMk`k>gt=@yn$|pJwl^*g}1A%q3JRHlaovFKxX{9Bma#j`ng7S zvpWy!sIIxK89(z_-_dbv&Dy$J1w+lg4mS2CM{XVX&r6O?{kSjCfuDwuqBhqDAa}X1 zQ;3TnE}`GCEMHYSAHOHpuw76sgFe>yI*U8;`;+xuoj!sJdNeda8@*+|$D12`Cb-a^ zuSx#g-PkN{!FcrZ6?JX&^LQ&(tXo;u>}~Nibgpb~>}tXAH@W-cHfU-0Hr92v*EMp+ zgg{_V@F}W77NdmA+5?KxwwjW=O4Z(wUjU5DS-F#2~z1MQ98PA^-)--Sa8E3JSe zWgI0)7BqBp=0AK3+KBrLqqno;HMRNp+uxhD=N+<}7xS@0{dVge`pd3jf&G~8dk{oq zs%7fwicC3iH)UyG^zqCBJO{tfhgKe;@Gi?B|gF mb6E~g_9ItN?NtuF4+C4rrY2`P=p8>iF6AdS=t*AI_WvJhAr-O! literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf3c4db3cd960b261d0847ebaf22df84c.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kf3c4db3cd960b261d0847ebaf22df84c.o new file mode 100644 index 0000000000000000000000000000000000000000..35f54b6d4f73d3f5c1821bb58bcbdb06af7238ad GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZNH)yGv#l3GG9V!kug$DzNevR{(WfPu zmLVrXWF2ELm@%`l&&C-t@nmQjGR{tB9L?rzVH-Je^s|`f4OYMGF{V{DjIv0#cB%5SY~hR-otM_ zEESY))>AWJtNVGj+xCCxg|^SmVC>M4ExYesyw2LP=l%<>=kWSh+@tA4MN8LfJAV-R z?KpPz!}WUTgDP9-nvEU$e>T>4q=Ov-?ctsbR`|VaPydG{?CoFa?9hzVo|-eZ$$dk% ziPYEOo^)3De|1m)Fxu&w?d>{@J#zAy!OC;{2e)5(W^l!+X9mBid#-}^S-cL-Iq~gq z-|=JN$DH2@AKQOCJTm`8xNrW+@V-^2!WZ|Q3Xkl&6n=XCd*O##&xMbjd@r0b^O4ZV z_WgrTKl99B>ima8b6UO=K0ohRc<-g3hM#u66>dI$G2FM}66gv;p#uj7w;kUixr^a# zmuS2{4u$4pyj2I#&++iw)^CSLfUi7vGCb$RIq**X{84CR-+{rW=RY$TI(a;N9hu(WDoU-Lac+R;C z;Ojzo+{{NqBZOZDUiX~{?@c_1c7K9)`vxlj+nabIJn|r5TK1KnB!0DK*lN zlK&~+8-VY<3g5ivdv~_)_1wVsj*Q`ZM=ZWaKB@d~z~X<(UvK1t*LMCe6e*ef_V5Gx zq4WP#eAR7dX%~J~e0AjIvi@JMVg0F?A7^HboVZ|LlG2>c1>z{pZry)$>X0Y6@tJFxQ6n zB=ise6YC#-Ks$8df9h8+yvq&^|4l+@_*?AIg$Ast6IlPqk0t|_7#eOE8 zp)>1uKT2~AJRiEY^EK0466?E==K}aAS%$$YjrktQ1Q{ZCU>aoPa=ZsQ zjPjEVBd;cgMj#J#(SLpdJ9H5`8F@J=#I&Bcaygji>Ku%ptg(JtH$JZVVTfSALChL6 z_JJqTRZ}9aX#a85Xnh&`V>e5m>j}~LL_b6G^z$)N;*W7HZTgjut2FLou1wF>lzE;| zYC%!R`Gw*T*@kRFcB1~?ge=fw&K(eQQJNo}7uB-8?#C;F0`*A&v*Z;=-DO~@J z#)rM$eJ&wngI$&X*~YW0??V5snR>_t`&@wElAV4H8<5S9 zvfn*@nDy~IoCki)U-WjX_3HO(|3PCecDVtapy*7~D8L70onXuHa-j3=xO(tP!YiG* zXxLF@yD!k@Yw2Lsz~mBE$IR1Han&_u8*kHe8)NoySn_1CIhh@|Ulw_7kwWO~e+ii+ z+2(q2bLKybtG=Bjde{<{0FI^*<${i;j?O@*w;p%u+TAMZ{94!1)Ycee3)yIyp-ZZqg`n{cOL8mX+xj=|k*U;73;_c|D^R~2X^>#M-0(A}E zj?OxX?ew*Ev+l60heWVaz}IqDcpK#K-W7s0`s%y3 z)@=!RTYbVhR)BYb34&?XKubfw>u=(d&L3#+Z13#$qdDQ)+d4YEZJl{{f%`Ui{jI)0 z!(Cx!n1;(@wIv72rIQ=b2)GMusKhb*EQj7TQ&`!S3v@jL%b;Rhi&eW0kw?f(}J$c9FCJhOL+-B?-AB@*a0zB#m-K3AMgLvEZ8*QM zmIz9&1kybf@V$$!X{oz=dS)|6WOvVO2ft;E;Pb4V$hzX${&fIETl;A`1feQkhw-BN zs&y5m6-Dk6?Ei6&A_^Un5b48sZ6T1qGYDTS>1y=4>2#sF#oO83-sbLXZ*S?C=f=$T zc{_aW{KX3j=DF9=WkKOmR@~Lx(&+B!?&$Qjx?8(CI^B)FEzNB{_u8te6>I9&mDQ}N zs9IH5wqb4As?xGjHowZf*gfAHXlQbGwBkJA5w9CV6R&i>g3muJyl!Vt0&ZXEVB`sT zbe}Hnam(x(vHP1j&apM6GW!-g?1HwB*hSy>(dY~wh?Akq>C7(ccCz&4qG;VlqZ{fv zbDQdPR%OIfpH#P@TietK(^ue_sm`X^=cIC*l3l6TXJdRA#gzhK~{Xi0X;~<+AOtAfC%A4DH zeJI;r7}RZqXN>TWFE}xhjatHFXdC0r3MT1M`V(m^Juoq0BK1FbFo~TdddBIKh#vUd zpnW2pb&CFpo=-DDkN%+#8;Rqu{($qJ3;7K6;3rMm3;duzqUZX{=+uZG)OC{2Umdm) zze&MKup+l_g2D^>FeAJx;G9JKrk;r;vB5M}_-E)JbuRClygz3eYW{lkhyI`1%OYeC zy94$ds$%!=Bw8wv?_wJJO858(HwWQ!ozRayeqC*CEtmJrY{)xVCtr#; zd1Pd@qQ+3IA9g(9@{Ow2wJIu7L1$+pnPy^sfhO%t?4s0@*h>lMf#m0RKa2 z_Ehp`ER_Du&=2q#!Q7d&Zo#y_q!tKH5AKH_??C78K^Nvj~&w8HGx*Cb>;r7pjeWL~RMXFBr zhJRo_=kobV^Z6HPtoQ{@kdyxcU;e_+WOGp;-Jh8r{+dhnp!AAbZ7ll&`;a^zqqy9zyGd+REo6h~`AlY+z)W3$D zT47*XLeyWe9?T3*n@04BF@KwuNpYK*1EB_oSfqps#Ue4#y`j=?g9{Qh2>Q??i z`sbfW5gy}}zp%vAU+>P0n7-@VEc&MTZO}KLuNhq58B%}n|BR@<>$9<5xGeg1xnlaR zB7M1H^PA5Pw3qru{~1AO3j#z2dd2!n^Me~u>MP4EFZCB)4~B|ZdDd)`d}HaWcPH|J zGiXBVSI(b6ew)GHNPdUK-|Reo+DZPCw4T{T{y^(PJ*}6by^3!$gAPvbi20j?>x2C5 ztfr6VPk1X|k1_w2!+xTO`DcxXWZ%<7AlL80;P~-me?C6qMR{^Auc>&@yAw;lieG+` zPmjr$<3zj^pOxOn`AyRcDgO9vFd41iG=Fp+FLlD>%ME{~_|Qv>!F3(@_?Npw;{gkI@y=UcjQ0tXMA)_ z{@jy9>uY(k5V_`1<@mRQ{A~dk4#z8V~E46I%t5PxJLG*+$WZ`v>GDJYVlJ z{EXx+>;;|=Ap9qNmo1s+#}1w!+jE1Cn?IWk*~Z_1^L8! zk|O9ew1>JimCqz*1d~w*Lj&ZW)L#mQi{kAW!EvY?@icvn_yp9A_(Y^zMtlIeT{9`XoPAbYjFh9?rO7?~Qk?&`cJBoNn_MZ~7Km6Z3 zKNoU(ZcmyM&}U(Anoi3tBmJHJ3tGRXC9=yAp5F{jhkqq!M*VAOA@&0~nGhcZ^ux5@ zaB%-gj_yyga-#I-Xush|=Jq20;rkOer_UmK;B$k?j;KE&zfBLgv)t%CLiFTMyBvId z#CT~$4}Z!u`de-=%WZ65xXbOP`NrwFKiV9yj_^l}Pwk`r_+c7*Hq4PMawd{S{1x_s zUbIi5d%Bkmi$dK4ASZ`8)8Y z`2f7h-+?#G2k0j^X!d{j`*gA|&v(FYM$j-HATQyg{(n)Udd14mu=TQ~$ob;WD?T9g+ zCni$=DxN0<#cX0V7zO#m?rw$#D0Du8{nx>6EVZKOv_HrGSR7)~{_sui@1&iGzgVtk z$pw;J@RuIYQ|ZJc(nIz?s4a`;3j+VsAa@4F$A^FQf*ALSMI&&*Xio`HY`)uk_3F znW+E8@)>?UaW>6dYe+M%*O;$_ zkFM8a<|}-DTk;j!j8hT*$oCV<{;(&`A2TNd8*yR2pZXN@74nC;`O1tj@)h@t8|N!o zJYR9g&sQK%mYd>xG+zlC@{Q5?3gW3dE1ItiLw}z#ICJMk*I|Hk@e-fu6&=L=))w|8-I)cVBZGaaeX=S4XbPwwOMeey&|Ih5uy z#V4$vrv3F6icg7rf6n6>@}u-!cAh_BKaj@WK>oTbA(7&F(!?|erH;!*pxW7-0AKwr^rcCAgFSQ>Z9p6mnKe@rFQ!Mc!A>KJ@LlWD6JDp?fzj2^K z<#ys-)otv*)3I8DL9t&&dl%}_{V4La3C8m#;4_TpQRM%bly8dtC!Y_Q(fudif0I9< zMC0A$$;^#9<*VlwLZ2?(pikI8UAO4dv>zEefBgf__lpVU{1y3a`QBTLZ^(}f-w#M* zE9tO_7ehWvOM=`_=XbIQ#Y6sv^#`99^YdPDo?$o-KLVngI1fLH^Kel1B)*Shm8LYy z{yI&^exH2}1l*so|GXi8_&gl`AkNpb=<`$ld8AGsm?u&l{I0a@xA}f8DxZ^)eEfWo zgcR$mog9R>A5N$K>EjmJpB6j+&$~H!&F2Sc?2F(>>}Syd&VS0gKZ@li`24_>pL_;& zNOiB=)4)9q+|$544cybfr>+4yJV&SP!FNK}{@vFc8 zUF01*E2~{STzzFq4 zFVT+kHqOc$nz0yuPl^>!>5jg{x#$YpjL(zsJ=GK>h`m}OSdNx>_^4uC@!L}_&x<=g z9?8EbU?5)qJl7`*Pkg+{wSb3W;<#r3X zRE#znvyN5`s6mgT@Wc)2V70EVd=PWb%{{1>HfOz>#|Eq_Mt%gWs9sOea~z4k<12+W z%DZT6wW3&8j2KVX)1c}%{t6Upzel&q>@tiS&$dy|1YJD5vI6xpT57|{qqK>QAArm?U}(s#6_C6Y&Ri zy5&nB@LPj#Zmha-Cico0#Jx0k6y=)-Ic6?c(1S zHmqG{`gbhtmmq64_UEzGt1We8tJhz-xr!nSd7d7K?|JiY4wdYuM z;|#kO!^Ijhmd-vY-l+3?G2Fb0i&&xJ!|1p}d(8MJX_nPpx32yaFAO_5O}m0DM(yNe z#8u;(Ec6G7OEhvUHrWzx%>AB(d;8Eo%v_DPy5H*jzx`q^mp|B zlJjpq%lo{Y?S0<$Yno5S9h->tv8cCa)Q-&q4~oU?Ji1WkN6CWj$|$8=)%WM_yO_#II;bY^*{P#oJ(NnU)Om5J&HH_-kGs~ ziC1X;XKt%?t-^)#1)5KyoVMe@CeAaAv>QIds6aKhTzi%a06wyTKxc5`As=K^T^gml zndkq+eYVdgH+%ZUoR_});>1?=y}7^n?FQRl&6+nO?UMrm?4a8$_ih<9uI3PR%Rnc@E%tJ^+ zIb)e(YJAmB+5d4gzEOKcN9~Eu+Meb)slei6OB~ggH zSt74As4V;sT@7^te-~ayJW>CqBHMdRK#P+BN9zmmUu>bnBm8;!nW5u&_VdFr@%MFP zKk4zJ-I+fY*Mj@SH47J#-$aD^e*%!kM=_x$ZWKW3*jOZO6touY_rw|hE3$pAq@?kw zSk%XSvc1PL{!!Up`7gC6m$tSqz~p^8&iG&71Y#67e`m}(zVf3=wC_cGk7fJ;*f`z)>YrSG!H!Z>lZy-yRCTDw#E%L9UC{SdSXTEs{F?C+OEdsEjutq8~7++SH7;R zVbRiV8n1DEWqWbOj?((Y>+)L4w^l~SE#JDzTeP&jtbFTQNmIVDZC&ez#p|}+6y2iy zvFMgk9*M$E?_~VI%mi zFJJaVLw8{_-+p4^V)^$zw zn`+uiHhODo{WVKlN}AeB*Ke)%7MHACR=z5KQ+Zj}#@3duP33qkUfH#2d8NOxbzL{~ zSFt*O-4l(=D_b_MUj%#C%>|Dte^b}G z)|Tc#-S#bu8wwih7dJF6eK-$h%x}zFQt*hc-n(VdqQ)&t3L5GJT|@I0Z-ei8Ke4iw z|F#PKstSJXcc1x@mx)RXAKr)qSbc^_>0hy_i2^*{g{#iiCxa2|O$K*a~D z#CzWJTHZAp#VPTgGcD+OL>$$^W*Ao?1$~SkGhFV36cATIZ`^qcWIHYyhMk*|i^3$q zmH!KUG4@T!rC_25@lIbVCwh+rD!&gyq4DTfUI+erw2wzm`JF` z)V$KJ!~Y9NyokP9n&5RQ2aOqT+&87XZfOEFznVc${aluC)pn7Dt4(UD>_12LQ!P!V z^!)dde>`1^7d5Y*lYG7+;c7m;F8djf22}9%fTycL(bHDgdG%;F^$vYa?2rrd#QvZrKB3&Wj)_yif_;gD@N?+8E@sy~?&_6xrM3v_aa3lS( z>%gxV;6{0KrQWT4&X#biyaUoeR(ao$aBDw5lW?oNzcln?i67%?_Pm5YWPuMD=#BP$UBa#W(^GV)5IPPWH0&F(G#zYH{o@{B3glo)UN@&^ z>r+&b`|mmHh=abKVVKMf&8ZLSYM?s8wJ z5EnmOLce2KzN&UUeowApyP#SIeXQ|y7IotHC+oX9eFPQsXlQ~qdP{weH#hi9aG^b4 zll-~6v02=L@#yC(>e}e%@m8!@x3aX^+v02JT-n~()q>$~a`(q=(9-U0tm|yAYvhax zfxw>NQ&fd4MhTa-2Nc1rj>U2tZ^CBYz}Aks4!^Hq^zVuW+8e!{UbcY03x^a|S^-H) zIZBW$Xz1w7fB2@gafj@MZ{E0Ze=zlSHt?o4pL{nj=3|HY?bbW=mtDmI`w8FoAc)9R z%hb~qnV?RU`G-QvWm;7k?ehsH(}F16BkxYjIHnLPUa9&l+)&EpQ2=@M?+`c7%Y2MJ zTcda8D}Rmj%IT7*8l%i^}nkAzp7iezgoV0sjg{E6^*^dVl@S2EVD0m@8P!| zmI_KY>#aR*tN(en+xEZdg|^SmVC>+KExZ3cyw2LP=l%<>=kWSh+@tA4MN8LfJAN4Y z?KpPzqjh@d!)jaTnvEU&zc$u?xRV_O?V(K>tmymMo`H`_**m||*})m9y|rg-llzBk z6REF5o6=d)|Is}I!)T{#ws-0=_Q=U+2CL5P8{Bs3nZf0!o*DeM?zsxuXYo2X=fun5 z{$oeOk2${^KDzH%cx3*GaR2<1;k_$Qg)iYVz__#CD0XxLi_g*ZaubdaL@ds z;i^-|!Y%Em!u|W+4v#!^F?@d8MbIn?RXX<%o}d5Z;GUJ}XYY5zbJ{Te$+yFm=Prh~ zUZU~-I24+X@mB6fKgYsz+g=Wj0AF?PWO&YrbKsr&`J>Rt-u;75&wpkxbn;mEJYFrA z-VV<_aUxvlyc9k^{~TzRgyyt8IoJ!>qqqdB535TezzTYQgvQu*I;i~lKq9m@x=?f6kBQabsa;Rp1C z=l`+fs@u-eF8r$G>c}hQ1HWF)22wFc&dg$0hkvABeG|Stl+CW5DV=n1s8Kt3=5O?C z!+Y5P%_XKCe{>n}k>}Vzx9toci6!TjaXABuz`^uPX;V8H2eTN_^%Iu_P-^C z&aB(@D9tzUeDKe2c#_Q!6PKGzeX@rizh<{98)q{JWNTH5q0pHyqy$6T47t10t5q0~ji zA?KG$LS!4V3E7GIdk=a&3;nqeKLg30foDY=!;de)X8#8J#r%OCynFz5HQ2Fvy3hyt=_l32p=(JcSIB<% z^kLS=^Kc&cF@Mq9t=4PaulswAx!C1Kbb_KYO`{+mly!nF!^?rrx8dr+D+#Z3=AvOo zm2JL2yRWsA)c})ASRFG@PsLT&m~FgG(`}5|$6?8n#pYyo+)J#UET)VscTQ0sPk)mXLEZ~kS%EM=<+RSe|$mL zwwCtI9kF_2x2!+DAbQi-(dPGdu?1beVAld6T76@8SF5+Pv)$9y#Zg--6KfA*Xav94vAo;fUotg@HWWdy(EyqSq{Bxrm(Ut7wCEhmO;h17OQrhB9D-n$c-=e(vW*;$eorJ z#&P|7X~-8Wdp9jxWF0Hu=F^Zne;=b3>DAa^E$G<1xzpE`$Lj0bAFq#MR+O)(7nu4y zY}H!4ojiI+@zfIe%UzX|-6%560n&!uvb8MX@I)(^{V9i!9|d2J&AN^6*v+V?I2jF%mNyAP*D;S$WgDe2YFivD|m+i-qiEfJJl38Z@} z;Cm-s(^7Z!_ReOG$gbYm4t~oR!RJ{!k#)th{hI)Yw)WF>2trl74&g=hm1`@@DvRBv z*#F}kMHD(DA<~EO+DssSR}j8f+TG-J)9FG>tGBDAqut%r(b3vD&yAVw^LF~&`GpG> z&2z7%%YvdMtfaf8waMMt)7j-~bGLPOcDb8;n_Jp_?lslb%U9R0Ew5c&S-rBpeEpj8 zm1X5+Y<{)7&^_N9Xl!qp^XyTR5SMd3Vh1YHDNx-f z>XYg=bZeU$Vfu0$Gu7KP`@Lv2 zVEE5p_UXdQY*}lVJ;lMuyL)Z8e^|n|;b^N3uPs2J&^UDcL|(7S>&Lj35dww$V)Sfd ztZ#=FVZQ>B3^bI+vU-HR_$0}3vZLD*Bm4j^1r!ZVwtx=lQ~`ZzfThyl{+i1+_KR%R zx7QwFmjNKAo1ARdoA z&luq$UvOe18?}VV&^E@K6-?5j^e56-dSGJ0MCyO=KoUDk^o-Ld5k2s^LHk5H>lFPH zJ)dTR9{ocfHWJ5Q^C9Oy7xEeC!B3jB7x+PcM9=k?(WMbTsOu!3zb0%Wev^WeU`1}< z1cev$VMcgYz&VNdO+6DyVuNX{=+DqU>RjHpcz@0`)cg(T5B)#2hegO9b_eV^RL$<+ zL9|pL-^DccwVv@2ZVtldI-wtZ{Mx#@Ixg>9*^qa#PQDazKtJd|H#m7R`P+D|e^F?f7rvxot*3eeu@HY;v_!4sQ2wih_DX@qQ+3IA9Y(9@{Ov`;ucu7L1$+pnPy^sfhO%t?4s0@*h>lMf#`4*rMI?5X6>SSbCQ zpda8fg1Iwk-GqPf_PIfrW4KP6O!5t8vx*lE0ZRqrpC|eUXI_a!v>5#yC+RODU+0|0)VR_#n5JCdMlY%$mdHq4uZJFhA!-*cr5^`II*u^|=vtQPvwSX)}ZK z=FoWah(Fk8SC3;JpKte1Pi24UWHW@=nNNw-VS=KZA5>> z$?|C4_OZ8q*p%HM)>=8!ls z{H$5g`Wt+{aC`9ihxVi|JNoDRylc}dvYuzOu0|rex&1R?-)I4Sv8vPh1^>W&&gJu! z=JPMoSjh{TASeF?zT$;{lFdbZbbn@g_-iiNllv>iBmde&Byv7v1ZU2q_{aV6=hXi) zQ~!NCrpNqgXg}7s1RMDqUldL)gnwbZkNOw+&-4TuZ#ws%17y$XQU4lpYDIx*2~mH= zdN4CMZ5q)h#{6wsCe^vW?M8bS+ADusN&21UGWnYWc(<_Uk?9=oKwY#af3yQH{SE#N z{mzW)SJ|5wAbY<{`lfpnsc#z3uKaVT6#D5BcsZX(>tCW}d+2{AsayF6>7RchMR<%? z@xo$Le|ynjhSNQeRnSd8xnXdN5SXDzavqzjFQr z^4kplM)Er>{$}U#(@yfAr1i`$@&{TU8fd)~?Nxl68FX-ZN6g9B>SEw0-28#1;>vk`}6S;FDjCAc}>NGz8zTlRs8ake0og294F$X z_^k9k&TpDtMDfROgUM+9run1uc&QT}UvBs_#ow!-L;QS2)oIk@sCUziEWq#YVLI)4 z5{oB{hx}}`{E{}dKFRz5`5WfXeBwtH)Q@h5{hk1T@&Wh{#ot%SA*ua`B>&-9e1|?E zk0rkAhWMTxC!Qk*CX-10>VeEyydGMpEeqHwavS2iu9JOvd`JERf5u107ws*0dsqwI$Ik8m``7~e8l5G@ixPL%i!t?bm!_P?GqCVjH z0K$LLciNJ9e(d1+u{}5FNS^%3%ZN`xJ%+ceXMBCLyJS2W z)4!`3)@$w`{!O$$fq(3w$w@`|2j=G)RLQ=uKl1%daz_yl$^KJf_J{wQ=jTFB&+SQb z0{ScpPSa_*Wu(8;|CQFSX^HG|gy%Oy)8SvqnNj~5T8RBXPA0@h0sS!THyqr5lB4^R ztehzQIofYHlDWOefB62y&FQm<9{Ai~vLotG$ZyjF?kqQYj}Sfi(@qCpA2D7U(ZioI zjsBJ!%yJvs7wvSrX})oK?vFMHtRwsp<5T;nKYor<&d850QOW4xR!)cJlB{lUJO zDLO>@1;ooL|KS1-zI3KY7nEV}h(|iEl6 z<^%MT8#Mbr{Czsvm*+d+HzR1650IDeQUAZFQ9cF#LZSHh626Ft{EN9!C;gsXh&s{* z>X76)Py07VLQKEpAIS+8{btfqhq|Icux9G#CF6O&l3}=e-+OY zf?_tYT8x7HVOI}B0~9(R!T#$&50+X{blRU|e=H6$X@B?@_jl4x#9t!Uv*bmRT=173 z&{OHeB+^UvKcFp*<_iMa<$^?@Jl=mQ_~PIx_maefH!Cg#gd2aXC*Y*;T& zV*H)L9{z?8#<5>gynTb8?b6LPB4+lh5RRrumGYbFcKv^O>ms#qt?` zK5;h9e#86)@$I_#%b&vk5Z^xU{AF~$;z)kgi}X|EBbaY42l*pEUx9y`^Oc3b8`o!A zudMlsVf{tC$ijM!7F{y^hkOJ=UYxerrfGuh*EbgpaP*W9BP- zep~Vt+Kf{X{>b+e%Koq?&L1-;0~>K+zMuLG^A+-kxcSPAG4d7nj2q`GSv+5H$In+F zPnMhFdo*7O8uE?N`3mBxJ1d&63`2jPJ|D@V^_9m<^k>OO5HC&nhy&xzaD(4Zk&k4F z1)t*csQl>8%}2<8{=dve=9%Kx=>0ck7btT7y)Ve@1mXDy&hIO_UmY`E9X7=)!}&V= z=XT;%bpMU@&Ai`Uh|d?s+HddV;;8kB$7ecHqtAo&V$pr%tiNx7hiQb$^TbdlUQZDaQC#6r7+l ze1IkPhcrJjCrpHe#Qr-Q`|qp?^!c1&|2>WNi-vgTqzy@I|Lt^+vH!+_3YFW5ch$GC z|4zqh2?oV}742Q9NB5)1*CrUxn}E+So=1`YXHvc?_Mdz{WJdR&eE&`Ugc6N+lP5Db z>XfgZTL^u+bb~%&|8(7=Pt$&6?ELi)INvWOnDbZUw-tMCExsW?GJHQEjjf==CSDBr zEG-FgKb_yjA`}n#8`d9uUd+#X#d(I|Jp3?-ZsI)r2+qSn*_-$Qj#Zk|Ec@#;9s7Ou z4G?gD#{Tn${NeL(_=7lK&!W#y`R9>3ePEtQb@02wvft+WwWxefM)L9VMG{i1uXb_} z-hMcp`lpXuWPe)h{6FvJ@o)eB=T+nWs_EnNe;qkc z#Af$D-RG)v_s`8a(%;Y6fW7Xykq`d4KJpEYN?Dcgm;E_?{rxHI^iN*<{qG|0+F5zs zsu@T0Rdorg>aZI@NdMVvitG9G#ZgU@Us~jKRAUd$0saaBl%U!;d&(6ap^JH-j@32% z%`$i&=S=w>wzrJ!)|McUYYhzav7+ZV6E#)~fKBs=#5;=g;7%Kti1w5LXxcp9pBnG= z1Uu>_sU?Z@Do*v-wTg|C1uvjR3k9b!uCNk7ibsu6Lp?u4Jn=EEybeaFFM5e~oVRgS z-q4K2@Vis2cuIHl#m<5&3{UPs=}SR^*rz3e2FdK7iDdTC z9z~WgaVBY^oI$!9aW;AmnmM=WR?g$u5EA z8uT~{Pu!3WR_XeR2Ql~D+=KdPbJnMMY{05wnBn z?^^1MH8b90sn=NQG#>TOmk9K~UYuwXh>w56I(fvcw)wR5#i?cXTkRuWo15YNG+$B2 z8trC!8mkX0K3<=nA2(L+#;O}09(oET)~KOVix1Eh}qNK4IYm%08F@X9P=?Xp?cF8*y{!`fx0f7jA} zF|uZ3e;!M{#!^SNdi}M_xY`Tf8!NuuC>oICNH6&Tv+L7QpiaHcK%KfGUDK#Jsvg^C z*mN4DICYJ^*!6hI&5cI6$XL3Gm#Yw`w~`CE!LC8aIzkyBu|gUl9goUlFm?ujXJ*h!rY6jE+0B$BciHW?9{J>*`PO!myLmv@6JB)J{%DTs5xALVu9B zL?g#ylP%%K-0x1fw-5cp%++|S`>oFZ+m8mH#pqUUL&ffS^2E;#G_Z6J6dr;e@E{xIsfLfyf50> z-WOfJruk&tv58n8i+a09?btlC$|5w{zspTa|sOn>l*LBSMf&QJ2Un#@e0lV%x#sf zRl0D#K=VnI({>!##Ce91cEe{FEmF-b*WP6UfRAh-&>5U~$OjqKltn3T=J`KypY5~J z&7OWS=cVtxII)d=f9`L7yWaL!v*yjn`|0$b97s#8Y)x5+A-@36|EYo0Zv0VQUM}oW z1)lKypI4VMwy6StTaMTDp8rD%B3sZk-m`rO{!4m8vEd{r`qetrm3AKmoFB|{s_Vba z;eGI9rEE+*>BWc?&*_Twqx?lZr%TbIIWOWl zU6~3Ty+)tYB_WDFN^qMuG}HMpou$1;)*GWAC<{PrWe|1R1jK9q6WQ)77S&qYEZ&SZT>GA272;}voZ2zIG)A;mV6Ki|&dvr01|KoCsxDD;;R<&1lJb;b~<{_k^oUu$XHNI-6 z?Eg3#->AK!qxM8+ZBO%@RA6ya#>g>IdUM2;+7nH$s7BxMCj-3C(kMjUES6UqR2Ke+ zu7)~+zXz`)o~VCQk?lPupvB36qxFUOFR{?!5&pdVlcD2y_VdFr@%MFPKk4zJ-I+fY z*G2SbF_bL4NPZI$>i+>i8Xv`knz%s#sbgc2v{BF+wBH?P{IALOxssB`r(#hb^U3xe z%lJoRd*#2>o?P16ei0_`({aZC=0*^sxcR$~pKeuqeAQ0D zrvebK{kw76)0D8bryq<$)bYd2CjDQQ?TKe&vFd*{+W+S`@~ifjWn;K0x4(6K@(I$( zrUpr9xn#-#)m618@a?waQC!YX%W%>8s^+b6JmOW&3u`-7_ecX9+i{SbPl`@m)%;Rd z)ox4i>b0H4<;9|ut|%#J?QSeAY28q`x^8Q6>l*L6rtS^vYx5i1*41q*TGin#E?rT& zwxqeSZE4rW^|h^y{?%(*E4MFgENBL7bsgc$T9>Tau)eCfVcpuijq9tHd)F0hYptrT zTfMzvY3tI*>YDwvt6P^9)ICfj)ZR^IirTGoZMB9?qC9MtXOY(dhdB4lo7cA{= zXj_s89{ddj57%uiS-G`oeQoE4^(&uP-nKHosiLmCX<6%bjL{B0D%Mu4?QSes(nI4l zt*h!NsoY-HP`Ea)wPH(Ebli$9E4{@_I?5}ytdTSo8`{^ltuI`=^`__w^2efEy0I^lnKy4F-Q{M4MEm@hrp>0d~*5cZp4eN>mamTFOzJ3GvZm3xLL}O1; z3*>EuuAXQpEz0w)FY!0D`Cy;5RSoT{o>)OPShlvsTfBNjN&CvY#io4!T1%Td%GPbE@s^aXSX!|%e`7^?_lCCC?u`|A6|U&sxU9bBa&Tk3yeL(8H^mA|QLeOqfwpnlut zM;eN}OA3683KwnO+~{pAD0;-3SNQOff`-CHg$?uzC7iCYWwW=@cfFrjS;K!@g??29 zzjksvmAo4~n>@d8(V|6#d3lRD%~Ee`r>|bTm)7A2WqjVYykH(HPS>WTX5&FF`29#_KwNVT_w#UX++%)?7F;T8Qt$UXrysKa zm&8AJd?WQ+&KE_`MBl^-#2X%~CCYW^X{Z7sX0T zbgef*r@FUZUqV|u`#>E@n9%aUaVD6MLn%9kA{I39)dK-+5|=tV!FlB202Lpo67PA> zYk1da6sN>{&a|NC5ph%tn_*mq6!bBE%y79AQb1eaI)kR7;W7b z#n`tYmx75N#5;Ydoaj9gsQf+*g~p>_aUJ;Y(LNqM?Qi12AGr?vhkz45r@0*%Q}ar@ z4*xG8@gn*fX@WPT95iOUao>{ix}^!!{AvL`^>bOmRoemySDVx_*?*4gr$(Ag>G|&@ z|9H9-FKS*rC;5C;!qt3wL-uoA8c@O40iLb~MNg|xr-G;Wa`+}${|5nP-@%pW^DqvT z6XL?&!Ij`wB-}3HO1=r`Px%g^CQAGy3p`K4rvR?n66nhkJ{54|mT{*6ZcG)ANH~p) z)5K^wD(`4qBitk9r2Z9eM7m7Et^J7kg6D}^`uVbr13PD{8|Ui<|KFVv5b|0@Q#QQnDi{t&$}M$Wqq z{E&nbKgQMUc?o~W0zYn`H`@0N3Agf3Ptl=5{7ezI{C$yxt8tYtHcPk~SHTY%=#Aq( zFX7g4e<|VCaZ}~IrE!h(#Ulv28*xJE_6J-o(Pv%D&}iW|>c;3_^Gx4=~#7`DLG(yPutRsU+epAG(~D0r{5 z-!cnat@nNlT&?#^q}`qM4K3~7Ko7$np~<`4+uh#S{22eq$#Oi98Ncqxf8&XMu94mB z&VxFtZ)tDA&pbACcHUaE_U<;pP>ZjVjlIc{TL=F0l4Dap?hAC{ry-=MEe!$4UE%8z z;^K!(=yxnDR@TkO@5wc86I4r~kJY}e;x7FDWJ7nCkD!7c4NcG{Z<+7$mPVfmF0|)s zmOpnlHjA4v9{qepeLMX;-ty&ZSCqAQTYZgPD>|CGTQU4i?*6z9T06W=^<5qHO`I_y z5ZDuZimH&sDB;qMfFiinu~=@?P1wvE*wR_w>Gw5`{$0^PN0Yb9%NFo=;gG^g%OOb_ zM+uSzjh$Wj58t#l?vTCk%^NrF52oJE2Hw=>)9>cReC$-e-Fk=qva48NKjr&w1QD5P znR>b+6V$0P|4>M|OsguReLlftS`dYM z#jm2b*8f6|f4%k}TM&wG)n7VDp7GS_;F}7I(Z5G)EYzT{E78@q5EN|Gs+mme~gF8_~ zO35;bp?z)AnPzxtrh{lmna=!L+{rX<(_fONX`1rpP4hbQn^HoFp;;IwLv0$O_dj>} z^nIUYn=UV}lgIT>zTdg$+;h)8=iGA_T}k(@+qB+lu`rb^?At5`sZz#;{4zhrxC|BGIz+c$@?Bj;?{C*Hv8ye)gtuaIBB>rFhPYKEev{MvzMBfm*z zBUh@dk@vUSBG+u}$bZ_{iDRAY2xz~+rm&U&knQciQp$e%3oAP^#~rFXYs)-w&Xz&E zzQEGi%AZ-i{e!4ywb*{@L)&@oN6wsh_`x%ak-zoOgJ=GS)jI;(KD>@JUwUTf#49I< z_P+DX(9_8CHveSk?stAXv>15{@(qu^J+$$ei$ixH--O&X|K`XK-+AcF-4`A@<6gQd z(tPsip%at7F|^^;9}ivn^(#Y>hu zLnj{o@zBMuzddwm?^~c*5qaQ|2hW^X`tX?u+P?Ior-z!+=B3TA4sAeNCzcY;CnFc1 ze(=nLEsvbZ`|-)42mj)kp$Af499oq6>d?i9-Wgi-V`}RYkv!lR;nlq5X~_Tep}dw? zhwgs*)uG5+zaDz<5u#ZWx%kk-XHJ0Tf&6a_En5Gzp`>SC8FH1qGSm#2FYUbmn$;1~ z)y8N3;>-iNPY*467;WXhGPE)GM?*gxe0%7@6N5uZdACF^-f`&6qLY7d=HeY_EB|Xl z7bpE>Xrtpr$n)0FrNtLOb8BRS<8$Eiz?q8=oful=er720%8wz}tLWd`LyI=Q3!2*^ z8y@|OGm(c+a2%B`~LO!{7SknHNUv6W8P%wi2y*22f8 zTD&7?ZBtJWuExq%YW@J6@Q11JH{laC7Pj&w_=LqmxTz=3S)F%KyrW<)h?}$``%x#c#t`Z}{TJ%NIlA_~Nk% zeDRpc7s*GJKMt7uQTgw9K6!Cof9u84!k=C{V2OOtZjD?kWk)Vn<=L<9u|*yNeGk5U{Q2U3r`I!^+3l$_211pQpLs`4|9zyP}z zJJR%w4ZOpZ%}-iJyjutD$3G96eAIanxTBywSlnNee(=_&Oq4x`vO_5A1kH!m{^Q>U zydQ8K;0u7i0k{qD9f1D@@K*sh0{$f67Xbey;7R{}ON|;5z^>0Q?Z(Re(zYX97M@ z0=aGdkY}XnD&U)uUqQYc`4IAD$lpP}82KgS3z1(yJ`efNk!K@6gM23PSCLOc{xb3@ z$X`I7iv0V?laYTHc_Q-fAh#j^7V;?Mcn0}zk)J^R8{|(Qe-HVWk-v-l5#+x{9!CBq z@_op+K_6d4KNch3hx|wA*G}Yhz|RHVhkQ2jcI2xpBhiEGrw0%pYpnT4>aB%G8qO6Q zY5ab{kv&fq94SG0bSCTnCFb$y6xJWLvHssCvwn)LsCUGF;rqANocsQ*erwuE|2ufT zM9--w{a5fjM9(QF{U74_Dm_m+N%fq$i20sSWtKNo+5AkxHP7Zj`%BrO^qy}ejMV&l z_*Pba+M;gORkL;;kdSnjdQ!?I6 zvXYFDkK}wq&K+FNZ$XA{k__&X{_o=XJM^4<(*FXUzfaGulm1un{4zZ!aXAN5Lh1W= zCyWF?oiNh8CE*&$#a=r$fB#O%S*c{og-nYe(;~>U2r@0wbVjm}u1FTrAI3VL%eXv; z&{mk*a-Q@*g69Y6Iq{_bi+Dat&yJJ+C-D4bdQRZ-G|dhj?YUwfX@18JSupmvj?T`n z`E%&(Q_p?2X2>q}G#@f8giH$|(?ZC!(4eO`u$I#6Nweinv{y;(*-!dEjpyC;Y&+?1 z#B(h@TTlAi@O(EtTZAn!UniwKUU>&}|1w_958AKw9<`4!%fPL9>{y`bT-uSnMM?d4 zf5P5>PYGT}?fuP<*hd}!ZPORgm$1FRrX+l;*ZZ8kX1(>AFNys(KV#c3V(plObz^gQ zZU61EEDv?;R7UN9=LY1iq)=t%n)+b!@@YsJY8 zjE#5_*h#;2>q$?71^E`_=#zCba@$EG{ic%}?27&lNxxCjZ-_%*esY6T(U(d3^^$&F z9Qw5~%9a&V&e(Hn#dsmh8Z%kt&i?L?fn^`=v80)Bui1ix268H>> zUl0f12>f)3&mWCnZOJ}O{5H;FBO1@y#UmR%sAFzxF3NM_l{Un}c< zI!?WR0lr+~8{^==2>hlvd@q#ryDVwL+?H)|>U{|OeGSA@cW8KT=xNQ_M0%uEg7TBA0hfUpos?hzla zycb*ado?;wh#s_IZR`Y15@=nhlZ?7+K)G`t_H|uDq*Q&wlXkmdLfQ7WWS>Ov=0X6Yt0% z)^8vFt9Rs=Z+J(}{iwMA4XnS;zfs)(E3C_Y^&5s~3(DBYh4+g4FT7{zKldX`|C?CD zU7m^dqip0q>Wce^9-{heY>$nPB)IS8Kjdi$^L*%%rAE$jcFeFOAT9vS>wTV!x~N&iJ# zN&j!2V6ybXC5KM$HWuqHZaqq81Sv z{c!7Z5h}x8=EQ_&OBUfXO%CL!|KZl}8TRtzA9{Z5dg*h#5o(|4=UBXaz1!r9-^R5A z-+}L0;CtAcVGbSk+2DV6_+LUKYD@X46TDnYiX-k7C6Sa{N;%&@f7v^d2HyW}^Y))Q zID6PqT%6}j@?!sV({OGQ$`WtMJ6@W7Iu|s@ZL@~Ud4ALIhB?J~k>_WfKK}g7(;GnF zMCE_`iAWRb{1C6>r87<+rm}-s!-qMXb^17mrw5vx0vAPr=E7^tmk+qub;tXI`NGak>dxaa9NL+PC1?ADT^!ueI95c zrRk@WUh$5cIG8@{v3UO}3FTBba&YqS@zTlQSI+sJIG8rKvysa-SSRp<%OtLfaCgM&m&y7IQ6Ij}>)%W?n>3nR!2grP zcOjK;j$DGD&Olif%2p%48M!lOOC%G|EAevXY>i|fU&x;;A`ay9`SYC-JMww_xiVr! z4xa)2wg~x85nfFTwnru-FW}ERBB{vp`SZ@mB;-r@b5$f6Iq9)!K{fSrBoD8%Kbjdi z_qiF7^N(glE_`WvAsJQ{oya2en!fU(CN{2bsaz&60x0}tK{xDN1p3^qdkYvkA~ z4|byb9>D(!_;$d18GO0o^x%&e{1o}~$PZQw5BgAEdgtlEzbE+4;lV0`D^Cx8mEg+Z z!L0;uJB_#kc-!z`8Nu664<00V`|#lHfNeX52NBv2t(a{PI zDfFPot$Ufn%T}`lNHl{e^E;b5y8>PQT0E(|r&Yi__jNY4)dyLATYFbSe%rqMuD#7| zjqNeGu3N(U^0kM$_SO!67t8Nz2zKR*uKDV^yITC6oxaxPzSf{`PebcIpTul$XzOeb zZ1#6_-qXB)e@lY~Zxe85VbSVZ-*R7lS5HSn9cLn$)%rUdeD$3zKs2}Q+0oqAwc6L! z>i0Fa2U`7I8lScPF8_L7RI+RPI$x2mwms;pZ*C>VK6+`CCJB56=%?t{$DkXXP5wYb z{l}*xfrie8z&`W{RthwYXl)48eJsojv+=%|ZOIE|=;Q{}7kmtC zsJdhHSzh|F8N$l8yhCFdSWYVXSgh)GigkoocuesB#xmr_GUUJ7vT(HP=lFbBgJLP~ zZ}AoQ0M&KZ`+c3Qg#|%hSKXe5wt|(uJ>C95eZb$`;_Jd(=c{XP@zpk9_3Eo{!2H)< z*HG`nQrj13=<3Fnpx#&8GWTJ@1-V|ZLJNh`R$F3oefHv%fPAff{}t z{yI(nu=?S#UiTyrzdFq+LIOd7N<(org7 zbGeMOWLcBURlCYoR80K;*zLV8+cWzB730JUm&+M@4Toi0c5Ya^q1aQ}9_WBCiA3%~ z=V+_&YNVo$t{}#HDdt%ZojNtQ_-Sx^y4u@YI+u75JsWUFwCcGv*VV<7`Pmjw;* z9M)lgJ*>$9XR{guJeyS;;G0;50iMOm4e(6nHNZ1i0pMayzgf&$ zUAF4ssqCx)&S0+@;3@1Sz%hHLv(pCrWOm#Dr?F!O`c&3uz^AZ~0iMJ<46vKk7~o`9 zVSrt%9B@pYB<2Mi^EW43ZGaP5fq~w^Jb+{ROJHdR7_ry@+gQlLE?cb1-w`Veu!V(~ z9?u5}(Qmc$F=;7=0dVkcxfA)!D6?+Dk-N`ku@}18n$y5gf`8F!hJSnKAh} zJYB#fFNbFcnB?OyCPLy*@(|pSD`1kxVX?D{MFJ+3J1j0%0bA(h2}xMVb4tQWp0g4r zeG>mPuc)u&@k&_9(;;Cck8O1-v(L(5r{FJX2=O7j`k)gAIX!rJ@j3-R@uD3&5s+H| zO)vXwI2)2JYuILYIavyxLpD6GlI45xHd8BJdx#oxAyNLJ%&*D(1DUT7WDF+a^VVx# zqGs#|BGbx`c}2lmQ@I4Scfk}NReo#?dWARV;RBW^`vpp!c%4gSSv`_s4)XjI_IwYN zL+LPHwJ9td1fl3UW3Q$HPb*=h`FK5Q3oUj zKX;JLCLMHCV9x%RR92SN3y4z4#xA9@r|+>xSu;Qv=;xCB4sKUYg8Jbg7KS7+C4zfX z6v+Aa_o&|50Z(RM3Z~gZAXV~zDVT02hB4S}59#2(pdHJWG#33O`u`I3FTtKn^`-oe z2kmz1uY%DYc_tFIhs$6+_@tl_hJ#WP_!l|_#Q%4suy1xpopuHVThNVKkAMpl%mhpW zjpHRuflUw7z|+HC6HEh9k1sdDG%)q}3KL8NRgb5@riW?Z>S1yqJxmTna6Mzs^+Zz` z!DPjA%UD@nmsE(b|FR$-jwk~R`%YK>O;oSLUmn! zJ1O{Cg9#7?iS`%gQ{n#!ximbu|6_c-(JkgkFmJ+}TEZf?r=S!^s|t z#g~VJjE}Dl&*fxxhV<=pa(j`!5g%UEl%M}yHk(noY4l*(QpIME);CRpG<7j#4!ZIWoa27I_Tf$tvybA3^K z3jt8^uP^B40an4`-bp6<6h{aR#_*{oe42wM!5DtB37_r=8SqnLc=+$>ZbnBKP#?)R znZsP498Tac51<@QBwEC&+sh}&D^dGefdzj}ZiWgdf zJ;R7cSgeaVgW%pw<&e@|lbsa)ReMhbr%fY$#rRKE`)N*(_zHEY_?n9O&+P~iJ)h6B z(cXN6yy4y)Lx1~%X}AWh=t-U_2rN8FdD2}WfJ&ae;B+VZa!kH(Zx-(S#OTAlGZ>bq zlw|#x)42c2^~FDQGY2L>EbtWmCMQj2G5)ihu(bwemwvoa|E6pFZ8ZPpFjBDAzxhsD z9>nfVMGt@Koy?b#ieI=lJ*Hpyb6;@EGzv$GKgr|K_*uky zfCSLw$#T+iM9|ZE6#fMRtNP*InWp~EaOvX@$(y6`r}p7L$5j2TkoP8*g~n8S;ojLM zd9$Zcb=4lp8?#R;$y>lkKuuow+mnhv>0=ICO58R4ylFJTb^IKfJurU~|CoL`{~|^P zm-VSXOH_X(y{5nO7PeWN4{3hM=Id*!+);6vub&X#F@BVMqz~BdDGAg1J1LD3djwRz z{x-~CuwOprdo5x8WuDJQuh$VRa%2tEe=0a7*BGy8Vb7C{2a}vNn!SlW#S}kN4e^lt zOHSA-o=h=}PxzxeC6{=tFYed(KQkqS=9N9dy;BYJ;ofQLbYIq=H%0HiUkW;O`-g1A zSG#Hdu-eJKrf=_)Q^+2%_STr%`-Ivs$o6h=vcvlN%chWhV)bt|)xX^!@8iMSX6pA_ zeEb}b@uT>x*AK;KxnYs*W4t^ZqwfpeJVo!nG=7ZrvrX;Ko@Zfo2zCoV8V1bi;K~G<3 zqF-)mzvyOD`*v-8!sFozCwoTEZ>5RfDnt7uKbH3xo)|W$517AAqJtec3NhfZ{MW+{ zEXVb*-D89uIY!vN#0WbIjIe!$g5$)Wvb?j5%PZr_{a9Z+vHyVdd_VtqFe!@Vs@Okc z{rg-u%4xka89B})UJmfpASkfPzT3!Z?d^8&Azc@=yV<28CO+L!BPS|97<$2W=P?{%9~GV?a+IOlD5Ojh{b! zDUf2P^{DdizTo6ZWN!sidwl--K&}U|-ZRSMg1X=j5|8tV6n#AVQZOTh`WxeyiRD!^ z>KGRM4EvpzQrVk5%wZSuK|dA_r<8*MA5RG!POmV+8Px_jQ@s0_mc-5_2>Z}_i0pGd z1^bm-V)nT7=&1dpK_2e!c83@2h5XrU9p)==!{ILGoA)vQJWvCGl9+$!yoszO=(l3L zCK%S2eZk~OA>j)Gc8NJ$(I=3<;*1OSHSLdJFSGrM)%t?$b;&3Ef!2@Y z4>%tw>;8dLoG4;^#yL+g^E22V=Oc0LABx$R+h4byC;L-;`zYM$G87L%)Z?I+J1xP@8n4|pM8|}|C1a4 z#|-v#WE_{Uvi~UwEBl`|(Es7~r~Qv+|9^AN=j{K8UNfg-IZ`27DYCNfH5{fGD4 z_2K$GD3k~gIeLWS| zDUiha`Dic~@6W~BKN?(=pzr^sdvTQ^J%c%HA^!di;3cy&Ih;)G;e5eCoz?WwyMjSL zN?e|B@5-2b*l(czQV+I_;K%EezsB^FE%r2t2*u!4pD zi)N&(mh~<0msy7KL*t9=C+g$8IhbYeM|yui?{DGWJRSTf`Jcf*;jadNfN-BKsNbWwZyy7o)#~dl#`>@IxZ|;Qd9c zcXa2IeZd8eQ$WV_zeGo$4Sr_($FXl;Y&|vFzH=R?b^J%$D__?howpgso3Mw{e*)w& z&cDFVpOvZf&&S__*)FBuB+#43-#o`lI(f)m9u3Z>@jKU(lCRl^{OvUjPyUzT7|>v{ zkA6Iu^-uOPj5o_@`x*7i?PEB&}zO|6bSRrFdu@A7*=*?GxV~ zhVe1aI6jQ$Sg^^uUoJAh=JC-NoaY!e)L&wvH;x~(zM&7qpB&+TSZ|X*YWfv0 z{9F1v*5AO-i|NmF-iCN%J|8olx0&we#JNv$v6Cgz`88fRpN;>1Agz}>YVm$xociQn z*;JpdQN6wl`qb_-Iq7~1&gbo{3G4kd;Q4vq`?325o7JMXh zp9=Q_&F#B;nVaG%-v8w5BO^YWt?aeh#C-;Ser_lJw}YOy|Ic(Dxq_G@iTy}cj}-qf ziqM`_-8ba-!Em1p_htBsm|*lLGdRh;9dBXq_sAe`4*KUsRhyOimrby(0B{PypkE0% z1obky&&BJzd3}1%k@{;5*aR=SuLk|B3|if3gd=+3|LHT#P5QvC4>udM5NvfS80URu zJrBzZilRT^-ekHPMoHOMEa=I;=-;UOP~Dde_O`0~F|z--&y-8jbA7{J(0^3ue~9a! zzdyVe1oA#-PVgq)e>e3X^b4V%ub()v2=`fde`e8rRV3%-VH@G^@5#7{k%H(=y!sUG zALI|*KYl>?>A*iL@K2?%$$@Mekw#yBl=Hv8$9)s)ll3p8kp8n7lq=h_-`|r4ysST$ z^gAn?ZsT$N;=CRAS*>IX{(k!TRD1YHP}$lO6XR?asi|1hJkVnBRxR7u}$KE$+jf%eK3Na1{<;;qTu&1KD()kCKC7z2OXa z66E~_z%W8!x}QC7D(I(1 z<$6cbI}&GddWFx-ApP^i=`*5IK2AR;kQf#GK_U1}^$2R!zLV@D+JBVf%aHw1^%F6C zMB?&hIfcLT_AR%~bwsJZQ7Yv4v?TFxR5m~8wvxT<+<&t%{!^Vq&uQR4a{>ubEXYMd zeX=k2CyK|M-}GcU5kTVd5MIPXS`lx`;>!AM!?wyp$vOO5W z7*+Ll?oWj7$b?d)|MB>h*`>kU-!lV@-}hfr4r)qbKDsS14GfiiZfD^GlcOvEj${~crp4oIN{s*mfd4h`HIQClilS1lQL3WynV={>km7G&lB+RnuYOxK5^9g z#<)Htd$^}+_Lw`_Vz3v+hZawC_Tl3{GpN~z_nz_gfs z&OO3z3rvLo%6{1vgS}us=r>*H_kPm1H9eR%RqHSOJ*H1AOU^GNdDDbGX+EU&V4BIE zles<9ls%7&^H*+9(8GSy$bM<0{}&!b{d8`>sd7CD`_Lzb;LpHgy%n?1{rvMDI{V=L zajw5~O&-ke>1a=rAN9#zQz`y%dzsevs6X0XIDS>*=gGPJK12m6h{o3-^qr;C_a(%q zz*KQw%jY|vX?#3l93NQ!V0`H1(~S>4pYZh)Z*Hvd@t97Z@MqXh++Q0Ak1;-U`Zd@S z^Q{;kQh#V)jE`~jzZLe-#zzt#ACPCXz7dbFTi+ONO5e3kK0XZk9u}MX>RZI4M7?~mcyvAb-pbM+OT=(9#G_O(UdGaQDi$>%w=+g8pt%8_fQlhzcrxq(a}RvG_7K)oO_6_`HxIzSNSxyJGU5 zr+BT4_b+Uvc;eFZLF0F{zEgDiM*K*@pjYuf)slt#L^Slq(RYl$@b463d`Xe<|9%<& zQ&s$z{yj?HDTyrfyXf0FuD*AT*0=Ky(zkOWee?VJ{PTbD^^KoRfxSgMq4l##-(vk4 z)3-~_wszj`R6KFwf*z8lZHXv+v;GrNLFs?1aeeY6K3{quL2nP} zSA71>O8SMqaXu9DcgRojhU9`|>vI8f8{Z?_8lhY??B{!Flv{%skL;yh5uqxp}}zdaG(hj6|67oT@4 z0Y~%#`bYms9C02ZVE&0~KHky)rH6r33HZ(0W>Kv7>GMxs9sYj({b0oN!Y=MEj4%4m z`|$ZfM=rOYysv9G&%=8D{vKPDUyzpfi_;;W)?a*{IV;FwpI4^+K)_1FQGC71$)*$=w4L$Ui1-w@ORr0sH2t*S7Q9WGzAJ*;2lg=28~la*5&Kz@M18~kbJ(k_dsuhg zc{|QeYuIZDr}%s>|9lLmH{6HT(?>yndNjS^zO0^}_!qE$HuE>!uQby?HJaXVAJ0rb zoxN`6Z@9l@rhj-ez2Ux(nVzwK)A7Fx`W60C!07Y3^!aPjHy_XNuZo^mtj}`#=ea)3 z^fcahlIPtOJp%^%vYrnZeK!NQ$AIbmlm8E3p`084Z)o6#25xBJh6ZkE;D!cnXyAqh zZfM}6(Ez>Ij{We!hghZ$K8Jtje{lo9FaKNM;DN~VGs~-P z|KAI)T&cPjYx{RU{O-x&=$pPbzj5se{2%#Q&$aGP-}{xH`QCi*8+E^k{`K1*{QO&Q zXIql(u3^60DCV1v7>uu*!Zy;9>~LM>c#CU^0WNmYA&MnA1$a6RuvtEVO-pggY?syT zO1a7}PuT31VXH-b*+fXHH-7lyoa@Bu+YI&jAp&DRwOD4ZLaW<&t0Buklv-7< zJ1y3kpLSej#Uzv!Uu+?3$;MA%|2+=AU;;dCTgR)vdICIcW5=uSod8eU+VSd_Pk^Uw z?s)YpCcx9Scf9)56X0nZJYM~#3Gf{p|76is=6wq8$G#WE8H9@tHbf$PaXr2=v8+Y< zwJ0yHv%BbJ-Ex1?FsrX2`tL;tqRfjgXB-eTbczxeFCCMPq>9V-xrl%9T6T+WY*^^T z$FjkVHobxw{#(izy!7vMuu-NvEtWDGUB%Q9o-Ad%GsS2`ZIqZ{-ZO2)9ZaARXGw9c zy~b0`4HEhuhRI47J;7wrR1glMtH?$uzcmN-v`ucXSorUdV^LKb1EbJJE zKfW9a{ZluP_z2eXPxL0TaN^kfon9wCtTYaPqx^h?>-i_4>2T86{6+u9;g7R&$j{$V zgSvSH(DN7lADh4E-#Gk3#D75X=Mh5BKiQki!pURvkE0*%=d}!@DVAb>C}x_Nj~%q= zXNfqohyfpx^MTQaLssg88i+hf&6~x2l;4uF!M!n9^izpdkC!$W6EA#t?D{EQyp58g z^Kc9cJ^xfMu8pLQU0?d}nC&Ycjj7YfU)o^I`pO5#sxNFjw)}3mU)Vji{FA(sSa_15 ze(}spF6+#5i1fvOL(M`qvJrxXOcd$eZhJ|t+d(g#KVVf6wy9NzqL1Ni33@z2C{HDe z91creJ$#Ozy^FHC6-6bc*LX5f?BTv^<|AcPvQk_8*~FXL!mN;Fg%mMcjMpMXqmQ;l ziccSHixjaw+7>B>eY7o7#2K?i@)qSW<9U)A33VA6S6NyGT-ahwM7#^x2kdxhOHDX*@#7VNPQku<2>w$+30@lEX<#4o5sW+*bOA zzYxhWDGuK24s$%@2*<;R++hdRCmB+-`h0}N)0;symLTdvN8u!`ZcO%&Q|ORnbH%|A zIE4;Lws?4!By>o!C2RG^wC7l|#ckW0q{$Yy?NE{?o33p`-<0jbI@y$unRPdoY;pBA zFoE7!Ot!ePqi;$!o$R9`sFHOo*{r{hO?7&UD;xS2N4C+CTa4%>Sf78E-*6wGy(*vA z&e9G>&1?9IckDy>jKlZ`g+|Xqm6)3vF)OHdZZWj^97FdOL@IW#wa0ev_c7Y77q7{1 zmAFeBDlb`WchHOW$XtR5)ujQhU;XehZPjnHticp%O|xc*_+jx_hRKapG!d;+E7rLw z_*_^C3ckm@n5^}wfTVGI?SiHfD4WGhLshJ+Q;McD4_|ayZMcH`FkaQW~#4N0IWvvcN+27KFHJ;w|!zO(tyC59W9kjQj^Q`90V< zp?|qXJl^M*sRCy*rd4rF=f!rfN$2=Q4Q8F|#90XG``{>j;}>p;Vw7PgftQR0EnU*Qc(eLM1M)%2d%=}$CpY) zAWf>X6G$~`jQx(fx-O~0SPLL7dNrvKf#O+kf|0L9>*DxQYmwm~6S@*!KIRz2K^jGC zO{!z@q|%*eSupRg;#%vdvpW4OMj_3zm`(yH3R{MWyJfKxr)CmHNGz^%@^d}Lw%6Ez z=jVDMh!X;Uq|Wt5;RiUHM?CX+;dpy4hPC>5czyyk&d!yeAdQ0;XS`$Yd}%*4UiBo3q!NL<#?zRy#SS`#$>3IJ-!bD4#`ioU!k=|HyHo;?VXkUJUY_8 zd)R|tnfuh=eI=un{llX7ep7Axn}ti}6#Q`Zi;t$dH?+8x;q96mzum?af8)2?P8)vP z?8a}mg?{hfZtJMWckAMH7Oy|XZ?|>W8N>SvEQCYF%Xrmn#_zHrOcmhG%Tqg$S0mqz z{8RX7rH`?STE;we`0QOHCew`}{@ZPs-o*uF#BKWRwi*c+NH`>6ns*3)Ov0-rJRo73 zcld9=N!Tl4hDnnyn9{sMc#nk3CG3?j%{#n(30FusBw?C&c>5BrmhgatY2M-OOSnnG zG?i2P9f}_J-3exZ^}n@7{Ui8nw!l+=2=>g2$xm?20s)h}1Ru*2Fv-X3FA*@w!|N9d znB<|~ZcAG$U=o3TyR87W(96>&VI|LH2`hOjFzFFJ>67@MlCY8|4YMWTl{_61R`RId zZo{{xGcP_kPQPqY1Dzc6VgrQNWxP(+!cWldSiju{_Iw7Ue5~JPd#i!4bj{wvfB%Gh zj?!P5$_FI;3fiQESz8^hbNFEg&qkF$3%gP>)2megm5c8k5OlY|Jt*BO z)?vi&Ax~ziziM?Oeh+yvQ~L4BI^y?`;Vh;keh*puMhXpE4bju_QEioKsGA}~NsN|I;br?7LB;7$PqEn7|^9&w^D?AfG*H;-N`aAR4-Yuf$Ors)i!WJgMekX^L_a(gZT{J1L?;jMt>; zpC)FyWY#?GgDQsc1}@VqVh|;rVtbPReAc?+6LP3X{MlkkNNj9gNd{^?jng@_%Zd zl1Y86LDu)01@Y=XChIHzCHW~Nnd`4a0cN8P_>V#VX%C2oJtEWooAtj9^*98@K|(z0v$wjFmDw`})U)pzfz zt}Lu;t=h47<(=*R;?m8fJ4>4CTGw~&uC8pU>)5uvWy3us^(!mei#Ob}wYpG;+uPMq z+qw<)DmNAHs@k!mV_R9v`a5zS@p$o7WX>DXcFGHq|ZP+`Vi0&Vt?BRu(i=m!ORX(C^-{ zyK3eB+U0jpdxg7N_pH0Sc+2v-BL5D5ajVv@zqqh{ZPA`P{Uuu@&H5EBWqUR@byRL^ zDKFY_N7&o_QudmwJUb5kywGHK4S|^|>Y1m!aUb@SVK31-2DQ#+BTeYXc zUsBS(Zu!n;fAO}>CCi%{>x@MFrfjqkl{8fc5b!`*Tb#&KP6*ktFtqD-S#thb=#R@Y zX0Ot!%8uHWH3hq>Hg@iwh_;o}w)?9p_wKG*HGzFa`!rtS^sr^`vwP>7yXz+EciYB& zb!9szv`@)$LAR@I+XVgD-BXKrR=vDz0=gDSSF*3RsAmG&4oSP^?p;;EmNE3M=}Gn- zdM%ttr|asnm|3`blf;; zrW-3R8Ryr;_ zj_cxAS9E>y)fG)B-v&O`%GS5-t{%%*H|oboOWjy=_gbu*3UY5w4#y)y!lqLy$1#>^)Iq;d!pf82_g}g^8cH z%vqySoD}ceTF~>I>ZlqvUAqb?=wtF2%W=qHN`jttmI=UbH2_`Gcl`g+jBlatEgm0t!Q;@>BcYW(6Xg_l>*VGWY1TZCt#uaV_zIhL`V zlFunwuIzF~^;edwar)1)Uup7uR*la~l8=|ZtQAR>KZNqh>VzBR4@j5k1Ai*-lkg>8 zi?QdBQ+Ws%kSM(%GWIrdDleCD&MwPU|L99Om9G&hL*l2J$_r%q43w+7RCJdtpNVqW zWnR*&E0Kz+J$ z`u<>*j_b&o@?w@Cjs^Y7A?>*-(9 zmFxLCr85y9J-zs{N8vyE=$#|!7lU43zDYX0S^w3t+^qk*WI445%2-@z^KJiNAh44(Q71+*+mgb>%YDa1{SicRUdvy*#%_Wtq!WU+kc# zp`hBPd5V%M_i_j~`DL!k&&u)hn5kUF``1n7Do&@NJxYp?ipK?}autu&`M07!EBoJJ zqF3?wxT##lV!9A(5a?`fZIBc#M!QES+E1*;Tkoi(Yb+$2l007|l&hTl$VafkOom`Twy0-^RQnms|!$9X}A@ zXXQFaA?0+SKt|^bRHk1O5YkKkkec+13&3h#0OLhnnzZ4z(3ZHT#|j7ZNr&SghW zJC~BJry|*+=D!=>m(D`Za5&%eA@74P96ELG*yE=@=zQYT$>WDk{h8tY9I}UT&AmSU zLiFgdwGoZQSr;@p%+d@Q=j-ay5`IW(O*1#CVF!G<7nEd z_499h^!TZa^4j@5XC6D%{=)I-^Uy!H=a8oqzk3R9lsXfT= z(eh)_j$a**-Z$~9=<|nu89n#>N6`=Vddg zM<->TGcg(MO_+ixV z;Zu*k`om~?@(-hHU?&}NkKTGF`q-;vr+&Vp>(HsA<%drFqU?v!N6U{#e{tqy^xVl` zMxQ_aA#9$Brj~D*e;+o|SJ%zIj`AFYjn|={w(73=gRp-N?YH{O^U*Z4)j_mPd)LY6 z>u8fV%AmV?9O+5@X~Iiv%hA(D zN(bp35u)a&;1#ouwDlAC!Kd9#^S|wDgkO0-zqs#}=dAv@kiUKEbga=nNB*UK>jZr3 zui$$(eCrR}w<1^Yts__Qts^$yBA?U#b;9Of+8?jvV;A@RWIpCx_U2jVEpzX`>zFHY ziL5p7&%bjz=R0u!@CRb9ac>jPo5 zhj1OozhU+J7b`vdsO0Wp`^YRp$cE-O=f z{_(qF4zhC}+-}T&RPC5Qdt1^x`4IWjC-BXW?{#t(MDF7H7R&mQ ze|_2AFn@7T;|1!U|KI9spfTWwd_{2os`S>6djFF{WQ(Ueq0IUY2Th=EMH6omt`y`z zOENF6MYtA=Y|8eMv?map2#igOtw2f$H4r&Q2JVI-oHscghEoVvI=WLOXQpuB@jYJgaNMWBP+($uGB_0qj@GW*JsudJtg7^xyn*4$T4MUgMt4tC)=pM=Jf7NhwUz7m zx`H5yTmi1SstKP5^_2Cx4yrLd;tvLfuAhno1Ji-v3~Gd43I@im3vWX`{MSWAh5`e7 zcl(Be{_%k99XrEck-$Ni4f@B%b_e}aBOG*7!O75MXm$$fgiTINhx`*E&voFwgZ`=U zKydK7=w@^c*TvqK%v3`scc7JV9rU5DPNL5;_qqvT-M3t!%XP4tRIFpM)|-~=2)z`! z;{C=lGA5YZ)kMJ7o5PNcR2Kb zZ@9Y3H$66aKNiWD>~5@G{=k;}31c1gj-7GsYnQM8=0}h0+pq(xm>eFS4um|y=bL~# ztM&J`&X!J}MBw$cATF6~-HWOksa3^-z_EgXOWIJ1Yi}^QQ!v8rA zCJb?tIG-z_Y&t|hdK{a7C!YXFT7UkS`DvOo?|%#al5C6pA_@Hoib#)MB?&Q3FM{tr zl4bH0TfC{{b=F$`4ERo*huBjLsYEg9nG(wVZMtV=JQRtPij>$xkZL+ZM|shX1XHY|`&WPha8o=wWr$aWba**H{@>0c606m$M$p3;cCUDEc%#>tF$ z3fHgEA6+{(H*aomH{k$-&yfhHLmuj?T3o}p#8fDZuxP@3;HJ}&(J_B$baKKSnw%V) zE^}j$3HYZ2?#k+lb!F}ybgQVT7ma&I$A;X~v(ur#xO;r>bjUpv7#^JnxVLwAZ`tPC z(bBVRb9a}oW#{&muI83zQQqyYc9;8ugCp+gabKlpz2A+jsjPH9gzraeW!)nl2H&+d zh0FTS>3MOA?pl6U?j~GIqzpu^>1fI|?1DeQ-g=kJ;~P{8A(Dkhi1UkuOVLT96m*eE zyM#M|c8Z(?+8`gZ$UCTJ7Me2w?ZiQvGe0JbEe_%IIUTOmrQEmVCs#|Z(o%6GU5FF7 z&g1f+=n-5mW}?4woyV1p16vO+qaAfDaEW&ipRVi-5kUc{@9A=_kkR4HT_;>=DFUWM z#EIux#ovP?zHwZ;0U%Mi=>Da;UsU%maDRmWBn$`KS2`A&nH%RTQXIJqU*rF;?)&#S zV&ZLBBL_R5C4M%Gfq>E@xQa7H?yQQYBG3bwVt*KQm(>^Ibe1@Fe{zfuNYh~9&q=%B}d7jz|&clyKv45Wt6YqdQL)__1=%>WQA2FRGDl$bm9m47a`xRFD zS6R?Ue&--S`o&pdRoLjKq8s9gMdIX>;-Oi?;2|Y{MTRKu7A4{31%$^4KfP8g3zx^T zXM`vRkIJ7EhJR9up~pcl3zsooMm)yz=*fb+8Fv%!J(4eCnwm$_yO`cZ^ecpSG2Lb2 zE16zt(#1>{oAd^zc`Ri6cQH+e|I&Uf(=;wg`|FurZ{podcbjx0(~Txw#dMWPS2JC0 z(k)E45Y6QmOcN>d-^lbvlfIkjyG^=}={}S8GVL|#R;F8tX8UbSx0!e|(;OFE|0brJ zO#BX}?=b0hrV+-fd>u@85Y6_tFuldZJDKKVezw1f=}jiSnd!|YUB`5tN!K$?>i}tg zE7KURRQ~rceUC}^Fx_L)+nC;F(%YHdZqmI>_Y%$JZ(zE?#P4MKPNLzT89~ARJ~i-( z6aAA^5>DkIH)fE(XStSz7qL9g1CU=8b{RAz%XlC>nQ@L8vKJNxhO5|A;~%-it68EZ zn3RO?H<4iPjV!Tf#^vHcj#!|V1f8zsc#M4koZHjI_IQW@z4+VNPL98n{aNC*LePty zWG^Y>$rLH?WOVy@nV`L(kpt0ZiFSd; z97A*|$!CB@g^3;^It8@bLg#=kmGl6`tGk@W2S}j(yD7fQt3b!&dxMv78J|z0{qFJ- z9*@uRGIM<4_NZziTE!dWtD8ukst^H+L_>L^H$MOA2ma#0HltnoE z1M4?s60XJrlqWfpaP|qtn{OgqjUT`l-9$M1KjW<#gj>d+48r5%PhC3UD&C=ApH4W( zJL_*sBV3Iaz&EE69v?5Zq!P~K%`4M{f-pEdAwr#4f%wtcn5xG zKH(hijBn2)oW~2c-XZn!2!eP6K5!R1*Mu2^n>uAuP<;|a=BOuiVmXKbL@WE%Dm{}tx= zFwFJ4i(*yh_t0!Ma5Y{qZq(K$=r0AX#t(_Fr&x^ZmjHLm@k840rdW>SrNGtrBJoD@ zeM@<(Xl$_1)igG+KDTdb3&mobHe3`darsmy#c~{%qjelNHquyNq3@=#z#`vAV~K_K zdK2YaDOTe8scjT1ae1SeVl9pvizwFOcxn^Hs)bI5=UC3nV;P+6hSnwW-H4DCtV$FhY zr&x{SMlZ#x1#f6br0=9yXB_kE!3R?fQRvPP0-KhSOi>hEVaOha&>Mw>uMmY^3tqUw z#7`GuJxc0;f?Li1$`{1b`h$AKi+_Y``x{3~k_|0`nR3>Xyu!=zv7 z5rsI=AbqUgY5hU#fvK(ModWr-3K#TKjw?7m*TP<s}#G{ul= zuR5Z6el+d7P5YH0_zx0ougL!5Lxy2(zvcS~r33-&Klks;iHWly$=`}sh(A8Kg4{{g zAO3pq!9w`Eiie>Jw0|KOD!s85cm@1h#luh+@QT8U1bl6U>Hk9%PirgICg7g6rhhZ; zS!>1v+s`NeH~rNJFBi`rT~jZzUkRcju}8D+(H(e4XEG zYlSh3P=`FT;QuQ_`FUgyN@{-1Gf18r5%ybV9sCd0!oYg?4D5#sut7cXU%8QhufqIE z&pGD&n3qpGcnER%+72T0InjC=`D6d<#`MSK$Nb+s8)8?3O{X}QWf;LcDxity%(All zKzDG{gTLtMWkOv{Bh zUG!Zek7J)&0OMiDOkR4TemaJ>xO~S<+07=7{7S;KyihpWH)Yz8?Y|UFg!U{6({vcu zFA39hXr*bovC=f%SZSKB70vn4^1zDI^1w>d^lYVRIS{AYgJ}kZDvx*=uS>!-ohlsm z+k=^g+r+7UG#w}C7f5{wy+?S#%0xV41%gc`#+MhHIP7=K6fZH`kMT?_H{mdMhn@-M;i&x&Unvi$H0F5k_K zCXVvA2dfNP|Hbof#BwP?KPR#NO&Hp(_!2|rukF+FMB&^%1>Z<4KP^|{`tZ+Xi8w7^ z;&^*-gF)-*c=>2~WwoDkOQJq4uj2Y0Gs~7H;*Rje9Xx z$IIU_Q@h%<2lHfVFQ@Pdt znl*{~4G^^IZ#OU?Xnj{ncmtOER=m_GjpHc43(JAH{q|t9(VT$OO2Vq&XuzS7kS)%e zHU=xPK&15%J}`c@2e%ps7$l8*R+WH?<2?o(QsOv2Xb)~N5P*{A{@G+;z>qZKbq4Rv zWqUxr)xdyc!S66=KcU(SxYzK4iqjhnJ{uu@wx8hNkazL=M9EWpR>}5<3G`Qcuymbl zFHNsmBimQgd3mxuK%@Q}gNyLZg|)spInAIsF`j(;8W8dJC@VAlk?}S8GW&dmZ=EdYnalE!x`isVwVL@lbySk*m zY5bPkZnfapAN1#7zt8&vu3yr6if^_Lf7lpwbz_2%H2QmEu(XZp%Q(|Z@UQ&w{K%e5 z?`PQF;+rxw&hpvmG{435t5Z`n&H3HFyg<{?Cwo-ic=^6jPwmP2uS(p5rG?x-NxBrP zJuAIb==FD&=rq@Ba($K?&W(Ko_6&8N(gOTEg*2_-&eHnQ5HC7FJ3ym6OTziNG``^b z8#+%GMld(N9>n=XO}HSR&WrW=F!mD#7a>2@e14JmACTXlC)ciOecX0X_b%+?JU1Qt z4m}h>9+LF zXuq5rE{OXx^p}JSxP94Qf!72J3*>%Fet%ID&MQQ9kiDeguX$g>)YQGQKiUpP$TE`j zH{gYgH!3`JOu`nN!bSIITWaVM{4DZgoc+h3szkVaZ_)P~BMI^=gL$%kC&1?ZD=2ic z1jnBX{!>V8H`Nm*f3(n_()T~8e;(rDK^d>y9yEPKrEE z2IVUVKS|nR>h39~ssE_`@(73hMHrumMso4@OX1vHq6;pdJX9Z)m*+zokGOwx3%ES0 zKbHgz3Lvw-ZL>6*Qj+7%lCY)yxqXO^JjmZ)q5N&La*^IIC1$9R|^&*h=ngpw?O3gQ=g^Oztt><1Idi~Nd0R3AD| zK&vfSUb2Ox%3B%Y;a#53pgp$EUdZ{M6dj}CFOmb7ICAvp(a%nQ7W?dj*k8xK`}XfIu6Lxn(k=*Ee>Cv%gCjjT?RS!Plwnqn z$X?eR)YYiVthylXU5rOzEW)Y6 zeg}V-PBi_C!4+}RxspCQ{E9dl`-(UUctzZe-frt(CjFaw9`dGtI+MLN|Ae@>$Kgmd z;7u$Gnl#8`8}a10#QQwpHe?K^UG>V!Olz=ST8DV_Y{m4mUB)h_aecSzX?cCu_&@R8 z+C%%7**wVB!y2d7=VbFRjnjCg(?a(3{srd)2_B;L>2!>f&4V;fV+kiWqCKNA*^!+L zpK#+010{n^@)8<1Dao>s6pbNt*Uw4|GF+b&!ma)#HLV;rD7G%5QK>`{<_P5!j&BqH za=(CBr{wLo$nBEjqM^%mFFw@IAcO+IA-fxJr$o<`D8I-*;+GIp(wxxw^SA4Yhk|^? z-w#~!+kk$zc=Va#XaC}vrQ_m9YyRQ0oz8!MTUn9k^%XyVGAm>ASlU|T@iqJXz}98h zapO9H%Yy^c7jd1(L6RE>Q(JLGaJ`6&zMP@o4@{v0FZcTa-IH`*=(hpHFXIC(a%czt zFUM9zZy*o}9+1#!CnX#I*COS0&oTeOCOS0qkBQ;V)|NcoU$T&vdVAyaUp;^;g%u!V_({W!&|L02YKhXJ`CE4|F28JcXB`HGZ#z=0ZKr%EU>*DAZ z+KGm++a$dsG*ThGC2!`ZU6%ZX{|CudjIipmW2UBju(x}?()ZemQKDSLE!B8sNoBX| z<4b2sdZ3dOKAV-kM{#r;+x2^ZKfHi`Na>fD7UhFozeqO7l-DNrw9a2O80AlU2D|?b6p?@eIZ zoDA5ruutW0w8`oGe`GPdQ2$L~yunmK{g-VuapF!%omR^jI;*DoCkTPQ8x2SKBPBq8 z-XNLp$ea18WKDWI^bahM{~xOgd6c9)E5D@qL_q0#ZTbI5>#Gb&pF-Nw9c+0WD)s0B z`M;++)Xh>t*z?zZ)DHdq(D&N%PidBl*78&ST2K3LJ{Sw>*D3u?mi#Fs?fUf6Ow2k{ z=;nm>k0^bmZr<$ezYY5TVuAW={S>8OzykHZJ%5TDYE%8bFFYos9Hp=CTA$eew|eA~ zW_%{(2BJtY)ZLmNl~u=ulDg#Pr9vTD^_ZpOS;FS0jxT-JdRkz2!?qpO_p~&~)YREn zHMV!Kx^Zk*^|s#o8pgK!`-b-J+SyY%INsO0r{Zu=V z8trUqt>4q%*FQ4U7xqZq{^9<<%KL{ljU%sZolTYXBLm|#Ged2y9)Dl`UgWcVSM?6h zKwIkrgR?dF4I%$66FoJ9ZO8-ijblAy0~1{{T&7JsM*R)o?d%!ZH69*={q~V5kYFt{L^NzM9Mc%6Ju-)}itG`*!tp z^!gjd+s0ZudWZX|Z7V06t9IXWU*izfZF<+vt_QY^w~q`~?HKOg)HR(@*WR(cL(RK9 zW9{9&st(QlI~%DF7ApVbeGNlx18wyqL(QGD7M-@i>YmyDzShu!dYkX>MSJhrwXJ5< z-&eB-*96*q&*1o2RsYTo_{T`|j-6ee!SNj<1O4b@wE0l`wpLjmy!~LTeil9g->Dhi z)mO83pt@&dsBLU!V5(=bWApv3+xlueoey-NuX`TYHM;ID@=4$L*l5tVXSlw)Cg8yf z?)(GQc*#Uv-C%8?et4+Lzpic|us%@dlX8Qj!~Vg*WnQ7To!_0OU!h^8Z12glK=RFa z7KY_rcnfpoOZR-00Z*;3%18RX?ZLePUsEq$8yN79d%}8%<&fJ~{I!1bMe!Z(ec$;G zYEoHI$@S5Ly5oT!?4?&Y3Ni}mS3P{O06t3e1MX$Gth@LT8n`KK&sgsLp7E6HUnT$R zlt0P%e&#dMGRY6Ifc*I8RpcqJr!4a3`gl|3;q>nt&+u3YoaA1<203g+`Da>MwAp;f z%>JJ7eSYPFOU)q!kzG#BalFKWm)dZWZ?@p)HNGYDNIGTkj2y8^ZtNQ5w7UKNEOcGs z6ZJ0ziKugd9geJrZtS=`6~V!~1<8A^Q{Me9;`0o~4C_3z(EB*HvuP+TkQUk>OL?`} zv*|c3`D#eY$IFw*r8VrA#oim%)GFID;V=2*wozftA%s4hM)&4^c& z9k=R0&F=$$u{HwP$3asH!9MX1EBbFFTl@fb;?shb(yQ_&-o%~wb;_mH&1{>SVNd%G zsp#$7EjGSK*~k_wCP?8ANx5og)Mxm{GrS6-?H$n^`-x=rerT?isIYr8&Mr( zuP>b!EM(^?#kbe@Pc3$=<$u}2x7PP93t#WCD8KWU;qO%OL-kT2z~#GF@o%^B|I{LH z?cZk=-(LP>icjr8|4^kgS~M)%-0Md27D3f2L&5SHGXgp6?-x9c#QC zR(yNByr}qON0){2d&k13eXUMQ)wn?N^#8g#9Z`H;U$+9DReW7v&A(ugx7POy#kbej zr7B~u?_tH)`Fd2oPbt36SM$$W#tT5<(i0 zH&b5E8&BK#*~*_z*!X%LIBVnU@m}xab-sGMFGcw%X+DNnPMd6eJ>E~*_f)Vq^xpEeuHJIIKX`DDlxjtL zYzu@MLU`f$z}`@RxKbX93EUxnb6{q4Fp$8fH#Yh!0we16##b-mF!H0_7a!OLx#ahGG$)Fbas(Fc$ zp=;)c4yzVlgQbP2cJ-2iJeZ=K)M6YRQ#grF2-5x2gF-gKD_eB%cou zPfgz@->aUoZCf-wZ)w5X@IXr`uV$$Gk$dIC(`ubX-wV>br=~~qyy7pxr2MUGY|`AS z^7Rs7NjiUh|DMt(+tzeg>FfQn#&rIg9>N3h>}^=rCnfCicKY9{{O$T)rN2SdU+1UG zt?9+6yuD8d`}RN65{Y-a?0rh#UjK;FKWtO5mp=e}q5i82$b8SM?5Y2#NwvOSOFxE; X?Rk?*HhqzXbXYYE6@z|nX4n5eQ3QOR literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kfafff4c85c826306b570c6022b95eb40.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kfafff4c85c826306b570c6022b95eb40.o new file mode 100644 index 0000000000000000000000000000000000000000..dffc2b576486af04adc25b6216250ee3da190df1 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10PxFuj!Y~uhwjLlP0}=x9+RTcU)F6Q#eOi)f z8FC^-)-eWy8M71nOq?YXPnMP;i z_xtP7eeYF+0DEG?Qb*nQ*8l%i^}nkAzp7iezr1Yaa$VDyDjIv0#cB%5SY}V`-otM_ zEESY))?M9itNnSl+xEZdg|^R5W9;ytExY$!yw2IO=lm#u_I54C9zONVK*jk313NB1GqCdXGXvk$J=Z|{EMA9apZs>X z_r&q=W6tk{j~_S@9-enH+&k}7c>n6t;Y<5ZhllrH4nIBbz3{^==flTOy%$cI{zzze z$AN*TpLu2=b>8C8?B?%;FU&n2-go(@;isK%g_}-X3iqzO47$Ql=-|PD?I#Wl?3;Hy zTygqDxT*DYxc9(Y;o*lbg)i*51e!&ma_7N;3-g{F*tZ(}?Eg-9b_>Qo^;Wq2{H5^r z%QW7fhC=f&-s*$s=R|l;%eTYBz*n3<6`p!y1?-SwBiBsVU=jm{0-lg!NR7v_CCyl>uH;0tZ0ySffOH9_-SJ8PTJOSmmMD~#G7IAQ+@@Gs#5TQpYq z27Ex%2sfd3PA?~zX`|LeE-pYqq6`QY_kKMX}mCcZuN zkbd~WKNnwf+gaMhUlm^)etB8nuh+7^RLqaFGugGFAL`fMfL{-0vukHdCLA8D*AAck zJN^35e%43xh-v+gt^hvr9P5kx3+ub|s($U=64rO_0e0Thll<)AvE+YcKBjF*3@yVZ}>+O0ZR-GJ;V18BCcrvapg#T8T(_mN}uZq(fC9^P4o2eF;e1>aXoG7)sHJR?qjY@&$X1f zo>1zdqLA|o#UZi{*@Wyw{k;plo`e2eh@XLEPu~vo|4m%i;8lV9Zd?!GdIGQih5OUE z{u_-Cd%gQyLdXWYD*v;MW!K(?{$11ckPG&?2)`ve{TenPn_qy<&n{$rgRmdjLHYFi zYgu2og!M7K|5383vj3HD!B+6)_u$8uVY7dO{bK&W4qrJ0yBh4+I92F_{Pg3>qR{oE z;;Ur8`}#2J<9Rp_{FuM!?RM)`@74UH#$4=5Jvu?rnWj;I56U{hR^a78=R0uq;FW|| zI&;ymBgzh6pw-vh&Z>aPC9IB_r>EkoYs@y*rs+1u>|?Ox$zpRdJ8r)$^4cPW(AobK zGD)&6b>imie-u}J2TSy@B`g6PO(M$q?TzgnfevpS?$oubMb!DVw!N{nA;{*pwsrXC zw?00;JJ|dVU$A4o5UsYpv!mJD-d^i%Zr%HwAwG!LmYi(}}_&Wkk3k$Zj_*%TpP1{=Y7q{it{`+DuKy7|uL2W_d(%SsIyu5|D ztt}{6l=n#f;w81Ud5}}cbq|Qq-slbZ8txrI0={-%;BiOSF-dWyqwQfMhJ0iNNC_0ydr2M32j<&Yu_PK7% zY@fH?=gwa^f6-j`db-RnT*`_&o0=Qk?Op91z7}^&XM2ac!MC-k)#qMUS-Em;?fPZa zYs)KF*Dl+*ZrSS6Wu3jvBe^_|k!JY)%zQDoA z6Y}UjUEJfA*)wAIw{o0gYfNSKt#;T2Z6C3Vz8|2`89WdtLs!z7UDoYn=_^Iix{XFR z)OF@I)#-yNHWl18q4Yu`r?x$$H|WGOpNdYxD-&-IoW(Vq*Dd-sR5Qs zgZrzl*w`gJ^K>veB4dhsqDC${isXGz@7pg{q+PfktB%5U)H6w zEyRo7Z4>-FfC|cVyv{k<;eg$a7A(?#FpUi^V7Wot15a(wXA%8i5_{tin;A^7{dUru zJ9vFC+g=#dZG>lx@Q^P!K9Y@E!bE5r63^a z_}rjN3;Hl4yer_GK>VhjjU=&wG*;Uzg^EJ<&hx;p9$E_5eSH0XBXDA79jab|pmE2LO-{!2dD= zwg`D-WTC(>>(ba2Vn7rAu_B|T4P>*j7mff+1>;{J`We%&Mj~2_ zezueJ7m@NU>%uZY6I=STi68h^VPM7#ZZA!YR~VQ%o6AG(Pp4si&W*6MXixJgZz}3@ zBJ7f^*Im}82j|YF@#YeLu+N?@$6P+&9-NxW{@Tf=2j|Qr{2Y#7z@F@4b4ay`4`R_7 zDgwP7^jO=7{+N^F?WhZU3Gi;<-6p&nb%9?7{1o8x1b#@gpMtu;e--#l;5`CAD~|e74= zbc%o6AAe5$uQ2uBvukS1p9T+NeM_*Bzwt%k^aA)7*88Y`k^f9hpz)@1|2ahVoEr77 zL8n$2n353nSF8uqgHxsuePYbtreso```cc$ccH!Vx7DQIDK3-0Ie>Qydmfw0@eb5Q zd-6v+@Y3Jl-_Y;$sD72bi2<_rJEU*AN0Iub@$AY!2TP!z4uO~Rd8Ga&TDFJ&r<1ys ze~|w9CsKsRcx5jvG4k$6X?`2@&F5+w`D=(>r4R=HU7u ze>o?6GoySX^@c44WUnu@w0UhG!%c@SJ_M_fO zH?jbKu#4%m>q#scHx}}%1LSX*Kl6wmRZu^^6ZU%o0Llm8KNNpoA%~>) zACUYEC9I&-UXxk#8vZ>L}lFIDd8h!|@{DA^W97*f|Wy{g>=> zbN_^Y^L)`E^GlM47nOemY!1FY|96!?CEMP?a;C<^dgjDdLFChXJxjJxwBh~%c?r+g zy9_@gc?)}h=K~1;N#AWt=J~OM=g0Qkpd)$KKSi>goKNH{Z&X4) zv7V#|dJXNNZcXJgi5bCU)WJ|c`6u<4g5jcgdq!{!>P9?GUn4#Nbt66z>6Q_ngnA5b zSL{BgUupQGfg}jXfLYNESI8NhAIW zdq6MRCsDqGeA?FkBzWcPTbc{`nLFAa zEY$ga6aB%yndB~hnh%h-FqoMcjb~PVi2ojpcVm8N{X#$|0^#3ge?LU?$v7Xd{+Rq7 zc+-3U-sJDV8|DM_lN&VqKm2_v*_Y=#;5Q>^m=BPb@KOK2q)|Qv|3abo_!7Q^i2RGW zQ78SLTYx&!1?rIGxj_3jM?y@$PEZ^ryi<3;nPt!;xopX@#9dB;UO>gSFR7pOLZ4G5c;uUKIe{-kMqeT;>(3f zb3A(q`QvVuO8yOh8pGbKB!5b!7XDJdUBr_wqDIBAUJRkiDJCii#JPQ+g< z*R$kBl3ehY9?(fpLBa@CN3~ zZU>GEP;6K)PGS7r!XEy*_s6hbQoMbgpYJLC+_pT}uk-bT{Ee^AXH53A=4YmS1ok@+ zU`SA;{jBF1SHNHAaR-=S8zT!xJ#f$V)36;VZDyhqdnH6*nVqJGq2Z} zuY`}T*Q4eue12Q<721qb5&p>c6UzRuC(a);CjuLBVZNXK6!R7Ghq(F5v{CXE_q3bm zD_J~WamUYBAWxQ?;(Ih-2^#W^k@*VZsXHs0uM9zdpFAJQqV<)>OY~>SM-VSf`G^DK zO>=|aPmqsfi3OkH^N9TD?#)NYfBwJBN9LO1*U0@hWfv%N|9v3H>;&QY2hQ)yI$s$z zUL7^XE5rFZ{O3;MRdoN2_07EBUVzURM%!=i=HjUJiN|L;Qlrm{awwiWz~}qK@sM&b z&1H&DSU*ks>#Y=@68Zj|$1~(d>AUSbf5LtsjlF^Vb$3D{#q*@`X%5PV3WIhE*9kn| z#rh}Ci;iw!@!}ccMf$(<{@WD4kWc9W{^>*X^9RC5;~Dr*M?9nbP$X02cefwUO#2OU z{KELA{RZ@Hif0^e-hY_(8^D|5+YrycKVdw(V)7SjJj3TnG5dqxbet#kAC&rm{ji^z zNT0{>_4yn=Z`?ILwm+Z2_vifll;_)TE&%;R!+!h_K5wv3>@hx`#W8L3k{@%iV zdy+A}6$Z!Y3?E>L{UObd%yHvkA+i6?#{N5N9DP1#*ndx<{h}e>IcY-@+kZQqqwK$N zphD$N;$7t(?7!2oT7p5bUqyQt>e2lu^0jfs^CsXkjOS70|Cy9;iv1^_51G;ZC*OaQ zKcPh9-NcE^jXLG4=NCYqF5RF{*gsvj=+m?x89jgfBhL3r3FiD2`EA+0+lz0=j||@r zNMoz$u!$E#K1)l2+)w9sun5IN{)Y7jpBMA#Lm{gts3`r~c{V7TKQ`JO9tSHF?eF2Wjk!;79Cd(E-kX%DO&^$BH9H~jv;)JDF>Q7LN@{<=4(r?)qSo%!*r zfB1dm9XnfAvu4^ceN9cmni}jz5Ym4-i{g4deQ{LN17OoUBJqwQJ-E}xB%(cK z0Gc+J_ov2tEy0d@acXfQy^2yjcCBplM8ONF(L%v#jH|2|kfIS|R8h|l6Hk1MtFM6( z>Wf~Y9p`O~l{YkFG5p>XE1uFFeTlQ+D%*n3lkh#&6eNf}S|V7EmU#H6VqNjuQ!dYo zJ3b!CzbIfJUjIDTCkjt|yvVhHhhpNjkMh4Irh8?Bc=S0&E>`6uR zY|v?v$Kns1irbQGvUh|nAu{83Gw5AU5_*ryO(Vs@N9A_K$4BLM3%TR<5tR#!kgQ5F z42fcp%>JoJW*_NMWC;^zk|xR-q`MhsBj=!*bDM7EJg&}l!C7=vbYir<&~;H+Yf8Ks zEA&*1HX5^zRt2a*k0bEJ4e4NwuCJPbx##8{)I*!I9?fF|Ruv;Z0#;O~r|3D3#NYFk zLL1>-G`3n%tSd&0ryFTd^&5W$inZUXTV-|`#*Js&sAqyM9$s02`WY#;VdN3oL<6I? z6XS8Fr&c?!^3AU;NxRy^7#?=Q3k`>Yc!7WF&~Sk9WmU&FT1@q)P*+S6JqFdO4x@?q zgF4;vr4M+pVWNM>QeUE(@g7UP%2KECsDHjhp#P2HM4Lc-{F~OvBW}0NC#5e+Ew$fn zAMx7U3h!t5iaOe8x6;#SeOU4F`uyy;v2r(B-8d6_WDMe7Cfthr{HA^tuf3WiG_^); zZc$Sn5@lIJk1Z?Rng<#4iBx9SCT>UA3G z)D`KPM$J+6*gnIi(+D6Y$5U=?G|ELr(@nfwg*d&PT)+)>4La5l$^eNK(g^8z zL>3z-*E4yqCC{UNjMf(;7)EB+k=-CM;P1zJ=mS1xTBARUHS*IK^RtVZ@wor&xUt%E zw7PMI-H+j74H->mpA>J@`TZDfUd2VMQ1M}8+@U>Y{F5}x>aJT?e~K4|ot&m!MHZuW zax&toaZMKbqr@c|ITo922{-3{Z^He3=pSdU##`NQcmCgbH25?|w{jb=&pYSP+WQ~7 zZknlyBToyh{KTWT?1xPG(1`ZYJtv=o3DFgVmLC)={oPhK=_k(0XO$-&z1rN-iVOO? zdVk6JH=pHw-p=+u@A@^(C*zJy#QIp&+dE>%=79&rV)kBUjCyL;zOC!4OR?8x{Onu1 z3h$%CaO}2K8Fd8yS&2t{hDArRmUDFB<0gE+_S+8-4H0 z*uTUpH2*iZHM&;r!ubNtCsEGWabOeY8AjSopJB8}HMd;5R|o(;vVlNnapEB#WK>le zrM#8r|HOT^&t^A!`o)}=zWd_%7WTb4zx~}t+uzKbJ1y@gQ-6FYEw#KkWdVl#96bN0 z3R1iAPjz{@utx=W!t;M#S5jUN?IF4=IRjLDzWC_96H$=?%q(lc4BV>rhwP zeH3thFwd!O{5G5S!HP873H2p2C!}G4Z4qBT_u4E7p(l z7xkPjMT_RVi05==Dsc1~eNLBzDEcVD9p2DP=f`xG_8wWUkA9#m0I`)p)M*nCulDu!aECBKlLiHjvw z(tcmIKW=DCw^qHA|4@66g^dPjayM{P$n%JIYx{CS;_I2>5_|uOY@aJZO2^jrw4Zx2 zPWxk)_Nr%V`)NWUevcxM*Hg0n2eMA%(|1j*?aA-a#R&fUiO$-d<~gar;--v|W1@8Dh%2=xnr=~zzT;m8c%LOv zh`d=MuQaGE{2yHnbpn4EUPnAp|E40_drUx!lL1HT3-MoUp~EBmdHI>4<9PP-!!hyy zb!0#3@uJy$ z<5RJykNISKk7fL0vc2+OYELe0ZNCVU_vtv}e|>eH9S6zz zr0CRD%`bIT?Y0%IUEf}`tVooSRmBC(o%IWgn>Q_7TeH2Wd7XDdL+7TA)%o=;8)|kG zu4(fYm8>dRU))&VvbI4wubPf%}dv8+E~$8w_$zW=8Y9A zy&DR5G*?vCtle3*ym|R!HI4r2waqIEY91@!S-!ELx_NWU^8C6LqHS^W(&oC2#d*HX zyx)}@3zm1*wJgm85B|D>#WmZDS8s3FSlzy9&TCi=6w|I5_rj~8Xwii`*ZQ4*6h&yKa&W)SE zcU{@?C+fQjn;>s9boE4CNnxIEW3j)k#RvPWuc&KX^TaB$!HV@w-lDauid$Fb)wir~ ztlM1OR)M*e&B9*Wiz~LG@4SYM6`fYQcuV|GY-(EcsPZ>; zt!-&;3e@h{>fO3^>!SK4h4o7c7B9?OTvzx=Uj5>{f`YoGg}%B)d9{MBzGu?hHQdj|y>XBE6fL+^ z)}=n^c}_oK|1XJu>G)deH=Hkuo{7GJ6NtCnqtMe>M-23@tNRtsqhr3Izi5|rqWgiu z;d)BlTj-_$eoo`!Eq*-kV7duh9VX;@znzXY!H_kJHdJ6;Q$pM zs1oma&+B;CXcVW!d(O0==Miy43!7nFg%tELe#~&W6H-821-)_SEs$-vWEggCN-heM z1Xunq^u^dWA(w)Q9>hC+shsFN5~%z>428y{Uv&fcAJ9G?J?(Ge!5_H+{0D#&KWDfd z7*q2~y8-_%An_vlDrth(r5rS7ym8-@^17u7)ck4!J@s=%!d2S>30IrcQrUlw?59ea zOzHU_B>#B26fbICJtz5mMZ(p5dR_L@FAb>R8vsvNgQBN3s8hjHd^voJtpAe$vv1)_ z^m!PE%1LoyZ{tevs}gRPa3$Y3^rw7>P~#d05_(J z$0VG_#c5)+9G7=At`S};<)r=r13ekS2od4Fl>$H@Oj2Dnk)OgVpOTw{#P zy8--&gcCo;)$Dl*f7k-=H_#jH`?`c%`KPDoP$7OMiCg}@NW#^)$`>0YT#c*XM-24F zai5oP>$ty^aO=3Ka^BLo#`)rs@DcMx!l@s{3kk5r0GCHRT&^Pqc=VrpfLSC5Q}Vhw zHDg~ez>RV>O1RR4qHmUPtGq8sxK-X#8FJLPd2-y92DowDpoFV&75#1rw~qTg3Ac_* zc?%WFU1&-v-BDblqTn80W2{kL6|PecjUkEL_gQa zZguBD9o06qHsNO;>)P9IuUTtni(sh9*Um=YURjL<2U-iF$aw%P{H zm=Fl;2|h(t$YPXmd0Rjc-0oN`x8W9S<_&CXuWk4H>PP;rXrQgZ+u>#N`MYpPVWpLj zq?Drs$^82Ej{L>9tc|;5FMR99jr)VCcd~&uwfW?`c`+Z`)o-`nrN8Vd7T8btz8676 zrdp<+uF3>;vdljeQZCb~%4nZYFqsxa;U0N+M#eFPQ1MFDXX1uZE{_7ptAD4sd0ysY z^w}E4I~k6nIoLk{NaGuGYij9|!Pi3-Ls8?a>vOU_^=&LwDtI>-p|SB5l+eVJRfkor zf8X4&kjfA>7JoAt@$oDi8|B)}fr@j8$g z_kMppy6?Se5MWPi7z(=Yt^fb4>VH-Je^s|`f4OYsa$VDyDjIu*#cB%5SY~hR-otM_ zEESY))>AWJtNVGj+xCCxg|^SkVC?XaExYd>yw2IO=l%<>=kfYi+@tA4MN8LfyM7S* z?KpPz!wq`qgDP9-nvEU)e>T>4w1XW6?U9}gR`|VaPydG{?5$tv?C^}#o|?0^$$dk% ziPYDTo^)3De|1m)Fxu&w?X5bDJ#y;l!OHUo26w#o^x(?VPY-@m_gn?-Gk6`IbMo8a zz7xm8k2=2-K7Qatcx3*`aNqn>;r**mhcE3v9Uj^LUihi`?}i_0Js&=P>fLb4%!flG zI}Qv!_4Lz&sq>eF=Cph#d|}@4@V@tc8h*<8X1MvprEuTM_dr(|3LQK+xc$U|!F}_O zhbvE?2sgK#4)+~+Gd%LprSOFvmq4>PRN*{0cwzn%gZoyapZ(tn&uPW@r``-#oWB&_ z{vM6@$Dz=CjJNtA`Z*Dv+xqSB2=JBXPle~4JP+QfpFawX>_0g8)cmIhL#IxJFW}Yu z-kagMCr^eeobQD%%s&sBWuZB(PYm`!7ZvA^qwf>p(1}ywO6TctX#S<}Q>%&RrJ>M? z1A}uq&ll!@JG^iHo8Sw2fb4yECC~Xnr~*lN zlK&~+8-VY<0^hvndv~_)_1wVsj*j7bM=icbKB@d~z~X<(UvK1t*LM9d6e*ef*6;)R z;S2v%eAR7dX%~N0e0Aidvi@JMW&Np`A7^K=tHVFkuf7hy9?E7{&z4L&Jk+2aKKr-& zwc-7&pXL$M1|C@feB@cyANgn2f9Vzd>N_Q@|J;4->c1>v{pZuz)eA}NY6@tJFxQ6n zCiD;g6YC#-Ks$W#f9h8+zQYa=|4l+@_*?Ao#RjaY6IlPqk0t|_7#eOE8 zp|czIJVJ8~JRiQc>s8ZS66?E==OXwgS%$$YjrlIg1Q{j|ZCU>aKu%ptg(JtH$JZVVTfSALChL6 z_JJqTRZ}9aX#a85Xnh&`V>e5m>j}~LL_b6G^z$)N;*W7HZTgjut2FLou1wF>lzE;| z>f)l1^9#ixvJKgU>_q*&1HGPu{#=Nkfn-nr4)p&`T-V@LiTfU058!$Num6qv)42W{ zjSqXh^K3%M2D>W%vyEq0-+}&JGxd-Q_PGeZB|H5ZHXxf{fX&Y?V*Nv~AK5|q^!saB zf4GG8GkxF@vZ=EFm$M3;r{|5WT{DB?5dZH(E+Vab!l=45u-t^J`s4Q(I$@Eof`+^et$6 zY(eLa=C-Zvv3f(7tUtCOdehL}>i2fC1)aWN=K>*GT|-xAi?^er&fC(m&D+`J3)D4u zJ38wmw$sb3^Ftv+ELE5N(J1i>_Gprs+;^*8ZJ=MS`Zws&^>(VTGYZ5^H7w$8k}zv`84Ft-^Zv$dM!3s3);7C?eKNxvAVjp$LgY( zRb{K{1g0(zTeW6y2an!SJhepra#!VKH;PPifV5$^Y%R+;Jkbhff6C$GN5MB@vu@)% zb~EZp4#!EMrMw7uhAY-jPmUm{2P-ph>yh*z<7F4%?xSf@xCFDWOL{f5qW>=7Hk@Br zO9UlX0_mO#_})#|wA4L4J+ql3vZrUZgWobn@OjowWL@!W|2hDot^Eugf>0H&BY07L z_4DoEADD;X>@mVcXaw%-K||6o$f~8*5)>!dtFu4%C&Xt%WBqE zRIRQn+qkZ5b!k~Cn_uN#DiB~#b!RH?qUU#r30Jkr6F!F>v zx=$DPxMlW?*!|5M=h&K3nSHArc0t>R?V|7dXmkb-#L3X*bY_=zJ6Za2QM7KO(G7K- zxlMICt1{xLPpaF{t!-+A=__%}RAk?y=#1iG=UK(N-&7+kil!ap?N7yk3*nk8mv| z1Pb}Z=-I|t?=CIEegz~MXefud}0 zqW9PYKlh=6G83-{jLkn4M&~{&D?m8CH4<@nK53yOn1lw<>ys?AV zhqCR3LET1p#t0Ajf)gXzs3lB>wlUtUV3HoCKbgkT0}~S_QvZX8lGr(-XPiEX=z-4- z+9%Rkr|6&P`7{&s=pXv9kvRV94>H#Gs3$9&Pl{?>e)yV8%$$`e}?{1=kmV6`*WtD=C4P8=>N%mEJF6M zJ7CYDDt7-aqNM`)E~T-rbdQg4a}Yk)3H|8f*Voq8a(Um#hP;z?@}-Ca`a%D>!O4@! z-^O$Oi#j{=lCJbqp7nKUUf2`;!yZoV^GYP5#pvfa zNq-S3Us*Sn37Xi_pH2M0zX}5nKFIB*iSY^pv*vJlsQu|Q%+Gldb{6evKIKhEeQtzZ zlJ)xcw3)$qb7;JI#2@Ulr`s`)&$s)hr?S6vvYElTvj{(z;}^0gy4hS(ZQ}h{bcTyS zZwEcrHljb~wCI4F_UIU)^_X~XeLipE%50ZS`|H;3CHXHXP zDa`Io`%b)+5Y%c1f`!mzSUvtTx++Q&s`PUXAk@F!VICCb& zKkkn|r~X%%`tRK}J?2kC2eG~-*vQ}bqHua4{0r-S)W67orYF#N)4Bf~B707c`qz+C zD-29ai25tmgPFl;(}+GX=5Nz7sm}dvFWS4%UisT<((g2v$=@8nyM;ZEP3L$A>Y_dQ zqaAqZZ}4yEcV<+-%HG5P+52tMH{GL1ebab$<)1?(&`+np%lSN7{}L_RL;o{L-O4{m z|NIjv!ehMh=a-uL>)n+R(|3KFMc*{P4f^KuHG}IrL+TIypApq}eKyt$mqp($S4`hk zq%T)&e)IW(_EO*IKO+cjL4e3WuULO+esBXyePx;DrT(Jp!B7z^&zfzLZ!CTF?m|9r z7ENgV%J~z>Z!`EC$?vfEo1Mo`JIQ~F)-$`vA8383r}a{_SMhCT(81{)F@JM#eUQJM z)AZ5&32)`=G3MV2*iRHO|E%$l?0bd?pCXxEp1DUaSJ+x3;5wKI_HpF*bC;RgFj{FDyjE|1V zpL>#MeJxK`0tkDW{g3o-Gsb88@tw#wlzjD+?>L;ly7u8jk?)ZGQX=de2IT%r_PMct z!oPXG=#cp($-|4vKLR!fU!VVn%Ab;LZ(})A<6%8>VyhtXX}+E%+bG&_|A4%N=j&aD zpOL(Uy}d$-b~Z^8HM5M-dOn{!?Q1hyR=B z=R!`;?MZV2`Ya4i(`mV7q`xzNLF?DFM0PpC^P8dR@UP^|sDBMD#C{+r6XK(Qewg+f z4(>n6(fvtQPL%#U?Kd3B++O5Ae1GER^jSm?d~Pt=5%nkJx9I_QmK(iCh@Sjuw}Y>b z7%z?J;ZK=Hf6EPKxsB}$ce~v*-#9(@N1FrI5&nqrseRNRKTKoKggKH$&PLLRzrtS7 zi}p#B?;xMH4Lku}`TCaTLVo6s_6PaX1Jm^>ppUpPpNFeb(*xQ1RH{$L1j6_jFDDCi zzTZTDux}>0i=XBL_mP(=Yi)a)L#_nY2{->qR^OztD!7pH-OO38+Ki(fOa49>b>r zpM<&*@4~5vYESrd)Q$Lz31<8R)MI$a%l(z>1M*Vc1rCIMY?#lvW8~v}a*6nIp~@W3 zUPS)5o28O}!=J{nH>${=QmKW%)NdE@}#q)%qm`$t(qac6S)6LKTh0aH?|2ou-rB)Q3_UG6ii$hG>AHKo;owO727t8f5 zd9frH{G|u3jUQb}09|F9N`Lf%A zqXHBg){9dZf48uQzyAGk?3Wa8U*qR{N+>0tJ+1kfDIbCT4g?qy z6lp)}dB$b%*LB=M@eurtllw7KzC-zpp9H4-gYx&}XPB^%(AVwcGkKqCKI7-yEB*3( zChC8&e1@M-oJ+IcFn>XOyKesSr|>_-x6e9%8J({aZ);GgDv zWg+m!^_kWyYrbMwe-SUTuwG+15%C`SGp*-XpH1sG>RDK?qx5Kx^(eOA8q&<`HRdbf zqwDpU`3j%kmVAXa<5Yw{^8JLeKkSL~$IQvVMqHThr$5Dfh5R9IzA|Hse8oNE#`#JX z&sW^>^A*UG<)-)^%~yhkd}DOJf_UoAismcB(BCJ|N3v*rr}#W7Ke}`C5%Qn^kom|wQ~Vme|EBB$Mee^31eu*6JpaJ?eR|LKTlv>%FOiu~@@VvT3`JSk>>@SBeFq=AD{Kd>M6Gn47_ z7`{HA!{?2=CdT&X5AyvvKR@O9_8SX9KiRM!KZMU4?34L^ocqUrq5XL5ya(f9{}emF z!umgYKR$KZjpN%C?(b9M$2Y`}DO36WOYO%;$2ZgYPi}DP6ia-Io&Q+(x0t^-vEQCz zjBka(2|B|ESYm%j^CNS@L|91dzq7Ib&YD1<&l&dL(`dhFh<8rfki_=iPUjf=Zycyl zxt(}dbsPKdbgY(OQ0!OH-i3N}KZ<;9g7Lfw_zdHD6#0K9<(p#v$>&36bpOfs-{emy z(Rep`GIOI&`Re(F(5Fi`=o9u&*Dd-q?MKGWU;lvf{ZfKCe?@*9QRj^ zA6@vH$e}_uyYH!9SFO8mZqBj3KF0d(wa<>c|Ic-iuW?k$nuNdX%jxawOJQez{L1ft z7kS&x%4*llIHs?uO;}Tl-3UVZPiIqH&!;brYMT7gBCn$wdvFf$R|ud4)yCOVuJ8z5 z!~=D#uHkQ%!TUI8%I~nfWo)mu41rv$XP}Q2J;#}-u^IqunnxtwQKScV+PFltrwl;T z=JEd2c&{edQ7=v{PNY{+s>iOCZ=Ni80X14EIE`_I6$4TjkEHGW-Nx^n_|UNx}z_37F=Ol@Ocuxr<#HUu~$n3%h3`KA62X?etXL0d2z?b zBl#Bv48-f7=lVq9iH{e#7VuC^y!KK4x5RX>Y!Ht=DvzJBHJX;JL(@?hhmbv~h@K5P zP4Za$fm3l?l1=uGvL!@j+-?TF>q$cIQMqZPIQXdCuK4(<+-@Ovygs6Gff15bNroX& z43gPD70K+QJ&G)0;!M&+IfHaJ;%xLBG;?m#t(?czxh^=1j*3oBC@r@Q!{YlgnlSGd}b*jT?BL1LG zw|waX9&DKC-?r43YG%C0Qm?kuX*}wmFA?Z}qd3tf5Fh`Bb@GT?ZSzU#i&9JNx7tU% zHaElj8NQ;9HQLSeG*%y0e7rtCJ8rDpja4_!#9kSLxR(jHB0s;OU&U*$CJ9ZgG25Hw zrJfX>mu|s;wS7EoTWLP;b)(MoI5Eh|e>{4X2S^?Jk(Qd*E_3}!;FVd#+GVq>UHsd^ zhPBH~|F)(5Qe@4>{ydg?wWW@1_4;d)rS3o(O?A5J$L7jS?fjV_X zx~5TcR6VxOu<0~PQR+H-k?XOPn;VUCk+F0WFIOQ>ZzUISgI$A;b%ZiNVudt9Iv$n9 z#>w?e-fPM8s2`*C#R!IxnRR40NDTOUu^#$>kD1o!&ti@IG{*ewqGmkqKRa%$_8hBj zoMHE3xL8BR(%C1)8+Cp!hMQM$5i3-D7#(+Lj~V|Y&9b`d*43Zlg<&VBX;+ZNsGXdQ zxN2OJh5jIMiAIjaCR@Udx!;>`Zy)-HnXB##_RL;IkfivhpwAu zYU1eALMuP<=q>vpQ$94ReRR*s=U_r~1)=2!#Y%st)lK?|v+`NxiAS$CceLVy{*K;X za{kR{d7rnlz0bRTP4mgPV-vAH7WMXy+Oc`yL9v*m;nze7~`sz~bwHZJA)~>+& z=rA0+ZB@n`fqz!w5uahvk*wt$UHG^OKOnvzC$|5w{zspTa|sOn>l*LBNAX7AJ2Un# z@e0lV%x#UXRk(1zK=VnIGj<%<#Ce91cEe{FEmqAf*PayufRAh-&{>>#$Ojo!mqsaX z=J`KypY5~R&7OK8=f&^7FtL?=Z|-k?yV3Snv*yjn`^oelA4*HDXh~U!AwLJt|EY%5 zZv0VQUM}oW37+u$pO=?2wxt|@TaMTDp8rD%B3sZk-m`rO{!4m8vEd{r`qetrm3AKi zoFB|{s_Vba;eGIBWc?&*_Twqx?lZ zr%TbIIWOWlU6~3Ty+)tYB_WDFN^qMuG}HMpou$1;)*GT9C<{PrWe|1R1jK9qW7+OY z7S&qYGA272;}vYZ2y6*)A;mV6Ki|&dvr01{{cBg+=ljatJ*6&9zw?i^AOTd z&RV9J8eg?j_J0hGZ`5AVQG24Zwx@YcDzLaIW8|19Jvrh^?TMyGRHN_slL6jmNfaV) zmdYy)DhvNZS3{k^--Xu^Pt?Du$o3u+(Bfpk(fUIC7hCA?2!CFFX6QJc{rqrD{CyqS zPkOv)clM9Pb@Bb;nuQC=Zz4kdKLJSNqnJ<=HwhqhY%G#C3R;Kud*h7%71=&lQquTT zEb3!E+1_Ir|CnsA{FmC3OIzD7#^ik}&iG&73}O^Fe`m}(zVf3=wC_cGk7fJ;*;#;j>0wV z-lCFKCF_fu8d{fkZr)ha(%@gauBBq<@`i#Yz}D6hzO-f8noS!ko9Z{L&)dAQa;0}e z;f|Kds@kt^Hf>z}_{!GR`HkhZU5zVRc4CY+@KL_Le0^6# z!Ln`|uW>_VdvV3i()vZ~^IFQcRYu1x-?rLYw5+|XeA_xnQ@*Kfee1?W>$l$&T|xd> zbjvrl2Aei7s)9aR%Sq;~n@M;1?WG0V)_99o=WlA=R<^yUrhC(d!a&?HD|T+&1itIb zmp|UnUDynHTcE4Q>q`prd>f1X^{qbGXMJUT+nUE$kquU?Z}t|gT~*w+I3GUe>j#wWVuwIbMrab!}cz>2GXZ-wpj$ ztjS;hc;kx7mQ5Q9VDFmzO>Jvyn>Gu3Z7;6eioWw2H&%98?cy!*KfbAX@gvIL)U~d) zr8!WyW2>(|&*#fuwzOf%!}&`KmgO%hEZo|V-&nt-z?WZ;@2eAZ4b5A<4ZiFB#L7DU z+bZ;{D)_aN+o|N;;MwH)ixw|lyeKbkDW_TPZRzmUiT9FP{Gg1_+nN{5V{BK>9sT&^ zoIi5!`qGzRocsm(+(c?I)*i26+V1JvwA5@o$OXS2sSJo~uHk+j?u~oQCuzZ@vM%+0 z&$Idw`=2NNH^9v$}${RO+M6WtFK4%bua z-aYpJ>~b-JF_zhNgl|HEGir+#@c;ByVZXto7U^rZ%T^jDEqI$s_~1H2$s zTB2*Y2|Cri_4*>(+S&W+NWz4c4~{dzgd9rQF%+?&iLV|AV57Ly+6m4h4+p6DK$UpU zdtS%8Mx!_--gBk}J&%Z^TG$NZDx{!~@neR|osa_JD(H z+I9GU9*GyxS4$JTCgq?pE>RHL>%Mz~U(`&My0ck)5-vD^J8WcUPL7fVo;>+P%Wc?onn0*UZqR+!P zR8EQudka^BUy*RTge&07p<+!|~agFd&DJS)>cq7u45^n8hql8bl^rQ4e{TNS)dJO&3b52xw&Hy*kAHNR# ziUDqvH&^Q2%I9nex5_&p4P=$~bqTlj^D_yz%KJ-0KSusPGQf@UX3F_P;~HaR-gV$d zB%Js$u4d0k_(K->fPvm<-`6DE%0E3thYImCMcnfD#S*T@Rle9H;c8q3KVqOaj{BU1 zTgUySgj>f=mGhRyHO?28gpZmp5>EXnUPyo~2Dm)p;c^`@z@vZM1I%JMn3C7csTuo% z0dAD5Ny3#L6n%??TjhOG!maX_%8;YR&6DG5k$W6$SV38e>iJs^BVaJZFKc_%L9Bt2i)hfvcrgoqwwS)p|c0{83Tx9%;W7 z7PwmP{T8@d@0m!uJL>A2+q{8phCM=~ccr(ht)b~r{*#kscpx)=-I4#s6a8ExyV;!w zbyU~f){LKdtncW!wPtNyt%9LuUk4j|lOwkd{O2Xdrhd#9=)g}yNKu>X1CYDi*D1us z50}vISeCD@osZv>YuF*EmO~$FeVs*}`2ETHu1+681w9&?ppD*A-($@UJ`-GM&(|b> z?rv-rw_-f{`HH$W`gy#SE7z|oZT7bK8ah|CH+HpP_?z7QaT~O>dmHOI+v^%RV?rRX zC-@XqA&XJMN_n;P1jAg_Tx9l2VQm zBnuikI`fy@v^MULz3|N&H|`In-p&Tz)aH}#=EZ#MP`}-JhyJpwSYSWl`(6YQnQEDO zx*`+QsWSgiNV!a_Dx-Zq!DLzxg?r@P85zeELd7dppM@JrxjYIWul}9l<~f;<(PwKE z?_@ZR=3svhAdPR#t*NC;244?V3`LEvuFuN$)VHxztKi*egvQ2KP(l+=RvlKi2@1up zqPN!nyBz;|?LV?06yK`9bO>*?3E@@-6ovU}rQI&ukCI=uf5y_l%D)fzc=mHh|G6xO jC;O2rsP-y{-j9K;V^fo}9rTVL9+&bH8}uYEYy1BX&50R< literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kfef3ffb9125b7358714c2ccf6aedd14e.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/kfef3ffb9125b7358714c2ccf6aedd14e.o new file mode 100644 index 0000000000000000000000000000000000000000..47eda6340e37a85de648ce3c3982a799bcc7a567 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yR^KsL<8Gpz>*$$*4FAlS@`mee4D9(`Jp zX&G`NMAk6|gBi0M`%Ii66HkVgA>-_1#?frfM&9@k$H5HNWV1e*oD(85j0Bj$9j^n4 zaqsunqx;^g1_AcOhNYnU-unN)s{U8i|5tVE_LrBfT&`;xQ$=I1vRF+)8O!XA-Fx`0 zhoyqj&3dW_Y;`}+cH90pz0mgA8H^novSs(Zi`Q9O_S}EP^&DRRhI=%fsA%bWZRZa{ zza7V}ez-voeNbr&U9+)6|JTO)j&!g?pgr7^!3w{Z?dkupguVSMogJEy+EabTHo0%e zHj(-|+>_1<|BvqJA4WS}v%Ou1u}4lmGgxtM|KRpZ&kU|S^~~Tmblae=qz{>$&i;lkbI7W1Uo9Or8I5Xim#_!sq853-7)3)9};Ix5CZGFNXV8UIJZVD0JY!;I`xY2lviD z7OpsTJlx!ND%`jKt?_;@>G{tLhE5(2pU12D z(p%xVCr*URotMJr=br=3($JjNCkOkWi}G{F(D(6h==jNSh4WN6H2-4w>D9#Zl2GXQ z{=vB?o*WE;=kxQw9o{?tE${_BK=wYolIMIOR1RI9e+V?tTT0@$!=d-y3a4y25uS7I z0{FTR9yjyR&3Hwiie+eJhtg*s3 z-~*aQxJi9Oy5liwcZ9LR*8t1mZA4!`gr9s+SrYniLoxi$bM@NJ*N+2%gg$Iy_WT-Vt$;N#jXzjP`~;H{CX&xT|HAW>CjMvcIeFC z>DPw$v3{CIOdEJ~1@MvQSbyYSSpUUW^{eleu>Q06v8(^Oi1nXKV^_~7v8ySdEy7$I z-jmQj{Lid^_yO(Eh5w;nz3?tOH2k*-q2X__Ll+vbrcPk}BR`r9SYl}S0e0x$9sup% zB!$jw*!?KYHSm1s+RoQZb4jf4LY@oYpJW*ZuQcX+Bok!32pOG_iDXNj#IF8?<|^bf z%Egk|H4o%D|98v!M<8FhlxqY$kAUY9@Juq2d>6pyIP~v@OwPa8LhnJ2kyjH!Baq`g z$YGS9WEgohF*E{spo{+V6WF1P(8{0t;}`nRM1Z{oTJuL|7v;JP2z<9Piq+@Her z-)Vf<>)q!PLN?e{`JZh(yZSEl@0zKHT(Hjt_$}G#*RTQE{5))aW)bTjg8j%2%BSC7 z%lgA5te@!vkCIK5{V#tDwt_Fe2S2_9oBccN7xM>p=<-3>)nLb_=|Ugmryo}qg{~zP zUm^S5(}!6f&%=4($NWWaw_2}yujU^$=3rm$P}T{y0xt(T-;S#XuOz(E znTv)URkr&AZN8QcRs~EhVRg(rJr!47W47@&O}8;-ABQDR7MqjVar@)w&yTaQbhxe`!q|sO3 zwY6?bz}xB**0BP-3rrA9vj$ok0$zU;pLG5}duMxRw;#<3*WT9A>22%Gy9?a6!Rv4J z1sd)OGs84o7OO2eP%fR^fJVSwU_&L2*=ITQu9?Ehwp^g=8CV7t<65lRb%;DdW+FGf z+)G35r6G4(S{TRm@1-H1v+UipY>{=WfSXT4?)-g>TBO%vgSDW2%a#sbXCAAoYkRyd zidnU6Rh_`pX3{q*Drf_ktr1GgSY4>DeM0`5MN7KKYN`?{o8Lo5340&c_k zg|$RbawU-NsetcYbWKa$-P1FhIU>7zW;^&TV+5aP?L^iU&-SkaAllkb(;)~|@j8qb z)mN`CFD)-}mtg;oa}-hNkc3Dd#%l|K{GCDgVo6t{*G;Dj%`M)}=JqysXM1}~$2>P? zw$Iz)bLTHwuy~$(JzW+QE@j1C%`J`Yj_!_5U#q*dtE1E1=-blV=5w#BtX#RaZvC?A zwdIwo>z387TeiA%St*-e>0acX?+r9GxjR~M9`K0QjiHHGI$y!(9~NG>vnK(!FLW^S zggm-W7x%bj_Kev5%^c_0no^m4iyd}B+ehr8@B3(U1`ou^(B*VymvuW?`f^dUZllo+ zb)C6Qbvmmu;;B!n+t96TYJ}-4am-X_)9iCnxlPHgRP3`cK8)f@hOIJiTLIptr7}Od zcfk>>@Upu=1B2l|f7z!CFSBK>WA+pWBk%69;r?L>-;SfLR=l-yNHWk+8q4Yy`r?x$$H|WENR03UxD-&-JJ|v{q*Dd-sR5Qs zgZry4+t@F%S?@l3gk1)Jm~M2k-Fp-5eB6!MsqC1W{is{Wz@7pg{q+VhktB%5U)HU& z&BTk|V-x(`hYHF}yv{n=p@7|v7A!JwAdL+zWVu1xeeZOZvxt5miM?@<%?c*ielz9G z?YuseZ7&S!Ho`MTc*qx=7|BL0VKTIh@n!{+^eFv_G?pHim@tw0A3T`E&JsQ2^hrby zd~VP_kzur{bh7HB^$>*;M z+lb$!;3Qa)+c!bs1$~$i-W708B7RfPM3UHG8Y}#B^p851_f6iPGYvI=J^DlcPwiz9 zvWMLPdk$5y`*#v8707onjeVtie1w~W@VQRtM<2hwrly9=`(`%eovf2DMI6u%`p*qc zo=pBWp6g%K+3A;crJu5_uS@g7p6DO;aB?Rndw`$90Gl|8k1y)II};-80|3Ye;C~qb zTZB9^vPj^Ub!+T0F`x{)2@EO6}nY3=gzj*uHAj~mbqfI9H2D4e&3x|QFg7ME2{ev^FL?T*@ zevXs$7m@NU>&7xc6I=STi68h^Vc@|BxxF+oUSVL?94-&FKb400IWNM_pgql}yy>XV zjj)TdUVllO8JstV#+yg{!9KgY9rO5nyMKBr`zt4#8Js(d@N+qSA$zi$%_Y?)-j79R zxCr!i&|_^Q`lC*cx1%obCBVCZcbo8T)CGPS@Y8_L6Zm1#ej4fm|5e~Kf%gdf>@?ng zCh7v8tnKc00gpwL#%~gM7wQ6^pbL36OMJHEf3w7Ez!U#|fv;Z(|9bF2l8^g8`B%_p zf6mW4HoYwCc}DAMB(jIwKNI$i7SI=| zI@ufkf%%-v=PS+UU!<|(7c@am{tJBB3qO<1MSXOCW_tK*F4>d&E5;-L+Ds&JK4b)E z&ZPLq{qg72{|Zz8y*sDJ{AuU_*0%&3`5RvpPA!CgVZD#~7x~Zh1R8HT_n(7g&*@SB z8ggodfoTa*f5mz*GdOJ;(I>|IZCWPPxxei}dl%X(e_KuZo#rz6n*(^au;tP@J2PVXu5YvGo94Gc-+aDiaD8V;{lWh;qWZ4S#(Lqh=-cIr z>ARBj<%-R3K0nZ2>KpxM1feYm5Eo3g@Za}H8EVI1SUvxbfDq>|>vrY1irLW$d z$Oq1#39Vl_e**b!27e>@9TtDH^Z02e`A^b%W*7Mbtq=9IUW)cAzRe6eIK3n0Zw{^x z^0%{^KAJz_t$aPk{96J0i6Z8oH6D_EPZNP$zYBxo$CLf}_=p!}$+^6y;z93DEd45e z`AI%KCSQ&d@lt$NdLQREO)sSQHz9p zbR!G!2fCR~yPm|N3F9F@8!f-2jjc~IKS2J5`7@vRQ3ds5J7B*j0HAyT{zLKi6>>;w z{{hK=Bo^PHPsn45@46wrXUB=>$brcuQonj2GZwFh7HTU3c8c7F_^#_@Umo9)|G=N| z(J}dRPZF)KWywkaVNbLFk^XJQ_-sG61NnxMub%Q9hx1q0KAb4>9kO3agq_8J+<(bF zH}+5XH_sOxGQT8wcv1OBz~8cz$fp4LXu%|5GH}N&bZPke`po=L@g9$oWLR@ef^~lb8`qMjZ?dkbhEtDHtw_w`T;$p>D*}^flrWP&eWek!~6BNvOy0 zmi3IUZ+4fACu90|CBu5n{lmYR_9yUbrAqxT5WlRxcp z@bwYnr4c>+Dbwh0xxp;Av3=n#x0~i0r|14?bHF;nA2B|)kNV?>Y3$iBN3zJ7NE-21 z*b92mK8f-jL;Uw*yzBEr>lXq#5eWY_`};wfPsaIx^~dDz zz?_XI$E>MRg&w1LvITB*}CI3iHu;@3FmMVX}hzH;o+EDYe67xF&btpVK{}a<= z_%z^?P&eXTIQ3BN37?L-5uY)^jGusd3=es^zjA#*UaGsmfzXc)^Er2ne4I}%5ns+% zn&a6^$RBsHRPt~5(>V5KCHYe-weXkv?INCh5j84?^M85*F_`3Ux32fMM_{CrR8=a%Kcex0u$QcKW55zD4+3@z?6Sb{+|3S6BZKsx}AI`?^DfZ{G5BG zU!KoI{V$f!@bihYY4#iDFNkl~&0qcu{)hPXS?4dK^A$((D_*3ZA|Jtgb2-Q#`S}X` z)10p?1m3tl(|TpiR}AYf;zbtLYb+-s-a~(;^&IQ7Y5hh$3+r{19__In#r9i6nt8p( zd?kEzy&f}P;q%*)uh3?kittCipHTLPJ#qe+IT_f93-kTdrGKX86u*7eGm z@#=^vUK!5U;Xk(%ucG^JtZ(N1_CkEVFxGy17Z*pZPdq-;ks5toltc04K0em_ur=Yg?vg6@J}D2pFa>j8qdIgI^r4aha#CGzq|E#X4-F< z;}^y^?KhxrQ#|8%^ZvuM-vHhe--dbq{R!jQWs|>H;~73rirF9hrsF(m;DFQ*?1%l# zWcoaYug_=kdE?HBvHkgje1Fc*PkFxm=0ea4H zx2G86TVZg5&hP=2*dNmT$eb_{783jKZ0x_YCeY_|hW+<6+AkX7os%{svHiExImZ4Q z2P#xCz4Qg#FWXi#|>Jk+JjFKj3`7m|)Idk>8f>y|ws;{K)YA zfHby>4x4x}IS$o+JFCyP)#*({$|j+1Eh8{Tchu8}f(G!{HC&d_9XkKjoiC>hytmBGtj~D$9PG@7JR8IT^{v z&lgEZvA){LL3sP&bn2fzZjt?IvGf1Do0Hdkevrn#2!6zV79HUHr>y&*78lb~-gxUdo%YpU$;@a;&{88kqFZWgbcGKVg8Z8u@#<;?Y0Vx_aMiuq^5b?yvxbhkp zp}y!P+Hu~-S$RV<7Q^pJvEnJ+(U&+2uCUGcJPF@ZO+kX#t0jWvXo-i9D%KUhJ>~Me zxZ~rI{EGqx;`PsSeWLKh$BSGGcqk@b`zZgLW4c#1h({ll$IsXrO-t6H=_rgt$evV0 z&jy_)c`W|Gskkl4CVNKN5+XBhH-p~wB%$}H+%!@gd{k~%e0)@Hw~#wtA5po$2+67> z!;mNj$?TtsWcJY>MV2seCTXIaLAo1pHhK=4Ik)Ln&g1G_7o0^$MJGnv3tbnLwWY+H zu|iM9XrnRfXjOn3^f(Gn+>j2|==!P$G56fugL-Lm)~k7Jz^Y>8N5G2e^%OnFk@!2l zQfQ;Ri^f(figm?^@pL^6s($0IK(Y3FbgRrR!?^Km8}&@k#ltHrP(P!kHjF$5nJmOZ{d{X+N)KdGc z_7Shm&G3Gjuc%{XjWx}n<&u{2g@!G3NLQ`wZ z_NIBMCq?I_TQFd4A5Ys>n$LUPs53oI46^bck6z^gQpbLzrRKHETz?99Wfrk^*(_@p z|F*DU?K0E9V`;wxS+lV}kELE^sUusx{@P_+?S=1+72j?Y4ajk%mwcbu_35bcfC{ai zMm^h5Cp^`;GPQB3$na?R#+2+UaN@fO<^P!1=1b*o-br(S2E zPF<0%Y1AB5kL@#TI*n44y3StYdOYRkMx$J0EZxM*RfyAD$pzeC*Pvq^p$w2%A&roZ zM`f{bay^sxTJk*V$7p>qf?;H49oY>M1O8sDhd$tArZxJrSR+4;F+aPg8ISwVjvK2z z$Eq7=*u5Ao){wDu_DS(Zo!^V$=2cw83Kbtl#~s>Z#y?53tnRvX^{04Y*vV!z8SIQq2E%1=Cc%YMj|4~=Rc-E;Cem=Ik-X!$|0(%)%ylYZi?d{%kl(W}iJt+=4S zqxYAbfAd-1=k09o^R8dhd@}CXM68cRy*;CLY#w+}EN1Uv#;B)e?c2J(x)ghD#?QXB zEAT!#499L;l`%))pOtvTXIOM3YdJ?3K5oJfi0{XV?SHKQ(I?|v0z?10#{2J4ywUg0 zjQvZzLi0a!Tcc~`E}SpWd=llf9S1gXo?)cj@EJynRddU=XN3UZBO3^G1}7f!K}J=j zQOcWn{!iRz`)qQvr(eu@>ANpZY-Qh@`==ivE2 zRgl_^KdQ^ig*__36Q2L`%5ugwm*H>A@w(pge@H=O3%bU8whzI7NpC1NoCHO`T8FyQ z?xTS7gLzJM{kJ*14}LUcB3jDfqk9m~>Edo}q8HEU$}qvG@)V|&jfp3{7?I*RU9o0f?;(qE4HDcYVLY_ywTicft5?{|0m)QGPWcyqRQaZM_r~TZM zaoQiXv{yY_+s_aR@p}}3yq=WpKah1ApT28iZBKrWE=KV`Ag74i(4KBpdu7Li=$K$0 zLK?~$%M?@Nt9Hu%kE8L8+ABJ0PjuGyG|x!|7B^*#922D{M_j2r(e#LF^c{aP!22wT zLgdX7d8I*R;eY6As1x|R@H*m&`ZpEX-eUq>c6~?DvLaDRRuvbtbTuq0ZrQkKZOyi#mUZ3@ja?gStMeOLH`Ht| zT+{9?Dp^&szPPEOb$REe+Uk}D|Jrpe^P!Q*Fgc z?}ozdEftkDYj>0_Z(06WO_RTRZOe*+n#am_l-Cwiw`^)%o?pL0v@LE~+EQOzoafub z`(0UEu)M3jb!i@W@YfeST(hls^|r>^>W+=ItDjiex;nqHtfs4RMavG1(FQ)s)|ajC zYA9IRP2)9gsAw-P-%(n>XnkHw+185axMf>cdyAH~FDu)+PSTWZY+K)2yJ-Eko1!bo zAB%4Jrq*E7rbU&|M{60$yk!&VF2B9BVCx!h@#_4Ity`CEE2{3^xS=o*cg*q~wHv{A zecAFS8oCRcA#V$G^+bJ1VVnxD+32mQ@mDWxDQRji-LSRFTU@egdD-gxO=ZivHnz5OZ7RcS(Wz`;`QPHw-LjmkvoxibdZB5fAVXtk)6#M`bh=xMCO2Kv|4{R-!iao^Biw97it{XpSx zJ*n<3bkhJot8loMs{2x>>pA@!cEa;N{FQL(mlp#**AR?mTku3*YQRT-6=|jOm2ot{ zi(;iEx|W-uQ{7vyFQKiSy|0cWOlbMwI1^0Bp_Cm%5eu65>VW_@ic5{1;5_nhfQk=P ziTAwcb-Zgdic{h}XIjwnh&ZZ+%`mP)3i=p7X1Lr5DIl(b-njD?$aY*Z3_CX^7llcJ zEB_bzV(gocOTk1B;+?)!PV^oLRDK_ZLgUe|x(@vJXdjQB_BZk1k6Z`-1Hg%&)7%b> zsd=SchyNFlcoBV-G{Ngq4jMDwxNk~%-O>bVel>%h`nfFOs%?RUt4(UD>_12LQzcEN z^!yK!e>`1^7d5Y*lYG7+;c7m;F8djf22}73fTycL(bF2#so*KT9KKoB|4D$^w{Ru; zJd8u-gt)M`aV7W_3AanQl5Ya~Q@%r}i4s4_0?(81DS)fC1p1DgdG%;F^$vYa?2rrd#QvZrKB3&ur)_!Uwe7dC{r7!BocuLe`=%1c*qRR6IxRL(Y zb>LSFaHG7rQtwtiXG^$M-T`SKtGsVWxV4|3Nw`(sUmE%`^8b+mZj?7u&L0}r7$ftp z13xU`#E)?`dtSmHvcLxn^hW!>F5y=G=_xu?h@UCqmcK8Sa5b*-#U=??<0|-J1HEzF z=Ox@a?k^?WI&P|*w=}MCzPKcO)O?X}>PPWH0&F(G#zYH{o@{B7R$kuylzg- z*cS|Nqg+iAuJoYjTO`~n?@JPHmA6!e95rs99CxJwZX7o#;c8q(ze~cc<9<)Vt>aSO zLWOb{no>%46xXOIxQEvmYm!$5S8?Nc3tYvA0SjEkfnf_=ExqdeQ}wUb``O@+ih}n@ z`>n9R)q3x@z}0%sMB3d^SKr*`4RkZ?5gNTKye$9;hg{4|6VwYfe3xyyW= zLR|cC3H^>`+3K44_&vFX?Sg7K^s(00S=5Q&pRDid^bu6hqoE1f=q>d<-rV3b!G-pG zP4egN#%6H~#-pFFsB5F2$6L8_{i@PtZ;P*?b5(m|R||%}$=x5fK});0v97bdu8}h) z1Oj`4Pf-=J7$scZ9#90gIu^@qya}6m16w=lI{dzd(Z4GiXm9j(df5X0E*w%=X(c2n z>?eHRgCHVP zEmKcdWP&7VM*-y3zeC(SFY___ zY>ncb49C$N>>mK6@r}7PwRFkg>!FIFsPWbHIoY22HkK+Cyc>Tqx zq4-tw*7{$_@vqnZBMU7Gn6FmaZL<9+`DOcOEe)*v`+$#UKZo?6 l%W`CH96Bk@A%p literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/disaggregation/base/backend.py b/tensorrt_llm/_torch/disaggregation/base/backend.py index 3e5c500826a1..286ff33adb9f 100644 --- a/tensorrt_llm/_torch/disaggregation/base/backend.py +++ b/tensorrt_llm/_torch/disaggregation/base/backend.py @@ -106,6 +106,13 @@ class Chunk: excluded_pool_views: Set[Tuple[int, int]] = field(default_factory=set) + draft_block_ids_per_layer_groups: Optional[List[np.ndarray]] = None + """The standalone drafter's context window, positional like the target's (one list per draft + layer group, ``ceil(prompt_len / draft_tpb)`` entries), or None when no draft set travels. + + Only the last piece carries it: a windowed draft pool releases the pages an earlier piece would + name. There is no cached-prefix mask: the draft pool is never paired with block reuse.""" + def __post_init__(self): if len(self.kind_per_layer_group) != len(self.block_ids_per_layer_groups): raise ValueError( diff --git a/tensorrt_llm/_torch/disaggregation/draft_kv.py b/tensorrt_llm/_torch/disaggregation/draft_kv.py new file mode 100644 index 000000000000..dbfd95778011 --- /dev/null +++ b/tensorrt_llm/_torch/disaggregation/draft_kv.py @@ -0,0 +1,347 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Whether the disagg transceiver ships the standalone drafter's context window. + +A DFlash / DSpark drafter with a separate draft KV pool keeps the prompt's drafter K/V in that pool +(the context window its block decode attends to). Without this transfer a generation worker starts +every request from an empty window. The window travels as a second block set in the same session +and write as the target's KV. + +The draft set ships only when all of these hold: + +* the target model opts in through ``RuntimeHooks.disagg_draft_kv`` (off for every model by default); +* a separate draft KV cache manager exists (both sides: the peers refuse each other otherwise); +* ``TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV`` is unset or false. + +This module reads no model or manager state at import, so the environment contract is testable alone. +""" + +import os +import threading +import time +from collections import Counter +from typing import List, Mapping, Optional, Tuple + +DISABLE_DISAGG_DRAFT_KV_ENV = "TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV" +# Test only: publish the context request's KV without waiting for its drafter-store event. This is the +# negative control of the ordering gate (it must break the draft-window oracle); never set it in serving. +TEST_SEND_BEFORE_EVENT_ENV = "TRTLLM_M31_F0A_TEST_SEND_BEFORE_EVENT" + +_TRUE_VALUES = frozenset({"1", "true", "on", "yes"}) +_FALSE_VALUES = frozenset({"", "0", "false", "off", "no"}) + + +def parse_strict_bool_env(name: str, environ: Optional[Mapping[str, str]] = None) -> bool: + """Read a boolean environment variable under the strict value contract. + + ``1`` / ``true`` / ``on`` / ``yes`` (any case) are True; unset, empty, ``0`` / ``false`` / + ``off`` / ``no`` are False; anything else raises ``ValueError`` naming the variable, so a typo + never silently keeps (or drops) a feature. + """ + env = os.environ if environ is None else environ + raw = env.get(name) + if raw is None: + return False + value = raw.strip().lower() + if value in _TRUE_VALUES: + return True + if value in _FALSE_VALUES: + return False + raise ValueError( + f"{name} must be one of 1/true/on/yes or 0/false/off/no (or empty), got {raw!r}" + ) + + +def disagg_draft_kv_disabled(environ: Optional[Mapping[str, str]] = None) -> bool: + """Whether ``TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV`` turns the draft-window transfer off.""" + return parse_strict_bool_env(DISABLE_DISAGG_DRAFT_KV_ENV, environ) + + +def send_before_draft_ctx_event(environ: Optional[Mapping[str, str]] = None) -> bool: + """Whether the test-only knob skips the context side's drafter-store wait.""" + return parse_strict_bool_env(TEST_SEND_BEFORE_EVENT_ENV, environ) + + +def select_disagg_draft_kv_cache_manager(model: object, draft_kv_cache_manager: Optional[object]): + """The draft KV cache manager the transceiver should ship, or None for the target-only path. + + ``model`` is the engine's model, whose ``runtime_hooks`` carry the opt-in. A model that opts in + must run a drafter whose context lives in the managed pool, unpaired with the target's block + reuse; anything else is refused here rather than mid-transfer. + """ + if draft_kv_cache_manager is None: + return None + # Imported lazily: the hook module is light, but this module stays importable on its own. + from tensorrt_llm._torch.pyexecutor.runtime_hooks import runtime_hooks_of + + if not runtime_hooks_of(model).disagg_draft_kv: + return None + if disagg_draft_kv_disabled(): + return None + worker = getattr(model, "spec_worker", None) + if worker is None or not hasattr(worker, "engage_disagg_draft_kv"): + # A separate draft pool behind a worker without a paged drafter context (e.g. a one-model + # Eagle3 layer): there is no context window to ship. + return None + if getattr(draft_kv_cache_manager, "enable_joint_kv_cache_reuse", False): + raise ValueError( + "the disagg draft-window transfer requires an unpaired draft KV pool " + "(enable_joint_kv_cache_reuse=False, i.e. TRTLLM_DRAFT_KEEP_JOINT_REUSE unset); " + f"set {DISABLE_DISAGG_DRAFT_KV_ENV}=1 to serve without it" + ) + return draft_kv_cache_manager + + +# --------------------------------------------------------------------------------------------- +# Per-request draft-window state and the fallback to an empty drafter window +# --------------------------------------------------------------------------------------------- + +# Why a request's window is not shipped. The request itself never fails: it runs as if the +# transfer did not exist (the generation side's drafter starts from an empty window). +FALLBACK_NO_STORE_EVENT = "no_store_event" # no drafter context store was recorded for it +FALLBACK_CTX_LEN_MISMATCH = "ctx_len_mismatch" # the drafter did not store the whole prompt +FALLBACK_REUSE_BASE = "reuse_base" # ctx_base outside the prompt's pages +FALLBACK_PAGE_MISMATCH = "page_mismatch" # the context side lacks a window page +FALLBACK_NO_STATE = "no_ctx_state" # generation side: no state arrived for the request +# The context TP ranks disagreed about a request (one shipped its slice, another did not), seen +# by the sender as a draft table on one side only, and by the generation ranks as a short draft +# write or an unusable state on some rank; every generation rank then falls back together. +FALLBACK_CTX_RANK_DIVERGENCE = "ctx_rank_divergence" +FALLBACK_RECEIVER_NOT_ASKED = "receiver_not_asked" # context side shipped, the receiver asked none +FALLBACK_UNVERIFIED_WRITE = "unverified_write" # the receive's byte attestation did not cover it +# The generation side receives synchronously (TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP=1) with TP +# ranks that share requests: that path has no cross-rank agreement on a window's usability, so it +# receives none (target only) rather than let its ranks draft from different windows. +FALLBACK_SYNC_PATH_TP = "sync_path_tp" + + +def draft_receive_disabled_reason(mapping, synchronous_receive: bool) -> Optional[str]: + """Why this rank must take no drafter window, or None. + + The synchronous receive decides a window's usability per rank with no consensus; TP ranks + that share every request (TP > 1 without attention DP) could then seed different windows. + """ + if synchronous_receive and mapping.tp_size > 1 and not mapping.enable_attention_dp: + return FALLBACK_SYNC_PATH_TP + return None + + +def encode_draft_ctx_state(ctx_len: int, ctx_base: int, valid: bool) -> List[int]: + """The context side's drafter state for a context-first response's ``draft_ctx_state``. + + It rides its own DisaggregatedParams field, never ``opaque_state``: that one becomes the C++ + ContextPhaseParams' serialized DataTransceiverState, which parses and re-serializes the bytes. + """ + return [int(ctx_len), int(ctx_base), 1 if valid else 0] + + +def decode_draft_ctx_state(state) -> Optional[Tuple[int, int, bool]]: + """``(ctx_len, ctx_base, valid)`` from ``encode_draft_ctx_state``, or None for anything else.""" + if state is None or isinstance(state, (bytes, bytearray, str)): + return None + try: + ctx_len, ctx_base, valid = (int(x) for x in state) + except (TypeError, ValueError): + return None + if ctx_len < 0 or ctx_base < 0 or valid not in (0, 1): + return None + return ctx_len, ctx_base, bool(valid) + + +def empty_window_state(prompt_len: int, tokens_per_block: int) -> Tuple[int, int]: + """Drafter ``(ctx_len, ctx_base)`` for a request whose window did not arrive. + + The generation side's draft pool holds only the pages behind the prompt end, so a drafter + numbering from position 0 would address released pages. Basing the slot past every prompt + page reads nothing and appends into the pages it holds: the drafter starts from an empty + window, exactly as without the transfer. + """ + return int(prompt_len), -(-int(prompt_len) // int(tokens_per_block)) + + +class DraftKvCounters: + """``draft_kv_sent`` / ``draft_kv_fallback{reason}`` for one side, logged periodically. + + One info line per ``interval_s`` (and on the first event) so the serving log carries the + fallback rate; each fallback is also logged at debug level. + """ + + def __init__(self, role: str, interval_s: float = 60.0): + self.role = role + self.interval_s = interval_s + self.sent = 0 + self.fallback: Counter = Counter() + self._last_log: Optional[float] = None + # The sender's worker threads count divergences too. + self._lock = threading.Lock() + + def record_sent(self) -> None: + with self._lock: + self.sent += 1 + self._maybe_log() + + def record_fallback(self, reason: str, detail: str = "") -> None: + with self._lock: + first = self.fallback[reason] == 0 + self.fallback[reason] += 1 + from tensorrt_llm.logger import logger + + message = f"F0a draft window ({self.role}) falls back: reason={reason} {detail}" + if first: + # The first of each reason at info, with its detail; the rest at debug. + logger.info(message) + else: + logger.debug(message) + self._maybe_log() + + def summary(self) -> str: + reasons = ",".join(f"{reason}={count}" for reason, count in sorted(self.fallback.items())) + return ( + f"F0a draft window ({self.role}): draft_kv_sent={self.sent} " + f"draft_kv_fallback={sum(self.fallback.values())} " + f"draft_kv_fallback_by_reason={{{reasons}}}" + ) + + def _maybe_log(self) -> None: + now = time.monotonic() + if self._last_log is not None and now - self._last_log < self.interval_s: + return + self._last_log = now + from tensorrt_llm.logger import logger + + logger.info(self.summary()) + + +# --------------------------------------------------------------------------------------------- +# Test only: draft-window dumps +# --------------------------------------------------------------------------------------------- + +# A directory: each side dumps the window it shipped / received (see ``write_window_dump``). +# Unset is fully inert: nothing reads the pools and nothing is written. +DUMP_DIR_ENV = "TRTLLM_M31_F0A_DUMP_DIR" +DUMP_FORMAT = "f0a_g1d_v1" + + +def draft_dump_dir(environ: Optional[Mapping[str, str]] = None) -> Optional[str]: + """The draft-window dump directory, or None when the test-only knob is unset or empty.""" + env = os.environ if environ is None else environ + return env.get(DUMP_DIR_ENV) or None + + +def _copy_to_host(dst, src_ptr: int, nbytes: int) -> None: + import torch + + if torch.cuda.is_available(): + try: + from cuda.bindings import runtime as cudart + except ImportError: + from cuda import cudart + + (err,) = cudart.cudaMemcpy( + dst.data_ptr(), src_ptr, nbytes, cudart.cudaMemcpyKind.cudaMemcpyDefault + ) + if err != cudart.cudaError_t.cudaSuccess: + raise RuntimeError( + f"F0a dump: cudaMemcpy of {nbytes} B from {src_ptr:#x} failed: {err}" + ) + else: + import ctypes + + ctypes.memmove(dst.data_ptr(), src_ptr, nbytes) + + +def gather_window_pages(page_table, heads_per_rank: int, block_ids_per_group, ctx_len: int): + """The window's held pages below ``ctx_len``, read through a draft block table. + + Returns ``(ordinals, layers)``: per layer group the block ordinals read (held and below + ``ceil(ctx_len / tpb)``), and per layer ``{"layer_group", "global_layer_id", "k", "v"}`` + with ``uint8 [pages, heads_per_rank, tokens_per_block, row_bytes]`` each, the raw HND + bytes of the K and V buffers (slot order) of those pages. + """ + import numpy as np + import torch + + tpb = int(page_table.tokens_per_block) + num_pages = -(-int(ctx_len) // tpb) + ordinals, layers = [], [] + for lg_idx, (group, block_ids) in enumerate(zip(page_table.layer_groups, block_ids_per_group)): + ids = np.asarray(block_ids, dtype=np.int64)[:num_pages] + held = np.flatnonzero(ids >= 0) + ordinals.append(torch.as_tensor(held, dtype=torch.int64)) + local_to_global = {ll.local_layer_id: ll.global_layer_id for ll in group.local_layers} + for view in group.pool_views: + pool = page_table.pool_groups[group.pool_group_idx].pools[view.pool_idx] + by_layer = {} + for entry in view.buffer_entries: + by_layer.setdefault(int(entry["local_layer_id"]), []).append( + (int(entry["offset"]), int(entry["size"])) + ) + for local, buffers in sorted(by_layer.items(), key=lambda item: min(item[1])): + buffers.sort() + if len(buffers) != 2: + raise ValueError(f"F0a dump: layer {local} has {len(buffers)} buffers, not K+V") + record = {"layer_group": lg_idx, "global_layer_id": int(local_to_global[local])} + for name, (offset, size) in zip(("k", "v"), buffers): + row_bytes = size // (heads_per_rank * tpb) + out = torch.empty( + (len(held), heads_per_rank, tpb, row_bytes), dtype=torch.uint8 + ) + for i, ordinal in enumerate(held): + src = ( + pool.base_address + int(ids[ordinal]) * pool.slot_stride_bytes + offset + ) + _copy_to_host(out[i], src, size) + record[name] = out + layers.append(record) + return ordinals, layers + + +def write_window_dump( + directory: str, + side: str, + rank: int, + match_id: int, + *, + request_id: int, + prompt_len: int, + ctx_len: int, + ctx_base: int, + shipped: bool, + reason: Optional[str] = None, + page_table=None, + heads_per_rank: int = 0, + block_ids_per_group=None, +) -> str: + """Write ``/__.pt`` (atomically: a temp name, then rename). + + ``shipped`` False writes a marker without pages (a context-side fallback). + """ + import torch + + payload = { + "format": DUMP_FORMAT, + "side": side, + "rank": int(rank), + "match_id": int(match_id), + "request_id": int(request_id), + "prompt_len": int(prompt_len), + "ctx_len": int(ctx_len), + "ctx_base": int(ctx_base), + "shipped": bool(shipped), + "reason": reason, + "heads_per_rank": int(heads_per_rank), + "tokens_per_block": None, + "ordinals": [], + "layers": [], + } + if shipped: + payload["tokens_per_block"] = int(page_table.tokens_per_block) + payload["ordinals"], payload["layers"] = gather_window_pages( + page_table, heads_per_rank, block_ids_per_group, ctx_len + ) + os.makedirs(directory, exist_ok=True) + final = os.path.join(directory, f"{side}_{int(rank)}_{int(match_id)}.pt") + temp = f"{final}.tmp.{os.getpid()}" + torch.save(payload, temp) + os.replace(temp, final) + return final diff --git a/tensorrt_llm/_torch/disaggregation/kv_cache_transceiver.py b/tensorrt_llm/_torch/disaggregation/kv_cache_transceiver.py index e93b8a819c59..7a690e2af2a4 100644 --- a/tensorrt_llm/_torch/disaggregation/kv_cache_transceiver.py +++ b/tensorrt_llm/_torch/disaggregation/kv_cache_transceiver.py @@ -148,8 +148,17 @@ def create_kv_cache_transceiver( kv_cache_manager: KVCacheManager, attention_type: AttentionTypeCpp, cache_transceiver_config: CacheTransceiverConfig, - mamba_cache_manager: Optional[BaseMambaCacheManager] = None + mamba_cache_manager: Optional[BaseMambaCacheManager] = None, + draft_kv_cache_manager: Optional[KVCacheManager] = None, + draft_ctx_owner: Optional[Any] = None, ) -> Optional["KvCacheTransceiver"]: + """Create the cache transceiver, or None when disaggregation is off. + + ``draft_kv_cache_manager`` (with ``draft_ctx_owner``, the speculative worker + that owns the drafter's per-request context state) ships a standalone + drafter's context window with the target's KV; pass it only when + ``disaggregation.draft_kv.select_disagg_draft_kv_cache_manager`` selected it. + """ resolve_cache_transceiver_config(cache_transceiver_config) if cache_transceiver_config is None or cache_transceiver_config.backend is None: logger.info("cache_transceiver is disabled") @@ -203,6 +212,12 @@ def create_kv_cache_transceiver( "kv_cache_config.use_kv_cache_manager_v2=False for a model " "that supports V1.") + if draft_kv_cache_manager is not None and not use_python_transceiver: + raise ValueError( + "the disagg drafter context-window transfer requires " + "transceiver_runtime='PYTHON'; set " + "TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV=1 to serve without it.") + if use_python_transceiver: if isinstance(mamba_cache_manager, CppMambaHybridCacheManager): raise ValueError( @@ -224,8 +239,16 @@ def create_kv_cache_transceiver( KvCacheTransceiverV2 logger.info("Using KvCacheTransceiverV2") # MixedMambaHybridCacheManager contains both the KV and Mamba pools. - return KvCacheTransceiverV2(mapping, dist, kv_cache_manager, - cache_transceiver_config) + if draft_kv_cache_manager is None: + return KvCacheTransceiverV2(mapping, dist, kv_cache_manager, + cache_transceiver_config) + return KvCacheTransceiverV2( + mapping, + dist, + kv_cache_manager, + cache_transceiver_config, + draft_kv_cache_manager=draft_kv_cache_manager, + draft_ctx_owner=draft_ctx_owner) # Explicit legacy runtime: transceiver_runtime is None or "CPP". return BindKvCacheTransceiver(mapping, dist, kv_cache_manager, diff --git a/tensorrt_llm/_torch/disaggregation/native/auxiliary.py b/tensorrt_llm/_torch/disaggregation/native/auxiliary.py index e7a3c411a7a2..1d6620f11ffb 100644 --- a/tensorrt_llm/_torch/disaggregation/native/auxiliary.py +++ b/tensorrt_llm/_torch/disaggregation/native/auxiliary.py @@ -4,7 +4,7 @@ from abc import ABC, abstractmethod from collections import deque, namedtuple from dataclasses import dataclass, field -from typing import Any +from typing import Any, Optional import numpy as np import torch @@ -169,7 +169,17 @@ def get_slot_data(self, slot: int) -> tuple[list[int], list[int], tuple[int, int class AuxBuffer(AuxBufferBase): - def __init__(self, max_slot_num: int, beam_width: int, max_draft_len: int, device: str = "cpu"): + # Workers built without __init__ carry no drafter-context slot. + _draft_ctx_buffer: Optional[torch.Tensor] = None + + def __init__( + self, + max_slot_num: int, + beam_width: int, + max_draft_len: int, + device: str = "cpu", + draft_ctx: bool = False, + ): # public constructor args remain the same, internals are private self._max_slot_num = int(max_slot_num) self._beam_width = int(beam_width) @@ -199,36 +209,27 @@ def __init__(self, max_slot_num: int, beam_width: int, max_draft_len: int, devic self._prompt_token_counts_buffer = torch.zeros( self._max_slot_num, 2, dtype=data_type, device=self._device ) + buffers = [ + self._first_tokens_buffer, + self._draft_tokens_buffer, + self._token_counts_buffer, + self._prompt_token_counts_buffer, + ] + # The context side's standalone-drafter (ctx_len, ctx_base, valid) per slot, only when + # the drafter context window ships (disaggregation/draft_kv.py); otherwise the layout is + # exactly the four buffers above. valid=0 means the window was not shipped for the + # request: the generation side's drafter starts empty. + if draft_ctx: + self._draft_ctx_buffer = torch.zeros( + self._max_slot_num, 3, dtype=data_type, device=self._device + ) + buffers.append(self._draft_ctx_buffer) self._meta = AuxBufferMeta( - ptrs=np.array( - [ - self._first_tokens_buffer.data_ptr(), - self._draft_tokens_buffer.data_ptr(), - self._token_counts_buffer.data_ptr(), - self._prompt_token_counts_buffer.data_ptr(), - ], - dtype=np.int64, - ), - size=np.array( - [ - self._first_tokens_buffer.numel() * self._first_tokens_buffer.element_size(), - self._draft_tokens_buffer.numel() * self._draft_tokens_buffer.element_size(), - self._token_counts_buffer.numel() * self._token_counts_buffer.element_size(), - self._prompt_token_counts_buffer.numel() - * self._prompt_token_counts_buffer.element_size(), - ], - dtype=np.int64, - ), + ptrs=np.array([buf.data_ptr() for buf in buffers], dtype=np.int64), + size=np.array([buf.numel() * buf.element_size() for buf in buffers], dtype=np.int64), item_sizes=np.array( - [ - self._first_tokens_buffer[0].numel() * self._first_tokens_buffer.element_size(), - self._draft_tokens_buffer[0].numel() * self._draft_tokens_buffer.element_size(), - self._token_counts_buffer[0].numel() * self._token_counts_buffer.element_size(), - self._prompt_token_counts_buffer[0].numel() - * self._prompt_token_counts_buffer.element_size(), - ], - dtype=np.int64, + [buf[0].numel() * buf.element_size() for buf in buffers], dtype=np.int64 ), device=self._device, ) @@ -305,6 +306,13 @@ def fill_slot(self, slot: int, request: LlmRequest) -> None: self._prompt_token_counts_buffer[slot].copy_( torch.tensor([prompt_tokens, cached_tokens], dtype=torch.int32, device=self._device) ) + if self._draft_ctx_buffer is not None: + ctx_len, ctx_base, valid = getattr(request, "py_draft_ctx_aux", None) or (0, 0, False) + self._draft_ctx_buffer[slot].copy_( + torch.tensor( + [ctx_len, ctx_base, 1 if valid else 0], dtype=torch.int32, device=self._device + ) + ) @staticmethod def _resolve_prompt_token_counts(request: LlmRequest) -> tuple[int, int]: @@ -335,3 +343,13 @@ def get_slot_data(self, slot: int) -> tuple[list[int], list[int], tuple[int, int first_gen_tokens, draft_tokens = self.get_slot_tokens(slot) prompt_tokens, cached_tokens = self._prompt_token_counts_buffer[slot].tolist() return first_gen_tokens, draft_tokens, (int(prompt_tokens), int(cached_tokens)) + + def get_slot_draft_ctx(self, slot: int) -> Optional[tuple[int, int, bool]]: + """The context side's drafter ``(ctx_len, ctx_base, valid)`` in the slot, or None when the + buffer has no drafter-context slot.""" + if self._draft_ctx_buffer is None: + return None + if slot not in self._occupied_slots: + raise ValueError(f"Cannot read slot {slot}: slot is not currently allocated.") + ctx_len, ctx_base, valid = self._draft_ctx_buffer[slot].tolist() + return int(ctx_len), int(ctx_base), bool(valid) diff --git a/tensorrt_llm/_torch/disaggregation/native/mixers/attention/peer.py b/tensorrt_llm/_torch/disaggregation/native/mixers/attention/peer.py index f141cce93ff4..70a3a22a4d19 100644 --- a/tensorrt_llm/_torch/disaggregation/native/mixers/attention/peer.py +++ b/tensorrt_llm/_torch/disaggregation/native/mixers/attention/peer.py @@ -27,7 +27,10 @@ ) from tensorrt_llm._torch.disaggregation.native.rank_info import RankInfo from tensorrt_llm._torch.disaggregation.resource.page import CacheKind, KVCachePageTable, MapperKind -from tensorrt_llm._torch.disaggregation.resource.utils import find_replicated_role_mismatch +from tensorrt_llm._torch.disaggregation.resource.utils import ( + find_replicated_role_mismatch, + find_replicated_size_mismatch, +) from tensorrt_llm._utils import nvtx_range @@ -561,11 +564,14 @@ def validate_peer_compatible( self_page_table: Optional[KVCachePageTable], peer_page_table: Optional[KVCachePageTable], ) -> None: - """Reject a peer whose PAGED groups declare different replicated roles. + """Reject a peer whose PAGED groups declare different replicated roles or sizes. Pool matching drops a view with no counterpart silently, so an index-key side cache the peer never declares would leave the receiver - holding zeroed state instead of raising. Raises ``ValueError``. + holding zeroed state instead of raising. A replicated view is copied + byte for byte, so one stored in another format on the peer (e.g. an + FP8 index-key cache against an NVFP4 one) is refused as well. Raises + ``ValueError``. """ differing = find_replicated_role_mismatch(self_page_table, peer_page_table, CacheKind.PAGED) if differing: @@ -573,6 +579,13 @@ def validate_peer_compatible( "AttentionPolicy.validate_peer_compatible: replicated roles differ on " f"overlapping layers: {differing}" ) + resized = find_replicated_size_mismatch(self_page_table, peer_page_table, CacheKind.PAGED) + if resized: + raise ValueError( + "AttentionPolicy.validate_peer_compatible: replicated roles differ in " + "bytes per layer (role, layer, local, peer); both sides must store them " + f"in the same format (e.g. the same indexer_kv_dtype): {resized}" + ) def check_peer_compatible(self, peer_ri: RankInfo) -> bool: a = self._ri.attention diff --git a/tensorrt_llm/_torch/disaggregation/native/rank_info.py b/tensorrt_llm/_torch/disaggregation/native/rank_info.py index 87731d303204..ee74cd5680ab 100644 --- a/tensorrt_llm/_torch/disaggregation/native/rank_info.py +++ b/tensorrt_llm/_torch/disaggregation/native/rank_info.py @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -from dataclasses import asdict, dataclass +from dataclasses import asdict, dataclass, replace from typing import List, Optional import msgpack @@ -49,6 +49,14 @@ class RankInfo: aux_meta: Optional[AuxBufferMeta] = None page_table: Optional[KVCachePageTable] = None bounded_replay_on_generation: bool = False + # A standalone drafter's separate KV pool (DFlash / DSpark context window), shipped as a second + # block set when the disagg draft-window transfer is engaged; None everywhere else. Serialized + # only when set, so a rank without it emits exactly the bytes it always did. + draft_attention: Optional[AttentionInfo] = None + draft_page_table: Optional[KVCachePageTable] = None + # Set on a rank that takes part in the draft-window exchange but will not RECEIVE a window + # (the reason, e.g. "sync_path_tp"); senders then ship it the target set only. + draft_receive_disabled: Optional[str] = None @property def tp_size_per_dp_group(self) -> int: @@ -56,24 +64,45 @@ def tp_size_per_dp_group(self) -> int: return self.tp_size return self.tp_size // self.dp_size if self.attention.enable_attention_dp else self.tp_size + @property + def has_draft_kv(self) -> bool: + return self.draft_attention is not None and self.draft_page_table is not None + + def draft_view(self) -> Optional["RankInfo"]: + """This rank with the draft pool's geometry in place of the target's, or None without one. + + Topology (TP/PP/DP/CP, layer split, endpoints) is unchanged, so every overlap, routing, + policy and mapper path built for the target serves the draft set as is. + """ + if not self.has_draft_kv: + return None + return replace( + self, + attention=self.draft_attention, + page_table=self.draft_page_table, + draft_attention=None, + draft_page_table=None, + draft_receive_disabled=None, + ) + def to_bytes(self) -> bytes: data = asdict(self) data["attention"] = self.attention.to_dict() if self.attention is not None else None data["aux_meta"] = self.aux_meta.to_dict() if self.aux_meta is not None else None data["page_table"] = self.page_table.to_dict() if self.page_table is not None else None + if self.has_draft_kv: + data["draft_attention"] = self.draft_attention.to_dict() + data["draft_page_table"] = self.draft_page_table.to_dict() + else: + # Absent rather than null: a peer built before these fields parses the message. + data.pop("draft_attention", None) + data.pop("draft_page_table", None) + if self.draft_receive_disabled is None: + data.pop("draft_receive_disabled", None) return msgpack.packb(data) - @classmethod - def from_kv_cache_manager( - cls, - instance_name: str, - kv_cache_manager: KVCacheManager, - device_id: int, - aux_buffer_meta: Optional[AuxBufferMeta] = None, - ) -> "RankInfo": - m = kv_cache_manager.mapping - kvm = kv_cache_manager - enable_attention_dp = m.enable_attention_dp + @staticmethod + def _attention_info(kvm: KVCacheManager, enable_attention_dp: bool) -> AttentionInfo: # Keep AttentionInfo on attention-free PP stages so it can still carry # the attention-DP topology used by Mamba transfers. A zero head count # means that this rank has no local attention cache; AttentionPolicy @@ -87,6 +116,32 @@ def from_kv_cache_manager( if bytes_for_eight_elements % 8 == 0 else bytes_for_eight_elements / 8 ) + return AttentionInfo( + kv_heads_per_rank=kv_heads_per_rank, + tokens_per_block=kvm.tokens_per_block, + dims_per_head=kvm.head_dim, + element_bytes=element_bytes, + enable_attention_dp=enable_attention_dp, + is_mla=kvm.kv_factor == 1, + ) + + @classmethod + def from_kv_cache_manager( + cls, + instance_name: str, + kv_cache_manager: KVCacheManager, + device_id: int, + aux_buffer_meta: Optional[AuxBufferMeta] = None, + draft_kv_cache_manager: Optional[KVCacheManager] = None, + ) -> "RankInfo": + m = kv_cache_manager.mapping + kvm = kv_cache_manager + enable_attention_dp = m.enable_attention_dp + draft_attention = None + draft_page_table = None + if draft_kv_cache_manager is not None: + draft_attention = cls._attention_info(draft_kv_cache_manager, enable_attention_dp) + draft_page_table = build_page_table_from_manager(draft_kv_cache_manager) return cls( instance_name=instance_name, instance_rank=m.rank, @@ -103,17 +158,12 @@ def from_kv_cache_manager( sender_endpoints=[], self_endpoint="", transfer_engine_info=bytes(), - attention=AttentionInfo( - kv_heads_per_rank=kv_heads_per_rank, - tokens_per_block=kvm.tokens_per_block, - dims_per_head=kvm.head_dim, - element_bytes=element_bytes, - enable_attention_dp=enable_attention_dp, - is_mla=kvm.kv_factor == 1, - ), + attention=cls._attention_info(kvm, enable_attention_dp), aux_meta=aux_buffer_meta, page_table=build_page_table_from_manager(kvm), bounded_replay_on_generation=getattr(kvm, "bounded_replay_on_generation", False), + draft_attention=draft_attention, + draft_page_table=draft_page_table, ) @classmethod @@ -125,4 +175,8 @@ def from_bytes(cls, data: bytes) -> "RankInfo": unpacked["page_table"] = KVCachePageTable.from_dict(unpacked["page_table"]) if unpacked.get("aux_meta") is not None: unpacked["aux_meta"] = AuxBufferMeta.from_dict(unpacked["aux_meta"]) + if unpacked.get("draft_attention") is not None: + unpacked["draft_attention"] = AttentionInfo.from_dict(unpacked["draft_attention"]) + if unpacked.get("draft_page_table") is not None: + unpacked["draft_page_table"] = KVCachePageTable.from_dict(unpacked["draft_page_table"]) return cls(**unpacked) diff --git a/tensorrt_llm/_torch/disaggregation/native/transfer.py b/tensorrt_llm/_torch/disaggregation/native/transfer.py index 56c61cf1ab59..c221351141c5 100644 --- a/tensorrt_llm/_torch/disaggregation/native/transfer.py +++ b/tensorrt_llm/_torch/disaggregation/native/transfer.py @@ -58,6 +58,7 @@ ) from tensorrt_llm._torch.disaggregation.native.auxiliary import ( AuxBuffer, + AuxBufferMeta, build_aux_transfer_layout, get_non_empty_aux_indices, ) @@ -79,7 +80,12 @@ from tensorrt_llm._torch.disaggregation.nixl.agent import NixlTransferAgent from tensorrt_llm._torch.disaggregation.resource.kv_extractor import KVRegionExtractorV1 from tensorrt_llm._torch.disaggregation.resource.page import CacheKind, KVCachePageTable, MapperKind -from tensorrt_llm._torch.disaggregation.resource.utils import get_unique_pool_memory_descs +from tensorrt_llm._torch.disaggregation.resource.utils import ( + get_layer_byte_ranges, + get_pool_view_global_layer_ids, + get_pool_view_slot_bytes, + get_unique_pool_memory_descs, +) from tensorrt_llm._torch.pyexecutor.llm_request import LlmRequest from tensorrt_llm._torch.pyexecutor.resource_manager import KVCacheManager from tensorrt_llm._utils import CUASSERT, nvtx_range @@ -122,23 +128,30 @@ class RecvReqInfo: aux_slot: Optional[int] = None slice_id: Optional[int] = None bounce_dst_base: Optional[int] = None + # The receiver's draft-pool slots per draft layer group (see Chunk), or None when no draft set + # travels. Sent only when set, so a receiver without one emits exactly the old message. + draft_block_ids_per_layer_groups: Optional[list[np.ndarray]] = None def to_bytes(self) -> bytes: - return msgpack.packb( - { - "sender_req_id": self.sender_req_id, - "instance_name": self.instance_name, - "instance_rank": self.instance_rank, - "block_ids_per_layer_groups": [ - arr.tobytes() for arr in self.block_ids_per_layer_groups - ], - "unique_rid": self.unique_rid, - "dst_start_token": self.dst_start_token, - "aux_slot": self.aux_slot, - "slice_id": self.slice_id, - "bounce_dst_base": self.bounce_dst_base, - } - ) + data = { + "sender_req_id": self.sender_req_id, + "instance_name": self.instance_name, + "instance_rank": self.instance_rank, + "block_ids_per_layer_groups": [ + arr.tobytes() for arr in self.block_ids_per_layer_groups + ], + "unique_rid": self.unique_rid, + "dst_start_token": self.dst_start_token, + "aux_slot": self.aux_slot, + "slice_id": self.slice_id, + "bounce_dst_base": self.bounce_dst_base, + } + if self.draft_block_ids_per_layer_groups is not None: + data["draft_block_ids_per_layer_groups"] = [ + np.asarray(arr, dtype=np.int64).tobytes() + for arr in self.draft_block_ids_per_layer_groups + ] + return msgpack.packb(data) @classmethod def from_bytes(cls, data: bytes) -> "RecvReqInfo": @@ -146,6 +159,11 @@ def from_bytes(cls, data: bytes) -> "RecvReqInfo": d["block_ids_per_layer_groups"] = [ np.frombuffer(b, dtype=np.int64).copy() for b in d["block_ids_per_layer_groups"] ] + if d.get("draft_block_ids_per_layer_groups") is not None: + d["draft_block_ids_per_layer_groups"] = [ + np.frombuffer(b, dtype=np.int64).copy() + for b in d["draft_block_ids_per_layer_groups"] + ] return cls(**d) @@ -911,12 +929,199 @@ class _SessionQuiescence: endpoint: Optional[str] +def _peer_has_draft_kv(peer_ri: RankInfo) -> bool: + # Strictly True: rank-info stand-ins without the property count as target-only peers. + return getattr(peer_ri, "has_draft_kv", False) is True + + +def check_draft_kv_peer( + draft_registrar: Optional[PeerRegistrar], peer_ri: RankInfo +) -> Optional[RankInfo]: + """The peer's draft view when both sides ship the drafter context window, else None. + + Never raises: a peer pair where only one side engages the window + (``draft_peer_not_engaged``) moves the target set only. + """ + if draft_registrar is None or not _peer_has_draft_kv(peer_ri): + return None + return peer_ri.draft_view() + + +def draft_peer_not_engaged( + draft_registrar: Optional[PeerRegistrar], peer_ri: RankInfo +) -> Optional[str]: + """Why the drafter window cannot travel because exactly one side engages it, or None. + + E.g. a context server on a model without ``RuntimeHooks.disagg_draft_kv`` (or with the kill + switch) facing a generation server that has it. The pair then degrades to target-only + transfers: the window is disabled toward that peer on both sides, the generation side seeds an + empty drafter window, and the byte accounting stays exact. Not a target-path incompatibility. + """ + local_has = draft_registrar is not None + peer_has = _peer_has_draft_kv(peer_ri) + if local_has == peer_has: + return None + return ( + f"local rank {'engages' if local_has else 'does not engage'} the drafter context window " + f"but peer {peer_ri.instance_name} (rank={peer_ri.instance_rank}) " + f"{'does not' if local_has else 'does'}" + ) + + +def aux_meta_without_draft_ctx(meta: AuxBufferMeta) -> AuxBufferMeta: + """The auxiliary layout without its trailing drafter-context buffer. + + A rank that engages the drafter window appends exactly one buffer, ``(ctx_len, ctx_base, + valid)`` per slot, after the common ones (``_make_aux_buffer(draft_ctx=True)``); toward a peer + that does not engage it only the common buffers travel. + """ + return AuxBufferMeta( + ptrs=meta.ptrs[:-1], + size=meta.size[:-1], + item_sizes=meta.item_sizes[:-1], + device=meta.device, + ) + + +# draft_kv_fallback reasons counted here (disaggregation/draft_kv.py names the others). +_PEER_DRAFT_GEOMETRY = "peer_draft_geometry" +_PEER_NOT_ENGAGED = "peer_not_engaged" +_DRAFT_FRAGMENT_ERROR = "draft_fragment_error" +_CTX_RANK_DIVERGENCE = "ctx_rank_divergence" +_RECEIVER_NOT_ASKED = "receiver_not_asked" + + +def draft_window_bytes(page_table: Optional[KVCachePageTable], draft_block_ids) -> int: + """Local-rank bytes of the drafter context window a draft table names (0 for none).""" + if draft_block_ids is None or page_table is None: + return 0 + total = 0 + for lg_id, block_ids in enumerate(draft_block_ids): + n = int((np.asarray(block_ids) >= 0).sum()) + if n: + for pool_view in page_table.layer_groups[lg_id].pool_views: + total += n * get_pool_view_slot_bytes(pool_view) + return total + + +def _draft_view_pairs(self_table: KVCachePageTable, peer_table: KVCachePageTable): + """Pair every draft view with exactly one peer view, both ways: ``(pairs, None)``, or + ``(None, why)``. + + The strict pairing of the upstream per-view transfer (#20709), for the drafter's tables only: + a view with no same-role peer view, several candidates, or a peer view left unpaired would + otherwise move a silently partial window. + """ + pairs, used = [], set() + for self_lg_idx, self_lg in enumerate(self_table.layer_groups): + for self_pi, self_pv in enumerate(self_lg.pool_views): + self_layers = set(get_pool_view_global_layer_ids(self_pv, self_lg)) + candidates = [ + (peer_lg_idx, peer_pi, peer_pv) + for peer_lg_idx, peer_lg in enumerate(peer_table.layer_groups) + for peer_pi, peer_pv in enumerate(peer_lg.pool_views) + if peer_lg.kind == self_lg.kind + and peer_pv.pool_role == self_pv.pool_role + and self_layers & set(get_pool_view_global_layer_ids(peer_pv, peer_lg)) + ] + if len(candidates) != 1: + return None, ( + f"draft view {sorted(self_pv.pool_role)} (lg={self_lg_idx}, pool={self_pi}) " + f"matches {len(candidates)} peer views" + ) + peer_lg_idx, peer_pi, peer_pv = candidates[0] + peer_layers = set( + get_pool_view_global_layer_ids(peer_pv, peer_table.layer_groups[peer_lg_idx]) + ) + if peer_layers != self_layers or peer_pv.mapper_kind != self_pv.mapper_kind: + return None, ( + f"draft view {sorted(self_pv.pool_role)} differs from its peer view in " + "layers or mapper kind" + ) + if (peer_lg_idx, peer_pi) in used: + return None, f"peer draft view (lg={peer_lg_idx}, pool={peer_pi}) paired twice" + used.add((peer_lg_idx, peer_pi)) + pairs.append(((self_lg_idx, self_pi, self_pv), (peer_lg_idx, peer_pi, peer_pv))) + peer_views = sum(len(lg.pool_views) for lg in peer_table.layer_groups) + if len(used) != peer_views: + return None, f"{peer_views - len(used)} peer draft view(s) left unpaired" + return pairs, None + + +def draft_peer_mismatch( + self_draft_ri: RankInfo, peer_draft_ri: RankInfo, overlaps=None +) -> Optional[str]: + """Why two peers' drafter windows cannot be exchanged, or None when they can. + + A pure function of both sides' draft geometry (and of the rank overlaps both sides compute + alike), so the sender's and the receiver's registration reach the same verdict. A mismatch + disables the draft set for that peer pair only; the target transfer is untouched. Ported from + the upstream per-view transfer's registration checks (#20709): bytes per head, strict view + pairing and the draft layer set, plus PP 1 and the draft/target overlap. + """ + if self_draft_ri.pp_size != 1 or peer_draft_ri.pp_size != 1: + # The draft view copies the target's PP split, which says nothing of the draft layers. + return f"pipeline parallelism (pp={self_draft_ri.pp_size}/{peer_draft_ri.pp_size})" + a, b = self_draft_ri.attention, peer_draft_ri.attention + if a.dims_per_head != b.dims_per_head: + return f"head_dim {a.dims_per_head} != {b.dims_per_head}" + if a.element_bytes != b.element_bytes or a.is_mla != b.is_mla: + return ( + f"dtype ({a.element_bytes} B, mla={a.is_mla}) != ({b.element_bytes} B, mla={b.is_mla})" + ) + if a.tokens_per_block != b.tokens_per_block: + return f"tokens_per_block {a.tokens_per_block} != {b.tokens_per_block}" + self_table, peer_table = self_draft_ri.page_table, peer_draft_ri.page_table + self_layers = {ll.global_layer_id for lg in self_table.layer_groups for ll in lg.local_layers} + peer_layers = {ll.global_layer_id for lg in peer_table.layer_groups for ll in lg.local_layers} + if self_layers != peer_layers: + return f"draft layers {sorted(self_layers)} != {sorted(peer_layers)}" + pairs, why = _draft_view_pairs(self_table, peer_table) + if pairs is None: + return why + for self_side, peer_side in pairs: + if self_side[2].mapper_kind == MapperKind.REPLICATED: + continue + per_head = [] + for (lg_idx, pool_idx, pool_view), heads in ( + (self_side, a.kv_heads_per_rank), + (peer_side, b.kv_heads_per_rank), + ): + layer_bytes = get_layer_byte_ranges(pool_view)[1] + buffers = PeerRegistrar._get_buffers_per_layer( + pool_view, layer_group_id=lg_idx, pool_idx=pool_idx + ) + if heads <= 0 or layer_bytes % (buffers * heads): + return f"draft view {sorted(pool_view.pool_role)} is not a whole number of heads" + per_head.append(layer_bytes // (buffers * heads)) + if per_head[0] != per_head[1]: + return f"bytes per draft head {per_head[0]} != {per_head[1]}" + if overlaps is not None and list(overlaps[0]) != list(overlaps[1]): + return f"draft overlap ranks {list(overlaps[1])} != target {list(overlaps[0])}" + return None + + +def draft_overlaps(target_registrar, draft_registrar, peer_ri: RankInfo, peer_draft_ri: RankInfo): + """Target and draft overlap ranks toward every DP group of a peer, as two flat lists.""" + target, draft = [], [] + for dp_rank in range(max(1, int(getattr(peer_ri, "dp_size", 1) or 1))): + target += target_registrar.get_peer_overlap(peer_ri, dp_rank).ranks + draft += draft_registrar.get_peer_overlap(peer_draft_ri, dp_rank).ranks + return target, draft + + class Sender(SenderBase): # Time-to-live for orphaned RecvReqInfo entries (seconds). # In gen-first ADP broadcast, non-assigned DP ranks accumulate # RecvReqInfo that never gets consumed. Entries older than this # are evicted during periodic sweeps. _STALE_REQ_INFO_TTL_S = 120.0 + # The standalone drafter's pool registrar (see TransferWorker); None ships the target set only. + _draft_registrar: Optional[PeerRegistrar] = None + # draft_kv counters of the context side (disaggregation/draft_kv.py), set by the transceiver. + _draft_kv_counters = None + # Peer key -> (counter reason, why) the drafter window is disabled toward that peer. + _draft_disabled_peers: Optional[Dict[str, tuple]] = None def __init__( self, @@ -925,8 +1130,10 @@ def __init__( bounce=None, enforce_physical_ownership: bool = False, retirement_watchdog: Optional[RetirementWatchdog] = None, + draft_registrar: Optional[PeerRegistrar] = None, ) -> None: self._registrar = peer_registrar + self._draft_registrar = draft_registrar self._device_id = peer_registrar.self_rank_info.device_id self._agent = agent self._bounce = bounce @@ -1706,54 +1913,32 @@ def _pair_ordinals( idx = np.flatnonzero(mask) return src_block_ids[idx], dst_block_ids[idx] - @nvtx_range("_build_kv_write_meta") - def _build_kv_write_meta(self, task: KVSendTask, req_info: RecvReqInfo) -> WriteMeta: - peer_ri = self._registrar.get_peer_rank_info(req_info.instance_name, req_info.instance_rank) - self_ri = self._registrar.self_rank_info - token_range = task._chunk.token_range - # Only a piece short of the whole prompt runs the chunk projection below, and that is the - # part with no answer under CP; a sole piece covering everything takes the untouched path. - is_partial = token_range.start > 0 or token_range.end < task._prompt_len - if is_partial and (self_ri.cp_size > 1 or peer_ri.cp_size > 1): - raise ValueError( - "enable_pipelined_transfer is not supported with context parallelism " - f"(sender cp_size={self_ri.cp_size}, receiver cp_size={peer_ri.cp_size})" - ) - timer = task._perf_timer - if timer: - timer.record_prepare_args_start(peer_ri.instance_rank) - targets = self._registrar.get_peer_overlap(peer_ri, peer_ri.dp_rank) - expected_transfers = len(targets.ranks) - # Recorded on the task as well: the count on the write meta settles the task, while the - # handle has to answer how many writes are still out after one of them has already failed. - task.expected_transfers = expected_transfers - - # Aggregate fragment pointers from all matching pool pairs. - # Each pool pair produces one or more region pairs (rp), each containing - # a numpy array of src/dst pointers and a uniform bytes_per_region. - src_frag_parts: list[np.ndarray] = [] - dst_frag_parts: list[np.ndarray] = [] - # Instead of calling np.full() per region pair to build a size array and - # then np.concatenate() all of them, we record (count, bytes_per_region) - # tuples and construct the final sizes array with a single np.repeat(). - # For 48k+ items this avoids many small allocations in the hot loop. - size_specs: list[tuple[int, int]] = [] - dst_device_id = peer_ri.device_id - extractor = self._registrar.self_extractor - peer_extractor = self._registrar.peer_extractor( - peer_ri.instance_name, peer_ri.instance_rank - ) - pool_mapping = self._registrar.get_pool_mapping(peer_ri) - dst_block_ids_per_groups = req_info.block_ids_per_layer_groups - src_block_ids_per_groups = task._chunk.block_ids_per_layer_groups + @staticmethod + def _collect_pool_fragments( + registrar: PeerRegistrar, + peer_ri: RankInfo, + targets: PeerOverlap, + src_block_ids_per_groups: list[np.ndarray], + dst_block_ids_per_groups: list[np.ndarray], + excluded_pool_views, + dst_start_token: Optional[int], + src_frag_parts: list[np.ndarray], + dst_frag_parts: list[np.ndarray], + size_specs: list[tuple[int, int]], + ) -> None: + """Append the fragments of every pool pair ``registrar`` matches with ``peer_ri``. - # Aggregate fragments from all matching pools using numpy concatenation. - # Send ownership is per pool: replicated pools elect one fan-in - # owner, sharded pools keep head-duplication routing. + Send ownership is per pool: replicated pools elect one fan-in owner, sharded pools keep + head-duplication routing. + """ + self_ri = registrar.self_rank_info + extractor = registrar.self_extractor + peer_extractor = registrar.peer_extractor(peer_ri.instance_name, peer_ri.instance_rank) + pool_mapping = registrar.get_pool_mapping(peer_ri) for (self_lg, self_pi), (peer_lg, peer_pi) in pool_mapping.items(): - if (self_lg, self_pi) in task._chunk.excluded_pool_views: + if (self_lg, self_pi) in excluded_pool_views: continue - if not self._registrar.should_send_pool(targets, peer_ri, self_lg, self_pi): + if not registrar.should_send_pool(targets, peer_ri, self_lg, self_pi): continue lg_info = extractor.page_table.layer_groups[self_lg] @@ -1764,7 +1949,7 @@ def _build_kv_write_meta(self, task: KVSendTask, req_info: RecvReqInfo) -> Write if lg_info.kind == CacheKind.STATE: if src_block_ids.size == 0 or dst_block_ids.size == 0: continue - if not MambaPolicy.is_paired(self._registrar.self_rank_info, peer_ri): + if not MambaPolicy.is_paired(self_ri, peer_ri): continue src_region = extractor.extract_slot(int(src_block_ids[0]), self_lg, self_pi) dst_region = peer_extractor.extract_slot(int(dst_block_ids[0]), peer_lg, peer_pi) @@ -1778,9 +1963,7 @@ def _build_kv_write_meta(self, task: KVSendTask, req_info: RecvReqInfo) -> Write # rejected under helix. src_block_ids = src_block_ids[peer_ri.cp_rank :: peer_ri.cp_size] tpb = extractor.page_table.tokens_per_block - dst_start_block = ( - req_info.dst_start_token // tpb if req_info.dst_start_token is not None else 0 - ) + dst_start_block = dst_start_token // tpb if dst_start_token is not None else 0 # Both tables are positional (index == block ordinal, -1 == not # held), so pairing is a plain intersection: SWA eviction, prefix # reuse, and pipelined chunk bounds are all already encoded as @@ -1794,7 +1977,7 @@ def _build_kv_write_meta(self, task: KVSendTask, req_info: RecvReqInfo) -> Write dst_region = peer_extractor.extract(dst_block_ids, peer_lg, peer_pi) # Map and collect fragments (same for all layer types) - mapper = self._registrar.get_kv_map(peer_ri, (self_lg, self_pi), (peer_lg, peer_pi)) + mapper = registrar.get_kv_map(peer_ri, (self_lg, self_pi), (peer_lg, peer_pi)) region_pair = mapper.map(src_region, dst_region) region_pairs = region_pair if isinstance(region_pair, list) else [region_pair] for rp in region_pairs: @@ -1802,6 +1985,169 @@ def _build_kv_write_meta(self, task: KVSendTask, req_info: RecvReqInfo) -> Write dst_frag_parts.append(rp.dst.memory.ptrs) size_specs.append((rp.src.memory.ptrs.size, rp.src.memory.bytes_per_region)) + def _collect_draft_fragments( + self, + task: KVSendTask, + req_info: RecvReqInfo, + peer_ri: RankInfo, + targets: PeerOverlap, + src_frag_parts: list[np.ndarray], + dst_frag_parts: list[np.ndarray], + size_specs: list[tuple[int, int]], + ) -> None: + """Append the standalone drafter's context window to the same write, on the last piece. + + The draft pool goes through its own registrar: its pools can share the target's role labels + and global layer ids, so one pool mapping would cross-match them. Its fragments always go + direct, since a bounce region is sized from the target's page table alone. + + A request whose window does not travel carries no draft table on the side that decided so + (the context side, signalled to the receiver ahead of its request in context-first). Any + other inconsistency also leaves the window out rather than failing the request: the target + write is unaffected and the generation side's drafter starts empty. + """ + src_draft = getattr(task._chunk, "draft_block_ids_per_layer_groups", None) + dst_draft = getattr(req_info, "draft_block_ids_per_layer_groups", None) + draft_registrar = self._draft_registrar + counters = self._draft_kv_counters + disabled = (self._draft_disabled_peers or {}).get( + peer_ri.instance_name + str(peer_ri.instance_rank) + ) + if draft_registrar is None: + # This rank does not engage the window. A peer that does (registration recorded it as + # peer_not_engaged) asks for none, so only the target set moves. + if not task._chunk.is_last: + return + if dst_draft is not None: + # The receiver disables the window toward this sender at registration, so this + # needs a peer that did not: leave the window out (its attestation then falls the + # window back) rather than fail the request. + logger.warning_once( + f"rid={req_info.unique_rid}: the receiver asks for the drafter context window " + "but this sender does not engage it; sending the target set only", + key=f"f0a-draft-unasked-{peer_ri.instance_name}", + ) + if counters is not None and (disabled is not None or dst_draft is not None): + counters.record_fallback( + disabled[0] if disabled is not None else _PEER_NOT_ENGAGED, + f"rid={req_info.unique_rid}", + ) + return + if not task._chunk.is_last: + return + if disabled is not None: + # Registration disabled the window toward this peer, and the receiver asked for none. + if src_draft is not None and counters is not None: + counters.record_fallback(disabled[0], f"rid={req_info.unique_rid} {disabled[1]}") + return + if src_draft is None or dst_draft is None: + if (src_draft is None) != (dst_draft is None) and counters is not None: + # Either this context rank decided otherwise than the rank whose response the + # receiver followed (it ships nothing the receiver asked for), or the receiver asked + # for none although this rank ships: its state never arrived (receiver_not_asked). + # No draft bytes move either way; the receiver's byte attestation catches the first. + counters.record_fallback( + _RECEIVER_NOT_ASKED if src_draft is not None else _CTX_RANK_DIVERGENCE, + f"rid={req_info.unique_rid} peer_rank={peer_ri.instance_rank} " + f"sender_ships={src_draft is not None}", + ) + return + if req_info.bounce_dst_base is not None: + logger.warning( + f"rid={req_info.unique_rid}: the receiver reserved a bounce region for a write " + "that carries the drafter context window; leaving the window out" + ) + return + peer_draft_ri = draft_registrar.get_peer_rank_info( + peer_ri.instance_name, peer_ri.instance_rank + ) + draft_targets = draft_registrar.get_peer_overlap(peer_draft_ri, peer_draft_ri.dp_rank) + # Registration already compared the overlaps (draft_peer_mismatch); a cheap guard. + if list(draft_targets.ranks) != list(targets.ranks): + logger.warning_once( + f"draft overlap ranks {draft_targets.ranks} differ from target overlap ranks " + f"{targets.ranks} (rid={req_info.unique_rid}); leaving the drafter window out", + key=f"f0a-draft-overlap-{peer_ri.instance_name}", + ) + return + marks = (len(src_frag_parts), len(dst_frag_parts), len(size_specs)) + try: + Sender._collect_pool_fragments( + draft_registrar, + peer_draft_ri, + draft_targets, + src_draft, + dst_draft, + (), + None, + src_frag_parts, + dst_frag_parts, + size_specs, + ) + except Exception as error: # noqa: BLE001 - only the drafter window may be lost here + # Drop whatever the draft set appended; the target fragments before it stay, and the + # receiver's byte attestation falls the window back on every generation rank. + del src_frag_parts[marks[0] :] + del dst_frag_parts[marks[1] :] + del size_specs[marks[2] :] + logger.error( + f"rid={req_info.unique_rid}: drafter window fragments failed ({error}); " + "sending the target set only" + ) + if counters is not None: + counters.record_fallback(_DRAFT_FRAGMENT_ERROR, f"rid={req_info.unique_rid}") + + @nvtx_range("_build_kv_write_meta") + def _build_kv_write_meta(self, task: KVSendTask, req_info: RecvReqInfo) -> WriteMeta: + peer_ri = self._registrar.get_peer_rank_info(req_info.instance_name, req_info.instance_rank) + self_ri = self._registrar.self_rank_info + token_range = task._chunk.token_range + # Only a piece short of the whole prompt runs the chunk projection below, and that is the + # part with no answer under CP; a sole piece covering everything takes the untouched path. + is_partial = token_range.start > 0 or token_range.end < task._prompt_len + if is_partial and (self_ri.cp_size > 1 or peer_ri.cp_size > 1): + raise ValueError( + "enable_pipelined_transfer is not supported with context parallelism " + f"(sender cp_size={self_ri.cp_size}, receiver cp_size={peer_ri.cp_size})" + ) + timer = task._perf_timer + if timer: + timer.record_prepare_args_start(peer_ri.instance_rank) + targets = self._registrar.get_peer_overlap(peer_ri, peer_ri.dp_rank) + expected_transfers = len(targets.ranks) + # Recorded on the task as well: the count on the write meta settles the task, while the + # handle has to answer how many writes are still out after one of them has already failed. + task.expected_transfers = expected_transfers + + # Aggregate fragment pointers from all matching pool pairs. + # Each pool pair produces one or more region pairs (rp), each containing + # a numpy array of src/dst pointers and a uniform bytes_per_region. + src_frag_parts: list[np.ndarray] = [] + dst_frag_parts: list[np.ndarray] = [] + # Instead of calling np.full() per region pair to build a size array and + # then np.concatenate() all of them, we record (count, bytes_per_region) + # tuples and construct the final sizes array with a single np.repeat(). + # For 48k+ items this avoids many small allocations in the hot loop. + size_specs: list[tuple[int, int]] = [] + dst_device_id = peer_ri.device_id + + # Aggregate fragments from all matching pools using numpy concatenation. + Sender._collect_pool_fragments( + self._registrar, + peer_ri, + targets, + task._chunk.block_ids_per_layer_groups, + req_info.block_ids_per_layer_groups, + task._chunk.excluded_pool_views, + req_info.dst_start_token, + src_frag_parts, + dst_frag_parts, + size_specs, + ) + self._collect_draft_fragments( + task, req_info, peer_ri, targets, src_frag_parts, dst_frag_parts, size_specs + ) + if src_frag_parts: src_frags = np.concatenate(src_frag_parts) dst_frags = np.concatenate(dst_frag_parts) @@ -1859,6 +2205,14 @@ def _build_aux_write_meta(self, task: AuxSendTask, req_info: RecvReqInfo) -> Wri peer_ri.instance_name, peer_ri.instance_rank ) if layout is None: + # Toward a peer that does not engage the drafter window (or from a sender that + # does not), only the common buffers travel; the drafter-context slot is not read + # (the generation side's window is disabled toward this peer). + self_has = _peer_has_draft_kv(self._registrar.self_rank_info) + if self_has and not _peer_has_draft_kv(peer_ri): + src_aux_meta = aux_meta_without_draft_ctx(src_aux_meta) + elif _peer_has_draft_kv(peer_ri) and not self_has: + peer_aux_meta = aux_meta_without_draft_ctx(peer_aux_meta) layout = build_aux_transfer_layout(src_aux_meta, peer_aux_meta) self._registrar.cache_aux_transfer_layout( peer_ri.instance_name, peer_ri.instance_rank, layout @@ -1975,7 +2329,14 @@ def _register_peer_rank(self, _send_id: bytes, message: list[bytes]): with self._loaded_remote_agents_lock: if self._shutdown_requested: return + # The drafter window never refuses a peer: a geometry mismatch or a peer that engages + # it on one side only disables the window toward that peer (target set only). + peer_draft_ri = check_draft_kv_peer(self._draft_registrar, ri) self._registrar.register(ri.instance_name, ri.instance_rank, ri) + if peer_draft_ri is not None: + self._register_draft_peer(ri, peer_draft_ri) + else: + self._note_draft_peer_not_engaged(ri) self._agent.load_remote_agent( agent_name, ri.transfer_engine_info, @@ -1985,6 +2346,57 @@ def _register_peer_rank(self, _send_id: bytes, message: list[bytes]): f"Completed handling REGISTER_RANK_INFO for instance='{ri.instance_name}', rank={ri.instance_rank}" ) + def _note_draft_peer_not_engaged(self, ri: RankInfo) -> None: + """Disable the drafter window toward a peer when only one side engages it. + + The receiver reaches the same verdict (``Receiver._check_draft_peer``) and asks for no + draft pages, so only the target set moves; counted per request as ``peer_not_engaged``. + """ + why = draft_peer_not_engaged(self._draft_registrar, ri) + if why is None: + return + if self._draft_disabled_peers is None: + self._draft_disabled_peers = {} + self._draft_disabled_peers[ri.instance_name + str(ri.instance_rank)] = ( + _PEER_NOT_ENGAGED, + why, + ) + logger.warning( + f"F0a: drafter context window disabled toward peer {ri.instance_name} " + f"(rank={ri.instance_rank}): {why}; transferring the target set only" + ) + + def _register_draft_peer(self, ri: RankInfo, peer_draft_ri: RankInfo) -> None: + """Register the peer's drafter window, or disable it toward this peer alone. + + The receiver runs the same check (``draft_peer_mismatch``) and then asks for no draft + pages, so both sides agree and the byte accounting stays exact. + """ + reason = _PEER_DRAFT_GEOMETRY + try: + why = draft_peer_mismatch( + self._draft_registrar.self_rank_info, + peer_draft_ri, + draft_overlaps(self._registrar, self._draft_registrar, ri, peer_draft_ri), + ) + if why is None and getattr(ri, "draft_receive_disabled", None): + # The peer takes part but receives no window (e.g. its synchronous receive path + # cannot keep its TP ranks in step about it). + reason = ri.draft_receive_disabled + why = f"the peer receives no drafter window ({reason})" + if why is None: + self._draft_registrar.register(ri.instance_name, ri.instance_rank, peer_draft_ri) + except Exception as error: # noqa: BLE001 - the target set must survive any draft issue + why = f"draft registration failed: {error}" + if why is not None: + if self._draft_disabled_peers is None: + self._draft_disabled_peers = {} + self._draft_disabled_peers[ri.instance_name + str(ri.instance_rank)] = (reason, why) + logger.warning( + f"F0a: drafter context window disabled toward peer {ri.instance_name} " + f"(rank={ri.instance_rank}): {why}; transferring the target set only" + ) + def _handle_cancel_session(self, message: list[bytes]): unique_rid = int(message[1]) session = None @@ -2795,6 +3207,9 @@ def __del__(self): class KVRecvTask(_LogicalTask): + # Why the drafter window was not asked for (set by Receiver.dispatch_task), else None. + draft_disabled_reason: Optional[str] = None + def __init__( self, unique_rid: Optional[int], @@ -2971,6 +3386,14 @@ def print_perf_info(self, peer_rank: int, instance_name: str, instance_rank: int class Receiver(ReceiverBase): + # The standalone drafter's pool registrar (see TransferWorker); None receives the target set only. + _draft_registrar: Optional[PeerRegistrar] = None + # Context info endpoint -> (counter reason, why) the drafter window is disabled toward that peer. + _draft_disabled_peers: Optional[Dict[str, tuple]] = None + # draft_kv counters of the generation side, set by the transceiver on a rank that does not + # engage the window (an engaged rank counts in the transceiver as it seeds). + _draft_kv_counters = None + def __init__( self, peer_registrar: PeerRegistrar, @@ -2978,8 +3401,10 @@ def __init__( bounce=None, enforce_physical_ownership: bool = False, retirement_watchdog: Optional[RetirementWatchdog] = None, + draft_registrar: Optional[PeerRegistrar] = None, ): self._registrar = peer_registrar + self._draft_registrar = draft_registrar self._agent = agent self._bounce = bounce # Internal component gate only. Production wiring is added after the @@ -3130,6 +3555,9 @@ def _build_recv_req_info(self, task: KVRecvTask) -> RecvReqInfo: dst_start_token=None, aux_slot=task._aux_slot, slice_id=task.slice_id, + draft_block_ids_per_layer_groups=getattr( + task._chunk, "draft_block_ids_per_layer_groups", None + ), ) @staticmethod @@ -3261,6 +3689,11 @@ def dispatch_task(self, task: KVRecvTask) -> None: self._registrar.self_extractor.page_table, ) ) + draft_ids = self._drop_disabled_draft_window(task, receiver_req, params) + if draft_ids is not None: + # The drafter context window rides the same write and always goes direct: a bounce + # region is sized from the target page table only. + allow_bounce = False # Recurrent (mamba/KDA) state rides the SAME coalesced write as the KV blocks, # so the bounce region must be sized for those bytes too. extra_bytes = 0 @@ -3411,6 +3844,7 @@ def _get_sender_info(self, params: DisaggregatedParams) -> RankInfo: self._incompatible_peers[info_endpoint] = msg raise PeerIncompatibleError(msg) from e + self._check_draft_peer(info_endpoint, sender_info) for endpoint in sender_info.sender_endpoints: dealer = self._get_or_connect_dealer(endpoint) rank_info = self._registrar.self_rank_info @@ -3422,6 +3856,84 @@ def _get_sender_info(self, params: DisaggregatedParams) -> RankInfo: else: return self._sender_ep_instance_map[info_endpoint] + def _drop_disabled_draft_window(self, task: KVRecvTask, receiver_req: RecvReqInfo, params): + """The draft table this request asks for, after dropping a disabled window. + + A window disabled toward the context peer (registration) or for this rank (it receives + none) is not asked for and its bytes are not expected, so the byte attestation stays exact; + the request then drafts from an empty window (the transceiver reads the reason back off + the task). + """ + draft_ids = getattr(receiver_req, "draft_block_ids_per_layer_groups", None) + peer_disabled = None + if self._draft_disabled_peers: + # Only a registered (so endpoint-carrying) peer can be disabled. Every other receive, + # draft or not, never needs the endpoint here: gen-first and ADP dispatches carry none. + endpoint = getattr(params, "ctx_info_endpoint", None) + if isinstance(endpoint, list): + endpoint = endpoint[0] if endpoint else None + if endpoint is not None: + peer_disabled = self._draft_disabled_peers.get(endpoint) + if draft_ids is None: + if ( + peer_disabled is not None + and self._draft_registrar is None + and self._draft_kv_counters is not None + ): + # This rank does not engage the window but the context peer does. + self._draft_kv_counters.record_fallback( + peer_disabled[0], f"rid={receiver_req.unique_rid}" + ) + return None + disabled = getattr(self._registrar.self_rank_info, "draft_receive_disabled", None) + if disabled is None and peer_disabled is not None: + disabled = peer_disabled[0] + if disabled is None: + return draft_ids + receiver_req.draft_block_ids_per_layer_groups = None + if task.expected_write_bytes is not None: + task.expected_write_bytes -= draft_window_bytes( + self._draft_registrar.self_extractor.page_table, draft_ids + ) + task.draft_disabled_reason = disabled + return None + + def _check_draft_peer(self, info_endpoint: str, sender_info: RankInfo) -> None: + """Disable the drafter window toward a context peer whose draft geometry does not match. + + The same verdict as the sender's (``draft_peer_mismatch``): requests to that peer then ask + for no draft pages, so neither side moves or expects any. + """ + not_engaged = draft_peer_not_engaged(self._draft_registrar, sender_info) + if not_engaged is not None: + if self._draft_disabled_peers is None: + self._draft_disabled_peers = {} + self._draft_disabled_peers[info_endpoint] = (_PEER_NOT_ENGAGED, not_engaged) + logger.warning( + f"F0a: drafter context window disabled toward context peer '{info_endpoint}': " + f"{not_engaged}; receiving the target set only" + ) + return + if self._draft_registrar is None: + return + try: + sender_draft = sender_info.draft_view() + why = draft_peer_mismatch( + self._draft_registrar.self_rank_info, + sender_draft, + draft_overlaps(self._registrar, self._draft_registrar, sender_info, sender_draft), + ) + except Exception as error: # noqa: BLE001 - the target set must survive any draft issue + why = f"draft geometry check failed: {error}" + if why is not None: + if self._draft_disabled_peers is None: + self._draft_disabled_peers = {} + self._draft_disabled_peers[info_endpoint] = (_PEER_DRAFT_GEOMETRY, why) + logger.warning( + f"F0a: drafter context window disabled toward context peer '{info_endpoint}': " + f"{why}; receiving the target set only" + ) + def send_cancel_to_senders(self, unique_rid: int, sender_endpoints: set[str]) -> set[str]: """Notify senders and return the endpoints whose cancel message was sent.""" message = [MessageType.CANCEL_SESSION, str(unique_rid).encode("ascii")] @@ -3836,6 +4348,30 @@ def prepare_receive( self._logical_outcomes.seal() return task + def kv_write_byte_counts(self) -> tuple[int, Optional[int]]: + """(attested, expected) destination bytes over this session's KV tasks; expected is None + when a task published no expected total.""" + with self.lock: + verified = sum(task.verified_write_bytes for task in self._kv_tasks) + expected = [task.expected_write_bytes for task in self._kv_tasks] + return verified, None if any(e is None for e in expected) else sum(expected) + + def draft_disabled_reason(self) -> Optional[str]: + """Why this session asked its context peer for no drafter window, or None.""" + with self.lock: + return next( + (t.draft_disabled_reason for t in self._kv_tasks if t.draft_disabled_reason), + None, + ) + + def peek_draft_ctx(self) -> Optional[tuple[int, int, bool]]: + """The drafter-context state in this session's aux slot, without consuming it.""" + get_draft_ctx = getattr(self._aux_buffer, "get_slot_draft_ctx", None) + if not callable(get_draft_ctx) or self.aux_slot is None: + return None + state = get_draft_ctx(self.aux_slot) + return state if isinstance(state, tuple) else None + def kv_write_verified(self) -> bool: """True when every KV task's published destination bytes are attested written. @@ -4156,6 +4692,11 @@ def unpack_aux(self, request: LlmRequest) -> None: ) request.py_first_gen_tokens = first_gen_tokens # type: ignore[attr-defined] request.py_draft_tokens = draft_tokens # type: ignore[attr-defined] + # The context side's drafter (ctx_len, ctx_base, valid), present only with that aux slot. + get_draft_ctx = getattr(self._aux_buffer, "get_slot_draft_ctx", None) + draft_ctx = get_draft_ctx(self.aux_slot) if callable(get_draft_ctx) else None + if isinstance(draft_ctx, tuple): + request.py_draft_ctx_aux = draft_ctx # type: ignore[attr-defined] if request.py_disaggregated_params is not None: request.py_disaggregated_params.ctx_usage = { "prompt_tokens": prompt_tokens, @@ -4476,17 +5017,22 @@ def _create_nixl_agent( def _make_aux_buffer( - kvm: KVCacheManager, max_slots: int, max_draft_len: Optional[int] = None + kvm: KVCacheManager, + max_slots: int, + max_draft_len: Optional[int] = None, + draft_ctx: bool = False, ) -> Optional[AuxBuffer]: if max_slots <= 0: return None if max_draft_len is None: max_draft_len = max(0, int(getattr(kvm, "max_draft_len", 0))) + kwargs = {"draft_ctx": True} if draft_ctx else {} return AuxBuffer( max_slot_num=max_slots, beam_width=max(1, int(getattr(kvm, "max_beam_width", 1))), max_draft_len=max_draft_len, device="cpu", + **kwargs, ) @@ -4513,9 +5059,18 @@ class TransferWorkerConfig: # trailing _BYTES, lowercased); dict > env > default. Ignored when # agent_buffer_size_mb == 0. agent_bounce_params: Optional[Dict[str, str]] = None + # A standalone drafter's separate KV pool whose context window ships with the + # target's KV (disaggregation/draft_kv.py decides); None moves the target only. + draft_kv_cache_manager: Optional[KVCacheManager] = None + # Why this rank takes part in the draft-window exchange but receives no window, or None. + draft_receive_disabled: Optional[str] = None class TransferWorker: + # Class-level defaults keep workers built without __init__ on the target-only path. + _draft_extractor: Optional[KVRegionExtractorV1] = None + _draft_registrar: Optional[PeerRegistrar] = None + def __init__(self, config: TransferWorkerConfig): self._config = config self._retirement_watchdog = None @@ -4529,15 +5084,22 @@ def __init__(self, config: TransferWorkerConfig): ) self._retirement_watchdog = RetirementWatchdog(self._on_quiescence_fatal) kvm = config.kv_cache_manager + draft_kvm = config.draft_kv_cache_manager self._aux_buffer = _make_aux_buffer( - kvm, config.max_concurrent_sessions, config.max_draft_len + kvm, + config.max_concurrent_sessions, + config.max_draft_len, + draft_ctx=draft_kvm is not None, ) self._rank_info = RankInfo.from_kv_cache_manager( config.instance_name, kvm, config.device_id, self._aux_buffer.meta if self._aux_buffer is not None else None, + draft_kv_cache_manager=draft_kvm, ) + if draft_kvm is not None and config.draft_receive_disabled: + self._rank_info.draft_receive_disabled = config.draft_receive_disabled self._setup_peer_infrastructure(kvm) self._setup_transfer_engine() if self._retirement_watchdog is not None: @@ -4602,6 +5164,12 @@ def _setup_peer_infrastructure(self, kvm: KVCacheManager): self._rank_info_server = RankInfoServer(self._rank_info) if kvm.mapping.rank == 0 else None self._kv_extractor = KVRegionExtractorV1(kvm) self._peer_registrar = PeerRegistrar(self._rank_info, self._kv_extractor) + draft_ri = self._rank_info.draft_view() + if draft_ri is not None: + # Separate from the target's: the draft pools can carry the same role labels and + # overlapping global layer ids, which one pool mapping would cross-match. + self._draft_extractor = KVRegionExtractorV1(draft_ri.page_table) + self._draft_registrar = PeerRegistrar(draft_ri, self._draft_extractor) def _setup_transfer_engine(self): torch.cuda.set_device(self._config.device_id) @@ -4649,6 +5217,7 @@ def _setup_transfer_engine(self): bounce=self._bounce, enforce_physical_ownership=self._config.enforce_physical_ownership, retirement_watchdog=self._retirement_watchdog, + draft_registrar=self._draft_registrar, ) self._receiver = Receiver( self._peer_registrar, @@ -4656,6 +5225,7 @@ def _setup_transfer_engine(self): bounce=self._bounce, enforce_physical_ownership=self._config.enforce_physical_ownership, retirement_watchdog=self._retirement_watchdog, + draft_registrar=self._draft_registrar, ) self._rank_info.transfer_engine_info = bytes(self._agent.get_local_agent_desc()) self._rank_info.self_endpoint = self._receiver.endpoint @@ -4677,6 +5247,14 @@ def _register_kv_cache(self): self._agent.register_memory(reg_memory_desc) logger.debug(f"Registered KV cache memory with transfer agent: {memory_descs}") self._registered_mem.append(reg_memory_desc) + draft_page_table = getattr(self._rank_info, "draft_page_table", None) + if draft_page_table is not None: + draft_descs = get_unique_pool_memory_descs(draft_page_table, self._rank_info.device_id) + if draft_descs: + reg_draft_desc = RegMemoryDescs("VRAM", draft_descs) + self._agent.register_memory(reg_draft_desc) + logger.debug(f"Registered draft KV cache memory with transfer agent: {draft_descs}") + self._registered_mem.append(reg_draft_desc) def _register_aux_buffer(self): assert self._aux_buffer is not None @@ -4707,6 +5285,12 @@ def page_table(self): assert self._rank_info is not None return self._rank_info.page_table + @property + def draft_page_table(self) -> Optional[KVCachePageTable]: + """The draft pool's page table when the drafter context window ships, else None.""" + rank_info = getattr(self, "_rank_info", None) + return getattr(rank_info, "draft_page_table", None) + def shutdown(self): if getattr(self, "_shutdown", False): return diff --git a/tensorrt_llm/_torch/disaggregation/orchestration/coordinator.py b/tensorrt_llm/_torch/disaggregation/orchestration/coordinator.py index c8aed02d61c2..0f695e72ab7e 100644 --- a/tensorrt_llm/_torch/disaggregation/orchestration/coordinator.py +++ b/tensorrt_llm/_torch/disaggregation/orchestration/coordinator.py @@ -69,6 +69,21 @@ def attach_ctx_usage(request: LlmRequest, response) -> None: response.result.ctx_usage = disagg_params.ctx_usage +class _ReceiveReadyEvents: + """The readiness events of one receive destination that spans two pools (the target's and a + standalone drafter's): ready only once every one has completed.""" + + def __init__(self, events: List["torch.cuda.Event"]) -> None: + self._events = events + + def query(self) -> bool: + return all(event.query() for event in self._events) + + def synchronize(self) -> None: + for event in self._events: + event.synchronize() + + class DisaggTransferCoordinator: """Disagg transfer entry points used by every executor loop variant. @@ -82,6 +97,7 @@ class DisaggTransferCoordinator: # Class-level default, so a coordinator assembled without ``__init__`` # keeps the blocking receive fence. _defer_unready_gen_receives = False + _draft_kv_cache_manager = None def __init__( self, @@ -350,6 +366,7 @@ def receive_gen_init(self, admitted: List[LlmRequest]) -> None: # does not block this thread on them; publication polls them instead. fences = self._take_receive_fences(admitted) if defer_unready else {} self._effects.prepare_gen_resources(admitted) + self._merge_draft_receive_fences(admitted, fences, defer_unready) # gen_only_no_context has no CTX worker, so mark each request as # transmission-complete immediately. @@ -396,6 +413,35 @@ def _take_receive_fences( return {} return {req.py_request_id: take(req) for req in requests} + def _merge_draft_receive_fences( + self, + requests: List[LlmRequest], + fences: Dict[int, Optional["torch.cuda.Event"]], + defer_unready: bool, + ) -> None: + """Fold the draft pool's readiness events into the receive fences. + + Only a draft pool that receives the drafter context window records one, during + ``prepare_gen_resources``. Deferred publication then polls both events; otherwise the + executor waits here, as the target pool's event is waited on in ``prepare_resources``. + """ + manager = self._draft_kv_cache_manager + if not getattr(manager, "_disagg_draft_kv_receive", False): + return + for req in requests: + draft_ready = manager.take_disagg_receive_ready(req) + if draft_ready is None: + continue + if not defer_unready: + draft_ready.synchronize() + continue + target_ready = fences.get(req.py_request_id) + fences[req.py_request_id] = ( + draft_ready + if target_ready is None + else _ReceiveReadyEvents([target_ready, draft_ready]) + ) + def _publish_ready_gen_receives(self) -> None: """Start the asynchronous receive of each waiting request whose admission readiness event has completed. diff --git a/tensorrt_llm/_torch/disaggregation/resource/utils.py b/tensorrt_llm/_torch/disaggregation/resource/utils.py index 153451b1b216..e4272c5e8b81 100644 --- a/tensorrt_llm/_torch/disaggregation/resource/utils.py +++ b/tensorrt_llm/_torch/disaggregation/resource/utils.py @@ -213,6 +213,46 @@ def find_replicated_role_mismatch( return sorted((sorted(role), gid) for role, gid in differing if gid in shared) +def get_replicated_role_layer_bytes( + page_table: KVCachePageTable, kind: CacheKind +) -> Dict[Tuple[FrozenSet[str], int], int]: + """``{(pool_role, global_layer_id): bytes_per_layer}`` of every REPLICATED view of *kind*.""" + sizes: Dict[Tuple[FrozenSet[str], int], int] = {} + for layer_group in page_table.layer_groups: + if layer_group.kind != kind: + continue + for pool_view in layer_group.pool_views: + if pool_view.mapper_kind != MapperKind.REPLICATED: + continue + _, bytes_per_layer = get_layer_byte_ranges(pool_view) + for global_layer_id in get_pool_view_global_layer_ids(pool_view, layer_group): + sizes[(pool_view.pool_role, global_layer_id)] = bytes_per_layer + return sizes + + +def find_replicated_size_mismatch( + self_page_table: Optional[KVCachePageTable], + peer_page_table: Optional[KVCachePageTable], + kind: CacheKind, +) -> List[Tuple[List[str], int, int, int]]: + """Replicated roles both sides declare on a layer but with different bytes per layer. + + A replicated view is copied byte for byte between pools of the same ``tokens_per_block``, + so the two sides must store it in the same format (e.g. the same index-key dtype). Returns + a sorted list of ``(sorted_role_names, global_layer_id, self_bytes, peer_bytes)``; empty + when the two sides agree or either page table is missing. + """ + if self_page_table is None or peer_page_table is None: + return [] + self_sizes = get_replicated_role_layer_bytes(self_page_table, kind) + peer_sizes = get_replicated_role_layer_bytes(peer_page_table, kind) + return sorted( + (sorted(role), gid, self_bytes, peer_sizes[(role, gid)]) + for (role, gid), self_bytes in self_sizes.items() + if (role, gid) in peer_sizes and self_bytes != peer_sizes[(role, gid)] + ) + + def get_layer_group_num_layers(layer_group: AttentionLayerGroup) -> int: """ Number of layers in *layer_group* diff --git a/tensorrt_llm/_torch/disaggregation/transceiver.py b/tensorrt_llm/_torch/disaggregation/transceiver.py index 3c9e4f245db7..a8bf771ad00d 100644 --- a/tensorrt_llm/_torch/disaggregation/transceiver.py +++ b/tensorrt_llm/_torch/disaggregation/transceiver.py @@ -17,6 +17,7 @@ import time import uuid from collections import Counter, defaultdict +from dataclasses import replace from itertools import chain from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, cast @@ -34,6 +35,23 @@ WaitResult, get_unique_rid, ) +from tensorrt_llm._torch.disaggregation.draft_kv import ( + FALLBACK_CTX_LEN_MISMATCH, + FALLBACK_CTX_RANK_DIVERGENCE, + FALLBACK_NO_STATE, + FALLBACK_NO_STORE_EVENT, + FALLBACK_PAGE_MISMATCH, + FALLBACK_REUSE_BASE, + FALLBACK_UNVERIFIED_WRITE, + TEST_SEND_BEFORE_EVENT_ENV, + DraftKvCounters, + decode_draft_ctx_state, + draft_dump_dir, + draft_receive_disabled_reason, + empty_window_state, + send_before_draft_ctx_event, + write_window_dump, +) from tensorrt_llm._torch.disaggregation.kv_cache_transceiver import ( CtxTransferStatus, GenTransferStatus, @@ -50,6 +68,7 @@ RxSession, TransferWorker, TransferWorkerConfig, + draft_window_bytes, ) from tensorrt_llm._torch.disaggregation.resource.cache_reuse import ( CacheReuseAdapter, @@ -231,6 +250,23 @@ def _validate_fp4_mla_bridge_profile( class KvCacheTransceiverV2(KvCacheTransceiver): + # The standalone drafter's context window (disaggregation/draft_kv.py). Class-level defaults + # keep a transceiver built without __init__, or without a draft manager, on the target-only path. + _draft_kv_cache_manager: Optional[KVCacheManager] = None + _draft_reuse_adapter: Optional[CacheReuseAdapter] = None + _draft_page_table = None + # The speculative worker holding the drafter's per-request context state (DFlashWorker). + _draft_ctx_owner: Optional[Any] = None + _send_before_draft_ctx_event: bool = False + # draft_kv_sent / draft_kv_fallback{reason}: what the context side shipped, and what the + # generation side seeded. + _draft_kv_ctx_counters: Optional[DraftKvCounters] = None + _draft_kv_gen_counters: Optional[DraftKvCounters] = None + # Test only (TRTLLM_M31_F0A_DUMP_DIR): draft-window dumps; None is fully inert. + _draft_dump_dir: Optional[str] = None + _draft_dump_rank: int = 0 + _draft_dump_heads: int = 0 + @property def consumes_transfer_buffer(self) -> bool: return False @@ -241,6 +277,8 @@ def __init__( dist: Distributed, kv_cache_manager: KVCacheManager, cache_transceiver_config: CacheTransceiverConfig, + draft_kv_cache_manager: Optional[KVCacheManager] = None, + draft_ctx_owner: Optional[Any] = None, ): self._shutdown_lock = threading.Lock() self._shutdown_complete = False @@ -261,6 +299,17 @@ def __init__( else None ) self._check_compatible() + if draft_kv_cache_manager is not None: + if draft_ctx_owner is None or not isinstance(draft_kv_cache_manager, KVCacheManagerV2): + raise ValueError( + "the disagg drafter context-window transfer needs a KVCacheManagerV2 draft " + "pool and the speculative worker that owns the drafter context" + ) + if draft_kv_cache_manager.enable_joint_kv_cache_reuse: + raise ValueError( + "the disagg drafter context-window transfer requires an unpaired draft pool " + "(enable_joint_kv_cache_reuse=False)" + ) enforce_physical_ownership = _validate_fp4_mla_bridge_profile( mapping, kv_cache_manager, cache_transceiver_config ) @@ -317,8 +366,18 @@ def __init__( else 0 ), agent_bounce_params=cache_transceiver_config.agent_bounce_params, + draft_kv_cache_manager=draft_kv_cache_manager, + draft_receive_disabled=( + self._draft_receive_disabled_reason(mapping) + if draft_kv_cache_manager is not None + else None + ), ) ) + if draft_kv_cache_manager is not None: + self._engage_draft_kv(draft_kv_cache_manager, draft_ctx_owner) + else: + self._count_draft_kv_peer_not_engaged() if enforce_physical_ownership: logger.info( "FP4 MLA KV ownership bridge ENABLED: " @@ -366,6 +425,76 @@ def __init__( self._ever_had_send_session: bool = False self._ever_had_recv_session: bool = False + @staticmethod + def _draft_receive_disabled_reason(mapping: Mapping) -> Optional[str]: + """``draft_kv.draft_receive_disabled_reason`` for this process's receive path, logged.""" + from tensorrt_llm._torch.disaggregation.orchestration.coordinator import ( + uses_async_gen_transfer, + ) + + reason = draft_receive_disabled_reason(mapping, not uses_async_gen_transfer()) + if reason is not None: + logger.warning( + f"F0a: this rank receives no drafter context window ({reason}): the synchronous " + "generation receive (TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP=1) has no cross-rank " + f"agreement on it under TP {mapping.tp_size} without attention DP; requests " + "draft from an empty window and the target transfer is unchanged" + ) + return reason + + def _count_draft_kv_peer_not_engaged(self) -> None: + """Count the requests whose drafter window a peer engages but this rank does not. + + Such a peer pair moves the target set only (``peer_not_engaged``); this rank has no drafter + context of its own, so its sender and receiver count the affected requests. + """ + self._draft_kv_ctx_counters = DraftKvCounters(f"context, rank {self._mapping.rank}") + self._draft_kv_gen_counters = DraftKvCounters(f"generation, rank {self._mapping.rank}") + sender = getattr(self._transfer_worker, "_sender", None) + if sender is not None: + sender._draft_kv_counters = self._draft_kv_ctx_counters + receiver = getattr(self._transfer_worker, "_receiver", None) + if receiver is not None: + receiver._draft_kv_counters = self._draft_kv_gen_counters + + def _engage_draft_kv(self, draft_kv_cache_manager: KVCacheManagerV2, owner: Any) -> None: + """Ship the standalone drafter's context window as a second block set.""" + self._draft_kv_cache_manager = draft_kv_cache_manager + self._draft_reuse_adapter = create_cache_reuse_adapter(draft_kv_cache_manager) + self._draft_page_table = self._transfer_worker.draft_page_table + self._draft_ctx_owner = owner + # A generation worker sizes the windowed draft pool for the received window, with a + # readiness event merged into the receive fences; a context worker records the + # drafter-store event its send waits on. + draft_kv_cache_manager.engage_disagg_draft_kv_receive() + owner.engage_disagg_draft_kv() + self._send_before_draft_ctx_event = send_before_draft_ctx_event() + # Per rank: under TP-attention every CTX (and GEN) TP rank counts each request once. + self._draft_kv_ctx_counters = DraftKvCounters(f"context, rank {self._mapping.rank}") + self._draft_kv_gen_counters = DraftKvCounters(f"generation, rank {self._mapping.rank}") + # The sender counts a context TP rank that decided otherwise than the response's rank. + self._transfer_worker._sender._draft_kv_counters = self._draft_kv_ctx_counters + self._draft_dump_dir = draft_dump_dir() + if self._draft_dump_dir is not None: + self._draft_dump_rank = self._mapping.rank + self._draft_dump_heads = ( + self._transfer_worker._rank_info.draft_attention.kv_heads_per_rank + ) + logger.warning( + f"TRTLLM_M31_F0A_DUMP_DIR={self._draft_dump_dir}: dumping every drafter context " + "window shipped or received (G1d; test only, synchronizes per request)" + ) + if self._send_before_draft_ctx_event: + logger.warning( + f"{TEST_SEND_BEFORE_EVENT_ENV}=1: context KV is published WITHOUT waiting for the " + "drafter context store (ordering negative control; test only)" + ) + logger.info( + "KvCacheTransceiverV2: shipping the standalone drafter context window " + f"({len(self._draft_page_table.layer_groups)} draft layer group(s), " + f"tokens_per_block={self._draft_page_table.tokens_per_block})" + ) + def _fail_unproven_transfer(self, event: QuiescenceFatalEvent) -> None: """Fail-stop while the watchdog retains this transceiver's resource roots. @@ -575,8 +704,37 @@ def _describe_local(self, req: LlmRequest) -> Chunk: token_range=TokenRange(start=0, end=transfer_len), is_last=True, excluded_pool_views=excluded_pool_views, + draft_block_ids_per_layer_groups=self._describe_draft_local(req), ) + def _describe_draft_local(self, req: LlmRequest) -> Optional[List[np.ndarray]]: + """The draft pool's blocks for the prompt, one positional list per draft layer group, or + None when the drafter context window does not ship. + + ``ceil(prompt_len / draft_tpb)`` entries with -1 where the windowed pool released or never + held a page. No cached-prefix mask: the draft pool is never paired with block reuse, so the + receiver's window is always written whole. + """ + page_table = self._draft_page_table + if page_table is None: + return None + if ( + getattr(req, "is_generation_only_request", False) is True + and not self._need_aux_transfer(req) + and self._received_draft_ctx_state(req) is None + ): + # Context-first: the context side said in its response that this request's window + # does not travel, so neither the request nor the expected bytes name its pages. + return None + adapter = self._draft_reuse_adapter + assert adapter is not None + tpb = adapter.tokens_per_block + prompt_blocks = (req.prompt_len + tpb - 1) // tpb + return [ + self._positional_window(adapter.get_block_ordinals(req, idx, lg), prompt_blocks, 0) + for idx, lg in enumerate(page_table.layer_groups) + ] + @staticmethod def _positional_window( ordinals: np.ndarray, prompt_blocks: int, cached_blocks: int @@ -629,8 +787,19 @@ def _chunk_num_bytes(self, chunk: Chunk) -> int: else: # Attention: n blocks, each covering the view's regions. total += n * view_bytes + # Read off the class: callers may pass a stand-in carrying only the page tables. + total += KvCacheTransceiverV2._draft_chunk_num_bytes( + getattr(self, "_draft_page_table", None), chunk + ) return total + @staticmethod + def _draft_chunk_num_bytes(page_table, chunk: Chunk) -> int: + """Local-rank bytes of the drafter context window a chunk covers (0 when none travels).""" + return draft_window_bytes( + page_table, getattr(chunk, "draft_block_ids_per_layer_groups", None) + ) + @staticmethod def _need_aux_transfer(req: LlmRequest) -> bool: params = req.py_disaggregated_params @@ -711,6 +880,7 @@ def _consensus_outcome( need_sync: bool, locally_quiesced=None, locally_verified=None, + locally_flagged=None, ): # CANCELLED/FAILED on any rank → global; COMPLETED only when ALL ranks agree. # Quiescence and write-verification, when requested, also require @@ -721,6 +891,8 @@ def _consensus_outcome( local_outcome.append(list(locally_quiesced)) if locally_verified is not None: local_outcome.append(list(locally_verified)) + if locally_flagged is not None: + local_outcome.append(list(locally_flagged)) if not need_sync: packed = [local_outcome] else: @@ -751,11 +923,18 @@ def _consensus_outcome( # the same tokens, so a request counts as write-verified only # when EVERY rank verified its local shard. all_verified = [p[idx] for p in packed] + idx += 1 global_verified = self._intersection(all_verified, n) result.append({rid for rid in to_process if rid in global_verified}) + if locally_flagged is not None: + # Flagged on any rank -> flagged everywhere. + global_flagged = self._union([p[idx] for p in packed]) + result.append({rid for rid in to_process if rid in global_flagged}) return tuple(result) - def _gen_consensus_outcome(self, to_process, cancelled, failed, completed, locally_verified): + def _gen_consensus_outcome( + self, to_process, cancelled, failed, completed, locally_verified, locally_draft_vetoed=None + ): # A failure/cancellation may be global, but reuse is safe only after # every participating rank has drained its local physical accessor. locally_retirable = [] @@ -763,25 +942,27 @@ def _gen_consensus_outcome(self, to_process, cancelled, failed, completed, local session = self._recv_sessions[rid] if not self._ownership_blocks_retirement(session): locally_retirable.append(rid) - new_cancelled, new_failed, new_completed, globally_retirable, verified = ( - self._consensus_outcome( - to_process, - cancelled, - failed, - completed, - self._gen_allgather, - self._gen_need_sync, - locally_retirable, - locally_verified, - ) + outcome = self._consensus_outcome( + to_process, + cancelled, + failed, + completed, + self._gen_allgather, + self._gen_need_sync, + locally_retirable, + locally_verified, + locally_draft_vetoed, ) + new_cancelled, new_failed, new_completed, globally_retirable, verified = outcome[:5] retirable = set(globally_retirable) - return ( + result = ( [rid for rid in new_cancelled if rid in retirable], [rid for rid in new_failed if rid in retirable], new_completed, verified, ) + # Only a transceiver shipping the drafter window passes vetoes (and gets them back). + return result if locally_draft_vetoed is None else result + (outcome[5],) def _ctx_consensus_outcome(self, to_process, cancelled, failed, completed, locally_quiesced): # TP first, then PP. A local timeout remains nonterminal, so it is @@ -977,6 +1158,9 @@ def _finalize_send(self, req: LlmRequest, session: TxSessionBase): if self._need_aux_transfer(req): session.pack_aux(req) session.send_aux() + # Context-first: the drafter window decision (req.py_draft_ctx_aux) reaches the generation + # request through the response's DisaggregatedParams.draft_ctx_state (LlmRequest + # create_response), never opaque_state, which C++ parses as a DataTransceiverState. req.context_phase_params = ContextPhaseParams( first_gen_tokens=[], req_id=rid, @@ -1053,9 +1237,233 @@ def _build_prefill_extent( token_range=TokenRange(start=chunk_start * tpb, end=chunk_end_token), is_last=is_last_chunk, excluded_pool_views=whole.excluded_pool_views, + # The windowed draft pool releases what an earlier piece would name: the window + # ships whole with the last piece only. + draft_block_ids_per_layer_groups=( + getattr(whole, "draft_block_ids_per_layer_groups", None) + if is_last_chunk + else None + ), ), ) + def _await_draft_ctx(self, req: LlmRequest, extent: CacheExtent) -> CacheExtent: + """Order this piece's publication after the drafter's context store, and on the last piece + decide whether the request's drafter context window travels. + + The drafter writes the context window on the execution stream inside the forward; RDMA + cannot observe stream order, so the event recorded right after that store is waited on + before any byte is published. The sampler's own host wait already covers it today, so the + wait returns at once; it keeps the guarantee explicit. + + A window that cannot be served exactly falls back to an empty drafter window on the + generation side; the request itself never fails. The decision rides the aux slot + (generation-first) or the response's ``draft_ctx_state`` (context-first). In context-first + the receiver learns it before it asks, so the window's pages are left out of this piece as + well; in generation-first the receiver has already asked for them, so they still travel + (keeping the byte accounting exact) and are marked unusable. + """ + owner = self._draft_ctx_owner + if owner is None: + return extent + ready = owner.take_disagg_draft_ctx_ready(req.py_request_id) + if ready is not None and not self._send_before_draft_ctx_event: + ready[0].synchronize() + chunk = extent.local + if not chunk.is_last: + return extent + reason = self._draft_ctx_fallback_reason(req, ready, chunk) + if reason is None: + _, ctx_len, ctx_base = ready + req.py_draft_ctx_aux = (int(ctx_len), int(ctx_base), True) + self._draft_kv_ctx_counters.record_sent() + if self._draft_dump_dir is not None: + self._dump_draft_window("ctx", req, get_unique_rid(req), chunk, None) + return extent + req.py_draft_ctx_aux = (0, 0, False) + self._draft_kv_ctx_counters.record_fallback( + reason, + f"rid={get_unique_rid(req)} prompt_len={req.prompt_len} " + f"state={None if ready is None else ready[1:]}", + ) + if self._draft_dump_dir is not None: + self._dump_draft_window("ctx", req, get_unique_rid(req), chunk, reason) + if self._need_aux_transfer(req): + return extent + return CacheExtent( + name=extent.name, local=replace(chunk, draft_block_ids_per_layer_groups=None) + ) + + def _draft_ctx_fallback_reason(self, req: LlmRequest, ready, chunk: Chunk) -> Optional[str]: + """Why the request's window cannot be served exactly, or None when it can. + + Exactly means the generation side's window holds what the context side stored, read the + same way. The drafter stores the whole prompt (``ctx_len == prompt_len``); on a prefix-reuse + hit an unpaired draft pool stores only the tail from the reused position on, numbered by + target position, and ``ctx_base`` pages below it are never read (the block decode shifts + the request's block-table row past them and shortens the context by as much, + ``dflash_rebased_ctx_view``). Those pages lie in the request's own draft table like every + other, so shipping the window pages and the same ``(ctx_len, ctx_base)`` reproduces the + context side's reads. The context side must hold every page the window behind the prompt + end covers. + """ + if ready is None: + return FALLBACK_NO_STORE_EVENT + _, ctx_len, ctx_base = ready + if ctx_len != req.prompt_len: + return FALLBACK_CTX_LEN_MISMATCH + tpb = self._draft_page_table.tokens_per_block + if not 0 <= ctx_base <= -(-req.prompt_len // tpb): + return FALLBACK_REUSE_BASE + draft_ids = chunk.draft_block_ids_per_layer_groups + if draft_ids is None: + return FALLBACK_PAGE_MISMATCH + for block_ids, group in zip(draft_ids, self._draft_page_table.layer_groups): + window = getattr(group, "sliding_window_size", None) + first = 0 if window is None else max(0, (req.prompt_len + 1 - window) // tpb) + if not (np.asarray(block_ids)[first:] >= 0).all(): + return FALLBACK_PAGE_MISMATCH + return None + + def _dump_draft_window( + self, side: str, req: LlmRequest, match_id, chunk, reason, state=None + ) -> None: + """Test only (TRTLLM_M31_F0A_DUMP_DIR): write this side's window for the cross-side window compare.""" + try: + ctx_len, ctx_base = state if state is not None else req.py_draft_ctx_aux[:2] + if reason is not None: + write_window_dump( + self._draft_dump_dir, + side, + self._draft_dump_rank, + match_id, + request_id=req.py_request_id, + prompt_len=req.prompt_len, + ctx_len=0, + ctx_base=0, + shipped=False, + reason=reason, + ) + return + block_ids = ( + chunk.draft_block_ids_per_layer_groups + if chunk is not None + else self._describe_draft_local(req) + ) + write_window_dump( + self._draft_dump_dir, + side, + self._draft_dump_rank, + match_id, + request_id=req.py_request_id, + prompt_len=req.prompt_len, + ctx_len=ctx_len, + ctx_base=ctx_base, + shipped=True, + page_table=self._draft_page_table, + heads_per_rank=self._draft_dump_heads, + block_ids_per_group=block_ids, + ) + except Exception as error: # noqa: BLE001 - a test dump must not take serving down + logger.error(f"F0a G1d dump ({side}, id={match_id}) failed: {error}") + + @staticmethod + def _received_draft_ctx_state(req: LlmRequest) -> Optional[tuple]: + """The ``(ctx_len, ctx_base)`` the context side shipped a window for, or None.""" + if KvCacheTransceiverV2._need_aux_transfer(req): + state = getattr(req, "py_draft_ctx_aux", None) + else: + params = req.py_disaggregated_params + state = decode_draft_ctx_state(getattr(params, "draft_ctx_state", None)) + if state is None or not state[2]: + return None + return state[0], state[1] + + def _seed_draft_ctx(self, req: LlmRequest, veto: Optional[str] = None) -> None: + """Hand the request's drafter context lengths to the worker, which seeds them on the + request's first step (DFlashSpecMetadata.prepare): the received ones, or an empty window + when the context side did not ship it or ``veto`` names why the received window cannot be + used (``_draft_window_vetoes``).""" + owner = self._draft_ctx_owner + if owner is None: + return + state = self._received_draft_ctx_state(req) + if veto is not None: + # The veto names the cause even when no state arrived (e.g. a peer that does not + # engage the window ships none: peer_not_engaged, not no_ctx_state). + state = None + reason = veto + else: + reason = FALLBACK_NO_STATE + shipped = state is not None + if state is None: + state = empty_window_state(req.prompt_len, self._draft_page_table.tokens_per_block) + self._draft_kv_gen_counters.record_fallback( + reason, f"rid={get_unique_rid(req)} prompt_len={req.prompt_len}" + ) + else: + self._draft_kv_gen_counters.record_sent() + ctx_len, ctx_base = state + hook = None + if self._draft_dump_dir is not None and shipped: + params = req.py_disaggregated_params + match_id = params.ctx_request_id + if match_id is None: + match_id = params.disagg_request_id + + def hook(): + # At the request's first step: the received rows, read through its block table. + self._dump_draft_window("gen", req, match_id, None, None, state=(ctx_len, ctx_base)) + + if hook is None: + owner.seed_disagg_draft_ctx(req.py_request_id, ctx_len, ctx_base, req.prompt_len) + else: + owner.seed_disagg_draft_ctx( + req.py_request_id, ctx_len, ctx_base, req.prompt_len, on_seed=hook + ) + + def _note_expected_draft_bytes(self, req: LlmRequest, chunk: Chunk) -> None: + """Record the draft bytes this rank asks for, which ``_local_draft_veto`` checks.""" + if self._draft_ctx_owner is not None: + req.py_draft_expected_bytes = KvCacheTransceiverV2._draft_chunk_num_bytes( + self._draft_page_table, chunk + ) + + def _local_draft_veto(self, req: LlmRequest, session) -> Optional[str]: + """Why this rank cannot use the request's received drafter window, or None. + + Generation-first: the aux slot this rank's context peer filled says not shipped. Either + style: the draft pages this rank asked for were not all attested written (a context TP + rank that left its slice out while the response's rank shipped). Flagged on any rank, + the window is dropped on every rank (the flags ride the receive consensus). + """ + disabled = getattr(session, "draft_disabled_reason", None) + reason = disabled() if callable(disabled) else None + if reason is not None: + # Registration disabled the window toward this context peer; nothing was asked for. + return reason + expected = getattr(req, "py_draft_expected_bytes", 0) + if expected: + verified, total = session.kv_write_byte_counts() + if total is None or verified != total: + if total is not None and total - verified == expected: + return FALLBACK_CTX_RANK_DIVERGENCE + return FALLBACK_UNVERIFIED_WRITE + if self._need_aux_transfer(req): + state = session.peek_draft_ctx() + if state is None or not state[2]: + return FALLBACK_NO_STATE + return None + + def _draft_window_vetoes(self, completed: List[int]) -> Dict[int, str]: + """This rank's ``_local_draft_veto`` of each completed receive that has one.""" + vetoes = {} + for rid in completed: + reason = self._local_draft_veto(self._recv_reqs[rid], self._recv_sessions[rid]) + if reason is not None: + vetoes[rid] = reason + return vetoes + @nvtx_range("KvCacheTransceiverV2.respond_and_send_async") def respond_and_send_async(self, req: LlmRequest) -> None: """Send the request's next KV slice to the generation server.""" @@ -1090,6 +1498,7 @@ def respond_and_send_async(self, req: LlmRequest) -> None: return else: extent = self._create_cache_extent(req) + extent = self._await_draft_ctx(req, extent) chunk = extent.local # The handle that comes back is the contract's answer about this piece. What retires # the request is the sweep over the session tables, as it was before. @@ -1125,6 +1534,7 @@ def request_and_receive_sync(self, req: LlmRequest) -> None: fetches = self._open_peer_source(req) # Same submission the asynchronous entry makes; what differs is who waits. The session # underneath is read back for the blocking wait, the auxiliary buffer and the close. + self._note_expected_draft_bytes(req, extent.local) fetches.fetch(extent, expected_write_bytes=self._chunk_num_bytes(extent.local)) session = self._legacy_session(fetches) self._recv_sessions[rid] = session @@ -1141,6 +1551,8 @@ def request_and_receive_sync(self, req: LlmRequest) -> None: ) if self._need_aux_transfer(req): self._apply_aux(session, req) + if self._draft_ctx_owner is not None: + self._seed_draft_ctx(req, self._local_draft_veto(req, session)) self._assert_disagg_history_declared(req) req.state = LlmRequestState.DISAGG_GENERATION_TRANS_COMPLETE else: @@ -1202,6 +1614,7 @@ def request_and_receive_async(self, req: LlmRequest) -> None: extent = self._create_cache_extent(req) chunk_bytes = self._chunk_num_bytes(extent.local) req.py_kv_cache_xfer_bytes = chunk_bytes * self._kv_size_rank_factor + self._note_expected_draft_bytes(req, extent.local) fetches = self._open_peer_source(req) # Claimed to be transferring only once there is something to transfer: a builder that # raises above leaves the request where it was, not in a state nothing advances. @@ -1370,9 +1783,23 @@ def check_gen_transfer_status(self, at_least_request_num: Optional[int]) -> GenT ] # All ranks must agree on per-rid outcome to avoid req.state divergence. - cancelled, failed, completed, verified = self._gen_consensus_outcome( - to_process, cancelled, failed, completed, locally_verified - ) + if self._draft_ctx_owner is None: + cancelled, failed, completed, verified = self._gen_consensus_outcome( + to_process, cancelled, failed, completed, locally_verified + ) + draft_vetoes, globally_vetoed = {}, set() + else: + # The drafter-window vetoes ride the same allgather: a window unusable on any rank + # is dropped on every rank, so the generation ranks never draft from different ones. + draft_vetoes = self._draft_window_vetoes(completed) + cancelled, failed, completed, verified, globally_vetoed = self._gen_consensus_outcome( + to_process, + cancelled, + failed, + completed, + locally_verified, + locally_draft_vetoed=list(draft_vetoes), + ) cancelled_reqs = [] for rid in cancelled: @@ -1407,6 +1834,11 @@ def check_gen_transfer_status(self, at_least_request_num: Optional[int]) -> GenT req.set_kv_cache_size(getattr(req, "py_kv_cache_xfer_bytes", 0)) if self._need_aux_transfer(req): self._apply_aux(session, req) + if self._draft_ctx_owner is not None: + veto = draft_vetoes.get(rid) + if veto is None and rid in globally_vetoed: + veto = FALLBACK_CTX_RANK_DIVERGENCE + self._seed_draft_ctx(req, veto) self._assert_disagg_history_declared(req) self._close_session_or_raise(session, rid, "completed") req.state = LlmRequestState.DISAGG_GENERATION_TRANS_COMPLETE diff --git a/tensorrt_llm/_torch/flashinfer_autotune_cache.py b/tensorrt_llm/_torch/flashinfer_autotune_cache.py new file mode 100644 index 000000000000..498546a1736d --- /dev/null +++ b/tensorrt_llm/_torch/flashinfer_autotune_cache.py @@ -0,0 +1,78 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""A per-rank file cache for FlashInfer's own autotuner (its CuTe DSL MXFP8 GEMM configs). + +FlashInfer keeps the configs it profiles inside ``flashinfer.autotune()`` in a process-wide table, so +every server start profiles them again and may pick a different config per shape. With +``TLLM_FLASHINFER_AUTOTUNE_CACHE_DIR`` set, every ``autotune(cache=None)`` this package opens reads +(and, while tuning, extends) ``

    /rank.json`` instead. With +``TLLM_FLASHINFER_AUTOTUNE_FROZEN=1`` the file is only loaded, never profiled into: the configs of a +previous run are served as they are. Without the directory, a model may install a bundled file +(``use_bundled_file``), which is always only loaded. Neither set, ``autotune`` is FlashInfer's own. +""" + +import os +from typing import Optional + +from tensorrt_llm._utils import mpi_rank +from tensorrt_llm.logger import logger + +CACHE_DIR_ENV = "TLLM_FLASHINFER_AUTOTUNE_CACHE_DIR" +FROZEN_ENV = "TLLM_FLASHINFER_AUTOTUNE_FROZEN" + +# FlashInfer keeps a loaded file's configs after the context exits, so a frozen process reads the +# file once; later calls pass no cache and see the same table. +_STATE = {"loaded": False, "logged": False, "bundled": None} + + +def use_bundled_file(path: Optional[str]) -> None: + """Serve this rank's FlashInfer configs from ``path`` (load only) when no cache directory is configured.""" + _STATE["bundled"] = path + + +def cache_file() -> Optional[str]: + """This rank's cache file: the configured directory's, else the bundled file, else None.""" + cache_dir = os.environ.get(CACHE_DIR_ENV) + if not cache_dir: + return _STATE["bundled"] + return os.path.join(cache_dir, f"rank{mpi_rank()}.json") + + +def frozen() -> bool: + """Whether the cache file is only loaded (never profiled into).""" + if not os.environ.get(CACHE_DIR_ENV): + return True + return os.environ.get(FROZEN_ENV, "0") == "1" + + +def autotune(tune_mode: bool = True, cache: Optional[str] = None, *args, **kwargs): + """``flashinfer.autotune`` with this rank's cache file filled in when none is given.""" + from flashinfer import autotune as _autotune + + path = cache_file() if cache is None else None + if path is not None: + if frozen(): + tune_mode = False + if _STATE["loaded"]: + return _autotune(tune_mode, None, *args, **kwargs) + _STATE["loaded"] = True + cache = path + if not _STATE["logged"]: + _STATE["logged"] = True + logger.info( + f"FlashInfer autotune cache: rank={mpi_rank()} file={path} " + f"exists={os.path.isfile(path)} frozen={frozen()}" + ) + return _autotune(tune_mode, cache, *args, **kwargs) diff --git a/tensorrt_llm/_torch/late_compile.py b/tensorrt_llm/_torch/late_compile.py new file mode 100644 index 000000000000..97eb4bf210ae --- /dev/null +++ b/tensorrt_llm/_torch/late_compile.py @@ -0,0 +1,90 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Late-compile detector: report kernel compiles that happen after the executor finished its warmup. + +A compile inside a served request costs that request its whole compile time, and in-memory compiles (CuTe DSL +handles, Triton specializations) leave no file in the JIT cache directories, so cache-file checks cannot see them. +After ``mark_server_ready()`` every ``cutlass.cute.compile`` call and every Triton JIT compile logs one warning line +``[late-compile] op= key=``. The executor arms it only for a model whose RuntimeHooks set +late_compile_detect. Kill switch: TRTLLM_LATE_COMPILE_DETECT=0. + +Not covered here: the TRTLLM-Gen FMHA runtime NVRTC compile (C++, an in-memory module cache warmed for generation +kernels only), whose warmup and persistence fix is tracked with that kernel library, and the fmha_sm100 package's +on-disk AOT cache, whose post-warmup "[aot_cache] Saved" lines are its own signal. +""" + +import os +from typing import Any + +from tensorrt_llm.logger import logger + +_ENABLED = os.environ.get("TRTLLM_LATE_COMPILE_DETECT", "1") != "0" +_ready = [False] + + +def note(op: str, key: Any) -> None: + """Log a compile of ``op`` for ``key`` when it happens after warmup.""" + if _ready[0]: + logger.warning(f"[late-compile] op={op} key={key}") + + +class _CuteCompile: + """Stands in for ``cutlass.cute.compile``: notes each call, then forwards it (``cute.compile[...]`` included).""" + + def __init__(self, inner: Any) -> None: + self._inner = inner + + def __call__(self, *args: Any, **kwargs: Any) -> Any: + target = args[0] if args else None + note("cute.compile", getattr(target, "__qualname__", type(target).__name__)) + return self._inner(*args, **kwargs) + + def __getitem__(self, item: Any) -> "_CuteCompile": + return _CuteCompile(self._inner[item]) + + def __getattr__(self, name: str) -> Any: + return getattr(self._inner, name) + + +def _hook_cute() -> None: + try: + import cutlass.cute as cute + except ImportError: + return + if not isinstance(cute.compile, _CuteCompile): + cute.compile = _CuteCompile(cute.compile) + + +def _hook_triton() -> None: + try: + import triton + except ImportError: + return + runtime = getattr(getattr(triton, "knobs", None), "runtime", None) + if runtime is None or not hasattr(runtime, "jit_post_compile_hook"): + logger.info( + "[late-compile] Triton post-compile hook unavailable; Triton compiles are not reported" + ) + return + previous = runtime.jit_post_compile_hook + + def hook(*args: Any, **kwargs: Any) -> Any: + try: + fn = kwargs.get("fn") + name = getattr(getattr(fn, "fn", fn), "__qualname__", "?") + note("triton", f"{name} {kwargs.get('key', '')}") + except Exception: # noqa: BLE001 (the detector must never break a compile) + pass + return previous(*args, **kwargs) if previous is not None else None + + runtime.jit_post_compile_hook = hook + + +def mark_server_ready() -> None: + """Arm the detector once the executor's warmup is done (idempotent).""" + if not _ENABLED or _ready[0]: + return + _hook_cute() + _hook_triton() + _ready[0] = True + logger.info("[late-compile] detector armed after warmup") diff --git a/tensorrt_llm/_torch/models/modeling_dflash.py b/tensorrt_llm/_torch/models/modeling_dflash.py index d1d80a29efe2..32c5c246baf8 100644 --- a/tensorrt_llm/_torch/models/modeling_dflash.py +++ b/tensorrt_llm/_torch/models/modeling_dflash.py @@ -11,8 +11,9 @@ from tensorrt_llm.logger import logger -from ...functional import RotaryScalingType +from ...functional import AllReduceFusionOp, RotaryScalingType from ..attention.rotary_embedding import RotaryEmbedding +from ..distributed import AllReduceParams try: from ..custom_ops import flashinfer_apply_rope_with_cos_sin_cache_inplace as _flashinfer_rope @@ -324,6 +325,30 @@ def forward( ) +# A row-parallel Linear / GatedMLP call that leaves its partial sum unreduced. +_NO_ALLREDUCE = AllReduceParams(enable_allreduce=False) + + +def _dflash_ar_residual_norm(linear, partial, residual, norm, op=None): + """All-reduce a row-parallel ``linear``'s partial sum, add ``residual`` and RMSNorm with ``norm``'s weight, in one + fused launch: a target's ``op`` when it takes the operands (see DFlashWorker.enable_fused_ar_norm_op), else the + Linear's own all-reduce. Returns (normed, residual + reduced), as ``norm(x, residual)`` does on the reduced x.""" + if op is not None: + out = op(partial, residual, norm.weight, norm.variance_epsilon) + if out is not None: + return out + out = linear.all_reduce( + partial, + all_reduce_params=AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual, + norm_weight=norm.weight, + eps=norm.variance_epsilon, + ), + ) + return out[0], out[1] + + class DFlashForCausalLM(nn.Module): """Draft model wrapper for DFlash speculative decoding. @@ -516,6 +541,17 @@ def __init__(self, draft_config, *, dflash_attention_backend: str = "AUTO"): self._dflash_fused_qk_norm_rope = None self._dflash_fused_qk_norm_rope_append = None self._dflash_attn_pdl = False + # Fused AR + norm (see DFlashWorker.enable_fused_ar_norm): each row-parallel o_proj / down_proj skips its own + # all-reduce and the next add + RMSNorm runs fused into it; the optional op is a target's one-launch variant. + self._dflash_fused_ar_norm = False + self._dflash_fused_ar_norm_op = None + # Windowed decode cap (see DFlashWorker.enable_window_capped_decode): batches of at most this many requests + # pass the TRTLLM-Gen decode a window-bounded max_seq_len (0 = off). + self._dflash_window_cap_max_batch = 0 + # fc split (see DFlashWorker.enable_fc_tp_split): (K start, K end, weight shard, all-reduce). + self._dflash_fc_split = None + # fc column shard: an object with ``applies(fc, rows)`` and ``__call__(x)`` returning ``fc(x)`` bit for bit. + self._dflash_fc_shard = None self._dflash_trtllm_gen_counters = None self.register_buffer("_dflash_batch_indices", None, persistent=False) self.register_buffer("_dflash_block_offsets", None, persistent=False) @@ -731,6 +767,14 @@ def project_target_hidden(self, hidden_states: torch.Tensor) -> torch.Tensor: normalize the per-aux features first by overriding this method. """ hidden_states = hidden_states.to(self.fc.weight.dtype) + split = self._dflash_fc_split + if split is not None: + k0, k1, shard, allreduce = split + partial = torch.mm(hidden_states[:, k0:k1], shard.t(), out_dtype=torch.float32) + return self.hidden_norm(allreduce(partial).to(self.fc.weight.dtype)) + shard = self._dflash_fc_shard + if shard is not None and shard.applies(self.fc, hidden_states.shape[0]): + return self.hidden_norm(shard(hidden_states)) return self.hidden_norm(self.fc(hidden_states)) def _validate_dflash2_config(self) -> None: @@ -1066,6 +1110,18 @@ def precompute_context_kv( self._fused_rope_inplace(k.view(N * L, nkv * hd), positions, N, L) return k, v + def _decode_max_seq_len( + self, capacity: int, window_left: int, block_size: int, page_size: int, batch: int + ) -> int: + """max_seq_len for the block decode: the slot capacity, or, with the windowed decode cap on for this batch size + and a finite left window, the window plus 64 rows rounded up to a page (the TRTLLM-Gen autotuner sizes the tile + and the KV split from it; the attended keys are the window either way).""" + limit = self._dflash_window_cap_max_batch + if limit <= 0 or batch > limit or window_left < 0 or block_size > 64: + return capacity + capped = -(-(window_left + 1 + 64) // page_size) * page_size + return min(capacity, capped) + def _get_cos_sin_cache(self) -> torch.Tensor: """Return the flashinfer-style cos/sin cache for the drafter. @@ -1633,6 +1689,10 @@ def dflash_forward( residual = None has_block_conv = self.has_block_conv + fuse_ar_norm = self._dflash_fused_ar_norm and not has_block_conv + # Fused AR + norm: the previous layer's down_proj, whose partial sum is still to be reduced into this layer's + # input_layernorm (or the final norm). + pending_ar = None for layer_idx, layer in enumerate(self.model.layers): attn_mod = layer.self_attn @@ -1644,7 +1704,17 @@ def dflash_forward( hs_normed_flat = layer.input_layernorm(hs_flat) else: res_flat = residual.reshape(-1, residual.shape[-1]) - hs_normed_flat, res_flat = layer.input_layernorm(hs_flat, res_flat) + if pending_ar is not None: + hs_normed_flat, res_flat = _dflash_ar_residual_norm( + pending_ar, + hs_flat, + res_flat, + layer.input_layernorm, + self._dflash_fused_ar_norm_op, + ) + pending_ar = None + else: + hs_normed_flat, res_flat = layer.input_layernorm(hs_flat, res_flat) residual = res_flat.reshape(B, block_size, -1) # DFlash 2 wraps each sublayer in a block-local convolution; both @@ -1810,7 +1880,9 @@ def dflash_forward( workspace_buffer=self._dflash_trtllm_gen_workspace, block_tables=block_tables, seq_lens=seq_lens_after, - max_seq_len=pages_per_slot * page_size, + max_seq_len=self._decode_max_seq_len( + pages_per_slot * page_size, window_left, block_size, page_size, B + ), bmm1_scale=head_dim**-0.5, bmm2_scale=1.0, window_left=window_left, @@ -1959,20 +2031,36 @@ def dflash_forward( ) # o_proj (flat 2D, handles all-reduce internally) - hidden_out = attn_mod.o_proj(attn_output) - if has_block_conv: - hidden_out = self.attention_convs[layer_idx].finish( - hidden_out, attn_conv_coefficients, block_size + res_flat = residual.reshape(-1, residual.shape[-1]) + if fuse_ar_norm: + # The all-reduce runs fused with the post-attention add + RMSNorm below. + hidden_out = attn_mod.o_proj(attn_output, all_reduce_params=_NO_ALLREDUCE) + hidden_out, res_flat = _dflash_ar_residual_norm( + attn_mod.o_proj, + hidden_out, + res_flat, + layer.post_attention_layernorm, + self._dflash_fused_ar_norm_op, ) + else: + hidden_out = attn_mod.o_proj(attn_output) + if has_block_conv: + hidden_out = self.attention_convs[layer_idx].finish( + hidden_out, attn_conv_coefficients, block_size + ) - # Post-attention layernorm + MLP (flat 2D) - res_flat = residual.reshape(-1, residual.shape[-1]) - hidden_out, res_flat = layer.post_attention_layernorm(hidden_out, res_flat) + # Post-attention layernorm + MLP (flat 2D) + hidden_out, res_flat = layer.post_attention_layernorm(hidden_out, res_flat) if has_block_conv: hidden_out, mlp_conv_coefficients = self.mlp_convs[layer_idx].prepare( hidden_out, block_size ) - hidden_out = layer.mlp(hidden_out) + if fuse_ar_norm: + # down_proj's partial sum is reduced into the next norm (next layer or the final norm). + hidden_out = layer.mlp(hidden_out, final_all_reduce_params=_NO_ALLREDUCE) + pending_ar = layer.mlp.down_proj + else: + hidden_out = layer.mlp(hidden_out) if has_block_conv: hidden_out = self.mlp_convs[layer_idx].finish( hidden_out, mlp_conv_coefficients, block_size @@ -1982,6 +2070,15 @@ def dflash_forward( residual = res_flat.reshape(B, block_size, -1) # Final norm + if pending_ar is not None: + hidden_states_out, _ = _dflash_ar_residual_norm( + pending_ar, + hidden_states.reshape(-1, hidden_states.shape[-1]), + residual.reshape(-1, residual.shape[-1]), + self.model.norm, + self._dflash_fused_ar_norm_op, + ) + return hidden_states_out hidden_states_out, _ = self.model.norm( hidden_states.reshape(-1, hidden_states.shape[-1]), residual.reshape(-1, residual.shape[-1]), diff --git a/tensorrt_llm/_torch/modules/linear.py b/tensorrt_llm/_torch/modules/linear.py index ad794d6b494d..7f123e148f88 100644 --- a/tensorrt_llm/_torch/modules/linear.py +++ b/tensorrt_llm/_torch/modules/linear.py @@ -3379,7 +3379,7 @@ def flashinfer_mxfp8_decode_graph_capture_active() -> bool: @contextmanager def flashinfer_mxfp8_autotune() -> Iterator[None]: """Tune FlashInfer MXFP8 tactics while enabling auto-dispatched calls.""" - from flashinfer import autotune + from ..flashinfer_autotune_cache import autotune token = _FLASHINFER_MXFP8_AUTOTUNE_ACTIVE.set(True) try: diff --git a/tensorrt_llm/_torch/pyexecutor/_util.py b/tensorrt_llm/_torch/pyexecutor/_util.py index cf91917ec042..2ac1c26d5f46 100644 --- a/tensorrt_llm/_torch/pyexecutor/_util.py +++ b/tensorrt_llm/_torch/pyexecutor/_util.py @@ -44,6 +44,7 @@ from tensorrt_llm.quantization import QuantAlgo from ..attention.backends import get_sparse_attn_kv_cache_manager +from ..disaggregation.draft_kv import select_disagg_draft_kv_cache_manager from ..disaggregation.kv_cache_transceiver import ( AttentionTypeCpp, create_kv_cache_transceiver, maybe_enable_fabric_memory_for_python_transceiver) @@ -827,6 +828,18 @@ def _derive_draft_max_attention_window( return None +def _arm_kv_cache_opt_ins(hooks) -> None: + """Arm the process-wide KV-cache options ``hooks`` opt into, before any + cache sizing or allocation. All are off for a model that declares none.""" + if hooks.draft_pool_fixes: + from .kv_cache import kv_cache_manager_v2 + kv_cache_manager_v2.arm_draft_pool_fixes() + if hooks.msa_kv_cache_fixes: + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import \ + cache_manager as msa_cache_manager + msa_cache_manager.arm_msa_kv_cache_fixes() + + class KvCacheCreator: """Groups together logic related to KV cache construction.""" @@ -865,6 +878,7 @@ def __init__( ): self._model_engine = model_engine self._draft_model_engine = draft_model_engine + _arm_kv_cache_opt_ins(runtime_hooks_of(model_engine.model)) self._mapping = mapping self._kv_cache_config = kv_cache_config self._max_kv_tokens_in = self._kv_cache_config.max_tokens @@ -4136,6 +4150,31 @@ def validate_seq_slot_pool_covers_admission(max_num_sequences: int, "shortfall means one of them was re-derived from max_batch_size.") +def engage_disagg_gen_init_window(model, draft_kv_cache_manager) -> bool: + """Size a disagg generation worker's gen-init draft mirrors for their window. + + Only for a model that opts in (RuntimeHooks.disagg_draft_kv) with a separate V2 draft pool + and a drafter that can start from an empty window; every other model keeps the stock mirror + path. Without it a windowed standalone draft pool is resized for the whole prompt and fails + past its funding. + """ + if draft_kv_cache_manager is None or not isinstance(draft_kv_cache_manager, + KVCacheManagerV2): + return False + if not runtime_hooks_of(model).disagg_draft_kv: + return False + worker = getattr(model, "spec_worker", None) + if worker is None or not hasattr(worker, "seed_disagg_empty_window"): + return False + worker.engage_disagg_gen_init_window() + draft_kv_cache_manager.engage_disagg_gen_init_window( + worker.seed_disagg_empty_window) + logger.info( + "Disagg: gen-init draft mirrors hold the drafter window behind the prompt end" + ) + return True + + def create_py_executor_instance( *, dist, @@ -4513,9 +4552,33 @@ def create_py_executor_instance( if isinstance(kv_cache_manager, BaseMambaCacheManager): mamba_cache_manager = kv_cache_manager - kv_cache_transceiver = create_kv_cache_transceiver( - mapping, dist, kv_cache_manager, attention_type, - cache_transceiver_config, mamba_cache_manager) + # A standalone drafter's context window ships with the prompt KV only for a + # model that opts in (RuntimeHooks.disagg_draft_kv) and has a separate draft + # pool; every other model passes None and keeps the target-only transfer. + # The generation side's window sizing does not depend on the transfer (nor on + # its kill switch): it is engaged first. + disagg_draft_kv_cache_manager = None + if is_disagg: + engage_disagg_gen_init_window( + model_engine.model, + resources.get(ResourceManagerType.DRAFT_KV_CACHE_MANAGER)) + disagg_draft_kv_cache_manager = select_disagg_draft_kv_cache_manager( + model_engine.model, + resources.get(ResourceManagerType.DRAFT_KV_CACHE_MANAGER)) + if disagg_draft_kv_cache_manager is None: + kv_cache_transceiver = create_kv_cache_transceiver( + mapping, dist, kv_cache_manager, attention_type, + cache_transceiver_config, mamba_cache_manager) + else: + kv_cache_transceiver = create_kv_cache_transceiver( + mapping, + dist, + kv_cache_manager, + attention_type, + cache_transceiver_config, + mamba_cache_manager, + draft_kv_cache_manager=disagg_draft_kv_cache_manager, + draft_ctx_owner=model_engine.model.spec_worker) waiting_queue_policy = (scheduler_config.waiting_queue_policy if scheduler_config is not None else diff --git a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py index b281c15d0f27..48a1c678eb41 100644 --- a/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py +++ b/tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py @@ -60,6 +60,7 @@ DEFAULT_RUNTIME_HOOKS, MsaRuntimePolicy, RuntimeHooks, + hook_work, runtime_hooks_of, ) from tensorrt_llm._torch.pyexecutor.sampler import SampleStateTensors @@ -372,6 +373,7 @@ def __init__( self._config = config self.runtime_hooks = runtime_hooks_of(model) + self._tactic_pins_installed = False self._h2d_cache = ( H2DSkipCache(changed_span=self.runtime_hooks.h2d_changed_span) if self.runtime_hooks.skip_unchanged_h2d @@ -383,6 +385,8 @@ def __init__( parallel_scatter=self.runtime_hooks.pack_parallel_scatter, in_graph=self.runtime_hooks.pack_in_graph, early_copy=self.runtime_hooks.pack_early_copy, + step_epoch=self.runtime_hooks.pack_step_epoch, + prologue_op=self.runtime_hooks.pack_prologue_op, ) if self.runtime_hooks.pack_step_inputs else None @@ -395,6 +399,14 @@ def __init__( logger.info( "[M3 host] step-input pack: window uploaded ahead of the replay on a side copy stream" ) + if self._step_pack is not None and self._step_pack.epoch is not None: + logger.info( + "[M3 host] step epoch: one device counter bumped by the decode graph's prologue" + ) + if self._step_pack is not None and self._step_pack.prologue_op is not None: + logger.info( + "[M3 host] step-input pack: the decode graph's prologue is one launch (scatter, header, epoch)" + ) if self._h2d_cache is not None and self._h2d_cache.changed_span: logger.info("[M3 host] changed metadata uploads stage only each row's changed span") @@ -1724,6 +1736,9 @@ def run_autotuner_pass(autotune_context: Any, synchronize_trtllm_cache: bool) -> AutoTuner.get().clean_pp_flag() return ran_forward + if self.runtime_hooks.install_tactic_pins is not None and not self._tactic_pins_installed: + self._tactic_pins_installed = True + self.runtime_hooks.install_tactic_pins(self.mapping.rank) cache_path = os.environ.get("TLLM_AUTOTUNER_CACHE_PATH", None) ran_native_forward = run_autotuner_pass( autotune(cache_path=cache_path), synchronize_trtllm_cache=True @@ -1738,6 +1753,15 @@ def run_autotuner_pass(autotune_context: Any, synchronize_trtllm_cache: bool) -> # serving different tactics for one shape: keep the fastest everywhere. # Collective; before the native MXFP8 cache below registers the tactics. AutoTuner.get().post_tune_merge_tactics() + if ( + self.runtime_hooks.persist_flashinfer_mxfp8_tactics + and ran_flashinfer_forward + and cache_path is not None + ): + # The FlashInfer MXFP8 pass tunes after the native pass's autotune() context saved the cache: + # save again so its tactics persist across starts instead of being re-profiled every start. + autotuner = AutoTuner.get() + autotuner.profiling_cache.save_cache(cache_path, autotuner.mapping.rank) if enable_flashinfer_mxfp8_autotuner: if ran_flashinfer_forward: @@ -2136,6 +2160,18 @@ def _run_cuda_graph_warmup(self, resource_manager: ResourceManager): # Tactics tuned while warming the decode graphs agree across # ranks before the capture pass reads them. AutoTuner.get().post_tune_merge_tactics() + cache_path = os.environ.get("TLLM_AUTOTUNER_CACHE_PATH", None) + if ( + self.runtime_hooks.persist_flashinfer_mxfp8_tactics + and is_warmup_only + and flashinfer_methods + and cache_path is not None + ): + # The decode-graph pass tunes after the warmup's autotune() context saved the + # cache: save again so these tactics (merged above when the model asks) persist + # across starts instead of being re-profiled on every server start. + autotuner = AutoTuner.get() + autotuner.profiling_cache.save_cache(cache_path, autotuner.mapping.rank) self._capture_additional_cuda_graphs(resource_manager) # Piecewise graphs have separate capture machinery and do not use the # whole-model attention workspace. Capture them only on the second pass. @@ -2888,6 +2924,12 @@ def _arm_draft_block_offset_guard( if not draft_kv_cache_manager.skip_unchanged_block_offsets: draft_kv_cache_manager.skip_unchanged_block_offsets = True logger.info("[M3 host] draft block-offset copy skipped when unchanged") + if ( + self.runtime_hooks.patch_unchanged_draft_block_offsets + and not draft_kv_cache_manager.patch_unchanged_block_offsets + ): + draft_kv_cache_manager.patch_unchanged_block_offsets = True + logger.info("[M3 host] draft block-offset changes patched with one row copy") def _arm_msa_policy(self, kv_cache_manager: Any) -> None: """Hand the model's MiniMax-M3 MSA preparation choices to the metadata and the KV cache.""" @@ -2913,6 +2955,7 @@ def _arm_msa_policy(self, kv_cache_manager: Any) -> None: kv_cache_manager.configure_decode_shortcuts( skip_unchanged_block_offsets=msa.skip_unchanged_block_offsets, reuse_pool_views=msa.reuse_pool_views, + block_ids_from_views=msa.block_ids_from_views, ) def _set_up_spec_metadata( @@ -2994,11 +3037,13 @@ def _preprocess_inputs( # RuntimeHooks.kv_lens_update_once, a step that corrects the KV lengths below (and then calls it again) calls # it only there: the second call rewrites everything the first one writes, and nothing reads the metadata # in between. - if attn_meta is not None and not ( - self.runtime_hooks.kv_lens_update_once - and self._kv_lens_corrected_in_preprocess(inputs, enable_spec_decode) - ): - attn_meta.on_update_kv_lens() + if attn_meta is not None: + if self.runtime_hooks.kv_lens_update_once and self._kv_lens_corrected_in_preprocess( + inputs, enable_spec_decode + ): + hook_work("KV_LENS_ONCE") + else: + attn_meta.on_update_kv_lens() if enable_spec_decode and not self._config.disable_overlap_scheduler: # When enabling overlap scheduler, the kv cache for draft tokens will @@ -3014,7 +3059,25 @@ def _preprocess_inputs( num_ctx_tokens = inputs["attn_metadata"].num_ctx_tokens num_chunked_ctx_requests = inputs["attn_metadata"].num_chunked_ctx_requests previous_batch_tokens = inputs["input_ids"].shape[0] - num_ctx_tokens - if inputs["position_ids"].ndim == 3: # mrope: [3, 1, N] + # RuntimeHooks.spec_offsets_op: both corrections of a generation-row step in one launch. + fused_offsets = ( + self.runtime_hooks.spec_offsets_op is not None + and inputs["position_ids"].ndim != 3 + and hasattr(inputs["attn_metadata"], "kv_lens_cuda") + and not ( + num_ctx_requests >= num_chunked_ctx_requests + and num_chunked_ctx_requests > 0 + ) + and self.runtime_hooks.spec_offsets_op( + inputs["position_ids"][0, num_ctx_tokens:], + self.previous_pos_id_offsets_cuda[:previous_batch_tokens], + inputs["attn_metadata"].kv_lens_cuda[num_ctx_requests:num_seqs], + self.previous_kv_lens_offsets_cuda[:num_gen_requests], + ) + ) + if fused_offsets: + pass + elif inputs["position_ids"].ndim == 3: # mrope: [3, 1, N] inputs["position_ids"][:, :, num_ctx_tokens:] += ( self.previous_pos_id_offsets_cuda[:previous_batch_tokens] ) @@ -3024,7 +3087,9 @@ def _preprocess_inputs( ] if hasattr(inputs["attn_metadata"], "kv_lens_cuda"): - if ( + if fused_offsets: + pass + elif ( num_ctx_requests >= num_chunked_ctx_requests and num_chunked_ctx_requests > 0 ): @@ -4208,19 +4273,20 @@ def previous_seq_slots_device(): if num_draft_tokens > 0: draft_tokens = torch.tensor(draft_tokens, dtype=torch.int, pin_memory=prefer_pinned()) self.draft_tokens_cuda[: len(draft_tokens)].copy_(draft_tokens, non_blocking=True) - if ( - self._config.is_spec_decode - and len(num_accepted_draft_tokens) > 0 - and not self.runtime_hooks.skip_accepted_draft_upload - ): - num_accepted_draft_tokens = torch.tensor( - num_accepted_draft_tokens, dtype=torch.int, pin_memory=prefer_pinned() - ) - copy_h2d_if_changed( - self.num_accepted_draft_tokens_cuda[: len(num_accepted_draft_tokens)], - num_accepted_draft_tokens, - pack=self.runtime_hooks.spec_small_h2d, - ) + if self._config.is_spec_decode and len(num_accepted_draft_tokens) > 0: + if self.runtime_hooks.skip_accepted_draft_upload: + hook_work("SKIP_ACCEPTED_UPLOAD") + else: + num_accepted_draft_tokens = torch.tensor( + num_accepted_draft_tokens, dtype=torch.int, pin_memory=prefer_pinned() + ) + if self.runtime_hooks.spec_small_h2d: + hook_work("SMALL_H2D") + copy_h2d_if_changed( + self.num_accepted_draft_tokens_cuda[: len(num_accepted_draft_tokens)], + num_accepted_draft_tokens, + pack=self.runtime_hooks.spec_small_h2d, + ) step_fold = self.runtime_hooks.spec_step_fold if self._config.is_spec_decode else None fold_rows = None fold_staged = False diff --git a/tensorrt_llm/_torch/pyexecutor/h2d_cache.py b/tensorrt_llm/_torch/pyexecutor/h2d_cache.py index cd7edb316bef..a9b2efff7cb2 100644 --- a/tensorrt_llm/_torch/pyexecutor/h2d_cache.py +++ b/tensorrt_llm/_torch/pyexecutor/h2d_cache.py @@ -31,6 +31,7 @@ from tensorrt_llm.logger import logger from . import step_inputs +from .runtime_hooks import hook_work class H2DSkipCache: @@ -129,6 +130,7 @@ def _copy_changed_span(self, dst: torch.Tensor, src: torch.Tensor, shadow: tuple return False for row, lo, hi in spans: step_inputs.copy_(dst[row, lo:hi], src[row, lo:hi]) + hook_work("H2D_CHANGED_SPAN") self.span_uploads += 1 self.span_bytes += staged * dst.element_size() self.span_full_bytes += dst.numel() * dst.element_size() diff --git a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py index 080c05303dec..e3b0ca1650be 100644 --- a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py +++ b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py @@ -16,6 +16,7 @@ import math import os import sys +import time from collections import OrderedDict, defaultdict, deque from dataclasses import dataclass, field, replace from enum import IntEnum @@ -558,6 +559,30 @@ def max_admissible_sequences( return max_num_sequences * (2 if needs_extra_leases else 1) +# DRAFT_KV_RESERVE: the draft KV cache manager reserves the DFlash / DSpark drafter's block width per +# generation step (see _get_generation_kv_capacity). Kill switch TRTLLM_M31_DISABLE_DRAFT_KV_RESERVE=1, +# read at import. +_DRAFT_KV_RESERVE = os.environ.get( + "TRTLLM_M31_DISABLE_DRAFT_KV_RESERVE", "0" +).strip().lower() not in ( + "1", + "true", + "on", + "yes", +) + + +# DRAFT_KV_GUARD: a DFlash / DSpark draft KV cache manager reserves the guard page (page 0) by +# default (see KVCacheManagerV2.__init__). Kill switch TRTLLM_M31_DISABLE_DRAFT_KV_GUARD=1, read at +# import; TRTLLM_KV_GUARD_PAGE, when set, still wins. +_DRAFT_KV_GUARD = os.environ.get("TRTLLM_M31_DISABLE_DRAFT_KV_GUARD", "0").strip().lower() not in ( + "1", + "true", + "on", + "yes", +) + + def _get_generation_kv_capacity(spec_config, *, is_draft: bool) -> tuple[int, int]: """Return draft-token reserve and total capacity headroom over history.""" from tensorrt_llm._torch.speculative import get_num_extra_kv_tokens @@ -576,6 +601,20 @@ def _get_generation_kv_capacity(spec_config, *, is_draft: bool) -> tuple[int, in headroom = ( get_num_extra_kv_tokens(spec_config) + spec_config.tokens_per_gen_step + dynamic_reserve ) + if ( + is_draft + and _DRAFT_KV_RESERVE + and (spec_config.spec_dec_mode.is_dflash() or spec_config.spec_dec_mode.is_dspark()) + ): + # A DFlash / DSpark drafter writes its per-step block K/V after its just-advanced context, + # max_draft_len rows that can run past the max_total_draft_tokens reserve. The cpp KV-v2 + # block-offset copy writes an unallocated entry as page 0, which another request owns, so + # requests that overflow in the same append launch race on that page and the drafted + # tokens differ run to run (M3.1 K3 agg, more than 8 requests). Reserving the block width + # keeps every drafter write inside its own pages; unaccepted slots are reclaimed after the + # step as before. + reserve += spec_config.max_draft_len + headroom += spec_config.max_draft_len return reserve, headroom @@ -638,6 +677,38 @@ def _resolve_v2_max_attention_window_vec( return [None if window == max_seq_len else window for window in local_windows] +# The standalone-drafter pool fixes (_resize_draft_context, the committed-history hints and the ceil(heads / tp) cost): +# a model opt-in (RuntimeHooks.draft_pool_fixes), armed process-wide by the KV-cache creator before any sizing +# (arm_draft_pool_fixes). Off, the previous resize and cost behavior. +_DRAFT_POOL_FIXES = False + + +def arm_draft_pool_fixes() -> None: + """Take the standalone-drafter pool fixes in this process (RuntimeHooks.draft_pool_fixes).""" + global _DRAFT_POOL_FIXES + _DRAFT_POOL_FIXES = True + + +def _draft_pool_fixes() -> bool: + return _DRAFT_POOL_FIXES + + +def _resize_draft_context(kv_cache, capacity: int, history: int) -> bool: + """Resize a draft mirror for a context chunk, releasing its stale window pages before allocating. + + KvCache::resize keeps the pages it drops out of the window held (for rollback) until it returns, and + allocates the new chunk while they are still held. With a host tier they are offloaded to make room; when + the GPU is the last cache level (host_cache_size=0) held pages cannot be evicted, so a windowed pool funded + for its window plus one chunk per mirror fails the resize ("Impossible to meet free-slot goal"). Committing + the history first, at the current capacity, allocates nothing and frees those pages on return; the chunk + then needs only the window plus itself. The end state is the same single resize's. + """ + if kv_cache.history_length < history <= kv_cache.capacity < capacity: + if not kv_cache.resize(kv_cache.capacity, history_length=history): + return False + return kv_cache.resize(capacity, history_length=history) + + def _extend_swa_windows_for_reuse( attention_windows: Sequence[Optional[int]], reuse_match_backoff: int, @@ -729,7 +800,15 @@ def _get_static_cache_size_layer_components( head_dim = getattr(config, "head_dim", None) if not isinstance(head_dim, int): head_dim = config.hidden_size // config.num_attention_heads - head_dim = head_dim * num_key_value_heads // tp_size + if is_external_draft and isinstance(num_key_value_heads, int) and _draft_pool_fixes(): + # A standalone drafter's pool is allocated with ceil(heads / tp) heads per rank (__init__'s + # num_kv_heads_per_layer): with fewer KV heads than ranks every rank keeps one whole head. Its windowed + # pool is funded in tokens and converted to bytes with this cost, so costing heads / tp instead funded + # a 4-head drafter at TP8 for half the tokens it holds ("Draft KV cache context resize failed"). + # Per-layer head lists (averaged above) keep the floor formula below; no standalone drafter uses them. + head_dim = head_dim * ((num_key_value_heads + tp_size - 1) // tp_size) + else: + head_dim = head_dim * num_key_value_heads // tp_size kv_factor = 2 cache_size_per_token = kv_factor * head_dim @@ -1255,6 +1334,14 @@ def _spec_recompute_claim_limit( class KVCacheManagerV2(BaseResourceManager): # Sparse managers attach a metadata batch during initialization. sparse_metadata_batch: Batch | None = None + # A standalone draft pool on a disagg generation worker that sizes gen-init requests for the + # window behind the prompt end; the callback seeds the drafter's empty window + # (engage_disagg_gen_init_window). Declared on the class so instances built without __init__ + # keep the plain mirror path. + _disagg_gen_init_window: Optional[Callable[[int, int, int], None]] = None + # The same pool also receives the drafter context window over the cache transceiver + # (engage_disagg_draft_kv_receive): its gen-init pages are then RDMA destinations. + _disagg_draft_kv_receive: bool = False # Filled lazily by _cold_pool_group_membership(); the grouping is fixed after construction. # Declared on the class so it is present even when an instance is built without running __init__. _cold_pool_group_membership_cache: Optional[tuple[tuple[int, frozenset[int]], ...]] = None @@ -1292,6 +1379,17 @@ class KVCacheManagerV2(BaseResourceManager): _block_offset_host: Optional[np.ndarray] = None block_offset_copies: int = 0 block_offset_skips: int = 0 + # With skip_unchanged_block_offsets, opt-in (RuntimeHooks.patch_unchanged_draft_block_offsets): a decode-only copy + # whose destination view and rows equal the last copy's and whose rows' page indices differ in at most + # BLOCK_OFFSET_PATCH_MAX entries rewrites those rows from host-transformed values with one host-to-device row + # copy per pool instead of the copy kernel (see _patch_block_offsets). + patch_unchanged_block_offsets: bool = False + BLOCK_OFFSET_PATCH_MAX = 64 + BLOCK_OFFSET_PATCH_SLOTS = 8 + _block_offset_patch_ring: Optional[list] = None + _block_offset_images: Optional[dict] = None + block_offset_patches: int = 0 + block_offset_patch_ns: int = 0 def __init__( self, @@ -1481,6 +1579,11 @@ def __init__( self._kv_reserve_draft_tokens, self._generation_kv_capacity_headroom = ( _get_generation_kv_capacity(spec_config, is_draft=self.is_draft) ) + if self.is_draft and spec_config is not None: + logger.info( + f"KVCacheManagerV2 (draft): generation reserve {self._kv_reserve_draft_tokens} tokens, " + f"DRAFT_KV_RESERVE {'on' if _DRAFT_KV_RESERVE else 'off'}" + ) self.event_buffer_max_size = kv_cache_config.event_buffer_max_size self.enable_stats = enable_stats @@ -1953,6 +2056,21 @@ def create_cold_page_codec(cache_config: object) -> Optional[object]: # set; with neither set nothing here changes any allocation, any page # content or any reported count. self._guard_page_fill = os.environ.get("TRTLLM_KV_GUARD_PAGE", "").strip().lower() + if ( + not self._guard_page_fill + and self.is_draft + and _DRAFT_KV_GUARD + and spec_config is not None + and (spec_config.spec_dec_mode.is_dflash() or spec_config.spec_dec_mode.is_dspark()) + ): + # DRAFT_KV_GUARD: the cpp block-offset copy writes an unallocated entry as page 0. A + # DFlash / DSpark drafter writes its per-step block through the tables of padding rows and + # of rows past their allocation, so page 0 is written every step; when page 0 belongs to + # a request, its drafter K/V is clobbered and the write order decides the drafts (M3.1 + # K3 agg c56). The guard page is reserved at + # init, before any request, so page 0 is owned by no request. Draft manager only: the + # target manager keeps its pages and CUDA-graph warmups unchanged. + self._guard_page_fill = "zero" self._guard_page_value = _parse_kv_fill_value(self._guard_page_fill, "TRTLLM_KV_GUARD_PAGE") self._guard_page_by_layer: dict[int, int] = {} self._fresh_page_fill = _parse_kv_fill_value( @@ -3997,7 +4115,11 @@ def try_allocate_draft_context(self, req: LlmRequest, num_tokens: int) -> bool: return False target = req.context_current_position + num_tokens + self.num_extra_kv_tokens - return kv_cache.resize(max(kv_cache.capacity, target)) + if not _draft_pool_fixes(): + return kv_cache.resize(max(kv_cache.capacity, target)) + # Same committed-history hint as the unpaired path (_prepare_draft_resources). + history = max(kv_cache.history_length, req.context_current_position) + return _resize_draft_context(kv_cache, max(kv_cache.capacity, target), history) def prepare_disagg_gen_init(self, req: LlmRequest) -> bool: """Prepare KV cache for a disagg generation init request. @@ -4709,6 +4831,51 @@ def _draft_pool_diagnostic(self) -> str: f"windows={self.max_attention_window_vec}, gpu_max_tokens={self._gpu_max_tokens}]" ) + def engage_disagg_gen_init_window(self, on_gen_init: Callable[[int, int, int], None]) -> None: + """Size disagg gen-init requests' draft mirrors for the window behind the prompt end. + + ``on_gen_init(request_id, prompt_len, tokens_per_block)`` runs for each such request; the + drafter worker seeds its empty window there (DFlashWorker.seed_disagg_empty_window). + """ + assert self.is_draft, "only a draft KV cache manager sizes the drafter window" + self._disagg_gen_init_window = on_gen_init + + def engage_disagg_draft_kv_receive(self) -> None: + """Receive the drafter context window into the gen-init window (cache transceiver only). + + The window sizing (``engage_disagg_gen_init_window``) is engaged first, for the same model. + """ + assert self.is_draft, "only a draft KV cache manager receives the drafter context window" + assert self._disagg_gen_init_window is not None, ( + "the drafter window transfer needs the gen-init window sizing engaged" + ) + self._disagg_draft_kv_receive = True + + def _prepare_disagg_gen_init_draft(self, req: LlmRequest, kv_cache) -> None: + """Hold the drafter window behind the prompt for a disagg gen-init request. + + Mirrors ``prepare_disagg_gen_init``: capacity for the prompt plus the draft tokens, and + ``history_length = prompt_len``, so a windowed pool keeps only the pages behind the prompt + end (the window the context side ships) instead of the whole prompt, which its funding + never covered. Its drafter starts from an empty window at the prompt end unless the window + arrives. When it can arrive, a readiness event joins the target's receive fence: RDMA cannot + observe the stream order of the recycled pages. + """ + target = req.prompt_len + get_draft_token_length(req) + self.num_extra_kv_tokens + capacity = max(kv_cache.capacity, target) + if not kv_cache.resize(capacity, history_length=req.prompt_len): + raise RuntimeError( + f"Draft KV cache disagg gen-init resize failed for request " + f"{req.py_request_id}: could not resize to {capacity} tokens" + f"{self._draft_pool_diagnostic()}" + ) + self._after_shrink(req.py_request_id, kv_cache) + self._disagg_gen_init_window(req.py_request_id, req.prompt_len, self.tokens_per_block) + if self._disagg_draft_kv_receive: + ready = torch.cuda.Event() + ready.record(self._stream) + self._disagg_receive_ready[req.py_request_id] = ready + def _prepare_draft_resources(self, scheduled_batch: ScheduledRequests): """Create/resize KV caches in the draft V2 manager for scheduled requests. @@ -4748,6 +4915,9 @@ def _prepare_draft_resources(self, scheduled_batch: ScheduledRequests): raise RuntimeError( f"Failed to resume draft KV cache for request {req.py_request_id}" ) + if self._disagg_gen_init_window is not None and req.is_disagg_generation_init_state: + self._prepare_disagg_gen_init_draft(req, kv_cache) + continue draft_len = get_draft_token_length(req) if req.is_last_context_chunk else 0 capacity = ( req.context_current_position @@ -4775,7 +4945,11 @@ def _prepare_draft_resources(self, scheduled_batch: ScheduledRequests): # plus the generation headroom the pool is funded for, # instead of one prompt-sized chunk that exceeds it. history = max(history, req.context_current_position + req.context_chunk_size) - if not kv_cache.resize(capacity, history): + if not ( + _resize_draft_context(kv_cache, capacity, history) + if _draft_pool_fixes() + else kv_cache.resize(capacity, history) + ): raise self._mirror_resize_error(req, "context", capacity) self._after_shrink(req.py_request_id, kv_cache) @@ -5695,7 +5869,11 @@ def release_resources( ): release_resources(req, free_draft_resources=True) return None - success = draft_kv_cache.resize(new_capacity) + success = ( + draft_kv_cache.resize(new_capacity, history_length=history_hint) + if _draft_pool_fixes() + else draft_kv_cache.resize(new_capacity) + ) if not success: release_resources(req, free_draft_resources=True) return None @@ -6640,6 +6818,8 @@ def _block_offsets_unchanged( storage = dst_tensor.untyped_storage().data_ptr() if num_contexts != 0: guard.pop(storage, None) + if self._block_offset_images is not None: + self._block_offset_images.pop(storage, None) return self._count_block_offset_copy(skipped=False) rows = copy_idx.tolist() key = ( @@ -6653,11 +6833,116 @@ def _block_offsets_unchanged( host = self._block_offset_host = self.host_kv_cache_block_offsets.numpy() current = host[:, rows, 0, :] last = guard.get(storage) - if last is not None and last[0] == key and np.array_equal(last[1], current): - return self._count_block_offset_copy(skipped=True) + if last is not None and last[0] == key: + changed = last[1] != current + if not changed.any(): + return self._count_block_offset_copy(skipped=True) + if ( + self.patch_unchanged_block_offsets + and np.count_nonzero(changed) <= self.BLOCK_OFFSET_PATCH_MAX + and self._patch_block_offsets(dst_tensor, storage, last[1], current, changed) + ): + guard[storage] = (key, current) + self.block_offset_patches += 1 + if self.block_offset_patches == 1: + logger.info( + "[M3 host] BLOCKOFF_PATCH engaged: a draft block-table copy was patched with one row copy " + "(first patch on this rank)" + ) + return self._count_block_offset_copy(skipped=True) guard[storage] = (key, current) + if self._block_offset_images is not None: + self._block_offset_images.pop(storage, None) return self._count_block_offset_copy(skipped=False) + def _block_offset_transform(self, values: np.ndarray, pools: np.ndarray) -> tuple: + """The copy kernel's K / V entries for field-0 page indices ``values`` of pools ``pools`` (int32 arrays of one + shape): K = 0 if v == BAD_PAGE_INDEX (-1) else index_scales[p] * v, V = 0 if v == -1 else K + kv_offset[p], + in int32 with two's-complement wrap, as kvCacheManagerV2Utils.cu writes them.""" + scales = self.index_scales.numpy().astype(np.int32, copy=False)[pools] + offsets = self.kv_offset.numpy().astype(np.int32, copy=False)[pools] + bad = values == -1 + with np.errstate(over="ignore"): + key_index = np.where(bad, np.int32(0), scales * values).astype(np.int32, copy=False) + value_index = np.where(bad, np.int32(0), key_index + offsets).astype( + np.int32, copy=False + ) + return key_index, value_index + + def _patch_block_offsets( + self, + dst_tensor: torch.Tensor, + storage: int, + previous: np.ndarray, + current: np.ndarray, + changed: np.ndarray, + ) -> bool: + """Rewrite the copied rows of ``dst_tensor`` (field-0 page indices ``previous`` -> ``current``, ``changed`` + where they differ) as one host-to-device copy per pool; False (the caller launches the copy kernel) when it + cannot. + + The rows' device image is kept on the host per destination storage: built once from ``previous`` with the copy + kernel's int32 transform (_block_offset_transform) when the destination was last written by the kernel, then + updated in only the changed entries, so the bytes it holds are the ones the kernel would write for + ``current``. The image is staged into one of BLOCK_OFFSET_PATCH_SLOTS pinned slots, each reused only after its + previous copy's event (one per slot, recorded again) completed, and copied over dst[p, :rows] on the manager's + stream. Used only when the destination is contiguous and no capture is in progress. + """ + start = time.perf_counter_ns() + if not dst_tensor.is_contiguous() or torch.cuda.is_current_stream_capturing(): + return False + pools, rows, width = current.shape + if ( + dst_tensor.dim() != 4 + or tuple(dst_tensor.shape[2:]) != (2, width) + or dst_tensor.shape[1] < rows + ): + return False + images = self._block_offset_images + if images is None: + images = self._block_offset_images = {} + image = images.get(storage) + if image is None or image.shape != (pools, rows, 2, width): + image = np.empty((pools, rows, 2, width), dtype=np.int32) + pool_ids = np.arange(pools, dtype=np.intp).reshape(pools, 1, 1) + key_index, value_index = self._block_offset_transform( + previous.astype(np.int32, copy=False), np.broadcast_to(pool_ids, previous.shape) + ) + image[:, :, 0, :] = key_index + image[:, :, 1, :] = value_index + images[storage] = image + pool, row, column = np.nonzero(changed) + key_index, value_index = self._block_offset_transform( + current[pool, row, column].astype(np.int32, copy=False), pool + ) + image[pool, row, 0, column] = key_index + image[pool, row, 1, column] = value_index + ring = self._block_offset_patch_ring + if ring is None: + ring = self._block_offset_patch_ring = [None] * self.BLOCK_OFFSET_PATCH_SLOTS + entry = ring[self.block_offset_patches % self.BLOCK_OFFSET_PATCH_SLOTS] + if entry is None or entry[0].shape != (pools, rows, 2, width): + staging = torch.empty( + (pools, rows, 2, width), dtype=torch.int32, pin_memory=prefer_pinned() + ) + entry = [staging, staging.numpy(), torch.cuda.Event(), False] + ring[self.block_offset_patches % self.BLOCK_OFFSET_PATCH_SLOTS] = entry + elif entry[3]: + entry[2].synchronize() + np.copyto(entry[1], image) + stream = self._stream + if stream == torch.cuda.current_stream(): + for p in range(pools): + dst_tensor[p, :rows].copy_(entry[0][p], non_blocking=True) + else: + with torch.cuda.stream(stream): + for p in range(pools): + dst_tensor[p, :rows].copy_(entry[0][p], non_blocking=True) + entry[2].record(stream) + entry[3] = True + self.block_offset_patch_ns += time.perf_counter_ns() - start + return True + def _count_block_offset_copy(self, skipped: bool) -> bool: """Count one guarded copy (``skipped``: not issued) and log the counts whenever the calls reach a power of two; returns ``skipped``.""" @@ -6669,6 +6954,12 @@ def _count_block_offset_copy(self, skipped: bool) -> bool: if calls & (calls - 1) == 0: logger.info( f"[M3 host] block-offset copies skipped when unchanged: {self.block_offset_skips} of {calls}" + + ( + f", patched {self.block_offset_patches} " + f"({self.block_offset_patch_ns / max(self.block_offset_patches, 1) / 1e3:.1f} us host each)" + if self.patch_unchanged_block_offsets + else "" + ) ) return skipped @@ -6812,6 +7103,46 @@ def probe_first_new_block_key(self, req: LlmRequest) -> bytes | None: scope = ReuseScope(lora_id=req.lora_task_id, salt=self._derive_reuse_salt(req.cache_salt)) return self.impl.probe_first_new_block_key(scope, tokens) + def prefetch_to_gpu(self, req) -> int: + """Migrate a waiting context request's reusable prefix to the GPU tier ahead of its admission. + + Uses a transient, never-resumed cache, as prefetch_for_context_tokens does: it takes no + IndexMapper slot, the migrated pages stay evictable (admission and max_util_for_resume are + unaffected) and the copies run on the cache manager's temporary streams, never on the + execution stream. Returns the bytes of the prefix that were off the GPU (0 when nothing + had to move); a request admitted later then finds its pages ready instead of making the + whole step wait for the copy. + """ + if not self.enable_block_reuse: + return 0 + tokens = self._context_reuse_tokens(req) + reuse_salt = self._derive_reuse_salt(req.cache_salt) + kv_cache = self.impl.create_kv_cache( + ReuseScope(lora_id=req.lora_task_id, salt=reuse_salt), + tokens, + ) + try: + # Reused tokens per cache level (the coldest page of each block), taken when the match + # is made. A held, never-resumed cache exposes no page levels in its page storage + # snapshot (only active caches do), so these counts are what tell where it lives. + off_tokens = sum(kv_cache.cached_tokens_by_level[GPU_LEVEL + 1 :]) + if off_tokens == 0: + return 0 + if not kv_cache.prefetch(GPU_LEVEL): + return 0 + return off_tokens * self._kv_token_bytes_estimate() + finally: + kv_cache.close() + + def _kv_token_bytes_estimate(self) -> int: + """KV bytes of one token over the local layers (for the prefetch hold length only).""" + if getattr(self, "_kv_token_bytes", None) is None: + try: + self._kv_token_bytes = max(1, int(self.get_cache_bytes_per_token())) + except Exception: # noqa: BLE001 - a rough size is enough here + self._kv_token_bytes = 32 << 10 + return self._kv_token_bytes + def prefetch_for_context_tokens(self, requests: list) -> bool: """Prefetch radix-tree blocks from disk→host for upcoming context requests. diff --git a/tensorrt_llm/_torch/pyexecutor/llm_request.py b/tensorrt_llm/_torch/pyexecutor/llm_request.py index d3b49b81c09b..aaf3ddfb001c 100644 --- a/tensorrt_llm/_torch/pyexecutor/llm_request.py +++ b/tensorrt_llm/_torch/pyexecutor/llm_request.py @@ -2,6 +2,7 @@ # SPDX-License-Identifier: Apache-2.0 """Python extensions for executor requests.""" +import os from copy import copy, deepcopy from dataclasses import dataclass, field from enum import Enum, auto @@ -769,6 +770,20 @@ def encoder_output(self) -> torch.Tensor | None: return self._encoder_output +def _response_draft_ctx_state(request) -> Optional[List[int]]: + """Context-first disagg: the drafter window decision ``[ctx_len, ctx_base, ok]`` the context + worker's response carries to the generation request (DisaggregatedParams.draft_ctx_state). + + Set by the KV transceiver's last-slice decision (``py_draft_ctx_aux``) before the response is + built; None for any other request. + """ + draft_ctx = getattr(request, "py_draft_ctx_aux", None) + if draft_ctx is None or not getattr(request, "is_context_only_request", + False): + return None + return [int(draft_ctx[0]), int(draft_ctx[1]), 1 if draft_ctx[2] else 0] + + class LlmResult: """LlmResult wraps `bindings.executor.Result` but detour some features to Python implementation""" py_result_properties = frozenset( @@ -797,6 +812,9 @@ def __init__(self, self.ctx_usage = None # Explicit context-to-generation remote-tail replay boundary. self.remote_tail_start: Optional[int] = None + # Context-first disagg: the standalone drafter's window decision [ctx_len, ctx_base, ok] + # for the generation request (disaggregation/draft_kv.py); never via opaque_state. + self.draft_ctx_state: Optional[List[int]] = None # Time breakdown metrics for performance analysis # Contains: step_metrics (list), ctx_gpu_forward_time (float), ctx_gpu_sample_time (float) self.time_breakdown_metrics = time_breakdown_metrics @@ -1295,6 +1313,7 @@ def create_response(self, ) if getattr(self, "py_csa2_remote_tail_mode", None) == "source": llm_result.remote_tail_start = self.py_csa2_remote_tail_start + llm_result.draft_ctx_state = _response_draft_ctx_state(self) llm_result.cached_tokens = self.cached_tokens return LlmResponse( request_id=self.py_request_id @@ -1511,6 +1530,26 @@ def is_multimodal_encoder_ready(request: LlmRequest) -> bool: return state is None or state.progress is MultimodalEncoderProgress.READY +# Hand the prompt tokens to LlmRequest as the executor request's int32 array (one copy each way) instead of a +# list[int] round trip, O(ISL) Python objects under the GIL. On by default for context-only requests (a disaggregated +# context server); TRTLLM_M31_ENABLE_L1_TOKEN_ARRAY=1 extends it to every request type. Kill switch (wins): +# TRTLLM_M31_DISABLE_L1_TOKEN_ARRAY=1. +_L1_TOKEN_ARRAY = os.environ.get("TRTLLM_M31_DISABLE_L1_TOKEN_ARRAY", + "0") != "1" +_L1_ALL_REQUESTS = os.environ.get("TRTLLM_M31_ENABLE_L1_TOKEN_ARRAY", + "0") == "1" + + +def _l1_input_tokens(executor_request: ExecutorRequest): + """The executor request's prompt tokens for LlmRequest: its int32 array where the array path applies, else the + list.""" + if (_L1_TOKEN_ARRAY and hasattr(executor_request, "input_token_ids_np") + and (_L1_ALL_REQUESTS or executor_request.request_type + == tllm_executor.RequestType.REQUEST_TYPE_CONTEXT_ONLY)): + return executor_request.input_token_ids_np + return executor_request.input_token_ids + + def executor_request_to_llm_request( req_id: int, executor_request: ExecutorRequest, @@ -1520,7 +1559,8 @@ def executor_request_to_llm_request( position_ids: Optional[List] = None) -> LlmRequest: sampling_config = executor_request.sampling_config - input_tokens = input_token_ids if input_token_ids is not None else executor_request.input_token_ids + input_tokens = (input_token_ids if input_token_ids is not None else + _l1_input_tokens(executor_request)) llm_request_type = REQUEST_TYPE_MAPPING[executor_request.request_type] # Kept in its native list[list[int]] form, like py_bad_words. The diff --git a/tensorrt_llm/_torch/pyexecutor/profiling.py b/tensorrt_llm/_torch/pyexecutor/profiling.py index e7bcbd5157fd..15ccb51c6094 100644 --- a/tensorrt_llm/_torch/pyexecutor/profiling.py +++ b/tensorrt_llm/_torch/pyexecutor/profiling.py @@ -55,6 +55,8 @@ from tensorrt_llm.tools.layer_wise_benchmarks import get_calibrator from tensorrt_llm.tools.profiler.host_profile_tools.host_profiler import get_global_profiler +from .runtime_hooks import runtime_hooks_of + # Environment-variable names used by both the env-var-driven legacy # path and the runtime HTTP-endpoint path. They are duplicated as # module-level constants here so callers that only need the manager @@ -608,6 +610,18 @@ def profile_step(self): start_event_2 = None end_event_2 = torch.cuda.Event(enable_timing=True) prev_device_step_time = None + # With RuntimeHooks.iteration_timing_one_event, boundary_events[k % 3] marks the boundary before + # iteration k, recorded on timing_stream, and replaces the start/end ping-pong above. + one_timing_event = runtime_hooks_of( + getattr(executor.model_engine, "model", None) + ).iteration_timing_one_event + boundary_events = ( + [torch.cuda.Event(enable_timing=True) for _ in range(3)] if one_timing_event else None + ) + boundary_count = 0 + timing_stream = torch.cuda.Stream() if one_timing_event else None + if one_timing_event: + logger.info("[M3 host] iteration timing: one side-stream event per iteration boundary") # Cumulative fetch counter as of the previous iter log line; the # per-iteration ``num_fetched`` on the log line is the delta. prev_fetch_requests_cur_rank = 0 @@ -663,6 +677,7 @@ def profile_step_fn(): nonlocal it, enabled, start_time nonlocal start_event_1, end_event_1, start_event_2, end_event_2 nonlocal prev_device_step_time, prev_fetch_requests_cur_rank + nonlocal boundary_count nonlocal torch_profiler, active_torch_trace_path nonlocal active_enable_torch_trace calibrator.post_step(it) @@ -718,9 +733,21 @@ def profile_step_fn(): # ``self`` lets the /metrics serializer pick up the # values without going through the log line. should_capture_timing = executor.print_log or executor.enable_iter_perf_stats + if should_capture_timing and one_timing_event: + # Iteration k - 2 spans the two previous boundaries: the step the ping-pong reports here. + timing_stream.wait_stream(torch.cuda.current_stream()) + boundary_events[boundary_count % 3].record(timing_stream) + if boundary_count >= 2: + boundary_events[(boundary_count - 1) % 3].synchronize() + prev_device_step_time = boundary_events[(boundary_count - 2) % 3].elapsed_time( + boundary_events[(boundary_count - 1) % 3] + ) + boundary_count += 1 if should_capture_timing and start_time is not None: end_time = time.time() - if it % 2 == 0: + if one_timing_event: + pass + elif it % 2 == 0: end_event_1.record() if start_event_2 is not None: end_event_2.synchronize() @@ -847,7 +874,7 @@ def profile_step_fn(): calibrator.pre_step(it) start_time = time.time() - if should_capture_timing: + if should_capture_timing and not one_timing_event: if it % 2 == 0: if start_event_1 is None: start_event_1 = torch.cuda.Event(enable_timing=True) diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor.py b/tensorrt_llm/_torch/pyexecutor/py_executor.py index 87d960f4d2e5..a3e16431d570 100644 --- a/tensorrt_llm/_torch/pyexecutor/py_executor.py +++ b/tensorrt_llm/_torch/pyexecutor/py_executor.py @@ -114,7 +114,7 @@ get_from_waiting_queue, merge_requests) from .resource_manager import (NoFreeSlotsError, ResourceManager, ResourceManagerType, request_context) -from .runtime_hooks import runtime_hooks_of +from .runtime_hooks import RuntimeHooks, runtime_hooks_of from .sampler import (AsyncWorkerMixin, Sampler, SamplerEvent, SampleState, SampleStateTensors) from .scheduler import (RequestScheduler, ScheduledRequests, @@ -133,6 +133,15 @@ _UNBOUNDED_STATS_MAX_LEN = -1 _V2_RETENTION_IGNORED_LOG_KEY = "kv_cache_manager_v2_retention_config_ignored" +# Waiting context requests have their host-tier prefix prefetched to the GPU off the +# execution stream and are admitted after a short, rank-agreed hold. Kill switch. +_KV_GPU_PREFETCH = os.environ.get("TRTLLM_M31_DISABLE_KV_GPU_PREFETCH", + "0") != "1" +_KV_GPU_PREFETCH_MAX = int(os.environ.get("TRTLLM_M31_KV_GPU_PREFETCH_MAX", + "2")) +_KV_GPU_PREFETCH_BYTES_PER_STEP = int(1.5e9) +_KV_GPU_PREFETCH_MAX_HOLD = 8 + class _ADPForwardIntent(IntEnum): # MAX reduction gives context precedence when ADP ranks have mixed work. @@ -379,6 +388,25 @@ class PendingEncoderStep: result: Optional[EncoderStepResult] = None +def _arm_runtime_opt_ins(hooks: RuntimeHooks) -> bool: + """Arm the process-wide features ``hooks`` opt into; return whether to arm + the late-compile detector after warmup. + + All are off for a model that declares none: the SM107 MXFP8 GEMM + tactic table (the GEMM runner does not know the model, so it is armed + here), the MSA saturating E4M3 casts and the late-compile detector. + """ + if hooks.mxfp8_pinned_tactics: + # Imported on use: only an opted-in model needs the table. + from tensorrt_llm._torch.custom_ops import mxfp8_cute_pinned_tactics + mxfp8_cute_pinned_tactics.arm() + if hooks.fp8_saturating_casts: + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import \ + paged_cache + paged_cache.arm_fp8_saturation() + return hooks.late_compile_detect + + class PyExecutor: # True while the overlap loop runs with the execution stream current # (_overlap_loop_stream_scope). @@ -565,6 +593,18 @@ def __init__( self._pending_num_new_active_requests = 0 self._pending_num_completed_requests = 0 self.stream_interval = self.llm_args.stream_interval + # Streaming: also emit on decoding iterations 2..N when the model's + # runtime hooks ask for it (RuntimeHooks.first_content_emit_iters; 1, + # the default, keeps the stock schedule). + model_hooks = runtime_hooks_of(model_engine.model) + self.first_content_emit_iters = max( + 1, model_hooks.first_content_emit_iters) + if self.stream_interval > 1 and self.first_content_emit_iters > 1: + logger.info( + f"first-content emission engaged: emit on decoding iters " + f"<= {self.first_content_emit_iters} (stream_interval " + f"{self.stream_interval}), rank {self.dist.rank}") + self._late_compile_detect = _arm_runtime_opt_ins(model_hooks) self.perf_manager = PerfMetricsManager( enabled=getattr(self.llm_args, 'return_perf_metrics', False)) self.attention_dp_enable_balance = ( @@ -907,6 +947,9 @@ def __init__( self._warmup_encoder_cuda_graphs_enc_dec).result() self.is_warmup = False + if self._late_compile_detect: + from tensorrt_llm._torch.late_compile import mark_server_ready + mark_server_ready() # Snapshot some cumulative KV cache counters so that stats reported to # users exclude blocks reused and missed during warmup dummy requests. @@ -2639,7 +2682,11 @@ def _update_iter_stats( if item.request is None: continue try: - token_count = len(item.request.input_token_ids) + # num_input_tokens skips building the token list; payloads + # without it (duck-typed requests) fall back to the list. + token_count = getattr(item.request, "num_input_tokens", None) + if token_count is None: + token_count = len(item.request.input_token_ids) except (AttributeError, TypeError) as e: # Unusual request shape with no usable token payload; # exclude from all queued counters so downstream consumers @@ -4036,13 +4083,94 @@ def _revert_ctx_alloc(self, dropped_context_requests): if self.enable_joint_kv_cache_reuse: self.draft_kv_cache_manager.free_resources(req) + def _kv_gpu_prefetch_state(self) -> str: + """Whether the host-tier prefix prefetch runs in this executor, and if not, why.""" + if not _KV_GPU_PREFETCH: + return "disabled (kill switch)" + if not self._is_kv_manager_v2: + return "inactive (KV cache manager v1)" + if not self.kv_cache_manager.enable_block_reuse: + return "inactive (block reuse off)" + return "enabled" + + @nvtx_range("_prefetch_context_prefixes_to_gpu") + def _prefetch_context_prefixes_to_gpu(self) -> None: + """Move a waiting request's host-tier prefix to the GPU off the execution stream. + + Admitting a request whose cached prefix sits in the host tier makes resume() put the + swap (evictions + onboarding, several GB for a long conversation) on the execution + stream, so every request in that step waits for it. Instead, the prefix is prefetched + on the cache manager's temporary streams when the request is fetched, and the request + is held out of scheduling for a few steps while the copy runs. The hold length is + derived from the bytes to move and agreed across ranks (MAX), so admission stays the + same on every rank; if the copy is not done by then, resume() still waits for it. + """ + if not hasattr(self, "_kv_gpu_prefetch_count"): + self._kv_gpu_prefetch_count = 0 + self._kv_gpu_prefetch_moved = 0 + self._kv_gpu_prefetch_steps = 0 + self._kv_gpu_prefetch_steps += 1 + # Engagement counter: printed every 200 steps even when no request was a candidate. + if self.dist.rank == 0 and self._kv_gpu_prefetch_steps % 200 == 0: + logger.info(f"[KVPF] steps {self._kv_gpu_prefetch_steps} " + f"candidates {self._kv_gpu_prefetch_count} " + f"prefetched {self._kv_gpu_prefetch_moved}") + for req in self.active_requests: + hold = getattr(req, "py_kv_gpu_hold", 0) + if type(hold) is int and hold > 0: + req.py_kv_gpu_hold = hold - 1 + candidates = [] + for req in self.active_requests: + if len(candidates) >= _KV_GPU_PREFETCH_MAX: + break + if (req.is_first_context_chunk and req.py_request_id + not in self.kv_cache_manager.kv_cache_map and getattr( + req, "py_kv_gpu_prefetched", False) is not True): + candidates.append(req) + if not candidates: + return + holds, moved = [], [] + for req in candidates: + req.py_kv_gpu_prefetched = True + nbytes = self.kv_cache_manager.prefetch_to_gpu(req) + moved.append(max(nbytes, 0)) + holds.append(0 if nbytes <= 0 else min( + _KV_GPU_PREFETCH_MAX_HOLD, + max(1, -(-nbytes // _KV_GPU_PREFETCH_BYTES_PER_STEP)))) + if self.dist.tp_size > 1: + gathered = self.dist.tp_allgather(holds) + holds = [ + max(rank_holds[i] for rank_holds in gathered) + for i in range(len(holds)) + ] + for req, hold, nbytes in zip(candidates, holds, moved): + req.py_kv_gpu_hold = hold + self._kv_gpu_prefetch_count += 1 + if hold: + self._kv_gpu_prefetch_moved += 1 + if self.dist.rank != 0: + continue + if hold and (self._kv_gpu_prefetch_moved <= 50 + or self._kv_gpu_prefetch_moved % 100 == 0): + logger.info(f"[KVPF] prefetch rid {req.py_request_id} off_GB " + f"{nbytes / 1e9:.2f} hold {hold} steps " + f"(#{self._kv_gpu_prefetch_moved})") + @nvtx_range("_prefetch_for_context_requests") def _prefetch_for_context_requests(self) -> None: """Pre-stage disk blocks to host for upcoming context requests with block reuse.""" + if not getattr(self, "_kv_gpu_prefetch_announced", False): + self._kv_gpu_prefetch_announced = True + if self.dist.rank == 0: + logger.info(f"[KVPF] startup: {self._kv_gpu_prefetch_state()} " + f"(max {_KV_GPU_PREFETCH_MAX} candidates per step; " + f"kill TRTLLM_M31_DISABLE_KV_GPU_PREFETCH=1)") if not self._is_kv_manager_v2: return if not self.kv_cache_manager.enable_block_reuse: return + if _KV_GPU_PREFETCH: + self._prefetch_context_prefixes_to_gpu() if self.kv_cache_manager.disk_prefetch_num_reqs <= 0: return max_prefetch = self.kv_cache_manager.disk_prefetch_num_reqs @@ -9083,6 +9211,24 @@ def _emit_first_token_responses(self, prev_scheduled_requests): self._enqueue_responses(new_responses) @nvtx_range("_handle_responses") + def _should_emit_response(self, + request: LlmRequest, + emit_first_iter: bool = True) -> bool: + """Whether ``_handle_responses`` emits a response for ``request`` now. + + Emits on decoding iteration 1, on finish, every ``stream_interval`` + iterations, and on iterations ``2..first_content_emit_iters``. + """ + decoding_iter = request.py_decoding_iter + if request.is_finished: + return True + if decoding_iter == 1: + # The early-emit prototype issues the (non-terminal) iter-1 + # response from `_emit_first_token_responses`; suppress it here. + return emit_first_iter + return (decoding_iter % self.stream_interval == 0 + or 1 < decoding_iter <= self.first_content_emit_iters) + def _handle_responses(self, emit_first_iter: bool = True): new_responses = [] requests_to_terminate = [] @@ -9174,15 +9320,7 @@ def _handle_responses(self, emit_first_iter: bool = True): "route_capture", None) if route_capture is not None: route_capture.attach_routes(request) # R3: append routes - should_emit = (request.py_decoding_iter == 1 or request.is_finished - or request.py_decoding_iter % self.stream_interval - == 0) - # The early-emit prototype issues the (non-terminal) iter-1 - # response from `_emit_first_token_responses`; suppress it here. - if (not emit_first_iter and request.py_decoding_iter == 1 - and not request.is_finished): - should_emit = False - if should_emit: + if self._should_emit_response(request, emit_first_iter): if request.return_perf_metrics: # Response creation may finalize and copy scalar ctx GPU totals. self.perf_manager.compute_batch_gpu_times( diff --git a/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py b/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py index 651777934a81..17735f9cda46 100644 --- a/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py +++ b/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py @@ -14,8 +14,9 @@ from __future__ import annotations +import atexit from dataclasses import dataclass -from typing import TYPE_CHECKING, Optional, Protocol +from typing import TYPE_CHECKING, Any, Callable, Optional, Protocol, Tuple if TYPE_CHECKING: import torch @@ -144,6 +145,25 @@ class MsaRuntimePolicy: skip_unchanged_block_offsets: bool = False # Build each KV pool view once per pool geometry instead of on every attention call. reuse_pool_views: bool = False + # Fill the per-request block-id table by slice-copying each request's page-index view instead of + # building a Python list of the whole padded row (max_seq_len / tokens_per_block entries) first. + block_ids_from_views: bool = False + # Expand the sub-page block table on the device inside the forward and invalidate the unused out-cache-loc tail + # there too, instead of between the decode graphs in prepare(); the tables a forward reads are the same. + forward_staging: bool = False + # Prefill indexer proxy plan (the max-score fmha_sm100 pass). Max-score plans need no split merge, so every option + # below gives the same score table bits. prefill_proxy_qo_tile: maxscore_qo_tile for a step whose longest planned + # query span exceeds 128 rows, when the loaded planner accepts it; prefill_proxy_kv_splits is then that plan's + # num_kv_splits. prefill_proxy_kv_splits_by_rows: ((max_rows, top_split, min_kv_pages_per_split), ...) in ascending + # max_rows: the first entry covering the step's longest planned query span gives S and F, and the plan takes S + # halved until ceil(max kv_len / page) / S >= F (down to 1), over every planned request. A span of at most 128 rows + # takes it under the planner's own tile; a longer span takes it only with the tile. + prefill_proxy_qo_tile: Optional[int] = None + prefill_proxy_kv_splits: Optional[int] = None + prefill_proxy_kv_splits_by_rows: Optional[Tuple[Tuple[int, int, int], ...]] = None + # Diagnostic, off by default: also score every proxy span under the plain plan (num_kv_splits 1, the planner's + # tile) and count the score and selection mismatches against the planned pass. + prefill_proxy_oracle: bool = False @dataclass(frozen=True) @@ -163,6 +183,13 @@ class RuntimeHooks: # previous graph still runs, so the prologue only waits on that copy and scatters device to device (no host copy # at the head of the replay). pack_early_copy: bool = False + # With pack_in_graph: the pack keeps a step epoch, one int64 device counter the graph's prologue increments once per + # replay and never resets, so in-graph per-step flags can be tagged with it instead of cleared between steps. + pack_step_epoch: bool = False + # With pack_early_copy: the model's one-launch prologue, called inside the capture as + # ``op(window, epoch_or_None, max_desc)``. It writes the window's host copies, then its device copies, zeroes the + # window's header and increments the epoch, which the default prologue does with two scatter launches and a fill. + pack_prologue_op: Optional[Callable[[torch.Tensor, Optional[torch.Tensor], int], None]] = None # Skip host-to-device metadata copies whose values the destination already holds. skip_unchanged_h2d: bool = False # With skip_unchanged_h2d: a changed packed 2-D upload stages only each row's changed column span (a page table that @@ -172,12 +199,47 @@ class RuntimeHooks: # after the forward's KV-length correction, instead of also before it: the first pass's device writes are all # rewritten by the second. kv_lens_update_once: bool = False + # The overlap scheduler's speculative step correction (position_ids += position offsets, kv_lens += KV-length + # offsets) as the model's one launch, called as ``op(positions, position_offsets, kv_lens, kv_lens_offsets)`` on the + # generation rows; it returns False to decline (the two in-tree adds run). + spec_offsets_op: Optional[ + Callable[[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor], bool] + ] = None # Run the overlap executor loop with the execution stream current, so the # loop's own stream work and the forward are ordered on one stream. loop_on_execution_stream: bool = False # After autotuning, keep the fastest profiled tactic per shape on every # tensor-parallel rank, so the ranks capture the same kernels. merge_tuned_tactics_across_ranks: bool = False + # Called once with this process's rank before the warmup tuning pass, to preload tuned tactics (the model's + # bundled pin files) into the autotuner; shapes it covers are served instead of profiled. + install_tactic_pins: Optional[Callable[[int], Any]] = None + # With TLLM_AUTOTUNER_CACHE_PATH set, save the autotuner cache again after the + # FlashInfer MXFP8 passes (warmup and the decode-graph pass) tune, so their + # tactics persist across starts instead of being re-profiled on every start. + persist_flashinfer_mxfp8_tactics: bool = False + # Streaming: also emit a response on decoding iterations 2..N, so the first + # counted text reaches the client one step after the first token instead of + # one stream interval later (a reasoning parser holds a leading reasoning + # tag back, so the iteration-1 chunk can carry no content). Emission only; + # tokens are unchanged. 1 keeps the stock schedule. + first_content_emit_iters: int = 1 + # On SM107, run FlashInfer's CuTe DSL MXFP8 GEMM above 32 rows at the pinned tuned tactics of + # custom_ops/mxfp8_cute_pinned_tactics (bitwise the default tactic's output, faster). The GEMM runner does not + # know the model, so the executor arms the table process-wide when the model opts in. + mxfp8_pinned_tactics: bool = False + # After warmup, log every CuTe DSL / Triton compile as a late compile (tensorrt_llm/_torch/late_compile.py). + late_compile_detect: bool = False + # Saturate the MSA backend's E4M3 cache writes and dense-decode query cast to +-448, as the CUDA FP8 writers do + # (bitwise the plain cast for every finite in-range value). Armed process-wide by the executor. + fp8_saturating_casts: bool = False + # MiniMax-M3 MSA KV cache: zero the KV pools once at allocation and price the manager's real per-token footprint + # in the pre-allocation estimate (attention/backends/sparse/minimax_m3/cache_manager.py). Armed process-wide by + # the KV-cache creator before any sizing. + msa_kv_cache_fixes: bool = False + # Standalone-drafter KV pool: ceil(heads / tp) per-rank cost and the history-first context resize + # (kv_cache_manager_v2.py). Armed process-wide by the KV-cache creator before any sizing. + draft_pool_fixes: bool = False # Skip the per-step Philox seed and offset uploads on all-greedy # one-model speculative batches (the argmax path never reads them). skip_greedy_rng_upload: bool = False @@ -204,12 +266,20 @@ class RuntimeHooks: # Copy the one-model speculative sampler's per-slot store to the host on the sampler's D2H side # stream, so those copies overlap the next step's input staging instead of preceding it. sampler_d2h_side_stream: bool = False + # Time the iteration log's device step with one timing event per iteration boundary, recorded on a side + # stream that waits for the current stream, instead of an end and a start event back to back on the + # current stream (the same instant, with no GPU work between them). The reported step times are unchanged. + iteration_timing_one_event: bool = False # MiniMax-M3 MSA metadata preparation. msa: MsaRuntimePolicy = MsaRuntimePolicy() # A separate draft KV cache manager (KVCacheManagerV2) skips a decode-only block-table copy whose destination, # copied rows and those rows' page indices equal the previous copy's (the standalone DSpark drafter's table is # unchanged on most decode steps; see KVCacheManagerV2._block_offsets_unchanged). skip_unchanged_draft_block_offsets: bool = False + # With skip_unchanged_draft_block_offsets: a decode-only draft block-table copy whose rows changed in a few page + # indices (the drafter's sliding window: a tail page allocated or re-allocated, a front page released) rewrites the + # rows with one host-to-device row copy instead of the copy kernel (KVCacheManagerV2._patch_block_offsets). + patch_unchanged_draft_block_offsets: bool = False # The one-model speculative sampler's tail bookkeeping without its no-op launches, bitwise the same outputs # (copied to spec_metadata.tail_bookkeep): at one decode request the draft block reads its sampling parameters as # views of the per-token buffers, a draft block with no d2t map skips the int32 -> int64 -> int32 round trip, and @@ -253,3 +323,45 @@ def runtime_hooks_of(model: object) -> RuntimeHooks: f"{type(model).__name__}.runtime_hooks must be a RuntimeHooks, got {type(hooks).__name__}" ) return hooks + + +# Work counters (log-only) for the step-loop items above. The code that does an armed hook's work calls +# ``hook_work(ITEM)`` on the path where the hook is taken (not where the flag is merely read); the hot-path cost is one +# dict increment. Counts are per process, in host code: under CUDA graphs a count is host executions (eager steps and +# captures; graph replays run no host code and are not counted), and every line says so. +HOOK_WORK_COUNTS: dict[str, int] = {} +HOOK_WORK_HOST_COUNT = ( + "host executions: eager steps + CUDA-graph captures, graph replays not counted" +) +_hook_work_exit = [False, False] # [atexit registered, final lines logged] + + +def _hook_work_log(message: str) -> None: + from tensorrt_llm._utils import mpi_rank + from tensorrt_llm.logger import logger + + logger.info(f"{message} (rank {mpi_rank()}; {HOOK_WORK_HOST_COUNT})") + + +def hook_work(item: str) -> None: + """Count one unit of ``item``'s work in this process. + + Logs ``[M3 host] engaged: work N`` at the first count and at every power of two, and (once per process, at + exit) ``[M3 host] engaged: final work N``. Log-only: no device work, no synchronization. + """ + count = HOOK_WORK_COUNTS.get(item, 0) + 1 + HOOK_WORK_COUNTS[item] = count + if count & (count - 1) == 0: + if not _hook_work_exit[0]: + _hook_work_exit[0] = True + atexit.register(log_hook_work_final) + _hook_work_log(f"[M3 host] {item} engaged: work {count}") + + +def log_hook_work_final() -> None: + """Log every counted item's final count, once per process (registered at exit by the first ``hook_work``).""" + if _hook_work_exit[1]: + return + _hook_work_exit[1] = True + for item, count in sorted(HOOK_WORK_COUNTS.items()): + _hook_work_log(f"[M3 host] {item} engaged: final work {count}") diff --git a/tensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.py b/tensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.py index 3b7e55c8fe33..15ce751f891d 100644 --- a/tensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.py +++ b/tensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.py @@ -579,6 +579,11 @@ def preempt_for_pages(req: LlmRequest) -> bool: continue if req.py_request_id in preempted_ids: continue + # Its host-tier prefix is being copied to the GPU off the execution stream + # (PyExecutor._prefetch_context_prefixes_to_gpu); admit it when the hold ends. + hold = getattr(req, "py_kv_gpu_hold", 0) + if type(hold) is int and hold > 0: + continue remote_mode = getattr(req, "py_csa2_remote_tail_mode", None) if defer_remote_tail_context and remote_mode is not None: continue diff --git a/tensorrt_llm/_torch/pyexecutor/step_inputs.py b/tensorrt_llm/_torch/pyexecutor/step_inputs.py index 4207c341cce7..3e11d32cfb50 100644 --- a/tensorrt_llm/_torch/pyexecutor/step_inputs.py +++ b/tensorrt_llm/_torch/pyexecutor/step_inputs.py @@ -39,6 +39,17 @@ ``prologue_done``, which the next upload waits for before it overwrites ``gdev``. The next fill of the pinned window waits for ``ghost_free``, which the upload records once it has read the window. +With ``step_epoch`` (``RuntimeHooks.pack_step_epoch``, on top of ``in_graph``) the pack keeps one int64 device counter, +``StepInputPack.epoch`` (``step_epoch()`` for the active pack), which the prologue's first scatter launch increments +once per replay (program 0, before any other kernel of the graph runs). It is never reset: it starts at 0, advances once +when the prologue kernel is compiled and then once per replay of any graph with the prologue, with or without a new pack +to scatter, and not on an eager forward. A kernel later in the graph reads it after its programmatic-dependent-launch +wait, so per-step device flags can be tagged with it instead of being cleared between steps. + +With ``prologue_op`` (``RuntimeHooks.pack_prologue_op``, on top of ``early_copy``) the prologue is the model's one +launch, ``prologue_op(gdev, epoch, MAX_DESC)``, in place of the two scatter launches and the header fill: it writes the +same bytes, zeroes the same header and increments the epoch the same once per replay (and once at warm-up). + Every in-graph attribute has a class-level default (off), so a pack built without its ``__init__`` and every pack built with both flags off takes exactly the path above (``flush()`` and ``flush(in_graph=True)`` are then the same call). """ @@ -47,7 +58,7 @@ import contextlib from contextvars import ContextVar -from typing import Iterator, Optional +from typing import Callable, Iterator, Optional import torch import triton @@ -79,12 +90,22 @@ def _parallel_scatter_kernel( # The in-graph prologue's scatter: no integer argument (the counts are read from the staged header and the grid is the -# fixed MAX_DESC), so the two DEVICE_SRC variants its warm-up compiles serve every replay and every capture. +# fixed MAX_DESC), so the variants its warm-up compiles serve every replay and every capture. @triton.jit -def _graph_scatter_kernel(stage_u8, stage_i64, DEVICE_SRC: tl.constexpr, BLOCK: tl.constexpr): +def _graph_scatter_kernel( + stage_u8, + stage_i64, + epoch, + DEVICE_SRC: tl.constexpr, + BLOCK: tl.constexpr, + BUMP_EPOCH: tl.constexpr, +): """Descriptor slot i of the host copies (``DEVICE_SRC`` False) or of the device copies, with the counts read from - the staged header; a slot past its count writes nothing.""" + the staged header; a slot past its count writes nothing. ``BUMP_EPOCH``: program 0 also increments ``epoch[0]``.""" i = tl.program_id(0) + if BUMP_EPOCH: + if i == 0: + tl.store(epoch, tl.load(epoch) + 1) num_h2d = tl.load(stage_i64) if DEVICE_SRC: count = tl.load(stage_i64 + 1) @@ -147,6 +168,8 @@ class StepInputPack: parallel_scatter = False in_graph = False early_copy = False + prologue_op: Optional[Callable[[torch.Tensor, Optional[torch.Tensor], int], None]] = None + epoch: Optional[torch.Tensor] = None # Engagement counters of this pack (one per engine and rank): packs the replay's prologue issued, packs issued # eagerly before a prologue graph (over the window or not order-free), and side-stream window uploads # (early_copy). Logged whenever the in-graph flushes (windows + fallbacks) reach a power of two. @@ -160,6 +183,8 @@ def __init__( parallel_scatter: bool = False, in_graph: bool = False, early_copy: bool = False, + step_epoch: bool = False, + prologue_op: Optional[Callable[[torch.Tensor, Optional[torch.Tensor], int], None]] = None, ) -> None: # Only an opted-in pack needs the native scatter fake registration. from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 import ( # noqa: F401 @@ -169,6 +194,14 @@ def __init__( self.parallel_scatter = parallel_scatter self.in_graph = in_graph and parallel_scatter self.early_copy = early_copy and self.in_graph + # The model's one-launch prologue (see the module docstring); None: the scatter launches and the header fill. + self.prologue_op = prologue_op if self.early_copy else None + # The step epoch (see the module docstring); None without step_epoch. + self.epoch = ( + torch.zeros(1, dtype=torch.int64, device=device) + if step_epoch and self.in_graph + else None + ) if self.early_copy: # The prologue's own device window: an eager flush (self.dev) can never leave a header in it. self.gdev = torch.zeros(self.GRAPH_WINDOW, dtype=torch.uint8, device=device) @@ -217,17 +250,27 @@ def _warm_up(self, device: torch.device) -> None: torch.cuda.synchronize(device) def _warm_up_graph_kernel(self) -> None: - """Compile both variants of the prologue's scatter now, not inside a capture (a zero-count header: it writes - nothing).""" + """Compile the prologue's scatter now (a zero-count header: it writes nothing).""" self.dev[: self.HEADER_BYTES].zero_() - self._launch_graph_scatter(self.dev) + if self.prologue_op is not None: + # The window it is captured with; its header is zero until the first upload. + self.prologue_op(self.gdev, self.epoch, self.MAX_DESC) + else: + self._launch_graph_scatter(self.dev) torch.cuda.synchronize(self.dev.device) def _launch_graph_scatter(self, window: torch.Tensor) -> None: stage_i64 = window.view(torch.int64) for device_src in (False, True): + bump = self.epoch is not None and not device_src _graph_scatter_kernel[(self.MAX_DESC,)]( - window, stage_i64, DEVICE_SRC=device_src, BLOCK=256, num_warps=1 + window, + stage_i64, + self.epoch if bump else stage_i64, + DEVICE_SRC=device_src, + BLOCK=256, + BUMP_EPOCH=bump, + num_warps=1, ) def capture_prologue(self) -> bool: @@ -237,9 +280,12 @@ def capture_prologue(self) -> bool: return False if self.early_copy: torch.cuda.current_stream().wait_event(self.staged_ready) - self._launch_graph_scatter(self.gdev) - # Consumed: a replay with no new upload scatters nothing. - self.gdev[: self.HEADER_BYTES].zero_() + if self.prologue_op is not None: + self.prologue_op(self.gdev, self.epoch, self.MAX_DESC) + else: + self._launch_graph_scatter(self.gdev) + # Consumed: a replay with no new upload scatters nothing. + self.gdev[: self.HEADER_BYTES].zero_() self.prologue_done.record() return True self.dev[: self.GRAPH_WINDOW].copy_(self.ghost, non_blocking=True) @@ -464,6 +510,13 @@ def flush(in_graph: bool = False) -> None: pack.flush(in_graph=in_graph) +def step_epoch() -> Optional[torch.Tensor]: + """The active pack's step epoch (an int64 [1] device tensor), or None when there is none (see the module + docstring).""" + pack = _active.get() + return pack.epoch if pack is not None else None + + def capture_prologue() -> Optional[StepInputPack]: """Inside a graph capture: add the active pack's in-graph prologue; the pack whose window the graph issues, or None when the graph has no prologue (no active pack, or one without ``in_graph``: nothing is captured).""" diff --git a/tensorrt_llm/_torch/speculative/dflash.py b/tensorrt_llm/_torch/speculative/dflash.py index 6c9dc7ab938e..388812061097 100644 --- a/tensorrt_llm/_torch/speculative/dflash.py +++ b/tensorrt_llm/_torch/speculative/dflash.py @@ -16,7 +16,7 @@ import os from collections import deque from dataclasses import dataclass -from typing import TYPE_CHECKING, Callable, List, Optional, Tuple +from typing import TYPE_CHECKING, Callable, Dict, List, Optional, Tuple import torch from torch import nn @@ -30,6 +30,7 @@ from ..pyexecutor.kv_cache.mamba_cache_manager import MambaHybridCacheManager from ..pyexecutor.llm_request import ATTENTION_DP_DUMMY_REQUEST_ID from ..pyexecutor.resource_manager import BaseResourceManager +from ..pyexecutor.runtime_hooks import hook_work from .accept_stats import maybe_create_recorder from .dflash_attention import ( get_dflash_paged_append, @@ -519,6 +520,34 @@ def dflash_allocated_ctx_limit( return (block_counts * page_size - block_size).clamp(min=0) +_HOST_CTX_ALLOC_DISABLED = os.environ.get("TRTLLM_DFLASH_DISABLE_HOST_CTX_ALLOC", "0") == "1" +_HOST_CTX_ALLOC_CHECK = os.environ.get("TRTLLM_DFLASH_CHECK_HOST_CTX_ALLOC", "0") == "1" +# Bound on the disagg drafter-context maps (context-side store events, generation-side seeds): an +# entry is normally consumed within an iteration, so only requests that failed or were cancelled in +# between linger, and the oldest are dropped first. +_DISAGG_DRAFT_CTX_MAP_CAP = 8192 + + +def _block_counts_are_full_rows(draft_kv_cache_manager) -> bool: + """Whether every row of the draft block table counts full in ``_ctx_block_counts``. + + ``KVCacheManagerV2.copy_batch_block_offsets`` fills the table with one + ``copyBatchBlockOffsetsToDevice`` launch, which writes ``BAD_PAGE_INDEX`` as + 0 and every page as ``scale * index`` (kvCacheManagerV2Utils.cu), so no entry + is negative and ``(entry >= 0).sum()`` is the table width. A subclass that + overrides the copy, or the per-layer page-table path, may leave placeholders + negative, so neither counts as full. + """ + from ..pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 + + return ( + isinstance(draft_kv_cache_manager, KVCacheManagerV2) + and type(draft_kv_cache_manager).copy_batch_block_offsets + is KVCacheManagerV2.copy_batch_block_offsets + and not getattr(draft_kv_cache_manager, "_use_per_layer_page_tables", True) + ) + + def dflash_rebased_ctx_rows(rows: torch.Tensor, base_pages: torch.Tensor) -> torch.Tensor: """Block-table rows shifted left by ``base_pages`` entries each. @@ -557,6 +586,16 @@ def dflash_rebased_ctx_view( return dflash_rebased_ctx_rows(rows, rebase_pages), num_ctx +def _context_onehot_vocab(spec_metadata, num_gens: int, draft_logits_vocab: int): + """The draft-prob width of a mixed step's context one-hot rows: the width the gen scatter published to + ``draft_probs`` (what acceptance reads next iteration), not the draft logits' width. A vocab-sharded drafter + gathers its logits to the full vocab before the scatter, so its shard width would leave stale columns in the + new requests' rows. None when the step has no generation requests (the writer then uses the buffer's width).""" + if num_gens <= 0: + return None + return spec_metadata.draft_probs_last_dim or draft_logits_vocab + + @dataclass class DFlashSpecMetadata(SpecMetadata): """Metadata for DFlash speculative decoding. @@ -580,6 +619,10 @@ class DFlashSpecMetadata(SpecMetadata): _batch_indices_uploaded = 0 def __post_init__(self): + # Preserve the initial slot capacity across CUDA graph copies, whose + # max_num_requests is narrowed to the captured graph bucket. + self.num_seq_slots = self.num_seq_slots or self.max_num_requests + self.batch_indices_cuda = torch.empty( [self.max_num_requests], dtype=torch.int, @@ -630,7 +673,9 @@ def prepare(self): changed_only = getattr(self, "batch_map_changed_only", False) # The batch indices are an arange, so with batch_map_changed_only a buffer already holding a long enough # prefix is not re-uploaded (nothing else writes batch_indices_cuda). - if not (changed_only and num_seqs <= self._batch_indices_uploaded): + if changed_only and num_seqs <= self._batch_indices_uploaded: + hook_work("BATCH_CHANGED_ONLY") + else: batch_indices = torch.arange( num_seqs, dtype=torch.int, device="cpu", pin_memory=prefer_pinned() ) @@ -655,26 +700,56 @@ def prepare(self): # prefilling, so _store_prefill_context -- the only other assigner -- # never runs and every request would land on the dummy slot. num_contexts = max(0, len(self.request_ids) - self.num_generations) + # A disagg generation request's drafter context lengths (seed_disagg_draft_ctx: the + # window received with the prompt KV, or an empty window at the prompt end), applied + # when the request first holds a real slot: a slot taken earlier would have been + # evicted above, and the dummy slot is never seeded. + seeds = getattr(worker, "_disagg_draft_ctx_seeds", None) + seeded_lens, seeded_bases, seeded_rids = {}, {}, [] for rid in self.request_ids[num_contexts:]: if ( rid != ATTENTION_DP_DUMMY_REQUEST_ID and rid < worker._graph_dummy_id_floor and rid not in worker._req_to_slot ): - if worker._assign_slot(rid) is None: + slot = worker._assign_slot(rid) + if slot is None: # Not fatal -- spec dec is lossless, so the dummy-slot # fallback only costs acceptance. Warn so it is not silent. logger.warning( f"DFlash: no free drafter slot for generation request {rid}; " "it shares the dummy slot and will draft poorly." ) + elif seeds: + seed = seeds.pop(rid, None) + if seed is not None: + ctx_len, ctx_base, prompt_len = seed + seeded_lens[slot] = ctx_len + seeded_bases[slot] = ctx_base + seeded_rids.append(rid) + worker._req_ctx_pos[rid] = prompt_len + if seeded_lens and getattr(worker, "_disagg_draft_kv_engaged", False): + # A received window lives in the draft pool: the drafter must be pool-bound. + worker._check_disagg_draft_ctx_pool_bound() + if seeded_lens and worker._ctx_block_tables is not None: + # Only a pool-bound context addresses the draft pool's pages; the private arena + # numbers every slot from 0 and has nothing to rebase. + worker._write_ctx_len(seeded_lens, seeded_bases) + hooks = getattr(worker, "_disagg_draft_ctx_seed_hooks", None) + if hooks: + for rid in seeded_rids: + hook = hooks.pop(rid, None) + if hook is not None: + hook() # Route unknown request IDs (cuda graph padding or warmup dummies) # to dummy slot to avoid corrupting real request's context slots = [worker._req_to_slot.get(rid, worker._dummy_slot) for rid in self.request_ids] # With batch_map_changed_only the map is re-uploaded only when it differs from the last upload (see # DFlashWorker._batch_to_slot_uploaded). - if not (changed_only and getattr(worker, "_batch_to_slot_uploaded", None) == slots): + if changed_only and getattr(worker, "_batch_to_slot_uploaded", None) == slots: + hook_work("BATCH_CHANGED_ONLY") + else: mapping = torch.tensor( slots, dtype=torch.long, @@ -761,6 +836,20 @@ class DFlashWorker(SpecWorkerBase): # The map DFlashSpecMetadata.prepare last uploaded into _batch_to_slot[:len(map)] under batch_map_changed_only, or # None when unknown. Every other writer of _batch_to_slot (its allocation, the context-step rebuild) resets it. _batch_to_slot_uploaded: Optional[List[int]] = None + # Disaggregated serving with the drafter context window shipped (disaggregation/draft_kv.py), + # engaged by the cache transceiver only. Class-level so workers built without __init__ stay off. + _disagg_draft_kv_engaged: bool = False + # Context side: request id -> (store event, ctx_len, ctx_base) of its latest prefill store. + _disagg_draft_ctx_ready: Optional[Dict[int, Tuple[torch.cuda.Event, int, int]]] = None + # Generation side of disaggregated serving: request id -> (ctx_len, ctx_base, prompt_len) its + # drafter slot starts from (seed_disagg_draft_ctx). Class-level so workers built without + # __init__ never seed. + _disagg_draft_ctx_seeds: Optional[Dict[int, Tuple[int, int, int]]] = None + # Test only (TRTLLM_M31_F0A_DUMP_DIR): request id -> callback run right after its seed lands. + _disagg_draft_ctx_seed_hooks: Optional[Dict[int, Callable[[], None]]] = None + + # Context arena sized by num_seq_slots (enable_full_slot_arena) instead of max_num_requests. + _full_slot_arena = False def __init__( self, @@ -804,6 +893,7 @@ def __init__( self._kv_rewind_nc = None self._kv_rewind_bs = None self._batch_to_slot = None + self._batch_to_slot_uploaded = None self._max_ctx = 0 self._ctx_k_buf = None # [max_batch+1, L, max_ctx+block, nkv, hd] self._ctx_v_buf = None @@ -831,6 +921,11 @@ def __init__( # Pages the manager handed this request, per batch row. Saturates at # the table width on the default backend -- dflash_allocated_ctx_limit. self._ctx_block_counts = None + # Width of the draft block table last decoded, and whether every row of + # it counts full (see _host_ctx_alloc): then the per-request allocation + # the prefill store caps at is known on the host, with no device sync. + self._ctx_table_width = 0 + self._ctx_counts_full_row = False self._ctx_pool_idx = 0 self._ctx_block_divisor = 1 # encoded offset -> raw pool block index self._ctx_kv_indptr = None @@ -1076,6 +1171,7 @@ def _init_ctx_block_tables( 0, (num_slots + 1) * max_blocks, max_blocks, dtype=torch.int32, device="cuda" ) self._ctx_block_counts = torch.zeros(num_slots, dtype=torch.long, device="cuda") + self._ctx_counts_full_row = _block_counts_are_full_rows(mgr) logger.info( f"DFlash: ctx block tables {tuple(self._ctx_block_tables.shape)}, " f"pool_idx={self._ctx_pool_idx}, divisor={self._ctx_block_divisor}, " @@ -1105,6 +1201,7 @@ def _refresh_ctx_block_tables(self, attn_metadata, num_seqs: int) -> bool: # pool, and the drafter addresses blocks, not planes. # clone(): .to() is a no-op for an int64 source, and the in-place ops # below would then edit attn_metadata's own tensor. + self._ctx_table_width = int(src.shape[-1]) if self._fused_block_refresh is not None and self._fused_block_refresh( src, self._ctx_pool_idx, @@ -1152,7 +1249,15 @@ def _lazy_init_ctx_buffers( self._ctx_block_tables = None self._ctx_block_tables_read = None self._ctx_block_counts = None - max_batch = spec_metadata.max_num_requests + # Worker-owned and allocated once, then reused for every later batch + # shape. With enable_full_slot_arena it spans num_seq_slots, the + # surviving proxy for the pre-graph capacity after max_num_requests is + # narrowed to a captured graph bucket (2 * max_num_requests under + # disagg-ADP: the large context K/V buffers are overallocated to keep + # one safe capacity across graph buckets); otherwise max_num_requests. + max_batch = ( + spec_metadata.num_seq_slots if self._full_slot_arena else spec_metadata.max_num_requests + ) # ctx_len is 1:1 with the target's positions, so max_seq_len bounds it. # max_position_embeddings does not: K3's drafter advertises 1048576, @@ -1380,6 +1485,27 @@ def _ctx_paged_index_args(self): indptr = self._ctx_kv_indptr return table.flatten().contiguous(), indptr + def _host_ctx_alloc(self, num_contexts: int) -> List[int]: + """Per-context-request allocation (positions) the prefill store caps at. + + It is ``_ctx_block_counts * page``, counted on the device from the draft + block table. Reading it back is a device sync once per context chunk: the + host waits for the target's whole chunk forward, then runs the rest of + the store while the GPU idles. When every row counts full + (``_block_counts_are_full_rows``) the value is ``width * page`` for every + request, known on the host. Otherwise, or with + ``TRTLLM_DFLASH_DISABLE_HOST_CTX_ALLOC=1``, it is read back as before. + ``TRTLLM_DFLASH_CHECK_HOST_CTX_ALLOC=1`` reads it back as well and + asserts the two agree. + """ + if self._ctx_counts_full_row and self._ctx_table_width > 0 and not _HOST_CTX_ALLOC_DISABLED: + alloc = [self._ctx_table_width * self._ctx_page_size] * num_contexts + if _HOST_CTX_ALLOC_CHECK: + device = (self._ctx_block_counts[:num_contexts] * self._ctx_page_size).tolist() + assert alloc == device, f"DFlash host ctx allocation {alloc} != device {device}" + return alloc + return (self._ctx_block_counts[:num_contexts] * self._ctx_page_size).tolist() + def _ctx_store_start(self, cur: int, first_pos: int) -> int: """Position at which a context chunk's drafter K/V is stored. @@ -1438,12 +1564,203 @@ def clear(slot: int) -> None: clear(slot) return self._req_to_slot[req_id] + def enable_full_slot_arena(self) -> None: + """Size the worker-owned context arena by the full seq-slot range (num_seq_slots) instead of + max_num_requests; call before the first prepare. Off by default.""" + self._full_slot_arena = True + + def enable_window_capped_decode(self, draft_model, max_batch: int) -> None: + """Bound the drafter's windowed block decode max_seq_len by the window (batches of at most ``max_batch`` + requests; 0 = off, the default). See DFlashForCausalLM._decode_max_seq_len.""" + if hasattr(draft_model, "_dflash_window_cap_max_batch"): + draft_model._dflash_window_cap_max_batch = int(max_batch) + + def enable_fused_ar_norm(self, draft_model, enabled: bool = True) -> bool: + """Fuse each drafter layer's row-parallel all-reduce (o_proj, down_proj) with the add + RMSNorm after it. Off + by default. Returns whether it is on: it needs a TP group, no block convolutions, plain (non-Gemma) RMSNorms and + Linears that own their all-reduce.""" + if not hasattr(draft_model, "_dflash_fused_ar_norm"): + return False + draft_model._dflash_fused_ar_norm = False + model = getattr(draft_model, "model", None) + layers = list(getattr(model, "layers", []) or []) + if ( + not enabled + or self.mapping.tp_size <= 1 + or not layers + or getattr(draft_model, "has_block_conv", False) + ): + return False + norms = [model.norm] + [ + n for layer in layers for n in (layer.input_layernorm, layer.post_attention_layernorm) + ] + linears = [m for layer in layers for m in (layer.self_attn.o_proj, layer.mlp.down_proj)] + if any(getattr(n, "use_gemma", False) or getattr(n, "weight", None) is None for n in norms): + return False + if any(getattr(m, "all_reduce", None) is None for m in linears): + return False + draft_model._dflash_fused_ar_norm = True + return True + + def enable_fused_ar_norm_op(self, draft_model, op) -> bool: + """Install a target's one-launch all-reduce + add + RMSNorm for the fused AR + norm (None, the default, keeps + each Linear's own fused all-reduce). Only while the fused AR + norm is on (enable_fused_ar_norm). + + ``op(partial, residual, norm_weight, eps)`` must return (normed, residual + reduced) under the fused AR + norm's + contract (rule 1b: the reduced sum is not rounded before the add), identical on every TP rank, or None for + operands it does not take (decided by shapes / layouts only, so every rank agrees), and then the Linear's + all-reduce runs.""" + if not hasattr(draft_model, "_dflash_fused_ar_norm_op"): + return False + draft_model._dflash_fused_ar_norm_op = ( + op if getattr(draft_model, "_dflash_fused_ar_norm", False) else None + ) + return draft_model._dflash_fused_ar_norm_op is not None + + def enable_fc_tp_split(self, draft_model, enabled: bool = True) -> bool: + """Split the drafter's replicated fc row-parallel over the TP group: each rank reads 1/TP of its weight, and one + fp32 all-reduce sums the partials. Off by default; collective (every rank, outside capture). Returns whether it + is on: it needs a TP group, a plain fc and an input width the TP size divides.""" + fc = getattr(draft_model, "fc", None) + if fc is None: + return False + draft_model._dflash_fc_split = None + tp, rank = self.mapping.tp_size, self.mapping.tp_rank + if not enabled or tp <= 1 or getattr(fc, "bias", None) is not None: + return False + k = fc.weight.shape[1] + if k % tp: + return False + from tensorrt_llm.functional import AllReduceStrategy + + from ..distributed import AllReduce + + ks = k // tp + shard = fc.weight.detach()[:, rank * ks : (rank + 1) * ks].contiguous() + # MNNVL by name, with the fp32 dtype it needs to set up: its one-shot / two-shot kernels sum the partials in + # rank order, so every rank gets the same bits. AUTO would leave the choice to the autotuned all-reduce, + # whose tactic is picked per process start. + draft_model._dflash_fc_split = ( + rank * ks, + (rank + 1) * ks, + shard, + AllReduce(mapping=self.mapping, strategy=AllReduceStrategy.MNNVL, dtype=torch.float32), + ) + return True + def _get_ctx_paged_append(self) -> Callable[..., None]: """flashinfer paged append, shared by the TRTLLM and FA4 backends.""" if self._ctx_paged_append is None: self._ctx_paged_append = get_dflash_paged_append() return self._ctx_paged_append + def engage_disagg_gen_init_window(self) -> None: + """Seed disagg generation requests' drafter slots (see ``seed_disagg_draft_ctx``).""" + if self._disagg_draft_ctx_seeds is None: + self._disagg_draft_ctx_seeds = {} + + def engage_disagg_draft_kv(self) -> None: + """Track the drafter context for the disagg draft-window transfer (cache transceiver only). + + A context worker then records an event after each prefill store, which the send waits on + before publishing; a generation worker seeds each received request's context length over + the empty window its gen-init sizing seeded. + """ + self._disagg_draft_kv_engaged = True + self._disagg_draft_ctx_ready = {} + self.engage_disagg_gen_init_window() + + def _check_disagg_draft_ctx_pool_bound(self) -> None: + """The transferred window lives in the draft KV pool; a private arena is out of reach.""" + if self._ctx_block_tables is None: + raise RuntimeError( + "DFlash: the disagg drafter context-window transfer needs the drafter context in " + "the managed draft KV pool, but it is on the private arena (FA4 or a pool " + "shape mismatch); set TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV=1 to serve without it" + ) + + @staticmethod + def _bound_disagg_draft_ctx_map(entries: dict) -> None: + while len(entries) > _DISAGG_DRAFT_CTX_MAP_CAP: + entries.pop(next(iter(entries))) + + def _record_disagg_draft_ctx_ready(self, spec_metadata, num_contexts: int) -> None: + """Context side of the disagg draft window: an event after this batch's drafter store. + + Recorded on the current (execution) stream right after ``_store_prefill_context``, and + never while capturing a graph (context batches run eagerly). Each context request keeps + its latest store's event with the slot's host ``(ctx_len, ctx_base)``, read here because + the slot may be recycled before the send. No host synchronization. + """ + if not self._disagg_draft_kv_engaged or num_contexts <= 0: + return + if torch.cuda.is_current_stream_capturing(): + return + request_ids = spec_metadata.request_ids + if not request_ids: + return + self._check_disagg_draft_ctx_pool_bound() + event = torch.cuda.Event() + event.record() + ready = self._disagg_draft_ctx_ready + for rid in request_ids[:num_contexts]: + slot = self._req_to_slot.get(rid) + if slot is None: + ctx_len, ctx_base = 0, 0 + else: + ctx_len, ctx_base = self._ctx_len_host[slot], self._ctx_base_host[slot] + # Re-inserted so the dict stays ordered by the latest store. + ready.pop(rid, None) + ready[rid] = (event, int(ctx_len), int(ctx_base)) + self._bound_disagg_draft_ctx_map(ready) + + def take_disagg_draft_ctx_ready( + self, request_id: int + ) -> Optional[Tuple[torch.cuda.Event, int, int]]: + """Hand over a context request's latest drafter-store event with its (ctx_len, ctx_base).""" + ready = self._disagg_draft_ctx_ready + return None if ready is None else ready.pop(request_id, None) + + def seed_disagg_draft_ctx( + self, + request_id: int, + ctx_len: int, + ctx_base: int, + prompt_len: int, + on_seed: Optional[Callable[[], None]] = None, + ) -> None: + """Generation side: the drafter state a disagg generation request's slot starts from. + + Applied by ``DFlashSpecMetadata.prepare`` when the request first holds a real slot, which + then runs ``on_seed`` (test only: the G1d dump of the window as the first step reads it). + """ + seeds = self._disagg_draft_ctx_seeds + if seeds is None: + return + seeds.pop(request_id, None) + seeds[request_id] = (int(ctx_len), int(ctx_base), int(prompt_len)) + self._bound_disagg_draft_ctx_map(seeds) + if on_seed is not None: + if self._disagg_draft_ctx_seed_hooks is None: + self._disagg_draft_ctx_seed_hooks = {} + self._disagg_draft_ctx_seed_hooks[request_id] = on_seed + self._bound_disagg_draft_ctx_map(self._disagg_draft_ctx_seed_hooks) + + def seed_disagg_empty_window( + self, request_id: int, prompt_len: int, tokens_per_block: int + ) -> None: + """Start a disagg generation request's drafter from an empty window at the prompt end. + + Its windowed draft mirror holds only the pages behind the prompt end (the low pages are + released), so a drafter numbering from position 0 would address released pages. Basing + the slot past every prompt page (``dflash_rebased_ctx_view``) reads nothing and appends + into pages the mirror holds. + """ + prompt_len = int(prompt_len) + self.seed_disagg_draft_ctx( + request_id, prompt_len, -(-prompt_len // int(tokens_per_block)), prompt_len + ) + def _static_arange(self, start: int, end: int) -> torch.Tensor: """torch.arange(start, end) (long, cuda). With lean glue on, cached per (start, end) outside CUDA-graph capture (read-only uses only); never cached during capture, so no graph-pool tensor outlives its graph.""" @@ -1766,9 +2083,7 @@ def _store_prefill_context( # One tolist() for the whole loop: the lengths themselves come from # _ctx_len_host, so the loop body below stays sync-free. ctx_alloc = ( - (self._ctx_block_counts[:num_contexts] * self._ctx_page_size).tolist() - if self._ctx_block_tables is not None - else None + self._host_ctx_alloc(num_contexts) if self._ctx_block_tables is not None else None ) for i in range(num_contexts): req_id = spec_metadata.request_ids[i] @@ -1991,6 +2306,7 @@ def _forward_impl( self._store_prefill_context( draft_model, spec_metadata, attn_metadata, position_ids, total_target_tokens ) + self._record_disagg_draft_ctx_ready(spec_metadata, num_contexts) # Rebuild batch_to_slot after prefill assigns new slots if self._ctx_buf_inited and spec_metadata.request_ids: num_seqs = len(spec_metadata.request_ids) @@ -2125,7 +2441,9 @@ def _forward_impl( # Context requests are not drafted by the block worker (zero placeholder # token); fill their draft-prob slot rows with a one-hot placeholder so # they are a legal distribution when they become gen requests next iter. - gen_vocab = vocab_size if num_gens > 0 else None + gen_vocab = ( + _context_onehot_vocab(spec_metadata, num_gens, vocab_size) if num_gens > 0 else None + ) self.write_context_onehot_draft_probs(spec_metadata, num_contexts, num_gens, K, gen_vocab) if num_contexts > 0 and num_gens > 0: diff --git a/tensorrt_llm/_torch/speculative/dspark.py b/tensorrt_llm/_torch/speculative/dspark.py index f1273926d683..dc1b027b1dce 100644 --- a/tensorrt_llm/_torch/speculative/dspark.py +++ b/tensorrt_llm/_torch/speculative/dspark.py @@ -177,6 +177,10 @@ class DSparkSpecMetadata(SpecMetadata): captured_hidden_states: Optional[torch.Tensor] = None def __post_init__(self): + # Preserve the initial slot capacity across CUDA graph copies, whose + # max_num_requests is narrowed to the captured graph bucket. + self.num_seq_slots = self.num_seq_slots or self.max_num_requests + self.batch_indices_cuda = torch.empty( [self.max_num_requests], dtype=torch.int, diff --git a/tensorrt_llm/_torch/speculative/interface.py b/tensorrt_llm/_torch/speculative/interface.py index d0bfd76c28cc..19b81228f26f 100644 --- a/tensorrt_llm/_torch/speculative/interface.py +++ b/tensorrt_llm/_torch/speculative/interface.py @@ -34,6 +34,7 @@ from ..flashinfer_utils import IS_FLASHINFER_AVAILABLE from ..pyexecutor import step_inputs from ..pyexecutor.resource_manager import ResourceManagerType +from ..pyexecutor.runtime_hooks import hook_work if TYPE_CHECKING: from ..pyexecutor.guided_decoder import CapturableGuidedDecoder @@ -58,6 +59,27 @@ Optional[tuple[torch.Tensor, torch.Tensor]]] _CHAIN_SPEC_OVERRIDE: Optional[_ChainSpecOverride] = None +# A target's slot-indexed form of the chain speculative sampling (rejection) op: +# fn(draft_buf, slot_ids, num_draft, draft_token_ids, target_probs, seed, offset) +# reads request r's draft distributions in place at draft_buf[slot_ids[r], +# :num_draft, :vocab] (the spec metadata's slot-indexed buffer) instead of a +# gathered copy, and returns the same (accepted_tokens, num_tokens), or None +# (the gathered copy and rejection_sampling_one_model then run). None (the +# default) keeps the gather. +_ChainSpecSlotOverride = Callable[[ + torch.Tensor, torch.Tensor, int, torch.Tensor, torch.Tensor, torch. + Tensor, torch.Tensor +], Optional[tuple[torch.Tensor, torch.Tensor]]] +_CHAIN_SPEC_SLOT_OVERRIDE: Optional[_ChainSpecSlotOverride] = None + + +def set_chain_speculative_sampling_slot_override( + fn: Optional[_ChainSpecSlotOverride]) -> None: + """Install (or, with None, remove) the slot-indexed chain speculative + sampling override (see _ChainSpecSlotOverride).""" + global _CHAIN_SPEC_SLOT_OVERRIDE + _CHAIN_SPEC_SLOT_OVERRIDE = fn + def set_chain_speculative_sampling_override( fn: Optional[_ChainSpecOverride]) -> None: @@ -507,16 +529,21 @@ def from_string(name: Optional[str]) -> "SpeculativeDecodingMode": ) # The tail bookkeeping items that already logged their first use. _TAIL_BOOKKEEP_LOGGED: set = set() +_TAIL_BOOKKEEP_USES: dict = {} def tail_bookkeep_log_once(item: str, message: str) -> None: - """Log a tail bookkeeping item's engagement line once per process. + """Log a tail bookkeeping item's engagement line once per process, and its use count at every power of two. - Host only, so a captured step logs it while it is captured. + Host only, so a captured step logs it while it is captured: the count is the item's host executions (eager steps + and graph captures), not graph replays. """ + uses = _TAIL_BOOKKEEP_USES[item] = _TAIL_BOOKKEEP_USES.get(item, 0) + 1 if item not in _TAIL_BOOKKEEP_LOGGED: _TAIL_BOOKKEEP_LOGGED.add(item) logger.info(f"[M3 host] tail bookkeeping {item}: {message}") + elif uses & (uses - 1) == 0: + logger.info(f"[M3 host] tail bookkeeping {item}: used {uses} times") def tail_bookkeep_draft_len(spec_metadata, num_contexts: int, @@ -1044,6 +1071,8 @@ def _upload(dst: torch.Tensor, values: list[int]) -> None: if self.skip_greedy_rng_upload and self.is_all_greedy_sample: # The offset window counters above still advanced. return + if self.pack_rng_uploads: + hook_work("PHILOX_PACK") _upload(self.request_seeds, request_seeds) _upload(self.request_offsets, request_offsets) _upload(self.seeds, flat_seeds) @@ -1778,6 +1807,12 @@ def _padded_gen_draft_tokens(spec_metadata: SpecMetadata, num_gens: int, ).to(torch.int32) +# Context one-hot draft-prob rows are written with index_fill_ (non-blocking) instead +# of an advanced-index put that synchronizes the host with the GPU. Kill switch. +_ONEHOT_INDEX_FILL = os.environ.get("TRTLLM_M31_DISABLE_ONEHOT_INDEX_FILL", + "0") != "1" + + class SpecWorkerBase(nn.Module, ABC): """ Base class for speculative decoding workers. @@ -1787,6 +1822,9 @@ class SpecWorkerBase(nn.Module, ABC): # Per-worker target-installed collective; None retains the stock gather. _draft_logits_gather_op: Optional[Callable[[torch.Tensor], Optional[torch.Tensor]]] = None + # Keep a block sampler's int32 tokens as int32 when no d2t map applies + # (skips an int32 -> int64 -> int32 round trip); False keeps the widening. + _block_tokens_keep_int32: bool = False def __init__(self, use_separate_draft_kv_cache: bool = False): super().__init__() @@ -2312,11 +2350,23 @@ def _accept_draft_tokens(self, logits, draft_tokens, num_contexts, # step indexed by py_seq_slot. gen_slot_ids = spec_metadata.batch_slot_ids[ num_contexts:batch_size] - draft_probs = spec_metadata.draft_probs[ - gen_slot_ids, :draft_len, :stored_vocab] + if _CHAIN_SPEC_SLOT_OVERRIDE is not None: + # Gathered only if the slot-indexed op cannot take them. + draft_probs = None + slot_rows = (spec_metadata.draft_probs, gen_slot_ids, + draft_len, stored_vocab) + else: + draft_probs = spec_metadata.draft_probs[ + gen_slot_ids, :draft_len, :stored_vocab] + slot_rows = None accepted = self._sample_and_accept_draft_tokens_rejection( - logits, draft_tokens, draft_probs, num_contexts, batch_size, - spec_metadata) + logits, + draft_tokens, + draft_probs, + num_contexts, + batch_size, + spec_metadata, + draft_probs_slot_rows=slot_rows) return self._commit_occurrence_counts(accepted, batch_size, spec_metadata) accepted = self._sample_and_accept_draft_tokens_base( @@ -2722,10 +2772,11 @@ def _sample_and_accept_draft_tokens_rejection( self, logits: torch.Tensor, draft_tokens: torch.Tensor, - draft_probs: torch.Tensor, + draft_probs: Optional[torch.Tensor], num_contexts: int, batch_size: int, spec_metadata, + draft_probs_slot_rows: Optional[tuple] = None, ): """ Rejection-sampling acceptance for one-model speculative decoding. @@ -2810,51 +2861,69 @@ def _sample_and_accept_draft_tokens_rejection( target_probs = target_probs_flat.reshape( num_gens, runtime_draft_len + 1, vocab_size) - draft_vocab_size = draft_probs.shape[-1] - assert draft_probs.shape[0] == num_gens, ( - f"draft_probs batch mismatch: {draft_probs.shape[0]} != " - f"num_gens={num_gens}") - assert draft_probs.shape[1] == runtime_draft_len, ( - f"draft_probs draft length mismatch: {draft_probs.shape[1]} != " - f"{runtime_draft_len}") - d2t = self._d2t.data if self._d2t is not None else None - if draft_vocab_size != vocab_size: - if spec_metadata.full_draft_probs is not None: - # Slice to runtime_draft_len so the max_draft_len buffer - # never passes stale extra rows to the rejection kernel. - full_draft_probs = spec_metadata.full_draft_probs[: - num_gens, : - runtime_draft_len] - else: - # Buffer not pre-allocated (e.g. rejection off at prepare()): - # fall back to a per-iter allocation. - full_draft_probs = torch.zeros( - (num_gens, runtime_draft_len, vocab_size), - dtype=torch.float32, - device=device) - if d2t is not None: - assert d2t.numel() == draft_vocab_size, ( - f"d2t size mismatch: {d2t.numel()} != {draft_vocab_size}" - ) - # d2t is model-static; compute target_indices once and - # cache on spec_metadata to skip the arange + add + mod - # kernel sequence on every iter. - target_indices = spec_metadata.d2t_target_indices - if target_indices is None: - source_indices = torch.arange(draft_vocab_size, - device=device, - dtype=torch.long) - target_indices = (source_indices + - d2t.to(device=device)) % vocab_size - spec_metadata.d2t_target_indices = target_indices - full_draft_probs[:, :runtime_draft_len, - target_indices] = draft_probs + # Slot-indexed draft probs (draft_probs None): kept in place for the + # slot-indexed op when no expansion / padding needs a copy anyway, + # else gathered here exactly as the caller would have. + slot_rows = None + if draft_probs is None: + buf, slot_ids, slot_draft_len, slot_vocab = draft_probs_slot_rows + if (slot_vocab == vocab_size and not is_ragged + and self._d2t is None + and slot_draft_len == runtime_draft_len + and slot_ids.shape[0] == num_gens): + slot_rows = (buf, slot_ids) else: - assert draft_vocab_size < vocab_size - full_draft_probs[:, :runtime_draft_len, : - draft_vocab_size] = (draft_probs) + draft_probs = buf[slot_ids, :slot_draft_len, :slot_vocab] + if slot_rows is not None: + # Kept in place: no tensor stands for these rows here (they are + # read by the slot call, or gathered below if it declines). + full_draft_probs = None else: - full_draft_probs = draft_probs + draft_vocab_size = draft_probs.shape[-1] + assert draft_probs.shape[0] == num_gens, ( + f"draft_probs batch mismatch: {draft_probs.shape[0]} != " + f"num_gens={num_gens}") + assert draft_probs.shape[1] == runtime_draft_len, ( + f"draft_probs draft length mismatch: {draft_probs.shape[1]} != " + f"{runtime_draft_len}") + d2t = self._d2t.data if self._d2t is not None else None + if draft_vocab_size != vocab_size: + if spec_metadata.full_draft_probs is not None: + # Slice to runtime_draft_len so the max_draft_len buffer + # never passes stale extra rows to the rejection kernel. + full_draft_probs = spec_metadata.full_draft_probs[: + num_gens, : + runtime_draft_len] + else: + # Buffer not pre-allocated (e.g. rejection off at prepare()): + # fall back to a per-iter allocation. + full_draft_probs = torch.zeros( + (num_gens, runtime_draft_len, vocab_size), + dtype=torch.float32, + device=device) + if d2t is not None: + assert d2t.numel() == draft_vocab_size, ( + f"d2t size mismatch: {d2t.numel()} != {draft_vocab_size}" + ) + # d2t is model-static; compute target_indices once and + # cache on spec_metadata to skip the arange + add + mod + # kernel sequence on every iter. + target_indices = spec_metadata.d2t_target_indices + if target_indices is None: + source_indices = torch.arange(draft_vocab_size, + device=device, + dtype=torch.long) + target_indices = (source_indices + d2t.to( + device=device)) % vocab_size + spec_metadata.d2t_target_indices = target_indices + full_draft_probs[:, :runtime_draft_len, + target_indices] = draft_probs + else: + assert draft_vocab_size < vocab_size + full_draft_probs[:, :runtime_draft_len, : + draft_vocab_size] = (draft_probs) + else: + full_draft_probs = draft_probs if is_ragged: # Missing draft IDs are zero, so their proposal must be a @@ -2878,14 +2947,27 @@ def _sample_and_accept_draft_tokens_rejection( batch_size, slot=spec_metadata.rng_accept_slot) - gen_accepted, gen_num_accepted = rejection_sampling_one_model( - draft_probs=full_draft_probs, - draft_token_ids=full_draft_tokens, - target_probs=target_probs, - deterministic=True, - seed=seed, - offset=offset, - ) + slot_out = None + if slot_rows is not None: + slot_out = _CHAIN_SPEC_SLOT_OVERRIDE(slot_rows[0], slot_rows[1], + runtime_draft_len, + full_draft_tokens, + target_probs, seed, offset) + if slot_out is None: + # Declined: the gathered copy, as without the override. + full_draft_probs = slot_rows[0][ + slot_rows[1], :runtime_draft_len, :vocab_size] + if slot_out is not None: + gen_accepted, gen_num_accepted = slot_out + else: + gen_accepted, gen_num_accepted = rejection_sampling_one_model( + draft_probs=full_draft_probs, + draft_token_ids=full_draft_tokens, + target_probs=target_probs, + deterministic=True, + seed=seed, + offset=offset, + ) if self.force_num_accepted_tokens != 0.0: # Fill gen_accepted positions 1..runtime_draft_len with all draft tokens @@ -3097,28 +3179,21 @@ def greedy_sample_draft_with_tp_gather(self, # _draft_logits_are_sharded). Plain argmax; caller applies d2t. return torch.argmax(logits, dim=-1).type(torch.int32) - def advanced_sample_draft_block(self, gen_logits: torch.Tensor, - spec_metadata: "SpecMetadata", - num_contexts: int, batch_size: int): - """Block counterpart of ``advanced_sample_draft`` for gen-only workers - (PARD, DFLASH): produces all ``K`` draft positions per gen request in one - forward from ``[num_gens, K, vocab]`` logits. - - With rejection enabled, samples via - ``sampling_batch_spec_dec_one_model_for_rejection`` and scatters the K - proposal rows into ``draft_probs[gen_slot_ids, 0:K, :]``; otherwise uses - ``sample_from_logits_op`` (tokens only). Only called for a - non-greedy batch (the all-greedy path is handled by the caller). Returns - ``[num_gens, K]`` int32 tokens in draft-vocab space (the caller applies - d2t); stored probs likewise stay in draft-vocab space. - """ - num_gens, K, vocab = gen_logits.shape - if num_gens == 0: - return torch.empty((0, K), - dtype=torch.int32, - device=gen_logits.device) - - if (num_gens == 1 and tail_bookkeep_draft_len( + @staticmethod + def _store_block_draft_probs(draft_probs, slot_ids, probs): + """Store each request's block proposal at its stable slot.""" + _, K, vocab = probs.shape + draft_probs[slot_ids, :K, :vocab] = probs + + def _block_sampling_params(self, spec_metadata: "SpecMetadata", + num_contexts: int, batch_size: int, K: int): + """Per-row sampling parameters of a block sampler: each gen request's + temperature / top_k / top_p / min_p repeated ``K`` times to line up with + the flattened ``[num_gens*K, vocab]`` logits, and its Philox (seed, + offset) for the step's ``K`` draft slots. Returns ``(temps, top_ks, + top_ps, min_ps, seed, offset)``, each ``[num_gens*K]``. Model workers + may override it with an equivalent implementation.""" + if (batch_size - num_contexts == 1 and tail_bookkeep_draft_len( spec_metadata, num_contexts, batch_size) == K): # Tail bookkeeping B1: the one request's per-token rows already # hold its values K times (tail_bookkeep_draft_len), so read-only @@ -3144,18 +3219,44 @@ def advanced_sample_draft_block(self, gen_logits: torch.Tensor, min_ps = spec_metadata.request_min_ps[ num_contexts:batch_size].repeat_interleave(K) - flat_logits = gen_logits.reshape(num_gens * K, vocab) - # A block sampler emits all K draft positions in one launch; they take - # the step's K draft slots. - assert K <= spec_metadata.max_draft_len, ( - f"{K} draft rows per request exceed max_draft_len=" - f"{spec_metadata.max_draft_len}") seed, offset = self._rng_state_per_request( spec_metadata, num_contexts, batch_size, repeat=K, slot=spec_metadata.rng_draft_slot(0)) + return temps, top_ks, top_ps, min_ps, seed, offset + + def advanced_sample_draft_block(self, gen_logits: torch.Tensor, + spec_metadata: "SpecMetadata", + num_contexts: int, batch_size: int): + """Block counterpart of ``advanced_sample_draft`` for gen-only workers + (PARD, DFLASH): produces all ``K`` draft positions per gen request in one + forward from ``[num_gens, K, vocab]`` logits. + + With rejection enabled, samples via + ``sampling_batch_spec_dec_one_model_for_rejection`` and scatters the K + proposal rows into ``draft_probs[gen_slot_ids, 0:K, :]``; otherwise uses + ``sample_from_logits_op`` (tokens only). Only called for a + non-greedy batch (the all-greedy path is handled by the caller). Returns + ``[num_gens, K]`` int32 tokens in draft-vocab space (the caller applies + d2t); stored probs likewise stay in draft-vocab space. + """ + num_gens, K, vocab = gen_logits.shape + if num_gens == 0: + return torch.empty((0, K), + dtype=torch.int32, + device=gen_logits.device) + + flat_logits = gen_logits.reshape(num_gens * K, vocab) + # A block sampler emits all K draft positions in one launch; they take + # the step's K draft slots. + assert K <= spec_metadata.max_draft_len, ( + f"{K} draft rows per request exceed max_draft_len=" + f"{spec_metadata.max_draft_len}") + temps, top_ks, top_ps, min_ps, seed, offset = ( + self._block_sampling_params(spec_metadata, num_contexts, batch_size, + K)) if getattr(spec_metadata, "use_rejection_sampling", False): flat_logits = self._zero_padding_rows(flat_logits, spec_metadata, @@ -3177,7 +3278,8 @@ def advanced_sample_draft_block(self, gen_logits: torch.Tensor, gen_slot_ids = spec_metadata.batch_slot_ids[ num_contexts:batch_size] probs = flat_probs.reshape(num_gens, K, vocab) - spec_metadata.draft_probs[gen_slot_ids, :K, :vocab] = probs + self._store_block_draft_probs(spec_metadata.draft_probs, + gen_slot_ids, probs) spec_metadata.draft_probs_last_dim = vocab else: flat_tokens = self._sample_from_logits( @@ -3222,6 +3324,15 @@ def write_context_onehot_draft_probs(self, else: onehot_vocab = spec_metadata.draft_probs_vocab_size spec_metadata.draft_probs_last_dim = onehot_vocab + if _ONEHOT_INDEX_FILL: + # index_fill_ on the [:, :draft_len, :onehot_vocab] view writes exactly + # the same elements; the advanced-index put below blocks the host until + # the GPU drains (16-67 ms per mixed step), exposing every later launch. + spec_metadata.draft_probs[:, :draft_len, :onehot_vocab].index_fill_( + 0, ctx_slot_ids, 0.0) + spec_metadata.draft_probs[:, :draft_len, + 0].index_fill_(0, ctx_slot_ids, 1.0) + return spec_metadata.draft_probs[ctx_slot_ids, :draft_len, :onehot_vocab] = 0.0 spec_metadata.draft_probs[ctx_slot_ids, :draft_len, 0] = 1.0 @@ -3295,7 +3406,7 @@ def sample_draft_tokens(self, tail_bookkeep_log_once( "S1", "int32 draft tokens without the int64 round " "trip") - else: + elif not (self._block_tokens_keep_int32 and self._d2t is None): tokens = tokens.long() else: tokens = self.advanced_sample_draft(logits, diff --git a/tensorrt_llm/_torch/speculative/spec_sampler_base.py b/tensorrt_llm/_torch/speculative/spec_sampler_base.py index 1c81e24be184..d46563e7515f 100644 --- a/tensorrt_llm/_torch/speculative/spec_sampler_base.py +++ b/tensorrt_llm/_torch/speculative/spec_sampler_base.py @@ -31,7 +31,7 @@ from ...sampling_params import SamplingParams from ..pyexecutor.llm_request import LlmRequest, LlmRequestState, get_draft_token_length from ..pyexecutor.resource_manager import BaseResourceManager -from ..pyexecutor.runtime_hooks import SpecStepFold +from ..pyexecutor.runtime_hooks import SpecStepFold, hook_work from ..pyexecutor.sampler import ( DEFAULT_BEAM_IDX, AsyncWorkerMixin, @@ -74,6 +74,11 @@ "Drop top_p_decay / top_p_min from the request, or disable " "speculative decoding." ) +UNSUPPORTED_LOGITS_POST_PROCESSOR_MSG = ( + "logits_processor (a logits post-processor callback) is not supported with one-model " + "speculative decoding. Drop logits_processor from the request, or disable speculative " + "decoding." +) UNSUPPORTED_MIN_P_MSG = ( "min_p requires 'advanced_sampling_mode: full' in the speculative decoding " "config when using one-model speculative decoding. Restore that default, " @@ -114,6 +119,10 @@ def one_model_sampling_rejection_reason( return UNSUPPORTED_NO_REPEAT_NGRAM_MSG if sampling_params.embedding_bias is not None: return UNSUPPORTED_EMBEDDING_BIAS_MSG + # A per-request callback becomes py_logits_post_processors; the one-model path never + # calls it (a fused verify may not even materialize the full logits). + if sampling_params.logits_processor: + return UNSUPPORTED_LOGITS_POST_PROCESSOR_MSG # The same predicate top_p_decay_active() delegates to, so "active" cannot # mean one thing here and another in the sampler. if SamplingParams.params_imply_top_p_decay_active(sampling_params.top_p_decay): @@ -240,8 +249,8 @@ def _validate_unsupported_logits_processors(request: LlmRequest) -> None: """Reject logits-side sampling features the one-model path cannot apply. TorchSampler implements min_length, bad_words, no_repeat_ngram_size, - embedding_bias and top_p_decay by editing the target logits before - sampling. The one-model path has no logits-editing hook, so each is + embedding_bias, top_p_decay and logits post-processor callbacks by editing + the target logits before sampling. The one-model path has no logits-editing hook, so each is rejected here instead of being silently dropped. Each check gates on a non-neutral value rather than on presence, since a @@ -260,6 +269,11 @@ def _validate_unsupported_logits_processors(request: LlmRequest) -> None: raise ValueError(UNSUPPORTED_NO_REPEAT_NGRAM_MSG) if getattr(request, "py_embedding_bias", None) is not None: raise ValueError(UNSUPPORTED_EMBEDDING_BIAS_MSG) + # The one-model sampler never calls a request's logits post-processors, and a fused + # verify (e.g. the MiniMax-M3 shard verify) does not even gather the full logits, so + # the callback would be skipped silently. + if getattr(request, "py_logits_post_processors", None): + raise ValueError(UNSUPPORTED_LOGITS_POST_PROCESSOR_MSG) # Reuse the handler's own predicate so "active" cannot drift between paths. if top_p_decay_active(_request_get_sampling_params(request)): raise ValueError(UNSUPPORTED_TOP_P_DECAY_MSG) @@ -379,6 +393,7 @@ def _copy_outputs_to_host( ) return host, None verify_lens = self._copy_to_host(o_verify_lens) if o_verify_lens is not None else None + hook_work("HOST_D2H_SIDE") with self._make_side_stream_copier() as copier: host = SampleStateTensorsSpec( new_tokens=copier.stage_copy_to_host(self.store.new_tokens), @@ -488,6 +503,7 @@ def _store_outputs( ) -> None: """Store one step's worker outputs into the per-slot store.""" if self.static_slot_ids: + hook_work("SMALL_H2D") slots = self._device_slot_ids(seq_slots) else: slots = torch.as_tensor(seq_slots, dtype=torch.long) diff --git a/tensorrt_llm/disaggregated_params.py b/tensorrt_llm/disaggregated_params.py index f65a3694d87b..eff5febf9c1e 100644 --- a/tensorrt_llm/disaggregated_params.py +++ b/tensorrt_llm/disaggregated_params.py @@ -48,6 +48,9 @@ class DisaggregatedParams: usage accounting on the generation server. remote_tail_start (int): Prompt-token boundary transferred by a remote-tail context worker. The generation worker replays the suffix starting here. + draft_ctx_state (List[int]): Context-first only: the standalone drafter's context-window + decision ``[ctx_len, ctx_base, ok]`` for the generation request (not ``opaque_state``, + which carries the C++ DataTransceiverState). multimodal_embedding_handles (List[Dict[str, Any]]): The resulting multimodal embedding handles from ViT. multimodal_hashes (List[List[int]]): The multimodal hashes of each multimodal item in the request. multimodal_positions (List[int]): Prompt-token offset of each multimodal item. @@ -69,6 +72,7 @@ class DisaggregatedParams: schedule_style: Optional[DisaggScheduleStyle] = None ctx_usage: Optional[Dict[str, Any]] = None remote_tail_start: Optional[int] = None + draft_ctx_state: Optional[List[int]] = None multimodal_positions: Optional[List[int]] = None multimodal_lengths: Optional[List[int]] = None diff --git a/tensorrt_llm/executor/result.py b/tensorrt_llm/executor/result.py index 5cceccd9ef14..0d7cae328baa 100644 --- a/tensorrt_llm/executor/result.py +++ b/tensorrt_llm/executor/result.py @@ -683,6 +683,8 @@ def _handle_response(self, ctx_dp_rank=context_phase_params.ctx_dp_rank, ctx_info_endpoint=context_phase_params.disagg_info_endpoint, remote_tail_start=remote_tail_start, + draft_ctx_state=getattr(response_result, "draft_ctx_state", + None), multimodal_embedding_handles=None, ) diff --git a/tensorrt_llm/llmapi/llm_args.py b/tensorrt_llm/llmapi/llm_args.py index 01f76b2886f9..41f3c3976f73 100644 --- a/tensorrt_llm/llmapi/llm_args.py +++ b/tensorrt_llm/llmapi/llm_args.py @@ -1094,12 +1094,17 @@ class MiniMaxM3SparseAttentionConfig(BaseSparseAttentionConfig): default=True, description="If True, skip the index V branch (M3 checkpoint default).", ) - indexer_kv_dtype: Literal["bf16", "fp8"] = Field( + indexer_kv_dtype: Literal["bf16", "fp8", "nvfp4"] = Field( default="bf16", description= "Storage and score-compute dtype for normalized index Q/K. 'fp8' uses " "unscaled E4M3 values with FP32 score accumulation and is supported only " - "by the MSA implementation.", + "by the MSA implementation. 'nvfp4' stores index-K as packed NVFP4 pages " + "(E2M1 with E4M3 16-element scales, no global scale; 72 bytes per token " + "instead of 128) with E4M3 index-Q: decode scores the pages directly, " + "prefill dequantizes them to E4M3 first. It is not exact against 'fp8' " + "(it changes which blocks are selected). MSA only; " + "TRTLLM_M31_DISABLE_AGG_NVFP4_INDEXK=1 falls back to 'fp8'.", status="prototype", ) fuse_qkv_index_projection: bool = Field( @@ -1109,8 +1114,8 @@ class MiniMaxM3SparseAttentionConfig(BaseSparseAttentionConfig): "is sharded with the KV heads and index-K is replicated. MSA batches " "also use a horizontal norm/RoPE/cache-insertion producer for prefill, " "mixed, and CUDA-graph decode execution. The MiniMax-M3-specific path " - "requires the MSA implementation, indexer_kv_dtype='fp8', and an FP8 " - "or NVFP4 main KV cache.", + "requires the MSA implementation, indexer_kv_dtype='fp8' or 'nvfp4', " + "and an FP8 or NVFP4 main KV cache.", status="prototype", ) num_attention_heads: Optional[int] = Field( @@ -1137,25 +1142,43 @@ class MiniMaxM3SparseAttentionConfig(BaseSparseAttentionConfig): @model_validator(mode="after") def _validate_msa_configuration(self): + if self.indexer_kv_dtype == "nvfp4" and os.environ.get( + "TRTLLM_M31_DISABLE_AGG_NVFP4_INDEXK", + "0").strip().lower() in ("1", "true", "on", "yes"): + # Kill switch of the NVFP4 index-K opt-in: every component reads + # the configured dtype, so falling back here keeps them consistent. + logger.warning( + "[M3.1] AGG_NVFP4_INDEXK killed (TRTLLM_M31_DISABLE_AGG_NVFP4_INDEXK=1): " + "indexer_kv_dtype 'nvfp4' -> 'fp8'") + self.indexer_kv_dtype = "fp8" if self.implementation == "msa" and self.sparse_block_size != 128: raise ValueError( "MiniMax-M3 'msa' implementation requires sparse_block_size == " f"128, got {self.sparse_block_size}.") - if self.indexer_kv_dtype == "fp8" and self.implementation != "msa": + if self.indexer_kv_dtype in ("fp8", + "nvfp4") and self.implementation != "msa": + raise ValueError( + f"MiniMax-M3 indexer_kv_dtype={self.indexer_kv_dtype!r} currently " + "requires the 'msa' implementation.") + if self.indexer_kv_dtype in ( + "fp8", "nvfp4") and not self.sparse_disable_index_value: + raise ValueError( + f"MiniMax-M3 indexer_kv_dtype={self.indexer_kv_dtype!r} requires " + "sparse_disable_index_value=True.") + if self.indexer_kv_dtype == "nvfp4" and not self.fuse_qkv_index_projection: raise ValueError( - "MiniMax-M3 indexer_kv_dtype='fp8' currently requires the " - "'msa' implementation.") - if self.indexer_kv_dtype == "fp8" and not self.sparse_disable_index_value: - raise ValueError("MiniMax-M3 indexer_kv_dtype='fp8' requires " - "sparse_disable_index_value=True.") + "MiniMax-M3 indexer_kv_dtype='nvfp4' requires " + "fuse_qkv_index_projection=True (its index-K writer follows the " + "fused producer).") if self.fuse_qkv_index_projection and self.implementation != "msa": raise ValueError( "MiniMax-M3 fuse_qkv_index_projection=True currently requires " "the 'msa' implementation.") - if self.fuse_qkv_index_projection and self.indexer_kv_dtype != "fp8": + if self.fuse_qkv_index_projection and self.indexer_kv_dtype not in ( + "fp8", "nvfp4"): raise ValueError( "MiniMax-M3 fuse_qkv_index_projection=True currently requires " - "indexer_kv_dtype='fp8'.") + "indexer_kv_dtype='fp8' or 'nvfp4'.") return self def supports_backend(self, backend: str) -> bool: diff --git a/tensorrt_llm/serve/openai_protocol.py b/tensorrt_llm/serve/openai_protocol.py index 7c2ab0812bf2..7a2362c38602 100644 --- a/tensorrt_llm/serve/openai_protocol.py +++ b/tensorrt_llm/serve/openai_protocol.py @@ -312,6 +312,11 @@ class DisaggregatedParams(OpenAIBaseModel): default=None, description= "Prompt-token boundary transferred by a remote-tail context worker.") + draft_ctx_state: Optional[List[NonNegativeInt]] = Field( + default=None, + description= + "Context-first only: the standalone drafter's context-window decision " + "[ctx_len, ctx_base, ok] for the generation request.") multimodal_positions: Optional[List[NonNegativeInt]] = Field( default=None, description="Prompt-token offset of each multimodal item.") @@ -1996,6 +2001,7 @@ def to_disaggregated_params( schedule_style=tllm_disagg_params.schedule_style, ctx_usage=ctx_usage, remote_tail_start=tllm_disagg_params.remote_tail_start, + draft_ctx_state=tllm_disagg_params.draft_ctx_state, multimodal_positions=tllm_disagg_params.multimodal_positions, multimodal_lengths=tllm_disagg_params.multimodal_lengths, ) @@ -2023,6 +2029,7 @@ def to_llm_disaggregated_params( schedule_style=disaggregated_params.schedule_style, ctx_usage=None if ctx_usage is None else ctx_usage.model_dump(), remote_tail_start=disaggregated_params.remote_tail_start, + draft_ctx_state=disaggregated_params.draft_ctx_state, multimodal_positions=disaggregated_params.multimodal_positions, multimodal_lengths=disaggregated_params.multimodal_lengths, ) diff --git a/tensorrt_llm/usage/llm_args_golden_manifest.json b/tensorrt_llm/usage/llm_args_golden_manifest.json index f6c872897b8f..4b6941b8f505 100644 --- a/tensorrt_llm/usage/llm_args_golden_manifest.json +++ b/tensorrt_llm/usage/llm_args_golden_manifest.json @@ -1420,7 +1420,8 @@ { "allowed_values": [ "bf16", - "fp8" + "fp8", + "nvfp4" ], "capture_policy": "literal", "kind": "categorical", diff --git a/tests/integration/test_lists/test-db/l0_cpu.yml b/tests/integration/test_lists/test-db/l0_cpu.yml index 8ff779ac101d..a3a7566ced3d 100644 --- a/tests/integration/test_lists/test-db/l0_cpu.yml +++ b/tests/integration/test_lists/test-db/l0_cpu.yml @@ -54,9 +54,10 @@ l0_cpu: - unittest/_torch/peft - unittest/_torch/memory - unittest/_torch/modeling - # modeling_v2 routing and routing-table claims: config shapes and a monkeypatched - # compute capability, no device. The rest of unittest/_torch/modeling_v2 needs a GPU. + # modeling_v2 routing, routing-table claims and the MiniMax-M3.1 tactic pins: config shapes, a monkeypatched + # compute capability and stub runners, no device. The rest of unittest/_torch/modeling_v2 needs a GPU. - unittest/_torch/modeling_v2/test_modeling_v2_claims.py + - unittest/_torch/modeling_v2/test_m31_pins.py - unittest/_torch/modeling_v2/test_modeling_v2_routing.py - unittest/_torch/models/test_minimax_m3.py::test_minimax_m3_fp8_indexer_rejects_different_qk_norm_epsilons - unittest/_torch/models/test_minimax_m3.py::test_minimax_m3_moe_reduces_only_local_terms @@ -117,6 +118,8 @@ l0_cpu: - unittest/_torch/speculative/hw_agnostic # l0_h100 selects with -m "not cpu_only", so this entry is what runs it. - unittest/_torch/speculative/test_spec_slot_pool_sizing.py + - unittest/_torch/test_cute_aot.py + - unittest/_torch/test_flashinfer_autotune_cache.py - unittest/_torch/test_hf_quant_config.py - unittest/_torch/test_mmap_utils.py - unittest/_torch/test_model_config.py diff --git a/tests/integration/test_lists/test-db/l0_dgx_b200.yml b/tests/integration/test_lists/test-db/l0_dgx_b200.yml index 7deeb59447c5..68504b79332c 100644 --- a/tests/integration/test_lists/test-db/l0_dgx_b200.yml +++ b/tests/integration/test_lists/test-db/l0_dgx_b200.yml @@ -23,6 +23,9 @@ l0_dgx_b200: - unittest/_torch/multi_gpu/test_deepseek_v41_attention_dp.py::test_attention_dp_decoder_bounded_replay[world2] - unittest/_torch/misc/test_autotuner.py::test_autotuner_distributed_strategy - unittest/_torch/multi_gpu/test_minimax_m3_piecewise.py::test_minimax_m3_piecewise_empty_adp_rank_preserves_caches + # MiniMax-M3.1 TP2 two-rank natives: bitwise to the stock one-shot all-reduce chains. + - unittest/_torch/multi_gpu/test_minimax_m31_f3_ar_prepushed.py + - unittest/_torch/multi_gpu/test_minimax_m31_s2_moe_tail.py # One process: the MNNVL workspace's capture guard raises before any MNNVL call. - unittest/_torch/multi_gpu/test_mnnvl_allreduce.py::test_mnnvl_workspace_creation_refuses_graph_capture - accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_bf16[tp2-TRTLLM] diff --git a/tests/unittest/_torch/executor/kv_cache/test_disagg_gen_init_draft_window.py b/tests/unittest/_torch/executor/kv_cache/test_disagg_gen_init_draft_window.py new file mode 100644 index 000000000000..8fda1ff70f6c --- /dev/null +++ b/tests/unittest/_torch/executor/kv_cache/test_disagg_gen_init_draft_window.py @@ -0,0 +1,361 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""A disagg generation worker sizes a gen-init request's windowed draft mirror for its window. + +The standalone DSpark drafter's separate pool is windowed and funded per request for its window +plus the step's tokens (here 16 requests x 4,276 tokens). A disagg generation worker receives the +prompt instead of prefilling it, so the mirror takes the whole prompt as history and holds only the +window behind the prompt end. With the model's ``disagg_draft_kv`` hook the drafter also starts +from an empty window there, never reading a page the window released. No GPU needed. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor.kv_cache import kv_cache_manager_v2 as mgr_mod +from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 +from tensorrt_llm._torch.speculative.dflash import DFlashWorker, dflash_rebased_ctx_view + +pytestmark = pytest.mark.cpu_only + +TPB = 32 +WINDOW = 4096 +FUNDED_TOKENS = 16 * 4276 +PROMPT = 100_013 # not a whole number of pages: the base rounds up +DRAFT = 7 +EXTRA = 1 + + +def _blocks(tokens): + return -(-tokens // TPB) + + +class _WindowedPool: + """A windowed draft pool's page budget: a cache holds its pages from the window start on.""" + + def __init__(self, funded_tokens=FUNDED_TOKENS): + self.budget_pages = funded_tokens // TPB + self.caches = [] + self.peak_pages = 0 + + def live_pages(self): + return sum(cache.live_pages() for cache in self.caches) + + +class _Cache: + def __init__(self, pool): + self.pool = pool + self.capacity = 0 + self.history_length = 0 + pool.caches.append(self) + + def live_pages(self, capacity=None, history=None): + capacity = self.capacity if capacity is None else capacity + history = self.history_length if history is None else history + released = max(0, (history + 1 - WINDOW) // TPB) + return max(0, _blocks(capacity) - released) + + def first_held_ordinal(self): + return max(0, (self.history_length + 1 - WINDOW) // TPB) + + def resize(self, capacity, history_length=None): + history = self.history_length if history_length is None else history_length + others = self.pool.live_pages() - self.live_pages() + if others + self.live_pages(capacity, history) > self.pool.budget_pages: + return False + self.capacity, self.history_length = capacity, history + self.pool.peak_pages = max(self.pool.peak_pages, self.pool.live_pages()) + return True + + +def _manager(cache): + m = object.__new__(KVCacheManagerV2) + m.is_draft = True + m.enable_joint_kv_cache_reuse = False + # A one-engine external drafter (DSpark): its mirror follows the target's progress. + m._mirror_uses_draft_view = False + m.num_extra_kv_tokens = EXTRA + m.tokens_per_block = TPB + m.kv_cache_map = {1: cache} + m._allocated_draft_lens = {} + m._mirror_draft_kv_cache = lambda req: cache + m._resume_and_restore = lambda request_id, kv_cache: True + m._after_shrink = lambda request_id, kv_cache: None + m._draft_pool_diagnostic = lambda: "" + return m + + +def _gen_init_request(): + # What a disagg generation worker hands the draft manager for a received prompt. + return SimpleNamespace( + py_request_id=1, + prompt_len=PROMPT, + context_current_position=0, + context_chunk_size=PROMPT, + is_last_context_chunk=True, + is_disagg_generation_init_state=True, + py_draft_tokens=[0] * DRAFT, + ) + + +def _batch(req): + return SimpleNamespace( + context_requests=[req], generation_requests=[], all_requests=lambda: [req] + ) + + +def _worker(): + worker = DFlashWorker.__new__(DFlashWorker) + worker.engage_disagg_gen_init_window() + return worker + + +def test_stock_mirror_path_holds_the_window_without_a_seed(monkeypatch): + """Without the hook the mirror already takes the received prompt as history, so it holds the + window and stays inside the pool's funding; only the hook seeds the drafter's empty window.""" + monkeypatch.setattr(mgr_mod, "get_draft_token_length", lambda req: DRAFT) + pool = _WindowedPool() + cache = _Cache(pool) + m = _manager(cache) + assert m._disagg_gen_init_window is None + + m._prepare_draft_resources(_batch(_gen_init_request())) # no resize failure + + assert (cache.capacity, cache.history_length) == (PROMPT + DRAFT + EXTRA, PROMPT) + assert pool.peak_pages <= _blocks(WINDOW + DRAFT + EXTRA) + 1 + + +def test_gen_init_mirror_holds_only_the_window_and_seeds_an_empty_one(monkeypatch): + monkeypatch.setattr(mgr_mod, "get_draft_token_length", lambda req: DRAFT) + pool = _WindowedPool() + cache = _Cache(pool) + m = _manager(cache) + worker = _worker() + m.engage_disagg_gen_init_window(worker.seed_disagg_empty_window) + + m._prepare_draft_resources(_batch(_gen_init_request())) # no resize failure + + assert (cache.capacity, cache.history_length) == (PROMPT + DRAFT + EXTRA, PROMPT) + # Peak live pages: the window behind the prompt end plus the draft / extra tail. + assert pool.peak_pages <= _blocks(WINDOW + DRAFT + EXTRA) + 1 + assert pool.peak_pages <= pool.budget_pages + base = _blocks(PROMPT) + assert worker._disagg_draft_ctx_seeds == {1: (PROMPT, base, PROMPT)} + + # The drafter never reads a page the window released: its reads start at base * page, past + # the prompt end and so past the window start L - W + 1. + rows = torch.arange(base + 4, dtype=torch.long).unsqueeze(0) + for accepted in (0, 1, 9, 40): + num_ctx = torch.tensor([PROMPT + accepted]) + shifted, readable = dflash_rebased_ctx_view(rows, torch.tensor([base]), num_ctx, TPB) + first_read_position = int(num_ctx - readable) # rows below it are not read + assert first_read_position >= PROMPT > PROMPT - WINDOW + 1 + first_page = int(shifted[0, 0]) + assert first_page >= cache.first_held_ordinal() + + +def test_seed_applies_once_at_first_sight_in_prepare(monkeypatch): + from tensorrt_llm._torch.speculative import dflash as dflash_module + from tensorrt_llm._torch.speculative.dflash import DFlashSpecMetadata + + monkeypatch.setattr(dflash_module, "prefer_pinned", lambda: False) + worker = _worker() + worker.seed_disagg_empty_window(7, PROMPT, TPB) + writes = [] + worker._ctx_buf_inited = True + worker._req_to_slot = {} + worker._req_ctx_pos = {} + worker._free_slots = __import__("collections").deque([0, 1]) + worker._graph_dummy_id_floor = 1 << 40 + worker._dummy_slot = 2 + worker._ctx_block_tables = torch.zeros(1) # the drafter context is pool-bound + worker._batch_to_slot = torch.zeros(4, dtype=torch.long) + worker._write_ctx_len = lambda updates, bases=None: writes.append((dict(updates), bases)) + meta = SimpleNamespace( + request_ids=[7], + num_generations=1, + batch_indices_cuda=torch.zeros(4, dtype=torch.int), + _dflash_worker=worker, + ) + DFlashSpecMetadata.prepare(meta) + slot = worker._req_to_slot[7] + assert writes[-1] == ({slot: PROMPT}, {slot: _blocks(PROMPT)}) + DFlashSpecMetadata.prepare(meta) + assert sum(1 for _, bases in writes if bases) == 1 + + +def test_without_the_hook_nothing_is_engaged(): + """Inert unless the model sets RuntimeHooks.disagg_draft_kv.""" + from tensorrt_llm._torch.pyexecutor._util import engage_disagg_gen_init_window + from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + + m = object.__new__(KVCacheManagerV2) + m.is_draft = True + worker = DFlashWorker.__new__(DFlashWorker) + assert not engage_disagg_gen_init_window(SimpleNamespace(spec_worker=worker), m) + assert m._disagg_gen_init_window is None and worker._disagg_draft_ctx_seeds is None + hooked = SimpleNamespace(spec_worker=worker, runtime_hooks=RuntimeHooks(disagg_draft_kv=True)) + assert not engage_disagg_gen_init_window(hooked, None) + assert engage_disagg_gen_init_window(hooked, m) + assert m._disagg_gen_init_window == worker.seed_disagg_empty_window + assert worker._disagg_draft_ctx_seeds == {} + + +def test_kill_switch_moves_no_window_but_keeps_the_gen_init_sizing(monkeypatch): + """TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV means no draft transfer only: a hooked model still sizes + the generation side's draft mirror for its window and seeds the empty window.""" + from tensorrt_llm._torch.disaggregation.draft_kv import ( + DISABLE_DISAGG_DRAFT_KV_ENV, + select_disagg_draft_kv_cache_manager, + ) + from tensorrt_llm._torch.pyexecutor._util import engage_disagg_gen_init_window + from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + + monkeypatch.setenv(DISABLE_DISAGG_DRAFT_KV_ENV, "1") + monkeypatch.setattr(mgr_mod, "get_draft_token_length", lambda req: DRAFT) + pool = _WindowedPool() + m = _manager(_Cache(pool)) + worker = DFlashWorker.__new__(DFlashWorker) + model = SimpleNamespace(spec_worker=worker, runtime_hooks=RuntimeHooks(disagg_draft_kv=True)) + + assert select_disagg_draft_kv_cache_manager(model, m) is None # no draft transfer + assert engage_disagg_gen_init_window(model, m) # the window sizing stays engaged + assert m._disagg_draft_kv_receive is False and worker._disagg_draft_kv_engaged is False + + m._prepare_draft_resources(_batch(_gen_init_request())) # 100K prompt: no resize failure + assert pool.peak_pages <= _blocks(WINDOW + DRAFT + EXTRA) + 1 + assert worker._disagg_draft_ctx_seeds == {1: (PROMPT, _blocks(PROMPT), PROMPT)} + assert not getattr(m, "_disagg_receive_ready", {}) # no receive fence without a transfer + + +def _run_context_only_flow(monkeypatch, engaged): + """A disagg context-only worker: chunked context requests through the draft mirror and the + drafter's metadata prepare, with or without the gen-init window sizing engaged.""" + from collections import deque + + from tensorrt_llm._torch.speculative import dflash as dflash_module + from tensorrt_llm._torch.speculative.dflash import DFlashSpecMetadata + + monkeypatch.setattr(mgr_mod, "get_draft_token_length", lambda req: DRAFT) + monkeypatch.setattr(dflash_module, "prefer_pinned", lambda: False) + pool = _WindowedPool() + caches = {rid: _Cache(pool) for rid in (1, 2)} + resizes = [] + for rid, cache in caches.items(): + plain_resize = cache.resize + + def resize(capacity, history_length=None, rid=rid, plain_resize=plain_resize): + resizes.append((rid, capacity, history_length)) + return plain_resize(capacity, history_length) + + cache.resize = resize + m = _manager(None) + m._mirror_draft_kv_cache = lambda req: caches[req.py_request_id] + worker = DFlashWorker.__new__(DFlashWorker) + worker._ctx_buf_inited = True + worker._req_to_slot = {} + worker._req_ctx_pos = {} + worker._free_slots = deque([0, 1, 2]) + worker._graph_dummy_id_floor = 1 << 40 + worker._dummy_slot = 3 + worker._ctx_block_tables = torch.zeros(1) # pool-bound: a seed would be written + worker._batch_to_slot = torch.zeros(4, dtype=torch.long) + writes = [] + worker._write_ctx_len = lambda updates, bases=None: writes.append((dict(updates), bases)) + gen_init_calls = [] + if engaged: + from tensorrt_llm._torch.pyexecutor._util import engage_disagg_gen_init_window + from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + + model = SimpleNamespace( + spec_worker=worker, runtime_hooks=RuntimeHooks(disagg_draft_kv=True) + ) + assert engage_disagg_gen_init_window(model, m) + seed = m._disagg_gen_init_window + m._disagg_gen_init_window = lambda *args: (gen_init_calls.append(args), seed(*args)) + + chunk = 8192 + prompts = {1: 3 * chunk, 2: 1000} + positions = {1: 0, 2: 0} + while any(positions[rid] < prompts[rid] for rid in prompts): + reqs = [] + for rid, prompt in prompts.items(): + pos = positions[rid] + if pos >= prompt: + continue + size = min(chunk, prompt - pos) + # A context-only request: never in DISAGG_GENERATION_INIT. + reqs.append( + SimpleNamespace( + py_request_id=rid, + prompt_len=prompt, + context_current_position=pos, + context_chunk_size=size, + is_last_context_chunk=pos + size == prompt, + is_disagg_generation_init_state=False, + py_draft_tokens=[0] * DRAFT, + ) + ) + positions[rid] = pos + size + m._prepare_draft_resources(SimpleNamespace(context_requests=reqs, generation_requests=[])) + meta = SimpleNamespace( + request_ids=[req.py_request_id for req in reqs], + num_generations=0, + batch_indices_cuda=torch.zeros(4, dtype=torch.int), + _dflash_worker=worker, + ) + DFlashSpecMetadata.prepare(meta) + # _store_prefill_context assigns the context requests' slots after the forward. + for req in reqs: + worker._assign_slot(req.py_request_id) + # The context-only request has finished: it leaves the batch. + DFlashSpecMetadata.prepare( + SimpleNamespace( + request_ids=[], + num_generations=0, + batch_indices_cuda=torch.zeros(4, dtype=torch.int), + _dflash_worker=worker, + ) + ) + state = { + "resizes": resizes, + "caches": {rid: (c.capacity, c.history_length) for rid, c in caches.items()}, + "peak_pages": pool.peak_pages, + "writes": writes, + "req_to_slot": dict(worker._req_to_slot), + "req_ctx_pos": dict(worker._req_ctx_pos), + "free_slots": list(worker._free_slots), + } + return state, worker, m, gen_init_calls + + +def test_gen_init_sizing_is_inert_on_a_context_only_worker(monkeypatch): + """A CTX worker of a hooked model engages the gen-init window sizing too (is_disagg), but it + never holds a DISAGG_GENERATION_INIT request: the draft mirror's sizing, the drafter's slots + and context lengths are exactly the stock ones, and no seed is ever recorded.""" + stock, stock_worker, stock_m, _ = _run_context_only_flow(monkeypatch, engaged=False) + engaged, worker, m, gen_init_calls = _run_context_only_flow(monkeypatch, engaged=True) + + assert m._disagg_gen_init_window is not None and stock_m._disagg_gen_init_window is None + assert engaged == stock + assert gen_init_calls == [] + assert worker._disagg_draft_ctx_seeds == {} # engaged, never filled + assert stock_worker._disagg_draft_ctx_seeds is None + assert not getattr(m, "_disagg_receive_ready", {}) + # The stock chunked-prefill sizing: chunk start as committed history, draft tail on the last. + assert engaged["caches"][1] == (3 * 8192 + DRAFT + EXTRA, 2 * 8192) + assert engaged["caches"][2] == (1000 + DRAFT + EXTRA, 0) diff --git a/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py b/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py new file mode 100644 index 000000000000..6bb018884122 --- /dev/null +++ b/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py @@ -0,0 +1,167 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +"""Host-tier prefix prefetch to the GPU for waiting context requests, with a rank-agreed admission hold.""" + +from types import SimpleNamespace +from unittest import mock + +import pytest + +import tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 as kvm +import tensorrt_llm._torch.pyexecutor.py_executor as pe + +pytestmark = pytest.mark.cpu_only + + +def _req(rid, first=True): + return SimpleNamespace(py_request_id=rid, is_first_context_chunk=first) + + +def _executor(reqs, nbytes_by_rid, tp_size=1, peer_holds=None): + kvc = SimpleNamespace( + kv_cache_map={}, + prefetch_to_gpu=mock.Mock(side_effect=lambda r: nbytes_by_rid.get(r.py_request_id, 0)), + ) + + def allgather(holds): + return [holds] + [peer_holds or holds for _ in range(tp_size - 1)] + + dist = SimpleNamespace(tp_size=tp_size, rank=0, tp_allgather=mock.Mock(side_effect=allgather)) + return SimpleNamespace(active_requests=reqs, kv_cache_manager=kvc, dist=dist) + + +def test_hold_from_bytes_and_cap(monkeypatch): + monkeypatch.setattr(pe, "_KV_GPU_PREFETCH_MAX", 2) + reqs = [_req(1), _req(2), _req(3)] + ex = _executor(reqs, {1: 0, 2: 5_200_000_000, 3: 1}) + pe.PyExecutor._prefetch_context_prefixes_to_gpu(ex) + assert reqs[0].py_kv_gpu_hold == 0 # already on GPU: no hold + assert reqs[1].py_kv_gpu_hold == 4 # ceil(5.2e9 / 1.5e9) + assert not hasattr(reqs[2], "py_kv_gpu_hold") # beyond the per-step cap + assert ex.kv_cache_manager.prefetch_to_gpu.call_count == 2 + + +def test_hold_clamped_and_counts_down(monkeypatch): + monkeypatch.setattr(pe, "_KV_GPU_PREFETCH_MAX", 2) + r = _req(7) + ex = _executor([r], {7: 100_000_000_000}) + pe.PyExecutor._prefetch_context_prefixes_to_gpu(ex) + assert r.py_kv_gpu_hold == pe._KV_GPU_PREFETCH_MAX_HOLD + for want in range(pe._KV_GPU_PREFETCH_MAX_HOLD - 1, -1, -1): + pe.PyExecutor._prefetch_context_prefixes_to_gpu(ex) # once per step + assert r.py_kv_gpu_hold == want + assert ex.kv_cache_manager.prefetch_to_gpu.call_count == 1 # prefetched once + + +def test_hold_is_max_over_ranks(monkeypatch): + monkeypatch.setattr(pe, "_KV_GPU_PREFETCH_MAX", 2) + a, b = _req(1), _req(2) + ex = _executor([a, b], {1: 0, 2: 1_000}, tp_size=2, peer_holds=[3, 0]) + pe.PyExecutor._prefetch_context_prefixes_to_gpu(ex) + assert (a.py_kv_gpu_hold, b.py_kv_gpu_hold) == (3, 1) + + +def test_skips_non_first_chunks_and_admitted(monkeypatch): + monkeypatch.setattr(pe, "_KV_GPU_PREFETCH_MAX", 2) + a, b = _req(1, first=False), _req(2) + ex = _executor([a, b], {1: 10**9, 2: 10**9}) + ex.kv_cache_manager.kv_cache_map[2] = object() + pe.PyExecutor._prefetch_context_prefixes_to_gpu(ex) + assert ex.kv_cache_manager.prefetch_to_gpu.call_count == 0 + + +def _kvm(tokens_by_level, prefetch_ok=True): + cache = mock.Mock() + cache.cached_tokens_by_level = tokens_by_level + cache.prefetch.return_value = prefetch_ok + impl = SimpleNamespace(create_kv_cache=mock.Mock(return_value=cache)) + self = SimpleNamespace( + enable_block_reuse=True, + impl=impl, + _context_reuse_tokens=lambda r: [1, 2, 3], + _derive_reuse_salt=lambda s: None, + _kv_token_bytes_estimate=lambda: 1000, + ) + return self, cache + + +def test_prefetch_only_when_off_gpu_and_always_closes(): + req = SimpleNamespace(cache_salt=None, lora_task_id=None) + self, cache = _kvm([96, 0]) + assert kvm.KVCacheManagerV2.prefetch_to_gpu(self, req) == 0 + cache.prefetch.assert_not_called() + cache.close.assert_called_once() + self, cache = _kvm([32, 48, 16]) + assert kvm.KVCacheManagerV2.prefetch_to_gpu(self, req) == (48 + 16) * 1000 + cache.prefetch.assert_called_once_with(kvm.GPU_LEVEL) + cache.close.assert_called_once() + + +def test_prefetch_failure_means_no_hold(): + req = SimpleNamespace(cache_salt=None, lora_task_id=None) + self, cache = _kvm([0, 64], prefetch_ok=False) + assert kvm.KVCacheManagerV2.prefetch_to_gpu(self, req) == 0 + cache.close.assert_called_once() + + +def test_kill_switch_default_on(): + assert pe._KV_GPU_PREFETCH is True + + +@pytest.mark.parametrize("hold", [0, 1, 3]) +def test_scheduler_guard_present(hold): + # The scheduler skips a pending context request while py_kv_gpu_hold > 0. + import inspect + + import tensorrt_llm._torch.pyexecutor.scheduler.scheduler_v2 as sv2 + + assert 'hold = getattr(req, "py_kv_gpu_hold", 0)' in inspect.getsource(sv2) + assert "type(hold) is int and hold > 0" in inspect.getsource(sv2) + assert (getattr(SimpleNamespace(py_kv_gpu_hold=hold), "py_kv_gpu_hold", 0) > 0) == (hold > 0) + + +def test_mock_requests_are_never_held(): + # Stock scheduler tests drive Mock requests: getattr on a Mock returns a Mock, which must not count as a hold. + hold = getattr(mock.Mock(), "py_kv_gpu_hold", 0) + assert not (type(hold) is int and hold > 0) + + +def test_counter_prints_every_200_steps_even_without_candidates(monkeypatch): + log = mock.Mock() + monkeypatch.setattr(pe, "logger", log) + ex = _executor([], {}) + for _ in range(400): + pe.PyExecutor._prefetch_context_prefixes_to_gpu(ex) + lines = [c.args[0] for c in log.info.call_args_list] + assert lines == [ + "[KVPF] steps 200 candidates 0 prefetched 0", + "[KVPF] steps 400 candidates 0 prefetched 0", + ] + assert ex.kv_cache_manager.prefetch_to_gpu.call_count == 0 + + +@pytest.mark.parametrize( + "switch,v2,reuse,want", + [ + (True, True, True, "enabled"), + (False, True, True, "disabled (kill switch)"), + (True, False, True, "inactive (KV cache manager v1)"), + (True, True, False, "inactive (block reuse off)"), + ], +) +def test_startup_line_once_with_state(monkeypatch, switch, v2, reuse, want): + log = mock.Mock() + monkeypatch.setattr(pe, "logger", log) + monkeypatch.setattr(pe, "_KV_GPU_PREFETCH", switch) + ex = SimpleNamespace( + dist=SimpleNamespace(rank=0), + _is_kv_manager_v2=v2, + kv_cache_manager=SimpleNamespace(enable_block_reuse=reuse, disk_prefetch_num_reqs=0), + _prefetch_context_prefixes_to_gpu=mock.Mock(), + ) + ex._kv_gpu_prefetch_state = lambda: pe.PyExecutor._kv_gpu_prefetch_state(ex) + for _ in range(3): + pe.PyExecutor._prefetch_for_context_requests(ex) + startup = [c.args[0] for c in log.info.call_args_list if "[KVPF] startup" in c.args[0]] + assert len(startup) == 1 and startup[0].startswith(f"[KVPF] startup: {want} (") + assert ex._prefetch_context_prefixes_to_gpu.call_count == (3 if switch and v2 and reuse else 0) diff --git a/tests/unittest/_torch/executor/test_request_token_array.py b/tests/unittest/_torch/executor/test_request_token_array.py new file mode 100644 index 000000000000..0e96ef0ee583 --- /dev/null +++ b/tests/unittest/_torch/executor/test_request_token_array.py @@ -0,0 +1,147 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Prompt tokens handed to LlmRequest as the executor request's int32 array: which requests take it (context-only by +default, every type with the opt-in, none with the kill switch), and the same request as the list path.""" + +import gc +from types import SimpleNamespace + +import numpy as np +import pytest + +from tensorrt_llm._torch.pyexecutor import llm_request as lr_mod +from tensorrt_llm.bindings import executor as tllm + +pytestmark = pytest.mark.cpu_only + +PROMPT = [7, 0, 151643, 2**31 - 1, 42, 3] + + +def _executor_request(tokens=PROMPT): + return tllm.Request(input_token_ids=list(tokens), max_tokens=4) + + +def test_input_token_ids_np_is_an_owned_int32_copy(): + req = _executor_request() + arr = req.input_token_ids_np + assert arr.dtype == np.int32 and arr.ndim == 1 and arr.flags["C_CONTIGUOUS"] + assert arr.tolist() == PROMPT == req.input_token_ids + arr[0] = -5 # the copy is the caller's: the request keeps its tokens + assert req.input_token_ids == PROMPT + del req + gc.collect() + assert arr.tolist() == [-5] + PROMPT[1:] # and it outlives the request + + +def test_input_token_ids_np_reads_again(): + # Each read returns its own owned copy; a borrowed (reference_internal) property would raise on the second read. + req = _executor_request() + first, second = req.input_token_ids_np, req.input_token_ids_np + assert first.tolist() == second.tolist() == PROMPT + assert not np.shares_memory(first, second) + + +def test_empty_prompt_is_rejected(): + # The executor request needs a prompt, so the array accessor never sees an empty one. + with pytest.raises(RuntimeError): + tllm.Request(input_token_ids=[], max_tokens=4) + + +RT = tllm.RequestType + + +def _typed_request(request_type, with_array=True): + req = SimpleNamespace(request_type=request_type, input_token_ids=list(PROMPT)) + if with_array: + req.input_token_ids_np = np.asarray(PROMPT, dtype=np.int32) + return req + + +def _scope(monkeypatch, on=True, all_requests=False): + monkeypatch.setattr(lr_mod, "_L1_TOKEN_ARRAY", on) + monkeypatch.setattr(lr_mod, "_L1_ALL_REQUESTS", all_requests) + + +@pytest.mark.parametrize( + "request_type,array", + [ + (RT.REQUEST_TYPE_CONTEXT_ONLY, True), + (RT.REQUEST_TYPE_CONTEXT_AND_GENERATION, False), + (RT.REQUEST_TYPE_GENERATION_ONLY, False), + ], +) +def test_default_scope_is_context_only(monkeypatch, request_type, array): + _scope(monkeypatch) + out = lr_mod._l1_input_tokens(_typed_request(request_type)) + assert isinstance(out, np.ndarray) is array + assert list(out) == PROMPT + + +def test_opt_in_covers_every_request_type(monkeypatch): + _scope(monkeypatch, all_requests=True) + for request_type in (RT.REQUEST_TYPE_CONTEXT_AND_GENERATION, RT.REQUEST_TYPE_GENERATION_ONLY): + assert isinstance(lr_mod._l1_input_tokens(_typed_request(request_type)), np.ndarray) + + +def test_kill_switch_wins_over_the_opt_in(monkeypatch): + _scope(monkeypatch, on=False, all_requests=True) + assert lr_mod._l1_input_tokens(_typed_request(RT.REQUEST_TYPE_CONTEXT_ONLY)) == PROMPT + + +def test_request_without_the_array_uses_the_list(monkeypatch): + _scope(monkeypatch, all_requests=True) + out = lr_mod._l1_input_tokens(_typed_request(RT.REQUEST_TYPE_CONTEXT_ONLY, with_array=False)) + assert out == PROMPT + + +@pytest.mark.parametrize("all_requests", [True, False]) +def test_llm_request_tokens_match_the_list_path(monkeypatch, all_requests): + # A context-and-generation request: the array path with the opt-in, the list path without it. + _scope(monkeypatch, all_requests=all_requests) + llm_req = lr_mod.executor_request_to_llm_request( + req_id=1, + executor_request=_executor_request(), + child_req_ids=[], + exclude_last_generation_logits=False, + ) + assert llm_req.get_tokens(0) == PROMPT + assert llm_req.orig_prompt_len == len(PROMPT) + + +def test_explicit_input_token_ids_win(monkeypatch): + _scope(monkeypatch, all_requests=True) + llm_req = lr_mod.executor_request_to_llm_request( + req_id=2, + executor_request=_executor_request(), + child_req_ids=[], + exclude_last_generation_logits=False, + input_token_ids=[9, 8, 7], + ) + assert llm_req.get_tokens(0) == [9, 8, 7] + + +def test_llm_request_accepts_an_int32_array_and_a_list(): + def make(tokens): + return lr_mod.LlmRequest( + request_id=3, + max_new_tokens=4, + input_tokens=tokens, + sampling_config=tllm.SamplingConfig(1), + is_streaming=False, + ) + + from_array = make(np.asarray(PROMPT, dtype=np.int32)) + from_list = make(list(PROMPT)) + assert from_array.get_tokens(0) == from_list.get_tokens(0) == PROMPT diff --git a/tests/unittest/_torch/executor/test_runtime_hooks.py b/tests/unittest/_torch/executor/test_runtime_hooks.py index 2e392661d69f..e72b5b33fdf1 100644 --- a/tests/unittest/_torch/executor/test_runtime_hooks.py +++ b/tests/unittest/_torch/executor/test_runtime_hooks.py @@ -34,6 +34,7 @@ def test_model_without_hooks_gets_the_all_off_default(): assert not hooks.skip_unchanged_h2d assert not hooks.loop_on_execution_stream assert not hooks.merge_tuned_tactics_across_ranks + assert not hooks.persist_flashinfer_mxfp8_tactics assert not hooks.skip_greedy_rng_upload assert not hooks.pack_spec_rng_uploads assert hooks.spec_step_fold is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/activation/test_modeling_v2_m31_tp2_swq.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/activation/test_modeling_v2_m31_tp2_swq.py new file mode 100644 index 000000000000..4866dba21341 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/activation/test_modeling_v2_m31_tp2_swq.py @@ -0,0 +1,151 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SWQ at TP2 (the sm_107 TP8 lever, ported): the m3_swq entry writes the shared expert's SwiGLU-OAI as the down +projection's MXFP8 activation, bitwise to silu_and_mul + mxfp8_quantize, at the TP2 shared width 1536 (the cells of +17 and 32 rows: the rows the shared down's GEMV does not serve) and at the TP8 width 384 it was certified on; the +The scale-code-0 exception keeps the code-0 blocks finite. The dispatch takes it on SHARED_DOWN (6144, 1536) at +17..32 rows; TRTLLM_M31_DISABLE_SWQ=1 empties SWQ_ROWS.""" + +import importlib +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + m31_switches, + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.activation.m3_swq import ( # noqa: E501 + m3_swq as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +OAI = (7.0, 1.702, 1.0) +_gpu = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + + +def test_swq_rows_default_and_kill(monkeypatch) -> None: + """CPU: the shared down at the TP2 width, 17..32 rows, none of them a GEMV row; the kill switch empties it.""" + assert modeling.SWQ is True + assert modeling.SWQ_SHARED_DOWN == (6144, 1536) == modeling.SHARED_DOWN + assert modeling.SWQ_ROWS == {(6144, 1536): frozenset(range(17, 33))} + assert not modeling.SWQ_ROWS[(6144, 1536)] & modeling.MX_GEMV_ROWS[modeling.SHARED_DOWN] + monkeypatch.setenv("TRTLLM_M31_DISABLE_SWQ", "1") + try: + importlib.reload(m31_switches) + killed = importlib.reload(modeling) + assert killed.SWQ is False and killed.SWQ_ROWS == {} + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_SWQ", raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) + + +def test_the_cells_cover_the_tp2_rows() -> None: + """CPU: the entry's cells include the TP2 width at the first and last SWQ row.""" + cells = {(c.spec["m"], c.spec["inter"]) for c in op.CELLS} + assert {(17, 1536), (32, 1536)} <= cells + + +def _assert_bitwise(x): + q, sf = op(x, *OAI) + rq, rsf = op.reference(x, *OAI) + assert q.shape == rq.shape and q.dtype == torch.float8_e4m3fn + assert torch.equal(q.view(torch.uint8), rq.view(torch.uint8)) + assert sf.numel() <= rsf.numel() and torch.equal(sf, rsf.view(-1)[: sf.numel()]) + + +@_gpu +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = (torch.randn(cell.spec["m"], 2 * cell.spec["inter"], generator=g, device="cuda") * 4.0).to( + torch.bfloat16 + ) + with validating(op): + _assert_bitwise(x) + + +@_gpu +@pytest.mark.parametrize("inter", [1536, 384]) +@pytest.mark.parametrize("m", [17, 24, 32]) +@pytest.mark.parametrize("scale", [1e-3, 1.0, 1e3]) +def test_bitwise_across_magnitudes_and_edges(inter, m, scale) -> None: + """Scaled inputs, gates past the clamp, an all-zero row and a -0.0 row: q bytes and every scale byte.""" + g = torch.Generator(device="cuda").manual_seed(m * 1000 + inter + int(scale * 10) % 997) + x = torch.randn(m, 2 * inter, generator=g, device="cuda") * scale + x[1, :inter] = x[1, :inter].abs() + OAI[0] + x[2] = 0.0 + x[3] = -0.0 + _assert_bitwise(x.to(torch.bfloat16)) + + +@_gpu +@pytest.mark.parametrize("m", [17, 32]) +def test_bitwise_under_block_scale_stress(m) -> None: + """Every 32-element block of gate and up scaled by its own 2^-12..2^+12 at the TP2 width: q and scale bytes.""" + inter = 1536 + g = torch.Generator(device="cuda").manual_seed(4242 + m) + x = torch.randn(m, 2 * inter, generator=g, device="cuda") + block = torch.randint(-12, 13, (m, 2 * inter // 32), generator=g, device="cuda").float() + x = x * torch.exp2(block).repeat_interleave(32, dim=1) + _assert_bitwise(x.to(torch.bfloat16)) + + +def _scale_bytes(sf, m, nb): + """Un-swizzle the 128x4 scale layout to [m, nb] bytes.""" + nbp = (nb + 3) // 4 * 4 + r = torch.arange(m, device=sf.device).view(-1, 1) + c = torch.arange(nb, device=sf.device).view(1, -1) + idx = ( + (c % 4) + (c // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + (r // 128) * (128 * nbp) + ) + return sf.view(-1)[idx] + + +@_gpu +@pytest.mark.parametrize("m", [17, 32]) +def test_tiny_amax_blocks_stay_finite(m) -> None: + """Scale code 0 at TP2 width: a block whose scale code is 0 (amax / 448 <= 2^-127) is native's inf/NaN; SWQ keeps + native's scale byte, writes data quantized with the exact reciprocal 2^127, and stays bitwise to native on every + other block.""" + inter = 1536 + nb = inter // 32 + g = torch.Generator(device="cuda").manual_seed(151 + m) + gate = torch.randn(m, inter, generator=g, device="cuda") + up = torch.randn(m, inter, generator=g, device="cuda") + pattern = torch.tensor([1.0, -0.75, 0.5, -0.25, 0.375, 0.0, -1.0, 0.625] * 4, device="cuda") + for r in range(m): + for j, c in enumerate([2.0**-130, 2.0**-127, 448 * 2.0**-128, 448 * 2.0**-127, 2.0**-118]): + blk = (r * 5 + j * 3) % nb + gate[r, blk * 32 : (blk + 1) * 32] = pattern * (2 * c) + up[r, blk * 32 : (blk + 1) * 32] = pattern * (2 * c) + x = torch.cat([gate, up], 1).to(torch.bfloat16) + q, sf = op(x, *OAI) + rq, rsf = op.reference(x, *OAI) + h = torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=OAI[0], swiglu_alpha=OAI[1], swiglu_beta=OAI[2] + ).float() + byte, rbyte = _scale_bytes(sf, m, nb), _scale_bytes(rsf, m, nb) + assert torch.equal(byte, rbyte), "scale bytes are native's on every block" + deq = q.float().view(m, nb, 32) * torch.exp2(byte.float() - 127.0).unsqueeze(-1) + rdeq = rq.float().view(m, nb, 32) * torch.exp2(rbyte.float() - 127.0).unsqueeze(-1) + native_finite = torch.isfinite(rdeq).all(-1) + code0 = (byte == 0) & (h.view(m, nb, 32).abs().amax(-1) != 0) + assert bool(code0.any()), "the planted blocks reach scale code 0" + assert not bool(native_finite[code0].any()), ( + "native is inf/NaN on code-0 blocks (the bug this guards)" + ) + assert bool(torch.isfinite(deq).all()), "SWQ is finite everywhere" + qb, rqb = q.view(torch.uint8).view(m, nb, 32), rq.view(torch.uint8).view(m, nb, 32) + assert bool((qb == rqb).all(-1)[native_finite].all()), ( + "bitwise to native wherever native is finite" + ) + exact = (h.view(m, nb, 32) * 2.0**127).to(torch.float8_e4m3fn).view(torch.uint8) + assert bool((qb == exact).all(-1)[code0].all()), "code-0 data = RN-e4m3(h * 2^127)" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/activation/test_modeling_v2_m3_silu_and_mul.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/activation/test_modeling_v2_m3_silu_and_mul.py new file mode 100644 index 000000000000..90c69cc46038 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/activation/test_modeling_v2_m3_silu_and_mul.py @@ -0,0 +1,49 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_silu_and_mul entry of the MiniMax-M3.1 tp2 target: SwiGLU-OAI and plain SwiGLU.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.activation.m3_silu_and_mul import ( # noqa: E501 + m3_silu_and_mul as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_silu_and_mul requires a CUDA device" + +OAI = (7.0, 1.702, 1.0) + + +def _build(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + # Gate and up spread past the clamp (|x| up to ~12) so both clamps engage. + x = (torch.randn(spec["m"], 2 * spec["inter"], generator=g, device="cuda") * 4.0).to( + torch.bfloat16 + ) + return (x, *OAI) if spec["oai"] else (x, None, None, None) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """fp32 arithmetic and one bf16 rounding on both sides: the kernel's fast sigmoid moves the fp32 value by a few + fp32 ulps, which flips the bf16 rounding of an element by at most one step (1 unit of ULP[bf16] of the row); + measured on GB300: worst element 0.5 unit, relative RMS 0.01.""" + args = _build(cell.spec, zlib.crc32(cell.why.encode()) % 2**31) + with validating(op): + out = op(*args) + assert out.shape == (cell.spec["m"], cell.spec["inter"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(*args), element_ulp=1.0, rms_ulp=0.1) + + +def test_the_gate_sees_a_missing_clamp() -> None: + """Dropping the SwiGLU-OAI clamps moves every element whose gate or up passed the limit far outside the band.""" + spec = op.CELLS[0].spec + x, *oai = _build(dict(spec, m=64), 3) + out = op(x, *oai) + with pytest.raises(AssertionError): + assert_within_ulp(out, op.reference(x, None, oai[1], oai[2]), element_ulp=1.0, rms_ulp=0.1) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/_m3_attn_runtime_tp2.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/_m3_attn_runtime_tp2.py new file mode 100644 index 000000000000..1eadd10e0944 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/_m3_attn_runtime_tp2.py @@ -0,0 +1,275 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Real MiniMax-M3 attention runtime state at the TP2 per-rank geometry, for the backend-step entries' GPU tests. + +One hybrid NVFP4 KV cache manager (``MiniMaxM3KVCacheManagerV2``) laid out as the target's: a dense layer in the FP8 +half, a sparse MSA layer in the NVFP4 half with its FP8 index-K cache, and (``draft=True``) the one-model Eagle3 +drafter's shared layer, read through the manager's FP8 draft view as the runtime's draft swap reads it. Metadata is +the runtime's own type, filled by its own ``prepare()``; the backends are built the way the target builds them. + +The geometry is one rank's: 32 query heads over two KV heads and two index heads of dim 128 (one index-K head). +The mapping is a single rank holding exactly those heads, so nothing is sharded again. + +History rows are written by the in-tree scatters (``fused_write_layer_caches`` for the FP8 layers, +``fused_write_layer_caches_nvfp4`` for the NVFP4 layer and its index-K), at the slots the manager assigned, which are +not in request order. +""" + +from __future__ import annotations + +from typing import Sequence + +import torch + +from tensorrt_llm._torch.attention.backends.interface import PositionalEmbeddingParams, RopeParams +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import ( + MiniMaxM3KVCacheManagerV2, + MiniMaxM3MsaSparseAttention, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_scatter import ( + fused_write_layer_caches, + fused_write_layer_caches_nvfp4, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.msa_backend import ( + MiniMaxM3MsaSparseAttentionMetadata, +) +from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention, TrtllmAttentionMetadata +from tensorrt_llm._torch.attention.backends.utils import create_attention +from tensorrt_llm._torch.metadata import KVCacheParams +from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy +from tensorrt_llm.bindings import DataType +from tensorrt_llm.bindings.internal.batch_manager import CacheType +from tensorrt_llm.functional import PositionEmbeddingType +from tensorrt_llm.llmapi.llm_args import ( + Eagle3DecodingConfig, + KvCacheConfig, + MiniMaxM3SparseAttentionConfig, +) +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.models.modeling_utils import QuantConfig +from tensorrt_llm.quantization.mode import QuantAlgo + +HEADS = 32 +KV_HEADS = 2 +INDEX_HEADS = 2 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 +DENSE, SPARSE, DRAFT = 0, 1, 2 +MAX_BATCH = 8 +MAX_SEQ_LEN = 8192 +MAX_NUM_TOKENS = 8192 +#: The Eagle3 drafter checkpoint's RoPE: NeoX over the whole head, theta 5e6. +DRAFT_ROPE_THETA = 5.0e6 + +#: E4M3-exact planted index levels, 1/8 apart in relative terms (0.25 .. 15): 48 distinct block scores. +LEVELS = tuple(2.0**e * (1 + m / 8) for e in range(-2, 4) for m in range(8)) +#: The index value of a block's non-peak rows; its score (8) is below every level's (>= 16). +LOW = 0.125 + + +def mapping() -> Mapping: + return Mapping(world_size=1, rank=0, tp_size=1, pp_size=1) + + +def sparse_attention_config() -> MiniMaxM3SparseAttentionConfig: + """The target's runtime: MSA, the FP8 indexer, the fused five-way projection (M3's checkpoint selection rule: + no forced initial blocks, the query's own block forced).""" + return MiniMaxM3SparseAttentionConfig( + implementation="msa", + indexer_kv_dtype="fp8", + fuse_qkv_index_projection=True, + sparse_num_index_heads=INDEX_HEADS, + num_attention_heads=HEADS, + num_key_value_heads=KV_HEADS, + ) + + +def make_manager(*, draft: bool) -> MiniMaxM3KVCacheManagerV2: + torch.cuda.init() + return MiniMaxM3KVCacheManagerV2( + KvCacheConfig( + max_tokens=4 * MAX_SEQ_LEN, + enable_block_reuse=False, + enable_swa_scratch_reuse=False, + host_cache_size=0, + event_buffer_max_size=0, + ), + CacheType.SELF, + num_layers=2, + num_kv_heads=[KV_HEADS] * (3 if draft else 2), + head_dim=HEAD_DIM, + tokens_per_block=PAGE, + max_seq_len=MAX_SEQ_LEN, + max_batch_size=MAX_BATCH, + max_num_tokens=MAX_NUM_TOKENS, + mapping=mapping(), + dtype=DataType.NVFP4, + vocab_size=16, + spec_config=( + Eagle3DecodingConfig(max_draft_len=3, speculative_model="unused") if draft else None + ), + sparse_layer_ids=[SPARSE], + sparse_attention_config=sparse_attention_config(), + ) + + +def make_msa_backend(layer: int) -> MiniMaxM3MsaSparseAttention: + """One layer's MSA backend as the tp2 target's ``MiniMaxM3Tp2ModelingV2`` builds it (NVFP4 KV quant config on every + layer).""" + quant = QuantConfig(kv_cache_quant_algo=QuantAlgo.NVFP4) + backend = create_attention( + "TRTLLM", + layer, + HEADS, + HEAD_DIM, + KV_HEADS, + pos_embd_params=None, + quant_config=quant, + q_scaling=1.0, + sparse_params=sparse_attention_config().to_sparse_params(layer_idx=layer), + ) + backend.update_quant_config(quant) + assert isinstance(backend, MiniMaxM3MsaSparseAttention), type(backend).__name__ + return backend + + +def make_draft_backend() -> TrtllmAttention: + """The drafter layer's attention as ``M3Eagle3Draft`` builds it: fused NeoX RoPE, an FP8 KV cache (the target + overrides the drafter's KV dtype to FP8), no other quantization.""" + quant = QuantConfig(kv_cache_quant_algo=QuantAlgo.FP8) + backend = create_attention( + "TRTLLM", + DRAFT, + HEADS, + HEAD_DIM, + KV_HEADS, + pos_embd_params=PositionalEmbeddingParams( + type=PositionEmbeddingType.rope_gpt_neox, + rope=RopeParams(dim=HEAD_DIM, theta=DRAFT_ROPE_THETA, max_positions=MAX_SEQ_LEN), + ), + quant_config=quant, + q_scaling=1.0, + attention_chunk_size=None, + sparse_params=None, + ) + backend.update_quant_config(quant) + assert type(backend) is TrtllmAttention, type(backend).__name__ + return backend + + +def make_msa_metadata(manager) -> MiniMaxM3MsaSparseAttentionMetadata: + metadata = MiniMaxM3MsaSparseAttentionMetadata( + max_num_requests=MAX_BATCH, + max_num_tokens=MAX_NUM_TOKENS, + max_num_sequences=MAX_BATCH, + kv_cache_manager=manager, + mapping=mapping(), + sparse_metadata_params=sparse_attention_config().to_sparse_metadata_params(), + ) + # The target's MSA preparation choice that changes what prepare() stages (the identity tables of short steps). + metadata.msa_policy = MsaRuntimePolicy(short_sequence_identity=True) + return metadata + + +def make_draft_metadata(manager) -> TrtllmAttentionMetadata: + """Drafter metadata over the shared manager's FP8 draft view, the manager the runtime's draft swap installs.""" + view = manager.get_draft_kv_cache_view() + assert view is not None and view.dtype == DataType.FP8 + return TrtllmAttentionMetadata( + max_num_requests=MAX_BATCH, + max_num_tokens=MAX_NUM_TOKENS, + kv_cache_manager=view, + mapping=mapping(), + ) + + +def allocate(manager, request_ids: Sequence[int], qlens: Sequence[int], cached: Sequence[int]): + manager.add_dummy_requests(list(request_ids), token_nums=[c + n for c, n in zip(cached, qlens)]) + + +def prepare(metadata, request_ids, qlens, cached, num_contexts: int) -> None: + metadata.seq_lens = torch.tensor(list(qlens), dtype=torch.int) + metadata.num_contexts = num_contexts + metadata.request_ids = list(request_ids) + metadata.prompt_lens = [ + c + n if row < num_contexts else c for row, (c, n) in enumerate(zip(cached, qlens)) + ] + metadata.kv_cache_params = KVCacheParams(use_cache=True, num_cached_tokens_per_seq=list(cached)) + metadata.prepare() + + +def flat_slots(manager, layer: int, request_id: int, positions: torch.Tensor) -> torch.Tensor: + """The cache slots (page slot id * PAGE + offset) of a request's positions in ``layer``.""" + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0], + dtype=torch.int64, + device="cuda", + ) + positions = positions.to(device="cuda", dtype=torch.int64) + return (pages[positions // PAGE] * PAGE + positions % PAGE).to(torch.int32) + + +def write_fp8_rows(manager, layer, request_id, positions, k, v) -> None: + """K / V rows [n, 128] (E4M3) into an FP8 layer's cache at ``positions``.""" + view = manager.get_buffers(layer, "HND") + assert view.dtype == torch.float8_e4m3fn + slots = flat_slots(manager, layer, request_id, positions) + assert fused_write_layer_caches(view[:, 0], view[:, 1], None, slots, k, v, None) + + +def write_nvfp4_rows(manager, metadata, layer, request_id, positions, k, v, idx_k, quant) -> None: + """BF16 K / V rows quantized into the NVFP4 layer's cache, and index-K rows into its FP8 index cache.""" + data = manager.get_buffers(layer, "HND") + scales = manager.get_block_scale_buffers(layer, "HND") + slots = flat_slots(manager, layer, request_id, positions) + assert fused_write_layer_caches_nvfp4( + data[:, 0], + data[:, 1], + scales[:, 0], + scales[:, 1], + metadata.msa_idx_k_cache(layer), + slots, + k, + v, + idx_k, + quant, + ) + + +def index_directions(gen: torch.Generator) -> tuple[torch.Tensor, torch.Tensor]: + """Two orthogonal ±1 index directions (the halves of the 128 dims), E4M3-exact.""" + signs = torch.randint(0, 2, (HEAD_DIM,), generator=gen, device="cuda").float() * 2 - 1 + u0 = torch.zeros(HEAD_DIM, device="cuda") + u1 = torch.zeros(HEAD_DIM, device="cuda") + u0[: HEAD_DIM // 2] = signs[: HEAD_DIM // 2] + u1[HEAD_DIM // 2 :] = signs[HEAD_DIM // 2 :] + return u0, u1 + + +def planted_index_keys(length: int, u0, u1, gen: torch.Generator) -> torch.Tensor: + """Index-K rows [length, 128] (bf16, E4M3-exact) whose per-block maximum score is unambiguous. + + Every block has one peak row at a random position, ``a_b u0 + c_b u1`` with ``a`` and ``c`` two independent + permutations of ``LEVELS``; its other rows are ``LOW (u0 + u1)``. A query along ``u0`` then ranks the blocks by + ``a``, one along ``u1`` by ``c``, every score 1/8 or more apart from the next.""" + blocks = (length + PAGE - 1) // PAGE + assert blocks <= len(LEVELS) + levels = torch.tensor(LEVELS, device="cuda") + a = levels[torch.randperm(len(LEVELS), generator=gen, device="cuda")[:blocks]] + c = levels[torch.randperm(len(LEVELS), generator=gen, device="cuda")[:blocks]] + rows = (LOW * (u0 + u1)).expand(length, HEAD_DIM).clone() + for b in range(blocks): + span = min(PAGE, length - b * PAGE) + peak = b * PAGE + int(torch.randint(0, span, (1,), generator=gen, device="cuda")) + rows[peak] = a[b] * u0 + c[b] * u1 + return rows.to(torch.bfloat16) + + +def nvfp4_quant(*values: torch.Tensor) -> torch.Tensor: + """The (Q, K, V) quantization scales ``fused_write_layer_caches_nvfp4`` takes: each role's amax at the top of the + E2M1 x E4M3 range.""" + quant = torch.ones(3, dtype=torch.float32, device="cuda") + for role, x in enumerate(values, start=1): + quant[role] = (448.0 * 6.0) / x.abs().max().float() + return quant diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/_m3_nvfp4_cache.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/_m3_nvfp4_cache.py new file mode 100644 index 000000000000..99255895e5b5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/_m3_nvfp4_cache.py @@ -0,0 +1,108 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""A MiniMax-M3.1 TP2 sparse-layer decode step over a packed NVFP4 cache, for the attention entries' tests. + +The cache is written by the in-tree scatter (``fused_write_layer_caches_nvfp4``, +which quantizes with ``torch.ops.trtllm.fp4_quantize``), into one +``[pages, 2 (K, V), 2 (KV heads), 128, .]`` pool per role exactly as ``msa_paged_kv`` and +``get_block_scale_buffers`` hand the K and V views over. Pages are shuffled so +a kernel that ignored the block table would read the wrong ones. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from typing import Sequence + +import torch + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_scatter import ( + fused_write_layer_caches_nvfp4, +) + +NUM_HEADS = 32 +NUM_KV_HEADS = 2 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 + + +def token_kv_len(seq_len: int, decode_query_len: int, offset: int) -> int: + """The KV extent query token ``offset`` of a request attends, as every decode kernel derives it.""" + return max(seq_len - decode_query_len + offset + 1, 0) + + +def build_step( + seq_lens: Sequence[int], decode_query_len: int, *, identity: bool, seed: int +) -> SimpleNamespace: + """One decode step: E4M3 q [T, 32, 128], the packed NVFP4 K/V views and their scales, a [2, T, 16] + head-major top-k table (the identity table, or a random ascending selection with a -1 tail per KV head), the + block table and per-request lengths.""" + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + total_q = batch * decode_query_len + max_blocks = max(1, max((s + PAGE - 1) // PAGE for s in seq_lens)) + num_pages = batch * max_blocks + num_slots = num_pages * PAGE + block_table = ( + torch.randperm(num_pages, device="cuda", generator=g) + .to(torch.int32) + .reshape(batch, max_blocks) + ) + data_pool = torch.zeros( + num_pages, 2, NUM_KV_HEADS, PAGE, HEAD_DIM // 2, dtype=torch.uint8, device="cuda" + ) + scale_pool = torch.zeros( + num_pages, 2, NUM_KV_HEADS, PAGE, HEAD_DIM // 16, dtype=torch.uint8, device="cuda" + ) + quant = torch.ones(3, dtype=torch.float32, device="cuda") + values = [] + for role in (0, 1): + x = torch.randn( + num_slots, NUM_KV_HEADS * HEAD_DIM, device="cuda", generator=g, dtype=torch.float32 + ).to(torch.bfloat16) + quant[role + 1] = (448.0 * 6.0) / x.abs().max().float().item() + values.append(x) + assert fused_write_layer_caches_nvfp4( + data_pool[:, 0], + data_pool[:, 1], + scale_pool[:, 0], + scale_pool[:, 1], + None, + torch.arange(num_slots, device="cuda", dtype=torch.int32), + values[0], + values[1], + None, + quant, + ) + q = torch.randn(total_q, NUM_HEADS, HEAD_DIM, device="cuda", generator=g).to( + torch.float8_e4m3fn + ) + topk = torch.full((NUM_KV_HEADS, total_q, TOPK), -1, dtype=torch.int32, device="cuda") + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + nblocks = (token_kv_len(seq_lens[req], decode_query_len, offset) + PAGE - 1) // PAGE + for head in range(NUM_KV_HEADS): + if identity: + assert nblocks <= TOPK, "an identity step has every request within 16 blocks" + chosen = torch.arange(nblocks, device="cuda") + else: + chosen = torch.randperm(nblocks, device="cuda", generator=g)[:TOPK].sort().values + topk[head, tok, : chosen.numel()] = chosen.to(torch.int32) + return SimpleNamespace( + q=q, + k_paged=data_pool[:, 0], + v_paged=data_pool[:, 1], + k_block_scale=scale_pool[:, 0], + v_block_scale=scale_pool[:, 1], + k_global_scale=(1.0 / quant[1:2]).contiguous(), + v_global_scale=(1.0 / quant[2:3]).contiguous(), + topk=topk, + block_table=block_table, + seq_lens=torch.tensor(list(seq_lens), dtype=torch.int32, device="cuda"), + decode_query_len=decode_query_len, + sm_scale=HEAD_DIM**-0.5, + source=values, + quant=quant, + num_slots=num_slots, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_gqamix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_gqamix.py new file mode 100644 index 000000000000..e557cc5b0cee --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_gqamix.py @@ -0,0 +1,204 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GQAMIX: a mixed step's sparse generation rows (MsaDecodeFmha) through ops/gqamix_sparse_decode. + +The copy must be the in-tree kernel module except for the two decode kernels' specialization (no row count, no slice +alignment) and the one import it cannot make relatively, and must write the in-tree kernel's output bit for bit on a +mixed step's [num_ctx:] slices, here at the TP2 per-rank geometry (32 query heads over 2 KV heads, each KV head with +its own selection). The shared MsaDecodeFmha hook must change nothing while it is unset. This target registers the hook +from the MSA call over the live rows (``_msa_attention_live``), which PrefillTarget runs eagerly or, under a breakable +prefill graph, inside its eager attention boundary. +""" + +import inspect +import os +import subprocess +import sys +import types +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.triton_sparse_decode as in_tree +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as m, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops import ( + gqamix_sparse_decode as copy, +) +from tensorrt_llm._torch.attention.backends.fmha.msa_decode import MsaDecodeFmha + +HEADS = 32 # query heads per rank at tp2 +KV_HEADS = 2 # KV heads per rank at tp2 +D, PAGE, TOPK, QLEN, SF_VEC = 128, 128, 16, 4, 16 + +_JIT_OLD = '@triton.jit(do_not_specialize=["decode_query_len"])' +_JIT_NEW = ( + "@triton.jit(\n" + ' do_not_specialize=["decode_query_len", "total_q"],\n' + ' do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"],\n' + ")" +) +_IMPORT_OLD = "from .msa_utils import check_decode_span_shape" +_IMPORT_NEW = ( + "from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import (\n" + " check_decode_span_shape,\n" + ")" +) + + +def _code_lines(text: str) -> list: + return [line for line in text.splitlines() if line.strip()] + + +def test_the_copy_is_the_in_tree_module_but_for_its_specialization() -> None: + """Line for line (blank lines aside: the import sorts into a different block) the in-tree module.""" + ours = Path(copy.__file__).read_text() + theirs = Path(in_tree.__file__).read_text() + assert ( + ours.count(_JIT_NEW) == 2 + and ours.count(_IMPORT_NEW) == 1 + and theirs.count(_IMPORT_OLD) == 1 + ) + ours = ours.replace(_JIT_NEW, _JIT_OLD).replace(_IMPORT_NEW, "") + assert _code_lines(ours) == _code_lines(theirs.replace(_IMPORT_OLD, "")), ( + "ops/gqamix_sparse_decode.py must stay a copy of the in-tree triton_sparse_decode.py: re-copy it" + ) + + +def test_register_sets_the_hook_once_and_only_where_it_exists() -> None: + lib = types.SimpleNamespace(sparse_decode_hook=None) + other_lib = types.SimpleNamespace() # an FMHA library without the hook + manager = types.SimpleNamespace(fmha_libs=[other_lib, lib]) + m._gqamix_register(types.SimpleNamespace(_fmha_manager=manager)) + assert lib.sparse_decode_hook is m._gqamix_decode + assert not hasattr(other_lib, "sparse_decode_hook") + mine = object() + lib.sparse_decode_hook = mine + m._gqamix_register(types.SimpleNamespace(_fmha_manager=manager)) + assert lib.sparse_decode_hook is mine + m._gqamix_register(types.SimpleNamespace()) # a backend without an FMHA manager + + +def test_the_kill_switch_leaves_the_pre_hook_decode() -> None: + """TRTLLM_M31_DISABLE_GQAMIX=1: GQAMIX is off, so no layer registers the hook and MsaDecodeFmha runs the in-tree + decode (whose output the copy matches bit for bit below).""" + target = "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2" + probe = f"import {target}.modeling as m; print(m.GQAMIX)" + env = dict(os.environ, TRTLLM_M31_DISABLE_GQAMIX="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False" + live = inspect.getsource(m._msa_attention_live) + assert "if GQAMIX and sparse:\n _gqamix_register(core.attn[i])" in live + assert live.count("_gqamix_register(") == 1 + # Every PrefillTarget MSA call goes through it (eager, or inside the breakable graph's eager boundary). + source = inspect.getsource(m.PrefillTarget) + assert "_gqamix_register(" not in source + assert "_msa_attention_live(" in source + assert "_msa_attention_live(" in inspect.getsource(m._prefill_attention_boundary) + + +class _Attn: + is_mla_enable = False + kv_lora_rank = None + v_head_dim = None + head_dim = D + num_heads = HEADS + layer_idx = 3 + q_scaling = 1.0 + + +def _params(attn, num_tokens): + return types.SimpleNamespace( + attn=attn, + num_tokens=num_tokens, + meta=types.SimpleNamespace(kv_cache_manager=object()), + fwd=types.SimpleNamespace(kv_scale_quant_orig=None), + query_input=torch.zeros(num_tokens, HEADS * D), + output=torch.zeros(num_tokens, HEADS * D), + token_offset=2, + input_seq_length=QLEN, + ) + + +@pytest.mark.parametrize("hooked", [False, True], ids=["unset", "registered"]) +def test_the_shared_decode_hook_is_inert_unset(monkeypatch, hooked) -> None: + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import msa_utils + + calls = [] + monkeypatch.setattr(msa_utils, "msa_paged_kv", lambda manager, layer: ("k", "v")) + monkeypatch.setattr( + in_tree, "minimax_m3_sparse_attn_decode", lambda *a, **k: calls.append(("in_tree", a, k)) + ) + attn = _Attn() + fmha = MsaDecodeFmha(attn) + assert fmha.sparse_decode_hook is None + if hooked: + fmha.sparse_decode_hook = lambda *a, **k: calls.append(("hook", a, k)) + indexes = torch.zeros(10, 1, TOPK, dtype=torch.int32) + fmha._run_sparse(_params(attn, QLEN), indexes, "bt", "sl") + assert [c[0] for c in calls] == ["hook" if hooked else "in_tree"] + _, args, kwargs = calls[0] + assert args[1:3] == ("k", "v") and args[4:] == ("bt", "sl") + assert args[3].shape == (1, QLEN, TOPK) + assert kwargs["decode_query_len"] == QLEN and kwargs["sm_scale"] == D**-0.5 + + +def _mixed_step(nreq, seq_len, row_first, seed): + """A mixed step's generation rows: NVFP4 caches, FP8 q, and block-table / length slices past row_first.""" + g = torch.Generator(device="cuda").manual_seed(seed) + nb = seq_len // PAGE + pages = (nreq + row_first) * nb + block_table = torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(-1, nb) + seq_lens = torch.full((nreq + row_first,), seq_len, device="cuda", dtype=torch.int32) + total_q = nreq * QLEN + q = torch.randn(total_q, HEADS, D, device="cuda", generator=g).to(torch.float8_e4m3fn) + kv = [ + torch.randint( + 0, 256, (pages, KV_HEADS, PAGE, D // 2), device="cuda", generator=g, dtype=torch.uint8 + ) + for _ in "kv" + ] + sc = [ + (torch.rand(pages, KV_HEADS, PAGE, D // SF_VEC, device="cuda", generator=g) + 0.5).to( + torch.float8_e4m3fn + ) + for _ in "kv" + ] + # Head-major [KV_HEADS, total_q, TOPK]: each KV head selects its own blocks. + topk = torch.stack( + [ + torch.stack( + [torch.randperm(nb, device="cuda", generator=g)[:TOPK] for _ in range(total_q)] + ) + for _ in range(KV_HEADS) + ] + ) + args = (q, kv[0], kv[1], topk.to(torch.int32).contiguous()) + args += (block_table[row_first:], seq_lens[row_first:]) + kwargs = dict( + sm_scale=D**-0.5, + decode_query_len=QLEN, + k_block_scale=sc[0], + v_block_scale=sc[1], + k_global_scale=torch.tensor([0.02], device="cuda"), + v_global_scale=torch.tensor([0.03], device="cuda"), + ) + return args, kwargs, total_q + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA device") +@pytest.mark.parametrize("nreq,row_first", [(1, 0), (1, 1), (3, 2), (5, 3), (8, 1)]) +@pytest.mark.parametrize("seq_len", [16384, 131072]) +def test_the_copy_writes_the_in_tree_output_on_a_mixed_step(nreq, row_first, seq_len) -> None: + args, kwargs, total_q = _mixed_step(nreq, seq_len, row_first, 31 * nreq + row_first) + ref = torch.empty(total_q, HEADS, D, device="cuda", dtype=torch.bfloat16) + out = torch.empty_like(ref) + in_tree.minimax_m3_sparse_attn_decode(*args, output=ref, **kwargs) + copy.minimax_m3_sparse_attn_decode(*args, output=out, **kwargs) + torch.cuda.synchronize() + assert torch.isfinite(ref.float()).all() + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_lc_tp2_default.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_lc_tp2_default.py new file mode 100644 index 000000000000..3ff963a85550 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_lc_tp2_default.py @@ -0,0 +1,44 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""LC_TP2 (the tp2 baseline): the long-context rows run the Triton sparse decode; the C++ +``sparse_decode_lc`` route is off by default, ``TRTLLM_M31_ENABLE_LC_TP2=1`` opts in and +``TRTLLM_M31_DISABLE_LC_TP2=1`` always wins. The switch sets are patched and the kernel module re-imported (read at +import); no GPU needed.""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + triton_sparse_decode, +) + +pytestmark = pytest.mark.cpu_only + + +def test_lc_tp2_is_off_by_default() -> None: + assert triton_sparse_decode.LC_TP2 is False + assert triton_sparse_decode.LC_TRITON_PATH == "m3_sparse_decode_lc_tp2_triton" + + +@pytest.mark.parametrize( + "enabled, disabled, expected", + [ + (frozenset(), frozenset(), False), + (frozenset({"LC_TP2"}), frozenset(), True), + (frozenset({"LC_TP2"}), frozenset({"LC_TP2"}), False), + (frozenset(), frozenset({"LC_TP2"}), False), + ], + ids=["default", "enabled", "enabled_and_killed", "killed"], +) +def test_lc_tp2_switch(monkeypatch, enabled, disabled, expected) -> None: + monkeypatch.setattr(m31_switches, "_ENABLED", enabled) + monkeypatch.setattr(m31_switches, "_DISABLED", disabled) + try: + assert importlib.reload(triton_sparse_decode).LC_TP2 is expected + finally: + monkeypatch.undo() + importlib.reload(triton_sparse_decode) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_tp2wide_intree_scorer.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_tp2wide_intree_scorer.py new file mode 100644 index 000000000000..939cd27c9de3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m31_tp2wide_intree_scorer.py @@ -0,0 +1,254 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2 WIDE in-tree scorer (``modeling.INDEX_SCORE_INTREE``) of the MiniMax-M3.1 sm_107 tp2 target: from +``INDEX_SCORE_INTREE_MIN_REQUESTS`` requests on, at steps of ``TP2_WIDE_MIN_ROWS`` rows or more, the sparse decode +scores with the in-tree CuTe DSL scorer (``cute_dsl_minimax_m3_index_decode_score``, its 256-CTA split) instead of the +split-512 ``m3_index_score``. The claim is the same scores bit for bit (measured on VR). + +GPU (SM107): both scorers on the dispatch's argument layouts (the [heads, blocks, tokens] selector buffer passed +transposed, the index-K cache squeezed from [pages, 1, page, dim]) at 2 index heads, qlen 8, 5 / 8 / 16 / 24 requests of +mixed lengths (tile / page edges, past 256 pages, 16K), seeds 0..3, FP8 q / k; and the same at 2^+-12 scaled operands +that saturate at the FP8 max and underflow into the subnormals. The whole backing buffer (``-inf`` padding included) is +compared as int32. + +CPU: the dispatch constants, the dispatch condition (evaluated from modeling.py's source: no step of <= 32 rows and no +step of < 5 requests takes the in-tree scorer) and the kill switch ``TRTLLM_M31_DISABLE_TP2WIDE_INTREE_SCORER=1`` +(evaluated from source against a private copy of m31_switches, as the S4-logits default test does). +""" + +import ast +import importlib.util +import os +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +_DIR = Path(_family.__file__).resolve().parent / "minimax_m31_nvfp4__sm_107__tp2" +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + +_sm107 = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +HEADS = 2 # index heads per rank at TP2 (4 index heads over 2 ranks) +HEAD_DIM = 128 # sparse_index_dim +QLEN = 8 # K7 verify rows +REQUESTS = (5, 8, 16, 24) +SEEDS = (0, 1, 2, 3) +FP8_MAX = 448.0 +# Per-request lengths, rotated by the seed: the shortest K7 request (qlen), one past it, the page / tile edges, 255..257 +# pages around the in-tree split (its CTAs loop past 256 pages), 16K, and a 4-block mid length. +LENGTHS = (8, 129, 16384, 127, 128, 9, 32767, 32768, 32769, 255, 256, 257, 512, 16383, 16385, 4097) + + +def _lengths(requests: int, seed: int): + return tuple(LENGTHS[(seed * 5 + i) % len(LENGTHS)] for i in range(requests)) + + +def _operands(seq_lens, seed: int, stress: bool): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_index_score import ( # noqa: E501 + PAGE, + ) + + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + tokens = batch * QLEN + max_blocks = (max(seq_lens) + PAGE - 1) // PAGE + stride = ((max_blocks + 15) // 16) * 16 + pages = batch * max_blocks + block_table = ( + torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(batch, max_blocks) + ) + + def fp8(shape): + x = torch.randn(shape, device="cuda", generator=g) + if stress: + # 2^+-12 per element: past the FP8 max (saturated to +-448 before the cast) and below its min normal 2^-6. + e = torch.randint(-12, 13, shape, device="cuda", generator=g).float() + x = (x * torch.exp2(e)).clamp(-FP8_MAX, FP8_MAX) + return x.to(torch.float8_e4m3fn) + + # The dispatch's layouts: idx_q viewed from the projection's [tokens, heads * dim], index-K [pages, 1, page, dim]. + idx_q = fp8((tokens, HEADS * HEAD_DIM)) + idx_k_cache = fp8((pages, 1, PAGE, HEAD_DIM)) + seq = torch.tensor(seq_lens, device="cuda", dtype=torch.int32) + return idx_q, idx_k_cache, block_table, seq, stride + + +def _max_score(stride: int, tokens: int): + """The selector buffer [heads, blocks, tokens]; the dispatch passes ``max_score.transpose(1, 2)``.""" + return torch.full((HEADS, stride, tokens), -float("inf"), device="cuda", dtype=torch.float32) + + +def _check(seq_lens, seed: int, stress: bool) -> None: + import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_index_score import ( # noqa: E501 + SPLIT_K, + m3_index_score, + ) + from tensorrt_llm._torch.custom_ops.cute_dsl_custom_ops import ( + CuteDSLMiniMaxM3IndexDecodeScoreRunner as in_tree, + ) + + assert SPLIT_K == 512 and in_tree.SPLIT_K == 256 + assert in_tree.is_supported( + q_dtype=torch.float8_e4m3fn, + num_heads=HEADS, + head_dim=HEAD_DIM, + page_size=in_tree.SUPPORTED_PAGE_SIZE, + max_decode_query_len=QLEN, + ) + idx_q, idx_k_cache, block_table, seq_lens_t, stride = _operands(seq_lens, seed, stress) + num_tokens = idx_q.shape[0] + assert in_tree.SUPPORTED_PAGE_SIZE == idx_k_cache.shape[2] + idx_q_view = idx_q.view(num_tokens, HEADS, HEAD_DIM) + base, wide = _max_score(stride, num_tokens), _max_score(stride, num_tokens) + # The split-512 branch (early = rows <= INDEX_SCORE_EARLY_MAX_REQUESTS = 2: False from 5 requests on). + m3_index_score( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens_t, + base.transpose(1, 2), + QLEN, + SPLIT_K, + False, + ) + # The TP2 WIDE branch. + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens_t, + wide.transpose(1, 2), + QLEN, + ) + torch.cuda.synchronize() + if not stress: + ref = m3_index_score.reference( + idx_q_view, + idx_k_cache.squeeze(1), + block_table, + seq_lens_t, + base.transpose(1, 2), + QLEN, + SPLIT_K, + False, + ) + torch.testing.assert_close(wide.transpose(1, 2), ref, rtol=1e-5, atol=1e-4) + diff = base.view(torch.int32) != wide.view(torch.int32) + assert not bool(diff.any()), ( + f"{int(diff.sum())} score words differ from split 512 (seq_lens={seq_lens}, seed={seed}, stress={stress}); " + f"first at [head, block, token] {diff.nonzero()[0].tolist()}" + ) + assert bool(torch.isfinite(wide).any()), "nothing was scored" + + +@_sm107 +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("requests", REQUESTS) +def test_in_tree_scorer_is_split_512_bitwise(requests, seed) -> None: + _check(_lengths(requests, seed), seed, stress=False) + + +@_sm107 +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("requests", REQUESTS) +def test_in_tree_scorer_is_split_512_bitwise_at_fp8_range_edges(requests, seed) -> None: + _check(_lengths(requests, seed), 100 + seed, stress=True) + + +# --------------------------------------------------------------------------------------------- CPU: the dispatch + + +def _tree(): + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _module_assign(tree, name: str) -> ast.Assign: + found = [ + node + for node in tree.body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id == name + ] + assert len(found) == 1, name + return found[0] + + +def _constant(name: str) -> int: + return ast.literal_eval(_module_assign(_tree(), name).value) + + +def _dispatch_condition(): + """The ``elif`` test that sends a sparse decode step to the in-tree scorer.""" + tests = [ + node.test + for node in ast.walk(_tree()) + if isinstance(node, ast.If) + and isinstance(node.test, ast.BoolOp) + and isinstance(node.test.op, ast.And) + and any(isinstance(v, ast.Name) and v.id == "INDEX_SCORE_INTREE" for v in node.test.values) + ] + assert len(tests) == 1 + return compile(ast.Expression(tests[0]), str(_MODELING), "eval") + + +@pytest.mark.cpu_only +def test_dispatch_constants(): + assert _constant("INDEX_SCORE_INTREE_MIN_REQUESTS") == 5 + assert _constant("TP2_WIDE_MIN_ROWS") == 33 + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("on", [True, False], ids=["on", "killed"]) +def test_no_step_of_32_rows_or_fewer_takes_the_in_tree_scorer(on): + condition = _dispatch_condition() + names = { + "INDEX_SCORE_INTREE": on, + "INDEX_SCORE_INTREE_MIN_REQUESTS": _constant("INDEX_SCORE_INTREE_MIN_REQUESTS"), + "TP2_WIDE_MIN_ROWS": _constant("TP2_WIDE_MIN_ROWS"), + } + taken = set() + for rows in range(1, 65): + for qlen in range(1, 9): + num_tokens = rows * qlen + if eval(condition, {}, dict(names, rows=rows, num_tokens=num_tokens, qlen=qlen)): + assert on and num_tokens > 32 and rows >= 5, (rows, qlen) + taken.add((rows, qlen)) + else: + assert not on or num_tokens <= 32 or rows < 5, (rows, qlen) + if on: # the K7 cells of the MB: 5 / 8 / 16 / 24 requests at qlen 8; 4 requests (32 rows) keep split 512 + assert {(5, 8), (8, 8), (16, 8), (24, 8)} <= taken and (4, 8) not in taken + + +def _intree_on(monkeypatch, env) -> bool: + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_intree_{id(env)}", _SWITCHES) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + statement = _module_assign(_tree(), "INDEX_SCORE_INTREE") + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module([statement], []), str(_MODELING), "exec"), namespace) + return namespace["INDEX_SCORE_INTREE"] + + +@pytest.mark.cpu_only +def test_on_by_default(monkeypatch): + assert _intree_on(monkeypatch, {}) is True + + +@pytest.mark.cpu_only +def test_the_kill_switch_turns_it_off(monkeypatch): + assert _intree_on(monkeypatch, {"TRTLLM_M31_DISABLE_TP2WIDE_INTREE_SCORER": "1"}) is False diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py new file mode 100644 index 000000000000..72cff7fe324a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_fp8_indexer_qk_norm_rope entry of the MiniMax-M3.1 sm_107 tp2 target: index-Q out and index-K in +the cache against the in-place fused norm + RoPE followed by the E4M3 cast, byte for byte up to the 0.1% budget the +in-tree test of the kernel allows. The entry's cells (one index head, its TP4 lineage) and the TP2 calls (two index +heads per rank, the M3.1 RoPE base 1e7: the unfused index projection of a prefill or mixed step).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_fp8_indexer_qk_norm_rope import ( # noqa: E501 + m3_fp8_indexer_qk_norm_rope as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +DIM, ROTARY, EPS, THETA, PAGE = 128, 64, 1e-6, 5e6, 128 +# The TP2 calls: two index heads per rank, the M3.1 checkpoint's RoPE base. +TP2_INDEX_HEADS, M31_THETA = 2, 1e7 + + +def _bytes_close(a: torch.Tensor, b: torch.Tensor) -> None: + assert a.shape == b.shape and a.dtype == b.dtype == torch.float8_e4m3fn + differ = int((a.view(torch.uint8) != b.view(torch.uint8)).sum()) + assert differ <= (a.numel() - 1) // 1000, f"{differ} of {a.numel()} E4M3 bytes differ" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + _check(cell.spec["m"], cell.spec["heads"], THETA, zlib.crc32(cell.why.encode()) % 2**31) + + +@pytest.mark.parametrize("m", [1, 8, 32, 2048]) +def test_tp2_index_heads(m) -> None: + _check(m, TP2_INDEX_HEADS, M31_THETA, 1000 + m) + + +def _check(m: int, heads: int, theta: float, seed: int) -> None: + g = torch.Generator(device="cuda").manual_seed(seed) + qk = torch.randn(m, (heads + 1) * DIM, generator=g, device="cuda").to(torch.bfloat16) + qw = (torch.randn(DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + kw = (torch.randn(DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + pos = torch.randint(0, 200000, (m,), generator=g, device="cuda", dtype=torch.int32) + pages = (m + PAGE - 1) // PAGE + 1 + cache = torch.zeros(pages, 1, PAGE, DIM, dtype=torch.float8_e4m3fn, device="cuda") + slots = torch.randperm(pages * PAGE, generator=g, device="cuda")[:m].to(torch.int32) + before = qk.clone() + args = (qk, cache, slots, heads, DIM, ROTARY, EPS, qw, kw, theta, pos) + with validating(op): + q = op(*args) + assert torch.equal(qk, before), "the BF16 input is left untouched" + q_ref, k_ref = op.reference(*args) + _bytes_close(q.reshape(m, heads, DIM), q_ref) + flat = cache.view(-1, DIM) + _bytes_close(flat[slots.long()], k_ref) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_fp8_saturate.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_fp8_saturate.py new file mode 100644 index 000000000000..f58dc5f3f0ce --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_fp8_saturate.py @@ -0,0 +1,102 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""paged_cache.fp8_e4m3_saturate (FP8_SATURATE): the E4M3 cache writes and the transient FP8 query casts saturate to ++-448 like the CUDA writers (__NV_SATFINITE); every finite in-range value keeps the plain cast's bits, values already +in E4M3 pass through, and the kill switch is exactly the plain cast. Where torch's cast already saturates (this image) +the conversion is the cast alone, one kernel; otherwise it clamps first, the same bytes.""" + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import paged_cache + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + + +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float32]) +def test_fp8_e4m3_saturate_matches_satfinite(dtype, monkeypatch) -> None: + g = torch.Generator(device="cuda").manual_seed(0) + inside = (torch.randn(4096, generator=g, device="cuda") * 100).clamp(-448, 448).to(dtype) + plain = inside.to(torch.float8_e4m3fn) + assert torch.equal( + paged_cache.fp8_e4m3_saturate(inside).view(torch.uint8), plain.view(torch.uint8) + ) + edge = torch.tensor( + [449.0, 463.0, 464.0, 1e4, float("inf"), -500.0, -float("inf")], dtype=dtype, device="cuda" + ) + want = torch.tensor([448.0, 448.0, 448.0, 448.0, 448.0, -448.0, -448.0], device="cuda") + assert torch.equal(paged_cache.fp8_e4m3_saturate(edge).float(), want) + nan = paged_cache.fp8_e4m3_saturate(torch.tensor([float("nan")], dtype=dtype, device="cuda")) + assert torch.isnan(nan.float()).all() + already = edge.clamp(-448, 448).to(torch.float8_e4m3fn) + assert paged_cache.fp8_e4m3_saturate(already) is already + monkeypatch.setattr(paged_cache, "FP8_SATURATE", False) + killed = paged_cache.fp8_e4m3_saturate(edge) + assert torch.equal(killed.view(torch.uint8), edge.to(torch.float8_e4m3fn).view(torch.uint8)) + + +def test_cache_write_saturates() -> None: + """The cache-dtype conversion every write_kv_slots branch applies (compared as bytes: torch has no float8 + index_copy_ kernel), and an E4M3 source passing through untouched.""" + values = torch.tensor([[1.0, 500.0, -1e5, float("inf")]], dtype=torch.bfloat16, device="cuda") + out = paged_cache._to_cache_dtype(values, torch.float8_e4m3fn) + assert out.dtype == torch.float8_e4m3fn + want = torch.tensor([[1.0, 448.0, -448.0, 448.0]], device="cuda").to(torch.float8_e4m3fn) + assert torch.equal(out.view(torch.uint8), want.view(torch.uint8)) + assert paged_cache._to_cache_dtype(out, torch.float8_e4m3fn) is out + bf = paged_cache._to_cache_dtype(values, torch.bfloat16) + assert bf.dtype == torch.bfloat16 and torch.equal(bf, values) + + +_EDGES = [ + 448.0, + 449.0, + 463.0, + 464.0, + 1e4, + float("inf"), + -448.0, + -449.0, + -464.0, + -1e4, + -float("inf"), + float("nan"), + 0.0, +] + + +def _cuda_kernels(fn) -> int: + from torch.profiler import ProfilerActivity, profile + + torch.cuda.synchronize() + with profile(activities=[ProfilerActivity.CUDA]) as prof: + fn() + torch.cuda.synchronize() + return sum(1 for e in prof.events() if e.device_type.name == "CUDA") + + +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16, torch.float32]) +@pytest.mark.parametrize("cast_saturates", [True, False], ids=["cast_only", "clamp_fallback"]) +def test_fp8_e4m3_saturate_bytes_equal_clamp_then_cast(dtype, cast_saturates, monkeypatch) -> None: + """Both paths give the bytes of clamp(+-448) followed by the cast, edge values included.""" + monkeypatch.setattr(paged_cache, "_CAST_SATURATES", cast_saturates) + x = torch.tensor(_EDGES, dtype=dtype, device="cuda") + want = x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn) + assert torch.equal(paged_cache.fp8_e4m3_saturate(x).view(torch.uint8), want.view(torch.uint8)) + + +def test_fp8_e4m3_saturate_takes_the_cast_only_path_here(monkeypatch) -> None: + """This image's torch cast saturates, on the CPU (the import-time probe) and on CUDA, so the conversion is one + kernel; with the probe failing it falls back to clamp + cast, two kernels.""" + assert paged_cache._CAST_SATURATES + x = torch.tensor(_EDGES, dtype=torch.bfloat16, device="cuda") + assert torch.equal( + x.to(torch.float8_e4m3fn).view(torch.uint8), + x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn).view(torch.uint8), + ) + assert _cuda_kernels(lambda: paged_cache.fp8_e4m3_saturate(x)) == 1 + monkeypatch.setattr(paged_cache, "_CAST_SATURATES", False) + assert _cuda_kernels(lambda: paged_cache.fp8_e4m3_saturate(x)) == 2 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_index_score.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_index_score.py new file mode 100644 index 000000000000..b1bbdc8e3161 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_index_score.py @@ -0,0 +1,163 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_index_score entry of the MiniMax-M3.1 sm_107 tp2 target: the MSA index scorer with +its pre-wait schedule, at the TP2 per-rank geometry (two index heads over one index-K head), the entry's step shapes and +the K7 verify rows (decode query length 8). + +Every cell is held to the fp32 reference (a few fp32 ulps: the kernel sums the 128 exact products in its k-split order) +and to the in-tree scorer (``cute_dsl_minimax_m3_index_decode_score``) bit for bit, which the pre-wait prefetch and the +early tile issue must not change. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_index_score import ( # noqa: E501 + PAGE, + SPLIT_K, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_index_score import ( # noqa: E501 + m3_index_score as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +HEADS = 2 # index heads per rank at TP2 (4 index heads over 2 ranks) +HEAD_DIM = 128 + + +def _inputs(seq_lens, qlen: int, seed: int, dtype=torch.float8_e4m3fn): + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + max_blocks = (max(seq_lens) + PAGE - 1) // PAGE + stride = ((max_blocks + 15) // 16) * 16 + pages = batch * max_blocks + block_table = ( + torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(batch, max_blocks) + ) + idx_q = torch.randn(batch * qlen, HEADS, HEAD_DIM, device="cuda", generator=g).to(dtype) + index_k = torch.randn(pages, PAGE, HEAD_DIM, device="cuda", generator=g).to(dtype) + seq = torch.tensor(seq_lens, device="cuda", dtype=torch.int32) + return idx_q, index_k, block_table, seq, stride + + +def _score(stride: int, total_q: int): + backing = torch.full( + (HEADS, stride, total_q), -float("inf"), device="cuda", dtype=torch.float32 + ) + return backing.transpose(1, 2) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + qlen = spec["qlen"] + idx_q, index_k, block_table, seq, stride = _inputs( + spec["seq_lens"], qlen, zlib.crc32(cell.why.encode()) + ) + score = _score(stride, idx_q.shape[0]) + args = (idx_q, index_k, block_table, seq, score, qlen, SPLIT_K, spec["early"]) + certifying(op) + ref = op.reference(*args) + op(*args) + stock = _score(stride, idx_q.shape[0]) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, qlen, SPLIT_K + ) + torch.cuda.synchronize() + torch.testing.assert_close(score, ref, rtol=1e-5, atol=1e-4) + assert torch.equal(score, stock), "the scores differ from the in-tree scorer" + + +# The K7 verify steps at TP2 (8 query rows per request, two index heads: 16 rows of the kernel's 32-row Q block). +K7_STEPS = ( + ( + (165521,), + True, + "one request verifying 1 + 7 rows over 165521 tokens (AgentX ISL): early issue", + ), + ((165521, 20000), True, "two requests verifying 1 + 7 rows each: early issue"), + ( + (165521, 3000, 129, 70000), + False, + "four requests verifying 1 + 7 rows each: the prefetching kernel", + ), +) + + +@pytest.mark.parametrize("seq_lens,early,why", K7_STEPS, ids=[w[:44] for *_, w in K7_STEPS]) +def test_k7_verify_rows_are_the_in_tree_scorer(seq_lens, early, why) -> None: + idx_q, index_k, block_table, seq, stride = _inputs(seq_lens, 8, zlib.crc32(why.encode())) + score, stock = _score(stride, idx_q.shape[0]), _score(stride, idx_q.shape[0]) + args = (idx_q, index_k, block_table, seq, score, 8, SPLIT_K, early) + ref = op.reference(*args) + op(*args) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, 8, SPLIT_K + ) + torch.cuda.synchronize() + torch.testing.assert_close(score, ref, rtol=1e-5, atol=1e-4) + assert torch.equal(score, stock), "the scores differ from the in-tree scorer" + + +# The 2-request decode step also runs the early-issue kernel on this target (dispatch.INDEX_SCORE_EARLY_MAX_REQUESTS). +TWO_REQUEST_EARLY = ( + ((20000, 3000), "two requests verifying 1 + 3 rows each over 20000 and 3000 tokens"), + ( + (70000, 129), + "two requests: one past 512 blocks (its CTAs loop), one whose early tiles are drained", + ), +) + + +def test_the_two_request_step_takes_the_early_kernel() -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling as dispatch, + ) + + assert dispatch.INDEX_SCORE_EARLY_MAX_REQUESTS == 2 + + +@pytest.mark.parametrize( + "seq_lens,why", TWO_REQUEST_EARLY, ids=[w[:44] for _, w in TWO_REQUEST_EARLY] +) +def test_two_request_early_issue_is_the_in_tree_scorer(seq_lens, why) -> None: + idx_q, index_k, block_table, seq, stride = _inputs(seq_lens, 4, zlib.crc32(why.encode())) + score, stock = _score(stride, idx_q.shape[0]), _score(stride, idx_q.shape[0]) + args = (idx_q, index_k, block_table, seq, score, 4, SPLIT_K, True) + ref = op.reference(*args) + op(*args) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, 4, SPLIT_K + ) + torch.cuda.synchronize() + torch.testing.assert_close(score, ref, rtol=1e-5, atol=1e-4) + assert torch.equal(score, stock), "the scores differ from the in-tree scorer" + + +@pytest.mark.parametrize("early", [True, False], ids=["early", "prefetch"]) +def test_bf16_index_is_the_in_tree_scorer(early) -> None: + idx_q, index_k, block_table, seq, stride = _inputs((5000,), 4, 11, dtype=torch.bfloat16) + score, stock = _score(stride, 4), _score(stride, 4) + op(idx_q, index_k, block_table, seq, score, 4, SPLIT_K, early) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, 4, SPLIT_K + ) + assert torch.equal(score, stock) + + +def test_guard_refuses_a_request_past_its_block_table() -> None: + idx_q, index_k, block_table, seq, stride = _inputs((1000,), 1, 5) + seq = seq + 4 * PAGE + with pytest.raises(AssertionError, match="past its block-table row"): + with validating(op): + op(idx_q, index_k, block_table, seq, _score(stride + 16, 1), 1, SPLIT_K, True) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_index_score_t2sc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_index_score_t2sc.py new file mode 100644 index 000000000000..dd09b4947c9c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_index_score_t2sc.py @@ -0,0 +1,136 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_index_score_t2sc entry (T2SC, dispatch.INDEX_SCORE_T2SC) of the MiniMax-M3.1 sm_107 +tp2 target: the TP2 / TP8 MSA index scorer with its pre-wait schedule. The entry's cells (1 index head, the TP4 lineage +of this copy) and every cell again at the TP2 per-rank geometry the target runs it at (2 index heads), plus the K7 +verify rows (decode query length 8) at TP2. + +Every cell is held to the fp32 reference (a few fp32 ulps: the kernel sums the 128 exact products in its k-split order) +and to the in-tree scorer (``cute_dsl_minimax_m3_index_decode_score`` at its own 256-CTA split) bit for bit, which the +pre-wait prefetch, the early tile issue and the cell's split must not change. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_index_score_t2sc import ( # noqa: E501 + PAGE, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_index_score_t2sc import ( # noqa: E501 + m3_index_score_t2sc as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +HEADS = 2 # index heads per rank at TP2 (4 index heads over 2 ranks) +HEAD_DIM = 128 + + +def _inputs(seq_lens, qlen: int, seed: int, dtype=torch.float8_e4m3fn, heads: int = HEADS): + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + max_blocks = (max(seq_lens) + PAGE - 1) // PAGE + stride = ((max_blocks + 15) // 16) * 16 + pages = batch * max_blocks + block_table = ( + torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(batch, max_blocks) + ) + idx_q = torch.randn(batch * qlen, heads, HEAD_DIM, device="cuda", generator=g).to(dtype) + index_k = torch.randn(pages, PAGE, HEAD_DIM, device="cuda", generator=g).to(dtype) + seq = torch.tensor(seq_lens, device="cuda", dtype=torch.int32) + return idx_q, index_k, block_table, seq, stride + + +def _score(stride: int, total_q: int, heads: int = HEADS): + backing = torch.full( + (heads, stride, total_q), -float("inf"), device="cuda", dtype=torch.float32 + ) + return backing.transpose(1, 2) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + certifying(op) + _check(cell.spec, cell.spec["heads"], zlib.crc32(cell.why.encode())) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_cells_at_tp2_index_heads(cell) -> None: + """Every cell's step and schedule at the two index heads per rank the TP2 target runs.""" + _check(cell.spec, HEADS, zlib.crc32(cell.why.encode()) + 1) + + +# The target's schedule (modeling.INDEX_SCORE_T2SC_SPLIT_K): 1272 CTAs at one request, 768 per request at two. +_K7 = ( + dict(seq_lens=(165521,), qlen=8, split_k=1272, early=True), + dict(seq_lens=(129,), qlen=8, split_k=1272, early=True), + dict(seq_lens=(165521, 20000), qlen=8, split_k=768, early=True), +) + + +@pytest.mark.parametrize("spec", _K7, ids=[f"{len(s['seq_lens'])}req_{s['split_k']}" for s in _K7]) +def test_k7_verify_rows_at_tp2(spec) -> None: + _check(spec, HEADS, 77 + len(spec["seq_lens"])) + + +def _check(spec, heads: int, seed: int) -> None: + qlen = spec["qlen"] + idx_q, index_k, block_table, seq, stride = _inputs(spec["seq_lens"], qlen, seed, heads=heads) + score = _score(stride, idx_q.shape[0], heads) + args = (idx_q, index_k, block_table, seq, score, qlen, spec["split_k"], spec["early"]) + ref = op.reference(*args) + op(*args) + stock = _score(stride, idx_q.shape[0], heads) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, qlen + ) + torch.cuda.synchronize() + torch.testing.assert_close(score, ref, rtol=1e-5, atol=1e-4) + assert torch.equal(score, stock), "the scores differ from the in-tree scorer" + + +@pytest.mark.parametrize("early", [True, False], ids=["early", "prefetch"]) +def test_bf16_index_is_the_in_tree_scorer(early) -> None: + idx_q, index_k, block_table, seq, stride = _inputs((5000,), 4, 11, dtype=torch.bfloat16) + score, stock = _score(stride, 4), _score(stride, 4) + op(idx_q, index_k, block_table, seq, score, 4, 1536, early) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, 4 + ) + assert torch.equal(score, stock) + + +@pytest.mark.parametrize("qlen", [1, 2, 3, 4]) +def test_prestaged_blocks_at_the_stable_edge_are_the_in_tree_scorer(qlen) -> None: + """Lengths around the edge between pre-staged (all-old) and post-wait first blocks, 1 and 2 requests.""" + for k in (1, 2, 1294): + for length in (k * PAGE - qlen - 1, k * PAGE - qlen, k * PAGE - qlen + 1, k * PAGE + 1): + for seq_lens, split in ( + ((length,), 1536), + ((length,), 1272), + ((length, max(qlen, length // 3)), 768), + ): + idx_q, index_k, block_table, seq, stride = _inputs(seq_lens, qlen, 31 + length) + score, stock = _score(stride, idx_q.shape[0]), _score(stride, idx_q.shape[0]) + op(idx_q, index_k, block_table, seq, score, 4, split, True) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, qlen + ) + assert torch.equal(score, stock), (qlen, seq_lens) + + +def test_guard_refuses_a_request_past_its_block_table() -> None: + idx_q, index_k, block_table, seq, stride = _inputs((1000,), 1, 5) + seq = seq + 4 * PAGE + with pytest.raises(AssertionError, match="past its block-table row"): + with validating(op): + op(idx_q, index_k, block_table, seq, _score(stride + 16, 1), 1, 1536, True) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_msa_attention.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_msa_attention.py new file mode 100644 index 000000000000..0bb1b4cfca2a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_msa_attention.py @@ -0,0 +1,232 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_msa_attention entry of the MiniMax-M3.1 sm_107 tp2 target: one layer's MSA backend step at the +TP2 per-rank geometry (32 query heads over 2 KV heads, 2 index heads) against its float64 reference. + +The entry's own ``reference`` mirrors the rank's grouped KV heads (ported from the M3 sm_107 tp2 entry: query head h +reads KV head h // 16, each KV head selecting its own top-k blocks from its own index head), and it is the reference +every cell is held to (the certifier requires the entry's own reference to run). + +Every cell runs on the runtime's own objects (``_m3_attn_runtime_tp2``): a hybrid NVFP4 ``MiniMaxM3KVCacheManagerV2`` +with the requests allocated in it, ``MiniMaxM3MsaSparseAttentionMetadata`` filled by its ``prepare()``, and the layer's +``MiniMaxM3MsaSparseAttention`` built as the target builds it. Earlier rows (and, on a sparse layer, the step's own +rows, which the target's producer writes before the call) go into the cache through the in-tree scatters, at the +slots the manager assigned. The sparse layers' index-K rows are planted so every block's score is 1/8 or more apart +from every other's: the top-k selection the reference recomputes is then unambiguous. + +The cells are the target's calls (``_attention_core`` in modeling.py): a dense layer with live bf16 K / V in a +prefill chunk and live FP8 K / V in a decode step (the dense layers sit in the hybrid cache's FP8 half), a sparse +layer whose producer already wrote the caches (``k = v = None``) in a prefill chunk beyond 16 blocks and in a mixed +step, both with the NVFP4 scales the target passes. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_msa_attention import ( # noqa: E501 + m3_msa_attention as op, +) + +__extra_import_path__ = ["../../../..", "."] +import _m3_attn_runtime_tp2 as rt # noqa: E402 — needs the path declared above +from _validating import ULP, assert_within_ulp, certifying # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +# Bands, in BF16 ulps (2^-8) of each token's [heads x 128] row's largest magnitude (element) and of the relative RMS, +# per kernel the step reaches. The reference reads the same quantized cache rows (and casts Q to E4M3 where the FP8 +# FMHA does), so what is left is each FMHA's own arithmetic, measured on GB300 over these cells (max element / RMS): +# * fmha_sm100 over the FP8 pool (dense context rows): 10.1 / 6.1. It runs E4M3 x E4M3 and its softmax +# probabilities are rounded before the PV product. +# * trtllm-gen over the FP8 pool (dense generation rows): 7.0 / 6.1. It rounds the probabilities to E4M3 (scaled by +# 448 against the running maximum): emulating that brings a single-tile request to 0.6 RMS. +# * the NVFP4 CSR prefill (sparse context rows): 34.4 / 11.8 at 4096 rows, 26.8 / 9.1 at 512. It stages the +# reconstructed K / V in E4M3 (the in-tree A/B test, test_minimax_m3_sparse_attn_decode, notes it and holds it +# to 3e-2). +# * the Triton NVFP4 sparse decode (sparse generation rows): 1.0 / 0.25 -- the band of its own entry +# (m3_sparse_decode): exact operands, FP16 probabilities, BF16 rounding. +# Each band leaves about 1.5x the measured maximum; a shifted or wrongly selected reference is far outside them. +_FP8_CONTEXT = dict(element_ulp=16.0, rms_ulp=9.0) +_FP8_DECODE = dict(element_ulp=12.0, rms_ulp=9.0) +_NVFP4_CONTEXT = dict(element_ulp=48.0, rms_ulp=16.0) +_NVFP4_DECODE = dict(element_ulp=2.0, rms_ulp=0.5) +_BANDS = { + ("dense", "context"): (_FP8_CONTEXT, None), + ("dense", "generation"): (None, _FP8_DECODE), + ("sparse", "context"): (_NVFP4_CONTEXT, None), + ("sparse", "mixed"): (_NVFP4_CONTEXT, _NVFP4_DECODE), +} + + +def _assert_in_bands(output, ref, bands, context_rows: int) -> None: + """Gate the context rows and the generation rows each on their own kernel's band.""" + context_band, decode_band = bands + if context_rows: + assert_within_ulp(output[:context_rows], ref[:context_rows], **context_band) + if context_rows < output.shape[0]: + assert_within_ulp(output[context_rows:], ref[context_rows:], **decode_band) + + +# (request rows, cached tokens per request, new tokens per request, context rows) per cell. +_SHAPES = { + ("dense", "context"): ((384,), (1024,), 1), + ("dense", "generation"): ((127, 128, 1000, 2500), (1, 1, 1, 1), 0), + ("sparse", "context"): ((0,), (4096,), 1), + ("sparse", "mixed"): ((0, 2999, 4500), (512, 1, 1), 1), +} + + +def _ulp_stats(out: torch.Tensor, ref: torch.Tensor) -> tuple[float, float]: + """The two quantities ``assert_within_ulp`` gates, in BF16 ulps.""" + o, r = out.float(), ref.float() + scale = r.abs().amax(dim=1, keepdim=True).clamp_min(1e-9) + element = ((o - r).abs() / scale).max().item() / ULP[torch.bfloat16] + rms = ((o - r).pow(2).mean().sqrt() / r.pow(2).mean().sqrt()).item() / ULP[torch.bfloat16] + return element, rms + + +def _shift(x: torch.Tensor) -> torch.Tensor: + """``x``'s rows moved down by one (byte-wise, so E4M3 rows move too).""" + return x.view(torch.uint8).roll(1, 0).view(x.dtype) if x.element_size() == 1 else x.roll(1, 0) + + +def _shape(cell): + spec = cell.spec + cached, qlens, contexts = _SHAPES[spec["layer"], spec["phase"]] + if spec["phase"] == "context": + assert sum(qlens) == spec["tokens"] + elif spec["phase"] == "generation": + assert len(qlens) == spec["reqs"] + else: + assert sum(qlens[:contexts]) == spec["tokens"] and len(qlens) - contexts == spec["reqs"] + return cached, qlens, contexts + + +def _run_dense(cell, gen, manager, backend, metadata): + cached, qlens, contexts = _shape(cell) + rids = list(range(len(qlens))) + rt.allocate(manager, rids, qlens, cached) + for rid, c in zip(rids, cached): + if c: + hist = torch.randn(2, c, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to( + torch.float8_e4m3fn + ) + rt.write_fp8_rows(manager, rt.DENSE, rid, torch.arange(c), hist[0], hist[1]) + rt.prepare(metadata, rids, qlens, cached, contexts) + total = sum(qlens) + # A prefill chunk arrives in BF16 (the hybrid cache's dense layers take FP8 Q / K / V only on decode steps). + dtype = torch.bfloat16 if contexts else torch.float8_e4m3fn + q = torch.randn(total, rt.HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to(dtype) + k = torch.randn(total, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to(dtype) + v = torch.randn(total, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to(dtype) + # The NVFP4 target passes its layer scales on every layer; an FP8 layer's write and FMHA take none. + inv = torch.tensor([1.0, 0.5, 2.0], device="cuda") + fwd = 1.0 / inv + return (q, k, v, None, inv, fwd), dict(rids=rids, cached=cached, qlens=qlens, contexts=contexts) + + +def _run_sparse(cell, gen, manager, backend, metadata): + cached, qlens, contexts = _shape(cell) + rids = list(range(len(qlens))) + rt.allocate(manager, rids, qlens, cached) + u0, u1 = rt.index_directions(gen) + lengths = [c + n for c, n in zip(cached, qlens)] + keys = [ + torch.randn(n, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda") for n in lengths + ] + values = [ + torch.randn(n, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda") for n in lengths + ] + quant = rt.nvfp4_quant(torch.cat(keys), torch.cat(values)) + for rid, n, kk, vv in zip(rids, lengths, keys, values): + # Every row, the step's own included: the producer writes a sparse layer's caches before the call. + idx_k = rt.planted_index_keys(n, u0, u1, gen) + rt.write_nvfp4_rows( + manager, + metadata, + rt.SPARSE, + rid, + torch.arange(n), + kk.to(torch.bfloat16), + vv.to(torch.bfloat16), + idx_k, + quant, + ) + rt.prepare(metadata, rids, qlens, cached, contexts) + total = sum(qlens) + q = torch.randn(total, rt.HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to( + torch.float8_e4m3fn + ) + # Query tokens alternate between the two planted directions, so neighbouring tokens select different blocks; the + # rank's index heads take alternating directions, so with more than one KV head they select different blocks too. + turn = torch.arange(total, device="cuda") % 2 + directions = torch.stack([u0, u1]) + idx_q = torch.cat([directions[(turn + h) % 2] for h in range(rt.INDEX_HEADS)], dim=1) + idx_q = idx_q.to(torch.float8_e4m3fn) + return (q, None, None, idx_q, quant, 1.0 / quant), dict( + rids=rids, cached=cached, qlens=qlens, contexts=contexts + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + sparse = spec["layer"] == "sparse" + gen = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + manager = rt.make_manager(draft=False) + try: + backend = rt.make_msa_backend(rt.SPARSE if sparse else rt.DENSE) + metadata = rt.make_msa_metadata(manager) + run = _run_sparse if sparse else _run_dense + (q, k, v, idx_q, inv, fwd), step = run(cell, gen, manager, backend, metadata) + output = torch.full( + (q.shape[0], rt.HEADS * rt.HEAD_DIM), float("nan"), dtype=torch.bfloat16, device="cuda" + ) + certifying(op) + result = op(backend, q, k, v, idx_q, metadata, output, sparse, inv, fwd) + assert result is output + ref = op.reference(backend, q, k, v, idx_q, metadata, output, sparse, inv, fwd) + assert torch.isfinite(output.float()).all(), "every live row is written" + bands = _BANDS[spec["layer"], spec["phase"]] + context_rows = sum(step["qlens"][: step["contexts"]]) + for name, rows in ( + ("context", slice(0, context_rows)), + ("generation", slice(context_rows, None)), + ): + if output[rows].numel(): + element, rms = _ulp_stats(output[rows], ref[rows]) + print( + f"\n[{cell.why}] {name} rows: max element {element:.2f} ulp, rms {rms:.3f} ulp" + ) + _assert_in_bands(output, ref, bands, context_rows) + + if k is not None: + # The call's own cache write: the live K / V rows land, as a plain E4M3 cast, at their positions. + pool = manager.get_buffers(rt.DENSE, "HND") + first = 0 + for rid, c, n in zip(step["rids"], step["cached"], step["qlens"]): + slots = rt.flat_slots(manager, rt.DENSE, rid, torch.arange(c, c + n)).long() + for role, src in ((0, k), (1, v)): + for h in range(rt.KV_HEADS): + got = pool[slots // rt.PAGE, role, h, slots % rt.PAGE] + want = src[first : first + n, h * rt.HEAD_DIM : (h + 1) * rt.HEAD_DIM] + want = want.to(torch.float8_e4m3fn) + assert torch.equal(got.view(torch.uint8), want.view(torch.uint8)) + first += n + + # The band discriminates: a reference whose query rows are shifted by one token is outside it. + shifted = op.reference(backend, _shift(q), k, v, idx_q, metadata, output, sparse, inv, fwd) + with pytest.raises(AssertionError): + _assert_in_bands(output, shifted, bands, context_rows) + if sparse: + # And the selection is load-bearing: attending every causal block instead is outside it too. + dense = op.reference(backend, q, k, v, idx_q, metadata, output, False, inv, fwd) + with pytest.raises(AssertionError): + _assert_in_bands(output, dense, bands, context_rows) + finally: + manager.shutdown() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_n_valid_contract.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_n_valid_contract.py new file mode 100644 index 000000000000..3c5b75f264ed --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_n_valid_contract.py @@ -0,0 +1,138 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The per-token valid-block count is written only before the step's first layer. + +The long-row select (``kernels/m3_select_long_dsl.py``, ``N_VALID_PREWAIT``) reads its row's entry of +``MsaMetadata.msa_n_valid_blocks`` BEFORE its PDL grid-dependency wait. That is exact only while every write to the +buffer happens before the step's first layer: + +- the allocation (``_alloc_msa_proxy_scratch``); +- the host-to-device copy in ``prepare()`` (``_build_step_plans``); +- ``on_update_kv_lens``, which the runner calls before the forward. It writes through the plain + ``<<<>>>`` launch of ``msa_kv_update_kernel`` or through torch copies, so any PDL grid of the step starts only + after it completes. + +This test scans ``tensorrt_llm/_torch`` and fails on any other write site (including writes through a local alias of +the buffer), so a new writer has to revisit the pre-wait read. A buffer handed to an opaque kernel launch is not +tracked; such a launch inside a layer must be reviewed by hand. + +CPU only: it parses the sources. +""" + +import ast +from pathlib import Path + +import pytest + +import tensorrt_llm + +pytestmark = pytest.mark.cpu_only + +_ROOT = Path(tensorrt_llm.__file__).resolve().parent / "_torch" +_NAME = "msa_n_valid_blocks" +# In-place tensor methods that write their receiver. +_MUTATORS = { + "copy_", + "fill_", + "zero_", + "index_put_", + "index_copy_", + "scatter_", + "masked_fill_", + "add_", + "sub_", +} +# Calls whose first argument is the destination they write: ``step_inputs.copy_(dst, src)`` and the H2D helpers. +_COPY_HELPERS = {"copy_", "copy_h2d_if_changed", "copy_h2d_pool_shared"} +_BACKEND = "attention/backends/sparse/minimax_m3/msa_backend.py" +# (file relative to tensorrt_llm/_torch, enclosing function) -> writes allowed there +_ALLOWED = { + (_BACKEND, "_alloc_msa_proxy_scratch"), # the allocation + (_BACKEND, "_build_step_plans"), # the H2D copy in prepare() + (_BACKEND, "on_update_kv_lens"), # the device-side correction before the forward +} + + +def _base(node): + """The innermost receiver of an attribute / subscript / call chain.""" + while isinstance(node, (ast.Subscript, ast.Attribute, ast.Call)): + if isinstance(node, ast.Attribute) and node.attr == _NAME: + return node + node = node.func if isinstance(node, ast.Call) else node.value + return node + + +def _mentions(node, aliases) -> bool: + """Whether the expression's receiver chain is ``*.msa_n_valid_blocks`` or a local alias of it.""" + b = _base(node) + if isinstance(b, ast.Attribute): + return b.attr == _NAME + return isinstance(b, ast.Name) and (b.id == _NAME or b.id in aliases) + + +def _refers(node, aliases) -> bool: + """Whether any sub-expression (e.g. a branch of a conditional) names the buffer or an alias.""" + return any( + (isinstance(n, ast.Attribute) and n.attr == _NAME) + or (isinstance(n, ast.Name) and (n.id == _NAME or n.id in aliases)) + for n in ast.walk(node) + ) + + +def _write_sites(tree: ast.AST): + sites = [] + + def visit(node, func, aliases): + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + func, aliases = node.name, set() + if isinstance(node, (ast.Assign, ast.AugAssign, ast.AnnAssign)): + targets = node.targets if isinstance(node, ast.Assign) else [node.target] + value = node.value + for t in targets: + if isinstance(t, ast.Name) and value is not None and _refers(value, aliases): + aliases.add(t.id) # a local view of the buffer + elif ( + not isinstance(t, ast.Name) + and _mentions(t, aliases) + and not isinstance(node, ast.AnnAssign) + ): + sites.append((func, node.lineno)) + elif ( + isinstance(t, ast.Attribute) + and t.attr == _NAME + and isinstance(node, ast.AnnAssign) + ): + pass # a dataclass field declaration + if isinstance(node, ast.Call): + f = node.func + if isinstance(f, ast.Attribute) and f.attr in _MUTATORS and _mentions(f.value, aliases): + sites.append((func, node.lineno)) + name = ( + f.id + if isinstance(f, ast.Name) + else (f.attr if isinstance(f, ast.Attribute) else "") + ) + if name in _COPY_HELPERS and node.args and _mentions(node.args[0], aliases): + sites.append((func, node.lineno)) + for child in ast.iter_child_nodes(node): + visit(child, func, aliases) + + visit(tree, None, set()) + return sites + + +def test_only_the_pre_forward_paths_write_the_valid_block_count() -> None: + found = set() + for path in sorted(_ROOT.rglob("*.py")): + src = path.read_text() + if _NAME not in src: + continue + rel = path.relative_to(_ROOT).as_posix() + for func, line in _write_sites(ast.parse(src)): + assert (rel, func) in _ALLOWED, ( + f"{rel}:{line} ({func}) writes msa_n_valid_blocks; the long-row select reads it before its PDL grid " + "wait (kernels/m3_select_long_dsl.py, N_VALID_PREWAIT), so a writer inside the " + "step must be ordered before the select's grid or the read moved after the wait" + ) + found.add((rel, func)) + assert found == _ALLOWED, f"expected write sites {_ALLOWED}, found {found}" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_producer.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_producer.py new file mode 100644 index 000000000000..8c35591ac197 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_producer.py @@ -0,0 +1,229 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_producer entry of the MiniMax-M3.1 sm_107 tp2 target: the pre-wait-staged NVFP4 QKV + +indexer producer at the TP2 per-rank heads (32 query, 2 KV, 2 index; packed row 39 x 128 = 4992 columns), on this +target's op (``minimax_m3_sm107_tp2_producer_pw``, the M3 tp2 target's wrapper). + +Every cell is held to the native producer (``minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert``) bit for bit on q, +index-q and every K / V / scale / index-K cache byte, and to the fp32 torch reference: the FP8 outputs within one E4M3 +step, the decoded NVFP4 K / V within one E2M1 step of their block scale. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_producer import ( # noqa: E501 + FP4_BLOCK, + HEADS_INDEX, + HEADS_KV, + HEADS_PACKED, + HEADS_Q, + PAGE, + decode_nvfp4_token, + scale_index, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_producer import ( # noqa: E501 + m3_producer as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +PAGES = 1024 +MAX_POSITIONS = 262144 +EPS = 1e-6 +KV_QUANT = (1.0, 0.37, 0.61) + + +def _caches(): + return ( + torch.zeros(PAGES, 2, HEADS_KV, PAGE, 64, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 2, HEADS_KV, PAGE, 8, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 1, PAGE, 128, dtype=torch.float8_e4m3fn, device="cuda"), + ) + + +def _inputs(tokens: int, scale: float, bad_slots: bool, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + rotary = torch.randn(MAX_POSITIONS, 2, 32, device="cuda", generator=g) + weights = [(torch.randn(128, device="cuda", generator=g) * 0.3).bfloat16() for _ in range(4)] + packed = ( + torch.randn(tokens, HEADS_PACKED * 128, device="cuda", generator=g) * scale + ).bfloat16() + slots = torch.randperm(PAGES * PAGE, device="cuda", generator=g)[:tokens].int() + if bad_slots: + slots[1] = -1 + slots[2] = PAGES * PAGE + 5 + positions = torch.randint( + 0, MAX_POSITIONS, (tokens,), device="cuda", dtype=torch.int32, generator=g + ) + quant = torch.tensor(KV_QUANT, device="cuda") + return packed, slots, quant, weights, rotary, positions + + +def _args(inputs, caches, early: bool = True): + packed, slots, quant, weights, rotary, positions = inputs + return (packed, *caches, slots, quant, EPS, *weights, rotary, positions, HEADS_Q, True, early) + + +def _native(inputs, caches): + packed, slots, quant, weights, rotary, positions = inputs + return torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed, + *caches, + slots, + quant, + HEADS_Q, + HEADS_KV, + HEADS_INDEX, + 128, + 64, + EPS, + *weights, + rotary, + positions, + ) + + +def _within_fp8(out: torch.Tensor, ref: torch.Tensor, name: str) -> None: + # One E4M3 step (3 mantissa bits) of the value, or its smallest subnormal step. + err = (out.float() - ref).abs() + assert bool((err <= ref.abs() * 2.0**-3 + 2.0**-9).all()), (name, float(err.max())) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + early = spec.get("early", True) + inputs = _inputs( + spec["tokens"], spec["scale"], spec.get("bad_slots", False), zlib.crc32(cell.why.encode()) + ) + stock_caches, caches = _caches(), _caches() + stock = _native(inputs, stock_caches) + args = _args(inputs, caches, early) + certifying(op) + ref = op.reference(*args) + out = op(*args) + torch.cuda.synchronize() + names = ("q", "index_q", "kv_data", "kv_scale", "index_k") + for name, s, o in zip(names, (*stock, *stock_caches), (*out, *caches)): + assert torch.equal(s.view(torch.uint8), o.view(torch.uint8)), ( + f"{name} differs from the native producer" + ) + + _within_fp8(out[0], ref["q"], "q") + _within_fp8(out[1], ref["index_q"], "index_q") + kv_data, kv_scale, index_k = caches + slots = inputs[1] + checked = 0 + for t in range(spec["tokens"]): + slot = int(slots[t]) + if slot < 0 or slot >= PAGES * PAGE: + continue + page, within = divmod(slot, PAGE) + _within_fp8(index_k[page, 0, within], ref["index_k"][t], f"index_k token {t}") + for (plane, name), head in ((p, h) for p in ((0, "k"), (1, "v")) for h in range(HEADS_KV)): + scales = kv_scale[page, plane, head].reshape(-1)[scale_index(within, plane).cuda()] + dq = decode_nvfp4_token(kv_data[page, plane, head, within], scales, KV_QUANT[1 + plane]) + r = ref[name][t, head].bfloat16().float() + # Round to nearest E2M1 under the block's E4M3-rounded scale s (in value units s / quant_scale): at most + # half the widest E2M1 step (one unit), plus the clipping at 6 units when s rounded down; a scale that + # underflows E4M3 zeroes the block. A slack of one E4M3 step covers the kernel's approximate reciprocals. + qs = KV_QUANT[1 + plane] + amax = r.view(-1, FP4_BLOCK).abs().amax(dim=1) + unit = (qs * amax / 6.0).to(torch.float8_e4m3fn).float() / qs + bound = torch.where( + unit > 0, 1.15 * unit + (amax - 6.0 * unit / 1.15).clamp_min(0.0), amax + ) + bound = bound.repeat_interleave(FP4_BLOCK) + 1e-12 + assert bool(((dq - r).abs() <= bound).all()), (name, t, head) + checked += 1 + assert checked >= spec["tokens"] - 2 + + +def test_early_trigger_does_not_change_the_bits() -> None: + inputs = _inputs(48, 1.0, False, 7) + a, b = _caches(), _caches() + out_a = op(*_args(inputs, a, early=True)) + out_b = op(*_args(inputs, b, early=False)) + for x, y in zip((*out_a, *a), (*out_b, *b)): + assert torch.equal(x.view(torch.uint8), y.view(torch.uint8)) + + +@pytest.mark.parametrize("tokens", [1, 4, 8, 48]) +def test_prewait_step_inputs_do_not_change_the_bits(tokens: int) -> None: + """Reading the position ids, slots and RoPE rows before the grid wait leaves every output and cache byte as is, + padded rows (slot -1) and a slot past the pool included.""" + inputs = _inputs(tokens, 1.0, tokens >= 3, 11) + a, b = _caches(), _caches() + out_a = op(*_args(inputs, a), prewait_step_inputs=True) + out_b = op(*_args(inputs, b), prewait_step_inputs=False) + for x, y in zip((*out_a, *a), (*out_b, *b)): + assert torch.equal(x.view(torch.uint8), y.view(torch.uint8)) + + +def test_guard_refuses_a_repeated_slot() -> None: + inputs = list(_inputs(4, 1.0, False, 3)) + inputs[1] = inputs[1].clone() + inputs[1][1] = inputs[1][0] + with pytest.raises(AssertionError, match="must not repeat a slot"): + with validating(op): + op(*_args(tuple(inputs), _caches())) + + +def test_op_refuses_an_operand_off_the_device() -> None: + inputs = list(_inputs(4, 1.0, False, 9)) + inputs[5] = inputs[5].cpu() # position ids on the host + with pytest.raises(RuntimeError, match="packed's CUDA device"): + op(*_args(tuple(inputs), _caches())) + + +def test_op_refuses_an_unaligned_index_k_page_stride() -> None: + """The index-K rows are written as 32-bit words: a page stride that is not a multiple of 4 is refused.""" + kv_data, kv_scale, _ = _caches() + backing = torch.zeros(PAGES * (128 * 128 + 1), dtype=torch.float8_e4m3fn, device="cuda") + index_k = backing.as_strided((PAGES, 1, PAGE, 128), (128 * 128 + 1, 128 * 128, 128, 1)) + with pytest.raises(RuntimeError, match="index_k"): + op(*_args(_inputs(4, 1.0, False, 13), (kv_data, kv_scale, index_k))) + + +def test_op_refuses_unaligned_weights_or_rope_table() -> None: + """The norm weights are read as 8-byte words and the RoPE rows as float4: offset views are refused.""" + packed, slots, quant, weights, rotary, positions = _inputs(4, 1.0, False, 17) + offset_weight = torch.zeros(129, dtype=torch.bfloat16, device="cuda")[1:] + with pytest.raises(RuntimeError, match="norm weights"): + op( + *_args( + (packed, slots, quant, [offset_weight, *weights[1:]], rotary, positions), _caches() + ) + ) + flat = torch.zeros(rotary.numel() + 1, device="cuda") + offset_rope = flat[1:].view(rotary.shape) + with pytest.raises(RuntimeError, match="rotary_cos_sin"): + op(*_args((packed, slots, quant, weights, offset_rope, positions), _caches())) + + +def test_tp2_geometry() -> None: + """The wrapper is built for the TP2 per-rank heads: 32 query, 2 KV, 2 index heads, a 39-head packed row.""" + assert (HEADS_Q, HEADS_KV, HEADS_INDEX, HEADS_PACKED) == (32, 2, 2, 39) + assert HEADS_PACKED * 128 == 4992 + + +def test_each_kv_head_lands_in_its_own_block() -> None: + """K / V of KV head 1 go to head block 1 of the HND pools and differ from head 0's (distinct packed rows).""" + inputs = _inputs(4, 1.0, False, 21) + caches = _caches() + op(*_args(inputs, caches)) + torch.cuda.synchronize() + kv_data = caches[0] + page, within = divmod(int(inputs[1][0]), PAGE) + for plane in (0, 1): + assert not torch.equal(kv_data[page, plane, 0, within], kv_data[page, plane, 1, within]) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..e1748d2dbf0b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_qk_norm_rope_fp8 entry of the MiniMax-M3.1 sm_107 tp2 target: the out-of-place FP8 form of the +fused per-head Gemma RMSNorm + partial NeoX RoPE against the in-place kernel followed by the E4M3 cast. V is a cast, so +it matches bit for bit; Q and K round once from fp32 where the composition rounds through bf16 first, so each element +is within one E4M3 step of it. The entry's cells (16 q / 1 kv, its TP4 lineage) and the TP2 calls (32 q / 2 kv per +rank: the unfused sparse projection of a prefill or mixed step).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_qk_norm_rope_fp8 import ( # noqa: E501 + m3_qk_norm_rope_fp8 as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +HEAD_DIM, ROTARY_DIM, EPS, THETA = 128, 64, 1e-6, 5e6 + + +def _args(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + q, kv = spec["q"], spec["kv"] + qkv = torch.randn(spec["m"], (q + 2 * kv) * HEAD_DIM, generator=g, device="cuda").to( + torch.bfloat16 + ) + qw = (torch.randn(HEAD_DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + kw = (torch.randn(HEAD_DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + pos = torch.randint(0, 200000, (spec["m"],), generator=g, device="cuda", dtype=torch.int32) + return ( + qkv, + q, + kv, + kv, + HEAD_DIM, + ROTARY_DIM, + EPS, + qw, + kw, + THETA, + True, + pos, + 1.0, + 0.0, + 0.0, + 1.0, + True, + True, + False, + 0, + 0, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + _check(cell.spec, zlib.crc32(cell.why.encode()) % 2**31) + + +@pytest.mark.parametrize("m", [1, 8, 32, 2048]) +def test_tp2_heads(m) -> None: + """The TP2 per-rank heads: 32 query, 2 KV.""" + _check(dict(m=m, q=32, kv=2), 2000 + m) + + +def _check(spec, seed) -> None: + args = _args(spec, seed) + before = args[0].clone() + with validating(op): + out = op(*args) + assert torch.equal(args[0], before), "the FP8 form must leave qkv untouched" + ref = op.reference(*args) + assert out.dtype == torch.float8_e4m3fn and out.shape == ref.shape + v_start = (spec["q"] + spec["kv"]) * HEAD_DIM + assert torch.equal(out[:, v_start:].view(torch.uint8), ref[:, v_start:].view(torch.uint8)) + assert _e4m3_steps(out[:, :v_start], ref[:, :v_start]) <= 1 + + +def _e4m3_steps(a, b) -> int: + """The largest distance between two E4M3 tensors in representable values (E4M3 codes order by magnitude).""" + + def signed(t): + c = t.view(torch.uint8).int() + # Sign-magnitude to a line: -0 and +0 both map to 0. + return torch.where(c >= 128, 128 - c, c) + + return int((signed(a) - signed(b)).abs().max()) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_qkv_insert.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_qkv_insert.py new file mode 100644 index 000000000000..5288c8eb66ac --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_qkv_insert.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_qkv_insert entry: the generic horizontal producer on an FP8 main cache. + +Q, index-Q and every written K / V / index-K cache row are held within one E4M3 step of the fp32 reference. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_qkv_insert import ( # noqa: E501 + m3_qkv_insert as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_qkv_insert requires a CUDA device" + +PAGES, PAGE, HEADS, KV, INDEX, D, ROTARY = 256, 128, 32, 2, 2, 128, 64 # the TP2 per-rank heads +MAX_POSITIONS = 262144 +EPS = 1e-6 + + +def _within_fp8(out: torch.Tensor, ref: torch.Tensor, name: str) -> None: + # One E4M3 step (3 mantissa bits) of the value, or its smallest subnormal step. + err = (out.float() - ref).abs() + assert bool((err <= ref.abs() * 2.0**-3 + 2.0**-9).all()), (name, float(err.max())) + + +def _args(tokens: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + packed = torch.randn( + tokens, (HEADS + 2 * KV + INDEX + 1) * D, device="cuda", generator=g + ).bfloat16() + kv = torch.zeros(PAGES, 2, KV, PAGE, D, dtype=torch.float8_e4m3fn, device="cuda") + index_k = torch.zeros(PAGES, 1, PAGE, D, dtype=torch.float8_e4m3fn, device="cuda") + slots = torch.randperm(PAGES * PAGE, device="cuda", generator=g)[:tokens].int() + weights = [(torch.randn(D, device="cuda", generator=g) * 0.3).bfloat16() for _ in range(4)] + rotary = torch.randn(MAX_POSITIONS, 2, ROTARY // 2, device="cuda", generator=g) + positions = torch.randint( + 0, MAX_POSITIONS, (tokens,), device="cuda", dtype=torch.int32, generator=g + ) + return ( + packed, + kv, + None, + index_k, + slots, + None, + HEADS, + KV, + INDEX, + D, + ROTARY, + EPS, + *weights, + rotary, + positions, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _args(cell.spec["m"], zlib.crc32(cell.why.encode())) + certifying(op) + ref = op.reference(*args) + q, index_q = op(*args) + torch.cuda.synchronize() + _within_fp8(q.view(ref["q"].shape), ref["q"], "q") + _within_fp8(index_q.view(ref["index_q"].shape), ref["index_q"], "index_q") + kv, index_k, slots = args[1], args[3], args[4] + for t in range(cell.spec["m"]): + page, within = divmod(int(slots[t]), PAGE) + _within_fp8(kv[page, 0, :, within], ref["k"][t], f"k token {t}") + _within_fp8(kv[page, 1, :, within], ref["v"][t], f"v token {t}") + _within_fp8(index_k[page, 0, within], ref["index_k"][t], f"index_k token {t}") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_blocks.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_blocks.py new file mode 100644 index 000000000000..01c73dc577c8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_blocks.py @@ -0,0 +1,45 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_select_blocks entry of the MiniMax-M3.1 tp2 target: MSA top-16 block selection with the forced +init / local blocks and per-row valid-block masks.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_select_blocks import ( # noqa: E501 + m3_select_blocks as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_select_blocks requires a CUDA device" + +TOPK, INIT, LOCAL = 16, 0, 1 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + n_blocks, total_q = spec["n_blocks"], spec["total_q"] + # Distinct scores (a permutation) so the selection has no ties to break. + scores = ( + torch.randperm(n_blocks * total_q, generator=g, device="cuda") + .float() + .view(1, n_blocks, total_q) + ) + if spec["mixed"]: + n_valid = torch.randint( + 17, n_blocks + 1, (total_q,), generator=g, device="cuda", dtype=torch.int32 + ) + else: + n_valid = torch.full((total_q,), n_blocks, dtype=torch.int32, device="cuda") + with validating(op): + out = op(scores, TOPK, n_valid, INIT, LOCAL, True) + ref = op.reference(scores, TOPK, n_valid, INIT, LOCAL, True) + assert out.shape == ref.shape + assert torch.equal(out, ref) + assert out.permute(1, 0, 2).is_contiguous(), "head-major backing" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_long.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_long.py new file mode 100644 index 000000000000..35ba8850a6b9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_long.py @@ -0,0 +1,230 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_select_long entry of the MiniMax-M3.1 sm_107 tp2 target (top-16 MSA block selection for long +decode rows): the entry's cells (one KV head, its TP4 lineage: this target calls the sm_107 TP4 build, whose KV-head +count is a runtime argument) and every cell again at the two KV heads per rank the TP2 target runs, plus the K7 rows.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_select_long import ( # noqa: E501 + TOPK, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_select_long import ( # noqa: E501 + m3_select_long as op, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + select_blocks_from_maxscore, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +# Valid-block counts that land a row on each of the kernel's paths: <= 16 (all picked), the 2-, 4- and 9-key +# widths, past the row (clamped), and zero (a CUDA-graph padding row). +_MIXED_VALID = (0, 5, 16, 17, 200, 2048, 2049, 4096, 4097, 8193, 9000, 1, 300, 6000, 129, 3000) + + +def _scores(spec, g): + nb, tq = spec["num_blocks"], spec["total_q"] + s = torch.randn(spec["num_kv_heads"], nb, tq, generator=g, device="cuda", dtype=torch.float32) + kind = spec["scores"] + if kind == "ties": + # Even tokens: a flat row with ten winners, so thousands of keys tie at the 16th-largest score (the + # re-read path). Odd tokens: scores on a coarse grid, so ties are broken by block index. + s[:, :, 0::2] = 0.25 + winners = torch.randint(0, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda") + s[:, :, 0::2].scatter_(1, winners[:, :, 0::2], 3.0) + s[:, :, 1::2] = (s[:, :, 1::2] * 2).round() / 2 + elif kind == "nonfinite": + s[:, 7::50, :] = float("inf") + s[:, 11::70, :] = float("nan") + s[:, 3::40, :] = -0.0 + s[:, 4::40, :] = 0.0 + # Odd tokens: almost everything -inf, so picks run past the finite scores. + s[:, :, 1::2] = float("-inf") + s[:, 0:5, 1::2] = 1.0 + return s + + +def _valid(spec, g): + tq, nb = spec["total_q"], spec["num_blocks"] + if spec["valid"] == "mixed": + vals = [_MIXED_VALID[i % len(_MIXED_VALID)] for i in range(tq)] + else: + vals = (nb - torch.randint(0, 12, (tq,), generator=g, device="cuda")).tolist() + vals[0] = nb + return torch.tensor(vals, dtype=torch.int32, device="cuda") + + +def _case(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + scores = _scores(spec, g) + nvb = _valid(spec, g) + out = torch.empty( + spec["num_kv_heads"], spec["total_q"], TOPK, dtype=torch.int32, device="cuda" + ).permute(1, 0, 2) + return scores, nvb, out, spec["init_blocks"], spec["local_blocks"], False + + +def _mass_tie_rows(scores, nvb, init_blocks, local_blocks): + """[total_q, num_kv_heads] mask of rows where more than 1024 valid blocks share the 16th-largest score.""" + num_kv_heads, num_blocks, total_q = scores.shape + blocks = torch.arange(num_blocks, device=scores.device) + raw = nvb[:total_q].long() + valid = raw.clamp(0, num_blocks) + forced = scores.permute(2, 0, 1).clone() # [total_q, heads, blocks] + forced = torch.where(blocks < init_blocks, 1.0e30, forced) + forced = torch.where(blocks >= (raw - local_blocks).clamp_min(0)[:, None, None], 1.0e29, forced) + in_row = (blocks[None, :] < valid[:, None])[:, None, :] + ranked = torch.where(in_row, forced, float("-inf")).sort(dim=-1, descending=True).values + kth = ranked[..., min(TOPK, num_blocks) - 1 : min(TOPK, num_blocks)] + return ((forced == kth) & in_row).sum(dim=-1) > 1024 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + certifying(op) + out = op(*args) + ref = op.reference(*args) + + # Selection is an exact claim: the same blocks or a different attention. + assert out.dtype is torch.int32 + assert torch.equal(out, ref), "selected blocks differ from the reference" + # And it is the in-tree selector's answer, bit for bit (the op the target falls back to) -- except on rows + # where more than 1024 blocks tie at the 16th-largest score: there the in-tree op does not break the tie to + # the smallest blocks, while this op and the reference do. + scores, nvb, _, init_blocks, local_blocks, _ = args + in_tree = select_blocks_from_maxscore( + scores, + topk=TOPK, + n_valid_blocks=nvb, + init_blocks=init_blocks, + local_blocks=local_blocks, + head_major_output=True, + ) + mass_tie = _mass_tie_rows(scores, nvb, init_blocks, local_blocks) + assert torch.equal(out[~mass_tie], in_tree[~mass_tie]), ( + "selected blocks differ from the in-tree minimax_m3_select_blocks" + ) + + +# The TP2 geometry the target runs this op at: two KV heads per rank (each head selects its own blocks), every cell's +# step again, and the K7 verify rows of c1..c4 over the AgentX 165521-token context (1294 blocks). +_TP2_SPECS = [dict(c.spec, num_kv_heads=2) for c in op.CELLS] + [ + dict( + num_blocks=1294, + total_q=tq, + valid="near_full", + scores="random", + num_kv_heads=2, + init_blocks=0, + local_blocks=1, + ) + for tq in (8, 16, 24, 32) +] + + +@pytest.mark.parametrize( + "spec", + _TP2_SPECS, + ids=[f"kv2_{s['num_blocks']}x{s['total_q']}_{s['scores']}" for s in _TP2_SPECS], +) +def test_two_kv_heads(spec) -> None: + """At two KV heads: the reference's blocks, and the in-tree selector's outside mass-tie rows.""" + args = _case(spec, seed=zlib.crc32(repr(spec).encode()) % 2**31) + out = op(*args) + ref = op.reference(*args) + assert torch.equal(out, ref), "selected blocks differ from the reference" + scores, nvb, _, init_blocks, local_blocks, _ = args + in_tree = select_blocks_from_maxscore( + scores, + topk=TOPK, + n_valid_blocks=nvb, + init_blocks=init_blocks, + local_blocks=local_blocks, + head_major_output=True, + ) + mass_tie = _mass_tie_rows(scores, nvb, init_blocks, local_blocks) + assert torch.equal(out[~mass_tie], in_tree[~mass_tie]) + # The heads select independently: their tables differ on these random scores. + assert not torch.equal(out[:, 0], out[:, 1]) + + +def test_pdl_launch_selects_the_same() -> None: + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=1) + pdl_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, False) + op(scores, nvb, pdl_out, init_blocks, local_blocks, True) + assert torch.equal(out, pdl_out) + + +def test_early_trigger_selects_the_same() -> None: + """Releasing the dependent grid right after the select's own wait changes only when the next kernel may launch.""" + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=2) + early_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False) + op(scores, nvb, early_out, init_blocks, local_blocks, True, True) + assert torch.equal(out, early_out) + + +@pytest.mark.parametrize("cell", [0, 1, 2]) +def test_trigger_phase_selects_the_same(cell: int) -> None: + """Releasing the dependent grid once the row's scores are loaded changes only when the next kernel may launch.""" + spec = op.CELLS[cell].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=3 + cell) + loaded_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False, 0) + op(scores, nvb, loaded_out, init_blocks, local_blocks, True, False, 1) + assert torch.equal(out, loaded_out) + + +@pytest.mark.parametrize("cell", range(len(op.CELLS))) +def test_warp_tail_selects_the_same(cell: int) -> None: + """The warp-0 tail (rows of at most 64 candidates) and the all-thread rank pick the same blocks + (dispatch.SELECT_LONG_SL3 and its kill value).""" + spec = op.CELLS[cell].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=7 + cell) + rank_out = torch.empty_like(out) + for phase in (0, 1): + op(scores, nvb, out, init_blocks, local_blocks, True, False, phase, True) + op(scores, nvb, rank_out, init_blocks, local_blocks, True, False, phase, False) + assert torch.equal(out, rank_out) + + +def test_op_refuses_a_row_past_its_width() -> None: + scores = torch.zeros(1, 9217, 1, device="cuda") + nvb = torch.ones(1, dtype=torch.int32, device="cuda") + out = torch.empty(1, 1, TOPK, dtype=torch.int32, device="cuda") + with pytest.raises(RuntimeError, match="at most 9216"): + op(scores, nvb, out, 0, 1, False) + + +def test_guard_refuses_negative_forcing() -> None: + args = list(_case(op.CELLS[0].spec, seed=2)) + args[4] = -1 + with pytest.raises(AssertionError, match="non-negative"): + with validating(op): + op(*args) + + +def test_guard_refuses_out_aliasing_scores() -> None: + scores, nvb, _, init_blocks, local_blocks, _ = _case(op.CELLS[0].spec, seed=3) + aliased = ( + scores.view(torch.int32) + .reshape(-1)[: scores.shape[2] * TOPK] + .view(scores.shape[2], 1, TOPK) + ) + with pytest.raises(AssertionError, match="alias"): + with validating(op): + op(scores, nvb, aliased, init_blocks, local_blocks, False) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_long_dsl.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_long_dsl.py new file mode 100644 index 000000000000..681afc63e84f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_select_long_dsl.py @@ -0,0 +1,228 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_select_long_dsl entry of the MiniMax-M3.1 sm_107 tp2 target: the long-row top-16 select from one +384-thread CTA per row, against the reference and bit for bit against the 1024-thread m3_select_long on every cell, at +the entry's one KV head (its TP4 lineage) and again at the two KV heads per rank the TP2 target runs, plus the K7 +rows.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( + m3_select_long_dsl as select_kernels, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_select_long import ( # noqa: E501 + TOPK, + m3_select_long, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_select_long_dsl import ( # noqa: E501 + m3_select_long_dsl as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +# Valid-block counts that land a row on each of the kernel's paths: <= 16 (all picked), the 4-, 11- and 24-key +# widths and their edges, past the row (clamped), and zero (a CUDA-graph padding row). +_MIXED_VALID = (0, 5, 16, 17, 200, 1536, 1537, 4224, 4225, 8193, 9000, 1, 300, 6000, 129, 3000) + + +def _scores(spec, g): + nb, tq = spec["num_blocks"], spec["total_q"] + s = torch.randn(spec["num_kv_heads"], nb, tq, generator=g, device="cuda", dtype=torch.float32) + kind = spec["scores"] + if kind == "ties": + # Even tokens: a flat row with ten winners, so thousands of keys tie at the 16th-largest score (the + # whole-row re-rank). Odd tokens: scores on a coarse grid, so ties are broken by block index. + s[:, :, 0::2] = 0.25 + winners = torch.randint(0, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda") + s[:, :, 0::2].scatter_(1, winners[:, :, 0::2], 3.0) + s[:, :, 1::2] = (s[:, :, 1::2] * 2).round() / 2 + elif kind == "ties1500": + s = torch.full_like(s, -1.0) + s[:, :1500, :] = 0.25 + winners = torch.randint( + 1500, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda" + ) + s.scatter_(1, winners, 3.0) + elif kind == "nonfinite": + s[:, 7::50, :] = float("inf") + s[:, 11::70, :] = float("nan") + s[:, 3::40, :] = -0.0 + s[:, 4::40, :] = 0.0 + s[:, :, 1::2] = float("-inf") + s[:, 0:5, 1::2] = 1.0 + return s + + +def _valid(spec, g): + tq, nb = spec["total_q"], spec["num_blocks"] + if spec["valid"] == "mixed": + vals = [_MIXED_VALID[i % len(_MIXED_VALID)] for i in range(tq)] + else: + vals = (nb - torch.randint(0, 12, (tq,), generator=g, device="cuda")).tolist() + vals[0] = nb + return torch.tensor(vals, dtype=torch.int32, device="cuda") + + +def _case(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + scores = _scores(spec, g) + nvb = _valid(spec, g) + out = torch.empty( + spec["num_kv_heads"], spec["total_q"], TOPK, dtype=torch.int32, device="cuda" + ).permute(1, 0, 2) + return scores, nvb, out, spec["init_blocks"], spec["local_blocks"], False + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + certifying(op) + out = op(*args) + ref = op.reference(*args) + assert out.dtype is torch.int32 + assert torch.equal(out, ref), "selected blocks differ from the reference" + # Every row, mass ties included, is the 1024-thread kernel's answer. + scores, nvb, _, init_blocks, local_blocks, _ = args + long_out = torch.empty_like(out) + m3_select_long(scores, nvb, long_out, init_blocks, local_blocks, False) + assert torch.equal(out, long_out), "selected blocks differ from m3_select_long" + + +# The TP2 geometry the target runs this op at: two KV heads per rank (each head selects its own blocks), every cell's +# step again, and the K7 verify rows of c1..c4 over the AgentX 165521-token context (1294 blocks). +_TP2_SPECS = [dict(c.spec, num_kv_heads=2) for c in op.CELLS] + [ + dict( + num_blocks=1294, + total_q=tq, + valid="near_full", + scores="random", + num_kv_heads=2, + init_blocks=0, + local_blocks=1, + ) + for tq in (8, 16, 24, 32) +] + + +@pytest.mark.parametrize( + "spec", + _TP2_SPECS, + ids=[f"kv2_{s['num_blocks']}x{s['total_q']}_{s['scores']}" for s in _TP2_SPECS], +) +def test_two_kv_heads(spec) -> None: + """At two KV heads: the reference's blocks and the 1024-thread kernel's, bit for bit, on every row.""" + args = _case(spec, seed=zlib.crc32(repr(spec).encode()) % 2**31) + out = op(*args) + assert torch.equal(out, op.reference(*args)), "selected blocks differ from the reference" + scores, nvb, _, init_blocks, local_blocks, _ = args + long_out = torch.empty_like(out) + m3_select_long(scores, nvb, long_out, init_blocks, local_blocks, False) + assert torch.equal(out, long_out), "selected blocks differ from m3_select_long" + + +@pytest.mark.parametrize("layout", ["kv_blocks_q", "kv_q_blocks"]) +@pytest.mark.parametrize("valid", [17, 1294, 2524, 3907, 9216]) +def test_layouts_and_widths_match_the_1024_thread_select(layout, valid) -> None: + """Both score layouts the scorer can hand over, at a valid count in each key width.""" + g = torch.Generator(device="cuda").manual_seed(valid) + nb = max(valid, 129) + s = torch.randn(1, nb, 8, generator=g, device="cuda") + if layout == "kv_q_blocks": + s = s.permute(0, 2, 1).contiguous().permute(0, 2, 1) + nvb = torch.tensor([valid - (t % 2) for t in range(8)], dtype=torch.int32, device="cuda") + out = torch.empty(1, 8, TOPK, dtype=torch.int32, device="cuda").permute(1, 0, 2) + ref = torch.empty_like(out) + op(s, nvb, out, 0, 1, False) + m3_select_long(s, nvb, ref, 0, 1, False) + assert torch.equal(out, ref) + + +def test_pdl_launch_selects_the_same() -> None: + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=1) + pdl_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, False) + op(scores, nvb, pdl_out, init_blocks, local_blocks, True) + assert torch.equal(out, pdl_out) + + +def test_early_trigger_selects_the_same() -> None: + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=2) + early_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False) + op(scores, nvb, early_out, init_blocks, local_blocks, True, True) + assert torch.equal(out, early_out) + + +@pytest.mark.parametrize("cell", [0, 1, 2]) +def test_trigger_phase_selects_the_same(cell: int) -> None: + spec = op.CELLS[cell].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=3 + cell) + loaded_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False, 0) + op(scores, nvb, loaded_out, init_blocks, local_blocks, True, False, 1) + assert torch.equal(out, loaded_out) + + +@pytest.mark.parametrize("early_trigger", [False, True]) +@pytest.mark.parametrize("cell", range(len(op.CELLS))) +def test_valid_count_read_before_the_wait_selects_the_same(cell: int, early_trigger: bool) -> None: + """kernels/m3_select_long_dsl.N_VALID_PREWAIT: the kernel that reads its row's valid-block count before the grid + wait selects exactly the blocks of the one that reads it after, on every certified cell (padding, clamped and tied + rows).""" + phase = 0 if early_trigger else 1 + spec = op.CELLS[cell].spec + scores, nvb, before, init_blocks, local_blocks, _ = _case(spec, seed=11 + cell) + after = torch.empty_like(before) + select_kernels._compile((True, early_trigger, phase, True))( + scores, nvb, before, init_blocks, local_blocks + ) + select_kernels._compile((True, early_trigger, phase, False))( + scores, nvb, after, init_blocks, local_blocks + ) + assert torch.equal(before, after) + + +def test_graph_replay_selects_the_same() -> None: + spec = op.CELLS[1].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=9) + ref = op.reference(scores, nvb, out, init_blocks, local_blocks) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + op(scores, nvb, out, init_blocks, local_blocks, True, False, 1) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(scores, nvb, out, init_blocks, local_blocks, True, False, 1) + for _ in range(3): + out.fill_(-7) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out, ref) + + +def test_op_refuses_a_row_past_its_width() -> None: + scores = torch.zeros(1, 9217, 1, device="cuda") + nvb = torch.ones(1, dtype=torch.int32, device="cuda") + out = torch.empty(1, 1, TOPK, dtype=torch.int32, device="cuda") + with pytest.raises(RuntimeError, match="at most 9216"): + op(scores, nvb, out, 0, 1, False) + + +def test_guard_refuses_negative_forcing() -> None: + args = list(_case(op.CELLS[0].spec, seed=2)) + args[4] = -1 + with pytest.raises(AssertionError, match="non-negative"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode.py new file mode 100644 index 000000000000..cd92449d1f1f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode.py @@ -0,0 +1,359 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_sparse_decode entry of the MiniMax-M3.1 sm_107 tp2 target at its per-rank geometry: 32 query +heads over 2 KV heads, head_dim 128, FP8 q, a packed NVFP4 cache [pages, 2, 2, 128, 64] (``_m3_nvfp4_cache``, the TP2 +helper). + +LC_TP2 (off by default) + TP2WIDE_LC2: steps of at most 32 rows run the Triton block-sparse decode, never the C++ +``sparse_decode_lc`` op (LC2 takes selected tables at 33..128 rows only; its tests are +``test_modeling_v2_m3_sparse_decode_lc.py``). The tests check that route at TP2 shapes of at most 32 rows: it never +takes the LC (even with LC_TP2 forced on, now that ``lc_applies`` accepts the TP2 q), reports its own engagement path, +and writes the stock in-tree decode's rows (bit for bit on the +identity and short-table routes, as the target's Triton copy takes the stock split-K), within the band of the float64 +reference. The entry's own ``reference`` mirrors one KV head (its TP4 lineage), so the band is checked against a float64 +reference built per KV head. LC2_ALLROWS and LC2_IDENTITY (default ON: LC2 from one row and on identity steps) are +killed in this file. +""" + +import zlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_attn_lc, + m3_prequant, + m3_report, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + triton_sparse_decode as target_kernel, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_sparse_decode import ( # noqa: E501 + m3_sparse_decode as op, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_sparse_decode_lc import ( # noqa: E501 + sparse_attention_reference, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.triton_sparse_decode import ( + minimax_m3_sparse_attn_decode as stock_decode, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_nvfp4_cache import ( # noqa: E402 — needs the path declared above + HEAD_DIM, + NUM_HEADS, + NUM_KV_HEADS, + build_step, +) +from _validating import assert_within_ulp, validating # noqa: E402 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + + +@pytest.fixture(autouse=True) +def _lc2_band_only(monkeypatch): + """These tests cover the Triton route of at most 32 rows: LC2_ALLROWS (LC2 from one row) and LC2_IDENTITY (LC2 on + identity steps) are killed here; their defaults are tested in test_modeling_v2_m3_sparse_decode_lc.py.""" + monkeypatch.setattr(target_kernel, "LC2_ALLROWS", False) + monkeypatch.setattr(target_kernel, "LC2_IDENTITY", False) + monkeypatch.setattr(target_kernel, "LC_KEYS", target_kernel.LC_KEYS - {"plain"}) + + +# Band, in BF16 ulps (2^-8) of each [heads, 128] row's largest magnitude, and of the relative RMS (the TP4 entry's +# band: exact operands, BF16 rounding on both sides, FP16 probabilities, FP32 split-K partials merged by their +# log2-sum-exp, approximate exp2 / log2). +_ELEMENT_ULP = 2.0 +_RMS_ULP = 0.5 + +# The entry's certified step shapes (op.CELLS' lengths and query lengths), rebuilt at the TP2 geometry, plus the K7 +# verify rows (decode_query_len 8) of c1..c4 and a 165521-token AgentX request. +_STEPS = [ + *((c.spec["seq_lens"], c.spec["decode_query_len"], c.spec["identity"]) for c in op.CELLS), + ((165521,), 8, False), + ((165521, 20000), 8, False), + ((900, 2048, 130), 8, True), + ((40000, 3000, 165521, 129), 8, False), +] + + +def _call(case, output, identity): + return op( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + case.sm_scale, + output, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + identity, + ) + + +def _reference(case, topk, output): + """Float64 reference per KV head: KV head ``h`` serves query heads ``[h * group, (h + 1) * group)`` with its own + top-k row (the TP2 geometry: two KV heads).""" + kv_heads = case.k_paged.shape[1] + group = case.q.shape[1] // kv_heads + heads = [ + sparse_attention_reference( + case.q[:, h * group : (h + 1) * group], + case.k_paged.view(torch.uint8), + case.v_paged.view(torch.uint8), + case.k_block_scale.view(torch.uint8), + case.v_block_scale.view(torch.uint8), + case.k_global_scale, + case.v_global_scale, + topk[h], + case.block_table, + case.seq_lens, + case.decode_query_len, + case.sm_scale, + kv_head=h, + ) + for h in range(kv_heads) + ] + return torch.cat(heads, dim=1).to(output.dtype) + + +def _stock(case, output): + stock_decode( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + sm_scale=case.sm_scale, + output=output, + decode_query_len=case.decode_query_len, + k_block_scale=case.k_block_scale, + v_block_scale=case.v_block_scale, + k_global_scale=case.k_global_scale, + v_global_scale=case.v_global_scale, + ) + + +def _empty(case): + return torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + + +def test_tp2_geometry() -> None: + case = build_step((3000,), 4, identity=False, seed=1) + assert tuple(case.q.shape[1:]) == (32, 128) and NUM_KV_HEADS == 2 + assert int(case.k_paged.shape[1]) == 2 and int(case.topk.shape[0]) == 2 + + +@pytest.mark.parametrize("step", _STEPS, ids=[f"{len(s[0])}req_q{s[1]}_{s[2]}" for s in _STEPS]) +def test_triton_route_at_tp2_shapes(step, monkeypatch) -> None: + """Every step runs the Triton route (never the LC op), reports the LC_TP2 Triton path, stays finite on padding + rows and lies in the band of the float64 reference.""" + seq_lens, qlen, identity = step + # The Triton route itself: K3 steps from 9 rows can opt into LC2 (TP2_LC2_SMALL, its own tests), so pin it off. + monkeypatch.setattr(target_kernel, "TP2_LC2_SMALL", False) + assert len(seq_lens) * qlen < target_kernel.lc_min_rows(qlen), ( + "this test drives the Triton band (<= 32 rows)" + ) + case = build_step( + seq_lens, qlen, identity=identity, seed=zlib.crc32(repr(step).encode()) % 2**31 + ) + taken = [] + monkeypatch.setattr(m3_report, "taken", lambda path, rows, what="": taken.append(path)) + lc_calls = [] + real = m3_attn_lc.m3_sparse_decode_lc + monkeypatch.setattr( + m3_attn_lc, + "m3_sparse_decode_lc", + lambda *a, **k: (lc_calls.append(1), real(*a, **k))[1], + ) + output = _empty(case) + _call(case, output, identity) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert not lc_calls, "the C++ long-context op ran at TP2" + assert "m3_sparse_decode_lc" not in taken, taken + assert target_kernel.LC_TRITON_PATH in taken, taken + assert torch.isfinite(output.float()).all(), "padding rows must be zeros, not NaN" + ref = _reference(case, case.topk, output) + assert_within_ulp(output, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +def test_rows_up_to_32_keep_the_triton_route_even_when_forced_on(monkeypatch) -> None: + """``lc_applies`` now accepts the TP2 q [T, 32, 128] on two KV heads (LC2), but at most 32 rows the dispatch never + asks it: with LC_TP2 forced on the step still runs the Triton route, bit for bit the default (LC_TP2 off) rows. + (8 rows at the K3 verify: below the opt-in lc_min_rows(4) = 9.)""" + case = build_step((165521, 3000), 4, identity=False, seed=31) + assert 2 * 4 < target_kernel.lc_min_rows(4) + assert m3_attn_lc.lc_applies( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.k_block_scale, + case.v_block_scale, + _empty(case), + ) + lc_calls = [] + real = m3_attn_lc.m3_sparse_decode_lc + monkeypatch.setattr( + m3_attn_lc, + "m3_sparse_decode_lc", + lambda *a, **k: (lc_calls.append(1), real(*a, **k))[1], + ) + default = _empty(case) + _call(case, default, False) + monkeypatch.setattr(target_kernel, "LC_TP2", True) + forced = _empty(case) + _call(case, forced, False) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert not lc_calls, "the C++ long-context op ran at 8 rows" + assert torch.equal(default.view(torch.int16), forced.view(torch.int16)) + + +@pytest.mark.parametrize("requests", [1, 2, 4]) +@pytest.mark.parametrize("qlen", [4, 8]) +def test_identity_route_is_bitwise_the_stock_decode(requests, qlen) -> None: + """The target's Triton producer + split-K merge takes the stock in-tree decode's split-K and writes its rows bit for + bit at the TP2 geometry (per-row producer: the multi-query producer needs one KV head).""" + case = build_step((1500, 2048, 900, 130)[:requests], qlen, identity=True, seed=60 + requests) + target, stock = _empty(case), _empty(case) + _call(case, target, True) + _stock(case, stock) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)), case.seq_lens.tolist() + + +def _identity_table( + seq_lens: torch.Tensor, decode_query_len: int, heads: int, lag: int = 0 +) -> torch.Tensor: + """The identity selection of every verify token, staged from lengths `lag` tokens behind seq_lens.""" + seq = (seq_lens.cpu().to(torch.int64) - lag).clamp(min=0).repeat_interleave(decode_query_len) + intra = torch.arange(seq.shape[0]) % decode_query_len + n_valid = ((seq - decode_query_len + intra + 1).clamp(min=0) + 127) // 128 + ar = torch.arange(16) + table = torch.where(ar[None, :] < n_valid[:, None], ar[None, :], -1).to(torch.int32) + return table.unsqueeze(0).expand(heads, -1, -1).contiguous().cuda() + + +# Each crosses a 128-token edge within its four verify tokens or between the stale and the true lengths. +_EDGE_LENS = (1284, 1287, 1281, 129, 132, 2048, 1028, 5, 643, 1920, 256, 1) + + +@pytest.mark.parametrize("fault", ["stale_one_step", "all_minus_one", "random_ids"]) +@pytest.mark.parametrize("requests", [1, 2, 3, 4]) +def test_identity_route_reads_its_own_lengths_not_the_table(requests, fault) -> None: + """The Triton identity route derives block i for slot i from the kv lengths the kernel reads: a staged table that + lags the lengths by a verify step, or is garbage, gives the stock decode's rows over the true identity selection, + bit for bit, on both KV heads.""" + lens = tuple(_EDGE_LENS[(requests * 5 + k) % len(_EDGE_LENS)] for k in range(requests)) + case = build_step(lens, 4, identity=True, seed=400 + requests) + heads = case.topk.shape[0] + true_table = _identity_table(case.seq_lens, 4, heads) + assert torch.equal(case.topk, true_table), ( + "build_step's identity table is the identity selection" + ) + if fault == "stale_one_step": + staged = _identity_table(case.seq_lens, 4, heads, lag=4) + elif fault == "all_minus_one": + staged = torch.full_like(true_table, -1) + else: + g = torch.Generator().manual_seed(requests) + staged = ( + torch.randint(-1, case.block_table.shape[1], true_table.shape, generator=g) + .to(torch.int32) + .cuda() + ) + target, stock = _empty(case), _empty(case) + _stock(case, stock) + _call(SimpleNamespace(**{**vars(case), "topk": staged}), target, True) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)), ( + fault, + case.seq_lens.tolist(), + ) + + +def _short_lens(requests: int, g: torch.Generator) -> tuple: + """Short contexts of 1..16 blocks of 128 tokens (the identity-route regime), a few exactly on a block edge.""" + lens = torch.randint(4, 16 * 128 + 1, (requests,), generator=g).tolist() + if requests > 1: + lens[0] = 128 * int(torch.randint(1, 17, (1,), generator=g)) + return tuple(lens) + + +@pytest.mark.parametrize("seed", range(6)) +@pytest.mark.parametrize("requests", [1, 2, 3, 4]) +def test_short_table_stress_is_bitwise_the_stock_decode(requests, seed, monkeypatch) -> None: + """Short contexts (1..16 blocks) at 1..4 requests x 4 verify rows, the identity route and a random selected route + with -1 tails past each token's blocks (each KV head its own selection): the stock decode's rows bit for bit. + The Triton route's contract: 3 / 4 requests (12 / 16 rows) can opt into LC2 (TP2_LC2_SMALL, in the reference + band, not bitwise; test_modeling_v2_m3_sparse_decode_lc.py), so it is pinned off here.""" + monkeypatch.setattr(target_kernel, "TP2_LC2_SMALL", False) + g = torch.Generator().manual_seed(1000 * requests + seed) + for identity in (True, False): + case = build_step(_short_lens(requests, g), 4, identity=identity, seed=7 * seed + requests) + target, stock = _empty(case), _empty(case) + _call(case, target, identity) + _stock(case, stock) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)), ( + identity, + case.seq_lens.tolist(), + ) + + +@pytest.mark.parametrize("head", [0, 1]) +@pytest.mark.parametrize("fault", ["neg_in_window", "id_past_row"]) +def test_invalid_slot_fails_closed(fault, head, monkeypatch) -> None: + """A corrupt selection slot on either KV head (a -1 inside a token's block window, or an id past the block-table + row) is read as an absent block instead of a wild KV read: the rows equal the reference over the remaining + blocks. The Triton route's contract (12 rows can opt into LC2 under TP2_LC2_SMALL, which does not mask an id + past the row, as at its 33..128 rows; the selector never writes one), so TP2_LC2_SMALL is pinned off here.""" + monkeypatch.setattr(target_kernel, "TP2_LC2_SMALL", False) + case = build_step((1500, 2048, 700), 4, identity=False, seed=303 + head) + tok, slot = 5, 2 + case.topk[head, tok, slot] = -1 if fault == "neg_in_window" else case.block_table.shape[1] + 7 + output = _empty(case) + _call(case, output, False) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + ref_topk = case.topk.clone() + ref_topk[head, tok, slot] = -1 + assert torch.isfinite(output.float()).all() + assert_within_ulp( + output, _reference(case, ref_topk, output), element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP + ) + + +def test_guard_refuses_a_padded_selection_slot() -> None: + """The Triton producers index the block table with every one of a token's first min(16, blocks) ids, -1 included, + on every KV head.""" + case = build_step((1500,), 1, identity=True, seed=23) + case.topk[1, 0, 3] = -1 + with pytest.raises(AssertionError, match="blocks of the request"): + with validating(op): + _call(case, _empty(case), True) + + +def test_guard_refuses_a_false_identity_promise() -> None: + """identity_blocks promises every token's table holds every block of its request in order.""" + case = build_step((1500, 1600, 1700, 1800), 4, identity=True, seed=29) + case.topk[1, 5, :2] = case.topk[1, 5, :2].flip(0) + with pytest.raises(AssertionError, match="identity_blocks promises"): + with validating(op): + _call(case, _empty(case), True) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode_lc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode_lc.py new file mode 100644 index 000000000000..991946072c5a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode_lc.py @@ -0,0 +1,875 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU tests for trtllm::minimax_m3_sm107_tp2_sparse_decode_lc (LC2: one-launch long-context sparse decode, TP2) as the +MiniMax-M3.1 sm_107 tp2 target takes it (TP2WIDE_LC2), ported from the M3 sm_107 tp2 target's test. + +Contracts: +- the BF16 rows sit in the same band around a float64 reference as the stock Triton sparse decode, and every variant + (linear softmax, pre-wait block-table staging, MXFP8 hand-off on or off) writes the same rows as the stock + log-domain recurrence where the arithmetic is shared; +- the MXFP8 hand-off (q8, sf) is bitwise torch.ops.trtllm.mxfp8_quantize(out, True) of the rows the kernel wrote: the + E4M3 bytes row-major and the UE8M0 scales in its 128x4-swizzled layout. The o_proj consumes this as its + prequantized activation, so this test is the interface contract; +- the target dispatch takes LC2 for selected tables at lc_min_rows(qlen)..LC_MAX_ROWS rows (9 for K7, 33 otherwise; + 9 when opted in for K3 under TP2_LC2_SMALL). Every narrower step (and every + step under TRTLLM_M31_DISABLE_TP2WIDE_LC2=1, emulated by LC_KEYS = frozenset()) is bitwise the Triton producer + + merge, and the qlen-8 (K7) wide steps and the K3 steps from 12 rows sit in the band. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm # noqa: F401 (registers torch.ops.trtllm) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_sparse_decode_lc import ( # noqa: E501 + m3_sparse_decode_lc as op, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_sparse_decode_lc import ( # noqa: E501 + mxfp8_handoff_reference, +) + +__extra_import_path__ = ["../../../.."] +from _validating import ( # noqa: E402, F401 — needs the path declared above + assert_within_ulp, + certifying, +) + +PAGE, DIM, TOPK, GROUP, KV_HEADS = 128, 128, 16, 16, 2 +WIDTH = KV_HEADS * GROUP * DIM +SM_SCALE = DIM**-0.5 +# Band, in BF16 ulps (2^-8) of each output row's largest magnitude, and of the relative RMS. Same as the sm_103 TP4 +# kernel; on SM107 the stock Triton decode and this kernel both measure at most 1.03 ulps and 0.26 RMS. +_ELEMENT_ULP = 2.0 +_RMS_ULP = 0.5 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +_E2M1 = [0, 0.5, 1, 1.5, 2, 3, 4, 6, -0.0, -0.5, -1, -1.5, -2, -3, -4, -6] + + +def _op(): + return torch.ops.trtllm.minimax_m3_sm107_tp2_sparse_decode_lc + + +def _e4m3_bytes(shape, gen, lo=0x28, hi=0x48): + mag = torch.randint(lo, hi, shape, dtype=torch.int32, device="cuda", generator=gen) + sign = torch.randint(0, 2, shape, dtype=torch.int32, device="cuda", generator=gen) << 7 + return (mag | sign).to(torch.uint8) + + +def _case(seq_lens, seed, dql=4, overlap=14, q_hi=0x48, tie=False): + """A packed-NVFP4 paged cache, ascending top-16 tables with a -1 tail, and FP8 queries for TP2 geometry.""" + gen = torch.Generator(device="cuda").manual_seed(seed) + reqs, total_q = len(seq_lens), len(seq_lens) * dql + nblk = [(s + PAGE - 1) // PAGE for s in seq_lens] + pages = sum(nblk) + 8 + k = torch.randint( + 0, 256, (pages, KV_HEADS, PAGE, DIM // 2), dtype=torch.uint8, device="cuda", generator=gen + ) + v = torch.randint( + 0, 256, (pages, KV_HEADS, PAGE, DIM // 2), dtype=torch.uint8, device="cuda", generator=gen + ) + if tie: + k[:] = k[:1, :, :1, :] + ksc = _e4m3_bytes((pages, KV_HEADS, PAGE, DIM // 16), gen) + vsc = _e4m3_bytes((pages, KV_HEADS, PAGE, DIM // 16), gen) + perm = torch.randperm(pages, device="cuda", generator=gen).int() + bt = torch.full((reqs, max(nblk)), -1, dtype=torch.int32, device="cuda") + start = 0 + for r, n in enumerate(nblk): + bt[r, :n] = perm[start : start + n] + start += n + topk = torch.full((KV_HEADS, total_q, TOPK), -1, dtype=torch.int32, device="cuda") + for r, s in enumerate(seq_lens): + for h in range(KV_HEADS): + last = (s + PAGE - 1) // PAGE + base = torch.randperm(max(last - 1, 1), device="cuda", generator=gen)[ + : TOPK - 1 + ].tolist() + for qo in range(dql): + nb = (s - dql + qo + 1 + PAGE - 1) // PAGE + sel = [b for b in base[:overlap] if b < nb - 1] + for b in torch.randperm(max(nb - 1, 1), device="cuda", generator=gen).tolist(): + if len(sel) >= min(TOPK, nb) - 1: + break + if b not in sel and b < nb - 1: + sel.append(b) + sel = sorted(set(sel + [nb - 1]))[: min(TOPK, nb)] + topk[h, r * dql + qo, : len(sel)] = torch.tensor( + sel, dtype=torch.int32, device="cuda" + ) + q = _e4m3_bytes((total_q, KV_HEADS * GROUP, DIM), gen, lo=0x20, hi=q_hi).view( + torch.float8_e4m3fn + ) + return dict( + q=q, + k=k, + v=v, + ksc=ksc, + vsc=vsc, + kgs=torch.tensor([0.03], device="cuda"), + vgs=torch.tensor([0.05], device="cuda"), + topk=topk, + bt=bt, + seq=torch.tensor(seq_lens, dtype=torch.int32, device="cuda"), + dql=dql, + ) + + +def _dequant(packed, sc, quad): + lut = torch.tensor(_E2M1, dtype=torch.float64, device="cuda") + vals = torch.stack([lut[(packed & 0xF).long()], lut[(packed >> 4).long()]], dim=-1).reshape( + *packed.shape[:3], DIM + ) + if ( + quad + ): # V scales: token t, column c at byte (t // 4) * 32 + t % 4 + 4 * c within the page/head + t = torch.arange(PAGE, device="cuda")[:, None] + idx = (t // 4) * 32 + t % 4 + 4 * torch.arange(8, device="cuda")[None, :] + sc = sc.reshape(*sc.shape[:2], PAGE * 8)[:, :, idx] + return vals * sc.contiguous().view(torch.float8_e4m3fn).to(torch.float64).repeat_interleave( + 16, dim=-1 + ) + + +def _reference(c): + q = c["q"].to(torch.float64) + out = torch.zeros(q.shape, dtype=torch.float64, device="cuda") + dql = c["dql"] + for tok in range(q.shape[0]): + r, qo = divmod(tok, dql) + kv_len = max(int(c["seq"][r]) - dql + qo + 1, 0) + for h in range(KV_HEADS): + blks = [b for b in c["topk"][h, tok].tolist() if b >= 0][ + : min(TOPK, (kv_len + PAGE - 1) // PAGE) + ] + if not blks: + continue + pg = c["bt"][r, blks].long() + kk = _dequant(c["k"][pg, h : h + 1], c["ksc"][pg, h : h + 1], False)[:, 0].reshape( + -1, DIM + ) + vv = _dequant(c["v"][pg, h : h + 1], c["vsc"][pg, h : h + 1], True)[:, 0].reshape( + -1, DIM + ) + pos = ( + torch.tensor(blks, device="cuda")[:, None] * PAGE + + torch.arange(PAGE, device="cuda") + ).reshape(-1) + s = (q[tok, h * GROUP : (h + 1) * GROUP] @ kk.T) * (SM_SCALE * c["kgs"].item()) + s[:, pos >= kv_len] = float("-inf") + out[tok, h * GROUP : (h + 1) * GROUP] = (torch.softmax(s, dim=-1) @ vv) * c[ + "vgs" + ].item() + return out.reshape(q.shape[0], WIDTH) + + +def _band(x, ref): + x = x.to(torch.float64).reshape(-1, DIM) + ref = ref.reshape(-1, DIM) + rowmax = ref.abs().amax(dim=-1, keepdim=True) + live = rowmax[:, 0] > 0 + unit = (rowmax * 2.0**-8).clamp_min(1e-300) + err = ((x - ref).abs() / unit)[live] + rms = (((x - ref) ** 2).mean(dim=-1).sqrt() / unit[:, 0])[live] + return (float(err.max()) if err.numel() else 0.0), (float(rms.max()) if rms.numel() else 0.0) + + +def _run(c, linear=False, prewait=False, mxfp8=False, n1=None, use_pdl=False, out=None): + total_q = c["q"].shape[0] + if out is None: + out = torch.empty(total_q, WIDTH, dtype=torch.bfloat16, device="cuda") + q8 = torch.zeros(total_q, WIDTH, dtype=torch.uint8, device="cuda") if mxfp8 else None + sf_bytes = 128 * (WIDTH // 32) * ((total_q + 127) // 128) + sf = torch.zeros(sf_bytes, dtype=torch.uint8, device="cuda") if mxfp8 else None + _op()( + c["q"].view(torch.uint8), + c["k"], + c["v"], + c["ksc"], + c["vsc"], + c["kgs"], + c["vgs"], + c["topk"], + c["bt"], + c["seq"], + out, + q8, + sf, + WIDTH // 128, + c["dql"], + SM_SCALE, + linear, + use_pdl, + prewait, + n1, + ) + return out, q8, sf + + +_CASES = { + "b1-165k": dict(seq_lens=[165521]), + "b2-mixed": dict(seq_lens=[165521, 98765]), + "b2-short": dict(seq_lens=[130, 5000]), + "b1-one-token-tail": dict(seq_lens=[128 * 40 + 1]), + "b1-padded-topk": dict(seq_lens=[700]), + "b1-tiny": dict(seq_lens=[4]), + "b2-large-logits": dict(seq_lens=[165521, 64000], q_hi=0x70), + "b1-ties": dict(seq_lens=[20000], tie=True), + "b2-disjoint": dict(seq_lens=[165521, 165521], overlap=0), + "b4": dict(seq_lens=[165521, 120000, 80000, 30000]), +} + +# The TP2WIDE_LC2 steps of the K7 (qlen 8) verify: 5 / 8 / 12 / 16 requests = 40 / 64 / 96 / 128 rows, mixed lengths +# with 128-token edges inside the verify rows and a zero-length padding request. +_WIDE_CASES = { + "k7-c5": dict(seq_lens=[165521, 20000, 1285, 129, 0], dql=8), + "k7-c8": dict(seq_lens=[165521, 98765, 20000, 2048, 1285, 136, 4, 0], dql=8), + "k7-c12": dict( + seq_lens=[165521] + [(40000, 20000, 1285, 129, 2056)[i % 5] for i in range(10)] + [0], dql=8 + ), + "k7-c16": dict( + seq_lens=[165521] + [(40000, 20000, 1285, 129, 2056, 136)[i % 6] for i in range(14)] + [0], + dql=8, + ), + # TP2WIDE_LC2_448: the agg TP2 Y steps at K3 (4 rows / request: c46 / c56 / c64 = 184 / 224 / 256 rows) + # and the K7 steps at 24 / 46 / 56 requests (192 / 368 / 448 rows); 184 and 368 end in a partial scale tile. + **{ + f"k{dql - 1}-c{reqs}": dict( + seq_lens=[165521] + + [(40000, 20000, 1285, 129, 2056, 136)[i % 6] for i in range(reqs - 2)] + + [0], + dql=dql, + ) + for dql, reqs in ((4, 46), (4, 56), (4, 64), (8, 24), (8, 46), (8, 56)) + }, +} + + +@pytest.mark.parametrize("name", list(_CASES) + list(_WIDE_CASES)) +@pytest.mark.parametrize("seed", [0, 1]) +def test_band_vs_float64(name, seed) -> None: + c = _case(seed=seed, **{**_CASES, **_WIDE_CASES}[name]) + out, _, _ = _run(c) + elem, rms = _band(out, _reference(c)) + assert torch.isfinite(out.float()).all() + assert elem <= _ELEMENT_ULP and rms <= _RMS_ULP, (elem, rms) + + +@pytest.mark.parametrize("name", ["b1-165k", "b2-mixed", "b1-padded-topk", "b4"]) +def test_variants_write_the_same_rows(name) -> None: + c = _case(seed=7, **_CASES[name]) + base, _, _ = _run(c) + for prewait in (False, True): + for mxfp8 in (False, True): + out, _, _ = _run(c, prewait=prewait, mxfp8=mxfp8) + assert torch.equal(out.view(torch.int16), base.view(torch.int16)), (prewait, mxfp8) + lin, _, _ = _run(c, linear=True) + elem, rms = _band(lin, _reference(c)) + assert elem <= _ELEMENT_ULP and rms <= _RMS_ULP, (elem, rms) + + +@pytest.mark.parametrize("name", list(_CASES) + list(_WIDE_CASES)) +@pytest.mark.parametrize("linear", [False, True]) +def test_mxfp8_handoff_is_stock_mxfp8_quantize(name, linear) -> None: + """The o_proj interface: (q8, sf) == mxfp8_quantize(out, is_sf_swizzled_layout=True), bitwise.""" + c = _case(seed=3, **{**_CASES, **_WIDE_CASES}[name]) + out, q8, sf = _run(c, linear=linear, mxfp8=True) + ref_q, ref_sf = torch.ops.trtllm.mxfp8_quantize(out, True) + assert torch.equal(q8, ref_q.view(torch.uint8).reshape(q8.shape)) + ref_sf = ref_sf.view(torch.uint8).reshape(-1) + assert torch.equal(sf[: ref_sf.numel()], ref_sf) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """The catalog entry's cells through its own __call__ (guarded by is_valid) and its own reference.""" + spec = cell.spec + c = _case( + list(spec["seq_lens"]), + seed=zlib.crc32(cell.why.encode()) % 2**31, + dql=spec["decode_query_len"], + ) + total_q = c["q"].shape[0] + # the hand-off buffers hold whole 128-row tiles (MergeMxfp8.MAX_ROWS) + alloc = (total_q + 127) // 128 * 128 + out = torch.empty(total_q, WIDTH, dtype=torch.bfloat16, device="cuda") + q8 = torch.zeros(alloc, WIDTH, dtype=torch.uint8, device="cuda") if spec["mxfp8"] else None + sf = ( + torch.zeros(alloc * (WIDTH // 32), dtype=torch.uint8, device="cuda") + if spec["mxfp8"] + else None + ) + args = ( + c["q"].view(torch.uint8), + c["k"], + c["v"], + c["ksc"], + c["vsc"], + c["kgs"], + c["vgs"], + c["topk"], + c["bt"], + c["seq"], + out, + q8, + sf, + WIDTH // 128, + c["dql"], + SM_SCALE, + spec["linear_softmax"], + False, + spec["prewait"], + ) + certifying(op) + op(*args) + ref = op.reference(*args) + assert_within_ulp(out, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + assert torch.isfinite(out.float()).all() + if spec["mxfp8"]: + ref_q8, ref_sf = mxfp8_handoff_reference(out, alloc) + assert torch.equal(q8[:total_q], ref_q8) + assert torch.equal(sf, ref_sf) + stock_q, stock_sf = torch.ops.trtllm.mxfp8_quantize(out, True) + assert torch.equal(q8[:total_q], stock_q.view(torch.uint8).reshape(total_q, WIDTH)) + assert torch.equal(sf[: stock_sf.numel()], stock_sf.view(torch.uint8).reshape(-1)) + + +# LC on vs the killed range (SPARSE_DECODE_LC = frozenset(), TRTLLM_M31_DISABLE_TP2WIDE_LC2=1: this target's Triton +# producer + merge) through the target's +# minimax_m3_sparse_attn_decode on the same inputs. Not bitwise: the split-K merge sums the 16 partials in Triton's +# reduction order there and in chunk order here (the per-block math is the same), so the rows differ by summation +# order only. Band, in BF16 ulps of each row's largest magnitude: measured on SM107 at most 1.92 per element and +# 0.18 RMS, so 2 / 0.25. +_LC_VS_KILLED_ELEMENT_ULP = 2.0 +_LC_VS_KILLED_RMS_ULP = 0.25 +# The K7 wide cells (40..128 rows; this target's Triton route at 1-2 top-k chunks, unlike the cells above): LC and the +# killed Triton path each sit within the float64-reference band (_ELEMENT_ULP / _RMS_ULP, asserted below for both), so +# their distance is bounded by twice that band. Measured on SM107: at most 1.97 per element and 0.263 RMS. +_LC_VS_KILLED_WIDE_ELEMENT_ULP = 2 * _ELEMENT_ULP +_LC_VS_KILLED_WIDE_RMS_ULP = 2 * _RMS_ULP + + +def _target_decode(c, killed): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_prequant, + triton_sparse_decode, + ) + + landed = triton_sparse_decode.LC_KEYS + if killed: + triton_sparse_decode.LC_KEYS = frozenset() + try: + total_q = c["q"].shape[0] + out = torch.empty(total_q, NUM_HEADS_TOTAL, DIM, dtype=torch.bfloat16, device="cuda") + triton_sparse_decode.minimax_m3_sparse_attn_decode( + c["q"], + c["k"], + c["v"], + c["topk"], + c["bt"], + c["seq"], + sm_scale=SM_SCALE, + output=out, + decode_query_len=c["dql"], + k_block_scale=c["ksc"], + v_block_scale=c["vsc"], + k_global_scale=c["kgs"], + v_global_scale=c["vgs"], + ) + torch.cuda.synchronize() + finally: + triton_sparse_decode.LC_KEYS = landed + m3_prequant.SLOT.clear() + return out.view(total_q, WIDTH) + + +NUM_HEADS_TOTAL = KV_HEADS * GROUP + + +@pytest.mark.parametrize("name", list(_CASES) + list(_WIDE_CASES)) +@pytest.mark.parametrize("seed", [0, 1, 2]) +def test_lc_vs_killed_stock_path(name, seed) -> None: + """The landed cell and its kill switch on the same step: rows within summation-order distance of each other + (bitwise equal at most 32 rows, where both arms run the Triton route).""" + c = _case(seed=seed, **{**_CASES, **_WIDE_CASES}[name]) + lc = _target_decode(c, killed=False) + stock = _target_decode(c, killed=True) + assert torch.isfinite(lc.float()).all() and torch.isfinite(stock.float()).all() + elem, rms = _band(lc, stock.to(torch.float64)) + if name in _WIDE_CASES: + assert elem <= _LC_VS_KILLED_WIDE_ELEMENT_ULP and rms <= _LC_VS_KILLED_WIDE_RMS_ULP, ( + elem, + rms, + ) + else: + assert elem <= _LC_VS_KILLED_ELEMENT_ULP and rms <= _LC_VS_KILLED_RMS_ULP, (elem, rms) + ref = _reference(c) + for out in (lc, stock): + e, r = _band(out, ref) + assert e <= _ELEMENT_ULP and r <= _RMS_ULP, (e, r) + + +# N1 (n1_ids): the layer's previous-step block ids, prefetched into L2 before the grid wait and overwritten with this +# step's. A hint: the rows must be bitwise the rows without it, whatever the buffer holds. + + +def _n1_expected(c, rows, before=None): + """The buffer after a launch: row ``tok`` < T holds topk[h, tok, r] for r below the token's block count, else -1; + rows at or past T keep what they held (``before``, default -1).""" + total_q, dql = c["q"].shape[0], c["dql"] + exp = ( + before.clone() + if before is not None + else torch.full((KV_HEADS, rows, TOPK), -1, dtype=torch.int32, device="cuda") + ) + for tok in range(min(total_q, rows)): + r, qo = divmod(tok, dql) + kv_len = max(int(c["seq"][r]) - dql + qo + 1, 0) + real = min(TOPK, (kv_len + PAGE - 1) // PAGE) + exp[:, tok] = -1 + exp[:, tok, :real] = c["topk"][:, tok, :real] + return exp + + +def _n1_garbage(rows, seed): + gen = torch.Generator(device="cuda").manual_seed(seed) + ids = torch.randint( + -5, 1 << 20, (KV_HEADS, rows, TOPK), dtype=torch.int32, device="cuda", generator=gen + ) + ids[0, :, 0] = 2**31 - 1 + ids[1, :, 0] = -(2**31) + return ids + + +# In-model request counts (1, 2, 4, 5, 8 Eagle3 verify requests of 4 rows), a full 128-row step, a plain decode. +_N1_CASES = { + "m1": dict(seq_lens=[165521]), + "m2": dict(seq_lens=[165521, 98765]), + "m4": dict(seq_lens=[165521, 120000, 700, 30000]), + "m5": dict(seq_lens=[165521, 4, 130, 98765, 5000]), + "m8": dict(seq_lens=[165521, 98765, 64000, 30000, 20000, 5000, 700, 130]), + "m32": dict(seq_lens=[4096 * (i + 1) + 77 * i for i in range(32)]), + "m2-dql1": dict(seq_lens=[165521, 5000], dql=1), + "k7-c5": _WIDE_CASES["k7-c5"], + "k7-c16": _WIDE_CASES["k7-c16"], +} + + +@pytest.mark.parametrize("name", list(_N1_CASES)) +def test_n1_rows_bitwise_and_ids_recorded(name) -> None: + c = _case(seed=11, **_N1_CASES[name]) + total_q = c["q"].shape[0] + for prewait in (False, True): + for mxfp8 in (False, True): + base, bq8, bsf = _run(c, prewait=prewait, mxfp8=mxfp8) + ids = torch.full((KV_HEADS, 128, TOPK), -1, dtype=torch.int32, device="cuda") + # First step (all -1: no prefetch), second (last step's ids: prefetch), then stale garbage ids. + for step, init in enumerate((None, None, _n1_garbage(128, 3))): + if init is not None: + ids.copy_(init) + exp = _n1_expected(c, 128, ids) + out, q8, sf = _run(c, prewait=prewait, mxfp8=mxfp8, n1=ids) + torch.cuda.synchronize() + assert torch.equal(out.view(torch.int16), base.view(torch.int16)), ( + prewait, + mxfp8, + step, + ) + if mxfp8: + assert torch.equal(q8, bq8) and torch.equal(sf, bsf), (prewait, mxfp8, step) + assert torch.equal(ids, exp), (prewait, mxfp8, step) + if total_q > 1: + # A buffer with fewer rows than the step: those rows record, the rest skip both prefetch and record. + rows = total_q // 2 + ids = torch.full((KV_HEADS, rows, TOPK), -1, dtype=torch.int32, device="cuda") + out, _, _ = _run(c, n1=ids) + assert torch.equal(out.view(torch.int16), _run(c)[0].view(torch.int16)) + assert torch.equal(ids, _n1_expected(c, rows)) + + +def test_n1_pdl_stress_and_graph_replay() -> None: + """Back-to-back PDL launches of two layers (each with its own id buffer, alternating steps), eager and replayed + from a CUDA graph: every launch writes the rows of the launch without N1, and the buffers end at the last step's + ids.""" + cases = [_case(seed=21, **_N1_CASES["m2"]), _case(seed=22, **_N1_CASES["m5"])] + bases = [_run(c, mxfp8=True)[0] for c in cases] + ids = [torch.full((KV_HEADS, 128, TOPK), -1, dtype=torch.int32, device="cuda") for _ in cases] + iters = 200 + outs = [[torch.empty_like(b) for _ in range(iters)] for b in bases] + for i in range(iters): + for j, c in enumerate(cases): + _run(c, mxfp8=True, n1=ids[j], use_pdl=True, out=outs[j][i]) + torch.cuda.synchronize() + for j in range(len(cases)): + assert all(torch.equal(o.view(torch.int16), bases[j].view(torch.int16)) for o in outs[j]), j + assert torch.equal(ids[j], _n1_expected(cases[j], 128)), j + for t in ids: + t.fill_(-1) + gouts = [torch.empty_like(b) for b in bases] + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream): + for j, c in enumerate(cases): # warm the launch configuration outside capture + _run(c, mxfp8=True, n1=ids[j], use_pdl=True, out=gouts[j]) + torch.cuda.synchronize() + with torch.cuda.graph(graph, stream=stream): + for _ in range(4): + for j, c in enumerate(cases): + _run(c, mxfp8=True, n1=ids[j], use_pdl=True, out=gouts[j]) + garbage = _n1_garbage(128, 5) + for t in ids: + t.copy_(garbage) + for _ in range(25): + for o in gouts: + o.zero_() + graph.replay() + torch.cuda.synchronize() + for j in range(len(cases)): + assert torch.equal(gouts[j].view(torch.int16), bases[j].view(torch.int16)), j + assert torch.equal(ids[j], _n1_expected(cases[j], 128, garbage)), j + + +# ---------------------------------------------------------------------------------- the target's TP2WIDE_LC2 dispatch + + +def _kernels(): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_attn_lc, + m3_prequant, + m3_report, + triton_sparse_decode, + ) + + return m3_attn_lc, m3_prequant, m3_report, triton_sparse_decode + + +# LC2_ALLROWS (LC2 from one row and past LC_MAX_ROWS) and LC2_IDENTITY (LC2 on identity steps) are default ON; the +# band / Triton-route contracts below are tested with both killed, their defaults in +# test_allrows_and_identity_defaults. +_BAND_KILLS = frozenset({"LC2_ALLROWS", "LC2_IDENTITY"}) + + +@pytest.fixture(autouse=True) +def _lc2_band_only(monkeypatch): + _, _, _, tsd = _kernels() + monkeypatch.setattr(tsd, "LC2_ALLROWS", False) + monkeypatch.setattr(tsd, "LC2_IDENTITY", False) + monkeypatch.setattr(tsd, "LC_KEYS", tsd.LC_KEYS - {"plain"}) + + +def _dispatch(c, monkeypatch, killed=False, n1=None): + """One step through the target's minimax_m3_sparse_attn_decode: (BF16 rows, LC2 calls, report paths taken).""" + m3_attn_lc, m3_prequant, m3_report, tsd = _kernels() + if killed: + monkeypatch.setattr(tsd, "LC_KEYS", frozenset()) + calls, taken = [], [] + real = m3_attn_lc.m3_sparse_decode_lc + monkeypatch.setattr( + m3_attn_lc, "m3_sparse_decode_lc", lambda *a, **k: (calls.append(a[-1]), real(*a, **k))[1] + ) + monkeypatch.setattr(m3_report, "taken", lambda path, rows, what="": taken.append(path)) + m3_attn_lc.N1_IDS.ids = n1 + total_q = c["q"].shape[0] + out = torch.empty(total_q, NUM_HEADS_TOTAL, DIM, dtype=torch.bfloat16, device="cuda") + try: + tsd.minimax_m3_sparse_attn_decode( + c["q"], + c["k"], + c["v"], + c["topk"], + c["bt"], + c["seq"], + sm_scale=SM_SCALE, + output=out, + decode_query_len=c["dql"], + k_block_scale=c["ksc"], + v_block_scale=c["vsc"], + k_global_scale=c["kgs"], + v_global_scale=c["vgs"], + ) + torch.cuda.synchronize() + finally: + m3_attn_lc.N1_IDS.ids = None + m3_prequant.SLOT.clear() + monkeypatch.undo() + return out.view(total_q, WIDTH), calls, taken + + +# Below lc_min_rows: 1 / 2 / 4 requests of the K7 verify (8 / 16 / 32 rows), 1 / 2 requests of the K3 verify (4 / 8 +# rows), a plain decode at 32 requests (32 rows). +_NARROW_CASES = { + "k7-c1": dict(seq_lens=[165521], dql=8), + "k7-c2": dict(seq_lens=[165521, 20000], dql=8), + "k7-c4": dict(seq_lens=[165521, 20000, 1285, 0], dql=8), + "k3-c1": dict(seq_lens=[165521], dql=4), + "k3-c2": dict(seq_lens=[165521, 20000], dql=4), + "dql1-c32": dict(seq_lens=[4096 * (i + 1) + 77 * i for i in range(31)] + [0], dql=1), +} +# TP2_LC2_SMALL: the K3 verify (4 query rows per request) at 3 / 4 / 8 requests (12 / 16 / 32 rows) takes LC2. +_K3_SMALL_CASES = { + "k3-c3": dict(seq_lens=[165521, 20000, 1285], dql=4), + "k3-c4": dict(seq_lens=[165521, 20000, 1285, 0], dql=4), + "k3-c8": dict(seq_lens=[165521, 98765, 20000, 2048, 1285, 136, 4, 0], dql=4), +} + +_K7_SMALL_CASES = { + "k7-c2": dict(seq_lens=[165521, 20000], dql=8), + "k7-c3": dict(seq_lens=[165521, 20000, 1285], dql=8), + "k7-c4": dict(seq_lens=[165521, 20000, 1285, 0], dql=8), +} + + +@pytest.mark.parametrize("name", list(_NARROW_CASES) + list(_K3_SMALL_CASES)) +def test_rows_up_to_32_keep_the_triton_route_bitwise(name, monkeypatch) -> None: + """With the small-row exception killed, dispatch never calls LC2 below 33 rows, and + writes the rows of the killed dispatch bit for bit; the N1 buffer is not touched.""" + _, _, _, tsd = _kernels() + monkeypatch.setattr(tsd, "TP2_LC2_SMALL", False) + c = _case(seed=41, **{**_NARROW_CASES, **_K3_SMALL_CASES}[name]) + assert c["q"].shape[0] < tsd.lc_min_rows(c["dql"]) + n1 = torch.full((KV_HEADS, 128, TOPK), -1, dtype=torch.int32, device="cuda") + on, calls, taken = _dispatch(c, monkeypatch, n1=n1) + killed, kcalls, _ = _dispatch(c, monkeypatch, killed=True) + assert not calls and not kcalls, "LC2 ran at most 32 rows" + assert tsd.LC_PATH not in taken and tsd.LC_TRITON_PATH in taken, taken + assert torch.equal(on.view(torch.int16), killed.view(torch.int16)) + assert bool((n1 == -1).all()), "the Triton route must not record N1 ids" + + +@pytest.fixture +def full_lc2_dispatch(): + """Exercise the full operator band independently of a serving vehicle's release hold.""" + import importlib + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + m31_switches, + ) + + _, _, _, tsd = _kernels() + try: + with pytest.MonkeyPatch.context() as patch: + patch.setattr( + m31_switches, + "_DISABLED", + (m31_switches._DISABLED - {"TP2WIDE_LC2", "TP2WIDE_LC2_448", "TP2_LC2_SMALL"}) + | _BAND_KILLS, + ) + patch.setattr(m31_switches, "_ENABLED", m31_switches._ENABLED - {"TP2_LC2_SMALL"}) + importlib.reload(tsd) + yield + finally: + importlib.reload(tsd) + + +@pytest.mark.parametrize("name", list(_WIDE_CASES) + list(_K3_SMALL_CASES) + list(_K7_SMALL_CASES)) +def test_wide_rows_take_lc2_with_n1_and_handoff(name, monkeypatch, full_lc2_dispatch) -> None: + """lc_min_rows(qlen)..LC_MAX_ROWS rows take LC2 with log-domain recurrence, MXFP8 hand-off and N1 ids. The + rows in the band of the float64 reference and bitwise the direct op call with the same settings; the kill switch + routes the step back to the Triton producer + merge.""" + _, _, _, tsd = _kernels() + if name in _K3_SMALL_CASES or name in _K7_SMALL_CASES: + monkeypatch.setattr(tsd, "TP2_LC2_SMALL", True) + monkeypatch.setattr(tsd, "LC2_SMALL_QLENS", frozenset({4, 8})) + c = _case(seed=43, **{**_WIDE_CASES, **_K3_SMALL_CASES, **_K7_SMALL_CASES}[name]) + total_q = c["q"].shape[0] + assert tsd.lc_min_rows(c["dql"]) <= total_q <= tsd.LC_MAX_ROWS + n1 = torch.full((KV_HEADS, tsd.LC_MAX_ROWS, TOPK), -1, dtype=torch.int32, device="cuda") + on, calls, taken = _dispatch(c, monkeypatch, n1=n1) + assert len(calls) == 1 and calls[0] is n1, "LC2 must run once and take the staged N1 buffer" + assert tsd.LC_PATH in taken and tsd.LC_TRITON_PATH not in taken, taken + assert total_q in tsd._TP2WIDE_LC2_LOGGED + direct, _, _ = _run(c, mxfp8=True) + assert torch.equal(on.view(torch.int16), direct.view(torch.int16)) + assert torch.equal(n1, _n1_expected(c, tsd.LC_MAX_ROWS)) + ref = _reference(c) + elem, rms = _band(on, ref) + assert elem <= _ELEMENT_ULP and rms <= _RMS_ULP, (elem, rms) + killed, kcalls, ktaken = _dispatch(c, monkeypatch, killed=True) + assert not kcalls and tsd.LC_PATH not in ktaken and tsd.LC_TRITON_PATH in ktaken, ktaken + elem, rms = _band(killed, ref) + assert elem <= _ELEMENT_ULP and rms <= _RMS_ULP, (elem, rms) + + +def test_identity_steps_never_take_lc2(monkeypatch) -> None: + """With LC2_IDENTITY killed, the identity selection (every request within 16 blocks) keeps the Triton producer + at every width.""" + m3_attn_lc, m3_prequant, _, tsd = _kernels() + c = _case(seed=45, seq_lens=[900, 2048, 130, 1500, 700], dql=8) + calls = [] + real = m3_attn_lc.m3_sparse_decode_lc + monkeypatch.setattr( + m3_attn_lc, "m3_sparse_decode_lc", lambda *a, **k: (calls.append(1), real(*a, **k))[1] + ) + out = torch.empty(c["q"].shape[0], NUM_HEADS_TOTAL, DIM, dtype=torch.bfloat16, device="cuda") + try: + tsd.minimax_m3_sparse_attn_decode( + c["q"], + c["k"], + c["v"], + c["topk"], + c["bt"], + c["seq"], + sm_scale=SM_SCALE, + output=out, + decode_query_len=c["dql"], + k_block_scale=c["ksc"], + v_block_scale=c["vsc"], + k_global_scale=c["kgs"], + v_global_scale=c["vgs"], + identity_blocks=True, + ) + torch.cuda.synchronize() + finally: + m3_prequant.SLOT.clear() + assert not calls + + +def test_kill_switch_contract(monkeypatch, full_lc2_dispatch) -> None: + """TRTLLM_M31_DISABLE_TP2WIDE_LC2=1 empties LC_KEYS (no step takes LC2) at import; unset, LC2 covers "mxfp8".""" + import importlib + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + ) + + _, _, _, tsd = _kernels() + assert tsd.TP2WIDE_LC2 and tsd.LC_KEYS == frozenset({"mxfp8"}) and tsd.LC_MIN_ROWS == 33 + assert ( + tsd.LC_MAX_ROWS == tsd.MergeMxfp8.MAX_ROWS == 448 + and tsd.MergeMxfp8.TRITON_MAX_ROWS == 128 + and tsd.TP2_LC2_SMALL + and tsd.lc_min_rows(8) == 9 + and tsd.lc_min_rows(4) == tsd.lc_min_rows(1) == 33 + ) + monkeypatch.setattr(m31_switches, "_DISABLED", frozenset({"TP2WIDE_LC2"})) + try: + m = importlib.reload(tsd) + assert not m.TP2WIDE_LC2 and m.LC_KEYS == frozenset() + finally: + monkeypatch.undo() + importlib.reload(tsd) + + +def test_lc2_448_kill_switch_contract(monkeypatch) -> None: + """TRTLLM_M31_DISABLE_TP2WIDE_LC2_448=1 restores the 128-row LC2 bound at import (LC_KEYS unchanged); the hand-off + buffers keep their 448-row capacity either way.""" + import importlib + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + ) + + _, _, _, tsd = _kernels() + monkeypatch.setattr(m31_switches, "_DISABLED", frozenset({"TP2WIDE_LC2_448"}) | _BAND_KILLS) + try: + m = importlib.reload(tsd) + assert m.TP2WIDE_LC2 and m.LC_KEYS == frozenset({"mxfp8"}) + assert m.LC_MAX_ROWS == m.MergeMxfp8.TRITON_MAX_ROWS == 128 and m.MergeMxfp8.MAX_ROWS == 448 + finally: + monkeypatch.undo() + importlib.reload(tsd) + + +@pytest.mark.parametrize("name", ["k3-c56", "k7-c46"]) +def test_lc2_448_killed_restores_the_triton_route(name, monkeypatch) -> None: + """Past 128 rows with the 448 bound killed (LC_MAX_ROWS = 128), the step takes the Triton producer + merge, bitwise + the step with LC2 killed entirely, and the Triton route publishes no MXFP8 hand-off there (its own bound, + MergeMxfp8.TRITON_MAX_ROWS, is unchanged by TP2WIDE_LC2_448).""" + _, m3_prequant, _, tsd = _kernels() + c = _case(seed=47, **_WIDE_CASES[name]) + total_q = c["q"].shape[0] + assert tsd.MergeMxfp8.TRITON_MAX_ROWS < total_q <= tsd.MergeMxfp8.MAX_ROWS + published = [] + real_publish = m3_prequant.SLOT.publish + monkeypatch.setattr( + m3_prequant.SLOT, "publish", lambda *a, **k: (published.append(1), real_publish(*a, **k))[1] + ) + monkeypatch.setattr(tsd, "LC_MAX_ROWS", tsd.MergeMxfp8.TRITON_MAX_ROWS) + bound, calls, taken = _dispatch(c, monkeypatch) + assert not calls and tsd.LC_PATH not in taken and tsd.LC_TRITON_PATH in taken, taken + assert not published, "the Triton route must not publish a hand-off past TRITON_MAX_ROWS" + killed, kcalls, _ = _dispatch(c, monkeypatch, killed=True) + assert not kcalls + assert torch.equal(bound.view(torch.int16), killed.view(torch.int16)) + + +def test_handoff_buffers_cover_whole_scale_tiles() -> None: + """The hand-off buffers hold every row and every whole 128-row scale tile up to LC_MAX_ROWS (448 is not a multiple + of 128: the last tile is written in full), and each step's hand-off slice is its whole tiles, never truncated.""" + _, _, _, tsd = _kernels() + q8, sf = tsd.MERGE_MXFP8.buffers(WIDTH, torch.device("cuda")) + assert q8.shape[0] >= tsd.LC_MAX_ROWS + assert sf.numel() >= (tsd.LC_MAX_ROWS + 127) // 128 * 128 * (WIDTH // 32) + for rows in (1, 33, 128, 129, 184, 224, 256, 368, tsd.LC_MAX_ROWS): + hq, hs = tsd.MergeMxfp8.handoff((q8, sf), rows) + assert hq.shape[0] == rows and hs.numel() == (rows + 127) // 128 * 128 * (WIDTH // 32), rows + + +@pytest.mark.parametrize( + "enabled,disabled,k3_min,k7_min", + [(False, False, 33, 9), (True, False, 9, 9), (True, True, 33, 33), (False, True, 33, 33)], +) +def test_lc2_small_opt_in_contract(enabled, disabled, k3_min, k7_min, monkeypatch) -> None: + """K7 defaults ON, K3 requires opt-in, and the kill switch restores both lower bounds.""" + import importlib + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + m31_switches, + ) + + _, _, _, tsd = _kernels() + monkeypatch.setattr( + m31_switches, "_ENABLED", frozenset({"TP2_LC2_SMALL"}) if enabled else frozenset() + ) + monkeypatch.setattr( + m31_switches, + "_DISABLED", + (frozenset({"TP2_LC2_SMALL"}) if disabled else frozenset()) | _BAND_KILLS, + ) + try: + m = importlib.reload(tsd) + assert m.TP2_LC2_SMALL == (not disabled) + assert m.lc_min_rows(4) == k3_min + assert m.lc_min_rows(8) == k7_min + assert m.lc_min_rows(1) == m.lc_min_rows(2) == 33 + assert m.LC_MAX_ROWS == 448 and m.LC_KEYS == frozenset({"mxfp8"}) + finally: + monkeypatch.undo() + importlib.reload(tsd) + + +def test_allrows_and_identity_defaults(monkeypatch) -> None: + """Default ON: LC2 from one row with the "plain" key past LC_MAX_ROWS (LC2_ALLROWS) and on identity steps + (LC2_IDENTITY); each kill switch turns only its own item off, and TP2WIDE_LC2 killed turns both off.""" + import importlib + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + ) + + _, _, _, tsd = _kernels() + try: + for disabled, allrows, identity in ( + (frozenset(), True, True), + (frozenset({"LC2_ALLROWS"}), False, True), + (frozenset({"LC2_IDENTITY"}), True, False), + (frozenset({"TP2WIDE_LC2"}), False, False), + ): + monkeypatch.setattr(m31_switches, "_DISABLED", disabled) + monkeypatch.setattr(m31_switches, "_ENABLED", frozenset()) + m = importlib.reload(tsd) + assert (m.LC2_ALLROWS, m.LC2_IDENTITY) == (allrows, identity), disabled + if allrows: + assert m.LC_KEYS == frozenset({"mxfp8", "plain"}) + assert m.lc_min_rows(1) == m.lc_min_rows(4) == m.lc_min_rows(8) == 1 + elif m.TP2WIDE_LC2: + assert m.LC_KEYS == frozenset({"mxfp8"}) and m.lc_min_rows(8) == 9 + finally: + monkeypatch.undo() + importlib.reload(tsd) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode_lc_split.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode_lc_split.py new file mode 100644 index 000000000000..e0569e7fca26 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_sparse_decode_lc_split.py @@ -0,0 +1,139 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU tests for TP2WIDE_LC2_SPLIT: trtllm::minimax_m3_sm107_tp2_sparse_decode_lc runs LC2-split (LC2 without its +thread-block cluster: a per-(row, block) grid storing LC2's partials, then a merge grid in LC2's slot order) unless +TRTLLM_M31_DISABLE_TP2WIDE_LC2_SPLIT=1. trtllm::minimax_m3_sm107_tp2_sparse_decode_lc_cluster always runs the cluster +kernel. + +Contract: on every case of test_modeling_v2_m3_sparse_decode_lc.py (the M3 cases, the K3 / K7 wide steps to 448 rows, +the N1 cases), in every variant (log-domain / linear softmax, pre-wait staging, MXFP8 hand-off on / off, N1 on / off, +PDL on / off), the split writes the same BF16 rows, MXFP8 bytes, swizzled scales and N1 ids as the cluster kernel, bit +for bit, eagerly and replayed from a CUDA graph. +""" + +import importlib.util +import os +import sys + +import pytest +import torch + +import tensorrt_llm # noqa: F401 (registers torch.ops.trtllm) + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +# The sibling LC2 test's case builder; it imports _validating from the tests root, as its __extra_import_path__ says. +sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../.."))) +_spec = importlib.util.spec_from_file_location( + "lc2_cases", os.path.join(os.path.dirname(__file__), "test_modeling_v2_m3_sparse_decode_lc.py") +) +lc = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(lc) + +_ALL = {**lc._CASES, **lc._WIDE_CASES, **{f"n1-{k}": v for k, v in lc._N1_CASES.items()}} + + +def _split_enabled(): + return os.environ.get("TRTLLM_M31_DISABLE_TP2WIDE_LC2_SPLIT", "0") != "1" + + +def _call(op, c, out, q8, sf, linear, use_pdl, prewait, n1): + op( + c["q"].view(torch.uint8), + c["k"], + c["v"], + c["ksc"], + c["vsc"], + c["kgs"], + c["vgs"], + c["topk"], + c["bt"], + c["seq"], + out, + q8, + sf, + lc.WIDTH // 128, + c["dql"], + lc.SM_SCALE, + linear, + use_pdl, + prewait, + n1, + ) + + +def _buffers(total_q, mxfp8): + out = torch.full((total_q, lc.WIDTH), 7.0, dtype=torch.bfloat16, device="cuda") + if not mxfp8: + return out, None, None + q8 = torch.full((total_q, lc.WIDTH), 0x55, dtype=torch.uint8, device="cuda") + sf = torch.full( + (128 * (lc.WIDTH // 32) * ((total_q + 127) // 128),), 0x33, dtype=torch.uint8, device="cuda" + ) + return out, q8, sf + + +def _both(c, linear, use_pdl, prewait, mxfp8, n1_init): + total_q = c["q"].shape[0] + res = [] + for op in ( + torch.ops.trtllm.minimax_m3_sm107_tp2_sparse_decode_lc, + torch.ops.trtllm.minimax_m3_sm107_tp2_sparse_decode_lc_cluster, + ): + out, q8, sf = _buffers(total_q, mxfp8) + n1 = n1_init.clone() if n1_init is not None else None + _call(op, c, out, q8, sf, linear, use_pdl, prewait, n1) + torch.cuda.synchronize() + res.append((out, q8, sf, n1)) + return res + + +def _assert_same(a, b, what): + for x, y, name in zip(a, b, ("out", "q8", "sf", "n1")): + if x is None: + continue + assert torch.equal(x.view(torch.uint8), y.view(torch.uint8)), (what, name) + + +@pytest.mark.skipif(not _split_enabled(), reason="TRTLLM_M31_DISABLE_TP2WIDE_LC2_SPLIT=1") +@pytest.mark.parametrize("name", list(_ALL)) +@pytest.mark.parametrize("linear", [False, True]) +@pytest.mark.parametrize("prewait", [False, True]) +def test_split_bitwise_vs_cluster(name, linear, prewait) -> None: + c = lc._case(seed=7, **_ALL[name]) + total_q = c["q"].shape[0] + for mxfp8 in (False, True): + for use_pdl in (False, True): + for n1_init in (None, lc._n1_garbage(max(total_q, 1), 5)): + split, cluster = _both(c, linear, use_pdl, prewait, mxfp8, n1_init) + _assert_same( + split, cluster, (name, linear, prewait, mxfp8, use_pdl, n1_init is not None) + ) + + +@pytest.mark.skipif(not _split_enabled(), reason="TRTLLM_M31_DISABLE_TP2WIDE_LC2_SPLIT=1") +@pytest.mark.parametrize("name", ["k7-c24", "k3-c56", "k7-c56", "b4"]) +def test_split_graph_replay_bitwise(name) -> None: + """The split captured in a CUDA graph (its workspace is allocated by the eager warmup call) and replayed twice + writes what the cluster kernel writes eagerly.""" + c = lc._case(seed=9, **_ALL[name]) + total_q = c["q"].shape[0] + ref = _both(c, False, True, False, True, None)[1] + out, q8, sf = _buffers(total_q, True) + op = torch.ops.trtllm.minimax_m3_sm107_tp2_sparse_decode_lc + _call(op, c, out, q8, sf, False, True, False, None) # eager warmup + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + _call(op, c, out, q8, sf, False, True, False, None) + for _ in range(2): + out.fill_(7.0) + q8.fill_(0x55) + sf.fill_(0x33) + graph.replay() + torch.cuda.synchronize() + _assert_same((out, q8, sf, None), ref, name) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_tokmap_guard.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_tokmap_guard.py new file mode 100644 index 000000000000..49a516aea0a5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_m3_tokmap_guard.py @@ -0,0 +1,226 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7) for kernels/m3_tokmap_guard: a one-request multi-token call of +thop.trtllm_gen_generation_preprocess (rows padded to the bucket, the request's generation length 1..bucket) with a +stale token map in the workspace gives, through the guard, the bytes the op gives with a freshly built map: the +appended FP8 K/V pages and the processed query's live rows, bit for bit, eager and from one captured graph replayed for +every length. Without the guard the stale map moves rows (the negative control), and a several-request call passes +through with its own map untouched. + +The fresh-map reference forces the op's own build (a scaled RoPE type with no inv-freq cache, at rotary dim 0, so no +row is rotated either way). +""" + +import math + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_tokmap_guard as guard, +) +from tensorrt_llm.bindings.internal import thop + +assert torch.cuda.is_available(), "m3_tokmap_guard requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp2 target is certified on SM 10.7 only", allow_module_level=True) + +H, KVH, D, PAGE = 32, 2, 128, 128 # the TP2 per-rank heads (64 / 2 query, 4 / 2 KV) +BUCKET = 4 +_PEER_EXTRA = 777 # the second request's extra length behind the first +FP8_KV_QUANT_MODE = 128 +_DEV = torch.device("cuda") # unindexed on purpose: install() and the lookup normalize it +_TAKEN = [] + + +@pytest.fixture(scope="module", autouse=True) +def _installed(): + guard.install(_DEV, 48, _TAKEN.append) + yield + + +class _Step: + """One request (or ``nreq``) of BUCKET padded rows over a fresh FP8 paged cache, persistent across calls like a + real step's buffers.""" + + def __init__(self, kv_len: int, nreq: int = 1, seed: int = 0): + g = torch.Generator(device=_DEV).manual_seed(seed + kv_len) + self.nreq = nreq + # Per-request inputs (page tables, generation lengths, position offsets) for at least two requests, the page + # tables at the peer's longer length (_seq_with_peer): the stale map sends rows to request 1, which must have + # real entries behind it, also on the unguarded calls. + tables = max(nreq, 2) + pages = math.ceil((kv_len + _PEER_EXTRA + BUCKET) / PAGE) + self.nblocks = tables * pages + 2 + self.pool0 = ( + torch.randn(self.nblocks, 2, KVH, PAGE, D, device=_DEV, generator=g) + .clamp_(-4, 4) + .to(torch.float8_e4m3fn) + ) + self.pool = self.pool0.clone() + perm = torch.randperm(self.nblocks, device=_DEV, generator=g)[: tables * pages].view( + tables, pages + ) + self.offsets = torch.zeros(1, tables, 2, pages, dtype=torch.int32, device=_DEV) + self.offsets[0, :, 0] = 2 * perm + self.offsets[0, :, 1] = 2 * perm + 1 + self.pointers = torch.tensor([[self.pool.data_ptr(), 0]], dtype=torch.int64) + self.mapping = torch.zeros(1, 2, dtype=torch.int32) + self.qkv = torch.randn( + nreq * BUCKET, (H + 2 * KVH) * D, dtype=torch.bfloat16, device=_DEV, generator=g + ) + # Request lengths after this step's append; generation lengths (accepted rows) set per case. + self.seq = torch.full((nreq,), kv_len, dtype=torch.int32, device=_DEV) + self.gen = torch.full((tables,), BUCKET, dtype=torch.int32, device=_DEV) + self.pos = ( + torch.arange(BUCKET, dtype=torch.int32, device=_DEV).repeat(tables, 1).contiguous() + ) + self.one = torch.ones(1, dtype=torch.float32, device=_DEV) + self.ws = torch.empty(64 << 20, dtype=torch.uint8, device=_DEV) + self.kv_len = kv_len + + def call(self, op, fresh_map: bool = False): + """One generation preprocess; returns a copy of the processed query.""" + out = op( + self.qkv, + self.ws, + self.seq, + self.gen, + self.pos, + self.offsets, + self.pointers, + self.mapping, + self.one, + self.one, + None, + None, + None, + None, + 0, + 0, + H, + KVH, + D, + PAGE, + FP8_KV_QUANT_MODE, + 1 << 20, + 1 << 20, + self.nreq * BUCKET, + self.nreq, + BUCKET, + self.kv_len, + 0, + 10000.0, + 1 if fresh_map else 0, + 1.0, + 1 << 20, + 0, + 1.0, + 1.0, + False, + BUCKET, + 0, + torch.cuda.get_device_properties(0).multi_processor_count, + self.nblocks, + 2, + True, + False, + ) + return out[0].clone() + + def poison_map(self) -> None: + """A stale multi-request map in the op's one-request map slice: rows 1.. on request 1.""" + layout = thop.get_trtllm_gen_generation_workspace_layout( + self.qkv.dtype, 1, self.nreq * BUCKET, H, D, 0, KVH, 0, False, False + ) + off = int(layout["tokens_info_offset"]) + stale = torch.tensor( + [[0, 0]] + [[1, t] for t in range(BUCKET - 1)], dtype=torch.int32, device=_DEV + ) + self.ws[off : off + 8 * BUCKET].view(torch.int32).view(BUCKET, 2).copy_(stale) + + def reset(self) -> None: + self.pool.copy_(self.pool0) + + +def _seq_with_peer(step: _Step) -> None: + # A second request's length behind the first, so a row the stale map moves to request 1 reads a real length. + step.seq = torch.tensor( + [step.kv_len, step.kv_len + _PEER_EXTRA], dtype=torch.int32, device=_DEV + ) + + +@pytest.mark.parametrize("kv_len", [130, 8191, 34117]) +@pytest.mark.parametrize("n", [1, 2, 3, 4]) +def test_guard_equals_a_fresh_map(kv_len, n) -> None: + step = _Step(kv_len) + _seq_with_peer(step) + step.gen.fill_(n) + step.poison_map() + step.reset() + q_ref = step.call(guard._original, fresh_map=True) + pool_ref = step.pool.clone() + + step.poison_map() + step.reset() + q = step.call(thop.trtllm_gen_generation_preprocess) + # The processed query's n live rows (rows past the request's length are padding the op leaves unwritten). + assert torch.equal(q[:n].view(torch.uint8), q_ref[:n].view(torch.uint8)) + assert torch.equal(step.pool.view(torch.uint8), pool_ref.view(torch.uint8)) + assert BUCKET in _TAKEN + + +def test_stale_map_moves_rows_without_the_guard() -> None: + step = _Step(34117) + _seq_with_peer(step) + step.gen.fill_(BUCKET) + step.poison_map() + step.reset() + step.call(guard._original, fresh_map=True) + pool_ref = step.pool.clone() + step.poison_map() + step.reset() + step.call(guard._original) + assert not torch.equal(step.pool.view(torch.uint8), pool_ref.view(torch.uint8)) + + +def test_one_captured_graph_serves_every_length() -> None: + step = _Step(34000) + _seq_with_peer(step) + refs = {} + for n in range(1, BUCKET + 1): + step.gen.fill_(n) + step.poison_map() + step.reset() + q = step.call(guard._original, fresh_map=True) + refs[n] = (q, step.pool.clone()) + step.gen.fill_(BUCKET) + step.call(thop.trtllm_gen_generation_preprocess) # eager first, as the warmup runs it + torch.cuda.synchronize() + g = torch.cuda.CUDAGraph() + with torch.cuda.graph(g): + out = thop.trtllm_gen_generation_preprocess + q_static = step.call(out) + for n in range(1, BUCKET + 1): + step.gen.fill_(n) + step.poison_map() + step.reset() + g.replay() + torch.cuda.synchronize() + q_ref, pool_ref = refs[n] + assert torch.equal(q_static[:n].view(torch.uint8), q_ref[:n].view(torch.uint8)), n + assert torch.equal(step.pool.view(torch.uint8), pool_ref.view(torch.uint8)), n + + +def test_several_requests_pass_through() -> None: + step = _Step(8191, nreq=2) + layout = thop.get_trtllm_gen_generation_workspace_layout( + step.qkv.dtype, 2, 2 * BUCKET, H, D, 0, KVH, 0, False, False + ) + step.reset() + step.call(thop.trtllm_gen_generation_preprocess) + pool_guarded = step.pool.clone() + step.reset() + step.call(guard._original) + assert torch.equal(step.pool.view(torch.uint8), pool_guarded.view(torch.uint8)) + assert int(layout["tokens_info_size"]) >= 8 * 2 * BUCKET diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_msa_kv_update.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_msa_kv_update.py new file mode 100644 index 000000000000..d27df3f8e655 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/attention/test_modeling_v2_msa_kv_update.py @@ -0,0 +1,133 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the msa_kv_update entry (the one-launch MSA KV-length correction).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.msa_kv_update import ( # noqa: E501 + msa_kv_update as hook, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.msa_kv_update import ( # noqa: E501 + TOKENS_PER_BLOCK, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.msa_kv_update import ( # noqa: E501 + msa_kv_update as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "msa_kv_update requires a CUDA device" + +#: Spare capacity past the live rows, as the metadata's buffers have: the op must leave it alone. +_SPARE = 3 + + +def _case(spec, seed): + """The metadata's buffers for one step, with stale values in every output so a skipped write shows.""" + g = torch.Generator(device="cpu").manual_seed(seed) + batch, qlen = spec["batch"], spec["query_len"] + total_q = batch * qlen + staged = torch.randint(qlen, 5000, (batch,), generator=g, dtype=torch.int32) + accept = spec["accept"] + if accept == "full": + kv = staged.clone() + elif accept == "page_edge": + staged = torch.tensor([130, 257, 2 * TOKENS_PER_BLOCK + qlen][:batch], dtype=torch.int32) + kv = staged - torch.tensor([0, 1, qlen - 1][:batch], dtype=torch.int32) + else: + kv = staged - torch.randint(0, qlen, (batch,), generator=g, dtype=torch.int32) + if accept == "padded": + kv[batch // 2 :] = 0 + staged[batch // 2 :] = 0 + ncols = (5000 + qlen) // TOKENS_PER_BLOCK + 2 + table = ( + torch.randperm(batch * ncols + 7, generator=g)[: batch * ncols] + .to(torch.int32) + .reshape(batch, ncols) + ) + dev = dict(device="cuda", dtype=torch.int32) + stale = -12345 + return ( + torch.cat([kv, torch.full((_SPARE,), 99, dtype=torch.int32)]).cuda(), + torch.cat([staged, torch.full((_SPARE,), 99, dtype=torch.int32)]).cuda(), + torch.full((batch + _SPARE,), qlen, **dev), + torch.arange(batch, **dev).repeat_interleave(qlen), + torch.arange(qlen, **dev).repeat(batch), + table.cuda(), + TOKENS_PER_BLOCK, + batch, + total_q, + torch.full((batch + _SPARE,), stale, **dev), + torch.full((batch + 1 + _SPARE,), stale, **dev) if spec["cu_kv"] else None, + torch.full((total_q + _SPARE,), stale, **dev), + torch.full((total_q + _SPARE,), stale, **dev) if spec["n_valid"] else None, + ) + + +def _check(args, ref): + seq, cu_kv, loc, n_valid = args[9], args[10], args[11], args[12] + batch, total_q = args[7], args[8] + ref_seq, ref_cu, ref_loc, ref_nv = ref + # Integer bookkeeping: exact, and nothing past the live rows is touched. + assert torch.equal(seq[:batch], ref_seq) + assert torch.equal(loc[:total_q], ref_loc) + assert bool((seq[batch:] == -12345).all()) and bool((loc[total_q:] == -12345).all()) + assert (cu_kv is None) == (ref_cu is None) and (n_valid is None) == (ref_nv is None) + if cu_kv is not None: + assert torch.equal(cu_kv[: batch + 1], ref_cu) + assert bool((cu_kv[batch + 1 :] == -12345).all()) + if n_valid is not None: + assert torch.equal(n_valid[:total_q], ref_nv) + assert bool((n_valid[total_q:] == -12345).all()) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + certifying(op) + op(*args) + _check(args, op.reference(*args)) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_the_hook_takes_every_cell(cell) -> None: + """The target's hook (what the metadata calls) takes each certified step and writes what the op does.""" + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31 + 1) + assert hook(*args) is True + _check(args, op.reference(*args)) + + +def test_the_hook_declines_what_the_op_would_misread() -> None: + """A strided operand is read wrongly by the op, so the hook declines it (the metadata then runs its own + torch update) and writes nothing.""" + args = list(_case(op.CELLS[2].spec, seed=5)) + args[0] = torch.stack([args[0], args[0]], dim=1)[:, 0] + before = args[11].clone() + assert hook(*args) is False + assert torch.equal(args[11], before) + + +def test_guard_refuses_a_strided_operand() -> None: + args = list(_case(op.CELLS[2].spec, seed=6)) + args[0] = torch.stack([args[0], args[0]], dim=1)[:, 0] + with pytest.raises(AssertionError, match="contiguous"): + with validating(op): + op(*args) + + +def test_guard_refuses_a_request_index_past_the_batch() -> None: + args = list(_case(op.CELLS[2].spec, seed=7)) + args[3] = args[3].clone() + args[3][-1] = args[7] + with pytest.raises(AssertionError, match="q_batch_row"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_m3_allreduce_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_m3_allreduce_op_matrix.py new file mode 100644 index 000000000000..cb4316cf4703 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_m3_allreduce_op_matrix.py @@ -0,0 +1,440 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU certification matrix for the ``m3_allreduce`` entry, 2 ranks (SM107): its cells against the entry's reference, +and the fused draft loop's layer boundaries built on it (``M3Eagle3Draft.ar_norm``: the plain all-reduce, then +FlashInfer's fused add + RMSNorm) against stock Eagle3's, bit for bit. + +Stock ``Eagle3DecoderLayer`` all-reduces each projection plainly (BF16 out) and runs the ``RMSNorm`` module with the +residual: FlashInfer's ``fused_add_rmsnorm``, which adds in FP32 and normalizes the FP32 sum. The draft loop's +boundaries must be that path exactly, under every all-reduce strategy the single-node drafter can run (AUTO, +ONESHOT, TWOSHOT, NCCL; MNNVL is multi-node only), at tp2 shapes (H = 6144, eps = 1e-6, 1 to 32 rows). The inputs are +random rows and rows built so a BF16 rounding of the residual sum would move the norm the most, including the +review's case: unit weights, sums 1 + 2^-8 on 6143 coordinates and 2 - 2^-8 on the last, both BF16 rounding +midpoints that round to even in opposite directions. + +Asserts the normalized rows and the residual sum bitwise equal to the stock path's, and reports the max ulp of each +against an fp64 reference (ulps of BF16 at the larger magnitude; a correctly rounded output is within 0.5). + +Its own 2-rank launcher, as ``_m3_mc_allreduce_norm_op_matrix.py`` is; the collected entry point is +``test_modeling_v2_m3_allreduce.py``. + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _m3_allreduce_op_matrix.py +""" + +import math +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Callable, List, Optional, Tuple + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "the fused all-reduce + RMSNorm requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 + +HIDDEN = 6144 +EPS = 1e-6 +TOKENS = (1, 4, 8, 32) +RESULT_TAG = "[ulp]" +"""Prefix of rank 0's result lines; the pytest entry reports them.""" + +RANK = 0 +WORLD = 1 + +Inputs = Tuple[List[torch.Tensor], torch.Tensor, torch.Tensor] +"""(every rank's [m, H] BF16 partial rows, the [m, H] BF16 residual, the [H] BF16 norm weight).""" + + +# ---------------------------------------------------------------- inputs (every rank builds every rank's rows) + + +def _gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def _random(seed: int, m: int, x_scale: float, r_scale: float, unit_w: bool) -> Inputs: + g = _gen(seed) + xs = [ + (torch.randn(m, HIDDEN, generator=g, device="cuda") * x_scale).bfloat16() + for _ in range(WORLD) + ] + residual = (torch.randn(m, HIDDEN, generator=g, device="cuda") * r_scale).bfloat16() + if unit_w: + w = torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16) + else: + w = (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).bfloat16() + return xs, residual, w + + +def _on_rank0(m: int, x: torch.Tensor) -> List[torch.Tensor]: + """``x`` as rank 0's partial, zeros elsewhere (the cross-rank sum is exact).""" + return [x.bfloat16()] + [ + torch.zeros(m, HIDDEN, device="cuda", dtype=torch.bfloat16) for _ in range(WORLD - 1) + ] + + +def _review_case(m: int) -> Inputs: + """Sums 1 + 2^-8 on the first H - 1 coordinates and 2 - 2^-8 on the last, unit weights. + + Both sums are BF16 midpoints: 1 + 2^-8 rounds (to even) down to 1 and 2 - 2^-8 up to 2, so the fused path's + norm sees a residual moved in opposite directions on the bulk and on the last coordinate. + """ + residual = torch.ones(m, HIDDEN, device="cuda") + residual[:, -1] = 2.0 - 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + return ( + _on_rank0(m, x), + residual.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _review_case_flipped(m: int) -> Inputs: + """The mirror: the bulk on odd-mantissa midpoints (1 + 3 * 2^-8 rounds up), the last one rounding down.""" + residual = torch.full((m, HIDDEN), 1.0 + 2.0**-7, device="cuda") + residual[:, -1] = 2.0 - 2.0 * 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + return ( + _on_rank0(m, x), + residual.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _midpoints(seed: int, m: int, outliers: int) -> Inputs: + """Every sum a BF16 midpoint: residual r random in [1, 2) and x = half its ulp (+2^-8), split over the ranks. + + The bulk has an odd mantissa (rounds up); ``outliers`` coordinates per row carry an even one (rounds down) and + a larger magnitude, so they weigh in the RMS while rounding the other way. + """ + g = _gen(seed) + mant = ( + torch.randint(0, 64, (m, HIDDEN), generator=g, device="cuda") * 2 + 1 + ) # odd 7-bit mantissa + r = 1.0 + mant.float() * 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + if outliers: + cols = torch.randperm(HIDDEN, generator=g, device="cuda")[:outliers] + even = torch.randint(0, 64, (m, outliers), generator=g, device="cuda") * 2 + r[:, cols] = 8.0 * (1.0 + even.float() * 2.0**-7) + x[:, cols] = 8.0 * 2.0**-8 + # Split x over the ranks in exact BF16 pieces: x/2 on each rank. + parts = [x / 2, x / 2] + return ( + [p.bfloat16() for p in parts[:WORLD]], + r.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _cancelling(seed: int, m: int) -> Inputs: + """Large partials that cancel across the ranks (+A + B, -A - B + d) onto a unit residual.""" + g = _gen(seed) + a = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 64).bfloat16().float() + b = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 64).bfloat16().float() + d = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 2.0**-6).bfloat16().float() + parts = [a + b, (-a - b + d)] + residual = (1.0 + torch.rand(m, HIDDEN, generator=g, device="cuda")).bfloat16() + w = (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).bfloat16() + return [p.bfloat16() for p in parts[:WORLD]], residual, w + + +def _cases() -> List[Tuple[str, Callable[[int], Inputs]]]: + cases: List[Tuple[str, Callable[[int], Inputs]]] = [ + ("review: 1+2^-8 x6143, 2-2^-8 x1", _review_case), + ("review mirrored", _review_case_flipped), + ("all midpoints, no outliers", lambda m: _midpoints(11, m, 0)), + ("all midpoints, 1 outlier", lambda m: _midpoints(12, m, 1)), + ("all midpoints, 64 outliers", lambda m: _midpoints(13, m, 64)), + ("cancelling partials", lambda m: _cancelling(14, m)), + ] + for i, (xs, rs, unit) in enumerate( + ((1.0, 4.0, False), (1.0, 1.0, True), (0.05, 8.0, False), (8.0, 0.5, False)) + ): + cases.append( + ( + f"random x{xs} r{rs}{' unit-w' if unit else ''}", + lambda m, i=i, xs=xs, rs=rs, unit=unit: _random(100 + i, m, xs, rs, unit), + ) + ) + return cases + + +# ---------------------------------------------------------------- the three paths + + +def _fp64(xs: List[torch.Tensor], residual: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + s = residual.double() + for x in xs: + s = s + x.double() + rms = torch.rsqrt(s.square().mean(-1, keepdim=True) + EPS) + return s * rms * w.double() + + +def _ulp_at(v: torch.Tensor) -> torch.Tensor: + """BF16's ulp at |v| (fp64): 2^(e - 8) for |v| in [2^(e-1), 2^e), floored at the smallest normal's.""" + _, e = torch.frexp(v.abs().double()) + return torch.ldexp(torch.ones_like(v, dtype=torch.float64), torch.clamp(e - 8, min=-133)) + + +def _ulps(out: torch.Tensor, ref: torch.Tensor) -> float: + """Max |out - ref| in BF16 ulps at the larger magnitude of the two.""" + o, r = out.double(), ref.double() + return ((o - r).abs() / _ulp_at(torch.maximum(o.abs(), r.abs()))).max().item() + + +def _ulps_at(out: torch.Tensor, ref: torch.Tensor, scale: torch.Tensor) -> float: + """Max |out - ref| in BF16 ulps at the magnitude ``scale`` (the operands'), not at the result's.""" + return ((out.double() - ref.double()).abs() / _ulp_at(scale.abs())).max().item() + + +def _target(ar: Any, x: torch.Tensor, residual: torch.Tensor, w: torch.Tensor): + """The draft loop's boundary (``M3Eagle3Draft.ar_norm``): ``m3_allreduce`` plain, then the catalog's + ``flashinfer_fused_add_rmsnorm``.""" + from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( # noqa: E501 + flashinfer_fused_add_rmsnorm, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.comm.m3_allreduce import ( # noqa: E501 + m3_allreduce, + ) + + return flashinfer_fused_add_rmsnorm(m3_allreduce(x, ar, None), residual.clone(), w, EPS) + + +def _check_cells(mapping: Any) -> None: + """Every cell of ``m3_allreduce.CELLS`` (AUTO strategy, as the target builds it) against the entry's reference: + rank-order fp32 sums with one BF16 rounding of the sum and one of the norm. A strategy may round the cross-rank sum + to BF16 before adding the residual, so the residual sum is bounded at its operands' magnitude + u = ulp(max(|residual|, |sum|)): half an ulp for that rounding plus half an output ulp for each of the two final + roundings, an output ulp being at most 2u, so 2.5u (where the two cancel, the output's own ulp is far smaller). + The normed rows are the norm of the op's own residual sum, within 2 ulps of the reference norm of that sum.""" + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.comm.m3_allreduce import ( # noqa: E501 + m3_allreduce, + ) + from tensorrt_llm._torch.distributed import AllReduce, AllReduceFusionOp, AllReduceParams + from tensorrt_llm.functional import AllReduceStrategy + + sys.path.insert(0, str(Path(__file__).resolve().parents[4])) + from _validating import validating + + ar = AllReduce(mapping=mapping, strategy=AllReduceStrategy.AUTO, dtype=torch.bfloat16) + for i, cell in enumerate(m3_allreduce.CELLS): + spec = cell.spec + xs, residual, w = _random(200 + i, spec["m"], 1.0, 4.0, False) + params = ( + None + if spec.get("plain") + else AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual.clone(), + norm_weight=w, + eps=EPS, + trigger_completion_at_end=False, + ) + ) + # The reference first: the fused path may write the updated residual into ``params.residual``. + ref = m3_allreduce.reference(xs, ar, params) + with validating(m3_allreduce): + out = m3_allreduce(xs[RANK], ar, params) + torch.cuda.synchronize() + total = sum(x.double() for x in xs) + if params is None: + err = _ulps_at(out, ref, total) + assert err <= 1.0, (cell.why, err) + else: + err = _ulps_at(out[1], ref[1], torch.maximum(total.abs(), residual.double().abs())) + assert err <= 2.5, (cell.why, "residual", err) + h = out[1].double() + own = h * torch.rsqrt(h.square().mean(-1, keepdim=True) + EPS) * w.double() + assert _ulps(out[0], own) <= 2.0, (cell.why, "normed", _ulps(out[0], own)) + dist.barrier() + if RANK == 0: + print(f"{RESULT_TAG} cell {cell.why} passed", flush=True) + + +def _stock(ar: Any, x: torch.Tensor, residual: torch.Tensor, w: torch.Tensor): + """Stock Eagle3: ``o_proj``'s plain all-reduce (BF16 out), then ``RMSNorm(h, residual)``'s FlashInfer + ``fused_add_rmsnorm`` (FP32 add, norm of the FP32 sum; in place on both).""" + h = ar(x) + res = residual.clone() + torch.ops.trtllm.flashinfer_fused_add_rmsnorm(h, res, w, EPS) + return h, res + + +# ---------------------------------------------------------------- the matrix + + +def _single_node_cluster_info() -> None: + """``TorchDist`` (the communicator under ``TLLM_DISABLE_MPI``) asks Ray for each rank's node and GPU to build its + node-local group. These ranks are spawned without Ray on one node, so every rank reports the same node and its + own device.""" + from tensorrt_llm._torch.distributed.communicator import TorchDist + + def cluster_info(self): + if self.cluster_info is None: + self.cluster_info = [("127.0.0.1", r) for r in range(dist.get_world_size())] + return self.cluster_info + + TorchDist._get_cluster_info = cluster_info + + +def _run_matrix() -> None: + import tensorrt_llm._torch.custom_ops # noqa: F401 -- registers trtllm::flashinfer_fused_add_rmsnorm + from tensorrt_llm._torch.distributed import AllReduce + from tensorrt_llm._torch.flashinfer_utils import IS_FLASHINFER_AVAILABLE + from tensorrt_llm.functional import AllReduceStrategy + from tensorrt_llm.mapping import Mapping + + assert IS_FLASHINFER_AVAILABLE, "stock Eagle3's add + RMSNorm is FlashInfer's fused_add_rmsnorm" + _single_node_cluster_info() + mapping = Mapping(world_size=WORLD, tp_size=WORLD, rank=RANK, gpus_per_node=WORLD) + _check_cells(mapping) + strategies = ( + AllReduceStrategy.AUTO, + AllReduceStrategy.ONESHOT, + AllReduceStrategy.TWOSHOT, + AllReduceStrategy.NCCL, + ) + worst = 0.0 + mismatches: List[str] = [] + for strategy in strategies: + ar = AllReduce(mapping=mapping, strategy=strategy, dtype=torch.bfloat16) + assert ar.mnnvl_allreduce is None and ar.symm_mem_allreduce is None, strategy + for name, build in _cases(): + ulp, res_ulp = 0.0, 0.0 + for m in TOKENS: + xs, residual, w = build(m) + target_out, target_res = _target(ar, xs[RANK], residual, w) + stock_out, stock_res = _stock(ar, xs[RANK], residual, w) + torch.cuda.synchronize() + for what, a, b in ( + ("normed", target_out, stock_out), + ("residual", target_res, stock_res), + ): + bad = int((a.view(torch.int16) != b.view(torch.int16)).sum().item()) + if bad: + mismatches.append( + f"{strategy.name} {name} m={m}: {bad} {what} elements differ from stock" + ) + ref = _fp64(xs, residual, w) + ref_res = residual.double() + sum(x.double() for x in xs) + u, ur = _ulps(target_out, ref), _ulps(target_res, ref_res) + assert math.isfinite(u) and math.isfinite(ur), (strategy, name, m) + ulp, res_ulp = max(ulp, u), max(res_ulp, ur) + dist.barrier() + worst = max(worst, ulp) + if RANK == 0: + print( + f"{RESULT_TAG} {strategy.name:7s} | {name:34s} | target = stock vs fp64: normed {ulp:.3f}" + f" | residual {res_ulp:.3f}", + flush=True, + ) + if RANK == 0: + scope = f"all cases, m in {TOKENS}" + print( + f"{RESULT_TAG} MAX | {scope:34s} | target = stock vs fp64: normed {worst:.3f}", + flush=True, + ) + assert not mismatches, "\n".join(mismatches) + + +def _run_one_rank() -> int: + global RANK, WORLD + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(rank) + dist.init_process_group( + "cpu:gloo,cuda:nccl", init_method="env://", device_id=torch.device("cuda", rank) + ) + RANK, WORLD = rank, world + assert WORLD == 2, f"the drafter runs at tp2, got {WORLD} ranks" + try: + _run_matrix() + except BaseException: + import traceback # only the rank subprocess's failure path needs it + + print(f"[rank {RANK}] FAILED", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + dist.barrier() + torch.cuda.synchronize() + print(f"[rank {RANK}] passed", flush=True) + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + """One worker per claimed device, each in its own session; on the first nonzero exit or the deadline every + session is killed (as ``_m3_mc_allreduce_norm_op_matrix.py`` does).""" + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the 2 devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size == 2, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); the matrix runs at 2" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +if __name__ == "__main__": + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank()) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_m3_mc_allreduce_norm_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_m3_mc_allreduce_norm_op_matrix.py new file mode 100644 index 000000000000..b973e385dca9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_m3_mc_allreduce_norm_op_matrix.py @@ -0,0 +1,755 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU certification matrix for ``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm``, 2 ranks (SM107). + +The tp2 target runs the TP4-named op (``trtllm::minimax_m3_sm107_tp4_mc_allreduce_norm``) at world 2: its C++ +invoke takes the row kernel at world 4 only, so every call here runs the 8-CTA cluster kernel (the target sets +``MC_AR_CLUSTER_MAX_ROWS`` 0: no one-shot cluster option), including the opts-16 drafter form S9B-MC uses (opts +23 / 31 at world 2; option 128 only routes a world-4 call). + +Its own launcher, as the catalog's collective matrices are: run plainly it spawns +one rank process per device in CUDA_VISIBLE_DEVICES and kills them all on the +first failure or on the deadline. A rank whose peers never push polls forever, +so the deadline is what keeps a broken exchange from wedging the calling run. + + CUDA_VISIBLE_DEVICES=0,1 python _m3_mc_allreduce_norm_op_matrix.py + +Each rank builds the three multicast workspaces (8 / 16 / 48 tokens) the target +builds, over the torch.distributed group (the non-MPI path of +``_make_mnnvl_mcast_buffer``). Every rank can regenerate every rank's inputs from +the shared seed convention, which is what keeps each reference arithmetic rather +than a second collective. Producer pushes (``prepushed`` 2 / 3) are emulated by +storing the canonicalized rows through the multicast mapping at the call parity +the flags select, which is exactly what the producing kernels store. + +Not a pytest module, despite the ``check_*`` bodies: they are one fixed sequence +inside a single 2-rank job whose workspace state (call parities) carries from +one to the next. The collected entry point is +``test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py``. +""" + +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Dict, List, Optional + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "m3_mc_allreduce_norm requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 + +HIDDEN = 6144 +EPS = 1e-6 +CLUSTER = 8 +BUCKETS = (8, 16, 48) +EMPTY_WORD = -2147483648 + +# Bound inside the rank body: importing the entries pulls in tensorrt_llm. +allreduce_entry: Any = None +finalize_entry: Any = None +ar_module: Any = None +validating: Any = None +RANK = 0 +WORLD = 1 +WORKSPACES: Dict[int, Any] = {} + + +# ---------------------------------------------------------------- inputs + + +def _gen(seed: int, rank: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed * 1009 + rank + 1) + + +def _rows(seed: int, rank: int, m: int, scale: float = 1.0) -> torch.Tensor: + """One rank's [m, H] BF16 rows; ``rank`` -1 for tensors every rank holds alike.""" + g = _gen(seed, rank + 1) + return (torch.randn(m, HIDDEN, generator=g, device="cuda") * scale).to(torch.bfloat16) + + +def _residual(seed: int, m: int) -> torch.Tensor: + return _rows(seed + 7, -1, m, 4.0) + + +def _norm_w(seed: int) -> torch.Tensor: + g = _gen(seed + 11, 0) + return (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).to(torch.bfloat16) + + +def _finalize_inputs(seed: int, rank: int, m: int, topk: int, dropped: int): + """One rank's unfinalized MoE outputs: fc2 [m * topk, H], perm [m, topk], fp32 scales, shared [m, H].""" + g = _gen(seed + 23, rank) + rows = m * topk + fc2 = (torch.randn(rows, HIDDEN, generator=g, device="cuda") * 0.5).to(torch.bfloat16) + perm = torch.randperm(rows, generator=g, device="cuda").to(torch.int32) + if dropped: + drop = torch.randperm(rows, generator=g, device="cuda")[:dropped] + perm[drop] = -1 + perm = perm.view(m, topk).contiguous() + scales = torch.rand(m, topk, generator=g, device="cuda", dtype=torch.float32) + shared = _rows(seed + 29, rank, m, 0.5) + return fc2, perm, scales, shared + + +# ---------------------------------------------------------------- workspace / producer emulation + + +def _make_workspace(maxt: int): + """One bucket's exchange, as ``McAllReduceWorkspace.create`` builds it, over the torch.distributed group.""" + from tensorrt_llm._torch.distributed.ops import _make_mnnvl_mcast_buffer + from tensorrt_llm.mapping import Mapping + + mapping = Mapping(world_size=WORLD, tp_size=WORLD, rank=RANK) + words = 4 * maxt * WORLD * (HIDDEN // 2) + group = dist.group.WORLD + handle = _make_mnnvl_mcast_buffer(group, words * 4, mapping, False) + uc = handle.get_uc_buffer(RANK, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + dist.barrier() + ws = ar_module.McAllReduceWorkspace( + uc=uc, + mc=mc, + flags={}, + rank=RANK, + world_size=WORLD, + hidden=HIDDEN, + handle=handle, + comm=group, + max_tokens=maxt, + ) + ws.flags_for(CLUSTER) + return ws + + +def _push(ws, rows: torch.Tensor, exchange_set: int) -> None: + """Store this rank's rows into every rank's exchange, as a producer kernel does before a prepushed call. + + Slot of (token t, cluster CTA c): set * 2 * half + parity(t, c) * half + (t * world + rank) * H / 2 + c * H / 16 + words, parity read from the CTA's flag; -0.0 halves are stored as +0.0 (0x80000000 is the empty word). + """ + m = rows.shape[0] + torch.cuda.synchronize() + parity = (ws.flags_for(CLUSTER).view(ws.max_tokens, CLUSTER)[:m] & 1).long() + half = ws.max_tokens * WORLD * (HIDDEN // 2) + chunk = HIDDEN // 2 // CLUSTER + h16 = rows.contiguous().view(torch.int16) + h16 = torch.where(h16 == -32768, torch.zeros_like(h16), h16) + words = h16.view(torch.int32).view(m, CLUSTER, chunk) + t = torch.arange(m, device="cuda").view(m, 1, 1) + c = torch.arange(CLUSTER, device="cuda").view(1, CLUSTER, 1) + w = torch.arange(chunk, device="cuda").view(1, 1, chunk) + index = ( + exchange_set * 2 * half + + parity.view(m, CLUSTER, 1) * half + + (t * WORLD + RANK) * (HIDDEN // 2) + + c * chunk + + w + ) + ws.mc[index.reshape(-1)] = words.reshape(-1) + torch.cuda.synchronize() + dist.barrier() + + +# ---------------------------------------------------------------- gates + + +def _assert_bitwise(out: torch.Tensor, ref: torch.Tensor, where: str) -> None: + assert out.dtype == ref.dtype and out.shape == ref.shape, ( + where, + out.dtype, + ref.dtype, + out.shape, + ) + a, b = out.contiguous().view(torch.uint8), ref.contiguous().view(torch.uint8) + bad = (a != b).sum().item() + assert bad == 0, f"{where}: {bad} of {a.numel()} bytes differ" + + +#: Share of normed elements one BF16 step off the float64 reference: measured at most 1.1e-4 per call over +#: every call of this matrix on GB300 (see ``_assert_normed``). The gate sits ~18x above that and far below what a +#: wrong statistic produces (most elements move). +NORMED_STEP_FRACTION = 0.002 +NORMED_STEP_SEEN: List[float] = [] + + +def _assert_normed(out: torch.Tensor, ref: torch.Tensor, where: str) -> None: + """At most one BF16 rounding step from the float64 reference, and rarely that. + + The kernel's RMS statistic is a shuffle / warp / cluster tree of FP32 partial sums and its ``rsqrtf`` is + approximate (~2^-22 relative): the scaled value moves by ~1e-6 relative, which changes the BF16 rounding only + for values within that distance of a rounding midpoint -- one step, never two. The fraction bound catches a + statistic that is systematically off (a missing rank, a wrong H), which moves most elements. + """ + assert out.dtype == ref.dtype == torch.bfloat16 and out.shape == ref.shape, where + o, r = out.float(), ref.float() + _, exponent = torch.frexp(torch.maximum(o.abs(), r.abs())) + step = torch.ldexp(torch.ones_like(o), exponent - 8) + diff = (o - r).abs() + assert bool((diff <= step).all()), f"{where}: max diff {diff.max().item()} beyond one BF16 step" + frac = (diff != 0).float().mean().item() + NORMED_STEP_SEEN.append(frac) + assert frac <= NORMED_STEP_FRACTION, f"{where}: {frac:.4f} of normed elements off by one step" + + +def _assert_quant(outs, ref, m: int, where: str) -> None: + """The MXFP8 copy: bit-exact against the recipe applied to the op's own normed rows (a pure function of + them), and equal to the reference's wherever the reference's normed 32-column block is the op's.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.comm.m3_mc_allreduce_norm import ( # noqa: E501 + mxfp8_rows, + sf_written, + ) + + q, sf = outs[2], outs[3] + own_q, own_sf = mxfp8_rows(outs[0]) + written = sf_written(m, HIDDEN // 32, sf.device) + _assert_bitwise(q, own_q, f"{where} e4m3 rows") + _assert_bitwise(sf[written], own_sf[written], f"{where} UE8M0 scales") + same_block = (outs[0].view(m, -1, 32) == ref[0].view(m, -1, 32)).all(-1) + qb = q.view(torch.uint8).view(m, -1, 32) + rqb = ref[2].view(torch.uint8).view(m, -1, 32) + assert bool((qb == rqb).all(-1)[same_block].all()), ( + f"{where}: e4m3 differs on a block whose normed rows agree" + ) + + +def _check_outputs(outs, ref, m: int, quant: bool, where: str) -> None: + assert len(outs) == (4 if quant else 2), (where, len(outs)) + _assert_bitwise(outs[1], ref[1], f"{where} updated residual") + _assert_normed(outs[0], ref[0], f"{where} normed") + if quant: + _assert_quant(outs, ref, m, where) + + +# ---------------------------------------------------------------- calls + + +def _call_plain(ws, x, x2, residual, norm_w, quant, prepushed, opts): + return allreduce_entry.m3_mc_allreduce_norm( + x, + x2, + residual, + norm_w, + EPS, + ws.uc, + ws.mc.data_ptr(), + ws.flags_for(CLUSTER), + RANK, + WORLD, + CLUSTER, + False, + ws.max_tokens, + quant, + prepushed, + opts, + ) + + +def _plain_case(spec: Dict[str, Any], seed: int, where: str) -> None: + m, maxt, quant, prepushed = spec["m"], spec["maxt"], spec["quant"], spec["prepushed"] + ws = WORKSPACES[maxt] + xs = [_rows(seed, r, m) for r in range(WORLD)] + x2s = [_rows(seed + 3, r, m) for r in range(WORLD)] if spec["x2"] else None + residual, norm_w = _residual(seed, m), _norm_w(seed) + x, x2 = xs[RANK], (None if x2s is None else x2s[RANK]) + if prepushed >= 2: + _push(ws, x, 0) + if prepushed == 3: + _push(ws, x2, 1) + with validating(allreduce_entry.m3_mc_allreduce_norm): + outs = _call_plain(ws, x, x2, residual, norm_w, quant, prepushed, spec["opts"]) + torch.cuda.synchronize() + ref = allreduce_entry.m3_mc_allreduce_norm.reference( + xs, x2s, residual, norm_w, EPS, quant, spec.get("drafter", False) + ) + _check_outputs(outs, ref, m, quant, where) + dist.barrier() + + +# ---------------------------------------------------------------- checks + + +def check_plain_cells() -> None: + """Every cell of ``m3_mc_allreduce_norm.CELLS`` through ``__call__`` under ``validating``, gated on its + ``reference``.""" + for i, cell in enumerate(allreduce_entry.m3_mc_allreduce_norm.CELLS): + _plain_case(cell.spec, 1000 + 17 * i, cell.why) + + +def check_finalize_cells() -> None: + """Every cell of ``m3_mc_finalize_allreduce_norm.CELLS``, gated on its ``reference``.""" + op = finalize_entry.m3_mc_finalize_allreduce_norm + for i, cell in enumerate(op.CELLS): + spec = cell.spec + m, maxt, quant, topk = spec["m"], spec["maxt"], spec["quant"], spec["topk"] + seed = 2000 + 17 * i + ws = WORKSPACES[maxt] + per_rank = [_finalize_inputs(seed, r, m, topk, spec["dropped"]) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + fc2, perm, scales, shared = per_rank[RANK] + with validating(op): + outs = op( + fc2, + perm, + scales, + shared, + residual, + norm_w, + EPS, + ws.uc, + ws.mc.data_ptr(), + ws.flags_for(CLUSTER), + RANK, + WORLD, + CLUSTER, + False, + maxt, + quant, + spec["opts"], + ) + torch.cuda.synchronize() + ref = op.reference( + [p[0] for p in per_rank], + [p[1] for p in per_rank], + [p[2] for p in per_rank], + [p[3] for p in per_rank], + residual, + norm_w, + EPS, + quant, + ) + _check_outputs(outs, ref, m, quant, cell.why) + dist.barrier() + + +def check_the_target_wrappers_select_bucket_and_options() -> None: + """The target's own wrappers (bucket by token count, option table) on top of the entries, every token count of + the 8-token bucket plus one of each larger bucket, and the finalize wrapper; the MXFP8 publication is the + prequant slot's business and is left off.""" + buckets = ar_module.McBuckets([WORKSPACES[b] for b in BUCKETS]) + for m in (*range(1, 9), 11, 40): + seed = 3000 + m + xs = [_rows(seed, r, m) for r in range(WORLD)] + x2s = [_rows(seed + 3, r, m) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + normed, updated = ar_module.m3_mc_allreduce_norm( + xs[RANK], residual, norm_w, EPS, buckets, x2=x2s[RANK] + ) + torch.cuda.synchronize() + ref = allreduce_entry.m3_mc_allreduce_norm.reference(xs, x2s, residual, norm_w, EPS, False) + _check_outputs([normed, updated], ref, m, False, f"wrapper m={m}") + dist.barrier() + for m in (9, 33): + seed = 3100 + m + per_rank = [_finalize_inputs(seed, r, m, 4, 2) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + fc2, perm, scales, shared = per_rank[RANK] + normed, updated = ar_module.m3_mc_finalize_allreduce_norm( + fc2, perm, scales, shared, residual, norm_w, EPS, buckets + ) + torch.cuda.synchronize() + ref = finalize_entry.m3_mc_finalize_allreduce_norm.reference( + *[[p[j] for p in per_rank] for j in range(4)], residual, norm_w, EPS, False + ) + _check_outputs([normed, updated], ref, m, False, f"finalize wrapper m={m}") + dist.barrier() + + +def check_cuda_graph_replays() -> None: + """Two boundaries captured in one graph (8- and 16-token buckets), replayed with fresh inputs. + + The decode graph holds every boundary call; each replay rotates the same per-CTA parities the eager calls do, + so a replay that reused a stale half or skipped a re-arm reads the previous replay's rows and fails the + bitwise residual gate (every replay gets new inputs). + """ + shapes = ((3, 8, 15), (12, 16, 15)) + norm_w = _norm_w(4000) + bufs = [] + for m, maxt, opts in shapes: + bufs.append( + dict( + m=m, + maxt=maxt, + opts=opts, + x=torch.empty(m, HIDDEN, dtype=torch.bfloat16, device="cuda"), + x2=torch.empty(m, HIDDEN, dtype=torch.bfloat16, device="cuda"), + residual=torch.empty(m, HIDDEN, dtype=torch.bfloat16, device="cuda"), + ) + ) + + def body(): + return [ + _call_plain( + WORKSPACES[b["maxt"]], b["x"], b["x2"], b["residual"], norm_w, False, 0, b["opts"] + ) + for b in bufs + ] + + def fill(seed: int): + refs = [] + for j, b in enumerate(bufs): + s = seed + 31 * j + xs = [_rows(s, r, b["m"]) for r in range(WORLD)] + x2s = [_rows(s + 3, r, b["m"]) for r in range(WORLD)] + b["x"].copy_(xs[RANK]) + b["x2"].copy_(x2s[RANK]) + b["residual"].copy_(_residual(s, b["m"])) + refs.append( + allreduce_entry.m3_mc_allreduce_norm.reference( + xs, x2s, b["residual"].clone(), norm_w, EPS, False + ) + ) + torch.cuda.synchronize() + dist.barrier() + return refs + + refs = fill(4100) + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + warm = body() + torch.cuda.current_stream().wait_stream(side) + torch.cuda.synchronize() + for j, b in enumerate(bufs): + _check_outputs(warm[j], refs[j], b["m"], False, f"graph warm-up m={b['m']}") + dist.barrier() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = body() + for rep in range(4): + refs = fill(4200 + 100 * rep) + graph.replay() + torch.cuda.synchronize() + for j, b in enumerate(bufs): + _check_outputs(captured[j], refs[j], b["m"], False, f"replay {rep} m={b['m']}") + dist.barrier() + del graph + dist.barrier() + + +def check_the_gate_sees_a_missing_rank() -> None: + """The residual gate is not blind to the reduction: dropping any one rank's partial from the reference + moves most elements of the updated residual (each rank's rows are distinct by construction).""" + m, seed = 4, 5000 + xs = [_rows(seed, r, m) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + full = allreduce_entry.m3_mc_allreduce_norm.reference(xs, None, residual, norm_w, EPS, False) + for r in range(WORLD): + partial = [x if i != r else torch.zeros_like(x) for i, x in enumerate(xs)] + wrong = allreduce_entry.m3_mc_allreduce_norm.reference( + partial, None, residual, norm_w, EPS, False + ) + frac = (wrong[1] != full[1]).float().mean().item() + assert frac > 0.9, f"dropping rank {r} changes only {frac:.3f} of the residual" + dist.barrier() + + +def _expect_rejected(fn, match: str, where: str) -> None: + try: + fn() + except AssertionError as exc: + assert match in str(exc), f"{where}: rejected for the wrong reason: {exc}" + else: + raise AssertionError(f"{where}: the guard admitted it") + + +def check_guards() -> None: + """``is_valid`` refuses what the op accepts and answers wrongly. Nothing here reaches the kernel: the guard + raises first, on every rank alike, so no rank is left polling.""" + m, seed = 4, 6000 + ws8, ws16 = WORKSPACES[8], WORKSPACES[16] + x, residual, norm_w = _rows(seed, RANK, m), _residual(seed, m), _norm_w(seed) + op = allreduce_entry.m3_mc_allreduce_norm + + def call(ws_uc, flags, norm): + with validating(op): + op( + x, + None, + residual, + norm, + EPS, + ws_uc, + ws8.mc.data_ptr(), + flags, + RANK, + WORLD, + CLUSTER, + False, + 8, + False, + 0, + 15, + ) + + strided = torch.empty(2 * HIDDEN, dtype=torch.bfloat16, device="cuda")[::2] + strided.copy_(norm_w) + _expect_rejected( + lambda: call(ws8.uc, ws8.flags_for(CLUSTER), strided), "contiguous", "strided norm_w" + ) + _expect_rejected( + lambda: call(ws8.uc[: ws8.uc.numel() // 2], ws8.flags_for(CLUSTER), norm_w), + "words", + "short workspace", + ) + _expect_rejected( + lambda: call(ws8.uc, ws16.flags_for(CLUSTER), norm_w), "counters", "another bucket's flags" + ) + + fin = finalize_entry.m3_mc_finalize_allreduce_norm + fc2, perm, scales, shared = _finalize_inputs(seed, RANK, 9, 4, 0) + bad = perm.clone() + bad[0, 0] = fc2.shape[0] + + def call_fin(): + with validating(fin): + fin( + fc2, + bad, + scales, + shared, + _residual(seed, 9), + norm_w, + EPS, + ws16.uc, + ws16.mc.data_ptr(), + ws16.flags_for(CLUSTER), + RANK, + WORLD, + CLUSTER, + False, + 16, + False, + 15, + ) + + _expect_rejected(call_fin, "perm spans", "perm past the FC2 rows") + dist.barrier() + + +def _nccl_updated(partials: List[torch.Tensor], residual: torch.Tensor) -> torch.Tensor: + """The stock boundary's updated residual from an NCCL all-reduce: this rank's BF16 partial summed by + ``dist.all_reduce`` (BF16 SUM), then the residual added and rounded to BF16. At 2 ranks every element is one + a + b of two BF16 values (fp32 add, one rounding), the same arithmetic as the entry's reference.""" + total = partials[RANK].clone() + dist.all_reduce(total) + torch.cuda.synchronize() + return (total.float() + residual.float()).to(torch.bfloat16) + + +def check_world2_bitwise_vs_reference_and_nccl() -> None: + """codex review item 5: at world 2 the decode boundary (the cluster kernel) at 1 / 8 / 32 rows, x only, x + x2 + with the MXFP8 copy, and pushed (prepushed 2, at the pushed rows 1 / 8), is bit for bit the reference's updated + residual AND the NCCL all-reduce's (the stock collective), its normed rows within one BF16 step of the float64 + reference, its MXFP8 copy exactly the recipe on its own normed rows. The bucket and options are the target's + (kernels/m3_mc_allreduce: 1 / 8 rows the 8-token bucket at option 7, 32 rows the 48-token bucket at 15).""" + for m in (1, 8, 32): + maxt = 8 if m <= 8 else 48 + opts = ar_module._ar_opts(m, maxt) + variants = [(False, False, 0), (True, True, 0)] + if m <= 8: + variants.append((False, False, 2)) + for with_x2, quant, prepushed in variants: + seed = 7000 + 97 * m + 11 * int(with_x2) + 3 * prepushed + spec = dict(m=m, maxt=maxt, opts=opts, x2=with_x2, quant=quant, prepushed=prepushed) + where = f"world 2 m={m} x2={with_x2} quant={quant} prepushed={prepushed}" + ws = WORKSPACES[maxt] + xs = [_rows(seed, r, m) for r in range(WORLD)] + x2s = [_rows(seed + 3, r, m) for r in range(WORLD)] if with_x2 else None + residual, norm_w = _residual(seed, m), _norm_w(seed) + x, x2 = xs[RANK], (None if x2s is None else x2s[RANK]) + if prepushed == 2: + _push(ws, x, 0) + with validating(allreduce_entry.m3_mc_allreduce_norm): + outs = _call_plain(ws, x, x2, residual, norm_w, quant, prepushed, spec["opts"]) + torch.cuda.synchronize() + ref = allreduce_entry.m3_mc_allreduce_norm.reference( + xs, x2s, residual, norm_w, EPS, quant + ) + _check_outputs(outs, ref, m, quant, where) + partials = ( + xs + if x2s is None + else [(a.float() + b.float()).to(torch.bfloat16) for a, b in zip(xs, x2s)] + ) + _assert_bitwise(outs[1], _nccl_updated(partials, residual), f"{where} updated vs NCCL") + dist.barrier() + + +def check_s9b_mc_drafter_form_at_world2() -> None: + """S9B-MC (dspark._s9b_mc_op): the target wrapper with ``drafter_norm`` at 1..8 rows (the 8-token bucket) runs + option 16 | the bucket's option, i.e. 23 at 1 / 2 / 7 / 8 rows and 31 at 3..6, on the cluster kernel at world 2. + Exact-class to the stock drafter boundary (a BF16 all-reduce, then FlashInfer fused_add_rmsnorm on the unrounded + FP32 sum): the updated residual bit for bit, the normed rows within one BF16 step of the float64 drafter + reference. Wider calls decline (None on every rank: S9b's fused all-reduce runs).""" + buckets = ar_module.McBuckets([WORKSPACES[b] for b in BUCKETS]) + for m in range(1, 9): + opts = 16 | ar_module._ar_opts(m, 8) + assert opts == (23 if m in (1, 2, 7, 8) else 31), (m, opts) + seed = 8000 + m + xs = [_rows(seed, r, m) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + out = ar_module.m3_mc_allreduce_norm( + xs[RANK], residual, norm_w, EPS, buckets, drafter_norm=True + ) + assert out is not None, f"S9B-MC declined {m} rows" + torch.cuda.synchronize() + ref = allreduce_entry.m3_mc_allreduce_norm.reference( + xs, None, residual, norm_w, EPS, False, True + ) + _check_outputs(list(out), ref, m, False, f"S9B-MC drafter form m={m}") + _assert_bitwise(out[1], _nccl_updated(xs, residual), f"S9B-MC m={m} updated vs NCCL") + dist.barrier() + wide = _rows(8100, RANK, 9) + assert ( + ar_module.m3_mc_allreduce_norm( + wide, _residual(8100, 9), _norm_w(8100), EPS, buckets, drafter_norm=True + ) + is None + ), "the drafter form must decline past the 8-token bucket" + dist.barrier() + + +CHECKS = ( + check_plain_cells, + check_finalize_cells, + check_the_target_wrappers_select_bucket_and_options, + check_cuda_graph_replays, + check_the_gate_sees_a_missing_rank, + check_world2_bitwise_vs_reference_and_nccl, + check_s9b_mc_drafter_form_at_world2, + check_guards, +) + + +def _run_one_rank() -> int: + """Body of one spawned rank: run every check; a failing rank exits nonzero and the launcher kills its + siblings.""" + global RANK, WORLD, allreduce_entry, finalize_entry, ar_module, validating + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(rank) + dist.init_process_group( + "cpu:gloo,cuda:nccl", init_method="env://", device_id=torch.device("cuda", rank) + ) + RANK, WORLD = rank, world + assert WORLD == 2, f"the tp2 target certifies the entries at 2 ranks, got {WORLD}" + + import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.m3_mc_allreduce as _ar # noqa: E501 + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.comm import ( # noqa: E501 + m3_mc_allreduce_norm as _plain, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.comm import ( # noqa: E501 + m3_mc_finalize_allreduce_norm as _finalize, + ) + + allreduce_entry, finalize_entry, ar_module = _plain, _finalize, _ar + # Spawned by file path, not collected by pytest: reach the sibling helper the plain way (confined to the + # rank processes). /comm -> tests/unittest/_torch/modeling_v2. + sys.path.insert(0, str(Path(__file__).resolve().parents[4])) + from _validating import validating as _validating_cm + + validating = _validating_cm + for maxt in BUCKETS: + WORKSPACES[maxt] = _make_workspace(maxt) + + for check in CHECKS: + try: + check() + except BaseException: + import traceback # only the rank subprocess's failure path needs it + + print(f"[rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + # A rank that leaves early wedges its peers' polls; the launcher kills them on this exit. + os._exit(13) + print(f"[rank {RANK}] {check.__name__} passed", flush=True) + dist.barrier() + torch.cuda.synchronize() + print( + f"[rank {RANK}] {len(CHECKS)} checks passed; normed one-step fraction max " + f"{max(NORMED_STEP_SEEN):.5f} over {len(NORMED_STEP_SEEN)} calls", + flush=True, + ) + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + """One worker process per claimed device, each in its own session; on the first nonzero exit or on the + deadline every session is killed.""" + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the 2 devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size == 2, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); the matrix runs at 2" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + # Drop a launching srun step's PMIx / MPI namespace: children registered in it get the whole step SIGKILLed + # when the first one exits hard. These ranks rendezvous through torch.distributed's env://. + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + print(f"[launcher] spawning {world_size} ranks, rendezvous on :{port}", flush=True) + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +if __name__ == "__main__": + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank()) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_rank_job.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_rank_job.py new file mode 100644 index 000000000000..2d024dc2d3e4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/_rank_job.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Run this target's collective rank job from pytest, and report what it did. + +The same contract as ``tests/unittest/_torch/modeling_v2/comm/_rank_job.py``: the +launcher beside this file owns the per-device rank processes and the deadline +that kills a wedged collective; this only selects the devices, starts it in a +fresh interpreter (by file path: this tree is not a package) and turns its exit +code into an assertion. +""" + +from __future__ import annotations + +import importlib.util +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +WORLD_SIZE = 2 +"""tp2's world size -- the topology these entries are certified for.""" + +_LAUNCHER_GRACE_S = 300 +"""Headroom over the launcher's own deadline, so its message wins the race.""" + + +def _devices() -> str: + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + if visible: + devices = [d for d in visible.split(",") if d.strip()] + else: + import torch + + devices = [str(i) for i in range(torch.cuda.device_count())] + if len(devices) < WORLD_SIZE: + pytest.skip( + f"this entry is certified at world size {WORLD_SIZE}; only {len(devices)} device(s) are visible" + ) + return ",".join(devices[:WORLD_SIZE]) + + +def _load_launcher_constants(launcher: Path): + """Read the launcher's module-level budget without importing it by name (it imports only torch at + module scope).""" + spec = importlib.util.spec_from_file_location(launcher.stem, launcher) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def run(entry: str) -> str: + """Run ``__op_matrix``'s launcher over ``WORLD_SIZE`` devices; returns its stdout.""" + launcher = Path(__file__).resolve().with_name(f"_{entry}_op_matrix.py") + env = dict(os.environ, CUDA_VISIBLE_DEVICES=_devices()) + + # The ranks import tensorrt_llm absolutely. From a source checkout that is not installed, put the repo root + # (tests/unittest/_torch/modeling_v2/models/minimax_m3//comm -> repo root) on their path; from a tests + # tree run against an installed package, they import the installed one. + repo_root = Path(__file__).resolve().parents[8] + if (repo_root / "tensorrt_llm").is_dir(): + env["PYTHONPATH"] = os.pathsep.join( + p for p in (str(repo_root), env.get("PYTHONPATH", "")) if p + ) + else: + assert importlib.util.find_spec("tensorrt_llm") is not None, ( + f"no tensorrt_llm/ at the repo root {repo_root} and no installed tensorrt_llm" + ) + + constants = _load_launcher_constants(launcher) + timeout = constants.DEADLINE_S + _LAUNCHER_GRACE_S + + completed = subprocess.run( + [sys.executable, str(launcher)], + env=env, + capture_output=True, + text=True, + timeout=timeout, + ) + if completed.returncode != 0: + raise AssertionError( + f"{launcher.name} exited {completed.returncode}\n" + f"--- stdout ---\n{completed.stdout}\n" + f"--- stderr ---\n{completed.stderr}" + ) + return completed.stdout diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_allreduce.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_allreduce.py new file mode 100644 index 000000000000..2e3d69a51316 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_allreduce.py @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for the ``m3_allreduce`` certification matrix. + +The matrix is ``_m3_allreduce_op_matrix.py`` beside this file, its own 2-rank launcher; see ``_rank_job``. It drives +every cell of the entry against its reference, then requires the fused draft loop's boundaries built on it (plain +all-reduce, then the fused add + RMSNorm) to equal stock Eagle3's bit for bit under every strategy, and reports both +against an fp64 reference as a warning, one line per strategy and case. +""" + +import warnings + +import _rank_job +import pytest +import torch + +pytestmark = [ + pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", + ), + pytest.mark.skipif( + torch.cuda.device_count() < _rank_job.WORLD_SIZE, + reason=f"certified at world size {_rank_job.WORLD_SIZE}", + ), +] + +_RESULT_TAG = "[ulp]" + + +# Spawns 2 rank processes over the visible devices; under xdist workers would fight for them. +@pytest.mark.no_xdist +def test_m3_allreduce_op_matrix() -> None: + out = _rank_job.run("m3_allreduce") + lines = [line for line in out.splitlines() if line.startswith(_RESULT_TAG)] + assert lines, f"the launcher reported no measurements:\n{out}" + warnings.warn( + "draft boundary (= stock Eagle3) vs fp64, max BF16 ulps\n" + "\n".join(lines), stacklevel=1 + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py new file mode 100644 index 000000000000..1684ac44ffea --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for the multicast all-reduce + RMSNorm entries' certification matrix. + +Covers both ``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm`` (one +kernel, one exchange protocol). The matrix is ``_m3_mc_allreduce_norm_op_matrix.py`` +beside this file, its own 2-rank launcher (world 2: the TP4-named op's 8-CTA cluster +kernel; the row kernel takes world 4 only); see ``_rank_job`` for why. +""" + +import _rank_job +import pytest +import torch + +pytestmark = [ + pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", + ), + pytest.mark.skipif( + torch.cuda.device_count() < _rank_job.WORLD_SIZE, + reason=f"certified at world size {_rank_job.WORLD_SIZE}", + ), +] + + +# Spawns 2 rank processes over the visible devices; under xdist workers would fight for them. +@pytest.mark.no_xdist +def test_m3_mc_allreduce_norm_op_matrix() -> None: + _rank_job.run("m3_mc_allreduce_norm") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_mc_finalize_allreduce_norm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_mc_finalize_allreduce_norm.py new file mode 100644 index 000000000000..8b5cc070ea91 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/comm/test_modeling_v2_m3_mc_finalize_allreduce_norm.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests of the ``m3_mc_finalize_allreduce_norm`` entry that need no collective. + +Its cells are certified, with ``m3_mc_allreduce_norm``'s (one kernel, one exchange protocol), by the 2-rank matrix +``_m3_mc_allreduce_norm_op_matrix.py`` that ``test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py`` runs. +""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.comm.m3_mc_finalize_allreduce_norm import ( # noqa: E501 + finalize_partial, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.comm.m3_mc_finalize_allreduce_norm import ( # noqa: E501 + m3_mc_finalize_allreduce_norm as op, +) + + +def test_finalize_partial_rounds_the_routed_sum_before_the_shared_expert() -> None: + """As stock: the finalized routed output is BF16, and the shared expert is added to it (shared.add_(routed)). + Four routed rows of 1.0 at FP32 routing weight 0.2509765625 sum to 1.00390625, which rounds to BF16 1.0; plus the + shared -1 that is 0, where adding before the rounding would leave 0.00390625.""" + fc2 = torch.ones((4, 8), dtype=torch.bfloat16) + perm = torch.arange(4, dtype=torch.int32).view(1, 4) + scales = torch.full((1, 4), 0.2509765625, dtype=torch.float32) + shared = torch.full((1, 8), -1.0, dtype=torch.bfloat16) + out = finalize_partial(fc2, perm, scales, shared) + routed = finalize_partial(fc2, perm, scales, None) + assert torch.equal(out, shared.clone().add_(routed)) + assert torch.equal(out, torch.zeros(1, 8, dtype=torch.bfloat16)) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_each_cells_partial_is_the_stock_finalize_then_shared_add(cell) -> None: + """Per cell shape (rows, top-k, dropped slots), every rank's partial in the entry's reference is the stock MoE's: + the finalized routed rows rounded to BF16, then ``shared.add_(routed)``. The 2-rank matrix drives the cells on the + GPU against this reference.""" + spec = cell.spec + g = torch.Generator().manual_seed(spec["m"] * 31 + spec["dropped"]) + m, topk, hidden = spec["m"], spec["topk"], 64 + fc2 = torch.randn(m * topk, hidden, generator=g).to(torch.bfloat16) + perm = torch.randperm(m * topk, generator=g).view(m, topk).to(torch.int32) + perm.view(-1)[: spec["dropped"]] = -1 + scales = torch.rand(m, topk, generator=g) + shared = torch.randn(m, hidden, generator=g).to(torch.bfloat16) + routed = finalize_partial(fc2, perm, scales, None) + assert torch.equal(finalize_partial(fc2, perm, scales, shared), shared.clone().add_(routed)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/embedding/test_modeling_v2_m3_embedding.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/embedding/test_modeling_v2_m3_embedding.py new file mode 100644 index 000000000000..9606963d29ad --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/embedding/test_modeling_v2_m3_embedding.py @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_embedding entry of the MiniMax-M3.1 tp2 target: an exact gather with zero rows outside the +table.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.embedding.m3_embedding import ( # noqa: E501 + m3_embedding as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_embedding requires a CUDA device" + +VOCAB, HIDDEN = 4096, 6144 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + table = torch.randn(VOCAB, HIDDEN, generator=g, device="cuda").to(torch.bfloat16) + ids = torch.randint( + 0, VOCAB, (spec["rows"],), generator=g, device="cuda", dtype=spec["id_dtype"] + ) + if spec["pad"]: + ids[1::3] = -1 + with validating(op): + out = op(ids, table) + assert torch.equal(out, op.reference(ids, table)) + if spec["pad"]: + assert not out[1::3].any(), "padded ids must embed to zero rows" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/_m3_mxfp8_operands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/_m3_mxfp8_operands.py new file mode 100644 index 000000000000..e44ad56d743d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/_m3_mxfp8_operands.py @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MXFP8 operands for the MXFP8 GEMM entries' tests: E4M3 values and 128x4-swizzled UE8M0 block scales.""" + +import torch + + +def swizzle(byte: torch.Tensor) -> torch.Tensor: + """[N, K / 32] scale bytes into the 128x4-swizzled buffer of ``mxfp8_quantize(w, True)``.""" + rows, cols = byte.shape + padded_cols, padded_rows = (cols + 3) // 4 * 4, (rows + 127) // 128 * 128 + r = torch.arange(rows, device=byte.device).view(-1, 1) + c = torch.arange(cols, device=byte.device).view(1, -1) + index = ( + (c % 4) + + (c // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * padded_cols) + ) + out = torch.zeros(padded_rows * padded_cols, dtype=torch.uint8, device=byte.device) + out[index.reshape(-1)] = byte.reshape(-1) + return out + + +def mxfp8(rows: int, k: int, g: torch.Generator, lo: int = 118, hi: int = 125): + """An MXFP8 operand: E4M3 values of a unit normal, UE8M0 block scales 2^(lo-127) .. 2^(hi-128).""" + v = torch.randn(rows, k, generator=g, device="cuda").to(torch.float8_e4m3fn) + sf = torch.randint(lo, hi, (rows, k // 32), generator=g, device="cuda", dtype=torch.int32) + return v, swizzle(sf.to(torch.uint8)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_tp2_gemvcfg.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_tp2_gemvcfg.py new file mode 100644 index 000000000000..bd73633ff804 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_tp2_gemvcfg.py @@ -0,0 +1,108 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2_GEMVCFG (kernels/mx_gemv/op._CONFIGS_BY_M): the TP2 per-row GEMV config of the shared down projection. + +- the shared down projection (6144, 1536) at 9..16 rows takes ring 2 with the table's split 2 and contiguous k-tiles: + bit for bit the table's (2, 12) output, on a bf16 and an MXFP8 activation; +- every other (projection, rows) keeps the table, the shared gate_up's bf16 rows stay 16 (its swept split-8 config + slowed the 32-row step in-model), and the kill switch restores the table. +""" + +import importlib +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mx_gemv import ( # noqa: E501 + SHARED_DOWN, + SHARED_GATE_UP, + m3_mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mx_gemv_bf16 import ( # noqa: E501 + m3_mx_gemv_bf16 as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +# The element band of test_modeling_v2_m3_mx_gemv_bf16 (the same fp32 sums against fp64). +ELEMENT_ULP = 1.5 +TABLE = {SHARED_DOWN: (2, 12), SHARED_GATE_UP: (4, 12)} + + +def _dispatch(): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as dispatch, + ) + + return dispatch + + +def _x(m: int, k: int, g: torch.Generator, stress: bool) -> torch.Tensor: + x = torch.randn(m, k, generator=g, device="cuda") + if stress: + block = torch.exp2(torch.randint(-12, 13, (m, k // 32), generator=g, device="cuda").float()) + x = x * block.repeat_interleave(32, dim=1) + return x.to(torch.bfloat16) + + +def test_config_table() -> None: + """The shared down at 9..16 rows takes (2, 2, 1); everything else keeps the table; the bf16 caps are unchanged.""" + assert mx_gemv.TP2_GEMVCFG + for m in range(1, 49): + down = mx_gemv.get_config(*SHARED_DOWN, m) + assert down == ((2, 2, 1) if 9 <= m <= 16 else TABLE[SHARED_DOWN]), (m, down) + assert mx_gemv.get_config(*SHARED_GATE_UP, m) == TABLE[SHARED_GATE_UP], m + assert set(mx_gemv._CONFIGS_BY_M) == {(*SHARED_DOWN, m) for m in range(9, 17)} + assert _dispatch().MX_GEMV_BF16_MAX_TOKENS[SHARED_GATE_UP] == 16 + + +def test_kill_switch_restores_the_table(monkeypatch) -> None: + """TRTLLM_M31_DISABLE_TP2_GEMVCFG=1: no per-row configs.""" + monkeypatch.setattr(m31_switches, "_DISABLED", frozenset({"TP2_GEMVCFG"})) + try: + m = importlib.reload(mx_gemv) + assert not m.TP2_GEMVCFG and m._CONFIGS_BY_M == {} + assert m.get_config(*SHARED_DOWN, 16) == TABLE[SHARED_DOWN] + finally: + monkeypatch.undo() + importlib.reload(mx_gemv) + + +@pytest.mark.parametrize("m", [9, 12, 16]) +@pytest.mark.parametrize("stress", [False, True]) +def test_shared_down_ring2_is_bitwise_the_table(m, stress) -> None: + """Ring depth only stages k-tiles: (2, 2, 1) writes the table's (2, 12) rows bit for bit, bf16 and MXFP8 in.""" + n, k = SHARED_DOWN + g = torch.Generator(device="cuda").manual_seed( + zlib.crc32(f"gemvcfg-down-{m}-{stress}".encode()) + ) + x = _x(m, k, g, stress) + w, wsf = mxfp8(n, k, g, 115, 140) + new = op(x, w, wsf, use_pdl=True) + old = op(x, w, wsf, cfg=TABLE[SHARED_DOWN], use_pdl=True) + assert new is not None and old is not None + assert torch.equal(new.view(torch.int16), old.view(torch.int16)) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + pre_new = m3_mx_gemv(xq.view(m, k), xsf, w, wsf, use_pdl=True) + pre_old = m3_mx_gemv(xq.view(m, k), xsf, w, wsf, cfg=TABLE[SHARED_DOWN], use_pdl=True) + assert torch.equal(pre_new.view(torch.int16), pre_old.view(torch.int16)) + + +@pytest.mark.parametrize("m", [17, 24, 32]) +def test_shared_gate_up_bf16_rows_past_16_keep_the_gemm(m) -> None: + d = _dispatch() + assert d.linear_path(*SHARED_GATE_UP, m, False, False) == d.GENERIC_GEMM_PATH diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_tp2_qkv_producer_fold.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_tp2_qkv_producer_fold.py new file mode 100644 index 000000000000..8be0e2ac4662 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_tp2_qkv_producer_fold.py @@ -0,0 +1,285 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Oracle for QKV_PRODUCER_FOLD on the sm_107 TP2 target: the sparse QKV + index GEMV with the producer in its owner +epilogue (mx_gemv ``m3_mx_gemv_producer``) writes q, index-q and every cache byte (NVFP4 K / V data and scales of both +KV heads, FP8 index-K) bit for bit as today's chain: the in-model GEMV (split-8 override ``mx_gemv_cfg``) storing the +packed BF16 row, then the target's producer on it. Both producer builds are references: the target's own +``m3_producer`` (``minimax_m3_sm107_tp2_producer_pw``, with its pre-wait step-input reads) and the native +``minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert``.""" + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.mx_gemv import ( # noqa: E501 + op as gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_producer import ( # noqa: E501 + m3_producer, +) + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +N, K = 39 * 128, 6144 +HEADS_Q, HEADS_KV, HEADS_INDEX = 32, 2, 2 +PAGE = 128 +PAGES = 512 +MAX_POSITIONS = 262144 +EPS = 1e-6 +KV_QUANT = (1.0, 0.37, 0.61) +CFG = modeling.mx_gemv_cfg(N, K, 8) # the in-model split-8 override + + +def _block_scaled(rows, cols, g, lo, hi, base_scale=1.0): + """bf16 [rows, cols], every 32-element block scaled by 2^U{lo..hi} (block-scale stress when lo / hi are wide).""" + base = torch.randn(rows, cols, device="cuda", generator=g) * base_scale + e = torch.randint(lo, hi + 1, (rows, cols // 32), device="cuda", generator=g).float() + return (base * torch.exp2(e).repeat_interleave(32, dim=1)).to(torch.bfloat16) + + +def _side(tokens, g, bad_slots=False): + rotary = torch.randn(MAX_POSITIONS, 2, 32, device="cuda", generator=g) + norms = [(torch.randn(128, device="cuda", generator=g) * 0.3).bfloat16() for _ in range(4)] + slots = torch.randperm(PAGES * PAGE, device="cuda", generator=g)[:tokens].int() + if bad_slots and tokens >= 3: + slots[1] = -1 + slots[2] = PAGES * PAGE + 5 + positions = torch.randint( + 0, MAX_POSITIONS, (tokens,), device="cuda", dtype=torch.int32, generator=g + ) + return slots, torch.tensor(KV_QUANT, device="cuda"), norms, rotary, positions + + +def _case(tokens, seed, scale=1.0, bad_slots=False, stress=False): + g = torch.Generator(device="cuda").manual_seed(seed) + lo, hi = (-12, 12) if stress else (0, 0) + w = _block_scaled(N, K, g, lo, hi, scale / K**0.5) + x = _block_scaled(tokens, K, g, lo, hi) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + return (xq.view(tokens, K), xsf, wq.view(N, K), wsf, *_side(tokens, g, bad_slots)) + + +def _caches(): + return ( + torch.zeros(PAGES, 2, HEADS_KV, PAGE, 64, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 2, HEADS_KV, PAGE, 8, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 1, PAGE, 128, dtype=torch.float8_e4m3fn, device="cuda"), + ) + + +def _chain(case, caches, producer, eps=EPS): + xq, xsf, wq, wsf, slots, quant, norms, rotary, positions = case + packed = gemv.m3_mx_gemv(xq, xsf, wq, wsf, True, CFG, True, None) + assert packed is not None, "the in-model sparse QKV GEMV must serve <= 8 tokens" + if producer == "target": + q, iq = m3_producer( + packed, *caches, slots, quant, eps, *norms, rotary, positions, HEADS_Q, True, True, True + ) # fmt: skip + else: + q, iq = torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed, *caches, slots, quant, HEADS_Q, HEADS_KV, HEADS_INDEX, 128, 64, eps, *norms, rotary, positions + ) # fmt: skip + torch.cuda.synchronize() + return q, iq + + +def _fused(case, caches, eps=EPS, cfg=CFG): + xq, xsf, wq, wsf, slots, quant, norms, rotary, positions = case + tokens = xq.shape[0] + q = torch.zeros(tokens, HEADS_Q, 128, dtype=torch.float8_e4m3fn, device="cuda") + iq = torch.zeros(tokens, HEADS_INDEX, 128, dtype=torch.float8_e4m3fn, device="cuda") + ok = gemv.m3_mx_gemv_producer( + xq, xsf, wq, wsf, q, iq, *caches, slots, quant, eps, *norms, rotary, positions, cfg=cfg, use_pdl=True + ) # fmt: skip + torch.cuda.synchronize() + return ok, q, iq + + +def _assert_bytes_equal(tag, ref, out): + names = ("q", "index_q", "kv_data", "kv_scale", "index_k") + for name, a, b in zip(names, ref, out): + a8, b8 = a.contiguous().view(torch.uint8), b.contiguous().view(torch.uint8) + diff = (a8 != b8).nonzero() + assert diff.numel() == 0, ( + f"{tag}: {name}: {diff.shape[0]} bytes differ, first at {tuple(diff[0].tolist())}" + ) + + +CASES = [ + (1, 21, 1.0, False, False), + (2, 22, 1.0, False, False), + (4, 23, 1.0, False, False), + (4, 24, 40.0, False, False), + (8, 25, 1.0, False, False), + (8, 26, 0.01, False, False), + (8, 27, 1.0, True, False), + (1, 28, 1.0, False, True), + (5, 29, 1.0, False, True), + (8, 30, 1.0, True, True), +] + + +@pytest.mark.parametrize( + "tokens, seed, scale, bad_slots, stress", + CASES, + ids=[ + "t1", + "t2", + "t4", + "t4-large", + "t8", + "t8-tiny", + "t8-bad-slots", + "t1-stress", + "t5-stress", + "t8-stress-bad", + ], +) +@pytest.mark.parametrize("producer", ["target", "native"]) +def test_fold_is_bitwise_the_chain(tokens, seed, scale, bad_slots, stress, producer) -> None: + """q, index-q (both heads) and every cache byte (both KV heads' NVFP4 data and scales, index-K), on natural and + 2^+-12 block-scale stress operands.""" + case = _case(tokens, seed, scale, bad_slots, stress) + ref_caches = _caches() + rq, riq = _chain(case, ref_caches, producer) + caches = _caches() + ok, q, iq = _fused(case, caches) + assert ok, ( + "the producer GEMV must serve the sparse QKV shape at <= 8 tokens on the split-8 config" + ) + for name, a in zip(("q", "index_q", "kv_data", "kv_scale", "index_k"), (rq, riq, *ref_caches)): + assert bool(a.contiguous().view(torch.uint8).any()), f"reference {name} is all zero" + # Both KV heads and both index-q heads carry data (the fold's head mapping, not only head 0). + for h in range(HEADS_KV): + assert bool(ref_caches[0][:, :, h].any()), f"reference KV head {h} is empty" + for h in range(HEADS_INDEX): + assert bool(riq[:, h].contiguous().view(torch.uint8).any()), ( + f"reference index-q head {h} is empty" + ) + _assert_bytes_equal(f"{producer} t{tokens}", (rq, riq, *ref_caches), (q, iq, *caches)) + + +def _edge_case(tokens, seed, edge): + """The producer's arithmetic edges on the packed row: the GEMV weight is an identity on the first N inputs, so the + packed BF16 row is the MXFP8-rounded input row.""" + g = torch.Generator(device="cuda").manual_seed(seed) + rows = torch.randn(tokens, N // 128, 128, device="cuda", generator=g) + if edge == "zero_rows": + rows[0] = 0.0 + rows[1, HEADS_Q + 2] = 0.0 # a zero V row of KV head 0: an all-zero NVFP4 block + rows[1, HEADS_Q + 3] = 0.0 # and of KV head 1 + elif edge == "denormal_rms": + rows[:2] = ( + rows[:2] * 1e-20 + ) # mean square below FLT_MIN with eps = 0: the rsqrt's denormal path + elif edge == "saturate": + rows[0] = ( + rows[0] * 1e4 + ) # past E4M3 448 after the norm's (1 + w), and saturating NVFP4 block scales + rows[1, :, ::7] = 3e38 + elif edge == "nan_inf": + rows[0, :, 5] = float("nan") + rows[1, :, 9] = float("inf") + rows[2, :, 40] = -float("inf") + x = torch.zeros(tokens, K, device="cuda") + x[:, :N] = rows.reshape(tokens, N) + w = torch.zeros(N, K, device="cuda") + w[:, :N] = torch.eye(N, device="cuda") + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w.to(torch.bfloat16), True) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x.to(torch.bfloat16), True) + slots, quant, norms, rotary, positions = _side(tokens, g) + positions[0] = 0 + positions[-1] = MAX_POSITIONS - 1 # the RoPE table's first and last rows + if edge == "rope_edges": + positions[1] = 1 + positions[2] = MAX_POSITIONS - 2 + return xq.view(tokens, K), xsf, wq.view(N, K), wsf, slots, quant, norms, rotary, positions + + +EDGES = [ + ("zero_rows", 1e-6), + ("denormal_rms", 0.0), + ("saturate", 1e-6), + ("nan_inf", 1e-6), + ("rope_edges", 1e-6), +] + + +@pytest.mark.parametrize("edge, eps", EDGES, ids=[e[0] for e in EDGES]) +def test_fold_edge_rows(edge, eps) -> None: + """Zero rows, a denormal RMS argument (eps 0), E4M3 / NVFP4 saturation, NaN / infinity and the RoPE table's edge + rows: the fold writes the chain's bytes (target producer).""" + case = _edge_case(8, 50 + len(edge), edge) + ref_caches = _caches() + rq, riq = _chain(case, ref_caches, "target", eps) + caches = _caches() + ok, q, iq = _fused(case, caches, eps) + assert ok + _assert_bytes_equal(edge, (rq, riq, *ref_caches), (q, iq, *caches)) + + +def test_fold_is_deterministic_and_graph_replays() -> None: + """Three eager runs and a captured replay write the same bytes.""" + case = _case(4, 31) + outs = [] + for _ in range(3): + caches = _caches() + _, q, iq = _fused(case, caches) + outs.append([t.contiguous().view(torch.uint8).clone() for t in (q, iq, *caches)]) + xq, xsf, wq, wsf, slots, quant, norms, rotary, positions = case + q = torch.zeros(4, HEADS_Q, 128, dtype=torch.float8_e4m3fn, device="cuda") + iq = torch.zeros(4, HEADS_INDEX, 128, dtype=torch.float8_e4m3fn, device="cuda") + caches = _caches() + stream = torch.cuda.Stream() + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + gemv.m3_mx_gemv_producer( + xq, xsf, wq, wsf, q, iq, *caches, slots, quant, EPS, *norms, rotary, positions, cfg=CFG, use_pdl=True + ) # fmt: skip + graph.replay() + torch.cuda.synchronize() + outs.append([t.contiguous().view(torch.uint8).clone() for t in (q, iq, *caches)]) + for run in outs[1:]: + for a, b in zip(outs[0], run): + assert torch.equal(a, b) + + +@pytest.mark.parametrize("tokens", [9, 16, 32]) +def test_declines_past_8_rows(tokens) -> None: + """9..32 rows (and any row count on a non-split-8 config): nothing is launched; the step keeps the plain GEMV + + producer, so those rows stay bitwise today's.""" + case = _case(tokens, 40 + tokens) + caches = _caches() + ok, q, iq = _fused(case, caches) + assert not ok + for t in (q, iq, *caches): + assert not bool(t.contiguous().view(torch.uint8).any()), "a declined call wrote output" + + +def test_declines_other_configs_and_shapes() -> None: + case = _case(8, 41) + ok, _, _ = _fused(case, _caches(), cfg=(4, 12)) + assert not ok, "split 4 cannot run the one-token-per-cluster-rank exchange" + xq, xsf, wq, wsf, *rest = case + narrow = (xq, xsf, wq[: 38 * 128].contiguous(), wsf[: 38 * 128 * K // 32].contiguous(), *rest) + ok, _, _ = _fused(narrow, _caches()) + assert not ok, "only the 39-head packed projection" + + +def test_fold_rows_and_kill_switch(monkeypatch) -> None: + """The dispatch: the fold serves the packed projection at 1..8 rows on its split-8 override, nothing past 8 rows or + on any other projection, and nothing when killed.""" + for rows in range(1, 9): + assert modeling._fold_cfg(N, K, rows) == CFG + for rows in range(9, 33): + assert modeling._fold_cfg(N, K, rows) is None + assert modeling._fold_cfg(*modeling.QKV, 4) is None + monkeypatch.setattr(modeling, "QKV_PRODUCER_FOLD", False) + assert modeling._fold_cfg(N, K, 4) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_unfused_index.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_unfused_index.py new file mode 100644 index 000000000000..a0d7a669e855 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m31_unfused_index.py @@ -0,0 +1,117 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test: the unfused sparse layers' separate MXFP8 index projection (``fuse_qkv_index_projection=False``) of the +tp2 target equals the index columns of the fused five-way projection on the same checkpoint tensors. + +Its weight rows are the five-way weight's index rows byte for byte, its block scales are applied (not dropped), and +on every engine MXFP8 backend its output is bitwise the five-way GEMM's index columns. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + weights as tw, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mxfp8_linear import ( # noqa: E501 + m3_mxfp8_linear as op, +) + +assert torch.cuda.is_available(), "the MXFP8 index projection requires a CUDA device" + +TP = 2 +HIDDEN = 6144 +BLOCK = 32 +# Text-model rows of the attention projections (64 query / 4 KV heads, 4 index heads, head and index dim 128): at TP2 +# the five-way projection is 32 q + 2 k + 2 v + 2 index-q heads + the replicated index-k = 4992 rows per rank. +ROWS = {"q": 64 * 128, "k": 4 * 128, "v": 4 * 128, "index_q": 4 * 128, "index_k": 128} +_FLAGS = { + "native": (False, False, False, False), + "flashinfer": (False, False, True, False), + "tuned": (True, False, False, False), +} + + +def _checkpoint(seed: int, lo: int = 121, hi: int = 133): + """E4M3 weights and UE8M0 block scales per projection; the scales spread over 2^(lo-127) .. 2^(hi-128) (default + 2^-6 .. 2^5; the stress 115..140 spans 2^-12 .. 2^+12).""" + g = torch.Generator(device="cuda").manual_seed(seed) + out = {} + for name, rows in ROWS.items(): + w = (torch.randn(rows, HIDDEN, generator=g, device="cuda") * 2).to(torch.float8_e4m3fn) + s = torch.randint( + lo, hi, (rows, HIDDEN // BLOCK), generator=g, device="cuda", dtype=torch.uint8 + ) + out[name] = (w.cpu(), s.cpu()) + return out + + +def _ns(rank: int) -> SimpleNamespace: + return SimpleNamespace(tp=TP, rank=rank, dev=torch.device("cuda"), head_dim=128, index_dim=128) + + +def _packs(ck, rank): + d = _ns(rank) + names = ("q", "k", "v", "index_q", "index_k") + w5 = tw._qkv_sparse_w(tuple(ck[n][0] for n in names), d) + s5 = tw._qkv_sparse_s( + tuple(ck[n][1] for n in names), d + ) # UE8M0 bytes, as the checkpoint stores them + wi = tw._index_qk_w((ck["index_q"][0], ck["index_k"][0]), d) + si = tw._index_qk_s((ck["index_q"][1], ck["index_k"][1]), d) + return w5, s5.view(torch.uint8), wi, si.view(torch.uint8) + + +@pytest.mark.parametrize("rank", [0, TP - 1]) +def test_weight_rows_are_the_five_way_index_rows(rank) -> None: + ck = _checkpoint(1234 + rank) + w5, _, wi, _ = _packs(ck, rank) + n = wi.shape[0] + assert w5.shape == (4992, HIDDEN), "the TP2 five-way projection's rows per rank" + assert n == 2 * 128 + 128, "this rank's 2 index-q heads and the replicated index-k" + assert wi.dtype == torch.float8_e4m3fn and wi.shape == (n, HIDDEN) + assert torch.equal(wi.view(torch.uint8), w5[-n:].view(torch.uint8)) + + +@pytest.mark.parametrize("scales", [(121, 133), (115, 140)], ids=["narrow", "stress"]) +@pytest.mark.parametrize("path", list(_FLAGS)) +@pytest.mark.parametrize("rows", [1, 4, 8, 32, 64, 256]) +def test_unfused_output_is_bitwise_the_five_way_index_columns(path, rows, scales) -> None: + if path == "flashinfer" and getattr(torch.ops.trtllm, "flashinfer_mm_mxfp8", None) is None: + pytest.skip("this build has no trtllm::flashinfer_mm_mxfp8") + ck = _checkpoint(77, *scales) + w5, s5, wi, si = _packs(ck, 0) + n = wi.shape[0] + g = torch.Generator(device="cuda").manual_seed(rows) + x = torch.randn(rows, HIDDEN, generator=g, device="cuda").to(torch.bfloat16) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + out5 = op(x, w5, s5, alpha, torch.bfloat16, *_FLAGS[path]) + outi = op(x, wi, si, alpha, torch.bfloat16, *_FLAGS[path]) + assert outi.shape == (rows, n) + assert torch.equal(outi, out5[:, -n:]), ( + f"{path}: max |diff| {(outi.float() - out5[:, -n:].float()).abs().max().item()}" + ) + + +def test_block_scales_are_applied() -> None: + """The output follows the op's reference with the checkpoint scales and differs from a unit-scale (dropped) + projection by far more than rounding.""" + ck = _checkpoint(5) + _, _, wi, si = _packs(ck, 0) + unit = tw._index_qk_s( + (torch.full_like(ck["index_q"][1], 127), torch.full_like(ck["index_k"][1], 127)), _ns(0) + ).view(torch.uint8) + x = torch.randn(8, HIDDEN, device="cuda").to(torch.bfloat16) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + out = op(x, wi, si, alpha, torch.bfloat16, *_FLAGS["native"]) + ref = op.reference(x, wi, si, alpha, torch.bfloat16, *_FLAGS["native"]) + dropped = op(x, wi, unit, alpha, torch.bfloat16, *_FLAGS["native"]) + scale = ref.float().abs().mean() + # The engine GEMM rounds once to BF16 after an fp32 accumulation in its own order: within 2 BF16 ulps (2^-7 + # relative) of the reference per element. + assert bool( + ((out.float() - ref.float()).abs() <= 2.0**-7 * ref.float().abs() + 1e-3 * scale).all() + ) + assert (out.float() - dropped.float()).abs().mean() > 0.5 * scale diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_bf16_linear.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_bf16_linear.py new file mode 100644 index 000000000000..923a969adf1e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_bf16_linear.py @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_bf16_linear entry of the MiniMax-M3.1 tp2 target (the drafter's projections).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_bf16_linear import ( # noqa: E501 + m3_bf16_linear as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_bf16_linear requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """Accumulation order (cuBLAS split-K) then one bf16 rounding on both sides: at most 2 units of ULP[bf16] of the + row, RMS well under one; measured on GB300: worst 0.5 unit.""" + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = torch.randn(spec["m"], spec["k"], generator=g, device="cuda").to(torch.bfloat16) + w = (torch.randn(spec["n"], spec["k"], generator=g, device="cuda") * 0.02).to(torch.bfloat16) + with validating(op): + out = op(x, w) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(x, w), element_ulp=2.0, rms_ulp=0.25) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_dense_swiglu.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_dense_swiglu.py new file mode 100644 index 000000000000..e5fb00c825a3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_dense_swiglu.py @@ -0,0 +1,143 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Bitwise oracle of the dense MLP's swigluoai C++ op (ops/gemm/m3_dense_swiglu, +trtllm::minimax_m3_sm107_tp4_swiglu_oai: the TP4 build has no width constant; run here at the TP2 width d 6144) against +the Triton silu_and_mul kernel the dense MLP runs today (modules/swiglu.py at limit 7, alpha 1.702, beta 1): every +bf16 gate bit pattern against special and random up values, every up bit pattern against special and random gates, +random pairs, and the dense MLP's shapes at every decode row count, eager and replayed from a CUDA graph, with and +without the programmatic launch.""" + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.silu_and_mul +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_dense_swiglu import ( # noqa: E501 + ALPHA, + BETA, + LIMIT, + m3_dense_swiglu, +) + +assert torch.cuda.is_available(), "the dense swigluoai op requires a CUDA device" +if not hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp4_swiglu_oai"): + pytest.skip( + "needs the TP4 op library with minimax_m3_sm107_tp4_swiglu_oai", allow_module_level=True + ) + +# The TP2 dense MLP's activation width (gate_up 12288 -> d 6144). +DENSE_D = dispatch.DENSE_DOWN[1] + + +def _triton(x: torch.Tensor) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=LIMIT, swiglu_alpha=ALPHA, swiglu_beta=BETA + ) + + +def _op(x: torch.Tensor, pdl: bool = True) -> torch.Tensor: + out = m3_dense_swiglu(x, LIMIT, ALPHA, BETA, pdl) + assert out is not None, "the op declined a contiguous bf16 [rows, 2d] input" + return out + + +def _row(gate: torch.Tensor, up: torch.Tensor) -> torch.Tensor: + return torch.cat([gate.view(1, -1), up.view(1, -1)], dim=1).contiguous() + + +def _all_bf16() -> torch.Tensor: + return ( + torch.arange(1 << 16, dtype=torch.int32, device="cuda").to(torch.int16).view(torch.bfloat16) + ) + + +def _specials(n: int, seed: int) -> list: + g = torch.Generator(device="cuda").manual_seed(seed) + vals = [0.0, -0.0, 1.0, -1.0, 6.99, 7.0, 7.01, -7.0, -7.01, 1e-38, -1e-38, 1e-40, 88.0, -88.0, 3e38, -3e38, + float("inf"), float("-inf"), float("nan")] # fmt: skip + rows = [torch.full((n,), v, dtype=torch.bfloat16, device="cuda") for v in vals] + for scale in (0.05, 1.0, 4.0, 30.0): + rows.append((scale * torch.randn(n, generator=g, device="cuda")).to(torch.bfloat16)) + rows.append( + torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + .to(torch.int16) + .view(torch.bfloat16) + ) + return rows + + +def _assert_bitwise(got: torch.Tensor, ref: torch.Tensor, what: str) -> None: + bad = (got.view(torch.int16) != ref.view(torch.int16)).nonzero().flatten() + assert bad.numel() == 0, f"{what}: {bad.numel()} mismatches, first at {bad[:4].tolist()}" + + +def test_every_gate_pattern_is_bitwise_the_triton_kernel() -> None: + gate = _all_bf16() + for j, up in enumerate(_specials(gate.numel(), 1)): + x = _row(gate, up) + _assert_bitwise(_op(x), _triton(x), f"up set {j}") + + +def test_every_up_pattern_is_bitwise_the_triton_kernel() -> None: + up = _all_bf16() + for j, gate in enumerate(_specials(up.numel(), 2)): + x = _row(gate, up) + _assert_bitwise(_op(x), _triton(x), f"gate set {j}") + + +@pytest.mark.parametrize("seed", range(4)) +def test_random_pairs_are_bitwise_the_triton_kernel(seed: int) -> None: + g = torch.Generator(device="cuda").manual_seed(100 + seed) + n = 1 << 22 + + def pattern() -> torch.Tensor: + bits = torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + return bits.to(torch.int16).view(torch.bfloat16) + + x = _row(pattern(), pattern()) + _assert_bitwise(_op(x), _triton(x), f"seed {seed}") + + +@pytest.mark.parametrize("pdl", [True, False], ids=["pdl", "plain"]) +@pytest.mark.parametrize("rows", range(1, dispatch.DECODE_MAX_TOKENS + 1)) +def test_dense_shape_eager_and_graph_are_bitwise_the_triton_kernel(rows: int, pdl: bool) -> None: + """The TP2 dense MLP's gate_up rows (a GEMV output's range plus values past the clamps), eager and replayed from a + CUDA graph after fresh rows are written in place, against the Triton kernel on the same rows.""" + g = torch.Generator(device="cuda").manual_seed(2000 + rows) + x = (3.0 * torch.randn(rows, 2 * DENSE_D, generator=g, device="cuda")).to(torch.bfloat16) + x[:, :8] = torch.tensor([8.0, -8.0, 7.0, -7.0, 0.0, -0.0, 50.0, -50.0], dtype=torch.bfloat16) + x[:, DENSE_D : DENSE_D + 8] = x[:, :8] + _assert_bitwise(_op(x, pdl), _triton(x), f"eager rows {rows}") + out = torch.empty(rows, DENSE_D, dtype=torch.bfloat16, device="cuda") + _op(x, pdl) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out.copy_(_op(x, pdl)) + x.copy_((3.0 * torch.randn(rows, 2 * DENSE_D, generator=g, device="cuda")).to(torch.bfloat16)) + graph.replay() + torch.cuda.synchronize() + _assert_bitwise(out, _triton(x), f"graph rows {rows}") + + +def test_the_op_declines_other_parameters_and_layouts() -> None: + x = torch.zeros(2, 2 * DENSE_D, dtype=torch.bfloat16, device="cuda") + assert m3_dense_swiglu(x, 8.0, ALPHA, BETA, True) is None + assert m3_dense_swiglu(x, LIMIT, 1.0, BETA, True) is None + assert m3_dense_swiglu(x, LIMIT, ALPHA, 0.0, True) is None + assert m3_dense_swiglu(x.float(), LIMIT, ALPHA, BETA, True) is None + assert m3_dense_swiglu(x[:, : 2 * DENSE_D - 2], LIMIT, ALPHA, BETA, True) is None + + +@pytest.mark.parametrize("cell", m3_dense_swiglu.CELLS, ids=lambda c: c.why) +def test_every_cell_is_bitwise_the_triton_kernel(cell) -> None: + """Each declared cell (the dense MLP's activation at its decode rows) through the op, bit for bit the Triton + kernel on gate_up rows in a GEMV output's range plus values past the clamps.""" + rows, d = cell.spec["rows"], cell.spec["d"] + g = torch.Generator(device="cuda").manual_seed(3000 + rows) + x = (3.0 * torch.randn(rows, 2 * d, generator=g, device="cuda")).to(torch.bfloat16) + x[:, :8] = torch.tensor([8.0, -8.0, 7.0, -7.0, 0.0, -0.0, 50.0, -50.0], dtype=torch.bfloat16) + x[:, d : d + 8] = x[:, :8] + _assert_bitwise(_op(x), _triton(x), cell.why) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_fp32_linear.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_fp32_linear.py new file mode 100644 index 000000000000..fda318dd3d95 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_fp32_linear.py @@ -0,0 +1,32 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_fp32_linear entry of the MiniMax-M3.1 tp2 target (the router beyond the decode routers' rows).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_fp32_linear import ( # noqa: E501 + m3_fp32_linear as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_fp32_linear requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """torch's default fp32 matmul precision on this hardware may be TF32 (~2^-11 relative per product): the band is + stated in fp32 ulps of each row's largest logit and covers it (K = 6144 TF32 products: ~2^13 ulps worst element); + measured on GB300 with the engine's defaults: worst element ~1700 ulps, RMS ~300.""" + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = (torch.randn(spec["m"], spec["k"], generator=g, device="cuda") * 2.0).to(torch.bfloat16) + w = torch.randn(spec["e"], spec["k"], generator=g, device="cuda") * 0.02 + with validating(op): + out = op(x, w) + assert out.dtype == torch.float32 and out.shape == (spec["m"], spec["e"]) + assert_within_ulp(out, op.reference(x, w), element_ulp=8192.0, rms_ulp=2048.0) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv.py new file mode 100644 index 000000000000..6f0a5b52c9e2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv.py @@ -0,0 +1,266 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gemm/m3_mx_gemv entry: the decode MXFP8 GEMV on a producer-quantized activation. + +Operands as the target holds them: the weight through mxfp8_quantize with the swizzled scale layout (the MXFP8 linear +method's storage), the activation through mxfp8_quantize with the swizzled layout (what the boundary all-reduce and +the attention merge publish). The tp2 target's per-rank projections: sparse QKV + index 4992x6144, dense QKV +4608x6144, o_proj 6144x4096, shared gate_up / down 3072x6144 / 6144x1536, dense gate_up / down 12288x6144 / 6144x6144, +and the DSpark drafter's QKV 5120x6144 (MXDRAFT). Besides the dense operands, every TP2 projection runs on operands +whose UE8M0 block scales span 2^-12..2^+12 (the block-scale stress: dense, narrow-scale data passes a split / config +claim falsely), for the in-band, deterministic and graph-replay claims. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mx_gemv import ( # noqa: E501 + m3_mx_gemv as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_mx_gemv requires a CUDA device" + + +# Element band, in ULP units of the row scale: see compare. +ELEMENT_ULP = 1.5 + + +def compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """Element ELEMENT_ULP of the row scale, relative RMS 0.1 ulp. + + Kernel and reference multiply bit-identical dequantized operands; the kernel sums in fp32 (tensor-core blocks, + then the split-K partials), the reference in fp64. Over K <= 6144 that order difference is ~1e-6 relative, so the + two bf16 roundings disagree only where the sum sits within that of a rounding boundary: one bf16 ulp of that + element, which for an element in its row maximum's binade is up to two of ``assert_within_ulp``'s units of the row + scale, and a few in ten thousand elements in aggregate. Any fp32 sum shows it: torch's IEEE fp32 matmul of the + same dequantized operands measured at most 1.22 units against the fp64 reference over every row count 1..48 on + the o_proj and the QKV (6 seeds each, GB300), the kernel at most 1.25, both at most 0.009 ulp RMS. The band + leaves the margin between that and a full extra ulp. + """ + assert_within_ulp(out, ref, element_ulp=ELEMENT_ULP, rms_ulp=0.1) + + +def build(n: int, k: int, m: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = torch.randn(m, k, generator=g, device="cuda").to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + return x, xq.view(m, k), xsf, wq.view(n, k), wsf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + (n, k), m = cell.spec["nk"], cell.spec["m"] + _, xq, xsf, w, wsf = build(n, k, m, seed=zlib.crc32(cell.why.encode()) % 2**31) + with validating(op): + out = op(xq, xsf, w, wsf, use_pdl=True) # the target launches with PDL + assert out is not None, "a certified shape must be served" + assert out.shape == (m, n) and out.dtype is torch.bfloat16 + compare(out, op.reference(xq, xsf, w, wsf)) + + +def test_an_unserved_shape_returns_none() -> None: + """Shapes outside the tuned table fall back at the call site; the op says so rather than computing.""" + _, xq, xsf, w, wsf = build(256, 512, 1, seed=1) + assert op(xq, xsf, w, wsf) is None + + +def test_guard_refuses_linear_activation_scales() -> None: + """A linear [M, K / 32] scale buffer is read as the swizzled one.""" + x, _, _, w, wsf = build(6144, 4096, 4, seed=2) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, False) + with pytest.raises(AssertionError, match="swizzled"): + with validating(op): + op(xq.view(4, 4096), xsf, w, wsf) + + +def test_guard_refuses_a_strided_weight() -> None: + _, xq, xsf, w, wsf = build(6144, 4096, 1, seed=3) + wide = torch.zeros(6144, 8192, dtype=w.dtype, device="cuda") + wide[:, :4096] = w + with pytest.raises(AssertionError, match="contiguous e4m3"): + with validating(op): + op(xq, xsf, wide[:, :4096], wsf) + + +@pytest.mark.parametrize("nk", [(4992, 6144), (4608, 6144)], ids=["qkv_sparse", "qkv_dense"]) +@pytest.mark.parametrize("m", range(1, 9)) +def test_qkv_split8_config(nk, m) -> None: + """The dispatch's (split, ring) override (the TP4 measurement, kept at the TP2 widths) for the QKV at decode rows: + within the band of the fp64 reference, and the graph replay bit-identical to the eager call (one split-K order per + config).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as dispatch, + ) + + cfg = dispatch.mx_gemv_cfg(nk[0], nk[1], m) + assert cfg == (8, 8) + _, xq, xsf, wq, wsf = build(nk[0], nk[1], m, zlib.crc32(f"qkv8-{nk}-{m}".encode())) + eager = op(xq, xsf, wq, wsf, cfg=cfg) + assert eager is not None, "the GEMV declined the split-8 config" + compare(eager, op.reference(xq, xsf, wq, wsf)) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = op(xq, xsf, wq, wsf, cfg=cfg) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(captured.view(torch.int16), eager.view(torch.int16)) + + +def test_qkv_split8_config_is_bounded(monkeypatch) -> None: + """The overrides cover only the two QKV projections (split 8) and the o_proj (strided k-tiles, rule 1b) at decode + rows; every other call, and every call with its kill knob off, keeps the shared table.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as dispatch, + ) + + assert dispatch.mx_gemv_cfg(4992, 6144, 9) is None + assert dispatch.mx_gemv_cfg(6144, 4096, 4) == (2, 8, 0) + assert dispatch.mx_gemv_cfg(6144, 4096, 9) is None + assert dispatch.mx_gemv_cfg(6144, 6144, 4) is None + assert dispatch.mx_gemv_cfg(5120, 6144, 4) is None # the drafter's QKV keeps the shared table + monkeypatch.setattr(dispatch, "GEMV_OPROJ_STRIDED_TP4", False) + assert dispatch.mx_gemv_cfg(6144, 4096, 4) is None + assert dispatch.mx_gemv_cfg(4992, 6144, 4) == (8, 8) + monkeypatch.setattr(dispatch, "GEMV_QKV_SPLIT8_TP4", False) + assert dispatch.mx_gemv_cfg(4992, 6144, 4) is None + + +@pytest.mark.parametrize( + "nk", + [(4992, 6144), (4608, 6144), (6144, 4096)], + ids=["qkv_sparse", "qkv_dense", "o_proj_strided"], +) +def test_prebuild_covers_the_split8_config_while_frozen(nk) -> None: + """The core's prebuild (``prebuild_kernels``: ``_gemv_variants`` + ``_precompile_cfg``) compiles every (rows, cfg) + ``_linear`` can pick for this projection, the (split, ring) override included: with compiles frozen afterwards + (as the target freezes them before warmup), every decode-row MXFP8 call at the override is served, none + declines.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as dispatch, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, + ) + + n, k = nk + _, _, _, wq, wsf = build(n, k, 1, zlib.crc32(f"prebuild-{nk}".encode())) + mx_gemv.freeze(False) + mx_gemv.ensure_compiled(wq, wsf, True) + for rows, x_bf16, ar in dispatch._gemv_variants(n, k, None, 2): + cfg = None if x_bf16 else dispatch.mx_gemv_cfg(n, k, rows) + if cfg is not None: + dispatch._precompile_cfg(wq, wsf, rows, cfg, True, ar=ar) + else: + mx_gemv.precompile(wq, wsf, rows, x_bf16, use_pdl=True, ar=ar) + picked = [ + (rows, dispatch.mx_gemv_cfg(n, k, rows)) + for rows in sorted(dispatch.MX_GEMV_ROWS[nk]) + if dispatch.mx_gemv_cfg(n, k, rows) is not None + ] + assert picked, "the override covers decode rows of this projection" + for rows, cfg in picked: + resolved = mx_gemv._resolve(rows, n, k, False, True, cfg, True, None) + assert resolved is not None and mx_gemv.is_compiled(resolved[0]), (rows, cfg) + mx_gemv.freeze(True) + try: + for rows, cfg in picked: + _, xq, xsf, _, _ = build(n, k, rows, rows) + out = mx_gemv.m3_mx_gemv(xq, xsf, wq, wsf, cfg=cfg, use_pdl=True) + assert out is not None, (rows, cfg, mx_gemv.last_uncompiled) + finally: + mx_gemv.freeze(False) + + +# Every TP2 projection (and the drafter's QKV) at the rows each build differs in: the MMA widths 8 / 16 / 32 / 48, the +# decode override rows (<= 8) and the c1..c4 K7 steps (8 / 16 / 24 / 32). +TP2_SHAPES = { + "qkv_sparse": (4992, 6144), + "qkv_dense": (4608, 6144), + "o_proj": (6144, 4096), + "shared_gate_up": (3072, 6144), + "shared_down": (6144, 1536), + "dense_gate_up": (12288, 6144), + "dense_down": (6144, 6144), + "drafter_qkv": (5120, 6144), +} +STRESS_ROWS = (1, 7, 8, 16, 24, 32, 48) +STRESS_CASES = [(name, m) for name in TP2_SHAPES for m in STRESS_ROWS] +# UE8M0 codes 115..139: block scales 2^-12 .. 2^+12 on both operands. +STRESS_LO, STRESS_HI = 115, 140 + + +def test_tp2_projections_are_served() -> None: + """The TP2 table serves every TP2 projection at one token (an unserved one would decline to the GEMM silently).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, + ) + + for name, (n, k) in TP2_SHAPES.items(): + assert mx_gemv.serves(n, k, 1, False), name + + +@pytest.mark.parametrize("name, m", STRESS_CASES, ids=[f"{n}-m{m}" for n, m in STRESS_CASES]) +def test_tp2_block_scale_stress(name, m) -> None: + """Block scales 2^-12..2^+12 on the weight and the activation, at every config the dispatch can pick (the table's, + and the decode override where it applies): within the fp64 reference's band, the same bits on 3 eager calls and on + a CUDA-graph replay. Rows the GEMV does not serve for a projection (the shared down at 17..32) skip.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as dispatch, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, + ) + + n, k = TP2_SHAPES[name] + if not mx_gemv.serves(n, k, m, False): + pytest.skip(f"{name} is not served at {m} rows (the dispatch runs the MXFP8 GEMM)") + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(f"stress-{name}-{m}".encode())) + xq, xsf = mxfp8(m, k, g, STRESS_LO, STRESS_HI) + w, wsf = mxfp8(n, k, g, STRESS_LO, STRESS_HI) + ref = op.reference(xq, xsf, w, wsf) + for cfg in dict.fromkeys((None, dispatch.mx_gemv_cfg(n, k, m))): + outs = [op(xq, xsf, w, wsf, True, cfg, True, None) for _ in range(3)] + assert all(o is not None for o in outs), (name, m, cfg) + compare(outs[0], ref) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = op(xq, xsf, w, wsf, True, cfg, True, None) + graph.replay() + torch.cuda.synchronize() + for o in outs[1:] + [captured]: + assert torch.equal(outs[0].view(torch.int16), o.view(torch.int16)), ( + f"{name} m={m} cfg {cfg}: not deterministic (eager / graph)" + ) + + +def test_the_mxfp8_row_table_is_served_at_tp2() -> None: + """Every (projection, rows) ``MX_GEMV_ROWS`` routes to the MXFP8-in GEMV is one the GEMV serves at the TP2 widths + (a declined row is a ``missed`` fast path).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as dispatch, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, + ) + + declined = [ + (nk, m) + for nk, rows in sorted(dispatch.MX_GEMV_ROWS.items()) + for m in sorted(rows) + if not mx_gemv.serves(nk[0], nk[1], m, False) + ] + assert not declined, f"MXFP8-in rows the dispatch routes but the GEMV declines: {declined}" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv_bf16.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv_bf16.py new file mode 100644 index 000000000000..888402db5809 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv_bf16.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gemm/m3_mx_gemv_bf16 entry: the decode MXFP8 GEMV quantizing a bf16 activation in its prologue. + +The weight as the MXFP8 linear method stores it (mxfp8_quantize, swizzled scales). Besides the reference band, the +prologue's quantization is held to mxfp8_quantize itself: the GEMV on a bf16 x must equal, bit for bit, the +prequantized GEMV on mxfp8_quantize(x) -- the claim the target relies on when it mixes the two paths. That claim is +also held at the tp2 target's projections on activations and weights whose 32-element blocks span 2^-12..2^+12 (the +block-scale stress), at the rows the bf16 path serves (modeling.MX_GEMV_BF16_MAX_TOKENS). +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mx_gemv import ( # noqa: E501 + m3_mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mx_gemv_bf16 import ( # noqa: E501 + m3_mx_gemv_bf16 as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_mx_gemv_bf16 requires a CUDA device" + +# The element band of test_modeling_v2_m3_mx_gemv.compare (derived there): the same fp32 sums against fp64. +ELEMENT_ULP = 1.5 + + +def build(n: int, k: int, m: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = torch.randn(m, k, generator=g, device="cuda").to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + return x, wq.view(n, k), wsf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + (n, k), m = cell.spec["nk"], cell.spec["m"] + x, w, wsf = build(n, k, m, seed=zlib.crc32(cell.why.encode()) % 2**31) + with validating(op): + out = op(x, w, wsf, use_pdl=True) # the target launches with PDL + assert out is not None, "a certified shape must be served" + assert out.shape == (m, n) and out.dtype is torch.bfloat16 + # The band of test_modeling_v2_m3_mx_gemv.compare: the quantized operands are the reference's exactly (checked + # below), the sums differ only in order. + assert_within_ulp(out, op.reference(x, w, wsf), element_ulp=ELEMENT_ULP, rms_ulp=0.1) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + pre = m3_mx_gemv(xq.view(m, k), xsf, w, wsf, use_pdl=True) + assert pre is not None and torch.equal(out.view(torch.int16), pre.view(torch.int16)), ( + "the prologue's MXFP8 quantization differs from mxfp8_quantize" + ) + + +def test_a_strided_activation_is_declined() -> None: + """The prologue reads x densely; the op declines a strided x (None) rather than computing on it.""" + x, w, wsf = build(6144, 4096, 2, seed=1) + wide = torch.zeros(2, 8192, dtype=torch.bfloat16, device="cuda") + wide[:, :4096] = x + assert op(wide[:, :4096], w, wsf) is None + + +def _dispatch(): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as dispatch, + ) + + return dispatch + + +def _stress_cases(): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, + ) + + return [ + (nk, m) + for nk, top in sorted(_dispatch().MX_GEMV_BF16_MAX_TOKENS.items()) + for m in sorted({1, 8, 16, top}) + if m <= top and mx_gemv.serves(nk[0], nk[1], m, True) + ] + + +def test_the_bf16_row_table_is_served_at_tp2() -> None: + """Every (projection, rows) the decode dispatch sends to the bf16-in GEMV (``MX_GEMV_BF16_MAX_TOKENS``) is one + the GEMV serves at the TP2 width; a row it declines is a ``missed`` fast path (raises under require). The prologue + pays one 32-element block per thread and round, at most 8 rounds per CTA: at TP2 the wider K halves the rows + each split serves relative to TP4.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, + ) + + declined = [ + (nk, m) + for nk, top in sorted(_dispatch().MX_GEMV_BF16_MAX_TOKENS.items()) + for m in range(1, top + 1) + if not mx_gemv.serves(nk[0], nk[1], m, True) + ] + assert not declined, f"bf16-in rows the dispatch routes but the GEMV declines: {declined}" + + +STRESS_CASES = _stress_cases() + + +@pytest.mark.parametrize("nk, m", STRESS_CASES, ids=[f"{n}x{k}-m{m}" for (n, k), m in STRESS_CASES]) +def test_prologue_quantize_is_mxfp8_quantize_under_block_scale_stress(nk, m) -> None: + """bf16 activation blocks scaled by 2^-12..2^+12 and weight block scales 2^-12..2^+12: the bf16-in GEMV equals the + prequantized GEMV on mxfp8_quantize(x) bit for bit, and stays in the fp64 reference's band.""" + n, k = nk + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(f"bf16-stress-{nk}-{m}".encode())) + block = torch.exp2(torch.randint(-12, 13, (m, k // 32), generator=g, device="cuda").float()) + x = (torch.randn(m, k, generator=g, device="cuda") * block.repeat_interleave(32, dim=1)).to( + torch.bfloat16 + ) + w, wsf = mxfp8(n, k, g, 115, 140) + out = op(x, w, wsf, use_pdl=True) + assert out is not None, f"the bf16 path does not serve {nk} at {m} rows" + assert_within_ulp(out, op.reference(x, w, wsf), element_ulp=ELEMENT_ULP, rms_ulp=0.1) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + pre = m3_mx_gemv(xq.view(m, k), xsf, w, wsf, use_pdl=True) + assert pre is not None and torch.equal(out.view(torch.int16), pre.view(torch.int16)), ( + "the prologue's MXFP8 quantization differs from mxfp8_quantize under block-scale stress" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv_order.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv_order.py new file mode 100644 index 000000000000..cb5e9370ac4b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mx_gemv_order.py @@ -0,0 +1,92 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode GEMV order overrides (modeling.mx_gemv_cfg: QKV split 8, o_proj strided k-tiles; TP4-measured, kept at +the TP2 widths 4992 / 4608 x 6144 and 6144 x 4096): for every overridden (projection, rows) the output is the same on 3 +eager calls and on a CUDA graph replay (deterministic), and its max abs / max rel error vs the fp64 product of the +dequantized operands stays within the shipped order's band (the shared table's config on the same operands, over +several seeds). Two operand families: the narrow-scale one and the block-scale stress (weight block scales 2^-12..2^+12, +activation 32-element blocks scaled by 2^-12..2^+12).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mx_gemv import ( # noqa: E501 + O_PROJ, + QKV, + QKV_SPARSE, + m3_mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.moe.m3_moe_engine import ( # noqa: E501 + mxfp8_dequant, +) + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +CASES = [ + (nk, m, scales) + for nk in (QKV_SPARSE, QKV, O_PROJ) + for m in (1, 2, 4, 8) + for scales in ("narrow", "stress") +] + + +def _err(y, ref): + d = (y.double() - ref).abs() + return float(d.max()), float((d / ref.abs().amax(dim=1, keepdim=True).clamp_min(1e-30)).max()) + + +@pytest.mark.parametrize( + "nk, m, scales", CASES, ids=[f"{n}x{k}-m{m}-{sc}" for (n, k), m, sc in CASES] +) +def test_order_override_is_deterministic_and_in_band(nk, m, scales) -> None: + n, k = nk + cfg = dispatch.mx_gemv_cfg(n, k, m) + assert cfg is not None, "the override is off or does not cover this call" + worst_new, worst_ship = [0.0, 0.0], [0.0, 0.0] + for seed in range(4): + g = torch.Generator(device="cuda").manual_seed(1000 * seed + m) + w = (torch.randn(n, k, device="cuda", generator=g) * 2).to(torch.float8_e4m3fn) + lo, hi = (118, 124) if scales == "narrow" else (115, 140) + ws = torch.randint(lo, hi, (n * k // 32,), dtype=torch.uint8, device="cuda", generator=g) + x = torch.randn(m, k, device="cuda", generator=g) * (0.5 + seed) + if scales == "stress": + block = torch.randint(-12, 13, (m, k // 32), device="cuda", generator=g).float() + x = x * torch.exp2(block).repeat_interleave(32, dim=1) + x = x.to(torch.bfloat16) + q, s = torch.ops.trtllm.mxfp8_quantize(x, True) + ref = ( + mxfp8_dequant(q.view(torch.uint8), s).double() + @ mxfp8_dequant(w.view(torch.uint8), ws).double().t() + ) + ys = [m3_mx_gemv(q, s, w, ws, True, cfg, True, None) for _ in range(3)] + assert all(y is not None for y in ys), "the GEMV declined the override" + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + out = [] + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out.append(m3_mx_gemv(q, s, w, ws, True, cfg, True, None)) + graph.replay() + torch.cuda.synchronize() + for y in ys[1:] + [out[0]]: + assert torch.equal(ys[0].view(torch.int16), y.view(torch.int16)), ( + "not deterministic (eager / graph)" + ) + ship = m3_mx_gemv(q, s, w, ws, True, None, True, None) + torch.cuda.synchronize() + for acc, y in ((worst_new, ys[0]), (worst_ship, ship)): + e = _err(y, ref) + acc[0], acc[1] = max(acc[0], e[0]), max(acc[1], e[1]) + print( + f"\nR1B1 tp2 {n}x{k} m={m} {scales} cfg {cfg}: max abs / rel vs fp64 {worst_new[0]:.3e} / " + f"{worst_new[1]:.3e} | shipped order {worst_ship[0]:.3e} / {worst_ship[1]:.3e}", + flush=True, + ) + # The band: the shipped order's own error (bf16 output rounding dominates), with 25 % headroom. + assert worst_new[0] <= 1.25 * worst_ship[0] and worst_new[1] <= 1.25 * worst_ship[1] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_gemm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_gemm.py new file mode 100644 index 000000000000..2873fcce20ed --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_gemm.py @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_gemm entry of the MiniMax-M3.1 tp2 target (the decode-graph GEMM on a prequantized +activation).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mxfp8_gemm import ( # noqa: E501 + m3_mxfp8_gemm as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_gemm requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """Same E4M3 operands and power-of-two scales on both sides; accumulation order then one bf16 rounding: at most + 2 units of ULP[bf16] of the row (the low-M GEMM's band), RMS well under one.""" + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + act, act_sf = mxfp8(spec["m"], spec["k"], g) + w, w_sf = mxfp8(spec["n"], spec["k"], g) + with validating(op): + out = op(act, act_sf, w, w_sf, torch.bfloat16, False) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp( + out, + op.reference(act, act_sf, w, w_sf, torch.bfloat16, False), + element_ulp=2.0, + rms_ulp=0.25, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_linear.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_linear.py new file mode 100644 index 000000000000..30f6132a201a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_linear.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_linear entry of the MiniMax-M3.1 tp2 target: MXFP8 quantize + GEMM on each backend the +engine selects (native CUTLASS, FlashInfer, the per-bucket tuned quantize + GEMM).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mxfp8_linear import ( # noqa: E501 + m3_mxfp8_linear as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_linear requires a CUDA device" + +_FLAGS = { + "native": (False, False, False, False), + "flashinfer": (False, False, True, False), + "tuned": (True, False, False, False), +} + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """The activation's MXFP8 round trip is exact on both sides (same block rule); then the accumulation order and + one bf16 rounding: at most 2 units of ULP[bf16] of the row, RMS well under one.""" + spec = cell.spec + if ( + spec["path"] == "flashinfer" + and getattr(torch.ops.trtllm, "flashinfer_mm_mxfp8", None) is None + ): + pytest.skip("this build has no trtllm::flashinfer_mm_mxfp8") + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = torch.randn(spec["m"], spec["k"], generator=g, device="cuda").to(torch.bfloat16) + w, w_sf = mxfp8(spec["n"], spec["k"], g) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + args = (x, w, w_sf, alpha, torch.bfloat16, *_FLAGS[spec["path"]]) + with validating(op): + out = op(*args) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(*args), element_ulp=2.0, rms_ulp=0.25) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_pinned_gemm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_pinned_gemm.py new file mode 100644 index 000000000000..4d0a9e14fa63 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_pinned_gemm.py @@ -0,0 +1,90 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_pinned_gemm entry (MX-TAC): the pinned decode-graph MXFP8 GEMM gives the same bits as +every candidate stock's AutoTuner can pick (CuTe DSL, CUTLASS), the same on every call, within the MXFP8 band of the +dequantized product.""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mxfp8_pinned_gemm import ( # noqa: E501 + m3_mxfp8_pinned_gemm as op, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import FlashInferMXFP8GemmRunner + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_pinned_gemm requires a CUDA device" + +TACTIC = modeling.MXFP8_DECODE_GEMM_TACTIC + +# The TP2 projections at rows the GEMVs do not serve (SHARED_DOWN 17..32 -- SWQ's rows --, over 48) and the bf16 +# activations over the GEMVs' bf16 bounds. +CASES = [ + ((6144, 1536), 17), + ((6144, 1536), 32), + ((4992, 6144), 24), + ((4608, 6144), 64), + ((6144, 4096), 49), + ((3072, 6144), 20), + ((6144, 6144), 12), + ((12288, 6144), 56), +] + + +def test_the_pinned_tactic_is_cute_dsl() -> None: + assert TACTIC == FlashInferMXFP8GemmRunner.CUTE_DSL + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + act, act_sf = mxfp8(spec["m"], spec["k"], g) + w, w_sf = mxfp8(spec["n"], spec["k"], g) + with validating(op): + out = op(act, act_sf, w, w_sf, torch.bfloat16, TACTIC) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp( + out, + op.reference(act, act_sf, w, w_sf, torch.bfloat16, TACTIC), + element_ulp=2.0, + rms_ulp=0.25, + ) + + +@pytest.mark.parametrize("nk,m", CASES, ids=[f"{n}x{k}-m{m}" for (n, k), m in CASES]) +@pytest.mark.parametrize("scale", [1e-3, 1.0, 1e3, "stress"]) +def test_pinned_decision_matches_every_stock_candidate(nk, m, scale) -> None: + """``stress``: every 32-element block of the weight and the activation scaled by its own 2^-12..2^+12.""" + n, k = nk + g = torch.Generator(device="cuda").manual_seed(n + k + m) + w = torch.randn(n, k, generator=g, device="cuda") / k**0.5 + x = torch.randn(m, k, generator=g, device="cuda") + if scale == "stress": + for t in (w, x): + block = torch.randint(-12, 13, (t.shape[0], k // 32), generator=g, device="cuda") + t.mul_(torch.exp2(block.float()).repeat_interleave(32, dim=1)) + else: + x = x * scale + w, x = w.to(torch.bfloat16), x.to(torch.bfloat16) + x[0, ::32] *= 1e3 # one outlier per 32-element block on the first row + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + wq = wq.view(n, k) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + xq = xq.view(m, k) + out = op(xq, xsf, wq, wsf, torch.bfloat16, TACTIC) + runner = FlashInferMXFP8GemmRunner(torch.bfloat16) + for tactic in (FlashInferMXFP8GemmRunner.CUTE_DSL, FlashInferMXFP8GemmRunner.CUTLASS): + ref = runner.forward([xq, xsf, wq, wsf], tactic=tactic) + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)), tactic + again = op(xq, xsf, wq, wsf, torch.bfloat16, TACTIC) + assert torch.equal(out.view(torch.int16), again.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_pinned_quantize.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_pinned_quantize.py new file mode 100644 index 000000000000..e25245246e5d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/gemm/test_modeling_v2_m3_mxfp8_pinned_quantize.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_pinned_quantize entry (MX-TAC): the pinned decode-graph MXFP8 activation quantize is the +stock quantize bit for bit, at every tactic stock's AutoTuner can pick.""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mxfp8_pinned_quantize import ( # noqa: E501 + m3_mxfp8_pinned_quantize as op, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_pinned_quantize requires a CUDA device" + +TACTIC = modeling.MXFP8_DECODE_QUANT_TACTIC + + +def test_the_pinned_tactic_is_cute_dsl() -> None: + assert TACTIC == MXFP8QuantizeRunner.CUTE_DSL + + +def _check(x: torch.Tensor) -> None: + m, k = x.shape + with validating(op): + xq, xsf = op(x, TACTIC) + tq, tsf = op.reference(x, TACTIC) + assert torch.equal(xq.view(torch.uint8).view(m, k), tq.view(torch.uint8).view(m, k)) + assert torch.equal(xsf.view(torch.uint8).flatten(), tsf.view(torch.uint8).flatten()) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +@pytest.mark.parametrize("scale", [1e-3, 1.0, 1e3]) +def test_certified_cells(cell, scale) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = (torch.randn(spec["m"], spec["k"], generator=g, device="cuda") * scale).to(torch.bfloat16) + x[0, ::32] *= 1e3 # one outlier per 32-element block on the first row + _check(x) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/_m3_moe_operands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/_m3_moe_operands.py new file mode 100644 index 000000000000..818926985778 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/_m3_moe_operands.py @@ -0,0 +1,171 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One tp2 rank's MiniMax-M3.1 MoE layer operands (expert width 1536 per rank, the shared expert's too) for the +decode-engine entries' tests. + +The routed experts are random checkpoint-format NVFP4 tensors ([w3 (up); w1 (gate)] rows, e4m3 block scales) put +through TRT-LLM's own TRTLLM-Gen NVFP4 loader shuffle (``NVFP4TRTLLMGenFusedMoEMethod._shuffle_all_experts``), which +writes the buffers the engines read in place. The per-expert scalars follow the loader's conventions (act_beta and +act_clamp divided by fc31_alpha, fc31_scale_c = fc2_input_scale x fc31_alpha) and are sized so the FC1 sums land +around the SwiGLU-OAI clamp and the outputs near unit scale. The shared expert's MXFP8 weights come from +mxfp8_quantize with the swizzled scale layout, as the MXFP8 linear method stores them. +""" + +import functools +from types import SimpleNamespace + +import torch + +H, I_TP, NUM_EXPERTS, TOP_K = 6144, 1536, 128, 4 +SHARED_I = 1536 +SWIGLU_ALPHA, SWIGLU_LIMIT = 1.702, 7.0 +ROUTED_SCALING = 2.0 +MINIMAX2_ROUTING = 5 # RoutingMethodType.MiniMax2 +# NVFP4 global scales: 448 x 6 over the amax each side is sized for. +FC31_INPUT_SCALE = 448.0 * 6.0 / 8.0 +FC2_INPUT_SCALE = 448.0 * 6.0 / 24.0 + + +def _rand_e4m3_scales(shape, gen) -> torch.Tensor: + """E4M3 block scales in [0.5, 2): biased exponents 6 and 7, any mantissa.""" + return torch.randint(0x30, 0x40, shape, dtype=torch.uint8, device="cuda", generator=gen).view( + torch.float8_e4m3fn + ) + + +@functools.lru_cache(maxsize=None) +def experts(seed: int = 20261013): + """The routed experts and the router of one MoE layer, in the layout and conventions of the TRTLLM-Gen loader. + + Returns a SimpleNamespace: ``tensors`` (the ten expert operands in ``M3MoeState.layer``'s order), ``bias``, + ``router_weight``, ``fc31_input_scale`` and ``shared`` (gate_up, its scales, down, its scales). + """ + from tensorrt_llm._torch.moe.fused_moe.quantization import NVFP4TRTLLMGenFusedMoEMethod + + gen = torch.Generator(device="cuda").manual_seed(seed) + e = NUM_EXPERTS + w31 = torch.randint( + 0, 256, (e, 2 * I_TP, H // 2), dtype=torch.uint8, device="cuda", generator=gen + ) + w2 = torch.randint(0, 256, (e, H, I_TP // 2), dtype=torch.uint8, device="cuda", generator=gen) + w31s = _rand_e4m3_scales((e, 2 * I_TP, H // 16), gen) + w2s = _rand_e4m3_scales((e, H, I_TP // 16), gen) + module = SimpleNamespace( + w3_w1_weight=w31, + w2_weight=w2, + w3_w1_weight_scale=w31s, + w2_weight_scale=w2s, + is_gated_activation=True, + bias=False, + ) + NVFP4TRTLLMGenFusedMoEMethod()._shuffle_all_experts(module) + torch.cuda.synchronize() + + # Raw FC1 sums: rms(x) x FC31_INPUT_SCALE x rms(E2M1 value) x rms(block scale) x sqrt(H) ~ 1.0e5 at rms(x) = 1; + # fc31_alpha brings them to ~1.5 in the dequantized domain, the FC2 alpha the outputs to ~1. + base = 1.5 / (FC31_INPUT_SCALE * 2.93 * 1.3 * H**0.5) + fc31_alpha = base * (0.8 + 0.4 * torch.rand(e, device="cuda", generator=gen)) + fc2_alpha = (0.8 + 0.4 * torch.rand(e, device="cuda", generator=gen)) / ( + FC2_INPUT_SCALE * 2.93 * 1.3 * I_TP**0.5 + ) + tensors = ( + w31, + w31s, + w2, + w2s, + fc31_alpha, + FC2_INPUT_SCALE * fc31_alpha, # fc31_scale_c + torch.full((e,), SWIGLU_ALPHA, device="cuda"), # act_alpha + 1.0 / fc31_alpha, # act_beta: the (up + 1) offset over fc31_alpha + SWIGLU_LIMIT / fc31_alpha, # act_clamp over fc31_alpha + fc2_alpha, + ) + bias = (0.02 * torch.randn(e, device="cuda", generator=gen)).float() + router_weight = (torch.randn(e, H, device="cuda", generator=gen) / H**0.5).float().contiguous() + fc31_input_scale = torch.tensor([FC31_INPUT_SCALE], dtype=torch.float32, device="cuda") + + gu = (0.02 * torch.randn(2 * SHARED_I, H, device="cuda", generator=gen)).to(torch.bfloat16) + dn = (0.05 * torch.randn(H, SHARED_I, device="cuda", generator=gen)).to(torch.bfloat16) + gu_q, gu_sf = torch.ops.trtllm.mxfp8_quantize(gu, True) + dn_q, dn_sf = torch.ops.trtllm.mxfp8_quantize(dn, True) + shared = (gu_q.view(2 * SHARED_I, H), gu_sf, dn_q.view(H, SHARED_I), dn_sf) + return SimpleNamespace( + tensors=tensors, + bias=bias, + router_weight=router_weight, + fc31_input_scale=fc31_input_scale, + shared=shared, + ) + + +# Routing cases: random (up to 4 M distinct experts), and every token on the same four experts (the bias outweighs +# any sigmoid difference), which leaves later slot groups empty. +CONCENTRATED = (5, 40, 77, 126) + + +def layer_for(state, case: str, shared: bool = True): + """One layer on ``state`` over :func:`experts` for a routing case. ``shared=False``: the shared expert's weights + are zero, so the engine returns the routed part alone (bf16(0 + bf16(routed))).""" + ops = experts() + bias = ops.bias.clone() + if case == "concentrated": + bias[list(CONCENTRATED)] += 1.0 + shared_weights = ops.shared + if not shared: + gu, gu_sf, dn, dn_sf = ops.shared + shared_weights = (torch.zeros_like(gu), gu_sf, torch.zeros_like(dn), dn_sf) + return state.layer( + *ops.tensors, + bias, + router_weight=ops.router_weight, + fc31_input_scale=ops.fc31_input_scale, + shared_weights=shared_weights, + ) + + +def stock_routed( + hidden: torch.Tensor, topk_ids: torch.Tensor, topk_weights: torch.Tensor +) -> torch.Tensor: + """The stock TRTLLM-Gen NVFP4 MoE (fp4_quantize, then fp4_block_scale_moe_runner with SwiGLU-OAI) on the given + routing, over :func:`experts`: what the built-in MiniMaxM3MoE computes for the routed part.""" + import tensorrt_llm._torch.custom_ops # noqa: F401 -- registers the stock ops + + ops = experts() + w31, w31s, w2, w2s, fc31_alpha, scale_c, act_alpha, act_beta, act_clamp, fc2_alpha = ops.tensors + xq, xsf = torch.ops.trtllm.fp4_quantize(hidden, ops.fc31_input_scale, 16, False, False) + out = torch.ops.trtllm.fp4_block_scale_moe_runner( + None, None, xq, xsf.view(torch.float8_e4m3fn).flatten(), + w31, w31s, None, act_alpha, act_beta, act_clamp, w2, w2s, None, + scale_c, fc31_alpha, fc2_alpha, + NUM_EXPERTS, TOP_K, None, None, I_TP, 0, NUM_EXPERTS, ROUTED_SCALING, MINIMAX2_ROUTING, True, 0, + topk_weights=topk_weights.to(torch.bfloat16), topk_ids=topk_ids.to(torch.int32), + ) # fmt: skip + return out[0] if isinstance(out, (list, tuple)) else out + + +def hidden_states(m: int, seed: int) -> torch.Tensor: + gen = torch.Generator(device="cuda").manual_seed(seed) + return torch.randn(m, H, device="cuda", generator=gen).to(torch.bfloat16) + + +def check_routed_part(op, state, m: int, seed: int) -> None: + """Exact, independent of the reference: the engine's routing (its route_dbg record: ids, then the bf16 weights' + fp32 bits) is the reference routing bit for bit, and its routed part is the stock TRTLLM-Gen MoE's output on that + routing bit for bit -- the claim the engines are built on.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.moe.m3_moe_engine import ( # noqa: E501 + moe_route, + ) + + layer = layer_for(state, "random", shared=False) + hidden = hidden_states(m, seed=seed) + out = op(hidden, layer) + ids = state.route_dbg[:, :4].long() + weights = state.route_dbg[:, 4:].contiguous().view(torch.float32) + ref_ids, ref_weights = moe_route(hidden.float(), layer.router_weight, layer.bias) + assert torch.equal(ids, ref_ids) and torch.equal(weights, ref_weights), ( + "routing differs from the reference" + ) + stock = stock_routed(hidden, ids, weights) + assert torch.equal(out.view(torch.int16), stock.view(torch.int16)), ( + "routed part differs from TRTLLM-Gen's" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/_m3_router.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/_m3_router.py new file mode 100644 index 000000000000..0d980f190b04 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/_m3_router.py @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Shared operands and gates of the router entry's tests (m3_router_tc: the sm_107 TP8 wrapper, 2..48 rows, with and +without the NVFP4 activation).""" + +import zlib + +import torch + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp # noqa: E402 — needs the path declared above + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.moe.m3_router_tc import ( # noqa: E501 + _E2M1_MIDPOINTS, + nvfp4_quantize, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.moe.m3_router_tc import ( # noqa: E501 + m3_router_tc as tc, +) + +__all__ = ["SF_SCALE", "_build", "_compare_fp4", "_compare_logits", "_seed", "tc"] + +#: The experts' NVFP4 input scale the tc cells use: 448 * 6 / 8, block scales well inside E4M3. +SF_SCALE = 336.0 +#: The kernel builds each block's output scale through two ``rcp.approx.ftz.f32`` reciprocals (~2^-23 relative): a +#: value this close to an e2m1 midpoint may round to either neighbour, everything further out must match (the sm_107 +#: TP8 router's TIE_WINDOW; the tp2 wrapper carries the reference quantize but not the window). +TIE_WINDOW = 2.0**-20 + + +def _build(m: int, e: int, k: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + x = (torch.randn(m, k, generator=g, device="cuda") * 2.0).to(torch.bfloat16) + w = torch.randn(e, k, generator=g, device="cuda") * 0.02 + return x, w + + +def _seed(cell) -> int: + return zlib.crc32(cell.why.encode()) % 2**31 + + +def _compare_logits(out: torch.Tensor, ref: torch.Tensor) -> None: + """fp32 logits that differ from the reference only in summation order. + + Over K = 6144 products a reordered fp32 sum moves by a few ulps of the sum of magnitudes, which near + cancellation is ~sqrt(K) times the row's largest logit's ulp at most. Gated in fp32 ulps of each row's largest + logit: 64 element-wise, 8 in relative RMS. The tensor-core logits against a true fp32 product (the TF32 rounding + the band must see) sit at 2200+ / 1700 ulp. + """ + assert out.dtype == ref.dtype == torch.float32 and out.shape == ref.shape + assert_within_ulp(out, ref, element_ulp=64.0, rms_ulp=8.0) + + +def _near_tie(scaled: torch.Tensor) -> torch.Tensor: + """Elements whose e2m1 rounding the kernel's approximate reciprocals decide (see ``TIE_WINDOW``).""" + mid = torch.tensor(_E2M1_MIDPOINTS, device=scaled.device, dtype=torch.float32) + a = scaled.abs().unsqueeze(-1) + return ((a - mid).abs() <= TIE_WINDOW * a).any(-1) + + +def _codes(packed: torch.Tensor) -> torch.Tensor: + """[M, K / 2] packed e2m1 bytes -> [M, K] 4-bit codes, low nibble first.""" + lo, hi = packed & 0xF, packed >> 4 + return torch.stack((lo, hi), dim=-1).reshape(packed.shape[0], -1) + + +def _compare_fp4( + q: torch.Tensor, sf: torch.Tensor, x: torch.Tensor, sf_scale: torch.Tensor +) -> None: + """Scales bit-exact; codes bit-exact except within ``TIE_WINDOW`` of an e2m1 midpoint, where the kernel's + approximate reciprocals choose the side (and then only by one code).""" + ref_q, ref_sf, scaled = nvfp4_quantize(x, sf_scale) + assert q.dtype == torch.uint8 and q.shape == ref_q.shape + assert torch.equal(sf, ref_sf), f"{(sf != ref_sf).sum().item()} block scales differ" + got, want = _codes(q), _codes(ref_q) + differ = got != want + assert not bool((differ & ~_near_tie(scaled)).any()), ( + f"{(differ & ~_near_tie(scaled)).sum().item()} e2m1 codes differ away from a rounding tie" + ) + magnitude = ((got & 7).int() - (want & 7).int()).abs() + assert bool((magnitude[differ] <= 1).all()), "a tie resolved more than one code away" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_moe16_tp2_off.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_moe16_tp2_off.py new file mode 100644 index 000000000000..857c86f2789e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_moe16_tp2_off.py @@ -0,0 +1,78 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MOE16_TP2: the 9..13-row MoE engine is off in the tp2 baseline. + +The engine's kernel (kernels/m3_moe/m3_moe16_kernel.py) is the TP4 build, its routed and shared expert widths fixed at +the TP4 768 per rank; the tp2 target's experts are 1536 wide. So by default MOE16_ENGINE covers no rows, nothing builds +it and moe_path sends 9..13-row steps to the generic MoE; the opt-in (TRTLLM_M31_ENABLE_MOE16_TP2) restores the 9..13 +rows, and TRTLLM_M31_DISABLE_MOE16_TP2 wins over it. CPU only. +""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + modeling, +) + +_ENV = ("TRTLLM_M31_ENABLE_MOE16_TP2", "TRTLLM_M31_DISABLE_MOE16_TP2") + + +def test_moe16_is_off_by_default() -> None: + assert modeling.MOE16_TP2 is False + assert modeling.MOE16_MAX_TOKENS == 0 + assert modeling.MOE16_ENGINE.keys == frozenset() + for rows in range(9, 14): + assert not modeling.MOE16_ENGINE.covers(rows) + assert modeling.moe_path(rows) != modeling.MOE16_ENGINE.path + # The MC finalize takes them (TP2_MC_FINALIZE_SMALL: from 9 rows with MoE16 off); with its kill (33) or + # TP2W_MC_FINALIZE's (no rows) the generic MoE runs. + want = ( + modeling.MC_FINALIZE.path + if modeling.TP2W_MC_FINALIZE and modeling.MC_FINALIZE_MIN_TOKENS <= 9 + else None + ) + assert all(modeling.moe_path(rows) == want for rows in range(9, 14)) + + +def test_the_moe16_kernel_is_the_tp4_width_build() -> None: + """Why it is off: the kernel's expert widths are TP4's, not the target's INTER.""" + kernel = pytest.importorskip( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels" + ".m3_moe.m3_moe16_kernel" + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.m3_moe import ( # noqa: E501 + op as moe_op, + ) + + assert kernel.I_TP == 768 and kernel.SH_I == 768 + assert moe_op.INTER == 1536 + + +def _reload(monkeypatch, enable: bool, disable: bool): + for name, on in zip(_ENV, (enable, disable)): + if on: + monkeypatch.setenv(name, "1") + else: + monkeypatch.delenv(name, raising=False) + importlib.reload(m31_switches) + return importlib.reload(modeling) + + +@pytest.mark.parametrize( + "enable, disable, rows", + [(False, False, ()), (True, False, tuple(range(9, 14))), (True, True, ())], + ids=["default", "opt_in", "kill_wins"], +) +def test_switches(monkeypatch, enable, disable, rows) -> None: + try: + m = _reload(monkeypatch, enable, disable) + assert m.MOE16_TP2 is bool(rows) + assert m.MOE16_ENGINE.keys == frozenset(rows) + finally: + for name in _ENV: + monkeypatch.delenv(name, raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_finalize.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_finalize.py new file mode 100644 index 000000000000..25ab50872ec1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_finalize.py @@ -0,0 +1,73 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2W_MOE_FINALIZE (a port of the M3 tp2 test): kernels/m3_moe_finalize finalizes the routed experts (+ the shared +expert) from the TRTLLM-Gen runner's unfinalized parts, bitwise to the runner's finalizeKernelVecLoad order (acc = 0; +acc += w_k * fc2[idx_k] for k in order, skipping -1; one BF16 rounding) and to ``shared.add_(routed)``, at the rows the +target routes there (49..16K), with dropped (-1) slots, padded FC2 rows, eager and CUDA-graph replay.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.m3_moe_finalize import ( # noqa: E501 + m3_moe_finalize, +) + +H, TOP_K = 6144, 4 + + +def _parts(rows: int, seed: int, pad: int = 128): + g = torch.Generator(device="cuda").manual_seed(seed) + n = rows * TOP_K + fc2 = torch.randn((n + pad, H), device="cuda", generator=g).to(torch.bfloat16) + idx = torch.randperm(n, device="cuda", generator=g).to(torch.int32).reshape(rows, TOP_K) + drop = torch.rand((rows, TOP_K), device="cuda", generator=g) < 0.02 + idx = torch.where(drop, torch.full_like(idx, -1), idx) + w = torch.rand((rows, TOP_K), device="cuda", generator=g).to(torch.bfloat16) + shared = torch.randn((rows, H), device="cuda", generator=g).to(torch.bfloat16) + return fc2, w, idx.contiguous(), shared + + +def _reference(fc2, w, idx): + acc = torch.zeros((w.shape[0], H), dtype=torch.float32, device=fc2.device) + for k in range(TOP_K): + valid = (idx[:, k] >= 0).unsqueeze(1) + x = fc2[idx[:, k].clamp(min=0).long()].float() + acc = torch.where(valid, acc + w[:, k : k + 1].float() * x, acc) + return acc.to(torch.bfloat16) + + +@pytest.mark.parametrize("rows", [49, 64, 65, 192, 1000, 4096, 16384]) +def test_bitwise_to_the_runner_order(rows: int) -> None: + fc2, w, idx, shared = _parts(rows, rows) + ref = _reference(fc2, w, idx) + out = m3_moe_finalize(fc2, w, idx) + assert out is not None and torch.equal(out.view(torch.int16), ref.view(torch.int16)) + fused = m3_moe_finalize(fc2, w, idx, shared) + assert torch.equal(fused.view(torch.int16), shared.clone().add_(ref).view(torch.int16)) + + +def test_graph_replay_matches_eager() -> None: + fc2, w, idx, shared = _parts(512, 7) + eager = m3_moe_finalize(fc2, w, idx, shared) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = m3_moe_finalize(fc2, w, idx, shared) + for seed in (8, 9): + f2, w2, i2, s2 = _parts(512, seed) + fc2.copy_(f2) + w.copy_(w2) + idx.copy_(i2) + shared.copy_(s2) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out, m3_moe_finalize(fc2, w, idx, shared)) + assert eager.shape == out.shape + + +def test_declines_other_layouts() -> None: + fc2, w, idx, shared = _parts(64, 3) + assert m3_moe_finalize(fc2.float(), w, idx) is None + assert m3_moe_finalize(fc2, w, idx.long()) is None + assert m3_moe_finalize(fc2, w, idx, shared[:, :128].contiguous()) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_finalize_rows.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_finalize_rows.py new file mode 100644 index 000000000000..ffb4d703221a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_finalize_rows.py @@ -0,0 +1,199 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2W_MOE_FINALIZE (TP2 WIDE F2): who finalizes the tp2 target's MoE at each row count. From +MOE_TRITON_FINALIZE_MIN_TOKENS (the first row past the MC finalize's 48), a finalized MoE forward in one runner call +takes the runner's FC2 output unfinalized and kernels/m3_moe_finalize adds it up with the shared expert; the MC finalize +keeps 33..48, the baseline <= 32, a forward that returns its parts (the boundary folds the add) or pushes keeps the +runner's finalize, and the kill switch restores the runner's finalize everywhere. + +CPU only: the runner, the shared expert and the finalize kernel are stubs. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) + +H, TOP_K = 64, 4 + + +def test_the_rows_meet_the_mc_finalize() -> None: + assert modeling.MOE_TRITON_FINALIZE_MIN_TOKENS == 49 + assert modeling.MOE_TRITON_FINALIZE_MIN_TOKENS == modeling.DECODE_MAX_TOKENS + 1 + if modeling.TP2W_MC_FINALIZE: + assert modeling.MOE_TRITON_FINALIZE_MIN_TOKENS == modeling.MC_FINALIZE_MAX_TOKENS + 1 + + +def _core(max_rows: int = 8192): + return SimpleNamespace( + moe_events=[(None, None)] * 4, + aux_stream_moe=None, + moe_max_num_tokens=max_rows, + engine=lambda *a, **k: None, + ) + + +def _parts(rows: int): + fc2 = torch.randn(rows * TOP_K + 8, H).to(torch.bfloat16) + scales = torch.rand(rows * TOP_K).to(torch.bfloat16) # flat, as the runner returns them + perm = torch.randperm(rows * TOP_K).to(torch.int32) + return [fc2, scales, perm] + + +@pytest.fixture +def calls(monkeypatch): + seen = [] + + def finalize(fc2, weights, idx, shared=None): + seen.append((weights.shape, idx.dtype, shared is not None)) + acc = torch.zeros(weights.shape[0], fc2.shape[1]) + for k in range(weights.shape[1]): + acc += weights[:, k : k + 1].float() * fc2[idx[:, k].long()].float() + out = acc.to(torch.bfloat16) + return out if shared is None else (shared.float() + out.float()).to(torch.bfloat16) + + monkeypatch.setattr(modeling, "m3_moe_finalize", finalize) + monkeypatch.setattr(modeling, "flashinfer_mxfp8_autotune_active", lambda: False) + return seen + + +def _decode(rows: int, max_rows: int = 8192): + shared = torch.randn(rows, H).to(torch.bfloat16) + parts = _parts(rows) + routed_calls = [] + + def routed(i, h, do_finalize): + routed_calls.append(do_finalize) + return parts if not do_finalize else torch.zeros(rows, H, dtype=torch.bfloat16) + + target = SimpleNamespace( + core=_core(max_rows), + _routed=routed, + _swiglu_mlp=lambda h, prefix, low_m, push=None: (shared.clone(), False), + ) + return target, shared, routed_calls + + +@pytest.mark.parametrize("rows", [49, 64, 192]) +def test_decode_target_finalizes_the_wide_rows(calls, rows: int) -> None: + target, shared, routed_calls = _decode(rows) + h = torch.zeros(rows, H, dtype=torch.bfloat16) + out, routed, engine_pushed, shared_pushed = modeling.DecodeTarget._moe( + target, 3, h, return_parts=False + ) + assert routed_calls == [False] and len(calls) == 1 + assert calls[0] == ((rows, TOP_K), torch.int32, True) + assert routed is None and not engine_pushed and not shared_pushed + assert out.shape == (rows, H) + + +@pytest.mark.parametrize( + "rows, return_parts, max_rows", + [ + (48, False, 8192), # the MC finalize / baseline rows + (32, False, 8192), + (64, True, 8192), # the boundary folds shared + routed (multicast all-reduce) + (64, False, 32), # more rows than one runner call + ], +) +def test_decode_target_keeps_the_runner_finalize(calls, rows, return_parts, max_rows) -> None: + target, shared, routed_calls = _decode(rows, max_rows) + h = torch.zeros(rows, H, dtype=torch.bfloat16) + modeling.DecodeTarget._moe(target, 3, h, return_parts=return_parts) + assert routed_calls == [True] and calls == [] + + +def test_kill_switch_keeps_the_runner_finalize(calls, monkeypatch) -> None: + monkeypatch.setattr(modeling, "TP2W_MOE_FINALIZE", False) + target, shared, routed_calls = _decode(64) + modeling.DecodeTarget._moe( + target, 3, torch.zeros(64, H, dtype=torch.bfloat16), return_parts=False + ) + assert routed_calls == [True] and calls == [] + + +def test_the_unfinalized_call_keeps_its_parts(calls) -> None: + """The MC finalize's own do_finalize=False call still gets the raw parts.""" + target, shared, routed_calls = _decode(64) + out, *_ = modeling.DecodeTarget._moe( + target, 3, torch.zeros(64, H, dtype=torch.bfloat16), return_parts=False, do_finalize=False + ) + assert routed_calls == [False] and calls == [] and len(out) == 4 + + +@pytest.mark.parametrize( + "keyed, enabled, ctxnormq, capturing, slot_present", + [ + (False, True, True, False, True), + (False, False, True, False, True), + (False, True, False, False, True), + (True, False, False, False, True), + (False, True, True, True, True), + (False, True, True, False, False), + ], +) +def test_prefill_target_finalizes_past_the_decode_rows( + calls, monkeypatch, keyed, enabled, ctxnormq, capturing, slot_present +) -> None: + """SP placeholders require their slot even with the item killed; only real uncaptured rows can run the oracle.""" + rows = 512 + parts = _parts(rows) + shared = torch.randn(rows, H).to(torch.bfloat16) + h = torch.zeros(rows, H, dtype=torch.bfloat16) + fp4 = ( + torch.zeros(rows, H // 2, dtype=torch.uint8), + torch.zeros(rows, H // 16, dtype=torch.uint8), + ) + finalize_flags, inputs, takes, checks = [], [], [], [] + + def take(value): + assert value is h + takes.append(value) + return fp4 if slot_present else None + + def quantize(value, *args): + assert value is h and not keyed and not capturing + checks.append(value) + return fp4 + + def routed_experts(core, i, x, x_sf, logits, do_finalize): + finalize_flags.append(do_finalize) + inputs.append((x, x_sf)) + return parts + + monkeypatch.setattr(modeling, "TP2W_FP4SLOT_FINALIZE", enabled) + monkeypatch.setattr(modeling, "CTXNORMQ", ctxnormq) + monkeypatch.setattr(modeling, "_FP4SLOT_CHECK", True) + monkeypatch.setattr(modeling, "_FP4SLOT_STATE", {"taken": 0, "checked": 0, "diff": 0}) + monkeypatch.setattr(modeling, "_sp_keyed", lambda value: keyed) + monkeypatch.setattr(modeling, "_CTXNORMQ_FP4", SimpleNamespace(take=take)) + monkeypatch.setattr(modeling, "fp4_quantize", quantize) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: capturing) + monkeypatch.setattr(modeling, "_routed_experts", routed_experts) + monkeypatch.setattr(modeling, "_router_logits", lambda core, i, h: None) + core = _core() + core.moe = [SimpleNamespace(fc31_input_scale=1)] * 4 + target = SimpleNamespace( + core=core, _swiglu_mlp=lambda h, prefix, low_m: shared.clone(), _routed=None + ) + out, routed = modeling.PrefillTarget._moe(target, 3, h, return_parts=False) + consume = keyed or (enabled and ctxnormq) + assert len(takes) == int(consume) + assert len(checks) == int(consume and slot_present and not keyed and not capturing) + expected = fp4 if consume and slot_present else (h, None) + assert inputs[0][0] is expected[0] and inputs[0][1] is expected[1] + assert finalize_flags == [False] and len(calls) == 1 and routed is None + + +def test_a_non_runner_layout_raises() -> None: + with pytest.raises(RuntimeError, match="not the trtllm-gen layout"): + modeling._triton_finalize(64, torch.zeros(64, H), None) + + +def test_the_decode_layer_asks_for_parts_only_where_the_boundary_folds_them() -> None: + with open(modeling.__file__) as f: + assert "return_parts=plan.moe_mc_ar or moe_push is not None" in f.read() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_tactic_pin.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_tactic_pin.py new file mode 100644 index 000000000000..786270127736 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_tactic_pin.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Local/QA dispatch contract; GPU LOGITS-route equivalence is a separate gate.""" + +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.moe import ( + m3_moe_tactic_pin as pin, +) + +pytestmark = pytest.mark.cpu_only + + +def _fixture(monkeypatch, rows=96, valid=True): + calls = [] + + class Runner: + def __init__(self, *args, **kwargs): + calls.append(("init", args, kwargs)) + + def get_runner(self): + return self + + def get_valid_configs(self, *args): + calls.append(("valid", args)) + return [[16, 128], [32, 64], [32, 96]] if valid else [[-1, -1]] + + def forward(self, inputs, tactic): + calls.append(("forward", inputs, tactic)) + return ["result"] + + monkeypatch.setattr(pin, "FP4BlockScaleMoERunner", Runner) + monkeypatch.setattr(pin, "ENABLED", True) + core = SimpleNamespace( + mapping=SimpleNamespace(tp_size=2, tp_rank=1), + hidden=6144, + moe_inter=1536, + num_experts=128, + top_k=4, + routed_scaling_factor=1.0, + max_num_tokens=16384, + ) + inputs = [object() for _ in range(16)] + inputs[2] = SimpleNamespace(shape=(rows, 3072), device="cuda:1") + return core, inputs, calls + + +@pytest.mark.parametrize( + "rows,expected", + [ + (32, None), + (95, None), + (96, (16, 128)), + (112, (16, 128)), + (127, (16, 128)), + (128, None), + (231, None), + (232, (32, 64)), + (233, None), + (255, None), + (256, (32, 96)), + (257, None), + (448, None), + ], +) +def test_table_boundaries(rows, expected): + assert pin.tactic_for_rows(rows) == expected + + +@pytest.mark.parametrize("finalize", [False, True]) +def test_preserves_inputs_and_validates_once(monkeypatch, finalize): + core, inputs, calls = _fixture(monkeypatch) + for _ in range(2): + assert pin.run_pinned(core, inputs, finalize, 4, 0) == ["result"] + assert sum(c[0] == "valid" for c in calls) == 1 + for c in calls: + if c[0] == "forward": + assert all(a is b for a, b in zip(c[1][:16], inputs)) + assert c[1][-2:] == [None, None] + assert c[2] == [16, 128] + assert calls[0][1][-2:] == (finalize, 0) + + +def test_invalid_tactic_declines_and_caches(monkeypatch): + core, inputs, calls = _fixture(monkeypatch, valid=False) + for _ in range(2): + assert pin.run_pinned(core, inputs, False, 4, 0) is None + assert [c[0] for c in calls] == ["init", "valid"] + + +def test_kill_switch_does_not_construct_runner(monkeypatch): + core, inputs, calls = _fixture(monkeypatch) + monkeypatch.setattr(pin, "ENABLED", False) + assert pin.run_pinned(core, inputs, False, 4, 0) is None + assert not calls + + +@pytest.mark.parametrize("rows", [1, 32, 64, 128, 224, 448]) +def test_unserved_rows_keep_tuner(monkeypatch, rows): + core, inputs, calls = _fixture(monkeypatch, rows=rows) + assert pin.run_pinned(core, inputs, False, 4, 0) is None + assert not calls + + +def test_other_geometry_keeps_tuner(monkeypatch): + core, inputs, calls = _fixture(monkeypatch) + core.mapping.tp_size = 4 + assert pin.run_pinned(core, inputs, False, 4, 0) is None + assert not calls diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_tactic_pin_gpu.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_tactic_pin_gpu.py new file mode 100644 index 000000000000..3fdacd36608c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_tp2w_moe_tactic_pin_gpu.py @@ -0,0 +1,135 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""LOGITS-route bitwise gate against frozen K3 tactics and CTX's MB references. + +Uses the existing loader-shuffled M3.1 operands. Both native finalization and the +production unfinalized-parts/Triton-finalize path are checked, eager and captured. +Local/QA gate, not a performance benchmark or a model-level numerical gate. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.m3_moe_finalize import ( # noqa: E501 + m3_moe_finalize, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.moe import ( + m3_moe_tactic_pin as pin, +) +from tensorrt_llm._torch.custom_ops.trtllm_gen_custom_ops import FP4BlockScaleMoERunner + +__extra_import_path__ = ["."] +from _m3_moe_operands import ( # noqa: E402 + CONCENTRATED, + MINIMAX2_ROUTING, + ROUTED_SCALING, + experts, + hidden_states, +) + + +def _inputs(rows, seed, concentrated): + ops = experts() + w31, s31, w2, s2, alpha31, scale_c, act_alpha, beta, clamp, alpha2 = ops.tensors + hidden = hidden_states(rows, seed) + x, sf = torch.ops.trtllm.fp4_quantize(hidden, ops.fc31_input_scale, 16, False, False) + logits = torch.nn.functional.linear(hidden.float(), ops.router_weight).contiguous() + bias = ops.bias.clone() + if concentrated: + bias[list(CONCENTRATED)] += 1.0 + return [ + logits, + bias, + x, + sf.view(torch.float8_e4m3fn).flatten(), + w31, + s31, + None, + act_alpha, + beta, + clamp, + w2, + s2, + None, + scale_c, + alpha31, + alpha2, + ] + + +def _finish(parts, finalize): + return parts[0] if finalize else m3_moe_finalize(*parts) + + +@pytest.mark.parametrize("rows", [96, 104, 112, 120, 127, 232, 256]) +@pytest.mark.parametrize("finalize", [False, True]) +def test_logits_route_bitwise_eager_and_graph(monkeypatch, rows, finalize): + assert torch.cuda.is_available() and torch.cuda.get_device_capability() == (10, 7) + monkeypatch.setattr(pin, "ENABLED", True) + core = SimpleNamespace( + mapping=SimpleNamespace(tp_size=2, tp_rank=0), + hidden=6144, + moe_inter=1536, + num_experts=128, + top_k=4, + routed_scaling_factor=ROUTED_SCALING, + max_num_tokens=16384, + ) + reference = FP4BlockScaleMoERunner( + 128, + 4, + None, + None, + 1536, + 0, + 128, + ROUTED_SCALING, + MINIMAX2_ROUTING, + finalize, + 0, + tune_max_num_tokens=16384, + use_dp=False, + ) + # Frozen843057 both-rank cache09a8fe22, buckets64/128/256. + stored = [16, 196] if rows == 256 else ([16, 236] if rows == 232 else [16, 226]) + ctx = [16, 130] if rows == 256 else ([16, 96] if rows == 232 else [8, 187]) + valid = reference.get_runner().get_valid_configs(4, 6144, 1536, 128, rows) + assert all( + t in [list(v) for v in valid] for t in (stored, ctx, list(pin.tactic_for_rows(rows))) + ) + + def candidate(inputs): + parts = pin.run_pinned(core, inputs, finalize, MINIMAX2_ROUTING, 0) + assert parts is not None, "pinned path declined" + return _finish(parts, finalize) + + inputs = _inputs(rows, 1, False) + candidate(inputs) # eager initialization before capture (runner and finalize) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + captured = candidate(inputs) + torch.cuda.current_stream().wait_stream(stream) + for seed in (2, 3, 4): + for concentrated in (False, True): + fresh = _inputs(rows, seed, concentrated) + for index in (0, 1, 2, 3): + inputs[index].copy_(fresh[index]) + eager = candidate(inputs).clone() + graph.replay() + torch.cuda.synchronize() + assert torch.equal(captured.view(torch.int16), eager.view(torch.int16)) + for tactic in (stored, ctx): + expected = _finish( + reference.forward([*inputs, None, None], tactic=tactic), finalize + ) + assert torch.equal(eager.view(torch.int16), expected.view(torch.int16)), ( + rows, + finalize, + seed, + concentrated, + tactic, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_x1.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_x1.py new file mode 100644 index 000000000000..b32acfead803 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m31_x1.py @@ -0,0 +1,213 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The X1 router payload reaches the real expert call without changing fallback routing.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as m, +) + + +@pytest.mark.parametrize("rows", [9, 16, 24, 32, 48]) +@pytest.mark.parametrize("scale_value", [0.125, 1.0, 336.0, 4096.0]) +@pytest.mark.parametrize("kind", ["random", "zero", "tiny", "large", "midpoints"]) +def test_native_payload_exact_eager_and_graph(rows, scale_value, kind): + """Donor payload contract, without a rounding-tie tolerance, against today's quantizer.""" + assert torch.cuda.get_device_capability() == (10, 7) + gen = torch.Generator(device="cuda").manual_seed(20623 + rows) + x = torch.randn(rows, 6144, device="cuda", generator=gen).to(torch.bfloat16) + w = torch.randn(128, 6144, device="cuda", generator=gen) / 6144**0.5 + scale = torch.tensor([scale_value], dtype=torch.float32, device="cuda") + if kind == "zero": + x.zero_() + x[:, 1::2] = -0.0 + elif kind == "tiny": + x.mul_(2**-12) + elif kind == "large": + x.mul_(2**12) + elif kind == "midpoints": + values = torch.tensor( + [0, 0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5, 6, -0.25, -0.75, -1.25, -1.75, -2.5, -3.5, -5], + device="cuda", + dtype=torch.bfloat16, + ) + x.copy_(values.repeat(rows, 384)) + native = torch.ops.trtllm.minimax_m3_router_tc + ref = [native(x, w, None, True)[0], *m.fp4_quantize(x, scale, 16, False, False)] + actual = native(x, w, scale, True) + assert len(actual) == len(ref) == 3 + for a, b in zip(actual, ref): + assert torch.equal(a.view(torch.uint8).flatten(), b.view(torch.uint8).flatten()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = native(x, w, scale, True) + for _ in range(3): + graph.replay() + torch.cuda.synchronize() + for a, b in zip(captured, ref): + assert torch.equal(a.view(torch.uint8).flatten(), b.view(torch.uint8).flatten()) + + +@pytest.mark.parametrize("rows", [8, 9, 16, 24, 32, 48, 49, 176]) +@pytest.mark.parametrize("cluster", [False, True]) +@pytest.mark.parametrize("enabled", [False, True]) +@pytest.mark.parametrize("finalize", [False, True]) +def test_payload_and_fallback_consumer(monkeypatch, rows, cluster, enabled, finalize): + h = torch.zeros(rows, 64, dtype=torch.bfloat16) + scale = torch.ones(1) + logits = torch.zeros(rows, 128) + q = torch.zeros(rows, 32, dtype=torch.uint8) + sf = torch.zeros(rows * 4, dtype=torch.uint8) + core = SimpleNamespace( + s1b_pins={}, + router_tc_cluster16=cluster, + w={"l3_router": torch.zeros(128, 64)}, + moe={3: SimpleNamespace(fc31_input_scale=scale)}, + moe_max_num_tokens=16384, + ) + calls = [] + + def router(x, weight, input_scale, pdl): + assert x is h and weight is core.w["l3_router"] + calls.append(("router", input_scale)) + return [logits] if input_scale is None else [logits, q, sf] + + def experts(actual_core, layer, x, x_sf, actual_logits, do_finalize, topk=None): + assert actual_core is core and layer == 3 and actual_logits is logits + calls.append(("experts", x, x_sf, do_finalize)) + return "experts" + + target = SimpleNamespace( + core=core, + _routed_chunks=lambda layer, x, router_logits, topk=None: calls.append(("chunks", x)) + or "chunks", + ) + monkeypatch.setattr(m, "X1", enabled) + monkeypatch.setattr(m, "m3_router_tc", router) + monkeypatch.setattr(m, "m3_fp32_linear", lambda *args: logits) + monkeypatch.setattr(m, "_routed_experts", experts) + monkeypatch.setattr(m, "taken", lambda *args: None) + # The parked decode-router-F32 slot is absent from the frozen vehicle. + if hasattr(m, "_DECODE_ROUTER_F32"): + m._DECODE_ROUTER_F32.clear() + out = m.DecodeTarget._routed(target, 3, h, finalize) + fused = enabled and cluster and 9 <= rows <= m.ROUTER_TC_GEN_MAX_ROWS + if fused: + assert calls[0][1] is scale + assert calls[-1][1] is q and calls[-1][2] is sf + assert calls[-1][3] is finalize and out == "experts" + elif finalize: + assert calls[-1][0] == "chunks" and calls[-1][1] is h + assert out == "chunks" + else: + assert calls[-1][1] is h and calls[-1][2] is None + assert calls[-1][3] is False and out == "experts" + + +@pytest.mark.parametrize("rows", [9, 16, 24, 32, 48]) +@pytest.mark.parametrize("finalize", [False, True]) +def test_real_experts_match_current_path_eager_and_graph(monkeypatch, rows, finalize): + """Donor 780d57e886 payload/expert contract through M31's actual routed call.""" + from _m3_moe_operands import experts, hidden_states + + from tensorrt_llm._torch.autotuner import AutoTuner + from tensorrt_llm.mapping import Mapping + + assert torch.cuda.get_device_capability() == (10, 7) + assert torch.ops.trtllm.minimax_m3_router_tc_cluster16(6144) + AutoTuner.get().setup_distributed_state(Mapping()) + ops = experts() + w31, s31, w2, s2, a31, scale_c, act_a, act_b, act_c, a2 = ops.tensors + core = SimpleNamespace( + s1b_pins={}, + router_tc_cluster16=True, + w={ + "l3_router": ops.router_weight, + "l3_router_bias": ops.bias, + "l3_e_w31": w31, + "l3_e_s31": s31, + "l3_e_w2": w2, + "l3_e_s2": s2, + }, + moe={ + 3: SimpleNamespace( + fc31_input_scale=ops.fc31_input_scale, + act_alpha=act_a, + act_beta=act_b, + act_clamp=act_c, + fc31_scale_c=scale_c, + fc31_alpha=a31, + fc2_alpha=a2, + ) + }, + mxfp8=SimpleNamespace( + quant_method=SimpleNamespace( + tune_decode_graph_backends=False, + backend="auto", + _flashinfer_autotuned=False, + needs_native_autotune=False, + ) + ), + moe_max_num_tokens=16384, + max_num_tokens=16384, + num_experts=128, + top_k=4, + moe_inter=1536, + routed_scaling_factor=2.0, + ) + target = object.__new__(m.DecodeTarget) + target.core = core + h = hidden_states(rows, 20623 + rows) + # The parked decode-router-F32 slot is absent from the frozen vehicle. + if hasattr(m, "_DECODE_ROUTER_F32"): + m._DECODE_ROUTER_F32.clear() + + def call(): + output = target._routed(3, h, finalize) + return output if finalize else m._triton_finalize(rows, output, None) + + monkeypatch.setattr(m, "X1", False) + reference = call() + monkeypatch.setattr(m, "X1", True) + actual = call() + assert torch.equal(actual.view(torch.uint8), reference.view(torch.uint8)) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = call() + for _ in range(3): + graph.replay() + torch.cuda.synchronize() + assert torch.equal(captured.view(torch.uint8), reference.view(torch.uint8)) + + +@pytest.mark.parametrize("enabled", [False, True]) +@pytest.mark.parametrize("finalize", [False, True]) +def test_x1_preserves_wide_s1b_topk(monkeypatch, enabled, finalize): + """A wide S1b route must retain its top-k payload with either X1 policy.""" + h = torch.zeros(176, 64, dtype=torch.bfloat16) + topk = (torch.zeros(176, 4, dtype=torch.int32), torch.ones(176, 4)) + core = SimpleNamespace(moe_max_num_tokens=448) + + def route(actual_core, layer, x, fp4=None): + assert actual_core is core and layer == 3 and x is h and fp4 is None + return "topk", topk + + def experts(actual_core, layer, x, sf, logits, do_finalize, actual_topk=None): + assert actual_core is core and x is h and sf is None and logits is None + assert not do_finalize and actual_topk is topk + return topk + + def chunks(layer, x, logits, actual_topk=None): + assert layer == 3 and x is h and logits is None and actual_topk is topk + return topk + + target = SimpleNamespace(core=core, _routed_chunks=chunks) + monkeypatch.setattr(m, "X1", enabled) + monkeypatch.setattr(m, "S1B_FOLD_CHECK", False) + monkeypatch.setattr(m, "_decode_route", route) + monkeypatch.setattr(m, "_routed_experts", experts) + assert m.DecodeTarget._routed(target, 3, h, finalize) is topk diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_moe_engine.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_moe_engine.py new file mode 100644 index 000000000000..67c7f5f66018 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_moe_engine.py @@ -0,0 +1,180 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the moe/m3_moe_engine entry: the 1..8-token MoE engine on its caller-owned state. + +One rank of the tp2 deployment: 128 experts (all local, intermediate 1536 per rank, the shared expert's too), top 4, +hidden 6144, the shared expert folded in, ``num_ctas`` = SMs - 24 as the target builds it (SH20 held off). Operands: +``_m3_moe_operands``. The M3.1 engine levers carried at TP2 are each held bitwise to their kill switch at the token +counts they change: MOE_M8CFG (8 tokens), MOE_VRPOLL (7, 8), MOE_HALFPLAN (every token count; it changes the FC1 +ranges only where the CTA shares are equal), and each state records the items it runs for the gate-e counters. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.moe.m3_moe_engine import ( # noqa: E501 + M3MoeState, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.moe.m3_moe_engine import ( # noqa: E501 + m3_moe_engine as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_moe_operands import H, check_routed_part, hidden_states, layer_for # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_moe_engine requires a CUDA device" + +_STATES = {} + + +def _state(m: int) -> M3MoeState: + if m not in _STATES: + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + _STATES[m] = M3MoeState(dev, num_tokens=m, num_ctas=sms - 24, shared=True) + return _STATES[m] + + +def compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """The engine against the plain-torch MoE: BAND_ELEMENT bf16 ulp of the row's largest magnitude per element, + BAND_RMS ulp relative RMS. + + Both sides quantize the same operands with the same recipes (the activation's and the intermediate's NVFP4 codes + and block scales, the shared expert's MXFP8); they differ in summation order (tensor-core fp32 against fp64) and in + the PTX approximations (rcp / ex2 / tanh). Such a difference changes the output only where it moves a value across + a rounding boundary: one bf16 ulp of a slot output or of the sum, or, rarely, one intermediate NVFP4 code, which + reaches the output as one weight column's share. Measured over every cell, both routing cases and 3 seeds each + (GB300): at most 1.48 ulp element, 0.18 ulp RMS. The band leaves ~3x on both; a misread recipe (the activation's + 1 / (sf x (1 / sc)) spelling of the reciprocal on ~0.06% of its codes) measured 12 ulp element, 7 ulp RMS. + """ + assert_within_ulp(out, ref, element_ulp=BAND_ELEMENT, rms_ulp=BAND_RMS) + + +BAND_ELEMENT = 4.0 +BAND_RMS = 0.5 + + +@pytest.mark.parametrize("case", ("random", "concentrated")) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, case) -> None: + m = cell.spec["tokens"] + layer = layer_for(_state(m), case) + hidden = hidden_states(m, seed=zlib.crc32((cell.why + case).encode()) % 2**31) + with validating(op): + out = op(hidden, layer) + again = op(hidden, layer) + assert out.shape == (m, H) and out.dtype is torch.bfloat16 + # The workspace is tagged per call and never reset: a repeated call is the same call. + assert torch.equal(out.view(torch.int16), again.view(torch.int16)), ( + "a repeated call changed the output" + ) + compare(out, op.reference(hidden, layer)) + + +@pytest.mark.parametrize("m", (1, 2, 8)) +def test_routed_part_is_trtllm_gen_bit_for_bit(m) -> None: + check_routed_part(op, _state(m), m, seed=100 + m) + + +def test_the_engine_is_built_at_the_tp2_width() -> None: + assert moe_op.INTER == moe_op.SHARED_INTER == 1536 + state = _state(1) + assert state.inter == 1536 and state.config["i_tp"] == 1536 + assert state.config.get("sh_i") == 1536 + + +def _killed_state(m: int, item: str, monkeypatch) -> M3MoeState: + """A fresh state for ``m`` tokens with ``item``'s kill switch set (the module constant or switch read at build).""" + if item == "MOE_VRPOLL": + monkeypatch.setattr(moe_op, "VRPOLL", False) + elif item == "MOE_HALFPLAN": + monkeypatch.setattr(moe_op, "HALFPLAN", False) + elif item == "MOE_M8CFG": + monkeypatch.setattr(moe_op, "m31_disabled", lambda item_id: item_id == "MOE_M8CFG") + else: + raise ValueError(item) + sms = torch.cuda.get_device_properties(0).multi_processor_count + return M3MoeState(torch.device("cuda"), num_tokens=m, num_ctas=sms - 24, shared=True) + + +LEVER_CASES = [("MOE_M8CFG", 8), ("MOE_VRPOLL", 7), ("MOE_VRPOLL", 8)] + [ + ("MOE_HALFPLAN", m) for m in range(1, 9) +] + + +@pytest.mark.parametrize("case", ("random", "concentrated")) +@pytest.mark.parametrize("item, m", LEVER_CASES, ids=[f"{i}-m{m}" for i, m in LEVER_CASES]) +def test_each_lever_is_bitwise_its_kill_switch(item, m, case, monkeypatch) -> None: + """The lever's state and the killed state give the same bits on the same layer operands (eager, and the default + state also from a CUDA-graph replay); the default state records the lever where its config carries it, the killed + state never does.""" + on = _state(m) + off = _killed_state(m, item, monkeypatch) + assert item not in off.items + if item == "MOE_M8CFG": + assert item in on.items and on.config["b2img"] == 0 + elif item == "MOE_VRPOLL": + assert item in on.items and on.config["lg_repl"] == moe_op.VRPOLL_LG_REPL + assert off.config.get("lg_repl") != moe_op.VRPOLL_LG_REPL + hidden = hidden_states(m, seed=zlib.crc32(f"{item}-{m}-{case}".encode()) % 2**31) + ref = layer_for(off, case)(hidden) + out = layer_for(on, case)(hidden) + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)), f"{item} at {m} tokens" + layer = layer_for(on, case) + layer(hidden) # compiled before the capture + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = layer(hidden) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(captured.view(torch.int16), ref.view(torch.int16)), ( + f"{item} at {m} tokens (graph)" + ) + + +def test_layers_on_one_state_do_not_leak_into_each_other() -> None: + """Two layers share one state's workspace, in stream order: A, B, A gives A's output twice, bit for bit, and the + caller's ``out`` receives the same rows as a fresh tensor.""" + state = _state(4) + a, b = layer_for(state, "random"), layer_for(state, "concentrated") + hidden = hidden_states(4, seed=7) + first = op(hidden, a) + op(hidden, b) + out = torch.empty(4, H, dtype=torch.bfloat16, device="cuda") + second = op(hidden, a, out=out) + assert second.data_ptr() == out.data_ptr() + assert torch.equal(first.view(torch.int16), out.view(torch.int16)) + + +@pytest.mark.parametrize( + "bad", + ("misaligned hidden", "fp32 out", "short out", "long out", "int32 arp"), +) +def test_malformed_buffers_are_rejected_by_the_call(bad) -> None: + """Why the entry has no is_valid: the compiled call checks each buffer against its signature and raises.""" + state = _state(2) + layer = layer_for(state, "random") + hidden, kw = hidden_states(2, seed=3), {} + if bad == "misaligned hidden": + hidden = torch.randn(2 * H + 8, device="cuda").to(torch.bfloat16)[2 : 2 + 2 * H].view(2, H) + elif bad == "fp32 out": + kw["out"] = torch.empty(2, H, dtype=torch.float32, device="cuda") + elif bad == "short out": + kw["out"] = torch.empty(1, H, dtype=torch.bfloat16, device="cuda") + elif bad == "long out": + kw["out"] = torch.empty(3, H, dtype=torch.bfloat16, device="cuda") + else: + kw["arp"] = torch.zeros(8, dtype=torch.int32, device="cuda") + with pytest.raises(ValueError): + op(hidden, layer, **kw) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_moe_flat_gate_tp2.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_moe_flat_gate_tp2.py new file mode 100644 index 000000000000..6d345006f570 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_moe_flat_gate_tp2.py @@ -0,0 +1,74 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The TP2 MoE engine's FLAT stack (the TP4 engine at the TP2 width) is applied only up to FLAT_MAX_TOKENS (4) tokens. + +At 8 tokens FLAT was +2.27 us/call slower than the per-m tables, so above the cap the engine keeps the per-m +configuration (bitwise the same outputs). CPU only: M3MoeState is constructed with its device +queries stubbed and stopped at the kernel build, so the test reads the engine config and the engagement lines +without a GPU. +""" + +import types + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.m3_moe import ( # noqa: E501 + op, +) + + +class _Built(Exception): + """Raised by the stubbed kernel build: the config is final by then.""" + + +def _engine(monkeypatch, num_tokens): + """(config, info lines) of M3MoeState(num_tokens) up to its kernel build.""" + lines = [] + seen = {} + + def build(config, path): + seen["config"] = dict(config) + raise _Built() + + monkeypatch.setattr(op, "is_supported", lambda device=None: (True, "")) + monkeypatch.setattr(op, "_kernel_module", build) + monkeypatch.setattr(op.torch.cuda, "is_current_stream_capturing", lambda: False) + monkeypatch.setattr( + op.torch.cuda, + "get_device_properties", + lambda device: types.SimpleNamespace(multi_processor_count=op.HIDDEN // 32), + ) + from tensorrt_llm.logger import logger + + monkeypatch.setattr(logger, "info", lambda msg, *a, **k: lines.append(str(msg))) + with pytest.raises(_Built): + op.M3MoeState("cuda", num_tokens, num_ctas=op.HIDDEN // 32) + return seen["config"], [line for line in lines if "MiniMax-M3 MoE engine FLAT (TP2)" in line] + + +def _has_flat(config): + return all(config.get(k) == v for k, v in op.FLAT_STACK.items()) + + +@pytest.mark.parametrize("num_tokens", range(1, 9)) +def test_flat_applies_only_up_to_the_cap(monkeypatch, num_tokens): + config, lines = _engine(monkeypatch, num_tokens) + assert op.FLAT_MAX_TOKENS == 4 + if num_tokens <= 4: + assert _has_flat(config), config + assert lines == [ + "MiniMax-M3 MoE engine FLAT (TP2): b2img=0 lg_repl=16 ar_pre=0 late_trigger=0 " + f"tokens={num_tokens}" + ] + else: + # above the cap: the per-m tables + assert config.get("b2img") == op.B2IMG_BY_M[num_tokens - 1] + assert config.get("ar_pre") == op.AR_PRE_BY_M[num_tokens - 1] + assert lines == [ + f"MiniMax-M3 MoE engine FLAT (TP2) off above 4 tokens (per-m tables): tokens={num_tokens}" + ] + + +def test_flat_applies_helper(): + """op.flat_applies: the FLAT stack up to FLAT_MAX_TOKENS tokens, the per-m tables above.""" + assert [op.flat_applies(m) for m in range(1, 9)] == [True] * 4 + [False] * 4 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_router_tc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_router_tc.py new file mode 100644 index 000000000000..beac41b4cbce --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/moe/test_modeling_v2_m3_router_tc.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_router_tc entry of the MiniMax-M3.1 tp2 target (ROUTER48: the sm_107 TP8 wrapper, cells +2..48 rows): the TF32 tensor-core router of the unfused MoE's generation steps, plain logits and logits + the NVFP4 +activation. Every certified cell runs; the K7 c3 / c4 steps (24 / 32 rows) are also driven explicitly in both forms, +since ROUTER48 is what takes them at TP2 (ROUTER_TC_GEN 9..48).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import true_fp32_matmul + +__extra_import_path__ = ["../../../..", "."] +from _m3_router import SF_SCALE, _build, _compare_fp4, _compare_logits, _seed, tc # noqa: E402 +from _validating import validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + + +def _run(m: int, fp4: bool, seed: int) -> None: + x, w = _build(m, 128, 6144, seed) + sf_scale = torch.tensor([SF_SCALE], dtype=torch.float32, device="cuda") if fp4 else None + with validating(tc): + outs = tc(x, w, sf_scale, False) + ref = tc.reference(x, w, sf_scale, False) + assert len(outs) == len(ref) == (3 if fp4 else 1) + _compare_logits(outs[0], ref[0]) + if fp4: + _compare_fp4(outs[1], outs[2], x, sf_scale) + + +@pytest.mark.parametrize("cell", tc.CELLS, ids=[c.why[:48] for c in tc.CELLS]) +def test_tc_certified_cells(cell) -> None: + _run(cell.spec["m"], cell.spec["fp4"], _seed(cell)) + + +def test_the_cells_certify_every_router48_row() -> None: + """The target's ROUTER_TC_GEN row range (9..48 with ROUTER48) is inside the wrapper's certified cells.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as dispatch, + ) + + assert dispatch.ROUTER48 and dispatch.ROUTER_TC_GEN_MAX_ROWS == 48 + assert dispatch.ROUTER_TC_GEN.keys <= dispatch.ROUTER_TC_GEN.certified() + assert {24, 32} <= dispatch.ROUTER_TC_GEN.keys + + +@pytest.mark.parametrize("fp4", [False, True], ids=["logits", "logits_nvfp4"]) +@pytest.mark.parametrize("m", [24, 32], ids=["c3_24rows", "c4_32rows"]) +def test_tc_k7_c3_c4_rows(m, fp4) -> None: + for seed in range(3): + _run(m, fp4, 7000 + 100 * m + seed) + + +def test_tc_is_the_built_in_gates_router_call() -> None: + """The target's entry is the op the built-in gate calls at these rows, with the same arguments: the same logits + bit for bit.""" + for m in (14, 24, 32): + x, w = _build(m, 128, 6144, 7) + ours = tc(x, w, None, True)[0] + gate = torch.ops.trtllm.minimax_m3_router_tc(x, w, None, True)[0] + assert torch.equal(ours.view(torch.int32), gate.view(torch.int32)), m + + +def test_tc_reference_is_tf32_and_the_gate_sees_it() -> None: + """The tensor-core router rounds the weight to TF32: against a true fp32 product its logits sit far outside + the band, so the TF32 rounding in ``reference`` is load-bearing rather than decoration.""" + x, w = _build(16, 128, 6144, 5) + out = tc(x, w, None, False)[0] + with true_fp32_matmul(): + fp32 = x.float() @ w.t() + with pytest.raises(AssertionError): + _compare_logits(out, fp32) + + +def test_guard_refuses_a_misaligned_activation() -> None: + """A row view starting 8 bytes into the buffer keeps a legal row stride but not a legal bulk-copy source.""" + x, w = _build(9, 128, 6144, 3) + flat = torch.empty(9 * 6144 + 4, dtype=torch.bfloat16, device="cuda") + shifted = flat[4:].view(9, 6144) + shifted.copy_(x) + with pytest.raises(AssertionError, match="16-byte aligned"): + with validating(tc): + tc(shifted, w, None, False) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_dispatch_lines.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_dispatch_lines.py new file mode 100644 index 000000000000..8d3284099cf9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_dispatch_lines.py @@ -0,0 +1,121 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The backend lines of the native SwiGLU-OAI and MoE finalize ops on the tp2 target. + +Each wrapper logs, once per (item, site, outcome) per process, "[M3 dispatch] rank N: NATIVE_SWIGLU / NATIVE_FINALIZE +taken ()" when the native op serves, or "... not taken (): " when the caller's fallback does, so a +serving log shows which backend ran. CPU only: the op is a stand-in. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_report, +) + +pytestmark = pytest.mark.cpu_only + +CORE = SimpleNamespace(swiglu_limit=7.0, swiglu_alpha=1.702, swiglu_beta=1.0) +SERVED = object() + + +@pytest.fixture +def lines(monkeypatch): + """Every line modeling logs, with rank 0, a stand-in native op and fresh once-per-process state.""" + out = [] + monkeypatch.setattr(m3_report, "_info", lambda message: None) + monkeypatch.setattr(m3_report, "_rank", lambda: 0) + monkeypatch.setattr(modeling, "mpi_rank", lambda: 0) + monkeypatch.setattr(modeling, "logger", SimpleNamespace(info=out.append, warning=out.append)) + monkeypatch.setattr(modeling, "_native_op", lambda name: lambda *args: SERVED) + m3_report.reset_work() + modeling.reset_logged() + yield out + m3_report.reset_work() + modeling.reset_logged() + + +def _dispatch(out): + return [line for line in out if "NATIVE_" in line] + + +def _gu(rows=8, d=16): + return torch.zeros(rows, 2 * d, dtype=torch.bfloat16) + + +def _finalize_args(rows=8, shared=False): + fc2 = torch.zeros(rows * 2, 16, dtype=torch.bfloat16) + scales = torch.zeros(rows, 2, dtype=torch.float32) + perm = torch.zeros(rows, 2, dtype=torch.int32) + return ( + rows, + fc2, + scales, + perm, + (torch.zeros(rows, 16, dtype=torch.bfloat16) if shared else None), + ) + + +@pytest.mark.parametrize("dense,site", [(False, "shared"), (True, "dense")]) +def test_swiglu_taken_once_per_site(lines, dense, site): + for _ in range(3): + assert modeling._native_swiglu(CORE, _gu(), dense) is SERVED + assert _dispatch(lines) == [f"[M3 dispatch] rank 0: NATIVE_SWIGLU taken ({site})"] + + +@pytest.mark.parametrize( + "patch,gu,why", + [ + ({"NATIVE_SWIGLU": False}, _gu(), "killed (TRTLLM_M31_DISABLE_NATIVE_SWIGLU=1)"), + ({"_native_op": lambda name: None}, _gu(), "the op is not registered"), + ({}, _gu().float(), "input is not a contiguous 2-D bf16 tensor"), + ({}, _gu().t(), "input is not a contiguous 2-D bf16 tensor"), + ], +) +def test_swiglu_not_taken_names_the_reason(lines, monkeypatch, patch, gu, why): + for name, value in patch.items(): + monkeypatch.setattr(modeling, name, value) + assert modeling._native_swiglu(CORE, gu, False) is None + assert modeling._native_swiglu(CORE, gu, False) is None + assert _dispatch(lines) == [f"[M3 dispatch] rank 0: NATIVE_SWIGLU not taken (shared): {why}"] + + +def test_swiglu_other_constants_are_not_taken(lines): + core = SimpleNamespace(swiglu_limit=10.0, swiglu_alpha=1.702, swiglu_beta=1.0) + assert modeling._native_swiglu(core, _gu(), True) is None + assert _dispatch(lines) == [ + "[M3 dispatch] rank 0: NATIVE_SWIGLU not taken (dense): SwiGLU constants other than (7, 1.702, 1)" + ] + + +@pytest.mark.parametrize("shared,site", [(False, "routed"), (True, "shared add")]) +def test_finalize_taken_once_per_site(lines, shared, site): + for _ in range(3): + assert modeling._native_finalize(*_finalize_args(shared=shared)) is SERVED + assert _dispatch(lines) == [f"[M3 dispatch] rank 0: NATIVE_FINALIZE taken ({site})"] + + +def test_finalize_kill_switch_and_missing_op(lines, monkeypatch): + monkeypatch.setattr(modeling, "NATIVE_FINALIZE", False) + assert modeling._native_finalize(*_finalize_args()) is None + monkeypatch.setattr(modeling, "NATIVE_FINALIZE", True) + monkeypatch.setattr(modeling, "_native_op", lambda name: None) + assert modeling._native_finalize(*_finalize_args()) is None + assert _dispatch(lines) == [ + "[M3 dispatch] rank 0: NATIVE_FINALIZE not taken (routed): killed (TRTLLM_M31_DISABLE_NATIVE_FINALIZE=1)", + "[M3 dispatch] rank 0: NATIVE_FINALIZE not taken (routed): the op is not registered", + ] + + +def test_finalize_layout_is_not_taken(lines): + rows, fc2, scales, perm, shared = _finalize_args() + assert modeling._native_finalize(rows, fc2.float(), scales, perm, shared) is None + assert _dispatch(lines) == [ + "[M3 dispatch] rank 0: NATIVE_FINALIZE not taken (routed): operand layout or alignment" + ] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_finalize.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_finalize.py new file mode 100644 index 000000000000..2247a8339b34 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_finalize.py @@ -0,0 +1,94 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""NATIVE_FINALIZE: trtllm::minimax_m31_moe_finalize is bit for bit the Triton kernels/m3_moe_finalize it replaces: the +routed experts' finalize from the trtllm-gen runner's unfinalized layout (fc2 permuted rows, expanded -> permuted +index with -1 = dropped, bf16 or fp32 weights), with and without the shared expert's add, for every work-split table +entry, at decode and prefill row counts, with denormal products and dropped slots.""" + +import pytest +import torch +from torch._subclasses.fake_tensor import FakeTensorMode + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import m31_fake_ops +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.m3_moe_finalize import ( # noqa: E501 + m3_moe_finalize, +) + +_ops = torch.ops.trtllm +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or not hasattr(_ops, "minimax_m31_moe_finalize"), + reason="needs a CUDA device and the native finalize op", +) +H = 6144 + + +def _parts(tokens, topk, kind, seed, wdtype): + g = torch.Generator(device="cuda").manual_seed(seed) + p = tokens * topk + 128 + fc2 = (torch.randn(p, H, generator=g, device="cuda") * 2).to(torch.bfloat16) + perm = ( + torch.randperm(p, generator=g, device="cuda")[: tokens * topk] + .to(torch.int32) + .view(tokens, topk) + ) + w = torch.rand(tokens, topk, generator=g, device="cuda") / topk + if kind == "dropped": + perm = torch.where(torch.rand(perm.shape, generator=g, device="cuda") < 0.05, -1, perm).to( + torch.int32 + ) + if kind == "denormal": + w = w * 2.0**-126 + shared = (torch.randn(tokens, H, generator=g, device="cuda") * 0.05).to(torch.bfloat16) + return fc2, w.to(wdtype).contiguous(), perm.contiguous(), shared + + +def _bits_equal(a, b): + return torch.equal(a.contiguous().view(torch.int16), b.contiguous().view(torch.int16)) + + +@pytest.mark.parametrize("tokens", [1, 49, 176, 224, 448, 2048, 4096]) +@pytest.mark.parametrize("topk", [4, 8]) +@pytest.mark.parametrize("kind", ["randn", "dropped", "denormal"]) +@pytest.mark.parametrize("wdtype", [torch.bfloat16, torch.float32]) +@pytest.mark.parametrize("with_shared", [False, True]) +def test_every_cfg_is_the_triton_finalize(tokens, topk, kind, wdtype, with_shared): + fc2, w, perm, shared = _parts(tokens, topk, kind, tokens * 31 + topk, wdtype) + sh = shared if with_shared else None + ref = m3_moe_finalize(fc2, w, perm, sh) + for cfg in (-1, 0, 1, 2, 3, 4, 5, 6, 7): + assert _bits_equal(_ops.minimax_m31_moe_finalize(fc2, w, perm, sh, cfg), ref), f"cfg {cfg}" + + +def test_graph_replay(): + fc2, w, perm, shared = _parts(2048, 8, "dropped", 5, torch.bfloat16) + ref = m3_moe_finalize(fc2, w, perm, shared) + st = torch.cuda.Stream() + st.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(st): + _ops.minimax_m31_moe_finalize(fc2, w, perm, shared, 4) + torch.cuda.current_stream().wait_stream(st) + gr = torch.cuda.CUDAGraph() + with torch.cuda.graph(gr, stream=st): + out = _ops.minimax_m31_moe_finalize(fc2, w, perm, shared, 4) + for _ in range(3): + gr.replay() + torch.cuda.synchronize() + assert _bits_equal(out, ref) + + +@pytest.mark.parametrize("with_shared", [False, True]) +def test_fake_impl_matches_the_op(with_shared): + """The registered fake (catalog m31_fake_ops) gives the op's output shape and dtype.""" + m31_fake_ops.register() + fc2, w, perm, shared = _parts(176, 8, "randn", 3, torch.bfloat16) + sh = shared if with_shared else None + real = _ops.minimax_m31_moe_finalize(fc2, w, perm, sh, -1) + with FakeTensorMode(allow_non_fake_inputs=True) as mode: + fake = _ops.minimax_m31_moe_finalize( + mode.from_tensor(fc2), + mode.from_tensor(w), + mode.from_tensor(perm), + None if sh is None else mode.from_tensor(sh), + -1, + ) + assert (tuple(fake.shape), fake.dtype) == (tuple(real.shape), real.dtype) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_swiglu.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_swiglu.py new file mode 100644 index 000000000000..fbffad23d8f7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/native/test_modeling_v2_m31_native_swiglu.py @@ -0,0 +1,99 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""NATIVE_SWIGLU: trtllm::minimax_m31_swiglu_oai_vec is bit for bit the Triton silu_and_mul it replaces (limit 7, alpha +1.702, beta 1) on every bf16 gate pattern x the special up values and the reverse, random cells at the TP2 widths +(shared expert d 1536, dense d 6144) and the pair fallback (d % 8 != 0), for every vector width and with or without +the programmatic launch, eager and replayed from a CUDA graph.""" + +import pytest +import torch +from torch._subclasses.fake_tensor import FakeTensorMode + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.silu_and_mul +from tensorrt_llm._torch._experimental.modeling_v2.catalog.fakes import m31_fake_ops + +_ops = torch.ops.trtllm +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or not hasattr(_ops, "minimax_m31_swiglu_oai_vec"), + reason="needs a CUDA device and the native SwiGLU op", +) +LIMIT, ALPHA, BETA = 7.0, 1.702, 1.0 + + +def _triton(x): + return _ops.silu_and_mul(x, swiglu_limit=LIMIT, swiglu_alpha=ALPHA, swiglu_beta=BETA) + + +def _bits_equal(a, b): + return torch.equal(a.contiguous().view(torch.int16), b.contiguous().view(torch.int16)) + + +def _specials(n, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + vals = [0.0, -0.0, 1.0, -1.0, 6.99, 7.0, 7.01, -7.0, -7.01, 1e-38, -1e-38, 1e-40, 88.0, -88.0, 3e38, -3e38, + float("inf"), float("-inf"), float("nan")] # fmt: skip + rows = [torch.full((n,), v, dtype=torch.bfloat16, device="cuda") for v in vals] + rows += [ + (s * torch.randn(n, generator=g, device="cuda")).to(torch.bfloat16) + for s in (0.05, 1.0, 4.0, 30.0) + ] + rows.append( + torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + .to(torch.int16) + .view(torch.bfloat16) + ) + return rows + + +@pytest.mark.parametrize("nv", [1, 2, 4]) +@pytest.mark.parametrize("pdl", [False, True]) +@pytest.mark.parametrize("gate_side", [True, False]) +def test_every_bf16_pattern(nv, pdl, gate_side): + allp = ( + torch.arange(1 << 16, dtype=torch.int32, device="cuda").to(torch.int16).view(torch.bfloat16) + ) + for j, other in enumerate(_specials(1 << 16, 1)): + g, u = (allp, other) if gate_side else (other, allp) + for rows in (1, 256): + x = torch.cat([g.view(rows, -1), u.view(rows, -1)], 1).contiguous() + assert _bits_equal(_ops.minimax_m31_swiglu_oai_vec(x, pdl, nv), _triton(x)), ( + f"set {j} rows {rows}" + ) + + +@pytest.mark.parametrize("d", [1536, 6144, 1532]) +@pytest.mark.parametrize("rows", [1, 8, 49, 224, 448, 2048, 4096]) +@pytest.mark.parametrize("nv", [1, 2, 4]) +@pytest.mark.parametrize("pdl", [False, True]) +def test_tp2_shapes(d, rows, nv, pdl): + g = torch.Generator(device="cuda").manual_seed(d * 7 + rows) + x = (1.5 * torch.randn(rows, 2 * d, generator=g, device="cuda")).to(torch.bfloat16) + assert _bits_equal(_ops.minimax_m31_swiglu_oai_vec(x, pdl, nv), _triton(x)) + + +def test_graph_replay(): + x = (1.5 * torch.randn(2048, 2 * 1536, device="cuda")).to(torch.bfloat16) + ref = _triton(x) + st = torch.cuda.Stream() + st.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(st): + _ops.minimax_m31_swiglu_oai_vec(x, True, 2) + torch.cuda.current_stream().wait_stream(st) + gr = torch.cuda.CUDAGraph() + with torch.cuda.graph(gr, stream=st): + out = _ops.minimax_m31_swiglu_oai_vec(x, True, 2) + for _ in range(3): + gr.replay() + torch.cuda.synchronize() + assert _bits_equal(out, ref) + + +@pytest.mark.parametrize("rows,d", [(1, 1536), (2048, 6144)]) +def test_fake_impl_matches_the_op(rows, d): + """The registered fake (catalog m31_fake_ops) gives the op's output shape and dtype.""" + m31_fake_ops.register() + x = torch.randn(rows, 2 * d, device="cuda").to(torch.bfloat16) + real = _ops.minimax_m31_swiglu_oai_vec(x, True, 1) + with FakeTensorMode(allow_non_fake_inputs=True) as mode: + fake = _ops.minimax_m31_swiglu_oai_vec(mode.from_tensor(x), True, 1) + assert (tuple(fake.shape), fake.dtype) == (tuple(real.shape), real.dtype) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/norm/test_modeling_v2_m31_router_f32.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/norm/test_modeling_v2_m31_router_f32.py new file mode 100644 index 000000000000..e3873eb0c6e1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/norm/test_modeling_v2_m31_router_f32.py @@ -0,0 +1,94 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for TP2AGG_ROUTER_F32: the MoE boundary's CTXNORMQ _f32 variant writes the norm widened to float32, and the +prefill router reads it instead of casting. Every output must equal the base op's bit for bit, the fp32 copy must be +norm.float() bit for bit, and the router logits must equal m3_fp32_linear's. Skipped when either op is not built.""" + +import types + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as m, +) + +assert torch.cuda.is_available(), "TP2AGG_ROUTER_F32 requires a CUDA device" + +H, E = 6144, 128 +EPS = 1e-6 +FP4_SCALE = 448.0 * 6.0 / 8.0 + +op = m._ctxnormq_op() +f32_op = m._ctxnormq_f32_op() +pytestmark = pytest.mark.skipif( + op is None or f32_op is None, reason="the CTXNORMQ op or its _f32 variant is not built" +) + + +def _operands(rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + inter = ( + torch.randn(rows, H, generator=g, device="cuda") + * torch.pow(2.0, torch.randint(-6, 7, (rows, 1), generator=g, device="cuda").float()) + ).to(torch.bfloat16) + residual = (2 * torch.randn(rows, H, generator=g, device="cuda")).to(torch.bfloat16) + weight = (1.0 + 0.1 * torch.randn(H, generator=g, device="cuda")).to(torch.bfloat16) + router = 0.02 * torch.randn(E, H, generator=g, device="cuda") + return inter, residual, weight, router + + +def _bytes_equal(a, b): + return torch.equal(a.contiguous().view(torch.uint8), b.contiguous().view(torch.uint8)) + + +@pytest.mark.parametrize("rows", [1, 127, 1024, 1160, 4096, 12697]) +@pytest.mark.parametrize("fp4", [False, True], ids=["nofp4", "moe"]) +def test_f32_variant_equals_the_base_op(rows, fp4) -> None: + inter, residual, weight, _ = _operands(rows, 20261031 + rows) + scale = torch.tensor([FP4_SCALE], device="cuda", dtype=torch.float32) if fp4 else None + inter_a, inter_b = inter.clone(), inter.clone() + base = op(inter_a, residual, weight, EPS, True, scale, False) + outs = f32_op(inter_b, residual, weight, EPS, scale, False) + assert _bytes_equal(inter_a, inter_b) + for a, b in zip(base, outs[:5]): + assert _bytes_equal(a, b) + assert outs[5].dtype == torch.float32 + assert torch.equal(outs[5].view(torch.int32), base[0].float().view(torch.int32)) + + +def test_router_reads_the_boundary_f32() -> None: + """_router_logits takes the published fp32 copy for the same tensor (bitwise m3_fp32_linear), once; any other + tensor falls back to the cast.""" + rows = 2048 + inter, residual, weight, router = _operands(rows, 5) + scale = torch.tensor([FP4_SCALE], device="cuda", dtype=torch.float32) + normed, _, _, _, _, normed_f32 = f32_op(inter, residual, weight, EPS, scale, False) + core = types.SimpleNamespace(w={"l3_router": router}) + ref = m.m3_fp32_linear(normed, router) + m._CTXNORMQ_F32.publish(normed, normed_f32, None) + got = m._router_logits(core, 3, normed) + assert torch.equal(got.view(torch.int32), ref.view(torch.int32)) + assert m._CTXNORMQ_F32.pending is None + m._CTXNORMQ_F32.publish(normed, normed_f32, None) + other = normed.clone() + assert torch.equal(m._router_logits(core, 3, other).view(torch.int32), ref.view(torch.int32)) + assert m._CTXNORMQ_F32.pending is None + + +def test_fake_impl_matches_the_op() -> None: + from torch._subclasses.fake_tensor import FakeTensorMode + + inter, residual, weight, _ = _operands(1160, 7) + scale = torch.tensor([FP4_SCALE], device="cuda") + real = f32_op(inter.clone(), residual, weight, EPS, scale, False) + with FakeTensorMode(allow_non_fake_inputs=True) as mode: + fake = f32_op( + mode.from_tensor(inter), + mode.from_tensor(residual), + mode.from_tensor(weight), + EPS, + mode.from_tensor(scale), + False, + ) + assert [(tuple(t.shape), t.dtype) for t in fake] == [(tuple(t.shape), t.dtype) for t in real] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/norm/test_modeling_v2_m3_embedding_firstnorm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/norm/test_modeling_v2_m3_embedding_firstnorm.py new file mode 100644 index 000000000000..e70f460126b9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/norm/test_modeling_v2_m3_embedding_firstnorm.py @@ -0,0 +1,88 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the norm/m3_embedding_firstnorm entry: the decode embedding gather + the first layer's RMSNorm. + +The table is the model's full bf16 [200064, 6144] embedding shape (random values at the checkpoint's scale), eps the +checkpoint's 1e-6. +""" + +import functools +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.norm.m3_embedding_firstnorm import ( # noqa: E501 + VOCAB, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.norm.m3_embedding_firstnorm import ( # noqa: E501 + m3_embedding_firstnorm as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_embedding_firstnorm requires a CUDA device" + +H = 6144 +EPS = 1e-6 + + +@functools.lru_cache(maxsize=None) +def _operands(): + g = torch.Generator(device="cuda").manual_seed(20261013) + table = (0.02 * torch.randn(VOCAB, H, generator=g, device="cuda")).to(torch.bfloat16) + weight = (1.0 + 0.1 * torch.randn(H, generator=g, device="cuda")).to(torch.bfloat16) + return table, weight + + +def _buffers(rows): + return (torch.empty(rows, H, dtype=torch.bfloat16, device="cuda"), + torch.empty(rows, H, dtype=torch.bfloat16, device="cuda")) # fmt: skip + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + rows, id_dtype = cell.spec["rows"], cell.spec["id_dtype"] + table, weight = _operands() + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + ids = torch.randint(0, VOCAB, (rows,), generator=g, device="cuda").to(id_dtype) + raw, out = _buffers(rows) + with validating(op): + normed, residual = op(ids, table, weight, EPS, raw, out) + assert normed.data_ptr() == out.data_ptr() and residual.data_ptr() == raw.data_ptr() + ref_normed, ref_raw = op.reference(ids, table, weight, EPS, raw, out) + # The gather is a copy: exact. + assert torch.equal(raw.view(torch.int16), ref_raw.view(torch.int16)) + # The norm: the kernel's fp32 sum of squares (warp butterflies, then across 4 warps) and fast rsqrt against an + # fp64 reference, then x * rstd * w rounded once to bf16. The fp32 side is ~1e-6 relative off, so the two roundings + # disagree only next to a bf16 rounding boundary: one ulp at most per element, a small fraction in aggregate. + # Measured over these cells (3 seeds each, GB300): at most 0.51 ulp element, 0.026 ulp RMS. + assert_within_ulp(out, ref_normed, element_ulp=1.0, rms_ulp=0.1) + + +def test_cells_cover_the_targets_own_row_bound() -> None: + """The tp2 target ships its own copy of the fused kernel (kernels/embedding_firstnorm, MAX_ROWS 8; the shared + copy's later 32-row bound is not this target's): every row count 1..MAX_ROWS is a cell, none past it.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.embedding_firstnorm import ( # noqa: E501 + op as kernel_op, + ) + + assert kernel_op.MAX_ROWS == 8 + rows = {cell.spec["rows"] for cell in op.CELLS} + assert rows == set(range(1, kernel_op.MAX_ROWS + 1)) + + +def test_ids_outside_the_table_give_zero_rows() -> None: + """Documented, not refused: an id outside [0, V) gathers a zero row whose norm is zero.""" + table, weight = _operands() + ids = torch.tensor([-1, 5, VOCAB], dtype=torch.int64, device="cuda") + raw, out = _buffers(3) + op(ids, table, weight, EPS, raw, out) + assert not raw[0].any() and not raw[2].any() and not out[0].any() and not out[2].any() + assert torch.equal(raw[1], table[5]) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_chain_spec_mc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_chain_spec_mc.py new file mode 100644 index 000000000000..be511325c8ff --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_chain_spec_mc.py @@ -0,0 +1,292 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for REJ_MC: the spec sampler's chain speculative sampling as +``torch.ops.trtllm.minimax_m31_chain_speculative_sampling_mc`` through ``ops/sampling/m31_chain_spec_mc``, installed as +the rejection sampler's override. + +It must equal the stock ``torch.ops.trtllm.chain_speculative_sampling`` bit for bit, tokens and counts (the microbench +generators: realistic / same / far / ties / onehot drafts and targets, K7). The shared hook in +``speculative/interface.rejection_sampling_one_model`` must be stock while no override is set, and fall back to the +stock op whenever the override declines. TRTLLM_M31_DISABLE_REJ_MC=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.sampling import ( # noqa: E501 + m31_chain_spec_mc as entry, +) +from tensorrt_llm._torch.speculative import interface as spec_interface + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2" +) +CASES = ("realistic", "same", "far", "ties", "onehot") +op = entry.m31_chain_spec_mc + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("REJ_MC is built for sm_107 here") + if entry.chain_spec_mc_op() is None: + pytest.skip("minimax_m31_chain_speculative_sampling_mc is not built") + if not hasattr(torch.ops.trtllm, "chain_speculative_sampling"): + pytest.skip("the stock chain_speculative_sampling op is not built") + pytest.importorskip("flashinfer.sampling") + + +def _probs(logits, top_p=0.95): + """softmax then FlashInfer's top-p renorm, as the sampler feeds the rejection step.""" + import flashinfer.sampling + + v = logits.shape[-1] + p = torch.softmax(logits, -1) + renormed = flashinfer.sampling.top_p_renorm_probs(p.view(-1, v).contiguous(), top_p) + return renormed.view(logits.shape).contiguous() + + +def _make(case, batch, k, v, seed): + """The microbenchmark generators.""" + g = torch.Generator(device="cuda").manual_seed(seed) + + def rl(shape, scale=3.0): + return torch.randn(*shape, v, generator=g, device="cuda") * scale + + dl = rl((batch, k)) + if case == "realistic": # independent draft / target + tl = torch.cat([dl, rl((batch, 1))], 1) + rl((batch, k + 1), 1.0) + elif case == "same": # target == draft: all-accept and zero-residual paths + tl = torch.cat([dl, rl((batch, 1))], 1) + elif case == "far": # target peaked elsewhere: early reject + tl = rl((batch, k + 1)) + tl[:, :, :8] += 20 + elif case == "ties": + dl = torch.round(dl * 2) / 2 + tl = torch.round(torch.cat([dl, rl((batch, 1))], 1) * 2) / 2 + elif case == "onehot": + tl = torch.full((batch, k + 1, v), -1e4, device="cuda") + tl[:, :, 1234] = 0 + else: + raise ValueError(case) + dp, tp = _probs(dl), _probs(tl) + ids = torch.multinomial(dp.view(-1, v), 1, generator=g).view(batch, k).to(torch.int32) + seeds = torch.randint(0, 2**31, (batch,), generator=g, device="cuda", dtype=torch.int64) + offsets = torch.randint(0, 2**20, (batch,), generator=g, device="cuda", dtype=torch.int64) + return dp, ids.contiguous(), tp, seeds, offsets + + +def _stock(*args): + return torch.ops.trtllm.chain_speculative_sampling(*args) + + +def _same(got, ref, what): + assert torch.equal(got[0], ref[0]), f"{what}: accepted tokens differ" + assert torch.equal(got[1], ref[1]), f"{what}: token counts differ" + + +@pytest.mark.parametrize("case", CASES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_stock_op(cell, case) -> None: + """At each certified (batch, K, vocab): the override equals the stock op (tokens and counts), and the explicit + entry call at ctas_per_row 1 / 4 / 8 gives the same result again.""" + _gpu() + batch, k, v = cell.spec["batch"], cell.spec["k"], cell.spec["v"] + for seed in range(4): + args = _make(case, batch, k, v, 1000 * seed + batch) + got = entry.chain_spec_mc(*args) + assert got is not None, "an in-contract input must be served" + _same(got, _stock(*args), f"{case} batch={batch} seed={seed}") + for ctas in (1, 4, 8): + if ctas * batch <= entry.sm_count(args[0].device): + _same(op(*args, entry.prebuild(args[0].device), ctas), got, f"ctas_per_row={ctas}") + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager result.""" + _gpu() + args = _make("realistic", 8, 7, 200064, 7) + entry.prebuild(args[0].device) + ref = entry.chain_spec_mc(*args) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.chain_spec_mc(*args) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.chain_spec_mc(*args) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out[0].fill_(-7) + out[1].fill_(-7) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """Non-fp32 probs, non-int32 draft tokens, non-int64 or mis-sized seed / offset, a vocab not divisible by 4, a + misaligned or non-contiguous probs, mismatched shapes, CPU inputs or more requests than MAX_BATCH return None (the + sampler runs the stock op).""" + _gpu() + dp, ids, tp, seeds, offsets = _make("realistic", 2, 3, 32768, 1) + assert entry.chain_spec_mc(dp, ids, tp, seeds, offsets) is not None + assert entry.chain_spec_mc(dp.double(), ids, tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids, tp.half(), seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids.long(), tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids, tp, seeds.int(), offsets) is None + assert entry.chain_spec_mc(dp, ids, tp, seeds[:1], offsets) is None + assert entry.chain_spec_mc(dp, ids, tp, seeds, offsets[:1]) is None + assert entry.chain_spec_mc(dp, ids[:, :2].contiguous(), tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids, tp[:, :3].contiguous(), seeds, offsets) is None + assert entry.chain_spec_mc(dp[..., :-4], ids, tp[..., :-4], seeds, offsets) is None + assert ( + entry.chain_spec_mc( + dp[..., :-2].contiguous(), ids, tp[..., :-2].contiguous(), seeds, offsets + ) + is None + ) + flat = torch.empty(dp.numel() + 1, device="cuda") + assert entry.chain_spec_mc(flat[1:].view(dp.shape), ids, tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp.cpu(), ids.cpu(), tp.cpu(), seeds.cpu(), offsets.cpu()) is None + b = entry.max_batch() + 1 + wide = _make("realistic", b, 1, 32768, 2) + assert entry.chain_spec_mc(*wide) is None + assert entry.ctas_per_row(1, 148) == 16 and entry.ctas_per_row(8, 148) == 16 + assert entry.ctas_per_row(16, 148) == 8 and entry.ctas_per_row(64, 148) == 2 + assert entry.ctas_per_row(200, 148) == 1 + # This device's own SM count keeps the largest grid co-resident. + sms = entry.sm_count(torch.device("cuda")) + assert entry.MAX_BATCH * entry.ctas_per_row(entry.MAX_BATCH, sms) <= sms + + +@pytest.fixture +def _fake_stock_op(monkeypatch): + """Record the feature FlashInfer fallback; retain real torch tensor semantics.""" + stock = (torch.zeros(2, 4, dtype=torch.int32), torch.ones(2, dtype=torch.int32)) + fake_op = mock.Mock(return_value=(stock[0], None, stock[1] - 1)) + fake_fi = types.SimpleNamespace( + __version__="0.6.4", sampling=types.SimpleNamespace(chain_speculative_sampling=fake_op) + ) + monkeypatch.setattr(spec_interface, "flashinfer", fake_fi, raising=False) + monkeypatch.setattr(spec_interface, "IS_FLASHINFER_AVAILABLE", True) + spec_interface.set_chain_speculative_sampling_override(None) + yield fake_op, stock + spec_interface.set_chain_speculative_sampling_override(None) + + +def _cpu_args(): + return ( + torch.rand(2, 3, 8), + torch.zeros(2, 3, dtype=torch.int32), + torch.rand(2, 4, 8), + torch.zeros(2, dtype=torch.int64), + torch.zeros(2, dtype=torch.int64), + ) + + +def test_hook_unset_is_stock(_fake_stock_op) -> None: + """No override: rejection_sampling_one_model makes exactly the stock op call and returns its result.""" + fake_op, stock = _fake_stock_op + args = _cpu_args() + assert spec_interface._CHAIN_SPEC_OVERRIDE is None + result = spec_interface.rejection_sampling_one_model(*args[:3], seed=args[3], offset=args[4]) + assert result[0] is stock[0] + assert torch.equal(result[1], stock[1]) + fake_op.assert_called_once() + assert all(a is b for a, b in zip(fake_op.call_args.args, args[:3])) + assert fake_op.call_args.kwargs["seed"] is args[3] + assert fake_op.call_args.kwargs["offset"] is args[4] + assert fake_op.call_args.kwargs["deterministic"] is True + assert fake_op.call_args.kwargs["generator"] is None + + +def test_hook_override_result_is_used(_fake_stock_op) -> None: + """An override's result replaces the stock op; the stock op does not run.""" + fake_op, _ = _fake_stock_op + mine = (torch.ones(2, 4, dtype=torch.int32), torch.full((2,), 4, dtype=torch.int32)) + override = mock.Mock(return_value=mine) + spec_interface.set_chain_speculative_sampling_override(override) + args = _cpu_args() + assert ( + spec_interface.rejection_sampling_one_model(*args[:3], seed=args[3], offset=args[4]) is mine + ) + override.assert_called_once_with(*args) + fake_op.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_stock_op) -> None: + """An override that returns None leaves the stock op call in place.""" + fake_op, stock = _fake_stock_op + override = mock.Mock(return_value=None) + spec_interface.set_chain_speculative_sampling_override(override) + args = _cpu_args() + result = spec_interface.rejection_sampling_one_model(*args[:3], seed=args[3], offset=args[4]) + assert result[0] is stock[0] + assert torch.equal(result[1], stock[1]) + assert override.call_count == 1 and fake_op.call_count == 1 + + +def test_the_target_installs_the_override() -> None: + """With REJ_MC on (the default) and the op built, the target installs ``chain_spec_mc`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, + ) + + if not modeling.REJ_MC: + pytest.skip("REJ_MC is switched off in this environment") + saved = dict(modeling._REJ_MC_STATE) + modeling._REJ_MC_STATE.clear() + try: + modeling._install_rej_mc() + assert spec_interface._CHAIN_SPEC_OVERRIDE is entry.chain_spec_mc + assert modeling._REJ_MC_STATE["installed"] is True + finally: + spec_interface.set_chain_speculative_sampling_override(None) + modeling._REJ_MC_STATE.clear() + modeling._REJ_MC_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_REJ_MC=1 (read at import): REJ_MC is off and the target leaves the rejection hook unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.speculative import interface as i\n" + "m._install_rej_mc()\n" + "print(m.REJ_MC, i._CHAIN_SPEC_OVERRIDE is None, m._REJ_MC_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_REJ_MC="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" + + +def test_the_wide_kill_switch_restores_the_c8_cap() -> None: + """MAX_BATCH 48 by default; TRTLLM_M31_DISABLE_REJ_MC_WIDE=1 restores the 8-request cap (a 9-request call keeps + the stock op).""" + _gpu() + assert entry.WIDE and entry.max_batch() == entry.MAX_BATCH == 48 + probe = ( + f"import importlib, torch; m = importlib.import_module('{TARGET}.ops.sampling.m31_chain_spec_mc'); " + "d = torch.softmax(torch.randn(9, 1, 32768, device='cuda'), -1); " + "t = torch.softmax(torch.randn(9, 2, 32768, device='cuda'), -1); " + "i = torch.zeros(9, 1, dtype=torch.int32, device='cuda'); " + "s = torch.zeros(9, dtype=torch.int64, device='cuda'); " + "print('CAP', m.max_batch(), m.chain_spec_mc(d, i, t, s, s.clone()) is None)" + ) + for flag, want in (("1", "CAP 8 True"), ("0", "CAP 48 False")): + out = subprocess.run( + [sys.executable, "-c", probe], + env=dict(os.environ, TRTLLM_M31_DISABLE_REJ_MC_WIDE=flag), + capture_output=True, + text=True, + check=True, + ) + assert [ln for ln in out.stdout.splitlines() if ln.startswith("CAP ")][-1] == want diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_rej_slots_default.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_rej_slots_default.py new file mode 100644 index 000000000000..01f43ab13350 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_rej_slots_default.py @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""REJ_SLOTS on the tp2 target: ON by default (exact: the rejection step reads the same draft floats by slot), and the +kill switch TRTLLM_M31_DISABLE_REJ_SLOTS=1, read at import, turns it off.""" + +import os +import subprocess +import sys + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling as dispatch, +) + + +def test_rej_slots_is_on_by_default(): + assert dispatch.REJ_SLOTS + + +def test_kill_switch(): + code = f"from {dispatch.__name__} import REJ_SLOTS\nprint(REJ_SLOTS)" + env = {**os.environ, "TRTLLM_M31_DISABLE_REJ_SLOTS": "1"} + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.split()[-1] == "False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_rej_slots_wide.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_rej_slots_wide.py new file mode 100644 index 000000000000..25c2c7cf7c22 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_rej_slots_wide.py @@ -0,0 +1,42 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Shape-scoped slot-indexed capacity; the gathered sampler retains its measured cap.""" + +import os +import subprocess +import sys + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.sampling import ( # noqa: E501 + m31_chain_spec_mc as entry, +) + + +@pytest.mark.parametrize( + "k,cap", [(0, 48), (1, 48), (2, 48), (3, 112), (4, 48), (6, 48), (7, 56), (8, 48)] +) +def test_slot_capacity_is_scoped(k, cap): + assert entry.SLOTS_WIDE + assert entry.max_slot_batch(k, 200064) == cap + assert entry.max_batch() == 48 + assert entry.WORKSPACE_MAX_BATCH == 112 + + +@pytest.mark.parametrize("vocab", [0, 16384, 200060, 200068, 262144]) +@pytest.mark.parametrize("k", [3, 7]) +def test_other_vocab_keeps_previous_capacity(k, vocab): + assert entry.max_slot_batch(k, vocab) == 48 + + +@pytest.mark.parametrize("disabled", [False, True]) +def test_slot_wide_kill_in_fresh_process(disabled): + env = {**os.environ, "TRTLLM_M31_DISABLE_REJ_SLOTS_WIDE": "1" if disabled else "0"} + code = ( + f"import {entry.__name__} as e; " + "print(e.max_slot_batch(3,200064),e.max_slot_batch(7,200064),e.max_batch())" + ) + result = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert result.stdout.splitlines()[-1] == ("48 48 48" if disabled else "112 56 48") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_softmax_mc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_softmax_mc.py new file mode 100644 index 000000000000..6b97cdb418e8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_softmax_mc.py @@ -0,0 +1,344 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for SOFTMAX_MC: the spec sampler's softmax as +``torch.ops.trtllm.minimax_m31_softmax_mc`` through ``ops/sampling/m31_softmax_mc``, installed as the sampler's softmax +override. + +It must equal ``flashinfer.sampling.softmax(logits, temperature)`` bit for bit (the microbenchmark generators: randn / +peaked / ties / near_uniform / all_equal / masked / wide rows, temperature 1 / per-row mixed / scalar). The shared +sampler hook must be byte-for-byte stock while no override is set, and fall back to the stock call whenever the +override declines. TRTLLM_M31_DISABLE_SOFTMAX_MC=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.sampling import ( # noqa: E501 + m31_softmax_mc as entry, +) +from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as fi_ops + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2" +) +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal", "masked", "wide") +TEMPS = ("t1", "mixed", "scalar") +op = entry.m31_softmax_mc + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("SOFTMAX_MC is built for sm_107 here") + if entry.softmax_mc_op() is None: + pytest.skip("minimax_m31_softmax_mc is not built") + pytest.importorskip("flashinfer.sampling") + + +def _make(cls, rows, d, seed): + """The microbenchmark logit generators.""" + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + return (torch.randn(rows, d, generator=g, device="cuda") * 3).contiguous() + if cls == "peaked": # few dominant tokens, long tail (real-like rows) + x = torch.randn(rows, d, generator=g, device="cuda") * 2 + idx = torch.randint(0, d, (rows, 8), generator=g, device="cuda") + x.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + return x.contiguous() + if cls == "ties": + return (torch.round(torch.randn(rows, d, generator=g, device="cuda") * 2) / 2).contiguous() + if cls == "near_uniform": + return (torch.randn(rows, d, generator=g, device="cuda") * 1e-3).contiguous() + if cls == "all_equal": + return torch.zeros(rows, d, device="cuda") + if cls == "masked": + x = torch.randn(rows, d, generator=g, device="cuda") * 3 + x[:, d // 2 :] = -1e4 + return x.contiguous() + if cls == "wide": + return (torch.randn(rows, d, generator=g, device="cuda") * 30).contiguous() + raise ValueError(cls) + + +def _temperature(mode, rows, seed): + """FlashInfer's temperature argument: an fp32 [rows] tensor, or a Python float.""" + g = torch.Generator(device="cuda").manual_seed(seed + 3) + if mode == "t1": + return torch.ones(rows, device="cuda") + if mode == "mixed": + return (0.3 + 1.2 * torch.rand(rows, generator=g, device="cuda")).contiguous() + if mode == "scalar": + return 0.7 + raise ValueError(mode) + + +def _fi(logits, temperature): + import flashinfer.sampling + + return flashinfer.sampling.softmax(logits, temperature) + + +def _same(a, b, what): + """Bit for bit.""" + assert torch.equal(a.view(torch.int32), b.view(torch.int32)), f"{what}: fp32 bits differ" + + +@pytest.mark.parametrize("mode", TEMPS) +@pytest.mark.parametrize("cls", CLASSES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_flashinfer_softmax(cell, cls, mode) -> None: + """At each certified (rows, d): the override equals FlashInfer's softmax bit for bit, and the explicit entry call + at write_ctas 4 / 8 gives the same bits again.""" + _gpu() + rows, d = cell.spec["rows"], cell.spec["d"] + for seed in range(2): + logits = _make(cls, rows, d, 100 * seed + rows) + temperature = _temperature(mode, rows, seed) + got = entry.softmax_mc(logits, temperature) + assert got is not None, "an in-contract input must be served" + _same(got, _fi(logits, temperature), f"{cls} {mode} rows={rows} seed={seed}") + for write_ctas in (4, 8): + again = op(logits, temperature, entry.prebuild(logits.device), write_ctas) + _same(again, got, f"write_ctas={write_ctas}") + + +def test_no_temperature_is_one() -> None: + """A None temperature (softmax_op's Optional) is FlashInfer's default, 1.0.""" + _gpu() + logits = _make("peaked", 8, 200064, 11) + got = entry.softmax_mc(logits, None) + assert got is not None + _same(got, _fi(logits, None), "temperature None") + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager bits.""" + _gpu() + logits = _make("peaked", 8, 200064, 5) + temperature = _temperature("mixed", 8, 5) + entry.prebuild(logits.device) + ref = entry.softmax_mc(logits, temperature) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.softmax_mc(logits, temperature) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.softmax_mc(logits, temperature) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """A non-fp32 / misaligned / non-contiguous logits, a d % 4 width, a vocab below FlashInfer's Path A, a mis-sized, + non-fp32 or 2-D temperature, a non-number temperature or more rows than MAX_ROWS return None (the sampler runs + stock FlashInfer).""" + _gpu() + d = 200064 + logits = _make("randn", 4, d, 1) + temperature = torch.full((4,), 0.9, device="cuda") + assert entry.softmax_mc(logits.half(), temperature) is None + assert entry.softmax_mc(logits, temperature.double()) is None + assert entry.softmax_mc(logits, temperature[:3]) is None + assert entry.softmax_mc(logits, temperature.view(4, 1)) is None + assert entry.softmax_mc(logits, "0.9") is None + assert entry.softmax_mc(logits[:, : d - 4], temperature) is None + assert entry.softmax_mc(logits[:, : d - 2].contiguous(), temperature) is None + assert ( + entry.softmax_mc(logits[:, : entry.PATH_A_MIN_VOCAB - 4].contiguous(), temperature) is None + ) + flat = torch.empty(4 * d + 1, device="cuda") + assert entry.softmax_mc(flat[1:].view(4, d), temperature) is None + wide = _make("randn", 449, d, 2) + assert entry.softmax_mc(wide, torch.full((449,), 0.9, device="cuda")) is None + assert entry.softmax_mc(logits.cpu(), temperature.cpu()) is None + + +@pytest.fixture +def _fake_flashinfer(monkeypatch): + """The sampler ops module with a stand-in flashinfer whose calls are recorded, and no override afterwards.""" + stock = torch.full((2, 8), 0.125) + sampling = types.SimpleNamespace( + softmax=mock.Mock(return_value=stock), + top_p_renorm_probs=mock.Mock(side_effect=lambda probs, top_p: probs), + top_k_mask_logits=mock.Mock(side_effect=lambda logits, top_k: logits), + ) + monkeypatch.setattr(fi_ops, "flashinfer", types.SimpleNamespace(sampling=sampling)) + fi_ops.set_softmax_override(None) + fi_ops.set_top_p_renorm_override(None) + yield sampling, stock + fi_ops.set_softmax_override(None) + + +def test_hook_unset_is_stock(_fake_flashinfer) -> None: + """No override: both ops make exactly the stock flashinfer call and return its result.""" + sampling, stock = _fake_flashinfer + logits, temperature = torch.rand(2, 8), torch.ones(2) + assert fi_ops._SOFTMAX_OVERRIDE is None + assert fi_ops.softmax_op(logits, temperature) is stock + sampling.softmax.assert_called_once_with(logits, temperature, enable_pdl=mock.ANY) + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, None) is stock + assert sampling.softmax.call_count == 2 + + +def test_hook_override_result_is_used(_fake_flashinfer) -> None: + """An override's tensor replaces the stock call in both ops; the stock kernel does not run, and a top_p renorm + after it gets the override's probs.""" + sampling, _ = _fake_flashinfer + mine = torch.zeros(2, 8) + override = mock.Mock(return_value=mine) + fi_ops.set_softmax_override(override) + logits, temperature = torch.rand(2, 8), torch.ones(2) + assert fi_ops.softmax_op(logits, temperature) is mine + override.assert_called_once_with(logits, temperature) + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, None) is mine + top_p = torch.full((2,), 0.9) + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, top_p) is mine + sampling.top_p_renorm_probs.assert_called_once_with(mine, top_p) + assert override.call_count == 3 + sampling.softmax.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_flashinfer) -> None: + """An override that returns None leaves the stock call in place in both ops.""" + sampling, stock = _fake_flashinfer + override = mock.Mock(return_value=None) + fi_ops.set_softmax_override(override) + logits, temperature = torch.rand(2, 8), torch.ones(2) + assert fi_ops.softmax_op(logits, temperature) is stock + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, None) is stock + assert override.call_count == 2 and sampling.softmax.call_count == 2 + + +def test_the_target_installs_the_override() -> None: + """With SOFTMAX_MC on (the default) and the op built, the target installs ``softmax_mc`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, + ) + + if not modeling.SOFTMAX_MC: + pytest.skip("SOFTMAX_MC is switched off in this environment") + saved = dict(modeling._SOFTMAX_MC_STATE) + modeling._SOFTMAX_MC_STATE.clear() + try: + modeling._install_softmax_mc() + assert fi_ops._SOFTMAX_OVERRIDE is entry.softmax_mc + assert modeling._SOFTMAX_MC_STATE["installed"] is True + finally: + fi_ops.set_softmax_override(None) + modeling._SOFTMAX_MC_STATE.clear() + modeling._SOFTMAX_MC_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_SOFTMAX_MC=1 (read at import): SOFTMAX_MC is off and the target leaves the sampler hook + unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as f\n" + "m._install_softmax_mc()\n" + "print(m.SOFTMAX_MC, f._SOFTMAX_OVERRIDE is None, m._SOFTMAX_MC_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_SOFTMAX_MC="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" + + +def test_write_ctas_follow_the_rows() -> None: + """16 write CTAs up to 12 rows (every c1 call unchanged), 8 up to 24, 4 up to MAX_ROWS (128).""" + assert entry.WIDE and entry.max_rows() == entry.MAX_ROWS == 128 + for rows, ctas in ( + (1, 16), + (8, 16), + (12, 16), + (13, 8), + (24, 8), + (25, 4), + (48, 4), + (96, 4), + (112, 4), + (113, 4), + (128, 4), + ): + assert entry.write_ctas(rows) == ctas, rows + + +@pytest.mark.parametrize("rows", [9, 13, 24, 25, 56, 96, 97, 112, 113, 127, 128]) +def test_wide_rows_bitwise_flashinfer(rows) -> None: + """Between and at the table's edges (the K7 draft's 7c rows included): the override equals FlashInfer's softmax + bit for bit, with every temperature mode.""" + _gpu() + for mode in TEMPS: + for cls in ("randn", "peaked", "masked"): + logits = _make(cls, rows, 200064, 7 * rows + len(cls)) + temperature = _temperature(mode, rows, rows) + got = entry.softmax_mc(logits, temperature) + assert got is not None, "a wide in-contract input must be served" + _same(got, _fi(logits, temperature), f"{cls} {mode} rows={rows}") + + +@pytest.mark.parametrize("rows", [48, 113, 128]) +def test_wide_graph_replay_equals_eager(rows) -> None: + """Wide rows replay the eager bits with fresh logits copied in, including the Path-A cap.""" + _gpu() + logits = _make("peaked", rows, 200064, 8).clone() + temperature = _temperature("mixed", rows, 8) + entry.prebuild(logits.device) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.softmax_mc(logits, temperature) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.softmax_mc(logits, temperature) + assert out is not None + for k, cls in enumerate(("randn", "near_uniform")): + logits.copy_(_make(cls, rows, 200064, 60 + k)) + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + _same(out, _fi(logits, temperature), f"graph replay {cls}") + + +def test_the_wide_kill_switch_restores_the_c1_form() -> None: + """The wide kill restores eight rows; the 128-row item kill restores 112. Log each cap once.""" + _gpu() + probe = ( + f"import importlib, torch; m = importlib.import_module('{TARGET}.ops.sampling.m31_softmax_mc'); " + "m.softmax_mc_op(); x = torch.randn(9, 200064, device='cuda'); " + "y = torch.randn(128, 200064, device='cuda'); " + "print('CAP', m.max_rows(), m.write_ctas(40), m.softmax_mc(x, None) is None, " + "m.softmax_mc(y, None) is None)" + ) + for flag, cap_flag, want, line in ( + ("1", "0", "CAP 8 16 True True", "<= 8, write CTAs 16"), + ("0", "0", "CAP 128 4 False False", "<= 128,"), + ("0", "1", "CAP 112 4 False True", "<= 112,"), + ): + out = subprocess.run( + [sys.executable, "-c", probe], + env=dict( + os.environ, + TLLM_LOG_LEVEL="INFO", + TRTLLM_M31_DISABLE_SOFTMAX_MC_WIDE=flag, + TRTLLM_M31_DISABLE_SOFTMAX_MC_128=cap_flag, + ), + capture_output=True, + text=True, + check=True, + ) + assert [ln for ln in out.stdout.splitlines() if ln.startswith("CAP ")][-1] == want + lines = [ln for ln in (out.stdout + out.stderr).splitlines() if "SOFTMAX_MC rows:" in ln] + assert len(lines) == 1 and line in lines[0], lines diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_softmax_pathb.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_softmax_pathb.py new file mode 100644 index 000000000000..e9783fb47614 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_softmax_pathb.py @@ -0,0 +1,144 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SM107 QA gates for the Path-B catalog, routing and capture lifecycle.""" + +import importlib +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.sampling import ( # noqa: E501 + m31_softmax_mc as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.sampling import ( # noqa: E501 + m31_softmax_pathb as entry, +) + + +def _gpu() -> None: + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("SM107 required") + assert entry.softmax_pathb_op() is not None, "Path-B native library must be installed" + + +def _same(actual: torch.Tensor, expected: torch.Tensor) -> None: + assert torch.isfinite(actual).all() + assert torch.equal(actual.view(torch.int32), expected.view(torch.int32)) + + +@pytest.mark.parametrize( + "cell", entry.m31_softmax_pathb.CELLS, ids=lambda cell: str(cell.spec["rows"]) +) +def test_every_integer_row(cell) -> None: + """Each catalog cell runs through the actual sampler override and stock oracle.""" + _gpu() + rows = cell.spec["rows"] + g = torch.Generator(device="cuda").manual_seed(1700 + rows) + logits = torch.randn(rows, cell.spec["d"], device="cuda", generator=g) + temperature = torch.linspace(0.1, 2.0, rows, device="cuda") + with mock.patch.object(dispatch, "m31_softmax_mc", side_effect=AssertionError("Path A called")): + actual = dispatch.softmax_mc(logits, temperature) + assert actual is not None + _same(actual, entry.m31_softmax_pathb.reference(logits, temperature)) + + +@pytest.mark.parametrize("rows", [129, 224, 448]) +def test_graph_changed_inputs_without_loading(rows) -> None: + """A preloaded operation captures without loading libraries or building a workspace.""" + _gpu() + logits = torch.randn(rows, entry.VOCAB, device="cuda") + temperature = torch.ones(rows, device="cuda") + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + dispatch.softmax_mc(logits, temperature) + torch.cuda.current_stream().wait_stream(stream) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with ( + mock.patch.object( + torch.ops, "load_library", side_effect=AssertionError("load inside capture") + ), + mock.patch.object( + dispatch, "prebuild", side_effect=AssertionError("Path-A workspace requested") + ), + ): + with torch.cuda.graph(graph, stream=stream): + actual = dispatch.softmax_mc(logits, temperature) + assert actual is not None + for seed in (17, 83): + g = torch.Generator(device="cuda").manual_seed(seed) + logits.normal_(generator=g) + temperature.uniform_(0.1, 3.0, generator=g) + actual.fill_(float("nan")) + graph.replay() + _same(actual, entry.m31_softmax_pathb.reference(logits, temperature)) + + +def test_capture_first_use_does_not_load_or_poison_state(monkeypatch) -> None: + """An unprepared captured call declines; a later uncaptured load remains possible.""" + monkeypatch.setattr(entry, "_STATE", {}) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: True) + with mock.patch.object(torch.ops, "load_library", side_effect=AssertionError("capture load")): + assert entry.softmax_pathb_op() is None + assert entry._STATE == {} + + +@pytest.mark.parametrize( + "kill", ["SOFTMAX_PATHB", "SOFTMAX_MC_WIDE", "SOFTMAX_MC", "SOFTMAX_MC_128"] +) +def test_import_time_kill_hierarchy(kill, monkeypatch) -> None: + _gpu() + logits = torch.zeros(129, entry.VOCAB, device="cuda") + switches = importlib.import_module( + entry.__package__.rsplit(".ops.sampling", 1)[0] + ".m31_switches" + ) + try: + with monkeypatch.context() as patch: + patch.setenv("TRTLLM_M31_DISABLE_" + kill, "1") + importlib.reload(switches) + importlib.reload(entry) + assert entry.ENABLED == (kill == "SOFTMAX_MC_128") + assert (dispatch.softmax_mc(logits, None) is not None) == (kill == "SOFTMAX_MC_128") + finally: + importlib.reload(switches) + importlib.reload(entry) + + +def test_invalid_inputs_and_native_alias_guards() -> None: + _gpu() + x = torch.zeros(129, entry.VOCAB, device="cuda") + op = entry.m31_softmax_pathb + assert op(x[:128], None) is None + assert op(torch.empty(449, entry.VOCAB, device="cuda"), None) is None + assert op(x.half(), None) is None + assert op(x[:, :-4], None) is None + assert op(x[:, :-4].contiguous(), None) is None + flat = torch.empty(x.numel() + 1, device="cuda") + assert op(flat[1:].view_as(x), None) is None + for temp in ( + True, + "1.0", + torch.ones(128, device="cuda"), + torch.ones(129), + torch.ones(129, device="cuda", dtype=torch.float64), + ): + assert op(x, temp) is None + native = entry.softmax_pathb_op() + with pytest.raises(RuntimeError, match="not overlap"): + native(x, None, 1.0, x) + out = torch.empty_like(x) + with pytest.raises(RuntimeError, match="nonoverlapping"): + native(x, out.view(-1)[:129], 0.0, out) + + +def test_native_fake_schema() -> None: + _gpu() + x = torch.randn(129, entry.VOCAB, device="cuda") + out = torch.empty_like(x) + torch.library.opcheck( + entry.softmax_pathb_op(), + (x, None, 1.0, out), + test_utils=("test_schema", "test_faketensor"), + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_topp_det.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_topp_det.py new file mode 100644 index 000000000000..a4e675a67cc0 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_topp_det.py @@ -0,0 +1,273 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for TOPP_DET: the spec sampler's top-p renorm as +``torch.ops.trtllm.minimax_m31_topp_renorm_det`` through ``ops/sampling/m31_topp_det``, installed as the sampler's +top-p renorm override. + +On rows with top_p < 1 it must equal FlashInfer's ``top_p_renorm_probs(probs, top_p, is_deterministic=True)`` bit for +bit (the microbenchmark generators: randn / peaked / ties / near_uniform / all_equal rows); rows with top_p >= 1 +keep every token (a deliberate deviation: FlashInfer drops tail buckets there in ~43% of rows). The shared sampler +hook must be byte-for-byte stock while no override is set, and fall back to the stock call whenever the override +declines. TRTLLM_M31_DISABLE_TOPP_DET=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.sampling import ( # noqa: E501 + m31_topp_det as entry, +) +from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as fi_ops + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2" +) +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal") +FLT_MIN = torch.finfo(torch.float32).tiny +op = entry.m31_topp_det + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("TOPP_DET is built for sm_107 here") + if entry.topp_det_op() is None: + pytest.skip("minimax_m31_topp_renorm_det is not built") + pytest.importorskip("flashinfer.sampling") + + +def _make(cls, rows, d, seed): + """The microbenchmark row generators: softmax of the class's logits.""" + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + logits = torch.randn(rows, d, generator=g, device="cuda") * 3 + elif cls == "peaked": # few dominant tokens, long tail (real-like draft rows) + logits = torch.randn(rows, d, generator=g, device="cuda") * 2 + idx = torch.randint(0, d, (rows, 8), generator=g, device="cuda") + logits.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + elif cls == "ties": + logits = torch.round(torch.randn(rows, d, generator=g, device="cuda") * 2) / 2 + elif cls == "near_uniform": + logits = torch.randn(rows, d, generator=g, device="cuda") * 1e-3 + elif cls == "all_equal": + logits = torch.zeros(rows, d, device="cuda") + else: + raise ValueError(cls) + return torch.softmax(logits, -1).contiguous() + + +def _top_p(mode, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed + 7) + if mode == "p95": + return torch.full((rows,), 0.95, device="cuda") + if mode == "mixed": + return (0.05 + 0.95 * torch.rand(rows, generator=g, device="cuda")).contiguous() + raise ValueError(mode) + + +def _fi_det(probs, top_p): + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + +def _same(a, b, what): + """Bit for bit.""" + assert torch.equal(a.view(torch.int32), b.view(torch.int32)), f"{what}: fp32 bits differ" + + +@pytest.mark.parametrize("mode", ["p95", "mixed"]) +@pytest.mark.parametrize("cls", CLASSES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_flashinfer_det_below_one(cell, cls, mode) -> None: + """Rows with top_p < 1 at each certified (rows, d): the override equals FlashInfer's det renorm bit for bit, and + the explicit entry call gives the same bits again (deterministic).""" + _gpu() + rows, d = cell.spec["rows"], cell.spec["d"] + for seed in range(2): + probs = _make(cls, rows, d, 1000 * rows + seed) + top_p = _top_p(mode, rows, seed) + got = entry.top_p_renorm_det(probs, top_p) + assert got is not None, "an in-contract input must be served" + ref = _fi_det(probs, top_p) + _same(got, ref, f"{cls} {mode} rows={rows} seed={seed}") + again = op( + probs, + top_p, + entry.prebuild(probs.device), + entry.ctas_per_row(rows, entry.sm_count(probs.device)), + ) + _same(again, got, "repeat") + + +@pytest.mark.parametrize("cls", CLASSES) +def test_keep_all_rows_in_a_mixed_batch(cls) -> None: + """A batch of 8 rows, a third at top_p 1: the top_p 1 rows keep every element whose renormed value is a normal float + (output nonzero <=> ftz(p * norm) != 0, the --use_fast_math semantics FlashInfer's apply shares) and sum to 1 within + 1e-5; the top_p < 1 rows are FlashInfer's det renorm bit for bit.""" + _gpu() + rows, d = 8, 200064 + probs = _make(cls, rows, d, 77) + top_p = _top_p("mixed", rows, 3) + keep = torch.zeros(rows, dtype=torch.bool, device="cuda") + keep[::3] = True + top_p[keep] = 1.0 + got = entry.top_p_renorm_det(probs, top_p) + assert got is not None + ref = _fi_det(probs, top_p) + _same(got[~keep], ref[~keep], f"{cls}: top_p < 1 rows") + for r in keep.nonzero().flatten().tolist(): + p64 = probs[r].double() + norm = 1.0 / float(p64[probs[r] >= FLT_MIN].sum()) + # kept <=> ftz(p * norm) != 0; elements within 1e-6 of the FLT_MIN boundary are left to the rounding + live = p64 * norm >= FLT_MIN * (1 + 1e-6) + dead = probs[r] < FLT_MIN + assert bool((got[r][live] != 0).all()), f"{cls} row {r}: a kept token was dropped" + assert bool((got[r][dead] == 0).all()), f"{cls} row {r}: a sub-FLT_MIN token is nonzero" + total = float(got[r].double().sum()) + assert abs(total - 1.0) <= 1e-5, f"{cls} row {r}: sum {total}" + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager bits.""" + _gpu() + probs = _make("peaked", 8, 200064, 5) + top_p = _top_p("mixed", 8, 5) + entry.prebuild(probs.device) + ref = entry.top_p_renorm_det(probs, top_p) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.top_p_renorm_det(probs, top_p) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.top_p_renorm_det(probs, top_p) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """A float top_p, a non-fp32 / misaligned / non-contiguous probs, a d % 4 width, a mis-sized top_p or more rows + than the workspace holds return None (the sampler runs stock FlashInfer).""" + _gpu() + d = 200064 + probs = _make("randn", 4, d, 1) + top_p = torch.full((4,), 0.9, device="cuda") + assert entry.top_p_renorm_det(probs, 0.9) is None + assert entry.top_p_renorm_det(probs.half(), top_p) is None + assert entry.top_p_renorm_det(probs, top_p.double()) is None + assert entry.top_p_renorm_det(probs, top_p[:3]) is None + assert entry.top_p_renorm_det(probs[:, : d - 2], top_p) is None + assert entry.top_p_renorm_det(probs[:, : d - 2].contiguous(), top_p) is None + flat = torch.empty(4 * d + 1, device="cuda") + assert entry.top_p_renorm_det(flat[1:].view(4, d), top_p) is None + wide = _make("randn", entry.max_rows() + 1, d, 2) + assert ( + entry.top_p_renorm_det(wide, torch.full((entry.max_rows() + 1,), 0.9, device="cuda")) + is None + ) + assert entry.ctas_per_row(1, 148) == 16 and entry.ctas_per_row(8, 148) == 16 + assert entry.ctas_per_row(16, 148) == 8 and entry.ctas_per_row(64, 148) == 2 + assert entry.ctas_per_row(200, 148) == 1 + + +@pytest.fixture +def _fake_flashinfer(monkeypatch): + """The sampler ops module with a stand-in flashinfer whose calls are recorded, and no override afterwards.""" + stock = torch.full((2, 8), 0.125) + sampling = types.SimpleNamespace( + top_p_renorm_probs=mock.Mock(return_value=stock), + softmax=mock.Mock(side_effect=lambda logits, temps, enable_pdl=False: logits.softmax(-1)), + top_k_mask_logits=mock.Mock(side_effect=lambda logits, top_k: logits), + ) + monkeypatch.setattr(fi_ops, "flashinfer", types.SimpleNamespace(sampling=sampling)) + fi_ops.set_top_p_renorm_override(None) + yield sampling, stock + fi_ops.set_top_p_renorm_override(None) + + +def test_hook_unset_is_stock(_fake_flashinfer) -> None: + """No override: both ops make exactly the stock flashinfer call and return its result.""" + sampling, stock = _fake_flashinfer + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops._TOP_P_RENORM_OVERRIDE is None + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is stock + sampling.top_p_renorm_probs.assert_called_once_with(probs, top_p) + assert ( + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is stock + ) + assert sampling.top_p_renorm_probs.call_count == 2 + + +def test_hook_override_result_is_used(_fake_flashinfer) -> None: + """An override's tensor replaces the stock call in both ops; the stock kernel does not run.""" + sampling, _ = _fake_flashinfer + mine = torch.zeros(2, 8) + override = mock.Mock(return_value=mine) + fi_ops.set_top_p_renorm_override(override) + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is mine + override.assert_called_once_with(probs, top_p) + assert fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is mine + assert override.call_count == 2 + sampling.top_p_renorm_probs.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_flashinfer) -> None: + """An override that returns None leaves the stock call in place; without top_p the override is never asked.""" + sampling, stock = _fake_flashinfer + override = mock.Mock(return_value=None) + fi_ops.set_top_p_renorm_override(override) + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is stock + assert ( + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is stock + ) + assert override.call_count == 2 and sampling.top_p_renorm_probs.call_count == 2 + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, None) + assert override.call_count == 2 + + +def test_the_target_installs_the_override() -> None: + """With TOPP_DET on (the default) and the op built, the target installs ``top_p_renorm_det`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, + ) + + if not modeling.TOPP_DET: + pytest.skip("TOPP_DET is switched off in this environment") + saved = dict(modeling._TOPP_DET_STATE) + modeling._TOPP_DET_STATE.clear() + try: + modeling._install_topp_det() + assert fi_ops._TOP_P_RENORM_OVERRIDE is entry.top_p_renorm_det + assert modeling._TOPP_DET_STATE["installed"] is True + finally: + fi_ops.set_top_p_renorm_override(None) + modeling._TOPP_DET_STATE.clear() + modeling._TOPP_DET_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_TOPP_DET=1 (read at import): TOPP_DET is off and the target leaves the sampler hook unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as f\n" + "m._install_topp_det()\n" + "print(m.TOPP_DET, f._TOP_P_RENORM_OVERRIDE is None, m._TOPP_DET_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_TOPP_DET="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_topp_det_wide.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_topp_det_wide.py new file mode 100644 index 000000000000..303c4631baaa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/sampling/test_modeling_v2_m31_topp_det_wide.py @@ -0,0 +1,236 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for TOPP_DET at the TP2 WIDE row counts (9..448 rows, ``m31_topp_det.MAX_ROWS``): V4K8 under every switch +combination equals v1.2 (``minimax_m31_topp_renorm_det``) bit for bit and FlashInfer's det renorm on the top_p < 1 +rows, at 2 / 1 CTAs per row, on one workspace alternating narrow and wide calls and in CUDA-graph replays. The entry +takes up to 448 rows with v3 and keeps v1.2 at 8 rows (TRTLLM_M31_DISABLE_TOPPDET_V3=1), and logs the cap it uses. +""" + +import os +import subprocess +import sys + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.sampling import ( # noqa: E501 + m31_topp_det as entry, +) + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2" +) +D = 200064 +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal", "one_hot", "tiny_tail") +COMBOS = [(True, True), (True, False), (False, True), (False, False)] +WIDE = (9, 16, 48, 64, 128, 192, 256, 448) + + +def _ops(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("this TOPP_DET target is sm_107") + v12 = entry._load_op("minimax_m31_topp_renorm_det") + v3 = entry._load_op("minimax_m31_topp_renorm_det_v3") + if v12 is None or v3 is None or "list_path" not in str(v3.default._schema): + pytest.skip("needs minimax_m31_topp_renorm_det and a V4K8 minimax_m31_topp_renorm_det_v3") + pytest.importorskip("flashinfer.sampling") + return v12, v3 + + +def _make(cls, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + logits = torch.randn(rows, D, generator=g, device="cuda") * 3 + elif cls == "peaked": + logits = torch.randn(rows, D, generator=g, device="cuda") * 2 + idx = torch.randint(0, D, (rows, 8), generator=g, device="cuda") + logits.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + elif cls == "ties": + logits = torch.round(torch.randn(rows, D, generator=g, device="cuda") * 2) / 2 + elif cls == "near_uniform": + logits = torch.randn(rows, D, generator=g, device="cuda") * 1e-3 + elif cls == "all_equal": + logits = torch.zeros(rows, D, device="cuda") + elif cls == "one_hot": + logits = torch.full((rows, D), -1e4, device="cuda") + logits[torch.arange(rows), torch.randint(0, D, (rows,), generator=g, device="cuda")] = 0 + elif cls == "tiny_tail": + logits = torch.randn(rows, D, generator=g, device="cuda") * 30 + else: + raise ValueError(cls) + return torch.softmax(logits, -1).contiguous() + + +def _top_p(mode, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed + 7) + if mode == "p95": + return torch.full((rows,), 0.95, device="cuda") + if mode == "mixed": + return (0.05 + 0.95 * torch.rand(rows, generator=g, device="cuda")).contiguous() + tp = torch.full((rows,), 0.95, device="cuda") + tp[seed % 2 :: 2] = 1.0 # with_p1: keep-all rows interleaved + return tp + + +class _Runner: + """v1.2 and V4K8 on separate zeroed workspaces sized for MAX_ROWS rows.""" + + def __init__(self, v12, v3): + self.v12, self.v3 = v12, v3 + n12 = int(torch.ops.trtllm.minimax_m31_topp_renorm_det_ws_bytes()) + n3 = int(torch.ops.trtllm.minimax_m31_topp_renorm_det_v3_ws_bytes()) + self.ws12 = torch.zeros(entry.MAX_ROWS * n12, dtype=torch.uint8, device="cuda") + self.ws3 = torch.zeros(entry.MAX_ROWS * n3, dtype=torch.uint8, device="cuda") + self.sms = torch.cuda.get_device_properties(0).multi_processor_count + + def g(self, rows): + return entry.ctas_per_row(rows, self.sms) + + def ref(self, probs, tp): + out = torch.empty_like(probs) + self.v12(probs, tp, self.ws12, out, self.g(probs.shape[0])) + return out + + def new(self, probs, tp, pdl, combo, out=None): + out = torch.empty_like(probs) if out is None else out + self.v3(probs, tp, self.ws3, out, self.g(probs.shape[0]), pdl, True, combo[0], combo[1]) + return out + + +def _bad_rows(a, b): + return int((a.view(torch.int32) != b.view(torch.int32)).any(1).sum()) + + +def _fi_det(probs, top_p): + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + +@pytest.mark.parametrize("combo", COMBOS) +@pytest.mark.parametrize("rows", WIDE) +def test_wide_bitwise_v12_and_flashinfer(rows, combo): + """Every class and top_p mode: all rows equal v1.2's, and the top_p < 1 rows FlashInfer's det renorm.""" + r = _Runner(*_ops()) + bad_v12 = bad_fi = 0 + for cls in CLASSES: + for mode in ("p95", "mixed", "with_p1"): + probs, tp = _make(cls, rows, 31 * rows + len(mode)), _top_p(mode, rows, rows) + got = r.new(probs, tp, True, combo) + bad_v12 += _bad_rows(got, r.ref(probs, tp)) + below = tp < 1 + bad_fi += _bad_rows(got[below], _fi_det(probs, tp)[below]) + assert bad_v12 == 0 and bad_fi == 0, (bad_v12, bad_fi) + + +def test_wide_alternating_rows_one_workspace(): + """48 launches on one workspace alternating 8 / 64 / 448 / 9 rows (16 / 2 / 1 / 16 CTAs per row), the four + combinations and PDL: the barrier counters, parity buffers and lists stay consistent.""" + r = _Runner(*_ops()) + for i in range(48): + rows = (8, 64, 448, 9)[i % 4] + probs = _make(CLASSES[i % len(CLASSES)], rows, 500 + i) + tp = _top_p(("p95", "mixed", "with_p1")[i % 3], rows, i) + got = r.new(probs, tp, bool(i % 2), COMBOS[(i // 4) % 4]) + assert _bad_rows(got, r.ref(probs, tp)) == 0, (i, rows) + + +def test_wide_graph_replay_equals_eager(): + """The entry at 96 rows captured in a CUDA graph (workspace prebuilt outside the capture) gives the eager bits.""" + _ops() + if not entry.is_v3(): + pytest.skip("v3 is not the engaged op") + rows = 96 + probs = _make("peaked", rows, 5).clone() + tp = _top_p("mixed", rows, 5).clone() + entry.prebuild(probs.device) + ref = entry.top_p_renorm_det(probs, tp) + assert ref is not None, "96 rows are within the v3 cap" + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.top_p_renorm_det(probs, tp) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.top_p_renorm_det(probs, tp) + assert out is not None, "a prebuilt workspace must serve a captured call" + for k, cls in enumerate(("randn", "near_uniform", "peaked")): + probs.copy_(_make(cls, rows, 70 + k)) + tp.copy_(_top_p("with_p1" if k == 2 else "p95", rows, k)) + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + assert _bad_rows(out, entry.top_p_renorm_det(probs, tp)) == 0, cls + + +def _probe(env): + code = ( + f"import importlib, torch; m = importlib.import_module('{TARGET}.ops.sampling.m31_topp_det'); " + "m.topp_det_op(); p = torch.softmax(torch.randn(9, 200064, device='cuda'), -1); " + "t = torch.full((9,), 0.9, device='cuda'); " + "print('CAP', m.max_rows(), m.top_p_renorm_det(p, t) is not None, " + "m.prebuild().numel() == m.max_rows() * m.ws_bytes_per_row())" + ) + out = subprocess.run( + [sys.executable, "-c", code], + env={**os.environ, "TLLM_LOG_LEVEL": "INFO", **env}, + capture_output=True, + text=True, + check=True, + ) + text = out.stdout + out.stderr + lines = [ln for ln in text.splitlines() if "TOPP_DET row cap:" in ln] + return [ln for ln in out.stdout.splitlines() if ln.startswith("CAP ")][-1], lines + + +def test_row_cap_follows_the_engaged_op(): + """v3 takes 9 rows (cap 448, workspace for 448 rows); v1.2 declines them (cap 8); one cap line per process.""" + _ops() + v3_cap, v3_lines = _probe({"TRTLLM_M31_DISABLE_TOPPDET_V3": "0"}) + v12_cap, v12_lines = _probe({"TRTLLM_M31_DISABLE_TOPPDET_V3": "1"}) + assert v3_cap == f"CAP {entry.MAX_ROWS} True True" and entry.MAX_ROWS == 448 + assert v12_cap == f"CAP {entry.V12_MAX_ROWS} False True" and entry.V12_MAX_ROWS == 8 + assert len(v3_lines) == 1 and "row cap: 448 rows" in v3_lines[0] + assert len(v12_lines) == 1 and "row cap: 8 rows" in v12_lines[0] + + +def _engagement_lines(env, rows_seq): + code = ( + f"import importlib, torch; m = importlib.import_module('{TARGET}.ops.sampling.m31_topp_det'); " + f"[m.top_p_renorm_det(torch.softmax(torch.randn(r, {D}, device='cuda'), -1), " + "torch.full((r,), 0.9, device='cuda')) " + f"for r in {list(rows_seq)}]; torch.cuda.synchronize()" + ) + out = subprocess.run( + [sys.executable, "-c", code], + env={**os.environ, "TLLM_LOG_LEVEL": "INFO", **env}, + capture_output=True, + text=True, + check=True, + ) + text = out.stdout + out.stderr + return [ + ln for ln in text.splitlines() if "TOPP_DET taken at" in ln or "TOPP_DET stock at" in ln + ] + + +def test_taken_line_per_distinct_row_count(): + """One "TOPP_DET taken at N rows" line per distinct served row count, one "TOPP_DET stock at N rows" line per + declined one (above the cap), never repeated; with v1.2 the 9-row call is logged stock.""" + _ops() + lines = _engagement_lines( + {"TRTLLM_M31_DISABLE_TOPPDET_V3": "0"}, (8, 8, 64, 9, 64, 449, 449, 8) + ) + taken = sorted( + ln.split("TOPP_DET taken at ")[1].split(" rows")[0] for ln in lines if "taken at" in ln + ) + stock = [ln for ln in lines if "TOPP_DET stock at" in ln] + assert taken == ["64", "8", "9"], lines + assert len(stock) == 1 and "stock at 449 rows (above the 448-row cap)" in stock[0], lines + assert all("ctas_per_row" in ln for ln in lines if "taken at" in ln) + v12 = _engagement_lines({"TRTLLM_M31_DISABLE_TOPPDET_V3": "1"}, (8, 9, 9)) + assert len(v12) == 2, v12 + assert ( + "TOPP_DET taken at 8 rows" in v12[0] + and "TOPP_DET stock at 9 rows (above the 8-row cap)" in v12[1] + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/_m3_spec_inputs.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/_m3_spec_inputs.py new file mode 100644 index 000000000000..ef0c03180476 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/_m3_spec_inputs.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Inputs shared by the tp2 target's spec entry tests: TP2 dimensions, single-device stand-ins for the cross-rank +exchanges, and the comparison bands.""" + +from __future__ import annotations + +import zlib + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.spec.m3_shard_accept import ( # noqa: E501 + SHARD_ROWS, + make_keys, +) + +#: MiniMax-M3 at TP2: hidden size, vocabulary, the target head's shard and RMSNorm epsilon. +HIDDEN = 6144 +VOCAB = 200064 +WORLD = 2 +TARGET_SHARD = VOCAB // WORLD +EPS = 1e-6 + + +def seed_of(why: str) -> int: + return zlib.crc32(why.encode()) % 2**31 + + +def gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def num_sms() -> int: + return torch.cuda.get_device_properties(torch.cuda.current_device()).multi_processor_count + + +def randn(shape, g: torch.Generator, std: float = 1.0, dtype=torch.bfloat16) -> torch.Tensor: + return (torch.randn(shape, generator=g, device="cuda") * std).to(dtype) + + +def head_workspace(world: int, call: int = 0, slot_rows: int = SHARD_ROWS): + """(ws_uc int64 [2 * slot_rows * world], keys int64 [slot_rows], counters int32 [2]) of a fresh call ``call``.""" + uc = torch.zeros(2 * slot_rows * world, dtype=torch.int64, device="cuda") + keys = torch.zeros(slot_rows, dtype=torch.int64, device="cuda") + counters = torch.tensor([0, call], dtype=torch.int32, device="cuda") + return uc, keys, counters + + +def plant_peers( + uc: torch.Tensor, + counters: torch.Tensor, + own_best: torch.Tensor, + rank: int, + world: int, + shard: int, + g: torch.Generator, + slot_rows: int = SHARD_ROWS, + ties: bool = False, +) -> None: + """Write the other ranks' keys for each row into this rank's slots, as their multicast pushes would. + + Each peer's maximum is the own row maximum shifted by +-0.25 (well clear of the accumulation-order band), so + some rows are won by a peer and some by this rank; with ``ties`` a peer on an even row reports exactly the own + maximum, and the lower global index must win. + """ + if world == 1: + return + m = own_best.shape[0] + half = int(counters[1].item()) & 1 + for r in range(world): + if r == rank: + continue + sign = torch.where(torch.rand(m, generator=g, device="cuda") < 0.5, -1.0, 1.0) + value = own_best.float() + 0.25 * sign + if ties: + value = torch.where(torch.arange(m, device="cuda") % 2 == 0, own_best.float(), value) + index = r * shard + torch.randint(0, shard, (m,), generator=g, device="cuda") + keys = make_keys(value, index) + for t in range(m): + uc[(half * slot_rows + t) * world + r] = keys[t] + + +def assert_bf16_close(out: torch.Tensor, ref: torch.Tensor, max_frac: float = 1e-3) -> None: + """At most one bf16 ulp per element, and only on a ``max_frac`` share of the elements. + + Kernel and reference compute the same fp32 expression and round once; they differ only where a different + summation order (or rsqrtf) puts the fp32 value on the other side of a bf16 rounding boundary. + """ + assert out.shape == ref.shape and out.dtype == ref.dtype == torch.bfloat16 + o, r = out.float(), ref.float() + ulp = torch.ldexp(torch.ones_like(r), torch.frexp(r).exponent - 8) + ulp = torch.where(r == 0, torch.full_like(r, 2.0**-133), ulp) + diff = (o - r).abs() + assert bool((diff <= ulp * 1.0001).all()), f"max |diff| / ulp = {(diff / ulp).max().item():.2f}" + frac = (diff > 0).float().mean().item() + assert frac <= max_frac, f"{frac:.2e} of the elements moved by one ulp (allowed {max_frac})" + + +def craft_drafts(tokens: torch.Tensor, nreq: int, d: int) -> torch.Tensor: + """Drafts matching request b's verify tokens for its first (b + 2) % (d + 1) positions and differing from + there on, so the acceptance counts vary across requests.""" + t = tokens.view(nreq, d + 1)[:, :d].clone() + for b in range(nreq): + n = (b + 2) % (d + 1) + t[b, n:] += 1 + return t.contiguous() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dsfold.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dsfold.py new file mode 100644 index 000000000000..bcbe75fd3dbb --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dsfold.py @@ -0,0 +1,168 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""DSFOLD: under the standalone DSpark worker the TP2 target (as TP4 / TP8) folds the engine's previous-batch input +gathers into one ``step_gather`` launch (``M3DSparkStepFold``). Oracle: the decoder runner's own eager gathers +(engine/runners/decoder/inputs.py, the ``previous_batch_len > 0 and not inputs_staged`` branch) on the same inputs; +every destination buffer must match bit for bit, including the rows the gathers do not own.""" + +import importlib +import inspect +import os +import subprocess +import sys +import zlib + +import pytest +import torch + +assert torch.cuda.is_available(), "DSFOLD requires a CUDA device" + +_TARGETS = ("minimax_m31_nvfp4__sm_107__tp2",) +_PKG = "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3" + +#: Engine buffers sized for the max batch (8 requests x up to 1 + 7 tokens) and the stale fill they start with. +MAX_BATCH = 8 +MAX_TOKENS = MAX_BATCH * 8 +STALE = -4 +VOCAB = 200064 + + +def _modeling(target): + return importlib.import_module(f"{_PKG}.{target}.modeling") + + +def _engine_eager( + new_tokens, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 +): + """The decoder runner's gathers for P previous requests, transcribed from inputs.py (offsets zeroed first).""" + pos_off.zero_() + kv_off.zero_() + s = slots[:P] + ids[ids0 : ids0 + P * T].copy_(new_tokens.transpose(0, 1)[s, :T].flatten()) + if D > 0: + draft[d0 : d0 + P * D].copy_(next_draft[s, :D].flatten()) + kv_len_offsets = lens - T + pos_off[b0 * T : (b0 + P) * T].copy_(lens[ppi[: P * T]]) + kv_off[b0 : b0 + P].copy_(kv_len_offsets[s]) + + +def _case(k, store_k, prev, new_before, dummies, seed): + """K drafts per step (T = K + 1 tokens) read from a store sized for ``store_k`` drafts; ``prev`` previous requests + in shuffled slots after ``new_before`` requests without a previous tensor, then ``dummies`` CUDA-graph padding + rows. Slots include the padding dummy's slot (the store holds MAX_BATCH + 1 slots).""" + g = torch.Generator(device="cuda").manual_seed(seed) + i32 = dict(dtype=torch.int32, device="cuda") + T, D, nslots = k + 1, k, MAX_BATCH + 1 + assert new_before + prev + dummies <= MAX_BATCH + new_tokens = torch.randint(0, VOCAB, (store_k + 1, nslots, 1), generator=g, **i32) + next_draft = torch.randint(0, VOCAB, (nslots, store_k), generator=g, **i32) + lens = torch.randint(1, T + 1, (nslots,), generator=g, **i32) + slots = torch.randperm(nslots, generator=g, device="cuda")[:prev].to(torch.int32) + ppi = slots.repeat_interleave(T).contiguous() + ids0, d0 = new_before * T, new_before * D + bufs = dict( + ids=torch.full((MAX_TOKENS,), STALE, **i32), + draft=torch.full((MAX_TOKENS,), STALE, **i32), + pos_off=torch.full((MAX_TOKENS,), STALE, **i32), + kv_off=torch.full((MAX_BATCH,), STALE, **i32), + ) + return new_tokens, next_draft, lens, slots, ppi, prev, T, D, ids0, d0, new_before, bufs + + +def _run(fold_or_none, case): + new_tokens, next_draft, lens, slots, ppi, P, T, D, ids0, d0, b0, bufs = case + b = {n: t.clone() for n, t in bufs.items()} + if fold_or_none is None: + _engine_eager( + new_tokens, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + b["ids"], + ids0, + b["draft"], + d0, + b["pos_off"], + b["kv_off"], + b0, + ) + else: + fold_or_none.bind_engine_buffers(b["ids"], b["draft"], b["pos_off"], b["kv_off"], MAX_BATCH) + # The engine's call (inputs.py): first_previous_row = requests without a previous tensor before them. + assert fold_or_none.gather(new_tokens, next_draft, lens, slots, ppi, P, T, D, ids0, d0, b0) + torch.cuda.synchronize() + return b + + +# (K, store K, previous requests, new requests before them, padding dummies after them) +CELLS = [ + (3, 3, 1, 0, 0), # c1, K3: one steady request + (3, 3, 1, 0, 1), # c1 padded to the batch-2 graph + (3, 3, 8, 0, 0), # c8 steady, shuffled slots + (3, 3, 5, 2, 1), # mixed: 2 new rows before, 1 dummy after + (3, 7, 6, 1, 1), # K3 run from a store sized for K7 (dynamic draft width) + (7, 7, 1, 0, 0), # K7 c1 + (7, 7, 8, 0, 0), # K7 c8 + (7, 7, 4, 3, 1), # K7 mixed + dummy +] + + +@pytest.mark.parametrize("target", _TARGETS) +@pytest.mark.parametrize( + "cell", CELLS, ids=[f"K{c[0]}_store{c[1]}_p{c[2]}_new{c[3]}_dummy{c[4]}" for c in CELLS] +) +def test_fold_matches_engine_gathers(target, cell): + case = _case(*cell, seed=zlib.crc32(f"{target}{cell}".encode()) % 2**31) + ref = _run(None, case) + fold = _modeling(target).M3DSparkStepFold() + got = _run(fold, case) + for name in ("ids", "draft", "pos_off", "kv_off"): + assert torch.equal(got[name], ref[name]), f"{target} {cell}: {name}" + assert (fold.taken, fold.declined) == (1, 0) + + +@pytest.mark.parametrize("target", _TARGETS) +def test_fold_declines_and_engine_copies_run(target): + """A layout the kernel does not take (int64 slots) is declined with nothing written: the engine then gathers.""" + case = list(_case(3, 3, 4, 0, 0, seed=7)) + case[3] = case[3].long() + new_tokens, next_draft, lens, slots, ppi, P, T, D, ids0, d0, b0, bufs = case + fold = _modeling(target).M3DSparkStepFold() + b = {n: t.clone() for n, t in bufs.items()} + fold.bind_engine_buffers(b["ids"], b["draft"], b["pos_off"], b["kv_off"], MAX_BATCH) + assert not fold.gather(new_tokens, next_draft, lens, slots, ppi, P, T, D, ids0, d0, b0) + assert all(torch.equal(b[n], bufs[n]) for n in bufs) + assert (fold.taken, fold.declined) == (0, 1) + + +@pytest.mark.parametrize("target", _TARGETS) +def test_scatter_stays_in_tree(target): + """The sampler's slot stores are never folded under DSpark (the store path stays in-tree).""" + fold = _modeling(target).M3DSparkStepFold() + fold.bind_sampler_store(object()) + assert fold.scatter(None, None, None, None, None) is False + assert fold.begin_step([0], 1) is False + assert fold.sampler_rows_stored({}, 0, [0]) is False + + +@pytest.mark.parametrize("target", _TARGETS) +def test_kill_switch(target): + """TRTLLM_M31_DISABLE_DSFOLD=1 (read once at import): DSFOLD is off, so a DSpark run keeps spec_fold None and the + engine's own gathers; the Eagle3 path still builds M3SpecStepFold.""" + probe = f"import {_PKG}.{target}.modeling as m; print(m.DSFOLD)" + for value, want in (("1", "False"), ("0", "True")): + env = dict(os.environ, TRTLLM_M31_DISABLE_DSFOLD=value) + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == want + source = inspect.getsource(_modeling(target)) + # The sm_107 targets wire the fold differently (TP8: self._spec_fold; TP4: behind STEP_SPEC_FOLD), so check the + # DSpark branch rather than one exact text: DSFOLD gates M3DSparkStepFold, the Eagle3 path keeps M3SpecStepFold. + assert ( + "elif DSFOLD" in source and "M3DSparkStepFold()" in source and "M3SpecStepFold(" in source + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_n3a.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_n3a.py new file mode 100644 index 000000000000..dc66f835cdf8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_n3a.py @@ -0,0 +1,571 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""N3 A2 in this target's DSpark worker: the fused strict acceptance and next-step tokens equal the in-tree path +(N3 P and A1, the verify logits, at the end). + +The worker's two overrides are driven on a bare worker (no model) against the in-tree methods of its base class on +the same inputs: +- eligible steps (K3 / K7, padded drafts, graph-padding dummy rows) are bitwise; +- forced acceptance (fractional, so the RNG pool path runs) over a run that alternates eligible and ineligible + steps keeps the same accepted counts and the same RNG counter as an all-in-tree run; +- the sampler runs exactly once per step; +- the switch and an ineligible step take the in-tree path. +""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import strict_accept +from tensorrt_llm._torch.speculative.dspark import DSparkWorker + +TARGET = "minimax_m31_nvfp4__sm_107__tp2" +dspark = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.dspark" +) + +VOCAB = 200064 + +assert torch.cuda.is_available(), "the N3 A2 worker test requires a CUDA device" + + +def _worker(force: float = 0.0): + w = dspark.M3DSparkWorker.__new__(dspark.M3DSparkWorker) + nn.Module.__init__(w) + w.force_num_accepted_tokens = force + w._force_accept_rng_counter = None + w._force_accept_rng_pool = None + w.calls = 0 + return w + + +def _metadata(ragged: bool = False, graph: bool = False): + return SimpleNamespace(is_ragged_verify=ragged, is_cuda_graph=graph) + + +def _fixed_sampler(tokens: torch.Tensor): + """The in-tree sampler stands in as a fixed function of the logits: the overrides must call it once.""" + + def sample(self, logits, spec_metadata, num_contexts, batch_size): + self.calls += 1 + return tokens + + return sample + + +def _step(n_req: int, draft_len: int, gen: torch.Generator, pad: bool = False, int64: bool = False): + k1 = draft_len + 1 + tokens = torch.randint(0, VOCAB, (n_req * k1,), generator=gen, device="cuda") + tokens = tokens if int64 else tokens.to(torch.int32) + draft = tokens.view(n_req, k1)[:, :draft_len].clone().to(torch.int32) + cut = torch.randint(0, draft_len + 1, (n_req,), generator=gen, device="cuda") + pos = torch.arange(draft_len, device="cuda").unsqueeze(0) + draft = torch.where(pos == cut.unsqueeze(1), (draft + 1) % VOCAB, draft) + if pad: + draft[::2, draft_len // 2 :] = -1 + logits = torch.zeros( + n_req * k1, 8, device="cuda" + ) # shape only: the sampler stand-in ignores the values + return logits, draft, tokens + + +def _in_tree(w, logits, draft, num_contexts, batch_size, md): + return DSparkWorker._sample_and_accept_draft_tokens_base( + w, logits, draft, num_contexts, batch_size, md + ) + + +@pytest.fixture(autouse=True) +def _a1_opted_in(monkeypatch): + """A1 is on by default (switch TRTLLM_M31_DISABLE_A1); pin it on so the environment cannot turn it off here + (test_prebuild_a1_off turns it off again).""" + monkeypatch.setattr(dspark, "A1_DISABLED", False) + + +@pytest.fixture(autouse=True) +def _compiled(): + strict_accept.precompile((3, 7), torch.device("cuda")) + + +@pytest.mark.parametrize( + "n_req,draft_len,pad,int64", + [ + (1, 3, False, False), + (8, 3, False, False), + (12, 3, True, False), + (8, 7, False, False), + (8, 3, False, True), + ], +) +def test_eligible_steps_are_bitwise(monkeypatch, n_req, draft_len, pad, int64): + gen = torch.Generator(device="cuda").manual_seed(3) + logits, draft, tokens = _step(n_req, draft_len, gen, pad=pad, int64=int64) + monkeypatch.setattr(dspark.M3DSparkWorker, "_sample_tokens_for_batch", _fixed_sampler(tokens)) + w, ref = _worker(), _worker() + md = _metadata() + assert w._n3a_eligible(logits, draft, 0, n_req, md) + acc, num = w._sample_and_accept_draft_tokens_base(logits, draft, 0, n_req, md) + racc, rnum = _in_tree(ref, logits, draft, 0, n_req, md) + assert w.calls == 1 and ref.calls == 1 + assert acc.dtype == racc.dtype and num.dtype == rnum.dtype + assert torch.equal(acc, racc) and torch.equal(num, rnum) + + +def test_forced_acceptance_alternating_steps_matches_in_tree(monkeypatch): + """Fractional forced AL (the RNG pool path) over eligible, ragged (ineligible) and graph-warmup steps: the + counts and the RNG counter follow the all-in-tree run step for step.""" + gen = torch.Generator(device="cuda").manual_seed(5) + w, ref = _worker(force=1.6), _worker(force=1.6) + served = 0 + for step in range(12): + logits, draft, tokens = _step(8, 3, gen) + num_contexts, batch_size = 0, 8 + if step % 3 == 2: + # An ineligible step: one context row ahead of the 8 generation requests (the in-tree path runs). + ctx_token = torch.randint(0, VOCAB, (1,), generator=gen, device="cuda").to(tokens.dtype) + tokens = torch.cat([ctx_token, tokens]) + logits = torch.zeros(tokens.numel(), 8, device="cuda") + num_contexts, batch_size = 1, 9 + monkeypatch.setattr( + dspark.M3DSparkWorker, "_sample_tokens_for_batch", _fixed_sampler(tokens) + ) + md = _metadata(graph=(step == 4)) + eligible = w._n3a_eligible(logits, draft, num_contexts, batch_size, md) + assert eligible == (num_contexts == 0), step + served += int(eligible) + acc, num = w._sample_and_accept_draft_tokens_base( + logits, draft, num_contexts, batch_size, md + ) + racc, rnum = _in_tree(ref, logits, draft, num_contexts, batch_size, md) + # A context row defines only column 0 (the in-tree path leaves the rest uninitialized). + assert torch.equal(acc[:num_contexts, 0], racc[:num_contexts, 0]), step + assert torch.equal(acc[num_contexts:], racc[num_contexts:]) and torch.equal(num, rnum), step + if ref._force_accept_rng_counter is not None: + assert torch.equal(w._force_accept_rng_counter, ref._force_accept_rng_counter), step + assert served == 8 + # A ragged verify step is declined (the in-tree ragged path keeps it). + logits, draft, _ = _step(8, 3, gen) + assert not w._n3a_eligible(logits, draft, 0, 8, _metadata(ragged=True)) + + +def test_ineligible_and_switch_take_the_in_tree_path(monkeypatch): + gen = torch.Generator(device="cuda").manual_seed(9) + logits, draft, tokens = _step(4, 3, gen) + w = _worker() + # A context row, a row-count mismatch, too many requests: declined. + assert not w._n3a_eligible(logits, draft, 1, 4, _metadata()) + assert not w._n3a_eligible(logits[:-1], draft, 0, 4, _metadata()) + big_logits, big_draft, _ = _step(strict_accept.max_requests() + 1, 3, gen) + assert not w._n3a_eligible(big_logits, big_draft, 0, big_draft.shape[0], _metadata()) + monkeypatch.setattr(dspark, "N3A_DISABLED", True) + assert not w._n3a_eligible(logits, draft, 0, 4, _metadata()) + + +@pytest.mark.parametrize( + "draft_len,wide,shuffle", [(3, False, False), (3, True, True), (7, False, True)] +) +def test_next_new_tokens_matches_in_tree(draft_len, wide, shuffle): + gen = torch.Generator(device="cuda").manual_seed(13) + n_req = 8 + width = draft_len + 1 + (4 if wide else 0) + accepted = torch.randint( + 0, VOCAB, (n_req, width), generator=gen, device="cuda", dtype=torch.int32 + ) + num = torch.randint(1, draft_len + 2, (n_req,), generator=gen, device="cuda", dtype=torch.int32) + batch_idx = torch.arange(32, device="cuda", dtype=torch.int32) + if shuffle: + batch_idx[:n_req] = torch.randperm(n_req, generator=gen, device="cuda").to(torch.int32) + next_draft = torch.randint( + 0, VOCAB, (n_req, draft_len), generator=gen, device="cuda", dtype=torch.int32 + ) + w = _worker() + out = w._prepare_next_new_tokens(accepted, next_draft, batch_idx, n_req, num) + ref = DSparkWorker._prepare_next_new_tokens(w, accepted, next_draft, batch_idx, n_req, num) + assert out.dtype == ref.dtype and torch.equal(out, ref) + + +# N3 P-gid + P-cat: the worker's target logits equal the logits processor's. + + +@pytest.mark.parametrize("tp", [2, 4, 8]) +def test_rank_major_strided_copy_equals_vocab_concat_and_cast(tp): + """The all-gather along dim 0 + one strided copy writes what the vocab concat + .float() writes.""" + gen = torch.Generator(device="cuda").manual_seed(tp) + rows, shard = 32, 25008 + shards = [ + torch.randn(rows, shard, generator=gen, device="cuda").to(torch.bfloat16) for _ in range(tp) + ] + gathered = torch.cat(shards, dim=0) # what the dim-0 all-gather returns, rank-major + out = torch.empty((rows, tp * shard), dtype=torch.float32, device="cuda") + out.view(rows, tp, shard).copy_(gathered.view(tp, rows, shard).transpose(0, 1)) + assert torch.equal(out, torch.cat(shards, dim=-1).float()) + + +def _lm_head(vocab=50016, hidden=6144): + from tensorrt_llm._torch.modules.embedding import LMHead + from tensorrt_llm._torch.modules.linear import TensorParallelMode + from tensorrt_llm.mapping import Mapping + + head = LMHead( + vocab, + hidden, + dtype=torch.bfloat16, + mapping=Mapping(world_size=1, tp_size=1, rank=0), + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=True, + ).cuda() + with torch.no_grad(): + head.weight.copy_( + torch.randn_like(head.weight, dtype=torch.float32).to(torch.bfloat16) * 0.02 + ) + return head + + +def test_target_logits_equal_the_processor_at_tp1(): + from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor + + head = _lm_head() + rows = 32 + hidden = torch.randn(rows, 6144, device="cuda").to(torch.bfloat16) + md = SimpleNamespace( + is_ragged_verify=False, gather_ids=torch.arange(rows, device="cuda", dtype=torch.int32) + ) + attn = SimpleNamespace(num_contexts=0) + out = _worker().target_logits(hidden, attn, md, head) + ref = LogitsProcessor().forward(hidden[md.gather_ids], head, attn, True) + assert out is not None and out.dtype == ref.dtype and torch.equal(out, ref) + + +def test_target_logits_decline_to_the_processor(monkeypatch): + head = _lm_head(vocab=1024, hidden=256) + hidden = torch.randn(8, 256, device="cuda").to(torch.bfloat16) + ids = torch.arange(8, device="cuda", dtype=torch.int32) + w = _worker() + # A context row, a gather that is not every row, a ragged verify, a padded head: the processor runs. + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=1), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids[:4]), + head, + ) + is None + ) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=True, gather_ids=ids), + head, + ) + is None + ) + monkeypatch.setattr(head, "padding_size", 3) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + monkeypatch.setattr(head, "padding_size", 0) + monkeypatch.setattr(dspark, "N3P_DISABLED", True) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + + +# N3 A1: the verify logits' all-gather + concat + cast in one launch (catalog/comm/allgather_cast) when set up. + + +def _a1_step(monkeypatch, rows: int, compiled: bool = True): + """A TP2-shaped call: the head reports tp 2, A1 is set up, the op and the NCCL all-gather are recorded (K7: the A1 + path also counts A1_K7).""" + head = _lm_head(vocab=1024, hidden=256) + monkeypatch.setattr(head, "tp_size", 2) + calls = {"a1": 0, "nccl": 0} + sentinel = torch.full((rows, 2 * 1024), 7.0, device="cuda") + + def fake_a1(shard, **ws): + assert ws == {"ws": "a1"} and shard.shape == (rows, 1024) + calls["a1"] += 1 + return sentinel + + def fake_allgather(shard, mapping, dim=0): + calls["nccl"] += 1 + return torch.cat([shard] * 2, dim=0) + + monkeypatch.setattr(dspark._allgather_cast, "allgather_cast", fake_a1) + monkeypatch.setattr(dspark._allgather_cast, "is_compiled", lambda *a, **k: compiled) + monkeypatch.setattr(dspark, "allgather", fake_allgather) + w = _worker() + w.spec_config = SimpleNamespace(max_draft_len=7) # max_draft_len reads it (A1_K7's counter) + w._a1 = {"ws": "a1"} + hidden = torch.randn(rows, 256, device="cuda").to(torch.bfloat16) + md = SimpleNamespace( + is_ragged_verify=False, gather_ids=torch.arange(rows, device="cuda", dtype=torch.int32) + ) + out = w.target_logits(hidden, SimpleNamespace(num_contexts=0), md, head) + return out, sentinel, calls + + +@pytest.mark.parametrize("rows", [4, 32, 64, 184, 256, 257, 320, 448]) +def test_target_logits_take_a1_when_set_up(monkeypatch, rows): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_report, + ) + + m3_report.reset_work() + out, sentinel, calls = _a1_step(monkeypatch, rows) + assert out is sentinel and calls == {"a1": 1, "nccl": 0} + # Gate e: the verify-row logits (N3P), A1 and, at K7, A1_K7 each count the call. + assert {k: m3_report.work_count(k) for k in ("N3P", "A1", "A1_K7")} == { + "N3P": 1, + "A1": 1, + "A1_K7": 1, + } + + +@pytest.mark.parametrize("rows,compiled", [(449, True), (4, False)]) +def test_target_logits_fall_back_to_nccl_past_a1(monkeypatch, rows, compiled): + """More rows than the workspace holds, or a shape never compiled: the NCCL all-gather + strided copy.""" + out, _, calls = _a1_step(monkeypatch, rows, compiled=compiled) + assert ( + calls == {"a1": 0, "nccl": 1} + and out.shape == (rows, 2 * 1024) + and out.dtype == torch.float32 + ) + + +def _a1_prebuild_worker(monkeypatch, tp: int = 2, draft_len: int = 3): + w = _worker() + w.mapping = SimpleNamespace(tp_size=tp, tp_rank=0, enable_attention_dp=False) + w.spec_config = SimpleNamespace(max_draft_len=draft_len) # max_draft_len reads it + # Eligibility needs every SM up to the workspace's grid; any CUDA device passes with a 1-CTA workspace grid. + monkeypatch.setattr(dspark._allgather_cast, "MAX_GRID", 1) + monkeypatch.setattr(dspark._allgather_cast, "configure_capacity", lambda rows: None) + monkeypatch.setattr(dspark.M3DSparkWorker, "_a1_all_ok", lambda self, local: local) + return w + + +def test_prebuild_a1_sets_up_k3_rows(monkeypatch): + head = _lm_head(vocab=1024, hidden=256) + seen = {} + + def fake_precompile(mapping, cols, rows, device, group=None, agree=None): + seen.update(cols=cols, rows=rows, agreed=agree(True)) + return True + + ws = {"ws_uc": torch.zeros(4, dtype=torch.int32), "rank": 0} + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr( + dspark._allgather_cast, "workspace_args", lambda mapping, cols, group=None: ws + ) + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is ws + # K3 verify rows of batches 1..112 fit the extended workspace. + assert dspark.A1_WIDE and dspark.A1_WIDE_448 and dspark.A1_MAX_ROWS == 448 + assert seen == {"cols": 1024, "rows": tuple(range(4, 449, 4)), "agreed": True} + + +@pytest.mark.parametrize("draft_len", [3, 7]) +def test_prebuild_a1_sets_up_the_verify_rows_of_k3_and_k7(monkeypatch, draft_len): + """K3: rows 4..448 (batches 1..112); K7: rows 8..448 (batches 1..56).""" + head = _lm_head(vocab=1024, hidden=256) + seen = {} + + def fake_precompile(mapping, cols, rows, device, group=None, agree=None): + seen.update(rows=rows, agreed=agree(True)) + return True + + ws = {"ws_uc": torch.zeros(4, dtype=torch.int32), "rank": 0} + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr( + dspark._allgather_cast, "workspace_args", lambda mapping, cols, group=None: ws + ) + assert _a1_prebuild_worker(monkeypatch, draft_len=draft_len)._prebuild_a1(head) is ws + k1 = draft_len + 1 + assert seen == {"rows": tuple(range(k1, 449, k1)), "agreed": True} + + +def test_prebuild_a1_raises_the_workspace_capacity_first(monkeypatch): + """configure_capacity(A1_MAX_ROWS) runs before the compiles and the workspace.""" + head = _lm_head(vocab=1024, hidden=256) + order = [] + w = _a1_prebuild_worker(monkeypatch) + monkeypatch.setattr( + dspark._allgather_cast, "configure_capacity", lambda rows: order.append(("cap", rows)) + ) + monkeypatch.setattr( + dspark._allgather_cast, "precompile", lambda *a, **k: order.append(("compile",)) or True + ) + ws = {"ws_uc": torch.zeros(4, dtype=torch.int32), "rank": 0} + monkeypatch.setattr( + dspark._allgather_cast, + "workspace_args", + lambda mapping, cols, group=None: order.append(("ws",)) or ws, + ) + assert w._prebuild_a1(head) is ws + assert order == [("cap", dspark.A1_MAX_ROWS), ("compile",), ("ws",)] + + +def test_configure_capacity(monkeypatch): + """The capacity (and the fp32 half) is set before any workspace; afterwards only the same value passes.""" + ag = dspark._allgather_cast + monkeypatch.setattr(ag, "MAX_ROWS", 64) + monkeypatch.setattr(ag, "MAX_ROWS_FP32", 32) + monkeypatch.setattr(ag, "_workspaces", {}) + ag.configure_capacity(256) + assert (ag.MAX_ROWS, ag.MAX_ROWS_FP32) == (256, 128) + monkeypatch.setattr(ag, "_workspaces", {"built": object()}) + ag.configure_capacity(256) # the same value: no-op + with pytest.raises(RuntimeError): + ag.configure_capacity(64) + with pytest.raises(ValueError): + ag.configure_capacity(255) + + +def test_a1_wide_kill_switch_keeps_64_rows(): + """TRTLLM_M31_DISABLE_A1_WIDE=1: A1_MAX_ROWS 64 (read at import).""" + import os + import subprocess + import sys + + probe = ( + f"import importlib; m = importlib.import_module('{dspark.__name__}'); " + "print('CAP', m.A1_WIDE, m.A1_MAX_ROWS)" + ) + for flag, want in (("1", "CAP False 64"), ("0", "CAP True 448")): + out = subprocess.run( + [sys.executable, "-c", probe], + env=dict( + os.environ, TRTLLM_M31_DISABLE_A1_WIDE=flag, TRTLLM_M31_DISABLE_A1_WIDE_448="0" + ), + capture_output=True, + text=True, + check=True, + ) + assert [ln for ln in out.stdout.splitlines() if ln.startswith("CAP ")][-1] == want + + +def test_prebuild_a1_off(monkeypatch): + """The switch, another draft length, one rank, a rank that is not eligible, or a compile that failed on any rank: + no A1, no workspace.""" + head = _lm_head(vocab=1024, hidden=256) + built = {"n": 0} + + def fake_precompile(*a, **k): + built["n"] += 1 + return False # a failed compile somewhere + + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr(dspark, "A1_K7_DISABLED", True) + assert _a1_prebuild_worker(monkeypatch, draft_len=7)._prebuild_a1(head) is None + monkeypatch.setattr(dspark, "A1_K7_DISABLED", False) + assert ( + _a1_prebuild_worker(monkeypatch, draft_len=448)._prebuild_a1(head) is None + ) # 449 rows at batch 1 + assert _a1_prebuild_worker(monkeypatch, tp=1)._prebuild_a1(head) is None + assert built["n"] == 0 + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is None + assert built["n"] == 1 + # Another rank not eligible: off here too, nothing compiled. + w = _a1_prebuild_worker(monkeypatch) + monkeypatch.setattr(dspark.M3DSparkWorker, "_a1_all_ok", lambda self, local: False) + assert w._prebuild_a1(head) is None and built["n"] == 1 + monkeypatch.setattr(dspark, "A1_DISABLED", True) + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is None + assert built["n"] == 1 + + +# N3A-KV: the draft-forward KV bookkeeping in one launch, through the DFlash hook (bitwise to the stock lines). + + +def _kv_step(nc: int, bs: int, gen: torch.Generator): + num = torch.randint(1, 5, (bs + 2,), generator=gen, device="cuda", dtype=torch.int32) + kv = torch.randint(0, 1 << 20, (bs + 2,), generator=gen, device="cuda", dtype=torch.int32) + return num, kv + + +def _kv_worker(fused: bool): + w = _worker() + w._fused_kv_rewind = None + if fused: + strict_accept.precompile_kv_rewind(torch.device("cuda")) + w._n3a_kv_buf = torch.empty(strict_accept.max_requests(), dtype=torch.int32, device="cuda") + w._fused_kv_rewind = w._n3a_kv_rewind + return w + + +@pytest.mark.parametrize("nc,bs", [(0, 1), (0, 8), (2, 6), (3, 3)]) +def test_kv_rewind_hook_equals_the_stock_lines(nc, bs): + gen = torch.Generator(device="cuda").manual_seed(5) + num, kv = _kv_step(nc, bs, gen) + outs = [] + for fused in (False, True): + w = _kv_worker(fused) + attn = SimpleNamespace(kv_lens_cuda=kv.clone(), update_for_spec_dec=lambda: None) + w._prepare_kv_for_draft_forward(attn, num, nc, bs) + outs.append( + ( + w._kv_rewind_amount, + attn.kv_lens_cuda, + w._kv_rewind_nc, + w._kv_rewind_bs, + w._kv_rewind_pending, + ) + ) + (r0, k0, *rest0), (r1, k1, *rest1) = outs + assert torch.equal(k0, k1) and rest0 == rest1 + assert r0.dtype == r1.dtype and r0.shape == r1.shape and torch.equal(r0, r1) + + +def test_kv_rewind_hook_declines_unsupported_operands(): + w = _kv_worker(True) + num64 = torch.ones(4, dtype=torch.int64, device="cuda") # not the buffer's dtype + kv = torch.zeros(4, dtype=torch.int32, device="cuda") + assert w._n3a_kv_rewind(num64, kv, 0, 4) is None and torch.equal(kv, torch.zeros_like(kv)) + w._n3a_kv_buf = None + assert w._n3a_kv_rewind(torch.ones(4, dtype=torch.int32, device="cuda"), kv, 0, 4) is None + + +@pytest.mark.parametrize( + "wide_kill,extension_kill,capacity", + [("0", "0", 448), ("0", "1", 256), ("1", "0", 64), ("1", "1", 64)], +) +def test_a1_capacity_kill_hierarchy(wide_kill, extension_kill, capacity): + import os + import subprocess + import sys + + code = f"import {dspark.__name__} as m; print('CAP', m.A1_MAX_ROWS)" + env = dict( + os.environ, + TRTLLM_M31_DISABLE_A1_WIDE=wide_kill, + TRTLLM_M31_DISABLE_A1_WIDE_448=extension_kill, + ) + result = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert [line for line in result.stdout.splitlines() if line.startswith("CAP ")][ + -1 + ] == f"CAP {capacity}" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_s4.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_s4.py new file mode 100644 index 000000000000..368febb71937 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_s4.py @@ -0,0 +1,460 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp2 target's worker-side routing of the one-launch Markov chain (``dspark_s4.S4WorkerMixin``), without GPUs +or a model: which steps it serves, that the kill switch sets nothing up, that a declined step is the in-tree path, that +the step's logits processor only takes the block rows, and that the chain's tokens replace the in-tree sampling for +that one call. The chain itself is the catalog's ``spec_decode/dspark_markov_chain``, certified by +``spec_decode/_dspark_markov_chain_op_matrix.py``; the in-model equality by the call-site gate.""" + +import importlib +import os +import subprocess +import sys +from types import SimpleNamespace + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark_s4 as w1, +) + + +class _Base: + # The worker's TP mapping (the S4 work counter's line names the rank). + mapping = SimpleNamespace(tp_rank=0) + + def __init__(self): + self.calls = [] + + def _forward_impl(self, *args): + self.calls.append("forward") + return "in-tree" + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + self.calls.append("refine") + return gen_logits + + def sample_draft_tokens(self, logits, spec_metadata, batch_size, **kwargs): + self.calls.append("sample") + return torch.full(logits.shape[:2], -1, dtype=torch.int32) + + def _apply_kv_rewind_after_draft(self, attn_metadata, spec_metadata): + self.calls.append("stock_rewind") + self._kv_rewind_pending = False + + +class _Worker(w1.S4WorkerMixin, _Base): + pass + + +def _state(block=3, shard=8, rows=8): + st = w1._S4State(None, rows, shard, 0, None, None, block, {}) + for b in range(1, w1.MAX_FAST_BATCH + 1): + st.bufs[(b, block)] = torch.full((b, block, rows), float("-inf")) + return st + + +def _meta(advanced=False): + return SimpleNamespace(wants_advanced_draft_sampling=advanced) + + +def test_serves_only_eligible_steps(monkeypatch): + wk = _Worker() + draft = SimpleNamespace(lilicorr=None, has_candidate_selector=False) + assert not wk._s4_serves(1, _meta(), draft) # not set up + wk._s4 = _state() + assert wk._s4_serves(1, _meta(), draft) + assert wk._s4_serves(w1.MAX_FAST_BATCH, _meta(), draft) + assert not wk._s4_serves(0, _meta(), draft) + assert not wk._s4_serves(w1.MAX_FAST_BATCH + 1, _meta(), draft) + assert not wk._s4_serves(1, _meta(advanced=True), draft) + wk._d2t = torch.zeros(1) + assert not wk._s4_serves(1, _meta(), draft) + + +def test_kill_switch_sets_nothing_up(monkeypatch): + wk = _Worker() + wk._s4 = _state() + monkeypatch.setattr(w1, "_DISABLED", True) + assert not wk.prebuild_s4(None, SimpleNamespace(has_markov_head=True)) + assert wk._s4 is None + assert not wk._s4_serves( + 1, _meta(), SimpleNamespace(lilicorr=None, has_candidate_selector=False) + ) + + +def test_declined_step_is_the_in_tree_path(): + wk = _Worker() + attn = SimpleNamespace(num_seqs=w1.MAX_FAST_BATCH + 1, num_contexts=0) # above MAX_FAST_BATCH + draft = SimpleNamespace(lilicorr=None, has_candidate_selector=False, logits_processor="proc") + wk._s4 = _state() + assert wk._forward_impl(None, None, None, None, attn, _meta(), draft) == "in-tree" + assert draft.logits_processor == "proc" + logits = torch.zeros(2, 3, 16) # full-vocab width: not this step's shard + assert wk._refine_block_logits(draft, logits, {}, None) is logits + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() + assert wk.calls == ["forward", "refine", "sample"] + + +def test_sharded_logits_processor_takes_only_block_rows(): + st = _state(block=3, shard=8, rows=12) + inner_calls = [] + + def inner(h, head, meta, ctx): + inner_calls.append(h.shape) + return "inner" + + head = SimpleNamespace(apply_linear=lambda h, b: h[:, :8] * 2) + proc = w1._ShardedDraftLogits(inner, st, batch=2) + h = torch.arange(6 * 10, dtype=torch.float32).view(6, 10) + out = proc(h, head, None, True) + assert torch.equal(out, h[:, :8] * 2) + buf = st.buffer(2) + assert torch.equal(buf[:, :, :8], (h[:, :8] * 2).view(2, 3, 8)) + assert torch.isinf(buf[:, :, 8:]).all() + assert proc(torch.zeros(5, 10), head, None, True) == "inner" and inner_calls == [ + torch.Size([5, 10]) + ] + + +def test_chain_tokens_replace_sampling_once(monkeypatch): + wk = _Worker() + st = _state(block=3, shard=8, rows=8) + wk._s4_step = st + toks = torch.tensor([[1, 2, 3], [4, 5, 6]]) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, + dspark_markov_chain=lambda *a, **k: toks.int())) # fmt: skip + logits = torch.zeros(2, 3, 8) + assert wk._refine_block_logits(None, logits, {"first_prev_tokens": torch.zeros(2, dtype=torch.long)}, None) \ + is logits # fmt: skip + out = wk.sample_draft_tokens(logits, None, 2) + assert out.dtype == torch.int32 and torch.equal(out, toks.int()) + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() # used once + assert "refine" not in wk.calls + + +def test_chain_launch_counts_its_work(monkeypatch): + """Gate e: every chain launch counts S4 (and S4_FP32SUM with the fp32 sum, S4B8 at batch 1 with its kernel on).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_report, + ) + + m3_report.reset_work() + wk = _Worker() + wk._s4_fp32, wk._s4b8 = True, True + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, + dspark_markov_chain=lambda *a, **k: None)) # fmt: skip + for batch in (1, 2): + wk._s4_chain(_state(), torch.zeros(batch, 3, 8), torch.zeros(batch, dtype=torch.long), None) + assert ( + m3_report.work_count("S4"), + m3_report.work_count("S4_FP32SUM"), + m3_report.work_count("S4B8"), + ) == (2, 2, 1) + + +def test_unsupported_shapes_fall_back_on_the_shards(monkeypatch): + wk = _Worker() + wk._s4_step = _state(block=3, shard=8, rows=8) + + def _never(*a, **k): + raise AssertionError("an unsupported step must not reach the op") + + monkeypatch.setattr( + w1, "chain", SimpleNamespace(supports=lambda *a: False, dspark_markov_chain=_never) + ) + logits = torch.zeros(2, 3, 8) + wk._refine_block_logits( + None, logits, {"first_prev_tokens": torch.zeros(2, dtype=torch.long)}, None + ) + assert wk.calls == ["refine"] + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() + + +def test_switch_reads_any_value_but_empty_or_zero(monkeypatch): + """Read at import, like the targets' m31_switches: "true" turns S4 off, not only "1".""" + try: + for value, off in ( + ("1", True), + ("true", True), + ("yes", True), + ("0", False), + ("", False), + (" 0 ", False), + ): + monkeypatch.setenv(w1.DISABLE_ENV, value) + importlib.reload(w1) + assert w1.disabled() is off, value + finally: + monkeypatch.delenv(w1.DISABLE_ENV, raising=False) + importlib.reload(w1) + assert not w1.disabled() + + +def _rewind_worker(monkeypatch, batch=2, nc=1, on=True): + """A worker in a served step with a pending KV-length rewind of ``batch`` rows from ``nc``, and a chain fake that + records which launch ran.""" + wk = _Worker() + wk.mapping = SimpleNamespace(tp_rank=5) + wk._s4_step = _state(block=3, shard=8, rows=8) + wk._s4_step_attn = SimpleNamespace(kv_lens_cuda=torch.zeros(nc + batch + 1, dtype=torch.int32)) + wk._kv_rewind_pending = True + wk._kv_rewind_amount = torch.zeros(batch, dtype=torch.int32) + wk._kv_rewind_nc, wk._kv_rewind_bs = nc, nc + batch + launches, lines = [], [] + toks = torch.zeros(batch, 3, dtype=torch.int32) + + def plain(*a, **k): + launches.append("chain") + return toks + + def with_rewind(base, first_prev, w1_, w2s, offset, kv_lens, amount, first, **k): + launches.append( + ( + "rewind", + kv_lens is wk._s4_step_attn.kv_lens_cuda, + amount is wk._kv_rewind_amount, + first, + ) + ) + return toks + + monkeypatch.setattr(w1, "TAILBK_SPEC", on) + monkeypatch.setattr(w1.logger, "info", lambda msg, *a, **k: lines.append(msg)) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, supports_rewind=lambda *a: True, + dspark_markov_chain=plain, + dspark_markov_chain_rewind=with_rewind, + ll_exchange_engaged=lambda b: False)) # fmt: skip + return wk, launches, lines, {"first_prev_tokens": torch.zeros(batch, dtype=torch.long)} + + +def test_tailbk_h_rewind_in_chain_launch(monkeypatch): + """TAILBK_SPEC H: a real step's chain launch applies the pending rewind (this step's kv_lens, amounts, first + row) and the stock rewind lines are skipped for that step; the engagement line is logged once per rank.""" + wk, launches, lines, inputs = _rewind_worker(monkeypatch) + meta = SimpleNamespace(is_cuda_graph=False) + for _ in range(2): + wk._kv_rewind_pending = True + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert not wk._kv_rewind_pending and not wk._tailbk_rewound + assert launches == [("rewind", True, True, 1)] * 2, launches + assert "stock_rewind" not in wk.calls + assert [m for m in lines if "TAILBK_SPEC H" in m] == [ + "[M3 dispatch] TAILBK_SPEC H: KV-length rewind in the Markov chain launch on rank 5" + ] + + +def test_tailbk_h_warmup_keeps_stock_path(monkeypatch): + """The eager warmup of a graph batch (is_cuda_graph, not capturing): the stock method skips the rewind there, so + the chain must not apply it either.""" + wk, launches, _, inputs = _rewind_worker(monkeypatch) + meta = SimpleNamespace(is_cuda_graph=True) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def test_tailbk_h_off_or_mismatch_keeps_stock_path(monkeypatch): + """Kill switch off, no pending rewind, or amounts that are not this chain call's rows: the stock rewind runs.""" + meta = SimpleNamespace(is_cuda_graph=False) + wk, launches, _, inputs = _rewind_worker(monkeypatch, on=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + wk, launches, _, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_pending = False + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert launches == ["chain"] + wk, launches, _, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_amount = torch.zeros(3, dtype=torch.int32) # not the 2 rows of this call + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def test_tailbk_h_unsupported_operands_keep_stock_path(monkeypatch): + """supports_rewind False (e.g. int64 amounts): the plain chain launch and the stock rewind.""" + wk, launches, _, inputs = _rewind_worker(monkeypatch) + monkeypatch.setattr(w1.chain, "supports_rewind", lambda *a: False) + meta = SimpleNamespace(is_cuda_graph=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def _tailbk_in_subprocess(value): + env = {k: v for k, v in os.environ.items() if k != "TRTLLM_M31_DISABLE_TAILBK_SPEC"} + if value is not None: + env["TRTLLM_M31_DISABLE_TAILBK_SPEC"] = value + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_107__tp2 import dspark_s4 as m; print('TAILBK_SPEC=' + str(m.TAILBK_SPEC))" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=600 + ) + assert out.returncode == 0, out.stderr + return [ln for ln in out.stdout.splitlines() if ln.startswith("TAILBK_SPEC=")][-1] + + +def _cap_in_subprocess(value): + env = {k: v for k, v in os.environ.items() if k != "TRTLLM_M31_DISABLE_S4CAP12"} + if value is not None: + env["TRTLLM_M31_DISABLE_S4CAP12"] = value + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_107__tp2 import dspark_s4 as m; print('MAX_FAST_BATCH=' + str(m.MAX_FAST_BATCH))" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=600 + ) + assert out.returncode == 0, out.stderr + return [ln for ln in out.stdout.splitlines() if ln.startswith("MAX_FAST_BATCH=")][-1] + + +def test_s4cap12_default_and_kill_switch_subprocess(): + """S4CAP12: the TP2 target serves up to 12 requests by default; TRTLLM_M31_DISABLE_S4CAP12=1 (read at import) + restores 8.""" + assert _cap_in_subprocess(None) == "MAX_FAST_BATCH=12" + assert _cap_in_subprocess("0") == "MAX_FAST_BATCH=12" + assert _cap_in_subprocess("1") == "MAX_FAST_BATCH=8" + + +def test_tailbk_spec_kill_switch_subprocess(): + """TAILBK_SPEC is on by default; TRTLLM_M31_DISABLE_TAILBK_SPEC=1 (read at import) turns it off.""" + assert _tailbk_in_subprocess(None) == "TAILBK_SPEC=True" + assert _tailbk_in_subprocess("0") == "TAILBK_SPEC=True" + assert _tailbk_in_subprocess("1") == "TAILBK_SPEC=False" + + +def test_tailbk_c_engagement_logged_once_per_rank(monkeypatch): + """TAILBK_SPEC C's "[M3 dispatch]" line: once per rank, only after a chain call took the cached inputs.""" + wk, launches, lines, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_pending = False # the plain chain launch + taken = [False] + monkeypatch.setattr(w1.chain, "cached_inputs_taken", lambda: taken[0], raising=False) + meta = SimpleNamespace(is_cuda_graph=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert not [m for m in lines if "TAILBK_SPEC C" in m] + taken[0] = True + for _ in range(3): + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._kv_rewind_pending = True # and through the rewind launch + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert [m for m in lines if "TAILBK_SPEC C" in m] == [ + "[M3 dispatch] TAILBK_SPEC C: chain accepted inputs from precompile cache (taken) on rank 5" + ] + + +def _decline_worker(monkeypatch, rows=25088, agree=True): + """A worker whose S4 set-up runs to the rank agreement on stand-ins: the padded rows, the head check and the + agreement are stubbed; the logged lines are returned.""" + lines = [] + monkeypatch.setattr(w1, "logger", SimpleNamespace(info=lines.append, warning=lines.append)) + monkeypatch.setattr(w1, "_DISABLED", False) + monkeypatch.setattr(w1, "_head_is_sharded", lambda head, vocab, tp: True) + monkeypatch.setattr(w1.chain, "padded_rows", lambda shard, max_batch, device=None: rows) + wk = _Worker() + wk.mapping = SimpleNamespace(tp_rank=1, tp_size=2, enable_attention_dp=False) + wk.max_draft_len = 7 + wk._s4_ranks_agree = lambda local: local and agree + return wk, lines + + +def _markov_draft(vocab=200064, dtype=torch.bfloat16, rank=w1.MARKOV_RANK): + w = torch.empty(vocab, rank, dtype=dtype, device="meta") + return SimpleNamespace(has_markov_head=True, markov_w1=w, markov_w2=w) + + +def _only_decline(lines): + (line,) = lines + assert line.startswith("M3.1 DSpark S4: one-launch Markov chain off on rank 1/2 (") + assert line.endswith( + "; the in-tree Markov chain runs (TAILBK_SPEC chain sites C / H / S4L not taken)" + ) + return line + + +def test_c6_every_decline_logs_its_reason(monkeypatch): + """C6: no silent decline: each reason the set-up stops for is the one line it logs, and nothing is set up.""" + wk, lines = _decline_worker(monkeypatch) + assert wk.prebuild_s4(None, SimpleNamespace(has_markov_head=False)) is False + assert "the drafter has no Markov head" in _only_decline(lines) and wk._s4 is None + + wk, lines = _decline_worker(monkeypatch) + assert wk.prebuild_s4(None, _markov_draft(dtype=torch.float32)) is False + assert "not bf16" in _only_decline(lines) + + wk, lines = _decline_worker(monkeypatch) + assert wk.prebuild_s4(None, _markov_draft(rank=128)) is False + assert "Markov rank 128" in _only_decline(lines) + + wk, lines = _decline_worker(monkeypatch) + monkeypatch.setattr(w1, "_head_is_sharded", lambda head, vocab, tp: False) + assert wk.prebuild_s4(None, _markov_draft()) is False + assert "not a vocab-sharded" in _only_decline(lines) + + wk, lines = _decline_worker(monkeypatch, agree=False) + assert wk.prebuild_s4(None, _markov_draft()) is False + assert "off on another rank" in _only_decline(lines) + + wk, lines = _decline_worker(monkeypatch) + wk.max_draft_len = w1.chain.MAX_BLOCK + 1 + assert wk.prebuild_s4(None, _markov_draft()) is False + assert f"draft length {w1.chain.MAX_BLOCK + 1}" in _only_decline(lines) + + +def test_s4stream_the_tp2_shard_is_padded_and_streamed(monkeypatch): + """S4STREAM (was the C6 root cause at TP2): the chain kernel keeps at most SMEM_ROW_BUDGET markov_w2 rows per CTA + in shared memory and streams the rest, so the TP2 vocab shard (200064 / 2 = 100032 rows) is padded to 101376 = + 144 CTAs x 704 rows; the shapes that fit keep their padding and grid (TP4 50688 on 132 CTAs, TP8 25088 on 98).""" + monkeypatch.setattr(w1.chain, "_sm_count", lambda device: 212) # VR200 + kern = w1.chain._kernel_module() + tp2 = w1.chain.padded_rows(100032, w1.MAX_FAST_BATCH, "cpu") + assert tp2 == 101376 + assert ( + w1.chain.padded_rows(100032, 8, "cpu") == tp2 + ) # S4CAP12 keeps the S4STREAM padding of max batch 8 + assert w1.chain.pick_grid(tp2, 7, w1.MAX_FAST_BATCH, "cpu") > 0 + assert w1.chain.pick_grid(tp2, 7, 1, "cpu") == 144 and not kern.fits(tp2, 144) + tp4 = w1.chain.padded_rows(50016, w1.MAX_FAST_BATCH, "cpu") + assert tp4 == 50688 and w1.chain.pick_grid(tp4, 7, 1, "cpu") == 132 and kern.fits(tp4, 132) + tp8 = w1.chain.padded_rows(25008, w1.MAX_FAST_BATCH, "cpu") + assert tp8 == 25088 and w1.chain.pick_grid(tp8, 7, 1, "cpu") == 98 and kern.fits(tp8, 98) + + +def test_s4stream_capacity_decline_names_the_numbers(monkeypatch): + """A shard past MAX_GRID x MAX_ROWS_PER_CTA rows still declines with the numbers.""" + wk, lines = _decline_worker(monkeypatch, rows=0) + assert wk.prebuild_s4(None, _markov_draft()) is False + cap = w1.chain._kernel_module().MAX_ROWS_PER_CTA + line = _only_decline(lines) + assert ( + f"vocab shard 100032 rows past the chain kernel's capacity of {w1.chain.MAX_GRID} co-resident CTAs x " + f"{cap} markov_w2 rows = {w1.chain.MAX_GRID * cap} rows" in line + ) + + +def test_s4cap12_one_shared_logits_buffer(monkeypatch): + """S4CAP12: one [MAX_FAST_BATCH, K, rows] -inf-padded buffer; batch b's buffer is its leading b rows (contiguous + views of the same storage), so the 12 batches cost one buffer, not 78 rows of them.""" + wk, lines = _decline_worker(monkeypatch, rows=40) + monkeypatch.setattr(w1.chain, "pad_w2_shard", lambda w2, rows: w2) + monkeypatch.setattr(w1.chain, "workspace_args", lambda mapping: {}) + monkeypatch.setattr(w1.chain, "precompile", lambda *a, **k: None) + for name in ("enable_fp32_sum", "enable_batch1_variant", "enable_cached_inputs"): + monkeypatch.setattr(w1.chain, name, lambda on: on) + monkeypatch.setattr(w1.atexit, "register", lambda fn: None) + draft = SimpleNamespace(has_markov_head=True, markov_w1=torch.zeros(64, w1.MARKOV_RANK, dtype=torch.bfloat16), + markov_w2=torch.zeros(64, w1.MARKOV_RANK, dtype=torch.bfloat16), + lm_head=None) # fmt: skip + assert wk.prebuild_s4(None, draft) is True, lines + st = wk._s4 + whole = st.bufs[(w1.MAX_FAST_BATCH, 7)] + assert whole.shape == (w1.MAX_FAST_BATCH, 7, 40) and whole.isneginf().all() + for b in range(1, w1.MAX_FAST_BATCH + 1): + buf = st.buffer(b) + assert buf.shape == (b, 7, 40) and buf.is_contiguous() + assert buf.data_ptr() == whole.data_ptr() + assert f"batches 1..{w1.MAX_FAST_BATCH};" in lines[-1] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_w0.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_w0.py new file mode 100644 index 000000000000..710890d81e56 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_dspark_w0.py @@ -0,0 +1,171 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-model DSpark on the tp2 target, phase W0 (``dspark.py``): the shell's calls on the DSpark worker take no +fused path, the capture is the in-tree DFlash capture, the drafter's borrowed embedding is the target's exact gather, +and the FP8 draft-KV override (an Eagle3 need) is never applied to a DSpark run, so the drafter's context pool keeps +the projection dtype (BF16) and binds to the separate draft KV cache. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_target_head, +) +from tensorrt_llm.quantization.mode import QuantAlgo + + +def _spec( + *, dspark_mode: bool, eagle3: bool = False, one_engine: bool = True, embedded: bool = False +): + mode = SimpleNamespace( + use_one_engine=lambda: one_engine, + is_dspark=lambda: dspark_mode, + is_eagle3_one_model=lambda: eagle3, + ) + return SimpleNamespace(spec_dec_mode=mode, draft_is_embedded_in_target=embedded) + + +def _worker() -> dspark.M3DSparkWorker: + # The shell-facing calls read no worker state, so the worker is not constructed (that needs a mapping and a + # DSpark spec config). + return dspark.M3DSparkWorker.__new__(dspark.M3DSparkWorker) + + +@pytest.mark.parametrize( + "spec, expected", + [ + (_spec(dspark_mode=True), True), + (_spec(dspark_mode=True, embedded=True), False), + (_spec(dspark_mode=True, one_engine=False), False), + (_spec(dspark_mode=False, eagle3=True), False), + (None, False), + ], +) +def test_standalone_dspark_detection(spec, expected): + assert dspark.is_standalone_dspark(spec) is expected + + +def test_fp8_draft_kv_override_is_eagle3_only(): + assert dspark.draft_kv_override_applies(_spec(dspark_mode=False, eagle3=True)) + assert dspark.draft_kv_override_applies(None) + assert not dspark.draft_kv_override_applies(_spec(dspark_mode=True)) + + +@pytest.mark.parametrize("target_algo", [QuantAlgo.NVFP4, QuantAlgo.FP8, None]) +def test_drafter_kv_pool_keeps_the_drafter_dtype(target_algo): + from tensorrt_llm.models.modeling_utils import QuantConfig + + whole, layer = ( + QuantConfig(kv_cache_quant_algo=target_algo), + QuantConfig(kv_cache_quant_algo=target_algo), + ) + _ = whole.quant_mode # cached before the drop, as a loaded config may have it + cfg = SimpleNamespace(quant_config=whole, quant_config_dict={"model.layers.0": layer}) + dspark.keep_drafter_kv_dtype(cfg) + assert whole.kv_cache_quant_algo is None and layer.kv_cache_quant_algo is None + assert not whole.quant_mode.has_fp8_kv_cache() and not whole.quant_mode.has_fp4_kv_cache() + + +def test_shell_calls_take_no_fused_path(monkeypatch): + def _forbidden(*args, **kwargs): + raise AssertionError("W0 must never reach the coded target head") + + monkeypatch.setattr(m3_target_head, "prepare_head", _forbidden) + monkeypatch.setattr(m3_target_head, "logits", _forbidden) + worker = _worker() + assert worker.tail_eligible(None, None, None) is False + assert worker.arm_capture_norm(None, None, None, None) is None + assert worker.prebuild(None, None) is None + + +def test_capture_is_the_in_tree_capture(): + calls = [] + meta = SimpleNamespace(maybe_capture_hidden_states=lambda *a: calls.append(a)) + hidden, residual = object(), object() + _worker().capture_hidden_states(meta, 17, hidden, residual) + _worker().capture_hidden_states(meta, 59, hidden) + assert calls == [(17, hidden, residual), (59, hidden, None)] + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +def test_embed_tokens_is_the_target_gather(): + gen = torch.Generator(device="cuda").manual_seed(31) + table = torch.randn(1000, 64, generator=gen, device="cuda").to(torch.bfloat16) + core = SimpleNamespace(w={"embed": table}) + embed = dspark.M3EmbedTokens(core) + ids = torch.tensor([0, 5, 999, 200, -1], dtype=torch.long, device="cuda") + out = embed(ids) + ref = table[ids.clamp(min=0)] * (ids >= 0).unsqueeze(-1).to(table.dtype) + assert embed.weight is table + assert torch.equal(out, ref) + assert not list(embed.parameters()), ( + "the table belongs to the target, not to the borrowed module" + ) + + +def test_borrowed_embedding_owns_no_parameters(): + # The weights walker asserts every shell parameter is fed: the borrowed module must add none. + model = torch.nn.Module() + model.w = torch.nn.ParameterDict( + {"embed": torch.nn.Parameter(torch.zeros(8, 4), requires_grad=False)} + ) + before = {n for n, _ in model.named_parameters()} + model.embed_tokens = dspark.M3EmbedTokens(model) + assert {n for n, _ in model.named_parameters()} == before == {"w.embed"} + assert model.embed_tokens.weight is model.w["embed"] + + +class _FakeDrafter: + def __init__(self, borrows: bool = True) -> None: + self.loaded = None + self.borrows = borrows + self.lm_head = None + self.draft_model_full = SimpleNamespace(model=SimpleNamespace(embed_tokens=None)) + + def load_weights(self, weights, weight_mapper=None): + self.loaded = dict(weights) + + def load_weights_from_target_model(self, target): + if self.borrows: + self.lm_head = target.lm_head + self.draft_model_full.model.embed_tokens = target.model.embed_tokens + + +def _shell(drafter): + return SimpleNamespace( + draft_model=drafter, + lm_head=object(), + model=SimpleNamespace(embed_tokens=object()), + named_parameters=lambda: iter([("lm_head.weight", None), ("model.w.embed", None)]), + ) + + +@pytest.mark.parametrize( + "name", + ["model.embed_tokens.weight", "language_model.model.embed_tokens.weight", "lm_head.weight"], +) +def test_draft_load_refuses_borrowed_tensors(name): + shell = _shell(_FakeDrafter()) + with pytest.raises(ValueError, match="lends it"): + dspark.load_dspark_draft_weights(shell, {name: torch.zeros(1), "fc.weight": torch.zeros(1)}) + assert shell.draft_model.loaded is None + + +def test_draft_load_borrows_target_modules(): + shell = _shell(_FakeDrafter()) + weights = { + "fc.weight": torch.zeros(1), + "markov_head.markov_w1.weight": torch.zeros(1), + "confidence_head.proj.weight": torch.zeros(1), + } + dspark.load_dspark_draft_weights(shell, weights) + assert shell.draft_model.loaded == weights + assert shell.draft_model.lm_head is shell.lm_head + with pytest.raises(RuntimeError, match="does not borrow"): + dspark.load_dspark_draft_weights(_shell(_FakeDrafter(borrows=False)), weights) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_s4fe.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_s4fe.py new file mode 100644 index 000000000000..1d447bd0ab31 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_s4fe.py @@ -0,0 +1,184 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""S4FE on the tp2 target: the steps S4's chain declines run the in-tree Markov chain with the fused tail +(``ops/spec/m31_markov_epilogue.py``). CPU: the routing (only full-vocab, unsharded steps with a Markov head while S4 is +not serving; the in-tree path otherwise) and the kill switch. GPU (SM107, the op built): the certified cells and the +edge cases bitwise the in-tree chain, eager and under graph replay, and the counters left zero.""" + +import os +import subprocess +import sys +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark_s4 as w1, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.spec import ( # noqa: E501 + m31_markov_epilogue as epi, +) + +VOCAB = 16 + + +class _Base: + mapping = SimpleNamespace(tp_rank=0) + + def __init__(self, sharded=False): + self.calls = [] + self.sharded = sharded + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + self.calls.append("in-tree") + return "in-tree" + + def _draft_logits_are_sharded(self, logits, spec_metadata): + return self.sharded + + +class _Worker(w1.S4WorkerMixin, _Base): + pass + + +def _draft(markov=True, vocab=VOCAB): + return SimpleNamespace( + has_markov_head=markov, markov_w1=torch.zeros(vocab, 4), markov_w2=torch.zeros(vocab, 4) + ) + + +def _route(monkeypatch, wk, draft, logits, result="fused"): + seen = [] + monkeypatch.setattr(w1, "m31_markov_chain_fused", lambda *a: seen.append(a) or result) + out = wk._refine_block_logits( + draft, logits, {"first_prev_tokens": torch.zeros(logits.shape[0])}, None + ) + return out, seen + + +def test_s4fe_is_on_by_default(): + assert w1.S4FE + + +def test_a_declined_full_vocab_step_takes_the_fused_chain(monkeypatch): + wk = _Worker() + out, seen = _route(monkeypatch, wk, _draft(), torch.zeros(13, 3, VOCAB)) + assert out == "fused" and len(seen) == 1 and wk.calls == [] + + +@pytest.mark.parametrize( + "why, worker, draft, width", + [ + ("no Markov head", _Worker(), _draft(markov=False), VOCAB), + ("sharded logits", _Worker(sharded=True), _draft(), VOCAB), + ("a vocab shard", _Worker(), _draft(), VOCAB // 2), + ], +) +def test_other_steps_keep_the_in_tree_chain(monkeypatch, why, worker, draft, width): + out, seen = _route(monkeypatch, worker, draft, torch.zeros(13, 3, width)) + assert out == "in-tree" and seen == [] and worker.calls == ["in-tree"], why + + +def test_an_op_decline_keeps_the_in_tree_chain(monkeypatch): + wk = _Worker() + out, seen = _route(monkeypatch, wk, _draft(), torch.zeros(13, 3, VOCAB), result=None) + assert out == "in-tree" and len(seen) == 1 and wk.calls == ["in-tree"] + + +def test_a_served_s4_step_never_reaches_s4fe(monkeypatch): + wk = _Worker() + wk._s4_step = SimpleNamespace( + shard=VOCAB // 2 + ) # an S4 step whose logits are not its shard: S4 falls through + out, seen = _route(monkeypatch, wk, _draft(), torch.zeros(13, 3, VOCAB)) + assert out == "in-tree" and seen == [] + + +def test_kill_switch(): + code = f"from {w1.__name__} import S4FE\nprint(S4FE)" + env = {**os.environ, "TRTLLM_M31_DISABLE_S4FE": "1"} + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.split()[-1] == "False" + + +_GPU = pytest.mark.skipif( + not torch.cuda.is_available() + or torch.cuda.get_device_capability() != (10, 7) + or epi.markov_epilogue_op() is None, + reason="the S4FE op runs on SM107 with the op built", +) +_RANK, _VOCAB = 256, 200064 + + +def _inputs(b, k, seed, case="randn"): + g = torch.Generator(device="cuda").manual_seed(seed) + ws = {"lo": 2.0**-12, "hi": 2.0**12}.get(case, 1.0) + base = torch.randn(b, k, _VOCAB, device="cuda", generator=g) * 3 * ws + if case == "ties": + base = torch.round(base * 2) / 2 + w1_ = (torch.randn(_VOCAB, _RANK, device="cuda", generator=g) * 0.05 * ws).bfloat16() + w2_ = (torch.randn(_VOCAB, _RANK, device="cuda", generator=g) * 0.05 * ws).bfloat16() + prev = torch.randint(0, _VOCAB, (b,), device="cuda", generator=g) + if case == "special": + base[0, :, 77] = float("nan") + base[0, :, 5] = float("nan") + base[1, :, 1000:150000] = float("-inf") + base[2] = float("-inf") + prev[3] = -1 + return base.contiguous(), prev, w1_, w2_ + + +def _same(a, b): + return torch.equal(a.view(torch.int32), b.view(torch.int32)) + + +@_GPU +@pytest.mark.parametrize( + "cell", epi.M31MarkovChainFused.CELLS, ids=[c.why[:40] for c in epi.M31MarkovChainFused.CELLS] +) +def test_cells_bitwise_the_in_tree_chain(cell): + op = epi.m31_markov_chain_fused + for seed in range(2): + args = _inputs(cell.spec["b"], cell.spec["k"], seed) + assert _same(op(*args), op.reference(*args)) + + +@_GPU +@pytest.mark.parametrize("case", ["ties", "lo", "hi", "special"]) +def test_edge_cases_bitwise_the_in_tree_chain(case): + op = epi.m31_markov_chain_fused + args = _inputs(16, 3, 11, case) + assert _same(op(*args), op.reference(*args)) + + +@_GPU +@pytest.mark.parametrize("b,k", [(16, 3), (32, 3), (56, 3), (112, 3), (56, 7)]) +def test_graph_replay_matches_eager_and_leaves_the_counters_zero(b, k): + op = epi.m31_markov_chain_fused + args = _inputs(b, k, 5) + eager = op(*args) + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s), torch.cuda.graph(g, stream=s): + out = op(*args) + torch.cuda.current_stream().wait_stream(s) + for _ in range(3): + g.replay() + torch.cuda.synchronize() + assert _same(out, eager) and _same(out, op.reference(*args)) + _, counters = epi.prebuild(args[0].device) + assert int(counters.abs().sum()) == 0 + + +def test_wider_steps_decline(): + base = torch.zeros(epi.MAX_ROWS + 1, 3, 16) + assert ( + epi.m31_markov_chain_fused( + base, torch.zeros(epi.MAX_ROWS + 1), torch.zeros(16, 4), torch.zeros(16, 4) + ) + is None + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail16_g2.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail16_g2.py new file mode 100644 index 000000000000..a8aade425447 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail16_g2.py @@ -0,0 +1,97 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TAIL16_G2 on the tp2 target: the worker's _zero_padding_rows in place and in one launch +(trtllm::minimax_m31_zero_padding_rows). CPU: the stock fallbacks and the kill switch. GPU (SM107, op built): +bitwise the stock SpecWorkerBase._zero_padding_rows result with 0 / 1 / all padding requests at 3 and 4 rows +per request.""" + +import os +import subprocess +import sys +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark_s4 as w1, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.spec import ( # noqa: E501 + m31_markov_epilogue as epi, +) +from tensorrt_llm._torch.speculative.interface import SpecWorkerBase + +DUMMY = 7 + + +def _meta(slots, ragged=False): + return SimpleNamespace(batch_slot_ids=slots, dummy_slot_row=DUMMY, is_ragged_verify=ragged) + + +def test_g2_is_on_by_default(): + assert w1.TAIL16_G2 + + +def test_kill_switch(): + code = f"from {w1.__name__} import TAIL16_G2\nprint(TAIL16_G2)" + env = {**os.environ, "TRTLLM_M31_DISABLE_TAIL16_G2": "1"} + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.split()[-1] == "False" + + +@pytest.mark.parametrize( + "why, kwargs", [("ragged verify", dict(packed_verify=True)), ("row mismatch", dict(rpr=5))] +) +def test_other_layouts_take_the_stock_path(monkeypatch, why, kwargs): + seen = [] + monkeypatch.setattr(SpecWorkerBase, "_zero_padding_rows", + staticmethod(lambda *a, **k: seen.append(why) or "stock")) # fmt: skip + monkeypatch.setattr(w1, "markov_epilogue_op", lambda: object()) + slots = torch.tensor([1, DUMMY, 2, 3]) + logits = torch.zeros(16, 8) + out = w1.S4WorkerMixin._zero_padding_rows(logits, _meta(slots, ragged=True), 0, 4, kwargs.get("rpr", 4), + packed_verify=kwargs.get("packed_verify", False)) # fmt: skip + assert out == "stock" and seen == [why] + + +_GPU = pytest.mark.skipif( + not torch.cuda.is_available() + or torch.cuda.get_device_capability() != (10, 7) + or epi.markov_epilogue_op() is None, + reason="the TAIL16_G2 op runs on SM107 with the op built", +) + + +@_GPU +@pytest.mark.parametrize("rpr", [3, 4]) +@pytest.mark.parametrize("npad", [0, 1, 4]) +def test_bitwise_the_stock_zeroing(rpr, npad): + nc, ng, v = 2, 4, 200064 + slots = torch.arange(10, 10 + nc + ng, dtype=torch.int64, device="cuda") + slots[nc + ng - npad : nc + ng] = DUMMY + logits = torch.randn(ng * rpr, v, device="cuda") + logits[0, 5] = float("nan") + ref = SpecWorkerBase._zero_padding_rows(logits.clone(), _meta(slots), nc, nc + ng, rpr) + got = w1.S4WorkerMixin._zero_padding_rows(logits.clone(), _meta(slots), nc, nc + ng, rpr) + assert torch.equal(ref.view(torch.int32), got.view(torch.int32)) + + +@_GPU +def test_graph_replay(): + nc, ng, rpr, v = 0, 4, 4, 200064 + slots = torch.tensor([3, 4, DUMMY, 5], dtype=torch.int64, device="cuda") + base = torch.randn(ng * rpr, v, device="cuda") + ref = SpecWorkerBase._zero_padding_rows(base.clone(), _meta(slots), nc, ng, rpr) + x = base.clone() + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s), torch.cuda.graph(g, stream=s): + w1.S4WorkerMixin._zero_padding_rows(x, _meta(slots), nc, ng, rpr) + torch.cuda.current_stream().wait_stream(s) + x.copy_(base) + g.replay() + torch.cuda.synchronize() + assert torch.equal(ref.view(torch.int32), x.view(torch.int32)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail16_g3.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail16_g3.py new file mode 100644 index 000000000000..8c6165fe722b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail16_g3.py @@ -0,0 +1,167 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TAIL16_G3: block draft probabilities stored with minimax_m31_scatter_rows. +CPU: the stock fallback and the kill switch. GPU (SM107, op built): bitwise the stock index_put store (random, NaN, -inf +rows; K 3 and 7; padded last dim), and under graph replay.""" + +import os +import subprocess +import sys + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark_s4 as w1, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.spec import ( # noqa: E501 + m31_markov_epilogue as epi, +) +from tensorrt_llm._torch.speculative.interface import SpecWorkerBase + + +def test_g3_is_on_by_default(): + assert w1.TAIL16_G3 + + +def test_kill_switch(): + code = f"from {w1.__name__} import TAIL16_G3\nprint(TAIL16_G3)" + env = {**os.environ, "TRTLLM_M31_DISABLE_TAIL16_G3": "1"} + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.split()[-1] == "False" + + +def test_unsupported_dtype_takes_the_stock_store(monkeypatch): + seen = [] + monkeypatch.setattr( + SpecWorkerBase, "_store_block_draft_probs", staticmethod(lambda *a: seen.append(1)) + ) + monkeypatch.setattr(w1, "markov_epilogue_op", lambda: object()) + w1.S4WorkerMixin._store_block_draft_probs(torch.zeros(8, 3, 16, dtype=torch.float64), torch.tensor([1, 2]), + torch.zeros(2, 3, 16, dtype=torch.float64)) # fmt: skip + assert seen == [1] + + +_GPU = pytest.mark.skipif( + not torch.cuda.is_available() + or torch.cuda.get_device_capability() != (10, 7) + or epi.markov_epilogue_op() is None, + reason="the TAIL16_G3 op runs on SM107 with the op built", +) + + +@_GPU +@pytest.mark.parametrize("k", [3, 7]) +@pytest.mark.parametrize("pad", [0, 8]) +def test_bitwise_the_stock_store(k, pad): + v = 200064 + dst0 = torch.randn(64, 7, v + pad, device="cuda") + probs = torch.rand(4, k, v, device="cuda") + probs[0, 0, 5] = float("nan") + probs[1, -1, :100] = float("-inf") + slots = torch.tensor([3, 17, 40, 9], dtype=torch.int64, device="cuda") + ref, got = dst0.clone(), dst0.clone() + SpecWorkerBase._store_block_draft_probs(ref, slots, probs) + w1.S4WorkerMixin._store_block_draft_probs(got, slots, probs) + assert torch.equal(ref.view(torch.int32), got.view(torch.int32)) + + +@_GPU +def test_graph_replay(): + v = 200064 + dst0 = torch.zeros(64, 3, v, device="cuda") + probs = torch.rand(4, 3, v, device="cuda") + slots = torch.tensor([3, 17, 40, 9], dtype=torch.int64, device="cuda") + ref = dst0.clone() + SpecWorkerBase._store_block_draft_probs(ref, slots, probs) + got = dst0.clone() + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + g = torch.cuda.CUDAGraph() + with torch.cuda.stream(s), torch.cuda.graph(g, stream=s): + w1.S4WorkerMixin._store_block_draft_probs(got, slots, probs) + torch.cuda.current_stream().wait_stream(s) + got.zero_() + g.replay() + torch.cuda.synchronize() + assert torch.equal(ref.view(torch.int32), got.view(torch.int32)) + + +def test_cpu_fp32_uses_stock(monkeypatch): + monkeypatch.setattr(w1, "markov_epilogue_op", lambda: object()) + dst = torch.zeros(4, 3, 16) + src = torch.randn(2, 3, 16) + slots = torch.tensor([3, 1]) + ref = dst.clone() + SpecWorkerBase._store_block_draft_probs(ref, slots, src) + w1.S4WorkerMixin._store_block_draft_probs(dst, slots, src) + assert torch.equal(ref, dst) + + +def test_old_native_library_uses_stock(monkeypatch): + monkeypatch.setattr(w1, "markov_epilogue_op", lambda: object()) + monkeypatch.setattr(torch.ops.trtllm, "minimax_m31_scatter_rows", None, raising=False) + seen = [] + monkeypatch.setattr( + SpecWorkerBase, "_store_block_draft_probs", staticmethod(lambda *a: seen.append(a)) + ) + w1.S4WorkerMixin._store_block_draft_probs(None, None, None) + assert seen == [(None, None, None)] + + +@_GPU +@pytest.mark.parametrize( + "n,k,v,pad,offset", + [ + (56, 7, 200064, 8, 0), + (112, 3, 200064, 0, 0), + (9, 3, 16383, 1, 0), + (9, 3, 16385, 3, 0), + (9, 7, 16384, 0, 1), + (9, 7, 16388, 1, 0), + ], +) +def test_wide_and_scalar_layouts(n, k, v, pad, offset): + assert getattr(torch.ops.trtllm, "minimax_m31_scatter_rows", None) is not None + backing = torch.zeros((n + 4) * (k + 1) * (v + pad) + offset, device="cuda") + dst = backing[offset:].view(n + 4, k + 1, v + pad) + src_backing = torch.randn(n * k * v + offset, device="cuda") + src = src_backing[offset:].view(n, k, v) + src.view(-1)[0:4] = torch.tensor( + [float("nan"), float("inf"), float("-inf"), -0.0], device="cuda" + ) + slots = torch.randperm(n + 4, device="cuda")[:n] + ref = dst.clone() + SpecWorkerBase._store_block_draft_probs(ref, slots, src) + w1.S4WorkerMixin._store_block_draft_probs(dst, slots, src) + assert torch.equal(ref.view(torch.int32), dst.view(torch.int32)) + + +@_GPU +def test_graph_changed_inputs_and_duplicate_padding(): + assert getattr(torch.ops.trtllm, "minimax_m31_scatter_rows", None) is not None + n, k, v = 56, 3, 16385 + dst = torch.zeros(n + 4, k + 1, v + 1, device="cuda") + src = torch.randn(n, k, v, device="cuda") + slots = torch.arange(n, device="cuda") + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + w1.S4WorkerMixin._store_block_draft_probs(dst, slots, src) + torch.cuda.current_stream().wait_stream(stream) + for seed in (17, 83, 137): + torch.manual_seed(seed) + src.normal_() + slots.copy_(torch.randperm(n + 4, device="cuda")[:n]) + # Graph padding can share a dummy slot; identical source bits are required. + slots[-1].copy_(slots[-2]) + src[-1].copy_(src[-2]) + dst.zero_() + ref = dst.clone() + SpecWorkerBase._store_block_draft_probs(ref, slots, src) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(ref.view(torch.int32), dst.view(torch.int32)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail_bk3.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail_bk3.py new file mode 100644 index 000000000000..4451cab404ce --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m31_tail_bk3.py @@ -0,0 +1,123 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TAIL_BK3: the S4FE position-0 Markov embedding (minimax_m31_markov_prev_embed), the block sampler's per-row +parameters (minimax_m31_expand_block_sampling, behind SpecWorkerBase._block_sampling_params) and the int32 block tokens. +CPU: defaults, the kill switch, the stock fallbacks. GPU (SM107, ops built): bitwise the in-tree torch sequences.""" + +import os +import subprocess +import sys +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark_s4 as w1, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.spec import ( # noqa: E501 + m31_markov_epilogue as epi, +) +from tensorrt_llm._torch.models.modeling_speculative import markov_prev_embeddings +from tensorrt_llm._torch.speculative.interface import _RNG_SLOT_SPAN, SpecWorkerBase + + +def test_on_by_default(): + assert w1.TAIL_BK3 + assert w1.S4WorkerMixin._block_tokens_keep_int32 + assert not SpecWorkerBase._block_tokens_keep_int32 + + +def test_kill_switch(): + code = f"from {w1.__name__} import TAIL_BK3, S4WorkerMixin\nprint(TAIL_BK3, S4WorkerMixin._block_tokens_keep_int32)" + env = {**os.environ, "TRTLLM_M31_DISABLE_TAIL_BK3": "1"} + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.split()[-2:] == ["False", "False"] + + +class _Base: + def _block_sampling_params(self, spec_metadata, num_contexts, batch_size, K): + return "stock" + + +class _Worker(w1.S4WorkerMixin, _Base): + pass + + +def _meta(n, device="cpu", top_ks_dtype=torch.int32): + return SimpleNamespace( + request_temperatures=torch.rand(n, device=device), + request_top_ks=torch.zeros(n, dtype=top_ks_dtype, device=device), + request_top_ps=torch.rand(n, device=device), + request_min_ps=torch.zeros(n, device=device), + request_seeds=torch.randint(0, 2**40, (n,), device=device), + request_offsets=torch.randint(0, 2**40, (n,), device=device), + rng_draft_slot=lambda step: 4 + step, + ) + + +def test_cpu_buffers_take_the_stock_params(): + assert _Worker()._block_sampling_params(_meta(8), 0, 8, 3) == "stock" + + +def test_unsupported_dtype_takes_the_stock_params(monkeypatch): + monkeypatch.setattr(w1, "markov_epilogue_op", lambda: object()) + meta = _meta(8, top_ks_dtype=torch.int64) + assert _Worker()._block_sampling_params(meta, 0, 8, 3) == "stock" + + +_GPU = pytest.mark.skipif( + not torch.cuda.is_available() + or torch.cuda.get_device_capability() != (10, 7) + or epi.markov_epilogue_op() is None + or not hasattr(torch.ops.trtllm, "minimax_m31_expand_block_sampling"), + reason="the TAIL_BK3 ops run on SM107 with the ops built", +) + + +def _same(a, b): + if a.dtype != b.dtype or a.shape != b.shape: + return False + if a.dtype == torch.bfloat16: + return torch.equal(a.view(torch.int16), b.view(torch.int16)) + if a.dtype == torch.float32: + return torch.equal(a.view(torch.int32), b.view(torch.int32)) + return torch.equal(a, b) + + +@_GPU +@pytest.mark.parametrize("n", [1, 22, 28, 112]) +def test_markov_embedding_bitwise(n): + v, rank = 200064, 256 + w = (torch.randn(v, rank, device="cuda") * 0.05).to(torch.bfloat16) + w[7, 3], w[11, :] = float("nan"), -0.0 + prev = torch.randint(0, v, (n,), device="cuda") + edge = torch.tensor([-1, v, 0, v - 1, 7, 11], device="cuda") + prev[: min(n, 6)] = edge[: min(n, 6)] + assert _same(markov_prev_embeddings(prev, w), epi._prev_embeddings(prev, w)) + + +@_GPU +@pytest.mark.parametrize("n", [1, 24, 112]) +@pytest.mark.parametrize("k", [3, 7]) +def test_sampling_params_bitwise(n, k): + meta = _meta(n + 2, device="cuda") + meta.request_temperatures[2] = float("nan") + meta.request_offsets[-1] = 2**63 - 40 + got = _Worker()._block_sampling_params(meta, 2, n + 2, k) + slot = meta.rng_draft_slot(0) + sl = slice(2, n + 2) + offsets = (meta.request_offsets[sl] + slot * _RNG_SLOT_SPAN).repeat_interleave( + k + ) + torch.arange(k, dtype=torch.int64, device="cuda").repeat(n) * _RNG_SLOT_SPAN + ref = ( + meta.request_temperatures[sl].repeat_interleave(k), + meta.request_top_ks[sl].repeat_interleave(k), + meta.request_top_ps[sl].repeat_interleave(k), + meta.request_min_ps[sl].repeat_interleave(k), + meta.request_seeds[sl].repeat_interleave(k), + offsets, + ) + assert got != "stock" and all(_same(a, b) for a, b in zip(ref, got)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_spec_offsets.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_spec_offsets.py new file mode 100644 index 000000000000..eeb8d317b576 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_spec_offsets.py @@ -0,0 +1,138 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for m3_spec_offsets (RuntimeHooks.spec_offsets_op): the overlap scheduler's two speculative step corrections +in one plain launch, bit for bit the in-place ATen adds, eager and in a CUDA graph.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.spec.m3_spec_offsets import ( # noqa: E501 + m3_spec_offsets as op, +) + +assert torch.cuda.is_available(), "m3_spec_offsets requires a CUDA device" + +I32 = torch.iinfo(torch.int32) + + +def _inputs(tokens, seqs, seed, wrap=False): + g = torch.Generator(device="cuda").manual_seed(seed) + lo, hi = (I32.min, I32.max) if wrap else (0, 1 << 20) + pos = torch.randint(lo, hi, (tokens,), generator=g, device="cuda", dtype=torch.int32) + kv = torch.randint(lo, hi, (seqs,), generator=g, device="cuda", dtype=torch.int32) + pos_off = torch.randint( + -8 if not wrap else lo, + 8 if not wrap else hi, + (tokens,), + generator=g, + device="cuda", + dtype=torch.int32, + ) + kv_off = torch.randint( + -8 if not wrap else lo, + 8 if not wrap else hi, + (seqs,), + generator=g, + device="cuda", + dtype=torch.int32, + ) + if wrap: + pos[:2], pos_off[:2] = I32.max, 1 # overflow wraps, as the ATen add does + kv[:1], kv_off[:1] = I32.min, -1 + return pos, pos_off, kv, kv_off + + +CELLS = [c.spec for c in op.CELLS] + [ + dict(tokens=1, seqs=1), + dict(tokens=300, seqs=75), + dict(tokens=4, seqs=0), +] + + +@pytest.mark.parametrize("spec", CELLS, ids=lambda s: "-".join(f"{k}{v}" for k, v in s.items())) +def test_bitwise_aten_adds(spec) -> None: + for trial in range(3): + pos, pos_off, kv, kv_off = _inputs( + spec["tokens"], spec["seqs"], 97 * trial + spec["tokens"], spec.get("wrap", False) + ) + ref_pos, ref_kv = pos.clone(), kv.clone() + op.reference(ref_pos, pos_off, ref_kv, kv_off) + assert op(pos, pos_off, kv, kv_off) is True + torch.cuda.synchronize() + assert torch.equal(pos, ref_pos) and torch.equal(kv, ref_kv), f"trial {trial}" + + +def test_graph_replays_read_the_live_offsets() -> None: + pos, pos_off, kv, kv_off = _inputs(32, 8, 5) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(pos, pos_off, kv, kv_off) + for step in range(3): + fresh = _inputs(32, 8, 11 + step) + for dst, src in zip((pos, pos_off, kv, kv_off), fresh): + dst.copy_(src) + ref_pos, ref_kv = pos.clone(), kv.clone() + op.reference(ref_pos, pos_off, ref_kv, kv_off) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(pos, ref_pos) and torch.equal(kv, ref_kv), f"step {step}" + + +def test_runner_views_are_accepted() -> None: + """The runner passes position_ids[0, ctx:] and kv_lens_cuda[ctx_reqs:seqs]: 1-D slices of contiguous buffers.""" + position_ids = torch.arange(40, dtype=torch.int32, device="cuda").view(1, 40) + kv_lens = torch.arange(12, dtype=torch.int32, device="cuda") + pos_off = torch.full((40,), 2, dtype=torch.int32, device="cuda") + kv_off = torch.full((12,), 3, dtype=torch.int32, device="cuda") + assert op(position_ids[0, 8:], pos_off[:32], kv_lens[2:10], kv_off[:8]) is True + torch.cuda.synchronize() + assert torch.equal( + position_ids[0], torch.cat([torch.arange(8), torch.arange(8, 40) + 2]).int().cuda() + ) + assert torch.equal( + kv_lens, + torch.cat([torch.arange(2), torch.arange(2, 10) + 3, torch.arange(10, 12)]).int().cuda(), + ) + + +@pytest.mark.parametrize( + "bad", + ["int64", "strided", "length", "2d"], +) +def test_declines_other_layouts_without_writing(bad) -> None: + pos, pos_off, kv, kv_off = _inputs(8, 2, 3) + if bad == "int64": + kv = kv.long() + elif bad == "strided": + pos = torch.zeros(16, dtype=torch.int32, device="cuda")[::2] + elif bad == "length": + pos_off = pos_off[:7] + else: + pos = pos.view(2, 4) + before = (pos.clone(), kv.clone()) + assert op(pos, pos_off, kv, kv_off) is False + torch.cuda.synchronize() + assert torch.equal(pos, before[0]) and torch.equal(kv, before[1]) + + +def test_launches_plainly() -> None: + """It writes step inputs that pre-wait readers take before their grid wait (MSA contract, dispatch + PRODUCER_PREWAIT_STEP_INPUTS): its kernel node carries no programmatic (PDL) serialization.""" + from cuda.bindings import runtime as rt + + pos, pos_off, kv, kv_off = _inputs(32, 8, 7) + graph = torch.cuda.CUDAGraph(keep_graph=True) + with torch.cuda.graph(graph): + op(pos, pos_off, kv, kv_off) + raw = rt.cudaGraph_t(init_value=graph.raw_cuda_graph()) + err, _, n = rt.cudaGraphGetNodes(raw, 0) + err, nodes, n = rt.cudaGraphGetNodes(raw, n) + flags = [] + for node in nodes[:n]: + err, kind = rt.cudaGraphNodeGetType(node) + if kind == rt.cudaGraphNodeType.cudaGraphNodeTypeKernel: + err, attr = rt.cudaGraphKernelNodeGetAttribute( + node, rt.cudaLaunchAttributeID.cudaLaunchAttributeProgrammaticStreamSerialization + ) + flags.append(0 if attr is None else int(attr.programmaticStreamSerializationAllowed)) + assert flags == [0], flags diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_step_gather.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_step_gather.py new file mode 100644 index 000000000000..cd96134bb58b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_step_gather.py @@ -0,0 +1,85 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_step_gather entry (the overlap scheduler's step-input gathers in one launch).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.spec.m3_step_gather import ( # noqa: E501 + m3_step_gather as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import VOCAB, gen, seed_of # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP2 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_step_gather requires a CUDA device" + +#: Rows of the engine's flat buffers (the max batch), and their stale fill. +MAX_ROWS = 8 +STALE = -4 + + +def _case(spec, seed): + g = gen(seed) + p, t, d, nslots, b0 = spec["p"], spec["t"], spec["d"], spec["slots"], spec["b0"] + i32 = dict(dtype=torch.int32, device="cuda") + new_tok = torch.randint(0, VOCAB, (t, nslots, 1), generator=g, device="cuda", dtype=torch.int32) + if spec.get("two_d"): + new_tok = new_tok.squeeze(-1) + next_draft = torch.randint(0, VOCAB, (nslots, d), generator=g, device="cuda", dtype=torch.int32) + lens = torch.randint(1, t + 1, (nslots,), generator=g, device="cuda", dtype=torch.int32) + slots = torch.randperm(nslots, generator=g, device="cuda")[:p].to(torch.int32) + ppi = slots.repeat_interleave(t) + ids = torch.full((MAX_ROWS * t,), STALE, **i32) + draft = torch.full((MAX_ROWS * d,), STALE, **i32) + pos_off = torch.full((MAX_ROWS * t,), STALE, **i32) + kv_off = torch.full((MAX_ROWS,), STALE, **i32) + return ( + new_tok, + next_draft, + lens, + slots, + ppi, + p, + t, + d, + ids, + spec["ids0"], + draft, + spec["d0"], + pos_off, + kv_off, + b0, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Index copies: exact, including the zero-filled offsets and the untouched rest of ids / draft.""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + for name, got in ( + ("ids", args[8]), + ("draft", args[10]), + ("pos_off", args[12]), + ("kv_off", args[13]), + ): + assert torch.equal(got, ref[name]), name + + +def test_guard_refuses_a_slot_past_the_store() -> None: + args = list(_case(op.CELLS[0].spec, 1)) + args[3] = args[3].clone() + args[3][0] = 8 + with pytest.raises(AssertionError, match="slots must index the store"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_step_prologue.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_step_prologue.py new file mode 100644 index 000000000000..c36a223c22c5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/spec/test_modeling_v2_m3_step_prologue.py @@ -0,0 +1,192 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_step_prologue entry (the decode graph's step-input prologue in one launch).""" + +import random +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.spec.m3_step_prologue import ( # noqa: E501 + m3_step_prologue as op, +) +from tensorrt_llm._torch.pyexecutor import step_inputs +from tensorrt_llm._torch.pyexecutor.step_inputs import StepInputPack + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_step_prologue requires a CUDA device" + +MAX_DESC = StepInputPack.MAX_DESC +WINDOW = StepInputPack.GRAPH_WINDOW +DATA_START = StepInputPack.HEADER_BYTES + StepInputPack.DESC_BYTES * MAX_DESC +ARENA = 1 << 20 + + +def seed_of(why: str) -> int: + return zlib.crc32(why.encode()) % 2**31 + + +def gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def _layout(spec, seed): + """Copies as (dst, src, bytes) arena offsets (a host copy's src: a window offset) and the host copies' bytes.""" + rng = random.Random(seed) + h2d, d2d, data = [], [], [] + used = DATA_START + pos = 0 + for _ in range(spec["h2d"]): + n = rng.choice([1, 3, 4, 7, 16, 24, 32, 48, 64, 5, 12]) + pos += rng.randint(0, 40) # gaps and every alignment + h2d.append((pos, used, n)) + data.append(bytes(rng.getrandbits(8) for _ in range(n))) + pos += n + used += -(-n // 16) * 16 + assert pos < ARENA // 2 and used <= WINDOW + # Device copies write the last quarter; unchained ones read the third. + dpos, spos = 3 * ARENA // 4, ARENA // 2 + for i in range(spec["d2d"]): + if spec.get("chained") and h2d: + dst_h, _, n_h = h2d[i % len(h2d)] + off = rng.randint(0, n_h - 1) + src, n = dst_h + off, n_h - off + else: + n = rng.choice([4, 8, 16, 32, 6, 64]) + spos += rng.randint(0, 24) + src = spos + spos += n + dpos += rng.randint(0, 24) + d2d.append((dpos, src, n)) + dpos += n + assert dpos < ARENA and spos < 3 * ARENA // 4 + return h2d, d2d, data + + +def _window(layout, base: int) -> torch.Tensor: + """The window of ``layout`` for an arena at device address ``base``.""" + h2d, d2d, data = layout + w = torch.zeros(WINDOW, dtype=torch.uint8) + words = [len(h2d), len(d2d)] + for dst, src, n in h2d: + words += [base + dst, src, n] + for dst, src, n in d2d: + words += [base + dst, base + src, n] + w[: 8 * len(words)] = torch.tensor(words, dtype=torch.int64).view(torch.uint8) + for (_, off, n), b in zip(h2d, data): + w[off : off + n] = torch.frombuffer(bytearray(b), dtype=torch.uint8) + return w.cuda() + + +def _arenas(seed): + a = torch.randint(0, 256, (ARENA,), generator=gen(seed), device="cuda", dtype=torch.int32) + a = a.to(torch.uint8) + return a, a.clone() + + +@pytest.mark.parametrize("with_epoch", [True, False], ids=["epoch", "no_epoch"]) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, with_epoch, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Byte for byte the default prologue's destinations, header clear and epoch.""" + seed = seed_of(cell.why) + layout = _layout(cell.spec, seed) + arena, ref_arena = _arenas(seed) + window, ref_window = _window(layout, arena.data_ptr()), _window(layout, ref_arena.data_ptr()) + epoch = torch.full((1,), 41, dtype=torch.int64, device="cuda") if with_epoch else None + ref_epoch = epoch.clone() if with_epoch else None + certifying(op) + op(window, epoch, MAX_DESC) + op.reference(ref_window, ref_epoch, MAX_DESC) + torch.cuda.synchronize() + assert torch.equal(arena, ref_arena) + # Each window holds its own arena's addresses: past the consumed header, both are left as staged. + for w, a in ((window, arena), (ref_window, ref_arena)): + assert not w[:16].any(), "the header is consumed" + assert torch.equal(w[16:], _window(layout, a.data_ptr())[16:]) + if with_epoch: + assert int(epoch.item()) == int(ref_epoch.item()) == 42 + + +def test_graph_replays_scatter_each_new_window() -> None: + """Captured once on a static window: every replay writes that replay's window and bumps the epoch once.""" + arena, ref_arena = _arenas(7) + window = torch.zeros(WINDOW, dtype=torch.uint8, device="cuda") + epoch = torch.zeros(1, dtype=torch.int64, device="cuda") + ref_epoch = epoch.clone() + op(window, epoch, MAX_DESC) # the empty window: writes nothing + op.reference(window.clone(), ref_epoch, MAX_DESC) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(window, epoch, MAX_DESC) + for step, spec in enumerate( + [dict(h2d=24, d2d=8, chained=True), dict(h2d=0, d2d=0), dict(h2d=40, d2d=4)] + ): + layout = _layout(spec, 100 + step) + window.copy_(_window(layout, arena.data_ptr())) + graph.replay() + op.reference(_window(layout, ref_arena.data_ptr()), ref_epoch, MAX_DESC) + torch.cuda.synchronize() + assert torch.equal(arena, ref_arena), step + assert not window[:16].any(), step + assert int(epoch.item()) == int(ref_epoch.item()) == step + 2, step + + +def _pack_case(prologue_op): + pack = StepInputPack( + torch.device("cuda", torch.cuda.current_device()), + parallel_scatter=True, + in_graph=True, + early_copy=True, + step_epoch=True, + prologue_op=prologue_op, + ) + a, b, c = (torch.zeros(16, dtype=torch.int32, device="cuda") for _ in range(3)) + out = torch.zeros(48, dtype=torch.int32, device="cuda") + graph = torch.cuda.CUDAGraph() + with step_inputs.active(pack): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + assert step_inputs.capture_prologue() is pack + out.copy_(torch.cat([a, b, c])) + return pack, graph, (a, b, c), out + + +def _pack_step(pack, graph, dsts, values, overlap=False, flush=True): + a, b, c = dsts + + def host(v): + return torch.tensor(v, dtype=torch.int32).pin_memory() + + with step_inputs.active(pack): + step_inputs.begin() + step_inputs.copy_(a, host(values[0])) + step_inputs.copy_(b, host(values[1])) + if overlap: + step_inputs.copy_(a[:4], host([-1, -2, -3, -4])) + step_inputs.copy_(c, a) # a device copy reading a host copy's destination + if not flush: + step_inputs.discard() + step_inputs.flush(in_graph=True) + graph.replay() + + +def test_pack_with_the_one_launch_prologue_matches_the_default() -> None: + """The same pack steps through the default prologue and the one-launch one: the same outputs and epochs, and the + one-launch prologue is engaged (the default's warm-up compile is not what bumped its epoch).""" + packs = [_pack_case(None), _pack_case(op)] + assert packs[0][0].prologue_op is None and packs[1][0].prologue_op is op + for step in range(6): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + outs = [] + for pack, graph, dsts, out in packs: + # A replay with a new pack, one with an eagerly issued pack, and one with nothing to scatter. + _pack_step(pack, graph, dsts, (va, vb), overlap=step == 2, flush=step != 3) + torch.cuda.synchronize() + outs.append((out.tolist(), int(pack.epoch.item()))) + assert outs[0] == outs[1], step + assert outs[1][1] == step + 2, step # one bump at warm-up, then one per replay diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_agg_qkv_a1.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_agg_qkv_a1.py new file mode 100644 index 000000000000..5c0229e3de66 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_agg_qkv_a1.py @@ -0,0 +1,77 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2AGG_QKV_A1 dispatch: which steps take the A1 record, that it never touches the CTXNORMQ slot when it declines +before the take, and that its placeholder carries the packed shape without storage. The kernel's numerics are the +tp2ctx A1 UTs and the agg MB (rule 1b).""" + +import types + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as m, +) + + +def _core(): + return types.SimpleNamespace( + heads=32, + kv_heads=2, + index_heads=2, + mxfp8=None, + w={"l3_qkv_w": torch.zeros(4992, 64), "l3_qkv_s": torch.zeros(1)}, + ) + + +@pytest.fixture +def a1_on(monkeypatch): + monkeypatch.setattr(m, "QKV_A1", True) + monkeypatch.setattr(m, "CTXNORMQ", True) + mode = types.SimpleNamespace(decode_graph=False, tuned_quantize_gemm=False) + monkeypatch.setattr(m, "_mx_mode", lambda _: mode) + monkeypatch.setattr(m, "is_torch_compiling", lambda: False) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + return mode + + +def test_off_by_default_without_opt_in() -> None: + """Rule 1b until its gates pass: the process takes A1 only with TRTLLM_M31_ENABLE_TP2AGG_QKV_A1=1.""" + assert m.QKV_A1 == (m.m31_enabled("TP2AGG_QKV_A1") and not m.m31_disabled("TP2AGG_QKV_A1")) + + +@pytest.mark.parametrize("rows", [1, 1024]) +def test_small_steps_keep_the_slot(a1_on, rows) -> None: + hidden = torch.zeros(rows, 64, dtype=torch.bfloat16) + m._CTXNORMQ_MX.publish(hidden, "act", "sf") + assert m._a1_defer(_core(), 3, hidden) is None + assert m._CTXNORMQ_MX.pending is not None + m._CTXNORMQ_MX.clear() + + +def test_takes_the_published_activation(a1_on) -> None: + hidden = torch.zeros(1025, 64, dtype=torch.bfloat16) + act = torch.zeros(1025, 64, dtype=torch.uint8) + m._CTXNORMQ_MX.publish(hidden, act, "sf") + rec = m._a1_defer(_core(), 3, hidden) + assert isinstance(rec, m._DeferredA1) + assert rec.act is act and rec.act_scale == "sf" + assert m._CTXNORMQ_MX.pending is None + assert rec.placeholder.shape == (1025, 4992) and rec.placeholder.dtype == torch.bfloat16 + assert rec.placeholder.untyped_storage().nbytes() == 2 + + +def test_unpublished_or_other_tensor_declines(a1_on) -> None: + hidden = torch.zeros(2048, 64, dtype=torch.bfloat16) + assert m._a1_defer(_core(), 3, hidden) is None + m._CTXNORMQ_MX.publish(hidden.clone(), "act", "sf") + assert m._a1_defer(_core(), 3, hidden) is None + + +@pytest.mark.parametrize("field", ["decode_graph", "tuned_quantize_gemm"]) +def test_modes_without_the_ctxnormq_gemm_decline(a1_on, field) -> None: + setattr(a1_on, field, True) + hidden = torch.zeros(2048, 64, dtype=torch.bfloat16) + m._CTXNORMQ_MX.publish(hidden, "act", "sf") + assert m._a1_defer(_core(), 3, hidden) is None + m._CTXNORMQ_MX.clear() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_ar_det.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_ar_det.py new file mode 100644 index 000000000000..584f4de7e5e7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_ar_det.py @@ -0,0 +1,49 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""AR-DET: the target defaults the AllReduce autotune off for its process (the C++ heuristic), keeps an explicit +TLLM_DISABLE_ALLREDUCE_AUTOTUNE as given, and leaves the autotune on with TRTLLM_M31_ENABLE_AR_AUTOTUNE=1.""" + +import os + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) + +KEY = "TLLM_DISABLE_ALLREDUCE_AUTOTUNE" + + +def test_autotune_defaults_off(monkeypatch) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_report, + ) + + m3_report.reset_work() + monkeypatch.setattr(os, "environ", {}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", False) + assert modeling._pin_allreduce_heuristic(0) is True + assert os.environ[KEY] == "1" + assert m3_report.work_count("AR_DET") == 1 # gate e: the setting counts once per process + + +def test_escape_hatch_keeps_autotune_on(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", True) + assert modeling._pin_allreduce_heuristic(0) is False + assert KEY not in os.environ + + +def test_explicit_setting_is_kept(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {KEY: "0"}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", False) + assert modeling._pin_allreduce_heuristic(0) is False + assert os.environ[KEY] == "0" + + +def test_escape_hatch_follows_the_switch_contract() -> None: + """TRTLLM_M31_ENABLE_AR_AUTOTUNE is parsed by m31_switches (1/true/on/yes set; ""/0/false/off/no unset), not by + a separate loose read.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + ) + + assert modeling.AR_AUTOTUNE_ENABLED == m31_switches.m31_enabled("AR_AUTOTUNE") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_blockoff_guard.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_blockoff_guard.py new file mode 100644 index 000000000000..f8ccf9972a70 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_blockoff_guard.py @@ -0,0 +1,165 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""BLOCKOFF_GUARD on this target: ``_runtime_hooks`` turns on RuntimeHooks.skip_unchanged_draft_block_offsets by default +and ``TRTLLM_M31_DISABLE_BLOCKOFF_GUARD`` turns it off. The constant is evaluated from modeling.py's source against a +private copy of m31_switches loaded under each environment, so the heavy modeling module is not reloaded. CPU only, +except the last test, which imports the in-tree module. +""" + +import ast +import importlib +import importlib.util +import os +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp2" +CONSTANT = "STEP_BLOCKOFF_GUARD" +FIELD = "skip_unchanged_draft_block_offsets" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + + +def _module_tree() -> ast.Module: + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _value(monkeypatch, env) -> bool: + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_blockoff_{id(env)}", _SWITCHES) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + statements = [ + node + for node in _module_tree().body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id == CONSTANT + ] + assert len(statements) == 1 + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + return namespace[CONSTANT] + + +@pytest.mark.cpu_only +def test_on_by_default(monkeypatch): + assert _value(monkeypatch, {}) is True + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("value", ["1", "true", "ON", " yes "]) +def test_the_kill_switch_turns_it_off(monkeypatch, value): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_BLOCKOFF_GUARD": value}) is False + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("value", ["", "0", "false", "off", "no"]) +def test_an_unset_kill_switch_keeps_it(monkeypatch, value): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_BLOCKOFF_GUARD": value}) is True + + +@pytest.mark.cpu_only +def test_other_switches_leave_it_on(monkeypatch): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_HOSTTAIL": "1"}) is True + + +@pytest.mark.cpu_only +def test_runtime_hooks_passes_the_constant(): + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + calls = [ + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1 + keywords = {kw.arg: kw.value for kw in calls[0].keywords} + assert isinstance(keywords.get(FIELD), ast.Name) and keywords[FIELD].id == CONSTANT + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="imports the target's modeling module") +def test_the_in_tree_hook_follows_the_switch(): + modeling = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.modeling" + ) + hooks = modeling._runtime_hooks(None) + disabled = os.environ.get("TRTLLM_M31_DISABLE_BLOCKOFF_GUARD", "").strip().lower() in ( + "1", + "true", + "on", + "yes", + ) + assert getattr(hooks, FIELD) is getattr(modeling, CONSTANT) is (not disabled) + assert hooks.skips_unchanged_metadata_uploads + + +def _patch_value(monkeypatch, env) -> bool: + """STEP_BLOCKOFF_PATCH as modeling.py computes it under ``env`` (it needs BLOCKOFF_GUARD).""" + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location( + f"_m31_switches_blockoff_patch_{id(env)}", _SWITCHES + ) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + statements = [ + node + for node in _module_tree().body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id in (CONSTANT, "STEP_BLOCKOFF_PATCH") + ] + assert [node.targets[0].id for node in statements] == [CONSTANT, "STEP_BLOCKOFF_PATCH"] + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + return namespace["STEP_BLOCKOFF_PATCH"] + + +@pytest.mark.cpu_only +def test_the_patch_is_on_by_default(monkeypatch): + assert _patch_value(monkeypatch, {}) is True + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", ["BLOCKOFF_PATCH", "BLOCKOFF_GUARD"]) +def test_the_patch_kill_switch_and_the_guard_kill_switch_turn_it_off(monkeypatch, item): + assert _patch_value(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: "1"}) is False + + +@pytest.mark.cpu_only +def test_runtime_hooks_passes_the_patch_constant(): + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + call = next( + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ) + keywords = {kw.arg: kw.value for kw in call.keywords} + field = keywords.get("patch_unchanged_draft_block_offsets") + assert isinstance(field, ast.Name) and field.id == "STEP_BLOCKOFF_PATCH" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_draft_a1.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_draft_a1.py new file mode 100644 index 000000000000..c730619a31c0 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_draft_a1.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""QA-only draft-head routing checks; multi-rank numerics require the separate GPU gate.""" + +from types import SimpleNamespace + +import pytest +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + dspark as d, +) + + +class Inner(nn.Module): + def __init__(self): + super().__init__() + self.calls = [] + + def forward(self, *args): + self.calls.append(args) + return "stock" + + +def make_wrapper(monkeypatch): + inner = Inner() + calls = [] + head = SimpleNamespace(apply_linear=lambda x, bias: calls.append((x, bias)) or x) + monkeypatch.setattr(d._allgather_cast, "allgather_cast", lambda x, **ws: (x, ws)) + monkeypatch.setattr(d.m3_report, "work", lambda *args: None) + return d._DraftA1Logits(inner, head, {"world": 2}, (66, 84, 154, 196)), head, calls + + +@pytest.mark.parametrize("rows", [66, 84, 154, 196]) +def test_compiled_draft_shape_uses_head_and_existing_workspace(monkeypatch, rows): + wrapper, head, calls = make_wrapper(monkeypatch) + hidden = torch.empty((rows, 8), dtype=torch.bfloat16) + result = wrapper(hidden, head, SimpleNamespace(num_contexts=0), True) + assert result[0] is hidden and result[1] == wrapper._workspace + assert calls == [(hidden, None)] + assert not wrapper.inner.calls + + +@pytest.mark.parametrize( + "reason", ["small", "uncompiled", "context", "last_token", "other_head", "dtype"] +) +def test_declines_before_local_head(monkeypatch, reason): + wrapper, head, calls = make_wrapper(monkeypatch) + rows = 64 if reason == "small" else (67 if reason == "uncompiled" else 66) + hidden = torch.empty((rows, 8), dtype=torch.float32 if reason == "dtype" else torch.bfloat16) + metadata = SimpleNamespace(num_contexts=int(reason == "context")) + supplied_head = object() if reason == "other_head" else head + result = wrapper(hidden, supplied_head, metadata, reason != "last_token") + assert result == "stock" and len(wrapper.inner.calls) == 1 + assert not calls + + +def test_default_on_and_kill_policy(): + assert d.DRAFT_A1 == (not d.m31_disabled("DRAFT_A1")) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_dspark_draft_ar.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_dspark_draft_ar.py new file mode 100644 index 000000000000..6d227030e517 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_dspark_draft_ar.py @@ -0,0 +1,148 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""S9b-VR-a: the DSpark drafter's row-parallel all-reduces take the one-shot all-reduce up to +``DRAFT_AR_ONESHOT_MAX_ROWS`` rows and keep their AUTO all-reduce above (``dspark.install_draft_oneshot_allreduce``). + +The all-reduce and linear classes are stand-ins (the routing is the unit under test; the one-shot kernel has its own +tests), so this runs on the CPU without a process group. TP2: on by default with the TP8 drafter +levers; the kill switch TRTLLM_M31_DISABLE_S9BVR follows the m31_switches value contract (strict parse); each one-shot +call counts the S9BVR work counter. +""" + +import importlib +import os +from types import SimpleNamespace + +import pytest +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark, + m31_switches, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_report, +) + +Strategy = dspark.AllReduceStrategy + + +class FakeAllReduce(nn.Module): + """Records each call; ``strategy`` and ``mapping`` as the real module exposes them.""" + + def __init__(self, mapping, strategy=Strategy.AUTO, dtype=None): + super().__init__() + self.mapping = mapping + self.strategy = strategy + self.dtype = dtype + self.calls = [] + + def uses_nccl_symmetric_memory_window(self): + return self.strategy == Strategy.AUTO + + def forward(self, input, *, all_reduce_params=None): + self.calls.append((input.shape[0], all_reduce_params)) + return input + 1 + + +class FakeLinear(nn.Module): + def __init__(self, tp_mode, all_reduce, dtype=torch.bfloat16): + super().__init__() + self.tp_mode = tp_mode + self.all_reduce = all_reduce + self.dtype = dtype + + +@pytest.fixture +def fakes(monkeypatch): + monkeypatch.setattr(dspark, "AllReduce", FakeAllReduce) + monkeypatch.setattr(dspark, "Linear", FakeLinear) + m3_report.reset_work() + return SimpleNamespace(mapping=SimpleNamespace(tp_size=2, tp_rank=0)) + + +def _drafter(mapping): + ROW, COL = dspark.TensorParallelMode.ROW, dspark.TensorParallelMode.COLUMN + d = nn.Module() + d.o_proj = FakeLinear(ROW, FakeAllReduce(mapping)) + d.down_proj = FakeLinear(ROW, FakeAllReduce(mapping)) + d.qkv_proj = FakeLinear(COL, None) + d.nccl_row = FakeLinear( + ROW, FakeAllReduce(mapping, Strategy.NCCL) + ) # an explicit strategy is left alone + d.no_reduce = FakeLinear(ROW, None) + return d + + +def test_wraps_only_row_parallel_auto_linears(fakes): + d = _drafter(fakes.mapping) + assert dspark.install_draft_oneshot_allreduce(d, 32) == 2 + assert isinstance(d.o_proj.all_reduce, dspark.DraftOneshotAllReduce) + assert isinstance(d.down_proj.all_reduce, dspark.DraftOneshotAllReduce) + assert isinstance(d.nccl_row.all_reduce, FakeAllReduce) + assert d.qkv_proj.all_reduce is None and d.no_reduce.all_reduce is None + # the one-shot module is the explicit ONESHOT strategy, never TWOSHOT + assert d.o_proj.all_reduce.oneshot.strategy == Strategy.ONESHOT + + +@pytest.mark.parametrize( + "rows, takes_oneshot", [(1, True), (4, True), (32, True), (33, False), (4096, False)] +) +def test_routes_by_rows(fakes, rows, takes_oneshot): + d = _drafter(fakes.mapping) + dspark.install_draft_oneshot_allreduce(d, 32) + w = d.o_proj.all_reduce + params = object() + out = w(torch.zeros(rows, 8), all_reduce_params=params) + assert torch.equal(out, torch.ones(rows, 8)) + taken, other = (w.oneshot, w.auto) if takes_oneshot else (w.auto, w.oneshot) + assert taken.calls == [(rows, params)] and other.calls == [] + assert m3_report.work_count("S9BVR") == (1 if takes_oneshot else 0) + + +def test_output_buffer_choice_stays_the_original(fakes): + d = _drafter(fakes.mapping) + dspark.install_draft_oneshot_allreduce(d, 32) + assert ( + d.o_proj.all_reduce.uses_nccl_symmetric_memory_window() is True + ) # the AUTO module's answer + + +def test_zero_rows_installs_nothing(fakes): + d = _drafter(fakes.mapping) + assert dspark.install_draft_oneshot_allreduce(d, 0) == 0 + assert isinstance(d.o_proj.all_reduce, FakeAllReduce) + + +def test_default_on_at_32_rows(): + assert dspark.S9BVR is (not m31_switches.m31_disabled("S9BVR")) + assert dspark.DRAFT_AR_ONESHOT_MAX_ROWS == (32 if dspark.S9BVR else 0) + + +@pytest.mark.parametrize("value, expected", [("1", 0), ("on", 0), ("0", 32), ("", 32), ("off", 32)]) +def test_kill_switch(monkeypatch, value, expected): + """The switch is read through m31_switches at import (reloaded here with dspark).""" + monkeypatch.setenv("TRTLLM_M31_DISABLE_S9BVR", value) + try: + importlib.reload(m31_switches) + mod = importlib.reload(dspark) + assert mod.S9BVR is (expected == 32) + assert mod.DRAFT_AR_ONESHOT_MAX_ROWS == expected + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_S9BVR", raising=False) + importlib.reload(m31_switches) + importlib.reload(dspark) + assert os.environ.get("TRTLLM_M31_DISABLE_S9BVR") is None + + +def test_kill_switch_rejects_a_bad_value(monkeypatch): + """A typo fails loudly at import (m31_switches' strict parse) instead of silently leaving the item on.""" + monkeypatch.setenv("TRTLLM_M31_DISABLE_S9BVR", "maybe") + try: + with pytest.raises(ValueError, match="TRTLLM_M31_DISABLE_S9BVR"): + importlib.reload(m31_switches) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_S9BVR", raising=False) + importlib.reload(m31_switches) + importlib.reload(dspark) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_hook_work_counters.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_hook_work_counters.py new file mode 100644 index 000000000000..7ba42407e345 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_hook_work_counters.py @@ -0,0 +1,383 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""runtime_hooks.hook_work: the log-only work counters of the shared step-loop items. + +* The counter: the first call logs "[M3 host] engaged: work 1 (...)", every power of two logs the running count, + nothing else is logged, the exit hook is registered once, and the final lines ("[M3 host] engaged: final work + N") are logged once per process. Every line says the count is host executions (graph replays not counted), and the + lines match the engagement patterns. +* The call sites: each of the seven items counts only on the path its RuntimeHooks flag takes, never when the flag is + off (the sampler, metadata and H2D items run on small stubs; the three DecoderRunner items are checked at their + source: each count sits under its own flag). CPU only. +""" + +import ast +import inspect +import re +from contextlib import contextmanager +from types import SimpleNamespace + +import pytest +import torch + +import tensorrt_llm._utils as trtllm_utils +from tensorrt_llm._torch.pyexecutor import h2d_cache +from tensorrt_llm._torch.pyexecutor import runtime_hooks as rh +from tensorrt_llm._torch.pyexecutor.engine.runners.decoder import runner as runner_module +from tensorrt_llm._torch.pyexecutor.h2d_cache import H2DSkipCache +from tensorrt_llm._torch.pyexecutor.runtime_hooks import hook_work +from tensorrt_llm._torch.speculative.dflash import DFlashSpecMetadata +from tensorrt_llm._torch.speculative.interface import SpecMetadata +from tensorrt_llm._torch.speculative.spec_sampler_base import SpecSampler +from tensorrt_llm.logger import logger + +pytestmark = pytest.mark.cpu_only + +ITEMS = ( + "HOST_D2H_SIDE", + "SMALL_H2D", + "PHILOX_PACK", + "KV_LENS_ONCE", + "H2D_CHANGED_SPAN", + "BATCH_CHANGED_ONLY", + "SKIP_ACCEPTED_UPLOAD", +) + + +class _Lines(list): + """The logged lines, plus the functions hook_work registered at exit.""" + + registered: list + + +@pytest.fixture +def lines(monkeypatch): + """Every line hook_work logs, with rank 0, fresh counters and the exit registrations recorded instead of made.""" + out = _Lines() + registered = [] + monkeypatch.setattr(logger, "info", out.append) + monkeypatch.setattr(trtllm_utils, "mpi_rank", lambda: 0) + monkeypatch.setattr(rh.atexit, "register", registered.append) + monkeypatch.setattr(rh, "HOOK_WORK_COUNTS", {}) + monkeypatch.setattr(rh, "_hook_work_exit", [False, False]) + out.registered = registered + return out + + +def _line(text: str) -> str: + return f"{text} (rank 0; {rh.HOOK_WORK_HOST_COUNT})" + + +# --------------------------------------------------------------------------- +# The counter +# --------------------------------------------------------------------------- + + +def test_first_call_and_powers_of_two_log_the_count(lines): + for _ in range(9): + hook_work("UT_ITEM") + assert lines == [ + _line("[M3 host] UT_ITEM engaged: work 1"), + _line("[M3 host] UT_ITEM engaged: work 2"), + _line("[M3 host] UT_ITEM engaged: work 4"), + _line("[M3 host] UT_ITEM engaged: work 8"), + ] + assert rh.HOOK_WORK_COUNTS == {"UT_ITEM": 9} + assert "graph replays not counted" in rh.HOOK_WORK_HOST_COUNT + + +def test_items_count_independently(lines): + hook_work("X1") + hook_work("X2") + hook_work("X1") + assert rh.HOOK_WORK_COUNTS == {"X1": 2, "X2": 1} + + +def test_the_final_lines_are_logged_once_per_process(lines): + for item, n in (("B_ITEM", 3), ("A_ITEM", 1)): + for _ in range(n): + hook_work(item) + assert lines.registered == [rh.log_hook_work_final] + lines.clear() + rh.log_hook_work_final() + rh.log_hook_work_final() + assert lines == [ + _line("[M3 host] A_ITEM engaged: final work 1"), + _line("[M3 host] B_ITEM engaged: final work 3"), + ] + + +def test_nothing_counted_logs_nothing_at_exit(lines): + rh.log_hook_work_final() + assert lines == [] and lines.registered == [] + + +@pytest.mark.parametrize("item", ITEMS) +def test_the_lines_match_the_eng_kit(lines, item): + """eng_tp2_v4's first and final patterns for each item.""" + hook_work(item) + rh.log_hook_work_final() + first, final = lines + assert re.search(rf"\[M3 host\] {item} engaged: work 1 \(", first) + assert re.search(rf"\[M3 host\] {item} engaged: final work [0-9]+", final) + assert not re.search(rf"\[M3 host\] {item} engaged: final work", first) + + +# --------------------------------------------------------------------------- +# HOST_D2H_SIDE: SpecSampler._copy_outputs_to_host +# --------------------------------------------------------------------------- + + +class _Copier: + event = None + + def stage_copy_to_host(self, t): + return t + + +def _sampler(**attrs): + @contextmanager + def copier(): + yield _Copier() + + store = SimpleNamespace( + new_tokens=torch.zeros(2), new_tokens_lens=torch.zeros(2), next_draft_tokens=torch.zeros(2) + ) + return SimpleNamespace( + store=store, + _copy_to_host=lambda t: t, + _make_side_stream_copier=copier, + _async_worker_active=lambda: False, + **attrs, + ) + + +@pytest.mark.parametrize( + "flag,on_side_stream,counted", [(True, True, 1), (True, False, 0), (False, True, 0)] +) +def test_host_d2h_side_counts_the_side_stream_copies(lines, flag, on_side_stream, counted): + sampler = _sampler(d2h_side_stream_copies=flag) + SpecSampler._copy_outputs_to_host(sampler, None, store_copies_on_side_stream=on_side_stream) + assert rh.HOOK_WORK_COUNTS.get("HOST_D2H_SIDE", 0) == counted + + +# --------------------------------------------------------------------------- +# SMALL_H2D: the sampler's static slot ids (SpecSampler._store_outputs) +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("flag", [True, False]) +def test_small_h2d_counts_the_static_slot_ids(lines, monkeypatch, flag): + # A fold that takes the store, so _store_outputs returns right after the slot ids. + fold = SimpleNamespace(scatter=lambda *args: True) + sampler = _sampler( + static_slot_ids=flag, spec_step_fold=fold, _device_slot_ids=lambda slots: "device" + ) + # The stock path uploads to "cuda"; keep it on the host here. + monkeypatch.setattr( + torch, "as_tensor", lambda *args, **kwargs: SimpleNamespace(to=lambda **kw: "uploaded") + ) + SpecSampler._store_outputs(sampler, [0, 1], None, None, None, None) + assert rh.HOOK_WORK_COUNTS.get("SMALL_H2D", 0) == int(flag) + + +# --------------------------------------------------------------------------- +# SKIP_ACCEPTED_UPLOAD, SMALL_H2D (accepted counts packed) and KV_LENS_ONCE: DecoderRunner +# --------------------------------------------------------------------------- + + +def _guarding_flags(module, item: str) -> list[set]: + """For each ``hook_work(item)`` call in ``module``: the ``self.runtime_hooks.`` names its enclosing ``if`` + tests (the branch the count sits on).""" + tree = ast.parse(inspect.getsource(module)) + parents = {child: node for node in ast.walk(tree) for child in ast.iter_child_nodes(node)} + found = [] + for node in ast.walk(tree): + if not ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "hook_work" + and node.args[0].value == item + ): + continue + flags, child, parent = set(), node, parents.get(node) + while parent is not None: + if isinstance(parent, ast.If) and child in parent.body: + flags |= { + a.attr + for a in ast.walk(parent.test) + if isinstance(a, ast.Attribute) + and isinstance(a.value, ast.Attribute) + and a.value.attr == "runtime_hooks" + } + child, parent = parent, parents.get(parent) + found.append(flags) + return found + + +@pytest.mark.parametrize( + "item,flag", + [ + ("SKIP_ACCEPTED_UPLOAD", "skip_accepted_draft_upload"), + ("SMALL_H2D", "spec_small_h2d"), + ("KV_LENS_ONCE", "kv_lens_update_once"), + ], +) +def test_the_runner_items_count_only_under_their_flag(item, flag): + (flags,) = _guarding_flags(runner_module, item) + assert flag in flags, (item, flags) + + +# --------------------------------------------------------------------------- +# H2D_CHANGED_SPAN: H2DSkipCache._copy_changed_span +# --------------------------------------------------------------------------- + + +@pytest.fixture +def staged(monkeypatch): + copies = [] + monkeypatch.setattr(h2d_cache.step_inputs, "is_open", lambda: True) + monkeypatch.setattr( + h2d_cache.step_inputs, "copy_", lambda dst, src: copies.append(src.tolist()) + ) + return copies + + +def _span(dst: torch.Tensor, old: torch.Tensor, src: torch.Tensor) -> bool: + key = (dst.storage_offset(), tuple(dst.shape), tuple(dst.stride()), dst.dtype) + return H2DSkipCache(changed_span=True)._copy_changed_span(dst, src, (key, old)) + + +def test_changed_span_counts_a_staged_span(lines, staged): + dst = torch.zeros(4, 16, dtype=torch.int32) + old = torch.zeros(4, 16, dtype=torch.int32) + src = old.clone() + src[1, 3] = 7 + assert _span(dst, old, src) is True + assert staged == [[7]] + assert rh.HOOK_WORK_COUNTS == {"H2D_CHANGED_SPAN": 1} + + +def test_changed_span_does_not_count_a_declined_span(lines, staged): + dst = torch.zeros(2, 4, dtype=torch.int32) + old = torch.zeros(2, 4, dtype=torch.int32) + src = torch.ones(2, 4, dtype=torch.int32) # every column changed: over the fraction + assert _span(dst, old, src) is False + assert staged == [] and rh.HOOK_WORK_COUNTS == {} + + +def test_without_the_flag_the_span_is_never_tried(): + """The span path (the only caller of the counter in h2d_cache) is reached only with changed_span on.""" + src = inspect.getsource(H2DSkipCache.copy) + assert "and self.changed_span" in src and "self._copy_changed_span(" in src + assert H2DSkipCache().changed_span is False + + +# --------------------------------------------------------------------------- +# PHILOX_PACK: SpecMetadata._populate_request_rng_state +# --------------------------------------------------------------------------- + + +def _rng_meta(pack: bool, greedy_skip: bool = False) -> SpecMetadata: + meta = SpecMetadata(max_num_requests=4, max_draft_len=3, max_total_draft_tokens=3) + meta.pack_rng_uploads = pack + meta.skip_greedy_rng_upload = greedy_skip + # Host buffers large enough that nothing is allocated on the device. + for name in ("request_seeds", "request_offsets", "seeds", "offsets"): + setattr(meta, name, torch.zeros(16, dtype=torch.int64)) + return meta + + +def _rng_request(slot: int, seed): + return SimpleNamespace( + py_seq_slot=slot, + py_decoding_iter=0, + py_request_id=slot, + sampling_config=SimpleNamespace(seed=seed), + ) + + +@pytest.mark.parametrize( + "pack,greedy_skip,all_greedy,counted", + [(True, False, False, 1), (False, False, False, 0), (True, True, True, 0)], + ids=["packed", "flag off", "all-greedy skip"], +) +def test_philox_pack_counts_a_packed_step(lines, pack, greedy_skip, all_greedy, counted): + meta = _rng_meta(pack, greedy_skip) + meta.is_all_greedy_sample = all_greedy + requests = [_rng_request(2, 1234), _rng_request(0, None)] + meta._populate_request_rng_state(requests, [(0.7, 0, 0.9, 0.0, 4) for _ in requests]) + assert rh.HOOK_WORK_COUNTS.get("PHILOX_PACK", 0) == counted + + +# --------------------------------------------------------------------------- +# BATCH_CHANGED_ONLY: DFlashSpecMetadata.prepare +# --------------------------------------------------------------------------- + + +def _dflash_meta(changed_only: bool, worker=None): + return SimpleNamespace( + request_ids=[11, 12], + num_generations=2, + batch_map_changed_only=changed_only, + _batch_indices_uploaded=0, + batch_indices_cuda=torch.full((8,), -5, dtype=torch.int), + _dflash_worker=worker, + ) + + +def _dflash_worker(): + w = SimpleNamespace( + _ctx_buf_inited=True, + _req_to_slot={11: 0, 12: 1}, + _req_ctx_pos={}, + _free_slots=[], + _graph_dummy_id_floor=1 << 40, + _dummy_slot=8, + _batch_to_slot=torch.full((8,), -7, dtype=torch.long), + _batch_to_slot_uploaded=None, + _write_ctx_len=lambda evicted: None, + ) + return w + + +@pytest.mark.parametrize("changed_only", [True, False]) +def test_batch_changed_only_counts_each_skipped_upload(lines, changed_only): + md = _dflash_meta(changed_only, _dflash_worker()) + DFlashSpecMetadata.prepare(md) # first step: both maps uploaded + assert rh.HOOK_WORK_COUNTS.get("BATCH_CHANGED_ONLY", 0) == 0 + DFlashSpecMetadata.prepare(md) # steady: both skipped with the flag + assert rh.HOOK_WORK_COUNTS.get("BATCH_CHANGED_ONLY", 0) == (2 if changed_only else 0) + assert md.batch_indices_cuda[:2].tolist() == [0, 1] + assert md._dflash_worker._batch_to_slot[:2].tolist() == [0, 1] + + +# --------------------------------------------------------------------------- +# Every item has exactly its call sites (the tests above check each one's branch). +# --------------------------------------------------------------------------- + + +def _hook_work_items(module) -> list[str]: + tree = ast.parse(inspect.getsource(module)) + return sorted( + node.args[0].value + for node in ast.walk(tree) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "hook_work" + ) + + +def test_the_seven_items_have_their_call_sites(): + from tensorrt_llm._torch.speculative import dflash, interface, spec_sampler_base + + sites = { + h2d_cache: ["H2D_CHANGED_SPAN"], + runner_module: ["KV_LENS_ONCE", "SKIP_ACCEPTED_UPLOAD", "SMALL_H2D"], + spec_sampler_base: ["HOST_D2H_SIDE", "SMALL_H2D"], + interface: ["PHILOX_PACK"], + dflash: ["BATCH_CHANGED_ONLY", "BATCH_CHANGED_ONLY"], + } + for module, items in sites.items(): + assert _hook_work_items(module) == items, module.__name__ + assert {item for items in sites.values() for item in items} == set(ITEMS) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_host_tail_sm107_tp2.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_host_tail_sm107_tp2.py new file mode 100644 index 000000000000..3315d367daa3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_host_tail_sm107_tp2.py @@ -0,0 +1,274 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The kill switches of this target's host-side step-loop items: ``_runtime_hooks`` turns on all ten RuntimeHooks fields +in ITEMS by default; ``TRTLLM_M31_DISABLE_`` turns off its own field (and, for the step-input pack, the fields that +need it: PACK_IN_GRAPH needs PACK_PSCATTER, PACK_EARLY_COPY needs PACK_IN_GRAPH); ``TRTLLM_M31_DISABLE_HOSTTAIL`` +turns off the six items of the sm_103 targets' HOSTTAIL bundle. + +The ``STEP_*`` expressions are evaluated from modeling.py's source against a private copy of m31_switches loaded +under each environment (as test_modeling_v2_m31_switches_sm107_tp2 does), so neither the in-tree switches nor the +heavy modeling module is reloaded. CPU only, except the last test, which imports the in-tree module. + +The TP2 target carries the TP4 base's step-loop block (some of these constants first defined there, without the bundle +switch) and, after it, the TP8 HOSTTAIL block that redefines them; module order makes the later definitions the +values ``_runtime_hooks`` reads, and the source is executed here in that same order. +""" + +import ast +import importlib +import importlib.util +import os +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.cache_manager import ( + MiniMaxM3KVCacheManagerV2, +) +from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import ( + BaseMambaCacheManager, + MambaHybridCacheManager, +) +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + +TARGET = "minimax_m31_nvfp4__sm_107__tp2" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + +# Kill-switch id -> (the module constant, the RuntimeHooks field it sets). +ITEMS = { + "SKIP_ACCEPTED_UPLOAD": ("STEP_SKIP_ACCEPTED_UPLOAD", "skip_accepted_draft_upload"), + "BATCH_CHANGED_ONLY": ("STEP_BATCH_CHANGED_ONLY", "spec_batch_map_changed_only"), + "HOST_D2H_SIDE": ("STEP_HOST_D2H_SIDE", "sampler_d2h_side_stream"), + "SMALL_H2D": ("STEP_SMALL_H2D", "spec_small_h2d"), + "PHILOX_PACK": ("STEP_PHILOX_PACK", "pack_spec_rng_uploads"), + "KV_LENS_ONCE": ("STEP_KV_LENS_ONCE", "kv_lens_update_once"), + "PACK_PSCATTER": ("STEP_PACK_PSCATTER", "pack_parallel_scatter"), + "PACK_IN_GRAPH": ("STEP_PACK_IN_GRAPH", "pack_in_graph"), + "PACK_EARLY_COPY": ("STEP_PACK_EARLY_COPY", "pack_early_copy"), + "H2D_CHANGED_SPAN": ("STEP_H2D_CHANGED_SPAN", "h2d_changed_span"), +} +# The sm_103 HOSTTAIL bundle's six items, all turned off by TRTLLM_M31_DISABLE_HOSTTAIL. +HOSTTAIL = ( + "HOST_D2H_SIDE", + "SMALL_H2D", + "BATCH_CHANGED_ONLY", + "SKIP_ACCEPTED_UPLOAD", + "PHILOX_PACK", + "KV_LENS_ONCE", +) +# Item -> the items it needs (off with it). +NEEDS = {"PACK_IN_GRAPH": ("PACK_PSCATTER",), "PACK_EARLY_COPY": ("PACK_IN_GRAPH", "PACK_PSCATTER")} + + +def _off_with(item: str) -> set: + """The items a kill switch of ``item`` turns off: its own and every item that needs it.""" + return {item} | {other for other, needs in NEEDS.items() if item in needs} + + +def _module_tree() -> ast.Module: + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _item_statements(tree: ast.Module) -> list[ast.stmt]: + """modeling.py's module-level assignments of ``_HOSTTAIL_OFF`` and the ITEMS constants, in source order.""" + constants = {"_HOSTTAIL_OFF"} | {constant for constant, _ in ITEMS.values()} + return [ + node + for node in tree.body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id in constants + ] + + +def _switches(monkeypatch, env): + """A private copy of m31_switches imported under ``env`` (every other TRTLLM_M31_* variable cleared).""" + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_host_tail_{id(env)}", _SWITCHES) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _values(monkeypatch, env) -> dict[str, bool]: + """The ITEMS constants modeling.py computes under ``env``, by item id.""" + switches = _switches(monkeypatch, env) + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec( + compile(ast.Module(_item_statements(_module_tree()), []), str(_MODELING), "exec"), + namespace, + ) + return {item: namespace[constant] for item, (constant, _) in ITEMS.items()} + + +@pytest.mark.cpu_only +def test_the_switches_are_the_ten_items(): + names = [node.targets[0].id for node in _item_statements(_module_tree())] + final = ["_HOSTTAIL_OFF", *(constant for constant, _ in ITEMS.values())] + + # The last definitions (the TP8 HOSTTAIL block) are the ten items, in order; any earlier definition is the TP4 + # base's for the same constant, overridden by them. + assert names[-len(final) :] == final + assert set(names[: -len(final)]) <= set(final[1:]) + for _, field in ITEMS.values(): + assert getattr(RuntimeHooks(), field) is False, field + + +@pytest.mark.cpu_only +def test_every_item_is_on_by_default(monkeypatch): + assert _values(monkeypatch, {}) == {item: True for item in ITEMS} + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", sorted(ITEMS)) +@pytest.mark.parametrize("value", ["1", "true", "ON", " yes "]) +def test_each_kill_switch_turns_off_only_its_own_item(monkeypatch, item, value): + values = _values(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: value}) + + assert values == {other: other not in _off_with(item) for other in ITEMS} + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", sorted(ITEMS)) +@pytest.mark.parametrize("value", ["", "0", "false", "off", "no"]) +def test_an_unset_kill_switch_keeps_the_item(monkeypatch, item, value): + assert _values(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: value}) == { + other: True for other in ITEMS + } + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", sorted(ITEMS)) +def test_a_typo_fails_at_import_naming_the_variable(monkeypatch, item): + with pytest.raises(ValueError, match="TRTLLM_M31_DISABLE_" + item): + _values(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: "2"}) + + +@pytest.mark.cpu_only +def test_an_enable_variable_does_not_turn_an_item_off_or_on(monkeypatch): + env = {"TRTLLM_M31_ENABLE_" + item: "1" for item in ITEMS} + assert _values(monkeypatch, env) == {item: True for item in ITEMS} + + +@pytest.mark.cpu_only +def test_the_bundle_switch_turns_off_the_six_hosttail_items(monkeypatch): + """HOSTTAIL turns off the sm_103 bundle's six items, as on those targets; the pack items stay on.""" + assert _values(monkeypatch, {"TRTLLM_M31_DISABLE_HOSTTAIL": "1"}) == { + item: item not in HOSTTAIL for item in ITEMS + } + + +@pytest.mark.cpu_only +def test_runtime_hooks_wires_each_field_to_its_switch(): + """``_runtime_hooks`` passes each constant to its RuntimeHooks field, and nothing else sets those.""" + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + calls = [ + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1 + keywords = {kw.arg: kw.value for kw in calls[0].keywords} + for constant, field in ITEMS.values(): + assert isinstance(keywords.get(field), ast.Name), field + assert keywords[field].id == constant, field + + +def _module_constant(tree: ast.Module, name: str): + """The value of modeling.py's last module-level ``name = `` (the definition module order makes final).""" + values = [ + node.value + for node in tree.body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id == name + ] + assert values, f"modeling.py defines no module-level {name}" + assert isinstance(values[-1], ast.Constant), f"{name} is not a literal" + return values[-1].value + + +def _always_on(tree: ast.Module, node) -> bool: + """A literal True, or (the TP4 base's spelling) a module constant that is a literal True.""" + if isinstance(node, ast.Constant): + return node.value is True + return isinstance(node, ast.Name) and _module_constant(tree, node.id) is True + + +@pytest.mark.cpu_only +def test_the_step_input_pack_and_its_epoch_stay_on(): + """The fields without a switch here: the pack itself, its step epoch and the one-launch prologue (the pack keeps + the epoch only with pack_in_graph and calls the prologue op only with pack_early_copy, so the switches above + turn those off with them). The TP2 target passes the TP4 base's switch-free constants (STEP_PACK_INPUTS, + STEP_PACK_EPOCH, STEP_PACK_CPP_PROLOGUE: literal True) where the TP8 target passes True itself.""" + tree = _module_tree() + fn = next( + node + for node in tree.body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + call = next( + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ) + keywords = {kw.arg: kw.value for kw in call.keywords} + for field in ("pack_step_inputs", "pack_step_epoch"): + assert _always_on(tree, keywords.get(field)), field + prologue = keywords.get("pack_prologue_op") + if isinstance(prologue, ast.IfExp): + # `` if else None``: the op, always + assert _always_on(tree, prologue.test), "pack_prologue_op" + assert isinstance(prologue.orelse, ast.Constant) and prologue.orelse.value is None + prologue = prologue.body + assert isinstance(prologue, ast.Name), "pack_prologue_op" + + +@pytest.mark.cpu_only +def test_the_target_kv_cache_is_not_a_mamba_cache(): + """skip_accepted_draft_upload is on only together with a non-mamba cache: the mamba state update is the one + consumer of accepted counts on the drafter path (it takes them as an argument, not from the spec metadata).""" + assert not issubclass(MiniMaxM3KVCacheManagerV2, BaseMambaCacheManager) + assert not issubclass(MiniMaxM3KVCacheManagerV2, MambaHybridCacheManager) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="imports the target's modeling module") +def test_the_in_tree_hooks_follow_the_switches(): + modeling = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.modeling" + ) + hooks = modeling._runtime_hooks(None) + + for constant, field in ITEMS.values(): + assert getattr(hooks, field) is getattr(modeling, constant), field + + def disabled(item): + return os.environ.get("TRTLLM_M31_DISABLE_" + item, "").strip().lower() in ( + "1", + "true", + "on", + "yes", + ) + + for item, (_, field) in ITEMS.items(): + off = disabled(item) or any(disabled(need) for need in NEEDS.get(item, ())) + off = off or (item in HOSTTAIL and disabled("HOSTTAIL")) + assert getattr(hooks, field) is not off, item diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_mxdraft_tp2_scope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_mxdraft_tp2_scope.py new file mode 100644 index 000000000000..377f7dff3e3a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_mxdraft_tp2_scope.py @@ -0,0 +1,50 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR TP2 MXDRAFT scope (the TP8 default): ON by default on this target; +TRTLLM_M31_DISABLE_MXDRAFT=1 turns it off (an ENABLE variable changes nothing); a value outside the m31_switches +contract fails at import. Read at import, so each case runs in a fresh process.""" + +import os +import subprocess +import sys + +import pytest + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2" +) + + +def _env(**switches: str) -> dict: + env = { + k: v + for k, v in os.environ.items() + if not k.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")) + } + env.update(switches) + return env + + +@pytest.mark.parametrize( + "enable, disable, on", + [("", "", True), ("1", "", True), ("1", "1", False), ("", "1", False), ("", "0", True)], +) +def test_mxdraft_tp2_default_on_and_kill_switch(enable: str, disable: str, on: bool) -> None: + out = subprocess.run( + [sys.executable, "-c", f"from {TARGET} import dspark\nprint(dspark.MXDRAFT)"], + env=_env(TRTLLM_M31_ENABLE_MXDRAFT=enable, TRTLLM_M31_DISABLE_MXDRAFT=disable), + capture_output=True, + text=True, + check=True, + ) + assert out.stdout.split()[-1] == str(on) + + +def test_mxdraft_kill_switch_typo_fails_at_import() -> None: + out = subprocess.run( + [sys.executable, "-c", f"from {TARGET} import dspark"], + env=_env(TRTLLM_M31_DISABLE_MXDRAFT="nope"), + capture_output=True, + text=True, + ) + assert out.returncode != 0 and "TRTLLM_M31_DISABLE_MXDRAFT" in out.stderr diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_onediv_cap.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_onediv_cap.py new file mode 100644 index 000000000000..79838aa48563 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_onediv_cap.py @@ -0,0 +1,55 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The ONE_DIV row cap the TP2 target carries from the TP4 base: the LC merge divides once up to 16 rows; +TRTLLM_M31_DISABLE_ONEDIV_CAP=1 keeps it on at every row count, TRTLLM_M31_DISABLE_LC_ONEDIV=1 turns it off. Inert at +TP2 in the baseline (LC_TP2 off: the long-context rows run the Triton sparse decode); the switch contract is kept.""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_attn_lc, +) + + +def test_onediv_cap_default() -> None: + assert m3_attn_lc.ONE_DIV_MAX_ROWS == 16 + assert m3_attn_lc.one_div_applies(4) and m3_attn_lc.one_div_applies(16) + assert not m3_attn_lc.one_div_applies(17) and not m3_attn_lc.one_div_applies(48) + + +@pytest.mark.parametrize("disable, applies_at_32", [("", False), ("0", False), ("1", True)]) +def test_onediv_cap_switch( + monkeypatch: pytest.MonkeyPatch, disable: str, applies_at_32: bool +) -> None: + monkeypatch.setenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", disable) + try: + importlib.reload(m31_switches) + lc = importlib.reload(m3_attn_lc) + assert lc.one_div_applies(1) and lc.one_div_applies(16) + assert lc.one_div_applies(32) is applies_at_32 + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", raising=False) + importlib.reload(m31_switches) + importlib.reload(m3_attn_lc) + + +@pytest.mark.parametrize("cap", ["", "1"]) +def test_onediv_kill_switch(monkeypatch: pytest.MonkeyPatch, cap: str) -> None: + """TRTLLM_M31_DISABLE_LC_ONEDIV=1 turns the one-division merge off at every row count, whatever the cap switch.""" + monkeypatch.setenv("TRTLLM_M31_DISABLE_LC_ONEDIV", "1") + monkeypatch.setenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", cap) + try: + importlib.reload(m31_switches) + lc = importlib.reload(m3_attn_lc) + assert lc.ONE_DIV_MAX_ROWS == 0 + assert not any(lc.one_div_applies(rows) for rows in (1, 8, 16, 32)) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_LC_ONEDIV", raising=False) + monkeypatch.delenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", raising=False) + importlib.reload(m31_switches) + importlib.reload(m3_attn_lc) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_prefill_ar_mnnvl.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_prefill_ar_mnnvl.py new file mode 100644 index 000000000000..48acae0d6dfc --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_prefill_ar_mnnvl.py @@ -0,0 +1,64 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""PREFILL-MNNVL: the sm_107 TP2 prefill / mixed-step boundaries' built-in all-reduce is MNNVL up to 8192 rows; larger +steps and every generation step keep the target's AUTO all-reduce. TRTLLM_M31_DISABLE_PREFILL_AR_MNNVL=1 restores +AUTO everywhere.""" + +import importlib +import inspect + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + modeling, +) + +AUTO, MNNVL = object(), object() + + +class _Core: + def __init__(self, mnnvl) -> None: + self.allreduce = AUTO + self.allreduce_mnnvl = mnnvl + + +@pytest.fixture +def prefill_mnnvl_off(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("TRTLLM_M31_DISABLE_PREFILL_AR_MNNVL", "1") + importlib.reload(m31_switches) + try: + yield importlib.reload(modeling) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_PREFILL_AR_MNNVL", raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) + + +def test_default_on_up_to_8192_rows() -> None: + assert modeling.PREFILL_AR_MNNVL and modeling.PREFILL_AR_MNNVL_MAX_ROWS == 8192 + + +@pytest.mark.parametrize("rows", [1, 33, 64, 128, 1024, 8192, 8193, 16384]) +def test_rows_pick_the_all_reduce(rows: int) -> None: + want = MNNVL if rows <= 8192 else AUTO + assert modeling._prefill_allreduce(_Core(MNNVL), rows) is want + assert modeling._prefill_allreduce(_Core(None), rows) is AUTO + + +def test_only_the_prefill_target_takes_it() -> None: + """Every built-in boundary all-reduce of the prefill target goes through ``_prefill_allreduce``; the decode target + (every generation step, the 33..64-row ones included) never does.""" + prefill = inspect.getsource(modeling.PrefillTarget) + # The two CTXNORMQ boundaries and the three native fused ones each pass ``_prefill_allreduce``. + assert prefill.count("_prefill_allreduce(core, ") == 5 + assert prefill.count("_ctxnormq_allreduce_norm(") == 2 + assert prefill.count("_native_allreduce_norm(") == 3 + assert "_prefill_allreduce" not in inspect.getsource(modeling.DecodeTarget) + + +def test_kill_switch(prefill_mnnvl_off) -> None: + m = prefill_mnnvl_off + assert not m.PREFILL_AR_MNNVL + assert "AllReduceStrategy.MNNVL" in inspect.getsource(m.MiniMaxM3Tp2ModelingV2.__init__) + assert m._prefill_allreduce(_Core(None), 128) is AUTO diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_proxy_prebuild.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_proxy_prebuild.py new file mode 100644 index 000000000000..0c2ba96a2904 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_proxy_prebuild.py @@ -0,0 +1,59 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""PROXYPB on the sm_107 TP2 target: every fmha_sm100 variant the MSA indexer's proxy pass can pick (OnlyScore over +the FP8 paged index-K, page 128, no split-KV) is built once before warmup. Two index heads per rank pack rows of up to +32 tokens 2x (packed <= 64 only, so the 2x-packed 65..128 variant is never reached); longer rows run unpacked (<= 64, +65..128, > 128). Variant names are fmha_sm100's dispatch indices (dtype, tile, single warpgroup, sparse mode, page, +split-KV, pack factor: 1 / 2 / 4 at index 0 / 1 / 2). + +The variant builds are recorded rather than compiled (an nvcc build each); the core is a stand-in with the attributes +the prebuild reads. +""" + +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + require_msa_module, +) + +INDEX_HEADS_PER_RANK = 2 +WANT = {"1_0_0_2_1_0_1", "1_0_0_2_1_0_0", "1_0_1_2_1_0_0", "1_1_1_2_1_0_0"} + + +@pytest.fixture +def proxy_builds(monkeypatch): + require_msa_module() # its cutlass compatibility shim must precede the fmha_sm100 import, as in the target + from fmha_sm100 import jit as fmha_jit + + built = [] + + def record(*key): + built.append(fmha_jit._variant_key_from_runtime(*key)[0]) + return SimpleNamespace(run=None) + + monkeypatch.setattr(fmha_jit, "get_fmha_variant", record) + return built + + +def _core(): + attn = SimpleNamespace( + indexer_kv_dtype="fp8", m3_config=SimpleNamespace(num_index_heads=INDEX_HEADS_PER_RANK) + ) + return SimpleNamespace(sparse_layers=frozenset({3, 4, 5}), attn={3: attn, 4: attn, 5: attn}) + + +def test_builds_every_proxy_variant_once(proxy_builds) -> None: + names = modeling.MiniMaxM3Tp2ModelingV2._prebuild_msa_proxy(_core()) + assert set(names) == WANT + assert sorted(proxy_builds) == sorted(WANT), ( + "each variant is built once, across every sparse layer" + ) + + +def test_prebuild_is_on_by_default() -> None: + assert modeling.MSA_PROXY_PREBUILD is True diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_r6_dense_prequant.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_r6_dense_prequant.py new file mode 100644 index 000000000000..2261fa34e030 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_r6_dense_prequant.py @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""R6 on the sm_107 TP2 target: the dense-MLP layer boundary hands the next layer's QKV its MXFP8 activation by +VR: OFF by default (not bitwise on sm_107): TRTLLM_M31_ENABLE_R6=1 +turns it on, TRTLLM_M31_DISABLE_R6=1 forces it off. The engagement itself ("[M3 dispatch] r6_dense_prequant taken" / +"off") is checked in-model.""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) + + +def test_r6_is_off_by_default_on_vr() -> None: + assert modeling.R6_DENSE_PREQUANT is False + + +@pytest.mark.parametrize( + ("enable", "disable", "expected"), + [("1", "", True), ("1", "1", False), ("0", "", False), ("", "1", False)], +) +def test_r6_switches( + monkeypatch: pytest.MonkeyPatch, enable: str, disable: str, expected: bool +) -> None: + monkeypatch.setenv("TRTLLM_M31_ENABLE_R6", enable) + monkeypatch.setenv("TRTLLM_M31_DISABLE_R6", disable) + try: + assert importlib.reload(modeling).R6_DENSE_PREQUANT is expected + finally: + monkeypatch.delenv("TRTLLM_M31_ENABLE_R6", raising=False) + monkeypatch.delenv("TRTLLM_M31_DISABLE_R6", raising=False) + importlib.reload(modeling) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s10_scope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s10_scope.py new file mode 100644 index 000000000000..81c7aaef77de --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s10_scope.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR S10 scope (tp2, the sm_107 TP8 drafter levers): the lean drafter glue at every draft length (TP8: K3 and K7 +gated token-identical).""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark, +) + + +@pytest.mark.parametrize("k", [1, 3, 4, 7, 15]) +def test_s10_applies_at_every_draft_length(k: int) -> None: + assert dspark.S10_MAX_DRAFT_LEN == 63 + assert dspark.s10_applies(k) is True diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s2_tail.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s2_tail.py new file mode 100644 index 000000000000..4c0780fcd074 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s2_tail.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2W_S2_TAIL dispatch: default-on policy, row band, and declines that keep the stock boundary. +The fused op's numerics are its 2-rank microbench (bitwise to finalize + one-shot Lamport AR + norm + mxfp8_quantize at +1..224 rows) and, in-model, TRTLLM_M31_TP2W_S2_CHECK.""" + +import types + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as m, +) + + +def test_default_on_with_kill_switch() -> None: + assert m.TP2W_S2_TAIL == (not m.m31_disabled("TP2W_S2_TAIL")) + + +def test_row_band_is_past_the_mc_finalize_and_within_the_mb() -> None: + assert m.S2_TAIL_MIN_ROWS == m.MOE_TRITON_FINALIZE_MIN_TOKENS + assert m.S2_TAIL_MAX_ROWS == 224 + + +def test_boundary_declines_without_the_op(monkeypatch) -> None: + monkeypatch.setattr(m, "_s2_op", lambda: None) + parts = ( + torch.zeros(4, 8, dtype=torch.bfloat16), + torch.zeros(1, 4), + torch.zeros(1, 4, dtype=torch.int32), + ) + assert m._s2_boundary(types.SimpleNamespace(), 1, parts, None, None, None, None) is None + + +def test_boundary_declines_parts_not_the_runner_layout(monkeypatch) -> None: + monkeypatch.setattr(m, "_s2_op", lambda: object()) + assert ( + m._s2_boundary(types.SimpleNamespace(), 1, torch.zeros(1, 8), None, None, None, None) + is None + ) + + +def test_decode_tail_uses_existing_moe_parts_and_stock_decline(monkeypatch) -> None: + final_norm = object() + norm = object() + core = types.SimpleNamespace(w={"final_norm": final_norm}, allreduce=object()) + target = m.DecodeTarget(core) + h = torch.zeros(88, 8, dtype=torch.bfloat16) + residual = torch.ones_like(h) + parts = [object(), object(), object(), object()] + calls = [] + monkeypatch.setattr(m, "_triton_finalize_rows", lambda *args: True) + monkeypatch.setattr(m, "is_torch_compiling", lambda: False) + monkeypatch.setattr(m, "_s2_op", lambda: object()) + monkeypatch.setattr(target, "_moe", lambda *args, **kwargs: (parts, None, False, False)) + + def boundary(c, rows, routed, shared, r, n, ar): + assert c is core and rows == 88 and tuple(routed) == tuple(parts[1:]) + assert shared is parts[0] and r is residual and n is norm and ar is core.allreduce + calls.append("fused") + return None + + monkeypatch.setattr(m, "_s2_boundary", boundary) + monkeypatch.setattr(m, "_triton_finalize", lambda *args: h) + monkeypatch.setattr(m, "_native_allreduce_norm", lambda *args: (h, residual)) + result = target._s2_tail(3, h, residual, norm) + assert result[0] is h and result[1] is residual + assert calls == ["fused"] + assert target._s2_tail(3, h, residual, final_norm) is None + assert target._s2_tail(3, h[:48], residual[:48], norm) is None + assert target._s2_tail(3, h.float(), residual, norm) is None + assert calls == ["fused"] + + +def _decode_graph_core(monkeypatch, pinned: bool): + monkeypatch.setattr(m, "TP2W_S2_TAIL", True) + monkeypatch.setattr(m, "MX_TAC_TP4", pinned) + monkeypatch.setattr( + m, + "_mx_mode", + lambda state: m.MxMode( + decode_graph=True, + autotune=False, + tuned_quantize_gemm=True, + flashinfer=True, + native_tuning=False, + ), + ) + monkeypatch.setattr(m, "linear_path", lambda *a: m.GENERIC_GEMM_PATH) + monkeypatch.setattr(m.m3_prequant.SLOT, "pending", None) + method = types.SimpleNamespace(tune_decode_graph_backends=True) + return types.SimpleNamespace( + w={"w": torch.zeros(16, 32), "s": torch.zeros(16, 1)}, + mxfp8=types.SimpleNamespace(quant_method=method), + ) + + +def test_decode_graph_qkv_takes_the_tail_mxfp8_on_the_pinned_path(monkeypatch) -> None: + core = _decode_graph_core(monkeypatch, pinned=True) + calls = [] + monkeypatch.setattr( + m, "m3_mxfp8_pinned_gemm", lambda a, sf, w, s, dt, t: calls.append((a, sf)) or "gemm" + ) + monkeypatch.setattr( + m, "m3_mxfp8_pinned_quantize", lambda *a: (_ for _ in ()).throw(AssertionError("quantized")) + ) + x = torch.zeros(64, 32, dtype=torch.bfloat16) + q, sf = torch.ones(64, 32), torch.ones(64) + m._S2_MX.publish(x, q, sf) + out, pushed = m._linear(core, x, "w", "s", True) + assert out == "gemm" and not pushed + assert len(calls) == 1 and calls[0][0] is q and calls[0][1] is sf + assert m._S2_MX.pending is None + + +def test_decode_graph_off_the_pinned_path_keeps_the_stock_quantize(monkeypatch) -> None: + core = _decode_graph_core(monkeypatch, pinned=False) + monkeypatch.setattr( + m, "m3_mxfp8_pinned_gemm", lambda *a: (_ for _ in ()).throw(AssertionError("pinned")) + ) + monkeypatch.setattr(m, "_mxfp8_linear", lambda core, x, w, s, mode: "stock") + x = torch.zeros(64, 32, dtype=torch.bfloat16) + m._S2_MX.publish(x, torch.ones(64, 32), torch.ones(64)) + out, _ = m._linear(core, x, "w", "s", True) + assert out == "stock" + assert m._S2_MX.pending is None + + +def test_tail_mxfp8_of_another_tensor_is_not_taken(monkeypatch) -> None: + core = _decode_graph_core(monkeypatch, pinned=True) + monkeypatch.setattr(m, "m3_mxfp8_pinned_gemm", lambda a, sf, w, s, dt, t: ("gemm", a.dtype)) + monkeypatch.setattr( + m, "m3_mxfp8_pinned_quantize", lambda x, t: (torch.zeros(1, dtype=torch.int8), None) + ) + m._S2_MX.publish(torch.zeros(64, 32, dtype=torch.bfloat16), torch.ones(64, 32), torch.ones(64)) + out, _ = m._linear(core, torch.zeros(64, 32, dtype=torch.bfloat16), "w", "s", True) + assert out == ("gemm", torch.int8) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s4_logits_default.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s4_logits_default.py new file mode 100644 index 000000000000..e8d54c61c654 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s4_logits_default.py @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""S4-logits on the sm_107 TP2 target: on by default (S4STREAM, the TP8 recipe with MCAG off); +``TRTLLM_M31_DISABLE_S4_LOGITS=1`` turns it off (also over ``TRTLLM_M31_ENABLE_S4_LOGITS=1``). The assignment is +evaluated from dspark_s4.py's source against a private copy of m31_switches. CPU only. +""" + +import ast +import importlib.util +import os +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +_DIR = Path(_family.__file__).resolve().parent / "minimax_m31_nvfp4__sm_107__tp2" +_SWITCHES = _DIR / "m31_switches.py" +_S4 = _DIR / "dspark_s4.py" + + +class _Worker: + pass + + +def _value(monkeypatch, env) -> bool: + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_s4_logits_{id(env)}", _SWITCHES) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + tree = ast.parse(_S4.read_text(), filename=str(_S4)) + statements = [ + node + for node in ast.walk(tree) + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Attribute) + and node.targets[0].attr == "_s4_logits_on" + ] + assert len(statements) == 1 + worker = _Worker() + namespace = { + "m31_disabled": switches.m31_disabled, + "m31_enabled": switches.m31_enabled, + "self": worker, + } + exec(compile(ast.Module(statements, []), str(_S4), "exec"), namespace) + return worker._s4_logits_on + + +@pytest.mark.cpu_only +def test_on_by_default(monkeypatch): + assert _value(monkeypatch, {}) is True + + +@pytest.mark.cpu_only +def test_the_kill_switch_turns_it_off(monkeypatch): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_S4_LOGITS": "1"}) is False + + +@pytest.mark.cpu_only +def test_the_kill_switch_wins(monkeypatch): + env = {"TRTLLM_M31_ENABLE_S4_LOGITS": "1", "TRTLLM_M31_DISABLE_S4_LOGITS": "1"} + assert _value(monkeypatch, env) is False diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s5_scope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s5_scope.py new file mode 100644 index 000000000000..6d40483c9351 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_s5_scope.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR S5 scope (tp2): the TP8 table of the window-capped drafter decode (OFF by default, K3-bundle gate c; a TP2 +table needs its own microbench), only up to the measured batch per draft length.""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark, +) + + +@pytest.mark.parametrize("k, expected", [(1, 0), (3, 0), (5, 0), (7, 2), (15, 2)]) +def test_s5_max_batch_by_draft_length(k: int, expected: int) -> None: + assert dspark.S5_MAX_BATCH_BY_K == {3: 0, 7: 2} + assert dspark.s5_max_batch_for(k) == expected diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_slot_ids_changed_default.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_slot_ids_changed_default.py new file mode 100644 index 000000000000..14dfc579e3d1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_slot_ids_changed_default.py @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SLOT_IDS_CHANGED (BP-3) on this target: off by default, ``TRTLLM_M31_ENABLE_SLOT_IDS_CHANGED`` turns it on and +``TRTLLM_M31_DISABLE_SLOT_IDS_CHANGED`` still wins; ``_runtime_hooks`` passes the constant as +RuntimeHooks.spec_slot_ids_changed_only. The constant is evaluated from modeling.py's source against a private copy of +m31_switches loaded under each environment, so the heavy modeling module is not reloaded. CPU only. +""" + +import ast +import importlib.util +import os +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp2" +CONSTANT = "SLOT_IDS_CHANGED" +FIELD = "spec_slot_ids_changed_only" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + + +def _module_tree() -> ast.Module: + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _value(monkeypatch, env) -> bool: + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_slot_ids_{id(env)}", _SWITCHES) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + statements = [ + node + for node in _module_tree().body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id == CONSTANT + ] + assert len(statements) == 1 + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + return namespace[CONSTANT] + + +@pytest.mark.cpu_only +def test_off_by_default(monkeypatch): + assert _value(monkeypatch, {}) is False + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("value", ["1", "true", "ON", " yes "]) +def test_the_opt_in_turns_it_on(monkeypatch, value): + assert _value(monkeypatch, {"TRTLLM_M31_ENABLE_SLOT_IDS_CHANGED": value}) is True + + +@pytest.mark.cpu_only +def test_the_kill_switch_wins(monkeypatch): + env = {"TRTLLM_M31_ENABLE_SLOT_IDS_CHANGED": "1", "TRTLLM_M31_DISABLE_SLOT_IDS_CHANGED": "1"} + assert _value(monkeypatch, env) is False + + +@pytest.mark.cpu_only +def test_runtime_hooks_passes_the_constant(): + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + calls = [ + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1 + keywords = {kw.arg: kw.value for kw in calls[0].keywords} + assert isinstance(keywords.get(FIELD), ast.Name) and keywords[FIELD].id == CONSTANT diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_switches_sm107_tp2.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_switches_sm107_tp2.py new file mode 100644 index 000000000000..b4b4927a6d73 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_switches_sm107_tp2.py @@ -0,0 +1,83 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The value contract of this target's kill switches (m31_switches): 1 / true / on / yes set TRTLLM_M31_DISABLE_ / +TRTLLM_M31_ENABLE_, empty / 0 / false / off / no leave it unset (case-insensitive), anything else raises at import +naming the variable; the environment is read once, at import. + +Each case loads a private copy of the module from its file, so the in-tree module (read by every other module of the +target at import) is never reloaded. CPU only. +""" + +import importlib.util +import os +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp2" +_SWITCHES = Path(_family.__file__).resolve().parent / TARGET / "m31_switches.py" + +pytestmark = pytest.mark.cpu_only + +KINDS = [("TRTLLM_M31_DISABLE_", "m31_disabled"), ("TRTLLM_M31_ENABLE_", "m31_enabled")] +ITEM = "UTCONTRACT" # an id no item uses, so the real switches of the test process cannot interfere + + +def _load(monkeypatch, env): + """A private copy of m31_switches, imported under ``env`` (every other TRTLLM_M31_* variable cleared).""" + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_contract_{id(env)}", _SWITCHES) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@pytest.mark.parametrize("prefix,query", KINDS) +@pytest.mark.parametrize( + "value", ["", " ", "0", " 0 ", "false", "FALSE", "off", " Off ", "no", "No"] +) +def test_unset_spellings_leave_the_switch_unset(monkeypatch, prefix, query, value): + assert not getattr(_load(monkeypatch, {}), query)(ITEM) + assert not getattr(_load(monkeypatch, {prefix + ITEM: value}), query)(ITEM) + + +@pytest.mark.parametrize("prefix,query", KINDS) +@pytest.mark.parametrize("value", ["1", " 1 ", "true", "TRUE", "on", "On", "yes", "YES"]) +def test_set_spellings_set_the_switch(monkeypatch, prefix, query, value): + assert getattr(_load(monkeypatch, {prefix + ITEM: value}), query)(ITEM) + + +@pytest.mark.parametrize("prefix", [k for k, _ in KINDS]) +@pytest.mark.parametrize("value", ["2", "-1", "1.0", "y", "n", "enable", "disabled", "truee", "of"]) +def test_any_other_value_fails_at_import_naming_the_variable(monkeypatch, prefix, value): + """Typos must fail loudly: a loose parse would count any value but "" / "0" as set (DISABLE_X=false disabling X).""" + with pytest.raises(ValueError, match=prefix + ITEM): + _load(monkeypatch, {prefix + ITEM: value}) + + +def test_the_id_is_matched_exactly(monkeypatch): + switches = _load(monkeypatch, {"TRTLLM_M31_DISABLE_" + ITEM: "1"}) + assert switches.m31_disabled(ITEM) + for other in (ITEM.lower(), ITEM + "_X", ITEM[:-1], ""): + assert not switches.m31_disabled(other), other + + +def test_disable_and_enable_are_independent(monkeypatch): + """Both read: an item's own code applies "DISABLE wins" (e.g. B1, L-DS1); the module reports each separately.""" + switches = _load( + monkeypatch, {"TRTLLM_M31_DISABLE_" + ITEM: "1", "TRTLLM_M31_ENABLE_" + ITEM: "1"} + ) + assert switches.m31_disabled(ITEM) and switches.m31_enabled(ITEM) + + +def test_the_environment_is_read_once_at_import(monkeypatch): + switches = _load(monkeypatch, {}) + monkeypatch.setenv("TRTLLM_M31_DISABLE_" + ITEM, "1") + monkeypatch.setenv("TRTLLM_M31_ENABLE_" + ITEM, "1") + assert not switches.m31_disabled(ITEM) and not switches.m31_enabled(ITEM) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_dspark_scope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_dspark_scope.py new file mode 100644 index 000000000000..b3947678e2c5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_dspark_scope.py @@ -0,0 +1,170 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""What the tp2 target's DSpark worker sets up at prebuild (its defaults), on a stub worker whose set-up calls +are recorded: S9b ON with an installed fused AR + norm op (S9B-MC's multicast form inside it, ON), S5 and S2 OFF, the +S10c hooks installed wrapped with their work counters, EX11 installed, MXDRAFT asked for the decode rows; each kill +switch turns its own item off. CPU only. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_report, +) + +pytestmark = pytest.mark.cpu_only + + +def _prebuild(monkeypatch, disabled=(), lm_head=None, a1=None, s4=None, s4_logits=False): + """Run the real M3DSparkWorker.prebuild on a stub; returns what it asked for. With ``lm_head`` the N3 A1 / MCAG + tail runs too (A1's set-up stubbed to ``a1``; the S4 set-up stubbed to leave ``s4`` and ``s4_logits``).""" + asked = {"lines": []} + monkeypatch.setattr( + dspark, + "logger", + SimpleNamespace(info=asked["lines"].append, warning=asked["lines"].append), + ) + monkeypatch.setattr(dspark, "N3A_DISABLED", True) + monkeypatch.setattr(dspark, "N3A_KV_DISABLED", True) + monkeypatch.setattr(dspark, "MCAG", "MCAG" not in disabled) + monkeypatch.setattr(m3_report, "_rank", lambda: 0) + monkeypatch.setattr(m3_report, "_info", lambda message: None) + monkeypatch.setattr(dspark, "m31_disabled", lambda i: i in disabled) + monkeypatch.setattr(dspark, "m31_enabled", lambda i: False) + monkeypatch.setattr(dspark, "MXDRAFT", "MXDRAFT" not in disabled) + monkeypatch.setattr(dspark, "S9B_MC", "S9B_MC" not in disabled) + monkeypatch.setattr( + dspark, + "route_mx_linears", + lambda model, rows: asked.setdefault("MXDRAFT", list(rows)) and 0, + ) + worker = object.__new__(dspark.M3DSparkWorker) + worker.mapping = SimpleNamespace(tp_rank=0, tp_size=2, enable_attention_dp=False) + worker.spec_config = SimpleNamespace(max_draft_len=7) + + def prebuild_s4(*a, **k): + worker._s4 = s4 + worker._s4_logits_on = s4_logits + + worker.prebuild_s4 = prebuild_s4 + worker.enable_lean_glue = lambda model, on: asked.__setitem__("S10", on) + worker.enable_small_m_linears = lambda *a, **k: False + worker.enable_window_capped_decode = lambda model, max_batch: asked.__setitem__("S5", max_batch) + worker.enable_fc_tp_split = lambda model, on=True: asked.__setitem__("S2", on) or on + worker.enable_fused_ar_norm = lambda model, on=True: asked.__setitem__("S9B", on) or on + worker.enable_fused_ar_norm_op = lambda model, op: ( + asked.__setitem__("S9B_OP", op) or (op is not None) + ) + for name, key in ( + ("enable_fused_ctx_store", "S10C_A"), + ("enable_fused_drafter_inputs", "S10C_C"), + ("enable_fused_ctx_advance", "S10C_D"), + ("enable_fused_block_refresh", "S10C_B"), + ): + setattr(worker, name, lambda fn, _k=key: asked.__setitem__(_k, fn) or fn is not None) + worker.enable_fused_forward_indices = lambda model, fn: ( + asked.__setitem__("S10C_E", fn) or (fn is not None) + ) + worker.enable_fused_qk_norm_rope = lambda model, fn: ( + asked.__setitem__("EX11", fn) or (fn is not None) + ) + worker._prebuild_a1 = lambda lm_head: a1 + worker._prebuild_draft_a1 = lambda lm_head, draft_model: asked.__setitem__("DRAFT_A1", True) + worker._prebuild_mcag = lambda lm_head: asked.__setitem__("MCAG_BUILD", True) or 100032 + worker.enable_draft_logits_gather_op = lambda op: asked.__setitem__("MCAG_OP", op) or ( + op is not None + ) + dspark.M3DSparkWorker.prebuild(worker, lm_head, SimpleNamespace(_dflash_fc_split=None)) + return asked + + +def test_the_tp2_defaults(monkeypatch): + asked = _prebuild(monkeypatch) + assert asked["S9B"] is True and asked["S5"] == 0 and asked["S2"] is False + assert asked["S10"] is True + assert callable(asked["S9B_OP"]) + for key in ("S10C_A", "S10C_B", "S10C_C", "S10C_D", "S10C_E", "EX11"): + assert callable(asked[key]), key + assert getattr(asked[key], "__wrapped__", None) is not None, key # the work-counted wrapper + assert asked["MXDRAFT"] == [7 * b for b in range(1, dspark.MAX_FAST_BATCH + 1)] + + +def test_s9b_kill_installs_no_op(monkeypatch): + asked = _prebuild(monkeypatch, disabled={"S9B"}) + assert asked["S9B"] is False and asked["S9B_OP"] is None + + +def test_s9b_mc_kill_keeps_the_counting_op_without_the_multicast_form(monkeypatch): + """With S9B-MC killed the installed op never takes the operands (the stock fused all-reduce runs).""" + asked = _prebuild(monkeypatch, disabled={"S9B_MC"}) + op = asked["S9B_OP"] + assert op(torch.zeros(4, 8), torch.zeros(4, 8), torch.ones(8), 1e-6) is None + + +@pytest.mark.parametrize("item", ["S10C_A", "S10C_B", "S10C_C", "S10C_D", "S10C_E", "EX11"]) +def test_each_hook_kill_switch(monkeypatch, item): + asked = _prebuild(monkeypatch, disabled={item}) + assert asked[item] is None + + +def test_s10c_kills_every_s10c_hook(monkeypatch): + asked = _prebuild(monkeypatch, disabled={"S10C"}) + assert all(asked[k] is None for k in ("S10C_A", "S10C_B", "S10C_C", "S10C_D", "S10C_E")) + + +def test_mxdraft_kill(monkeypatch): + asked = _prebuild(monkeypatch, disabled={"MXDRAFT"}) + assert "MXDRAFT" not in asked + + +def _mcag_line(asked): + return [s for s in asked["lines"] if s.startswith("M3.1 MCAG:")] + + +def test_c6_mcag_declines_with_its_reason_when_s4_is_not_set_up(monkeypatch): + """C6 (TP2: the vocab shard is past the S4 chain's capacity, so S4 is never set up): no vocab-sharded draft logits + reach the gather, MCAG is not compiled, no op is installed, and the line says why.""" + asked = _prebuild(monkeypatch, lm_head=SimpleNamespace(), a1={"world": 2}) + assert "MCAG_BUILD" not in asked and asked["MCAG_OP"] is None + (line,) = _mcag_line(asked) + assert line.startswith( + "M3.1 MCAG: draft-logits all-gather on the multicast kernel off on rank 0 (" + ) + assert "S4 is not set up" in line + + +def test_c6_mcag_declines_when_s4_logits_is_off(monkeypatch): + asked = _prebuild(monkeypatch, lm_head=SimpleNamespace(), a1={"world": 2}, s4=object()) + assert "MCAG_BUILD" not in asked and asked["MCAG_OP"] is None + assert "S4-logits is off" in _mcag_line(asked)[0] + + +def test_c6_mcag_builds_when_sharded_draft_logits_reach_it(monkeypatch): + asked = _prebuild( + monkeypatch, lm_head=SimpleNamespace(), a1={"world": 2}, s4=object(), s4_logits=True + ) + assert asked["MCAG_BUILD"] is True and callable(asked["MCAG_OP"]) + assert _mcag_line(asked) == [ + "M3.1 MCAG: draft-logits all-gather on the multicast kernel on on rank 0" + ] + + +def test_c6_mcag_kill_switch_still_wins(monkeypatch): + asked = _prebuild( + monkeypatch, + disabled={"MCAG"}, + lm_head=SimpleNamespace(), + a1={"world": 2}, + s4=object(), + s4_logits=True, + ) + assert "MCAG_BUILD" not in asked and asked["MCAG_OP"] is None + assert _mcag_line(asked) == [ + "M3.1 MCAG: draft-logits all-gather on the multicast kernel off on rank 0" + ] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_fused_projection_default.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_fused_projection_default.py new file mode 100644 index 000000000000..02239736a76b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_fused_projection_default.py @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The M3.1 sm_107 TP2 target packs the sparse layers' QKV + index projection by default (a measured +win); an explicit ``fuse_qkv_index_projection`` in the deployment's config wins, so False is the kill switch.""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + weights as tp_weights, +) +from tensorrt_llm.llmapi.llm_args import MiniMaxM3SparseAttentionConfig + +_MSA = dict(implementation="msa", indexer_kv_dtype="fp8") + + +@pytest.mark.parametrize( + ("explicit", "expected"), + [ + ({}, True), + ({"fuse_qkv_index_projection": True}, True), + ({"fuse_qkv_index_projection": False}, False), + ], +) +def test_fused_projection_default_and_kill(explicit: dict, expected: bool) -> None: + config = MiniMaxM3SparseAttentionConfig(**_MSA, **explicit) + assert tp_weights.fused_projection(config) is expected + + +def test_no_sparse_config_is_unfused() -> None: + assert tp_weights.fused_projection(None) is False + + +@pytest.mark.parametrize("value", [True, False]) +def test_a_config_without_set_fields_is_taken_at_its_word(value: bool) -> None: + from types import SimpleNamespace + + assert tp_weights.fused_projection(SimpleNamespace(fuse_qkv_index_projection=value)) is value diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_prefill_brk.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_prefill_brk.py new file mode 100644 index 000000000000..ddc0a8c5aac8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_prefill_brk.py @@ -0,0 +1,172 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""PREFILL_BRK (Host's B1 on the M3.1 tp2 target): a step with context rows under a breakable prefill CUDA graph. + +The wiring, with the kernels stubbed (CPU tensors only): +* under a breakable graph the attention boundary is handed to the eager-on-graph callable: a sparse layer passes its + five-way projection (the producer runs inside the boundary), the attention core passes q / k / v; nothing calls the + MSA backend directly, and the o_proj reads the boundary's output buffer [rows, heads * head_dim]; +* the boundary reads the live row count from the step's metadata at replay: the producer sees packed[:live] and + position_ids[..., :live], the backend q[:live] / output[:live]; +* the unfused index projection still declines breakable graphs; +* outside a breakable graph the step runs the MSA backend directly, as before; +* TRTLLM_M31_DISABLE_PREFILL_BRK (PREFILL_BRK False) restores the raise. CPU only. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) + +pytestmark = pytest.mark.cpu_only + +HEADS, HEAD_DIM, ROWS, LIVE = 32, 128, 16, 11 +PACKED = 4992 # the TP2 five-way projection: 39 packed heads of 128 + + +def _core(fused=True): + return SimpleNamespace( + heads=HEADS, + head_dim=HEAD_DIM, + sparse_layers=frozenset({0, 1}), + fused_projection=fused, + main_kv_is_nvfp4=False, + attn={0: "attn0", 1: "attn1"}, + inv_kv_scales={}, + kv_scales={}, + ) + + +def _target(core): + target = modeling.PrefillTarget(core) + target._contract_pending = False + return target + + +@pytest.fixture +def stubs(monkeypatch): + calls = {"eager": [], "live": [], "linear": [], "producer": [], "msa": []} + + def eager(*args): + calls["eager"].append(args) + + def live(core, i, q, k, v, idx_q, md, output): + calls["live"].append((i, q, k, v, idx_q, md, output)) + + def linear(core, x, wk, sk, low_m_gemm, push=None): + calls["linear"].append((wk, x)) + n = PACKED if wk.endswith("_qkv_w") else 6144 + return torch.zeros(x.shape[0], n, dtype=torch.bfloat16), False + + monkeypatch.setattr(modeling, "_prefill_attention_eager", eager) + monkeypatch.setattr(modeling, "_msa_attention_live", live) + monkeypatch.setattr(modeling, "_linear", linear) + return calls + + +def _brk(monkeypatch, on: bool): + monkeypatch.setattr(modeling, "is_in_breakable_cuda_graph", lambda: on) + + +def test_sparse_layer_hands_the_projection_to_the_eager_boundary(monkeypatch, stubs): + _brk(monkeypatch, True) + target = _target(_core()) + hidden = torch.zeros(ROWS, 6144, dtype=torch.bfloat16) + pos = torch.arange(ROWS) + out, _ = target._attention(1, pos, hidden, step=None) + assert len(stubs["eager"]) == 1 and not stubs["live"] + packed, position_ids, q, k, v, idx_q, output, key, i = stubs["eager"][0] + assert packed.shape == (ROWS, PACKED) and position_ids is pos + assert (q, k, v, idx_q) == (None, None, None, None) + assert output.shape == (ROWS, HEADS * HEAD_DIM) and output.dtype == torch.bfloat16 + assert key == target._bcg_key and modeling._BCG_CORES[key] is target.core and i == 1 + # the o_proj reads the boundary's output buffer + assert stubs["linear"][-1][0] == "l1_o_w" and stubs["linear"][-1][1] is output + assert out.shape == (ROWS, 6144) + + +def test_attention_core_hands_q_k_v_to_the_eager_boundary(monkeypatch, stubs): + _brk(monkeypatch, True) + target = _target(_core()) + q = torch.zeros(ROWS, HEADS * HEAD_DIM, dtype=torch.bfloat16) + k = torch.zeros(ROWS, 2 * HEAD_DIM, dtype=torch.bfloat16) + v = torch.zeros_like(k) + output = target._attention_core(0, q, k, v, None, {"metadata": "md"}) + assert not stubs["live"] and len(stubs["eager"]) == 1 + args = stubs["eager"][0] + assert args[:2] == (None, None) and args[2] is q and args[3] is k and args[4] is v + assert args[6] is output and output.shape == (ROWS, HEADS * HEAD_DIM) + + +def test_without_a_breakable_graph_the_backend_runs_directly(monkeypatch, stubs): + _brk(monkeypatch, False) + target = _target(_core()) + q = torch.zeros(ROWS, HEADS * HEAD_DIM, dtype=torch.bfloat16) + target._attention_core(1, q, None, None, None, {"metadata": "md"}) + assert not stubs["eager"] and len(stubs["live"]) == 1 and stubs["live"][0][5] == "md" + + +def test_the_unfused_index_projection_declines_breakable_graphs(monkeypatch, stubs): + _brk(monkeypatch, True) + target = _target(_core(fused=False)) + with pytest.raises(RuntimeError, match="five-way projection"): + target._attention(1, None, torch.zeros(ROWS, 6144, dtype=torch.bfloat16), step=None) + + +def test_the_boundary_slices_the_live_rows_from_the_steps_metadata(monkeypatch, stubs): + core = _core() + modeling._BCG_CORES["ut_core"] = core + md = SimpleNamespace( + num_tokens=LIVE, kv_cache_manager="kv", msa_idx_k_cache="idx", msa_out_cache_loc="loc" + ) + monkeypatch.setattr( + modeling, "get_model_extra_attrs", lambda: {"attention_metadata": lambda: md} + ) + seen = {} + + def producer(core_, i, packed, position_ids, manager, idx_cache, out_loc): + seen.update(packed=packed, pos=position_ids, args=(manager, idx_cache, out_loc)) + return torch.zeros(LIVE, HEADS, HEAD_DIM), torch.zeros(LIVE, 2, HEAD_DIM) + + monkeypatch.setattr(modeling, "_sparse_qkv_producer", producer) + packed = torch.zeros(ROWS, PACKED, dtype=torch.bfloat16) + pos = torch.arange(ROWS).view(1, ROWS) + output = torch.zeros(ROWS, HEADS * HEAD_DIM, dtype=torch.bfloat16) + try: + modeling._prefill_attention_boundary( + packed, pos, None, None, None, None, output, "ut_core", 1 + ) + finally: + del modeling._BCG_CORES["ut_core"] + assert seen["packed"].shape[0] == LIVE and seen["pos"].shape == (1, LIVE) + assert seen["args"] == ("kv", "idx", "loc") + (i, q, k, v, idx_q, md_seen, out_seen) = stubs["live"][0] + assert i == 1 and k is None and v is None and md_seen is md and out_seen is output + + +def test_the_kill_switch_restores_the_raise(monkeypatch, stubs): + _brk(monkeypatch, True) + monkeypatch.setattr(modeling, "PREFILL_BRK", False) + target = _target(_core()) + with pytest.raises(RuntimeError, match="TRTLLM_M31_DISABLE_PREFILL_BRK"): + target.forward(SimpleNamespace(), None, None, None, None, None, None) + + +def test_a_breakable_step_logs_its_engagement_and_counts(monkeypatch, stubs): + """With PREFILL_BRK on, the forward reports the brk path (and its work counter) before the layers run.""" + _brk(monkeypatch, True) + seen = [] + monkeypatch.setattr(modeling, "taken", lambda path, rows, what="": seen.append(path)) + + class _Stop(Exception): + pass + + target = _target(_core()) + monkeypatch.setattr(target, "step_args", lambda md: (_ for _ in ()).throw(_Stop())) + with pytest.raises(_Stop): + target.forward(SimpleNamespace(), None, None, None, None, None, None) + assert seen == [modeling.PREFILL_BRK_PATH] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_r1vr8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_r1vr8.py new file mode 100644 index 000000000000..7939a3a1b818 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_r1vr8.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""R1VR8 on the sm_107 TP2 target: the decode layer takes pure generation steps up to the TP4 base's 48 rows on the +M3 tp2 row buckets (tiny 8 / b16 16 / b48 48), and up to 64 with VR-BANDS; TRTLLM_M31_DISABLE_R1VR8=1 restores the +TP2-lineage 8-row bound (one "tiny" plan, every GEMV row table at 1..8, SWQ (17..32 rows) and VR-BANDS off), as the +sm_107 TP8 target's kill does. The switch follows the m31_switches value contract (read at import), so the module is +reloaded under each environment. CPU only. +""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mx_gemv import ( # noqa: E501 + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, +) + +ALL = (QKV_SPARSE, QKV, O_PROJ, SHARED_GATE_UP, SHARED_DOWN, DENSE_GATE_UP, DENSE_DOWN) + + +@pytest.fixture +def r1vr8_off(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("TRTLLM_M31_DISABLE_R1VR8", "1") + importlib.reload(m31_switches) + try: + yield importlib.reload(modeling) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_R1VR8", raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) + + +def test_default_bounds() -> None: + m = modeling + assert m.TP2_ROWS == 8 and m.R1VR8 is True and m.DECODE_MAX_TOKENS == 48 + assert m.TP2_BUCKETS == (8, 16, 48) + assert [p.name for p in m.DECODE_PLANS][:3] == ["tiny", "b16", "b48"] + assert [p.max_tokens for p in m.DECODE_PLANS][:3] == [8, 16, 48] + assert m.is_decode_step(48) and m.is_decode_step(33) + + +def test_swq_takes_the_shared_down_rows_past_its_gemv() -> None: + """SWQ: the shared down projection (6144, 1536) at 17..32 rows, the rows its decode GEMV does not serve.""" + m = modeling + assert SHARED_DOWN == (6144, 1536) == m.SWQ_SHARED_DOWN + assert m.SWQ is True + assert m.SWQ_ROWS == {SHARED_DOWN: frozenset(range(17, 33))} + assert not (m.SWQ_ROWS[SHARED_DOWN] & m.MX_GEMV_ROWS[SHARED_DOWN]) + + +def test_kill_switch_restores_the_8_row_bound(r1vr8_off) -> None: + m = r1vr8_off + assert not m.R1VR8 and m.DECODE_MAX_TOKENS == 8 + assert not m.VR_BANDS and m.DECODE_STEP_MAX == 8 + assert m.TP2_BUCKETS == (8,) and [p.name for p in m.DECODE_PLANS] == ["tiny"] + assert m.is_decode_step(8) and not m.is_decode_step(9) + assert all(m.MX_GEMV_ROWS[nk] == frozenset(range(1, 9)) for nk in ALL) + assert not any(m.SWQ_ROWS.values()) + + +def test_vr_bands_kill_alone_keeps_r1vr8(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("TRTLLM_M31_DISABLE_VR_BANDS", "1") + importlib.reload(m31_switches) + try: + m = importlib.reload(modeling) + assert m.R1VR8 and not m.VR_BANDS and m.DECODE_STEP_MAX == m.DECODE_MAX_TOKENS == 48 + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_VR_BANDS", raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_rulings.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_rulings.py new file mode 100644 index 000000000000..a70737bfac75 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_rulings.py @@ -0,0 +1,160 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp2 target's default constants and TP2 row-band items, default and switched, each read at import. + +* LC_TP2 (kernels/triton_sparse_decode): OFF; ENABLE opts in, DISABLE wins. +* MXDRAFT, S9B (S9B_DEFAULT_ON), S9B_MC, S9BVR (dspark): ON; each kill switch turns only its own item off + (S9BVR's one-shot drafter all-reduce bound 32 -> 0). +* MOE16_TP2 (modeling): OFF, MOE16_ENGINE covers no rows and moe_path never picks it; ENABLE builds 9..13, DISABLE wins. +* PREFILL_BRK (modeling): ON; DISABLE turns it off. +* R1VR8: the decode bound 48 (and VR_BANDS' 64-row step); DISABLE restores the 8-row bound, VR_BANDS off with it. +* SWQ: the shared down (6144, 1536) at 17..32 rows (the rows its GEMV does not serve); DISABLE empties it. +* ROUTER48: the TF32 router at 9..48 rows and its op's cells certify every one of them; DISABLE keeps 9..16. + +Each switched case reloads m31_switches and the module under the environment, then restores both. CPU only. +""" + +import importlib +import os + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark, + m31_switches, + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + triton_sparse_decode, +) + +pytestmark = pytest.mark.cpu_only + + +@pytest.fixture +def reloaded(monkeypatch): + """``reloaded(module, **env)``: ``module`` re-imported with only ``env`` among the TRTLLM_M31_* switches; every + module touched is reloaded clean again at teardown.""" + touched = [] + + def load(module, **env): + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + importlib.reload(m31_switches) + touched.append(module) + return importlib.reload(module) + + yield load + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + importlib.reload(m31_switches) + for module in dict.fromkeys(touched): + importlib.reload(module) + + +# ---- defaults (the modules as imported) + + +def test_lc_tp2_is_off_by_default(): + assert triton_sparse_decode.LC_TP2 is False + + +def test_the_drafter_rulings_are_on_by_default(): + assert dspark.MXDRAFT and dspark.S9B_DEFAULT_ON and dspark.S9B_MC and dspark.S9BVR + assert dspark.DRAFT_AR_ONESHOT_MAX_ROWS == 32 + assert dspark.S9B_MC_MAX_ROWS == 8 + + +def test_moe16_is_off_by_default(): + m = modeling + assert m.MOE16_TP2 is False and m.MOE16_MAX_TOKENS == 0 + assert not m.MOE16_ENGINE.keys + assert all(m.moe_path(rows) != m.MOE16_ENGINE.path for rows in range(1, 65)) + + +def test_prefill_brk_is_on_by_default(): + assert modeling.PREFILL_BRK is True + + +def test_the_decode_bound_is_48_with_r1vr8(): + m = modeling + assert m.R1VR8 and m.TP2_ROWS == 8 and m.DECODE_MAX_TOKENS == 48 + assert m.TP2_BUCKETS == (8, 16, 48) + assert [p.name for p in m.DECODE_PLANS][:3] == ["tiny", "b16", "b48"] + + +def test_swq_covers_the_shared_down_rows_its_gemv_does_not(): + m = modeling + assert m.SWQ and m.SWQ_SHARED_DOWN == m.SHARED_DOWN == (6144, 1536) + assert m.SWQ_ROWS == {(6144, 1536): frozenset(range(17, 33))} + gemv = m.MX_GEMV_ROWS[m.SHARED_DOWN] + assert not (gemv & m.SWQ_ROWS[m.SHARED_DOWN]) + + +def test_router48_rows_are_certified_by_the_ops_cells(): + m = modeling + assert m.ROUTER48 and m.ROUTER_TC_GEN.keys == frozenset(range(9, 49)) + assert m.ROUTER_TC_GEN.keys <= m.ROUTER_TC_GEN.certified() + certified = m.ROUTER_TC_GEN.certified() + assert {24, 32} <= certified + + +def test_mc_ar_at_world_2_takes_no_oneshot_cluster_rows(): + assert modeling.MC_AR_CLUSTER_MAX_ROWS == 0 + + +# ---- switched + + +def test_lc_tp2_opt_in_and_kill(reloaded): + assert reloaded(triton_sparse_decode, TRTLLM_M31_ENABLE_LC_TP2="1").LC_TP2 is True + m = reloaded(triton_sparse_decode, TRTLLM_M31_ENABLE_LC_TP2="1", TRTLLM_M31_DISABLE_LC_TP2="1") + assert m.LC_TP2 is False + + +@pytest.mark.parametrize("item", ["MXDRAFT", "S9B_MC", "S9BVR"]) +def test_each_drafter_kill_switch_turns_only_its_item_off(reloaded, item): + m = reloaded(dspark, **{f"TRTLLM_M31_DISABLE_{item}": "1"}) + flags = {"MXDRAFT": m.MXDRAFT, "S9B_MC": m.S9B_MC, "S9BVR": m.S9BVR} + assert flags == {k: k != item for k in flags} + assert m.DRAFT_AR_ONESHOT_MAX_ROWS == (0 if item == "S9BVR" else 32) + + +def test_s9bvr_kill_switch_parses_strictly(reloaded): + with pytest.raises(ValueError, match="TRTLLM_M31_DISABLE_S9BVR"): + reloaded(dspark, TRTLLM_M31_DISABLE_S9BVR="maybe") + + +def test_moe16_opt_in_and_kill(reloaded): + m = reloaded(modeling, TRTLLM_M31_ENABLE_MOE16_TP2="1") + assert m.MOE16_TP2 and m.MOE16_ENGINE.keys == frozenset(range(9, 14)) + assert m.moe_path(9) == m.MOE16_ENGINE.path + m = reloaded(modeling, TRTLLM_M31_ENABLE_MOE16_TP2="1", TRTLLM_M31_DISABLE_MOE16_TP2="1") + assert not m.MOE16_TP2 and not m.MOE16_ENGINE.keys + + +def test_prefill_brk_kill(reloaded): + assert reloaded(modeling, TRTLLM_M31_DISABLE_PREFILL_BRK="1").PREFILL_BRK is False + + +def test_r1vr8_kill_restores_the_8_row_bound(reloaded): + m = reloaded(modeling, TRTLLM_M31_DISABLE_R1VR8="1") + assert not m.R1VR8 and not m.VR_BANDS + assert m.DECODE_MAX_TOKENS == m.DECODE_STEP_MAX == 8 + assert m.TP2_BUCKETS == (8,) and [p.name for p in m.DECODE_PLANS] == ["tiny"] + assert m.is_decode_step(8) and not m.is_decode_step(9) + assert m.SWQ_ROWS == {(6144, 1536): frozenset()} + + +def test_swq_kill(reloaded): + m = reloaded(modeling, TRTLLM_M31_DISABLE_SWQ="1") + assert not m.SWQ and m.SWQ_ROWS == {} + + +def test_router48_kill(reloaded): + m = reloaded(modeling, TRTLLM_M31_DISABLE_ROUTER48="1") + assert not m.ROUTER48 and m.ROUTER_TC_GEN.keys == frozenset(range(9, 17)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_weights_layout.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_weights_layout.py new file mode 100644 index 000000000000..7ee8a2d27272 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_weights_layout.py @@ -0,0 +1,90 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP2 weight table: the per-rank shapes at tp 2, layer kinds from the config, packed-NVFP4 expert +scalars as FP32 divisors.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + weights as W, +) + +pytestmark = pytest.mark.cpu_only + + +def _core(sparse_freq, moe_freq, fused=True, tp=2): + cfg = SimpleNamespace( + num_hidden_layers=len(sparse_freq), + moe_layer_freq=moe_freq, + sparse_attention_config=dict( + sparse_attention_freq=sparse_freq, sparse_num_index_heads=4, sparse_index_dim=128 + ), + head_dim=128, + num_attention_heads=64, + num_key_value_heads=4, + hidden_size=6144, + dense_intermediate_size=12288, + shared_intermediate_size=3072, + n_shared_experts=1, + intermediate_size=3072, + num_local_experts=128, + vocab_size=200064, + torch_dtype=torch.bfloat16, + ) + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=cfg, + mapping=SimpleNamespace(tp_size=tp, tp_rank=0), + sparse_attention_config=SimpleNamespace(fuse_qkv_index_projection=fused), + ) + ) + + +def test_the_tp2_per_rank_shapes(): + """32 query / 2 KV / 2 index heads per rank: the five-way projection is 4992 rows (the producer's 39 packed heads), + the three-way 4608, o_proj reads 4096 columns; dense MLP 6144 and shared / routed experts 1536 per rank (the + m3_mx_gemv TP2 shapes and the MoE engine's INTER / SHARED_INTER).""" + d = W.MODEL_WEIGHTS.dims(_core([1] * 4, [0, 1, 1, 1])) + assert (d.tp, d.heads, d.kv_heads, d.index_heads, d.head_dim, d.index_dim) == ( + 2, + 32, + 2, + 2, + 128, + 128, + ) + assert d.qkv_sparse_rows == 4992 == 39 * 128 + assert d.qkv_dense_rows == 4608 + assert d.q_width == 4096 + assert (2 * d.dense_inter, d.dense_inter) == (12288, 6144) + assert (2 * d.shared_inter, d.shared_inter) == (3072, 1536) + assert d.moe_inter == 1536 + + +def test_layer_kinds_come_from_the_config_independently(): + """Sparse attention on every layer while layers 0-1 keep the dense MLP.""" + d = W.MODEL_WEIGHTS.dims(_core([1] * 6, [0, 0, 1, 1, 1, 1])) + assert list(W._sparse_attn_layers(d)) == [0, 1, 2, 3, 4, 5] + assert list(W._dense_mlp_layers(d)) == [0, 1] + assert list(W._moe_layers(d)) == [2, 3, 4, 5] + assert list(W._five_way_qkv_layers(d)) == [0, 1, 2, 3, 4, 5] + assert list(W._three_way_qkv_layers(d)) == [] + + +def test_unfused_projection_uses_the_three_way_qkv_everywhere(): + d = W.MODEL_WEIGHTS.dims(_core([0, 0, 1, 1], [0, 0, 1, 1], fused=False)) + assert list(W._three_way_qkv_layers(d)) == [0, 1, 2, 3] + assert list(W._unfused_sparse_layers(d)) == [2, 3] + assert list(W._five_way_qkv_layers(d)) == [] + + +def test_expert_scalars_are_the_fp32_reciprocal_of_the_stored_multiplier(): + stored = torch.tensor(12345.678, dtype=torch.float32) + out = W._divisor(stored, None) + assert out.dtype == torch.float32 + assert out.view(torch.int32).item() == torch.reciprocal(stored).view(torch.int32).item() + with pytest.raises(ValueError): + W._divisor(torch.tensor(0.0), None) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_work_counters.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_work_counters.py new file mode 100644 index 000000000000..df4979f89ce9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2_work_counters.py @@ -0,0 +1,182 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Gate e on the tp2 target: the per-rank engagement line and work-done counter of every ported item. + +* kernels/m3_report.work: the first call logs "[M3 sm107 tp2] rank N: engaged (); work 1 (host count: + ...)", every power of two logs the running count, nothing else is logged, and the exit line holds every item's final + count; every line says what the count counts (captures + eager calls, replays not counted). +* modeling.taken counts every taken fast path under its path name; the multicast all-reduce's line carries world size, + kernel (world 2: the 8-CTA cluster kernel) and the per-rank call count. +* the DSpark hooks: a wrapped hook counts only the calls that did the work (and the items riding on it); the S9b op + counts S9b on every call and S9B-MC only when the multicast form took the operands. +* the target tag line and the eng kit's patterns (eng_tp2_v3) match what the code formats. CPU only. +""" + +import re +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark, + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_report, +) + +pytestmark = pytest.mark.cpu_only + + +@pytest.fixture +def lines(monkeypatch): + """Every work / engagement line logged (m3_report's and modeling's), with rank 0 and fresh counters.""" + out = [] + monkeypatch.setattr(m3_report, "_info", out.append) + monkeypatch.setattr(m3_report, "_rank", lambda: 0) + monkeypatch.setattr(modeling, "mpi_rank", lambda: 0) + monkeypatch.setattr(modeling, "logger", SimpleNamespace(info=out.append, warning=out.append)) + m3_report.reset_work() + modeling.reset_logged() + yield out + m3_report.reset_work() + modeling.reset_logged() + + +def test_first_call_and_powers_of_two_log_the_count(lines): + for _ in range(9): + m3_report.work("UT_ITEM", "unit detail") + assert lines == [ + f"[M3 sm107 tp2] rank 0: UT_ITEM engaged (unit detail); work 1 ({m3_report.HOST_COUNT})", + f"[M3 sm107 tp2] rank 0: UT_ITEM work 2 ({m3_report.HOST_COUNT})", + f"[M3 sm107 tp2] rank 0: UT_ITEM work 4 ({m3_report.HOST_COUNT})", + f"[M3 sm107 tp2] rank 0: UT_ITEM work 8 ({m3_report.HOST_COUNT})", + ] + assert m3_report.work_count("UT_ITEM") == 9 + assert "replays not counted" in m3_report.HOST_COUNT + + +def test_the_exit_line_holds_every_final_count(lines): + for item, n in (("B_ITEM", 3), ("A_ITEM", 1)): + for _ in range(n): + m3_report.work(item) + lines.clear() + m3_report.log_work_totals() + assert lines == [ + f"[M3 sm107 tp2] rank 0: work-done final ({m3_report.HOST_COUNT}): A_ITEM=1, B_ITEM=3" + ] + + +def test_items_count_independently(lines): + m3_report.work("X1") + m3_report.work("X2") + m3_report.work("X1") + assert m3_report.work_totals() == {"X1": 2, "X2": 1} + + +def test_taken_counts_the_path(lines): + for _ in range(3): + modeling.taken("m3_ut_path", 8, "detail") + assert m3_report.work_count("m3_ut_path") == 3 + assert "[M3 dispatch] rank 0: m3_ut_path taken at 8 rows (detail)" in lines + assert any( + re.match(r"\[M3 sm107 tp2\] rank 0: m3_ut_path engaged \(8 rows: detail\)", s) + for s in lines + ) + + +@pytest.mark.parametrize( + "world,rows,kernel", + [(2, 1, "8-CTA cluster kernel"), (2, 32, "8-CTA cluster kernel"), (4, 16, "row kernel")], +) +def test_the_mc_allreduce_line_names_world_kernel_and_call(lines, world, rows, kernel): + mapping = SimpleNamespace(tp_size=world) + modeling._mc_taken(mapping, rows, "x") + modeling._mc_taken(mapping, rows, "x") + taken = [s for s in lines if "mc_allreduce_norm taken" in s] + assert taken == [ + f"[M3 dispatch] rank 0: mc_allreduce_norm taken at {rows} rows (x, world {world}, {kernel}, call 1)" + ] + assert m3_report.work_count(modeling.MC_ALLREDUCE.path) == 2 + eng_x = ( + r"\[M3 dispatch\] rank [0-9]+: mc_allreduce_norm taken at [0-9]+ rows " + r"\(x.*, world 2, 8-CTA cluster kernel, call [0-9]+\)" + ) + assert bool(re.search(eng_x, taken[0])) is (world == 2) + + +def test_the_prepushed_line_matches_the_eng_kit(lines): + modeling._mc_taken(SimpleNamespace(tp_size=2), 8, "prepushed 2") + eng = ( + r"\[M3 dispatch\] rank [0-9]+: mc_allreduce_norm taken at [0-9]+ rows " + r"\(prepushed 2, world 2, 8-CTA cluster kernel, call [0-9]+\)" + ) + assert any(re.search(eng, s) for s in lines) + + +@pytest.mark.parametrize( + "result,counted", [(object(), True), (True, True), (None, False), (False, False)] +) +def test_a_wrapped_hook_counts_only_the_work_it_did(lines, result, counted): + hook = dspark._counted("UT_HOOK", lambda *a, **k: result, ("UT_RIDER", True), ("UT_OFF", False)) + assert hook(1, x=2) is result + assert m3_report.work_count("UT_HOOK") == int(counted) + assert m3_report.work_count("UT_RIDER") == int(counted) + assert m3_report.work_count("UT_OFF") == 0 + assert dspark._counted("UT_NONE", None) is None + + +def test_the_s9b_op_counts_s9b_always_and_s9b_mc_when_taken(lines): + mapping = SimpleNamespace(tp_size=2) + taken_out = (torch.zeros(2, 4), torch.zeros(2, 4)) + op = dspark._s9b_count_op(mapping, lambda p, r, w, e: taken_out if p.shape[0] <= 8 else None) + assert op(torch.zeros(8, 4), None, None, 1e-6) is taken_out + assert op(torch.zeros(16, 4), None, None, 1e-6) is None + assert dspark._s9b_count_op(mapping, None)(torch.zeros(4, 4), None, None, 1e-6) is None + assert m3_report.work_count("S9B") == 3 and m3_report.work_count("S9B_MC") == 1 + assert any( + re.search( + r"S9B_MC engaged .*world 2, drafter form \(opts 16\) on the 8-CTA cluster kernel", s + ) + for s in lines + ) + + +def test_the_target_tag_line_matches_the_eng_kit(): + """The tag line is formatted in the producer's decode branch; the eng kit's ``target`` pattern matches it.""" + with open(modeling.__file__) as f: + text = f.read() + assert ( + 'f"[M3 sm107 tp2] fast path engaged: m3_producer (decode, rank {mpi_rank()}, {rows} rows' + in text + ) + sample = ( + "[M3 sm107 tp2] fast path engaged: m3_producer (decode, rank 1, 8 rows, nvfp4_tp2: ...)" + ) + assert re.search(r"\[M3 sm107 tp2\] fast path engaged: m3_producer \(decode", sample) + + +def test_the_tag_is_the_eng_kits(): + assert m3_report.TAG == "[M3 sm107 tp2]" + + +def test_c6_the_producer_covers_the_tp2_heads(): + """C6: the producer gate holds this target's per-rank heads (32 query, 2 KV, 2 index: the TP2 op's), not the TP4 + heads (16, 1, 1) it was copied with, under which no TP2 sparse layer took ``m3_producer`` (no target tag line).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention import ( # noqa: E501 + m3_producer as op, + ) + + assert modeling.PRODUCER_HEADS == (op.HEADS_Q, op.HEADS_KV, op.HEADS_INDEX) == (32, 2, 2) + tp2 = SimpleNamespace(main_kv_is_nvfp4=True, heads=32, kv_heads=2, index_heads=2) + assert modeling._producer_covered(tp2) + tp4 = SimpleNamespace(main_kv_is_nvfp4=True, heads=16, kv_heads=1, index_heads=1) + assert not modeling._producer_covered(tp4) + fp8 = SimpleNamespace(main_kv_is_nvfp4=False, heads=32, kv_heads=2, index_heads=2) + assert not modeling._producer_covered(fp8) + with open(modeling.__file__) as f: + text = f.read() + # Both the coverage and the call site read the one constant; the TP4 heads are gone. + assert text.count("(core.heads, core.kv_heads, core.index_heads) == PRODUCER_HEADS") == 2 + assert "(core.heads, core.kv_heads, core.index_heads) == (16, 1, 1)" not in text diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2agg_sp.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2agg_sp.py new file mode 100644 index 000000000000..5ee6c96fcc5a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2agg_sp.py @@ -0,0 +1,197 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2AGG_SP on CPU stand-ins: the band, the owned-row split and its 128-row scale-factor tiles (the tp2ctx_ipc +workspace's split_row / own_sf_slice), the "logits" gather kind, the opt-in dispatch (TRTLLM_M31_ENABLE_TP2AGG_SP=1), +the residual unshard, the router / finalize reads of the gathered products and the SP_OVL hand-off. The collectives +and the bitwise boundary are gated on GPU (det + TRTLLM_M31_TP2CTX_SP_CHECK).""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + modeling as m, +) + +H = 6144 + + +def test_opt_in_with_both_kill_switches() -> None: + assert m.TP2AGG_SP == ( + m.m31_enabled("TP2AGG_SP") + and not m.m31_disabled("TP2AGG_SP") + and not m.m31_disabled("AGG_SP") + ) + + +@pytest.mark.parametrize("on", [False, True]) +def test_band(monkeypatch, on) -> None: + monkeypatch.setattr(m, "TP2AGG_SP", on) + assert not m._sp_supports(m.SP_MIN_ROWS - 1) + assert m._sp_supports(m.SP_MIN_ROWS) == on + assert m._sp_supports(m.SP_MAX_ROWS) == on + assert not m._sp_supports(m.SP_MAX_ROWS + 1) + + +@pytest.mark.parametrize("rows", [4097, 5461, 5462, 8192, 12288, 12697, 16383, 16384]) +def test_owned_rows_tile_the_step(rows) -> None: + (o0, c0), (o1, c1) = m._sp_own_rows(rows, 0), m._sp_own_rows(rows, 1) + assert o0 == 0 and o1 == c0 and o1 + c1 == rows + assert c0 % m.SP_SF_TILE_ROWS == 0 and abs(c0 - rows / 2) <= m.SP_SF_TILE_ROWS / 2 + if rows % 256 == 0: + assert c0 == c1 == rows // 2 + + +@pytest.mark.parametrize("rows", [5462, 12697, 16384]) +def test_sf_bytes_are_whole_tiles_and_cover_the_padded_step(rows) -> None: + per = (H // 32 + 3) // 4 * 4 + a, b = m._sp_own_sf_bytes(rows, 0, H), m._sp_own_sf_bytes(rows, 1, H) + rows_pad = (rows + 127) // 128 * 128 + assert a[0] == 0 and a[1] == b[0] and b[1] == rows_pad * per + assert a[1] % (128 * per) == 0 + + +@pytest.mark.parametrize("rows", [5462, 12697]) +def test_logits_kind_is_512_bytes_per_owned_row(rows) -> None: + a, b = m._sp_logits_bytes(rows, 0), m._sp_logits_bytes(rows, 1) + assert m.SP_LOGITS_ROW_BYTES == 128 * torch.tensor([], dtype=torch.float32).element_size() + assert a[0] == 0 and a[1] == b[0] and b[1] == rows * 512 + + +class _FakeWs: + """CPU stand-in for tp2ctx_ipc.Tp2ctxIpcWorkspace: the same row / scale ownership, local views only.""" + + def __init__(self, rank, max_tokens=16384, hidden=8): + self.rank, self.max_tokens, self.hidden = rank, max_tokens, hidden + self.views = {} + self.pushed = [] + + def supports(self, rows, *, nccl_only=True): + return 256 <= rows <= self.max_tokens + + def own_rows(self, rows): + return m._sp_own_rows(rows, self.rank) + + def own_sf_slice(self, rows): + return slice(*m._sp_own_sf_bytes(rows, self.rank, self.hidden * 32)) + + def gather_view(self, kind, rows): + shapes = { + "logits": ((self.max_tokens, 128), torch.float32), + "bf16": ((self.max_tokens, self.hidden), torch.bfloat16), + } + shape, dtype = shapes.get(kind, ((self.max_tokens, self.hidden), torch.uint8)) + if kind not in self.views: + self.views[kind] = torch.zeros(shape, dtype=dtype) + return self.views[kind][:rows] + + def push_owned(self, kinds, rows): + self.pushed.append((tuple(kinds), rows)) + + +@pytest.fixture +def sp_on(monkeypatch): + monkeypatch.setattr(m, "TP2AGG_SP", True) + monkeypatch.setattr(m, "is_torch_compiling", lambda: False) + monkeypatch.setattr(m, "is_in_breakable_cuda_graph", lambda: False) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + state = {} + monkeypatch.setattr(m, "_SP_STATE", state) + return state + + +def test_no_workspace_keeps_todays_boundary(sp_on) -> None: + assert m._sp_ws(torch.zeros(8192, 4)) is None + + +@pytest.mark.parametrize("rank", [0, 1]) +def test_ws_dispatch_is_the_band(sp_on, rank) -> None: + sp_on["ws"] = _FakeWs(rank) + assert m._sp_ws(torch.zeros(8192, 4)) is sp_on["ws"] + assert m._sp_ws(torch.zeros(m.SP_MIN_ROWS - 1, 4)) is None + assert m._sp_ws(torch.zeros(8192)) is None + + +def test_ws_dispatch_off_when_capturing(sp_on, monkeypatch) -> None: + sp_on["ws"] = _FakeWs(0) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: True) + assert m._sp_ws(torch.zeros(8192, 4)) is None + + +@pytest.mark.parametrize("rank", [0, 1]) +def test_logits_put_writes_only_the_owned_rows(sp_on, rank) -> None: + ws, rows = _FakeWs(rank), 12697 + first, count = ws.own_rows(rows) + owned = torch.randn(count, 128) + view = m._sp_put(ws, "logits", rows, owned) + assert view.dtype == torch.float32 and view.shape == (rows, 128) + assert torch.equal(view[first : first + count], owned) + others = torch.cat([view[:first], view[first + count :]]) + assert int(others.count_nonzero()) == 0 + + +@pytest.mark.parametrize("rank", [0, 1]) +def test_unshard_gathers_an_owned_residual(sp_on, rank) -> None: + ws = sp_on["ws"] = _FakeWs(rank) + rows = 8192 + first, count = ws.own_rows(rows) + x = torch.zeros(rows, 8, dtype=torch.bfloat16) + owned = torch.randn(count, 8).to(torch.bfloat16) + full = m._sp_unshard(owned, x) + assert full.shape == (rows, 8) and torch.equal(full[first : first + count], owned) + assert ws.pushed == [(("bf16",), rows)] + # a full residual (or one outside an SP step) is left alone + assert m._sp_unshard(full, x) is full + small = owned[:7] + assert m._sp_unshard(small, x) is small and ws.pushed == [(("bf16",), rows)] + + +def test_router_and_finalize_read_the_gathered_products(sp_on) -> None: + keyed = torch.zeros(8192, 8, dtype=torch.bfloat16) + logits = torch.randn(8192, 128) + m._SP_LOGITS.publish(keyed, logits, None) + assert m._router_logits(None, 3, keyed) is logits + assert m._SP_LOGITS.pending is None + sp_on["keyed"] = keyed.data_ptr() + assert m._sp_keyed(keyed) and not m._sp_keyed(keyed.clone()) + + +def test_ovl_take_matches_the_keyed_view(sp_on) -> None: + class _Ev: + pass + + keyed = torch.zeros(12288, 8, dtype=torch.bfloat16) + ovl = m._SpOvl(_Ev(), (0, 6144), (6144, 6144), slice(0, 1), slice(1, 2)) + sp_on["ovl"] = (keyed.data_ptr(), ovl) + assert m._sp_ovl_take(keyed) is ovl + assert sp_on["ovl"] is None and m._sp_ovl_take(keyed) is None + + +@pytest.mark.parametrize( + "fused,decode,tuned,allowed", + [ + (True, False, False, True), + (False, False, False, False), + (True, True, False, False), + (True, False, True, False), + ], +) +def test_placeholder_requires_prequant_consumers(monkeypatch, fused, decode, tuned, allowed): + core = SimpleNamespace(fused_projection=fused, mxfp8=None) + mode = SimpleNamespace(decode_graph=decode, tuned_quantize_gemm=tuned) + workspace = object() + monkeypatch.setattr(m, "_mx_mode", lambda _: mode) + monkeypatch.setattr(m, "_sp_ws", lambda _: workspace) + assert m._sp_consumer_ws(core, None) is (workspace if allowed else None) + + +@pytest.mark.parametrize("rows", [4097, 5461, 5462, 8192, 12288, 16384, 16385]) +def test_agg_band_override_preserves_workspace_capacity(monkeypatch, rows): + from tensorrt_llm._torch._experimental.modeling_v2.catalog.comm import tp2ctx_ipc + + monkeypatch.setattr(tp2ctx_ipc, "_nccl_fallback_bytes", lambda: 64 * 1024 * 1024) + workspace = SimpleNamespace(max_tokens=16384, hidden=6144) + supports = tp2ctx_ipc.Tp2ctxIpcWorkspace.supports + assert supports(workspace, rows, nccl_only=False) == (rows <= 16384) + assert supports(workspace, rows) == (5462 <= rows <= 16384) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_decode.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_decode.py new file mode 100644 index 000000000000..c32c7fe229cd --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_decode.py @@ -0,0 +1,90 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Decode-plan dispatch coverage. Numerical and per-rank model gates are separate.""" + +import importlib +import itertools +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( + m31_switches, + modeling, +) + +pytestmark = pytest.mark.cpu_only + + +@pytest.mark.parametrize("r1,bands,wide", itertools.product((False, True), repeat=3)) +def test_decode_plan_switches_and_all_rows(monkeypatch, r1, bands, wide): + try: + with monkeypatch.context() as patch: + for name, enabled in (("R1VR8", r1), ("VR_BANDS", bands), ("TP2WIDE_DECODE", wide)): + patch.setenv("TRTLLM_M31_DISABLE_" + name, "0" if enabled else "1") + importlib.reload(m31_switches) + m = importlib.reload(modeling) + bound = 8 if not r1 else 48 if not bands else 64 if not wide else 448 + assert m.DECODE_STEP_MAX == bound + for rows in range(1, 450): + assert m.is_decode_step(rows) == (rows <= bound) + if rows > bound: + with pytest.raises(AssertionError): + m.plan_for(rows) + continue + plan = m.plan_for(rows) + if rows <= 64: + expected = ( + ("tiny", 8) + if rows <= 8 + else ("b16", 16) + if rows <= 16 + else ("b48", 48) + if rows <= 48 + else ("wide", 64) + ) + assert (plan.name, plan.max_tokens) == expected + else: + assert (plan.name, plan.max_tokens) == ("wide448", 448) + flags = (plan.attn_mc_ar, plan.moe_mc_ar, plan.qkv_prequant) + assert flags == ((True, True, True) if rows <= 48 else (False, False, False)) + finally: + importlib.reload(m31_switches) + importlib.reload(modeling) + + +@pytest.mark.parametrize("rows", [64, 65, 128, 184, 224, 256, 448, 449]) +@pytest.mark.parametrize("enabled", [False, True]) +def test_wide_input_keeps_original_operands_and_narrow_route(monkeypatch, rows, enabled): + ids, table, weight, raw, normed = (object() for _ in range(5)) + calls = [] + + def embedding(actual_ids, actual_table): + assert actual_ids is ids and actual_table is table + calls.append("embedding") + return raw + + def rmsnorm(actual_raw, actual_weight, eps): + assert actual_raw is raw and actual_weight is weight and eps == 1e-6 + calls.append("norm") + return normed + + monkeypatch.setattr(modeling, "TP2WIDE_DECODE", enabled) + monkeypatch.setattr(modeling, "DECODE_STEP_MAX", 448 if enabled else 64) + monkeypatch.setattr(modeling, "m3_embedding", embedding) + monkeypatch.setattr(modeling, "flashinfer_rmsnorm", rmsnorm) + monkeypatch.setattr(modeling, "taken", lambda *args: None) + monkeypatch.setattr(modeling, "uncovered", lambda *args: None) + target = modeling.DecodeTarget.__new__(modeling.DecodeTarget) + target.core = SimpleNamespace( + w={"embed": table, "l0_in_norm": weight}, + rms_eps=1e-6, + mapping=SimpleNamespace(tp_rank=1), + ) + out, residual = target._embed(ids, rows) + if enabled and 64 < rows <= 448: + assert out is normed and residual is raw + assert calls == ["embedding", "norm"] + else: + assert out is raw and residual is None + assert calls == ["embedding"] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_gemm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_gemm.py new file mode 100644 index 000000000000..1958449b066d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_gemm.py @@ -0,0 +1,90 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2WIDE_GEMM: at 33..48 rows the dense MLP's MXFP8 projections leave the GEMV for the pinned FlashInfer GEMM +(MX-TAC), every other projection keeps its GEMV rows, rows <= 32 are unchanged, and ``linear_path`` never names a +GEMM ``_linear`` does not run (the VR_GEMV48 / LINEAR_CUBLASLT tables are empty at TP2). +TRTLLM_M31_DISABLE_TP2WIDE_GEMM=1 restores the GEMV rows and the two tables.""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + modeling, +) + +DENSE = (modeling.DENSE_GATE_UP, modeling.DENSE_DOWN) +OTHERS = ( + modeling.QKV_SPARSE, + modeling.QKV, + modeling.O_PROJ, + modeling.SHARED_GATE_UP, + modeling.SHARED_DOWN, +) +STEP_ROWS = range(1, modeling.DECODE_STEP_MAX + 1) + + +def _tables(m) -> dict: + return { + name: dict(getattr(m, name)) + for name in ("MX_GEMV_ROWS", "GEMV48_ROWS", "LINEAR_CUBLASLT_ROWS") + } + + +# The default module's view, recorded at collection before any reload. +_ON = _tables(modeling) + + +@pytest.fixture +def killed(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("TRTLLM_M31_DISABLE_TP2WIDE_GEMM", "1") + importlib.reload(m31_switches) + try: + yield importlib.reload(modeling) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_TP2WIDE_GEMM", raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) + + +def test_dense_mlp_takes_the_gemm_at_33_to_48_rows() -> None: + m = modeling + assert m.TP2WIDE_GEMM + wide = frozenset(range(33, m.DECODE_MAX_TOKENS + 1)) + for nk in DENSE: + assert m.MX_GEMV_ROWS[nk] == frozenset(range(1, 33)), nk + for rows in wide: + assert m.linear_path(*nk, rows, True, False) == m.PREQUANT_GEMM_PATH, (nk, rows) + for rows in range(1, 33): + assert m.linear_path(*nk, rows, True, False) == m.MX_GEMV_PATH, (nk, rows) + + +def test_other_projections_and_small_rows_unchanged(killed) -> None: + """Every projection but the dense MLP keeps its GEMV rows; rows <= 32 route the same with the item killed.""" + off = _tables(killed) + for nk in OTHERS: + assert _ON["MX_GEMV_ROWS"][nk] == off["MX_GEMV_ROWS"][nk], nk + for nk in DENSE: + assert _ON["MX_GEMV_ROWS"][nk] == off["MX_GEMV_ROWS"][nk] - frozenset(range(33, 49)), nk + + +def test_no_dead_wide_path() -> None: + """``_linear`` has no branch for the GEMV48 / cuBLASLt paths at TP2: no row count may select them.""" + m = modeling + assert not m.GEMV48_ROWS and not m.LINEAR_CUBLASLT_ROWS + for nk in DENSE + OTHERS: + for rows in STEP_ROWS: + for prequant in (True, False): + assert m.linear_path(*nk, rows, prequant, False) not in ( + m.GEMV48_PATH, + m.CUBLASLT_GEMM_PATH, + ), (nk, rows, prequant) + + +def test_kill_switch_restores_today(killed) -> None: + m = killed + assert not m.TP2WIDE_GEMM + for nk in DENSE: + assert m.MX_GEMV_ROWS[nk] == frozenset(range(1, m.DECODE_MAX_TOKENS + 1)), nk + assert bool(m.LINEAR_CUBLASLT_ROWS) == m.LINEAR_CUBLASLT diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_prefill_attn.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_prefill_attn.py new file mode 100644 index 000000000000..ec902253c091 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_prefill_attn.py @@ -0,0 +1,266 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TP2WIDE_PREFILL_ATTN (MSA): the agg target's NVFP4 sparse prefill takes the tp2ctx items of the shared msa_prefill. + +* TP2WIDE_COMBINE_MXFP8: with the in-place output and CTXNORMQ, every backend gets the o_proj MXFP8 sink (12288-row + minimum); the sink publishes in the CTXNORMQ slot, taken only by the combine's own output. CPU only. + +* Every layer's attention backend opts in to the in-place combine (``msa_sparse_out_in_place``) and the heaviest-first + k2q schedule (``msa_k2q_schedule_order = "lpt"``); TRTLLM_M31_DISABLE_TP2WIDE_SPARSE_OUT / + TRTLLM_M31_DISABLE_TP2WIDE_SCHED_LPT each leave its attribute unset (the stock copy / order). CPU only. +* On sm_107 with the MSA build loaded: run_msa_nvfp4_sparse_gqa with the opt-ins, and with the E4M3X forward (when the + BF16X fork with E4M3X is installed), writes the same bytes as the stock call (E4M3X killed, no opt-ins) at a 16K chunk + past a long prefix, a 4-request mixed chunk and a short chunk. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + modeling, +) + +HQ, HKV, D, PAGE, TOPK = 32, 2, 128, 128, 16 + + +@pytest.mark.cpu_only +def test_switches_are_read_once_at_import() -> None: + assert modeling.TP2WIDE_SPARSE_OUT == (not m31_switches.m31_disabled("TP2WIDE_SPARSE_OUT")) + assert modeling.TP2WIDE_SCHED_LPT == (not m31_switches.m31_disabled("TP2WIDE_SCHED_LPT")) + assert modeling.TP2WIDE_COMBINE_MXFP8 == ( + not m31_switches.m31_disabled("TP2WIDE_COMBINE_MXFP8") + ) + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("sparse_out", [True, False]) +@pytest.mark.parametrize("combine_mxfp8", [True, False]) +@pytest.mark.parametrize("ctxnormq", [True, False]) +def test_combine_mxfp8_sink_follows_its_switches( + monkeypatch, sparse_out, combine_mxfp8, ctxnormq +) -> None: + """TP2WIDE_COMBINE_MXFP8: the sink is set only with the in-place output, its own switch and CTXNORMQ (its slot's + consumer), and carries the 12288-row minimum msa_prefill reads.""" + monkeypatch.setattr(modeling, "TP2WIDE_SPARSE_OUT", sparse_out) + monkeypatch.setattr(modeling, "TP2WIDE_COMBINE_MXFP8", combine_mxfp8) + monkeypatch.setattr(modeling, "CTXNORMQ", ctxnormq) + backends = [SimpleNamespace() for _ in range(3)] + modeling._prefill_attention_opt_ins(backends) + for b in backends: + sink = getattr(b, "msa_sparse_out_mxfp8_sink", None) + if sparse_out and combine_mxfp8 and ctxnormq: + assert sink is modeling._combine_mxfp8_sink + assert sink.min_rows == 12288 + else: + assert sink is None + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("context_rows,gen_rows", [(128, 0), (130, 3), (128, 8)]) +def test_combine_mxfp8_publishes_full_step_after_tail_repair( + monkeypatch, context_rows, gen_rows +) -> None: + """Context publication waits for the generation rows, including a partial 128-row scale tile.""" + out = torch.zeros(context_rows + gen_rows, 128, dtype=torch.bfloat16) + state = {"pending": {"output": out, "context_rows": context_rows}} + monkeypatch.setattr(modeling, "_COMBINE_MXFP8_STATE", state) + monkeypatch.setattr(modeling, "_COMBINE_MXFP8_CHECK", False) + modeling._CTXNORMQ_MX.clear() + quantized_rows = [] + + def quantize(tail, swizzled): + assert swizzled + quantized_rows.append(tail.shape[0]) + return ( + torch.full(tail.shape, 7, dtype=torch.uint8).view(torch.float8_e4m3fn), + torch.full((((tail.shape[0] + 127) // 128 * 128) * 4,), 9, dtype=torch.uint8), + ) + + monkeypatch.setattr(torch.ops.trtllm, "mxfp8_quantize", quantize) + context = out[:context_rows] + xq, xsf = modeling._combine_mxfp8_allocate(context) + xq.fill_(3) + xsf.fill_(4) + modeling._combine_mxfp8_sink(context, xq, xsf) + assert modeling._CTXNORMQ_MX.pending is None + modeling._combine_mxfp8_finish(out) + assert "pending" not in state + pair = modeling._CTXNORMQ_MX.take(out) + assert pair is not None and pair[0].shape == out.shape + tail_start = context_rows // 128 * 128 + assert torch.all(pair[0][:tail_start].view(torch.uint8) == 3) + if tail_start < out.shape[0]: + assert quantized_rows == [out.shape[0] - tail_start] + assert torch.all(pair[0][tail_start:].view(torch.uint8) == 7) + assert torch.all(pair[1][tail_start * 4 :] == 9) + else: + assert not quantized_rows + assert modeling._CTXNORMQ_MX.take(out) is None + + +@pytest.mark.cpu_only +def test_combine_mxfp8_decline_and_missing_callback_cannot_publish(monkeypatch) -> None: + out = torch.zeros(128, 128, dtype=torch.bfloat16) + state = {} + monkeypatch.setattr(modeling, "_COMBINE_MXFP8_STATE", state) + modeling._CTXNORMQ_MX.clear() + assert modeling._combine_mxfp8_allocate(out) is None + state["pending"] = {"output": out, "context_rows": 128} + modeling._combine_mxfp8_allocate(out) + modeling._combine_mxfp8_finish(out) + assert modeling._CTXNORMQ_MX.take(out) is None + assert "pending" not in state + + +@pytest.mark.cpu_only +def test_combine_mxfp8_refuses_a_different_context_tensor(monkeypatch) -> None: + out = torch.zeros(128, 128, dtype=torch.bfloat16) + monkeypatch.setattr( + modeling, "_COMBINE_MXFP8_STATE", {"pending": {"output": out, "context_rows": 128}} + ) + with pytest.raises(RuntimeError, match="context output differs"): + modeling._combine_mxfp8_allocate(out.clone()) + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("sparse_out", [True, False]) +@pytest.mark.parametrize("sched_lpt", [True, False]) +def test_opt_ins_follow_their_kill_switches(monkeypatch, sparse_out, sched_lpt) -> None: + monkeypatch.setattr(modeling, "TP2WIDE_SPARSE_OUT", sparse_out) + monkeypatch.setattr(modeling, "TP2WIDE_SCHED_LPT", sched_lpt) + backends = [SimpleNamespace() for _ in range(3)] + modeling._prefill_attention_opt_ins(backends) + for b in backends: + assert getattr(b, "msa_sparse_out_in_place", False) is sparse_out + assert getattr(b, "msa_k2q_schedule_order", None) == ("lpt" if sched_lpt else None) + + +def _selection(qn, offset, gen): + """[qn, HKV, TOPK] ascending block ids with the row's local block, -1 padded (correlated across rows).""" + nvalid = (offset + torch.arange(qn, device="cuda")) // PAGE + 1 + nb = int(nvalid.max()) + out = torch.full((HKV, qn, TOPK), -1, dtype=torch.int32, device="cuda") + for h in range(HKV): + score = torch.randn(nb, generator=gen, device="cuda")[None, :].expand(qn, nb).clone() + score += 0.5 * torch.randn(qn, nb, generator=gen, device="cuda") + score[torch.arange(nb, device="cuda")[None, :] >= (nvalid - 1)[:, None]] = float("-inf") + k = min(TOPK - 1, nb - 1) + sel = (nvalid - 1)[:, None] + if k > 0: + pick = torch.topk(score, k, dim=1).indices + pick = torch.where( + torch.gather(score, 1, pick) == float("-inf"), torch.full_like(pick, -1), pick + ) + sel = torch.cat([pick, sel], dim=1) + sel, _ = torch.sort(torch.where(sel < 0, torch.full_like(sel, 1 << 30), sel), dim=1) + out[h, :, : sel.shape[1]] = torch.where( + sel == (1 << 30), torch.full_like(sel, -1), sel + ).int() + return out.permute(1, 0, 2).contiguous() + + +def _case(reqs, seed): + gen = torch.Generator(device="cuda").manual_seed(seed) + qo = [q for q, _ in reqs] + kv = [p + q for q, p in reqs] + pages_per = [-(-k // PAGE) for k in kv] + pages = sum(pages_per) + 8 + q = (torch.randn(sum(qo), HQ, D, generator=gen, device="cuda") * 0.5).to(torch.float8_e4m3fn) + k = torch.randint( + 0, 256, (pages, HKV, PAGE, D // 2), dtype=torch.uint8, generator=gen, device="cuda" + ) + v = torch.randint( + 0, 256, (pages, HKV, PAGE, D // 2), dtype=torch.uint8, generator=gen, device="cuda" + ) + sc = (torch.rand(pages, 2, HKV, PAGE, D // 16, generator=gen, device="cuda") + 0.5).to( + torch.float8_e4m3fn + ) + perm = torch.randperm(pages, generator=gen, device="cuda").int() + bt = torch.zeros((len(reqs), max(pages_per)), dtype=torch.int32, device="cuda") + c = 0 + for b, n in enumerate(pages_per): + bt[b, :n] = perm[c : c + n] + c += n + scales = torch.zeros((2, 4), dtype=torch.float32, device="cuda") + scales[0, 0], scales[1, 0] = 0.02, 0.03 + md = SimpleNamespace( + _msa_live_batch=len(reqs), + _msa_max_q_len=max(qo), + _msa_max_kv_len_all=max(kv), + _msa_total_k=sum(kv), + _msa_total_k_rows=sum(pages_per), + _msa_prefill_identity=None, + _msa_identity_csr=None, + _msa_context_prefix_bounds=None, + msa_cu_q_lens=torch.tensor( + [sum(qo[:i]) for i in range(len(qo) + 1)], dtype=torch.int32, device="cuda" + ), + msa_cu_kv_lens=torch.tensor( + [sum(kv[:i]) for i in range(len(kv) + 1)], dtype=torch.int32, device="cuda" + ), + msa_block_table=bt, + msa_seq_lens_cuda=torch.tensor(kv, dtype=torch.int32, device="cuda"), + ) + kbi = torch.cat([_selection(qn, p, gen) for qn, p in reqs], dim=0) + return q, k, v, sc.view(torch.uint8), kbi, md, scales[0, :1], scales[1, :1] + + +@pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="sm_107 only", +) +@pytest.mark.parametrize( + "reqs", + [ + [(16384, 98304)], + [(4096, 98304), (4096, 90112), (4096, 106496), (4096, 102400)], + [(2048, 1000)], + ], + ids=["c16k-p96k", "mix4x4k", "c2k-short"], +) +def test_opt_ins_and_e4m3x_are_bitwise_the_stock_call(monkeypatch, reqs) -> None: + from tensorrt_llm._torch.attention.backends.fmha import msa_prefill as mp + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + require_msa_module, + ) + + require_msa_module() + q, k, v, sc, kbi, md, kg, vg = _case(reqs, seed=sum(p for _, p in reqs) + len(reqs)) + outs = {} + arms = { + "stock": (dict(), "1"), + "opt_ins": (dict(in_place=True, schedule_order="lpt"), "1"), + "opt_ins_e4m3x": (dict(in_place=True, schedule_order="lpt"), "0"), + } + for arm, (kw, kill) in arms.items(): + monkeypatch.setenv("TRTLLM_M31_DISABLE_TP2CTX_E4M3X", kill) + out = torch.full((q.shape[0], HQ, D), float("nan"), dtype=torch.bfloat16, device="cuda") + taken_before = mp._TP2CTX_BF16X_STATE["e4m3x_taken"] + mp.run_msa_nvfp4_sparse_gqa( + q, + k, + v, + sc, + kbi, + md, + sm_scale=D**-0.5, + k_global_scale=kg, + v_global_scale=vg, + out=out, + **kw, + ) + torch.cuda.synchronize() + # E4M3X engages only where the BF16X fork is loaded and is an E4M3X build; elsewhere the arm runs the + # CuTe-DSL forward and must still match the stock bytes. + fork_e4m3x = mp._tp2ctx_bf16x_run()[0] is not None and mp._tp2ctx_e4m3x_ok() is None + assert mp._TP2CTX_BF16X_STATE["e4m3x_taken"] - taken_before == ( + 1 if arm == "opt_ins_e4m3x" and fork_e4m3x else 0 + ), f"{arm}: E4M3X engagement differs from the requested path" + outs[arm] = out + ref = outs["stock"] + assert torch.isfinite(ref.float()).all() + for arm in ("opt_ins", "opt_ins_e4m3x"): + assert torch.equal(outs[arm].view(torch.int16), ref.view(torch.int16)), arm diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_bands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_bands.py new file mode 100644 index 000000000000..ef2c64328afa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_bands.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""VR-BANDS: the sm_107 TP2 decode step bound is 64 rows while every op keeps its own row range (the TP4 base's 48). + +Steps of 49..64 rows (K7 c7..c8) take the "wide" plan (the built-in all-reduce at both boundaries); each op arms +exactly one path at every row count 1..64 (its fast path inside its range, its generic form past it), and no op's +range moves. TRTLLM_M31_DISABLE_VR_BANDS=1 restores the 48-row step bound; VR_BANDS is also off with the R1VR8 kill +(test_modeling_v2_m31_tp2_r1vr8.py), which restores the 8-row bound. +""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + modeling, +) + +OP_BOUND = 48 +STEP_ROWS = range(1, 64 + 1) +ROW_TABLES = ("MX_GEMV_ROWS", "GEMV48_ROWS", "LINEAR_CUBLASLT_ROWS") + + +def _ranges(m) -> dict: + return {r.path: r.keys for r in m.FAST_PATHS} + + +def _projections(m) -> set: + tables = ROW_TABLES + ("MX_GEMV_BF16_MAX_TOKENS",) + return {nk for name in tables for nk in getattr(m, name, {})} + + +def _linear_paths(m) -> dict: + return { + nk: [ + m.linear_path(*nk, rows, prequant, False) + for rows in STEP_ROWS + for prequant in (True, False) + ] + for nk in _projections(m) + } + + +# The default (VR-BANDS on) module's view, recorded at collection before any reload. +_ON_RANGES = _ranges(modeling) +_ON_LINEAR_PATHS = _linear_paths(modeling) + + +@pytest.fixture +def vr_bands_off(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("TRTLLM_M31_DISABLE_VR_BANDS", "1") + importlib.reload(m31_switches) + try: + yield importlib.reload(modeling) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_VR_BANDS", raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) + + +def test_the_legacy_wide_plan_and_op_bound_stay() -> None: + m = modeling + assert m.R1VR8 and m.VR_BANDS and m.DECODE_MAX_TOKENS == OP_BOUND + assert m.plan_for(64).max_tokens == 64 + assert all(m.is_decode_step(n) for n in STEP_ROWS) + + +@pytest.mark.parametrize("rows", STEP_ROWS) +def test_every_step_row_count_takes_exactly_one_plan(rows: int) -> None: + m = modeling + plan = m.plan_for(rows) + assert plan is [p for p in m.DECODE_PLANS if rows <= p.max_tokens][0] + if rows > OP_BOUND: + assert plan.name == "wide" and plan.max_tokens == 64 + assert (plan.attn_mc_ar, plan.moe_mc_ar, plan.qkv_prequant) == (False, False, False) + else: + assert plan.name != "wide" + + +@pytest.mark.parametrize("rows", STEP_ROWS) +def test_every_op_arms_exactly_one_path(rows: int) -> None: + """Each projection's GEMM row tables never overlap and it selects one GEMM per activation form; at most one MoE + path (the engine, the 9..13-row engine where built, the unfinalized MoE) covers the row count, and the MoE takes it + there and the generic MoE elsewhere.""" + m = modeling + for nk in _projections(m): + armed = [name for name in ROW_TABLES if rows in getattr(m, name, {}).get(nk, ())] + assert len(armed) <= 1, (nk, rows, armed) + for prequant in (True, False): + assert isinstance(m.linear_path(*nk, rows, prequant, False), str) + moe_ranges = [ + getattr(m, n) for n in ("MOE_ENGINE", "MOE16_ENGINE", "MC_FINALIZE") if hasattr(m, n) + ] + covering = [r.path for r in moe_ranges if r.covers(rows)] + assert len(covering) <= 1, (rows, covering) + assert m.moe_path(rows) == (covering[0] if covering else None) + + +def test_no_op_range_moves(vr_bands_off) -> None: + """The bound is the step's alone: every fast path's range and every projection's GEMM per row count is the same + with VR-BANDS off.""" + assert _ranges(vr_bands_off) == _ON_RANGES + assert _linear_paths(vr_bands_off) == _ON_LINEAR_PATHS + + +def test_kill_switch(vr_bands_off) -> None: + m = vr_bands_off + assert not m.VR_BANDS and m.DECODE_STEP_MAX == m.DECODE_MAX_TOKENS == OP_BOUND + assert "wide" not in [p.name for p in m.DECODE_PLANS] + assert m.is_decode_step(OP_BOUND) and not m.is_decode_step(OP_BOUND + 1) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_draft_fmha_sm100f.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_draft_fmha_sm100f.py new file mode 100644 index 000000000000..101b9ae6f79a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_draft_fmha_sm100f.py @@ -0,0 +1,99 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""VR_DRAFT_FMHA_SM100F: ``M3DrafterSlidingFiFmha`` is FlashInfer's trtllm-gen FMHA admitted on SM 10.7 only, and +the target installs it on the standalone DSpark drafter by default; ``TRTLLM_M31_DISABLE_VR_DRAFT_FMHA_SM100F`` turns +it off. The constant is evaluated from modeling.py's source against a private copy of m31_switches. CPU only. +""" + +import ast +import importlib.util +import os +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp2" +CONSTANT = "VR_DRAFT_FMHA_SM100F" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + + +def _value(monkeypatch, env) -> bool: + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_draft_fmha_{id(env)}", _SWITCHES) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + tree = ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + statements = [ + node + for node in tree.body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id == CONSTANT + ] + assert len(statements) == 1 + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + return namespace[CONSTANT] + + +@pytest.mark.cpu_only +def test_on_by_default(monkeypatch): + assert _value(monkeypatch, {}) is True + + +@pytest.mark.cpu_only +def test_the_kill_switch_turns_it_off(monkeypatch): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_VR_DRAFT_FMHA_SM100F": "1"}) is False + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("sm,available", [(100, False), (103, False), (107, True)]) +def test_the_library_is_sm107_only(monkeypatch, sm, available): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_drafter_attn as mod, + ) + + monkeypatch.setattr(mod, "get_sm_version", lambda: sm) + monkeypatch.setattr(mod, "IS_FLASHINFER_AVAILABLE", True) + monkeypatch.setattr( + mod.M3DrafterSlidingFiFmha, "_missing_fused_nanobind_ops", staticmethod(lambda: []) + ) + assert mod.M3DrafterSlidingFiFmha.is_environment_available() is available + + +@pytest.mark.cpu_only +def test_c6_not_installed_where_the_decode_bypasses_the_modules(): + """C6: a DFlash-family drafter (the standalone DSpark drafter, also wrapped) runs its block decode on its own + attention ops, so the library is not installed there and the prebuild logs why; a drafter without that attribute + keeps the install.""" + from torch import nn + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_drafter_attn as mod, + ) + + class _DFlash(nn.Module): + def __init__(self): + super().__init__() + self.dflash_attention_backend = "TRTLLM" + self.proj = nn.Linear(2, 2) + + for draft in (_DFlash(), nn.Sequential(nn.Linear(2, 2), _DFlash())): + reason = mod.sliding_fi_bypass_reason(draft) + assert reason is not None and "DFlash attention backend TRTLLM" in reason + assert "never its attention modules' FMHA list" in reason + assert mod.sliding_fi_bypass_reason(nn.Sequential(nn.Linear(2, 2))) is None + text = _MODELING.read_text() + assert text.index("sliding_fi_bypass_reason(self.draft_model)") < text.index( + "install_sliding_fi(self.draft_model)" + ) + assert '"[M3 prebuild] VR_DRAFT_FMHA_SM100F: not installed: {bypass}"' in text diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_tp2_mcag.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_tp2_mcag.py new file mode 100644 index 000000000000..02be22a202ff --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_vr_tp2_mcag.py @@ -0,0 +1,155 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MCAG on the sm_107 TP2 target (world 2; VR port of the sm_103 test). + +The rejection sampler's draft-logits gather (SpecWorkerBase.maybe_gather_sharded_draft_logits) takes a target-installed +one-launch gather when it returns a tensor and the stock all-gather otherwise; the target's op declines by dtype / +columns / uncompiled rows. Default OFF, TRTLLM_M31_ENABLE_MCAG=1 turns it on and TRTLLM_M31_DISABLE_MCAG=1 still +wins. CPU only: the gathers are stand-ins.""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + dspark, + m31_switches, +) +from tensorrt_llm._torch.distributed import ops as dist_ops +from tensorrt_llm._torch.speculative.interface import SpecWorkerBase + +STOCK = "stock-gathered" + + +def _worker(op=None): + return SimpleNamespace( + mapping="tp2", + _draft_logits_are_sharded=lambda logits, md: True, + _draft_logits_gather_op=op, + ) + + +def test_the_hook_is_a_class_default(): + """Every stock spec worker reads the class attribute (None), no per-step attribute miss.""" + assert SpecWorkerBase._draft_logits_gather_op is None + assert "_draft_logits_gather_op" in SpecWorkerBase.__dict__ + + +@pytest.fixture +def stock(monkeypatch): + calls = [] + monkeypatch.setattr(dist_ops, "allgather", lambda x, mapping, dim: calls.append(dim) or STOCK) + return calls + + +def _gather(worker, logits, greedy=False): + md = SimpleNamespace(is_all_greedy_sample=greedy) + return SpecWorkerBase.maybe_gather_sharded_draft_logits(worker, logits, md) + + +def test_the_hook_is_inert_by_default(stock): + assert _gather(_worker(), torch.zeros(1, 7, 8)) == STOCK + assert stock == [-1] + + +def test_the_op_runs_and_a_declined_call_falls_back(stock): + x = torch.zeros(1, 7, 8) + assert _gather(_worker(lambda logits: "op-gathered"), x) == "op-gathered" + assert stock == [] + assert _gather(_worker(lambda logits: None), x) == STOCK + assert stock == [-1] + + +def test_greedy_batches_never_gather(stock): + x = torch.zeros(1, 7, 8) + assert _gather(_worker(lambda logits: "op"), x, greedy=True) is x + assert stock == [] + + +def test_the_setter(): + w = SimpleNamespace() + assert SpecWorkerBase.enable_draft_logits_gather_op(w, None) is False + op = object() + assert ( + SpecWorkerBase.enable_draft_logits_gather_op(w, op) is True + and w._draft_logits_gather_op is op + ) + + +@pytest.fixture +def mcag_target(monkeypatch): + calls = [] + compiled = {(7, 100032, 2)} + monkeypatch.setattr( + dspark._allgather_cast, + "is_compiled", + lambda rows, cols, world, grid=0, device=None, fp32=False: ( + fp32 and (rows, cols, world) in compiled + ), + ) + monkeypatch.setattr( + dspark._allgather_cast, + "gather_last_dim_f32", + lambda logits, **ws: calls.append((tuple(logits.shape), ws["world"])) or "mc-gathered", + ) + target = SimpleNamespace( + _mcag_cols=100032, + _a1={"world": 2, "ws_uc": None, "ws_mc": None, "ws_flags": None, "rank": 0}, + ) + return target, calls + + +def test_the_target_op_takes_compiled_fp32_rows(mcag_target): + target, calls = mcag_target + assert dspark.M3DSparkWorker._mcag_gather(target, torch.zeros(1, 7, 100032)) == "mc-gathered" + assert calls == [((1, 7, 100032), 2)] + + +@pytest.mark.parametrize( + "logits", + [ + torch.zeros(1, 7, 100032, dtype=torch.bfloat16), # not fp32 + torch.zeros(1, 7, 100000), # other columns + torch.zeros(2, 7, 100032), # rows not compiled + ], +) +def test_the_target_op_declines(mcag_target, logits): + target, calls = mcag_target + assert dspark.M3DSparkWorker._mcag_gather(target, logits) is None + assert calls == [] + + +@pytest.mark.parametrize( + "env, expected", + [ + ({}, False), + ({"TRTLLM_M31_ENABLE_MCAG": "1"}, True), + ({"TRTLLM_M31_ENABLE_MCAG": "1", "TRTLLM_M31_DISABLE_MCAG": "1"}, False), + ], +) +def test_default_off_opt_in_and_kill_switch(monkeypatch: pytest.MonkeyPatch, env, expected) -> None: + monkeypatch.delenv("TRTLLM_M31_DISABLE_MCAG", raising=False) + monkeypatch.delenv("TRTLLM_M31_ENABLE_MCAG", raising=False) + for name, value in env.items(): + monkeypatch.setenv(name, value) + importlib.reload(m31_switches) + try: + assert importlib.reload(dspark).MCAG is expected + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_MCAG", raising=False) + monkeypatch.delenv("TRTLLM_M31_ENABLE_MCAG", raising=False) + importlib.reload(m31_switches) + importlib.reload(dspark) + assert dspark.MCAG is False + + +def test_c6_mcag_is_unreachable_without_sharded_draft_logits(): + """C6: only the S4-logits path hands the sampler vocab-sharded draft logits; without S4 set up, or with S4-logits + off, the gather MCAG replaces is never called, and MCAG says why instead of compiling.""" + off = dspark.mcag_unreachable(SimpleNamespace(_s4=None, _s4_logits_on=True)) + assert off is not None and "S4 is not set up" in off + off = dspark.mcag_unreachable(SimpleNamespace(_s4=object(), _s4_logits_on=False)) + assert off is not None and "S4-logits is off" in off + assert dspark.mcag_unreachable(SimpleNamespace(_s4=object(), _s4_logits_on=True)) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_capture_boundary.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_capture_boundary.py new file mode 100644 index 000000000000..7adc0cc92652 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_capture_boundary.py @@ -0,0 +1,283 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The feed-forward exit on Eagle3 capture layers: the generic layer in both targets' bodies (``PrefillTarget._layer``, +``DecodeTarget._generic_layer``), and the decode fast path's capture layers (``CAPTURE_LAYERS_FAST``). + +Every layer's MoE / MLP output goes unreduced to the fused boundary (all-reduce + residual + the next layer's input +norm, the final norm after the last layer), and a capture layer captures the reduced residual sum that boundary +writes, with no residual of its own. CPU only: the attention, the feed-forward and the boundaries are stand-ins. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) + +LAYERS = 4 + + +class _Rows(tuple): + """A stand-in activation: compares as its tuple, and has the rows (``shape[0]``) the prefill boundaries read to + pick their all-reduce (PREFILL-MNNVL).""" + + shape = (2,) + + +class _Spec: + def __init__(self, layers): + self.layers = set(layers) + + def is_layer_capture(self, i): + return i in self.layers + + +@pytest.fixture(params=["prefill", "decode"]) +def layer(request, monkeypatch): + """(core, the target's generic layer bound to a target over ``core``).""" + calls = [] + w = {f"l{i}_{n}": f"{n}{i}" for i in range(LAYERS) for n in ("in_norm", "post_norm", "router")} + w["final_norm"] = "final" + c = SimpleNamespace( + w=w, + num_layers=LAYERS, + moe_layers=frozenset({2, 3}), + sparse_layers=frozenset({1, 3}), + rms_eps=1e-6, + calls=calls, + # The prefill boundaries pick their built-in all-reduce from these (PREFILL-MNNVL); the stand-ins ignore it. + allreduce=None, + allreduce_mnnvl=None, + # The prefill attention boundary passes a MoE layer's NVFP4 input scale to CTXNORMQ. + moe={i: SimpleNamespace(fc31_input_scale=None) for i in (2, 3)}, + ) + monkeypatch.setattr(modeling, "_can_mc", lambda x, folded=False: False) + # The prefill body tries the CTXNORMQ boundary first; it declines here (no op on these stand-ins). + monkeypatch.setattr(modeling, "_ctxnormq_allreduce_norm", lambda *args, **kwargs: None) + monkeypatch.setattr( + modeling, + "_native_allreduce_norm", + lambda core, x, r, norm, allreduce=None: (("normed", x, norm), ("updated", x, r)), + ) + + def boundary(self, x, x2, residual, norm_w, qkv_a1_next=False): + calls.append((x, x2, norm_w, qkv_a1_next)) + return ("next", norm_w), ("sum", x, x2) + + if request.param == "prefill": + cls = modeling.PrefillTarget + monkeypatch.setattr(cls, "_moe", lambda self, i, h, **k: (("moe", i), ("routed", i))) + monkeypatch.setattr( + cls, "_swiglu_mlp", lambda self, h, prefix, low_m, dense=False: ("mlp", prefix) + ) + else: + cls = modeling.DecodeTarget + monkeypatch.setattr( + cls, "_moe", lambda self, i, h, **k: (("moe", i), ("routed", i), False, False) + ) + monkeypatch.setattr( + cls, + "_swiglu_mlp", + lambda self, h, prefix, low_m, push=None, dense=False: (("mlp", prefix), False), + ) + monkeypatch.setattr(cls, "_attention", lambda self, i, *a, **k: (_Rows(("attn", i)), False)) + monkeypatch.setattr(cls, "_generic_boundary", boundary) + target = cls(c) + return c, target._layer if request.param == "prefill" else target._generic_layer + + +@pytest.mark.parametrize("i", range(LAYERS)) +def test_a_capture_layer_captures_the_boundary_residual_sum(layer, i) -> None: + core, run = layer + captured = [] + out = run( + i, + None, + torch.ones(2), + torch.zeros(2), + None, + _Spec([i]), + lambda spec, layer, h, r: captured.append((layer, h, r)), + ) + norm = core.w[f"l{i + 1}_in_norm"] if i + 1 < LAYERS else "final" + assert core.calls[-1][2] == norm, ( + "the boundary applies the next layer's norm (the final norm after the last)" + ) + assert captured == [(i, out[1], None)] + assert core.calls[-1][3] == ( + isinstance(run.__self__, modeling.PrefillTarget) and (i + 1) in core.sparse_layers + ) + if i in core.moe_layers: + assert core.calls[-1][1] == ("routed", i), "the routed part folds into the boundary" + + +def test_a_layer_without_capture_captures_nothing(layer) -> None: + _, run = layer + captured = [] + run( + 1, + None, + torch.ones(2), + torch.zeros(2), + None, + _Spec([]), + lambda *a: captured.append(a), + ) + assert not captured + + +class _FastCore: + num_layers = 3 + w = {"embed": None} + + +@pytest.mark.parametrize("written", [True, False]) +def test_a_decode_capture_layer_takes_the_fast_layer_with_the_capture_norm( + monkeypatch, written +) -> None: + """A decode step's capture layer (past the first) runs the fast decode layer with the worker's capture target, and + captures the boundary's updated residual; ``written=True`` when the boundary all-reduce wrote the capture norm + (``CAPTURE_NORM_IN_BOUNDARY``).""" + assert modeling.CAPTURE_LAYERS_FAST and modeling.CAPTURE_FUSED_BOUNDARY + assert modeling.CAPTURE_NORM_IN_BOUNDARY + target = modeling.DecodeTarget(_FastCore()) + seen, captured = [], [] + monkeypatch.setattr(target, "_check_step_contract", lambda md: None) + monkeypatch.setattr(target, "step_args", lambda md: {"num_tokens": 4}) + monkeypatch.setattr(target, "_embed", lambda ids, n: ("h", "r")) + monkeypatch.setattr(modeling, "plan_for", lambda n: SimpleNamespace(name="plan")) + monkeypatch.setattr(modeling, "_record", lambda *a: None) + + def fast(i, plan, pos, h, r, step, capture=None): + seen.append((i, capture)) + if capture is not None: + target._capture_written = written + return ("h", i), ("r", i) + + def generic(*a): + raise AssertionError("a capture layer ran the generic layer on a decode step") + + monkeypatch.setattr(target, "_layer", fast) + monkeypatch.setattr(target, "_generic_layer", generic) + out = target.forward( + None, + torch.zeros(4, dtype=torch.int32), + None, + None, + _Spec([1]), + lambda spec, i, h, r, written=False: captured.append((i, h, r, written)), + lambda spec, i, n: ("capture_out", i, n), + ) + assert out == ("h", 2) + assert seen == [(0, None), (1, ("capture_out", 1, 4)), (2, None)] + assert captured == [(1, ("r", 1), None, written)] + + +@pytest.fixture +def ctxnormq_prefill(monkeypatch): + """(core, the prefill target's generic layer, the boundary calls), with CTXNORMQ taking every boundary offered. + + Only the collectives are stand-ins: the layer and its exit boundary (``PrefillTarget._generic_boundary``) are the + target's own, so the test reads which boundary each result came from and where the layer uses it. + """ + calls = [] + w = {f"l{i}_{n}": f"{n}{i}" for i in range(LAYERS) for n in ("in_norm", "post_norm", "router")} + w["final_norm"] = "final" + c = SimpleNamespace( + w=w, + num_layers=LAYERS, + moe_layers=frozenset({2, 3}), + sparse_layers=frozenset({1, 3}), + rms_eps=1e-6, + allreduce=None, + allreduce_mnnvl=None, + moe={i: SimpleNamespace(fc31_input_scale=f"fp4_{i}") for i in (2, 3)}, + ) + monkeypatch.setattr(modeling, "_can_mc", lambda x, folded=False: False) + + def ctxnormq( + core, + x, + residual, + norm_w, + fp4_scale, + allreduce=None, + router_w=None, + qkv_a1_next=False, + pre_moe=False, + ): + calls.append(("ctxnormq", x, residual, norm_w, fp4_scale, router_w, qkv_a1_next, pre_moe)) + return ("ctxnormq_normed", norm_w), ("ctxnormq_updated", norm_w) + + def native(core, x, residual, norm_w, allreduce=None): + calls.append(("native", x, residual, norm_w)) + return ("native_normed", norm_w), ("native_updated", norm_w) + + monkeypatch.setattr(modeling, "_ctxnormq_allreduce_norm", ctxnormq) + monkeypatch.setattr(modeling, "_native_allreduce_norm", native) + cls = modeling.PrefillTarget + monkeypatch.setattr(cls, "_attention", lambda self, i, *a, **k: (_Rows(("attn", i)), False)) + monkeypatch.setattr(cls, "_moe", lambda self, i, h, **k: (_Rows(("moe", i, h)), None)) + monkeypatch.setattr( + cls, "_swiglu_mlp", lambda self, h, prefix, *a, **k: _Rows(("mlp", prefix, h)) + ) + return c, cls(c)._layer, calls + + +@pytest.mark.parametrize("i", range(LAYERS)) +def test_prefill_uses_the_ctxnormq_boundary_in_place_of_the_fused_allreduce( + ctxnormq_prefill, i +) -> None: + """A prefill boundary CTXNORMQ takes is used where the native fused all-reduce's would be. + + The attention boundary (with the MoE's NVFP4 input scale on a MoE layer) feeds the feed-forward, and its updated + residual goes to the exit boundary. The exit boundary is CTXNORMQ's into every next input norm; the final norm + keeps the native fused all-reduce. Its (normed, updated) is what the layer returns, and its updated is what a + capture layer captures. + """ + core, run, calls = ctxnormq_prefill + residual = torch.zeros(2) + captured = [] + out = run( + i, + None, + torch.ones(2), + residual, + None, + _Spec([i]), + lambda spec, layer, h, r: captured.append((layer, h, r)), + ) + post = core.w[f"l{i}_post_norm"] + kind, x, res, norm, fp4, router, next_sparse, pre_moe = calls[0] + assert (kind, x, norm) == ("ctxnormq", ("attn", i), post), ( + "the attention boundary is offered to CTXNORMQ" + ) + assert res is residual + assert fp4 == (f"fp4_{i}" if i in core.moe_layers else None), "NVFP4 for the MoE's input only" + assert router == (core.w[f"l{i}_router"] if i in core.moe_layers else None) + assert pre_moe and not next_sparse + normed = ("ctxnormq_normed", post) + ff = ("moe", i, normed) if i in core.moe_layers else ("mlp", f"l{i}_mlp", normed) + if i + 1 < LAYERS: + nxt = core.w[f"l{i + 1}_in_norm"] + assert calls[1] == ( + "ctxnormq", + ff, + ("ctxnormq_updated", post), + nxt, + None, + None, + (i + 1) in core.sparse_layers, + False, + ) + assert out == (("ctxnormq_normed", nxt), ("ctxnormq_updated", nxt)) + else: + assert calls[1] == ("native", ff, ("ctxnormq_updated", post), "final"), ( + "the final norm keeps the native fused all-reduce" + ) + assert out == (("native_normed", "final"), ("native_updated", "final")) + assert len(calls) == 2 + assert captured == [(i, out[1], None)] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_dispatch.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_dispatch.py new file mode 100644 index 000000000000..dda36d3a9a18 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_dispatch.py @@ -0,0 +1,412 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The M3.1 tp2 (Rubin) decode target's fast-path table (``modeling.py``: the row ranges and the selection functions): +its row ranges against the catalog cells that certify them, the selection for every decode row count, and the phase +split, at the TP2 per-rank geometry (32 query / 2 KV / 2 index heads, expert width 1536). + +The plans are the M3 tp2 row buckets (tiny 8 / b16 16 / b48 48) with the TP4 base's one setting each; MoE16 is off +(the TP4 kernel is untested at width 1536), ROUTER48 takes the tensor-core router at 9..48 rows, the +multicast all-reduce + norm runs at world 2 without the one-shot cluster option. The ones not yet driven on SM 10.7 +(the long-row select, the capped dense decode split, the early drafter fc, the MC finalize) cover nothing. The dense +layers' full-context decode keeps its TP4 build (M3.1 has no dense attention layers). + +CPU only: nothing here launches a kernel. +""" + +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_report, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + triton_sparse_decode as tsd, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm import ( # noqa: E501 + m3_mx_gemv, +) + +ROWS = range(1, dispatch.DECODE_STEP_MAX + 2) # every decode row count and the first one past it + +# The TP2 per-rank MXFP8 projections (N, K): 32 query heads and 2 KV heads, expert width 1536 per rank. +TP2_PROJECTIONS = ( + m3_mx_gemv.QKV_SPARSE, + m3_mx_gemv.QKV, + m3_mx_gemv.O_PROJ, + m3_mx_gemv.SHARED_GATE_UP, + m3_mx_gemv.SHARED_DOWN, + m3_mx_gemv.DENSE_GATE_UP, + m3_mx_gemv.DENSE_DOWN, +) + + +@pytest.fixture(autouse=True) +def _policy(): + """Every test starts outside require mode with nothing logged or counted, and leaves it so.""" + dispatch.set_require(False) + dispatch.reset_logged() + m3_report.reset_work() + yield + dispatch.set_require(False) + dispatch.reset_logged() + m3_report.reset_work() + + +@pytest.mark.parametrize("fast_path", dispatch.FAST_PATHS, ids=lambda r: r.path) +def test_every_covered_shape_is_a_certified_cell(fast_path) -> None: + uncertified = fast_path.keys - fast_path.certified() + assert not uncertified, ( + f"{fast_path.path} covers {sorted(uncertified, key=str)} but " + f"{type(fast_path.entry).__name__}.CELLS certifies none of them: add a cell or narrow the range" + ) + + +def test_the_fast_path_names_are_unique() -> None: + names = [r.path for r in dispatch.FAST_PATHS] + assert len(names) == len(set(names)) + + +def test_the_decode_target_keeps_the_tp2_row_bounds() -> None: + """Up to 48 decode-layer rows (the TP4 base's bound; R1VR8's kill restores the TP2-lineage 8), up to 64 with + VR_BANDS (off with the R1VR8 kill); a wider step runs the generic layers.""" + assert dispatch.DECODE_MAX_TOKENS == (48 if dispatch.R1VR8 else dispatch.TP2_ROWS) + assert dispatch.TP2_ROWS == 8 + assert dispatch.DECODE_STEP_MAX == ( + 448 if dispatch.TP2WIDE_DECODE else 64 if dispatch.VR_BANDS else dispatch.DECODE_MAX_TOKENS + ) + assert not dispatch.VR_BANDS or dispatch.R1VR8 + assert [dispatch.is_decode_step(n) for n in ROWS] == [True] * dispatch.DECODE_STEP_MAX + [False] + + +# ------------------------------------------------------------------ the paths still off on SM 10.7 + +OFF_ON_SM107 = ( + dispatch.DENSE_DECODE_SPLIT, + dispatch.LONG_SELECT, +) + (() if dispatch.TP2W_MC_FINALIZE else (dispatch.MC_FINALIZE,)) + + +@pytest.mark.parametrize("fast_path", OFF_ON_SM107, ids=lambda r: r.path) +def test_paths_without_an_sm107_build_cover_nothing(fast_path) -> None: + assert not fast_path.keys, f"{fast_path.path} covers {sorted(fast_path.keys, key=str)}" + + +def test_decode_boundaries_follow_the_multicast_all_reduce_kill() -> None: + """MC_AR_TP4 moves every decode boundary (and the o_proj / MoE pushes into it) onto the SM 10.7 TP4 multicast + all-reduce; off, the built-in all-reduce runs and nothing pushes.""" + on = dispatch.MC_AR_TP4 + for plan in dispatch.DECODE_PLANS: + want = (False, False, False) if plan.name in ("wide", "wide448") else (on, on, on) + assert (plan.attn_mc_ar, plan.moe_mc_ar, plan.qkv_prequant) == want, plan + assert dispatch.TP2_BUCKETS == tuple(b for b in (8, 16, 48) if b <= dispatch.DECODE_MAX_TOKENS) + assert tuple(p.name for p in dispatch.DECODE_PLANS) == ("tiny", "b16", "b48")[ + : len(dispatch.TP2_BUCKETS) + ] + (("wide",) if dispatch.VR_BANDS else ()) + (("wide448",) if dispatch.TP2WIDE_DECODE else ()) + assert tuple(p.max_tokens for p in dispatch.DECODE_PLANS) == dispatch.TP2_BUCKETS + ( + (64,) if dispatch.VR_BANDS else () + ) + ((448,) if dispatch.TP2WIDE_DECODE else ()) + # World 2: the TP4 op's row kernel takes world 4 only; the one-shot cluster option is off at every row count. + assert dispatch.MC_AR_CLUSTER_MAX_ROWS == 0 + assert bool(dispatch.MC_ALLREDUCE.keys) == on + assert dispatch.DRAFT_MC_AR.keys == (frozenset(range(1, 9)) if on else frozenset()) + assert dispatch.ATTN_PUSH_MAX_TOKENS == (8 if on else 0) + assert dispatch.DENSE_PUSH_MAX_TOKENS == (8 if on and dispatch.DENSE_PUSH_TP4 else 0) + assert dispatch.MOE_PUSH_MAX_TOKENS == (dispatch.MOE_ENGINE_MAX_TOKENS if on else 0) + assert dispatch.GENERIC_MC_AR_MAX_TOKENS == 0 + assert dispatch.GENERIC_MC_AR_FOLDED_MAX_TOKENS == 0 + + +def test_the_dense_decode_split_is_the_built_in_one() -> None: + assert dict(dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH) == {} + + +# ------------------------------------------------------------------ on: the dense layers' full-context decode + + +def test_the_dense_layers_keep_the_fullctx_decode_tables() -> None: + """One or two requests, 1 + 3 verify rows or one draft-pass row, each at its own pinned recipe.""" + assert dispatch.FULLCTX_DECODE_ENABLED + assert dispatch.FULLCTX_DECODE.keys == {(4, 1), (1, 1), (4, 2), (1, 2)} + assert set(dispatch.FULLCTX_DECODE_RECIPES) == set(dispatch.FULLCTX_DECODE.keys) + for recipe in dispatch.FULLCTX_DECODE_RECIPES.values(): + assert 2 <= recipe["splits"] <= 128 # the standalone reducer's range + + +def test_the_fullctx_plans_keep_their_tp4_build() -> None: + """The dense-attention kernel is the TP4 build, unchanged: M3.1 runs MSA on every layer (no dense attention), so + no TP2 step reaches it.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_fullctx_decode as kernel, + ) + + assert (kernel.HEADS, kernel.KV_HEADS, kernel.HEAD_DIM) == (16, 1, 128) + assert kernel.HEADS_PER_KV == 16 + + +def test_the_fullctx_entry_cells_run_at_the_forward_plan_capacity() -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention import ( # noqa: E501 + m3_fullctx_decode as entry, + ) + + assert entry.MAX_KV_LEN == dispatch.FULLCTX_DECODE_MAX_KV_LEN + + +def test_the_fullctx_capacity_covers_the_agentx_cache_bound() -> None: + """The forward compares the cache's length bound (max_seq_len plus the speculative rows it reserves) with the plan + capacity; at the AgentX max_seq_len of 1 << 20 with up to 8 draft rows the dense layers must still be taken.""" + assert dispatch.FULLCTX_DECODE_MAX_KV_LEN >= (1 << 20) + 8 + + +def test_the_drafter_takes_the_fullctx_decode_at_the_same_shapes() -> None: + assert dispatch.DRAFTER_FULLCTX_ENABLED + assert dispatch.DRAFTER_FULLCTX.keys == dispatch.FULLCTX_DECODE.keys + + +def test_the_dense_fp8_qkv_and_kv_insert_kill_constants_default_on() -> None: + assert dispatch.DENSE_QKV_FP8_VIA_BF16 is True + assert dispatch.DENSE_KV_INSERT_FUSED is True + assert dispatch.FULLCTX_DECODE_OWN_REDUCER is True + assert dispatch.FULLCTX_DECODE_MXFP8 is True + + +def test_the_fullctx_taken_lines_log_without_raising_under_require() -> None: + dispatch.set_require(True) + dispatch.taken(dispatch.FULLCTX_DECODE.path, 4) + dispatch.taken(dispatch.DENSE_KV_INSERT_PATH, 4) + dispatch.taken(dispatch.FULLCTX_DECODE_MXFP8_PATH, 4) + + +# ------------------------------------------------------------------ the paths on + + +@pytest.mark.parametrize("rows", range(1, dispatch.DECODE_MAX_TOKENS + 1)) +def test_the_moe_takes_the_engines_and_the_generation_router(rows) -> None: + """1..8 rows the MoE engine (router in-kernel); 9..13 the MoE16 engine only with MOE16_TP2 (off by default: the + generic MoE); the unfused MoE of a pure generation step takes the TF32 tensor-core router at 9..48 rows (ROUTER48; + 9..16 with its kill), the fp32 router else.""" + expected = ( + dispatch.MOE_ENGINE.path + if rows <= 8 + else dispatch.MOE16_ENGINE.path + if rows <= 13 and dispatch.MOE16_TP2 + else dispatch.MC_FINALIZE.path + if dispatch.MC_FINALIZE_MIN_TOKENS <= rows <= 48 and dispatch.TP2W_MC_FINALIZE + else None + ) + assert dispatch.moe_path(rows) == expected + assert dispatch.ROUTER_TC_GEN_MAX_ROWS == (48 if dispatch.ROUTER48 else 16) + assert dispatch.ROUTER_TC_GEN.covers(rows) == (9 <= rows <= dispatch.ROUTER_TC_GEN_MAX_ROWS) + + +def test_moe16_is_off_at_the_tp2_width() -> None: + assert dispatch.MOE16_TP2 is False + assert dispatch.MOE16_MAX_TOKENS == 0 + assert not dispatch.MOE16_ENGINE.keys + + +def _router_core(cluster16: bool): + import torch + + return SimpleNamespace(router_tc_cluster16=cluster16, w={"l3_router": torch.zeros(128, 64)}) + + +@pytest.mark.parametrize("rows", [1, 8, 9, 13, 14, 16, 17, 24, 32, 48, 49]) +@pytest.mark.parametrize("cluster16", [True, False]) +def test_the_generation_router_takes_router_tc_only_on_its_rows( + monkeypatch, rows, cluster16 +) -> None: + """The built-in gate's choice: router_tc on 9..48 bf16 rows (ROUTER48) when the device co-schedules 16-CTA clusters; + the fp32 router otherwise. Context steps (``_router_logits``) never take it.""" + import torch + + calls = [] + monkeypatch.setattr(dispatch, "m3_router_tc", lambda *a: calls.append("tc") or [a[0]]) + monkeypatch.setattr(dispatch, "m3_fp32_linear", lambda *a: calls.append("fp32") or a[0]) + h = torch.zeros(rows, 64, dtype=torch.bfloat16) + dispatch._decode_router_logits(_router_core(cluster16), 3, h) + assert calls == ["tc" if cluster16 and 9 <= rows <= dispatch.ROUTER_TC_GEN_MAX_ROWS else "fp32"] + # ROUTER48's work counter: the router rows past the TP4 router's 16. + assert m3_report.work_count("ROUTER48") == (1 if cluster16 and 16 < rows <= 48 else 0) + calls.clear() + dispatch._router_logits(_router_core(cluster16), 3, h) + assert calls == ["fp32"] + + +def test_the_core_selects_the_target_on_the_context_count_alone() -> None: + """Every pure generation step is decode's, whatever its width; a step with context rows (a mixed one too) is + prefill's.""" + core = SimpleNamespace(_prefill="prefill", _decode="decode") + select = dispatch.MiniMaxM3Tp2ModelingV2._select_target + assert select(core, SimpleNamespace(num_contexts=0)) == "decode" + assert select(core, SimpleNamespace(num_contexts=1)) == "prefill" + assert select(core, SimpleNamespace(num_contexts=7)) == "prefill" + + +@pytest.mark.parametrize("rows", range(1, dispatch.DECODE_MAX_TOKENS + 1)) +def test_the_projections_take_the_gemvs_else_the_mxfp8_gemm(rows) -> None: + for nk in TP2_PROJECTIONS: + prequant = dispatch.linear_path(*nk, rows, prequant=True, low_m_gemm=True) + assert prequant == ( + dispatch.MX_GEMV_PATH + if rows in dispatch.MX_GEMV_ROWS[nk] + else dispatch.PREQUANT_GEMM_PATH + ) + bf16 = dispatch.linear_path(*nk, rows, prequant=False, low_m_gemm=True) + assert bf16 == ( + dispatch.MX_GEMV_BF16_PATH + if rows <= dispatch.MX_GEMV_BF16_MAX_TOKENS[nk] + else dispatch.GENERIC_GEMM_PATH + ) + + +def test_the_attention_producer_and_scorer_are_on() -> None: + assert dispatch.PRODUCER.covers("nvfp4_tp2") + assert dispatch.INDEX_SCORE.keys == {"early", "prefetch"} + assert dispatch.SPARSE_DECODE_PRODUCER.keys == {"per_row", "multi_query"} + assert dispatch.EMBED_FIRSTNORM.keys == frozenset(range(1, 9)) + + +def test_speculation_runs_the_fused_tail_with_the_bf16_draft_head() -> None: + requests = dispatch.DECODE_MAX_TOKENS // (dispatch.SHARD_ACCEPT_DRAFT_LEN + 1) + assert dispatch.SHARD_ACCEPT_MAX_REQUESTS == requests + assert dispatch.SAMPLED_VERIFY_MAX_REQUESTS == requests + assert dispatch.FIRST_DRAFTER_INPUTS_MAX_REQUESTS == requests + assert dispatch.TARGET_HEAD_MAX_ROWS == 0 # the in-tree BF16 head (Eagle3 only) + assert dispatch.FC_AFTER_HEAD_MAX_ROWS == 8 + assert dispatch.DRAFT_STEP_MAX_REQUESTS == 8 + assert dispatch.FC_EARLY_MAX_ROWS == 0 + assert not dispatch.DRAFT_HEAD_CODED + + +def test_a_miss_raises_only_under_require() -> None: + dispatch.missed("some_path", 1, "a precondition") + dispatch.set_require(True) + with pytest.raises(dispatch.FastPathUnavailable, match="some_path"): + dispatch.missed("some_path", 1, "a precondition") + + +def test_taken_logs_without_raising_under_require() -> None: + dispatch.set_require(True) + dispatch.taken(dispatch.INDEX_SCORE.path, 1) + dispatch.taken(dispatch.MOE_ENGINE.path, 8) + + +def test_taken_logs_once_per_path_and_rows(monkeypatch) -> None: + lines = [] + monkeypatch.setattr(dispatch.logger, "info", lines.append) + for _ in range(3): + dispatch.taken(dispatch.MX_GEMV_PATH, 4, "[4992, 6144] mxfp8 in") + dispatch.taken(dispatch.MX_GEMV_PATH, 8, "[4992, 6144] mxfp8 in") + taken_lines = [line for line in lines if "taken at" in line] + assert len(taken_lines) == 2 and all("m3_mx_gemv taken at" in line for line in taken_lines) + # Gate e: each taken call is also counted (powers of two logged: 1, 2, 4). + work = [line for line in lines if line.startswith("[M3 sm107 tp2] rank")] + assert len(work) == 3 and "m3_mx_gemv engaged" in work[0] and "work 4 (host count" in work[-1] + assert m3_report.work_count(dispatch.MX_GEMV_PATH) == 4 + + +@pytest.mark.parametrize( + "world, rows, kernel", + [(2, 1, "8-CTA cluster kernel"), (2, 32, "8-CTA cluster kernel"), (4, 8, "row kernel")], +) +def test_the_mc_allreduce_line_carries_world_kernel_and_call_count( + monkeypatch, world, rows, kernel +) -> None: + """The multicast all-reduce + norm's engagement line names the world size, the kernel the TP4 op runs there (its + row kernel at world 4 only: TP2 runs the 8-CTA cluster kernel) and this rank's running call count; logged once per + (rows, detail), counted on every call.""" + lines = [] + monkeypatch.setattr(dispatch.logger, "info", lines.append) + mapping = SimpleNamespace(tp_size=world) + for _ in range(3): + dispatch._mc_taken(mapping, rows, "x") + taken_lines = [line for line in lines if "taken at" in line] + assert taken_lines == [ + f"[M3 dispatch] rank {taken_lines[0].split()[3][:-1]}: mc_allreduce_norm taken at {rows} rows " + f"(x, world {world}, {kernel}, call 1)" + ] + assert m3_report.work_count(dispatch.MC_ALLREDUCE.path) == 3 + + +def test_the_sparse_decode_writes_the_mxfp8_handoff() -> None: + """Every decode step (at most DECODE_MAX_TOKENS rows) can take the MXFP8 o_proj hand-off of the sparse decode.""" + assert dispatch.MERGE_MXFP8_HANDOFF is True + assert tsd.MERGE_MXFP8_MIN_TOKENS <= 1 and dispatch.DECODE_MAX_TOKENS <= tsd.MergeMxfp8.MAX_ROWS + + +def test_the_decode_mxfp8_decision_is_pinned_to_cute_dsl() -> None: + from tensorrt_llm._torch.custom_ops.torch_custom_ops import ( + FlashInferMXFP8GemmRunner, + MXFP8QuantizeRunner, + ) + + assert dispatch.MX_TAC_TP4 + assert dispatch.MXFP8_DECODE_QUANT_TACTIC == MXFP8QuantizeRunner.CUTE_DSL + assert dispatch.MXFP8_DECODE_GEMM_TACTIC == FlashInferMXFP8GemmRunner.CUTE_DSL + + +def _runtime(fused: bool, indexer: str = "fp8", kv: str = "fp8"): + """A model config with the runtime fields ``_validate_runtime`` reads.""" + sparse = SimpleNamespace( + algorithm="minimax_m3", + implementation="msa", + fuse_qkv_index_projection=fused, + indexer_kv_dtype=indexer, + ) + mode = SimpleNamespace( + has_fp8_kv_cache=lambda: kv == "fp8", has_fp4_kv_cache=lambda: kv == "nvfp4" + ) + return SimpleNamespace( + sparse_attention_config=sparse, quant_config=SimpleNamespace(quant_mode=mode) + ) + + +@pytest.mark.parametrize("fused", [True, False], ids=["five_way", "unfused"]) +def test_the_target_takes_both_projection_layouts(fused) -> None: + """The packed five-way projection and the separate QKV + MXFP8 index projections + (fuse_qkv_index_projection=False) are both this target's; a BF16 indexer or a BF16 KV cache is not.""" + dispatch._validate_runtime(_runtime(fused)) + dispatch._validate_runtime(_runtime(fused, kv="nvfp4")) + with pytest.raises(ValueError, match="indexer_kv_dtype='fp8'"): + dispatch._validate_runtime(_runtime(fused, indexer="bf16")) + with pytest.raises(ValueError, match="FP8 or NVFP4 KV cache"): + dispatch._validate_runtime(_runtime(fused, kv="bf16")) + + +def test_tp2w_mc_finalize_takes_rows_past_the_moe_engines() -> None: + """TP2W_MC_FINALIZE: the unfinalized MoE + the finalize in the multicast boundary from the first row past the MoE + engines (TP2_MC_FINALIZE_SMALL: 9, 14 with MoE16 on; 33 with its kill) to 48 (the op's certified 48-token bucket); + the engines' rows keep their path; the kill switch leaves no rows.""" + if dispatch.TP2W_MC_FINALIZE: + lo = max(9, dispatch.MOE16_MAX_TOKENS + 1) if dispatch.TP2_MC_FINALIZE_SMALL else 33 + assert (dispatch.MC_FINALIZE_MIN_TOKENS, dispatch.MC_FINALIZE_MAX_TOKENS) == (lo, 48) + assert not any(dispatch.MC_FINALIZE.covers(r) for r in range(1, lo)) + assert all(dispatch.MC_FINALIZE.covers(r) for r in range(lo, 49)) + else: + assert not dispatch.MC_FINALIZE.keys + src = open(dispatch.__file__).read() + assert 'TP2W_MC_FINALIZE = not m31_disabled("TP2W_MC_FINALIZE")' in src + + +def test_tp2_mc_finalize_small_kill_switch(monkeypatch) -> None: + """TRTLLM_M31_DISABLE_TP2_MC_FINALIZE_SMALL=1: the MC finalize back to the WIDE rows 33..48.""" + import importlib + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + m31_switches, + ) + + assert dispatch.TP2_MC_FINALIZE_SMALL and dispatch.MC_FINALIZE_MIN_TOKENS == 9 + monkeypatch.setattr(m31_switches, "_DISABLED", frozenset({"TP2_MC_FINALIZE_SMALL"})) + try: + m = importlib.reload(dispatch) + assert not m.TP2_MC_FINALIZE_SMALL + assert (m.MC_FINALIZE_MIN_TOKENS, m.MC_FINALIZE_MAX_TOKENS) == (33, 48) + assert not any(m.MC_FINALIZE.covers(r) for r in range(1, 33)) + finally: + monkeypatch.undo() + importlib.reload(dispatch) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_prebuild.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_prebuild.py new file mode 100644 index 000000000000..6b545e29e865 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_prebuild.py @@ -0,0 +1,372 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode kernels the core builds before the executor's warmup (``prebuild_kernels``), and that a forward after it +never compiles one: the MoE engines of every ``MOE_ENGINE`` / ``MOE16_ENGINE`` token count, the decode GEMVs with +their push variants, the index scorer with and without its early-issue schedule at every decode query length (and +the T2SC scorer, the long-row select and the full-context plans where they apply), and the 9..48-row generation +router's device decision (ROUTER48). At TP2 MOE16 is off (MOE16_TP2): no 9..13-row engine is built. + +The engine builds are stubbed (a real one needs a loaded checkpoint's expert buffers; the engine kernels have their +own tests); the GEMVs and the scorer compile for real on the GPU. +""" + +from types import SimpleNamespace + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers the trtllm ops +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_index_score as index_score_kernel, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_index_score_t2sc as index_score_t2sc_kernel, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.attention.m3_index_score import ( # noqa: E501 + m3_index_score, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.ops.gemm.m3_mx_gemv import ( # noqa: E501 + O_PROJ, + SHARED_DOWN, + SHARED_GATE_UP, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + require_msa_module, +) + +dispatch = modeling + +assert torch.cuda.is_available(), "the prebuild checks require a CUDA device" + +INDEX_HEADS = 2 # the TP2 per-rank index heads +INDEX_DIM = 128 +DRAFT_LEN = 7 # K7 (the TP2 baseline's draft length) +_CORE_METHODS = ( + "prebuild_kernels", + "_prebuild_engines", + "_try_build", + "engine", + "moe16_engine", + "_lookup", + "_scorer_variants", + "_prebuild_index_scorer", + "_prebuild_index_scorer_t2sc", + "_prebuild_index_scorer_intree", + "_prebuild_msa_proxy", + "decode_query_lens", +) + + +@pytest.fixture(autouse=True) +def _proxy_builds(monkeypatch): + """The MSA indexer proxy's fmha_sm100 variants are recorded rather than compiled (an nvcc build each).""" + require_msa_module() # its cutlass compatibility shim must precede the fmha_sm100 import, as in the target + from fmha_sm100 import jit as fmha_jit + + built = [] + + def record(*key): + built.append(fmha_jit._variant_key_from_runtime(*key)[0]) + return SimpleNamespace(run=None) + + monkeypatch.setattr(fmha_jit, "get_fmha_variant", record) + return built + + +@pytest.fixture(autouse=True) +def _restore_modes(): + require, frozen = dispatch.require_mode(), mx_gemv.frozen() + yield + dispatch.set_require(require) + mx_gemv.freeze(frozen) + dispatch.reset_logged() + + +class _FakeSparseAttn: + def __init__(self, indexer_kv_dtype: str): + self.indexer_kv_dtype = indexer_kv_dtype + self.m3_config = SimpleNamespace(num_index_heads=INDEX_HEADS, sparse_index_dim=INDEX_DIM) + + +def _core(num_layers: int = 3, indexer_kv_dtype: str = "fp8", built=None, fail=False): + """The parts of ``MiniMaxM3Tp2ModelingV2`` the prebuild reads: one rank (the build agreement takes no + collective), MoE + sparse layers 1.. (layer 0 dense), no projections (the GEMV prebuild has its own test), the + speculative config; the engine builds stubbed.""" + core = SimpleNamespace( + mapping=SimpleNamespace(tp_size=1, tp_rank=0, enable_attention_dp=False), + num_experts=128, + hidden=6144, + moe_layers=frozenset(range(1, num_layers)), + sparse_layers=frozenset(range(1, num_layers)), + attn=[_FakeSparseAttn(indexer_kv_dtype) for _ in range(num_layers)], + w={"embed": torch.empty(1, device="cuda")}, + mxfp8=SimpleNamespace(quant_method=SimpleNamespace(use_cutlass=True)), + model_config=SimpleNamespace( + spec_config=SimpleNamespace(max_draft_len=DRAFT_LEN, max_total_draft_tokens=DRAFT_LEN) + ), + moe_engines={}, + moe16_engines={}, + projections=lambda: [], + ) + for name in _CORE_METHODS: + setattr(core, name, getattr(modeling.MiniMaxM3Tp2ModelingV2, name).__get__(core)) + built = [] if built is None else built + + def build(i, num_tokens, shared=True): + if fail: + raise ValueError("the routed experts are not TRTLLM-Gen NVFP4 SwiGLU-OAI") + built.append(("moe", i, num_tokens, shared)) + return ("engine", i, num_tokens) + + def build16(i, num_tokens): + if fail: + raise ValueError("the routed experts are not TRTLLM-Gen NVFP4 SwiGLU-OAI") + built.append(("moe16", i, num_tokens)) + return ("engine16", i, num_tokens) + + core._build_engine, core._build_moe16_engine = build, build16 + return core + + +def _forbid_compiles(monkeypatch, core=None) -> None: + """Every compile / build entry point a forward could reach raises.""" + import cutlass.cute as cute + + def boom(*args, **kwargs): + raise AssertionError("a compile entry point ran after prebuild") + + monkeypatch.setattr(cute, "compile", boom) + monkeypatch.setattr(mx_gemv, "ensure_compiled", boom) + monkeypatch.setattr(mx_gemv, "precompile", boom) + monkeypatch.setattr(index_score_kernel.IndexScoreRunner, "_compile", boom) + monkeypatch.setattr(index_score_t2sc_kernel.IndexScoreRunner, "_compile", boom) + if core is not None: + core._build_engine = core._build_moe16_engine = boom + + +def test_decode_query_lens_follow_the_speculative_config() -> None: + core = _core() + assert core.decode_query_lens() == [1, 1 + DRAFT_LEN] + core.model_config.spec_config = None + assert core.decode_query_lens() == [1] + core.model_config.spec_config = SimpleNamespace( + max_draft_len=4, max_total_draft_tokens=4, draft_len_schedule={1: 4, 8: 2} + ) + assert core.decode_query_lens() == [1, 3, 5] + + +def test_prebuild_builds_every_engine_count_and_scorer_variant(monkeypatch) -> None: + built = [] + core = _core(built=built) + # The S1B fold's router pins probe cuBLASLt on the router weights (their own tests); here only that the prebuild + # asks for every MoE layer's router. + asked = [] + monkeypatch.setattr( + modeling.m31_s1b_fold, + "prebuild", + lambda routers, rows: asked.append((len(routers), list(rows))) or {}, + ) + core.w.update({f"l{i}_router": torch.empty(0) for i in core.moe_layers}) + times = core.prebuild_kernels() + assert asked and asked[0][0] == len(core.moe_layers) + + assert mx_gemv.frozen(), "prebuild must close the GEMV compiles" + + # Layer 0 is dense; every MoE layer gets every count, in layer order. + def per_layer(i): + engines = [("moe", i, n, True) for n in range(1, dispatch.MOE_ENGINE_MAX_TOKENS + 1)] + return engines + [ + ("moe16", i, n) for n in range(dispatch.MOE16_MIN_TOKENS, dispatch.MOE16_MAX_TOKENS + 1) + ] + + assert built == per_layer(1) + per_layer(2) + if not dispatch.MOE16_TP2: + assert not any(b[0] == "moe16" for b in built) + assert set(core.moe_engines[2]) == {(n, True, True) for n in dispatch.MOE_ENGINE.keys} + assert set(core.moe16_engines[2]) == set(dispatch.MOE16_ENGINE.keys) + + want = { + (torch.float8_e4m3fn, INDEX_HEADS, qlen, INDEX_DIM, modeling.INDEX_SCORE_SPLIT_K, early) + for qlen in (1, 1 + DRAFT_LEN) + for early in (True, False) + } + assert set(times["scorer_keys"]) == want + assert want <= set(index_score_kernel.IndexScoreRunner.kernel_cache) + if modeling.INDEX_SCORE_INTREE: + in_tree = modeling._in_tree_scorer() + want_intree = { + (torch.float8_e4m3fn, INDEX_HEADS, qlen, INDEX_DIM, in_tree.SPLIT_K) + for qlen in core.decode_query_lens() + if max(qlen * modeling.INDEX_SCORE_INTREE_MIN_REQUESTS, modeling.TP2_WIDE_MIN_ROWS) + <= modeling.DECODE_STEP_MAX + and in_tree.is_supported( + q_dtype=torch.float8_e4m3fn, + num_heads=INDEX_HEADS, + head_dim=INDEX_DIM, + page_size=in_tree.SUPPORTED_PAGE_SIZE, + max_decode_query_len=qlen, + ) + } + assert set(times["scorer_intree_keys"]) == want_intree + assert want_intree <= set(in_tree.kernel_cache) + else: + assert times["scorer_intree_keys"] == [] + for key in ("gemv_s", "moe_s", "scorer_s", "proxy_s"): + assert times[key] >= 0.0 + + # The generation router's device decision, as the built-in gate makes it. + assert core.router_tc_cluster16 == bool(torch.ops.trtllm.minimax_m3_router_tc_cluster16(6144)) + + # A second build builds no engine again. + built.clear() + core.prebuild_kernels() + assert built == [] + + +def test_engine_lookups_after_prebuild_never_build(monkeypatch) -> None: + core = _core() + core._prebuild_engines(5) + _forbid_compiles(monkeypatch, core) + for require in (False, True): + dispatch.set_require(require) + for n in sorted(dispatch.MOE_ENGINE.keys): + assert core.engine(5, n, shared=True) == ("engine", 5, n) + for n in sorted(dispatch.MOE16_ENGINE.keys): + assert core.moe16_engine(5, n) == ("engine16", 5, n) + + +def test_missing_engine_is_a_miss_not_a_build(monkeypatch) -> None: + core = _core() + core.moe_engines[7], core.moe16_engines[7] = {}, {} + _forbid_compiles(monkeypatch, core) + dispatch.set_require(False) + assert core.engine(7, 3, shared=True) is None + assert core.moe16_engine(7, 10) is None + dispatch.set_require(True) + with pytest.raises(dispatch.FastPathUnavailable, match="not built before warmup"): + core.engine(7, 3, shared=True) + if 10 in dispatch.MOE16_ENGINE.keys: + with pytest.raises(dispatch.FastPathUnavailable, match="not built before warmup"): + core.moe16_engine(7, 10) + else: + # MOE16 off at TP2 (MOE16_TP2): 10 rows are not a 9..13-row engine row, so the lookup is no fast path to + # miss (the generic MoE serves them), not a miss even under require. + assert core.moe16_engine(7, 10) is None + + +def test_failed_engine_build_is_logged_then_missed() -> None: + core = _core(fail=True) + core._prebuild_engines(9) + assert all(v is None for v in core.moe_engines[9].values()) and len(core.moe_engines[9]) == 8 + dispatch.set_require(True) + with pytest.raises(dispatch.FastPathUnavailable, match="build failed"): + core.engine(9, 1, shared=True) + + +def _projection(nk, push): + n, k = nk + w = (torch.randn(n, k, device="cuda") / k**0.5).to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + return SimpleNamespace(weight=wq.view(n, k), weight_scale=wsf, push=push) + + +@pytest.mark.parametrize( + "nk,push", + ((O_PROJ, "attn"), (SHARED_DOWN, "moe"), (SHARED_GATE_UP, None)), + ids=("o_proj", "shared_down", "shared_gate_up"), +) +def test_gemvs_after_prebuild_never_compile(monkeypatch, nk, push) -> None: + use_pdl = True + module = _projection(nk, push) + variants = modeling._gemv_variants(*nk, push, 2) + pushes = [v for v in variants if v[2] is not None] + if push is None: + assert not pushes + else: + covered = dispatch.ATTN_PUSH.keys if push == "attn" else dispatch.MOE_PUSH.keys + assert {rows for rows, _, _ in pushes} == set(covered) + assert {p[1] for _, _, p in pushes} == {b for b in (8, 16) if b <= max(covered)} + mx_gemv.freeze(False) + # As prebuild_kernels compiles one projection. + mx_gemv.ensure_compiled(module.weight, module.weight_scale, use_pdl) + for rows, x_bf16, ar in variants: + mx_gemv.precompile(module.weight, module.weight_scale, rows, x_bf16, use_pdl=use_pdl, ar=ar) + mx_gemv.freeze(True) + _forbid_compiles(monkeypatch) + + n, k = nk + for rows, x_bf16, ar in variants: + if ar is not None: + # A push needs a multicast exchange to launch: its kernel is compiled (the forward finds it). + resolved = mx_gemv._resolve(rows, n, k, x_bf16, True, None, use_pdl, (0, 0, 0, *ar)) + if resolved is not None: + assert mx_gemv.is_compiled(resolved[0]), (rows, x_bf16, ar) + continue + if x_bf16: + x = torch.randn(rows, k, device="cuda").to(torch.bfloat16) + out = mx_gemv.m3_mx_gemv_bf16(x, module.weight, module.weight_scale, use_pdl=use_pdl) + else: + xq, xsf = torch.ops.trtllm.mxfp8_quantize( + torch.randn(rows, k, device="cuda").to(torch.bfloat16), True + ) + out = mx_gemv.m3_mx_gemv( + xq.view(rows, k), xsf, module.weight, module.weight_scale, use_pdl=use_pdl + ) + served = mx_gemv.serves(n, k, rows, x_bf16) + assert (out is not None) == served, (rows, x_bf16) + assert mx_gemv.last_uncompiled is None + torch.cuda.synchronize() + + +def test_frozen_gemv_declines_an_uncompiled_kernel(monkeypatch) -> None: + module = _projection(SHARED_DOWN, None) + n, k = SHARED_DOWN # the TP2 shared down (6144, 1536) + mx_gemv.freeze(True) + _forbid_compiles(monkeypatch) + # A launch option nothing prebuilt (no early trigger, without a push): declined, never compiled. + xq, xsf = torch.ops.trtllm.mxfp8_quantize( + torch.randn(3, k, device="cuda").to(torch.bfloat16), True + ) + key = mx_gemv._resolve(3, n, k, False, False, None, True, None)[0] + if mx_gemv.is_compiled(key): + pytest.skip("this process already compiled the probe kernel") + out = mx_gemv.m3_mx_gemv( + xq.view(3, k), xsf, module.weight, module.weight_scale, trigger_early=False, use_pdl=True + ) + assert out is None and mx_gemv.last_uncompiled == key + + +@pytest.mark.parametrize("indexer_kv_dtype", ("fp8", "bf16")) +def test_scorer_after_prebuild_never_compiles(monkeypatch, indexer_kv_dtype) -> None: + core = _core(indexer_kv_dtype=indexer_kv_dtype) + keys = core._prebuild_index_scorer(core.decode_query_lens(), torch.device("cuda")) + assert len(keys) == 4 + _forbid_compiles(monkeypatch, core) + dtype = torch.float8_e4m3fn if indexer_kv_dtype == "fp8" else torch.bfloat16 + for _, heads, qlen, dim, split_k, early in keys: + rows = 1 if early else 2 + blocks = 3 + idx_q = torch.randn(rows * qlen, heads, dim, device="cuda").to(dtype) + idx_k = torch.randn(rows * blocks, 1, 128, dim, device="cuda").to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device="cuda").view( + rows, blocks + ) + seq_lens = torch.full((rows,), blocks * 128 - 5, dtype=torch.int32, device="cuda") + score = torch.empty(heads, blocks, rows * qlen, dtype=torch.float32, device="cuda") + m3_index_score( + idx_q, + idx_k.squeeze(1), + block_table, + seq_lens, + score.transpose(1, 2), + qlen, + split_k, + early, + ) + torch.cuda.synchronize() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_rank_agreement.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_rank_agreement.py new file mode 100644 index 000000000000..1ff8c39abd05 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m3_rank_agreement.py @@ -0,0 +1,195 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Decode robustness without a GPU kernel (the TP2 target, world 2): the multicast all-reduce epilogue options keep the +cluster-barrier bit (and the drafter form S9B-MC asks for at world 2 is a cluster-kernel instance), an engine build that +fails on one TP rank drops that engine on every rank (two ranks simulated with threads), and the Eagle3 fc fork's +output is recorded on the stream that reads it.""" + +import threading +from types import SimpleNamespace +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels import ( # noqa: E501 + m3_mc_allreduce as mc, +) + + +@pytest.fixture(autouse=True) +def _restore_modes(): + require = modeling.require_mode() + yield + modeling.set_require(require) + modeling.reset_logged() + + +@pytest.mark.parametrize("finalize", [False, True]) +@pytest.mark.parametrize("bucket", [8, 16, 32, 48]) +def test_ar_opts_keep_cluster_barrier_bit(monkeypatch, bucket, finalize): + # Tables without bit 2 and with buckets missing: every lookup, fallback included, keeps the barrier. + monkeypatch.setattr(mc, "AR_OPTS_BY_M", (0, 1, 4, 5, 8, 9, 12, 13)) + monkeypatch.setattr(mc, "AR_OPTS_BY_BUCKET", {}) + monkeypatch.setattr(mc, "AR_FIN_OPTS_BY_BUCKET", {}) + for m in range(1, min(bucket, 8) + 1): + opts = mc._ar_opts(m, bucket, finalize=finalize) + assert opts & mc._OPT_CLUSTER_BARRIER and 0 <= opts <= 15, (m, bucket, finalize, opts) + + +def test_ar_opts_shipped_tables_unchanged(): + for m in range(1, 9): + assert mc._ar_opts(m, 8) == mc.AR_OPTS_BY_M[m - 1] + for b, o in mc.AR_OPTS_BY_BUCKET.items(): + assert mc._ar_opts(b, b) == o + for b, o in mc.AR_FIN_OPTS_BY_BUCKET.items(): + assert mc._ar_opts(b, b, finalize=True) == o + + +@pytest.mark.parametrize("m", range(1, 9)) +def test_s9b_mc_drafter_opts_are_cluster_kernel_instances(m): + """S9B-MC (drafter_norm at world 2) launches opts 16 | the 8-token bucket's options: 23 or 31 after the row + kernel's carveout bit 64 is stripped, both instantiated by the 8-CTA cluster kernel (mcAllreduceNormTp4.cu + isSupported: 7 / 15 / 23 / 31 / 39 / 47 at bucket 8). The TP4 row kernel takes world 4 only.""" + opts = (16 | mc._ar_opts(m, 8) | mc._OPT_ROW_CARVEOUT) & ~(64 | 128) + assert opts in (23, 31) + + +class _FakeComm: + """An MPI-like TP communicator for ``world`` threads: allreduce sums the ranks' ints.""" + + def __init__(self, world): + self.world = world + self.barrier = threading.Barrier(world) + self.lock = threading.Lock() + self.vals = [] + self.calls = 0 + + def allreduce(self, v): + with self.lock: + self.vals.append(v) + self.calls += 1 + self.barrier.wait() + total = sum(self.vals) + self.barrier.wait() + with self.lock: + self.vals.clear() + self.barrier.wait() + return total + + def Get_size(self): # noqa: N802 (MPI name) + return self.world + + +def _mapping(tp_size=2, adp=False): + return SimpleNamespace(tp_size=tp_size, enable_attention_dp=adp, tp_group=list(range(tp_size))) + + +def _run_ranks(fns): + out, errs = [None] * len(fns), [] + + def run(i): + try: + out[i] = fns[i]() + except BaseException as e: # noqa: BLE001 + errs.append(e) + + threads = [threading.Thread(target=run, args=(i,)) for i in range(len(fns))] + for t in threads: + t.start() + for t in threads: + t.join(timeout=30) + assert not errs, errs + return out + + +@pytest.fixture +def two_rank_comm(monkeypatch): + from tensorrt_llm._torch.distributed import ops + + comm = _FakeComm(2) + monkeypatch.setattr(mc, "_tp_comm", lambda mapping: comm) + monkeypatch.setattr(ops, "mpi_disabled", lambda: False) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + return comm + + +def test_tp_agree_single_rank_and_adp_take_no_collective(monkeypatch): + monkeypatch.setattr(mc, "_tp_comm", mock.Mock(side_effect=AssertionError("no collective"))) + assert mc.tp_agree(_mapping(1), True) is True + assert mc.tp_agree(_mapping(1), False) is False + assert mc.tp_agree(_mapping(4, adp=True), False) is False + + +def test_tp_agree_refuses_capture(monkeypatch): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: True) + with pytest.raises(RuntimeError): + mc.tp_agree(_mapping(2), True) + + +@pytest.mark.parametrize("oks", [(True, True), (True, False), (False, True), (False, False)]) +def test_tp_agree_built_two_ranks(two_rank_comm, oks): + built = [object() if ok else None for ok in oks] + out = _run_ranks([lambda i=i: mc.tp_agree_built(_mapping(2), built[i], "t") for i in range(2)]) + assert out == (built if all(oks) else [None, None]) + + +def _core(fail: bool) -> SimpleNamespace: + """The engine-building half of the core (``MiniMaxM3Tp2ModelingV2``) on one of two simulated TP ranks, its + engine builds stubbed (and failing on one rank).""" + core = SimpleNamespace(mapping=_mapping(2), moe_engines={}, moe16_engines={}) + cls = modeling.MiniMaxM3Tp2ModelingV2 + + def build(i, num_tokens, shared=True): + if fail: + raise RuntimeError("compile failed on this rank") + return ("engine", num_tokens) + + core._build_engine = build + core._build_moe16_engine = lambda i, num_tokens: build(i, num_tokens) + for name in ("_prebuild_engines", "_try_build", "engine", "moe16_engine", "_lookup"): + setattr(core, name, getattr(cls, name).__get__(core)) + return core + + +@pytest.mark.parametrize("fail", [(False, False), (False, True)]) +def test_engine_build_agreed_across_ranks(two_rank_comm, fail): + cores = [_core(f) for f in fail] + _run_ranks([lambda c=c: c._prebuild_engines(3) for c in cores]) + keys = sorted(cores[0].moe_engines[3]) + sorted(cores[0].moe16_engines[3]) + assert keys and two_rank_comm.calls == 2 * len(keys) + # MoE16 is off at the TP2 width (MOE16_TP2): no 9..13-row engine is built or agreed. + assert sorted(cores[0].moe16_engines[3]) == sorted(modeling.MOE16_ENGINE.keys) + if not modeling.MOE16_TP2: + assert cores[0].moe16_engines[3] == {} + for engines in ("moe_engines", "moe16_engines"): + a, b = getattr(cores[0], engines)[3], getattr(cores[1], engines)[3] + assert a.keys() == b.keys() + for key in a: + assert (a[key] is None) == (b[key] is None) == any(fail), (engines, key) + # Both ranks then take the same path in the forward: the engine, or (after a miss) the generic MoE. + modeling.set_require(False) + for n in sorted(modeling.MOE_ENGINE.keys): + got = [c.engine(3, n, shared=True) for c in cores] + assert (got[0] is None) == (got[1] is None) + + +@pytest.mark.parametrize("capturing", [False, True]) +def test_fc_join_records_output_on_main(monkeypatch, capturing): + M3Eagle3Worker = modeling.M3Eagle3Worker + main, fork = mock.Mock(name="main"), mock.Mock(name="fork") + hidden = mock.Mock(name="hidden", is_cuda=True) + monkeypatch.setattr(torch.cuda, "current_stream", lambda *a: main) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: capturing) + worker = SimpleNamespace(_m3_fc_hidden=hidden) + M3Eagle3Worker._m3_fc_join(worker, fork) + main.wait_stream.assert_called_once_with(fork) + if capturing: + hidden.record_stream.assert_not_called() + else: + hidden.record_stream.assert_called_once_with(main) + worker._m3_fc_hidden = None + M3Eagle3Worker._m3_fc_join(worker, fork) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/activation/test_modeling_v2_m3_silu_and_mul.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/activation/test_modeling_v2_m3_silu_and_mul.py new file mode 100644 index 000000000000..4d35b9dc05c2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/activation/test_modeling_v2_m3_silu_and_mul.py @@ -0,0 +1,49 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_silu_and_mul entry of the MiniMax-M3 tp4 target: SwiGLU-OAI and plain SwiGLU.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.activation.m3_silu_and_mul import ( # noqa: E501 + m3_silu_and_mul as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_silu_and_mul requires a CUDA device" + +OAI = (7.0, 1.702, 1.0) + + +def _build(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + # Gate and up spread past the clamp (|x| up to ~12) so both clamps engage. + x = (torch.randn(spec["m"], 2 * spec["inter"], generator=g, device="cuda") * 4.0).to( + torch.bfloat16 + ) + return (x, *OAI) if spec["oai"] else (x, None, None, None) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """fp32 arithmetic and one bf16 rounding on both sides: the kernel's fast sigmoid moves the fp32 value by a few + fp32 ulps, which flips the bf16 rounding of an element by at most one step (1 unit of ULP[bf16] of the row); + measured on GB300: worst element 0.5 unit, relative RMS 0.01.""" + args = _build(cell.spec, zlib.crc32(cell.why.encode()) % 2**31) + with validating(op): + out = op(*args) + assert out.shape == (cell.spec["m"], cell.spec["inter"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(*args), element_ulp=1.0, rms_ulp=0.1) + + +def test_the_gate_sees_a_missing_clamp() -> None: + """Dropping the SwiGLU-OAI clamps moves every element whose gate or up passed the limit far outside the band.""" + spec = op.CELLS[0].spec + x, *oai = _build(dict(spec, m=64), 3) + out = op(x, *oai) + with pytest.raises(AssertionError): + assert_within_ulp(out, op.reference(x, None, oai[1], oai[2]), element_ulp=1.0, rms_ulp=0.1) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/_m3_attn_runtime_tp4.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/_m3_attn_runtime_tp4.py new file mode 100644 index 000000000000..7bfd8de384f9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/_m3_attn_runtime_tp4.py @@ -0,0 +1,274 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Real MiniMax-M3 attention runtime state at the TP4 per-rank geometry, for the backend-step entries' GPU tests. + +One hybrid NVFP4 KV cache manager (``MiniMaxM3KVCacheManagerV2``) laid out as the target's: a dense layer in the FP8 +half, a sparse MSA layer in the NVFP4 half with its FP8 index-K cache, and (``draft=True``) the one-model Eagle3 +drafter's shared layer, read through the manager's FP8 draft view as the runtime's draft swap reads it. Metadata is +the runtime's own type, filled by its own ``prepare()``; the backends are built the way the target builds them. + +The geometry is one rank's: 16 query heads over one KV head and one index head of dim 128. The mapping is a single +rank holding exactly those heads, so nothing is sharded again. + +History rows are written by the in-tree scatters (``fused_write_layer_caches`` for the FP8 layers, +``fused_write_layer_caches_nvfp4`` for the NVFP4 layer and its index-K), at the slots the manager assigned, which are +not in request order. +""" + +from __future__ import annotations + +from typing import Sequence + +import torch + +from tensorrt_llm._torch.attention.backends.interface import PositionalEmbeddingParams, RopeParams +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import ( + MiniMaxM3KVCacheManagerV2, + MiniMaxM3MsaSparseAttention, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_scatter import ( + fused_write_layer_caches, + fused_write_layer_caches_nvfp4, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.msa_backend import ( + MiniMaxM3MsaSparseAttentionMetadata, +) +from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention, TrtllmAttentionMetadata +from tensorrt_llm._torch.attention.backends.utils import create_attention +from tensorrt_llm._torch.metadata import KVCacheParams +from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy +from tensorrt_llm.bindings import DataType +from tensorrt_llm.bindings.internal.batch_manager import CacheType +from tensorrt_llm.functional import PositionEmbeddingType +from tensorrt_llm.llmapi.llm_args import ( + Eagle3DecodingConfig, + KvCacheConfig, + MiniMaxM3SparseAttentionConfig, +) +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.models.modeling_utils import QuantConfig +from tensorrt_llm.quantization.mode import QuantAlgo + +HEADS = 16 +KV_HEADS = 1 +INDEX_HEADS = 1 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 +DENSE, SPARSE, DRAFT = 0, 1, 2 +MAX_BATCH = 8 +MAX_SEQ_LEN = 8192 +MAX_NUM_TOKENS = 8192 +#: The Eagle3 drafter checkpoint's RoPE: NeoX over the whole head, theta 5e6. +DRAFT_ROPE_THETA = 5.0e6 + +#: E4M3-exact planted index levels, 1/8 apart in relative terms (0.25 .. 15): 48 distinct block scores. +LEVELS = tuple(2.0**e * (1 + m / 8) for e in range(-2, 4) for m in range(8)) +#: The index value of a block's non-peak rows; its score (8) is below every level's (>= 16). +LOW = 0.125 + + +def mapping() -> Mapping: + return Mapping(world_size=1, rank=0, tp_size=1, pp_size=1) + + +def sparse_attention_config() -> MiniMaxM3SparseAttentionConfig: + """The target's runtime: MSA, the FP8 indexer, the fused five-way projection (M3's checkpoint selection rule: + no forced initial blocks, the query's own block forced).""" + return MiniMaxM3SparseAttentionConfig( + implementation="msa", + indexer_kv_dtype="fp8", + fuse_qkv_index_projection=True, + sparse_num_index_heads=INDEX_HEADS, + num_attention_heads=HEADS, + num_key_value_heads=KV_HEADS, + ) + + +def make_manager(*, draft: bool) -> MiniMaxM3KVCacheManagerV2: + torch.cuda.init() + return MiniMaxM3KVCacheManagerV2( + KvCacheConfig( + max_tokens=4 * MAX_SEQ_LEN, + enable_block_reuse=False, + enable_swa_scratch_reuse=False, + host_cache_size=0, + event_buffer_max_size=0, + ), + CacheType.SELF, + num_layers=2, + num_kv_heads=[KV_HEADS] * (3 if draft else 2), + head_dim=HEAD_DIM, + tokens_per_block=PAGE, + max_seq_len=MAX_SEQ_LEN, + max_batch_size=MAX_BATCH, + max_num_tokens=MAX_NUM_TOKENS, + mapping=mapping(), + dtype=DataType.NVFP4, + vocab_size=16, + spec_config=( + Eagle3DecodingConfig(max_draft_len=3, speculative_model="unused") if draft else None + ), + sparse_layer_ids=[SPARSE], + sparse_attention_config=sparse_attention_config(), + ) + + +def make_msa_backend(layer: int) -> MiniMaxM3MsaSparseAttention: + """One layer's MSA backend as ``MiniMaxM3Tp4ModelingV2`` builds it (NVFP4 KV quant config on every layer).""" + quant = QuantConfig(kv_cache_quant_algo=QuantAlgo.NVFP4) + backend = create_attention( + "TRTLLM", + layer, + HEADS, + HEAD_DIM, + KV_HEADS, + pos_embd_params=None, + quant_config=quant, + q_scaling=1.0, + sparse_params=sparse_attention_config().to_sparse_params(layer_idx=layer), + ) + backend.update_quant_config(quant) + assert isinstance(backend, MiniMaxM3MsaSparseAttention), type(backend).__name__ + return backend + + +def make_draft_backend() -> TrtllmAttention: + """The drafter layer's attention as ``M3Eagle3Draft`` builds it: fused NeoX RoPE, an FP8 KV cache (the target + overrides the drafter's KV dtype to FP8), no other quantization.""" + quant = QuantConfig(kv_cache_quant_algo=QuantAlgo.FP8) + backend = create_attention( + "TRTLLM", + DRAFT, + HEADS, + HEAD_DIM, + KV_HEADS, + pos_embd_params=PositionalEmbeddingParams( + type=PositionEmbeddingType.rope_gpt_neox, + rope=RopeParams(dim=HEAD_DIM, theta=DRAFT_ROPE_THETA, max_positions=MAX_SEQ_LEN), + ), + quant_config=quant, + q_scaling=1.0, + attention_chunk_size=None, + sparse_params=None, + ) + backend.update_quant_config(quant) + assert type(backend) is TrtllmAttention, type(backend).__name__ + return backend + + +def make_msa_metadata(manager) -> MiniMaxM3MsaSparseAttentionMetadata: + metadata = MiniMaxM3MsaSparseAttentionMetadata( + max_num_requests=MAX_BATCH, + max_num_tokens=MAX_NUM_TOKENS, + max_num_sequences=MAX_BATCH, + kv_cache_manager=manager, + mapping=mapping(), + sparse_metadata_params=sparse_attention_config().to_sparse_metadata_params(), + ) + # The target's MSA preparation choice that changes what prepare() stages (the identity tables of short steps). + metadata.msa_policy = MsaRuntimePolicy(short_sequence_identity=True) + return metadata + + +def make_draft_metadata(manager) -> TrtllmAttentionMetadata: + """Drafter metadata over the shared manager's FP8 draft view, the manager the runtime's draft swap installs.""" + view = manager.get_draft_kv_cache_view() + assert view is not None and view.dtype == DataType.FP8 + return TrtllmAttentionMetadata( + max_num_requests=MAX_BATCH, + max_num_tokens=MAX_NUM_TOKENS, + kv_cache_manager=view, + mapping=mapping(), + ) + + +def allocate(manager, request_ids: Sequence[int], qlens: Sequence[int], cached: Sequence[int]): + manager.add_dummy_requests(list(request_ids), token_nums=[c + n for c, n in zip(cached, qlens)]) + + +def prepare(metadata, request_ids, qlens, cached, num_contexts: int) -> None: + metadata.seq_lens = torch.tensor(list(qlens), dtype=torch.int) + metadata.num_contexts = num_contexts + metadata.request_ids = list(request_ids) + metadata.prompt_lens = [ + c + n if row < num_contexts else c for row, (c, n) in enumerate(zip(cached, qlens)) + ] + metadata.kv_cache_params = KVCacheParams(use_cache=True, num_cached_tokens_per_seq=list(cached)) + metadata.prepare() + + +def flat_slots(manager, layer: int, request_id: int, positions: torch.Tensor) -> torch.Tensor: + """The cache slots (page slot id * PAGE + offset) of a request's positions in ``layer``.""" + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0], + dtype=torch.int64, + device="cuda", + ) + positions = positions.to(device="cuda", dtype=torch.int64) + return (pages[positions // PAGE] * PAGE + positions % PAGE).to(torch.int32) + + +def write_fp8_rows(manager, layer, request_id, positions, k, v) -> None: + """K / V rows [n, 128] (E4M3) into an FP8 layer's cache at ``positions``.""" + view = manager.get_buffers(layer, "HND") + assert view.dtype == torch.float8_e4m3fn + slots = flat_slots(manager, layer, request_id, positions) + assert fused_write_layer_caches(view[:, 0], view[:, 1], None, slots, k, v, None) + + +def write_nvfp4_rows(manager, metadata, layer, request_id, positions, k, v, idx_k, quant) -> None: + """BF16 K / V rows quantized into the NVFP4 layer's cache, and index-K rows into its FP8 index cache.""" + data = manager.get_buffers(layer, "HND") + scales = manager.get_block_scale_buffers(layer, "HND") + slots = flat_slots(manager, layer, request_id, positions) + assert fused_write_layer_caches_nvfp4( + data[:, 0], + data[:, 1], + scales[:, 0], + scales[:, 1], + metadata.msa_idx_k_cache(layer), + slots, + k, + v, + idx_k, + quant, + ) + + +def index_directions(gen: torch.Generator) -> tuple[torch.Tensor, torch.Tensor]: + """Two orthogonal ±1 index directions (the halves of the 128 dims), E4M3-exact.""" + signs = torch.randint(0, 2, (HEAD_DIM,), generator=gen, device="cuda").float() * 2 - 1 + u0 = torch.zeros(HEAD_DIM, device="cuda") + u1 = torch.zeros(HEAD_DIM, device="cuda") + u0[: HEAD_DIM // 2] = signs[: HEAD_DIM // 2] + u1[HEAD_DIM // 2 :] = signs[HEAD_DIM // 2 :] + return u0, u1 + + +def planted_index_keys(length: int, u0, u1, gen: torch.Generator) -> torch.Tensor: + """Index-K rows [length, 128] (bf16, E4M3-exact) whose per-block maximum score is unambiguous. + + Every block has one peak row at a random position, ``a_b u0 + c_b u1`` with ``a`` and ``c`` two independent + permutations of ``LEVELS``; its other rows are ``LOW (u0 + u1)``. A query along ``u0`` then ranks the blocks by + ``a``, one along ``u1`` by ``c``, every score 1/8 or more apart from the next.""" + blocks = (length + PAGE - 1) // PAGE + assert blocks <= len(LEVELS) + levels = torch.tensor(LEVELS, device="cuda") + a = levels[torch.randperm(len(LEVELS), generator=gen, device="cuda")[:blocks]] + c = levels[torch.randperm(len(LEVELS), generator=gen, device="cuda")[:blocks]] + rows = (LOW * (u0 + u1)).expand(length, HEAD_DIM).clone() + for b in range(blocks): + span = min(PAGE, length - b * PAGE) + peak = b * PAGE + int(torch.randint(0, span, (1,), generator=gen, device="cuda")) + rows[peak] = a[b] * u0 + c[b] * u1 + return rows.to(torch.bfloat16) + + +def nvfp4_quant(*values: torch.Tensor) -> torch.Tensor: + """The (Q, K, V) quantization scales ``fused_write_layer_caches_nvfp4`` takes: each role's amax at the top of the + E2M1 x E4M3 range.""" + quant = torch.ones(3, dtype=torch.float32, device="cuda") + for role, x in enumerate(values, start=1): + quant[role] = (448.0 * 6.0) / x.abs().max().float() + return quant diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/_m3_nvfp4_cache.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/_m3_nvfp4_cache.py new file mode 100644 index 000000000000..2254f6c0b68c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/_m3_nvfp4_cache.py @@ -0,0 +1,104 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""A MiniMax-M3 TP4 sparse-layer decode step over a packed NVFP4 cache, for the attention entries' tests. + +The cache is written by the in-tree scatter (``fused_write_layer_caches_nvfp4``, +which quantizes with ``torch.ops.trtllm.fp4_quantize``), into one +``[pages, 2 (K, V), 1, 128, .]`` pool per role exactly as ``msa_paged_kv`` and +``get_block_scale_buffers`` hand the K and V views over. Pages are shuffled so +a kernel that ignored the block table would read the wrong ones. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from typing import Sequence + +import torch + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_scatter import ( + fused_write_layer_caches_nvfp4, +) + +NUM_HEADS = 16 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 + + +def token_kv_len(seq_len: int, decode_query_len: int, offset: int) -> int: + """The KV extent query token ``offset`` of a request attends, as every decode kernel derives it.""" + return max(seq_len - decode_query_len + offset + 1, 0) + + +def build_step( + seq_lens: Sequence[int], decode_query_len: int, *, identity: bool, seed: int +) -> SimpleNamespace: + """One decode step: E4M3 q [T, 16, 128], the packed NVFP4 K/V views and their scales, a [1, T, 16] + head-major top-k table (the identity table, or a random ascending selection with a -1 tail), the block + table and per-request lengths.""" + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + total_q = batch * decode_query_len + max_blocks = max(1, max((s + PAGE - 1) // PAGE for s in seq_lens)) + num_pages = batch * max_blocks + num_slots = num_pages * PAGE + block_table = ( + torch.randperm(num_pages, device="cuda", generator=g) + .to(torch.int32) + .reshape(batch, max_blocks) + ) + data_pool = torch.zeros(num_pages, 2, 1, PAGE, HEAD_DIM // 2, dtype=torch.uint8, device="cuda") + scale_pool = torch.zeros( + num_pages, 2, 1, PAGE, HEAD_DIM // 16, dtype=torch.uint8, device="cuda" + ) + quant = torch.ones(3, dtype=torch.float32, device="cuda") + values = [] + for role in (0, 1): + x = torch.randn(num_slots, HEAD_DIM, device="cuda", generator=g, dtype=torch.float32).to( + torch.bfloat16 + ) + quant[role + 1] = (448.0 * 6.0) / x.abs().max().float().item() + values.append(x) + assert fused_write_layer_caches_nvfp4( + data_pool[:, 0], + data_pool[:, 1], + scale_pool[:, 0], + scale_pool[:, 1], + None, + torch.arange(num_slots, device="cuda", dtype=torch.int32), + values[0], + values[1], + None, + quant, + ) + q = torch.randn(total_q, NUM_HEADS, HEAD_DIM, device="cuda", generator=g).to( + torch.float8_e4m3fn + ) + topk = torch.full((1, total_q, TOPK), -1, dtype=torch.int32, device="cuda") + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + nblocks = (token_kv_len(seq_lens[req], decode_query_len, offset) + PAGE - 1) // PAGE + if identity: + assert nblocks <= TOPK, "an identity step has every request within 16 blocks" + chosen = torch.arange(nblocks, device="cuda") + else: + chosen = torch.randperm(nblocks, device="cuda", generator=g)[:TOPK].sort().values + topk[0, tok, : chosen.numel()] = chosen.to(torch.int32) + return SimpleNamespace( + q=q, + k_paged=data_pool[:, 0], + v_paged=data_pool[:, 1], + k_block_scale=scale_pool[:, 0], + v_block_scale=scale_pool[:, 1], + k_global_scale=(1.0 / quant[1:2]).contiguous(), + v_global_scale=(1.0 / quant[2:3]).contiguous(), + topk=topk, + block_table=block_table, + seq_lens=torch.tensor(list(seq_lens), dtype=torch.int32, device="cuda"), + decode_query_len=decode_query_len, + sm_scale=HEAD_DIM**-0.5, + source=values, + quant=quant, + num_slots=num_slots, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m31_gqamix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m31_gqamix.py new file mode 100644 index 000000000000..b2b0294ae235 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m31_gqamix.py @@ -0,0 +1,189 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GQAMIX: a mixed step's sparse generation rows (MsaDecodeFmha) through ops/gqamix_sparse_decode. + +The copy must be the in-tree kernel module except for the two decode kernels' specialization (no row count, no slice +alignment) and the one import it cannot make relatively, and must write the in-tree kernel's output bit for bit on a +mixed step's [num_ctx:] slices. The shared MsaDecodeFmha hook must change nothing while it is unset. +""" + +import inspect +import os +import subprocess +import sys +import types +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.triton_sparse_decode as in_tree +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + modeling as m, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops import ( + gqamix_sparse_decode as copy, +) +from tensorrt_llm._torch.attention.backends.fmha.msa_decode import MsaDecodeFmha + +HEADS = 16 # query heads per rank at tp4, one KV head +D, PAGE, TOPK, QLEN, SF_VEC = 128, 128, 16, 4, 16 + +_JIT_OLD = '@triton.jit(do_not_specialize=["decode_query_len"])' +_JIT_NEW = ( + "@triton.jit(\n" + ' do_not_specialize=["decode_query_len", "total_q"],\n' + ' do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"],\n' + ")" +) +_IMPORT_OLD = "from .msa_utils import check_decode_span_shape" +_IMPORT_NEW = ( + "from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import (\n" + " check_decode_span_shape,\n" + ")" +) + + +def _code_lines(text: str) -> list: + return [line for line in text.splitlines() if line.strip()] + + +def test_the_copy_is_the_in_tree_module_but_for_its_specialization() -> None: + """Line for line (blank lines aside: the import sorts into a different block) the in-tree module.""" + ours = Path(copy.__file__).read_text() + theirs = Path(in_tree.__file__).read_text() + assert ( + ours.count(_JIT_NEW) == 2 + and ours.count(_IMPORT_NEW) == 1 + and theirs.count(_IMPORT_OLD) == 1 + ) + ours = ours.replace(_JIT_NEW, _JIT_OLD).replace(_IMPORT_NEW, "") + assert _code_lines(ours) == _code_lines(theirs.replace(_IMPORT_OLD, "")), ( + "ops/gqamix_sparse_decode.py must stay a copy of the in-tree triton_sparse_decode.py: re-copy it" + ) + + +def test_register_sets_the_hook_once_and_only_where_it_exists() -> None: + lib = types.SimpleNamespace(sparse_decode_hook=None) + other_lib = types.SimpleNamespace() # an FMHA library without the hook + manager = types.SimpleNamespace(fmha_libs=[other_lib, lib]) + m._gqamix_register(types.SimpleNamespace(_fmha_manager=manager)) + assert lib.sparse_decode_hook is m._gqamix_decode + assert not hasattr(other_lib, "sparse_decode_hook") + mine = object() + lib.sparse_decode_hook = mine + m._gqamix_register(types.SimpleNamespace(_fmha_manager=manager)) + assert lib.sparse_decode_hook is mine + m._gqamix_register(types.SimpleNamespace()) # a backend without an FMHA manager + + +def test_the_kill_switch_leaves_the_pre_hook_decode() -> None: + """TRTLLM_M31_DISABLE_GQAMIX=1: GQAMIX is off, so no layer registers the hook and MsaDecodeFmha runs the in-tree + decode (whose output the copy matches bit for bit below).""" + target = "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4" + probe = f"import {target}.modeling as m; print(m.GQAMIX)" + env = dict(os.environ, TRTLLM_M31_DISABLE_GQAMIX="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False" + source = inspect.getsource(m.PrefillTarget) + assert "if GQAMIX and sparse:\n _gqamix_register(core.attn[i])" in source + assert source.count("_gqamix_register(") == 1 + + +class _Attn: + is_mla_enable = False + kv_lora_rank = None + v_head_dim = None + head_dim = D + num_heads = HEADS + layer_idx = 3 + q_scaling = 1.0 + + +def _params(attn, num_tokens): + return types.SimpleNamespace( + attn=attn, + num_tokens=num_tokens, + meta=types.SimpleNamespace(kv_cache_manager=object()), + fwd=types.SimpleNamespace(kv_scale_quant_orig=None), + query_input=torch.zeros(num_tokens, HEADS * D), + output=torch.zeros(num_tokens, HEADS * D), + token_offset=2, + input_seq_length=QLEN, + ) + + +@pytest.mark.parametrize("hooked", [False, True], ids=["unset", "registered"]) +def test_the_shared_decode_hook_is_inert_unset(monkeypatch, hooked) -> None: + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import msa_utils + + calls = [] + monkeypatch.setattr(msa_utils, "msa_paged_kv", lambda manager, layer: ("k", "v")) + monkeypatch.setattr( + in_tree, "minimax_m3_sparse_attn_decode", lambda *a, **k: calls.append(("in_tree", a, k)) + ) + attn = _Attn() + fmha = MsaDecodeFmha(attn) + assert fmha.sparse_decode_hook is None + if hooked: + fmha.sparse_decode_hook = lambda *a, **k: calls.append(("hook", a, k)) + indexes = torch.zeros(10, 1, TOPK, dtype=torch.int32) + fmha._run_sparse(_params(attn, QLEN), indexes, "bt", "sl") + assert [c[0] for c in calls] == ["hook" if hooked else "in_tree"] + _, args, kwargs = calls[0] + assert args[1:3] == ("k", "v") and args[4:] == ("bt", "sl") + assert args[3].shape == (1, QLEN, TOPK) + assert kwargs["decode_query_len"] == QLEN and kwargs["sm_scale"] == D**-0.5 + + +def _mixed_step(nreq, seq_len, row_first, seed): + """A mixed step's generation rows: NVFP4 caches, FP8 q, and block-table / length slices past row_first.""" + g = torch.Generator(device="cuda").manual_seed(seed) + nb = seq_len // PAGE + pages = (nreq + row_first) * nb + block_table = torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(-1, nb) + seq_lens = torch.full((nreq + row_first,), seq_len, device="cuda", dtype=torch.int32) + total_q = nreq * QLEN + q = torch.randn(total_q, HEADS, D, device="cuda", generator=g).to(torch.float8_e4m3fn) + kv = [ + torch.randint( + 0, 256, (pages, 1, PAGE, D // 2), device="cuda", generator=g, dtype=torch.uint8 + ) + for _ in "kv" + ] + sc = [ + (torch.rand(pages, 1, PAGE, D // SF_VEC, device="cuda", generator=g) + 0.5).to( + torch.float8_e4m3fn + ) + for _ in "kv" + ] + topk = torch.stack( + [torch.randperm(nb, device="cuda", generator=g)[:TOPK] for _ in range(total_q)] + ) + args = (q, kv[0], kv[1], topk.to(torch.int32).view(total_q, 1, TOPK).permute(1, 0, 2)) + args += (block_table[row_first:], seq_lens[row_first:]) + kwargs = dict( + sm_scale=D**-0.5, + decode_query_len=QLEN, + k_block_scale=sc[0], + v_block_scale=sc[1], + k_global_scale=torch.tensor([0.02], device="cuda"), + v_global_scale=torch.tensor([0.03], device="cuda"), + ) + return args, kwargs, total_q + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA device") +@pytest.mark.parametrize("nreq,row_first", [(1, 0), (1, 1), (3, 2), (5, 3), (8, 1)]) +@pytest.mark.parametrize("seq_len", [16384, 131072]) +def test_the_copy_writes_the_in_tree_output_on_a_mixed_step(nreq, row_first, seq_len) -> None: + args, kwargs, total_q = _mixed_step(nreq, seq_len, row_first, 31 * nreq + row_first) + ref = torch.empty(total_q, HEADS, D, device="cuda", dtype=torch.bfloat16) + out = torch.empty_like(ref) + in_tree.minimax_m3_sparse_attn_decode(*args, output=ref, **kwargs) + copy.minimax_m3_sparse_attn_decode(*args, output=out, **kwargs) + torch.cuda.synchronize() + assert torch.isfinite(ref.float()).all() + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_dense_decode.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_dense_decode.py new file mode 100644 index 000000000000..096e0333f2c4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_dense_decode.py @@ -0,0 +1,153 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_dense_decode entry: the dense layers' trtllm-gen decode at the KV split this target +sizes per request count. + +Every cell runs against the fp32 reference, and a capped cell also against the same call at the model maximum (the +split the in-tree step hands it): the two differ only by the order the CTAs' partial softmax sums are merged in, held +here to a few BF16 ulps. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_dense_decode import ( # noqa: E501 + CAPPED, + HEAD_DIM, + HEADS, + KV_HEADS, + MODEL_MAX, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_dense_decode import ( # noqa: E501 + m3_dense_decode as op, +) +from tensorrt_llm._torch.attention.backends.fmha.msa_decode import dense_decode_max_seq_len + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_dense_decode requires a CUDA device" + +PAGE = 32 +SUBPAGES = 2 +SM_SCALE = HEAD_DIM**-0.5 +# The FP8-KV band of the in-tree dense decode test (test_minimax_m3_dense_decode.test_matches_reference). +FP8_TOL = 6e-2 + + +class _Pool: + """The one method the kernel asks of a KV cache manager.""" + + def __init__(self, pool: torch.Tensor): + self.pool = pool + + def get_kv_subpage_pool(self, layer_idx: int, kv_layout: str = "HND"): + assert kv_layout == "HND" + return self.pool, SUBPAGES + + +def _inputs(kv_lens, qlen: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(kv_lens) + pages = max((n + PAGE - 1) // PAGE for n in kv_lens) + slots = batch * pages + 3 + pool = torch.randn( + (slots * SUBPAGES, KV_HEADS, PAGE, HEAD_DIM), device="cuda", generator=g + ).clamp(-4, 4) + # Slots handed out non-contiguously, as the block manager does. + perm = torch.randperm(slots, generator=g, device="cuda")[: batch * pages] + block_table = perm.reshape(batch, pages).to(torch.int32) + # The last third of every request's values is shifted by +1: a third of each row's softmax mass sits there, far + # outside the band, so a split that missed the tail of a context longer than its max_seq_len fails. + for b, n in enumerate(kv_lens): + start = 2 * n // 3 + first_full = (start + PAGE - 1) // PAGE + last = (n + PAGE - 1) // PAGE + if first_full < last: + v_idx = block_table[b, first_full:last].long() * SUBPAGES + 1 + pool[v_idx] += 1.0 + if start % PAGE: + v0 = int(block_table[b, start // PAGE]) * SUBPAGES + 1 + pool[v0, :, start % PAGE :] += 1.0 + q = torch.randn((batch * qlen, HEADS, HEAD_DIM), device="cuda", generator=g).to(torch.bfloat16) + return ( + q, + _Pool(pool.to(torch.float8_e4m3fn)), + block_table, + torch.tensor(kv_lens, device="cuda", dtype=torch.int32), + ) + + +def _args(q, manager, block_table, seq_lens, qlen, max_seq_len): + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + return ( + q, + manager, + 0, + block_table, + seq_lens, + SM_SCALE, + out, + qlen, + max_seq_len, + seq_lens.shape[0], + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + q, manager, block_table, seq_lens = _inputs( + spec["kv_lens"], spec["qlen"], zlib.crc32(cell.why.encode()) % 2**31 + ) + args = _args(q, manager, block_table, seq_lens, spec["qlen"], spec["max_seq_len"]) + certifying(op) + ref = op.reference(*args) + out = op(*args).clone() + torch.testing.assert_close(out.float(), ref, rtol=FP8_TOL, atol=FP8_TOL) + if spec["capped"]: + uncapped = op(*_args(q, manager, block_table, seq_lens, spec["qlen"], MODEL_MAX)).clone() + # The split reorders only the merge of the CTAs' fp32 partial sums before the one BF16 rounding: every element + # is within 4 of its own BF16 spacing (2^(e - 8) for |x| = m * 2^e, m in [0.5, 1)). + ref_bf16 = uncapped.float() + _, exp = torch.frexp(ref_bf16.abs()) + spacing = torch.where( + ref_bf16 == 0, + torch.full_like(ref_bf16, 2.0**-133), + torch.ldexp(torch.ones_like(ref_bf16), exp - 8), + ) + assert bool(((out.float() - ref_bf16).abs() <= 4 * spacing).all()) + assert_within_ulp(out, uncapped, element_ulp=4.0, rms_ulp=1.0) + + +@pytest.mark.skipif( + not dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH, + reason="the TP4 SM 10.7 target has no dense-decode cap until one is measured and certified on Rubin", +) +def test_the_cap_is_the_dispatch_constant() -> None: + """The entry's capped cells run at the split the target hands trtllm-gen at one request, and the in-tree step + resolves the cap from the attention module only at the request counts it names.""" + assert dict(dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH) == {1: CAPPED} + attn = type( + "Attn", + (), + {"dense_decode_max_seq_len_by_batch": dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH}, + )() + assert dense_decode_max_seq_len(attn, MODEL_MAX, 1) == CAPPED + assert dense_decode_max_seq_len(attn, MODEL_MAX, 2) == MODEL_MAX + assert dense_decode_max_seq_len(attn, 4096, 1) == 4096 + assert dense_decode_max_seq_len(object(), MODEL_MAX, 1) == MODEL_MAX + + +def test_guard_refuses_more_requests_than_the_counters() -> None: + q, manager, block_table, seq_lens = _inputs((3000, 3000), 1, 3) + args = list(_args(q, manager, block_table, seq_lens, 1, CAPPED)) + args[9] = 1 + with pytest.raises(AssertionError, match="counters cover"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_dense_qk_norm_rope_fp8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_dense_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..b41186a93406 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_dense_qk_norm_rope_fp8.py @@ -0,0 +1,180 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7): the TP4 dense layers' FP8 q/k/v (minimax_m3_sm107tp4_dense_qk_norm_rope_fp8, head-count +generic, at the TP4 per-rank 16 query heads over 1 KV head) carries the bits of the stock path, the BF16 +fused_qk_norm_rope output cast to E4M3, for every norm / RoPE variant: Q and K are rounded to BF16 before E4M3 (K is +stored in the FP8 cache), V is cast.""" + +import pytest +import torch + +import tensorrt_llm # noqa: F401 (registers the trtllm ops) + +assert torch.cuda.is_available(), "the dense FP8 q/k/v op requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp4 target is certified on SM 10.7 only", allow_module_level=True) + +HQ, HK, HV, D, ROT = 16, 1, 1, 128, 64 + + +def _entry(): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_dense_qk_norm_rope_fp8 import ( # noqa: E501 + m3_dense_qk_norm_rope_fp8, + ) + + return m3_dense_qk_norm_rope_fp8 + + +def test_certified_cells() -> None: + """The entry (ops/attention/m3_dense_qk_norm_rope_fp8) at every cell: its output is its reference's bit for bit; + with ``kv_write`` the fused-insert form also returns the plain form's bytes and the compact Q.""" + op = _entry() + for cell in op.CELLS: + spec = cell.spec + m, hq, hk = spec["m"], spec["q"], spec["kv"] + g = torch.Generator(device="cuda").manual_seed(m * 7 + hq) + qkv = (torch.randn(m, (hq + 2 * hk) * D, device="cuda", generator=g) * 2).to(torch.bfloat16) + pos = torch.randint(0, 1 << 20, (m,), device="cuda", generator=g, dtype=torch.int32) + qw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + kw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + args = (qkv, hq, hk, hk, ROT, 1e-6, qw, kw, 5e6, True, pos, True) + ref = op.reference(*args) + if spec.get("kv_write"): + pages, tpb = 4, 128 + cache = torch.zeros(pages, 2, hk, tpb, D, device="cuda", dtype=torch.float8_e4m3fn) + slots = torch.randperm(pages * tpb, device="cuda", generator=g)[:m].to(torch.int32) + q_compact = torch.empty(m, hq, D, device="cuda", dtype=torch.float8_e4m3fn) + out = op(*args, cache, slots, m, q_compact) + assert torch.equal( + q_compact.view(m, hq * D).view(torch.uint8), out[:, : hq * D].view(torch.uint8) + ), cell.why + else: + out = op(*args) + assert torch.equal(out.view(torch.uint8), ref.view(torch.uint8)), cell.why + + +@pytest.mark.parametrize("tokens", [1, 4, 8, 4096]) +@pytest.mark.parametrize("use_gemma", [False, True], ids=["rmsnorm", "gemma"]) +@pytest.mark.parametrize("is_neox", [True, False], ids=["neox", "interleaved"]) +def test_the_dense_fp8_qkv_is_the_bf16_kernel_cast(tokens, use_gemma, is_neox) -> None: + if not hasattr(torch.ops.trtllm, "minimax_m3_sm107tp4_dense_qk_norm_rope_fp8"): + pytest.skip("this build does not carry minimax_m3_sm107tp4_dense_qk_norm_rope_fp8") + g = torch.Generator(device="cuda").manual_seed(tokens * 4 + int(use_gemma) * 2 + int(is_neox)) + qkv = (torch.randn(tokens, (HQ + HK + HV) * D, device="cuda", generator=g) * 2).to( + torch.bfloat16 + ) + pos = torch.randint(0, 1 << 20, (tokens,), device="cuda", generator=g, dtype=torch.int32) + qw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + kw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + ours = torch.ops.trtllm.minimax_m3_sm107tp4_dense_qk_norm_rope_fp8( + qkv, HQ, HK, HV, ROT, 1e-6, qw, kw, 5e6, is_neox, pos, use_gemma + ) + stock = qkv.clone() + torch.ops.trtllm.fused_qk_norm_rope( + stock, HQ, HK, HV, D, ROT, 1e-6, qw, kw, 5e6, is_neox, pos, + 1.0, 0.0, 0.0, 1.0, True, use_gemma, False, 0, 0, + ) # fmt: skip + assert torch.equal(ours.view(torch.uint8), stock.to(torch.float8_e4m3fn).view(torch.uint8)) + + +@pytest.mark.parametrize("tokens,live", [(1, 1), (4, 4), (8, 5), (64, 64)]) +@pytest.mark.parametrize("tokens_per_block", [32, 128]) +@pytest.mark.parametrize("is_neox", [True, False], ids=["neox", "interleaved"]) +def test_the_fused_kv_insert_stores_the_bytes_of_write_kv_slots_and_the_compact_q( + tokens, live, tokens_per_block, is_neox +) -> None: + if not hasattr(torch.ops.trtllm, "minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv"): + pytest.skip("this build does not carry minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv") + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + write_kv_slots, + ) + + g = torch.Generator(device="cuda").manual_seed(tokens * 1000 + tokens_per_block + int(is_neox)) + qkv = (torch.randn(tokens, (HQ + HK + HV) * D, device="cuda", generator=g) * 2).to( + torch.bfloat16 + ) + pos = torch.randint(0, 1 << 20, (tokens,), device="cuda", generator=g, dtype=torch.int32) + qw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + kw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + pages, layers, layer = 16, 3, 1 + # The layer's HND view of a multi-layer pool, strided as get_buffers returns it. + raw = torch.randint( + 0, + 256, + (pages, layers, 2, HK * tokens_per_block * D), + device="cuda", + generator=g, + dtype=torch.uint8, + ) + slots = torch.randperm(pages * tokens_per_block, device="cuda", generator=g)[:tokens].to( + torch.int32 + ) + slots[live:] = -1 # padded rows own no slot + + def view(t): + return t.view(torch.float8_e4m3fn)[:, layer].view(pages, 2, HK, tokens_per_block, D) + + ours_raw = raw.clone() + args = (qkv, HQ, HK, HV, ROT, 1e-6, qw, kw, 5e6, is_neox, pos, False) + q_compact = torch.empty(tokens, HQ, D, device="cuda", dtype=torch.float8_e4m3fn) + ours = torch.ops.trtllm.minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv( + *args, view(ours_raw), slots, live, q_compact + ) + plain = torch.ops.trtllm.minimax_m3_sm107tp4_dense_qk_norm_rope_fp8(*args) + assert torch.equal(ours.view(torch.uint8), plain.view(torch.uint8)) + assert torch.equal( + q_compact.view(tokens, HQ * D).view(torch.uint8), plain[:, : HQ * D].view(torch.uint8) + ) + + ref_raw = raw.clone() + k = plain[:, HQ * D : (HQ + HK) * D].reshape(tokens, HK, D) + v = plain[:, (HQ + HK) * D :].reshape(tokens, HV, D) + ref = view(ref_raw) + write_kv_slots(ref[:, 0], slots, k, live, layout="HND") + write_kv_slots(ref[:, 1], slots, v, live, layout="HND") + assert torch.equal(ours_raw, ref_raw) + + +def test_the_fused_kv_insert_skips_a_negative_slot() -> None: + """A live row whose slot is the -1 sentinel stores nothing (no write before the pool row).""" + if not hasattr(torch.ops.trtllm, "minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv"): + pytest.skip("this build does not carry minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv") + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + write_kv_slots, + ) + + tokens, tokens_per_block, pages = 4, 32, 8 + g = torch.Generator(device="cuda").manual_seed(7) + qkv = (torch.randn(tokens, (HQ + HK + HV) * D, device="cuda", generator=g) * 2).to( + torch.bfloat16 + ) + pos = torch.randint(0, 1 << 20, (tokens,), device="cuda", generator=g, dtype=torch.int32) + qw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + kw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + raw = torch.randint( + 0, + 256, + (pages, 3, 2, HK * tokens_per_block * D), + device="cuda", + generator=g, + dtype=torch.uint8, + ) + slots = torch.tensor([5, -1, 40, 77], device="cuda", dtype=torch.int32) + + def view(t): + return t.view(torch.float8_e4m3fn)[:, 1].view(pages, 2, HK, tokens_per_block, D) + + ours_raw = raw.clone() + args = (qkv, HQ, HK, HV, ROT, 1e-6, qw, kw, 5e6, True, pos, False) + torch.ops.trtllm.minimax_m3_sm107tp4_dense_qk_norm_rope_fp8_kv( + *args, view(ours_raw), slots, tokens + ) + plain = torch.ops.trtllm.minimax_m3_sm107tp4_dense_qk_norm_rope_fp8(*args) + keep = torch.tensor([0, 2, 3], device="cuda") + ref_raw = raw.clone() + ref = view(ref_raw) + k = plain[:, HQ * D : (HQ + HK) * D].reshape(tokens, HK, D)[keep] + v = plain[:, (HQ + HK) * D :].reshape(tokens, HV, D)[keep] + write_kv_slots(ref[:, 0], slots[keep], k, 3, layout="HND") + write_kv_slots(ref[:, 1], slots[keep], v, 3, layout="HND") + assert torch.equal(ours_raw, ref_raw) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_draft_attention.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_draft_attention.py new file mode 100644 index 000000000000..1cd948182a98 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_draft_attention.py @@ -0,0 +1,138 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_draft_attention entry: the Eagle3 drafter layer's attention against its float64 reference. + +Every cell runs on the runtime's own objects (``_m3_attn_runtime_tp4``): the target's hybrid NVFP4 +``MiniMaxM3KVCacheManagerV2`` with its one-model Eagle3 draft layer, the requests allocated in it, drafter +``TrtllmAttentionMetadata`` over the manager's FP8 draft view (the manager the runtime's draft swap installs) filled +by its ``prepare()``, and the drafter's ``TrtllmAttention`` built as ``M3Eagle3Draft`` builds it (fused NeoX RoPE at +the drafter checkpoint's theta, FP8 KV). Earlier rows go into the draft layer's cache through the in-tree scatter at +the slots the manager assigned; the call rotates and appends the step's own. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_draft_attention import ( # noqa: E501 + m3_draft_attention as op, +) + +__extra_import_path__ = ["../../../..", "."] +import _m3_attn_runtime_tp4 as rt # noqa: E402 — needs the path declared above +from _validating import ULP, assert_within_ulp, certifying # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_draft_attention requires a CUDA device" + +# Band, in BF16 ulps (2^-8) of each token's [heads x 128] row's largest magnitude (element) and of the relative RMS. +# The reference reads the same FP8 cache rows the call attends, so what is left is the FP8 FMHA's own arithmetic: +# it takes the rotated Q in E4M3 (rounding the reference's Q to E4M3 takes the RMS gap from ~9 to ~7 ulps) and rounds +# the softmax probabilities before the PV product. Measured on GB300 over these cells (max element / RMS): 13.6 / 8.6 +# (4 x 4 rows), 9.9 / 7.2 (8 x 1), 20.4 / 7.7 (512 context rows). The band leaves about 1.5x; a reference shifted +# by one row is far outside it. +_ELEMENT_ULP = 30.0 +_RMS_ULP = 12.0 + +# Cached tokens per request of each cell; the new rows per request come from the cell. +_CACHED = { + (4, 4): (200, 129, 1000, 64), + (8, 1): (5, 127, 128, 300, 1000, 2047, 64, 900), + (1, None): (0,), +} +#: Rows past the live ones in the fused QKV, as a padded step hands them over. +_PAD = 3 + + +def _ulp_stats(out: torch.Tensor, ref: torch.Tensor) -> tuple[float, float]: + o, r = out.float(), ref.float() + scale = r.abs().amax(dim=1, keepdim=True).clamp_min(1e-9) + element = ((o - r).abs() / scale).max().item() / ULP[torch.bfloat16] + rms = ((o - r).pow(2).mean().sqrt() / r.pow(2).mean().sqrt()).item() / ULP[torch.bfloat16] + return element, rms + + +def _rope(x: torch.Tensor, positions: torch.Tensor) -> torch.Tensor: + """NeoX RoPE over the whole head at the drafter's theta, float64.""" + half = rt.HEAD_DIM // 2 + inv_freq = rt.DRAFT_ROPE_THETA ** ( + -torch.arange(0, rt.HEAD_DIM, 2, dtype=torch.float64, device=x.device) / rt.HEAD_DIM + ) + angles = positions.to(torch.float64)[:, None] * inv_freq[None, :] + x = x.to(torch.float64) + x1, x2 = x[..., :half], x[..., half:] + return torch.cat( + [x1 * angles.cos() - x2 * angles.sin(), x2 * angles.cos() + x1 * angles.sin()], -1 + ) + + +# Non-paged context FMHA on SM107 does not apply RoPE (this fixture's path; the served drafter runs the paged context +# FMHA, which rotates): the context cell is a strict expected failure until the runtime rotates there too. +_UNROTATED_CONTEXT = pytest.mark.xfail( + strict=True, + reason="non-paged context FMHA on SM107 does not apply RoPE (shared runtime)", +) +_CELLS = [ + pytest.param(c, marks=_UNROTATED_CONTEXT) if c.spec.get("context") else c for c in op.CELLS +] + + +@pytest.mark.parametrize("cell", _CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + context = spec.get("context") + cached = _CACHED[spec["reqs"], spec.get("qlen")] + qlens = (context,) if context else (spec["qlen"],) * spec["reqs"] + assert len(cached) == spec["reqs"] + contexts = 1 if context else 0 + rids = list(range(spec["reqs"])) + gen = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + manager = rt.make_manager(draft=True) + try: + backend = rt.make_draft_backend() + metadata = rt.make_draft_metadata(manager) + rt.allocate(manager, rids, qlens, cached) + for rid, c in zip(rids, cached): + if c: + hist = torch.randn(2, c, rt.HEAD_DIM, generator=gen, device="cuda") + hist = hist.to(torch.float8_e4m3fn) + rt.write_fp8_rows(manager, rt.DRAFT, rid, torch.arange(c), hist[0], hist[1]) + rt.prepare(metadata, rids, qlens, cached, contexts) + total = sum(qlens) + assert int(metadata.num_tokens) == total + width = (rt.HEADS + 2 * rt.KV_HEADS) * rt.HEAD_DIM + qkv = torch.randn(total + _PAD, width, generator=gen, device="cuda").to(torch.bfloat16) + + certifying(op) + out = op(backend, qkv, metadata) + ref = op.reference(backend, qkv, metadata) + assert out.shape == (total, rt.HEADS * rt.HEAD_DIM) and out.dtype == torch.bfloat16 + assert torch.isfinite(out.float()).all() + element, rms = _ulp_stats(out, ref) + print(f"\n[{cell.why}] max element {element:.2f} ulp, rms {rms:.3f} ulp") + assert_within_ulp(out, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + # The call's append: each live row's rotated K and its V at the row's position, E4M3 (unit KV scale), to + # within one E4M3 step of the float64 rotation (the kernel rotates in FP32). + pool = manager.get_buffers(rt.DRAFT, "HND") + q_size = rt.HEADS * rt.HEAD_DIM + first = 0 + for rid, c, n in zip(rids, cached, qlens): + positions = torch.arange(c, c + n, device="cuda") + slots = rt.flat_slots(manager, rt.DRAFT, rid, positions).long() + rows = qkv[first : first + n] + want_k = _rope(rows[:, q_size : q_size + rt.HEAD_DIM].float(), positions) + want_v = rows[:, q_size + rt.HEAD_DIM :].to(torch.float8_e4m3fn) + got_k = pool[slots // rt.PAGE, 0, 0, slots % rt.PAGE].to(torch.float64) + got_v = pool[slots // rt.PAGE, 1, 0, slots % rt.PAGE] + step = want_k.abs().clamp_min(2.0**-6) * 2.0**-3 + assert bool(((got_k - want_k).abs() <= step).all()), "K is appended rotated, in E4M3" + assert torch.equal(got_v.view(torch.uint8), want_v.view(torch.uint8)) + first += n + + # The band discriminates: a reference whose rows are shifted by one token is outside it. + shifted = op.reference(backend, qkv.roll(1, 0), metadata) + with pytest.raises(AssertionError): + assert_within_ulp(out, shifted, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + finally: + manager.shutdown() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_drafter_fullctx.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_drafter_fullctx.py new file mode 100644 index 000000000000..d168c91a16c3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_drafter_fullctx.py @@ -0,0 +1,289 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7) for kernels/m3_drafter_attn: the Eagle3 drafter layer's generation pass on the full-context +PrimTS decode, driven through ``M3DrafterFullCtxFmha.run_generation`` (in-tree preprocess with the step's FP8 K/V +append, the K / V sub-page views of the drafter cache, the pinned plan) at every recipe the target pins: the 4-row +first draft pass and the 1-row later passes of one request. + +Lengths sweep the KV tile edges each query row in turn ends on (the rows of a Q CTA with no key in its trailing +splits); every call is held to the fp32 reference over the cache after the append (E4M3 query, unit KV scales, no +RoPE) within the FP8-KV band, finite, for both split-KV reducers. +""" + +import math +import weakref +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_drafter_attn as drafter, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_fullctx_decode as fullctx, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_tokmap_guard as tokmap_guard, +) + +assert torch.cuda.is_available(), "m3_drafter_attn requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp4 target is certified on SM 10.7 only", allow_module_level=True) + +H, KVH, D, PAGE = fullctx.HEADS, fullctx.KV_HEADS, fullctx.HEAD_DIM, 128 +MAX_LEN = 1 << 20 +FP8_KV_QUANT_MODE = 128 +FP8_TOL = 6e-2 +# Persistent like the FMHA manager's workspace and the spec-decoding metadata of a real step: the in-tree preprocess +# kernels must not see these move between calls. +_PERSISTENT = {} +# Row counts of the one-request token maps the guard wrote; the model installs it in prebuild_kernels. +_TOKMAP_TAKEN = [] + + +@pytest.fixture(scope="module", autouse=True) +def _tokmap_guard_installed(): + # A one-request spec pass places its rows by the workspace token map, which the op builds only for several + # requests; without the guard the call reads whatever the workspace holds. + tokmap_guard.install(torch.device("cuda"), dispatch.DECODE_MAX_TOKENS, _TOKMAP_TAKEN.append) + yield + + +def _persistent(name: str, make): + if name not in _PERSISTENT: + _PERSISTENT[name] = make() + return _PERSISTENT[name] + + +def _edge_lens(plan, qlen: int): + tile, insts, splits = plan.tile_kv, plan.insts_kv, int(plan.recipe["splits"]) + grow = splits * insts + tiles = sorted({1, 2, 8, 9, grow - 1, grow, grow + 1, grow + 2 * insts, 2 * grow}) + return [k * tile + d for k in tiles for d in range(-1, qlen)] + + +def _library(qlen: int, nreq: int, own_reducer: bool): + lib = drafter.M3DrafterFullCtxFmha.__new__(drafter.M3DrafterFullCtxFmha) + lib._recipes = {(qlen, nreq): dispatch.FULLCTX_DECODE_RECIPES[(qlen, nreq)]} + lib._own_reducer = own_reducer + lib._max_kv_len = MAX_LEN + lib._multi_processor_count = torch.cuda.get_device_properties(0).multi_processor_count + lib._on_taken = lambda rows: None + # The in-tree preprocess pass (its RoPE-free cache append); the one-launch preprocess has its own bitwise test. + lib._fused_rope_mode = None + lib._pdl = False + lib._on_rope_taken = None + return lib + + +def _attn(qlen: int): + return SimpleNamespace( + num_heads=H, + num_kv_heads=KVH, + head_dim=D, + q_scaling=1.0, + attention_chunk_size=None, + quant_mode=FP8_KV_QUANT_MODE, + local_layer_idx=0, + position_embedding_type=0, + rotary_inv_freq=None, + rotary_cos_sin=None, + predicted_tokens_per_seq=qlen, + rope_params=SimpleNamespace( + dim=0, theta=10000.0, scale_type=0, scale=1.0, max_positions=1024 + ), + ) + + +def _run(lib, qlen: int, kv_lens, seed: int, spec: bool = True): + """One drafter generation pass over a fresh FP8 cache; returns (output, fp32 reference). ``spec`` passes the + spec-decoding lengths/positions (a multi-token generation step); without them a multi-row pass is the drafter's + plain generation step of input_seq_length rows, as it runs in-model.""" + dev = torch.device("cuda") + nreq = len(kv_lens) + g = torch.Generator(device=dev).manual_seed(seed) + pages = math.ceil(max(kv_lens) / PAGE) + nblocks = nreq * pages + 2 + pool = torch.randn(nblocks, 2, KVH, PAGE, D, device=dev, generator=g).clamp_(-4, 4) + perm = torch.randperm(nblocks, device=dev, generator=g)[: nreq * pages].view(nreq, pages) + for r, n in enumerate( + kv_lens + ): # shift the tail third of V, so a missed trailing split fails the band + start = 2 * n // 3 + for pg in range(start // PAGE, math.ceil(n / PAGE)): + pool[int(perm[r, pg]), 1, :, (start % PAGE if pg == start // PAGE else 0) :] += 1.0 + pool = pool.to(torch.float8_e4m3fn) + offsets = torch.zeros(1, nreq, 2, pages, dtype=torch.int32, device=dev) + offsets[0, :, 0] = 2 * perm + offsets[0, :, 1] = 2 * perm + 1 + pointers = torch.tensor([[pool.data_ptr(), 0]], dtype=torch.int64) + ntok = nreq * qlen + qkv = torch.randn(ntok, (H + 2 * KVH) * D, dtype=torch.bfloat16, device=dev, generator=g) + seq = torch.tensor(kv_lens, dtype=torch.int32, device=dev) + one = torch.ones(1, dtype=torch.float32, device=dev) + multi = qlen > 1 and spec + out = torch.zeros(ntok, H * D, dtype=torch.bfloat16, device=dev) + params = SimpleNamespace( + qkv_input=qkv, + query_input=None, + attn=_attn(qlen), + meta=SimpleNamespace( + kv_cache_block_offsets=offsets, + host_kv_cache_pool_pointers=pointers, + host_kv_cache_pool_mapping=torch.zeros(1, 2, dtype=torch.int32), + ), + fwd=SimpleNamespace( + kv_scale_orig_quant=one, + kv_scale_quant_orig=one, + out_scale=None, + mrope_position_deltas=None, + ), + workspace=_persistent( + "workspace", lambda: torch.empty(512 << 20, dtype=torch.uint8, device=dev) + ), + sequence_lengths=seq, + spec_decoding_generation_lengths=( + _persistent( + f"gen_lengths{qlen}x{nreq}", + lambda: torch.full((nreq,), qlen, dtype=torch.int32, device=dev), + ) + if multi + else None + ), + spec_decoding_position_offsets=( + _persistent( + f"position_offsets{qlen}x{nreq}", + lambda: ( + torch.arange(qlen, dtype=torch.int32, device=dev).repeat(nreq, 1).contiguous() + ), + ) + if multi + else None + ), + seq_offset=0, + tokens_per_block=PAGE, + max_attention_window_size=MAX_LEN, + cyclic_attention_window_size=MAX_LEN, + num_tokens=ntok, + batch_size=nreq, + input_seq_length=qlen, + max_past_kv_length=max(kv_lens), + total_num_blocks=nblocks, + kv_factor=2, + is_cross=False, + output=out, + ) + taken = len(_TOKMAP_TAKEN) + lib.run_generation(params) + torch.cuda.synchronize() + if multi and nreq == 1: + assert _TOKMAP_TAKEN[taken:] == [ntok], ( + "the one-request spec pass ran without its token map" + ) + # fp32 reference over the cache after the append: E4M3 query, unit scales, causal over the step's rows. + q = qkv[:, : H * D].float().view(nreq, qlen, H, D).to(torch.float8_e4m3fn).float() + flat = pool.float() + ref = torch.empty(nreq, qlen, H, D, device=dev) + for r, n in enumerate(kv_lens): + blocks = perm[r, : math.ceil(n / PAGE)] + k = flat[blocks, 0].permute(1, 0, 2, 3).reshape(KVH, -1, D)[:, :n] + v = flat[blocks, 1].permute(1, 0, 2, 3).reshape(KVH, -1, D)[:, :n] + k = k.repeat_interleave(H // KVH, dim=0) + v = v.repeat_interleave(H // KVH, dim=0) + s = torch.einsum("qhd,hkd->hqk", q[r], k) * D**-0.5 + visible = ( + torch.arange(n, device=dev)[None, :] + < (n - qlen + 1 + torch.arange(qlen, device=dev))[:, None] + ) + s = s.masked_fill(~visible[None], float("-inf")) + ref[r] = torch.einsum("hqk,hkd->qhd", torch.softmax(s, dim=-1), v) + return out.float().view(nreq, qlen, H, D), ref + + +@pytest.mark.parametrize("qlen,nreq", sorted(dispatch.FULLCTX_DECODE_RECIPES)) +def test_the_drafter_pass_at_tile_edges(qlen, nreq) -> None: + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + recipe = dispatch.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + reducers = (True, False) if fullctx.has_own_reducer() else (False,) + for own in reducers: + fullctx.prebuild( + torch.device("cuda"), {(qlen, nreq): recipe}, MAX_LEN, own, get_env_enable_pdl() + ) + lens = _edge_lens(fullctx.plan_for(torch.device("cuda"), qlen, nreq, False), qlen) + lens = [n for n in lens if n > qlen] + batches = [[n] for n in lens] if nreq == 1 else [[a, b] for a, b in zip(lens, lens[::-1])] + batches.append([1300, 1281][:nreq]) + failures = [] + for i, kv_lens in enumerate(batches): + # The first draft pass runs in-model as a plain multi-row generation step; every other batch also takes the + # spec-decoding form. + spec = qlen > 1 and i % 2 == 1 + for own in reducers: + got, ref = _run(_library(qlen, nreq, own), qlen, kv_lens, sum(kv_lens), spec=spec) + if not bool(torch.isfinite(got).all()) or not torch.allclose( + got, ref, rtol=FP8_TOL, atol=FP8_TOL + ): + failures.append((kv_lens, "bf16-merge" if own else "primts-reducer")) + assert not failures, failures + + +def _drafter_model(scale: float): + """A drafter whose one attention module holds a TrtllmAttention backend as ``attn`` (the backend is no module).""" + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + backend = TrtllmAttention.__new__(TrtllmAttention) + backend.kv_cache_scaling_factor = torch.full((1,), scale, device="cuda") + backend.is_mla_enable, backend.kv_lora_rank, backend.v_head_dim, backend.head_dim = ( + False, + None, + None, + D, + ) + backend._fmha_manager = SimpleNamespace(fmha_libs=[]) + model = torch.nn.Module() + model.weight = torch.nn.Parameter(torch.zeros(1, device="cuda")) + model.self_attn = torch.nn.Module() + model.self_attn.attn = backend + return model, backend + + +def test_install_declines_a_scaled_kv_cache() -> None: + """At unit KV scales the arithmetic is the stock decode's; a scaled cache keeps the stock path.""" + reasons = [] + model, layer = _drafter_model(0.5) + assert drafter.install(model, {}, MAX_LEN, False, False, None, reasons.append, None) == 0 + assert layer._fmha_manager.fmha_libs == [] and reasons == ["a scaled FP8 KV cache"] + model, layer = _drafter_model(1.0) + assert drafter.install(model, {}, MAX_LEN, False, False, None, reasons.append, None) == 1 + assert isinstance(layer._fmha_manager.fmha_libs[0], drafter.M3DrafterFullCtxFmha) + + +def test_a_cache_bound_past_the_plan_capacity_keeps_the_stock_path() -> None: + """The plans are compiled for ``max_kv_len`` of K/V: a step whose cache length bound is longer falls through.""" + + from tensorrt_llm._torch.attention.backends.fmha.interface import FmhaPhase + + class _Backend: + num_heads, num_kv_heads, head_dim, quant_mode = H, KVH, D, FP8_KV_QUANT_MODE + kv_scale_orig_quant = kv_scale_quant_orig = None + + backend = _Backend() + backend.kv_scale_orig_quant, backend.kv_scale_quant_orig = object(), object() + reasons = [] + lib = _library(1, 1, False) + lib._on_uncovered = reasons.append + lib._attn_ref = weakref.ref(backend) + fwd = SimpleNamespace( + attention_input_type=None, + kv_scale_orig_quant=backend.kv_scale_orig_quant, + kv_scale_quant_orig=backend.kv_scale_quant_orig, + ) + meta = SimpleNamespace(num_contexts=0, num_generations=1, max_seq_len=MAX_LEN + 8) + q = torch.empty(1, H * D, device="cuda") + assert not lib._is_supported(q, None, None, meta, fwd, phase=FmhaPhase.GENERATION) + assert reasons == ["the cache length bound is beyond the plans' K/V capacity"] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_drafter_rope_fp8_kv.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_drafter_rope_fp8_kv.py new file mode 100644 index 000000000000..45a719b6c690 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_drafter_rope_fp8_kv.py @@ -0,0 +1,371 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7) for minimax_m3_sm107tp4_drafter_rope_fp8_kv, the TP4 drafter layer's one-launch generation +preprocess: bitwise against the in-tree ``thop.trtllm_gen_generation_preprocess`` followed by the torch E4M3 cast +of its BF16 query, on the query bytes and on every byte of the paged cache, with GPT-NeoX RoPE from a cos/sin table +(full and partial rotary width), multi-token (spec-decoding positions) and plain passes, one and two requests, and +query values past the E4M3 range. The library's fused pass is also held bitwise to its preprocess pass end to end. +""" + +import math +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_drafter_attn as drafter, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_fullctx_decode as fullctx, +) +from tensorrt_llm._torch.attention.backends.fmha.utils import ( + get_attention_chunk_size, + get_bmm1_scale, +) +from tensorrt_llm.bindings.internal import thop + +assert torch.cuda.is_available(), "the drafter preprocess op requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp4 target is certified on SM 10.7 only", allow_module_level=True) +if not hasattr(torch.ops.trtllm, "minimax_m3_sm107tp4_drafter_rope_fp8_kv"): + pytest.skip("minimax_m3_sm107tp4_drafter_rope_fp8_kv is not built", allow_module_level=True) + +H, KVH, D, PAGE = fullctx.HEADS, fullctx.KV_HEADS, fullctx.HEAD_DIM, 128 +MAX_LEN = 1 << 20 +FP8_KV_QUANT_MODE = 128 +ROPE_GPT_NEOX = 2 +_PERSISTENT = {} + + +def _persistent(name, make): + if name not in _PERSISTENT: + _PERSISTENT[name] = make() + return _PERSISTENT[name] + + +def _case(qlen, kv_lens, rotary_dim, multi, seed): + dev = torch.device("cuda") + nreq = len(kv_lens) + g = torch.Generator(device=dev).manual_seed(seed) + pages = math.ceil(max(kv_lens) / PAGE) + nblocks = nreq * pages + 2 + pool = (torch.randn(nblocks, 2, KVH, PAGE, D, device=dev, generator=g) * 2).to( + torch.float8_e4m3fn + ) + perm = torch.randperm(nblocks, device=dev, generator=g)[: nreq * pages].view(nreq, pages) + offsets = torch.zeros(1, nreq, 2, pages, dtype=torch.int32, device=dev) + offsets[0, :, 0] = 2 * perm + offsets[0, :, 1] = 2 * perm + 1 + ntok = nreq * qlen + qkv = torch.randn(ntok, (H + 2 * KVH) * D, device=dev, generator=g) * 4 + # A few query / key values near and past the E4M3 range (both casts saturate at 448). + hot = torch.rand(qkv.shape, device=dev, generator=g) < 0.01 + qkv = torch.where( + hot, qkv.sign() * (430 + 80 * torch.rand(qkv.shape, device=dev, generator=g)), qkv + ) + qkv = qkv.to(torch.bfloat16).contiguous() + max_pos = max(kv_lens) + 64 + cos_sin = torch.randn(max_pos, rotary_dim // 2, 2, device=dev, generator=g).float().contiguous() + attn = SimpleNamespace( + num_heads=H, + num_kv_heads=KVH, + head_dim=D, + q_scaling=1.0, + attention_chunk_size=None, + quant_mode=FP8_KV_QUANT_MODE, + local_layer_idx=0, + position_embedding_type=ROPE_GPT_NEOX, + rotary_inv_freq=None, + rotary_cos_sin=cos_sin, + predicted_tokens_per_seq=qlen, + rope_params=SimpleNamespace( + dim=rotary_dim, theta=10000.0, scale_type=0, scale=1.0, max_positions=max_pos + ), + ) + one = torch.ones(1, dtype=torch.float32, device=dev) + spec_offsets = None + if multi: + spec_offsets = _persistent( + f"pos{qlen}x{nreq}", + lambda: torch.arange(qlen, dtype=torch.int32, device=dev).repeat(nreq, 1).contiguous(), + ) + # Spec positions are not the plain row index: shift them per request. + spec_offsets.copy_( + torch.arange(qlen, dtype=torch.int32, device=dev).repeat(nreq, 1) + + torch.arange(nreq, dtype=torch.int32, device=dev)[:, None] + ) + params = SimpleNamespace( + qkv_input=qkv, + query_input=None, + attn=attn, + meta=SimpleNamespace( + kv_cache_block_offsets=offsets, + host_kv_cache_pool_pointers=torch.tensor([[pool.data_ptr(), 0]], dtype=torch.int64), + host_kv_cache_pool_mapping=torch.zeros(1, 2, dtype=torch.int32), + ), + fwd=SimpleNamespace( + kv_scale_orig_quant=one, + kv_scale_quant_orig=one, + out_scale=None, + mrope_position_deltas=None, + ), + workspace=_persistent( + "workspace", lambda: torch.empty(512 << 20, dtype=torch.uint8, device=dev) + ), + sequence_lengths=torch.tensor(kv_lens, dtype=torch.int32, device=dev), + spec_decoding_generation_lengths=( + _persistent( + f"len{qlen}x{nreq}", + lambda: torch.full((nreq,), qlen, dtype=torch.int32, device=dev), + ) + if multi + else None + ), + spec_decoding_position_offsets=spec_offsets, + seq_offset=0, + tokens_per_block=PAGE, + max_attention_window_size=MAX_LEN, + cyclic_attention_window_size=MAX_LEN, + num_tokens=ntok, + batch_size=nreq, + input_seq_length=qlen, + max_past_kv_length=max(kv_lens), + total_num_blocks=nblocks, + kv_factor=2, + is_cross=False, + output=torch.zeros(ntok, H * D, dtype=torch.bfloat16, device=dev), + ) + return params, pool + + +def _preprocess(params, mpc): + """run_generation's in-tree preprocess call and the E4M3 cast of its query.""" + attn, meta, fwd, rope = params.attn, params.meta, params.fwd, params.attn.rope_params + q_processed = thop.trtllm_gen_generation_preprocess( + params.qkv_input, + params.workspace, + params.sequence_lengths, + params.spec_decoding_generation_lengths, + params.spec_decoding_position_offsets, + meta.kv_cache_block_offsets, + meta.host_kv_cache_pool_pointers, + meta.host_kv_cache_pool_mapping, + fwd.kv_scale_orig_quant, + fwd.kv_scale_quant_orig, + fwd.out_scale, + attn.rotary_inv_freq, + attn.rotary_cos_sin, + fwd.mrope_position_deltas, + attn.local_layer_idx, + params.seq_offset, + attn.num_heads, + attn.num_kv_heads, + attn.head_dim, + params.tokens_per_block, + attn.quant_mode, + params.max_attention_window_size, + params.cyclic_attention_window_size, + params.num_tokens, + params.batch_size, + params.input_seq_length, + params.max_past_kv_length, + rope.dim, + rope.theta, + int(rope.scale_type), + rope.scale, + rope.max_positions, + attn.position_embedding_type, + get_bmm1_scale(attn), + 1.0, + False, + attn.predicted_tokens_per_seq, + get_attention_chunk_size(attn), + mpc, + params.total_num_blocks, + params.kv_factor, + True, + params.is_cross, + )[0] + rows = params.num_tokens * attn.num_heads * attn.head_dim + return ( + q_processed.flatten()[:rows] + .to(torch.float8_e4m3fn) + .view(params.num_tokens, attn.num_heads, attn.head_dim) + ) + + +def _op(params, pool, mode): + attn = params.attn + nreq, qlen = params.batch_size, params.input_seq_length + kv_pages = pool.view(-1, KVH, PAGE, D) + tables = params.meta.kv_cache_block_offsets[0, :nreq] + q = torch.empty( + params.num_tokens, + attn.num_heads, + attn.head_dim, + dtype=torch.float8_e4m3fn, + device=pool.device, + ) + multi = params.spec_decoding_generation_lengths is not None and qlen > 1 + torch.ops.trtllm.minimax_m3_sm107tp4_drafter_rope_fp8_kv( + params.qkv_input, + attn.rotary_cos_sin, + params.sequence_lengths, + params.spec_decoding_position_offsets if multi else None, + kv_pages, + tables, + q, + qlen, + attn.num_heads, + attn.num_kv_heads, + attn.rope_params.dim, + PAGE, + mode, + True, + ) + return q + + +# One-request multi-token passes keep the in-tree preprocess (it reads an unbuilt token map), so the spec cases hold +# two requests. +CASES = [ + (qlen, kv_lens, rotary, multi) + for qlen, multi in ((1, False), (4, True), (4, False)) + for kv_lens in (((4, 130) if multi else (4,)) if qlen > 1 else (130,), (8191, 131073)) + for rotary in (128, 64) +] + + +def _bytes(t): + return t.contiguous().view(torch.uint8) + + +def _matches(case, seed): + """Per rope mode: (query bytes equal, cache bytes equal) against the in-tree preprocess + cast.""" + qlen, kv_lens, rotary, multi = case + kv_lens = [max(n, qlen) for n in kv_lens] + mpc = torch.cuda.get_device_properties(0).multi_processor_count + params, pool = _case(qlen, kv_lens, rotary, multi, seed) + start = pool.clone() + q_ref = _preprocess(params, mpc) + torch.cuda.synchronize() + pool_ref = pool.clone() + out = {} + for mode in (0, 1, 2): + pool.copy_(start) + q = _op(params, pool, mode) + torch.cuda.synchronize() + out[mode] = ( + bool(torch.equal(_bytes(q), _bytes(q_ref))), + bool(torch.equal(_bytes(pool), _bytes(pool_ref))), + ) + return out + + +@pytest.mark.parametrize( + "case", + CASES, + ids=lambda c: f"q{c[0]}-{'x'.join(map(str, c[1]))}-r{c[2]}-{'spec' if c[3] else 'plain'}", +) +def test_the_pinned_rope_mode_is_bitwise(case) -> None: + result = _matches(case, seed=7) + print(f"[rope modes] {case}: {result}", flush=True) + assert result[dispatch.DRAFTER_ROPE_MODE] == (True, True), result + + +def test_the_fused_pass_matches_the_preprocess_pass() -> None: + mpc = torch.cuda.get_device_properties(0).multi_processor_count + for qlen, nreq in sorted(dispatch.FULLCTX_DECODE_RECIPES): + for multi in (False, True) if qlen > 1 else (False,): + kv_lens = [70001 + 517 * r for r in range(nreq)] + outs = [] + for fused in (False, True): + params, pool = _case(qlen, kv_lens, 128, multi, seed=11) + lib = drafter.M3DrafterFullCtxFmha.__new__(drafter.M3DrafterFullCtxFmha) + lib._recipes = {(qlen, nreq): dispatch.FULLCTX_DECODE_RECIPES[(qlen, nreq)]} + lib._own_reducer = True + lib._max_kv_len = MAX_LEN + lib._multi_processor_count = mpc + lib._fused_rope_mode = dispatch.DRAFTER_ROPE_MODE if fused else None + lib._pdl = True + taken = [] + lib._on_rope_taken = lambda rows, what: taken.append(what) + lib._on_taken = lambda rows: None + fullctx.prebuild(torch.device("cuda"), lib._recipes, MAX_LEN, True, True) + lib.run_generation(params) + torch.cuda.synchronize() + assert bool(taken) == (fused and not (multi and nreq == 1)) + outs.append((params.output.clone(), pool.clone())) + assert torch.equal(_bytes(outs[0][1]), _bytes(outs[1][1])), (qlen, nreq, multi, "cache") + assert torch.equal(_bytes(outs[0][0]), _bytes(outs[1][0])), ( + qlen, + nreq, + multi, + "output", + ) + + +def _round_bf16(x): + u = x.view(torch.int32).to(torch.int64) & 0xFFFFFFFF + u = (u + 0x7FFF + ((u >> 16) & 1)) & 0xFFFF0000 + return u.to(torch.int64).to(torch.int32).view(torch.float32) + + +def _discriminating(n, seed): + """(cos, sin, x, pair) quadruples whose rotated first-half output's E4M3 byte differs between the rope modes + (fp32 CPU emulation; the GPU comparison is the verdict).""" + g = torch.Generator().manual_seed(seed) + keep = [] + while sum(k.shape[0] for k in keep) < n: + m = 1 << 20 + c, s = torch.randn(m, generator=g), torch.randn(m, generator=g) + x = (torch.randn(m, generator=g) * 4).to(torch.bfloat16).float() + xp = (torch.randn(m, generator=g) * 4).to(torch.bfloat16).float() + sn = -s + r0 = (c * x) + (sn * xp) + r1 = ((c.double() * x.double()) + (sn * xp).double()).float() + r2 = ((sn.double() * xp.double()) + (c * x).double()).float() + b = [ + _round_bf16(r.contiguous()).to(torch.float8_e4m3fn).view(torch.uint8) + for r in (r0, r1, r2) + ] + diff = (b[0] != b[1]) | (b[0] != b[2]) | (b[1] != b[2]) + diff &= (r0.abs() < 400) & (r1.abs() < 400) + keep.append(torch.stack([c, s, x, xp], 1)[diff]) + return torch.cat(keep)[:n] + + +def test_crafted_inputs_pin_the_rope_mode() -> None: + """Rotations whose E4M3 byte depends on how the two products are summed: the in-tree preprocess and exactly the + pinned mode agree on every one.""" + mpc = torch.cuda.get_device_properties(0).multi_processor_count + params, pool = _case(1, [130], 128, False, seed=3) + half = 64 + quads = _discriminating(H * half, seed=5).cuda() + pos = 129 # the one row's rope position: cache length 130, one query row + table = params.attn.rotary_cos_sin.view(-1, half, 2) + # One (cos, sin) per dim: every head shares the position's table row, so the heads take the same dims' quads with + # their own (x, pair). + table[pos, :, 0] = quads[:half, 0] + table[pos, :, 1] = quads[:half, 1] + qkv = params.qkv_input.view(-1, H + 2 * KVH, D) + for h in range(H): + qkv[0, h, :half] = quads[:half, 2].to(torch.bfloat16) + qkv[0, h, half:] = quads[:half, 3].to(torch.bfloat16) + start = pool.clone() + q_ref = _preprocess(params, mpc) + torch.cuda.synchronize() + agree = {} + for mode in (0, 1, 2): + pool.copy_(start) + q = _op(params, pool, mode) + torch.cuda.synchronize() + agree[mode] = int((_bytes(q)[0, :, :half] == _bytes(q_ref)[0, :, :half]).sum()) + print( + f"[rope modes crafted] agreeing first-half query bytes of {H * half}: {agree}", flush=True + ) + assert agree[dispatch.DRAFTER_ROPE_MODE] == H * half, agree diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py new file mode 100644 index 000000000000..f0c79ec020e0 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py @@ -0,0 +1,49 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_fp8_indexer_qk_norm_rope entry of the MiniMax-M3 tp4 target: index-Q out and index-K in the +cache against the in-place fused norm + RoPE followed by the E4M3 cast, byte for byte up to the 0.1% budget the +in-tree test of the kernel allows.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_fp8_indexer_qk_norm_rope import ( # noqa: E501 + m3_fp8_indexer_qk_norm_rope as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_fp8_indexer_qk_norm_rope requires a CUDA device" + +DIM, ROTARY, EPS, THETA, PAGE = 128, 64, 1e-6, 5e6, 128 + + +def _bytes_close(a: torch.Tensor, b: torch.Tensor) -> None: + assert a.shape == b.shape and a.dtype == b.dtype == torch.float8_e4m3fn + differ = int((a.view(torch.uint8) != b.view(torch.uint8)).sum()) + assert differ <= (a.numel() - 1) // 1000, f"{differ} of {a.numel()} E4M3 bytes differ" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + m, heads = cell.spec["m"], cell.spec["heads"] + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + qk = torch.randn(m, (heads + 1) * DIM, generator=g, device="cuda").to(torch.bfloat16) + qw = (torch.randn(DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + kw = (torch.randn(DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + pos = torch.randint(0, 200000, (m,), generator=g, device="cuda", dtype=torch.int32) + pages = (m + PAGE - 1) // PAGE + 1 + cache = torch.zeros(pages, 1, PAGE, DIM, dtype=torch.float8_e4m3fn, device="cuda") + slots = torch.randperm(pages * PAGE, generator=g, device="cuda")[:m].to(torch.int32) + before = qk.clone() + args = (qk, cache, slots, heads, DIM, ROTARY, EPS, qw, kw, THETA, pos) + with validating(op): + q = op(*args) + assert torch.equal(qk, before), "the BF16 input is left untouched" + q_ref, k_ref = op.reference(*args) + _bytes_close(q.reshape(m, heads, DIM), q_ref) + flat = cache.view(-1, DIM) + _bytes_close(flat[slots.long()], k_ref) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fp8_saturate.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fp8_saturate.py new file mode 100644 index 000000000000..2188ca0c7b7b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fp8_saturate.py @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""paged_cache.fp8_e4m3_saturate (FP8_SATURATE): the E4M3 cache writes and the transient FP8 query casts saturate to ++-448 like the CUDA writers (__NV_SATFINITE); every finite in-range value keeps the plain cast's bits, values already +in E4M3 pass through, and the kill switch is exactly the plain cast. Where torch's cast already saturates (this image) +the conversion is the cast alone, one kernel; otherwise it clamps first, the same bytes.""" + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import paged_cache + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + + +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float32]) +def test_fp8_e4m3_saturate_matches_satfinite(dtype, monkeypatch) -> None: + g = torch.Generator(device="cuda").manual_seed(0) + inside = (torch.randn(4096, generator=g, device="cuda") * 100).clamp(-448, 448).to(dtype) + plain = inside.to(torch.float8_e4m3fn) + assert torch.equal( + paged_cache.fp8_e4m3_saturate(inside).view(torch.uint8), plain.view(torch.uint8) + ) + edge = torch.tensor( + [449.0, 463.0, 464.0, 1e4, float("inf"), -500.0, -float("inf")], dtype=dtype, device="cuda" + ) + want = torch.tensor([448.0, 448.0, 448.0, 448.0, 448.0, -448.0, -448.0], device="cuda") + assert torch.equal(paged_cache.fp8_e4m3_saturate(edge).float(), want) + nan = paged_cache.fp8_e4m3_saturate(torch.tensor([float("nan")], dtype=dtype, device="cuda")) + assert torch.isnan(nan.float()).all() + already = edge.clamp(-448, 448).to(torch.float8_e4m3fn) + assert paged_cache.fp8_e4m3_saturate(already) is already + monkeypatch.setattr(paged_cache, "FP8_SATURATE", False) + killed = paged_cache.fp8_e4m3_saturate(edge) + assert torch.equal(killed.view(torch.uint8), edge.to(torch.float8_e4m3fn).view(torch.uint8)) + + +def test_cache_write_saturates(monkeypatch) -> None: + """The cache-dtype conversion every write_kv_slots branch applies once a model arms it (compared as bytes: torch + has no float8 index_copy_ kernel), and an E4M3 source passing through untouched; unarmed it is the plain cast.""" + values = torch.tensor([[1.0, 500.0, -1e5, float("inf")]], dtype=torch.bfloat16, device="cuda") + assert not paged_cache.SHARED_FP8_SATURATE + plain = paged_cache._to_cache_dtype(values, torch.float8_e4m3fn) + assert torch.equal(plain.view(torch.uint8), values.to(torch.float8_e4m3fn).view(torch.uint8)) + monkeypatch.setattr(paged_cache, "SHARED_FP8_SATURATE", True) + values = torch.tensor([[1.0, 500.0, -1e5, float("inf")]], dtype=torch.bfloat16, device="cuda") + out = paged_cache._to_cache_dtype(values, torch.float8_e4m3fn) + assert out.dtype == torch.float8_e4m3fn + want = torch.tensor([[1.0, 448.0, -448.0, 448.0]], device="cuda").to(torch.float8_e4m3fn) + assert torch.equal(out.view(torch.uint8), want.view(torch.uint8)) + assert paged_cache._to_cache_dtype(out, torch.float8_e4m3fn) is out + bf = paged_cache._to_cache_dtype(values, torch.bfloat16) + assert bf.dtype == torch.bfloat16 and torch.equal(bf, values) + + +_EDGES = [ + 448.0, + 449.0, + 463.0, + 464.0, + 1e4, + float("inf"), + -448.0, + -449.0, + -464.0, + -1e4, + -float("inf"), + float("nan"), + 0.0, +] + + +def _cuda_kernels(fn) -> int: + from torch.profiler import ProfilerActivity, profile + + torch.cuda.synchronize() + with profile(activities=[ProfilerActivity.CUDA]) as prof: + fn() + torch.cuda.synchronize() + return sum(1 for e in prof.events() if e.device_type.name == "CUDA") + + +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16, torch.float32]) +@pytest.mark.parametrize("cast_saturates", [True, False], ids=["cast_only", "clamp_fallback"]) +def test_fp8_e4m3_saturate_bytes_equal_clamp_then_cast(dtype, cast_saturates, monkeypatch) -> None: + """Both paths give the bytes of clamp(+-448) followed by the cast, edge values included.""" + monkeypatch.setattr(paged_cache, "_CAST_SATURATES", cast_saturates) + x = torch.tensor(_EDGES, dtype=dtype, device="cuda") + want = x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn) + assert torch.equal(paged_cache.fp8_e4m3_saturate(x).view(torch.uint8), want.view(torch.uint8)) + + +def test_fp8_e4m3_saturate_takes_the_cast_only_path_here(monkeypatch) -> None: + """This image's torch cast saturates, on the CPU (the import-time probe) and on CUDA, so the conversion is one + kernel; with the probe failing it falls back to clamp + cast, two kernels.""" + assert paged_cache._CAST_SATURATES + x = torch.tensor(_EDGES, dtype=torch.bfloat16, device="cuda") + assert torch.equal( + x.to(torch.float8_e4m3fn).view(torch.uint8), + x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn).view(torch.uint8), + ) + assert _cuda_kernels(lambda: paged_cache.fp8_e4m3_saturate(x)) == 1 + monkeypatch.setattr(paged_cache, "_CAST_SATURATES", False) + assert _cuda_kernels(lambda: paged_cache.fp8_e4m3_saturate(x)) == 2 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fullctx_decode.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fullctx_decode.py new file mode 100644 index 000000000000..bbc3196701ba --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_fullctx_decode.py @@ -0,0 +1,312 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7) for the attention/m3_fullctx_decode entry: the dense layers' full-context FP8 decode on PrimTS +split-KV decode at the recipe the target pins per (query rows per request, requests). + +Every cell runs against the fp32 reference (q rounded to E4M3 first, as the kernel runs it), eager and replayed from a +CUDA graph (the same bits), and twice (the same bits). The K/V pages are read through the strided views of a flat +sub-page pool, as the KV cache manager hands them out. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_fullctx_decode import ( # noqa: E501 + HEAD_DIM, + HEADS, + KV_HEADS, + MAX_KV_LEN, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_fullctx_decode import ( # noqa: E501 + m3_fullctx_decode as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_fullctx_decode requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp4 target is certified on SM 10.7 only", allow_module_level=True) + +PAGE = 128 +SM_SCALE = HEAD_DIM**-0.5 +# The FP8-KV band of the in-tree dense decode test (test_minimax_m3_dense_decode.test_matches_reference). +FP8_TOL = 6e-2 + + +class _Pool: + """The one method the entry asks of a KV cache manager.""" + + def __init__(self, pool: torch.Tensor, subpages: int): + self.pool = pool + self.subpages = subpages + + def get_kv_subpage_pool(self, layer_idx: int, kv_layout: str = "HND"): + assert kv_layout == "HND" + return self.pool, self.subpages + + +def _inputs(kv_lens, qlen: int, seed: int, subpages: int): + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(kv_lens) + pages = max((n + PAGE - 1) // PAGE for n in kv_lens) + slots = batch * pages + 3 + pool = torch.randn( + (slots * subpages, KV_HEADS, PAGE, HEAD_DIM), device="cuda", generator=g + ).clamp(-4, 4) + perm = torch.randperm(slots, generator=g, device="cuda")[: batch * pages] + block_table = perm.reshape(batch, pages).to(torch.int32) + # The last third of every request's values is shifted by +1, so a split that missed the tail fails the band. + for b, n in enumerate(kv_lens): + start = 2 * n // 3 + for page in range(start // PAGE, (n + PAGE - 1) // PAGE): + v = int(block_table[b, page]) * subpages + 1 + pool[v, :, (start % PAGE if page == start // PAGE else 0) :] += 1.0 + # q as the layer hands it over on decode steps: the FP8 query columns of the fused norm/RoPE kernel's + # [q | k | v] output (row stride wider than q), so no cast makes it compact. + qkv = torch.randn( + (batch * qlen, (HEADS + 2 * KV_HEADS) * HEAD_DIM), device="cuda", generator=g + ).to(torch.float8_e4m3fn) + q = qkv[:, : HEADS * HEAD_DIM].view(batch * qlen, HEADS, HEAD_DIM) + seq_lens = torch.tensor(kv_lens, device="cuda", dtype=torch.int32) + return q, _Pool(pool.to(torch.float8_e4m3fn), subpages), block_table, seq_lens + + +@pytest.mark.parametrize("subpages", [2, 3], ids=["kv", "kv+index"]) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, subpages, certifying) -> None: # noqa: F811 + spec = cell.spec + qlen, nreq = spec["qlen"], len(spec["kv_lens"]) + recipe = dispatch.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + q, manager, block_table, seq_lens = _inputs( + spec["kv_lens"], qlen, zlib.crc32(cell.why.encode()) % 2**31, subpages + ) + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + args = (q, manager, 0, block_table, seq_lens, SM_SCALE, out, qlen, recipe) + certifying(op) + ref = op.reference(*args) + eager = op(*args).clone() + torch.testing.assert_close(eager.float(), ref, rtol=FP8_TOL, atol=FP8_TOL) + again = op(*args).clone() + assert torch.equal(again, eager), "not deterministic" + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(*args) + out.zero_() + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out, eager), "the CUDA-graph replay differs from the eager call" + + +# Plans compiled at other K/V stage depths stay alive for the whole session: a compiled decode is cached per config +# and is only ever run with the buffers of a live plan. +_OTHER_DEPTH_PLANS = [] + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_the_kv_stage_depth_keeps_the_bits(cell) -> None: + """kv_stages only sets how many K/V tiles the load pipeline keeps in flight: the recipe's plan and plans compiled + at 12 and at 4 stages give the same bits.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_fullctx_decode as kernel, + ) + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + if not kernel.has_own_reducer(): + pytest.skip("this build does not carry minimax_m3_sm107tp4_fullctx_reduce") + spec = cell.spec + qlen, nreq = spec["qlen"], len(spec["kv_lens"]) + recipe = dispatch.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + q, manager, block_table, seq_lens = _inputs(spec["kv_lens"], qlen, 13, 2) + k_pages, v_pages = kernel.kv_page_views(manager.pool, manager.subpages) + kernel.prebuild(q.device, {(qlen, nreq): recipe}, MAX_KV_LEN, True, get_env_enable_pdl()) + plans = [kernel.plan_for(q.device, qlen, nreq, True)] + for stages in (12, 4): + config = {**recipe["config"], "kv_stages": stages} + plans.append( + kernel.FullCtxPlan( + q.device, + qlen, + nreq, + MAX_KV_LEN, + {**recipe, "config": config}, + True, + get_env_enable_pdl(), + ) + ) + _OTHER_DEPTH_PLANS.append(plans[-1]) + outs = [] + for plan in plans: + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + plan.run(q, k_pages, v_pages, block_table, seq_lens, SM_SCALE, out) + outs.append(out) + assert torch.equal(outs[0], outs[1]) and torch.equal(outs[0], outs[2]) + + +def test_the_cells_cover_the_dispatch_rows() -> None: + certified = {(c.spec["qlen"], len(c.spec["kv_lens"])) for c in op.CELLS} + assert set(dispatch.FULLCTX_DECODE.keys) <= certified + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_the_bf16_merge_is_the_primts_reducer_to_one_ulp(cell) -> None: + """The BF16 split-KV merge op against PrimTS's own FP16 reducer and the cast: the same partials, merged once + in fp32 and rounded once, so within one BF16 spacing of the FP16-then-BF16 result.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_fullctx_decode as kernel, + ) + + if not kernel.has_own_reducer(): + pytest.skip("this build does not carry minimax_m3_sm107tp4_fullctx_reduce") + spec = cell.spec + qlen, nreq = spec["qlen"], len(spec["kv_lens"]) + recipe = dispatch.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + q, manager, block_table, seq_lens = _inputs(spec["kv_lens"], qlen, 11, 2) + outs = [] + for own in (True, False): + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + op(q, manager, 0, block_table, seq_lens, SM_SCALE, out, qlen, recipe, own_reducer=own) + outs.append(out.float()) + ours, primts = outs + _, exp = torch.frexp(primts.abs()) + spacing = torch.where( + primts == 0, + torch.full_like(primts, 2.0**-133), + torch.ldexp(torch.ones_like(primts), exp - 8), + ) + assert bool(((ours - primts).abs() <= spacing).all()) + + +def _tile_edge_kv_lens(plan, qlen: int): + """KV lengths around every kind of boundary the split ranges have, so each query row of the Q CTA in turn ends + its keys on a KV tile edge (k * tile + d, d in -1 .. qlen - 1): the first tiles, the reducer's 8-group stride, the + lengths where a split's range grows (splits * insts tiles) and a split edge past it.""" + tile, insts, splits = plan.tile_kv, plan.insts_kv, int(plan.recipe["splits"]) + grow = splits * insts + tiles = sorted({1, 2, 8, 9, grow - 1, grow, grow + 1, grow + 2 * insts, 2 * grow}) + return [k * tile + d for k in tiles for d in range(-1, qlen)] + + +@pytest.mark.parametrize("qlen,nreq", sorted(dispatch.FULLCTX_DECODE_RECIPES)) +def test_rows_with_no_key_in_a_trailing_split(qlen, nreq) -> None: + """A row whose keys end before its Q CTA's causal union does has trailing splits holding none of its keys + (log2-LSE -inf); both reducers must leave them out. [1300, 1281] at 4 query rows is the batch a GSM8K decode + step first hit (NaN rows from exp2(-inf - -inf)).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_fullctx_decode as kernel, + ) + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + recipe = dispatch.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + reducers = (True, False) if kernel.has_own_reducer() else (False,) + kernel.prebuild( + torch.device("cuda"), {(qlen, nreq): recipe}, MAX_KV_LEN, False, get_env_enable_pdl() + ) + lens = _tile_edge_kv_lens(kernel.plan_for(torch.device("cuda"), qlen, nreq, False), qlen) + batches = [[n] for n in lens] if nreq == 1 else [[a, b] for a, b in zip(lens, lens[::-1])] + batches.append([1300, 1281][:nreq]) + failures = [] + for kv_lens in batches: + q, manager, block_table, seq_lens = _inputs(kv_lens, qlen, sum(kv_lens), 2) + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + args = (q, manager, 0, block_table, seq_lens, SM_SCALE, out, qlen, recipe) + ref = op.reference(*args) + for own in reducers: + out.zero_() + op(*args, own_reducer=own) + got = out.float() + if not bool(torch.isfinite(got).all()) or not torch.allclose( + got, ref, rtol=FP8_TOL, atol=FP8_TOL + ): + failures.append((kv_lens, "bf16-merge" if own else "primts-reducer")) + assert not failures, failures + + +@pytest.mark.parametrize("qlen,nreq", sorted(dispatch.FULLCTX_DECODE_RECIPES)) +def test_kv_lengths_past_the_policy_length(qlen, nreq) -> None: + """The plans are compiled for MAX_KV_LEN of K/V with PrimTS's policy sized at 1 << 20 (POLICY_KV_LEN): caches just + past the policy length (the AgentX bound is 1 << 20 plus the speculative rows) and well past it stay exact.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_fullctx_decode as kernel, + ) + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + recipe = dispatch.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + reducers = (True, False) if kernel.has_own_reducer() else (False,) + lengths = [(1 << 20) + 1, (1 << 20) + 8] + ([1_500_001] if nreq == 1 else []) + batches = [[n] * nreq if nreq == 1 else [n, (1 << 20) - 3] for n in lengths] + failures = [] + for kv_lens in batches: + q, manager, block_table, seq_lens = _inputs(kv_lens, qlen, 13, 2) + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + args = (q, manager, 0, block_table, seq_lens, SM_SCALE, out, qlen, recipe) + ref = op.reference(*args) + for own in reducers: + kernel.prebuild( + torch.device("cuda"), {(qlen, nreq): recipe}, MAX_KV_LEN, own, get_env_enable_pdl() + ) + out.zero_() + op(*args, own_reducer=own) + got = out.float() + if not bool(torch.isfinite(got).all()) or not torch.allclose( + got, ref, rtol=FP8_TOL, atol=FP8_TOL + ): + failures.append((kv_lens, "bf16-merge" if own else "primts-reducer")) + assert not failures, failures + + +def _swizzled_scale_offsets(rows: int, width: int) -> torch.Tensor: + """Offsets of rows [0, rows)'s UE8M0 scales in the 128x4-swizzled layout of mxfp8_quantize(x, True).""" + k_sf = width // 32 + k_tiles = (k_sf + 3) // 4 + r = torch.arange(rows, device="cuda").view(-1, 1) + k = torch.arange(k_sf, device="cuda").view(1, -1) + offsets = ( + (r // 128) * k_tiles * 512 + (k // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + k % 4 + ) + return offsets.reshape(-1) + + +@pytest.mark.parametrize("splits,rows,seq_len_q", [(108, 8, 1), (64, 8, 1), (96, 32, 4)]) +def test_the_merge_mxfp8_copy_is_stock_mxfp8_quantize(splits, rows, seq_len_q) -> None: + """The TP4 merge's MXFP8 copy (the dense o_proj hand-off) equals mxfp8_quantize(out, True) bit for bit: E4M3 rows + and the real rows' swizzled UE8M0 scales, with an all-zero head row (amax 0) and saturating outliers; the BF16 + output is the merge without the copy.""" + ops = torch.ops.trtllm + if not hasattr(ops, "minimax_m3_sm107tp4_fullctx_reduce") or "q8" not in str( + ops.minimax_m3_sm107tp4_fullctx_reduce.default._schema + ): + pytest.skip("needs the TP4 merge op with the MXFP8 copy") + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(splits * 7 + rows) + nreq, kvh = 2, 1 + tokens = nreq * seq_len_q + width = kvh * rows // seq_len_q * 128 + for dtype in (torch.float16, torch.bfloat16): + po = torch.randn(nreq, kvh, splits, rows, 128, device=dev, generator=g) * 3 + po[0, 0, :, 1] = 0.0 + po[1, 0, :, 0, 5] = 6.0e4 + po = po.to(dtype).contiguous() + ps = (torch.randn(nreq, kvh, splits, rows, device=dev, generator=g) * 4).contiguous() + seq = torch.tensor([165521, 120011], dtype=torch.int32, device=dev) + args = (po, ps, seq) + tail = (seq_len_q, seq_len_q, 128, 1, seq_len_q > 1, False) + plain = torch.zeros(tokens, width, dtype=torch.bfloat16, device=dev) + ops.minimax_m3_sm107tp4_fullctx_reduce(*args, plain, *tail) + out = torch.zeros(tokens, width, dtype=torch.bfloat16, device=dev) + q8 = torch.empty(128, width, dtype=torch.float8_e4m3fn, device=dev) + sf = torch.zeros(128 * (width // 32), dtype=torch.uint8, device=dev) + ops.minimax_m3_sm107tp4_fullctx_reduce(*args, out, *tail, q8, sf) + torch.cuda.synchronize() + assert torch.equal(out.view(torch.int16), plain.view(torch.int16)) + stock_q, stock_sf = ops.mxfp8_quantize(out, True) + assert torch.equal( + q8[:tokens].view(torch.uint8), stock_q.view(torch.uint8).reshape(tokens, -1) + ), dtype + at = _swizzled_scale_offsets(tokens, width) + assert torch.equal(sf[at], stock_sf.view(torch.uint8).reshape(-1)[at]), dtype diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_index_score.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_index_score.py new file mode 100644 index 000000000000..13845e1a5ede --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_index_score.py @@ -0,0 +1,133 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_index_score entry: the MSA index scorer with its pre-wait schedule. + +Every cell is held to the fp32 reference (a few fp32 ulps: the kernel sums the 128 exact products in its k-split order) +and to the in-tree scorer (``cute_dsl_minimax_m3_index_decode_score``) bit for bit, which the pre-wait prefetch and the +early tile issue must not change. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_index_score import ( # noqa: E501 + PAGE, + SPLIT_K, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_index_score import ( # noqa: E501 + m3_index_score as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_index_score requires a CUDA device" + +HEADS = 1 # index heads per rank at TP4 +HEAD_DIM = 128 + + +def _inputs(seq_lens, qlen: int, seed: int, dtype=torch.float8_e4m3fn): + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + max_blocks = (max(seq_lens) + PAGE - 1) // PAGE + stride = ((max_blocks + 15) // 16) * 16 + pages = batch * max_blocks + block_table = ( + torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(batch, max_blocks) + ) + idx_q = torch.randn(batch * qlen, HEADS, HEAD_DIM, device="cuda", generator=g).to(dtype) + index_k = torch.randn(pages, PAGE, HEAD_DIM, device="cuda", generator=g).to(dtype) + seq = torch.tensor(seq_lens, device="cuda", dtype=torch.int32) + return idx_q, index_k, block_table, seq, stride + + +def _score(stride: int, total_q: int): + backing = torch.full( + (HEADS, stride, total_q), -float("inf"), device="cuda", dtype=torch.float32 + ) + return backing.transpose(1, 2) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + qlen = spec["qlen"] + idx_q, index_k, block_table, seq, stride = _inputs( + spec["seq_lens"], qlen, zlib.crc32(cell.why.encode()) + ) + score = _score(stride, idx_q.shape[0]) + args = (idx_q, index_k, block_table, seq, score, qlen, SPLIT_K, spec["early"]) + certifying(op) + ref = op.reference(*args) + op(*args) + stock = _score(stride, idx_q.shape[0]) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, qlen, SPLIT_K + ) + torch.cuda.synchronize() + torch.testing.assert_close(score, ref, rtol=1e-5, atol=1e-4) + assert torch.equal(score, stock), "the scores differ from the in-tree scorer" + + +# The 2-request decode step also runs the early-issue kernel on this target (dispatch.INDEX_SCORE_EARLY_MAX_REQUESTS). +# The op file is the sm_103 copy (test_modeling_v2_m3_tp4_drift), so these cells live here. +TWO_REQUEST_EARLY = ( + ((20000, 3000), "two requests verifying 1 + 3 rows each over 20000 and 3000 tokens"), + ( + (70000, 129), + "two requests: one past 512 blocks (its CTAs loop), one whose early tiles are drained", + ), +) + + +def test_the_two_request_step_takes_the_early_kernel() -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling as dispatch, + ) + + assert dispatch.INDEX_SCORE_EARLY_MAX_REQUESTS == 2 + + +@pytest.mark.parametrize( + "seq_lens,why", TWO_REQUEST_EARLY, ids=[w[:44] for _, w in TWO_REQUEST_EARLY] +) +def test_two_request_early_issue_is_the_in_tree_scorer(seq_lens, why) -> None: + idx_q, index_k, block_table, seq, stride = _inputs(seq_lens, 4, zlib.crc32(why.encode())) + score, stock = _score(stride, idx_q.shape[0]), _score(stride, idx_q.shape[0]) + args = (idx_q, index_k, block_table, seq, score, 4, SPLIT_K, True) + ref = op.reference(*args) + op(*args) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, 4, SPLIT_K + ) + torch.cuda.synchronize() + torch.testing.assert_close(score, ref, rtol=1e-5, atol=1e-4) + assert torch.equal(score, stock), "the scores differ from the in-tree scorer" + + +@pytest.mark.parametrize("early", [True, False], ids=["early", "prefetch"]) +def test_bf16_index_is_the_in_tree_scorer(early) -> None: + idx_q, index_k, block_table, seq, stride = _inputs((5000,), 4, 11, dtype=torch.bfloat16) + score, stock = _score(stride, 4), _score(stride, 4) + op(idx_q, index_k, block_table, seq, score, 4, SPLIT_K, early) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, 4, SPLIT_K + ) + assert torch.equal(score, stock) + + +def test_guard_refuses_a_request_past_its_block_table() -> None: + idx_q, index_k, block_table, seq, stride = _inputs((1000,), 1, 5) + seq = seq + 4 * PAGE + with pytest.raises(AssertionError, match="past its block-table row"): + with validating(op): + op(idx_q, index_k, block_table, seq, _score(stride + 16, 1), 1, SPLIT_K, True) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_index_score_t2sc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_index_score_t2sc.py new file mode 100644 index 000000000000..c19ead23a083 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_index_score_t2sc.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_index_score_t2sc entry (T2SC, dispatch.INDEX_SCORE_T2SC): the TP2 / TP8 MSA index +scorer with its pre-wait schedule, as the TP4 target's copy at its 1 index head per rank. + +Every cell is held to the fp32 reference (a few fp32 ulps: the kernel sums the 128 exact products in its k-split order) +and to the in-tree scorer (``cute_dsl_minimax_m3_index_decode_score`` at its own 256-CTA split) bit for bit, which the +pre-wait prefetch, the early tile issue and the cell's split must not change. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_index_score_t2sc import ( # noqa: E501 + PAGE, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_index_score_t2sc import ( # noqa: E501 + m3_index_score_t2sc as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_index_score requires a CUDA device" + +HEADS = 1 # index heads per rank at TP4 (4 index heads over 4 KV groups) +HEAD_DIM = 128 + + +def _inputs(seq_lens, qlen: int, seed: int, dtype=torch.float8_e4m3fn): + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + max_blocks = (max(seq_lens) + PAGE - 1) // PAGE + stride = ((max_blocks + 15) // 16) * 16 + pages = batch * max_blocks + block_table = ( + torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(batch, max_blocks) + ) + idx_q = torch.randn(batch * qlen, HEADS, HEAD_DIM, device="cuda", generator=g).to(dtype) + index_k = torch.randn(pages, PAGE, HEAD_DIM, device="cuda", generator=g).to(dtype) + seq = torch.tensor(seq_lens, device="cuda", dtype=torch.int32) + return idx_q, index_k, block_table, seq, stride + + +def _score(stride: int, total_q: int): + backing = torch.full( + (HEADS, stride, total_q), -float("inf"), device="cuda", dtype=torch.float32 + ) + return backing.transpose(1, 2) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + qlen = spec["qlen"] + idx_q, index_k, block_table, seq, stride = _inputs( + spec["seq_lens"], qlen, zlib.crc32(cell.why.encode()) + ) + score = _score(stride, idx_q.shape[0]) + assert spec["heads"] == HEADS + args = (idx_q, index_k, block_table, seq, score, qlen, spec["split_k"], spec["early"]) + certifying(op) + ref = op.reference(*args) + op(*args) + stock = _score(stride, idx_q.shape[0]) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, qlen + ) + torch.cuda.synchronize() + torch.testing.assert_close(score, ref, rtol=1e-5, atol=1e-4) + assert torch.equal(score, stock), "the scores differ from the in-tree scorer" + + +@pytest.mark.parametrize("early", [True, False], ids=["early", "prefetch"]) +def test_bf16_index_is_the_in_tree_scorer(early) -> None: + idx_q, index_k, block_table, seq, stride = _inputs((5000,), 4, 11, dtype=torch.bfloat16) + score, stock = _score(stride, 4), _score(stride, 4) + op(idx_q, index_k, block_table, seq, score, 4, 1536, early) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, 4 + ) + assert torch.equal(score, stock) + + +@pytest.mark.parametrize("qlen", [1, 2, 3, 4]) +def test_prestaged_blocks_at_the_stable_edge_are_the_in_tree_scorer(qlen) -> None: + """Lengths around the edge between pre-staged (all-old) and post-wait first blocks, 1 and 2 requests.""" + for k in (1, 2, 1294): + for length in (k * PAGE - qlen - 1, k * PAGE - qlen, k * PAGE - qlen + 1, k * PAGE + 1): + for seq_lens, split in ( + ((length,), 1536), + ((length,), 1272), + ((length, max(qlen, length // 3)), 768), + ): + idx_q, index_k, block_table, seq, stride = _inputs(seq_lens, qlen, 31 + length) + score, stock = _score(stride, idx_q.shape[0]), _score(stride, idx_q.shape[0]) + op(idx_q, index_k, block_table, seq, score, 4, split, True) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, qlen + ) + assert torch.equal(score, stock), (qlen, seq_lens) + + +def test_guard_refuses_a_request_past_its_block_table() -> None: + idx_q, index_k, block_table, seq, stride = _inputs((1000,), 1, 5) + seq = seq + 4 * PAGE + with pytest.raises(AssertionError, match="past its block-table row"): + with validating(op): + op(idx_q, index_k, block_table, seq, _score(stride + 16, 1), 1, 1536, True) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_msa_attention.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_msa_attention.py new file mode 100644 index 000000000000..034bd7a8d322 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_msa_attention.py @@ -0,0 +1,214 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_msa_attention entry: one layer's MSA backend step against its float64 reference. + +Every cell runs on the runtime's own objects (``_m3_attn_runtime_tp4``): a hybrid NVFP4 ``MiniMaxM3KVCacheManagerV2`` +with the requests allocated in it, ``MiniMaxM3MsaSparseAttentionMetadata`` filled by its ``prepare()``, and the layer's +``MiniMaxM3MsaSparseAttention`` built as the target builds it. Earlier rows (and, on a sparse layer, the step's own +rows, which the target's producer writes before the call) go into the cache through the in-tree scatters, at the +slots the manager assigned. The sparse layers' index-K rows are planted so every block's score is 1/8 or more apart +from every other's: the top-k selection the reference recomputes is then unambiguous. + +The cells are the target's calls (``_attention_core`` in modeling.py): a dense layer with live bf16 K / V in a +prefill chunk and live FP8 K / V in a decode step (the dense layers sit in the hybrid cache's FP8 half), a sparse +layer whose producer already wrote the caches (``k = v = None``) in a prefill chunk beyond 16 blocks and in a mixed +step, both with the NVFP4 scales the target passes. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_msa_attention import ( # noqa: E501 + m3_msa_attention as op, +) + +__extra_import_path__ = ["../../../..", "."] +import _m3_attn_runtime_tp4 as rt # noqa: E402 — needs the path declared above +from _validating import ULP, assert_within_ulp, certifying # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_msa_attention requires a CUDA device" + +# Bands, in BF16 ulps (2^-8) of each token's [heads x 128] row's largest magnitude (element) and of the relative RMS, +# per kernel the step reaches. The reference reads the same quantized cache rows (and casts Q to E4M3 where the FP8 +# FMHA does), so what is left is each FMHA's own arithmetic, measured on GB300 over these cells (max element / RMS): +# * fmha_sm100 over the FP8 pool (dense context rows): 10.1 / 6.1. It runs E4M3 x E4M3 and its softmax +# probabilities are rounded before the PV product. +# * trtllm-gen over the FP8 pool (dense generation rows): 7.0 / 6.1. It rounds the probabilities to E4M3 (scaled by +# 448 against the running maximum): emulating that brings a single-tile request to 0.6 RMS. +# * the NVFP4 CSR prefill (sparse context rows): 34.4 / 11.8 at 4096 rows, 26.8 / 9.1 at 512. It stages the +# reconstructed K / V in E4M3 (the in-tree A/B test, test_minimax_m3_sparse_attn_decode, notes it and holds it +# to 3e-2). +# * the Triton NVFP4 sparse decode (sparse generation rows): 1.0 / 0.25 -- the band of its own entry +# (m3_sparse_decode): exact operands, FP16 probabilities, BF16 rounding. +# Each band leaves about 1.5x the measured maximum; a shifted or wrongly selected reference is far outside them. +_FP8_CONTEXT = dict(element_ulp=16.0, rms_ulp=9.0) +_FP8_DECODE = dict(element_ulp=12.0, rms_ulp=9.0) +_NVFP4_CONTEXT = dict(element_ulp=48.0, rms_ulp=16.0) +_NVFP4_DECODE = dict(element_ulp=2.0, rms_ulp=0.5) +_BANDS = { + ("dense", "context"): (_FP8_CONTEXT, None), + ("dense", "generation"): (None, _FP8_DECODE), + ("sparse", "context"): (_NVFP4_CONTEXT, None), + ("sparse", "mixed"): (_NVFP4_CONTEXT, _NVFP4_DECODE), +} + + +def _assert_in_bands(output, ref, bands, context_rows: int) -> None: + """Gate the context rows and the generation rows each on their own kernel's band.""" + context_band, decode_band = bands + if context_rows: + assert_within_ulp(output[:context_rows], ref[:context_rows], **context_band) + if context_rows < output.shape[0]: + assert_within_ulp(output[context_rows:], ref[context_rows:], **decode_band) + + +# (request rows, cached tokens per request, new tokens per request, context rows) per cell. +_SHAPES = { + ("dense", "context"): ((384,), (1024,), 1), + ("dense", "generation"): ((127, 128, 1000, 2500), (1, 1, 1, 1), 0), + ("sparse", "context"): ((0,), (4096,), 1), + ("sparse", "mixed"): ((0, 2999, 4500), (512, 1, 1), 1), +} + + +def _ulp_stats(out: torch.Tensor, ref: torch.Tensor) -> tuple[float, float]: + """The two quantities ``assert_within_ulp`` gates, in BF16 ulps.""" + o, r = out.float(), ref.float() + scale = r.abs().amax(dim=1, keepdim=True).clamp_min(1e-9) + element = ((o - r).abs() / scale).max().item() / ULP[torch.bfloat16] + rms = ((o - r).pow(2).mean().sqrt() / r.pow(2).mean().sqrt()).item() / ULP[torch.bfloat16] + return element, rms + + +def _shift(x: torch.Tensor) -> torch.Tensor: + """``x``'s rows moved down by one (byte-wise, so E4M3 rows move too).""" + return x.view(torch.uint8).roll(1, 0).view(x.dtype) if x.element_size() == 1 else x.roll(1, 0) + + +def _shape(cell): + spec = cell.spec + cached, qlens, contexts = _SHAPES[spec["layer"], spec["phase"]] + if spec["phase"] == "context": + assert sum(qlens) == spec["tokens"] + elif spec["phase"] == "generation": + assert len(qlens) == spec["reqs"] + else: + assert sum(qlens[:contexts]) == spec["tokens"] and len(qlens) - contexts == spec["reqs"] + return cached, qlens, contexts + + +def _run_dense(cell, gen, manager, backend, metadata): + cached, qlens, contexts = _shape(cell) + rids = list(range(len(qlens))) + rt.allocate(manager, rids, qlens, cached) + for rid, c in zip(rids, cached): + if c: + hist = torch.randn(2, c, rt.HEAD_DIM, generator=gen, device="cuda").to( + torch.float8_e4m3fn + ) + rt.write_fp8_rows(manager, rt.DENSE, rid, torch.arange(c), hist[0], hist[1]) + rt.prepare(metadata, rids, qlens, cached, contexts) + total = sum(qlens) + # A prefill chunk arrives in BF16 (the hybrid cache's dense layers take FP8 Q / K / V only on decode steps). + dtype = torch.bfloat16 if contexts else torch.float8_e4m3fn + q = torch.randn(total, rt.HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to(dtype) + k = torch.randn(total, rt.HEAD_DIM, generator=gen, device="cuda").to(dtype) + v = torch.randn(total, rt.HEAD_DIM, generator=gen, device="cuda").to(dtype) + # The NVFP4 target passes its layer scales on every layer; an FP8 layer's write and FMHA take none. + inv = torch.tensor([1.0, 0.5, 2.0], device="cuda") + fwd = 1.0 / inv + return (q, k, v, None, inv, fwd), dict(rids=rids, cached=cached, qlens=qlens, contexts=contexts) + + +def _run_sparse(cell, gen, manager, backend, metadata): + cached, qlens, contexts = _shape(cell) + rids = list(range(len(qlens))) + rt.allocate(manager, rids, qlens, cached) + u0, u1 = rt.index_directions(gen) + lengths = [c + n for c, n in zip(cached, qlens)] + keys = [torch.randn(n, rt.HEAD_DIM, generator=gen, device="cuda") for n in lengths] + values = [torch.randn(n, rt.HEAD_DIM, generator=gen, device="cuda") for n in lengths] + quant = rt.nvfp4_quant(torch.cat(keys), torch.cat(values)) + for rid, n, kk, vv in zip(rids, lengths, keys, values): + # Every row, the step's own included: the producer writes a sparse layer's caches before the call. + idx_k = rt.planted_index_keys(n, u0, u1, gen) + rt.write_nvfp4_rows( + manager, + metadata, + rt.SPARSE, + rid, + torch.arange(n), + kk.to(torch.bfloat16), + vv.to(torch.bfloat16), + idx_k, + quant, + ) + rt.prepare(metadata, rids, qlens, cached, contexts) + total = sum(qlens) + q = torch.randn(total, rt.HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to( + torch.float8_e4m3fn + ) + # Query tokens alternate between the two planted directions, so neighbouring tokens select different blocks. + idx_q = torch.stack([u0, u1])[torch.arange(total, device="cuda") % 2].to(torch.float8_e4m3fn) + return (q, None, None, idx_q, quant, 1.0 / quant), dict( + rids=rids, cached=cached, qlens=qlens, contexts=contexts + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + sparse = spec["layer"] == "sparse" + gen = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + manager = rt.make_manager(draft=False) + try: + backend = rt.make_msa_backend(rt.SPARSE if sparse else rt.DENSE) + metadata = rt.make_msa_metadata(manager) + run = _run_sparse if sparse else _run_dense + (q, k, v, idx_q, inv, fwd), step = run(cell, gen, manager, backend, metadata) + output = torch.full( + (q.shape[0], rt.HEADS * rt.HEAD_DIM), float("nan"), dtype=torch.bfloat16, device="cuda" + ) + certifying(op) + result = op(backend, q, k, v, idx_q, metadata, output, sparse, inv, fwd) + assert result is output + ref = op.reference(backend, q, k, v, idx_q, metadata, output, sparse, inv, fwd) + assert torch.isfinite(output.float()).all(), "every live row is written" + bands = _BANDS[spec["layer"], spec["phase"]] + context_rows = sum(step["qlens"][: step["contexts"]]) + for name, rows in ( + ("context", slice(0, context_rows)), + ("generation", slice(context_rows, None)), + ): + if output[rows].numel(): + element, rms = _ulp_stats(output[rows], ref[rows]) + print( + f"\n[{cell.why}] {name} rows: max element {element:.2f} ulp, rms {rms:.3f} ulp" + ) + _assert_in_bands(output, ref, bands, context_rows) + + if k is not None: + # The call's own cache write: the live K / V rows land, as a plain E4M3 cast, at their positions. + pool = manager.get_buffers(rt.DENSE, "HND") + first = 0 + for rid, c, n in zip(step["rids"], step["cached"], step["qlens"]): + slots = rt.flat_slots(manager, rt.DENSE, rid, torch.arange(c, c + n)).long() + for role, src in ((0, k), (1, v)): + got = pool[slots // rt.PAGE, role, 0, slots % rt.PAGE] + want = src[first : first + n].to(torch.float8_e4m3fn) + assert torch.equal(got.view(torch.uint8), want.view(torch.uint8)) + first += n + + # The band discriminates: a reference whose query rows are shifted by one token is outside it. + shifted = op.reference(backend, _shift(q), k, v, idx_q, metadata, output, sparse, inv, fwd) + with pytest.raises(AssertionError): + _assert_in_bands(output, shifted, bands, context_rows) + if sparse: + # And the selection is load-bearing: attending every causal block instead is outside it too. + dense = op.reference(backend, q, k, v, idx_q, metadata, output, False, inv, fwd) + with pytest.raises(AssertionError): + _assert_in_bands(output, dense, bands, context_rows) + finally: + manager.shutdown() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_n_valid_contract.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_n_valid_contract.py new file mode 100644 index 000000000000..3c5b75f264ed --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_n_valid_contract.py @@ -0,0 +1,138 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The per-token valid-block count is written only before the step's first layer. + +The long-row select (``kernels/m3_select_long_dsl.py``, ``N_VALID_PREWAIT``) reads its row's entry of +``MsaMetadata.msa_n_valid_blocks`` BEFORE its PDL grid-dependency wait. That is exact only while every write to the +buffer happens before the step's first layer: + +- the allocation (``_alloc_msa_proxy_scratch``); +- the host-to-device copy in ``prepare()`` (``_build_step_plans``); +- ``on_update_kv_lens``, which the runner calls before the forward. It writes through the plain + ``<<<>>>`` launch of ``msa_kv_update_kernel`` or through torch copies, so any PDL grid of the step starts only + after it completes. + +This test scans ``tensorrt_llm/_torch`` and fails on any other write site (including writes through a local alias of +the buffer), so a new writer has to revisit the pre-wait read. A buffer handed to an opaque kernel launch is not +tracked; such a launch inside a layer must be reviewed by hand. + +CPU only: it parses the sources. +""" + +import ast +from pathlib import Path + +import pytest + +import tensorrt_llm + +pytestmark = pytest.mark.cpu_only + +_ROOT = Path(tensorrt_llm.__file__).resolve().parent / "_torch" +_NAME = "msa_n_valid_blocks" +# In-place tensor methods that write their receiver. +_MUTATORS = { + "copy_", + "fill_", + "zero_", + "index_put_", + "index_copy_", + "scatter_", + "masked_fill_", + "add_", + "sub_", +} +# Calls whose first argument is the destination they write: ``step_inputs.copy_(dst, src)`` and the H2D helpers. +_COPY_HELPERS = {"copy_", "copy_h2d_if_changed", "copy_h2d_pool_shared"} +_BACKEND = "attention/backends/sparse/minimax_m3/msa_backend.py" +# (file relative to tensorrt_llm/_torch, enclosing function) -> writes allowed there +_ALLOWED = { + (_BACKEND, "_alloc_msa_proxy_scratch"), # the allocation + (_BACKEND, "_build_step_plans"), # the H2D copy in prepare() + (_BACKEND, "on_update_kv_lens"), # the device-side correction before the forward +} + + +def _base(node): + """The innermost receiver of an attribute / subscript / call chain.""" + while isinstance(node, (ast.Subscript, ast.Attribute, ast.Call)): + if isinstance(node, ast.Attribute) and node.attr == _NAME: + return node + node = node.func if isinstance(node, ast.Call) else node.value + return node + + +def _mentions(node, aliases) -> bool: + """Whether the expression's receiver chain is ``*.msa_n_valid_blocks`` or a local alias of it.""" + b = _base(node) + if isinstance(b, ast.Attribute): + return b.attr == _NAME + return isinstance(b, ast.Name) and (b.id == _NAME or b.id in aliases) + + +def _refers(node, aliases) -> bool: + """Whether any sub-expression (e.g. a branch of a conditional) names the buffer or an alias.""" + return any( + (isinstance(n, ast.Attribute) and n.attr == _NAME) + or (isinstance(n, ast.Name) and (n.id == _NAME or n.id in aliases)) + for n in ast.walk(node) + ) + + +def _write_sites(tree: ast.AST): + sites = [] + + def visit(node, func, aliases): + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + func, aliases = node.name, set() + if isinstance(node, (ast.Assign, ast.AugAssign, ast.AnnAssign)): + targets = node.targets if isinstance(node, ast.Assign) else [node.target] + value = node.value + for t in targets: + if isinstance(t, ast.Name) and value is not None and _refers(value, aliases): + aliases.add(t.id) # a local view of the buffer + elif ( + not isinstance(t, ast.Name) + and _mentions(t, aliases) + and not isinstance(node, ast.AnnAssign) + ): + sites.append((func, node.lineno)) + elif ( + isinstance(t, ast.Attribute) + and t.attr == _NAME + and isinstance(node, ast.AnnAssign) + ): + pass # a dataclass field declaration + if isinstance(node, ast.Call): + f = node.func + if isinstance(f, ast.Attribute) and f.attr in _MUTATORS and _mentions(f.value, aliases): + sites.append((func, node.lineno)) + name = ( + f.id + if isinstance(f, ast.Name) + else (f.attr if isinstance(f, ast.Attribute) else "") + ) + if name in _COPY_HELPERS and node.args and _mentions(node.args[0], aliases): + sites.append((func, node.lineno)) + for child in ast.iter_child_nodes(node): + visit(child, func, aliases) + + visit(tree, None, set()) + return sites + + +def test_only_the_pre_forward_paths_write_the_valid_block_count() -> None: + found = set() + for path in sorted(_ROOT.rglob("*.py")): + src = path.read_text() + if _NAME not in src: + continue + rel = path.relative_to(_ROOT).as_posix() + for func, line in _write_sites(ast.parse(src)): + assert (rel, func) in _ALLOWED, ( + f"{rel}:{line} ({func}) writes msa_n_valid_blocks; the long-row select reads it before its PDL grid " + "wait (kernels/m3_select_long_dsl.py, N_VALID_PREWAIT), so a writer inside the " + "step must be ordered before the select's grid or the read moved after the wait" + ) + found.add((rel, func)) + assert found == _ALLOWED, f"expected write sites {_ALLOWED}, found {found}" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_producer.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_producer.py new file mode 100644 index 000000000000..9900882200c3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_producer.py @@ -0,0 +1,209 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_producer entry: the pre-wait-staged NVFP4 QKV + indexer producer. + +Every cell is held to the native producer (``minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert``) bit for bit on q, +index-q and every K / V / scale / index-K cache byte, and to the fp32 torch reference: the FP8 outputs within one E4M3 +step, the decoded NVFP4 K / V within one E2M1 step of their block scale. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_producer import ( # noqa: E501 + FP4_BLOCK, + HEADS_Q, + PAGE, + decode_nvfp4_token, + scale_index, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_producer import ( # noqa: E501 + m3_producer as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_producer requires a CUDA device" + +PAGES = 1024 +MAX_POSITIONS = 262144 +EPS = 1e-6 +KV_QUANT = (1.0, 0.37, 0.61) + + +def _caches(): + return ( + torch.zeros(PAGES, 2, 1, PAGE, 64, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 2, 1, PAGE, 8, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 1, PAGE, 128, dtype=torch.float8_e4m3fn, device="cuda"), + ) + + +def _inputs(tokens: int, scale: float, bad_slots: bool, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + rotary = torch.randn(MAX_POSITIONS, 2, 32, device="cuda", generator=g) + weights = [(torch.randn(128, device="cuda", generator=g) * 0.3).bfloat16() for _ in range(4)] + packed = ( + torch.randn(tokens, (HEADS_Q + 4) * 128, device="cuda", generator=g) * scale + ).bfloat16() + slots = torch.randperm(PAGES * PAGE, device="cuda", generator=g)[:tokens].int() + if bad_slots: + slots[1] = -1 + slots[2] = PAGES * PAGE + 5 + positions = torch.randint( + 0, MAX_POSITIONS, (tokens,), device="cuda", dtype=torch.int32, generator=g + ) + quant = torch.tensor(KV_QUANT, device="cuda") + return packed, slots, quant, weights, rotary, positions + + +def _args(inputs, caches, early: bool = True): + packed, slots, quant, weights, rotary, positions = inputs + return (packed, *caches, slots, quant, EPS, *weights, rotary, positions, HEADS_Q, True, early) + + +def _native(inputs, caches): + packed, slots, quant, weights, rotary, positions = inputs + return torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed, *caches, slots, quant, HEADS_Q, 1, 1, 128, 64, EPS, *weights, rotary, positions + ) + + +def _within_fp8(out: torch.Tensor, ref: torch.Tensor, name: str) -> None: + # One E4M3 step (3 mantissa bits) of the value, or its smallest subnormal step. + err = (out.float() - ref).abs() + assert bool((err <= ref.abs() * 2.0**-3 + 2.0**-9).all()), (name, float(err.max())) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + early = spec.get("early", True) + inputs = _inputs( + spec["tokens"], spec["scale"], spec.get("bad_slots", False), zlib.crc32(cell.why.encode()) + ) + stock_caches, caches = _caches(), _caches() + stock = _native(inputs, stock_caches) + args = _args(inputs, caches, early) + certifying(op) + ref = op.reference(*args) + out = op(*args) + torch.cuda.synchronize() + names = ("q", "index_q", "kv_data", "kv_scale", "index_k") + for name, s, o in zip(names, (*stock, *stock_caches), (*out, *caches)): + assert torch.equal(s.view(torch.uint8), o.view(torch.uint8)), ( + f"{name} differs from the native producer" + ) + + _within_fp8(out[0], ref["q"], "q") + _within_fp8(out[1][:, 0], ref["index_q"], "index_q") + kv_data, kv_scale, index_k = caches + slots = inputs[1] + checked = 0 + for t in range(spec["tokens"]): + slot = int(slots[t]) + if slot < 0 or slot >= PAGES * PAGE: + continue + page, within = divmod(slot, PAGE) + _within_fp8(index_k[page, 0, within], ref["index_k"][t], f"index_k token {t}") + for plane, name in ((0, "k"), (1, "v")): + scales = kv_scale[page, plane, 0].reshape(-1)[scale_index(within, plane).cuda()] + dq = decode_nvfp4_token(kv_data[page, plane, 0, within], scales, KV_QUANT[1 + plane]) + r = ref[name][t].bfloat16().float() + # Round to nearest E2M1 under the block's E4M3-rounded scale s (in value units s / quant_scale): at most + # half the widest E2M1 step (one unit), plus the clipping at 6 units when s rounded down; a scale that + # underflows E4M3 zeroes the block. A slack of one E4M3 step covers the kernel's approximate reciprocals. + qs = KV_QUANT[1 + plane] + amax = r.view(-1, FP4_BLOCK).abs().amax(dim=1) + unit = (qs * amax / 6.0).to(torch.float8_e4m3fn).float() / qs + bound = torch.where( + unit > 0, 1.15 * unit + (amax - 6.0 * unit / 1.15).clamp_min(0.0), amax + ) + bound = bound.repeat_interleave(FP4_BLOCK) + 1e-12 + assert bool(((dq - r).abs() <= bound).all()), (name, t) + checked += 1 + assert checked >= spec["tokens"] - 2 + + +def test_early_trigger_does_not_change_the_bits() -> None: + inputs = _inputs(48, 1.0, False, 7) + a, b = _caches(), _caches() + out_a = op(*_args(inputs, a, early=True)) + out_b = op(*_args(inputs, b, early=False)) + for x, y in zip((*out_a, *a), (*out_b, *b)): + assert torch.equal(x.view(torch.uint8), y.view(torch.uint8)) + + +@pytest.mark.parametrize( + "tokens, scale, bad_slots, early", + [ + (1, 1.0, False, True), + (4, 3.0, False, True), + (16, 1e-3, False, True), + (32, 40.0, True, True), + (48, 1.0, False, True), + (1000, 1.0, False, False), + ], +) +def test_prewait_step_inputs_is_bitwise_the_shared_op(tokens, scale, bad_slots, early) -> None: + """dispatch.PRODUCER_PREWAIT_STEP_INPUTS: this target's copy (minimax_m3_sm107_tp4_producer_pw, the position ids, + slots and RoPE rows read before its grid wait) writes exactly the shared op's q, index-q and cache bytes.""" + assert hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp4_producer_pw"), ( + "the target's producer op is not loaded (its TP4 extension)" + ) + inputs = _inputs(tokens, scale, bad_slots, 11) + a, b = _caches(), _caches() + out_a = op(*_args(inputs, a, early=early)) + out_b = op(*_args(inputs, b, early=early), prewait_step_inputs=True) + for x, y in zip((*out_a, *a), (*out_b, *b)): + assert torch.equal(x.view(torch.uint8), y.view(torch.uint8)) + + +def test_guard_refuses_a_repeated_slot() -> None: + inputs = list(_inputs(4, 1.0, False, 3)) + inputs[1] = inputs[1].clone() + inputs[1][1] = inputs[1][0] + with pytest.raises(AssertionError, match="must not repeat a slot"): + with validating(op): + op(*_args(tuple(inputs), _caches())) + + +def test_op_refuses_an_operand_off_the_device() -> None: + inputs = list(_inputs(4, 1.0, False, 9)) + inputs[5] = inputs[5].cpu() # position ids on the host + with pytest.raises(RuntimeError, match="packed's CUDA device"): + op(*_args(tuple(inputs), _caches())) + + +def test_op_refuses_an_unaligned_index_k_page_stride() -> None: + """The index-K rows are written as 32-bit words: a page stride that is not a multiple of 4 is refused.""" + kv_data, kv_scale, _ = _caches() + backing = torch.zeros(PAGES * (128 * 128 + 1), dtype=torch.float8_e4m3fn, device="cuda") + index_k = backing.as_strided((PAGES, 1, PAGE, 128), (128 * 128 + 1, 128 * 128, 128, 1)) + with pytest.raises(RuntimeError, match="index_k"): + op(*_args(_inputs(4, 1.0, False, 13), (kv_data, kv_scale, index_k))) + + +def test_op_refuses_unaligned_weights_or_rope_table() -> None: + """The norm weights are read as 8-byte words and the RoPE rows as float4: offset views are refused.""" + packed, slots, quant, weights, rotary, positions = _inputs(4, 1.0, False, 17) + offset_weight = torch.zeros(129, dtype=torch.bfloat16, device="cuda")[1:] + with pytest.raises(RuntimeError, match="norm weights"): + op( + *_args( + (packed, slots, quant, [offset_weight, *weights[1:]], rotary, positions), _caches() + ) + ) + flat = torch.zeros(rotary.numel() + 1, device="cuda") + offset_rope = flat[1:].view(rotary.shape) + with pytest.raises(RuntimeError, match="rotary_cos_sin"): + op(*_args((packed, slots, quant, weights, offset_rope, positions), _caches())) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..387f944008ed --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_qk_norm_rope_fp8 entry of the MiniMax-M3 tp4 target: the out-of-place FP8 form of the fused +per-head Gemma RMSNorm + partial NeoX RoPE against the in-place kernel followed by the E4M3 cast. V is a cast, so it +matches bit for bit; Q and K round once from fp32 where the composition rounds through bf16 first, so each element is +within one E4M3 step of it.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_qk_norm_rope_fp8 import ( # noqa: E501 + m3_qk_norm_rope_fp8 as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_qk_norm_rope_fp8 requires a CUDA device" + +HEAD_DIM, ROTARY_DIM, EPS, THETA = 128, 64, 1e-6, 5e6 + + +def _args(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + q, kv = spec["q"], spec["kv"] + qkv = torch.randn(spec["m"], (q + 2 * kv) * HEAD_DIM, generator=g, device="cuda").to( + torch.bfloat16 + ) + qw = (torch.randn(HEAD_DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + kw = (torch.randn(HEAD_DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + pos = torch.randint(0, 200000, (spec["m"],), generator=g, device="cuda", dtype=torch.int32) + return ( + qkv, + q, + kv, + kv, + HEAD_DIM, + ROTARY_DIM, + EPS, + qw, + kw, + THETA, + True, + pos, + 1.0, + 0.0, + 0.0, + 1.0, + True, + True, + False, + 0, + 0, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + args = _args(cell.spec, zlib.crc32(cell.why.encode()) % 2**31) + before = args[0].clone() + with validating(op): + out = op(*args) + assert torch.equal(args[0], before), "the FP8 form must leave qkv untouched" + ref = op.reference(*args) + assert out.dtype == torch.float8_e4m3fn and out.shape == ref.shape + v_start = (cell.spec["q"] + cell.spec["kv"]) * HEAD_DIM + assert torch.equal(out[:, v_start:].view(torch.uint8), ref[:, v_start:].view(torch.uint8)) + assert _e4m3_steps(out[:, :v_start], ref[:, :v_start]) <= 1 + + +def _e4m3_steps(a, b) -> int: + """The largest distance between two E4M3 tensors in representable values (E4M3 codes order by magnitude).""" + + def signed(t): + c = t.view(torch.uint8).int() + # Sign-magnitude to a line: -0 and +0 both map to 0. + return torch.where(c >= 128, 128 - c, c) + + return int((signed(a) - signed(b)).abs().max()) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_qkv_insert.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_qkv_insert.py new file mode 100644 index 000000000000..37f493baf6bb --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_qkv_insert.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_qkv_insert entry: the generic horizontal producer on an FP8 main cache. + +Q, index-Q and every written K / V / index-K cache row are held within one E4M3 step of the fp32 reference. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_qkv_insert import ( # noqa: E501 + m3_qkv_insert as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_qkv_insert requires a CUDA device" + +PAGES, PAGE, HEADS, KV, INDEX, D, ROTARY = 256, 128, 16, 1, 1, 128, 64 +MAX_POSITIONS = 262144 +EPS = 1e-6 + + +def _within_fp8(out: torch.Tensor, ref: torch.Tensor, name: str) -> None: + # One E4M3 step (3 mantissa bits) of the value, or its smallest subnormal step. + err = (out.float() - ref).abs() + assert bool((err <= ref.abs() * 2.0**-3 + 2.0**-9).all()), (name, float(err.max())) + + +def _args(tokens: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + packed = torch.randn( + tokens, (HEADS + 2 * KV + INDEX + 1) * D, device="cuda", generator=g + ).bfloat16() + kv = torch.zeros(PAGES, 2, KV, PAGE, D, dtype=torch.float8_e4m3fn, device="cuda") + index_k = torch.zeros(PAGES, 1, PAGE, D, dtype=torch.float8_e4m3fn, device="cuda") + slots = torch.randperm(PAGES * PAGE, device="cuda", generator=g)[:tokens].int() + weights = [(torch.randn(D, device="cuda", generator=g) * 0.3).bfloat16() for _ in range(4)] + rotary = torch.randn(MAX_POSITIONS, 2, ROTARY // 2, device="cuda", generator=g) + positions = torch.randint( + 0, MAX_POSITIONS, (tokens,), device="cuda", dtype=torch.int32, generator=g + ) + return ( + packed, + kv, + None, + index_k, + slots, + None, + HEADS, + KV, + INDEX, + D, + ROTARY, + EPS, + *weights, + rotary, + positions, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _args(cell.spec["m"], zlib.crc32(cell.why.encode())) + certifying(op) + ref = op.reference(*args) + q, index_q = op(*args) + torch.cuda.synchronize() + _within_fp8(q.view(ref["q"].shape), ref["q"], "q") + _within_fp8(index_q.view(ref["index_q"].shape), ref["index_q"], "index_q") + kv, index_k, slots = args[1], args[3], args[4] + for t in range(cell.spec["m"]): + page, within = divmod(int(slots[t]), PAGE) + _within_fp8(kv[page, 0, :, within], ref["k"][t], f"k token {t}") + _within_fp8(kv[page, 1, :, within], ref["v"][t], f"v token {t}") + _within_fp8(index_k[page, 0, within], ref["index_k"][t], f"index_k token {t}") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_blocks.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_blocks.py new file mode 100644 index 000000000000..c47275a63d0b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_blocks.py @@ -0,0 +1,45 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_select_blocks entry of the MiniMax-M3 tp4 target: MSA top-16 block selection with the forced +init / local blocks and per-row valid-block masks.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_select_blocks import ( # noqa: E501 + m3_select_blocks as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_select_blocks requires a CUDA device" + +TOPK, INIT, LOCAL = 16, 0, 1 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + n_blocks, total_q = spec["n_blocks"], spec["total_q"] + # Distinct scores (a permutation) so the selection has no ties to break. + scores = ( + torch.randperm(n_blocks * total_q, generator=g, device="cuda") + .float() + .view(1, n_blocks, total_q) + ) + if spec["mixed"]: + n_valid = torch.randint( + 17, n_blocks + 1, (total_q,), generator=g, device="cuda", dtype=torch.int32 + ) + else: + n_valid = torch.full((total_q,), n_blocks, dtype=torch.int32, device="cuda") + with validating(op): + out = op(scores, TOPK, n_valid, INIT, LOCAL, True) + ref = op.reference(scores, TOPK, n_valid, INIT, LOCAL, True) + assert out.shape == ref.shape + assert torch.equal(out, ref) + assert out.permute(1, 0, 2).is_contiguous(), "head-major backing" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_long.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_long.py new file mode 100644 index 000000000000..608f8ca5f564 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_long.py @@ -0,0 +1,183 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_select_long entry (top-16 MSA block selection for long decode rows).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_select_long import ( # noqa: E501 + TOPK, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_select_long import ( # noqa: E501 + m3_select_long as op, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + select_blocks_from_maxscore, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +assert torch.cuda.is_available(), "m3_select_long requires a CUDA device" + +# Valid-block counts that land a row on each of the kernel's paths: <= 16 (all picked), the 2-, 4- and 9-key +# widths, past the row (clamped), and zero (a CUDA-graph padding row). +_MIXED_VALID = (0, 5, 16, 17, 200, 2048, 2049, 4096, 4097, 8193, 9000, 1, 300, 6000, 129, 3000) + + +def _scores(spec, g): + nb, tq = spec["num_blocks"], spec["total_q"] + s = torch.randn(spec["num_kv_heads"], nb, tq, generator=g, device="cuda", dtype=torch.float32) + kind = spec["scores"] + if kind == "ties": + # Even tokens: a flat row with ten winners, so thousands of keys tie at the 16th-largest score (the + # re-read path). Odd tokens: scores on a coarse grid, so ties are broken by block index. + s[:, :, 0::2] = 0.25 + winners = torch.randint(0, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda") + s[:, :, 0::2].scatter_(1, winners[:, :, 0::2], 3.0) + s[:, :, 1::2] = (s[:, :, 1::2] * 2).round() / 2 + elif kind == "nonfinite": + s[:, 7::50, :] = float("inf") + s[:, 11::70, :] = float("nan") + s[:, 3::40, :] = -0.0 + s[:, 4::40, :] = 0.0 + # Odd tokens: almost everything -inf, so picks run past the finite scores. + s[:, :, 1::2] = float("-inf") + s[:, 0:5, 1::2] = 1.0 + return s + + +def _valid(spec, g): + tq, nb = spec["total_q"], spec["num_blocks"] + if spec["valid"] == "mixed": + vals = [_MIXED_VALID[i % len(_MIXED_VALID)] for i in range(tq)] + else: + vals = (nb - torch.randint(0, 12, (tq,), generator=g, device="cuda")).tolist() + vals[0] = nb + return torch.tensor(vals, dtype=torch.int32, device="cuda") + + +def _case(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + scores = _scores(spec, g) + nvb = _valid(spec, g) + out = torch.empty( + spec["num_kv_heads"], spec["total_q"], TOPK, dtype=torch.int32, device="cuda" + ).permute(1, 0, 2) + return scores, nvb, out, spec["init_blocks"], spec["local_blocks"], False + + +def _mass_tie_rows(scores, nvb, init_blocks, local_blocks): + """[total_q, num_kv_heads] mask of rows where more than 1024 valid blocks share the 16th-largest score.""" + num_kv_heads, num_blocks, total_q = scores.shape + blocks = torch.arange(num_blocks, device=scores.device) + raw = nvb[:total_q].long() + valid = raw.clamp(0, num_blocks) + forced = scores.permute(2, 0, 1).clone() # [total_q, heads, blocks] + forced = torch.where(blocks < init_blocks, 1.0e30, forced) + forced = torch.where(blocks >= (raw - local_blocks).clamp_min(0)[:, None, None], 1.0e29, forced) + in_row = (blocks[None, :] < valid[:, None])[:, None, :] + ranked = torch.where(in_row, forced, float("-inf")).sort(dim=-1, descending=True).values + kth = ranked[..., min(TOPK, num_blocks) - 1 : min(TOPK, num_blocks)] + return ((forced == kth) & in_row).sum(dim=-1) > 1024 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + certifying(op) + out = op(*args) + ref = op.reference(*args) + + # Selection is an exact claim: the same blocks or a different attention. + assert out.dtype is torch.int32 + assert torch.equal(out, ref), "selected blocks differ from the reference" + # And it is the in-tree selector's answer, bit for bit (the op the target falls back to) -- except on rows + # where more than 1024 blocks tie at the 16th-largest score: there the in-tree op does not break the tie to + # the smallest blocks, while this op and the reference do. + scores, nvb, _, init_blocks, local_blocks, _ = args + in_tree = select_blocks_from_maxscore( + scores, + topk=TOPK, + n_valid_blocks=nvb, + init_blocks=init_blocks, + local_blocks=local_blocks, + head_major_output=True, + ) + mass_tie = _mass_tie_rows(scores, nvb, init_blocks, local_blocks) + assert torch.equal(out[~mass_tie], in_tree[~mass_tie]), ( + "selected blocks differ from the in-tree minimax_m3_select_blocks" + ) + + +def test_pdl_launch_selects_the_same() -> None: + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=1) + pdl_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, False) + op(scores, nvb, pdl_out, init_blocks, local_blocks, True) + assert torch.equal(out, pdl_out) + + +def test_early_trigger_selects_the_same() -> None: + """Releasing the dependent grid right after the select's own wait changes only when the next kernel may launch.""" + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=2) + early_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False) + op(scores, nvb, early_out, init_blocks, local_blocks, True, True) + assert torch.equal(out, early_out) + + +@pytest.mark.parametrize("cell", [0, 1, 2]) +def test_trigger_phase_selects_the_same(cell: int) -> None: + """Releasing the dependent grid once the row's scores are loaded changes only when the next kernel may launch.""" + spec = op.CELLS[cell].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=3 + cell) + loaded_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False, 0) + op(scores, nvb, loaded_out, init_blocks, local_blocks, True, False, 1) + assert torch.equal(out, loaded_out) + + +@pytest.mark.parametrize("cell", range(len(op.CELLS))) +def test_warp_tail_selects_the_same(cell: int) -> None: + """The warp-0 tail (rows of at most 64 candidates) and the all-thread rank pick the same blocks + (dispatch.SELECT_LONG_SL3 and its kill value).""" + spec = op.CELLS[cell].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=7 + cell) + rank_out = torch.empty_like(out) + for phase in (0, 1): + op(scores, nvb, out, init_blocks, local_blocks, True, False, phase, True) + op(scores, nvb, rank_out, init_blocks, local_blocks, True, False, phase, False) + assert torch.equal(out, rank_out) + + +def test_op_refuses_a_row_past_its_width() -> None: + scores = torch.zeros(1, 9217, 1, device="cuda") + nvb = torch.ones(1, dtype=torch.int32, device="cuda") + out = torch.empty(1, 1, TOPK, dtype=torch.int32, device="cuda") + with pytest.raises(RuntimeError, match="at most 9216"): + op(scores, nvb, out, 0, 1, False) + + +def test_guard_refuses_negative_forcing() -> None: + args = list(_case(op.CELLS[0].spec, seed=2)) + args[4] = -1 + with pytest.raises(AssertionError, match="non-negative"): + with validating(op): + op(*args) + + +def test_guard_refuses_out_aliasing_scores() -> None: + scores, nvb, _, init_blocks, local_blocks, _ = _case(op.CELLS[0].spec, seed=3) + aliased = ( + scores.view(torch.int32) + .reshape(-1)[: scores.shape[2] * TOPK] + .view(scores.shape[2], 1, TOPK) + ) + with pytest.raises(AssertionError, match="alias"): + with validating(op): + op(scores, nvb, aliased, init_blocks, local_blocks, False) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_long_dsl.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_long_dsl.py new file mode 100644 index 000000000000..4458934b21b6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_select_long_dsl.py @@ -0,0 +1,191 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the tp4 (Rubin) m3_select_long_dsl entry: the long-row top-16 select from one 384-thread CTA per row, +against the reference and bit for bit against the 1024-thread m3_select_long on every cell.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( + m3_select_long_dsl as select_kernels, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_select_long import ( # noqa: E501 + TOPK, + m3_select_long, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_select_long_dsl import ( # noqa: E501 + m3_select_long_dsl as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +assert torch.cuda.is_available(), "m3_select_long_dsl requires a CUDA device" + +# Valid-block counts that land a row on each of the kernel's paths: <= 16 (all picked), the 4-, 11- and 24-key +# widths and their edges, past the row (clamped), and zero (a CUDA-graph padding row). +_MIXED_VALID = (0, 5, 16, 17, 200, 1536, 1537, 4224, 4225, 8193, 9000, 1, 300, 6000, 129, 3000) + + +def _scores(spec, g): + nb, tq = spec["num_blocks"], spec["total_q"] + s = torch.randn(spec["num_kv_heads"], nb, tq, generator=g, device="cuda", dtype=torch.float32) + kind = spec["scores"] + if kind == "ties": + # Even tokens: a flat row with ten winners, so thousands of keys tie at the 16th-largest score (the + # whole-row re-rank). Odd tokens: scores on a coarse grid, so ties are broken by block index. + s[:, :, 0::2] = 0.25 + winners = torch.randint(0, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda") + s[:, :, 0::2].scatter_(1, winners[:, :, 0::2], 3.0) + s[:, :, 1::2] = (s[:, :, 1::2] * 2).round() / 2 + elif kind == "ties1500": + s = torch.full_like(s, -1.0) + s[:, :1500, :] = 0.25 + winners = torch.randint( + 1500, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda" + ) + s.scatter_(1, winners, 3.0) + elif kind == "nonfinite": + s[:, 7::50, :] = float("inf") + s[:, 11::70, :] = float("nan") + s[:, 3::40, :] = -0.0 + s[:, 4::40, :] = 0.0 + s[:, :, 1::2] = float("-inf") + s[:, 0:5, 1::2] = 1.0 + return s + + +def _valid(spec, g): + tq, nb = spec["total_q"], spec["num_blocks"] + if spec["valid"] == "mixed": + vals = [_MIXED_VALID[i % len(_MIXED_VALID)] for i in range(tq)] + else: + vals = (nb - torch.randint(0, 12, (tq,), generator=g, device="cuda")).tolist() + vals[0] = nb + return torch.tensor(vals, dtype=torch.int32, device="cuda") + + +def _case(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + scores = _scores(spec, g) + nvb = _valid(spec, g) + out = torch.empty( + spec["num_kv_heads"], spec["total_q"], TOPK, dtype=torch.int32, device="cuda" + ).permute(1, 0, 2) + return scores, nvb, out, spec["init_blocks"], spec["local_blocks"], False + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + certifying(op) + out = op(*args) + ref = op.reference(*args) + assert out.dtype is torch.int32 + assert torch.equal(out, ref), "selected blocks differ from the reference" + # Every row, mass ties included, is the 1024-thread kernel's answer. + scores, nvb, _, init_blocks, local_blocks, _ = args + long_out = torch.empty_like(out) + m3_select_long(scores, nvb, long_out, init_blocks, local_blocks, False) + assert torch.equal(out, long_out), "selected blocks differ from m3_select_long" + + +@pytest.mark.parametrize("layout", ["kv_blocks_q", "kv_q_blocks"]) +@pytest.mark.parametrize("valid", [17, 1294, 2524, 3907, 9216]) +def test_layouts_and_widths_match_the_1024_thread_select(layout, valid) -> None: + """Both score layouts the scorer can hand over, at a valid count in each key width.""" + g = torch.Generator(device="cuda").manual_seed(valid) + nb = max(valid, 129) + s = torch.randn(1, nb, 8, generator=g, device="cuda") + if layout == "kv_q_blocks": + s = s.permute(0, 2, 1).contiguous().permute(0, 2, 1) + nvb = torch.tensor([valid - (t % 2) for t in range(8)], dtype=torch.int32, device="cuda") + out = torch.empty(1, 8, TOPK, dtype=torch.int32, device="cuda").permute(1, 0, 2) + ref = torch.empty_like(out) + op(s, nvb, out, 0, 1, False) + m3_select_long(s, nvb, ref, 0, 1, False) + assert torch.equal(out, ref) + + +def test_pdl_launch_selects_the_same() -> None: + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=1) + pdl_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, False) + op(scores, nvb, pdl_out, init_blocks, local_blocks, True) + assert torch.equal(out, pdl_out) + + +def test_early_trigger_selects_the_same() -> None: + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=2) + early_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False) + op(scores, nvb, early_out, init_blocks, local_blocks, True, True) + assert torch.equal(out, early_out) + + +@pytest.mark.parametrize("cell", [0, 1, 2]) +def test_trigger_phase_selects_the_same(cell: int) -> None: + spec = op.CELLS[cell].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=3 + cell) + loaded_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False, 0) + op(scores, nvb, loaded_out, init_blocks, local_blocks, True, False, 1) + assert torch.equal(out, loaded_out) + + +@pytest.mark.parametrize("early_trigger", [False, True]) +@pytest.mark.parametrize("cell", range(len(op.CELLS))) +def test_valid_count_read_before_the_wait_selects_the_same(cell: int, early_trigger: bool) -> None: + """kernels/m3_select_long_dsl.N_VALID_PREWAIT: the kernel that reads its row's valid-block count before the grid + wait selects exactly the blocks of the one that reads it after, on every certified cell (padding, clamped and tied + rows).""" + phase = 0 if early_trigger else 1 + spec = op.CELLS[cell].spec + scores, nvb, before, init_blocks, local_blocks, _ = _case(spec, seed=11 + cell) + after = torch.empty_like(before) + select_kernels._compile((True, early_trigger, phase, True))( + scores, nvb, before, init_blocks, local_blocks + ) + select_kernels._compile((True, early_trigger, phase, False))( + scores, nvb, after, init_blocks, local_blocks + ) + assert torch.equal(before, after) + + +def test_graph_replay_selects_the_same() -> None: + spec = op.CELLS[1].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=9) + ref = op.reference(scores, nvb, out, init_blocks, local_blocks) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + op(scores, nvb, out, init_blocks, local_blocks, True, False, 1) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(scores, nvb, out, init_blocks, local_blocks, True, False, 1) + for _ in range(3): + out.fill_(-7) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out, ref) + + +def test_op_refuses_a_row_past_its_width() -> None: + scores = torch.zeros(1, 9217, 1, device="cuda") + nvb = torch.ones(1, dtype=torch.int32, device="cuda") + out = torch.empty(1, 1, TOPK, dtype=torch.int32, device="cuda") + with pytest.raises(RuntimeError, match="at most 9216"): + op(scores, nvb, out, 0, 1, False) + + +def test_guard_refuses_negative_forcing() -> None: + args = list(_case(op.CELLS[0].spec, seed=2)) + args[4] = -1 + with pytest.raises(AssertionError, match="non-negative"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_sparse_decode.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_sparse_decode.py new file mode 100644 index 000000000000..4f75ef1e70e6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_sparse_decode.py @@ -0,0 +1,446 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_sparse_decode entry (the target's Triton block-sparse decode and its routes).""" + +import zlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_attn_lc, + m3_report, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_sparse_decode import ( # noqa: E501 + m3_sparse_decode as op, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.triton_sparse_decode import ( + minimax_m3_sparse_attn_decode as stock_decode, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_nvfp4_cache import ( # noqa: E402 — needs the path declared above + HEAD_DIM, + NUM_HEADS, + build_step, +) +from _validating import assert_within_ulp, certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_sparse_decode requires a CUDA device" + +# Band, in BF16 ulps (2^-8) of each [16, 128] head row's largest magnitude, and of the relative RMS. Every route +# computes from exact operands (E4M3 q and E2M1 x E4M3 K/V are exact in FP16), so the gap to the float64 reference +# is the BF16 rounding on both sides (one ulp of the row at most), the FP16 probabilities before PV, the split-K +# partials kept in FP32 and merged by their log2-sum-exp, and approximate exp2/log2: 2 ulps per element, the two +# roundings (each at most half an ulp) dominating the RMS, so half an ulp. Measured on GB300 over the cells: at most +# 1.80 ulps per element and 0.20 ulps RMS. +_ELEMENT_ULP = 2.0 +_RMS_ULP = 0.5 + + +def _call(case, output, identity): + return op( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + case.sm_scale, + output, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + identity, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + case = build_step( + spec["seq_lens"], + spec["decode_query_len"], + identity=spec["identity"], + seed=zlib.crc32(cell.why.encode()) % 2**31, + ) + output = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + certifying(op) + _call(case, output, spec["identity"]) + ref = op.reference( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + case.sm_scale, + output, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + spec["identity"], + ) + assert torch.isfinite(output.float()).all(), "padding rows must be zeros, not NaN" + assert_within_ulp(output, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +def test_identity_flag_does_not_change_the_answer() -> None: + """On an identity table the identity routes (the producer, or the multi-query producer from 17 rows) and the + selected-table route (the long-context kernel) compute the same attention.""" + case = build_step((2048, 130, 5, 0, 1777, 900, 2000, 0), 4, identity=True, seed=17) + outs = [] + for identity in (True, False): + output = torch.empty( + case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda" + ) + _call(case, output, identity) + outs.append(output) + assert_within_ulp(outs[0], outs[1], element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +def test_guard_refuses_a_padded_selection_slot() -> None: + """The Triton producers index the block table with every one of a token's first min(16, blocks) ids, + -1 included.""" + case = build_step((1500,), 1, identity=True, seed=23) + case.topk[0, 0, 3] = -1 + output = torch.empty(1, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + with pytest.raises(AssertionError, match="blocks of the request"): + with validating(op): + _call(case, output, True) + + +def test_guard_refuses_a_false_identity_promise() -> None: + """identity_blocks lets the multi-query producer read one table for the whole request.""" + case = build_step((1500, 1600, 1700, 1800, 1500, 1600, 1700, 1800), 4, identity=True, seed=29) + case.topk[0, 5, :2] = case.topk[0, 5, :2].flip(0) + output = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + with pytest.raises(AssertionError, match="identity_blocks promises"): + with validating(op): + _call(case, output, True) + + +def _stock(case, output): + stock_decode( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + sm_scale=case.sm_scale, + output=output, + decode_query_len=case.decode_query_len, + k_block_scale=case.k_block_scale, + v_block_scale=case.v_block_scale, + k_global_scale=case.k_global_scale, + v_global_scale=case.v_global_scale, + ) + + +def _identity_table( + seq_lens: torch.Tensor, decode_query_len: int, heads: int, lag: int = 0 +) -> torch.Tensor: + """The identity selection of every verify token, staged from lengths `lag` tokens behind seq_lens.""" + seq = (seq_lens.cpu().to(torch.int64) - lag).clamp(min=0).repeat_interleave(decode_query_len) + intra = torch.arange(seq.shape[0]) % decode_query_len + n_valid = ((seq - decode_query_len + intra + 1).clamp(min=0) + 127) // 128 + ar = torch.arange(16) + table = torch.where(ar[None, :] < n_valid[:, None], ar[None, :], -1).to(torch.int32) + return table.unsqueeze(0).expand(heads, -1, -1).contiguous().cuda() + + +# Each crosses a 128-token edge within its four verify tokens or between the stale and the true lengths. +_EDGE_LENS = (1284, 1287, 1281, 129, 132, 2048, 1028, 5, 643, 1920, 256, 1) + + +@pytest.mark.parametrize("fault", ["stale_one_step", "all_minus_one", "random_ids"]) +@pytest.mark.parametrize("requests", [1, 2, 3, 4]) +def test_identity_route_reads_its_own_lengths_not_the_table(requests, fault, monkeypatch) -> None: + """The Triton identity route (single / pair producer, up to 16 rows) derives block i for slot i from the kv lengths + the kernel reads: a staged table that lags the lengths by a verify step (a 128-token edge crossed), or is garbage, + gives the stock decode's rows over the true identity selection bit for bit. (The LC identity route has its own test + below.)""" + _triton_route(monkeypatch) + lens = tuple(_EDGE_LENS[(requests * 5 + k) % len(_EDGE_LENS)] for k in range(requests)) + case = build_step(lens, 4, identity=True, seed=400 + requests) + heads = case.topk.shape[0] + true_table = _identity_table(case.seq_lens, 4, heads) + assert torch.equal(case.topk, true_table), ( + "build_step's identity table is the identity selection" + ) + if fault == "stale_one_step": + staged = _identity_table(case.seq_lens, 4, heads, lag=4) + elif fault == "all_minus_one": + staged = torch.full_like(true_table, -1) + else: + g = torch.Generator().manual_seed(requests) + staged = ( + torch.randint(-1, case.block_table.shape[1], true_table.shape, generator=g) + .to(torch.int32) + .cuda() + ) + rows = case.q.shape[0] + target = torch.empty(rows, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + stock = torch.empty_like(target) + _stock(case, stock) + case_staged = SimpleNamespace(**{**vars(case), "topk": staged}) + _call(case_staged, target, True) + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)), ( + fault, + case.seq_lens.tolist(), + ) + + +@pytest.mark.parametrize("requests", [1, 2, 4]) +def test_identity_route_is_bitwise_the_stock_decode(requests, monkeypatch) -> None: + """The target's Triton producer + split-K merge takes the stock in-tree decode's split-K at every row count + (4 / 8 / 16 identity rows here) and so writes its rows bit for bit.""" + _triton_route(monkeypatch) + case = build_step((1500, 2048, 900, 130)[:requests], 4, identity=True, seed=60 + requests) + rows = case.q.shape[0] + target = torch.empty(rows, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + stock = torch.empty_like(target) + _call(case, target, True) + _stock(case, stock) + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)), case.seq_lens.tolist() + + +def _triton_route(monkeypatch) -> None: + """Route selected steps to the target's Triton producer + merge (this target takes the LC whenever it applies).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( + m3_attn_lc, + ) + + monkeypatch.setattr(m3_attn_lc, "lc_applies", lambda *args, **kwargs: False) + + +def _short_lens(requests: int, g: torch.Generator) -> tuple: + """Short contexts of 1..16 blocks of 128 tokens (the identity-route regime), a few exactly on a block edge.""" + lens = torch.randint(4, 16 * 128 + 1, (requests,), generator=g).tolist() + if requests > 1: + lens[0] = 128 * int(torch.randint(1, 17, (1,), generator=g)) + return tuple(lens) + + +@pytest.mark.parametrize("seed", range(12)) +@pytest.mark.parametrize("requests", [1, 2, 3, 4]) +def test_short_table_stress_is_bitwise_the_stock_decode(requests, seed, monkeypatch) -> None: + """The fail-closed id / page guard leaves valid inputs alone: short contexts (1..16 blocks) at 1..4 requests x 4 + verify rows, the identity route and a random selected route with -1 tails past each token's blocks, write the + stock decode's rows bit for bit.""" + _triton_route(monkeypatch) + g = torch.Generator().manual_seed(1000 * requests + seed) + for identity in (True, False): + case = build_step(_short_lens(requests, g), 4, identity=identity, seed=7 * seed + requests) + target = torch.empty( + case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda" + ) + stock = torch.empty_like(target) + _call(case, target, identity) + _stock(case, stock) + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)), ( + identity, + case.seq_lens.tolist(), + ) + + +@pytest.mark.parametrize("fault", ["neg_in_window", "id_past_row", "page_out_of_pool"]) +def test_invalid_slot_fails_closed(fault, monkeypatch) -> None: + """A corrupt decode input (a -1 inside a token's block window, an id past the block-table row, or a block-table + page outside the pool) is read as an absent block instead of a wild KV read: the rows equal the reference over the + remaining blocks.""" + _triton_route(monkeypatch) + case = build_step((1500, 2048, 700), 4, identity=False, seed=303) + tok, slot = 5, 2 + if fault == "neg_in_window": + case.topk[0, tok, slot] = -1 + elif fault == "id_past_row": + case.topk[0, tok, slot] = case.block_table.shape[1] + 7 + else: + req = tok // 4 + blk = int(case.topk[0, tok, slot]) + case.block_table = case.block_table.clone() + case.block_table[req, blk] = case.k_paged.shape[0] + 1000 + # Every token of the request that selected that block loses it. + hits = (case.topk[:, req * 4 : req * 4 + 4] == blk).nonzero().tolist() + output = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + _call(case, output, False) + torch.cuda.synchronize() + # The reference walks each token's first min(16, blocks) entries and drops -1: the corrupt slots as -1. + ref_topk = case.topk.clone() + if fault == "page_out_of_pool": + for h, t, k in hits: + ref_topk[h, req * 4 + t, k] = -1 + else: + ref_topk[0, tok, slot] = -1 + ref = op.reference( + case.q, + case.k_paged, + case.v_paged, + ref_topk, + case.block_table, + case.seq_lens, + case.sm_scale, + output, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + False, + ) + assert torch.isfinite(output.float()).all() + assert_within_ulp(output, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +@pytest.mark.parametrize("fault", ["none", "stale_one_step", "all_minus_one", "random_ids"]) +@pytest.mark.parametrize("requests", [1, 2, 3, 4]) +def test_lc_identity_route_reads_its_own_lengths_not_the_table( + requests, fault, monkeypatch +) -> None: + """An identity step runs the long-context kernel, which takes block i for CTA i from the kv lengths it reads. + Whatever the staged table holds (true, lagging a verify step, all -1, garbage), the rows are the LC's on the true + identity table (the selected-table route) bit for bit, and in the band of the float64 reference.""" + lens = tuple(_EDGE_LENS[(requests * 5 + k) % len(_EDGE_LENS)] for k in range(requests)) + case = build_step(lens, 4, identity=True, seed=700 + requests) + heads = case.topk.shape[0] + true_table = _identity_table(case.seq_lens, 4, heads) + if fault == "none": + staged = true_table.clone() + elif fault == "stale_one_step": + staged = _identity_table(case.seq_lens, 4, heads, lag=4) + elif fault == "all_minus_one": + staged = torch.full_like(true_table, -1) + else: + g = torch.Generator().manual_seed(requests) + staged = ( + torch.randint(-1, case.block_table.shape[1], true_table.shape, generator=g) + .to(torch.int32) + .cuda() + ) + rows = case.q.shape[0] + target = torch.empty(rows, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + selected = torch.empty_like(target) + case_staged = SimpleNamespace(**{**vars(case), "topk": staged}) + case_true = SimpleNamespace(**{**vars(case), "topk": true_table}) + taken = [] + monkeypatch.setattr(m3_report, "taken", lambda path, rows, what="": taken.append(path)) + _call(case_staged, target, True) + monkeypatch.undo() + # The LC engagement, plus its one-division merge's at these rows (m3_attn_lc.one_div_applies). + assert sorted(taken) == sorted( + ["m3_sparse_decode_lc"] + + (["sparse_decode_lc_one_div_merge"] if m3_attn_lc.one_div_applies(rows) else []) + ), taken + _call(case_true, selected, False) + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), selected.view(torch.int16)), ( + fault, + case.seq_lens.tolist(), + ) + ref = op.reference( + case.q, + case.k_paged, + case.v_paged, + true_table, + case.block_table, + case.seq_lens, + case.sm_scale, + target, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + False, + ) + assert_within_ulp(target, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +@pytest.mark.parametrize("requests", [1, 2]) +def test_lc_identity_step_reads_q_after_the_wait(requests, monkeypatch) -> None: + """An identity step has no select, so the LC must read q after its grid wait: the op gets identity_step (and the + device identity selection) on every identity step.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_attn_lc, + m3_prequant, + ) + + lens = tuple(_EDGE_LENS[(requests * 3 + k) % len(_EDGE_LENS)] for k in range(requests)) + case = build_step(lens, 4, identity=True, seed=900 + requests) + true_table = _identity_table(case.seq_lens, 4, case.topk.shape[0]) + case_true = SimpleNamespace(**{**vars(case), "topk": true_table}) + rows = case.q.shape[0] + seen = [] + real = m3_attn_lc.m3_sparse_decode_lc + + def spy(*args, **kwargs): + seen.append((kwargs.get("identity", False), kwargs.get("identity_step", False))) + return real(*args, **kwargs) + + monkeypatch.setattr(m3_attn_lc, "m3_sparse_decode_lc", spy) + out = torch.empty(rows, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + _call(case_true, out, True) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert seen[0] == (True, True), seen + assert bool(torch.isfinite(out.float()).all()) + + +@pytest.mark.parametrize("requests", [2, 3]) +def test_lc_fails_closed_on_ids_past_the_row(requests) -> None: + """A selected id past the token's row (a stale table, e.g. a padded row keeping a 500K request's n_valid) is + skipped, as the Triton producer skips it: the rows equal those with that slot at -1, bit for bit, and stay finite + even though the block-table tail it would reach points at a page whose block scales are NaN (0xFF).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_prequant, + ) + + lens = (165521, 700, 3000)[:requests] + case = build_step(lens, 4, identity=False, seed=1300 + requests) + dql = case.decode_query_len + good = case.topk.clone() + bad = case.topk.clone() + for tok in range(case.q.shape[0]): + req, qoff = divmod(tok, dql) + kv_len = max(int(case.seq_lens[req]) - dql + qoff + 1, 0) + nblk = (kv_len + 127) // 128 + past = nblk + 1 + if 2 <= nblk < 16 and past < case.block_table.shape[1]: + bad[:, tok, nblk - 1] = past + good[:, tok, nblk - 1] = -1 + page = int(case.block_table[req, past]) + case.k_block_scale[page].fill_(0xFF) + case.v_block_scale[page].fill_(0xFF) + assert not torch.equal(bad, good) + outs = [] + for table in (bad, good): + out = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + _call(SimpleNamespace(**{**vars(case), "topk": table}), out, False) + outs.append(out) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert torch.isfinite(outs[0].float()).all() + assert torch.equal(outs[0].view(torch.int16), outs[1].view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_sparse_decode_lc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_sparse_decode_lc.py new file mode 100644 index 000000000000..afee00bbdafa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_sparse_decode_lc.py @@ -0,0 +1,526 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_sparse_decode_lc entry (one-launch long-context sparse decode).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_sparse_decode_lc import ( # noqa: E501 + WIDTH, + dequant_nvfp4_pages, + mxfp8_handoff_reference, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_sparse_decode_lc import ( # noqa: E501 + m3_sparse_decode_lc as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_nvfp4_cache import build_step # noqa: E402 — needs the path declared above +from _validating import assert_within_ulp, certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_sparse_decode_lc requires a CUDA device" + +#: Hand-off buffers are sized as MERGE_MXFP8 sizes them. +_HANDOFF_ROWS = 128 + +# Band, in BF16 ulps (2^-8) of each output row's largest magnitude, and of the relative RMS. +# +# The reference is float64 attention over the exactly dequantized blocks, rounded once to BF16. The kernel's +# operands are exact too (E4M3 q and E2M1 x E4M3 K/V are exact in FP16), so what separates the two is: the final +# BF16 rounding on both sides (up to one ulp of the element, at most one of the row's scale), the softmax +# probabilities rounded to FP16 before PV (2^-11 relative each, averaging down over the block), ex2.approx / +# lg2.approx (~2^-22) and FP32 accumulation order. Hence 2 ulps per element; the RMS is dominated by the two +# roundings (each at most half an ulp, about 0.3 in aggregate), so half an ulp. Measured on GB300 over the cells: +# at most 1.91 ulps per element and 0.25 ulps RMS. +_ELEMENT_ULP = 2.0 +_RMS_ULP = 0.5 + + +def _call(case, spec, out, q8, sf): + return op( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + WIDTH // 128, + case.decode_query_len, + case.sm_scale, + spec["linear_softmax"], + False, + True, + ) + + +def _buffers(total_q, mxfp8): + out = torch.empty(total_q, WIDTH, dtype=torch.bfloat16, device="cuda") + if not mxfp8: + return out, None, None + q8 = torch.zeros(_HANDOFF_ROWS, WIDTH, dtype=torch.uint8, device="cuda") + sf = torch.zeros(_HANDOFF_ROWS * (WIDTH // 32), dtype=torch.uint8, device="cuda") + return out, q8, sf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + case = build_step( + spec["seq_lens"], + spec["decode_query_len"], + identity=False, + seed=zlib.crc32(cell.why.encode()) % 2**31, + ) + total_q = case.q.shape[0] + out, q8, sf = _buffers(total_q, spec["mxfp8"]) + certifying(op) + _call(case, spec, out, q8, sf) + ref = op.reference( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + WIDTH // 128, + case.decode_query_len, + case.sm_scale, + spec["linear_softmax"], + False, + True, + ) + assert_within_ulp(out, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + # Padding rows attend nothing and must be exact zeros, not NaN. + assert torch.isfinite(out.float()).all() + if spec["mxfp8"]: + # The hand-off is a deterministic function of the op's own BF16 rows: exact. + ref_q8, ref_sf = mxfp8_handoff_reference(out, _HANDOFF_ROWS) + assert torch.equal(q8[:total_q], ref_q8), ( + "MXFP8 rows differ from the recipe applied to the BF16 rows" + ) + assert torch.equal(sf, ref_sf), "UE8M0 scales differ from the recipe / swizzle" + + +def test_reference_dequant_matches_the_quantizer() -> None: + """The reference reads the paged bytes through its own layout claims (K scales token-major, V scales in + 4x4 token quads). Hold them to the quantizer's own flat output: every slot was written in order, so slot + ``s`` is page ``s // 128``, token ``s % 128``, and the flat scales are token-major by construction.""" + case = build_step((3000,), 1, identity=False, seed=5) + pages = torch.arange(case.k_paged.shape[0], device="cuda") + views = ((case.k_paged, case.k_block_scale, False), (case.v_paged, case.v_block_scale, True)) + for role, (data, scale, quad) in enumerate(views): + flat_data, flat_scale = torch.ops.trtllm.fp4_quantize( + case.source[role].reshape(case.num_slots, 1, -1).contiguous(), + case.quant[role + 1 : role + 2], + 16, + False, + False, + ) + expected = dequant_nvfp4_pages( + flat_data.view(torch.uint8).reshape(-1, 1, 128, 64), + flat_scale.view(torch.uint8).reshape(-1, 1, 128, 8), + pages, + quad_scales=False, + ) + got = dequant_nvfp4_pages(data, scale, pages, quad_scales=quad) + assert torch.equal(got, expected), ( + f"role {role}: paged dequantization disagrees with the quantizer" + ) + + +def test_the_variants_agree() -> None: + """The two softmax recurrences and the hand-off switch must not change the BF16 rows beyond rounding.""" + spec = dict(seq_lens=(5000, 12345), decode_query_len=4) + case = build_step(spec["seq_lens"], 4, identity=False, seed=11) + outs = [] + for linear in (False, True): + for mxfp8 in (False, True): + out, q8, sf = _buffers(case.q.shape[0], mxfp8) + _call(case, dict(linear_softmax=linear), out, q8, sf) + outs.append(out) + for other in outs[1:]: + assert_within_ulp(other, outs[0], element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + torch.testing.assert_close(outs[1], outs[0], rtol=0, atol=0) # hand-off on/off: same rows + + +def test_op_refuses_the_uninstantiated_late_trigger() -> None: + case = build_step((3000,), 1, identity=False, seed=3) + out, q8, sf = _buffers(1, True) + with pytest.raises(RuntimeError, match="early_trigger"): + op( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + WIDTH // 128, + 1, + case.sm_scale, + True, + False, + False, + ) + + +def test_guard_refuses_a_block_past_the_request() -> None: + """A selected id at or past the request's block count reads a page the token cannot attend; with every + position masked the CTA's softmax is NaN.""" + case = build_step((3000,), 1, identity=False, seed=4) + case.topk[0, 0, 0] = 40 + out, q8, sf = _buffers(1, True) + with pytest.raises(AssertionError, match="block count"): + with validating(op): + _call(case, dict(linear_softmax=False), out, q8, sf) + + +def test_guard_refuses_strided_page_rows() -> None: + """The kernel walks a page's token rows at a fixed 64-byte pitch; a padded row stride is misread.""" + case = build_step((3000,), 1, identity=False, seed=6) + wide = torch.zeros(*case.k_paged.shape[:3], 80, dtype=torch.uint8, device="cuda")[..., :64] + wide.copy_(case.k_paged) + case.k_paged = wide + out, q8, sf = _buffers(1, True) + with pytest.raises(AssertionError, match="64-byte rows"): + with validating(op): + _call(case, dict(linear_softmax=False), out, q8, sf) + + +def test_guard_refuses_a_narrow_handoff() -> None: + case = build_step((3000,), 1, identity=False, seed=7) + out, q8, sf = _buffers(1, True) + with pytest.raises(AssertionError, match="num_sf_k_tiles"): + with validating(op): + op( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + 8, + 1, + case.sm_scale, + False, + False, + True, + ) + + +@pytest.mark.parametrize( + "seq_lens, decode_query_len", + [ + ((165521,), 4), + ((5000, 12345), 4), + ((700,), 1), + ((1500, 300, 90000), 2), + ((200000, 3000, 64, 70000), 4), + ], + ids=["1x165k", "2req", "short-1row", "3req-mixed", "4req-incl-64"], +) +def test_merge_once_is_bitwise_the_image_op(seq_lens, decode_query_len) -> None: + """dispatch.SPARSE_DECODE_LC_MERGE_ONCE: the target's op (merge weights formed once per CTA) writes exactly the + image op's rows and hand-off, at both softmax recurrences, including requests with fewer than 16 selected blocks + (empty chunks, -inf LSE). With SPARSE_DECODE_LC_PREFETCH_OPROJ it also pulls two ranges the size of the TP4 + o_proj weight and scales (one not a multiple of 16 bytes) into L2: the same bits again. With + SPARSE_DECODE_LC_PREFETCH_NEXT_KV it records each row's selection (the first min(16, blocks) ids, -1 after) and + prefetches a next layer's recorded blocks, stale ids past a request's blocks included: the same bits, and the + record.""" + assert hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp4_sparse_decode_lc"), ( + "the target's sparse decode op is not loaded (its TP4 extension)" + ) + case = build_step( + seq_lens, decode_query_len, identity=False, seed=zlib.crc32(repr(seq_lens).encode()) + ) + o_proj_weight = torch.empty(6144, 2048, dtype=torch.uint8, device="cuda") + o_proj_scale = torch.empty(6144 * 64 + 9, dtype=torch.uint8, device="cuda") + rows = case.q.shape[0] + n1_self = torch.full((128, 16), -7, dtype=torch.int32, device="cuda") + n1_next = torch.randint( + -1, 2 * (max(seq_lens) // 128 + 1), (128, 16), dtype=torch.int32, device="cuda" + ) + next_kv = (n1_self, n1_next, case.k_paged, case.v_paged, case.k_block_scale, case.v_block_scale) + variants = ( + (False, None, None), + (True, None, None), + (True, (o_proj_weight, o_proj_scale), None), + (True, (o_proj_weight, o_proj_scale), next_kv), + ) + for linear in (False, True): + for mxfp8 in (False, True): + got = [] + for merge_once, prefetch, n1 in variants: + out, q8, sf = _buffers(case.q.shape[0], mxfp8) + op( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + WIDTH // 128, + case.decode_query_len, + case.sm_scale, + linear, + False, + True, + merge_once=merge_once, + prefetch=prefetch, + next_kv=n1, + ) + got.append((out, q8, sf)) + torch.cuda.synchronize() + a, aq, asf = got[0] + for i, (b, bq, bsf) in enumerate(got[1:], 1): + assert torch.equal(a.view(torch.int16), b.view(torch.int16)), (linear, mxfp8, i) + if mxfp8: + assert torch.equal(aq, bq) and torch.equal(asf, bsf), (linear, mxfp8, i) + # The record: each row's first min(16, blocks) selected ids, -1 past them; rows past the step untouched. + want = case.topk[0].clone() + for tok in range(rows): + req, qoff = divmod(tok, case.decode_query_len) + kv_len = max(int(case.seq_lens[req]) - case.decode_query_len + qoff + 1, 0) + want[tok, min(16, (kv_len + 127) // 128) :] = -1 + assert torch.equal(n1_self[:rows], want), "the recorded selection" + assert bool((n1_self[rows:] == -7).all()), "rows past the step" + + +def _tp4_op(case, out, q8, sf, linear, use_pdl, one_div): + torch.ops.trtllm.minimax_m3_sm107_tp4_sparse_decode_lc( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + WIDTH // 128, + case.decode_query_len, + case.sm_scale, + linear, + use_pdl, + True, + one_div=one_div, + ) + + +_ONE_DIV_STEPS = [ + ((165521,), 4), + ((5000, 12345), 4), + ((700,), 1), + ((1500, 300, 90000), 2), + ((200000, 3000, 64, 70000), 4), +] + + +@pytest.mark.parametrize("linear", [False, True]) +@pytest.mark.parametrize( + "seq_lens, decode_query_len", + _ONE_DIV_STEPS, + ids=["1x165k", "2req", "short-1row", "3req-mixed", "4req-incl-64"], +) +def test_one_div_merge_is_deterministic_and_in_band(seq_lens, decode_query_len, linear) -> None: + """The one-division merge (on in this target): it writes the same rows on every run + (5 eager runs, PDL on and off) and under CUDA-graph replay (3 replays == eager), MXFP8 hand-off included; the rows + are within the float64 reference's band, and within summation-order distance of the per-chunk merge.""" + case = build_step( + seq_lens, decode_query_len, identity=False, seed=zlib.crc32(repr(seq_lens).encode()) + 1 + ) + total_q = case.q.shape[0] + runs = [] + for i in range(6): + out, q8, sf = _buffers(total_q, True) + _tp4_op(case, out, q8, sf, linear, i == 5, True) + runs.append((out, q8, sf)) + torch.cuda.synchronize() + a, aq, asf = runs[0] + for b, bq, bsf in runs[1:]: + assert torch.equal(a.view(torch.int16), b.view(torch.int16)) + assert torch.equal(aq, bq) and torch.equal(asf, bsf) + out = torch.empty_like(a) + _tp4_op(case, out, None, None, linear, True, True) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + _tp4_op(case, out, None, None, linear, True, True) + for _ in range(3): + out.zero_() + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out.view(torch.int16), a.view(torch.int16)) + per, _, _ = _buffers(total_q, False) + _tp4_op(case, per, None, None, linear, False, False) + ref = op.reference( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + a, + None, + None, + WIDTH // 128, + case.decode_query_len, + case.sm_scale, + linear, + False, + True, + ) + assert torch.isfinite(a.float()).all() + assert_within_ulp(a, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + assert_within_ulp(a, per, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +_BT_PREWAIT_STEPS = [ + ((165521,), 4), + ((5000, 12345), 4), + ((700,), 1), + ((1500, 300, 90000), 2), + ((200000, 3000, 64, 70000), 4), + ((524288,), 4), # exactly the 4096-entry staging cap + ((524289, 2000), 4), # one block past the cap: the global lookup, beside a staged request + ((600000,), 1), +] + + +@pytest.mark.parametrize( + "seq_lens, decode_query_len", + _BT_PREWAIT_STEPS, + ids=[ + "1x165k", + "2req", + "short-1row", + "3req-mixed", + "4req-incl-64", + "at-cap", + "past-cap+short", + "600k-1row", + ], +) +def test_bt_prewait_is_bitwise(seq_lens, decode_query_len) -> None: + """The op's ``bt_prewait`` option (not used by this target): staging the request's block-table row in shared memory + before the grid wait writes exactly the rows and MXFP8 hand-off of the global lookup, at both softmax recurrences, + with and without the one-division merge, PDL on and off, eager and under CUDA-graph replay; rows at the 4096-entry + cap are staged, longer ones keep the global lookup; the o_proj / next-KV prefetch hints alongside change nothing.""" + case = build_step( + seq_lens, decode_query_len, identity=False, seed=zlib.crc32(repr(seq_lens).encode()) + 7 + ) + total_q = case.q.shape[0] + n1_self = torch.full((128, 16), -7, dtype=torch.int32, device="cuda") + n1_next = torch.randint( + -1, 2 * (max(seq_lens) // 128 + 1), (128, 16), dtype=torch.int32, device="cuda" + ) + next_kv = (n1_self, n1_next, case.k_paged, case.v_paged, case.k_block_scale, case.v_block_scale) + + def run(out, q8, sf, linear, one_div, bt, use_pdl, hints=False): + torch.ops.trtllm.minimax_m3_sm107_tp4_sparse_decode_lc( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + WIDTH // 128, + case.decode_query_len, + case.sm_scale, + linear, + use_pdl, + True, + *((None, None) + next_kv if hints else ()), + one_div=one_div, + bt_prewait=bt, + ) + + for linear in (False, True): + for one_div in (False, True): + base = _buffers(total_q, True) + run(*base, linear, one_div, False, False) + arms = [] + for bt, use_pdl, hints in ( + (True, False, False), + (True, True, False), + (True, True, True), + ): + bufs = _buffers(total_q, True) + run(*bufs, linear, one_div, bt, use_pdl, hints) + arms.append(bufs) + torch.cuda.synchronize() + a, aq, asf = base + for i, (b, bq, bsf) in enumerate(arms): + assert torch.equal(a.view(torch.int16), b.view(torch.int16)), (linear, one_div, i) + assert torch.equal(aq, bq) and torch.equal(asf, bsf), (linear, one_div, i) + out = torch.empty_like(a) + run(out, None, None, linear, one_div, True, True) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + run(out, None, None, linear, one_div, True, True) + for _ in range(3): + out.zero_() + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out.view(torch.int16), a.view(torch.int16)), ( + linear, + one_div, + "graph", + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_tokmap_guard.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_tokmap_guard.py new file mode 100644 index 000000000000..8af4b4357963 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_m3_tokmap_guard.py @@ -0,0 +1,226 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7) for kernels/m3_tokmap_guard: a one-request multi-token call of +thop.trtllm_gen_generation_preprocess (rows padded to the bucket, the request's generation length 1..bucket) with a +stale token map in the workspace gives, through the guard, the bytes the op gives with a freshly built map: the +appended FP8 K/V pages and the processed query's live rows, bit for bit, eager and from one captured graph replayed for +every length. Without the guard the stale map moves rows (the negative control), and a several-request call passes +through with its own map untouched. + +The fresh-map reference forces the op's own build (a scaled RoPE type with no inv-freq cache, at rotary dim 0, so no +row is rotated either way). +""" + +import math + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_tokmap_guard as guard, +) +from tensorrt_llm.bindings.internal import thop + +assert torch.cuda.is_available(), "m3_tokmap_guard requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp4 target is certified on SM 10.7 only", allow_module_level=True) + +H, KVH, D, PAGE = 16, 1, 128, 128 +BUCKET = 4 +_PEER_EXTRA = 777 # the second request's extra length behind the first +FP8_KV_QUANT_MODE = 128 +_DEV = torch.device("cuda") # unindexed on purpose: install() and the lookup normalize it +_TAKEN = [] + + +@pytest.fixture(scope="module", autouse=True) +def _installed(): + guard.install(_DEV, 48, _TAKEN.append) + yield + + +class _Step: + """One request (or ``nreq``) of BUCKET padded rows over a fresh FP8 paged cache, persistent across calls like a + real step's buffers.""" + + def __init__(self, kv_len: int, nreq: int = 1, seed: int = 0): + g = torch.Generator(device=_DEV).manual_seed(seed + kv_len) + self.nreq = nreq + # Per-request inputs (page tables, generation lengths, position offsets) for at least two requests, the page + # tables at the peer's longer length (_seq_with_peer): the stale map sends rows to request 1, which must have + # real entries behind it, also on the unguarded calls. + tables = max(nreq, 2) + pages = math.ceil((kv_len + _PEER_EXTRA + BUCKET) / PAGE) + self.nblocks = tables * pages + 2 + self.pool0 = ( + torch.randn(self.nblocks, 2, KVH, PAGE, D, device=_DEV, generator=g) + .clamp_(-4, 4) + .to(torch.float8_e4m3fn) + ) + self.pool = self.pool0.clone() + perm = torch.randperm(self.nblocks, device=_DEV, generator=g)[: tables * pages].view( + tables, pages + ) + self.offsets = torch.zeros(1, tables, 2, pages, dtype=torch.int32, device=_DEV) + self.offsets[0, :, 0] = 2 * perm + self.offsets[0, :, 1] = 2 * perm + 1 + self.pointers = torch.tensor([[self.pool.data_ptr(), 0]], dtype=torch.int64) + self.mapping = torch.zeros(1, 2, dtype=torch.int32) + self.qkv = torch.randn( + nreq * BUCKET, (H + 2 * KVH) * D, dtype=torch.bfloat16, device=_DEV, generator=g + ) + # Request lengths after this step's append; generation lengths (accepted rows) set per case. + self.seq = torch.full((nreq,), kv_len, dtype=torch.int32, device=_DEV) + self.gen = torch.full((tables,), BUCKET, dtype=torch.int32, device=_DEV) + self.pos = ( + torch.arange(BUCKET, dtype=torch.int32, device=_DEV).repeat(tables, 1).contiguous() + ) + self.one = torch.ones(1, dtype=torch.float32, device=_DEV) + self.ws = torch.empty(64 << 20, dtype=torch.uint8, device=_DEV) + self.kv_len = kv_len + + def call(self, op, fresh_map: bool = False): + """One generation preprocess; returns a copy of the processed query.""" + out = op( + self.qkv, + self.ws, + self.seq, + self.gen, + self.pos, + self.offsets, + self.pointers, + self.mapping, + self.one, + self.one, + None, + None, + None, + None, + 0, + 0, + H, + KVH, + D, + PAGE, + FP8_KV_QUANT_MODE, + 1 << 20, + 1 << 20, + self.nreq * BUCKET, + self.nreq, + BUCKET, + self.kv_len, + 0, + 10000.0, + 1 if fresh_map else 0, + 1.0, + 1 << 20, + 0, + 1.0, + 1.0, + False, + BUCKET, + 0, + torch.cuda.get_device_properties(0).multi_processor_count, + self.nblocks, + 2, + True, + False, + ) + return out[0].clone() + + def poison_map(self) -> None: + """A stale multi-request map in the op's one-request map slice: rows 1.. on request 1.""" + layout = thop.get_trtllm_gen_generation_workspace_layout( + self.qkv.dtype, 1, self.nreq * BUCKET, H, D, 0, KVH, 0, False, False + ) + off = int(layout["tokens_info_offset"]) + stale = torch.tensor( + [[0, 0]] + [[1, t] for t in range(BUCKET - 1)], dtype=torch.int32, device=_DEV + ) + self.ws[off : off + 8 * BUCKET].view(torch.int32).view(BUCKET, 2).copy_(stale) + + def reset(self) -> None: + self.pool.copy_(self.pool0) + + +def _seq_with_peer(step: _Step) -> None: + # A second request's length behind the first, so a row the stale map moves to request 1 reads a real length. + step.seq = torch.tensor( + [step.kv_len, step.kv_len + _PEER_EXTRA], dtype=torch.int32, device=_DEV + ) + + +@pytest.mark.parametrize("kv_len", [130, 8191, 34117]) +@pytest.mark.parametrize("n", [1, 2, 3, 4]) +def test_guard_equals_a_fresh_map(kv_len, n) -> None: + step = _Step(kv_len) + _seq_with_peer(step) + step.gen.fill_(n) + step.poison_map() + step.reset() + q_ref = step.call(guard._original, fresh_map=True) + pool_ref = step.pool.clone() + + step.poison_map() + step.reset() + q = step.call(thop.trtllm_gen_generation_preprocess) + # The processed query's n live rows (rows past the request's length are padding the op leaves unwritten). + assert torch.equal(q[:n].view(torch.uint8), q_ref[:n].view(torch.uint8)) + assert torch.equal(step.pool.view(torch.uint8), pool_ref.view(torch.uint8)) + assert BUCKET in _TAKEN + + +def test_stale_map_moves_rows_without_the_guard() -> None: + step = _Step(34117) + _seq_with_peer(step) + step.gen.fill_(BUCKET) + step.poison_map() + step.reset() + step.call(guard._original, fresh_map=True) + pool_ref = step.pool.clone() + step.poison_map() + step.reset() + step.call(guard._original) + assert not torch.equal(step.pool.view(torch.uint8), pool_ref.view(torch.uint8)) + + +def test_one_captured_graph_serves_every_length() -> None: + step = _Step(34000) + _seq_with_peer(step) + refs = {} + for n in range(1, BUCKET + 1): + step.gen.fill_(n) + step.poison_map() + step.reset() + q = step.call(guard._original, fresh_map=True) + refs[n] = (q, step.pool.clone()) + step.gen.fill_(BUCKET) + step.call(thop.trtllm_gen_generation_preprocess) # eager first, as the warmup runs it + torch.cuda.synchronize() + g = torch.cuda.CUDAGraph() + with torch.cuda.graph(g): + out = thop.trtllm_gen_generation_preprocess + q_static = step.call(out) + for n in range(1, BUCKET + 1): + step.gen.fill_(n) + step.poison_map() + step.reset() + g.replay() + torch.cuda.synchronize() + q_ref, pool_ref = refs[n] + assert torch.equal(q_static[:n].view(torch.uint8), q_ref[:n].view(torch.uint8)), n + assert torch.equal(step.pool.view(torch.uint8), pool_ref.view(torch.uint8)), n + + +def test_several_requests_pass_through() -> None: + step = _Step(8191, nreq=2) + layout = thop.get_trtllm_gen_generation_workspace_layout( + step.qkv.dtype, 2, 2 * BUCKET, H, D, 0, KVH, 0, False, False + ) + step.reset() + step.call(thop.trtllm_gen_generation_preprocess) + pool_guarded = step.pool.clone() + step.reset() + step.call(guard._original) + assert torch.equal(step.pool.view(torch.uint8), pool_guarded.view(torch.uint8)) + assert int(layout["tokens_info_size"]) >= 8 * 2 * BUCKET diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_msa_kv_update.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_msa_kv_update.py new file mode 100644 index 000000000000..8d7e115734fb --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/attention/test_modeling_v2_msa_kv_update.py @@ -0,0 +1,133 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the msa_kv_update entry (the one-launch MSA KV-length correction).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.msa_kv_update import ( # noqa: E501 + msa_kv_update as hook, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.msa_kv_update import ( # noqa: E501 + TOKENS_PER_BLOCK, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.msa_kv_update import ( # noqa: E501 + msa_kv_update as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "msa_kv_update requires a CUDA device" + +#: Spare capacity past the live rows, as the metadata's buffers have: the op must leave it alone. +_SPARE = 3 + + +def _case(spec, seed): + """The metadata's buffers for one step, with stale values in every output so a skipped write shows.""" + g = torch.Generator(device="cpu").manual_seed(seed) + batch, qlen = spec["batch"], spec["query_len"] + total_q = batch * qlen + staged = torch.randint(qlen, 5000, (batch,), generator=g, dtype=torch.int32) + accept = spec["accept"] + if accept == "full": + kv = staged.clone() + elif accept == "page_edge": + staged = torch.tensor([130, 257, 2 * TOKENS_PER_BLOCK + qlen][:batch], dtype=torch.int32) + kv = staged - torch.tensor([0, 1, qlen - 1][:batch], dtype=torch.int32) + else: + kv = staged - torch.randint(0, qlen, (batch,), generator=g, dtype=torch.int32) + if accept == "padded": + kv[batch // 2 :] = 0 + staged[batch // 2 :] = 0 + ncols = (5000 + qlen) // TOKENS_PER_BLOCK + 2 + table = ( + torch.randperm(batch * ncols + 7, generator=g)[: batch * ncols] + .to(torch.int32) + .reshape(batch, ncols) + ) + dev = dict(device="cuda", dtype=torch.int32) + stale = -12345 + return ( + torch.cat([kv, torch.full((_SPARE,), 99, dtype=torch.int32)]).cuda(), + torch.cat([staged, torch.full((_SPARE,), 99, dtype=torch.int32)]).cuda(), + torch.full((batch + _SPARE,), qlen, **dev), + torch.arange(batch, **dev).repeat_interleave(qlen), + torch.arange(qlen, **dev).repeat(batch), + table.cuda(), + TOKENS_PER_BLOCK, + batch, + total_q, + torch.full((batch + _SPARE,), stale, **dev), + torch.full((batch + 1 + _SPARE,), stale, **dev) if spec["cu_kv"] else None, + torch.full((total_q + _SPARE,), stale, **dev), + torch.full((total_q + _SPARE,), stale, **dev) if spec["n_valid"] else None, + ) + + +def _check(args, ref): + seq, cu_kv, loc, n_valid = args[9], args[10], args[11], args[12] + batch, total_q = args[7], args[8] + ref_seq, ref_cu, ref_loc, ref_nv = ref + # Integer bookkeeping: exact, and nothing past the live rows is touched. + assert torch.equal(seq[:batch], ref_seq) + assert torch.equal(loc[:total_q], ref_loc) + assert bool((seq[batch:] == -12345).all()) and bool((loc[total_q:] == -12345).all()) + assert (cu_kv is None) == (ref_cu is None) and (n_valid is None) == (ref_nv is None) + if cu_kv is not None: + assert torch.equal(cu_kv[: batch + 1], ref_cu) + assert bool((cu_kv[batch + 1 :] == -12345).all()) + if n_valid is not None: + assert torch.equal(n_valid[:total_q], ref_nv) + assert bool((n_valid[total_q:] == -12345).all()) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + certifying(op) + op(*args) + _check(args, op.reference(*args)) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_the_hook_takes_every_cell(cell) -> None: + """The target's hook (what the metadata calls) takes each certified step and writes what the op does.""" + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31 + 1) + assert hook(*args) is True + _check(args, op.reference(*args)) + + +def test_the_hook_declines_what_the_op_would_misread() -> None: + """A strided operand is read wrongly by the op, so the hook declines it (the metadata then runs its own + torch update) and writes nothing.""" + args = list(_case(op.CELLS[2].spec, seed=5)) + args[0] = torch.stack([args[0], args[0]], dim=1)[:, 0] + before = args[11].clone() + assert hook(*args) is False + assert torch.equal(args[11], before) + + +def test_guard_refuses_a_strided_operand() -> None: + args = list(_case(op.CELLS[2].spec, seed=6)) + args[0] = torch.stack([args[0], args[0]], dim=1)[:, 0] + with pytest.raises(AssertionError, match="contiguous"): + with validating(op): + op(*args) + + +def test_guard_refuses_a_request_index_past_the_batch() -> None: + args = list(_case(op.CELLS[2].spec, seed=7)) + args[3] = args[3].clone() + args[3][-1] = args[7] + with pytest.raises(AssertionError, match="q_batch_row"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_allreduce_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_allreduce_op_matrix.py new file mode 100644 index 000000000000..d0a02412264f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_allreduce_op_matrix.py @@ -0,0 +1,440 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU certification matrix for the ``m3_allreduce`` entry, 4 ranks: its cells against the entry's reference, and the +fused draft loop's layer boundaries built on it (``M3Eagle3Draft.ar_norm``: the plain all-reduce, then FlashInfer's +fused add + RMSNorm) against stock Eagle3's, bit for bit. + +Stock ``Eagle3DecoderLayer`` all-reduces each projection plainly (BF16 out) and runs the ``RMSNorm`` module with the +residual: FlashInfer's ``fused_add_rmsnorm``, which adds in FP32 and normalizes the FP32 sum. The draft loop's +boundaries must be that path exactly, under every all-reduce strategy the single-node drafter can run (AUTO, +ONESHOT, TWOSHOT, NCCL; MNNVL is multi-node only), at tp4 shapes (H = 6144, eps = 1e-6, 1 to 32 rows). The inputs are +random rows and rows built so a BF16 rounding of the residual sum would move the norm the most, including the +review's case: unit weights, sums 1 + 2^-8 on 6143 coordinates and 2 - 2^-8 on the last, both BF16 rounding +midpoints that round to even in opposite directions. + +Asserts the normalized rows and the residual sum bitwise equal to the stock path's, and reports the max ulp of each +against an fp64 reference (ulps of BF16 at the larger magnitude; a correctly rounded output is within 0.5). + +Its own 4-rank launcher, as ``_m3_mc_allreduce_norm_op_matrix.py`` is; the collected entry point is +``test_modeling_v2_m3_allreduce.py``. + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _m3_allreduce_op_matrix.py +""" + +import math +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Callable, List, Optional, Tuple + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "the fused all-reduce + RMSNorm requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 + +HIDDEN = 6144 +EPS = 1e-6 +TOKENS = (1, 4, 8, 32) +RESULT_TAG = "[ulp]" +"""Prefix of rank 0's result lines; the pytest entry reports them.""" + +RANK = 0 +WORLD = 1 + +Inputs = Tuple[List[torch.Tensor], torch.Tensor, torch.Tensor] +"""(every rank's [m, H] BF16 partial rows, the [m, H] BF16 residual, the [H] BF16 norm weight).""" + + +# ---------------------------------------------------------------- inputs (every rank builds every rank's rows) + + +def _gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def _random(seed: int, m: int, x_scale: float, r_scale: float, unit_w: bool) -> Inputs: + g = _gen(seed) + xs = [ + (torch.randn(m, HIDDEN, generator=g, device="cuda") * x_scale).bfloat16() + for _ in range(WORLD) + ] + residual = (torch.randn(m, HIDDEN, generator=g, device="cuda") * r_scale).bfloat16() + if unit_w: + w = torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16) + else: + w = (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).bfloat16() + return xs, residual, w + + +def _on_rank0(m: int, x: torch.Tensor) -> List[torch.Tensor]: + """``x`` as rank 0's partial, zeros elsewhere (the cross-rank sum is exact).""" + return [x.bfloat16()] + [ + torch.zeros(m, HIDDEN, device="cuda", dtype=torch.bfloat16) for _ in range(WORLD - 1) + ] + + +def _review_case(m: int) -> Inputs: + """Sums 1 + 2^-8 on the first H - 1 coordinates and 2 - 2^-8 on the last, unit weights. + + Both sums are BF16 midpoints: 1 + 2^-8 rounds (to even) down to 1 and 2 - 2^-8 up to 2, so the fused path's + norm sees a residual moved in opposite directions on the bulk and on the last coordinate. + """ + residual = torch.ones(m, HIDDEN, device="cuda") + residual[:, -1] = 2.0 - 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + return ( + _on_rank0(m, x), + residual.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _review_case_flipped(m: int) -> Inputs: + """The mirror: the bulk on odd-mantissa midpoints (1 + 3 * 2^-8 rounds up), the last one rounding down.""" + residual = torch.full((m, HIDDEN), 1.0 + 2.0**-7, device="cuda") + residual[:, -1] = 2.0 - 2.0 * 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + return ( + _on_rank0(m, x), + residual.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _midpoints(seed: int, m: int, outliers: int) -> Inputs: + """Every sum a BF16 midpoint: residual r random in [1, 2) and x = half its ulp (+2^-8), split over the ranks. + + The bulk has an odd mantissa (rounds up); ``outliers`` coordinates per row carry an even one (rounds down) and + a larger magnitude, so they weigh in the RMS while rounding the other way. + """ + g = _gen(seed) + mant = ( + torch.randint(0, 64, (m, HIDDEN), generator=g, device="cuda") * 2 + 1 + ) # odd 7-bit mantissa + r = 1.0 + mant.float() * 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + if outliers: + cols = torch.randperm(HIDDEN, generator=g, device="cuda")[:outliers] + even = torch.randint(0, 64, (m, outliers), generator=g, device="cuda") * 2 + r[:, cols] = 8.0 * (1.0 + even.float() * 2.0**-7) + x[:, cols] = 8.0 * 2.0**-8 + # Split x over the ranks in exact BF16 pieces: x/2 on rank 0, x/4 on ranks 1 and 2, 0 on rank 3. + parts = [x / 2, x / 4, x / 4] + [torch.zeros_like(x)] * (WORLD - 3) + return ( + [p.bfloat16() for p in parts[:WORLD]], + r.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _cancelling(seed: int, m: int) -> Inputs: + """Large partials that cancel across the ranks (+A, -A, +B, -B + d) onto a unit residual.""" + g = _gen(seed) + a = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 64).bfloat16().float() + b = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 64).bfloat16().float() + d = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 2.0**-6).bfloat16().float() + parts = [a, -a, b, (-b + d)] + residual = (1.0 + torch.rand(m, HIDDEN, generator=g, device="cuda")).bfloat16() + w = (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).bfloat16() + return [p.bfloat16() for p in parts[:WORLD]], residual, w + + +def _cases() -> List[Tuple[str, Callable[[int], Inputs]]]: + cases: List[Tuple[str, Callable[[int], Inputs]]] = [ + ("review: 1+2^-8 x6143, 2-2^-8 x1", _review_case), + ("review mirrored", _review_case_flipped), + ("all midpoints, no outliers", lambda m: _midpoints(11, m, 0)), + ("all midpoints, 1 outlier", lambda m: _midpoints(12, m, 1)), + ("all midpoints, 64 outliers", lambda m: _midpoints(13, m, 64)), + ("cancelling partials", lambda m: _cancelling(14, m)), + ] + for i, (xs, rs, unit) in enumerate( + ((1.0, 4.0, False), (1.0, 1.0, True), (0.05, 8.0, False), (8.0, 0.5, False)) + ): + cases.append( + ( + f"random x{xs} r{rs}{' unit-w' if unit else ''}", + lambda m, i=i, xs=xs, rs=rs, unit=unit: _random(100 + i, m, xs, rs, unit), + ) + ) + return cases + + +# ---------------------------------------------------------------- the three paths + + +def _fp64(xs: List[torch.Tensor], residual: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + s = residual.double() + for x in xs: + s = s + x.double() + rms = torch.rsqrt(s.square().mean(-1, keepdim=True) + EPS) + return s * rms * w.double() + + +def _ulp_at(v: torch.Tensor) -> torch.Tensor: + """BF16's ulp at |v| (fp64): 2^(e - 8) for |v| in [2^(e-1), 2^e), floored at the smallest normal's.""" + _, e = torch.frexp(v.abs().double()) + return torch.ldexp(torch.ones_like(v, dtype=torch.float64), torch.clamp(e - 8, min=-133)) + + +def _ulps(out: torch.Tensor, ref: torch.Tensor) -> float: + """Max |out - ref| in BF16 ulps at the larger magnitude of the two.""" + o, r = out.double(), ref.double() + return ((o - r).abs() / _ulp_at(torch.maximum(o.abs(), r.abs()))).max().item() + + +def _ulps_at(out: torch.Tensor, ref: torch.Tensor, scale: torch.Tensor) -> float: + """Max |out - ref| in BF16 ulps at the magnitude ``scale`` (the operands'), not at the result's.""" + return ((out.double() - ref.double()).abs() / _ulp_at(scale.abs())).max().item() + + +def _target(ar: Any, x: torch.Tensor, residual: torch.Tensor, w: torch.Tensor): + """The draft loop's boundary (``M3Eagle3Draft.ar_norm``): ``m3_allreduce`` plain, then the catalog's + ``flashinfer_fused_add_rmsnorm``.""" + from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( # noqa: E501 + flashinfer_fused_add_rmsnorm, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.comm.m3_allreduce import ( # noqa: E501 + m3_allreduce, + ) + + return flashinfer_fused_add_rmsnorm(m3_allreduce(x, ar, None), residual.clone(), w, EPS) + + +def _check_cells(mapping: Any) -> None: + """Every cell of ``m3_allreduce.CELLS`` (AUTO strategy, as the target builds it) against the entry's reference: + rank-order fp32 sums with one BF16 rounding of the sum and one of the norm. A strategy may round the cross-rank sum + to BF16 before adding the residual, so the residual sum is bounded at its operands' magnitude + u = ulp(max(|residual|, |sum|)): half an ulp for that rounding plus half an output ulp for each of the two final + roundings, an output ulp being at most 2u, so 2.5u (where the two cancel, the output's own ulp is far smaller). + The normed rows are the norm of the op's own residual sum, within 2 ulps of the reference norm of that sum.""" + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.comm.m3_allreduce import ( # noqa: E501 + m3_allreduce, + ) + from tensorrt_llm._torch.distributed import AllReduce, AllReduceFusionOp, AllReduceParams + from tensorrt_llm.functional import AllReduceStrategy + + sys.path.insert(0, str(Path(__file__).resolve().parents[4])) + from _validating import validating + + ar = AllReduce(mapping=mapping, strategy=AllReduceStrategy.AUTO, dtype=torch.bfloat16) + for i, cell in enumerate(m3_allreduce.CELLS): + spec = cell.spec + xs, residual, w = _random(200 + i, spec["m"], 1.0, 4.0, False) + params = ( + None + if spec.get("plain") + else AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual.clone(), + norm_weight=w, + eps=EPS, + trigger_completion_at_end=False, + ) + ) + # The reference first: the fused path may write the updated residual into ``params.residual``. + ref = m3_allreduce.reference(xs, ar, params) + with validating(m3_allreduce): + out = m3_allreduce(xs[RANK], ar, params) + torch.cuda.synchronize() + total = sum(x.double() for x in xs) + if params is None: + err = _ulps_at(out, ref, total) + assert err <= 1.0, (cell.why, err) + else: + err = _ulps_at(out[1], ref[1], torch.maximum(total.abs(), residual.double().abs())) + assert err <= 2.5, (cell.why, "residual", err) + h = out[1].double() + own = h * torch.rsqrt(h.square().mean(-1, keepdim=True) + EPS) * w.double() + assert _ulps(out[0], own) <= 2.0, (cell.why, "normed", _ulps(out[0], own)) + dist.barrier() + if RANK == 0: + print(f"{RESULT_TAG} cell {cell.why} passed", flush=True) + + +def _stock(ar: Any, x: torch.Tensor, residual: torch.Tensor, w: torch.Tensor): + """Stock Eagle3: ``o_proj``'s plain all-reduce (BF16 out), then ``RMSNorm(h, residual)``'s FlashInfer + ``fused_add_rmsnorm`` (FP32 add, norm of the FP32 sum; in place on both).""" + h = ar(x) + res = residual.clone() + torch.ops.trtllm.flashinfer_fused_add_rmsnorm(h, res, w, EPS) + return h, res + + +# ---------------------------------------------------------------- the matrix + + +def _single_node_cluster_info() -> None: + """``TorchDist`` (the communicator under ``TLLM_DISABLE_MPI``) asks Ray for each rank's node and GPU to build its + node-local group. These ranks are spawned without Ray on one node, so every rank reports the same node and its + own device.""" + from tensorrt_llm._torch.distributed.communicator import TorchDist + + def cluster_info(self): + if self.cluster_info is None: + self.cluster_info = [("127.0.0.1", r) for r in range(dist.get_world_size())] + return self.cluster_info + + TorchDist._get_cluster_info = cluster_info + + +def _run_matrix() -> None: + import tensorrt_llm._torch.custom_ops # noqa: F401 -- registers trtllm::flashinfer_fused_add_rmsnorm + from tensorrt_llm._torch.distributed import AllReduce + from tensorrt_llm._torch.flashinfer_utils import IS_FLASHINFER_AVAILABLE + from tensorrt_llm.functional import AllReduceStrategy + from tensorrt_llm.mapping import Mapping + + assert IS_FLASHINFER_AVAILABLE, "stock Eagle3's add + RMSNorm is FlashInfer's fused_add_rmsnorm" + _single_node_cluster_info() + mapping = Mapping(world_size=WORLD, tp_size=WORLD, rank=RANK, gpus_per_node=WORLD) + _check_cells(mapping) + strategies = ( + AllReduceStrategy.AUTO, + AllReduceStrategy.ONESHOT, + AllReduceStrategy.TWOSHOT, + AllReduceStrategy.NCCL, + ) + worst = 0.0 + mismatches: List[str] = [] + for strategy in strategies: + ar = AllReduce(mapping=mapping, strategy=strategy, dtype=torch.bfloat16) + assert ar.mnnvl_allreduce is None and ar.symm_mem_allreduce is None, strategy + for name, build in _cases(): + ulp, res_ulp = 0.0, 0.0 + for m in TOKENS: + xs, residual, w = build(m) + target_out, target_res = _target(ar, xs[RANK], residual, w) + stock_out, stock_res = _stock(ar, xs[RANK], residual, w) + torch.cuda.synchronize() + for what, a, b in ( + ("normed", target_out, stock_out), + ("residual", target_res, stock_res), + ): + bad = int((a.view(torch.int16) != b.view(torch.int16)).sum().item()) + if bad: + mismatches.append( + f"{strategy.name} {name} m={m}: {bad} {what} elements differ from stock" + ) + ref = _fp64(xs, residual, w) + ref_res = residual.double() + sum(x.double() for x in xs) + u, ur = _ulps(target_out, ref), _ulps(target_res, ref_res) + assert math.isfinite(u) and math.isfinite(ur), (strategy, name, m) + ulp, res_ulp = max(ulp, u), max(res_ulp, ur) + dist.barrier() + worst = max(worst, ulp) + if RANK == 0: + print( + f"{RESULT_TAG} {strategy.name:7s} | {name:34s} | target = stock vs fp64: normed {ulp:.3f}" + f" | residual {res_ulp:.3f}", + flush=True, + ) + if RANK == 0: + scope = f"all cases, m in {TOKENS}" + print( + f"{RESULT_TAG} MAX | {scope:34s} | target = stock vs fp64: normed {worst:.3f}", + flush=True, + ) + assert not mismatches, "\n".join(mismatches) + + +def _run_one_rank() -> int: + global RANK, WORLD + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(rank) + dist.init_process_group( + "cpu:gloo,cuda:nccl", init_method="env://", device_id=torch.device("cuda", rank) + ) + RANK, WORLD = rank, world + assert WORLD == 4, f"the drafter runs at tp4, got {WORLD} ranks" + try: + _run_matrix() + except BaseException: + import traceback # only the rank subprocess's failure path needs it + + print(f"[rank {RANK}] FAILED", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + dist.barrier() + torch.cuda.synchronize() + print(f"[rank {RANK}] passed", flush=True) + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + """One worker per claimed device, each in its own session; on the first nonzero exit or the deadline every + session is killed (as ``_m3_mc_allreduce_norm_op_matrix.py`` does).""" + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the 4 devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size == 4, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); the matrix runs at 4" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +if __name__ == "__main__": + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank()) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_capture_in_boundary_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_capture_in_boundary_op_matrix.py new file mode 100644 index 000000000000..9d04b6a0db98 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_capture_in_boundary_op_matrix.py @@ -0,0 +1,307 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CAPTURE_NORM_IN_BOUNDARY on the tp4 multicast all-reduce + RMSNorm op, at 4 ranks. + +With ``cap_out`` the boundary also writes the Eagle3 capture of its updated residual. Asserted bit for bit against +``m3_spec_tail.capture_norm`` of the op's own updated residual, in a strided [M, H] block of a [rows, 3H] capture +buffer (the other bytes untouched), and the op's normed / updated / MXFP8 outputs bitwise the same call's without +``cap_out``. Every kernel a boundary can run: + +- the 8-CTA cluster kernel in the stock oneshot order (options 39 / 47, plain calls up to 8 rows, x and x + x2, + prepushed 2); +- the row kernel (plain options 7 / 15 and the carveout instance 71 / 79, 1 to 8 rows on the 8-token bucket, 12 rows + on the 16-token bucket, 32 rows on the 48-token bucket, prepushed 2); + +(The tp4 MoE finalize runs the shared GB300 op, which has no capture epilogue.) + +at residual scales 1e-3 to 1e3 and the fc_norm eps 1e-6 / 1e-5. + +Its own 4-rank launcher, as ``_m3_draft_ar_norm_op_matrix.py`` is; the collected entry point is +``test_modeling_v2_m3_capture_in_boundary.py``. + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _m3_capture_in_boundary_op_matrix.py +""" + +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Dict, Optional + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "m3_mc_allreduce_norm requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 + +HIDDEN = 6144 +EPS = 1e-6 +CLUSTER = 8 +BUCKETS = (8, 16, 48) +EMPTY_WORD = -2147483648 + +# Bound inside the rank body: importing the entries pulls in tensorrt_llm. +allreduce_entry: Any = None +ar_module: Any = None +RANK = 0 +WORLD = 1 +WORKSPACES: Dict[int, Any] = {} + + +def _gen(seed: int, rank: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed * 1009 + rank + 1) + + +def _rows(seed: int, rank: int, m: int, scale: float = 1.0) -> torch.Tensor: + g = _gen(seed, rank + 1) + return (torch.randn(m, HIDDEN, generator=g, device="cuda") * scale).to(torch.bfloat16) + + +def _norm_w(seed: int) -> torch.Tensor: + g = _gen(seed + 11, 0) + return (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).to(torch.bfloat16) + + +def _make_workspace(maxt: int): + """One bucket's exchange, as ``McAllReduceWorkspace.create`` builds it, over the torch.distributed group.""" + from tensorrt_llm._torch.distributed.ops import _make_mnnvl_mcast_buffer + from tensorrt_llm.mapping import Mapping + + mapping = Mapping(world_size=WORLD, tp_size=WORLD, rank=RANK) + words = 4 * maxt * WORLD * (HIDDEN // 2) + handle = _make_mnnvl_mcast_buffer(dist.group.WORLD, words * 4, mapping, False) + uc = handle.get_uc_buffer(RANK, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + dist.barrier() + ws = ar_module.McAllReduceWorkspace( + uc=uc, + mc=mc, + flags={}, + rank=RANK, + world_size=WORLD, + hidden=HIDDEN, + handle=handle, + comm=dist.group.WORLD, + max_tokens=maxt, + ) + ws.flags_for(CLUSTER) + return ws + + +def _push(ws, rows: torch.Tensor) -> None: + """Store this rank's rows into every rank's exchange (set A), as a producer does before a prepushed call.""" + m = rows.shape[0] + torch.cuda.synchronize() + parity = (ws.flags_for(CLUSTER).view(ws.max_tokens, CLUSTER)[:m] & 1).long() + half = ws.max_tokens * WORLD * (HIDDEN // 2) + chunk = HIDDEN // 2 // CLUSTER + h16 = rows.contiguous().view(torch.int16) + h16 = torch.where(h16 == -32768, torch.zeros_like(h16), h16) + words = h16.view(torch.int32).view(m, CLUSTER, chunk) + t = torch.arange(m, device="cuda").view(m, 1, 1) + c = torch.arange(CLUSTER, device="cuda").view(1, CLUSTER, 1) + w = torch.arange(chunk, device="cuda").view(1, 1, chunk) + index = parity.view(m, CLUSTER, 1) * half + (t * WORLD + RANK) * (HIDDEN // 2) + c * chunk + w + ws.mc[index.reshape(-1)] = words.reshape(-1) + torch.cuda.synchronize() + dist.barrier() + + +def _assert_bitwise(out: torch.Tensor, ref: torch.Tensor, where: str) -> None: + assert out.dtype == ref.dtype and out.shape == ref.shape, (where, out.dtype, ref.dtype) + a, b = out.contiguous().view(torch.uint8), ref.contiguous().view(torch.uint8) + bad = (a != b).sum().item() + assert bad == 0, f"{where}: {bad} of {a.numel()} bytes differ" + + +def _capture_reference(updated, weight, eps, out) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_spec_tail, + ) + + m3_spec_tail.capture_norm(updated, None, weight, eps, out) + + +def _one_case( + kind: str, m: int, maxt: int, opts: int, scale: float, quant: bool, seed: int +) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.comm.m3_mc_allreduce_norm import ( # noqa: E501 + sf_written, + ) + + cap_eps = (1e-6, 1e-5)[seed % 2] + ws = WORKSPACES[maxt] + residual = (_rows(seed + 7, -1, m).float() * 4.0 * scale).to(torch.bfloat16) + norm_w, cap_w = _norm_w(seed), _norm_w(seed + 101) + results = [] + for with_cap in (False, True): + buf = torch.full((m + 3, 3 * HIDDEN), 7.0, dtype=torch.bfloat16, device="cuda") + cap = buf[:m, HIDDEN : 2 * HIDDEN] if with_cap else None + cw = cap_w if with_cap else None + flags = ws.flags_for(CLUSTER) + x = _rows(seed, RANK, m, scale) + x2 = _rows(seed + 3, RANK, m, scale) if kind == "x2" else None + prepushed = 2 if kind == "prepushed" else 0 + if prepushed: + _push(ws, x) + outs = allreduce_entry.m3_mc_allreduce_norm( + x, x2, residual, norm_w, EPS, ws.uc, ws.mc.data_ptr(), flags, RANK, WORLD, CLUSTER, False, maxt, + quant, prepushed, opts, cap, cw, cap_eps, + ) # fmt: skip + torch.cuda.synchronize() + results.append((outs, buf)) + dist.barrier() + (plain, _), (capped, buf) = results + where = f"{kind} opts {opts} bucket {maxt} rows {m} scale {scale} quant {quant}" + assert len(plain) == len(capped), where + for i, (a, b) in enumerate(zip(plain, capped)): + if i == 3: + written = sf_written(m, HIDDEN // 32, a.device) + a, b = a[written], b[written] + _assert_bitwise(b, a, f"{where}: output {i} with vs without cap_out") + ref = torch.empty((m, HIDDEN), dtype=torch.bfloat16, device="cuda") + _capture_reference(capped[1], cap_w, cap_eps, ref) + torch.cuda.synchronize() + _assert_bitwise(buf[:m, HIDDEN : 2 * HIDDEN], ref, f"{where}: capture vs capture_norm") + untouched = buf.clone() + untouched[:m, HIDDEN : 2 * HIDDEN] = 7.0 + assert bool((untouched == 7.0).all()), f"{where}: bytes outside the capture block changed" + dist.barrier() + + +def _cases(): + """(kind, rows, bucket, opts, scale, quant).""" + cases = [] + scales = (1.0, 1e3, 1e-3) + # The cluster kernel in the oneshot order (plain calls up to 8 rows): options 39 and 47. + for i, m in enumerate((1, 4, 8)): + for opts in (39, 47): + cases.append(("x" if i != 1 else "x2", m, 8, opts, scales[i], opts == 47)) + cases.append(("prepushed", 4, 8, 39, 1.0, False)) + # The row kernel: plain options 7 / 15 and the carveout instance (+ 64). + for i, (m, opts) in enumerate(((1, 7), (4, 15), (8, 7 | 64), (3, 15 | 64))): + cases.append(("x2" if i % 2 else "x", m, 8, opts, scales[i % 3], i == 1)) + cases.append(("prepushed", 8, 8, 7, 1e-3, False)) + cases.append(("x", 12, 16, 15, 1.0, True)) + cases.append(("x2", 12, 16, 15 | 64, 1e3, False)) + cases.append(("x", 32, 48, 15, 1e-3, False)) + return cases + + +def _run_matrix() -> None: + for n, (kind, m, maxt, opts, scale, quant) in enumerate(_cases()): + _one_case(kind, m, maxt, opts, scale, quant, 7000 + 37 * n) + if RANK == 0: + print( + f"[case] {kind} opts {opts} bucket {maxt} rows {m} scale {scale}: bitwise", + flush=True, + ) + + +def _run_one_rank() -> int: + global RANK, WORLD, allreduce_entry, ar_module + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(rank) + dist.init_process_group( + "cpu:gloo,cuda:nccl", init_method="env://", device_id=torch.device("cuda", rank) + ) + RANK, WORLD = rank, world + assert WORLD == 4, f"the tp4 boundary runs at 4 ranks, got {WORLD}" + try: + import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.m3_mc_allreduce as _ar # noqa: E501 + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.comm import ( # noqa: E501 + m3_mc_allreduce_norm as _plain, + ) + + allreduce_entry, ar_module = _plain, _ar + for maxt in BUCKETS: + WORKSPACES[maxt] = _make_workspace(maxt) + _run_matrix() + except BaseException: + import traceback # only the rank subprocess's failure path needs it + + print(f"[rank {RANK}] FAILED", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + dist.barrier() + torch.cuda.synchronize() + print(f"[rank {RANK}] passed", flush=True) + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + """One worker per claimed device, each in its own session; on the first nonzero exit or the deadline every + session is killed (as ``_m3_draft_ar_norm_op_matrix.py`` does).""" + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the 4 devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size == 4, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); the matrix runs at 4" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +if __name__ == "__main__": + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank()) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_draft_ar_norm_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_draft_ar_norm_op_matrix.py new file mode 100644 index 000000000000..e4841a190906 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_draft_ar_norm_op_matrix.py @@ -0,0 +1,533 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The fused draft loop's layer boundaries (``modeling.M3Eagle3Worker._m3_draft_ar_norm``) against stock Eagle3's, +bit for bit. + +Stock ``Eagle3DecoderLayer`` all-reduces each projection plainly (BF16 out) and runs the ``RMSNorm`` module with the +residual: FlashInfer's ``fused_add_rmsnorm``, which adds in FP32 and normalizes the FP32 sum. The draft loop's +boundaries must be that path exactly, under every all-reduce strategy the single-node drafter can run (AUTO, +ONESHOT, TWOSHOT, NCCL; MNNVL is multi-node only), at tp4 shapes (H = 6144, eps = 1e-6, 1 to 32 rows). The inputs are +random rows and rows built so a BF16 rounding of the residual sum would move the norm the most, including the +review's case: unit weights, sums 1 + 2^-8 on 6143 coordinates and 2 - 2^-8 on the last, both BF16 rounding +midpoints that round to even in opposite directions. + +The worker method runs with stand-ins for what it reads (the worker's ``mapping``; the drafter's ``w[norm_key]``, +``rms_eps``, ``allreduce`` and its own ``M3Eagle3Draft.ar_norm``), over the multicast workspace the target's +``m3_mc_allreduce.get_workspace`` returns (built here over the torch.distributed group, as the sibling multicast +matrices build it, and registered where ``get_workspace`` finds it). Two paths per strategy, case and row count: + +- ``boundary``: the method as the target runs it. At the rows ``DRAFT_MC_AR`` covers (1..``DRAFT_MC_AR_MAX_ROWS``) + the multicast all-reduce + residual + RMSNorm in its drafter form (row kernel, opts bit 16; the strategy is not + used); above them the built-in path, ``M3Eagle3Draft.ar_norm`` on the strategy's all-reduce. +- ``declined``: the method with ``get_workspace`` returning None (no multicast workspace), so the covered rows take + its fallback (``missed``, then ``M3Eagle3Draft.ar_norm`` on the strategy's all-reduce) as well. + +Which of the two ran is checked per call (the drafter-form multicast call returned rows exactly at the covered rows +of ``boundary``, never in ``declined``). Not driven: ``pushed=True`` (``DRAFT_AR_PUSH``, prepushed 2; it needs the +projection's split-K push) and ``DRAFT_MC_AR_CLUSTER`` (off; exact-class, not bitwise). + +Asserts the normalized rows and the residual sum bitwise equal to the stock path's, and reports the max ulp of each +against an fp64 reference (ulps of BF16 at the larger magnitude; a correctly rounded output is within 0.5). + +Its own 4-rank launcher, as ``_m3_mc_allreduce_norm_op_matrix.py`` is; the collected entry point is +``test_modeling_v2_m3_draft_ar_norm.py``. + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _m3_draft_ar_norm_op_matrix.py +""" + +import contextlib +import math +import os +import signal +import socket +import subprocess +import sys +import time +import types +from pathlib import Path +from typing import Any, Callable, Iterator, List, Optional, Tuple + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "the fused all-reduce + RMSNorm requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 + +HIDDEN = 6144 +EPS = 1e-6 +TOKENS = (1, 2, 3, 4, 5, 6, 7, 8, 9, 32) +"""Every row count of the drafter-form multicast boundary (1..8: its option table differs by row), the first one +past it and a wide one (the built-in path).""" +NORM_KEYS = ("post_norm", "final_norm") +"""The drafter's two boundaries per pass (o_proj -> post-attention norm, down -> final norm), alternated by row +count: the method reads only ``w[norm_key]``, so they differ in the weight's key alone.""" +CLUSTER = 8 +EMPTY_WORD = -2147483648 +RESULT_TAG = "[ulp]" +"""Prefix of rank 0's result lines; the pytest entry reports them.""" + +RANK = 0 +WORLD = 1 + +# Bound inside the rank body: importing the target pulls in tensorrt_llm. +modeling: Any = None +ar_module: Any = None + +Inputs = Tuple[List[torch.Tensor], torch.Tensor, torch.Tensor] +"""(every rank's [m, H] BF16 partial rows, the [m, H] BF16 residual, the [H] BF16 norm weight).""" + + +# ---------------------------------------------------------------- inputs (every rank builds every rank's rows) + + +def _gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def _random(seed: int, m: int, x_scale: float, r_scale: float, unit_w: bool) -> Inputs: + g = _gen(seed) + xs = [ + (torch.randn(m, HIDDEN, generator=g, device="cuda") * x_scale).bfloat16() + for _ in range(WORLD) + ] + residual = (torch.randn(m, HIDDEN, generator=g, device="cuda") * r_scale).bfloat16() + if unit_w: + w = torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16) + else: + w = (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).bfloat16() + return xs, residual, w + + +def _on_rank0(m: int, x: torch.Tensor) -> List[torch.Tensor]: + """``x`` as rank 0's partial, zeros elsewhere (the cross-rank sum is exact).""" + return [x.bfloat16()] + [ + torch.zeros(m, HIDDEN, device="cuda", dtype=torch.bfloat16) for _ in range(WORLD - 1) + ] + + +def _review_case(m: int) -> Inputs: + """Sums 1 + 2^-8 on the first H - 1 coordinates and 2 - 2^-8 on the last, unit weights. + + Both sums are BF16 midpoints: 1 + 2^-8 rounds (to even) down to 1 and 2 - 2^-8 up to 2, so the fused path's + norm sees a residual moved in opposite directions on the bulk and on the last coordinate. + """ + residual = torch.ones(m, HIDDEN, device="cuda") + residual[:, -1] = 2.0 - 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + return ( + _on_rank0(m, x), + residual.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _review_case_flipped(m: int) -> Inputs: + """The mirror: the bulk on odd-mantissa midpoints (1 + 3 * 2^-8 rounds up), the last one rounding down.""" + residual = torch.full((m, HIDDEN), 1.0 + 2.0**-7, device="cuda") + residual[:, -1] = 2.0 - 2.0 * 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + return ( + _on_rank0(m, x), + residual.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _midpoints(seed: int, m: int, outliers: int) -> Inputs: + """Every sum a BF16 midpoint: residual r random in [1, 2) and x = half its ulp (+2^-8), split over the ranks. + + The bulk has an odd mantissa (rounds up); ``outliers`` coordinates per row carry an even one (rounds down) and + a larger magnitude, so they weigh in the RMS while rounding the other way. + """ + g = _gen(seed) + mant = ( + torch.randint(0, 64, (m, HIDDEN), generator=g, device="cuda") * 2 + 1 + ) # odd 7-bit mantissa + r = 1.0 + mant.float() * 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + if outliers: + cols = torch.randperm(HIDDEN, generator=g, device="cuda")[:outliers] + even = torch.randint(0, 64, (m, outliers), generator=g, device="cuda") * 2 + r[:, cols] = 8.0 * (1.0 + even.float() * 2.0**-7) + x[:, cols] = 8.0 * 2.0**-8 + # Split x over the ranks in exact BF16 pieces: x/2 on rank 0, x/4 on ranks 1 and 2, 0 on rank 3. + parts = [x / 2, x / 4, x / 4] + [torch.zeros_like(x)] * (WORLD - 3) + return ( + [p.bfloat16() for p in parts[:WORLD]], + r.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _cancelling(seed: int, m: int) -> Inputs: + """Large partials that cancel across the ranks (+A, -A, +B, -B + d) onto a unit residual.""" + g = _gen(seed) + a = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 64).bfloat16().float() + b = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 64).bfloat16().float() + d = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 2.0**-6).bfloat16().float() + parts = [a, -a, b, (-b + d)] + residual = (1.0 + torch.rand(m, HIDDEN, generator=g, device="cuda")).bfloat16() + w = (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).bfloat16() + return [p.bfloat16() for p in parts[:WORLD]], residual, w + + +def _cases() -> List[Tuple[str, Callable[[int], Inputs]]]: + cases: List[Tuple[str, Callable[[int], Inputs]]] = [ + ("review: 1+2^-8 x6143, 2-2^-8 x1", _review_case), + ("review mirrored", _review_case_flipped), + ("all midpoints, no outliers", lambda m: _midpoints(11, m, 0)), + ("all midpoints, 1 outlier", lambda m: _midpoints(12, m, 1)), + ("all midpoints, 64 outliers", lambda m: _midpoints(13, m, 64)), + ("cancelling partials", lambda m: _cancelling(14, m)), + ] + for i, (xs, rs, unit) in enumerate( + ((1.0, 4.0, False), (1.0, 1.0, True), (0.05, 8.0, False), (8.0, 0.5, False)) + ): + cases.append( + ( + f"random x{xs} r{rs}{' unit-w' if unit else ''}", + lambda m, i=i, xs=xs, rs=rs, unit=unit: _random(100 + i, m, xs, rs, unit), + ) + ) + return cases + + +# ---------------------------------------------------------------- the three paths + + +def _fp64(xs: List[torch.Tensor], residual: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + s = residual.double() + for x in xs: + s = s + x.double() + rms = torch.rsqrt(s.square().mean(-1, keepdim=True) + EPS) + return s * rms * w.double() + + +def _ulp_at(v: torch.Tensor) -> torch.Tensor: + """BF16's ulp at |v| (fp64): 2^(e - 8) for |v| in [2^(e-1), 2^e), floored at the smallest normal's.""" + _, e = torch.frexp(v.abs().double()) + return torch.ldexp(torch.ones_like(v, dtype=torch.float64), torch.clamp(e - 8, min=-133)) + + +def _ulps(out: torch.Tensor, ref: torch.Tensor) -> float: + """Max |out - ref| in BF16 ulps at the larger magnitude of the two.""" + o, r = out.double(), ref.double() + return ((o - r).abs() / _ulp_at(torch.maximum(o.abs(), r.abs()))).max().item() + + +class _McCalls: + """Wraps ``m3_mc_allreduce.m3_mc_allreduce_norm`` (the method reads it off the module at each call): counts the + drafter-form calls that returned rows, i.e. the boundaries the multicast kernel ran.""" + + def __init__(self, fn: Callable) -> None: + self.fn = fn + self.ran = 0 + + def __call__(self, *args, **kwargs): + out = self.fn(*args, **kwargs) + if out is not None and kwargs.get("drafter_norm"): + self.ran += 1 + return out + + +@contextlib.contextmanager +def _no_workspace() -> Iterator[None]: + """``get_workspace`` returning None, as the target sees it with no multicast workspace: the method's decline.""" + real = ar_module.get_workspace + ar_module.get_workspace = lambda mapping, hidden: None + try: + yield + finally: + ar_module.get_workspace = real + + +def _stand_ins(mapping: Any, ar: Any, w: torch.Tensor) -> Tuple[Any, Any]: + """(worker, drafter) with what ``M3Eagle3Worker._m3_draft_ar_norm`` reads: the worker's ``mapping``; the + drafter's norm weights, ``rms_eps``, its all-reduce and its own ``ar_norm`` (the target's method, bound).""" + worker = types.SimpleNamespace(mapping=mapping) + draft = types.SimpleNamespace( + w={key: w for key in NORM_KEYS}, rms_eps=EPS, allreduce=ar, dtype=torch.bfloat16 + ) + draft.ar_norm = types.MethodType(modeling.M3Eagle3Draft.ar_norm, draft) + return worker, draft + + +def _target( + mapping: Any, + ar: Any, + x: torch.Tensor, + residual: torch.Tensor, + w: torch.Tensor, + norm_key: str, + mc: _McCalls, + declined: bool, +) -> Tuple[torch.Tensor, torch.Tensor, bool]: + """The draft loop's boundary as the target runs it (``M3Eagle3Worker._m3_draft_ar_norm``, ``pushed`` False); + with ``declined`` the multicast workspace is unavailable. Returns (normed, residual sum, whether the multicast + drafter form ran).""" + worker, draft = _stand_ins(mapping, ar, w) + before = mc.ran + with _no_workspace() if declined else contextlib.nullcontext(): + out, res = modeling.M3Eagle3Worker._m3_draft_ar_norm( + worker, draft, x, residual.clone(), norm_key + ) + return out, res, mc.ran > before + + +def _stock(ar: Any, x: torch.Tensor, residual: torch.Tensor, w: torch.Tensor): + """Stock Eagle3: ``o_proj``'s plain all-reduce (BF16 out), then ``RMSNorm(h, residual)``'s FlashInfer + ``fused_add_rmsnorm`` (FP32 add, norm of the FP32 sum; in place on both).""" + h = ar(x) + res = residual.clone() + torch.ops.trtllm.flashinfer_fused_add_rmsnorm(h, res, w, EPS) + return h, res + + +# ---------------------------------------------------------------- the matrix + + +def _single_node_cluster_info() -> None: + """``TorchDist`` (the communicator under ``TLLM_DISABLE_MPI``) asks Ray for each rank's node and GPU to build its + node-local group. These ranks are spawned without Ray on one node, so every rank reports the same node and its + own device.""" + from tensorrt_llm._torch.distributed.communicator import TorchDist + + def cluster_info(self): + if self.cluster_info is None: + self.cluster_info = [("127.0.0.1", r) for r in range(dist.get_world_size())] + return self.cluster_info + + TorchDist._get_cluster_info = cluster_info + + +def _register_workspace(mapping: Any) -> Any: + """The multicast workspace ``m3_mc_allreduce.get_workspace(mapping, HIDDEN)`` returns: one bucket per + ``BUCKETS`` entry, each built as ``McAllReduceWorkspace.create`` builds it but over the torch.distributed group + (these ranks have no MPI), with the 8-CTA call counters ``get_workspace`` creates; registered in its cache.""" + from tensorrt_llm._torch.distributed.ops import _make_mnnvl_mcast_buffer + + wss = [] + for maxt in ar_module.BUCKETS: + words = 4 * maxt * WORLD * (HIDDEN // 2) + handle = _make_mnnvl_mcast_buffer(dist.group.WORLD, words * 4, mapping, False) + uc = handle.get_uc_buffer(RANK, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + dist.barrier() + ws = ar_module.McAllReduceWorkspace( + uc=uc, + mc=mc, + flags={}, + rank=RANK, + world_size=WORLD, + hidden=HIDDEN, + handle=handle, + comm=dist.group.WORLD, + max_tokens=maxt, + ) + ws.flags_for(CLUSTER) + wss.append(ws) + buckets = ar_module.McBuckets(wss) + ar_module._WORKSPACE[HIDDEN] = buckets + assert ar_module.get_workspace(mapping, HIDDEN) is buckets + return buckets + + +def _run_matrix() -> None: + import tensorrt_llm._torch.custom_ops # noqa: F401 -- registers trtllm::flashinfer_fused_add_rmsnorm + from tensorrt_llm._torch.distributed import AllReduce + from tensorrt_llm._torch.flashinfer_utils import IS_FLASHINFER_AVAILABLE + from tensorrt_llm.functional import AllReduceStrategy + from tensorrt_llm.mapping import Mapping + + assert IS_FLASHINFER_AVAILABLE, "stock Eagle3's add + RMSNorm is FlashInfer's fused_add_rmsnorm" + _single_node_cluster_info() + mapping = Mapping(world_size=WORLD, tp_size=WORLD, rank=RANK, gpus_per_node=WORLD) + # The covered rows are 1..DRAFT_MC_AR_MAX_ROWS, every one of them driven; the row kernel's drafter form, the one + # the bitwise claim is about (the cluster form is exact-class). + covered = set(modeling.DRAFT_MC_AR.keys) + assert covered == set(range(1, modeling.DRAFT_MC_AR_MAX_ROWS + 1)), sorted(covered) + assert covered <= set(TOKENS), (sorted(covered), TOKENS) + assert any(m not in covered for m in TOKENS), ( + "drive at least one row count past the multicast boundary" + ) + assert not modeling.DRAFT_MC_AR_CLUSTER, ( + "the cluster drafter form is exact-class, not bitwise stock" + ) + # A declined multicast boundary logs (``missed``) and falls back instead of raising. + modeling.set_require(False) + _register_workspace(mapping) + mc = _McCalls(ar_module.m3_mc_allreduce_norm) + ar_module.m3_mc_allreduce_norm = mc + if RANK == 0: + print( + f"{RESULT_TAG} multicast drafter form at rows {sorted(covered)} (DRAFT_MC_AR_MAX_ROWS" + f" {modeling.DRAFT_MC_AR_MAX_ROWS}); built-in path at the others and when declined", + flush=True, + ) + strategies = ( + AllReduceStrategy.AUTO, + AllReduceStrategy.ONESHOT, + AllReduceStrategy.TWOSHOT, + AllReduceStrategy.NCCL, + ) + worst = 0.0 + mismatches: List[str] = [] + # The multicast drafter kernel takes no strategy: it sums the ranks in FP32 in rank order, which is stock's ONESHOT + # (the strategy a served drafter resolves AUTO to). Under NCCL its rows are therefore compared with stock ONESHOT; + # NCCL itself (which can round cancelling partials differently) stays the reference of the built-in rows. + oneshot = AllReduce(mapping=mapping, strategy=AllReduceStrategy.ONESHOT, dtype=torch.bfloat16) + for strategy in strategies: + ar = AllReduce(mapping=mapping, strategy=strategy, dtype=torch.bfloat16) + assert ar.mnnvl_allreduce is None and ar.symm_mem_allreduce is None, strategy + for name, build in _cases(): + for declined in (False, True): + path = "declined" if declined else "boundary" + ulp, res_ulp = 0.0, 0.0 + for i, m in enumerate(TOKENS): + xs, residual, w = build(m) + target_out, target_res, ran_mc = _target( + mapping, ar, xs[RANK], residual, w, NORM_KEYS[i % 2], mc, declined + ) + expect_mc = not declined and m in covered + stock_ar = oneshot if expect_mc and strategy == AllReduceStrategy.NCCL else ar + stock_out, stock_res = _stock(stock_ar, xs[RANK], residual, w) + torch.cuda.synchronize() + assert ran_mc == expect_mc, ( + f"{strategy.name} {path} {name} m={m}: the multicast drafter form " + f"{'ran' if ran_mc else 'did not run'}" + ) + for what, a, b in ( + ("normed", target_out, stock_out), + ("residual", target_res, stock_res), + ): + bad = int((a.view(torch.int16) != b.view(torch.int16)).sum().item()) + if bad: + mismatches.append( + f"{strategy.name} {path} ({'multicast' if ran_mc else 'built-in'}) {name}" + f" m={m}: {bad} {what} elements differ from stock" + ) + ref = _fp64(xs, residual, w) + ref_res = residual.double() + sum(x.double() for x in xs) + u, ur = _ulps(target_out, ref), _ulps(target_res, ref_res) + assert math.isfinite(u) and math.isfinite(ur), (strategy, path, name, m) + ulp, res_ulp = max(ulp, u), max(res_ulp, ur) + dist.barrier() + worst = max(worst, ulp) + if RANK == 0: + print( + f"{RESULT_TAG} {strategy.name:7s} | {path:8s} | {name:34s} | target = stock vs fp64:" + f" normed {ulp:.3f} | residual {res_ulp:.3f}", + flush=True, + ) + if RANK == 0: + scope = f"all cases, m in {TOKENS}" + print( + f"{RESULT_TAG} MAX | both | {scope:34s} | target = stock vs fp64: normed {worst:.3f}", + flush=True, + ) + assert not mismatches, "\n".join(mismatches) + + +def _run_one_rank() -> int: + global RANK, WORLD, modeling, ar_module + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(rank) + dist.init_process_group( + "cpu:gloo,cuda:nccl", init_method="env://", device_id=torch.device("cuda", rank) + ) + RANK, WORLD = rank, world + assert WORLD == 4, f"the drafter runs at tp4, got {WORLD} ranks" + try: + import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.m3_mc_allreduce as _ar # noqa: E501 + import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.modeling as _modeling # noqa: E501 + + modeling, ar_module = _modeling, _ar + assert modeling.m3_mc_allreduce is ar_module, ( + "the method calls the kernels module this patches" + ) + _run_matrix() + except BaseException: + import traceback # only the rank subprocess's failure path needs it + + print(f"[rank {RANK}] FAILED", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + dist.barrier() + torch.cuda.synchronize() + print(f"[rank {RANK}] passed", flush=True) + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + """One worker per claimed device, each in its own session; on the first nonzero exit or the deadline every + session is killed (as ``_m3_mc_allreduce_norm_op_matrix.py`` does).""" + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the 4 devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size == 4, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); the matrix runs at 4" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +if __name__ == "__main__": + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank()) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_mc_allreduce_norm_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_mc_allreduce_norm_op_matrix.py new file mode 100644 index 000000000000..f1962a878faf --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_m3_mc_allreduce_norm_op_matrix.py @@ -0,0 +1,665 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU certification matrix for ``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm``, 4 ranks. + +Its own launcher, as the catalog's collective matrices are: run plainly it spawns +one rank process per device in CUDA_VISIBLE_DEVICES and kills them all on the +first failure or on the deadline. A rank whose peers never push polls forever, +so the deadline is what keeps a broken exchange from wedging the calling run. + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _m3_mc_allreduce_norm_op_matrix.py + +Each rank builds the three multicast workspaces (8 / 16 / 48 tokens) the target +builds, over the torch.distributed group (the non-MPI path of +``_make_mnnvl_mcast_buffer``). Every rank can regenerate every rank's inputs from +the shared seed convention, which is what keeps each reference arithmetic rather +than a second collective. Producer pushes (``prepushed`` 2 / 3) are emulated by +storing the canonicalized rows through the multicast mapping at the call parity +the flags select, which is exactly what the producing kernels store. + +Not a pytest module, despite the ``check_*`` bodies: they are one fixed sequence +inside a single 4-rank job whose workspace state (call parities) carries from +one to the next. The collected entry point is +``test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py``. +""" + +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Dict, List, Optional + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "m3_mc_allreduce_norm requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 + +HIDDEN = 6144 +EPS = 1e-6 +CLUSTER = 8 +BUCKETS = (8, 16, 48) +EMPTY_WORD = -2147483648 + +# Bound inside the rank body: importing the entries pulls in tensorrt_llm. +allreduce_entry: Any = None +finalize_entry: Any = None +ar_module: Any = None +validating: Any = None +RANK = 0 +WORLD = 1 +WORKSPACES: Dict[int, Any] = {} + + +# ---------------------------------------------------------------- inputs + + +def _gen(seed: int, rank: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed * 1009 + rank + 1) + + +def _rows(seed: int, rank: int, m: int, scale: float = 1.0) -> torch.Tensor: + """One rank's [m, H] BF16 rows; ``rank`` -1 for tensors every rank holds alike.""" + g = _gen(seed, rank + 1) + return (torch.randn(m, HIDDEN, generator=g, device="cuda") * scale).to(torch.bfloat16) + + +def _residual(seed: int, m: int) -> torch.Tensor: + return _rows(seed + 7, -1, m, 4.0) + + +def _norm_w(seed: int) -> torch.Tensor: + g = _gen(seed + 11, 0) + return (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).to(torch.bfloat16) + + +def _finalize_inputs(seed: int, rank: int, m: int, topk: int, dropped: int): + """One rank's unfinalized MoE outputs: fc2 [m * topk, H], perm [m, topk], fp32 scales, shared [m, H].""" + g = _gen(seed + 23, rank) + rows = m * topk + fc2 = (torch.randn(rows, HIDDEN, generator=g, device="cuda") * 0.5).to(torch.bfloat16) + perm = torch.randperm(rows, generator=g, device="cuda").to(torch.int32) + if dropped: + drop = torch.randperm(rows, generator=g, device="cuda")[:dropped] + perm[drop] = -1 + perm = perm.view(m, topk).contiguous() + scales = torch.rand(m, topk, generator=g, device="cuda", dtype=torch.float32) + shared = _rows(seed + 29, rank, m, 0.5) + return fc2, perm, scales, shared + + +# ---------------------------------------------------------------- workspace / producer emulation + + +def _make_workspace(maxt: int): + """One bucket's exchange, as ``McAllReduceWorkspace.create`` builds it, over the torch.distributed group.""" + from tensorrt_llm._torch.distributed.ops import _make_mnnvl_mcast_buffer + from tensorrt_llm.mapping import Mapping + + mapping = Mapping(world_size=WORLD, tp_size=WORLD, rank=RANK) + words = 4 * maxt * WORLD * (HIDDEN // 2) + group = dist.group.WORLD + handle = _make_mnnvl_mcast_buffer(group, words * 4, mapping, False) + uc = handle.get_uc_buffer(RANK, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + dist.barrier() + ws = ar_module.McAllReduceWorkspace( + uc=uc, + mc=mc, + flags={}, + rank=RANK, + world_size=WORLD, + hidden=HIDDEN, + handle=handle, + comm=group, + max_tokens=maxt, + ) + ws.flags_for(CLUSTER) + return ws + + +def _push(ws, rows: torch.Tensor, exchange_set: int) -> None: + """Store this rank's rows into every rank's exchange, as a producer kernel does before a prepushed call. + + Slot of (token t, cluster CTA c): set * 2 * half + parity(t, c) * half + (t * world + rank) * H / 2 + c * H / 16 + words, parity read from the CTA's flag; -0.0 halves are stored as +0.0 (0x80000000 is the empty word). + """ + m = rows.shape[0] + torch.cuda.synchronize() + parity = (ws.flags_for(CLUSTER).view(ws.max_tokens, CLUSTER)[:m] & 1).long() + half = ws.max_tokens * WORLD * (HIDDEN // 2) + chunk = HIDDEN // 2 // CLUSTER + h16 = rows.contiguous().view(torch.int16) + h16 = torch.where(h16 == -32768, torch.zeros_like(h16), h16) + words = h16.view(torch.int32).view(m, CLUSTER, chunk) + t = torch.arange(m, device="cuda").view(m, 1, 1) + c = torch.arange(CLUSTER, device="cuda").view(1, CLUSTER, 1) + w = torch.arange(chunk, device="cuda").view(1, 1, chunk) + index = ( + exchange_set * 2 * half + + parity.view(m, CLUSTER, 1) * half + + (t * WORLD + RANK) * (HIDDEN // 2) + + c * chunk + + w + ) + ws.mc[index.reshape(-1)] = words.reshape(-1) + torch.cuda.synchronize() + dist.barrier() + + +# ---------------------------------------------------------------- gates + + +def _assert_bitwise(out: torch.Tensor, ref: torch.Tensor, where: str) -> None: + assert out.dtype == ref.dtype and out.shape == ref.shape, ( + where, + out.dtype, + ref.dtype, + out.shape, + ) + a, b = out.contiguous().view(torch.uint8), ref.contiguous().view(torch.uint8) + bad = (a != b).sum().item() + assert bad == 0, f"{where}: {bad} of {a.numel()} bytes differ" + + +#: Share of normed elements one BF16 step off the float64 reference: measured at most 1.1e-4 per call over +#: every call of this matrix on GB300 (see ``_assert_normed``). The gate sits ~18x above that and far below what a +#: wrong statistic produces (most elements move). +NORMED_STEP_FRACTION = 0.002 +NORMED_STEP_SEEN: List[float] = [] + + +def _assert_normed(out: torch.Tensor, ref: torch.Tensor, where: str) -> None: + """At most one BF16 rounding step from the float64 reference, and rarely that. + + The kernel's RMS statistic is a shuffle / warp / cluster tree of FP32 partial sums and its ``rsqrtf`` is + approximate (~2^-22 relative): the scaled value moves by ~1e-6 relative, which changes the BF16 rounding only + for values within that distance of a rounding midpoint -- one step, never two. The fraction bound catches a + statistic that is systematically off (a missing rank, a wrong H), which moves most elements. + """ + assert out.dtype == ref.dtype == torch.bfloat16 and out.shape == ref.shape, where + o, r = out.float(), ref.float() + _, exponent = torch.frexp(torch.maximum(o.abs(), r.abs())) + step = torch.ldexp(torch.ones_like(o), exponent - 8) + diff = (o - r).abs() + assert bool((diff <= step).all()), f"{where}: max diff {diff.max().item()} beyond one BF16 step" + frac = (diff != 0).float().mean().item() + NORMED_STEP_SEEN.append(frac) + assert frac <= NORMED_STEP_FRACTION, f"{where}: {frac:.4f} of normed elements off by one step" + + +def _assert_quant(outs, ref, m: int, where: str) -> None: + """The MXFP8 copy: bit-exact against the recipe applied to the op's own normed rows (a pure function of + them), and equal to the reference's wherever the reference's normed 32-column block is the op's.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.comm.m3_mc_allreduce_norm import ( # noqa: E501 + mxfp8_rows, + sf_written, + ) + + q, sf = outs[2], outs[3] + own_q, own_sf = mxfp8_rows(outs[0]) + written = sf_written(m, HIDDEN // 32, sf.device) + _assert_bitwise(q, own_q, f"{where} e4m3 rows") + _assert_bitwise(sf[written], own_sf[written], f"{where} UE8M0 scales") + same_block = (outs[0].view(m, -1, 32) == ref[0].view(m, -1, 32)).all(-1) + qb = q.view(torch.uint8).view(m, -1, 32) + rqb = ref[2].view(torch.uint8).view(m, -1, 32) + assert bool((qb == rqb).all(-1)[same_block].all()), ( + f"{where}: e4m3 differs on a block whose normed rows agree" + ) + + +def _check_outputs(outs, ref, m: int, quant: bool, where: str) -> None: + assert len(outs) == (4 if quant else 2), (where, len(outs)) + _assert_bitwise(outs[1], ref[1], f"{where} updated residual") + _assert_normed(outs[0], ref[0], f"{where} normed") + if quant: + _assert_quant(outs, ref, m, where) + + +# ---------------------------------------------------------------- calls + + +def _call_plain(ws, x, x2, residual, norm_w, quant, prepushed, opts): + return allreduce_entry.m3_mc_allreduce_norm( + x, + x2, + residual, + norm_w, + EPS, + ws.uc, + ws.mc.data_ptr(), + ws.flags_for(CLUSTER), + RANK, + WORLD, + CLUSTER, + False, + ws.max_tokens, + quant, + prepushed, + opts, + ) + + +def _plain_case(spec: Dict[str, Any], seed: int, where: str) -> None: + m, maxt, quant, prepushed = spec["m"], spec["maxt"], spec["quant"], spec["prepushed"] + ws = WORKSPACES[maxt] + xs = [_rows(seed, r, m) for r in range(WORLD)] + x2s = [_rows(seed + 3, r, m) for r in range(WORLD)] if spec["x2"] else None + residual, norm_w = _residual(seed, m), _norm_w(seed) + x, x2 = xs[RANK], (None if x2s is None else x2s[RANK]) + if prepushed >= 2: + _push(ws, x, 0) + if prepushed == 3: + _push(ws, x2, 1) + with validating(allreduce_entry.m3_mc_allreduce_norm): + outs = _call_plain(ws, x, x2, residual, norm_w, quant, prepushed, spec["opts"]) + torch.cuda.synchronize() + ref = allreduce_entry.m3_mc_allreduce_norm.reference( + xs, x2s, residual, norm_w, EPS, quant, spec.get("drafter", False) + ) + _check_outputs(outs, ref, m, quant, where) + dist.barrier() + + +# ---------------------------------------------------------------- checks + + +def check_plain_cells() -> None: + """Every cell of ``m3_mc_allreduce_norm.CELLS`` through ``__call__`` under ``validating``, gated on its + ``reference``.""" + for i, cell in enumerate(allreduce_entry.m3_mc_allreduce_norm.CELLS): + _plain_case(cell.spec, 1000 + 17 * i, cell.why) + + +def check_finalize_cells() -> None: + """Every cell of ``m3_mc_finalize_allreduce_norm.CELLS``, gated on its ``reference``.""" + op = finalize_entry.m3_mc_finalize_allreduce_norm + for i, cell in enumerate(op.CELLS): + spec = cell.spec + m, maxt, quant, topk = spec["m"], spec["maxt"], spec["quant"], spec["topk"] + seed = 2000 + 17 * i + ws = WORKSPACES[maxt] + per_rank = [_finalize_inputs(seed, r, m, topk, spec["dropped"]) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + fc2, perm, scales, shared = per_rank[RANK] + with validating(op): + outs = op( + fc2, + perm, + scales, + shared, + residual, + norm_w, + EPS, + ws.uc, + ws.mc.data_ptr(), + ws.flags_for(CLUSTER), + RANK, + WORLD, + CLUSTER, + False, + maxt, + quant, + spec["opts"], + ) + torch.cuda.synchronize() + ref = op.reference( + [p[0] for p in per_rank], + [p[1] for p in per_rank], + [p[2] for p in per_rank], + [p[3] for p in per_rank], + residual, + norm_w, + EPS, + quant, + ) + _check_outputs(outs, ref, m, quant, cell.why) + dist.barrier() + + +def check_the_target_wrappers_select_bucket_and_options() -> None: + """The target's own wrappers (bucket by token count, option table) on top of the entries, every token count of + the 8-token bucket plus one of each larger bucket, and the finalize wrapper; the MXFP8 publication is the + prequant slot's business and is left off.""" + buckets = ar_module.McBuckets([WORKSPACES[b] for b in BUCKETS]) + for m in (*range(1, 9), 11, 40): + seed = 3000 + m + xs = [_rows(seed, r, m) for r in range(WORLD)] + x2s = [_rows(seed + 3, r, m) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + normed, updated = ar_module.m3_mc_allreduce_norm( + xs[RANK], residual, norm_w, EPS, buckets, x2=x2s[RANK] + ) + torch.cuda.synchronize() + ref = allreduce_entry.m3_mc_allreduce_norm.reference(xs, x2s, residual, norm_w, EPS, False) + _check_outputs([normed, updated], ref, m, False, f"wrapper m={m}") + dist.barrier() + for m in (9, 33): + seed = 3100 + m + per_rank = [_finalize_inputs(seed, r, m, 4, 2) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + fc2, perm, scales, shared = per_rank[RANK] + normed, updated = ar_module.m3_mc_finalize_allreduce_norm( + fc2, perm, scales, shared, residual, norm_w, EPS, buckets + ) + torch.cuda.synchronize() + ref = finalize_entry.m3_mc_finalize_allreduce_norm.reference( + *[[p[j] for p in per_rank] for j in range(4)], residual, norm_w, EPS, False + ) + _check_outputs([normed, updated], ref, m, False, f"finalize wrapper m={m}") + dist.barrier() + + +def check_cuda_graph_replays() -> None: + """Two boundaries captured in one graph (8- and 16-token buckets), replayed with fresh inputs. + + The decode graph holds every boundary call; each replay rotates the same per-CTA parities the eager calls do, + so a replay that reused a stale half or skipped a re-arm reads the previous replay's rows and fails the + bitwise residual gate (every replay gets new inputs). + """ + shapes = ((3, 8, 15), (12, 16, 15)) + norm_w = _norm_w(4000) + bufs = [] + for m, maxt, opts in shapes: + bufs.append( + dict( + m=m, + maxt=maxt, + opts=opts, + x=torch.empty(m, HIDDEN, dtype=torch.bfloat16, device="cuda"), + x2=torch.empty(m, HIDDEN, dtype=torch.bfloat16, device="cuda"), + residual=torch.empty(m, HIDDEN, dtype=torch.bfloat16, device="cuda"), + ) + ) + + def body(): + return [ + _call_plain( + WORKSPACES[b["maxt"]], b["x"], b["x2"], b["residual"], norm_w, False, 0, b["opts"] + ) + for b in bufs + ] + + def fill(seed: int): + refs = [] + for j, b in enumerate(bufs): + s = seed + 31 * j + xs = [_rows(s, r, b["m"]) for r in range(WORLD)] + x2s = [_rows(s + 3, r, b["m"]) for r in range(WORLD)] + b["x"].copy_(xs[RANK]) + b["x2"].copy_(x2s[RANK]) + b["residual"].copy_(_residual(s, b["m"])) + refs.append( + allreduce_entry.m3_mc_allreduce_norm.reference( + xs, x2s, b["residual"].clone(), norm_w, EPS, False + ) + ) + torch.cuda.synchronize() + dist.barrier() + return refs + + refs = fill(4100) + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + warm = body() + torch.cuda.current_stream().wait_stream(side) + torch.cuda.synchronize() + for j, b in enumerate(bufs): + _check_outputs(warm[j], refs[j], b["m"], False, f"graph warm-up m={b['m']}") + dist.barrier() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = body() + for rep in range(4): + refs = fill(4200 + 100 * rep) + graph.replay() + torch.cuda.synchronize() + for j, b in enumerate(bufs): + _check_outputs(captured[j], refs[j], b["m"], False, f"replay {rep} m={b['m']}") + dist.barrier() + del graph + dist.barrier() + + +def check_the_gate_sees_a_missing_rank() -> None: + """The residual gate is not blind to the reduction: dropping any one rank's partial from the reference + moves most elements of the updated residual (each rank's rows are distinct by construction).""" + m, seed = 4, 5000 + xs = [_rows(seed, r, m) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + full = allreduce_entry.m3_mc_allreduce_norm.reference(xs, None, residual, norm_w, EPS, False) + for r in range(WORLD): + partial = [x if i != r else torch.zeros_like(x) for i, x in enumerate(xs)] + wrong = allreduce_entry.m3_mc_allreduce_norm.reference( + partial, None, residual, norm_w, EPS, False + ) + frac = (wrong[1] != full[1]).float().mean().item() + assert frac > 0.9, f"dropping rank {r} changes only {frac:.3f} of the residual" + dist.barrier() + + +def _expect_rejected(fn, match: str, where: str) -> None: + try: + fn() + except AssertionError as exc: + assert match in str(exc), f"{where}: rejected for the wrong reason: {exc}" + else: + raise AssertionError(f"{where}: the guard admitted it") + + +def check_guards() -> None: + """``is_valid`` refuses what the op accepts and answers wrongly. Nothing here reaches the kernel: the guard + raises first, on every rank alike, so no rank is left polling.""" + m, seed = 4, 6000 + ws8, ws16 = WORKSPACES[8], WORKSPACES[16] + x, residual, norm_w = _rows(seed, RANK, m), _residual(seed, m), _norm_w(seed) + op = allreduce_entry.m3_mc_allreduce_norm + + def call(ws_uc, flags, norm): + with validating(op): + op( + x, + None, + residual, + norm, + EPS, + ws_uc, + ws8.mc.data_ptr(), + flags, + RANK, + WORLD, + CLUSTER, + False, + 8, + False, + 0, + 15, + ) + + strided = torch.empty(2 * HIDDEN, dtype=torch.bfloat16, device="cuda")[::2] + strided.copy_(norm_w) + _expect_rejected( + lambda: call(ws8.uc, ws8.flags_for(CLUSTER), strided), "contiguous", "strided norm_w" + ) + _expect_rejected( + lambda: call(ws8.uc[: ws8.uc.numel() // 2], ws8.flags_for(CLUSTER), norm_w), + "words", + "short workspace", + ) + _expect_rejected( + lambda: call(ws8.uc, ws16.flags_for(CLUSTER), norm_w), "counters", "another bucket's flags" + ) + + fin = finalize_entry.m3_mc_finalize_allreduce_norm + fc2, perm, scales, shared = _finalize_inputs(seed, RANK, 9, 4, 0) + bad = perm.clone() + bad[0, 0] = fc2.shape[0] + + def call_fin(): + with validating(fin): + fin( + fc2, + bad, + scales, + shared, + _residual(seed, 9), + norm_w, + EPS, + ws16.uc, + ws16.mc.data_ptr(), + ws16.flags_for(CLUSTER), + RANK, + WORLD, + CLUSTER, + False, + 16, + False, + 15, + ) + + _expect_rejected(call_fin, "perm spans", "perm past the FC2 rows") + dist.barrier() + + +CHECKS = ( + check_plain_cells, + check_finalize_cells, + check_the_target_wrappers_select_bucket_and_options, + check_cuda_graph_replays, + check_the_gate_sees_a_missing_rank, + check_guards, +) + + +def _run_one_rank() -> int: + """Body of one spawned rank: run every check; a failing rank exits nonzero and the launcher kills its + siblings.""" + global RANK, WORLD, allreduce_entry, finalize_entry, ar_module, validating + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(rank) + dist.init_process_group( + "cpu:gloo,cuda:nccl", init_method="env://", device_id=torch.device("cuda", rank) + ) + RANK, WORLD = rank, world + assert WORLD == 4, f"the entries are certified at 4 ranks, got {WORLD}" + + import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.m3_mc_allreduce as _ar # noqa: E501 + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.comm import ( # noqa: E501 + m3_mc_allreduce_norm as _plain, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.comm import ( # noqa: E501 + m3_mc_finalize_allreduce_norm as _finalize, + ) + + allreduce_entry, finalize_entry, ar_module = _plain, _finalize, _ar + # Spawned by file path, not collected by pytest: reach the sibling helper the plain way (confined to the + # rank processes). /comm -> tests/unittest/_torch/modeling_v2. + sys.path.insert(0, str(Path(__file__).resolve().parents[4])) + from _validating import validating as _validating_cm + + validating = _validating_cm + for maxt in BUCKETS: + WORKSPACES[maxt] = _make_workspace(maxt) + + for check in CHECKS: + try: + check() + except BaseException: + import traceback # only the rank subprocess's failure path needs it + + print(f"[rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + # A rank that leaves early wedges its peers' polls; the launcher kills them on this exit. + os._exit(13) + print(f"[rank {RANK}] {check.__name__} passed", flush=True) + dist.barrier() + torch.cuda.synchronize() + print( + f"[rank {RANK}] {len(CHECKS)} checks passed; normed one-step fraction max " + f"{max(NORMED_STEP_SEEN):.5f} over {len(NORMED_STEP_SEEN)} calls", + flush=True, + ) + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + """One worker process per claimed device, each in its own session; on the first nonzero exit or on the + deadline every session is killed.""" + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the 4 devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size == 4, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); the matrix runs at 4" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + # Drop a launching srun step's PMIx / MPI namespace: children registered in it get the whole step SIGKILLed + # when the first one exits hard. These ranks rendezvous through torch.distributed's env://. + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + print(f"[launcher] spawning {world_size} ranks, rendezvous on :{port}", flush=True) + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +if __name__ == "__main__": + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank()) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_rank_job.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_rank_job.py new file mode 100644 index 000000000000..1115ff3b5976 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/_rank_job.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Run this target's collective rank job from pytest, and report what it did. + +The same contract as ``tests/unittest/_torch/modeling_v2/comm/_rank_job.py``: the +launcher beside this file owns the per-device rank processes and the deadline +that kills a wedged collective; this only selects the devices, starts it in a +fresh interpreter (by file path: this tree is not a package) and turns its exit +code into an assertion. +""" + +from __future__ import annotations + +import importlib.util +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +WORLD_SIZE = 4 +"""tp4's world size -- the topology these entries are certified for.""" + +_LAUNCHER_GRACE_S = 300 +"""Headroom over the launcher's own deadline, so its message wins the race.""" + + +def _devices() -> str: + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + if visible: + devices = [d for d in visible.split(",") if d.strip()] + else: + import torch + + devices = [str(i) for i in range(torch.cuda.device_count())] + if len(devices) < WORLD_SIZE: + pytest.skip( + f"this entry is certified at world size {WORLD_SIZE}; only {len(devices)} device(s) are visible" + ) + return ",".join(devices[:WORLD_SIZE]) + + +def _load_launcher_constants(launcher: Path): + """Read the launcher's module-level budget without importing it by name (it imports only torch at + module scope).""" + spec = importlib.util.spec_from_file_location(launcher.stem, launcher) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def run(entry: str) -> str: + """Run ``__op_matrix``'s launcher over ``WORLD_SIZE`` devices; returns its stdout.""" + launcher = Path(__file__).resolve().with_name(f"_{entry}_op_matrix.py") + env = dict(os.environ, CUDA_VISIBLE_DEVICES=_devices()) + + # The ranks import tensorrt_llm absolutely. From a source checkout that is not installed, put the repo root + # (tests/unittest/_torch/modeling_v2/models/minimax_m3//comm -> repo root) on their path; from a tests + # tree run against an installed package, they import the installed one. + repo_root = Path(__file__).resolve().parents[8] + if (repo_root / "tensorrt_llm").is_dir(): + env["PYTHONPATH"] = os.pathsep.join( + p for p in (str(repo_root), env.get("PYTHONPATH", "")) if p + ) + else: + assert importlib.util.find_spec("tensorrt_llm") is not None, ( + f"no tensorrt_llm/ at the repo root {repo_root} and no installed tensorrt_llm" + ) + + constants = _load_launcher_constants(launcher) + timeout = constants.DEADLINE_S + _LAUNCHER_GRACE_S + + completed = subprocess.run( + [sys.executable, str(launcher)], + env=env, + capture_output=True, + text=True, + timeout=timeout, + ) + if completed.returncode != 0: + raise AssertionError( + f"{launcher.name} exited {completed.returncode}\n" + f"--- stdout ---\n{completed.stdout}\n" + f"--- stderr ---\n{completed.stderr}" + ) + return completed.stdout diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_allreduce.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_allreduce.py new file mode 100644 index 000000000000..27d45f08edc4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_allreduce.py @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for the ``m3_allreduce`` certification matrix. + +The matrix is ``_m3_allreduce_op_matrix.py`` beside this file, its own 4-rank launcher; see ``_rank_job``. It drives +every cell of the entry against its reference, then requires the fused draft loop's boundaries built on it (plain +all-reduce, then the fused add + RMSNorm) to equal stock Eagle3's bit for bit under every strategy, and reports both +against an fp64 reference as a warning, one line per strategy and case. +""" + +import warnings + +import _rank_job +import pytest +import torch + +pytestmark = [ + pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", + ), + pytest.mark.skipif( + torch.cuda.device_count() < _rank_job.WORLD_SIZE, + reason=f"certified at world size {_rank_job.WORLD_SIZE}", + ), +] + +_RESULT_TAG = "[ulp]" + + +# Spawns 4 rank processes over the visible devices; under xdist workers would fight for them. +@pytest.mark.no_xdist +def test_m3_allreduce_op_matrix() -> None: + out = _rank_job.run("m3_allreduce") + lines = [line for line in out.splitlines() if line.startswith(_RESULT_TAG)] + assert lines, f"the launcher reported no measurements:\n{out}" + warnings.warn( + "draft boundary (= stock Eagle3) vs fp64, max BF16 ulps\n" + "\n".join(lines), stacklevel=1 + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_capture_in_boundary.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_capture_in_boundary.py new file mode 100644 index 000000000000..ad7539d27829 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_capture_in_boundary.py @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for CAPTURE_NORM_IN_BOUNDARY on the tp4 multicast all-reduce + RMSNorm op. + +The matrix is ``_m3_capture_in_boundary_op_matrix.py`` beside this file, its own 4-rank launcher; see ``_rank_job``. +It requires the boundary's capture output to equal ``m3_spec_tail.capture_norm`` of its updated residual bit for bit, +on both kernels the tp4 op runs (the oneshot cluster kernel, the row kernel), and the op's other outputs to be +unchanged by it. +""" + +import _rank_job +import pytest +import torch + +assert torch.cuda.is_available(), "the multicast all-reduce requires CUDA devices" + + +# Spawns 4 rank processes over the visible devices; under xdist workers would fight for them. +@pytest.mark.no_xdist +def test_m3_capture_norm_in_boundary_is_capture_norm() -> None: + out = _rank_job.run("m3_capture_in_boundary") + assert "[case]" in out, f"the launcher ran no case:\n{out}" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_draft_ar_norm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_draft_ar_norm.py new file mode 100644 index 000000000000..3cc1592bc3c5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_draft_ar_norm.py @@ -0,0 +1,32 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for the fast draft loop's layer boundaries against stock Eagle3's. + +The matrix is ``_m3_draft_ar_norm_op_matrix.py`` beside this file, its own 4-rank launcher; see ``_rank_job``. It +requires the target's drafter boundary (``modeling.M3Eagle3Worker._m3_draft_ar_norm``: the multicast all-reduce + +residual + RMSNorm in its drafter form at the rows ``DRAFT_MC_AR`` covers, else and when that declines +``M3Eagle3Draft.ar_norm`` on the strategy's all-reduce) to equal stock Eagle3's boundary (plain all-reduce, then the +``RMSNorm`` module with the residual) bit for bit under every strategy, and reports both against an fp64 reference as a +warning, one line per strategy, path and case. +""" + +import warnings + +import _rank_job +import pytest +import torch + +assert torch.cuda.is_available(), "the fused all-reduce + RMSNorm requires CUDA devices" + +_RESULT_TAG = "[ulp]" + + +# Spawns 4 rank processes over the visible devices; under xdist workers would fight for them. +@pytest.mark.no_xdist +def test_m3_draft_ar_norm_is_stock_eagle3() -> None: + out = _rank_job.run("m3_draft_ar_norm") + lines = [line for line in out.splitlines() if line.startswith(_RESULT_TAG)] + assert lines, f"the launcher reported no measurements:\n{out}" + warnings.warn( + "draft boundary (= stock Eagle3) vs fp64, max BF16 ulps\n" + "\n".join(lines), stacklevel=1 + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py new file mode 100644 index 000000000000..31a054642e89 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for the multicast all-reduce + RMSNorm entries' certification matrix. + +Covers both ``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm`` (one +kernel, one exchange protocol). The matrix is ``_m3_mc_allreduce_norm_op_matrix.py`` +beside this file, its own 4-rank launcher; see ``_rank_job`` for why. +""" + +import _rank_job +import pytest +import torch + +assert torch.cuda.is_available(), "m3_mc_allreduce_norm requires CUDA devices" + + +# Spawns 4 rank processes over the visible devices; under xdist workers would fight for them. +@pytest.mark.no_xdist +def test_m3_mc_allreduce_norm_op_matrix() -> None: + _rank_job.run("m3_mc_allreduce_norm") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_mc_finalize_allreduce_norm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_mc_finalize_allreduce_norm.py new file mode 100644 index 000000000000..e6f9013f854c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/comm/test_modeling_v2_m3_mc_finalize_allreduce_norm.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests of the ``m3_mc_finalize_allreduce_norm`` entry that need no collective. + +Its cells are certified, with ``m3_mc_allreduce_norm``'s (one kernel, one exchange protocol), by the 4-rank matrix +``_m3_mc_allreduce_norm_op_matrix.py`` that ``test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py`` runs. +""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.comm.m3_mc_finalize_allreduce_norm import ( # noqa: E501 + finalize_partial, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.comm.m3_mc_finalize_allreduce_norm import ( # noqa: E501 + m3_mc_finalize_allreduce_norm as op, +) + + +def test_finalize_partial_rounds_the_routed_sum_before_the_shared_expert() -> None: + """As stock: the finalized routed output is BF16, and the shared expert is added to it (shared.add_(routed)). + Four routed rows of 1.0 at FP32 routing weight 0.2509765625 sum to 1.00390625, which rounds to BF16 1.0; plus the + shared -1 that is 0, where adding before the rounding would leave 0.00390625.""" + fc2 = torch.ones((4, 8), dtype=torch.bfloat16) + perm = torch.arange(4, dtype=torch.int32).view(1, 4) + scales = torch.full((1, 4), 0.2509765625, dtype=torch.float32) + shared = torch.full((1, 8), -1.0, dtype=torch.bfloat16) + out = finalize_partial(fc2, perm, scales, shared) + routed = finalize_partial(fc2, perm, scales, None) + assert torch.equal(out, shared.clone().add_(routed)) + assert torch.equal(out, torch.zeros(1, 8, dtype=torch.bfloat16)) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_each_cells_partial_is_the_stock_finalize_then_shared_add(cell) -> None: + """Per cell shape (rows, top-k, dropped slots), every rank's partial in the entry's reference is the stock MoE's: + the finalized routed rows rounded to BF16, then ``shared.add_(routed)``. The 4-rank matrix drives the cells on the + GPU against this reference.""" + spec = cell.spec + g = torch.Generator().manual_seed(spec["m"] * 31 + spec["dropped"]) + m, topk, hidden = spec["m"], spec["topk"], 64 + fc2 = torch.randn(m * topk, hidden, generator=g).to(torch.bfloat16) + perm = torch.randperm(m * topk, generator=g).view(m, topk).to(torch.int32) + perm.view(-1)[: spec["dropped"]] = -1 + scales = torch.rand(m, topk, generator=g) + shared = torch.randn(m, hidden, generator=g).to(torch.bfloat16) + routed = finalize_partial(fc2, perm, scales, None) + assert torch.equal(finalize_partial(fc2, perm, scales, shared), shared.clone().add_(routed)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/embedding/test_modeling_v2_m3_embedding.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/embedding/test_modeling_v2_m3_embedding.py new file mode 100644 index 000000000000..fdf8805cdd7d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/embedding/test_modeling_v2_m3_embedding.py @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_embedding entry of the MiniMax-M3 tp4 target: an exact gather with zero rows outside the +table.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.embedding.m3_embedding import ( # noqa: E501 + m3_embedding as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_embedding requires a CUDA device" + +VOCAB, HIDDEN = 4096, 6144 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + table = torch.randn(VOCAB, HIDDEN, generator=g, device="cuda").to(torch.bfloat16) + ids = torch.randint( + 0, VOCAB, (spec["rows"],), generator=g, device="cuda", dtype=spec["id_dtype"] + ) + if spec["pad"]: + ids[1::3] = -1 + with validating(op): + out = op(ids, table) + assert torch.equal(out, op.reference(ids, table)) + if spec["pad"]: + assert not out[1::3].any(), "padded ids must embed to zero rows" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/_m3_mxfp8_operands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/_m3_mxfp8_operands.py new file mode 100644 index 000000000000..e44ad56d743d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/_m3_mxfp8_operands.py @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MXFP8 operands for the MXFP8 GEMM entries' tests: E4M3 values and 128x4-swizzled UE8M0 block scales.""" + +import torch + + +def swizzle(byte: torch.Tensor) -> torch.Tensor: + """[N, K / 32] scale bytes into the 128x4-swizzled buffer of ``mxfp8_quantize(w, True)``.""" + rows, cols = byte.shape + padded_cols, padded_rows = (cols + 3) // 4 * 4, (rows + 127) // 128 * 128 + r = torch.arange(rows, device=byte.device).view(-1, 1) + c = torch.arange(cols, device=byte.device).view(1, -1) + index = ( + (c % 4) + + (c // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * padded_cols) + ) + out = torch.zeros(padded_rows * padded_cols, dtype=torch.uint8, device=byte.device) + out[index.reshape(-1)] = byte.reshape(-1) + return out + + +def mxfp8(rows: int, k: int, g: torch.Generator, lo: int = 118, hi: int = 125): + """An MXFP8 operand: E4M3 values of a unit normal, UE8M0 block scales 2^(lo-127) .. 2^(hi-128).""" + v = torch.randn(rows, k, generator=g, device="cuda").to(torch.float8_e4m3fn) + sf = torch.randint(lo, hi, (rows, k // 32), generator=g, device="cuda", dtype=torch.int32) + return v, swizzle(sf.to(torch.uint8)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m31_unfused_index.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m31_unfused_index.py new file mode 100644 index 000000000000..23b7bc17a215 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m31_unfused_index.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test: the unfused sparse layers' separate MXFP8 index projection (``fuse_qkv_index_projection=False``) of the +tp4 target equals the index columns of the fused five-way projection on the same checkpoint tensors. + +Its weight rows are the five-way weight's index rows byte for byte, its block scales are applied (not dropped), and +on every engine MXFP8 backend its output is bitwise the five-way GEMM's index columns. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + weights as tw, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mxfp8_linear import ( # noqa: E501 + m3_mxfp8_linear as op, +) + +assert torch.cuda.is_available(), "the MXFP8 index projection requires a CUDA device" + +TP = 4 +HIDDEN = 6144 +BLOCK = 32 +# Text-model rows of the attention projections (64 query / 4 KV heads, 4 index heads, head and index dim 128). +ROWS = {"q": 64 * 128, "k": 4 * 128, "v": 4 * 128, "index_q": 4 * 128, "index_k": 128} +_FLAGS = { + "native": (False, False, False, False), + "flashinfer": (False, False, True, False), + "tuned": (True, False, False, False), +} + + +def _checkpoint(seed: int): + """E4M3 weights and UE8M0 block scales per projection; the scales spread over 2^-6 .. 2^5 (bytes 121..132).""" + g = torch.Generator(device="cuda").manual_seed(seed) + out = {} + for name, rows in ROWS.items(): + w = (torch.randn(rows, HIDDEN, generator=g, device="cuda") * 2).to(torch.float8_e4m3fn) + s = torch.randint( + 121, 133, (rows, HIDDEN // BLOCK), generator=g, device="cuda", dtype=torch.uint8 + ) + out[name] = (w.cpu(), s.cpu()) + return out + + +def _ns(rank: int) -> SimpleNamespace: + return SimpleNamespace(tp=TP, rank=rank, dev=torch.device("cuda"), head_dim=128, index_dim=128) + + +def _packs(ck, rank): + d = _ns(rank) + names = ("q", "k", "v", "index_q", "index_k") + w5 = tw._qkv_sparse_w(tuple(ck[n][0] for n in names), d) + s5 = tw._qkv_sparse_s( + tuple(ck[n][1] for n in names), d + ) # UE8M0 bytes, as the checkpoint stores them + wi = tw._index_qk_w((ck["index_q"][0], ck["index_k"][0]), d) + si = tw._index_qk_s((ck["index_q"][1], ck["index_k"][1]), d) + return w5, s5.view(torch.uint8), wi, si.view(torch.uint8) + + +@pytest.mark.parametrize("rank", [0, TP - 1]) +def test_weight_rows_are_the_five_way_index_rows(rank) -> None: + ck = _checkpoint(1234 + rank) + w5, _, wi, _ = _packs(ck, rank) + n = wi.shape[0] + assert wi.dtype == torch.float8_e4m3fn and wi.shape == (n, HIDDEN) + assert torch.equal(wi.view(torch.uint8), w5[-n:].view(torch.uint8)) + + +@pytest.mark.parametrize("path", list(_FLAGS)) +@pytest.mark.parametrize("rows", [1, 4, 8, 32, 64, 256]) +def test_unfused_output_is_bitwise_the_five_way_index_columns(path, rows) -> None: + if path == "flashinfer" and getattr(torch.ops.trtllm, "flashinfer_mm_mxfp8", None) is None: + pytest.skip("this build has no trtllm::flashinfer_mm_mxfp8") + ck = _checkpoint(77) + w5, s5, wi, si = _packs(ck, 0) + n = wi.shape[0] + g = torch.Generator(device="cuda").manual_seed(rows) + x = torch.randn(rows, HIDDEN, generator=g, device="cuda").to(torch.bfloat16) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + out5 = op(x, w5, s5, alpha, torch.bfloat16, *_FLAGS[path]) + outi = op(x, wi, si, alpha, torch.bfloat16, *_FLAGS[path]) + assert outi.shape == (rows, n) + assert torch.equal(outi, out5[:, -n:]), ( + f"{path}: max |diff| {(outi.float() - out5[:, -n:].float()).abs().max().item()}" + ) + + +def test_block_scales_are_applied() -> None: + """The output follows the op's reference with the checkpoint scales and differs from a unit-scale (dropped) + projection by far more than rounding.""" + ck = _checkpoint(5) + _, _, wi, si = _packs(ck, 0) + unit = tw._index_qk_s( + (torch.full_like(ck["index_q"][1], 127), torch.full_like(ck["index_k"][1], 127)), _ns(0) + ).view(torch.uint8) + x = torch.randn(8, HIDDEN, device="cuda").to(torch.bfloat16) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + out = op(x, wi, si, alpha, torch.bfloat16, *_FLAGS["native"]) + ref = op.reference(x, wi, si, alpha, torch.bfloat16, *_FLAGS["native"]) + dropped = op(x, wi, unit, alpha, torch.bfloat16, *_FLAGS["native"]) + scale = ref.float().abs().mean() + # The engine GEMM rounds once to BF16 after an fp32 accumulation in its own order: within 2 BF16 ulps (2^-7 + # relative) of the reference per element. + assert bool( + ((out.float() - ref.float()).abs() <= 2.0**-7 * ref.float().abs() + 1e-3 * scale).all() + ) + assert (out.float() - dropped.float()).abs().mean() > 0.5 * scale diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_bf16_linear.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_bf16_linear.py new file mode 100644 index 000000000000..59606c7ca0ee --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_bf16_linear.py @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_bf16_linear entry of the MiniMax-M3 tp4 target (the drafter's projections).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_bf16_linear import ( # noqa: E501 + m3_bf16_linear as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_bf16_linear requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """Accumulation order (cuBLAS split-K) then one bf16 rounding on both sides: at most 2 units of ULP[bf16] of the + row, RMS well under one; measured on GB300: worst 0.5 unit.""" + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = torch.randn(spec["m"], spec["k"], generator=g, device="cuda").to(torch.bfloat16) + w = (torch.randn(spec["n"], spec["k"], generator=g, device="cuda") * 0.02).to(torch.bfloat16) + with validating(op): + out = op(x, w) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(x, w), element_ulp=2.0, rms_ulp=0.25) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_dense_swiglu.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_dense_swiglu.py new file mode 100644 index 000000000000..335dad1bc616 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_dense_swiglu.py @@ -0,0 +1,142 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Bitwise oracle of the dense MLP's swigluoai C++ op (ops/gemm/m3_dense_swiglu, +trtllm::minimax_m3_sm107_tp4_swiglu_oai) against the Triton silu_and_mul kernel the dense MLP runs today +(modules/swiglu.py at limit 7, alpha 1.702, beta 1): every bf16 gate bit pattern against special and random up values, +every up bit pattern against special and random gates, random pairs, and the dense MLP's shapes at every decode row +count, eager and replayed from a CUDA graph, with and without the programmatic launch.""" + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.silu_and_mul +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_dense_swiglu import ( # noqa: E501 + ALPHA, + BETA, + LIMIT, + m3_dense_swiglu, +) + +assert torch.cuda.is_available(), "the dense swigluoai op requires a CUDA device" +if not hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp4_swiglu_oai"): + pytest.skip( + "needs the TP4 op library with minimax_m3_sm107_tp4_swiglu_oai", allow_module_level=True + ) + +# The TP4 dense MLP's activation width (gate_up 3072 -> d 1536). +DENSE_D = dispatch.DENSE_DOWN[1] + + +def _triton(x: torch.Tensor) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=LIMIT, swiglu_alpha=ALPHA, swiglu_beta=BETA + ) + + +def _op(x: torch.Tensor, pdl: bool = True) -> torch.Tensor: + out = m3_dense_swiglu(x, LIMIT, ALPHA, BETA, pdl) + assert out is not None, "the op declined a contiguous bf16 [rows, 2d] input" + return out + + +def _row(gate: torch.Tensor, up: torch.Tensor) -> torch.Tensor: + return torch.cat([gate.view(1, -1), up.view(1, -1)], dim=1).contiguous() + + +def _all_bf16() -> torch.Tensor: + return ( + torch.arange(1 << 16, dtype=torch.int32, device="cuda").to(torch.int16).view(torch.bfloat16) + ) + + +def _specials(n: int, seed: int) -> list: + g = torch.Generator(device="cuda").manual_seed(seed) + vals = [0.0, -0.0, 1.0, -1.0, 6.99, 7.0, 7.01, -7.0, -7.01, 1e-38, -1e-38, 1e-40, 88.0, -88.0, 3e38, -3e38, + float("inf"), float("-inf"), float("nan")] # fmt: skip + rows = [torch.full((n,), v, dtype=torch.bfloat16, device="cuda") for v in vals] + for scale in (0.05, 1.0, 4.0, 30.0): + rows.append((scale * torch.randn(n, generator=g, device="cuda")).to(torch.bfloat16)) + rows.append( + torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + .to(torch.int16) + .view(torch.bfloat16) + ) + return rows + + +def _assert_bitwise(got: torch.Tensor, ref: torch.Tensor, what: str) -> None: + bad = (got.view(torch.int16) != ref.view(torch.int16)).nonzero().flatten() + assert bad.numel() == 0, f"{what}: {bad.numel()} mismatches, first at {bad[:4].tolist()}" + + +def test_every_gate_pattern_is_bitwise_the_triton_kernel() -> None: + gate = _all_bf16() + for j, up in enumerate(_specials(gate.numel(), 1)): + x = _row(gate, up) + _assert_bitwise(_op(x), _triton(x), f"up set {j}") + + +def test_every_up_pattern_is_bitwise_the_triton_kernel() -> None: + up = _all_bf16() + for j, gate in enumerate(_specials(up.numel(), 2)): + x = _row(gate, up) + _assert_bitwise(_op(x), _triton(x), f"gate set {j}") + + +@pytest.mark.parametrize("seed", range(4)) +def test_random_pairs_are_bitwise_the_triton_kernel(seed: int) -> None: + g = torch.Generator(device="cuda").manual_seed(100 + seed) + n = 1 << 22 + + def pattern() -> torch.Tensor: + bits = torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + return bits.to(torch.int16).view(torch.bfloat16) + + x = _row(pattern(), pattern()) + _assert_bitwise(_op(x), _triton(x), f"seed {seed}") + + +@pytest.mark.parametrize("pdl", [True, False], ids=["pdl", "plain"]) +@pytest.mark.parametrize("rows", range(1, dispatch.DECODE_MAX_TOKENS + 1)) +def test_dense_shape_eager_and_graph_are_bitwise_the_triton_kernel(rows: int, pdl: bool) -> None: + """The TP4 dense MLP's gate_up rows (a GEMV output's range plus values past the clamps), eager and replayed from a + CUDA graph after fresh rows are written in place, against the Triton kernel on the same rows.""" + g = torch.Generator(device="cuda").manual_seed(2000 + rows) + x = (3.0 * torch.randn(rows, 2 * DENSE_D, generator=g, device="cuda")).to(torch.bfloat16) + x[:, :8] = torch.tensor([8.0, -8.0, 7.0, -7.0, 0.0, -0.0, 50.0, -50.0], dtype=torch.bfloat16) + x[:, DENSE_D : DENSE_D + 8] = x[:, :8] + _assert_bitwise(_op(x, pdl), _triton(x), f"eager rows {rows}") + out = torch.empty(rows, DENSE_D, dtype=torch.bfloat16, device="cuda") + _op(x, pdl) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out.copy_(_op(x, pdl)) + x.copy_((3.0 * torch.randn(rows, 2 * DENSE_D, generator=g, device="cuda")).to(torch.bfloat16)) + graph.replay() + torch.cuda.synchronize() + _assert_bitwise(out, _triton(x), f"graph rows {rows}") + + +def test_the_op_declines_other_parameters_and_layouts() -> None: + x = torch.zeros(2, 2 * DENSE_D, dtype=torch.bfloat16, device="cuda") + assert m3_dense_swiglu(x, 8.0, ALPHA, BETA, True) is None + assert m3_dense_swiglu(x, LIMIT, 1.0, BETA, True) is None + assert m3_dense_swiglu(x, LIMIT, ALPHA, 0.0, True) is None + assert m3_dense_swiglu(x.float(), LIMIT, ALPHA, BETA, True) is None + assert m3_dense_swiglu(x[:, : 2 * DENSE_D - 2], LIMIT, ALPHA, BETA, True) is None + + +@pytest.mark.parametrize("cell", m3_dense_swiglu.CELLS, ids=lambda c: c.why) +def test_every_cell_is_bitwise_the_triton_kernel(cell) -> None: + """Each declared cell (the dense MLP's activation at its decode rows) through the op, bit for bit the Triton + kernel on gate_up rows in a GEMV output's range plus values past the clamps.""" + rows, d = cell.spec["rows"], cell.spec["d"] + g = torch.Generator(device="cuda").manual_seed(3000 + rows) + x = (3.0 * torch.randn(rows, 2 * d, generator=g, device="cuda")).to(torch.bfloat16) + x[:, :8] = torch.tensor([8.0, -8.0, 7.0, -7.0, 0.0, -0.0, 50.0, -50.0], dtype=torch.bfloat16) + x[:, d : d + 8] = x[:, :8] + _assert_bitwise(_op(x), _triton(x), cell.why) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_draft_lowm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_draft_lowm.py new file mode 100644 index 000000000000..65f31f3be3bd --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_draft_lowm.py @@ -0,0 +1,182 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gemm/m3_draft_lowm entry: the Eagle3 drafter's BF16 linears on FlashInfer's fixed-tactic low-M +kernels (direct SIMT and split-K tensor-core), at the TP4 drafter shards, at the rows its table certifies. + +Every cell is held to the fp64 reference within the derived band. The drafter's projections (``M3Eagle3Draft._linear`` +over ``kernels/m3_draft_lowm``) are held on the drafter's weight table: a covered row count is the entry's call bit for +bit, an uncovered one the plain GEMM's. +""" + +import zlib + +import pytest +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_draft_lowm, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_draft_lowm import ( # noqa: E501 + ROWS, + TACTICS, + tactic_for, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_draft_lowm import ( # noqa: E501 + m3_draft_lowm as op, +) +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_draft_lowm requires a CUDA device" + +# Element band, in BF16 ulps of the row scale, and the relative RMS (``assert_within_ulp``). +# +# Kernel and reference multiply the same BF16 operands (every product exact in fp32); the kernel sums in fp32 (the +# direct kernel one dot product in K order, the split-K kernel tensor-core blocks and then its split partials), the +# reference in fp64, and both round once to BF16. With w ~ N(0, 1 / K) and x ~ N(0, 1) the outputs are O(1) and the +# fp32 order difference is ~sqrt(K) * 2^-24 of the summed magnitudes, a few 1e-6 at K = 18432: far below half a BF16 +# ulp, so the two roundings disagree by at most one ulp of the element -- up to two units of the row scale for an +# element in its row maximum's binade -- and only where the sum sits that close to a rounding boundary, a few in ten +# thousand elements, hence 0.1 ulp RMS. The same derivation as the decode GEMV's band (test_modeling_v2_m3_mx_gemv), +# with no slack under it: a second ulp would mean a wrong sum. +ELEMENT_ULP = 2.0 +RMS_ULP = 0.1 + +# (N, K) -> the drafter's weight key (M3Eagle3Draft.w). +DRAFTER_LINEARS = { + (2304, 12288): "qkv", + (6144, 2048): "o", + (9216, 6144): "gu", + (6144, 4608): "dn", + (6144, 18432): "fc", +} + + +@pytest.fixture(autouse=True) +def _policy(): + dispatch.set_require(False) + dispatch.reset_logged() + yield + dispatch.set_require(False) + dispatch.reset_logged() + + +def build(n: int, k: int, m: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = torch.randn(m, k, generator=g, device="cuda").to(torch.bfloat16) + return x, w + + +def test_the_table_is_the_tp4_drafter() -> None: + assert set(TACTICS) == set(DRAFTER_LINEARS) + assert {c.spec["nk"] for c in op.CELLS} == set(TACTICS) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + (n, k), m = cell.spec["nk"], cell.spec["m"] + x, w = build(n, k, m, seed=zlib.crc32(cell.why.encode()) % 2**31) + kind, tactic = tactic_for(n, k, m) + out = torch.full((m, n), float("nan"), dtype=torch.bfloat16, device="cuda") + args = ( + x, + w, + out, + kind, + tactic, + True, + ) # the target launches with PDL when the environment enables it + certifying(op) + assert op(*args) is out + ref = op.reference(*args) + assert out.shape == (m, n) and out.dtype is torch.bfloat16 + assert_within_ulp(out, ref, element_ulp=ELEMENT_ULP, rms_ulp=RMS_ULP) + # PDL only changes the launch attribute. + plain = torch.empty_like(out) + op(x, w, plain, kind, tactic, False) + assert torch.equal(plain.view(torch.int16), out.view(torch.int16)) + + +def _draft(linears=DRAFTER_LINEARS): + """A drafter holding only its projection weights (the module is not built).""" + draft = object.__new__(dispatch.M3Eagle3Draft) + draft.w = {key: build(*nk, 1, seed=nk[0] + nk[1])[1] for nk, key in linears.items()} + draft._lowm_keys = frozenset() + return draft + + +@pytest.fixture(scope="module") +def prebuilt(): + draft = _draft() + draft.prebuild_linears() # 0 when an earlier test in this process compiled them + assert draft._lowm_keys == frozenset(DRAFTER_LINEARS.values()) + assert draft.prebuild_linears() == 0, "a second prebuild finds nothing left to compile" + return draft + + +@pytest.mark.parametrize("nk", sorted(DRAFTER_LINEARS), ids=lambda nk: DRAFTER_LINEARS[nk]) +def test_the_projection_is_the_entry_and_falls_back_past_its_rows(prebuilt, nk) -> None: + n, k = nk + key = DRAFTER_LINEARS[nk] + w = prebuilt.w[key] + use_pdl = get_env_enable_pdl() + dispatch.set_require(True) + for m in ROWS[nk]: + assert dispatch.DRAFT_LOWM.covers((nk, m)) + x, _ = build(n, k, m, seed=m) + got = prebuilt._linear(x, key) + kind, tactic = tactic_for(n, k, m) + want = op(x, w, torch.empty(m, n, dtype=x.dtype, device="cuda"), kind, tactic, use_pdl) + assert torch.equal(got.view(torch.int16), want.view(torch.int16)), m + # Rows the table does not certify (past its rows): the plain GEMM, silently, even under require. + for m in sorted(set(range(1, dispatch.DECODE_MAX_TOKENS + 2)) - set(ROWS[nk])): + assert not dispatch.DRAFT_LOWM.covers((nk, m)) + x, _ = build(n, k, m, seed=m) + got = prebuilt._linear(x, key) + assert torch.equal(got.view(torch.int16), F.linear(x, w).view(torch.int16)), m + + +def test_a_projection_outside_the_table_is_the_plain_gemm() -> None: + assert not m3_draft_lowm.installed(torch.empty(256, 512, dtype=torch.bfloat16)) + draft = _draft({(256, 512): "o"}) + draft._lowm_keys = frozenset(k for k in ("o",) if m3_draft_lowm.installed(draft.w[k])) + x, _ = build(256, 512, 1, seed=3) + got = draft._linear(x, "o") + assert torch.equal(got.view(torch.int16), F.linear(x, draft.w["o"]).view(torch.int16)) + + +def test_a_covered_call_not_prebuilt_raises_under_require(monkeypatch) -> None: + monkeypatch.setattr(m3_draft_lowm, "_BUILT", set()) + monkeypatch.setattr(m3_draft_lowm, "_KEYS", frozenset(dispatch.DRAFT_LOWM.keys)) + w = build(6144, 2048, 1, seed=4)[1] + x, _ = build(6144, 2048, 1, seed=5) + dispatch.set_require(True) + with pytest.raises(dispatch.FastPathUnavailable, match="not compiled before warmup"): + m3_draft_lowm.draft_linear(w, x, None) + # Outside require the drafter falls back to the plain GEMM. + dispatch.set_require(False) + assert m3_draft_lowm.draft_linear(w, x, None) is None + + +def test_guard_refuses_a_strided_activation() -> None: + x, w = build(6144, 2048, 2, seed=6) + wide = torch.zeros(2, 4096, dtype=torch.bfloat16, device="cuda") + wide[:, :2048] = x + kind, tactic = tactic_for(6144, 2048, 2) + with pytest.raises(AssertionError, match="contiguous"): + with validating(op): + op( + wide[:, :2048], + w, + torch.empty(2, 6144, dtype=torch.bfloat16, device="cuda"), + kind, + tactic, + False, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_fp32_linear.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_fp32_linear.py new file mode 100644 index 000000000000..b6b1cdcbe2a6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_fp32_linear.py @@ -0,0 +1,32 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_fp32_linear entry of the MiniMax-M3 tp4 target (the router beyond the decode routers' rows).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_fp32_linear import ( # noqa: E501 + m3_fp32_linear as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_fp32_linear requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """torch's default fp32 matmul precision on this hardware may be TF32 (~2^-11 relative per product): the band is + stated in fp32 ulps of each row's largest logit and covers it (K = 6144 TF32 products: ~2^13 ulps worst element); + measured on GB300 with the engine's defaults: worst element ~1700 ulps, RMS ~300.""" + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = (torch.randn(spec["m"], spec["k"], generator=g, device="cuda") * 2.0).to(torch.bfloat16) + w = torch.randn(spec["e"], spec["k"], generator=g, device="cuda") * 0.02 + with validating(op): + out = op(x, w) + assert out.dtype == torch.float32 and out.shape == (spec["m"], spec["e"]) + assert_within_ulp(out, op.reference(x, w), element_ulp=8192.0, rms_ulp=2048.0) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv.py new file mode 100644 index 000000000000..6d45a96bce88 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gemm/m3_mx_gemv entry: the decode MXFP8 GEMV on a producer-quantized activation. + +Operands as the target holds them: the weight through mxfp8_quantize with the swizzled scale layout (the MXFP8 linear +method's storage), the activation through mxfp8_quantize with the swizzled layout (what the boundary all-reduce and +the attention merge publish). +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mx_gemv import ( # noqa: E501 + m3_mx_gemv as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_mx_gemv requires a CUDA device" + + +# Element band, in ULP units of the row scale: see compare. +ELEMENT_ULP = 1.5 + + +def compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """Element ELEMENT_ULP of the row scale, relative RMS 0.1 ulp. + + Kernel and reference multiply bit-identical dequantized operands; the kernel sums in fp32 (tensor-core blocks, + then the split-K partials), the reference in fp64. Over K <= 6144 that order difference is ~1e-6 relative, so the + two bf16 roundings disagree only where the sum sits within that of a rounding boundary: one bf16 ulp of that + element, which for an element in its row maximum's binade is up to two of ``assert_within_ulp``'s units of the row + scale, and a few in ten thousand elements in aggregate. Any fp32 sum shows it: torch's IEEE fp32 matmul of the + same dequantized operands measured at most 1.22 units against the fp64 reference over every row count 1..48 on + the o_proj and the QKV (6 seeds each, GB300), the kernel at most 1.25, both at most 0.009 ulp RMS. The band + leaves the margin between that and a full extra ulp. + """ + assert_within_ulp(out, ref, element_ulp=ELEMENT_ULP, rms_ulp=0.1) + + +def build(n: int, k: int, m: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = torch.randn(m, k, generator=g, device="cuda").to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + return x, xq.view(m, k), xsf, wq.view(n, k), wsf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + (n, k), m = cell.spec["nk"], cell.spec["m"] + _, xq, xsf, w, wsf = build(n, k, m, seed=zlib.crc32(cell.why.encode()) % 2**31) + with validating(op): + out = op(xq, xsf, w, wsf, use_pdl=True) # the target launches with PDL + assert out is not None, "a certified shape must be served" + assert out.shape == (m, n) and out.dtype is torch.bfloat16 + compare(out, op.reference(xq, xsf, w, wsf)) + + +def test_an_unserved_shape_returns_none() -> None: + """Shapes outside the tuned table fall back at the call site; the op says so rather than computing.""" + _, xq, xsf, w, wsf = build(256, 512, 1, seed=1) + assert op(xq, xsf, w, wsf) is None + + +def test_guard_refuses_linear_activation_scales() -> None: + """A linear [M, K / 32] scale buffer is read as the swizzled one.""" + x, _, _, w, wsf = build(6144, 2048, 4, seed=2) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, False) + with pytest.raises(AssertionError, match="swizzled"): + with validating(op): + op(xq.view(4, 2048), xsf, w, wsf) + + +def test_guard_refuses_a_strided_weight() -> None: + _, xq, xsf, w, wsf = build(6144, 2048, 1, seed=3) + wide = torch.zeros(6144, 4096, dtype=w.dtype, device="cuda") + wide[:, :2048] = w + with pytest.raises(AssertionError, match="contiguous e4m3"): + with validating(op): + op(xq, xsf, wide[:, :2048], wsf) + + +@pytest.mark.parametrize("nk", [(2560, 6144), (2304, 6144)], ids=["qkv_sparse", "qkv_dense"]) +@pytest.mark.parametrize("m", range(1, 9)) +def test_qkv_split8_config(nk, m) -> None: + """The TP4 dispatch's (split, ring) override for the QKV at decode rows: within the band of the fp64 reference, + and the graph replay bit-identical to the eager call (one split-K order per config).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + modeling as dispatch, + ) + + cfg = dispatch.mx_gemv_cfg(nk[0], nk[1], m) + assert cfg == (8, 8) + _, xq, xsf, wq, wsf = build(nk[0], nk[1], m, zlib.crc32(f"qkv8-{nk}-{m}".encode())) + eager = op(xq, xsf, wq, wsf, cfg=cfg) + assert eager is not None, "the GEMV declined the split-8 config" + compare(eager, op.reference(xq, xsf, wq, wsf)) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = op(xq, xsf, wq, wsf, cfg=cfg) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(captured.view(torch.int16), eager.view(torch.int16)) + + +def test_qkv_split8_config_is_bounded(monkeypatch) -> None: + """The overrides cover only the two QKV projections (split 8) and the o_proj (strided k-tiles, rule 1b) at decode + rows; every other call, and every call with its kill knob off, keeps the shared table.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + modeling as dispatch, + ) + + assert dispatch.mx_gemv_cfg(2560, 6144, 9) is None + assert dispatch.mx_gemv_cfg(6144, 2048, 4) == (2, 8, 0) + assert dispatch.mx_gemv_cfg(6144, 2048, 9) is None + assert dispatch.mx_gemv_cfg(6144, 3072, 4) is None + monkeypatch.setattr(dispatch, "GEMV_OPROJ_STRIDED_TP4", False) + assert dispatch.mx_gemv_cfg(6144, 2048, 4) is None + assert dispatch.mx_gemv_cfg(2560, 6144, 4) == (8, 8) + + +@pytest.mark.parametrize( + "nk", + [(2560, 6144), (2304, 6144), (6144, 2048)], + ids=["qkv_sparse", "qkv_dense", "o_proj_strided"], +) +def test_prebuild_covers_the_split8_config_while_frozen(nk) -> None: + """The core's prebuild (``prebuild_kernels``: ``_gemv_variants`` + ``_precompile_cfg``) compiles every (rows, cfg) + ``_linear`` can pick for this projection, the TP4 (split, ring) override included: with compiles frozen afterwards + (as the target freezes them before warmup), every decode-row MXFP8 call at the override is served, none + declines.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + modeling as dispatch, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, + ) + + n, k = nk + _, _, _, wq, wsf = build(n, k, 1, zlib.crc32(f"prebuild-{nk}".encode())) + mx_gemv.freeze(False) + mx_gemv.ensure_compiled(wq, wsf, True) + for rows, x_bf16, ar in dispatch._gemv_variants(n, k, None, 4): + cfg = None if x_bf16 else dispatch.mx_gemv_cfg(n, k, rows) + if cfg is not None: + dispatch._precompile_cfg(wq, wsf, rows, cfg, True, ar=ar) + else: + mx_gemv.precompile(wq, wsf, rows, x_bf16, use_pdl=True, ar=ar) + picked = [ + (rows, dispatch.mx_gemv_cfg(n, k, rows)) + for rows in sorted(dispatch.MX_GEMV_ROWS[nk]) + if dispatch.mx_gemv_cfg(n, k, rows) is not None + ] + assert picked, "the override covers decode rows of this projection" + for rows, cfg in picked: + resolved = mx_gemv._resolve(rows, n, k, False, True, cfg, True, None) + assert resolved is not None and mx_gemv.is_compiled(resolved[0]), (rows, cfg) + mx_gemv.freeze(True) + try: + for rows, cfg in picked: + _, xq, xsf, _, _ = build(n, k, rows, rows) + out = mx_gemv.m3_mx_gemv(xq, xsf, wq, wsf, cfg=cfg, use_pdl=True) + assert out is not None, (rows, cfg, mx_gemv.last_uncompiled) + finally: + mx_gemv.freeze(False) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv_bf16.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv_bf16.py new file mode 100644 index 000000000000..20e91120fe16 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv_bf16.py @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gemm/m3_mx_gemv_bf16 entry: the decode MXFP8 GEMV quantizing a bf16 activation in its prologue. + +The weight as the MXFP8 linear method stores it (mxfp8_quantize, swizzled scales). Besides the reference band, the +prologue's quantization is held to mxfp8_quantize itself: the GEMV on a bf16 x must equal, bit for bit, the +prequantized GEMV on mxfp8_quantize(x) -- the claim the target relies on when it mixes the two paths. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mx_gemv import ( # noqa: E501 + m3_mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mx_gemv_bf16 import ( # noqa: E501 + m3_mx_gemv_bf16 as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_mx_gemv_bf16 requires a CUDA device" + +# The element band of test_modeling_v2_m3_mx_gemv.compare (derived there): the same fp32 sums against fp64. +ELEMENT_ULP = 1.5 + + +def build(n: int, k: int, m: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = torch.randn(m, k, generator=g, device="cuda").to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + return x, wq.view(n, k), wsf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + (n, k), m = cell.spec["nk"], cell.spec["m"] + x, w, wsf = build(n, k, m, seed=zlib.crc32(cell.why.encode()) % 2**31) + with validating(op): + out = op(x, w, wsf, use_pdl=True) # the target launches with PDL + assert out is not None, "a certified shape must be served" + assert out.shape == (m, n) and out.dtype is torch.bfloat16 + # The band of test_modeling_v2_m3_mx_gemv.compare: the quantized operands are the reference's exactly (checked + # below), the sums differ only in order. + assert_within_ulp(out, op.reference(x, w, wsf), element_ulp=ELEMENT_ULP, rms_ulp=0.1) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + pre = m3_mx_gemv(xq.view(m, k), xsf, w, wsf, use_pdl=True) + assert pre is not None and torch.equal(out.view(torch.int16), pre.view(torch.int16)), ( + "the prologue's MXFP8 quantization differs from mxfp8_quantize" + ) + + +def test_a_strided_activation_is_declined() -> None: + """The prologue reads x densely; the op declines a strided x (None) rather than computing on it.""" + x, w, wsf = build(6144, 2048, 2, seed=1) + wide = torch.zeros(2, 4096, dtype=torch.bfloat16, device="cuda") + wide[:, :2048] = x + assert op(wide[:, :2048], w, wsf) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv_order.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv_order.py new file mode 100644 index 000000000000..dcc9a4930467 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mx_gemv_order.py @@ -0,0 +1,78 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The TP4 decode GEMV order overrides (modeling.mx_gemv_cfg: QKV split 8, o_proj strided k-tiles): +for every overridden (projection, rows) the output is the same on 3 eager calls and on a CUDA graph replay +(deterministic), and its max abs / max rel error vs the fp64 product of the dequantized operands stays within the +shipped order's band (the shared table's config on the same operands, over several seeds).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mx_gemv import ( # noqa: E501 + O_PROJ, + QKV, + QKV_SPARSE, + m3_mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe_engine import ( # noqa: E501 + mxfp8_dequant, +) + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +CASES = [(nk, m) for nk in (QKV_SPARSE, QKV, O_PROJ) for m in (1, 2, 4, 8)] + + +def _err(y, ref): + d = (y.double() - ref).abs() + return float(d.max()), float((d / ref.abs().amax(dim=1, keepdim=True).clamp_min(1e-30)).max()) + + +@pytest.mark.parametrize("nk, m", CASES, ids=[f"{n}x{k}-m{m}" for (n, k), m in CASES]) +def test_order_override_is_deterministic_and_in_band(nk, m) -> None: + n, k = nk + cfg = dispatch.mx_gemv_cfg(n, k, m) + assert cfg is not None, "the override is off or does not cover this call" + worst_new, worst_ship = [0.0, 0.0], [0.0, 0.0] + for seed in range(4): + g = torch.Generator(device="cuda").manual_seed(1000 * seed + m) + w = (torch.randn(n, k, device="cuda", generator=g) * 2).to(torch.float8_e4m3fn) + ws = torch.randint(118, 124, (n * k // 32,), dtype=torch.uint8, device="cuda", generator=g) + x = (torch.randn(m, k, device="cuda", generator=g) * (0.5 + seed)).to(torch.bfloat16) + q, s = torch.ops.trtllm.mxfp8_quantize(x, True) + ref = ( + mxfp8_dequant(q.view(torch.uint8), s).double() + @ mxfp8_dequant(w.view(torch.uint8), ws).double().t() + ) + ys = [m3_mx_gemv(q, s, w, ws, True, cfg, True, None) for _ in range(3)] + assert all(y is not None for y in ys), "the GEMV declined the override" + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + out = [] + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out.append(m3_mx_gemv(q, s, w, ws, True, cfg, True, None)) + graph.replay() + torch.cuda.synchronize() + for y in ys[1:] + [out[0]]: + assert torch.equal(ys[0].view(torch.int16), y.view(torch.int16)), ( + "not deterministic (eager / graph)" + ) + ship = m3_mx_gemv(q, s, w, ws, True, None, True, None) + torch.cuda.synchronize() + for acc, y in ((worst_new, ys[0]), (worst_ship, ship)): + e = _err(y, ref) + acc[0], acc[1] = max(acc[0], e[0]), max(acc[1], e[1]) + print( + f"\nR1B1 tp4 {n}x{k} m={m} cfg {cfg}: max abs / rel vs fp64 {worst_new[0]:.3e} / {worst_new[1]:.3e} | shipped " + f"order {worst_ship[0]:.3e} / {worst_ship[1]:.3e}", + flush=True, + ) + # The band: the shipped order's own error (bf16 output rounding dominates), with 25 % headroom. + assert worst_new[0] <= 1.25 * worst_ship[0] and worst_new[1] <= 1.25 * worst_ship[1] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_gemm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_gemm.py new file mode 100644 index 000000000000..9e8bffc5c6a1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_gemm.py @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_gemm entry of the MiniMax-M3 tp4 target (the decode-graph GEMM on a prequantized +activation).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mxfp8_gemm import ( # noqa: E501 + m3_mxfp8_gemm as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_gemm requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """Same E4M3 operands and power-of-two scales on both sides; accumulation order then one bf16 rounding: at most + 2 units of ULP[bf16] of the row (the low-M GEMM's band), RMS well under one.""" + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + act, act_sf = mxfp8(spec["m"], spec["k"], g) + w, w_sf = mxfp8(spec["n"], spec["k"], g) + with validating(op): + out = op(act, act_sf, w, w_sf, torch.bfloat16, False) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp( + out, + op.reference(act, act_sf, w, w_sf, torch.bfloat16, False), + element_ulp=2.0, + rms_ulp=0.25, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_linear.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_linear.py new file mode 100644 index 000000000000..8331ec65dfb4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_linear.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_linear entry of the MiniMax-M3 tp4 target: MXFP8 quantize + GEMM on each backend the +engine selects (native CUTLASS, FlashInfer, the per-bucket tuned quantize + GEMM).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mxfp8_linear import ( # noqa: E501 + m3_mxfp8_linear as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_linear requires a CUDA device" + +_FLAGS = { + "native": (False, False, False, False), + "flashinfer": (False, False, True, False), + "tuned": (True, False, False, False), +} + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """The activation's MXFP8 round trip is exact on both sides (same block rule); then the accumulation order and + one bf16 rounding: at most 2 units of ULP[bf16] of the row, RMS well under one.""" + spec = cell.spec + if ( + spec["path"] == "flashinfer" + and getattr(torch.ops.trtllm, "flashinfer_mm_mxfp8", None) is None + ): + pytest.skip("this build has no trtllm::flashinfer_mm_mxfp8") + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = torch.randn(spec["m"], spec["k"], generator=g, device="cuda").to(torch.bfloat16) + w, w_sf = mxfp8(spec["n"], spec["k"], g) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + args = (x, w, w_sf, alpha, torch.bfloat16, *_FLAGS[spec["path"]]) + with validating(op): + out = op(*args) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(*args), element_ulp=2.0, rms_ulp=0.25) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_pinned_gemm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_pinned_gemm.py new file mode 100644 index 000000000000..347a39dd4f6b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_pinned_gemm.py @@ -0,0 +1,82 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_pinned_gemm entry (MX-TAC): the pinned decode-graph MXFP8 GEMM gives the same bits as +every candidate stock's AutoTuner can pick (CuTe DSL, CUTLASS), the same on every call, within the MXFP8 band of the +dequantized product.""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mxfp8_pinned_gemm import ( # noqa: E501 + m3_mxfp8_pinned_gemm as op, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import FlashInferMXFP8GemmRunner + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_pinned_gemm requires a CUDA device" + +TACTIC = modeling.MXFP8_DECODE_GEMM_TACTIC + +# The TP4 projections at rows the GEMVs do not serve (SHARED_DOWN 17..32, over 48) and the bf16 activations over the +# GEMVs' bf16 bounds. +CASES = [ + ((6144, 768), 17), + ((6144, 768), 32), + ((2560, 6144), 24), + ((2304, 6144), 64), + ((6144, 2048), 49), + ((1536, 6144), 20), + ((6144, 6144), 12), + ((6144, 3072), 56), +] + + +def test_the_pinned_tactic_is_cute_dsl() -> None: + assert TACTIC == FlashInferMXFP8GemmRunner.CUTE_DSL + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + act, act_sf = mxfp8(spec["m"], spec["k"], g) + w, w_sf = mxfp8(spec["n"], spec["k"], g) + with validating(op): + out = op(act, act_sf, w, w_sf, torch.bfloat16, TACTIC) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp( + out, + op.reference(act, act_sf, w, w_sf, torch.bfloat16, TACTIC), + element_ulp=2.0, + rms_ulp=0.25, + ) + + +@pytest.mark.parametrize("nk,m", CASES, ids=[f"{n}x{k}-m{m}" for (n, k), m in CASES]) +@pytest.mark.parametrize("scale", [1e-3, 1.0, 1e3]) +def test_pinned_decision_matches_every_stock_candidate(nk, m, scale) -> None: + n, k = nk + g = torch.Generator(device="cuda").manual_seed(n + k + m) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = (torch.randn(m, k, generator=g, device="cuda") * scale).to(torch.bfloat16) + x[0, ::32] *= 1e3 # one outlier per 32-element block on the first row + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + wq = wq.view(n, k) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + xq = xq.view(m, k) + out = op(xq, xsf, wq, wsf, torch.bfloat16, TACTIC) + runner = FlashInferMXFP8GemmRunner(torch.bfloat16) + for tactic in (FlashInferMXFP8GemmRunner.CUTE_DSL, FlashInferMXFP8GemmRunner.CUTLASS): + ref = runner.forward([xq, xsf, wq, wsf], tactic=tactic) + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)), tactic + again = op(xq, xsf, wq, wsf, torch.bfloat16, TACTIC) + assert torch.equal(out.view(torch.int16), again.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_pinned_quantize.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_pinned_quantize.py new file mode 100644 index 000000000000..e3779113718c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_mxfp8_pinned_quantize.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_pinned_quantize entry (MX-TAC): the pinned decode-graph MXFP8 activation quantize is the +stock quantize bit for bit, at every tactic stock's AutoTuner can pick.""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mxfp8_pinned_quantize import ( # noqa: E501 + m3_mxfp8_pinned_quantize as op, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import MXFP8QuantizeRunner + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_pinned_quantize requires a CUDA device" + +TACTIC = modeling.MXFP8_DECODE_QUANT_TACTIC + + +def test_the_pinned_tactic_is_cute_dsl() -> None: + assert TACTIC == MXFP8QuantizeRunner.CUTE_DSL + + +def _check(x: torch.Tensor) -> None: + m, k = x.shape + with validating(op): + xq, xsf = op(x, TACTIC) + tq, tsf = op.reference(x, TACTIC) + assert torch.equal(xq.view(torch.uint8).view(m, k), tq.view(torch.uint8).view(m, k)) + assert torch.equal(xsf.view(torch.uint8).flatten(), tsf.view(torch.uint8).flatten()) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +@pytest.mark.parametrize("scale", [1e-3, 1.0, 1e3]) +def test_certified_cells(cell, scale) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = (torch.randn(spec["m"], spec["k"], generator=g, device="cuda") * scale).to(torch.bfloat16) + x[0, ::32] *= 1e3 # one outlier per 32-element block on the first row + _check(x) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_target_head.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_target_head.py new file mode 100644 index 000000000000..3ac75b391452 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/gemm/test_modeling_v2_m3_target_head.py @@ -0,0 +1,253 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gemm/m3_target_head entry: the target lm_head shard GEMV over the lossless 12-bit coding of its +BF16 weight, against its reference, the in-tree BF16 head GEMM (``F.linear``) and fp64. + +* The coding round-trips bit for bit (code values and exception lists), and a corrupted coding fails the check. +* Integer-valued data (fp32 sums exact in any order): the kernel's logits equal the in-tree GEMM's bit for bit. +* Random data: the kernel's error against fp64 is the in-tree GEMM's (the same products, another fp32 order, one + BF16 rounding), its argmax is the in-tree one and its result is the same run to run. +""" + +import zlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_target_head as target_head_kernels, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_target_head import ( # noqa: E501 + CTA_WARPS, + HIDDEN, + TARGET_SHARD, + decode, + encode_chunked, + round_trip_ok, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_target_head import ( # noqa: E501 + m3_target_head as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_target_head requires a CUDA device" + + +def _seed(why: str) -> int: + return zlib.crc32(why.encode()) % 2**31 + + +def _num_sms() -> int: + return torch.cuda.get_device_properties(torch.cuda.current_device()).multi_processor_count + + +def _weight(n: int, k: int, seed: int, exceptions: bool = False) -> torch.Tensor: + """A [n, k] head of trained-weight scale; with ``exceptions``, values on both sides of the exponent window, exact + zeros and a large outlier in many tiles.""" + g = torch.Generator(device="cuda").manual_seed(seed) + w = torch.randn(n, k, device="cuda", generator=g) * 0.02 + if exceptions: + w[::97, ::131] = 1e-9 + w[::89, 7] = 0.0 + w[3, 5] = 7.5 + w[n - 1, k - 1] = -3.0 + return w.to(torch.bfloat16) + + +_CODED: dict = {} + + +def _coded(n: int, k: int, seed: int, exceptions: bool): + """(weight, encode_chunked(weight)), built once per shape.""" + key = (n, k, seed, exceptions) + if key not in _CODED: + _CODED.clear() + w = _weight(n, k, seed, exceptions) + _CODED[key] = (w, encode_chunked(w)) + return _CODED[key] + + +def _args(coded, x: torch.Tensor, n: int, grid: int): + packed, base, exc_off, exc_pos, exc_w = coded + out = torch.full((x.shape[0], n), float("nan"), dtype=torch.bfloat16, device="cuda") + part = torch.zeros(grid * CTA_WARPS * 256, dtype=torch.float32, device="cuda") + counter = torch.zeros((n + 15) // 16, dtype=torch.int32, device="cuda") + return (packed, x, out, part, counter, exc_off, exc_pos, exc_w.view(-1), base, n, grid, True) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """The kernel against its reference and the in-tree BF16 GEMM: the error against fp64 is the in-tree GEMM's + (both are one BF16 rounding of an fp32 sum of the same exact products), the argmax is the in-tree one, nearly + every logit is the in-tree one bit for bit, and a second call gives the same bits.""" + spec = cell.spec + m, n, k = spec["m"], spec["n"], spec["k"] + exceptions = spec.get("exceptions", False) + w, coded = _coded(n, k, 1, exceptions) + if exceptions: + assert coded[3].numel() > 0, "the cell must exercise the exception lists" + g = torch.Generator(device="cuda").manual_seed(_seed(cell.why)) + x = torch.randn(m, k, device="cuda", generator=g).to(torch.bfloat16) + args = _args(coded, x, n, _num_sms()) + certifying(op) + ref = op.reference(*args) + out = op(*args).clone() + again = op(*_args(coded, x, n, _num_sms())).clone() + torch.cuda.synchronize() + assert not bool(args[4].any()), "the split-tile counters are not re-armed" + assert torch.equal(out, again), "the split-tile reduction is not deterministic" + stock = torch.nn.functional.linear(x, w) + exact = x.double() @ w.double().t() + # Each side: one BF16 rounding (<= half an ulp of the logit) of an fp32 sum whose order error is far below it. + err_out = (out.double() - exact).abs().max() + err_stock = (stock.double() - exact).abs().max() + err_ref = (ref.double() - exact).abs().max() + assert err_out <= 1.01 * max(err_stock, err_ref) + 1e-6, (err_out, err_stock, err_ref) + assert (out == stock).float().mean() > 0.99 + assert (out == ref).float().mean() > 0.99 + assert torch.equal(out.float().argmax(-1), stock.float().argmax(-1)) + + +@pytest.mark.parametrize("m", [1, 4, 8]) +def test_integer_data_is_the_bf16_head_bit_for_bit(m) -> None: + """Small-integer weights and activations: every fp32 sum is exact in any order, so the coded head's logits are + the in-tree BF16 GEMM's bit for bit (exceptions included: zeros and integers outside the exponent window).""" + g = torch.Generator(device="cuda").manual_seed(_seed(f"integer {m}")) + n, k = 4096, HIDDEN + w = torch.randint(-3, 4, (n, k), generator=g, device="cuda").to(torch.bfloat16) + w[::7, ::5] = 96.0 # far above the window of the {1, 2, 3} exponents: exceptions + x = torch.randint(-2, 3, (m, k), generator=g, device="cuda").to(torch.bfloat16) + coded = encode_chunked(w) + assert coded[3].numel() > 0 + assert round_trip_ok(*coded, w) + with validating(op): + out = op(*_args(coded, x, n, _num_sms())) + assert torch.equal(out, torch.nn.functional.linear(x, w)) + + +def test_the_head_writes_only_the_shards_columns() -> None: + """Integer data over a 4000-row shard: every logit is the in-tree GEMM's bit for bit, and the columns past n of a + wider output are left untouched.""" + n, k, m = 4000, 1024, 2 + g = torch.Generator(device="cuda").manual_seed(_seed("partial last tile")) + # Small integers: every fp32 sum is exact in any order, so the logits are comparable bit for bit. + w = torch.randint(-3, 4, (n, k), generator=g, device="cuda").to(torch.bfloat16) + w[::7, ::5] = 96.0 # exceptions, including in the partial tile + x = torch.randint(-2, 3, (m, k), generator=g, device="cuda").to(torch.bfloat16) + coded = encode_chunked(w) + assert round_trip_ok(*coded, w) + args = list(_args(coded, x, n, _num_sms())) + args[2] = torch.full((m, n + 16), float("nan"), dtype=torch.bfloat16, device="cuda") + with validating(op): + op(*args) + out = args[2] + assert torch.isnan(out[:, n:]).all(), "the head wrote past the shard's last column" + stock = torch.nn.functional.linear(x, w) + assert torch.equal(out[:, :n], stock) + + +def test_a_shard_of_partial_tiles_keeps_the_bf16_head() -> None: + """The kernel streams whole 16-row tiles: a shard whose rows are not a multiple of 16 is not taken (the fused + verify keeps the in-tree GEMM), and the op refuses it.""" + w = torch.zeros(4004, 1024, dtype=torch.bfloat16, device="cuda") + assert not target_head_kernels.head_supported(SimpleNamespace(weight=w, bias=None)) + assert target_head_kernels.head_supported(SimpleNamespace(weight=w[:4000], bias=None)) + + +@pytest.mark.parametrize("n,k", [(TARGET_SHARD, HIDDEN), (4004, 1024)]) +def test_round_trip_is_bitwise(n, k) -> None: + w, coded = _coded(n, k, 1, True) + assert coded[3].numel() > 0 + assert round_trip_ok(*coded, w) + assert torch.equal(decode(*coded, n, k).view(torch.int16), w.view(torch.int16)) + + +def test_a_corrupted_coding_fails_closed() -> None: + """A flipped sign/mantissa bit, a wrong exception value or a wrong base each fail the bitwise round trip.""" + w = _weight(4004, 1024, 2, exceptions=True) + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + assert round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w) + bad = packed.clone() + bad[100] ^= 1 + assert not round_trip_ok(bad, base, exc_off, exc_pos, exc_w, w) + bad_w = exc_w.clone() + bad_w[0, 0] += 1e-3 + assert not round_trip_ok(packed, base, exc_off, exc_pos, bad_w, w) + assert not round_trip_ok(packed, base + 1, exc_off, exc_pos, exc_w, w) + + +def test_prepare_head_keeps_only_a_bitwise_coding(monkeypatch) -> None: + """prepare_head keeps the coded shard when it round-trips and runs it as the entry; a coding that does not + round-trip is dropped (the in-tree GEMM stays the path).""" + w = _weight(4000, 1024, 3, exceptions=True) + head = target_head_kernels.prepare_head(SimpleNamespace(weight=w, bias=None)) + assert head.coded and head.weight is w + x = torch.randn(4, 1024, device="cuda").to(torch.bfloat16) + out = target_head_kernels.logits(head, x) + stock = torch.nn.functional.linear(x, w) + assert torch.equal(out.float().argmax(-1), stock.float().argmax(-1)) + assert (out == stock).float().mean() > 0.99 + + def corrupt(weight, *args, **kwargs): + packed, base, exc_off, exc_pos, exc_w = encode_chunked(weight) + packed = packed.clone() + packed[7] ^= 0x40 + return packed, base, exc_off, exc_pos, exc_w + + monkeypatch.setattr(target_head_kernels, "encode_chunked", corrupt) + dropped = target_head_kernels.prepare_head(SimpleNamespace(weight=w, bias=None)) + assert not dropped.coded and dropped.w is None + + +def test_guard_refuses_a_dirty_counter() -> None: + w, coded = _coded(4000, 1024, 1, True) + args = list(_args(coded, torch.zeros(2, 1024, dtype=torch.bfloat16, device="cuda"), 4000, 8)) + args[4][3] = 1 + with pytest.raises(AssertionError, match="counter must be zero"): + with validating(op): + op(*args) + + +def _tiny_exceptions_head(n: int = 4096, k: int = 1024) -> torch.Tensor: + """Weights of 1 (exponent window base 113) with column 0 zero except 2^-30 / 2^-29 on rows 0 / 1: exceptions far + below the window whose exact products are the only non-zero logits.""" + w = torch.ones((n, k), dtype=torch.bfloat16, device="cuda") + w[:, 0] = 0.0 + w[0, 0] = 2.0**-30 + w[1, 0] = 2.0**-29 + return w + + +@pytest.mark.parametrize("m", [1, 4, 8]) +def test_tiny_exception_products_are_exact(m) -> None: + """An exception adds exactly x * w: with a one-hot activation on column 0 the logits are 0, 2^-30 and 2^-29 bit + for bit as the BF16 head's, and row 1 wins (a correction against a non-zero placeholder cancels them to 0).""" + w = _tiny_exceptions_head() + coded = encode_chunked(w) + assert coded[1] == 113 and round_trip_ok(*coded, w) + x = torch.zeros((m, w.shape[1]), dtype=torch.bfloat16, device="cuda") + x[:, 0] = 1.0 + with validating(op): + out = op(*_args(coded, x, w.shape[0], _num_sms())) + stock = torch.nn.functional.linear(x, w) + assert torch.equal(out.view(torch.int16), stock.view(torch.int16)) + assert out.float().argmax(-1).tolist() == [1] * m + + +def test_a_shard_smaller_than_the_grid_is_stored() -> None: + """A shard with fewer coded units than warps of the requested grid: the launch narrows the grid so every warp owns + a unit, and every logit is stored.""" + w = _tiny_exceptions_head(n=256) + coded = encode_chunked(w) + x = torch.zeros((2, w.shape[1]), dtype=torch.bfloat16, device="cuda") + x[:, 0] = 1.0 + out = op(*_args(coded, x, w.shape[0], _num_sms())) + assert torch.equal(out.view(torch.int16), torch.nn.functional.linear(x, w).view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/_m3_moe_operands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/_m3_moe_operands.py new file mode 100644 index 000000000000..e47f54bfad65 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/_m3_moe_operands.py @@ -0,0 +1,170 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One tp4 rank's MiniMax-M3 MoE layer operands for the decode-engine entries' tests. + +The routed experts are random checkpoint-format NVFP4 tensors ([w3 (up); w1 (gate)] rows, e4m3 block scales) put +through TRT-LLM's own TRTLLM-Gen NVFP4 loader shuffle (``NVFP4TRTLLMGenFusedMoEMethod._shuffle_all_experts``), which +writes the buffers the engines read in place. The per-expert scalars follow the loader's conventions (act_beta and +act_clamp divided by fc31_alpha, fc31_scale_c = fc2_input_scale x fc31_alpha) and are sized so the FC1 sums land +around the SwiGLU-OAI clamp and the outputs near unit scale. The shared expert's MXFP8 weights come from +mxfp8_quantize with the swizzled scale layout, as the MXFP8 linear method stores them. +""" + +import functools +from types import SimpleNamespace + +import torch + +H, I_TP, NUM_EXPERTS, TOP_K = 6144, 768, 128, 4 +SHARED_I = 768 +SWIGLU_ALPHA, SWIGLU_LIMIT = 1.702, 7.0 +ROUTED_SCALING = 2.0 +MINIMAX2_ROUTING = 5 # RoutingMethodType.MiniMax2 +# NVFP4 global scales: 448 x 6 over the amax each side is sized for. +FC31_INPUT_SCALE = 448.0 * 6.0 / 8.0 +FC2_INPUT_SCALE = 448.0 * 6.0 / 24.0 + + +def _rand_e4m3_scales(shape, gen) -> torch.Tensor: + """E4M3 block scales in [0.5, 2): biased exponents 6 and 7, any mantissa.""" + return torch.randint(0x30, 0x40, shape, dtype=torch.uint8, device="cuda", generator=gen).view( + torch.float8_e4m3fn + ) + + +@functools.lru_cache(maxsize=None) +def experts(seed: int = 20261013): + """The routed experts and the router of one MoE layer, in the layout and conventions of the TRTLLM-Gen loader. + + Returns a SimpleNamespace: ``tensors`` (the ten expert operands in ``M3MoeState.layer``'s order), ``bias``, + ``router_weight``, ``fc31_input_scale`` and ``shared`` (gate_up, its scales, down, its scales). + """ + from tensorrt_llm._torch.moe.fused_moe.quantization import NVFP4TRTLLMGenFusedMoEMethod + + gen = torch.Generator(device="cuda").manual_seed(seed) + e = NUM_EXPERTS + w31 = torch.randint( + 0, 256, (e, 2 * I_TP, H // 2), dtype=torch.uint8, device="cuda", generator=gen + ) + w2 = torch.randint(0, 256, (e, H, I_TP // 2), dtype=torch.uint8, device="cuda", generator=gen) + w31s = _rand_e4m3_scales((e, 2 * I_TP, H // 16), gen) + w2s = _rand_e4m3_scales((e, H, I_TP // 16), gen) + module = SimpleNamespace( + w3_w1_weight=w31, + w2_weight=w2, + w3_w1_weight_scale=w31s, + w2_weight_scale=w2s, + is_gated_activation=True, + bias=False, + ) + NVFP4TRTLLMGenFusedMoEMethod()._shuffle_all_experts(module) + torch.cuda.synchronize() + + # Raw FC1 sums: rms(x) x FC31_INPUT_SCALE x rms(E2M1 value) x rms(block scale) x sqrt(H) ~ 1.0e5 at rms(x) = 1; + # fc31_alpha brings them to ~1.5 in the dequantized domain, the FC2 alpha the outputs to ~1. + base = 1.5 / (FC31_INPUT_SCALE * 2.93 * 1.3 * H**0.5) + fc31_alpha = base * (0.8 + 0.4 * torch.rand(e, device="cuda", generator=gen)) + fc2_alpha = (0.8 + 0.4 * torch.rand(e, device="cuda", generator=gen)) / ( + FC2_INPUT_SCALE * 2.93 * 1.3 * I_TP**0.5 + ) + tensors = ( + w31, + w31s, + w2, + w2s, + fc31_alpha, + FC2_INPUT_SCALE * fc31_alpha, # fc31_scale_c + torch.full((e,), SWIGLU_ALPHA, device="cuda"), # act_alpha + 1.0 / fc31_alpha, # act_beta: the (up + 1) offset over fc31_alpha + SWIGLU_LIMIT / fc31_alpha, # act_clamp over fc31_alpha + fc2_alpha, + ) + bias = (0.02 * torch.randn(e, device="cuda", generator=gen)).float() + router_weight = (torch.randn(e, H, device="cuda", generator=gen) / H**0.5).float().contiguous() + fc31_input_scale = torch.tensor([FC31_INPUT_SCALE], dtype=torch.float32, device="cuda") + + gu = (0.02 * torch.randn(2 * SHARED_I, H, device="cuda", generator=gen)).to(torch.bfloat16) + dn = (0.05 * torch.randn(H, SHARED_I, device="cuda", generator=gen)).to(torch.bfloat16) + gu_q, gu_sf = torch.ops.trtllm.mxfp8_quantize(gu, True) + dn_q, dn_sf = torch.ops.trtllm.mxfp8_quantize(dn, True) + shared = (gu_q.view(2 * SHARED_I, H), gu_sf, dn_q.view(H, SHARED_I), dn_sf) + return SimpleNamespace( + tensors=tensors, + bias=bias, + router_weight=router_weight, + fc31_input_scale=fc31_input_scale, + shared=shared, + ) + + +# Routing cases: random (up to 4 M distinct experts), and every token on the same four experts (the bias outweighs +# any sigmoid difference), which leaves later slot groups empty. +CONCENTRATED = (5, 40, 77, 126) + + +def layer_for(state, case: str, shared: bool = True): + """One layer on ``state`` over :func:`experts` for a routing case. ``shared=False``: the shared expert's weights + are zero, so the engine returns the routed part alone (bf16(0 + bf16(routed))).""" + ops = experts() + bias = ops.bias.clone() + if case == "concentrated": + bias[list(CONCENTRATED)] += 1.0 + shared_weights = ops.shared + if not shared: + gu, gu_sf, dn, dn_sf = ops.shared + shared_weights = (torch.zeros_like(gu), gu_sf, torch.zeros_like(dn), dn_sf) + return state.layer( + *ops.tensors, + bias, + router_weight=ops.router_weight, + fc31_input_scale=ops.fc31_input_scale, + shared_weights=shared_weights, + ) + + +def stock_routed( + hidden: torch.Tensor, topk_ids: torch.Tensor, topk_weights: torch.Tensor +) -> torch.Tensor: + """The stock TRTLLM-Gen NVFP4 MoE (fp4_quantize, then fp4_block_scale_moe_runner with SwiGLU-OAI) on the given + routing, over :func:`experts`: what the built-in MiniMaxM3MoE computes for the routed part.""" + import tensorrt_llm._torch.custom_ops # noqa: F401 -- registers the stock ops + + ops = experts() + w31, w31s, w2, w2s, fc31_alpha, scale_c, act_alpha, act_beta, act_clamp, fc2_alpha = ops.tensors + xq, xsf = torch.ops.trtllm.fp4_quantize(hidden, ops.fc31_input_scale, 16, False, False) + out = torch.ops.trtllm.fp4_block_scale_moe_runner( + None, None, xq, xsf.view(torch.float8_e4m3fn).flatten(), + w31, w31s, None, act_alpha, act_beta, act_clamp, w2, w2s, None, + scale_c, fc31_alpha, fc2_alpha, + NUM_EXPERTS, TOP_K, None, None, I_TP, 0, NUM_EXPERTS, ROUTED_SCALING, MINIMAX2_ROUTING, True, 0, + topk_weights=topk_weights.to(torch.bfloat16), topk_ids=topk_ids.to(torch.int32), + ) # fmt: skip + return out[0] if isinstance(out, (list, tuple)) else out + + +def hidden_states(m: int, seed: int) -> torch.Tensor: + gen = torch.Generator(device="cuda").manual_seed(seed) + return torch.randn(m, H, device="cuda", generator=gen).to(torch.bfloat16) + + +def check_routed_part(op, state, m: int, seed: int) -> None: + """Exact, independent of the reference: the engine's routing (its route_dbg record: ids, then the bf16 weights' + fp32 bits) is the reference routing bit for bit, and its routed part is the stock TRTLLM-Gen MoE's output on that + routing bit for bit -- the claim the engines are built on.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe_engine import ( # noqa: E501 + moe_route, + ) + + layer = layer_for(state, "random", shared=False) + hidden = hidden_states(m, seed=seed) + out = op(hidden, layer) + ids = state.route_dbg[:, :4].long() + weights = state.route_dbg[:, 4:].contiguous().view(torch.float32) + ref_ids, ref_weights = moe_route(hidden.float(), layer.router_weight, layer.bias) + assert torch.equal(ids, ref_ids) and torch.equal(weights, ref_weights), ( + "routing differs from the reference" + ) + stock = stock_routed(hidden, ids, weights) + assert torch.equal(out.view(torch.int16), stock.view(torch.int16)), ( + "routed part differs from TRTLLM-Gen's" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/_m3_router.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/_m3_router.py new file mode 100644 index 000000000000..e8beded4b53b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/_m3_router.py @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Shared operands and gates of the router entry's tests (m3_router_tc).""" + +import zlib + +import torch + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp # noqa: E402 — needs the path declared above + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_router_tc import ( # noqa: E501 + m3_router_tc as tc, +) + +__all__ = ["_build", "_compare_logits", "_seed", "tc"] + + +def _build(m: int, e: int, k: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + x = (torch.randn(m, k, generator=g, device="cuda") * 2.0).to(torch.bfloat16) + w = torch.randn(e, k, generator=g, device="cuda") * 0.02 + return x, w + + +def _seed(cell) -> int: + return zlib.crc32(cell.why.encode()) % 2**31 + + +def _compare_logits(out: torch.Tensor, ref: torch.Tensor) -> None: + """fp32 logits that differ from the reference only in summation order. + + Over K = 6144 products a reordered fp32 sum moves by a few ulps of the sum of magnitudes, which near + cancellation is ~sqrt(K) times the row's largest logit's ulp at most. Gated in fp32 ulps of each row's largest + logit: 64 element-wise, 8 in relative RMS. The tensor-core logits against a true fp32 product (the TF32 rounding + the band must see) sit at 2200+ / 1700 ulp. + """ + assert out.dtype == ref.dtype == torch.float32 and out.shape == ref.shape + assert_within_ulp(out, ref, element_ulp=64.0, rms_ulp=8.0) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m31_vr_tp4_moe_out_skip.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m31_vr_tp4_moe_out_skip.py new file mode 100644 index 000000000000..c12c8be8fe81 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m31_vr_tp4_moe_out_skip.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MOE-FINALIZE-DEADSTORE (OUT_SKIP, default on) on the sm_107 tp4 MoE engines (m3_moe and +m3_moe16): a call that pushes into the boundary all-reduce (arp enabled) does not store its bf16 rows into +``out`` (the pushed all-reduce never reads them); a call without a push stores as before; +TRTLLM_M31_DISABLE_MOE_OUT_SKIP=1 keeps the store. The pushed words are the same bits either way. The push target +is a plain device buffer laid out as set A of the multicast exchange (world 1, rank 0, parity 0): word t * H / 2 ++ c / 2 holds channels c, c + 1 of row t as bf16x2 (-0 pushed as +0).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_moe_operands import hidden_states, layer_for # noqa: E402 + +H = 6144 +EMPTY = -(2**31) # 0x80000000 +SENTINEL = 1234.0 + + +def _check_push(engine, state, num_tokens: int, bucket: int, killed: bool, counter=None) -> None: + dev = torch.device("cuda") + layer = layer_for(state, "random") + hidden = hidden_states(num_tokens, seed=11 + num_tokens) + ref = engine(hidden, layer).clone() # no push: stores out (either config) + torch.cuda.synchronize() + buf = torch.full( + (2 * 2 * bucket * H // 2,), EMPTY, dtype=torch.int32, device=dev + ) # sets A and B + flags = torch.zeros( + bucket * 8, dtype=torch.int32, device=dev + ) # parity 0 for every (row, AR CTA) + arp = moe_op.ar_push_desc(buf.data_ptr(), flags.data_ptr(), 0, 1, bucket, dev) + out = torch.full((num_tokens, H), SENTINEL, dtype=torch.bfloat16, device=dev) + counter = moe_op.OUT_SKIP_TAKEN if counter is None else counter + before = counter[0] + engine(hidden, layer, out, arp) + torch.cuda.synchronize() + assert counter[0] - before == int(not killed), ( + "the OUT_SKIP work counter counts pushing calls that skip" + ) + pushed = buf[: num_tokens * H // 2].view(torch.bfloat16).view(num_tokens, H) + ref_pos0 = torch.where(ref == 0, torch.zeros_like(ref), ref) # the push writes -0 as +0 + assert torch.equal(pushed.view(torch.int16), ref_pos0.view(torch.int16)), "pushed words changed" + assert bool((buf[num_tokens * H // 2 : bucket * H // 2] == EMPTY).all()), ( + "a push beyond the rows" + ) + if killed: + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)), ( + "the kill switch keeps the store" + ) + else: + assert bool((out == SENTINEL).all()), "a pushing call still stored out" + out2 = torch.full((num_tokens, H), SENTINEL, dtype=torch.bfloat16, device=dev) + engine(hidden, layer, out2, None) # a call without a push stores as before + torch.cuda.synchronize() + assert torch.equal(out2.view(torch.int16), ref.view(torch.int16)) + + +def _state(monkeypatch, num_tokens: int, killed: bool): + monkeypatch.setattr( + moe_op, "m31_disabled", lambda item_id, k=killed: k and item_id == "MOE_OUT_SKIP" + ) + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + return moe_op.M3MoeState(dev, num_tokens=num_tokens, num_ctas=sms - 24, shared=True) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("killed", [False, True]) +def test_config_and_kill_switch(monkeypatch: pytest.MonkeyPatch, killed: bool) -> None: + state = _state(monkeypatch, 8, killed) + assert state.config.get("out_skip", 0) == int(not killed) + assert bool(state.mod.OUT_SKIP) == (not killed) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("num_tokens", [1, 2, 3, 8]) +@pytest.mark.parametrize("killed", [False, True]) +def test_push_bits_unchanged_and_out_store_skipped( + monkeypatch: pytest.MonkeyPatch, num_tokens: int, killed: bool +) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe_engine import ( # noqa: E501 + m3_moe_engine, + ) + + _check_push(m3_moe_engine, _state(monkeypatch, num_tokens, killed), num_tokens, 8, killed) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("num_tokens", [9, 16]) +@pytest.mark.parametrize("killed", [False, True]) +def test_moe16_push_bits_unchanged_and_out_store_skipped( + monkeypatch: pytest.MonkeyPatch, num_tokens: int, killed: bool +) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.m3_moe import ( # noqa: E501 + op16, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe16_engine import ( # noqa: E501 + m3_moe16_engine, + ) + + monkeypatch.setattr( + op16, "m31_disabled", lambda item_id, k=killed: k and item_id == "MOE_OUT_SKIP" + ) + state = op16.M3Moe16State(torch.device("cuda"), num_tokens=num_tokens) + assert state.config.get("out_skip", 0) == int(not killed) + _check_push(m3_moe16_engine, state, num_tokens, 16, killed, op16.OUT_SKIP_TAKEN) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m31_vr_tp4_moe_tail_trigger.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m31_vr_tp4_moe_tail_trigger.py new file mode 100644 index 000000000000..9bb4430aeb8b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m31_vr_tp4_moe_tail_trigger.py @@ -0,0 +1,101 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MOE_TAIL_TRIGGER (default on; the TP8 port) on the sm_107 tp4 MoE engines (m3_moe and m3_moe16): +on a call that pushes into the boundary all-reduce, each routed CTA releases the PDL dependents when its weights +producer is done (the kernels' PROD_TRIGGER), not at teardown; TRTLLM_M31_DISABLE_MOE_TAIL_TRIGGER=1 restores the +teardown release. Scheduling only: the pushed words and ``out`` are the same bits either way, and every CTA's call +epoch (stored early with the release) advances exactly once per call. The push target is a plain device buffer laid +out as set A of the multicast exchange (world 1, rank 0, parity 0): word t * H / 2 + c / 2 holds channels c, c + 1 of +row t as bf16x2 (-0 pushed as +0).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_moe_operands import hidden_states, layer_for # noqa: E402 + +H = 6144 +EMPTY = -(2**31) # 0x80000000 + + +def _kill(monkeypatch, killed: bool) -> None: + monkeypatch.setattr( + moe_op, "m31_disabled", lambda item_id, k=killed: k and item_id == "MOE_TAIL_TRIGGER" + ) + + +def _check(engine, state, num_tokens: int, bucket: int, killed: bool) -> None: + dev = torch.device("cuda") + assert state.config.get("prod_trigger", 0) == int(not killed) + assert bool(state.mod.PROD_TRIGGER) == (not killed) + assert bool(state.mod.EARLY_EPOCH) == (not killed) + layer = layer_for(state, "random") + hidden = hidden_states(num_tokens, seed=31 + num_tokens) + ref = engine(hidden, layer).clone() # no push: the release point does not apply + torch.cuda.synchronize() + ref_pos0 = torch.where(ref == 0, torch.zeros_like(ref), ref) # the push writes -0 as +0 + for _ in range(2): + buf = torch.full((2 * 2 * bucket * H // 2,), EMPTY, dtype=torch.int32, device=dev) + flags = torch.zeros(bucket * 8, dtype=torch.int32, device=dev) + arp = moe_op.ar_push_desc(buf.data_ptr(), flags.data_ptr(), 0, 1, bucket, dev) + before_ep = state.epochs.clone() + before = moe_op.TAIL_TRIGGER_TAKEN[0] + out = engine(hidden, layer, None, arp) + torch.cuda.synchronize() + assert moe_op.TAIL_TRIGGER_TAKEN[0] - before == int(not killed) + assert torch.equal(state.epochs, before_ep + 1), ( + "every CTA's epoch advances exactly once per call" + ) + pushed = buf[: num_tokens * H // 2].view(torch.bfloat16).view(num_tokens, H) + assert torch.equal(pushed.view(torch.int16), ref_pos0.view(torch.int16)), ( + "pushed words changed" + ) + assert bool((buf[num_tokens * H // 2 : bucket * H // 2] == EMPTY).all()), ( + "a push beyond the rows" + ) + if not state.config.get("out_skip"): # (OUT_SKIP: a pushing call leaves out unwritten) + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)) + before_ep = state.epochs.clone() + again = engine(hidden, layer) # a call without a push after pushing ones + torch.cuda.synchronize() + assert torch.equal(state.epochs, before_ep + 1) + assert torch.equal(again.view(torch.int16), ref.view(torch.int16)) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("num_tokens", [1, 2, 4, 8]) +@pytest.mark.parametrize("killed", [False, True]) +def test_push_bits_and_epochs_unchanged( + monkeypatch: pytest.MonkeyPatch, num_tokens: int, killed: bool +) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe_engine import ( # noqa: E501 + m3_moe_engine, + ) + + _kill(monkeypatch, killed) + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + state = moe_op.M3MoeState(dev, num_tokens=num_tokens, num_ctas=sms - 24, shared=True) + _check(m3_moe_engine, state, num_tokens, 8, killed) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("num_tokens", [9, 16]) +@pytest.mark.parametrize("killed", [False, True]) +def test_moe16_push_bits_and_epochs_unchanged( + monkeypatch: pytest.MonkeyPatch, num_tokens: int, killed: bool +) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.m3_moe import ( # noqa: E501 + op16, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe16_engine import ( # noqa: E501 + m3_moe16_engine, + ) + + _kill(monkeypatch, killed) + state = op16.M3Moe16State(torch.device("cuda"), num_tokens=num_tokens) + _check(m3_moe16_engine, state, num_tokens, 16, killed) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe16_engine.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe16_engine.py new file mode 100644 index 000000000000..5171e46217c4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe16_engine.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the moe/m3_moe16_engine entry: the 9..13-row MoE engine on its caller-owned state. + +One GB300 rank of the tp4 deployment, the target's build (128 routed CTAs + 24 shared-expert CTAs). Operands and the +band: ``_m3_moe_operands`` and ``test_modeling_v2_m3_moe_engine`` (the same math). +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe16_engine import ( # noqa: E501 + M3Moe16State, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe16_engine import ( # noqa: E501 + m3_moe16_engine as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_moe_operands import H, check_routed_part, hidden_states, layer_for # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_moe16_engine requires a CUDA device" + +# As test_modeling_v2_m3_moe_engine.compare (same arithmetic, same quantization boundaries). +BAND_ELEMENT = 4.0 +BAND_RMS = 0.5 + +_STATES = {} + + +def _state(m: int) -> M3Moe16State: + if m not in _STATES: + _STATES[m] = M3Moe16State(torch.device("cuda"), num_tokens=m) + return _STATES[m] + + +@pytest.mark.parametrize("case", ("random", "concentrated")) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, case) -> None: + m = cell.spec["tokens"] + layer = layer_for(_state(m), case) + hidden = hidden_states(m, seed=zlib.crc32((cell.why + case).encode()) % 2**31) + with validating(op): + out = op(hidden, layer) + again = op(hidden, layer) + assert out.shape == (m, H) and out.dtype is torch.bfloat16 + assert torch.equal(out.view(torch.int16), again.view(torch.int16)), ( + "a repeated call changed the output" + ) + assert_within_ulp(out, op.reference(hidden, layer), element_ulp=BAND_ELEMENT, rms_ulp=BAND_RMS) + + +@pytest.mark.parametrize("m", (9, 13)) +def test_routed_part_is_trtllm_gen_bit_for_bit(m) -> None: + check_routed_part(op, _state(m), m, seed=100 + m) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe_engine.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe_engine.py new file mode 100644 index 000000000000..da6c5d5cc542 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe_engine.py @@ -0,0 +1,139 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the moe/m3_moe_engine entry: the 1..8-token MoE engine on its caller-owned state. + +One GB300 rank of the tp4 deployment: 128 experts (all local, intermediate 768 per rank), top 4, hidden 6144, the +shared expert folded in, ``num_ctas`` = SMs - 24 as the target builds it. Operands: ``_m3_moe_operands``. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe_engine import ( # noqa: E501 # noqa: E501 + M3MoeState, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.moe.m3_moe_engine import ( # noqa: E501 + m3_moe_engine as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_moe_operands import H, check_routed_part, hidden_states, layer_for # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_moe_engine requires a CUDA device" + +_STATES = {} + + +def _state(m: int) -> M3MoeState: + if m not in _STATES: + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + _STATES[m] = M3MoeState(dev, num_tokens=m, num_ctas=sms - 24, shared=True) + return _STATES[m] + + +def compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """The engine against the plain-torch MoE: BAND_ELEMENT bf16 ulp of the row's largest magnitude per element, + BAND_RMS ulp relative RMS. + + Both sides quantize the same operands with the same recipes (the activation's and the intermediate's NVFP4 codes + and block scales, the shared expert's MXFP8); they differ in summation order (tensor-core fp32 against fp64) and in + the PTX approximations (rcp / ex2 / tanh). Such a difference changes the output only where it moves a value across + a rounding boundary: one bf16 ulp of a slot output or of the sum, or, rarely, one intermediate NVFP4 code, which + reaches the output as one weight column's share. Measured over every cell, both routing cases and 3 seeds each + (GB300): at most 1.48 ulp element, 0.18 ulp RMS. The band leaves ~3x on both; a misread recipe (the activation's + 1 / (sf x (1 / sc)) spelling of the reciprocal on ~0.06% of its codes) measured 12 ulp element, 7 ulp RMS. + """ + assert_within_ulp(out, ref, element_ulp=BAND_ELEMENT, rms_ulp=BAND_RMS) + + +BAND_ELEMENT = 4.0 +BAND_RMS = 0.5 + + +@pytest.mark.parametrize("case", ("random", "concentrated")) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, case) -> None: + m = cell.spec["tokens"] + layer = layer_for(_state(m), case) + hidden = hidden_states(m, seed=zlib.crc32((cell.why + case).encode()) % 2**31) + with validating(op): + out = op(hidden, layer) + again = op(hidden, layer) + assert out.shape == (m, H) and out.dtype is torch.bfloat16 + # The workspace is tagged per call and never reset: a repeated call is the same call. + assert torch.equal(out.view(torch.int16), again.view(torch.int16)), ( + "a repeated call changed the output" + ) + compare(out, op.reference(hidden, layer)) + + +@pytest.mark.parametrize("m", (1, 2, 8)) +def test_routed_part_is_trtllm_gen_bit_for_bit(m) -> None: + check_routed_part(op, _state(m), m, seed=100 + m) + + +_GB_STATES = {} + + +@pytest.mark.parametrize("case", ("random", "concentrated")) +@pytest.mark.parametrize("m", range(1, 9)) +def test_engine_is_the_gb_tp4_engine_bit_for_bit(m, case) -> None: + """The target's engine (the Rubin m3_moe at the TP4 widths) against the GB TP4 engine it replaces, same operands + and CTA count: the per-expert accumulation order is the same, so the outputs are equal bit for bit.""" + # The shared GB300 engine leaves the tree once no target imports it. + gb_op = pytest.importorskip("tensorrt_llm._torch.cute_dsl_kernels.minimax_m3.moe.op") + if m not in _GB_STATES: + sms = torch.cuda.get_device_properties(0).multi_processor_count + _GB_STATES[m] = gb_op.M3MoeState( + torch.device("cuda"), num_tokens=m, num_ctas=sms - 24, shared=True + ) + hidden = hidden_states(m, seed=300 + m) + out = layer_for(_state(m), case)(hidden) + ref = layer_for(_GB_STATES[m], case)(hidden) + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)) + + +def test_layers_on_one_state_do_not_leak_into_each_other() -> None: + """Two layers share one state's workspace, in stream order: A, B, A gives A's output twice, bit for bit, and the + caller's ``out`` receives the same rows as a fresh tensor.""" + state = _state(4) + a, b = layer_for(state, "random"), layer_for(state, "concentrated") + hidden = hidden_states(4, seed=7) + first = op(hidden, a) + op(hidden, b) + out = torch.empty(4, H, dtype=torch.bfloat16, device="cuda") + second = op(hidden, a, out=out) + assert second.data_ptr() == out.data_ptr() + assert torch.equal(first.view(torch.int16), out.view(torch.int16)) + + +@pytest.mark.parametrize( + "bad", + ("misaligned hidden", "fp32 out", "short out", "long out", "int32 arp"), +) +def test_malformed_buffers_are_rejected_by_the_call(bad) -> None: + """Why the entry has no is_valid: the compiled call checks each buffer against its signature and raises.""" + state = _state(2) + layer = layer_for(state, "random") + hidden, kw = hidden_states(2, seed=3), {} + if bad == "misaligned hidden": + hidden = torch.randn(2 * H + 8, device="cuda").to(torch.bfloat16)[2 : 2 + 2 * H].view(2, H) + elif bad == "fp32 out": + kw["out"] = torch.empty(2, H, dtype=torch.float32, device="cuda") + elif bad == "short out": + kw["out"] = torch.empty(1, H, dtype=torch.bfloat16, device="cuda") + elif bad == "long out": + kw["out"] = torch.empty(3, H, dtype=torch.bfloat16, device="cuda") + else: + kw["arp"] = torch.zeros(8, dtype=torch.int32, device="cuda") + with pytest.raises(ValueError): + op(hidden, layer, **kw) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe_flat_gate_tp4.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe_flat_gate_tp4.py new file mode 100644 index 000000000000..09a2ec51c8a5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_moe_flat_gate_tp4.py @@ -0,0 +1,74 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The TP4 MoE engine's FLAT stack is applied only up to FLAT_MAX_TOKENS (4) tokens per call. + +At 8 tokens FLAT was +2.27 us/call slower than the per-m tables, so above the cap the engine keeps the per-m +configuration (bitwise the same outputs). CPU only: M3MoeState is constructed with its device +queries stubbed and stopped at the kernel build, so the test reads the engine config and the engagement lines +without a GPU. +""" + +import types + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.m3_moe import ( # noqa: E501 + op, +) + + +class _Built(Exception): + """Raised by the stubbed kernel build: the config is final by then.""" + + +def _engine(monkeypatch, num_tokens): + """(config, info lines) of M3MoeState(num_tokens) up to its kernel build.""" + lines = [] + seen = {} + + def build(config, path): + seen["config"] = dict(config) + raise _Built() + + monkeypatch.setattr(op, "is_supported", lambda device=None: (True, "")) + monkeypatch.setattr(op, "_kernel_module", build) + monkeypatch.setattr(op.torch.cuda, "is_current_stream_capturing", lambda: False) + monkeypatch.setattr( + op.torch.cuda, + "get_device_properties", + lambda device: types.SimpleNamespace(multi_processor_count=op.HIDDEN // 32), + ) + from tensorrt_llm.logger import logger + + monkeypatch.setattr(logger, "info", lambda msg, *a, **k: lines.append(str(msg))) + with pytest.raises(_Built): + op.M3MoeState("cuda", num_tokens, num_ctas=op.HIDDEN // 32) + return seen["config"], [line for line in lines if "MiniMax-M3 MoE engine FLAT (TP4)" in line] + + +def _has_flat(config): + return all(config.get(k) == v for k, v in op.FLAT_STACK.items()) + + +@pytest.mark.parametrize("num_tokens", range(1, 9)) +def test_flat_applies_only_up_to_the_cap(monkeypatch, num_tokens): + config, lines = _engine(monkeypatch, num_tokens) + assert op.FLAT_MAX_TOKENS == 4 + if num_tokens <= 4: + assert _has_flat(config), config + assert lines == [ + "MiniMax-M3 MoE engine FLAT (TP4): b2img=0 lg_repl=16 ar_pre=0 late_trigger=0 " + f"tokens={num_tokens}" + ] + else: + # above the cap: the per-m tables + assert config.get("b2img") == op.B2IMG_BY_M[num_tokens - 1] + assert config.get("ar_pre") == op.AR_PRE_BY_M[num_tokens - 1] + assert lines == [ + f"MiniMax-M3 MoE engine FLAT (TP4) off above 4 tokens (per-m tables): tokens={num_tokens}" + ] + + +def test_flat_applies_helper(): + """op.flat_applies: the FLAT stack up to FLAT_MAX_TOKENS tokens, the per-m tables above.""" + assert [op.flat_applies(m) for m in range(1, 9)] == [True] * 4 + [False] * 4 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_router_tc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_router_tc.py new file mode 100644 index 000000000000..e43c57f109a4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/moe/test_modeling_v2_m3_router_tc.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_router_tc entry of the MiniMax-M3 tp4 target: the TF32 tensor-core router of the unfused MoE's +9..16-row pure generation steps (the arithmetic the built-in gate takes at these rows).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog._op import true_fp32_matmul + +__extra_import_path__ = ["../../../..", "."] +from _m3_router import _build, _compare_logits, _seed, tc # noqa: E402 +from _validating import validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + + +@pytest.mark.parametrize("cell", tc.CELLS, ids=[c.why[:48] for c in tc.CELLS]) +def test_tc_certified_cells(cell) -> None: + spec = cell.spec + x, w = _build(spec["m"], 128, spec["k"], _seed(cell)) + with validating(tc): + outs = tc(x, w, None, False) + ref = tc.reference(x, w, None, False) + assert len(outs) == len(ref) == 1 + _compare_logits(outs[0], ref[0]) + + +def test_tc_is_the_built_in_gates_router_call() -> None: + """The target's entry is the op the built-in gate calls at these rows, with the same arguments: the same logits + bit for bit.""" + x, w = _build(14, 128, 6144, 7) + ours = tc(x, w, None, True)[0] + gate = torch.ops.trtllm.minimax_m3_router_tc(x, w, None, True)[0] + assert torch.equal(ours.view(torch.int32), gate.view(torch.int32)) + + +def test_tc_reference_is_tf32_and_the_gate_sees_it() -> None: + """The tensor-core router rounds the weight to TF32: against a true fp32 product its logits sit far outside + the band, so the TF32 rounding in ``reference`` is load-bearing rather than decoration.""" + x, w = _build(16, 128, 6144, 5) + out = tc(x, w, None, False)[0] + with true_fp32_matmul(): + fp32 = x.float() @ w.t() + with pytest.raises(AssertionError): + _compare_logits(out, fp32) + + +def test_guard_refuses_a_misaligned_activation() -> None: + """A row view starting 8 bytes into the buffer keeps a legal row stride but not a legal bulk-copy source.""" + x, w = _build(9, 128, 6144, 3) + flat = torch.empty(9 * 6144 + 4, dtype=torch.bfloat16, device="cuda") + shifted = flat[4:].view(9, 6144) + shifted.copy_(x) + with pytest.raises(AssertionError, match="16-byte aligned"): + with validating(tc): + tc(shifted, w, None, False) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/norm/test_modeling_v2_m31_ctxnormq.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/norm/test_modeling_v2_m31_ctxnormq.py new file mode 100644 index 000000000000..86a55c66a4f6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/norm/test_modeling_v2_m31_ctxnormq.py @@ -0,0 +1,101 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for CTXNORMQ: the prefill boundary's residual add + RMSNorm that also writes the next projection's MXFP8 +activation and the routed experts' NVFP4 input (torch.ops.trtllm.minimax_m31_ctx_residual_rmsnorm_mxfp8). + +The quantized outputs must equal mxfp8_quantize / fp4_quantize of the op's own norm bit for bit; the norm and the +updated residual are checked against an fp32 reference. Skipped when the op is not built (no overlay library). +""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + modeling as m, +) + +assert torch.cuda.is_available(), "CTXNORMQ requires a CUDA device" + +H = 6144 +EPS = 1e-6 +FP4_SCALE = 448.0 * 6.0 / 8.0 + +op = m._ctxnormq_op() +pytestmark = pytest.mark.skipif( + op is None, reason="minimax_m31_ctx_residual_rmsnorm_mxfp8 is not built" +) + + +def _operands(rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + inter = torch.randn(rows, H, generator=g, device="cuda").to(torch.bfloat16) + residual = (2 * torch.randn(rows, H, generator=g, device="cuda")).to(torch.bfloat16) + weight = (1.0 + 0.1 * torch.randn(H, generator=g, device="cuda")).to(torch.bfloat16) + return inter, residual, weight + + +@pytest.mark.parametrize("rows", [1024, 1160, 4096]) +@pytest.mark.parametrize("fp4", [False, True], ids=["attn", "moe"]) +def test_quantized_outputs_equal_quantizing_the_norm(rows, fp4) -> None: + inter, residual, weight = _operands(rows, 20261021 + rows) + scale = torch.tensor([FP4_SCALE], device="cuda", dtype=torch.float32) if fp4 else None + ref_updated = (inter.float() + residual.float()).to(torch.bfloat16) + normed, q, sf, f4, f4_sf = op(inter, residual, weight, EPS, True, scale, False) + # The updated residual is written in place into inter: one bf16 rounding of the fp32 sum, exact. + assert torch.equal(inter.view(torch.int16), ref_updated.view(torch.int16)) + u = ref_updated.float() + ref_normed = u * torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + EPS) * weight.float() + # fp32 sum of squares + fast rsqrt vs torch's: at most one bf16 rounding apart. + torch.testing.assert_close(normed.float(), ref_normed, rtol=2**-7, atol=2**-7) + q_ref, sf_ref = torch.ops.trtllm.mxfp8_quantize(normed, True) + assert torch.equal(q.view(torch.uint8), q_ref.view(torch.uint8)) + assert torch.equal(sf.view(torch.uint8)[: sf_ref.numel()], sf_ref.view(torch.uint8).flatten()) + if fp4: + f4_ref, f4_sf_ref = torch.ops.trtllm.fp4_quantize(normed, scale, 16, False, False) + assert torch.equal(f4.view(torch.uint8).flatten(), f4_ref.view(torch.uint8).flatten()) + assert torch.equal( + f4_sf.view(torch.uint8).flatten(), + f4_sf_ref.view(torch.uint8).flatten()[: f4_sf.numel()], + ) + # _routed_chunks splits the NVFP4 scales per row chunk. + assert f4_sf.numel() == rows * H // 16 + + +def test_deterministic() -> None: + outs = [] + for _ in range(3): + inter, residual, weight = _operands(2048, 7) + scale = torch.tensor([FP4_SCALE], device="cuda", dtype=torch.float32) + outs.append((inter,) + tuple(op(inter, residual, weight, EPS, True, scale, False))) + for o in outs[1:]: + assert all( + torch.equal(a.view(torch.uint8), b.view(torch.uint8)) for a, b in zip(outs[0], o) + ) + + +def test_small_boundaries_keep_the_fused_allreduce() -> None: + """Below CTXNORMQ_MIN_ROWS the boundary is not taken (the microbench crossover); decided before any collective.""" + x = torch.zeros(m.CTXNORMQ_MIN_ROWS - 1, H, dtype=torch.bfloat16, device="cuda") + assert m._ctxnormq_allreduce_norm(None, x, x, x[0], None) is None + + +@pytest.mark.parametrize("fp4", [False, True], ids=["attn", "moe"]) +@pytest.mark.parametrize("write_norm", [False, True]) +def test_fake_impl_matches_the_op(fp4, write_norm) -> None: + """The registered fake (catalog m31_fake_ops) gives every output the op's shape and dtype.""" + from torch._subclasses.fake_tensor import FakeTensorMode + + inter, residual, weight = _operands(1160, 7) + scale = torch.tensor([FP4_SCALE], device="cuda") if fp4 else None + real = op(inter.clone(), residual, weight, EPS, write_norm, scale, False) + with FakeTensorMode(allow_non_fake_inputs=True) as mode: + fake = op( + mode.from_tensor(inter), + mode.from_tensor(residual), + mode.from_tensor(weight), + EPS, + write_norm, + None if scale is None else mode.from_tensor(scale), + False, + ) + assert [(tuple(t.shape), t.dtype) for t in fake] == [(tuple(t.shape), t.dtype) for t in real] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/norm/test_modeling_v2_m3_embedding_firstnorm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/norm/test_modeling_v2_m3_embedding_firstnorm.py new file mode 100644 index 000000000000..d9e418552ce3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/norm/test_modeling_v2_m3_embedding_firstnorm.py @@ -0,0 +1,76 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the norm/m3_embedding_firstnorm entry: the decode embedding gather + the first layer's RMSNorm. + +The table is the model's full bf16 [200064, 6144] embedding shape (random values at the checkpoint's scale), eps the +checkpoint's 1e-6. +""" + +import functools +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.norm.m3_embedding_firstnorm import ( # noqa: E501 + VOCAB, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.norm.m3_embedding_firstnorm import ( # noqa: E501 + m3_embedding_firstnorm as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_embedding_firstnorm requires a CUDA device" + +H = 6144 +EPS = 1e-6 + + +@functools.lru_cache(maxsize=None) +def _operands(): + g = torch.Generator(device="cuda").manual_seed(20261013) + table = (0.02 * torch.randn(VOCAB, H, generator=g, device="cuda")).to(torch.bfloat16) + weight = (1.0 + 0.1 * torch.randn(H, generator=g, device="cuda")).to(torch.bfloat16) + return table, weight + + +def _buffers(rows): + return (torch.empty(rows, H, dtype=torch.bfloat16, device="cuda"), + torch.empty(rows, H, dtype=torch.bfloat16, device="cuda")) # fmt: skip + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + rows, id_dtype = cell.spec["rows"], cell.spec["id_dtype"] + table, weight = _operands() + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + ids = torch.randint(0, VOCAB, (rows,), generator=g, device="cuda").to(id_dtype) + raw, out = _buffers(rows) + with validating(op): + normed, residual = op(ids, table, weight, EPS, raw, out) + assert normed.data_ptr() == out.data_ptr() and residual.data_ptr() == raw.data_ptr() + ref_normed, ref_raw = op.reference(ids, table, weight, EPS, raw, out) + # The gather is a copy: exact. + assert torch.equal(raw.view(torch.int16), ref_raw.view(torch.int16)) + # The norm: the kernel's fp32 sum of squares (warp butterflies, then across 4 warps) and fast rsqrt against an + # fp64 reference, then x * rstd * w rounded once to bf16. The fp32 side is ~1e-6 relative off, so the two roundings + # disagree only next to a bf16 rounding boundary: one ulp at most per element, a small fraction in aggregate. + # Measured over these cells (3 seeds each, GB300): at most 0.51 ulp element, 0.026 ulp RMS. + assert_within_ulp(out, ref_normed, element_ulp=1.0, rms_ulp=0.1) + + +def test_ids_outside_the_table_give_zero_rows() -> None: + """Documented, not refused: an id outside [0, V) gathers a zero row whose norm is zero.""" + table, weight = _operands() + ids = torch.tensor([-1, 5, VOCAB], dtype=torch.int64, device="cuda") + raw, out = _buffers(3) + op(ids, table, weight, EPS, raw, out) + assert not raw[0].any() and not raw[2].any() and not out[0].any() and not out[2].any() + assert torch.equal(raw[1], table[5]) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/sampling/test_modeling_v2_m31_topp_det.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/sampling/test_modeling_v2_m31_topp_det.py new file mode 100644 index 000000000000..41b6c02dda12 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/sampling/test_modeling_v2_m31_topp_det.py @@ -0,0 +1,271 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for TOPP_DET: the spec sampler's top-p renorm as +``torch.ops.trtllm.minimax_m31_topp_renorm_det`` through ``ops/sampling/m31_topp_det``, installed as the sampler's +top-p renorm override. + +On rows with top_p < 1 it must equal FlashInfer's ``top_p_renorm_probs(probs, top_p, is_deterministic=True)`` bit for +bit (the microbench generators: randn / peaked / ties / near_uniform / all_equal rows); rows with top_p >= 1 keep every +token (an approved exception). The shared sampler hook must be byte-for-byte stock while no override is set, and +fall back to the stock call whenever the override declines. TRTLLM_M31_DISABLE_TOPP_DET=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.sampling import ( # noqa: E501 + m31_topp_det as entry, +) +from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as fi_ops + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4" +) +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal") +FLT_MIN = torch.finfo(torch.float32).tiny +op = entry.m31_topp_det + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("TOPP_DET is built for sm_107 here") + if entry.topp_det_op() is None: + pytest.skip("minimax_m31_topp_renorm_det is not built") + pytest.importorskip("flashinfer.sampling") + + +def _make(cls, rows, d, seed): + """The microbench row generators: softmax of the class's logits.""" + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + logits = torch.randn(rows, d, generator=g, device="cuda") * 3 + elif cls == "peaked": # few dominant tokens, long tail (real-like draft rows) + logits = torch.randn(rows, d, generator=g, device="cuda") * 2 + idx = torch.randint(0, d, (rows, 8), generator=g, device="cuda") + logits.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + elif cls == "ties": + logits = torch.round(torch.randn(rows, d, generator=g, device="cuda") * 2) / 2 + elif cls == "near_uniform": + logits = torch.randn(rows, d, generator=g, device="cuda") * 1e-3 + elif cls == "all_equal": + logits = torch.zeros(rows, d, device="cuda") + else: + raise ValueError(cls) + return torch.softmax(logits, -1).contiguous() + + +def _top_p(mode, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed + 7) + if mode == "p95": + return torch.full((rows,), 0.95, device="cuda") + if mode == "mixed": + return (0.05 + 0.95 * torch.rand(rows, generator=g, device="cuda")).contiguous() + raise ValueError(mode) + + +def _fi_det(probs, top_p): + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + +def _same(a, b, what): + """Bit for bit.""" + assert torch.equal(a.view(torch.int32), b.view(torch.int32)), f"{what}: fp32 bits differ" + + +@pytest.mark.parametrize("mode", ["p95", "mixed"]) +@pytest.mark.parametrize("cls", CLASSES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_flashinfer_det_below_one(cell, cls, mode) -> None: + """Rows with top_p < 1 at each certified (rows, d): the override equals FlashInfer's det renorm bit for bit, and + the explicit entry call gives the same bits again (deterministic).""" + _gpu() + rows, d = cell.spec["rows"], cell.spec["d"] + for seed in range(2): + probs = _make(cls, rows, d, 1000 * rows + seed) + top_p = _top_p(mode, rows, seed) + got = entry.top_p_renorm_det(probs, top_p) + assert got is not None, "an in-contract input must be served" + ref = _fi_det(probs, top_p) + _same(got, ref, f"{cls} {mode} rows={rows} seed={seed}") + again = op( + probs, + top_p, + entry.prebuild(probs.device), + entry.ctas_per_row(rows, entry.sm_count(probs.device)), + ) + _same(again, got, "repeat") + + +@pytest.mark.parametrize("cls", CLASSES) +def test_keep_all_rows_in_a_mixed_batch(cls) -> None: + """A batch of 8 rows, a third at top_p 1: the top_p 1 rows keep every element whose renormed value is a normal float + (output nonzero <=> ftz(p * norm) != 0, the --use_fast_math semantics FlashInfer's apply shares) and sum to 1 within + 1e-5; the top_p < 1 rows are FlashInfer's det renorm bit for bit.""" + _gpu() + rows, d = 8, 200064 + probs = _make(cls, rows, d, 77) + top_p = _top_p("mixed", rows, 3) + keep = torch.zeros(rows, dtype=torch.bool, device="cuda") + keep[::3] = True + top_p[keep] = 1.0 + got = entry.top_p_renorm_det(probs, top_p) + assert got is not None + ref = _fi_det(probs, top_p) + _same(got[~keep], ref[~keep], f"{cls}: top_p < 1 rows") + for r in keep.nonzero().flatten().tolist(): + p64 = probs[r].double() + norm = 1.0 / float(p64[probs[r] >= FLT_MIN].sum()) + # kept <=> ftz(p * norm) != 0; elements within 1e-6 of the FLT_MIN boundary are left to the rounding + live = p64 * norm >= FLT_MIN * (1 + 1e-6) + dead = probs[r] < FLT_MIN + assert bool((got[r][live] != 0).all()), f"{cls} row {r}: a kept token was dropped" + assert bool((got[r][dead] == 0).all()), f"{cls} row {r}: a sub-FLT_MIN token is nonzero" + total = float(got[r].double().sum()) + assert abs(total - 1.0) <= 1e-5, f"{cls} row {r}: sum {total}" + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager bits.""" + _gpu() + probs = _make("peaked", 8, 200064, 5) + top_p = _top_p("mixed", 8, 5) + entry.prebuild(probs.device) + ref = entry.top_p_renorm_det(probs, top_p) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.top_p_renorm_det(probs, top_p) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.top_p_renorm_det(probs, top_p) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """A float top_p, a non-fp32 / misaligned / non-contiguous probs, a d % 4 width, a mis-sized top_p or more rows + than the workspace holds return None (the sampler runs stock FlashInfer).""" + _gpu() + d = 200064 + probs = _make("randn", 4, d, 1) + top_p = torch.full((4,), 0.9, device="cuda") + assert entry.top_p_renorm_det(probs, 0.9) is None + assert entry.top_p_renorm_det(probs.half(), top_p) is None + assert entry.top_p_renorm_det(probs, top_p.double()) is None + assert entry.top_p_renorm_det(probs, top_p[:3]) is None + assert entry.top_p_renorm_det(probs[:, : d - 2], top_p) is None + assert entry.top_p_renorm_det(probs[:, : d - 2].contiguous(), top_p) is None + flat = torch.empty(4 * d + 1, device="cuda") + assert entry.top_p_renorm_det(flat[1:].view(4, d), top_p) is None + wide = _make("randn", entry.MAX_ROWS + 1, d, 2) + assert ( + entry.top_p_renorm_det(wide, torch.full((entry.MAX_ROWS + 1,), 0.9, device="cuda")) is None + ) + assert entry.ctas_per_row(1, 148) == 16 and entry.ctas_per_row(8, 148) == 16 + assert entry.ctas_per_row(16, 148) == 8 and entry.ctas_per_row(64, 148) == 2 + assert entry.ctas_per_row(200, 148) == 1 + + +@pytest.fixture +def _fake_flashinfer(monkeypatch): + """The sampler ops module with a stand-in flashinfer whose calls are recorded, and no override afterwards.""" + stock = torch.full((2, 8), 0.125) + sampling = types.SimpleNamespace( + top_p_renorm_probs=mock.Mock(return_value=stock), + softmax=mock.Mock(side_effect=lambda logits, temps, enable_pdl=False: logits.softmax(-1)), + top_k_mask_logits=mock.Mock(side_effect=lambda logits, top_k: logits), + ) + monkeypatch.setattr(fi_ops, "flashinfer", types.SimpleNamespace(sampling=sampling)) + fi_ops.set_top_p_renorm_override(None) + yield sampling, stock + fi_ops.set_top_p_renorm_override(None) + + +def test_hook_unset_is_stock(_fake_flashinfer) -> None: + """No override: both ops make exactly the stock flashinfer call and return its result.""" + sampling, stock = _fake_flashinfer + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops._TOP_P_RENORM_OVERRIDE is None + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is stock + sampling.top_p_renorm_probs.assert_called_once_with(probs, top_p) + assert ( + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is stock + ) + assert sampling.top_p_renorm_probs.call_count == 2 + + +def test_hook_override_result_is_used(_fake_flashinfer) -> None: + """An override's tensor replaces the stock call in both ops; the stock kernel does not run.""" + sampling, _ = _fake_flashinfer + mine = torch.zeros(2, 8) + override = mock.Mock(return_value=mine) + fi_ops.set_top_p_renorm_override(override) + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is mine + override.assert_called_once_with(probs, top_p) + assert fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is mine + assert override.call_count == 2 + sampling.top_p_renorm_probs.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_flashinfer) -> None: + """An override that returns None leaves the stock call in place; without top_p the override is never asked.""" + sampling, stock = _fake_flashinfer + override = mock.Mock(return_value=None) + fi_ops.set_top_p_renorm_override(override) + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is stock + assert ( + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is stock + ) + assert override.call_count == 2 and sampling.top_p_renorm_probs.call_count == 2 + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, None) + assert override.call_count == 2 + + +def test_the_target_installs_the_override() -> None: + """With TOPP_DET on (the default) and the op built, the target installs ``top_p_renorm_det`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, + ) + + if not modeling.TOPP_DET: + pytest.skip("TOPP_DET is switched off in this environment") + saved = dict(modeling._TOPP_DET_STATE) + modeling._TOPP_DET_STATE.clear() + try: + modeling._install_topp_det() + assert fi_ops._TOP_P_RENORM_OVERRIDE is entry.top_p_renorm_det + assert modeling._TOPP_DET_STATE["installed"] is True + finally: + fi_ops.set_top_p_renorm_override(None) + modeling._TOPP_DET_STATE.clear() + modeling._TOPP_DET_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_TOPP_DET=1 (read at import): TOPP_DET is off and the target leaves the sampler hook unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as f\n" + "m._install_topp_det()\n" + "print(m.TOPP_DET, f._TOP_P_RENORM_OVERRIDE is None, m._TOPP_DET_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_TOPP_DET="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/_m3_spec_inputs.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/_m3_spec_inputs.py new file mode 100644 index 000000000000..a2b4b89e3774 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/_m3_spec_inputs.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Inputs shared by the tp4 target's spec entry tests: TP4 dimensions, single-device stand-ins for the cross-rank +exchanges, and the comparison bands.""" + +from __future__ import annotations + +import zlib + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_shard_accept import ( # noqa: E501 + SHARD_ROWS, + make_keys, +) + +#: MiniMax-M3 at TP4: hidden size, vocabulary, the target head's shard and RMSNorm epsilon. +HIDDEN = 6144 +VOCAB = 200064 +WORLD = 4 +TARGET_SHARD = VOCAB // WORLD +EPS = 1e-6 + + +def seed_of(why: str) -> int: + return zlib.crc32(why.encode()) % 2**31 + + +def gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def num_sms() -> int: + return torch.cuda.get_device_properties(torch.cuda.current_device()).multi_processor_count + + +def randn(shape, g: torch.Generator, std: float = 1.0, dtype=torch.bfloat16) -> torch.Tensor: + return (torch.randn(shape, generator=g, device="cuda") * std).to(dtype) + + +def head_workspace(world: int, call: int = 0, slot_rows: int = SHARD_ROWS): + """(ws_uc int64 [2 * slot_rows * world], keys int64 [slot_rows], counters int32 [2]) of a fresh call ``call``.""" + uc = torch.zeros(2 * slot_rows * world, dtype=torch.int64, device="cuda") + keys = torch.zeros(slot_rows, dtype=torch.int64, device="cuda") + counters = torch.tensor([0, call], dtype=torch.int32, device="cuda") + return uc, keys, counters + + +def plant_peers( + uc: torch.Tensor, + counters: torch.Tensor, + own_best: torch.Tensor, + rank: int, + world: int, + shard: int, + g: torch.Generator, + slot_rows: int = SHARD_ROWS, + ties: bool = False, +) -> None: + """Write the other ranks' keys for each row into this rank's slots, as their multicast pushes would. + + Each peer's maximum is the own row maximum shifted by +-0.25 (well clear of the accumulation-order band), so + some rows are won by a peer and some by this rank; with ``ties`` a peer on an even row reports exactly the own + maximum, and the lower global index must win. + """ + if world == 1: + return + m = own_best.shape[0] + half = int(counters[1].item()) & 1 + for r in range(world): + if r == rank: + continue + sign = torch.where(torch.rand(m, generator=g, device="cuda") < 0.5, -1.0, 1.0) + value = own_best.float() + 0.25 * sign + if ties: + value = torch.where(torch.arange(m, device="cuda") % 2 == 0, own_best.float(), value) + index = r * shard + torch.randint(0, shard, (m,), generator=g, device="cuda") + keys = make_keys(value, index) + for t in range(m): + uc[(half * slot_rows + t) * world + r] = keys[t] + + +def assert_bf16_close(out: torch.Tensor, ref: torch.Tensor, max_frac: float = 1e-3) -> None: + """At most one bf16 ulp per element, and only on a ``max_frac`` share of the elements. + + Kernel and reference compute the same fp32 expression and round once; they differ only where a different + summation order (or rsqrtf) puts the fp32 value on the other side of a bf16 rounding boundary. + """ + assert out.shape == ref.shape and out.dtype == ref.dtype == torch.bfloat16 + o, r = out.float(), ref.float() + ulp = torch.ldexp(torch.ones_like(r), torch.frexp(r).exponent - 8) + ulp = torch.where(r == 0, torch.full_like(r, 2.0**-133), ulp) + diff = (o - r).abs() + assert bool((diff <= ulp * 1.0001).all()), f"max |diff| / ulp = {(diff / ulp).max().item():.2f}" + frac = (diff > 0).float().mean().item() + assert frac <= max_frac, f"{frac:.2e} of the elements moved by one ulp (allowed {max_frac})" + + +def craft_drafts(tokens: torch.Tensor, nreq: int, d: int) -> torch.Tensor: + """Drafts matching request b's verify tokens for its first (b + 2) % (d + 1) positions and differing from + there on, so the acceptance counts vary across requests.""" + t = tokens.view(nreq, d + 1)[:, :d].clone() + for b in range(nreq): + n = (b + 2) % (d + 1) + t[b, n:] += 1 + return t.contiguous() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_n3a.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_n3a.py new file mode 100644 index 000000000000..cb426e6537f5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_n3a.py @@ -0,0 +1,473 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""N3 A2 in this target's DSpark worker: the fused strict acceptance and next-step tokens equal the in-tree path +(N3 P and A1, the verify logits, at the end). + +The worker's two overrides are driven on a bare worker (no model) against the in-tree methods of its base class on +the same inputs: +- eligible steps (K3 / K7, padded drafts, graph-padding dummy rows) are bitwise; +- forced acceptance (fractional, so the RNG pool path runs) over a run that alternates eligible and ineligible + steps keeps the same accepted counts and the same RNG counter as an all-in-tree run; +- the sampler runs exactly once per step; +- the switch and an ineligible step take the in-tree path. +""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import strict_accept +from tensorrt_llm._torch.speculative.dspark import DSparkWorker + +TARGET = "minimax_m31_nvfp4__sm_107__tp4" +dspark = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.dspark" +) + +VOCAB = 200064 + +assert torch.cuda.is_available(), "the N3 A2 worker test requires a CUDA device" + + +def _worker(force: float = 0.0): + w = dspark.M3DSparkWorker.__new__(dspark.M3DSparkWorker) + nn.Module.__init__(w) + w.force_num_accepted_tokens = force + w._force_accept_rng_counter = None + w._force_accept_rng_pool = None + w.calls = 0 + return w + + +def _metadata(ragged: bool = False, graph: bool = False): + return SimpleNamespace(is_ragged_verify=ragged, is_cuda_graph=graph) + + +def _fixed_sampler(tokens: torch.Tensor): + """The in-tree sampler stands in as a fixed function of the logits: the overrides must call it once.""" + + def sample(self, logits, spec_metadata, num_contexts, batch_size): + self.calls += 1 + return tokens + + return sample + + +def _step(n_req: int, draft_len: int, gen: torch.Generator, pad: bool = False, int64: bool = False): + k1 = draft_len + 1 + tokens = torch.randint(0, VOCAB, (n_req * k1,), generator=gen, device="cuda") + tokens = tokens if int64 else tokens.to(torch.int32) + draft = tokens.view(n_req, k1)[:, :draft_len].clone().to(torch.int32) + cut = torch.randint(0, draft_len + 1, (n_req,), generator=gen, device="cuda") + pos = torch.arange(draft_len, device="cuda").unsqueeze(0) + draft = torch.where(pos == cut.unsqueeze(1), (draft + 1) % VOCAB, draft) + if pad: + draft[::2, draft_len // 2 :] = -1 + logits = torch.zeros( + n_req * k1, 8, device="cuda" + ) # shape only: the sampler stand-in ignores the values + return logits, draft, tokens + + +def _in_tree(w, logits, draft, num_contexts, batch_size, md): + return DSparkWorker._sample_and_accept_draft_tokens_base( + w, logits, draft, num_contexts, batch_size, md + ) + + +@pytest.fixture(autouse=True) +def _a1_opted_in(monkeypatch): + """A1 is on by default (switch TRTLLM_M31_DISABLE_A1); pin it on so the environment cannot turn it off here + (test_prebuild_a1_off turns it off again).""" + monkeypatch.setattr(dspark, "A1_DISABLED", False) + + +@pytest.fixture(autouse=True) +def _compiled(): + strict_accept.precompile((3, 7), torch.device("cuda")) + + +@pytest.mark.parametrize( + "n_req,draft_len,pad,int64", + [ + (1, 3, False, False), + (8, 3, False, False), + (12, 3, True, False), + (8, 7, False, False), + (8, 3, False, True), + ], +) +def test_eligible_steps_are_bitwise(monkeypatch, n_req, draft_len, pad, int64): + gen = torch.Generator(device="cuda").manual_seed(3) + logits, draft, tokens = _step(n_req, draft_len, gen, pad=pad, int64=int64) + monkeypatch.setattr(dspark.M3DSparkWorker, "_sample_tokens_for_batch", _fixed_sampler(tokens)) + w, ref = _worker(), _worker() + md = _metadata() + assert w._n3a_eligible(logits, draft, 0, n_req, md) + acc, num = w._sample_and_accept_draft_tokens_base(logits, draft, 0, n_req, md) + racc, rnum = _in_tree(ref, logits, draft, 0, n_req, md) + assert w.calls == 1 and ref.calls == 1 + assert acc.dtype == racc.dtype and num.dtype == rnum.dtype + assert torch.equal(acc, racc) and torch.equal(num, rnum) + + +def test_forced_acceptance_alternating_steps_matches_in_tree(monkeypatch): + """Fractional forced AL (the RNG pool path) over eligible, ragged (ineligible) and graph-warmup steps: the + counts and the RNG counter follow the all-in-tree run step for step.""" + gen = torch.Generator(device="cuda").manual_seed(5) + w, ref = _worker(force=1.6), _worker(force=1.6) + served = 0 + for step in range(12): + logits, draft, tokens = _step(8, 3, gen) + num_contexts, batch_size = 0, 8 + if step % 3 == 2: + # An ineligible step: one context row ahead of the 8 generation requests (the in-tree path runs). + ctx_token = torch.randint(0, VOCAB, (1,), generator=gen, device="cuda").to(tokens.dtype) + tokens = torch.cat([ctx_token, tokens]) + logits = torch.zeros(tokens.numel(), 8, device="cuda") + num_contexts, batch_size = 1, 9 + monkeypatch.setattr( + dspark.M3DSparkWorker, "_sample_tokens_for_batch", _fixed_sampler(tokens) + ) + md = _metadata(graph=(step == 4)) + eligible = w._n3a_eligible(logits, draft, num_contexts, batch_size, md) + assert eligible == (num_contexts == 0), step + served += int(eligible) + acc, num = w._sample_and_accept_draft_tokens_base( + logits, draft, num_contexts, batch_size, md + ) + racc, rnum = _in_tree(ref, logits, draft, num_contexts, batch_size, md) + # A context row defines only column 0 (the in-tree path leaves the rest uninitialized). + assert torch.equal(acc[:num_contexts, 0], racc[:num_contexts, 0]), step + assert torch.equal(acc[num_contexts:], racc[num_contexts:]) and torch.equal(num, rnum), step + if ref._force_accept_rng_counter is not None: + assert torch.equal(w._force_accept_rng_counter, ref._force_accept_rng_counter), step + assert served == 8 + # A ragged verify step is declined (the in-tree ragged path keeps it). + logits, draft, _ = _step(8, 3, gen) + assert not w._n3a_eligible(logits, draft, 0, 8, _metadata(ragged=True)) + + +def test_ineligible_and_switch_take_the_in_tree_path(monkeypatch): + gen = torch.Generator(device="cuda").manual_seed(9) + logits, draft, tokens = _step(4, 3, gen) + w = _worker() + # A context row, a row-count mismatch, too many requests: declined. + assert not w._n3a_eligible(logits, draft, 1, 4, _metadata()) + assert not w._n3a_eligible(logits[:-1], draft, 0, 4, _metadata()) + big_logits, big_draft, _ = _step(strict_accept.max_requests() + 1, 3, gen) + assert not w._n3a_eligible(big_logits, big_draft, 0, big_draft.shape[0], _metadata()) + monkeypatch.setattr(dspark, "N3A_DISABLED", True) + assert not w._n3a_eligible(logits, draft, 0, 4, _metadata()) + + +@pytest.mark.parametrize( + "draft_len,wide,shuffle", [(3, False, False), (3, True, True), (7, False, True)] +) +def test_next_new_tokens_matches_in_tree(draft_len, wide, shuffle): + gen = torch.Generator(device="cuda").manual_seed(13) + n_req = 8 + width = draft_len + 1 + (4 if wide else 0) + accepted = torch.randint( + 0, VOCAB, (n_req, width), generator=gen, device="cuda", dtype=torch.int32 + ) + num = torch.randint(1, draft_len + 2, (n_req,), generator=gen, device="cuda", dtype=torch.int32) + batch_idx = torch.arange(32, device="cuda", dtype=torch.int32) + if shuffle: + batch_idx[:n_req] = torch.randperm(n_req, generator=gen, device="cuda").to(torch.int32) + next_draft = torch.randint( + 0, VOCAB, (n_req, draft_len), generator=gen, device="cuda", dtype=torch.int32 + ) + w = _worker() + out = w._prepare_next_new_tokens(accepted, next_draft, batch_idx, n_req, num) + ref = DSparkWorker._prepare_next_new_tokens(w, accepted, next_draft, batch_idx, n_req, num) + assert out.dtype == ref.dtype and torch.equal(out, ref) + + +# N3 P-gid + P-cat: the worker's target logits equal the logits processor's. + + +@pytest.mark.parametrize("tp", [2, 4, 8]) +def test_rank_major_strided_copy_equals_vocab_concat_and_cast(tp): + """The all-gather along dim 0 + one strided copy writes what the vocab concat + .float() writes.""" + gen = torch.Generator(device="cuda").manual_seed(tp) + rows, shard = 32, 25008 + shards = [ + torch.randn(rows, shard, generator=gen, device="cuda").to(torch.bfloat16) for _ in range(tp) + ] + gathered = torch.cat(shards, dim=0) # what the dim-0 all-gather returns, rank-major + out = torch.empty((rows, tp * shard), dtype=torch.float32, device="cuda") + out.view(rows, tp, shard).copy_(gathered.view(tp, rows, shard).transpose(0, 1)) + assert torch.equal(out, torch.cat(shards, dim=-1).float()) + + +def _lm_head(vocab=50016, hidden=6144): + from tensorrt_llm._torch.modules.embedding import LMHead + from tensorrt_llm._torch.modules.linear import TensorParallelMode + from tensorrt_llm.mapping import Mapping + + head = LMHead( + vocab, + hidden, + dtype=torch.bfloat16, + mapping=Mapping(world_size=1, tp_size=1, rank=0), + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=True, + ).cuda() + with torch.no_grad(): + head.weight.copy_( + torch.randn_like(head.weight, dtype=torch.float32).to(torch.bfloat16) * 0.02 + ) + return head + + +def test_target_logits_equal_the_processor_at_tp1(): + from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor + + head = _lm_head() + rows = 32 + hidden = torch.randn(rows, 6144, device="cuda").to(torch.bfloat16) + md = SimpleNamespace( + is_ragged_verify=False, gather_ids=torch.arange(rows, device="cuda", dtype=torch.int32) + ) + attn = SimpleNamespace(num_contexts=0) + out = _worker().target_logits(hidden, attn, md, head) + ref = LogitsProcessor().forward(hidden[md.gather_ids], head, attn, True) + assert out is not None and out.dtype == ref.dtype and torch.equal(out, ref) + + +def test_target_logits_decline_to_the_processor(monkeypatch): + head = _lm_head(vocab=1024, hidden=256) + hidden = torch.randn(8, 256, device="cuda").to(torch.bfloat16) + ids = torch.arange(8, device="cuda", dtype=torch.int32) + w = _worker() + # A context row, a gather that is not every row, a ragged verify, a padded head: the processor runs. + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=1), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids[:4]), + head, + ) + is None + ) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=True, gather_ids=ids), + head, + ) + is None + ) + monkeypatch.setattr(head, "padding_size", 3) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + monkeypatch.setattr(head, "padding_size", 0) + monkeypatch.setattr(dspark, "N3P_DISABLED", True) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + + +# N3 A1: the verify logits' all-gather + concat + cast in one launch (catalog/comm/allgather_cast) when set up. + + +def _a1_step(monkeypatch, rows: int, compiled: bool = True): + """A TP4-shaped call: the head reports tp 4, A1 is set up, the op and the NCCL all-gather are recorded.""" + head = _lm_head(vocab=1024, hidden=256) + monkeypatch.setattr(head, "tp_size", 4) + calls = {"a1": 0, "nccl": 0} + sentinel = torch.full((rows, 4 * 1024), 7.0, device="cuda") + + def fake_a1(shard, **ws): + assert ws == {"ws": "a1"} and shard.shape == (rows, 1024) + calls["a1"] += 1 + return sentinel + + def fake_allgather(shard, mapping, dim=0): + calls["nccl"] += 1 + return torch.cat([shard] * 4, dim=0) + + monkeypatch.setattr(dspark._allgather_cast, "allgather_cast", fake_a1) + monkeypatch.setattr(dspark._allgather_cast, "is_compiled", lambda *a, **k: compiled) + monkeypatch.setattr(dspark, "allgather", fake_allgather) + w = _worker() + w._a1 = {"ws": "a1"} + hidden = torch.randn(rows, 256, device="cuda").to(torch.bfloat16) + md = SimpleNamespace( + is_ragged_verify=False, gather_ids=torch.arange(rows, device="cuda", dtype=torch.int32) + ) + out = w.target_logits(hidden, SimpleNamespace(num_contexts=0), md, head) + return out, sentinel, calls + + +@pytest.mark.parametrize("rows", [4, 32, 64]) +def test_target_logits_take_a1_when_set_up(monkeypatch, rows): + out, sentinel, calls = _a1_step(monkeypatch, rows) + assert out is sentinel and calls == {"a1": 1, "nccl": 0} + + +@pytest.mark.parametrize("rows,compiled", [(68, True), (4, False)]) +def test_target_logits_fall_back_to_nccl_past_a1(monkeypatch, rows, compiled): + """More rows than the workspace holds, or a shape never compiled: the NCCL all-gather + strided copy.""" + out, _, calls = _a1_step(monkeypatch, rows, compiled=compiled) + assert ( + calls == {"a1": 0, "nccl": 1} + and out.shape == (rows, 4 * 1024) + and out.dtype == torch.float32 + ) + + +def _a1_prebuild_worker(monkeypatch, tp: int = 4, draft_len: int = 3): + w = _worker() + w.mapping = SimpleNamespace(tp_size=tp, tp_rank=0, enable_attention_dp=False) + w.spec_config = SimpleNamespace(max_draft_len=draft_len) # max_draft_len reads it + # Eligibility needs every SM up to the workspace's grid; any CUDA device passes with a 1-CTA workspace grid. + monkeypatch.setattr(dspark._allgather_cast, "MAX_GRID", 1) + monkeypatch.setattr(dspark.M3DSparkWorker, "_a1_all_ok", lambda self, local: local) + return w + + +def test_prebuild_a1_sets_up_k3_rows(monkeypatch): + head = _lm_head(vocab=1024, hidden=256) + seen = {} + + def fake_precompile(mapping, cols, rows, device, group=None, agree=None): + seen.update(cols=cols, rows=rows, agreed=agree(True)) + return True + + ws = {"ws_uc": torch.zeros(4, dtype=torch.int32), "rank": 0} + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr( + dspark._allgather_cast, "workspace_args", lambda mapping, cols, group=None: ws + ) + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is ws + # K3 verify rows of batches 1..16, all within the workspace's 64 rows. + assert seen == {"cols": 1024, "rows": tuple(range(4, 65, 4)), "agreed": True} + + +@pytest.mark.parametrize("draft_len", [3, 7]) +def test_prebuild_a1_sets_up_the_verify_rows_of_k3_and_k7(monkeypatch, draft_len): + """K3: rows 4..64 (batches 1..16); K7, the headline: rows 8..64 (batches 1..8).""" + head = _lm_head(vocab=1024, hidden=256) + seen = {} + + def fake_precompile(mapping, cols, rows, device, group=None, agree=None): + seen.update(rows=rows, agreed=agree(True)) + return True + + ws = {"ws_uc": torch.zeros(4, dtype=torch.int32), "rank": 0} + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr( + dspark._allgather_cast, "workspace_args", lambda mapping, cols, group=None: ws + ) + assert _a1_prebuild_worker(monkeypatch, draft_len=draft_len)._prebuild_a1(head) is ws + k1 = draft_len + 1 + assert seen == {"rows": tuple(range(k1, 65, k1)), "agreed": True} + + +def test_prebuild_a1_off(monkeypatch): + """The switch, another draft length, one rank, a rank that is not eligible, or a compile that failed on any rank: + no A1, no workspace.""" + head = _lm_head(vocab=1024, hidden=256) + built = {"n": 0} + + def fake_precompile(*a, **k): + built["n"] += 1 + return False # a failed compile somewhere + + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr(dspark, "A1_K7_DISABLED", True) + assert _a1_prebuild_worker(monkeypatch, draft_len=7)._prebuild_a1(head) is None + monkeypatch.setattr(dspark, "A1_K7_DISABLED", False) + assert ( + _a1_prebuild_worker(monkeypatch, draft_len=64)._prebuild_a1(head) is None + ) # 65 rows at batch 1 + assert _a1_prebuild_worker(monkeypatch, tp=1)._prebuild_a1(head) is None + assert built["n"] == 0 + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is None + assert built["n"] == 1 + # Another rank not eligible: off here too, nothing compiled. + w = _a1_prebuild_worker(monkeypatch) + monkeypatch.setattr(dspark.M3DSparkWorker, "_a1_all_ok", lambda self, local: False) + assert w._prebuild_a1(head) is None and built["n"] == 1 + monkeypatch.setattr(dspark, "A1_DISABLED", True) + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is None + assert built["n"] == 1 + + +# N3A-KV: the draft-forward KV bookkeeping in one launch, through the DFlash hook (bitwise to the stock lines). + + +def _kv_step(nc: int, bs: int, gen: torch.Generator): + num = torch.randint(1, 5, (bs + 2,), generator=gen, device="cuda", dtype=torch.int32) + kv = torch.randint(0, 1 << 20, (bs + 2,), generator=gen, device="cuda", dtype=torch.int32) + return num, kv + + +def _kv_worker(fused: bool): + w = _worker() + w._fused_kv_rewind = None + if fused: + strict_accept.precompile_kv_rewind(torch.device("cuda")) + w._n3a_kv_buf = torch.empty(strict_accept.max_requests(), dtype=torch.int32, device="cuda") + w._fused_kv_rewind = w._n3a_kv_rewind + return w + + +@pytest.mark.parametrize("nc,bs", [(0, 1), (0, 8), (2, 6), (3, 3)]) +def test_kv_rewind_hook_equals_the_stock_lines(nc, bs): + gen = torch.Generator(device="cuda").manual_seed(5) + num, kv = _kv_step(nc, bs, gen) + outs = [] + for fused in (False, True): + w = _kv_worker(fused) + attn = SimpleNamespace(kv_lens_cuda=kv.clone(), update_for_spec_dec=lambda: None) + w._prepare_kv_for_draft_forward(attn, num, nc, bs) + outs.append( + ( + w._kv_rewind_amount, + attn.kv_lens_cuda, + w._kv_rewind_nc, + w._kv_rewind_bs, + w._kv_rewind_pending, + ) + ) + (r0, k0, *rest0), (r1, k1, *rest1) = outs + assert torch.equal(k0, k1) and rest0 == rest1 + assert r0.dtype == r1.dtype and r0.shape == r1.shape and torch.equal(r0, r1) + + +def test_kv_rewind_hook_declines_unsupported_operands(): + w = _kv_worker(True) + num64 = torch.ones(4, dtype=torch.int64, device="cuda") # not the buffer's dtype + kv = torch.zeros(4, dtype=torch.int32, device="cuda") + assert w._n3a_kv_rewind(num64, kv, 0, 4) is None and torch.equal(kv, torch.zeros_like(kv)) + w._n3a_kv_buf = None + assert w._n3a_kv_rewind(torch.ones(4, dtype=torch.int32, device="cuda"), kv, 0, 4) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_s4.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_s4.py new file mode 100644 index 000000000000..1227ebad00d9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_s4.py @@ -0,0 +1,347 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp4 target's worker-side routing of the one-launch Markov chain (``dspark_s4.S4WorkerMixin``), without GPUs +or a model: which steps it serves, that the kill switch sets nothing up, that a declined step is the in-tree path, that +the step's logits processor only takes the block rows, and that the chain's tokens replace the in-tree sampling for +that one call. The chain itself is the catalog's ``spec_decode/dspark_markov_chain``, certified by +``spec_decode/_dspark_markov_chain_op_matrix.py``; the in-model equality by the call-site gate.""" + +import importlib +import os +import subprocess +import sys +from types import SimpleNamespace + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + dspark_s4 as w1, +) + + +class _Base: + def __init__(self): + self.calls = [] + + def _forward_impl(self, *args): + self.calls.append("forward") + return "in-tree" + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + self.calls.append("refine") + return gen_logits + + def sample_draft_tokens(self, logits, spec_metadata, batch_size, **kwargs): + self.calls.append("sample") + return torch.full(logits.shape[:2], -1, dtype=torch.int32) + + def _apply_kv_rewind_after_draft(self, attn_metadata, spec_metadata): + self.calls.append("stock_rewind") + self._kv_rewind_pending = False + + +class _Worker(w1.S4WorkerMixin, _Base): + pass + + +def _state(block=3, shard=8, rows=8): + st = w1._S4State(None, rows, shard, 0, None, None, block, {}) + for b in range(1, w1.MAX_FAST_BATCH + 1): + st.bufs[(b, block)] = torch.full((b, block, rows), float("-inf")) + return st + + +def _meta(advanced=False): + return SimpleNamespace(wants_advanced_draft_sampling=advanced) + + +def test_serves_only_eligible_steps(monkeypatch): + wk = _Worker() + draft = SimpleNamespace(lilicorr=None, has_candidate_selector=False) + assert not wk._s4_serves(1, _meta(), draft) # not set up + wk._s4 = _state() + assert wk._s4_serves(1, _meta(), draft) + assert wk._s4_serves(w1.MAX_FAST_BATCH, _meta(), draft) + assert not wk._s4_serves(0, _meta(), draft) + assert not wk._s4_serves(w1.MAX_FAST_BATCH + 1, _meta(), draft) + assert not wk._s4_serves(1, _meta(advanced=True), draft) + wk._d2t = torch.zeros(1) + assert not wk._s4_serves(1, _meta(), draft) + + +def test_kill_switch_sets_nothing_up(monkeypatch): + wk = _Worker() + wk._s4 = _state() + monkeypatch.setattr(w1, "_DISABLED", True) + assert not wk.prebuild_s4(None, SimpleNamespace(has_markov_head=True)) + assert wk._s4 is None + assert not wk._s4_serves( + 1, _meta(), SimpleNamespace(lilicorr=None, has_candidate_selector=False) + ) + + +def test_declined_step_is_the_in_tree_path(): + wk = _Worker() + attn = SimpleNamespace(num_seqs=12, num_contexts=0) # above MAX_FAST_BATCH + draft = SimpleNamespace(lilicorr=None, has_candidate_selector=False, logits_processor="proc") + wk._s4 = _state() + assert wk._forward_impl(None, None, None, None, attn, _meta(), draft) == "in-tree" + assert draft.logits_processor == "proc" + logits = torch.zeros(2, 3, 16) # full-vocab width: not this step's shard + assert wk._refine_block_logits(draft, logits, {}, None) is logits + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() + assert wk.calls == ["forward", "refine", "sample"] + + +def test_sharded_logits_processor_takes_only_block_rows(): + st = _state(block=3, shard=8, rows=12) + inner_calls = [] + + def inner(h, head, meta, ctx): + inner_calls.append(h.shape) + return "inner" + + head = SimpleNamespace(apply_linear=lambda h, b: h[:, :8] * 2) + proc = w1._ShardedDraftLogits(inner, st, batch=2) + h = torch.arange(6 * 10, dtype=torch.float32).view(6, 10) + out = proc(h, head, None, True) + assert torch.equal(out, h[:, :8] * 2) + buf = st.buffer(2) + assert torch.equal(buf[:, :, :8], (h[:, :8] * 2).view(2, 3, 8)) + assert torch.isinf(buf[:, :, 8:]).all() + assert proc(torch.zeros(5, 10), head, None, True) == "inner" and inner_calls == [ + torch.Size([5, 10]) + ] + + +def test_chain_tokens_replace_sampling_once(monkeypatch): + wk = _Worker() + st = _state(block=3, shard=8, rows=8) + wk._s4_step = st + toks = torch.tensor([[1, 2, 3], [4, 5, 6]]) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, + dspark_markov_chain=lambda *a, **k: toks.int(), + ll_exchange_engaged=lambda batch: False)) # fmt: skip + logits = torch.zeros(2, 3, 8) + assert wk._refine_block_logits(None, logits, {"first_prev_tokens": torch.zeros(2, dtype=torch.long)}, None) \ + is logits # fmt: skip + out = wk.sample_draft_tokens(logits, None, 2) + assert out.dtype == torch.int32 and torch.equal(out, toks.int()) + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() # used once + assert "refine" not in wk.calls + + +def test_unsupported_shapes_fall_back_on_the_shards(monkeypatch): + wk = _Worker() + wk._s4_step = _state(block=3, shard=8, rows=8) + + def _never(*a, **k): + raise AssertionError("an unsupported step must not reach the op") + + monkeypatch.setattr( + w1, "chain", SimpleNamespace(supports=lambda *a: False, dspark_markov_chain=_never) + ) + logits = torch.zeros(2, 3, 8) + wk._refine_block_logits( + None, logits, {"first_prev_tokens": torch.zeros(2, dtype=torch.long)}, None + ) + assert wk.calls == ["refine"] + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() + + +def test_switch_reads_any_value_but_empty_or_zero(monkeypatch): + """Read at import, like the targets' m31_switches: "true" turns S4 off, not only "1".""" + try: + for value, off in ( + ("1", True), + ("true", True), + ("yes", True), + ("0", False), + ("", False), + (" 0 ", False), + ): + monkeypatch.setenv(w1.DISABLE_ENV, value) + importlib.reload(w1) + assert w1.disabled() is off, value + finally: + monkeypatch.delenv(w1.DISABLE_ENV, raising=False) + importlib.reload(w1) + assert not w1.disabled() + + +def _rewind_worker(monkeypatch, batch=2, nc=1, on=True): + """A worker in a served step with a pending KV-length rewind of ``batch`` rows from ``nc``, and a chain fake that + records which launch ran.""" + wk = _Worker() + wk.mapping = SimpleNamespace(tp_rank=5) + wk._s4_step = _state(block=3, shard=8, rows=8) + wk._s4_step_attn = SimpleNamespace(kv_lens_cuda=torch.zeros(nc + batch + 1, dtype=torch.int32)) + wk._kv_rewind_pending = True + wk._kv_rewind_amount = torch.zeros(batch, dtype=torch.int32) + wk._kv_rewind_nc, wk._kv_rewind_bs = nc, nc + batch + launches, lines = [], [] + toks = torch.zeros(batch, 3, dtype=torch.int32) + + def plain(*a, **k): + launches.append("chain") + return toks + + def with_rewind(base, first_prev, w1_, w2s, offset, kv_lens, amount, first, **k): + launches.append( + ( + "rewind", + kv_lens is wk._s4_step_attn.kv_lens_cuda, + amount is wk._kv_rewind_amount, + first, + ) + ) + return toks + + monkeypatch.setattr(w1, "TAILBK_SPEC", on) + monkeypatch.setattr(w1.logger, "info", lambda msg, *a, **k: lines.append(msg)) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, supports_rewind=lambda *a: True, + dspark_markov_chain=plain, + dspark_markov_chain_rewind=with_rewind, + ll_exchange_engaged=lambda b: False)) # fmt: skip + return wk, launches, lines, {"first_prev_tokens": torch.zeros(batch, dtype=torch.long)} + + +def test_tailbk_h_rewind_in_chain_launch(monkeypatch): + """TAILBK_SPEC H: a real step's chain launch applies the pending rewind (this step's kv_lens, amounts, first + row) and the stock rewind lines are skipped for that step; the engagement line is logged once per rank.""" + wk, launches, lines, inputs = _rewind_worker(monkeypatch) + meta = SimpleNamespace(is_cuda_graph=False) + for _ in range(2): + wk._kv_rewind_pending = True + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert not wk._kv_rewind_pending and not wk._tailbk_rewound + assert launches == [("rewind", True, True, 1)] * 2, launches + assert "stock_rewind" not in wk.calls + assert [m for m in lines if "TAILBK_SPEC H" in m] == [ + "[M3 dispatch] TAILBK_SPEC H: KV-length rewind in the Markov chain launch on rank 5" + ] + + +def test_tailbk_h_warmup_keeps_stock_path(monkeypatch): + """The eager warmup of a graph batch (is_cuda_graph, not capturing): the stock method skips the rewind there, so + the chain must not apply it either.""" + wk, launches, _, inputs = _rewind_worker(monkeypatch) + meta = SimpleNamespace(is_cuda_graph=True) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def test_tailbk_h_off_or_mismatch_keeps_stock_path(monkeypatch): + """Kill switch off, no pending rewind, or amounts that are not this chain call's rows: the stock rewind runs.""" + meta = SimpleNamespace(is_cuda_graph=False) + wk, launches, _, inputs = _rewind_worker(monkeypatch, on=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + wk, launches, _, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_pending = False + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert launches == ["chain"] + wk, launches, _, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_amount = torch.zeros(3, dtype=torch.int32) # not the 2 rows of this call + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def test_tailbk_h_unsupported_operands_keep_stock_path(monkeypatch): + """supports_rewind False (e.g. int64 amounts): the plain chain launch and the stock rewind.""" + wk, launches, _, inputs = _rewind_worker(monkeypatch) + monkeypatch.setattr(w1.chain, "supports_rewind", lambda *a: False) + meta = SimpleNamespace(is_cuda_graph=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def _tailbk_in_subprocess(value): + env = {k: v for k, v in os.environ.items() if k != "TRTLLM_M31_DISABLE_TAILBK_SPEC"} + if value is not None: + env["TRTLLM_M31_DISABLE_TAILBK_SPEC"] = value + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_107__tp4 import dspark_s4 as m; print('TAILBK_SPEC=' + str(m.TAILBK_SPEC))" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=600 + ) + assert out.returncode == 0, out.stderr + return [ln for ln in out.stdout.splitlines() if ln.startswith("TAILBK_SPEC=")][-1] + + +def test_tailbk_spec_kill_switch_subprocess(): + """TAILBK_SPEC is on by default; TRTLLM_M31_DISABLE_TAILBK_SPEC=1 (read at import) turns it off.""" + assert _tailbk_in_subprocess(None) == "TAILBK_SPEC=True" + assert _tailbk_in_subprocess("0") == "TAILBK_SPEC=True" + assert _tailbk_in_subprocess("1") == "TAILBK_SPEC=False" + + +def test_tailbk_c_engagement_logged_once_per_rank(monkeypatch): + """TAILBK_SPEC C's "[M3 dispatch]" line: once per rank, only after a chain call took the cached inputs.""" + wk, launches, lines, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_pending = False # the plain chain launch + taken = [False] + monkeypatch.setattr(w1.chain, "cached_inputs_taken", lambda: taken[0], raising=False) + meta = SimpleNamespace(is_cuda_graph=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert not [m for m in lines if "TAILBK_SPEC C" in m] + taken[0] = True + for _ in range(3): + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._kv_rewind_pending = True # and through the rewind launch + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert [m for m in lines if "TAILBK_SPEC C" in m] == [ + "[M3 dispatch] TAILBK_SPEC C: chain accepted inputs from precompile cache (taken) on rank 5" + ] + + +def test_markov_ll_engagement_logged_once_per_rank(monkeypatch): + """MARKOV_LL's "[M3 dispatch]" line: once per rank, on the first served step that took the LL kernel.""" + wk = _Worker() + wk.mapping = SimpleNamespace(tp_rank=3) + wk._s4_step = _state(block=3, shard=8, rows=8) + lines = [] + monkeypatch.setattr(w1.logger, "info", lambda msg, *a, **k: lines.append(msg)) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, + dspark_markov_chain=lambda *a, **k: torch.zeros( + a[0].shape[0], 3, dtype=torch.int32), + ll_exchange_engaged=lambda batch: batch == 1)) # fmt: skip + inputs2 = {"first_prev_tokens": torch.zeros(2, dtype=torch.long)} + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs2, None) # batch 2: not the LL kernel + assert not [m for m in lines if "MARKOV_LL" in m] + inputs1 = {"first_prev_tokens": torch.zeros(1, dtype=torch.long)} + for _ in range(3): + wk._refine_block_logits(None, torch.zeros(1, 3, 8), inputs1, None) + ll = [m for m in lines if "MARKOV_LL" in m] + assert ll == [ + "[M3 dispatch] MARKOV_LL: k3_markov LL flag-in-payload exchange (batch 1) on rank 3" + ], ll + + +def _markov_ll_in_subprocess(value): + env = {k: v for k, v in os.environ.items() if k != "TRTLLM_M31_DISABLE_MARKOV_LL"} + if value is not None: + env["TRTLLM_M31_DISABLE_MARKOV_LL"] = value + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_107__tp4 import dspark_s4 as m; print('MARKOV_LL=' + str(m.MARKOV_LL))" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=600 + ) + assert out.returncode == 0, out.stderr + return [ln for ln in out.stdout.splitlines() if ln.startswith("MARKOV_LL=")][-1] + + +def test_markov_ll_kill_switch_subprocess(): + """MARKOV_LL is on by default; TRTLLM_M31_DISABLE_MARKOV_LL=1 (read at import) turns it off.""" + assert _markov_ll_in_subprocess(None) == "MARKOV_LL=True" + assert _markov_ll_in_subprocess("0") == "MARKOV_LL=True" + assert _markov_ll_in_subprocess("1") == "MARKOV_LL=False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_w0.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_w0.py new file mode 100644 index 000000000000..01fc0de89e64 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_dspark_w0.py @@ -0,0 +1,171 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-model DSpark on the tp4 target, phase W0 (``dspark.py``): the shell's calls on the DSpark worker take no +fused path, the capture is the in-tree DFlash capture, the drafter's borrowed embedding is the target's exact gather, +and the FP8 draft-KV override (an Eagle3 need) is never applied to a DSpark run, so the drafter's context pool keeps +the projection dtype (BF16) and binds to the separate draft KV cache. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + dspark, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_target_head, +) +from tensorrt_llm.quantization.mode import QuantAlgo + + +def _spec( + *, dspark_mode: bool, eagle3: bool = False, one_engine: bool = True, embedded: bool = False +): + mode = SimpleNamespace( + use_one_engine=lambda: one_engine, + is_dspark=lambda: dspark_mode, + is_eagle3_one_model=lambda: eagle3, + ) + return SimpleNamespace(spec_dec_mode=mode, draft_is_embedded_in_target=embedded) + + +def _worker() -> dspark.M3DSparkWorker: + # The shell-facing calls read no worker state, so the worker is not constructed (that needs a mapping and a + # DSpark spec config). + return dspark.M3DSparkWorker.__new__(dspark.M3DSparkWorker) + + +@pytest.mark.parametrize( + "spec, expected", + [ + (_spec(dspark_mode=True), True), + (_spec(dspark_mode=True, embedded=True), False), + (_spec(dspark_mode=True, one_engine=False), False), + (_spec(dspark_mode=False, eagle3=True), False), + (None, False), + ], +) +def test_standalone_dspark_detection(spec, expected): + assert dspark.is_standalone_dspark(spec) is expected + + +def test_fp8_draft_kv_override_is_eagle3_only(): + assert dspark.draft_kv_override_applies(_spec(dspark_mode=False, eagle3=True)) + assert dspark.draft_kv_override_applies(None) + assert not dspark.draft_kv_override_applies(_spec(dspark_mode=True)) + + +@pytest.mark.parametrize("target_algo", [QuantAlgo.NVFP4, QuantAlgo.FP8, None]) +def test_drafter_kv_pool_keeps_the_drafter_dtype(target_algo): + from tensorrt_llm.models.modeling_utils import QuantConfig + + whole, layer = ( + QuantConfig(kv_cache_quant_algo=target_algo), + QuantConfig(kv_cache_quant_algo=target_algo), + ) + _ = whole.quant_mode # cached before the drop, as a loaded config may have it + cfg = SimpleNamespace(quant_config=whole, quant_config_dict={"model.layers.0": layer}) + dspark.keep_drafter_kv_dtype(cfg) + assert whole.kv_cache_quant_algo is None and layer.kv_cache_quant_algo is None + assert not whole.quant_mode.has_fp8_kv_cache() and not whole.quant_mode.has_fp4_kv_cache() + + +def test_shell_calls_take_no_fused_path(monkeypatch): + def _forbidden(*args, **kwargs): + raise AssertionError("W0 must never reach the coded target head") + + monkeypatch.setattr(m3_target_head, "prepare_head", _forbidden) + monkeypatch.setattr(m3_target_head, "logits", _forbidden) + worker = _worker() + assert worker.tail_eligible(None, None, None) is False + assert worker.arm_capture_norm(None, None, None, None) is None + assert worker.prebuild(None, None) is None + + +def test_capture_is_the_in_tree_capture(): + calls = [] + meta = SimpleNamespace(maybe_capture_hidden_states=lambda *a: calls.append(a)) + hidden, residual = object(), object() + _worker().capture_hidden_states(meta, 17, hidden, residual) + _worker().capture_hidden_states(meta, 59, hidden) + assert calls == [(17, hidden, residual), (59, hidden, None)] + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +def test_embed_tokens_is_the_target_gather(): + gen = torch.Generator(device="cuda").manual_seed(31) + table = torch.randn(1000, 64, generator=gen, device="cuda").to(torch.bfloat16) + core = SimpleNamespace(w={"embed": table}) + embed = dspark.M3EmbedTokens(core) + ids = torch.tensor([0, 5, 999, 200, -1], dtype=torch.long, device="cuda") + out = embed(ids) + ref = table[ids.clamp(min=0)] * (ids >= 0).unsqueeze(-1).to(table.dtype) + assert embed.weight is table + assert torch.equal(out, ref) + assert not list(embed.parameters()), ( + "the table belongs to the target, not to the borrowed module" + ) + + +def test_borrowed_embedding_owns_no_parameters(): + # The weights walker asserts every shell parameter is fed: the borrowed module must add none. + model = torch.nn.Module() + model.w = torch.nn.ParameterDict( + {"embed": torch.nn.Parameter(torch.zeros(8, 4), requires_grad=False)} + ) + before = {n for n, _ in model.named_parameters()} + model.embed_tokens = dspark.M3EmbedTokens(model) + assert {n for n, _ in model.named_parameters()} == before == {"w.embed"} + assert model.embed_tokens.weight is model.w["embed"] + + +class _FakeDrafter: + def __init__(self, borrows: bool = True) -> None: + self.loaded = None + self.borrows = borrows + self.lm_head = None + self.draft_model_full = SimpleNamespace(model=SimpleNamespace(embed_tokens=None)) + + def load_weights(self, weights, weight_mapper=None): + self.loaded = dict(weights) + + def load_weights_from_target_model(self, target): + if self.borrows: + self.lm_head = target.lm_head + self.draft_model_full.model.embed_tokens = target.model.embed_tokens + + +def _shell(drafter): + return SimpleNamespace( + draft_model=drafter, + lm_head=object(), + model=SimpleNamespace(embed_tokens=object()), + named_parameters=lambda: iter([("lm_head.weight", None), ("model.w.embed", None)]), + ) + + +@pytest.mark.parametrize( + "name", + ["model.embed_tokens.weight", "language_model.model.embed_tokens.weight", "lm_head.weight"], +) +def test_draft_load_refuses_borrowed_tensors(name): + shell = _shell(_FakeDrafter()) + with pytest.raises(ValueError, match="lends it"): + dspark.load_dspark_draft_weights(shell, {name: torch.zeros(1), "fc.weight": torch.zeros(1)}) + assert shell.draft_model.loaded is None + + +def test_draft_load_borrows_target_modules(): + shell = _shell(_FakeDrafter()) + weights = { + "fc.weight": torch.zeros(1), + "markov_head.markov_w1.weight": torch.zeros(1), + "confidence_head.proj.weight": torch.zeros(1), + } + dspark.load_dspark_draft_weights(shell, weights) + assert shell.draft_model.loaded == weights + assert shell.draft_model.lm_head is shell.lm_head + with pytest.raises(RuntimeError, match="does not borrow"): + dspark.load_dspark_draft_weights(_shell(_FakeDrafter(borrows=False)), weights) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_vr_draft_append_fuse.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_vr_draft_append_fuse.py new file mode 100644 index 000000000000..d0ea64059ccf --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m31_vr_draft_append_fuse.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""DRAFT_APPEND_FUSE on the sm_107 tp4 target: default ON, kill switch +TRTLLM_M31_DISABLE_DRAFT_APPEND_FUSE, and the target hands the catalog qk_norm_rope_append (through its counting +wrapper) to the DSpark worker only with EX-11. The fused append's bits are covered by +modeling_v2/spec_decode/test_modeling_v2_dspark_qk_norm_rope.py.""" + +import ast +import os +import subprocess +import sys +from pathlib import Path + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp4" +_DSPARK = Path(_family.__file__).resolve().parent / TARGET / "dspark.py" + + +def test_on_by_default_and_off_by_its_kill_switch() -> None: + probe = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + f"{TARGET} import dspark as d; print(d.DRAFT_APPEND_FUSE)" + ) + for value, expected in ((None, "True"), ("1", "False")): + env = {k: v for k, v in os.environ.items() if "DRAFT_APPEND_FUSE" not in k} + if value is not None: + env["TRTLLM_M31_DISABLE_DRAFT_APPEND_FUSE"] = value + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == expected, value + + +def test_wired_after_ex11_with_the_counting_wrapper() -> None: + tree = ast.parse(_DSPARK.read_text()) + calls = [ + n + for n in ast.walk(tree) + if isinstance(n, ast.Call) + and isinstance(n.func, ast.Attribute) + and n.func.attr == "enable_fused_qk_norm_rope_append" + ] + assert len(calls) == 1 + src = ast.unparse(calls[0].args[1]) + assert src == "_qk_norm_rope_append_logged if ex11 and DRAFT_APPEND_FUSE else None", src diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_capture_norm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_capture_norm.py new file mode 100644 index 000000000000..172dc21fae02 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_capture_norm.py @@ -0,0 +1,62 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_capture_norm entry (Eagle3 capture with fc_norm folded in).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_capture_norm import ( # noqa: E501 + m3_capture_norm as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 — needs the path declared above + EPS, + HIDDEN, + assert_bf16_close, + gen, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_capture_norm requires a CUDA device" + +LAYERS = 3 + + +def _case(spec, seed): + g = gen(seed) + m, slot = spec["m"], spec["slot"] + h = randn((m, HIDDEN), g, 2.0) + r = randn((m, HIDDEN), g, 2.0) if spec["residual"] else None + w = (1.0 + randn((HIDDEN,), g, 0.1).float()).to(torch.bfloat16) + buf = torch.full((m, LAYERS * HIDDEN), 7.0, dtype=torch.bfloat16, device="cuda") + dst = buf[:, slot * HIDDEN : (slot + 1) * HIDDEN] + return (h, r, w, EPS, dst), buf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Within one bf16 ulp per element on at most 0.1% of the elements; the other layer slots of the buffer + untouched.""" + args, buf = _case(cell.spec, seed_of(cell.why)) + certifying(op) + out = op(*args) + assert_bf16_close(out, op.reference(*args)) + slot = cell.spec["slot"] + others = torch.cat([buf[:, : slot * HIDDEN], buf[:, (slot + 1) * HIDDEN :]], dim=1) + assert bool((others == 7.0).all()) + + +def test_guard_refuses_a_strided_weight() -> None: + (h, r, w, eps, dst), _ = _case(op.CELLS[0].spec, 1) + w2 = torch.stack([w, w], dim=1)[:, 0] + with pytest.raises(AssertionError, match="dense"): + with validating(op): + op(h, r, w2, eps, dst) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_capture_normed.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_capture_normed.py new file mode 100644 index 000000000000..2bc79ad05b7b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_capture_normed.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CPU test: a first-draft-inputs reader that skips the drafter's fc_norm (the capture norm is armed) gets the captured +rows unchanged when every captured layer was written with the step's norm, and otherwise a copy with only the raw +layers normalized by their own fc_norm (M3Eagle3Worker._m3_renorm_raw_captures); the per-forward reset clears the +record and the unconsumed fc fork result.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) + +M3Eagle3Worker = modeling.M3Eagle3Worker + +H = 4 +LAYERS = (1, 29, 56) + + +def _worker(normed): + w = object.__new__(M3Eagle3Worker) + w._m3_capture_normed = set(normed) + return w + + +@pytest.fixture(autouse=True) +def _scaling_norm(monkeypatch): + """The drafter's RMSNorm as a stand-in that multiplies by its weight (CPU, no FlashInfer).""" + monkeypatch.setattr(modeling, "flashinfer_rmsnorm", lambda x, w, eps: x * w) + + +def _drafter(): + # fc_norm i scales by (i + 2): enough to tell which blocks were normalized here. + return SimpleNamespace(w={f"fc_norm{i}": i + 2 for i in range(len(LAYERS))}, rms_eps=1e-6) + + +def test_all_captured_layers_normed_returns_the_rows_as_they_lie() -> None: + hs = torch.randn(4, H * len(LAYERS)) + out = _worker(set(LAYERS))._m3_renorm_raw_captures( + hs, SimpleNamespace(layers_to_capture=LAYERS, hidden_size=H), _drafter() + ) + assert out is hs + + +@pytest.mark.parametrize("normed", [set(), {1, 29}, {29, 56}, {1, 56}]) +def test_raw_layers_get_their_own_fc_norm(normed) -> None: + hs = torch.randn(4, H * len(LAYERS)) + ref = hs.clone() + out = _worker(normed)._m3_renorm_raw_captures( + hs, SimpleNamespace(layers_to_capture=LAYERS, hidden_size=H), _drafter() + ) + assert torch.equal(hs, ref) # the capture buffer itself is not modified + for i, layer_id in enumerate(LAYERS): + block, src = out[:, i * H : (i + 1) * H], ref[:, i * H : (i + 1) * H] + assert torch.equal(block, src if layer_id in normed else src * (i + 2)), layer_id + + +def test_arm_resets_the_record_and_the_fork_result() -> None: + w = _worker(set(LAYERS)) + w._m3_fc_hidden = object() + w._m3_fc_early_stream = None # no fork pending + w._m3_join_pending = lambda: None # (tp8) nothing else pending + w.arm_capture_norm( + None, None, None, None + ) # no drafter: resets the per-forward state and returns + assert w._m3_capture_normed == set() + assert w._m3_fc_hidden is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_chunk_norm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_chunk_norm.py new file mode 100644 index 000000000000..65cd86101457 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_chunk_norm.py @@ -0,0 +1,55 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_chunk_norm entry (the Eagle3 fc_norm over the captured layers).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_chunk_norm import ( # noqa: E501 + m3_chunk_norm as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 — needs the path declared above + EPS, + assert_bf16_close, + gen, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_chunk_norm requires a CUDA device" + + +def _case(spec, seed): + g = gen(seed) + m, c, h = spec["m"], spec["c"], spec["h"] + buf = randn((m, c * h + spec["extra_cols"]), g, 2.0) + src = buf[:, : c * h + spec["extra_cols"]] + w = (1.0 + randn((c, h), g, 0.1).float()).to(torch.bfloat16) + dst = torch.full((m, c * h), 7.0, dtype=torch.bfloat16, device="cuda") + return (src, w, EPS, dst, True) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Within one bf16 ulp per element on at most 0.1% of the elements (fp32 statistic, rsqrtf).""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + out = op(*args) + assert_bf16_close(out, op.reference(*args)) + + +def test_guard_refuses_a_strided_weight() -> None: + src, w, eps, dst, pdl = _case(op.CELLS[0].spec, 1) + wt = torch.empty((w.shape[1], w.shape[0]), dtype=w.dtype, device="cuda").t() + wt.copy_(w) + with pytest.raises(AssertionError, match="dense"): + with validating(op): + op(src, wt, eps, dst, pdl) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_input.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_input.py new file mode 100644 index 000000000000..204d0192c73c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_input.py @@ -0,0 +1,103 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_draft_input entry (embedding + both RMSNorms + concat in one launch), against its reference and +against the in-tree drafter's modules.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_spec_tail, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_draft_input import ( # noqa: E501 + m3_draft_input as op, +) +from tensorrt_llm._torch.modules.embedding import Embedding +from tensorrt_llm._torch.modules.rms_norm import RMSNorm + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 — needs the path declared above + EPS, + HIDDEN, + VOCAB, + assert_bf16_close, + gen, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_draft_input requires a CUDA device" + + +def _case(spec, seed): + g = gen(seed) + m, h = spec["m"], spec["h"] + table = randn((VOCAB, h), g) + ids = torch.randint(0, VOCAB, (m,), generator=g, device="cuda", dtype=torch.int32) + if spec["invalid_id"]: + ids[0] = -1 + hidden = randn((4 * m if spec["gather"] else m, h), g) + rows = torch.arange(m, device="cuda", dtype=torch.int64) * 4 + 2 if spec["gather"] else None + w_e = (1.0 + randn((h,), g, 0.1).float()).to(torch.bfloat16) + w_h = (1.0 + randn((h,), g, 0.1).float()).to(torch.bfloat16) + out = torch.full((m, 2 * h), 7.0, dtype=torch.bfloat16, device="cuda") + residual = torch.full((m, h), 7.0, dtype=torch.bfloat16, device="cuda") + return (ids, table, hidden, rows, w_e, w_h, EPS, EPS, out, residual, True) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Normalized halves within one bf16 ulp per element on at most 0.1% of the elements (fp32 statistic in a + different summation order, rsqrtf); the gathered residual and the zero embedding of an invalid id exact.""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + out, residual = op(*args) + ref_out, ref_res = op.reference(*args) + torch.cuda.synchronize() + assert_bf16_close(out, ref_out) + assert torch.equal(residual, ref_res) + if cell.spec["invalid_id"]: + assert not bool(out[0, : cell.spec["h"]].float().any()) + + +def test_guard_refuses_rows_past_hidden() -> None: + gather = next(c for c in op.CELLS if c.spec["gather"]) + args = list(_case(gather.spec, 1)) + args[3] = args[3] + 100 + with pytest.raises(AssertionError, match="rows must index hidden"): + with validating(op): + op(*args) + + +@pytest.mark.parametrize("m,gather", [(4, False), (32, False), (8, True)], ids=str) +def test_the_in_tree_draft_input(m, gather) -> None: + """Against the in-tree drafter's own modules: the Embedding lookup, then the layer's input_layernorm and + hidden_norm (in-tree RMSNorm) and the concat of Eagle3DecoderLayer.forward. Within one bf16 ulp on at most 0.1% of + the elements (the RMS statistic summed in another order); the residual (the carried hidden rows) exact.""" + g = gen(seed_of(f"in-tree draft input {m} {gather}")) + h = HIDDEN + embed = Embedding(VOCAB, h, dtype=torch.bfloat16).cuda() + embed.weight.data = randn((VOCAB, h), g) + norms = [] + for _ in range(2): + norm = RMSNorm(hidden_size=h, eps=EPS, dtype=torch.bfloat16).cuda() + norm.weight.data = (1.0 + randn((h,), g, 0.1).float()).to(torch.bfloat16) + norms.append(norm) + input_layernorm, hidden_norm = norms + ids = torch.randint(0, VOCAB, (m,), generator=g, device="cuda", dtype=torch.int32) + hidden = randn((4 * m if gather else m, h), g, 2.0) + rows = torch.arange(m, device="cuda", dtype=torch.int64) * 4 + 3 if gather else None + carried = hidden[rows] if gather else hidden + stock = torch.cat([input_layernorm(embed(ids)), hidden_norm(carried)], dim=-1) + out, residual = m3_spec_tail.draft_input( + ids, embed.weight, hidden, rows, input_layernorm.weight, EPS, hidden_norm.weight, EPS + ) + torch.cuda.synchronize() + assert_bf16_close(out, stock) + assert torch.equal(residual, carried) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_push_take.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_push_take.py new file mode 100644 index 000000000000..fae1e3af872f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_push_take.py @@ -0,0 +1,42 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The drafter boundary push bookkeeping (kernels/m3_draft_lowm arm_push / take_pushed, dispatch.DRAFT_AR_PUSH): the +boundary all-reduce runs prepushed only when the armed call pushed, and only on that call's own output; any other +input raises instead of reducing rows the exchange does not hold. Host state only (no GPU).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_draft_lowm, +) + + +@pytest.fixture(autouse=True) +def _clean(): + m3_draft_lowm._PUSH_ARM, m3_draft_lowm._PUSHED = None, 0 + yield + m3_draft_lowm._PUSH_ARM, m3_draft_lowm._PUSHED = None, 0 + + +def test_not_pushed_is_false_and_disarms() -> None: + m3_draft_lowm._PUSH_ARM = (1, 2, 3, 0, 8, 8) + assert m3_draft_lowm.take_pushed(torch.zeros(4)) is False + assert m3_draft_lowm._PUSH_ARM is None + + +def test_pushed_output_is_true_once() -> None: + out = torch.zeros(4, 8) + m3_draft_lowm._PUSHED = out.data_ptr() + assert m3_draft_lowm.take_pushed(out) is True + assert m3_draft_lowm._PUSHED == 0 + assert m3_draft_lowm.take_pushed(out) is False + + +def test_other_input_raises() -> None: + out = torch.zeros(4, 8) + other = out.clone() + m3_draft_lowm._PUSHED = out.data_ptr() + with pytest.raises(RuntimeError, match="not the boundary all-reduce's input"): + m3_draft_lowm.take_pushed(other) + assert m3_draft_lowm._PUSHED == 0 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_silu.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_silu.py new file mode 100644 index 000000000000..9a167f3928a6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_silu.py @@ -0,0 +1,98 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_draft_silu entry (the drafter MLP's plain SwiGLU as one C++ launch): bit for bit the Triton +silu_and_mul at its defaults, over every bf16 gate and every bf16 up pattern.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_draft_silu import ( # noqa: E501 + m3_draft_silu as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + + +def seed_of(why: str) -> int: + return zlib.crc32(why.encode()) % 2**31 + + +def gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +assert torch.cuda.is_available(), "m3_draft_silu requires a CUDA device" + +ALL = torch.arange(-(1 << 15), 1 << 15, dtype=torch.int32).to(torch.int16).view(torch.bfloat16) + + +def _rows(gate: torch.Tensor, up: torch.Tensor) -> torch.Tensor: + """[rows, 2d] gate | up from equally long flat gate and up values, d = 4608 (the TP4 drafter shard).""" + d = 4608 + n = gate.numel() + rows = -(-n // d) + pad = rows * d - n + g = torch.cat([gate, gate[:pad]]).view(rows, d) + u = torch.cat([up, up[:pad]]).view(rows, d) + return torch.cat([g, u], dim=1).contiguous() + + +def _assert_bitwise(x: torch.Tensor, what: str) -> None: + got = op(x) + assert got is not None, what + ref = op.reference(x) + torch.cuda.synchronize() + diff = got.view(torch.int16) != ref.view(torch.int16) + assert not bool(diff.any()), f"{what}: {int(diff.sum())} of {diff.numel()} differ" + + +@pytest.mark.parametrize("up", [0.0, 1.0, -1.0, 3.5, -0.0, 1e-3], ids=lambda v: f"up={v}") +def test_every_gate_pattern_is_bitwise_the_triton_kernel(up) -> None: + gate = ALL.cuda() + _assert_bitwise(_rows(gate, torch.full_like(gate, up)), f"gate sweep, up {up}") + + +@pytest.mark.parametrize("gate", [1.0, -1.0, 0.5, -3.0, 20.0, -0.0], ids=lambda v: f"gate={v}") +def test_every_up_pattern_is_bitwise_the_triton_kernel(gate) -> None: + up = ALL.cuda() + _assert_bitwise(_rows(torch.full_like(up, gate), up), f"up sweep, gate {gate}") + + +@pytest.mark.parametrize("seed", range(4)) +def test_random_pairs_are_bitwise_the_triton_kernel(seed) -> None: + g = gen(100 + seed) + n = 1 << 22 + gate = torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda").to(torch.int16) + up = torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda").to(torch.int16) + _assert_bitwise(_rows(gate.view(torch.bfloat16), up.view(torch.bfloat16)), f"seed {seed}") + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """The drafter's gate_up output range (and past it), in a CUDA graph as the drafter runs it.""" + g = gen(seed_of(cell.why)) + rows, d = cell.spec["rows"], cell.spec["d"] + x = (4.0 * torch.randn(rows, 2 * d, generator=g, device="cuda")).to(torch.bfloat16) + x[:, :6] = torch.tensor([60.0, -60.0, 0.0, -0.0, 88.0, -88.0], dtype=torch.bfloat16) + certifying(op) + ref = op.reference(x) + eager = op(x) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = op(x) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(eager.view(torch.int16), ref.view(torch.int16)) + assert torch.equal(captured.view(torch.int16), ref.view(torch.int16)) + + +def test_declines_what_it_does_not_cover() -> None: + x = torch.zeros(2, 2 * 4608, dtype=torch.bfloat16, device="cuda") + assert op(x[:, 8:]) is None # strided rows + assert ( + op(torch.zeros(2, 2 * 12, dtype=torch.bfloat16, device="cuda")) is None + ) # d not a multiple of 8 + assert op(x.float()) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_step.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_step.py new file mode 100644 index 000000000000..440354a6b632 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_step.py @@ -0,0 +1,694 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_draft_step entry (one fused greedy draft pass: the BF16 draft head, streamed from its lossless +12-bit coding or as BF16 fragments, the cross-rank argmax and the draft loop's bookkeeping), and its tokens against the +in-tree drafter's head and greedy draft sampling. + +The in-tree path: each rank's LM head shard GEMM (BF16 out), ``LogitsProcessor``'s float cast, then either the argmax +of the gathered logits (``torch.argmax``) or ``greedy_sample_draft_with_tp_gather``'s per-rank ``torch.max`` and the +pick among the ranks' (index, value) pairs. The edge cases use integer data, whose fp32 sums are exact in any order, +so the kernel and the in-tree path see the same BF16 logits and must agree bit for bit; every in-tree comparison runs +with the coded head (the default) and with the BF16 fragments, and on integer data the coded head's winning keys are +the BF16 fragments' bit for bit. +""" + +import warnings +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_draft_step as draft_step_kernels, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_target_head import ( # noqa: E501 + encode_chunked, + round_trip_ok, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_draft_step import ( # noqa: E501 + DRAFT_LEN, + DRAFT_SHARD, + HIDDEN, + MAX_TOKENS, + bf16_fragments, + from_fragments, + head_weight, + shard_logits, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_draft_step import ( # noqa: E501 + m3_draft_step as op, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_shard_accept import ( # noqa: E501 + exchange_tokens, + local_argmax, + make_keys, +) +from tensorrt_llm._torch.speculative.eagle3 import Eagle3OneModelWorker + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 — needs the path declared above + WORLD, + gen, + head_workspace, + num_sms, + plant_peers, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_draft_step requires a CUDA device" + +T = DRAFT_LEN + 1 +_LOW32 = 0xFFFFFFFF + +# ------------------------------------------------------------------ the per-row rule against the in-tree token +# +# The kernel and the in-tree head (a cuBLAS BF16 GEMM) both round an FP32 sum of the same exact BF16 products to BF16, +# in different summation orders. A row's token can differ from the in-tree one only where some candidate's FP32 sum +# can land on either side of a BF16 rounding midpoint. Each row is held to: +# +# kernel token == in-tree token, or the row is provably undecidable: with the fp64 reference logit L_i (exact BF16 +# products, fp64 sum) and the FP32-order bound E_i = gamma_K * sum_k |w_ik * x_k|, the BF16 value of any FP32 sum of +# candidate i lies in [lo_i, hi_i] = [bf16(L_i - E_i), bf16(L_i + E_i)]; the kernel's token must win for some +# choice of values within those intervals (larger value, then the lower global id), and a BF16 rounding midpoint +# must lie within E of the kernel's or the in-tree token's L. +# +# gamma_K = K * 2^-22 is la-xh2's bound (4x the round-to-nearest unit roundoff 2^-24 per addition: any summation +# order, and a truncating tensor-core accumulator); K counts the row's non-zero products (adding an exact zero does +# not round). lo / hi take one more FP32 step outward before the BF16 rounding, which covers rounding L -+ E to FP32. +# Each undecidable row is reported as a warning with its midpoint distances, and each test reports its row count; any +# other mismatch fails. +GAMMA_UNIT = 2.0**-22 + + +def _bf16_outward(v: torch.Tensor, direction: float) -> torch.Tensor: + f = v.float() + f = torch.nextafter(f, torch.full_like(f, direction)) + b = f.to(torch.bfloat16).float() + return torch.where(b == 0, torch.zeros_like(b), b) + + +def bounds(h: torch.Tensor, w: torch.Tensor): + """(fp64 logits L, bound E, lo, hi) [rows, N] of the head input rows h against the head rows w.""" + cols = (h != 0).any(dim=0) # zero columns add exact zeros + hd, wd = h[:, cols].double(), w[:, cols].double() + logits = hd @ wd.t() + bound = (h != 0).sum(dim=1, keepdim=True).double() * GAMMA_UNIT * (hd.abs() @ wd.abs().t()) + del wd + inf = float("inf") + return logits, bound, _bf16_outward(logits - bound, -inf), _bf16_outward(logits + bound, inf) + + +def midpoint_distance(v: float) -> float: + """Distance of v to the nearest BF16 rounding midpoint.""" + b = torch.tensor(v, dtype=torch.float64).float().to(torch.bfloat16).float() + if float(b) == 0.0: + return abs(v) + ulp = 2.0 ** (int(torch.frexp(b).exponent) - 8) + return min(abs(v - (float(b) + ulp / 2)), abs(v - (float(b) - ulp / 2))) + + +def _winnable(tok: int, ids: torch.Tensor, lo: torch.Tensor, hi: torch.Tensor) -> bool: + """Whether candidate ``tok`` wins the row for some values within the candidates' [lo, hi].""" + k = int((ids == tok).nonzero()[0, 0]) + others = ids != tok + beat = (hi[k] > lo[others]) | ((hi[k] == lo[others]) & (tok < ids[others])) + return bool(beat.all()) + + +def check_row(name: str, kernel_tok: int, stock_tok: int, ids, logits, bound, lo, hi) -> bool: + """The per-row rule for one row over the candidates ``ids`` [C] (logits / bound / lo / hi [C]); True when the + row is undecidable (reported as a warning), False when the tokens match; a mismatch outside the bound fails.""" + if kernel_tok == stock_tok: + return False + assert _winnable(kernel_tok, ids, lo, hi), ( + f"{name}: token {kernel_tok} differs from the in-tree {stock_tok} outside the FP32-order bound" + ) + ik = int((ids == kernel_tok).nonzero()[0, 0]) + i_stock = int((ids == stock_tok).nonzero()[0, 0]) + lk, ls, ek, es = ( + float(logits[ik]), + float(logits[i_stock]), + float(bound[ik]), + float(bound[i_stock]), + ) + dk, ds = midpoint_distance(lk), midpoint_distance(ls) + assert dk <= ek or ds <= es, f"{name}: neither logit is within its bound of a BF16 midpoint" + warnings.warn( + f"[draft-step undecidable] {name}: kernel {kernel_tok} (L {lk:.9g}, midpoint distance {dk:.3e}, bound " + f"{ek:.3e}) vs in-tree {stock_tok} (L {ls:.9g}, midpoint distance {ds:.3e}, bound {es:.3e})" + ) + return True + + +def report(name: str, rows: int, undecidable: int) -> None: + warnings.warn(f"[draft-step rows] {name}: {rows} rows, {undecidable} undecidable") + + +def key_value(keys: torch.Tensor) -> torch.Tensor: + """The fp32 value a key orders by (the inverse of the kernel's order-preserving image).""" + u = (keys >> 32) & _LOW32 + bits = torch.where(u >= 0x80000000, u - 0x80000000, _LOW32 - u) + bits = torch.where(bits >= 2**31, bits - 2**32, bits) # as a signed 32-bit pattern + return bits.to(torch.int32).view(torch.float32) + + +def key_index(keys: torch.Tensor) -> torch.Tensor: + return _LOW32 - (keys & _LOW32) + + +def _bookkeeping(nreq: int, step: int, n_x_rows: int, g: torch.Generator): + num_acc = ( + torch.randint(1, T + 1, (nreq,), generator=g, device="cuda", dtype=torch.int32) + if step == 0 + else None + ) + seq_lens = torch.full((nreq,), T, dtype=torch.int32, device="cuda") if step == 0 else None + next_new = torch.full((nreq, T), -5, dtype=torch.int32, device="cuda") + pos_src = torch.randint(100, 3000, (n_x_rows,), generator=g, device="cuda", dtype=torch.int32) + kv_lens = torch.randint(200, 3000, (nreq,), generator=g, device="cuda", dtype=torch.int32) + ids_next = torch.full((nreq,), -9, dtype=torch.int32, device="cuda") + pos_next = torch.full((nreq,), -9, dtype=torch.int32, device="cuda") + return num_acc, seq_lens, next_new, pos_src, kv_lens, ids_next, pos_next + + +def _head(w: torch.Tensor, coded: bool): + """(wf, (exc_off, exc_pos, exc_w, exp_base)) of a BF16 shard: its 12-bit coding (checked to round-trip) or its + BF16 fragments.""" + if not coded: + return bf16_fragments(w), (None, None, None, 0) + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + assert round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w) + return packed, (exc_off, exc_pos, exc_w.view(-1), base) + + +def _args( + x, + x_rows, + nreq, + wf, + rank, + world, + step, + g, + uc=None, + keys=None, + counters=None, + num_acc=None, + coded=(None, None, None, 0), +): + """The op's argument tuple for one rank (fresh workspace and bookkeeping buffers unless given); ``coded`` = the + coded head's (exc_off, exc_pos, exc_w, exp_base).""" + acc, seq_lens, next_new, pos_src, kv_lens, ids_next, pos_next = _bookkeeping( + nreq, step, x.shape[0], g + ) + if num_acc is None: + num_acc = acc + if uc is None: + uc, keys, counters = head_workspace(world, slot_rows=MAX_TOKENS) + out_keys = torch.zeros(nreq, dtype=torch.int64, device="cuda") + tiles = wf.shape[0] if coded[0] is None else coded[0].numel() - 1 + return (x, x_rows, nreq, wf, rank * tiles * 16, keys, counters, uc, uc.data_ptr(), rank, world, + next_new[:, 1:], step, ids_next, pos_src, pos_next, kv_lens, num_acc, seq_lens, DRAFT_LEN, num_sms(), + True, out_keys, *coded) # fmt: skip + + +def _case(spec, seed): + g = gen(seed) + nreq, step, rank, world, vs, k = ( + spec["nreq"], + spec["step"], + spec["rank"], + spec["world"], + spec["vs"], + spec["k"], + ) + pass0 = step == 0 + x = randn((nreq * T if pass0 else nreq, k), g) + wf, coded = _head(randn((vs, k), g, 0.02), spec["coded"]) + num_acc = ( + torch.randint(1, T + 1, (nreq,), generator=g, device="cuda", dtype=torch.int32) + if pass0 + else None + ) + x_rows = (torch.arange(nreq, device="cuda") * T + num_acc.long() - 1) if pass0 else None + uc, keys, counters = head_workspace(world, slot_rows=MAX_TOKENS) + h = x[x_rows] if pass0 else x + own = shard_logits(h, wf, *coded).max(dim=1).values + plant_peers( + uc, counters, own, rank, world, vs, g, slot_rows=MAX_TOKENS, ties=spec.get("ties", False) + ) + return _args(x, x_rows, nreq, wf, rank, world, step, g, uc, keys, counters, num_acc, coded) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """The bookkeeping is exact on every row (next positions, KV and sequence lengths). Each row's token is the + in-tree one (this rank's BF16 shard logits from the head's GEMM, their first maximum, against the peers' keys) or + the row is provably undecidable (check_row); the winning key's index is the token, and its value lies in the + token's [lo, hi] (or is the peer's pushed key when a peer wins).""" + args = _case(cell.spec, seed_of(cell.why)) + x, x_rows, nreq, wf, offset, _, counters, uc = args[:8] + rank, world = args[9], args[10] + half = int(counters[1].item()) & 1 + pushed = ( + uc[half * MAX_TOKENS * world : (half * MAX_TOKENS + nreq) * world].view(nreq, world).clone() + ) + h = x[x_rows] if x_rows is not None else x[:nreq] + w = head_weight(wf, x.shape[1], *args[23:27]) + best, idx = local_argmax(torch.nn.functional.linear(h, w).float(), offset) + stock = exchange_tokens(make_keys(best, idx), uc, counters, rank, world, MAX_TOKENS) + certifying(op) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + assert args[6].tolist() == [0, 1] and not bool(args[5].any()), "counters / keys not re-armed" + step = args[12] + others = [c for c in range(args[11].shape[1]) if c != step] + assert torch.equal(args[11][:, others], ref["draft_out"][:, others]) + assert torch.equal(args[11][:, step], args[13]), "draft_out and ids_next disagree" + assert torch.equal(args[15], ref["pos_next"]) + assert torch.equal(args[16], ref["kv_lens"]) + if args[18] is not None: + assert torch.equal(args[18], ref["seq_lens"]) + # Candidates per row: this rank's head rows (fp64 bounds) and each peer's pushed key (an exact value). + logits, bound, lo, hi = bounds(h, w) + own_ids = offset + torch.arange(w.shape[0], device="cuda") + peers = [p for p in range(world) if p != rank] + undecidable = 0 + for r in range(nreq): + pv = key_value(pushed[r, peers]).double() + ids = torch.cat([own_ids, key_index(pushed[r, peers])]) + undecidable += check_row( + f"{cell.why[:40]} row {r}", + int(args[13][r]), + int(stock[r]), + ids, + torch.cat([logits[r], pv]), + torch.cat([bound[r], torch.zeros_like(pv)]), + torch.cat([lo[r], pv.float()]), + torch.cat([hi[r], pv.float()]), + ) + report(cell.why[:40], nreq, undecidable) + out_keys = args[22] + assert torch.equal(key_index(out_keys), args[13].long()), "the winning key's index is the token" + got = key_value(out_keys) + for r in range(nreq): + c = int(args[13][r]) - offset + if 0 <= c < w.shape[0]: + assert float(lo[r, c]) <= float(got[r]) <= float(hi[r, c]), ( + f"row {r}: winning value outside [lo, hi]" + ) + else: + assert int(out_keys[r]) in pushed[r].tolist(), ( + f"row {r}: a peer won with a key it did not push" + ) + + +def test_fragments_round_trip() -> None: + g = gen(seed_of("fragments")) + w = randn((64, 96), g) + wf = bf16_fragments(w) + assert wf.shape == (4, 6, 256) + assert torch.equal(from_fragments(wf), w) + head = SimpleNamespace(weight=w, bias=None) + # K 96: never coded, the BF16 fragments. + prepared = draft_step_kernels.prepare_head(head, 3, True) + assert not prepared.coded + assert torch.equal(prepared.wf, wf) and prepared.vocab_offset == 3 * 64 and prepared.weight is w + + +# ------------------------------------------------------------------ against the in-tree drafter, bit for bit + + +def _stock_tokens(shards: list, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """The in-tree greedy draft tokens of the rows x over the vocab-sharded BF16 head: (argmax of the gathered float + logits, greedy_sample_draft_with_tp_gather's per-rank (index, value) pick).""" + logits = [torch.nn.functional.linear(x, w).float() for w in shards] + gathered = torch.argmax(torch.cat(logits, dim=-1), dim=-1).to(torch.int32) + combined = [] + for rank, shard_logit in enumerate(logits): + worker = SimpleNamespace(mapping=SimpleNamespace(tp_rank=rank)) + combined.append(Eagle3OneModelWorker._get_local_max_and_combined(worker, shard_logit)) + pick = Eagle3OneModelWorker._get_draft_tokens_from_gathered + pick = getattr(pick, "_torchdynamo_orig_callable", pick) + tp_gather = pick(None, torch.cat(combined, dim=-1)).to(torch.int32) + return gathered, tp_gather + + +def _fused_tokens(shards: list, x: torch.Tensor, coded: bool, step: int = 1) -> list: + """Each rank's fused draft tokens: every rank's own key from a one-rank run, then each rank's four-rank pass with + its peers' keys in its slots, as their multicast pushes would put them.""" + g = gen(seed_of("fused tokens")) + nreq = x.shape[0] + heads = [_head(w, coded) for w in shards] + own_keys = [] + for rank, (wf, cd) in enumerate(heads): + args = list(_args(x, None, nreq, wf, rank, 1, step, g, coded=cd)) + args[9] = 0 # one-rank run: this rank's slot only, global ids of `rank` + op(*args) + own_keys.append(args[22].clone()) + tokens = [] + for rank, (wf, cd) in enumerate(heads): + uc, keys, counters = head_workspace(WORLD, slot_rows=MAX_TOKENS) + for peer in range(WORLD): + if peer != rank: + for t in range(nreq): + uc[t * WORLD + peer] = own_keys[peer][t] + args = _args(x, None, nreq, wf, rank, WORLD, step, g, uc, keys, counters, coded=cd) + with validating(op): + op(*args) + torch.cuda.synchronize() + assert torch.equal(args[11][:, step], args[13]), "draft_out and ids_next disagree" + tokens.append(args[13].clone()) + return tokens + + +def _integer_head(g: torch.Generator, rows: int = WORLD * DRAFT_SHARD) -> torch.Tensor: + """A [rows, 6144] head of small integers over the first 4 columns (zero elsewhere): its logits against integer + rows are exact integers in any summation order, many of which share a BF16 value.""" + w = torch.zeros((rows, HIDDEN), dtype=torch.bfloat16, device="cuda") + w[:, :4] = torch.randint(-60, 61, (rows, 4), generator=g, device="cuda").to(torch.bfloat16) + return w + + +def _shards(w: torch.Tensor) -> list: + return [w[r * DRAFT_SHARD : (r + 1) * DRAFT_SHARD].contiguous() for r in range(WORLD)] + + +def _assert_stock(w: torch.Tensor, x: torch.Tensor, coded: bool, expect=None) -> None: + shards = _shards(w) + gathered, tp_gather = _stock_tokens(shards, x) + assert torch.equal(gathered, tp_gather), "the two in-tree pickers disagree" + if expect is not None: + assert gathered.tolist() == expect, (gathered.tolist(), expect) + for rank, tokens in enumerate(_fused_tokens(shards, x, coded)): + assert torch.equal(tokens, gathered), ( + f"rank {rank}: {tokens.tolist()} vs stock {gathered.tolist()}" + ) + + +def _ones_rows(nreq: int) -> torch.Tensor: + x = torch.zeros((nreq, HIDDEN), dtype=torch.bfloat16, device="cuda") + x[:, :4] = 1.0 + return x + + +@pytest.mark.parametrize("coded", [True, False], ids=["coded", "bf16"]) +def test_forced_bf16_ties_take_the_lowest_global_id(coded) -> None: + """A head row copied onto every rank that wins every row of ones (logit 244, every other at most 240): the lowest + global id must win, inside a shard and across ranks. Then integer rows in [-2, 2], whose logits (up to 480 in + magnitude, BF16 spacing 2 above 256) collide often, exactly or after BF16 rounding.""" + g = gen(seed_of("forced ties")) + w = _integer_head(g) + win = torch.tensor([61.0, 61.0, 61.0, 61.0], device="cuda").to(torch.bfloat16) + for gid in (3 * DRAFT_SHARD + 5, DRAFT_SHARD + 77, DRAFT_SHARD + 9000, 2 * DRAFT_SHARD + 1): + w[gid, :4] = win + _assert_stock(w, _ones_rows(5), coded, expect=[DRAFT_SHARD + 77] * 5) + # Random integer rows: ties after BF16 rounding happen naturally. + x = torch.randint(-2, 3, (8, HIDDEN), generator=g, device="cuda").to(torch.bfloat16) + x[:, 4:] = 0 + _assert_stock(w, x, coded) + + +@pytest.mark.parametrize("coded", [True, False], ids=["coded", "bf16"]) +def test_signed_zero_maxima_take_the_lowest_global_id(coded) -> None: + """Rows whose maximum is zero: an all-zero head input (every logit +-0) and logits <= 0 with zeros by cancellation + on several ranks. -0 and +0 compare equal, so the lowest global id reaching zero wins.""" + w = torch.zeros((WORLD * DRAFT_SHARD, HIDDEN), dtype=torch.bfloat16, device="cuda") + w[:, 0] = -1.0 + w[:, 1] = -1.0 + for gid in (2 * DRAFT_SHARD + 5, DRAFT_SHARD + 40000, 3 * DRAFT_SHARD + 7, 49999): + w[gid, 0] = 1.0 + w[gid, 1] = -1.0 + x = torch.zeros((3, HIDDEN), dtype=torch.bfloat16, device="cuda") + x[1, :2] = 1.0 + x[2, :2] = 2.0 + x[2, 2] = -1.0 # w[:, 2] is zero: a -0 product on every row + _assert_stock(w, x, coded, expect=[0, 49999, 49999]) + + +@pytest.mark.parametrize( + "lower,higher,winner", + [ + (1000, 1001, "lower"), # 1001 rounds to 1000 in BF16: a tie, the lower id wins + (1001, 1002, "lower"), # 1001 and the tie-to-even midpoint 1002 both round to 1000 + (1000, 1004, "higher"), # adjacent BF16 values: the larger wins + (1003, 1000, "lower"), # 1003 rounds up to 1004 > 1000 + ], + ids=["one_fp32_step", "midpoint", "adjacent_bf16", "rounds_up"], +) +@pytest.mark.parametrize("where", ["in_shard", "across_ranks"]) +@pytest.mark.parametrize("coded", [True, False], ids=["coded", "bf16"]) +def test_near_equal_top2_follow_the_bf16_logits(lower, higher, winner, where, coded) -> None: + """Top-2 logits one to four fp32 steps apart: the winner is the one the BF16 logits decide (the in-tree logits' + dtype), not the fp32 sum. The pair sits inside rank 3's shard or on ranks 1 and 2; every other logit is at most + 240 in magnitude.""" + g = gen(seed_of(f"near top2 {lower} {higher} {where}")) + w = _integer_head(g) + if where == "in_shard": + lo_id, hi_id = 3 * DRAFT_SHARD + 100, 3 * DRAFT_SHARD + 200 + else: + lo_id, hi_id = DRAFT_SHARD + 49000, 2 * DRAFT_SHARD + 3 + for gid, value in ((lo_id, lower), (hi_id, higher)): + q, r = divmod(value, 4) # four integers <= 251, exact in BF16 + w[gid, :4] = torch.tensor([q + (i < r) for i in range(4)], device="cuda").to(torch.bfloat16) + expect = lo_id if winner == "lower" else hi_id + _assert_stock(w, _ones_rows(2), coded, expect=[expect, expect]) + + +# ------------------------------------------------------------------ non-integer rows at a BF16 rounding midpoint + +#: A BF16 rounding midpoint (between 6.0 and 6.03125; a tie there rounds to the even 6.0) and the FP32 spacing there. +MIDPOINT = 6.0 + 2.0**-6 +FP32_ULP = 2.0**-21 + + +def _near_midpoint_head(g: torch.Generator, targets: dict): + """A full-vocabulary head [4 * 50016, 6144] and one head-input row x with 65 non-zero entries: 1, 2^-12, 2^-20 + and 2^-28 on columns 0..3 and 61 random non-integer BF16 values. Every head row carries random non-integer BF16 + values on those 61 columns (logits about 0.2 in magnitude); each row of ``targets`` (global id -> fp64 value) also + gets columns 0..3 set so its fp64 logit is the target to within 2^-30 (a coarse value, then three finer + corrections, each BF16-rounded).""" + x = torch.zeros((1, HIDDEN), dtype=torch.bfloat16, device="cuda") + scales = (1.0, 2.0**-12, 2.0**-20, 2.0**-28) + for col, scale in enumerate(scales): + x[0, col] = scale + x[0, 4:65] = randn((61,), g, 0.5) + w = torch.zeros((WORLD * DRAFT_SHARD, HIDDEN), dtype=torch.bfloat16, device="cuda") + w[:, 4:65] = randn((WORLD * DRAFT_SHARD, 61), g, 0.05) + xd = x[0].double() + for gid, target in targets.items(): + rest = float(target - (w[gid].double() * xd).sum()) + for col, scale in enumerate(scales): + v = torch.tensor(rest / scale, dtype=torch.float64).to(torch.bfloat16) + w[gid, col] = v + rest -= float(v) * scale + assert abs(rest) <= 2.0**-30, (gid, rest) + return w, x + + +def _check_fused_against_stock( + name: str, w: torch.Tensor, x: torch.Tensor, coded: bool, expect=None +) -> int: + """Each rank's fused tokens against the in-tree tokens under the per-row rule (full-vocabulary candidates); + ``expect`` (a token per row) additionally pins rows the bound decides. Returns the undecidable rows.""" + shards = _shards(w) + gathered, tp_gather = _stock_tokens(shards, x) + assert torch.equal(gathered, tp_gather), "the two in-tree pickers disagree" + fused = _fused_tokens(shards, x, coded) + for tokens in fused[1:]: + assert torch.equal(tokens, fused[0]), "the ranks disagree on the global token" + logits, bound, lo, hi = bounds(x, w) + ids = torch.arange(w.shape[0], device="cuda") + undecidable = 0 + for r in range(x.shape[0]): + undecidable += check_row( + f"{name} row {r}", + int(fused[0][r]), + int(gathered[r]), + ids, + logits[r], + bound[r], + lo[r], + hi[r], + ) + if expect is not None: + assert int(gathered[r]) == expect[r] and int(fused[0][r]) == expect[r], ( + name, + int(gathered[r]), + int(fused[0][r]), + expect[r], + ) + others = ids != expect[r] + assert _winnable(expect[r], ids, lo[r], hi[r]), ( + f"{name}: the bound does not let {expect[r]} win" + ) + rivals = [int(i) for i in ids[others][hi[r][others] >= lo[r][expect[r]]]] + assert not any(_winnable(i, ids, lo[r], hi[r]) for i in rivals), ( + f"{name}: the row is not decided" + ) + report(name, x.shape[0], undecidable) + return undecidable + + +@pytest.mark.parametrize( + "lower,higher,winner", + [ + ( + -2, + 2, + None, + ), # straddling the midpoint by 2 FP32 ulps: either side is in-tree up to the order + (2, -2, None), + (0, -2, None), # exactly on the midpoint (ties to the even 6.0) against just below it + ( + -2048, + 2048, + "higher", + ), # ~1e-3 either side, well outside the bound: decided by BF16 rounding + ( + -4096, + -2048, + "lower", + ), # both below the midpoint, ~1e-3 apart: a BF16 tie the lower id wins + ], + ids=["straddle", "straddle_reversed", "on_midpoint", "decided_up", "decided_tie"], +) +@pytest.mark.parametrize("where", ["in_shard", "across_ranks"]) +@pytest.mark.parametrize("coded", [True, False], ids=["coded", "bf16"]) +def test_top2_at_a_bf16_midpoint(lower, higher, winner, where, coded) -> None: + """Non-integer rows whose top-2 fp64 logits sit the given number of FP32 ulps from a BF16 rounding midpoint, inside + rank 3's shard or on ranks 1 and 2, every other logit far below. The per-row rule holds on every row; outside the + bound the token is the one the BF16 rounding decides, bit for bit.""" + g = gen(seed_of(f"midpoint {lower} {higher} {where}")) + if where == "in_shard": + lo_id, hi_id = 3 * DRAFT_SHARD + 100, 3 * DRAFT_SHARD + 200 + else: + lo_id, hi_id = DRAFT_SHARD + 49000, 2 * DRAFT_SHARD + 3 + w, x = _near_midpoint_head( + g, {lo_id: MIDPOINT + lower * FP32_ULP, hi_id: MIDPOINT + higher * FP32_ULP} + ) + expect = None if winner is None else [lo_id if winner == "lower" else hi_id] + _check_fused_against_stock(f"midpoint {where} {lower:+d}/{higher:+d} ulp", w, x, coded, expect) + + +# ------------------------------------------------------------------ the coded head against the BF16 fragments + + +@pytest.mark.parametrize("nreq", [1, 2, 8]) +def test_coded_head_is_the_bf16_head_bit_for_bit(nreq) -> None: + """Integer data (fp32 sums exact in any order) with exceptions on both sides of the exponent window: the coded + head's winning keys (value and global id) and its bookkeeping are the BF16 fragments' bit for bit, on one rank and + across the ranks.""" + g = gen(seed_of(f"coded vs bf16 {nreq}")) + sign = torch.randint(0, 2, (DRAFT_SHARD, HIDDEN), generator=g, device="cuda") * 2 - 1 + w = (torch.randint(1, 41, (DRAFT_SHARD, HIDDEN), generator=g, device="cuda") * sign).to( + torch.bfloat16 + ) + # Exceptions on both sides of the window of 1..40: exact zeros, and 2^15 (fp32 sums stay below 2^24). + w[::5, ::512] = 0.0 + w[::13, 7::600] = 32768.0 + x = torch.randint(-2, 3, (nreq, HIDDEN), generator=g, device="cuda").to(torch.bfloat16) + wf_c, cd = _head(w, True) + assert cd[1].numel() > 0, "the shard must carry exceptions" + wf_b, cb = _head(w, False) + outs = [] + for wf, c in ((wf_c, cd), (wf_b, cb)): + args = _args(x, None, nreq, wf, 2, 1, 1, gen(seed_of("bookkeeping")), coded=c) + args = list(args) + args[9] = 0 + with validating(op): + op(*args) + torch.cuda.synchronize() + outs.append(args) + coded_args, bf16_args = outs + for i in (11, 13, 15, 16, 22): # draft_out, ids_next, pos_next, kv_lens, out_keys + assert torch.equal(coded_args[i], bf16_args[i]), i + best, idx = local_argmax(torch.nn.functional.linear(x, w).float(), 2 * DRAFT_SHARD) + assert torch.equal(coded_args[22], make_keys(best, idx)), ( + "the winning keys are not the in-tree ones" + ) + + +@pytest.mark.parametrize("nreq", [1, 2, 8]) +def test_tiny_exception_products_are_exact(nreq) -> None: + """Weights of 1 (exponent window base 113) with column 0 zero except 2^-30 / 2^-29 on rows 0 / 1, and a one-hot + activation on column 0: the exceptions' exact products are the only non-zero logits, so the coded head must pick + row 1 with the BF16 head's key bit for bit (a correction against a non-zero placeholder cancels them to 0).""" + n, k = 256, 1024 + w = torch.ones((n, k), dtype=torch.bfloat16, device="cuda") + w[:, 0] = 0.0 + w[0, 0] = 2.0**-30 + w[1, 0] = 2.0**-29 + x = torch.zeros((nreq, k), dtype=torch.bfloat16, device="cuda") + x[:, 0] = 1.0 + wf_c, cd = _head(w, True) + assert cd[3] == 113 + keys = [] + for wf, c in ((wf_c, cd), _head(w, False)): + args = _args(x, None, nreq, wf, 0, 1, 1, gen(seed_of("tiny exceptions")), coded=c) + with validating(op): + # 16 tiles on an all-SM grid: every tile is streamed by all 16 warps, which only the fixed-order sum + # adds deterministically (is_valid rejects the atomics there). + op(*args, fixed_order=True) + torch.cuda.synchronize() + assert args[13].tolist() == [1] * nreq + keys.append(args[22]) + best, idx = local_argmax(torch.nn.functional.linear(x, w).float(), 0) + assert torch.equal(keys[0], keys[1]) and torch.equal(keys[0], make_keys(best, idx)) + + +def test_prepare_head_codes_the_shard_and_fails_closed(monkeypatch) -> None: + """prepare_head keeps the coded shard when it decodes to the weight bit for bit; a coding that does not is + dropped for the BF16 fragments.""" + g = gen(seed_of("prepare coded")) + w = randn((4096, 1024), g, 0.02) + head = SimpleNamespace(weight=w, bias=None) + prepared = draft_step_kernels.prepare_head(head, 2, True) + assert prepared.coded and prepared.vocab_offset == 2 * 4096 and prepared.weight is w + assert torch.equal(head_weight(prepared.wf, 1024, prepared.exc_off, prepared.exc_pos, prepared.exc_w, + prepared.exp_base), w) # fmt: skip + + def corrupt(weight, *args, **kwargs): + packed, base, exc_off, exc_pos, exc_w = encode_chunked(weight) + packed = packed.clone() + packed[11] ^= 0x10 + return packed, base, exc_off, exc_pos, exc_w + + monkeypatch.setattr(draft_step_kernels, "encode_chunked", corrupt) + dropped = draft_step_kernels.prepare_head(head, 2, True) + assert not dropped.coded and dropped.exc_off is None + assert torch.equal(dropped.wf, bf16_fragments(w)) + + +def test_prepare_head_keeps_bf16_when_coding_is_off(monkeypatch) -> None: + """``coded`` False (``DRAFT_HEAD_CODED`` off) keeps the BF16 fragments even for a shard whose coding would decode + bit for bit.""" + g = gen(seed_of("prepare bf16")) + w = randn((4096, 1024), g, 0.02) + prepared = draft_step_kernels.prepare_head(SimpleNamespace(weight=w, bias=None), 2, False) + assert not prepared.coded and prepared.exc_off is None + assert torch.equal(prepared.wf, bf16_fragments(w)) + + +def test_guard_refuses_head_rows_past_x() -> None: + args = list(_case(op.CELLS[0].spec, 1)) + args[1] = args[1] + 1000 + with pytest.raises(AssertionError, match="x_rows must index x"): + with validating(op): + op(*args) + + +def test_guard_refuses_too_few_ctas() -> None: + args = list(_case(op.CELLS[1].spec, 2)) + args[20] = 8 + with pytest.raises(AssertionError, match="tiles per CTA"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_step_fixed_order.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_step_fixed_order.py new file mode 100644 index 000000000000..b7423b3d1c65 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_draft_step_fixed_order.py @@ -0,0 +1,148 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The draft step's tile pieces and their summation order (kernels/m3_draft_step: the fixed warp order). + +A CTA's (tile, chunk) range is cut into 16 equal warp segments; the shared-memory atomics add a tile's pieces in +arrival order, which is deterministic only while no tile takes more than 2 pieces. ``fixed_order`` sums them in warp +order instead. The geometry tests enumerate the real grid (the shard on 212 SMs) so a grid or shard change cannot +silently bring back an order-dependent sum; the GPU tests hold the fixed order to the atomics on exact data and to the +warp-order oracle on a planted tile whose sum depends on the order. +""" + +import types + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( + m3_draft_step as kds, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_draft_step import ( # noqa: E501 + DRAFT_SHARD, + MAX_TOKENS, + bf16_fragments, + max_tile_pieces, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import gen, head_workspace # noqa: E402 + +OP = "minimax_m3_sm107tp4_draft_step" +SMS = 212 # Rubin's SMs: the grid the draft step launches on +K = 6144 +CHUNKS = K // 16 +# This target's shard on Rubin: 14-15 tiles per CTA, at most 2 pieces per tile (the atomics are deterministic and the +# fixed order stays off). +ORDER_DEPENDENT = False + +gpu = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + + +@pytest.mark.parametrize("grid", [SMS, kds.GRID]) +def test_real_grid_pieces(grid) -> None: + """One CTA per SM and the launch the target uses (kernels/m3_draft_step.GRID).""" + pieces = max_tile_pieces(DRAFT_SHARD // 16, grid, CHUNKS) + assert (pieces > 2) == ORDER_DEPENDENT, (grid, pieces) + + +def test_ds_grid_launches_the_configured_grid(monkeypatch) -> None: + monkeypatch.setattr(kds, "_num_sms", lambda: SMS) + assert kds.GRID == 209 + assert kds._grid() == min(kds.GRID, SMS) + + +def test_max_tile_pieces_counts_warp_segments() -> None: + # One tile per CTA: all 16 warps stream it. 7 tiles per CTA: a 168-chunk segment can sit inside a tile (TP8). + assert max_tile_pieces(16, 16, CHUNKS) == 16 + assert max_tile_pieces(7, 1, CHUNKS) == 4 + assert max_tile_pieces(8, 1, CHUNKS) == 2 + assert max_tile_pieces(15, 1, CHUNKS) == 2 + + +def test_the_fixed_order_engages_where_the_geometry_needs_it() -> None: + flag = True + x = torch.empty(1, K) + + def head(n_tiles): + return types.SimpleNamespace(wf=torch.empty(n_tiles, 1), exc_off=None, bnd=None) + + assert kds._fixed_order(1, head(DRAFT_SHARD // 16), x, SMS) == (flag and ORDER_DEPENDENT) + assert kds._fixed_order(1, head(DRAFT_SHARD // 16), x, kds.GRID) == (flag and ORDER_DEPENDENT) + assert kds._fixed_order(1, head(16), x, 16) == flag # 16 pieces per tile + + +def _launcher(wf, x, fixed, nctas): + """(launch, ids, keys): one pass's op launch on buffers made here (outside any capture), world 1.""" + uc, keys, counters = head_workspace(1, slot_rows=MAX_TOKENS) + m = x.shape[0] + i32 = dict(dtype=torch.int32, device="cuda") + draft_out = torch.zeros(m, 4, **i32) + ids = torch.zeros(m, **i32) + pos_src, pos_next, kv_lens = ( + torch.full((m,), 100, **i32), + torch.zeros(m, **i32), + torch.full((m,), 500, **i32), + ) + out = torch.zeros(m, dtype=torch.int64, device="cuda") + + def launch(): + getattr(torch.ops.trtllm, OP)(x, None, m, wf, 0, keys, counters, uc, uc.data_ptr(), 0, 1, draft_out, 1, ids, + pos_src, pos_next, kv_lens, None, None, 3, nctas, False, out, + fixed_order=fixed) # fmt: skip + + return launch, ids, out + + +@gpu +@pytest.mark.parametrize("nctas,vs", [(SMS, DRAFT_SHARD), (16, 256)]) +@pytest.mark.parametrize("m", [1, 4, 8]) +def test_fixed_order_equals_the_atomics_on_exact_data(nctas, vs, m) -> None: + """Integer data: every fp32 sum is exact in any order, so both summations give the same bits.""" + g = gen(17 + m) + w = torch.randint(-1, 2, (vs, K), generator=g, device="cuda").to(torch.bfloat16) + x = torch.randint(-1, 2, (m, K), generator=g, device="cuda").to(torch.bfloat16) + wf = bf16_fragments(w) + out = {} + for fixed in (False, True): + launch, ids, keys = _launcher(wf, x, fixed, nctas) + launch() + torch.cuda.synchronize() + out[fixed] = (keys, ids) + assert torch.equal(out[False][0], out[True][0]) + ref = (x.double() @ w.double().t()).float().bfloat16().float().argmax(dim=1).int() + assert torch.equal(out[True][1], ref) + + +@gpu +@pytest.mark.parametrize("graph", [False, True]) +def test_fixed_order_is_the_warp_order_sum_on_a_planted_tile(graph) -> None: + """One tile per CTA (16 pieces): row 0's pieces are +2^25 (warp 0), +1 (warps 1..14), -2^25 (warp 15). In warp + order the +1s are absorbed by 2^25 and the row sums to 0, so the competitor (row 255, logit 0.5) wins; arrival + order could give up to 14. Every launch must give the warp-order result.""" + w = torch.zeros(256, K, device="cuda") + seg = CHUNKS // 16 * 16 # one warp segment in K + w[0, 0] = 2.0**25 + for s in range(1, 15): + w[0, s * seg] = 1.0 + w[0, 15 * seg] = -(2.0**25) + w[255, 0] = 0.5 + wf = bf16_fragments(w.to(torch.bfloat16)) + x = torch.ones(1, K, device="cuda", dtype=torch.bfloat16) + launch, ids, _ = _launcher(wf, x, True, 16) + run = launch + if graph: + launch() + torch.cuda.synchronize() + gr = torch.cuda.CUDAGraph() + cs = torch.cuda.Stream() + with torch.cuda.stream(cs), torch.cuda.graph(gr, stream=cs): + launch() + run = gr.replay + for _ in range(16): + ids.zero_() + run() + torch.cuda.synchronize() + assert int(ids[0]) == 255 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_exact_numerics.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_exact_numerics.py new file mode 100644 index 000000000000..8857b6be4945 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_exact_numerics.py @@ -0,0 +1,289 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp4 target's speculative step against the in-tree (stock) Eagle3 numerics, at the real TP4 shapes. + +* Verify: the target logits are the in-tree BF16 LM head's own shard GEMM, and the fused shard argmax + acceptance + gives the in-tree greedy acceptance of the gathered logits, bit for bit, on every rank (peers' keys are the real + per-shard maxima, computed by the same kernel). +* Drafter: the worker runs the in-tree drafter's modules and weights; its own drafter arithmetic is the first pass's + fc_norm (chunk_norm, or capture_norm folded into the capture), held here to the in-tree ``apply_eagle3_fc`` RMSNorm + modules within one bf16 ulp (its effect on the fc output and on the full-vocabulary BF16 head's draft tokens is + measured too), and the fused draft loop's draft input and BF16 head, held to the in-tree modules and greedy draft + sampling by their own tests (test_modeling_v2_m3_draft_input.py, test_modeling_v2_m3_draft_step.py). + +Synthetic weights: hidden 6144, vocabulary 200064 (4 x 50016 shards), 3 captured layers, draft length 3. +""" + +import importlib.util +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_spec_tail, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_shard_accept import ( # noqa: E501 + SHARD_ROWS, + make_keys, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_shard_accept import ( # noqa: E501 + m3_shard_accept as shard_accept_op, +) +from tensorrt_llm._torch.models.modeling_speculative import Eagle3ForCausalLM +from tensorrt_llm._torch.modules.embedding import LMHead +from tensorrt_llm._torch.modules.linear import Linear +from tensorrt_llm._torch.modules.rms_norm import RMSNorm +from tensorrt_llm._torch.speculative.eagle3 import Eagle3OneModelWorker + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 + EPS, + HIDDEN, + TARGET_SHARD, + VOCAB, + WORLD, + assert_bf16_close, + craft_drafts, + gen, + head_workspace, + num_sms, + randn, + seed_of, +) + +assert torch.cuda.is_available(), "the exact-numerics checks require a CUDA device" + +DRAFT_LEN = 3 +CAPTURED_LAYERS = 3 +#: Largest share of fc_norm outputs one bf16 ulp off the in-tree RMSNorm (fp32 statistic summed in another order). +NORM_ULP_FRAC = 1e-3 + + +def _stock_accept(logits: torch.Tensor, draft: torch.Tensor): + """The in-tree greedy strict acceptance (``SpecWorkerBase._sample_and_accept_draft_tokens_base``) of the gathered + float logits of a decode-only batch, no forcing.""" + md = SimpleNamespace(is_all_greedy_sample=True, is_ragged_verify=False, is_cuda_graph=False) + worker = SimpleNamespace(force_num_accepted_tokens=0.0) + worker._sample_tokens_for_batch = lambda *a: Eagle3OneModelWorker._sample_tokens_for_batch( + worker, *a + ) + worker._apply_force_accepted_tokens = lambda *a, **k: ( + Eagle3OneModelWorker._apply_force_accepted_tokens(worker, *a, **k) + ) + nreq = draft.shape[0] + return Eagle3OneModelWorker._sample_and_accept_draft_tokens_base( + worker, logits, draft, 0, nreq, md + ) + + +def _lm_head_shard(weight: torch.Tensor, rank: int) -> LMHead: + """Rank ``rank``'s local part of the target's vocab-sharded BF16 LM head at TP4: an LM head over that rank's + 50016 rows (the TP4 module's local GEMM is this module's; its all-gather is the caller's).""" + head = LMHead(num_embeddings=TARGET_SHARD, embedding_dim=HIDDEN, dtype=torch.bfloat16) + head.weight.data = weight[rank * TARGET_SHARD : (rank + 1) * TARGET_SHARD].contiguous() + return head + + +def _run_shard_accept(logits, rank, world, draft, keys_of_peers=None, call=0): + """The fused verify of one rank: its shard logits, the peers' keys planted in its slots.""" + nreq = draft.shape[0] + rows = logits.shape[0] + uc, keys, counters = head_workspace(world, call=call, slot_rows=SHARD_ROWS) + half = call & 1 + for peer, peer_keys in (keys_of_peers or {}).items(): + for t in range(rows): + uc[(half * SHARD_ROWS + t) * world + peer] = peer_keys[t] + acc = torch.full((nreq, DRAFT_LEN + 1), -7, dtype=torch.int32, device="cuda") + num = torch.full((nreq,), -7, dtype=torch.int32, device="cuda") + gather = torch.full((nreq,), -7, dtype=torch.int64, device="cuda") + next_new = torch.full((nreq, DRAFT_LEN + 1), -7, dtype=torch.int32, device="cuda") + batch_idx = torch.arange(nreq, dtype=torch.int32, device="cuda") + shard_accept_op( + logits, rank * TARGET_SHARD if world > 1 else 0, keys, counters, uc, uc.data_ptr(), rank, world, + draft, batch_idx, acc, num, gather, 0, next_new, num_sms(), None, + ) # fmt: skip + torch.cuda.synchronize() + return acc, num, gather, next_new + + +def _local_keys(logits: torch.Tensor, rank: int) -> torch.Tensor: + """What rank ``rank`` pushes: the key of its own shard maximum per row, found by the same kernel at world 1.""" + rows = logits.shape[0] + nreq = rows // (DRAFT_LEN + 1) + dummy = torch.full((nreq, DRAFT_LEN), -1, dtype=torch.int32, device="cuda") + acc, _, _, _ = _run_shard_accept(logits, 0, 1, dummy) + local = acc.view(-1).to(torch.int64) + value = logits.float().gather(1, local[:, None]).squeeze(1) + return make_keys(value, local + rank * TARGET_SHARD) + + +@pytest.mark.parametrize( + "nreq,ties", [(1, False), (2, False), (4, False), (8, True), (12, False)], ids=str +) +def test_fused_verify_is_the_stock_greedy_acceptance(nreq, ties) -> None: + """Target logits (in-tree BF16 head) and verify outputs bit-identical to the in-tree path, on all four ranks.""" + g = gen(seed_of(f"verify {nreq} {ties}")) + rows = nreq * (DRAFT_LEN + 1) + x = randn((rows, HIDDEN), g, 1.0) + weight = randn((VOCAB, HIDDEN), g, 0.02) + if ties: + # Two equal head rows on ranks 1 and 3 that win every row: the lower global id must be the token. + win = (x.float().sum(0) / x.float().sum(0).norm() * 4.0).to(torch.bfloat16) + weight[TARGET_SHARD + 77] = win + weight[3 * TARGET_SHARD + 5] = win + heads = [_lm_head_shard(weight, r) for r in range(WORLD)] + # Stock: each rank's LMHead forward, all-gathered, then LogitsProcessor's float cast. + shard_logits = [head(x) for head in heads] + for head, stock in zip(heads, shard_logits): + assert torch.equal(head.apply_linear(x, None), stock), ( + "the verify's shard GEMM is not the head's" + ) + gathered = torch.cat(shard_logits, dim=1).float() + target_tokens = gathered.argmax(dim=-1).to(torch.int32) + if ties: + assert bool((target_tokens == TARGET_SHARD + 77).all()) + draft = craft_drafts(target_tokens, nreq, DRAFT_LEN) + stock_acc, stock_num = _stock_accept(gathered, draft) + peer_keys = {r: _local_keys(shard_logits[r], r) for r in range(WORLD)} + for rank in range(WORLD): + peers = {r: k for r, k in peer_keys.items() if r != rank} + acc, num, gather, next_new = _run_shard_accept( + shard_logits[rank], rank, WORLD, draft, peers + ) + assert torch.equal(acc, stock_acc.to(torch.int32)), f"rank {rank}: accepted tokens" + assert torch.equal(num, stock_num.to(torch.int32)), f"rank {rank}: accepted counts" + expect_gather = torch.arange(nreq, device="cuda") * (DRAFT_LEN + 1) + num.long() - 1 + assert torch.equal(gather, expect_gather), f"rank {rank}: first-pass gather rows" + assert torch.equal(next_new[:, 0], target_tokens[expect_gather]), f"rank {rank}: next_new" + + +class _Drafter: + """The in-tree Eagle3 first-pass pieces at the target's shapes: per-layer fc_norm RMSNorms and the fc.""" + + def __init__(self, g: torch.Generator): + self.norms = [] + for _ in range(CAPTURED_LAYERS): + norm = RMSNorm(hidden_size=HIDDEN, eps=EPS, dtype=torch.bfloat16).cuda() + norm.weight.data = (1.0 + 0.1 * torch.randn(HIDDEN, generator=g, device="cuda")).to( + torch.bfloat16 + ) + self.norms.append(norm) + self.fc = Linear(HIDDEN * CAPTURED_LAYERS, HIDDEN, bias=False, dtype=torch.bfloat16).cuda() + self.fc.weight.data = randn((HIDDEN, HIDDEN * CAPTURED_LAYERS), g, 0.02) + self.model = SimpleNamespace( + dtype=torch.bfloat16, + hidden_size=HIDDEN, + fc_norm=torch.nn.ModuleList(self.norms), + _norm_before_fc=False, + fc=self.fc, + ) + self.stacked = torch.stack([n.weight.detach() for n in self.norms]).contiguous() + + def stock_normed(self, hs: torch.Tensor) -> torch.Tensor: + """``apply_eagle3_fc``'s fc input: each captured layer through its own in-tree RMSNorm.""" + chunks = hs.chunk(CAPTURED_LAYERS, dim=-1) + return torch.cat([norm(c) for norm, c in zip(self.norms, chunks)], dim=-1) + + def stock_fc(self, hs: torch.Tensor) -> torch.Tensor: + return Eagle3ForCausalLM.apply_eagle3_fc(self, hs) + + +def _captures(rows: int, g: torch.Generator): + """Per captured layer (hidden, residual) and the in-tree capture buffer of bf16(hidden + residual).""" + layers = [(randn((rows, HIDDEN), g, 1.5), randn((rows, HIDDEN), g, 3.0)) for _ in range(3)] + buf = torch.cat([h + r for h, r in layers], dim=-1) + return layers, buf + + +def _row_identical(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + return (a == b).all(dim=-1) + + +@pytest.mark.parametrize("rows", (4, 8, 32, 48), ids=lambda r: f"rows{r}") +def test_first_drafter_inputs_are_the_stock_fc_norm(rows) -> None: + """chunk_norm and capture_norm against ``apply_eagle3_fc``'s in-tree RMSNorms: within one bf16 ulp on at most + NORM_ULP_FRAC of the elements; the fc (the in-tree module in both paths) then differs only on rows whose input + did.""" + g = gen(seed_of(f"fc_norm {rows}")) + drafter = _Drafter(g) + layers, buf = _captures(rows, g) + stock_in = drafter.stock_normed(buf) + stock_out = drafter.stock_fc(buf) + assert torch.equal(drafter.fc(stock_in), stock_out) + + chunk_in = m3_spec_tail.chunk_norm(buf, drafter.stacked, EPS) + capture_in = torch.empty_like(buf) + for i, (h, r) in enumerate(layers): + m3_spec_tail.capture_norm( + h, r, drafter.stacked[i], EPS, capture_in[:, i * HIDDEN : (i + 1) * HIDDEN] + ) + torch.cuda.synchronize() + for name, ours in (("chunk_norm", chunk_in), ("capture_norm", capture_in)): + assert_bf16_close(ours, stock_in, max_frac=NORM_ULP_FRAC) + out = drafter.fc(ours) + same_rows = _row_identical(ours, stock_in) + assert torch.equal(out[same_rows], stock_out[same_rows]), ( + f"{name}: fc moved an unchanged row" + ) + moved = (ours != stock_in).float().mean().item() + print( + f"[exact-numerics] rows={rows} {name}: {moved:.2e} of fc inputs one ulp off, " + f"{int((~same_rows).sum())}/{rows} rows differ, fc max |diff| " + f"{(out.float() - stock_out.float()).abs().max().item():.3e}" + ) + assert torch.equal(chunk_in, capture_in), "capture_norm and chunk_norm disagree" + + +@pytest.mark.parametrize("rows", (4, 32), ids=lambda r: f"rows{r}") +def test_draft_tokens_through_the_full_vocab_head(rows) -> None: + """The fc_norm's ulp-level differences carried through the full-vocabulary BF16 head (a proxy of the draft head + input): the greedy draft token equals the stock one on every row whose stock top-2 margin exceeds the logit + perturbation.""" + g = gen(seed_of(f"draft head {rows}")) + drafter = _Drafter(g) + _, buf = _captures(rows, g) + weight = randn((VOCAB, HIDDEN), g, 0.02) + head = LMHead(num_embeddings=VOCAB, embedding_dim=HIDDEN, dtype=torch.bfloat16) + head.weight.data = weight + stock_logits = head(drafter.stock_fc(buf)).float() + ours_logits = head(drafter.fc(m3_spec_tail.chunk_norm(buf, drafter.stacked, EPS))).float() + perturb = (ours_logits - stock_logits).abs().max(dim=-1).values + top2 = stock_logits.topk(2, dim=-1).values + decidable = (top2[:, 0] - top2[:, 1]) > 2 * perturb + stock_tok, ours_tok = stock_logits.argmax(-1), ours_logits.argmax(-1) + assert torch.equal(ours_tok[decidable], stock_tok[decidable]) + print( + f"[exact-numerics] rows={rows} draft head: {int((ours_tok == stock_tok).sum())}/{rows} tokens equal, " + f"{int(decidable.sum())} decidable, max logit |diff| {perturb.max().item():.3e}" + ) + + +def test_the_drafter_is_the_in_tree_drafter() -> None: + """Outside the fused draft loop's steps the worker drafts with the in-tree loop, forward and sampling, and nothing + re-quantizes the drafter's linears or the LM heads: no hook of a quantized / pruned head or drafter projection is + left in the target, and the fused head streams the BF16 head weight itself.""" + assert ( + modeling.M3Eagle3Worker._forward_draft_loop is not Eagle3OneModelWorker._forward_draft_loop + ) + for name in ( + "_forward_linear_draft_loop", + "_run_draft_forward", + "sample_draft_tokens", + "sample_and_accept_draft_tokens", + ): + assert getattr(modeling.M3Eagle3Worker, name) is getattr(Eagle3OneModelWorker, name), name + for module, name in ( + (modeling, "M3DraftGemvLinearMethod"), + (modeling, "adopt_draft_linears"), + (modeling, "greedy_head_outputs"), + (modeling, "M3FP8LMHeadMethod"), + (modeling, "adopt_fp8_lm_heads"), + ): + assert not hasattr(module, name), f"{module.__name__}.{name}" + pkg = modeling.__name__.rsplit(".", 1)[0] + for gone in ("kernels.m3_draft_head", "ops.spec.m3_draft_head", "ops.gemm.m3_draft_gemv"): + assert importlib.util.find_spec(f"{pkg}.{gone}") is None, gone diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_fold_tail.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_fold_tail.py new file mode 100644 index 000000000000..3c24955f9944 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_fold_tail.py @@ -0,0 +1,302 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the forward-tail spec fold (kernels/m3_fold_tail and M3SpecStepFold.in_forward). + +The tail must write exactly what the sampler's store (slot_scatter) and then the engine's next-step gathers +(step_gather, rows = the stored rows in order, first previous row 0) write, for the rows the step armed, and nothing +when the step was not armed. +""" + +import types + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + modeling as spec_fold, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( + m3_fold_tail, + m3_spec_tail, +) + +assert torch.cuda.is_available(), "m3_fold_tail requires a CUDA device" + +VOCAB = 200064 +NUM_SLOTS = 9 +MAX_ROWS = 8 +STALE = -4 + + +def _store(widths, g): + wn, wx, wd = widths + i32 = dict(dtype=torch.int32, device="cuda") + return types.SimpleNamespace( + new_tokens=torch.randint( + 0, VOCAB, (wn, NUM_SLOTS, 1), generator=g, device="cuda", dtype=torch.int32 + ), + next_new_tokens=torch.randint( + 0, VOCAB, (wx, NUM_SLOTS, 1), generator=g, device="cuda", dtype=torch.int32 + ), + next_draft_tokens=torch.randint( + 0, VOCAB, (NUM_SLOTS, wd), generator=g, device="cuda", dtype=torch.int32 + ), + new_tokens_lens=torch.randint(1, 5, (NUM_SLOTS,), generator=g, **i32), + ) + + +def _clone(ns): + return types.SimpleNamespace(**{k: v.clone() for k, v in vars(ns).items()}) + + +def _outputs(rows, t, g): + d = t - 1 + return { + "new_tokens": torch.randint( + 0, VOCAB, (rows, t), generator=g, device="cuda", dtype=torch.int32 + ), + "next_new_tokens": torch.randint( + 0, VOCAB, (rows, t), generator=g, device="cuda", dtype=torch.int32 + ), + "next_draft_tokens": torch.randint( + 0, VOCAB, (rows, d), generator=g, device="cuda", dtype=torch.int32 + ), + "new_tokens_lens": torch.randint( + 1, t + 1, (rows,), generator=g, device="cuda", dtype=torch.int32 + ), + } + + +def _engine(t): + i32 = dict(dtype=torch.int32, device="cuda") + return ( + torch.full((MAX_ROWS * t,), STALE, **i32), + torch.full((MAX_ROWS * (t - 1),), STALE, **i32), + torch.full((MAX_ROWS * t,), STALE, **i32), + torch.full((MAX_ROWS,), STALE, **i32), + ) + + +def _reference(store, engine, slots, out, t): + """slot_scatter into the store, then step_gather of the next step's inputs from it.""" + assert m3_spec_tail.slot_scatter( + slots, + out["new_tokens"], + store.new_tokens.squeeze(-1).T, + out["next_new_tokens"], + store.next_new_tokens.squeeze(-1).T, + out["new_tokens_lens"], + store.new_tokens_lens, + out["next_draft_tokens"], + store.next_draft_tokens, + ) + p = slots.numel() + ids, draft, pos_off, kv_off = engine + slots32 = slots.to(torch.int32) + assert m3_spec_tail.step_gather( + store.next_new_tokens, + store.next_draft_tokens, + store.new_tokens_lens, + slots32, + slots32.repeat_interleave(t), + p, + t, + t - 1, + ids, + 0, + draft, + 0, + pos_off, + kv_off, + 0, + ) + + +def _tail(store, engine, ctrl, slots_buf, out, t, rows, cpp=False): + ids, draft, pos_off, kv_off = engine + assert m3_fold_tail.fold_tail( + ctrl, + slots_buf, + rows, + out["new_tokens"], + store.new_tokens.squeeze(-1).T, + out["next_new_tokens"], + store.next_new_tokens.squeeze(-1).T, + out["next_draft_tokens"], + store.next_draft_tokens, + out["new_tokens_lens"], + store.new_tokens_lens, + ids, + draft, + pos_off, + kv_off, + t, + cpp=cpp, + ) + + +@pytest.mark.parametrize("cpp", [False, True], ids=["triton", "cpp"]) +@pytest.mark.parametrize("rows", [1, 2, 4, 8]) +@pytest.mark.parametrize("widths", [(4, 4, 3), (8, 8, 7)], ids=["store=step", "store wider"]) +def test_tail_matches_scatter_then_gather(rows, widths, cpp) -> None: + """Store bit for bit; engine rows [0, P*T) / [0, P) bit for bit (the forward reads only those rows).""" + t = 4 + g = torch.Generator(device="cuda").manual_seed(1000 * rows + widths[0]) + store = _store(widths, g) + out = _outputs(rows, t, g) + slots = torch.randperm(NUM_SLOTS, generator=g, device="cuda")[:rows] + ref_store, tail_store = _clone(store), _clone(store) + ref_engine, tail_engine = _engine(t), _engine(t) + _reference(ref_store, ref_engine, slots, out, t) + ctrl = torch.tensor([rows], dtype=torch.int32, device="cuda") + slots_buf = torch.zeros(MAX_ROWS, dtype=torch.long, device="cuda") + slots_buf[:rows] = slots + _tail(tail_store, tail_engine, ctrl, slots_buf, out, t, rows, cpp) + torch.cuda.synchronize() + for name in vars(store): + assert torch.equal(getattr(ref_store, name), getattr(tail_store, name)), name + spans = (rows * t, rows * (t - 1), rows * t, rows) + for k, (ref, got, n) in enumerate(zip(ref_engine, tail_engine, spans)): + assert torch.equal(ref[:n], got[:n]), k + assert torch.all(got[n:] == STALE), k + + +@pytest.mark.parametrize("cpp", [False, True], ids=["triton", "cpp"]) +def test_unarmed_tail_writes_nothing(cpp) -> None: + """ctrl[0] = 0 (the step was not armed): the store and the engine buffers are untouched.""" + t, rows = 4, 2 + g = torch.Generator(device="cuda").manual_seed(7) + store = _store((4, 4, 3), g) + out = _outputs(rows, t, g) + before, engine = _clone(store), _engine(t) + ctrl = torch.zeros(1, dtype=torch.int32, device="cuda") + slots_buf = torch.arange(MAX_ROWS, dtype=torch.long, device="cuda") + _tail(store, engine, ctrl, slots_buf, out, t, rows, cpp) + torch.cuda.synchronize() + for name in vars(store): + assert torch.equal(getattr(before, name), getattr(store, name)), name + for buf in engine: + assert torch.all(buf == STALE) + + +def _fold(): + fold = spec_fold.M3SpecStepFold(in_forward=True) + t = 4 + engine = _engine(t) + fold.bind_engine_buffers(*engine, max_batch_size=MAX_ROWS) + g = torch.Generator(device="cuda").manual_seed(11) + fold.bind_sampler_store(_store((4, 4, 3), g)) + return fold, g, t + + +def test_fold_handshake_stages_only_confirmed_rows() -> None: + """begin_step arms; the sampler confirms only the armed rows from the tail's outputs; the next step is staged only + for exactly those rows in that order.""" + fold, g, t = _fold() + assert fold.begin_step([3, 5], 2) is False # nothing staged before the first forward + out = _outputs(2, t, g) + fold.store_in_forward(out) + assert fold.sampler_rows_stored(out, 0, [3, 5]) is True + assert fold.begin_step([3, 5], 2) is True + out = _outputs(2, t, g) + fold.store_in_forward(out) + assert ( + fold.sampler_rows_stored(out, 0, [5, 3]) is False + ) # other order: the sampler stores itself + assert fold.begin_step([5, 3], 2) is False + + +def test_fold_handshake_needs_the_tail_outputs() -> None: + """Outputs the tail did not run on (another dict, a mixed step, an unarmed step) are never reported stored.""" + fold, g, t = _fold() + fold.begin_step([1], 1) + fold.store_in_forward(_outputs(1, t, g)) + assert fold.sampler_rows_stored(_outputs(1, t, g), 0, [1]) is False + fold.begin_step(None, 1) + out = _outputs(1, t, g) + fold.store_in_forward(out) + assert fold.sampler_rows_stored(out, 0, [1]) is False + fold.begin_step([1], 1) + out = _outputs(1, t, g) + fold.store_in_forward(out) + assert fold.sampler_rows_stored(out, 1, [1]) is False # rows were skipped (chunked context) + + +def test_a_padded_step_after_an_armed_one_writes_nothing() -> None: + """K3 #150 check #5: CUDA-graph padding rows carry no seq slot, so the engine passes rows=None for a padded step. + Even right after an armed step, that step uploads ctrl 0 and its forward tail leaves the store and the engine + buffers untouched (a dummy row never writes a live slot).""" + fold, g, t = _fold() + fold.begin_step([2, 4], 2) + fold.store_in_forward(_outputs(2, t, g)) + torch.cuda.synchronize() + assert int(fold._ctrl.item()) == 2 + + fold.begin_step(None, 4) + torch.cuda.synchronize() + assert int(fold._ctrl.item()) == 0 + store_before = _clone(fold.store) + engine_before = [buf.clone() for buf in fold.step_inputs] + fold.store_in_forward(_outputs(4, t, g)) + torch.cuda.synchronize() + for name in vars(store_before): + assert torch.equal(getattr(store_before, name), getattr(fold.store, name)), name + for before, buf in zip(engine_before, fold.step_inputs): + assert torch.equal(before, buf) + + +def test_fold_off_by_default() -> None: + fold = spec_fold.M3SpecStepFold() + assert not fold.in_forward + assert fold.begin_step([0], 1) is False + + +@pytest.mark.parametrize( + "in_forward, store_copy_wait, expected", + [(True, True, True), (True, False, False), (False, True, False)], +) +def test_the_store_copy_event_exists_only_with_the_tail_and_the_wait( + in_forward, store_copy_wait, expected +): + """STEP_SPEC_STORE_COPY_SIDE: an external event the sampler records after its side-stream store copies and the + forward's tail waits on; only a fold that stores in the forward needs it.""" + fold = spec_fold.M3SpecStepFold(in_forward=in_forward, store_copy_wait=store_copy_wait) + device = torch.device("cuda") + fold.bind_engine_buffers( + torch.zeros(8, dtype=torch.int32, device=device), + torch.zeros(8, dtype=torch.int32, device=device), + torch.zeros(8, dtype=torch.int32, device=device), + torch.zeros(8, dtype=torch.int32, device=device), + 4, + ) + assert (fold.store_copies_done is not None) == expected + if expected: + fold.store_copies_done.synchronize() + + +def test_the_store_wait_precedes_only_a_plain_launch_under_capture(): + """The store wait is an external-event wait node under graph capture, which drops the next kernel's PDL edge: + it may precede the fold tail (a plain launch); any other launch fails the capture (eager steps are not + checked).""" + fold = spec_fold.M3SpecStepFold(in_forward=True, store_copy_wait=True) + device = torch.device("cuda") + fold.bind_engine_buffers( + *(torch.zeros(8, dtype=torch.int32, device=device) for _ in range(4)), 4 + ) + assert fold.STORE_WAIT_BEFORE == frozenset({"fold_tail"}) + fold._wait_store_copies(before="draft_step") # eager: allowed + x = torch.zeros(4, device=device) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + fold._wait_store_copies(before="fold_tail") + x.add_(1) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(x, torch.ones(4, device=device)) + bad = torch.cuda.CUDAGraph() + with pytest.raises(RuntimeError, match="PDL edge"): + with torch.cuda.graph(bad, stream=stream): + fold._wait_store_copies(before="draft_step") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_force_vec.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_force_vec.py new file mode 100644 index 000000000000..5f58290c222e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_force_vec.py @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_force_vec entry of the MiniMax-M3 tp4 target: the fractional forced-acceptance draw.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_force_vec import ( # noqa: E501 + m3_force_vec as op, +) +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_force_vec requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(5) + pool = torch.rand(_FORCE_ACCEPT_RNG_POOL_SIZE, generator=g, device="cuda") + counter = torch.tensor([41], dtype=torch.int64, device="cuda") + out = torch.empty(spec["batch"], dtype=torch.int32, device="cuda") + args = ( + pool, + counter, + out, + spec["batch"], + 2, + 4, + spec["frac"], + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + spec["block"], + ) + ref = op.reference(*args) + with validating(op): + op(*args) + assert int(counter.item()) == 42, "the draw bumps the step counter once" + assert torch.equal(out, ref) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_force_vec_req.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_force_vec_req.py new file mode 100644 index 000000000000..e0a13a0c523b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_force_vec_req.py @@ -0,0 +1,93 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_force_vec_req entry of the MiniMax-M3 tp4 target: the per-request fractional forced-acceptance +draw (FORCE_AL_PER_REQUEST: one draw per request at its own key, its KV length at the verify, no shared counter), so +a request's forced count does not depend on the call history, on its slot in the batch, or on the other requests.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_force_vec_req import ( # noqa: E501 + m3_force_vec_req as op, +) +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, + _FORCE_ACCEPT_RNG_SEED, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_force_vec_req requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(7) + pool = torch.rand(_FORCE_ACCEPT_RNG_POOL_SIZE, generator=g, device="cuda") + # Each request's key is its KV length at the verify: long, distinct, arbitrary. + keys = torch.randint( + 1, 1 << 20, (spec["batch"],), generator=g, device="cuda", dtype=torch.int64 + ).to(torch.int32) + out = torch.empty(spec["batch"], dtype=torch.int32, device="cuda") + args = ( + pool, + keys, + out, + spec["batch"], + 2, + 4, + spec["frac"], + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + spec["block"], + ) + ref = op.reference(*args) + with validating(op): + op(*args) + assert torch.equal(out, ref) + + +def _pool(): + g = torch.Generator(device="cpu") + g.manual_seed(_FORCE_ACCEPT_RNG_SEED) + return torch.rand(_FORCE_ACCEPT_RNG_POOL_SIZE, dtype=torch.float32, generator=g).cuda() + + +def _draw(pool, keys, base_total=2, max_total=4, frac=0.6): + out = torch.empty(keys.numel(), dtype=torch.int32, device="cuda") + return op( + pool, + keys, + out, + keys.numel(), + base_total, + max_total, + frac, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + max(16, 1 << (keys.numel() - 1).bit_length()), + ) + + +def test_a_request_draws_the_same_whatever_ran_before_or_beside_it() -> None: + pool = _pool() + keys = torch.tensor([165_521, 40_003, 323_007, 7], dtype=torch.int32, device="cuda") + first = _draw(pool, keys) + for _ in range(5): # no shared counter: repeated calls draw the same + assert torch.equal(_draw(pool, keys), first) + perm = torch.tensor([2, 0, 3, 1], device="cuda") + assert torch.equal(_draw(pool, keys[perm]), first[perm]) # slot-independent + alone = torch.cat([_draw(pool, keys[i : i + 1]) for i in range(4)]) + assert torch.equal(alone, first) # independent of the other requests + + +def test_the_counts_follow_the_forced_fraction() -> None: + pool = _pool() + keys = torch.arange(100_000, 100_000 + 64 * 400, dtype=torch.int32, device="cuda") + counts = torch.cat([_draw(pool, keys[i : i + 64]) for i in range(0, keys.numel(), 64)]) + assert set(counts.tolist()) <= {2, 3} + extra = (counts == 3).float().mean().item() + assert abs(extra - 0.6) < 0.02, extra diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_lowm_splitk.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_lowm_splitk.py new file mode 100644 index 000000000000..07eac340f8a6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_lowm_splitk.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the target's copy of FlashInfer's split-K low-M GEMM (kernels/m3_lowm_splitk, +kernels/m3_draft_lowm): bit for bit FlashInfer's kernel at every certified split-K cell of the drafter's +linears, eager and in a CUDA graph, with and without PDL.""" + +import pytest +import torch +from flashinfer.gemm.kernels.dense_bf16_gemm_sm100_splitk import SplitKTactic as FiTactic +from flashinfer.gemm.kernels.dense_bf16_gemm_sm100_splitk import run_splitk_dense as fi_run + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.m3_lowm_splitk import ( # noqa: E501 + SplitKTactic, + run_splitk_dense, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_draft_lowm import ( # noqa: E501 + ROWS, + tactic_for, +) + +assert torch.cuda.is_available(), "m3_lowm_splitk requires a CUDA device" + +# Every certified (N, K, rows) of the TP4 drafter table that a split-K tactic serves: (N, K, rows, tactic). +CELLS = [ + (nk[0], nk[1], m, tactic_for(nk[0], nk[1], m)[1]) + for nk, rows in sorted(ROWS.items()) + for m in rows + if tactic_for(nk[0], nk[1], m)[0] == "s" +] + + +def test_the_table_has_the_certified_split_k_cells() -> None: + assert sorted((n, k, m) for n, k, m, _ in CELLS) == [ + (2304, 12288, 4), + (6144, 2048, 4), + (6144, 4608, 1), + (6144, 4608, 4), + (9216, 6144, 4), + ] + + +@pytest.mark.parametrize("use_pdl", [False, True], ids=["plain", "pdl"]) +@pytest.mark.parametrize("cell", CELLS, ids=[f"{n}x{k}@{m}" for n, k, m, _ in CELLS]) +def test_bitwise_flashinfer(cell, use_pdl) -> None: + n, k, m, tactic = cell + g = torch.Generator(device="cuda").manual_seed(n + 7 * k + m) + for trial in range(3): + x = (torch.randn(m, k, generator=g, device="cuda") * (1.0 + trial)).to(torch.bfloat16) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + ref = torch.empty(m, n, dtype=torch.bfloat16, device="cuda") + got = torch.empty(m, n, dtype=torch.bfloat16, device="cuda") + fi_run(x, w.t(), None, ref, use_pdl, FiTactic(*tactic)) + run_splitk_dense(x, w.t(), None, got, use_pdl, SplitKTactic(*tactic)) + torch.cuda.synchronize() + assert torch.equal(got.view(torch.int16), ref.view(torch.int16)), f"trial {trial}" + # In a CUDA graph, as the drafter runs it. + cap = torch.empty(m, n, dtype=torch.bfloat16, device="cuda") + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + run_splitk_dense(x, w.t(), None, cap, use_pdl, SplitKTactic(*tactic)) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(cap.view(torch.int16), ref.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_sampled_accept.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_sampled_accept.py new file mode 100644 index 000000000000..87b46e887e9a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_sampled_accept.py @@ -0,0 +1,297 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_sampled_accept entry (fused temperature + top-p verify) and its workspace size.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_sampled_accept import ( # noqa: E501 + m3_sampled_accept as op, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_sampled_accept_ws_words import ( # noqa: E501 + m3_sampled_accept_ws_words as ws_words, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import gen, seed_of # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_sampled_accept requires a CUDA device" + +DRAFT_LEN = 3 +#: Hot tokens of the distribution cells; every other logit sits 22 below the top (mass ~ 1e-5 of the total). +HEAD = 12 +#: Draws per distribution cell, and the total-variation band against the reference distribution. With 12 +#: categories and >= 4000 draws the expected TV of an exact sampler is ~0.02 (sd ~0.006); 0.06 is ~6 sd above it +#: while a wrong nucleus cut (one category too many or too few, >= 1 / 12 of the mass moved) lands at >= 0.08. +DRAWS = 4000 +TV_BAND = 0.06 + + +def _logits(spec, g, rows): + vs = spec["vs"] + if spec["check"] == "flat": + return (torch.randn((rows, vs), generator=g, device="cuda") * 0.3).to(torch.bfloat16) + if spec["check"] == "argmax": + return (torch.randn((rows, vs), generator=g, device="cuda") * 3).to(torch.bfloat16) + row = torch.full((vs,), -20.0, device="cuda") + hot = torch.randperm(vs, generator=g, device="cuda")[:HEAD] + row[hot] = torch.linspace(2.0, 0.2, HEAD, device="cuda") + return row.to(torch.bfloat16).expand(rows, vs).contiguous() + + +def _case(spec, seed): + g = gen(seed) + nreq = spec["nreq"] + rows = nreq * (DRAFT_LEN + 1) + logits = _logits(spec, g, rows) + words = ws_words(1) + uc = torch.zeros(words, dtype=torch.int64, device="cuda") + counters = torch.zeros(2, dtype=torch.int32, device="cuda") + tokens = torch.zeros(48, dtype=torch.int32, device="cuda") + extra = torch.zeros(1, dtype=torch.int32, device="cuda") + temperatures = torch.full((rows,), spec["temperature"], dtype=torch.float32, device="cuda") + top_ps = torch.full((rows,), spec["top_p"], dtype=torch.float32, device="cuda") + seeds = torch.randint(0, 2**62, (rows,), generator=g, device="cuda", dtype=torch.int64) + offsets = torch.zeros(rows, dtype=torch.int64, device="cuda") + # Drafts drawn from the likeliest tokens, so some prefixes match. + top = logits[0].float().topk(3).indices.to(torch.int32) + spec["vocab_offset"] + draft = top[torch.randint(0, 3, (nreq, DRAFT_LEN), generator=g, device="cuda")].contiguous() + batch_idx = torch.arange(nreq, dtype=torch.int32, device="cuda") + acc = torch.full((nreq, DRAFT_LEN + 1), -7, dtype=torch.int32, device="cuda") + num = torch.full((nreq,), -7, dtype=torch.int32, device="cuda") + gather = torch.full((nreq,), -7, dtype=torch.int64, device="cuda") + next_new = torch.full((nreq, DRAFT_LEN + 1), -7, dtype=torch.int32, device="cuda") + force = spec.get("force", "none") + force_total = 3 if force == "total" else 0 + force_vec = (torch.randint(1, DRAFT_LEN + 2, (nreq,), generator=g, device="cuda", dtype=torch.int32) + if force == "vec" else None) # fmt: skip + return [logits, spec["vocab_offset"], temperatures, top_ps, seeds, offsets, uc, uc.data_ptr(), counters, tokens, + 0, 1, draft, batch_idx, acc, num, gather, force_total, next_new, force_vec, extra, True] # fmt: skip + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Statistical for the draw, exact for everything derived from it. + + Every drawn token has nonzero reference probability (lies in the nucleus); the acceptance outputs equal the + reference's strict acceptance of the drawn tokens, every call; the call counter advances once per call; and, + for the distribution cells, the draws over many calls (fresh Philox offsets) are within TV_BAND of the + reference distribution (see DRAWS / TV_BAND). + """ + spec = cell.spec + args = _case(spec, seed_of(cell.why)) + logits, offset, rows = args[0], spec["vocab_offset"], args[0].shape[0] + certifying(op) + ref = op.reference(*args) + probs = ref["probs"] + calls = 1 if spec["check"] == "argmax" else -(-DRAWS // rows) + counts = torch.zeros(logits.shape[1], dtype=torch.float64, device="cuda") + for c in range(calls): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * 1000) + op(*args) + drawn = args[9][:rows].clone() + local = drawn.long() - offset + assert bool(((local >= 0) & (local < logits.shape[1])).all()), "token outside the shard" + assert bool((probs.gather(1, local[:, None]) > 0).all()), ( + "a drawn token outside the nucleus" + ) + acc = ref["accept"](drawn) + assert torch.equal(args[14], acc["accepted"]) + assert torch.equal(args[15], acc["num_accepted"]) + assert torch.equal(args[16], acc["gather_ids"]) + assert torch.equal(args[18], acc["next_new"]) + assert args[8].tolist() == [0, c + 1] + counts.index_add_(0, local, torch.ones(rows, dtype=torch.float64, device="cuda")) + if spec["check"] == "distribution": + emp = counts / counts.sum() + tv = 0.5 * (emp - probs[0].double()).abs().sum().item() + assert tv <= TV_BAND, f"total variation {tv:.3f} against the reference nucleus distribution" + + +@pytest.mark.parametrize("top_p", [0.001, 0.0002]) +def test_long_rejection_chains_stay_in_the_nucleus(top_p) -> None: + """A tiny nucleus over a flat tail: rows that need more exchange rounds than the workspace has slots. + + Each round's candidate is drawn over the tokens heavier than the last one, so a row takes ~ln(1 / top_p) + 1 + rounds on average, with a tail past the 16 exchange slots (about 1% of the rows at top_p 0.001). Rounds reuse the + slots cyclically, told apart by a lap-parity bit in the wire tags, and no round accepts unconditionally: every + draw, however many rounds it took, lies in the reference nucleus. World 1 (one rank's 8 CTAs per row exchange + through the same slot ring); the cross-rank path needs the multi-device job. + """ + spec = dict(nreq=1, vs=50016, vocab_offset=50016, check="flat", top_p=top_p, temperature=1.0) + args = _case(spec, seed_of(f"flat tail top_p {top_p}")) + logits, rows = args[0], args[0].shape[0] + probs = op.reference(*args)["probs"] + calls = -(-DRAWS // rows) + outside = torch.zeros((), dtype=torch.int64, device="cuda") + for c in range(calls): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * 1000) + op(*args) + local = args[9][:rows].long() - spec["vocab_offset"] + assert bool(((local >= 0) & (local < logits.shape[1])).all()), "token outside the shard" + outside += (probs.gather(1, local[:, None]) == 0).sum() + assert args[8].tolist() == [0, c + 1] + draws = calls * rows + rounds = 2.0 + args[20].item() / draws + kept = int((probs > 0).sum(1).min().item()) + assert outside.item() == 0, ( + f"{outside.item()} of {draws} draws outside the nucleus (kept >= {kept} of {logits.shape[1]}, " + f"{rounds:.2f} rounds per row on average)" + ) + assert rounds > 4.0, ( + f"only {rounds:.2f} rounds per row: the rows no longer stress the slot ring" + ) + + +@pytest.mark.parametrize("cell", ws_words.CELLS, ids=[c.why[:44] for c in ws_words.CELLS]) +def test_workspace_words(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + certifying(ws_words) + world = cell.spec["world"] + assert ws_words(world) == ws_words.reference(world) + + +def test_guard_refuses_top_k_rows() -> None: + args = _case(op.CELLS[0].spec, 1) + top_ks = torch.full((args[0].shape[0],), 20, dtype=torch.int32, device="cuda") + with pytest.raises(AssertionError, match="top-k rows"): + with validating(op): + op(*args, top_ks=top_ks) + + +def test_guard_refuses_fp64_temperatures() -> None: + args = _case(op.CELLS[0].spec, 2) + args[2] = args[2].double() + with pytest.raises(AssertionError, match="fp32 sampling parameters"): + with validating(op): + op(*args) + + +def test_the_verify_calls_this_targets_op_pair() -> None: + """The verify calls this target's op pair (the sm_107 tp4 kernel), whose workspace sizes as the shared one.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_sampled_accept as kernels, + ) + + assert kernels._sampled_accept_op is op + assert kernels._ws_words_op is ws_words + assert int(torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept_ws_words(4, 0)) == int( + torch.ops.trtllm.minimax_m3_sampled_accept_ws_words(4) + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_prewait_parameter_loads_change_nothing(cell) -> None: + """dispatch.SAMPLED_ACCEPT_PREWAIT_PARAMS moves only where the sampling parameters load (before or after the PDL + wait): the same draws and acceptance outputs, bit for bit, call after call.""" + outs = [] + for prewait in (True, False): + args = _case(cell.spec, seed_of(cell.why)) + rows = args[0].shape[0] + got = [] + for c in range(3): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * 1000) + torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept(*args, prewait) + got.append([args[i].clone() for i in (9, 14, 15, 16, 18)]) + outs.append(got) + for call, (a, b) in enumerate(zip(*outs)): + for x, y in zip(a, b): + assert torch.equal(x, y), f"call {call}: prewait changes {x.tolist()} -> {y.tolist()}" + + +def _walk_outputs(args, one_round_k, ilp, calls=3, offset_step=1000): + """The draws and acceptance outputs of `calls` calls with the owner walk one token (ilp False) or 8 tokens at a + time (ilp True), at SA-1R variant `one_round_k`, each call at its own Philox offsets.""" + if one_round_k: + words = torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept_ws_words(1, one_round_k) + args[6] = torch.zeros(words, dtype=torch.int64, device="cuda") + args[7] = args[6].data_ptr() + rows = args[0].shape[0] + got = [] + for c in range(calls): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * offset_step) + torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept( + *args, True, False, one_round_k, None, ilp + ) + got.append([args[i].clone() for i in (9, 14, 15, 16, 18)]) + return got + + +@pytest.mark.parametrize("one_round_k", [0, 82]) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_the_ilp_walk_changes_nothing(cell, one_round_k) -> None: + """The owner walk's 8-token form (``ilp_walk``; off in this target) only computes the owner's token weights 8 at a + time; the walk adds them in the same order with the same early exit, so the draws and acceptance outputs are the + one-token walk's, bit for bit, call after call, with SA-1R off and on.""" + one = _walk_outputs(_case(cell.spec, seed_of(cell.why)), one_round_k, False) + ilp = _walk_outputs(_case(cell.spec, seed_of(cell.why)), one_round_k, True) + for call, (a, b) in enumerate(zip(one, ilp)): + for x, y in zip(a, b): + assert torch.equal(x, y), ( + f"call {call}: the 8-token walk changes {x.tolist()} -> {y.tolist()}" + ) + + +@pytest.mark.parametrize("top_p", [0.001, 0.95]) +def test_the_ilp_walk_changes_nothing_over_many_rounds(top_p) -> None: + """A flat tail (many tokens of equal weight) under a tiny and a wide nucleus: rows take many exchange rounds and + each round's walk exits at a different token; the 8-token walk draws the same tokens as the one-token walk.""" + spec = dict(nreq=1, vs=50016, vocab_offset=50016, check="flat", top_p=top_p, temperature=1.0) + one = _walk_outputs(_case(spec, seed_of(f"ilp walk flat {top_p}")), 0, False, calls=24) + ilp = _walk_outputs(_case(spec, seed_of(f"ilp walk flat {top_p}")), 0, True, calls=24) + for call, (a, b) in enumerate(zip(one, ilp)): + for x, y in zip(a, b): + assert torch.equal(x, y), ( + f"call {call}: the 8-token walk changes {x.tolist()} -> {y.tolist()}" + ) + + +def test_the_ilp_walk_is_the_same_under_graph_replay() -> None: + """The 8-token walk captured in a CUDA graph and replayed draws what the one-token walk draws eagerly.""" + cell = op.CELLS[0] + eager = _walk_outputs(_case(cell.spec, seed_of(cell.why)), 0, False) + _walk_outputs( + _case(cell.spec, seed_of("ilp walk warm-up")), 0, True, calls=1 + ) # first launch outside the capture + args = _case(cell.spec, seed_of(cell.why)) + rows = args[0].shape[0] + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept(*args, True, False, 0, None, True) + torch.cuda.current_stream().wait_stream(stream) + for c in range(3): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * 1000) + graph.replay() + torch.cuda.synchronize() + for x, y in zip(eager[c], [args[i] for i in (9, 14, 15, 16, 18)]): + assert torch.equal(x, y), ( + f"replay {c}: the graphed 8-token walk changes {x.tolist()} -> {y.tolist()}" + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_the_early_trigger_changes_nothing(cell) -> None: + """dispatch.SAMPLED_ACCEPT_EARLY_TRIGGER moves only where the grid releases its dependents: the same draws and + acceptance outputs, bit for bit, call after call, PDL launch on.""" + outs = [] + for early in (True, False): + args = _case(cell.spec, seed_of(cell.why)) + rows = args[0].shape[0] + got = [] + for c in range(3): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * 1000) + torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept(*args, True, early) + got.append([args[i].clone() for i in (9, 14, 15, 16, 18)]) + outs.append(got) + for call, (a, b) in enumerate(zip(*outs)): + for x, y in zip(a, b): + assert torch.equal(x, y), ( + f"call {call}: the early trigger changes {x.tolist()} -> {y.tolist()}" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_sampled_accept_1r.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_sampled_accept_1r.py new file mode 100644 index 000000000000..42bacb1467be --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_sampled_accept_1r.py @@ -0,0 +1,186 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SA-1R (dispatch.SAMPLED_ACCEPT_1R_K) on trtllm::minimax_m3_sm107tp4_sampled_accept, world 1, the TP4 shard. + +The one-round variants (one_round_k 9: exact top-9 list; 82: pooled top-8 + bound) must return the two-round +kernel's (one_round_k 0) sampled tokens, accepted tokens, accepted counts, gather ids and extra-attempt count BIT FOR +BIT on the same Philox stream, over random AgentX-like rows (several temperatures / top-p, broad and heavy-tailed +heads) and over rows built so the decision sits within ulps of top_p * Z (the margin fallback) or a slice holds more +heavy tokens than its list (the incomplete-list fallback), eager and under CUDA-graph replay. The round-0 decided +counter must engage on the AgentX-like rows and stay below the row count on the boundary rows (fallbacks fire). +Ported from the prototype test (exact on 39,968 rows). +""" + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* + +VS, D = 50016, 3 +KX = (9, 82) + +if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the sm_107 tp4 sampled accept runs on SM 10.7", allow_module_level=True) + + +class Runner: + """One variant with its own workspace / counters (call tag sequence) and outputs.""" + + def __init__(self, k, rows): + self.k, self.rows = k, rows + words = int(torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept_ws_words(1, k)) + self.ws = torch.zeros(words, dtype=torch.int64, device="cuda") + self.counters = torch.zeros(2, dtype=torch.int32, device="cuda") + self.tokens = torch.zeros(48, dtype=torch.int32, device="cuda") + self.extra = torch.zeros(1, dtype=torch.int32, device="cuda") + self.decided = torch.zeros(1, dtype=torch.int32, device="cuda") + nreq = rows // (D + 1) + self.acc = torch.zeros((nreq, D + 1), dtype=torch.int32, device="cuda") + self.nacc = torch.zeros(nreq, dtype=torch.int32, device="cuda") + self.gids = torch.zeros(nreq, dtype=torch.int64, device="cuda") + + def __call__(self, lg, temps, tps, seeds, offs, draft, bidx): + torch.ops.trtllm.minimax_m3_sm107tp4_sampled_accept( + lg, 0, temps, tps, seeds, offs, self.ws, self.ws.data_ptr(), self.counters, self.tokens, 0, 1, draft, + bidx, self.acc, self.nacc, self.gids, 0, None, None, self.extra, False, True, False, self.k, + self.decided) # fmt: skip + + def out(self): + return (self.tokens[: self.rows].clone(), self.acc.clone(), self.nacc.clone(), self.gids.clone(), + self.extra.clone()) # fmt: skip + + +def _params(rows, temp, top_p, seed0): + temps = torch.full((rows,), temp, dtype=torch.float32, device="cuda") + tps = ( + (top_p if torch.is_tensor(top_p) else torch.full((rows,), top_p)) + .float() + .cuda() + .contiguous() + ) + seeds = torch.arange(rows, dtype=torch.int64, device="cuda") * 7919 + seed0 + offs = torch.zeros(rows, dtype=torch.int64, device="cuda") + return temps, tps, seeds, offs + + +def _sets(n, rows, g, kind): + out = [] + for _ in range(n): + lg = torch.randn(rows, VS, device="cuda", generator=g) * (1.0 if kind == "flat" else 2.0) + if kind == "lm": # a few heavy tokens over a broad tail + hot = torch.randint(0, VS, (rows, 8), device="cuda", generator=g) + lg.scatter_(1, hot, 12.0 + 3.0 * torch.rand(rows, 8, device="cuda", generator=g)) + elif kind == "tail": # 64 near-equal top tokens: lists overflow + hot = torch.randint(0, VS, (rows, 64), device="cuda", generator=g) + lg.scatter_(1, hot, 9.0 + 0.5 * torch.rand(rows, 64, device="cuda", generator=g)) + out.append(lg.to(torch.bfloat16).contiguous()) + return out + + +def _draft_from(lg, g): + nreq = lg.shape[0] // (D + 1) + am = lg.float().argmax(1).view(nreq, D + 1)[:, :D].int() + rnd = torch.randint(0, VS, (nreq, D), device="cuda", generator=g, dtype=torch.int32) + return torch.where(torch.rand(nreq, D, device="cuda", generator=g) < 0.7, am, rnd).contiguous() + + +def _compare(ref, got, what): + for n, x, y in zip( + ("tokens", "acc_tokens", "num_acc", "gather_ids", "extra_attempts"), ref, got + ): + assert torch.equal(x, y), f"{what}: {n} differs" + + +CASES = [(1.0, 0.95, "lm"), (0.7, 0.8, "lm"), (1.3, 0.5, "lm"), (1.0, 1.0, "lm"), (1.0, 0.95, "flat"), + (1.0, 0.95, "tail"), (0.6, 0.99, "tail")] # fmt: skip + + +def test_one_round_is_two_round_bitwise(): + g = torch.Generator(device="cuda").manual_seed(2) + total = 0 + decided = {k: 0 for k in KX} + for rows, calls in ((32, 20), (4, 60)): + for ci, (t, p, kind) in enumerate(CASES): + runners = {k: Runner(k, rows) for k in (0, *KX)} + temps, tps, seeds, offs = _params(rows, t, p, 1000 + ci) + sets = _sets(4, rows, g, kind) + bidx = torch.arange(rows // (D + 1), dtype=torch.int32, device="cuda") + for i in range(calls): + lg = sets[i % 4] + draft = _draft_from(lg, g) + offs.fill_(i) + for r in runners.values(): + r(lg, temps, tps, seeds, offs, draft, bidx) + for k in KX: + _compare( + runners[0].out(), runners[k].out(), f"K={k} rows={rows} case={ci} call={i}" + ) + total += rows + if kind == "lm" and p < 1.0: + for k in KX: + decided[k] += int(runners[k].decided.item()) + assert int(runners[0].decided.item()) == 0 + assert total >= 5000 + for k in KX: + assert decided[k] > 0, ( + f"K={k}: no row decided in round 0 on the AgentX-like rows (SA-1R not engaged)" + ) + + +def _boundary_rows(rows, n_heavy_same_slice): + """Heavy tokens heavier than a 50-token candidate group; top_p swept across ulps of S / Z (margin fallback), or + n heavy tokens in one slice (more than its list: incomplete fallback).""" + lg = torch.full((rows, VS), -40.0, device="cuda") + sl = ((VS + 7) // 8 + 7) // 8 * 8 + heavy = list(range(5, 5 + n_heavy_same_slice)) if n_heavy_same_slice else [5, 3 * sl + 11] + grp = [2 * sl + 3 + 37 * j for j in range(25)] + [5 * sl + 7 + 41 * j for j in range(25)] + lg[:, heavy] = 5.0 + lg[:, grp] = 4.0 + lg = lg.to(torch.bfloat16).contiguous() + w = torch.exp(lg[0].double() - lg[0].double().max()) + crit = float(w[heavy].sum()) / float(w.sum()) + ks = torch.arange(rows, dtype=torch.float64) - rows // 2 + return lg, (crit * (1.0 + ks * 2.0**-24)).float() + + +@pytest.mark.parametrize("graph", [False, True]) +def test_one_round_boundary_and_incomplete(graph): + rows = 32 + g = torch.Generator(device="cuda").manual_seed(3) + seen = {k: [0, 0] for k in KX} # decided rows, rows + for n_same in (0, 12): + lg, tps_rows = _boundary_rows(rows, n_same) + for shift in range(-4, 4): + tps = (tps_rows.double() * (1.0 + shift * 32 * 2.0**-24)).float() + runners = {k: Runner(k, rows) for k in (0, *KX)} + temps, tp_t, seeds, offs = _params(rows, 1.0, tps, 5000 + shift) + bidx = torch.arange(rows // (D + 1), dtype=torch.int32, device="cuda") + draft = _draft_from(lg, g) + graphs = {} + if graph: + for r in runners.values(): + r(lg, temps, tp_t, seeds, offs, draft, bidx) + torch.cuda.synchronize() + for k, r in runners.items(): + graphs[k] = torch.cuda.CUDAGraph() + with torch.cuda.graph(graphs[k]): + r(lg, temps, tp_t, seeds, offs, draft, bidx) + for r in runners.values(): + r.decided.zero_() + for i in range(12): + offs.fill_(i) + for k, r in runners.items(): + if graph: + graphs[k].replay() + else: + r(lg, temps, tp_t, seeds, offs, draft, bidx) + for k in KX: + _compare(runners[0].out(), runners[k].out(), + f"boundary n_same={n_same} shift={shift} call={i} K={k}") # fmt: skip + for k in KX: + seen[k][0] += int(runners[k].decided.item()) + seen[k][1] += 12 * rows + for k in KX: + assert 0 < seen[k][0] < seen[k][1], ( + f"K={k}: decided {seen[k]}: the boundary sweep missed both sides" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_shard_accept.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_shard_accept.py new file mode 100644 index 000000000000..d5fae82473e9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_shard_accept.py @@ -0,0 +1,125 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_shard_accept entry (greedy verify from this rank's logits shard).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_shard_accept import ( # noqa: E501 + SHARD_ROWS, + exchange_tokens, + local_argmax, + make_keys, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_shard_accept import ( # noqa: E501 + m3_shard_accept as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 + craft_drafts, + gen, + head_workspace, + num_sms, + plant_peers, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_shard_accept requires a CUDA device" + + +def _case(spec, seed, edit_logits=None): + g = gen(seed) + nreq, d, vs, world, rank = ( + spec["nreq"], + spec["draft_len"], + spec["vs"], + spec["world"], + spec["rank"], + ) + m = nreq * (d + 1) + if spec["strided"]: + logits = randn((m, vs + 72), g, 3.0)[:, :vs] + else: + logits = randn((m, vs), g, 3.0) + if edit_logits is not None: + edit_logits(logits) + uc, keys, counters = head_workspace(world, slot_rows=SHARD_ROWS) + own = logits.float().max(dim=1).values + plant_peers( + uc, counters, own, rank, world, vs, g, slot_rows=SHARD_ROWS, ties=spec.get("ties", False) + ) + offset = rank * vs + best, idx = local_argmax(logits.float(), offset) + tokens = exchange_tokens(make_keys(best, idx), uc, counters, rank, world, SHARD_ROWS) + draft = craft_drafts(tokens, nreq, d) + batch_idx = torch.arange(nreq, dtype=torch.int32, device="cuda") + acc = torch.full((nreq, d + 1), -7, dtype=torch.int32, device="cuda") + num = torch.full((nreq,), -7, dtype=torch.int32, device="cuda") + gather = torch.full((nreq,), -7, dtype=torch.int64, device="cuda") + next_new = torch.full((nreq, d + 1), -7, dtype=torch.int32, device="cuda") + force_total = 3 if spec["force"] == "total" else 0 + force_vec = (torch.randint(1, d + 2, (nreq,), generator=g, device="cuda", dtype=torch.int32) + if spec["force"] == "vec" else None) # fmt: skip + return (logits, offset, keys, counters, uc, uc.data_ptr(), rank, world, draft, batch_idx, acc, num, gather, + force_total, next_new, num_sms(), force_vec) # fmt: skip + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Bit-exact: the op reduces BF16 logits without arithmetic, ties (within the shard and across ranks) go to + the lowest global index on both sides.""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + assert torch.equal(args[10], ref["accepted"]) + assert torch.equal(args[11], ref["num_accepted"]) + assert torch.equal(args[12], ref["gather_ids"]) + assert torch.equal(args[14], ref["next_new"]) + assert args[3].tolist() == [0, 1] and not bool(args[2].any()) + + +def test_guard_refuses_nan_logits() -> None: + args = list(_case(op.CELLS[0].spec, 1)) + args[0][1, 7] = float("nan") + with pytest.raises(AssertionError, match="NaN"): + with validating(op): + op(*args) + + +def test_guard_refuses_a_batch_index_past_the_requests() -> None: + args = list(_case(op.CELLS[0].spec, 2)) + args[9] = args[9] + 1 + with pytest.raises(AssertionError, match="batch_idx"): + with validating(op): + op(*args) + + +def _signed_zero_maxima(logits: torch.Tensor) -> None: + """Every row's maximum is zero, reached as -0.0 at a lower index and +0.0 at a higher one.""" + logits.fill_(-1.0) + logits[:, 3] = -0.0 + logits[:, 9] = 0.0 + + +@pytest.mark.parametrize("ties", [False, True], ids=["peer_rows", "peer_ties"]) +def test_signed_zero_maxima_take_the_lower_index(ties) -> None: + """-0.0 and +0.0 compare equal, so the lower index wins as with torch.argmax (also across ranks).""" + spec = dict(op.CELLS[1].spec, ties=ties) + args = _case(spec, seed_of(f"signed-zero-{ties}"), edit_logits=_signed_zero_maxima) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + assert torch.equal(args[10], ref["accepted"]) + assert torch.equal(args[11], ref["num_accepted"]) + assert torch.equal(args[12], ref["gather_ids"]) + assert torch.equal(args[14], ref["next_new"]) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_slot_scatter.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_slot_scatter.py new file mode 100644 index 000000000000..0e0482bb1111 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_slot_scatter.py @@ -0,0 +1,62 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_slot_scatter entry (the spec sampler's per-slot stores in one launch).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_slot_scatter import ( # noqa: E501 + m3_slot_scatter as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import VOCAB, gen, seed_of # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_slot_scatter requires a CUDA device" + +STALE = -4 + + +def _case(spec, seed): + """Worker outputs and the store's transposed [slots, width] views of its [width, slots, 1] buffers.""" + g = gen(seed) + n, nslots = spec["n"], spec["slots"] + + def rnd(shape): + return torch.randint(0, VOCAB, shape, generator=g, device="cuda", dtype=torch.int32) + + store_new = torch.full((spec["store_w"], nslots, 1), STALE, dtype=torch.int32, device="cuda") + store_next = torch.full((spec["store_w"], nslots, 1), STALE, dtype=torch.int32, device="cuda") + store_lens = torch.full((nslots,), STALE, dtype=torch.int32, device="cuda") + store_draft = torch.full((nslots, spec["store_d"]), STALE, dtype=torch.int32, device="cuda") + slots = torch.randperm(nslots, generator=g, device="cuda")[:n] + lens = torch.randint(1, 5, (n,), generator=g, device="cuda", dtype=torch.int32) + return (slots, rnd((n, spec["src_w"])), store_new.squeeze(-1).T, rnd((n, spec["src_w"])), + store_next.squeeze(-1).T, lens, store_lens, rnd((n, spec["src_d"])), store_draft) # fmt: skip + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Index copies: exact, zero padding included, other slots untouched.""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + for name, got in (("da", args[2]), ("db", args[4]), ("dl", args[6]), ("dd", args[8])): + assert torch.equal(got, ref[name]), name + + +def test_guard_refuses_repeated_slots() -> None: + args = list(_case(op.CELLS[0].spec, 1)) + args[0] = args[0].clone() + args[0][1] = args[0][0] + with pytest.raises(AssertionError, match="distinct"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_spec_offsets.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_spec_offsets.py new file mode 100644 index 000000000000..ca30e632b701 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_spec_offsets.py @@ -0,0 +1,138 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for m3_spec_offsets (RuntimeHooks.spec_offsets_op): the overlap scheduler's two speculative step corrections +in one plain launch, bit for bit the in-place ATen adds, eager and in a CUDA graph.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_spec_offsets import ( # noqa: E501 + m3_spec_offsets as op, +) + +assert torch.cuda.is_available(), "m3_spec_offsets requires a CUDA device" + +I32 = torch.iinfo(torch.int32) + + +def _inputs(tokens, seqs, seed, wrap=False): + g = torch.Generator(device="cuda").manual_seed(seed) + lo, hi = (I32.min, I32.max) if wrap else (0, 1 << 20) + pos = torch.randint(lo, hi, (tokens,), generator=g, device="cuda", dtype=torch.int32) + kv = torch.randint(lo, hi, (seqs,), generator=g, device="cuda", dtype=torch.int32) + pos_off = torch.randint( + -8 if not wrap else lo, + 8 if not wrap else hi, + (tokens,), + generator=g, + device="cuda", + dtype=torch.int32, + ) + kv_off = torch.randint( + -8 if not wrap else lo, + 8 if not wrap else hi, + (seqs,), + generator=g, + device="cuda", + dtype=torch.int32, + ) + if wrap: + pos[:2], pos_off[:2] = I32.max, 1 # overflow wraps, as the ATen add does + kv[:1], kv_off[:1] = I32.min, -1 + return pos, pos_off, kv, kv_off + + +CELLS = [c.spec for c in op.CELLS] + [ + dict(tokens=1, seqs=1), + dict(tokens=300, seqs=75), + dict(tokens=4, seqs=0), +] + + +@pytest.mark.parametrize("spec", CELLS, ids=lambda s: "-".join(f"{k}{v}" for k, v in s.items())) +def test_bitwise_aten_adds(spec) -> None: + for trial in range(3): + pos, pos_off, kv, kv_off = _inputs( + spec["tokens"], spec["seqs"], 97 * trial + spec["tokens"], spec.get("wrap", False) + ) + ref_pos, ref_kv = pos.clone(), kv.clone() + op.reference(ref_pos, pos_off, ref_kv, kv_off) + assert op(pos, pos_off, kv, kv_off) is True + torch.cuda.synchronize() + assert torch.equal(pos, ref_pos) and torch.equal(kv, ref_kv), f"trial {trial}" + + +def test_graph_replays_read_the_live_offsets() -> None: + pos, pos_off, kv, kv_off = _inputs(32, 8, 5) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(pos, pos_off, kv, kv_off) + for step in range(3): + fresh = _inputs(32, 8, 11 + step) + for dst, src in zip((pos, pos_off, kv, kv_off), fresh): + dst.copy_(src) + ref_pos, ref_kv = pos.clone(), kv.clone() + op.reference(ref_pos, pos_off, ref_kv, kv_off) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(pos, ref_pos) and torch.equal(kv, ref_kv), f"step {step}" + + +def test_runner_views_are_accepted() -> None: + """The runner passes position_ids[0, ctx:] and kv_lens_cuda[ctx_reqs:seqs]: 1-D slices of contiguous buffers.""" + position_ids = torch.arange(40, dtype=torch.int32, device="cuda").view(1, 40) + kv_lens = torch.arange(12, dtype=torch.int32, device="cuda") + pos_off = torch.full((40,), 2, dtype=torch.int32, device="cuda") + kv_off = torch.full((12,), 3, dtype=torch.int32, device="cuda") + assert op(position_ids[0, 8:], pos_off[:32], kv_lens[2:10], kv_off[:8]) is True + torch.cuda.synchronize() + assert torch.equal( + position_ids[0], torch.cat([torch.arange(8), torch.arange(8, 40) + 2]).int().cuda() + ) + assert torch.equal( + kv_lens, + torch.cat([torch.arange(2), torch.arange(2, 10) + 3, torch.arange(10, 12)]).int().cuda(), + ) + + +@pytest.mark.parametrize( + "bad", + ["int64", "strided", "length", "2d"], +) +def test_declines_other_layouts_without_writing(bad) -> None: + pos, pos_off, kv, kv_off = _inputs(8, 2, 3) + if bad == "int64": + kv = kv.long() + elif bad == "strided": + pos = torch.zeros(16, dtype=torch.int32, device="cuda")[::2] + elif bad == "length": + pos_off = pos_off[:7] + else: + pos = pos.view(2, 4) + before = (pos.clone(), kv.clone()) + assert op(pos, pos_off, kv, kv_off) is False + torch.cuda.synchronize() + assert torch.equal(pos, before[0]) and torch.equal(kv, before[1]) + + +def test_launches_plainly() -> None: + """It writes step inputs that pre-wait readers take before their grid wait (MSA contract, dispatch + PRODUCER_PREWAIT_STEP_INPUTS): its kernel node carries no programmatic (PDL) serialization.""" + from cuda.bindings import runtime as rt + + pos, pos_off, kv, kv_off = _inputs(32, 8, 7) + graph = torch.cuda.CUDAGraph(keep_graph=True) + with torch.cuda.graph(graph): + op(pos, pos_off, kv, kv_off) + raw = rt.cudaGraph_t(init_value=graph.raw_cuda_graph()) + err, _, n = rt.cudaGraphGetNodes(raw, 0) + err, nodes, n = rt.cudaGraphGetNodes(raw, n) + flags = [] + for node in nodes[:n]: + err, kind = rt.cudaGraphNodeGetType(node) + if kind == rt.cudaGraphNodeType.cudaGraphNodeTypeKernel: + err, attr = rt.cudaGraphKernelNodeGetAttribute( + node, rt.cudaLaunchAttributeID.cudaLaunchAttributeProgrammaticStreamSerialization + ) + flags.append(0 if attr is None else int(attr.programmaticStreamSerializationAllowed)) + assert flags == [0], flags diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_step_gather.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_step_gather.py new file mode 100644 index 000000000000..d058f1e5d39f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_step_gather.py @@ -0,0 +1,85 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_step_gather entry (the overlap scheduler's step-input gathers in one launch).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_step_gather import ( # noqa: E501 + m3_step_gather as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import VOCAB, gen, seed_of # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP4 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_step_gather requires a CUDA device" + +#: Rows of the engine's flat buffers (the max batch), and their stale fill. +MAX_ROWS = 8 +STALE = -4 + + +def _case(spec, seed): + g = gen(seed) + p, t, d, nslots, b0 = spec["p"], spec["t"], spec["d"], spec["slots"], spec["b0"] + i32 = dict(dtype=torch.int32, device="cuda") + new_tok = torch.randint(0, VOCAB, (t, nslots, 1), generator=g, device="cuda", dtype=torch.int32) + if spec.get("two_d"): + new_tok = new_tok.squeeze(-1) + next_draft = torch.randint(0, VOCAB, (nslots, d), generator=g, device="cuda", dtype=torch.int32) + lens = torch.randint(1, t + 1, (nslots,), generator=g, device="cuda", dtype=torch.int32) + slots = torch.randperm(nslots, generator=g, device="cuda")[:p].to(torch.int32) + ppi = slots.repeat_interleave(t) + ids = torch.full((MAX_ROWS * t,), STALE, **i32) + draft = torch.full((MAX_ROWS * d,), STALE, **i32) + pos_off = torch.full((MAX_ROWS * t,), STALE, **i32) + kv_off = torch.full((MAX_ROWS,), STALE, **i32) + return ( + new_tok, + next_draft, + lens, + slots, + ppi, + p, + t, + d, + ids, + spec["ids0"], + draft, + spec["d0"], + pos_off, + kv_off, + b0, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Index copies: exact, including the zero-filled offsets and the untouched rest of ids / draft.""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + for name, got in ( + ("ids", args[8]), + ("draft", args[10]), + ("pos_off", args[12]), + ("kv_off", args[13]), + ): + assert torch.equal(got, ref[name]), name + + +def test_guard_refuses_a_slot_past_the_store() -> None: + args = list(_case(op.CELLS[0].spec, 1)) + args[3] = args[3].clone() + args[3][0] = 8 + with pytest.raises(AssertionError, match="slots must index the store"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_step_prologue.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_step_prologue.py new file mode 100644 index 000000000000..296b8680efec --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/spec/test_modeling_v2_m3_step_prologue.py @@ -0,0 +1,192 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_step_prologue entry (the decode graph's step-input prologue in one launch).""" + +import random +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.spec.m3_step_prologue import ( # noqa: E501 + m3_step_prologue as op, +) +from tensorrt_llm._torch.pyexecutor import step_inputs +from tensorrt_llm._torch.pyexecutor.step_inputs import StepInputPack + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_step_prologue requires a CUDA device" + +MAX_DESC = StepInputPack.MAX_DESC +WINDOW = StepInputPack.GRAPH_WINDOW +DATA_START = StepInputPack.HEADER_BYTES + StepInputPack.DESC_BYTES * MAX_DESC +ARENA = 1 << 20 + + +def seed_of(why: str) -> int: + return zlib.crc32(why.encode()) % 2**31 + + +def gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def _layout(spec, seed): + """Copies as (dst, src, bytes) arena offsets (a host copy's src: a window offset) and the host copies' bytes.""" + rng = random.Random(seed) + h2d, d2d, data = [], [], [] + used = DATA_START + pos = 0 + for _ in range(spec["h2d"]): + n = rng.choice([1, 3, 4, 7, 16, 24, 32, 48, 64, 5, 12]) + pos += rng.randint(0, 40) # gaps and every alignment + h2d.append((pos, used, n)) + data.append(bytes(rng.getrandbits(8) for _ in range(n))) + pos += n + used += -(-n // 16) * 16 + assert pos < ARENA // 2 and used <= WINDOW + # Device copies write the last quarter; unchained ones read the third. + dpos, spos = 3 * ARENA // 4, ARENA // 2 + for i in range(spec["d2d"]): + if spec.get("chained") and h2d: + dst_h, _, n_h = h2d[i % len(h2d)] + off = rng.randint(0, n_h - 1) + src, n = dst_h + off, n_h - off + else: + n = rng.choice([4, 8, 16, 32, 6, 64]) + spos += rng.randint(0, 24) + src = spos + spos += n + dpos += rng.randint(0, 24) + d2d.append((dpos, src, n)) + dpos += n + assert dpos < ARENA and spos < 3 * ARENA // 4 + return h2d, d2d, data + + +def _window(layout, base: int) -> torch.Tensor: + """The window of ``layout`` for an arena at device address ``base``.""" + h2d, d2d, data = layout + w = torch.zeros(WINDOW, dtype=torch.uint8) + words = [len(h2d), len(d2d)] + for dst, src, n in h2d: + words += [base + dst, src, n] + for dst, src, n in d2d: + words += [base + dst, base + src, n] + w[: 8 * len(words)] = torch.tensor(words, dtype=torch.int64).view(torch.uint8) + for (_, off, n), b in zip(h2d, data): + w[off : off + n] = torch.frombuffer(bytearray(b), dtype=torch.uint8) + return w.cuda() + + +def _arenas(seed): + a = torch.randint(0, 256, (ARENA,), generator=gen(seed), device="cuda", dtype=torch.int32) + a = a.to(torch.uint8) + return a, a.clone() + + +@pytest.mark.parametrize("with_epoch", [True, False], ids=["epoch", "no_epoch"]) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, with_epoch, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Byte for byte the default prologue's destinations, header clear and epoch.""" + seed = seed_of(cell.why) + layout = _layout(cell.spec, seed) + arena, ref_arena = _arenas(seed) + window, ref_window = _window(layout, arena.data_ptr()), _window(layout, ref_arena.data_ptr()) + epoch = torch.full((1,), 41, dtype=torch.int64, device="cuda") if with_epoch else None + ref_epoch = epoch.clone() if with_epoch else None + certifying(op) + op(window, epoch, MAX_DESC) + op.reference(ref_window, ref_epoch, MAX_DESC) + torch.cuda.synchronize() + assert torch.equal(arena, ref_arena) + # Each window holds its own arena's addresses: past the consumed header, both are left as staged. + for w, a in ((window, arena), (ref_window, ref_arena)): + assert not w[:16].any(), "the header is consumed" + assert torch.equal(w[16:], _window(layout, a.data_ptr())[16:]) + if with_epoch: + assert int(epoch.item()) == int(ref_epoch.item()) == 42 + + +def test_graph_replays_scatter_each_new_window() -> None: + """Captured once on a static window: every replay writes that replay's window and bumps the epoch once.""" + arena, ref_arena = _arenas(7) + window = torch.zeros(WINDOW, dtype=torch.uint8, device="cuda") + epoch = torch.zeros(1, dtype=torch.int64, device="cuda") + ref_epoch = epoch.clone() + op(window, epoch, MAX_DESC) # the empty window: writes nothing + op.reference(window.clone(), ref_epoch, MAX_DESC) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(window, epoch, MAX_DESC) + for step, spec in enumerate( + [dict(h2d=24, d2d=8, chained=True), dict(h2d=0, d2d=0), dict(h2d=40, d2d=4)] + ): + layout = _layout(spec, 100 + step) + window.copy_(_window(layout, arena.data_ptr())) + graph.replay() + op.reference(_window(layout, ref_arena.data_ptr()), ref_epoch, MAX_DESC) + torch.cuda.synchronize() + assert torch.equal(arena, ref_arena), step + assert not window[:16].any(), step + assert int(epoch.item()) == int(ref_epoch.item()) == step + 2, step + + +def _pack_case(prologue_op): + pack = StepInputPack( + torch.device("cuda", torch.cuda.current_device()), + parallel_scatter=True, + in_graph=True, + early_copy=True, + step_epoch=True, + prologue_op=prologue_op, + ) + a, b, c = (torch.zeros(16, dtype=torch.int32, device="cuda") for _ in range(3)) + out = torch.zeros(48, dtype=torch.int32, device="cuda") + graph = torch.cuda.CUDAGraph() + with step_inputs.active(pack): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + assert step_inputs.capture_prologue() is pack + out.copy_(torch.cat([a, b, c])) + return pack, graph, (a, b, c), out + + +def _pack_step(pack, graph, dsts, values, overlap=False, flush=True): + a, b, c = dsts + + def host(v): + return torch.tensor(v, dtype=torch.int32).pin_memory() + + with step_inputs.active(pack): + step_inputs.begin() + step_inputs.copy_(a, host(values[0])) + step_inputs.copy_(b, host(values[1])) + if overlap: + step_inputs.copy_(a[:4], host([-1, -2, -3, -4])) + step_inputs.copy_(c, a) # a device copy reading a host copy's destination + if not flush: + step_inputs.discard() + step_inputs.flush(in_graph=True) + graph.replay() + + +def test_pack_with_the_one_launch_prologue_matches_the_default() -> None: + """The same pack steps through the default prologue and the one-launch one: the same outputs and epochs, and the + one-launch prologue is engaged (the default's warm-up compile is not what bumped its epoch).""" + packs = [_pack_case(None), _pack_case(op)] + assert packs[0][0].prologue_op is None and packs[1][0].prologue_op is op + for step in range(6): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + outs = [] + for pack, graph, dsts, out in packs: + # A replay with a new pack, one with an eagerly issued pack, and one with nothing to scatter. + _pack_step(pack, graph, dsts, (va, vb), overlap=step == 2, flush=step != 3) + torch.cuda.synchronize() + outs.append((out.tolist(), int(pack.epoch.item()))) + assert outs[0] == outs[1], step + assert outs[1][1] == step + 2, step # one bump at warm-up, then one per replay diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_ar_det.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_ar_det.py new file mode 100644 index 000000000000..e3e11ebd2025 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_ar_det.py @@ -0,0 +1,54 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""AR-DET: the target defaults the AllReduce autotune off for its process (the C++ heuristic), keeps an explicit +TLLM_DISABLE_ALLREDUCE_AUTOTUNE as given, and leaves the autotune on with TRTLLM_M31_ENABLE_AR_AUTOTUNE=1.""" + +import os + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) + +KEY = "TLLM_DISABLE_ALLREDUCE_AUTOTUNE" + + +def test_autotune_defaults_off(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", False) + assert modeling._pin_allreduce_heuristic(0) is True + assert os.environ[KEY] == "1" + + +def test_escape_hatch_keeps_autotune_on(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", True) + assert modeling._pin_allreduce_heuristic(0) is False + assert KEY not in os.environ + + +def test_explicit_setting_is_kept(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {KEY: "0"}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", False) + assert modeling._pin_allreduce_heuristic(0) is False + assert os.environ[KEY] == "0" + + +def test_escape_hatch_follows_the_switch_contract() -> None: + """TRTLLM_M31_ENABLE_AR_AUTOTUNE is parsed by m31_switches (1/true/on/yes set; ""/0/false/off/no unset), not by + a separate loose read.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + m31_switches, + ) + + assert modeling.AR_AUTOTUNE_ENABLED == m31_switches.m31_enabled("AR_AUTOTUNE") + + +def test_engagement_line_only_when_off(monkeypatch) -> None: + lines = [] + monkeypatch.setattr(modeling, "logger", type("L", (), {"info": staticmethod(lines.append)})) + for env, enabled, engaged in (({}, False, True), ({}, True, False), ({KEY: "0"}, False, False)): + lines.clear() + monkeypatch.setattr(os, "environ", dict(env)) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", enabled) + modeling._pin_allreduce_heuristic(3) + assert any("rank 3: AR_DET engaged" in line for line in lines) == engaged, (env, enabled) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_dspark_draft_ar.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_dspark_draft_ar.py new file mode 100644 index 000000000000..24052c98df95 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_dspark_draft_ar.py @@ -0,0 +1,119 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""S9b-VR-a: the DSpark drafter's row-parallel all-reduces take the one-shot all-reduce up to +``DRAFT_AR_ONESHOT_MAX_ROWS`` rows and keep their AUTO all-reduce above (``dspark.install_draft_oneshot_allreduce``). + +The all-reduce and linear classes are stand-ins (the routing is the unit under test; the one-shot kernel has its own +tests), so this runs on the CPU without a process group. +""" + +import importlib +import os +from types import SimpleNamespace + +import pytest +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + dspark, +) + +Strategy = dspark.AllReduceStrategy + + +class FakeAllReduce(nn.Module): + """Records each call; ``strategy`` and ``mapping`` as the real module exposes them.""" + + def __init__(self, mapping, strategy=Strategy.AUTO, dtype=None): + super().__init__() + self.mapping = mapping + self.strategy = strategy + self.dtype = dtype + self.calls = [] + + def uses_nccl_symmetric_memory_window(self): + return self.strategy == Strategy.AUTO + + def forward(self, input, *, all_reduce_params=None): + self.calls.append((input.shape[0], all_reduce_params)) + return input + 1 + + +class FakeLinear(nn.Module): + def __init__(self, tp_mode, all_reduce, dtype=torch.bfloat16): + super().__init__() + self.tp_mode = tp_mode + self.all_reduce = all_reduce + self.dtype = dtype + + +@pytest.fixture +def fakes(monkeypatch): + monkeypatch.setattr(dspark, "AllReduce", FakeAllReduce) + monkeypatch.setattr(dspark, "Linear", FakeLinear) + return SimpleNamespace(mapping=SimpleNamespace(tp_size=4, tp_rank=0)) + + +def _drafter(mapping): + ROW, COL = dspark.TensorParallelMode.ROW, dspark.TensorParallelMode.COLUMN + d = nn.Module() + d.o_proj = FakeLinear(ROW, FakeAllReduce(mapping)) + d.down_proj = FakeLinear(ROW, FakeAllReduce(mapping)) + d.qkv_proj = FakeLinear(COL, None) + d.nccl_row = FakeLinear( + ROW, FakeAllReduce(mapping, Strategy.NCCL) + ) # an explicit strategy is left alone + d.no_reduce = FakeLinear(ROW, None) + return d + + +def test_wraps_only_row_parallel_auto_linears(fakes): + d = _drafter(fakes.mapping) + assert dspark.install_draft_oneshot_allreduce(d, 32) == 2 + assert isinstance(d.o_proj.all_reduce, dspark.DraftOneshotAllReduce) + assert isinstance(d.down_proj.all_reduce, dspark.DraftOneshotAllReduce) + assert isinstance(d.nccl_row.all_reduce, FakeAllReduce) + assert d.qkv_proj.all_reduce is None and d.no_reduce.all_reduce is None + # the one-shot module is the explicit ONESHOT strategy, never TWOSHOT + assert d.o_proj.all_reduce.oneshot.strategy == Strategy.ONESHOT + + +@pytest.mark.parametrize( + "rows, takes_oneshot", [(1, True), (4, True), (32, True), (33, False), (4096, False)] +) +def test_routes_by_rows(fakes, rows, takes_oneshot): + d = _drafter(fakes.mapping) + dspark.install_draft_oneshot_allreduce(d, 32) + w = d.o_proj.all_reduce + params = object() + out = w(torch.zeros(rows, 8), all_reduce_params=params) + assert torch.equal(out, torch.ones(rows, 8)) + taken, other = (w.oneshot, w.auto) if takes_oneshot else (w.auto, w.oneshot) + assert taken.calls == [(rows, params)] and other.calls == [] + + +def test_output_buffer_choice_stays_the_original(fakes): + d = _drafter(fakes.mapping) + dspark.install_draft_oneshot_allreduce(d, 32) + assert ( + d.o_proj.all_reduce.uses_nccl_symmetric_memory_window() is True + ) # the AUTO module's answer + + +def test_zero_rows_installs_nothing(fakes): + d = _drafter(fakes.mapping) + assert dspark.install_draft_oneshot_allreduce(d, 0) == 0 + assert isinstance(d.o_proj.all_reduce, FakeAllReduce) + + +@pytest.mark.parametrize("value, expected", [("1", 0), ("0", 32), ("", 32)]) +def test_kill_switch(monkeypatch, value, expected): + monkeypatch.setenv("TRTLLM_M31_DISABLE_S9BVR", value) + try: + mod = importlib.reload(dspark) + assert mod.DRAFT_AR_ONESHOT_MAX_ROWS == expected + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_S9BVR", raising=False) + importlib.reload(dspark) + assert os.environ.get("TRTLLM_M31_DISABLE_S9BVR") is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_fc_shard.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_fc_shard.py new file mode 100644 index 000000000000..6d55ef887af1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_fc_shard.py @@ -0,0 +1,168 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""FC_SHARD on the sm_107 TP4 target: the drafter's fc as a column shard per rank + an all-gather. + +Each 1536-column shard computed by ``trtllm::minimax_m31_fc_shard_gemm`` with the pinned algo equals the matching +columns of ``torch.nn.functional.linear(x, w)``, for every shard, every row count up to MAX_ROWS, real-scale and +2^+-12 block-scaled inputs, eager and in a CUDA graph; the shards gathered in rank order are the full fc. The wrapper +takes a shard only for the fc it was built from and the checked row counts, and the DSpark drafter's +project_target_hidden takes it only within those rows. Default on; TRTLLM_M31_DISABLE_FC_SHARD=1 turns it off.""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm import ( # noqa: E501 + m31_fc_shard, +) + +K, N, TP = 30720, 6144, 4 +COLS = N // TP + + +def _op_or_skip(): + if not torch.cuda.is_available(): + pytest.skip("needs a GPU") + op = m31_fc_shard.fc_shard_op() + if op is None: + pytest.skip("the minimax_m31_fc_shard_gemm op is not built") + return op + + +@pytest.fixture(scope="module") +def weight(): + g = torch.Generator(device="cpu").manual_seed(2026) + return (torch.randn(N, K, generator=g) * 0.02).to(torch.bfloat16).cuda() + + +def _inputs(rows, stress): + g = torch.Generator(device="cpu").manual_seed(100 + rows) + x = torch.randn(rows, K, generator=g) + if stress: + e = torch.randint(-12, 13, (rows, K // 128), generator=g).float() + x = x * torch.pow(2.0, e).repeat_interleave(128, dim=1) + return x.to(torch.bfloat16).cuda() + + +@pytest.mark.parametrize("stress", [False, True]) +@pytest.mark.parametrize("rows", list(range(1, m31_fc_shard.MAX_ROWS + 1))) +def test_every_shard_is_the_full_fc_columns(weight, rows, stress): + op = _op_or_skip() + x = _inputs(rows, stress) + full = F.linear(x, weight) + cfg = list(m31_fc_shard.PINNED_CFG) + parts = [] + for r in range(TP): + got = op(x, weight[r * COLS : (r + 1) * COLS].contiguous(), cfg) + assert torch.equal(got, full[:, r * COLS : (r + 1) * COLS]), r + parts.append(got) + assert torch.equal(torch.cat(parts, dim=-1), full) + + +def test_the_shard_in_a_cuda_graph(weight): + op = _op_or_skip() + x = _inputs(8, True) + w3 = weight[3 * COLS : 4 * COLS].contiguous() + cfg = list(m31_fc_shard.PINNED_CFG) + out = torch.empty(8, COLS, dtype=torch.bfloat16, device="cuda") + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + out.copy_(op(x, w3, cfg)) # set the algo up outside the capture + torch.cuda.current_stream().wait_stream(s) + g = torch.cuda.CUDAGraph() + with torch.cuda.graph(g): + out.copy_(op(x, w3, cfg)) + out.zero_() + g.replay() + torch.cuda.synchronize() + assert torch.equal(out, F.linear(x, weight)[:, 3 * COLS : 4 * COLS]) + + +def test_the_wrapper_shards_checks_and_gathers(weight, monkeypatch): + _op_or_skip() + fc = torch.nn.Linear(K, N, bias=False, device="cuda", dtype=torch.bfloat16) + fc.weight.data.copy_(weight) + x = _inputs(7, False) + shards = [] + for r in range(TP): + shard, why = m31_fc_shard.FcShard._check(fc, TP, r, SimpleNamespace(tp_size=TP, tp_rank=r)) + assert why is None and shard is not None, why + shards.append(shard) + s0 = shards[0] + assert s0.applies(fc, 7) and s0.applies(fc, m31_fc_shard.MAX_ROWS) + assert not s0.applies(fc, m31_fc_shard.MAX_ROWS + 1) and not s0.applies(fc, 0) + other = torch.nn.Linear(K, N, bias=False, device="cuda", dtype=torch.bfloat16) + assert not s0.applies(other, 7) + parts = [s.partial(x) for s in shards] + assert all(p.shape == (7, COLS) for p in parts) + # The all-gather concatenates the ranks' tensors in rank order along dim. + monkeypatch.setattr( + m31_fc_shard, "allgather", lambda y, mapping, dim: torch.cat(parts, dim=dim) + ) + assert torch.equal(s0.gather(parts[0]), fc(x)) + + +def test_a_non_applicable_fc_keeps_the_stock_path(): + _op_or_skip() + fc = torch.nn.Linear(K, N, bias=True, device="cuda", dtype=torch.bfloat16) + shard, why = m31_fc_shard.FcShard._check(fc, TP, 0, SimpleNamespace(tp_size=TP, tp_rank=0)) + assert shard is None and "do not apply" in why + + +class _HalfShard: + """A stand-in shard: the fc as two column halves, concatenated (the all-gather of a 2-rank group).""" + + def __init__(self, fc, rows): + self.fc, self.rows, self.calls = fc, rows, 0 + + def applies(self, fc, rows): + return fc is self.fc and 0 < rows <= self.rows + + def __call__(self, x): + self.calls += 1 + h = self.fc.weight.shape[0] // 2 + return torch.cat([F.linear(x, self.fc.weight[:h]), F.linear(x, self.fc.weight[h:])], dim=-1) + + +@pytest.mark.parametrize("rows", [1, 8, 9, 24]) +def test_project_target_hidden_takes_the_shard_within_its_rows(rows): + from tensorrt_llm._torch.models.modeling_dflash import DFlashForCausalLM + + g = torch.Generator().manual_seed(11) + fc = torch.nn.Linear(96, 32, bias=False, dtype=torch.bfloat16) + norm = torch.nn.RMSNorm(32, eps=1e-6, dtype=torch.bfloat16) + with torch.no_grad(): + fc.weight.copy_(torch.randn(32, 96, generator=g).bfloat16()) + norm.weight.copy_((1 + 0.1 * torch.randn(32, generator=g)).bfloat16()) + shard = _HalfShard(fc, 8) + model = SimpleNamespace(fc=fc, hidden_norm=norm, _dflash_fc_split=None, _dflash_fc_shard=shard) + x = torch.randn(rows, 96, generator=g) + out = DFlashForCausalLM.project_target_hidden(model, x) + assert torch.equal(out, norm(fc(x.to(torch.bfloat16)))) + assert shard.calls == (1 if rows <= 8 else 0) + + +@pytest.mark.parametrize( + "env, expected", [({}, True), ({"TRTLLM_M31_DISABLE_FC_SHARD": "1"}, False)] +) +def test_fc_shard_default_on_and_kill_switch(monkeypatch, env, expected): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + dspark, + m31_switches, + ) + + monkeypatch.delenv("TRTLLM_M31_DISABLE_FC_SHARD", raising=False) + for name, value in env.items(): + monkeypatch.setenv(name, value) + importlib.reload(m31_switches) + try: + assert importlib.reload(dspark).FC_SHARD is expected + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_FC_SHARD", raising=False) + importlib.reload(m31_switches) + importlib.reload(dspark) + assert dspark.FC_SHARD is True diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_host_tail_sm107_tp4.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_host_tail_sm107_tp4.py new file mode 100644 index 000000000000..6b2c0d2958c9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_host_tail_sm107_tp4.py @@ -0,0 +1,187 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The kill switches of this target's two host-tail items ported from the sm_103 HOSTTAIL bundle: ``_runtime_hooks`` +turns on RuntimeHooks.skip_accepted_draft_upload (``SKIP_ACCEPTED_UPLOAD``) and +RuntimeHooks.spec_batch_map_changed_only (``BATCH_CHANGED_ONLY``) by default, and +``TRTLLM_M31_DISABLE_`` turns off only its own item (this target has no bundle switch). + +The ``STEP_*`` expressions are evaluated from modeling.py's source against a private copy of m31_switches loaded +under each environment (as test_modeling_v2_m31_switches_sm107_tp4 does), so neither the in-tree switches nor the +heavy modeling module is reloaded. CPU only, except the last test, which imports the in-tree module. +""" + +import ast +import importlib +import importlib.util +import os +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.cache_manager import ( + MiniMaxM3KVCacheManagerV2, +) +from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import ( + BaseMambaCacheManager, + MambaHybridCacheManager, +) +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + +TARGET = "minimax_m31_nvfp4__sm_107__tp4" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + +# Kill-switch id -> (the module constant, the RuntimeHooks field it sets). +ITEMS = { + "SKIP_ACCEPTED_UPLOAD": ("STEP_SKIP_ACCEPTED_UPLOAD", "skip_accepted_draft_upload"), + "BATCH_CHANGED_ONLY": ("STEP_BATCH_CHANGED_ONLY", "spec_batch_map_changed_only"), +} + + +def _module_tree() -> ast.Module: + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _item_statements(tree: ast.Module) -> list[ast.stmt]: + """modeling.py's module-level assignments of the ITEMS constants, in source order.""" + constants = {constant for constant, _ in ITEMS.values()} + return [ + node + for node in tree.body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id in constants + ] + + +def _switches(monkeypatch, env): + """A private copy of m31_switches imported under ``env`` (every other TRTLLM_M31_* variable cleared).""" + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_host_tail_{id(env)}", _SWITCHES) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _values(monkeypatch, env) -> dict[str, bool]: + """The ITEMS constants modeling.py computes under ``env``, by item id.""" + switches = _switches(monkeypatch, env) + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec( + compile(ast.Module(_item_statements(_module_tree()), []), str(_MODELING), "exec"), + namespace, + ) + return {item: namespace[constant] for item, (constant, _) in ITEMS.items()} + + +@pytest.mark.cpu_only +def test_the_switches_are_the_two_items(): + names = [node.targets[0].id for node in _item_statements(_module_tree())] + + assert names == [constant for constant, _ in ITEMS.values()] + for _, field in ITEMS.values(): + assert getattr(RuntimeHooks(), field) is False, field + + +@pytest.mark.cpu_only +def test_every_item_is_on_by_default(monkeypatch): + assert _values(monkeypatch, {}) == {item: True for item in ITEMS} + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", sorted(ITEMS)) +@pytest.mark.parametrize("value", ["1", "true", "ON", " yes "]) +def test_each_kill_switch_turns_off_only_its_own_item(monkeypatch, item, value): + values = _values(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: value}) + + assert values == {other: other != item for other in ITEMS} + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", sorted(ITEMS)) +@pytest.mark.parametrize("value", ["", "0", "false", "off", "no"]) +def test_an_unset_kill_switch_keeps_the_item(monkeypatch, item, value): + assert _values(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: value}) == { + other: True for other in ITEMS + } + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", sorted(ITEMS)) +def test_a_typo_fails_at_import_naming_the_variable(monkeypatch, item): + with pytest.raises(ValueError, match="TRTLLM_M31_DISABLE_" + item): + _values(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: "2"}) + + +@pytest.mark.cpu_only +def test_an_enable_variable_does_not_turn_an_item_off_or_on(monkeypatch): + env = {"TRTLLM_M31_ENABLE_" + item: "1" for item in ITEMS} + assert _values(monkeypatch, env) == {item: True for item in ITEMS} + + +@pytest.mark.cpu_only +def test_no_bundle_switch_on_this_target(monkeypatch): + """HOSTTAIL is the sm_103 targets' bundle switch; this target has none, so it changes nothing here.""" + assert _values(monkeypatch, {"TRTLLM_M31_DISABLE_HOSTTAIL": "1"}) == { + item: True for item in ITEMS + } + + +@pytest.mark.cpu_only +def test_runtime_hooks_wires_each_field_to_its_switch(): + """``_runtime_hooks`` passes each constant to its RuntimeHooks field, and nothing else sets those.""" + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + calls = [ + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1 + keywords = {kw.arg: kw.value for kw in calls[0].keywords} + for constant, field in ITEMS.values(): + assert isinstance(keywords.get(field), ast.Name), field + assert keywords[field].id == constant, field + + +@pytest.mark.cpu_only +def test_the_target_kv_cache_is_not_a_mamba_cache(): + """skip_accepted_draft_upload is on only together with a non-mamba cache: the mamba state update is the one + consumer of accepted counts on the drafter path (it takes them as an argument, not from the spec metadata).""" + assert not issubclass(MiniMaxM3KVCacheManagerV2, BaseMambaCacheManager) + assert not issubclass(MiniMaxM3KVCacheManagerV2, MambaHybridCacheManager) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="imports the target's modeling module") +def test_the_in_tree_hooks_follow_the_switches(): + modeling = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.modeling" + ) + hooks = modeling._runtime_hooks(None) + + for constant, field in ITEMS.values(): + assert getattr(hooks, field) is getattr(modeling, constant), field + + def disabled(item): + return os.environ.get("TRTLLM_M31_DISABLE_" + item, "").strip().lower() in ( + "1", + "true", + "on", + "yes", + ) + + for item, (_, field) in ITEMS.items(): + assert getattr(hooks, field) is not disabled(item), item diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_mxdraft_tp4_scope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_mxdraft_tp4_scope.py new file mode 100644 index 000000000000..9a565e04471c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_mxdraft_tp4_scope.py @@ -0,0 +1,34 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR TP4 MXDRAFT scope: default off on this target; TRTLLM_M31_ENABLE_MXDRAFT=1 opts in and +TRTLLM_M31_DISABLE_MXDRAFT=1 still wins (both read at import, so each case runs in a fresh process).""" + +import os +import subprocess +import sys + +import pytest + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4" +) + + +@pytest.mark.parametrize( + "enable, disable, on", [("", "", False), ("1", "", True), ("1", "1", False), ("", "1", False)] +) +def test_mxdraft_tp4_default_off_and_switches(enable: str, disable: str, on: bool) -> None: + env = { + k: v + for k, v in os.environ.items() + if not k.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")) + } + env.update(TRTLLM_M31_ENABLE_MXDRAFT=enable, TRTLLM_M31_DISABLE_MXDRAFT=disable) + out = subprocess.run( + [sys.executable, "-c", f"from {TARGET} import dspark\nprint(dspark.MXDRAFT)"], + env=env, + capture_output=True, + text=True, + check=True, + ) + assert out.stdout.split()[-1] == str(on) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_onediv_cap.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_onediv_cap.py new file mode 100644 index 000000000000..30c96cbb3f18 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_onediv_cap.py @@ -0,0 +1,54 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR TP4 ONE_DIV row cap: the LC merge divides once up to 16 rows; TRTLLM_M31_DISABLE_ONEDIV_CAP=1 keeps it on at +every row count, TRTLLM_M31_DISABLE_LC_ONEDIV=1 turns it off.""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + m31_switches, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_attn_lc, +) + + +def test_onediv_cap_default() -> None: + assert m3_attn_lc.ONE_DIV_MAX_ROWS == 16 + assert m3_attn_lc.one_div_applies(4) and m3_attn_lc.one_div_applies(16) + assert not m3_attn_lc.one_div_applies(17) and not m3_attn_lc.one_div_applies(48) + + +@pytest.mark.parametrize("disable, applies_at_32", [("", False), ("0", False), ("1", True)]) +def test_onediv_cap_switch( + monkeypatch: pytest.MonkeyPatch, disable: str, applies_at_32: bool +) -> None: + monkeypatch.setenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", disable) + try: + importlib.reload(m31_switches) + lc = importlib.reload(m3_attn_lc) + assert lc.one_div_applies(1) and lc.one_div_applies(16) + assert lc.one_div_applies(32) is applies_at_32 + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", raising=False) + importlib.reload(m31_switches) + importlib.reload(m3_attn_lc) + + +@pytest.mark.parametrize("cap", ["", "1"]) +def test_onediv_kill_switch(monkeypatch: pytest.MonkeyPatch, cap: str) -> None: + """TRTLLM_M31_DISABLE_LC_ONEDIV=1 turns the one-division merge off at every row count, whatever the cap switch.""" + monkeypatch.setenv("TRTLLM_M31_DISABLE_LC_ONEDIV", "1") + monkeypatch.setenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", cap) + try: + importlib.reload(m31_switches) + lc = importlib.reload(m3_attn_lc) + assert lc.ONE_DIV_MAX_ROWS == 0 + assert not any(lc.one_div_applies(rows) for rows in (1, 8, 16, 32)) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_LC_ONEDIV", raising=False) + monkeypatch.delenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", raising=False) + importlib.reload(m31_switches) + importlib.reload(m3_attn_lc) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_prefill_ar_mnnvl.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_prefill_ar_mnnvl.py new file mode 100644 index 000000000000..0dddaa6e1745 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_prefill_ar_mnnvl.py @@ -0,0 +1,64 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""PREFILL-MNNVL: the sm_107 TP4 prefill / mixed-step boundaries' built-in all-reduce is MNNVL up to 8192 rows; larger +steps and every generation step keep the target's AUTO all-reduce. TRTLLM_M31_DISABLE_PREFILL_AR_MNNVL=1 restores +AUTO everywhere.""" + +import importlib +import inspect + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + m31_switches, + modeling, +) + +AUTO, MNNVL = object(), object() + + +class _Core: + def __init__(self, mnnvl) -> None: + self.allreduce = AUTO + self.allreduce_mnnvl = mnnvl + + +@pytest.fixture +def prefill_mnnvl_off(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("TRTLLM_M31_DISABLE_PREFILL_AR_MNNVL", "1") + importlib.reload(m31_switches) + try: + yield importlib.reload(modeling) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_PREFILL_AR_MNNVL", raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) + + +def test_default_on_up_to_8192_rows() -> None: + assert modeling.PREFILL_AR_MNNVL and modeling.PREFILL_AR_MNNVL_MAX_ROWS == 8192 + + +@pytest.mark.parametrize("rows", [1, 33, 64, 128, 1024, 8192, 8193, 16384]) +def test_rows_pick_the_all_reduce(rows: int) -> None: + want = MNNVL if rows <= 8192 else AUTO + assert modeling._prefill_allreduce(_Core(MNNVL), rows) is want + assert modeling._prefill_allreduce(_Core(None), rows) is AUTO + + +def test_only_the_prefill_target_takes_it() -> None: + """Every built-in boundary all-reduce of the prefill target goes through ``_prefill_allreduce``; the decode target + (every generation step, the 33..64-row ones included) never does.""" + prefill = inspect.getsource(modeling.PrefillTarget) + # The two CTXNORMQ boundaries and the three native fused ones each pass ``_prefill_allreduce``. + assert prefill.count("_prefill_allreduce(core, ") == 5 + assert prefill.count("_ctxnormq_allreduce_norm(") == 2 + assert prefill.count("_native_allreduce_norm(") == 3 + assert "_prefill_allreduce" not in inspect.getsource(modeling.DecodeTarget) + + +def test_kill_switch(prefill_mnnvl_off) -> None: + m = prefill_mnnvl_off + assert not m.PREFILL_AR_MNNVL + assert "AllReduceStrategy.MNNVL" in inspect.getsource(m.MiniMaxM3Tp4ModelingV2.__init__) + assert m._prefill_allreduce(_Core(None), 128) is AUTO diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_proxy_prebuild.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_proxy_prebuild.py new file mode 100644 index 000000000000..b32e62a810f3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_proxy_prebuild.py @@ -0,0 +1,57 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""PROXYPB on the sm_107 TP4 target: every fmha_sm100 variant the MSA indexer's proxy pass can pick (OnlyScore over +the FP8 paged index-K, page 128, no split-KV) is built once before warmup. Four index heads per rank pack rows of up to +32 tokens 4x (packed <= 64 or 65..128); longer rows run unpacked (<= 64, 65..128, > 128). + +The variant builds are recorded rather than compiled (an nvcc build each); the core is a stand-in with the attributes +the prebuild reads. +""" + +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + require_msa_module, +) + +INDEX_HEADS_PER_RANK = 4 +WANT = {"1_0_0_2_1_0_2", "1_0_1_2_1_0_2", "1_0_0_2_1_0_0", "1_0_1_2_1_0_0", "1_1_1_2_1_0_0"} + + +@pytest.fixture +def proxy_builds(monkeypatch): + require_msa_module() # its cutlass compatibility shim must precede the fmha_sm100 import, as in the target + from fmha_sm100 import jit as fmha_jit + + built = [] + + def record(*key): + built.append(fmha_jit._variant_key_from_runtime(*key)[0]) + return SimpleNamespace(run=None) + + monkeypatch.setattr(fmha_jit, "get_fmha_variant", record) + return built + + +def _core(): + attn = SimpleNamespace( + indexer_kv_dtype="fp8", m3_config=SimpleNamespace(num_index_heads=INDEX_HEADS_PER_RANK) + ) + return SimpleNamespace(sparse_layers=frozenset({3, 4, 5}), attn={3: attn, 4: attn, 5: attn}) + + +def test_builds_every_proxy_variant_once(proxy_builds) -> None: + names = modeling.MiniMaxM3Tp4ModelingV2._prebuild_msa_proxy(_core()) + assert set(names) == WANT + assert sorted(proxy_builds) == sorted(WANT), ( + "each variant is built once, across every sparse layer" + ) + + +def test_prebuild_is_on_by_default() -> None: + assert modeling.MSA_PROXY_PREBUILD is True diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_r6_dense_prequant.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_r6_dense_prequant.py new file mode 100644 index 000000000000..54525aa72bf2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_r6_dense_prequant.py @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""R6 on the sm_107 TP4 target: the dense-MLP layer boundary hands the next layer's QKV its MXFP8 activation by +VR: OFF by default (not bitwise on sm_107; no passing gate c): TRTLLM_M31_ENABLE_R6=1 +turns it on, TRTLLM_M31_DISABLE_R6=1 forces it off. The engagement itself ("[M3 dispatch] r6_dense_prequant taken" / +"off") is checked in-model.""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) + + +def test_r6_is_off_by_default_on_vr() -> None: + assert modeling.R6_DENSE_PREQUANT is False + + +@pytest.mark.parametrize( + ("enable", "disable", "expected"), + [("1", "", True), ("1", "1", False), ("0", "", False), ("", "1", False)], +) +def test_r6_switches( + monkeypatch: pytest.MonkeyPatch, enable: str, disable: str, expected: bool +) -> None: + monkeypatch.setenv("TRTLLM_M31_ENABLE_R6", enable) + monkeypatch.setenv("TRTLLM_M31_DISABLE_R6", disable) + try: + assert importlib.reload(modeling).R6_DENSE_PREQUANT is expected + finally: + monkeypatch.delenv("TRTLLM_M31_ENABLE_R6", raising=False) + monkeypatch.delenv("TRTLLM_M31_DISABLE_R6", raising=False) + importlib.reload(modeling) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_s10_scope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_s10_scope.py new file mode 100644 index 000000000000..e2185669c3f1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_s10_scope.py @@ -0,0 +1,32 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR S10 scope (tp4): the lean drafter glue only up to the measured draft length (K3).""" + +import subprocess +import sys + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + dspark, +) + + +@pytest.mark.parametrize("k, on", [(1, True), (3, True), (4, False), (7, False), (15, False)]) +def test_s10_applies_up_to_k3(k: int, on: bool) -> None: + assert dspark.S10_MAX_DRAFT_LEN == 3 + assert dspark.s10_applies(k) is on + + +def test_s10_k7_gate_opt_in() -> None: + """TRTLLM_M31_ENABLE_S10_K7=1 (gate-only, read at import) lifts the bound to every draft length (default K3).""" + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_107__tp4 import dspark\n" + "print(dspark.S10_MAX_DRAFT_LEN, dspark.s10_applies(7))" + ) + env = {**__import__("os").environ, "TRTLLM_M31_ENABLE_S10_K7": "1"} + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.split()[-2:] == ["63", "True"] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_s5_scope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_s5_scope.py new file mode 100644 index 000000000000..ceca4c75492e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_s5_scope.py @@ -0,0 +1,15 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR S5 scope (tp4): the window-capped drafter decode only up to the measured batch per draft length.""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + dspark, +) + + +@pytest.mark.parametrize("k, expected", [(1, 4), (3, 4), (5, 4), (7, 4), (15, 4)]) +def test_s5_max_batch_by_draft_length(k: int, expected: int) -> None: + assert dspark.S5_MAX_BATCH_BY_K == {3: 4, 7: 4} + assert dspark.s5_max_batch_for(k) == expected diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_switches_sm107_tp4.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_switches_sm107_tp4.py new file mode 100644 index 000000000000..c68a4742e1d0 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_switches_sm107_tp4.py @@ -0,0 +1,83 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The value contract of this target's kill switches (m31_switches): 1 / true / on / yes set TRTLLM_M31_DISABLE_ / +TRTLLM_M31_ENABLE_, empty / 0 / false / off / no leave it unset (case-insensitive), anything else raises at import +naming the variable; the environment is read once, at import. + +Each case loads a private copy of the module from its file, so the in-tree module (read by every other module of the +target at import) is never reloaded. CPU only. +""" + +import importlib.util +import os +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp4" +_SWITCHES = Path(_family.__file__).resolve().parent / TARGET / "m31_switches.py" + +pytestmark = pytest.mark.cpu_only + +KINDS = [("TRTLLM_M31_DISABLE_", "m31_disabled"), ("TRTLLM_M31_ENABLE_", "m31_enabled")] +ITEM = "UTCONTRACT" # an id no item uses, so the real switches of the test process cannot interfere + + +def _load(monkeypatch, env): + """A private copy of m31_switches, imported under ``env`` (every other TRTLLM_M31_* variable cleared).""" + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_contract_{id(env)}", _SWITCHES) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@pytest.mark.parametrize("prefix,query", KINDS) +@pytest.mark.parametrize( + "value", ["", " ", "0", " 0 ", "false", "FALSE", "off", " Off ", "no", "No"] +) +def test_unset_spellings_leave_the_switch_unset(monkeypatch, prefix, query, value): + assert not getattr(_load(monkeypatch, {}), query)(ITEM) + assert not getattr(_load(monkeypatch, {prefix + ITEM: value}), query)(ITEM) + + +@pytest.mark.parametrize("prefix,query", KINDS) +@pytest.mark.parametrize("value", ["1", " 1 ", "true", "TRUE", "on", "On", "yes", "YES"]) +def test_set_spellings_set_the_switch(monkeypatch, prefix, query, value): + assert getattr(_load(monkeypatch, {prefix + ITEM: value}), query)(ITEM) + + +@pytest.mark.parametrize("prefix", [k for k, _ in KINDS]) +@pytest.mark.parametrize("value", ["2", "-1", "1.0", "y", "n", "enable", "disabled", "truee", "of"]) +def test_any_other_value_fails_at_import_naming_the_variable(monkeypatch, prefix, value): + """Typos must fail loudly: a loose parse would count any value but "" / "0" as set (DISABLE_X=false disabling X).""" + with pytest.raises(ValueError, match=prefix + ITEM): + _load(monkeypatch, {prefix + ITEM: value}) + + +def test_the_id_is_matched_exactly(monkeypatch): + switches = _load(monkeypatch, {"TRTLLM_M31_DISABLE_" + ITEM: "1"}) + assert switches.m31_disabled(ITEM) + for other in (ITEM.lower(), ITEM + "_X", ITEM[:-1], ""): + assert not switches.m31_disabled(other), other + + +def test_disable_and_enable_are_independent(monkeypatch): + """Both read: an item's own code applies "DISABLE wins" (e.g. B1, L-DS1); the module reports each separately.""" + switches = _load( + monkeypatch, {"TRTLLM_M31_DISABLE_" + ITEM: "1", "TRTLLM_M31_ENABLE_" + ITEM: "1"} + ) + assert switches.m31_disabled(ITEM) and switches.m31_enabled(ITEM) + + +def test_the_environment_is_read_once_at_import(monkeypatch): + switches = _load(monkeypatch, {}) + monkeypatch.setenv("TRTLLM_M31_DISABLE_" + ITEM, "1") + monkeypatch.setenv("TRTLLM_M31_ENABLE_" + ITEM, "1") + assert not switches.m31_disabled(ITEM) and not switches.m31_enabled(ITEM) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_tp4_fused_projection_default.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_tp4_fused_projection_default.py new file mode 100644 index 000000000000..26b56be45b54 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_tp4_fused_projection_default.py @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The M3.1 sm_107 TP4 target packs the sparse layers' QKV + index projection by default (a measured +win); an explicit ``fuse_qkv_index_projection`` in the deployment's config wins, so False is the kill switch.""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + weights as tp_weights, +) +from tensorrt_llm.llmapi.llm_args import MiniMaxM3SparseAttentionConfig + +_MSA = dict(implementation="msa", indexer_kv_dtype="fp8") + + +@pytest.mark.parametrize( + ("explicit", "expected"), + [ + ({}, True), + ({"fuse_qkv_index_projection": True}, True), + ({"fuse_qkv_index_projection": False}, False), + ], +) +def test_fused_projection_default_and_kill(explicit: dict, expected: bool) -> None: + config = MiniMaxM3SparseAttentionConfig(**_MSA, **explicit) + assert tp_weights.fused_projection(config) is expected + + +def test_no_sparse_config_is_unfused() -> None: + assert tp_weights.fused_projection(None) is False + + +@pytest.mark.parametrize("value", [True, False]) +def test_a_config_without_set_fields_is_taken_at_its_word(value: bool) -> None: + from types import SimpleNamespace + + assert tp_weights.fused_projection(SimpleNamespace(fuse_qkv_index_projection=value)) is value diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_tp4_weights_layout.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_tp4_weights_layout.py new file mode 100644 index 000000000000..a2fc876acad1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_tp4_weights_layout.py @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP4 weight table: layer kinds from the config, packed-NVFP4 expert scalars as FP32 divisors.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( + weights as W, +) + +pytestmark = pytest.mark.cpu_only + + +def _core(sparse_freq, moe_freq, fused=True): + cfg = SimpleNamespace( + num_hidden_layers=len(sparse_freq), + moe_layer_freq=moe_freq, + sparse_attention_config=dict( + sparse_attention_freq=sparse_freq, sparse_num_index_heads=4, sparse_index_dim=128 + ), + head_dim=128, + num_attention_heads=64, + num_key_value_heads=4, + hidden_size=6144, + dense_intermediate_size=12288, + shared_intermediate_size=3072, + n_shared_experts=1, + intermediate_size=3072, + num_local_experts=128, + vocab_size=200064, + torch_dtype=torch.bfloat16, + ) + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=cfg, + mapping=SimpleNamespace(tp_size=4, tp_rank=0), + sparse_attention_config=SimpleNamespace(fuse_qkv_index_projection=fused), + ) + ) + + +def test_layer_kinds_come_from_the_config_independently(): + """Sparse attention on every layer while layers 0-1 keep the dense MLP.""" + d = W.MODEL_WEIGHTS.dims(_core([1] * 6, [0, 0, 1, 1, 1, 1])) + assert list(W._sparse_attn_layers(d)) == [0, 1, 2, 3, 4, 5] + assert list(W._dense_mlp_layers(d)) == [0, 1] + assert list(W._moe_layers(d)) == [2, 3, 4, 5] + assert list(W._five_way_qkv_layers(d)) == [0, 1, 2, 3, 4, 5] + assert list(W._three_way_qkv_layers(d)) == [] + + +def test_unfused_projection_uses_the_three_way_qkv_everywhere(): + d = W.MODEL_WEIGHTS.dims(_core([0, 0, 1, 1], [0, 0, 1, 1], fused=False)) + assert list(W._three_way_qkv_layers(d)) == [0, 1, 2, 3] + assert list(W._unfused_sparse_layers(d)) == [2, 3] + assert list(W._five_way_qkv_layers(d)) == [] + + +def test_expert_scalars_are_the_fp32_reciprocal_of_the_stored_multiplier(): + stored = torch.tensor(12345.678, dtype=torch.float32) + out = W._divisor(stored, None) + assert out.dtype == torch.float32 + assert out.view(torch.int32).item() == torch.reciprocal(stored).view(torch.int32).item() + with pytest.raises(ValueError): + W._divisor(torch.tensor(0.0), None) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_vr_bands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_vr_bands.py new file mode 100644 index 000000000000..e9d431cc29c7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m31_vr_bands.py @@ -0,0 +1,109 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""VR-BANDS: the sm_107 TP4 decode step bound is 64 rows while every op keeps its own row range. + +Steps of 49..64 rows (K7 c7..c8) take the "wide" plan (the built-in all-reduce at both boundaries); each op arms +exactly one path at every row count 1..64 (its fast path inside its range, its generic form past it), and no op's +range moves. TRTLLM_M31_DISABLE_VR_BANDS=1 restores the 48-row step bound. +""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + m31_switches, + modeling, +) + +OP_BOUND = 48 +STEP_ROWS = range(1, 64 + 1) +ROW_TABLES = ("MX_GEMV_ROWS", "GEMV48_ROWS", "LINEAR_CUBLASLT_ROWS") + + +def _ranges(m) -> dict: + return {r.path: r.keys for r in m.FAST_PATHS} + + +def _projections(m) -> set: + tables = ROW_TABLES + ("MX_GEMV_BF16_MAX_TOKENS",) + return {nk for name in tables for nk in getattr(m, name, {})} + + +def _linear_paths(m) -> dict: + return { + nk: [ + m.linear_path(*nk, rows, prequant, False) + for rows in STEP_ROWS + for prequant in (True, False) + ] + for nk in _projections(m) + } + + +# The default (VR-BANDS on) module's view, recorded at collection before any reload. +_ON_RANGES = _ranges(modeling) +_ON_LINEAR_PATHS = _linear_paths(modeling) + + +@pytest.fixture +def vr_bands_off(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("TRTLLM_M31_DISABLE_VR_BANDS", "1") + importlib.reload(m31_switches) + try: + yield importlib.reload(modeling) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_VR_BANDS", raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) + + +def test_the_step_bound_is_64_and_the_op_bound_stays() -> None: + m = modeling + assert m.VR_BANDS and m.DECODE_STEP_MAX == 64 and m.DECODE_MAX_TOKENS == OP_BOUND + assert [m.is_decode_step(n) for n in range(1, 66)] == [True] * 64 + [False] + + +@pytest.mark.parametrize("rows", STEP_ROWS) +def test_every_step_row_count_takes_exactly_one_plan(rows: int) -> None: + m = modeling + plan = m.plan_for(rows) + assert plan is [p for p in m.DECODE_PLANS if rows <= p.max_tokens][0] + if rows > OP_BOUND: + assert plan.name == "wide" and plan.max_tokens == 64 + assert (plan.attn_mc_ar, plan.moe_mc_ar, plan.qkv_prequant) == (False, False, False) + else: + assert plan.name != "wide" + + +@pytest.mark.parametrize("rows", STEP_ROWS) +def test_every_op_arms_exactly_one_path(rows: int) -> None: + """Each projection's GEMM row tables never overlap and it selects one GEMM per activation form; at most one MoE + path (the engine, the 9..13-row engine where built, the unfinalized MoE) covers the row count, and the MoE takes it + there and the generic MoE elsewhere.""" + m = modeling + for nk in _projections(m): + armed = [name for name in ROW_TABLES if rows in getattr(m, name, {}).get(nk, ())] + assert len(armed) <= 1, (nk, rows, armed) + for prequant in (True, False): + assert isinstance(m.linear_path(*nk, rows, prequant, False), str) + moe_ranges = [ + getattr(m, n) for n in ("MOE_ENGINE", "MOE16_ENGINE", "MC_FINALIZE") if hasattr(m, n) + ] + covering = [r.path for r in moe_ranges if r.covers(rows)] + assert len(covering) <= 1, (rows, covering) + assert m.moe_path(rows) == (covering[0] if covering else None) + + +def test_no_op_range_moves(vr_bands_off) -> None: + """The bound is the step's alone: every fast path's range and every projection's GEMM per row count is the same + with VR-BANDS off.""" + assert _ranges(vr_bands_off) == _ON_RANGES + assert _linear_paths(vr_bands_off) == _ON_LINEAR_PATHS + + +def test_kill_switch(vr_bands_off) -> None: + m = vr_bands_off + assert not m.VR_BANDS and m.DECODE_STEP_MAX == m.DECODE_MAX_TOKENS == OP_BOUND + assert "wide" not in [p.name for p in m.DECODE_PLANS] + assert m.is_decode_step(OP_BOUND) and not m.is_decode_step(OP_BOUND + 1) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_capture_boundary.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_capture_boundary.py new file mode 100644 index 000000000000..f591ee201425 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_capture_boundary.py @@ -0,0 +1,257 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The feed-forward exit on Eagle3 capture layers: the generic layer in both targets' bodies (``PrefillTarget._layer``, +``DecodeTarget._generic_layer``), and the decode fast path's capture layers (``CAPTURE_LAYERS_FAST``). + +Every layer's MoE / MLP output goes unreduced to the fused boundary (all-reduce + residual + the next layer's input +norm, the final norm after the last layer), and a capture layer captures the reduced residual sum that boundary +writes, with no residual of its own. CPU only: the attention, the feed-forward and the boundaries are stand-ins. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) + +LAYERS = 4 + + +class _Rows(tuple): + """A stand-in activation: compares as its tuple, and has the rows (``shape[0]``) the prefill boundaries read to + pick their all-reduce (PREFILL-MNNVL).""" + + shape = (2,) + + +class _Spec: + def __init__(self, layers): + self.layers = set(layers) + + def is_layer_capture(self, i): + return i in self.layers + + +@pytest.fixture(params=["prefill", "decode"]) +def layer(request, monkeypatch): + """(core, the target's generic layer bound to a target over ``core``).""" + calls = [] + w = {f"l{i}_{n}": f"{n}{i}" for i in range(LAYERS) for n in ("in_norm", "post_norm")} + w["final_norm"] = "final" + c = SimpleNamespace( + w=w, + num_layers=LAYERS, + moe_layers=frozenset({2, 3}), + rms_eps=1e-6, + calls=calls, + # The prefill boundaries pick their built-in all-reduce from these (PREFILL-MNNVL); the stand-ins ignore it. + allreduce=None, + allreduce_mnnvl=None, + # The prefill attention boundary passes a MoE layer's NVFP4 input scale to CTXNORMQ. + moe={i: SimpleNamespace(fc31_input_scale=None) for i in (2, 3)}, + ) + monkeypatch.setattr(modeling, "_can_mc", lambda x, folded=False: False) + # The prefill body tries the CTXNORMQ boundary first; it declines here (no op on these stand-ins). + monkeypatch.setattr(modeling, "_ctxnormq_allreduce_norm", lambda *args, **kwargs: None) + monkeypatch.setattr( + modeling, + "_native_allreduce_norm", + lambda core, x, r, norm, allreduce=None: (("normed", x, norm), ("updated", x, r)), + ) + + def boundary(self, x, x2, residual, norm_w): + calls.append((x, x2, norm_w)) + return ("next", norm_w), ("sum", x, x2) + + if request.param == "prefill": + cls = modeling.PrefillTarget + monkeypatch.setattr(cls, "_moe", lambda self, i, h, **k: (("moe", i), ("routed", i))) + monkeypatch.setattr( + cls, "_swiglu_mlp", lambda self, h, prefix, low_m, dense=False: ("mlp", prefix) + ) + else: + cls = modeling.DecodeTarget + monkeypatch.setattr( + cls, "_moe", lambda self, i, h, **k: (("moe", i), ("routed", i), False, False) + ) + monkeypatch.setattr( + cls, + "_swiglu_mlp", + lambda self, h, prefix, low_m, push=None, dense=False: (("mlp", prefix), False), + ) + monkeypatch.setattr(cls, "_attention", lambda self, i, *a, **k: (_Rows(("attn", i)), False)) + monkeypatch.setattr(cls, "_generic_boundary", boundary) + target = cls(c) + return c, target._layer if request.param == "prefill" else target._generic_layer + + +@pytest.mark.parametrize("i", range(LAYERS)) +def test_a_capture_layer_captures_the_boundary_residual_sum(layer, i) -> None: + core, run = layer + captured = [] + out = run( + i, + None, + torch.ones(2), + torch.zeros(2), + None, + _Spec([i]), + lambda spec, layer, h, r: captured.append((layer, h, r)), + ) + norm = core.w[f"l{i + 1}_in_norm"] if i + 1 < LAYERS else "final" + assert core.calls[-1][2] == norm, ( + "the boundary applies the next layer's norm (the final norm after the last)" + ) + assert captured == [(i, out[1], None)] + if i in core.moe_layers: + assert core.calls[-1][1] == ("routed", i), "the routed part folds into the boundary" + + +def test_a_layer_without_capture_captures_nothing(layer) -> None: + _, run = layer + captured = [] + run( + 1, + None, + torch.ones(2), + torch.zeros(2), + None, + _Spec([]), + lambda *a: captured.append(a), + ) + assert not captured + + +class _FastCore: + num_layers = 3 + w = {"embed": None} + + +@pytest.mark.parametrize("written", [True, False]) +def test_a_decode_capture_layer_takes_the_fast_layer_with_the_capture_norm( + monkeypatch, written +) -> None: + """A decode step's capture layer (past the first) runs the fast decode layer with the worker's capture target, and + captures the boundary's updated residual; ``written=True`` when the boundary all-reduce wrote the capture norm + (``CAPTURE_NORM_IN_BOUNDARY``).""" + assert modeling.CAPTURE_LAYERS_FAST and modeling.CAPTURE_FUSED_BOUNDARY + assert modeling.CAPTURE_NORM_IN_BOUNDARY + target = modeling.DecodeTarget(_FastCore()) + seen, captured = [], [] + monkeypatch.setattr(target, "_check_step_contract", lambda md: None) + monkeypatch.setattr(target, "step_args", lambda md: {"num_tokens": 4}) + monkeypatch.setattr(target, "_embed", lambda ids, n: ("h", "r")) + monkeypatch.setattr(modeling, "plan_for", lambda n: SimpleNamespace(name="plan")) + monkeypatch.setattr(modeling, "_record", lambda *a: None) + + def fast(i, plan, pos, h, r, step, capture=None): + seen.append((i, capture)) + if capture is not None: + target._capture_written = written + return ("h", i), ("r", i) + + def generic(*a): + raise AssertionError("a capture layer ran the generic layer on a decode step") + + monkeypatch.setattr(target, "_layer", fast) + monkeypatch.setattr(target, "_generic_layer", generic) + out = target.forward( + None, + torch.zeros(4, dtype=torch.int32), + None, + None, + _Spec([1]), + lambda spec, i, h, r, written=False: captured.append((i, h, r, written)), + lambda spec, i, n: ("capture_out", i, n), + ) + assert out == ("h", 2) + assert seen == [(0, None), (1, ("capture_out", 1, 4)), (2, None)] + assert captured == [(1, ("r", 1), None, written)] + + +@pytest.fixture +def ctxnormq_prefill(monkeypatch): + """(core, the prefill target's generic layer, the boundary calls), with CTXNORMQ taking every boundary offered. + + Only the collectives are stand-ins: the layer and its exit boundary (``PrefillTarget._generic_boundary``) are the + target's own, so the test reads which boundary each result came from and where the layer uses it. + """ + calls = [] + w = {f"l{i}_{n}": f"{n}{i}" for i in range(LAYERS) for n in ("in_norm", "post_norm")} + w["final_norm"] = "final" + c = SimpleNamespace( + w=w, + num_layers=LAYERS, + moe_layers=frozenset({2, 3}), + rms_eps=1e-6, + allreduce=None, + allreduce_mnnvl=None, + moe={i: SimpleNamespace(fc31_input_scale=f"fp4_{i}") for i in (2, 3)}, + ) + monkeypatch.setattr(modeling, "_can_mc", lambda x, folded=False: False) + + def ctxnormq(core, x, residual, norm_w, fp4_scale, allreduce=None): + calls.append(("ctxnormq", x, residual, norm_w, fp4_scale)) + return ("ctxnormq_normed", norm_w), ("ctxnormq_updated", norm_w) + + def native(core, x, residual, norm_w, allreduce=None): + calls.append(("native", x, residual, norm_w)) + return ("native_normed", norm_w), ("native_updated", norm_w) + + monkeypatch.setattr(modeling, "_ctxnormq_allreduce_norm", ctxnormq) + monkeypatch.setattr(modeling, "_native_allreduce_norm", native) + cls = modeling.PrefillTarget + monkeypatch.setattr(cls, "_attention", lambda self, i, *a, **k: (_Rows(("attn", i)), False)) + monkeypatch.setattr(cls, "_moe", lambda self, i, h, **k: (_Rows(("moe", i, h)), None)) + monkeypatch.setattr( + cls, "_swiglu_mlp", lambda self, h, prefix, *a, **k: _Rows(("mlp", prefix, h)) + ) + return c, cls(c)._layer, calls + + +@pytest.mark.parametrize("i", range(LAYERS)) +def test_prefill_uses_the_ctxnormq_boundary_in_place_of_the_fused_allreduce( + ctxnormq_prefill, i +) -> None: + """A prefill boundary CTXNORMQ takes is used where the native fused all-reduce's would be. + + The attention boundary (with the MoE's NVFP4 input scale on a MoE layer) feeds the feed-forward, and its updated + residual goes to the exit boundary. The exit boundary is CTXNORMQ's into every next input norm; the final norm + keeps the native fused all-reduce. Its (normed, updated) is what the layer returns, and its updated is what a + capture layer captures. + """ + core, run, calls = ctxnormq_prefill + residual = torch.zeros(2) + captured = [] + out = run( + i, + None, + torch.ones(2), + residual, + None, + _Spec([i]), + lambda spec, layer, h, r: captured.append((layer, h, r)), + ) + post = core.w[f"l{i}_post_norm"] + kind, x, res, norm, fp4 = calls[0] + assert (kind, x, norm) == ("ctxnormq", ("attn", i), post), ( + "the attention boundary is offered to CTXNORMQ" + ) + assert res is residual + assert fp4 == (f"fp4_{i}" if i in core.moe_layers else None), "NVFP4 for the MoE's input only" + normed = ("ctxnormq_normed", post) + ff = ("moe", i, normed) if i in core.moe_layers else ("mlp", f"l{i}_mlp", normed) + if i + 1 < LAYERS: + nxt = core.w[f"l{i + 1}_in_norm"] + assert calls[1] == ("ctxnormq", ff, ("ctxnormq_updated", post), nxt, None) + assert out == (("ctxnormq_normed", nxt), ("ctxnormq_updated", nxt)) + else: + assert calls[1] == ("native", ff, ("ctxnormq_updated", post), "final"), ( + "the final norm keeps the native fused all-reduce" + ) + assert out == (("native_normed", "final"), ("native_updated", "final")) + assert len(calls) == 2 + assert captured == [(i, out[1], None)] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_dispatch.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_dispatch.py new file mode 100644 index 000000000000..3c57ac837d01 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_dispatch.py @@ -0,0 +1,318 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp4 (Rubin) decode target's fast-path table (``modeling.py``: the row ranges and the selection functions): its +row ranges against the catalog cells that certify them, the selection for every decode row count, and the phase split. + +The paths whose cells pass on SM 10.7 take the sm_103 tp4 rows; the ones built on kernels with no SM 10.7 build (the +the fp32 routers, the low-M GEMM, the fused / mega / c1 attention) and the ones not yet driven on +SM 10.7 (the long-row select, the capped dense decode split, the early drafter fc) cover nothing. The dense layers' and +the drafter's full-context decode take the one-request shapes measured at TP4. + +CPU only: nothing here launches a kernel. +""" + +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + triton_sparse_decode as tsd, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm import ( # noqa: E501 + m3_mx_gemv, +) + +ROWS = range(1, dispatch.DECODE_STEP_MAX + 2) # every decode row count and the first one past it + +# The TP4 per-rank MXFP8 projections (N, K): 16 query heads and 1 KV head, expert width 768 per rank. +TP4_PROJECTIONS = ( + m3_mx_gemv.QKV_SPARSE, + m3_mx_gemv.QKV, + m3_mx_gemv.O_PROJ, + m3_mx_gemv.SHARED_GATE_UP, + m3_mx_gemv.SHARED_DOWN, + m3_mx_gemv.DENSE_GATE_UP, + m3_mx_gemv.DENSE_DOWN, +) + + +@pytest.fixture(autouse=True) +def _policy(): + """Every test starts outside require mode with nothing logged, and leaves it so.""" + dispatch.set_require(False) + dispatch.reset_logged() + yield + dispatch.set_require(False) + dispatch.reset_logged() + + +@pytest.mark.parametrize("fast_path", dispatch.FAST_PATHS, ids=lambda r: r.path) +def test_every_covered_shape_is_a_certified_cell(fast_path) -> None: + uncertified = fast_path.keys - fast_path.certified() + assert not uncertified, ( + f"{fast_path.path} covers {sorted(uncertified, key=str)} but " + f"{type(fast_path.entry).__name__}.CELLS certifies none of them: add a cell or narrow the range" + ) + + +def test_the_fast_path_names_are_unique() -> None: + names = [r.path for r in dispatch.FAST_PATHS] + assert len(names) == len(set(names)) + + +def test_the_decode_target_keeps_the_tp4_geometry() -> None: + """Up to 8 requests verifying 1 + 3 drafts and their padding rows; a wider step is the prefill target's.""" + assert dispatch.DECODE_MAX_TOKENS == 48 + assert dispatch.DECODE_STEP_MAX == (64 if dispatch.VR_BANDS else 48) + assert [dispatch.is_decode_step(n) for n in ROWS] == [True] * dispatch.DECODE_STEP_MAX + [False] + + +# ------------------------------------------------------------------ the paths still off on SM 10.7 + +OFF_ON_SM107 = ( + dispatch.DENSE_DECODE_SPLIT, + dispatch.LONG_SELECT, + dispatch.MC_FINALIZE, +) + + +@pytest.mark.parametrize("fast_path", OFF_ON_SM107, ids=lambda r: r.path) +def test_paths_without_an_sm107_build_cover_nothing(fast_path) -> None: + assert not fast_path.keys, f"{fast_path.path} covers {sorted(fast_path.keys, key=str)}" + + +def test_decode_boundaries_follow_the_multicast_all_reduce_kill() -> None: + """MC_AR_TP4 moves every decode boundary (and the o_proj / MoE pushes into it) onto the SM 10.7 TP4 multicast + all-reduce; off, the built-in all-reduce runs and nothing pushes.""" + on = dispatch.MC_AR_TP4 + for plan in dispatch.DECODE_PLANS: + want = (False, False, False) if plan.name == "wide" else (on, on, on) + assert (plan.attn_mc_ar, plan.moe_mc_ar, plan.qkv_prequant) == want, plan + assert tuple(p.max_tokens for p in dispatch.DECODE_PLANS) == (dispatch.DECODE_MAX_TOKENS,) + ( + (64,) if dispatch.VR_BANDS else () + ) + assert bool(dispatch.MC_ALLREDUCE.keys) == on + assert dispatch.DRAFT_MC_AR.keys == (frozenset(range(1, 9)) if on else frozenset()) + assert dispatch.ATTN_PUSH_MAX_TOKENS == (8 if on else 0) + assert dispatch.DENSE_PUSH_MAX_TOKENS == (8 if on and dispatch.DENSE_PUSH_TP4 else 0) + assert dispatch.MOE_PUSH_MAX_TOKENS == (dispatch.MOE_ENGINE_MAX_TOKENS if on else 0) + assert dispatch.GENERIC_MC_AR_MAX_TOKENS == 0 + assert dispatch.GENERIC_MC_AR_FOLDED_MAX_TOKENS == 0 + + +def test_the_dense_decode_split_is_the_built_in_one() -> None: + assert dict(dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH) == {} + + +# ------------------------------------------------------------------ on: the dense layers' full-context decode + + +def test_the_dense_layers_take_the_fullctx_decode_at_the_measured_tp4_shapes() -> None: + """One or two requests, 1 + 3 verify rows or one draft-pass row, each at its own pinned recipe.""" + assert dispatch.FULLCTX_DECODE_ENABLED + assert dispatch.FULLCTX_DECODE.keys == {(4, 1), (1, 1), (4, 2), (1, 2)} + assert set(dispatch.FULLCTX_DECODE_RECIPES) == set(dispatch.FULLCTX_DECODE.keys) + for recipe in dispatch.FULLCTX_DECODE_RECIPES.values(): + assert 2 <= recipe["splits"] <= 128 # the standalone reducer's range + + +def test_the_fullctx_plans_keep_the_tp4_geometry() -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_fullctx_decode as kernel, + ) + + assert (kernel.HEADS, kernel.KV_HEADS, kernel.HEAD_DIM) == (16, 1, 128) + assert kernel.HEADS_PER_KV == 16 + + +def test_the_fullctx_entry_cells_run_at_the_forward_plan_capacity() -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention import ( # noqa: E501 + m3_fullctx_decode as entry, + ) + + assert entry.MAX_KV_LEN == dispatch.FULLCTX_DECODE_MAX_KV_LEN + + +def test_the_fullctx_capacity_covers_the_agentx_cache_bound() -> None: + """The forward compares the cache's length bound (max_seq_len plus the speculative rows it reserves) with the plan + capacity; at the AgentX max_seq_len of 1 << 20 with up to 8 draft rows the dense layers must still be taken.""" + assert dispatch.FULLCTX_DECODE_MAX_KV_LEN >= (1 << 20) + 8 + + +def test_the_drafter_takes_the_fullctx_decode_at_the_same_shapes() -> None: + assert dispatch.DRAFTER_FULLCTX_ENABLED + assert dispatch.DRAFTER_FULLCTX.keys == dispatch.FULLCTX_DECODE.keys + + +def test_the_dense_fp8_qkv_and_kv_insert_kill_constants_default_on() -> None: + assert dispatch.DENSE_QKV_FP8_VIA_BF16 is True + assert dispatch.DENSE_KV_INSERT_FUSED is True + assert dispatch.FULLCTX_DECODE_OWN_REDUCER is True + assert dispatch.FULLCTX_DECODE_MXFP8 is True + + +def test_the_fullctx_taken_lines_log_without_raising_under_require() -> None: + dispatch.set_require(True) + dispatch.taken(dispatch.FULLCTX_DECODE.path, 4) + dispatch.taken(dispatch.DENSE_KV_INSERT_PATH, 4) + dispatch.taken(dispatch.FULLCTX_DECODE_MXFP8_PATH, 4) + + +# ------------------------------------------------------------------ the paths on + + +@pytest.mark.parametrize("rows", range(1, dispatch.DECODE_MAX_TOKENS + 1)) +def test_the_moe_takes_the_engines_and_the_generation_router(rows) -> None: + """1..8 rows the MoE engine, 9..13 the MoE16 engine (routers in-kernel); the unfused MoE of a pure generation step + takes the TF32 tensor-core router at 9..16 rows (14..16 always, 9..13 on an MoE16 miss), the fp32 router else.""" + expected = ( + dispatch.MOE_ENGINE.path + if rows <= 8 + else dispatch.MOE16_ENGINE.path + if rows <= 13 + else None + ) + assert dispatch.moe_path(rows) == expected + assert dispatch.ROUTER_TC_GEN.covers(rows) == (9 <= rows <= 16) + + +def _router_core(cluster16: bool): + import torch + + return SimpleNamespace(router_tc_cluster16=cluster16, w={"l3_router": torch.zeros(128, 64)}) + + +@pytest.mark.parametrize("rows", [1, 8, 9, 13, 14, 16, 17, 48]) +@pytest.mark.parametrize("cluster16", [True, False]) +def test_the_generation_router_takes_router_tc_only_on_its_rows( + monkeypatch, rows, cluster16 +) -> None: + """The built-in gate's choice: router_tc on 9..16 bf16 rows when the device co-schedules 16-CTA clusters; the fp32 + router otherwise. Context steps (``_router_logits``) never take it.""" + import torch + + calls = [] + monkeypatch.setattr(dispatch, "m3_router_tc", lambda *a: calls.append("tc") or [a[0]]) + monkeypatch.setattr(dispatch, "m3_fp32_linear", lambda *a: calls.append("fp32") or a[0]) + h = torch.zeros(rows, 64, dtype=torch.bfloat16) + dispatch._decode_router_logits(_router_core(cluster16), 3, h) + assert calls == ["tc" if cluster16 and 9 <= rows <= 16 else "fp32"] + calls.clear() + dispatch._router_logits(_router_core(cluster16), 3, h) + assert calls == ["fp32"] + + +def test_the_core_selects_the_target_on_the_context_count_alone() -> None: + """Every pure generation step is decode's, whatever its width; a step with context rows (a mixed one too) is + prefill's.""" + core = SimpleNamespace(_prefill="prefill", _decode="decode") + select = dispatch.MiniMaxM3Tp4ModelingV2._select_target + assert select(core, SimpleNamespace(num_contexts=0)) == "decode" + assert select(core, SimpleNamespace(num_contexts=1)) == "prefill" + assert select(core, SimpleNamespace(num_contexts=7)) == "prefill" + + +@pytest.mark.parametrize("rows", range(1, dispatch.DECODE_MAX_TOKENS + 1)) +def test_the_projections_take_the_gemvs_else_the_mxfp8_gemm(rows) -> None: + for nk in TP4_PROJECTIONS: + prequant = dispatch.linear_path(*nk, rows, prequant=True, low_m_gemm=True) + assert prequant == ( + dispatch.MX_GEMV_PATH + if rows in dispatch.MX_GEMV_ROWS[nk] + else dispatch.PREQUANT_GEMM_PATH + ) + bf16 = dispatch.linear_path(*nk, rows, prequant=False, low_m_gemm=True) + assert bf16 == ( + dispatch.MX_GEMV_BF16_PATH + if rows <= dispatch.MX_GEMV_BF16_MAX_TOKENS[nk] + else dispatch.GENERIC_GEMM_PATH + ) + + +def test_the_attention_producer_and_scorer_are_on() -> None: + assert dispatch.PRODUCER.covers("nvfp4_tp4") + assert dispatch.INDEX_SCORE.keys == {"early", "prefetch"} + assert dispatch.SPARSE_DECODE_PRODUCER.keys == {"per_row", "multi_query"} + assert dispatch.EMBED_FIRSTNORM.keys == frozenset(range(1, 9)) + + +def test_speculation_runs_the_fused_tail_with_the_bf16_draft_head() -> None: + requests = dispatch.DECODE_MAX_TOKENS // (dispatch.SHARD_ACCEPT_DRAFT_LEN + 1) + assert dispatch.SHARD_ACCEPT_MAX_REQUESTS == requests + assert dispatch.SAMPLED_VERIFY_MAX_REQUESTS == requests + assert dispatch.FIRST_DRAFTER_INPUTS_MAX_REQUESTS == requests + assert dispatch.TARGET_HEAD_MAX_ROWS == 0 # the in-tree BF16 head is faster at the TP4 shard + assert dispatch.FC_AFTER_HEAD_MAX_ROWS == 8 + assert dispatch.DRAFT_STEP_MAX_REQUESTS == 8 + assert dispatch.FC_EARLY_MAX_ROWS == 0 + assert not dispatch.DRAFT_HEAD_CODED + + +def test_a_miss_raises_only_under_require() -> None: + dispatch.missed("some_path", 1, "a precondition") + dispatch.set_require(True) + with pytest.raises(dispatch.FastPathUnavailable, match="some_path"): + dispatch.missed("some_path", 1, "a precondition") + + +def test_taken_logs_without_raising_under_require() -> None: + dispatch.set_require(True) + dispatch.taken(dispatch.INDEX_SCORE.path, 1) + dispatch.taken(dispatch.MOE_ENGINE.path, 8) + + +def test_taken_logs_once_per_path_and_rows(monkeypatch) -> None: + lines = [] + monkeypatch.setattr(dispatch.logger, "info", lines.append) + for _ in range(3): + dispatch.taken(dispatch.MX_GEMV_PATH, 4, "[2560, 6144] mxfp8 in") + dispatch.taken(dispatch.MX_GEMV_PATH, 8, "[2560, 6144] mxfp8 in") + assert len(lines) == 2 and all("m3_mx_gemv taken at" in line for line in lines) + + +def test_the_sparse_decode_writes_the_mxfp8_handoff() -> None: + """Every decode step (at most DECODE_MAX_TOKENS rows) can take the MXFP8 o_proj hand-off of the sparse decode.""" + assert dispatch.MERGE_MXFP8_HANDOFF is True + assert tsd.MERGE_MXFP8_MIN_TOKENS <= 1 and dispatch.DECODE_MAX_TOKENS <= tsd.MergeMxfp8.MAX_ROWS + + +def test_the_decode_mxfp8_decision_is_pinned_to_cute_dsl() -> None: + from tensorrt_llm._torch.custom_ops.torch_custom_ops import ( + FlashInferMXFP8GemmRunner, + MXFP8QuantizeRunner, + ) + + assert dispatch.MX_TAC_TP4 + assert dispatch.MXFP8_DECODE_QUANT_TACTIC == MXFP8QuantizeRunner.CUTE_DSL + assert dispatch.MXFP8_DECODE_GEMM_TACTIC == FlashInferMXFP8GemmRunner.CUTE_DSL + + +def _runtime(fused: bool, indexer: str = "fp8", kv: str = "fp8"): + """A model config with the runtime fields ``_validate_runtime`` reads.""" + sparse = SimpleNamespace( + algorithm="minimax_m3", + implementation="msa", + fuse_qkv_index_projection=fused, + indexer_kv_dtype=indexer, + ) + mode = SimpleNamespace( + has_fp8_kv_cache=lambda: kv == "fp8", has_fp4_kv_cache=lambda: kv == "nvfp4" + ) + return SimpleNamespace( + sparse_attention_config=sparse, quant_config=SimpleNamespace(quant_mode=mode) + ) + + +@pytest.mark.parametrize("fused", [True, False], ids=["five_way", "unfused"]) +def test_the_target_takes_both_projection_layouts(fused) -> None: + """The packed five-way projection and the separate QKV + MXFP8 index projections + (fuse_qkv_index_projection=False) are both this target's; a BF16 indexer or a BF16 KV cache is not.""" + dispatch._validate_runtime(_runtime(fused)) + dispatch._validate_runtime(_runtime(fused, kv="nvfp4")) + with pytest.raises(ValueError, match="indexer_kv_dtype='fp8'"): + dispatch._validate_runtime(_runtime(fused, indexer="bf16")) + with pytest.raises(ValueError, match="FP8 or NVFP4 KV cache"): + dispatch._validate_runtime(_runtime(fused, kv="bf16")) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_prebuild.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_prebuild.py new file mode 100644 index 000000000000..59ed0a7140ae --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_prebuild.py @@ -0,0 +1,334 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The decode kernels the core builds before the executor's warmup (``prebuild_kernels``), and that a forward after it +never compiles one: the MoE engines of every ``MOE_ENGINE`` / ``MOE16_ENGINE`` token count, the decode GEMVs with +their push variants, the index scorer with and without its early-issue schedule at every decode query length (and +the T2SC scorer, the long-row select and the full-context plans where they apply), and the 9..16-row generation +router's device decision. + +The engine builds are stubbed (a real one needs a loaded checkpoint's expert buffers; the engine kernels have their +own tests); the GEMVs and the scorer compile for real on the GPU. +""" + +from types import SimpleNamespace + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers the trtllm ops +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_index_score as index_score_kernel, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_index_score_t2sc as index_score_t2sc_kernel, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels.mx_gemv import ( # noqa: E501 + op as mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.attention.m3_index_score import ( # noqa: E501 + m3_index_score, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.ops.gemm.m3_mx_gemv import ( # noqa: E501 + O_PROJ, + SHARED_DOWN, + SHARED_GATE_UP, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + require_msa_module, +) + +dispatch = modeling + +assert torch.cuda.is_available(), "the prebuild checks require a CUDA device" + +INDEX_HEADS = 4 +INDEX_DIM = 128 +DRAFT_LEN = 3 +_CORE_METHODS = ( + "prebuild_kernels", + "_prebuild_engines", + "_try_build", + "engine", + "moe16_engine", + "_lookup", + "_scorer_variants", + "_prebuild_index_scorer", + "_prebuild_index_scorer_t2sc", + "_prebuild_msa_proxy", + "decode_query_lens", +) + + +@pytest.fixture(autouse=True) +def _proxy_builds(monkeypatch): + """The MSA indexer proxy's fmha_sm100 variants are recorded rather than compiled (an nvcc build each).""" + require_msa_module() # its cutlass compatibility shim must precede the fmha_sm100 import, as in the target + from fmha_sm100 import jit as fmha_jit + + built = [] + + def record(*key): + built.append(fmha_jit._variant_key_from_runtime(*key)[0]) + return SimpleNamespace(run=None) + + monkeypatch.setattr(fmha_jit, "get_fmha_variant", record) + return built + + +@pytest.fixture(autouse=True) +def _restore_modes(): + require, frozen = dispatch.require_mode(), mx_gemv.frozen() + yield + dispatch.set_require(require) + mx_gemv.freeze(frozen) + dispatch.reset_logged() + + +class _FakeSparseAttn: + def __init__(self, indexer_kv_dtype: str): + self.indexer_kv_dtype = indexer_kv_dtype + self.m3_config = SimpleNamespace(num_index_heads=INDEX_HEADS, sparse_index_dim=INDEX_DIM) + + +def _core(num_layers: int = 3, indexer_kv_dtype: str = "fp8", built=None, fail=False): + """The parts of ``MiniMaxM3Tp4ModelingV2`` the prebuild reads: one rank (the build agreement takes no + collective), MoE + sparse layers 1.. (layer 0 dense), no projections (the GEMV prebuild has its own test), the + speculative config; the engine builds stubbed.""" + core = SimpleNamespace( + mapping=SimpleNamespace(tp_size=1, enable_attention_dp=False), + num_experts=128, + hidden=6144, + moe_layers=frozenset(range(1, num_layers)), + sparse_layers=frozenset(range(1, num_layers)), + attn=[_FakeSparseAttn(indexer_kv_dtype) for _ in range(num_layers)], + w={"embed": torch.empty(1, device="cuda")}, + mxfp8=SimpleNamespace(quant_method=SimpleNamespace(use_cutlass=True)), + model_config=SimpleNamespace( + spec_config=SimpleNamespace(max_draft_len=DRAFT_LEN, max_total_draft_tokens=DRAFT_LEN) + ), + moe_engines={}, + moe16_engines={}, + projections=lambda: [], + ) + for name in _CORE_METHODS: + setattr(core, name, getattr(modeling.MiniMaxM3Tp4ModelingV2, name).__get__(core)) + built = [] if built is None else built + + def build(i, num_tokens, shared=True): + if fail: + raise ValueError("the routed experts are not TRTLLM-Gen NVFP4 SwiGLU-OAI") + built.append(("moe", i, num_tokens, shared)) + return ("engine", i, num_tokens) + + def build16(i, num_tokens): + if fail: + raise ValueError("the routed experts are not TRTLLM-Gen NVFP4 SwiGLU-OAI") + built.append(("moe16", i, num_tokens)) + return ("engine16", i, num_tokens) + + core._build_engine, core._build_moe16_engine = build, build16 + return core + + +def _forbid_compiles(monkeypatch, core=None) -> None: + """Every compile / build entry point a forward could reach raises.""" + import cutlass.cute as cute + + def boom(*args, **kwargs): + raise AssertionError("a compile entry point ran after prebuild") + + monkeypatch.setattr(cute, "compile", boom) + monkeypatch.setattr(mx_gemv, "ensure_compiled", boom) + monkeypatch.setattr(mx_gemv, "precompile", boom) + monkeypatch.setattr(index_score_kernel.IndexScoreRunner, "_compile", boom) + monkeypatch.setattr(index_score_t2sc_kernel.IndexScoreRunner, "_compile", boom) + if core is not None: + core._build_engine = core._build_moe16_engine = boom + + +def test_decode_query_lens_follow_the_speculative_config() -> None: + core = _core() + assert core.decode_query_lens() == [1, 1 + DRAFT_LEN] + core.model_config.spec_config = None + assert core.decode_query_lens() == [1] + core.model_config.spec_config = SimpleNamespace( + max_draft_len=4, max_total_draft_tokens=4, draft_len_schedule={1: 4, 8: 2} + ) + assert core.decode_query_lens() == [1, 3, 5] + + +def test_prebuild_builds_every_engine_count_and_scorer_variant() -> None: + built = [] + core = _core(built=built) + times = core.prebuild_kernels() + + assert mx_gemv.frozen(), "prebuild must close the GEMV compiles" + + # Layer 0 is dense; every MoE layer gets every count, in layer order. + def per_layer(i): + engines = [("moe", i, n, True) for n in range(1, dispatch.MOE_ENGINE_MAX_TOKENS + 1)] + return engines + [ + ("moe16", i, n) for n in range(dispatch.MOE16_MIN_TOKENS, dispatch.MOE16_MAX_TOKENS + 1) + ] + + assert built == per_layer(1) + per_layer(2) + assert set(core.moe_engines[2]) == {(n, True, True) for n in dispatch.MOE_ENGINE.keys} + assert set(core.moe16_engines[2]) == set(dispatch.MOE16_ENGINE.keys) + + want = { + (torch.float8_e4m3fn, INDEX_HEADS, qlen, INDEX_DIM, modeling.INDEX_SCORE_SPLIT_K, early) + for qlen in (1, 1 + DRAFT_LEN) + for early in (True, False) + } + assert set(times["scorer_keys"]) == want + assert want <= set(index_score_kernel.IndexScoreRunner.kernel_cache) + for key in ("gemv_s", "moe_s", "scorer_s", "proxy_s"): + assert times[key] >= 0.0 + + # The generation router's device decision, as the built-in gate makes it. + assert core.router_tc_cluster16 == bool(torch.ops.trtllm.minimax_m3_router_tc_cluster16(6144)) + + # A second build builds no engine again. + built.clear() + core.prebuild_kernels() + assert built == [] + + +def test_engine_lookups_after_prebuild_never_build(monkeypatch) -> None: + core = _core() + core._prebuild_engines(5) + _forbid_compiles(monkeypatch, core) + for require in (False, True): + dispatch.set_require(require) + for n in sorted(dispatch.MOE_ENGINE.keys): + assert core.engine(5, n, shared=True) == ("engine", 5, n) + for n in sorted(dispatch.MOE16_ENGINE.keys): + assert core.moe16_engine(5, n) == ("engine16", 5, n) + + +def test_missing_engine_is_a_miss_not_a_build(monkeypatch) -> None: + core = _core() + core.moe_engines[7], core.moe16_engines[7] = {}, {} + _forbid_compiles(monkeypatch, core) + dispatch.set_require(False) + assert core.engine(7, 3, shared=True) is None + assert core.moe16_engine(7, 10) is None + dispatch.set_require(True) + with pytest.raises(dispatch.FastPathUnavailable, match="not built before warmup"): + core.engine(7, 3, shared=True) + with pytest.raises(dispatch.FastPathUnavailable, match="not built before warmup"): + core.moe16_engine(7, 10) + + +def test_failed_engine_build_is_logged_then_missed() -> None: + core = _core(fail=True) + core._prebuild_engines(9) + assert all(v is None for v in core.moe_engines[9].values()) and len(core.moe_engines[9]) == 8 + dispatch.set_require(True) + with pytest.raises(dispatch.FastPathUnavailable, match="build failed"): + core.engine(9, 1, shared=True) + + +def _projection(nk, push): + n, k = nk + w = (torch.randn(n, k, device="cuda") / k**0.5).to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + return SimpleNamespace(weight=wq.view(n, k), weight_scale=wsf, push=push) + + +@pytest.mark.parametrize( + "nk,push", + ((O_PROJ, "attn"), (SHARED_DOWN, "moe"), (SHARED_GATE_UP, None)), + ids=("o_proj", "shared_down", "shared_gate_up"), +) +def test_gemvs_after_prebuild_never_compile(monkeypatch, nk, push) -> None: + use_pdl = True + module = _projection(nk, push) + variants = modeling._gemv_variants(*nk, push, 4) + pushes = [v for v in variants if v[2] is not None] + if push is None: + assert not pushes + else: + covered = dispatch.ATTN_PUSH.keys if push == "attn" else dispatch.MOE_PUSH.keys + assert {rows for rows, _, _ in pushes} == set(covered) + assert {p[1] for _, _, p in pushes} == {b for b in (8, 16) if b <= max(covered)} + mx_gemv.freeze(False) + # As prebuild_kernels compiles one projection. + mx_gemv.ensure_compiled(module.weight, module.weight_scale, use_pdl) + for rows, x_bf16, ar in variants: + mx_gemv.precompile(module.weight, module.weight_scale, rows, x_bf16, use_pdl=use_pdl, ar=ar) + mx_gemv.freeze(True) + _forbid_compiles(monkeypatch) + + n, k = nk + for rows, x_bf16, ar in variants: + if ar is not None: + # A push needs a multicast exchange to launch: its kernel is compiled (the forward finds it). + resolved = mx_gemv._resolve(rows, n, k, x_bf16, True, None, use_pdl, (0, 0, 0, *ar)) + if resolved is not None: + assert mx_gemv.is_compiled(resolved[0]), (rows, x_bf16, ar) + continue + if x_bf16: + x = torch.randn(rows, k, device="cuda").to(torch.bfloat16) + out = mx_gemv.m3_mx_gemv_bf16(x, module.weight, module.weight_scale, use_pdl=use_pdl) + else: + xq, xsf = torch.ops.trtllm.mxfp8_quantize( + torch.randn(rows, k, device="cuda").to(torch.bfloat16), True + ) + out = mx_gemv.m3_mx_gemv( + xq.view(rows, k), xsf, module.weight, module.weight_scale, use_pdl=use_pdl + ) + served = mx_gemv.serves(n, k, rows, x_bf16) + assert (out is not None) == served, (rows, x_bf16) + assert mx_gemv.last_uncompiled is None + torch.cuda.synchronize() + + +def test_frozen_gemv_declines_an_uncompiled_kernel(monkeypatch) -> None: + module = _projection(SHARED_DOWN, None) + mx_gemv.freeze(True) + _forbid_compiles(monkeypatch) + # A launch option nothing prebuilt (no early trigger, without a push): declined, never compiled. + xq, xsf = torch.ops.trtllm.mxfp8_quantize( + torch.randn(3, 768, device="cuda").to(torch.bfloat16), True + ) + key = mx_gemv._resolve(3, 6144, 768, False, False, None, True, None)[0] + if mx_gemv.is_compiled(key): + pytest.skip("this process already compiled the probe kernel") + out = mx_gemv.m3_mx_gemv( + xq.view(3, 768), xsf, module.weight, module.weight_scale, trigger_early=False, use_pdl=True + ) + assert out is None and mx_gemv.last_uncompiled == key + + +@pytest.mark.parametrize("indexer_kv_dtype", ("fp8", "bf16")) +def test_scorer_after_prebuild_never_compiles(monkeypatch, indexer_kv_dtype) -> None: + core = _core(indexer_kv_dtype=indexer_kv_dtype) + keys = core._prebuild_index_scorer(core.decode_query_lens(), torch.device("cuda")) + assert len(keys) == 4 + _forbid_compiles(monkeypatch, core) + dtype = torch.float8_e4m3fn if indexer_kv_dtype == "fp8" else torch.bfloat16 + for _, heads, qlen, dim, split_k, early in keys: + rows = 1 if early else 2 + blocks = 3 + idx_q = torch.randn(rows * qlen, heads, dim, device="cuda").to(dtype) + idx_k = torch.randn(rows * blocks, 1, 128, dim, device="cuda").to(dtype) + block_table = torch.arange(rows * blocks, dtype=torch.int32, device="cuda").view( + rows, blocks + ) + seq_lens = torch.full((rows,), blocks * 128 - 5, dtype=torch.int32, device="cuda") + score = torch.empty(heads, blocks, rows * qlen, dtype=torch.float32, device="cuda") + m3_index_score( + idx_q, + idx_k.squeeze(1), + block_table, + seq_lens, + score.transpose(1, 2), + qlen, + split_k, + early, + ) + torch.cuda.synchronize() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_rank_agreement.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_rank_agreement.py new file mode 100644 index 000000000000..b10b8b93fb8f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/test_modeling_v2_m3_rank_agreement.py @@ -0,0 +1,181 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Decode robustness without a GPU kernel: the multicast all-reduce epilogue options keep the cluster-barrier bit, +an engine build that fails on one TP rank drops that engine on every rank (two ranks simulated with threads), and +the Eagle3 fc fork's output is recorded on the stream that reads it.""" + +import threading +from types import SimpleNamespace +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp4.kernels import ( # noqa: E501 + m3_mc_allreduce as mc, +) + + +@pytest.fixture(autouse=True) +def _restore_modes(): + require = modeling.require_mode() + yield + modeling.set_require(require) + modeling.reset_logged() + + +@pytest.mark.parametrize("finalize", [False, True]) +@pytest.mark.parametrize("bucket", [8, 16, 32, 48]) +def test_ar_opts_keep_cluster_barrier_bit(monkeypatch, bucket, finalize): + # Tables without bit 2 and with buckets missing: every lookup, fallback included, keeps the barrier. + monkeypatch.setattr(mc, "AR_OPTS_BY_M", (0, 1, 4, 5, 8, 9, 12, 13)) + monkeypatch.setattr(mc, "AR_OPTS_BY_BUCKET", {}) + monkeypatch.setattr(mc, "AR_FIN_OPTS_BY_BUCKET", {}) + for m in range(1, min(bucket, 8) + 1): + opts = mc._ar_opts(m, bucket, finalize=finalize) + assert opts & mc._OPT_CLUSTER_BARRIER and 0 <= opts <= 15, (m, bucket, finalize, opts) + + +def test_ar_opts_shipped_tables_unchanged(): + for m in range(1, 9): + assert mc._ar_opts(m, 8) == mc.AR_OPTS_BY_M[m - 1] + for b, o in mc.AR_OPTS_BY_BUCKET.items(): + assert mc._ar_opts(b, b) == o + for b, o in mc.AR_FIN_OPTS_BY_BUCKET.items(): + assert mc._ar_opts(b, b, finalize=True) == o + + +class _FakeComm: + """An MPI-like TP communicator for ``world`` threads: allreduce sums the ranks' ints.""" + + def __init__(self, world): + self.world = world + self.barrier = threading.Barrier(world) + self.lock = threading.Lock() + self.vals = [] + self.calls = 0 + + def allreduce(self, v): + with self.lock: + self.vals.append(v) + self.calls += 1 + self.barrier.wait() + total = sum(self.vals) + self.barrier.wait() + with self.lock: + self.vals.clear() + self.barrier.wait() + return total + + def Get_size(self): # noqa: N802 (MPI name) + return self.world + + +def _mapping(tp_size=2, adp=False): + return SimpleNamespace(tp_size=tp_size, enable_attention_dp=adp, tp_group=list(range(tp_size))) + + +def _run_ranks(fns): + out, errs = [None] * len(fns), [] + + def run(i): + try: + out[i] = fns[i]() + except BaseException as e: # noqa: BLE001 + errs.append(e) + + threads = [threading.Thread(target=run, args=(i,)) for i in range(len(fns))] + for t in threads: + t.start() + for t in threads: + t.join(timeout=30) + assert not errs, errs + return out + + +@pytest.fixture +def two_rank_comm(monkeypatch): + from tensorrt_llm._torch.distributed import ops + + comm = _FakeComm(2) + monkeypatch.setattr(mc, "_tp_comm", lambda mapping: comm) + monkeypatch.setattr(ops, "mpi_disabled", lambda: False) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + return comm + + +def test_tp_agree_single_rank_and_adp_take_no_collective(monkeypatch): + monkeypatch.setattr(mc, "_tp_comm", mock.Mock(side_effect=AssertionError("no collective"))) + assert mc.tp_agree(_mapping(1), True) is True + assert mc.tp_agree(_mapping(1), False) is False + assert mc.tp_agree(_mapping(4, adp=True), False) is False + + +def test_tp_agree_refuses_capture(monkeypatch): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: True) + with pytest.raises(RuntimeError): + mc.tp_agree(_mapping(2), True) + + +@pytest.mark.parametrize("oks", [(True, True), (True, False), (False, True), (False, False)]) +def test_tp_agree_built_two_ranks(two_rank_comm, oks): + built = [object() if ok else None for ok in oks] + out = _run_ranks([lambda i=i: mc.tp_agree_built(_mapping(2), built[i], "t") for i in range(2)]) + assert out == (built if all(oks) else [None, None]) + + +def _core(fail: bool) -> SimpleNamespace: + """The engine-building half of the core (``MiniMaxM3Tp4ModelingV2``) on one of two simulated TP ranks, its + engine builds stubbed (and failing on one rank).""" + core = SimpleNamespace(mapping=_mapping(2), moe_engines={}, moe16_engines={}) + cls = modeling.MiniMaxM3Tp4ModelingV2 + + def build(i, num_tokens, shared=True): + if fail: + raise RuntimeError("compile failed on this rank") + return ("engine", num_tokens) + + core._build_engine = build + core._build_moe16_engine = lambda i, num_tokens: build(i, num_tokens) + for name in ("_prebuild_engines", "_try_build", "engine", "moe16_engine", "_lookup"): + setattr(core, name, getattr(cls, name).__get__(core)) + return core + + +@pytest.mark.parametrize("fail", [(False, False), (False, True)]) +def test_engine_build_agreed_across_ranks(two_rank_comm, fail): + cores = [_core(f) for f in fail] + _run_ranks([lambda c=c: c._prebuild_engines(3) for c in cores]) + keys = sorted(cores[0].moe_engines[3]) + sorted(cores[0].moe16_engines[3]) + assert keys and two_rank_comm.calls == 2 * len(keys) + for engines in ("moe_engines", "moe16_engines"): + a, b = getattr(cores[0], engines)[3], getattr(cores[1], engines)[3] + assert a.keys() == b.keys() + for key in a: + assert (a[key] is None) == (b[key] is None) == any(fail), (engines, key) + # Both ranks then take the same path in the forward: the engine, or (after a miss) the generic MoE. + modeling.set_require(False) + for n in sorted(modeling.MOE_ENGINE.keys): + got = [c.engine(3, n, shared=True) for c in cores] + assert (got[0] is None) == (got[1] is None) + + +@pytest.mark.parametrize("capturing", [False, True]) +def test_fc_join_records_output_on_main(monkeypatch, capturing): + M3Eagle3Worker = modeling.M3Eagle3Worker + main, fork = mock.Mock(name="main"), mock.Mock(name="fork") + hidden = mock.Mock(name="hidden", is_cuda=True) + monkeypatch.setattr(torch.cuda, "current_stream", lambda *a: main) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: capturing) + worker = SimpleNamespace(_m3_fc_hidden=hidden) + M3Eagle3Worker._m3_fc_join(worker, fork) + main.wait_stream.assert_called_once_with(fork) + if capturing: + hidden.record_stream.assert_not_called() + else: + hidden.record_stream.assert_called_once_with(main) + worker._m3_fc_hidden = None + M3Eagle3Worker._m3_fc_join(worker, fork) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/activation/test_modeling_v2_m31_tp8_swq.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/activation/test_modeling_v2_m31_tp8_swq.py new file mode 100644 index 000000000000..82ae653efa0f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/activation/test_modeling_v2_m31_tp8_swq.py @@ -0,0 +1,99 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_swq entry of the MiniMax-M3.1 TP8 target (shared expert intermediate 384): SwiGLU-OAI +written as MXFP8, bitwise to silu_and_mul + mxfp8_quantize.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.activation.m3_swq import ( # noqa: E501 + m3_swq as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_swq requires a CUDA device" + +OAI = (7.0, 1.702, 1.0) + + +def _assert_bitwise(x): + q, sf = op(x, *OAI) + rq, rsf = op.reference(x, *OAI) + assert q.shape == rq.shape and q.dtype == torch.float8_e4m3fn + assert torch.equal(q.view(torch.uint8), rq.view(torch.uint8)) + assert sf.numel() <= rsf.numel() and torch.equal(sf, rsf.view(-1)[: sf.numel()]) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = (torch.randn(cell.spec["m"], 2 * cell.spec["inter"], generator=g, device="cuda") * 4.0).to( + torch.bfloat16 + ) + with validating(op): + _assert_bitwise(x) + + +@pytest.mark.parametrize("m", [9, 16, 32, 48]) +@pytest.mark.parametrize("scale", [1e-3, 1.0, 1e3]) +def test_bitwise_across_magnitudes_and_edges(m, scale) -> None: + """Scaled inputs, gates past the clamp, an all-zero row and a -0.0 row: q bytes and every scale byte.""" + g = torch.Generator(device="cuda").manual_seed(m * 1000 + int(scale * 10) % 997) + x = torch.randn(m, 2 * 384, generator=g, device="cuda") * scale + x[1, :384] = x[1, :384].abs() + OAI[0] + x[2] = 0.0 + x[3] = -0.0 + _assert_bitwise(x.to(torch.bfloat16)) + + +def _scale_bytes(sf, m, nb): + """Un-swizzle the 128x4 scale layout to [m, nb] bytes.""" + nbp = (nb + 3) // 4 * 4 + r = torch.arange(m, device=sf.device).view(-1, 1) + c = torch.arange(nb, device=sf.device).view(1, -1) + idx = ( + (c % 4) + (c // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + (r // 128) * (128 * nbp) + ) + return sf.view(-1)[idx] + + +@pytest.mark.parametrize("m", [9, 48]) +def test_tiny_amax_blocks_stay_finite(m) -> None: + """A block whose scale code is 0 (amax / 448 <= 2^-127) is native's inf/NaN; SWQ keeps native's scale + byte, writes data quantized with the exact reciprocal 2^127, and stays bitwise to native on every other block.""" + g = torch.Generator(device="cuda").manual_seed(151 + m) + gate = torch.randn(m, 384, generator=g, device="cuda") + up = torch.randn(m, 384, generator=g, device="cuda") + pattern = torch.tensor([1.0, -0.75, 0.5, -0.25, 0.375, 0.0, -1.0, 0.625] * 4, device="cuda") + for r in range(m): + for j, c in enumerate([2.0**-130, 2.0**-127, 448 * 2.0**-128, 448 * 2.0**-127, 2.0**-118]): + blk = (r * 5 + j * 3) % 12 + gate[r, blk * 32 : (blk + 1) * 32] = pattern * (2 * c) + up[r, blk * 32 : (blk + 1) * 32] = pattern * (2 * c) + x = torch.cat([gate, up], 1).to(torch.bfloat16) + q, sf = op(x, *OAI) + rq, rsf = op.reference(x, *OAI) + h = torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=OAI[0], swiglu_alpha=OAI[1], swiglu_beta=OAI[2] + ).float() + byte, rbyte = _scale_bytes(sf, m, 12), _scale_bytes(rsf, m, 12) + assert torch.equal(byte, rbyte), "scale bytes are native's on every block" + deq = q.float().view(m, 12, 32) * torch.exp2(byte.float() - 127.0).unsqueeze(-1) + rdeq = rq.float().view(m, 12, 32) * torch.exp2(rbyte.float() - 127.0).unsqueeze(-1) + native_finite = torch.isfinite(rdeq).all(-1) + code0 = (byte == 0) & (h.view(m, 12, 32).abs().amax(-1) != 0) + assert bool(code0.any()), "the planted blocks reach scale code 0" + assert not bool(native_finite[code0].any()), ( + "native is inf/NaN on code-0 blocks (the bug this guards)" + ) + assert bool(torch.isfinite(deq).all()), "SWQ is finite everywhere" + qb, rqb = q.view(torch.uint8).view(m, 12, 32), rq.view(torch.uint8).view(m, 12, 32) + assert bool((qb == rqb).all(-1)[native_finite].all()), ( + "bitwise to native wherever native is finite" + ) + exact = (h.view(m, 12, 32) * 2.0**127).to(torch.float8_e4m3fn).view(torch.uint8) + assert bool((qb == exact).all(-1)[code0].all()), "code-0 data = RN-e4m3(h * 2^127)" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/activation/test_modeling_v2_m3_silu_and_mul.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/activation/test_modeling_v2_m3_silu_and_mul.py new file mode 100644 index 000000000000..5be4a5c752f1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/activation/test_modeling_v2_m3_silu_and_mul.py @@ -0,0 +1,49 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_silu_and_mul entry of the MiniMax-M3 tp8 target: SwiGLU-OAI and plain SwiGLU.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.activation.m3_silu_and_mul import ( # noqa: E501 + m3_silu_and_mul as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_silu_and_mul requires a CUDA device" + +OAI = (7.0, 1.702, 1.0) + + +def _build(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + # Gate and up spread past the clamp (|x| up to ~12) so both clamps engage. + x = (torch.randn(spec["m"], 2 * spec["inter"], generator=g, device="cuda") * 4.0).to( + torch.bfloat16 + ) + return (x, *OAI) if spec["oai"] else (x, None, None, None) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """fp32 arithmetic and one bf16 rounding on both sides: the kernel's fast sigmoid moves the fp32 value by a few + fp32 ulps, which flips the bf16 rounding of an element by at most one step (1 unit of ULP[bf16] of the row); + measured on GB300: worst element 0.5 unit, relative RMS 0.01.""" + args = _build(cell.spec, zlib.crc32(cell.why.encode()) % 2**31) + with validating(op): + out = op(*args) + assert out.shape == (cell.spec["m"], cell.spec["inter"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(*args), element_ulp=1.0, rms_ulp=0.1) + + +def test_the_gate_sees_a_missing_clamp() -> None: + """Dropping the SwiGLU-OAI clamps moves every element whose gate or up passed the limit far outside the band.""" + spec = op.CELLS[0].spec + x, *oai = _build(dict(spec, m=64), 3) + out = op(x, *oai) + with pytest.raises(AssertionError): + assert_within_ulp(out, op.reference(x, None, oai[1], oai[2]), element_ulp=1.0, rms_ulp=0.1) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/_m3_attn_runtime_tp8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/_m3_attn_runtime_tp8.py new file mode 100644 index 000000000000..e9aaad5c65fa --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/_m3_attn_runtime_tp8.py @@ -0,0 +1,274 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Real MiniMax-M3 attention runtime state at the TP8 per-rank geometry, for the backend-step entries' GPU tests. + +One hybrid NVFP4 KV cache manager (``MiniMaxM3KVCacheManagerV2``) laid out as the target's: a dense layer in the FP8 +half, a sparse MSA layer in the NVFP4 half with its FP8 index-K cache, and (``draft=True``) the one-model Eagle3 +drafter's shared layer, read through the manager's FP8 draft view as the runtime's draft swap reads it. Metadata is +the runtime's own type, filled by its own ``prepare()``; the backends are built the way the target builds them. + +The geometry is one rank's: 8 query heads over one KV head (M3's 4 KV heads, each duplicated on a rank pair) and +one index head of dim 128. The mapping is a single rank holding exactly those heads, so nothing is sharded again. + +History rows are written by the in-tree scatters (``fused_write_layer_caches`` for the FP8 layers, +``fused_write_layer_caches_nvfp4`` for the NVFP4 layer and its index-K), at the slots the manager assigned, which are +not in request order. +""" + +from __future__ import annotations + +from typing import Sequence + +import torch + +from tensorrt_llm._torch.attention.backends.interface import PositionalEmbeddingParams, RopeParams +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import ( + MiniMaxM3KVCacheManagerV2, + MiniMaxM3MsaSparseAttention, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_scatter import ( + fused_write_layer_caches, + fused_write_layer_caches_nvfp4, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.msa_backend import ( + MiniMaxM3MsaSparseAttentionMetadata, +) +from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention, TrtllmAttentionMetadata +from tensorrt_llm._torch.attention.backends.utils import create_attention +from tensorrt_llm._torch.metadata import KVCacheParams +from tensorrt_llm._torch.pyexecutor.runtime_hooks import MsaRuntimePolicy +from tensorrt_llm.bindings import DataType +from tensorrt_llm.bindings.internal.batch_manager import CacheType +from tensorrt_llm.functional import PositionEmbeddingType +from tensorrt_llm.llmapi.llm_args import ( + Eagle3DecodingConfig, + KvCacheConfig, + MiniMaxM3SparseAttentionConfig, +) +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.models.modeling_utils import QuantConfig +from tensorrt_llm.quantization.mode import QuantAlgo + +HEADS = 8 +KV_HEADS = 1 +INDEX_HEADS = 1 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 +DENSE, SPARSE, DRAFT = 0, 1, 2 +MAX_BATCH = 8 +MAX_SEQ_LEN = 8192 +MAX_NUM_TOKENS = 8192 +#: The Eagle3 drafter checkpoint's RoPE: NeoX over the whole head, theta 5e6. +DRAFT_ROPE_THETA = 5.0e6 + +#: E4M3-exact planted index levels, 1/8 apart in relative terms (0.25 .. 15): 48 distinct block scores. +LEVELS = tuple(2.0**e * (1 + m / 8) for e in range(-2, 4) for m in range(8)) +#: The index value of a block's non-peak rows; its score (8) is below every level's (>= 16). +LOW = 0.125 + + +def mapping() -> Mapping: + return Mapping(world_size=1, rank=0, tp_size=1, pp_size=1) + + +def sparse_attention_config() -> MiniMaxM3SparseAttentionConfig: + """The target's runtime: MSA, the FP8 indexer, the fused five-way projection (M3's checkpoint selection rule: + no forced initial blocks, the query's own block forced).""" + return MiniMaxM3SparseAttentionConfig( + implementation="msa", + indexer_kv_dtype="fp8", + fuse_qkv_index_projection=True, + sparse_num_index_heads=INDEX_HEADS, + num_attention_heads=HEADS, + num_key_value_heads=KV_HEADS, + ) + + +def make_manager(*, draft: bool) -> MiniMaxM3KVCacheManagerV2: + torch.cuda.init() + return MiniMaxM3KVCacheManagerV2( + KvCacheConfig( + max_tokens=4 * MAX_SEQ_LEN, + enable_block_reuse=False, + enable_swa_scratch_reuse=False, + host_cache_size=0, + event_buffer_max_size=0, + ), + CacheType.SELF, + num_layers=2, + num_kv_heads=[KV_HEADS] * (3 if draft else 2), + head_dim=HEAD_DIM, + tokens_per_block=PAGE, + max_seq_len=MAX_SEQ_LEN, + max_batch_size=MAX_BATCH, + max_num_tokens=MAX_NUM_TOKENS, + mapping=mapping(), + dtype=DataType.NVFP4, + vocab_size=16, + spec_config=( + Eagle3DecodingConfig(max_draft_len=3, speculative_model="unused") if draft else None + ), + sparse_layer_ids=[SPARSE], + sparse_attention_config=sparse_attention_config(), + ) + + +def make_msa_backend(layer: int) -> MiniMaxM3MsaSparseAttention: + """One layer's MSA backend as ``MiniMaxM3Tp8ModelingV2`` builds it (NVFP4 KV quant config on every layer).""" + quant = QuantConfig(kv_cache_quant_algo=QuantAlgo.NVFP4) + backend = create_attention( + "TRTLLM", + layer, + HEADS, + HEAD_DIM, + KV_HEADS, + pos_embd_params=None, + quant_config=quant, + q_scaling=1.0, + sparse_params=sparse_attention_config().to_sparse_params(layer_idx=layer), + ) + backend.update_quant_config(quant) + assert isinstance(backend, MiniMaxM3MsaSparseAttention), type(backend).__name__ + return backend + + +def make_draft_backend() -> TrtllmAttention: + """The drafter layer's attention as ``M3Eagle3Draft`` builds it: fused NeoX RoPE, an FP8 KV cache (the target + overrides the drafter's KV dtype to FP8), no other quantization.""" + quant = QuantConfig(kv_cache_quant_algo=QuantAlgo.FP8) + backend = create_attention( + "TRTLLM", + DRAFT, + HEADS, + HEAD_DIM, + KV_HEADS, + pos_embd_params=PositionalEmbeddingParams( + type=PositionEmbeddingType.rope_gpt_neox, + rope=RopeParams(dim=HEAD_DIM, theta=DRAFT_ROPE_THETA, max_positions=MAX_SEQ_LEN), + ), + quant_config=quant, + q_scaling=1.0, + attention_chunk_size=None, + sparse_params=None, + ) + backend.update_quant_config(quant) + assert type(backend) is TrtllmAttention, type(backend).__name__ + return backend + + +def make_msa_metadata(manager) -> MiniMaxM3MsaSparseAttentionMetadata: + metadata = MiniMaxM3MsaSparseAttentionMetadata( + max_num_requests=MAX_BATCH, + max_num_tokens=MAX_NUM_TOKENS, + max_num_sequences=MAX_BATCH, + kv_cache_manager=manager, + mapping=mapping(), + sparse_metadata_params=sparse_attention_config().to_sparse_metadata_params(), + ) + # The target's MSA preparation choice that changes what prepare() stages (the identity tables of short steps). + metadata.msa_policy = MsaRuntimePolicy(short_sequence_identity=True) + return metadata + + +def make_draft_metadata(manager) -> TrtllmAttentionMetadata: + """Drafter metadata over the shared manager's FP8 draft view, the manager the runtime's draft swap installs.""" + view = manager.get_draft_kv_cache_view() + assert view is not None and view.dtype == DataType.FP8 + return TrtllmAttentionMetadata( + max_num_requests=MAX_BATCH, + max_num_tokens=MAX_NUM_TOKENS, + kv_cache_manager=view, + mapping=mapping(), + ) + + +def allocate(manager, request_ids: Sequence[int], qlens: Sequence[int], cached: Sequence[int]): + manager.add_dummy_requests(list(request_ids), token_nums=[c + n for c, n in zip(cached, qlens)]) + + +def prepare(metadata, request_ids, qlens, cached, num_contexts: int) -> None: + metadata.seq_lens = torch.tensor(list(qlens), dtype=torch.int) + metadata.num_contexts = num_contexts + metadata.request_ids = list(request_ids) + metadata.prompt_lens = [ + c + n if row < num_contexts else c for row, (c, n) in enumerate(zip(cached, qlens)) + ] + metadata.kv_cache_params = KVCacheParams(use_cache=True, num_cached_tokens_per_seq=list(cached)) + metadata.prepare() + + +def flat_slots(manager, layer: int, request_id: int, positions: torch.Tensor) -> torch.Tensor: + """The cache slots (page slot id * PAGE + offset) of a request's positions in ``layer``.""" + pages = torch.tensor( + manager.get_batch_cache_indices([request_id], layer_idx=layer)[0], + dtype=torch.int64, + device="cuda", + ) + positions = positions.to(device="cuda", dtype=torch.int64) + return (pages[positions // PAGE] * PAGE + positions % PAGE).to(torch.int32) + + +def write_fp8_rows(manager, layer, request_id, positions, k, v) -> None: + """K / V rows [n, 128] (E4M3) into an FP8 layer's cache at ``positions``.""" + view = manager.get_buffers(layer, "HND") + assert view.dtype == torch.float8_e4m3fn + slots = flat_slots(manager, layer, request_id, positions) + assert fused_write_layer_caches(view[:, 0], view[:, 1], None, slots, k, v, None) + + +def write_nvfp4_rows(manager, metadata, layer, request_id, positions, k, v, idx_k, quant) -> None: + """BF16 K / V rows quantized into the NVFP4 layer's cache, and index-K rows into its FP8 index cache.""" + data = manager.get_buffers(layer, "HND") + scales = manager.get_block_scale_buffers(layer, "HND") + slots = flat_slots(manager, layer, request_id, positions) + assert fused_write_layer_caches_nvfp4( + data[:, 0], + data[:, 1], + scales[:, 0], + scales[:, 1], + metadata.msa_idx_k_cache(layer), + slots, + k, + v, + idx_k, + quant, + ) + + +def index_directions(gen: torch.Generator) -> tuple[torch.Tensor, torch.Tensor]: + """Two orthogonal ±1 index directions (the halves of the 128 dims), E4M3-exact.""" + signs = torch.randint(0, 2, (HEAD_DIM,), generator=gen, device="cuda").float() * 2 - 1 + u0 = torch.zeros(HEAD_DIM, device="cuda") + u1 = torch.zeros(HEAD_DIM, device="cuda") + u0[: HEAD_DIM // 2] = signs[: HEAD_DIM // 2] + u1[HEAD_DIM // 2 :] = signs[HEAD_DIM // 2 :] + return u0, u1 + + +def planted_index_keys(length: int, u0, u1, gen: torch.Generator) -> torch.Tensor: + """Index-K rows [length, 128] (bf16, E4M3-exact) whose per-block maximum score is unambiguous. + + Every block has one peak row at a random position, ``a_b u0 + c_b u1`` with ``a`` and ``c`` two independent + permutations of ``LEVELS``; its other rows are ``LOW (u0 + u1)``. A query along ``u0`` then ranks the blocks by + ``a``, one along ``u1`` by ``c``, every score 1/8 or more apart from the next.""" + blocks = (length + PAGE - 1) // PAGE + assert blocks <= len(LEVELS) + levels = torch.tensor(LEVELS, device="cuda") + a = levels[torch.randperm(len(LEVELS), generator=gen, device="cuda")[:blocks]] + c = levels[torch.randperm(len(LEVELS), generator=gen, device="cuda")[:blocks]] + rows = (LOW * (u0 + u1)).expand(length, HEAD_DIM).clone() + for b in range(blocks): + span = min(PAGE, length - b * PAGE) + peak = b * PAGE + int(torch.randint(0, span, (1,), generator=gen, device="cuda")) + rows[peak] = a[b] * u0 + c[b] * u1 + return rows.to(torch.bfloat16) + + +def nvfp4_quant(*values: torch.Tensor) -> torch.Tensor: + """The (Q, K, V) quantization scales ``fused_write_layer_caches_nvfp4`` takes: each role's amax at the top of the + E2M1 x E4M3 range.""" + quant = torch.ones(3, dtype=torch.float32, device="cuda") + for role, x in enumerate(values, start=1): + quant[role] = (448.0 * 6.0) / x.abs().max().float() + return quant diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/_m3_nvfp4_cache.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/_m3_nvfp4_cache.py new file mode 100644 index 000000000000..bfa03c09da03 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/_m3_nvfp4_cache.py @@ -0,0 +1,105 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""A MiniMax-M3 TP8 sparse-layer decode step over a packed NVFP4 cache, for the attention entries' tests. + +The cache is written by the in-tree scatter (``fused_write_layer_caches_nvfp4``, +which quantizes with ``torch.ops.trtllm.fp4_quantize``), into one +``[pages, 2 (K, V), 1, 128, .]`` pool per role exactly as ``msa_paged_kv`` and +``get_block_scale_buffers`` hand the K and V views over. Pages are shuffled so +a kernel that ignored the block table would read the wrong ones. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from typing import Sequence + +import torch + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_scatter import ( + fused_write_layer_caches_nvfp4, +) + +#: Query heads per rank at TP8 (on one KV head). +NUM_HEADS = 8 +HEAD_DIM = 128 +PAGE = 128 +TOPK = 16 + + +def token_kv_len(seq_len: int, decode_query_len: int, offset: int) -> int: + """The KV extent query token ``offset`` of a request attends, as every decode kernel derives it.""" + return max(seq_len - decode_query_len + offset + 1, 0) + + +def build_step( + seq_lens: Sequence[int], decode_query_len: int, *, identity: bool, seed: int +) -> SimpleNamespace: + """One decode step: E4M3 q [T, 8, 128], the packed NVFP4 K/V views and their scales, a [1, T, 16] + head-major top-k table (the identity table, or a random ascending selection with a -1 tail), the block + table and per-request lengths.""" + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + total_q = batch * decode_query_len + max_blocks = max(1, max((s + PAGE - 1) // PAGE for s in seq_lens)) + num_pages = batch * max_blocks + num_slots = num_pages * PAGE + block_table = ( + torch.randperm(num_pages, device="cuda", generator=g) + .to(torch.int32) + .reshape(batch, max_blocks) + ) + data_pool = torch.zeros(num_pages, 2, 1, PAGE, HEAD_DIM // 2, dtype=torch.uint8, device="cuda") + scale_pool = torch.zeros( + num_pages, 2, 1, PAGE, HEAD_DIM // 16, dtype=torch.uint8, device="cuda" + ) + quant = torch.ones(3, dtype=torch.float32, device="cuda") + values = [] + for role in (0, 1): + x = torch.randn(num_slots, HEAD_DIM, device="cuda", generator=g, dtype=torch.float32).to( + torch.bfloat16 + ) + quant[role + 1] = (448.0 * 6.0) / x.abs().max().float().item() + values.append(x) + assert fused_write_layer_caches_nvfp4( + data_pool[:, 0], + data_pool[:, 1], + scale_pool[:, 0], + scale_pool[:, 1], + None, + torch.arange(num_slots, device="cuda", dtype=torch.int32), + values[0], + values[1], + None, + quant, + ) + q = torch.randn(total_q, NUM_HEADS, HEAD_DIM, device="cuda", generator=g).to( + torch.float8_e4m3fn + ) + topk = torch.full((1, total_q, TOPK), -1, dtype=torch.int32, device="cuda") + for tok in range(total_q): + req, offset = divmod(tok, decode_query_len) + nblocks = (token_kv_len(seq_lens[req], decode_query_len, offset) + PAGE - 1) // PAGE + if identity: + assert nblocks <= TOPK, "an identity step has every request within 16 blocks" + chosen = torch.arange(nblocks, device="cuda") + else: + chosen = torch.randperm(nblocks, device="cuda", generator=g)[:TOPK].sort().values + topk[0, tok, : chosen.numel()] = chosen.to(torch.int32) + return SimpleNamespace( + q=q, + k_paged=data_pool[:, 0], + v_paged=data_pool[:, 1], + k_block_scale=scale_pool[:, 0], + v_block_scale=scale_pool[:, 1], + k_global_scale=(1.0 / quant[1:2]).contiguous(), + v_global_scale=(1.0 / quant[2:3]).contiguous(), + topk=topk, + block_table=block_table, + seq_lens=torch.tensor(list(seq_lens), dtype=torch.int32, device="cuda"), + decode_query_len=decode_query_len, + sm_scale=HEAD_DIM**-0.5, + source=values, + quant=quant, + num_slots=num_slots, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m31_gqamix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m31_gqamix.py new file mode 100644 index 000000000000..6c87a50ca136 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m31_gqamix.py @@ -0,0 +1,189 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GQAMIX: a mixed step's sparse generation rows (MsaDecodeFmha) through ops/gqamix_sparse_decode. + +The copy must be the in-tree kernel module except for the two decode kernels' specialization (no row count, no slice +alignment) and the one import it cannot make relatively, and must write the in-tree kernel's output bit for bit on a +mixed step's [num_ctx:] slices. The shared MsaDecodeFmha hook must change nothing while it is unset. +""" + +import inspect +import os +import subprocess +import sys +import types +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.triton_sparse_decode as in_tree +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( + modeling as m, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops import ( + gqamix_sparse_decode as copy, +) +from tensorrt_llm._torch.attention.backends.fmha.msa_decode import MsaDecodeFmha + +HEADS = 8 # query heads per rank at tp8, one KV head +D, PAGE, TOPK, QLEN, SF_VEC = 128, 128, 16, 4, 16 + +_JIT_OLD = '@triton.jit(do_not_specialize=["decode_query_len"])' +_JIT_NEW = ( + "@triton.jit(\n" + ' do_not_specialize=["decode_query_len", "total_q"],\n' + ' do_not_specialize_on_alignment=["block_table_ptr", "seq_lens"],\n' + ")" +) +_IMPORT_OLD = "from .msa_utils import check_decode_span_shape" +_IMPORT_NEW = ( + "from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import (\n" + " check_decode_span_shape,\n" + ")" +) + + +def _code_lines(text: str) -> list: + return [line for line in text.splitlines() if line.strip()] + + +def test_the_copy_is_the_in_tree_module_but_for_its_specialization() -> None: + """Line for line (blank lines aside: the import sorts into a different block) the in-tree module.""" + ours = Path(copy.__file__).read_text() + theirs = Path(in_tree.__file__).read_text() + assert ( + ours.count(_JIT_NEW) == 2 + and ours.count(_IMPORT_NEW) == 1 + and theirs.count(_IMPORT_OLD) == 1 + ) + ours = ours.replace(_JIT_NEW, _JIT_OLD).replace(_IMPORT_NEW, "") + assert _code_lines(ours) == _code_lines(theirs.replace(_IMPORT_OLD, "")), ( + "ops/gqamix_sparse_decode.py must stay a copy of the in-tree triton_sparse_decode.py: re-copy it" + ) + + +def test_register_sets_the_hook_once_and_only_where_it_exists() -> None: + lib = types.SimpleNamespace(sparse_decode_hook=None) + other_lib = types.SimpleNamespace() # an FMHA library without the hook + manager = types.SimpleNamespace(fmha_libs=[other_lib, lib]) + m._gqamix_register(types.SimpleNamespace(_fmha_manager=manager)) + assert lib.sparse_decode_hook is m._gqamix_decode + assert not hasattr(other_lib, "sparse_decode_hook") + mine = object() + lib.sparse_decode_hook = mine + m._gqamix_register(types.SimpleNamespace(_fmha_manager=manager)) + assert lib.sparse_decode_hook is mine + m._gqamix_register(types.SimpleNamespace()) # a backend without an FMHA manager + + +def test_the_kill_switch_leaves_the_pre_hook_decode() -> None: + """TRTLLM_M31_DISABLE_GQAMIX=1: GQAMIX is off, so no layer registers the hook and MsaDecodeFmha runs the in-tree + decode (whose output the copy matches bit for bit below).""" + target = "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8" + probe = f"import {target}.modeling as m; print(m.GQAMIX)" + env = dict(os.environ, TRTLLM_M31_DISABLE_GQAMIX="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False" + source = inspect.getsource(m.PrefillTarget) + assert "if GQAMIX and sparse:\n _gqamix_register(core.attn[i])" in source + assert source.count("_gqamix_register(") == 1 + + +class _Attn: + is_mla_enable = False + kv_lora_rank = None + v_head_dim = None + head_dim = D + num_heads = HEADS + layer_idx = 3 + q_scaling = 1.0 + + +def _params(attn, num_tokens): + return types.SimpleNamespace( + attn=attn, + num_tokens=num_tokens, + meta=types.SimpleNamespace(kv_cache_manager=object()), + fwd=types.SimpleNamespace(kv_scale_quant_orig=None), + query_input=torch.zeros(num_tokens, HEADS * D), + output=torch.zeros(num_tokens, HEADS * D), + token_offset=2, + input_seq_length=QLEN, + ) + + +@pytest.mark.parametrize("hooked", [False, True], ids=["unset", "registered"]) +def test_the_shared_decode_hook_is_inert_unset(monkeypatch, hooked) -> None: + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import msa_utils + + calls = [] + monkeypatch.setattr(msa_utils, "msa_paged_kv", lambda manager, layer: ("k", "v")) + monkeypatch.setattr( + in_tree, "minimax_m3_sparse_attn_decode", lambda *a, **k: calls.append(("in_tree", a, k)) + ) + attn = _Attn() + fmha = MsaDecodeFmha(attn) + assert fmha.sparse_decode_hook is None + if hooked: + fmha.sparse_decode_hook = lambda *a, **k: calls.append(("hook", a, k)) + indexes = torch.zeros(10, 1, TOPK, dtype=torch.int32) + fmha._run_sparse(_params(attn, QLEN), indexes, "bt", "sl") + assert [c[0] for c in calls] == ["hook" if hooked else "in_tree"] + _, args, kwargs = calls[0] + assert args[1:3] == ("k", "v") and args[4:] == ("bt", "sl") + assert args[3].shape == (1, QLEN, TOPK) + assert kwargs["decode_query_len"] == QLEN and kwargs["sm_scale"] == D**-0.5 + + +def _mixed_step(nreq, seq_len, row_first, seed): + """A mixed step's generation rows: NVFP4 caches, FP8 q, and block-table / length slices past row_first.""" + g = torch.Generator(device="cuda").manual_seed(seed) + nb = seq_len // PAGE + pages = (nreq + row_first) * nb + block_table = torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(-1, nb) + seq_lens = torch.full((nreq + row_first,), seq_len, device="cuda", dtype=torch.int32) + total_q = nreq * QLEN + q = torch.randn(total_q, HEADS, D, device="cuda", generator=g).to(torch.float8_e4m3fn) + kv = [ + torch.randint( + 0, 256, (pages, 1, PAGE, D // 2), device="cuda", generator=g, dtype=torch.uint8 + ) + for _ in "kv" + ] + sc = [ + (torch.rand(pages, 1, PAGE, D // SF_VEC, device="cuda", generator=g) + 0.5).to( + torch.float8_e4m3fn + ) + for _ in "kv" + ] + topk = torch.stack( + [torch.randperm(nb, device="cuda", generator=g)[:TOPK] for _ in range(total_q)] + ) + args = (q, kv[0], kv[1], topk.to(torch.int32).view(total_q, 1, TOPK).permute(1, 0, 2)) + args += (block_table[row_first:], seq_lens[row_first:]) + kwargs = dict( + sm_scale=D**-0.5, + decode_query_len=QLEN, + k_block_scale=sc[0], + v_block_scale=sc[1], + k_global_scale=torch.tensor([0.02], device="cuda"), + v_global_scale=torch.tensor([0.03], device="cuda"), + ) + return args, kwargs, total_q + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA device") +@pytest.mark.parametrize("nreq,row_first", [(1, 0), (1, 1), (3, 2), (5, 3), (8, 1)]) +@pytest.mark.parametrize("seq_len", [16384, 131072]) +def test_the_copy_writes_the_in_tree_output_on_a_mixed_step(nreq, row_first, seq_len) -> None: + args, kwargs, total_q = _mixed_step(nreq, seq_len, row_first, 31 * nreq + row_first) + ref = torch.empty(total_q, HEADS, D, device="cuda", dtype=torch.bfloat16) + out = torch.empty_like(ref) + in_tree.minimax_m3_sparse_attn_decode(*args, output=ref, **kwargs) + copy.minimax_m3_sparse_attn_decode(*args, output=out, **kwargs) + torch.cuda.synchronize() + assert torch.isfinite(ref.float()).all() + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_dense_qk_norm_rope_fp8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_dense_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..bef0db5ce41b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_dense_qk_norm_rope_fp8.py @@ -0,0 +1,144 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7): the TP8 dense layers' FP8 q/k/v (minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8, head-count +generic, at the TP8 per-rank 8 query heads over 1 KV head) carries the bits of the stock path, the BF16 +fused_qk_norm_rope output cast to E4M3, for every norm / RoPE variant: Q and K are rounded to BF16 before E4M3 (K is +stored in the FP8 cache), V is cast.""" + +import pytest +import torch + +import tensorrt_llm # noqa: F401 (registers the trtllm ops) + +assert torch.cuda.is_available(), "the dense FP8 q/k/v op requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp8 target is certified on SM 10.7 only", allow_module_level=True) + +HQ, HK, HV, D, ROT = 8, 1, 1, 128, 64 + + +@pytest.mark.parametrize("tokens", [1, 4, 8, 4096]) +@pytest.mark.parametrize("use_gemma", [False, True], ids=["rmsnorm", "gemma"]) +@pytest.mark.parametrize("is_neox", [True, False], ids=["neox", "interleaved"]) +def test_the_dense_fp8_qkv_is_the_bf16_kernel_cast(tokens, use_gemma, is_neox) -> None: + if not hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8"): + pytest.skip("this build does not carry minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8") + g = torch.Generator(device="cuda").manual_seed(tokens * 4 + int(use_gemma) * 2 + int(is_neox)) + qkv = (torch.randn(tokens, (HQ + HK + HV) * D, device="cuda", generator=g) * 2).to( + torch.bfloat16 + ) + pos = torch.randint(0, 1 << 20, (tokens,), device="cuda", generator=g, dtype=torch.int32) + qw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + kw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + ours = torch.ops.trtllm.minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8( + qkv, HQ, HK, HV, ROT, 1e-6, qw, kw, 5e6, is_neox, pos, use_gemma + ) + stock = qkv.clone() + torch.ops.trtllm.fused_qk_norm_rope( + stock, HQ, HK, HV, D, ROT, 1e-6, qw, kw, 5e6, is_neox, pos, + 1.0, 0.0, 0.0, 1.0, True, use_gemma, False, 0, 0, + ) # fmt: skip + assert torch.equal(ours.view(torch.uint8), stock.to(torch.float8_e4m3fn).view(torch.uint8)) + + +@pytest.mark.parametrize("tokens,live", [(1, 1), (4, 4), (8, 5), (64, 64)]) +@pytest.mark.parametrize("tokens_per_block", [32, 128]) +@pytest.mark.parametrize("is_neox", [True, False], ids=["neox", "interleaved"]) +def test_the_fused_kv_insert_stores_the_bytes_of_write_kv_slots_and_the_compact_q( + tokens, live, tokens_per_block, is_neox +) -> None: + if not hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv"): + pytest.skip("this build does not carry minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv") + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + write_kv_slots, + ) + + g = torch.Generator(device="cuda").manual_seed(tokens * 1000 + tokens_per_block + int(is_neox)) + qkv = (torch.randn(tokens, (HQ + HK + HV) * D, device="cuda", generator=g) * 2).to( + torch.bfloat16 + ) + pos = torch.randint(0, 1 << 20, (tokens,), device="cuda", generator=g, dtype=torch.int32) + qw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + kw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + pages, layers, layer = 16, 3, 1 + # The layer's HND view of a multi-layer pool, strided as get_buffers returns it. + raw = torch.randint( + 0, + 256, + (pages, layers, 2, HK * tokens_per_block * D), + device="cuda", + generator=g, + dtype=torch.uint8, + ) + slots = torch.randperm(pages * tokens_per_block, device="cuda", generator=g)[:tokens].to( + torch.int32 + ) + slots[live:] = -1 # padded rows own no slot + + def view(t): + return t.view(torch.float8_e4m3fn)[:, layer].view(pages, 2, HK, tokens_per_block, D) + + ours_raw = raw.clone() + args = (qkv, HQ, HK, HV, ROT, 1e-6, qw, kw, 5e6, is_neox, pos, False) + q_compact = torch.empty(tokens, HQ, D, device="cuda", dtype=torch.float8_e4m3fn) + ours = torch.ops.trtllm.minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv( + *args, view(ours_raw), slots, live, q_compact + ) + plain = torch.ops.trtllm.minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8(*args) + assert torch.equal(ours.view(torch.uint8), plain.view(torch.uint8)) + assert torch.equal( + q_compact.view(tokens, HQ * D).view(torch.uint8), plain[:, : HQ * D].view(torch.uint8) + ) + + ref_raw = raw.clone() + k = plain[:, HQ * D : (HQ + HK) * D].reshape(tokens, HK, D) + v = plain[:, (HQ + HK) * D :].reshape(tokens, HV, D) + ref = view(ref_raw) + write_kv_slots(ref[:, 0], slots, k, live, layout="HND") + write_kv_slots(ref[:, 1], slots, v, live, layout="HND") + assert torch.equal(ours_raw, ref_raw) + + +def test_the_fused_kv_insert_skips_a_negative_slot() -> None: + """A live row whose slot is the -1 sentinel stores nothing (no write before the pool row).""" + if not hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv"): + pytest.skip("this build does not carry minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv") + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.paged_cache import ( + write_kv_slots, + ) + + tokens, tokens_per_block, pages = 4, 32, 8 + g = torch.Generator(device="cuda").manual_seed(7) + qkv = (torch.randn(tokens, (HQ + HK + HV) * D, device="cuda", generator=g) * 2).to( + torch.bfloat16 + ) + pos = torch.randint(0, 1 << 20, (tokens,), device="cuda", generator=g, dtype=torch.int32) + qw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + kw = (1 + 0.1 * torch.randn(D, device="cuda", generator=g)).to(torch.bfloat16) + raw = torch.randint( + 0, + 256, + (pages, 3, 2, HK * tokens_per_block * D), + device="cuda", + generator=g, + dtype=torch.uint8, + ) + slots = torch.tensor([5, -1, 40, 77], device="cuda", dtype=torch.int32) + + def view(t): + return t.view(torch.float8_e4m3fn)[:, 1].view(pages, 2, HK, tokens_per_block, D) + + ours_raw = raw.clone() + args = (qkv, HQ, HK, HV, ROT, 1e-6, qw, kw, 5e6, True, pos, False) + torch.ops.trtllm.minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8_kv( + *args, view(ours_raw), slots, tokens + ) + plain = torch.ops.trtllm.minimax_m3_sm107_tp2_dense_qk_norm_rope_fp8(*args) + keep = torch.tensor([0, 2, 3], device="cuda") + ref_raw = raw.clone() + ref = view(ref_raw) + k = plain[:, HQ * D : (HQ + HK) * D].reshape(tokens, HK, D)[keep] + v = plain[:, (HQ + HK) * D :].reshape(tokens, HV, D)[keep] + write_kv_slots(ref[:, 0], slots[keep], k, 3, layout="HND") + write_kv_slots(ref[:, 1], slots[keep], v, 3, layout="HND") + assert torch.equal(ours_raw, ref_raw) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_draft_attention.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_draft_attention.py new file mode 100644 index 000000000000..6249b4e7b4fb --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_draft_attention.py @@ -0,0 +1,148 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_draft_attention entry: the Eagle3 drafter layer's attention against its float64 reference. + +Every cell runs on the runtime's own objects (``_m3_attn_runtime_tp8``): the target's hybrid NVFP4 +``MiniMaxM3KVCacheManagerV2`` with its one-model Eagle3 draft layer, the requests allocated in it, drafter +``TrtllmAttentionMetadata`` over the manager's FP8 draft view (the manager the runtime's draft swap installs) filled +by its ``prepare()``, and the drafter's ``TrtllmAttention`` built as ``M3Eagle3Draft`` builds it (fused NeoX RoPE at +the drafter checkpoint's theta, FP8 KV). Earlier rows go into the draft layer's cache through the in-tree scatter at +the slots the manager assigned; the call rotates and appends the step's own. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_draft_attention import ( # noqa: E501 + m3_draft_attention as op, +) + +__extra_import_path__ = ["../../../..", "."] +import _m3_attn_runtime_tp8 as rt # noqa: E402 — needs the path declared above +from _validating import ULP, assert_within_ulp, certifying # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_draft_attention requires a CUDA device" + +# Band, in BF16 ulps (2^-8) of each token's [heads x 128] row's largest magnitude (element) and of the relative RMS. +# The reference reads the same FP8 cache rows the call attends, so what is left is the FP8 FMHA's own arithmetic: +# it takes the rotated Q in E4M3 (rounding the reference's Q to E4M3 takes the RMS gap from ~9 to ~7 ulps) and rounds +# the softmax probabilities before the PV product. Measured on GB300 over these cells (max element / RMS): 14.8 / 8.7 +# (4 x 4 rows), 13.3 / 7.4 (8 x 1), 19.1 / 7.7 (512 context rows). The band leaves about 1.5x; a reference shifted +# by one row is far outside it. +_ELEMENT_ULP = 30.0 +_RMS_ULP = 12.0 + +# Cached tokens per request of each cell; the new rows per request come from the cell. +_CACHED = { + (2, 4): (200, 1000), + (2, 1): (127, 2047), + (1, None): (0,), +} +#: Rows past the live ones in the fused QKV, as a padded step hands them over. +_PAD = 3 + + +def _ulp_stats(out: torch.Tensor, ref: torch.Tensor) -> tuple[float, float]: + o, r = out.float(), ref.float() + scale = r.abs().amax(dim=1, keepdim=True).clamp_min(1e-9) + element = ((o - r).abs() / scale).max().item() / ULP[torch.bfloat16] + rms = ((o - r).pow(2).mean().sqrt() / r.pow(2).mean().sqrt()).item() / ULP[torch.bfloat16] + return element, rms + + +def _rope(x: torch.Tensor, positions: torch.Tensor) -> torch.Tensor: + """NeoX RoPE over the whole head at the drafter's theta, float64.""" + half = rt.HEAD_DIM // 2 + inv_freq = rt.DRAFT_ROPE_THETA ** ( + -torch.arange(0, rt.HEAD_DIM, 2, dtype=torch.float64, device=x.device) / rt.HEAD_DIM + ) + angles = positions.to(torch.float64)[:, None] * inv_freq[None, :] + x = x.to(torch.float64) + x1, x2 = x[..., :half], x[..., half:] + return torch.cat( + [x1 * angles.cos() - x2 * angles.sin(), x2 * angles.cos() + x1 * angles.sin()], -1 + ) + + +# Non-paged context FMHA on SM107 does not apply RoPE (this fixture's path; the served drafter runs the paged context +# FMHA, which rotates): the context cell is a strict expected failure until the runtime rotates there too. +_UNROTATED_CONTEXT = pytest.mark.xfail( + strict=True, + reason="non-paged context FMHA on SM107 does not apply RoPE (shared runtime)", +) +_CELLS = [ + pytest.param(c, marks=_UNROTATED_CONTEXT) if c.spec.get("context") else c for c in op.CELLS +] + + +@pytest.mark.parametrize("cell", _CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + context = spec.get("context") + cached = _CACHED[spec["reqs"], spec.get("qlen")] + qlens = (context,) if context else (spec["qlen"],) * spec["reqs"] + assert len(cached) == spec["reqs"] + contexts = 1 if context else 0 + rids = list(range(spec["reqs"])) + gen = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + manager = rt.make_manager(draft=True) + try: + backend = rt.make_draft_backend() + metadata = rt.make_draft_metadata(manager) + rt.allocate(manager, rids, qlens, cached) + for rid, c in zip(rids, cached): + if c: + hist = torch.randn(2, c, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda") + hist = hist.to(torch.float8_e4m3fn) + rt.write_fp8_rows(manager, rt.DRAFT, rid, torch.arange(c), hist[0], hist[1]) + rt.prepare(metadata, rids, qlens, cached, contexts) + total = sum(qlens) + assert int(metadata.num_tokens) == total + width = (rt.HEADS + 2 * rt.KV_HEADS) * rt.HEAD_DIM + qkv = torch.randn(total + _PAD, width, generator=gen, device="cuda").to(torch.bfloat16) + + certifying(op) + out = op(backend, qkv, metadata) + ref = op.reference(backend, qkv, metadata) + assert out.shape == (total, rt.HEADS * rt.HEAD_DIM) and out.dtype == torch.bfloat16 + assert torch.isfinite(out.float()).all() + element, rms = _ulp_stats(out, ref) + print(f"\n[{cell.why}] max element {element:.2f} ulp, rms {rms:.3f} ulp") + group = rt.HEADS // rt.KV_HEADS + for h in range(rt.KV_HEADS): + cols = slice(h * group * rt.HEAD_DIM, (h + 1) * group * rt.HEAD_DIM) + element, rms = _ulp_stats(out[:, cols], ref[:, cols]) + print(f"[{cell.why}] KV head {h}: max element {element:.2f} ulp, rms {rms:.3f} ulp") + assert_within_ulp(out, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + # The call's append: each live row's rotated K and its V at the row's position, E4M3 (unit KV scale), to + # within one E4M3 step of the float64 rotation (the kernel rotates in FP32). + pool = manager.get_buffers(rt.DRAFT, "HND") + q_size = rt.HEADS * rt.HEAD_DIM + first = 0 + for rid, c, n in zip(rids, cached, qlens): + positions = torch.arange(c, c + n, device="cuda") + slots = rt.flat_slots(manager, rt.DRAFT, rid, positions).long() + rows = qkv[first : first + n] + for h in range(rt.KV_HEADS): + k_cols = q_size + h * rt.HEAD_DIM + v_cols = q_size + (rt.KV_HEADS + h) * rt.HEAD_DIM + want_k = _rope(rows[:, k_cols : k_cols + rt.HEAD_DIM].float(), positions) + want_v = rows[:, v_cols : v_cols + rt.HEAD_DIM].to(torch.float8_e4m3fn) + got_k = pool[slots // rt.PAGE, 0, h, slots % rt.PAGE].to(torch.float64) + got_v = pool[slots // rt.PAGE, 1, h, slots % rt.PAGE] + step = want_k.abs().clamp_min(2.0**-6) * 2.0**-3 + assert bool(((got_k - want_k).abs() <= step).all()), ( + f"K head {h} is appended rotated, in E4M3" + ) + assert torch.equal(got_v.view(torch.uint8), want_v.view(torch.uint8)), f"V head {h}" + first += n + + # The band discriminates: a reference whose rows are shifted by one token is outside it. + shifted = op.reference(backend, qkv.roll(1, 0), metadata) + with pytest.raises(AssertionError): + assert_within_ulp(out, shifted, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + finally: + manager.shutdown() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_drafter_fullctx.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_drafter_fullctx.py new file mode 100644 index 000000000000..22731402afd7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_drafter_fullctx.py @@ -0,0 +1,271 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7) for kernels/m3_drafter_attn: the Eagle3 drafter layer's generation pass on the full-context +PrimTS decode, driven through ``M3DrafterFullCtxFmha.run_generation`` (in-tree preprocess with the step's FP8 K/V +append, the K / V sub-page views of the drafter cache, the pinned plan) at every recipe the target pins: the 4-row +first draft pass and the 1-row later passes of 1-2 requests. + +Lengths sweep the KV tile edges each query row in turn ends on (the rows of a Q CTA with no key in its trailing +splits), with mixed per-request lengths; every call is held to the fp32 reference over the cache after the append +(E4M3 query, unit KV scales, no RoPE) within the FP8-KV band, finite, for both split-KV reducers. +""" + +import math +import weakref +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_drafter_attn as drafter, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_fullctx_decode as fullctx, +) + +assert torch.cuda.is_available(), "m3_drafter_attn requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp8 target is certified on SM 10.7 only", allow_module_level=True) + +H, KVH, D, PAGE = fullctx.HEADS, fullctx.KV_HEADS, fullctx.HEAD_DIM, 128 +MAX_LEN = 1 << 20 +FP8_KV_QUANT_MODE = 128 +FP8_TOL = 6e-2 +# Persistent like the FMHA manager's workspace and the spec-decoding metadata of a real step: the in-tree preprocess +# kernels must not see these move between calls. +_PERSISTENT = {} + + +def _persistent(name: str, make): + if name not in _PERSISTENT: + _PERSISTENT[name] = make() + return _PERSISTENT[name] + + +def _edge_lens(plan, qlen: int): + tile, insts, splits = plan.tile_kv, plan.insts_kv, int(plan.recipe["splits"]) + grow = splits * insts + tiles = sorted({1, 2, 8, 9, grow - 1, grow, grow + 1, grow + 2 * insts, 2 * grow}) + return [k * tile + d for k in tiles for d in range(-1, qlen)] + + +def _library(qlen: int, nreq: int, own_reducer: bool): + lib = drafter.M3DrafterFullCtxFmha.__new__(drafter.M3DrafterFullCtxFmha) + lib._recipes = {(qlen, nreq): modeling.FULLCTX_DECODE_RECIPES[(qlen, nreq)]} + lib._own_reducer = own_reducer + lib._max_kv_len = MAX_LEN + lib._multi_processor_count = torch.cuda.get_device_properties(0).multi_processor_count + lib._on_taken = lambda rows: None + # The in-tree preprocess pass (its RoPE-free cache append); the one-launch preprocess has its own bitwise test. + lib._fused_rope_mode = None + lib._pdl = False + lib._on_rope_taken = None + return lib + + +def _attn(qlen: int): + return SimpleNamespace( + num_heads=H, + num_kv_heads=KVH, + head_dim=D, + q_scaling=1.0, + attention_chunk_size=None, + quant_mode=FP8_KV_QUANT_MODE, + local_layer_idx=0, + position_embedding_type=0, + rotary_inv_freq=None, + rotary_cos_sin=None, + predicted_tokens_per_seq=qlen, + rope_params=SimpleNamespace( + dim=0, theta=10000.0, scale_type=0, scale=1.0, max_positions=1024 + ), + ) + + +def _run(lib, qlen: int, kv_lens, seed: int, spec: bool = True): + """One drafter generation pass over a fresh FP8 cache; returns (output, fp32 reference). ``spec`` passes the + spec-decoding lengths/positions (a multi-token generation step); without them a multi-row pass is the drafter's + plain generation step of input_seq_length rows, as it runs in-model.""" + dev = torch.device("cuda") + nreq = len(kv_lens) + g = torch.Generator(device=dev).manual_seed(seed) + pages = math.ceil(max(kv_lens) / PAGE) + nblocks = nreq * pages + 2 + pool = torch.randn(nblocks, 2, KVH, PAGE, D, device=dev, generator=g).clamp_(-4, 4) + perm = torch.randperm(nblocks, device=dev, generator=g)[: nreq * pages].view(nreq, pages) + for r, n in enumerate( + kv_lens + ): # shift the tail third of V, so a missed trailing split fails the band + start = 2 * n // 3 + for pg in range(start // PAGE, math.ceil(n / PAGE)): + pool[int(perm[r, pg]), 1, :, (start % PAGE if pg == start // PAGE else 0) :] += 1.0 + pool = pool.to(torch.float8_e4m3fn) + offsets = torch.zeros(1, nreq, 2, pages, dtype=torch.int32, device=dev) + offsets[0, :, 0] = 2 * perm + offsets[0, :, 1] = 2 * perm + 1 + pointers = torch.tensor([[pool.data_ptr(), 0]], dtype=torch.int64) + ntok = nreq * qlen + qkv = torch.randn(ntok, (H + 2 * KVH) * D, dtype=torch.bfloat16, device=dev, generator=g) + seq = torch.tensor(kv_lens, dtype=torch.int32, device=dev) + one = torch.ones(1, dtype=torch.float32, device=dev) + multi = qlen > 1 and spec + out = torch.zeros(ntok, H * D, dtype=torch.bfloat16, device=dev) + params = SimpleNamespace( + qkv_input=qkv, + query_input=None, + attn=_attn(qlen), + meta=SimpleNamespace( + kv_cache_block_offsets=offsets, + host_kv_cache_pool_pointers=pointers, + host_kv_cache_pool_mapping=torch.zeros(1, 2, dtype=torch.int32), + ), + fwd=SimpleNamespace( + kv_scale_orig_quant=one, + kv_scale_quant_orig=one, + out_scale=None, + mrope_position_deltas=None, + ), + workspace=_persistent( + "workspace", lambda: torch.empty(512 << 20, dtype=torch.uint8, device=dev) + ), + sequence_lengths=seq, + spec_decoding_generation_lengths=( + _persistent( + f"gen_lengths{qlen}x{nreq}", + lambda: torch.full((nreq,), qlen, dtype=torch.int32, device=dev), + ) + if multi + else None + ), + spec_decoding_position_offsets=( + _persistent( + f"position_offsets{qlen}x{nreq}", + lambda: ( + torch.arange(qlen, dtype=torch.int32, device=dev).repeat(nreq, 1).contiguous() + ), + ) + if multi + else None + ), + seq_offset=0, + tokens_per_block=PAGE, + max_attention_window_size=MAX_LEN, + cyclic_attention_window_size=MAX_LEN, + num_tokens=ntok, + batch_size=nreq, + input_seq_length=qlen, + max_past_kv_length=max(kv_lens), + total_num_blocks=nblocks, + kv_factor=2, + is_cross=False, + output=out, + ) + lib.run_generation(params) + torch.cuda.synchronize() + # fp32 reference over the cache after the append: E4M3 query, unit scales, causal over the step's rows. + q = qkv[:, : H * D].float().view(nreq, qlen, H, D).to(torch.float8_e4m3fn).float() + flat = pool.float() + ref = torch.empty(nreq, qlen, H, D, device=dev) + for r, n in enumerate(kv_lens): + blocks = perm[r, : math.ceil(n / PAGE)] + k = flat[blocks, 0].permute(1, 0, 2, 3).reshape(KVH, -1, D)[:, :n] + v = flat[blocks, 1].permute(1, 0, 2, 3).reshape(KVH, -1, D)[:, :n] + k = k.repeat_interleave(H // KVH, dim=0) + v = v.repeat_interleave(H // KVH, dim=0) + s = torch.einsum("qhd,hkd->hqk", q[r], k) * D**-0.5 + visible = ( + torch.arange(n, device=dev)[None, :] + < (n - qlen + 1 + torch.arange(qlen, device=dev))[:, None] + ) + s = s.masked_fill(~visible[None], float("-inf")) + ref[r] = torch.einsum("hqk,hkd->qhd", torch.softmax(s, dim=-1), v) + return out.float().view(nreq, qlen, H, D), ref + + +@pytest.mark.parametrize("qlen,nreq", sorted(modeling.FULLCTX_DECODE_RECIPES)) +def test_the_drafter_pass_at_tile_edges(qlen, nreq) -> None: + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + recipe = modeling.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + reducers = (True, False) if fullctx.has_own_reducer() else (False,) + for own in reducers: + fullctx.prebuild( + torch.device("cuda"), {(qlen, nreq): recipe}, MAX_LEN, own, get_env_enable_pdl() + ) + lens = _edge_lens(fullctx.plan_for(torch.device("cuda"), qlen, nreq, False), qlen) + lens = [n for n in lens if n > qlen] + batches = [[n] for n in lens] if nreq == 1 else [[a, b] for a, b in zip(lens, lens[::-1])] + batches.append([1300, 1281][:nreq]) + failures = [] + for i, kv_lens in enumerate(batches): + # The first draft pass runs in-model as a plain multi-row generation step; every other batch also takes the + # spec-decoding form. + spec = qlen > 1 and i % 2 == 1 + for own in reducers: + got, ref = _run(_library(qlen, nreq, own), qlen, kv_lens, sum(kv_lens), spec=spec) + if not bool(torch.isfinite(got).all()) or not torch.allclose( + got, ref, rtol=FP8_TOL, atol=FP8_TOL + ): + failures.append((kv_lens, "bf16-merge" if own else "primts-reducer")) + assert not failures, failures + + +def _drafter_model(scale: float): + """A drafter whose one attention module holds a TrtllmAttention backend as ``attn`` (the backend is no module).""" + from tensorrt_llm._torch.attention.backends.trtllm import TrtllmAttention + + backend = TrtllmAttention.__new__(TrtllmAttention) + backend.kv_cache_scaling_factor = torch.full((1,), scale, device="cuda") + backend.is_mla_enable, backend.kv_lora_rank, backend.v_head_dim, backend.head_dim = ( + False, + None, + None, + D, + ) + backend._fmha_manager = SimpleNamespace(fmha_libs=[]) + model = torch.nn.Module() + model.weight = torch.nn.Parameter(torch.zeros(1, device="cuda")) + model.self_attn = torch.nn.Module() + model.self_attn.attn = backend + return model, backend + + +def test_install_declines_a_scaled_kv_cache() -> None: + """At unit KV scales the arithmetic is the stock decode's; a scaled cache keeps the stock path.""" + reasons = [] + model, layer = _drafter_model(0.5) + assert drafter.install(model, {}, MAX_LEN, False, False, None, reasons.append, None) == 0 + assert layer._fmha_manager.fmha_libs == [] and reasons == ["a scaled FP8 KV cache"] + model, layer = _drafter_model(1.0) + assert drafter.install(model, {}, MAX_LEN, False, False, None, reasons.append, None) == 1 + assert isinstance(layer._fmha_manager.fmha_libs[0], drafter.M3DrafterFullCtxFmha) + + +def test_a_cache_bound_past_the_plan_capacity_keeps_the_stock_path() -> None: + """The plans are compiled for ``max_kv_len`` of K/V: a step whose cache length bound is longer falls through.""" + + from tensorrt_llm._torch.attention.backends.fmha.interface import FmhaPhase + + class _Backend: + num_heads, num_kv_heads, head_dim, quant_mode = H, KVH, D, FP8_KV_QUANT_MODE + kv_scale_orig_quant = kv_scale_quant_orig = None + + backend = _Backend() + backend.kv_scale_orig_quant, backend.kv_scale_quant_orig = object(), object() + reasons = [] + lib = _library(1, 1, False) + lib._on_uncovered = reasons.append + lib._attn_ref = weakref.ref(backend) + fwd = SimpleNamespace( + attention_input_type=None, + kv_scale_orig_quant=backend.kv_scale_orig_quant, + kv_scale_quant_orig=backend.kv_scale_quant_orig, + ) + meta = SimpleNamespace(num_contexts=0, num_generations=1, max_seq_len=MAX_LEN + 8) + q = torch.empty(1, H * D, device="cuda") + assert not lib._is_supported(q, None, None, meta, fwd, phase=FmhaPhase.GENERATION) + assert reasons == ["the cache length bound is beyond the plans' K/V capacity"] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_drafter_rope_fp8_kv.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_drafter_rope_fp8_kv.py new file mode 100644 index 000000000000..19747e3cb4a7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_drafter_rope_fp8_kv.py @@ -0,0 +1,371 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7) for minimax_m3_sm107tp8_drafter_rope_fp8_kv, the TP8 drafter layer's one-launch generation +preprocess: bitwise against the in-tree ``thop.trtllm_gen_generation_preprocess`` followed by the torch E4M3 cast +of its BF16 query, on the query bytes and on every byte of the paged cache, with GPT-NeoX RoPE from a cos/sin table +(full and partial rotary width), multi-token (spec-decoding positions) and plain passes, one and two requests, and +query values past the E4M3 range. The library's fused pass is also held bitwise to its preprocess pass end to end. +""" + +import math +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_drafter_attn as drafter, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_fullctx_decode as fullctx, +) +from tensorrt_llm._torch.attention.backends.fmha.utils import ( + get_attention_chunk_size, + get_bmm1_scale, +) +from tensorrt_llm.bindings.internal import thop + +assert torch.cuda.is_available(), "the drafter preprocess op requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp8 target is certified on SM 10.7 only", allow_module_level=True) +if not hasattr(torch.ops.trtllm, "minimax_m3_sm107tp8_drafter_rope_fp8_kv"): + pytest.skip("minimax_m3_sm107tp8_drafter_rope_fp8_kv is not built", allow_module_level=True) + +H, KVH, D, PAGE = fullctx.HEADS, fullctx.KV_HEADS, fullctx.HEAD_DIM, 128 +MAX_LEN = 1 << 20 +FP8_KV_QUANT_MODE = 128 +ROPE_GPT_NEOX = 2 +_PERSISTENT = {} + + +def _persistent(name, make): + if name not in _PERSISTENT: + _PERSISTENT[name] = make() + return _PERSISTENT[name] + + +def _case(qlen, kv_lens, rotary_dim, multi, seed): + dev = torch.device("cuda") + nreq = len(kv_lens) + g = torch.Generator(device=dev).manual_seed(seed) + pages = math.ceil(max(kv_lens) / PAGE) + nblocks = nreq * pages + 2 + pool = (torch.randn(nblocks, 2, KVH, PAGE, D, device=dev, generator=g) * 2).to( + torch.float8_e4m3fn + ) + perm = torch.randperm(nblocks, device=dev, generator=g)[: nreq * pages].view(nreq, pages) + offsets = torch.zeros(1, nreq, 2, pages, dtype=torch.int32, device=dev) + offsets[0, :, 0] = 2 * perm + offsets[0, :, 1] = 2 * perm + 1 + ntok = nreq * qlen + qkv = torch.randn(ntok, (H + 2 * KVH) * D, device=dev, generator=g) * 4 + # A few query / key values near and past the E4M3 range (both casts saturate at 448). + hot = torch.rand(qkv.shape, device=dev, generator=g) < 0.01 + qkv = torch.where( + hot, qkv.sign() * (430 + 80 * torch.rand(qkv.shape, device=dev, generator=g)), qkv + ) + qkv = qkv.to(torch.bfloat16).contiguous() + max_pos = max(kv_lens) + 64 + cos_sin = torch.randn(max_pos, rotary_dim // 2, 2, device=dev, generator=g).float().contiguous() + attn = SimpleNamespace( + num_heads=H, + num_kv_heads=KVH, + head_dim=D, + q_scaling=1.0, + attention_chunk_size=None, + quant_mode=FP8_KV_QUANT_MODE, + local_layer_idx=0, + position_embedding_type=ROPE_GPT_NEOX, + rotary_inv_freq=None, + rotary_cos_sin=cos_sin, + predicted_tokens_per_seq=qlen, + rope_params=SimpleNamespace( + dim=rotary_dim, theta=10000.0, scale_type=0, scale=1.0, max_positions=max_pos + ), + ) + one = torch.ones(1, dtype=torch.float32, device=dev) + spec_offsets = None + if multi: + spec_offsets = _persistent( + f"pos{qlen}x{nreq}", + lambda: torch.arange(qlen, dtype=torch.int32, device=dev).repeat(nreq, 1).contiguous(), + ) + # Spec positions are not the plain row index: shift them per request. + spec_offsets.copy_( + torch.arange(qlen, dtype=torch.int32, device=dev).repeat(nreq, 1) + + torch.arange(nreq, dtype=torch.int32, device=dev)[:, None] + ) + params = SimpleNamespace( + qkv_input=qkv, + query_input=None, + attn=attn, + meta=SimpleNamespace( + kv_cache_block_offsets=offsets, + host_kv_cache_pool_pointers=torch.tensor([[pool.data_ptr(), 0]], dtype=torch.int64), + host_kv_cache_pool_mapping=torch.zeros(1, 2, dtype=torch.int32), + ), + fwd=SimpleNamespace( + kv_scale_orig_quant=one, + kv_scale_quant_orig=one, + out_scale=None, + mrope_position_deltas=None, + ), + workspace=_persistent( + "workspace", lambda: torch.empty(512 << 20, dtype=torch.uint8, device=dev) + ), + sequence_lengths=torch.tensor(kv_lens, dtype=torch.int32, device=dev), + spec_decoding_generation_lengths=( + _persistent( + f"len{qlen}x{nreq}", + lambda: torch.full((nreq,), qlen, dtype=torch.int32, device=dev), + ) + if multi + else None + ), + spec_decoding_position_offsets=spec_offsets, + seq_offset=0, + tokens_per_block=PAGE, + max_attention_window_size=MAX_LEN, + cyclic_attention_window_size=MAX_LEN, + num_tokens=ntok, + batch_size=nreq, + input_seq_length=qlen, + max_past_kv_length=max(kv_lens), + total_num_blocks=nblocks, + kv_factor=2, + is_cross=False, + output=torch.zeros(ntok, H * D, dtype=torch.bfloat16, device=dev), + ) + return params, pool + + +def _preprocess(params, mpc): + """run_generation's in-tree preprocess call and the E4M3 cast of its query.""" + attn, meta, fwd, rope = params.attn, params.meta, params.fwd, params.attn.rope_params + q_processed = thop.trtllm_gen_generation_preprocess( + params.qkv_input, + params.workspace, + params.sequence_lengths, + params.spec_decoding_generation_lengths, + params.spec_decoding_position_offsets, + meta.kv_cache_block_offsets, + meta.host_kv_cache_pool_pointers, + meta.host_kv_cache_pool_mapping, + fwd.kv_scale_orig_quant, + fwd.kv_scale_quant_orig, + fwd.out_scale, + attn.rotary_inv_freq, + attn.rotary_cos_sin, + fwd.mrope_position_deltas, + attn.local_layer_idx, + params.seq_offset, + attn.num_heads, + attn.num_kv_heads, + attn.head_dim, + params.tokens_per_block, + attn.quant_mode, + params.max_attention_window_size, + params.cyclic_attention_window_size, + params.num_tokens, + params.batch_size, + params.input_seq_length, + params.max_past_kv_length, + rope.dim, + rope.theta, + int(rope.scale_type), + rope.scale, + rope.max_positions, + attn.position_embedding_type, + get_bmm1_scale(attn), + 1.0, + False, + attn.predicted_tokens_per_seq, + get_attention_chunk_size(attn), + mpc, + params.total_num_blocks, + params.kv_factor, + True, + params.is_cross, + )[0] + rows = params.num_tokens * attn.num_heads * attn.head_dim + return ( + q_processed.flatten()[:rows] + .to(torch.float8_e4m3fn) + .view(params.num_tokens, attn.num_heads, attn.head_dim) + ) + + +def _op(params, pool, mode): + attn = params.attn + nreq, qlen = params.batch_size, params.input_seq_length + kv_pages = pool.view(-1, KVH, PAGE, D) + tables = params.meta.kv_cache_block_offsets[0, :nreq] + q = torch.empty( + params.num_tokens, + attn.num_heads, + attn.head_dim, + dtype=torch.float8_e4m3fn, + device=pool.device, + ) + multi = params.spec_decoding_generation_lengths is not None and qlen > 1 + torch.ops.trtllm.minimax_m3_sm107tp8_drafter_rope_fp8_kv( + params.qkv_input, + attn.rotary_cos_sin, + params.sequence_lengths, + params.spec_decoding_position_offsets if multi else None, + kv_pages, + tables, + q, + qlen, + attn.num_heads, + attn.num_kv_heads, + attn.rope_params.dim, + PAGE, + mode, + True, + ) + return q + + +# One-request multi-token passes keep the in-tree preprocess (it reads an unbuilt token map), so the spec cases hold +# two requests. +CASES = [ + (qlen, kv_lens, rotary, multi) + for qlen, multi in ((1, False), (4, True), (4, False)) + for kv_lens in (((4, 130) if multi else (4,)) if qlen > 1 else (130,), (8191, 131073)) + for rotary in (128, 64) +] + + +def _bytes(t): + return t.contiguous().view(torch.uint8) + + +def _matches(case, seed): + """Per rope mode: (query bytes equal, cache bytes equal) against the in-tree preprocess + cast.""" + qlen, kv_lens, rotary, multi = case + kv_lens = [max(n, qlen) for n in kv_lens] + mpc = torch.cuda.get_device_properties(0).multi_processor_count + params, pool = _case(qlen, kv_lens, rotary, multi, seed) + start = pool.clone() + q_ref = _preprocess(params, mpc) + torch.cuda.synchronize() + pool_ref = pool.clone() + out = {} + for mode in (0, 1, 2): + pool.copy_(start) + q = _op(params, pool, mode) + torch.cuda.synchronize() + out[mode] = ( + bool(torch.equal(_bytes(q), _bytes(q_ref))), + bool(torch.equal(_bytes(pool), _bytes(pool_ref))), + ) + return out + + +@pytest.mark.parametrize( + "case", + CASES, + ids=lambda c: f"q{c[0]}-{'x'.join(map(str, c[1]))}-r{c[2]}-{'spec' if c[3] else 'plain'}", +) +def test_the_pinned_rope_mode_is_bitwise(case) -> None: + result = _matches(case, seed=7) + print(f"[rope modes] {case}: {result}", flush=True) + assert result[modeling.DRAFTER_ROPE_MODE] == (True, True), result + + +def test_the_fused_pass_matches_the_preprocess_pass() -> None: + mpc = torch.cuda.get_device_properties(0).multi_processor_count + for qlen, nreq in sorted(modeling.FULLCTX_DECODE_RECIPES): + for multi in (False, True) if qlen > 1 else (False,): + kv_lens = [70001 + 517 * r for r in range(nreq)] + outs = [] + for fused in (False, True): + params, pool = _case(qlen, kv_lens, 128, multi, seed=11) + lib = drafter.M3DrafterFullCtxFmha.__new__(drafter.M3DrafterFullCtxFmha) + lib._recipes = {(qlen, nreq): modeling.FULLCTX_DECODE_RECIPES[(qlen, nreq)]} + lib._own_reducer = True + lib._max_kv_len = MAX_LEN + lib._multi_processor_count = mpc + lib._fused_rope_mode = modeling.DRAFTER_ROPE_MODE if fused else None + lib._pdl = True + taken = [] + lib._on_rope_taken = lambda rows, what: taken.append(what) + lib._on_taken = lambda rows: None + fullctx.prebuild(torch.device("cuda"), lib._recipes, MAX_LEN, True, True) + lib.run_generation(params) + torch.cuda.synchronize() + assert bool(taken) == (fused and not (multi and nreq == 1)) + outs.append((params.output.clone(), pool.clone())) + assert torch.equal(_bytes(outs[0][1]), _bytes(outs[1][1])), (qlen, nreq, multi, "cache") + assert torch.equal(_bytes(outs[0][0]), _bytes(outs[1][0])), ( + qlen, + nreq, + multi, + "output", + ) + + +def _round_bf16(x): + u = x.view(torch.int32).to(torch.int64) & 0xFFFFFFFF + u = (u + 0x7FFF + ((u >> 16) & 1)) & 0xFFFF0000 + return u.to(torch.int64).to(torch.int32).view(torch.float32) + + +def _discriminating(n, seed): + """(cos, sin, x, pair) quadruples whose rotated first-half output's E4M3 byte differs between the rope modes + (fp32 CPU emulation; the GPU comparison is the verdict).""" + g = torch.Generator().manual_seed(seed) + keep = [] + while sum(k.shape[0] for k in keep) < n: + m = 1 << 20 + c, s = torch.randn(m, generator=g), torch.randn(m, generator=g) + x = (torch.randn(m, generator=g) * 4).to(torch.bfloat16).float() + xp = (torch.randn(m, generator=g) * 4).to(torch.bfloat16).float() + sn = -s + r0 = (c * x) + (sn * xp) + r1 = ((c.double() * x.double()) + (sn * xp).double()).float() + r2 = ((sn.double() * xp.double()) + (c * x).double()).float() + b = [ + _round_bf16(r.contiguous()).to(torch.float8_e4m3fn).view(torch.uint8) + for r in (r0, r1, r2) + ] + diff = (b[0] != b[1]) | (b[0] != b[2]) | (b[1] != b[2]) + diff &= (r0.abs() < 400) & (r1.abs() < 400) + keep.append(torch.stack([c, s, x, xp], 1)[diff]) + return torch.cat(keep)[:n] + + +def test_crafted_inputs_pin_the_rope_mode() -> None: + """Rotations whose E4M3 byte depends on how the two products are summed: the in-tree preprocess and exactly the + pinned mode agree on every one.""" + mpc = torch.cuda.get_device_properties(0).multi_processor_count + params, pool = _case(1, [130], 128, False, seed=3) + half = 64 + quads = _discriminating(H * half, seed=5).cuda() + pos = 129 # the one row's rope position: cache length 130, one query row + table = params.attn.rotary_cos_sin.view(-1, half, 2) + # One (cos, sin) per dim: every head shares the position's table row, so the heads take the same dims' quads with + # their own (x, pair). + table[pos, :, 0] = quads[:half, 0] + table[pos, :, 1] = quads[:half, 1] + qkv = params.qkv_input.view(-1, H + 2 * KVH, D) + for h in range(H): + qkv[0, h, :half] = quads[:half, 2].to(torch.bfloat16) + qkv[0, h, half:] = quads[:half, 3].to(torch.bfloat16) + start = pool.clone() + q_ref = _preprocess(params, mpc) + torch.cuda.synchronize() + agree = {} + for mode in (0, 1, 2): + pool.copy_(start) + q = _op(params, pool, mode) + torch.cuda.synchronize() + agree[mode] = int((_bytes(q)[0, :, :half] == _bytes(q_ref)[0, :, :half]).sum()) + print( + f"[rope modes crafted] agreeing first-half query bytes of {H * half}: {agree}", flush=True + ) + assert agree[modeling.DRAFTER_ROPE_MODE] == H * half, agree diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py new file mode 100644 index 000000000000..b0478ee7a464 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fp8_indexer_qk_norm_rope.py @@ -0,0 +1,49 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_fp8_indexer_qk_norm_rope entry of the MiniMax-M3 tp8 target: index-Q out and index-K in the +cache against the in-place fused norm + RoPE followed by the E4M3 cast, byte for byte up to the 0.1% budget the +in-tree test of the kernel allows.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_fp8_indexer_qk_norm_rope import ( # noqa: E501 + m3_fp8_indexer_qk_norm_rope as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_fp8_indexer_qk_norm_rope requires a CUDA device" + +DIM, ROTARY, EPS, THETA, PAGE = 128, 64, 1e-6, 5e6, 128 + + +def _bytes_close(a: torch.Tensor, b: torch.Tensor) -> None: + assert a.shape == b.shape and a.dtype == b.dtype == torch.float8_e4m3fn + differ = int((a.view(torch.uint8) != b.view(torch.uint8)).sum()) + assert differ <= (a.numel() - 1) // 1000, f"{differ} of {a.numel()} E4M3 bytes differ" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + m, heads = cell.spec["m"], cell.spec["heads"] + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + qk = torch.randn(m, (heads + 1) * DIM, generator=g, device="cuda").to(torch.bfloat16) + qw = (torch.randn(DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + kw = (torch.randn(DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + pos = torch.randint(0, 200000, (m,), generator=g, device="cuda", dtype=torch.int32) + pages = (m + PAGE - 1) // PAGE + 1 + cache = torch.zeros(pages, 1, PAGE, DIM, dtype=torch.float8_e4m3fn, device="cuda") + slots = torch.randperm(pages * PAGE, generator=g, device="cuda")[:m].to(torch.int32) + before = qk.clone() + args = (qk, cache, slots, heads, DIM, ROTARY, EPS, qw, kw, THETA, pos) + with validating(op): + q = op(*args) + assert torch.equal(qk, before), "the BF16 input is left untouched" + q_ref, k_ref = op.reference(*args) + _bytes_close(q.reshape(m, heads, DIM), q_ref) + flat = cache.view(-1, DIM) + _bytes_close(flat[slots.long()], k_ref) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fp8_saturate.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fp8_saturate.py new file mode 100644 index 000000000000..381aa7b7c202 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fp8_saturate.py @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""paged_cache.fp8_e4m3_saturate (FP8_SATURATE): the E4M3 cache writes and the transient FP8 query casts saturate to ++-448 like the CUDA writers (__NV_SATFINITE); every finite in-range value keeps the plain cast's bits, values already +in E4M3 pass through, and the kill switch is exactly the plain cast. Where torch's cast already saturates (this image) +the conversion is the cast alone, one kernel; otherwise it clamps first, the same bytes.""" + +import pytest +import torch + +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import paged_cache + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + + +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float32]) +def test_fp8_e4m3_saturate_matches_satfinite(dtype, monkeypatch) -> None: + g = torch.Generator(device="cuda").manual_seed(0) + inside = (torch.randn(4096, generator=g, device="cuda") * 100).clamp(-448, 448).to(dtype) + plain = inside.to(torch.float8_e4m3fn) + assert torch.equal( + paged_cache.fp8_e4m3_saturate(inside).view(torch.uint8), plain.view(torch.uint8) + ) + edge = torch.tensor( + [449.0, 463.0, 464.0, 1e4, float("inf"), -500.0, -float("inf")], dtype=dtype, device="cuda" + ) + want = torch.tensor([448.0, 448.0, 448.0, 448.0, 448.0, -448.0, -448.0], device="cuda") + assert torch.equal(paged_cache.fp8_e4m3_saturate(edge).float(), want) + nan = paged_cache.fp8_e4m3_saturate(torch.tensor([float("nan")], dtype=dtype, device="cuda")) + assert torch.isnan(nan.float()).all() + already = edge.clamp(-448, 448).to(torch.float8_e4m3fn) + assert paged_cache.fp8_e4m3_saturate(already) is already + monkeypatch.setattr(paged_cache, "FP8_SATURATE", False) + killed = paged_cache.fp8_e4m3_saturate(edge) + assert torch.equal(killed.view(torch.uint8), edge.to(torch.float8_e4m3fn).view(torch.uint8)) + + +def test_cache_write_saturates(monkeypatch) -> None: + """The cache-dtype conversion every write_kv_slots branch applies once a model arms it (compared as bytes: torch + has no float8 index_copy_ kernel), and an E4M3 source passing through untouched; unarmed it is the plain cast.""" + values = torch.tensor([[1.0, 500.0, -1e5, float("inf")]], dtype=torch.bfloat16, device="cuda") + assert not paged_cache.SHARED_FP8_SATURATE + plain = paged_cache._to_cache_dtype(values, torch.float8_e4m3fn) + assert torch.equal(plain.view(torch.uint8), values.to(torch.float8_e4m3fn).view(torch.uint8)) + monkeypatch.setattr(paged_cache, "SHARED_FP8_SATURATE", True) + values = torch.tensor([[1.0, 500.0, -1e5, float("inf")]], dtype=torch.bfloat16, device="cuda") + out = paged_cache._to_cache_dtype(values, torch.float8_e4m3fn) + assert out.dtype == torch.float8_e4m3fn + want = torch.tensor([[1.0, 448.0, -448.0, 448.0]], device="cuda").to(torch.float8_e4m3fn) + assert torch.equal(out.view(torch.uint8), want.view(torch.uint8)) + assert paged_cache._to_cache_dtype(out, torch.float8_e4m3fn) is out + bf = paged_cache._to_cache_dtype(values, torch.bfloat16) + assert bf.dtype == torch.bfloat16 and torch.equal(bf, values) + + +_EDGES = [ + 448.0, + 449.0, + 463.0, + 464.0, + 1e4, + float("inf"), + -448.0, + -449.0, + -464.0, + -1e4, + -float("inf"), + float("nan"), + 0.0, +] + + +def _cuda_kernels(fn) -> int: + from torch.profiler import ProfilerActivity, profile + + torch.cuda.synchronize() + with profile(activities=[ProfilerActivity.CUDA]) as prof: + fn() + torch.cuda.synchronize() + return sum(1 for e in prof.events() if e.device_type.name == "CUDA") + + +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16, torch.float32]) +@pytest.mark.parametrize("cast_saturates", [True, False], ids=["cast_only", "clamp_fallback"]) +def test_fp8_e4m3_saturate_bytes_equal_clamp_then_cast(dtype, cast_saturates, monkeypatch) -> None: + """Both paths give the bytes of clamp(+-448) followed by the cast, edge values included.""" + monkeypatch.setattr(paged_cache, "_CAST_SATURATES", cast_saturates) + x = torch.tensor(_EDGES, dtype=dtype, device="cuda") + want = x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn) + assert torch.equal(paged_cache.fp8_e4m3_saturate(x).view(torch.uint8), want.view(torch.uint8)) + + +def test_fp8_e4m3_saturate_takes_the_cast_only_path_here(monkeypatch) -> None: + """This image's torch cast saturates, on the CPU (the import-time probe) and on CUDA, so the conversion is one + kernel; with the probe failing it falls back to clamp + cast, two kernels.""" + assert paged_cache._CAST_SATURATES + x = torch.tensor(_EDGES, dtype=torch.bfloat16, device="cuda") + assert torch.equal( + x.to(torch.float8_e4m3fn).view(torch.uint8), + x.clamp(-448.0, 448.0).to(torch.float8_e4m3fn).view(torch.uint8), + ) + assert _cuda_kernels(lambda: paged_cache.fp8_e4m3_saturate(x)) == 1 + monkeypatch.setattr(paged_cache, "_CAST_SATURATES", False) + assert _cuda_kernels(lambda: paged_cache.fp8_e4m3_saturate(x)) == 2 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fullctx_decode.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fullctx_decode.py new file mode 100644 index 000000000000..c6144b82abab --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_fullctx_decode.py @@ -0,0 +1,310 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7) for the attention/m3_fullctx_decode entry: the dense layers' full-context FP8 decode on PrimTS +split-KV decode at the recipe the target pins per (query rows per request, requests). + +Every cell runs against the fp32 reference (q rounded to E4M3 first, as the kernel runs it), eager and replayed from a +CUDA graph (the same bits), and twice (the same bits). The K/V pages are read through the strided views of a flat +sub-page pool, as the KV cache manager hands them out. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_fullctx_decode import ( # noqa: E501 + HEAD_DIM, + HEADS, + KV_HEADS, + MAX_KV_LEN, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_fullctx_decode import ( # noqa: E501 + m3_fullctx_decode as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_fullctx_decode requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp8 target is certified on SM 10.7 only", allow_module_level=True) + +PAGE = 128 +SM_SCALE = HEAD_DIM**-0.5 +# The FP8-KV band of the in-tree dense decode test (test_minimax_m3_dense_decode.test_matches_reference). +FP8_TOL = 6e-2 + + +class _Pool: + """The one method the entry asks of a KV cache manager.""" + + def __init__(self, pool: torch.Tensor, subpages: int): + self.pool = pool + self.subpages = subpages + + def get_kv_subpage_pool(self, layer_idx: int, kv_layout: str = "HND"): + assert kv_layout == "HND" + return self.pool, self.subpages + + +def _inputs(kv_lens, qlen: int, seed: int, subpages: int): + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(kv_lens) + pages = max((n + PAGE - 1) // PAGE for n in kv_lens) + slots = batch * pages + 3 + pool = torch.randn( + (slots * subpages, KV_HEADS, PAGE, HEAD_DIM), device="cuda", generator=g + ).clamp(-4, 4) + perm = torch.randperm(slots, generator=g, device="cuda")[: batch * pages] + block_table = perm.reshape(batch, pages).to(torch.int32) + # The last third of every request's values is shifted by +1, so a split that missed the tail fails the band. + for b, n in enumerate(kv_lens): + start = 2 * n // 3 + for page in range(start // PAGE, (n + PAGE - 1) // PAGE): + v = int(block_table[b, page]) * subpages + 1 + pool[v, :, (start % PAGE if page == start // PAGE else 0) :] += 1.0 + # q as the layer hands it over on decode steps: the FP8 query columns of the fused norm/RoPE kernel's + # [q | k | v] output (row stride wider than q), so no cast makes it compact. + qkv = torch.randn( + (batch * qlen, (HEADS + 2 * KV_HEADS) * HEAD_DIM), device="cuda", generator=g + ).to(torch.float8_e4m3fn) + q = qkv[:, : HEADS * HEAD_DIM].view(batch * qlen, HEADS, HEAD_DIM) + seq_lens = torch.tensor(kv_lens, device="cuda", dtype=torch.int32) + return q, _Pool(pool.to(torch.float8_e4m3fn), subpages), block_table, seq_lens + + +@pytest.mark.parametrize("subpages", [2, 3], ids=["kv", "kv+index"]) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, subpages, certifying) -> None: # noqa: F811 + spec = cell.spec + qlen, nreq = spec["qlen"], len(spec["kv_lens"]) + recipe = modeling.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + q, manager, block_table, seq_lens = _inputs( + spec["kv_lens"], qlen, zlib.crc32(cell.why.encode()) % 2**31, subpages + ) + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + args = (q, manager, 0, block_table, seq_lens, SM_SCALE, out, qlen, recipe) + certifying(op) + ref = op.reference(*args) + eager = op(*args).clone() + torch.testing.assert_close(eager.float(), ref, rtol=FP8_TOL, atol=FP8_TOL) + again = op(*args).clone() + assert torch.equal(again, eager), "not deterministic" + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(*args) + out.zero_() + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out, eager), "the CUDA-graph replay differs from the eager call" + + +def test_the_cells_cover_the_target_rows() -> None: + certified = {(c.spec["qlen"], len(c.spec["kv_lens"])) for c in op.CELLS} + assert set(modeling.FULLCTX_DECODE.keys) <= certified + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_the_bf16_merge_is_the_primts_reducer_to_one_ulp(cell) -> None: + """The BF16 split-KV merge op against PrimTS's own FP16 reducer and the cast: the same partials, merged once + in fp32 and rounded once, so within one BF16 spacing of the FP16-then-BF16 result.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_fullctx_decode as kernel, + ) + + if not kernel.has_own_reducer(): + pytest.skip("this build does not carry minimax_m3_sm107_tp2_fullctx_reduce") + spec = cell.spec + qlen, nreq = spec["qlen"], len(spec["kv_lens"]) + recipe = modeling.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + q, manager, block_table, seq_lens = _inputs(spec["kv_lens"], qlen, 11, 2) + outs = [] + for own in (True, False): + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + op(q, manager, 0, block_table, seq_lens, SM_SCALE, out, qlen, recipe, own_reducer=own) + outs.append(out.float()) + ours, primts = outs + _, exp = torch.frexp(primts.abs()) + spacing = torch.where( + primts == 0, + torch.full_like(primts, 2.0**-133), + torch.ldexp(torch.ones_like(primts), exp - 8), + ) + assert bool(((ours - primts).abs() <= spacing).all()) + + +def _tile_edge_kv_lens(plan, qlen: int): + """KV lengths around every kind of boundary the split ranges have, so each query row of the Q CTA in turn ends + its keys on a KV tile edge (k * tile + d, d in -1 .. qlen - 1): the first tiles, the reducer's 8-group stride, the + lengths where a split's range grows (splits * insts tiles) and a split edge past it.""" + tile, insts, splits = plan.tile_kv, plan.insts_kv, int(plan.recipe["splits"]) + grow = splits * insts + tiles = sorted({1, 2, 8, 9, grow - 1, grow, grow + 1, grow + 2 * insts, 2 * grow}) + return [k * tile + d for k in tiles for d in range(-1, qlen)] + + +@pytest.mark.parametrize("qlen,nreq", sorted(modeling.FULLCTX_DECODE_RECIPES)) +def test_rows_with_no_key_in_a_trailing_split(qlen, nreq) -> None: + """A row whose keys end before its Q CTA's causal union does has trailing splits holding none of its keys + (log2-LSE -inf); both reducers must leave them out. [1300, 1281] at 4 query rows is the batch a GSM8K decode + step first hit (NaN rows from exp2(-inf - -inf)).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_fullctx_decode as kernel, + ) + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + recipe = modeling.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + reducers = (True, False) if kernel.has_own_reducer() else (False,) + kernel.prebuild( + torch.device("cuda"), {(qlen, nreq): recipe}, MAX_KV_LEN, False, get_env_enable_pdl() + ) + lens = _tile_edge_kv_lens(kernel.plan_for(torch.device("cuda"), qlen, nreq, False), qlen) + batches = [[n] for n in lens] if nreq == 1 else [[a, b] for a, b in zip(lens, lens[::-1])] + batches.append([1300, 1281][:nreq]) + failures = [] + for kv_lens in batches: + q, manager, block_table, seq_lens = _inputs(kv_lens, qlen, sum(kv_lens), 2) + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + args = (q, manager, 0, block_table, seq_lens, SM_SCALE, out, qlen, recipe) + ref = op.reference(*args) + for own in reducers: + out.zero_() + op(*args, own_reducer=own) + got = out.float() + if not bool(torch.isfinite(got).all()) or not torch.allclose( + got, ref, rtol=FP8_TOL, atol=FP8_TOL + ): + failures.append((kv_lens, "bf16-merge" if own else "primts-reducer")) + assert not failures, failures + + +@pytest.mark.parametrize("qlen,nreq", sorted(modeling.FULLCTX_DECODE_RECIPES)) +def test_kv_lengths_past_the_policy_length(qlen, nreq) -> None: + """The plans are compiled for MAX_KV_LEN of K/V with PrimTS's policy sized at 1 << 20 (POLICY_KV_LEN): caches just + past the policy length (the AgentX bound is 1 << 20 plus the speculative rows) and well past it stay exact.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_fullctx_decode as kernel, + ) + from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + + recipe = modeling.FULLCTX_DECODE_RECIPES[(qlen, nreq)] + reducers = (True, False) if kernel.has_own_reducer() else (False,) + lengths = [(1 << 20) + 1, (1 << 20) + 8] + ([1_500_001] if nreq == 1 else []) + batches = [[n] * nreq if nreq == 1 else [n, (1 << 20) - 3] for n in lengths] + failures = [] + for kv_lens in batches: + q, manager, block_table, seq_lens = _inputs(kv_lens, qlen, 13, 2) + out = torch.zeros(q.shape, dtype=torch.bfloat16, device="cuda") + args = (q, manager, 0, block_table, seq_lens, SM_SCALE, out, qlen, recipe) + ref = op.reference(*args) + for own in reducers: + kernel.prebuild( + torch.device("cuda"), {(qlen, nreq): recipe}, MAX_KV_LEN, own, get_env_enable_pdl() + ) + out.zero_() + op(*args, own_reducer=own) + got = out.float() + if not bool(torch.isfinite(got).all()) or not torch.allclose( + got, ref, rtol=FP8_TOL, atol=FP8_TOL + ): + failures.append((kv_lens, "bf16-merge" if own else "primts-reducer")) + assert not failures, failures + + +@pytest.mark.parametrize( + "splits,rows,seq_len_q", [(108, 8, 1), (64, 8, 1), (96, 32, 4), (96, 32, 4)] +) +def test_the_tp8_merge_is_the_tp2_merge_bitwise(splits, rows, seq_len_q) -> None: + """The TP8 copy of the BF16 split-KV merge (one row per CTA, early dependent release) returns the image TP2 op's + bits on random 16-bit partials with empty (-inf log2-LSE) splits, both partial dtypes, two requests.""" + ops = torch.ops.trtllm + if not ( + hasattr(ops, "minimax_m3_sm107tp8_fullctx_reduce") + and hasattr(ops, "minimax_m3_sm107_tp2_fullctx_reduce") + ): + pytest.skip("needs both the TP8 merge copy and the image's TP2 merge op") + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(splits * 131 + rows) + nreq, kvh = 2, 1 + for dtype in (torch.float16, torch.bfloat16): + po = ( + (torch.randn(nreq, kvh, splits, rows, 128, device=dev, generator=g) * 3) + .to(dtype) + .contiguous() + ) + ps = torch.randn(nreq, kvh, splits, rows, device=dev, generator=g) * 4 + ps[torch.rand(ps.shape, device=dev, generator=g) < 0.05] = float("-inf") + seq = torch.tensor([165521, 120011], dtype=torch.int32, device=dev) + outs = [] + for reduce in ( + ops.minimax_m3_sm107_tp2_fullctx_reduce, + ops.minimax_m3_sm107tp8_fullctx_reduce, + ): + out = torch.zeros( + nreq * seq_len_q, kvh * rows // seq_len_q, 128, dtype=torch.bfloat16, device=dev + ) + reduce( + po, ps.contiguous(), seq, out, seq_len_q, seq_len_q, 128, 1, seq_len_q > 1, False + ) + outs.append(out) + torch.cuda.synchronize() + assert torch.equal(outs[0].view(torch.int16), outs[1].view(torch.int16)), ( + splits, + rows, + seq_len_q, + dtype, + ) + + +def _swizzled_scale_offsets(rows: int, width: int) -> torch.Tensor: + """Offsets of rows [0, rows)'s UE8M0 scales in the 128x4-swizzled layout of mxfp8_quantize(x, True).""" + k_sf = width // 32 + k_tiles = (k_sf + 3) // 4 + r = torch.arange(rows, device="cuda").view(-1, 1) + k = torch.arange(k_sf, device="cuda").view(1, -1) + offsets = ( + (r // 128) * k_tiles * 512 + (k // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + k % 4 + ) + return offsets.reshape(-1) + + +@pytest.mark.parametrize("splits,rows,seq_len_q", [(108, 8, 1), (64, 8, 1), (96, 32, 4)]) +def test_the_tp8_merge_mxfp8_copy_is_stock_mxfp8_quantize(splits, rows, seq_len_q) -> None: + """The TP8 merge's MXFP8 copy (the dense o_proj hand-off) equals mxfp8_quantize(out, True) bit for bit: E4M3 rows + and the real rows' swizzled UE8M0 scales, with an all-zero head row (amax 0) and saturating outliers; the BF16 + output is the merge without the copy.""" + ops = torch.ops.trtllm + if not hasattr(ops, "minimax_m3_sm107tp8_fullctx_reduce"): + pytest.skip("needs the TP8 merge copy") + dev = torch.device("cuda") + g = torch.Generator(device=dev).manual_seed(splits * 7 + rows) + nreq, kvh = 2, 1 + tokens = nreq * seq_len_q + width = kvh * rows // seq_len_q * 128 + for dtype in (torch.float16, torch.bfloat16): + po = torch.randn(nreq, kvh, splits, rows, 128, device=dev, generator=g) * 3 + po[0, 0, :, 1] = 0.0 + po[1, 0, :, 0, 5] = 6.0e4 + po = po.to(dtype).contiguous() + ps = (torch.randn(nreq, kvh, splits, rows, device=dev, generator=g) * 4).contiguous() + seq = torch.tensor([165521, 120011], dtype=torch.int32, device=dev) + args = (po, ps, seq) + tail = (seq_len_q, seq_len_q, 128, 1, seq_len_q > 1, False) + plain = torch.zeros(tokens, width, dtype=torch.bfloat16, device=dev) + ops.minimax_m3_sm107tp8_fullctx_reduce(*args, plain, *tail) + out = torch.zeros(tokens, width, dtype=torch.bfloat16, device=dev) + q8 = torch.empty(128, width, dtype=torch.float8_e4m3fn, device=dev) + sf = torch.zeros(128 * (width // 32), dtype=torch.uint8, device=dev) + ops.minimax_m3_sm107tp8_fullctx_reduce(*args, out, *tail, q8, sf) + torch.cuda.synchronize() + assert torch.equal(out.view(torch.int16), plain.view(torch.int16)) + stock_q, stock_sf = ops.mxfp8_quantize(out, True) + assert torch.equal( + q8[:tokens].view(torch.uint8), stock_q.view(torch.uint8).reshape(tokens, -1) + ), dtype + at = _swizzled_scale_offsets(tokens, width) + assert torch.equal(sf[at], stock_sf.view(torch.uint8).reshape(-1)[at]), dtype diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_identity_selection.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_identity_selection.py new file mode 100644 index 000000000000..8819b1fcfb73 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_identity_selection.py @@ -0,0 +1,103 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test: the short-sequence identity selection (runtime policy ``short_sequence_identity``) is the stock table. + +On a step whose every query token has at most 16 valid blocks, this target skips the index scorer and the select and +stages the table (i < n_valid ? i : -1). Stock runs the in-tree CuTe DSL scorer and ``select_blocks_from_maxscore``, +which returns ascending ids and drops a valid block only when its max score is -inf. The scorer's scores are FP8 +products summed in FP32 (|score| <= 448^2 * 128), so -inf comes only from the causal mask, which n_valid excludes. +This test drives the real scorer with extreme FP8 operands at the TP8 geometry (1 index head per rank) and checks +both claims. +""" + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + select_blocks_from_maxscore, +) +from tensorrt_llm._torch.custom_ops import ( + cute_dsl_custom_ops, # noqa: F401 — registers the scorer op +) + +PAGE, DIM, TOPK, HEADS, QLEN = 128, 128, 16, 1, 4 +E4M3_MAX = 448.0 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + + +def _fill(kind, shape, gen): + if kind == "random": + x = torch.randn(shape, generator=gen, device="cuda") * 64 + elif kind == "max": + x = torch.full(shape, E4M3_MAX, device="cuda") + elif kind == "negmax": + x = torch.full(shape, -E4M3_MAX, device="cuda") + elif kind == "alternating": + x = torch.full(shape, E4M3_MAX, device="cuda") + x[..., 1::2] = -E4M3_MAX + elif kind == "zero": + x = torch.zeros(shape, device="cuda") + else: # the smallest E4M3 subnormal + x = torch.full(shape, 2.0**-9, device="cuda") + return x.clamp(-E4M3_MAX, E4M3_MAX).to(torch.float8_e4m3fn) + + +_KINDS = ("random", "max", "negmax", "alternating", "zero", "tiny") + + +@pytest.mark.parametrize( + "seq_lens", [(4,), (129,), (2048,), (700, 2048), (5, 1500, 2047, 300), (1, 128, 256, 1024)] +) +@pytest.mark.parametrize("q_kind", _KINDS) +@pytest.mark.parametrize("k_kind", _KINDS) +def test_stock_selection_of_a_short_step_is_the_identity_table(seq_lens, q_kind, k_kind) -> None: + gen = torch.Generator(device="cuda").manual_seed(0) + reqs, total_q = len(seq_lens), len(seq_lens) * QLEN + nblk = [(s + PAGE - 1) // PAGE for s in seq_lens] + pages = sum(nblk) + 4 + perm = torch.randperm(pages, generator=gen, device="cuda").int() + block_table = torch.zeros((reqs, max(nblk)), dtype=torch.int32, device="cuda") + start = 0 + for r, n in enumerate(nblk): + block_table[r, :n] = perm[start : start + n] + start += n + n_valid = torch.tensor( + [(max(s - QLEN + o + 1, 0) + PAGE - 1) // PAGE for s in seq_lens for o in range(QLEN)], + dtype=torch.int32, + device="cuda", + ) + idx_q = _fill(q_kind, (total_q, HEADS, DIM), gen) + idx_k = _fill(k_kind, (pages, PAGE, DIM), gen) + max_score = torch.full((HEADS, max(nblk), total_q), float("-inf"), device="cuda") + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, + idx_k, + block_table, + torch.tensor(seq_lens, dtype=torch.int32, device="cuda"), + max_score.transpose(1, 2), + QLEN, + ) + for tok, nv in enumerate(n_valid.tolist()): + assert bool(torch.isfinite(max_score[:, :nv, tok]).all()), ( + f"token {tok}: a causally valid block scored non-finite" + ) + table = select_blocks_from_maxscore( + max_score, + topk=TOPK, + n_valid_blocks=n_valid, + init_blocks=1, + local_blocks=4, + head_major_output=True, + ).to(torch.int32) + # The selector returns [tokens, kv_heads, topk]; the decode reads it as [kv_heads, tokens, topk] (layers.py). + table = table.permute(1, 0, 2) + i = torch.arange(TOPK, dtype=torch.int32, device="cuda") + identity = torch.where( + i[None, :] < n_valid[:, None], i[None, :], torch.full_like(i[None, :], -1) + ) + assert torch.equal(table, identity[None].expand(HEADS, total_q, TOPK)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_index_score.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_index_score.py new file mode 100644 index 000000000000..56595cd6350f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_index_score.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_index_score entry: the MSA index scorer with its pre-wait schedule. + +Every cell is held to the fp32 reference (a few fp32 ulps: the kernel sums the 128 exact products in its k-split order) +and to the in-tree scorer (``cute_dsl_minimax_m3_index_decode_score`` at its own 256-CTA split) bit for bit, which the +pre-wait prefetch, the early tile issue and the cell's split must not change. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_index_score import ( # noqa: E501 + PAGE, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_index_score import ( # noqa: E501 + m3_index_score as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_index_score requires a CUDA device" + +HEADS = 1 # index heads per rank at TP8 (4 over 4 KV groups, each group on a rank pair) +HEAD_DIM = 128 + + +def _inputs(seq_lens, qlen: int, seed: int, dtype=torch.float8_e4m3fn): + g = torch.Generator(device="cuda").manual_seed(seed) + batch = len(seq_lens) + max_blocks = (max(seq_lens) + PAGE - 1) // PAGE + stride = ((max_blocks + 15) // 16) * 16 + pages = batch * max_blocks + block_table = ( + torch.randperm(pages, device="cuda", generator=g).to(torch.int32).view(batch, max_blocks) + ) + idx_q = torch.randn(batch * qlen, HEADS, HEAD_DIM, device="cuda", generator=g).to(dtype) + index_k = torch.randn(pages, PAGE, HEAD_DIM, device="cuda", generator=g).to(dtype) + seq = torch.tensor(seq_lens, device="cuda", dtype=torch.int32) + return idx_q, index_k, block_table, seq, stride + + +def _score(stride: int, total_q: int): + backing = torch.full( + (HEADS, stride, total_q), -float("inf"), device="cuda", dtype=torch.float32 + ) + return backing.transpose(1, 2) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + qlen = spec["qlen"] + idx_q, index_k, block_table, seq, stride = _inputs( + spec["seq_lens"], qlen, zlib.crc32(cell.why.encode()) + ) + score = _score(stride, idx_q.shape[0]) + assert spec["heads"] == HEADS + args = (idx_q, index_k, block_table, seq, score, qlen, spec["split_k"], spec["early"]) + certifying(op) + ref = op.reference(*args) + op(*args) + stock = _score(stride, idx_q.shape[0]) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, qlen + ) + torch.cuda.synchronize() + torch.testing.assert_close(score, ref, rtol=1e-5, atol=1e-4) + assert torch.equal(score, stock), "the scores differ from the in-tree scorer" + + +@pytest.mark.parametrize("early", [True, False], ids=["early", "prefetch"]) +def test_bf16_index_is_the_in_tree_scorer(early) -> None: + idx_q, index_k, block_table, seq, stride = _inputs((5000,), 4, 11, dtype=torch.bfloat16) + score, stock = _score(stride, 4), _score(stride, 4) + op(idx_q, index_k, block_table, seq, score, 4, 1536, early) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, 4 + ) + assert torch.equal(score, stock) + + +@pytest.mark.parametrize("qlen", [1, 2, 3, 4]) +def test_prestaged_blocks_at_the_stable_edge_are_the_in_tree_scorer(qlen) -> None: + """Lengths around the edge between pre-staged (all-old) and post-wait first blocks, 1 and 2 requests.""" + for k in (1, 2, 1294): + for length in (k * PAGE - qlen - 1, k * PAGE - qlen, k * PAGE - qlen + 1, k * PAGE + 1): + for seq_lens, split in ( + ((length,), 1536), + ((length,), 1272), + ((length, max(qlen, length // 3)), 768), + ): + idx_q, index_k, block_table, seq, stride = _inputs(seq_lens, qlen, 31 + length) + score, stock = _score(stride, idx_q.shape[0]), _score(stride, idx_q.shape[0]) + op(idx_q, index_k, block_table, seq, score, 4, split, True) + torch.ops.trtllm.cute_dsl_minimax_m3_index_decode_score( + idx_q, index_k, block_table, seq, stock, qlen + ) + assert torch.equal(score, stock), (qlen, seq_lens) + + +def test_guard_refuses_a_request_past_its_block_table() -> None: + idx_q, index_k, block_table, seq, stride = _inputs((1000,), 1, 5) + seq = seq + 4 * PAGE + with pytest.raises(AssertionError, match="past its block-table row"): + with validating(op): + op(idx_q, index_k, block_table, seq, _score(stride + 16, 1), 1, 1536, True) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_msa_attention.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_msa_attention.py new file mode 100644 index 000000000000..75226ff2bb9b --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_msa_attention.py @@ -0,0 +1,224 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_msa_attention entry: one layer's MSA backend step against its float64 reference. + +Every cell runs on the runtime's own objects (``_m3_attn_runtime_tp8``): a hybrid NVFP4 ``MiniMaxM3KVCacheManagerV2`` +with the requests allocated in it, ``MiniMaxM3MsaSparseAttentionMetadata`` filled by its ``prepare()``, and the layer's +``MiniMaxM3MsaSparseAttention`` built as the target builds it. Earlier rows (and, on a sparse layer, the step's own +rows, which the target's producer writes before the call) go into the cache through the in-tree scatters, at the +slots the manager assigned. The sparse layers' index-K rows are planted so every block's score is 1/8 or more apart +from every other's: the top-k selection the reference recomputes is then unambiguous. + +The cells are the target's calls (``_attention_core`` in modeling.py): a dense layer with live bf16 K / V in a +prefill chunk and live FP8 K / V in a decode step (the dense layers sit in the hybrid cache's FP8 half), a sparse +layer whose producer already wrote the caches (``k = v = None``) in a prefill chunk beyond 16 blocks and in a mixed +step, both with the NVFP4 scales the target passes. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_msa_attention import ( # noqa: E501 + m3_msa_attention as op, +) + +__extra_import_path__ = ["../../../..", "."] +import _m3_attn_runtime_tp8 as rt # noqa: E402 — needs the path declared above +from _validating import ULP, assert_within_ulp, certifying # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_msa_attention requires a CUDA device" + +# Bands, in BF16 ulps (2^-8) of each token's [heads x 128] row's largest magnitude (element) and of the relative RMS, +# per kernel the step reaches. The reference reads the same quantized cache rows (and casts Q to E4M3 where the FP8 +# FMHA does), so what is left is each FMHA's own arithmetic, measured on GB300 over these cells (max element / RMS): +# * fmha_sm100 over the FP8 pool (dense context rows): 10.7 / 6.0. It runs E4M3 x E4M3 and its softmax +# probabilities are rounded before the PV product. +# * trtllm-gen over the FP8 pool (dense generation rows): 7.7 / 6.1. It rounds the probabilities to E4M3 (scaled by +# 448 against the running maximum): emulating that brings a single-tile request to 0.6 RMS. +# * the NVFP4 CSR prefill (sparse context rows): 33.6 / 11.8 at 4096 rows, 24.3 / 9.1 at 512. It stages the +# reconstructed K / V in E4M3 (the in-tree A/B test, test_minimax_m3_sparse_attn_decode, notes it and holds it +# to 3e-2). +# * the Triton NVFP4 sparse decode (sparse generation rows): 1.0 / 0.19 -- the band of its own entry +# (m3_sparse_decode): exact operands, FP16 probabilities, BF16 rounding. +# Each band leaves about 1.5x the measured maximum; a shifted or wrongly selected reference is far outside them. +_FP8_CONTEXT = dict(element_ulp=16.0, rms_ulp=9.0) +_FP8_DECODE = dict(element_ulp=12.0, rms_ulp=9.0) +_NVFP4_CONTEXT = dict(element_ulp=48.0, rms_ulp=16.0) +_NVFP4_DECODE = dict(element_ulp=2.0, rms_ulp=0.5) +_BANDS = { + ("dense", "context"): (_FP8_CONTEXT, None), + ("dense", "generation"): (None, _FP8_DECODE), + ("sparse", "context"): (_NVFP4_CONTEXT, None), + ("sparse", "mixed"): (_NVFP4_CONTEXT, _NVFP4_DECODE), +} + + +def _assert_in_bands(output, ref, bands, context_rows: int) -> None: + """Gate the context rows and the generation rows each on their own kernel's band.""" + context_band, decode_band = bands + if context_rows: + assert_within_ulp(output[:context_rows], ref[:context_rows], **context_band) + if context_rows < output.shape[0]: + assert_within_ulp(output[context_rows:], ref[context_rows:], **decode_band) + + +# (request rows, cached tokens per request, new tokens per request, context rows) per cell. +_SHAPES = { + ("dense", "context"): ((384,), (1024,), 1), + ("dense", "generation"): ((128, 2500), (1, 1), 0), + ("sparse", "context"): ((0,), (4096,), 1), + ("sparse", "mixed"): ((0, 2999, 4500), (512, 1, 1), 1), +} + + +def _ulp_stats(out: torch.Tensor, ref: torch.Tensor) -> tuple[float, float]: + """The two quantities ``assert_within_ulp`` gates, in BF16 ulps.""" + o, r = out.float(), ref.float() + scale = r.abs().amax(dim=1, keepdim=True).clamp_min(1e-9) + element = ((o - r).abs() / scale).max().item() / ULP[torch.bfloat16] + rms = ((o - r).pow(2).mean().sqrt() / r.pow(2).mean().sqrt()).item() / ULP[torch.bfloat16] + return element, rms + + +def _shift(x: torch.Tensor) -> torch.Tensor: + """``x``'s rows moved down by one (byte-wise, so E4M3 rows move too).""" + return x.view(torch.uint8).roll(1, 0).view(x.dtype) if x.element_size() == 1 else x.roll(1, 0) + + +def _shape(cell): + spec = cell.spec + cached, qlens, contexts = _SHAPES[spec["layer"], spec["phase"]] + if spec["phase"] == "context": + assert sum(qlens) == spec["tokens"] + elif spec["phase"] == "generation": + assert len(qlens) == spec["reqs"] + else: + assert sum(qlens[:contexts]) == spec["tokens"] and len(qlens) - contexts == spec["reqs"] + return cached, qlens, contexts + + +def _run_dense(cell, gen, manager, backend, metadata): + cached, qlens, contexts = _shape(cell) + rids = list(range(len(qlens))) + rt.allocate(manager, rids, qlens, cached) + for rid, c in zip(rids, cached): + if c: + hist = torch.randn(2, c, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to( + torch.float8_e4m3fn + ) + rt.write_fp8_rows(manager, rt.DENSE, rid, torch.arange(c), hist[0], hist[1]) + rt.prepare(metadata, rids, qlens, cached, contexts) + total = sum(qlens) + # A prefill chunk arrives in BF16 (the hybrid cache's dense layers take FP8 Q / K / V only on decode steps). + dtype = torch.bfloat16 if contexts else torch.float8_e4m3fn + q = torch.randn(total, rt.HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to(dtype) + k = torch.randn(total, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to(dtype) + v = torch.randn(total, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to(dtype) + # The NVFP4 target passes its layer scales on every layer; an FP8 layer's write and FMHA take none. + inv = torch.tensor([1.0, 0.5, 2.0], device="cuda") + fwd = 1.0 / inv + return (q, k, v, None, inv, fwd), dict(rids=rids, cached=cached, qlens=qlens, contexts=contexts) + + +def _run_sparse(cell, gen, manager, backend, metadata): + cached, qlens, contexts = _shape(cell) + rids = list(range(len(qlens))) + rt.allocate(manager, rids, qlens, cached) + u0, u1 = rt.index_directions(gen) + lengths = [c + n for c, n in zip(cached, qlens)] + keys = [ + torch.randn(n, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda") for n in lengths + ] + values = [ + torch.randn(n, rt.KV_HEADS * rt.HEAD_DIM, generator=gen, device="cuda") for n in lengths + ] + quant = rt.nvfp4_quant(torch.cat(keys), torch.cat(values)) + for rid, n, kk, vv in zip(rids, lengths, keys, values): + # Every row, the step's own included: the producer writes a sparse layer's caches before the call. + idx_k = rt.planted_index_keys(n, u0, u1, gen) + rt.write_nvfp4_rows( + manager, + metadata, + rt.SPARSE, + rid, + torch.arange(n), + kk.to(torch.bfloat16), + vv.to(torch.bfloat16), + idx_k, + quant, + ) + rt.prepare(metadata, rids, qlens, cached, contexts) + total = sum(qlens) + q = torch.randn(total, rt.HEADS * rt.HEAD_DIM, generator=gen, device="cuda").to( + torch.float8_e4m3fn + ) + # Query tokens alternate between the two planted directions, so neighbouring tokens select different blocks; the + # rank's index heads take alternating directions, so with more than one KV head they select different blocks too. + turn = torch.arange(total, device="cuda") % 2 + directions = torch.stack([u0, u1]) + idx_q = torch.cat([directions[(turn + h) % 2] for h in range(rt.INDEX_HEADS)], dim=1) + idx_q = idx_q.to(torch.float8_e4m3fn) + return (q, None, None, idx_q, quant, 1.0 / quant), dict( + rids=rids, cached=cached, qlens=qlens, contexts=contexts + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + sparse = spec["layer"] == "sparse" + gen = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + manager = rt.make_manager(draft=False) + try: + backend = rt.make_msa_backend(rt.SPARSE if sparse else rt.DENSE) + metadata = rt.make_msa_metadata(manager) + run = _run_sparse if sparse else _run_dense + (q, k, v, idx_q, inv, fwd), step = run(cell, gen, manager, backend, metadata) + output = torch.full( + (q.shape[0], rt.HEADS * rt.HEAD_DIM), float("nan"), dtype=torch.bfloat16, device="cuda" + ) + certifying(op) + result = op(backend, q, k, v, idx_q, metadata, output, sparse, inv, fwd) + assert result is output + ref = op.reference(backend, q, k, v, idx_q, metadata, output, sparse, inv, fwd) + assert torch.isfinite(output.float()).all(), "every live row is written" + bands = _BANDS[spec["layer"], spec["phase"]] + context_rows = sum(step["qlens"][: step["contexts"]]) + for name, rows in ( + ("context", slice(0, context_rows)), + ("generation", slice(context_rows, None)), + ): + if output[rows].numel(): + element, rms = _ulp_stats(output[rows], ref[rows]) + print( + f"\n[{cell.why}] {name} rows: max element {element:.2f} ulp, rms {rms:.3f} ulp" + ) + _assert_in_bands(output, ref, bands, context_rows) + + if k is not None: + # The call's own cache write: the live K / V rows land, as a plain E4M3 cast, at their positions. + pool = manager.get_buffers(rt.DENSE, "HND") + first = 0 + for rid, c, n in zip(step["rids"], step["cached"], step["qlens"]): + slots = rt.flat_slots(manager, rt.DENSE, rid, torch.arange(c, c + n)).long() + for role, src in ((0, k), (1, v)): + for h in range(rt.KV_HEADS): + got = pool[slots // rt.PAGE, role, h, slots % rt.PAGE] + want = src[first : first + n, h * rt.HEAD_DIM : (h + 1) * rt.HEAD_DIM] + want = want.to(torch.float8_e4m3fn) + assert torch.equal(got.view(torch.uint8), want.view(torch.uint8)) + first += n + + # The band discriminates: a reference whose query rows are shifted by one token is outside it. + shifted = op.reference(backend, _shift(q), k, v, idx_q, metadata, output, sparse, inv, fwd) + with pytest.raises(AssertionError): + _assert_in_bands(output, shifted, bands, context_rows) + if sparse: + # And the selection is load-bearing: attending every causal block instead is outside it too. + dense = op.reference(backend, q, k, v, idx_q, metadata, output, False, inv, fwd) + with pytest.raises(AssertionError): + _assert_in_bands(output, dense, bands, context_rows) + finally: + manager.shutdown() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_n_valid_contract.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_n_valid_contract.py new file mode 100644 index 000000000000..baf8ef7a2b73 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_n_valid_contract.py @@ -0,0 +1,138 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The per-token valid-block count is written only before the step's first layer. + +The long-row select (``kernels/m3_select_long_dsl.py``, ``SELECT_N_VALID_PREWAIT``) reads its row's entry of +``MsaMetadata.msa_n_valid_blocks`` BEFORE its PDL grid-dependency wait. That is exact only while every write to the +buffer happens before the step's first layer: + +- the allocation (``_alloc_msa_proxy_scratch``); +- the host-to-device copy in ``prepare()`` (``_build_step_plans``); +- ``on_update_kv_lens``, which the runner calls before the forward. It writes through the plain + ``<<<>>>`` launch of ``msa_kv_update_kernel`` or through torch copies, so any PDL grid of the step starts only + after it completes. + +This test scans ``tensorrt_llm/_torch`` and fails on any other write site (including writes through a local alias of +the buffer), so a new writer has to revisit the pre-wait read. A buffer handed to an opaque kernel launch is not +tracked; such a launch inside a layer must be reviewed by hand. + +CPU only: it parses the sources. +""" + +import ast +from pathlib import Path + +import pytest + +import tensorrt_llm + +pytestmark = pytest.mark.cpu_only + +_ROOT = Path(tensorrt_llm.__file__).resolve().parent / "_torch" +_NAME = "msa_n_valid_blocks" +# In-place tensor methods that write their receiver. +_MUTATORS = { + "copy_", + "fill_", + "zero_", + "index_put_", + "index_copy_", + "scatter_", + "masked_fill_", + "add_", + "sub_", +} +# Calls whose first argument is the destination they write: ``step_inputs.copy_(dst, src)`` and the H2D helpers. +_COPY_HELPERS = {"copy_", "copy_h2d_if_changed", "copy_h2d_pool_shared"} +_BACKEND = "attention/backends/sparse/minimax_m3/msa_backend.py" +# (file relative to tensorrt_llm/_torch, enclosing function) -> writes allowed there +_ALLOWED = { + (_BACKEND, "_alloc_msa_proxy_scratch"), # the allocation + (_BACKEND, "_build_step_plans"), # the H2D copy in prepare() + (_BACKEND, "on_update_kv_lens"), # the device-side correction before the forward +} + + +def _base(node): + """The innermost receiver of an attribute / subscript / call chain.""" + while isinstance(node, (ast.Subscript, ast.Attribute, ast.Call)): + if isinstance(node, ast.Attribute) and node.attr == _NAME: + return node + node = node.func if isinstance(node, ast.Call) else node.value + return node + + +def _mentions(node, aliases) -> bool: + """Whether the expression's receiver chain is ``*.msa_n_valid_blocks`` or a local alias of it.""" + b = _base(node) + if isinstance(b, ast.Attribute): + return b.attr == _NAME + return isinstance(b, ast.Name) and (b.id == _NAME or b.id in aliases) + + +def _refers(node, aliases) -> bool: + """Whether any sub-expression (e.g. a branch of a conditional) names the buffer or an alias.""" + return any( + (isinstance(n, ast.Attribute) and n.attr == _NAME) + or (isinstance(n, ast.Name) and (n.id == _NAME or n.id in aliases)) + for n in ast.walk(node) + ) + + +def _write_sites(tree: ast.AST): + sites = [] + + def visit(node, func, aliases): + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + func, aliases = node.name, set() + if isinstance(node, (ast.Assign, ast.AugAssign, ast.AnnAssign)): + targets = node.targets if isinstance(node, ast.Assign) else [node.target] + value = node.value + for t in targets: + if isinstance(t, ast.Name) and value is not None and _refers(value, aliases): + aliases.add(t.id) # a local view of the buffer + elif ( + not isinstance(t, ast.Name) + and _mentions(t, aliases) + and not isinstance(node, ast.AnnAssign) + ): + sites.append((func, node.lineno)) + elif ( + isinstance(t, ast.Attribute) + and t.attr == _NAME + and isinstance(node, ast.AnnAssign) + ): + pass # a dataclass field declaration + if isinstance(node, ast.Call): + f = node.func + if isinstance(f, ast.Attribute) and f.attr in _MUTATORS and _mentions(f.value, aliases): + sites.append((func, node.lineno)) + name = ( + f.id + if isinstance(f, ast.Name) + else (f.attr if isinstance(f, ast.Attribute) else "") + ) + if name in _COPY_HELPERS and node.args and _mentions(node.args[0], aliases): + sites.append((func, node.lineno)) + for child in ast.iter_child_nodes(node): + visit(child, func, aliases) + + visit(tree, None, set()) + return sites + + +def test_only_the_pre_forward_paths_write_the_valid_block_count() -> None: + found = set() + for path in sorted(_ROOT.rglob("*.py")): + src = path.read_text() + if _NAME not in src: + continue + rel = path.relative_to(_ROOT).as_posix() + for func, line in _write_sites(ast.parse(src)): + assert (rel, func) in _ALLOWED, ( + f"{rel}:{line} ({func}) writes msa_n_valid_blocks; the long-row select reads it before its PDL grid " + "wait (kernels/m3_select_long_dsl.py, SELECT_N_VALID_PREWAIT), so a writer inside the " + "step must be ordered before the select's grid or the read moved after the wait" + ) + found.add((rel, func)) + assert found == _ALLOWED, f"expected write sites {_ALLOWED}, found {found}" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_producer.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_producer.py new file mode 100644 index 000000000000..a0ef31a4ad97 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_producer.py @@ -0,0 +1,209 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_producer entry: the pre-wait-staged NVFP4 QKV + indexer producer. + +Every cell is held to the native producer (``minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert``) bit for bit on q, +index-q and every K / V / scale / index-K cache byte, and to the fp32 torch reference: the FP8 outputs within one E4M3 +step, the decoded NVFP4 K / V within one E2M1 step of their block scale. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_producer import ( # noqa: E501 + FP4_BLOCK, + HEADS_Q, + PAGE, + decode_nvfp4_token, + scale_index, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_producer import ( # noqa: E501 + m3_producer as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_producer requires a CUDA device" + +PAGES = 1024 +MAX_POSITIONS = 262144 +EPS = 1e-6 +KV_QUANT = (1.0, 0.37, 0.61) + + +def _caches(): + return ( + torch.zeros(PAGES, 2, 1, PAGE, 64, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 2, 1, PAGE, 8, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 1, PAGE, 128, dtype=torch.float8_e4m3fn, device="cuda"), + ) + + +def _inputs(tokens: int, scale: float, bad_slots: bool, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + rotary = torch.randn(MAX_POSITIONS, 2, 32, device="cuda", generator=g) + weights = [(torch.randn(128, device="cuda", generator=g) * 0.3).bfloat16() for _ in range(4)] + packed = ( + torch.randn(tokens, (HEADS_Q + 4) * 128, device="cuda", generator=g) * scale + ).bfloat16() + slots = torch.randperm(PAGES * PAGE, device="cuda", generator=g)[:tokens].int() + if bad_slots: + slots[1] = -1 + slots[2] = PAGES * PAGE + 5 + positions = torch.randint( + 0, MAX_POSITIONS, (tokens,), device="cuda", dtype=torch.int32, generator=g + ) + quant = torch.tensor(KV_QUANT, device="cuda") + return packed, slots, quant, weights, rotary, positions + + +def _args(inputs, caches, early: bool = True): + packed, slots, quant, weights, rotary, positions = inputs + return (packed, *caches, slots, quant, EPS, *weights, rotary, positions, HEADS_Q, True, early) + + +def _native(inputs, caches): + packed, slots, quant, weights, rotary, positions = inputs + return torch.ops.trtllm.minimax_m3_nvfp4_qkv_indexer_norm_rope_kv_insert( + packed, *caches, slots, quant, HEADS_Q, 1, 1, 128, 64, EPS, *weights, rotary, positions + ) + + +def _within_fp8(out: torch.Tensor, ref: torch.Tensor, name: str) -> None: + # One E4M3 step (3 mantissa bits) of the value, or its smallest subnormal step. + err = (out.float() - ref).abs() + assert bool((err <= ref.abs() * 2.0**-3 + 2.0**-9).all()), (name, float(err.max())) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + early = spec.get("early", True) + inputs = _inputs( + spec["tokens"], spec["scale"], spec.get("bad_slots", False), zlib.crc32(cell.why.encode()) + ) + stock_caches, caches = _caches(), _caches() + stock = _native(inputs, stock_caches) + args = _args(inputs, caches, early) + certifying(op) + ref = op.reference(*args) + out = op(*args) + torch.cuda.synchronize() + names = ("q", "index_q", "kv_data", "kv_scale", "index_k") + for name, s, o in zip(names, (*stock, *stock_caches), (*out, *caches)): + assert torch.equal(s.view(torch.uint8), o.view(torch.uint8)), ( + f"{name} differs from the native producer" + ) + + _within_fp8(out[0], ref["q"], "q") + _within_fp8(out[1][:, 0], ref["index_q"], "index_q") + kv_data, kv_scale, index_k = caches + slots = inputs[1] + checked = 0 + for t in range(spec["tokens"]): + slot = int(slots[t]) + if slot < 0 or slot >= PAGES * PAGE: + continue + page, within = divmod(slot, PAGE) + _within_fp8(index_k[page, 0, within], ref["index_k"][t], f"index_k token {t}") + for plane, name in ((0, "k"), (1, "v")): + scales = kv_scale[page, plane, 0].reshape(-1)[scale_index(within, plane).cuda()] + dq = decode_nvfp4_token(kv_data[page, plane, 0, within], scales, KV_QUANT[1 + plane]) + r = ref[name][t].bfloat16().float() + # Round to nearest E2M1 under the block's E4M3-rounded scale s (in value units s / quant_scale): at most + # half the widest E2M1 step (one unit), plus the clipping at 6 units when s rounded down; a scale that + # underflows E4M3 zeroes the block. A slack of one E4M3 step covers the kernel's approximate reciprocals. + qs = KV_QUANT[1 + plane] + amax = r.view(-1, FP4_BLOCK).abs().amax(dim=1) + unit = (qs * amax / 6.0).to(torch.float8_e4m3fn).float() / qs + bound = torch.where( + unit > 0, 1.15 * unit + (amax - 6.0 * unit / 1.15).clamp_min(0.0), amax + ) + bound = bound.repeat_interleave(FP4_BLOCK) + 1e-12 + assert bool(((dq - r).abs() <= bound).all()), (name, t) + checked += 1 + assert checked >= spec["tokens"] - 2 + + +def test_early_trigger_does_not_change_the_bits() -> None: + inputs = _inputs(48, 1.0, False, 7) + a, b = _caches(), _caches() + out_a = op(*_args(inputs, a, early=True)) + out_b = op(*_args(inputs, b, early=False)) + for x, y in zip((*out_a, *a), (*out_b, *b)): + assert torch.equal(x.view(torch.uint8), y.view(torch.uint8)) + + +@pytest.mark.parametrize( + "tokens, scale, bad_slots, early", + [ + (1, 1.0, False, True), + (4, 3.0, False, True), + (16, 1e-3, False, True), + (32, 40.0, True, True), + (48, 1.0, False, True), + (1000, 1.0, False, False), + ], +) +def test_prewait_step_inputs_is_bitwise_the_shared_op(tokens, scale, bad_slots, early) -> None: + """modeling's PRODUCER_PREWAIT_STEP_INPUTS: this target's copy (minimax_m3_sm107_tp8_producer_pw, the position ids, + slots and RoPE rows read before its grid wait) writes exactly the shared op's q, index-q and cache bytes.""" + assert hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp8_producer_pw"), ( + "the target's producer op is not loaded (its TP8 extension)" + ) + inputs = _inputs(tokens, scale, bad_slots, 11) + a, b = _caches(), _caches() + out_a = op(*_args(inputs, a, early=early)) + out_b = op(*_args(inputs, b, early=early), prewait_step_inputs=True) + for x, y in zip((*out_a, *a), (*out_b, *b)): + assert torch.equal(x.view(torch.uint8), y.view(torch.uint8)) + + +def test_guard_refuses_a_repeated_slot() -> None: + inputs = list(_inputs(4, 1.0, False, 3)) + inputs[1] = inputs[1].clone() + inputs[1][1] = inputs[1][0] + with pytest.raises(AssertionError, match="must not repeat a slot"): + with validating(op): + op(*_args(tuple(inputs), _caches())) + + +def test_op_refuses_an_operand_off_the_device() -> None: + inputs = list(_inputs(4, 1.0, False, 9)) + inputs[5] = inputs[5].cpu() # position ids on the host + with pytest.raises(RuntimeError, match="packed's CUDA device"): + op(*_args(tuple(inputs), _caches())) + + +def test_op_refuses_an_unaligned_index_k_page_stride() -> None: + """The index-K rows are written as 32-bit words: a page stride that is not a multiple of 4 is refused.""" + kv_data, kv_scale, _ = _caches() + backing = torch.zeros(PAGES * (128 * 128 + 1), dtype=torch.float8_e4m3fn, device="cuda") + index_k = backing.as_strided((PAGES, 1, PAGE, 128), (128 * 128 + 1, 128 * 128, 128, 1)) + with pytest.raises(RuntimeError, match="index_k"): + op(*_args(_inputs(4, 1.0, False, 13), (kv_data, kv_scale, index_k))) + + +def test_op_refuses_unaligned_weights_or_rope_table() -> None: + """The norm weights are read as 8-byte words and the RoPE rows as float4: offset views are refused.""" + packed, slots, quant, weights, rotary, positions = _inputs(4, 1.0, False, 17) + offset_weight = torch.zeros(129, dtype=torch.bfloat16, device="cuda")[1:] + with pytest.raises(RuntimeError, match="norm weights"): + op( + *_args( + (packed, slots, quant, [offset_weight, *weights[1:]], rotary, positions), _caches() + ) + ) + flat = torch.zeros(rotary.numel() + 1, device="cuda") + offset_rope = flat[1:].view(rotary.shape) + with pytest.raises(RuntimeError, match="rotary_cos_sin"): + op(*_args((packed, slots, quant, weights, offset_rope, positions), _caches())) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py new file mode 100644 index 000000000000..d2ec8991a48d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_qk_norm_rope_fp8.py @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_qk_norm_rope_fp8 entry of the MiniMax-M3 tp8 target: the out-of-place FP8 form of the fused +per-head Gemma RMSNorm + partial NeoX RoPE against the in-place kernel followed by the E4M3 cast. V is a cast, so it +matches bit for bit; Q and K round once from fp32 where the composition rounds through bf16 first, so each element is +within one E4M3 step of it.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_qk_norm_rope_fp8 import ( # noqa: E501 + m3_qk_norm_rope_fp8 as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_qk_norm_rope_fp8 requires a CUDA device" + +HEAD_DIM, ROTARY_DIM, EPS, THETA = 128, 64, 1e-6, 5e6 + + +def _args(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + q, kv = spec["q"], spec["kv"] + qkv = torch.randn(spec["m"], (q + 2 * kv) * HEAD_DIM, generator=g, device="cuda").to( + torch.bfloat16 + ) + qw = (torch.randn(HEAD_DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + kw = (torch.randn(HEAD_DIM, generator=g, device="cuda") * 0.1).to(torch.bfloat16) + pos = torch.randint(0, 200000, (spec["m"],), generator=g, device="cuda", dtype=torch.int32) + return ( + qkv, + q, + kv, + kv, + HEAD_DIM, + ROTARY_DIM, + EPS, + qw, + kw, + THETA, + True, + pos, + 1.0, + 0.0, + 0.0, + 1.0, + True, + True, + False, + 0, + 0, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + args = _args(cell.spec, zlib.crc32(cell.why.encode()) % 2**31) + before = args[0].clone() + with validating(op): + out = op(*args) + assert torch.equal(args[0], before), "the FP8 form must leave qkv untouched" + ref = op.reference(*args) + assert out.dtype == torch.float8_e4m3fn and out.shape == ref.shape + v_start = (cell.spec["q"] + cell.spec["kv"]) * HEAD_DIM + assert torch.equal(out[:, v_start:].view(torch.uint8), ref[:, v_start:].view(torch.uint8)) + assert _e4m3_steps(out[:, :v_start], ref[:, :v_start]) <= 1 + + +def _e4m3_steps(a, b) -> int: + """The largest distance between two E4M3 tensors in representable values (E4M3 codes order by magnitude).""" + + def signed(t): + c = t.view(torch.uint8).int() + # Sign-magnitude to a line: -0 and +0 both map to 0. + return torch.where(c >= 128, 128 - c, c) + + return int((signed(a) - signed(b)).abs().max()) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_qkv_insert.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_qkv_insert.py new file mode 100644 index 000000000000..05065902858f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_qkv_insert.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the attention/m3_qkv_insert entry: the generic horizontal producer on an FP8 main cache. + +Q, index-Q and every written K / V / index-K cache row are held within one E4M3 step of the fp32 reference. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_qkv_insert import ( # noqa: E501 + m3_qkv_insert as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_qkv_insert requires a CUDA device" + +PAGES, PAGE, HEADS, KV, INDEX, D, ROTARY = 256, 128, 8, 1, 1, 128, 64 +MAX_POSITIONS = 262144 +EPS = 1e-6 + + +def _within_fp8(out: torch.Tensor, ref: torch.Tensor, name: str) -> None: + # One E4M3 step (3 mantissa bits) of the value, or its smallest subnormal step. + err = (out.float() - ref).abs() + assert bool((err <= ref.abs() * 2.0**-3 + 2.0**-9).all()), (name, float(err.max())) + + +def _args(tokens: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + packed = torch.randn( + tokens, (HEADS + 2 * KV + INDEX + 1) * D, device="cuda", generator=g + ).bfloat16() + kv = torch.zeros(PAGES, 2, KV, PAGE, D, dtype=torch.float8_e4m3fn, device="cuda") + index_k = torch.zeros(PAGES, 1, PAGE, D, dtype=torch.float8_e4m3fn, device="cuda") + slots = torch.randperm(PAGES * PAGE, device="cuda", generator=g)[:tokens].int() + weights = [(torch.randn(D, device="cuda", generator=g) * 0.3).bfloat16() for _ in range(4)] + rotary = torch.randn(MAX_POSITIONS, 2, ROTARY // 2, device="cuda", generator=g) + positions = torch.randint( + 0, MAX_POSITIONS, (tokens,), device="cuda", dtype=torch.int32, generator=g + ) + return ( + packed, + kv, + None, + index_k, + slots, + None, + HEADS, + KV, + INDEX, + D, + ROTARY, + EPS, + *weights, + rotary, + positions, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _args(cell.spec["m"], zlib.crc32(cell.why.encode())) + certifying(op) + ref = op.reference(*args) + q, index_q = op(*args) + torch.cuda.synchronize() + _within_fp8(q.view(ref["q"].shape), ref["q"], "q") + _within_fp8(index_q.view(ref["index_q"].shape), ref["index_q"], "index_q") + kv, index_k, slots = args[1], args[3], args[4] + for t in range(cell.spec["m"]): + page, within = divmod(int(slots[t]), PAGE) + _within_fp8(kv[page, 0, :, within], ref["k"][t], f"k token {t}") + _within_fp8(kv[page, 1, :, within], ref["v"][t], f"v token {t}") + _within_fp8(index_k[page, 0, within], ref["index_k"][t], f"index_k token {t}") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_blocks.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_blocks.py new file mode 100644 index 000000000000..033468e3cdb4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_blocks.py @@ -0,0 +1,45 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_select_blocks entry of the MiniMax-M3 tp8 target: MSA top-16 block selection with the forced +init / local blocks and per-row valid-block masks.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_select_blocks import ( # noqa: E501 + m3_select_blocks as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_select_blocks requires a CUDA device" + +TOPK, INIT, LOCAL = 16, 0, 1 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + n_blocks, total_q = spec["n_blocks"], spec["total_q"] + # Distinct scores (a permutation) so the selection has no ties to break. + scores = ( + torch.randperm(n_blocks * total_q, generator=g, device="cuda") + .float() + .view(1, n_blocks, total_q) + ) + if spec["mixed"]: + n_valid = torch.randint( + 17, n_blocks + 1, (total_q,), generator=g, device="cuda", dtype=torch.int32 + ) + else: + n_valid = torch.full((total_q,), n_blocks, dtype=torch.int32, device="cuda") + with validating(op): + out = op(scores, TOPK, n_valid, INIT, LOCAL, True) + ref = op.reference(scores, TOPK, n_valid, INIT, LOCAL, True) + assert out.shape == ref.shape + assert torch.equal(out, ref) + assert out.permute(1, 0, 2).is_contiguous(), "head-major backing" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_long.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_long.py new file mode 100644 index 000000000000..3e69ac2ac780 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_long.py @@ -0,0 +1,179 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the tp8 (Rubin) m3_select_long entry: top-16 MSA block selection for long decode rows, 1 KV head.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_select_long import ( # noqa: E501 + TOPK, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_select_long import ( # noqa: E501 + m3_select_long as op, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + select_blocks_from_maxscore, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +assert torch.cuda.is_available(), "m3_select_long requires a CUDA device" + +# Valid-block counts that land a row on each of the kernel's paths: <= 16 (all picked), the 2-, 4- and 9-key +# widths, past the row (clamped), and zero (a CUDA-graph padding row). +_MIXED_VALID = (0, 5, 16, 17, 200, 2048, 2049, 4096, 4097, 8193, 9000, 1, 300, 6000, 129, 3000) + + +def _scores(spec, g): + nb, tq = spec["num_blocks"], spec["total_q"] + s = torch.randn(spec["num_kv_heads"], nb, tq, generator=g, device="cuda", dtype=torch.float32) + kind = spec["scores"] + if kind == "ties": + # Even tokens: a flat row with ten winners, so thousands of keys tie at the 16th-largest score (the + # re-read path). Odd tokens: scores on a coarse grid, so ties are broken by block index. + s[:, :, 0::2] = 0.25 + winners = torch.randint(0, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda") + s[:, :, 0::2].scatter_(1, winners[:, :, 0::2], 3.0) + s[:, :, 1::2] = (s[:, :, 1::2] * 2).round() / 2 + elif kind == "ties1500": + # Every row: 10 winners above, ~1500 blocks tied at the 16th score, the rest below; the in-tree op stages up to + # 2048 boundary candidates, so it must break this tie to the smaller block like this op. + s = torch.full_like(s, -1.0) + s[:, :1500, :] = 0.25 + winners = torch.randint( + 1500, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda" + ) + s.scatter_(1, winners, 3.0) + elif kind == "nonfinite": + s[:, 7::50, :] = float("inf") + s[:, 11::70, :] = float("nan") + s[:, 3::40, :] = -0.0 + s[:, 4::40, :] = 0.0 + # Odd tokens: almost everything -inf, so picks run past the finite scores. + s[:, :, 1::2] = float("-inf") + s[:, 0:5, 1::2] = 1.0 + return s + + +def _valid(spec, g): + tq, nb = spec["total_q"], spec["num_blocks"] + if spec["valid"] == "mixed": + vals = [_MIXED_VALID[i % len(_MIXED_VALID)] for i in range(tq)] + else: + vals = (nb - torch.randint(0, 12, (tq,), generator=g, device="cuda")).tolist() + vals[0] = nb + return torch.tensor(vals, dtype=torch.int32, device="cuda") + + +def _case(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + scores = _scores(spec, g) + nvb = _valid(spec, g) + out = torch.empty( + spec["num_kv_heads"], spec["total_q"], TOPK, dtype=torch.int32, device="cuda" + ).permute(1, 0, 2) + return scores, nvb, out, spec["init_blocks"], spec["local_blocks"], False + + +def _mass_tie_rows(scores, nvb, init_blocks, local_blocks): + """[total_q, num_kv_heads] mask of rows where more than 2048 valid blocks share the 16th-largest score.""" + num_kv_heads, num_blocks, total_q = scores.shape + blocks = torch.arange(num_blocks, device=scores.device) + raw = nvb[:total_q].long() + valid = raw.clamp(0, num_blocks) + forced = scores.permute(2, 0, 1).clone() # [total_q, heads, blocks] + forced = torch.where(blocks < init_blocks, 1.0e30, forced) + forced = torch.where(blocks >= (raw - local_blocks).clamp_min(0)[:, None, None], 1.0e29, forced) + in_row = (blocks[None, :] < valid[:, None])[:, None, :] + ranked = torch.where(in_row, forced, float("-inf")).sort(dim=-1, descending=True).values + kth = ranked[..., min(TOPK, num_blocks) - 1 : min(TOPK, num_blocks)] + return ((forced == kth) & in_row).sum(dim=-1) > 2048 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + certifying(op) + out = op(*args) + ref = op.reference(*args) + + # Selection is an exact claim: the same blocks or a different attention. + assert out.dtype is torch.int32 + assert torch.equal(out, ref), "selected blocks differ from the reference" + # And it is the in-tree selector's answer, bit for bit (the op the target falls back to) -- except on rows + # where more than 2048 blocks tie at the 16th-largest score: there the in-tree op does not break the tie to + # the smallest blocks, while this op and the reference do. + scores, nvb, _, init_blocks, local_blocks, _ = args + in_tree = select_blocks_from_maxscore( + scores, + topk=TOPK, + n_valid_blocks=nvb, + init_blocks=init_blocks, + local_blocks=local_blocks, + head_major_output=True, + ) + mass_tie = _mass_tie_rows(scores, nvb, init_blocks, local_blocks) + assert torch.equal(out[~mass_tie], in_tree[~mass_tie]), ( + "selected blocks differ from the in-tree minimax_m3_select_blocks" + ) + + +def test_pdl_launch_selects_the_same() -> None: + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=1) + pdl_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, False) + op(scores, nvb, pdl_out, init_blocks, local_blocks, True) + assert torch.equal(out, pdl_out) + + +def test_early_trigger_selects_the_same() -> None: + """Releasing the dependent grid right after the select's own wait changes only when the next kernel may launch.""" + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=2) + early_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False) + op(scores, nvb, early_out, init_blocks, local_blocks, True, True) + assert torch.equal(out, early_out) + + +@pytest.mark.parametrize("cell", [0, 1, 2]) +def test_trigger_phase_selects_the_same(cell: int) -> None: + """Releasing the dependent grid once the row's scores are loaded changes only when the next kernel may launch.""" + spec = op.CELLS[cell].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=3 + cell) + loaded_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False, 0) + op(scores, nvb, loaded_out, init_blocks, local_blocks, True, False, 1) + assert torch.equal(out, loaded_out) + + +def test_op_refuses_a_row_past_its_width() -> None: + scores = torch.zeros(1, 9217, 1, device="cuda") + nvb = torch.ones(1, dtype=torch.int32, device="cuda") + out = torch.empty(1, 1, TOPK, dtype=torch.int32, device="cuda") + with pytest.raises(RuntimeError, match="at most 9216"): + op(scores, nvb, out, 0, 1, False) + + +def test_guard_refuses_negative_forcing() -> None: + args = list(_case(op.CELLS[0].spec, seed=2)) + args[4] = -1 + with pytest.raises(AssertionError, match="non-negative"): + with validating(op): + op(*args) + + +def test_guard_refuses_out_aliasing_scores() -> None: + scores, nvb, _, init_blocks, local_blocks, _ = _case(op.CELLS[0].spec, seed=3) + aliased = ( + scores.view(torch.int32) + .reshape(-1)[: scores.shape[2] * TOPK] + .view(scores.shape[2], 1, TOPK) + ) + with pytest.raises(AssertionError, match="alias"): + with validating(op): + op(scores, nvb, aliased, init_blocks, local_blocks, False) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_long_dsl.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_long_dsl.py new file mode 100644 index 000000000000..ca7b03cee5f2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_select_long_dsl.py @@ -0,0 +1,191 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the tp8 (Rubin) m3_select_long_dsl entry: the long-row top-16 select from one 384-thread CTA per row, +against the reference and bit for bit against the 1024-thread m3_select_long on every cell.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( + m3_select_long_dsl as select_kernels, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_select_long import ( # noqa: E501 + TOPK, + m3_select_long, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_select_long_dsl import ( # noqa: E501 + m3_select_long_dsl as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +assert torch.cuda.is_available(), "m3_select_long_dsl requires a CUDA device" + +# Valid-block counts that land a row on each of the kernel's paths: <= 16 (all picked), the 4-, 11- and 24-key +# widths and their edges, past the row (clamped), and zero (a CUDA-graph padding row). +_MIXED_VALID = (0, 5, 16, 17, 200, 1536, 1537, 4224, 4225, 8193, 9000, 1, 300, 6000, 129, 3000) + + +def _scores(spec, g): + nb, tq = spec["num_blocks"], spec["total_q"] + s = torch.randn(spec["num_kv_heads"], nb, tq, generator=g, device="cuda", dtype=torch.float32) + kind = spec["scores"] + if kind == "ties": + # Even tokens: a flat row with ten winners, so thousands of keys tie at the 16th-largest score (the + # whole-row re-rank). Odd tokens: scores on a coarse grid, so ties are broken by block index. + s[:, :, 0::2] = 0.25 + winners = torch.randint(0, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda") + s[:, :, 0::2].scatter_(1, winners[:, :, 0::2], 3.0) + s[:, :, 1::2] = (s[:, :, 1::2] * 2).round() / 2 + elif kind == "ties1500": + s = torch.full_like(s, -1.0) + s[:, :1500, :] = 0.25 + winners = torch.randint( + 1500, nb, (spec["num_kv_heads"], 10, tq), generator=g, device="cuda" + ) + s.scatter_(1, winners, 3.0) + elif kind == "nonfinite": + s[:, 7::50, :] = float("inf") + s[:, 11::70, :] = float("nan") + s[:, 3::40, :] = -0.0 + s[:, 4::40, :] = 0.0 + s[:, :, 1::2] = float("-inf") + s[:, 0:5, 1::2] = 1.0 + return s + + +def _valid(spec, g): + tq, nb = spec["total_q"], spec["num_blocks"] + if spec["valid"] == "mixed": + vals = [_MIXED_VALID[i % len(_MIXED_VALID)] for i in range(tq)] + else: + vals = (nb - torch.randint(0, 12, (tq,), generator=g, device="cuda")).tolist() + vals[0] = nb + return torch.tensor(vals, dtype=torch.int32, device="cuda") + + +def _case(spec, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + scores = _scores(spec, g) + nvb = _valid(spec, g) + out = torch.empty( + spec["num_kv_heads"], spec["total_q"], TOPK, dtype=torch.int32, device="cuda" + ).permute(1, 0, 2) + return scores, nvb, out, spec["init_blocks"], spec["local_blocks"], False + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + certifying(op) + out = op(*args) + ref = op.reference(*args) + assert out.dtype is torch.int32 + assert torch.equal(out, ref), "selected blocks differ from the reference" + # Every row, mass ties included, is the 1024-thread kernel's answer. + scores, nvb, _, init_blocks, local_blocks, _ = args + long_out = torch.empty_like(out) + m3_select_long(scores, nvb, long_out, init_blocks, local_blocks, False) + assert torch.equal(out, long_out), "selected blocks differ from m3_select_long" + + +@pytest.mark.parametrize("layout", ["kv_blocks_q", "kv_q_blocks"]) +@pytest.mark.parametrize("valid", [17, 1294, 2524, 3907, 9216]) +def test_layouts_and_widths_match_the_1024_thread_select(layout, valid) -> None: + """Both score layouts the scorer can hand over, at a valid count in each key width.""" + g = torch.Generator(device="cuda").manual_seed(valid) + nb = max(valid, 129) + s = torch.randn(1, nb, 8, generator=g, device="cuda") + if layout == "kv_q_blocks": + s = s.permute(0, 2, 1).contiguous().permute(0, 2, 1) + nvb = torch.tensor([valid - (t % 2) for t in range(8)], dtype=torch.int32, device="cuda") + out = torch.empty(1, 8, TOPK, dtype=torch.int32, device="cuda").permute(1, 0, 2) + ref = torch.empty_like(out) + op(s, nvb, out, 0, 1, False) + m3_select_long(s, nvb, ref, 0, 1, False) + assert torch.equal(out, ref) + + +def test_pdl_launch_selects_the_same() -> None: + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=1) + pdl_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, False) + op(scores, nvb, pdl_out, init_blocks, local_blocks, True) + assert torch.equal(out, pdl_out) + + +def test_early_trigger_selects_the_same() -> None: + spec = op.CELLS[2].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=2) + early_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False) + op(scores, nvb, early_out, init_blocks, local_blocks, True, True) + assert torch.equal(out, early_out) + + +@pytest.mark.parametrize("cell", [0, 1, 2]) +def test_trigger_phase_selects_the_same(cell: int) -> None: + spec = op.CELLS[cell].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=3 + cell) + loaded_out = torch.empty_like(out) + op(scores, nvb, out, init_blocks, local_blocks, True, False, 0) + op(scores, nvb, loaded_out, init_blocks, local_blocks, True, False, 1) + assert torch.equal(out, loaded_out) + + +@pytest.mark.parametrize("early_trigger", [False, True]) +@pytest.mark.parametrize("cell", range(len(op.CELLS))) +def test_valid_count_read_before_the_wait_selects_the_same(cell: int, early_trigger: bool) -> None: + """kernels/m3_select_long_dsl.SELECT_N_VALID_PREWAIT: the kernel that reads its row's valid-block count before the + grid wait selects exactly the blocks of the one that reads it after, on every certified cell (padding, clamped and + tied rows).""" + phase = 0 if early_trigger else 1 + spec = op.CELLS[cell].spec + scores, nvb, before, init_blocks, local_blocks, _ = _case(spec, seed=11 + cell) + after = torch.empty_like(before) + select_kernels._compile((True, early_trigger, phase, True))( + scores, nvb, before, init_blocks, local_blocks + ) + select_kernels._compile((True, early_trigger, phase, False))( + scores, nvb, after, init_blocks, local_blocks + ) + assert torch.equal(before, after) + + +def test_graph_replay_selects_the_same() -> None: + spec = op.CELLS[1].spec + scores, nvb, out, init_blocks, local_blocks, _ = _case(spec, seed=9) + ref = op.reference(scores, nvb, out, init_blocks, local_blocks) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + op(scores, nvb, out, init_blocks, local_blocks, True, False, 1) + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(scores, nvb, out, init_blocks, local_blocks, True, False, 1) + for _ in range(3): + out.fill_(-7) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out, ref) + + +def test_op_refuses_a_row_past_its_width() -> None: + scores = torch.zeros(1, 9217, 1, device="cuda") + nvb = torch.ones(1, dtype=torch.int32, device="cuda") + out = torch.empty(1, 1, TOPK, dtype=torch.int32, device="cuda") + with pytest.raises(RuntimeError, match="at most 9216"): + op(scores, nvb, out, 0, 1, False) + + +def test_guard_refuses_negative_forcing() -> None: + args = list(_case(op.CELLS[0].spec, seed=2)) + args[4] = -1 + with pytest.raises(AssertionError, match="non-negative"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_sparse_decode.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_sparse_decode.py new file mode 100644 index 000000000000..5599fc6efc74 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_sparse_decode.py @@ -0,0 +1,626 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_sparse_decode entry (the target's Triton block-sparse decode and its routes) at the TP8 heads. + +Besides the band of every route around a float64 reference, the Triton merge's MXFP8 o_proj hand-off (published to +``m3_prequant.SLOT``) is held to ``torch.ops.trtllm.mxfp8_quantize(output, True)`` bit for bit: on identity steps and on +selected steps with the long-context range killed (the routes the o_proj consumes when LC8 does not run). +""" + +import zlib +from types import SimpleNamespace + +import pytest +import torch + +import tensorrt_llm # noqa: F401 (registers torch.ops.trtllm) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( + modeling, # noqa: E501 +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_attn_lc8, + m3_prequant, + m3_report, + triton_sparse_decode, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_sparse_decode import ( # noqa: E501 + m3_sparse_decode as op, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.triton_sparse_decode import ( + minimax_m3_sparse_attn_decode as stock_decode, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_nvfp4_cache import ( # noqa: E402 — needs the path declared above + HEAD_DIM, + NUM_HEADS, + build_step, +) +from _validating import assert_within_ulp, certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + +# Band, in BF16 ulps (2^-8) of each [8, 128] row's largest magnitude, and of the relative RMS. Every route +# computes from exact operands (E4M3 q and E2M1 x E4M3 K/V are exact in FP16), so the gap to the float64 reference +# is the BF16 rounding on both sides (one ulp of the row at most), the FP16 probabilities before PV, the split-K +# partials kept in FP32 and merged by their log2-sum-exp, and approximate exp2/log2: 2 ulps per element, the two +# roundings (each at most half an ulp) dominating the RMS, so half an ulp. (GB300 TP4: at most 1.80 ulps per element +# and 0.20 ulps RMS over the cells.) +_ELEMENT_ULP = 2.0 +_RMS_ULP = 0.5 + + +def _call(case, output, identity): + return op( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + case.sm_scale, + output, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + identity, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + case = build_step( + spec["seq_lens"], + spec["decode_query_len"], + identity=spec["identity"], + seed=zlib.crc32(cell.why.encode()) % 2**31, + ) + output = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + certifying(op) + _call(case, output, spec["identity"]) + ref = op.reference( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + case.sm_scale, + output, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + spec["identity"], + ) + assert torch.isfinite(output.float()).all(), "padding rows must be zeros, not NaN" + assert_within_ulp(output, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +def test_identity_flag_does_not_change_the_answer() -> None: + """On an identity table the identity route (the per-row producer) and the selected-table route (the + long-context kernel) compute the same attention.""" + case = build_step((2048, 130), 4, identity=True, seed=17) + outs = [] + for identity in (True, False): + output = torch.empty( + case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda" + ) + _call(case, output, identity) + outs.append(output) + assert_within_ulp(outs[0], outs[1], element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +def test_guard_refuses_a_padded_selection_slot() -> None: + """The Triton producers index the block table with every one of a token's first min(16, blocks) ids, + -1 included.""" + case = build_step((1500,), 1, identity=True, seed=23) + case.topk[0, 0, 3] = -1 + output = torch.empty(1, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + with pytest.raises(AssertionError, match="blocks of the request"): + with validating(op): + _call(case, output, True) + + +def test_guard_refuses_a_false_identity_promise() -> None: + """identity_blocks lets the multi-query producer read one table for the whole request.""" + case = build_step((1500, 1600, 1700, 1800, 1500, 1600, 1700, 1800), 4, identity=True, seed=29) + case.topk[0, 5, :2] = case.topk[0, 5, :2].flip(0) + output = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + with pytest.raises(AssertionError, match="identity_blocks promises"): + with validating(op): + _call(case, output, True) + + +def _swizzled_scale_offsets(rows: int, width: int) -> torch.Tensor: + """Offsets of rows [0, rows)'s UE8M0 scales in the 128x4-swizzled layout of mxfp8_quantize(x, True).""" + k_sf = width // 32 + k_tiles = (k_sf + 3) // 4 + r = torch.arange(rows, device="cuda").view(-1, 1) + k = torch.arange(k_sf, device="cuda").view(1, -1) + offsets = ( + (r // 128) * k_tiles * 512 + (k // 4) * 512 + (r % 32) * 16 + ((r % 128) // 32) * 4 + k % 4 + ) + return offsets.reshape(-1) + + +def _mxfp8_handoff_matches_stock(output) -> None: + """The real rows' E4M3 bytes and UE8M0 scales equal mxfp8_quantize's. The merge writes only the step's rows into + its persistent 128-row scale tile; the padding rows' scales are never read by the o_proj and are not compared.""" + rows = output.shape[0] + flat = output.view(rows, -1) + taken = m3_prequant.SLOT.take(flat) + assert taken is not None, "the route published no MXFP8 hand-off for its output" + q8, sf = taken + stock_q, stock_sf = torch.ops.trtllm.mxfp8_quantize(flat, True) + assert torch.equal( + q8.view(torch.uint8).reshape(rows, -1), stock_q.view(torch.uint8).reshape(rows, -1) + ) + at = _swizzled_scale_offsets(rows, flat.shape[1]) + assert torch.equal( + sf.view(torch.uint8).reshape(-1)[at], stock_sf.view(torch.uint8).reshape(-1)[at] + ) + + +@pytest.mark.parametrize("seq_lens", [(1900,), (2048, 0), (700, 1500)]) +def test_identity_merge_mxfp8_handoff_is_stock_mxfp8_quantize(seq_lens) -> None: + """The Triton merge writes the o_proj hand-off (E4M3 rows, 128x4-swizzled UE8M0 scales) as mxfp8_quantize.""" + case = build_step(seq_lens, 4, identity=True, seed=31 + len(seq_lens)) + output = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + _call(case, output, True) + _mxfp8_handoff_matches_stock(output) + + +@pytest.mark.parametrize("seq_lens", [(40000,), (9000, 40000)]) +def test_killed_long_context_route(seq_lens) -> None: + """With SPARSE_DECODE_LC8 killed a selected table runs the Triton producer + merge: in the band of the reference, + and its MXFP8 hand-off is mxfp8_quantize of the rows it wrote.""" + case = build_step(seq_lens, 4, identity=False, seed=41 + len(seq_lens)) + landed = triton_sparse_decode.LC8_KEYS + triton_sparse_decode.LC8_KEYS = frozenset() + try: + output = torch.empty( + case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda" + ) + _call(case, output, False) + _mxfp8_handoff_matches_stock(output) + finally: + triton_sparse_decode.LC8_KEYS = landed + ref = op.reference( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + case.sm_scale, + output, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + False, + ) + assert_within_ulp(output, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +def test_identity_route_is_the_certified_producer_range() -> None: + """Every identity step at the TP8 heads (one KV head) takes the per-row producer below 17 rows and the multi-query + producer from 17 rows; modeling.SPARSE_DECODE_PRODUCER covers both, and the cells certify both.""" + assert modeling.SPARSE_DECODE_PRODUCER.covers("per_row") + assert modeling.SPARSE_DECODE_PRODUCER.covers("multi_query") + seen = set() + for cell in op.CELLS: + keys = tuple(modeling._sparse_decode_producer(cell.spec)) + rows = cell.spec["decode_query_len"] * len(cell.spec["seq_lens"]) + want = (("multi_query" if rows >= 17 else "per_row"),) if cell.spec["identity"] else () + assert keys == want, (cell.why, keys) + seen.update(keys) + assert seen == {"per_row", "multi_query"} + + +_IDENTITY_LENS = (1900, 2048, 700, 1500, 1200, 2000, 300, 1024) +_SELECTED_LENS = (9000, 40000, 20000, 5000, 7000, 12000, 30000, 3000) + + +def _stock(case, output): + stock_decode( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + sm_scale=case.sm_scale, + output=output, + decode_query_len=case.decode_query_len, + k_block_scale=case.k_block_scale, + v_block_scale=case.v_block_scale, + k_global_scale=case.k_global_scale, + v_global_scale=case.v_global_scale, + ) + + +@pytest.mark.parametrize("identity", [True, False], ids=["identity", "selected_lc_killed"]) +@pytest.mark.parametrize("requests", [1, 2, 4, 8]) +def test_triton_route_is_bitwise_the_stock_decode(requests, identity) -> None: + """The target's per-row Triton producer + split-K merge writes the stock in-tree MSA decode's rows bit for bit: on + SM107 it takes stock's split-K factor at every row count (4 / 8 / 16 / 32 verify rows here). An identity step of + at least SPARSE_DECODE_MQ_MIN_TOKENS rows takes the multi-query producer (one KV head), which stock does not run: + exact-class instead, in the band of the float64 reference and the same bits on every run.""" + lens = (_IDENTITY_LENS if identity else _SELECTED_LENS)[:requests] + case = build_step(lens, 4, identity=identity, seed=50 + requests + 100 * identity) + rows = case.q.shape[0] + multi_query = identity and rows >= modeling.SPARSE_DECODE_MQ_MIN_TOKENS + target = torch.empty(rows, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + again = torch.empty_like(target) + stock = torch.empty_like(target) + landed = triton_sparse_decode.LC8_KEYS + triton_sparse_decode.LC8_KEYS = frozenset() + try: + _call(case, target, identity) + _call(case, again, identity) + finally: + triton_sparse_decode.LC8_KEYS = landed + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), again.view(torch.int16)), "run to run" + if not multi_query: + _stock(case, stock) + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)) + return + ref = op.reference( + case.q, + case.k_paged, + case.v_paged, + case.topk, + case.block_table, + case.seq_lens, + case.sm_scale, + target, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + True, + ) + assert_within_ulp(target, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +def test_plain_route_is_bitwise_the_stock_decode() -> None: + """Above MergeMxfp8.MAX_ROWS (here 160 verify rows) the Triton producer + merge writes no hand-off; with + SPARSE_DECODE_LC8 killed (its "plain" key covers these rows) it writes the stock decode's rows bit for bit.""" + case = build_step((3000,) * 40, 4, identity=False, seed=77) + target = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + stock = torch.empty_like(target) + landed = triton_sparse_decode.LC8_KEYS + triton_sparse_decode.LC8_KEYS = frozenset() + try: + _call(case, target, False) + finally: + triton_sparse_decode.LC8_KEYS = landed + _stock(case, stock) + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)) + + +@pytest.mark.parametrize("identity", [True, False], ids=["triton_identity", "lc8_selected"]) +def test_handoff_kill_publishes_nothing_and_keeps_the_rows(identity) -> None: + """MERGE_MXFP8_HANDOFF False: no route publishes the MXFP8 hand-off (the o_proj then quantizes its BF16 input), and + the attention rows are the same bits as with the hand-off.""" + case = build_step((1900,) if identity else (40000,), 4, identity=identity, seed=91 + identity) + on = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + off = torch.empty_like(on) + _call(case, on, identity) + assert m3_prequant.SLOT.take(on.view(on.shape[0], -1)) is not None + triton_sparse_decode.MERGE_MXFP8_HANDOFF = False + try: + _call(case, off, identity) + assert m3_prequant.SLOT.take(off.view(off.shape[0], -1)) is None + finally: + triton_sparse_decode.MERGE_MXFP8_HANDOFF = True + assert torch.equal(on.view(torch.int16), off.view(torch.int16)) + + +def _short_lens(requests: int, g: torch.Generator) -> tuple: + """Short contexts of 1..16 blocks of 128 tokens (the GSM / identity-route regime), a few exactly on a block edge.""" + lens = torch.randint(4, 16 * 128 + 1, (requests,), generator=g).tolist() + if requests > 1: + lens[0] = 128 * int(torch.randint(1, 17, (1,), generator=g)) + return tuple(lens) + + +@pytest.mark.parametrize("seed", range(12)) +@pytest.mark.parametrize("requests", [1, 2, 3, 4]) +def test_short_table_stress_is_bitwise_the_stock_decode(requests, seed) -> None: + """Short contexts (1..16 blocks) at 1..4 requests x 4 verify rows, the identity route and a random selected route + with -1 tails past each token's blocks: the target's Triton route writes the stock decode's rows bit for bit.""" + g = torch.Generator().manual_seed(1000 * requests + seed) + for identity in (True, False): + case = build_step(_short_lens(requests, g), 4, identity=identity, seed=7 * seed + requests) + target = torch.empty( + case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda" + ) + stock = torch.empty_like(target) + landed = triton_sparse_decode.LC8_KEYS + triton_sparse_decode.LC8_KEYS = frozenset() + try: + _call(case, target, identity) + finally: + triton_sparse_decode.LC8_KEYS = landed + _stock(case, stock) + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)), ( + identity, + case.seq_lens.tolist(), + ) + + +@pytest.mark.parametrize("fault", ["neg_in_window", "id_past_row", "page_out_of_pool"]) +def test_invalid_slot_fails_closed(fault) -> None: + """A corrupt decode input (a -1 inside a token's block window, an id past the block-table row, or a block-table + page outside the pool) is read as an absent block instead of a wild KV read: the rows equal the reference over the + remaining blocks.""" + case = build_step((1500, 2048, 700), 4, identity=False, seed=303) + tok, slot = 5, 2 + if fault == "neg_in_window": + case.topk[0, tok, slot] = -1 + elif fault == "id_past_row": + case.topk[0, tok, slot] = case.block_table.shape[1] + 7 + else: + req = tok // 4 + blk = int(case.topk[0, tok, slot]) + bad_page = case.k_paged.shape[0] + 1000 + case.block_table = case.block_table.clone() + case.block_table[req, blk] = bad_page + # Every token of the request that selected that block loses it. + hits = (case.topk[:, req * 4 : req * 4 + 4] == blk).nonzero().tolist() + output = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + landed = triton_sparse_decode.LC8_KEYS + triton_sparse_decode.LC8_KEYS = frozenset() + try: + _call(case, output, False) + finally: + triton_sparse_decode.LC8_KEYS = landed + torch.cuda.synchronize() + # The reference walks each token's first min(16, blocks) entries and drops -1: the corrupt slots as -1. + ref_topk = case.topk.clone() + if fault == "page_out_of_pool": + for h, t, k in hits: + ref_topk[h, req * 4 + t, k] = -1 + else: + ref_topk[0, tok, slot] = -1 + ref = op.reference( + case.q, + case.k_paged, + case.v_paged, + ref_topk, + case.block_table, + case.seq_lens, + case.sm_scale, + output, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + False, + ) + assert torch.isfinite(output.float()).all() + assert_within_ulp(output, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +def _identity_table( + seq_lens: torch.Tensor, decode_query_len: int, heads: int, lag: int = 0 +) -> torch.Tensor: + """The identity selection of every verify token, staged from lengths `lag` tokens behind seq_lens.""" + seq = (seq_lens.cpu().to(torch.int64) - lag).clamp(min=0).repeat_interleave(decode_query_len) + intra = torch.arange(seq.shape[0]) % decode_query_len + n_valid = ((seq - decode_query_len + intra + 1).clamp(min=0) + 127) // 128 + ar = torch.arange(16) + table = torch.where(ar[None, :] < n_valid[:, None], ar[None, :], -1).to(torch.int32) + return table.unsqueeze(0).expand(heads, -1, -1).contiguous().cuda() + + +# Each crosses a 128-token edge within its four verify tokens or between the stale and the true lengths. +_EDGE_LENS = (1284, 1287, 1281, 129, 132, 2048, 1028, 5, 643, 1920, 256, 1) + + +@pytest.mark.parametrize("fault", ["stale_one_step", "all_minus_one", "random_ids"]) +@pytest.mark.parametrize("requests", [1, 2, 3, 4]) +def test_identity_route_reads_its_own_lengths_not_the_table(requests, fault) -> None: + """The identity route derives block i for slot i from the kv lengths the kernel reads: a staged table that lags + the lengths by a verify step (a 128-token edge crossed), or is garbage, gives the stock decode's rows over the + true identity selection bit for bit.""" + lens = tuple(_EDGE_LENS[(requests * 5 + k) % len(_EDGE_LENS)] for k in range(requests)) + case = build_step(lens, 4, identity=True, seed=400 + requests) + heads = case.topk.shape[0] + true_table = _identity_table(case.seq_lens, 4, heads) + assert torch.equal(case.topk, true_table), ( + "build_step's identity table is the identity selection" + ) + if fault == "stale_one_step": + staged = _identity_table(case.seq_lens, 4, heads, lag=4) + elif fault == "all_minus_one": + staged = torch.full_like(true_table, -1) + else: + g = torch.Generator().manual_seed(requests) + staged = ( + torch.randint(-1, case.block_table.shape[1], true_table.shape, generator=g) + .to(torch.int32) + .cuda() + ) + rows = case.q.shape[0] + target = torch.empty(rows, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + stock = torch.empty_like(target) + _stock(case, stock) + case_staged = SimpleNamespace(**{**vars(case), "topk": staged}) + # The Triton identity route (triton_sparse_decode.LC8_IDENTITY off); the LC8 identity route has its own test below. + triton_sparse_decode.LC8_IDENTITY = False + try: + _call(case_staged, target, True) + finally: + triton_sparse_decode.LC8_IDENTITY = True + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), stock.view(torch.int16)), ( + fault, + case.seq_lens.tolist(), + ) + + +@pytest.mark.parametrize("fault", ["none", "stale_one_step", "all_minus_one", "random_ids"]) +@pytest.mark.parametrize("requests", [1, 2, 3, 4]) +def test_lc8_identity_route_reads_its_own_lengths_not_the_table( + requests, fault, monkeypatch +) -> None: + """triton_sparse_decode.LC8_IDENTITY: an identity step runs the LC8, which takes block i for CTA i from the kv + lengths it reads. Whatever the staged table holds (true, lagging a verify step, all -1, garbage), the rows and the + MXFP8 hand-off are the LC8's on the true identity table (the selected-table route) bit for bit, and in the band of + the float64 reference.""" + lens = tuple(_EDGE_LENS[(requests * 5 + k) % len(_EDGE_LENS)] for k in range(requests)) + case = build_step(lens, 4, identity=True, seed=700 + requests) + heads = case.topk.shape[0] + true_table = _identity_table(case.seq_lens, 4, heads) + if fault == "none": + staged = true_table.clone() + elif fault == "stale_one_step": + staged = _identity_table(case.seq_lens, 4, heads, lag=4) + elif fault == "all_minus_one": + staged = torch.full_like(true_table, -1) + else: + g = torch.Generator().manual_seed(requests) + staged = ( + torch.randint(-1, case.block_table.shape[1], true_table.shape, generator=g) + .to(torch.int32) + .cuda() + ) + rows = case.q.shape[0] + target = torch.empty(rows, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + selected = torch.empty_like(target) + case_staged = SimpleNamespace(**{**vars(case), "topk": staged}) + case_true = SimpleNamespace(**{**vars(case), "topk": true_table}) + taken = [] + monkeypatch.setattr(m3_report, "taken", lambda path, rows, what="": taken.append(path)) + _call(case_staged, target, True) + monkeypatch.undo() + # The LC8 engagement, plus its one-division merge's (m3_attn_lc8.one_div_applies) at these rows. + assert sorted(taken) == sorted( + [triton_sparse_decode.LC8_PATH] + + ([triton_sparse_decode.ONE_DIV_MERGE_PATH] if m3_attn_lc8.one_div_applies(rows) else []) + ), taken + target_handoff = m3_prequant.SLOT.take(target.view(rows, -1)) + _call(case_true, selected, False) + selected_handoff = m3_prequant.SLOT.take(selected.view(rows, -1)) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert torch.equal(target.view(torch.int16), selected.view(torch.int16)), ( + fault, + case.seq_lens.tolist(), + ) + assert (target_handoff is None) == (selected_handoff is None) + if target_handoff is not None: + for a, b in zip(target_handoff, selected_handoff): + assert torch.equal(a.view(torch.uint8), b.view(torch.uint8)) + ref = op.reference( + case.q, + case.k_paged, + case.v_paged, + true_table, + case.block_table, + case.seq_lens, + case.sm_scale, + target, + case.decode_query_len, + None, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + None, + False, + ) + assert_within_ulp(target, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + + +@pytest.mark.parametrize("requests", [1, 2]) +def test_lc8_identity_step_reads_q_after_the_wait(requests, monkeypatch) -> None: + """An identity step has no select, so the LC8 must read q after its grid wait (a route that skipped it + read q before the producer wrote it): the op gets identity and identity_step on every identity step, and the rows + are the same call after call.""" + lens = tuple(_EDGE_LENS[(requests * 3 + k) % len(_EDGE_LENS)] for k in range(requests)) + case = build_step(lens, 4, identity=True, seed=900 + requests) + true_table = _identity_table(case.seq_lens, 4, case.topk.shape[0]) + case_true = SimpleNamespace(**{**vars(case), "topk": true_table}) + rows = case.q.shape[0] + seen = [] + real = m3_attn_lc8.m3_sparse_decode_lc8 + + def spy(*args, **kwargs): + seen.append((kwargs.get("identity", False), kwargs.get("identity_step", False))) + return real(*args, **kwargs) + + monkeypatch.setattr(m3_attn_lc8, "m3_sparse_decode_lc8", spy) + out = torch.empty(rows, NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + ref = torch.empty_like(out) + _call(case_true, out, True) + _call(case_true, ref, True) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert seen[0] == (True, True), seen + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)) + + +@pytest.mark.parametrize("requests", [2, 3]) +def test_lc_fails_closed_on_ids_past_the_row(requests) -> None: + """A selected id past the token's row (a stale table, e.g. a padded row keeping a 500K request's n_valid) is + skipped, as the Triton producer skips it: the rows equal those with that slot at -1, bit for bit, and stay finite + even though the block-table tail it would reach points at a page whose block scales are NaN (0xFF).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_prequant, + ) + + lens = (165521, 700, 3000)[:requests] + case = build_step(lens, 4, identity=False, seed=1300 + requests) + dql = case.decode_query_len + good = case.topk.clone() + bad = case.topk.clone() + for tok in range(case.q.shape[0]): + req, qoff = divmod(tok, dql) + kv_len = max(int(case.seq_lens[req]) - dql + qoff + 1, 0) + nblk = (kv_len + 127) // 128 + past = nblk + 1 + if 2 <= nblk < 16 and past < case.block_table.shape[1]: + bad[:, tok, nblk - 1] = past + good[:, tok, nblk - 1] = -1 + page = int(case.block_table[req, past]) + case.k_block_scale[page].fill_(0xFF) + case.v_block_scale[page].fill_(0xFF) + assert not torch.equal(bad, good) + outs = [] + for table in (bad, good): + out = torch.empty(case.q.shape[0], NUM_HEADS, HEAD_DIM, dtype=torch.bfloat16, device="cuda") + _call(SimpleNamespace(**{**vars(case), "topk": table}), out, False) + outs.append(out) + m3_prequant.SLOT.clear() + torch.cuda.synchronize() + assert torch.isfinite(outs[0].float()).all() + assert torch.equal(outs[0].view(torch.int16), outs[1].view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_sparse_decode_lc8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_sparse_decode_lc8.py new file mode 100644 index 000000000000..9978cd63b13e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_sparse_decode_lc8.py @@ -0,0 +1,360 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_sparse_decode_lc8 entry (one-launch long-context sparse decode at the TP8 8 query heads, +``torch.ops.trtllm.minimax_m3_tp8_sparse_decode_lc8``). + +Every cell is held to the float64 reference within the derived band, its MXFP8 hand-off to the recipe bit for bit, and +the two ``prewait`` variants (staging before or after the grid wait) to each other bit for bit. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_sparse_decode_lc8 import ( # noqa: E501 + NUM_HEADS, + WIDTH, + dequant_nvfp4_pages, + mxfp8_handoff_reference, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.m3_sparse_decode_lc8 import ( # noqa: E501 + m3_sparse_decode_lc8 as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_nvfp4_cache import NUM_HEADS as CASE_HEADS # noqa: E402 — needs the path declared above +from _m3_nvfp4_cache import build_step # noqa: E402 +from _validating import assert_within_ulp, certifying, validating # noqa: E402, F401 + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + +assert torch.cuda.is_available(), "m3_sparse_decode_lc8 requires a CUDA device" + +#: Hand-off buffers are sized as MERGE_MXFP8 sizes them. +_HANDOFF_ROWS = 128 + +# Band, in BF16 ulps (2^-8) of each output row's largest magnitude, and of the relative RMS. +# +# The reference is float64 attention over the exactly dequantized blocks, rounded once to BF16. The kernel's +# operands are exact too (E4M3 q and E2M1 x E4M3 K/V are exact in FP16), so what separates the two is: the final +# BF16 rounding on both sides (up to one ulp of the element, at most one of the row's scale), the softmax +# probabilities rounded to FP16 before PV (2^-11 relative each, averaging down over the block), ex2.approx / +# lg2.approx (~2^-22) and FP32 accumulation order. Hence 2 ulps per element; the RMS is dominated by the two +# roundings (each at most half an ulp, about 0.3 in aggregate), so half an ulp. None of these depends on the head +# count or on the MMA orientation: the tp4 target's 16-head kernel measured at most 1.91 ulps per element and 0.25 +# ulps RMS on GB300 over the same cells. +_ELEMENT_ULP = 2.0 +_RMS_ULP = 0.5 + + +def _args(case, out, q8, sf, linear_softmax, prewait, num_sf_k_tiles=WIDTH // 128): + return ( + case.q.view(torch.uint8), + case.k_paged, + case.v_paged, + case.k_block_scale, + case.v_block_scale, + case.k_global_scale, + case.v_global_scale, + case.topk[0], + case.block_table, + case.seq_lens, + out, + q8, + sf, + num_sf_k_tiles, + case.decode_query_len, + case.sm_scale, + linear_softmax, + False, + prewait, + ) + + +def _buffers(total_q, mxfp8): + out = torch.empty(total_q, WIDTH, dtype=torch.bfloat16, device="cuda") + if not mxfp8: + return out, None, None + q8 = torch.zeros(_HANDOFF_ROWS, WIDTH, dtype=torch.uint8, device="cuda") + sf = torch.zeros(_HANDOFF_ROWS * (WIDTH // 32), dtype=torch.uint8, device="cuda") + return out, q8, sf + + +def _case(spec, seed): + return build_step(spec["seq_lens"], spec["decode_query_len"], identity=False, seed=seed) + + +def test_the_cases_are_the_tp8_geometry() -> None: + assert CASE_HEADS == NUM_HEADS == 8 and WIDTH == 1024 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + spec = cell.spec + case = _case(spec, zlib.crc32(cell.why.encode()) % 2**31) + total_q = case.q.shape[0] + out, q8, sf = _buffers(total_q, spec["mxfp8"]) + args = _args(case, out, q8, sf, spec["linear_softmax"], spec["prewait"]) + certifying(op) + op(*args) + ref = op.reference(*args) + assert_within_ulp(out, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + # Padding rows attend nothing and must be exact zeros, not NaN. + assert torch.isfinite(out.float()).all() + if spec["mxfp8"]: + # The hand-off is a deterministic function of the op's own BF16 rows: exact. + ref_q8, ref_sf = mxfp8_handoff_reference(out, _HANDOFF_ROWS) + assert torch.equal(q8[:total_q], ref_q8), ( + "MXFP8 rows differ from the recipe applied to the BF16 rows" + ) + assert torch.equal(sf, ref_sf), "UE8M0 scales differ from the recipe / swizzle" + + # The other staging variant: the same arithmetic, so the same bits (rows and hand-off). + other_out, other_q8, other_sf = _buffers(total_q, spec["mxfp8"]) + op(*_args(case, other_out, other_q8, other_sf, spec["linear_softmax"], not spec["prewait"])) + assert torch.equal(other_out.view(torch.int16), out.view(torch.int16)), ( + "prewait and plain staging wrote different rows" + ) + if spec["mxfp8"]: + assert torch.equal(other_q8, q8) and torch.equal(other_sf, sf) + + +def test_reference_dequant_matches_the_quantizer() -> None: + """The reference reads the paged bytes through its own layout claims (K scales token-major, V scales in + 4x4 token quads). Hold them to the quantizer's own flat output: every slot was written in order, so slot + ``s`` is page ``s // 128``, token ``s % 128``, and the flat scales are token-major by construction.""" + case = build_step((3000,), 1, identity=False, seed=5) + pages = torch.arange(case.k_paged.shape[0], device="cuda") + views = ((case.k_paged, case.k_block_scale, False), (case.v_paged, case.v_block_scale, True)) + for role, (data, scale, quad) in enumerate(views): + flat_data, flat_scale = torch.ops.trtllm.fp4_quantize( + case.source[role].reshape(case.num_slots, 1, -1).contiguous(), + case.quant[role + 1 : role + 2], + 16, + False, + False, + ) + expected = dequant_nvfp4_pages( + flat_data.view(torch.uint8).reshape(-1, 1, 128, 64), + flat_scale.view(torch.uint8).reshape(-1, 1, 128, 8), + pages, + quad_scales=False, + ) + got = dequant_nvfp4_pages(data, scale, pages, quad_scales=quad) + assert torch.equal(got, expected), ( + f"role {role}: paged dequantization disagrees with the quantizer" + ) + + +def test_the_variants_agree() -> None: + """The two softmax recurrences, the hand-off switch and the staging must not change the BF16 rows beyond + rounding; the hand-off switch and the staging not at all.""" + case = build_step((5000, 12345), 4, identity=False, seed=11) + outs = {} + for linear in (False, True): + for mxfp8 in (False, True): + for prewait in (False, True): + out, q8, sf = _buffers(case.q.shape[0], mxfp8) + op(*_args(case, out, q8, sf, linear, prewait)) + outs[(linear, mxfp8, prewait)] = out + base = outs[(False, False, False)] + for key, other in outs.items(): + assert_within_ulp(other, base, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + same_recurrence = outs[(key[0], False, False)] + torch.testing.assert_close(other, same_recurrence, rtol=0, atol=0) + + +def test_guard_refuses_a_block_past_the_request() -> None: + """A selected id at or past the request's block count reads a page the token cannot attend; with every + position masked the CTA's softmax is NaN.""" + case = build_step((3000,), 1, identity=False, seed=4) + case.topk[0, 0, 0] = 40 + out, q8, sf = _buffers(1, True) + with pytest.raises(AssertionError, match="block count"): + with validating(op): + op(*_args(case, out, q8, sf, False, True)) + + +def test_guard_refuses_strided_page_rows() -> None: + """The kernel walks a page's token rows at a fixed 64-byte pitch; a padded row stride is misread.""" + case = build_step((3000,), 1, identity=False, seed=6) + wide = torch.zeros(*case.k_paged.shape[:3], 80, dtype=torch.uint8, device="cuda")[..., :64] + wide.copy_(case.k_paged) + case.k_paged = wide + out, q8, sf = _buffers(1, True) + with pytest.raises(AssertionError, match="64-byte rows"): + with validating(op): + op(*_args(case, out, q8, sf, False, True)) + + +def test_guard_refuses_a_narrow_handoff() -> None: + """The hand-off rows are 1024 wide (8 scale k-tiles); a smaller count overlaps them.""" + case = build_step((3000,), 1, identity=False, seed=7) + out, q8, sf = _buffers(1, True) + with pytest.raises(AssertionError, match="num_sf_k_tiles"): + with validating(op): + op(*_args(case, out, q8, sf, False, True, num_sf_k_tiles=4)) + + +@pytest.mark.parametrize("prewait", [True, False], ids=["prewait", "plain"]) +@pytest.mark.parametrize("mxfp8", [True, False], ids=["handoff", "no_handoff"]) +def test_run_to_run_deterministic(prewait, mxfp8) -> None: + """The exact-class reference for the TP8 sparse decode (band vs fp64, as the TP4 LC): every launch of the same + step writes the same bits, in eager repeats and in a CUDA-graph replay.""" + spec = dict(seq_lens=(165521, 5000), decode_query_len=4) + case = _case(spec, seed=91) + for linear in (False, True): + runs = [] + for _ in range(3): + out, q8, sf = _buffers(case.q.shape[0], mxfp8) + op(*_args(case, out, q8, sf, linear, prewait)) + runs.append((out, q8, sf)) + g_out, g_q8, g_sf = _buffers(case.q.shape[0], mxfp8) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(*_args(case, g_out, g_q8, g_sf, linear, prewait)) + graph.replay() + torch.cuda.synchronize() + runs.append((g_out, g_q8, g_sf)) + base = runs[0] + for other in runs[1:]: + assert torch.equal(other[0].view(torch.int16), base[0].view(torch.int16)), ( + linear, + prewait, + mxfp8, + ) + if mxfp8: + assert torch.equal(other[1], base[1]) and torch.equal(other[2], base[2]), ( + linear, + prewait, + ) + + +@pytest.mark.parametrize( + "seq_lens, decode_query_len", + [((165521,), 4), ((5000, 12345), 4), ((700,), 1), ((200000, 3000, 64, 70000), 4)], + ids=["1x165k", "2req", "short-1row", "4req-incl-64"], +) +def test_merge_once_is_bitwise_the_image_op(seq_lens, decode_query_len) -> None: + """kernels/m3_attn_lc8.MERGE_ONCE: the target's op (merge weights formed once per CTA) writes exactly the + image op's rows and hand-off, at both softmax recurrences and both staging variants, including requests with fewer + than 16 selected blocks (empty chunks, -inf LSE). With SPARSE_DECODE_LC8_PREFETCH_OPROJ it also pulls two ranges + the size of the TP8 o_proj weight and scales (one not a multiple of 16 bytes) into L2: the same bits again. With + SPARSE_DECODE_LC8_PREFETCH_NEXT_KV it records each row's selection (the first min(16, blocks) ids, -1 after) and + prefetches a next layer's recorded blocks, stale ids past a request's blocks included: the same bits, and the + record.""" + assert hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp8_sparse_decode_lc8"), ( + "the target's LC8 op is not loaded (its TP8 extension)" + ) + case = build_step( + seq_lens, decode_query_len, identity=False, seed=zlib.crc32(repr(seq_lens).encode()) + ) + o_proj_weight = torch.empty(6144, 1024, dtype=torch.uint8, device="cuda") + o_proj_scale = torch.empty(6144 * 32 + 9, dtype=torch.uint8, device="cuda") + rows = case.q.shape[0] + n1_next = torch.randint( + -1, 2 * (max(seq_lens) // 128 + 1), (128, 16), dtype=torch.int32, device="cuda" + ) + for linear in (False, True): + for prewait in (False, True): + for mxfp8 in (False, True): + n1_self = torch.full((128, 16), -7, dtype=torch.int32, device="cuda") + next_kv = ( + n1_self, + n1_next, + case.k_paged.view(torch.uint8), + case.v_paged.view(torch.uint8), + case.k_block_scale.view(torch.uint8), + case.v_block_scale.view(torch.uint8), + ) + variants = ( + (False, None, None), + (True, None, None), + (True, (o_proj_weight, o_proj_scale), None), + (True, (o_proj_weight, o_proj_scale), next_kv), + ) + got = [] + for merge_once, prefetch, n1 in variants: + out, q8, sf = _buffers(case.q.shape[0], mxfp8) + op( + *_args(case, out, q8, sf, linear, prewait), + merge_once=merge_once, + prefetch=prefetch, + next_kv=n1, + ) + got.append((out, q8, sf)) + torch.cuda.synchronize() + a, aq, asf = got[0] + for i, (b, bq, bsf) in enumerate(got[1:], 1): + assert torch.equal(a.view(torch.int16), b.view(torch.int16)), ( + linear, + prewait, + mxfp8, + i, + ) + if mxfp8: + assert torch.equal(aq, bq) and torch.equal(asf, bsf), (linear, prewait, i) + # The record: each row's first min(16, blocks) selected ids, -1 past them; rows past the step untouched. + want = case.topk[0].clone() + for tok in range(rows): + req, qoff = divmod(tok, case.decode_query_len) + kv_len = max(int(case.seq_lens[req]) - case.decode_query_len + qoff + 1, 0) + want[tok, min(16, (kv_len + 127) // 128) :] = -1 + assert torch.equal(n1_self[:rows], want), "the recorded selection" + assert bool((n1_self[rows:] == -7).all()), "rows past the step" + + +@pytest.mark.parametrize("linear", [False, True]) +@pytest.mark.parametrize("prewait", [True, False], ids=["prewait", "plain"]) +@pytest.mark.parametrize( + "seq_lens, decode_query_len", + [((165521,), 4), ((5000, 12345), 4), ((700,), 1), ((200000, 3000, 64, 70000), 4)], + ids=["1x165k", "2req", "short-1row", "4req-incl-64"], +) +def test_one_div_merge_is_deterministic_and_in_band( + seq_lens, decode_query_len, prewait, linear +) -> None: + """kernels/m3_attn_lc8.ONE_DIV_MERGE: the one-division merge writes the same rows on + every run (5 eager runs, PDL on and off) and under CUDA-graph replay (3 replays == eager), MXFP8 hand-off included; + the rows are within the float64 reference's band, and within summation-order distance of the per-chunk merge.""" + case = build_step( + seq_lens, decode_query_len, identity=False, seed=zlib.crc32(repr(seq_lens).encode()) + 3 + ) + total_q = case.q.shape[0] + + def run(out, q8, sf, use_pdl, one_div): + args = list(_args(case, out, q8, sf, linear, prewait)) + args[18] = use_pdl + op(*args, merge_once=True, one_div=one_div) + + runs = [] + for i in range(6): + out, q8, sf = _buffers(total_q, True) + run(out, q8, sf, i == 5, True) + runs.append((out, q8, sf)) + torch.cuda.synchronize() + a, aq, asf = runs[0] + for b, bq, bsf in runs[1:]: + assert torch.equal(a.view(torch.int16), b.view(torch.int16)) + assert torch.equal(aq, bq) and torch.equal(asf, bsf) + out = torch.empty_like(a) + run(out, None, None, True, True) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + run(out, None, None, True, True) + for _ in range(3): + out.zero_() + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out.view(torch.int16), a.view(torch.int16)) + per, _, _ = _buffers(total_q, False) + run(per, None, None, False, False) + ref = op.reference(*_args(case, a, None, None, linear, prewait)) + assert torch.isfinite(a.float()).all() + assert_within_ulp(a, ref, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) + assert_within_ulp(a, per, element_ulp=_ELEMENT_ULP, rms_ulp=_RMS_ULP) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_tokmap_guard.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_tokmap_guard.py new file mode 100644 index 000000000000..c4c44f7aa360 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_m3_tokmap_guard.py @@ -0,0 +1,226 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test (SM 10.7) for kernels/m3_tokmap_guard: a one-request multi-token call of +thop.trtllm_gen_generation_preprocess (rows padded to the bucket, the request's generation length 1..bucket) with a +stale token map in the workspace gives, through the guard, the bytes the op gives with a freshly built map: the +appended FP8 K/V pages and the processed query's live rows, bit for bit, eager and from one captured graph replayed for +every length. Without the guard the stale map moves rows (the negative control), and a several-request call passes +through with its own map untouched. + +The fresh-map reference forces the op's own build (a scaled RoPE type with no inv-freq cache, at rotary dim 0, so no +row is rotated either way). +""" + +import math + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_tokmap_guard as guard, +) +from tensorrt_llm.bindings.internal import thop + +assert torch.cuda.is_available(), "m3_tokmap_guard requires a CUDA device" +if torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the tp8 target is certified on SM 10.7 only", allow_module_level=True) + +H, KVH, D, PAGE = 8, 1, 128, 128 +BUCKET = 4 +_PEER_EXTRA = 777 # the second request's extra length behind the first +FP8_KV_QUANT_MODE = 128 +_DEV = torch.device("cuda") # unindexed on purpose: install() and the lookup normalize it +_TAKEN = [] + + +@pytest.fixture(scope="module", autouse=True) +def _installed(): + guard.install(_DEV, 48, _TAKEN.append) + yield + + +class _Step: + """One request (or ``nreq``) of BUCKET padded rows over a fresh FP8 paged cache, persistent across calls like a + real step's buffers.""" + + def __init__(self, kv_len: int, nreq: int = 1, seed: int = 0): + g = torch.Generator(device=_DEV).manual_seed(seed + kv_len) + self.nreq = nreq + # Per-request inputs (page tables, generation lengths, position offsets) for at least two requests, the page + # tables at the peer's longer length (_seq_with_peer): the stale map sends rows to request 1, which must have + # real entries behind it, also on the unguarded calls. + tables = max(nreq, 2) + pages = math.ceil((kv_len + _PEER_EXTRA + BUCKET) / PAGE) + self.nblocks = tables * pages + 2 + self.pool0 = ( + torch.randn(self.nblocks, 2, KVH, PAGE, D, device=_DEV, generator=g) + .clamp_(-4, 4) + .to(torch.float8_e4m3fn) + ) + self.pool = self.pool0.clone() + perm = torch.randperm(self.nblocks, device=_DEV, generator=g)[: tables * pages].view( + tables, pages + ) + self.offsets = torch.zeros(1, tables, 2, pages, dtype=torch.int32, device=_DEV) + self.offsets[0, :, 0] = 2 * perm + self.offsets[0, :, 1] = 2 * perm + 1 + self.pointers = torch.tensor([[self.pool.data_ptr(), 0]], dtype=torch.int64) + self.mapping = torch.zeros(1, 2, dtype=torch.int32) + self.qkv = torch.randn( + nreq * BUCKET, (H + 2 * KVH) * D, dtype=torch.bfloat16, device=_DEV, generator=g + ) + # Request lengths after this step's append; generation lengths (accepted rows) set per case. + self.seq = torch.full((nreq,), kv_len, dtype=torch.int32, device=_DEV) + self.gen = torch.full((tables,), BUCKET, dtype=torch.int32, device=_DEV) + self.pos = ( + torch.arange(BUCKET, dtype=torch.int32, device=_DEV).repeat(tables, 1).contiguous() + ) + self.one = torch.ones(1, dtype=torch.float32, device=_DEV) + self.ws = torch.empty(64 << 20, dtype=torch.uint8, device=_DEV) + self.kv_len = kv_len + + def call(self, op, fresh_map: bool = False): + """One generation preprocess; returns a copy of the processed query.""" + out = op( + self.qkv, + self.ws, + self.seq, + self.gen, + self.pos, + self.offsets, + self.pointers, + self.mapping, + self.one, + self.one, + None, + None, + None, + None, + 0, + 0, + H, + KVH, + D, + PAGE, + FP8_KV_QUANT_MODE, + 1 << 20, + 1 << 20, + self.nreq * BUCKET, + self.nreq, + BUCKET, + self.kv_len, + 0, + 10000.0, + 1 if fresh_map else 0, + 1.0, + 1 << 20, + 0, + 1.0, + 1.0, + False, + BUCKET, + 0, + torch.cuda.get_device_properties(0).multi_processor_count, + self.nblocks, + 2, + True, + False, + ) + return out[0].clone() + + def poison_map(self) -> None: + """A stale multi-request map in the op's one-request map slice: rows 1.. on request 1.""" + layout = thop.get_trtllm_gen_generation_workspace_layout( + self.qkv.dtype, 1, self.nreq * BUCKET, H, D, 0, KVH, 0, False, False + ) + off = int(layout["tokens_info_offset"]) + stale = torch.tensor( + [[0, 0]] + [[1, t] for t in range(BUCKET - 1)], dtype=torch.int32, device=_DEV + ) + self.ws[off : off + 8 * BUCKET].view(torch.int32).view(BUCKET, 2).copy_(stale) + + def reset(self) -> None: + self.pool.copy_(self.pool0) + + +def _seq_with_peer(step: _Step) -> None: + # A second request's length behind the first, so a row the stale map moves to request 1 reads a real length. + step.seq = torch.tensor( + [step.kv_len, step.kv_len + _PEER_EXTRA], dtype=torch.int32, device=_DEV + ) + + +@pytest.mark.parametrize("kv_len", [130, 8191, 34117]) +@pytest.mark.parametrize("n", [1, 2, 3, 4]) +def test_guard_equals_a_fresh_map(kv_len, n) -> None: + step = _Step(kv_len) + _seq_with_peer(step) + step.gen.fill_(n) + step.poison_map() + step.reset() + q_ref = step.call(guard._original, fresh_map=True) + pool_ref = step.pool.clone() + + step.poison_map() + step.reset() + q = step.call(thop.trtllm_gen_generation_preprocess) + # The processed query's n live rows (rows past the request's length are padding the op leaves unwritten). + assert torch.equal(q[:n].view(torch.uint8), q_ref[:n].view(torch.uint8)) + assert torch.equal(step.pool.view(torch.uint8), pool_ref.view(torch.uint8)) + assert BUCKET in _TAKEN + + +def test_stale_map_moves_rows_without_the_guard() -> None: + step = _Step(34117) + _seq_with_peer(step) + step.gen.fill_(BUCKET) + step.poison_map() + step.reset() + step.call(guard._original, fresh_map=True) + pool_ref = step.pool.clone() + step.poison_map() + step.reset() + step.call(guard._original) + assert not torch.equal(step.pool.view(torch.uint8), pool_ref.view(torch.uint8)) + + +def test_one_captured_graph_serves_every_length() -> None: + step = _Step(34000) + _seq_with_peer(step) + refs = {} + for n in range(1, BUCKET + 1): + step.gen.fill_(n) + step.poison_map() + step.reset() + q = step.call(guard._original, fresh_map=True) + refs[n] = (q, step.pool.clone()) + step.gen.fill_(BUCKET) + step.call(thop.trtllm_gen_generation_preprocess) # eager first, as the warmup runs it + torch.cuda.synchronize() + g = torch.cuda.CUDAGraph() + with torch.cuda.graph(g): + out = thop.trtllm_gen_generation_preprocess + q_static = step.call(out) + for n in range(1, BUCKET + 1): + step.gen.fill_(n) + step.poison_map() + step.reset() + g.replay() + torch.cuda.synchronize() + q_ref, pool_ref = refs[n] + assert torch.equal(q_static[:n].view(torch.uint8), q_ref[:n].view(torch.uint8)), n + assert torch.equal(step.pool.view(torch.uint8), pool_ref.view(torch.uint8)), n + + +def test_several_requests_pass_through() -> None: + step = _Step(8191, nreq=2) + layout = thop.get_trtllm_gen_generation_workspace_layout( + step.qkv.dtype, 2, 2 * BUCKET, H, D, 0, KVH, 0, False, False + ) + step.reset() + step.call(thop.trtllm_gen_generation_preprocess) + pool_guarded = step.pool.clone() + step.reset() + step.call(guard._original) + assert torch.equal(step.pool.view(torch.uint8), pool_guarded.view(torch.uint8)) + assert int(layout["tokens_info_size"]) >= 8 * 2 * BUCKET diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_msa_kv_update.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_msa_kv_update.py new file mode 100644 index 000000000000..d00e93fef6a7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/attention/test_modeling_v2_msa_kv_update.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the msa_kv_update entry (the one-launch MSA KV-length correction).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.msa_kv_update import ( # noqa: E501 + msa_kv_update as hook, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.msa_kv_update import ( # noqa: E501 + TOKENS_PER_BLOCK, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention.msa_kv_update import ( # noqa: E501 + msa_kv_update as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import certifying, validating # noqa: E402, F401 — needs the path declared above + +assert torch.cuda.is_available(), "msa_kv_update requires a CUDA device" + +#: Spare capacity past the live rows, as the metadata's buffers have: the op must leave it alone. +_SPARE = 3 + + +def _case(spec, seed): + """The metadata's buffers for one step, with stale values in every output so a skipped write shows.""" + g = torch.Generator(device="cpu").manual_seed(seed) + batch, qlen = spec["batch"], spec["query_len"] + total_q = batch * qlen + staged = torch.randint(qlen, 5000, (batch,), generator=g, dtype=torch.int32) + accept = spec["accept"] + if accept == "full": + kv = staged.clone() + elif accept == "page_edge": + staged = torch.tensor([130, 257, 2 * TOKENS_PER_BLOCK + qlen][:batch], dtype=torch.int32) + kv = staged - torch.tensor([0, 1, qlen - 1][:batch], dtype=torch.int32) + else: + kv = staged - torch.randint(0, qlen, (batch,), generator=g, dtype=torch.int32) + if accept == "padded": + kv[batch // 2 :] = 0 + staged[batch // 2 :] = 0 + ncols = (5000 + qlen) // TOKENS_PER_BLOCK + 2 + table = ( + torch.randperm(batch * ncols + 7, generator=g)[: batch * ncols] + .to(torch.int32) + .reshape(batch, ncols) + ) + dev = dict(device="cuda", dtype=torch.int32) + stale = -12345 + return ( + torch.cat([kv, torch.full((_SPARE,), 99, dtype=torch.int32)]).cuda(), + torch.cat([staged, torch.full((_SPARE,), 99, dtype=torch.int32)]).cuda(), + torch.full((batch + _SPARE,), qlen, **dev), + torch.arange(batch, **dev).repeat_interleave(qlen), + torch.arange(qlen, **dev).repeat(batch), + table.cuda(), + TOKENS_PER_BLOCK, + batch, + total_q, + torch.full((batch + _SPARE,), stale, **dev), + torch.full((batch + 1 + _SPARE,), stale, **dev) if spec["cu_kv"] else None, + torch.full((total_q + _SPARE,), stale, **dev), + torch.full((total_q + _SPARE,), stale, **dev) if spec["n_valid"] else None, + ) + + +def _check(args, ref): + seq, cu_kv, loc, n_valid = args[9], args[10], args[11], args[12] + batch, total_q = args[7], args[8] + ref_seq, ref_cu, ref_loc, ref_nv = ref + # Integer bookkeeping: exact, and nothing past the live rows is touched. + assert torch.equal(seq[:batch], ref_seq) + assert torch.equal(loc[:total_q], ref_loc) + assert bool((seq[batch:] == -12345).all()) and bool((loc[total_q:] == -12345).all()) + assert (cu_kv is None) == (ref_cu is None) and (n_valid is None) == (ref_nv is None) + if cu_kv is not None: + assert torch.equal(cu_kv[: batch + 1], ref_cu) + assert bool((cu_kv[batch + 1 :] == -12345).all()) + if n_valid is not None: + assert torch.equal(n_valid[:total_q], ref_nv) + assert bool((n_valid[total_q:] == -12345).all()) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31) + certifying(op) + op(*args) + _check(args, op.reference(*args)) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_the_hook_takes_every_cell(cell) -> None: + """The target's hook (what the metadata calls) takes each certified step and writes what the op does.""" + args = _case(cell.spec, seed=zlib.crc32(cell.why.encode()) % 2**31 + 1) + assert hook(*args) is True + _check(args, op.reference(*args)) + + +def test_the_hook_declines_what_the_op_would_misread() -> None: + """A strided operand is read wrongly by the op, so the hook declines it (the metadata then runs its own + torch update) and writes nothing.""" + args = list(_case(op.CELLS[2].spec, seed=5)) + args[0] = torch.stack([args[0], args[0]], dim=1)[:, 0] + before = args[11].clone() + assert hook(*args) is False + assert torch.equal(args[11], before) + + +def test_guard_refuses_a_strided_operand() -> None: + args = list(_case(op.CELLS[2].spec, seed=6)) + args[0] = torch.stack([args[0], args[0]], dim=1)[:, 0] + with pytest.raises(AssertionError, match="contiguous"): + with validating(op): + op(*args) + + +def test_guard_refuses_a_request_index_past_the_batch() -> None: + args = list(_case(op.CELLS[2].spec, seed=7)) + args[3] = args[3].clone() + args[3][-1] = args[7] + with pytest.raises(AssertionError, match="q_batch_row"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_allreduce_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_allreduce_op_matrix.py new file mode 100644 index 000000000000..8ebb79ea4d18 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_allreduce_op_matrix.py @@ -0,0 +1,454 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU certification matrix for the ``m3_allreduce`` entry, 2 ranks (SM107): its cells against the entry's reference, +and the fused draft loop's layer boundaries built on it (``M3Eagle3Draft.ar_norm``: the plain all-reduce, then +FlashInfer's fused add + RMSNorm) against stock Eagle3's, bit for bit. + +Stock ``Eagle3DecoderLayer`` all-reduces each projection plainly (BF16 out) and runs the ``RMSNorm`` module with the +residual: FlashInfer's ``fused_add_rmsnorm``, which adds in FP32 and normalizes the FP32 sum. The draft loop's +boundaries must be that path exactly, under every all-reduce strategy a single-node job can run (AUTO, ONESHOT, +TWOSHOT, NCCL; MNNVL, which the tp8 drafter's two-tray group may take, is multi-node only and not driven here), at the +drafter's shapes (H = 6144, eps = 1e-6, 1 to 32 rows), at the job's world size (2, 4 or 8 ranks on one node; the tp8 +drafter runs at 8 over two trays, which one node cannot form, and nothing here depends on the rank count). The inputs +are +random rows and rows built so a BF16 rounding of the residual sum would move the norm the most, including the +review's case: unit weights, sums 1 + 2^-8 on 6143 coordinates and 2 - 2^-8 on the last, both BF16 rounding +midpoints that round to even in opposite directions. + +Asserts the normalized rows and the residual sum bitwise equal to the stock path's, and reports the max ulp of each +against an fp64 reference (ulps of BF16 at the larger magnitude; a correctly rounded output is within 0.5). + +Its own launcher, one rank per device in CUDA_VISIBLE_DEVICES, as ``_m3_mc_allreduce_norm_op_matrix.py`` is; the +collected entry point is ``test_modeling_v2_m3_allreduce.py``. + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _m3_allreduce_op_matrix.py +""" + +import math +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Callable, List, Optional, Tuple + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "the fused all-reduce + RMSNorm requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 + +HIDDEN = 6144 +EPS = 1e-6 +TOKENS = (1, 4, 8, 32) +RESULT_TAG = "[ulp]" +"""Prefix of rank 0's result lines; the pytest entry reports them.""" + +#: World sizes the matrix runs at (one node): see ``_rank_job.WORLD_SIZES``. +WORLD_SIZES = (2, 4, 8) +RANK = 0 +WORLD = 1 + +Inputs = Tuple[List[torch.Tensor], torch.Tensor, torch.Tensor] +"""(every rank's [m, H] BF16 partial rows, the [m, H] BF16 residual, the [H] BF16 norm weight).""" + + +# ---------------------------------------------------------------- inputs (every rank builds every rank's rows) + + +def _gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def _spread(parts: List[torch.Tensor]) -> List[torch.Tensor]: + """Every rank's BF16 partial: ``parts`` on the first ranks, zeros on the rest.""" + assert len(parts) <= WORLD, (len(parts), WORLD) + zeros = [torch.zeros_like(parts[0]) for _ in range(WORLD - len(parts))] + return [p.bfloat16() for p in parts] + [z.bfloat16() for z in zeros] + + +def _random(seed: int, m: int, x_scale: float, r_scale: float, unit_w: bool) -> Inputs: + g = _gen(seed) + xs = [ + (torch.randn(m, HIDDEN, generator=g, device="cuda") * x_scale).bfloat16() + for _ in range(WORLD) + ] + residual = (torch.randn(m, HIDDEN, generator=g, device="cuda") * r_scale).bfloat16() + if unit_w: + w = torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16) + else: + w = (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).bfloat16() + return xs, residual, w + + +def _on_rank0(m: int, x: torch.Tensor) -> List[torch.Tensor]: + """``x`` as rank 0's partial, zeros elsewhere (the cross-rank sum is exact).""" + return [x.bfloat16()] + [ + torch.zeros(m, HIDDEN, device="cuda", dtype=torch.bfloat16) for _ in range(WORLD - 1) + ] + + +def _review_case(m: int) -> Inputs: + """Sums 1 + 2^-8 on the first H - 1 coordinates and 2 - 2^-8 on the last, unit weights. + + Both sums are BF16 midpoints: 1 + 2^-8 rounds (to even) down to 1 and 2 - 2^-8 up to 2, so the fused path's + norm sees a residual moved in opposite directions on the bulk and on the last coordinate. + """ + residual = torch.ones(m, HIDDEN, device="cuda") + residual[:, -1] = 2.0 - 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + return ( + _on_rank0(m, x), + residual.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _review_case_flipped(m: int) -> Inputs: + """The mirror: the bulk on odd-mantissa midpoints (1 + 3 * 2^-8 rounds up), the last one rounding down.""" + residual = torch.full((m, HIDDEN), 1.0 + 2.0**-7, device="cuda") + residual[:, -1] = 2.0 - 2.0 * 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + return ( + _on_rank0(m, x), + residual.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _midpoints(seed: int, m: int, outliers: int) -> Inputs: + """Every sum a BF16 midpoint: residual r random in [1, 2) and x = half its ulp (+2^-8), split over the ranks. + + The bulk has an odd mantissa (rounds up); ``outliers`` coordinates per row carry an even one (rounds down) and + a larger magnitude, so they weigh in the RMS while rounding the other way. + """ + g = _gen(seed) + mant = ( + torch.randint(0, 64, (m, HIDDEN), generator=g, device="cuda") * 2 + 1 + ) # odd 7-bit mantissa + r = 1.0 + mant.float() * 2.0**-7 + x = torch.full((m, HIDDEN), 2.0**-8, device="cuda") + if outliers: + cols = torch.randperm(HIDDEN, generator=g, device="cuda")[:outliers] + even = torch.randint(0, 64, (m, outliers), generator=g, device="cuda") * 2 + r[:, cols] = 8.0 * (1.0 + even.float() * 2.0**-7) + x[:, cols] = 8.0 * 2.0**-8 + # Split x over the ranks in exact BF16 pieces: x/2 on rank 0, x/4 on ranks 1 and 2 (x/2 on rank 1 at two ranks), + # 0 on the rest. + parts = [x / 2, x / 2] if WORLD == 2 else [x / 2, x / 4, x / 4] + return ( + _spread(parts), + r.bfloat16(), + torch.ones(HIDDEN, device="cuda", dtype=torch.bfloat16), + ) + + +def _cancelling(seed: int, m: int) -> Inputs: + """Large partials that cancel across the ranks (+A, -A, +B, -B + d; +A, -A + d at two or three ranks) onto a unit + residual.""" + g = _gen(seed) + a = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 64).bfloat16().float() + b = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 64).bfloat16().float() + d = (torch.randn(m, HIDDEN, generator=g, device="cuda") * 2.0**-6).bfloat16().float() + parts = [a, -a, b, (-b + d)] if WORLD >= 4 else [a, (-a + d)] + residual = (1.0 + torch.rand(m, HIDDEN, generator=g, device="cuda")).bfloat16() + w = (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).bfloat16() + return _spread(parts), residual, w + + +def _cases() -> List[Tuple[str, Callable[[int], Inputs]]]: + cases: List[Tuple[str, Callable[[int], Inputs]]] = [ + ("review: 1+2^-8 x6143, 2-2^-8 x1", _review_case), + ("review mirrored", _review_case_flipped), + ("all midpoints, no outliers", lambda m: _midpoints(11, m, 0)), + ("all midpoints, 1 outlier", lambda m: _midpoints(12, m, 1)), + ("all midpoints, 64 outliers", lambda m: _midpoints(13, m, 64)), + ("cancelling partials", lambda m: _cancelling(14, m)), + ] + for i, (xs, rs, unit) in enumerate( + ((1.0, 4.0, False), (1.0, 1.0, True), (0.05, 8.0, False), (8.0, 0.5, False)) + ): + cases.append( + ( + f"random x{xs} r{rs}{' unit-w' if unit else ''}", + lambda m, i=i, xs=xs, rs=rs, unit=unit: _random(100 + i, m, xs, rs, unit), + ) + ) + return cases + + +# ---------------------------------------------------------------- the three paths + + +def _fp64(xs: List[torch.Tensor], residual: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + s = residual.double() + for x in xs: + s = s + x.double() + rms = torch.rsqrt(s.square().mean(-1, keepdim=True) + EPS) + return s * rms * w.double() + + +def _ulp_at(v: torch.Tensor) -> torch.Tensor: + """BF16's ulp at |v| (fp64): 2^(e - 8) for |v| in [2^(e-1), 2^e), floored at the smallest normal's.""" + _, e = torch.frexp(v.abs().double()) + return torch.ldexp(torch.ones_like(v, dtype=torch.float64), torch.clamp(e - 8, min=-133)) + + +def _ulps(out: torch.Tensor, ref: torch.Tensor) -> float: + """Max |out - ref| in BF16 ulps at the larger magnitude of the two.""" + o, r = out.double(), ref.double() + return ((o - r).abs() / _ulp_at(torch.maximum(o.abs(), r.abs()))).max().item() + + +def _ulps_at(out: torch.Tensor, ref: torch.Tensor, scale: torch.Tensor) -> float: + """Max |out - ref| in BF16 ulps at the magnitude ``scale`` (the operands'), not at the result's.""" + return ((out.double() - ref.double()).abs() / _ulp_at(scale.abs())).max().item() + + +def _target(ar: Any, x: torch.Tensor, residual: torch.Tensor, w: torch.Tensor): + """The draft loop's boundary (``M3Eagle3Draft.ar_norm``): ``m3_allreduce`` plain, then the catalog's + ``flashinfer_fused_add_rmsnorm``.""" + from tensorrt_llm._torch._experimental.modeling_v2.catalog.norm.flashinfer_fused_add_rmsnorm import ( # noqa: E501 + flashinfer_fused_add_rmsnorm, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.comm.m3_allreduce import ( # noqa: E501 + m3_allreduce, + ) + + return flashinfer_fused_add_rmsnorm(m3_allreduce(x, ar, None), residual.clone(), w, EPS) + + +def _check_cells(mapping: Any) -> None: + """Every cell of ``m3_allreduce.CELLS`` (AUTO strategy, as the target builds it) against the entry's reference: + rank-order fp32 sums with one BF16 rounding of the sum and one of the norm. A strategy may round the cross-rank sum + to BF16 before adding the residual, so the residual sum is bounded at its operands' magnitude + u = ulp(max(|residual|, |sum|)): half an ulp for that rounding plus half an output ulp for each of the two final + roundings, an output ulp being at most 2u, so 2.5u (where the two cancel, the output's own ulp is far smaller). + The normed rows are the norm of the op's own residual sum, within 2 ulps of the reference norm of that sum.""" + + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.comm.m3_allreduce import ( # noqa: E501 + m3_allreduce, + ) + from tensorrt_llm._torch.distributed import AllReduce, AllReduceFusionOp, AllReduceParams + from tensorrt_llm.functional import AllReduceStrategy + + sys.path.insert(0, str(Path(__file__).resolve().parents[4])) + from _validating import validating + + ar = AllReduce(mapping=mapping, strategy=AllReduceStrategy.AUTO, dtype=torch.bfloat16) + for i, cell in enumerate(m3_allreduce.CELLS): + spec = cell.spec + xs, residual, w = _random(200 + i, spec["m"], 1.0, 4.0, False) + params = ( + None + if spec.get("plain") + else AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual.clone(), + norm_weight=w, + eps=EPS, + trigger_completion_at_end=False, + ) + ) + # The reference first: the fused path may write the updated residual into ``params.residual``. + ref = m3_allreduce.reference(xs, ar, params) + with validating(m3_allreduce): + out = m3_allreduce(xs[RANK], ar, params) + torch.cuda.synchronize() + total = sum(x.double() for x in xs) + if params is None: + err = _ulps_at(out, ref, total) + assert err <= 1.0, (cell.why, err) + else: + err = _ulps_at(out[1], ref[1], torch.maximum(total.abs(), residual.double().abs())) + assert err <= 2.5, (cell.why, "residual", err) + h = out[1].double() + own = h * torch.rsqrt(h.square().mean(-1, keepdim=True) + EPS) * w.double() + assert _ulps(out[0], own) <= 2.0, (cell.why, "normed", _ulps(out[0], own)) + dist.barrier() + if RANK == 0: + print(f"{RESULT_TAG} cell {cell.why} passed", flush=True) + + +def _stock(ar: Any, x: torch.Tensor, residual: torch.Tensor, w: torch.Tensor): + """Stock Eagle3: ``o_proj``'s plain all-reduce (BF16 out), then ``RMSNorm(h, residual)``'s FlashInfer + ``fused_add_rmsnorm`` (FP32 add, norm of the FP32 sum; in place on both).""" + h = ar(x) + res = residual.clone() + torch.ops.trtllm.flashinfer_fused_add_rmsnorm(h, res, w, EPS) + return h, res + + +# ---------------------------------------------------------------- the matrix + + +def _single_node_cluster_info() -> None: + """``TorchDist`` (the communicator under ``TLLM_DISABLE_MPI``) asks Ray for each rank's node and GPU to build its + node-local group. These ranks are spawned without Ray on one node, so every rank reports the same node and its + own device.""" + from tensorrt_llm._torch.distributed.communicator import TorchDist + + def cluster_info(self): + if self.cluster_info is None: + self.cluster_info = [("127.0.0.1", r) for r in range(dist.get_world_size())] + return self.cluster_info + + TorchDist._get_cluster_info = cluster_info + + +def _run_matrix() -> None: + import tensorrt_llm._torch.custom_ops # noqa: F401 -- registers trtllm::flashinfer_fused_add_rmsnorm + from tensorrt_llm._torch.distributed import AllReduce + from tensorrt_llm._torch.flashinfer_utils import IS_FLASHINFER_AVAILABLE + from tensorrt_llm.functional import AllReduceStrategy + from tensorrt_llm.mapping import Mapping + + assert IS_FLASHINFER_AVAILABLE, "stock Eagle3's add + RMSNorm is FlashInfer's fused_add_rmsnorm" + _single_node_cluster_info() + mapping = Mapping(world_size=WORLD, tp_size=WORLD, rank=RANK, gpus_per_node=WORLD) + _check_cells(mapping) + strategies = ( + AllReduceStrategy.AUTO, + AllReduceStrategy.ONESHOT, + AllReduceStrategy.TWOSHOT, + AllReduceStrategy.NCCL, + ) + worst = 0.0 + mismatches: List[str] = [] + for strategy in strategies: + ar = AllReduce(mapping=mapping, strategy=strategy, dtype=torch.bfloat16) + assert ar.mnnvl_allreduce is None and ar.symm_mem_allreduce is None, strategy + for name, build in _cases(): + ulp, res_ulp = 0.0, 0.0 + for m in TOKENS: + xs, residual, w = build(m) + target_out, target_res = _target(ar, xs[RANK], residual, w) + stock_out, stock_res = _stock(ar, xs[RANK], residual, w) + torch.cuda.synchronize() + for what, a, b in ( + ("normed", target_out, stock_out), + ("residual", target_res, stock_res), + ): + bad = int((a.view(torch.int16) != b.view(torch.int16)).sum().item()) + if bad: + mismatches.append( + f"{strategy.name} {name} m={m}: {bad} {what} elements differ from stock" + ) + ref = _fp64(xs, residual, w) + ref_res = residual.double() + sum(x.double() for x in xs) + u, ur = _ulps(target_out, ref), _ulps(target_res, ref_res) + assert math.isfinite(u) and math.isfinite(ur), (strategy, name, m) + ulp, res_ulp = max(ulp, u), max(res_ulp, ur) + dist.barrier() + worst = max(worst, ulp) + if RANK == 0: + print( + f"{RESULT_TAG} {strategy.name:7s} | {name:34s} | target = stock vs fp64: normed {ulp:.3f}" + f" | residual {res_ulp:.3f}", + flush=True, + ) + if RANK == 0: + scope = f"all cases, m in {TOKENS}" + print( + f"{RESULT_TAG} MAX | {scope:34s} | target = stock vs fp64: normed {worst:.3f}", + flush=True, + ) + assert not mismatches, "\n".join(mismatches) + + +def _run_one_rank() -> int: + global RANK, WORLD + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(rank) + dist.init_process_group( + "cpu:gloo,cuda:nccl", init_method="env://", device_id=torch.device("cuda", rank) + ) + RANK, WORLD = rank, world + assert WORLD in WORLD_SIZES, f"the matrix runs at {WORLD_SIZES} ranks on one node, got {WORLD}" + try: + _run_matrix() + except BaseException: + import traceback # only the rank subprocess's failure path needs it + + print(f"[rank {RANK}] FAILED", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(13) + dist.barrier() + torch.cuda.synchronize() + print(f"[rank {RANK}] passed", flush=True) + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + """One worker per claimed device, each in its own session; on the first nonzero exit or the deadline every + session is killed (as ``_m3_mc_allreduce_norm_op_matrix.py`` does).""" + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the devices this run owns (one rank each)" + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size in WORLD_SIZES, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); the matrix runs at {WORLD_SIZES}" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +if __name__ == "__main__": + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank()) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_mc_allreduce_norm_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_mc_allreduce_norm_op_matrix.py new file mode 100644 index 000000000000..a8616e5d2d5e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_mc_allreduce_norm_op_matrix.py @@ -0,0 +1,788 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU certification matrix for ``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm``, at the job's ranks. + +Its own launcher, as the catalog's collective matrices are: run plainly it spawns +one rank process per device in CUDA_VISIBLE_DEVICES and kills them all on the +first failure or on the deadline. A rank whose peers never push polls forever, +so the deadline is what keeps a broken exchange from wedging the calling run. + + CUDA_VISIBLE_DEVICES=0,1,2,3 python _m3_mc_allreduce_norm_op_matrix.py + +World size: one rank per visible device, 2, 4 or 8. The tp8 target runs these +entries at 8 ranks over two trays, which one 4-device node cannot form; the +world size is a runtime argument of the kernel and no cell's spec names one, so +the cells are certified at the world size of the job (``_rank_job.WORLD_SIZES``). + +Op: the entries call the image's op by default and this target's own copy +(``trtllm::minimax_m3_sm107_tp8_mc_allreduce_norm``) with ``own_op``, which is +what the target's boundaries call when it is loaded. Every cell is driven +through each op the build has. + +Each rank builds the three multicast workspaces (8 / 16 / 48 tokens) the target +builds, over the torch.distributed group (the non-MPI path of +``_make_mnnvl_mcast_buffer``). Every rank can regenerate every rank's inputs from +the shared seed convention, which is what keeps each reference arithmetic rather +than a second collective. Producer pushes (``prepushed`` 2 / 3) are emulated by +storing the canonicalized rows through the multicast mapping at the call parity +the flags select, which is exactly what the producing kernels store. + +Not a pytest module, despite the ``check_*`` bodies: they are one fixed sequence +inside a single multi-rank job whose workspace state (call parities) carries from +one to the next. The collected entry point is +``test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py``. +""" + +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "m3_mc_allreduce_norm requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 900 + +HIDDEN = 6144 +EPS = 1e-6 +CLUSTER = 8 +BUCKETS = (8, 16, 48) +EMPTY_WORD = -2147483648 + +# Bound inside the rank body: importing the entries pulls in tensorrt_llm. +allreduce_entry: Any = None +finalize_entry: Any = None +ar_module: Any = None +validating: Any = None +#: World sizes the matrix runs at (one node): see ``_rank_job.WORLD_SIZES``. +WORLD_SIZES = (2, 4, 8) +#: The ``own_op`` values the cells are driven with: False (the image's op), then True when this target's op is +#: loaded. The last one is the op the target's boundaries call, which the other checks use. +OWN_OPS: Tuple[bool, ...] = (False,) +RANK = 0 +WORLD = 1 +WORKSPACES: Dict[int, Any] = {} + + +# ---------------------------------------------------------------- inputs + + +def _gen(seed: int, rank: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed * 1009 + rank + 1) + + +def _rows(seed: int, rank: int, m: int, scale: float = 1.0) -> torch.Tensor: + """One rank's [m, H] BF16 rows; ``rank`` -1 for tensors every rank holds alike.""" + g = _gen(seed, rank + 1) + return (torch.randn(m, HIDDEN, generator=g, device="cuda") * scale).to(torch.bfloat16) + + +def _residual(seed: int, m: int) -> torch.Tensor: + return _rows(seed + 7, -1, m, 4.0) + + +def _norm_w(seed: int) -> torch.Tensor: + g = _gen(seed + 11, 0) + return (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).to(torch.bfloat16) + + +def _finalize_inputs(seed: int, rank: int, m: int, topk: int, dropped: int): + """One rank's unfinalized MoE outputs: fc2 [m * topk, H], perm [m, topk], fp32 scales, shared [m, H].""" + g = _gen(seed + 23, rank) + rows = m * topk + fc2 = (torch.randn(rows, HIDDEN, generator=g, device="cuda") * 0.5).to(torch.bfloat16) + perm = torch.randperm(rows, generator=g, device="cuda").to(torch.int32) + if dropped: + drop = torch.randperm(rows, generator=g, device="cuda")[:dropped] + perm[drop] = -1 + perm = perm.view(m, topk).contiguous() + scales = torch.rand(m, topk, generator=g, device="cuda", dtype=torch.float32) + shared = _rows(seed + 29, rank, m, 0.5) + return fc2, perm, scales, shared + + +# ---------------------------------------------------------------- workspace / producer emulation + + +def _make_workspace(maxt: int): + """One bucket's exchange, as ``McAllReduceWorkspace.create`` builds it, over the torch.distributed group.""" + from tensorrt_llm._torch.distributed.ops import _make_mnnvl_mcast_buffer + from tensorrt_llm.mapping import Mapping + + mapping = Mapping(world_size=WORLD, tp_size=WORLD, rank=RANK) + words = 4 * maxt * WORLD * (HIDDEN // 2) + group = dist.group.WORLD + handle = _make_mnnvl_mcast_buffer(group, words * 4, mapping, False) + uc = handle.get_uc_buffer(RANK, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + dist.barrier() + ws = ar_module.McAllReduceWorkspace( + uc=uc, + mc=mc, + flags={}, + rank=RANK, + world_size=WORLD, + hidden=HIDDEN, + handle=handle, + comm=group, + max_tokens=maxt, + ) + ws.flags_for(CLUSTER) + return ws + + +def _push(ws, rows: torch.Tensor, exchange_set: int) -> None: + """Store this rank's rows into every rank's exchange, as a producer kernel does before a prepushed call. + + Slot of (token t, cluster CTA c): set * 2 * half + parity(t, c) * half + (t * world + rank) * H / 2 + c * H / 16 + words, parity read from the CTA's flag; -0.0 halves are stored as +0.0 (0x80000000 is the empty word). + """ + m = rows.shape[0] + torch.cuda.synchronize() + parity = (ws.flags_for(CLUSTER).view(ws.max_tokens, CLUSTER)[:m] & 1).long() + half = ws.max_tokens * WORLD * (HIDDEN // 2) + chunk = HIDDEN // 2 // CLUSTER + h16 = rows.contiguous().view(torch.int16) + h16 = torch.where(h16 == -32768, torch.zeros_like(h16), h16) + words = h16.view(torch.int32).view(m, CLUSTER, chunk) + t = torch.arange(m, device="cuda").view(m, 1, 1) + c = torch.arange(CLUSTER, device="cuda").view(1, CLUSTER, 1) + w = torch.arange(chunk, device="cuda").view(1, 1, chunk) + index = ( + exchange_set * 2 * half + + parity.view(m, CLUSTER, 1) * half + + (t * WORLD + RANK) * (HIDDEN // 2) + + c * chunk + + w + ) + ws.mc[index.reshape(-1)] = words.reshape(-1) + torch.cuda.synchronize() + dist.barrier() + + +# ---------------------------------------------------------------- gates + + +def _assert_bitwise(out: torch.Tensor, ref: torch.Tensor, where: str) -> None: + assert out.dtype == ref.dtype and out.shape == ref.shape, ( + where, + out.dtype, + ref.dtype, + out.shape, + ) + a, b = out.contiguous().view(torch.uint8), ref.contiguous().view(torch.uint8) + bad = (a != b).sum().item() + assert bad == 0, f"{where}: {bad} of {a.numel()} bytes differ" + + +#: Share of normed elements one BF16 step off the float64 reference: measured at most 1.1e-4 per call over +#: every call of this matrix on GB300 (see ``_assert_normed``). The gate sits ~18x above that and far below what a +#: wrong statistic produces (most elements move). +NORMED_STEP_FRACTION = 0.002 +NORMED_STEP_SEEN: List[float] = [] + + +def _assert_normed(out: torch.Tensor, ref: torch.Tensor, where: str) -> None: + """At most one BF16 rounding step from the float64 reference, and rarely that. + + The kernel's RMS statistic is a shuffle / warp / cluster tree of FP32 partial sums and its ``rsqrtf`` is + approximate (~2^-22 relative): the scaled value moves by ~1e-6 relative, which changes the BF16 rounding only + for values within that distance of a rounding midpoint -- one step, never two. The fraction bound catches a + statistic that is systematically off (a missing rank, a wrong H), which moves most elements. + """ + assert out.dtype == ref.dtype == torch.bfloat16 and out.shape == ref.shape, where + o, r = out.float(), ref.float() + _, exponent = torch.frexp(torch.maximum(o.abs(), r.abs())) + step = torch.ldexp(torch.ones_like(o), exponent - 8) + diff = (o - r).abs() + assert bool((diff <= step).all()), f"{where}: max diff {diff.max().item()} beyond one BF16 step" + frac = (diff != 0).float().mean().item() + NORMED_STEP_SEEN.append(frac) + assert frac <= NORMED_STEP_FRACTION, f"{where}: {frac:.4f} of normed elements off by one step" + + +def _assert_quant(outs, ref, m: int, where: str) -> None: + """The MXFP8 copy: bit-exact against the recipe applied to the op's own normed rows (a pure function of + them), and equal to the reference's wherever the reference's normed 32-column block is the op's.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.comm.m3_mc_allreduce_norm import ( # noqa: E501 + mxfp8_rows, + sf_written, + ) + + q, sf = outs[2], outs[3] + own_q, own_sf = mxfp8_rows(outs[0]) + written = sf_written(m, HIDDEN // 32, sf.device) + _assert_bitwise(q, own_q, f"{where} e4m3 rows") + _assert_bitwise(sf[written], own_sf[written], f"{where} UE8M0 scales") + same_block = (outs[0].view(m, -1, 32) == ref[0].view(m, -1, 32)).all(-1) + qb = q.view(torch.uint8).view(m, -1, 32) + rqb = ref[2].view(torch.uint8).view(m, -1, 32) + assert bool((qb == rqb).all(-1)[same_block].all()), ( + f"{where}: e4m3 differs on a block whose normed rows agree" + ) + + +def _check_outputs(outs, ref, m: int, quant: bool, where: str) -> None: + assert len(outs) == (4 if quant else 2), (where, len(outs)) + _assert_bitwise(outs[1], ref[1], f"{where} updated residual") + _assert_normed(outs[0], ref[0], f"{where} normed") + if quant: + _assert_quant(outs, ref, m, where) + + +# ---------------------------------------------------------------- calls + + +def _call_plain(ws, x, x2, residual, norm_w, quant, prepushed, opts, own_op=None): + own_op = OWN_OPS[-1] if own_op is None else own_op + return allreduce_entry.m3_mc_allreduce_norm( + x, + x2, + residual, + norm_w, + EPS, + ws.uc, + ws.mc.data_ptr(), + ws.flags_for(CLUSTER), + RANK, + WORLD, + CLUSTER, + False, + ws.max_tokens, + quant, + prepushed, + opts, + own_op=own_op, + ) + + +def _plain_case(spec: Dict[str, Any], seed: int, where: str, own_op: bool) -> None: + m, maxt, quant, prepushed = spec["m"], spec["maxt"], spec["quant"], spec["prepushed"] + ws = WORKSPACES[maxt] + xs = [_rows(seed, r, m) for r in range(WORLD)] + x2s = [_rows(seed + 3, r, m) for r in range(WORLD)] if spec["x2"] else None + residual, norm_w = _residual(seed, m), _norm_w(seed) + x, x2 = xs[RANK], (None if x2s is None else x2s[RANK]) + if prepushed >= 2: + _push(ws, x, 0) + if prepushed == 3: + _push(ws, x2, 1) + with validating(allreduce_entry.m3_mc_allreduce_norm): + outs = _call_plain(ws, x, x2, residual, norm_w, quant, prepushed, spec["opts"], own_op) + torch.cuda.synchronize() + ref = allreduce_entry.m3_mc_allreduce_norm.reference( + xs, x2s, residual, norm_w, EPS, quant, spec.get("drafter", False) + ) + _check_outputs(outs, ref, m, quant, where) + dist.barrier() + + +# ---------------------------------------------------------------- checks + + +def _op_name(own_op: bool) -> str: + return "own op" if own_op else "image op" + + +def check_plain_cells() -> None: + """Every cell of ``m3_mc_allreduce_norm.CELLS`` through ``__call__`` under ``validating``, gated on its + ``reference``, with each op of ``OWN_OPS``.""" + for own_op in OWN_OPS: + for i, cell in enumerate(allreduce_entry.m3_mc_allreduce_norm.CELLS): + _plain_case(cell.spec, 1000 + 17 * i, f"{cell.why} ({_op_name(own_op)})", own_op) + + +def check_finalize_cells() -> None: + """Every cell of ``m3_mc_finalize_allreduce_norm.CELLS``, gated on its ``reference``, with each op of + ``OWN_OPS``.""" + for own_op in OWN_OPS: + _finalize_cells(own_op) + + +def _finalize_cells(own_op: bool) -> None: + op = finalize_entry.m3_mc_finalize_allreduce_norm + for i, cell in enumerate(op.CELLS): + spec = cell.spec + m, maxt, quant, topk = spec["m"], spec["maxt"], spec["quant"], spec["topk"] + seed = 2000 + 17 * i + ws = WORKSPACES[maxt] + per_rank = [_finalize_inputs(seed, r, m, topk, spec["dropped"]) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + fc2, perm, scales, shared = per_rank[RANK] + with validating(op): + outs = op( + fc2, + perm, + scales, + shared, + residual, + norm_w, + EPS, + ws.uc, + ws.mc.data_ptr(), + ws.flags_for(CLUSTER), + RANK, + WORLD, + CLUSTER, + False, + maxt, + quant, + spec["opts"], + own_op=own_op, + ) + torch.cuda.synchronize() + ref = op.reference( + [p[0] for p in per_rank], + [p[1] for p in per_rank], + [p[2] for p in per_rank], + [p[3] for p in per_rank], + residual, + norm_w, + EPS, + quant, + ) + _check_outputs(outs, ref, m, quant, f"{cell.why} ({_op_name(own_op)})") + dist.barrier() + + +def _capture_reference( + updated: torch.Tensor, weight: torch.Tensor, eps: float, out: torch.Tensor +) -> None: + """The capture the boundary replaces: m3_spec_tail.capture_norm of the updated residual (no second addend).""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_spec_tail, + ) + + m3_spec_tail.capture_norm(updated, None, weight, eps, out) + + +def check_capture_norm_in_boundary() -> None: + """CAPTURE_NORM_IN_BOUNDARY: with cap_out the boundary also writes the capture, BITWISE equal to capture_norm of + its own updated residual, into a strided [M, H] block of a [rows, 3H] capture buffer (other bytes untouched); its + normed / updated / MXFP8 outputs are bitwise the ones of the same call without cap_out. The row kernel (the plain + entry: x, x + x2, prepushed) and the cluster kernel (MoE finalize), 1 / 4 / 8 rows, residual scales from 1e-3 to + 1e3 (the statistic's range), the fc_norm eps 1e-6 and 1e-5; on the op the target's boundaries call.""" + cap_eps_values = (1e-6, 1e-5) + own_op = OWN_OPS[-1] + for case, (m, scale, kind) in enumerate( + ( + (1, 1.0, "plain"), + (4, 1e3, "plain_x2"), + (8, 1e-3, "prepushed"), + (4, 1.0, "prepushed"), + (1, 4.0, "finalize"), + (4, 1e3, "finalize"), + (8, 1e-3, "finalize"), + ) + ): + seed = 5000 + 31 * case + cap_eps = cap_eps_values[case % 2] + ws = WORKSPACES[8] + residual = (_residual(seed, m).float() * scale).to(torch.bfloat16) + norm_w = _norm_w(seed) + cap_w = _norm_w(seed + 101) + quant = case % 3 == 1 + outs_by = [] + for with_cap in (False, True): + buf = torch.full((m + 3, 3 * HIDDEN), 7.0, dtype=torch.bfloat16, device="cuda") + cap = buf[:m, HIDDEN : 2 * HIDDEN] if with_cap else None + if kind == "finalize": + fc2, perm, scales, shared = _finalize_inputs(seed, RANK, m, 4, 1) + fc2 = (fc2.float() * scale).to(torch.bfloat16) + outs = finalize_entry.m3_mc_finalize_allreduce_norm( + fc2, perm, scales, shared, residual, norm_w, EPS, ws.uc, ws.mc.data_ptr(), + ws.flags_for(CLUSTER), RANK, WORLD, CLUSTER, False, 8, quant, 7, + cap, cap_w if with_cap else None, cap_eps, own_op=own_op, + ) # fmt: skip + else: + x = (_rows(seed, RANK, m).float() * scale).to(torch.bfloat16) + x2 = ( + (_rows(seed + 3, RANK, m).float() * scale).to(torch.bfloat16) + if kind == "plain_x2" + else None + ) + prepushed = 2 if kind == "prepushed" else 0 + if prepushed: + _push(ws, x, 0) + outs = allreduce_entry.m3_mc_allreduce_norm( + x, x2, residual, norm_w, EPS, ws.uc, ws.mc.data_ptr(), ws.flags_for(CLUSTER), RANK, WORLD, + CLUSTER, False, 8, quant, prepushed, (7, 7, 15, 15, 15, 15, 7, 7)[m - 1], + cap, cap_w if with_cap else None, cap_eps, own_op=own_op, + ) # fmt: skip + torch.cuda.synchronize() + outs_by.append((outs, buf)) + dist.barrier() + (plain, _), (capped, buf) = outs_by + where = f"capture in boundary, {kind}, {m} rows, scale {scale}" + assert len(plain) == len(capped), where + for i, (a, b) in enumerate(zip(plain, capped)): + if i == 3: + # The scale buffer is torch.empty: compare the bytes the kernel writes (the valid rows'). + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.comm.m3_mc_allreduce_norm import ( # noqa: E501 + sf_written, + ) + + written = sf_written(m, HIDDEN // 32, a.device) + a, b = a[written], b[written] + _assert_bitwise(b, a, f"{where}: output {i} with vs without cap_out") + ref = torch.empty((m, HIDDEN), dtype=torch.bfloat16, device="cuda") + _capture_reference(capped[1], cap_w, cap_eps, ref) + torch.cuda.synchronize() + _assert_bitwise(buf[:m, HIDDEN : 2 * HIDDEN], ref, f"{where}: capture vs capture_norm") + untouched = buf.clone() + untouched[:m, HIDDEN : 2 * HIDDEN] = 7.0 + assert bool((untouched == 7.0).all()), f"{where}: bytes outside the capture block changed" + dist.barrier() + + +def check_the_target_wrappers_select_bucket_and_options() -> None: + """The target's own wrappers (bucket by token count, option table) on top of the entries, every token count of + the 8-token bucket plus one of each larger bucket, and the finalize wrapper; the MXFP8 publication is the + prequant slot's business and is left off.""" + buckets = ar_module.McBuckets([WORKSPACES[b] for b in BUCKETS]) + for m in (*range(1, 9), 11, 40): + seed = 3000 + m + xs = [_rows(seed, r, m) for r in range(WORLD)] + x2s = [_rows(seed + 3, r, m) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + normed, updated = ar_module.m3_mc_allreduce_norm( + xs[RANK], residual, norm_w, EPS, buckets, x2=x2s[RANK] + ) + torch.cuda.synchronize() + ref = allreduce_entry.m3_mc_allreduce_norm.reference(xs, x2s, residual, norm_w, EPS, False) + _check_outputs([normed, updated], ref, m, False, f"wrapper m={m}") + dist.barrier() + for m in (9, 33): + seed = 3100 + m + per_rank = [_finalize_inputs(seed, r, m, 4, 2) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + fc2, perm, scales, shared = per_rank[RANK] + normed, updated = ar_module.m3_mc_finalize_allreduce_norm( + fc2, perm, scales, shared, residual, norm_w, EPS, buckets + ) + torch.cuda.synchronize() + ref = finalize_entry.m3_mc_finalize_allreduce_norm.reference( + *[[p[j] for p in per_rank] for j in range(4)], residual, norm_w, EPS, False + ) + _check_outputs([normed, updated], ref, m, False, f"finalize wrapper m={m}") + dist.barrier() + + +def check_cuda_graph_replays() -> None: + """Two boundaries captured in one graph (8- and 16-token buckets), replayed with fresh inputs. + + The decode graph holds every boundary call; each replay rotates the same per-CTA parities the eager calls do, + so a replay that reused a stale half or skipped a re-arm reads the previous replay's rows and fails the + bitwise residual gate (every replay gets new inputs). + """ + shapes = ((3, 8, 15), (12, 16, 15)) + norm_w = _norm_w(4000) + bufs = [] + for m, maxt, opts in shapes: + bufs.append( + dict( + m=m, + maxt=maxt, + opts=opts, + x=torch.empty(m, HIDDEN, dtype=torch.bfloat16, device="cuda"), + x2=torch.empty(m, HIDDEN, dtype=torch.bfloat16, device="cuda"), + residual=torch.empty(m, HIDDEN, dtype=torch.bfloat16, device="cuda"), + ) + ) + + def body(): + return [ + _call_plain( + WORKSPACES[b["maxt"]], b["x"], b["x2"], b["residual"], norm_w, False, 0, b["opts"] + ) + for b in bufs + ] + + def fill(seed: int): + refs = [] + for j, b in enumerate(bufs): + s = seed + 31 * j + xs = [_rows(s, r, b["m"]) for r in range(WORLD)] + x2s = [_rows(s + 3, r, b["m"]) for r in range(WORLD)] + b["x"].copy_(xs[RANK]) + b["x2"].copy_(x2s[RANK]) + b["residual"].copy_(_residual(s, b["m"])) + refs.append( + allreduce_entry.m3_mc_allreduce_norm.reference( + xs, x2s, b["residual"].clone(), norm_w, EPS, False + ) + ) + torch.cuda.synchronize() + dist.barrier() + return refs + + refs = fill(4100) + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(side): + warm = body() + torch.cuda.current_stream().wait_stream(side) + torch.cuda.synchronize() + for j, b in enumerate(bufs): + _check_outputs(warm[j], refs[j], b["m"], False, f"graph warm-up m={b['m']}") + dist.barrier() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = body() + for rep in range(4): + refs = fill(4200 + 100 * rep) + graph.replay() + torch.cuda.synchronize() + for j, b in enumerate(bufs): + _check_outputs(captured[j], refs[j], b["m"], False, f"replay {rep} m={b['m']}") + dist.barrier() + del graph + dist.barrier() + + +def check_the_gate_sees_a_missing_rank() -> None: + """The residual gate is not blind to the reduction: dropping any one rank's partial from the reference + moves most elements of the updated residual (each rank's rows are distinct by construction).""" + m, seed = 4, 5000 + xs = [_rows(seed, r, m) for r in range(WORLD)] + residual, norm_w = _residual(seed, m), _norm_w(seed) + full = allreduce_entry.m3_mc_allreduce_norm.reference(xs, None, residual, norm_w, EPS, False) + for r in range(WORLD): + partial = [x if i != r else torch.zeros_like(x) for i, x in enumerate(xs)] + wrong = allreduce_entry.m3_mc_allreduce_norm.reference( + partial, None, residual, norm_w, EPS, False + ) + frac = (wrong[1] != full[1]).float().mean().item() + assert frac > 0.9, f"dropping rank {r} changes only {frac:.3f} of the residual" + dist.barrier() + + +def _expect_rejected(fn, match: str, where: str) -> None: + try: + fn() + except AssertionError as exc: + assert match in str(exc), f"{where}: rejected for the wrong reason: {exc}" + else: + raise AssertionError(f"{where}: the guard admitted it") + + +def check_guards() -> None: + """``is_valid`` refuses what the op accepts and answers wrongly. Nothing here reaches the kernel: the guard + raises first, on every rank alike, so no rank is left polling.""" + m, seed = 4, 6000 + ws8, ws16 = WORKSPACES[8], WORKSPACES[16] + x, residual, norm_w = _rows(seed, RANK, m), _residual(seed, m), _norm_w(seed) + op = allreduce_entry.m3_mc_allreduce_norm + + def call(ws_uc, flags, norm): + with validating(op): + op( + x, + None, + residual, + norm, + EPS, + ws_uc, + ws8.mc.data_ptr(), + flags, + RANK, + WORLD, + CLUSTER, + False, + 8, + False, + 0, + 15, + ) + + strided = torch.empty(2 * HIDDEN, dtype=torch.bfloat16, device="cuda")[::2] + strided.copy_(norm_w) + _expect_rejected( + lambda: call(ws8.uc, ws8.flags_for(CLUSTER), strided), "contiguous", "strided norm_w" + ) + _expect_rejected( + lambda: call(ws8.uc[: ws8.uc.numel() // 2], ws8.flags_for(CLUSTER), norm_w), + "words", + "short workspace", + ) + _expect_rejected( + lambda: call(ws8.uc, ws16.flags_for(CLUSTER), norm_w), "counters", "another bucket's flags" + ) + + fin = finalize_entry.m3_mc_finalize_allreduce_norm + fc2, perm, scales, shared = _finalize_inputs(seed, RANK, 9, 4, 0) + bad = perm.clone() + bad[0, 0] = fc2.shape[0] + + def call_fin(): + with validating(fin): + fin( + fc2, + bad, + scales, + shared, + _residual(seed, 9), + norm_w, + EPS, + ws16.uc, + ws16.mc.data_ptr(), + ws16.flags_for(CLUSTER), + RANK, + WORLD, + CLUSTER, + False, + 16, + False, + 15, + ) + + _expect_rejected(call_fin, "perm spans", "perm past the FC2 rows") + dist.barrier() + + +CHECKS = ( + check_plain_cells, + check_finalize_cells, + check_capture_norm_in_boundary, + check_the_target_wrappers_select_bucket_and_options, + check_cuda_graph_replays, + check_the_gate_sees_a_missing_rank, + check_guards, +) + + +def _run_one_rank() -> int: + """Body of one spawned rank: run every check; a failing rank exits nonzero and the launcher kills its + siblings.""" + global RANK, WORLD, OWN_OPS, allreduce_entry, finalize_entry, ar_module, validating + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(rank) + dist.init_process_group( + "cpu:gloo,cuda:nccl", init_method="env://", device_id=torch.device("cuda", rank) + ) + RANK, WORLD = rank, world + assert WORLD in WORLD_SIZES, f"the matrix runs at {WORLD_SIZES} ranks on one node, got {WORLD}" + + import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_mc_allreduce as _ar # noqa: E501 + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.comm import ( # noqa: E501 + m3_mc_allreduce_norm as _plain, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.comm import ( # noqa: E501 + m3_mc_finalize_allreduce_norm as _finalize, + ) + + allreduce_entry, finalize_entry, ar_module = _plain, _finalize, _ar + # The same library is loaded on every rank, so the ranks agree on the ops they drive. + OWN_OPS = (False, True) if _ar.own_op() else (False,) + print(f"[rank {RANK}] ops: {', '.join(_op_name(o) for o in OWN_OPS)}", flush=True) + # Spawned by file path, not collected by pytest: reach the sibling helper the plain way (confined to the + # rank processes). /comm -> tests/unittest/_torch/modeling_v2. + sys.path.insert(0, str(Path(__file__).resolve().parents[4])) + from _validating import validating as _validating_cm + + validating = _validating_cm + for maxt in BUCKETS: + WORKSPACES[maxt] = _make_workspace(maxt) + + for check in CHECKS: + try: + check() + except BaseException: + import traceback # only the rank subprocess's failure path needs it + + print(f"[rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + # A rank that leaves early wedges its peers' polls; the launcher kills them on this exit. + os._exit(13) + print(f"[rank {RANK}] {check.__name__} passed", flush=True) + dist.barrier() + torch.cuda.synchronize() + print( + f"[rank {RANK}] {len(CHECKS)} checks passed; normed one-step fraction max " + f"{max(NORMED_STEP_SEEN):.5f} over {len(NORMED_STEP_SEEN)} calls", + flush=True, + ) + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + """One worker process per claimed device, each in its own session; on the first nonzero exit or on the + deadline every session is killed.""" + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the devices this run owns (one rank each)" + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size in WORLD_SIZES, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); the matrix runs at {WORLD_SIZES}" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + # Drop a launching srun step's PMIx / MPI namespace: children registered in it get the whole step SIGKILLed + # when the first one exits hard. These ranks rendezvous through torch.distributed's env://. + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + print(f"[launcher] spawning {world_size} ranks, rendezvous on :{port}", flush=True) + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +if __name__ == "__main__": + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank()) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_moe_push_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_moe_push_op_matrix.py new file mode 100644 index 000000000000..5515b24d9228 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_m3_moe_push_op_matrix.py @@ -0,0 +1,403 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU certification of the MoE engine's all-reduce push (``m3_moe_engine`` with ``arp``), 8 ranks, two trays (SM107). + +The decode forward's MoE boundary (MOE_PUSH rows): the engine pushes its bf16 sum (routed + shared) into set A of +the boundary's multicast exchange while it writes it, and the boundary all-reduce runs with ``prepushed`` 2, reading +the pushed words instead of loading and pushing the rows itself. Certified against the unpushed boundary the target +runs otherwise: the same engine call without ``arp``, then ``m3_mc_allreduce_norm`` with ``prepushed`` 0. Every +output must be bit for bit the same: the engine's own output tensor, the updated residual, the normed rows and, with +``quant``, the MXFP8 hand-off it publishes. Both read the same bf16 words, so the reduction and the norm see equal +inputs; any difference is a push that stored the wrong word, slot or parity. + +Each rank holds its own expert shard (operand seed per rank) and the same hidden rows, residual and norm weight, as +the TP2 decode step does. Eager calls alternate pushed and unpushed in both orders so both call parities of the +exchange are exercised; a CUDA graph of the pushed engine + boundary is replayed on fresh rows against eager +unpushed references. + +Its own launcher, as ``_m3_mc_allreduce_norm_op_matrix.py`` is: run plainly it spawns one rank process per device +in CUDA_VISIBLE_DEVICES and kills them all on the first failure or on the deadline (a rank whose peer never pushes +polls forever). The collected entry point is ``test_modeling_v2_m3_moe_push_op_matrix.py``. + + CUDA_VISIBLE_DEVICES=0,1 python _m3_moe_push_op_matrix.py +""" + +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Optional + +import torch +import torch.distributed as dist + +assert torch.cuda.is_available(), "m3_moe_engine requires CUDA devices" + +_WORKER_FLAG = "--rank-worker" +DEADLINE_S = 1200 + +HIDDEN = 6144 +EPS = 1e-6 +CLUSTER = 8 +MAX_TOKENS = 8 +EMPTY_WORD = -2147483648 +RESERVED_SMS = 24 +OPERAND_SEED = 20261013 + +# Bound inside the rank body: importing the entries pulls in tensorrt_llm. +ar = None +sf_written: Any = None +engine_op: Any = None +ar_push_desc: Any = None +moe_state: Any = None +operands: Any = None +prequant: Any = None +RANK = 0 +WORLD = 1 +WS: Any = None + + +def _gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def _residual(seed: int, m: int) -> torch.Tensor: + return (torch.randn(m, HIDDEN, generator=_gen(seed + 7), device="cuda") * 4.0).to( + torch.bfloat16 + ) + + +def _norm_w(seed: int) -> torch.Tensor: + g = _gen(seed + 11) + return (1.0 + 0.25 * torch.randn(HIDDEN, generator=g, device="cuda")).to(torch.bfloat16) + + +def _make_workspace(): + """The 8-token bucket's exchange, as ``McAllReduceWorkspace.create`` builds it, over the torch.distributed + group.""" + from tensorrt_llm._torch.distributed.ops import _make_mnnvl_mcast_buffer + from tensorrt_llm.mapping import Mapping + + mapping = Mapping(world_size=WORLD, tp_size=WORLD, rank=RANK, gpus_per_node=4) + words = 4 * MAX_TOKENS * WORLD * (HIDDEN // 2) + group = dist.group.WORLD + handle = _make_mnnvl_mcast_buffer(group, words * 4, mapping, False) + uc = handle.get_uc_buffer(RANK, (words,), torch.int32, 0) + mc = handle.get_mc_buffer((words,), torch.int32, 0) + uc.fill_(EMPTY_WORD) + torch.cuda.synchronize() + dist.barrier() + ws = ar.McAllReduceWorkspace( + uc=uc, + mc=mc, + flags={}, + rank=RANK, + world_size=WORLD, + hidden=HIDDEN, + handle=handle, + comm=group, + max_tokens=MAX_TOKENS, + ) + ws.flags_for(CLUSTER) + return ws + + +def _assert_bitwise(out: torch.Tensor, ref: torch.Tensor, where: str) -> None: + assert out.dtype == ref.dtype and out.shape == ref.shape, ( + where, + out.dtype, + ref.dtype, + out.shape, + ) + a, b = out.contiguous().view(torch.uint8), ref.contiguous().view(torch.uint8) + bad = (a != b).sum().item() + assert bad == 0, f"{where}: {bad} of {a.numel()} bytes differ" + + +_STATES = {} +_LAYERS = {} + + +def _layer(m: int, case: str): + """This rank's engine layer for ``m`` tokens: its own expert shard; ``concentrated`` routes every token to the + same four experts.""" + if m not in _STATES: + sms = torch.cuda.get_device_properties(0).multi_processor_count + _STATES[m] = moe_state( + torch.device("cuda"), num_tokens=m, num_ctas=sms - RESERVED_SMS, shared=True + ) + key = (m, case) + if key not in _LAYERS: + ops = operands.experts(OPERAND_SEED + RANK) + bias = ops.bias.clone() + if case == "concentrated": + bias[list(operands.CONCENTRATED)] += 1.0 + _LAYERS[key] = _STATES[m].layer( + *ops.tensors, + bias, + router_weight=ops.router_weight, + fc31_input_scale=ops.fc31_input_scale, + shared_weights=ops.shared, + ) + return _LAYERS[key] + + +def _desc(): + flags = WS.flags_for(CLUSTER) + return ar_push_desc( + WS.mc.data_ptr(), flags.data_ptr(), RANK, WORLD, WS.max_tokens, flags.device + ) + + +def _boundary(x, residual, norm_w, quant, prepushed): + prequant.SLOT.clear() + outs = ar.m3_mc_allreduce_norm( + x, residual, norm_w, EPS, WS, use_pdl=True, quant=quant, prepushed=prepushed + ) + assert outs is not None, "the boundary all-reduce declined its operands" + pub = prequant.SLOT.pending + prequant.SLOT.clear() + return outs, (None if pub is None else (pub[3].clone(), pub[4].clone())) + + +def _unpushed(hidden, layer, residual, norm_w, quant): + out = engine_op(hidden, layer) + (normed, res), pub = _boundary(out, residual.clone(), norm_w, quant, 0) + return out.clone(), normed.clone(), res.clone(), pub + + +def _pushed(hidden, layer, residual, norm_w, quant): + out = engine_op(hidden, layer, arp=_desc()) + (normed, res), pub = _boundary(out, residual.clone(), norm_w, quant, 2) + return out.clone(), normed.clone(), res.clone(), pub + + +def _compare(got, ref, quant: bool, where: str) -> None: + _assert_bitwise(got[0], ref[0], f"{where} engine output") + _assert_bitwise(got[2], ref[2], f"{where} updated residual") + _assert_bitwise(got[1], ref[1], f"{where} normed") + if quant: + assert got[3] is not None and ref[3] is not None, f"{where}: no MXFP8 hand-off published" + _assert_bitwise(got[3][0], ref[3][0], f"{where} e4m3 rows") + # Only the scale atoms of the m rows are written; the swizzled buffer's padding is not. + written = sf_written(got[0].shape[0], HIDDEN // 32, got[3][1].device) + _assert_bitwise(got[3][1][written], ref[3][1][written], f"{where} UE8M0 scales") + + +# ---------------------------------------------------------------- checks + + +def check_pushed_boundary_is_the_unpushed_one() -> None: + """Every token count, both routing cases, with and without the MXFP8 hand-off, both call orders.""" + for m in range(1, MAX_TOKENS + 1): + for case in ("random", "concentrated"): + layer = _layer(m, case) + for quant in (False, True): + for seed in (1, 2): + s = ( + 1000 * m + + 10 * seed + + (5 if quant else 0) + + (500 if case == "concentrated" else 0) + ) + hidden = operands.hidden_states(m, s) + residual, norm_w = _residual(s, m), _norm_w(s) + where = f"m={m} {case} quant={quant} seed={seed}" + if seed == 1: + ref = _unpushed(hidden, layer, residual, norm_w, quant) + got = _pushed(hidden, layer, residual, norm_w, quant) + else: + got = _pushed(hidden, layer, residual, norm_w, quant) + ref = _unpushed(hidden, layer, residual, norm_w, quant) + torch.cuda.synchronize() + _compare(got, ref, quant, where) + + +def check_graph_replays() -> None: + """A captured pushed engine + prepushed boundary, replayed on fresh rows, equals the eager unpushed path.""" + for m in (1, 4, 8): + layer = _layer(m, "random") + hidden = operands.hidden_states(m, 90000 + m).clone() + residual = _residual(90000 + m, m).clone() + norm_w = _norm_w(90000 + m) + desc = _desc() + for _ in range( + 2 + ): # warm both parities eagerly; every push is consumed by its prepushed boundary + _pushed(hidden, layer, residual, norm_w, False) + _unpushed(hidden, layer, residual, norm_w, False) + torch.cuda.synchronize() + dist.barrier() + res_in = residual.clone() + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + g_out = engine_op(hidden, layer, arp=desc) + g_normed, g_res = ar.m3_mc_allreduce_norm( + g_out, res_in, norm_w, EPS, WS, use_pdl=True, prepushed=2 + ) + torch.cuda.synchronize() + dist.barrier() + for rep in range(6): + s = 91000 + 100 * m + rep + fresh_h, fresh_r = operands.hidden_states(m, s), _residual(s, m) + hidden.copy_(fresh_h) + res_in.copy_(fresh_r) + torch.cuda.synchronize() + dist.barrier() + graph.replay() + torch.cuda.synchronize() + got = (g_out.clone(), g_normed.clone(), g_res.clone(), None) + ref = _unpushed(fresh_h, layer, fresh_r, norm_w, False) + torch.cuda.synchronize() + _compare(got, ref, False, f"graph m={m} replay {rep}") + dist.barrier() + + +CHECKS = [check_pushed_boundary_is_the_unpushed_one, check_graph_replays] + + +def _run_one_rank() -> int: + """Body of one spawned rank: run every check; a failing rank exits nonzero and the launcher kills its + siblings.""" + global RANK, WORLD, ar, sf_written, engine_op, ar_push_desc, moe_state, operands, prequant, WS + os.environ.setdefault("TLLM_DISABLE_MPI", "1") + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + local = int(os.environ.get("LOCAL_RANK", rank)) + torch.cuda.set_device(local) + dist.init_process_group( + "cpu:gloo,cuda:nccl", init_method="env://", device_id=torch.device("cuda", local) + ) + RANK, WORLD = rank, world + assert WORLD == 8, f"the TP8 push is certified at 8 ranks, got {WORLD}" + + import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_mc_allreduce as _ar # noqa: E501 + import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_prequant as _pq # noqa: E501 + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.comm.m3_mc_allreduce_norm import ( # noqa: E501 + sf_written as _sf_written, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + M3MoeState, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + ar_push_desc as _desc_fn, + ) + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + m3_moe_engine as _op, + ) + + ar, prequant, moe_state, ar_push_desc, engine_op = _ar, _pq, M3MoeState, _desc_fn, _op + sf_written = _sf_written + # Spawned by file path, not collected by pytest: the sibling moe/ operands the plain way. + sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "moe")) + import _m3_moe_operands as _operands + + operands = _operands + WS = _make_workspace() + + for check in CHECKS: + try: + check() + except BaseException: + import traceback # only the rank subprocess's failure path needs it + + print(f"[rank {RANK}] FAILED {check.__name__}", flush=True) + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + # A rank that leaves early wedges its peer's polls; the launcher kills them on this exit. + os._exit(13) + print(f"[rank {RANK}] {check.__name__} passed", flush=True) + dist.barrier() + torch.cuda.synchronize() + print(f"[rank {RANK}] {len(CHECKS)} checks passed", flush=True) + sys.stdout.flush() + os._exit(0) + + +def _spawn_ranks() -> None: + """One worker process per claimed device, each in its own session; on the first nonzero exit or on the + deadline every session is killed.""" + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + assert visible, "set CUDA_VISIBLE_DEVICES to the 2 devices this run owns" + world_size = len([d for d in visible.split(",") if d.strip()]) + assert world_size == 2, ( + f"CUDA_VISIBLE_DEVICES names {world_size} device(s); the matrix runs at 2" + ) + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + env = dict( + os.environ, + MASTER_ADDR="127.0.0.1", + MASTER_PORT=str(port), + WORLD_SIZE=str(world_size), + TLLM_DISABLE_MPI="1", + ) + # Drop a launching srun step's PMIx / MPI namespace: children registered in it get the whole step SIGKILLed + # when the first one exits hard. These ranks rendezvous through torch.distributed's env://. + env = {k: v for k, v in env.items() if not k.startswith(("PMIX_", "PMI_", "OMPI_", "SLURM_"))} + print(f"[launcher] spawning {world_size} ranks, rendezvous on :{port}", flush=True) + children = [ + subprocess.Popen( + [sys.executable, str(Path(__file__).resolve()), _WORKER_FLAG], + env=dict(env, RANK=str(r), LOCAL_RANK=str(r)), + start_new_session=True, + ) + for r in range(world_size) + ] + + def _kill_all() -> None: + for p in children: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + for p in children: + p.wait() + + deadline = time.monotonic() + DEADLINE_S + live = set(range(world_size)) + failure: Optional[str] = None + while live and failure is None: + if time.monotonic() > deadline: + failure = f"did not finish in {DEADLINE_S}s (wedged)" + break + for r in sorted(live): + code = children[r].poll() + if code is None: + continue + live.discard(r) + if code != 0: + failure = f"rank {r} exited {code}" + break + time.sleep(0.5) + if failure is not None: + _kill_all() + raise AssertionError(f"the {world_size}-rank run failed: {failure}") + + +_SRUN_FLAG = "--srun-rank" + + +def _srun_rank() -> int: + """One rank per Slurm task over two trays (srun -N2 --ntasks-per-node 4): RANK / WORLD_SIZE / LOCAL_RANK from + Slurm; the rendezvous is MASTER_ADDR / MASTER_PORT exported by the job (the first node).""" + os.environ["RANK"] = os.environ["SLURM_PROCID"] + os.environ["WORLD_SIZE"] = os.environ["SLURM_NTASKS"] + os.environ["LOCAL_RANK"] = os.environ["SLURM_LOCALID"] + return _run_one_rank() + + +if __name__ == "__main__": + if _SRUN_FLAG in sys.argv: + sys.exit(_srun_rank()) + if _WORKER_FLAG in sys.argv: + sys.exit(_run_one_rank()) + _spawn_ranks() + print("OK") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_rank_job.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_rank_job.py new file mode 100644 index 000000000000..dedb967cd195 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/_rank_job.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Run this target's collective rank job from pytest, and report what it did. + +The same contract as ``tests/unittest/_torch/modeling_v2/comm/_rank_job.py``: the +launcher beside this file owns the per-device rank processes and the deadline +that kills a wedged collective; this only selects the devices, starts it in a +fresh interpreter (by file path: this tree is not a package) and turns its exit +code into an assertion. + +World size. The tp8 target runs 8 ranks over two GB300 trays of 4 GPUs (one +NVLink domain); a single-node unit-test job cannot form that group. The +collective entries take the world size as a runtime argument and their cells' +specs do not name one (token counts, buckets, push modes, epilogue options), so +the jobs here certify every cell at each world size in ``WORLD_SIZES`` the +node's visible devices can form: 2 and 4 on a GB300 tray, 8 only on a node with +8 devices. A world size the node cannot form is skipped with that reason, never +emulated; the 8-rank, two-tray group itself is exercised only by the target's +multi-node runs. +""" + +from __future__ import annotations + +import importlib.util +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +TARGET_WORLD_SIZE = 8 +"""tp8's world size: 8 ranks over two GB300 trays, which one node's job cannot form.""" + +WORLD_SIZES = (2, 4, TARGET_WORLD_SIZE) +"""The world sizes the jobs certify the (world-independent) cells at, where the node has that many devices.""" + +_LAUNCHER_GRACE_S = 300 +"""Headroom over the launcher's own deadline, so its message wins the race.""" + + +def visible_devices() -> list: + visible = os.environ.get("CUDA_VISIBLE_DEVICES") + if visible: + return [d for d in visible.split(",") if d.strip()] + import torch + + return [str(i) for i in range(torch.cuda.device_count())] + + +def require_world(world_size: int) -> str: + """The first ``world_size`` visible devices; skips (with the reason) when the node has fewer.""" + devices = visible_devices() + if len(devices) < world_size: + where = ( + "the tp8 target's own 8-rank group spans two trays and is exercised only by its multi-node runs" + if world_size == TARGET_WORLD_SIZE + else "run this job on a node with more devices" + ) + pytest.skip( + f"{world_size} ranks need {world_size} devices; {len(devices)} visible on this node ({where})" + ) + return ",".join(devices[:world_size]) + + +def _load_launcher_constants(launcher: Path): + """Read the launcher's module-level budget without importing it by name (it imports only torch at + module scope).""" + spec = importlib.util.spec_from_file_location(launcher.stem, launcher) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def run(entry: str, world_size: int) -> str: + """Run ``__op_matrix``'s launcher over ``world_size`` devices (skipped when the node has fewer); returns + its stdout.""" + assert world_size in WORLD_SIZES, world_size + launcher = Path(__file__).resolve().with_name(f"_{entry}_op_matrix.py") + env = dict(os.environ, CUDA_VISIBLE_DEVICES=require_world(world_size)) + + # The ranks import tensorrt_llm absolutely. From a source checkout that is not installed, put the repo root + # (tests/unittest/_torch/modeling_v2/models/minimax_m3//comm -> repo root) on their path; from a tests + # tree run against an installed package, they import the installed one. + repo_root = Path(__file__).resolve().parents[8] + if (repo_root / "tensorrt_llm").is_dir(): + env["PYTHONPATH"] = os.pathsep.join( + p for p in (str(repo_root), env.get("PYTHONPATH", "")) if p + ) + else: + assert importlib.util.find_spec("tensorrt_llm") is not None, ( + f"no tensorrt_llm/ at the repo root {repo_root} and no installed tensorrt_llm" + ) + + constants = _load_launcher_constants(launcher) + timeout = constants.DEADLINE_S + _LAUNCHER_GRACE_S + + completed = subprocess.run( + [sys.executable, str(launcher)], + env=env, + capture_output=True, + text=True, + timeout=timeout, + ) + if completed.returncode != 0: + raise AssertionError( + f"{launcher.name} exited {completed.returncode} at world size {world_size}\n" + f"--- stdout ---\n{completed.stdout}\n" + f"--- stderr ---\n{completed.stderr}" + ) + return completed.stdout diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/test_modeling_v2_m3_allreduce.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/test_modeling_v2_m3_allreduce.py new file mode 100644 index 000000000000..c19a7461a368 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/test_modeling_v2_m3_allreduce.py @@ -0,0 +1,42 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for the ``m3_allreduce`` certification matrix. + +The matrix is ``_m3_allreduce_op_matrix.py`` beside this file, its own launcher; see ``_rank_job``. It drives every +cell of the entry against its reference, then requires the fused draft loop's boundaries built on it (plain +all-reduce, then the fused add + RMSNorm) to equal stock Eagle3's bit for bit under every single-node strategy, and +reports both against an fp64 reference as a warning, one line per strategy and case. + +The tp8 drafter runs at 8 ranks over two trays; a single-node job cannot form that group, so the matrix runs at each +world size of ``_rank_job.WORLD_SIZES`` the node's devices can form (the claim, draft boundary = stock boundary, does +not depend on the rank count) and skips the others. MNNVL, the strategy the two-tray group may take, is multi-node +only and not driven here. +""" + +import warnings + +import _rank_job +import pytest +import torch + +# Per world size, _rank_job.require_world skips the sizes this node's devices cannot form. +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + +_RESULT_TAG = "[ulp]" + + +# Spawns one rank process per device of the world size; under xdist workers would fight for them. +@pytest.mark.no_xdist +@pytest.mark.parametrize("world_size", _rank_job.WORLD_SIZES, ids=lambda w: f"world{w}") +def test_m3_allreduce_op_matrix(world_size) -> None: + out = _rank_job.run("m3_allreduce", world_size) + lines = [line for line in out.splitlines() if line.startswith(_RESULT_TAG)] + assert lines, f"the launcher reported no measurements:\n{out}" + warnings.warn( + f"draft boundary (= stock Eagle3) vs fp64 at {world_size} ranks, max BF16 ulps\n" + + "\n".join(lines), + stacklevel=1, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py new file mode 100644 index 000000000000..6d743c4c9a6e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/comm/test_modeling_v2_m3_mc_allreduce_norm_op_matrix.py @@ -0,0 +1,62 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Collected entry point for the multicast all-reduce + RMSNorm entries' certification matrix. + +Covers both ``m3_mc_allreduce_norm`` and ``m3_mc_finalize_allreduce_norm`` (one +kernel, one exchange protocol). The matrix is ``_m3_mc_allreduce_norm_op_matrix.py`` +beside this file, its own launcher; it drives every cell of each entry's +``CELLS`` against the entry's ``reference``. See ``_rank_job`` for why. + +The tp8 target runs these entries at 8 ranks over two trays, which a single-node +job cannot form. The world size is a runtime argument of the kernel and no cell's +spec names one, so the matrix certifies every cell at each world size of +``_rank_job.WORLD_SIZES`` the node's devices can form (2 and 4 on a 4-device tray, +8 on an 8-device node) and skips the others; it does not emulate the two-tray group. +""" + +import _rank_job +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.comm.m3_mc_finalize_allreduce_norm import ( # noqa: E501 + finalize_partial, +) + +assert torch.cuda.is_available(), "m3_mc_allreduce_norm requires CUDA devices" + + +# Spawns one rank process per device of the world size; under xdist workers would fight for them. +@pytest.mark.no_xdist +@pytest.mark.parametrize("world_size", _rank_job.WORLD_SIZES, ids=lambda w: f"world{w}") +def test_m3_mc_allreduce_norm_op_matrix(world_size) -> None: + _rank_job.run("m3_mc_allreduce_norm", world_size) + + +def test_the_cells_are_world_independent() -> None: + """What lets a single-node job certify the tp8 entries: no cell's spec names a world size.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.comm import ( # noqa: E501 + m3_mc_allreduce_norm, + m3_mc_finalize_allreduce_norm, + ) + + for op in ( + m3_mc_allreduce_norm.m3_mc_allreduce_norm, + m3_mc_finalize_allreduce_norm.m3_mc_finalize_allreduce_norm, + ): + assert op.CELLS + for cell in op.CELLS: + assert not {"world", "world_size", "rank"} & set(cell.spec), cell.why + + +def test_finalize_partial_rounds_the_routed_sum_before_the_shared_expert() -> None: + """As stock: the finalized routed output is BF16, and the shared expert is added to it (shared.add_(routed)). + Four routed rows of 1.0 at FP32 routing weight 0.2509765625 sum to 1.00390625, which rounds to BF16 1.0; plus the + shared -1 that is 0, where adding before the rounding would leave 0.00390625.""" + fc2 = torch.ones((4, 8), dtype=torch.bfloat16) + perm = torch.arange(4, dtype=torch.int32).view(1, 4) + scales = torch.full((1, 4), 0.2509765625, dtype=torch.float32) + shared = torch.full((1, 8), -1.0, dtype=torch.bfloat16) + out = finalize_partial(fc2, perm, scales, shared) + routed = finalize_partial(fc2, perm, scales, None) + assert torch.equal(out, shared.clone().add_(routed)) + assert torch.equal(out, torch.zeros(1, 8, dtype=torch.bfloat16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/embedding/test_modeling_v2_m3_embedding.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/embedding/test_modeling_v2_m3_embedding.py new file mode 100644 index 000000000000..3163ca2bf294 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/embedding/test_modeling_v2_m3_embedding.py @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_embedding entry of the MiniMax-M3 tp8 target: an exact gather with zero rows outside the +table.""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.embedding.m3_embedding import ( # noqa: E501 + m3_embedding as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_embedding requires a CUDA device" + +VOCAB, HIDDEN = 4096, 6144 + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + table = torch.randn(VOCAB, HIDDEN, generator=g, device="cuda").to(torch.bfloat16) + ids = torch.randint( + 0, VOCAB, (spec["rows"],), generator=g, device="cuda", dtype=spec["id_dtype"] + ) + if spec["pad"]: + ids[1::3] = -1 + with validating(op): + out = op(ids, table) + assert torch.equal(out, op.reference(ids, table)) + if spec["pad"]: + assert not out[1::3].any(), "padded ids must embed to zero rows" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/_m3_mxfp8_operands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/_m3_mxfp8_operands.py new file mode 100644 index 000000000000..e44ad56d743d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/_m3_mxfp8_operands.py @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MXFP8 operands for the MXFP8 GEMM entries' tests: E4M3 values and 128x4-swizzled UE8M0 block scales.""" + +import torch + + +def swizzle(byte: torch.Tensor) -> torch.Tensor: + """[N, K / 32] scale bytes into the 128x4-swizzled buffer of ``mxfp8_quantize(w, True)``.""" + rows, cols = byte.shape + padded_cols, padded_rows = (cols + 3) // 4 * 4, (rows + 127) // 128 * 128 + r = torch.arange(rows, device=byte.device).view(-1, 1) + c = torch.arange(cols, device=byte.device).view(1, -1) + index = ( + (c % 4) + + (c // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (r // 128) * (128 * padded_cols) + ) + out = torch.zeros(padded_rows * padded_cols, dtype=torch.uint8, device=byte.device) + out[index.reshape(-1)] = byte.reshape(-1) + return out + + +def mxfp8(rows: int, k: int, g: torch.Generator, lo: int = 118, hi: int = 125): + """An MXFP8 operand: E4M3 values of a unit normal, UE8M0 block scales 2^(lo-127) .. 2^(hi-128).""" + v = torch.randn(rows, k, generator=g, device="cuda").to(torch.float8_e4m3fn) + sf = torch.randint(lo, hi, (rows, k // 32), generator=g, device="cuda", dtype=torch.int32) + return v, swizzle(sf.to(torch.uint8)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_mxfp8_cublaslt.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_mxfp8_cublaslt.py new file mode 100644 index 000000000000..3941e4f773cb --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_mxfp8_cublaslt.py @@ -0,0 +1,141 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_cublaslt entry of the MiniMax-M3.1 sm_107 tp8 target (cuBLASLt's MXFP8 GEMM for the +generation steps past the decode bound, LINEAR_CUBLASLT).""" + +import os +import subprocess +import sys +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm import ( # noqa: E501 + m3_mxfp8_cublaslt as entry, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_cublaslt requires a CUDA device" +op = entry.m3_mxfp8_cublaslt + + +def _operands(cell): + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + w, w_sf = mxfp8(spec["n"], spec["k"], g) + if spec["prequant"]: + act, act_sf = mxfp8(spec["m"], spec["k"], g) + else: + act = (torch.randn(spec["m"], spec["k"], generator=g, device="cuda") / 4).to(torch.bfloat16) + act_sf = None + return act, act_sf, w, w_sf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """Same E4M3 operands and power-of-two scales on both sides; accumulation order then one bf16 rounding: at most + 2 units of ULP[bf16] of the row, RMS well under one.""" + act, act_sf, w, w_sf = _operands(cell) + with validating(op): + out = op(act, act_sf, w, w_sf) + assert out.shape == (cell.spec["m"], cell.spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(act, act_sf, w, w_sf), element_ulp=2.0, rms_ulp=0.25) + + +def test_the_quantize_is_the_native_one() -> None: + """The bf16 path's CuTe DSL quantize gives the bytes of ``torch.ops.trtllm.mxfp8_quantize(x, True)``, so the + cuBLASLt GEMM sees the operands FlashInfer's quantize + GEMM would.""" + g = torch.Generator(device="cuda").manual_seed(7) + x = (torch.randn(64, 6144, generator=g, device="cuda") / 4).to(torch.bfloat16) + act, act_sf = entry.mxfp8_quantize(x) + ref, ref_sf = torch.ops.trtllm.mxfp8_quantize(x, True) + assert torch.equal(act.view(torch.uint8), ref.view(64, 6144).view(torch.uint8)) + assert torch.equal(act_sf.view(torch.uint8).reshape(-1), ref_sf.view(torch.uint8).reshape(-1)) + + +@pytest.mark.parametrize("nk", entry.SHAPES, ids=str) +def test_repeats_and_graph_replay_are_bit_identical(nk) -> None: + """Three eager calls and a CUDA-graph replay (captured on a fresh stream) give the same bits: cuBLASLt's + algorithm is a pure function of the shape and the process's workspace.""" + n, k = nk + g = torch.Generator(device="cuda").manual_seed(n + k) + w, w_sf = mxfp8(n, k, g) + act, act_sf = mxfp8(entry.MAX_TOKENS, k, g) + first = op(act, act_sf, w, w_sf) + for _ in range(2): + assert torch.equal(op(act, act_sf, w, w_sf), first) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = op(act, act_sf, w, w_sf) + torch.cuda.current_stream().wait_stream(stream) + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(out, first) + + +# A fresh process: the target's prebuild warm-up (one eager call per shape, rows and activation), then the first +# capture of the entry on a fresh stream; prints the sha256 of the eager and the replayed output per (shape, rows). +_FRESH = """ +import hashlib, sys, torch +sys.path[:0] = sys.argv[1:3] +from {mod} import m3_mxfp8_cublaslt as op, mxfp8_quantize, SHAPES, MIN_TOKENS, MAX_TOKENS +from _m3_mxfp8_operands import mxfp8 +for (n, k) in SHAPES: + g = torch.Generator(device="cuda").manual_seed(n * 7 + k) + w, w_sf = mxfp8(n, k, g) + for m in (MIN_TOKENS, MAX_TOKENS): + x = (torch.randn(m, k, generator=g, device="cuda") / 4).to(torch.bfloat16) + act, act_sf = mxfp8_quantize(torch.zeros(m, k, dtype=torch.bfloat16, device="cuda")) + op(torch.zeros_like(x), None, w, w_sf) + op(act, act_sf, w, w_sf) + eager = op(x, None, w, w_sf) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = op(x, None, w, w_sf) + torch.cuda.current_stream().wait_stream(stream) + graph.replay() + torch.cuda.synchronize() + h = lambda t: hashlib.sha256(t.contiguous().view(torch.uint8).cpu().numpy().tobytes()).hexdigest()[:16] + print("SHA", n, k, m, h(eager), h(out)) +""" + + +def _fresh_process() -> dict: + here = os.path.dirname(os.path.abspath(__file__)) + code = _FRESH.format(mod=entry.__name__) + out = subprocess.run( + [sys.executable, "-c", code, here, os.path.join(here, "../../../..")], + capture_output=True, + text=True, + check=True, + ) + rows = [line.split() for line in out.stdout.splitlines() if line.startswith("SHA ")] + assert len(rows) == 2 * len(entry.SHAPES), out.stdout + out.stderr + return {tuple(r[1:4]): (r[4], r[5]) for r in rows} + + +def test_capture_after_the_prebuild_warmup_and_across_processes() -> None: + """In two fresh processes: after the prebuild-style eager warm-up, the entry's first capture works and replays the + eager bits (capture safety), and both processes give the same bits at every shape and its first / last row + (cross-process determinism: cuBLASLt's heuristic algorithm with torch's fixed default workspace, no timing).""" + first, second = _fresh_process(), _fresh_process() + for key, (eager, replay) in first.items(): + assert eager == replay, f"{key}: the replay differs from the eager call" + assert second[key] == (eager, replay), f"{key}: another process gave other bits" + + +def test_the_range_is_past_the_decode_bound() -> None: + """The entry's rows start past the R1VR8 decode bound and stop at the K7 verify rows at batch 8.""" + assert entry.MIN_TOKENS == 33 + assert entry.MAX_TOKENS == 64 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_unfused_index.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_unfused_index.py new file mode 100644 index 000000000000..4ae5f839edba --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_unfused_index.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test: the unfused sparse layers' separate MXFP8 index projection (``fuse_qkv_index_projection=False``) of the +tp8 target equals the index columns of the fused five-way projection on the same checkpoint tensors. + +Its weight rows are the five-way weight's index rows byte for byte, its block scales are applied (not dropped), and +on every engine MXFP8 backend its output is bitwise the five-way GEMM's index columns. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( + weights as tw, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_mxfp8_linear import ( # noqa: E501 + m3_mxfp8_linear as op, +) + +assert torch.cuda.is_available(), "the MXFP8 index projection requires a CUDA device" + +TP = 8 +HIDDEN = 6144 +BLOCK = 32 +# Text-model rows of the attention projections (64 query / 4 KV heads, 4 index heads, head and index dim 128). +ROWS = {"q": 64 * 128, "k": 4 * 128, "v": 4 * 128, "index_q": 4 * 128, "index_k": 128} +_FLAGS = { + "native": (False, False, False, False), + "flashinfer": (False, False, True, False), + "tuned": (True, False, False, False), +} + + +def _checkpoint(seed: int): + """E4M3 weights and UE8M0 block scales per projection; the scales spread over 2^-6 .. 2^5 (bytes 121..132).""" + g = torch.Generator(device="cuda").manual_seed(seed) + out = {} + for name, rows in ROWS.items(): + w = (torch.randn(rows, HIDDEN, generator=g, device="cuda") * 2).to(torch.float8_e4m3fn) + s = torch.randint( + 121, 133, (rows, HIDDEN // BLOCK), generator=g, device="cuda", dtype=torch.uint8 + ) + out[name] = (w.cpu(), s.cpu()) + return out + + +def _ns(rank: int) -> SimpleNamespace: + return SimpleNamespace(tp=TP, rank=rank, dev=torch.device("cuda"), head_dim=128, index_dim=128) + + +def _packs(ck, rank): + d = _ns(rank) + names = ("q", "k", "v", "index_q", "index_k") + w5 = tw._qkv_sparse_w(tuple(ck[n][0] for n in names), d) + s5 = tw._qkv_sparse_s( + tuple(ck[n][1] for n in names), d + ) # UE8M0 bytes, as the checkpoint stores them + wi = tw._index_qk_w((ck["index_q"][0], ck["index_k"][0]), d) + si = tw._index_qk_s((ck["index_q"][1], ck["index_k"][1]), d) + return w5, s5.view(torch.uint8), wi, si.view(torch.uint8) + + +@pytest.mark.parametrize("rank", [0, TP - 1]) +def test_weight_rows_are_the_five_way_index_rows(rank) -> None: + ck = _checkpoint(1234 + rank) + w5, _, wi, _ = _packs(ck, rank) + n = wi.shape[0] + assert wi.dtype == torch.float8_e4m3fn and wi.shape == (n, HIDDEN) + assert torch.equal(wi.view(torch.uint8), w5[-n:].view(torch.uint8)) + + +@pytest.mark.parametrize("path", list(_FLAGS)) +@pytest.mark.parametrize("rows", [1, 4, 8, 32, 64, 256]) +def test_unfused_output_is_bitwise_the_five_way_index_columns(path, rows) -> None: + if path == "flashinfer" and getattr(torch.ops.trtllm, "flashinfer_mm_mxfp8", None) is None: + pytest.skip("this build has no trtllm::flashinfer_mm_mxfp8") + ck = _checkpoint(77) + w5, s5, wi, si = _packs(ck, 0) + n = wi.shape[0] + g = torch.Generator(device="cuda").manual_seed(rows) + x = torch.randn(rows, HIDDEN, generator=g, device="cuda").to(torch.bfloat16) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + out5 = op(x, w5, s5, alpha, torch.bfloat16, *_FLAGS[path]) + outi = op(x, wi, si, alpha, torch.bfloat16, *_FLAGS[path]) + assert outi.shape == (rows, n) + assert torch.equal(outi, out5[:, -n:]), ( + f"{path}: max |diff| {(outi.float() - out5[:, -n:].float()).abs().max().item()}" + ) + + +def test_block_scales_are_applied() -> None: + """The output follows the op's reference with the checkpoint scales and differs from a unit-scale (dropped) + projection by far more than rounding.""" + ck = _checkpoint(5) + _, _, wi, si = _packs(ck, 0) + unit = tw._index_qk_s( + (torch.full_like(ck["index_q"][1], 127), torch.full_like(ck["index_k"][1], 127)), _ns(0) + ).view(torch.uint8) + x = torch.randn(8, HIDDEN, device="cuda").to(torch.bfloat16) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + out = op(x, wi, si, alpha, torch.bfloat16, *_FLAGS["native"]) + ref = op.reference(x, wi, si, alpha, torch.bfloat16, *_FLAGS["native"]) + dropped = op(x, wi, unit, alpha, torch.bfloat16, *_FLAGS["native"]) + scale = ref.float().abs().mean() + # The engine GEMM rounds once to BF16 after an fp32 accumulation in its own order: within 2 BF16 ulps (2^-7 + # relative) of the reference per element. + assert bool( + ((out.float() - ref.float()).abs() <= 2.0**-7 * ref.float().abs() + 1e-3 * scale).all() + ) + assert (out.float() - dropped.float()).abs().mean() > 0.5 * scale diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_vr_mxgemv_vrcfg.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_vr_mxgemv_vrcfg.py new file mode 100644 index 000000000000..909c38fac927 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m31_vr_mxgemv_vrcfg.py @@ -0,0 +1,76 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""VRCFG-SWI: the sm_107 TP8 mx_gemv's swiglu-prologue calls of the dense down projection (6144, 1536) at up to 8 rows +run ring 3 instead of the table's (ring 8, clamped to the 6 k-tiles a rank owns), bit for bit the table config's +output (same split, same k-tile order); other calls keep the table; TRTLLM_M31_DISABLE_MXGEMV_VRCFG restores the table +for every call.""" + +import importlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 -- registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + m31_switches, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.mx_gemv import ( # noqa: E501 + op as gemv, +) + +N, K = 6144, 1536 + + +def _ring(m: int, x_swiglu: bool): + res = gemv._resolve(m, N, K, True, True, None, True, None, x_swiglu, False) + return None if res is None else res[0][3] + + +def test_vrcfg_scope(): + assert gemv.MXGEMV_VRCFG and gemv._CONFIGS_SWIGLU == {(N, K): (2, 3, 0)} + table = gemv.get_config(N, K, 8) + assert table == (2, 8, 0) + for m in range(1, 9): + assert _ring(m, True) == 3 + assert _ring(m, False) == 6 + assert _ring(16, True) == 6 # past the swept rows: the table + + +def test_kill_switch(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("TRTLLM_M31_DISABLE_MXGEMV_VRCFG", "1") + importlib.reload(m31_switches) + try: + g = importlib.reload(gemv) + assert not g.MXGEMV_VRCFG and g._CONFIGS_SWIGLU == {} + res = g._resolve(8, N, K, True, True, None, True, None, True, False) + assert res is not None and res[0][3] == 6 + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_MXGEMV_VRCFG", raising=False) + importlib.reload(m31_switches) + importlib.reload(gemv) + assert gemv.MXGEMV_VRCFG + + +def _mxfp8_weight(seed): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(N, K, generator=g, device="cuda") / K**0.5).to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + return wq.view(N, K), wsf + + +@pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target's GEMV runs on SM107 only", +) +@pytest.mark.parametrize("m", [1, 4, 7, 8]) +def test_ring3_is_bitwise_the_table(m): + wq, wsf = _mxfp8_weight(m) + g = torch.Generator(device="cuda").manual_seed(100 + m) + gate_up = ( + (torch.randn(m, 2 * K, generator=g, device="cuda") * 3).to(torch.bfloat16).contiguous() + ) + got = gemv.m3_mx_gemv_swiglu(gate_up, wq, wsf, use_pdl=True) + table = gemv.m3_mx_gemv_swiglu(gate_up, wq, wsf, cfg=gemv.get_config(N, K, m), use_pdl=True) + torch.cuda.synchronize() + assert got is not None and table is not None and bool(table.any()) + assert torch.equal(got, table) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_bf16_linear.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_bf16_linear.py new file mode 100644 index 000000000000..7f5344eb551c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_bf16_linear.py @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_bf16_linear entry of the MiniMax-M3 tp8 target (the drafter's projections).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_bf16_linear import ( # noqa: E501 + m3_bf16_linear as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_bf16_linear requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """Accumulation order (cuBLAS split-K) then one bf16 rounding on both sides: at most 2 units of ULP[bf16] of the + row, RMS well under one; measured on GB300: worst 0.5 unit.""" + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = torch.randn(spec["m"], spec["k"], generator=g, device="cuda").to(torch.bfloat16) + w = (torch.randn(spec["n"], spec["k"], generator=g, device="cuda") * 0.02).to(torch.bfloat16) + with validating(op): + out = op(x, w) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(x, w), element_ulp=2.0, rms_ulp=0.25) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_dense_swiglu.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_dense_swiglu.py new file mode 100644 index 000000000000..f11d10a99a9d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_dense_swiglu.py @@ -0,0 +1,142 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Bitwise oracle of the dense MLP's swigluoai C++ op (ops/gemm/m3_dense_swiglu, +trtllm::minimax_m3_sm107_tp8_swiglu_oai) against the Triton silu_and_mul kernel the dense MLP runs today +(modules/swiglu.py at limit 7, alpha 1.702, beta 1): every bf16 gate bit pattern against special and random up values, +every up bit pattern against special and random gates, random pairs, and the dense MLP's shapes at every decode row +count, eager and replayed from a CUDA graph, with and without the programmatic launch.""" + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.silu_and_mul +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_dense_swiglu import ( # noqa: E501 + ALPHA, + BETA, + LIMIT, + m3_dense_swiglu, +) + +assert torch.cuda.is_available(), "the dense swigluoai op requires a CUDA device" +if not hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp8_swiglu_oai"): + pytest.skip( + "needs the TP8 op library with minimax_m3_sm107_tp8_swiglu_oai", allow_module_level=True + ) + +# The TP8 dense MLP's activation width (gate_up 3072 -> d 1536). +DENSE_D = modeling.DENSE_DOWN[1] + + +def _triton(x: torch.Tensor) -> torch.Tensor: + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=LIMIT, swiglu_alpha=ALPHA, swiglu_beta=BETA + ) + + +def _op(x: torch.Tensor, pdl: bool = True) -> torch.Tensor: + out = m3_dense_swiglu(x, LIMIT, ALPHA, BETA, pdl) + assert out is not None, "the op declined a contiguous bf16 [rows, 2d] input" + return out + + +def _row(gate: torch.Tensor, up: torch.Tensor) -> torch.Tensor: + return torch.cat([gate.view(1, -1), up.view(1, -1)], dim=1).contiguous() + + +def _all_bf16() -> torch.Tensor: + return ( + torch.arange(1 << 16, dtype=torch.int32, device="cuda").to(torch.int16).view(torch.bfloat16) + ) + + +def _specials(n: int, seed: int) -> list: + g = torch.Generator(device="cuda").manual_seed(seed) + vals = [0.0, -0.0, 1.0, -1.0, 6.99, 7.0, 7.01, -7.0, -7.01, 1e-38, -1e-38, 1e-40, 88.0, -88.0, 3e38, -3e38, + float("inf"), float("-inf"), float("nan")] # fmt: skip + rows = [torch.full((n,), v, dtype=torch.bfloat16, device="cuda") for v in vals] + for scale in (0.05, 1.0, 4.0, 30.0): + rows.append((scale * torch.randn(n, generator=g, device="cuda")).to(torch.bfloat16)) + rows.append( + torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + .to(torch.int16) + .view(torch.bfloat16) + ) + return rows + + +def _assert_bitwise(got: torch.Tensor, ref: torch.Tensor, what: str) -> None: + bad = (got.view(torch.int16) != ref.view(torch.int16)).nonzero().flatten() + assert bad.numel() == 0, f"{what}: {bad.numel()} mismatches, first at {bad[:4].tolist()}" + + +def test_every_gate_pattern_is_bitwise_the_triton_kernel() -> None: + gate = _all_bf16() + for j, up in enumerate(_specials(gate.numel(), 1)): + x = _row(gate, up) + _assert_bitwise(_op(x), _triton(x), f"up set {j}") + + +def test_every_up_pattern_is_bitwise_the_triton_kernel() -> None: + up = _all_bf16() + for j, gate in enumerate(_specials(up.numel(), 2)): + x = _row(gate, up) + _assert_bitwise(_op(x), _triton(x), f"gate set {j}") + + +@pytest.mark.parametrize("seed", range(4)) +def test_random_pairs_are_bitwise_the_triton_kernel(seed: int) -> None: + g = torch.Generator(device="cuda").manual_seed(100 + seed) + n = 1 << 22 + + def pattern() -> torch.Tensor: + bits = torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + return bits.to(torch.int16).view(torch.bfloat16) + + x = _row(pattern(), pattern()) + _assert_bitwise(_op(x), _triton(x), f"seed {seed}") + + +@pytest.mark.parametrize("pdl", [True, False], ids=["pdl", "plain"]) +@pytest.mark.parametrize("rows", range(1, modeling.DECODE_MAX_TOKENS + 1)) +def test_dense_shape_eager_and_graph_are_bitwise_the_triton_kernel(rows: int, pdl: bool) -> None: + """The TP8 dense MLP's gate_up rows (a GEMV output's range plus values past the clamps), eager and replayed from a + CUDA graph after fresh rows are written in place, against the Triton kernel on the same rows.""" + g = torch.Generator(device="cuda").manual_seed(2000 + rows) + x = (3.0 * torch.randn(rows, 2 * DENSE_D, generator=g, device="cuda")).to(torch.bfloat16) + x[:, :8] = torch.tensor([8.0, -8.0, 7.0, -7.0, 0.0, -0.0, 50.0, -50.0], dtype=torch.bfloat16) + x[:, DENSE_D : DENSE_D + 8] = x[:, :8] + _assert_bitwise(_op(x, pdl), _triton(x), f"eager rows {rows}") + out = torch.empty(rows, DENSE_D, dtype=torch.bfloat16, device="cuda") + _op(x, pdl) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out.copy_(_op(x, pdl)) + x.copy_((3.0 * torch.randn(rows, 2 * DENSE_D, generator=g, device="cuda")).to(torch.bfloat16)) + graph.replay() + torch.cuda.synchronize() + _assert_bitwise(out, _triton(x), f"graph rows {rows}") + + +def test_the_op_declines_other_parameters_and_layouts() -> None: + x = torch.zeros(2, 2 * DENSE_D, dtype=torch.bfloat16, device="cuda") + assert m3_dense_swiglu(x, 8.0, ALPHA, BETA, True) is None + assert m3_dense_swiglu(x, LIMIT, 1.0, BETA, True) is None + assert m3_dense_swiglu(x, LIMIT, ALPHA, 0.0, True) is None + assert m3_dense_swiglu(x.float(), LIMIT, ALPHA, BETA, True) is None + assert m3_dense_swiglu(x[:, : 2 * DENSE_D - 2], LIMIT, ALPHA, BETA, True) is None + + +@pytest.mark.parametrize("cell", m3_dense_swiglu.CELLS, ids=lambda c: c.why) +def test_every_cell_is_bitwise_the_triton_kernel(cell) -> None: + """Each declared cell (the dense MLP's activation at its decode rows) through the op, bit for bit the Triton + kernel on gate_up rows in a GEMV output's range plus values past the clamps.""" + rows, d = cell.spec["rows"], cell.spec["d"] + g = torch.Generator(device="cuda").manual_seed(3000 + rows) + x = (3.0 * torch.randn(rows, 2 * d, generator=g, device="cuda")).to(torch.bfloat16) + x[:, :8] = torch.tensor([8.0, -8.0, 7.0, -7.0, 0.0, -0.0, 50.0, -50.0], dtype=torch.bfloat16) + x[:, d : d + 8] = x[:, :8] + _assert_bitwise(_op(x), _triton(x), cell.why) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_draft_lowm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_draft_lowm.py new file mode 100644 index 000000000000..87be936710b4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_draft_lowm.py @@ -0,0 +1,177 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gemm/m3_draft_lowm entry: the Eagle3 drafter's BF16 linears on FlashInfer's fixed-tactic low-M +kernels (direct SIMT and split-K tensor-core), at the TP8 drafter shards, at the rows its table certifies. + +Every cell is held to the fp64 reference within the derived band. The drafter's projection call +(``M3Eagle3Draft.linear``) at a covered row count is the entry's call bit for bit, at an uncovered one the in-tree +GEMM's. +""" + +import zlib + +import pytest +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_draft_lowm import ( # noqa: E501 + ROWS, + TACTICS, + tactic_for, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_draft_lowm import ( # noqa: E501 + m3_draft_lowm as op, +) +from tensorrt_llm._torch.flashinfer_utils import get_env_enable_pdl + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_draft_lowm requires a CUDA device" + +# Element band, in BF16 ulps of the row scale, and the relative RMS (``assert_within_ulp``). +# +# Kernel and reference multiply the same BF16 operands (every product exact in fp32); the kernel sums in fp32 (the +# direct kernel one dot product in K order, the split-K kernel tensor-core blocks and then its split partials), the +# reference in fp64, and both round once to BF16. With w ~ N(0, 1 / K) and x ~ N(0, 1) the outputs are O(1) and the +# fp32 order difference is ~sqrt(K) * 2^-24 of the summed magnitudes, a few 1e-6 at K = 18432: far below half a BF16 +# ulp, so the two roundings disagree by at most one ulp of the element -- up to two units of the row scale for an +# element in its row maximum's binade -- and only where the sum sits that close to a rounding boundary, a few in ten +# thousand elements, hence 0.1 ulp RMS. The same derivation as the decode GEMV's band (test_modeling_v2_m3_mx_gemv), +# with no slack under it: a second ulp would mean a wrong sum. +ELEMENT_ULP = 2.0 +RMS_ULP = 0.1 + +#: The drafter's projections (N, K) and their weight keys on ``M3Eagle3Draft.w``. +DRAFTER_LINEARS = { + (1280, 12288): "qkv", + (6144, 1024): "o", + (4608, 6144): "gu", + (6144, 2304): "dn", + (6144, 18432): "fc", +} + + +@pytest.fixture(autouse=True) +def _policy(): + modeling.set_require(False) + modeling.reset_logged() + yield + modeling.set_require(False) + modeling.reset_logged() + + +def build(n: int, k: int, m: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = torch.randn(m, k, generator=g, device="cuda").to(torch.bfloat16) + return x, w + + +def test_the_table_is_the_tp8_drafter() -> None: + assert set(TACTICS) == set(DRAFTER_LINEARS) + assert {c.spec["nk"] for c in op.CELLS} == set(TACTICS) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + (n, k), m = cell.spec["nk"], cell.spec["m"] + x, w = build(n, k, m, seed=zlib.crc32(cell.why.encode()) % 2**31) + kind, tactic = tactic_for(n, k, m) + out = torch.full((m, n), float("nan"), dtype=torch.bfloat16, device="cuda") + args = ( + x, + w, + out, + kind, + tactic, + True, + ) # the target launches with PDL when the environment enables it + certifying(op) + assert op(*args) is out + ref = op.reference(*args) + assert out.shape == (m, n) and out.dtype is torch.bfloat16 + assert_within_ulp(out, ref, element_ulp=ELEMENT_ULP, rms_ulp=RMS_ULP) + # PDL only changes the launch attribute. + plain = torch.empty_like(out) + op(x, w, plain, kind, tactic, False) + assert torch.equal(plain.view(torch.int16), out.view(torch.int16)) + + +def _draft(linears=DRAFTER_LINEARS): + """A drafter holding only its projection weights and the low-M bookkeeping (the module is not built).""" + draft = object.__new__(modeling.M3Eagle3Draft) + draft.w = {key: build(*nk, 1, seed=nk[0] + nk[1])[1] for nk, key in linears.items()} + draft._lowm_built, draft._lowm_engaged = set(), set() + draft._push_arm, draft._pushed = None, 0 + return draft + + +@pytest.fixture(scope="module") +def prebuilt(): + draft = _draft() + assert draft.prebuild_linears() == len(op.CELLS) + assert draft.prebuild_linears() == 0, "a second prebuild finds nothing left to compile" + return draft + + +@pytest.mark.parametrize("nk", sorted(DRAFTER_LINEARS), ids=lambda nk: DRAFTER_LINEARS[nk]) +def test_the_projection_is_the_entry_and_falls_back_past_its_rows(prebuilt, nk) -> None: + n, k = nk + key = DRAFTER_LINEARS[nk] + w = prebuilt.w[key] + use_pdl = get_env_enable_pdl() + modeling.set_require(True) + for m in ROWS[nk]: + assert modeling.DRAFT_LOWM.covers((nk, m)) + x, _ = build(n, k, m, seed=m) + got = prebuilt.linear(x, key) + kind, tactic = tactic_for(n, k, m) + want = op(x, w, torch.empty(m, n, dtype=x.dtype, device="cuda"), kind, tactic, use_pdl) + assert torch.equal(got.view(torch.int16), want.view(torch.int16)), m + # Rows the table does not certify (the b2 rows, and past it): the in-tree GEMM, silently, even under require. + for m in sorted(set(range(1, modeling.DECODE_MAX_TOKENS + 2)) - set(ROWS[nk])): + assert not modeling.DRAFT_LOWM.covers((nk, m)) + x, _ = build(n, k, m, seed=m) + got = prebuilt.linear(x, key) + assert torch.equal(got.view(torch.int16), F.linear(x, w).view(torch.int16)), m + + +def test_a_projection_outside_the_table_is_the_in_tree_gemm() -> None: + draft = _draft({(256, 512): "o"}) + x, _ = build(256, 512, 1, seed=3) + assert draft._lowm_linear(draft.w["o"], x) is None + got = draft.linear(x, "o") + assert torch.equal(got.view(torch.int16), F.linear(x, draft.w["o"]).view(torch.int16)) + + +def test_a_covered_call_not_prebuilt_raises_under_require() -> None: + draft = _draft({(6144, 1024): "o"}) + x, _ = build(6144, 1024, 1, seed=5) + modeling.set_require(True) + with pytest.raises(modeling.FastPathUnavailable, match="not compiled before warmup"): + draft.linear(x, "o") + # Outside require it falls back to the in-tree GEMM. + modeling.set_require(False) + got = draft.linear(x, "o") + assert torch.equal(got.view(torch.int16), F.linear(x, draft.w["o"]).view(torch.int16)) + + +def test_guard_refuses_a_strided_activation() -> None: + x, w = build(6144, 1024, 2, seed=6) + wide = torch.zeros(2, 2048, dtype=torch.bfloat16, device="cuda") + wide[:, :1024] = x + kind, tactic = tactic_for(6144, 1024, 2) + with pytest.raises(AssertionError, match="contiguous"): + with validating(op): + op( + wide[:, :1024], + w, + torch.empty(2, 6144, dtype=torch.bfloat16, device="cuda"), + kind, + tactic, + False, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_fp32_linear.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_fp32_linear.py new file mode 100644 index 000000000000..aab8e920657a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_fp32_linear.py @@ -0,0 +1,32 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_fp32_linear entry of the MiniMax-M3 tp8 target (the router beyond the decode routers' rows).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_fp32_linear import ( # noqa: E501 + m3_fp32_linear as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_fp32_linear requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """torch's default fp32 matmul precision on this hardware may be TF32 (~2^-11 relative per product): the band is + stated in fp32 ulps of each row's largest logit and covers it (K = 6144 TF32 products: ~2^13 ulps worst element); + measured on GB300 with the engine's defaults: worst element ~1700 ulps, RMS ~300.""" + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = (torch.randn(spec["m"], spec["k"], generator=g, device="cuda") * 2.0).to(torch.bfloat16) + w = torch.randn(spec["e"], spec["k"], generator=g, device="cuda") * 0.02 + with validating(op): + out = op(x, w) + assert out.dtype == torch.float32 and out.shape == (spec["m"], spec["e"]) + assert_within_ulp(out, op.reference(x, w), element_ulp=8192.0, rms_ulp=2048.0) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv.py new file mode 100644 index 000000000000..64f44b74934d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gemm/m3_mx_gemv entry: the decode MXFP8 GEMV on a producer-quantized activation. + +Operands as the target holds them: the weight through mxfp8_quantize with the swizzled scale layout (the MXFP8 linear +method's storage), the activation through mxfp8_quantize with the swizzled layout (what the boundary all-reduce and +the attention merge publish). +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_mx_gemv import ( # noqa: E501 + m3_mx_gemv as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mx_gemv requires a CUDA device" + + +# Element band, in ULP units of the row scale: see compare. +ELEMENT_ULP = 1.5 + + +def compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """Element ELEMENT_ULP of the row scale, relative RMS 0.1 ulp. + + Kernel and reference multiply bit-identical dequantized operands; the kernel sums in fp32 (tensor-core blocks, + then the split-K partials), the reference in fp64. Over K <= 6144 that order difference is ~1e-6 relative, so the + two bf16 roundings disagree only where the sum sits within that of a rounding boundary: one bf16 ulp of that + element, which for an element in its row maximum's binade is up to two of ``assert_within_ulp``'s units of the row + scale, and a few in ten thousand elements in aggregate. Any fp32 sum shows it: torch's IEEE fp32 matmul of the + same dequantized operands measured at most 1.22 units against the fp64 reference over every row count 1..48 on + the o_proj and the QKV (6 seeds each, GB300), the kernel at most 1.25, both at most 0.009 ulp RMS. The band + leaves the margin between that and a full extra ulp. The kernel is the same on SM107 (the tp2 table only picks + split and ring), and its TP2 projections' K is 1536..6144. + """ + assert_within_ulp(out, ref, element_ulp=ELEMENT_ULP, rms_ulp=0.1) + + +def build(n: int, k: int, m: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = torch.randn(m, k, generator=g, device="cuda").to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + return x, xq.view(m, k), xsf, wq.view(n, k), wsf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + (n, k), m = cell.spec["nk"], cell.spec["m"] + _, xq, xsf, w, wsf = build(n, k, m, seed=zlib.crc32(cell.why.encode()) % 2**31) + with validating(op): + out = op(xq, xsf, w, wsf, use_pdl=True) # the target launches with PDL + assert out is not None, "a certified shape must be served" + assert out.shape == (m, n) and out.dtype is torch.bfloat16 + compare(out, op.reference(xq, xsf, w, wsf)) + + +def test_an_unserved_shape_returns_none() -> None: + """Shapes outside the tuned table fall back at the call site; the op says so rather than computing.""" + _, xq, xsf, w, wsf = build(256, 512, 1, seed=1) + assert op(xq, xsf, w, wsf) is None + + +def test_guard_refuses_linear_activation_scales() -> None: + """A linear [M, K / 32] scale buffer is read as the swizzled one.""" + x, _, _, w, wsf = build(6144, 4096, 4, seed=2) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, False) + with pytest.raises(AssertionError, match="swizzled"): + with validating(op): + op(xq.view(4, 4096), xsf, w, wsf) + + +def test_guard_refuses_a_strided_weight() -> None: + _, xq, xsf, w, wsf = build(6144, 4096, 1, seed=3) + wide = torch.zeros(6144, 8192, dtype=w.dtype, device="cuda") + wide[:, :4096] = w + with pytest.raises(AssertionError, match="contiguous e4m3"): + with validating(op): + op(xq, xsf, wide[:, :4096], wsf) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv_bf16.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv_bf16.py new file mode 100644 index 000000000000..8b76baea4c50 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv_bf16.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the gemm/m3_mx_gemv_bf16 entry: the decode MXFP8 GEMV quantizing a bf16 activation in its prologue. + +The weight as the MXFP8 linear method stores it (mxfp8_quantize, swizzled scales). Besides the reference band, the +prologue's quantization is held to mxfp8_quantize itself: the GEMV on a bf16 x must equal, bit for bit, the +prequantized GEMV on mxfp8_quantize(x) -- the claim the target relies on when it mixes the two paths. +""" + +import zlib + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_mx_gemv import ( # noqa: E501 + m3_mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_mx_gemv_bf16 import ( # noqa: E501 + m3_mx_gemv_bf16 as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mx_gemv_bf16 requires a CUDA device" + +# The element band of test_modeling_v2_m3_mx_gemv.compare (derived there): the same fp32 sums against fp64. +ELEMENT_ULP = 1.5 + + +def build(n: int, k: int, m: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = torch.randn(m, k, generator=g, device="cuda").to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + return x, wq.view(n, k), wsf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + (n, k), m = cell.spec["nk"], cell.spec["m"] + x, w, wsf = build(n, k, m, seed=zlib.crc32(cell.why.encode()) % 2**31) + with validating(op): + out = op(x, w, wsf, use_pdl=True) # the target launches with PDL + assert out is not None, "a certified shape must be served" + assert out.shape == (m, n) and out.dtype is torch.bfloat16 + # The band of test_modeling_v2_m3_mx_gemv.compare: the quantized operands are the reference's exactly (checked + # below), the sums differ only in order. + assert_within_ulp(out, op.reference(x, w, wsf), element_ulp=ELEMENT_ULP, rms_ulp=0.1) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + pre = m3_mx_gemv(xq.view(m, k), xsf, w, wsf, use_pdl=True) + assert pre is not None and torch.equal(out.view(torch.int16), pre.view(torch.int16)), ( + "the prologue's MXFP8 quantization differs from mxfp8_quantize" + ) + + +def test_a_strided_activation_is_declined() -> None: + """The prologue reads x densely; the op declines a strided x (None) rather than computing on it.""" + x, w, wsf = build(6144, 4096, 2, seed=1) + wide = torch.zeros(2, 8192, dtype=torch.bfloat16, device="cuda") + wide[:, :4096] = x + assert op(wide[:, :4096], w, wsf) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv_producer.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv_producer.py new file mode 100644 index 000000000000..3ac3588a2cfc --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mx_gemv_producer.py @@ -0,0 +1,236 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Oracle for #117, the sparse QKV + index GEMV with the producer in its owner epilogue (mx_gemv +``m3_mx_gemv_producer``): q, index-q and every cache byte (NVFP4 K / V data and scales, FP8 index-K) bit for bit equal +to today's chain, the GEMV storing the packed BF16 row and then the producer kernel on it. Both producer builds are +references: the library's ``minimax_m3_producer_pw`` and, when the target's extension is loaded, its own +``minimax_m3_sm107_tp8_producer_pw`` (the copy the target dispatches, with its pre-wait step-input reads).""" + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.mx_gemv import ( # noqa: E501 + op as gemv, +) + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + +N, K = 12 * 128, 6144 +HEADS_Q = 8 +PAGE = 128 +PAGES = 512 +MAX_POSITIONS = 262144 +EPS = 1e-6 +KV_QUANT = (1.0, 0.37, 0.61) + + +def _case(tokens, seed, scale=1.0, bad_slots=False): + g = torch.Generator(device="cuda").manual_seed(seed) + w = (torch.randn(N, K, generator=g, device="cuda") * (scale / K**0.5)).to(torch.bfloat16) + x = torch.randn(tokens, K, generator=g, device="cuda").to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x, True) + rotary = torch.randn(MAX_POSITIONS, 2, 32, device="cuda", generator=g) + norms = [(torch.randn(128, device="cuda", generator=g) * 0.3).bfloat16() for _ in range(4)] + slots = torch.randperm(PAGES * PAGE, device="cuda", generator=g)[:tokens].int() + if bad_slots and tokens >= 3: + slots[1] = -1 + slots[2] = PAGES * PAGE + 5 + positions = torch.randint( + 0, MAX_POSITIONS, (tokens,), device="cuda", dtype=torch.int32, generator=g + ) + quant = torch.tensor(KV_QUANT, device="cuda") + return xq.view(tokens, K), xsf, wq.view(N, K), wsf, slots, quant, norms, rotary, positions + + +def _caches(): + return ( + torch.zeros(PAGES, 2, 1, PAGE, 64, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 2, 1, PAGE, 8, dtype=torch.uint8, device="cuda"), + torch.zeros(PAGES, 1, PAGE, 128, dtype=torch.float8_e4m3fn, device="cuda"), + ) + + +def _chain(case, caches, producer, eps=EPS): + xq, xsf, wq, wsf, slots, quant, norms, rotary, positions = case + packed = gemv.m3_mx_gemv(xq, xsf, wq, wsf, use_pdl=True) + assert packed is not None, "the sparse QKV shape must be served" + if producer == "lib": + q, iq = torch.ops.trtllm.minimax_m3_producer_pw( + packed, *caches, slots, quant, eps, *norms, rotary, positions, HEADS_Q, True, True + ) + else: + q, iq = torch.ops.trtllm.minimax_m3_sm107_tp8_producer_pw( + packed, *caches, slots, quant, eps, *norms, rotary, positions, True, True, True + ) + torch.cuda.synchronize() + return q, iq + + +def _fused(case, caches, eps=EPS): + xq, xsf, wq, wsf, slots, quant, norms, rotary, positions = case + tokens = xq.shape[0] + q = torch.zeros(tokens, HEADS_Q, 128, dtype=torch.float8_e4m3fn, device="cuda") + iq = torch.zeros(tokens, 1, 128, dtype=torch.float8_e4m3fn, device="cuda") + kv_data, kv_scale, index_k = caches + ok = gemv.m3_mx_gemv_producer( + xq, xsf, wq, wsf, q, iq, kv_data, kv_scale, index_k, slots, quant, eps, *norms, rotary, positions, + use_pdl=True, + ) # fmt: skip + assert ok, "the producer GEMV must serve the sparse QKV shape at <= 8 tokens" + torch.cuda.synchronize() + return q, iq + + +def _producers(): + out = ["lib"] + if hasattr(torch.ops.trtllm, "minimax_m3_sm107_tp8_producer_pw"): + out.append("tp8") + return out + + +@pytest.mark.parametrize( + "tokens, seed, scale, bad_slots", + [ + (1, 21, 1.0, False), + (2, 22, 1.0, False), + (4, 23, 1.0, False), + (4, 24, 40.0, False), + (8, 25, 1.0, False), + (8, 26, 0.01, False), + (8, 27, 1.0, True), + ], + ids=["t1", "t2", "t4", "t4-large", "t8", "t8-tiny", "t8-bad-slots"], +) +@pytest.mark.parametrize("producer", _producers()) +def test_producer_gemv_is_bitwise_the_chain(tokens, seed, scale, bad_slots, producer) -> None: + case = _case(tokens, seed, scale, bad_slots) + ref_caches = _caches() + rq, riq = _chain(case, ref_caches, producer) + caches = _caches() + q, iq = _fused(case, caches) + names = ("q", "index_q", "kv_data", "kv_scale", "index_k") + for name, a in zip(names, (rq, riq, *ref_caches)): + assert bool(a.contiguous().view(torch.uint8).any()), f"reference {name} is all zero" + for name, a, b in zip(names, (rq, riq, *ref_caches), (q, iq, *caches)): + a8, b8 = a.contiguous().view(torch.uint8), b.contiguous().view(torch.uint8) + diff = (a8 != b8).nonzero() + assert diff.numel() == 0, ( + f"{name}: {diff.shape[0]} bytes differ, first at {tuple(diff[0].tolist())}" + ) + + +def _edge_case(tokens, seed, edge): + """The producer's arithmetic edges on the packed row: the GEMV weight is an identity on the first 1536 inputs, so + the packed BF16 row is the MXFP8-rounded input row.""" + g = torch.Generator(device="cuda").manual_seed(seed) + rows = torch.randn(tokens, N // 128, 128, device="cuda", generator=g) + if edge == "zero_rows": + rows[0] = 0.0 + rows[1, HEADS_Q + 1] = 0.0 # a zero v row: an all-zero NVFP4 block + elif edge == "denormal_rms": + rows[:2] = ( + rows[:2] * 1e-20 + ) # mean square below FLT_MIN with eps = 0: the rsqrt's denormal path + elif edge == "saturate": + rows[0] = ( + rows[0] * 1e4 + ) # past E4M3 448 after the norm's (1 + w), and saturating NVFP4 block scales + rows[1, :, ::7] = 3e38 + elif edge == "nan_inf": + rows[0, :, 5] = float("nan") + rows[1, :, 9] = float("inf") + rows[2, :, 40] = -float("inf") + x = torch.zeros(tokens, K, device="cuda") + x[:, :N] = rows.reshape(tokens, N) + w = torch.zeros(N, K, device="cuda") + w[:, :N] = torch.eye(N, device="cuda") + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w.to(torch.bfloat16), True) + xq, xsf = torch.ops.trtllm.mxfp8_quantize(x.to(torch.bfloat16), True) + rotary = torch.randn(MAX_POSITIONS, 2, 32, device="cuda", generator=g) + norms = [(torch.randn(128, device="cuda", generator=g) * 0.3).bfloat16() for _ in range(4)] + slots = torch.randperm(PAGES * PAGE, device="cuda", generator=g)[:tokens].int() + positions = torch.randint( + 0, MAX_POSITIONS, (tokens,), device="cuda", dtype=torch.int32, generator=g + ) + positions[0] = 0 + positions[-1] = MAX_POSITIONS - 1 # the RoPE table's first and last rows + if edge == "rope_edges": + positions[1] = 1 + positions[2] = MAX_POSITIONS - 2 + quant = torch.tensor(KV_QUANT, device="cuda") + return xq.view(tokens, K), xsf, wq.view(N, K), wsf, slots, quant, norms, rotary, positions + + +EDGES = [ + ("zero_rows", 1e-6), + ("denormal_rms", 0.0), + ("saturate", 1e-6), + ("nan_inf", 1e-6), + ("rope_edges", 1e-6), +] + + +@pytest.mark.parametrize("edge, eps", EDGES, ids=[e[0] for e in EDGES]) +@pytest.mark.parametrize("producer", _producers()) +def test_producer_gemv_edge_rows(edge, eps, producer) -> None: + """Zero rows, a denormal RMS argument (eps 0), E4M3 / NVFP4 saturation, NaN / infinity and the RoPE table's edge + rows: the fused producer writes the chain's bytes.""" + case = _edge_case(8, 50 + len(edge), edge) + ref_caches = _caches() + rq, riq = _chain(case, ref_caches, producer, eps) + caches = _caches() + q, iq = _fused(case, caches, eps) + names = ("q", "index_q", "kv_data", "kv_scale", "index_k") + for name, a, b in zip(names, (rq, riq, *ref_caches), (q, iq, *caches)): + a8, b8 = a.contiguous().view(torch.uint8), b.contiguous().view(torch.uint8) + diff = (a8 != b8).nonzero() + assert diff.numel() == 0, ( + f"{edge}: {name}: {diff.shape[0]} bytes differ, first at {tuple(diff[0].tolist())}" + ) + + +def test_producer_gemv_is_deterministic_and_graph_replays() -> None: + """Three eager runs and a captured replay write the same bytes.""" + case = _case(4, 31) + outs = [] + for _ in range(3): + caches = _caches() + q, iq = _fused(case, caches) + outs.append([t.contiguous().view(torch.uint8).clone() for t in (q, iq, *caches)]) + xq, xsf, wq, wsf, slots, quant, norms, rotary, positions = case + q = torch.zeros(4, HEADS_Q, 128, dtype=torch.float8_e4m3fn, device="cuda") + iq = torch.zeros(4, 1, 128, dtype=torch.float8_e4m3fn, device="cuda") + caches = _caches() + stream = torch.cuda.Stream() + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + gemv.m3_mx_gemv_producer( + xq, xsf, wq, wsf, q, iq, *caches, slots, quant, EPS, *norms, rotary, positions, use_pdl=True + ) # fmt: skip + graph.replay() + torch.cuda.synchronize() + outs.append([t.contiguous().view(torch.uint8).clone() for t in (q, iq, *caches)]) + for run in outs[1:]: + for a, b in zip(outs[0], run): + assert torch.equal(a, b) + + +def test_declines_outside_its_shape() -> None: + """More than 8 tokens, or a projection other than the 12-head packed one, launches nothing.""" + case = _case(8, 41) + xq, xsf, wq, wsf, slots, quant, norms, rotary, positions = case + wide = torch.cat([xq, xq], dim=0) + q = torch.zeros(16, HEADS_Q, 128, dtype=torch.float8_e4m3fn, device="cuda") + iq = torch.zeros(16, 1, 128, dtype=torch.float8_e4m3fn, device="cuda") + xsf16 = torch.ops.trtllm.mxfp8_quantize( + torch.zeros(16, K, dtype=torch.bfloat16, device="cuda"), True + )[1] + caches = _caches() + assert not gemv.m3_mx_gemv_producer( + wide, xsf16, wq, wsf, q, iq, *caches, slots, quant, EPS, *norms, rotary, positions + ) # fmt: skip diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mxfp8_gemm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mxfp8_gemm.py new file mode 100644 index 000000000000..183fc17f2d19 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mxfp8_gemm.py @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_gemm entry of the MiniMax-M3 tp8 target (the decode-graph GEMM on a prequantized +activation).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_mxfp8_gemm import ( # noqa: E501 + m3_mxfp8_gemm as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_gemm requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """Same E4M3 operands and power-of-two scales on both sides; accumulation order then one bf16 rounding: at most + 2 units of ULP[bf16] of the row (the low-M GEMM's band), RMS well under one.""" + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + act, act_sf = mxfp8(spec["m"], spec["k"], g) + w, w_sf = mxfp8(spec["n"], spec["k"], g) + with validating(op): + out = op(act, act_sf, w, w_sf, torch.bfloat16, False) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp( + out, + op.reference(act, act_sf, w, w_sf, torch.bfloat16, False), + element_ulp=2.0, + rms_ulp=0.25, + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mxfp8_linear.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mxfp8_linear.py new file mode 100644 index 000000000000..3847e2c8185a --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_mxfp8_linear.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_mxfp8_linear entry of the MiniMax-M3 tp8 target: MXFP8 quantize + GEMM on each backend the +engine selects (native CUTLASS, FlashInfer, the per-bucket tuned quantize + GEMM).""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_mxfp8_linear import ( # noqa: E501 + m3_mxfp8_linear as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_mxfp8_operands import mxfp8 # noqa: E402 +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_mxfp8_linear requires a CUDA device" + +_FLAGS = { + "native": (False, False, False, False), + "flashinfer": (False, False, True, False), + "tuned": (True, False, False, False), +} + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + """The activation's MXFP8 round trip is exact on both sides (same block rule); then the accumulation order and + one bf16 rounding: at most 2 units of ULP[bf16] of the row, RMS well under one.""" + spec = cell.spec + if ( + spec["path"] == "flashinfer" + and getattr(torch.ops.trtllm, "flashinfer_mm_mxfp8", None) is None + ): + pytest.skip("this build has no trtllm::flashinfer_mm_mxfp8") + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + x = torch.randn(spec["m"], spec["k"], generator=g, device="cuda").to(torch.bfloat16) + w, w_sf = mxfp8(spec["n"], spec["k"], g) + alpha = torch.ones(1, dtype=torch.float32, device="cuda") + args = (x, w, w_sf, alpha, torch.bfloat16, *_FLAGS[spec["path"]]) + with validating(op): + out = op(*args) + assert out.shape == (spec["m"], spec["n"]) and out.dtype == torch.bfloat16 + assert_within_ulp(out, op.reference(*args), element_ulp=2.0, rms_ulp=0.25) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_pinned_gemm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_pinned_gemm.py new file mode 100644 index 000000000000..77a87904dc08 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_pinned_gemm.py @@ -0,0 +1,52 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for MX-TAC: the pinned decode-graph MXFP8 decision (CuTe DSL quantize + CuTe DSL GEMM) gives the same bits +as every other candidate stock's AutoTuner can pick (TRT-LLM quantize, CUTLASS GEMM), the same on every call.""" + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.mxfp8_quantize +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.modeling import ( # noqa: E501 + _pinned_mxfp8_gemm, + _pinned_mxfp8_quantize, +) +from tensorrt_llm._torch.custom_ops.torch_custom_ops import FlashInferMXFP8GemmRunner + +assert torch.cuda.is_available(), "MX-TAC requires a CUDA device" + +# The TP8 per-rank projections (ops/gemm/m3_mx_gemv) at decode rows the GEMVs do not serve (9..16) and a few beyond. +CASES = [ + ((1536, 6144), 16), + ((1280, 6144), 12), + ((6144, 1024), 9), + ((768, 6144), 16), + ((6144, 384), 12), + ((3072, 6144), 24), + ((6144, 1536), 32), + ((1536, 6144), 48), +] + + +@pytest.mark.parametrize("nk,m", CASES, ids=[f"{n}x{k}-m{m}" for (n, k), m in CASES]) +@pytest.mark.parametrize("scale", [1e-3, 1.0, 1e3]) +def test_pinned_decision_matches_every_stock_candidate(nk, m, scale) -> None: + n, k = nk + g = torch.Generator(device="cuda").manual_seed(n + k + m) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + x = (torch.randn(m, k, generator=g, device="cuda") * scale).to(torch.bfloat16) + x[0, ::32] *= 1e3 # one outlier per 32-element block on the first row + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + wq = wq.view(n, k) + xq, xsf = _pinned_mxfp8_quantize(x) + tq, tsf = torch.ops.trtllm.mxfp8_quantize(x, True) + assert torch.equal(xq.view(torch.uint8).view(m, k), tq.view(torch.uint8).view(m, k)) + assert torch.equal(xsf.view(torch.uint8).flatten(), tsf.view(torch.uint8).flatten()) + xq = xq.view(m, k) + out = _pinned_mxfp8_gemm(xq, xsf, wq, wsf, torch.bfloat16) + runner = FlashInferMXFP8GemmRunner(torch.bfloat16) + for tactic in (FlashInferMXFP8GemmRunner.CUTE_DSL, FlashInferMXFP8GemmRunner.CUTLASS): + ref = runner.forward([xq, xsf, wq, wsf], tactic=tactic) + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)), tactic + again = _pinned_mxfp8_gemm(xq, xsf, wq, wsf, torch.bfloat16) + assert torch.equal(out.view(torch.int16), again.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_swiglu_oai_bf16x2.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_swiglu_oai_bf16x2.py new file mode 100644 index 000000000000..8db99da569da --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_modeling_v2_m3_swiglu_oai_bf16x2.py @@ -0,0 +1,149 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Bitwise oracle of the mx_gemv swigluoai prologue arithmetic (kernels/mx_gemv/_swiglu_oai_bf16x2) against the +Triton silu_and_mul kernel the dense MLP runs today (modules/swiglu.py, alpha 1.702, limit 7, beta 1): every bf16 gate +bit pattern against special and random up values, and every up bit pattern against special and random gates.""" + +import cutlass +import cutlass.cute as cute +import pytest +import torch +from cutlass.cute.runtime import from_dlpack + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.mx_gemv.m3_mx_gemv_kernel import ( # noqa: E501 + SWIGLU_OAI_ALPHA, + SWIGLU_OAI_LIMIT, + _swiglu_oai_bf16x2, +) + +assert torch.cuda.is_available(), "the swigluoai prologue oracle requires a CUDA device" + +THREADS = 256 + + +@cute.kernel +def _swiglu_words_kernel(gate: cute.Tensor, up: cute.Tensor, out: cute.Tensor, n: cutlass.Int32): + tidx, _, _ = cute.arch.thread_idx() + bidx, _, _ = cute.arch.block_idx() + i = bidx * cutlass.Int32(THREADS) + tidx + if i < n: + out[i] = _swiglu_oai_bf16x2(gate[i], up[i]) + + +@cute.jit +def _swiglu_words(gate: cute.Tensor, up: cute.Tensor, out: cute.Tensor, n: cutlass.Int32): + _swiglu_words_kernel(gate, up, out, n).launch( + grid=((n + THREADS - 1) // THREADS, 1, 1), block=(THREADS, 1, 1) + ) + + +_compiled = {} + + +def _prologue(gate_bf16: torch.Tensor, up_bf16: torch.Tensor) -> torch.Tensor: + """The prologue helper over flat bf16 tensors of even length (pairs packed into int32 words).""" + g = gate_bf16.contiguous().view(torch.int32) + u = up_bf16.contiguous().view(torch.int32) + out = torch.empty_like(g) + args = [from_dlpack(t, assumed_align=4) for t in (g, u, out)] + fn = _compiled.get("f") + if fn is None: + fn = _compiled["f"] = cute.compile(_swiglu_words, *args, cutlass.Int32(g.numel())) + fn(*args, cutlass.Int32(g.numel())) + return out.view(torch.bfloat16) + + +def _triton(gate_bf16: torch.Tensor, up_bf16: torch.Tensor) -> torch.Tensor: + """The dense MLP's activation today: torch.ops.trtllm.silu_and_mul on one [1, 2d] row per call site layout.""" + x = torch.cat([gate_bf16.view(1, -1), up_bf16.view(1, -1)], dim=1) + return torch.ops.trtllm.silu_and_mul( + x, swiglu_limit=SWIGLU_OAI_LIMIT, swiglu_alpha=SWIGLU_OAI_ALPHA, swiglu_beta=1.0 + ).view(-1) + + +def _all_bf16() -> torch.Tensor: + return ( + torch.arange(1 << 16, dtype=torch.int32, device="cuda").to(torch.int16).view(torch.bfloat16) + ) + + +def _specials(n: int, seed: int) -> list: + g = torch.Generator(device="cuda").manual_seed(seed) + vals = [0.0, -0.0, 1.0, -1.0, 6.99, 7.0, 7.01, -7.0, -7.01, 1e-38, -1e-38, 1e-40, 88.0, -88.0, 3e38, -3e38, + float("inf"), float("-inf"), float("nan")] # fmt: skip + rows = [torch.full((n,), v, dtype=torch.bfloat16, device="cuda") for v in vals] + for scale in (0.05, 1.0, 4.0, 30.0): + rows.append((scale * torch.randn(n, generator=g, device="cuda")).to(torch.bfloat16)) + rows.append( + torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + .to(torch.int16) + .view(torch.bfloat16) + ) + return rows + + +def _assert_bitwise(got: torch.Tensor, ref: torch.Tensor, what: str) -> None: + bad = (got.view(torch.int16) != ref.view(torch.int16)).nonzero().flatten() + assert bad.numel() == 0, f"{what}: {bad.numel()} mismatches, first at {bad[:4].tolist()}" + + +def test_every_gate_pattern_is_bitwise_the_triton_kernel() -> None: + gate = _all_bf16() + for j, up in enumerate(_specials(gate.numel(), 1)): + _assert_bitwise(_prologue(gate, up), _triton(gate, up), f"up set {j}") + + +def test_every_up_pattern_is_bitwise_the_triton_kernel() -> None: + up = _all_bf16() + for j, gate in enumerate(_specials(up.numel(), 2)): + _assert_bitwise(_prologue(gate, up), _triton(gate, up), f"gate set {j}") + + +@pytest.mark.parametrize("seed", range(4)) +def test_random_pairs_are_bitwise_the_triton_kernel(seed: int) -> None: + g = torch.Generator(device="cuda").manual_seed(100 + seed) + n = 1 << 22 + gate = ( + torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + .to(torch.int16) + .view(torch.bfloat16) + ) + up = ( + torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda") + .to(torch.int16) + .view(torch.bfloat16) + ) + _assert_bitwise(_prologue(gate, up), _triton(gate, up), f"seed {seed}") + + +# The GEMV with the activation in its prologue against today's two launches (Triton silu_and_mul, then the bf16 GEMV): +# bit for bit on the dense MLP's down projection at every decode row count. +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E402 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.mx_gemv import ( # noqa: E402, E501 + op as gemv, +) + + +@pytest.mark.parametrize("m", range(1, modeling.DECODE_MAX_TOKENS + 1)) +def test_down_gemv_with_the_activation_prologue_is_bitwise_two_launches(m: int) -> None: + n, k = modeling.DENSE_DOWN + g = torch.Generator(device="cuda").manual_seed(1000 + m) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + wq, wsf = torch.ops.trtllm.mxfp8_quantize(w, True) + wq = wq.view(n, k) + # A gate_up GEMV output's range, plus values past the clamps. + gate_up = (3.0 * torch.randn(m, 2 * k, generator=g, device="cuda")).to(torch.bfloat16) + gate_up[:, :8] = torch.tensor( + [8.0, -8.0, 7.0, -7.0, 0.0, -0.0, 50.0, -50.0], dtype=torch.bfloat16 + ) + act = torch.ops.trtllm.silu_and_mul( + gate_up, swiglu_limit=SWIGLU_OAI_LIMIT, swiglu_alpha=SWIGLU_OAI_ALPHA, swiglu_beta=1.0 + ) + ref = gemv.m3_mx_gemv_bf16(act.contiguous(), wq, wsf, use_pdl=True) + out = gemv.m3_mx_gemv_swiglu(gate_up, wq, wsf, use_pdl=True) + assert ref is not None and out is not None, ( + "the dense down projection must be served at every decode row count" + ) + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_mxdraft_gu_swiglu.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_mxdraft_gu_swiglu.py new file mode 100644 index 000000000000..3ca98b1f1656 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_mxdraft_gu_swiglu.py @@ -0,0 +1,123 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""DRAFT_GU_SWIGLU_EPI (F5): the drafter MLP's gate_up GEMV with the swigluoai in its epilogue is bit for bit today's +two launches, m3_mx_gemv_bf16(x, gate_up) followed by the Triton silu_and_mul (limit 7, alpha 1.702, beta 1).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( + mxdraft, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.mx_gemv import ( # noqa: E501 + op as gemv, +) +from tensorrt_llm._torch.modules.swiglu import swiglu + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="sm_107 kernel (certified on sm_107 only)", +) + +F, K = 1536, 6144 # the drafter MLP's per-rank gate_up at TP8: [2 F, K] = [3072, 6144] +LIMIT, ALPHA, BETA = 7.0, 1.702, 1.0 + + +def _mxfp8_exact_bf16(n: int, k: int, scale_lo: int, scale_hi: int, seed: int) -> torch.Tensor: + """A bf16 weight that is exactly an MXFP8 dequantization (E4M3 codes x 2^e per 32 columns), as the checkpoint's.""" + g = torch.Generator(device="cuda").manual_seed(seed) + codes = ( + (torch.randn(n, k, device="cuda", generator=g) * 64) + .clamp(-448, 448) + .to(torch.float8_e4m3fn) + ) + exp = torch.randint(scale_lo, scale_hi + 1, (n, k // 32), device="cuda", generator=g) + return (codes.float() * torch.exp2(exp.float()).repeat_interleave(32, dim=1)).to(torch.bfloat16) + + +def _forms(w_bf16: torch.Tensor): + plain = mxdraft.mxfp8_from_bf16(w_bf16) + inter = mxdraft.mxfp8_from_bf16(gemv.gu_interleave(w_bf16)) + assert plain is not None and inter is not None, "weight is not an exact MXFP8 dequantization" + return ( + (plain[0], mxdraft.swizzle_sf(plain[1])), + (inter[0], mxdraft.swizzle_sf(inter[1])), + ) + + +def _two_launch(x, w8, sf): + y = gemv.m3_mx_gemv_bf16(x, w8, sf) + assert y is not None + return swiglu(y, swiglu_limit=LIMIT, swiglu_alpha=ALPHA, swiglu_beta=BETA) + + +def test_interleave_is_a_row_permutation(): + # float32: row ids up to 3071 are exact (bf16 holds integers exactly only to 256); gu_interleave is dtype-agnostic. + w = ( + torch.arange(2 * F, device="cuda", dtype=torch.float32) + .view(-1, 1) + .expand(-1, 32) + .contiguous() + ) + wi = gemv.gu_interleave(w) + rows = wi[:, 0].float().long() + tile = torch.arange(2 * F, device="cuda") // 128 + pos = torch.arange(2 * F, device="cuda") % 128 + expect = torch.where(pos < 64, tile * 64 + pos, F + tile * 64 + pos - 64) + assert torch.equal(rows, expect) + + +@pytest.mark.parametrize("m", [1, 2, 3, 4, 5, 6, 7, 8]) +@pytest.mark.parametrize("case", ["natural", "clamps"]) +def test_epilogue_bitwise_vs_two_launches(m, case): + # natural: small outputs (no clamp). clamps: scales large enough that many gate > 7 and |up| > 7 (both clamps). + lo, hi = (-14, -10) if case == "natural" else (-6, -2) + w = _mxfp8_exact_bf16(2 * F, K, lo, hi, seed=1000 + m) + (w8, sf), (w8i, sfi) = _forms(w) + g = torch.Generator(device="cuda").manual_seed(m) + x = (torch.randn(m, K, device="cuda", generator=g) * (1.0 if case == "natural" else 4.0)).to( + torch.bfloat16 + ) + ref = _two_launch(x, w8, sf) + out = gemv.m3_mx_gemv_gu_swiglu(x, w8i, sfi) + assert out is not None and out.shape == (m, F) and out.dtype == torch.bfloat16 + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)), ( + f"m={m} {case}: {int((out.view(torch.int16) != ref.view(torch.int16)).sum())} differing elements" + ) + if case == "clamps": + y = gemv.m3_mx_gemv_bf16(x, w8, sf).float() + assert int((y[:, :F] > LIMIT).sum()) > 0 and int((y[:, F:].abs() > LIMIT).sum()) > 0, ( + "clamps not exercised" + ) + + +def test_cuda_graph_replay_matches_eager(): + m = 8 + w = _mxfp8_exact_bf16(2 * F, K, -12, -8, seed=7) + (_, _), (w8i, sfi) = _forms(w) + x = torch.randn(m, K, device="cuda").to(torch.bfloat16) + eager = gemv.m3_mx_gemv_gu_swiglu(x, w8i, sfi).clone() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = gemv.m3_mx_gemv_gu_swiglu(x, w8i, sfi) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(captured.view(torch.int16), eager.view(torch.int16)) + + +def test_declines(): + w = _mxfp8_exact_bf16(2 * F, K, -12, -8, seed=3) + (_, _), (w8i, sfi) = _forms(w) + # more rows than the epilogue's 128 owner threads take at split 4 (16 pairs x 8 tokens) + x = torch.randn(9, K, device="cuda").to(torch.bfloat16) + assert gemv.m3_mx_gemv_gu_swiglu(x, w8i, sfi) is None + # a non-bf16 activation + assert gemv.m3_mx_gemv_gu_swiglu(x[:8].float(), w8i, sfi) is None + + +def test_kill_switch_source(): + import inspect + + src = inspect.getsource(mxdraft.route_gu_swiglu) + assert "TRTLLM_M31_DISABLE_DRAFT_GU_SWIGLU_EPI" in src + assert mxdraft.DRAFT_GU_SWIGLU_EPI_PATH == "DRAFT_GU_SWIGLU_EPI" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_r1b1_gemv_order_gate_a.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_r1b1_gemv_order_gate_a.py new file mode 100644 index 000000000000..15c2f7515e31 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/gemm/test_r1b1_gemv_order_gate_a.py @@ -0,0 +1,85 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Rule 1b gate a for the TP8 decode GEMV order override (kernels/mx_gemv/op.STRIDED_K_TILES: strided k-tiles): +for every overridden (projection, rows) the output is the same on 3 eager calls and on a CUDA graph replay +(deterministic), and its max abs / max rel error vs the fp64 product of the dequantized operands stays within the +shipped order's band (the shared table's config on the same operands, over several seeds).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.mx_gemv import ( # noqa: E501 + op, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_mx_gemv import ( # noqa: E501 + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + m3_mx_gemv, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + mxfp8_dequant, +) + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + +CASES = [ + (nk, m) for nk in (QKV_SPARSE, QKV, O_PROJ, DENSE_GATE_UP, DENSE_DOWN) for m in (1, 2, 4, 8) +] + + +def _err(y, ref): + d = (y.double() - ref).abs() + return float(d.max()), float((d / ref.abs().amax(dim=1, keepdim=True).clamp_min(1e-30)).max()) + + +@pytest.mark.parametrize("nk, m", CASES, ids=[f"{n}x{k}-m{m}" for (n, k), m in CASES]) +def test_gate_a(nk, m) -> None: + n, k = nk + cfg = op.get_config(n, k, m) + assert cfg is not None and len(cfg) > 2 and cfg[2] == 0, ( + "the override is off or does not cover this call" + ) + shipped = cfg[:2] # the contiguous order + worst_new, worst_ship = [0.0, 0.0], [0.0, 0.0] + for seed in range(4): + g = torch.Generator(device="cuda").manual_seed(1000 * seed + m) + w = (torch.randn(n, k, device="cuda", generator=g) * 2).to(torch.float8_e4m3fn) + ws = torch.randint(118, 124, (n * k // 32,), dtype=torch.uint8, device="cuda", generator=g) + x = (torch.randn(m, k, device="cuda", generator=g) * (0.5 + seed)).to(torch.bfloat16) + q, s = torch.ops.trtllm.mxfp8_quantize(x, True) + ref = ( + mxfp8_dequant(q.view(torch.uint8), s).double() + @ mxfp8_dequant(w.view(torch.uint8), ws).double().t() + ) + ys = [m3_mx_gemv(q, s, w, ws, True, cfg, True, None) for _ in range(3)] + assert all(y is not None for y in ys), "the GEMV declined the override" + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + out = [] + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out.append(m3_mx_gemv(q, s, w, ws, True, cfg, True, None)) + graph.replay() + torch.cuda.synchronize() + for y in ys[1:] + [out[0]]: + assert torch.equal(ys[0].view(torch.int16), y.view(torch.int16)), ( + "not deterministic (eager / graph)" + ) + ship = m3_mx_gemv(q, s, w, ws, True, shipped, True, None) + torch.cuda.synchronize() + for acc, y in ((worst_new, ys[0]), (worst_ship, ship)): + e = _err(y, ref) + acc[0], acc[1] = max(acc[0], e[0]), max(acc[1], e[1]) + print( + f"\nR1B1 tp8 {n}x{k} m={m} cfg {cfg}: max abs / rel vs fp64 {worst_new[0]:.3e} / {worst_new[1]:.3e} | shipped " + f"order {worst_ship[0]:.3e} / {worst_ship[1]:.3e}", + flush=True, + ) + # The band: the shipped order's own error (bf16 output rounding dominates), with 25 % headroom. + assert worst_new[0] <= 1.25 * worst_ship[0] and worst_new[1] <= 1.25 * worst_ship[1] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/_m3_moe_operands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/_m3_moe_operands.py new file mode 100644 index 000000000000..d737e414f318 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/_m3_moe_operands.py @@ -0,0 +1,172 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One tp8 rank's MiniMax-M3 MoE layer operands for the decode-engine entries' tests. + +The routed experts are random checkpoint-format NVFP4 tensors ([w3 (up); w1 (gate)] rows, e4m3 block scales) put +through TRT-LLM's own TRTLLM-Gen NVFP4 loader shuffle (``NVFP4TRTLLMGenFusedMoEMethod._shuffle_all_experts``), which +writes the buffers the engines read in place. The per-expert scalars follow the loader's conventions (act_beta and +act_clamp divided by fc31_alpha, fc31_scale_c = fc2_input_scale x fc31_alpha) and are sized so the FC1 sums land +around the SwiGLU-OAI clamp and the outputs near unit scale. The shared expert's MXFP8 weights come from +mxfp8_quantize with the swizzled scale layout, as the MXFP8 linear method stores them. +""" + +import functools +from types import SimpleNamespace + +import torch + +# Hidden size, the routed experts' intermediate per rank at TP8 (1536 / 8 x 2: every expert on every rank), experts, +# top-k, and the shared expert's intermediate per rank. +H, I_TP, NUM_EXPERTS, TOP_K = 6144, 384, 128, 4 +SHARED_I = 384 +SWIGLU_ALPHA, SWIGLU_LIMIT = 1.702, 7.0 +ROUTED_SCALING = 2.0 +MINIMAX2_ROUTING = 5 # RoutingMethodType.MiniMax2 +# NVFP4 global scales: 448 x 6 over the amax each side is sized for. +FC31_INPUT_SCALE = 448.0 * 6.0 / 8.0 +FC2_INPUT_SCALE = 448.0 * 6.0 / 24.0 + + +def _rand_e4m3_scales(shape, gen) -> torch.Tensor: + """E4M3 block scales in [0.5, 2): biased exponents 6 and 7, any mantissa.""" + return torch.randint(0x30, 0x40, shape, dtype=torch.uint8, device="cuda", generator=gen).view( + torch.float8_e4m3fn + ) + + +@functools.lru_cache(maxsize=None) +def experts(seed: int = 20261013): + """The routed experts and the router of one MoE layer, in the layout and conventions of the TRTLLM-Gen loader. + + Returns a SimpleNamespace: ``tensors`` (the ten expert operands in ``M3MoeState.layer``'s order), ``bias``, + ``router_weight``, ``fc31_input_scale`` and ``shared`` (gate_up, its scales, down, its scales). + """ + from tensorrt_llm._torch.moe.fused_moe.quantization import NVFP4TRTLLMGenFusedMoEMethod + + gen = torch.Generator(device="cuda").manual_seed(seed) + e = NUM_EXPERTS + w31 = torch.randint( + 0, 256, (e, 2 * I_TP, H // 2), dtype=torch.uint8, device="cuda", generator=gen + ) + w2 = torch.randint(0, 256, (e, H, I_TP // 2), dtype=torch.uint8, device="cuda", generator=gen) + w31s = _rand_e4m3_scales((e, 2 * I_TP, H // 16), gen) + w2s = _rand_e4m3_scales((e, H, I_TP // 16), gen) + module = SimpleNamespace( + w3_w1_weight=w31, + w2_weight=w2, + w3_w1_weight_scale=w31s, + w2_weight_scale=w2s, + is_gated_activation=True, + bias=False, + ) + NVFP4TRTLLMGenFusedMoEMethod()._shuffle_all_experts(module) + torch.cuda.synchronize() + + # Raw FC1 sums: rms(x) x FC31_INPUT_SCALE x rms(E2M1 value) x rms(block scale) x sqrt(H) ~ 1.0e5 at rms(x) = 1; + # fc31_alpha brings them to ~1.5 in the dequantized domain, the FC2 alpha the outputs to ~1. + base = 1.5 / (FC31_INPUT_SCALE * 2.93 * 1.3 * H**0.5) + fc31_alpha = base * (0.8 + 0.4 * torch.rand(e, device="cuda", generator=gen)) + fc2_alpha = (0.8 + 0.4 * torch.rand(e, device="cuda", generator=gen)) / ( + FC2_INPUT_SCALE * 2.93 * 1.3 * I_TP**0.5 + ) + tensors = ( + w31, + w31s, + w2, + w2s, + fc31_alpha, + FC2_INPUT_SCALE * fc31_alpha, # fc31_scale_c + torch.full((e,), SWIGLU_ALPHA, device="cuda"), # act_alpha + 1.0 / fc31_alpha, # act_beta: the (up + 1) offset over fc31_alpha + SWIGLU_LIMIT / fc31_alpha, # act_clamp over fc31_alpha + fc2_alpha, + ) + bias = (0.02 * torch.randn(e, device="cuda", generator=gen)).float() + router_weight = (torch.randn(e, H, device="cuda", generator=gen) / H**0.5).float().contiguous() + fc31_input_scale = torch.tensor([FC31_INPUT_SCALE], dtype=torch.float32, device="cuda") + + gu = (0.02 * torch.randn(2 * SHARED_I, H, device="cuda", generator=gen)).to(torch.bfloat16) + dn = (0.05 * torch.randn(H, SHARED_I, device="cuda", generator=gen)).to(torch.bfloat16) + gu_q, gu_sf = torch.ops.trtllm.mxfp8_quantize(gu, True) + dn_q, dn_sf = torch.ops.trtllm.mxfp8_quantize(dn, True) + shared = (gu_q.view(2 * SHARED_I, H), gu_sf, dn_q.view(H, SHARED_I), dn_sf) + return SimpleNamespace( + tensors=tensors, + bias=bias, + router_weight=router_weight, + fc31_input_scale=fc31_input_scale, + shared=shared, + ) + + +# Routing cases: random (up to 4 M distinct experts), and every token on the same four experts (the bias outweighs +# any sigmoid difference), which leaves later slot groups empty. +CONCENTRATED = (5, 40, 77, 126) + + +def layer_for(state, case: str, shared: bool = True): + """One layer on ``state`` over :func:`experts` for a routing case. ``shared=False``: the shared expert's weights + are zero, so the engine returns the routed part alone (bf16(0 + bf16(routed))).""" + ops = experts() + bias = ops.bias.clone() + if case == "concentrated": + bias[list(CONCENTRATED)] += 1.0 + shared_weights = ops.shared + if not shared: + gu, gu_sf, dn, dn_sf = ops.shared + shared_weights = (torch.zeros_like(gu), gu_sf, torch.zeros_like(dn), dn_sf) + return state.layer( + *ops.tensors, + bias, + router_weight=ops.router_weight, + fc31_input_scale=ops.fc31_input_scale, + shared_weights=shared_weights, + ) + + +def stock_routed( + hidden: torch.Tensor, topk_ids: torch.Tensor, topk_weights: torch.Tensor +) -> torch.Tensor: + """The stock TRTLLM-Gen NVFP4 MoE (fp4_quantize, then fp4_block_scale_moe_runner with SwiGLU-OAI) on the given + routing, over :func:`experts`: what the built-in MiniMaxM3MoE computes for the routed part.""" + import tensorrt_llm._torch.custom_ops # noqa: F401 -- registers the stock ops + + ops = experts() + w31, w31s, w2, w2s, fc31_alpha, scale_c, act_alpha, act_beta, act_clamp, fc2_alpha = ops.tensors + xq, xsf = torch.ops.trtllm.fp4_quantize(hidden, ops.fc31_input_scale, 16, False, False) + out = torch.ops.trtllm.fp4_block_scale_moe_runner( + None, None, xq, xsf.view(torch.float8_e4m3fn).flatten(), + w31, w31s, None, act_alpha, act_beta, act_clamp, w2, w2s, None, + scale_c, fc31_alpha, fc2_alpha, + NUM_EXPERTS, TOP_K, None, None, I_TP, 0, NUM_EXPERTS, ROUTED_SCALING, MINIMAX2_ROUTING, True, 0, + topk_weights=topk_weights.to(torch.bfloat16), topk_ids=topk_ids.to(torch.int32), + ) # fmt: skip + return out[0] if isinstance(out, (list, tuple)) else out + + +def hidden_states(m: int, seed: int) -> torch.Tensor: + gen = torch.Generator(device="cuda").manual_seed(seed) + return torch.randn(m, H, device="cuda", generator=gen).to(torch.bfloat16) + + +def check_routed_part(op, state, m: int, seed: int) -> None: + """Exact, independent of the reference: the engine's routing (its route_dbg record: ids, then the bf16 weights' + fp32 bits) is the reference routing bit for bit, and its routed part is the stock TRTLLM-Gen MoE's output on that + routing bit for bit -- the claim the engines are built on.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + moe_route, + ) + + layer = layer_for(state, "random", shared=False) + hidden = hidden_states(m, seed=seed) + out = op(hidden, layer) + ids = state.route_dbg[:, :4].long() + weights = state.route_dbg[:, 4:].contiguous().view(torch.float32) + ref_ids, ref_weights = moe_route(hidden.float(), layer.router_weight, layer.bias) + assert torch.equal(ids, ref_ids) and torch.equal(weights, ref_weights), ( + "routing differs from the reference" + ) + stock = stock_routed(hidden, ids, weights) + assert torch.equal(out.view(torch.int16), stock.view(torch.int16)), ( + "routed part differs from TRTLLM-Gen's" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/_m3_router.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/_m3_router.py new file mode 100644 index 000000000000..a7df7ed8bd80 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/_m3_router.py @@ -0,0 +1,79 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Shared operands and gates of the router entry's tests (m3_router_tc).""" + +import zlib + +import torch + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp # noqa: E402 — needs the path declared above + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_router_tc import ( # noqa: E501 + fp32_logits, + near_tie, + nvfp4_quantize, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_router_tc import ( # noqa: E501 + m3_router_tc as tc, +) + +__all__ = [ + "SF_SCALE", + "_build", + "_compare_fp4", + "_compare_logits", + "_seed", + "fp32_logits", + "tc", +] + +#: The experts' NVFP4 input scale the tc cells use: 448 * 6 / 8, block scales well inside E4M3. +SF_SCALE = 336.0 + + +def _build(m: int, e: int, k: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + x = (torch.randn(m, k, generator=g, device="cuda") * 2.0).to(torch.bfloat16) + w = torch.randn(e, k, generator=g, device="cuda") * 0.02 + return x, w + + +def _seed(cell) -> int: + return zlib.crc32(cell.why.encode()) % 2**31 + + +def _compare_logits(out: torch.Tensor, ref: torch.Tensor) -> None: + """fp32 logits that differ from a true fp32 product only in summation order. + + Over K = 6144 products a reordered fp32 sum moves by a few ulps of the sum of magnitudes, which near + cancellation is ~sqrt(K) times the row's largest logit's ulp at most. Gated in fp32 ulps of each row's largest + logit: 64 element-wise, 8 in relative RMS. Measured on GB300: GEMV 1.6 / 1.4 ulp, tensor-core router 3.8 / 2.0 + ulp at 2..48 tokens; the tensor-core logits against a true fp32 product (the TF32 rounding the band must see) + sit at 2200+ / 1700 ulp. + """ + assert out.dtype == ref.dtype == torch.float32 and out.shape == ref.shape + assert_within_ulp(out, ref, element_ulp=64.0, rms_ulp=8.0) + + +def _codes(packed: torch.Tensor) -> torch.Tensor: + """[M, K / 2] packed e2m1 bytes -> [M, K] 4-bit codes, low nibble first.""" + lo, hi = packed & 0xF, packed >> 4 + return torch.stack((lo, hi), dim=-1).reshape(packed.shape[0], -1) + + +def _compare_fp4( + q: torch.Tensor, sf: torch.Tensor, x: torch.Tensor, sf_scale: torch.Tensor +) -> None: + """Scales bit-exact; codes bit-exact except within ``TIE_WINDOW`` of an e2m1 midpoint, where the kernel's + approximate reciprocals choose the side (and then only by one code).""" + ref_q, ref_sf, scaled = nvfp4_quantize(x, sf_scale) + assert q.dtype == torch.uint8 and q.shape == ref_q.shape + assert torch.equal(sf, ref_sf), f"{(sf != ref_sf).sum().item()} block scales differ" + got, want = _codes(q), _codes(ref_q) + differ = got != want + assert not bool((differ & ~near_tie(scaled)).any()), ( + f"{(differ & ~near_tie(scaled)).sum().item()} e2m1 codes differ away from a rounding tie" + ) + magnitude = ((got & 7).int() - (want & 7).int()).abs() + assert bool((magnitude[differ] <= 1).all()), "a tie resolved more than one code away" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_early_epoch.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_early_epoch.py new file mode 100644 index 000000000000..55d78ada347f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_early_epoch.py @@ -0,0 +1,59 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""EARLY_EPOCH: the sm_107 TP8 MoE engine stores its per-CTA call epoch early whenever it releases its PDL +dependents early (EARLY_AR, the default): the next call on the same state reads the epoch right after its own grid +wait, which no longer implies this call finished. Without early release the epoch is stored at teardown, as before. +Outputs are unchanged and repeated calls (epochs advancing) stay bit-identical.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_moe_operands import hidden_states, layer_for # noqa: E402 + + +def _state(num_tokens: int, early_release: bool): + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + return moe_op.M3MoeState( + dev, + num_tokens=num_tokens, + num_ctas=sms - moe_op.shared_ctas_for(num_tokens, True), + shared=True, + early_release=early_release, + ) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("early_release", [True, False]) +def test_early_epoch_follows_the_release_point(early_release: bool) -> None: + state = _state(8, early_release) + early = bool(state.mod.EARLY_AR) or bool( + getattr(state.mod, "TAIL_TRIGGER", False) + ) # any early release point + assert bool(state.mod.EARLY_EPOCH) == early + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("num_tokens", [1, 8]) +def test_epochs_advance_once_per_call_and_outputs_repeat(num_tokens: int) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + m3_moe_engine, + ) + + state = _state(num_tokens, True) + layer = layer_for(state, "random") + hidden = hidden_states(num_tokens, seed=3 + num_tokens) + outs = [] + for _ in range(3): + before = state.epochs.clone() + outs.append(m3_moe_engine(hidden, layer).clone()) + torch.cuda.synchronize() + assert torch.equal(state.epochs, before + 1), ( + "every CTA's epoch advances exactly once per call" + ) + assert all(torch.equal(o.view(torch.int16), outs[0].view(torch.int16)) for o in outs) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_halfplan.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_halfplan.py new file mode 100644 index 000000000000..f949e0497ad4 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_halfplan.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sm_107 TP8 MoE phase plan's half-unit FC1 ranges (MOE_HALFPLAN): with equal per-CTA shares (SH20's 192 routed +CTAs at 8 tokens) a split unit is halved instead of extended to its end, so the partial last FC1 group puts at most +half a unit on any CTA; unequal shares keep the extended ranges; TRTLLM_M31_DISABLE_MOE_HALFPLAN restores them.""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) + +M_MAX = 8 + + +def _ranges(plan, g: int, num_ctas: int): + """Per group p of G = g: the CTAs' [i0, i1) FC1 item ranges.""" + out = [] + for p in range((g + 3) // 4): + words = [int(plan[g, c, p]) for c in range(num_ctas)] + out.append([(w & 0xFFFF, w >> 16) if w else (0, 0) for w in words]) + return out + + +def _max_items(plan, g: int, num_ctas: int) -> int: + return max(sum(i1 - i0 for i0, i1 in col) for col in zip(*_ranges(plan, g, num_ctas))) + + +@pytest.mark.parametrize("killed", [False, True]) +def test_halfplan_last_group(monkeypatch: pytest.MonkeyPatch, killed: bool) -> None: + monkeypatch.setattr(moe_op, "HALFPLAN", not killed) + plan, _ = moe_op.make_phase_plan(192, M_MAX) + # base: the partial last group's 1 + 5 splits; half units: 3 + 3 + expected = {29: 26, 30: 25, 31: 26} if killed else {29: 24, 30: 24, 31: 24} + for g, most in expected.items(): + assert _max_items(plan, g, 192) == most + for g in (28, 32): # full groups only: 3 items per CTA either way + assert _max_items(plan, g, 192) == (21 if g == 28 else 24) + + +@pytest.mark.parametrize("num_ctas", [192, 188]) +def test_halfplan_ranges_valid(num_ctas: int) -> None: + plan, _ = moe_op.make_phase_plan(num_ctas, M_MAX) + s1 = moe_op._S1 + for g in range(moe_op.TOP_K, moe_op.TOP_K * M_MAX + 1): + for p, ranges in enumerate(_ranges(plan, g, num_ctas)): + n_p = min(4, g - 4 * p) + total = n_p * moe_op.U1 * s1 + pos = 0 + for i0, i1 in ranges: # contiguous in CTA order, covering the group + if i1 > i0: + assert i0 == pos + pos = i1 + assert pos == total + # every unit has at most one CTA boundary strictly inside it, and a range that starts inside a unit + # runs to (at least) its end + inner = [i0 for i0, i1 in ranges if i1 > i0 and i0 % s1] + assert len({i0 // s1 for i0 in inner}) == len(inner) + assert all( + i1 >= (i0 // s1 + 1) * s1 or i1 == total for i0, i1 in ranges if i1 > i0 and i0 % s1 + ) + + +def test_halfplan_unequal_shares_unchanged(monkeypatch: pytest.MonkeyPatch) -> None: + # 188 routed CTAs (SH20 off): 4 CTAs carry two FC2 blocks, the shares differ, the plan is the base plan + on, _ = moe_op.make_phase_plan(188, M_MAX) + monkeypatch.setattr(moe_op, "HALFPLAN", False) + off, _ = moe_op.make_phase_plan(188, M_MAX) + assert bool((on == off).all()) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_m8cfg.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_m8cfg.py new file mode 100644 index 000000000000..57205f383c59 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_m8cfg.py @@ -0,0 +1,27 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sm_107 TP8 MoE engine's 8-token override (MOE_M8CFG): the tagged-record FC1 -> FC2 hand-off instead of the B +image, ported from the GB300 target (measured faster there with bit-identical outputs); other token counts keep the +tables; TRTLLM_M31_DISABLE_MOE_M8CFG restores the tables.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) + + +def test_m8cfg_scope() -> None: + assert moe_op.M8_CFG_BY_M == {8: {"b2img": 0}} + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("killed", [False, True]) +def test_m8cfg_config_and_kill_switch(monkeypatch: pytest.MonkeyPatch, killed: bool) -> None: + monkeypatch.setattr(moe_op, "m31_disabled", lambda item_id: killed and item_id == "MOE_M8CFG") + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + state = moe_op.M3MoeState(dev, num_tokens=8, num_ctas=sms - 24, shared=True) + assert state.config["b2img"] == (moe_op.B2IMG_BY_M[7] if killed else 0) + assert bool(state.mod.B2IMG) == bool(killed and moe_op.B2IMG_BY_M[7]) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_out_skip.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_out_skip.py new file mode 100644 index 000000000000..157b09489d1c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_out_skip.py @@ -0,0 +1,96 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MOE-FINALIZE-DEADSTORE (OUT_SKIP, default on) on the sm_107 tp8 MoE engine: a call that pushes +into the boundary all-reduce (arp enabled) does not store its bf16 rows into ``out`` (the pushed all-reduce never +reads them); a call without a push stores as before; TRTLLM_M31_DISABLE_MOE_OUT_SKIP=1 keeps the store. The +pushed words are the same bits either way. The push target is a plain device buffer laid out as set A of the +multicast exchange (world 1, rank 0, parity 0): word t * H / 2 + c / 2 holds channels c, c + 1 of row t as bf16x2 +(-0 pushed as +0).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_moe_operands import hidden_states, layer_for # noqa: E402 + +H = 6144 +EMPTY = -(2**31) # 0x80000000 +SENTINEL = 1234.0 + + +def _check_push(engine, state, num_tokens: int, bucket: int, killed: bool, counter=None) -> None: + dev = torch.device("cuda") + layer = layer_for(state, "random") + hidden = hidden_states(num_tokens, seed=11 + num_tokens) + ref = engine(hidden, layer).clone() # no push: stores out (either config) + torch.cuda.synchronize() + buf = torch.full( + (2 * 2 * bucket * H // 2,), EMPTY, dtype=torch.int32, device=dev + ) # sets A and B + flags = torch.zeros( + bucket * 8, dtype=torch.int32, device=dev + ) # parity 0 for every (row, AR CTA) + arp = moe_op.ar_push_desc(buf.data_ptr(), flags.data_ptr(), 0, 1, bucket, dev) + out = torch.full((num_tokens, H), SENTINEL, dtype=torch.bfloat16, device=dev) + counter = moe_op.OUT_SKIP_TAKEN if counter is None else counter + before = counter[0] + engine(hidden, layer, out, arp) + torch.cuda.synchronize() + assert counter[0] - before == int(not killed), ( + "the OUT_SKIP work counter counts pushing calls that skip" + ) + pushed = buf[: num_tokens * H // 2].view(torch.bfloat16).view(num_tokens, H) + ref_pos0 = torch.where(ref == 0, torch.zeros_like(ref), ref) # the push writes -0 as +0 + assert torch.equal(pushed.view(torch.int16), ref_pos0.view(torch.int16)), "pushed words changed" + assert bool((buf[num_tokens * H // 2 : bucket * H // 2] == EMPTY).all()), ( + "a push beyond the rows" + ) + if killed: + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)), ( + "the kill switch keeps the store" + ) + else: + assert bool((out == SENTINEL).all()), "a pushing call still stored out" + out2 = torch.full((num_tokens, H), SENTINEL, dtype=torch.bfloat16, device=dev) + engine(hidden, layer, out2, None) # a call without a push stores as before + torch.cuda.synchronize() + assert torch.equal(out2.view(torch.int16), ref.view(torch.int16)) + + +def _state(monkeypatch, num_tokens: int, killed: bool): + monkeypatch.setattr( + moe_op, "m31_disabled", lambda item_id, k=killed: k and item_id == "MOE_OUT_SKIP" + ) + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + return moe_op.M3MoeState( + dev, + num_tokens=num_tokens, + num_ctas=sms - moe_op.shared_ctas_for(num_tokens, True), + shared=True, + ) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("killed", [False, True]) +def test_config_and_kill_switch(monkeypatch: pytest.MonkeyPatch, killed: bool) -> None: + state = _state(monkeypatch, 8, killed) + assert state.config.get("out_skip", 0) == int(not killed) + assert bool(state.mod.OUT_SKIP) == (not killed) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("num_tokens", [1, 2, 3, 8]) +@pytest.mark.parametrize("killed", [False, True]) +def test_push_bits_unchanged_and_out_store_skipped( + monkeypatch: pytest.MonkeyPatch, num_tokens: int, killed: bool +) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + m3_moe_engine, + ) + + _check_push(m3_moe_engine, _state(monkeypatch, num_tokens, killed), num_tokens, 8, killed) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_sh20.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_sh20.py new file mode 100644 index 000000000000..d63bb464f22c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_sh20.py @@ -0,0 +1,59 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sm_107 TP8 MoE engine's SH20 layout: at 8 tokens the shared expert's 24 work units run on 20 shared CTAs, so the +routed grid has 192 CTAs (one FC2 32-row block each); other token counts keep 24 shared CTAs, and an engine without the +shared expert has none. TRTLLM_M31_DISABLE_MOE_SH20 restores 24 everywhere.""" + +import importlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + m31_switches, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) + + +def test_sh20_scope() -> None: + assert moe_op.SH20 and moe_op.SH20_CTAS == 20 and moe_op.SH20_TOKENS == frozenset({8}) + assert moe_op.SHARED_CTAS == 24 + # 6144 / 32 FC2 blocks: every routed CTA owns exactly one at 212 - 20 routed CTAs + assert moe_op.HIDDEN // 32 == 212 - moe_op.SH20_CTAS + + +@pytest.mark.parametrize("num_tokens", range(1, 9)) +def test_shared_ctas_for(num_tokens: int) -> None: + want = 20 if num_tokens == 8 else 24 + assert moe_op.shared_ctas_for(num_tokens, True) == want + assert moe_op.shared_ctas_for(num_tokens, False) == 0 + + +def test_kill_switch(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("TRTLLM_M31_DISABLE_MOE_SH20", "1") + importlib.reload(m31_switches) + try: + m = importlib.reload(moe_op) + assert not m.SH20 + assert [m.shared_ctas_for(n, True) for n in range(1, 9)] == [24] * 8 + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_MOE_SH20", raising=False) + importlib.reload(m31_switches) + importlib.reload(moe_op) + assert moe_op.SH20 + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("killed", [False, True]) +def test_sh20_config(monkeypatch: pytest.MonkeyPatch, killed: bool) -> None: + monkeypatch.setattr(moe_op, "SH20", not killed) + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + sh = moe_op.shared_ctas_for(8, True) + state = moe_op.M3MoeState(dev, num_tokens=8, num_ctas=sms - sh, shared=True) + assert state.config.get("sh_ctas", 24) == (24 if killed else 20) + assert state.mod.NSHC == (24 if killed else 20) and state.mod.NSH == 24 + assert state.mod.SH_NU == (1 if killed else 2) + assert state.epochs.numel() == state.num_ctas + state.mod.NSHC diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_tail_trigger.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_tail_trigger.py new file mode 100644 index 000000000000..1dc7a21c2793 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_tail_trigger.py @@ -0,0 +1,93 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MOE_TAIL_TRIGGER (default on) on the sm_107 TP8 MoE engine: on a call that pushes into the boundary +all-reduce, each routed CTA releases the PDL dependents when its weights producer is done, in place of +the release early_release picks (True: right after the grid wait, EARLY_AR; False, the model's TP8 default: +teardown), which TRTLLM_M31_DISABLE_MOE_TAIL_TRIGGER=1 restores. Scheduling only: the pushed words and ``out`` are +the same bits in every configuration, and the call epoch is stored early whenever a release point is early. The push +target is a plain device buffer laid out as set A of the multicast exchange (world 1, rank 0, parity 0): word t * H / +2 + c / 2 holds channels c, c + 1 of row t as bf16x2 (-0 pushed as +0).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_moe_operands import hidden_states, layer_for # noqa: E402 + +H = 6144 +EMPTY = -(2**31) # 0x80000000 + + +def _state(monkeypatch, num_tokens: int, killed: bool, early_release: bool = True): + monkeypatch.setattr( + moe_op, "m31_disabled", lambda item_id, k=killed: k and item_id == "MOE_TAIL_TRIGGER" + ) + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + return moe_op.M3MoeState( + dev, + num_tokens=num_tokens, + num_ctas=sms - moe_op.shared_ctas_for(num_tokens, True), + shared=True, + early_release=early_release, + ) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize( + "killed, early_release, tail, early", + [(False, True, 1, 0), (True, True, 0, 1), (False, False, 1, 0), (True, False, 0, 0)], +) +def test_release_config( + monkeypatch: pytest.MonkeyPatch, killed: bool, early_release: bool, tail: int, early: int +) -> None: + state = _state(monkeypatch, 8, killed, early_release) + assert int(bool(state.mod.TAIL_TRIGGER)) == tail + assert int(bool(state.mod.EARLY_AR)) == early + assert bool(state.mod.EARLY_EPOCH) == bool(tail or early) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("num_tokens", [1, 2, 3, 8]) +@pytest.mark.parametrize("killed", [False, True]) +@pytest.mark.parametrize("early_release", [False, True]) +def test_push_bits_and_epochs_unchanged( + monkeypatch: pytest.MonkeyPatch, num_tokens: int, killed: bool, early_release: bool +) -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + m3_moe_engine, + ) + + dev = torch.device("cuda") + bucket = 8 + state = _state(monkeypatch, num_tokens, killed, early_release) + layer = layer_for(state, "random") + hidden = hidden_states(num_tokens, seed=21 + num_tokens) + ref = m3_moe_engine(hidden, layer).clone() # no push: the release point does not apply + torch.cuda.synchronize() + ref_pos0 = torch.where(ref == 0, torch.zeros_like(ref), ref) # the push writes -0 as +0 + for _ in range(2): + buf = torch.full((2 * 2 * bucket * H // 2,), EMPTY, dtype=torch.int32, device=dev) + flags = torch.zeros(bucket * 8, dtype=torch.int32, device=dev) + arp = moe_op.ar_push_desc(buf.data_ptr(), flags.data_ptr(), 0, 1, bucket, dev) + before_ep = state.epochs.clone() + before = moe_op.TAIL_TRIGGER_TAKEN[0] + out = m3_moe_engine(hidden, layer, None, arp) + torch.cuda.synchronize() + assert moe_op.TAIL_TRIGGER_TAKEN[0] - before == int(not killed) + assert torch.equal(state.epochs, before_ep + 1), ( + "every CTA's epoch advances exactly once per call" + ) + pushed = buf[: num_tokens * H // 2].view(torch.bfloat16).view(num_tokens, H) + assert torch.equal(pushed.view(torch.int16), ref_pos0.view(torch.int16)), ( + "pushed words changed" + ) + assert bool((buf[num_tokens * H // 2 : bucket * H // 2] == EMPTY).all()), ( + "a push beyond the rows" + ) + if not state.config.get("out_skip"): # (OUT_SKIP: a pushing call leaves out unwritten) + assert torch.equal(out.view(torch.int16), ref.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_vrpoll.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_vrpoll.py new file mode 100644 index 000000000000..1b91576de7da --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m31_vr_tp8_moe_vrpoll.py @@ -0,0 +1,56 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The sm_107 TP8 MoE engine's VRPOLL override: 16 router logit replicas at 7 and 8 tokens instead of the kernel's 32; +up to 4 tokens keep LG_REPL; TRTLLM_M31_DISABLE_MOE_VRPOLL restores the kernel default.""" + +import importlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + m31_switches, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_moe import ( # noqa: E501 + op as moe_op, +) + + +def test_vrpoll_scope() -> None: + assert ( + moe_op.VRPOLL and moe_op.VRPOLL_LG_REPL == 16 and moe_op.VRPOLL_TOKENS == frozenset({7, 8}) + ) + assert max(moe_op.VRPOLL_TOKENS) > moe_op.LG_REPL_MAX_TOKENS >= 4 + + +def test_kill_switch(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("TRTLLM_M31_DISABLE_MOE_VRPOLL", "1") + importlib.reload(m31_switches) + try: + assert not importlib.reload(moe_op).VRPOLL + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_MOE_VRPOLL", raising=False) + importlib.reload(m31_switches) + importlib.reload(moe_op) + assert moe_op.VRPOLL + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="the engine state compiles the kernel") +@pytest.mark.parametrize("num_tokens", [4, 7, 8]) +@pytest.mark.parametrize("killed", [False, True]) +def test_vrpoll_config(monkeypatch: pytest.MonkeyPatch, num_tokens: int, killed: bool) -> None: + monkeypatch.setattr(moe_op, "VRPOLL", not killed) + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + state = moe_op.M3MoeState( + dev, + num_tokens=num_tokens, + num_ctas=sms - moe_op.shared_ctas_for(num_tokens, True), + shared=True, + ) + if num_tokens <= moe_op.LG_REPL_MAX_TOKENS: + want = moe_op.LG_REPL + else: + want = 32 if killed else 16 + assert state.mod.LG_REPL == want + assert state.lgbuf.numel() == want * state.mod.LG_STRIDE diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m3_moe_engine.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m3_moe_engine.py new file mode 100644 index 000000000000..07a3ea62395e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m3_moe_engine.py @@ -0,0 +1,139 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the moe/m3_moe_engine entry: the 1..8-token MoE engine on its caller-owned state. + +One Rubin (SM107) rank of the tp8 deployment: 128 experts (all local, routed and shared intermediate 384 per +rank), top 4, hidden 6144, the shared expert folded in, ``num_ctas`` = SMs - 24 as the target builds it. Operands: +``_m3_moe_operands``. +""" + +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + HIDDEN, + INTER, + SHARED_INTER, + M3MoeState, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.moe.m3_moe_engine import ( # noqa: E501 + m3_moe_engine as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_moe_operands import ( # noqa: E402 + I_TP, + SHARED_I, + H, + check_routed_part, + hidden_states, + layer_for, +) +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_moe_engine requires a CUDA device" + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + +_STATES = {} + + +def test_the_operands_are_the_tp8_engine_geometry() -> None: + assert (H, I_TP, SHARED_I) == (HIDDEN, INTER, SHARED_INTER) == (6144, 384, 384) + + +def test_the_engine_refuses_the_tp4_width() -> None: + with pytest.raises(ValueError, match="384"): + M3MoeState(torch.device("cuda"), num_tokens=1, inter=768) + + +def _state(m: int) -> M3MoeState: + if m not in _STATES: + dev = torch.device("cuda") + sms = torch.cuda.get_device_properties(dev).multi_processor_count + _STATES[m] = M3MoeState(dev, num_tokens=m, num_ctas=sms - 24, shared=True) + return _STATES[m] + + +def compare(out: torch.Tensor, ref: torch.Tensor) -> None: + """The engine against the plain-torch MoE: BAND_ELEMENT bf16 ulp of the row's largest magnitude per element, + BAND_RMS ulp relative RMS. + + Both sides quantize the same operands with the same recipes (the activation's and the intermediate's NVFP4 codes + and block scales, the shared expert's MXFP8); they differ in summation order (tensor-core fp32 against fp64) and in + the PTX approximations (rcp / ex2 / tanh). Such a difference changes the output only where it moves a value across + a rounding boundary: one bf16 ulp of a slot output or of the sum, or, rarely, one intermediate NVFP4 code, which + reaches the output as one weight column's share. Measured over every cell, both routing cases and 3 seeds each + on the tp4 engine (GB300, intermediate 768): at most 1.48 ulp element, 0.18 ulp RMS; the tp8 engine runs the + same recipes over half the intermediate. The band leaves ~3x on both; a misread recipe (the activation's + 1 / (sf x (1 / sc)) spelling of the reciprocal on ~0.06% of its codes) measured 12 ulp element, 7 ulp RMS. + """ + assert_within_ulp(out, ref, element_ulp=BAND_ELEMENT, rms_ulp=BAND_RMS) + + +BAND_ELEMENT = 4.0 +BAND_RMS = 0.5 + + +@pytest.mark.parametrize("case", ("random", "concentrated")) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, case) -> None: + m = cell.spec["tokens"] + layer = layer_for(_state(m), case) + hidden = hidden_states(m, seed=zlib.crc32((cell.why + case).encode()) % 2**31) + with validating(op): + out = op(hidden, layer) + again = op(hidden, layer) + assert out.shape == (m, H) and out.dtype is torch.bfloat16 + # The workspace is tagged per call and never reset: a repeated call is the same call. + assert torch.equal(out.view(torch.int16), again.view(torch.int16)), ( + "a repeated call changed the output" + ) + compare(out, op.reference(hidden, layer)) + + +@pytest.mark.parametrize("m", (1, 2, 8)) +def test_routed_part_is_trtllm_gen_bit_for_bit(m) -> None: + check_routed_part(op, _state(m), m, seed=100 + m) + + +def test_layers_on_one_state_do_not_leak_into_each_other() -> None: + """Two layers share one state's workspace, in stream order: A, B, A gives A's output twice, bit for bit, and the + caller's ``out`` receives the same rows as a fresh tensor.""" + state = _state(4) + a, b = layer_for(state, "random"), layer_for(state, "concentrated") + hidden = hidden_states(4, seed=7) + first = op(hidden, a) + op(hidden, b) + out = torch.empty(4, H, dtype=torch.bfloat16, device="cuda") + second = op(hidden, a, out=out) + assert second.data_ptr() == out.data_ptr() + assert torch.equal(first.view(torch.int16), out.view(torch.int16)) + + +@pytest.mark.parametrize( + "bad", + ("misaligned hidden", "fp32 out", "short out", "long out", "int32 arp"), +) +def test_malformed_buffers_are_rejected_by_the_call(bad) -> None: + """Why the entry has no is_valid: the compiled call checks each buffer against its signature and raises.""" + state = _state(2) + layer = layer_for(state, "random") + hidden, kw = hidden_states(2, seed=3), {} + if bad == "misaligned hidden": + hidden = torch.randn(2 * H + 8, device="cuda").to(torch.bfloat16)[2 : 2 + 2 * H].view(2, H) + elif bad == "fp32 out": + kw["out"] = torch.empty(2, H, dtype=torch.float32, device="cuda") + elif bad == "short out": + kw["out"] = torch.empty(1, H, dtype=torch.bfloat16, device="cuda") + elif bad == "long out": + kw["out"] = torch.empty(3, H, dtype=torch.bfloat16, device="cuda") + else: + kw["arp"] = torch.zeros(8, dtype=torch.int32, device="cuda") + with pytest.raises(ValueError): + op(hidden, layer, **kw) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m3_router_tc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m3_router_tc.py new file mode 100644 index 000000000000..5d649c1ccb7c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/moe/test_modeling_v2_m3_router_tc.py @@ -0,0 +1,57 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_router_tc entry of the MiniMax-M3 tp8 target.""" + +import pytest +import torch + +__extra_import_path__ = ["../../../..", "."] +from _m3_router import ( # noqa: E402, F401 + SF_SCALE, + _build, + _compare_fp4, + _compare_logits, + _seed, + fp32_logits, + tc, +) +from _validating import validating # noqa: E402 — needs the path declared above + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + + +@pytest.mark.parametrize("cell", tc.CELLS, ids=[c.why[:48] for c in tc.CELLS]) +def test_tc_certified_cells(cell) -> None: + spec = cell.spec + x, w = _build(spec["m"], 128, spec["k"], _seed(cell)) + sf_scale = torch.tensor([SF_SCALE], dtype=torch.float32, device="cuda") if spec["fp4"] else None + with validating(tc): + outs = tc(x, w, sf_scale, False) + ref = tc.reference(x, w, sf_scale, False) + assert len(outs) == len(ref) == (3 if spec["fp4"] else 1) + _compare_logits(outs[0], ref[0]) + if spec["fp4"]: + _compare_fp4(outs[1], outs[2], x, sf_scale) + + +def test_tc_reference_is_tf32_and_the_gate_sees_it() -> None: + """The tensor-core router rounds the weight to TF32: against a true fp32 product its logits sit far outside + the band, so the TF32 rounding in ``reference`` is load-bearing rather than decoration.""" + x, w = _build(16, 128, 6144, 5) + out = tc(x, w, None, False)[0] + with pytest.raises(AssertionError): + _compare_logits(out, fp32_logits(x, w)) + + +def test_guard_refuses_a_misaligned_activation() -> None: + """A row view starting 8 bytes into the buffer keeps a legal row stride but not a legal bulk-copy source.""" + x, w = _build(2, 128, 6144, 3) + flat = torch.empty(2 * 6144 + 4, dtype=torch.bfloat16, device="cuda") + shifted = flat[4:].view(2, 6144) + shifted.copy_(x) + with pytest.raises(AssertionError, match="16-byte aligned"): + with validating(tc): + tc(shifted, w, None, False) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/norm/test_modeling_v2_m31_ctxnormq.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/norm/test_modeling_v2_m31_ctxnormq.py new file mode 100644 index 000000000000..19639601bcc0 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/norm/test_modeling_v2_m31_ctxnormq.py @@ -0,0 +1,79 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for CTXNORMQ: the prefill boundary's residual add + RMSNorm that also writes the next projection's MXFP8 +activation and the routed experts' NVFP4 input (torch.ops.trtllm.minimax_m31_ctx_residual_rmsnorm_mxfp8). + +The quantized outputs must equal mxfp8_quantize / fp4_quantize of the op's own norm bit for bit; the norm and the +updated residual are checked against an fp32 reference. Skipped when the op is not built (no overlay library). +""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( + modeling as m, +) + +assert torch.cuda.is_available(), "CTXNORMQ requires a CUDA device" + +H = 6144 +EPS = 1e-6 +FP4_SCALE = 448.0 * 6.0 / 8.0 + +op = m._ctxnormq_op() +pytestmark = pytest.mark.skipif( + op is None, reason="minimax_m31_ctx_residual_rmsnorm_mxfp8 is not built" +) + + +def _operands(rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + inter = torch.randn(rows, H, generator=g, device="cuda").to(torch.bfloat16) + residual = (2 * torch.randn(rows, H, generator=g, device="cuda")).to(torch.bfloat16) + weight = (1.0 + 0.1 * torch.randn(H, generator=g, device="cuda")).to(torch.bfloat16) + return inter, residual, weight + + +@pytest.mark.parametrize("rows", [1024, 1160, 4096]) +@pytest.mark.parametrize("fp4", [False, True], ids=["attn", "moe"]) +def test_quantized_outputs_equal_quantizing_the_norm(rows, fp4) -> None: + inter, residual, weight = _operands(rows, 20261021 + rows) + scale = torch.tensor([FP4_SCALE], device="cuda", dtype=torch.float32) if fp4 else None + ref_updated = (inter.float() + residual.float()).to(torch.bfloat16) + normed, q, sf, f4, f4_sf = op(inter, residual, weight, EPS, True, scale, False) + # The updated residual is written in place into inter: one bf16 rounding of the fp32 sum, exact. + assert torch.equal(inter.view(torch.int16), ref_updated.view(torch.int16)) + u = ref_updated.float() + ref_normed = u * torch.rsqrt(u.pow(2).mean(-1, keepdim=True) + EPS) * weight.float() + # fp32 sum of squares + fast rsqrt vs torch's: at most one bf16 rounding apart. + torch.testing.assert_close(normed.float(), ref_normed, rtol=2**-7, atol=2**-7) + q_ref, sf_ref = torch.ops.trtllm.mxfp8_quantize(normed, True) + assert torch.equal(q.view(torch.uint8), q_ref.view(torch.uint8)) + assert torch.equal(sf.view(torch.uint8)[: sf_ref.numel()], sf_ref.view(torch.uint8).flatten()) + if fp4: + f4_ref, f4_sf_ref = torch.ops.trtllm.fp4_quantize(normed, scale, 16, False, False) + assert torch.equal(f4.view(torch.uint8).flatten(), f4_ref.view(torch.uint8).flatten()) + assert torch.equal( + f4_sf.view(torch.uint8).flatten(), + f4_sf_ref.view(torch.uint8).flatten()[: f4_sf.numel()], + ) + # _routed_chunks splits the NVFP4 scales per row chunk. + assert f4_sf.numel() == rows * H // 16 + + +def test_deterministic() -> None: + outs = [] + for _ in range(3): + inter, residual, weight = _operands(2048, 7) + scale = torch.tensor([FP4_SCALE], device="cuda", dtype=torch.float32) + outs.append((inter,) + tuple(op(inter, residual, weight, EPS, True, scale, False))) + for o in outs[1:]: + assert all( + torch.equal(a.view(torch.uint8), b.view(torch.uint8)) for a, b in zip(outs[0], o) + ) + + +def test_small_boundaries_keep_the_fused_allreduce() -> None: + """Below CTXNORMQ_MIN_ROWS the boundary is not taken (the microbench crossover); decided before any collective.""" + x = torch.zeros(m.CTXNORMQ_MIN_ROWS - 1, H, dtype=torch.bfloat16, device="cuda") + assert m._ctxnormq_allreduce_norm(None, x, x, x[0], None) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/norm/test_modeling_v2_m3_embedding_firstnorm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/norm/test_modeling_v2_m3_embedding_firstnorm.py new file mode 100644 index 000000000000..4f552ef387d2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/norm/test_modeling_v2_m3_embedding_firstnorm.py @@ -0,0 +1,102 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the norm/m3_embedding_firstnorm entry: the decode embedding gather + the first layer's RMSNorm. + +The table is the model's full bf16 [200064, 6144] embedding shape (random values at the checkpoint's scale), eps the +checkpoint's 1e-6. +""" + +import functools +import zlib + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.norm.m3_embedding_firstnorm import ( # noqa: E501 + VOCAB, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.norm.m3_embedding_firstnorm import ( # noqa: E501 + m3_embedding_firstnorm as op, +) + +__extra_import_path__ = ["../../../.."] +from _validating import assert_within_ulp, validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_embedding_firstnorm requires a CUDA device" + +H = 6144 +EPS = 1e-6 + + +@functools.lru_cache(maxsize=None) +def _operands(): + g = torch.Generator(device="cuda").manual_seed(20261013) + table = (0.02 * torch.randn(VOCAB, H, generator=g, device="cuda")).to(torch.bfloat16) + weight = (1.0 + 0.1 * torch.randn(H, generator=g, device="cuda")).to(torch.bfloat16) + return table, weight + + +def _buffers(rows): + return (torch.empty(rows, H, dtype=torch.bfloat16, device="cuda"), + torch.empty(rows, H, dtype=torch.bfloat16, device="cuda")) # fmt: skip + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + rows, id_dtype = cell.spec["rows"], cell.spec["id_dtype"] + table, weight = _operands() + g = torch.Generator(device="cuda").manual_seed(zlib.crc32(cell.why.encode()) % 2**31) + ids = torch.randint(0, VOCAB, (rows,), generator=g, device="cuda").to(id_dtype) + raw, out = _buffers(rows) + with validating(op): + normed, residual = op(ids, table, weight, EPS, raw, out) + assert normed.data_ptr() == out.data_ptr() and residual.data_ptr() == raw.data_ptr() + ref_normed, ref_raw = op.reference(ids, table, weight, EPS, raw, out) + # The gather is a copy: exact. + assert torch.equal(raw.view(torch.int16), ref_raw.view(torch.int16)) + # The norm: the kernel's fp32 sum of squares (warp butterflies, then across 4 warps) and fast rsqrt against an + # fp64 reference, then x * rstd * w rounded once to bf16. The fp32 side is ~1e-6 relative off, so the two roundings + # disagree only next to a bf16 rounding boundary: one ulp at most per element, a small fraction in aggregate. + # Measured over these cells (3 seeds each, GB300): at most 0.51 ulp element, 0.026 ulp RMS. + assert_within_ulp(out, ref_normed, element_ulp=1.0, rms_ulp=0.1) + + +def test_ids_outside_the_table_give_zero_rows() -> None: + """Documented, not refused: an id outside [0, V) gathers a zero row whose norm is zero.""" + table, weight = _operands() + ids = torch.tensor([-1, 5, VOCAB], dtype=torch.int64, device="cuda") + raw, out = _buffers(3) + op(ids, table, weight, EPS, raw, out) + assert not raw[0].any() and not raw[2].any() and not out[0].any() and not out[2].any() + assert torch.equal(raw[1], table[5]) + + +@pytest.mark.parametrize("rows", range(1, 9)) +def test_against_the_stock_embedding_and_rmsnorm(rows) -> None: + """What this target certifies on SM107 against the built-in path it replaces (the model's Embedding gather, then + the first layer's RMSNorm module), including a constant row, large and tiny rows and both vocab ends: the gather + bit for bit, the norm within one bf16 ulp (the sum-of-squares tree differs from flashinfer's: measured ~5e-6 of + the elements one ulp off over 12000 random rows).""" + from tensorrt_llm._torch.modules.rms_norm import RMSNorm + + table, weight = _operands() + table = table.clone() + g = torch.Generator(device="cuda").manual_seed(rows) + table[1] = 3.0 + table[2] = (300.0 * torch.randn(H, generator=g, device="cuda")).to(torch.bfloat16) + table[3] = (1e-4 * torch.randn(H, generator=g, device="cuda")).to(torch.bfloat16) + norm = RMSNorm(hidden_size=H, eps=EPS, dtype=torch.bfloat16).cuda() + with torch.no_grad(): + norm.weight.copy_(weight) + for ids in ( + torch.randint(0, VOCAB, (rows,), generator=g, device="cuda"), + torch.tensor([1, 2, 3, 0, VOCAB - 1, 1, 2, 3][:rows], device="cuda"), + ): + raw, out = _buffers(rows) + normed, residual = op(ids, table, weight, EPS, raw, out) + x = torch.nn.functional.embedding(ids, table) + assert torch.equal(residual.view(torch.int16), x.view(torch.int16)) + stock = norm(x) + off = normed != stock + assert int(off.sum()) <= 4, f"{int(off.sum())} normed elements differ from stock" + assert_within_ulp(normed, stock, element_ulp=1.0, rms_ulp=0.05) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_chain_spec_mc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_chain_spec_mc.py new file mode 100644 index 000000000000..7f77ccfc65c2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_chain_spec_mc.py @@ -0,0 +1,527 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for REJ_MC: the spec sampler's chain speculative sampling as +``torch.ops.trtllm.minimax_m31_chain_speculative_sampling_mc`` through ``ops/sampling/m31_chain_spec_mc``, installed as +the rejection sampler's override. + +It must equal the stock ``torch.ops.trtllm.chain_speculative_sampling`` bit for bit, tokens and counts (the microbench +generators: realistic / same / far / ties / onehot drafts and targets, K7). The shared hook in +``speculative/interface.rejection_sampling_one_model`` must be stock while no override is set, and fall back to the +stock op whenever the override declines. TRTLLM_M31_DISABLE_REJ_MC=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.sampling import ( # noqa: E501 + m31_chain_spec_mc as entry, +) +from tensorrt_llm._torch.speculative import interface as spec_interface + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8" +) +CASES = ("realistic", "same", "far", "ties", "onehot") +op = entry.m31_chain_spec_mc + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("REJ_MC is built for sm_107 here") + if entry.chain_spec_mc_op() is None: + pytest.skip("minimax_m31_chain_speculative_sampling_mc is not built") + if not hasattr(torch.ops.trtllm, "chain_speculative_sampling"): + pytest.skip("the stock chain_speculative_sampling op is not built") + pytest.importorskip("flashinfer.sampling") + + +def _probs(logits, top_p=0.95): + """softmax then FlashInfer's top-p renorm, as the sampler feeds the rejection step.""" + import flashinfer.sampling + + v = logits.shape[-1] + p = torch.softmax(logits, -1) + renormed = flashinfer.sampling.top_p_renorm_probs(p.view(-1, v).contiguous(), top_p) + return renormed.view(logits.shape).contiguous() + + +def _make(case, batch, k, v, seed): + """The microbench generators.""" + g = torch.Generator(device="cuda").manual_seed(seed) + + def rl(shape, scale=3.0): + return torch.randn(*shape, v, generator=g, device="cuda") * scale + + dl = rl((batch, k)) + if case == "realistic": # independent draft / target + tl = torch.cat([dl, rl((batch, 1))], 1) + rl((batch, k + 1), 1.0) + elif case == "same": # target == draft: all-accept and zero-residual paths + tl = torch.cat([dl, rl((batch, 1))], 1) + elif case == "far": # target peaked elsewhere: early reject + tl = rl((batch, k + 1)) + tl[:, :, :8] += 20 + elif case == "ties": + dl = torch.round(dl * 2) / 2 + tl = torch.round(torch.cat([dl, rl((batch, 1))], 1) * 2) / 2 + elif case == "onehot": + tl = torch.full((batch, k + 1, v), -1e4, device="cuda") + tl[:, :, 1234] = 0 + else: + raise ValueError(case) + dp, tp = _probs(dl), _probs(tl) + ids = torch.multinomial(dp.view(-1, v), 1, generator=g).view(batch, k).to(torch.int32) + seeds = torch.randint(0, 2**31, (batch,), generator=g, device="cuda", dtype=torch.int64) + offsets = torch.randint(0, 2**20, (batch,), generator=g, device="cuda", dtype=torch.int64) + return dp, ids.contiguous(), tp, seeds, offsets + + +def _stock(*args): + return torch.ops.trtllm.chain_speculative_sampling(*args) + + +def _same(got, ref, what): + assert torch.equal(got[0], ref[0]), f"{what}: accepted tokens differ" + assert torch.equal(got[1], ref[1]), f"{what}: token counts differ" + + +@pytest.mark.parametrize("case", CASES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_stock_op(cell, case) -> None: + """At each certified (batch, K, vocab): the override equals the stock op (tokens and counts), and the explicit + entry call at ctas_per_row 1 / 4 / 8 / 16 / 32 / 64 gives the same result again (REJ v2: the grid size never changes + the result).""" + _gpu() + batch, k, v = cell.spec["batch"], cell.spec["k"], cell.spec["v"] + for seed in range(4): + args = _make(case, batch, k, v, 1000 * seed + batch) + got = entry.chain_spec_mc(*args) + assert got is not None, "an in-contract input must be served" + _same(got, _stock(*args), f"{case} batch={batch} seed={seed}") + for ctas in (1, 4, 8, 16, 32, 64): + if ctas > 1 and batch * ctas > entry.sm_count(args[0].device): + continue + _same(op(*args, entry.prebuild(args[0].device), ctas), got, f"ctas_per_row={ctas}") + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager result.""" + _gpu() + args = _make("realistic", 8, 7, 200064, 7) + entry.prebuild(args[0].device) + ref = entry.chain_spec_mc(*args) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.chain_spec_mc(*args) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.chain_spec_mc(*args) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out[0].fill_(-7) + out[1].fill_(-7) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """Non-fp32 probs, non-int32 draft tokens, non-int64 or mis-sized seed / offset, a vocab not divisible by 4, a + misaligned or non-contiguous probs, mismatched shapes, CPU inputs or more requests than MAX_BATCH return None (the + sampler runs the stock op).""" + _gpu() + dp, ids, tp, seeds, offsets = _make("realistic", 2, 3, 32768, 1) + assert entry.chain_spec_mc(dp, ids, tp, seeds, offsets) is not None + assert entry.chain_spec_mc(dp.double(), ids, tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids, tp.half(), seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids.long(), tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids, tp, seeds.int(), offsets) is None + assert entry.chain_spec_mc(dp, ids, tp, seeds[:1], offsets) is None + assert entry.chain_spec_mc(dp, ids, tp, seeds, offsets[:1]) is None + assert entry.chain_spec_mc(dp, ids[:, :2].contiguous(), tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp, ids, tp[:, :3].contiguous(), seeds, offsets) is None + assert entry.chain_spec_mc(dp[..., :-4], ids, tp[..., :-4], seeds, offsets) is None + assert ( + entry.chain_spec_mc( + dp[..., :-2].contiguous(), ids, tp[..., :-2].contiguous(), seeds, offsets + ) + is None + ) + flat = torch.empty(dp.numel() + 1, device="cuda") + assert entry.chain_spec_mc(flat[1:].view(dp.shape), ids, tp, seeds, offsets) is None + assert entry.chain_spec_mc(dp.cpu(), ids.cpu(), tp.cpu(), seeds.cpu(), offsets.cpu()) is None + b = entry.MAX_BATCH + 1 + wide = _make("realistic", b, 1, 32768, 2) + assert entry.chain_spec_mc(*wide) is None + # More draft tokens than the kernel's shared acceptance probabilities hold. + deep = _make("realistic", 1, entry.MAX_DRAFT + 1, 32768, 3) + assert entry.chain_spec_mc(*deep) is None + edge = _make("realistic", 1, entry.MAX_DRAFT, 32768, 4) + _same(entry.chain_spec_mc(*edge), _stock(*edge), "k = MAX_DRAFT") + assert entry.ctas_per_row(1, 148) == 64 and entry.ctas_per_row(2, 148) == 64 + assert entry.ctas_per_row(4, 148) == 32 and entry.ctas_per_row(8, 148) == 16 + assert entry.ctas_per_row(16, 148) == 8 and entry.ctas_per_row(64, 148) == 2 + assert entry.ctas_per_row(200, 148) == 1 + # This device's own SM count keeps the largest grid co-resident. + sms = entry.sm_count(torch.device("cuda")) + assert entry.MAX_BATCH * entry.ctas_per_row(entry.MAX_BATCH, sms) <= sms + + +@pytest.fixture +def _fake_stock_op(monkeypatch): + """Record the feature FlashInfer fallback; retain real torch tensor semantics.""" + stock = (torch.zeros(2, 4, dtype=torch.int32), torch.ones(2, dtype=torch.int32)) + fake_op = mock.Mock(return_value=(stock[0], None, stock[1] - 1)) + fake_fi = types.SimpleNamespace( + __version__="0.6.4", sampling=types.SimpleNamespace(chain_speculative_sampling=fake_op) + ) + monkeypatch.setattr(spec_interface, "flashinfer", fake_fi, raising=False) + monkeypatch.setattr(spec_interface, "IS_FLASHINFER_AVAILABLE", True) + spec_interface.set_chain_speculative_sampling_override(None) + yield fake_op, stock + spec_interface.set_chain_speculative_sampling_override(None) + + +def _cpu_args(): + return ( + torch.rand(2, 3, 8), + torch.zeros(2, 3, dtype=torch.int32), + torch.rand(2, 4, 8), + torch.zeros(2, dtype=torch.int64), + torch.zeros(2, dtype=torch.int64), + ) + + +def test_hook_unset_is_stock(_fake_stock_op) -> None: + """No override: rejection_sampling_one_model makes exactly the stock op call and returns its result.""" + fake_op, stock = _fake_stock_op + args = _cpu_args() + assert spec_interface._CHAIN_SPEC_OVERRIDE is None + result = spec_interface.rejection_sampling_one_model(*args[:3], seed=args[3], offset=args[4]) + assert result[0] is stock[0] + assert torch.equal(result[1], stock[1]) + fake_op.assert_called_once() + assert all(a is b for a, b in zip(fake_op.call_args.args, args[:3])) + assert fake_op.call_args.kwargs["seed"] is args[3] + assert fake_op.call_args.kwargs["offset"] is args[4] + assert fake_op.call_args.kwargs["deterministic"] is True + assert fake_op.call_args.kwargs["generator"] is None + + +def test_hook_override_result_is_used(_fake_stock_op) -> None: + """An override's result replaces the stock op; the stock op does not run.""" + fake_op, _ = _fake_stock_op + mine = (torch.ones(2, 4, dtype=torch.int32), torch.full((2,), 4, dtype=torch.int32)) + override = mock.Mock(return_value=mine) + spec_interface.set_chain_speculative_sampling_override(override) + args = _cpu_args() + assert ( + spec_interface.rejection_sampling_one_model(*args[:3], seed=args[3], offset=args[4]) is mine + ) + override.assert_called_once_with(*args) + fake_op.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_stock_op) -> None: + """An override that returns None leaves the stock op call in place.""" + fake_op, stock = _fake_stock_op + override = mock.Mock(return_value=None) + spec_interface.set_chain_speculative_sampling_override(override) + args = _cpu_args() + result = spec_interface.rejection_sampling_one_model(*args[:3], seed=args[3], offset=args[4]) + assert result[0] is stock[0] + assert torch.equal(result[1], stock[1]) + assert override.call_count == 1 and fake_op.call_count == 1 + + +def test_the_target_installs_the_override() -> None: + """With REJ_MC on (the default) and the op built, the target installs ``chain_spec_mc`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, + ) + + if not modeling.REJ_MC: + pytest.skip("REJ_MC is switched off in this environment") + saved = dict(modeling._REJ_MC_STATE) + modeling._REJ_MC_STATE.clear() + try: + modeling._install_rej_mc() + assert spec_interface._CHAIN_SPEC_OVERRIDE is entry.chain_spec_mc + assert modeling._REJ_MC_STATE["installed"] is True + if modeling.REJ_SLOTS and entry.chain_spec_mc_slots_op() is not None: + assert spec_interface._CHAIN_SPEC_SLOT_OVERRIDE is entry.chain_spec_mc_slots + finally: + spec_interface.set_chain_speculative_sampling_override(None) + spec_interface.set_chain_speculative_sampling_slot_override(None) + modeling._REJ_MC_STATE.clear() + modeling._REJ_MC_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_REJ_MC=1 (read at import): REJ_MC is off and the target leaves the rejection hook unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.speculative import interface as i\n" + "m._install_rej_mc()\n" + "print(m.REJ_MC, i._CHAIN_SPEC_OVERRIDE is None, m._REJ_MC_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_REJ_MC="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" + + +def _slot_case(case, batch, k, v, seed, slots=16, max_draft=8, v_pad=64): + """A spec-metadata-like slot buffer [slots, max_draft, v + v_pad] with the request rows' draft probs at random + distinct slots (the other cells hold junk), plus the gathered contiguous copy the sampler would pass.""" + dp, ids, tp, seeds, offs = _make(case, batch, k, v, seed) + g = torch.Generator(device="cuda").manual_seed(seed + 1) + buf = torch.rand(slots, max_draft, v + v_pad, generator=g, device="cuda") + slot_ids = ( + torch.randperm(slots, generator=g, device="cuda")[:batch].to(torch.int64).contiguous() + ) + buf[slot_ids, :k, :v] = dp + gathered = buf[slot_ids, :k, :v].contiguous() + assert torch.equal(gathered, dp) + return buf, slot_ids, gathered, ids, tp, seeds, offs + + +def _slots_gpu(): + _gpu() + if entry.chain_spec_mc_slots_op() is None: + pytest.skip("minimax_m31_chain_speculative_sampling_mc_slots is not built") + + +@pytest.mark.parametrize("case", CASES) +@pytest.mark.parametrize("batch", [1, 2, 8]) +def test_slots_bitwise_the_gathered_call(case, batch) -> None: + """REJ_SLOTS: reading the draft rows in place by slot (padded draft length and vocab stride, junk in the other + cells) gives exactly the gathered call's tokens and counts, and the stock op's.""" + _slots_gpu() + k, v = 7, 200064 + for seed in range(3): + buf, slot_ids, gathered, ids, tp, seeds, offs = _slot_case( + case, batch, k, v, 100 * seed + batch + ) + got = entry.chain_spec_mc_slots(buf, slot_ids, k, ids, tp, seeds, offs) + assert got is not None, "an in-contract slot call must be served" + _same( + got, + entry.chain_spec_mc(gathered, ids, tp, seeds, offs), + f"{case} batch={batch} seed={seed} vs gathered", + ) + _same( + got, + _stock(gathered, ids, tp, seeds, offs), + f"{case} batch={batch} seed={seed} vs stock", + ) + + +def test_slots_graph_replay_with_new_slots() -> None: + """Captured with the slot ids as a device tensor: a replay after the slots change reads the new rows.""" + _slots_gpu() + batch, k, v = 2, 7, 200064 + buf, slot_ids, gathered, ids, tp, seeds, offs = _slot_case("realistic", batch, k, v, 5) + entry.prebuild(buf.device) + entry.chain_spec_mc_slots(buf, slot_ids, k, ids, tp, seeds, offs) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.chain_spec_mc_slots(buf, slot_ids, k, ids, tp, seeds, offs) + for rep in range(3): + g = torch.Generator(device="cuda").manual_seed(50 + rep) + new_slots = torch.randperm(buf.shape[0], generator=g, device="cuda")[:batch].to(torch.int64) + slot_ids.copy_(new_slots) + graph.replay() + torch.cuda.synchronize() + ref = _stock(buf[slot_ids, :k, :v].contiguous(), ids, tp, seeds, offs) + _same(out, ref, f"graph replay {rep}") + + +def test_slots_outside_the_contract_decline() -> None: + _slots_gpu() + buf, slot_ids, gathered, ids, tp, seeds, offs = _slot_case("realistic", 2, 7, 200064, 9) + assert ( + entry.chain_spec_mc_slots(buf, slot_ids.int(), 7, ids, tp, seeds, offs) is None + ) # int32 slots + assert entry.chain_spec_mc_slots(buf.double(), slot_ids, 7, ids, tp, seeds, offs) is None + assert ( + entry.chain_spec_mc_slots(buf, slot_ids, 9, ids, tp, seeds, offs) is None + ) # num_draft > max draft + assert ( + entry.chain_spec_mc_slots(buf[:, :, :1000], slot_ids, 7, ids, tp, seeds, offs) is None + ) # narrow / strided + + +def test_the_slots_kill_switch_installs_no_slot_hook() -> None: + """TRTLLM_M31_DISABLE_REJ_SLOTS=1 (read at import): REJ_MC still installs, the slot-indexed hook stays unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.speculative import interface as i\n" + "m._install_rej_mc()\n" + "print(m.REJ_SLOTS, i._CHAIN_SPEC_SLOT_OVERRIDE is None)" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_REJ_SLOTS="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True" + + +def test_the_slots_hook_is_off_by_default_and_opt_in() -> None: + """REJ_SLOTS is off by default (read at import); TRTLLM_M31_ENABLE_REJ_SLOTS=1 turns it on; the disable switch + wins.""" + probe = f"import {TARGET}.modeling as m\nprint(m.REJ_SLOTS)" + base = {k: v for k, v in os.environ.items() if not k.endswith("_REJ_SLOTS")} + for extra, want in ( + ({}, "False"), + ({"TRTLLM_M31_ENABLE_REJ_SLOTS": "1"}, "True"), + ({"TRTLLM_M31_ENABLE_REJ_SLOTS": "1", "TRTLLM_M31_DISABLE_REJ_SLOTS": "1"}, "False"), + ): + out = subprocess.run( + [sys.executable, "-c", probe], + env=dict(base, **extra), + capture_output=True, + text=True, + check=True, + ) + assert out.stdout.strip().splitlines()[-1] == want, (extra, out.stdout[-200:]) + + +def test_rej_v2_taken_line_once_per_grid(monkeypatch) -> None: + """REJ_V2's taken line names the grid a served call launched with, once per distinct ctas_per_row.""" + _gpu() + from tensorrt_llm.logger import logger + + lines = [] + monkeypatch.setattr(logger, "info", lambda msg, *a, **k: lines.append(msg)) + monkeypatch.setattr(entry, "_V2_TAKEN", set()) + args = _make("realistic", 1, 7, 32768, 5) + g = entry.ctas_per_row(1, entry.sm_count(args[0].device)) + for _ in range(2): + assert entry.chain_spec_mc(*args) is not None + taken = [m for m in lines if "REJ_V2 taken" in m] + assert taken == [ + f"[M3 dispatch] REJ_V2 taken (max_ctas_per_row {entry.MAX_CTAS_PER_ROW}, ctas_per_row {g} at batch 1)" + ], taken + + +def test_rej_v3_taken_line_once() -> None: + """REJ v3 (the default kernel) logs one REJ_V3 taken line per process, with the lib's workspace bytes.""" + _gpu() + from tensorrt_llm.logger import logger + + lines = [] + with ( + mock.patch.object(logger, "info", lambda msg, *a, **k: lines.append(msg)), + mock.patch.object(entry, "_V2_TAKEN", set()), + ): + args = _make("realistic", 1, 7, 32768, 6) + for _ in range(2): + assert entry.chain_spec_mc(*args) is not None + if entry.kernel_version() != 3: + pytest.skip("this lib launches the v2 kernel") + assert [m for m in lines if "REJ_V3 taken" in m] == [ + f"[M3 dispatch] REJ_V3 taken (ws_bytes_per_row {entry.ws_bytes_per_row()})" + ], lines + assert not [m for m in lines if "REJ_V2_KERNEL" in m], lines + + +def _v3_adversarial(case, batch, k, v, seed): + """Rows aimed at v3's moved steps: the staged chain tails, both tile variants and the leader's tile walk.""" + g = torch.Generator(device="cuda").manual_seed(seed) + dp = torch.softmax(torch.randn(batch, k, v, generator=g, device="cuda") * 3.0, -1) + bonus = torch.softmax(torch.randn(batch, 1, v, generator=g, device="cuda") * 3.0, -1) + if ( + case == "allaccept" + ): # target == draft: every draft token accepted, the target-only (bonus) path + tp = torch.cat([dp, bonus], 1) + elif case == "lasttile": # the residual has mass only in the last tile + tp = torch.cat([dp, bonus], 1) + tp[:, :, (v - 1) // 4096 * 4096 :] += 1e-3 + tp = tp / tp.sum(-1, keepdim=True) + elif ( + case == "dyadic" + ): # power-of-two masses: tile sums and prefixes land exactly on the threshold grid + tp = torch.zeros(batch, k + 1, v, device="cuda") + tp[:, :, : 4 * 4096] = 1.0 / (4 * 4096) + dp = torch.zeros(batch, k, v, device="cuda") + dp[:, :, 4096 : 3 * 4096] = 1.0 / (2 * 4096) + elif case == "zerotail": # the target has no mass past the third tile + tp = torch.softmax(torch.randn(batch, k + 1, v, generator=g, device="cuda") * 3.0, -1) + tp[:, :, 3 * 4096 :] = 0.0 + tp = tp / tp.sum(-1, keepdim=True) + else: + raise ValueError(case) + dp, tp = dp.contiguous(), tp.contiguous() + ids = ( + torch.multinomial(dp.clamp_min(1e-12).view(-1, v), 1, generator=g) + .view(batch, k) + .to(torch.int32) + ) + seeds = torch.randint(0, 2**31, (batch,), generator=g, device="cuda", dtype=torch.int64) + offsets = torch.randint(0, 2**20, (batch,), generator=g, device="cuda", dtype=torch.int64) + return dp, ids.contiguous(), tp, seeds, offsets + + +@pytest.mark.parametrize("case", ["allaccept", "lasttile", "dyadic", "zerotail"]) +@pytest.mark.parametrize("v", [200064, 32768 + 4, 4096 * 3 + 8]) +def test_v3_adversarial_rows_bitwise(case, v) -> None: + """REJ v3 equals the stock op on rows aimed at its moved steps, at every grid (1..64 CTAs per request), batch 1 / 2 + / 8, K 7.""" + _gpu() + if case == "dyadic" and v < 4 * 4096: + pytest.skip("the dyadic rows need 4 tiles") + for batch in (1, 2, 8): + for seed in range(8): + args = _v3_adversarial(case, batch, 7, v, 100 * seed + batch) + ref = _stock(*args) + for ctas in (1, 2, 8, 16, 32, 64): + if ctas > 1 and batch * ctas > entry.sm_count(args[0].device): + continue + _same( + op(*args, entry.prebuild(args[0].device), ctas), + ref, + f"{case} v={v} batch={batch} seed={seed} ctas={ctas}", + ) + + +def test_rej_v3_kill_switch_runs_v2_bitwise() -> None: + """TRTLLM_M31_DISABLE_REJ_V3=1 (read once by the lib): the op launches the v2 kernel, still bitwise the stock op.""" + _gpu() + probe = ( + "import sys, torch\n" + f"sys.path.insert(0, {os.path.dirname(os.path.abspath(__file__))!r})\n" + "import test_modeling_v2_m31_chain_spec_mc as t\n" + "ok = True\n" + "for case in t.CASES:\n" + " args = t._make(case, 2, 7, 200064, 77)\n" + " got = t.op(*args, t.entry.prebuild(args[0].device), 16)\n" + " ref = t._stock(*args)\n" + " ok &= bool(torch.equal(got[0], ref[0]) and torch.equal(got[1], ref[1]))\n" + "print(t.entry.kernel_version(), ok)" + ) + for extra, want in (({"TRTLLM_M31_DISABLE_REJ_V3": "1"}, "2 True"), ({}, "3 True")): + env = {kk: vv for kk, vv in os.environ.items() if kk != "TRTLLM_M31_DISABLE_REJ_V3"} + out = subprocess.run( + [sys.executable, "-c", probe], + env=dict(env, **extra), + capture_output=True, + text=True, + check=True, + ) + assert out.stdout.strip().splitlines()[-1] == want, ( + extra, + out.stdout[-300:], + out.stderr[-500:], + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_softmax_mc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_softmax_mc.py new file mode 100644 index 000000000000..c512ba36d224 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_softmax_mc.py @@ -0,0 +1,256 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for SOFTMAX_MC: the spec sampler's softmax as +``torch.ops.trtllm.minimax_m31_softmax_mc`` through ``ops/sampling/m31_softmax_mc``, installed as the sampler's softmax +override. + +It must equal ``flashinfer.sampling.softmax(logits, temperature)`` bit for bit (the microbench generators: randn / +peaked / ties / near_uniform / all_equal / masked / wide rows, temperature 1 / per-row mixed / scalar). The shared +sampler hook must be byte-for-byte stock while no override is set, and fall back to the stock call whenever the +override declines. TRTLLM_M31_DISABLE_SOFTMAX_MC=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.sampling import ( # noqa: E501 + m31_softmax_mc as entry, +) +from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as fi_ops + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8" +) +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal", "masked", "wide") +TEMPS = ("t1", "mixed", "scalar") +op = entry.m31_softmax_mc + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("SOFTMAX_MC is built for sm_107 here") + if entry.softmax_mc_op() is None: + pytest.skip("minimax_m31_softmax_mc is not built") + pytest.importorskip("flashinfer.sampling") + + +def _make(cls, rows, d, seed): + """The microbench logit generators.""" + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + return (torch.randn(rows, d, generator=g, device="cuda") * 3).contiguous() + if cls == "peaked": # few dominant tokens, long tail (real-like rows) + x = torch.randn(rows, d, generator=g, device="cuda") * 2 + idx = torch.randint(0, d, (rows, 8), generator=g, device="cuda") + x.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + return x.contiguous() + if cls == "ties": + return (torch.round(torch.randn(rows, d, generator=g, device="cuda") * 2) / 2).contiguous() + if cls == "near_uniform": + return (torch.randn(rows, d, generator=g, device="cuda") * 1e-3).contiguous() + if cls == "all_equal": + return torch.zeros(rows, d, device="cuda") + if cls == "masked": + x = torch.randn(rows, d, generator=g, device="cuda") * 3 + x[:, d // 2 :] = -1e4 + return x.contiguous() + if cls == "wide": + return (torch.randn(rows, d, generator=g, device="cuda") * 30).contiguous() + raise ValueError(cls) + + +def _temperature(mode, rows, seed): + """FlashInfer's temperature argument: an fp32 [rows] tensor, or a Python float.""" + g = torch.Generator(device="cuda").manual_seed(seed + 3) + if mode == "t1": + return torch.ones(rows, device="cuda") + if mode == "mixed": + return (0.3 + 1.2 * torch.rand(rows, generator=g, device="cuda")).contiguous() + if mode == "scalar": + return 0.7 + raise ValueError(mode) + + +def _fi(logits, temperature): + import flashinfer.sampling + + return flashinfer.sampling.softmax(logits, temperature) + + +def _same(a, b, what): + """Bit for bit.""" + assert torch.equal(a.view(torch.int32), b.view(torch.int32)), f"{what}: fp32 bits differ" + + +@pytest.mark.parametrize("mode", TEMPS) +@pytest.mark.parametrize("cls", CLASSES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_flashinfer_softmax(cell, cls, mode) -> None: + """At each certified (rows, d): the override equals FlashInfer's softmax bit for bit, and the explicit entry call + at write_ctas 4 / 8 gives the same bits again.""" + _gpu() + rows, d = cell.spec["rows"], cell.spec["d"] + for seed in range(2): + logits = _make(cls, rows, d, 100 * seed + rows) + temperature = _temperature(mode, rows, seed) + got = entry.softmax_mc(logits, temperature) + assert got is not None, "an in-contract input must be served" + _same(got, _fi(logits, temperature), f"{cls} {mode} rows={rows} seed={seed}") + for write_ctas in (4, 8): + again = op(logits, temperature, entry.prebuild(logits.device), write_ctas) + _same(again, got, f"write_ctas={write_ctas}") + + +def test_no_temperature_is_one() -> None: + """A None temperature (softmax_op's Optional) is FlashInfer's default, 1.0.""" + _gpu() + logits = _make("peaked", 8, 200064, 11) + got = entry.softmax_mc(logits, None) + assert got is not None + _same(got, _fi(logits, None), "temperature None") + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager bits.""" + _gpu() + logits = _make("peaked", 8, 200064, 5) + temperature = _temperature("mixed", 8, 5) + entry.prebuild(logits.device) + ref = entry.softmax_mc(logits, temperature) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.softmax_mc(logits, temperature) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.softmax_mc(logits, temperature) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """A non-fp32 / misaligned / non-contiguous logits, a d % 4 width, a vocab below FlashInfer's Path A, a mis-sized, + non-fp32 or 2-D temperature, a non-number temperature or more rows than MAX_ROWS return None (the sampler runs + stock FlashInfer).""" + _gpu() + d = 200064 + logits = _make("randn", 4, d, 1) + temperature = torch.full((4,), 0.9, device="cuda") + assert entry.softmax_mc(logits.half(), temperature) is None + assert entry.softmax_mc(logits, temperature.double()) is None + assert entry.softmax_mc(logits, temperature[:3]) is None + assert entry.softmax_mc(logits, temperature.view(4, 1)) is None + assert entry.softmax_mc(logits, "0.9") is None + assert entry.softmax_mc(logits[:, : d - 4], temperature) is None + assert entry.softmax_mc(logits[:, : d - 2].contiguous(), temperature) is None + assert ( + entry.softmax_mc(logits[:, : entry.PATH_A_MIN_VOCAB - 4].contiguous(), temperature) is None + ) + flat = torch.empty(4 * d + 1, device="cuda") + assert entry.softmax_mc(flat[1:].view(4, d), temperature) is None + wide = _make("randn", entry.MAX_ROWS + 1, d, 2) + assert entry.softmax_mc(wide, torch.full((entry.MAX_ROWS + 1,), 0.9, device="cuda")) is None + assert entry.softmax_mc(logits.cpu(), temperature.cpu()) is None + + +@pytest.fixture +def _fake_flashinfer(monkeypatch): + """The sampler ops module with a stand-in flashinfer whose calls are recorded, and no override afterwards.""" + stock = torch.full((2, 8), 0.125) + sampling = types.SimpleNamespace( + softmax=mock.Mock(return_value=stock), + top_p_renorm_probs=mock.Mock(side_effect=lambda probs, top_p: probs), + top_k_mask_logits=mock.Mock(side_effect=lambda logits, top_k: logits), + ) + monkeypatch.setattr(fi_ops, "flashinfer", types.SimpleNamespace(sampling=sampling)) + fi_ops.set_softmax_override(None) + fi_ops.set_top_p_renorm_override(None) + yield sampling, stock + fi_ops.set_softmax_override(None) + + +def test_hook_unset_is_stock(_fake_flashinfer) -> None: + """No override: both ops make exactly the stock flashinfer call and return its result.""" + sampling, stock = _fake_flashinfer + logits, temperature = torch.rand(2, 8), torch.ones(2) + assert fi_ops._SOFTMAX_OVERRIDE is None + assert fi_ops.softmax_op(logits, temperature) is stock + sampling.softmax.assert_called_once_with(logits, temperature, enable_pdl=mock.ANY) + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, None) is stock + assert sampling.softmax.call_count == 2 + + +def test_hook_override_result_is_used(_fake_flashinfer) -> None: + """An override's tensor replaces the stock call in both ops; the stock kernel does not run, and a top_p renorm + after it gets the override's probs.""" + sampling, _ = _fake_flashinfer + mine = torch.zeros(2, 8) + override = mock.Mock(return_value=mine) + fi_ops.set_softmax_override(override) + logits, temperature = torch.rand(2, 8), torch.ones(2) + assert fi_ops.softmax_op(logits, temperature) is mine + override.assert_called_once_with(logits, temperature) + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, None) is mine + top_p = torch.full((2,), 0.9) + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, top_p) is mine + sampling.top_p_renorm_probs.assert_called_once_with(mine, top_p) + assert override.call_count == 3 + sampling.softmax.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_flashinfer) -> None: + """An override that returns None leaves the stock call in place in both ops.""" + sampling, stock = _fake_flashinfer + override = mock.Mock(return_value=None) + fi_ops.set_softmax_override(override) + logits, temperature = torch.rand(2, 8), torch.ones(2) + assert fi_ops.softmax_op(logits, temperature) is stock + assert fi_ops.compute_probs_from_logits_op(logits, temperature, None, None) is stock + assert override.call_count == 2 and sampling.softmax.call_count == 2 + + +def test_the_target_installs_the_override() -> None: + """With SOFTMAX_MC on (the default) and the op built, the target installs ``softmax_mc`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, + ) + + if not modeling.SOFTMAX_MC: + pytest.skip("SOFTMAX_MC is switched off in this environment") + saved = dict(modeling._SOFTMAX_MC_STATE) + modeling._SOFTMAX_MC_STATE.clear() + try: + modeling._install_softmax_mc() + assert fi_ops._SOFTMAX_OVERRIDE is entry.softmax_mc + assert modeling._SOFTMAX_MC_STATE["installed"] is True + finally: + fi_ops.set_softmax_override(None) + modeling._SOFTMAX_MC_STATE.clear() + modeling._SOFTMAX_MC_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_SOFTMAX_MC=1 (read at import): SOFTMAX_MC is off and the target leaves the sampler hook + unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as f\n" + "m._install_softmax_mc()\n" + "print(m.SOFTMAX_MC, f._SOFTMAX_OVERRIDE is None, m._SOFTMAX_MC_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_SOFTMAX_MC="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det.py new file mode 100644 index 000000000000..5997699961d5 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det.py @@ -0,0 +1,271 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for TOPP_DET: the spec sampler's top-p renorm as +``torch.ops.trtllm.minimax_m31_topp_renorm_det`` through ``ops/sampling/m31_topp_det``, installed as the sampler's +top-p renorm override. + +On rows with top_p < 1 it must equal FlashInfer's ``top_p_renorm_probs(probs, top_p, is_deterministic=True)`` bit for +bit (the microbench generators: randn / peaked / ties / near_uniform / all_equal rows); rows with top_p >= 1 keep every +token (an approved exception). The shared sampler hook must be byte-for-byte stock while no override is set, and +fall back to the stock call whenever the override declines. TRTLLM_M31_DISABLE_TOPP_DET=1 installs nothing. +""" + +import os +import subprocess +import sys +import types +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.sampling import ( # noqa: E501 + m31_topp_det as entry, +) +from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as fi_ops + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8" +) +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal") +FLT_MIN = torch.finfo(torch.float32).tiny +op = entry.m31_topp_det + + +def _gpu(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("TOPP_DET is built for sm_107 here") + if entry.topp_det_op() is None: + pytest.skip("minimax_m31_topp_renorm_det is not built") + pytest.importorskip("flashinfer.sampling") + + +def _make(cls, rows, d, seed): + """The microbench row generators: softmax of the class's logits.""" + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + logits = torch.randn(rows, d, generator=g, device="cuda") * 3 + elif cls == "peaked": # few dominant tokens, long tail (real-like draft rows) + logits = torch.randn(rows, d, generator=g, device="cuda") * 2 + idx = torch.randint(0, d, (rows, 8), generator=g, device="cuda") + logits.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + elif cls == "ties": + logits = torch.round(torch.randn(rows, d, generator=g, device="cuda") * 2) / 2 + elif cls == "near_uniform": + logits = torch.randn(rows, d, generator=g, device="cuda") * 1e-3 + elif cls == "all_equal": + logits = torch.zeros(rows, d, device="cuda") + else: + raise ValueError(cls) + return torch.softmax(logits, -1).contiguous() + + +def _top_p(mode, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed + 7) + if mode == "p95": + return torch.full((rows,), 0.95, device="cuda") + if mode == "mixed": + return (0.05 + 0.95 * torch.rand(rows, generator=g, device="cuda")).contiguous() + raise ValueError(mode) + + +def _fi_det(probs, top_p): + import flashinfer.sampling + + return flashinfer.sampling.top_p_renorm_probs(probs, top_p, is_deterministic=True) + + +def _same(a, b, what): + """Bit for bit.""" + assert torch.equal(a.view(torch.int32), b.view(torch.int32)), f"{what}: fp32 bits differ" + + +@pytest.mark.parametrize("mode", ["p95", "mixed"]) +@pytest.mark.parametrize("cls", CLASSES) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:40] for c in op.CELLS]) +def test_bitwise_flashinfer_det_below_one(cell, cls, mode) -> None: + """Rows with top_p < 1 at each certified (rows, d): the override equals FlashInfer's det renorm bit for bit, and + the explicit entry call gives the same bits again (deterministic).""" + _gpu() + rows, d = cell.spec["rows"], cell.spec["d"] + for seed in range(2): + probs = _make(cls, rows, d, 1000 * rows + seed) + top_p = _top_p(mode, rows, seed) + got = entry.top_p_renorm_det(probs, top_p) + assert got is not None, "an in-contract input must be served" + ref = _fi_det(probs, top_p) + _same(got, ref, f"{cls} {mode} rows={rows} seed={seed}") + again = op( + probs, + top_p, + entry.prebuild(probs.device), + entry.ctas_per_row(rows, entry.sm_count(probs.device)), + ) + _same(again, got, "repeat") + + +@pytest.mark.parametrize("cls", CLASSES) +def test_keep_all_rows_in_a_mixed_batch(cls) -> None: + """A batch of 8 rows, a third at top_p 1: the top_p 1 rows keep every element whose renormed value is a normal float + (output nonzero <=> ftz(p * norm) != 0, the --use_fast_math semantics FlashInfer's apply shares) and sum to 1 within + 1e-5; the top_p < 1 rows are FlashInfer's det renorm bit for bit.""" + _gpu() + rows, d = 8, 200064 + probs = _make(cls, rows, d, 77) + top_p = _top_p("mixed", rows, 3) + keep = torch.zeros(rows, dtype=torch.bool, device="cuda") + keep[::3] = True + top_p[keep] = 1.0 + got = entry.top_p_renorm_det(probs, top_p) + assert got is not None + ref = _fi_det(probs, top_p) + _same(got[~keep], ref[~keep], f"{cls}: top_p < 1 rows") + for r in keep.nonzero().flatten().tolist(): + p64 = probs[r].double() + norm = 1.0 / float(p64[probs[r] >= FLT_MIN].sum()) + # kept <=> ftz(p * norm) != 0; elements within 1e-6 of the FLT_MIN boundary are left to the rounding + live = p64 * norm >= FLT_MIN * (1 + 1e-6) + dead = probs[r] < FLT_MIN + assert bool((got[r][live] != 0).all()), f"{cls} row {r}: a kept token was dropped" + assert bool((got[r][dead] == 0).all()), f"{cls} row {r}: a sub-FLT_MIN token is nonzero" + total = float(got[r].double().sum()) + assert abs(total - 1.0) <= 1e-5, f"{cls} row {r}: sum {total}" + + +def test_graph_replay_equals_eager() -> None: + """Captured in a CUDA graph (workspace prebuilt outside the capture), the replay gives the eager bits.""" + _gpu() + probs = _make("peaked", 8, 200064, 5) + top_p = _top_p("mixed", 8, 5) + entry.prebuild(probs.device) + ref = entry.top_p_renorm_det(probs, top_p) + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + entry.top_p_renorm_det(probs, top_p) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = entry.top_p_renorm_det(probs, top_p) + assert out is not None, "a prebuilt workspace must serve a captured call" + for _ in range(3): + out.fill_(float("nan")) + graph.replay() + torch.cuda.synchronize() + _same(out, ref, "graph replay") + + +def test_outside_the_contract_the_override_declines() -> None: + """A float top_p, a non-fp32 / misaligned / non-contiguous probs, a d % 4 width, a mis-sized top_p or more rows + than the workspace holds return None (the sampler runs stock FlashInfer).""" + _gpu() + d = 200064 + probs = _make("randn", 4, d, 1) + top_p = torch.full((4,), 0.9, device="cuda") + assert entry.top_p_renorm_det(probs, 0.9) is None + assert entry.top_p_renorm_det(probs.half(), top_p) is None + assert entry.top_p_renorm_det(probs, top_p.double()) is None + assert entry.top_p_renorm_det(probs, top_p[:3]) is None + assert entry.top_p_renorm_det(probs[:, : d - 2], top_p) is None + assert entry.top_p_renorm_det(probs[:, : d - 2].contiguous(), top_p) is None + flat = torch.empty(4 * d + 1, device="cuda") + assert entry.top_p_renorm_det(flat[1:].view(4, d), top_p) is None + wide = _make("randn", entry.MAX_ROWS + 1, d, 2) + assert ( + entry.top_p_renorm_det(wide, torch.full((entry.MAX_ROWS + 1,), 0.9, device="cuda")) is None + ) + assert entry.ctas_per_row(1, 148) == 16 and entry.ctas_per_row(8, 148) == 16 + assert entry.ctas_per_row(16, 148) == 8 and entry.ctas_per_row(64, 148) == 2 + assert entry.ctas_per_row(200, 148) == 1 + + +@pytest.fixture +def _fake_flashinfer(monkeypatch): + """The sampler ops module with a stand-in flashinfer whose calls are recorded, and no override afterwards.""" + stock = torch.full((2, 8), 0.125) + sampling = types.SimpleNamespace( + top_p_renorm_probs=mock.Mock(return_value=stock), + softmax=mock.Mock(side_effect=lambda logits, temps, enable_pdl=False: logits.softmax(-1)), + top_k_mask_logits=mock.Mock(side_effect=lambda logits, top_k: logits), + ) + monkeypatch.setattr(fi_ops, "flashinfer", types.SimpleNamespace(sampling=sampling)) + fi_ops.set_top_p_renorm_override(None) + yield sampling, stock + fi_ops.set_top_p_renorm_override(None) + + +def test_hook_unset_is_stock(_fake_flashinfer) -> None: + """No override: both ops make exactly the stock flashinfer call and return its result.""" + sampling, stock = _fake_flashinfer + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops._TOP_P_RENORM_OVERRIDE is None + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is stock + sampling.top_p_renorm_probs.assert_called_once_with(probs, top_p) + assert ( + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is stock + ) + assert sampling.top_p_renorm_probs.call_count == 2 + + +def test_hook_override_result_is_used(_fake_flashinfer) -> None: + """An override's tensor replaces the stock call in both ops; the stock kernel does not run.""" + sampling, _ = _fake_flashinfer + mine = torch.zeros(2, 8) + override = mock.Mock(return_value=mine) + fi_ops.set_top_p_renorm_override(override) + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is mine + override.assert_called_once_with(probs, top_p) + assert fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is mine + assert override.call_count == 2 + sampling.top_p_renorm_probs.assert_not_called() + + +def test_hook_override_declining_falls_back(_fake_flashinfer) -> None: + """An override that returns None leaves the stock call in place; without top_p the override is never asked.""" + sampling, stock = _fake_flashinfer + override = mock.Mock(return_value=None) + fi_ops.set_top_p_renorm_override(override) + probs, top_p = torch.rand(2, 8), torch.full((2,), 0.9) + assert fi_ops.top_p_renorm_probs_op(probs, top_p) is stock + assert ( + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, top_p) is stock + ) + assert override.call_count == 2 and sampling.top_p_renorm_probs.call_count == 2 + fi_ops.compute_probs_from_logits_op(torch.rand(2, 8), torch.ones(2), None, None) + assert override.call_count == 2 + + +def test_the_target_installs_the_override() -> None: + """With TOPP_DET on (the default) and the op built, the target installs ``top_p_renorm_det`` once per process.""" + _gpu() + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, + ) + + if not modeling.TOPP_DET: + pytest.skip("TOPP_DET is switched off in this environment") + saved = dict(modeling._TOPP_DET_STATE) + modeling._TOPP_DET_STATE.clear() + try: + modeling._install_topp_det() + assert fi_ops._TOP_P_RENORM_OVERRIDE is entry.top_p_renorm_det + assert modeling._TOPP_DET_STATE["installed"] is True + finally: + fi_ops.set_top_p_renorm_override(None) + modeling._TOPP_DET_STATE.clear() + modeling._TOPP_DET_STATE.update(saved) + + +def test_the_kill_switch_installs_nothing() -> None: + """TRTLLM_M31_DISABLE_TOPP_DET=1 (read at import): TOPP_DET is off and the target leaves the sampler hook unset.""" + probe = ( + f"import {TARGET}.modeling as m\n" + "from tensorrt_llm._torch.pyexecutor.sampler.ops import flashinfer as f\n" + "m._install_topp_det()\n" + "print(m.TOPP_DET, f._TOP_P_RENORM_OVERRIDE is None, m._TOPP_DET_STATE['installed'])" + ) + env = dict(os.environ, TRTLLM_M31_DISABLE_TOPP_DET="1") + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == "False True False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det_v3.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det_v3.py new file mode 100644 index 000000000000..1d13172219a8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det_v3.py @@ -0,0 +1,249 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for TOPP_DET v3 (``torch.ops.trtllm.minimax_m31_topp_renorm_det_v3``): its output must equal v1.2's +(``minimax_m31_topp_renorm_det``) bit for bit on every row (top_p < 1 and the top_p >= 1 keep-all rows), with and +without the programmatic dependent launch, across back-to-back launches and CUDA-graph replays that reuse one +workspace; ``ops/sampling/m31_topp_det`` takes v3 when it is built unless TRTLLM_M31_DISABLE_TOPPDET_V3 is set. +""" + +import os +import subprocess +import sys + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.sampling import ( # noqa: E501 + m31_topp_det as entry, +) + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8" +) +D = 200064 +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal", "one_hot", "tiny_tail") + + +def _ops(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("this TOPP_DET target is sm_107") + # The entry loads only the op it engages; load both (in-tree or from tensorrt_llm/libs) for the comparison. + v12 = entry._load_op("minimax_m31_topp_renorm_det") + v3 = entry._load_op("minimax_m31_topp_renorm_det_v3") + if v12 is None or v3 is None: + pytest.skip("needs both minimax_m31_topp_renorm_det and minimax_m31_topp_renorm_det_v3") + return v12, v3 + + +def _make(cls, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + logits = torch.randn(rows, D, generator=g, device="cuda") * 3 + elif cls == "peaked": + logits = torch.randn(rows, D, generator=g, device="cuda") * 2 + idx = torch.randint(0, D, (rows, 8), generator=g, device="cuda") + logits.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + elif cls == "ties": + logits = torch.round(torch.randn(rows, D, generator=g, device="cuda") * 2) / 2 + elif cls == "near_uniform": + logits = torch.randn(rows, D, generator=g, device="cuda") * 1e-3 + elif cls == "all_equal": + logits = torch.zeros(rows, D, device="cuda") + elif cls == "one_hot": + logits = torch.full((rows, D), -1e4, device="cuda") + logits[torch.arange(rows), torch.randint(0, D, (rows,), generator=g, device="cuda")] = 0 + elif cls == "tiny_tail": + logits = torch.randn(rows, D, generator=g, device="cuda") * 30 + else: + raise ValueError(cls) + return torch.softmax(logits, -1).contiguous() + + +def _top_p(mode, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed + 7) + if mode == "p95": + return torch.full((rows,), 0.95, device="cuda") + if mode == "mixed": + return (0.05 + 0.95 * torch.rand(rows, generator=g, device="cuda")).contiguous() + tp = torch.full((rows,), 0.95, device="cuda") + tp[seed % 2 :: 2] = 1.0 # with_p1: keep-all rows interleaved + return tp + + +class _Runner: + """v1.2 and v3 on separate zeroed workspaces sized for 8 rows.""" + + def __init__(self, v12, v3): + self.v12, self.v3 = v12, v3 + self.ws12 = torch.zeros( + 8 * int(torch.ops.trtllm.minimax_m31_topp_renorm_det_ws_bytes()), + dtype=torch.uint8, + device="cuda", + ) + self.ws3 = torch.zeros( + 8 * int(torch.ops.trtllm.minimax_m31_topp_renorm_det_v3_ws_bytes()), + dtype=torch.uint8, + device="cuda", + ) + self.sms = torch.cuda.get_device_properties(0).multi_processor_count + + def g(self, rows): + return entry.ctas_per_row(rows, self.sms) + + def ref(self, probs, tp): + out = torch.empty_like(probs) + self.v12(probs, tp, self.ws12, out, self.g(probs.shape[0])) + return out + + def new(self, probs, tp, pdl, out=None, cand=True): + out = torch.empty_like(probs) if out is None else out + self.v3(probs, tp, self.ws3, out, self.g(probs.shape[0]), pdl, cand) + return out + + +def _bad_rows(a, b): + return int((a.view(torch.int32) != b.view(torch.int32)).any(1).sum()) + + +@pytest.mark.parametrize("cand", [False, True]) +@pytest.mark.parametrize("pdl", [False, True]) +@pytest.mark.parametrize("rows", [1, 7, 8]) +def test_v3_bitwise_v12(rows, pdl, cand): + r = _Runner(*_ops()) + bad = 0 + for cls in CLASSES: + for mode in ("p95", "mixed", "with_p1"): + for seed in range(4): + probs, tp = _make(cls, rows, 100 * seed + rows), _top_p(mode, rows, seed) + bad += _bad_rows(r.new(probs, tp, pdl, cand=cand), r.ref(probs, tp)) + assert bad == 0 + + +def test_v3_back_to_back_graph_replays(): + """100 PDL launches captured back to back on one workspace, replayed twice with fresh inputs copied in.""" + r = _Runner(*_ops()) + rows = 7 + probs = _make("peaked", rows, 1).clone() + tp = _top_p("mixed", rows, 1).clone() + outs = [torch.empty_like(probs) for _ in range(100)] + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + r.new(probs, tp, True, outs[0]) + torch.cuda.current_stream().wait_stream(s) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + for o in outs: + r.new(probs, tp, True, o) + for k, cls in enumerate(("randn", "near_uniform")): + probs.copy_(_make(cls, rows, 50 + k)) + tp.copy_(_top_p("with_p1", rows, k)) + graph.replay() + torch.cuda.synchronize() + ref = r.ref(probs, tp) + assert sum(_bad_rows(o, ref) for o in outs) == 0 + + +def test_v3_alternating_grid_one_workspace(): + """60 launches on one workspace alternating rows (1 / 7 / 8), ctas_per_row (16 / 8 / 4), PDL and the candidate + list: the barrier counters and the list must stay consistent; every output bitwise v1.2's at the same rows.""" + r = _Runner(*_ops()) + bad = 0 + for i in range(60): + rows = (1, 7, 8)[i % 3] + g = (16, 8, 4)[(i // 3) % 3] + probs = _make(CLASSES[i % len(CLASSES)], rows, 300 + i) + tp = _top_p(("p95", "mixed", "with_p1")[i % 3], rows, i) + out = torch.empty_like(probs) + r.v3(probs, tp, r.ws3, out, g, bool(i % 2), bool((i // 2) % 2)) + bad += _bad_rows(out, r.ref(probs, tp)) + assert bad == 0 + + +def _cap_row(members, seed): + """A probability row whose top-p cut lands mid-way into a pass-0 radix bucket (the top 11 key bits) of exactly + ``members`` elements: 10 head elements of 0.05, ``members`` values in [2^-14, 1.15 * 2^-14) (one bucket), and a + tiny tail in other buckets; returns (row, top_p).""" + g = torch.Generator(device="cuda").manual_seed(seed) + row = torch.full((D,), 1.0e-6, device="cuda") + pos = torch.randperm(D, generator=g, device="cuda") + row[pos[:10]] = 0.05 + c = 2.0**-14 + row[pos[10 : 10 + members]] = c * (1 + 0.15 * torch.arange(members, device="cuda") / members) + bucket_mass = float(row[pos[10 : 10 + members]].sum()) + return row, 0.5 + 0.5 * bucket_mass + + +def _cut_bucket_size(row, top_p): + vals, _ = torch.sort(row, descending=True) + cut = int( + torch.searchsorted( + torch.cumsum(vals.double(), 0), torch.tensor(top_p, dtype=torch.float64, device="cuda") + ) + ) + keys = row.view(torch.int32) >> 21 + return int((keys == (vals[cut : cut + 1].view(torch.int32) >> 21)).sum()) + + +@pytest.mark.parametrize("members", [4095, 4096, 4097]) +def test_t3_candidate_cap_boundary(members): + """The candidate list holds a pass-0 bucket of at most 4096: rows whose cut bucket has 4095 / 4096 (list) and 4097 + (fallback to the full passes) members, mixed with randn rows, must all be bitwise v1.2's, list on and off.""" + r = _Runner(*_ops()) + rows = 8 + probs = _make("randn", rows, 900 + members).clone() + tp = _top_p("p95", rows, members).clone() + for i in (0, 3, 7): + row, p = _cap_row(members, 10 * members + i) + assert _cut_bucket_size(row, p) == members # the construction's precondition + probs[i], tp[i] = row, p + ref = r.ref(probs, tp) + for pdl in (False, True): + for cand in (False, True): + assert _bad_rows(r.new(probs, tp, pdl, cand=cand), ref) == 0 + + +def _t3_line(env): + code = f"import importlib; m = importlib.import_module('{TARGET}.ops.sampling.m31_topp_det'); m.topp_det_op()" + out = subprocess.run( + [sys.executable, "-c", code], + env={**os.environ, "TLLM_LOG_LEVEL": "INFO", **env}, + capture_output=True, + text=True, + check=True, + ) + lines = [ + ln for ln in (out.stdout + out.stderr).splitlines() if "TOPP_DET v3 T3 candidate-list" in ln + ] + return lines + + +def test_t3_engagement_line(): + """Rule e: one line per process naming the value actually used.""" + _ops() + on = _t3_line({"TRTLLM_M31_DISABLE_TOPPDET_T3": "0"}) + off = _t3_line({"TRTLLM_M31_DISABLE_TOPPDET_T3": "1"}) + assert len(on) == 1 and "candidate-list fused passes: on (cap 4096)" in on[0] + assert len(off) == 1 and "candidate-list fused passes: off (cap 4096)" in off[0] + + +def _engaged(env): + code = ( + f"import importlib; m = importlib.import_module('{TARGET}.ops.sampling.m31_topp_det'); " + "m.topp_det_op(); print('V3' if m.is_v3() else 'V12')" + ) + out = subprocess.run( + [sys.executable, "-c", code], + env={**os.environ, **env}, + capture_output=True, + text=True, + check=True, + ) + return out.stdout.strip().splitlines()[-1] + + +def test_switch_precedence(): + _ops() # both ops present in this process: the subprocesses see the same build / libs + assert _engaged({"TRTLLM_M31_DISABLE_TOPPDET_V3": "0"}) == "V3" + assert _engaged({"TRTLLM_M31_DISABLE_TOPPDET_V3": "1"}) == "V12" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det_v4k8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det_v4k8.py new file mode 100644 index 000000000000..6b74c9fb3b60 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/sampling/test_modeling_v2_m31_topp_det_v4k8.py @@ -0,0 +1,211 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Tests for TOPP_DET V4K8 (``minimax_m31_topp_renorm_det_v3`` with ``list_path`` / ``wide_hist``): every switch +combination's output must equal v1.2's (``minimax_m31_topp_renorm_det``) bit for bit, including rows whose >= 2^-12 +list overflows (unnormalized rows), thresholds on either side of the 2^-12 bucket edge, keep-all rows, mixed +combinations on one workspace and CUDA-graph replays; ``ops/sampling/m31_topp_det`` logs the values it uses and +honours TRTLLM_M31_DISABLE_TOPPDET_V4 / _K8. +""" + +import os +import subprocess +import sys + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.sampling import ( # noqa: E501 + m31_topp_det as entry, +) + +TARGET = ( + "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8" +) +D = 200064 +EDGE = 2.0**-12 +CLASSES = ("randn", "peaked", "ties", "near_uniform", "all_equal", "one_hot", "tiny_tail") +COMBOS = [(True, True), (True, False), (False, True), (False, False)] + + +def _ops(): + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("this TOPP_DET target is sm_107") + v12 = entry._load_op("minimax_m31_topp_renorm_det") + v3 = entry._load_op("minimax_m31_topp_renorm_det_v3") + if v12 is None or v3 is None or "list_path" not in str(v3.default._schema): + pytest.skip("needs minimax_m31_topp_renorm_det and a V4K8 minimax_m31_topp_renorm_det_v3") + return v12, v3 + + +def _make(cls, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + if cls == "randn": + logits = torch.randn(rows, D, generator=g, device="cuda") * 3 + elif cls == "peaked": + logits = torch.randn(rows, D, generator=g, device="cuda") * 2 + idx = torch.randint(0, D, (rows, 8), generator=g, device="cuda") + logits.scatter_(1, idx, 14 + 3 * torch.rand(rows, 8, generator=g, device="cuda")) + elif cls == "ties": + logits = torch.round(torch.randn(rows, D, generator=g, device="cuda") * 2) / 2 + elif cls == "near_uniform": + logits = torch.randn(rows, D, generator=g, device="cuda") * 1e-3 + elif cls == "all_equal": + logits = torch.zeros(rows, D, device="cuda") + elif cls == "one_hot": + logits = torch.full((rows, D), -1e4, device="cuda") + logits[torch.arange(rows), torch.randint(0, D, (rows,), generator=g, device="cuda")] = 0 + elif cls == "tiny_tail": + logits = torch.randn(rows, D, generator=g, device="cuda") * 30 + else: + raise ValueError(cls) + return torch.softmax(logits, -1).contiguous() + + +def _top_p(mode, rows, seed): + g = torch.Generator(device="cuda").manual_seed(seed + 7) + if mode == "p95": + return torch.full((rows,), 0.95, device="cuda") + if mode == "mixed": + return (0.05 + 0.95 * torch.rand(rows, generator=g, device="cuda")).contiguous() + tp = torch.full((rows,), 0.95, device="cuda") + tp[seed % 2 :: 2] = 1.0 # with_p1: keep-all rows interleaved + return tp + + +class _Runner: + def __init__(self, v12, v3): + self.v12, self.v3 = v12, v3 + n12 = int(torch.ops.trtllm.minimax_m31_topp_renorm_det_ws_bytes()) + n3 = int(torch.ops.trtllm.minimax_m31_topp_renorm_det_v3_ws_bytes()) + self.ws12 = torch.zeros(8 * n12, dtype=torch.uint8, device="cuda") + self.ws3 = torch.zeros(8 * n3, dtype=torch.uint8, device="cuda") + self.sms = torch.cuda.get_device_properties(0).multi_processor_count + + def g(self, rows): + return entry.ctas_per_row(rows, self.sms) + + def ref(self, probs, tp): + out = torch.empty_like(probs) + self.v12(probs, tp, self.ws12, out, self.g(probs.shape[0])) + return out + + def new(self, probs, tp, pdl, combo, out=None): + out = torch.empty_like(probs) if out is None else out + self.v3(probs, tp, self.ws3, out, self.g(probs.shape[0]), pdl, True, combo[0], combo[1]) + return out + + +def _bad_rows(a, b): + return int((a.view(torch.int32) != b.view(torch.int32)).any(1).sum()) + + +@pytest.mark.parametrize("combo", COMBOS) +@pytest.mark.parametrize("pdl", [False, True]) +@pytest.mark.parametrize("rows", [1, 7, 8]) +def test_v4k8_bitwise_v12(rows, pdl, combo): + r = _Runner(*_ops()) + bad = 0 + for cls in CLASSES: + for mode in ("p95", "mixed", "with_p1"): + for seed in range(3): + probs, tp = _make(cls, rows, 100 * seed + rows), _top_p(mode, rows, seed) + bad += _bad_rows(r.new(probs, tp, pdl, combo), r.ref(probs, tp)) + assert bad == 0 + + +@pytest.mark.parametrize("listed", [4095, 4096, 4097, 6000]) +def test_v4k8_list_overflow_unnormalized(listed): + """Rows (not summing to 1) with `listed` elements >= 2^-12 and the threshold above the edge: at most 4096 fit the + list, the rest take the radix fallback; both bitwise v1.2.""" + r = _Runner(*_ops()) + g = torch.Generator(device="cuda").manual_seed(listed) + probs = torch.rand(8, D, generator=g, device="cuda") * 1e-6 + for i in range(8): + idx = torch.randperm(D, generator=g, device="cuda")[:listed] + probs[i, idx] = EDGE * (1.5 + torch.rand(listed, generator=g, device="cuda")) + tp = torch.full((8,), 0.9, device="cuda") + ref = r.ref(probs, tp) + for combo in COMBOS: + assert _bad_rows(r.new(probs, tp, True, combo), ref) == 0 + + +@pytest.mark.parametrize("side", ["at_edge", "below_edge"]) +def test_v4k8_threshold_at_the_edge(side): + """The kept set ends inside the [2^-12, ...) bucket (list path) or just below 2^-12 (fallback).""" + r = _Runner(*_ops()) + g = torch.Generator(device="cuda").manual_seed(3) + probs = torch.rand(8, D, generator=g, device="cuda") * 1e-7 + v = EDGE if side == "at_edge" else EDGE * 0.999 + for i in range(8): + idx = torch.randperm(D, generator=g, device="cuda")[:3000] + probs[i, idx] = v + probs[i, idx[:4]] = 0.05 + probs = probs.contiguous() + tp = torch.full((8,), 0.6, device="cuda") + ref = r.ref(probs, tp) + for combo in COMBOS: + assert _bad_rows(r.new(probs, tp, True, combo), ref) == 0 + + +def test_v4k8_mixed_combos_one_workspace(): + """64 launches on one workspace cycling the four combinations, keep-all rows and row counts (the pass-0 parity + buffers and list counters must stay consistent across modes).""" + r = _Runner(*_ops()) + for i in range(64): + rows = (1, 7, 8)[i % 3] + probs = _make(CLASSES[i % len(CLASSES)], rows, 900 + i) + tp = _top_p(("p95", "mixed", "with_p1")[i % 3], rows, i) + assert ( + _bad_rows(r.new(probs, tp, bool(i % 2), COMBOS[(i // 2) % 4]), r.ref(probs, tp)) == 0 + ), i + + +def test_v4k8_graph_replays(): + """48 launches (all combinations) captured back to back on one workspace, replayed with fresh inputs copied in.""" + r = _Runner(*_ops()) + rows = 8 + probs = _make("peaked", rows, 1).clone() + tp = _top_p("mixed", rows, 1).clone() + outs = [torch.empty_like(probs) for _ in range(48)] + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + for i, o in enumerate(outs): + r.new(probs, tp, True, COMBOS[i % 4], out=o) + torch.cuda.current_stream().wait_stream(s) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + for i, o in enumerate(outs): + r.new(probs, tp, True, COMBOS[i % 4], out=o) + for k, cls in enumerate(("randn", "peaked", "one_hot")): + probs.copy_(_make(cls, rows, 50 + k)) + tp.copy_(_top_p("with_p1" if k == 2 else "p95", rows, k)) + graph.replay() + torch.cuda.synchronize() + ref = r.ref(probs, tp) + assert all(_bad_rows(o, ref) == 0 for o in outs), cls + + +def _v4k8_line(env): + code = f"import importlib; m = importlib.import_module('{TARGET}.ops.sampling.m31_topp_det'); m.topp_det_op()" + out = subprocess.run( + [sys.executable, "-c", code], + env={**os.environ, "TLLM_LOG_LEVEL": "INFO", **env}, + capture_output=True, + text=True, + check=True, + ) + return [ln for ln in (out.stdout + out.stderr).splitlines() if "TOPP_DET V4K8:" in ln] + + +def test_v4k8_engagement_and_switches(): + """Rule e: one line per process naming the values actually used; each kill switch turns only its item off.""" + _ops() + base = {"TRTLLM_M31_DISABLE_TOPPDET_V4": "0", "TRTLLM_M31_DISABLE_TOPPDET_K8": "0"} + on = _v4k8_line(base) + no_list = _v4k8_line({**base, "TRTLLM_M31_DISABLE_TOPPDET_V4": "1"}) + no_k8 = _v4k8_line({**base, "TRTLLM_M31_DISABLE_TOPPDET_K8": "1"}) + assert len(on) == 1 and "list path on, 8 pass histograms on" in on[0] + assert len(no_list) == 1 and "list path off, 8 pass histograms on" in no_list[0] + assert len(no_k8) == 1 and "list path on, 8 pass histograms off" in no_k8[0] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/_m3_spec_inputs.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/_m3_spec_inputs.py new file mode 100644 index 000000000000..2e2c1968edea --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/_m3_spec_inputs.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Inputs shared by the tp4 target's spec entry tests: TP4 dimensions, single-device stand-ins for the cross-rank +exchanges, and the comparison bands.""" + +from __future__ import annotations + +import zlib + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_shard_accept import ( # noqa: E501 + SHARD_ROWS, + make_keys, +) + +#: MiniMax-M3 at TP8: hidden size, vocabulary, the target head's shard and RMSNorm epsilon. +HIDDEN = 6144 +VOCAB = 200064 +WORLD = 8 +TARGET_SHARD = VOCAB // WORLD +EPS = 1e-6 + + +def seed_of(why: str) -> int: + return zlib.crc32(why.encode()) % 2**31 + + +def gen(seed: int) -> torch.Generator: + return torch.Generator(device="cuda").manual_seed(seed) + + +def num_sms() -> int: + return torch.cuda.get_device_properties(torch.cuda.current_device()).multi_processor_count + + +def randn(shape, g: torch.Generator, std: float = 1.0, dtype=torch.bfloat16) -> torch.Tensor: + return (torch.randn(shape, generator=g, device="cuda") * std).to(dtype) + + +def head_workspace(world: int, call: int = 0, slot_rows: int = SHARD_ROWS): + """(ws_uc int64 [2 * slot_rows * world], keys int64 [slot_rows], counters int32 [2]) of a fresh call ``call``.""" + uc = torch.zeros(2 * slot_rows * world, dtype=torch.int64, device="cuda") + keys = torch.zeros(slot_rows, dtype=torch.int64, device="cuda") + counters = torch.tensor([0, call], dtype=torch.int32, device="cuda") + return uc, keys, counters + + +def plant_peers( + uc: torch.Tensor, + counters: torch.Tensor, + own_best: torch.Tensor, + rank: int, + world: int, + shard: int, + g: torch.Generator, + slot_rows: int = SHARD_ROWS, + ties: bool = False, +) -> None: + """Write the other ranks' keys for each row into this rank's slots, as their multicast pushes would. + + Each peer's maximum is the own row maximum shifted by +-0.25 (well clear of the accumulation-order band), so + some rows are won by a peer and some by this rank; with ``ties`` a peer on an even row reports exactly the own + maximum, and the lower global index must win. + """ + if world == 1: + return + m = own_best.shape[0] + half = int(counters[1].item()) & 1 + for r in range(world): + if r == rank: + continue + sign = torch.where(torch.rand(m, generator=g, device="cuda") < 0.5, -1.0, 1.0) + value = own_best.float() + 0.25 * sign + if ties: + value = torch.where(torch.arange(m, device="cuda") % 2 == 0, own_best.float(), value) + index = r * shard + torch.randint(0, shard, (m,), generator=g, device="cuda") + keys = make_keys(value, index) + for t in range(m): + uc[(half * slot_rows + t) * world + r] = keys[t] + + +def assert_bf16_close(out: torch.Tensor, ref: torch.Tensor, max_frac: float = 1e-3) -> None: + """At most one bf16 ulp per element, and only on a ``max_frac`` share of the elements. + + Kernel and reference compute the same fp32 expression and round once; they differ only where a different + summation order (or rsqrtf) puts the fp32 value on the other side of a bf16 rounding boundary. + """ + assert out.shape == ref.shape and out.dtype == ref.dtype == torch.bfloat16 + o, r = out.float(), ref.float() + ulp = torch.ldexp(torch.ones_like(r), torch.frexp(r).exponent - 8) + ulp = torch.where(r == 0, torch.full_like(r, 2.0**-133), ulp) + diff = (o - r).abs() + assert bool((diff <= ulp * 1.0001).all()), f"max |diff| / ulp = {(diff / ulp).max().item():.2f}" + frac = (diff > 0).float().mean().item() + assert frac <= max_frac, f"{frac:.2e} of the elements moved by one ulp (allowed {max_frac})" + + +def craft_drafts(tokens: torch.Tensor, nreq: int, d: int) -> torch.Tensor: + """Drafts matching request b's verify tokens for its first (b + 2) % (d + 1) positions and differing from + there on, so the acceptance counts vary across requests.""" + t = tokens.view(nreq, d + 1)[:, :d].clone() + for b in range(nreq): + n = (b + 2) % (d + 1) + t[b, n:] += 1 + return t.contiguous() diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dsfold.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dsfold.py new file mode 100644 index 000000000000..4aa2687ab2e2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dsfold.py @@ -0,0 +1,168 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""DSFOLD: under the standalone DSpark worker the TP4 / TP8 targets fold the engine's previous-batch input gathers into +one ``step_gather`` launch (``M3DSparkStepFold``). Oracle: the decoder runner's own eager gathers +(engine/runners/decoder/inputs.py, the ``previous_batch_len > 0 and not inputs_staged`` branch) on the same inputs; +every destination buffer must match bit for bit, including the rows the gathers do not own.""" + +import importlib +import inspect +import os +import subprocess +import sys +import zlib + +import pytest +import torch + +assert torch.cuda.is_available(), "DSFOLD requires a CUDA device" + +_TARGETS = ("minimax_m31_nvfp4__sm_107__tp8", "minimax_m31_nvfp4__sm_107__tp4") +_PKG = "tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3" + +#: Engine buffers sized for the max batch (8 requests x up to 1 + 7 tokens) and the stale fill they start with. +MAX_BATCH = 8 +MAX_TOKENS = MAX_BATCH * 8 +STALE = -4 +VOCAB = 200064 + + +def _modeling(target): + return importlib.import_module(f"{_PKG}.{target}.modeling") + + +def _engine_eager( + new_tokens, next_draft, lens, slots, ppi, P, T, D, ids, ids0, draft, d0, pos_off, kv_off, b0 +): + """The decoder runner's gathers for P previous requests, transcribed from inputs.py (offsets zeroed first).""" + pos_off.zero_() + kv_off.zero_() + s = slots[:P] + ids[ids0 : ids0 + P * T].copy_(new_tokens.transpose(0, 1)[s, :T].flatten()) + if D > 0: + draft[d0 : d0 + P * D].copy_(next_draft[s, :D].flatten()) + kv_len_offsets = lens - T + pos_off[b0 * T : (b0 + P) * T].copy_(lens[ppi[: P * T]]) + kv_off[b0 : b0 + P].copy_(kv_len_offsets[s]) + + +def _case(k, store_k, prev, new_before, dummies, seed): + """K drafts per step (T = K + 1 tokens) read from a store sized for ``store_k`` drafts; ``prev`` previous requests + in shuffled slots after ``new_before`` requests without a previous tensor, then ``dummies`` CUDA-graph padding + rows. Slots include the padding dummy's slot (the store holds MAX_BATCH + 1 slots).""" + g = torch.Generator(device="cuda").manual_seed(seed) + i32 = dict(dtype=torch.int32, device="cuda") + T, D, nslots = k + 1, k, MAX_BATCH + 1 + assert new_before + prev + dummies <= MAX_BATCH + new_tokens = torch.randint(0, VOCAB, (store_k + 1, nslots, 1), generator=g, **i32) + next_draft = torch.randint(0, VOCAB, (nslots, store_k), generator=g, **i32) + lens = torch.randint(1, T + 1, (nslots,), generator=g, **i32) + slots = torch.randperm(nslots, generator=g, device="cuda")[:prev].to(torch.int32) + ppi = slots.repeat_interleave(T).contiguous() + ids0, d0 = new_before * T, new_before * D + bufs = dict( + ids=torch.full((MAX_TOKENS,), STALE, **i32), + draft=torch.full((MAX_TOKENS,), STALE, **i32), + pos_off=torch.full((MAX_TOKENS,), STALE, **i32), + kv_off=torch.full((MAX_BATCH,), STALE, **i32), + ) + return new_tokens, next_draft, lens, slots, ppi, prev, T, D, ids0, d0, new_before, bufs + + +def _run(fold_or_none, case): + new_tokens, next_draft, lens, slots, ppi, P, T, D, ids0, d0, b0, bufs = case + b = {n: t.clone() for n, t in bufs.items()} + if fold_or_none is None: + _engine_eager( + new_tokens, + next_draft, + lens, + slots, + ppi, + P, + T, + D, + b["ids"], + ids0, + b["draft"], + d0, + b["pos_off"], + b["kv_off"], + b0, + ) + else: + fold_or_none.bind_engine_buffers(b["ids"], b["draft"], b["pos_off"], b["kv_off"], MAX_BATCH) + # The engine's call (inputs.py): first_previous_row = requests without a previous tensor before them. + assert fold_or_none.gather(new_tokens, next_draft, lens, slots, ppi, P, T, D, ids0, d0, b0) + torch.cuda.synchronize() + return b + + +# (K, store K, previous requests, new requests before them, padding dummies after them) +CELLS = [ + (3, 3, 1, 0, 0), # c1, K3: one steady request + (3, 3, 1, 0, 1), # c1 padded to the batch-2 graph + (3, 3, 8, 0, 0), # c8 steady, shuffled slots + (3, 3, 5, 2, 1), # mixed: 2 new rows before, 1 dummy after + (3, 7, 6, 1, 1), # K3 run from a store sized for K7 (dynamic draft width) + (7, 7, 1, 0, 0), # K7 c1 + (7, 7, 8, 0, 0), # K7 c8 + (7, 7, 4, 3, 1), # K7 mixed + dummy +] + + +@pytest.mark.parametrize("target", _TARGETS) +@pytest.mark.parametrize( + "cell", CELLS, ids=[f"K{c[0]}_store{c[1]}_p{c[2]}_new{c[3]}_dummy{c[4]}" for c in CELLS] +) +def test_fold_matches_engine_gathers(target, cell): + case = _case(*cell, seed=zlib.crc32(f"{target}{cell}".encode()) % 2**31) + ref = _run(None, case) + fold = _modeling(target).M3DSparkStepFold() + got = _run(fold, case) + for name in ("ids", "draft", "pos_off", "kv_off"): + assert torch.equal(got[name], ref[name]), f"{target} {cell}: {name}" + assert (fold.taken, fold.declined) == (1, 0) + + +@pytest.mark.parametrize("target", _TARGETS) +def test_fold_declines_and_engine_copies_run(target): + """A layout the kernel does not take (int64 slots) is declined with nothing written: the engine then gathers.""" + case = list(_case(3, 3, 4, 0, 0, seed=7)) + case[3] = case[3].long() + new_tokens, next_draft, lens, slots, ppi, P, T, D, ids0, d0, b0, bufs = case + fold = _modeling(target).M3DSparkStepFold() + b = {n: t.clone() for n, t in bufs.items()} + fold.bind_engine_buffers(b["ids"], b["draft"], b["pos_off"], b["kv_off"], MAX_BATCH) + assert not fold.gather(new_tokens, next_draft, lens, slots, ppi, P, T, D, ids0, d0, b0) + assert all(torch.equal(b[n], bufs[n]) for n in bufs) + assert (fold.taken, fold.declined) == (0, 1) + + +@pytest.mark.parametrize("target", _TARGETS) +def test_scatter_stays_in_tree(target): + """The sampler's slot stores are never folded under DSpark (the store path stays in-tree).""" + fold = _modeling(target).M3DSparkStepFold() + fold.bind_sampler_store(object()) + assert fold.scatter(None, None, None, None, None) is False + assert fold.begin_step([0], 1) is False + assert fold.sampler_rows_stored({}, 0, [0]) is False + + +@pytest.mark.parametrize("target", _TARGETS) +def test_kill_switch(target): + """TRTLLM_M31_DISABLE_DSFOLD=1 (read once at import): DSFOLD is off, so a DSpark run keeps spec_fold None and the + engine's own gathers; the Eagle3 path still builds M3SpecStepFold.""" + probe = f"import {_PKG}.{target}.modeling as m; print(m.DSFOLD)" + for value, want in (("1", "False"), ("0", "True")): + env = dict(os.environ, TRTLLM_M31_DISABLE_DSFOLD=value) + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == want + source = inspect.getsource(_modeling(target)) + # The sm_107 targets wire the fold differently (TP8: self._spec_fold; TP4: behind STEP_SPEC_FOLD), so check the + # DSpark branch rather than one exact text: DSFOLD gates M3DSparkStepFold, the Eagle3 path keeps M3SpecStepFold. + assert ( + "elif DSFOLD" in source and "M3DSparkStepFold()" in source and "M3SpecStepFold(" in source + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_n3a.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_n3a.py new file mode 100644 index 000000000000..573b04c622a6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_n3a.py @@ -0,0 +1,473 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""N3 A2 in this target's DSpark worker: the fused strict acceptance and next-step tokens equal the in-tree path +(N3 P and A1, the verify logits, at the end). + +The worker's two overrides are driven on a bare worker (no model) against the in-tree methods of its base class on +the same inputs: +- eligible steps (K3 / K7, padded drafts, graph-padding dummy rows) are bitwise; +- forced acceptance (fractional, so the RNG pool path runs) over a run that alternates eligible and ineligible + steps keeps the same accepted counts and the same RNG counter as an all-in-tree run; +- the sampler runs exactly once per step; +- the switch and an ineligible step take the in-tree path. +""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch +from torch import nn + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import strict_accept +from tensorrt_llm._torch.speculative.dspark import DSparkWorker + +TARGET = "minimax_m31_nvfp4__sm_107__tp8" +dspark = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.dspark" +) + +VOCAB = 200064 + +assert torch.cuda.is_available(), "the N3 A2 worker test requires a CUDA device" + + +def _worker(force: float = 0.0): + w = dspark.M3DSparkWorker.__new__(dspark.M3DSparkWorker) + nn.Module.__init__(w) + w.force_num_accepted_tokens = force + w._force_accept_rng_counter = None + w._force_accept_rng_pool = None + w.calls = 0 + return w + + +def _metadata(ragged: bool = False, graph: bool = False): + return SimpleNamespace(is_ragged_verify=ragged, is_cuda_graph=graph) + + +def _fixed_sampler(tokens: torch.Tensor): + """The in-tree sampler stands in as a fixed function of the logits: the overrides must call it once.""" + + def sample(self, logits, spec_metadata, num_contexts, batch_size): + self.calls += 1 + return tokens + + return sample + + +def _step(n_req: int, draft_len: int, gen: torch.Generator, pad: bool = False, int64: bool = False): + k1 = draft_len + 1 + tokens = torch.randint(0, VOCAB, (n_req * k1,), generator=gen, device="cuda") + tokens = tokens if int64 else tokens.to(torch.int32) + draft = tokens.view(n_req, k1)[:, :draft_len].clone().to(torch.int32) + cut = torch.randint(0, draft_len + 1, (n_req,), generator=gen, device="cuda") + pos = torch.arange(draft_len, device="cuda").unsqueeze(0) + draft = torch.where(pos == cut.unsqueeze(1), (draft + 1) % VOCAB, draft) + if pad: + draft[::2, draft_len // 2 :] = -1 + logits = torch.zeros( + n_req * k1, 8, device="cuda" + ) # shape only: the sampler stand-in ignores the values + return logits, draft, tokens + + +def _in_tree(w, logits, draft, num_contexts, batch_size, md): + return DSparkWorker._sample_and_accept_draft_tokens_base( + w, logits, draft, num_contexts, batch_size, md + ) + + +@pytest.fixture(autouse=True) +def _a1_opted_in(monkeypatch): + """A1 is on by default (switch TRTLLM_M31_DISABLE_A1); pin it on so the environment cannot turn it off here + (test_prebuild_a1_off turns it off again).""" + monkeypatch.setattr(dspark, "A1_DISABLED", False) + + +@pytest.fixture(autouse=True) +def _compiled(): + strict_accept.precompile((3, 7), torch.device("cuda")) + + +@pytest.mark.parametrize( + "n_req,draft_len,pad,int64", + [ + (1, 3, False, False), + (8, 3, False, False), + (12, 3, True, False), + (8, 7, False, False), + (8, 3, False, True), + ], +) +def test_eligible_steps_are_bitwise(monkeypatch, n_req, draft_len, pad, int64): + gen = torch.Generator(device="cuda").manual_seed(3) + logits, draft, tokens = _step(n_req, draft_len, gen, pad=pad, int64=int64) + monkeypatch.setattr(dspark.M3DSparkWorker, "_sample_tokens_for_batch", _fixed_sampler(tokens)) + w, ref = _worker(), _worker() + md = _metadata() + assert w._n3a_eligible(logits, draft, 0, n_req, md) + acc, num = w._sample_and_accept_draft_tokens_base(logits, draft, 0, n_req, md) + racc, rnum = _in_tree(ref, logits, draft, 0, n_req, md) + assert w.calls == 1 and ref.calls == 1 + assert acc.dtype == racc.dtype and num.dtype == rnum.dtype + assert torch.equal(acc, racc) and torch.equal(num, rnum) + + +def test_forced_acceptance_alternating_steps_matches_in_tree(monkeypatch): + """Fractional forced AL (the RNG pool path) over eligible, ragged (ineligible) and graph-warmup steps: the + counts and the RNG counter follow the all-in-tree run step for step.""" + gen = torch.Generator(device="cuda").manual_seed(5) + w, ref = _worker(force=1.6), _worker(force=1.6) + served = 0 + for step in range(12): + logits, draft, tokens = _step(8, 3, gen) + num_contexts, batch_size = 0, 8 + if step % 3 == 2: + # An ineligible step: one context row ahead of the 8 generation requests (the in-tree path runs). + ctx_token = torch.randint(0, VOCAB, (1,), generator=gen, device="cuda").to(tokens.dtype) + tokens = torch.cat([ctx_token, tokens]) + logits = torch.zeros(tokens.numel(), 8, device="cuda") + num_contexts, batch_size = 1, 9 + monkeypatch.setattr( + dspark.M3DSparkWorker, "_sample_tokens_for_batch", _fixed_sampler(tokens) + ) + md = _metadata(graph=(step == 4)) + eligible = w._n3a_eligible(logits, draft, num_contexts, batch_size, md) + assert eligible == (num_contexts == 0), step + served += int(eligible) + acc, num = w._sample_and_accept_draft_tokens_base( + logits, draft, num_contexts, batch_size, md + ) + racc, rnum = _in_tree(ref, logits, draft, num_contexts, batch_size, md) + # A context row defines only column 0 (the in-tree path leaves the rest uninitialized). + assert torch.equal(acc[:num_contexts, 0], racc[:num_contexts, 0]), step + assert torch.equal(acc[num_contexts:], racc[num_contexts:]) and torch.equal(num, rnum), step + if ref._force_accept_rng_counter is not None: + assert torch.equal(w._force_accept_rng_counter, ref._force_accept_rng_counter), step + assert served == 8 + # A ragged verify step is declined (the in-tree ragged path keeps it). + logits, draft, _ = _step(8, 3, gen) + assert not w._n3a_eligible(logits, draft, 0, 8, _metadata(ragged=True)) + + +def test_ineligible_and_switch_take_the_in_tree_path(monkeypatch): + gen = torch.Generator(device="cuda").manual_seed(9) + logits, draft, tokens = _step(4, 3, gen) + w = _worker() + # A context row, a row-count mismatch, too many requests: declined. + assert not w._n3a_eligible(logits, draft, 1, 4, _metadata()) + assert not w._n3a_eligible(logits[:-1], draft, 0, 4, _metadata()) + big_logits, big_draft, _ = _step(strict_accept.max_requests() + 1, 3, gen) + assert not w._n3a_eligible(big_logits, big_draft, 0, big_draft.shape[0], _metadata()) + monkeypatch.setattr(dspark, "N3A_DISABLED", True) + assert not w._n3a_eligible(logits, draft, 0, 4, _metadata()) + + +@pytest.mark.parametrize( + "draft_len,wide,shuffle", [(3, False, False), (3, True, True), (7, False, True)] +) +def test_next_new_tokens_matches_in_tree(draft_len, wide, shuffle): + gen = torch.Generator(device="cuda").manual_seed(13) + n_req = 8 + width = draft_len + 1 + (4 if wide else 0) + accepted = torch.randint( + 0, VOCAB, (n_req, width), generator=gen, device="cuda", dtype=torch.int32 + ) + num = torch.randint(1, draft_len + 2, (n_req,), generator=gen, device="cuda", dtype=torch.int32) + batch_idx = torch.arange(32, device="cuda", dtype=torch.int32) + if shuffle: + batch_idx[:n_req] = torch.randperm(n_req, generator=gen, device="cuda").to(torch.int32) + next_draft = torch.randint( + 0, VOCAB, (n_req, draft_len), generator=gen, device="cuda", dtype=torch.int32 + ) + w = _worker() + out = w._prepare_next_new_tokens(accepted, next_draft, batch_idx, n_req, num) + ref = DSparkWorker._prepare_next_new_tokens(w, accepted, next_draft, batch_idx, n_req, num) + assert out.dtype == ref.dtype and torch.equal(out, ref) + + +# N3 P-gid + P-cat: the worker's target logits equal the logits processor's. + + +@pytest.mark.parametrize("tp", [2, 4, 8]) +def test_rank_major_strided_copy_equals_vocab_concat_and_cast(tp): + """The all-gather along dim 0 + one strided copy writes what the vocab concat + .float() writes.""" + gen = torch.Generator(device="cuda").manual_seed(tp) + rows, shard = 32, 25008 + shards = [ + torch.randn(rows, shard, generator=gen, device="cuda").to(torch.bfloat16) for _ in range(tp) + ] + gathered = torch.cat(shards, dim=0) # what the dim-0 all-gather returns, rank-major + out = torch.empty((rows, tp * shard), dtype=torch.float32, device="cuda") + out.view(rows, tp, shard).copy_(gathered.view(tp, rows, shard).transpose(0, 1)) + assert torch.equal(out, torch.cat(shards, dim=-1).float()) + + +def _lm_head(vocab=50016, hidden=6144): + from tensorrt_llm._torch.modules.embedding import LMHead + from tensorrt_llm._torch.modules.linear import TensorParallelMode + from tensorrt_llm.mapping import Mapping + + head = LMHead( + vocab, + hidden, + dtype=torch.bfloat16, + mapping=Mapping(world_size=1, tp_size=1, rank=0), + tensor_parallel_mode=TensorParallelMode.COLUMN, + gather_output=True, + ).cuda() + with torch.no_grad(): + head.weight.copy_( + torch.randn_like(head.weight, dtype=torch.float32).to(torch.bfloat16) * 0.02 + ) + return head + + +def test_target_logits_equal_the_processor_at_tp1(): + from tensorrt_llm._torch.modules.logits_processor import LogitsProcessor + + head = _lm_head() + rows = 32 + hidden = torch.randn(rows, 6144, device="cuda").to(torch.bfloat16) + md = SimpleNamespace( + is_ragged_verify=False, gather_ids=torch.arange(rows, device="cuda", dtype=torch.int32) + ) + attn = SimpleNamespace(num_contexts=0) + out = _worker().target_logits(hidden, attn, md, head) + ref = LogitsProcessor().forward(hidden[md.gather_ids], head, attn, True) + assert out is not None and out.dtype == ref.dtype and torch.equal(out, ref) + + +def test_target_logits_decline_to_the_processor(monkeypatch): + head = _lm_head(vocab=1024, hidden=256) + hidden = torch.randn(8, 256, device="cuda").to(torch.bfloat16) + ids = torch.arange(8, device="cuda", dtype=torch.int32) + w = _worker() + # A context row, a gather that is not every row, a ragged verify, a padded head: the processor runs. + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=1), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids[:4]), + head, + ) + is None + ) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=True, gather_ids=ids), + head, + ) + is None + ) + monkeypatch.setattr(head, "padding_size", 3) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + monkeypatch.setattr(head, "padding_size", 0) + monkeypatch.setattr(dspark, "N3P_DISABLED", True) + assert ( + w.target_logits( + hidden, + SimpleNamespace(num_contexts=0), + SimpleNamespace(is_ragged_verify=False, gather_ids=ids), + head, + ) + is None + ) + + +# N3 A1: the verify logits' all-gather + concat + cast in one launch (catalog/comm/allgather_cast) when set up. + + +def _a1_step(monkeypatch, rows: int, compiled: bool = True): + """A TP8-shaped call: the head reports tp 8, A1 is set up, the op and the NCCL all-gather are recorded.""" + head = _lm_head(vocab=1024, hidden=256) + monkeypatch.setattr(head, "tp_size", 8) + calls = {"a1": 0, "nccl": 0} + sentinel = torch.full((rows, 8 * 1024), 7.0, device="cuda") + + def fake_a1(shard, **ws): + assert ws == {"ws": "a1"} and shard.shape == (rows, 1024) + calls["a1"] += 1 + return sentinel + + def fake_allgather(shard, mapping, dim=0): + calls["nccl"] += 1 + return torch.cat([shard] * 8, dim=0) + + monkeypatch.setattr(dspark._allgather_cast, "allgather_cast", fake_a1) + monkeypatch.setattr(dspark._allgather_cast, "is_compiled", lambda *a, **k: compiled) + monkeypatch.setattr(dspark, "allgather", fake_allgather) + w = _worker() + w._a1 = {"ws": "a1"} + hidden = torch.randn(rows, 256, device="cuda").to(torch.bfloat16) + md = SimpleNamespace( + is_ragged_verify=False, gather_ids=torch.arange(rows, device="cuda", dtype=torch.int32) + ) + out = w.target_logits(hidden, SimpleNamespace(num_contexts=0), md, head) + return out, sentinel, calls + + +@pytest.mark.parametrize("rows", [4, 32, 64]) +def test_target_logits_take_a1_when_set_up(monkeypatch, rows): + out, sentinel, calls = _a1_step(monkeypatch, rows) + assert out is sentinel and calls == {"a1": 1, "nccl": 0} + + +@pytest.mark.parametrize("rows,compiled", [(68, True), (4, False)]) +def test_target_logits_fall_back_to_nccl_past_a1(monkeypatch, rows, compiled): + """More rows than the workspace holds, or a shape never compiled: the NCCL all-gather + strided copy.""" + out, _, calls = _a1_step(monkeypatch, rows, compiled=compiled) + assert ( + calls == {"a1": 0, "nccl": 1} + and out.shape == (rows, 8 * 1024) + and out.dtype == torch.float32 + ) + + +def _a1_prebuild_worker(monkeypatch, tp: int = 8, draft_len: int = 3): + w = _worker() + w.mapping = SimpleNamespace(tp_size=tp, tp_rank=0, enable_attention_dp=False) + w.spec_config = SimpleNamespace(max_draft_len=draft_len) # max_draft_len reads it + # Eligibility needs every SM up to the workspace's grid; any CUDA device passes with a 1-CTA workspace grid. + monkeypatch.setattr(dspark._allgather_cast, "MAX_GRID", 1) + monkeypatch.setattr(dspark.M3DSparkWorker, "_a1_all_ok", lambda self, local: local) + return w + + +def test_prebuild_a1_sets_up_k3_rows(monkeypatch): + head = _lm_head(vocab=1024, hidden=256) + seen = {} + + def fake_precompile(mapping, cols, rows, device, group=None, agree=None): + seen.update(cols=cols, rows=rows, agreed=agree(True)) + return True + + ws = {"ws_uc": torch.zeros(4, dtype=torch.int32), "rank": 0} + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr( + dspark._allgather_cast, "workspace_args", lambda mapping, cols, group=None: ws + ) + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is ws + # K3 verify rows of batches 1..16, all within the workspace's 64 rows. + assert seen == {"cols": 1024, "rows": tuple(range(4, 65, 4)), "agreed": True} + + +@pytest.mark.parametrize("draft_len", [3, 7]) +def test_prebuild_a1_sets_up_the_verify_rows_of_k3_and_k7(monkeypatch, draft_len): + """K3: rows 4..64 (batches 1..16); K7, the headline: rows 8..64 (batches 1..8).""" + head = _lm_head(vocab=1024, hidden=256) + seen = {} + + def fake_precompile(mapping, cols, rows, device, group=None, agree=None): + seen.update(rows=rows, agreed=agree(True)) + return True + + ws = {"ws_uc": torch.zeros(4, dtype=torch.int32), "rank": 0} + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr( + dspark._allgather_cast, "workspace_args", lambda mapping, cols, group=None: ws + ) + assert _a1_prebuild_worker(monkeypatch, draft_len=draft_len)._prebuild_a1(head) is ws + k1 = draft_len + 1 + assert seen == {"rows": tuple(range(k1, 65, k1)), "agreed": True} + + +def test_prebuild_a1_off(monkeypatch): + """The switch, another draft length, one rank, a rank that is not eligible, or a compile that failed on any rank: + no A1, no workspace.""" + head = _lm_head(vocab=1024, hidden=256) + built = {"n": 0} + + def fake_precompile(*a, **k): + built["n"] += 1 + return False # a failed compile somewhere + + monkeypatch.setattr(dspark._allgather_cast, "precompile", fake_precompile) + monkeypatch.setattr(dspark, "A1_K7_DISABLED", True) + assert _a1_prebuild_worker(monkeypatch, draft_len=7)._prebuild_a1(head) is None + monkeypatch.setattr(dspark, "A1_K7_DISABLED", False) + assert ( + _a1_prebuild_worker(monkeypatch, draft_len=64)._prebuild_a1(head) is None + ) # 65 rows at batch 1 + assert _a1_prebuild_worker(monkeypatch, tp=1)._prebuild_a1(head) is None + assert built["n"] == 0 + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is None + assert built["n"] == 1 + # Another rank not eligible: off here too, nothing compiled. + w = _a1_prebuild_worker(monkeypatch) + monkeypatch.setattr(dspark.M3DSparkWorker, "_a1_all_ok", lambda self, local: False) + assert w._prebuild_a1(head) is None and built["n"] == 1 + monkeypatch.setattr(dspark, "A1_DISABLED", True) + assert _a1_prebuild_worker(monkeypatch)._prebuild_a1(head) is None + assert built["n"] == 1 + + +# N3A-KV: the draft-forward KV bookkeeping in one launch, through the DFlash hook (bitwise to the stock lines). + + +def _kv_step(nc: int, bs: int, gen: torch.Generator): + num = torch.randint(1, 5, (bs + 2,), generator=gen, device="cuda", dtype=torch.int32) + kv = torch.randint(0, 1 << 20, (bs + 2,), generator=gen, device="cuda", dtype=torch.int32) + return num, kv + + +def _kv_worker(fused: bool): + w = _worker() + w._fused_kv_rewind = None + if fused: + strict_accept.precompile_kv_rewind(torch.device("cuda")) + w._n3a_kv_buf = torch.empty(strict_accept.max_requests(), dtype=torch.int32, device="cuda") + w._fused_kv_rewind = w._n3a_kv_rewind + return w + + +@pytest.mark.parametrize("nc,bs", [(0, 1), (0, 8), (2, 6), (3, 3)]) +def test_kv_rewind_hook_equals_the_stock_lines(nc, bs): + gen = torch.Generator(device="cuda").manual_seed(5) + num, kv = _kv_step(nc, bs, gen) + outs = [] + for fused in (False, True): + w = _kv_worker(fused) + attn = SimpleNamespace(kv_lens_cuda=kv.clone(), update_for_spec_dec=lambda: None) + w._prepare_kv_for_draft_forward(attn, num, nc, bs) + outs.append( + ( + w._kv_rewind_amount, + attn.kv_lens_cuda, + w._kv_rewind_nc, + w._kv_rewind_bs, + w._kv_rewind_pending, + ) + ) + (r0, k0, *rest0), (r1, k1, *rest1) = outs + assert torch.equal(k0, k1) and rest0 == rest1 + assert r0.dtype == r1.dtype and r0.shape == r1.shape and torch.equal(r0, r1) + + +def test_kv_rewind_hook_declines_unsupported_operands(): + w = _kv_worker(True) + num64 = torch.ones(4, dtype=torch.int64, device="cuda") # not the buffer's dtype + kv = torch.zeros(4, dtype=torch.int32, device="cuda") + assert w._n3a_kv_rewind(num64, kv, 0, 4) is None and torch.equal(kv, torch.zeros_like(kv)) + w._n3a_kv_buf = None + assert w._n3a_kv_rewind(torch.ones(4, dtype=torch.int32, device="cuda"), kv, 0, 4) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_s4.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_s4.py new file mode 100644 index 000000000000..762c61f43ec2 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_s4.py @@ -0,0 +1,347 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp8 target's worker-side routing of the one-launch Markov chain (``dspark_s4.S4WorkerMixin``), without GPUs +or a model: which steps it serves, that the kill switch sets nothing up, that a declined step is the in-tree path, that +the step's logits processor only takes the block rows, and that the chain's tokens replace the in-tree sampling for +that one call. The chain itself is the catalog's ``spec_decode/dspark_markov_chain``, certified by +``spec_decode/_dspark_markov_chain_op_matrix.py``; the in-model equality by the call-site gate.""" + +import importlib +import os +import subprocess +import sys +from types import SimpleNamespace + +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + dspark_s4 as w1, +) + + +class _Base: + def __init__(self): + self.calls = [] + + def _forward_impl(self, *args): + self.calls.append("forward") + return "in-tree" + + def _refine_block_logits(self, draft_model, gen_logits, inputs, spec_metadata): + self.calls.append("refine") + return gen_logits + + def sample_draft_tokens(self, logits, spec_metadata, batch_size, **kwargs): + self.calls.append("sample") + return torch.full(logits.shape[:2], -1, dtype=torch.int32) + + def _apply_kv_rewind_after_draft(self, attn_metadata, spec_metadata): + self.calls.append("stock_rewind") + self._kv_rewind_pending = False + + +class _Worker(w1.S4WorkerMixin, _Base): + pass + + +def _state(block=3, shard=8, rows=8): + st = w1._S4State(None, rows, shard, 0, None, None, block, {}) + for b in range(1, w1.MAX_FAST_BATCH + 1): + st.bufs[(b, block)] = torch.full((b, block, rows), float("-inf")) + return st + + +def _meta(advanced=False): + return SimpleNamespace(wants_advanced_draft_sampling=advanced) + + +def test_serves_only_eligible_steps(monkeypatch): + wk = _Worker() + draft = SimpleNamespace(lilicorr=None, has_candidate_selector=False) + assert not wk._s4_serves(1, _meta(), draft) # not set up + wk._s4 = _state() + assert wk._s4_serves(1, _meta(), draft) + assert wk._s4_serves(w1.MAX_FAST_BATCH, _meta(), draft) + assert not wk._s4_serves(0, _meta(), draft) + assert not wk._s4_serves(w1.MAX_FAST_BATCH + 1, _meta(), draft) + assert not wk._s4_serves(1, _meta(advanced=True), draft) + wk._d2t = torch.zeros(1) + assert not wk._s4_serves(1, _meta(), draft) + + +def test_kill_switch_sets_nothing_up(monkeypatch): + wk = _Worker() + wk._s4 = _state() + monkeypatch.setattr(w1, "_DISABLED", True) + assert not wk.prebuild_s4(None, SimpleNamespace(has_markov_head=True)) + assert wk._s4 is None + assert not wk._s4_serves( + 1, _meta(), SimpleNamespace(lilicorr=None, has_candidate_selector=False) + ) + + +def test_declined_step_is_the_in_tree_path(): + wk = _Worker() + attn = SimpleNamespace(num_seqs=12, num_contexts=0) # above MAX_FAST_BATCH + draft = SimpleNamespace(lilicorr=None, has_candidate_selector=False, logits_processor="proc") + wk._s4 = _state() + assert wk._forward_impl(None, None, None, None, attn, _meta(), draft) == "in-tree" + assert draft.logits_processor == "proc" + logits = torch.zeros(2, 3, 16) # full-vocab width: not this step's shard + assert wk._refine_block_logits(draft, logits, {}, None) is logits + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() + assert wk.calls == ["forward", "refine", "sample"] + + +def test_sharded_logits_processor_takes_only_block_rows(): + st = _state(block=3, shard=8, rows=12) + inner_calls = [] + + def inner(h, head, meta, ctx): + inner_calls.append(h.shape) + return "inner" + + head = SimpleNamespace(apply_linear=lambda h, b: h[:, :8] * 2) + proc = w1._ShardedDraftLogits(inner, st, batch=2) + h = torch.arange(6 * 10, dtype=torch.float32).view(6, 10) + out = proc(h, head, None, True) + assert torch.equal(out, h[:, :8] * 2) + buf = st.buffer(2) + assert torch.equal(buf[:, :, :8], (h[:, :8] * 2).view(2, 3, 8)) + assert torch.isinf(buf[:, :, 8:]).all() + assert proc(torch.zeros(5, 10), head, None, True) == "inner" and inner_calls == [ + torch.Size([5, 10]) + ] + + +def test_chain_tokens_replace_sampling_once(monkeypatch): + wk = _Worker() + st = _state(block=3, shard=8, rows=8) + wk._s4_step = st + toks = torch.tensor([[1, 2, 3], [4, 5, 6]]) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, + dspark_markov_chain=lambda *a, **k: toks.int(), + ll_exchange_engaged=lambda batch: False)) # fmt: skip + logits = torch.zeros(2, 3, 8) + assert wk._refine_block_logits(None, logits, {"first_prev_tokens": torch.zeros(2, dtype=torch.long)}, None) \ + is logits # fmt: skip + out = wk.sample_draft_tokens(logits, None, 2) + assert out.dtype == torch.int32 and torch.equal(out, toks.int()) + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() # used once + assert "refine" not in wk.calls + + +def test_unsupported_shapes_fall_back_on_the_shards(monkeypatch): + wk = _Worker() + wk._s4_step = _state(block=3, shard=8, rows=8) + + def _never(*a, **k): + raise AssertionError("an unsupported step must not reach the op") + + monkeypatch.setattr( + w1, "chain", SimpleNamespace(supports=lambda *a: False, dspark_markov_chain=_never) + ) + logits = torch.zeros(2, 3, 8) + wk._refine_block_logits( + None, logits, {"first_prev_tokens": torch.zeros(2, dtype=torch.long)}, None + ) + assert wk.calls == ["refine"] + assert wk.sample_draft_tokens(logits, None, 2).eq(-1).all() + + +def test_switch_reads_any_value_but_empty_or_zero(monkeypatch): + """Read at import, like the targets' m31_switches: "true" turns S4 off, not only "1".""" + try: + for value, off in ( + ("1", True), + ("true", True), + ("yes", True), + ("0", False), + ("", False), + (" 0 ", False), + ): + monkeypatch.setenv(w1.DISABLE_ENV, value) + importlib.reload(w1) + assert w1.disabled() is off, value + finally: + monkeypatch.delenv(w1.DISABLE_ENV, raising=False) + importlib.reload(w1) + assert not w1.disabled() + + +def _rewind_worker(monkeypatch, batch=2, nc=1, on=True): + """A worker in a served step with a pending KV-length rewind of ``batch`` rows from ``nc``, and a chain fake that + records which launch ran.""" + wk = _Worker() + wk.mapping = SimpleNamespace(tp_rank=5) + wk._s4_step = _state(block=3, shard=8, rows=8) + wk._s4_step_attn = SimpleNamespace(kv_lens_cuda=torch.zeros(nc + batch + 1, dtype=torch.int32)) + wk._kv_rewind_pending = True + wk._kv_rewind_amount = torch.zeros(batch, dtype=torch.int32) + wk._kv_rewind_nc, wk._kv_rewind_bs = nc, nc + batch + launches, lines = [], [] + toks = torch.zeros(batch, 3, dtype=torch.int32) + + def plain(*a, **k): + launches.append("chain") + return toks + + def with_rewind(base, first_prev, w1_, w2s, offset, kv_lens, amount, first, **k): + launches.append( + ( + "rewind", + kv_lens is wk._s4_step_attn.kv_lens_cuda, + amount is wk._kv_rewind_amount, + first, + ) + ) + return toks + + monkeypatch.setattr(w1, "TAILBK_SPEC", on) + monkeypatch.setattr(w1.logger, "info", lambda msg, *a, **k: lines.append(msg)) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, supports_rewind=lambda *a: True, + dspark_markov_chain=plain, + dspark_markov_chain_rewind=with_rewind, + ll_exchange_engaged=lambda b: False)) # fmt: skip + return wk, launches, lines, {"first_prev_tokens": torch.zeros(batch, dtype=torch.long)} + + +def test_tailbk_h_rewind_in_chain_launch(monkeypatch): + """TAILBK_SPEC H: a real step's chain launch applies the pending rewind (this step's kv_lens, amounts, first + row) and the stock rewind lines are skipped for that step; the engagement line is logged once per rank.""" + wk, launches, lines, inputs = _rewind_worker(monkeypatch) + meta = SimpleNamespace(is_cuda_graph=False) + for _ in range(2): + wk._kv_rewind_pending = True + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert not wk._kv_rewind_pending and not wk._tailbk_rewound + assert launches == [("rewind", True, True, 1)] * 2, launches + assert "stock_rewind" not in wk.calls + assert [m for m in lines if "TAILBK_SPEC H" in m] == [ + "[M3 dispatch] TAILBK_SPEC H: KV-length rewind in the Markov chain launch on rank 5" + ] + + +def test_tailbk_h_warmup_keeps_stock_path(monkeypatch): + """The eager warmup of a graph batch (is_cuda_graph, not capturing): the stock method skips the rewind there, so + the chain must not apply it either.""" + wk, launches, _, inputs = _rewind_worker(monkeypatch) + meta = SimpleNamespace(is_cuda_graph=True) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def test_tailbk_h_off_or_mismatch_keeps_stock_path(monkeypatch): + """Kill switch off, no pending rewind, or amounts that are not this chain call's rows: the stock rewind runs.""" + meta = SimpleNamespace(is_cuda_graph=False) + wk, launches, _, inputs = _rewind_worker(monkeypatch, on=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + wk, launches, _, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_pending = False + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert launches == ["chain"] + wk, launches, _, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_amount = torch.zeros(3, dtype=torch.int32) # not the 2 rows of this call + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def test_tailbk_h_unsupported_operands_keep_stock_path(monkeypatch): + """supports_rewind False (e.g. int64 amounts): the plain chain launch and the stock rewind.""" + wk, launches, _, inputs = _rewind_worker(monkeypatch) + monkeypatch.setattr(w1.chain, "supports_rewind", lambda *a: False) + meta = SimpleNamespace(is_cuda_graph=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._apply_kv_rewind_after_draft(None, meta) + assert launches == ["chain"] and wk.calls == ["stock_rewind"] + + +def _tailbk_in_subprocess(value): + env = {k: v for k, v in os.environ.items() if k != "TRTLLM_M31_DISABLE_TAILBK_SPEC"} + if value is not None: + env["TRTLLM_M31_DISABLE_TAILBK_SPEC"] = value + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_107__tp8 import dspark_s4 as m; print('TAILBK_SPEC=' + str(m.TAILBK_SPEC))" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=600 + ) + assert out.returncode == 0, out.stderr + return [ln for ln in out.stdout.splitlines() if ln.startswith("TAILBK_SPEC=")][-1] + + +def test_tailbk_spec_kill_switch_subprocess(): + """TAILBK_SPEC is on by default; TRTLLM_M31_DISABLE_TAILBK_SPEC=1 (read at import) turns it off.""" + assert _tailbk_in_subprocess(None) == "TAILBK_SPEC=True" + assert _tailbk_in_subprocess("0") == "TAILBK_SPEC=True" + assert _tailbk_in_subprocess("1") == "TAILBK_SPEC=False" + + +def test_tailbk_c_engagement_logged_once_per_rank(monkeypatch): + """TAILBK_SPEC C's "[M3 dispatch]" line: once per rank, only after a chain call took the cached inputs.""" + wk, launches, lines, inputs = _rewind_worker(monkeypatch) + wk._kv_rewind_pending = False # the plain chain launch + taken = [False] + monkeypatch.setattr(w1.chain, "cached_inputs_taken", lambda: taken[0], raising=False) + meta = SimpleNamespace(is_cuda_graph=False) + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert not [m for m in lines if "TAILBK_SPEC C" in m] + taken[0] = True + for _ in range(3): + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + wk._kv_rewind_pending = True # and through the rewind launch + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs, meta) + assert [m for m in lines if "TAILBK_SPEC C" in m] == [ + "[M3 dispatch] TAILBK_SPEC C: chain accepted inputs from precompile cache (taken) on rank 5" + ] + + +def test_markov_ll_engagement_logged_once_per_rank(monkeypatch): + """MARKOV_LL's "[M3 dispatch]" line: once per rank, on the first served step that took the LL kernel.""" + wk = _Worker() + wk.mapping = SimpleNamespace(tp_rank=3) + wk._s4_step = _state(block=3, shard=8, rows=8) + lines = [] + monkeypatch.setattr(w1.logger, "info", lambda msg, *a, **k: lines.append(msg)) + monkeypatch.setattr(w1, "chain", SimpleNamespace(supports=lambda *a: True, + dspark_markov_chain=lambda *a, **k: torch.zeros( + a[0].shape[0], 3, dtype=torch.int32), + ll_exchange_engaged=lambda batch: batch == 1)) # fmt: skip + inputs2 = {"first_prev_tokens": torch.zeros(2, dtype=torch.long)} + wk._refine_block_logits(None, torch.zeros(2, 3, 8), inputs2, None) # batch 2: not the LL kernel + assert not [m for m in lines if "MARKOV_LL" in m] + inputs1 = {"first_prev_tokens": torch.zeros(1, dtype=torch.long)} + for _ in range(3): + wk._refine_block_logits(None, torch.zeros(1, 3, 8), inputs1, None) + ll = [m for m in lines if "MARKOV_LL" in m] + assert ll == [ + "[M3 dispatch] MARKOV_LL: k3_markov LL flag-in-payload exchange (batch 1) on rank 3" + ], ll + + +def _markov_ll_in_subprocess(value): + env = {k: v for k, v in os.environ.items() if k != "TRTLLM_M31_DISABLE_MARKOV_LL"} + if value is not None: + env["TRTLLM_M31_DISABLE_MARKOV_LL"] = value + code = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + "minimax_m31_nvfp4__sm_107__tp8 import dspark_s4 as m; print('MARKOV_LL=' + str(m.MARKOV_LL))" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=600 + ) + assert out.returncode == 0, out.stderr + return [ln for ln in out.stdout.splitlines() if ln.startswith("MARKOV_LL=")][-1] + + +def test_markov_ll_kill_switch_subprocess(): + """MARKOV_LL is on by default; TRTLLM_M31_DISABLE_MARKOV_LL=1 (read at import) turns it off.""" + assert _markov_ll_in_subprocess(None) == "MARKOV_LL=True" + assert _markov_ll_in_subprocess("0") == "MARKOV_LL=True" + assert _markov_ll_in_subprocess("1") == "MARKOV_LL=False" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_w0.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_w0.py new file mode 100644 index 000000000000..613da16fbb93 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_dspark_w0.py @@ -0,0 +1,164 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""One-model DSpark on the tp8 target, phase W0 (``dspark.py``): the shell's calls on the DSpark worker take no +fused path, the capture is the in-tree DFlash capture, the drafter's borrowed embedding is the target's exact gather, +and the FP8 draft-KV override (an Eagle3 need) is never applied to a DSpark run, so the drafter's context pool keeps +the projection dtype (BF16) and binds to the separate draft KV cache. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + dspark, +) +from tensorrt_llm.quantization.mode import QuantAlgo + + +def _spec( + *, dspark_mode: bool, eagle3: bool = False, one_engine: bool = True, embedded: bool = False +): + mode = SimpleNamespace( + use_one_engine=lambda: one_engine, + is_dspark=lambda: dspark_mode, + is_eagle3_one_model=lambda: eagle3, + ) + return SimpleNamespace(spec_dec_mode=mode, draft_is_embedded_in_target=embedded) + + +def _worker() -> dspark.M3DSparkWorker: + # The shell-facing calls read no worker state, so the worker is not constructed (that needs a mapping and a + # DSpark spec config). + return dspark.M3DSparkWorker.__new__(dspark.M3DSparkWorker) + + +@pytest.mark.parametrize( + "spec, expected", + [ + (_spec(dspark_mode=True), True), + (_spec(dspark_mode=True, embedded=True), False), + (_spec(dspark_mode=True, one_engine=False), False), + (_spec(dspark_mode=False, eagle3=True), False), + (None, False), + ], +) +def test_standalone_dspark_detection(spec, expected): + assert dspark.is_standalone_dspark(spec) is expected + + +def test_fp8_draft_kv_override_is_eagle3_only(): + assert dspark.draft_kv_override_applies(_spec(dspark_mode=False, eagle3=True)) + assert dspark.draft_kv_override_applies(None) + assert not dspark.draft_kv_override_applies(_spec(dspark_mode=True)) + + +@pytest.mark.parametrize("target_algo", [QuantAlgo.NVFP4, QuantAlgo.FP8, None]) +def test_drafter_kv_pool_keeps_the_drafter_dtype(target_algo): + from tensorrt_llm.models.modeling_utils import QuantConfig + + whole, layer = ( + QuantConfig(kv_cache_quant_algo=target_algo), + QuantConfig(kv_cache_quant_algo=target_algo), + ) + _ = whole.quant_mode # cached before the drop, as a loaded config may have it + cfg = SimpleNamespace(quant_config=whole, quant_config_dict={"model.layers.0": layer}) + dspark.keep_drafter_kv_dtype(cfg) + assert whole.kv_cache_quant_algo is None and layer.kv_cache_quant_algo is None + assert not whole.quant_mode.has_fp8_kv_cache() and not whole.quant_mode.has_fp4_kv_cache() + + +def test_shell_calls_take_no_fused_path(): + # tp8 has no coded target head; the fused verify is the only path tail_eligible() could open. + worker = _worker() + assert worker.tail_eligible(None, None, None) is False + assert worker.arm_capture_norm(None, None, None, None) is None + assert worker.prebuild(None, None) is None + + +def test_capture_is_the_in_tree_capture(): + calls = [] + meta = SimpleNamespace(maybe_capture_hidden_states=lambda *a: calls.append(a)) + hidden, residual = object(), object() + _worker().capture_hidden_states(meta, 17, hidden, residual) + _worker().capture_hidden_states(meta, 59, hidden) + assert calls == [(17, hidden, residual), (59, hidden, None)] + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +def test_embed_tokens_is_the_target_gather(): + gen = torch.Generator(device="cuda").manual_seed(31) + table = torch.randn(1000, 64, generator=gen, device="cuda").to(torch.bfloat16) + core = SimpleNamespace(w={"embed": table}) + embed = dspark.M3EmbedTokens(core) + ids = torch.tensor([0, 5, 999, 200, -1], dtype=torch.long, device="cuda") + out = embed(ids) + ref = table[ids.clamp(min=0)] * (ids >= 0).unsqueeze(-1).to(table.dtype) + assert embed.weight is table + assert torch.equal(out, ref) + assert not list(embed.parameters()), ( + "the table belongs to the target, not to the borrowed module" + ) + + +def test_borrowed_embedding_owns_no_parameters(): + # The weights walker asserts every shell parameter is fed: the borrowed module must add none. + model = torch.nn.Module() + model.w = torch.nn.ParameterDict( + {"embed": torch.nn.Parameter(torch.zeros(8, 4), requires_grad=False)} + ) + before = {n for n, _ in model.named_parameters()} + model.embed_tokens = dspark.M3EmbedTokens(model) + assert {n for n, _ in model.named_parameters()} == before == {"w.embed"} + assert model.embed_tokens.weight is model.w["embed"] + + +class _FakeDrafter: + def __init__(self, borrows: bool = True) -> None: + self.loaded = None + self.borrows = borrows + self.lm_head = None + self.draft_model_full = SimpleNamespace(model=SimpleNamespace(embed_tokens=None)) + + def load_weights(self, weights, weight_mapper=None): + self.loaded = dict(weights) + + def load_weights_from_target_model(self, target): + if self.borrows: + self.lm_head = target.lm_head + self.draft_model_full.model.embed_tokens = target.model.embed_tokens + + +def _shell(drafter): + return SimpleNamespace( + draft_model=drafter, + lm_head=object(), + model=SimpleNamespace(embed_tokens=object()), + named_parameters=lambda: iter([("lm_head.weight", None), ("model.w.embed", None)]), + ) + + +@pytest.mark.parametrize( + "name", + ["model.embed_tokens.weight", "language_model.model.embed_tokens.weight", "lm_head.weight"], +) +def test_draft_load_refuses_borrowed_tensors(name): + shell = _shell(_FakeDrafter()) + with pytest.raises(ValueError, match="lends it"): + dspark.load_dspark_draft_weights(shell, {name: torch.zeros(1), "fc.weight": torch.zeros(1)}) + assert shell.draft_model.loaded is None + + +def test_draft_load_borrows_target_modules(): + shell = _shell(_FakeDrafter()) + weights = { + "fc.weight": torch.zeros(1), + "markov_head.markov_w1.weight": torch.zeros(1), + "confidence_head.proj.weight": torch.zeros(1), + } + dspark.load_dspark_draft_weights(shell, weights) + assert shell.draft_model.loaded == weights + assert shell.draft_model.lm_head is shell.lm_head + with pytest.raises(RuntimeError, match="does not borrow"): + dspark.load_dspark_draft_weights(_shell(_FakeDrafter(borrows=False)), weights) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_vr_draft_append_fuse.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_vr_draft_append_fuse.py new file mode 100644 index 000000000000..4a274cd90358 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m31_vr_draft_append_fuse.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""DRAFT_APPEND_FUSE on the sm_107 tp8 target: default ON, kill switch +TRTLLM_M31_DISABLE_DRAFT_APPEND_FUSE, and the target hands the catalog qk_norm_rope_append (through its counting +wrapper) to the DSpark worker only with EX-11. The fused append's bits are covered by +modeling_v2/spec_decode/test_modeling_v2_dspark_qk_norm_rope.py.""" + +import ast +import os +import subprocess +import sys +from pathlib import Path + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp8" +_DSPARK = Path(_family.__file__).resolve().parent / TARGET / "dspark.py" + + +def test_on_by_default_and_off_by_its_kill_switch() -> None: + probe = ( + "from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3." + f"{TARGET} import dspark as d; print(d.DRAFT_APPEND_FUSE)" + ) + for value, expected in ((None, "True"), ("1", "False")): + env = {k: v for k, v in os.environ.items() if "DRAFT_APPEND_FUSE" not in k} + if value is not None: + env["TRTLLM_M31_DISABLE_DRAFT_APPEND_FUSE"] = value + out = subprocess.run( + [sys.executable, "-c", probe], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.strip().splitlines()[-1] == expected, value + + +def test_wired_after_ex11_with_the_counting_wrapper() -> None: + tree = ast.parse(_DSPARK.read_text()) + calls = [ + n + for n in ast.walk(tree) + if isinstance(n, ast.Call) + and isinstance(n.func, ast.Attribute) + and n.func.attr == "enable_fused_qk_norm_rope_append" + ] + assert len(calls) == 1 + src = ast.unparse(calls[0].args[1]) + assert src == "_qk_norm_rope_append_logged if ex11 and DRAFT_APPEND_FUSE else None", src diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_norm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_norm.py new file mode 100644 index 000000000000..d2131b5dcc68 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_norm.py @@ -0,0 +1,57 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_capture_norm entry (Eagle3 capture with fc_norm folded in).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_capture_norm import ( # noqa: E501 + m3_capture_norm as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 — needs the path declared above + EPS, + HIDDEN, + assert_bf16_close, + gen, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_capture_norm requires a CUDA device" + +LAYERS = 3 + + +def _case(spec, seed): + g = gen(seed) + m, slot = spec["m"], spec["slot"] + h = randn((m, HIDDEN), g, 2.0) + r = randn((m, HIDDEN), g, 2.0) if spec["residual"] else None + w = (1.0 + randn((HIDDEN,), g, 0.1).float()).to(torch.bfloat16) + buf = torch.full((m, LAYERS * HIDDEN), 7.0, dtype=torch.bfloat16, device="cuda") + dst = buf[:, slot * HIDDEN : (slot + 1) * HIDDEN] + return (h, r, w, EPS, dst), buf + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Within one bf16 ulp per element on at most 0.1% of the elements; the other layer slots of the buffer + untouched.""" + args, buf = _case(cell.spec, seed_of(cell.why)) + certifying(op) + out = op(*args) + assert_bf16_close(out, op.reference(*args)) + slot = cell.spec["slot"] + others = torch.cat([buf[:, : slot * HIDDEN], buf[:, (slot + 1) * HIDDEN :]], dim=1) + assert bool((others == 7.0).all()) + + +def test_guard_refuses_a_strided_weight() -> None: + (h, r, w, eps, dst), _ = _case(op.CELLS[0].spec, 1) + w2 = torch.stack([w, w], dim=1)[:, 0] + with pytest.raises(AssertionError, match="dense"): + with validating(op): + op(h, r, w2, eps, dst) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_normed.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_normed.py new file mode 100644 index 000000000000..857a9512a4d1 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_normed.py @@ -0,0 +1,71 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CPU test: a first-draft-inputs reader that skips the drafter's fc_norm (the capture norm is armed) gets the captured +rows unchanged when every captured layer was written with the step's norm, and otherwise a copy with only the raw +layers normalized by their own fc_norm (M3Eagle3Worker._m3_renorm_raw_captures); the per-forward reset clears the +record and the unconsumed fc fork result.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) + +M3Eagle3Worker = modeling.M3Eagle3Worker + +H = 4 +LAYERS = (1, 29, 56) + + +def _worker(normed): + w = object.__new__(M3Eagle3Worker) + w._m3_capture_normed = set(normed) + return w + + +@pytest.fixture(autouse=True) +def _scaling_norm(monkeypatch): + """The RMSNorm the worker applies, as ``x * w`` (CPU): enough to tell which blocks were normalized here.""" + monkeypatch.setattr(modeling, "flashinfer_rmsnorm", lambda x, w, eps: x * w) + modeling.set_require(False) + + +def _drafter(): + # fc_norm i scales by (i + 2). + return SimpleNamespace(w={f"fc_norm{i}": i + 2 for i in range(len(LAYERS))}, rms_eps=1e-6) + + +def test_all_captured_layers_normed_returns_the_rows_as_they_lie() -> None: + hs = torch.randn(4, H * len(LAYERS)) + out = _worker(set(LAYERS))._m3_renorm_raw_captures( + hs, SimpleNamespace(layers_to_capture=LAYERS, hidden_size=H), _drafter() + ) + assert out is hs + + +@pytest.mark.parametrize("normed", [set(), {1, 29}, {29, 56}, {1, 56}]) +def test_raw_layers_get_their_own_fc_norm(normed) -> None: + hs = torch.randn(4, H * len(LAYERS)) + ref = hs.clone() + out = _worker(normed)._m3_renorm_raw_captures( + hs, SimpleNamespace(layers_to_capture=LAYERS, hidden_size=H), _drafter() + ) + assert torch.equal(hs, ref) # the capture buffer itself is not modified + for i, layer_id in enumerate(LAYERS): + block, src = out[:, i * H : (i + 1) * H], ref[:, i * H : (i + 1) * H] + assert torch.equal(block, src if layer_id in normed else src * (i + 2)), layer_id + + +def test_arm_resets_the_record_and_the_fork_result() -> None: + w = _worker(set(LAYERS)) + w._m3_fc_hidden = object() + w._m3_fc_early_stream = None # no fork pending + w._m3_join_pending = lambda: None # (tp8) nothing else pending + w.arm_capture_norm( + None, None, None, None + ) # no drafter: resets the per-forward state and returns + assert w._m3_capture_normed == set() + assert w._m3_fc_hidden is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_target.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_target.py new file mode 100644 index 000000000000..c0f68f89ec8e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_target.py @@ -0,0 +1,82 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The worker's capture target (the capture norm in the boundary all-reduce): the capture buffer's column block of a +captured layer with its fc_norm weight and eps; None without an armed capture norm, for a layer that is not captured, +past the buffer's rows, or when the loaded all-reduce op has no capture epilogue (then the capture_norm kernel runs).""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_mc_allreduce, +) + +H = 64 +LAYERS = (2, 30, 58) + + +def _setup(armed=True): + hidden = torch.arange(8 * 3 * H, dtype=torch.float32).reshape(8, 3 * H).to(torch.bfloat16) + weights = torch.randn(3, H).to(torch.bfloat16) + meta = SimpleNamespace(hidden_states=hidden, hidden_size=H, layers_to_capture=LAYERS) + worker = SimpleNamespace(_m3_capture_norm=(weights, 1e-5) if armed else None) + return worker, meta, hidden, weights + + +def _target(worker, meta, layer_id, num_tokens): + return modeling.M3Eagle3Worker.capture_target(worker, meta, layer_id, num_tokens) + + +@pytest.fixture +def supported(monkeypatch): + monkeypatch.setattr(m3_mc_allreduce, "capture_supported", lambda: True) + + +def test_column_block_of_each_captured_layer(supported) -> None: + worker, meta, hidden, weights = _setup() + for i, layer_id in enumerate(LAYERS): + out, w, eps = _target(worker, meta, layer_id, 4) + assert out.shape == (4, H) + assert out.data_ptr() == hidden[:4, i * H : (i + 1) * H].data_ptr() + assert out.stride() == hidden.stride() + assert torch.equal(w, weights[i]) + assert eps == 1e-5 + + +def test_none_when_not_capturable(supported) -> None: + worker, meta, _, _ = _setup() + assert _target(worker, meta, 3, 4) is None # not a captured layer + assert _target(worker, meta, LAYERS[0], 9) is None # past the buffer's rows + unarmed, meta2, _, _ = _setup(armed=False) + assert _target(unarmed, meta2, LAYERS[0], 4) is None + + +def test_none_without_the_capture_epilogue(monkeypatch) -> None: + monkeypatch.setattr(m3_mc_allreduce, "capture_supported", lambda: False) + worker, meta, _, _ = _setup() + assert _target(worker, meta, LAYERS[0], 4) is None + + +def test_capture_supported_reads_the_op_schema() -> None: + # The schema of the op the boundaries call: this target's copy (m3_mc_allreduce.OWN_OP), else the image's. + name = ( + "minimax_m3_sm107_tp8_mc_allreduce_norm" + if m3_mc_allreduce.own_op() + else "minimax_m3_sm107_tp2_mc_allreduce_norm" + ) + op = getattr(torch.ops.trtllm, name, None) + want = op is not None and any(a.name == "cap_out" for a in op.default._schema.arguments) + assert m3_mc_allreduce.capture_supported() is want + + +def test_own_op_follows_the_kernel_constant(monkeypatch) -> None: + monkeypatch.setattr(m3_mc_allreduce, "OWN_OP", False) + assert m3_mc_allreduce.own_op() is False + monkeypatch.setattr(m3_mc_allreduce, "OWN_OP", True) + loaded = getattr(torch.ops.trtllm, "minimax_m3_sm107_tp8_mc_allreduce_norm", None) is not None + assert m3_mc_allreduce.own_op() is loaded diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_workspaces.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_workspaces.py new file mode 100644 index 000000000000..21eb199b7173 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_capture_workspaces.py @@ -0,0 +1,56 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The lazily created exchange workspaces refuse to be created inside a CUDA-graph capture (K3 audit #150, check #10). + +The draft-step and sampled-accept workspaces are created collectively (both ranks, a multicast buffer) on the first +eager call. Creating one inside a capture would allocate and synchronize mid-capture, so a capture that finds it missing +reports a miss (``kernels/m3_report.capture_missed``) and gets None (the caller then takes its stock path) instead. +""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( + m3_draft_step, + m3_report, + m3_sampled_accept, +) + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available(), reason="CUDA graph capture needs a GPU" +) + + +def _refuse(*args, **kwargs): + raise AssertionError("a workspace must not be created inside a CUDA-graph capture") + + +@pytest.mark.parametrize( + "module, creator", + [ + (m3_draft_step, "create_workspace"), + (m3_sampled_accept, "SampledAcceptWorkspace.create"), + ], + ids=["draft_step", "sampled_accept"], +) +def test_a_capture_without_the_workspace_misses_instead_of_creating_it( + monkeypatch, module, creator +): + monkeypatch.setattr(module, "_WS", None) + owner, _, attr = creator.rpartition(".") + monkeypatch.setattr(getattr(module, owner) if owner else module, attr, _refuse) + misses = [] + monkeypatch.setattr(m3_report, "capture_missed", lambda *args: misses.append(args)) + + device = torch.device("cuda", torch.cuda.current_device()) + found = [] + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + found.append(module.workspace(None, device)) + torch.cuda.current_stream().wait_stream(stream) + + assert found == [None] + assert module._WS is None + assert len(misses) == 1 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_chunk_norm.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_chunk_norm.py new file mode 100644 index 000000000000..9a145ae0f9e7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_chunk_norm.py @@ -0,0 +1,50 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_chunk_norm entry (the Eagle3 fc_norm over the captured layers).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_chunk_norm import ( # noqa: E501 + m3_chunk_norm as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 — needs the path declared above + EPS, + assert_bf16_close, + gen, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_chunk_norm requires a CUDA device" + + +def _case(spec, seed): + g = gen(seed) + m, c, h = spec["m"], spec["c"], spec["h"] + buf = randn((m, c * h + spec["extra_cols"]), g, 2.0) + src = buf[:, : c * h + spec["extra_cols"]] + w = (1.0 + randn((c, h), g, 0.1).float()).to(torch.bfloat16) + dst = torch.full((m, c * h), 7.0, dtype=torch.bfloat16, device="cuda") + return (src, w, EPS, dst, True) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Within one bf16 ulp per element on at most 0.1% of the elements (fp32 statistic, rsqrtf).""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + out = op(*args) + assert_bf16_close(out, op.reference(*args)) + + +def test_guard_refuses_a_strided_weight() -> None: + src, w, eps, dst, pdl = _case(op.CELLS[0].spec, 1) + wt = torch.empty((w.shape[1], w.shape[0]), dtype=w.dtype, device="cuda").t() + wt.copy_(w) + with pytest.raises(AssertionError, match="dense"): + with validating(op): + op(src, wt, eps, dst, pdl) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_di_fc_flag.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_di_fc_flag.py new file mode 100644 index 000000000000..9391b18383da --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_di_fc_flag.py @@ -0,0 +1,91 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test of the fc fork's counter wait (modeling's DI_FC_FLAG, kernels/m3_fc_wait): the one-thread wait kernel +queued on the main stream releases its successor only after the fork's bump on another stream (so the successor reads +what the fork wrote, with no stream join before it), eagerly and on CUDA-graph replays; it counts an overrun instead +of hanging when no bump comes.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_fc_wait, +) + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA device") + +N = 1 << 16 + + +def _wait(counts, spins=m3_fc_wait.MAX_SPINS): + m3_fc_wait.fc_wait_kernel[(1,)](counts, MAX_SPINS=spins, num_warps=1, launch_pdl=True) + + +def _step(counts, aux, src, dst, value, delay_cycles): + """One fork / wait / consume: the fork (after a delay) writes ``value`` into src and bumps the counter; the main + stream waits on the counter, copies src to dst, then joins the fork (deferred, as the worker does).""" + main = torch.cuda.current_stream() + aux.wait_stream(main) + with torch.cuda.stream(aux): + torch.cuda._sleep(delay_cycles) + src.fill_(value) + counts[:1].add_(1) + _wait(counts) + dst.copy_(src) + main.wait_stream(aux) + + +@pytest.mark.parametrize("delay_cycles", [0, 200_000, 2_000_000]) +def test_wait_releases_after_the_fork(delay_cycles) -> None: + counts = torch.zeros(3, dtype=torch.int64, device="cuda") + src = torch.zeros(N, dtype=torch.float32, device="cuda") + dst = torch.empty_like(src) + aux = torch.cuda.Stream() + for it in range(1, 9): + _step(counts, aux, src, dst, float(it), delay_cycles) + torch.cuda.synchronize() + assert bool((dst == float(it)).all()), f"iteration {it}: read before the fork finished" + assert counts.tolist() == [8, 8, 0] + + +def test_wait_releases_on_graph_replays() -> None: + counts = torch.zeros(3, dtype=torch.int64, device="cuda") + src = torch.zeros(N, dtype=torch.float32, device="cuda") + dst = torch.empty_like(src) + val = torch.zeros(1, dtype=torch.float32, device="cuda") + aux = torch.cuda.Stream() + cap = torch.cuda.Stream() + + def body(): + main = torch.cuda.current_stream() + aux.wait_stream(main) + with torch.cuda.stream(aux): + torch.cuda._sleep(1_000_000) + src.copy_(val.expand(N)) + counts[:1].add_(1) + _wait(counts) + dst.copy_(src) + main.wait_stream(aux) + + with torch.cuda.stream(cap): + body() # eager warm-up (compiles the wait kernel outside capture) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(cap): + with torch.cuda.graph(graph, stream=cap): + body() + torch.cuda.synchronize() + base = counts.tolist() + for it in range(1, 11): + val.fill_(float(it)) + graph.replay() + torch.cuda.synchronize() + assert bool((dst == float(it)).all()), f"replay {it}: read before the fork finished" + assert counts.tolist() == [base[0] + 10, base[1] + 10, 0] + + +def test_wait_without_a_fork_counts_an_overrun() -> None: + counts = torch.zeros(3, dtype=torch.int64, device="cuda") + _wait(counts, spins=1000) + torch.cuda.synchronize() + assert counts.tolist() == [0, 1, 1] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_input.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_input.py new file mode 100644 index 000000000000..8c14e7d61d07 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_input.py @@ -0,0 +1,98 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_draft_input entry (embedding + both RMSNorms + concat in one launch), against its reference and +against the in-tree drafter's modules.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_spec_tail, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_draft_input import ( # noqa: E501 + m3_draft_input as op, +) +from tensorrt_llm._torch.modules.embedding import Embedding +from tensorrt_llm._torch.modules.rms_norm import RMSNorm + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 — needs the path declared above + EPS, + HIDDEN, + VOCAB, + assert_bf16_close, + gen, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_draft_input requires a CUDA device" + + +def _case(spec, seed): + g = gen(seed) + m, h = spec["m"], spec["h"] + table = randn((VOCAB, h), g) + ids = torch.randint(0, VOCAB, (m,), generator=g, device="cuda", dtype=torch.int32) + if spec["invalid_id"]: + ids[0] = -1 + hidden = randn((4 * m if spec["gather"] else m, h), g) + rows = torch.arange(m, device="cuda", dtype=torch.int64) * 4 + 2 if spec["gather"] else None + w_e = (1.0 + randn((h,), g, 0.1).float()).to(torch.bfloat16) + w_h = (1.0 + randn((h,), g, 0.1).float()).to(torch.bfloat16) + out = torch.full((m, 2 * h), 7.0, dtype=torch.bfloat16, device="cuda") + residual = torch.full((m, h), 7.0, dtype=torch.bfloat16, device="cuda") + return (ids, table, hidden, rows, w_e, w_h, EPS, EPS, out, residual, True) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Normalized halves within one bf16 ulp per element on at most 0.1% of the elements (fp32 statistic in a + different summation order, rsqrtf); the gathered residual and the zero embedding of an invalid id exact.""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + out, residual = op(*args) + ref_out, ref_res = op.reference(*args) + torch.cuda.synchronize() + assert_bf16_close(out, ref_out) + assert torch.equal(residual, ref_res) + if cell.spec["invalid_id"]: + assert not bool(out[0, : cell.spec["h"]].float().any()) + + +def test_guard_refuses_rows_past_hidden() -> None: + gather = next(c for c in op.CELLS if c.spec["gather"]) + args = list(_case(gather.spec, 1)) + args[3] = args[3] + 100 + with pytest.raises(AssertionError, match="rows must index hidden"): + with validating(op): + op(*args) + + +@pytest.mark.parametrize("m,gather", [(4, False), (32, False), (8, True)], ids=str) +def test_the_in_tree_draft_input(m, gather) -> None: + """Against the in-tree drafter's own modules: the Embedding lookup, then the layer's input_layernorm and + hidden_norm (in-tree RMSNorm) and the concat of Eagle3DecoderLayer.forward. Within one bf16 ulp on at most 0.1% of + the elements (the RMS statistic summed in another order); the residual (the carried hidden rows) exact.""" + g = gen(seed_of(f"in-tree draft input {m} {gather}")) + h = HIDDEN + embed = Embedding(VOCAB, h, dtype=torch.bfloat16).cuda() + embed.weight.data = randn((VOCAB, h), g) + norms = [] + for _ in range(2): + norm = RMSNorm(hidden_size=h, eps=EPS, dtype=torch.bfloat16).cuda() + norm.weight.data = (1.0 + randn((h,), g, 0.1).float()).to(torch.bfloat16) + norms.append(norm) + input_layernorm, hidden_norm = norms + ids = torch.randint(0, VOCAB, (m,), generator=g, device="cuda", dtype=torch.int32) + hidden = randn((4 * m if gather else m, h), g, 2.0) + rows = torch.arange(m, device="cuda", dtype=torch.int64) * 4 + 3 if gather else None + carried = hidden[rows] if gather else hidden + stock = torch.cat([input_layernorm(embed(ids)), hidden_norm(carried)], dim=-1) + out, residual = m3_spec_tail.draft_input( + ids, embed.weight, hidden, rows, input_layernorm.weight, EPS, hidden_norm.weight, EPS + ) + torch.cuda.synchronize() + assert_bf16_close(out, stock) + assert torch.equal(residual, carried) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_push_take.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_push_take.py new file mode 100644 index 000000000000..aa924d9520db --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_push_take.py @@ -0,0 +1,43 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The drafter boundary push bookkeeping (``M3Eagle3Draft.arm_push`` / ``take_pushed``, ``DRAFT_AR_PUSH``): the +boundary all-reduce runs prepushed only when the armed call pushed, and only on that call's own output; any other +input raises instead of reducing rows the exchange does not hold. Host state only (no GPU).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) + + +@pytest.fixture +def draft(): + """A drafter with only its push bookkeeping (the module and its weights are not built).""" + d = object.__new__(modeling.M3Eagle3Draft) + d._push_arm, d._pushed = None, 0 + return d + + +def test_not_pushed_is_false_and_disarms(draft) -> None: + draft._push_arm = (1, 2, 3, 0, 8, 8) + assert draft.take_pushed(torch.zeros(4)) is False + assert draft._push_arm is None + + +def test_pushed_output_is_true_once(draft) -> None: + out = torch.zeros(4, 8) + draft._pushed = out.data_ptr() + assert draft.take_pushed(out) is True + assert draft._pushed == 0 + assert draft.take_pushed(out) is False + + +def test_other_input_raises(draft) -> None: + out = torch.zeros(4, 8) + other = out.clone() + draft._pushed = out.data_ptr() + with pytest.raises(RuntimeError, match="not the boundary all-reduce's input"): + draft.take_pushed(other) + assert draft._pushed == 0 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_silu.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_silu.py new file mode 100644 index 000000000000..15283d10a797 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_silu.py @@ -0,0 +1,88 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_draft_silu entry (the drafter MLP's plain SwiGLU as one C++ launch): bit for bit the Triton +silu_and_mul at its defaults, over every bf16 gate and every bf16 up pattern.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_draft_silu import ( # noqa: E501 + m3_draft_silu as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import gen, seed_of # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_draft_silu requires a CUDA device" + +ALL = torch.arange(-(1 << 15), 1 << 15, dtype=torch.int32).to(torch.int16).view(torch.bfloat16) + + +def _rows(gate: torch.Tensor, up: torch.Tensor) -> torch.Tensor: + """[rows, 2d] gate | up from equally long flat gate and up values, d = 2304 (the TP8 drafter shard).""" + d = 2304 + n = gate.numel() + rows = -(-n // d) + pad = rows * d - n + g = torch.cat([gate, gate[:pad]]).view(rows, d) + u = torch.cat([up, up[:pad]]).view(rows, d) + return torch.cat([g, u], dim=1).contiguous() + + +def _assert_bitwise(x: torch.Tensor, what: str) -> None: + got = op(x) + assert got is not None, what + ref = op.reference(x) + torch.cuda.synchronize() + diff = got.view(torch.int16) != ref.view(torch.int16) + assert not bool(diff.any()), f"{what}: {int(diff.sum())} of {diff.numel()} differ" + + +@pytest.mark.parametrize("up", [0.0, 1.0, -1.0, 3.5, -0.0, 1e-3], ids=lambda v: f"up={v}") +def test_every_gate_pattern_is_bitwise_the_triton_kernel(up) -> None: + gate = ALL.cuda() + _assert_bitwise(_rows(gate, torch.full_like(gate, up)), f"gate sweep, up {up}") + + +@pytest.mark.parametrize("gate", [1.0, -1.0, 0.5, -3.0, 20.0, -0.0], ids=lambda v: f"gate={v}") +def test_every_up_pattern_is_bitwise_the_triton_kernel(gate) -> None: + up = ALL.cuda() + _assert_bitwise(_rows(torch.full_like(up, gate), up), f"up sweep, gate {gate}") + + +@pytest.mark.parametrize("seed", range(4)) +def test_random_pairs_are_bitwise_the_triton_kernel(seed) -> None: + g = gen(100 + seed) + n = 1 << 22 + gate = torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda").to(torch.int16) + up = torch.randint(-(1 << 15), 1 << 15, (n,), generator=g, device="cuda").to(torch.int16) + _assert_bitwise(_rows(gate.view(torch.bfloat16), up.view(torch.bfloat16)), f"seed {seed}") + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """The drafter's gate_up output range (and past it), in a CUDA graph as the drafter runs it.""" + g = gen(seed_of(cell.why)) + rows, d = cell.spec["rows"], cell.spec["d"] + x = (4.0 * torch.randn(rows, 2 * d, generator=g, device="cuda")).to(torch.bfloat16) + x[:, :6] = torch.tensor([60.0, -60.0, 0.0, -0.0, 88.0, -88.0], dtype=torch.bfloat16) + certifying(op) + ref = op.reference(x) + eager = op(x) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = op(x) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(eager.view(torch.int16), ref.view(torch.int16)) + assert torch.equal(captured.view(torch.int16), ref.view(torch.int16)) + + +def test_declines_what_it_does_not_cover() -> None: + x = torch.zeros(2, 2 * 2304, dtype=torch.bfloat16, device="cuda") + assert op(x[:, 8:]) is None # strided rows + assert ( + op(torch.zeros(2, 2 * 12, dtype=torch.bfloat16, device="cuda")) is None + ) # d not a multiple of 8 + assert op(x.float()) is None diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_step.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_step.py new file mode 100644 index 000000000000..b32811c2ee9c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_step.py @@ -0,0 +1,747 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_draft_step entry (one fused greedy draft pass: the BF16 draft head, streamed from its lossless +12-bit coding or as BF16 fragments, the cross-rank argmax and the draft loop's bookkeeping), and its tokens against the +in-tree drafter's head and greedy draft sampling. + +The in-tree path: each rank's LM head shard GEMM (BF16 out), ``LogitsProcessor``'s float cast, then either the argmax +of the gathered logits (``torch.argmax``) or ``greedy_sample_draft_with_tp_gather``'s per-rank ``torch.max`` and the +pick among the ranks' (index, value) pairs. The edge cases use integer data, whose fp32 sums are exact in any order, +so the kernel and the in-tree path see the same BF16 logits and must agree bit for bit; every in-tree comparison runs +with the coded head (the default) and with the BF16 fragments, and on integer data the coded head's winning keys are +the BF16 fragments' bit for bit. +""" + +import warnings +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_draft_step as draft_step_kernels, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_target_head import ( # noqa: E501 + encode_chunked, + round_trip_ok, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_draft_step import ( # noqa: E501 + DRAFT_LEN, + DRAFT_SHARD, + HIDDEN, + MAX_TOKENS, + bf16_fragments, + from_fragments, + head_weight, + max_tile_pieces, + shard_logits, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_draft_step import ( # noqa: E501 + m3_draft_step as op, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_shard_accept import ( # noqa: E501 + exchange_tokens, + local_argmax, + make_keys, +) +from tensorrt_llm._torch.speculative.eagle3 import Eagle3OneModelWorker + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 — needs the path declared above + WORLD, + gen, + head_workspace, + num_sms, + plant_peers, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_draft_step requires a CUDA device" + +T = DRAFT_LEN + 1 +_LOW32 = 0xFFFFFFFF + +# ------------------------------------------------------------------ the per-row rule against the in-tree token +# +# The kernel and the in-tree head (a cuBLAS BF16 GEMM) both round an FP32 sum of the same exact BF16 products to BF16, +# in different summation orders. A row's token can differ from the in-tree one only where some candidate's FP32 sum +# can land on either side of a BF16 rounding midpoint. Each row is held to: +# +# kernel token == in-tree token, or the row is provably undecidable: with the fp64 reference logit L_i (exact BF16 +# products, fp64 sum) and the FP32-order bound E_i = gamma_K * sum_k |w_ik * x_k|, the BF16 value of any FP32 sum of +# candidate i lies in [lo_i, hi_i] = [bf16(L_i - E_i), bf16(L_i + E_i)]; the kernel's token must win for some +# choice of values within those intervals (larger value, then the lower global id), and a BF16 rounding midpoint +# must lie within E of the kernel's or the in-tree token's L. +# +# gamma_K = K * 2^-22 is la-xh2's bound (4x the round-to-nearest unit roundoff 2^-24 per addition: any summation +# order, and a truncating tensor-core accumulator); K counts the row's non-zero products (adding an exact zero does +# not round). lo / hi take one more FP32 step outward before the BF16 rounding, which covers rounding L -+ E to FP32. +# Each undecidable row is reported as a warning with its midpoint distances, and each test reports its row count; any +# other mismatch fails. +GAMMA_UNIT = 2.0**-22 + + +def _bf16_outward(v: torch.Tensor, direction: float) -> torch.Tensor: + f = v.float() + f = torch.nextafter(f, torch.full_like(f, direction)) + b = f.to(torch.bfloat16).float() + return torch.where(b == 0, torch.zeros_like(b), b) + + +def bounds(h: torch.Tensor, w: torch.Tensor): + """(fp64 logits L, bound E, lo, hi) [rows, N] of the head input rows h against the head rows w.""" + cols = (h != 0).any(dim=0) # zero columns add exact zeros + hd, wd = h[:, cols].double(), w[:, cols].double() + logits = hd @ wd.t() + bound = (h != 0).sum(dim=1, keepdim=True).double() * GAMMA_UNIT * (hd.abs() @ wd.abs().t()) + del wd + inf = float("inf") + return logits, bound, _bf16_outward(logits - bound, -inf), _bf16_outward(logits + bound, inf) + + +def midpoint_distance(v: float) -> float: + """Distance of v to the nearest BF16 rounding midpoint.""" + b = torch.tensor(v, dtype=torch.float64).float().to(torch.bfloat16).float() + if float(b) == 0.0: + return abs(v) + ulp = 2.0 ** (int(torch.frexp(b).exponent) - 8) + return min(abs(v - (float(b) + ulp / 2)), abs(v - (float(b) - ulp / 2))) + + +def _winnable(tok: int, ids: torch.Tensor, lo: torch.Tensor, hi: torch.Tensor) -> bool: + """Whether candidate ``tok`` wins the row for some values within the candidates' [lo, hi].""" + k = int((ids == tok).nonzero()[0, 0]) + others = ids != tok + beat = (hi[k] > lo[others]) | ((hi[k] == lo[others]) & (tok < ids[others])) + return bool(beat.all()) + + +def check_row(name: str, kernel_tok: int, stock_tok: int, ids, logits, bound, lo, hi) -> bool: + """The per-row rule for one row over the candidates ``ids`` [C] (logits / bound / lo / hi [C]); True when the + row is undecidable (reported as a warning), False when the tokens match; a mismatch outside the bound fails.""" + if kernel_tok == stock_tok: + return False + assert _winnable(kernel_tok, ids, lo, hi), ( + f"{name}: token {kernel_tok} differs from the in-tree {stock_tok} outside the FP32-order bound" + ) + ik = int((ids == kernel_tok).nonzero()[0, 0]) + i_stock = int((ids == stock_tok).nonzero()[0, 0]) + lk, ls, ek, es = ( + float(logits[ik]), + float(logits[i_stock]), + float(bound[ik]), + float(bound[i_stock]), + ) + dk, ds = midpoint_distance(lk), midpoint_distance(ls) + assert dk <= ek or ds <= es, f"{name}: neither logit is within its bound of a BF16 midpoint" + warnings.warn( + f"[draft-step undecidable] {name}: kernel {kernel_tok} (L {lk:.9g}, midpoint distance {dk:.3e}, bound " + f"{ek:.3e}) vs in-tree {stock_tok} (L {ls:.9g}, midpoint distance {ds:.3e}, bound {es:.3e})" + ) + return True + + +def report(name: str, rows: int, undecidable: int) -> None: + warnings.warn(f"[draft-step rows] {name}: {rows} rows, {undecidable} undecidable") + + +def key_value(keys: torch.Tensor) -> torch.Tensor: + """The fp32 value a key orders by (the inverse of the kernel's order-preserving image).""" + u = (keys >> 32) & _LOW32 + bits = torch.where(u >= 0x80000000, u - 0x80000000, _LOW32 - u) + bits = torch.where(bits >= 2**31, bits - 2**32, bits) # as a signed 32-bit pattern + return bits.to(torch.int32).view(torch.float32) + + +def key_index(keys: torch.Tensor) -> torch.Tensor: + return _LOW32 - (keys & _LOW32) + + +def _bookkeeping(nreq: int, step: int, n_x_rows: int, g: torch.Generator): + num_acc = ( + torch.randint(1, T + 1, (nreq,), generator=g, device="cuda", dtype=torch.int32) + if step == 0 + else None + ) + seq_lens = torch.full((nreq,), T, dtype=torch.int32, device="cuda") if step == 0 else None + next_new = torch.full((nreq, T), -5, dtype=torch.int32, device="cuda") + pos_src = torch.randint(100, 3000, (n_x_rows,), generator=g, device="cuda", dtype=torch.int32) + kv_lens = torch.randint(200, 3000, (nreq,), generator=g, device="cuda", dtype=torch.int32) + ids_next = torch.full((nreq,), -9, dtype=torch.int32, device="cuda") + pos_next = torch.full((nreq,), -9, dtype=torch.int32, device="cuda") + return num_acc, seq_lens, next_new, pos_src, kv_lens, ids_next, pos_next + + +def _head(w: torch.Tensor, coded: bool): + """(wf, (exc_off, exc_pos, exc_w, exp_base)) of a BF16 shard: its 12-bit coding (checked to round-trip) or its + BF16 fragments.""" + if not coded: + return bf16_fragments(w), (None, None, None, 0) + packed, base, exc_off, exc_pos, exc_w = encode_chunked(w) + assert round_trip_ok(packed, base, exc_off, exc_pos, exc_w, w) + return packed, (exc_off, exc_pos, exc_w.view(-1), base) + + +def _args( + x, + x_rows, + nreq, + wf, + rank, + world, + step, + g, + uc=None, + keys=None, + counters=None, + num_acc=None, + coded=(None, None, None, 0), +): + """The op's argument tuple for one rank (fresh workspace and bookkeeping buffers unless given); ``coded`` = the + coded head's (exc_off, exc_pos, exc_w, exp_base).""" + acc, seq_lens, next_new, pos_src, kv_lens, ids_next, pos_next = _bookkeeping( + nreq, step, x.shape[0], g + ) + if num_acc is None: + num_acc = acc + if uc is None: + uc, keys, counters = head_workspace(world, slot_rows=MAX_TOKENS) + out_keys = torch.zeros(nreq, dtype=torch.int64, device="cuda") + tiles = wf.shape[0] if coded[0] is None else coded[0].numel() - 1 + return (x, x_rows, nreq, wf, rank * tiles * 16, keys, counters, uc, uc.data_ptr(), rank, world, + next_new[:, 1:], step, ids_next, pos_src, pos_next, kv_lens, num_acc, seq_lens, DRAFT_LEN, num_sms(), + True, out_keys, *coded) # fmt: skip + + +def _fixed(args) -> bool: + """fixed_order for these op arguments as dispatch picks it (kernels/m3_draft_step._fixed_order): the warp-order + sum where the launch grid lets a tile take more than 2 warp pieces (D-FIX; the TP8 shard on 212 SMs does).""" + x, wf, num_ctas, exc_off = args[0], args[3], args[20], args[23] + tiles = wf.shape[0] if exc_off is None else exc_off.numel() - 1 + return max_tile_pieces(tiles, num_ctas, x.shape[1] // (16 if exc_off is None else 64)) > 2 + + +def _case(spec, seed): + g = gen(seed) + nreq, step, rank, world, vs, k = ( + spec["nreq"], + spec["step"], + spec["rank"], + spec["world"], + spec["vs"], + spec["k"], + ) + pass0 = step == 0 + x = randn((nreq * T if pass0 else nreq, k), g) + wf, coded = _head(randn((vs, k), g, 0.02), spec["coded"]) + num_acc = ( + torch.randint(1, T + 1, (nreq,), generator=g, device="cuda", dtype=torch.int32) + if pass0 + else None + ) + x_rows = (torch.arange(nreq, device="cuda") * T + num_acc.long() - 1) if pass0 else None + uc, keys, counters = head_workspace(world, slot_rows=MAX_TOKENS) + h = x[x_rows] if pass0 else x + own = shard_logits(h, wf, *coded).max(dim=1).values + plant_peers( + uc, counters, own, rank, world, vs, g, slot_rows=MAX_TOKENS, ties=spec.get("ties", False) + ) + return _args(x, x_rows, nreq, wf, rank, world, step, g, uc, keys, counters, num_acc, coded) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """The bookkeeping is exact on every row (next positions, KV and sequence lengths). Each row's token is the + in-tree one (this rank's BF16 shard logits from the head's GEMM, their first maximum, against the peers' keys) or + the row is provably undecidable (check_row); the winning key's index is the token, and its value lies in the + token's [lo, hi] (or is the peer's pushed key when a peer wins).""" + args = _case(cell.spec, seed_of(cell.why)) + x, x_rows, nreq, wf, offset, _, counters, uc = args[:8] + rank, world = args[9], args[10] + half = int(counters[1].item()) & 1 + pushed = ( + uc[half * MAX_TOKENS * world : (half * MAX_TOKENS + nreq) * world].view(nreq, world).clone() + ) + h = x[x_rows] if x_rows is not None else x[:nreq] + w = head_weight(wf, x.shape[1], *args[23:27]) + best, idx = local_argmax(torch.nn.functional.linear(h, w).float(), offset) + stock = exchange_tokens(make_keys(best, idx), uc, counters, rank, world, MAX_TOKENS) + certifying(op) + ref = op.reference(*args) + op(*args, fixed_order=_fixed(args)) + torch.cuda.synchronize() + assert args[6].tolist() == [0, 1] and not bool(args[5].any()), "counters / keys not re-armed" + step = args[12] + others = [c for c in range(args[11].shape[1]) if c != step] + assert torch.equal(args[11][:, others], ref["draft_out"][:, others]) + assert torch.equal(args[11][:, step], args[13]), "draft_out and ids_next disagree" + assert torch.equal(args[15], ref["pos_next"]) + assert torch.equal(args[16], ref["kv_lens"]) + if args[18] is not None: + assert torch.equal(args[18], ref["seq_lens"]) + # Candidates per row: this rank's head rows (fp64 bounds) and each peer's pushed key (an exact value). + logits, bound, lo, hi = bounds(h, w) + own_ids = offset + torch.arange(w.shape[0], device="cuda") + peers = [p for p in range(world) if p != rank] + undecidable = 0 + for r in range(nreq): + pv = key_value(pushed[r, peers]).double() + ids = torch.cat([own_ids, key_index(pushed[r, peers])]) + undecidable += check_row( + f"{cell.why[:40]} row {r}", + int(args[13][r]), + int(stock[r]), + ids, + torch.cat([logits[r], pv]), + torch.cat([bound[r], torch.zeros_like(pv)]), + torch.cat([lo[r], pv.float()]), + torch.cat([hi[r], pv.float()]), + ) + report(cell.why[:40], nreq, undecidable) + out_keys = args[22] + assert torch.equal(key_index(out_keys), args[13].long()), "the winning key's index is the token" + got = key_value(out_keys) + for r in range(nreq): + c = int(args[13][r]) - offset + if 0 <= c < w.shape[0]: + assert float(lo[r, c]) <= float(got[r]) <= float(hi[r, c]), ( + f"row {r}: winning value outside [lo, hi]" + ) + else: + assert int(out_keys[r]) in pushed[r].tolist(), ( + f"row {r}: a peer won with a key it did not push" + ) + + +def test_fragments_round_trip() -> None: + g = gen(seed_of("fragments")) + w = randn((64, 96), g) + wf = bf16_fragments(w) + assert wf.shape == (4, 6, 256) + assert torch.equal(from_fragments(wf), w) + head = SimpleNamespace(weight=w, bias=None) + prepared = draft_step_kernels.prepare_head(head, 3) # K 96: no coding, the BF16 fragments + assert not prepared.coded + assert torch.equal(prepared.wf, wf) and prepared.vocab_offset == 3 * 64 and prepared.weight is w + + +# ------------------------------------------------------------------ against the in-tree drafter, bit for bit + + +def _stock_tokens(shards: list, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """The in-tree greedy draft tokens of the rows x over the vocab-sharded BF16 head: (argmax of the gathered float + logits, greedy_sample_draft_with_tp_gather's per-rank (index, value) pick).""" + logits = [torch.nn.functional.linear(x, w).float() for w in shards] + gathered = torch.argmax(torch.cat(logits, dim=-1), dim=-1).to(torch.int32) + combined = [] + for rank, shard_logit in enumerate(logits): + worker = SimpleNamespace(mapping=SimpleNamespace(tp_rank=rank)) + combined.append(Eagle3OneModelWorker._get_local_max_and_combined(worker, shard_logit)) + pick = Eagle3OneModelWorker._get_draft_tokens_from_gathered + pick = getattr(pick, "_torchdynamo_orig_callable", pick) + tp_gather = pick(None, torch.cat(combined, dim=-1)).to(torch.int32) + return gathered, tp_gather + + +def _fused_tokens(shards: list, x: torch.Tensor, coded: bool, step: int = 1) -> list: + """Each rank's fused draft tokens: every rank's own key from a one-rank run, then each rank's four-rank pass with + its peers' keys in its slots, as their multicast pushes would put them.""" + g = gen(seed_of("fused tokens")) + nreq = x.shape[0] + heads = [_head(w, coded) for w in shards] + own_keys = [] + for rank, (wf, cd) in enumerate(heads): + args = list(_args(x, None, nreq, wf, rank, 1, step, g, coded=cd)) + args[9] = 0 # one-rank run: this rank's slot only, global ids of `rank` + op(*args, fixed_order=_fixed(args)) + own_keys.append(args[22].clone()) + tokens = [] + for rank, (wf, cd) in enumerate(heads): + uc, keys, counters = head_workspace(WORLD, slot_rows=MAX_TOKENS) + for peer in range(WORLD): + if peer != rank: + for t in range(nreq): + uc[t * WORLD + peer] = own_keys[peer][t] + args = _args(x, None, nreq, wf, rank, WORLD, step, g, uc, keys, counters, coded=cd) + with validating(op): + op(*args, fixed_order=_fixed(args)) + torch.cuda.synchronize() + assert torch.equal(args[11][:, step], args[13]), "draft_out and ids_next disagree" + tokens.append(args[13].clone()) + return tokens + + +def _integer_head(g: torch.Generator, rows: int = WORLD * DRAFT_SHARD) -> torch.Tensor: + """A [rows, 6144] head of small integers over the first 4 columns (zero elsewhere): its logits against integer + rows are exact integers in any summation order, many of which share a BF16 value.""" + w = torch.zeros((rows, HIDDEN), dtype=torch.bfloat16, device="cuda") + w[:, :4] = torch.randint(-60, 61, (rows, 4), generator=g, device="cuda").to(torch.bfloat16) + return w + + +def _shards(w: torch.Tensor) -> list: + return [w[r * DRAFT_SHARD : (r + 1) * DRAFT_SHARD].contiguous() for r in range(WORLD)] + + +def _assert_stock(w: torch.Tensor, x: torch.Tensor, coded: bool, expect=None) -> None: + shards = _shards(w) + gathered, tp_gather = _stock_tokens(shards, x) + assert torch.equal(gathered, tp_gather), "the two in-tree pickers disagree" + if expect is not None: + assert gathered.tolist() == expect, (gathered.tolist(), expect) + for rank, tokens in enumerate(_fused_tokens(shards, x, coded)): + assert torch.equal(tokens, gathered), ( + f"rank {rank}: {tokens.tolist()} vs stock {gathered.tolist()}" + ) + + +def _ones_rows(nreq: int) -> torch.Tensor: + x = torch.zeros((nreq, HIDDEN), dtype=torch.bfloat16, device="cuda") + x[:, :4] = 1.0 + return x + + +@pytest.mark.parametrize("coded", [True, False], ids=["coded", "bf16"]) +def test_forced_bf16_ties_take_the_lowest_global_id(coded) -> None: + """A head row copied onto every rank that wins every row of ones (logit 244, every other at most 240): the lowest + global id must win, inside a shard and across ranks. Then integer rows in [-2, 2], whose logits (up to 480 in + magnitude, BF16 spacing 2 above 256) collide often, exactly or after BF16 rounding.""" + g = gen(seed_of("forced ties")) + w = _integer_head(g) + win = torch.tensor([61.0, 61.0, 61.0, 61.0], device="cuda").to(torch.bfloat16) + for gid in (DRAFT_SHARD + 5000, DRAFT_SHARD + 77, DRAFT_SHARD + 9000, 23750): + w[gid, :4] = win + _assert_stock(w, _ones_rows(5), coded, expect=[23750] * 5) + # Random integer rows: ties after BF16 rounding happen naturally. + x = torch.randint(-2, 3, (8, HIDDEN), generator=g, device="cuda").to(torch.bfloat16) + x[:, 4:] = 0 + _assert_stock(w, x, coded) + + +@pytest.mark.parametrize("coded", [True, False], ids=["coded", "bf16"]) +def test_signed_zero_maxima_take_the_lowest_global_id(coded) -> None: + """Rows whose maximum is zero: an all-zero head input (every logit +-0) and logits <= 0 with zeros by cancellation + on several ranks. -0 and +0 compare equal, so the lowest global id reaching zero wins.""" + w = torch.zeros((WORLD * DRAFT_SHARD, HIDDEN), dtype=torch.bfloat16, device="cuda") + w[:, 0] = -1.0 + w[:, 1] = -1.0 + for gid in (DRAFT_SHARD + 5, DRAFT_SHARD + 40000, DRAFT_SHARD + 70007, 24999): + w[gid, 0] = 1.0 + w[gid, 1] = -1.0 + x = torch.zeros((3, HIDDEN), dtype=torch.bfloat16, device="cuda") + x[1, :2] = 1.0 + x[2, :2] = 2.0 + x[2, 2] = -1.0 # w[:, 2] is zero: a -0 product on every row + _assert_stock(w, x, coded, expect=[0, 24999, 24999]) + + +@pytest.mark.parametrize( + "lower,higher,winner", + [ + (1000, 1001, "lower"), # 1001 rounds to 1000 in BF16: a tie, the lower id wins + (1001, 1002, "lower"), # 1001 and the tie-to-even midpoint 1002 both round to 1000 + (1000, 1004, "higher"), # adjacent BF16 values: the larger wins + (1003, 1000, "lower"), # 1003 rounds up to 1004 > 1000 + ], + ids=["one_fp32_step", "midpoint", "adjacent_bf16", "rounds_up"], +) +@pytest.mark.parametrize("where", ["in_shard", "across_ranks"]) +@pytest.mark.parametrize("coded", [True, False], ids=["coded", "bf16"]) +def test_near_equal_top2_follow_the_bf16_logits(lower, higher, winner, where, coded) -> None: + """Top-2 logits one to four fp32 steps apart: the winner is the one the BF16 logits decide (the in-tree logits' + dtype), not the fp32 sum. The pair sits inside rank 1's shard or on ranks 0 and 1; every other logit is at most + 240 in magnitude.""" + g = gen(seed_of(f"near top2 {lower} {higher} {where}")) + w = _integer_head(g) + if where == "in_shard": + lo_id, hi_id = DRAFT_SHARD + 100, DRAFT_SHARD + 200 + else: + lo_id, hi_id = DRAFT_SHARD - 1000, DRAFT_SHARD + 3 + for gid, value in ((lo_id, lower), (hi_id, higher)): + q, r = divmod(value, 4) # four integers <= 251, exact in BF16 + w[gid, :4] = torch.tensor([q + (i < r) for i in range(4)], device="cuda").to(torch.bfloat16) + expect = lo_id if winner == "lower" else hi_id + _assert_stock(w, _ones_rows(2), coded, expect=[expect, expect]) + + +# ------------------------------------------------------------------ non-integer rows at a BF16 rounding midpoint + +#: A BF16 rounding midpoint (between 6.0 and 6.03125; a tie there rounds to the even 6.0) and the FP32 spacing there. +MIDPOINT = 6.0 + 2.0**-6 +FP32_ULP = 2.0**-21 + + +def _near_midpoint_head(g: torch.Generator, targets: dict): + """A full-vocabulary head [8 * 25008, 6144] and one head-input row x with 65 non-zero entries: 1, 2^-12, 2^-20 + and 2^-28 on columns 0..3 and 61 random non-integer BF16 values. Every head row carries random non-integer BF16 + values on those 61 columns (logits about 0.2 in magnitude); each row of ``targets`` (global id -> fp64 value) also + gets columns 0..3 set so its fp64 logit is the target to within 2^-30 (a coarse value, then three finer + corrections, each BF16-rounded).""" + x = torch.zeros((1, HIDDEN), dtype=torch.bfloat16, device="cuda") + scales = (1.0, 2.0**-12, 2.0**-20, 2.0**-28) + for col, scale in enumerate(scales): + x[0, col] = scale + x[0, 4:65] = randn((61,), g, 0.5) + w = torch.zeros((WORLD * DRAFT_SHARD, HIDDEN), dtype=torch.bfloat16, device="cuda") + w[:, 4:65] = randn((WORLD * DRAFT_SHARD, 61), g, 0.05) + xd = x[0].double() + for gid, target in targets.items(): + rest = float(target - (w[gid].double() * xd).sum()) + for col, scale in enumerate(scales): + v = torch.tensor(rest / scale, dtype=torch.float64).to(torch.bfloat16) + w[gid, col] = v + rest -= float(v) * scale + assert abs(rest) <= 2.0**-30, (gid, rest) + return w, x + + +def _check_fused_against_stock( + name: str, w: torch.Tensor, x: torch.Tensor, coded: bool, expect=None +) -> int: + """Each rank's fused tokens against the in-tree tokens under the per-row rule (full-vocabulary candidates); + ``expect`` (a token per row) additionally pins rows the bound decides. Returns the undecidable rows.""" + shards = _shards(w) + gathered, tp_gather = _stock_tokens(shards, x) + assert torch.equal(gathered, tp_gather), "the two in-tree pickers disagree" + fused = _fused_tokens(shards, x, coded) + for tokens in fused[1:]: + assert torch.equal(tokens, fused[0]), "the ranks disagree on the global token" + logits, bound, lo, hi = bounds(x, w) + ids = torch.arange(w.shape[0], device="cuda") + undecidable = 0 + for r in range(x.shape[0]): + undecidable += check_row( + f"{name} row {r}", + int(fused[0][r]), + int(gathered[r]), + ids, + logits[r], + bound[r], + lo[r], + hi[r], + ) + if expect is not None: + assert int(gathered[r]) == expect[r] and int(fused[0][r]) == expect[r], ( + name, + int(gathered[r]), + int(fused[0][r]), + expect[r], + ) + others = ids != expect[r] + assert _winnable(expect[r], ids, lo[r], hi[r]), ( + f"{name}: the bound does not let {expect[r]} win" + ) + rivals = [int(i) for i in ids[others][hi[r][others] >= lo[r][expect[r]]]] + assert not any(_winnable(i, ids, lo[r], hi[r]) for i in rivals), ( + f"{name}: the row is not decided" + ) + report(name, x.shape[0], undecidable) + return undecidable + + +@pytest.mark.parametrize( + "lower,higher,winner", + [ + ( + -2, + 2, + None, + ), # straddling the midpoint by 2 FP32 ulps: either side is in-tree up to the order + (2, -2, None), + (0, -2, None), # exactly on the midpoint (ties to the even 6.0) against just below it + ( + -2048, + 2048, + "higher", + ), # ~1e-3 either side, well outside the bound: decided by BF16 rounding + ( + -4096, + -2048, + "lower", + ), # both below the midpoint, ~1e-3 apart: a BF16 tie the lower id wins + ], + ids=["straddle", "straddle_reversed", "on_midpoint", "decided_up", "decided_tie"], +) +@pytest.mark.parametrize("where", ["in_shard", "across_ranks"]) +@pytest.mark.parametrize("coded", [True, False], ids=["coded", "bf16"]) +def test_top2_at_a_bf16_midpoint(lower, higher, winner, where, coded) -> None: + """Non-integer rows whose top-2 fp64 logits sit the given number of FP32 ulps from a BF16 rounding midpoint, inside + rank 1's shard or on ranks 0 and 1, every other logit far below. The per-row rule holds on every row; outside the + bound the token is the one the BF16 rounding decides, bit for bit.""" + g = gen(seed_of(f"midpoint {lower} {higher} {where}")) + if where == "in_shard": + lo_id, hi_id = DRAFT_SHARD + 100, DRAFT_SHARD + 200 + else: + lo_id, hi_id = DRAFT_SHARD - 1000, DRAFT_SHARD + 3 + w, x = _near_midpoint_head( + g, {lo_id: MIDPOINT + lower * FP32_ULP, hi_id: MIDPOINT + higher * FP32_ULP} + ) + expect = None if winner is None else [lo_id if winner == "lower" else hi_id] + _check_fused_against_stock(f"midpoint {where} {lower:+d}/{higher:+d} ulp", w, x, coded, expect) + + +# ------------------------------------------------------------------ the coded head against the BF16 fragments + + +@pytest.mark.parametrize("nreq", [1, 2, 8]) +def test_coded_head_is_the_bf16_head_bit_for_bit(nreq) -> None: + """Integer data (fp32 sums exact in any order) with exceptions on both sides of the exponent window: the coded + head's winning keys (value and global id) and its bookkeeping are the BF16 fragments' bit for bit, on one rank and + across the ranks.""" + g = gen(seed_of(f"coded vs bf16 {nreq}")) + sign = torch.randint(0, 2, (DRAFT_SHARD, HIDDEN), generator=g, device="cuda") * 2 - 1 + w = (torch.randint(1, 41, (DRAFT_SHARD, HIDDEN), generator=g, device="cuda") * sign).to( + torch.bfloat16 + ) + # Exceptions on both sides of the window of 1..40: exact zeros, and 2^15 (fp32 sums stay below 2^24). + w[::5, ::512] = 0.0 + w[::13, 7::600] = 32768.0 + x = torch.randint(-2, 3, (nreq, HIDDEN), generator=g, device="cuda").to(torch.bfloat16) + wf_c, cd = _head(w, True) + assert cd[1].numel() > 0, "the shard must carry exceptions" + wf_b, cb = _head(w, False) + outs = [] + for wf, c in ((wf_c, cd), (wf_b, cb)): + args = _args(x, None, nreq, wf, 1, 1, 1, gen(seed_of("bookkeeping")), coded=c) + args = list(args) + args[9] = 0 + with validating(op): + op(*args, fixed_order=_fixed(args)) + torch.cuda.synchronize() + outs.append(args) + coded_args, bf16_args = outs + for i in (11, 13, 15, 16, 22): # draft_out, ids_next, pos_next, kv_lens, out_keys + assert torch.equal(coded_args[i], bf16_args[i]), i + best, idx = local_argmax(torch.nn.functional.linear(x, w).float(), DRAFT_SHARD) + assert torch.equal(coded_args[22], make_keys(best, idx)), ( + "the winning keys are not the in-tree ones" + ) + + +@pytest.mark.parametrize("nreq", [1, 2, 8]) +def test_tiny_exception_products_are_exact(nreq) -> None: + """Weights of 1 (exponent window base 113) with column 0 zero except 2^-30 / 2^-29 on rows 0 / 1, and a one-hot + activation on column 0: the exceptions' exact products are the only non-zero logits, so the coded head must pick + row 1 with the BF16 head's key bit for bit (a correction against a non-zero placeholder cancels them to 0).""" + n, k = 256, 1024 + w = torch.ones((n, k), dtype=torch.bfloat16, device="cuda") + w[:, 0] = 0.0 + w[0, 0] = 2.0**-30 + w[1, 0] = 2.0**-29 + x = torch.zeros((nreq, k), dtype=torch.bfloat16, device="cuda") + x[:, 0] = 1.0 + wf_c, cd = _head(w, True) + assert cd[3] == 113 + keys = [] + for wf, c in ((wf_c, cd), _head(w, False)): + args = _args(x, None, nreq, wf, 0, 1, 1, gen(seed_of("tiny exceptions")), coded=c) + with validating(op): + # 16 tiles on an all-SM grid: every tile is streamed by all 16 warps, which only the fixed-order sum + # (D-FIX) adds deterministically (is_valid rejects the atomics there). + op(*args, fixed_order=True) + torch.cuda.synchronize() + assert args[13].tolist() == [1] * nreq + keys.append(args[22]) + best, idx = local_argmax(torch.nn.functional.linear(x, w).float(), 0) + assert torch.equal(keys[0], keys[1]) and torch.equal(keys[0], make_keys(best, idx)) + + +def test_prepare_head_codes_the_shard_and_fails_closed(monkeypatch) -> None: + """With the coding on, prepare_head keeps the coded shard when it decodes to the weight bit for bit; a coding + that does not is dropped for the BF16 fragments. (The tp8 target ships HEAD_CODED = False.)""" + monkeypatch.setattr(draft_step_kernels, "HEAD_CODED", True) + g = gen(seed_of("prepare coded")) + w = randn((4096, 1024), g, 0.02) + head = SimpleNamespace(weight=w, bias=None) + prepared = draft_step_kernels.prepare_head(head, 2) + assert prepared.coded and prepared.vocab_offset == 2 * 4096 and prepared.weight is w + assert torch.equal(head_weight(prepared.wf, 1024, prepared.exc_off, prepared.exc_pos, prepared.exc_w, + prepared.exp_base), w) # fmt: skip + + def corrupt(weight, *args, **kwargs): + packed, base, exc_off, exc_pos, exc_w = encode_chunked(weight) + packed = packed.clone() + packed[11] ^= 0x10 + return packed, base, exc_off, exc_pos, exc_w + + monkeypatch.setattr(draft_step_kernels, "encode_chunked", corrupt) + dropped = draft_step_kernels.prepare_head(head, 2) + assert not dropped.coded and dropped.exc_off is None + assert torch.equal(dropped.wf, bf16_fragments(w)) + + +def test_prepare_head_keeps_bf16_when_coding_is_off(monkeypatch) -> None: + """HEAD_CODED = False keeps the BF16 fragments even for a shard whose coding would decode bit for bit.""" + g = gen(seed_of("prepare bf16")) + w = randn((4096, 1024), g, 0.02) + monkeypatch.setattr(draft_step_kernels, "HEAD_CODED", False) + prepared = draft_step_kernels.prepare_head(SimpleNamespace(weight=w, bias=None), 2) + assert not prepared.coded and prepared.exc_off is None + assert torch.equal(prepared.wf, bf16_fragments(w)) + + +def test_guard_refuses_head_rows_past_x() -> None: + args = list(_case(op.CELLS[0].spec, 1)) + args[1] = args[1] + 1000 + with pytest.raises(AssertionError, match="x_rows must index x"): + with validating(op): + op(*args, fixed_order=_fixed(args)) + + +def test_guard_refuses_too_few_ctas() -> None: + args = list(_case(op.CELLS[1].spec, 2)) + args[20] = 8 + with pytest.raises(AssertionError, match="tiles per CTA"): + with validating(op): + op(*args, fixed_order=_fixed(args)) + + +# ------------------------------------------------- one pass as two launches (uGPU, kernels/m3_draft_step.UGPU) + + +@pytest.mark.parametrize( + "cell", + [c for c in op.CELLS if not c.spec["coded"]], + ids=[c.why[:44] for c in op.CELLS if not c.spec["coded"]], +) +def test_two_launches_per_domain_are_the_one_launch_pass(cell) -> None: + """A pass split into two launches (tiles [0, split) and [split, n), one per locality domain stream, the exchange + and bookkeeping by the last of both launches' CTAs) writes exactly what the one-launch pass writes: tokens, + winning keys, draft columns, next positions, KV and sequence lengths, and re-arms the counters and keys.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_locality import ( # noqa: E501 + LocalityMap, + ) + + one = _case(cell.spec, seed_of(cell.why)) + two = _case(cell.spec, seed_of(cell.why)) + op(*one) + lm = LocalityMap.get(torch.cuda.current_device()) + streams = lm.domain_streams() + n_tiles = two[3].shape[0] + split = max(1, n_tiles // 2) + per = min(lm.sms_per_domain, split, n_tiles - split) + main = torch.cuda.current_stream() + fork = torch.cuda.Event() + fork.record(main) + for st, lo, hi in ((streams[0], 0, split), (streams[1], split, n_tiles)): + st.wait_event(fork) + with torch.cuda.stream(st): + op(*two[:20], per, False, *two[22:], tile_lo=lo, tile_hi=hi, total_ctas=2 * per) + for st in streams: + done = torch.cuda.Event() + done.record(st) + main.wait_event(done) + torch.cuda.synchronize() + for i, name in ( + (11, "draft_out"), + (13, "ids_next"), + (15, "pos_next"), + (16, "kv_lens"), + (22, "out_keys"), + ): + assert torch.equal(one[i], two[i]), name + if one[18] is not None: + assert torch.equal(one[18], two[18]), "seq_lens" + assert two[6].tolist() == [0, 1] and not bool(two[5].any()), "counters / keys not re-armed" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_step_fixed_order.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_step_fixed_order.py new file mode 100644 index 000000000000..5871685e4420 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_draft_step_fixed_order.py @@ -0,0 +1,150 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""D-FIX: the draft step's tile pieces and their summation order (kernels/m3_draft_step.FIXED_ORDER). + +A CTA's (tile, chunk) range is cut into 16 equal warp segments; the shared-memory atomics add a tile's pieces in +arrival order, which is deterministic only while no tile takes more than 2 pieces. ``fixed_order`` sums them in warp +order instead. The geometry tests enumerate the real grid (the shard on 212 SMs) so a grid or shard change cannot +silently bring back an order-dependent sum; the GPU tests hold the fixed order to the atomics on exact data and to the +warp-order oracle on a planted tile whose sum depends on the order. +""" + +import types + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( + m3_draft_step as kds, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_draft_step import ( # noqa: E501 + DRAFT_SHARD, + MAX_TOKENS, + bf16_fragments, + max_tile_pieces, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import gen, head_workspace # noqa: E402 + +OP = "minimax_m3_sm107tp8_draft_step" +SMS = 212 # Rubin's SMs: the grid the draft step launches on +K = 6144 +CHUNKS = K // 16 +# This target's shard on Rubin: 7-8 tiles per CTA; a 7-tile CTA's tiles take up to 4 pieces (the atomics' sum is +# order-dependent, so the fixed order is on). +ORDER_DEPENDENT = True + +gpu = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="the sm_107 TP8 target runs on SM107 only", +) + + +@pytest.mark.parametrize("grid", [SMS, kds.GRID]) +def test_real_grid_pieces(grid) -> None: + """One CTA per SM and the DS-GRID launch (kernels/m3_draft_step.GRID).""" + pieces = max_tile_pieces(DRAFT_SHARD // 16, grid, CHUNKS) + assert (pieces > 2) == ORDER_DEPENDENT, (grid, pieces) + + +def test_ds_grid_launches_the_configured_grid(monkeypatch) -> None: + monkeypatch.setattr(kds, "_num_sms", lambda: SMS) + assert kds._grid() == min(kds.GRID, SMS) + monkeypatch.setattr(kds, "GRID", 0) + assert kds._grid() == SMS + + +def test_max_tile_pieces_counts_warp_segments() -> None: + # One tile per CTA: all 16 warps stream it. 7 tiles per CTA: a 168-chunk segment can sit inside a tile (TP8). + assert max_tile_pieces(16, 16, CHUNKS) == 16 + assert max_tile_pieces(7, 1, CHUNKS) == 4 + assert max_tile_pieces(8, 1, CHUNKS) == 2 + assert max_tile_pieces(15, 1, CHUNKS) == 2 + + +@pytest.mark.parametrize("flag", [True, False]) +def test_the_kernel_engages_the_fixed_order_where_the_geometry_needs_it(flag, monkeypatch) -> None: + monkeypatch.setattr(kds, "FIXED_ORDER", flag) + x = torch.empty(1, K) + + def head(n_tiles): + return types.SimpleNamespace(wf=torch.empty(n_tiles, 1), exc_off=None, bnd=None) + + assert kds._fixed_order(1, head(DRAFT_SHARD // 16), x, SMS) == (flag and ORDER_DEPENDENT) + assert kds._fixed_order(1, head(DRAFT_SHARD // 16), x, kds.GRID) == (flag and ORDER_DEPENDENT) + assert kds._fixed_order(1, head(16), x, 16) == flag # 16 pieces per tile + + +def _launcher(wf, x, fixed, nctas): + """(launch, ids, keys): one pass's op launch on buffers made here (outside any capture), world 1.""" + uc, keys, counters = head_workspace(1, slot_rows=MAX_TOKENS) + m = x.shape[0] + i32 = dict(dtype=torch.int32, device="cuda") + draft_out = torch.zeros(m, 4, **i32) + ids = torch.zeros(m, **i32) + pos_src, pos_next, kv_lens = ( + torch.full((m,), 100, **i32), + torch.zeros(m, **i32), + torch.full((m,), 500, **i32), + ) + out = torch.zeros(m, dtype=torch.int64, device="cuda") + + def launch(): + getattr(torch.ops.trtllm, OP)(x, None, m, wf, 0, keys, counters, uc, uc.data_ptr(), 0, 1, draft_out, 1, ids, + pos_src, pos_next, kv_lens, None, None, 3, nctas, False, out, + fixed_order=fixed) # fmt: skip + + return launch, ids, out + + +@gpu +@pytest.mark.parametrize("nctas,vs", [(SMS, DRAFT_SHARD), (16, 256)]) +@pytest.mark.parametrize("m", [1, 4, 8]) +def test_fixed_order_equals_the_atomics_on_exact_data(nctas, vs, m) -> None: + """Integer data: every fp32 sum is exact in any order, so both summations give the same bits.""" + g = gen(17 + m) + w = torch.randint(-1, 2, (vs, K), generator=g, device="cuda").to(torch.bfloat16) + x = torch.randint(-1, 2, (m, K), generator=g, device="cuda").to(torch.bfloat16) + wf = bf16_fragments(w) + out = {} + for fixed in (False, True): + launch, ids, keys = _launcher(wf, x, fixed, nctas) + launch() + torch.cuda.synchronize() + out[fixed] = (keys, ids) + assert torch.equal(out[False][0], out[True][0]) + ref = (x.double() @ w.double().t()).float().bfloat16().float().argmax(dim=1).int() + assert torch.equal(out[True][1], ref) + + +@gpu +@pytest.mark.parametrize("graph", [False, True]) +def test_fixed_order_is_the_warp_order_sum_on_a_planted_tile(graph) -> None: + """One tile per CTA (16 pieces): row 0's pieces are +2^25 (warp 0), +1 (warps 1..14), -2^25 (warp 15). In warp + order the +1s are absorbed by 2^25 and the row sums to 0, so the competitor (row 255, logit 0.5) wins; arrival + order could give up to 14. Every launch must give the warp-order result.""" + w = torch.zeros(256, K, device="cuda") + seg = CHUNKS // 16 * 16 # one warp segment in K + w[0, 0] = 2.0**25 + for s in range(1, 15): + w[0, s * seg] = 1.0 + w[0, 15 * seg] = -(2.0**25) + w[255, 0] = 0.5 + wf = bf16_fragments(w.to(torch.bfloat16)) + x = torch.ones(1, K, device="cuda", dtype=torch.bfloat16) + launch, ids, _ = _launcher(wf, x, True, 16) + run = launch + if graph: + launch() + torch.cuda.synchronize() + gr = torch.cuda.CUDAGraph() + cs = torch.cuda.Stream() + with torch.cuda.stream(cs), torch.cuda.graph(gr, stream=cs): + launch() + run = gr.replay + for _ in range(16): + ids.zero_() + run() + torch.cuda.synchronize() + assert int(ids[0]) == 255 diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_fold_tail.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_fold_tail.py new file mode 100644 index 000000000000..913ff80e8ea3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_fold_tail.py @@ -0,0 +1,376 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the forward-tail spec fold (kernels/m3_fold_tail and M3SpecStepFold.in_forward). + +The tail must write exactly what the sampler's store (slot_scatter) and then the engine's next-step gathers +(step_gather, rows = the stored rows in order, first previous row 0) write, for the rows the step armed, and nothing +when the step was not armed. +""" + +import types + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( + m3_fold_tail, + m3_spec_tail, +) + +assert torch.cuda.is_available(), "m3_fold_tail requires a CUDA device" + +VOCAB = 200064 +NUM_SLOTS = 9 +MAX_ROWS = 8 +STALE = -4 + + +def _store(widths, g): + wn, wx, wd = widths + i32 = dict(dtype=torch.int32, device="cuda") + return types.SimpleNamespace( + new_tokens=torch.randint( + 0, VOCAB, (wn, NUM_SLOTS, 1), generator=g, device="cuda", dtype=torch.int32 + ), + next_new_tokens=torch.randint( + 0, VOCAB, (wx, NUM_SLOTS, 1), generator=g, device="cuda", dtype=torch.int32 + ), + next_draft_tokens=torch.randint( + 0, VOCAB, (NUM_SLOTS, wd), generator=g, device="cuda", dtype=torch.int32 + ), + new_tokens_lens=torch.randint(1, 5, (NUM_SLOTS,), generator=g, **i32), + ) + + +def _clone(ns): + return types.SimpleNamespace(**{k: v.clone() for k, v in vars(ns).items()}) + + +def _outputs(rows, t, g): + d = t - 1 + return { + "new_tokens": torch.randint( + 0, VOCAB, (rows, t), generator=g, device="cuda", dtype=torch.int32 + ), + "next_new_tokens": torch.randint( + 0, VOCAB, (rows, t), generator=g, device="cuda", dtype=torch.int32 + ), + "next_draft_tokens": torch.randint( + 0, VOCAB, (rows, d), generator=g, device="cuda", dtype=torch.int32 + ), + "new_tokens_lens": torch.randint( + 1, t + 1, (rows,), generator=g, device="cuda", dtype=torch.int32 + ), + } + + +def _engine(t): + i32 = dict(dtype=torch.int32, device="cuda") + return ( + torch.full((MAX_ROWS * t,), STALE, **i32), + torch.full((MAX_ROWS * (t - 1),), STALE, **i32), + torch.full((MAX_ROWS * t,), STALE, **i32), + torch.full((MAX_ROWS,), STALE, **i32), + ) + + +def _reference(store, engine, slots, out, t): + """slot_scatter into the store, then step_gather of the next step's inputs from it.""" + assert m3_spec_tail.slot_scatter( + slots, + out["new_tokens"], + store.new_tokens.squeeze(-1).T, + out["next_new_tokens"], + store.next_new_tokens.squeeze(-1).T, + out["new_tokens_lens"], + store.new_tokens_lens, + out["next_draft_tokens"], + store.next_draft_tokens, + ) + p = slots.numel() + ids, draft, pos_off, kv_off = engine + slots32 = slots.to(torch.int32) + assert m3_spec_tail.step_gather( + store.next_new_tokens, + store.next_draft_tokens, + store.new_tokens_lens, + slots32, + slots32.repeat_interleave(t), + p, + t, + t - 1, + ids, + 0, + draft, + 0, + pos_off, + kv_off, + 0, + ) + + +def _tail(store, engine, ctrl, slots_buf, out, t, rows, cpp=False, one_warp=False): + ids, draft, pos_off, kv_off = engine + assert m3_fold_tail.fold_tail( + ctrl, + slots_buf, + rows, + out["new_tokens"], + store.new_tokens.squeeze(-1).T, + out["next_new_tokens"], + store.next_new_tokens.squeeze(-1).T, + out["next_draft_tokens"], + store.next_draft_tokens, + out["new_tokens_lens"], + store.new_tokens_lens, + ids, + draft, + pos_off, + kv_off, + t, + cpp=cpp, + one_warp=one_warp, + ) + + +@pytest.mark.parametrize("cpp", [False, True], ids=["triton", "cpp"]) +@pytest.mark.parametrize("rows", [1, 2, 4, 8]) +@pytest.mark.parametrize("widths", [(4, 4, 3), (8, 8, 7)], ids=["store=step", "store wider"]) +def test_tail_matches_scatter_then_gather(rows, widths, cpp) -> None: + """Store bit for bit; engine rows [0, P*T) / [0, P) bit for bit (the forward reads only those rows).""" + t = 4 + g = torch.Generator(device="cuda").manual_seed(1000 * rows + widths[0]) + store = _store(widths, g) + out = _outputs(rows, t, g) + slots = torch.randperm(NUM_SLOTS, generator=g, device="cuda")[:rows] + ref_store, tail_store = _clone(store), _clone(store) + ref_engine, tail_engine = _engine(t), _engine(t) + _reference(ref_store, ref_engine, slots, out, t) + ctrl = torch.tensor([rows], dtype=torch.int32, device="cuda") + slots_buf = torch.zeros(MAX_ROWS, dtype=torch.long, device="cuda") + slots_buf[:rows] = slots + _tail(tail_store, tail_engine, ctrl, slots_buf, out, t, rows, cpp) + torch.cuda.synchronize() + for name in vars(store): + assert torch.equal(getattr(ref_store, name), getattr(tail_store, name)), name + spans = (rows * t, rows * (t - 1), rows * t, rows) + for k, (ref, got, n) in enumerate(zip(ref_engine, tail_engine, spans)): + assert torch.equal(ref[:n], got[:n]), k + assert torch.all(got[n:] == STALE), k + + +@pytest.mark.parametrize("cpp", [False, True], ids=["triton", "cpp"]) +def test_unarmed_tail_writes_nothing(cpp) -> None: + """ctrl[0] = 0 (the step was not armed): the store and the engine buffers are untouched.""" + t, rows = 4, 2 + g = torch.Generator(device="cuda").manual_seed(7) + store = _store((4, 4, 3), g) + out = _outputs(rows, t, g) + before, engine = _clone(store), _engine(t) + ctrl = torch.zeros(1, dtype=torch.int32, device="cuda") + slots_buf = torch.arange(MAX_ROWS, dtype=torch.long, device="cuda") + _tail(store, engine, ctrl, slots_buf, out, t, rows, cpp) + torch.cuda.synchronize() + for name in vars(store): + assert torch.equal(getattr(before, name), getattr(store, name)), name + for buf in engine: + assert torch.all(buf == STALE) + + +def _fold(): + fold = modeling.M3SpecStepFold() + t = 4 + engine = _engine(t) + fold.bind_engine_buffers(*engine, max_batch_size=MAX_ROWS) + g = torch.Generator(device="cuda").manual_seed(11) + fold.bind_sampler_store(_store((4, 4, 3), g)) + return fold, g, t + + +def test_fold_handshake_stages_only_confirmed_rows() -> None: + """begin_step arms; the sampler confirms only the armed rows from the tail's outputs; the next step is staged only + for exactly those rows in that order.""" + fold, g, t = _fold() + assert fold.begin_step([3, 5], 2) is False # nothing staged before the first forward + out = _outputs(2, t, g) + fold.store_in_forward(out) + assert fold.sampler_rows_stored(out, 0, [3, 5]) is True + assert fold.begin_step([3, 5], 2) is True + out = _outputs(2, t, g) + fold.store_in_forward(out) + assert ( + fold.sampler_rows_stored(out, 0, [5, 3]) is False + ) # other order: the sampler stores itself + assert fold.begin_step([5, 3], 2) is False + + +def test_fold_handshake_needs_the_tail_outputs() -> None: + """Outputs the tail did not run on (another dict, a mixed step, an unarmed step) are never reported stored.""" + fold, g, t = _fold() + fold.begin_step([1], 1) + fold.store_in_forward(_outputs(1, t, g)) + assert fold.sampler_rows_stored(_outputs(1, t, g), 0, [1]) is False + fold.begin_step(None, 1) + out = _outputs(1, t, g) + fold.store_in_forward(out) + assert fold.sampler_rows_stored(out, 0, [1]) is False + fold.begin_step([1], 1) + out = _outputs(1, t, g) + fold.store_in_forward(out) + assert fold.sampler_rows_stored(out, 1, [1]) is False # rows were skipped (chunked context) + + +def test_a_padded_step_after_an_armed_one_writes_nothing() -> None: + """K3 #150 check #5: CUDA-graph padding rows carry no seq slot, so the engine passes rows=None for a padded step. + Even right after an armed step, that step uploads ctrl 0 and its forward tail leaves the store and the engine + buffers untouched (a dummy row never writes a live slot).""" + fold, g, t = _fold() + fold.begin_step([2, 4], 2) + fold.store_in_forward(_outputs(2, t, g)) + torch.cuda.synchronize() + assert int(fold._ctrl.item()) == 2 + + fold.begin_step(None, 4) + torch.cuda.synchronize() + assert int(fold._ctrl.item()) == 0 + store_before = _clone(fold.store) + engine_before = [buf.clone() for buf in fold.step_inputs] + fold.store_in_forward(_outputs(4, t, g)) + torch.cuda.synchronize() + for name in vars(store_before): + assert torch.equal(getattr(store_before, name), getattr(fold.store, name)), name + for before, buf in zip(engine_before, fold.step_inputs): + assert torch.equal(before, buf) + + +def test_the_fold_stores_in_the_forward_once_bound() -> None: + """The fold always stores and stages at the forward's tail; before the engine binds its buffers nothing is armed.""" + fold = modeling.M3SpecStepFold() + assert fold.in_forward + assert fold.begin_step([0], 1) is False + + +def test_the_store_copy_event_is_recorded_at_binding(): + """An external event the sampler records after its side-stream store copies and the forward's tail waits on, + recorded once at binding so a captured wait has a completed record to wait on.""" + fold = modeling.M3SpecStepFold() + device = torch.device("cuda") + fold.bind_engine_buffers( + torch.zeros(8, dtype=torch.int32, device=device), + torch.zeros(8, dtype=torch.int32, device=device), + torch.zeros(8, dtype=torch.int32, device=device), + torch.zeros(8, dtype=torch.int32, device=device), + 4, + ) + assert fold.store_copies_done is not None + fold.store_copies_done.synchronize() + + +def test_the_cpp_tail_launches_plainly() -> None: + """The C++ tail writes the next step's inputs, which pre-wait readers take before their grid wait (MSA contract): + its kernel node carries no programmatic (PDL) serialization.""" + from cuda.bindings import runtime as rt + + t, rows = 4, 2 + g = torch.Generator(device="cuda").manual_seed(13) + store, out, engine = _store((4, 4, 3), g), _outputs(rows, t, g), _engine(t) + ctrl = torch.tensor([rows], dtype=torch.int32, device="cuda") + slots_buf = torch.arange(MAX_ROWS, dtype=torch.long, device="cuda") + _tail(store, engine, ctrl, slots_buf, out, t, rows, cpp=True) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph(keep_graph=True) + with torch.cuda.graph(graph): + _tail(store, engine, ctrl, slots_buf, out, t, rows, cpp=True) + raw = rt.cudaGraph_t(init_value=graph.raw_cuda_graph()) + # cuda-python changed the signature: (graph[, numNodes]) on older bindings, (graph, nodes, numNodes) on newer ones. + try: + err, _, n = rt.cudaGraphGetNodes(raw) + err, nodes, n = rt.cudaGraphGetNodes(raw, n) + except TypeError: + err, _, n = rt.cudaGraphGetNodes(raw, None, 0) + err, nodes, n = rt.cudaGraphGetNodes(raw, None, n) + flags = [] + for node in nodes[:n]: + err, kind = rt.cudaGraphNodeGetType(node) + if kind == rt.cudaGraphNodeType.cudaGraphNodeTypeKernel: + err, attr = rt.cudaGraphKernelNodeGetAttribute( + node, rt.cudaLaunchAttributeID.cudaLaunchAttributeProgrammaticStreamSerialization + ) + # Older bindings return no attribute for a kernel launched without one (a plain launch). + flags.append(0 if attr is None else int(attr.programmaticStreamSerializationAllowed)) + assert flags == [0], flags + + +@pytest.mark.parametrize("rows", [1, 2, 4, 8]) +@pytest.mark.parametrize("widths", [(4, 4, 3), (8, 8, 7)], ids=["store=step", "store wider"]) +def test_one_warp_form_is_bitwise_the_kernel(rows, widths) -> None: + """The in-kernel form (one warp over the rows: what the drafter's last-pass draft_step runs in its last CTA, H-R2-3) + writes exactly what the one-CTA-per-row kernel writes, armed and unarmed, eager and from a graph.""" + t = 4 + for trial in range(3): + g = torch.Generator(device="cuda").manual_seed(100 * rows + trial) + store, out, engine = _store(widths, g), _outputs(rows, t, g), _engine(t) + slots_buf = torch.randperm(NUM_SLOTS, generator=g, device="cuda")[:MAX_ROWS].to(torch.long) + for armed in (rows, 0): + ctrl = torch.tensor([armed], dtype=torch.int32, device="cuda") + a_store, a_engine = _clone(store), tuple(e.clone() for e in engine) + b_store, b_engine = _clone(store), tuple(e.clone() for e in engine) + _tail(a_store, a_engine, ctrl, slots_buf, out, t, rows, cpp=True) + _tail(b_store, b_engine, ctrl, slots_buf, out, t, rows, cpp=True, one_warp=True) + torch.cuda.synchronize() + for k in vars(a_store): + assert torch.equal(getattr(a_store, k), getattr(b_store, k)), (k, armed, trial) + for x, y in zip(a_engine, b_engine): + assert torch.equal(x, y), (armed, trial) + # From a captured graph, replayed on fresh outputs. + g = torch.Generator(device="cuda").manual_seed(7) + store, out, engine = _store(widths, g), _outputs(rows, t, g), _engine(t) + slots_buf = torch.arange(MAX_ROWS, dtype=torch.long, device="cuda") + ctrl = torch.tensor([rows], dtype=torch.int32, device="cuda") + b_store, b_engine = _clone(store), tuple(e.clone() for e in engine) + _tail(b_store, b_engine, ctrl, slots_buf, out, t, rows, cpp=True, one_warp=True) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + _tail(b_store, b_engine, ctrl, slots_buf, out, t, rows, cpp=True, one_warp=True) + for step in range(3): + fresh = _outputs(rows, t, torch.Generator(device="cuda").manual_seed(20 + step)) + for k in out: + out[k].copy_(fresh[k]) + a_store, a_engine = _clone(b_store), tuple(e.clone() for e in b_engine) + _tail(a_store, a_engine, ctrl, slots_buf, out, t, rows, cpp=True) + graph.replay() + torch.cuda.synchronize() + for k in vars(a_store): + assert torch.equal(getattr(a_store, k), getattr(b_store, k)), (k, step) + for x, y in zip(a_engine, b_engine): + assert torch.equal(x, y), step + + +def test_the_store_wait_precedes_only_a_plain_launch_under_capture(): + """The store wait is an external-event wait node under graph capture, which drops the next kernel's PDL edge: + it may precede the fold tail (a plain launch); any other launch fails the capture (eager steps are not checked).""" + fold = modeling.M3SpecStepFold() + device = torch.device("cuda") + fold.bind_engine_buffers( + *(torch.zeros(8, dtype=torch.int32, device=device) for _ in range(4)), 4 + ) + assert fold.STORE_WAIT_BEFORE == frozenset({"fold_tail"}) + fold._wait_store_copies(before="draft_step") # eager: allowed + x = torch.zeros(4, device=device) + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + fold._wait_store_copies(before="fold_tail") + x.add_(1) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(x, torch.ones(4, device=device)) + bad = torch.cuda.CUDAGraph() + with pytest.raises(RuntimeError, match="PDL edge"): + with torch.cuda.graph(bad, stream=stream): + fold._wait_store_copies(before="draft_step") diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_al.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_al.py new file mode 100644 index 000000000000..bbd97cb73146 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_al.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test of the per-request forced-acceptance draw (``FORCE_AL_PER_REQUEST``, the ``m3_force_vec_req`` entry, +synth only): each request's forced count comes from its own key (its KV length at the verify), so it does not depend +on the call history, on the request's slot in the batch, or on the other requests.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_force_vec import ( # noqa: E501 + m3_force_vec_req, +) +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, + _FORCE_ACCEPT_RNG_SEED, +) + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA device") + + +def _pool(): + g = torch.Generator(device="cpu") + g.manual_seed(_FORCE_ACCEPT_RNG_SEED) + return torch.rand(_FORCE_ACCEPT_RNG_POOL_SIZE, dtype=torch.float32, generator=g).cuda() + + +def _draw(pool, keys, base_total=2, max_total=4, frac=0.6): + out = torch.empty(keys.numel(), dtype=torch.int32, device="cuda") + m3_force_vec_req( + pool, + keys, + out, + keys.numel(), + base_total, + max_total, + frac, + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + max(16, 1 << (keys.numel() - 1).bit_length()), + ) + return out + + +def test_a_request_draws_the_same_whatever_ran_before_or_beside_it() -> None: + pool = _pool() + keys = torch.tensor([165_521, 40_003, 323_007, 7], dtype=torch.int32, device="cuda") + first = _draw(pool, keys) + for _ in range(5): # no shared counter: repeated calls draw the same + assert torch.equal(_draw(pool, keys), first) + perm = torch.tensor([2, 0, 3, 1], device="cuda") + assert torch.equal(_draw(pool, keys[perm]), first[perm]) # slot-independent + alone = torch.cat([_draw(pool, keys[i : i + 1]) for i in range(4)]) + assert torch.equal(alone, first) # independent of the other requests + + +def test_the_counts_follow_the_forced_fraction() -> None: + pool = _pool() + keys = torch.arange(100_000, 100_000 + 64 * 400, dtype=torch.int32, device="cuda") + counts = torch.cat([_draw(pool, keys[i : i + 64]) for i in range(0, keys.numel(), 64)]) + assert set(counts.tolist()) <= {2, 3} + extra = (counts == 3).float().mean().item() + assert abs(extra - 0.6) < 0.02, extra diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_vec.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_vec.py new file mode 100644 index 000000000000..622e5170ad69 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_vec.py @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_force_vec entry of the MiniMax-M3 tp8 target: the fractional forced-acceptance draw.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_force_vec import ( # noqa: E501 + m3_force_vec as op, +) +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_force_vec requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(5) + pool = torch.rand(_FORCE_ACCEPT_RNG_POOL_SIZE, generator=g, device="cuda") + counter = torch.tensor([41], dtype=torch.int64, device="cuda") + out = torch.empty(spec["batch"], dtype=torch.int32, device="cuda") + args = ( + pool, + counter, + out, + spec["batch"], + 2, + 4, + spec["frac"], + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_SLOT_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + spec["block"], + ) + ref = op.reference(*args) + with validating(op): + op(*args) + assert int(counter.item()) == 42, "the draw bumps the step counter once" + assert torch.equal(out, ref) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_vec_req.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_vec_req.py new file mode 100644 index 000000000000..78e24d3d42cf --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_force_vec_req.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_force_vec_req entry of the MiniMax-M3 tp8 target: the per-request fractional forced-acceptance +draw (one draw per request at its own key, no shared counter).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_force_vec import ( # noqa: E501 + m3_force_vec_req as op, +) +from tensorrt_llm._torch.speculative.interface import ( + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE, +) + +__extra_import_path__ = ["../../../.."] +from _validating import validating # noqa: E402 — needs the path declared above + +assert torch.cuda.is_available(), "m3_force_vec_req requires a CUDA device" + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:48] for c in op.CELLS]) +def test_certified_cells(cell) -> None: + spec = cell.spec + g = torch.Generator(device="cuda").manual_seed(7) + pool = torch.rand(_FORCE_ACCEPT_RNG_POOL_SIZE, generator=g, device="cuda") + # Each request's key is its KV length at the verify: long, distinct, arbitrary. + keys = torch.randint( + 1, 1 << 20, (spec["batch"],), generator=g, device="cuda", dtype=torch.int64 + ).to(torch.int32) + out = torch.empty(spec["batch"], dtype=torch.int32, device="cuda") + args = ( + pool, + keys, + out, + spec["batch"], + 2, + 4, + spec["frac"], + _FORCE_ACCEPT_RNG_COUNTER_STRIDE, + _FORCE_ACCEPT_RNG_POOL_SIZE - 1, + spec["block"], + ) + ref = op.reference(*args) + with validating(op): + op(*args) + assert torch.equal(out, ref) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_lowm_splitk.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_lowm_splitk.py new file mode 100644 index 000000000000..9ceb6e587031 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_lowm_splitk.py @@ -0,0 +1,98 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the target's copy of FlashInfer's split-K low-M GEMM (kernels/m3_lowm_splitk, +modeling.DRAFT_LOWM_OWN_SPLITK): bit for bit FlashInfer's kernel at every certified split-K cell of the drafter's +linears, eager and in a CUDA graph, with and without PDL.""" + +import pytest +import torch +from flashinfer.gemm.kernels.dense_bf16_gemm_sm100_splitk import SplitKTactic as FiTactic +from flashinfer.gemm.kernels.dense_bf16_gemm_sm100_splitk import run_splitk_dense as fi_run + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels.m3_lowm_splitk import ( # noqa: E501 + SplitKTactic, + run_splitk_dense, + run_splitk_dense_swiglu, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_draft_lowm import ( # noqa: E501 + TACTICS, +) + +assert torch.cuda.is_available(), "m3_lowm_splitk requires a CUDA device" + +# Every split-K cell of the TP8 drafter table: (N, K, max rows, tactic) for kind "s". +CELLS = [ + (nk[0], nk[1], rows, tactic) + for nk, entries in sorted(TACTICS.items()) + for rows, kind, tactic in entries + if kind == "s" +] + + +def test_the_table_has_the_four_certified_split_k_cells() -> None: + assert sorted((n, k, m) for n, k, m, _ in CELLS) == [ + (6144, 1024, 1), + (6144, 1024, 4), + (6144, 2304, 1), + (6144, 2304, 4), + ] + + +@pytest.mark.parametrize("use_pdl", [False, True], ids=["plain", "pdl"]) +@pytest.mark.parametrize("cell", CELLS, ids=[f"{n}x{k}@{m}" for n, k, m, _ in CELLS]) +def test_bitwise_flashinfer(cell, use_pdl) -> None: + n, k, m, tactic = cell + g = torch.Generator(device="cuda").manual_seed(n + 7 * k + m) + for trial in range(3): + x = (torch.randn(m, k, generator=g, device="cuda") * (1.0 + trial)).to(torch.bfloat16) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + ref = torch.empty(m, n, dtype=torch.bfloat16, device="cuda") + got = torch.empty(m, n, dtype=torch.bfloat16, device="cuda") + fi_run(x, w.t(), None, ref, use_pdl, FiTactic(*tactic)) + run_splitk_dense(x, w.t(), None, got, use_pdl, SplitKTactic(*tactic)) + torch.cuda.synchronize() + assert torch.equal(got.view(torch.int16), ref.view(torch.int16)), f"trial {trial}" + # In a CUDA graph, as the drafter runs it. + cap = torch.empty(m, n, dtype=torch.bfloat16, device="cuda") + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + run_splitk_dense(x, w.t(), None, cap, use_pdl, SplitKTactic(*tactic)) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(cap.view(torch.int16), ref.view(torch.int16)) + + +# The drafter MLP's down projection: the SwiGLU folded into the GEMM's B load (c2). +DOWN_CELLS = [c for c in CELLS if (c[0], c[1]) == (6144, 2304)] + + +@pytest.mark.parametrize("use_pdl", [False, True], ids=["plain", "pdl"]) +@pytest.mark.parametrize("cell", DOWN_CELLS, ids=[f"{n}x{k}@{m}" for n, k, m, _ in DOWN_CELLS]) +def test_swiglu_prologue_is_bitwise_silu_then_split_k(cell, use_pdl) -> None: + n, k, m, tactic = cell + g = torch.Generator(device="cuda").manual_seed(31 * n + m) + w = (torch.randn(n, k, generator=g, device="cuda") / k**0.5).to(torch.bfloat16) + scratch = torch.full((8, k), 7, dtype=torch.bfloat16, device="cuda") + for trial in range(3): + gate_up = (torch.randn(m, 2 * k, generator=g, device="cuda") * (2.0 + trial)).to( + torch.bfloat16 + ) + gate_up[:, :6] = torch.tensor([60.0, -60.0, 0.0, -0.0, 88.0, -88.0], dtype=torch.bfloat16) + act = torch.ops.trtllm.silu_and_mul(gate_up) + ref = torch.empty(m, n, dtype=torch.bfloat16, device="cuda") + fi_run(act, w.t(), None, ref, use_pdl, FiTactic(*tactic)) + got = torch.empty(m, n, dtype=torch.bfloat16, device="cuda") + run_splitk_dense_swiglu(gate_up, w.t(), got, use_pdl, SplitKTactic(*tactic), scratch) + torch.cuda.synchronize() + assert torch.equal(got.view(torch.int16), ref.view(torch.int16)), f"trial {trial}" + assert torch.equal(scratch[:m].view(torch.int16), act.view(torch.int16)), ( + "the B scratch holds the activation" + ) + cap = torch.empty(m, n, dtype=torch.bfloat16, device="cuda") + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + run_splitk_dense_swiglu(gate_up, w.t(), cap, use_pdl, SplitKTactic(*tactic), scratch) + scratch.fill_(3) # a replay rewrites what it reads + graph.replay() + torch.cuda.synchronize() + assert torch.equal(cap.view(torch.int16), ref.view(torch.int16)) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_sampled_accept.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_sampled_accept.py new file mode 100644 index 000000000000..38410da71a22 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_sampled_accept.py @@ -0,0 +1,267 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the tp8 m3_sampled_accept entry (fused temperature + top-p verify over the 25008-column TP8 shard) +and its workspace size.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_sampled_accept import ( # noqa: E501 + m3_sampled_accept as op, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_sampled_accept_ws_words import ( # noqa: E501 + m3_sampled_accept_ws_words as ws_words, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import gen, seed_of # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_sampled_accept requires a CUDA device" + +DRAFT_LEN = 3 +#: Hot tokens of the distribution cells; every other logit sits 22 below the top (mass ~ 1e-5 of the total). +HEAD = 12 +#: Draws per distribution cell, and the total-variation band against the reference distribution. With 12 +#: categories and >= 4000 draws the expected TV of an exact sampler is ~0.02 (sd ~0.006); 0.06 is ~6 sd above it +#: while a wrong nucleus cut (one category too many or too few, >= 1 / 12 of the mass moved) lands at >= 0.08. +DRAWS = 4000 +TV_BAND = 0.06 + + +def _logits(spec, g, rows): + vs = spec["vs"] + if spec["check"] == "flat": + return (torch.randn((rows, vs), generator=g, device="cuda") * 0.3).to(torch.bfloat16) + if spec["check"] == "argmax": + return (torch.randn((rows, vs), generator=g, device="cuda") * 3).to(torch.bfloat16) + row = torch.full((vs,), -20.0, device="cuda") + hot = torch.randperm(vs, generator=g, device="cuda")[:HEAD] + row[hot] = torch.linspace(2.0, 0.2, HEAD, device="cuda") + return row.to(torch.bfloat16).expand(rows, vs).contiguous() + + +def _case(spec, seed): + g = gen(seed) + nreq = spec["nreq"] + rows = nreq * (DRAFT_LEN + 1) + logits = _logits(spec, g, rows) + words = ws_words(1) + uc = torch.zeros(words, dtype=torch.int64, device="cuda") + counters = torch.zeros(2, dtype=torch.int32, device="cuda") + tokens = torch.zeros(48, dtype=torch.int32, device="cuda") + extra = torch.zeros(1, dtype=torch.int32, device="cuda") + temperatures = torch.full((rows,), spec["temperature"], dtype=torch.float32, device="cuda") + top_ps = torch.full((rows,), spec["top_p"], dtype=torch.float32, device="cuda") + seeds = torch.randint(0, 2**62, (rows,), generator=g, device="cuda", dtype=torch.int64) + offsets = torch.zeros(rows, dtype=torch.int64, device="cuda") + # Drafts drawn from the likeliest tokens, so some prefixes match. + top = logits[0].float().topk(3).indices.to(torch.int32) + spec["vocab_offset"] + draft = top[torch.randint(0, 3, (nreq, DRAFT_LEN), generator=g, device="cuda")].contiguous() + batch_idx = torch.arange(nreq, dtype=torch.int32, device="cuda") + acc = torch.full((nreq, DRAFT_LEN + 1), -7, dtype=torch.int32, device="cuda") + num = torch.full((nreq,), -7, dtype=torch.int32, device="cuda") + gather = torch.full((nreq,), -7, dtype=torch.int64, device="cuda") + next_new = torch.full((nreq, DRAFT_LEN + 1), -7, dtype=torch.int32, device="cuda") + force = spec.get("force", "none") + force_total = 3 if force == "total" else 0 + force_vec = (torch.randint(1, DRAFT_LEN + 2, (nreq,), generator=g, device="cuda", dtype=torch.int32) + if force == "vec" else None) # fmt: skip + return [logits, spec["vocab_offset"], temperatures, top_ps, seeds, offsets, uc, uc.data_ptr(), counters, tokens, + 0, 1, draft, batch_idx, acc, num, gather, force_total, next_new, force_vec, extra, True] # fmt: skip + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Statistical for the draw, exact for everything derived from it. + + Every drawn token has nonzero reference probability (lies in the nucleus); the acceptance outputs equal the + reference's strict acceptance of the drawn tokens, every call; the call counter advances once per call; and, + for the distribution cells, the draws over many calls (fresh Philox offsets) are within TV_BAND of the + reference distribution (see DRAWS / TV_BAND). + """ + spec = cell.spec + args = _case(spec, seed_of(cell.why)) + logits, offset, rows = args[0], spec["vocab_offset"], args[0].shape[0] + certifying(op) + ref = op.reference(*args) + probs = ref["probs"] + calls = 1 if spec["check"] == "argmax" else -(-DRAWS // rows) + counts = torch.zeros(logits.shape[1], dtype=torch.float64, device="cuda") + for c in range(calls): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * 1000) + op(*args) + drawn = args[9][:rows].clone() + local = drawn.long() - offset + assert bool(((local >= 0) & (local < logits.shape[1])).all()), "token outside the shard" + assert bool((probs.gather(1, local[:, None]) > 0).all()), ( + "a drawn token outside the nucleus" + ) + acc = ref["accept"](drawn) + assert torch.equal(args[14], acc["accepted"]) + assert torch.equal(args[15], acc["num_accepted"]) + assert torch.equal(args[16], acc["gather_ids"]) + assert torch.equal(args[18], acc["next_new"]) + assert args[8].tolist() == [0, c + 1] + counts.index_add_(0, local, torch.ones(rows, dtype=torch.float64, device="cuda")) + if spec["check"] == "distribution": + emp = counts / counts.sum() + tv = 0.5 * (emp - probs[0].double()).abs().sum().item() + assert tv <= TV_BAND, f"total variation {tv:.3f} against the reference nucleus distribution" + + +def _walk_outputs(args, one_round_k, ilp, calls=3, offset_step=1000): + """The draws and acceptance outputs of `calls` calls with the owner walk one token (ilp False) or 8 tokens at a + time (ilp True), at SA-1R variant `one_round_k`, each call at its own Philox offsets.""" + if one_round_k: + words = torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept_ws_words(1, one_round_k) + args[6] = torch.zeros(words, dtype=torch.int64, device="cuda") + args[7] = args[6].data_ptr() + rows = args[0].shape[0] + got = [] + for c in range(calls): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * offset_step) + torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept( + *args, True, False, one_round_k, None, ilp + ) + got.append([args[i].clone() for i in (9, 14, 15, 16, 18)]) + return got + + +@pytest.mark.parametrize("one_round_k", [0, 82]) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_the_ilp_walk_changes_nothing(cell, one_round_k) -> None: + """kernels/m3_sampled_accept.ILP_WALK (SA-PW) only computes the owner's token weights 8 at a time; the walk adds + them in the same order with the same early exit, so the draws and acceptance outputs are the one-token walk's, + bit for bit, call after call, with SA-1R off and on.""" + one = _walk_outputs(_case(cell.spec, seed_of(cell.why)), one_round_k, False) + ilp = _walk_outputs(_case(cell.spec, seed_of(cell.why)), one_round_k, True) + for call, (a, b) in enumerate(zip(one, ilp)): + for x, y in zip(a, b): + assert torch.equal(x, y), ( + f"call {call}: the 8-token walk changes {x.tolist()} -> {y.tolist()}" + ) + + +@pytest.mark.parametrize("top_p", [0.001, 0.95]) +def test_the_ilp_walk_changes_nothing_over_many_rounds(top_p) -> None: + """A flat tail (many tokens of equal weight) under a tiny and a wide nucleus: rows take many exchange rounds and + each round's walk exits at a different token; the 8-token walk draws the same tokens as the one-token walk.""" + spec = dict(nreq=1, vs=25008, vocab_offset=25008, check="flat", top_p=top_p, temperature=1.0) + one = _walk_outputs(_case(spec, seed_of(f"ilp walk flat {top_p}")), 0, False, calls=24) + ilp = _walk_outputs(_case(spec, seed_of(f"ilp walk flat {top_p}")), 0, True, calls=24) + for call, (a, b) in enumerate(zip(one, ilp)): + for x, y in zip(a, b): + assert torch.equal(x, y), ( + f"call {call}: the 8-token walk changes {x.tolist()} -> {y.tolist()}" + ) + + +def test_the_ilp_walk_is_the_same_under_graph_replay() -> None: + """The 8-token walk captured in a CUDA graph and replayed draws what the one-token walk draws eagerly.""" + cell = op.CELLS[0] + eager = _walk_outputs(_case(cell.spec, seed_of(cell.why)), 0, False) + _walk_outputs( + _case(cell.spec, seed_of("ilp walk warm-up")), 0, True, calls=1 + ) # first launch outside the capture + args = _case(cell.spec, seed_of(cell.why)) + rows = args[0].shape[0] + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + graph = torch.cuda.CUDAGraph() + with torch.cuda.stream(stream), torch.cuda.graph(graph, stream=stream): + torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept(*args, True, False, 0, None, True) + torch.cuda.current_stream().wait_stream(stream) + for c in range(3): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * 1000) + graph.replay() + torch.cuda.synchronize() + for x, y in zip(eager[c], [args[i] for i in (9, 14, 15, 16, 18)]): + assert torch.equal(x, y), ( + f"replay {c}: the graphed 8-token walk changes {x.tolist()} -> {y.tolist()}" + ) + + +@pytest.mark.parametrize("top_p", [0.001, 0.0002]) +def test_long_rejection_chains_stay_in_the_nucleus(top_p) -> None: + """A tiny nucleus over a flat tail: rows that need more exchange rounds than the workspace has slots. + + Each round's candidate is drawn over the tokens heavier than the last one, so a row takes ~ln(1 / top_p) + 1 + rounds on average, with a tail past the 16 exchange slots (about 1% of the rows at top_p 0.001). Rounds reuse the + slots cyclically, told apart by a lap-parity bit in the wire tags, and no round accepts unconditionally: every + draw, however many rounds it took, lies in the reference nucleus. World 1 (one rank's 8 CTAs per row exchange + through the same slot ring); the cross-rank path needs the multi-device job. + """ + spec = dict(nreq=1, vs=25008, vocab_offset=25008, check="flat", top_p=top_p, temperature=1.0) + args = _case(spec, seed_of(f"flat tail top_p {top_p}")) + logits, rows = args[0], args[0].shape[0] + probs = op.reference(*args)["probs"] + calls = -(-DRAWS // rows) + outside = torch.zeros((), dtype=torch.int64, device="cuda") + for c in range(calls): + args[5].copy_(torch.arange(rows, device="cuda", dtype=torch.int64) + c * 1000) + op(*args) + local = args[9][:rows].long() - spec["vocab_offset"] + assert bool(((local >= 0) & (local < logits.shape[1])).all()), "token outside the shard" + outside += (probs.gather(1, local[:, None]) == 0).sum() + assert args[8].tolist() == [0, c + 1] + draws = calls * rows + rounds = 2.0 + args[20].item() / draws + kept = int((probs > 0).sum(1).min().item()) + assert outside.item() == 0, ( + f"{outside.item()} of {draws} draws outside the nucleus (kept >= {kept} of {logits.shape[1]}, " + f"{rounds:.2f} rounds per row on average)" + ) + assert rounds > 4.0, ( + f"only {rounds:.2f} rounds per row: the rows no longer stress the slot ring" + ) + + +@pytest.mark.parametrize("cell", ws_words.CELLS, ids=[c.why[:44] for c in ws_words.CELLS]) +def test_workspace_words(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + certifying(ws_words) + world = cell.spec["world"] + assert ws_words(world) == ws_words.reference(world) + + +def test_guard_refuses_top_k_rows() -> None: + args = _case(op.CELLS[0].spec, 1) + top_ks = torch.full((args[0].shape[0],), 20, dtype=torch.int32, device="cuda") + with pytest.raises(AssertionError, match="top-k rows"): + with validating(op): + op(*args, top_ks=top_ks) + + +def test_guard_refuses_fp64_temperatures() -> None: + args = _case(op.CELLS[0].spec, 2) + args[2] = args[2].double() + with pytest.raises(AssertionError, match="fp32 sampling parameters"): + with validating(op): + op(*args) + + +@pytest.mark.parametrize( + "check,top_p", [("flat", 0.95), ("argmax", 0.95), ("flat", 0.001), ("distribution", 0.5)] +) +def test_the_copy_is_the_tp2_kernel_bit_for_bit(check, top_p) -> None: + """This target's own op (trtllm::minimax_m3_sm107tp8_sampled_accept) is the TP2 kernel it copies, bit for bit: + the same tokens, acceptance outputs, call counter and extra-round count from the same seeds and offsets.""" + tp2 = getattr(torch.ops.trtllm, "minimax_m3_sm107tp2_sampled_accept", None) + if tp2 is None: + pytest.skip("the TP2 op is not loaded") + spec = dict(nreq=1, vs=25008, vocab_offset=25008, check=check, top_p=top_p, temperature=1.0) + for call in range(8): + outs = [] + for which in ("tp8", "tp2"): + args = _case(spec, seed_of(f"copy {check} {top_p}")) + args[5].copy_( + torch.arange(args[0].shape[0], device="cuda", dtype=torch.int64) + call * 1000 + ) + if which == "tp8": + op(*args) + else: + tp2(*args[:21], args[21]) + torch.cuda.synchronize() + outs.append([args[i].clone() for i in (9, 14, 15, 16, 18, 20)] + [args[8].clone()]) + for a, b in zip(*outs): + assert torch.equal(a, b) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_sampled_accept_1r.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_sampled_accept_1r.py new file mode 100644 index 000000000000..462d227d44f9 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_sampled_accept_1r.py @@ -0,0 +1,186 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SA-1R (kernels/m3_sampled_accept.ONE_ROUND_K) on trtllm::minimax_m3_sm107tp8_sampled_accept, world 1, the TP8 shard. + +The one-round variants (one_round_k 9: exact top-9 list; 82: pooled top-8 + bound) must return the two-round +kernel's (one_round_k 0) sampled tokens, accepted tokens, accepted counts, gather ids and extra-attempt count BIT FOR +BIT on the same Philox stream, over random AgentX-like rows (several temperatures / top-p, broad and heavy-tailed +heads) and over rows built so the decision sits within ulps of top_p * Z (the margin fallback) or a slice holds more +heavy tokens than its list (the incomplete-list fallback), eager and under CUDA-graph replay. The round-0 decided +counter must engage on the AgentX-like rows and stay below the row count on the boundary rows (fallbacks fire). +Ported from the prototype test (exact on 39,968 rows). +""" + +import pytest +import torch + +import tensorrt_llm._torch.custom_ops # noqa: F401 — registers torch.ops.trtllm.* + +VS, D = 25008, 3 +KX = (9, 82) + +if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7): + pytest.skip("the sm_107 tp8 sampled accept runs on SM 10.7", allow_module_level=True) + + +class Runner: + """One variant with its own workspace / counters (call tag sequence) and outputs.""" + + def __init__(self, k, rows): + self.k, self.rows = k, rows + words = int(torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept_ws_words(1, k)) + self.ws = torch.zeros(words, dtype=torch.int64, device="cuda") + self.counters = torch.zeros(2, dtype=torch.int32, device="cuda") + self.tokens = torch.zeros(48, dtype=torch.int32, device="cuda") + self.extra = torch.zeros(1, dtype=torch.int32, device="cuda") + self.decided = torch.zeros(1, dtype=torch.int32, device="cuda") + nreq = rows // (D + 1) + self.acc = torch.zeros((nreq, D + 1), dtype=torch.int32, device="cuda") + self.nacc = torch.zeros(nreq, dtype=torch.int32, device="cuda") + self.gids = torch.zeros(nreq, dtype=torch.int64, device="cuda") + + def __call__(self, lg, temps, tps, seeds, offs, draft, bidx): + torch.ops.trtllm.minimax_m3_sm107tp8_sampled_accept( + lg, 0, temps, tps, seeds, offs, self.ws, self.ws.data_ptr(), self.counters, self.tokens, 0, 1, draft, + bidx, self.acc, self.nacc, self.gids, 0, None, None, self.extra, False, True, False, self.k, + self.decided) # fmt: skip + + def out(self): + return (self.tokens[: self.rows].clone(), self.acc.clone(), self.nacc.clone(), self.gids.clone(), + self.extra.clone()) # fmt: skip + + +def _params(rows, temp, top_p, seed0): + temps = torch.full((rows,), temp, dtype=torch.float32, device="cuda") + tps = ( + (top_p if torch.is_tensor(top_p) else torch.full((rows,), top_p)) + .float() + .cuda() + .contiguous() + ) + seeds = torch.arange(rows, dtype=torch.int64, device="cuda") * 7919 + seed0 + offs = torch.zeros(rows, dtype=torch.int64, device="cuda") + return temps, tps, seeds, offs + + +def _sets(n, rows, g, kind): + out = [] + for _ in range(n): + lg = torch.randn(rows, VS, device="cuda", generator=g) * (1.0 if kind == "flat" else 2.0) + if kind == "lm": # a few heavy tokens over a broad tail + hot = torch.randint(0, VS, (rows, 8), device="cuda", generator=g) + lg.scatter_(1, hot, 12.0 + 3.0 * torch.rand(rows, 8, device="cuda", generator=g)) + elif kind == "tail": # 64 near-equal top tokens: lists overflow + hot = torch.randint(0, VS, (rows, 64), device="cuda", generator=g) + lg.scatter_(1, hot, 9.0 + 0.5 * torch.rand(rows, 64, device="cuda", generator=g)) + out.append(lg.to(torch.bfloat16).contiguous()) + return out + + +def _draft_from(lg, g): + nreq = lg.shape[0] // (D + 1) + am = lg.float().argmax(1).view(nreq, D + 1)[:, :D].int() + rnd = torch.randint(0, VS, (nreq, D), device="cuda", generator=g, dtype=torch.int32) + return torch.where(torch.rand(nreq, D, device="cuda", generator=g) < 0.7, am, rnd).contiguous() + + +def _compare(ref, got, what): + for n, x, y in zip( + ("tokens", "acc_tokens", "num_acc", "gather_ids", "extra_attempts"), ref, got + ): + assert torch.equal(x, y), f"{what}: {n} differs" + + +CASES = [(1.0, 0.95, "lm"), (0.7, 0.8, "lm"), (1.3, 0.5, "lm"), (1.0, 1.0, "lm"), (1.0, 0.95, "flat"), + (1.0, 0.95, "tail"), (0.6, 0.99, "tail")] # fmt: skip + + +def test_one_round_is_two_round_bitwise(): + g = torch.Generator(device="cuda").manual_seed(2) + total = 0 + decided = {k: 0 for k in KX} + for rows, calls in ((32, 20), (4, 60)): + for ci, (t, p, kind) in enumerate(CASES): + runners = {k: Runner(k, rows) for k in (0, *KX)} + temps, tps, seeds, offs = _params(rows, t, p, 1000 + ci) + sets = _sets(4, rows, g, kind) + bidx = torch.arange(rows // (D + 1), dtype=torch.int32, device="cuda") + for i in range(calls): + lg = sets[i % 4] + draft = _draft_from(lg, g) + offs.fill_(i) + for r in runners.values(): + r(lg, temps, tps, seeds, offs, draft, bidx) + for k in KX: + _compare( + runners[0].out(), runners[k].out(), f"K={k} rows={rows} case={ci} call={i}" + ) + total += rows + if kind == "lm" and p < 1.0: + for k in KX: + decided[k] += int(runners[k].decided.item()) + assert int(runners[0].decided.item()) == 0 + assert total >= 5000 + for k in KX: + assert decided[k] > 0, ( + f"K={k}: no row decided in round 0 on the AgentX-like rows (SA-1R not engaged)" + ) + + +def _boundary_rows(rows, n_heavy_same_slice): + """Heavy tokens heavier than a 50-token candidate group; top_p swept across ulps of S / Z (margin fallback), or + n heavy tokens in one slice (more than its list: incomplete fallback).""" + lg = torch.full((rows, VS), -40.0, device="cuda") + sl = ((VS + 7) // 8 + 7) // 8 * 8 + heavy = list(range(5, 5 + n_heavy_same_slice)) if n_heavy_same_slice else [5, 3 * sl + 11] + grp = [2 * sl + 3 + 37 * j for j in range(25)] + [5 * sl + 7 + 41 * j for j in range(25)] + lg[:, heavy] = 5.0 + lg[:, grp] = 4.0 + lg = lg.to(torch.bfloat16).contiguous() + w = torch.exp(lg[0].double() - lg[0].double().max()) + crit = float(w[heavy].sum()) / float(w.sum()) + ks = torch.arange(rows, dtype=torch.float64) - rows // 2 + return lg, (crit * (1.0 + ks * 2.0**-24)).float() + + +@pytest.mark.parametrize("graph", [False, True]) +def test_one_round_boundary_and_incomplete(graph): + rows = 32 + g = torch.Generator(device="cuda").manual_seed(3) + seen = {k: [0, 0] for k in KX} # decided rows, rows + for n_same in (0, 12): + lg, tps_rows = _boundary_rows(rows, n_same) + for shift in range(-4, 4): + tps = (tps_rows.double() * (1.0 + shift * 32 * 2.0**-24)).float() + runners = {k: Runner(k, rows) for k in (0, *KX)} + temps, tp_t, seeds, offs = _params(rows, 1.0, tps, 5000 + shift) + bidx = torch.arange(rows // (D + 1), dtype=torch.int32, device="cuda") + draft = _draft_from(lg, g) + graphs = {} + if graph: + for r in runners.values(): + r(lg, temps, tp_t, seeds, offs, draft, bidx) + torch.cuda.synchronize() + for k, r in runners.items(): + graphs[k] = torch.cuda.CUDAGraph() + with torch.cuda.graph(graphs[k]): + r(lg, temps, tp_t, seeds, offs, draft, bidx) + for r in runners.values(): + r.decided.zero_() + for i in range(12): + offs.fill_(i) + for k, r in runners.items(): + if graph: + graphs[k].replay() + else: + r(lg, temps, tp_t, seeds, offs, draft, bidx) + for k in KX: + _compare(runners[0].out(), runners[k].out(), + f"boundary n_same={n_same} shift={shift} call={i} K={k}") # fmt: skip + for k in KX: + seen[k][0] += int(runners[k].decided.item()) + seen[k][1] += 12 * rows + for k in KX: + assert 0 < seen[k][0] < seen[k][1], ( + f"K={k}: decided {seen[k]}: the boundary sweep missed both sides" + ) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_shard_accept.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_shard_accept.py new file mode 100644 index 000000000000..eeff06f03fd0 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_shard_accept.py @@ -0,0 +1,120 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_shard_accept entry (greedy verify from this rank's logits shard).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_shard_accept import ( # noqa: E501 + SHARD_ROWS, + exchange_tokens, + local_argmax, + make_keys, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_shard_accept import ( # noqa: E501 + m3_shard_accept as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import ( # noqa: E402 + craft_drafts, + gen, + head_workspace, + num_sms, + plant_peers, + randn, + seed_of, +) +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_shard_accept requires a CUDA device" + + +def _case(spec, seed, edit_logits=None): + g = gen(seed) + nreq, d, vs, world, rank = ( + spec["nreq"], + spec["draft_len"], + spec["vs"], + spec["world"], + spec["rank"], + ) + m = nreq * (d + 1) + if spec["strided"]: + logits = randn((m, vs + 72), g, 3.0)[:, :vs] + else: + logits = randn((m, vs), g, 3.0) + if edit_logits is not None: + edit_logits(logits) + uc, keys, counters = head_workspace(world, slot_rows=SHARD_ROWS) + own = logits.float().max(dim=1).values + plant_peers( + uc, counters, own, rank, world, vs, g, slot_rows=SHARD_ROWS, ties=spec.get("ties", False) + ) + offset = rank * vs + best, idx = local_argmax(logits.float(), offset) + tokens = exchange_tokens(make_keys(best, idx), uc, counters, rank, world, SHARD_ROWS) + draft = craft_drafts(tokens, nreq, d) + batch_idx = torch.arange(nreq, dtype=torch.int32, device="cuda") + acc = torch.full((nreq, d + 1), -7, dtype=torch.int32, device="cuda") + num = torch.full((nreq,), -7, dtype=torch.int32, device="cuda") + gather = torch.full((nreq,), -7, dtype=torch.int64, device="cuda") + next_new = torch.full((nreq, d + 1), -7, dtype=torch.int32, device="cuda") + force_total = 3 if spec["force"] == "total" else 0 + force_vec = (torch.randint(1, d + 2, (nreq,), generator=g, device="cuda", dtype=torch.int32) + if spec["force"] == "vec" else None) # fmt: skip + return (logits, offset, keys, counters, uc, uc.data_ptr(), rank, world, draft, batch_idx, acc, num, gather, + force_total, next_new, num_sms(), force_vec) # fmt: skip + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Bit-exact: the op reduces BF16 logits without arithmetic, ties (within the shard and across ranks) go to + the lowest global index on both sides.""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + assert torch.equal(args[10], ref["accepted"]) + assert torch.equal(args[11], ref["num_accepted"]) + assert torch.equal(args[12], ref["gather_ids"]) + assert torch.equal(args[14], ref["next_new"]) + assert args[3].tolist() == [0, 1] and not bool(args[2].any()) + + +def test_guard_refuses_nan_logits() -> None: + args = list(_case(op.CELLS[0].spec, 1)) + args[0][1, 7] = float("nan") + with pytest.raises(AssertionError, match="NaN"): + with validating(op): + op(*args) + + +def test_guard_refuses_a_batch_index_past_the_requests() -> None: + args = list(_case(op.CELLS[0].spec, 2)) + args[9] = args[9] + 1 + with pytest.raises(AssertionError, match="batch_idx"): + with validating(op): + op(*args) + + +def _signed_zero_maxima(logits: torch.Tensor) -> None: + """Every row's maximum is zero, reached as -0.0 at a lower index and +0.0 at a higher one.""" + logits.fill_(-1.0) + logits[:, 3] = -0.0 + logits[:, 9] = 0.0 + + +@pytest.mark.parametrize("ties", [False, True], ids=["peer_rows", "peer_ties"]) +def test_signed_zero_maxima_take_the_lower_index(ties) -> None: + """-0.0 and +0.0 compare equal, so the lower index wins as with torch.argmax (also across ranks).""" + spec = dict(op.CELLS[1].spec, ties=ties) + args = _case(spec, seed_of(f"signed-zero-{ties}"), edit_logits=_signed_zero_maxima) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + assert torch.equal(args[10], ref["accepted"]) + assert torch.equal(args[11], ref["num_accepted"]) + assert torch.equal(args[12], ref["gather_ids"]) + assert torch.equal(args[14], ref["next_new"]) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_slot_scatter.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_slot_scatter.py new file mode 100644 index 000000000000..5278bb30cdb0 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_slot_scatter.py @@ -0,0 +1,57 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_slot_scatter entry (the spec sampler's per-slot stores in one launch).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_slot_scatter import ( # noqa: E501 + m3_slot_scatter as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import VOCAB, gen, seed_of # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_slot_scatter requires a CUDA device" + +STALE = -4 + + +def _case(spec, seed): + """Worker outputs and the store's transposed [slots, width] views of its [width, slots, 1] buffers.""" + g = gen(seed) + n, nslots = spec["n"], spec["slots"] + + def rnd(shape): + return torch.randint(0, VOCAB, shape, generator=g, device="cuda", dtype=torch.int32) + + store_new = torch.full((spec["store_w"], nslots, 1), STALE, dtype=torch.int32, device="cuda") + store_next = torch.full((spec["store_w"], nslots, 1), STALE, dtype=torch.int32, device="cuda") + store_lens = torch.full((nslots,), STALE, dtype=torch.int32, device="cuda") + store_draft = torch.full((nslots, spec["store_d"]), STALE, dtype=torch.int32, device="cuda") + slots = torch.randperm(nslots, generator=g, device="cuda")[:n] + lens = torch.randint(1, 5, (n,), generator=g, device="cuda", dtype=torch.int32) + return (slots, rnd((n, spec["src_w"])), store_new.squeeze(-1).T, rnd((n, spec["src_w"])), + store_next.squeeze(-1).T, lens, store_lens, rnd((n, spec["src_d"])), store_draft) # fmt: skip + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Index copies: exact, zero padding included, other slots untouched.""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + for name, got in (("da", args[2]), ("db", args[4]), ("dl", args[6]), ("dd", args[8])): + assert torch.equal(got, ref[name]), name + + +def test_guard_refuses_repeated_slots() -> None: + args = list(_case(op.CELLS[0].spec, 1)) + args[0] = args[0].clone() + args[0][1] = args[0][0] + with pytest.raises(AssertionError, match="distinct"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_spec_offsets.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_spec_offsets.py new file mode 100644 index 000000000000..00da4a8692f8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_spec_offsets.py @@ -0,0 +1,138 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for m3_spec_offsets (RuntimeHooks.spec_offsets_op): the overlap scheduler's two speculative step corrections +in one plain launch, bit for bit the in-place ATen adds, eager and in a CUDA graph.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_spec_offsets import ( # noqa: E501 + m3_spec_offsets as op, +) + +assert torch.cuda.is_available(), "m3_spec_offsets requires a CUDA device" + +I32 = torch.iinfo(torch.int32) + + +def _inputs(tokens, seqs, seed, wrap=False): + g = torch.Generator(device="cuda").manual_seed(seed) + lo, hi = (I32.min, I32.max) if wrap else (0, 1 << 20) + pos = torch.randint(lo, hi, (tokens,), generator=g, device="cuda", dtype=torch.int32) + kv = torch.randint(lo, hi, (seqs,), generator=g, device="cuda", dtype=torch.int32) + pos_off = torch.randint( + -8 if not wrap else lo, + 8 if not wrap else hi, + (tokens,), + generator=g, + device="cuda", + dtype=torch.int32, + ) + kv_off = torch.randint( + -8 if not wrap else lo, + 8 if not wrap else hi, + (seqs,), + generator=g, + device="cuda", + dtype=torch.int32, + ) + if wrap: + pos[:2], pos_off[:2] = I32.max, 1 # overflow wraps, as the ATen add does + kv[:1], kv_off[:1] = I32.min, -1 + return pos, pos_off, kv, kv_off + + +CELLS = [c.spec for c in op.CELLS] + [ + dict(tokens=1, seqs=1), + dict(tokens=300, seqs=75), + dict(tokens=4, seqs=0), +] + + +@pytest.mark.parametrize("spec", CELLS, ids=lambda s: "-".join(f"{k}{v}" for k, v in s.items())) +def test_bitwise_aten_adds(spec) -> None: + for trial in range(3): + pos, pos_off, kv, kv_off = _inputs( + spec["tokens"], spec["seqs"], 97 * trial + spec["tokens"], spec.get("wrap", False) + ) + ref_pos, ref_kv = pos.clone(), kv.clone() + op.reference(ref_pos, pos_off, ref_kv, kv_off) + assert op(pos, pos_off, kv, kv_off) is True + torch.cuda.synchronize() + assert torch.equal(pos, ref_pos) and torch.equal(kv, ref_kv), f"trial {trial}" + + +def test_graph_replays_read_the_live_offsets() -> None: + pos, pos_off, kv, kv_off = _inputs(32, 8, 5) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(pos, pos_off, kv, kv_off) + for step in range(3): + fresh = _inputs(32, 8, 11 + step) + for dst, src in zip((pos, pos_off, kv, kv_off), fresh): + dst.copy_(src) + ref_pos, ref_kv = pos.clone(), kv.clone() + op.reference(ref_pos, pos_off, ref_kv, kv_off) + graph.replay() + torch.cuda.synchronize() + assert torch.equal(pos, ref_pos) and torch.equal(kv, ref_kv), f"step {step}" + + +def test_runner_views_are_accepted() -> None: + """The runner passes position_ids[0, ctx:] and kv_lens_cuda[ctx_reqs:seqs]: 1-D slices of contiguous buffers.""" + position_ids = torch.arange(40, dtype=torch.int32, device="cuda").view(1, 40) + kv_lens = torch.arange(12, dtype=torch.int32, device="cuda") + pos_off = torch.full((40,), 2, dtype=torch.int32, device="cuda") + kv_off = torch.full((12,), 3, dtype=torch.int32, device="cuda") + assert op(position_ids[0, 8:], pos_off[:32], kv_lens[2:10], kv_off[:8]) is True + torch.cuda.synchronize() + assert torch.equal( + position_ids[0], torch.cat([torch.arange(8), torch.arange(8, 40) + 2]).int().cuda() + ) + assert torch.equal( + kv_lens, + torch.cat([torch.arange(2), torch.arange(2, 10) + 3, torch.arange(10, 12)]).int().cuda(), + ) + + +@pytest.mark.parametrize( + "bad", + ["int64", "strided", "length", "2d"], +) +def test_declines_other_layouts_without_writing(bad) -> None: + pos, pos_off, kv, kv_off = _inputs(8, 2, 3) + if bad == "int64": + kv = kv.long() + elif bad == "strided": + pos = torch.zeros(16, dtype=torch.int32, device="cuda")[::2] + elif bad == "length": + pos_off = pos_off[:7] + else: + pos = pos.view(2, 4) + before = (pos.clone(), kv.clone()) + assert op(pos, pos_off, kv, kv_off) is False + torch.cuda.synchronize() + assert torch.equal(pos, before[0]) and torch.equal(kv, before[1]) + + +def test_launches_plainly() -> None: + """It writes step inputs that pre-wait readers take before their grid wait (MSA contract, dispatch + PRODUCER_PREWAIT_STEP_INPUTS): its kernel node carries no programmatic (PDL) serialization.""" + from cuda.bindings import runtime as rt + + pos, pos_off, kv, kv_off = _inputs(32, 8, 7) + graph = torch.cuda.CUDAGraph(keep_graph=True) + with torch.cuda.graph(graph): + op(pos, pos_off, kv, kv_off) + raw = rt.cudaGraph_t(init_value=graph.raw_cuda_graph()) + err, _, n = rt.cudaGraphGetNodes(raw, 0) + err, nodes, n = rt.cudaGraphGetNodes(raw, n) + flags = [] + for node in nodes[:n]: + err, kind = rt.cudaGraphNodeGetType(node) + if kind == rt.cudaGraphNodeType.cudaGraphNodeTypeKernel: + err, attr = rt.cudaGraphKernelNodeGetAttribute( + node, rt.cudaLaunchAttributeID.cudaLaunchAttributeProgrammaticStreamSerialization + ) + flags.append(0 if attr is None else int(attr.programmaticStreamSerializationAllowed)) + assert flags == [0], flags diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_step_gather.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_step_gather.py new file mode 100644 index 000000000000..c75c45723b83 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_step_gather.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_step_gather entry (the overlap scheduler's step-input gathers in one launch).""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_step_gather import ( # noqa: E501 + m3_step_gather as op, +) + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import VOCAB, gen, seed_of # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_step_gather requires a CUDA device" + +#: Rows of the engine's flat buffers (the max batch), and their stale fill. +MAX_ROWS = 8 +STALE = -4 + + +def _case(spec, seed): + g = gen(seed) + p, t, d, nslots, b0 = spec["p"], spec["t"], spec["d"], spec["slots"], spec["b0"] + i32 = dict(dtype=torch.int32, device="cuda") + new_tok = torch.randint(0, VOCAB, (t, nslots, 1), generator=g, device="cuda", dtype=torch.int32) + if spec.get("two_d"): + new_tok = new_tok.squeeze(-1) + next_draft = torch.randint(0, VOCAB, (nslots, d), generator=g, device="cuda", dtype=torch.int32) + lens = torch.randint(1, t + 1, (nslots,), generator=g, device="cuda", dtype=torch.int32) + slots = torch.randperm(nslots, generator=g, device="cuda")[:p].to(torch.int32) + ppi = slots.repeat_interleave(t) + ids = torch.full((MAX_ROWS * t,), STALE, **i32) + draft = torch.full((MAX_ROWS * d,), STALE, **i32) + pos_off = torch.full((MAX_ROWS * t,), STALE, **i32) + kv_off = torch.full((MAX_ROWS,), STALE, **i32) + return ( + new_tok, + next_draft, + lens, + slots, + ppi, + p, + t, + d, + ids, + spec["ids0"], + draft, + spec["d0"], + pos_off, + kv_off, + b0, + ) + + +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Index copies: exact, including the zero-filled offsets and the untouched rest of ids / draft.""" + args = _case(cell.spec, seed_of(cell.why)) + certifying(op) + ref = op.reference(*args) + op(*args) + torch.cuda.synchronize() + for name, got in ( + ("ids", args[8]), + ("draft", args[10]), + ("pos_off", args[12]), + ("kv_off", args[13]), + ): + assert torch.equal(got, ref[name]), name + + +def test_guard_refuses_a_slot_past_the_store() -> None: + args = list(_case(op.CELLS[0].spec, 1)) + args[3] = args[3].clone() + args[3][0] = 8 + with pytest.raises(AssertionError, match="slots must index the store"): + with validating(op): + op(*args) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_step_prologue.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_step_prologue.py new file mode 100644 index 000000000000..ed30529df72c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/spec/test_modeling_v2_m3_step_prologue.py @@ -0,0 +1,228 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU test for the m3_step_prologue entry (the decode graph's step-input prologue in one launch).""" + +import random + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.spec.m3_step_prologue import ( # noqa: E501 + m3_step_prologue as op, +) +from tensorrt_llm._torch.pyexecutor import step_inputs +from tensorrt_llm._torch.pyexecutor.step_inputs import StepInputPack + +__extra_import_path__ = ["../../../..", "."] +from _m3_spec_inputs import gen, seed_of # noqa: E402 — needs the path declared above +from _validating import certifying, validating # noqa: E402, F401 + +assert torch.cuda.is_available(), "m3_step_prologue requires a CUDA device" +# The reference and the pack tests run this branch's step_inputs (the step epoch, prologue_op): a run that mounts only +# the target's files gets the image's older module, whose graph scatter has no epoch argument. +assert "epoch" in step_inputs._graph_scatter_kernel.arg_names, ( + "tensorrt_llm/_torch/pyexecutor/step_inputs.py is not this branch's (no step epoch): mount the shared pyexecutor " + "files (step_inputs.py, runtime_hooks.py, engine/runners/decoder/runner.py, h2d_cache.py) with the target" +) + +MAX_DESC = StepInputPack.MAX_DESC +WINDOW = StepInputPack.GRAPH_WINDOW +DATA_START = StepInputPack.HEADER_BYTES + StepInputPack.DESC_BYTES * MAX_DESC +ARENA = 1 << 20 + + +def _layout(spec, seed): + """Copies as (dst, src, bytes) arena offsets (a host copy's src: a window offset) and the host copies' bytes.""" + rng = random.Random(seed) + h2d, d2d, data = [], [], [] + used = DATA_START + pos = 0 + for _ in range(spec["h2d"]): + n = rng.choice([1, 3, 4, 7, 16, 24, 32, 48, 64, 5, 12]) + pos += rng.randint(0, 40) # gaps and every alignment + h2d.append((pos, used, n)) + data.append(bytes(rng.getrandbits(8) for _ in range(n))) + pos += n + used += -(-n // 16) * 16 + assert pos < ARENA // 2 and used <= WINDOW + # Device copies write the last quarter; unchained ones read the third. + dpos, spos = 3 * ARENA // 4, ARENA // 2 + for i in range(spec["d2d"]): + if spec.get("chained") and h2d: + dst_h, _, n_h = h2d[i % len(h2d)] + off = rng.randint(0, n_h - 1) + src, n = dst_h + off, n_h - off + else: + n = rng.choice([4, 8, 16, 32, 6, 64]) + spos += rng.randint(0, 24) + src = spos + spos += n + dpos += rng.randint(0, 24) + d2d.append((dpos, src, n)) + dpos += n + assert dpos < ARENA and spos < 3 * ARENA // 4 + return h2d, d2d, data + + +def _window(layout, base: int) -> torch.Tensor: + """The window of ``layout`` for an arena at device address ``base``.""" + h2d, d2d, data = layout + w = torch.zeros(WINDOW, dtype=torch.uint8) + words = [len(h2d), len(d2d)] + for dst, src, n in h2d: + words += [base + dst, src, n] + for dst, src, n in d2d: + words += [base + dst, base + src, n] + w[: 8 * len(words)] = torch.tensor(words, dtype=torch.int64).view(torch.uint8) + for (_, off, n), b in zip(h2d, data): + w[off : off + n] = torch.frombuffer(bytearray(b), dtype=torch.uint8) + return w.cuda() + + +def _arenas(seed): + a = torch.randint(0, 256, (ARENA,), generator=gen(seed), device="cuda", dtype=torch.int32) + a = a.to(torch.uint8) + return a, a.clone() + + +@pytest.mark.parametrize("with_epoch", [True, False], ids=["epoch", "no_epoch"]) +@pytest.mark.parametrize("cell", op.CELLS, ids=[c.why[:44] for c in op.CELLS]) +def test_certified_cells(cell, with_epoch, certifying) -> None: # noqa: F811 — pytest resolves the fixture by this name + """Byte for byte the default prologue's destinations, header clear and epoch.""" + seed = seed_of(cell.why) + layout = _layout(cell.spec, seed) + arena, ref_arena = _arenas(seed) + window, ref_window = _window(layout, arena.data_ptr()), _window(layout, ref_arena.data_ptr()) + epoch = torch.full((1,), 41, dtype=torch.int64, device="cuda") if with_epoch else None + ref_epoch = epoch.clone() if with_epoch else None + certifying(op) + op(window, epoch, MAX_DESC) + op.reference(ref_window, ref_epoch, MAX_DESC) + torch.cuda.synchronize() + assert torch.equal(arena, ref_arena) + # Each window holds its own arena's addresses: past the consumed header, both are left as staged. + for w, a in ((window, arena), (ref_window, ref_arena)): + assert not w[:16].any(), "the header is consumed" + assert torch.equal(w[16:], _window(layout, a.data_ptr())[16:]) + if with_epoch: + assert int(epoch.item()) == int(ref_epoch.item()) == 42 + + +def test_graph_replays_scatter_each_new_window() -> None: + """Captured once on a static window: every replay writes that replay's window and bumps the epoch once.""" + arena, ref_arena = _arenas(7) + window = torch.zeros(WINDOW, dtype=torch.uint8, device="cuda") + epoch = torch.zeros(1, dtype=torch.int64, device="cuda") + ref_epoch = epoch.clone() + op(window, epoch, MAX_DESC) # the empty window: writes nothing + op.reference(window.clone(), ref_epoch, MAX_DESC) + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + op(window, epoch, MAX_DESC) + for step, spec in enumerate( + [dict(h2d=24, d2d=8, chained=True), dict(h2d=0, d2d=0), dict(h2d=40, d2d=4)] + ): + layout = _layout(spec, 100 + step) + window.copy_(_window(layout, arena.data_ptr())) + graph.replay() + op.reference(_window(layout, ref_arena.data_ptr()), ref_epoch, MAX_DESC) + torch.cuda.synchronize() + assert torch.equal(arena, ref_arena), step + assert not window[:16].any(), step + assert int(epoch.item()) == int(ref_epoch.item()) == step + 2, step + + +def _pack_case(prologue_op): + pack = StepInputPack( + torch.device("cuda", torch.cuda.current_device()), + parallel_scatter=True, + in_graph=True, + early_copy=True, + step_epoch=True, + prologue_op=prologue_op, + ) + a, b, c = (torch.zeros(16, dtype=torch.int32, device="cuda") for _ in range(3)) + out = torch.zeros(48, dtype=torch.int32, device="cuda") + graph = torch.cuda.CUDAGraph() + with step_inputs.active(pack): + torch.cuda.synchronize() + with torch.cuda.graph(graph): + assert step_inputs.capture_prologue() is pack + out.copy_(torch.cat([a, b, c])) + return pack, graph, (a, b, c), out + + +def _pack_step(pack, graph, dsts, values, overlap=False, flush=True): + a, b, c = dsts + + def host(v): + return torch.tensor(v, dtype=torch.int32).pin_memory() + + with step_inputs.active(pack): + step_inputs.begin() + step_inputs.copy_(a, host(values[0])) + step_inputs.copy_(b, host(values[1])) + if overlap: + step_inputs.copy_(a[:4], host([-1, -2, -3, -4])) + step_inputs.copy_(c, a) # a device copy reading a host copy's destination + if not flush: + step_inputs.discard() + step_inputs.flush(in_graph=True) + graph.replay() + + +def test_pack_with_the_one_launch_prologue_matches_the_default() -> None: + """The same pack steps through the default prologue and the one-launch one: the same outputs and epochs, and the + one-launch prologue is engaged (the default's warm-up compile is not what bumped its epoch).""" + packs = [_pack_case(None), _pack_case(op)] + assert packs[0][0].prologue_op is None and packs[1][0].prologue_op is op + for step in range(6): + va = [step * 100 + i for i in range(16)] + vb = [step * 1000 - i for i in range(16)] + outs = [] + for pack, graph, dsts, out in packs: + # A replay with a new pack, one with an eagerly issued pack, and one with nothing to scatter. + _pack_step(pack, graph, dsts, (va, vb), overlap=step == 2, flush=step != 3) + torch.cuda.synchronize() + outs.append((out.tolist(), int(pack.epoch.item()))) + assert outs[0] == outs[1], step + assert outs[1][1] == step + 2, step # one bump at warm-up, then one per replay + + +def _kernel_nodes_launch_plainly(capture) -> list: + """The PDL attributes of every kernel node of a graph ``capture(stream)`` records: the step-input writers must be + plain launches (MSA pre-wait contract: no programmatic serialization or event on them).""" + from cuda.bindings import runtime as rt + + graph = torch.cuda.CUDAGraph(keep_graph=True) + with torch.cuda.graph(graph): + capture() + raw = rt.cudaGraph_t(init_value=graph.raw_cuda_graph()) + # cuda-python changed the signature: (graph[, numNodes]) on older bindings, (graph, nodes, numNodes) on newer ones. + try: + err, _, n = rt.cudaGraphGetNodes(raw) + err, nodes, n = rt.cudaGraphGetNodes(raw, n) + except TypeError: + err, _, n = rt.cudaGraphGetNodes(raw, None, 0) + err, nodes, n = rt.cudaGraphGetNodes(raw, None, n) + found = [] + for node in nodes[:n]: + err, kind = rt.cudaGraphNodeGetType(node) + if kind != rt.cudaGraphNodeType.cudaGraphNodeTypeKernel: + continue + err, attr = rt.cudaGraphKernelNodeGetAttribute( + node, rt.cudaLaunchAttributeID.cudaLaunchAttributeProgrammaticStreamSerialization + ) + # Older bindings return no attribute for a kernel launched without one (a plain launch). + found.append(0 if attr is None else int(attr.programmaticStreamSerializationAllowed)) + return found + + +def test_the_prologue_launches_plainly() -> None: + window = torch.zeros(WINDOW, dtype=torch.uint8, device="cuda") + epoch = torch.zeros(1, dtype=torch.int64, device="cuda") + op(window, epoch, MAX_DESC) + torch.cuda.synchronize() + flags = _kernel_nodes_launch_plainly(lambda: op(window, epoch, MAX_DESC)) + assert flags == [0], flags diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_ar_det.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_ar_det.py new file mode 100644 index 000000000000..7fca53cb9135 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_ar_det.py @@ -0,0 +1,54 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""AR-DET: the target defaults the AllReduce autotune off for its process (the C++ heuristic), keeps an explicit +TLLM_DISABLE_ALLREDUCE_AUTOTUNE as given, and leaves the autotune on with TRTLLM_M31_ENABLE_AR_AUTOTUNE=1.""" + +import os + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) + +KEY = "TLLM_DISABLE_ALLREDUCE_AUTOTUNE" + + +def test_autotune_defaults_off(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", False) + assert modeling._pin_allreduce_heuristic(0) is True + assert os.environ[KEY] == "1" + + +def test_escape_hatch_keeps_autotune_on(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", True) + assert modeling._pin_allreduce_heuristic(0) is False + assert KEY not in os.environ + + +def test_explicit_setting_is_kept(monkeypatch) -> None: + monkeypatch.setattr(os, "environ", {KEY: "0"}) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", False) + assert modeling._pin_allreduce_heuristic(0) is False + assert os.environ[KEY] == "0" + + +def test_escape_hatch_follows_the_switch_contract() -> None: + """TRTLLM_M31_ENABLE_AR_AUTOTUNE is parsed by m31_switches (1/true/on/yes set; ""/0/false/off/no unset), not by + a separate loose read.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + m31_switches, + ) + + assert modeling.AR_AUTOTUNE_ENABLED == m31_switches.m31_enabled("AR_AUTOTUNE") + + +def test_engagement_line_only_when_off(monkeypatch) -> None: + lines = [] + monkeypatch.setattr(modeling, "logger", type("L", (), {"info": staticmethod(lines.append)})) + for env, enabled, engaged in (({}, False, True), ({}, True, False), ({KEY: "0"}, False, False)): + lines.clear() + monkeypatch.setattr(os, "environ", dict(env)) + monkeypatch.setattr(modeling, "AR_AUTOTUNE_ENABLED", enabled) + modeling._pin_allreduce_heuristic(3) + assert any("rank 3: AR_DET engaged" in line for line in lines) == engaged, (env, enabled) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_blockoff_guard.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_blockoff_guard.py new file mode 100644 index 000000000000..849d61de7ffe --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_blockoff_guard.py @@ -0,0 +1,165 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""BLOCKOFF_GUARD on this target: ``_runtime_hooks`` turns on RuntimeHooks.skip_unchanged_draft_block_offsets by default +and ``TRTLLM_M31_DISABLE_BLOCKOFF_GUARD`` turns it off. The constant is evaluated from modeling.py's source against a +private copy of m31_switches loaded under each environment, so the heavy modeling module is not reloaded. CPU only, +except the last test, which imports the in-tree module. +""" + +import ast +import importlib +import importlib.util +import os +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp8" +CONSTANT = "STEP_BLOCKOFF_GUARD" +FIELD = "skip_unchanged_draft_block_offsets" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + + +def _module_tree() -> ast.Module: + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _value(monkeypatch, env) -> bool: + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_blockoff_{id(env)}", _SWITCHES) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + statements = [ + node + for node in _module_tree().body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id == CONSTANT + ] + assert len(statements) == 1 + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + return namespace[CONSTANT] + + +@pytest.mark.cpu_only +def test_on_by_default(monkeypatch): + assert _value(monkeypatch, {}) is True + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("value", ["1", "true", "ON", " yes "]) +def test_the_kill_switch_turns_it_off(monkeypatch, value): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_BLOCKOFF_GUARD": value}) is False + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("value", ["", "0", "false", "off", "no"]) +def test_an_unset_kill_switch_keeps_it(monkeypatch, value): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_BLOCKOFF_GUARD": value}) is True + + +@pytest.mark.cpu_only +def test_other_switches_leave_it_on(monkeypatch): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_HOSTTAIL": "1"}) is True + + +@pytest.mark.cpu_only +def test_runtime_hooks_passes_the_constant(): + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + calls = [ + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1 + keywords = {kw.arg: kw.value for kw in calls[0].keywords} + assert isinstance(keywords.get(FIELD), ast.Name) and keywords[FIELD].id == CONSTANT + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="imports the target's modeling module") +def test_the_in_tree_hook_follows_the_switch(): + modeling = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.modeling" + ) + hooks = modeling._runtime_hooks(None) + disabled = os.environ.get("TRTLLM_M31_DISABLE_BLOCKOFF_GUARD", "").strip().lower() in ( + "1", + "true", + "on", + "yes", + ) + assert getattr(hooks, FIELD) is getattr(modeling, CONSTANT) is (not disabled) + assert hooks.skips_unchanged_metadata_uploads + + +def _patch_value(monkeypatch, env) -> bool: + """STEP_BLOCKOFF_PATCH as modeling.py computes it under ``env`` (it needs BLOCKOFF_GUARD).""" + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location( + f"_m31_switches_blockoff_patch_{id(env)}", _SWITCHES + ) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + statements = [ + node + for node in _module_tree().body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id in (CONSTANT, "STEP_BLOCKOFF_PATCH") + ] + assert [node.targets[0].id for node in statements] == [CONSTANT, "STEP_BLOCKOFF_PATCH"] + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + return namespace["STEP_BLOCKOFF_PATCH"] + + +@pytest.mark.cpu_only +def test_the_patch_is_on_by_default(monkeypatch): + assert _patch_value(monkeypatch, {}) is True + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", ["BLOCKOFF_PATCH", "BLOCKOFF_GUARD"]) +def test_the_patch_kill_switch_and_the_guard_kill_switch_turn_it_off(monkeypatch, item): + assert _patch_value(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: "1"}) is False + + +@pytest.mark.cpu_only +def test_runtime_hooks_passes_the_patch_constant(): + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + call = next( + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ) + keywords = {kw.arg: kw.value for kw in call.keywords} + field = keywords.get("patch_unchanged_draft_block_offsets") + assert isinstance(field, ast.Name) and field.id == "STEP_BLOCKOFF_PATCH" diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_fc_shard.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_fc_shard.py new file mode 100644 index 000000000000..e441dd0db073 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_fc_shard.py @@ -0,0 +1,168 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""FC_SHARD on the sm_107 TP8 target: the drafter's fc as a column shard per rank + an all-gather. + +Each 768-column shard computed by ``trtllm::minimax_m31_fc_shard_gemm`` with the pinned algo equals the matching +columns of ``torch.nn.functional.linear(x, w)``, for every shard, every row count up to MAX_ROWS, real-scale and +2^+-12 block-scaled inputs, eager and in a CUDA graph; the shards gathered in rank order are the full fc. The wrapper +takes a shard only for the fc it was built from and the checked row counts, and the DSpark drafter's +project_target_hidden takes it only within those rows. Default on; TRTLLM_M31_DISABLE_FC_SHARD=1 turns it off.""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm import ( # noqa: E501 + m31_fc_shard, +) + +K, N, TP = 30720, 6144, 8 +COLS = N // TP + + +def _op_or_skip(): + if not torch.cuda.is_available(): + pytest.skip("needs a GPU") + op = m31_fc_shard.fc_shard_op() + if op is None: + pytest.skip("the minimax_m31_fc_shard_gemm op is not built") + return op + + +@pytest.fixture(scope="module") +def weight(): + g = torch.Generator(device="cpu").manual_seed(2026) + return (torch.randn(N, K, generator=g) * 0.02).to(torch.bfloat16).cuda() + + +def _inputs(rows, stress): + g = torch.Generator(device="cpu").manual_seed(100 + rows) + x = torch.randn(rows, K, generator=g) + if stress: + e = torch.randint(-12, 13, (rows, K // 128), generator=g).float() + x = x * torch.pow(2.0, e).repeat_interleave(128, dim=1) + return x.to(torch.bfloat16).cuda() + + +@pytest.mark.parametrize("stress", [False, True]) +@pytest.mark.parametrize("rows", list(range(1, m31_fc_shard.MAX_ROWS + 1))) +def test_every_shard_is_the_full_fc_columns(weight, rows, stress): + op = _op_or_skip() + x = _inputs(rows, stress) + full = F.linear(x, weight) + cfg = list(m31_fc_shard.PINNED_CFG) + parts = [] + for r in range(TP): + got = op(x, weight[r * COLS : (r + 1) * COLS].contiguous(), cfg) + assert torch.equal(got, full[:, r * COLS : (r + 1) * COLS]), r + parts.append(got) + assert torch.equal(torch.cat(parts, dim=-1), full) + + +def test_the_shard_in_a_cuda_graph(weight): + op = _op_or_skip() + x = _inputs(8, True) + w3 = weight[3 * COLS : 4 * COLS].contiguous() + cfg = list(m31_fc_shard.PINNED_CFG) + out = torch.empty(8, COLS, dtype=torch.bfloat16, device="cuda") + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + out.copy_(op(x, w3, cfg)) # set the algo up outside the capture + torch.cuda.current_stream().wait_stream(s) + g = torch.cuda.CUDAGraph() + with torch.cuda.graph(g): + out.copy_(op(x, w3, cfg)) + out.zero_() + g.replay() + torch.cuda.synchronize() + assert torch.equal(out, F.linear(x, weight)[:, 3 * COLS : 4 * COLS]) + + +def test_the_wrapper_shards_checks_and_gathers(weight, monkeypatch): + _op_or_skip() + fc = torch.nn.Linear(K, N, bias=False, device="cuda", dtype=torch.bfloat16) + fc.weight.data.copy_(weight) + x = _inputs(7, False) + shards = [] + for r in range(TP): + shard, why = m31_fc_shard.FcShard._check(fc, TP, r, SimpleNamespace(tp_size=TP, tp_rank=r)) + assert why is None and shard is not None, why + shards.append(shard) + s0 = shards[0] + assert s0.applies(fc, 7) and s0.applies(fc, m31_fc_shard.MAX_ROWS) + assert not s0.applies(fc, m31_fc_shard.MAX_ROWS + 1) and not s0.applies(fc, 0) + other = torch.nn.Linear(K, N, bias=False, device="cuda", dtype=torch.bfloat16) + assert not s0.applies(other, 7) + parts = [s.partial(x) for s in shards] + assert all(p.shape == (7, COLS) for p in parts) + # The all-gather concatenates the ranks' tensors in rank order along dim. + monkeypatch.setattr( + m31_fc_shard, "allgather", lambda y, mapping, dim: torch.cat(parts, dim=dim) + ) + assert torch.equal(s0.gather(parts[0]), fc(x)) + + +def test_a_non_applicable_fc_keeps_the_stock_path(): + _op_or_skip() + fc = torch.nn.Linear(K, N, bias=True, device="cuda", dtype=torch.bfloat16) + shard, why = m31_fc_shard.FcShard._check(fc, TP, 0, SimpleNamespace(tp_size=TP, tp_rank=0)) + assert shard is None and "do not apply" in why + + +class _HalfShard: + """A stand-in shard: the fc as two column halves, concatenated (the all-gather of a 2-rank group).""" + + def __init__(self, fc, rows): + self.fc, self.rows, self.calls = fc, rows, 0 + + def applies(self, fc, rows): + return fc is self.fc and 0 < rows <= self.rows + + def __call__(self, x): + self.calls += 1 + h = self.fc.weight.shape[0] // 2 + return torch.cat([F.linear(x, self.fc.weight[:h]), F.linear(x, self.fc.weight[h:])], dim=-1) + + +@pytest.mark.parametrize("rows", [1, 8, 9, 24]) +def test_project_target_hidden_takes_the_shard_within_its_rows(rows): + from tensorrt_llm._torch.models.modeling_dflash import DFlashForCausalLM + + g = torch.Generator().manual_seed(11) + fc = torch.nn.Linear(96, 32, bias=False, dtype=torch.bfloat16) + norm = torch.nn.RMSNorm(32, eps=1e-6, dtype=torch.bfloat16) + with torch.no_grad(): + fc.weight.copy_(torch.randn(32, 96, generator=g).bfloat16()) + norm.weight.copy_((1 + 0.1 * torch.randn(32, generator=g)).bfloat16()) + shard = _HalfShard(fc, 8) + model = SimpleNamespace(fc=fc, hidden_norm=norm, _dflash_fc_split=None, _dflash_fc_shard=shard) + x = torch.randn(rows, 96, generator=g) + out = DFlashForCausalLM.project_target_hidden(model, x) + assert torch.equal(out, norm(fc(x.to(torch.bfloat16)))) + assert shard.calls == (1 if rows <= 8 else 0) + + +@pytest.mark.parametrize( + "env, expected", [({}, True), ({"TRTLLM_M31_DISABLE_FC_SHARD": "1"}, False)] +) +def test_fc_shard_default_on_and_kill_switch(monkeypatch, env, expected): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + dspark, + m31_switches, + ) + + monkeypatch.delenv("TRTLLM_M31_DISABLE_FC_SHARD", raising=False) + for name, value in env.items(): + monkeypatch.setenv(name, value) + importlib.reload(m31_switches) + try: + assert importlib.reload(dspark).FC_SHARD is expected + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_FC_SHARD", raising=False) + importlib.reload(m31_switches) + importlib.reload(dspark) + assert dspark.FC_SHARD is True diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_host_tail_sm107_tp8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_host_tail_sm107_tp8.py new file mode 100644 index 000000000000..a93d65e03c95 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_host_tail_sm107_tp8.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The kill switches of this target's host-side step-loop items: ``_runtime_hooks`` turns on all ten RuntimeHooks fields +in ITEMS by default; ``TRTLLM_M31_DISABLE_`` turns off its own field (and, for the step-input pack, the fields that +need it: PACK_IN_GRAPH needs PACK_PSCATTER, PACK_EARLY_COPY needs PACK_IN_GRAPH); ``TRTLLM_M31_DISABLE_HOSTTAIL`` +turns off the six items of the sm_103 targets' HOSTTAIL bundle. + +The ``STEP_*`` expressions are evaluated from modeling.py's source against a private copy of m31_switches loaded +under each environment (as test_modeling_v2_m31_switches_sm107_tp8 does), so neither the in-tree switches nor the +heavy modeling module is reloaded. CPU only, except the last test, which imports the in-tree module. +""" + +import ast +import importlib +import importlib.util +import os +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.cache_manager import ( + MiniMaxM3KVCacheManagerV2, +) +from tensorrt_llm._torch.pyexecutor.kv_cache.mamba_cache_manager import ( + BaseMambaCacheManager, + MambaHybridCacheManager, +) +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + +TARGET = "minimax_m31_nvfp4__sm_107__tp8" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + +# Kill-switch id -> (the module constant, the RuntimeHooks field it sets). +ITEMS = { + "SKIP_ACCEPTED_UPLOAD": ("STEP_SKIP_ACCEPTED_UPLOAD", "skip_accepted_draft_upload"), + "BATCH_CHANGED_ONLY": ("STEP_BATCH_CHANGED_ONLY", "spec_batch_map_changed_only"), + "HOST_D2H_SIDE": ("STEP_HOST_D2H_SIDE", "sampler_d2h_side_stream"), + "SMALL_H2D": ("STEP_SMALL_H2D", "spec_small_h2d"), + "PHILOX_PACK": ("STEP_PHILOX_PACK", "pack_spec_rng_uploads"), + "KV_LENS_ONCE": ("STEP_KV_LENS_ONCE", "kv_lens_update_once"), + "PACK_PSCATTER": ("STEP_PACK_PSCATTER", "pack_parallel_scatter"), + "PACK_IN_GRAPH": ("STEP_PACK_IN_GRAPH", "pack_in_graph"), + "PACK_EARLY_COPY": ("STEP_PACK_EARLY_COPY", "pack_early_copy"), + "H2D_CHANGED_SPAN": ("STEP_H2D_CHANGED_SPAN", "h2d_changed_span"), +} +# The sm_103 HOSTTAIL bundle's six items, all turned off by TRTLLM_M31_DISABLE_HOSTTAIL. +HOSTTAIL = ( + "HOST_D2H_SIDE", + "SMALL_H2D", + "BATCH_CHANGED_ONLY", + "SKIP_ACCEPTED_UPLOAD", + "PHILOX_PACK", + "KV_LENS_ONCE", +) +# Item -> the items it needs (off with it). +NEEDS = {"PACK_IN_GRAPH": ("PACK_PSCATTER",), "PACK_EARLY_COPY": ("PACK_IN_GRAPH", "PACK_PSCATTER")} + + +def _off_with(item: str) -> set: + """The items a kill switch of ``item`` turns off: its own and every item that needs it.""" + return {item} | {other for other, needs in NEEDS.items() if item in needs} + + +def _module_tree() -> ast.Module: + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _item_statements(tree: ast.Module) -> list[ast.stmt]: + """modeling.py's module-level assignments of ``_HOSTTAIL_OFF`` and the ITEMS constants, in source order.""" + constants = {"_HOSTTAIL_OFF"} | {constant for constant, _ in ITEMS.values()} + return [ + node + for node in tree.body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id in constants + ] + + +def _switches(monkeypatch, env): + """A private copy of m31_switches imported under ``env`` (every other TRTLLM_M31_* variable cleared).""" + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_host_tail_{id(env)}", _SWITCHES) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _values(monkeypatch, env) -> dict[str, bool]: + """The ITEMS constants modeling.py computes under ``env``, by item id.""" + switches = _switches(monkeypatch, env) + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec( + compile(ast.Module(_item_statements(_module_tree()), []), str(_MODELING), "exec"), + namespace, + ) + return {item: namespace[constant] for item, (constant, _) in ITEMS.items()} + + +@pytest.mark.cpu_only +def test_the_switches_are_the_ten_items(): + names = [node.targets[0].id for node in _item_statements(_module_tree())] + + assert names == ["_HOSTTAIL_OFF", *(constant for constant, _ in ITEMS.values())] + for _, field in ITEMS.values(): + assert getattr(RuntimeHooks(), field) is False, field + + +@pytest.mark.cpu_only +def test_every_item_is_on_by_default(monkeypatch): + assert _values(monkeypatch, {}) == {item: True for item in ITEMS} + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", sorted(ITEMS)) +@pytest.mark.parametrize("value", ["1", "true", "ON", " yes "]) +def test_each_kill_switch_turns_off_only_its_own_item(monkeypatch, item, value): + values = _values(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: value}) + + assert values == {other: other not in _off_with(item) for other in ITEMS} + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", sorted(ITEMS)) +@pytest.mark.parametrize("value", ["", "0", "false", "off", "no"]) +def test_an_unset_kill_switch_keeps_the_item(monkeypatch, item, value): + assert _values(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: value}) == { + other: True for other in ITEMS + } + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("item", sorted(ITEMS)) +def test_a_typo_fails_at_import_naming_the_variable(monkeypatch, item): + with pytest.raises(ValueError, match="TRTLLM_M31_DISABLE_" + item): + _values(monkeypatch, {"TRTLLM_M31_DISABLE_" + item: "2"}) + + +@pytest.mark.cpu_only +def test_an_enable_variable_does_not_turn_an_item_off_or_on(monkeypatch): + env = {"TRTLLM_M31_ENABLE_" + item: "1" for item in ITEMS} + assert _values(monkeypatch, env) == {item: True for item in ITEMS} + + +@pytest.mark.cpu_only +def test_the_bundle_switch_turns_off_the_six_hosttail_items(monkeypatch): + """HOSTTAIL turns off the sm_103 bundle's six items, as on those targets; the pack items stay on.""" + assert _values(monkeypatch, {"TRTLLM_M31_DISABLE_HOSTTAIL": "1"}) == { + item: item not in HOSTTAIL for item in ITEMS + } + + +@pytest.mark.cpu_only +def test_runtime_hooks_wires_each_field_to_its_switch(): + """``_runtime_hooks`` passes each constant to its RuntimeHooks field, and nothing else sets those.""" + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + calls = [ + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1 + keywords = {kw.arg: kw.value for kw in calls[0].keywords} + for constant, field in ITEMS.values(): + assert isinstance(keywords.get(field), ast.Name), field + assert keywords[field].id == constant, field + + +@pytest.mark.cpu_only +def test_the_step_input_pack_and_its_epoch_stay_on(): + """The fields without a switch here: the pack itself, its step epoch and the one-launch prologue (the pack keeps + the epoch only with pack_in_graph and calls the prologue op only with pack_early_copy, so the switches above + turn those off with them).""" + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + call = next( + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ) + keywords = {kw.arg: kw.value for kw in call.keywords} + for field in ("pack_step_inputs", "pack_step_epoch"): + assert isinstance(keywords.get(field), ast.Constant), field + assert keywords[field].value is True, field + assert isinstance(keywords.get("pack_prologue_op"), ast.Name) + + +@pytest.mark.cpu_only +def test_the_target_kv_cache_is_not_a_mamba_cache(): + """skip_accepted_draft_upload is on only together with a non-mamba cache: the mamba state update is the one + consumer of accepted counts on the drafter path (it takes them as an argument, not from the spec metadata).""" + assert not issubclass(MiniMaxM3KVCacheManagerV2, BaseMambaCacheManager) + assert not issubclass(MiniMaxM3KVCacheManagerV2, MambaHybridCacheManager) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="imports the target's modeling module") +def test_the_in_tree_hooks_follow_the_switches(): + modeling = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.modeling" + ) + hooks = modeling._runtime_hooks(None) + + for constant, field in ITEMS.values(): + assert getattr(hooks, field) is getattr(modeling, constant), field + + def disabled(item): + return os.environ.get("TRTLLM_M31_DISABLE_" + item, "").strip().lower() in ( + "1", + "true", + "on", + "yes", + ) + + for item, (_, field) in ITEMS.items(): + off = disabled(item) or any(disabled(need) for need in NEEDS.get(item, ())) + off = off or (item in HOSTTAIL and disabled("HOSTTAIL")) + assert getattr(hooks, field) is not off, item diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_onediv_cap.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_onediv_cap.py new file mode 100644 index 000000000000..4e951599e8f8 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_onediv_cap.py @@ -0,0 +1,58 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR TP8 ONE_DIV merge: default off (TRTLLM_M31_ENABLE_LC8_ONEDIV=1 opts in, TRTLLM_M31_DISABLE_LC8_ONEDIV=1 still +wins); when on, the LC8 merge divides once up to 8 rows and TRTLLM_M31_DISABLE_ONEDIV_CAP=1 keeps it on at every row.""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + m31_switches, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_attn_lc8, +) + + +def test_onediv_default_off() -> None: + assert m3_attn_lc8.ONE_DIV_MERGE is False and m3_attn_lc8.ONE_DIV_MAX_ROWS == 8 + assert not any(m3_attn_lc8.one_div_applies(rows) for rows in (1, 4, 8, 9, 32)) + + +@pytest.mark.parametrize( + "enable, disable, on", [("1", "", True), ("1", "1", False), ("", "", False)] +) +def test_onediv_switches( + monkeypatch: pytest.MonkeyPatch, enable: str, disable: str, on: bool +) -> None: + monkeypatch.setenv("TRTLLM_M31_ENABLE_LC8_ONEDIV", enable) + monkeypatch.setenv("TRTLLM_M31_DISABLE_LC8_ONEDIV", disable) + try: + importlib.reload(m31_switches) + lc8 = importlib.reload(m3_attn_lc8) + assert lc8.ONE_DIV_MERGE is on + assert lc8.one_div_applies(8) is on and not lc8.one_div_applies(9) + finally: + monkeypatch.delenv("TRTLLM_M31_ENABLE_LC8_ONEDIV", raising=False) + monkeypatch.delenv("TRTLLM_M31_DISABLE_LC8_ONEDIV", raising=False) + importlib.reload(m31_switches) + importlib.reload(m3_attn_lc8) + + +@pytest.mark.parametrize("disable, applies_at_32", [("", False), ("0", False), ("1", True)]) +def test_onediv_cap_switch( + monkeypatch: pytest.MonkeyPatch, disable: str, applies_at_32: bool +) -> None: + monkeypatch.setenv("TRTLLM_M31_ENABLE_LC8_ONEDIV", "1") + monkeypatch.setenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", disable) + try: + importlib.reload(m31_switches) + lc8 = importlib.reload(m3_attn_lc8) + assert lc8.one_div_applies(1) and lc8.one_div_applies(8) + assert lc8.one_div_applies(32) is applies_at_32 + finally: + monkeypatch.delenv("TRTLLM_M31_ENABLE_LC8_ONEDIV", raising=False) + monkeypatch.delenv("TRTLLM_M31_DISABLE_ONEDIV_CAP", raising=False) + importlib.reload(m31_switches) + importlib.reload(m3_attn_lc8) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_proxy_prebuild.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_proxy_prebuild.py new file mode 100644 index 000000000000..116bf4355df3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_proxy_prebuild.py @@ -0,0 +1,57 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""PROXYPB on the sm_107 TP8 target: every fmha_sm100 variant the MSA indexer's proxy pass can pick (OnlyScore over +the FP8 paged index-K, page 128, no split-KV) is built once before warmup. One index head per rank packs nothing, so the +longest row alone picks the tile: <= 64, 65..128, > 128. + +The variant builds are recorded rather than compiled (an nvcc build each); the core is a stand-in with the attributes +the prebuild reads. +""" + +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels.msa_utils import ( + require_msa_module, +) + +INDEX_HEADS_PER_RANK = 1 +WANT = {"1_0_0_2_1_0_0", "1_0_1_2_1_0_0", "1_1_1_2_1_0_0"} + + +@pytest.fixture +def proxy_builds(monkeypatch): + require_msa_module() # its cutlass compatibility shim must precede the fmha_sm100 import, as in the target + from fmha_sm100 import jit as fmha_jit + + built = [] + + def record(*key): + built.append(fmha_jit._variant_key_from_runtime(*key)[0]) + return SimpleNamespace(run=None) + + monkeypatch.setattr(fmha_jit, "get_fmha_variant", record) + return built + + +def _core(): + attn = SimpleNamespace( + indexer_kv_dtype="fp8", m3_config=SimpleNamespace(num_index_heads=INDEX_HEADS_PER_RANK) + ) + return SimpleNamespace(sparse_layers=frozenset({3, 4, 5}), attn={3: attn, 4: attn, 5: attn}) + + +def test_builds_every_proxy_variant_once(proxy_builds) -> None: + names = modeling.MiniMaxM3Tp8ModelingV2._prebuild_msa_proxy(_core()) + assert set(names) == WANT + assert sorted(proxy_builds) == sorted(WANT), ( + "each variant is built once, across every sparse layer" + ) + + +def test_prebuild_is_on_by_default() -> None: + assert modeling.MSA_PROXY_PREBUILD is True diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_r6_dense_prequant.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_r6_dense_prequant.py new file mode 100644 index 000000000000..6e8b44f7d444 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_r6_dense_prequant.py @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""R6 on the sm_107 TP8 target: the dense-MLP layer boundary hands the next layer's QKV its MXFP8 activation by +VR: OFF by default (not bitwise on sm_107; no passing gate c): TRTLLM_M31_ENABLE_R6=1 +turns it on, TRTLLM_M31_DISABLE_R6=1 forces it off. The engagement itself ("[M3 dispatch] r6_dense_prequant taken" / +"off") is checked in-model.""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) + + +def test_r6_is_off_by_default_on_vr() -> None: + assert modeling.R6_DENSE_PREQUANT is False + + +@pytest.mark.parametrize( + ("enable", "disable", "expected"), + [("1", "", True), ("1", "1", False), ("0", "", False), ("", "1", False)], +) +def test_r6_switches( + monkeypatch: pytest.MonkeyPatch, enable: str, disable: str, expected: bool +) -> None: + monkeypatch.setenv("TRTLLM_M31_ENABLE_R6", enable) + monkeypatch.setenv("TRTLLM_M31_DISABLE_R6", disable) + try: + assert importlib.reload(modeling).R6_DENSE_PREQUANT is expected + finally: + monkeypatch.delenv("TRTLLM_M31_ENABLE_R6", raising=False) + monkeypatch.delenv("TRTLLM_M31_DISABLE_R6", raising=False) + importlib.reload(modeling) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s10_scope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s10_scope.py new file mode 100644 index 000000000000..e026b4fc1e04 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s10_scope.py @@ -0,0 +1,15 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR S10 scope (tp8): the lean drafter glue at every draft length (K3 and K7 gated token-identical).""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + dspark, +) + + +@pytest.mark.parametrize("k", [1, 3, 4, 7, 15]) +def test_s10_applies_at_every_draft_length(k: int) -> None: + assert dspark.S10_MAX_DRAFT_LEN == 63 + assert dspark.s10_applies(k) is True diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s5_scope.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s5_scope.py new file mode 100644 index 000000000000..6444334743a6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s5_scope.py @@ -0,0 +1,15 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR S5 scope (tp8): the window-capped drafter decode only up to the measured batch per draft length.""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + dspark, +) + + +@pytest.mark.parametrize("k, expected", [(1, 0), (3, 0), (5, 0), (7, 2), (15, 2)]) +def test_s5_max_batch_by_draft_length(k: int, expected: int) -> None: + assert dspark.S5_MAX_BATCH_BY_K == {3: 0, 7: 2} + assert dspark.s5_max_batch_for(k) == expected diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s9b_mc.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s9b_mc.py new file mode 100644 index 000000000000..162cd0e7622d --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_s9b_mc.py @@ -0,0 +1,124 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""S9B-MC: the sm_107 TP8 target runs S9b's fused drafter all-reduce + norm on its multicast all-reduce + norm (the +drafter-norm form, 8-token bucket) through the DFlash drafter's inert fused-AR-norm hook; other row counts, and every +drafter without the hook filled, keep the Linear's own fused all-reduce. Default off: TRTLLM_M31_ENABLE_S9B_MC=1 opts +in, TRTLLM_M31_DISABLE_S9B_MC=1 still wins.""" + +import importlib +import inspect +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + dspark, + m31_switches, +) +from tensorrt_llm._torch.models import modeling_dflash +from tensorrt_llm._torch.speculative.dflash import DFlashWorker + +STOCK, FROM_OP = ("stock-normed", "stock-res"), ("op-normed", "op-res") + + +class _Linear: + def __init__(self) -> None: + self.calls = 0 + + def all_reduce(self, partial, all_reduce_params): + self.calls += 1 + return STOCK + + +_NORM = SimpleNamespace(weight="w", variance_epsilon=1e-6) + + +def test_the_hook_is_inert_by_default() -> None: + linear = _Linear() + assert modeling_dflash._dflash_ar_residual_norm(linear, "x", "r", _NORM) == STOCK + assert modeling_dflash._dflash_ar_residual_norm(linear, "x", "r", _NORM, None) == STOCK + assert linear.calls == 2 + + +def test_the_op_runs_and_a_declined_call_falls_back() -> None: + seen = [] + + def op(partial, residual, norm_weight, eps): + seen.append((partial, residual, norm_weight, eps)) + return FROM_OP if partial == "take" else None + + linear = _Linear() + assert modeling_dflash._dflash_ar_residual_norm(linear, "take", "r", _NORM, op) == FROM_OP + assert linear.calls == 0 + assert modeling_dflash._dflash_ar_residual_norm(linear, "decline", "r", _NORM, op) == STOCK + assert linear.calls == 1 + assert seen[0] == ("take", "r", "w", 1e-6) + + +def test_every_fused_call_site_passes_the_hook() -> None: + src = inspect.getsource(modeling_dflash.DFlashForCausalLM) + assert src.count("_dflash_ar_residual_norm(") == 3 + assert ( + src.count("self._dflash_fused_ar_norm_op,") + src.count("self._dflash_fused_ar_norm_op\n") + >= 3 + ) + assert "self._dflash_fused_ar_norm_op = None" in src + + +@pytest.mark.parametrize("fused", [True, False]) +def test_the_setter_needs_the_fused_ar_norm(fused: bool) -> None: + model = SimpleNamespace(_dflash_fused_ar_norm=fused, _dflash_fused_ar_norm_op=None) + op = object() + assert DFlashWorker.enable_fused_ar_norm_op(None, model, op) is fused + assert model._dflash_fused_ar_norm_op is (op if fused else None) + assert DFlashWorker.enable_fused_ar_norm_op(None, SimpleNamespace(), op) is False + + +def test_default_off_with_the_8_token_bucket() -> None: + assert dspark.S9B_MC is False and dspark.S9B_MC_MAX_ROWS == 8 + + +@pytest.mark.parametrize("rows", [9, 16, 64]) +def test_the_op_declines_past_8_rows_without_a_workspace(monkeypatch, rows: int) -> None: + def no_workspace(*args, **kwargs): + raise AssertionError("the op must decline before asking for a workspace") + + monkeypatch.setattr(dspark.m3_mc_allreduce, "get_workspace", no_workspace) + op = dspark._s9b_mc_op(SimpleNamespace(tp_size=8)) + x = torch.zeros(rows, 16, dtype=torch.bfloat16) + assert op(x, x, torch.ones(16, dtype=torch.bfloat16), 1e-6) is None + + +def test_the_op_calls_the_drafter_norm_form(monkeypatch) -> None: + calls = [] + monkeypatch.setattr(dspark.m3_mc_allreduce, "get_workspace", lambda mapping, hidden: "ws") + + def mc(x, residual, w, eps, workspace, use_pdl=False, drafter_norm=False, **kwargs): + calls.append((tuple(x.shape), workspace, drafter_norm, kwargs)) + return FROM_OP + + monkeypatch.setattr(dspark.m3_mc_allreduce, "m3_mc_allreduce_norm", mc) + op = dspark._s9b_mc_op(SimpleNamespace(tp_size=8)) + x = torch.zeros(8, 16, dtype=torch.bfloat16) + assert op(x, x, torch.ones(16, dtype=torch.bfloat16), 1e-6) == FROM_OP + assert calls == [((8, 16), "ws", True, {})] + monkeypatch.setattr(dspark.m3_mc_allreduce, "get_workspace", lambda mapping, hidden: None) + assert op(x, x, torch.ones(16, dtype=torch.bfloat16), 1e-6) is None + + +@pytest.mark.parametrize( + "enable, disable, on", [("1", "", True), ("1", "1", False), ("", "1", False)] +) +def test_switches(monkeypatch: pytest.MonkeyPatch, enable: str, disable: str, on: bool) -> None: + monkeypatch.setenv("TRTLLM_M31_ENABLE_S9B_MC", enable) + monkeypatch.setenv("TRTLLM_M31_DISABLE_S9B_MC", disable) + importlib.reload(m31_switches) + try: + assert importlib.reload(dspark).S9B_MC is on + finally: + monkeypatch.delenv("TRTLLM_M31_ENABLE_S9B_MC", raising=False) + monkeypatch.delenv("TRTLLM_M31_DISABLE_S9B_MC", raising=False) + importlib.reload(m31_switches) + importlib.reload(dspark) + assert dspark.S9B_MC is False diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_slot_ids_changed_default.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_slot_ids_changed_default.py new file mode 100644 index 000000000000..62de4d1102b3 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_slot_ids_changed_default.py @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SLOT_IDS_CHANGED on this target: off by default, ``TRTLLM_M31_ENABLE_SLOT_IDS_CHANGED`` turns it on and +``TRTLLM_M31_DISABLE_SLOT_IDS_CHANGED`` still wins; ``_runtime_hooks`` passes the constant as +RuntimeHooks.spec_slot_ids_changed_only. The constant is evaluated from modeling.py's source against a private copy of +m31_switches loaded under each environment, so the heavy modeling module is not reloaded. CPU only. +""" + +import ast +import importlib.util +import os +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp8" +CONSTANT = "SLOT_IDS_CHANGED" +FIELD = "spec_slot_ids_changed_only" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + + +def _module_tree() -> ast.Module: + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _value(monkeypatch, env) -> bool: + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_slot_ids_{id(env)}", _SWITCHES) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + statements = [ + node + for node in _module_tree().body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id == CONSTANT + ] + assert len(statements) == 1 + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + return namespace[CONSTANT] + + +@pytest.mark.cpu_only +def test_off_by_default(monkeypatch): + assert _value(monkeypatch, {}) is False + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("value", ["1", "true", "ON", " yes "]) +def test_the_opt_in_turns_it_on(monkeypatch, value): + assert _value(monkeypatch, {"TRTLLM_M31_ENABLE_SLOT_IDS_CHANGED": value}) is True + + +@pytest.mark.cpu_only +def test_the_kill_switch_wins(monkeypatch): + env = {"TRTLLM_M31_ENABLE_SLOT_IDS_CHANGED": "1", "TRTLLM_M31_DISABLE_SLOT_IDS_CHANGED": "1"} + assert _value(monkeypatch, env) is False + + +@pytest.mark.cpu_only +def test_runtime_hooks_passes_the_constant(): + fn = next( + node + for node in _module_tree().body + if isinstance(node, ast.FunctionDef) and node.name == "_runtime_hooks" + ) + calls = [ + node + for node in ast.walk(fn) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1 + keywords = {kw.arg: kw.value for kw in calls[0].keywords} + assert isinstance(keywords.get(FIELD), ast.Name) and keywords[FIELD].id == CONSTANT diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_switches_sm107_tp8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_switches_sm107_tp8.py new file mode 100644 index 000000000000..1eef5c1dd667 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_switches_sm107_tp8.py @@ -0,0 +1,83 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The value contract of this target's kill switches (m31_switches): 1 / true / on / yes set TRTLLM_M31_DISABLE_ / +TRTLLM_M31_ENABLE_, empty / 0 / false / off / no leave it unset (case-insensitive), anything else raises at import +naming the variable; the environment is read once, at import. + +Each case loads a private copy of the module from its file, so the in-tree module (read by every other module of the +target at import) is never reloaded. CPU only. +""" + +import importlib.util +import os +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp8" +_SWITCHES = Path(_family.__file__).resolve().parent / TARGET / "m31_switches.py" + +pytestmark = pytest.mark.cpu_only + +KINDS = [("TRTLLM_M31_DISABLE_", "m31_disabled"), ("TRTLLM_M31_ENABLE_", "m31_enabled")] +ITEM = "UTCONTRACT" # an id no item uses, so the real switches of the test process cannot interfere + + +def _load(monkeypatch, env): + """A private copy of m31_switches, imported under ``env`` (every other TRTLLM_M31_* variable cleared).""" + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_contract_{id(env)}", _SWITCHES) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@pytest.mark.parametrize("prefix,query", KINDS) +@pytest.mark.parametrize( + "value", ["", " ", "0", " 0 ", "false", "FALSE", "off", " Off ", "no", "No"] +) +def test_unset_spellings_leave_the_switch_unset(monkeypatch, prefix, query, value): + assert not getattr(_load(monkeypatch, {}), query)(ITEM) + assert not getattr(_load(monkeypatch, {prefix + ITEM: value}), query)(ITEM) + + +@pytest.mark.parametrize("prefix,query", KINDS) +@pytest.mark.parametrize("value", ["1", " 1 ", "true", "TRUE", "on", "On", "yes", "YES"]) +def test_set_spellings_set_the_switch(monkeypatch, prefix, query, value): + assert getattr(_load(monkeypatch, {prefix + ITEM: value}), query)(ITEM) + + +@pytest.mark.parametrize("prefix", [k for k, _ in KINDS]) +@pytest.mark.parametrize("value", ["2", "-1", "1.0", "y", "n", "enable", "disabled", "truee", "of"]) +def test_any_other_value_fails_at_import_naming_the_variable(monkeypatch, prefix, value): + """Typos must fail loudly: a loose parse would count any value but "" / "0" as set (DISABLE_X=false disabling X).""" + with pytest.raises(ValueError, match=prefix + ITEM): + _load(monkeypatch, {prefix + ITEM: value}) + + +def test_the_id_is_matched_exactly(monkeypatch): + switches = _load(monkeypatch, {"TRTLLM_M31_DISABLE_" + ITEM: "1"}) + assert switches.m31_disabled(ITEM) + for other in (ITEM.lower(), ITEM + "_X", ITEM[:-1], ""): + assert not switches.m31_disabled(other), other + + +def test_disable_and_enable_are_independent(monkeypatch): + """Both read: an item's own code applies "DISABLE wins" (e.g. B1, L-DS1); the module reports each separately.""" + switches = _load( + monkeypatch, {"TRTLLM_M31_DISABLE_" + ITEM: "1", "TRTLLM_M31_ENABLE_" + ITEM: "1"} + ) + assert switches.m31_disabled(ITEM) and switches.m31_enabled(ITEM) + + +def test_the_environment_is_read_once_at_import(monkeypatch): + switches = _load(monkeypatch, {}) + monkeypatch.setenv("TRTLLM_M31_DISABLE_" + ITEM, "1") + monkeypatch.setenv("TRTLLM_M31_ENABLE_" + ITEM, "1") + assert not switches.m31_disabled(ITEM) and not switches.m31_enabled(ITEM) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_swq_rows.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_swq_rows.py new file mode 100644 index 000000000000..89efd1e9ae4e --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_swq_rows.py @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The VR TP8 SWQ rows: the shared expert's SwiGLU writes the down projection's MXFP8 activation past the MoE engine's +8-row fold up to R1VR8's 32-row step; TRTLLM_M31_DISABLE_SWQ=1 empties the table.""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) + + +def test_swq_rows_cover_the_unfolded_shared_down_rows() -> None: + if not modeling.R1VR8: + pytest.skip("R1-VR-TP8 is off: the decode step bound is the TP2 rows") + assert modeling.SWQ_ROWS == {modeling.SWQ_SHARED_DOWN: frozenset(range(9, 33))} + + +@pytest.mark.parametrize("disable, empty", [("", False), ("0", False), ("1", True)]) +def test_swq_kill_switch(monkeypatch: pytest.MonkeyPatch, disable: str, empty: bool) -> None: + monkeypatch.setenv("TRTLLM_M31_DISABLE_SWQ", disable) + try: + m = importlib.reload(modeling) + assert (m.SWQ_ROWS == {}) is empty + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_SWQ", raising=False) + importlib.reload(modeling) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tailbk.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tailbk.py new file mode 100644 index 000000000000..8014e2846c0c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tailbk.py @@ -0,0 +1,558 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""TAILBK on this target: ``TAILBK`` is on by default and ``TRTLLM_M31_DISABLE_TAILBK`` turns it off; ``_runtime_hooks`` +passes it as RuntimeHooks.spec_tail_bookkeep; the model logs the armed items once; and the worker's own items, R1 +(``M3DSparkWorker._rng_state_per_request``) and Z1 (``M3DSparkWorker._zero_padding_rows``), match the in-tree methods +bit for bit, eagerly and inside ``torch.cuda.graph``. + +The constant and the arm line are evaluated from modeling.py's source against a private copy of m31_switches loaded +under each environment, so the heavy modeling module is not reloaded (CPU only). The R1 / Z1 tests import the target's +dspark module and need CUDA. The shared items (B1, S1, A1) are tested in tests/unittest/_torch/speculative/ +test_tail_bookkeep.py. +""" + +import ast +import importlib +import importlib.util +import os +import types +from pathlib import Path + +import pytest +import torch + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp8" +CONSTANT = "TAILBK" +FIELD = "spec_tail_bookkeep" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" +ARM_PREFIX = "[M3 host] tail bookkeeping: " + +needs_cuda = pytest.mark.skipif( + not torch.cuda.is_available(), reason="imports the target's worker; CUDA buffers" +) + + +def _module_tree() -> ast.Module: + return ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + + +def _switches(monkeypatch, env): + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_tailbk_{id(env)}", _SWITCHES) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + return switches + + +def _top_level(names) -> list: + """modeling.py's module-level statements that bind one of ``names`` (assignments and function definitions).""" + out = [] + for node in _module_tree().body: + if isinstance(node, ast.FunctionDef) and node.name in names: + out.append(node) + elif isinstance(node, (ast.Assign, ast.AnnAssign)): + targets = node.targets if isinstance(node, ast.Assign) else [node.target] + if any(isinstance(t, ast.Name) and t.id in names for t in targets): + out.append(node) + return out + + +def _value(monkeypatch, env) -> bool: + switches = _switches(monkeypatch, env) + statements = _top_level({CONSTANT}) + assert len(statements) == 1 + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + return namespace[CONSTANT] + + +@pytest.mark.cpu_only +def test_on_by_default(monkeypatch): + assert _value(monkeypatch, {}) is True + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("value", ["1", "true", "ON", " yes "]) +def test_the_kill_switch_turns_it_off(monkeypatch, value): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_TAILBK": value}) is False + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("value", ["", "0", "false", "off", "no"]) +def test_an_unset_kill_switch_keeps_it(monkeypatch, value): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_TAILBK": value}) is True + + +@pytest.mark.cpu_only +@pytest.mark.parametrize( + "other", ["TAILBK_SPEC", "HOSTTAIL", "BLOCKOFF_GUARD", "SLOT_IDS_CHANGED", "R1", "R1VR8", "A1"] +) +def test_other_switches_leave_it_on(monkeypatch, other): + # A1 is another item's switch on this target (the verify all-gather) and R1VR8 the MSA-rows bound; R1 is the + # sm_103 targets' MSA-rows switch name. None of them is a TAILBK sub-item switch. + assert _value(monkeypatch, {f"TRTLLM_M31_DISABLE_{other}": "1"}) is True + + +MODEL_CLASS = "ModelingV2MiniMaxM31Nvfp4Sm107Tp8" + + +def _function(name: str, cls: str = None) -> ast.FunctionDef: + """A module-level function, or a method of the module-level class ``cls``.""" + scope = _module_tree().body + if cls is not None: + scope = next( + node for node in scope if isinstance(node, ast.ClassDef) and node.name == cls + ).body + return next(node for node in scope if isinstance(node, ast.FunctionDef) and node.name == name) + + +@pytest.mark.cpu_only +def test_runtime_hooks_passes_the_constant(): + calls = [ + node + for node in ast.walk(_function("_runtime_hooks")) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1 + keywords = {kw.arg: kw.value for kw in calls[0].keywords} + assert isinstance(keywords.get(FIELD), ast.Name) and keywords[FIELD].id == CONSTANT + + +@pytest.mark.cpu_only +def test_the_model_logs_the_arm_line_after_building_its_hooks(): + body = _function("__init__", MODEL_CLASS).body + names = [ + node.value.func.id + if isinstance(node, ast.Expr) + and isinstance(node.value, ast.Call) + and isinstance(node.value.func, ast.Name) + else None + for node in body + ] + assigns_hooks = [ + i + for i, node in enumerate(body) + if isinstance(node, ast.Assign) + and any(isinstance(t, ast.Attribute) and t.attr == "runtime_hooks" for t in node.targets) + ] + assert assigns_hooks and "_log_tailbk" in names + assert names.index("_log_tailbk") > assigns_hooks[0] + + +class _DSparkWorker: + """Stands in for dspark.M3DSparkWorker in the arm line's namespace.""" + + +def _arm_lines(monkeypatch, env, worker) -> list: + """The lines ``_log_tailbk(worker)`` logs on a fresh copy of the function (its once-per-process flag reset).""" + switches = _switches(monkeypatch, env) + statements = _top_level({CONSTANT, "_TAILBK_LOGGED", "_log_tailbk"}) + assert len(statements) == 3 + lines = [] + namespace = { + "m31_disabled": switches.m31_disabled, + "m31_enabled": switches.m31_enabled, + "_dspark": types.SimpleNamespace(M3DSparkWorker=_DSparkWorker), + "logger": types.SimpleNamespace(info=lines.append), + } + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + namespace["_log_tailbk"](worker) + namespace["_log_tailbk"](worker) # once per process + return lines + + +@pytest.mark.cpu_only +def test_arm_line_lists_the_dspark_workers_items_once(monkeypatch): + assert _arm_lines(monkeypatch, {}, _DSparkWorker()) == [ARM_PREFIX + "R1 B1 Z1 S1 A1"] + + +@pytest.mark.cpu_only +def test_arm_line_lists_only_the_accept_for_another_worker(monkeypatch): + assert _arm_lines(monkeypatch, {}, object()) == [ARM_PREFIX + "A1"] + + +@pytest.mark.cpu_only +def test_no_arm_line_when_off_or_without_speculation(monkeypatch): + assert _arm_lines(monkeypatch, {"TRTLLM_M31_DISABLE_TAILBK": "1"}, _DSparkWorker()) == [] + assert _arm_lines(monkeypatch, {}, None) == [] + + +@needs_cuda +def test_the_in_tree_hook_follows_the_switch(): + modeling = importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.modeling" + ) + hooks = modeling._runtime_hooks(None) + disabled = os.environ.get("TRTLLM_M31_DISABLE_TAILBK", "").strip().lower() in ( + "1", + "true", + "on", + "yes", + ) + assert getattr(hooks, FIELD) is getattr(modeling, CONSTANT) is (not disabled) + + +# --------------------------------------------------------------------------------------------------------------------- +# R1 and Z1 against the in-tree methods (CUDA: the real populate). +# --------------------------------------------------------------------------------------------------------------------- + +V = 64 +SLOT = 3 +# (temperature, top_k, top_p, min_p, seed): unseeded, int64-edge seed, a uint64 seed (negative as int64), seed 0. +REQUESTS = [ + (0.7, 0, 0.9, 0.0, None), + (1.3, 17, 0.5, 0.05, (1 << 63) - 1), + (0.25, 3, 1.0, 0.2, (1 << 64) - 1), + (1.0, 0, 0.95, 0.0, 0), +] + + +def _dspark(): + return importlib.import_module( + f"tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.{TARGET}.dspark" + ) + + +def _interface(): + return importlib.import_module("tensorrt_llm._torch.speculative.interface") + + +def _worker(attention_dp: bool = False): + worker = object.__new__(_dspark().M3DSparkWorker) # the overrides read only these fields + worker.mapping = types.SimpleNamespace(enable_attention_dp=attention_dp, tp_size=1, tp_rank=0) + worker._d2t = None + worker.force_num_accepted_tokens = 0.0 + return worker + + +def _request(params, request_id: int = 7): + from tensorrt_llm._torch.pyexecutor.llm_request import LlmRequestState + + temperature, top_k, top_p, min_p, seed = params + return types.SimpleNamespace( + sampling_config=types.SimpleNamespace( + temperature=temperature, top_k=top_k, top_p=top_p, min_p=min_p, seed=seed + ), + state=LlmRequestState.GENERATION_IN_PROGRESS, + py_seq_slot=SLOT, + py_request_id=request_id, + py_decoding_iter=0, + ) + + +def _meta(k: int, on: bool): + from tensorrt_llm.llmapi.llm_args import AdvancedSamplingMode + + interface = _interface() + meta = interface.SpecMetadata( + max_num_requests=8, + max_draft_len=k, + max_total_draft_tokens=k, + spec_dec_mode=interface.SpeculativeDecodingMode.DSPARK, + ) + meta.runtime_draft_len = k + meta.advanced_sampling_mode = AdvancedSamplingMode.NO_TOPK + meta.use_rejection_sampling = True + meta.vocab_size = V + meta.draft_vocab_size = V + meta.tail_bookkeep = on + return meta + + +def _populate(meta, params) -> None: + meta.populate_sampling_params_for_one_model([_request(params)]) + assert not meta.is_all_greedy_sample + + +def _stock_rng(worker, meta, k): + base = _interface().SpecWorkerBase._rng_state_per_request + accept = base(worker, meta, 0, 1, slot=meta.rng_accept_slot) + block = base(worker, meta, 0, 1, repeat=k, slot=meta.rng_draft_slot(0)) + return accept, block + + +def _r1_rng(worker, meta, k, stash: bool = True): + accept = worker._rng_state_per_request(meta, 0, 1, slot=meta.rng_accept_slot) + if not stash: + worker._tailbk_win = None + block = worker._rng_state_per_request(meta, 0, 1, repeat=k, slot=meta.rng_draft_slot(0)) + return accept, block + + +def _assert_rng_equal(got, want, what): + for (g_seed, g_off), (w_seed, w_off), part in zip(got, want, ("accept", "block")): + for g, w, name in ((g_seed, w_seed, "seed"), (g_off, w_off, "offset")): + assert g.dtype == w.dtype == torch.int64, f"{what} {part} {name} dtype" + assert g.shape == w.shape and g.is_contiguous(), f"{what} {part} {name} shape" + assert torch.equal(g, w), f"{what} {part} {name}: {g.tolist()} vs {w.tolist()}" + + +@needs_cuda +@pytest.mark.parametrize("k", range(1, 8)) +@pytest.mark.parametrize("stash", [True, False], ids=["stash", "no_stash"]) +def test_r1_eager_matches_the_in_tree_rows(k, stash): + worker = _worker() + on, off = _meta(k, True), _meta(k, False) + for params in REQUESTS: + _populate(on, params) + _populate(off, params) + _assert_rng_equal( + _r1_rng(worker, on, k, stash), _stock_rng(worker, off, k), f"K={k} {params}" + ) + assert worker._tailbk_win is None # the block consumed (or never had) the window + + +@needs_cuda +@pytest.mark.parametrize("k", [1, 3, 7]) +def test_r1_offsets_that_wrap_int64(k): + """The largest offset a host upload allows (o + K * SPAN = 2^63 - 1): the accept and draft rows wrap past 2^63 + in both, to the same bits (int64 addition is associative mod 2^64).""" + span = _interface()._RNG_SLOT_SPAN + worker = _worker() + on, off = _meta(k, True), _meta(k, False) + o = (1 << 63) - 1 - k * span + for meta in (on, off): + _populate(meta, REQUESTS[1]) + meta.request_offsets[0] = o + meta.offsets[: k + 1] = torch.tensor( + [o + r * span for r in range(k + 1)], dtype=torch.int64, device="cuda" + ) + got, want = _r1_rng(worker, on, k), _stock_rng(worker, off, k) + _assert_rng_equal(got, want, "wrap") + assert int(want[0][1][0]) < 0 # the accept row did wrap + + +@needs_cuda +def test_r1_counter_near_the_int64_edge(): + """A seeded slot whose window counter puts its offset base within one window of 2^63.""" + k = 7 + worker = _worker() + on, off = _meta(k, True), _meta(k, False) + window = (on.rng_accept_slot + 1) * _interface()._RNG_SLOT_SPAN + step = ((1 << 63) - 1 - k * _interface()._RNG_SLOT_SPAN) // window + for meta in (on, off): + meta._rng_slot_owner[SLOT] = 7 + meta._rng_window_counter[SLOT] = step + _populate(meta, REQUESTS[1]) + _assert_rng_equal(_r1_rng(worker, on, k), _stock_rng(worker, off, k), "edge counter") + + +@needs_cuda +def test_r1_stash_of_another_metadata_is_not_used(): + k = 3 + worker = _worker() + on, other, off = _meta(k, True), _meta(k, True), _meta(k, False) + _populate(on, REQUESTS[0]) + _populate(off, REQUESTS[0]) + _populate(other, REQUESTS[1]) + other_accept = worker._rng_state_per_request(other, 0, 1, slot=other.rng_accept_slot) + assert worker._tailbk_win is not None and worker._tailbk_win[0] is other + block = worker._rng_state_per_request(on, 0, 1, repeat=k, slot=on.rng_draft_slot(0)) + accept = worker._rng_state_per_request(on, 0, 1, slot=on.rng_accept_slot) + _assert_rng_equal((accept, block), _stock_rng(worker, off, k), "foreign stash") + assert other_accept[1].shape == (1,) + + +@needs_cuda +@pytest.mark.parametrize( + "case", ["flag_off", "two_requests", "context_row", "step_form_slot", "dynamic_draft_len"] +) +def test_r1_falls_back_to_the_in_tree_rows(case): + k = 3 + worker = _worker() + meta = _meta(k, case != "flag_off") + _populate(meta, REQUESTS[1]) + base = _interface().SpecWorkerBase._rng_state_per_request + args = { + "flag_off": ((0, 1), {"slot": meta.rng_accept_slot}), + "two_requests": ((0, 2), {"slot": meta.rng_accept_slot}), + "context_row": ((1, 2), {"slot": meta.rng_accept_slot}), + "step_form_slot": ((0, 1), {"slot": meta.rng_draft_slot(1)}), + "dynamic_draft_len": ((0, 1), {"repeat": k - 1, "slot": meta.rng_draft_slot(0)}), + }[case] + if case == "dynamic_draft_len": + meta.runtime_draft_len = k - 1 + got = worker._rng_state_per_request(meta, *args[0], **args[1]) + want = base(worker, meta, *args[0], **args[1]) + assert worker._tailbk_win is None + for g, w in zip(got, want): + assert torch.equal(g, w) + + +@needs_cuda +@pytest.mark.parametrize("k", [1, 4, 7]) +def test_r1_in_a_cuda_graph(k): + """Captured on the first request's populate, replayed after each later one (same buffers, new values).""" + worker = _worker() + on, off = _meta(k, True), _meta(k, False) + _populate(on, REQUESTS[0]) + _populate(off, REQUESTS[0]) + + def step(): + worker._tailbk_win = None # the forward's reset + return _r1_rng(worker, on, k) + + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + step() + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + captured = step() + for params in REQUESTS[1:]: + _populate(on, params) + _populate(off, params) + graph.replay() + want = _stock_rng(worker, off, k) + torch.cuda.synchronize() + _assert_rng_equal(captured, want, f"replay {params}") + + +@needs_cuda +def test_the_forward_resets_the_r1_window(monkeypatch): + dspark = _dspark() + seen = [] + monkeypatch.setattr( + dspark.S4WorkerMixin, + "_forward_impl", + lambda self, *a, **kw: seen.append(self._tailbk_win) or "out", + ) + worker = _worker() + worker._tailbk_win = ("stale", None) + assert worker._forward_impl(1, x=2) == "out" + assert seen == [None] and worker._tailbk_win is None + + +DUMMY_ROW = 16 # spec_metadata.dummy_slot_row + + +def _z1_meta(slots, on: bool = True, ragged: bool = False, fused: bool = False): + from tensorrt_llm.llmapi.llm_args import AdvancedSamplingMode + + return types.SimpleNamespace( + tail_bookkeep=on, + batch_slot_ids=torch.tensor(slots, dtype=torch.long, device="cuda"), + dummy_slot_row=DUMMY_ROW, + is_ragged_verify=ragged, + advanced_sampling_mode=AdvancedSamplingMode.FULL if fused else AdvancedSamplingMode.NO_TOPK, + ) + + +def _z1_logits(rows: int) -> torch.Tensor: + logits = torch.randn(rows, V, device="cuda") + logits[:, 0] = float("inf") + logits[:, 1] = -float("inf") + logits[:, 2] = -0.0 + return logits + + +def _capture(fn): + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + fn() + torch.cuda.current_stream().wait_stream(stream) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + out = fn() + return graph, out + + +def _stock_zero(*args, **kwargs): + return _interface().SpecWorkerBase._zero_padding_rows(*args, **kwargs) + + +@needs_cuda +@pytest.mark.parametrize("rows_per_request", [1, 7, 8]) +def test_z1_skips_only_while_capturing_batch_one(rows_per_request): + worker = _worker() + meta = _z1_meta([SLOT]) + logits = _z1_logits(rows_per_request) + eager = worker._zero_padding_rows(logits, meta, 0, 1, rows_per_request) + assert ( + eager is not logits + ) # eager steps (the graph warmup's dummy included) keep the in-tree zeroing + graph, captured = _capture( + lambda: worker._zero_padding_rows(logits, meta, 0, 1, rows_per_request) + ) + assert captured is logits + graph.replay() + # The in-tree result at the one real request is a bitwise copy of the logits. + stock = _stock_zero(logits, meta, 0, 1, rows_per_request) + torch.cuda.synchronize() + assert torch.equal(stock.view(torch.int32), logits.view(torch.int32)) + + +@needs_cuda +@pytest.mark.parametrize("batch", range(2, 9)) +@pytest.mark.parametrize("rows_per_request", [1, 8]) +def test_z1_keeps_the_zeroing_in_padded_graphs(batch, rows_per_request): + """Batches 2..8 can hold dummies: the captured graph zeroes their rows exactly as the in-tree method.""" + worker = _worker() + real = batch // 2 + meta = _z1_meta([SLOT + i for i in range(real)] + [DUMMY_ROW] * (batch - real)) + logits = _z1_logits(batch * rows_per_request) + graph, captured = _capture( + lambda: worker._zero_padding_rows(logits, meta, 0, batch, rows_per_request) + ) + assert captured is not logits + graph.replay() + stock = _stock_zero(logits, meta, 0, batch, rows_per_request) + torch.cuda.synchronize() + assert torch.equal(captured.view(torch.int32), stock.view(torch.int32)) + assert bool((captured[real * rows_per_request :] == 0).all()) + + +@needs_cuda +@pytest.mark.parametrize( + "case", + [ + "flag_off", + "attention_dp", + "fused_mode", + "ragged", + "context_row", + "strided_logits", + "unaligned_logits", + ], +) +def test_z1_keeps_the_zeroing_outside_its_condition(case): + worker = _worker(attention_dp=case == "attention_dp") + slots = [SLOT, SLOT + 1] if case == "context_row" else [SLOT] + meta = _z1_meta( + slots, on=case != "flag_off", ragged=case == "ragged", fused=case == "fused_mode" + ) + logits = _z1_logits(4) + if case == "strided_logits": # the copy would be contiguous: a consumer may dispatch on that + logits = torch.cat([logits, logits], dim=1)[:, :V] + elif case == "unaligned_logits": # the copy would be aligned: a consumer may dispatch on that + logits = torch.cat([torch.zeros(4, device="cuda"), logits.flatten()])[4:].view(4, V) + nc, bs = (1, 2) if case == "context_row" else (0, 1) + if case == "ragged": + meta.verify_lens = torch.tensor([4], dtype=torch.int32, device="cuda") + meta.total_verify_tokens = 4 + graph, captured = _capture( + lambda: worker._zero_padding_rows(logits, meta, nc, bs, 4, packed_verify=True) + ) + assert captured is not logits + graph.replay() + stock = _stock_zero(logits, meta, nc, bs, 4, packed_verify=True) + torch.cuda.synchronize() + assert torch.equal(captured.view(torch.int32), stock.view(torch.int32)) + + +if __name__ == "__main__": + import sys + + sys.exit(pytest.main([__file__, "-v"])) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_fused_projection_default.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_fused_projection_default.py new file mode 100644 index 000000000000..772c12075a7c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_fused_projection_default.py @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The M3.1 sm_107 TP8 target packs the sparse layers' QKV + index projection by default (a measured +win); an explicit ``fuse_qkv_index_projection`` in the deployment's config wins, so False is the kill switch.""" + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( + weights as tp_weights, +) +from tensorrt_llm.llmapi.llm_args import MiniMaxM3SparseAttentionConfig + +_MSA = dict(implementation="msa", indexer_kv_dtype="fp8") + + +@pytest.mark.parametrize( + ("explicit", "expected"), + [ + ({}, True), + ({"fuse_qkv_index_projection": True}, True), + ({"fuse_qkv_index_projection": False}, False), + ], +) +def test_fused_projection_default_and_kill(explicit: dict, expected: bool) -> None: + config = MiniMaxM3SparseAttentionConfig(**_MSA, **explicit) + assert tp_weights.fused_projection(config) is expected + + +def test_no_sparse_config_is_unfused() -> None: + assert tp_weights.fused_projection(None) is False + + +@pytest.mark.parametrize("value", [True, False]) +def test_a_config_without_set_fields_is_taken_at_its_word(value: bool) -> None: + from types import SimpleNamespace + + assert tp_weights.fused_projection(SimpleNamespace(fuse_qkv_index_projection=value)) is value diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_r1vr8.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_r1vr8.py new file mode 100644 index 000000000000..3061fb8664a6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_r1vr8.py @@ -0,0 +1,79 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""R1VR8: the sm_107 TP8 decode step bound is 32 rows (K3 c8) and every fast path keeps its own row range. + +The dispatch of a step of up to 8 rows is exactly the TP2-lineage one (the same plan, GEMV rows, pushes, embedding and +multicast ranges, linear paths); 9..32 rows take the 16- / 32-row plans, the GEMV on the QKV and gate_up projections up +to 32 rows and on the o_proj up to 16, the generic MoE above the engine's 8 rows; TRTLLM_M31_DISABLE_R1VR8=1 restores +the 8-row bound. +""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm.m3_mx_gemv import ( # noqa: E501 + DENSE_DOWN, + DENSE_GATE_UP, + O_PROJ, + QKV, + QKV_SPARSE, + SHARED_DOWN, + SHARED_GATE_UP, +) + +ALL = (QKV_SPARSE, QKV, O_PROJ, SHARED_GATE_UP, SHARED_DOWN, DENSE_GATE_UP, DENSE_DOWN) + + +def test_bounds() -> None: + assert modeling.TP2_ROWS == 8 and modeling.R1VR8 is True and modeling.DECODE_MAX_TOKENS == 32 + assert [p.name for p in modeling.DECODE_PLANS][:3] == ["tiny", "small", "verify"] + assert [p.max_tokens for p in modeling.DECODE_PLANS][:3] == [8, 16, 32] + + +@pytest.mark.parametrize("rows", range(1, 9)) +def test_up_to_8_rows_is_the_tp2_dispatch(rows: int) -> None: + m = modeling + assert m.is_decode_step(rows) and m.plan_for(rows).name == "tiny" + for rr in (m.EMBED_FIRSTNORM, m.ATTN_PUSH, m.DENSE_PUSH, m.MC_ALLREDUCE, m.MOE_ENGINE): + assert rr.covers(rows) == (rr.keys != frozenset()), rr.path + assert m.moe_path(rows) == (m.MOE_ENGINE.path if m.MOE_ENGINE.covers(rows) else None) + for nk in ALL: + assert rows in m.MX_GEMV_ROWS.get(nk, ()) or not m.MX_GEMV_TP2 + assert rows <= m.MX_GEMV_BF16_MAX_TOKENS.get(nk, 0) or not m.MX_GEMV_TP2 + assert m.linear_path(*nk, rows, True, False) == m.MX_GEMV_PATH + assert m.linear_path(*nk, rows, False, False) == m.MX_GEMV_BF16_PATH + + +@pytest.mark.parametrize("rows", range(9, 33)) +def test_9_to_32_rows(rows: int) -> None: + m = modeling + assert m.is_decode_step(rows) + assert m.plan_for(rows).name == ("small" if rows <= 16 else "verify") + assert m.MC_ALLREDUCE.covers(rows) + for rr in (m.EMBED_FIRSTNORM, m.ATTN_PUSH, m.DENSE_PUSH, m.MOE_ENGINE, m.MOE_PUSH): + assert not rr.covers(rows), rr.path # TP2-lineage paths stay at their 8 rows + assert m.moe_path(rows) is None + for nk in (QKV_SPARSE, QKV, SHARED_GATE_UP, DENSE_GATE_UP): + assert m.linear_path(*nk, rows, True, False) == m.MX_GEMV_PATH + assert m.linear_path(*O_PROJ, rows, True, False) == ( + m.MX_GEMV_PATH if rows <= 16 else m.PREQUANT_GEMM_PATH + ) + for nk in (SHARED_DOWN, DENSE_DOWN): # bf16 input: the pinned quantize + CuTe DSL GEMM + assert m.linear_path(*nk, rows, False, False) == m.GENERIC_GEMM_PATH + assert m.is_decode_step(33) is m.VR_BANDS and not m.is_decode_step(m.DECODE_STEP_MAX + 1) + + +def test_kill_switch(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("TRTLLM_M31_DISABLE_R1VR8", "1") + try: + m = importlib.reload(modeling) + assert m.DECODE_MAX_TOKENS == 8 and [p.name for p in m.DECODE_PLANS] == ["tiny"] + assert not m.is_decode_step(9) + assert all(m.MX_GEMV_ROWS[nk] == frozenset(range(1, 9)) for nk in ALL) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_R1VR8", raising=False) + importlib.reload(modeling) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_weights_layout.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_weights_layout.py new file mode 100644 index 000000000000..d043b46e0a40 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_tp8_weights_layout.py @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-M3.1 TP8 weight table: layer kinds from the config, packed-NVFP4 expert scalars as FP32 divisors.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( + weights as W, +) + +pytestmark = pytest.mark.cpu_only + + +def _core(sparse_freq, moe_freq, fused=True): + cfg = SimpleNamespace( + num_hidden_layers=len(sparse_freq), + moe_layer_freq=moe_freq, + sparse_attention_config=dict( + sparse_attention_freq=sparse_freq, sparse_num_index_heads=4, sparse_index_dim=128 + ), + head_dim=128, + num_attention_heads=64, + num_key_value_heads=4, + hidden_size=6144, + dense_intermediate_size=12288, + shared_intermediate_size=3072, + n_shared_experts=1, + intermediate_size=3072, + num_local_experts=128, + vocab_size=200064, + torch_dtype=torch.bfloat16, + ) + return SimpleNamespace( + model_config=SimpleNamespace( + pretrained_config=cfg, + mapping=SimpleNamespace(tp_size=8, tp_rank=0), + sparse_attention_config=SimpleNamespace(fuse_qkv_index_projection=fused), + ) + ) + + +def test_layer_kinds_come_from_the_config_independently(): + """Sparse attention on every layer while layers 0-1 keep the dense MLP.""" + d = W.MODEL_WEIGHTS.dims(_core([1] * 6, [0, 0, 1, 1, 1, 1])) + assert list(W._sparse_attn_layers(d)) == [0, 1, 2, 3, 4, 5] + assert list(W._dense_mlp_layers(d)) == [0, 1] + assert list(W._moe_layers(d)) == [2, 3, 4, 5] + assert list(W._five_way_qkv_layers(d)) == [0, 1, 2, 3, 4, 5] + assert list(W._three_way_qkv_layers(d)) == [] + + +def test_unfused_projection_uses_the_three_way_qkv_everywhere(): + d = W.MODEL_WEIGHTS.dims(_core([0, 0, 1, 1], [0, 0, 1, 1], fused=False)) + assert list(W._three_way_qkv_layers(d)) == [0, 1, 2, 3] + assert list(W._unfused_sparse_layers(d)) == [2, 3] + assert list(W._five_way_qkv_layers(d)) == [] + + +def test_expert_scalars_are_the_fp32_reciprocal_of_the_stored_multiplier(): + stored = torch.tensor(12345.678, dtype=torch.float32) + out = W._divisor(stored, None) + assert out.dtype == torch.float32 + assert out.view(torch.int32).item() == torch.reciprocal(stored).view(torch.int32).item() + with pytest.raises(ValueError): + W._divisor(torch.tensor(0.0), None) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_bands.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_bands.py new file mode 100644 index 000000000000..4c4fcc48fa0f --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_bands.py @@ -0,0 +1,109 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""VR-BANDS: the sm_107 TP8 decode step bound is 64 rows while every op keeps its own row range. + +Steps of 33..64 rows (K7 c5..c8) take the "wide" plan (the built-in all-reduce at both boundaries); each op arms +exactly one path at every row count 1..64 (its fast path inside its range, its generic form past it), and no op's +range moves. TRTLLM_M31_DISABLE_VR_BANDS=1 restores the 32-row step bound. +""" + +import importlib + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + m31_switches, + modeling, +) + +OP_BOUND = 32 +STEP_ROWS = range(1, 64 + 1) +ROW_TABLES = ("MX_GEMV_ROWS", "GEMV48_ROWS", "LINEAR_CUBLASLT_ROWS") + + +def _ranges(m) -> dict: + return {r.path: r.keys for r in m.FAST_PATHS} + + +def _projections(m) -> set: + tables = ROW_TABLES + ("MX_GEMV_BF16_MAX_TOKENS",) + return {nk for name in tables for nk in getattr(m, name, {})} + + +def _linear_paths(m) -> dict: + return { + nk: [ + m.linear_path(*nk, rows, prequant, False) + for rows in STEP_ROWS + for prequant in (True, False) + ] + for nk in _projections(m) + } + + +# The default (VR-BANDS on) module's view, recorded at collection before any reload. +_ON_RANGES = _ranges(modeling) +_ON_LINEAR_PATHS = _linear_paths(modeling) + + +@pytest.fixture +def vr_bands_off(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("TRTLLM_M31_DISABLE_VR_BANDS", "1") + importlib.reload(m31_switches) + try: + yield importlib.reload(modeling) + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_VR_BANDS", raising=False) + importlib.reload(m31_switches) + importlib.reload(modeling) + + +def test_the_step_bound_is_64_and_the_op_bound_stays() -> None: + m = modeling + assert m.VR_BANDS and m.DECODE_STEP_MAX == 64 and m.DECODE_MAX_TOKENS == OP_BOUND + assert [m.is_decode_step(n) for n in range(1, 66)] == [True] * 64 + [False] + + +@pytest.mark.parametrize("rows", STEP_ROWS) +def test_every_step_row_count_takes_exactly_one_plan(rows: int) -> None: + m = modeling + plan = m.plan_for(rows) + assert plan is [p for p in m.DECODE_PLANS if rows <= p.max_tokens][0] + if rows > OP_BOUND: + assert plan.name == "wide" and plan.max_tokens == 64 + assert (plan.attn_mc_ar, plan.moe_mc_ar, plan.qkv_prequant) == (False, False, False) + else: + assert plan.name != "wide" + + +@pytest.mark.parametrize("rows", STEP_ROWS) +def test_every_op_arms_exactly_one_path(rows: int) -> None: + """Each projection's GEMM row tables never overlap and it selects one GEMM per activation form; at most one MoE + path (the engine, the 9..13-row engine where built, the unfinalized MoE) covers the row count, and the MoE takes it + there and the generic MoE elsewhere.""" + m = modeling + for nk in _projections(m): + armed = [name for name in ROW_TABLES if rows in getattr(m, name, {}).get(nk, ())] + assert len(armed) <= 1, (nk, rows, armed) + for prequant in (True, False): + assert isinstance(m.linear_path(*nk, rows, prequant, False), str) + moe_ranges = [ + getattr(m, n) for n in ("MOE_ENGINE", "MOE16_ENGINE", "MC_FINALIZE") if hasattr(m, n) + ] + covering = [r.path for r in moe_ranges if r.covers(rows)] + assert len(covering) <= 1, (rows, covering) + assert m.moe_path(rows) == (covering[0] if covering else None) + + +def test_no_op_range_moves(vr_bands_off) -> None: + """The bound is the step's alone: every fast path's range and every projection's GEMM per row count is the same + with VR-BANDS off.""" + assert _ranges(vr_bands_off) == _ON_RANGES + assert _linear_paths(vr_bands_off) == _ON_LINEAR_PATHS + + +def test_kill_switch(vr_bands_off) -> None: + m = vr_bands_off + assert not m.VR_BANDS and m.DECODE_STEP_MAX == m.DECODE_MAX_TOKENS == OP_BOUND + assert "wide" not in [p.name for p in m.DECODE_PLANS] + assert m.is_decode_step(OP_BOUND) and not m.is_decode_step(OP_BOUND + 1) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_draft_fmha_sm100f.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_draft_fmha_sm100f.py new file mode 100644 index 000000000000..172079cece03 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_draft_fmha_sm100f.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""VR_DRAFT_FMHA_SM100F: ``M3DrafterSlidingFiFmha`` is FlashInfer's trtllm-gen FMHA admitted on SM 10.7 only, and +the target installs it on the standalone DSpark drafter by default; ``TRTLLM_M31_DISABLE_VR_DRAFT_FMHA_SM100F`` turns +it off. The constant is evaluated from modeling.py's source against a private copy of m31_switches. CPU only. +""" + +import ast +import importlib.util +import os +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +TARGET = "minimax_m31_nvfp4__sm_107__tp8" +CONSTANT = "VR_DRAFT_FMHA_SM100F" +_DIR = Path(_family.__file__).resolve().parent / TARGET +_SWITCHES = _DIR / "m31_switches.py" +_MODELING = _DIR / "modeling.py" + + +def _value(monkeypatch, env) -> bool: + for name in list(os.environ): + if name.startswith(("TRTLLM_M31_DISABLE_", "TRTLLM_M31_ENABLE_")): + monkeypatch.delenv(name) + for name, value in env.items(): + monkeypatch.setenv(name, value) + spec = importlib.util.spec_from_file_location(f"_m31_switches_draft_fmha_{id(env)}", _SWITCHES) + switches = importlib.util.module_from_spec(spec) + spec.loader.exec_module(switches) + tree = ast.parse(_MODELING.read_text(), filename=str(_MODELING)) + statements = [ + node + for node in tree.body + if isinstance(node, ast.Assign) + and len(node.targets) == 1 + and isinstance(node.targets[0], ast.Name) + and node.targets[0].id == CONSTANT + ] + assert len(statements) == 1 + namespace = {"m31_disabled": switches.m31_disabled, "m31_enabled": switches.m31_enabled} + exec(compile(ast.Module(statements, []), str(_MODELING), "exec"), namespace) + return namespace[CONSTANT] + + +@pytest.mark.cpu_only +def test_on_by_default(monkeypatch): + assert _value(monkeypatch, {}) is True + + +@pytest.mark.cpu_only +def test_the_kill_switch_turns_it_off(monkeypatch): + assert _value(monkeypatch, {"TRTLLM_M31_DISABLE_VR_DRAFT_FMHA_SM100F": "1"}) is False + + +@pytest.mark.cpu_only +@pytest.mark.parametrize("sm,available", [(100, False), (103, False), (107, True)]) +def test_the_library_is_sm107_only(monkeypatch, sm, available): + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_drafter_attn as mod, + ) + + monkeypatch.setattr(mod, "get_sm_version", lambda: sm) + monkeypatch.setattr(mod, "IS_FLASHINFER_AVAILABLE", True) + monkeypatch.setattr( + mod.M3DrafterSlidingFiFmha, "_missing_fused_nanobind_ops", staticmethod(lambda: []) + ) + assert mod.M3DrafterSlidingFiFmha.is_environment_available() is available diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_tp8_mcag.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_tp8_mcag.py new file mode 100644 index 000000000000..7b70c4ba23c6 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m31_vr_tp8_mcag.py @@ -0,0 +1,144 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""MCAG on the sm_107 TP8 target (VR port of the sm_103 test). + +The rejection sampler's draft-logits gather (SpecWorkerBase.maybe_gather_sharded_draft_logits) takes a target-installed +one-launch gather when it returns a tensor and the stock all-gather otherwise; the target's op declines by dtype / +columns / uncompiled rows. Default OFF, TRTLLM_M31_ENABLE_MCAG=1 turns it on and TRTLLM_M31_DISABLE_MCAG=1 still +wins. CPU only: the gathers are stand-ins.""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + dspark, + m31_switches, +) +from tensorrt_llm._torch.distributed import ops as dist_ops +from tensorrt_llm._torch.speculative.interface import SpecWorkerBase + +STOCK = "stock-gathered" + + +def _worker(op=None): + return SimpleNamespace( + mapping="tp8", + _draft_logits_are_sharded=lambda logits, md: True, + _draft_logits_gather_op=op, + ) + + +def test_the_hook_is_a_class_default(): + """Every stock spec worker reads the class attribute (None), no per-step attribute miss.""" + assert SpecWorkerBase._draft_logits_gather_op is None + assert "_draft_logits_gather_op" in SpecWorkerBase.__dict__ + + +@pytest.fixture +def stock(monkeypatch): + calls = [] + monkeypatch.setattr(dist_ops, "allgather", lambda x, mapping, dim: calls.append(dim) or STOCK) + return calls + + +def _gather(worker, logits, greedy=False): + md = SimpleNamespace(is_all_greedy_sample=greedy) + return SpecWorkerBase.maybe_gather_sharded_draft_logits(worker, logits, md) + + +def test_the_hook_is_inert_by_default(stock): + assert _gather(_worker(), torch.zeros(1, 7, 8)) == STOCK + assert stock == [-1] + + +def test_the_op_runs_and_a_declined_call_falls_back(stock): + x = torch.zeros(1, 7, 8) + assert _gather(_worker(lambda logits: "op-gathered"), x) == "op-gathered" + assert stock == [] + assert _gather(_worker(lambda logits: None), x) == STOCK + assert stock == [-1] + + +def test_greedy_batches_never_gather(stock): + x = torch.zeros(1, 7, 8) + assert _gather(_worker(lambda logits: "op"), x, greedy=True) is x + assert stock == [] + + +def test_the_setter(): + w = SimpleNamespace() + assert SpecWorkerBase.enable_draft_logits_gather_op(w, None) is False + op = object() + assert ( + SpecWorkerBase.enable_draft_logits_gather_op(w, op) is True + and w._draft_logits_gather_op is op + ) + + +@pytest.fixture +def mcag_target(monkeypatch): + calls = [] + compiled = {(7, 25008, 8)} + monkeypatch.setattr( + dspark._allgather_cast, + "is_compiled", + lambda rows, cols, world, grid=0, device=None, fp32=False: fp32 + and (rows, cols, world) in compiled, + ) + monkeypatch.setattr( + dspark._allgather_cast, + "gather_last_dim_f32", + lambda logits, **ws: calls.append((tuple(logits.shape), ws["world"])) or "mc-gathered", + ) + target = SimpleNamespace( + _mcag_cols=25008, + _a1={"world": 8, "ws_uc": None, "ws_mc": None, "ws_flags": None, "rank": 0}, + ) + return target, calls + + +def test_the_target_op_takes_compiled_fp32_rows(mcag_target): + target, calls = mcag_target + assert dspark.M3DSparkWorker._mcag_gather(target, torch.zeros(1, 7, 25008)) == "mc-gathered" + assert calls == [((1, 7, 25008), 8)] + + +@pytest.mark.parametrize( + "logits", + [ + torch.zeros(1, 7, 25008, dtype=torch.bfloat16), # not fp32 + torch.zeros(1, 7, 25000), # other columns + torch.zeros(2, 7, 25008), # rows not compiled + ], +) +def test_the_target_op_declines(mcag_target, logits): + target, calls = mcag_target + assert dspark.M3DSparkWorker._mcag_gather(target, logits) is None + assert calls == [] + + +@pytest.mark.parametrize( + "env, expected", + [ + ({}, False), + ({"TRTLLM_M31_ENABLE_MCAG": "1"}, True), + ({"TRTLLM_M31_ENABLE_MCAG": "1", "TRTLLM_M31_DISABLE_MCAG": "1"}, False), + ], +) +def test_default_off_opt_in_and_kill_switch(monkeypatch: pytest.MonkeyPatch, env, expected) -> None: + monkeypatch.delenv("TRTLLM_M31_DISABLE_MCAG", raising=False) + monkeypatch.delenv("TRTLLM_M31_ENABLE_MCAG", raising=False) + for name, value in env.items(): + monkeypatch.setenv(name, value) + importlib.reload(m31_switches) + try: + assert importlib.reload(dspark).MCAG is expected + finally: + monkeypatch.delenv("TRTLLM_M31_DISABLE_MCAG", raising=False) + monkeypatch.delenv("TRTLLM_M31_ENABLE_MCAG", raising=False) + importlib.reload(m31_switches) + importlib.reload(dspark) + assert dspark.MCAG is False diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_capture_boundary.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_capture_boundary.py new file mode 100644 index 000000000000..2c05cf103660 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_capture_boundary.py @@ -0,0 +1,192 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The generic layer's feed-forward exit on Eagle3 capture layers, in both targets' bodies (``PrefillTarget._layer``, +``DecodeTarget._generic_layer``). + +Every layer's MoE / MLP output goes unreduced to the fused boundary (all-reduce + residual + the next layer's input +norm, the final norm after the last layer), and a capture layer captures the reduced residual sum that boundary +writes, with no residual of its own. CPU only: the attention, the feed-forward and the boundaries are stand-ins. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) + +LAYERS = 4 + + +class _Spec: + def __init__(self, layers): + self.layers = set(layers) + + def is_layer_capture(self, i): + return i in self.layers + + +@pytest.fixture(params=["prefill", "decode"]) +def layer(request, monkeypatch): + """(core, the target's generic layer bound to a target over ``core``).""" + calls = [] + w = {f"l{i}_{n}": f"{n}{i}" for i in range(LAYERS) for n in ("in_norm", "post_norm")} + w["final_norm"] = "final" + c = SimpleNamespace( + w=w, + num_layers=LAYERS, + moe_layers=frozenset({2, 3}), + rms_eps=1e-6, + calls=calls, + # The prefill attention boundary passes a MoE layer's NVFP4 input scale to CTXNORMQ. + moe={i: SimpleNamespace(fc31_input_scale=None) for i in (2, 3)}, + ) + monkeypatch.setattr(modeling, "_can_mc", lambda x, folded=False: False) + # The prefill body tries the CTXNORMQ boundary first; it declines here (no op on these stand-ins). + monkeypatch.setattr(modeling, "_ctxnormq_allreduce_norm", lambda *args, **kwargs: None) + monkeypatch.setattr( + modeling, + "_native_allreduce_norm", + lambda core, x, r, norm: (("normed", x, norm), ("updated", x, r)), + ) + + def boundary(self, x, x2, residual, norm_w): + calls.append((x, x2, norm_w)) + return ("next", norm_w), ("sum", x, x2) + + if request.param == "prefill": + cls = modeling.PrefillTarget + monkeypatch.setattr(cls, "_moe", lambda self, i, h, **k: (("moe", i), ("routed", i))) + monkeypatch.setattr(cls, "_swiglu_mlp", lambda self, h, prefix, *a, **k: ("mlp", prefix)) + else: + cls = modeling.DecodeTarget + monkeypatch.setattr( + cls, "_moe", lambda self, i, h, **k: (("moe", i), ("routed", i), False, False) + ) + monkeypatch.setattr( + cls, "_swiglu_mlp", lambda self, h, prefix, *a, **k: (("mlp", prefix), False) + ) + monkeypatch.setattr(cls, "_attention", lambda self, i, *a, **k: (("attn", i), False)) + monkeypatch.setattr(cls, "_generic_boundary", boundary) + target = cls(c) + return c, target._layer if request.param == "prefill" else target._generic_layer + + +@pytest.mark.parametrize("i", range(LAYERS)) +def test_a_capture_layer_captures_the_boundary_residual_sum(layer, i) -> None: + core, run = layer + captured = [] + out = run( + i, + None, + torch.ones(2), + torch.zeros(2), + None, + _Spec([i]), + lambda spec, layer, h, r: captured.append((layer, h, r)), + ) + norm = core.w[f"l{i + 1}_in_norm"] if i + 1 < LAYERS else "final" + assert core.calls[-1][2] == norm, ( + "the boundary applies the next layer's norm (the final norm after the last)" + ) + assert captured == [(i, out[1], None)] + if i in core.moe_layers: + assert core.calls[-1][1] == ("routed", i), "the routed part folds into the boundary" + + +def test_a_layer_without_capture_captures_nothing(layer) -> None: + _, run = layer + captured = [] + run( + 1, + None, + torch.ones(2), + torch.zeros(2), + None, + _Spec([]), + lambda *a: captured.append(a), + ) + assert not captured + + +@pytest.fixture +def ctxnormq_prefill(monkeypatch): + """(core, the prefill target's generic layer, the boundary calls), with CTXNORMQ taking every boundary offered. + + Only the collectives are stand-ins: the layer and its exit boundary (``PrefillTarget._generic_boundary``) are the + target's own, so the test reads which boundary each result came from and where the layer uses it. + """ + calls = [] + w = {f"l{i}_{n}": f"{n}{i}" for i in range(LAYERS) for n in ("in_norm", "post_norm")} + w["final_norm"] = "final" + c = SimpleNamespace( + w=w, + num_layers=LAYERS, + moe_layers=frozenset({2, 3}), + rms_eps=1e-6, + moe={i: SimpleNamespace(fc31_input_scale=f"fp4_{i}") for i in (2, 3)}, + ) + monkeypatch.setattr(modeling, "_can_mc", lambda x, folded=False: False) + + def ctxnormq(core, x, residual, norm_w, fp4_scale): + calls.append(("ctxnormq", x, residual, norm_w, fp4_scale)) + return ("ctxnormq_normed", norm_w), ("ctxnormq_updated", norm_w) + + def native(core, x, residual, norm_w): + calls.append(("native", x, residual, norm_w)) + return ("native_normed", norm_w), ("native_updated", norm_w) + + monkeypatch.setattr(modeling, "_ctxnormq_allreduce_norm", ctxnormq) + monkeypatch.setattr(modeling, "_native_allreduce_norm", native) + cls = modeling.PrefillTarget + monkeypatch.setattr(cls, "_attention", lambda self, i, *a, **k: (("attn", i), False)) + monkeypatch.setattr(cls, "_moe", lambda self, i, h, **k: (("moe", i, h), None)) + monkeypatch.setattr(cls, "_swiglu_mlp", lambda self, h, prefix, *a, **k: ("mlp", prefix, h)) + return c, cls(c)._layer, calls + + +@pytest.mark.parametrize("i", range(LAYERS)) +def test_prefill_uses_the_ctxnormq_boundary_in_place_of_the_fused_allreduce( + ctxnormq_prefill, i +) -> None: + """A prefill boundary CTXNORMQ takes is used where the native fused all-reduce's would be. + + The attention boundary (with the MoE's NVFP4 input scale on a MoE layer) feeds the feed-forward, and its updated + residual goes to the exit boundary. The exit boundary is CTXNORMQ's into every next input norm; the final norm + keeps the native fused all-reduce. Its (normed, updated) is what the layer returns, and its updated is what a + capture layer captures. + """ + core, run, calls = ctxnormq_prefill + residual = torch.zeros(2) + captured = [] + out = run( + i, + None, + torch.ones(2), + residual, + None, + _Spec([i]), + lambda spec, layer, h, r: captured.append((layer, h, r)), + ) + post = core.w[f"l{i}_post_norm"] + kind, x, res, norm, fp4 = calls[0] + assert (kind, x, norm) == ("ctxnormq", ("attn", i), post), ( + "the attention boundary is offered to CTXNORMQ" + ) + assert res is residual + assert fp4 == (f"fp4_{i}" if i in core.moe_layers else None), "NVFP4 for the MoE's input only" + normed = ("ctxnormq_normed", post) + ff = ("moe", i, normed) if i in core.moe_layers else ("mlp", f"l{i}_mlp", normed) + if i + 1 < LAYERS: + nxt = core.w[f"l{i + 1}_in_norm"] + assert calls[1] == ("ctxnormq", ff, ("ctxnormq_updated", post), nxt, None) + assert out == (("ctxnormq_normed", nxt), ("ctxnormq_updated", nxt)) + else: + assert calls[1] == ("native", ff, ("ctxnormq_updated", post), "final"), ( + "the final norm keeps the native fused all-reduce" + ) + assert out == (("native_normed", "final"), ("native_updated", "final")) + assert len(calls) == 2 + assert captured == [(i, out[1], None)] diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_dispatch.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_dispatch.py new file mode 100644 index 000000000000..f31685544625 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_dispatch.py @@ -0,0 +1,612 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The tp8 (Rubin, two trays) decode target's fast-path table (``modeling.py``: the row ranges and the selection +functions): its row ranges against the catalog cells that certify them, and the selection for every decode row count. + +Every range's covered shapes are listed in ``ENABLED``; a range not listed covers nothing. + +CPU only: nothing here launches a kernel. +""" + +import inspect +import os +import subprocess +import sys +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling as dispatch, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_draft_step as draft_step_kernel, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + triton_sparse_decode as tsd, +) + +ROWS = range(1, dispatch.DECODE_STEP_MAX + 2) # every decode row count and the first one past it + + +@pytest.fixture(autouse=True) +def _policy(): + """Every test starts outside require mode with nothing logged, and leaves it so.""" + dispatch.set_require(False) + dispatch.reset_logged() + yield + dispatch.set_require(False) + dispatch.reset_logged() + + +@pytest.mark.parametrize("fast_path", dispatch.FAST_PATHS, ids=lambda r: r.path) +def test_every_covered_shape_is_a_certified_cell(fast_path) -> None: + uncertified = fast_path.keys - fast_path.certified() + assert not uncertified, ( + f"{fast_path.path} covers {sorted(uncertified, key=str)} but " + f"{type(fast_path.entry).__name__}.CELLS certifies none of them: add a cell or narrow the range" + ) + + +def test_the_fast_path_names_are_unique() -> None: + names = [r.path for r in dispatch.FAST_PATHS] + assert len(names) == len(set(names)) + + +def test_the_decode_target_takes_the_gen_workers_batch() -> None: + """The TP2-lineage paths' 8 rows (2 requests verifying 1 + 3 drafts); R1VR8 certifies the ops to 32 rows (8 + requests, K3 c8) and VR-BANDS raises the step bound to 64 (K7 c8); a wider step is the prefill target's.""" + assert dispatch.TP2_ROWS == 8 + assert dispatch.DECODE_MAX_TOKENS == _STEP_MAX + assert dispatch.DECODE_STEP_MAX == (64 if dispatch.VR_BANDS else _STEP_MAX) + assert [dispatch.is_decode_step(n) for n in ROWS] == [True] * dispatch.DECODE_STEP_MAX + [False] + + +# ------------------------------------------------------------------ the cells turned on + + +def test_the_moe_engine_takes_every_decode_step() -> None: + """The m3_moe engine (TP8 width 384, shared expert folded in) at 1..8 rows, unless killed; its sum + pushes into the multicast boundary all-reduce at the engine's rows while that all-reduce is on.""" + expected = set() if dispatch.KILL_MOE_ENGINE else set(range(1, 9)) + assert set(dispatch.MOE_ENGINE.keys) == expected + for rows in ROWS: + want = dispatch.MOE_ENGINE.path if rows in expected else None + assert dispatch.moe_path(rows) == want, rows + push_rows = dispatch.MOE_ENGINE_MAX_TOKENS if dispatch.MC_AR_TP2 else 0 + assert dispatch.MOE_PUSH_MAX_TOKENS == push_rows + assert set(dispatch.MOE_PUSH.keys) == set(range(1, push_rows + 1)) + + +# ------------------------------------------------------------------ on: the sparse layers' block selection + + +def test_the_index_scorer_takes_one_and_two_requests_with_their_schedules() -> None: + assert dispatch.INDEX_SCORE.keys == frozenset({1, 2}) + assert dispatch.index_score_schedule(1) == (1272, True) + assert dispatch.index_score_schedule(2) == (768, True) + # every covered request count has a split (the graph captures it) + assert set(dispatch.INDEX_SCORE.keys) <= set(dispatch.INDEX_SCORE_SPLIT_K) + + +def test_the_long_select_takes_every_key_width_up_to_its_maximum() -> None: + assert dispatch.LONG_SELECT.keys == frozenset(dispatch.LONG_SELECT_KEY_WIDTHS) + assert (dispatch.LONG_SELECT_MIN_BLOCKS, dispatch.LONG_SELECT_MAX_BLOCKS) == (129, 9216) + + +def test_the_384_thread_select_takes_every_key_width_with_its_launch_key() -> None: + assert dispatch.SELECT_LONG_DSL is True + assert dispatch.LONG_SELECT_DSL.keys == frozenset(dispatch.LONG_SELECT_DSL_KEY_WIDTHS) + keys = dispatch.long_select_dsl_keys() + assert [key[1:] for key in keys] == [(False, dispatch.SELECT_LONG_TRIGGER_PHASE)] + + +@pytest.mark.parametrize("num_blocks", [129, 1294, 5888, 9216]) +def test_a_long_select_decline_inside_its_range_is_a_miss(num_blocks) -> None: + """The kernel declining operands its cells certify must not fall back silently under require.""" + assert dispatch.long_select_covers(num_blocks) + # Outside require: logged once, the in-tree selector runs. + dispatch.long_select_fallback(num_blocks, 8) + dispatch.set_require(True) + with pytest.raises(dispatch.FastPathUnavailable, match="long_select"): + dispatch.long_select_fallback(num_blocks, 8) + + +@pytest.mark.parametrize("num_blocks", [1, 16, 128, 9217, 20000]) +def test_rows_outside_the_long_select_range_are_uncovered_even_under_require(num_blocks) -> None: + """Short rows and rows past LONG_SELECT_MAX_BLOCKS take the in-tree selector in every mode (long contexts must not + fail under require).""" + assert not dispatch.long_select_covers(num_blocks) + dispatch.set_require(True) + dispatch.long_select_fallback(num_blocks, 8) + + +def test_taken_logs_without_raising_under_require() -> None: + dispatch.set_require(True) + dispatch.taken(dispatch.INDEX_SCORE.path, 1) + dispatch.taken(dispatch.LONG_SELECT.path, 8) + + +def test_an_index_score_cell_of_another_schedule_certifies_nothing() -> None: + spec = dict(seq_lens=(165521,), qlen=4, heads=1, split_k=512, early=True) + assert tuple(dispatch.INDEX_SCORE.certifies(spec)) == () + assert tuple(dispatch.INDEX_SCORE.certifies(dict(spec, split_k=1272, heads=2))) == () + assert tuple(dispatch.INDEX_SCORE.certifies(dict(spec, split_k=1272))) == (1,) + + +#: Every TP8 per-rank MXFP8 projection (N, K): sparse-layer QKV + index, dense QKV, o_proj, shared gate_up / down, +#: dense gate_up / down. +TP8_PROJECTIONS = ( + (1536, 6144), + (1280, 6144), + (6144, 1024), + (768, 6144), + (6144, 384), + (3072, 6144), + (6144, 1536), +) +# R1VR8: the step bound is 32 rows; the TP2-lineage paths keep their 8 certified rows; +# on an MXFP8 activation the GEMV serves the QKV and both gate_up projections up to 32 rows and the o_proj up to 16 +# (the downs, bf16 input: 8). TRTLLM_M31_DISABLE_R1VR8=1: every range is 8 rows. +_STEP_MAX = 32 if dispatch.R1VR8 else 8 +_DECODE_ROWS = range(1, 8 + 1) # the TP2-lineage rows +_STEP_ROWS = range(1, _STEP_MAX + 1) +_GEMV_MAX = dict.fromkeys(TP8_PROJECTIONS, 8) +# Past the decode bound (VR_GEMV48 / LINEAR_CUBLASLT): the generation steps of 33..64 rows. +_WIDE_ROWS = range(max(33, dispatch.DECODE_MAX_TOKENS + 1), 64 + 1) +_GEMV48_ROWS = range(max(33, dispatch.DECODE_MAX_TOKENS + 1), 48 + 1) +_GEMV48_SHAPES = ( + (1536, 6144), + (1280, 6144), + (768, 6144), +) # sparse / dense-attention QKV, shared gate_up +_CUBLASLT_SHAPES = ( + (1536, 6144), + (6144, 1024), + (768, 6144), + (6144, 384), + (3072, 6144), + (6144, 1536), +) +if dispatch.R1VR8: + _GEMV_MAX.update( + {(1536, 6144): 32, (1280, 6144): 32, (768, 6144): 32, (3072, 6144): 32, (6144, 1024): 16} + ) + +#: Fast paths turned on, each with the shapes it covers (the rest of the skeleton stays off). +ENABLED = { + # Sampled (temperature + top-p) verify of 1-2 requests at draft length 3 (the shard kernel). + "m3_sampled_accept": frozenset({(3, 1), (3, 2)}), + # Greedy verify (shard argmax + accept) and the fused draft loop of 1-2 requests at draft length 3, + # and that loop's draft-input rows (1 / 2 later-pass rows, 4 / 8 first-pass rows). + "m3_shard_accept": frozenset({(3, 1), (3, 2)}), + "m3_draft_step": frozenset({(3, 1), (3, 2)}), + "m3_draft_input": frozenset({1, 2, 4, 8}), + # The drafter's BF16 linears on FlashInfer's low-M kernels (TP8 port) at the certified b1 rows. + "m3_draft_lowm": frozenset( + { + ((1280, 12288), 1), + ((1280, 12288), 4), + ((6144, 1024), 1), + ((6144, 1024), 4), + ((4608, 6144), 1), + ((4608, 6144), 4), + ((6144, 2304), 1), + ((6144, 2304), 4), + ((6144, 18432), 1), + } + ), + # The fused first drafter inputs of 1-2 fused-verify requests, the capture norm and the chunk norm at + # their 4 / 8 verify rows (own ranges and kills since the TP2 fast-path audit). + "m3_fused_first_inputs": frozenset({1, 2}), + "m3_capture_norm": frozenset({4, 8}), + "m3_chunk_norm": frozenset({4, 8}), + # MSA: the one-launch long-context sparse decode at the TP8 heads (pre-wait up to its rows, plain above), the + # identity-selection producer, and the sparse layers' NVFP4 QKV + indexer producer at the TP8 heads. + "m3_sparse_decode_lc8": frozenset({"plain", "prewait"}), + "m3_producer": frozenset({"nvfp4_tp8"}), + "sparse_decode_producer": frozenset({"multi_query", "per_row"}), + # MoE: the m3_moe engine (TP8 width 384, shared expert folded in) at 1..8 rows, unless killed. + "m3_moe_engine": frozenset() if dispatch.KILL_MOE_ENGINE else frozenset(range(1, 9)), + # The TF32 tensor-core router of 9..16-row pure generation steps on the generic layer (above the engine's rows). + "m3_router_tc": frozenset(range(9, (48 if dispatch.ROUTER48 else 16) + 1)), + # The index scorer at 1-2 requests, the long-row select at every key width (129..9216 blocks). + "m3_index_score": frozenset({1, 2}), + "long_select": frozenset({2, 4, 9}), + "long_select_dsl": frozenset({4, 11, 24}), + # The dense layers' full-context decode of 1-2 requests, 1 + 3 verify rows or one draft-pass row. + "m3_fullctx_decode": frozenset({(4, 1), (1, 1), (4, 2), (1, 2)}), + # The decode MXFP8 GEMVs at every TP8 projection and decode row, on an MXFP8 and a bf16 activation. + "m3_mx_gemv": frozenset((nk, m) for nk in TP8_PROJECTIONS for m in range(1, _GEMV_MAX[nk] + 1)) + if dispatch.MX_GEMV_TP2 + else frozenset(), + "m3_mx_gemv_bf16": frozenset((nk, m) for nk in TP8_PROJECTIONS for m in _DECODE_ROWS) + if dispatch.MX_GEMV_TP2 + else frozenset(), + # Past the decode bound, the GEMV (VR_GEMV48) on the QKV and shared gate_up projections at 33..48 rows and + # cuBLASLt's MXFP8 GEMM (LINEAR_CUBLASLT) on its six projections at every other row up to 64. + "m3_mx_gemv48": frozenset((nk, m) for nk in _GEMV48_SHAPES for m in _GEMV48_ROWS) + if (dispatch.MX_GEMV_TP2 and dispatch.VR_GEMV48) + else frozenset(), + "m3_mxfp8_cublaslt": frozenset( + (nk, m) + for nk in _CUBLASLT_SHAPES + for m in _WIDE_ROWS + if not ( + dispatch.MX_GEMV_TP2 + and dispatch.VR_GEMV48 + and nk in _GEMV48_SHAPES + and m in _GEMV48_ROWS + ) + ) + if dispatch.LINEAR_CUBLASLT + else frozenset(), + # The multicast boundary all-reduce, the o_proj's push into it, the fused embedding + first RMSNorm. + "mc_allreduce_norm": frozenset(_STEP_ROWS) if dispatch.MC_AR_TP2 else frozenset(), + "attn_push": frozenset(_DECODE_ROWS) + if (dispatch.MC_AR_TP2 and dispatch.ATTN_PUSH_TP2) + else frozenset(), + "dense_push": frozenset(_DECODE_ROWS) + if (dispatch.MC_AR_TP2 and dispatch.DENSE_PUSH_TP2) + else frozenset(), + # MoE: the engine's push into the multicast boundary all-reduce, at the engine's rows. + "moe_push": frozenset(range(1, dispatch.MOE_PUSH_MAX_TOKENS + 1)), + "embed_firstnorm": frozenset(_DECODE_ROWS) if dispatch.EMBED_FIRSTNORM_TP2 else frozenset(), + # The fused draft loop's all-reduce boundaries (1..8 rows) on the multicast all-reduce (opts 16). + "m3_draft_mc_allreduce_norm": frozenset(range(1, dispatch.DRAFT_MC_AR_MAX_ROWS + 1)), + # The Eagle3 drafter layer's generation passes of 1-2 requests on the same recipes. + "m3_drafter_fullctx": frozenset({(4, 1), (1, 1), (4, 2), (1, 2)}), +} + + +@pytest.mark.parametrize("fast_path", dispatch.FAST_PATHS, ids=lambda r: r.path) +def test_skeleton_no_fast_path_covers_a_shape(fast_path) -> None: + expected = ENABLED.get(fast_path.path, frozenset()) + assert fast_path.keys == expected, f"{fast_path.path} covers {sorted(fast_path.keys, key=str)}" + + +def test_the_sparse_decode_writes_the_mxfp8_handoff() -> None: + """The sparse decode (LC8, the Triton merge) writes the MXFP8 o_proj hand-off; the range keys the target's + SPARSE_DECODE_LC8 / SPARSE_DECODE_PRODUCER cover are the kernel module's.""" + assert tsd.MERGE_MXFP8_HANDOFF is True + assert dispatch.SPARSE_DECODE_LC8.keys is tsd.LC8_KEYS + assert dispatch.SPARSE_DECODE_PRODUCER.keys is tsd.PRODUCER_KEYS + + +def test_sparse_decode_lc8_covers_the_decode_steps_with_the_handoff() -> None: + """Every decode step (at most DECODE_MAX_TOKENS rows) writes the MXFP8 o_proj hand-off; the one-launch decode + stages its pre-wait reads up to m3_attn_lc8.PREWAIT_MAX_ROWS rows and merges once.""" + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_attn_lc8, + ) + + assert dispatch.SPARSE_DECODE_LC8.keys == frozenset({"plain", "prewait"}) + assert tsd.MERGE_MXFP8_MIN_TOKENS <= 1 and dispatch.DECODE_MAX_TOKENS <= tsd.MergeMxfp8.MAX_ROWS + assert ( + m3_attn_lc8.PREWAIT_MAX_ROWS == 8 + and m3_attn_lc8.MERGE_ONCE + and not m3_attn_lc8.ONE_DIV_MERGE + ) + # ONE_DIV is default off (TRTLLM_M31_ENABLE_LC8_ONEDIV=1 opts in); when on, up to 8 rows only ( + # TRTLLM_M31_DISABLE_ONEDIV_CAP=1 keeps it at every row count). + assert m3_attn_lc8.ONE_DIV_MAX_ROWS == 8 + assert m3_attn_lc8.one_div_applies( + 8 + ) is m3_attn_lc8.ONE_DIV_MERGE and not m3_attn_lc8.one_div_applies(9) + + +def test_the_dense_layers_take_the_fullctx_decode_at_the_gen_workers_shapes() -> None: + """One or two requests, 1 + 3 verify rows or one draft-pass row, each at its own pinned recipe.""" + assert dispatch.FULLCTX_DECODE_ENABLED + assert dispatch.FULLCTX_DECODE.keys == {(4, 1), (1, 1), (4, 2), (1, 2)} + assert set(dispatch.FULLCTX_DECODE_RECIPES) == set(dispatch.FULLCTX_DECODE.keys) + for recipe in dispatch.FULLCTX_DECODE_RECIPES.values(): + assert 2 <= recipe["splits"] <= 128 # the standalone reducer's range + + +def test_the_fullctx_entry_cells_run_at_the_forward_plan_capacity() -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.attention import ( # noqa: E501 + m3_fullctx_decode as entry, + ) + + assert entry.MAX_KV_LEN == dispatch.FULLCTX_DECODE_MAX_KV_LEN + + +def test_the_fullctx_capacity_covers_the_agentx_cache_bound() -> None: + """The forward compares the cache's length bound (max_seq_len plus the speculative rows it reserves) with the plan + capacity; at the AgentX max_seq_len of 1 << 20 with up to 8 draft rows the dense layers must still be taken.""" + assert dispatch.FULLCTX_DECODE_MAX_KV_LEN >= (1 << 20) + 8 + + +def test_the_drafter_takes_the_fullctx_decode_at_the_same_shapes() -> None: + assert dispatch.DRAFTER_FULLCTX_ENABLED + assert dispatch.DRAFTER_FULLCTX.keys == dispatch.FULLCTX_DECODE.keys + + +# ------------------------------------------------------------------ landed: the decode MXFP8 GEMVs + + +def test_the_gemv_rows_follow_the_kill_constant() -> None: + expected = set(TP8_PROJECTIONS) if dispatch.MX_GEMV_TP2 else set() + assert set(dispatch.MX_GEMV_ROWS) == expected + assert set(dispatch.MX_GEMV_BF16_MAX_TOKENS) == expected + + +@pytest.mark.parametrize("rows", _DECODE_ROWS) +@pytest.mark.parametrize("nk", TP8_PROJECTIONS, ids=str) +def test_every_projection_takes_the_gemv_at_every_decode_row(nk, rows) -> None: + """MXFP8 activation (a producer quantized it) -> the GEMV; bf16 -> the GEMV quantizing in its prologue (staged: + the stock decode-graph GEMMs while MX_GEMV_TP2 is off).""" + on = dispatch.MX_GEMV_TP2 + assert dispatch.linear_path(*nk, rows, prequant=True, low_m_gemm=True) == ( + dispatch.MX_GEMV_PATH if on else dispatch.PREQUANT_GEMM_PATH + ) + assert dispatch.linear_path(*nk, rows, prequant=False, low_m_gemm=True) == ( + dispatch.MX_GEMV_BF16_PATH if on else dispatch.GENERIC_GEMM_PATH + ) + + +@pytest.mark.parametrize("rows", [*_WIDE_ROWS, 65]) +@pytest.mark.parametrize("nk", TP8_PROJECTIONS, ids=str) +def test_wider_than_the_decode_step(nk, rows) -> None: + """Past the decode bound: the GEMV on GEMV48_SHAPES at 33..48 rows, cuBLASLt's MXFP8 GEMM on its shapes at the + other rows up to 64, else (and past 64) FlashInfer's GEMM; the same choice for either activation.""" + gemv48 = ( + (dispatch.MX_GEMV_TP2 and dispatch.VR_GEMV48) + and nk in _GEMV48_SHAPES + and rows in _GEMV48_ROWS + ) + cublaslt = ( + dispatch.LINEAR_CUBLASLT and nk in _CUBLASLT_SHAPES and rows in _WIDE_ROWS and not gemv48 + ) + for prequant, generic in ( + (True, dispatch.PREQUANT_GEMM_PATH), + (False, dispatch.GENERIC_GEMM_PATH), + ): + expected = ( + dispatch.GEMV48_PATH + if gemv48 + else (dispatch.CUBLASLT_GEMM_PATH if cublaslt else generic) + ) + assert dispatch.linear_path(*nk, rows, prequant=prequant, low_m_gemm=True) == expected + + +@pytest.mark.parametrize( + "switch, expected", + [ + # Without the GEMV past the bound, cuBLASLt takes those rows too; without cuBLASLt, FlashInfer's GEMM does. + ("TRTLLM_M31_DISABLE_VR_GEMV48", "0 m3_mxfp8_cublaslt m3_mxfp8_cublaslt"), + ("TRTLLM_M31_DISABLE_LINEAR_CUBLASLT", "0 m3_mx_gemv48 flashinfer_mxfp8_gemm"), + ], +) +def test_wide_kill_switches(switch, expected) -> None: + """Each switch, read at import, removes its rows (printed: the killed table's size, the 40-row QKV_SPARSE path, + the 56-row QKV_SPARSE path on an MXFP8 activation).""" + table = "GEMV48_ROWS" if "GEMV48" in switch else "LINEAR_CUBLASLT_ROWS" + code = ( + f"from {dispatch.__name__} import {table} as r, linear_path as p\n" + "print(len(r), p(1536, 6144, 40, True, True), p(1536, 6144, 56, True, True))" + ) + env = {**os.environ, switch: "1"} + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert " ".join(out.stdout.split()[-3:]) == expected + + +def test_router48_kill_switch() -> None: + """TRTLLM_M31_DISABLE_ROUTER48=1, read at import, keeps the tensor-core router at its 16-row range.""" + code = ( + f"from {dispatch.__name__} import ROUTER_TC_GEN as r\n" + "print(min(r.keys), max(r.keys), r.covers(32))" + ) + env = {**os.environ, "TRTLLM_M31_DISABLE_ROUTER48": "1"} + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.split()[-3:] == ["9", "16", "False"] + assert max(dispatch.ROUTER_TC_GEN.keys) == (48 if dispatch.ROUTER48 else 16) + + +def test_an_unknown_projection_is_the_generic_gemm() -> None: + assert ( + dispatch.linear_path(256, 512, 1, prequant=True, low_m_gemm=True) + == dispatch.PREQUANT_GEMM_PATH + ) + assert ( + dispatch.linear_path(256, 512, 1, prequant=False, low_m_gemm=True) + == dispatch.GENERIC_GEMM_PATH + ) + + +def test_taken_logs_once_per_path_and_rows(monkeypatch) -> None: + lines = [] + monkeypatch.setattr(dispatch.logger, "info", lines.append) + for _ in range(3): + dispatch.taken(dispatch.MX_GEMV_PATH, 4, "[6144, 4096] mxfp8 in") + dispatch.taken(dispatch.MX_GEMV_PATH, 8, "[6144, 4096] mxfp8 in") + assert len(lines) == 2 and all("m3_mx_gemv taken at" in line for line in lines) + + +def test_skeleton_the_dense_decode_split_is_the_built_in_one() -> None: + assert dict(dispatch.DENSE_DECODE_MAX_SEQ_LEN_BY_BATCH) == {} + + +def test_skeleton_speculation_runs_the_in_tree_loop() -> None: + # Fused greedy / sampled verify and the fused draft loop at b1/b2 (BF16 head; no coded heads). + assert dispatch.SHARD_ACCEPT_MAX_REQUESTS == 2 + assert dispatch.SAMPLED_VERIFY_MAX_REQUESTS == 2 + assert dispatch.DRAFT_STEP_MAX_REQUESTS == 2 + assert dispatch.FC_EARLY_MAX_ROWS == 0 + assert not draft_step_kernel.HEAD_CODED + + +def test_a_miss_raises_only_under_require() -> None: + dispatch.missed("some_path", 1, "a precondition") + dispatch.set_require(True) + with pytest.raises(dispatch.FastPathUnavailable, match="some_path"): + dispatch.missed("some_path", 1, "a precondition") + + +def test_skeleton_first_drafter_inputs_are_in_tree() -> None: + # The sampled verify is on (SAMPLED_VERIFY_MAX_REQUESTS 2, test_skeleton_speculation_...). + assert dispatch.FIRST_DRAFTER_INPUTS_MAX_REQUESTS == 0 + + +@pytest.mark.parametrize( + "rows, capture_normed, taken", + [ + (4, False, True), + (8, False, True), + (8, True, True), + (12, False, False), + (12, True, False), + (16, True, False), + ], +) +def test_the_fused_first_inputs_take_only_their_covered_rows(rows, capture_normed, taken) -> None: + """The fused first drafter inputs (and the fc fork) take a fused-verify step only at the requests + FUSED_FIRST_INPUTS covers and, unless the captures applied the fc_norm, the rows CHUNK_NORM covers; anything else + runs the in-tree first drafter inputs.""" + + worker = object.__new__(dispatch.M3Eagle3Worker) + worker._m3_capture_norm = (None, 1e-6) if capture_normed else None + meta = SimpleNamespace(runtime_draft_len=dispatch.SHARD_ACCEPT_DRAFT_LEN) + assert worker._m3_first_inputs_covered(rows, meta) is taken + + +# ------------------------------------------------------- landed: the multicast boundary all-reduce + + +def test_both_boundaries_take_the_multicast_all_reduce() -> None: + """The 8-token plan for the TP2 rows (R1VR8 adds the 16- and 32-row plans); every plan's boundaries take the + multicast all-reduce and the MoE boundary publishes the next QKV's MXFP8 rows (killed: every boundary on the + built-in all-reduce while MC_AR_TP2 is off).""" + on = dispatch.MC_AR_TP2 + assert tuple(p.max_tokens for p in dispatch.DECODE_PLANS) == ( + (8, 16, 32) if dispatch.R1VR8 else (8,) + ) + ((64,) if dispatch.VR_BANDS else ()) + for plan in dispatch.DECODE_PLANS: + want = (False, False, False) if plan.name == "wide" else (on, on, on) + assert (plan.attn_mc_ar, plan.moe_mc_ar, plan.qkv_prequant) == want, plan + assert dispatch.ATTN_PUSH_MAX_TOKENS == (8 if on and dispatch.ATTN_PUSH_TP2 else 0) + assert dispatch.DENSE_PUSH_MAX_TOKENS == (8 if on and dispatch.DENSE_PUSH_TP2 else 0) + for rows in _DECODE_ROWS: + assert dispatch.plan_for(rows) is dispatch.DECODE_PLANS[0] + # The MoE pushes only from its engine, at the engine's rows. + assert dispatch.MOE_PUSH_MAX_TOKENS == (dispatch.MOE_ENGINE_MAX_TOKENS if on else 0) + + +def test_the_generic_boundaries_take_the_multicast_all_reduce() -> None: + """Prefill, mixed steps and the Eagle3 capture layers: the same certified rows, alone and folded.""" + rows = 8 if dispatch.MC_AR_TP2 else 0 + assert dispatch.GENERIC_MC_AR_MAX_TOKENS == rows + assert dispatch.GENERIC_MC_AR_FOLDED_MAX_TOKENS == rows + + +# ------------------------------------------------------------------ landed: the fused embedding + first RMSNorm + + +def test_the_fused_embedding_rows_follow_the_kill_constant() -> None: + assert dispatch.EMBED_FIRSTNORM_MAX_TOKENS == (8 if dispatch.EMBED_FIRSTNORM_TP2 else 0) + + +# ------------------------------------------------------- landed: MX-TAC (pinned decode-graph MXFP8 GEMM tactic) + + +def test_the_decode_mxfp8_decision_is_pinned_to_cute_dsl() -> None: + from tensorrt_llm._torch.custom_ops.torch_custom_ops import ( + FlashInferMXFP8GemmRunner, + MXFP8QuantizeRunner, + ) + + assert dispatch.MXFP8_DECODE_QUANT_TACTIC == MXFP8QuantizeRunner.CUTE_DSL + assert dispatch.MXFP8_DECODE_GEMM_TACTIC == FlashInferMXFP8GemmRunner.CUTE_DSL + + +@pytest.mark.parametrize("target", ["DecodeTarget", "PrefillTarget"]) +def test_the_moe_forward_reports_the_engine_taken(target) -> None: + """The engagement evidence comes from the forward (not the engine build's warm call).""" + + src = inspect.getsource(getattr(dispatch, target)._moe) + assert src.index("taken(MOE_ENGINE.path") < src.index("m3_moe_engine(h.contiguous()") + + +# ------------------------------------------------------------------ the drafter's low-M linears (TP8 port) + + +def test_draft_lowm_covers_exactly_the_certified_rows_of_its_tactic_table() -> None: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.ops.gemm import ( # noqa: E501 + m3_draft_lowm as entry, + ) + + assert set(entry.TACTICS) == set(entry.ROWS) == set(dispatch.DRAFT_LOWM_ROWS) + assert dispatch.DRAFT_LOWM.keys == frozenset( + (nk, m) + for nk, rows in entry.ROWS.items() + for m in rows + if m <= dispatch.DRAFT_LOWM_MAX_ROWS + ) + for nk, table in entry.TACTICS.items(): + rows = [row[0] for row in table] + assert rows == sorted(rows) and len(set(rows)) == len(rows), nk + for m in entry.ROWS[nk]: + assert entry.tactic_for(*nk, m) is not None, (nk, m) + + +def test_prebuild_draft_head_is_idempotent_and_keeps_replaced_copies(monkeypatch) -> None: + """A repeated cache_derived_state with an unchanged head weight keeps the built (placed) copy; a changed weight + rebuilds it and keeps the old copy alive, so a CUDA graph captured earlier never reads unmapped memory.""" + + import torch + + built = [] + + def fake_prepare(lm_head, tp_rank): + head = SimpleNamespace(weight=lm_head.weight) + built.append(head) + return head + + monkeypatch.setattr(draft_step_kernel, "prepare_head", fake_prepare) + worker = object.__new__(dispatch.M3Eagle3Worker) + worker._m3_draft_head = None + worker._m3_draft_head_version = None + worker._m3_draft_head_graveyard = [] + worker.mapping = SimpleNamespace(tp_rank=0) + monkeypatch.setattr(worker, "_m3_drafter_supported", lambda dm: True, raising=False) + model = SimpleNamespace(lm_head=SimpleNamespace(weight=torch.zeros(4, 4))) + + assert worker.prebuild_draft_head(model) and len(built) == 1 + first = worker._m3_draft_head + assert worker.prebuild_draft_head(model) and len(built) == 1 and worker._m3_draft_head is first + model.lm_head.weight.add_(1.0) # new values: version bump + assert worker.prebuild_draft_head(model) and len(built) == 2 + assert worker._m3_draft_head is built[1] and worker._m3_draft_head_graveyard == [first] + + +def _runtime(fused: bool, indexer: str = "fp8", kv: str = "fp8"): + """A model config with the runtime fields ``_validate_runtime`` reads.""" + sparse = SimpleNamespace( + algorithm="minimax_m3", + implementation="msa", + fuse_qkv_index_projection=fused, + indexer_kv_dtype=indexer, + ) + mode = SimpleNamespace( + has_fp8_kv_cache=lambda: kv == "fp8", has_fp4_kv_cache=lambda: kv == "nvfp4" + ) + return SimpleNamespace( + sparse_attention_config=sparse, quant_config=SimpleNamespace(quant_mode=mode) + ) + + +@pytest.mark.parametrize("fused", [True, False], ids=["five_way", "unfused"]) +def test_the_target_takes_both_projection_layouts(fused) -> None: + """The packed five-way projection and the separate QKV + MXFP8 index projections + (fuse_qkv_index_projection=False) are both this target's; a BF16 indexer or a BF16 KV cache is not.""" + dispatch._validate_runtime(_runtime(fused)) + dispatch._validate_runtime(_runtime(fused, kv="nvfp4")) + with pytest.raises(ValueError, match="indexer_kv_dtype='fp8'"): + dispatch._validate_runtime(_runtime(fused, indexer="bf16")) + with pytest.raises(ValueError, match="FP8 or NVFP4 KV cache"): + dispatch._validate_runtime(_runtime(fused, kv="bf16")) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_rank_agreement.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_rank_agreement.py new file mode 100644 index 000000000000..8a07bf4630af --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/test_modeling_v2_m3_rank_agreement.py @@ -0,0 +1,190 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Decode robustness without a GPU kernel: the multicast all-reduce epilogue options keep the cluster-barrier bit, +an engine build that fails on one TP rank drops that engine on every rank (two ranks, and the tp8 target's eight, +simulated with threads), and the Eagle3 fc fork's output is recorded on the stream that reads it.""" + +import threading +from types import SimpleNamespace +from unittest import mock + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8 import ( # noqa: E501 + modeling, +) +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp8.kernels import ( # noqa: E501 + m3_mc_allreduce as mc, +) + + +@pytest.fixture(autouse=True) +def _restore_modes(): + require = modeling.require_mode() + yield + modeling.set_require(require) + modeling.reset_logged() + + +@pytest.mark.parametrize("finalize", [False, True]) +@pytest.mark.parametrize("bucket", [8, 16, 32, 48]) +def test_ar_opts_keep_cluster_barrier_bit(monkeypatch, bucket, finalize): + # Tables without bit 2 and with buckets missing: every lookup, fallback included, keeps the barrier. + monkeypatch.setattr(mc, "AR_OPTS_BY_M", (0, 1, 4, 5, 8, 9, 12, 13)) + monkeypatch.setattr(mc, "AR_OPTS_BY_BUCKET", {}) + monkeypatch.setattr(mc, "AR_FIN_OPTS_BY_BUCKET", {}) + for m in range(1, min(bucket, 8) + 1): + opts = mc._ar_opts(m, bucket, finalize=finalize) + assert opts & mc._OPT_CLUSTER_BARRIER and 0 <= opts <= 15, (m, bucket, finalize, opts) + + +def test_ar_opts_shipped_tables_unchanged(): + for m in range(1, 9): + assert mc._ar_opts(m, 8) == mc.AR_OPTS_BY_M[m - 1] + for b, o in mc.AR_OPTS_BY_BUCKET.items(): + assert mc._ar_opts(b, b) == o + for b, o in mc.AR_FIN_OPTS_BY_BUCKET.items(): + assert mc._ar_opts(b, b, finalize=True) == o + + +class _FakeComm: + """An MPI-like TP communicator for ``world`` threads: allreduce sums the ranks' ints.""" + + def __init__(self, world): + self.world = world + self.barrier = threading.Barrier(world) + self.lock = threading.Lock() + self.vals = [] + self.calls = 0 + + def allreduce(self, v): + with self.lock: + self.vals.append(v) + self.calls += 1 + self.barrier.wait() + total = sum(self.vals) + self.barrier.wait() + with self.lock: + self.vals.clear() + self.barrier.wait() + return total + + def Get_size(self): # noqa: N802 (MPI name) + return self.world + + +def _mapping(tp_size=2, adp=False): + return SimpleNamespace(tp_size=tp_size, enable_attention_dp=adp, tp_group=list(range(tp_size))) + + +def _run_ranks(fns): + out, errs = [None] * len(fns), [] + + def run(i): + try: + out[i] = fns[i]() + except BaseException as e: # noqa: BLE001 + errs.append(e) + + threads = [threading.Thread(target=run, args=(i,)) for i in range(len(fns))] + for t in threads: + t.start() + for t in threads: + t.join(timeout=30) + assert not errs, errs + return out + + +def _patch_comm(monkeypatch, world: int) -> _FakeComm: + from tensorrt_llm._torch.distributed import ops + + comm = _FakeComm(world) + monkeypatch.setattr(mc, "_tp_comm", lambda mapping: comm) + monkeypatch.setattr(ops, "mpi_disabled", lambda: False) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + return comm + + +@pytest.fixture +def two_rank_comm(monkeypatch): + return _patch_comm(monkeypatch, 2) + + +def test_tp_agree_single_rank_and_adp_take_no_collective(monkeypatch): + monkeypatch.setattr(mc, "_tp_comm", mock.Mock(side_effect=AssertionError("no collective"))) + assert mc.tp_agree(_mapping(1), True) is True + assert mc.tp_agree(_mapping(1), False) is False + assert mc.tp_agree(_mapping(8, adp=True), False) is False + + +def test_tp_agree_refuses_capture(monkeypatch): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: True) + with pytest.raises(RuntimeError): + mc.tp_agree(_mapping(2), True) + + +@pytest.mark.parametrize("oks", [(True, True), (True, False), (False, True), (False, False)]) +def test_tp_agree_built_two_ranks(two_rank_comm, oks): + built = [object() if ok else None for ok in oks] + out = _run_ranks([lambda i=i: mc.tp_agree_built(_mapping(2), built[i], "t") for i in range(2)]) + assert out == (built if all(oks) else [None, None]) + + +def _core(fail: bool, world: int = 2) -> SimpleNamespace: + """The engine-building half of the core (``MiniMaxM3Tp8ModelingV2``) on one of ``world`` simulated TP ranks, + its engine builds stubbed (and failing where ``fail``).""" + core = SimpleNamespace(mapping=_mapping(world), moe_engines={}) + cls = modeling.MiniMaxM3Tp8ModelingV2 + + def build(i, num_tokens, shared=True): + if fail: + raise RuntimeError("compile failed on this rank") + return ("engine", num_tokens) + + core._build_engine = build + for name in ("_prebuild_engines", "_try_build", "engine", "_lookup"): + setattr(core, name, getattr(cls, name).__get__(core)) + return core + + +@pytest.mark.parametrize( + "world,failing", + [(2, ()), (2, (1,)), (8, ()), (8, (5,)), (8, (0, 7))], + ids=["w2_ok", "w2_rank1_fails", "w8_ok", "w8_rank5_fails", "w8_ranks0_7_fail"], +) +def test_engine_build_agreed_across_ranks(monkeypatch, world, failing): + comm = _patch_comm(monkeypatch, world) + cores = [_core(rank in failing, world) for rank in range(world)] + _run_ranks([lambda c=c: c._prebuild_engines(3) for c in cores]) + keys = sorted(cores[0].moe_engines[3]) + assert keys and comm.calls == world * len(keys) + assert not any(hasattr(c, "moe16_engines") for c in cores) + for c in cores[1:]: + assert c.moe_engines[3].keys() == cores[0].moe_engines[3].keys() + for key in keys: + for c in cores: + assert (c.moe_engines[3][key] is None) == bool(failing), key + # Every rank then takes the same path in the forward: the engine, or (after a miss) the generic MoE. + modeling.set_require(False) + for n in sorted(modeling.MOE_ENGINE.keys): + got = [c.engine(3, n, shared=True) is None for c in cores] + assert len(set(got)) == 1, (n, got) + + +@pytest.mark.parametrize("capturing", [False, True]) +def test_fc_join_records_output_on_main(monkeypatch, capturing): + M3Eagle3Worker = modeling.M3Eagle3Worker + main, fork = mock.Mock(name="main"), mock.Mock(name="fork") + hidden = mock.Mock(name="hidden", is_cuda=True) + monkeypatch.setattr(torch.cuda, "current_stream", lambda *a: main) + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: capturing) + worker = SimpleNamespace(_m3_fc_hidden=hidden, _m3_fc_flagged=False) + M3Eagle3Worker._m3_fc_join(worker, fork) + main.wait_stream.assert_called_once_with(fork) + if capturing: + hidden.record_stream.assert_not_called() + else: + hidden.record_stream.assert_called_once_with(main) + worker._m3_fc_hidden = None + M3Eagle3Worker._m3_fc_join(worker, fork) diff --git a/tests/unittest/_torch/modeling_v2/test_m31_pins.py b/tests/unittest/_torch/modeling_v2/test_m31_pins.py new file mode 100644 index 000000000000..6a60ec43f43c --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/test_m31_pins.py @@ -0,0 +1,260 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""The MiniMax-M3.1 bundled tactic pins: what a name-keyed entry serves on its first AutoTuner lookup.""" + +import json + +import pytest +import torch + +from tensorrt_llm import _bootstrap +from tensorrt_llm._torch import flashinfer_autotune_cache as fac +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 import m31_pins +from tensorrt_llm._torch.autotuner import AutoTuner, TunableRunner, TuningConfig + +pytestmark = pytest.mark.cpu_only + +UNIFIED_CACHE_ENV = ( + "TRTLLM_CACHE_DIR" # tensorrt_llm defaults the AutoTuner cache path from it at import +) +OP = "m31_pins_test::gemm" + + +class _NamedRunner(TunableRunner): + """Tactics are indices into ``names``; a rebuild may reorder that list.""" + + def __init__(self, names): + self.names = list(names) + + def get_valid_tactics(self, inputs, profile, **kwargs): + return list(range(len(self.names))) + + def forward(self, inputs, tactic=-1, **kwargs): + return inputs[0] + + def tactic_name(self, tactic): + return "fallback" if tactic == -1 else self.names[tactic] + + def unique_id(self): + return ("bf16",) + + +@pytest.fixture +def tuner(monkeypatch): + t = AutoTuner.get() + t.clear_cache() + monkeypatch.setattr(t, "bundle_reporter", None) + monkeypatch.setattr(t, "is_tuning_mode", False) + for env in ( + m31_pins.DISABLE_ENV, + m31_pins.DIR_ENV, + m31_pins.AUTOTUNER_CACHE_ENV, + UNIFIED_CACHE_ENV, + ): + monkeypatch.delenv(env, raising=False) + # atexit is one module for the whole process: the stub must take whatever other callers pass. + monkeypatch.setattr(m31_pins.atexit, "register", lambda *args, **kwargs: None) + yield t + t.clear_cache() + + +def _inputs(): + return [torch.empty(4, 8), torch.empty(8, 16)] + + +def _write_bundle( + tmp_path, monkeypatch, tuner, runner, tactic, name, flashinfer=None, native_guard=None +): + """A tp2 rank-0 bundle holding one entry for ``runner`` at ``_inputs()``'s shapes.""" + config = TuningConfig() + key = tuner.profiling_cache.get_cache_key( + OP, runner, tuple(tuner._get_input_sizes(_inputs())), config + ) + bundle = { + "format": m31_pins.FORMAT, + "autotuner": [ + { + "op": key[0], + "runner": key[1], + "runner_key": key[2], + "shapes": [list(dims) for dims in key[3]], + "runner_id": 0, + "tactic": tactic, + "tactic_name": name, + "min_time": 0.01, + } + ], + "flashinfer": flashinfer, + } + if native_guard is not None: + bundle["native_guard"] = native_guard + (tmp_path / "tp2").mkdir() + (tmp_path / "tp2" / "rank_0.json").write_text(json.dumps(bundle)) + monkeypatch.setenv(m31_pins.DIR_ENV, str(tmp_path)) + return key, config + + +def test_entry_whose_tactic_keeps_its_name_is_served(tuner, tmp_path, monkeypatch): + runner = _NamedRunner(["a", "b", "c"]) + key, config = _write_bundle(tmp_path, monkeypatch, tuner, runner, 1, "b") + reporter = m31_pins.install("tp2", 0, tuner) + _, tactic = tuner.choose_one(OP, [runner], config, _inputs()) + assert tactic == 1 + assert reporter.unresolved_count == 0 + assert key not in tuner.profiling_cache.bundled_names + + +def test_reordered_build_serves_the_tactic_of_the_same_name(tuner, tmp_path, monkeypatch): + runner = _NamedRunner(["c", "a", "b"]) + key, config = _write_bundle(tmp_path, monkeypatch, tuner, runner, 1, "b") + reporter = m31_pins.install("tp2", 0, tuner) + _, tactic = tuner.choose_one(OP, [runner], config, _inputs()) + assert tactic == 2 + assert tuner.profiling_cache.cache[key][1] == 2 + assert reporter.unresolved_count == 0 + + +def test_renamed_tactic_is_unresolved_and_dropped(tuner, tmp_path, monkeypatch): + runner = _NamedRunner(["a", "b2", "c"]) + key, config = _write_bundle(tmp_path, monkeypatch, tuner, runner, 1, "b") + reporter = m31_pins.install("tp2", 0, tuner) + _, tactic = tuner.choose_one(OP, [runner], config, _inputs()) + assert tactic == -1 + assert key not in tuner.profiling_cache.cache + assert reporter.unresolved_count == 1 + + +def test_kill_switch_loads_nothing(tuner, tmp_path, monkeypatch): + _write_bundle(tmp_path, monkeypatch, tuner, _NamedRunner(["a", "b"]), 1, "b") + monkeypatch.setenv(m31_pins.DISABLE_ENV, "1") + assert m31_pins.install("tp2", 0, tuner) is None + assert not tuner.profiling_cache.cache + + +def test_explicit_autotuner_cache_takes_precedence(tuner, tmp_path, monkeypatch): + _write_bundle(tmp_path, monkeypatch, tuner, _NamedRunner(["a", "b"]), 1, "b") + monkeypatch.setenv(m31_pins.AUTOTUNER_CACHE_ENV, str(tmp_path / "cache.json")) + assert m31_pins.install("tp2", 0, tuner) is None + assert not tuner.profiling_cache.cache + + +@pytest.mark.parametrize( + "unified,cache,served", + [ + (False, None, True), # no AutoTuner cache: the bundle is served + (False, "explicit", False), # an explicit cache wins + (True, "default", True), # TRTLLM_CACHE_DIR's default path is not an explicit cache + (True, "explicit", False), # an explicit cache wins under TRTLLM_CACHE_DIR too + ], +) +def test_bundle_vs_autotuner_cache_path(tuner, tmp_path, monkeypatch, unified, cache, served): + runner = _NamedRunner(["a", "b"]) + _, config = _write_bundle(tmp_path, monkeypatch, tuner, runner, 1, "b") + root = str(tmp_path / "trtllm_cache") + if unified: + monkeypatch.setenv(UNIFIED_CACHE_ENV, root) + if cache == "default": + # The value tensorrt_llm sets at import from TRTLLM_CACHE_DIR. + path = _bootstrap._unified_cache_defaults(root)[m31_pins.AUTOTUNER_CACHE_ENV] + monkeypatch.setenv(m31_pins.AUTOTUNER_CACHE_ENV, path) + elif cache == "explicit": + monkeypatch.setenv(m31_pins.AUTOTUNER_CACHE_ENV, str(tmp_path / "elsewhere.json")) + assert (m31_pins.install("tp2", 0, tuner) is not None) is served + if served: + _, tactic = tuner.choose_one(OP, [runner], config, _inputs()) + assert tactic == 1 + else: + assert not tuner.profiling_cache.cache + + +def test_missing_bundle_loads_nothing(tuner, tmp_path, monkeypatch): + monkeypatch.setenv(m31_pins.DIR_ENV, str(tmp_path)) + assert m31_pins.install("tp8", 3, tuner) is None + + +def test_flashinfer_section_needs_its_guard(tuner, tmp_path, monkeypatch): + monkeypatch.setattr(fac, "_STATE", {"loaded": False, "logged": False, "bundled": None}) + have = {"flashinfer-python": "0.6.18", "nvidia-cutlass-dsl": "4.8.0"} + monkeypatch.setattr(m31_pins, "_flashinfer_versions", lambda: have) + fi_cache = {"_metadata": {}, "('mxfp8_gemm', (32, 4096))": ["CutlassMxfp8GemmRunner", 6]} + _write_bundle( + tmp_path, + monkeypatch, + tuner, + _NamedRunner(["a"]), + 0, + "a", + flashinfer={"guard": dict(have, **{"flashinfer-python": "0.6.17"}), "cache": fi_cache}, + ) + reporter = m31_pins.install("tp2", 0, tuner) + assert reporter.unresolved_count == 1 + assert fac._STATE["bundled"] is None + + +def test_flashinfer_section_with_matching_guard_is_served_load_only(tuner, tmp_path, monkeypatch): + monkeypatch.setattr(fac, "_STATE", {"loaded": False, "logged": False, "bundled": None}) + monkeypatch.delenv(fac.CACHE_DIR_ENV, raising=False) + have = {"flashinfer-python": "0.6.18", "nvidia-cutlass-dsl": "4.8.0"} + monkeypatch.setattr(m31_pins, "_flashinfer_versions", lambda: have) + fi_cache = {"_metadata": {}, "('mxfp8_gemm', (32, 4096))": ["CutlassMxfp8GemmRunner", 6]} + _write_bundle( + tmp_path, + monkeypatch, + tuner, + _NamedRunner(["a"]), + 0, + "a", + flashinfer={"guard": have, "cache": fi_cache}, + ) + reporter = m31_pins.install("tp2", 0, tuner) + assert reporter.unresolved_count == 0 + path = fac.cache_file() + assert path is not None and json.loads(open(path).read()) == fi_cache + assert fac.frozen() + + +GUARD = {"libtensorrt_llm.so": "aa", "libth_common.so": "bb"} + + +def test_converter_keeps_an_unnamed_runner_under_the_native_guard(tmp_path, monkeypatch): + monkeypatch.setattr(m31_pins, "_native_guard", lambda: dict(GUARD)) + cache = { + "shared": {}, + "rank_0": {"('op', 'SomeRunner', '()', ((1, 2),))": {"runner_id": 0, "tactic": "3"}}, + } + path = tmp_path / "cache.json" + path.write_text(json.dumps(cache)) + (written,) = m31_pins.convert(path, None, tmp_path / "out", 1) + bundle = json.loads(written.read_text()) + assert bundle["autotuner"][0]["tactic_name"] is None and bundle["autotuner"][0]["tactic"] == 3 + assert bundle["native_guard"] == GUARD + + +@pytest.mark.parametrize("same_build", [True, False]) +def test_unnamed_entry_is_served_only_on_the_same_native_build( + tuner, tmp_path, monkeypatch, same_build +): + monkeypatch.setattr( + m31_pins, + "_native_guard", + lambda: dict(GUARD) if same_build else {"libtensorrt_llm.so": "cc"}, + ) + runner = _NamedRunner(["a", "b"]) + key, config = _write_bundle(tmp_path, monkeypatch, tuner, runner, 1, None, native_guard=GUARD) + reporter = m31_pins.install("tp2", 0, tuner) + _, tactic = tuner.choose_one(OP, [runner], config, _inputs()) + assert tactic == (1 if same_build else -1) + assert reporter.unresolved_count == (0 if same_build else 1) + assert key not in tuner.profiling_cache.bundled_names diff --git a/tests/unittest/_torch/modeling_v2/test_m31_tp4_routing_source.py b/tests/unittest/_torch/modeling_v2/test_m31_tp4_routing_source.py index 6ac180b441f6..910b90dd6dff 100644 --- a/tests/unittest/_torch/modeling_v2/test_m31_tp4_routing_source.py +++ b/tests/unittest/_torch/modeling_v2/test_m31_tp4_routing_source.py @@ -82,7 +82,8 @@ def inputs(): class TestM31Tp4RoutingSource(unittest.TestCase): def test_declines_outside_scope(self): - cases = [("chip", (10, 0)), ("chip", (10, 7))] + # SM107 (10, 7) is in scope: it routes to the sm_107 targets (their own routing tests). + cases = [("chip", (10, 0)), ("chip", (12, 0))] cases += [ (name, value) for name, value in ( @@ -226,12 +227,14 @@ def load_file(name, path, package=False): # A route rejected after the parent import can still choose the built-in model. config, mapping = inputs() with self.assertRaises(NoTarget): - route()(config, mapping, (10, 7), None, Quant.NVFP4) + route()(config, mapping, (12, 0), None, Quant.NVFP4) load_file(catalog_name, SOURCE / "catalog/fakes/m31_fake_ops.py") fake_module = load_file( family_name + ".fake_ops", SOURCE / "models/minimax_m3/fake_ops.py" ) - self.assertEqual(len(fake_module.FAKES), 7) + # The sm_103 entries plus the sm_107 targets' (producer_pw and mc_allreduce_norm per TP width, and the + # four dense qk-norm-rope ops). + self.assertEqual(len(fake_module.FAKES), 7 + 3 * 2 + 4) names = list(fake_module.FAKES) for name in names: setattr(native, name, object()) @@ -239,9 +242,9 @@ def load_file(name, path, package=False): previous_fake = object() installed["trtllm::" + names[1]] = previous_fake fake_module.register() - self.assertEqual(len(calls), 5) + self.assertEqual(len(calls), len(names) - 2) fake_module.register() - self.assertEqual(len(calls), 5) + self.assertEqual(len(calls), len(names) - 2) self.assertIs(installed["trtllm::" + names[1]], previous_fake) self.assertEqual(installed["trtllm::" + names[0]], "Meta") diff --git a/tests/unittest/_torch/modeling_v2/test_m31_tp8_routing_source.py b/tests/unittest/_torch/modeling_v2/test_m31_tp8_routing_source.py index 57ca48949126..10e6a27559c5 100644 --- a/tests/unittest/_torch/modeling_v2/test_m31_tp8_routing_source.py +++ b/tests/unittest/_torch/modeling_v2/test_m31_tp8_routing_source.py @@ -82,7 +82,8 @@ def inputs(): class TestM31Tp8RoutingSource(unittest.TestCase): def test_declines_outside_scope(self): - cases = [("chip", (10, 0)), ("chip", (10, 7))] + # SM107 (10, 7) is in scope: it routes to the sm_107 targets (their own routing tests). + cases = [("chip", (10, 0)), ("chip", (12, 0))] cases += [ (name, value) for name, value in ( @@ -230,11 +231,13 @@ def load_file(name, path, package=False): fake_module = load_file( family_name + ".fake_ops", SOURCE / "models/minimax_m3/fake_ops.py" ) - self.assertEqual(len(fake_module.FAKES), 7) + # The sm_103 entries plus the sm_107 targets' (producer_pw and mc_allreduce_norm per TP width, and the + # four dense qk-norm-rope ops). + self.assertEqual(len(fake_module.FAKES), 7 + 3 * 2 + 4) # A route rejected after the parent import can still choose the built-in model. config, mapping = inputs() with self.assertRaises(NoTarget): - route()(config, mapping, (10, 7), None, Quant.NVFP4) + route()(config, mapping, (12, 0), None, Quant.NVFP4) names = list(fake_module.FAKES) for name in names: setattr(native, name, object()) @@ -242,9 +245,9 @@ def load_file(name, path, package=False): previous_fake = object() installed["trtllm::" + names[1]] = previous_fake fake_module.register() - self.assertEqual(len(calls), 5) + self.assertEqual(len(calls), len(names) - 2) fake_module.register() - self.assertEqual(len(calls), 5) + self.assertEqual(len(calls), len(names) - 2) self.assertIs(installed["trtllm::" + names[1]], previous_fake) self.assertEqual(installed["trtllm::" + names[0]], "Meta") diff --git a/tests/unittest/_torch/multi_gpu/test_minimax_m31_f3_ar_prepushed.py b/tests/unittest/_torch/multi_gpu/test_minimax_m31_f3_ar_prepushed.py new file mode 100644 index 000000000000..e7e2f2537fae --- /dev/null +++ b/tests/unittest/_torch/multi_gpu/test_minimax_m31_f3_ar_prepushed.py @@ -0,0 +1,181 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""trtllm::minimax_m31_f3_ar_prepushed (MiniMax-M3.1 TP2 F3 boundary) is bit for bit the stock one-shot Lamport +AllReduce(RESIDUAL_RMS_NORM) (+ mxfp8_quantize of the norm) when the producer has pre-pushed its rows. + +The serving producer is the o_proj GEMM's epilogue fork (kernels/m31_f3_gemm.py). Here a test producer does exactly +the stock push loop: the rank's allreduce input, with every 32-bit word 0x80000000 (-0, the Lamport "empty" marker) +replaced by +0, stored into this rank's section of every rank's data slot flag % 3. Two ranks; random, -0-word, zero-row +and signed-zero inputs; real-size and padded MXFP8 scale tiles. +""" + +import pickle +import sys +import traceback + +import cloudpickle +import pytest +import torch +from mpi4py import MPI + +import tensorrt_llm +import tensorrt_llm._torch.custom_ops # noqa: F401 +from tensorrt_llm._torch.distributed import ( + AllReduce, + AllReduceFusionOp, + AllReduceParams, + AllReduceStrategy, +) +from tensorrt_llm._torch.distributed.ops import get_allreduce_workspace +from tensorrt_llm.mapping import Mapping + +cloudpickle.register_pickle_by_value(sys.modules[__name__]) +MPI.pickle.__init__(cloudpickle.dumps, cloudpickle.loads, pickle.HIGHEST_PROTOCOL) + +# needed since we reuse the mpi executor pool, first test running will leak a thread +pytestmark = pytest.mark.threadleak(enabled=False) + +H = 6144 +EPS = 1e-5 +NR = 2 + + +class _DevBytes: + """A uint8 view of raw device memory (the Lamport slots of this rank and its peer).""" + + def __init__(self, ptr: int, nbytes: int): + self.__cuda_array_interface__ = { + "shape": (nbytes,), + "typestr": "|u1", + "data": (ptr, False), + "version": 3, + } + + +def _inputs(rows: int, kind: str, rank: int, seed: int): + g = torch.Generator(device="cuda").manual_seed(seed) + x = torch.randn(rows, H, device="cuda", generator=g) * 2.0 + if rank == 1: + x = x + torch.randn( + rows, H, device="cuda", generator=torch.Generator(device="cuda").manual_seed(seed + 7) + ) + x = x.to(torch.bfloat16) + res = ( + torch.randn( + rows, H, device="cuda", generator=torch.Generator(device="cuda").manual_seed(seed + 1) + ) + * 4 + ).to(torch.bfloat16) + if kind == "neg0_words": + # words whose high bf16 is -0 and low is +0: the push must store them as +0 + w = x.view(torch.int32) + w[:, ::5] = torch.tensor(-(2**31), dtype=torch.int32, device="cuda") + elif kind == "zero_rows": + # both ranks contribute zero rows and the residual is +0 / -0: all-zero normed rows (MXFP8 scale 0) + x[: rows // 3] = 0 + res[: rows // 4] = 0.0 + res[rows // 4 : rows // 3] = -0.0 + elif kind == "signed_zeros": + x = torch.where(x > 0, torch.zeros_like(x), -torch.zeros_like(x)) + res[: rows // 2] = -0.0 + return x.contiguous(), res.contiguous() + + +def _run(rows_list, kinds): + rank = tensorrt_llm.mpi_rank() + torch.cuda.set_device(rank) + comm = MPI.COMM_WORLD + mapping = Mapping(world_size=NR, rank=rank, gpus_per_node=NR, tp_size=NR) + # The reference is pinned to the one-shot Lamport kernel, the kernel F3 replaces (F3 declines a + # boundary that is not it). AUTO is not a stable reference here: with the AllReduce autotuner on + # it can pick NCCL, which keeps signed zeros where the Lamport push stores -0 words as +0 + # (serving runs AR_DET: autotune off). + ar = AllReduce(mapping=mapping, strategy=AllReduceStrategy.ONESHOT, dtype=torch.bfloat16) + ws = get_allreduce_workspace(mapping) + wsp = [int(v) for v in ws.cpu().tolist()] + # getattr keeps "ops" out of this function's code names: cloudpickle would otherwise pickle the torch.ops module + # object (a ModuleType subclass it cannot reduce) along with the function. + trtllm_ops = getattr(torch, "ops").trtllm + read = trtllm_ops.minimax_m31_f3_read_i64 + comm_size = read(wsp[NR * 3 + 1] + 8) + flag_addr = wsp[NR * 3] + 8 + nw = ( + torch.rand(H, device="cuda", generator=torch.Generator(device="cuda").manual_seed(3)) + 0.5 + ).to(torch.bfloat16) + bad = {} + for rows in rows_list: + for kind in kinds: + x, r0 = _inputs(rows, kind, rank, 1000 * rows + len(kind)) + n_s, r_s = ar( + x, + all_reduce_params=AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=r0, + norm_weight=nw, + eps=EPS, + trigger_completion_at_end=False, + ), + )[:2] + q_s, sf_s = trtllm_ops.mxfp8_quantize(n_s, True) + torch.cuda.synchronize() + comm.Barrier() + # the test producer: the stock push loop + slot = (read(flag_addr) & 0xFFFFFFFF) % 3 + words = x.view(torch.int32).clone() + words[words == -(2**31)] = 0 + src = words.view(torch.uint8).reshape(-1) + for dest in range(NR): + base = wsp[2 * NR + dest] + slot * comm_size + rank * src.numel() + # a view on the owning device (the peer's slot is on the peer GPU); as_tensor(device=) would copy it + torch.as_tensor(_DevBytes(base, src.numel())).copy_(src) + torch.cuda.synchronize() + comm.Barrier() + n_f, r_f = torch.empty_like(r0), torch.empty_like(r0) + q_f = torch.empty(rows, H, dtype=torch.uint8, device="cuda") + sf_f = torch.empty( + (rows + 127) // 128 * 128 * (H // 32), dtype=torch.uint8, device="cuda" + ) + trtllm_ops.minimax_m31_f3_ar_prepushed( + rank, ws, x, r0, r_f, n_f, nw, EPS, False, q_f, sf_f + ) + torch.cuda.synchronize() + + def neq(a, b): + a, b = ( + a.contiguous().view(torch.uint8).reshape(-1), + b.contiguous().view(torch.uint8).reshape(-1), + ) + return -1 if a.numel() != b.numel() else int((a != b).sum()) + + cell = { + "norm": neq(n_f, n_s), + "residual": neq(r_f, r_s), + "mxfp8": neq(q_f, q_s), + "mx_sf": neq(sf_f, sf_s), + } + if any(cell.values()): + bad[f"r{rows}_{kind}"] = cell + comm.Barrier() + return bad + + +def _run_rank(rows_list, kinds): + try: + return _run(rows_list, kinds) + except Exception: + traceback.print_exc() + raise + + +@pytest.mark.skipif(torch.cuda.device_count() < 2, reason="needs 2 GPUs") +@pytest.mark.skipif( + not hasattr(torch.ops.trtllm, "minimax_m31_f3_ar_prepushed"), + reason="the F3 prepushed all-reduce op is not built", +) +@pytest.mark.parametrize("mpi_pool_executor", [2], indirect=True) +def test_f3_ar_prepushed_is_the_stock_oneshot_allreduce(mpi_pool_executor): + rows_list = [1, 49, 104, 128, 176, 224, 256] + kinds = ["randn", "neg0_words", "zero_rows", "signed_zeros"] + results = mpi_pool_executor.map(_run_rank, *zip(*[(rows_list, kinds)] * 2)) + for rank, bad in enumerate(results): + assert not bad, f"rank {rank}: F3 boundary bytes differ from the stock chain: {bad}" diff --git a/tests/unittest/_torch/multi_gpu/test_minimax_m31_s2_moe_tail.py b/tests/unittest/_torch/multi_gpu/test_minimax_m31_s2_moe_tail.py new file mode 100644 index 000000000000..e43adf2c3c54 --- /dev/null +++ b/tests/unittest/_torch/multi_gpu/test_minimax_m31_s2_moe_tail.py @@ -0,0 +1,166 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""trtllm::minimax_m31_s2_moe_tail (TP2W_S2_TAIL) on 2 ranks, bit for bit the chain it fuses: +m3_moe_finalize -> AllReduce(ONESHOT, RESIDUAL_RMS_NORM) -> mxfp8_quantize(norm_out, swizzled). Every output is +compared: norm_out, residual_out, the MXFP8 bytes, the valid rows' scales; the pad rows' scales are 0.""" + +import pickle +import sys +import traceback + +import cloudpickle +import pytest +import torch +from mpi4py import MPI + +import tensorrt_llm +import tensorrt_llm._torch.custom_ops # noqa: F401 +from tensorrt_llm._torch.distributed import AllReduce, AllReduceFusionOp, AllReduceParams +from tensorrt_llm.functional import AllReduceStrategy +from tensorrt_llm.mapping import Mapping + +cloudpickle.register_pickle_by_value(sys.modules[__name__]) +MPI.pickle.__init__(cloudpickle.dumps, cloudpickle.loads, pickle.HIGHEST_PROTOCOL) + +# needed since we reuse the mpi executor pool, first test running will leak a thread +pytestmark = pytest.mark.threadleak(enabled=False) + +HIDDEN, TOPK, EPS = 6144, 4, 1e-6 +ROWS = (1, 8, 48, 64, 88, 112, 128, 136, 176, 224) + + +def _sf_valid_mask(rows, dev): + cols = HIDDEN // 32 + padded = (rows + 127) // 128 * 128 + m = torch.zeros(padded * cols, dtype=torch.bool, device=dev) + r = torch.arange(padded, device=dev).view(-1, 1).expand(padded, cols) + c = torch.arange(cols, device=dev).view(1, -1).expand(padded, cols) + off = ( + (r // 128) * (cols // 4 * 512) + + (c // 4) * 512 + + (r % 32) * 16 + + ((r % 128) // 32) * 4 + + (c % 4) + ) + m[off.reshape(-1)] = (r < rows).reshape(-1) + return m + + +def _run_rank(world): + rank = tensorrt_llm.mpi_rank() + torch.cuda.set_device(rank) + dev = torch.device("cuda", rank) + try: + from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3.minimax_m31_nvfp4__sm_107__tp2.kernels.m3_moe_finalize import ( # noqa: E501 + m3_moe_finalize, + ) + + # getattr keeps "ops" out of this function's code names: cloudpickle would otherwise pickle the torch.ops + # module object (a ModuleType subclass it cannot reduce) along with the function. + trtllm_ops = getattr(torch, "ops").trtllm + op = trtllm_ops.minimax_m31_s2_moe_tail + mapping = Mapping(world_size=world, rank=rank, gpus_per_node=world, tp_size=world) + ar = AllReduce(mapping=mapping, strategy=AllReduceStrategy.ONESHOT, dtype=torch.bfloat16) + g0 = torch.Generator(device=dev).manual_seed(7) + w_norm = (1.0 + 0.1 * torch.randn(HIDDEN, generator=g0, device=dev)).to(torch.bfloat16) + w_norm[::97] = -w_norm[::97] + bad = [] + for rows in ROWS: + for sdt in (torch.bfloat16, torch.float32): + for seed in (0, 1, 2): + g = torch.Generator(device=dev).manual_seed(1000 * seed + 17 * rank + rows) + n_perm = rows * TOPK + fc2 = ( + torch.randn(((n_perm + 127) // 128 * 128, HIDDEN), generator=g, device=dev) + * 0.5 + ) + if seed == 2: # 2^+-12 per-row stress + fc2 = fc2 * torch.pow( + 2.0, + torch.randint( + -12, 13, (fc2.shape[0], 1), generator=g, device=dev + ).float(), + ) + fc2 = fc2.to(torch.bfloat16) + perm = ( + torch.randperm(n_perm, generator=g, device=dev) + .to(torch.int32) + .view(rows, TOPK) + ) + if seed == 1: # dropped slots + drop = torch.rand((rows, TOPK), generator=g, device=dev) < 0.02 + perm = torch.where(drop, torch.full_like(perm, -1), perm) + perm = perm.contiguous() + scales = torch.softmax( + torch.randn((rows, TOPK), generator=g, device=dev), -1 + ).to(sdt) + shared = (torch.randn((rows, HIDDEN), generator=g, device=dev) * 0.5).to( + torch.bfloat16 + ) + residual = (torch.randn((rows, HIDDEN), generator=g, device=dev) * 2).to( + torch.bfloat16 + ) + o = m3_moe_finalize(fc2, scales.contiguous(), perm, shared) + n, r = ar( + o, + all_reduce_params=AllReduceParams( + fusion_op=AllReduceFusionOp.RESIDUAL_RMS_NORM, + residual=residual, + norm_weight=w_norm, + eps=EPS, + trigger_completion_at_end=False, + ), + ) + q, sf = trtllm_ops.mxfp8_quantize(n, True) + b = op( + fc2, + scales.contiguous(), + perm, + shared, + residual, + w_norm, + ar.workspace, + rank, + world, + EPS, + False, + ) + torch.cuda.synchronize() + valid = _sf_valid_mask(rows, dev) + diffs = ( + int((n.view(torch.int16) != b[0].view(torch.int16)).sum()), + int((r.view(torch.int16) != b[1].view(torch.int16)).sum()), + int((q.view(torch.uint8) != b[2].view(torch.uint8)).sum()), + int((sf[valid] != b[3][valid]).sum()), + int((b[3][~valid] != 0).sum()), + ) + if any(diffs): + bad.append((rows, str(sdt), seed, diffs)) + assert not bad, bad + except Exception: + traceback.print_exc() + raise + return True + + +@pytest.mark.skipif( + torch.cuda.device_count() < 2 or torch.cuda.get_device_capability()[0] < 10, + reason="needs 2 GPUs of SM100 or newer", +) +@pytest.mark.parametrize("mpi_pool_executor", [2], indirect=True) +def test_s2_moe_tail_is_the_fused_chain_bit_for_bit(mpi_pool_executor): + world = mpi_pool_executor.num_workers + for r in mpi_pool_executor.map(_run_rank, *zip(*[(world,)] * world)): + assert r is True diff --git a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_context_onehot.py b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_context_onehot.py new file mode 100644 index 000000000000..18854a900285 --- /dev/null +++ b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_context_onehot.py @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""DFlash block worker: a mixed step's context one-hot draft-prob rows span the width acceptance reads. + +With a vocab-sharded drafter the draft logits are a shard of the vocab, but the gen scatter gathers them to the full +vocab and publishes that width (``draft_probs_last_dim``). The new requests' one-hot rows must cover the same width, +or the columns past the shard keep the previous slot occupant's probabilities and the request's first rejection step +reads a corrupt draft distribution.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.speculative.dflash import _context_onehot_vocab +from tensorrt_llm._torch.speculative.interface import SpecWorkerBase + +pytestmark = pytest.mark.cpu_only + +SHARD, FULL, K, SLOTS = 4, 8, 3, 4 + + +def _metadata(published): + # Every slot row holds a stale distribution from an earlier occupant. + return SimpleNamespace( + use_rejection_sampling=True, + draft_probs=torch.full((SLOTS, K, FULL), 0.5), + # One context request (slot 2), then one generation request (slot 0). + batch_slot_ids=torch.tensor([2, 0]), + draft_probs_last_dim=published, + draft_probs_vocab_size=FULL, + ) + + +def test_width_is_the_published_one(): + assert _context_onehot_vocab(_metadata(FULL), num_gens=1, draft_logits_vocab=SHARD) == FULL + + +def test_width_falls_back_to_the_logits_before_any_scatter(): + assert _context_onehot_vocab(_metadata(0), num_gens=1, draft_logits_vocab=SHARD) == SHARD + + +def test_no_generation_requests_leaves_the_width_to_the_writer(): + assert _context_onehot_vocab(_metadata(FULL), num_gens=0, draft_logits_vocab=SHARD) is None + + +@pytest.mark.parametrize("index_fill", [True, False]) +def test_new_request_rows_are_a_full_width_one_hot(monkeypatch, index_fill): + from tensorrt_llm._torch.speculative import interface + + monkeypatch.setattr(interface, "_ONEHOT_INDEX_FILL", index_fill) + md = _metadata(FULL) + width = _context_onehot_vocab(md, num_gens=1, draft_logits_vocab=SHARD) + SpecWorkerBase.write_context_onehot_draft_probs(None, md, 1, 1, K, width) + row = md.draft_probs[2, :K, : md.draft_probs_last_dim] + expected = torch.zeros(K, FULL) + expected[:, 0] = 1.0 + # Every column acceptance reads, not just the shard; the generation request's row is untouched. + assert torch.equal(row, expected) + assert torch.equal(md.draft_probs[0], torch.full((K, FULL), 0.5)) diff --git a/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_disagg_draft_ctx.py b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_disagg_draft_ctx.py new file mode 100644 index 000000000000..1178cb760254 --- /dev/null +++ b/tests/unittest/_torch/speculative/hw_agnostic/test_dflash_disagg_draft_ctx.py @@ -0,0 +1,322 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""DFlash worker state for the disagg drafter context-window transfer (F0a). + +Context side: ``_forward_impl`` records one event right after ``_store_prefill_context`` when the +batch has context requests and the stream is not capturing, and each context request keeps it +with its slot's host ``(ctx_len, ctx_base)``. Generation side: received lengths are seeded by +``DFlashSpecMetadata.prepare`` when the request first holds a real slot, never on the dummy slot. +""" + +from collections import deque +from functools import partial +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest +import torch + +from tensorrt_llm._torch.speculative import dflash as dflash_module +from tensorrt_llm._torch.speculative.dflash import DFlashSpecMetadata, DFlashWorker + +pytestmark = pytest.mark.cpu_only + +FLOOR = 1 << 40 # request ids from here on are CUDA-graph padding dummies + + +class _StopAfterStore(Exception): + """Ends the forward right after the context store; drafting is not under test.""" + + +class _Event: + recorded = [] + + def __init__(self): + self.record_calls = 0 + + def record(self, stream=None): + self.record_calls += 1 + _Event.recorded.append(self) + + +@pytest.fixture +def events(monkeypatch): + _Event.recorded = [] + monkeypatch.setattr(torch.cuda, "Event", _Event) + return _Event.recorded + + +def _ctx_worker(engaged=True, pool_bound=True): + worker = DFlashWorker.__new__(DFlashWorker) + worker._ctx_buf_inited = False # skips the batch-to-slot rebuild after the store + worker._ctx_block_tables = torch.zeros(2, 4, dtype=torch.int32) if pool_bound else None + worker._ctx_len = torch.zeros(3, dtype=torch.long) + worker._ctx_base_pages = torch.zeros(3, dtype=torch.long) + worker._ctx_len_host = [0, 0, 0] + worker._ctx_base_host = [0, 0, 0] + worker._req_ctx_pos = {} + worker._req_to_slot = {} + worker._accept_stats = None + worker.get_draft_kv_cache_manager = Mock(return_value=None) + worker._lazy_init_ctx_buffers = Mock() + worker._refresh_ctx_block_tables = Mock(return_value=True) + worker._execute_guided_decoder_if_present = Mock() + worker.sample_and_accept_draft_tokens = Mock(return_value=(None, None)) + worker._prepare_attn_metadata_for_dflash = Mock() + worker._prepare_kv_for_draft_forward = Mock() + worker.prepare_1st_drafter_inputs = Mock(side_effect=_StopAfterStore) + + def store(draft_model, spec_metadata, attn_metadata, position_ids, total_target_tokens): + # What the real store leaves on the host: slots and their stored lengths. + for slot, rid in enumerate(spec_metadata.request_ids[: attn_metadata.num_contexts]): + worker._req_to_slot[rid] = slot + worker._ctx_len_host[slot] = 100 + slot + worker._ctx_base_host[slot] = slot + + worker._store_prefill_context = Mock(side_effect=store) + if engaged: + worker.engage_disagg_draft_kv() + return worker + + +def _forward(worker, request_ids, num_contexts): + num_tokens = 8 + attn_metadata = SimpleNamespace( + num_seqs=len(request_ids), num_contexts=num_contexts, kv_cache_manager=None + ) + spec_metadata = SimpleNamespace( + runtime_draft_len=3, is_cuda_graph=False, request_ids=list(request_ids) + ) + with pytest.raises(_StopAfterStore): + worker._forward_impl( + torch.zeros(num_tokens, dtype=torch.int32), + torch.arange(num_tokens), + None, + torch.zeros(num_tokens, 4), + attn_metadata, + spec_metadata, + SimpleNamespace(), + ) + + +def test_event_is_recorded_after_the_context_store(monkeypatch, events): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + worker = _ctx_worker() + order = [] + store = worker._store_prefill_context.side_effect + + def ordered_store(*args): + order.append("store") + store(*args) + + def ordered_record(event, stream=None): + order.append("record") + events.append(event) + + worker._store_prefill_context.side_effect = ordered_store + monkeypatch.setattr(_Event, "record", ordered_record) + + _forward(worker, [11, 12, 13], num_contexts=2) + + assert order == ["store", "record"] + assert len(events) == 1 + ready = worker.take_disagg_draft_ctx_ready(11) + assert ready == (events[0], 100, 0) + assert worker.take_disagg_draft_ctx_ready(12) == (events[0], 101, 1) + # The generation request in the batch gets nothing; a taken entry is gone. + assert worker.take_disagg_draft_ctx_ready(13) is None + assert worker.take_disagg_draft_ctx_ready(11) is None + + +def test_no_event_without_context_requests(monkeypatch, events): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + worker = _ctx_worker() + _forward(worker, [21, 22], num_contexts=0) + worker._store_prefill_context.assert_not_called() + assert events == [] + assert worker._disagg_draft_ctx_ready == {} + + +def test_no_event_while_capturing(monkeypatch, events): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: True) + worker = _ctx_worker() + _forward(worker, [31, 32], num_contexts=1) + worker._store_prefill_context.assert_called_once() + assert events == [] + assert worker._disagg_draft_ctx_ready == {} + + +def test_no_event_unless_engaged(monkeypatch, events): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + worker = _ctx_worker(engaged=False) + _forward(worker, [41], num_contexts=1) + worker._store_prefill_context.assert_called_once() + assert events == [] + assert worker.take_disagg_draft_ctx_ready(41) is None + + +def test_private_arena_is_refused(monkeypatch, events): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + worker = _ctx_worker(pool_bound=False) + with pytest.raises(RuntimeError, match="managed draft KV pool"): + worker._record_disagg_draft_ctx_ready(SimpleNamespace(request_ids=[51]), 1) + assert events == [] + + +def test_latest_chunk_store_wins(monkeypatch, events): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + worker = _ctx_worker() + _forward(worker, [61], num_contexts=1) + worker._ctx_len_host[0] = 4096 # the next chunk stored more of the prompt + worker._record_disagg_draft_ctx_ready(SimpleNamespace(request_ids=[61]), 1) + assert len(events) == 2 + assert worker.take_disagg_draft_ctx_ready(61) == (events[1], 4096, 0) + + +def test_event_map_is_bounded(monkeypatch, events): + monkeypatch.setattr(torch.cuda, "is_current_stream_capturing", lambda: False) + monkeypatch.setattr(dflash_module, "_DISAGG_DRAFT_CTX_MAP_CAP", 2) + worker = _ctx_worker() + for rid in (71, 72, 73): + worker._record_disagg_draft_ctx_ready(SimpleNamespace(request_ids=[rid]), 1) + assert list(worker._disagg_draft_ctx_ready) == [72, 73] + + +# --------------------------------------------------------------------------- +# Generation side: seeding at first sight in prepare() +# --------------------------------------------------------------------------- + + +def _gen_worker(num_slots=2, engaged=True): + worker = SimpleNamespace( + _ctx_buf_inited=True, + _req_to_slot={}, + _req_ctx_pos={}, + _free_slots=deque(range(num_slots)), + _graph_dummy_id_floor=FLOOR, + _dummy_slot=num_slots, + _ctx_block_tables=torch.zeros(1), + _batch_to_slot=torch.zeros(8, dtype=torch.long), + _disagg_draft_ctx_seeds={} if engaged else None, + writes=[], + ) + worker._write_ctx_len = lambda updates, bases=None: worker.writes.append( + (dict(updates), dict(bases or {})) + ) + worker._assign_slot = partial(DFlashWorker._assign_slot, worker) + worker._check_disagg_draft_ctx_pool_bound = partial( + DFlashWorker._check_disagg_draft_ctx_pool_bound, worker + ) + worker._bound_disagg_draft_ctx_map = DFlashWorker._bound_disagg_draft_ctx_map + worker.seed = partial(DFlashWorker.seed_disagg_draft_ctx, worker) + return worker + + +def _seed_writes(worker): + """The writes that carry base pages: only a seed does (slot clears and evictions do not).""" + return [write for write in worker.writes if write[1]] + + +def _prepare(monkeypatch, worker, request_ids, num_generations): + monkeypatch.setattr(dflash_module, "prefer_pinned", lambda: False) + meta = SimpleNamespace( + request_ids=request_ids, + num_generations=num_generations, + batch_indices_cuda=torch.zeros(8, dtype=torch.int), + _dflash_worker=worker, + ) + DFlashSpecMetadata.prepare(meta) + + +def test_received_lengths_are_seeded_when_the_request_first_holds_a_slot(monkeypatch): + worker = _gen_worker() + worker.seed(7, 9000, 3, 9000) + worker.seed(8, 33, 0, 33) + + _prepare(monkeypatch, worker, [7, FLOOR + 1], num_generations=2) + + slot7 = worker._req_to_slot[7] + # _assign_slot clears the slot, then the seed lands in one batched write. + assert _seed_writes(worker) == [({slot7: 9000}, {slot7: 3})] + assert worker.writes[-1] == ({slot7: 9000}, {slot7: 3}) + assert worker._req_ctx_pos[7] == 9000 + # A request not in the batch keeps its seed for its own first step. + assert worker._disagg_draft_ctx_seeds == {8: (33, 0, 33)} + # The padding dummy never holds a slot, so nothing seeds the dummy slot. + assert all(worker._dummy_slot not in lens for lens, _ in worker.writes) + + _prepare(monkeypatch, worker, [7, 8], num_generations=2) + slot8 = worker._req_to_slot[8] + assert _seed_writes(worker)[1:] == [({slot8: 33}, {slot8: 0})] + assert worker._disagg_draft_ctx_seeds == {} + # Seeds apply once: a later step leaves the slots to the drafter. + _prepare(monkeypatch, worker, [7, 8], num_generations=2) + assert len(_seed_writes(worker)) == 2 + + +def test_seed_waits_for_a_free_slot(monkeypatch): + worker = _gen_worker(num_slots=1) + worker.seed(5, 64, 0, 64) + worker.seed(6, 96, 0, 96) + _prepare(monkeypatch, worker, [5, 6], num_generations=2) + assert 6 not in worker._req_to_slot # shares the dummy slot this step, unseeded + assert worker._disagg_draft_ctx_seeds == {6: (96, 0, 96)} + _prepare(monkeypatch, worker, [6], num_generations=1) # request 5 left, its slot freed + assert worker._req_to_slot[6] == 0 + assert worker.writes[-1] == ({0: 96}, {0: 0}) + + +def test_without_engagement_prepare_writes_no_seed(monkeypatch): + worker = _gen_worker(engaged=False) + _prepare(monkeypatch, worker, [5], num_generations=1) + assert worker._req_to_slot == {5: 0} + assert worker._req_ctx_pos == {} + assert _seed_writes(worker) == [] + + +def test_seeding_a_received_window_requires_the_pool_bound_context(monkeypatch): + worker = _gen_worker() + worker._disagg_draft_kv_engaged = True # the window transfer is engaged + worker._ctx_block_tables = None + worker.seed(5, 64, 0, 64) + with pytest.raises(RuntimeError, match="managed draft KV pool"): + _prepare(monkeypatch, worker, [5], num_generations=1) + + +def test_private_arena_without_the_transfer_skips_the_seed(monkeypatch): + """Gen-init window sizing alone: a private-arena drafter numbers from 0, nothing to rebase.""" + worker = _gen_worker() + worker._ctx_block_tables = None + worker.seed(5, 64, 2, 64) + _prepare(monkeypatch, worker, [5], num_generations=1) + assert _seed_writes(worker) == [] + + +def test_dump_hook_runs_once_right_after_the_seed_lands(monkeypatch): + worker = _gen_worker() + worker._disagg_draft_ctx_seed_hooks = None + seen = [] + worker.seed(7, 4097, 0, 4097, on_seed=lambda: seen.append(list(worker.writes))) + worker.seed(8, 64, 0, 64) # no knob: no hook + + _prepare(monkeypatch, worker, [7, 8], num_generations=2) + + assert len(seen) == 1 + # The batch's seed write (both requests in one) already happened when the hook ran. + slot7, slot8 = worker._req_to_slot[7], worker._req_to_slot[8] + assert seen[0][-1] == ({slot7: 4097, slot8: 64}, {slot7: 0, slot8: 0}) + assert worker._disagg_draft_ctx_seed_hooks == {} + _prepare(monkeypatch, worker, [7, 8], num_generations=2) + assert len(seen) == 1 diff --git a/tests/unittest/_torch/speculative/test_dflash_forward_context_onehot.py b/tests/unittest/_torch/speculative/test_dflash_forward_context_onehot.py new file mode 100644 index 000000000000..42ee51feba18 --- /dev/null +++ b/tests/unittest/_torch/speculative/test_dflash_forward_context_onehot.py @@ -0,0 +1,183 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GPU unit tests for the context one-hot draft-prob rows written by ``DFlashWorker._forward_impl``. + +The worker's own ``_forward_impl`` and ``SpecWorkerBase.write_context_onehot_draft_probs`` run unchanged; every step +around them (KV / block-table bookkeeping, acceptance, the drafter forward and its sampling) is replaced on the +instance by a stand-in, and the metadata are ``types.SimpleNamespace`` objects carrying only the attributes those two +functions read. Covered: a step without generation requests (warmup, every disaggregated context step), a mixed step +with a vocab-sharded drafter (the context rows span the published width, not the shard), and a generation-only step. + +Requires CUDA because the worker allocates its draft-token tensors on ``device='cuda'``. +""" + +import contextlib +import types + +import pytest +import torch + +from tensorrt_llm._torch.speculative import interface +from tensorrt_llm._torch.speculative.dflash import DFlashWorker + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available(), reason="the worker allocates CUDA tensors" +) + +K, SHARD, FULL, SLOTS, HIDDEN = 3, 4, 8, 4, 16 +STALE = 0.5 + + +def _worker(): + """A DFlashWorker whose surrounding steps are stand-ins; ``_forward_impl`` itself is the real one.""" + w = DFlashWorker.__new__(DFlashWorker) + torch.nn.Module.__init__(w) + w._accept_stats = None + w._ctx_buf_inited = False + w._lean_glue = False + w._ctx_block_tables = None + w._compute_block_size = K + 1 + w._d2t = None + w._ctx_len = torch.zeros(SLOTS, dtype=torch.int32) + w._ctx_len_host = [0] * SLOTS + w._ctx_base_pages = torch.zeros(SLOTS, dtype=torch.int32) + w._ctx_base_host = [0] * SLOTS + w._req_ctx_pos = {} + + def noop(*args, **kwargs): + return None + + for name in ( + "_lazy_init_ctx_buffers", + "_refresh_ctx_block_tables", + "_execute_guided_decoder_if_present", + "_prepare_attn_metadata_for_dflash", + "_prepare_kv_for_draft_forward", + "_store_prefill_context", + "_record_disagg_draft_ctx_ready", + "_restore_attn_metadata_from_spec_dec", + "_apply_kv_rewind_after_draft", + "_rollback_guided_decoder_after_verify", + "_restore_ctx_len_device", + "_restore_ctx_len_host", + ): + setattr(w, name, noop) + w.get_draft_kv_cache_manager = lambda resource_manager: None + w.draft_kv_cache_context = lambda attn_metadata, manager: contextlib.nullcontext() + + def accept(logits, attn_metadata, spec_metadata): + n = attn_metadata.num_seqs + return ( + torch.zeros(n, K + 1, dtype=torch.int32, device="cuda"), + torch.ones(n, dtype=torch.int32, device="cuda"), + ) + + w.sample_and_accept_draft_tokens = accept + w.prepare_1st_drafter_inputs = lambda **kwargs: dict.fromkeys( + ( + "noise_embedding", + "query_positions", + "num_ctx_per_req", + "ctx_k_cache", + "ctx_v_cache", + "ctx_cache_batch_idx", + "ctx_kv_cache", + "ctx_page_table", + ) + ) + w._draft_slot_ids = lambda draft_model, num_gens, block_size, k: torch.arange( + num_gens * k, device="cuda" + ) + w._refine_block_logits = lambda draft_model, logits, inputs, spec_metadata: logits + w.sample_draft_tokens = lambda logits, spec_metadata, batch_size, num_contexts: logits.argmax( + -1 + ).to(torch.int32) + w._prepare_next_new_tokens = lambda accepted, drafts, batch_indices, batch_size, num_acc: drafts + return w + + +def _drafter(num_gens): + """A vocab-sharded drafter: its block outputs give SHARD-wide logits.""" + return types.SimpleNamespace( + lm_head=None, + dflash_forward=lambda **kwargs: torch.zeros(num_gens * (K + 1), HIDDEN, device="cuda"), + logits_processor=lambda hidden, lm_head, attn_metadata, flag: torch.zeros( + hidden.shape[0], SHARD, device="cuda" + ), + ) + + +def _metadata(num_contexts, num_gens, published): + attn = types.SimpleNamespace( + num_seqs=num_contexts + num_gens, num_contexts=num_contexts, kv_cache_manager=None + ) + # Context requests sit in slots 2, 3; generation requests in slots 0, 1. Every row starts stale. + slots = [2, 3][:num_contexts] + [0, 1][:num_gens] + spec = types.SimpleNamespace( + runtime_draft_len=K, + is_cuda_graph=False, + request_ids=None, + batch_indices_cuda=None, + use_rejection_sampling=True, + draft_probs=torch.full((SLOTS, K, FULL), STALE, device="cuda"), + batch_slot_ids=torch.tensor(slots, device="cuda"), + draft_probs_last_dim=published, + draft_probs_vocab_size=FULL, + ) + return attn, spec + + +def _run(num_contexts, num_gens, published=FULL): + attn, spec = _metadata(num_contexts, num_gens, published) + n = num_contexts + num_gens + out = _worker()._forward_impl( + input_ids=torch.zeros(n, dtype=torch.int32, device="cuda"), + position_ids=torch.zeros(1, n, dtype=torch.int32, device="cuda"), + hidden_states=torch.zeros(n, HIDDEN, device="cuda"), + logits=torch.zeros(n, FULL, device="cuda"), + attn_metadata=attn, + spec_metadata=spec, + draft_model=_drafter(num_gens), + ) + return out, spec + + +def _onehot(): + row = torch.zeros(K, FULL, device="cuda") + row[:, 0] = 1.0 + return row + + +@pytest.fixture(params=[True, False], ids=["index_fill", "put"]) +def onehot_write(request, monkeypatch): + monkeypatch.setattr(interface, "_ONEHOT_INDEX_FILL", request.param) + + +def test_step_without_generation_requests(onehot_write): + out, spec = _run(num_contexts=2, num_gens=0, published=0) + assert torch.equal( + out["next_draft_tokens"], torch.zeros(2, K, dtype=torch.int32, device="cuda") + ) + # The rows span the buffer's width, which is published for next iteration's acceptance. + assert spec.draft_probs_last_dim == FULL + for slot in (2, 3): + assert torch.equal(spec.draft_probs[slot], _onehot()) + for slot in (0, 1): + assert torch.equal(spec.draft_probs[slot], torch.full((K, FULL), STALE, device="cuda")) + + +def test_mixed_step_rows_span_the_published_width(onehot_write): + out, spec = _run(num_contexts=1, num_gens=1, published=FULL) + assert out["next_draft_tokens"].shape == (2, K) + # Past the drafter's SHARD-wide logits: every column acceptance reads is one-hot, none stale. + assert torch.equal(spec.draft_probs[2], _onehot()) + assert spec.draft_probs_last_dim == FULL + # Rows of slots without a context request are left to their own writers. + for slot in (0, 1, 3): + assert torch.equal(spec.draft_probs[slot], torch.full((K, FULL), STALE, device="cuda")) + + +def test_generation_only_step_writes_no_context_row(onehot_write): + out, spec = _run(num_contexts=0, num_gens=2, published=FULL) + assert out["next_draft_tokens"].shape == (2, K) + assert torch.equal(spec.draft_probs, torch.full((SLOTS, K, FULL), STALE, device="cuda")) diff --git a/tests/unittest/_torch/test_cute_aot.py b/tests/unittest/_torch/test_cute_aot.py new file mode 100644 index 000000000000..0225518253df --- /dev/null +++ b/tests/unittest/_torch/test_cute_aot.py @@ -0,0 +1,176 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Shipped CuTe DSL objects: when ``compile_or_load`` loads one, and when it compiles (and exports).""" + +import os + +import pytest + +from tensorrt_llm._torch import cute_aot + +pytestmark = pytest.mark.cpu_only + +TOOL = "cutedsl-4.8.0.dev0-sm107" + + +class _Compiled: + def __init__(self, tag): + self.tag = tag + self.exports = [] + + def export_to_c(self, path, name=None, **kwargs): + # Writes what CuTe DSL writes: /.o + .h (plain), or the given object file (TVM-FFI). + self.exports.append(kwargs) + if path.endswith(".o"): + open(path, "wb").write(b"obj") + else: + for ext in (".o", ".h"): + open(os.path.join(path, name + ext), "wb").write(b"obj") + + +@pytest.fixture +def aot(monkeypatch, tmp_path): + loads = [] + monkeypatch.setattr(cute_aot, "toolchain", lambda: TOOL) + monkeypatch.setattr(cute_aot, "_loaded", {}) + monkeypatch.setattr(cute_aot, "_counts", {"loaded": 0, "compiled": 0}) + + def load(path, symbol, tvm_ffi): + loads.append((path, symbol, tvm_ffi)) + if path.read_bytes() == b"stale": + raise RuntimeError("version mismatch") + return _Compiled("loaded") + + monkeypatch.setattr(cute_aot, "_load", load) + monkeypatch.setenv(cute_aot.DIR_ENV, str(tmp_path / "data")) + for env in (cute_aot.EXPORT_DIR_ENV, cute_aot.DISABLE_ENV): + monkeypatch.delenv(env, raising=False) + return loads + + +def _ship(tmp_path, key, content=b"obj", sources=()): + path = tmp_path / "data" / TOOL / f"{cute_aot.digest(cute_aot.full_key(key, sources))}.o" + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content) + return path + + +def test_shipped_object_is_loaded_not_compiled(aot, tmp_path): + path = _ship(tmp_path, "gemv m=8") + got = cute_aot.compile_or_load("gemv m=8", lambda: pytest.fail("compiled")) + assert got.tag == "loaded" + assert aot == [(path, cute_aot.digest(cute_aot.full_key("gemv m=8")), False)] + assert cute_aot.counts() == {"loaded": 1, "compiled": 0} + + +def test_missing_object_compiles_once_per_key(aot): + calls = [] + for _ in range(2): + got = cute_aot.compile_or_load("gemv m=16", lambda: calls.append(1) or _Compiled("jit")) + assert got.tag == "jit" and calls == [1] + assert cute_aot.counts() == {"loaded": 0, "compiled": 1} + + +def test_other_key_or_toolchain_is_not_served(aot, tmp_path, monkeypatch): + _ship(tmp_path, "gemv m=8") + assert cute_aot.compile_or_load("gemv m=32", lambda: _Compiled("jit")).tag == "jit" + monkeypatch.setattr(cute_aot, "toolchain", lambda: "cutedsl-4.9.0-sm107") + assert cute_aot.compile_or_load("gemv m=8", lambda: _Compiled("jit")).tag == "jit" + assert aot == [] + + +def test_stale_object_falls_back_to_compiling(aot, tmp_path): + _ship(tmp_path, "gemv m=8", b"stale") + assert cute_aot.compile_or_load("gemv m=8", lambda: _Compiled("jit")).tag == "jit" + assert cute_aot.counts() == {"loaded": 0, "compiled": 1} + + +def test_disable_always_compiles(aot, tmp_path, monkeypatch): + _ship(tmp_path, "gemv m=8") + monkeypatch.setenv(cute_aot.DISABLE_ENV, "1") + assert cute_aot.compile_or_load("gemv m=8", lambda: _Compiled("jit")).tag == "jit" + assert aot == [] + + +@pytest.mark.parametrize("tvm_ffi", [False, True]) +def test_export_writes_the_shipped_layout(aot, tmp_path, monkeypatch, tvm_ffi): + monkeypatch.setenv(cute_aot.EXPORT_DIR_ENV, str(tmp_path / "export")) + compiled = cute_aot.compile_or_load("moe rows=4", lambda: _Compiled("jit"), tvm_ffi=tvm_ffi) + symbol = cute_aot.digest(cute_aot.full_key("moe rows=4")) + out = tmp_path / "export" / TOOL + assert (out / f"{symbol}.key").read_text() == cute_aot.full_key("moe rows=4") + "\n" + expected = {f"{symbol}.o", f"{symbol}.key"} | (set() if tvm_ffi else {f"{symbol}.h"}) + assert set(os.listdir(out)) == expected # the private export directory is gone + if tvm_ffi: + assert compiled.exports == [{"function_name": symbol, "export_only_tvm_ffi_symbols": True}] + else: + assert compiled.exports == [{"function_prefix": symbol}] + + +def test_digest_is_a_c_identifier(): + d = cute_aot.digest("any key: with spaces / slashes") + assert d.isidentifier() and d == cute_aot.digest("any key: with spaces / slashes") + + +def test_edited_source_is_not_served(aot, tmp_path): + src = tmp_path / "kernel.py" + src.write_text("v1") + _ship(tmp_path, "gemv m=8", sources=[str(src)]) + assert ( + cute_aot.compile_or_load("gemv m=8", lambda: _Compiled("jit"), sources=[str(src)]).tag + == "loaded" + ) + edited = tmp_path / "edited" / "kernel.py" + edited.parent.mkdir() + edited.write_text("v2") + assert ( + cute_aot.compile_or_load("gemv m=8", lambda: _Compiled("jit"), sources=[str(edited)]).tag + == "jit" + ) + + +def test_source_digest_ignores_install_location(tmp_path): + a, b = tmp_path / "a" / "k.py", tmp_path / "b" / "k.py" + for f in (a, b): + f.parent.mkdir() + f.write_text("same") + assert cute_aot.full_key("k", [str(a)]) == cute_aot.full_key("k", [str(b)]) + assert cute_aot.py_files(str(tmp_path / "a")) == (str(a),) + + +def test_model_switch_or_dsl_setting_changes_the_key(aot, tmp_path, monkeypatch): + for k in [k for k in os.environ if k.startswith(cute_aot._ENV_PREFIXES)]: + monkeypatch.delenv(k) + _ship(tmp_path, "moe rows=4") + monkeypatch.setenv("CUTE_DSL_CACHE_DIR", "/tmp/x") # a directory: not a codegen setting + assert cute_aot.compile_or_load("moe rows=4", lambda: _Compiled("jit")).tag == "loaded" + monkeypatch.setattr(cute_aot, "_loaded", {}) + monkeypatch.setenv("TRTLLM_M31_DISABLE_MOE_SH20", "1") + assert cute_aot.compile_or_load("moe rows=4", lambda: _Compiled("jit")).tag == "jit" + + +def test_cached_compile_keys_on_bound_arguments(aot, tmp_path): + calls = [] + + @cute_aot.cached_compile("gemm", lambda: ()) + def build(dtype, tile, bias=False): + calls.append((dtype, tile, bias)) + return _Compiled("jit") + + _ship(tmp_path, "gemm dtype='bf16' tile=(128, 64) bias=True") + assert build("bf16", (128, 64), bias=True).tag == "loaded" + assert build("bf16", (128, 64)).tag == "jit" + assert build("bf16", tile=(128, 64)).tag == "jit" + assert calls == [("bf16", (128, 64), False)] diff --git a/tests/unittest/_torch/test_flashinfer_autotune_cache.py b/tests/unittest/_torch/test_flashinfer_autotune_cache.py new file mode 100644 index 000000000000..deaaace34239 --- /dev/null +++ b/tests/unittest/_torch/test_flashinfer_autotune_cache.py @@ -0,0 +1,71 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""The per-rank FlashInfer autotune cache file: what each ``autotune`` call hands FlashInfer.""" + +import sys +import types + +import pytest + +from tensorrt_llm._torch import flashinfer_autotune_cache as fac + +pytestmark = pytest.mark.cpu_only + + +@pytest.fixture +def calls(monkeypatch): + seen = [] + fake = types.ModuleType("flashinfer") + fake.autotune = lambda tune_mode=True, cache=None, *a, **k: seen.append((tune_mode, cache, k)) + monkeypatch.setitem(sys.modules, "flashinfer", fake) + monkeypatch.setattr(fac, "mpi_rank", lambda: 3) + monkeypatch.setattr(fac, "_STATE", {"loaded": False, "logged": False, "bundled": None}) + for env in (fac.CACHE_DIR_ENV, fac.FROZEN_ENV): + monkeypatch.delenv(env, raising=False) + return seen + + +def test_unset_is_flashinfers_own_call(calls): + fac.autotune(tune_mode=True) + fac.autotune(tune_mode=False, skip_ops="mxfp8_gemm") + assert calls == [(True, None, {}), (False, None, {"skip_ops": "mxfp8_gemm"})] + + +def test_learning_mode_tunes_into_this_ranks_file(calls, monkeypatch, tmp_path): + monkeypatch.setenv(fac.CACHE_DIR_ENV, str(tmp_path)) + fac.autotune(tune_mode=True) + fac.autotune(tune_mode=False) + path = str(tmp_path / "rank3.json") + assert calls == [(True, path, {}), (False, path, {})] + + +def test_frozen_mode_loads_the_file_once_and_never_tunes(calls, monkeypatch, tmp_path): + monkeypatch.setenv(fac.CACHE_DIR_ENV, str(tmp_path)) + monkeypatch.setenv(fac.FROZEN_ENV, "1") + fac.autotune(tune_mode=True) + fac.autotune(tune_mode=True) + fac.autotune(tune_mode=False, skip_ops="mxfp8_gemm") + path = str(tmp_path / "rank3.json") + assert calls == [ + (False, path, {}), + (False, None, {}), + (False, None, {"skip_ops": "mxfp8_gemm"}), + ] + + +def test_an_explicit_cache_is_left_alone(calls, monkeypatch, tmp_path): + monkeypatch.setenv(fac.CACHE_DIR_ENV, str(tmp_path)) + fac.autotune(True, "/elsewhere.json") + assert calls == [(True, "/elsewhere.json", {})] diff --git a/tests/unittest/_torch/thop/parallel/test_minimax_m31_dspark_glue_ops.py b/tests/unittest/_torch/thop/parallel/test_minimax_m31_dspark_glue_ops.py new file mode 100644 index 000000000000..300f799b2092 --- /dev/null +++ b/tests/unittest/_torch/thop/parallel/test_minimax_m31_dspark_glue_ops.py @@ -0,0 +1,135 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""DSPARK_GLUE_CPP: trtllm::minimax_m31_dspark_advance / _indices (kernels/minimaxM31/dsparkGlueSm107.cu) bit for bit +the catalog Triton kernels they replace (catalog/spec_decode/dspark_ctx_advance.py, dspark_forward_indices.py), on +every output, through the catalog wrappers (C++ path vs the forced Triton path); and the kill switch.""" + +import os +import subprocess +import sys + +import pytest +import torch + +import tensorrt_llm # noqa: F401 (loads the trtllm op library) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_ctx_advance as AD, +) +from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_forward_indices as IX, +) + +_GPU = pytest.mark.skipif( + not torch.cuda.is_available() + or torch.cuda.get_device_capability() != (10, 7) + or not hasattr(torch.ops.trtllm, "minimax_m31_dspark_advance") + or not hasattr(torch.ops.trtllm, "minimax_m31_dspark_indices"), + reason="needs an SM107 GPU and the DSPARK_GLUE_CPP ops", +) + + +def _ri(g, lo, hi, shape, dtype=torch.int64): + return torch.randint(lo, hi, shape, generator=g, device="cuda", dtype=dtype) + + +def _both(fn, monkeypatch, mod): + """(C++ result, Triton result) of fn() through the catalog wrapper.""" + got = fn() + monkeypatch.setattr(mod, "_GLUE_CPP_OFF", True) + ref = fn() + monkeypatch.setattr(mod, "_GLUE_CPP_OFF", False) + return got, ref + + +@_GPU +@pytest.mark.parametrize("width", [2048, 8192, 65536]) +@pytest.mark.parametrize("n", [1, 7, 22, 56]) +@pytest.mark.parametrize("nacc_dtype", [torch.int32, torch.int64]) +@pytest.mark.parametrize("parallel", [False, True]) +def test_advance_bitwise(monkeypatch, width, n, nacc_dtype, parallel): + g = torch.Generator(device="cuda").manual_seed(width + 31 * n) + nslots = nrows = 64 + page, block_size = 128, 7 + slots = torch.randperm(nslots - 1, generator=g, device="cuda")[:n].to(torch.int64) + if n > 2: + slots[-1] = nslots - 1 # a dummy (padding) request + rows = torch.randperm(nrows, generator=g, device="cuda")[:n].to(torch.int64) + nacc = _ri(g, 0, 9, (n,), nacc_dtype) + max_ctx = width * page - 1000 + ctx0 = _ri(g, 0, max_ctx, (nslots,)) + ctx0[:2] = max_ctx # at the clamp + counts = _ri(g, 0, width + 1, (nrows,)) + counts[0] = 0 # nothing allocated + base = _ri(g, 0, width // 4, (nslots,)) + tables = _ri(g, 0, 2**31 - 1, (nrows, width), torch.int32) + + def run(): + ctx, read = ctx0.clone(), torch.zeros_like(tables) + args = ( + slots, + rows, + nacc, + ctx, + max_ctx, + nslots - 1, + counts, + base, + tables, + read, + page, + block_size, + ) + out = AD.advance_ctx_and_read_view(*args, parallel=parallel) + torch.cuda.synchronize() + return out, ctx, read + + got, ref = _both(run, monkeypatch, AD) + if not parallel: # the engagement keys behind the "[M3 dispatch] DSPARK_GLUE_CPP ..." lines + assert ("minimax_m31_dspark_advance", None) in AD._glue_logged + assert ( + "minimax_m31_dspark_advance", + "killed (TRTLLM_M31_DISABLE_DSPARK_GLUE_CPP=1)", + ) in AD._glue_logged + for a, b in zip(got, ref): + assert a.dtype == b.dtype and torch.equal(a, b) + + +@_GPU +@pytest.mark.parametrize("B", [1, 7, 22, 56]) +@pytest.mark.parametrize("dtype", [torch.int32, torch.int64]) +@pytest.mark.parametrize("strided", [False, True]) +def test_indices_bitwise(monkeypatch, B, dtype, strided): + g = torch.Generator(device="cuda").manual_seed(7 * B + int(strided)) + block, pages_per_slot, page = 7, 8192, 128 + num_ctx = _ri(g, 0, 2**20, (B,), dtype) + num_ctx[0] = 0 # (after - 1) % page at the wrap + batch_idx = _ri(g, 0, 64, (B,), dtype) + q = _ri(g, 0, 2**20, (B, 2 * block), dtype) + qpos = q[:, ::2] if strided else q[:, :block].contiguous() + if dtype == torch.int64: + num_ctx[-1] = 2**33 + 5 # the int64 -> int32 cast keeps the low 32 bits + qpos[-1, -1] = -(2**34) - 3 + + def run(): + out = IX.forward_indices(num_ctx, batch_idx, qpos, block, pages_per_slot, page) + torch.cuda.synchronize() + return out + + got, ref = _both(run, monkeypatch, IX) + assert ("minimax_m31_dspark_indices", None) in IX._glue_logged + for a, b in zip(got, ref): + assert a.dtype == b.dtype and torch.equal(a, b) + + +def test_kill_switch(): + code = ( + "import torch\n" + "from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import dspark_ctx_advance as a, " + "dspark_forward_indices as i\n" + "print(a._GLUE_CPP_OFF, i._GLUE_CPP_OFF, a._glue_cpp_op('x', torch.device('cuda', 0)) is None)" + ) + env = {**os.environ, "TRTLLM_M31_DISABLE_DSPARK_GLUE_CPP": "1"} + out = subprocess.run( + [sys.executable, "-c", code], env=env, capture_output=True, text=True, check=True + ) + assert out.stdout.split()[-3:] == ["True", "True", "True"] diff --git a/tests/unittest/_torch/thop/parallel/test_mxfp8_cute_pinned_tactics.py b/tests/unittest/_torch/thop/parallel/test_mxfp8_cute_pinned_tactics.py new file mode 100644 index 000000000000..f9c38516d2fb --- /dev/null +++ b/tests/unittest/_torch/thop/parallel/test_mxfp8_cute_pinned_tactics.py @@ -0,0 +1,100 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""The pinned FlashInfer CuTe DSL MXFP8 GEMM tactics (custom_ops/mxfp8_cute_pinned_tactics): every entry is +non-split-K, each +pinned call equals FlashInfer's default (heuristic) CuTe DSL path bit for bit, the runner takes the pinned path above 32 +rows only, and the kill switch restores the default path.""" + +import hashlib + +import pytest +import torch + +from tensorrt_llm._torch.custom_ops import mxfp8_cute_pinned_tactics as pin +from tensorrt_llm._utils import get_sm_version + +sm107 = pytest.mark.skipif( + not torch.cuda.is_available() or get_sm_version() != 107, reason="SM107 (Rubin) only" +) + + +@pytest.fixture(autouse=True) +def _armed(monkeypatch): + """The table as an opted-in model's executor arms it (RuntimeHooks.mxfp8_pinned_tactics).""" + monkeypatch.setattr(pin, "_ARMED", [True]) + + +def _table_digest() -> str: + return hashlib.md5(repr(sorted(pin.TACTICS.items())).encode()).hexdigest()[:8] + + +def test_table_is_non_split_k_and_pinned(): + assert pin.TACTICS, "empty table" + for (n, k, m), (tile, cluster, swap, prefetch, split_k) in pin.TACTICS.items(): + assert split_k == 1, ((n, k, m), "a split-K tactic changes the K order (not bitwise)") + assert m >= 64 and m & (m - 1) == 0 and m <= pin.MAX_BUCKET, (n, k, m) + assert ( + len(tile) == 2 + and len(cluster) == 2 + and isinstance(swap, bool) + and isinstance(prefetch, bool) + ) + # Drift guard: a re-tuned table is a new measured artifact (bitwise + timing), not an edit in place. + assert _table_digest() == "d43d4a0d", _table_digest() + + +def test_bucket_and_row_gate(): + assert ( + pin._bucket(33) == 64 + and pin._bucket(64) == 64 + and pin._bucket(65) == 128 + and pin._bucket(184) == 256 + ) + assert pin.tactic_for(32, 4992, 6144) is None # 1..32 rows keep FlashInfer's own low-M path + + +def _operands(m, n, k, seed): + g = torch.Generator(device="cuda").manual_seed(seed) + x = (torch.randn(m, k, device="cuda", generator=g) * 0.5).to(torch.bfloat16) + w = (torch.randn(n, k, device="cuda", generator=g) * 0.05).to(torch.bfloat16) + q, s = torch.ops.trtllm.mxfp8_quantize(x, True) + wq, ws = torch.ops.trtllm.mxfp8_quantize(w, True) + return q, s, wq, ws + + +def _default(q, s, wq, ws): + from flashinfer import autotune, mm_mxfp8 + + with autotune(tune_mode=False, skip_ops="mxfp8_gemm"): + return mm_mxfp8( + q, wq.t(), s, ws, out_dtype=torch.bfloat16, use_8x4_sf_layout=False, backend="cute-dsl" + ) + + +@sm107 +@pytest.mark.parametrize("key", sorted(pin.TACTICS), ids=lambda k: f"N{k[0]}_K{k[1]}_M{k[2]}") +def test_pinned_equals_default_bitwise(key): + n, k, bucket = key + for m in sorted({bucket // 2 + 1 if bucket > 64 else 33, bucket}): # both ends of the bucket + q, s, wq, ws = _operands(m, n, k, seed=hash(key) % 1000 + m) + assert pin.tactic_for(m, n, k) == pin.TACTICS[key] + out = pin.run(q, s, wq, ws, torch.bfloat16) + assert out is not None, "the pinned path declined a table entry" + assert torch.equal(out, _default(q, s, wq, ws)), (key, m) + + +@sm107 +def test_runner_takes_pinned_above_32_rows_and_kill(monkeypatch): + from tensorrt_llm._torch.custom_ops.torch_custom_ops import FlashInferMXFP8GemmRunner + + runner = FlashInferMXFP8GemmRunner(torch.bfloat16) + calls = [] + real = pin.run + monkeypatch.setattr(pin, "run", lambda *a, **kw: calls.append(a[0].shape[0]) or real(*a, **kw)) + for m in (16, 80): + q, s, wq, ws = _operands(m, 4992, 6144, seed=m) + out = runner.forward([q, s, wq, ws], tactic=FlashInferMXFP8GemmRunner.CUTE_DSL) + assert torch.equal(out, _default(q, s, wq, ws)), m + assert calls == [16, 80] + monkeypatch.setattr(pin, "ENABLED", False) + assert pin.tactic_for(80, 4992, 6144) is None diff --git a/tests/unittest/disaggregated/f0a_g1d_compare.py b/tests/unittest/disaggregated/f0a_g1d_compare.py new file mode 100644 index 000000000000..18eb8f1298e5 --- /dev/null +++ b/tests/unittest/disaggregated/f0a_g1d_compare.py @@ -0,0 +1,183 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Draft-window compare for the F0a dumps (TRTLLM_M31_F0A_DUMP_DIR). Not a pytest module. + +Usage: ``python f0a_g1d_compare.py DUMP_DIR CTX_TP GEN_TP OUT_JSON`` + +``CTX_TP`` is the context side's TP ranks per attention-DP group: 1 under attention DP, where +one rank holds every drafter head and its dump is whichever rank wrote it; the TP size under +TP-attention (e.g. 2 for tp2 without attention DP), where TP rank ``c`` holds heads +``[c * H_ctx, (c + 1) * H_ctx)`` and each request has one dump per CTX rank, all with the same +``match_id``. ``GEN_TP`` is the generation side's. Dumps pair by +``match_id`` (the context request id both sides know). Generation rank ``r`` holds global +head ``r * H // GEN_TP`` (``H`` = the context side's ``heads_per_rank * CTX_TP``): head ``r`` +at TP4, ``r // 2`` at TP8 for four drafter heads, as ``duplicate_kv_weight`` replicates them. +Every row the generation window holds below ``ctx_len`` is compared bitwise with the context +side's row, for every layer and K/V; the seeded ``(ctx_len, ctx_base)`` must equal the shipped +one. A context-side fallback (no window shipped) is listed as ``fallback``. + +Verdict ``G1D_PASS`` when every compared request is bitwise equal with equal state and at least +one request was compared, else ``G1D_FAIL``. Exit status 0 on pass, 1 on fail. +""" + +import glob +import json +import os +import sys + +import torch + +FORMAT = "f0a_g1d_v1" + + +def _load(directory): + ctx, gen = {}, {} + for path in sorted(glob.glob(os.path.join(directory, "*.pt"))): + dump = torch.load(path, map_location="cpu", weights_only=False) + if not isinstance(dump, dict) or dump.get("format") != FORMAT: + continue + if dump["side"] == "ctx": + ctx.setdefault(dump["match_id"], []).append(dump) + elif dump["side"] == "gen": + gen.setdefault(dump["match_id"], []).append(dump) + return ctx, gen + + +def _rows_by_ordinal(dump, group): + return {int(o): i for i, o in enumerate(dump["ordinals"][group].tolist())} + + +def _global_head(gen_rank, local_head, gen_heads, gen_tp, total_heads): + """The drafter head generation rank ``gen_rank`` holds in ``local_head`` (HND mapper rule).""" + return (gen_rank * gen_heads + local_head) * total_heads // (gen_tp * gen_heads) % total_heads + + +def _compare_rank(ctx_dumps, gen_dump, gen_rank, ctx_tp, gen_tp): + """(K/V rows compared, mismatched K/V rows, problems) for one generation rank's dump. + + ``ctx_dumps`` maps a context TP rank within its DP group to its dump; under attention DP + (``ctx_tp == 1``) it is the one rank that dumped the request. + """ + problems = [] + ctx_heads = next(iter(ctx_dumps.values()))["heads_per_rank"] + total_heads = ctx_heads * ctx_tp + tpb = gen_dump["tokens_per_block"] + ctx_len = gen_dump["ctx_len"] + rows = mismatches = 0 + for gen_layer in gen_dump["layers"]: + key = (gen_layer["layer_group"], gen_layer["global_layer_id"]) + group = key[0] + for local_head in range(gen_dump["heads_per_rank"]): + head = _global_head( + gen_rank, local_head, gen_dump["heads_per_rank"], gen_tp, total_heads + ) + ctx_dump = ctx_dumps.get(head // ctx_heads) + if ctx_dump is None: + problems.append(f"no context dump holds head {head}") + continue + ctx_layer = { + (r["layer_group"], r["global_layer_id"]): r for r in ctx_dump["layers"] + }.get(key) + if ctx_layer is None: + problems.append(f"layer {key} missing on the context side") + continue + ctx_index = _rows_by_ordinal(ctx_dump, group) + ctx_head = head % ctx_heads + for gen_i, ordinal in enumerate(gen_dump["ordinals"][group].tolist()): + valid = max(0, min(tpb, ctx_len - int(ordinal) * tpb)) + if valid == 0: + continue + ctx_i = ctx_index.get(int(ordinal)) + if ctx_i is None: + problems.append(f"ordinal {ordinal} held on generation, not on context") + rows += 2 * valid + mismatches += 2 * valid + continue + for name in ("k", "v"): + got = gen_layer[name][gen_i, local_head, :valid] + want = ctx_layer[name][ctx_i, ctx_head, :valid] + rows += valid + mismatches += int((got != want).any(dim=-1).sum()) + return rows, mismatches, problems + + +def compare(directory, ctx_tp, gen_tp): + ctx, gen = _load(directory) + report = {"dump_dir": directory, "ctx_tp": ctx_tp, "gen_tp": gen_tp, "requests": {}} + compared = 0 + passed = True + for match_id in sorted(set(ctx) | set(gen)): + ctx_dumps = ctx.get(match_id, []) + entry = {"ctx_ranks": [d["rank"] for d in ctx_dumps]} + report["requests"][str(match_id)] = entry + shipped = [d for d in ctx_dumps if d["shipped"]] + if ctx_dumps and not shipped: + entry["status"] = "fallback" + entry["reason"] = ctx_dumps[0].get("reason") + continue + gen_dumps = gen.get(match_id, []) + if not shipped or not gen_dumps: + entry["status"] = "unpaired" + entry["gen_ranks"] = [d["rank"] for d in gen_dumps] + passed = False + continue + # Under attention DP the context rank holding the request is whichever dumped it. + ctx_dump = shipped[0] + ctx_by_tp_rank = {d["rank"] % ctx_tp: d for d in shipped} if ctx_tp > 1 else {0: ctx_dump} + entry.update( + status="compared", + ctx_len=ctx_dump["ctx_len"], + ctx_base=ctx_dump["ctx_base"], + gen_ranks={}, + ) + ok = True + for gen_dump in sorted(gen_dumps, key=lambda d: d["rank"]): + rows, mismatches, problems = _compare_rank( + ctx_by_tp_rank, gen_dump, gen_dump["rank"] % gen_tp, ctx_tp, gen_tp + ) + state_equal = (gen_dump["ctx_len"], gen_dump["ctx_base"]) == ( + ctx_dump["ctx_len"], + ctx_dump["ctx_base"], + ) + entry["gen_ranks"][str(gen_dump["rank"])] = { + "rows_compared": rows, + "mismatches": mismatches, + "ctx_len_equal": gen_dump["ctx_len"] == ctx_dump["ctx_len"], + "ctx_base_equal": gen_dump["ctx_base"] == ctx_dump["ctx_base"], + "problems": problems, + } + ok = ok and mismatches == 0 and state_equal and not problems + entry["bitwise"] = ok + passed = passed and ok + compared += 1 + report["requests_compared"] = compared + report["verdict"] = "G1D_PASS" if passed and compared > 0 else "G1D_FAIL" + return report + + +def main(argv): + if len(argv) != 5: + print(__doc__) + return 2 + _, directory, ctx_tp, gen_tp, out_json = argv + report = compare(directory, int(ctx_tp), int(gen_tp)) + with open(out_json, "w") as f: + json.dump(report, f, indent=1, sort_keys=True) + print(f"{report['verdict']}: {report['requests_compared']} request(s) compared") + return 0 if report["verdict"] == "G1D_PASS" else 1 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/tests/unittest/disaggregated/test_disagg_draft_ctx_state_carrier.py b/tests/unittest/disaggregated/test_disagg_draft_ctx_state_carrier.py new file mode 100644 index 000000000000..9972e3b02073 --- /dev/null +++ b/tests/unittest/disaggregated/test_disagg_draft_ctx_state_carrier.py @@ -0,0 +1,99 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""F0a context-first: the drafter window decision reaches the generation request in serving. + +It rode the response's opaque_state, which becomes the C++ ContextPhaseParams' serialized +DataTransceiverState: the bytes were parsed and re-serialized, so in serving every generation +request saw no state (no_ctx_state) and asked for no draft pages. The decision now rides its own +DisaggregatedParams field, draft_ctx_state, set on the context worker's LlmResult (like +remote_tail_start) and carried through the serving protocol. +""" + +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch.pyexecutor.llm_request import _response_draft_ctx_state +from tensorrt_llm.bindings import executor as tllm +from tensorrt_llm.disaggregated_params import DisaggregatedParams +from tensorrt_llm.executor.result import GenerationResultBase +from tensorrt_llm.sampling_params import SamplingParams + +pytestmark = pytest.mark.cpu_only + + +@pytest.mark.parametrize( + "aux, context_only, expected", + [ + ((9000, 3, True), True, [9000, 3, 1]), + ((0, 0, False), True, [0, 0, 0]), # the window did not travel: the receiver asks none + ((64, 0, True), False, None), # a generation request's own aux (generation-first) stays off + (None, True, None), # no window decision (F0a off or not engaged) + ], +) +def test_the_context_response_carries_the_decision(aux, context_only, expected): + request = SimpleNamespace(py_draft_ctx_aux=aux, is_context_only_request=context_only) + assert _response_draft_ctx_state(request) == expected + + +def _context_result(draft_ctx_state=None): + result = SimpleNamespace( + is_final=True, + decoding_iter=1, + avg_decoded_tokens_per_iter=1.0, + context_phase_params=SimpleNamespace( + first_gen_tokens=[7], + req_id=1, + opaque_state=None, + draft_tokens=None, + ctx_dp_rank=0, + disagg_info_endpoint=None, + ), + finish_reasons=[tllm.FinishReason.LENGTH], + output_token_ids=[[7]], + sequence_index=0, + cum_log_probs=None, + log_probs=None, + generation_logits=None, + context_logits=None, + request_perf_metrics=None, + additional_context_outputs=None, + additional_generation_outputs=None, + ) + if draft_ctx_state is not None: + result.draft_ctx_state = draft_ctx_state + return result + + +@pytest.mark.parametrize("state", [[9000, 3, 1], [0, 0, 0], None]) +def test_the_disaggregated_params_carry_draft_ctx_state(state): + result = GenerationResultBase(id=1, sampling_params=SamplingParams(max_tokens=4)) + result._handle_response(SimpleNamespace(result=_context_result(state), has_error=lambda: False)) + params = result.disaggregated_params + assert params.request_type == "context_only" + assert params.draft_ctx_state == state + assert params.opaque_state is None + + +def test_draft_ctx_state_never_enters_the_cpp_context_phase_params(): + """The generation request builds its C++ ContextPhaseParams from opaque_state only.""" + params = DisaggregatedParams( + request_type="generation_only", + ctx_request_id=1, + draft_ctx_state=[9000, 3, 1], + ) + cpp = params.get_context_phase_params() + assert cpp.opaque_state is None + assert params.draft_ctx_state == [9000, 3, 1] diff --git a/tests/unittest/disaggregated/test_disagg_draft_kv_hook_off.py b/tests/unittest/disaggregated/test_disagg_draft_kv_hook_off.py new file mode 100644 index 000000000000..3b00be40c15f --- /dev/null +++ b/tests/unittest/disaggregated/test_disagg_draft_kv_hook_off.py @@ -0,0 +1,635 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""The disagg drafter context window (F0a) off and on, around the write itself. + +Off (the hook False, the default): the transceiver's chunks, wire messages, byte accounting, aux +layout, send path and receive fences are exactly the target-only ones. On: the context side waits +on the drafter-store event before publishing (unless the test-only knob skips it), the aux slot +carries the drafter context lengths, the generation side hands them to the worker, and the draft +pool's readiness event joins the receive fence. +""" + +from types import SimpleNamespace +from unittest.mock import MagicMock, Mock + +import msgpack +import numpy as np +import pytest +import torch + +from tensorrt_llm import DisaggregatedParams +from tensorrt_llm._torch.disaggregation import transceiver as transceiver_module +from tensorrt_llm._torch.disaggregation.base import CacheExtent, CacheKind, Chunk, TokenRange +from tensorrt_llm._torch.disaggregation.draft_kv import ( + DraftKvCounters, + decode_draft_ctx_state, + encode_draft_ctx_state, +) +from tensorrt_llm._torch.disaggregation.native.auxiliary import AuxBuffer +from tensorrt_llm._torch.disaggregation.native.transfer import RecvReqInfo +from tensorrt_llm._torch.disaggregation.orchestration.coordinator import DisaggTransferCoordinator +from tensorrt_llm._torch.disaggregation.transceiver import KvCacheTransceiverV2 +from tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 import KVCacheManagerV2 +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks, runtime_hooks_of +from tensorrt_llm.disaggregated_params import DisaggScheduleStyle + +pytestmark = pytest.mark.cpu_only + +_RID = 42 + + +def _chunk(draft=None, is_last=True): + return Chunk( + block_ids_per_layer_groups=[np.array([3, 4], dtype=np.int64)], + kind_per_layer_group=[CacheKind.PAGED], + token_range=TokenRange(start=0, end=64), + is_last=is_last, + draft_block_ids_per_layer_groups=draft, + ) + + +# --------------------------------------------------------------------------- +# Off: identical to the target-only path +# --------------------------------------------------------------------------- + + +def test_hook_is_off_for_every_model_that_does_not_set_it(): + assert RuntimeHooks().disagg_draft_kv is False + assert runtime_hooks_of(object()).disagg_draft_kv is False + assert KVCacheManagerV2._disagg_draft_kv_receive is False + + +def test_chunk_and_transceiver_carry_no_draft_set_by_default(): + assert _chunk().draft_block_ids_per_layer_groups is None + transceiver = object.__new__(KvCacheTransceiverV2) + assert transceiver._draft_ctx_owner is None + assert transceiver._describe_draft_local(SimpleNamespace(prompt_len=64)) is None + + +def test_recv_req_info_without_a_draft_set_is_the_legacy_message(): + info = RecvReqInfo( + sender_req_id=7, + instance_name="gen", + instance_rank=3, + block_ids_per_layer_groups=[np.array([1, -1, 2], dtype=np.int64)], + unique_rid=_RID, + dst_start_token=None, + aux_slot=5, + slice_id=0, + bounce_dst_base=None, + ) + legacy = msgpack.packb( + { + "sender_req_id": 7, + "instance_name": "gen", + "instance_rank": 3, + "block_ids_per_layer_groups": [np.array([1, -1, 2], dtype=np.int64).tobytes()], + "unique_rid": _RID, + "dst_start_token": None, + "aux_slot": 5, + "slice_id": 0, + "bounce_dst_base": None, + } + ) + assert info.to_bytes() == legacy + assert RecvReqInfo.from_bytes(legacy).draft_block_ids_per_layer_groups is None + + +def test_recv_req_info_round_trips_the_draft_set(): + draft = [np.array([-1, -1, 9, 8], dtype=np.int64)] + info = RecvReqInfo( + sender_req_id=7, + instance_name="gen", + instance_rank=0, + block_ids_per_layer_groups=[np.array([1], dtype=np.int64)], + unique_rid=_RID, + draft_block_ids_per_layer_groups=draft, + ) + restored = RecvReqInfo.from_bytes(info.to_bytes()) + assert len(restored.draft_block_ids_per_layer_groups) == 1 + assert np.array_equal(restored.draft_block_ids_per_layer_groups[0], draft[0]) + + +def test_chunk_bytes_count_no_draft_pages_when_off(): + stand_in = SimpleNamespace(_page_table=None) + assert KvCacheTransceiverV2._chunk_num_bytes(stand_in, _chunk()) == 0 + transceiver = object.__new__(KvCacheTransceiverV2) + assert KvCacheTransceiverV2._draft_chunk_num_bytes(None, _chunk(draft=[np.array([1])])) == 0 + assert transceiver._draft_page_table is None + + +def _request(**kwargs): + defaults = dict( + py_request_id=_RID, + py_disaggregated_params=DisaggregatedParams( + disagg_request_id=_RID, schedule_style=DisaggScheduleStyle.GENERATION_FIRST + ), + prompt_len=64, + cached_tokens=0, + py_draft_tokens=[], + get_last_tokens=lambda: [17], + ) + defaults.update(kwargs) + return SimpleNamespace(**defaults) + + +def test_aux_layout_is_unchanged_without_the_draft_slot(): + aux = AuxBuffer(max_slot_num=4, beam_width=1, max_draft_len=3) + assert aux.meta.ptrs.size == 4 + slot = aux.alloc_slot().id + aux.fill_slot(slot, _request(py_draft_ctx_aux=(64, 1, True))) + assert aux.get_slot_draft_ctx(slot) is None + + +def test_aux_draft_slot_carries_the_context_state(): + aux = AuxBuffer(max_slot_num=4, beam_width=1, max_draft_len=3, draft_ctx=True) + plain = AuxBuffer(max_slot_num=4, beam_width=1, max_draft_len=3) + assert aux.meta.ptrs.size == 5 + assert np.array_equal(aux.meta.item_sizes[:4], plain.meta.item_sizes) + assert aux.meta.item_sizes[4] == 3 * 4 + slot = aux.alloc_slot().id + aux.fill_slot(slot, _request(py_draft_ctx_aux=(9000, 3, True))) + assert aux.get_slot_draft_ctx(slot) == (9000, 3, True) + aux.fill_slot(slot, _request(py_draft_ctx_aux=(0, 0, False))) # the window did not travel + assert aux.get_slot_draft_ctx(slot) == (0, 0, False) + aux.fill_slot(slot, _request()) # no decision at all reads as "not shipped" + assert aux.get_slot_draft_ctx(slot) == (0, 0, False) + + +def test_context_state_codec_round_trips_and_ignores_foreign_values(): + state = encode_draft_ctx_state(9000, 3, True) + assert state == [9000, 3, 1] + assert decode_draft_ctx_state(state) == (9000, 3, True) + assert decode_draft_ctx_state(encode_draft_ctx_state(64, 0, False)) == (64, 0, False) + for foreign in ( + None, + b"", + b"F0A1" + b"\x00" * 17, + "opaque", + [1, 2], + [1, 2, 3, 4], + [-1, 0, 1], + [1, 0, 2], + ["a", 0, 1], + ): + assert decode_draft_ctx_state(foreign) is None + + +# --------------------------------------------------------------------------- +# The context side's send: wait on the drafter-store event, decide, then publish +# --------------------------------------------------------------------------- + +_CONTEXT_FIRST = dict(schedule_style=DisaggScheduleStyle.CONTEXT_FIRST) +_GENERATION_FIRST = dict(schedule_style=DisaggScheduleStyle.GENERATION_FIRST) + + +class _ReadyEvent: + def __init__(self, order): + self._order = order + + def synchronize(self): + self._order.append("wait") + + +class _Owner: + def __init__(self, order, ctx_len, ctx_base): + self.ready = {} if ctx_len is None else {_RID: (_ReadyEvent(order), ctx_len, ctx_base)} + self.seeds = {} + + def take_disagg_draft_ctx_ready(self, request_id): + return self.ready.pop(request_id, None) + + def seed_disagg_draft_ctx(self, request_id, ctx_len, ctx_base, prompt_len): + self.seeds[request_id] = (ctx_len, ctx_base, prompt_len) + + +def _engage(transceiver, owner, knob=False): + transceiver._draft_ctx_owner = owner + transceiver._send_before_draft_ctx_event = knob + transceiver._draft_page_table = SimpleNamespace( + tokens_per_block=32, layer_groups=[SimpleNamespace(sliding_window_size=None)] + ) + transceiver._draft_kv_ctx_counters = DraftKvCounters("context") + transceiver._draft_kv_gen_counters = DraftKvCounters("generation") + + +def _send(monkeypatch, *, owner=None, request=None, knob=False, chunk=None): + order = [] if owner is None else owner.order + published = [] + chunk = chunk if chunk is not None else _chunk(draft=[np.array([1, 2])]) + + class _Publish: + def __init__(self, session, req): + pass + + def publish(self, extent): + order.append("publish") + published.append(extent) + + monkeypatch.setattr(transceiver_module, "PeerPublish", _Publish) + transceiver = object.__new__(KvCacheTransceiverV2) + session = MagicMock() + transceiver._enable_pipelined_transfer = False + transceiver._send_reqs = {} + transceiver._get_or_create_send_session = MagicMock(return_value=session) + transceiver._create_cache_extent = MagicMock(return_value=CacheExtent(name=_RID, local=chunk)) + transceiver._finalize_send = MagicMock() + if owner is not None: + _engage(transceiver, owner, knob) + request = request if request is not None else _request() + request.set_kv_cache_transfer_start = lambda _ts: None + KvCacheTransceiverV2.respond_and_send_async(transceiver, request) + session.set_exception.assert_not_called() + transceiver._finalize_send.assert_called_once() + return order, transceiver, request, published[0] + + +def _owner(ctx_len=64, ctx_base=0): + order = [] + owner = _Owner(order, ctx_len, ctx_base) + owner.order = order + return owner + + +def _params(**kwargs): + return DisaggregatedParams(disagg_request_id=_RID, **kwargs) + + +def test_send_without_the_draft_set_publishes_untouched(monkeypatch): + order, _, request, extent = _send(monkeypatch) + assert order == ["publish"] + assert not hasattr(request, "py_draft_ctx_aux") + assert extent.local.draft_block_ids_per_layer_groups is not None # nothing strips it + + +def test_send_waits_on_the_drafter_store_before_publishing(monkeypatch): + owner = _owner(ctx_len=64, ctx_base=0) + order, transceiver, request, extent = _send(monkeypatch, owner=owner) + assert order == ["wait", "publish"] + assert request.py_draft_ctx_aux == (64, 0, True) + assert extent.local.draft_block_ids_per_layer_groups is not None + assert owner.ready == {} + assert transceiver._draft_kv_ctx_counters.sent == 1 + assert not transceiver._draft_kv_ctx_counters.fallback + + +def test_a_reuse_hit_ships_its_base(monkeypatch): + owner = _owner(ctx_len=64, ctx_base=1) # the drafter stored the tail from position 20 + order, transceiver, request, extent = _send( + monkeypatch, + owner=owner, + request=_request(py_disaggregated_params=_params(**_CONTEXT_FIRST)), + ) + assert order == ["wait", "publish"] + assert request.py_draft_ctx_aux == (64, 1, True) + assert extent.local.draft_block_ids_per_layer_groups is not None + assert transceiver._draft_kv_ctx_counters.sent == 1 + + +def test_context_first_state_survives_the_serving_params_round_trip(): + """CTX response -> serving DisaggregatedParams (draft_ctx_state) -> generation request.""" + from tensorrt_llm.serve.openai_protocol import ( + to_disaggregated_params, + to_llm_disaggregated_params, + ) + + for state in ((9000, 256, True), (33, 0, True), (0, 0, False)): + ctx_side = _params( + **_CONTEXT_FIRST, + request_type="context_only", + draft_ctx_state=encode_draft_ctx_state(*state), + ) + wire = to_disaggregated_params(ctx_side) + assert wire.draft_ctx_state == encode_draft_ctx_state(*state) + assert wire.encoded_opaque_state is None # never the C++ transceiver-state carrier + # The serving JSON hop and back. + wire = type(wire).model_validate_json(wire.model_dump_json()) + gen_side = to_llm_disaggregated_params( + wire.model_copy(update={"request_type": "generation_only"}) + ) + assert decode_draft_ctx_state(gen_side.draft_ctx_state) == state + assert gen_side.opaque_state is None + gen_request = _gen_request(gen_side, prompt_len=state[0] or 64) + expected = state[:2] if state[2] else None + assert KvCacheTransceiverV2._received_draft_ctx_state(gen_request) == expected + + +def test_test_knob_publishes_before_the_drafter_store(monkeypatch): + owner = _owner() + order, _, request, _ = _send(monkeypatch, owner=owner, knob=True) + assert order == ["publish"] + assert request.py_draft_ctx_aux == (64, 0, True) + + +@pytest.mark.parametrize( + "ctx_len, ctx_base, draft, reason", + [ + (None, None, [np.array([1, 2])], "no_store_event"), + (0, 0, [np.array([1, 2])], "ctx_len_mismatch"), # the drafter stored nothing + (64, 3, [np.array([1, 2])], "reuse_base"), # a base past the prompt's two pages + (64, 0, [np.array([-1, 2])], "page_mismatch"), # a window page is missing + ], +) +@pytest.mark.parametrize("context_first", [True, False]) +def test_a_window_that_cannot_be_served_falls_back( + monkeypatch, context_first, ctx_len, ctx_base, draft, reason +): + owner = _owner(ctx_len=ctx_len, ctx_base=ctx_base) + params = _params(**_CONTEXT_FIRST) if context_first else None + request = _request(py_disaggregated_params=params) if params else _request() + order, transceiver, request, extent = _send( + monkeypatch, owner=owner, request=request, chunk=_chunk(draft=draft) + ) + + # The request is published, never failed; the target set is untouched. + assert order[-1] == "publish" + assert np.array_equal(extent.local.block_ids_per_layer_groups[0], [3, 4]) + assert request.py_draft_ctx_aux == (0, 0, False) + counters = transceiver._draft_kv_ctx_counters + assert counters.sent == 0 and counters.fallback == {reason: 1} + if context_first: + # The receiver learns it from the response first, so the window's pages are left out. + assert extent.local.draft_block_ids_per_layer_groups is None + else: + # The receiver already asked for the pages: they still travel, marked unusable. + assert extent.local.draft_block_ids_per_layer_groups is draft + + +def test_context_first_response_carries_the_decision(monkeypatch): + transceiver = object.__new__(KvCacheTransceiverV2) + transceiver._send_reqs = {} + transceiver._dp_rank = 0 + transceiver._context_info_endpoint = "tcp://ctx:1" + seen = {} + monkeypatch.setattr( + transceiver_module, "ContextPhaseParams", lambda **kwargs: seen.update(kwargs) or kwargs + ) + request = _request(py_disaggregated_params=_params(**_CONTEXT_FIRST)) + request.py_draft_ctx_aux = (0, 0, False) + KvCacheTransceiverV2._finalize_send(transceiver, request, MagicMock()) + assert seen["opaque_state"] is None # not engaged: today's response + + _engage(transceiver, _owner()) + for state in ((0, 0, False), (64, 0, True)): + request.py_draft_ctx_aux = state + KvCacheTransceiverV2._finalize_send(transceiver, request, MagicMock()) + # Never opaque_state: C++ ContextPhaseParams parses it as a DataTransceiverState and + # re-serializes it, so the drafter state rides DisaggregatedParams.draft_ctx_state. + assert seen["opaque_state"] is None + assert request.py_draft_ctx_aux == state + + gen_first = _request() + gen_first.py_draft_ctx_aux = (64, 0, True) + session = MagicMock() + KvCacheTransceiverV2._finalize_send(transceiver, gen_first, session) + assert seen["opaque_state"] is None # generation-first: the aux slot carries it + session.pack_aux.assert_called_once_with(gen_first) + + +# --------------------------------------------------------------------------- +# The generation side: expected pages and the seed follow the context side's decision +# --------------------------------------------------------------------------- + + +def _gen_request(params, prompt_len=64, **kwargs): + return _request( + py_disaggregated_params=params, + prompt_len=prompt_len, + is_generation_only_request=True, + **kwargs, + ) + + +def test_generation_seeds_the_received_state_or_an_empty_window(): + transceiver = object.__new__(KvCacheTransceiverV2) + transceiver._seed_draft_ctx(_request(py_draft_ctx_aux=(9000, 0, True))) # off: nothing + + owner = _owner() + _engage(transceiver, owner) + counters = transceiver._draft_kv_gen_counters + + # Generation-first: the aux slot. + transceiver._seed_draft_ctx( + _gen_request(_params(**_GENERATION_FIRST), 9000, py_draft_ctx_aux=(9000, 0, True)) + ) + assert owner.seeds[_RID] == (9000, 0, 9000) + transceiver._seed_draft_ctx( + _gen_request(_params(**_GENERATION_FIRST), 9000, py_draft_ctx_aux=(0, 0, False)) + ) + # Not shipped: based past every prompt page, the drafter reads nothing. + assert owner.seeds[_RID] == (9000, 282, 9000) + + # Context-first: the response's draft_ctx_state. + shipped = _params(**_CONTEXT_FIRST, draft_ctx_state=encode_draft_ctx_state(33, 0, True)) + transceiver._seed_draft_ctx(_gen_request(shipped, 33)) + assert owner.seeds[_RID] == (33, 0, 33) + for params in ( + _params(**_CONTEXT_FIRST, draft_ctx_state=encode_draft_ctx_state(0, 0, False)), + _params(**_CONTEXT_FIRST), # no state arrived at all + ): + transceiver._seed_draft_ctx(_gen_request(params, 33)) + assert owner.seeds[_RID] == (33, 2, 33) + + assert counters.sent == 2 + assert counters.fallback == {"no_ctx_state": 3} + + +def test_context_first_generation_expects_no_draft_pages_after_a_fallback(): + transceiver = object.__new__(KvCacheTransceiverV2) + _engage(transceiver, _owner()) + transceiver._draft_reuse_adapter = SimpleNamespace( + tokens_per_block=32, + get_block_ordinals=lambda req, idx, lg: np.array([7, 8], dtype=np.int64), + ) + shipped = _gen_request( + _params(**_CONTEXT_FIRST, draft_ctx_state=encode_draft_ctx_state(64, 0, True)) + ) + dropped = _gen_request( + _params(**_CONTEXT_FIRST, draft_ctx_state=encode_draft_ctx_state(0, 0, False)) + ) + assert np.array_equal(transceiver._describe_draft_local(shipped)[0], [7, 8]) + assert transceiver._describe_draft_local(dropped) is None + # Generation-first asks before the context side decides: always the whole window. + assert transceiver._describe_draft_local(_gen_request(_params(**_GENERATION_FIRST))) is not None + + +# --------------------------------------------------------------------------- +# Receive fences +# --------------------------------------------------------------------------- + + +class _Fence: + def __init__(self, done): + self.done = done + self.synchronized = 0 + + def query(self): + return self.done + + def synchronize(self): + self.synchronized += 1 + + +def _coordinator(manager): + coordinator = object.__new__(DisaggTransferCoordinator) + coordinator._draft_kv_cache_manager = manager + return coordinator + + +def test_fences_are_untouched_without_an_engaged_draft_pool(): + req = SimpleNamespace(py_request_id=1) + target = _Fence(True) + fences = {1: target} + object.__new__(DisaggTransferCoordinator)._merge_draft_receive_fences([req], fences, True) + manager = SimpleNamespace(take_disagg_receive_ready=Mock()) # not engaged + _coordinator(manager)._merge_draft_receive_fences([req], fences, True) + manager.take_disagg_receive_ready.assert_not_called() + assert fences == {1: target} + + +def test_draft_readiness_joins_the_deferred_receive_fence(): + draft_fences = {1: _Fence(False), 2: _Fence(True)} + manager = SimpleNamespace( + _disagg_draft_kv_receive=True, + take_disagg_receive_ready=lambda req: draft_fences.pop(req.py_request_id, None), + ) + target = _Fence(True) + fences = {1: target, 3: None} + reqs = [SimpleNamespace(py_request_id=i) for i in (1, 2, 3)] + _coordinator(manager)._merge_draft_receive_fences(reqs, fences, True) + assert fences[1].query() is False # target done, draft not yet + assert fences[2].query() is True # the draft event alone + assert fences[3] is None + + +def test_draft_readiness_is_waited_on_without_deferral(): + draft = _Fence(False) + manager = SimpleNamespace( + _disagg_draft_kv_receive=True, take_disagg_receive_ready=lambda req: draft + ) + fences = {} + _coordinator(manager)._merge_draft_receive_fences( + [SimpleNamespace(py_request_id=1)], fences, False + ) + assert draft.synchronized == 1 + assert fences == {} + + +# --------------------------------------------------------------------------- +# The draft pool on a generation worker +# --------------------------------------------------------------------------- + + +class _Cache: + def __init__(self, capacity=0, ok=True): + self.capacity = capacity + self.ok = ok + self.calls = [] + + def resize(self, capacity, history_length=None): + self.calls.append((capacity, history_length)) + return self.ok + + +def _draft_manager(monkeypatch): + class _Event: + def record(self, stream=None): + self.stream = stream + + monkeypatch.setattr(torch.cuda, "Event", _Event) + manager = object.__new__(KVCacheManagerV2) + manager.is_draft = True + manager.num_extra_kv_tokens = 1 + manager._stream = "execution-stream" + manager._disagg_receive_ready = {} + manager._sharing = None + manager.kv_cache_map = {} + manager._gpu_max_tokens = 0 + manager.max_attention_window_vec = [4096] # read by the resize-failure diagnostic + manager.tokens_per_block = 32 + return manager + + +def test_gen_init_holds_the_window_and_fences_a_received_one(monkeypatch): + manager = _draft_manager(monkeypatch) + seeded = [] + manager.engage_disagg_gen_init_window(lambda *args: seeded.append(args)) + manager.engage_disagg_draft_kv_receive() + cache = _Cache() + req = SimpleNamespace(py_request_id=9, prompt_len=9000, py_draft_tokens=[0, 0, 0]) + + manager._prepare_disagg_gen_init_draft(req, cache) + + assert cache.calls == [(9000 + 3 + 1, 9000)] + # The empty window at the prompt end, which a received window then replaces. + assert seeded == [(9, 9000, manager.tokens_per_block)] + assert manager._disagg_receive_ready[9].stream == "execution-stream" + assert manager.take_disagg_receive_ready(req) is not None + + +def test_gen_init_without_the_transfer_records_no_fence(monkeypatch): + manager = _draft_manager(monkeypatch) + manager.engage_disagg_gen_init_window(lambda *args: None) + manager._prepare_disagg_gen_init_draft( + SimpleNamespace(py_request_id=9, prompt_len=64, py_draft_tokens=None), _Cache() + ) + assert manager._disagg_receive_ready == {} + + +def test_the_transfer_needs_the_window_sizing_engaged(monkeypatch): + manager = _draft_manager(monkeypatch) + with pytest.raises(AssertionError, match="gen-init window sizing"): + manager.engage_disagg_draft_kv_receive() + + +def test_gen_init_resize_failure_is_loud(monkeypatch): + manager = _draft_manager(monkeypatch) + manager.engage_disagg_gen_init_window(lambda *args: None) + req = SimpleNamespace(py_request_id=9, prompt_len=64, py_draft_tokens=None) + with pytest.raises(RuntimeError, match="disagg gen-init resize failed"): + manager._prepare_disagg_gen_init_draft(req, _Cache(ok=False)) + assert manager._disagg_receive_ready == {} + + +def test_only_a_draft_pool_can_be_engaged(monkeypatch): + manager = _draft_manager(monkeypatch) + manager.is_draft = False + with pytest.raises(AssertionError): + manager.engage_disagg_draft_kv_receive() + assert manager._disagg_draft_kv_receive is False + + +def test_draft_vetoes_ride_the_generation_consensus_and_are_absent_when_off(): + transceiver = object.__new__(KvCacheTransceiverV2) + transceiver._recv_sessions = {1: object(), 2: object()} + transceiver._gen_need_sync = True + seen = [] + + def allgather(local): + seen.append(local) + other = [[], [], [1, 2], [1, 2], [1, 2]] + ([[2]] if len(local) == 6 else []) + return [local, other] + + transceiver._gen_allgather = allgather + # Off: the same four results and the same allgather payload as before. + result = transceiver._gen_consensus_outcome([1, 2], [], [], [1, 2], [1, 2]) + assert len(result) == 4 and len(seen[-1]) == 5 + # On: one more list in the same allgather; a veto on any rank is a veto on all. + result = transceiver._gen_consensus_outcome( + [1, 2], [], [], [1, 2], [1, 2], locally_draft_vetoed=[] + ) + assert len(seen) == 2 and len(seen[-1]) == 6 + cancelled, failed, completed, verified, vetoed = result + assert completed == [1, 2] and verified == {1, 2} and vetoed == {2} diff --git a/tests/unittest/disaggregated/test_disagg_draft_kv_switch.py b/tests/unittest/disaggregated/test_disagg_draft_kv_switch.py new file mode 100644 index 000000000000..d19c07aa3aad --- /dev/null +++ b/tests/unittest/disaggregated/test_disagg_draft_kv_switch.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""The disagg drafter context-window switch. + +The strict environment contract, and the gate: the model's opt-in hook AND a separate draft pool +AND the kill switch unset. +""" + +import inspect +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch.disaggregation import draft_kv +from tensorrt_llm._torch.disaggregation.draft_kv import ( + DISABLE_DISAGG_DRAFT_KV_ENV, + TEST_SEND_BEFORE_EVENT_ENV, + disagg_draft_kv_disabled, + parse_strict_bool_env, + select_disagg_draft_kv_cache_manager, + send_before_draft_ctx_event, +) +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + +pytestmark = pytest.mark.cpu_only + +_TRUE = ["1", "true", "TRUE", "True", "on", "ON", "yes", "Yes"] +_FALSE = ["", "0", "false", "FALSE", "off", "Off", "no", "NO"] +_INVALID = ["2", "-1", "enable", "y", "n", "disable", "truee", "1.0"] + + +def test_names_are_the_documented_variables(): + assert DISABLE_DISAGG_DRAFT_KV_ENV == "TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV" + assert TEST_SEND_BEFORE_EVENT_ENV == "TRTLLM_M31_F0A_TEST_SEND_BEFORE_EVENT" + + +def test_unset_is_not_disabled(): + assert disagg_draft_kv_disabled({}) is False + assert send_before_draft_ctx_event({}) is False + + +@pytest.mark.parametrize("value", _TRUE) +def test_true_values_disable(value): + assert disagg_draft_kv_disabled({DISABLE_DISAGG_DRAFT_KV_ENV: value}) is True + assert send_before_draft_ctx_event({TEST_SEND_BEFORE_EVENT_ENV: value}) is True + + +@pytest.mark.parametrize("value", _FALSE) +def test_false_values_do_not_disable(value): + assert disagg_draft_kv_disabled({DISABLE_DISAGG_DRAFT_KV_ENV: value}) is False + assert send_before_draft_ctx_event({TEST_SEND_BEFORE_EVENT_ENV: value}) is False + + +@pytest.mark.parametrize("value", _INVALID) +def test_anything_else_raises_naming_the_variable(value): + with pytest.raises(ValueError, match=DISABLE_DISAGG_DRAFT_KV_ENV): + disagg_draft_kv_disabled({DISABLE_DISAGG_DRAFT_KV_ENV: value}) + with pytest.raises(ValueError, match=TEST_SEND_BEFORE_EVENT_ENV): + send_before_draft_ctx_event({TEST_SEND_BEFORE_EVENT_ENV: value}) + + +def test_process_environment_is_read(monkeypatch): + monkeypatch.setenv(DISABLE_DISAGG_DRAFT_KV_ENV, "yes") + assert disagg_draft_kv_disabled() is True + monkeypatch.setenv(DISABLE_DISAGG_DRAFT_KV_ENV, "off") + assert disagg_draft_kv_disabled() is False + monkeypatch.setenv(DISABLE_DISAGG_DRAFT_KV_ENV, "maybe") + with pytest.raises(ValueError, match=DISABLE_DISAGG_DRAFT_KV_ENV): + disagg_draft_kv_disabled() + + +def test_parse_is_shared_code_not_the_targets_switches(): + # The M3.1 targets' m31_switches treats any non-empty, non-"0" value as set; the shared + # parse must not import it (frozen M3 and every other model never load a target module). + assert "m31_switches" not in inspect.getsource(draft_kv) + assert parse_strict_bool_env("X", {"X": " On "}) is True + + +class _Worker: + def engage_disagg_draft_kv(self): + raise AssertionError("selection must not engage the worker") + + +def _model(hook: bool, worker=None): + model = SimpleNamespace(spec_worker=_Worker() if worker is None else worker) + if hook: + model.runtime_hooks = RuntimeHooks(disagg_draft_kv=True) + return model + + +def _draft_manager(joint_reuse=False): + return SimpleNamespace(enable_joint_kv_cache_reuse=joint_reuse) + + +def test_hook_defaults_off(): + assert RuntimeHooks().disagg_draft_kv is False + + +def test_selected_only_with_hook_and_separate_pool(monkeypatch): + monkeypatch.delenv(DISABLE_DISAGG_DRAFT_KV_ENV, raising=False) + manager = _draft_manager() + assert select_disagg_draft_kv_cache_manager(_model(hook=True), manager) is manager + # Every other model: no hook (frozen M3, all other DFlash / DSpark deployments). + assert select_disagg_draft_kv_cache_manager(_model(hook=False), manager) is None + assert select_disagg_draft_kv_cache_manager(SimpleNamespace(), manager) is None + # No separate draft pool (one-model Eagle3 sharing the target's). + assert select_disagg_draft_kv_cache_manager(_model(hook=True), None) is None + # A worker with no paged drafter context to ship. + no_ctx = SimpleNamespace(runtime_hooks=RuntimeHooks(disagg_draft_kv=True), spec_worker=object()) + assert select_disagg_draft_kv_cache_manager(no_ctx, manager) is None + + +@pytest.mark.parametrize("value", ["1", "true", "on", "yes"]) +def test_kill_switch_wins(monkeypatch, value): + monkeypatch.setenv(DISABLE_DISAGG_DRAFT_KV_ENV, value) + assert select_disagg_draft_kv_cache_manager(_model(hook=True), _draft_manager()) is None + + +def test_invalid_kill_switch_value_fails_selection(monkeypatch): + monkeypatch.setenv(DISABLE_DISAGG_DRAFT_KV_ENV, "please") + with pytest.raises(ValueError, match=DISABLE_DISAGG_DRAFT_KV_ENV): + select_disagg_draft_kv_cache_manager(_model(hook=True), _draft_manager()) + + +def test_joint_reuse_is_refused(monkeypatch): + monkeypatch.delenv(DISABLE_DISAGG_DRAFT_KV_ENV, raising=False) + with pytest.raises(ValueError, match="enable_joint_kv_cache_reuse"): + select_disagg_draft_kv_cache_manager(_model(hook=True), _draft_manager(joint_reuse=True)) diff --git a/tests/unittest/disaggregated/test_disagg_draft_kv_targets.py b/tests/unittest/disaggregated/test_disagg_draft_kv_targets.py new file mode 100644 index 000000000000..8a2066af663f --- /dev/null +++ b/tests/unittest/disaggregated/test_disagg_draft_kv_targets.py @@ -0,0 +1,53 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""The MiniMax-M3.1 sm_107 targets opt in to the disagg drafter context window. + +Each target's ``RuntimeHooks`` is read from its modeling.py source (the targets need SM107 kernels +to import), so the opt-in is checked on any machine. +""" + +import ast +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family + +pytestmark = pytest.mark.cpu_only + +_FAMILY = Path(_family.__file__).resolve().parent +_SM107_TARGETS = ("tp2", "tp4", "tp8") + + +def _runtime_hooks_keywords(modeling: Path) -> dict: + tree = ast.parse(modeling.read_text(), filename=str(modeling)) + calls = [ + node + for node in ast.walk(tree) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "RuntimeHooks" + ] + assert len(calls) == 1, f"{modeling}: expected one RuntimeHooks(...) call, found {len(calls)}" + return {kw.arg: kw.value for kw in calls[0].keywords} + + +@pytest.mark.parametrize("tp", _SM107_TARGETS) +def test_sm107_target_opts_in_to_the_draft_window_transfer(tp): + modeling = _FAMILY / f"minimax_m31_nvfp4__sm_107__{tp}" / "modeling.py" + keywords = _runtime_hooks_keywords(modeling) + assert "disagg_draft_kv" in keywords, f"{modeling.parent.name} does not set disagg_draft_kv" + value = keywords["disagg_draft_kv"] + assert isinstance(value, ast.Constant) and value.value is True diff --git a/tests/unittest/disaggregated/test_disagg_draft_kv_transfer.py b/tests/unittest/disaggregated/test_disagg_draft_kv_transfer.py new file mode 100644 index 000000000000..431da5162a7d --- /dev/null +++ b/tests/unittest/disaggregated/test_disagg_draft_kv_transfer.py @@ -0,0 +1,1782 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Loopback of the standalone drafter's context window (F0a), at unit level. + +A context rank with one TP rank per attention-DP group holds all four drafter KV heads; generation +TP4 holds one head per rank (head r) and TP8 one duplicated head per rank (head r // 2). The real +rank-info serialization, peer registrars, extractors, head-mismatch mappers and +``Sender._build_kv_write_meta`` build the write; the write is then applied with host memcpy to +host-memory pools laid out like the V2 HND pools, so no NIXL agent or GPU is needed. +""" + +import ctypes +import importlib.util +import json +from collections import Counter +from dataclasses import replace +from pathlib import Path +from types import SimpleNamespace + +import msgpack +import numpy as np +import pytest +import torch + +from tensorrt_llm import DisaggregatedParams +from tensorrt_llm._torch.disaggregation.base import CacheExtent, CacheKind, Chunk, TokenRange +from tensorrt_llm._torch.disaggregation.draft_kv import ( + DUMP_DIR_ENV, + DraftKvCounters, + draft_dump_dir, + draft_receive_disabled_reason, + encode_draft_ctx_state, +) +from tensorrt_llm._torch.disaggregation.native.mixers.attention.spec import AttentionInfo +from tensorrt_llm._torch.disaggregation.native.peer import PeerOverlap, PeerRegistrar +from tensorrt_llm._torch.disaggregation.native.rank_info import RankInfo +from tensorrt_llm._torch.disaggregation.native.transfer import ( + KVSendTask, + Receiver, + RecvReqInfo, + Sender, + aux_meta_without_draft_ctx, + check_draft_kv_peer, + draft_peer_mismatch, + draft_peer_not_engaged, +) +from tensorrt_llm._torch.disaggregation.resource.kv_extractor import KVRegionExtractorV1 +from tensorrt_llm._torch.disaggregation.resource.page import ( + BUFFER_ENTRY_DTYPE, + AttentionLayerGroup, + KVCachePageTable, + LocalLayer, + MapperKind, + PhysicalPool, + PhysicalPoolGroup, + PoolView, +) +from tensorrt_llm._torch.disaggregation.transceiver import KvCacheTransceiverV2 +from tensorrt_llm.disaggregated_params import DisaggScheduleStyle + +pytestmark = pytest.mark.cpu_only + +# Target and draft pools deliberately share role labels and global layer ids: one pool mapping +# would cross-match them, so the draft set must go through its own registrar. +_ROLES = frozenset({"key", "value"}) +_INDEX_ROLES = frozenset({"index_key"}) +_LAYER_IDS = [0, 1] +_RID = 4242 +_SENTINEL = -1 + +_TARGET_TPB = 64 +_TARGET_DIM = 8 +_TARGET_HEADS = 8 # all heads on the context rank (attention DP) +_DRAFT_TPB = 32 +_DRAFT_DIM = 16 +_DRAFT_HEADS = 4 # the drafter's KV heads, all on the context rank (attention DP) +_DRAFT_WINDOW = 2048 +# The context side's last prefill chunk starts this far before the prompt end: its windowed draft +# pool still holds the window behind that chunk start, a superset of what generation keeps. +_CTX_LAST_CHUNK = 4096 + +PROMPT_LENS = [31, 32, 33, 4095, 4097, 9000] + + +def _blocks(tokens: int, tpb: int) -> int: + return (tokens + tpb - 1) // tpb + + +def _window_start_block(history: int, tpb: int) -> int: + """First ordinal a windowed pool still holds at ``history`` (released below it).""" + return max(0, (history + 1 - _DRAFT_WINDOW) // tpb) + + +def _value(layer, kv, head, pos, dim): + """The K/V the context side stored: f(layer, K/V, global head, position, dim) as int16.""" + return ((layer * 2 + kv) * 1009 + head * 131 + pos * 7 + dim * 3) % 30011 + + +def _index_value(layer, pos, dim): + """The MSA index key: one vector per token, the same on every TP rank (replicated weights).""" + return ((layer + 3) * 977 + pos * 5 + dim * 11) % 30011 + + +class _Pool: + """A host-memory paged pool laid out like a V2 HND attention pool. + + Shape ``[slot, layer, K/V, head, token, dim]``: one slot holds every layer's K then V buffer, + each buffer head-major. ``head_offset`` is the global index of this rank's first head. + ``index_dim`` > 0 adds MiniMax M3's MSA index-K cache as a second physical pool, one + ``[token, index_dim]`` vector per layer and block, declared REPLICATED. + """ + + def __init__( + self, + num_slots: int, + heads: int, + tpb: int, + head_dim: int, + window=None, + head_offset: int = 0, + index_dim: int = 0, + dtype=np.int16, + ): + self.window = window + self.head_offset = head_offset + self.array = np.full( + (num_slots, len(_LAYER_IDS), 2, heads, tpb, head_dim), _SENTINEL, dtype=dtype + ) + self.index = ( + np.full((num_slots, len(_LAYER_IDS), tpb, index_dim), _SENTINEL, dtype=np.int16) + if index_dim + else None + ) + + @property + def heads(self) -> int: + return self.array.shape[3] + + @property + def tpb(self) -> int: + return self.array.shape[4] + + def page_table(self) -> KVCachePageTable: + num_slots, layers, _, heads, tpb, dim = self.array.shape + buf = heads * tpb * dim * self.array.itemsize + entries = [(i, (2 * i + kv) * buf, buf) for i in range(layers) for kv in range(2)] + view = PoolView( + pool_idx=0, + buffer_entries=np.array(entries, dtype=BUFFER_ENTRY_DTYPE), + pool_role=_ROLES, + mapper_kind=MapperKind.INDEXED, + bytes_per_layer=2 * buf, + ) + views = [view] + pools = [ + PhysicalPool( + base_address=self.array.ctypes.data, + slot_bytes=self.array[0].nbytes, + num_slots=num_slots, + ) + ] + if self.index is not None: + index_buf = self.index[0, 0].nbytes + views.append( + PoolView( + pool_idx=1, + buffer_entries=np.array( + [(i, i * index_buf, index_buf) for i in range(layers)], + dtype=BUFFER_ENTRY_DTYPE, + ), + pool_role=_INDEX_ROLES, + mapper_kind=MapperKind.REPLICATED, + bytes_per_layer=index_buf, + ) + ) + pools.append( + PhysicalPool( + base_address=self.index.ctypes.data, + slot_bytes=self.index[0].nbytes, + num_slots=num_slots, + ) + ) + group = AttentionLayerGroup( + pool_group_idx=0, + kv_head_num_per_rank=heads, + sliding_window_size=self.window, + local_layers=[LocalLayer(i, g) for i, g in enumerate(_LAYER_IDS)], + pool_views=views, + ) + return KVCachePageTable( + tokens_per_block=tpb, + layer_groups=[group], + pool_groups=[PhysicalPoolGroup(pools=pools)], + ) + + def fill(self, table: np.ndarray) -> None: + """Store f(...) at every block the positional table holds.""" + _, layers, _, heads, tpb, dim = self.array.shape + layer, kv, h, t, d = np.indices((layers, 2, heads, tpb, dim)) + for ordinal, slot in enumerate(table): + if slot >= 0: + self.array[slot] = _value(layer, kv, h + self.head_offset, ordinal * tpb + t, d) + if self.index is not None: + il, it, idim = np.indices(self.index.shape[1:]) + self.index[slot] = _index_value(il, ordinal * tpb + it, idim) + + +def _attention(pool: "_Pool", adp: bool) -> AttentionInfo: + return AttentionInfo( + kv_heads_per_rank=pool.heads, + tokens_per_block=pool.tpb, + dims_per_head=pool.array.shape[-1], + element_bytes=pool.array.itemsize, + enable_attention_dp=adp, + is_mla=False, + ) + + +def _rank_info(name, rank, *, tp_size, tp_rank, dp_size, adp, target, draft) -> RankInfo: + ri = RankInfo( + instance_name=name, + instance_rank=rank, + tp_size=tp_size, + tp_rank=tp_rank, + pp_size=1, + pp_rank=0, + layer_num_per_pp=[len(_LAYER_IDS)], + sender_endpoints=[], + self_endpoint="", + transfer_engine_info=b"", + dp_size=dp_size, + dp_rank=tp_rank if adp else 0, + attention=_attention(target, adp), + page_table=target.page_table(), + ) + if draft is not None: + ri.draft_attention = _attention(draft, adp) + ri.draft_page_table = draft.page_table() + return ri + + +def _positional(num_blocks: int, first_held: int, slot_of) -> np.ndarray: + table = np.full(num_blocks, -1, dtype=np.int64) + for ordinal in range(first_held, num_blocks): + table[ordinal] = slot_of(ordinal) + return table + + +class _Context: + """One context rank of CTX TP2. + + ``ctx_tp`` 1 is the attention-DP layout: one TP rank per DP group holds every head (rank 0 + modelled). ``ctx_tp`` > 1 is the TP-attention layout: rank ``ctx_rank`` holds its slice of + every request's heads, ``heads // ctx_tp`` of the target's and of the drafter's. + """ + + def __init__( + self, + prompt_len: int, + with_draft: bool = True, + reuse_prefix: int = 0, + ctx_tp: int = 1, + ctx_rank: int = 0, + index_dim: int = 0, + ): + """Model one context rank's pools after its prefill. + + ``reuse_prefix`` > 0 is a prefix-reuse hit (turn 2 of a conversation): the target prefill + starts there, so the unpaired draft pool's history is that position and the drafter + stored only the tail ``[reuse_prefix, prompt_len)``. + """ + self.prompt_len = prompt_len + self.ctx_tp = ctx_tp + self.ctx_rank = ctx_rank + n_target = _blocks(prompt_len, _TARGET_TPB) + n_draft = _blocks(prompt_len, _DRAFT_TPB) + target_heads = _TARGET_HEADS // ctx_tp + draft_heads = _DRAFT_HEADS // ctx_tp + self.target = _Pool( + n_target + 5, + target_heads, + _TARGET_TPB, + _TARGET_DIM, + head_offset=ctx_rank * target_heads, + index_dim=index_dim, + ) + self.draft = _Pool( + n_draft + 7, + draft_heads, + _DRAFT_TPB, + _DRAFT_DIM, + _DRAFT_WINDOW, + head_offset=ctx_rank * draft_heads, + ) + self.target_table = _positional(n_target, 0, lambda o: o + 5) + last_chunk_start = max(reuse_prefix, prompt_len - _CTX_LAST_CHUNK) + self.draft_table = _positional( + n_draft, _window_start_block(last_chunk_start, _DRAFT_TPB), lambda o: o + 7 + ) + self.target.fill(self.target_table) + self.draft.fill(self.draft_table) + if reuse_prefix: + # Positions below the reused prefix were never stored by the drafter. + for ordinal, slot in enumerate(self.draft_table): + low = ordinal * _DRAFT_TPB + if slot >= 0 and low < reuse_prefix: + self.draft.array[slot][:, :, :, : min(reuse_prefix - low, _DRAFT_TPB)] = ( + _SENTINEL + ) + adp = ctx_tp == 1 + self.rank_info = _rank_info( + "ctx", + ctx_rank, + tp_size=2, + tp_rank=ctx_rank, + dp_size=2 if adp else 1, + adp=adp, + target=self.target, + draft=self.draft if with_draft else None, + ) + self.sender = Sender.__new__(Sender) + self.sender._registrar = PeerRegistrar( + self.rank_info, KVRegionExtractorV1(self.rank_info.page_table) + ) + if with_draft: + draft_ri = self.rank_info.draft_view() + self.sender._draft_registrar = PeerRegistrar( + draft_ri, KVRegionExtractorV1(draft_ri.page_table) + ) + + def register(self, gen_ri: RankInfo) -> None: + """What the sender's REGISTER_RANK_INFO handler does, minus the transfer agent.""" + gen_ri = RankInfo.from_bytes(gen_ri.to_bytes()) # as the wire delivers it + peer_draft_ri = check_draft_kv_peer(self.sender._draft_registrar, gen_ri) + self.sender._registrar.register(gen_ri.instance_name, gen_ri.instance_rank, gen_ri) + if peer_draft_ri is not None: + self.sender._register_draft_peer(gen_ri, peer_draft_ri) + else: + self.sender._note_draft_peer_not_engaged(gen_ri) + + def chunk(self, *, with_draft: bool = True, is_last: bool = True) -> Chunk: + return Chunk( + block_ids_per_layer_groups=[self.target_table], + kind_per_layer_group=[CacheKind.PAGED], + token_range=TokenRange(start=0, end=self.prompt_len), + is_last=is_last, + draft_block_ids_per_layer_groups=[self.draft_table] if with_draft else None, + ) + + def task(self, *, with_draft: bool = True, is_last: bool = True, chunk=None) -> KVSendTask: + return KVSendTask( + chunk if chunk is not None else self.chunk(with_draft=with_draft, is_last=is_last), + DisaggregatedParams(disagg_request_id=_RID), + slice_id=0, + prompt_len=self.prompt_len, + ) + + +class _Generation: + """One generation TP rank (no attention DP): 1 drafter head, target heads split by TP.""" + + def __init__( + self, + prompt_len: int, + tp_size: int, + tp_rank: int, + with_draft: bool = True, + index_dim: int = 0, + draft_dim: int = _DRAFT_DIM, + draft_dtype=np.int16, + ): + n_target = _blocks(prompt_len, _TARGET_TPB) + n_draft = _blocks(prompt_len, _DRAFT_TPB) + self.target = _Pool( + n_target + 2, + max(1, _TARGET_HEADS // tp_size), + _TARGET_TPB, + _TARGET_DIM, + index_dim=index_dim, + ) + self.draft = _Pool( + n_draft + 3, + max(1, _DRAFT_HEADS // tp_size), + _DRAFT_TPB, + draft_dim, + _DRAFT_WINDOW, + dtype=draft_dtype, + ) + # Reversed slot order, so a pointer-arithmetic slip cannot pass by accident. + self.target_table = _positional(n_target, 0, lambda o: n_target + 1 - o) + self.draft_table = _positional( + n_draft, _window_start_block(prompt_len, _DRAFT_TPB), lambda o: n_draft + 2 - o + ) + self.rank_info = _rank_info( + "gen", + tp_rank, + tp_size=tp_size, + tp_rank=tp_rank, + dp_size=1, + adp=False, + target=self.target, + draft=self.draft if with_draft else None, + ) + + def chunk(self, prompt_len: int, *, with_draft: bool = True) -> Chunk: + return Chunk( + block_ids_per_layer_groups=[self.target_table], + kind_per_layer_group=[CacheKind.PAGED], + token_range=TokenRange(start=0, end=prompt_len), + is_last=True, + draft_block_ids_per_layer_groups=[self.draft_table] if with_draft else None, + ) + + def req_info(self, *, with_draft: bool = True, bounce_dst_base=None) -> RecvReqInfo: + info = RecvReqInfo( + sender_req_id=_RID, + instance_name=self.rank_info.instance_name, + instance_rank=self.rank_info.instance_rank, + block_ids_per_layer_groups=[self.target_table], + unique_rid=_RID, + bounce_dst_base=bounce_dst_base, + draft_block_ids_per_layer_groups=[self.draft_table] if with_draft else None, + ) + return RecvReqInfo.from_bytes(info.to_bytes()) # as the wire delivers it + + +def _apply(write_meta) -> None: + for src, dst, size in zip(write_meta.src_ptrs, write_meta.dst_ptrs, write_meta.sizes): + ctypes.memmove(int(dst), int(src), int(size)) + + +def _chunk_num_bytes(gen: _Generation, chunk: Chunk) -> int: + transceiver = object.__new__(KvCacheTransceiverV2) + transceiver._page_table = gen.rank_info.page_table + transceiver._draft_page_table = gen.rank_info.draft_page_table + return transceiver._chunk_num_bytes(chunk) + + +@pytest.mark.parametrize("prompt_len", PROMPT_LENS) +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_draft_window_lands_head_sliced_on_every_generation_rank(gen_tp, prompt_len): + ctx = _Context(prompt_len) + for rank in range(gen_tp): + gen = _Generation(prompt_len, gen_tp, rank) + ctx.register(gen.rank_info) + + write_meta = ctx.sender._build_kv_write_meta(ctx.task(), gen.req_info()) + _apply(write_meta) + + # The fan-out reaches every generation rank of the DP group, target and draft alike. + assert write_meta.expected_transfers == gen_tp + head = rank if gen_tp == 4 else rank // 2 # duplicate_kv_weight at TP8 + common = (ctx.draft_table >= 0) & (gen.draft_table >= 0) + assert common.any() + for ordinal in np.flatnonzero(common): + received = gen.draft.array[gen.draft_table[ordinal]] + stored = ctx.draft.array[ctx.draft_table[ordinal]][:, :, head : head + 1] + assert np.array_equal(received, stored), f"ordinal {ordinal}, rank {rank}" + layer, kv, _, t, d = np.indices(received.shape) + assert np.array_equal(received, _value(layer, kv, head, ordinal * _DRAFT_TPB + t, d)) + # Released or never-held pages, and every slot no ordinal names, are untouched. + written = {int(gen.draft_table[o]) for o in np.flatnonzero(common)} + for slot in range(gen.draft.array.shape[0]): + if slot not in written: + assert (gen.draft.array[slot] == _SENTINEL).all(), f"slot {slot} was written" + + # Exact byte accounting: the receiver's expected-write total is what landed. + assert _chunk_num_bytes(gen, gen.chunk(prompt_len)) == int(write_meta.sizes.sum()) + + +@pytest.mark.parametrize("prompt_len", PROMPT_LENS) +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_target_fragments_are_unchanged_by_the_draft_set(gen_tp, prompt_len): + rank = gen_tp - 1 + on, off = _Context(prompt_len), _Context(prompt_len, with_draft=False) + gen_on = _Generation(prompt_len, gen_tp, rank) + gen_off = _Generation(prompt_len, gen_tp, rank, with_draft=False) + on.register(gen_on.rank_info) + off.register(gen_off.rank_info) + + meta_on = on.sender._build_kv_write_meta(on.task(), gen_on.req_info()) + meta_off = off.sender._build_kv_write_meta( + off.task(with_draft=False), gen_off.req_info(with_draft=False) + ) + + # The target fragments come first and are the feature-off fragments, rebased onto the other + # pools' (distinct host) addresses; the draft fragments follow, all inside the draft pools. + n = meta_off.src_ptrs.size + assert meta_on.src_ptrs.size > n + rebase_src = on.rank_info.page_table.pool_groups[0].pools[0].base_address - ( + off.rank_info.page_table.pool_groups[0].pools[0].base_address + ) + rebase_dst = gen_on.target.array.ctypes.data - gen_off.target.array.ctypes.data + assert np.array_equal(meta_on.src_ptrs[:n], meta_off.src_ptrs + rebase_src) + assert np.array_equal(meta_on.dst_ptrs[:n], meta_off.dst_ptrs + rebase_dst) + assert np.array_equal(meta_on.sizes[:n], meta_off.sizes) + assert meta_on.expected_transfers == meta_off.expected_transfers + draft_lo = gen_on.draft.array.ctypes.data + draft_hi = draft_lo + gen_on.draft.array.nbytes + assert ((meta_on.dst_ptrs[n:] >= draft_lo) & (meta_on.dst_ptrs[n:] < draft_hi)).all() + + _apply(meta_on) + _apply(meta_off) + assert np.array_equal(gen_on.target.array, gen_off.target.array) + assert _chunk_num_bytes(gen_off, gen_off.chunk(prompt_len, with_draft=False)) == int( + meta_off.sizes.sum() + ) + + +def test_draft_set_ships_only_with_the_last_piece(): + ctx = _Context(4097) + gen = _Generation(4097, 4, 1) + ctx.register(gen.rank_info) + last = ctx.sender._build_kv_write_meta(ctx.task(), gen.req_info()) + middle = ctx.sender._build_kv_write_meta(ctx.task(is_last=False), gen.req_info()) + + lo = gen.draft.array.ctypes.data + hi = lo + gen.draft.array.nbytes + + def into_draft_pool(write_meta): + return (write_meta.dst_ptrs >= lo) & (write_meta.dst_ptrs < hi) + + assert into_draft_pool(last).any() + assert not into_draft_pool(middle).any() + assert middle.src_ptrs.size == int((~into_draft_pool(last)).sum()) + + +def _target_only_meta(prompt_len, gen_tp, rank, **req_kwargs): + off = _Context(prompt_len, with_draft=False) + gen_off = _Generation(prompt_len, gen_tp, rank, with_draft=False) + off.register(gen_off.rank_info) + return ( + off, + gen_off, + off.sender._build_kv_write_meta( + off.task(with_draft=False), gen_off.req_info(with_draft=False, **req_kwargs) + ), + ) + + +def _same_target_write(meta, ref, ctx, ref_ctx, gen, ref_gen): + rebase_src = ctx.target.array.ctypes.data - ref_ctx.target.array.ctypes.data + rebase_dst = gen.target.array.ctypes.data - ref_gen.target.array.ctypes.data + return ( + np.array_equal(meta.src_ptrs, ref.src_ptrs + rebase_src) + and np.array_equal(meta.dst_ptrs, ref.dst_ptrs + rebase_dst) + and np.array_equal(meta.sizes, ref.sizes) + and meta.expected_transfers == ref.expected_transfers + ) + + +def test_a_bounce_reservation_leaves_the_window_out_not_the_request(): + ctx = _Context(4097) + gen = _Generation(4097, 4, 0) + ctx.register(gen.rank_info) + meta = ctx.sender._build_kv_write_meta(ctx.task(), gen.req_info(bounce_dst_base=0x1000)) + ref_ctx, ref_gen, ref = _target_only_meta(4097, 4, 0, bounce_dst_base=0x1000) + assert _same_target_write(meta, ref, ctx, ref_ctx, gen, ref_gen) + assert meta.bounce_dst_base == 0x1000 + + +def test_an_overlap_mismatch_leaves_the_window_out_not_the_request(monkeypatch): + ctx = _Context(4097) + gen = _Generation(4097, 4, 0) + ctx.register(gen.rank_info) + monkeypatch.setattr( + ctx.sender._draft_registrar, + "get_peer_overlap", + lambda *_args: PeerOverlap(ranks=[0, 1]), + ) + meta = ctx.sender._build_kv_write_meta(ctx.task(), gen.req_info()) + ref_ctx, ref_gen, ref = _target_only_meta(4097, 4, 0) + assert _same_target_write(meta, ref, ctx, ref_ctx, gen, ref_gen) + + +@pytest.mark.parametrize("ctx_has_draft", [True, False]) +def test_one_sided_draft_set_registers_the_target_and_disables_the_window(ctx_has_draft): + ctx = _Context(64, with_draft=ctx_has_draft) + gen = _Generation(64, 4, 0, with_draft=not ctx_has_draft) + ctx.register(gen.rank_info) # never raises + assert ctx.sender._registrar.get_peer_rank_info("gen", 0) is not None + assert ctx.sender._draft_disabled_peers["gen0"][0] == "peer_not_engaged" + assert check_draft_kv_peer(getattr(ctx.sender, "_draft_registrar", None), gen.rank_info) is None + assert draft_peer_not_engaged(None, gen.rank_info if not ctx_has_draft else ctx.rank_info) + + +def test_receiver_draft_set_without_a_sender_draft_registrar_sends_the_target_only(): + """A receiver that still asks for the window gets the target set only, not a failure. + + (It never registered the sender's verdict.) + """ + prompt_len = 4097 + ctx = _Context(prompt_len, with_draft=False) + gen = _Generation(prompt_len, 4, 0, with_draft=False) + ctx.register(gen.rank_info) + counters = DraftKvCounters("context") + ctx.sender._draft_kv_counters = counters + meta = ctx.sender._build_kv_write_meta(ctx.task(with_draft=False), gen.req_info()) + ref_ctx, ref_gen, ref = _target_only_meta(prompt_len, 4, 0) + assert _same_target_write(meta, ref, ctx, ref_ctx, gen, ref_gen) + assert counters.fallback == {"peer_not_engaged": 1} + + +def test_target_only_rank_info_has_no_draft_view(): + gen = _Generation(64, 4, 0, with_draft=False) + assert gen.rank_info.draft_view() is None + with_draft = _Generation(64, 4, 0).rank_info + view = with_draft.draft_view() + assert view.attention == with_draft.draft_attention + assert view.page_table is with_draft.draft_page_table + assert view.draft_view() is None + assert replace(view, draft_attention=None) == view + assert (view.tp_size, view.dp_size, view.tp_rank) == ( + with_draft.tp_size, + with_draft.dp_size, + with_draft.tp_rank, + ) + + +@pytest.mark.parametrize("prompt_len", [33, 4097, 100_013]) +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_hook_off_write_meta_is_the_target_only_write(gen_tp, prompt_len): + """With no draft set on either side the write is exactly the target pool pair's fragments. + + That is also the kill-switch case (TRTLLM_M31_DISABLE_DISAGG_DRAFT_KV): neither rank carries a + draft set, including on a 100K prompt whose draft mirror the generation side window-sizes. + """ + rank = gen_tp - 1 + ctx = _Context(prompt_len, with_draft=False) + gen = _Generation(prompt_len, gen_tp, rank, with_draft=False) + ctx.register(gen.rank_info) + info = gen.req_info(with_draft=False) + assert info.draft_block_ids_per_layer_groups is None + + write_meta = ctx.sender._build_kv_write_meta(ctx.task(with_draft=False), info) + + registrar = ctx.sender._registrar + peer = registrar.get_peer_rank_info("gen", rank) + targets = registrar.get_peer_overlap(peer, peer.dp_rank) + ((self_key, peer_key),) = registrar.get_pool_mapping(peer).items() + assert registrar.should_send_pool(targets, peer, *self_key) + src, dst = Sender._pair_ordinals(ctx.target_table, gen.target_table) + pair = registrar.get_kv_map(peer, self_key, peer_key).map( + registrar.self_extractor.extract(src, *self_key), + registrar.peer_extractor("gen", rank).extract(dst, *peer_key), + ) + assert np.array_equal(write_meta.src_ptrs, pair.src.memory.ptrs) + assert np.array_equal(write_meta.dst_ptrs, pair.dst.memory.ptrs) + assert (write_meta.sizes == pair.src.memory.bytes_per_region).all() + assert write_meta.expected_transfers == len(targets.ranks) == gen_tp + + +# --------------------------------------------------------------------------- +# The context side's per-request decision, end to end through the write +# --------------------------------------------------------------------------- + + +class _StoreEvent: + def synchronize(self): + pass + + +class _Worker: + """The drafter state the context side's forward left for the request, and the seeds.""" + + def __init__(self, ctx_len=None, ctx_base=0): + self.ready = {} if ctx_len is None else {_RID: (_StoreEvent(), ctx_len, ctx_base)} + self.seeds = {} + + def take_disagg_draft_ctx_ready(self, request_id): + return self.ready.pop(request_id, None) + + def seed_disagg_draft_ctx(self, request_id, ctx_len, ctx_base, prompt_len, on_seed=None): + self.seeds[request_id] = (ctx_len, ctx_base, prompt_len) + self.hooks = getattr(self, "hooks", {}) + if on_seed is not None: + self.hooks[request_id] = on_seed + + def first_step(self, request_id): + """What DFlashSpecMetadata.prepare does once the request holds its slot.""" + hook = getattr(self, "hooks", {}).pop(request_id, None) + if hook is not None: + hook() + + +def _transceiver(rank_info, worker) -> KvCacheTransceiverV2: + transceiver = object.__new__(KvCacheTransceiverV2) + transceiver._draft_ctx_owner = worker + transceiver._send_before_draft_ctx_event = False + transceiver._draft_page_table = rank_info.draft_page_table + transceiver._draft_kv_ctx_counters = DraftKvCounters("context") + transceiver._draft_kv_gen_counters = DraftKvCounters("generation") + return transceiver + + +def _request(prompt_len, style, **kwargs): + return SimpleNamespace( + py_request_id=_RID, + prompt_len=prompt_len, + is_generation_only_request=False, + py_disaggregated_params=DisaggregatedParams(disagg_request_id=_RID, schedule_style=style), + **kwargs, + ) + + +def _context_decision(ctx, worker, style): + transceiver = _transceiver(ctx.rank_info, worker) + request = _request(ctx.prompt_len, style) + extent = transceiver._await_draft_ctx(request, CacheExtent(name=_RID, local=ctx.chunk())) + return transceiver, request, extent.local + + +def _generation_request(prompt_len, style, ctx_request): + """The generation request as the context side's response (or aux slot) delivers it.""" + if style == DisaggScheduleStyle.CONTEXT_FIRST: + params = DisaggregatedParams( + disagg_request_id=_RID, + schedule_style=style, + draft_ctx_state=encode_draft_ctx_state(*ctx_request.py_draft_ctx_aux), + ) + return SimpleNamespace( + py_request_id=_RID, prompt_len=prompt_len, py_disaggregated_params=params + ) + params = DisaggregatedParams(disagg_request_id=_RID, schedule_style=style) + return SimpleNamespace( + py_request_id=_RID, + prompt_len=prompt_len, + py_disaggregated_params=params, + py_draft_ctx_aux=ctx_request.py_draft_ctx_aux, + ) + + +@pytest.mark.parametrize("style", list(DisaggScheduleStyle)) +@pytest.mark.parametrize("prompt_len, reuse_prefix", [(33, 16), (4097, 1000), (9000, 8200)]) +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_reuse_hit_window_lands_bitwise_with_its_base(gen_tp, prompt_len, reuse_prefix, style): + """Context-side reuse hit: the draft window and its base travel, rows < ctx_len match. + + Turn 2 of a conversation: the target reuses a cached prefix, so the unpaired draft pool + stored only the tail from the reused position on and the drafter reads past + ``ctx_base = ceil(reuse_prefix / page)`` pages. The generation side receives the window's + pages bitwise (head-sliced) and seeds the same ``(ctx_len, ctx_base)``, so every row its + block decode reads, ``[max(base * page, prompt_len - window + 1), prompt_len)``, is the + context side's. + """ + ctx = _Context(prompt_len, reuse_prefix=reuse_prefix) + base = _blocks(reuse_prefix, _DRAFT_TPB) + transceiver, ctx_request, chunk = _context_decision(ctx, _Worker(prompt_len, base), style) + assert transceiver._draft_kv_ctx_counters.sent == 1 + assert not transceiver._draft_kv_ctx_counters.fallback + assert ctx_request.py_draft_ctx_aux == (prompt_len, base, True) + assert chunk.draft_block_ids_per_layer_groups is not None + + gen_request = _generation_request(prompt_len, style, ctx_request) + assert KvCacheTransceiverV2._received_draft_ctx_state(gen_request) == (prompt_len, base) + for rank in range(gen_tp): + gen = _Generation(prompt_len, gen_tp, rank) + ctx.register(gen.rank_info) + meta = ctx.sender._build_kv_write_meta(ctx.task(chunk=chunk), gen.req_info()) + _apply(meta) + head = rank if gen_tp == 4 else rank // 2 + common = (ctx.draft_table >= 0) & (gen.draft_table >= 0) + assert (gen.draft_table >= 0).sum() == common.sum() # the context side covers it all + for ordinal in np.flatnonzero(common): + received = gen.draft.array[gen.draft_table[ordinal]] + stored = ctx.draft.array[ctx.draft_table[ordinal]][:, :, head : head + 1] + assert np.array_equal(received, stored), f"ordinal {ordinal}, rank {rank}" + # Rows the drafter stored (position >= reuse_prefix) are f(...) exactly. + layer, kv, _, t, d = np.indices(received.shape) + pos = ordinal * _DRAFT_TPB + t + expected = _value(layer, kv, head, pos, d) + mask = pos >= reuse_prefix + assert np.array_equal(received[mask], expected[mask]) + assert _chunk_num_bytes(gen, gen.chunk(prompt_len)) == int(meta.sizes.sum()) + + worker = _Worker() + gen_transceiver = _transceiver(gen.rank_info, worker) + gen_transceiver._seed_draft_ctx(gen_request) + assert worker.seeds[_RID] == (prompt_len, base, prompt_len) + assert gen_transceiver._draft_kv_gen_counters.sent == 1 + + +@pytest.mark.parametrize("prompt_len", [33, 4097, 9000]) +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_unservable_state_completes_with_no_draft_set_in_context_first(gen_tp, prompt_len): + """An unservable context state completes with an empty window and the target-only write. + + Here the base lies past the prompt's pages, which the generation side cannot reproduce. + """ + ctx = _Context(prompt_len, reuse_prefix=prompt_len // 2) + base = _blocks(prompt_len, _DRAFT_TPB) + 1 + transceiver, ctx_request, chunk = _context_decision( + ctx, _Worker(prompt_len, base), DisaggScheduleStyle.CONTEXT_FIRST + ) + assert transceiver._draft_kv_ctx_counters.fallback == {"reuse_base": 1} + assert transceiver._draft_kv_ctx_counters.sent == 0 + assert chunk.draft_block_ids_per_layer_groups is None + assert np.array_equal(chunk.block_ids_per_layer_groups[0], ctx.target_table) + + gen_request = _generation_request(prompt_len, DisaggScheduleStyle.CONTEXT_FIRST, ctx_request) + assert KvCacheTransceiverV2._received_draft_ctx_state(gen_request) is None + for rank in range(gen_tp): + gen = _Generation(prompt_len, gen_tp, rank) + ctx.register(gen.rank_info) + meta = ctx.sender._build_kv_write_meta( + ctx.task(chunk=chunk), gen.req_info(with_draft=False) + ) + ref_ctx, ref_gen, ref = _target_only_meta(prompt_len, gen_tp, rank) + assert _same_target_write(meta, ref, ctx, ref_ctx, gen, ref_gen) + _apply(meta) + assert (gen.draft.array == _SENTINEL).all() + assert _chunk_num_bytes(gen, gen.chunk(prompt_len, with_draft=False)) == int( + meta.sizes.sum() + ) + worker = _Worker() + _transceiver(gen.rank_info, worker)._seed_draft_ctx(gen_request) + assert worker.seeds[_RID] == (prompt_len, _blocks(prompt_len, _DRAFT_TPB), prompt_len) + + +@pytest.mark.parametrize("prompt_len", [33, 4097, 9000]) +def test_generation_first_fallback_still_moves_the_asked_bytes_but_seeds_empty(prompt_len): + """Generation-first fallback: the asked pages travel, marked unusable. + + The generation side asks for the window before the context side decides, so the pages still + move (the expected-byte total stays exact) and the aux slot marks them unusable. + """ + ctx = _Context(prompt_len) + transceiver, ctx_request, chunk = _context_decision( + ctx, _Worker(prompt_len - 1, 0), DisaggScheduleStyle.GENERATION_FIRST + ) + assert transceiver._draft_kv_ctx_counters.fallback == {"ctx_len_mismatch": 1} + assert ctx_request.py_draft_ctx_aux == (0, 0, False) + assert chunk.draft_block_ids_per_layer_groups is not None + + gen = _Generation(prompt_len, 4, 2) + ctx.register(gen.rank_info) + meta = ctx.sender._build_kv_write_meta(ctx.task(chunk=chunk), gen.req_info()) + assert _chunk_num_bytes(gen, gen.chunk(prompt_len)) == int(meta.sizes.sum()) + + gen_request = _generation_request(prompt_len, DisaggScheduleStyle.GENERATION_FIRST, ctx_request) + worker = _Worker() + gen_transceiver = _transceiver(gen.rank_info, worker) + gen_transceiver._seed_draft_ctx(gen_request) + assert worker.seeds[_RID] == (prompt_len, _blocks(prompt_len, _DRAFT_TPB), prompt_len) + assert gen_transceiver._draft_kv_gen_counters.fallback == {"no_ctx_state": 1} + + +@pytest.mark.parametrize("style", list(DisaggScheduleStyle)) +def test_a_whole_prompt_store_ships_and_seeds_the_context_state(style): + prompt_len = 4097 + ctx = _Context(prompt_len) + transceiver, ctx_request, chunk = _context_decision(ctx, _Worker(prompt_len, 0), style) + assert transceiver._draft_kv_ctx_counters.sent == 1 + assert not transceiver._draft_kv_ctx_counters.fallback + assert chunk.draft_block_ids_per_layer_groups is not None + gen_request = _generation_request(prompt_len, style, ctx_request) + assert KvCacheTransceiverV2._received_draft_ctx_state(gen_request) == (prompt_len, 0) + worker = _Worker() + _transceiver(ctx.rank_info, worker)._seed_draft_ctx(gen_request) + assert worker.seeds[_RID] == (prompt_len, 0, prompt_len) + + +# --------------------------------------------------------------------------- +# Test only: the draft-window dump knob (TRTLLM_M31_F0A_DUMP_DIR) and its compare script +# --------------------------------------------------------------------------- + + +def _compare_module(): + spec = importlib.util.spec_from_file_location( + "f0a_g1d_compare", Path(__file__).with_name("f0a_g1d_compare.py") + ) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _dumping(transceiver, directory, rank, heads): + transceiver._draft_dump_dir = str(directory) + transceiver._draft_dump_rank = rank + transceiver._draft_dump_heads = heads + return transceiver + + +def _serving_context(contexts, gen): + """The one context rank whose overlap reaches this generation rank (what its receiver asks).""" + serving = [ + ctx + for ctx in contexts + if gen.rank_info.instance_rank + in ctx.sender._registrar.get_peer_overlap(gen.rank_info, gen.rank_info.dp_rank).ranks + ] + assert len(serving) == 1, [ctx.ctx_rank for ctx in serving] + receiver = PeerRegistrar(gen.rank_info, KVRegionExtractorV1(gen.rank_info.page_table)) + assert receiver.get_peer_overlap(contexts[0].rank_info, 0).ranks == [serving[0].ctx_rank] + return serving[0] + + +def _dump_loopback(directory, gen_tp, prompt_len=4097, reuse_prefix=0, ctx_base=0, ctx_tp=1): + """Ship one request from every CTX rank to every GEN rank with dumps on.""" + contexts = [ + _Context(prompt_len, reuse_prefix=reuse_prefix, ctx_tp=ctx_tp, ctx_rank=c) + for c in range(ctx_tp) + ] + chunks, ctx_requests = [], [] + for ctx in contexts: + ctx_transceiver = _dumping( + _transceiver(ctx.rank_info, _Worker(prompt_len, ctx_base)), + directory, + 5 if ctx_tp == 1 else ctx.ctx_rank, # attention DP: whichever rank holds it + ctx.draft.heads, + ) + ctx_request = _request(prompt_len, DisaggScheduleStyle.CONTEXT_FIRST) + chunks.append( + ctx_transceiver._await_draft_ctx( + ctx_request, CacheExtent(name=_RID, local=ctx.chunk()) + ).local + ) + ctx_requests.append(ctx_request) + # Context-first: the response the generation request carries is TP rank 0's. + response = ctx_requests[0] + shipped = response.py_draft_ctx_aux[2] + for rank in range(gen_tp): + gen = _Generation(prompt_len, gen_tp, rank) + ctx = _serving_context(contexts, gen) + ctx.register(gen.rank_info) + _apply( + ctx.sender._build_kv_write_meta( + ctx.task(chunk=chunks[ctx.ctx_rank]), gen.req_info(with_draft=shipped) + ) + ) + worker = _Worker() + gen_transceiver = _dumping(_transceiver(gen.rank_info, worker), directory, rank, 1) + gen_transceiver._draft_reuse_adapter = SimpleNamespace( + tokens_per_block=_DRAFT_TPB, + get_block_ordinals=lambda req, idx, lg, table=gen.draft_table: table, + ) + gen_request = _generation_request(prompt_len, DisaggScheduleStyle.CONTEXT_FIRST, response) + gen_request.is_generation_only_request = True + gen_request.py_disaggregated_params.ctx_request_id = _RID + gen_request.py_disaggregated_params.disagg_request_id = None + gen_request.py_request_id = 9000 + rank # local ids differ; the match id does not + gen_request.request_id = gen_request.py_request_id + gen_transceiver._seed_draft_ctx(gen_request) + worker.first_step(gen_request.py_request_id) + + +def test_dump_knob_unset_writes_nothing(tmp_path, monkeypatch): + monkeypatch.delenv(DUMP_DIR_ENV, raising=False) + assert draft_dump_dir() is None + assert draft_dump_dir({DUMP_DIR_ENV: ""}) is None + ctx = _Context(4097) + worker = _Worker(4097, 0) + transceiver = _transceiver(ctx.rank_info, worker) + request = _request(4097, DisaggScheduleStyle.CONTEXT_FIRST) + transceiver._await_draft_ctx(request, CacheExtent(name=_RID, local=ctx.chunk())) + gen_worker = _Worker() + _transceiver(ctx.rank_info, gen_worker)._seed_draft_ctx( + _generation_request(4097, DisaggScheduleStyle.CONTEXT_FIRST, request) + ) + assert getattr(gen_worker, "hooks", {}) == {} + assert list(tmp_path.iterdir()) == [] + + +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_dumps_pair_and_the_compare_passes_then_catches_one_flip(tmp_path, gen_tp): + _dump_loopback(tmp_path, gen_tp) + names = sorted(p.name for p in tmp_path.iterdir()) + assert names == sorted([f"ctx_5_{_RID}.pt"] + [f"gen_{r}_{_RID}.pt" for r in range(gen_tp)]) + + ctx_dump = torch.load(tmp_path / f"ctx_5_{_RID}.pt", weights_only=False) + assert ctx_dump["format"] == "f0a_g1d_v1" and ctx_dump["shipped"] + assert (ctx_dump["ctx_len"], ctx_dump["ctx_base"], ctx_dump["prompt_len"]) == (4097, 0, 4097) + assert ctx_dump["heads_per_rank"] == _DRAFT_HEADS + assert len(ctx_dump["layers"]) == len(_LAYER_IDS) + first = ctx_dump["layers"][0] + assert first["k"].shape[1:] == (_DRAFT_HEADS, _DRAFT_TPB, _DRAFT_DIM * 2) + + compare = _compare_module() + report = compare.compare(str(tmp_path), 1, gen_tp) + assert report["verdict"] == "G1D_PASS", report + entry = report["requests"][str(_RID)] + assert entry["status"] == "compared" and entry["bitwise"] + assert all(r["rows_compared"] > 0 and r["mismatches"] == 0 for r in entry["gen_ranks"].values()) + + flipped = tmp_path / f"gen_{gen_tp - 1}_{_RID}.pt" + dump = torch.load(flipped, weights_only=False) + dump["layers"][1]["v"][-1, 0, 0, 0] ^= 1 + torch.save(dump, flipped) + report = compare.compare(str(tmp_path), 1, gen_tp) + assert report["verdict"] == "G1D_FAIL" + assert report["requests"][str(_RID)]["gen_ranks"][str(gen_tp - 1)]["mismatches"] == 1 + + out = tmp_path / "report.json" + assert compare.main(["f0a_g1d_compare.py", str(tmp_path), "1", str(gen_tp), str(out)]) == 1 + assert json.loads(out.read_text())["verdict"] == "G1D_FAIL" + + +def test_a_fallback_is_listed_and_alone_does_not_pass(tmp_path): + # A base past the prompt's pages cannot be served: the context side falls back. + _dump_loopback(tmp_path, 4, prompt_len=4097, ctx_base=_blocks(4097, _DRAFT_TPB) + 1) + assert sorted(p.name for p in tmp_path.iterdir()) == [f"ctx_5_{_RID}.pt"] + report = _compare_module().compare(str(tmp_path), 1, 4) + assert report["requests"][str(_RID)]["status"] == "fallback" + assert report["verdict"] == "G1D_FAIL" # nothing compared + + +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_reuse_hit_dumps_compare_bitwise_with_equal_base(tmp_path, gen_tp): + reuse_prefix = 8200 + base = _blocks(reuse_prefix, _DRAFT_TPB) + _dump_loopback(tmp_path, gen_tp, prompt_len=9000, reuse_prefix=reuse_prefix, ctx_base=base) + report = _compare_module().compare(str(tmp_path), 1, gen_tp) + assert report["verdict"] == "G1D_PASS", report + entry = report["requests"][str(_RID)] + assert (entry["ctx_len"], entry["ctx_base"]) == (9000, base) + assert all(r["ctx_base_equal"] for r in entry["gen_ranks"].values()) + + +# --------------------------------------------------------------------------- +# CTX TP-attention layout (tp2, no attention DP): each rank holds a slice of every request +# --------------------------------------------------------------------------- + +_INDEX_DIM = 8 +_TP_CASES = [(p, 0) for p in PROMPT_LENS] + [(33, 16), (4097, 1000), (9000, 8200)] + + +def _tp_contexts(prompt_len, ctx_tp=2, **kwargs): + return [_Context(prompt_len, ctx_tp=ctx_tp, ctx_rank=c, **kwargs) for c in range(ctx_tp)] + + +def _global_head(gen_rank, local_head, gen_heads, gen_tp, total_heads): + """The head generation rank ``gen_rank`` holds in ``local_head`` (the HND mapper's rule).""" + return (gen_rank * gen_heads + local_head) * total_heads // (gen_tp * gen_heads) % total_heads + + +@pytest.mark.parametrize("style", list(DisaggScheduleStyle)) +def test_a_valid_tp_attention_request_ships_on_every_ctx_rank(style): + """Both CTX TP ranks decide from the same host state, so both ship: zero fallbacks.""" + for ctx in _tp_contexts(4097): + transceiver, request, chunk = _context_decision(ctx, _Worker(4097, 0), style) + assert transceiver._draft_kv_ctx_counters.sent == 1 + assert not transceiver._draft_kv_ctx_counters.fallback + assert request.py_draft_ctx_aux == (4097, 0, True) + assert chunk.draft_block_ids_per_layer_groups is not None + + +@pytest.mark.parametrize("prompt_len, reuse_prefix", _TP_CASES) +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_tp_attention_ctx_window_lands_bitwise_on_every_generation_rank( + gen_tp, prompt_len, reuse_prefix +): + contexts = _tp_contexts(prompt_len, reuse_prefix=reuse_prefix) + base = _blocks(reuse_prefix, _DRAFT_TPB) + chunks = [] + for ctx in contexts: + transceiver, request, chunk = _context_decision( + ctx, _Worker(prompt_len, base), DisaggScheduleStyle.CONTEXT_FIRST + ) + assert transceiver._draft_kv_ctx_counters.sent == 1 + assert not transceiver._draft_kv_ctx_counters.fallback + assert request.py_draft_ctx_aux == (prompt_len, base, True) + chunks.append(chunk) + + ctx_heads = contexts[0].draft.heads + total_heads = ctx_heads * len(contexts) + for rank in range(gen_tp): + gen = _Generation(prompt_len, gen_tp, rank) + ctx = _serving_context(contexts, gen) + ctx.register(gen.rank_info) + peer = ctx.sender._registrar.get_peer_rank_info("gen", rank) + peer_draft = ctx.sender._draft_registrar.get_peer_rank_info("gen", rank) + # The draft view keeps the rank topology, so the draft set reaches exactly the target's + # peers: the overlap check never drops a valid window. + target_ranks = ctx.sender._registrar.get_peer_overlap(peer, peer.dp_rank).ranks + draft_ranks = ctx.sender._draft_registrar.get_peer_overlap(peer_draft, 0).ranks + assert draft_ranks == target_ranks and rank in target_ranks + assert len(target_ranks) == gen_tp // len(contexts) + + meta = ctx.sender._build_kv_write_meta(ctx.task(chunk=chunks[ctx.ctx_rank]), gen.req_info()) + assert meta.expected_transfers == len(target_ranks) + _apply(meta) + + head = _global_head(rank, 0, gen.draft.heads, gen_tp, total_heads) + local = head - ctx.ctx_rank * ctx_heads + assert 0 <= local < ctx_heads, (rank, head, ctx.ctx_rank) + common = (ctx.draft_table >= 0) & (gen.draft_table >= 0) + assert (gen.draft_table >= 0).sum() == common.sum() + for ordinal in np.flatnonzero(common): + received = gen.draft.array[gen.draft_table[ordinal]] + stored = ctx.draft.array[ctx.draft_table[ordinal]][:, :, local : local + 1] + assert np.array_equal(received, stored), f"ordinal {ordinal}, rank {rank}" + layer, kv, _, t, d = np.indices(received.shape) + pos = ordinal * _DRAFT_TPB + t + mask = pos >= reuse_prefix + assert np.array_equal(received[mask], _value(layer, kv, head, pos, d)[mask]) + assert _chunk_num_bytes(gen, gen.chunk(prompt_len)) == int(meta.sizes.sum()) + + +@pytest.mark.parametrize("prompt_len", [33, 4097]) +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_tp_attention_ctx_target_kv_and_replicated_index_k(gen_tp, prompt_len): + """Target K/V head mapping and MSA index-K (REPLICATED) from a TP2 CTX without ADP. + + Every CTX rank holds the same index-K (one replicated vector per token); each GEN rank is + written by exactly one CTX rank, its overlap source, and receives the full index-K bitwise. + """ + contexts = _tp_contexts(prompt_len, with_draft=False, index_dim=_INDEX_DIM) + assert np.array_equal(contexts[0].target.index, contexts[1].target.index) + ctx_heads = contexts[0].target.heads + total_heads = ctx_heads * len(contexts) + for rank in range(gen_tp): + gen = _Generation(prompt_len, gen_tp, rank, with_draft=False, index_dim=_INDEX_DIM) + writers = [] + for ctx in contexts: + ctx.register(gen.rank_info) + peer = ctx.sender._registrar.get_peer_rank_info("gen", rank) + if rank not in ctx.sender._registrar.get_peer_overlap(peer, peer.dp_rank).ranks: + continue + writers.append(ctx) + meta = ctx.sender._build_kv_write_meta( + ctx.task(with_draft=False), gen.req_info(with_draft=False) + ) + assert meta.expected_transfers == gen_tp // len(contexts) + _apply(meta) + assert _chunk_num_bytes(gen, gen.chunk(prompt_len, with_draft=False)) == int( + meta.sizes.sum() + ) + # One writer per generation rank: no duplicate index-K writes. + assert [w.ctx_rank for w in writers] == [_serving_context(contexts, gen).ctx_rank] + ctx = writers[0] + for ordinal, gen_slot in enumerate(gen.target_table): + ctx_slot = ctx.target_table[ordinal] + # Full index-K, bitwise, on every generation rank. + assert np.array_equal(gen.target.index[gen_slot], ctx.target.index[ctx_slot]) + il, it, idim = np.indices(gen.target.index[gen_slot].shape) + assert np.array_equal( + gen.target.index[gen_slot], _index_value(il, ordinal * _TARGET_TPB + it, idim) + ) + # Target K/V: this rank's heads, from the CTX rank that holds them. + for local in range(gen.target.heads): + head = _global_head(rank, local, gen.target.heads, gen_tp, total_heads) + assert head // ctx_heads == ctx.ctx_rank + received = gen.target.array[gen_slot][:, :, local] + layer, kv, t, d = np.indices(received.shape) + assert np.array_equal( + received, _value(layer, kv, head, ordinal * _TARGET_TPB + t, d) + ) + + +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_tp_attention_dumps_compare_per_ctx_rank_and_catch_one_flip(tmp_path, gen_tp): + _dump_loopback(tmp_path, gen_tp, prompt_len=9000, reuse_prefix=8200, ctx_base=257, ctx_tp=2) + names = sorted(p.name for p in tmp_path.iterdir()) + assert names == sorted( + [f"ctx_{c}_{_RID}.pt" for c in range(2)] + [f"gen_{r}_{_RID}.pt" for r in range(gen_tp)] + ) + compare = _compare_module() + report = compare.compare(str(tmp_path), 2, gen_tp) + assert report["verdict"] == "G1D_PASS", report + entry = report["requests"][str(_RID)] + assert sorted(entry["ctx_ranks"]) == [0, 1] + assert all(r["rows_compared"] > 0 for r in entry["gen_ranks"].values()) + + flipped = tmp_path / f"gen_0_{_RID}.pt" + dump = torch.load(flipped, weights_only=False) + dump["layers"][0]["k"][-1, 0, 3, 1] ^= 1 + torch.save(dump, flipped) + report = compare.compare(str(tmp_path), 2, gen_tp) + assert report["verdict"] == "G1D_FAIL" + assert report["requests"][str(_RID)]["gen_ranks"]["0"]["mismatches"] == 1 + + +# --------------------------------------------------------------------------- +# CTX TP-rank divergence: one CTX rank ships its slice, another does not +# --------------------------------------------------------------------------- + + +def _gen_receive_outcome(gen, transceiver, request, meta, aux_state): + """What this generation rank's completed receive reports: its local drafter-window veto.""" + written = int(meta.sizes.sum()) + expected = _chunk_num_bytes(gen, gen.chunk(request.prompt_len, with_draft=bool(request.asks))) + session = SimpleNamespace( + kv_write_byte_counts=lambda: (written, expected), peek_draft_ctx=lambda: aux_state + ) + return transceiver._local_draft_veto(request, session) + + +@pytest.mark.parametrize("style", list(DisaggScheduleStyle)) +@pytest.mark.parametrize("shipping_rank", [0, 1]) +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_ctx_rank_divergence_falls_back_on_every_generation_rank(gen_tp, shipping_rank, style): + """CTX TP ranks diverge: rank ``shipping_rank`` ships its slice, the other falls back. + + The other rank had no drafter-store event. No hang (the write completes with what was sent), + no write outside the pages the generation rank named, the target set intact, every generation + rank starts from an empty window, and the divergence is counted. + """ + prompt_len = 4097 + contexts = _tp_contexts(prompt_len) + decisions = [] + for ctx in contexts: + worker = _Worker(prompt_len, 0) if ctx.ctx_rank == shipping_rank else _Worker() + decisions.append(_context_decision(ctx, worker, style)) + assert decisions[shipping_rank][1].py_draft_ctx_aux[2] + assert not decisions[1 - shipping_rank][1].py_draft_ctx_aux[2] + # Context-first: the generation request carries TP rank 0's response. + response = decisions[0][1] + context_first = style == DisaggScheduleStyle.CONTEXT_FIRST + + outcomes, gens = {}, {} + for rank in range(gen_tp): + gen = _Generation(prompt_len, gen_tp, rank) + ctx = _serving_context(contexts, gen) + ctx_transceiver, ctx_request, chunk = decisions[ctx.ctx_rank] + ctx.register(gen.rank_info) + ctx.sender._draft_kv_counters = ctx_transceiver._draft_kv_ctx_counters + gen_request = _generation_request( + prompt_len, style, response if context_first else ctx_request + ) + gen_request.is_generation_only_request = True + gen_request.asks = ( + KvCacheTransceiverV2._received_draft_ctx_state(gen_request) is not None + if context_first + else True + ) + gen_transceiver = _transceiver(gen.rank_info, _Worker()) + gen_transceiver._note_expected_draft_bytes( + gen_request, gen.chunk(prompt_len, with_draft=gen_request.asks) + ) + + meta = ctx.sender._build_kv_write_meta( + ctx.task(chunk=chunk), gen.req_info(with_draft=gen_request.asks) + ) + # Never more than the pages this generation rank named: its own target and draft pools. + pools = [gen.target.array, gen.draft.array] + for dst, size in zip(meta.dst_ptrs, meta.sizes): + assert any( + pool.ctypes.data <= dst and dst + size <= pool.ctypes.data + pool.nbytes + for pool in pools + ) + _apply(meta) + ref_ctx, ref_gen, ref = _target_only_meta(prompt_len, gen_tp, rank) + n = ref.src_ptrs.size + assert np.array_equal(meta.sizes[:n], ref.sizes) # the target set, unaffected + _apply(ref) + assert np.array_equal(gen.target.array, ref_gen.target.array) + + aux = ctx_request.py_draft_ctx_aux + outcomes[rank] = _gen_receive_outcome(gen, gen_transceiver, gen_request, meta, aux) + gens[rank] = (gen, gen_transceiver, gen_request) + + # The vetoes ride the receive consensus: flagged on any generation rank, flagged on all. + def allgather(_local): + return [[[], [], [_RID], [_RID] if outcomes[r] else []] for r in range(gen_tp)] + + stand_in = object.__new__(KvCacheTransceiverV2) + *_, flagged = KvCacheTransceiverV2._consensus_outcome( + stand_in, [_RID], [], [], [_RID], allgather, True, locally_flagged=[] + ) + # Context-first with the response's rank (0) falling back: every generation rank already asked + # for nothing and starts empty, so nothing is vetoed; the shipping rank 1 counts it. + receivers_follow_fallback = context_first and shipping_rank == 1 + assert flagged == (set() if receivers_follow_fallback else {_RID}) + for rank, (gen, gen_transceiver, gen_request) in gens.items(): + veto = outcomes[rank] or ("ctx_rank_divergence" if flagged else None) + worker = gen_transceiver._draft_ctx_owner + gen_transceiver._seed_draft_ctx(gen_request, veto) + # A clean fallback: the drafter starts from an empty window on every rank. + assert worker.seeds[_RID] == (prompt_len, _blocks(prompt_len, _DRAFT_TPB), prompt_len) + assert gen_transceiver._draft_kv_gen_counters.sent == 0 + reasons = Counter() + for _, gen_transceiver, _ in gens.values(): + reasons.update(gen_transceiver._draft_kv_gen_counters.fallback) + if receivers_follow_fallback: + assert reasons == {"no_ctx_state": gen_tp} + else: + assert reasons["ctx_rank_divergence"] > 0 + assert sum(reasons.values()) == gen_tp + # The context rank whose decision the receivers did not follow counts it too: as a divergence + # when it ships nothing the receivers asked for, as receiver_not_asked when it ships and the + # receivers (following rank 0's fallback) asked for none. A rank without a drafter-store event + # also counted its own fallback once, when it decided. + if context_first: + divergent = contexts[1] + counters = decisions[1][0]._draft_kv_ctx_counters + if shipping_rank == 0: + expected = {"no_store_event": 1, "ctx_rank_divergence": gen_tp // len(contexts)} + else: + expected = {"receiver_not_asked": gen_tp // len(contexts)} + assert counters.fallback == expected + assert divergent.sender._draft_kv_counters is counters + + +# --------------------------------------------------------------------------- +# Registration-time draft validation (from the upstream per-view transfer, #20709): a mismatch +# disables the draft set for that peer pair only, both sides agreeing; the target is untouched +# --------------------------------------------------------------------------- + +_CTX_ENDPOINT = "tcp://ctx:1" + + +def _receiver(gen, ctx, counters=None): + """The generation rank's receiver after it fetched the context peer's rank info.""" + receiver = Receiver.__new__(Receiver) + receiver._registrar = PeerRegistrar( + gen.rank_info, KVRegionExtractorV1(gen.rank_info.page_table) + ) + draft_ri = gen.rank_info.draft_view() + if draft_ri is not None: + receiver._draft_registrar = PeerRegistrar( + draft_ri, KVRegionExtractorV1(draft_ri.page_table) + ) + receiver._draft_kv_counters = counters + receiver._check_draft_peer(_CTX_ENDPOINT, RankInfo.from_bytes(ctx.rank_info.to_bytes())) + return receiver + + +def _receive(receiver, gen, prompt_len): + """The receive's task and request info after the receiver's per-peer draft decision.""" + engaged = gen.rank_info.has_draft_kv + task = SimpleNamespace( + expected_write_bytes=_chunk_num_bytes(gen, gen.chunk(prompt_len, with_draft=engaged)), + draft_disabled_reason=None, + ) + info = gen.req_info(with_draft=engaged) + params = DisaggregatedParams(disagg_request_id=_RID, ctx_info_endpoint=_CTX_ENDPOINT) + receiver._drop_disabled_draft_window(task, info, params) + return task, info + + +@pytest.mark.parametrize("draft", [False, True], ids=["target_only", "draft_set"]) +@pytest.mark.parametrize("disabled_peers", [None, {_CTX_ENDPOINT: ("peer_not_engaged", "why")}]) +def test_receive_without_a_ctx_endpoint_never_needs_it(draft, disabled_peers): + """Gen-first and ADP dispatches carry no ctx_info_endpoint; the window decision never needs one. + + Looking it up on every receive raised "ctx_info_endpoint is required" and broke the stock + receive path (test_transfer_ownership_regressions). + """ + receiver = Receiver.__new__(Receiver) + receiver._registrar = SimpleNamespace( + self_rank_info=SimpleNamespace(draft_receive_disabled=None) + ) + receiver._draft_registrar = None + receiver._draft_kv_counters = None + receiver._draft_disabled_peers = disabled_peers + draft_ids = [[1, 2]] if draft else None + info = SimpleNamespace(draft_block_ids_per_layer_groups=draft_ids, unique_rid=_RID) + task = SimpleNamespace(expected_write_bytes=None, draft_disabled_reason=None) + params = DisaggregatedParams(disagg_request_id=_RID) + assert receiver._drop_disabled_draft_window(task, info, params) == draft_ids + assert task.draft_disabled_reason is None + + +def _disabled_window_round_trip(ctx, gen, prompt_len, gen_tp, rank, reason): + """Register both sides, move one request, and check the window-only fallback end to end.""" + counters = DraftKvCounters("context") + ctx.sender._draft_kv_counters = counters + ctx.register(gen.rank_info) # never raises: the target registers, the window is disabled + key = gen.rank_info.instance_name + str(gen.rank_info.instance_rank) + assert ctx.sender._draft_disabled_peers[key][0] == reason + assert ctx.sender._registrar.get_peer_rank_info("gen", rank) is not None + + receiver = _receiver(gen, ctx) + task, info = _receive(receiver, gen, prompt_len) + assert info.draft_block_ids_per_layer_groups is None + assert task.draft_disabled_reason == reason + assert task.expected_write_bytes == _chunk_num_bytes( + gen, gen.chunk(prompt_len, with_draft=False) + ) + + meta = ctx.sender._build_kv_write_meta(ctx.task(), info) + assert counters.fallback == {reason: 1} + ref_ctx, ref_gen, ref = _target_only_meta(prompt_len, gen_tp, rank) + assert _same_target_write(meta, ref, ctx, ref_ctx, gen, ref_gen) + _apply(meta) + _apply(ref) + assert np.array_equal(gen.target.array, ref_gen.target.array) + assert (gen.draft.array == _SENTINEL).all() + assert int(meta.sizes.sum()) == task.expected_write_bytes # exact: the write verifies + + # The generation rank drafts from an empty window, counted under the same reason. + worker = _Worker() + gen_transceiver = _transceiver(gen.rank_info, worker) + session = SimpleNamespace( + draft_disabled_reason=lambda: task.draft_disabled_reason, + kv_write_byte_counts=lambda: (int(meta.sizes.sum()), task.expected_write_bytes), + peek_draft_ctx=lambda: None, + ) + request = _request(prompt_len, DisaggScheduleStyle.GENERATION_FIRST) + request.py_draft_ctx_aux = (prompt_len, 0, True) + veto = gen_transceiver._local_draft_veto(request, session) + assert veto == reason + gen_transceiver._seed_draft_ctx(request, veto) + assert worker.seeds[_RID] == (prompt_len, _blocks(prompt_len, _DRAFT_TPB), prompt_len) + assert gen_transceiver._draft_kv_gen_counters.fallback == {reason: 1} + + +@pytest.mark.parametrize("drift", ["head_dim", "dtype"]) +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_draft_geometry_drift_disables_only_the_window(gen_tp, drift): + prompt_len = 4097 + rank = gen_tp - 1 + ctx = _Context(prompt_len) + kwargs = {"draft_dim": 2 * _DRAFT_DIM} if drift == "head_dim" else {"draft_dtype": np.int32} + gen = _Generation(prompt_len, gen_tp, rank, **kwargs) + reason = draft_peer_mismatch(ctx.rank_info.draft_view(), gen.rank_info.draft_view()) + assert reason is not None and ( + "head_dim" in reason if drift == "head_dim" else "dtype" in reason + ) + _disabled_window_round_trip(ctx, gen, prompt_len, gen_tp, rank, "peer_draft_geometry") + + +def _rename_roles(page_table, roles): + page_table.layer_groups[0].pool_views[0].pool_role = frozenset(roles) + + +def _duplicate_view(page_table): + group = page_table.layer_groups[0] + group.pool_views.append(replace(group.pool_views[0])) + + +def _shift_layers(page_table): + group = page_table.layer_groups[0] + group.local_layers = [ + LocalLayer(ll.local_layer_id, ll.global_layer_id + 100) for ll in group.local_layers + ] + + +@pytest.mark.parametrize( + "mutate, expected", + [ + (lambda pt: _rename_roles(pt, {"key", "value", "scale"}), "matches 0 peer views"), + (_duplicate_view, "matches 2 peer views"), + (_shift_layers, "draft layers"), + ], +) +def test_each_draft_view_check_disables_the_window_on_both_sides(mutate, expected): + prompt_len = 4097 + ctx = _Context(prompt_len) + gen = _Generation(prompt_len, 4, 1) + mutate(gen.rank_info.draft_page_table) + why = draft_peer_mismatch(ctx.rank_info.draft_view(), gen.rank_info.draft_view()) + assert why is not None and expected in why + # The receiver's view of the same pair reaches a verdict too. + assert draft_peer_mismatch(gen.rank_info.draft_view(), ctx.rank_info.draft_view()) is not None + _disabled_window_round_trip(ctx, gen, prompt_len, 4, 1, "peer_draft_geometry") + + +def test_draft_pipeline_parallelism_and_overlap_mismatch_are_refused(): + ctx = _Context(64) + gen = _Generation(64, 4, 0) + pp2 = replace(gen.rank_info.draft_view(), pp_size=2) + assert "pipeline parallelism" in draft_peer_mismatch(ctx.rank_info.draft_view(), pp2) + why = draft_peer_mismatch( + ctx.rank_info.draft_view(), gen.rank_info.draft_view(), ([0, 1, 2, 3], [0, 1]) + ) + assert "overlap" in why + assert draft_peer_mismatch(ctx.rank_info.draft_view(), gen.rank_info.draft_view()) is None + + +def test_a_draft_fragment_error_drops_only_the_window(monkeypatch): + prompt_len = 4097 + ctx = _Context(prompt_len) + gen = _Generation(prompt_len, 4, 2) + ctx.register(gen.rank_info) + counters = DraftKvCounters("context") + ctx.sender._draft_kv_counters = counters + + def broken(*_args, **_kwargs): + raise ValueError("HND bytes per head mismatch") + + monkeypatch.setattr(ctx.sender._draft_registrar, "get_kv_map", broken) + meta = ctx.sender._build_kv_write_meta(ctx.task(), gen.req_info()) # completes + assert counters.fallback == {"draft_fragment_error": 1} + ref_ctx, ref_gen, ref = _target_only_meta(prompt_len, 4, 2) + assert _same_target_write(meta, ref, ctx, ref_ctx, gen, ref_gen) + # The receiver asked for the window, so its attestation is short by exactly that: it falls back. + expected = _chunk_num_bytes(gen, gen.chunk(prompt_len)) + session = SimpleNamespace( + kv_write_byte_counts=lambda: (int(meta.sizes.sum()), expected), + peek_draft_ctx=lambda: None, + ) + gen_transceiver = _transceiver(gen.rank_info, _Worker()) + request = _request(prompt_len, DisaggScheduleStyle.CONTEXT_FIRST) + gen_transceiver._note_expected_draft_bytes(request, gen.chunk(prompt_len)) + assert gen_transceiver._local_draft_veto(request, session) == "ctx_rank_divergence" + + +# --------------------------------------------------------------------------- +# Synchronous generation receive with TP ranks: no window at all (sync_path_tp) +# --------------------------------------------------------------------------- + + +def test_sync_receive_with_tp_ranks_takes_no_window(): + tp4 = SimpleNamespace(tp_size=4, enable_attention_dp=False) + assert draft_receive_disabled_reason(tp4, synchronous_receive=True) == "sync_path_tp" + assert draft_receive_disabled_reason(tp4, synchronous_receive=False) is None + adp = SimpleNamespace(tp_size=4, enable_attention_dp=True) + assert draft_receive_disabled_reason(adp, synchronous_receive=True) is None + tp1 = SimpleNamespace(tp_size=1, enable_attention_dp=False) + assert draft_receive_disabled_reason(tp1, synchronous_receive=True) is None + + +def test_receive_disabled_rank_info_round_trips_and_is_absent_by_default(): + gen = _Generation(64, 4, 0) + assert "draft_receive_disabled" not in msgpack.unpackb( + gen.rank_info.to_bytes(), strict_map_key=False + ) + gen.rank_info.draft_receive_disabled = "sync_path_tp" + restored = RankInfo.from_bytes(gen.rank_info.to_bytes()) + assert restored.draft_receive_disabled == "sync_path_tp" + assert restored.draft_view().draft_receive_disabled is None + + +@pytest.mark.parametrize("gen_tp", [4, 8]) +def test_sync_receive_tp_ranks_all_seed_the_same_empty_window_with_a_divergent_ctx(gen_tp): + """GEN TP on the synchronous receive path seeds one empty window on every rank. + + CTX TP2 with rank 1 diverging: every generation rank takes the target set only and seeds the + same empty window; both sides count it. + """ + prompt_len = 4097 + contexts = _tp_contexts(prompt_len) + decisions = [ + _context_decision( + ctx, + _Worker(prompt_len, 0) if ctx.ctx_rank == 0 else _Worker(), + DisaggScheduleStyle.CONTEXT_FIRST, + ) + for ctx in contexts + ] + seeds = set() + for rank in range(gen_tp): + gen = _Generation(prompt_len, gen_tp, rank) + gen.rank_info.draft_receive_disabled = "sync_path_tp" + ctx = _serving_context(contexts, gen) + ctx_counters = decisions[ctx.ctx_rank][0]._draft_kv_ctx_counters + ctx.sender._draft_kv_counters = ctx_counters + ctx.register(gen.rank_info) + key = "gen" + str(rank) + assert ctx.sender._draft_disabled_peers[key][0] == "sync_path_tp" + receiver = _receiver(gen, ctx) + task, info = _receive(receiver, gen, prompt_len) + assert info.draft_block_ids_per_layer_groups is None + assert task.draft_disabled_reason == "sync_path_tp" + chunk = decisions[ctx.ctx_rank][2] + meta = ctx.sender._build_kv_write_meta(ctx.task(chunk=chunk), info) # no hang + ref_ctx, ref_gen, ref = _target_only_meta(prompt_len, gen_tp, rank) + assert np.array_equal(meta.sizes, ref.sizes) + assert int(meta.sizes.sum()) == task.expected_write_bytes + worker = _Worker() + gen_transceiver = _transceiver(gen.rank_info, worker) + session = SimpleNamespace( + draft_disabled_reason=lambda t=task: t.draft_disabled_reason, + kv_write_byte_counts=lambda m=meta, t=task: ( + int(m.sizes.sum()), + t.expected_write_bytes, + ), + peek_draft_ctx=lambda: None, + ) + request = _generation_request( + prompt_len, DisaggScheduleStyle.CONTEXT_FIRST, decisions[0][1] + ) + gen_transceiver._seed_draft_ctx( + request, gen_transceiver._local_draft_veto(request, session) + ) + seeds.add(worker.seeds[_RID]) + assert gen_transceiver._draft_kv_gen_counters.fallback == {"sync_path_tp": 1} + assert seeds == {(prompt_len, _blocks(prompt_len, _DRAFT_TPB), prompt_len)} + # Rank 0 shipped, so its sender counts the disabled peers; rank 1 fell back on its own. + assert decisions[0][0]._draft_kv_ctx_counters.fallback["sync_path_tp"] == gen_tp // 2 + + +# --------------------------------------------------------------------------- +# Mixed peers: only one side engages the drafter window +# --------------------------------------------------------------------------- +# +# E.g. a legacy (hook-off) MiniMax CTX TP2 serving a modeling_v2 (hook-on) GEN TP4 / TP8, or the +# reverse. The peer pair degrades to target-only transfers: no request fails, the target write +# is bitwise the target-only write, both sides count draft_kv_fallback{peer_not_engaged}, and an +# engaged generation side seeds the empty window at the prompt end. + + +def _target_only_reference(prompt_len, gen_tp, gen_rank, ctx_tp, ctx_rank): + ref_ctx = _Context(prompt_len, with_draft=False, ctx_tp=ctx_tp, ctx_rank=ctx_rank) + ref_gen = _Generation(prompt_len, gen_tp, gen_rank, with_draft=False) + ref_ctx.register(ref_gen.rank_info) + ref = ref_ctx.sender._build_kv_write_meta( + ref_ctx.task(with_draft=False), ref_gen.req_info(with_draft=False) + ) + return ref_ctx, ref_gen, ref + + +@pytest.mark.parametrize("prompt_len", [4097, 100_013]) +@pytest.mark.parametrize("style", list(DisaggScheduleStyle)) +@pytest.mark.parametrize("gen_tp", [4, 8]) +@pytest.mark.parametrize("ctx_layout", ["adp", "tp_attention"]) +@pytest.mark.parametrize("engaged", ["gen_only", "ctx_only"]) +def test_mixed_peers_degrade_to_target_only(engaged, ctx_layout, gen_tp, style, prompt_len): + ctx_engaged = engaged == "ctx_only" + ctx_tp = 1 if ctx_layout == "adp" else 2 + contexts = [ + _Context(prompt_len, with_draft=ctx_engaged, ctx_tp=ctx_tp, ctx_rank=c) + for c in range(ctx_tp) + ] + ctx_counters, chunks, ctx_requests = {}, {}, {} + for ctx in contexts: + if ctx_engaged: + # The engaged context side ships the whole-prompt window as usual. + transceiver, request, chunk = _context_decision(ctx, _Worker(prompt_len, 0), style) + assert chunk.draft_block_ids_per_layer_groups is not None + ctx_counters[ctx.ctx_rank] = transceiver._draft_kv_ctx_counters + ctx_requests[ctx.ctx_rank] = request + else: + chunk = ctx.chunk(with_draft=False) + ctx_counters[ctx.ctx_rank] = DraftKvCounters("context") + ctx.sender._draft_kv_counters = ctx_counters[ctx.ctx_rank] + chunks[ctx.ctx_rank] = chunk + + empty_window = (prompt_len, _blocks(prompt_len, _DRAFT_TPB), prompt_len) + for rank in range(gen_tp): + gen = _Generation(prompt_len, gen_tp, rank, with_draft=not ctx_engaged) + ctx = _serving_context(contexts, gen) if ctx_tp > 1 else contexts[0] + ctx.register(gen.rank_info) # never raises + assert ctx.sender._draft_disabled_peers[f"gen{rank}"][0] == "peer_not_engaged" + + gen_counters = DraftKvCounters("generation") + receiver = _receiver(gen, ctx, counters=None if not ctx_engaged else gen_counters) + assert receiver._draft_disabled_peers[_CTX_ENDPOINT][0] == "peer_not_engaged" + task, info = _receive(receiver, gen, prompt_len) + assert info.draft_block_ids_per_layer_groups is None + target_bytes = _chunk_num_bytes(gen, gen.chunk(prompt_len, with_draft=False)) + assert task.expected_write_bytes == target_bytes + + meta = ctx.sender._build_kv_write_meta(ctx.task(chunk=chunks[ctx.ctx_rank]), info) + ref_ctx, ref_gen, ref = _target_only_reference( + prompt_len, gen_tp, rank, ctx_tp, ctx.ctx_rank + ) + assert _same_target_write(meta, ref, ctx, ref_ctx, gen, ref_gen) + _apply(meta) + _apply(ref) + assert np.array_equal(gen.target.array, ref_gen.target.array) # bitwise target-only + assert (gen.draft.array == _SENTINEL).all() # no draft byte moved + assert int(meta.sizes.sum()) == task.expected_write_bytes # exact: the write verifies + + if ctx_engaged: + # The generation rank does not engage: its receiver counts the request. + assert gen_counters.fallback == {"peer_not_engaged": 1} + continue + # The engaged generation rank vetoes the window and seeds the empty one at the prompt end, + # even over a stale drafter-context aux slot (the legacy peer never writes it). + worker = _Worker() + gen_transceiver = _transceiver(gen.rank_info, worker) + session = SimpleNamespace( + draft_disabled_reason=lambda t=task: t.draft_disabled_reason, + kv_write_byte_counts=lambda m=meta, t=task: ( + int(m.sizes.sum()), + t.expected_write_bytes, + ), + peek_draft_ctx=lambda: (prompt_len, 0, True), + ) + request = _request(prompt_len, style) + request.py_draft_ctx_aux = (prompt_len, 0, True) + veto = gen_transceiver._local_draft_veto(request, session) + assert veto == "peer_not_engaged" + gen_transceiver._seed_draft_ctx(request, veto) + assert worker.seeds[_RID] == empty_window + assert gen_transceiver._draft_kv_gen_counters.fallback == {"peer_not_engaged": 1} + + # The context side counts every request toward a peer that does not engage the window + # (an engaged one also counted it as decided-to-ship before the per-peer veto). + assert sum(c.fallback["peer_not_engaged"] for c in ctx_counters.values()) == gen_tp + for counters in ctx_counters.values(): + assert set(counters.fallback) <= {"peer_not_engaged"} + + +@pytest.mark.parametrize("sender_engaged", [True, False]) +def test_mixed_peer_aux_write_moves_the_common_buffers_only(sender_engaged): + """Generation-first: only the common auxiliary buffers travel between mixed peers. + + The engaged side's layout carries one extra drafter-context buffer; toward a peer without it + the common buffers travel instead of failing on the layout mismatch. + """ + from tensorrt_llm._torch.disaggregation.native.auxiliary import ( + AuxBufferMeta, + build_aux_transfer_layout, + ) + + def aux_meta(base, engaged): + sizes = [8, 32, 8, 8] + ([12] if engaged else []) + ptrs = [base + 0x10000 * i for i in range(len(sizes))] + return AuxBufferMeta( + ptrs=np.array(ptrs, dtype=np.int64), + size=np.array([s * 16 for s in sizes], dtype=np.int64), + item_sizes=np.array(sizes, dtype=np.int64), + ) + + src = aux_meta(0x100000, sender_engaged) + dst = aux_meta(0x900000, not sender_engaged) + with pytest.raises(ValueError, match="do not match"): + build_aux_transfer_layout(src, dst) + self_ri = SimpleNamespace(has_draft_kv=sender_engaged, aux_meta=src) + peer_ri = SimpleNamespace( + has_draft_kv=not sender_engaged, + aux_meta=dst, + instance_name="gen", + instance_rank=0, + dp_rank=0, + self_endpoint="", + ) + cached = {} + registrar = SimpleNamespace( + self_rank_info=self_ri, + get_peer_rank_info=lambda name, rank: peer_ri, + get_peer_overlap=lambda ri, dp_rank: PeerOverlap(ranks=[0]), + should_send_aux=lambda ri: True, + get_aux_transfer_layout=lambda name, rank: cached.get((name, rank)), + cache_aux_transfer_layout=lambda name, rank, layout: cached.__setitem__( + (name, rank), layout + ), + ) + sender = Sender.__new__(Sender) + sender._registrar = registrar + task = SimpleNamespace(_perf_timer=None, _slot=2, _unique_rid=_RID) + meta = sender._build_aux_write_meta( + task, SimpleNamespace(instance_name="gen", instance_rank=0, aux_slot=3, unique_rid=_RID) + ) + common = aux_meta(0x100000, False) + assert np.array_equal(meta.sizes, common.item_sizes) + assert np.array_equal(meta.src_ptrs, common.ptrs + common.item_sizes * 2) + assert np.array_equal(meta.dst_ptrs, aux_meta(0x900000, False).ptrs + common.item_sizes * 3) + trimmed = aux_meta_without_draft_ctx(aux_meta(0x100000, True)) + assert np.array_equal(trimmed.ptrs, common.ptrs) + assert np.array_equal(trimmed.item_sizes, common.item_sizes) + + +def test_matched_peers_keep_the_aux_layout_unchanged(): + """Both engaged or both not: the auxiliary layout is passed through as before.""" + from tensorrt_llm._torch.disaggregation.native.auxiliary import AuxBufferMeta + + for engaged in (True, False): + n = 5 if engaged else 4 + meta = AuxBufferMeta( + ptrs=np.arange(1, n + 1, dtype=np.int64) * 0x10000, + size=np.full(n, 64, dtype=np.int64), + item_sizes=np.full(n, 4, dtype=np.int64), + ) + ri = SimpleNamespace( + has_draft_kv=engaged, + aux_meta=meta, + instance_name="gen", + instance_rank=0, + dp_rank=0, + self_endpoint="", + ) + registrar = SimpleNamespace( + self_rank_info=ri, + get_peer_rank_info=lambda name, rank, ri=ri: ri, + get_peer_overlap=lambda r, d: PeerOverlap(ranks=[0]), + should_send_aux=lambda r: True, + get_aux_transfer_layout=lambda name, rank: None, + cache_aux_transfer_layout=lambda *a: None, + ) + sender = Sender.__new__(Sender) + sender._registrar = registrar + meta_out = sender._build_aux_write_meta( + SimpleNamespace(_perf_timer=None, _slot=0, _unique_rid=_RID), + SimpleNamespace(instance_name="gen", instance_rank=0, aux_slot=0, unique_rid=_RID), + ) + assert meta_out.sizes.size == n diff --git a/tests/unittest/disaggregated/test_peer.py b/tests/unittest/disaggregated/test_peer.py index 7ff5f5e72e7c..fd335705f5f5 100644 --- a/tests/unittest/disaggregated/test_peer.py +++ b/tests/unittest/disaggregated/test_peer.py @@ -836,6 +836,26 @@ def test_replicated_mapper_ignores_kv_head_mismatch(): assert isinstance(mapper, ReplicatedMapper) +@pytest.mark.parametrize("peer_block_bytes", [1024, 512]) +def test_peer_registrar_rejects_replicated_role_of_another_size(peer_block_bytes): + """An index-key side cache stored in another format on the peer (e.g. FP8 vs NVFP4).""" + self_pt = make_page_table(global_layer_ids=[0]) + peer_pt = make_page_table(global_layer_ids=[0], block_bytes=[peer_block_bytes]) + for page_table in (self_pt, peer_pt): + view = page_table.layer_groups[0].pool_views[0] + view.pool_role = frozenset({"index_key"}) + view.mapper_kind = MapperKind.REPLICATED + reg = _make_peer_registrar(make_rankinfo(instance_name="local", page_table=self_pt)) + peer_ri = make_rankinfo( + instance_name="peer", instance_rank=3, layer_num_per_pp=[1], page_table=peer_pt + ) + if peer_block_bytes == 1024: + reg.register(peer_ri.instance_name, peer_ri.instance_rank, peer_ri) + return + with pytest.raises(ValueError, match="replicated roles differ in bytes per layer"): + reg.register(peer_ri.instance_name, peer_ri.instance_rank, peer_ri) + + @pytest.mark.parametrize("peer_dp_rank", [0, 1, 3, 7]) def test_replicated_pool_owner_rotates_by_destination_dp_rank(peer_dp_rank): """Exactly one owner per fan-in group, rotated by destination DP rank. diff --git a/tests/unittest/disaggregated/test_rank_info_draft_kv.py b/tests/unittest/disaggregated/test_rank_info_draft_kv.py new file mode 100644 index 000000000000..25f7e3b5e8b6 --- /dev/null +++ b/tests/unittest/disaggregated/test_rank_info_draft_kv.py @@ -0,0 +1,215 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""RankInfo's optional draft-pool geometry (disagg drafter context window). + +Wire round trip, the unchanged target-only wire format, and the draft view. +""" + +from dataclasses import asdict +from types import SimpleNamespace + +import msgpack +import numpy as np +import pytest + +from tensorrt_llm._torch.disaggregation.native import rank_info as rank_info_module +from tensorrt_llm._torch.disaggregation.native.mixers.attention.spec import AttentionInfo +from tensorrt_llm._torch.disaggregation.native.rank_info import RankInfo +from tensorrt_llm._torch.disaggregation.resource.page import ( + BUFFER_ENTRY_DTYPE, + AttentionLayerGroup, + KVCachePageTable, + LocalLayer, + PhysicalPool, + PhysicalPoolGroup, + PoolView, +) + +pytestmark = pytest.mark.cpu_only + + +def _attention(heads, tpb, dim): + return AttentionInfo( + kv_heads_per_rank=heads, + tokens_per_block=tpb, + dims_per_head=dim, + element_bytes=2, + enable_attention_dp=True, + is_mla=False, + ) + + +def _page_table(base, tpb, buf): + entries = np.array([(0, 0, buf), (0, buf, buf)], dtype=BUFFER_ENTRY_DTYPE) + return KVCachePageTable( + tokens_per_block=tpb, + layer_groups=[ + AttentionLayerGroup( + pool_group_idx=0, + kv_head_num_per_rank=4, + sliding_window_size=2048, + local_layers=[LocalLayer(0, 0)], + pool_views=[PoolView(pool_idx=0, buffer_entries=entries, bytes_per_layer=2 * buf)], + ) + ], + pool_groups=[ + PhysicalPoolGroup( + pools=[PhysicalPool(base_address=base, slot_bytes=2 * buf, num_slots=9)] + ) + ], + ) + + +def _rank_info(with_draft: bool) -> RankInfo: + ri = RankInfo( + instance_name="ctx", + instance_rank=1, + tp_size=2, + tp_rank=1, + pp_size=1, + pp_rank=0, + layer_num_per_pp=[62], + sender_endpoints=["tcp://a:1", "tcp://a:2"], + self_endpoint="tcp://a:3", + transfer_engine_info=b"\x01\x02", + dp_size=2, + dp_rank=1, + attention=_attention(8, 64, 128), + page_table=_page_table(0x10000, 64, 4096), + ) + if with_draft: + ri.draft_attention = _attention(4, 32, 128) + ri.draft_page_table = _page_table(0x90000, 32, 2048) + return ri + + +def _legacy_bytes(ri: RankInfo) -> bytes: + """What a rank built before the draft fields serialized: no draft keys at all.""" + data = asdict(ri) + data.pop("draft_attention") + data.pop("draft_page_table") + data.pop("draft_receive_disabled") + data["attention"] = ri.attention.to_dict() + data["aux_meta"] = None + data["page_table"] = ri.page_table.to_dict() + return msgpack.packb(data) + + +def test_target_only_rank_info_serializes_exactly_as_before(): + ri = _rank_info(with_draft=False) + data = ri.to_bytes() + assert data == _legacy_bytes(ri) + unpacked = msgpack.unpackb(data, strict_map_key=False) + assert "draft_attention" not in unpacked and "draft_page_table" not in unpacked + assert "draft_receive_disabled" not in unpacked + restored = RankInfo.from_bytes(data) + assert restored.draft_attention is None and restored.draft_page_table is None + assert not restored.has_draft_kv + assert restored.draft_view() is None + + +def test_legacy_peer_bytes_parse_without_draft_fields(): + legacy = _legacy_bytes(_rank_info(with_draft=False)) + restored = RankInfo.from_bytes(legacy) + assert restored.draft_view() is None + assert restored.page_table.to_dict() == _rank_info(with_draft=False).page_table.to_dict() + + +def test_draft_fields_round_trip(): + ri = _rank_info(with_draft=True) + restored = RankInfo.from_bytes(ri.to_bytes()) + assert restored.has_draft_kv + assert restored.draft_attention == ri.draft_attention + assert restored.draft_page_table.to_dict() == ri.draft_page_table.to_dict() + # The target geometry is untouched by the draft set. + assert restored.attention == ri.attention + assert restored.page_table.to_dict() == ri.page_table.to_dict() + + +def test_draft_view_swaps_only_the_geometry(): + ri = _rank_info(with_draft=True) + view = RankInfo.from_bytes(ri.to_bytes()).draft_view() + assert view.attention == ri.draft_attention + assert view.page_table.to_dict() == ri.draft_page_table.to_dict() + assert not view.has_draft_kv + for name in ( + "instance_name", + "instance_rank", + "tp_size", + "tp_rank", + "pp_size", + "pp_rank", + "dp_size", + "dp_rank", + "layer_num_per_pp", + "sender_endpoints", + "self_endpoint", + "transfer_engine_info", + ): + assert getattr(view, name) == getattr(ri, name), name + assert view.tp_size_per_dp_group == ri.tp_size_per_dp_group == 1 + + +def test_rank_info_built_without_init_has_no_draft_set(): + ri = object.__new__(RankInfo) + assert not ri.has_draft_kv + assert ri.draft_view() is None + + +def _manager(heads, tpb, head_dim, layers): + mapping = SimpleNamespace( + rank=0, + tp_size=2, + tp_rank=0, + pp_size=1, + pp_rank=0, + dp_size=1, + cp_size=1, + cp_rank=0, + enable_attention_dp=True, + ) + return SimpleNamespace( + mapping=mapping, + num_kv_heads_per_layer=[heads] * layers, + pp_layers=list(range(layers)), + tokens_per_block=tpb, + head_dim=head_dim, + dtype=None, + kv_factor=2, + ) + + +@pytest.mark.parametrize("with_draft", [False, True]) +def test_from_kv_cache_manager_reads_the_draft_pool_geometry(monkeypatch, with_draft): + tables = {} + + def build(manager): + tables[id(manager)] = object() + return tables[id(manager)] + + monkeypatch.setattr(rank_info_module, "build_page_table_from_manager", build) + monkeypatch.setattr(rank_info_module, "get_size_in_bytes", lambda n, _dtype: 2 * n) + target = _manager(heads=8, tpb=64, head_dim=128, layers=62) + draft = _manager(heads=4, tpb=32, head_dim=128, layers=1) if with_draft else None + + ri = RankInfo.from_kv_cache_manager("ctx", target, 0, draft_kv_cache_manager=draft) + + assert ri.attention == AttentionInfo(8, 64, 128, 2, True, False) + assert ri.page_table is tables[id(target)] + if with_draft: + assert ri.draft_attention == AttentionInfo(4, 32, 128, 2, True, False) + assert ri.draft_page_table is tables[id(draft)] + else: + assert ri.draft_attention is None and ri.draft_page_table is None diff --git a/tests/unittest/kv_cache_manager_v2_tests/test_kv_gpu_prefetch_runtime.py b/tests/unittest/kv_cache_manager_v2_tests/test_kv_gpu_prefetch_runtime.py new file mode 100644 index 000000000000..b3a21bda16db --- /dev/null +++ b/tests/unittest/kv_cache_manager_v2_tests/test_kv_gpu_prefetch_runtime.py @@ -0,0 +1,145 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +"""Real-tree test for KVCacheManagerV2.prefetch_to_gpu. + +A waiting request's reusable prefix that was evicted to the host tier is found through a +transient, never-resumed cache and migrated back to the GPU. The tier of a held cache's pages +is only visible through ``cached_tokens_by_level`` (taken at match time): the page storage +snapshot reports no levels for a cache that is not active, which is why it cannot be used. +""" + +import gc +import os +import sys +import unittest +from importlib.util import find_spec +from types import SimpleNamespace +from typing import TYPE_CHECKING, cast + +import tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 as kvm + +if not TYPE_CHECKING and find_spec("kv_cache_manager_v2") is not None: + from kv_cache_manager_v2 import GPU_LEVEL, CudaStream, KVCacheManager, ReuseScope, TokenId +else: + from tensorrt_llm.runtime.kv_cache_manager_v2 import ( + GPU_LEVEL, + CudaStream, + KVCacheManager, + ReuseScope, + TokenId, + ) + +_TEST_DIR = os.path.dirname(os.path.abspath(__file__)) +_ADDED_TEST_DIR = _TEST_DIR not in sys.path +if _ADDED_TEST_DIR: + sys.path.insert(0, _TEST_DIR) +try: + from cuda_test_utils import ( # noqa: E402 + TemporaryCudaStream, + init_cuda_once, + temporary_sys_path, + ) +finally: + if _ADDED_TEST_DIR: + sys.path.remove(_TEST_DIR) + +with temporary_sys_path(_TEST_DIR): + from test_kv_cache_manager_v2 import create_config + +TOKENS_PER_BLOCK = 32 +PROMPT_BLOCKS = 16 +# 2 layers x (K + V) x 8 KiB per block: 32 KiB per block, so the GPU tier holds about 128 blocks. +GPU_QUOTA = 4 << 20 +HOST_QUOTA = 64 << 20 + + +class TestKvGpuPrefetchRuntime(unittest.TestCase): + def setUp(self) -> None: + init_cuda_once() + gc.collect() + gc.disable() + self.manager = KVCacheManager( + create_config( + tokens_per_block=TOKENS_PER_BLOCK, + gpu_quota=GPU_QUOTA, + host_quota=HOST_QUOTA, + disk_quota=0, + num_layers=2, + window_size=None, + sink_tokens=0, + ) + ) + + def tearDown(self) -> None: + gc.enable() + if hasattr(self, "manager"): + self.manager.shutdown() + del self.manager + + @staticmethod + def tokens(start, count): + return [TokenId(start + i) for i in range(count)] + + def commit(self, tokens): + with TemporaryCudaStream([]) as stream_holder: + stream = cast(CudaStream, stream_holder.handle) + kv_cache = self.manager.create_kv_cache(ReuseScope(), tokens) + self.assertTrue(kv_cache.resume(stream)) + self.assertTrue(kv_cache.resize(len(tokens))) + uncommitted = tokens[kv_cache.num_committed_tokens :] + if uncommitted: + kv_cache.commit(uncommitted) + kv_cache.close() + stream_holder.take_finish_event().synchronize() + + def levels(self, tokens): + kv_cache = self.manager.create_kv_cache(ReuseScope(), tokens) + try: + return list(kv_cache.cached_tokens_by_level) + finally: + kv_cache.close() + + def kvm_view(self, tokens): + return SimpleNamespace( + enable_block_reuse=True, + impl=self.manager, + _context_reuse_tokens=lambda req: tokens, + _derive_reuse_salt=lambda salt: None, + _kv_token_bytes_estimate=lambda: 100, + ) + + def evict_to_host(self, prompt): + self.commit(prompt) + # Enough newer prompts to push the first one out of the GPU tier. + n = PROMPT_BLOCKS * TOKENS_PER_BLOCK + for i in range(1, 12): + self.commit(self.tokens(i * 100_000, n)) + before = self.levels(prompt) + self.assertGreater(sum(before[GPU_LEVEL + 1 :]), 0, before) + return before + + def test_held_cache_snapshot_has_no_levels(self): + prompt = self.tokens(0, PROMPT_BLOCKS * TOKENS_PER_BLOCK) + self.evict_to_host(prompt) + kv_cache = self.manager.create_kv_cache(ReuseScope(), prompt) + try: + levels = kv_cache.get_page_storage_snapshot(0).cache_levels + self.assertTrue(all(level is None for level in levels), levels) + finally: + kv_cache.close() + + def test_prefetch_to_gpu_moves_host_prefix(self): + prompt = self.tokens(0, PROMPT_BLOCKS * TOKENS_PER_BLOCK) + before = self.evict_to_host(prompt) + req = SimpleNamespace(cache_salt=None, lora_task_id=None) + nbytes = kvm.KVCacheManagerV2.prefetch_to_gpu(self.kvm_view(prompt), req) + self.assertEqual(nbytes, sum(before[GPU_LEVEL + 1 :]) * 100) + after = self.levels(prompt) + self.assertEqual(sum(after), sum(before)) + self.assertEqual(sum(after[GPU_LEVEL + 1 :]), 0, after) + # Already on the GPU: nothing to move, no hold. + self.assertEqual(kvm.KVCacheManagerV2.prefetch_to_gpu(self.kvm_view(prompt), req), 0) + + +if __name__ == "__main__": + unittest.main() From bc6d981029354c03b23c426c87a10212b669cd7a Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 10:34:57 -0700 Subject: [PATCH 23/38] [None][fix] Disaggregated generation: one-hot draft probs on the first verify under rejection sampling A generation-only request received from the context server entered with dummy (or context-server) draft tokens, so the one-hot placeholder row was never requested (py_needs_onehot_draft_probs is set only for requests without draft tokens) and its first rejection-sampling verify read the slot's stale or uninitialized draft-probability row. A stale row accepts the dummy tokens; an uninitialized one can make chain_speculative_sampling emit an out-of-range token id. disagg_gen_first_draft_tokens (speculative/utils.py) now picks the first draft tokens at receive: under rejection sampling always dummy zeros with the one-hot row requested (no proposal row exists on this server); otherwise the previous behaviour. CPU UT in speculative/hw_agnostic. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- tensorrt_llm/_torch/pyexecutor/py_executor.py | 21 +++--- tensorrt_llm/_torch/speculative/utils.py | 27 ++++++++ .../test_disagg_gen_first_verify_onehot.py | 69 +++++++++++++++++++ 3 files changed, 107 insertions(+), 10 deletions(-) create mode 100644 tests/unittest/_torch/speculative/hw_agnostic/test_disagg_gen_first_verify_onehot.py diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor.py b/tensorrt_llm/_torch/pyexecutor/py_executor.py index a3e16431d570..1873e32f0318 100644 --- a/tensorrt_llm/_torch/pyexecutor/py_executor.py +++ b/tensorrt_llm/_torch/pyexecutor/py_executor.py @@ -75,7 +75,7 @@ from ..speculative.drafter import Drafter from ..speculative.spec_sampler_base import SampleStateTensorsSpec from ..speculative.speculation_gate import SpeculationGate -from ..speculative.utils import update_draft_len +from ..speculative.utils import disagg_gen_first_draft_tokens, update_draft_len from .adp_iter_stats import ADPIterStatsBuffer from .connectors.kv_cache_connector import KvCacheConnectorManager from .connectors.kv_cache_layout import build_kv_cache_layout_v2 @@ -8125,15 +8125,16 @@ def _prepare_disagg_gen_transmission_complete(self, scheduled_batch): req.decoding_iter = 1 req.py_decoding_iter = 1 first_gen_tokens = req.context_phase_params.first_gen_tokens - ctx_draft_tokens = req.context_phase_params.draft_tokens - if not ctx_draft_tokens and self.model_engine.enable_spec_decode: - # CTX has no speculative decoding — fill dummy draft tokens - # so model_engine builds the correct input shape (1 + draft_len - # tokens per gen request). Dummies will be rejected on verify, - # and the draft model will produce real tokens for next step. - ctx_draft_tokens = [0 - ] * self.model_engine.max_total_draft_tokens - req.py_draft_tokens = [] if ctx_draft_tokens is None else ctx_draft_tokens + # Dummy draft tokens when the context server sent none; under + # rejection sampling always dummies, verified against the one-hot + # placeholder row (no proposal row exists on this server). + req.py_draft_tokens, needs_onehot = disagg_gen_first_draft_tokens( + req.context_phase_params.draft_tokens, + getattr(self.model_engine, "spec_config", None), + self.model_engine.enable_spec_decode, + getattr(self.model_engine, "max_total_draft_tokens", 0)) + if needs_onehot: + req.py_needs_onehot_draft_probs = True beam_width = req.py_beam_width if not self._update_sampler_state_for_disagg_gen_request( req, beam_width, first_gen_tokens): diff --git a/tensorrt_llm/_torch/speculative/utils.py b/tensorrt_llm/_torch/speculative/utils.py index 813ad824262a..2bb15ed1d727 100644 --- a/tensorrt_llm/_torch/speculative/utils.py +++ b/tensorrt_llm/_torch/speculative/utils.py @@ -1044,6 +1044,33 @@ def resolve_draft_len(spec_config: Optional["DecodingBaseConfig"], return draft_len +def disagg_gen_first_draft_tokens(ctx_draft_tokens, spec_config, + enable_spec_decode: bool, + max_total_draft_tokens: int): + """Draft tokens a generation-only request (received from the context + server) verifies on its first step, and whether its draft-probability row + must be the one-hot placeholder. + + Without speculative decoding: the context server's tokens (None -> []). + Without rejection sampling: those tokens, or dummy zeros when there are + none (strict acceptance rejects a dummy unless the target samples it). + With rejection sampling: always dummy zeros verified against the one-hot + row at token 0. No proposal row was written on this server (the prefill + and any drafting ran on the context server), and a stale or uninitialized + row would accept the dummies or make chain_speculative_sampling emit an + out-of-range token id. + + Returns ``(draft_tokens, needs_onehot_draft_probs)``. + """ + if not enable_spec_decode: + return ([] if ctx_draft_tokens is None else ctx_draft_tokens), False + if getattr(spec_config, "use_rejection_sampling", False): + return [0] * max_total_draft_tokens, True + if not ctx_draft_tokens: + return [0] * max_total_draft_tokens, False + return ctx_draft_tokens, False + + def update_draft_len(model_engine: "ModelEngine", scheduled_batch: "ScheduledRequests", *, diff --git a/tests/unittest/_torch/speculative/hw_agnostic/test_disagg_gen_first_verify_onehot.py b/tests/unittest/_torch/speculative/hw_agnostic/test_disagg_gen_first_verify_onehot.py new file mode 100644 index 000000000000..b2d328ac6f4e --- /dev/null +++ b/tests/unittest/_torch/speculative/hw_agnostic/test_disagg_gen_first_verify_onehot.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Disaggregated generation: a request received from the context server verifies its first draft tokens against a +draft-probability row that this server never wrote (the prefill and any drafting ran on the context server). Under +rejection sampling it must verify dummy tokens against the one-hot placeholder row, or the stale / uninitialized row +accepts the dummies or makes chain_speculative_sampling emit an out-of-range token id.""" + +from types import SimpleNamespace + +import pytest +import torch + +from tensorrt_llm._torch.speculative.interface import SpecMetadata +from tensorrt_llm._torch.speculative.utils import disagg_gen_first_draft_tokens + +pytestmark = pytest.mark.cpu_only + +K, SLOTS, VOCAB = 3, 4, 8 +REJ = SimpleNamespace(use_rejection_sampling=True) +STRICT = SimpleNamespace(use_rejection_sampling=False) + + +@pytest.mark.parametrize("ctx_drafts", [None, [], [5, 6, 7]]) +def test_rejection_always_verifies_dummies_against_the_one_hot_row(ctx_drafts): + assert disagg_gen_first_draft_tokens(ctx_drafts, REJ, True, K) == ([0] * K, True) + + +def test_strict_keeps_context_server_drafts(): + assert disagg_gen_first_draft_tokens([5, 6, 7], STRICT, True, K) == ([5, 6, 7], False) + + +@pytest.mark.parametrize("ctx_drafts", [None, []]) +def test_strict_without_drafts_gets_dummies(ctx_drafts): + assert disagg_gen_first_draft_tokens(ctx_drafts, STRICT, True, K) == ([0] * K, False) + + +@pytest.mark.parametrize("ctx_drafts, expected", [(None, []), ([5, 6, 7], [5, 6, 7])]) +def test_no_spec_decode_is_unchanged(ctx_drafts, expected): + assert disagg_gen_first_draft_tokens(ctx_drafts, REJ, False, K) == (expected, False) + + +def test_flagged_slot_gets_a_one_hot_row_at_the_dummy(): + # The flag makes input preparation hand the slot to write_padding_onehot_draft_probs; its row then matches the + # dummy draft (token 0) in full, and a slot that is not flagged keeps its stale row. + md = SimpleNamespace( + use_rejection_sampling=True, + draft_probs=torch.full((SLOTS, K, VOCAB), 0.5), + draft_probs_last_dim=VOCAB, + draft_probs_vocab_size=VOCAB, + ) + tokens, needs_onehot = disagg_gen_first_draft_tokens(None, REJ, True, K) + assert needs_onehot + SpecMetadata.write_padding_onehot_draft_probs(md, [2], K) + expected = torch.zeros(K, VOCAB) + expected[:, tokens[0]] = 1.0 + assert torch.equal(md.draft_probs[2], expected) + assert torch.equal(md.draft_probs[1], torch.full((K, VOCAB), 0.5)) From fd382342cd5b1471d4bfb1a21787e12d58bbd33d Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 12:05:36 -0700 Subject: [PATCH 24/38] [None][fix] Gate TOPP_V6 to sm_107 and keep the token-array copy to host arrays - toppRenormDetV3Sm103: the TOPP_V6 chain-prefetch summation now runs only on sm_107 devices (the same per-device compute-capability check as REJ v3); other devices, including GB300 callers, run the staged loop as before. TRTLLM_M31_DISABLE_TOPP_V6 still turns it off everywhere. Each kernel variant sets its own shared-memory attribute. - LlmRequest binding: the int32 token-array fast path accepts host arrays only (nb::device::cpu); a device array takes the element-wise cast instead of a memcpy from a device pointer. Unit test with a CUDA tensor. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../minimaxM31/toppRenormDetV3Sm103.cu | 45 ++++++++++++++++--- .../nanobind/batch_manager/bindings.cpp | 4 +- .../executor/test_request_token_array.py | 15 +++++++ 3 files changed, 55 insertions(+), 9 deletions(-) diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu index 2f16fe10712f..e5a67132c804 100644 --- a/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu +++ b/cpp/tensorrt_llm/kernels/minimaxM31/toppRenormDetV3Sm103.cu @@ -32,6 +32,7 @@ #include #include #include +#include #include #include @@ -873,10 +874,12 @@ __global__ void __launch_bounds__(kThreads, 1) toppRenormDetV3Kernel( namespace { -// TOPP_V6: on unless TRTLLM_M31_DISABLE_TOPP_V6 is set to 1 / true / on / yes (read once per process). +// TOPP_V6: the chain-prefetch summation on sm_107 devices unless TRTLLM_M31_DISABLE_TOPP_V6 is set to 1 / true / on / +// yes; other devices run the staged loop. The switch is read once per process and the compute capability once per +// device. bool chainPrefetchOn() { - static bool const on = [] + static bool const allowed = [] { char const* v = std::getenv("TRTLLM_M31_DISABLE_TOPP_V6"); if (v == nullptr) @@ -886,15 +889,34 @@ bool chainPrefetchOn() return !(std::strcmp(v, "1") == 0 || strcasecmp(v, "true") == 0 || strcasecmp(v, "on") == 0 || strcasecmp(v, "yes") == 0); }(); - return on; + if (!allowed) + { + return false; + } + static std::once_flag smOnce[64]; + static bool sm107[64] = {}; + int device = 0; + if (cudaGetDevice(&device) != cudaSuccess) + { + return false; + } + std::call_once(smOnce[device % 64], + [device] + { + int major = 0; + int minor = 0; + cudaDeviceGetAttribute(&major, cudaDevAttrComputeCapabilityMajor, device); + cudaDeviceGetAttribute(&minor, cudaDevAttrComputeCapabilityMinor, device); + sm107[device % 64] = major == 10 && minor == 7; + }); + return sm107[device % 64]; } -template -cudaError_t launchV3(cudaLaunchConfig_t& cfg, float const* probs, float* out, float const* topP, int d, +template +cudaError_t launchV3Kernel(cudaLaunchConfig_t& cfg, float const* probs, float* out, float const* topP, int d, toppv3::RowWorkspace* ws, int* keptOut) { - auto* kernel = chainPrefetchOn() ? toppv3::toppRenormDetV3Kernel - : toppv3::toppRenormDetV3Kernel; + auto* kernel = toppv3::toppRenormDetV3Kernel; static cudaError_t const attrErr = cudaFuncSetAttribute( kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, static_cast(toppv3::dynSmemBytes())); if (attrErr != cudaSuccess) @@ -905,6 +927,15 @@ cudaError_t launchV3(cudaLaunchConfig_t& cfg, float const* probs, float* out, fl return cudaLaunchKernelEx(&cfg, kernel, probs, out, topP, d, ws, keptOut); } +template +cudaError_t launchV3(cudaLaunchConfig_t& cfg, float const* probs, float* out, float const* topP, int d, + toppv3::RowWorkspace* ws, int* keptOut) +{ + return chainPrefetchOn() + ? launchV3Kernel(cfg, probs, out, topP, d, ws, keptOut) + : launchV3Kernel(cfg, probs, out, topP, d, ws, keptOut); +} + template cudaError_t launchV3Copies(cudaLaunchConfig_t& cfg, bool listPath, bool wideHist, float const* probs, float* out, float const* topP, int d, toppv3::RowWorkspace* ws, int* keptOut) diff --git a/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp b/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp index 30c30c17af68..828b2ef55469 100644 --- a/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp @@ -406,10 +406,10 @@ void initBindings(nb::module_& m) auto lora_config_tensor_ptr = makeOptionalTensor(lora_config); auto encoder_input_features_tensor_ptr = makeOptionalTensor(encoder_input_features); - // input_tokens: a 1-D contiguous int32 ndarray is memcpy'd (no per-element + // input_tokens: a 1-D contiguous int32 host ndarray is memcpy'd (no per-element // PyLong cast, O(ISL) under the GIL); anything else uses the sequence cast. std::vector input_tokens; - nb::ndarray, nb::c_contig> input_tokens_arr; + nb::ndarray, nb::c_contig, nb::device::cpu> input_tokens_arr; if (nb::try_cast(input_tokens_obj, input_tokens_arr, /*convert=*/false)) { input_tokens.resize(input_tokens_arr.shape(0)); diff --git a/tests/unittest/_torch/executor/test_request_token_array.py b/tests/unittest/_torch/executor/test_request_token_array.py index 0e96ef0ee583..6abcb1cab037 100644 --- a/tests/unittest/_torch/executor/test_request_token_array.py +++ b/tests/unittest/_torch/executor/test_request_token_array.py @@ -20,6 +20,7 @@ import numpy as np import pytest +import torch from tensorrt_llm._torch.pyexecutor import llm_request as lr_mod from tensorrt_llm.bindings import executor as tllm @@ -145,3 +146,17 @@ def make(tokens): from_array = make(np.asarray(PROMPT, dtype=np.int32)) from_list = make(list(PROMPT)) assert from_array.get_tokens(0) == from_list.get_tokens(0) == PROMPT + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA tensor") +def test_llm_request_reads_a_device_array_through_the_sequence_path(): + # Only a host int32 array is copied as raw memory; a CUDA tensor takes the element-wise cast. + tokens = torch.tensor(PROMPT, dtype=torch.int32, device="cuda") + llm_req = lr_mod.LlmRequest( + request_id=4, + max_new_tokens=4, + input_tokens=tokens, + sampling_config=tllm.SamplingConfig(1), + is_streaming=False, + ) + assert llm_req.get_tokens(0) == PROMPT From 0f4ab317039fb8590ce943b9ddb71c88ec14c987 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 13:34:43 -0700 Subject: [PATCH 25/38] [None][fix] Arm the shared MiniMax-M3.1 sm_107 paths only for the targets that opt in Shared paths this PR added were on for every model that reached them. Each is now a model opt-in that only the sm_107 MiniMax-M3.1 targets arm, and keeps its kill switch: - RuntimeHooks fields (default False): draft_kv_reserve_guard (armed by the KV-cache creator before sizing), kv_gpu_prefetch, context_token_array, msa_prefill_fork, proxy_ms_reuse, context_onehot_index_fill and dflash_host_ctx_alloc (armed by the executor before warmup). - Ahead-of-time CuTe DSL objects are served only after routing selects an sm_107 target (cute_aot.arm, before the model builds its kernels); an unarmed process compiles with no lookup, export or log line. - The TP2 / TP4 / TP8 targets arm all of them, so their behaviour is unchanged. Unit tests: the opt-in and kill-switch semantics, the unarmed AOT path, and a source check that every sm_107 target arms each shared opt-in. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../modeling.py | 7 ++ .../modeling.py | 7 ++ .../modeling.py | 7 ++ .../modeling_v2/models/minimax_m3/routing.py | 5 ++ .../attention/backends/fmha/msa_prefill.py | 14 +++- .../backends/sparse/minimax_m3/msa_indexer.py | 14 +++- tensorrt_llm/_torch/cute_aot.py | 17 +++++ tensorrt_llm/_torch/pyexecutor/_util.py | 3 + .../kv_cache/kv_cache_manager_v2.py | 38 +++++----- tensorrt_llm/_torch/pyexecutor/llm_request.py | 15 ++-- tensorrt_llm/_torch/pyexecutor/py_executor.py | 38 ++++++++-- .../_torch/pyexecutor/runtime_hooks.py | 24 +++++++ tensorrt_llm/_torch/speculative/dflash.py | 12 +++- tensorrt_llm/_torch/speculative/interface.py | 14 +++- .../integration/test_lists/test-db/l0_cpu.yml | 6 +- .../executor/kv_cache/test_kv_gpu_prefetch.py | 12 +++- ...st_modeling_v2_m31_tp2wide_prefill_attn.py | 2 + .../test_m31_sm107_shared_opt_ins.py | 69 +++++++++++++++++++ tests/unittest/_torch/test_cute_aot.py | 14 ++++ 19 files changed, 278 insertions(+), 40 deletions(-) create mode 100644 tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py index 2153e950db70..544a098835d3 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py @@ -8825,6 +8825,13 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: fp8_saturating_casts=True, msa_kv_cache_fixes=True, draft_pool_fixes=True, + draft_kv_reserve_guard=True, + kv_gpu_prefetch=True, + context_token_array=True, + msa_prefill_fork=True, + proxy_ms_reuse=True, + context_onehot_index_fill=True, + dflash_host_ctx_alloc=True, persist_flashinfer_mxfp8_tactics=True, # Bundled name-keyed tactic pins (m31_pins); kill: TRTLLM_M31_DISABLE_BUNDLED_PINS=1. install_tactic_pins=lambda rank: m31_pins.install("tp2", rank), diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py index 921c65470472..f73189312685 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py @@ -6100,6 +6100,13 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: fp8_saturating_casts=True, msa_kv_cache_fixes=True, draft_pool_fixes=True, + draft_kv_reserve_guard=True, + kv_gpu_prefetch=True, + context_token_array=True, + msa_prefill_fork=True, + proxy_ms_reuse=True, + context_onehot_index_fill=True, + dflash_host_ctx_alloc=True, first_content_emit_iters=FIRST_CONTENT_EMIT_ITERS, mxfp8_pinned_tactics=True, late_compile_detect=True, diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py index 184b2cc201f1..adc703bde461 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py @@ -6537,6 +6537,13 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: fp8_saturating_casts=True, msa_kv_cache_fixes=True, draft_pool_fixes=True, + draft_kv_reserve_guard=True, + kv_gpu_prefetch=True, + context_token_array=True, + msa_prefill_fork=True, + proxy_ms_reuse=True, + context_onehot_index_fill=True, + dflash_host_ctx_alloc=True, first_content_emit_iters=FIRST_CONTENT_EMIT_ITERS, mxfp8_pinned_tactics=True, late_compile_detect=True, diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py index 20bb8f2682cc..1572c8b60bfe 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py @@ -217,9 +217,14 @@ def route( and m31_spec_miss is None and kv_cache ): + from tensorrt_llm._torch import cute_aot + from .fake_ops import register register() + # The sm_107 targets ship ahead-of-time CuTe DSL objects (cute_aot_data); serve them before the model builds + # its kernels. + cute_aot.arm() if parallel == "tp2": from .minimax_m31_nvfp4__sm_107__tp2.modeling import ModelingV2MiniMaxM31Nvfp4Sm107Tp2 diff --git a/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py b/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py index a116c5c43632..7a5c0824792f 100644 --- a/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py +++ b/tensorrt_llm/_torch/attention/backends/fmha/msa_prefill.py @@ -316,14 +316,26 @@ def _combine_takes(fn, name: str) -> bool: _TP2CTX_BF16X_STATE: dict = {"taken": 0, "skip": {}, "q8kv4_taken": 0, "e4m3x_taken": 0} +# The BF16X / E4M3X fork is a model opt-in (RuntimeHooks.msa_prefill_fork), armed by the executor (arm_prefill_fork). +_PREFILL_FORK_ARMED = False + + +def arm_prefill_fork() -> None: + """Let the MSA NVFP4 sparse prefill take the BF16X / E4M3X fork in this process (RuntimeHooks.msa_prefill_fork).""" + global _PREFILL_FORK_ARMED + _PREFILL_FORK_ARMED = True + def _tp2ctx_bf16x_run(): """(run, reason): fmha_sm100.prefill_bf16x.run_prefill (BF16X fork m3b) when TP2CTX_BF16X may run here, else None. - Default ON; TRTLLM_M31_DISABLE_TP2CTX_BF16X=1 (read per call) keeps the CuTe-DSL forward. The probe is cached. + Armed by a model opt-in; TRTLLM_M31_DISABLE_TP2CTX_BF16X=1 (read per call) keeps the CuTe-DSL forward. The probe + is cached. """ import os + if not _PREFILL_FORK_ARMED: + return None, "not_armed" if os.environ.get("TRTLLM_M31_DISABLE_TP2CTX_BF16X", "0") == "1": return None, "disabled_by_env" if "run" not in _TP2CTX_BF16X_STATE: diff --git a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_indexer.py b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_indexer.py index 379723657668..8092d114fea7 100644 --- a/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_indexer.py +++ b/tensorrt_llm/_torch/attention/backends/sparse/minimax_m3/msa_indexer.py @@ -376,8 +376,18 @@ def _group_max_reduce( # The proxy writes every valid block and selection masks the rest. Reusing the -# step buffer avoids allocating and filling it at each sparse layer. -_PROXY_MS_REUSE = os.environ.get("TRTLLM_M31_DISABLE_PROXY_MS_REUSE", "0") != "1" +# step buffer avoids allocating and filling it at each sparse layer. A model +# opt-in (RuntimeHooks.proxy_ms_reuse), armed by the executor +# (arm_proxy_ms_reuse); kill switch TRTLLM_M31_DISABLE_PROXY_MS_REUSE=1. +_PROXY_MS_REUSE = False + + +def arm_proxy_ms_reuse() -> None: + """Reuse the step's proxy max-score buffer in this process (RuntimeHooks.proxy_ms_reuse), unless killed.""" + global _PROXY_MS_REUSE + _PROXY_MS_REUSE = os.environ.get("TRTLLM_M31_DISABLE_PROXY_MS_REUSE", "0") != "1" + + _proxy_ms_step: Optional[tuple] = None # (proxy_plan, tokens, max_score) diff --git a/tensorrt_llm/_torch/cute_aot.py b/tensorrt_llm/_torch/cute_aot.py index 7d4bd02d42ed..64033c5eba41 100644 --- a/tensorrt_llm/_torch/cute_aot.py +++ b/tensorrt_llm/_torch/cute_aot.py @@ -56,6 +56,17 @@ _counts = {"loaded": 0, "compiled": 0} +# Shipped objects are served only after a model that ships them arms the loader (``arm``, at target routing, before +# the model builds its kernels); until then every key compiles in-process with no lookup, export or log line. +_armed = False + + +def arm() -> None: + """Serve shipped objects (and export, with ``TRTLLM_M31_CUTE_AOT_EXPORT_DIR``) in this process.""" + global _armed + _armed = True + + def _disabled() -> bool: return os.environ.get(DISABLE_ENV, "0").strip().lower() in ("1", "true", "on", "yes") @@ -170,6 +181,12 @@ def compile_or_load( ) -> Any: """The compiled function of ``key`` built from ``sources``: the shipped object when there is one, else ``compile_fn()``.""" + if not _armed: + with _lock: + cached = _loaded.get(("", key)) + if cached is None: + cached = _loaded[("", key)] = compile_fn() + return cached tool = toolchain() key = full_key(key, sources) with _lock: diff --git a/tensorrt_llm/_torch/pyexecutor/_util.py b/tensorrt_llm/_torch/pyexecutor/_util.py index 2ac1c26d5f46..3f6f2bd8459a 100644 --- a/tensorrt_llm/_torch/pyexecutor/_util.py +++ b/tensorrt_llm/_torch/pyexecutor/_util.py @@ -834,6 +834,9 @@ def _arm_kv_cache_opt_ins(hooks) -> None: if hooks.draft_pool_fixes: from .kv_cache import kv_cache_manager_v2 kv_cache_manager_v2.arm_draft_pool_fixes() + if hooks.draft_kv_reserve_guard: + from .kv_cache import kv_cache_manager_v2 + kv_cache_manager_v2.arm_draft_kv_reserve_guard() if hooks.msa_kv_cache_fixes: from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import \ cache_manager as msa_cache_manager diff --git a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py index e3b0ca1650be..4370c306e0a4 100644 --- a/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py +++ b/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py @@ -559,28 +559,26 @@ def max_admissible_sequences( return max_num_sequences * (2 if needs_extra_leases else 1) -# DRAFT_KV_RESERVE: the draft KV cache manager reserves the DFlash / DSpark drafter's block width per -# generation step (see _get_generation_kv_capacity). Kill switch TRTLLM_M31_DISABLE_DRAFT_KV_RESERVE=1, -# read at import. -_DRAFT_KV_RESERVE = os.environ.get( - "TRTLLM_M31_DISABLE_DRAFT_KV_RESERVE", "0" -).strip().lower() not in ( - "1", - "true", - "on", - "yes", -) +def _m31_killed(name: str) -> bool: + return os.environ.get(name, "0").strip().lower() in ("1", "true", "on", "yes") -# DRAFT_KV_GUARD: a DFlash / DSpark draft KV cache manager reserves the guard page (page 0) by -# default (see KVCacheManagerV2.__init__). Kill switch TRTLLM_M31_DISABLE_DRAFT_KV_GUARD=1, read at -# import; TRTLLM_KV_GUARD_PAGE, when set, still wins. -_DRAFT_KV_GUARD = os.environ.get("TRTLLM_M31_DISABLE_DRAFT_KV_GUARD", "0").strip().lower() not in ( - "1", - "true", - "on", - "yes", -) +# DRAFT_KV_RESERVE: the draft KV cache manager reserves the DFlash / DSpark drafter's block width per +# generation step (see _get_generation_kv_capacity). DRAFT_KV_GUARD: a DFlash / DSpark draft KV cache +# manager reserves the guard page (page 0) (see KVCacheManagerV2.__init__); TRTLLM_KV_GUARD_PAGE, when +# set, still wins. Both are a model opt-in (RuntimeHooks.draft_kv_reserve_guard), armed process-wide by +# the KV-cache creator before any sizing (arm_draft_kv_reserve_guard). Kill switches +# TRTLLM_M31_DISABLE_DRAFT_KV_RESERVE=1 / TRTLLM_M31_DISABLE_DRAFT_KV_GUARD=1. +_DRAFT_KV_RESERVE = False +_DRAFT_KV_GUARD = False + + +def arm_draft_kv_reserve_guard() -> None: + """Take the drafter block-width reserve and the guard page in this process + (RuntimeHooks.draft_kv_reserve_guard), unless their kill switches are set.""" + global _DRAFT_KV_RESERVE, _DRAFT_KV_GUARD + _DRAFT_KV_RESERVE = not _m31_killed("TRTLLM_M31_DISABLE_DRAFT_KV_RESERVE") + _DRAFT_KV_GUARD = not _m31_killed("TRTLLM_M31_DISABLE_DRAFT_KV_GUARD") def _get_generation_kv_capacity(spec_config, *, is_draft: bool) -> tuple[int, int]: diff --git a/tensorrt_llm/_torch/pyexecutor/llm_request.py b/tensorrt_llm/_torch/pyexecutor/llm_request.py index aaf3ddfb001c..1337512a939c 100644 --- a/tensorrt_llm/_torch/pyexecutor/llm_request.py +++ b/tensorrt_llm/_torch/pyexecutor/llm_request.py @@ -1531,15 +1531,22 @@ def is_multimodal_encoder_ready(request: LlmRequest) -> bool: # Hand the prompt tokens to LlmRequest as the executor request's int32 array (one copy each way) instead of a -# list[int] round trip, O(ISL) Python objects under the GIL. On by default for context-only requests (a disaggregated -# context server); TRTLLM_M31_ENABLE_L1_TOKEN_ARRAY=1 extends it to every request type. Kill switch (wins): +# list[int] round trip, O(ISL) Python objects under the GIL. A model opt-in (RuntimeHooks.context_token_array), +# armed by the executor (arm_context_token_array): context-only requests (a disaggregated context server); +# TRTLLM_M31_ENABLE_L1_TOKEN_ARRAY=1 extends it to every request type. Kill switch (wins): # TRTLLM_M31_DISABLE_L1_TOKEN_ARRAY=1. -_L1_TOKEN_ARRAY = os.environ.get("TRTLLM_M31_DISABLE_L1_TOKEN_ARRAY", - "0") != "1" +_L1_TOKEN_ARRAY = False _L1_ALL_REQUESTS = os.environ.get("TRTLLM_M31_ENABLE_L1_TOKEN_ARRAY", "0") == "1" +def arm_context_token_array() -> None: + """Take the int32 prompt-token path in this process (RuntimeHooks.context_token_array), unless killed.""" + global _L1_TOKEN_ARRAY + _L1_TOKEN_ARRAY = os.environ.get("TRTLLM_M31_DISABLE_L1_TOKEN_ARRAY", + "0") != "1" + + def _l1_input_tokens(executor_request: ExecutorRequest): """The executor request's prompt tokens for LlmRequest: its int32 array where the array path applies, else the list.""" diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor.py b/tensorrt_llm/_torch/pyexecutor/py_executor.py index 1873e32f0318..a35e7c4d136b 100644 --- a/tensorrt_llm/_torch/pyexecutor/py_executor.py +++ b/tensorrt_llm/_torch/pyexecutor/py_executor.py @@ -134,9 +134,12 @@ _V2_RETENTION_IGNORED_LOG_KEY = "kv_cache_manager_v2_retention_config_ignored" # Waiting context requests have their host-tier prefix prefetched to the GPU off the -# execution stream and are admitted after a short, rank-agreed hold. Kill switch. -_KV_GPU_PREFETCH = os.environ.get("TRTLLM_M31_DISABLE_KV_GPU_PREFETCH", - "0") != "1" +# execution stream and are admitted after a short, rank-agreed hold. A model opt-in +# (RuntimeHooks.kv_gpu_prefetch), armed by _arm_runtime_opt_ins; kill switch +# TRTLLM_M31_DISABLE_KV_GPU_PREFETCH=1. +_KV_GPU_PREFETCH_KILLED = os.environ.get("TRTLLM_M31_DISABLE_KV_GPU_PREFETCH", + "0") == "1" +_KV_GPU_PREFETCH = False _KV_GPU_PREFETCH_MAX = int(os.environ.get("TRTLLM_M31_KV_GPU_PREFETCH_MAX", "2")) _KV_GPU_PREFETCH_BYTES_PER_STEP = int(1.5e9) @@ -394,8 +397,13 @@ def _arm_runtime_opt_ins(hooks: RuntimeHooks) -> bool: All are off for a model that declares none: the SM107 MXFP8 GEMM tactic table (the GEMM runner does not know the model, so it is armed - here), the MSA saturating E4M3 casts and the late-compile detector. + here), the MSA saturating E4M3 casts, the host-tier KV prefix prefetch, + the context-only int32 prompt tokens, the MSA NVFP4 sparse-prefill fork, + the indexer proxy max-score buffer reuse, the context one-hot draft-prob + index_fill, the DFlash host-side context allocation and the late-compile + detector. Each keeps its TRTLLM_M31_DISABLE_* kill switch. """ + global _KV_GPU_PREFETCH if hooks.mxfp8_pinned_tactics: # Imported on use: only an opted-in model needs the table. from tensorrt_llm._torch.custom_ops import mxfp8_cute_pinned_tactics @@ -404,6 +412,24 @@ def _arm_runtime_opt_ins(hooks: RuntimeHooks) -> bool: from tensorrt_llm._torch.attention.backends.sparse.minimax_m3.kernels import \ paged_cache paged_cache.arm_fp8_saturation() + if hooks.kv_gpu_prefetch: + _KV_GPU_PREFETCH = not _KV_GPU_PREFETCH_KILLED + if hooks.context_token_array: + from . import llm_request + llm_request.arm_context_token_array() + if hooks.msa_prefill_fork: + from tensorrt_llm._torch.attention.backends.fmha import msa_prefill + msa_prefill.arm_prefill_fork() + if hooks.proxy_ms_reuse: + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import \ + msa_indexer + msa_indexer.arm_proxy_ms_reuse() + if hooks.context_onehot_index_fill: + from tensorrt_llm._torch.speculative import interface + interface.arm_onehot_index_fill() + if hooks.dflash_host_ctx_alloc: + from tensorrt_llm._torch.speculative import dflash + dflash.arm_host_ctx_alloc() return hooks.late_compile_detect @@ -4085,8 +4111,10 @@ def _revert_ctx_alloc(self, dropped_context_requests): def _kv_gpu_prefetch_state(self) -> str: """Whether the host-tier prefix prefetch runs in this executor, and if not, why.""" - if not _KV_GPU_PREFETCH: + if _KV_GPU_PREFETCH_KILLED: return "disabled (kill switch)" + if not _KV_GPU_PREFETCH: + return "inactive (the model does not opt in)" if not self._is_kv_manager_v2: return "inactive (KV cache manager v1)" if not self.kv_cache_manager.enable_block_reuse: diff --git a/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py b/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py index 17735f9cda46..3a3786a65c64 100644 --- a/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py +++ b/tensorrt_llm/_torch/pyexecutor/runtime_hooks.py @@ -240,6 +240,30 @@ class RuntimeHooks: # Standalone-drafter KV pool: ceil(heads / tp) per-rank cost and the history-first context resize # (kv_cache_manager_v2.py). Armed process-wide by the KV-cache creator before any sizing. draft_pool_fixes: bool = False + # DFlash / DSpark draft KV manager: reserve the drafter's block width per generation step and keep page 0 as an + # unowned guard page, so drafter writes through unallocated block-table entries cannot land in another request's + # page (kv_cache_manager_v2.py). Armed process-wide by the KV-cache creator before any sizing. Kill switches + # TRTLLM_M31_DISABLE_DRAFT_KV_RESERVE / TRTLLM_M31_DISABLE_DRAFT_KV_GUARD. + draft_kv_reserve_guard: bool = False + # KV cache manager V2 with block reuse: prefetch waiting context requests' host-tier prefix to the GPU and admit + # them after a short rank-agreed hold (py_executor.py). Kill switch TRTLLM_M31_DISABLE_KV_GPU_PREFETCH. + kv_gpu_prefetch: bool = False + # Hand a context-only request's prompt tokens to LlmRequest as the executor request's int32 array instead of a + # list (llm_request.py). Kill switch TRTLLM_M31_DISABLE_L1_TOKEN_ARRAY. + context_token_array: bool = False + # On SM107, run the MSA NVFP4 sparse prefill through the fmha_sm100 BF16X / E4M3X fork when it is installed and + # the geometry matches (attention/backends/fmha/msa_prefill.py). Kill switch TRTLLM_M31_DISABLE_TP2CTX_BF16X. + msa_prefill_fork: bool = False + # Reuse the step's MSA indexer proxy max-score buffer across sparse layers (msa_indexer.py). Kill switch + # TRTLLM_M31_DISABLE_PROXY_MS_REUSE. + proxy_ms_reuse: bool = False + # Write the context one-hot draft-prob rows with index_fill_ instead of an advanced-index put that synchronizes + # the host (speculative/interface.py; the same elements). Kill switch TRTLLM_M31_DISABLE_ONEHOT_INDEX_FILL. + context_onehot_index_fill: bool = False + # DFlash / DSpark: compute a context chunk's draft allocation on the host when the draft block table counts full + # rows, instead of reading it back from the device (speculative/dflash.py). Kill switch + # TRTLLM_DFLASH_DISABLE_HOST_CTX_ALLOC. + dflash_host_ctx_alloc: bool = False # Skip the per-step Philox seed and offset uploads on all-greedy # one-model speculative batches (the argmax path never reads them). skip_greedy_rng_upload: bool = False diff --git a/tensorrt_llm/_torch/speculative/dflash.py b/tensorrt_llm/_torch/speculative/dflash.py index 388812061097..e5e3d22b2690 100644 --- a/tensorrt_llm/_torch/speculative/dflash.py +++ b/tensorrt_llm/_torch/speculative/dflash.py @@ -520,7 +520,17 @@ def dflash_allocated_ctx_limit( return (block_counts * page_size - block_size).clamp(min=0) -_HOST_CTX_ALLOC_DISABLED = os.environ.get("TRTLLM_DFLASH_DISABLE_HOST_CTX_ALLOC", "0") == "1" +# The host-side context allocation is a model opt-in (RuntimeHooks.dflash_host_ctx_alloc), armed by the executor +# (arm_host_ctx_alloc); kill switch TRTLLM_DFLASH_DISABLE_HOST_CTX_ALLOC=1. +_HOST_CTX_ALLOC_DISABLED = True + + +def arm_host_ctx_alloc() -> None: + """Compute the context allocation on the host where it is known (RuntimeHooks.dflash_host_ctx_alloc).""" + global _HOST_CTX_ALLOC_DISABLED + _HOST_CTX_ALLOC_DISABLED = os.environ.get("TRTLLM_DFLASH_DISABLE_HOST_CTX_ALLOC", "0") == "1" + + _HOST_CTX_ALLOC_CHECK = os.environ.get("TRTLLM_DFLASH_CHECK_HOST_CTX_ALLOC", "0") == "1" # Bound on the disagg drafter-context maps (context-side store events, generation-side seeds): an # entry is normally consumed within an iteration, so only requests that failed or were cancelled in diff --git a/tensorrt_llm/_torch/speculative/interface.py b/tensorrt_llm/_torch/speculative/interface.py index 19b81228f26f..987362acfb4b 100644 --- a/tensorrt_llm/_torch/speculative/interface.py +++ b/tensorrt_llm/_torch/speculative/interface.py @@ -1808,9 +1808,17 @@ def _padded_gen_draft_tokens(spec_metadata: SpecMetadata, num_gens: int, # Context one-hot draft-prob rows are written with index_fill_ (non-blocking) instead -# of an advanced-index put that synchronizes the host with the GPU. Kill switch. -_ONEHOT_INDEX_FILL = os.environ.get("TRTLLM_M31_DISABLE_ONEHOT_INDEX_FILL", - "0") != "1" +# of an advanced-index put that synchronizes the host with the GPU. A model opt-in +# (RuntimeHooks.context_onehot_index_fill), armed by the executor +# (arm_onehot_index_fill); kill switch TRTLLM_M31_DISABLE_ONEHOT_INDEX_FILL=1. +_ONEHOT_INDEX_FILL = False + + +def arm_onehot_index_fill() -> None: + """Write the context one-hot rows with index_fill_ in this process (RuntimeHooks.context_onehot_index_fill).""" + global _ONEHOT_INDEX_FILL + _ONEHOT_INDEX_FILL = os.environ.get("TRTLLM_M31_DISABLE_ONEHOT_INDEX_FILL", + "0") != "1" class SpecWorkerBase(nn.Module, ABC): diff --git a/tests/integration/test_lists/test-db/l0_cpu.yml b/tests/integration/test_lists/test-db/l0_cpu.yml index a3a7566ced3d..c9af143873af 100644 --- a/tests/integration/test_lists/test-db/l0_cpu.yml +++ b/tests/integration/test_lists/test-db/l0_cpu.yml @@ -54,10 +54,12 @@ l0_cpu: - unittest/_torch/peft - unittest/_torch/memory - unittest/_torch/modeling - # modeling_v2 routing, routing-table claims and the MiniMax-M3.1 tactic pins: config shapes, a monkeypatched - # compute capability and stub runners, no device. The rest of unittest/_torch/modeling_v2 needs a GPU. + # modeling_v2 routing, routing-table claims, the MiniMax-M3.1 tactic pins and the sm_107 targets' shared opt-ins: + # config shapes, a monkeypatched compute capability, stub runners and source parsing, no device. The rest of + # unittest/_torch/modeling_v2 needs a GPU. - unittest/_torch/modeling_v2/test_modeling_v2_claims.py - unittest/_torch/modeling_v2/test_m31_pins.py + - unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py - unittest/_torch/modeling_v2/test_modeling_v2_routing.py - unittest/_torch/models/test_minimax_m3.py::test_minimax_m3_fp8_indexer_rejects_different_qk_norm_epsilons - unittest/_torch/models/test_minimax_m3.py::test_minimax_m3_moe_reduces_only_local_terms diff --git a/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py b/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py index 6bb018884122..40e5673de231 100644 --- a/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py +++ b/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py @@ -9,6 +9,7 @@ import tensorrt_llm._torch.pyexecutor.kv_cache.kv_cache_manager_v2 as kvm import tensorrt_llm._torch.pyexecutor.py_executor as pe +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks pytestmark = pytest.mark.cpu_only @@ -104,8 +105,15 @@ def test_prefetch_failure_means_no_hold(): cache.close.assert_called_once() -def test_kill_switch_default_on(): - assert pe._KV_GPU_PREFETCH is True +@pytest.mark.parametrize( + "opt_in,killed,armed", [(False, False, False), (True, False, True), (True, True, False)] +) +def test_prefetch_is_a_model_opt_in(monkeypatch, opt_in, killed, armed): + # Off unless the model opts in (RuntimeHooks.kv_gpu_prefetch); the kill switch wins over the opt-in. + monkeypatch.setattr(pe, "_KV_GPU_PREFETCH", False) + monkeypatch.setattr(pe, "_KV_GPU_PREFETCH_KILLED", killed) + pe._arm_runtime_opt_ins(RuntimeHooks(kv_gpu_prefetch=opt_in)) + assert pe._KV_GPU_PREFETCH is armed @pytest.mark.parametrize("hold", [0, 1, 3]) diff --git a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_prefill_attn.py b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_prefill_attn.py index ec902253c091..c83dee9566b7 100644 --- a/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_prefill_attn.py +++ b/tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/test_modeling_v2_m31_tp2wide_prefill_attn.py @@ -228,6 +228,8 @@ def test_opt_ins_and_e4m3x_are_bitwise_the_stock_call(monkeypatch, reqs) -> None ) require_msa_module() + # The fork is a model opt-in (RuntimeHooks.msa_prefill_fork) the sm_107 targets arm. + monkeypatch.setattr(mp, "_PREFILL_FORK_ARMED", True) q, k, v, sc, kbi, md, kg, vg = _case(reqs, seed=sum(p for _, p in reqs) + len(reqs)) outs = {} arms = { diff --git a/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py b/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py new file mode 100644 index 000000000000..9bfc85b853e7 --- /dev/null +++ b/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""The shared-path opt-ins the MiniMax-M3.1 sm_107 targets arm: every target's ``_runtime_hooks`` passes each of them +as a literal True, and each is off in the default RuntimeHooks (so a model that declares none, GB300's included, keeps +the stock path). Read from modeling.py's source, so the heavy modeling modules are not imported. CPU only.""" + +import ast +import dataclasses +from pathlib import Path + +import pytest + +import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family +from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks + +pytestmark = pytest.mark.cpu_only + +SHARED_OPT_INS = ( + "draft_pool_fixes", + "msa_kv_cache_fixes", + "draft_kv_reserve_guard", + "kv_gpu_prefetch", + "context_token_array", + "msa_prefill_fork", + "proxy_ms_reuse", + "context_onehot_index_fill", + "dflash_host_ctx_alloc", +) +TARGETS = tuple(f"minimax_m31_nvfp4__sm_107__{tp}" for tp in ("tp2", "tp4", "tp8")) + + +def _hook_keywords(target: str) -> dict: + path = Path(_family.__file__).resolve().parent / target / "modeling.py" + tree = ast.parse(path.read_text(), filename=str(path)) + (fn,) = [n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == "_runtime_hooks"] + calls = [ + n + for n in ast.walk(fn) + if isinstance(n, ast.Call) and isinstance(n.func, ast.Name) and n.func.id == "RuntimeHooks" + ] + assert len(calls) == 1, f"{target}: expected one RuntimeHooks(...) in _runtime_hooks" + return {kw.arg: kw.value for kw in calls[0].keywords} + + +@pytest.mark.parametrize("target", TARGETS) +def test_target_arms_every_shared_opt_in(target): + keywords = _hook_keywords(target) + for field in SHARED_OPT_INS: + value = keywords.get(field) + assert isinstance(value, ast.Constant) and value.value is True, ( + f"{target} does not arm {field}" + ) + + +def test_shared_opt_ins_are_off_by_default(): + defaults = {f.name: f.default for f in dataclasses.fields(RuntimeHooks)} + assert all(defaults[field] is False for field in SHARED_OPT_INS) diff --git a/tests/unittest/_torch/test_cute_aot.py b/tests/unittest/_torch/test_cute_aot.py index 0225518253df..ab6b92d5638b 100644 --- a/tests/unittest/_torch/test_cute_aot.py +++ b/tests/unittest/_torch/test_cute_aot.py @@ -54,6 +54,7 @@ def load(path, symbol, tvm_ffi): return _Compiled("loaded") monkeypatch.setattr(cute_aot, "_load", load) + monkeypatch.setattr(cute_aot, "_armed", True) monkeypatch.setenv(cute_aot.DIR_ENV, str(tmp_path / "data")) for env in (cute_aot.EXPORT_DIR_ENV, cute_aot.DISABLE_ENV): monkeypatch.delenv(env, raising=False) @@ -75,6 +76,19 @@ def test_shipped_object_is_loaded_not_compiled(aot, tmp_path): assert cute_aot.counts() == {"loaded": 1, "compiled": 0} +def test_unarmed_process_only_compiles(aot, tmp_path, monkeypatch): + # A process whose model does not ship objects never looks one up or exports, even with the export directory set. + monkeypatch.setattr(cute_aot, "_armed", False) + monkeypatch.setenv(cute_aot.EXPORT_DIR_ENV, str(tmp_path / "export")) + _ship(tmp_path, "gemv m=8") + calls = [] + for _ in range(2): + got = cute_aot.compile_or_load("gemv m=8", lambda: calls.append(1) or _Compiled("jit")) + assert got.tag == "jit" and calls == [1] + assert aot == [] and not (tmp_path / "export").exists() + assert cute_aot.counts() == {"loaded": 0, "compiled": 0} + + def test_missing_object_compiles_once_per_key(aot): calls = [] for _ in range(2): From 0c88f870ea6d1b13d99aa2e9231ae0668adaf8c6 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 15:15:48 -0700 Subject: [PATCH 26/38] [None][test] Expect a device token array to be rejected by LlmRequest A CUDA int32 tensor no longer takes the host-array copy; it takes the sequence cast, which rejects it, as before the fast path existed. The test asserts that rejection instead of a read through the element-wise cast. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../executor/test_request_token_array.py | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/tests/unittest/_torch/executor/test_request_token_array.py b/tests/unittest/_torch/executor/test_request_token_array.py index 6abcb1cab037..36ee35cb7a62 100644 --- a/tests/unittest/_torch/executor/test_request_token_array.py +++ b/tests/unittest/_torch/executor/test_request_token_array.py @@ -149,14 +149,15 @@ def make(tokens): @pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA tensor") -def test_llm_request_reads_a_device_array_through_the_sequence_path(): - # Only a host int32 array is copied as raw memory; a CUDA tensor takes the element-wise cast. +def test_llm_request_rejects_a_device_array(): + # Only a host int32 array is copied as raw memory; a CUDA tensor takes the sequence cast, which rejects it (it is + # never read through a device pointer). tokens = torch.tensor(PROMPT, dtype=torch.int32, device="cuda") - llm_req = lr_mod.LlmRequest( - request_id=4, - max_new_tokens=4, - input_tokens=tokens, - sampling_config=tllm.SamplingConfig(1), - is_streaming=False, - ) - assert llm_req.get_tokens(0) == PROMPT + with pytest.raises(RuntimeError): + lr_mod.LlmRequest( + request_id=4, + max_new_tokens=4, + input_tokens=tokens, + sampling_config=tllm.SamplingConfig(1), + is_streaming=False, + ) From 3f48ba43216c531bc6bbfa132cc3eb09f927301a Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 15:43:39 -0700 Subject: [PATCH 27/38] [None][test] Cover the KV prefetch startup state when the model does not opt in The startup line reads "disabled (kill switch)" only when the kill switch is set; a model that does not opt in reads "inactive (the model does not opt in)". The parametrized state test covers both. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../executor/kv_cache/test_kv_gpu_prefetch.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py b/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py index 40e5673de231..baaf3eeae721 100644 --- a/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py +++ b/tests/unittest/_torch/executor/kv_cache/test_kv_gpu_prefetch.py @@ -149,18 +149,20 @@ def test_counter_prints_every_200_steps_even_without_candidates(monkeypatch): @pytest.mark.parametrize( - "switch,v2,reuse,want", + "switch,killed,v2,reuse,want", [ - (True, True, True, "enabled"), - (False, True, True, "disabled (kill switch)"), - (True, False, True, "inactive (KV cache manager v1)"), - (True, True, False, "inactive (block reuse off)"), + (True, False, True, True, "enabled"), + (False, True, True, True, "disabled (kill switch)"), + (False, False, True, True, "inactive (the model does not opt in)"), + (True, False, False, True, "inactive (KV cache manager v1)"), + (True, False, True, False, "inactive (block reuse off)"), ], ) -def test_startup_line_once_with_state(monkeypatch, switch, v2, reuse, want): +def test_startup_line_once_with_state(monkeypatch, switch, killed, v2, reuse, want): log = mock.Mock() monkeypatch.setattr(pe, "logger", log) monkeypatch.setattr(pe, "_KV_GPU_PREFETCH", switch) + monkeypatch.setattr(pe, "_KV_GPU_PREFETCH_KILLED", killed) ex = SimpleNamespace( dist=SimpleNamespace(rank=0), _is_kv_manager_v2=v2, From 6aaa96e855acf7e1f7d80c4c4da9f710bf741b8d Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 15:59:41 -0700 Subject: [PATCH 28/38] [None][fix] Arm the shared MiniMax-M3.1 sm_107 opt-ins at routing The shared opt-ins are now armed by routing, process-wide, when it selects an sm_107 target (arm_sm107_shared_opt_ins, before the model is built), instead of through the targets' RuntimeHooks. The target directories are the source digest of their shipped ahead-of-time CuTe DSL objects, so leaving them unedited keeps every shipped object valid. The RuntimeHooks fields stay as the opt-in for other models. Unit test: routing arms every opt-in before returning an sm_107 target, and the helper sets every flag. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../modeling.py | 7 -- .../modeling.py | 7 -- .../modeling.py | 7 -- .../modeling_v2/models/minimax_m3/routing.py | 30 ++++- tensorrt_llm/_torch/pyexecutor/py_executor.py | 11 +- .../test_m31_sm107_shared_opt_ins.py | 108 +++++++++++++----- 6 files changed, 113 insertions(+), 57 deletions(-) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py index 544a098835d3..2153e950db70 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp2/modeling.py @@ -8825,13 +8825,6 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: fp8_saturating_casts=True, msa_kv_cache_fixes=True, draft_pool_fixes=True, - draft_kv_reserve_guard=True, - kv_gpu_prefetch=True, - context_token_array=True, - msa_prefill_fork=True, - proxy_ms_reuse=True, - context_onehot_index_fill=True, - dflash_host_ctx_alloc=True, persist_flashinfer_mxfp8_tactics=True, # Bundled name-keyed tactic pins (m31_pins); kill: TRTLLM_M31_DISABLE_BUNDLED_PINS=1. install_tactic_pins=lambda rank: m31_pins.install("tp2", rank), diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py index f73189312685..921c65470472 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp4/modeling.py @@ -6100,13 +6100,6 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: fp8_saturating_casts=True, msa_kv_cache_fixes=True, draft_pool_fixes=True, - draft_kv_reserve_guard=True, - kv_gpu_prefetch=True, - context_token_array=True, - msa_prefill_fork=True, - proxy_ms_reuse=True, - context_onehot_index_fill=True, - dflash_host_ctx_alloc=True, first_content_emit_iters=FIRST_CONTENT_EMIT_ITERS, mxfp8_pinned_tactics=True, late_compile_detect=True, diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py index adc703bde461..184b2cc201f1 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m31_nvfp4__sm_107__tp8/modeling.py @@ -6537,13 +6537,6 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks: fp8_saturating_casts=True, msa_kv_cache_fixes=True, draft_pool_fixes=True, - draft_kv_reserve_guard=True, - kv_gpu_prefetch=True, - context_token_array=True, - msa_prefill_fork=True, - proxy_ms_reuse=True, - context_onehot_index_fill=True, - dflash_host_ctx_alloc=True, first_content_emit_iters=FIRST_CONTENT_EMIT_ITERS, mxfp8_pinned_tactics=True, late_compile_detect=True, diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py index 1572c8b60bfe..96b8bdc85743 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py @@ -151,6 +151,30 @@ def field(name): return None +def arm_sm107_shared_opt_ins() -> None: + """Arm, process-wide and before the model is built, the shared paths the MiniMax-M3.1 sm_107 targets take: the + shipped ahead-of-time CuTe DSL objects, the drafter KV reserve and guard page, the host-tier KV prefetch, the + context-only int32 prompt tokens, the BF16X / E4M3X sparse-prefill fork, the indexer proxy max-score buffer reuse, + the context one-hot index_fill and the DFlash host-side context allocation. Each keeps its kill switch. Armed here + rather than in a target's RuntimeHooks so that the target directories (the AOT objects' source digests) are not + edited.""" + from tensorrt_llm._torch import cute_aot + from tensorrt_llm._torch.attention.backends.fmha import msa_prefill + from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import msa_indexer + from tensorrt_llm._torch.pyexecutor import llm_request, py_executor + from tensorrt_llm._torch.pyexecutor.kv_cache import kv_cache_manager_v2 + from tensorrt_llm._torch.speculative import dflash, interface + + cute_aot.arm() + kv_cache_manager_v2.arm_draft_kv_reserve_guard() + py_executor.arm_kv_gpu_prefetch() + llm_request.arm_context_token_array() + msa_prefill.arm_prefill_fork() + msa_indexer.arm_proxy_ms_reuse() + interface.arm_onehot_index_fill() + dflash.arm_host_ctx_alloc() + + def route( pretrained_config: Any, mapping: Any, @@ -217,14 +241,10 @@ def route( and m31_spec_miss is None and kv_cache ): - from tensorrt_llm._torch import cute_aot - from .fake_ops import register register() - # The sm_107 targets ship ahead-of-time CuTe DSL objects (cute_aot_data); serve them before the model builds - # its kernels. - cute_aot.arm() + arm_sm107_shared_opt_ins() if parallel == "tp2": from .minimax_m31_nvfp4__sm_107__tp2.modeling import ModelingV2MiniMaxM31Nvfp4Sm107Tp2 diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor.py b/tensorrt_llm/_torch/pyexecutor/py_executor.py index a35e7c4d136b..8cb61ace557d 100644 --- a/tensorrt_llm/_torch/pyexecutor/py_executor.py +++ b/tensorrt_llm/_torch/pyexecutor/py_executor.py @@ -140,6 +140,14 @@ _KV_GPU_PREFETCH_KILLED = os.environ.get("TRTLLM_M31_DISABLE_KV_GPU_PREFETCH", "0") == "1" _KV_GPU_PREFETCH = False + + +def arm_kv_gpu_prefetch() -> None: + """Take the host-tier KV prefix prefetch in this process (RuntimeHooks.kv_gpu_prefetch), unless killed.""" + global _KV_GPU_PREFETCH + _KV_GPU_PREFETCH = not _KV_GPU_PREFETCH_KILLED + + _KV_GPU_PREFETCH_MAX = int(os.environ.get("TRTLLM_M31_KV_GPU_PREFETCH_MAX", "2")) _KV_GPU_PREFETCH_BYTES_PER_STEP = int(1.5e9) @@ -403,7 +411,6 @@ def _arm_runtime_opt_ins(hooks: RuntimeHooks) -> bool: index_fill, the DFlash host-side context allocation and the late-compile detector. Each keeps its TRTLLM_M31_DISABLE_* kill switch. """ - global _KV_GPU_PREFETCH if hooks.mxfp8_pinned_tactics: # Imported on use: only an opted-in model needs the table. from tensorrt_llm._torch.custom_ops import mxfp8_cute_pinned_tactics @@ -413,7 +420,7 @@ def _arm_runtime_opt_ins(hooks: RuntimeHooks) -> bool: paged_cache paged_cache.arm_fp8_saturation() if hooks.kv_gpu_prefetch: - _KV_GPU_PREFETCH = not _KV_GPU_PREFETCH_KILLED + arm_kv_gpu_prefetch() if hooks.context_token_array: from . import llm_request llm_request.arm_context_token_array() diff --git a/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py b/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py index 9bfc85b853e7..5f5b0e97c7b4 100644 --- a/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py +++ b/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py @@ -12,9 +12,9 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -"""The shared-path opt-ins the MiniMax-M3.1 sm_107 targets arm: every target's ``_runtime_hooks`` passes each of them -as a literal True, and each is off in the default RuntimeHooks (so a model that declares none, GB300's included, keeps -the stock path). Read from modeling.py's source, so the heavy modeling modules are not imported. CPU only.""" +"""The shared-path opt-ins the MiniMax-M3.1 sm_107 targets take: routing arms each of them, process-wide, before it +returns an sm_107 target (``arm_sm107_shared_opt_ins``), every one is off until armed, and the matching RuntimeHooks +fields default to off (so a model that declares none, GB300's included, keeps the stock path). CPU only.""" import ast import dataclasses @@ -23,13 +23,18 @@ import pytest import tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 as _family +from tensorrt_llm._torch import cute_aot +from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 import routing +from tensorrt_llm._torch.attention.backends.fmha import msa_prefill +from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import msa_indexer +from tensorrt_llm._torch.pyexecutor import llm_request, py_executor +from tensorrt_llm._torch.pyexecutor.kv_cache import kv_cache_manager_v2 from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks +from tensorrt_llm._torch.speculative import dflash, interface pytestmark = pytest.mark.cpu_only SHARED_OPT_INS = ( - "draft_pool_fixes", - "msa_kv_cache_fixes", "draft_kv_reserve_guard", "kv_gpu_prefetch", "context_token_array", @@ -38,30 +43,75 @@ "context_onehot_index_fill", "dflash_host_ctx_alloc", ) -TARGETS = tuple(f"minimax_m31_nvfp4__sm_107__{tp}" for tp in ("tp2", "tp4", "tp8")) - - -def _hook_keywords(target: str) -> dict: - path = Path(_family.__file__).resolve().parent / target / "modeling.py" - tree = ast.parse(path.read_text(), filename=str(path)) - (fn,) = [n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == "_runtime_hooks"] - calls = [ - n - for n in ast.walk(fn) - if isinstance(n, ast.Call) and isinstance(n.func, ast.Name) and n.func.id == "RuntimeHooks" - ] - assert len(calls) == 1, f"{target}: expected one RuntimeHooks(...) in _runtime_hooks" - return {kw.arg: kw.value for kw in calls[0].keywords} - - -@pytest.mark.parametrize("target", TARGETS) -def test_target_arms_every_shared_opt_in(target): - keywords = _hook_keywords(target) - for field in SHARED_OPT_INS: - value = keywords.get(field) - assert isinstance(value, ast.Constant) and value.value is True, ( - f"{target} does not arm {field}" - ) +ARM_CALLS = ( + "arm", # cute_aot + "arm_draft_kv_reserve_guard", + "arm_kv_gpu_prefetch", + "arm_context_token_array", + "arm_prefill_fork", + "arm_proxy_ms_reuse", + "arm_onehot_index_fill", + "arm_host_ctx_alloc", +) +# (module, flag, value once armed) for every process-wide flag the helper sets. +FLAGS = ( + (cute_aot, "_armed", True), + (kv_cache_manager_v2, "_DRAFT_KV_RESERVE", True), + (kv_cache_manager_v2, "_DRAFT_KV_GUARD", True), + (py_executor, "_KV_GPU_PREFETCH", True), + (llm_request, "_L1_TOKEN_ARRAY", True), + (msa_prefill, "_PREFILL_FORK_ARMED", True), + (msa_indexer, "_PROXY_MS_REUSE", True), + (interface, "_ONEHOT_INDEX_FILL", True), + (dflash, "_HOST_CTX_ALLOC_DISABLED", False), +) +KILL_SWITCHES = ( + "TRTLLM_M31_DISABLE_DRAFT_KV_RESERVE", + "TRTLLM_M31_DISABLE_DRAFT_KV_GUARD", + "TRTLLM_M31_DISABLE_L1_TOKEN_ARRAY", + "TRTLLM_M31_DISABLE_PROXY_MS_REUSE", + "TRTLLM_M31_DISABLE_ONEHOT_INDEX_FILL", + "TRTLLM_DFLASH_DISABLE_HOST_CTX_ALLOC", +) + + +def _routing_tree() -> ast.Module: + path = Path(_family.__file__).resolve().parent / "routing.py" + return ast.parse(path.read_text(), filename=str(path)) + + +def _function(tree: ast.Module, name: str) -> ast.FunctionDef: + (fn,) = [n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == name] + return fn + + +def _called_names(fn: ast.FunctionDef) -> set: + names = set() + for node in ast.walk(fn): + if isinstance(node, ast.Call): + func = node.func + names.add(func.attr if isinstance(func, ast.Attribute) else getattr(func, "id", None)) + return names + + +def test_helper_arms_every_shared_opt_in(): + called = _called_names(_function(_routing_tree(), "arm_sm107_shared_opt_ins")) + assert set(ARM_CALLS) <= called, f"missing arm calls: {set(ARM_CALLS) - called}" + + +def test_route_arms_before_returning_an_sm107_target(): + assert "arm_sm107_shared_opt_ins" in _called_names(_function(_routing_tree(), "route")) + + +def test_helper_sets_every_flag(monkeypatch): + for env in KILL_SWITCHES: + monkeypatch.delenv(env, raising=False) + monkeypatch.setattr(py_executor, "_KV_GPU_PREFETCH_KILLED", False) + for module, flag, armed in FLAGS: + monkeypatch.setattr(module, flag, not armed) + routing.arm_sm107_shared_opt_ins() + for module, flag, armed in FLAGS: + assert getattr(module, flag) is armed, f"{module.__name__}.{flag}" def test_shared_opt_ins_are_off_by_default(): From bb04179f2cccee2d97c190f8e6320c24bd1e97ca Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 17:14:20 -0700 Subject: [PATCH 29/38] [None][fix] Avoid batch-shape specialization in DSpark drafter inputs The DSpark drafter-inputs Triton kernel no longer specializes on the accepted-token buffer's row stride or the alignment of its post-context view. Both depend on the step's batch composition, so a specialization compiled a new variant at a batch shape warmup never ran (a late compile after the server was ready). Both feed scalar per-program loads only, so not specializing them costs nothing and the outputs are bitwise unchanged. No kill switch: this is a compile-specialization hint with no alternate path; turning it off would only reintroduce the late compile. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../catalog/spec_decode/dspark_drafter_inputs.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_drafter_inputs.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_drafter_inputs.py index 56d9cef29894..22bdbe58a21b 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_drafter_inputs.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/spec_decode/dspark_drafter_inputs.py @@ -32,7 +32,11 @@ import triton.language as tl -@triton.jit +# acc_stride and nacc_ptr depend on the step's batch composition (the accepted-token buffer's row stride; a view +# starting after the context requests), so Triton's divisibility-16 specialization of either would compile a new +# variant at a batch shape warmup never ran (a post-ready late compile, seen at 3 scheduled requests). Both feed +# scalar per-program loads only, so not specializing them costs nothing. +@triton.jit(do_not_specialize=["acc_stride"], do_not_specialize_on_alignment=["nacc_ptr"]) def _inputs_kernel( acc_ptr, # accepted tokens [n, acc_stride] (int32 / int64) acc_stride, From b3f097dd3ff781c20a437d575fd3f1a2112f34c0 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 17:28:51 -0700 Subject: [PATCH 30/38] [None][test] Cover DSpark input strides and context-view specializations Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../test_modeling_v2_dspark_drafter_inputs.py | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_drafter_inputs.py b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_drafter_inputs.py index 14e64c41d22c..bc8904b7245e 100644 --- a/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_drafter_inputs.py +++ b/tests/unittest/_torch/modeling_v2/spec_decode/test_modeling_v2_dspark_drafter_inputs.py @@ -103,3 +103,39 @@ def test_other_embeddings_keep_the_stock_path(): other = torch.nn.Embedding(VOCAB, HIDDEN).cuda().bfloat16() fn = _target("tp8").s10c_c_drafter_inputs assert fn(other, accepted, nacc, slots, ctx_len, 4096, MASK_ID, None, 3, 4) is None + + +@pytest.mark.parametrize("k", [3, 7]) +def test_batch_views_do_not_create_inputs_variants(k, monkeypatch): + """Accepted-token strides and context-offset count views must share one variant per K.""" + from tensorrt_llm._torch._experimental.modeling_v2.catalog.spec_decode import ( + dspark_drafter_inputs as inputs, + ) + + hashes = set() + launch = inputs._inputs_kernel.run + + def record_launch(*args, **kwargs): + compiled = launch(*args, **kwargs) + hashes.add(compiled.hash) + return compiled + + monkeypatch.setattr(inputs._inputs_kernel, "run", record_launch) + embed = _embed("tp8") + # Use the public target adapter: it supplies the actual embedding and mask-row contract. + fn = _target("tp8").s10c_c_drafter_inputs + for n in range(1, 17): + accepted, nacc, slots, ctx_len = _case(n, k, torch.int32, n, max_ctx=4096) + for stride in (k + 1, 16, 17): + storage = torch.empty((n, stride), dtype=accepted.dtype, device="cuda") + view = storage[:, : k + 1] + view.copy_(accepted) + for context_offset in range(4): + counts = torch.empty(n + context_offset, dtype=nacc.dtype, device="cuda") + counts_view = counts[context_offset:] + counts_view.copy_(nacc) + args = (embed, view, counts_view, slots, ctx_len, 4096) + got = fn(*args, MASK_ID, None, k, k + 1) + ref = _stock(*args, None, k, k + 1) + assert got is not None and _equal(got, ref), (n, stride, context_offset) + assert len(hashes) == 1, hashes From 6355f113686e7e56184f49dd0ebb996b43ad4b35 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 21:59:59 -0700 Subject: [PATCH 31/38] [None][fix] Await the iteration-stats RPC instead of blocking the event loop GenerationExecutorProxy.aget_stats and GenerationExecutorRpcProxy.aget_stats fetched stats with a blocking .remote() that waits up to `timeout` for a record. trtllm-serve calls them on the event loop that also sends every response and token, so whenever no record is ready the server stalls. They now issue the same RPC through remote_future() and await it, then drain the iteration-result queue as before: same records, same order. TRTLLM_M31_DISABLE_A4_ASYNC_STATS=1 restores the blocking call. Unit test: both proxies return the same records in the same order with and without the switch, and the event loop keeps running while the awaited RPC is in flight (it does not while the blocking one is). Ported from dongfengy/tekit b84c17672 (GLM-5 Rubin branch). Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- tensorrt_llm/executor/proxy.py | 39 +++++- tensorrt_llm/executor/rpc_proxy.py | 32 ++++- .../executor/test_proxy_aget_stats_async.py | 118 ++++++++++++++++++ 3 files changed, 180 insertions(+), 9 deletions(-) create mode 100644 tests/unittest/executor/test_proxy_aget_stats_async.py diff --git a/tensorrt_llm/executor/proxy.py b/tensorrt_llm/executor/proxy.py index 4026ccf9734b..c1931bf32adb 100644 --- a/tensorrt_llm/executor/proxy.py +++ b/tensorrt_llm/executor/proxy.py @@ -12,6 +12,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import asyncio import atexit import concurrent.futures import json @@ -22,7 +23,7 @@ import time import weakref from queue import Empty -from typing import Dict, List, Optional, Union +from typing import AsyncIterator, Dict, List, Optional, Union import torch import zmq @@ -59,6 +60,13 @@ "GenerationExecutorFrontendProxy", ] +# TRTLLM_M31_DISABLE_A4_ASYNC_STATS=1 restores the blocking stats RPC in aget_stats. +# By default the RPC is awaited, so trtllm-serve's event loop (which also sends +# every response and token) does not stall while it waits up to `timeout` for a +# stats record. Same records, same order either way. +_ASYNC_STATS_RPC = os.environ.get("TRTLLM_M31_DISABLE_A4_ASYNC_STATS", + "0") != "1" + # Methods that are explicitly implemented for multi-rank MPI/IPC executor # deployments and may be called via collective_rpc even when world_size > 1. # sleep() and wakeup() coordinate across all ranks via a dedicated control @@ -1372,14 +1380,16 @@ def get_startup_metrics(self) -> dict | None: logger.warning(f"Error fetching startup metrics via RPC: {e}") return None - def aget_stats(self, timeout: float) -> IterationResult: + def aget_stats( + self, + timeout: float) -> Union[IterationResult, AsyncIterator[dict]]: """Get iteration statistics from the runtime via RPC (async). Args: timeout (float): Max wait time in seconds for the RPC call. Returns: - IterationResult: An async iterable object containing runtime stats. + An async iterable object containing runtime stats. """ # Initialize iteration result if needed self._maybe_initialize_iteration_results() @@ -1389,6 +1399,10 @@ def aget_stats(self, timeout: float) -> IterationResult: from .executor import empty_async_iterable return empty_async_iterable() + self._iter_stats_result.set_timeout(timeout) + if _ASYNC_STATS_RPC: + return self._aiter_stats(timeout) + # Fetch stats via RPC and populate the result try: stats = self.rpc_client.fetch_stats_wait_async( @@ -1400,9 +1414,26 @@ def aget_stats(self, timeout: float) -> IterationResult: for stat in stats: self._iter_stats_result.queue.put(stat) - self._iter_stats_result.set_timeout(timeout) return self._iter_stats_result + async def _aiter_stats(self, timeout: float) -> AsyncIterator[dict]: + # The RPC waits up to `timeout` for a stats record. Await it instead of + # blocking the caller's event loop, which in trtllm-serve also sends + # every response. + try: + stats = await asyncio.wrap_future( + self.rpc_client.fetch_stats_wait_async( + timeout=timeout).remote_future()) + except Exception as e: + logger.debug(f"Error fetching stats via RPC: {e}") + stats = [] + + for stat in stats: + self._iter_stats_result.queue.put(stat) + + async for stat in self._iter_stats_result: + yield stat + def get_kv_events(self, timeout: float) -> List[dict]: """Get iteration KV events from the runtime via RPC. diff --git a/tensorrt_llm/executor/rpc_proxy.py b/tensorrt_llm/executor/rpc_proxy.py index 45a4c653a7d8..96fd78ab2d82 100644 --- a/tensorrt_llm/executor/rpc_proxy.py +++ b/tensorrt_llm/executor/rpc_proxy.py @@ -12,9 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import asyncio import json import threading -from typing import List, Optional, Union +from typing import AsyncIterator, List, Optional, Union from ..llmapi.mpi_session import (MpiPoolSession, MpiSession, validate_session_world_size) @@ -22,7 +23,7 @@ from ..logger import logger from .executor import GenerationExecutor from .postproc_worker import PostprocWorkerConfig -from .proxy import _check_collective_rpc_guard +from .proxy import _ASYNC_STATS_RPC, _check_collective_rpc_guard from .result import IterationResult from .rpc.rpc_common import RPCError from .rpc_proxy_mixin import RpcExecutorMixin @@ -164,14 +165,16 @@ def get_startup_metrics(self) -> dict | None: logger.warning(f"Error fetching startup metrics via RPC: {e}") return None - def aget_stats(self, timeout: float) -> IterationResult: + def aget_stats( + self, + timeout: float) -> Union[IterationResult, AsyncIterator[dict]]: """Get iteration statistics from the runtime via RPC (async). Args: timeout (float): Max wait time in seconds for the RPC call. Returns: - IterationResult: An async iterable object containing runtime stats. + An async iterable object containing runtime stats. """ self._maybe_initialize_iteration_results() @@ -181,6 +184,10 @@ def aget_stats(self, timeout: float) -> IterationResult: from .executor import empty_async_iterable return empty_async_iterable() + self._iter_stats_result.set_timeout(timeout) + if _ASYNC_STATS_RPC: + return self._aiter_stats(timeout) + # Fetch stats via RPC and populate the result try: stats = self.rpc_client.fetch_stats_wait_async( @@ -191,9 +198,24 @@ def aget_stats(self, timeout: float) -> IterationResult: for stat in stats: self._iter_stats_result.queue.put(stat) - self._iter_stats_result.set_timeout(timeout) return self._iter_stats_result + async def _aiter_stats(self, timeout: float) -> AsyncIterator[dict]: + # Await the stats RPC instead of blocking the caller's event loop + # (see GenerationExecutorProxy._aiter_stats). + try: + stats = await asyncio.wrap_future( + self.rpc_client.fetch_stats_wait_async( + timeout=timeout).remote_future()) + except Exception: + stats = [] + + for stat in stats: + self._iter_stats_result.queue.put(stat) + + async for stat in self._iter_stats_result: + yield stat + def get_kv_events(self, timeout: float) -> List[dict]: """Get iteration KV events from the runtime via RPC. diff --git a/tests/unittest/executor/test_proxy_aget_stats_async.py b/tests/unittest/executor/test_proxy_aget_stats_async.py new file mode 100644 index 000000000000..7519584dc559 --- /dev/null +++ b/tests/unittest/executor/test_proxy_aget_stats_async.py @@ -0,0 +1,118 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""``aget_stats`` awaits the stats RPC instead of blocking the event loop. + +trtllm-serve's stats collector calls ``get_stats_async`` after every finished +request on the same event loop that sends every response and token. The RPC +waits up to ``timeout`` for a stats record, so a blocking ``.remote()`` there +stalls the server. These tests pin, for the IPC and the RPC proxy: + +* the awaited path returns the same records in the same order as the blocking + path (``TRTLLM_M31_DISABLE_A4_ASYNC_STATS=1``); +* the event loop keeps running while the awaited RPC is in flight, and does not + run while the blocking one is. +""" + +import asyncio +import concurrent.futures +import json +import time + +import pytest + +from tensorrt_llm.executor import proxy as proxy_mod +from tensorrt_llm.executor import rpc_proxy as rpc_proxy_mod +from tensorrt_llm.executor.proxy import GenerationExecutorProxy +from tensorrt_llm.executor.rpc_proxy import GenerationExecutorRpcProxy + +_RECORDS = [{"iter": i, "numActiveRequests": i % 3} for i in range(5)] + + +class _FakeCall: + def __init__(self, pool, delay, records): + self._pool, self._delay, self._records = pool, delay, records + + def _run(self): + time.sleep(self._delay) + return [json.dumps(r) for r in self._records] + + def remote(self): + return self._run() + + def remote_future(self): + return self._pool.submit(self._run) + + +class _FakeRpcClient: + def __init__(self, pool, delay, records): + self._pool, self._delay, self._records = pool, delay, records + + def fetch_stats_wait_async(self, timeout=None): + return _FakeCall(self._pool, self._delay, self._records) + + +@pytest.fixture +def pool(): + p = concurrent.futures.ThreadPoolExecutor(max_workers=1) + yield p + p.shutdown(wait=True) + + +def _bare(cls, client): + executor = cls.__new__(cls) + executor._is_llm_executor = True + executor._iter_stats_result = None + executor._iter_kv_events_result = None + executor.rpc_client = client + return executor + + +async def _collect(executor, timeout): + return [stat async for stat in executor.aget_stats(timeout=timeout)] + + +@pytest.mark.parametrize("cls", [GenerationExecutorProxy, GenerationExecutorRpcProxy]) +def test_same_records_same_order(cls, pool, monkeypatch): + got = {} + for async_rpc in (False, True): + monkeypatch.setattr(proxy_mod, "_ASYNC_STATS_RPC", async_rpc) + monkeypatch.setattr(rpc_proxy_mod, "_ASYNC_STATS_RPC", async_rpc) + executor = _bare(cls, _FakeRpcClient(pool, 0.0, _RECORDS)) + got[async_rpc] = asyncio.run(_collect(executor, 0.05)) + assert got[False] == _RECORDS + assert got[True] == got[False] + + +@pytest.mark.parametrize("cls", [GenerationExecutorProxy, GenerationExecutorRpcProxy]) +@pytest.mark.parametrize("async_rpc", [False, True]) +def test_event_loop_runs_during_rpc(cls, async_rpc, pool, monkeypatch): + monkeypatch.setattr(proxy_mod, "_ASYNC_STATS_RPC", async_rpc) + monkeypatch.setattr(rpc_proxy_mod, "_ASYNC_STATS_RPC", async_rpc) + delay = 0.3 + + async def main(): + ticks = [] + stop = asyncio.Event() + + async def ticker(): + while not stop.is_set(): + ticks.append(time.monotonic()) + await asyncio.sleep(0.005) + + task = asyncio.create_task(ticker()) + await asyncio.sleep(0.02) + executor = _bare(cls, _FakeRpcClient(pool, delay, _RECORDS)) + t0 = time.monotonic() + records = await _collect(executor, 0.05) + stop.set() + await task + during = [t for t in ticks if t0 < t < t0 + delay] + return records, len(during) + + records, ticks_during_rpc = asyncio.run(main()) + assert records == _RECORDS + if async_rpc: + # ~60 ticks of 5 ms fit in the 300 ms RPC; leave a wide margin. + assert ticks_during_rpc >= 20 + else: + assert ticks_during_rpc <= 1 From 75dfb8911469535e9df313417b0f86fb18066f34 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 23:20:03 -0700 Subject: [PATCH 32/38] [None][feat] Add a phase-armed Nsight window and a per-rank host-path probe Two executor diagnostics, armed by environment only and off by default. Off, profile_step pays one None check per iteration. TRTLLM_M31_PHASE_NSYS_ARM= (phase_capture.py): rank 0 reads a JSON marker (schema_version 1, capture_at_unix_s, num_steps, max_seconds) at most once per second and, at the requested wall-clock time, starts a CUDA_PROFILER-only window through PyExecutorProfileManager.start_profile, whose broadcast starts every rank on the same iteration. The window stops after num_steps or max_seconds (a non-blocking stop: the scheduled stop config plus its broadcast). It covers a chosen phase of a running server for nsys --capture-range=cudaProfilerApi. Never during warmup; refused together with the iteration-range or torch-trace profiling variables. TRTLLM_M31_HOST_PATH_PROBE= (host_path_probe.py): per iteration and rank, host-only wall-clock stamps (loop top, forward issue and return, the wait on the previous step's sampler event) and the forward's context / generation request counts, written as int64 rows. The wait is the host's headroom over the GPU. No CUDA events and no added synchronization: the probe waits on the same event the sampler then waits on. Unit test (CPU): off by default with the executor untouched, a single None check in profile_step's per-iteration function, the marker poll rate, start and both stop reasons, malformed markers, warmup, rank and conflict guards, and the probe's rows and call passthrough. Adapted from dongfengy/tekit 8b5caa029 (GLM-5 phase-armed capture) and the per-rank host-path fixture its host commits report. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../_torch/pyexecutor/host_path_probe.py | 98 ++++++++ .../_torch/pyexecutor/phase_capture.py | 137 +++++++++++ tensorrt_llm/_torch/pyexecutor/profiling.py | 20 ++ .../test_phase_capture_host_path_probe.py | 215 ++++++++++++++++++ 4 files changed, 470 insertions(+) create mode 100644 tensorrt_llm/_torch/pyexecutor/host_path_probe.py create mode 100644 tensorrt_llm/_torch/pyexecutor/phase_capture.py create mode 100644 tests/unittest/_torch/executor/test_phase_capture_host_path_probe.py diff --git a/tensorrt_llm/_torch/pyexecutor/host_path_probe.py b/tensorrt_llm/_torch/pyexecutor/host_path_probe.py new file mode 100644 index 000000000000..440cccee0dc0 --- /dev/null +++ b/tensorrt_llm/_torch/pyexecutor/host_path_probe.py @@ -0,0 +1,98 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Per-rank host-path probe (diagnostic only, off by default). + +``TRTLLM_M31_HOST_PATH_PROBE=`` records, per executor iteration and rank, host-only wall-clock stamps (no +CUDA events, no extra synchronization point) into ``/hostpath-rank-pid

    .bin`` as int64 rows: + + t_top loop top (profile_step) + t_launch _forward_step entry + t_fwd_ret _forward_step return (the forward is issued) + t_wait0 before waiting on the previous step's sampler event (in _update_requests) + t_wait1 the previous step's sampler event has completed + n_ctx context requests in the forward + n_gen generation requests in the forward + +``t_wait1 - t_wait0`` is how long the host waited for the GPU: the host's headroom. A step with no wait was +host-bound. The probe waits on the same event the sampler then waits on (``SamplerEvent.synchronize`` is +idempotent), so it moves no work and adds no synchronization. Rows are flushed every 4096 steps and at exit. +Reader: the Host fixture's host_path_read_v1.py. Without the variable the executor installs nothing. +""" + +import array +import atexit +import os +import time +from typing import Optional + +from tensorrt_llm.logger import logger + +HOST_PATH_PROBE_ENV_VAR_NAME = "TRTLLM_M31_HOST_PATH_PROBE" +_FIELDS = 7 +_FLUSH_ROWS = 4096 + + +class HostPathProbe: + """One rank's per-iteration host stamps; installed on an executor by ``make_host_path_probe``.""" + + def __init__(self, out_dir: str, rank: int): + os.makedirs(out_dir, exist_ok=True) + self._path = os.path.join(out_dir, f"hostpath-rank{rank}-pid{os.getpid()}.bin") + self._rows = array.array("q") + self._row = [0] * _FIELDS + self._ns = time.time_ns + atexit.register(self.flush) + + def top(self) -> None: + row = self._row + if row[0]: + self._rows.extend(row) + if len(self._rows) >= _FLUSH_ROWS * _FIELDS: + self.flush() + self._row = [self._ns(), 0, 0, 0, 0, 0, 0] + + def flush(self) -> None: + if self._rows: + with open(self._path, "ab") as f: + self._rows.tofile(f) + del self._rows[:] + + def install(self, executor) -> None: + """Wrap this executor's _forward_step and _update_requests (instance attributes only).""" + forward_step = executor._forward_step + update_requests = executor._update_requests + ns = self._ns + + def _forward_step(scheduled_requests, *args, **kwargs): + row = self._row + row[1] = ns() + row[5] = scheduled_requests.num_context_requests + row[6] = len(scheduled_requests.generation_requests) + try: + return forward_step(scheduled_requests, *args, **kwargs) + finally: + row[2] = ns() + + def _update_requests(sample_state, *args, **kwargs): + row = self._row + event = getattr(sample_state, "sampler_event", None) + row[3] = ns() + if event is not None: + event.synchronize() + row[4] = ns() + return update_requests(sample_state, *args, **kwargs) + + executor._forward_step = _forward_step + executor._update_requests = _update_requests + + +def make_host_path_probe(executor) -> Optional[HostPathProbe]: + """The installed probe when TRTLLM_M31_HOST_PATH_PROBE is set, else None.""" + out_dir = os.environ.get(HOST_PATH_PROBE_ENV_VAR_NAME) + if not out_dir: + return None + rank = getattr(executor, "global_rank", getattr(getattr(executor, "dist", None), "rank", 0)) + probe = HostPathProbe(out_dir, rank) + probe.install(executor) + logger.info(f"[M31 host path probe] rank {rank} writing {probe._path}") + return probe diff --git a/tensorrt_llm/_torch/pyexecutor/phase_capture.py b/tensorrt_llm/_torch/pyexecutor/phase_capture.py new file mode 100644 index 000000000000..51ac89060443 --- /dev/null +++ b/tensorrt_llm/_torch/pyexecutor/phase_capture.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Phase-armed Nsight capture window (diagnostic only, off by default). + +``TRTLLM_M31_PHASE_NSYS_ARM=`` arms a capture of a chosen window of a running server, e.g. one phase +of an E2E benchmark, for nsys run with ``--capture-range=cudaProfilerApi``. Rank 0 reads the marker (JSON, +``{"schema_version": 1, "capture_at_unix_s": , "num_steps": <1..10000, default 200>, +"max_seconds": <0..120, default 10>}``) at most once per second. At ``capture_at_unix_s`` it starts a +``CUDA_PROFILER`` window through ``PyExecutorProfileManager.start_profile``, whose broadcast starts every rank on +the same iteration. The window stops after ``num_steps`` iterations or ``max_seconds``, whichever comes first. One +window per process; never during warmup; refused with the iteration-range or torch-trace profiling env set. + +Adapted from dongfengy/tekit 8b5caa029 (GLM-5: a per-rank marker poll calling cudaProfilerStart directly). +""" + +import json +import os +import time +from typing import Optional + +from tensorrt_llm.logger import logger + +PHASE_NSYS_ARM_ENV_VAR_NAME = "TRTLLM_M31_PHASE_NSYS_ARM" +_MAX_MARKER_BYTES = 4096 + + +class PhaseCapture: + """Rank 0's per-iteration driver of one phase-armed profile window.""" + + def __init__(self, manager, executor, marker_path: str): + self._manager = manager + self._executor = executor + self._path = marker_path + self._next_poll = 0.0 + self._capture_at: Optional[float] = None + self._num_steps = 200 + self._max_seconds = 10.0 + self._started_at: Optional[float] = None + self._seen_enabled = False + self.done = False + + def _read_marker(self) -> bool: + """Parse the marker once it exists; returns True once armed. Any malformed marker disarms for good.""" + try: + with open(self._path, encoding="utf-8") as f: + text = f.read(_MAX_MARKER_BYTES + 1) + except FileNotFoundError: + return False + except OSError as e: + logger.error(f"[M31 phase capture] marker read failed ({e}); disarmed") + self.done = True + return False + try: + if len(text) > _MAX_MARKER_BYTES: + raise ValueError("marker exceeds 4096 bytes") + control = json.loads(text) + if not isinstance(control, dict) or control.get("schema_version") != 1: + raise ValueError("unsupported marker schema") + capture_at = float(control["capture_at_unix_s"]) + num_steps = int(control.get("num_steps", 200)) + max_seconds = float(control.get("max_seconds", 10.0)) + if not ( + 0 < capture_at < float("inf") and 1 <= num_steps <= 10000 and 0 < max_seconds <= 120 + ): + raise ValueError("marker values out of range") + except (ValueError, KeyError, TypeError) as e: + logger.error(f"[M31 phase capture] bad marker {self._path} ({e}); disarmed") + self.done = True + return False + self._capture_at, self._num_steps, self._max_seconds = capture_at, num_steps, max_seconds + logger.info( + f"[M31 phase capture] armed at iteration {self._executor.iter_counter}: capture_at_unix_s=" + f"{capture_at:.3f} num_steps={num_steps} max_seconds={max_seconds}" + ) + return True + + def _finish(self, reason: str) -> None: + self.done = True + logger.info( + f"[M31 phase capture] window ended at iteration {self._executor.iter_counter}: reason={reason} " + f"seconds={time.monotonic() - self._started_at:.3f}" + ) + + def tick(self) -> None: + """Called once per executor iteration on rank 0.""" + if self.done or self._executor.is_warmup: + return + if self._capture_at is None: + now = time.monotonic() + if now < self._next_poll: + return + self._next_poll = now + 1.0 + if not self._read_marker(): + return + if self._started_at is None: + if time.time() < self._capture_at: + return + try: + self._manager.start_profile(num_steps=self._num_steps, activities=["CUDA_PROFILER"]) + except Exception as e: # noqa: BLE001 - a diagnostic must never take the server down + logger.error(f"[M31 phase capture] start_profile refused ({e}); disarmed") + self.done = True + return + self._started_at = time.monotonic() + logger.info( + f"[M31 phase capture] window scheduled at iteration {self._executor.iter_counter}" + ) + return + enabled = self._executor._profile_enabled + if enabled: + self._seen_enabled = True + if time.monotonic() - self._started_at >= self._max_seconds: + # The non-blocking half of stop_profile(): it would wait for this very thread to run the stop. + self._manager.apply_stop_config() + queue = getattr(self._executor, "executor_request_queue", None) + if queue is not None: + queue.enqueue_profile_stop_request() + self._finish("max_seconds") + elif self._seen_enabled: + self._finish("num_steps") + + +def make_phase_capture(manager, executor, conflicting_env: bool) -> Optional[PhaseCapture]: + """The rank-0 phase capture when TRTLLM_M31_PHASE_NSYS_ARM is set, else None.""" + path = os.environ.get(PHASE_NSYS_ARM_ENV_VAR_NAME) + if not path: + return None + if conflicting_env: + logger.error( + f"[M31 phase capture] {PHASE_NSYS_ARM_ENV_VAR_NAME} is exclusive with TLLM_PROFILE_START_STOP / " + "TLLM_TORCH_PROFILE_TRACE; disarmed" + ) + return None + if getattr(getattr(executor, "dist", None), "rank", 0) != 0: + return None + logger.info(f"[M31 phase capture] waiting for marker {path}") + return PhaseCapture(manager, executor, path) diff --git a/tensorrt_llm/_torch/pyexecutor/profiling.py b/tensorrt_llm/_torch/pyexecutor/profiling.py index 15ccb51c6094..4c584d1587d8 100644 --- a/tensorrt_llm/_torch/pyexecutor/profiling.py +++ b/tensorrt_llm/_torch/pyexecutor/profiling.py @@ -55,6 +55,8 @@ from tensorrt_llm.tools.layer_wise_benchmarks import get_calibrator from tensorrt_llm.tools.profiler.host_profile_tools.host_profiler import get_global_profiler +from .host_path_probe import make_host_path_probe +from .phase_capture import make_phase_capture from .runtime_hooks import runtime_hooks_of # Environment-variable names used by both the env-var-driven legacy @@ -637,6 +639,22 @@ def profile_step(self): env_torch_trace_path = f"{trace_base}-rank-{executor.global_rank}{trace_ext}" profile_start_stop = os.environ.get(PROFILE_START_STOP_ENV_VAR_NAME, None) env_enable_torch_trace = bool(env_torch_trace_path and profile_start_stop) + # Diagnostics armed by environment only (TRTLLM_M31_PHASE_NSYS_ARM, TRTLLM_M31_HOST_PATH_PROBE): when neither + # is set, per_step_diag is None and each iteration pays one None check. + phase_capture = make_phase_capture( + self, executor, conflicting_env=bool(env_torch_trace_path or profile_start_stop) + ) + host_path_probe = make_host_path_probe(executor) + per_step_diag = None + if phase_capture is not None or host_path_probe is not None: + + def _per_step_diag(): + if host_path_probe is not None: + host_path_probe.top() + if phase_capture is not None: + phase_capture.tick() + + per_step_diag = _per_step_diag if env_torch_trace_path and profile_start_stop is None: logger.warning( f"{PROFILE_START_STOP_ENV_VAR_NAME} environment variable " @@ -680,6 +698,8 @@ def profile_step_fn(): nonlocal boundary_count nonlocal torch_profiler, active_torch_trace_path nonlocal active_enable_torch_trace + if per_step_diag is not None: + per_step_diag() calibrator.post_step(it) if executor.iter_counter in executor.profile_stop_iters and not executor.is_warmup: if not enabled: diff --git a/tests/unittest/_torch/executor/test_phase_capture_host_path_probe.py b/tests/unittest/_torch/executor/test_phase_capture_host_path_probe.py new file mode 100644 index 000000000000..0bc41fb5c5c9 --- /dev/null +++ b/tests/unittest/_torch/executor/test_phase_capture_host_path_probe.py @@ -0,0 +1,215 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Diagnostics armed by environment only: the phase-armed Nsight window (TRTLLM_M31_PHASE_NSYS_ARM, +phase_capture.py) and the per-rank host-path probe (TRTLLM_M31_HOST_PATH_PROBE, host_path_probe.py). +Off, the executor's per-iteration cost is one None check (profiling.profile_step). CPU only.""" + +import array +import ast +import json +import time +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from tensorrt_llm._torch.pyexecutor import host_path_probe as hpp +from tensorrt_llm._torch.pyexecutor import phase_capture as pc +from tensorrt_llm._torch.pyexecutor import profiling + +pytestmark = pytest.mark.cpu_only + + +# ---- off path ------------------------------------------------------------------------------------------------- + + +def test_off_by_default(monkeypatch): + monkeypatch.delenv(pc.PHASE_NSYS_ARM_ENV_VAR_NAME, raising=False) + monkeypatch.delenv(hpp.HOST_PATH_PROBE_ENV_VAR_NAME, raising=False) + executor = SimpleNamespace(dist=SimpleNamespace(rank=0), global_rank=0) + fwd, upd = object(), object() + executor._forward_step, executor._update_requests = fwd, upd + assert pc.make_phase_capture(None, executor, conflicting_env=False) is None + assert hpp.make_host_path_probe(executor) is None + assert executor._forward_step is fwd and executor._update_requests is upd + + +def test_per_iteration_cost_is_one_none_check(): + """profile_step_fn touches the diagnostics only through `if per_step_diag is not None: per_step_diag()`.""" + tree = ast.parse(Path(profiling.__file__).read_text()) + (fn,) = [ + n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "profile_step_fn" + ] + names = [n.id for n in ast.walk(fn) if isinstance(n, ast.Name)] + assert "phase_capture" not in names and "host_path_probe" not in names + guards = [ + n + for n in ast.walk(fn) + if isinstance(n, ast.If) + and isinstance(n.test, ast.Compare) + and isinstance(n.test.left, ast.Name) + and n.test.left.id == "per_step_diag" + and isinstance(n.test.ops[0], ast.IsNot) + ] + assert len(guards) == 1 and names.count("per_step_diag") == 2 + + +# ---- phase capture -------------------------------------------------------------------------------------------- + + +class _Manager: + def __init__(self): + self.starts, self.stop_configs = [], 0 + + def start_profile(self, **kwargs): + self.starts.append(kwargs) + + def apply_stop_config(self): + self.stop_configs += 1 + + +def _executor(rank=0): + queue = SimpleNamespace(stops=0) + queue.enqueue_profile_stop_request = lambda: setattr(queue, "stops", queue.stops + 1) + return SimpleNamespace( + dist=SimpleNamespace(rank=rank), + iter_counter=0, + is_warmup=False, + _profile_enabled=False, + executor_request_queue=queue, + ) + + +class _Clock: + def __init__(self): + self.t = 1000.0 + + def monotonic(self): + return self.t + + +def _capture(tmp_path, monkeypatch, marker=None, rank=0): + path = tmp_path / "arm.json" + if marker is not None: + path.write_text(json.dumps(marker)) + monkeypatch.setenv(pc.PHASE_NSYS_ARM_ENV_VAR_NAME, str(path)) + clock = _Clock() + monkeypatch.setattr(pc.time, "monotonic", clock.monotonic) + manager, executor = _Manager(), _executor(rank) + return ( + pc.make_phase_capture(manager, executor, conflicting_env=False), + manager, + executor, + clock, + path, + ) + + +def test_rank0_only_and_conflicts(tmp_path, monkeypatch): + cap, *_ = _capture(tmp_path, monkeypatch, rank=1) + assert cap is None + monkeypatch.setenv(pc.PHASE_NSYS_ARM_ENV_VAR_NAME, str(tmp_path / "arm.json")) + assert pc.make_phase_capture(_Manager(), _executor(), conflicting_env=True) is None + + +def test_polls_at_most_once_per_second_then_starts(tmp_path, monkeypatch): + cap, manager, executor, clock, path = _capture(tmp_path, monkeypatch) + cap.tick() # no marker yet: polled once + path.write_text( + json.dumps({"schema_version": 1, "capture_at_unix_s": time.time() - 1, "num_steps": 50}) + ) + clock.t += 0.5 + cap.tick() # within the second: not re-read + assert manager.starts == [] + clock.t += 0.6 + cap.tick() # re-read and armed; capture time already passed: started + assert manager.starts == [{"num_steps": 50, "activities": ["CUDA_PROFILER"]}] + + +def test_waits_for_capture_time_and_skips_warmup(tmp_path, monkeypatch): + cap, manager, executor, clock, _ = _capture( + tmp_path, monkeypatch, {"schema_version": 1, "capture_at_unix_s": time.time() + 3600} + ) + executor.is_warmup = True + cap.tick() + assert cap._capture_at is None # nothing read during warmup + executor.is_warmup = False + cap.tick() + assert cap._capture_at is not None and manager.starts == [] + + +def test_stops_on_num_steps(tmp_path, monkeypatch): + cap, manager, executor, clock, _ = _capture( + tmp_path, monkeypatch, {"schema_version": 1, "capture_at_unix_s": 1.0} + ) + cap.tick() + executor._profile_enabled = True + cap.tick() + executor._profile_enabled = False # the broadcast stop fired after num_steps + cap.tick() + assert cap.done and manager.stop_configs == 0 + + +def test_stops_on_max_seconds_without_blocking(tmp_path, monkeypatch): + cap, manager, executor, clock, _ = _capture( + tmp_path, monkeypatch, {"schema_version": 1, "capture_at_unix_s": 1.0, "max_seconds": 2} + ) + cap.tick() + executor._profile_enabled = True + clock.t += 2.5 + cap.tick() + assert cap.done and manager.stop_configs == 1 and executor.executor_request_queue.stops == 1 + + +@pytest.mark.parametrize( + "marker", + [ + {"schema_version": 2, "capture_at_unix_s": 1.0}, + {"schema_version": 1}, + {"schema_version": 1, "capture_at_unix_s": 1.0, "num_steps": 0}, + {"schema_version": 1, "capture_at_unix_s": 1.0, "max_seconds": 500}, + ], +) +def test_bad_marker_disarms(tmp_path, monkeypatch, marker): + cap, manager, *_ = _capture(tmp_path, monkeypatch, marker) + cap.tick() + assert cap.done and manager.starts == [] + + +# ---- host path probe ------------------------------------------------------------------------------------------ + + +class _Event: + def __init__(self): + self.syncs = 0 + + def synchronize(self): + self.syncs += 1 + + +def test_probe_rows_and_passthrough(tmp_path, monkeypatch): + monkeypatch.setenv(hpp.HOST_PATH_PROBE_ENV_VAR_NAME, str(tmp_path)) + calls = [] + executor = SimpleNamespace(dist=SimpleNamespace(rank=0), global_rank=3) + executor._forward_step = lambda req, *a, **k: calls.append(("fwd", req, a, k)) or "out" + executor._update_requests = lambda state, *a, **k: calls.append(("upd", state, a, k)) or None + probe = hpp.make_host_path_probe(executor) + req = SimpleNamespace(num_context_requests=2, generation_requests=[1, 2, 3]) + event = _Event() + state = SimpleNamespace(sampler_event=event) + for _ in range(3): + probe.top() + assert executor._forward_step(req, "t", x=1) == "out" + executor._update_requests(state, "rm") + probe.top() + probe.flush() + (f,) = list(tmp_path.glob("hostpath-rank3-pid*.bin")) + rows = array.array("q") + rows.frombytes(f.read_bytes()) + assert len(rows) == 3 * 7 + for i in range(3): + t_top, t_launch, t_fwd, t_w0, t_w1, n_ctx, n_gen = rows[7 * i : 7 * i + 7] + assert t_top <= t_launch <= t_fwd <= t_w0 <= t_w1 + assert (n_ctx, n_gen) == (2, 3) + assert event.syncs == 3 + assert calls[0] == ("fwd", req, ("t",), {"x": 1}) and calls[1] == ("upd", state, ("rm",), {}) From a3f86ac155db90fc3b55e92be5501878bcf1da31 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 23:22:53 -0700 Subject: [PATCH 33/38] [None][test] Run the iteration-stats RPC test in the CPU test list The async stats test needs no device; list it with the neighbouring executor RPC tests so CI collects it. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- tests/integration/test_lists/test-db/l0_cpu.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/integration/test_lists/test-db/l0_cpu.yml b/tests/integration/test_lists/test-db/l0_cpu.yml index c9af143873af..0f4babd9d3ce 100644 --- a/tests/integration/test_lists/test-db/l0_cpu.yml +++ b/tests/integration/test_lists/test-db/l0_cpu.yml @@ -171,6 +171,7 @@ l0_cpu: - unittest/executor/test_fatal_error_health_check.py - unittest/executor/test_ipc.py - unittest/executor/test_rpc.py + - unittest/executor/test_proxy_aget_stats_async.py - unittest/executor/test_multi_frontend_routing.py - unittest/executor/test_event_loop_error_broadcast.py - unittest/executor/test_rpc_worker_mixin.py From 6cb186f317957e05cf6f5996b3b41feba6f8c695 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 23:51:49 -0700 Subject: [PATCH 34/38] [None][perf] Run the MiniMax-M3.1 TP2 router as an exact half-row TF32 GEMM at qualified row counts The TP2 target's sequence-parallel router computes each rank's owned half of the rows with a native TF32 GEMM (trtllm::m31_router_half_pin, built into th_common) instead of the full-shape router, at the full-step row counts that were qualified bitwise against the full-row cuBLAS reference; every other layout keeps the stock full-row path. Routing arms it when it selects the TP2 target. Kill switch: TRTLLM_M31_DISABLE_ROUTER_HALF_PIN=1. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- cpp/tensorrt_llm/thop/CMakeLists.txt | 1 + cpp/tensorrt_llm/thop/m31RouterHalfPinOp.cpp | 136 ++++++++++++++++++ .../catalog/gemm/m31_router_half_pin.py | 100 +++++++++++++ .../modeling_v2/models/minimax_m3/routing.py | 3 + 4 files changed, 240 insertions(+) create mode 100644 cpp/tensorrt_llm/thop/m31RouterHalfPinOp.cpp create mode 100644 tensorrt_llm/_torch/_experimental/modeling_v2/catalog/gemm/m31_router_half_pin.py diff --git a/cpp/tensorrt_llm/thop/CMakeLists.txt b/cpp/tensorrt_llm/thop/CMakeLists.txt index 5f29b51649d8..fb20068be8d5 100644 --- a/cpp/tensorrt_llm/thop/CMakeLists.txt +++ b/cpp/tensorrt_llm/thop/CMakeLists.txt @@ -165,6 +165,7 @@ target_sources( add_th_common_family( th_common_gemm_quant cutlassScaledMM.cpp + m31RouterHalfPinOp.cpp cublasScaledMM.cpp cublasFp4ScaledMM.cpp cudaNvfp4MM.cpp diff --git a/cpp/tensorrt_llm/thop/m31RouterHalfPinOp.cpp b/cpp/tensorrt_llm/thop/m31RouterHalfPinOp.cpp new file mode 100644 index 000000000000..568b418b40b7 --- /dev/null +++ b/cpp/tensorrt_llm/thop/m31RouterHalfPinOp.cpp @@ -0,0 +1,136 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 +#include +#include +#include +#include +#include +#include +#include +#include + +namespace +{ +void check(cublasStatus_t status) +{ + TORCH_CHECK(status == CUBLAS_STATUS_SUCCESS, "M31 router cuBLASLt status ", int(status)); +} + +struct Plan +{ + cublasLtHandle_t handle{}; + cublasLtMatmulDesc_t op{}; + cublasLtMatrixLayout_t weight{}, input{}, output{}; + cublasLtMatmulAlgo_t algo{}; + bool supported{false}; + + ~Plan() + { + if (output) + cublasLtMatrixLayoutDestroy(output); + if (input) + cublasLtMatrixLayoutDestroy(input); + if (weight) + cublasLtMatrixLayoutDestroy(weight); + if (op) + cublasLtMatmulDescDestroy(op); + if (handle) + cublasLtDestroy(handle); + } +}; + +std::mutex mutex; +std::map, std::unique_ptr> plans; + +int config(cublasLtMatmulAlgo_t const& algo, cublasLtMatmulAlgoConfigAttributes_t attr) +{ + int value{}; + size_t written{}; + check(cublasLtMatmulAlgoConfigGetAttribute(&algo, attr, &value, sizeof(value), &written)); + TORCH_CHECK(written == sizeof(value), "Unexpected cuBLASLt tactic attribute size"); + return value; +} + +Plan& prepare(int64_t rows, int64_t device) +{ + auto key = std::make_pair(device, rows); + auto found = plans.find(key); + if (found != plans.end()) + return *found->second; + c10::cuda::CUDAGuard guard{c10::Device(c10::kCUDA, device)}; + auto p = std::make_unique(); + check(cublasLtCreate(&p->handle)); + check(cublasLtMatmulDescCreate(&p->op, CUBLAS_COMPUTE_32F_FAST_TF32, CUDA_R_32F)); + cublasOperation_t transpose = CUBLAS_OP_T; + check(cublasLtMatmulDescSetAttribute(p->op, CUBLASLT_MATMUL_DESC_TRANSA, &transpose, sizeof(transpose))); + check(cublasLtMatrixLayoutCreate(&p->weight, CUDA_R_32F, 6144, 128, 6144)); + check(cublasLtMatrixLayoutCreate(&p->input, CUDA_R_32F, 6144, rows, 6144)); + check(cublasLtMatrixLayoutCreate(&p->output, CUDA_R_32F, 128, rows, 128)); + cublasLtMatmulPreference_t preference{}; + check(cublasLtMatmulPreferenceCreate(&preference)); + uint64_t workspace = 64ULL << 20; + auto status = cublasLtMatmulPreferenceSetAttribute( + preference, CUBLASLT_MATMUL_PREF_MAX_WORKSPACE_BYTES, &workspace, sizeof(workspace)); + cublasLtMatmulHeuristicResult_t candidates[64]{}; + int count{}; + if (status == CUBLAS_STATUS_SUCCESS) + status = cublasLtMatmulAlgoGetHeuristic( + p->handle, p->op, p->weight, p->input, p->output, p->output, preference, 64, candidates, &count); + cublasLtMatmulPreferenceDestroy(preference); + check(status); + for (int i = 0; i < count; ++i) + { + auto const& candidate = candidates[i]; + auto const& a = candidate.algo; + if (candidate.state == CUBLAS_STATUS_SUCCESS && candidate.workspaceSize == 0 + && config(a, CUBLASLT_ALGO_CONFIG_ID) == 66 + && config(a, CUBLASLT_ALGO_CONFIG_TILE_ID) == (rows <= 4096 ? 15 : 321) + && config(a, CUBLASLT_ALGO_CONFIG_STAGES_ID) == 34 && config(a, CUBLASLT_ALGO_CONFIG_SPLITK_NUM) == 1 + && config(a, CUBLASLT_ALGO_CONFIG_REDUCTION_SCHEME) == 0 + && config(a, CUBLASLT_ALGO_CONFIG_CTA_SWIZZLING) == 0 + && config(a, CUBLASLT_ALGO_CONFIG_CUSTOM_OPTION) == (rows <= 4096 ? 3 : 1)) + { + p->algo = a; + p->supported = true; + break; + } + } + return *plans.emplace(key, std::move(p)).first->second; +} + +bool supported(int64_t rows, int64_t device) +{ + std::lock_guard lock(mutex); + return prepare(rows, device).supported; +} + +at::Tensor router(at::Tensor const& x, at::Tensor const& w) +{ + TORCH_CHECK(x.is_cuda() && w.device() == x.device(), "router tensors must share a CUDA device"); + TORCH_CHECK(x.scalar_type() == at::kFloat && w.scalar_type() == at::kFloat && x.dim() == 2 && w.dim() == 2 + && x.size(1) == 6144 && w.size(0) == 128 && w.size(1) == 6144 && x.is_contiguous() && w.is_contiguous(), + "unsupported router layout"); + c10::cuda::CUDAGuard guard(x.device()); + auto output = at::empty({x.size(0), 128}, x.options()); + std::lock_guard lock(mutex); + auto found = plans.find({x.get_device(), x.size(0)}); + TORCH_CHECK(found != plans.end() && found->second->supported, "router plan must be prepared before capture"); + auto& p = *found->second; + float alpha = 1.0F, beta = 0.0F; + check( + cublasLtMatmul(p.handle, p.op, &alpha, w.data_ptr(), p.weight, x.data_ptr(), p.input, &beta, output.data_ptr(), + p.output, output.data_ptr(), p.output, &p.algo, nullptr, 0, at::cuda::getCurrentCUDAStream())); + return output; +} +} // namespace + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("m31_router_half_pin_prepare(int rows, int device) -> bool", &supported); + m.def("m31_router_half_pin(Tensor x, Tensor weight) -> Tensor"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("m31_router_half_pin", &router); +} diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/gemm/m31_router_half_pin.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/gemm/m31_router_half_pin.py new file mode 100644 index 000000000000..acfd7261feb4 --- /dev/null +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/gemm/m31_router_half_pin.py @@ -0,0 +1,100 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Exact, qualified half-row TF32 router dispatch for the M3.1 TP2 IPC boundary.""" + +import os +from pathlib import Path + +import torch + +# Only the full-step row counts qualified bitwise against the full-row cuBLAS reference. +QUALIFIED_ROWS = frozenset( + (8000, 16000, 16384, 16317, 16314, 16313, 16341, 16308, 16334, 16296, 16259) +) +_DISABLED = os.environ.get("TRTLLM_M31_DISABLE_ROUTER_HALF_PIN", "0") == "1" +_PREPARED = {} +_LOADED = False +_LOGGED = set() + + +def _load(): + global _LOADED + if _LOADED: + return hasattr(torch.ops.trtllm, "m31_router_half_pin") + import tensorrt_llm + + library = Path(tensorrt_llm.__file__).parent / "libs/libm31_router_half_pin.so" + if not hasattr(torch.ops.trtllm, "m31_router_half_pin") and library.is_file(): + torch.ops.load_library(str(library)) + if hasattr(torch.ops.trtllm, "m31_router_half_pin"): + + @torch.library.register_fake("trtllm::m31_router_half_pin") + def fake(x, weight): + return x.new_empty((x.shape[0], weight.shape[0])) + + _LOADED = True + return hasattr(torch.ops.trtllm, "m31_router_half_pin") + + +def try_router_half_pin(x, weight, total_rows): + """Return None for an unqualified layout/tactic, retaining the stock full-row path.""" + if ( + _DISABLED + or total_rows not in QUALIFIED_ROWS + or not x.is_cuda + or x.dtype != torch.float32 + or weight.dtype != torch.float32 + or x.dim() != 2 + or weight.shape != (128, 6144) + or x.shape[1] != 6144 + or not x.is_contiguous() + or not weight.is_contiguous() + or x.device != weight.device + or not torch.backends.cuda.matmul.allow_tf32 + ): + return None + split = min(max((total_rows // 2 + 64) // 128 * 128, 128), total_rows - 1) + if x.shape[0] not in (split, total_rows - split): + return None + key = (x.device.index, x.shape[0]) + if key not in _PREPARED: + if torch.cuda.is_current_stream_capturing(): + return None + if torch.cuda.get_device_capability(x.device) != (10, 7) or not _load(): + _PREPARED[key] = False + else: + _PREPARED[key] = torch.ops.trtllm.m31_router_half_pin_prepare( + x.shape[0], x.device.index + ) + if not _PREPARED[key]: + return None + if key not in _LOGGED: + from tensorrt_llm.logger import logger + + logger.info( + f"[M3.1] ROUTER_HALF_PIN taken device={x.device.index} rows={x.shape[0]} total={total_rows}" + ) + _LOGGED.add(key) + return torch.ops.trtllm.m31_router_half_pin(x, weight) + + +_ARMED = False + + +def arm_router_half_pin() -> None: + """Install only when routing the qualified TP2 model; keep digested comm source intact.""" + global _ARMED + if _ARMED or _DISABLED: + return + from ..comm.tp2ctx_ipc import Tp2ctxIpcWorkspace + + stock = Tp2ctxIpcWorkspace.route_owned_full_shape + + def route_owned(self, x_owned, weight, num_tokens): + pinned = try_router_half_pin(x_owned, weight, num_tokens) + if pinned is not None: + return pinned + return stock(self, x_owned, weight, num_tokens) + + Tp2ctxIpcWorkspace.route_owned_full_shape = route_owned + _ARMED = True diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py index 96b8bdc85743..5abc6573cec1 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py @@ -246,6 +246,9 @@ def route( register() arm_sm107_shared_opt_ins() if parallel == "tp2": + from ...catalog.gemm.m31_router_half_pin import arm_router_half_pin + + arm_router_half_pin() from .minimax_m31_nvfp4__sm_107__tp2.modeling import ModelingV2MiniMaxM31Nvfp4Sm107Tp2 return ModelingV2MiniMaxM31Nvfp4Sm107Tp2 From 41cd32fe4bc9301f31a1dbbae54cb6bf15e0b5a6 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 23:54:06 -0700 Subject: [PATCH 35/38] [None][test] Cover the MiniMax-M3.1 TP2 router half-row pin - unittest/others/test_m31_router_half_pin.py (CPU, collected by the l0_cpu unittest/others entry): arming, the kill switch, idempotence and the fallback to the full-row path. - unittest/_torch/thop/parallel/test_m31_router_half_pin.py (GPU, sm_107): bitwise against the full-row reference at every qualified shape and ownership, in eager mode and under CUDA graph replay with changing inputs, plus the unqualified-layout guards. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../thop/parallel/test_m31_router_half_pin.py | 66 +++++++++++++++ .../others/test_m31_router_half_pin.py | 81 +++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 tests/unittest/_torch/thop/parallel/test_m31_router_half_pin.py create mode 100644 tests/unittest/others/test_m31_router_half_pin.py diff --git a/tests/unittest/_torch/thop/parallel/test_m31_router_half_pin.py b/tests/unittest/_torch/thop/parallel/test_m31_router_half_pin.py new file mode 100644 index 000000000000..c2a457286e03 --- /dev/null +++ b/tests/unittest/_torch/thop/parallel/test_m31_router_half_pin.py @@ -0,0 +1,66 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""SM107 router pin: full-shape stock bytes on both ownership slices, eager and graph.""" + +import pytest +import torch + +from tensorrt_llm._torch._experimental.modeling_v2.catalog.gemm import m31_router_half_pin as pin + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() != (10, 7), + reason="Requires SM107", +) + + +@pytest.fixture(autouse=True) +def dispatch_state(monkeypatch): + monkeypatch.setattr(torch.backends.cuda.matmul, "allow_tf32", True) + monkeypatch.setattr(pin, "_DISABLED", False) + monkeypatch.setattr(pin, "_PREPARED", {}) + monkeypatch.setattr(pin, "_LOGGED", set()) + assert pin._load(), "SM107 CI must ship the native router pin operator" + + +@pytest.mark.parametrize("total_rows", sorted(pin.QUALIFIED_ROWS)) +@pytest.mark.parametrize("owner", [0, 1]) +def test_router_half_pin_exact_eager_and_graph(total_rows, owner): + split = min(max((total_rows // 2 + 64) // 128 * 128, 128), total_rows - 1) + start, rows = (0, split) if owner == 0 else (split, total_rows - split) + generator = torch.Generator(device="cuda").manual_seed(20623 + total_rows + owner) + x = torch.randn(rows, 6144, device="cuda", generator=generator) + weight = torch.randn(128, 6144, device="cuda", generator=generator) + full = torch.zeros(total_rows, 6144, device="cuda") + + def reference(): + full[start : start + rows].copy_(x) + return torch.nn.functional.linear(full, weight)[start : start + rows] + + expected = reference() + actual = pin.try_router_half_pin(x, weight, total_rows) + assert actual is not None, "Qualified native descriptor unexpectedly declined" + assert torch.equal(actual.view(torch.int32), expected.view(torch.int32)) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + actual = pin.try_router_half_pin(x, weight, total_rows) + assert actual is not None + for _ in range(3): + # Change replay inputs as well: stale eager results cannot satisfy this oracle. + x.mul_(0.5) + weight.add_(0.015625) + expected = reference() + graph.replay() + assert torch.equal(actual.view(torch.int32), expected.view(torch.int32)) + + +def test_router_half_pin_guards(monkeypatch): + x = torch.empty(8192, 6144, device="cuda") + weight = torch.empty(128, 6144, device="cuda") + monkeypatch.setattr(pin, "_DISABLED", True) + assert pin.try_router_half_pin(x, weight, 16384) is None + monkeypatch.setattr(pin, "_DISABLED", False) + assert pin.try_router_half_pin(x, weight, 16383) is None + assert pin.try_router_half_pin(x[:-1], weight, 16384) is None + monkeypatch.setattr(torch.backends.cuda.matmul, "allow_tf32", False) + assert pin.try_router_half_pin(x, weight, 16384) is None + assert pin._PREPARED == {}, "Declined inputs must not prepare a native descriptor" diff --git a/tests/unittest/others/test_m31_router_half_pin.py b/tests/unittest/others/test_m31_router_half_pin.py new file mode 100644 index 000000000000..b40fa868ad5c --- /dev/null +++ b/tests/unittest/others/test_m31_router_half_pin.py @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Device-free tests of the production router arm, with only GPU/IPC dependencies mocked.""" + +import importlib.util +import sys +from pathlib import Path +from types import ModuleType, SimpleNamespace + +import pytest + +pytestmark = pytest.mark.cpu_only +_ROOT = Path(__file__).resolve().parents[3] +_CATALOG = "tensorrt_llm._torch._experimental.modeling_v2.catalog" +_SOURCE = ( + _ROOT / "tensorrt_llm/_torch/_experimental/modeling_v2/catalog/gemm/m31_router_half_pin.py" +) + + +@pytest.fixture +def router(monkeypatch): + calls = [] + + class Workspace: + def route_owned_full_shape(self, x, weight, rows): + calls.append((self, x, weight, rows)) + return "stock" + + ipc = ModuleType(_CATALOG + ".comm.tp2ctx_ipc") + ipc.Tp2ctxIpcWorkspace = Workspace + monkeypatch.setitem(sys.modules, ipc.__name__, ipc) + monkeypatch.setitem(sys.modules, "torch", SimpleNamespace()) + monkeypatch.delenv("TRTLLM_M31_DISABLE_ROUTER_HALF_PIN", raising=False) + + def load(): + spec = importlib.util.spec_from_file_location( + _CATALOG + ".gemm._router_pin_under_test", _SOURCE + ) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + return load, Workspace, calls + + +def test_kill_switch_prevents_arming_and_dispatch(router, monkeypatch): + load, workspace, calls = router + monkeypatch.setenv("TRTLLM_M31_DISABLE_ROUTER_HALF_PIN", "1") + pin = load() + stock = workspace.route_owned_full_shape + pin.arm_router_half_pin() + assert workspace.route_owned_full_shape is stock + assert not pin._ARMED + # The kill switch must decline before touching tensor or CUDA properties. + assert pin.try_router_half_pin(object(), object(), 16384) is None + assert calls == [] + + +def test_default_arm_is_idempotent_and_preserves_fallback(router, monkeypatch): + load, workspace, calls = router + pin = load() + assert not pin._DISABLED + pinned = object() + dispatches = [] + + def dispatch(x, weight, rows): + dispatches.append((x, weight, rows)) + return pinned if rows == 16384 else None + + monkeypatch.setattr(pin, "try_router_half_pin", dispatch) + pin.arm_router_half_pin() + installed = workspace.route_owned_full_shape + pin.arm_router_half_pin() + assert workspace.route_owned_full_shape is installed + instance = workspace() + x, weight = object(), object() + assert instance.route_owned_full_shape(x, weight, 16384) is pinned + assert calls == [] + assert instance.route_owned_full_shape(x, weight, 31) == "stock" + assert calls == [(instance, x, weight, 31)] + assert dispatches == [(x, weight, 16384), (x, weight, 31)] From 82c5e772d92e3e9c75ec123e932cfd8a14890fdd Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 23:20:30 -0700 Subject: [PATCH 36/38] [None][perf] Time the executor with GPU timestamps on the MiniMax-M3.1 sm_107 targets The executor records timed CUDA events around every decode step: the per-request perf metrics' forward start, forward end and sample end, and the iteration log's device-step boundary. A timed record waits for its stream to drain, and while one is in flight the next decode graph's launch waits behind it: a few microseconds per record at every step boundary. GpuTimestamp (perf_metrics_manager.py) keeps the torch.cuda.Event surface the executor uses (record, query, synchronize, elapsed_time). record() makes the timestamp's own non-blocking follower stream wait for the recording stream, then stores %globaltimer into a device word with the new native op trtllm::minimax_m31_globaltimer_stamp (one thread, kernels/minimaxM31/globaltimerStamp.cu), copies it to pinned host memory and records an untimed event, all on the follower. Under stream capture it records nothing. timing_event() returns it for the perf metrics and the iteration log's events once armed. Routing arms it process-wide for the sm_107 targets (arm_sm107_shared_opt_ins -> arm_gpu_timestamps), without editing the target directories. It arms only when the native op is present and pinned memory is preferred; otherwise timed events stay. TRTLLM_M31_DISABLE_GPU_TIMESTAMPS=1 keeps timed events. Each process logs one "[M31] executor timing: ..." line. Measured (same node, untraced, c1 K7 decode with the perf metrics and the iteration log on, replay period of the dominant decode graph, the Triton stand-in stamp): TP4 -9.5 us median / -7.6 us trimmed mean per step, TP8 -6.7 / -11.2 us (about -0.2..-0.4% of the step). Unit tests: GpuTimestamp elapsed time matches timed events over a spin, query / unrecorded semantics, no record under capture, timing_event follows the arm, the kill switch; the shared opt-ins test covers the new arm call, flag and kill switch. Ported from dongfengy/tekit 5f03c8e89 (GLM-5), with the stamp as a native op. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../kernels/minimaxM31/globaltimerStamp.cu | 48 ++++++ .../kernels/minimaxM31/globaltimerStamp.h | 40 +++++ cpp/tensorrt_llm/thop/CMakeLists.txt | 1 + .../minimaxM31GlobaltimerStampOp.cpp | 53 +++++++ .../modeling_v2/models/minimax_m3/routing.py | 6 +- .../_torch/pyexecutor/perf_metrics_manager.py | 137 +++++++++++++++++- tensorrt_llm/_torch/pyexecutor/profiling.py | 25 ++-- .../_torch/executor/test_gpu_timestamp.py | 95 ++++++++++++ .../test_m31_sm107_shared_opt_ins.py | 7 +- 9 files changed, 394 insertions(+), 18 deletions(-) create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/globaltimerStamp.cu create mode 100644 cpp/tensorrt_llm/kernels/minimaxM31/globaltimerStamp.h create mode 100644 cpp/tensorrt_llm/thop/minimaxM31/minimaxM31GlobaltimerStampOp.cpp create mode 100644 tests/unittest/_torch/executor/test_gpu_timestamp.py diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/globaltimerStamp.cu b/cpp/tensorrt_llm/kernels/minimaxM31/globaltimerStamp.cu new file mode 100644 index 000000000000..7329dfeadcfa --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/globaltimerStamp.cu @@ -0,0 +1,48 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/globaltimerStamp.h" + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +namespace +{ + +__global__ void globaltimerStampKernel(int64_t* out) +{ + uint64_t t; + asm volatile("mov.u64 %0, %%globaltimer;" : "=l"(t)); + *out = static_cast(t); +} + +} // namespace + +cudaError_t invokeGlobaltimerStamp(int64_t* out, cudaStream_t stream) +{ + globaltimerStampKernel<<<1, 1, 0, stream>>>(out); + return cudaGetLastError(); +} + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/minimaxM31/globaltimerStamp.h b/cpp/tensorrt_llm/kernels/minimaxM31/globaltimerStamp.h new file mode 100644 index 000000000000..2250ea6e1b7a --- /dev/null +++ b/cpp/tensorrt_llm/kernels/minimaxM31/globaltimerStamp.h @@ -0,0 +1,40 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels +{ +namespace minimaxM31 +{ + +//! One thread stores the GPU's %globaltimer (nanoseconds) into out[0] when the stream reaches the launch: the +//! executor's timing stamp (perf metrics, the iteration log), which unlike a timed CUDA event record does not wait +//! for its stream to drain. out is an int64 word in device memory. +cudaError_t invokeGlobaltimerStamp(int64_t* out, cudaStream_t stream); + +} // namespace minimaxM31 +} // namespace kernels + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/thop/CMakeLists.txt b/cpp/tensorrt_llm/thop/CMakeLists.txt index fb20068be8d5..fe8ba6f8e9a6 100644 --- a/cpp/tensorrt_llm/thop/CMakeLists.txt +++ b/cpp/tensorrt_llm/thop/CMakeLists.txt @@ -84,6 +84,7 @@ target_sources( minimaxM31/minimaxM31DsparkGlueOp.cpp minimaxM31/minimaxM31F3ArOp.cpp minimaxM31/minimaxM31FcShardGemmOp.cpp + minimaxM31/minimaxM31GlobaltimerStampOp.cpp minimaxM31/minimaxM31MarkovEpilogueOp.cpp minimaxM31/minimaxM31MoeFinalizeOp.cpp minimaxM31/minimaxM31S1bFoldOp.cpp diff --git a/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31GlobaltimerStampOp.cpp b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31GlobaltimerStampOp.cpp new file mode 100644 index 000000000000..2a55dafad792 --- /dev/null +++ b/cpp/tensorrt_llm/thop/minimaxM31/minimaxM31GlobaltimerStampOp.cpp @@ -0,0 +1,53 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/minimaxM31/globaltimerStamp.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +//! Store the GPU's %globaltimer (ns) into out[0] on the current stream (see kernels/minimaxM31/globaltimerStamp.h). +//! out: an int64 CUDA tensor of >= 1 element. +void minimaxM31GlobaltimerStamp(torch::Tensor& out) +{ + TORCH_CHECK(out.is_cuda() && out.scalar_type() == torch::kInt64 && out.numel() >= 1, + "minimax_m31_globaltimer_stamp: out must be an int64 CUDA tensor of >= 1 element"); + at::cuda::CUDAGuard const guard(out.device()); + cudaError_t const e = tensorrt_llm::kernels::minimaxM31::invokeGlobaltimerStamp( + out.data_ptr(), at::cuda::getCurrentCUDAStream(out.get_device())); + TORCH_CHECK(e == cudaSuccess, "minimax_m31_globaltimer_stamp launch: ", cudaGetErrorString(e)); +} + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def("minimax_m31_globaltimer_stamp(Tensor(a!) out) -> ()"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("minimax_m31_globaltimer_stamp", &tensorrt_llm::torch_ext::minimaxM31GlobaltimerStamp); +} diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py index 5abc6573cec1..cb9f02f86316 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/routing.py @@ -155,13 +155,14 @@ def arm_sm107_shared_opt_ins() -> None: """Arm, process-wide and before the model is built, the shared paths the MiniMax-M3.1 sm_107 targets take: the shipped ahead-of-time CuTe DSL objects, the drafter KV reserve and guard page, the host-tier KV prefetch, the context-only int32 prompt tokens, the BF16X / E4M3X sparse-prefill fork, the indexer proxy max-score buffer reuse, - the context one-hot index_fill and the DFlash host-side context allocation. Each keeps its kill switch. Armed here + the context one-hot index_fill, the DFlash host-side context allocation and GPU timestamps for the executor's + timing events. Each keeps its kill switch. Armed here rather than in a target's RuntimeHooks so that the target directories (the AOT objects' source digests) are not edited.""" from tensorrt_llm._torch import cute_aot from tensorrt_llm._torch.attention.backends.fmha import msa_prefill from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import msa_indexer - from tensorrt_llm._torch.pyexecutor import llm_request, py_executor + from tensorrt_llm._torch.pyexecutor import llm_request, perf_metrics_manager, py_executor from tensorrt_llm._torch.pyexecutor.kv_cache import kv_cache_manager_v2 from tensorrt_llm._torch.speculative import dflash, interface @@ -173,6 +174,7 @@ def arm_sm107_shared_opt_ins() -> None: msa_indexer.arm_proxy_ms_reuse() interface.arm_onehot_index_fill() dflash.arm_host_ctx_alloc() + perf_metrics_manager.arm_gpu_timestamps() def route( diff --git a/tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py b/tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py index d129095b0045..03e04ceab460 100644 --- a/tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py +++ b/tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py @@ -8,17 +8,144 @@ readability and separation of concerns. """ +import os from contextlib import contextmanager from types import SimpleNamespace -from typing import Optional +from typing import Optional, Union import torch -from tensorrt_llm._utils import get_steady_clock_now_in_seconds +from tensorrt_llm._utils import get_steady_clock_now_in_seconds, prefer_pinned from tensorrt_llm.logger import logger from .llm_request import PerfTimingInfo +# GPU timestamps for the executor's timing events (the per-request perf metrics, the iteration log's device step): +# a model opt-in armed process-wide (arm_gpu_timestamps; the MiniMax-M3.1 sm_107 routing arms it). Kill switch +# TRTLLM_M31_DISABLE_GPU_TIMESTAMPS=1. TRTLLM_M31_GPU_TIMESTAMPS_TRITON_PROTO=1 is a measurement-only fallback that +# writes the stamp with a Triton kernel when the native stamp op is not built. +_GPU_TIMESTAMPS = False +_TRITON_PROTO = os.environ.get("TRTLLM_M31_GPU_TIMESTAMPS_TRITON_PROTO", "0") == "1" + + +def _native_stamp_op(): + try: + return torch.ops.trtllm.minimax_m31_globaltimer_stamp + except (AttributeError, RuntimeError): + return None + + +def arm_gpu_timestamps() -> bool: + """Time the executor with GpuTimestamp in this process, unless killed or no stamp op is available.""" + global _GPU_TIMESTAMPS + if os.environ.get("TRTLLM_M31_DISABLE_GPU_TIMESTAMPS", "0") == "1": + _GPU_TIMESTAMPS = False + else: + # The stamp lands in pinned host memory with a non-blocking copy; without pinned memory (Confidential + # Compute) that copy would block the host, so keep timed events there. + _GPU_TIMESTAMPS = (_native_stamp_op() is not None or _TRITON_PROTO) and prefer_pinned() + logger.info( + f"[M31] executor timing: {'GPU timestamps' if _GPU_TIMESTAMPS else 'timed CUDA events'}" + f"{' (Triton prototype stamp)' if _GPU_TIMESTAMPS and _native_stamp_op() is None else ''}" + ) + return _GPU_TIMESTAMPS + + +def timing_event() -> Union[torch.cuda.Event, "GpuTimestamp"]: + """A GPU timing event for the executor: a GpuTimestamp when armed, else a timed CUDA event.""" + if _GPU_TIMESTAMPS: + return GpuTimestamp() + return torch.cuda.Event(enable_timing=True) + + +class GpuTimestamp: + """The torch.cuda.Event surface the executor's timing uses (record, query, synchronize, elapsed_time), measured + with %globaltimer. + + A timed event record waits for its stream to drain before it takes its timestamp, and while one is in flight the + next decode graph's launch waits behind it (a few microseconds per record on Rubin, at every step boundary). + record() instead makes this timestamp's own non-blocking follower stream wait for the recording stream, then + stores %globaltimer there into a device word, copies it to pinned host memory and records an untimed event, all + on the follower. The stamp is taken when the follower passes the recording stream's position, as a timed record's + timestamp is. Each timestamp has its own follower, so stamps recorded at one position do not queue behind each + other. Under stream capture record() records nothing. Ported from dongfengy/tekit 5f03c8e89 (GLM-5), with the + stamp as a native op. + """ + + _triton_kernel = None + + def __init__(self): + self._follower: Optional[torch.cuda.Stream] = None + self._stamp: Optional[torch.Tensor] = None + self._host: Optional[torch.Tensor] = None + self._done: Optional[torch.cuda.Event] = None + self._recorded = False + + @staticmethod + def _new_stream(device: torch.device) -> torch.cuda.Stream: + """A non-blocking stream of its own (not one of PyTorch's pooled streams, which can alias the timed one).""" + from tensorrt_llm._utils import CUASSERT + + try: + from cuda.bindings import runtime as cudart + except ImportError: + from cuda import cudart + index = device.index if device.index is not None else torch.cuda.current_device() + with torch.cuda.device(index): + handle = CUASSERT(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))[0] + return torch.cuda.ExternalStream(int(handle), device=torch.device("cuda", index)) + + @classmethod + def _write_stamp(cls, out: torch.Tensor) -> None: + op = _native_stamp_op() + if op is not None: + op(out) + return + if cls._triton_kernel is None: + import triton + import triton.language as tl + + @triton.jit + def _globaltimer_kernel(out_ptr): + tl.store(out_ptr, tl.extra.cuda.globaltimer()) + + cls._triton_kernel = _globaltimer_kernel + cls._triton_kernel[(1,)](out) + + def record(self, stream: Optional[torch.cuda.Stream] = None) -> None: + if stream is None: + stream = torch.cuda.current_stream() + if torch.cuda.is_current_stream_capturing(): + return + if self._follower is None: + self._follower = self._new_stream(stream.device) + follower = self._follower + follower.wait_stream(stream) + with torch.cuda.stream(follower): + if self._stamp is None: + self._stamp = torch.empty(1, dtype=torch.int64, device=stream.device) + self._host = torch.empty(1, dtype=torch.int64, pin_memory=prefer_pinned()) + self._done = torch.cuda.Event() + self._write_stamp(self._stamp) + self._host.copy_(self._stamp, non_blocking=True) + self._done.record(follower) + self._recorded = True + + def query(self) -> bool: + return not self._recorded or self._done.query() + + def synchronize(self) -> None: + if self._recorded: + self._done.synchronize() + + def elapsed_time(self, end_event: "GpuTimestamp") -> float: + """Milliseconds from this stamp to ``end_event``'s, once both landed.""" + if not (self._recorded and end_event._recorded): + raise RuntimeError("GpuTimestamp.elapsed_time: a timestamp was not recorded") + self.synchronize() + end_event.synchronize() + return (int(end_event._host[0]) - int(self._host[0])) / 1e6 + class PerfMetricsManager: """Manages GPU/CPU timing instrumentation for PyExecutor iterations. @@ -56,8 +183,8 @@ def create_timing_events(self): return None, None, None if self._perf_events is None: self._perf_events = [ - tuple(torch.cuda.Event(enable_timing=True) for _ in range(3)), - tuple(torch.cuda.Event(enable_timing=True) for _ in range(3)), + tuple(timing_event() for _ in range(3)), + tuple(timing_event() for _ in range(3)), ] self._perf_event_idx = 0 events = self._perf_events[self._perf_event_idx % 2] @@ -68,7 +195,7 @@ def borrow_forward_timing_events(self): """Borrow a forward-only pair when the ping-pong perf events are unavailable.""" if self._forward_event_pool: return self._forward_event_pool.pop() - return (torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)) + return (timing_event(), timing_event()) def release_forward_timing_events(self, start_event, end_event) -> None: if start_event is not None and end_event is not None: diff --git a/tensorrt_llm/_torch/pyexecutor/profiling.py b/tensorrt_llm/_torch/pyexecutor/profiling.py index 4c584d1587d8..692afac76a5a 100644 --- a/tensorrt_llm/_torch/pyexecutor/profiling.py +++ b/tensorrt_llm/_torch/pyexecutor/profiling.py @@ -56,6 +56,7 @@ from tensorrt_llm.tools.profiler.host_profile_tools.host_profiler import get_global_profiler from .host_path_probe import make_host_path_probe +from .perf_metrics_manager import GpuTimestamp, timing_event from .phase_capture import make_phase_capture from .runtime_hooks import runtime_hooks_of @@ -608,20 +609,21 @@ def profile_step(self): # batch. We need two sets of start-end events so that the # ping-pong pattern works with the overlap scheduler. start_event_1 = None - end_event_1 = torch.cuda.Event(enable_timing=True) + end_event_1 = timing_event() start_event_2 = None - end_event_2 = torch.cuda.Event(enable_timing=True) + end_event_2 = timing_event() prev_device_step_time = None # With RuntimeHooks.iteration_timing_one_event, boundary_events[k % 3] marks the boundary before # iteration k, recorded on timing_stream, and replaces the start/end ping-pong above. one_timing_event = runtime_hooks_of( getattr(executor.model_engine, "model", None) ).iteration_timing_one_event - boundary_events = ( - [torch.cuda.Event(enable_timing=True) for _ in range(3)] if one_timing_event else None - ) + boundary_events = [timing_event() for _ in range(3)] if one_timing_event else None boundary_count = 0 - timing_stream = torch.cuda.Stream() if one_timing_event else None + # A GpuTimestamp (arm_gpu_timestamps) has its own follower stream: it records at the current stream's position + # without the side stream. + stamp_boundaries = one_timing_event and isinstance(boundary_events[0], GpuTimestamp) + timing_stream = torch.cuda.Stream() if one_timing_event and not stamp_boundaries else None if one_timing_event: logger.info("[M3 host] iteration timing: one side-stream event per iteration boundary") # Cumulative fetch counter as of the previous iter log line; the @@ -755,8 +757,11 @@ def profile_step_fn(): should_capture_timing = executor.print_log or executor.enable_iter_perf_stats if should_capture_timing and one_timing_event: # Iteration k - 2 spans the two previous boundaries: the step the ping-pong reports here. - timing_stream.wait_stream(torch.cuda.current_stream()) - boundary_events[boundary_count % 3].record(timing_stream) + if stamp_boundaries: + boundary_events[boundary_count % 3].record() + else: + timing_stream.wait_stream(torch.cuda.current_stream()) + boundary_events[boundary_count % 3].record(timing_stream) if boundary_count >= 2: boundary_events[(boundary_count - 1) % 3].synchronize() prev_device_step_time = boundary_events[(boundary_count - 2) % 3].elapsed_time( @@ -897,11 +902,11 @@ def profile_step_fn(): if should_capture_timing and not one_timing_event: if it % 2 == 0: if start_event_1 is None: - start_event_1 = torch.cuda.Event(enable_timing=True) + start_event_1 = timing_event() start_event_1.record() else: if start_event_2 is None: - start_event_2 = torch.cuda.Event(enable_timing=True) + start_event_2 = timing_event() start_event_2.record() try: diff --git a/tests/unittest/_torch/executor/test_gpu_timestamp.py b/tests/unittest/_torch/executor/test_gpu_timestamp.py new file mode 100644 index 000000000000..75a143d43eaa --- /dev/null +++ b/tests/unittest/_torch/executor/test_gpu_timestamp.py @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""GpuTimestamp keeps torch.cuda.Event's timing semantics (perf_metrics_manager; armed by arm_gpu_timestamps). + +The stamp comes from the native op (trtllm.minimax_m31_globaltimer_stamp) when it is built, otherwise from the +measurement-only Triton fallback, which these tests force when the op is missing. +""" + +import pytest +import torch + +from tensorrt_llm._torch.pyexecutor import perf_metrics_manager as pmm + +pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") + + +@pytest.fixture(autouse=True) +def _stamp_path(monkeypatch): + if pmm._native_stamp_op() is None: + monkeypatch.setattr(pmm, "_TRITON_PROTO", True) + + +# A fixed spin of tens of milliseconds on Rubin-class clocks; the tests compare against the span that actually ran +# (timed events), so no cycles-per-ms calibration is needed (torch.cuda._sleep calibration is unreliable). +_SPIN_CYCLES = 40_000_000 + + +def _spin() -> None: + torch.cuda._sleep(_SPIN_CYCLES) + + +def test_elapsed_matches_timed_events(): + # Warm the spin and the stamp kernel first, so neither's first launch lands inside the measured window. + _spin() + warm = pmm.GpuTimestamp() + warm.record() + warm.synchronize() + torch.cuda.synchronize() + s, e = pmm.GpuTimestamp(), pmm.GpuTimestamp() + ts, te = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True) + ts.record() + s.record() + _spin() + e.record() + te.record() + torch.cuda.synchronize() + ref = ts.elapsed_time(te) + got = s.elapsed_time(e) + assert ref > 1.0 + assert abs(got - ref) < 0.05 + 0.02 * ref, (got, ref) + + +def test_query_and_unrecorded_semantics(): + t = pmm.GpuTimestamp() + assert t.query() # unrecorded: True, like torch.cuda.Event + with pytest.raises(RuntimeError): + t.elapsed_time(pmm.GpuTimestamp()) + _spin() + t.record() + assert not t.query() # behind the spin + t.synchronize() + assert t.query() + + +def test_record_under_capture_records_nothing(): + t = pmm.GpuTimestamp() + x = torch.zeros(4, device="cuda") + g = torch.cuda.CUDAGraph() + s = torch.cuda.Stream() + s.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(s): + x.add_(1) # warm the op outside capture + torch.cuda.current_stream().wait_stream(s) + with torch.cuda.graph(g): + x.add_(1) + t.record() + assert not t._recorded + g.replay() + torch.cuda.synchronize() + assert t.query() + + +def test_timing_event_follows_the_arm(monkeypatch): + monkeypatch.setattr(pmm, "_GPU_TIMESTAMPS", False) + assert isinstance(pmm.timing_event(), torch.cuda.Event) + monkeypatch.setattr(pmm, "_GPU_TIMESTAMPS", True) + assert isinstance(pmm.timing_event(), pmm.GpuTimestamp) + + +def test_kill_switch(monkeypatch): + monkeypatch.setenv("TRTLLM_M31_DISABLE_GPU_TIMESTAMPS", "1") + assert pmm.arm_gpu_timestamps() is False + monkeypatch.delenv("TRTLLM_M31_DISABLE_GPU_TIMESTAMPS") + assert pmm.arm_gpu_timestamps() is True + monkeypatch.setattr(pmm, "_GPU_TIMESTAMPS", False) diff --git a/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py b/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py index 5f5b0e97c7b4..d6b4161b48d1 100644 --- a/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py +++ b/tests/unittest/_torch/modeling_v2/test_m31_sm107_shared_opt_ins.py @@ -27,7 +27,7 @@ from tensorrt_llm._torch._experimental.modeling_v2.models.minimax_m3 import routing from tensorrt_llm._torch.attention.backends.fmha import msa_prefill from tensorrt_llm._torch.attention.backends.sparse.minimax_m3 import msa_indexer -from tensorrt_llm._torch.pyexecutor import llm_request, py_executor +from tensorrt_llm._torch.pyexecutor import llm_request, perf_metrics_manager, py_executor from tensorrt_llm._torch.pyexecutor.kv_cache import kv_cache_manager_v2 from tensorrt_llm._torch.pyexecutor.runtime_hooks import RuntimeHooks from tensorrt_llm._torch.speculative import dflash, interface @@ -52,6 +52,7 @@ "arm_proxy_ms_reuse", "arm_onehot_index_fill", "arm_host_ctx_alloc", + "arm_gpu_timestamps", ) # (module, flag, value once armed) for every process-wide flag the helper sets. FLAGS = ( @@ -64,6 +65,7 @@ (msa_indexer, "_PROXY_MS_REUSE", True), (interface, "_ONEHOT_INDEX_FILL", True), (dflash, "_HOST_CTX_ALLOC_DISABLED", False), + (perf_metrics_manager, "_GPU_TIMESTAMPS", True), ) KILL_SWITCHES = ( "TRTLLM_M31_DISABLE_DRAFT_KV_RESERVE", @@ -72,6 +74,7 @@ "TRTLLM_M31_DISABLE_PROXY_MS_REUSE", "TRTLLM_M31_DISABLE_ONEHOT_INDEX_FILL", "TRTLLM_DFLASH_DISABLE_HOST_CTX_ALLOC", + "TRTLLM_M31_DISABLE_GPU_TIMESTAMPS", ) @@ -107,6 +110,8 @@ def test_helper_sets_every_flag(monkeypatch): for env in KILL_SWITCHES: monkeypatch.delenv(env, raising=False) monkeypatch.setattr(py_executor, "_KV_GPU_PREFETCH_KILLED", False) + # CPU only: stand in for the native stamp op, which GPU timestamps need to arm. + monkeypatch.setattr(perf_metrics_manager, "_native_stamp_op", lambda: object()) for module, flag, armed in FLAGS: monkeypatch.setattr(module, flag, not armed) routing.arm_sm107_shared_opt_ins() From f0af3b6318024b35b1e08eaac06658d9f1995061 Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Mon, 26 Oct 2026 23:33:46 -0700 Subject: [PATCH 37/38] [None][fix] Drop the Triton stand-in stamp from the executor's GPU timestamps GpuTimestamp now writes its stamp only with the native op (trtllm::minimax_m31_globaltimer_stamp), looked up once when routing arms GPU timestamps; without the op the executor keeps timed CUDA events. The measurement-only Triton stand-in (TRTLLM_M31_GPU_TIMESTAMPS_TRITON_PROTO) is removed from runtime code. The arm logs one line, "[M31] executor timing: GPU timestamps" or "timed CUDA events". The GPU unit test stands in a test-local %globaltimer store when the native op is not built, so the timestamp's stream and timing logic stays covered on such images. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../_torch/pyexecutor/perf_metrics_manager.py | 34 +++++-------------- .../_torch/executor/test_gpu_timestamp.py | 26 ++++++++++++-- 2 files changed, 32 insertions(+), 28 deletions(-) diff --git a/tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py b/tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py index 03e04ceab460..f767b28c515a 100644 --- a/tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py +++ b/tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py @@ -22,10 +22,9 @@ # GPU timestamps for the executor's timing events (the per-request perf metrics, the iteration log's device step): # a model opt-in armed process-wide (arm_gpu_timestamps; the MiniMax-M3.1 sm_107 routing arms it). Kill switch -# TRTLLM_M31_DISABLE_GPU_TIMESTAMPS=1. TRTLLM_M31_GPU_TIMESTAMPS_TRITON_PROTO=1 is a measurement-only fallback that -# writes the stamp with a Triton kernel when the native stamp op is not built. +# TRTLLM_M31_DISABLE_GPU_TIMESTAMPS=1. Needs the native stamp op (trtllm::minimax_m31_globaltimer_stamp). _GPU_TIMESTAMPS = False -_TRITON_PROTO = os.environ.get("TRTLLM_M31_GPU_TIMESTAMPS_TRITON_PROTO", "0") == "1" +_STAMP_OP = None def _native_stamp_op(): @@ -36,17 +35,17 @@ def _native_stamp_op(): def arm_gpu_timestamps() -> bool: - """Time the executor with GpuTimestamp in this process, unless killed or no stamp op is available.""" - global _GPU_TIMESTAMPS + """Time the executor with GpuTimestamp in this process, unless killed or the native stamp op is missing.""" + global _GPU_TIMESTAMPS, _STAMP_OP if os.environ.get("TRTLLM_M31_DISABLE_GPU_TIMESTAMPS", "0") == "1": _GPU_TIMESTAMPS = False else: + _STAMP_OP = _native_stamp_op() # The stamp lands in pinned host memory with a non-blocking copy; without pinned memory (Confidential # Compute) that copy would block the host, so keep timed events there. - _GPU_TIMESTAMPS = (_native_stamp_op() is not None or _TRITON_PROTO) and prefer_pinned() + _GPU_TIMESTAMPS = _STAMP_OP is not None and prefer_pinned() logger.info( f"[M31] executor timing: {'GPU timestamps' if _GPU_TIMESTAMPS else 'timed CUDA events'}" - f"{' (Triton prototype stamp)' if _GPU_TIMESTAMPS and _native_stamp_op() is None else ''}" ) return _GPU_TIMESTAMPS @@ -72,8 +71,6 @@ class GpuTimestamp: stamp as a native op. """ - _triton_kernel = None - def __init__(self): self._follower: Optional[torch.cuda.Stream] = None self._stamp: Optional[torch.Tensor] = None @@ -95,22 +92,9 @@ def _new_stream(device: torch.device) -> torch.cuda.Stream: handle = CUASSERT(cudart.cudaStreamCreateWithFlags(cudart.cudaStreamNonBlocking))[0] return torch.cuda.ExternalStream(int(handle), device=torch.device("cuda", index)) - @classmethod - def _write_stamp(cls, out: torch.Tensor) -> None: - op = _native_stamp_op() - if op is not None: - op(out) - return - if cls._triton_kernel is None: - import triton - import triton.language as tl - - @triton.jit - def _globaltimer_kernel(out_ptr): - tl.store(out_ptr, tl.extra.cuda.globaltimer()) - - cls._triton_kernel = _globaltimer_kernel - cls._triton_kernel[(1,)](out) + @staticmethod + def _write_stamp(out: torch.Tensor) -> None: + (_STAMP_OP or _native_stamp_op())(out) def record(self, stream: Optional[torch.cuda.Stream] = None) -> None: if stream is None: diff --git a/tests/unittest/_torch/executor/test_gpu_timestamp.py b/tests/unittest/_torch/executor/test_gpu_timestamp.py index 75a143d43eaa..8e4b208c4b19 100644 --- a/tests/unittest/_torch/executor/test_gpu_timestamp.py +++ b/tests/unittest/_torch/executor/test_gpu_timestamp.py @@ -2,8 +2,9 @@ # SPDX-License-Identifier: Apache-2.0 """GpuTimestamp keeps torch.cuda.Event's timing semantics (perf_metrics_manager; armed by arm_gpu_timestamps). -The stamp comes from the native op (trtllm.minimax_m31_globaltimer_stamp) when it is built, otherwise from the -measurement-only Triton fallback, which these tests force when the op is missing. +The stamp comes from the native op (trtllm.minimax_m31_globaltimer_stamp). Where the op is not built, these tests +stand in a test-local Triton kernel that does the same store, so the timestamp's stream and timing logic is still +covered; production code has no such fallback. """ import pytest @@ -14,10 +15,29 @@ pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a GPU") +_STAND_IN = None + + +def _stand_in_stamp_op(): + """A test-only %globaltimer store with the native op's signature, for images without the op.""" + global _STAND_IN + if _STAND_IN is None: + import triton + import triton.language as tl + + @triton.jit + def _globaltimer_test_kernel(out_ptr): + tl.store(out_ptr, tl.extra.cuda.globaltimer()) + + _STAND_IN = lambda out: _globaltimer_test_kernel[(1,)](out) # noqa: E731 + return _STAND_IN + + @pytest.fixture(autouse=True) def _stamp_path(monkeypatch): + monkeypatch.setattr(pmm, "_STAMP_OP", None) if pmm._native_stamp_op() is None: - monkeypatch.setattr(pmm, "_TRITON_PROTO", True) + monkeypatch.setattr(pmm, "_native_stamp_op", _stand_in_stamp_op) # A fixed spin of tens of milliseconds on Rubin-class clocks; the tests compare against the span that actually ran From 8b1d98e976bf6e622def3bae5fd7c543acfd0d7d Mon Sep 17 00:00:00 2001 From: peihengh <259410613+peihu-nv@users.noreply.github.com> Date: Tue, 27 Oct 2026 04:55:22 -0700 Subject: [PATCH 38/38] [None][chore] Drop the unused separate-library loader from the TP2 router half-row pin The router half-row op is built into th_common, so the fallback that loaded a separate libm31_router_half_pin.so (never built) and its imports are removed. No behaviour change. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> --- .../modeling_v2/catalog/gemm/m31_router_half_pin.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/gemm/m31_router_half_pin.py b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/gemm/m31_router_half_pin.py index acfd7261feb4..5d55c232c0ab 100644 --- a/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/gemm/m31_router_half_pin.py +++ b/tensorrt_llm/_torch/_experimental/modeling_v2/catalog/gemm/m31_router_half_pin.py @@ -3,7 +3,6 @@ """Exact, qualified half-row TF32 router dispatch for the M3.1 TP2 IPC boundary.""" import os -from pathlib import Path import torch @@ -21,11 +20,6 @@ def _load(): global _LOADED if _LOADED: return hasattr(torch.ops.trtllm, "m31_router_half_pin") - import tensorrt_llm - - library = Path(tensorrt_llm.__file__).parent / "libs/libm31_router_half_pin.so" - if not hasattr(torch.ops.trtllm, "m31_router_half_pin") and library.is_file(): - torch.ops.load_library(str(library)) if hasattr(torch.ops.trtllm, "m31_router_half_pin"): @torch.library.register_fake("trtllm::m31_router_half_pin")

  • ~ohr4Swiy`3G#zd;K%m)4V?8*xxRD`T*JME`Q(@+2aw%9>;liqs!%Qu>axm zFF+sI+Z8mUoca@vza{ZydtUkzd*Dxa?RNwI9@!IYBys(dMT3SxAK=gai1r7_A5rT~ z${!i;_-~NkRDUUzrwj7K=&F?So8&FmKSj=OmcNYuWYWKR><_d8zw$|1LP&pSs+|9v zpXeW9|3dl?4ao2EKe)XrDIX|5@bUW($X{&N5Bj{Xe}MU`^nTOW{e9)xcgM;zPyHA#&-xVl=V=f7 zk2t@USLk2M_R}tWT4_I+>-{>cZ}>ba`}Z7=Nz3t6@*Dby6^5N2drB`XyJN;V zay=94kI!$0bDtS_##7BtB5(DA^_FZad+7c3zAgX@F-thFw`oK!K7FX~^ZU>y&v z?YZo7k|+B8CN0lxl)k-M|8277U+#(*BEG8Ptt!4cw;TGF$`eu;AFStHCh^nbF>lTg z@R>>RKgb7)_fq%Go zziuRYH1YVuzr8>_&mX_@$Lm&_Qx~v3ZpR0`@xvg_AF@Bh>)Xm+p%=1|n_fflL2o>= z7W?HzgHz}6dX$<^O;)Ac zKPdSF{(jk1)(tjXI#uc;=AW=bFWcX*l=QRx{W2Fl$=@%P@VWebiVxcHJLOON{d>Rv ze<6S0jyJ-ehkfG2Bb9$|#}gH7#~T&w_wW7wza6hs`R#b7g6(*sg6()?nBK>Q_#itt zj(=L2=lH9XJ`n>;m-hF|^4)(=6GS9|4p0Lll=iDE8gC;15e!ThXD zYyAa%hxa?kcqrSS0i;(}rYS$oU+6dBQ`V3CIzEW_|3vmZDg9-7KiX;6`E=j=0MUN4 zUY{oX&*=5#WvCxo1nbp0UOIu+Kad~s>2Xy^)$wV6eAA=fkst9@3P;%SRsVXlC;o~4 zAU-;KLduDcs`n?__;9?F(rH;0`h)#zdcUkY-bwFw9TV@Q^$K^q(_XJo@lF}91O8PY zC)O`8zOa{^cxPIzPw03j2;_-(rinhTcqdSXzW?Cy&d(VgsEYLk#XFyq(nIqX$2(Jc z{oy?EPI}+2ig)hQ@5gb+GwFq}f6XFXwEVGi-la#wt_3jEs0uDI5> z{`e}z^VxvH93lQfyfgRsCg!hU&ho?)r*)Td{E|0Gooc{(zyywOO1Q|Nc&PLbFn-`S ztS^KKhR?|B8UFYs)=y^9`hqS`>mQ!{W8$HOh3<67bNU3HEzkA(i&I}|)Ro^b-0L6U zpH*gkF{Q(%=Jf%JU)u0_;*oYbuU8mx8%BKBv%Z1(bFG1TAg)&tnTe!uk!ig&u!E9z+d!4)6n@~(I&uV=W{Cm!~%Pf&fX^%pMhT3L;E8YY`pGOo=bO2{)6^f=`brJe zFV`=4eFXLj)-PCqP!VyI@K4D1HL!f={H%ZEdI!fNmA!!c@Sm{W!tu*VDt<}vSKePl z<+;C+S=2wdK0)g(#r4wOqw+Mq-gu>dy#(tO@ZZW+yfZ}c&|)odCjpgwK`@>$7yC>zp0YQ2Nr&#mNpOYE3GzUf(iEVeGYoaFaqLvSOh zzla}t{i_K+xXgjU?<>vlg3Ex9jAu4*Jkwj=>#y;8hW2OHQ9RS@ztQ`f)2Pn&XAocY z*5~!-C|>AISMO87RHXhuo=AVgj%P->el1@qUg<4gV@TNccO*TW_@<)2_IK=fBm5i47uY+} zKG*vl?RcR-o~U3u9;skEUa4T)e}cc!iu&z%s`5YVc&LJH|4G4iJW|zX$14?V`(rA< z?T;zg_LmfF`%B8-i50ReV1K3+{*6(D&3Cx8f0L*7XJ|M&p7*n;bkN`D@5p#5$2VVT zfPW+7rLK4;$4g!DOpcfSO5&L`zpy?fh*78v0tr!<9=WBh5Z|x*Y@nUr}bc=@!2?k%u)HHi2uHCd=>t& z^ZrV%zs?gs?Ykr7_+yBlz@Lbce-eVZ1pNSiCmeQQ!Y}H?TiM?!Erl^fGzefk5zj0u z^TGuMUbwK(hS`5`#RD<_(O#XaJ&M0N_#po2vCr9`se^Ip_GjvgZJ7NTm;DZYUxj#_ zovz}qHjMT?@m;q6UGZDC$Nllp@CD+be*eY2e{m1mbNMe^zsvurA^Sbzi-)@6*^Hmx zpUHE@TdDp@f+a2)#OR-AKjyRe$Qt~GdL*&DLefmpc0=pKV2blf_J(AN17w zBRSsr${xa(aYsk-zJT@5=g}{hyrQ8}CK^w4GdDG#|TM@mW3xpG}=dzUla^OP)Q5^>_Dr9lfuC zb!j2pq|Kl?KpACAwqyYOk{_hS}dy@=nhiFEJ#HIa_@YjQemtEBR=&$;5Y zzeIV@`!#ps{hBLW_3{2z&-*BQcjP(o-cGzPf!=RG-}@r@%l%pD=?}|)ijOP*65kWx z_@g|p$nTeo?!LUDc~ATq_J-&CKP*8lU2 z&l}k9Oz#7v4>l3~kstDb_6MYaG0V$*>_6pr%5iBAU27IbVUM`ec|UqRboRYUyqj&U93C;9tUm55r3`gmix!bi5J_h;$))&4C1{tLq@NB7c8@dx{^ z0sdnJ+%tk9Z`q$LD^%&A-yWn-gyqiZ0`G4Pw5+1j4RlHSjQ#a?f?e^+_VdRlSDOWd zfV^I*3a>T`%h8~MX+5#H%$BdO6d^rPCeP0y3+TDp441RKq7p|N07uK^{!GB2C#`=M zw39ppJ~;2Q+BC{Yb@Ija#mjhn=mF*j?{5LVod1=v>*dcrBa5<+v3`!6pXGXfHb{A* z=Rd~3W&0z}c%4IDGk>S3PU7#W-t~WGCBJ+4 z>wV{g%rE(y`0pt3_pf_?tas>rj``t_e|{kTKYkwm@1*f`@qg!kGyemhi{gLatNGva zy#$P>Xa6er`<(M3>C6Yb|E-kPdk_!Jr|-4V_oD6Z9bmlx@+fzG!TQ;7#`jF?IrIhg zN7(wpH=lsNM_*t)*&`334?X&$mgcX!J=QN_q31WJSGw@2!g?k4W29*OrM`kcC(jQn zed4#*JnNbGp0`)uS5Z7oo<~;tDq`_&NC*H(0h(WUUn#FAYI$!7c|KXm`=B{r>StPS zLH>xqg4DtGET#q}wFn9id@ewZ^WczEfv_6%QECf5toL6YCq^ClF- zpGk8ck={2M@pb(^h8m==D>f=z>2tfKd(QobKB=xLDZzTTUB3GIvKaP%*>G~+qzUpo zMhf}|{I_|cQ$P6AnBO2cos`VFB9W{qFOe%4$PfJ)GfNUu-+&&}-`||zlt+J!X^o!q zlZJVDiPCqh->a9E#L-_HpK0@qf(A6jgM#yGps(7G4f_T5NMAneHQj&0XWhhNK947W z&x^qSTD)#dr(^t#8|FLvk86S=dHsYkJ}*f5U)Ur1uEg?%&QJJ$dX96R5cIB2OibYX zp2DvI{Y{K#`$nzjPFrV$Dc;FC80YztiBff*8}+{e|J7W`?+-*3wBCgAOyr~gsy>YW z_$AKy#u|KiWmz%%$Aa*Ky%NFx7rxZ|s1A-Fk9edkPx@#1Tdgk~`FJo-0nSv)!yFk=n-Yg#bBD5c9EYZcSx<>y@mm}v-k)v~?PSOr-6-@Kk{ zU_&uXCj5{3;7`<%f2{YP)LM&X(0tMHrN?fMWYeecfb4hRvukHw%r3u<-iI@TzmLdo z(L{Z;f2ExWMC-kP>HA#ild3%WkMlQiyZmXwCz5Z&ui$&2_f}6&A$v9LCj|ZP=@NmN zNh%Mlke}eoYQ6B~bqYp%*CL*6+jH1&PWczkcJlAqS?HY4;rs>hFp~Ltb`S6=q5~OF zL7>k@#<#$RCBEZq_!RjIg;C^`7tZ(W+G!Np=Zm=h(gJV&u_8NN*6+N(K-TZP-+=2k z@F7Wc!Qa2u4DP#znmF`Z zrHHmugJe%P07ks7Q^DXLyia}+O5(zJV?LpO{C+@sB988%@l4p`iTQ+d=|55WsiHuM zX-uDv+pHnqhlKpn-lTNSKRayu3z!eIe-hLCTJArOPrWmKfXnbc+u{eE`KkO9FdyNM z{dQJt_6tw#Mu=YOra?}Zo4a>Ets`!|N#|3vzY#my<$yI3Dr@NWw`?}YXM zFGV_F>3^14^BnvzKMX654zYj5@ida>uKF-P!@T|wQtj1RSM&KeHNO$hTs)8B+j>31 zKztMBAwtqBKDWYOKj8z~Groi$wP)j7VqVD${MjUrM>uIN=kK>Dd5Qhg1|4s*<%?l0 zbJtJ(hlA+UZ&)i<0JiZ_>2`ZY*K)BO^|6qyfqt=;EtC1d=AfTpHM#2-NwYa072-T!hA(KpO4l1E$Mw@ z#O{WA-&WFGumBa}N^$#CAL)Y5ggpP-WL`n#(>mTv^~rS9g8ta~*GPUax|R}ysQyD= zo6Dn253V2aF>^ZMtLu~YruHvDFOvTx3OZMG2gtrKx)#s?$okeH-nI|?L9m>kM86f> zU#@?TNcPNI4t#N?`2GCJp5|s72EIvi*)qnD@*gV-Bzu<8fXn=xzO+fM2q2yGskvZY zgG?tF{4;MJ>3>bHq&dHNjvMY-y3|`=Q*)y`-JG+G^rb3~`Wly!zIE!GyENs%NN-q5 z`ctRV`@|p#@d5z+d#b2-wOKl^lwi6}HH0gyrL6y)^s;%d|Kx@KCGdVS5^wr^2i70v zh3)hzq%U1c^*ia&dA7V$<*`_n8r^E}?Y5tIbJsq=+k2$vfL3?=L+Wa80YX0oI zqbe(g&;!Z@eoElH0OPCpo!UzyiRBIC)oL?7M_F9d9@EF2ALsWL(2iE7G2=UBKHq_~T)uLNT^{XYzg=Sz=}Z$Y-yq{1cKO1_ z8l*E#P~TN%DW4bPIl%QBjZ`1ggwy5zUMD@?NcOk1&p6&9-#3eAgdf=>WR8de)B=5r z_f>FwMVI&ZcU=EOc|K2-e+1_tsqF}Vf3}t4EquOMmv8EsGDY$i;)z^7D)*~VdGufJ z_j2-AR=V>;pU#+ptV(`Uyk<%%y{}>!9s=K$W{lUz<@uW`tXEIJlJp0^j}Y`me$?mW zPgrromao8PCF1u)Ke-?30_WAhPfdN(J@Ti?ESNzGO3R~!mA|MH)#>>o<0pM!Nc{gQ zfB(CHY-hEfna&qnAz>-Maeiq!DiirMAK*{7#M%MT1^X524Oky{*~1OAzlYkl_g_}v z`=~@e!(Q)(fA|I-sP7M#*xz%8J_LPm9*yx0tNB6kV>@%9ogNK7Wu+;Zz&+O^KObKvU0Hfq#L|-8%%^?u|s!g9p`MqF0K9H9c)|K4fE2zI1p9;&{fB*Q59gi7Yema#$|NQg^ zeH{8>{rHleD|kGvQ0Lj2dZtge>Gf=@m!7}}^tyuWTcQ{0Z!$v$y)T6MAMYOn{jQ+- z8afyKUb!SE`ke*p?5`31X3}h;OY?t7ki6IaV1-rZoF~Nik$pq5mgk>>S+@AP+FUJ%;nW@M8_MUyA7$ zQr}Od^Gcu{a8B#_!|$`Gtib$vHSdDs&+CVFJVLD}+Uv2rJ`f~S*`7szU@rz~zIN7v zPk8*kDs{CQKg92NKAi77NAHeWC!V^l1m8b^dc;ua?_u@z%mT0R#r7oDTUowF%J6+M z@ISA&(gy`OUP1c>>LY_Yng3YdwQT>nA?<6#yWTAdc)nkb@j-pgei->a;jp0h+fjYB z;D4;&F5>1<{tU|ZxW2!q-#<(rY_Ri}d-6YMO=Wz;gfI9P`x)v{T{!!5@B`pJyuUP+ z5-Dzi;VKWj59bxLuKrNG*1kA@KFFn9V8Hl(MFG!G^}Qb|PrRVp+iz9!`h%wLek)PH z^Of?0zxP|osqXUhy*tupPX6Mlc7D+R48DIp6&fTON*9UKF@kH762|wV7_Kb!!ii!R z{92c&pz*~Y=o1rwu^{Zgyq+vnJgSz~)Bo8SPmcF`&zG+>BiULa5Wf%j-|_v%i2wVI z;4eL16km@-==^W?81ymtc?Mwnyf4jPz)TOCuSlow&k?M@KS$@!ZT>?1nA&s4!!bW< z-SK%Izpx;Zl0FAN=6lAw->T&8%bMO5>U(XuQpMJ#mE2Pv7hD@nDBynE27K z^XCzNasOTXR_^dy9`wT^q3FZ&wGaFIJ+L*)GhQ^Fk{{3_oo?YYqtR&OdvxD}Mm`LcTAQz88$~hJIUAMfnLn%lgQrKS*AkHspE(>5nXZkEV(1 zr}xWE0G`Y0tvsPzW*onH`XV4y?4KvGD!dH_V=1GDIe7Ljy&Hte1-JM zH7W0StVnt3yDa6RFZe$e+vt)P{pC}de#-tN^b_#Id`?Zp3K{Dgk{@*o;_mc)=AuPp zKlA_NB9Ia!0Q{FVi z%`g3R3m4ch+9!Uwg7c?se!0vGFPrCr!7m2UAAg!(iX1pX{DT0mPRIA$XuqJgXAs|N z*}la!o_l=qLGMR%?3I@7J+AyY$p=0P%Jf42&N1F4RfHeQGoJss%LCMRkF{vbczfl8 zvCuo-i{fNY>-l-{^1(aa=aUb%eNneyp*_B@I`RPN_V|)KaKS7OG<=Tnb=eE}ek_eA zogbw6%;VYU8lOFo2N4R0&~*-Zp!5a#S9bZup7I`f0DZF$@1NxPtIk*N!}saDa6*1x z)J|95XC(N{x%7>@y()1bc|!KBoZr$P@3;K&CGN-<);}qa{@H6rJo6v+Bim!zeuVx} z@`e9y@wdw#=J6iq*hBjJ z+FIXn`IkNA_nHNcJtX~UtX#V1(`irs28*zztB>QM4_Y-o`v~qWmOr(A80%jrxitU7 z_|`Cg*4Xw@0mE2N<@w6~l*_){Yeqfx&V$x8=1*Oo&MUa>WzgG^uSel8C4BypSySWX zPueb=w*B>fe2?DE@BLQ3$Nt-I)zx_QDjaz_z0wQUPIAHEx2T)n0K4=h+gmPu>6Wkf{vg(~ zG2ip$e22XQeEN-ikA2v(-HO@#N#6%`zkg2hS6HyTMSbf@|K985$Nt)TuwEtQ8HQ%L^ZFc4SZRA=zgf=6CpJ|YJRHqIyIlL{#5Vf*GctW{>1Ce zUG`lie|;YGIcqBaGU8d^E2weJCmJ7%mOIt_!T9V)Jj3xnS^*cx?}^#|UYz#_X?bDC zPwxd4v0mI%9txBF!2UAE>u$(mzKBWCUdikbA82HI9qU6)b)gW=SI&=g%#V5l5?acC zxgJ)?Fv^geq2TN<#=a4@%ou-elhIQ z2Ql#bY@W)X5v^yEK5+T71m`*SKlUfHePgeO(tLTxn92B({HOJfC=4LJ3?b(i+&Q;@ z6L#@G*)I`C|DJFCRog?g)LyZte8LL4-rp$KpUK~2`+?V=4@qp)`m;Je1sY?(Fdqsm zA*yhzzqis7d&HCfnH}Vf;^Og_>%}R4fPYZ_t{uT_IF0pY+W)qW&L>CUm_i;9KfFTz zeKZONr1%H?NpZXJ}01e>-dMC_a zkSb(+Wc>_t{Y>8!Qt6WApErs6bU`Yi{fZsbnz}H55@wk9LuBVmd+~0(Pmiy_{k*^Q zXcun}lFuOD{~GT`w2SFc(@qLCC>{pYQ z>_2_qOB}=#qBrQfVn$@KHK z_t=>B*ni{ph<_rEK9=b&J^3NGH~1m9cWg|1)tVnkf4k35YyNyd$kd)OzhUFWrRdj~Gi9;KhRy}B{&U9f*Jm{}j4uYZ(&-u4!Z zY0s;V&e6X^7w8|QpSQjCG3{Nje+YoSZ~rL$yzQ+S)86^#JL~sQSdSm{_c`>tOsDpc z?&TNK?}KC7o8aS5wtyUc8w%_3J70Tx{Lb0l(Ti{I1NE;!bo=ImfBcfVzj{7=p#Bw) zY43vc{6PI{AJg8iw}0!$wD;@n-*?Bf_v`K7lVjRDzkc`4zxDq4r{wkf%Fm>*eEfOA z`L6l%eC_G^hjj0JC;c#cOndykN826@$@6FK`L6l%eC@q&|GLMtcR_ynK>gb|ro9X5 zhk*;|2TDJ$yngcH+xtNJ{jo9a{d)Q34?pDg9{G^l`>^Tx-4D5cPkqSk9sQ8od*MTF z@5dia(nOi+MD3>k4jzZS*+hnC!8jG9(&Geq2Dt?r2pXmZ|{7d zL9 znb>Yuus(|bmlYt2mUUTE%&rOsbikz(@KV8g&x*6|V!%sPFz4ui0WE8`YH{j)-+S*f zbH8W)+4)10b2Q0#^gj1@?|tuk?|1M0zWHWmR(6ata{W!tw?7xy^terbe{XHewEhuvsB+_q&`iL+w@0%mh0OS9Hsig>tC7r zmaCqxG~KT071zHWs&5O$v-Eo87Q|Ec6QOv@e(7^T%%p2PB|RKZ-M^d-rbF#>_X8L9 zFQxTuLOiv;P&}o+&&c&p#8c}F#Z&8x1v8=gxP9*Zkm7!6^OoxIwCNS&nWXw&L-B0n z@!9wq(>{AWAbkF3_MbMrVtw)8c&L4TJoS2crFd4X&&9Ly^$D)8I6f=XS3RDwO;yL| zRf^|z9Pje|+f|QuO|N*oZwqEa@w|@XJ*CIHJwFfkpUf|xlH*;|wf|apKd(apJlH1AfDyhC*M!t;;Hq8-tW-*MuU@~`daDw^;`J(drNEec-r)e@$3jr zh3adh$~3ewQ&3Nc+Z#A7v4{liKl$OVU_tMwn@{q zeP!!gE2*#8fBODdnfee%a2#728)Ko-ernw7#Vq zZ=dPEitC^2ze?kCDL5S(FaG*4F5{(i{)|o6xZb1b+JA$5{C!EeKX!^_?rJuV(uKJ^r+RZlUA-mWY0TZ%ot6 z9uL1L@s!&-D4xak>G&+ue{t8oO7&@a+4}C3`r;+^728Mk{e#)R*#4=!KP&cM+4<~h zNqyYD!uz+V5B*nueyNM0|H{^Pxum|D?OQ@TdA@$PeZNuDE5>tIaE|(qua_6@M?w3V z-FT7XtL*Wh=@sismehCM@vKx|)p#aK>Z=*g!tucEd&b7I;_;y4bE>4i7v8>cTVK`o z4VTncvwasTjThHGO|RI0r%URqIbIT&Un;~?(<|0@vZTJ3H=d>SRg5RzZ>)Iyxp-DR z{+fejI{vtQh2szP;rJ_=UtE1vkH5x}`il22+4+<1&$9DX+4V!OV}|_tV`wABf0DaP|hdzkc?wGiSf` z)sO$_rGNa#Ym?vo?jL&Q%(dVA)`Oq?@0Xwd`M>_v4dB>{?*6-;j91j?yLXuwcq=V z9|=|teQEbs*023Y)ZEk`q%?f#TW^o)hw@|gbC{9z?H>x_a+9o0eCYN-6C?I_-%|4S z-G??sx+Hb>%5OIYQ}6lKcyoNZ_skcs{o-GI<75B$#jpS6C$Ik>DO8J=nxt;6Pi*N( z`Cn|=cFO-R1j!f={@`oUxb4v(5kw;PEB~!6W<+YW4@qAY;Uk{Ct^n-F&9R1cvjgt5 z?iB1u5Q&O=H*aqErb&M`+9vky#@(Aet!}enwl>&LY1KvEB~6MpHpp)VY6l#NMAoAb z`-WT)(|febrspL= z9{Q2L&PSeV=#=(%b_KhN{S>)LV;+4|{pN259}wzDBo&C@J!bXOXg6<_e$(*OZFxZY zr1R#MhChi!?${~n2St@!k^D7Zsf(5qzCd@<#2q>$2|+ZEpL5P zO`nl;zn5*lSMO7~-IdA=_qp6&TQyxSXH-p>%l%c;n`@*eYNWT;NKe*C@2HWUs*&DZ zBRyRsy|+eszoeJ;sjt{$xq2@T*Qg~^BYmnydag$LbdB_x8tJn&(&uZWpRAF-P$PY@ zMtZ(R`f5pfcxG9vkuGmqml*}s`&@23t9s-uNuB;UIugm1T_vj5vRvYMc>Y)~Ne{PN zE*FxQ;q;Cd(DGDCnPHzUl(f9|+P6~Dj&PZBn+CNboGx#pQ?zT>DVO@ImMND$gUEXu zWxtdvtPzn%KJq%d$FsYS$i6_*iw|!|DV6wc_68*@$jcG{5yHvi@8oFLu4#V~QQkM?3e|H|X9d2zrWpr$Bb}x$xep zBt382qG* zQ`iXU9s+UFS8s1tQeIxlE2hglO?$VKTA#!|ddHINx%8*bp1T(4w$XdoJy&IabkAb^ zs8WX6m%VB0=5@1&tc$Q8>Swqu<$c}#UAvpg#7DRPDJjA{G(H$Vk$d7XJ)h8#lZxb` z$Y0WC^DFYjMwhlZ7nM5lGMox^gv$=46AZP9;yfOy(=Vkdb6nPIKbrR^+|E$!;@VEX9UA*jN6keL(>W&g zmDdfd3v!)9uZ!#!+`QH5*?KL*kL69}eDrwp=4UbXN9uiE2C{j0yt#G6v&~!ep3VLk z9>++T@f;clI<};&Wog@cq+*l>+qBI~H@pKmbjEz{q|F&kPLm&N|hnP0MS3Gz}hZtEo` zp6}*ZENx%uvFNogC10nme!=Hxc)W)4`fV#SZVzJ~Qjh(11oh$lj<&;(OTo?RQ-$X zKdoE)#EfabeL+r-;XL*h7?Y*@n;>+ohx_iiST2^g?JLZ$<>#W}xbTn3aM_5%4Yhew zwK3aSFN^M`aKDD<0c|&YE3-a^j$3WJ>C0kUx0IWA>DUaFfi`NLeqX2R<=mrm&XEPR z`0$U_>zH#wYnWa;)H?MUiU*xj>E6VSSMOMmX+n=*jFs>hFflIdr>N|WeD3($h)d!8 zGb&^Ky2ndrV^=J4vtC!xmH$YCeyOBq*Q>2_mU7tZ;WcW`vvs=_c}+@&u>X)xC70W$ zhmM(5>>u}kVDoNyM)n^NKZ>iT){DCI)g05s>m9NU%gK99hvmJfmVVum$)3JZk0b11 zg1cpXCEgyN#k~dZiS5TI zQl9;6G$mK1is{cqgGfI5Vn^Z&8EM|E(O7QQ(4N;m!wAsVnxj!iFO5bWNjgf+NcBb@ zk4VF5eqsEPxLZb{+An?t!n0~4Q0J8w)EiZfYc12xccJ+#H1hS>FE`&8wK}&3l+OQ{ zn`_mf?~54rI_G)6x66$%)T^)3J2ypTekdGi{){iTgXD_4vhYfAv%gZL!YhTzS8YbN z@8pD{8rq5JyE4UN?O$h=(07mg$aU7GN>_<0qs6@IQ(V!vmOD1h@uBYo)L4zKXLT^g zovtEuY=(|O@0i5#Y0}-XsYh1n@u{k}O33$?=fdaV^uB|>NA^;WVSkLi_;HHS{W6bJ zStF{iA$d8gEspCKwuY39(@3pxD!UKwOj^gO>8-+8eNJ`<-gZc~3A+2ltXuJw#vPAC z%>OmVdf7DsD~}oFx)RazM#m>wQEd&;S?5OA5bcdxV_(ZHf0e#YuH$QeRKAYy6Se9w z``jwKIkQIU>{DeHGo!0Y874+jhPyWV%Q9HZ!R5VIT32o~dAzxvs%^QR_PFb?{|WVY zw!H>i=Derc%A+x$_J>v;eQ&-n)^3T*?nj=wo2s<-x*bwsykf^B=P0VVc7GoJo0+}L z`HP;<+!b8k`Al$?L!x*Uq!v;&JU}_N&7i;<4U2h&ODesu3cO>NAj7~ZHQn~>T zU0hB`8R7Jt`ADzG=FuYmhST$*^j1mNxi8G<4}=sSZH_6-k#1e~*5fU@U&MT=dpBlb z8^)yT^)b0tCC9h9Rz`EG*^d_PH}8y?_pm)i>nNTTKer`Can;K{=9s55TyK^TEzid{(4~ZN458k!Z(OxjNT> zt9o)+@0y5ikur*p(zCbPrN6k6%a2^$FGkPDt~KA<5qFpb02JX;Orf3_rlLQ2rM z*m$H&Jfb!pvgIy5%EY6}UdFDYm9J&$9A9RS5}J3X-TM~R=D}LmMCEu$JfinYM7ckU`NOd+6YEg^ zGU=gvTMB(qd>>Hzx=?>1-RrBvQGF|KOCWDZmhY>Z^j4nA>Cu~{FRG+VUrSiR`Tf30 z$^FE>9Y|m45wBxjpKc%hhtjR)fcIh-x)%kp_jn@}`SAO{h^5ENTvYiuLF_!<6ye$C z?Gw*7S-kqUN?+W{C1mPhV$AT z<}XnGjz~B?MRuC(P#dLwz3=JEQor%ZvrX_l+Z;EZjpG9Oaa_PQL&4+Cxc6-ID|j<* zy>yHNq?>Uk>94ABrS~$3&!L#p7z*t-bgy9A7#=fIG^So@W9U`YC)^*k>~eDh=eZt+ z^RV67*cfB2-1uRpI2RweRp#z^nYAZ8{tEksx)ATp^%eLdald?Tt`G6-AKTa?w#uXv zjoUrsP5GM&`x55a_U>ESn%WvfIUVKe1atUSxg}5bnAOVn`RJ{3d*Ti7hxPm2Yw-TQ zCnZ~D_Sa9($-aowX}$3Oq&${5u1(2YKNdG~MS z=E-oM$VDUX9>R@v*6hh&);%@PV=vY&cig?`GOu&ZEv`AwMpIjJrE5-MO%1QXc2{j; z9a#|;JlJ;(_%i(kK zm-L){W6n3n{VVD^3LoP)YL}a*ZnO;MulKyFILCL&44>M{<#=;&=)4L0yvlrdj{mlu zX9ZiTo(Glq?_bN;^Qap6{rTUVN8JE#&ZBDZ{(M|(J`Zm2JWB6zllE`S*DvFI&Eu=w zv0iOW)%WW*`smwm9_8Q7x+&O%J*w^!^`bHNYv?{7`F!4u@XyL8`fh~p)X#Kd0W0uMZeFu+TqwAFK!BkGm)8mcqk!g_k!t3#V z2d=ev`}EK~G*~9QbiKErnBM`#^ji1e)P?iAYca+AM>fm*C58KZ@P0|>o@hg8oYxB@ z*Iw+s88SD^!%vSk$a^G(^fLF^+!U>v9=+-CX3p>TX^!sA&}$)ndS0%5g!-voBJF)X zQuof)(skdaedx_?a(0z6-zO2uC(Y(-wA%lX_-6S8llE?;UT!m5*KtscvG&p7hALw~ z`oK#s?c;E|>qD(mzk1o!83gO$ad)In`n7PsP3gF*3#H?Hujum;xu2()ALF9)IV>MJK)M|-k-mC~VT-iqxmaGliP+~4Zy28Du&ziWGe1}}&yOE#O5gPg)yGSG-`W`#EH?sWq*yZFyRe9SCd=r>nYCo(n5cq#nYC?g`1f;n!@cfP$8 z@z8HtsouA>$5`zVv_XV>?q!QdJdoX&&g+|GPF^i{RLVCD6$$+;QEBh{#B#YZ?KLt)(zKXvE5w;YX}Cp`Rfw6A!O%^^;`PiP~l8r!1?N zQMQ&=zb*Rdvg&2LY<|f0%Z^hWh2{P5`*ynKjJsBL6#8Y<*Cc|n@$k!tyE5b&#jEZ9 zs#n!Dkyl%MEANXjE=`Lz?l>--Xi7X@?41f5Bl_iiCvhXbFJpWw&lp&zHfD?m$Wfsd|PIWC4X1CvB%ihW4uZ}5xTL$z2qxg=nnYGdklSVSkL@7df+AR zfsHlrWn2R{#_LNyUaxb;*f=ZkzAfOT9pM`@#!DN|(1_lcF}__h2Ck-W^uSBr0~<5O z#*ATaWRic)@fE%L0LVLK%Glt$5 zy>T7G95Ea3m4w&4SM(aVF=KqYW{kwfl}~&t;Kq!xF=K?^D+_q9=Bc1T}uX?!-9YYpnJ zJ)3K-H-B2D-Qp$xCO&B=%UD{HVk!B@6p)IgKT()PwcK4Ur~a)EtuJ>t*UHt^e_E=4 zouiSk(`3tVSIcMuj>>c&y7k0Y!?_2AG z%`Q#<#&4eL-}*(vCQP?^e;g@?sbLjR%W$fmi-5!9vL1 z8k`K3_nKh7F8a^K{EfkEDE(&rr4z4x4f=ayo_$jwzjWf+^}%$=mLC-ewNLs{(j!sC zTYfYk>f7@9+K?@0-4Q=s4{YDoU+M5|(}`OC`(tLzL^`EH86c7Gm)ARTL3Gb8L0w;6 zB(}Xd$hyY_pkOxto58mHONSCbMiYtY5CA<{B{YO z&X@85)!c9p%*!)ABAPr?KO6-8lcIk@ba^H|Dcm8+C(qzh^89Jh@z*FNq)WoRMe$AIYmc7v#At&)|=x40)#I znUQDwzX@|eUhj+q%ks>>NnYg{{oY&uyNP)6e*6dcxWIsX(<;&epY7?S5zLXGhe|IlI}}^Uh81>@`wf+ugC!C#fcFNh^&hB@1+Sxg0_d0vt+5OH=HrVoqot<`e#@QKXPdR(W z**Rw~ID6XJ%g&y0_L{S2ogJ5No!9ovJ3Hy@lg>^%d%@WmXD>Q?#@TsiFF1SE*~`ve zbM~6E^|MW;e`JZ#`t{RtrhlBhzWsgqP-DaJGI*O6`Qq6`Ei7lxnA(t-CI@mu2xY<;!?%Q2Pzm z*5ffE_L{Txc+vFy8*RECpPHUZ`1u=y{=5DB@^vxJ*5j{1K2pCT8!WBQmM@+AU6lXL ze!Ap$Hu5jL#ZQ-y=(qa$8-n?_hST%Tw)yLWM4O*pAEca(@{@1(%M%*Mc4(Go{Cj;~;`6=MwtPbne4o$DSG_;#^G(6Bvn_vH z(EmQ4ZwlhFhba1gTaa|N&c_swzkf1IzIBw*7>YKzRqQ4 z)TZlvr|H2#YwLWb@ySeDd&%WbWvz|!7i52-{-OM3XQTX;AGPTyKlhBaQGWbW)<*dm zXWRVJAI?Vp_@^z8^nPcfy!S=*K`^Wzu7<)?n#+BU!Rm$Ttt{5Qhs zX|Z+wvE^M0X2?#Qw!Fqq+tcrCji0t>!P#2Arl)_iSiVXBt#JPIZ(AGrGtRCz`Q`Zd z9hC_ zNyh7{>#uod>-^aek^X&`O$J#09&%Cp-o+RFFk00>!V(WSXK9Pz9vPCbfKl5UT(mQSa;OCuf z=X0rlw@tV6wfOJsLi&4bx}D!8-Pzz1T{hj$=aTO1LjH#+zq1SZAEx|btN(Vs7ylly z_8R)f+4VU7x@~%Z_&Gc7(v#m8&Yyq3wUIyFW9=o!uk>2m&bQZs%tym~u+Q44Z{|Qa zJ@JIK?R#GUDF10H-`R*?N-l?M zeBpoI*~nkl(GkS9$vWFCd&)OS-^68fN9EL2vgkARMjhSuq1X8IuWz~6=iTd7d7YEI z`p~aRluH=qI+WUbNY6O?(4Cf-T}$Ds43$5p*W2WQa?JHNu@8|x@9a~0aLI$qIpf@B zeN@_3mbdPha$aK*^RFD<=JB9w7p}vfa}0F6AJ9d*Gj-WE5Ae`Wx<6P)`i;KcBN^(< ziXG1RfYEg(3qL+8W?gV=SQP6E5~n)7@JWv^iyjpo?VWMyvRyDb;*)cFqii1Pg4?K+ zAlPN}+e`Fb(Qk_v{Yf6OfqR{}ewRmMRK7#>#C)lrY0=;A=?RxNTcI4U=oj>-dTEX` z%AIodez}z4KiqIANOBphZ2m91d>`6UK+PYV|FxpJp&bGfacd5{49dtQDmlLF1Ci^M zfO45iOuMd-X*TpAGthD8TnxFf?7IF`D7U^L6n^OV)ch%Oc}ngk4#RQYJO0Y`Uq0<4_xL?Hl7F++NpR}r64mRx%L884|0_R#9N;>+p zR!zG^luc&PBl(Y%ssH@XSrX%4>(>W`FBo$9Q{;dA=WTwBU(K&m0_Sg+`22X8`cq%9 z`4f^yA6!4i(f3OJ!;(Km_4m8{7ff!OU+YIcjNdy%D3`zE*KPe6zgoXeNu2+kBFJi# zk?|9X|ANb(kUajwmReLkDEU8Crv1wHP0-$+iycAMlF&O25#2 zGGv1Q$EU5P_^&%&1NZ9Jr@j_OYcWUYuYNZ`{Crj_-##2)ctyURr!zwjO-zhWxY8e* z$_(w#P9}D=Co%&Q0|$q)Llcu;&54f>4~-@E4~&kEj6MFYME}8ok+Ht)(AeboMBl*R zKt}M^kG(6=W0cm%#*%G`?K|bUqvNA(i36kK1KEzgq4tCK9T>Z(E%8L({(=3&Lms|e zP(xSB^3;uId7^Lf0O#5+-aI)nmhBKPo~n^P{$$qku(UzjcS<3XLm%rKl}3By4yi+` zxNnCf?|-sS2nG7Q=%^PU_KlA3f1)otuy53l$Bwqd*nJ-LvA*p16GLNtPdwFUGfYY^ zO%C;C1|A>kJ2-T3pY+!7NG9Zl6w&S#AsMnG1EYQ8o~zm@`9Ikq|Ac$XG!9uNvjbVb z=ukGP!t}3c?__3lB&%NTAAfR8#zbjdo<{>&89>>Q@v*+ik+CPd3N${&GRK}g*q0f8 zdU8ZYpBAlc7FYtfvt7rMiNR!cVq{PvSWt(D1_t{|R2xN-ZIKK=)mN^Fr(8!+5&I4v z+}<}S4vGuT^R|SDJ4(dCkqJ4_`aZV3Z)7am+}TiM;;$5q$dUk zvI7a5S^o9%n*;-EO%HD0nb`MqcF3O$g!lZj^%aXUWqRd}4UQaq@IIp!|MAiXhn^bQ zKh&qqd~pAhg9Ds@(4>UskRW*fli9@hfy6!;W`haI@v(a+hX*o4!DGpfs_C~UJD%7# zBm=`9a*2t7vB!rJkG*}z_V)YRJMP=rE`z`D_x-xx^4byw#J zb@9uLObTDAe>;-xCWB^@hZi5~|MnobTb?#p(v-oWse~czKVjYsoy5oPwe-D)_F(zb z<4ymxi{_?f7s()a>^|GRg6>*!_l_WN^KpUo$`gNW{e^tmGtx2YK_MT?as$K5Z~T#_ zh8|wJ9S>#}FCxXszTi@Dw>HW>ij%#U{!XNZ;^B=q>127xLQu@)vUvU!+Uv%kX{pJB zHcBZTPsdk#xx*{OoAGbfiy**y!wwbo;&8kB{$l(5g$Q}?_6D5ipNYS8ap>VKQ^+1l z_wpCwQS`|h04`UkJjCgNVv*Oo;K7;e=e_uBFHWGI|EB#qG_d@QqmrU~e*d`RGijpv zSi3!Dyu8Ht?h;EcW-a^V<)3p=e2jrbEiD-muH7O9|-Q=k(6g~7V!KvD}jdc z?j1YiX(tdr{lLh*of8uSPbUtHPb9Lsx`yU-qa-FrK4Di?+bLfDV)}!{@qv?<$R|D^ zcRIf@vaRt>a}}@I>(dtR?>70{n|^{5dV->lp2jz%emZ(Q_LN8-I4|&&7ZKwz;OyzU~*AH+8>$(|fm>4Epa0FMr|C?%taI zY)q~TO4?a3{d}u4hHI9;>3b7FpykKBQw+J9eO-@#CwXGQ|1sCMIfMKP3rprYh5YnM zL2q$?*{Qmy15lUjJ3CfMov}_nB&e2y9z^^flUUeyo-B_rt5s*t_^gh9|H@P;ndotw zoX~3dE6yeK*EgJ=(hHCBKtKGQxTSo*8x(&0%3W8s^_nwDTafo}UFp@J!1DGC(nFo@ zU1V{ImT&M;^Y`=89*lKA@p19LoPK)+UEba+%YT0b{Z}f~`^OdZ{~*ql^XIM#`ZE>u zdGY5q>6e`AFB})q3V!~6g>ueS;D4@y|Npjv{?!WQw^ZN-HxIep!Ce1@jS;K;-EcD^>m-_c2fZJF*m+B{!LEj_%}OU%ee!M zlGkBR*EIo;M%Nk2%dNJBN8IUhyFqYgL6zHWoDP4`2iu&EaRoi$@vcivX)PY_>#f8m zUEOWOLr*%rgZylFy0$Cf`QPp7cX|3=H~5r)lc%R$C$T>3bd7@^r&zz{oQ~rc`aJPz zx89u~9{Ne*Gmbw+JoK}~x8%h!c`OkR{T%U0$Lsys`hfq?&l7*;OSb$A#6!PGe9V=< zLOk@##KX@u;-Oz5KIh6`Cm#A$;^8L_PP7+#%pJ7oU-((?bnwudh=-re#6#akJp4=$ z550wW_}NN4^fuz*XOej6yNHLMDdM4b5f4ARiHE+2c=(wn9(oV)@UxeA=!3*3p0nd~ zn0V-;#G_pq;-P1W4_x_E#6v$seBALl;-OCypK<&#;-SwF4?m9+4}F&Sy{`N@;-SwI z4?j;35B(JJ!>;@V;-N1RKjZi_#6!;$k9M6U9{LjTXxBO7p)V7UcAY04`bFaN&i@tS zpIU?dm2T`X1uZt~Bw`dx%H7 zdWnbLPyA8m{~+NywlGUk9J)k z9{NS%vF@)B5B(DH@N<=T=$Dyyey%a^baUZL_OY0+;AhO8{9t~AUhj19@UxM4=yBrV zXA|+zo0)fhZe!l*t;EC6HsYZtiHD!<#6#~O9)9j39(sy-=Vur5PTxa(!s%(^q4yAv zcJ&eueJ}BWE5Dz3=!3-f{Jz^C5)VB?Jl4T1@z4(uzvBGK5fA++@o3jH@z9SEk9lK; zc<9H8N4sW;hdxg{*1;3RLqA14*8K(Ip`Rum?OG%r`WfQUt~~M3&k~PzEfEiWnRu*& z=ZS}Yk$ALgg?Q+fh)26tiHClfc(iMcc<5J%N4wUEhaSAaoYoiSLyW_i(=i`HuO}XU zHWCj#PCWcAp z=*NghyJm=oK1V#-HBUVB6U1ZPJV`wCQ^dp11>&KfCLVq+5)b_>@$hqrco9_>0!JoH84(XKPZL(dbBcAX_2 z`V#SI*E!;$pC=yU@B;DBFA@(wSBQsxiFo+AN<8$-#4ozzagBKB>%?PUbd`AMc#{wF zVPT$hZ~kFEgkJA-@P&Djc<9Zkc<6hHhoAk#L(dQoKeNO`pCTT99wHukj(GTa zlz8aV#KX^H#6v$$Jp7y`9{L>d@N=Gc=qHGWpC^fjeu{Ybxj;PhGsI(_$rBI#Eb(a9 z67kT_5s%|&nRw{uiI4w*d;gbs=qtoy-M>UU^i|>!x68ysUn3rIyFxtlAQ3+QMI2&I z$NUDpo_NHek$C8F;xW%O5f8nYc(iL9@z4{*x48JX5Dz`cyxV`aGw<{c;xW#55f42@ zJpAk;9(p(N@N*CG(0iG8e(q)7>HWmR&q3m$4-*eRM~R1?As&8aiHDwJ-uZcyd8bbk z4?m9)4}FGs_<5Xo=(EJb&pG0u&l8XHniIrBKS}(WTOUsm5B)Upn8y~0hkk~59H)8W zp`Rrl$LSLB(9aQ{bjQ~+@z5_2k9q7O@z7U@N55Pm9{MWr=$FgHLtkg!jjOB7J3Y87 zFc0kCaNQ~9bj*j)>xqY-jl@Hb6W`_H*+e|_1o7y{7UH3|5|4gtBOZE^c+A7?#6#~O z9_`viJoFUtXjd2U(9^`DT|LA@?mA5|4IG5fA+k@n~0$c<4uoN4utphdx6*#^G_|q0bSIeb+qk&`%JLcAX?1`YGbk zt_9+upC%sdS|lENo_Jh`JWD+ECE_s;pCcanGVz#)&l3;*BJqgB3h~e{5s!V>D)G=S z6AwSvh=+cK_*3q9StlNP{oUdHA^dD~I_5*@apKXBO~gZQCLa5)ZNx)Q5RZ1X5D&eT zc(kjHc<3F(qg}g*hn^xH>wXvU(7TC;pL>Xho+ciC_7D$UzXx3(SdZZ6Ao0+LiHDz~ z#6!;z4?nZSL!TlZejXwo`cdNH=QQ!qj}Z?)XNZS>oOt*-OFZ;B;^F5!@z75ZpK$Br z0`bsK6OVQ+5)b_h@qsHpPdxOq#IIw2NIdjq;#XYx=ZS}Yfq3|Nk$C7U#A6<~L_G9W z;^Sv!1168l#6!PAJl64b;-Oz99{nD)m}&~2e}NuzI*u>&dp+^cn}|monu&+Ljd;xS z3F4u*5D!0FiHF`se2a@`l6dI5n0MC=Qp`KOi+J>7H}TN-5RZ92O+54-;?b^N;-T*) z9_{KU9{MQru3Z`Cot`Bg^UM_S&<_z0KXb%GKT16OoF*Rnaps+$v&=hvj(GSvPdxM! z#KX^%#6v$tJp5cB9{L&Pou7H;oqm>h__;(p^mD|+&t>AFpC=xEULYR&3h|gHFA)!Y zm3Yhpmx+hIMm&zAE5t)zCqC(p_p8K1kA0V&_b|UH1Y6rk$C85h=-qf;-Q}-9)2zp5B)sx@bd!k&@U1XKUau{eu;Sa zxk^0rHR5O7@pXlG=R9=tg`zu`EFIUVyG^m^hm&Ywo&p*Inad9s;!=n3Kz zZol0^JoHxL(XKY)p(lwCT>0(9L+>CS?b<~=^e*C8T>0I^L*GL@{7e%My@z=8OE2-z z_Y$9X{nbxA^kL$09c+|%=o#WM&a=crpCTUP{1EZbr-?@#ju8)ihIp)N$BBnNOFaCX zBOdxZ@#kGUPY@4%fq3|Nnt14o#G@b25Dz_1Jl3_d#6w>q9_>0uJoIJa(XR8vLth~t z?Ycxf^i|?9uU;k|`Wo@@^9u3M*NKOpSBZyiel$a-MC|wQK7Q;i;qzbU^-c$$#(YRT z^k(7_hi$||PY{ndv=9%ym3YLVjdUYe(oV2dYX9n*+V?^ zUgF{BUgDt-5PTQCLZhNG2)>gCqCiU zqgmpi&k>Jy%@Ys(1o43@|0MCyPZ5uH>F-JE1N#l=i_E+7&oJ-wJn``JEb-8nh)2Ji zBOdxP@$vs^ufLrq9{NS%v2Lyq5B(DH80V|RL%&Qs#`zlY(617YI0UWX_(P959rGL3 z&3fXYHxds&oa667kT_ z5f49?iHClkc=Y21;-Oz8K8^cfh=;yPe9rNgiHE*MJmPkRcmS{)FD>bnq)~or)6=y@`1E*-Sk2ZN$UR1o6;Yh=-r8#6wRKk9De@c<8%` z&$;-dh=<-qJlfSwJoG)pqg`p@q4yAvcJ&euy`Om87codY^ikq7u75McL(dYAc1;lv z{SfhJSB`k-M~O$friq6>!@Rpc`8e}VpCcak`OFg!{RHu7*Gb}`pCTUZS|A?!Y2wkY zMdG37iO;+7d6szSOT@#^bHqbmCLVsCCm#9*;^&<|7m0^{iFoYuSBZzdMtt6l&nv`3 zUnd^zx=K9s!2NC%<~Ot}=5*}mq1O|Sb~O?Yy@_}ncg@5@PY{oB*g`z?R^rhv{GJy4 zhn^%J?P{m;p?46EcI_e_dKdBd9AY=|(DA!nh(Fem9^#?*Qu%1tUgDwm6OVQc5)XZt zc(iMjc<5Q;XWjUpA|Co7;<0bZ5fA++@m;R`Y2u+DCmwN_B_8@5@i-5fCm#9<;^F5> z;-Q}+{(|#=fq3YriO0Tck$C85h)?7FW#XZqBOY;BCLa2E;xSGx5D)z#@i@;~As+fA z;t{u1;-OzA9&uYE9{N?{(JuTR7S9i{x7mFljz{dn>zxiBdL!}hGfq78CgR~|Gx5;3 z5s!XP5D&eD_>^1sTZxC>PCVk%K|J(b#3K$V;-PmDk2rJ_4}A~uh(nrq=)J_l&%MM$ z?&KfCLVq+5)b_h@$fTGJoF{vaUOJzch0`br< z5|4JR5D)zl@o3j7@z5_5k9MsQ4}G0@?7OZK4?U(o93l_QhZ(p3tam!*L+Fjfqg`?0 zp*Inab~O_ZeH-y;SAuxxt;DC?cy1#edXjkf*-kw44&veGF5;o5h)+0wx`>Cqhj^S9 zq=|>#OFa5zFY(a(iATEziHAN+JlZu%JoF6lXjhha=!b~MILr|beVTa8GslRBK0`d( zb)0zUv&5rabHqcRCm!uOK|J(R#N)hRfq3YriO0ILNIdj2#ADsb6A%3y@rc7R@zBo` zkA3n5;-Oz89)7M65B(DHz3zBfB_8?};^F5y@zAdlkA4i^Zs$GBZ_r~-$9#x=Y(4SN z8;M7|;>1I5A|CB(CLVeV@n}~o@zC3d$37=XJoI+r;b#Z&(036JKU2g*?;;-i*lyyX z?;(EPo$sWHhrXA1#G#*f=!3)~4#UJlA0-}f$Pf=bOFZH*MLhJQ#KX^N;-McS9)8Xc z5B)gt@N<@U=ySxw&w1jZpCBInc#?SNr-)y1?-wi(4}Fn%?32$B4?Ryj;&zsJ=u5<7 z9ymul^kw4v-F$VPc<2|2$2zq_JoHP%!_QUXp5ba4uix)A0|HU_N$}BL!TlZaX3Ug z^c?Ys!%^a)PZN(g93vk34DpD=apIxR5|22{5f6Qy_!W0O?F8}APciSVM=db#^wY#6 zZi~c2KSMm`u{`n6&k{fH=JO@up)V7Ub@M#&&@T{=ez{0I^cCXKFPDgiewlduuHPE* z(611`;(jl0op|V1iHD!~AzaLd&|~fg5V4K#KX^R#6wRI4?kOohu%s& z{A?p0dXo5a&i{7ep{Iz4pIyX5?mBD5f4Al5f6Qt_>0c}^Tb16As&8SA|CoG@$mC9@zB?Zho4u7 zhrUkyiu3;}@z8_q;r$-gwV2Z}UqP=Y{){WXk$C9M#3K&dh=-ma9&uhWlQKhrUca{5(%Q^b5qp&x^!EUm+fT zULqd)D)Vl?dYO5ruM-bHuM!VE*b$zW;AhO~=r8E?#KX@<;-SZh$M@kk5f42#KX@+#6!;!4?m9*4}F?=e81c=;-SwH4?pLKhdxg{{5(NC^pnKH z&r`%hUmzaek93-N=y~Gd=UL*RFA)zv&k+xOnRxhlo_OdNh;MP9Tf9g-^i|^F=Vjud zuMrPFuMiJ?op|_pm3ZjEec|~HpEr*=9rGLXIPvhaiFoME#KX^R#6wRI4?kOohu%s& zKHt|yJoFCY;pZ;mp{Iz4pIyX5?)mwe|V92=vRnG9M*}4ewBE{ zA!xVr66PQ1F{fkyj5yR2551B2OD;Zf;-M#qho3FPLvJM>ezp-0JxM(LY$qOi2k|j? zp0bO0=qcjSk6pw=?T}B9oNng>z88XdUzz{uE0-@wGgz|%qB{wD_q_6$5Zwtx8h4EoSi&@$z??9fy;=o2J6 z)HgDgmGptpz9;S<=xE=ubLV~gcO2Nc|G@qo58S^mxo>d$14G+)>>oNXG-Qetek2?8 zeE89aAAavc6BFYT9~{blU|?i&Xz<~Q@q_RAV0I!sFp-@MwEDimk%7^n{n@^O$;qLK zY~SRQ`}Yq`P6qexlYgoe5xhq{>-ylH5A^g|tWe2&q*zle@{c|B$WT^l9U3_3+B~vv zLLwrnZFg|**m!p6-bdc|p|?+F2lhW>l@6DPuT847d$yOIzBM?VoNr?Y6{AZ_YIo3*mz1TZJ4Yn!SYQTSwG$nk53eV zo}sbH@rg$VG7pXIA0Hg*85tcK3|a<*dndCK*@1oX`t(8bIx#*tkR7m9?6EQ0BdzK( z;thvM!a6D=v+v-*$XH+2_|P{nIFOO!q3_{=Cx)zB4@+~i9S^lXdf&tE8ytCR`}UpN zcXaGXws*+O_G+A_@8}rW*{-?5FE2db3mOd=ihYwq*}fq?Xl2}vAMDHMk=oZGlY)us zz56G#+jkabmM6@#p;0wa8XVHT>3e)=tk0zsx^42~BcJ%hsEe0$;|HxhDb3&4cb^Q# zr!zzQO(CwZeFKw2eS?#uk}xv%xF6xZ?7@M)1LG412eO6YwR>f}h_w3$dmifB(YJ4W zN~V^BT7KX5oqgNQe+68;>BBT^C(jr0@88J0T~{W5|Bu@V{jN>nL^oX*#0TXs)ixIt z`$x*zqi!8SavXoSS6yFX@=Q4!FBi<}6a*{NR%w(M^{Czf&F=M4$*lF?=4`wyxOx*1 z+$-wG>px4On%}mz2rs+*!*1P23i84JqWqygaSS`I+DI`E|B?T8`9s(3O8J>*TgxvO zzr`Q3Ekc|$ep}>;{LtSfZdy+z(R-wv|DI(2WcH}b&+$Y41Z6<|DEApjFX#WV^MA&H f>VJbgkstp5y;RI~YnJ}!ZN*bAuxh=wO#c55?llN; literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k7ac2accb0dfe6f9715bd7db142efee09.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k7ac2accb0dfe6f9715bd7db142efee09.o new file mode 100644 index 0000000000000000000000000000000000000000..79de19b58a60953fe6c84c61e912e483806abbc2 GIT binary patch literal 63448 zcmeIb4}4VBwJ*MB=FDUgLJ~q$0)Y$=X+Q`Ge-tg^Bm@!_Bw}i9YdaZ8Fp>OoGU%kn z>kzpW6*W+AtF5ikOM3~$UfOzpSNrikhbONWt+j~0_qg}@ZX@M-QBi3gZG%PU_g#DM zHFM4m0n}b!pC31$;heSBcdxzn+Iz44=bV|`zHG&Ex636sxx_z;G{syY4jxb62jy*0 zWCICo-&7&C9`Y3L{4G9*J;k&B9oN_K`48L!T2LJD{CEDmf9ul&e;gxD{Nao4f#a(@ z1Mhpp*5e-6tlxXY&i*d3b!v8e^&w9I<$GXIhy~A(eAUu{KYXzY`GP0jzjg19geUYr z{;@dt)Dg(sF3(FLls~ZL-UB;dxc9(YKiYC&%j5SRc+?#{0Udq#Y@PY5$A|7N+ch*M z_s2s=@A>i28%KXRbob-G8v4^^WLLF7d-yp*MEDFm&UO ze>F6K@@|~*IxwFfc;oeZ4&05;8?f)q^*P`3Q5#z(CtK4(yzB z?}3@n^ZcC0ho1Wt%9``S&~w@E3}wCh7BCkNJO}yCx9>eL0DJDP`5OoBerNN6yJqeh+PUlTp@Ff`)AroZ8!O)#+PUJLq0E_!2cAQnZrpO; zfxF)R(a@V0?1Jotp}XJt<NO(1`W#pA#GvP#og_=Vn_T#ZrQ)UExY0)-8}yvJ^AKde)qN%L zPIBQsckn5fi)4y_$nC$9@(#Y8T(}4PX)<3r&uD#%`zl@EzxC$>Z`|Q|>12m%;Jq8% zvcC?8Jm`-W`mcB3vVZ^jw*P~ESbEYm?lK{68unN_8&-=G{>qILFSA5N=4~0{{o%s= zl0@y-cbpXCW_a(K_zZCV%D&Q7ay(zbX@bUSBxumWv~C+ zWU}53e6AH=%{pF#&fh`jx!_Yi<~IQQJoJX3_gRG+p^;}TncbzVF_ zKfZN!%f(9unl~0bxexZ#;d2*05z!wJwU1kKaKFzbp6YZJKPe9OpWjwF`N_z}$-}?O z6i+>Oxp?YVUlLC>d|8}m*i&>e7<|TRa21_Q%oOhzXNsG`@ZEcebD5~?zRQD8G4B%S zBtJCc`el6l<>ILwJ;84dA3Sg&}=pBZ2M zZ^>tdf`2vkyI=hLj_W$(lg`dwXD!_OMfXV;_8Rwu6fdY-#zXOR_`eRh3N_1SM_h^ynF z3DU@$dX2col!u@bVAc{+&l~-*T;Z>YSH*wpC<3|D8X%{8u(#5j>IkQqQ*L z<-rqvy8h5V&|IZ>NA`m*6DOLNuqC-+}T%>-ioXjz9aLY?>1i*JmDc7 zGnNlz&~*y9DMh{ghDa!dM?KPZLR@P`-W9GPMq*f7AM5U-o>QH2i?EH_;6(| znR&#c@>AY?;D$Viw^xe+!czWx`~iDDug*84w_upcro0(*#K8J#!4ro(@onEg zdzZS;85XsNEcamYz?ee!z?i_`fyxzcSz_K4Nr5Ip848&rA7uEMkG=xhJG}S7;fbQx*h$TB9S@mfFYp4pq0-FR*A#7uY5Fm$56>9)Rc zMve3f`GovIektt*4*s&^V5jmQFzmll_hg5zma^ap^08CbWU|kp>&6+iBg=G8IlD4@ zr#Q>R7#&fjyI7Z*9zPAUt{hl{F*mcfSo%P&E3lrR^?;i9mthXC9vBWS#k^iMaI&Lv z;Qd>FA=j2Xx6w7Z)b&#GPo9A{E(l_Lx@NtGzH%s63>-esHE_t|pY?jNd*F>p@Fnzq zFgvsV~)8&@iJk1);Ylwi(OgwJdzt6+VeS4pE)l0)Mc259)*s~Om)9;Cho5e zUYzxX;KgI+1iLSriFHC|?bjPJE5H64$;Jjx%!aN<_RN0k(OIH?nip${#I(2eTp;zG zz#QRO{R=!s`Thteh4L&s&_|iO^joD5cH?D<^;IWN!VB<4mgF}=b9Qa0zo-eTAtM|=CJZeH=bsTaep z^59c`ueh4l5szI0I+&){d>(Y2jlL9g1L_IvTIeZ;O&fu!Sb+L>I{ABmowpeLxXmxs zPQSeNMd$;M{j!kS&=(w^T;DywmG(L9oC-aQR>01FCx6c}Q9r*P{M}A|<#JKK;Ck>APW~S7FKh&Vzms2C zBkC`01%J@V-vj<7QSgsB`IWVz{-Pf2LB?bb1hEzwsu3q%^Ayhd-O@1w@668{cCXDh{xV2xOK za;Z3R@;AY!9`O{ee-!KZBj2ta7{q$_5Z1!KEnYG3$9H`L@4w|8IE=Ob@&9EF4FB3Q z@H*CyNB*aK;7G<51OGlo(0=&D8`G~Gcx^oH|IRh=(#hnv5wB)VqOxE&l{L6jEc|om z{yB7G-!p5_QzZ2UD#gOrJmY7TV%=Fh7W*Ew>Htp7s_WMyD6${{VjQ;ra_)zXID&=VP|zt7qv8)_)TCv(bw^ z(pllWXK5#w&-Gh}d>_U?pM=jpINLfKZ8iTx`rx}Cv7ExQET{h?>c?^78QbJ++QB|n zvSG`IX=k&aJcRtCjY%E@)SsuEWu2kytn^WgIh%1}+PnKK^F4N!I8*CZL)}LiQie(?g=hq}V$itFxWLn=8 zX>V%n5_J%jkl3B+5^h98x7#f|V?8dn$0e*W=(aj}as=X;xLrNhsbqm${1c7?utgD;WfD9Yj>NX9rd)eREmrW^G4XXE-X#qD`@AnW|c7?S^P;xT`Cq$!Jr1S4Sig4Yjtmg`ypi zwauZXSZ7nDrLCzw8g6atXl!b2X}=+K^E`Rm)z;G9(iV<|+UBS4Dnjiyuj^b83U#%G zD#|Ykg`%DFv?JT*SA^Eh58cqz*0y~9if~(JMD>sJ=O}BHZ1vA$sk~tFD|^fm9wUM?F=&J_Xh2Y7R%58b3W1i8OUJMQ%oo&`Ob} z)=x!mLp{Quii$Kgt=VuxXk8@S)}&g;WQ5Bc91OFy;nvn0BH_+vIp{hg9np?xcPFxw ztfRdv8g7r4e+s(q+HhxEQ)KO@qM6Y&vMy#@N~R8-vH|UkPeB`Ub)0=xaz8agnA=uX zXtV~_lZrhSbG|OMj!3OVPIvR!GUT&m2o}Jh|Cg49BfEY{DAdv(#S$~z8Y&NgTDze! zjMZsHc`OuNdqY!u`NGf*8^V#sNVo-c#atI!+tC_Y(?l!X#wN^v9c!B!Ls)8uB2Cc^ z*b+2`)^vv?U)YsboSpIwFa#>YQRqQI8`|2t%)&lfI{n4%I2@st7Pcni?3Yge@9({M zV!5WRsjaMI-MX%(Xt@YsV7@OSGk>YWumaX|7x*5W`&vsrG9xU1o3B^ zN;J(rv#tX}<+Wnd#{o|q=fFVehbedThw zt%yzf|J>yr)a99dfY>xw06x7AbK{1yP5(O^UD^@4qQu@vV|9mstqlzje)N2u4q$R zplw4}G|PgDfq8*BVH{Eg)KOD7 zfTGD(I(NhK3&Ym+;v3*wbNvF(^~3_YALke7Ix^8ck37033?(Z+eSbE`f7Ut+eG&OC zvQ&BJkwrpG62S~{Y^ty{Jy9fl;+V_LbfE}3=m{d>wad>FK?i-VNLY6HIU?ww3xpqZ zy1en?q(>a9NR^*2jydRY;wb2GDR~}w&_?HqR~`5qvEMyJVOLQr}O(n!ey6xMI3Zmj@aOkdqhImyV3Dz zJT;t;4+4J!2K4SFh{IRJg>`Yz>O?0>N+a-}t4PE(Ezgue_i0*2J<$D{oF31={`;OYx+S=2jE|_w_nrcn%<#lj5aCXt!bKb2>+y} zJJB{oCp1mdEYVcoESFyda_qDv5_xvIO!Vd3X-fpgJL%mH+9!f|f|1tm6Z;Li0S|5M z@?!=a$4lgPc^;nV*y(-;ZHc_eN>0_wa#4d}=iM-1&)^M-dBPrqr zeaxU6f=>K?gAQVR+x6`>=s=ZIPA?YO@nWfy?lfq;`y=tnUrA9{t!PvM^-qd~rdb|X zrsT?>Nzn%yLsHIdgcr+Y7?J&(=uXfiPiTFfMTNpDvcw=_ij|iw`7}?{$C^Tx17BF1 zVP*LRx-#Mm?l08*^?1zHhR+RzMmr$U^#gr^hAT6V$iyGsQeSS{}Kr8Q&SjKaNQogG3jo_+c#JAp*`@_TO?mYd+f^<55+Qk zV<1iXP*3E4DCSfBOVfSuhZFC$oOnz22hRVCE?qobZp;xs-H7=%7K@I%%uUpszFN27_)k=uU%<8FbvBHyd=HK|h$L z&Gt3vnhpLjG#)Yj#?bit=X8I7zyID<)2+LU?hnrVkZ;I#=D$rUqyJ=oxW79!)w*T> zGW1QpmGQ9RXfFDfRg?frj-MA+33m_n6$Hsu`f3WPaVQ~P0N)$)p{aF3|D*ouXZjbU z&xh{w3HR527(o5+v1zJ5?$Gp`Zb@6aH9gIv=qEM3#x0!w%^g9XzP;W(LwiB~XKt^W zOqwkysrXI)4@esSAwGPp@qe=#|5gIJv_C(de^!ap_vck|{*BZ4qWKi_&qFaQSq#!w zfQ32Y-`0p_Q9Md6!hDMQZHsxWIM0XZe_LY{RXlqha`e9~5#c2~=EpJA|F=fG=h2KW z=VOw`gpaa!R&~np0)5`p{7U|dOgazKPn_qIhhp9&orhRt5Ar`0J9iS%GJYNgzIURR z=3B-)+WQX}-yd^)()vWafda@sSf7+ptm1_F?YgBlDc+I{xgObdOHDGs^dVZWoRf?n z)$(bC&rS9-jr9xsO9NJ~SLUMrg)1izP4f-vOa4yz2jx-w(D-M6nEvjIU9Q@L=4;d6 z|AYK>ITjo?e`QAXgpB?Se?8lM0i;Nu!w2=Key*!8 zl7X(I`v6{zkHyJ=;ySL7Iz?Y#(nj-i~|n zC)z`{pM%a!iViJ5MB_cHQ2Il!U$#fQUb0{M8|&HLp4mc2Jn8S=o;mUiM`}1c7xsCj zeG-58eutbt_f2>^ZPf-4Ts5Wy}G2}9y z>__|j6#9tP>)mIT!H=}w8HX$UTeD#}o9vPPKTh?rk6)sv`|v##)KBT(8p#-s;{=fm zD!Mn8;}{>O9p9+`*2wtrc744)1&;BA`ci*|Kd`?5zArY$O8G_Sb=2@yrHcrTl~Wqh`Zwe+Bg6^=hoI z=2Cg*$?|AE03P#sS|8S5y)js~LuSB_b3k8rk{4jI{dV0_n%*D5|KzXXWby;vuaUjf z-;$l|?|`Bkbo<8g#>#k;TAz;{OYO()3;QP02Bfn}`48*)0{eK7^Le!m9}{C~L&N=V zYs6oeNBVU8_Qe7LvDcl_-`g`)sPT&agZ8*HR>tu_kEdz8p_25sV^d0lq)+z0x!7OL zC?$uny}dp8j`sg{tfVwyllR5)@!C$+)BOqMJ+VPd zksY!xAnogm<>c7g3-TEOs(_Y*F3BT0Z>nD$_J}9CL@8!0Q9c3m>t6B9LQ$6LpNMyT zv4Vn>KCExh{spvwl;cPC&&3`(lSwA;koyCozwWe;2h^X&gPfoA zcyQQ5{yAg(EAcz+vC{EI*<+>RhqOn{Z)tlh+5IK0?UDT#xHOd$Ms}P4P^hS7ZLWvS=UWr}o2ot0ps;-v41e_H6f3 zNRd8=57t{W|HWzk)BE3Dx0w6CWyb!`xxVduX6sP{ll`$mqEn) zJM7<O>{d(!pW7P}O~#ZFI#@6+@{ zu}N~j;n)6#Kh%CKfc@W#=%2AbRUF1}`}W1A5gkm^B}B*5^bDdCOg|JWkop6u_8c$i zX(>8i(j_T+oTO)Hx-WKFRiCZ=ZCIa!Y~cQ~EjAuc_w95(JrAj(97{BRt->MKUU7e9+^6TV#2q&mkXEDbKe^j`NP14gI$E`!(9%@8~bi^AB48 zXQum)YF~B0MCU_ty}j6$6~u?uJ0>E75vr@q)lPP!+{p}#M- zXtr$Ml)g*K>~wGJ^UnOSiyicTbc?FeIGIrccQD5i!HrI7$BqvAt zKQ5nNIo7v`w;ygjEA`u+U)+!S{<+q(%09BkzMkC{sjjB^NXCEo<4o(>nhizi^Bg=+ za`+STGx|qi%3o+t^ankEsrfGLU-%RLarS4>DSx6r!JcgqtJ>aww?=&F`5JP(uVEFF z{j^?G@ft7Kfsu;WNBtC!d4HRY{Sx&z>^D5(UtwQnr=aI?m;&YdI@$L3dq|GcUn-yF zj`yFif0`RptuN&=9$Pv%BMWH9^Ig~r`zOfy(0Wb!-X8!Bf6DR2`;mNpUuE%q83OU9WKaICMN#j4X*Kapi zs(v)yFu%Eb+=`a(C(K3uf^_>G{k7|>c%PzC=2!mu@cNGWQ}N+wuAu#oj|`&y56Ul$ zc|5v3@w|Pmcs`PmUN56P@H~QA+Sa~(j`n>X>nUj-y>=wq!>j5;OwjHgueg%#_4^lY z+6z$=(x8C6BC4L}$)EuLmc;z-07Q76nL+Y%W0tQTCCD`#`zoa=+yHNdd&m+H{<;h|2`p#$RBvE{GV9&Z`T@lp$ll4}1=-EK$w!T)3~68!(Kb5A`!9qqTs(SD@Qz8^N*vm&a_Q)TFMC-F|s^9)apTy@`0l+GuM_Tu1#;3a!}$#DR*~p@2J@|aKCvG6sps=}y!WQ2r5w`}KcARJ z^J78!yyg(SZ|RZeL+X9Y=`??-=fQ&F&nkJoBKOC%Ux2@y{x8G+yx?-$u<`R>$9x8V zA>J2IetCWadgOUchVA*hoZsa0bM<~euFLU!<1K{<^w!`r=graUr_J26w3WO%{RXEd@t=i3>{ET7M0$mbiY5FdyA zc-m9 zKJ@u8@>n0*8|LwO5c)%YEF(i8!07VC!q^O1o}l=(SB?icK=FJ=#xwbRs{;j6yg`5Y zT+Hja_6uM50@|D8?u9XQL^**-+FBSZrEiZQsr-38RtD{#ExkTRf0(cmw49I7KI8B_ zORg{c_sjCIzmoo6q4$S4k0_$P&+E5^u@XHYae0;%kO5PbNBSvTcjl}2i~3@dGv(tY zT|T|9fLNG1Pb$QD6HOpEpOEprBG^F_}X$NnQfR#MW5{N9)< zzl5G=!y!B$l2emKcn6D>_eki2#ku`EXE+=T&KQO-0 zp7MEEx;?G*`yQ~buNO4gqt{1dPwIUOfx`{1ebCvi;e|1jRY0qCDa`Z0e#AEEb&DPjIQoNGNF z!Ta4zL;pM71vFP9;mdOH2iAjQ2*B@?yo2{iF+E}ifgJG==N0MmA=Gypp1-Hw7o++^ z4!nwguov$)AYkbBqj(t0lJSZA#}$z&*#l20 zTI$Q4ME0chS?c=&WRH~R6q0>ud7wN1I&I&y@@Y&J`^ZV#D1)6ssEAK!;e z6$JORbl2-U9gr6c3f0 z$}0s$iPx*74?TeNVf;~fnBh|E|5NMtY5kc}zZ>whergy>cKxWoO~152?=McR->>xJ z{T9TNeDbH7U;f^;4D0tGRmctfM-abzEcrZEo`)jdECj9kJN)W z^q&wX6}^LKiccRX`UuhTaTVDszpn%P=uY_;?UAMW_$6T~;0=X&OA>W^O?6*;VCHyB7`R`4{Pp0GGqCYympMm|SdVik=5A|2* zFFYo*bNiw`_#s1k`#<26M}#LeU&#GZ&>`P-3*KgG0ZXsvasKa@@v&KrcgzRHj(M%8d{X;UYws&ek8e{q~}#mI$OC<%fB593uXj zA7dKldH>KQ0?jx%{~x3G5hqUE&2o%itasDQ6@_H$$p z>&N>6^7~0l{{Zi&()f`3>AiF5JW!rL$@6Q97ZGnE;zwNKLEjn4knf*^CF2S0mur!q z_sclX(91z}o`L-Gd8Nz`{}H~1@t{$@+HcW$0_F#-M;-4Q_Nn*VRr#ndy>Ce2JG~$0 z^LMt#_C6!wo<=eOZ~Y059pt7ZLsnH1mHA9!CNy}zgRukpSk{4e)ka{mE2?8DMH zecpimy^O!|{l&xgqy3){zvK4D``|T&YBkK`({Ub*_I8}7!TyS-lX4Oq37H8iIXF zmmjzv>l~dgVVYgiTDApROPkvuHIbWSGd_$fuK0NqgWQNkd-^_O~jj{^*XmMVuaR z)B5*C>=(LuK26B=SWP$AFO8?)_81Ls_9u$(B90Fw)OQZXQ!2sVD^v3U>HoEh>PzjV z>P!25r@#IV-*=$jE=lU=M_;`Q>c90MGV~$;T=92*t>sh=tQTwUI+9+0LSEDTq{%b2=R(B-ivv>>G3De_jLT1;|~ka)5bHBj~dTbh=WFXaz85P3pL(T^9PT2 z*mK(QNIq(LJ4Tdu#`yBF>NhN2b=2<^@e36^ZGA|ds?SklKDbT{I`rG-Q`ue~GhTXd zQhvJflsr|QIsfc3%Cqo$YDlT}^3vYI9-qBd|7pup@=?nx5l0;L6GFakq1O{M|H&1z z>W^;pNBQ$&(x2ONgmZpfMf0nYQ+bd(=hs!3UmL`6TY1-Gepl<6azW=i^gLK5eSTQi zUG08S;-8iCv5BvU%HtZfNJsyz>t5u**L0suw`amw&wdJM<*d*C|I7oQdEhe-eCC1A zJn)$ZKJ&n59{9`ypLyV;c!1so#7T8xF*YUflb6w}T_X5};vF~6dM1?Ucry9Sv+phK z`uoKvF8$x%=zRbE&oAj2KK9zPcjgu^S$y){-~7u5f4coha{AEMiuPD4C*Mro zmVE8yj(6YtO64Q(x(07g{?!*x3@6ur?bgpPDgD9D7k`mV{_&Z6?l0b)44%yRuits* z_7DEW_xt3p-#+=|AOG#vcmE0h{qf)al;IjON;W*kgCMWAXc`E90 zT@1TjV|<`#SLO22mxJ8Xa(u&LHE2(EP(8fIy-M`9K^niuOD`9?r}=&FN!&)4iHC1& z^2!l%Ivr2zbGhjB+HT(^pxxP7Iq%80M(%LAhTSf_2`y7lU2JlX+b-t^d$#-zd4h~X zO3OVSd;T~X=aApwkO%eF)0W?|=MR!go%tQ|#0a^^>&Pz;mhAT3;j_yVBjp)(c~BnA z3Q?IeCCg@`!-okQhHz*4yyS9Cx)8nP8r16!bf-kp&cUk^;9Tu;Px=~qL?zjmktKSy zpUD?0olZ`tOQ)0Pok30yZ%&s#a0Yql8RR&3IIaFgXOIWaAg>)EM~}Cqx--bHJA=I8 z4D#kP$UDy+){i3 zz)C&hp`qr>=c*Wrdw}e{{0>+ABlP=zXgOWK+%ZD#Z1>$~kUt6ee{Q``J-VG%CO(1{ zYW}D~&XM!SUT~}O7WmHGLZ{QR50TonMxf2pDLres@`J`KcM$lre9t&j{x1%BrPq|> z#Wj0=k179m$f;K8nQu>)4}DLSDQ8}{nP-1WCwXd*%aji~!DrToYMY!uwsl7qtvga>puF__c?Zq0S);E(?#Q&usRESJ z`8^JKLd#t<^DJ6}xrR4p#k1mLZ1^m@zSNrJY<)R>;*9siesOgkjBvpny{_Jj1ii7S zlG`r#2mPYg-~`D)5fzdXLlay#WE&|l~A*F)|`Gi`FiIoT-<)Rq%dZG!L2WQl}c zzR91pvi0~8J=)phUG5;df;~BV{75uQ~Md#Rd`W3|p zj^(MC?2l*A7?96*A>WOUo)ZGF)j!&tV6RVQCSsH`48&z!s6P5rS(<0rKckFe$Giel z9PwV(aqWfks_=2Iorrm>4iV5jrgscE7J(EGZT*x9DJ}2Kc4tG5<)N$BPZOL>Hu+fh zSjz7Xy4{&+c{-M#(g*4M?wsBn((g{Vd(-({Ys;$N6Mwyv3XA89yt8S`^cbu31U=4J zH$D}w;c464OZpvUW#BkL?*0E9V`Y@FkeU}=6Csh`nIL>k>?zaR3tvLvPoeQXW);U_weBHS()+ z&}l_wO?wvAcf@!!{G{xL%^$^{I0{0N_Dp&jbuJi?463t{lzzir?H6V5&1jp{{!7Nd z^qJHhB4}?I=nf*Fq}#XBM={*CmXl{(>2<~(BVzi8TxX0}?K@+-%Wc}n>kylb#!7ml zi&oNA*;Te3flJ16;yFHu)0Npli$14}YyP;8`o2Ea!R!IOdX0Rh^|wu)ibKerN>yuD z`&3+9?W@eq$@*BI>}sp3!SYq=E8Ww_*6tYKaUtc}2iRF;y0M2#k05X^&oXxUseU8# z+xmfJU(?&x*Ql%8O0DN<_#@fmbY_>5Cs;n+k9DtD8X&N%3bDeUTJsXiZI`>N+^3bx z{MP9CJyjn3>Z3z$tCxS2deJ_^pQ@Mc5Vm?zy;E}TFb=z=UG~-fsp@qaxwQK57`K1kY7M2Lo~2m z!!AtFt_%D)ztA*}Dk|6d<(w!)y>pI@BMTDG(IeyuDVB5M2piIUL4qpRxU*iux%stK z!=iE~3WYHk@U#xV{8tAIy_NnM;*<0Tts778b!xiPBhIXYR$U`?B=D1rX#F^~{;3f` z)jzNVr#<3nbfAyB;5LP*!>^B?>MuXLPWzXi7a;ek(*9<}wP@L91u&M>-*mPb)}`Q7 z-eOJBJtb^7KGt7=?$r9Y{9vjyyxEHM=Pqwh>(6UZH+8o1Q1OrPcaul_P2?{J#NFSU zw*4pHD`*o>%=$ln{QZypcW>PCy~dy2IPI}V5AXTAUX=IQ-%XbO|1|&I{!8-;W8i?_weP@hUR}I$uIs6@qSHHd=a3J zX_^Kz(Ro-q5uWyRM3-wipy@hImuk9G)8(4()AS-u@78os)B81DtLbB!mZOWx_o?!) z({lPD53;90({-AbK?wLxO?PT}pQdA)-mU4lruS=lv!;(}x=+)2@F|t|pr-r%itg8R zLeo1mJ*es3nx-$VQ~oD49Yos@ozOJSY9vkdrN5ipiG>8o6>W+A6YO-EXuym8c6l9M zw71jzcZyYhpP(OpNy~kyzgx`myns_KraEc-$#6&h!D&u<|8zTTi9Yb_@F2~P@Vp>V#{bKhbMJqi1PV(m!4Yxv{Pvk9e(hZlT^V8o=?yFIBzr;@z zbzic}@hu392QH8B0Zk_)9?opQSi`;Y4~}6BT7(U`L5nqp+@Qr;LvGOYLyL~`4O%oA za!o6Lp=Rrp994k7V85nWeoWKKpZL4TH>mH%GJOm*hM?%z`8%7HJfUg+yUF`QC})(v zn+&L&f$2)WTSQm-{VlrE?{@ScFD3e`Q5YUFJ`p|J;MZxo2W>{8cDwLLt;zo6KH`MlDoHu#zp@<%&CFrN84M!~cE!(L}1 zgZ@UVBP&%~f3wKJ|ElDVbMT1xiJR~UFh=@k@ZC6A<@yG-yu_pU>>nB&LO=cVmwXi?I0efZ6B8<55UrMch-`bXMX2mTxzKOoUUL`Tu1#UJ@X z-2^Tek%F4DmIje6>28(c&@<0d<^N}$?+ch~?8(eeKbnuG8Ib~ciq)($5Qe@A#pbZ* zu7g8Oos@I4Zi-q%Xpn;Xoi@>7#(v%OyGWTWar-fg`X}WN8fM_(7i;O>HyP5xT zCGXU*)a7BFf3fDUkInq_PQ^E{oJwWXHwSh85<_6-pQ;?v88mPwb$(h)P@?iFzL@zp zg8>yI%6|dV5~YCrKix+-KeyxkP)xj_#_{Et&d)TL&-u9hZ${?R<&S9vl%H^Beu^XH z0$okqsT9YWM5bDGq5PTn#1)zP{|GXZ&ZQ}izG>7~DpV&KpKUa;DPB>2((gz*e}jW* zbTY`cLm$~+W#IIW8;gCmgQiopUnk-ge#eg5FVC)Gf4ic7isb{PLVT$H4iM8x;r>tMQ!?Bqnsol4QGVXN5-zy^Qho{vX8r{j zymz0X{K0Rk`~h{5^`~3T&;95Mr8+;iAG)b*f3tiVH{_Gyhf_s3 zZg~B`SI$p%eAImCvLZD;9i@S&)eL>Lmq)hpxX`E(mj>Oy_I#4nI07meJihqK`Rcnn zo0df)9T6kjvRG%++Gtmx;=({@I1+AaiZ(^MY?#0;%}worwc*y*mi8Mi2{f#mAG#$H z?(A%el(tn~5~y!#@9Kz@ex<#9cA&lDtFr^^T06qgMWLnzZS&W)&zv1-tFnRSjRsn3 zfXbn1jZ{@}A(V8j3$s2!{adj7BaAREMMCx(;M$ia^n++2PjC z=5Qz~E08WeRfqZWP;h6ctGv}vN4(CFyoyucDo%l$cM9CRk+^lOVYu5?kC_d!RaUli zwS}W=o0}Q~Q8a0Qm>unbaO8#!ZB6Y_Nlz~)u1d25O|pf7S8WCMwc*wc+!>%R+}d(O zdrBT?gzo|>Gyb>bmjVfDbWT7(kA;%CfHJaAy-`{z~~*iRDPP z@}eDqHBIPc)Y0b83q->0H#7ylGG|`Jf{PX`nt$PfMY98``-^Cv{z`??M`c7KEsf+0 zTm5A@BlK4YF&!UOITY2{6bslaa^huE2cTcU%OqU1Ri*1R3SQP3_DL+;HXY8k;g$b* zn8Aj&rD?rvk_`|4XnS<|@TXm0TCROga?+V9f4-x9xEa2{*~IjP_*|$;w3XK!Zf|UD z3UoxMK}b6NB77FHWh3xL|6FjY?rFDI_CGX*BiYKg^{@G)>Qd@Asn zhtGU`7T|-5>BN(v>+7p=CO^ehlwB-$g*@*Ptc1b1&c+9>NhS4S3&{HH$-!^BAF%$$ z_bvaov%jD7J*Anj_9@}G-b9Gz?8yx&eAa{659NF(@4NXA7kvNR?@`%B_}Hh)lzVG) z>b`F&8=eM+)2njP2G7k8>tmvu*dMyZsU-b_>4@cPb1X&W~WzIu{T9Q+)QJ z@_d@l0Gj7ra;8_D2fwebhXEx$96H(Gh46t(?v@me=hI>+BK)8(g8lv9u<((5z0PHS z-mUfX!p-gvkIf!ciV!W(lV061S6xqaE}`(W5^*NQmn#j2K=Ek>?M$1@g_{=7gZI2E7X4LTLSXcT_!DCJHaMgQ%i@E?L6`XStae)*aE>+7T7e>e(%{U~-`Jc^yO zM&Um@ivDLu(GR#x#jqfq=%#R!RsK#$H@XX(A;5TXf76Z@x2Ms*i`LKa!e!qbikE;4^H}K5gVc^-$ zT?U@{yA3?s`GkRIe!{@BoqG*D^YeyxG$daW?<%&#-> zT(5cq&;08QJlE@b1JC?M1E2SpYOiJk&-_*c&vteic;-h9ywLh%2A=sn2LAY?%AUA^ zXFfmo;Qn(|Z8F=RJHt-zhpD^&u zf6~B5wf?6KJo6IyXMU-HXFF#bc;@RrVk1oZ7Z`Z9f03b|?Obf&nIAOpY-hEBXMUZ5XFKZ+JoB$J z@NDPx2A=s12A=I~H1N!CHt=j`tAS^Jr-A2nVAQ}fKW5+?9##FP$G|gxlY!@XWV3;1 z{uTqze(N*v%zwbZ$F)5V8hGYEY~UO9`0Y3F%-><)dEVS*;FKSJC7Q8<{vZg zecGSL4LtKt8hD;Jd2`P3kolJ0v~xV+apc#0#xp`L5Q)Yks|fXFptL;F*8D zf#-hGVBneGXyAEXZZ`1DZ#D4jw@w4k{HTFvzr_qZ^EVlIuGeM*&-^V0UOcMeZJ&W> z{sRV{?R?O{Gyh=&&vy13c;;_6@Z9b@3_SC98Tf#K?BeH!v>!1JYwLPf7HOUoyQD3^N$;NwsY9PGykN4XFKJW zh0v8azA@j@hmRZ&xgGtQ&v@o%8~A>XhX$Vc6Ab)*jW06r%nulN_FIX8XMU-H=QuFi zz%#$xz(1|+SzzFqzu3SZ)A*o)XMVMTXFF>RJo8r=c($|7z%#$zz_Xp#8F=P57iy>lHKb%?417bss_&Bqp7~E3c(yZP;F-VI zz!zzM?lbVrKWO0D&Q}dQ^9Kz)+j-c)GyjNzXFHD?c;+87@a5Y6;|8AjCk;H?NiStl z;{MEhOY=D%vYmbd&-`oy&vxb+c;*)v_(j_O2?n0|r3RkuoNeHlUvA*p&IJaZ`HKuZ z+qu}lGe2nH7ijyd4LtK}4Lr{yD-1mI>kNEA>#sNP%x^I8?1x4J&-`Ws&wgk%@XYTt z@a%`EfoFcqz+bQJ?=kSq-)!JHPHr*q%pvPiX(_Gw{scZ{T^I|B8WU{y_uJetXrx zGk?&)a~wWw;F*8Kzz=Hsj~aO9A2;y4&L1}L%s*-1xn1}r7>&yRFXg=G?{;bn{t{3y`4LtkvIs?!A>kT~nv%$bKztO;N z>Q}}#8+hi&3_RP}W8j$|H}GudCIip>%?6(B++yIF-)G?YdDa64p85R-p6%Rj;F-U} zz_XpZ3_SC98+f+!2?NjkCk;HGzddc>nZM7#vz;#(c;@dn@NDNR2A=r`4LsZVs)1+z zpn>Q0!(ju@{38aQ$Kz20&-`NszEAhh;|8Aj{9ZndFM9u(-^;b=i;hS%zU+Lz=F@xE z0f}(_hu>YpTj)p{$M4o!%-8Tm`pW!rO*b3-L8&bLo;KqL^?S?-1JCcj4;y^8N33s( zv^TYOh1RsRha=shEYj2(E?d{q-VzE&BH?ZkqOV-72ybX#+kB;jE{lnhn8rn$Vo?zS zB-#{`-?D2Dw}#eV6ka+P74!DQF`??D^}Nsl(v+j zrLbIm*HIVKesfJz6q=jDZMt49Ya*yMxT`V<3(EN`*Y#ewvNUo`c`G==6UkAs|~Jc3&%oj^V4?~ zq4t~CbuI{ny4pe&DuO3P-!fKXKmb*#A&-|P&9 zqoL69@cJh8ZM^03>zT_Iten66s>T-OjOA!!_^vXF?`f@pPl>B^z&{RdW4Ky=C(prG zJ*TNTw2r<$Dpstxc4c)-xD{ou>}cH3+9aqWg&Gx0wtlEpxj)p=z80l@=x!7WHNu;! zH?iIg?WkGf$JH!Nm5QPrA@!X*Nx zMmp%bYD(1#<%<=tzFIOv(VN>s>(;fLc|3h2*Rl3cS5q|9MBfNT!*#TU@a^AdS7=cx zE~rjMZ6mw2Ce_zv{{p&neM@Uo&P3$T-?o|Hc{CP$OE}!*Yf&8bk ozggSgr-5WYeG8BCv;9#x&TKo%68kUoKlTjTv1Grp1UAPYG#4ZGan8O z?>sp0)YDH7q|RR&n$!H<@Wpv2!u#Lbp#2lmfD z5iUP-GThX9Cfs}Qt?=+em%|r#UIxvQP?_`4z{UAb4D4Tvehz#$Jf{WYpMEP`cHwe( z$NMzipN2y7G2Yrk=;vg3Zp%yIVc^RzoDR=9bpgCnKYtP$K5%H@srgS2gifCfU&O2F z{kOt%Pn`;vIo}UooPPl{%R_Too*3wbF3K*PK;I|Bp_8Y><<2wV(EQ8cr`8hB%R-@( z2M6Y!dSW01o-fXSDZGFFTi^?Nfb6|^CC~X%s0_Ni_z-BIx0J+}!lC!x3a4y46`ph9 z68O3l9yjxm&@kawg4dmZzjOiZ{tWF743q)3Kk-y}_(8z5?5jUb{Cd^!rI!QAd$WI# zoY-BJ{91O9?C7zv0talQ>D^Uz@Tp0fXXKo1QZM1Q>Z~AYf8d1uC&9mj4{X&~!5i=a zO(Wc--a*~*D78DrSix(6FmNUn=Iuxmj#v=tug<8}RGFY&LSPc+!!!L;Di? zhW?TD4LzV8x%5BukxTEgBSU|k5E}XpJ94QWYw85nH~iztfF*{89$-iQ^DuZG2G1lT$#)5Sjzj-m$mIN6E%YAb7=AS&Gz>Z3gB(Wr zNrvH96GOw02fFCHIDs9x44n+WoD^bOcU-v~%ri0v<0osZkJgP(Dt;6q*l!WD#*BU7 ziFDPNh%4HEQZZUz#{SsN(&u_YG(OSK&^jFk9eTuYmN^^*#X`D!6^zm4lUyvlLkjq5>NPvZ5zaDN8Zf2HwZ zuXmqK2-#p)<$t#EY~)?&-!)SYxnQ45@LRIeZ(swm`9;|L++x-@2>X#8luv)Kp7n)` zSs&B;A0eA6`(ODEYz1F_4}SbUZ1%6PU(6rakt>H`SA!iJrwe_MpMFwN7`m2Jbd~IP zPakG|JO}52AM+Qz-D}+b?))uSRcgp%>3!*pmZ7qIp2V2nL3wA6JqSe-Sb~Jn2+iShe&D*^ljlMu_y|=xi zR$@DRt?g|Ae@CEc@uKZ5z7}tD)ArWbtQ7Dy-xb~lIlOm;APv5{&h53^ z0^Sy%u#OerU0{M>n$^?N5b*jN`K0p)+B(`gy8LKPxVF~z4sUBm-d*6n^H(wvRG}&fpY2O1~dZh0vjrE%s$JZcg+-5w&em{&%iRM7}sLeu3h92G84J+AfZa_8@3)FQne8>|Iw+qSj)I`UX;ZR=yTQOueZ zYib3iHV<31CT}~B-cdZYME-JD!&+M5Y&N{8MyUGdXVw58*ulrv?yGH**7J<8d}kR4{#gKFRUek zk}H99PX&DMp=(;|-tO+%%n{k!J=?)=86)^SYbUaF3d@b&l&h`#>gKt|?tIxfmqGI*>+Knr!)|XYR ztzA*GVa3{#6(wwbg?q7kzBf?c=x%SpdBDS7H-;u&>3jvBe^_|k$({h*zRnHMhOM z+4h2N@MAPi3t;_|AE6v>^#vkPM<{dz~=_- z6X~o|^iTABnhARJ4}I839Dn79oc~w(7dEeyyInz+{*P%c3|KxraA$!;z zu;*X}yMH&)Qh|Jz)7aO##z(k02%qbOe)RDhtE;QIyl-Yh-pM-oQp5rMp#R+9E8WoA`l$6$Bo9klRZW;}ryE&EfJ;`!i{npYtN@9NN=-%A1b*+z7iY z>vivIGlTQy(0KERKiFq)mt!8EZ}(46Wq;{pGlO$y5q>VmFJw=2vALw$#QU-63>AXj z4tlI@M1S1L@pjY&z8H8n@NN^{jk>_E0Dc+Hq+D}7W;J*%hCh#7CpPk10&qQ6| zleN8FF5t0<()f)6??PSR6LcZZR*BD+{BM;a?9vNb+(2C;tlCY}}WW zzXj*aA#q~(S+k<`*ZF+m_Tcjm?MYvD^w0Tu$EKHNJ;!KWjYRfw`)9(w(E|EHRVRDH zKQNzj`Fy4M{L3^}^t>j>$$x<_eg5aNxu}os&rA=0%_Vzsf5mv@Ut5Vp&WDWP%$XGb zxIg}a`d?-0zi0RKm_H33!upnABY)$I!kLBeFRb@b|04gHo^VK^UxQAq zATTW<>aSQ2W(KECBl^UczfH@eI`_AIXzxOM?{`SwbdMtSP2<^>e-0KyKOF)u=ksX&OSEhc{m&$IEB_$< z^G~D*kMT;MUuNpBXLm+S-*v4Pebf9l=$p^i46g4CsXzFCMpWN**;p@J7Ja*1F@0B% zzFe{S&F2T&OMRpNj3BfH0V4f9V*RE0!3`+&m1UNf`irgygN3X#Yqm+gvGmom8~MOF zG@!nz&+H<9p!K1S)=SY|#kZM32d8(${LR7jLH>4L z(?|0syp^xVn18EaKT*W|v&KWR?^z;{>vutL{CKiIA0P3eG&z^oR6OX}jiq13FF(nr z$K=a#B3_EmO7G+Rrs)L~fBZI>jMi_OKRS<>I^pr> zkHz9U^a*(^@m)8>_v|?F962zVMCw-$WX9t4;6iOxz)q3d5Z`s3?91aj@*ns!J~}3U z?oOihwKQ1?Ana-OKhnR=7@zGYb|K$T^3_qk<8c1^+D8*bzC-p)iLmn+kozy$=f?gC z|K|CkL*|zx4=*bJ2-qBaeg5w%e@eE!gXK((hxN>ft%As>`FfUYqiDnZ1M(7{uXh=K zM)DT)0M7>y{*%7Pmdx{G2hWe~xj{$r?0<-4JISBW9`f_?_~_GQhhQe5XQ%NIa#Rl z{U-W@eKW~j{4^gRZ$U6KGaApV{1E>=81MT0(E5ddP6Wcg&HjFv=96(gVEr-qJMgCY z0KCcHfj7(t=qER5_J8>Mbh0nccffB(&@dk$FX5y9e_5k^3jT#c@$p4`2@&}hbE8iB zJ--ljqzlv`$#aqRZ;phRe#t+Q6D<18q@~JVC*lG4g*Mdutib$EKphH?&i};p7(Na7 zB-D*~7fwA?d%~xqZp3FyFykkn9>YUk?yp=QkeBK%a3J(!!+g#iBOm9JOT?Fp73O&M zBJ#&QES3Bl{xpugSwa4kN-g{)e!GY#Uq+3JVZ9hamB)kWWWUtsP!;i<^qq+9h%ufg zCQ|<@o+kvwY+_Xy1^L6?E`|mubUuRp*WoTKwW8>>Kga%99AeV`@J;UTq@9SrNUmqe zOC-7AFFl~A(uql=o9usBTN%w41pa41?hK5N5C7@`KiY{0KIWb9dIID85a12Wmpu*~ z6`NY&Jzxbhv09V+>e>^9m;3?BrxS4l)ooG!-R!|zHTR<$@@(689(P<>6hm- zQU8nOGyHtwe471+`3vIPb@P`$hyNkIect)Y=zPVI{E8Rpr^rVz-&_vzM}EEn|1{?- z3xPMT&$M1y^A*GTi+GWR^%~2Gi1*N+X+6jKY+Aoj&%$~grAK?LN3s3Zpk`jLF<%KE zU9ZQ?SNQz490T#8Y=xG+!Bl{yu#^l11w)kC*7rl8+!>n(`3`#+%^= zzn>x>$r1}b#phA^(Vd%*kpKLDnUBmf#jnx(Z^|xEl2U9bfiX~7v)eqxsT8H$rB;vV4BMm zpRj(K_Sf4eJ|*(~Ige+^kJ9(pdH#g`KpJ}k`Rks9M2hE06Vn`&4;2LM6s{9^zKiuw zoEIJ2#Nx#>#EbNQOUm)1N&h=Gnqb* z;p_8xeBQWwVr+l@Am5+!^HZL0zqt_flMVav!}z?xKAG>wxqtiz+KIIKEBc{ysH+d_(-0GL`SY)P8()d^4T@zA5&fd_H7G_n&I7-Jp4G$!$H}d_yLYp8q+NM z>ogtvefA9yaDT@B^M?H4^KkfsIA71A&rkX1kve@~o=A1@yT-EL=KHm%d`?F4@$*Fz zQmn6bauD8rD4qJJk6UDaTI~Ek@8;w+pC6>LFM}VkpG5~a|0(VIIF_H_^8-_U@;THY z)xC001NSs=PXqTfa8CoDxd!O)9HF)!-*RBxzr6N`kA58a`pdnQzuWS+fB%c}aevkD z@x{N694=t9d!OoYRl9rV<{axg@3jOw>P4wViS#N=_1Lx2Et3T=phgP?r!lUwB0vg9jZsNGKSVt7F|NJ_MyM}( ziFTZ~aaP{YjK%Q#QmlANcl2e>MOWEYe4d2wsiq)7?9mdzaN;^RfG1w0fJuYHvNtufsz8^oiJ%HwBjou(!0&~y~WA!JV~qGy9n zlROrG;8fg}WRrcPYzdJWx0^xldXmt4RBjq64n8WkD?UCdw_C^^uaBr)V1#5` zgJky4L^Atmk0MK$IFmF{&LG{5I2%0&&79kGE9Y@_t_#khqoNa|?S-xjOIuUo%~+wQ zBDB$%b+k%A4SF1fCvHdw>vVn1gP416?m<1YIqT6pHegjT@*`k{b$W`P<4F8HUn#Uv z-bG`p6~($@#CW=%235cDSD;w?eY#aQ_cGyDqP^Yd) z*EDL5s>k*jHl0Q(Ox<8FbUl`GbE8o%GL~-Qgl3uxrqtl((a5pbWJ|a)_xlp=?L+@Kb2Z-Teyj8U_M^dPF}ju8czxbJht}Tz*mcuP zO&on%Xyqp!y=6aS%7;d^kM247988F=Ahi6TSn2Pyx=BBARz9md@#xj&j#gaI-_iR^ z&cFFA?~8V}_eIyQX+9ZuY$DdjqTaqyJ2nqIC>FE#F=NzIv-WLWUt5B`HsfdC+EsWT z9fo7Kt;(1q@Xty-;xjBdlC_+p3m-S(`^ER;#P&bd|LBu(E`gzcUE}?CE8ggPXU6^| zUZMG)xvkT+G8fJlXg-N@){X<4IL|QBZuktNC91jQ+Pz8u@R1D!I)@Vv`5>dpk|^cP zJpU)|vwgO>*;6m%y!gErCbqEe&;9N1YHWWsYu=2!pHBbD;k4AU=9GmP@(b|%pGrvW z#vj$?<-#82;0e$Fd1WPITTAh`<#=81`9Gu}vISk^J==%izoIu38%~0vU#&x3Y4;Jp z`N2G=y8hc7-UmM#G7&B3@X zQLVK-{ut77#wGe)MSrA;;*bq!y`!-dJ4!=j4{M*)u2N3;MyeQ!m6rTMekLxKR7v{- z+5Uu~E!|r6PX0sfJr*_^q{-dDQ6bMG-mUG+2#K#}ic9SMYqEW=1SuU`+tYsTi8$?# zTiUCht?g$Bh4?*+KweMF_8-bRjZfb-v9>3_M;D{`@0U}=ZD>!os=cz~VRTF|4pbcohFV6U1lkIaQC5=zT zqCV!6?LC(9kIVMTf2lpWw6*;bOx~yBjQ`CoAVzWXcgC#aD?chn`yRCSSjO*{?Unzj z@l`top9(;{_V2`LPgBC$o_;I}QO6H6oAiH0wwLTjFIN4pNBjR6M}F1*ifjot<@UFZ zPd-69-C8FJEtgC=pt`E|1isyNJc`TtX&Ej$U)8)djz_$zd0}m*>K^GJ#&#Se=aZsS zS2e%XRkhn*xPD`M;fg|0iq{k^YVNFGT-3aI@%ri=h0Pnhn;JSd*Hq=#w`{83S+K6n zTUflNcwrtY6d!*!pV1mozV5x4EXgv2N4Gye&25tG$~F zb~cw+RIlGvy0UrYqt%W6s`bsQ7F9o5wyUgWQC0JnmX-N+t3=zP=H<;wSRe{@iAP{%VvRyTs!FOHh z%E#-w3Ys8qGj#QMU2#F4ucpXf*W!bHHkQ}5u6uk9*THASs!^XgkRHr8#a zYAfFCt*-W0EpIMvY%AHcz0zA$yk=$T+Wal0D>^r~G|~Z&+2{ym`|i*t;r!bL;x*#x25LJBrG;q3^tgn(|JoUA)Eq$2T`Ec|`e}y4JQd zHw9{UKJ2Z3cv)Tk(uSp8?~>&{UqPLBN!~JVop1TJyhRV^Ev*%F^-bHn^}g%<#L5Q# z+bZ;{D)_aN+o|N;;MwH)ioQiocsm(+(c?I)*i27+Mem!wA5@o$OXS2sSJo~uHk+j?u~oQCuzZ@vLW?; z&$Ieb`+rLO&yH`Te#`lS=$YsnIDvTEGX_14b<{xry1HNGJT~rI`U`ehC%PXh9ImI; zy@hTX;O7+%*AjJK;&eT$f6GpI{)fL3PW|#?z~>r*(QFHz=t~Uv=&vHJcD^!>26#cN zv_#i@6LhM3>-9ynwX+Y@k%S2?9~@_b2|1LqV<=)l6JI?Lzy@)twiBF39u83PfhzHy z_q>63jYe@wyyr{{dL9u+wXhk+RY*Y}Gm&%FWBZ12A!%%2E`Zd>q{{ije(bN7W9{l0!z<&rh@pG2jfiX3& zwCnKyJQ6RWuaqWuUCKdY#vAudDX&|aK+Ufv&{IEGBwV#!B;jh4S|a<;k^NLklPNv_ zo#Y=+m*Pdut7j#juSmF>Pp`{<`lSICd=ud5YEbmF4s|MciZ6$6mGyrRVD=qci9QeG zP&p+o>}^~LepSNl60YQ%fc}*45Ne{tPqM)CBzy|ssx5)OEa6iDH*OjC7~sZK@wkN3 zxHwIWmJ{-h#x=rAq@2{h;*CgGOSrY48VR3n=||~{`Z1mobsPGp=bWhWoB?j6KXD!S zRRi28Z?4q4mCxA{Zk4xR8ptZ|8xn5q=jRe`mG@VMevJHoY=9f(&6M+p#x=&syz9V^ zN;vUjT+N=7@P{n$egnPHzOPHTm4AAQ4i(~Oin!(POC(&4t9-Fh!qvD6e$+s39QQd1 zw~qTO3Ac`$D(5YYYn(4G2_H3IB%JzDypRA}4RCqH!{s_^fJgth2bd*tFeR^>Q#1A@ z1KcQAql7CxDEejzx61pXgj?k;ks(Kon@uA-WS8-s-0#{3~I{#GttMz_1_@koW-O_%m zEO52n`z>&_-ZPPQx7XG+wR!_x410tI?`m&nYklLR{3j<@;DOBebw~aiPxNz*>}Gc! z)KP6yYZHFvv97)S)|$0;wg`rreC=%PO^)0;@Sm3)oBA74&Fmf;MF4oQuim((q{-XttM6FT*3j9E;cs&H$8FHu=546$Xsd1Dj0u6j zp5Rkdg)Bx1SGENd!L5$PavN^KX5PT|_S$y8uYUCJiU!&myd7S)fWHfe6joXdNlG|M zkSwTg@5o8eanr^@_8A>}fys*Lve1e0k&6z-9CXJs5y2oe4YX71OsB)NwnL~`rqiz_opz+vuXS2$sb8n{>;K=DqP4;LV4|amRqy||_T!wh z_q{ilD9)!X8*a{Bd+ojU+H0@1_TKC4bMEa+uU+PLxtK~W_AO>ZDwnb4opx@3X9rjo zO8QxN?FmoIFG~HM|Dq2{^-pE&z-dqEk>~Jv-cwrndt6_@=MVUfss*YnUEjIwI|FZK zve7p-x(8lg;~99z!w&q9haEZ8%MPIIXIMU)|BceX;2VqC&wlG>2c~9)*Pike9y#qP zAX=Ycd2IeK-GRX&)N{K$KWjnT6t&w;KNA00mO z)Y0LaUU_u*5nL-){dD-oSAH@)8`o}JS3L0Y@XANe3}1umwYX-?xNP7*U%B_08(+Bh znXEaN4s;!TWcbL0uMMww`X|FL{_&~df%{$_KJv)RC^vs#;Qsrcx$)?I&)hxdk>Qn3 zA06)6_4M$O`+hQf=BqCczj)J2C^K*1?)&d~=E$7;o*6*fFaG$E;V!iK;;N^ISD>vU zbEwP}17{w&=b3xD?|-J^Cr5|x`P)Z_@6P_=aAnTZ!)NY&Ww`Pu)Yjz#6~I^G)3y2$ z@c-4}iteX}Z+ztG;enU_IDF6jRA%A8nS1Yh<_OB%UH!G;%4J^}PJi^N;f$uIhP%M? zi#NT1G7APsRx2O<+h^`Be`L7wKD1T+)bPsk9}oZM(96U392pu;uefsH%r$pDQ+f1n zpE+|4+N%D_@RLdPz01x8PKE;>TERyUh(=>u>=A1;Ev37u$lvH4FzCtNPVU3BEM+jkAs zJH*)h?;&p*uOnzZ2EBQG&EkPKHa0>J1EcTU_QYYQKF8C~*FCOErK6;eN*4pr#cx4Z zFX-Z*moA3K(8WVZbn%d(i=?AUA5R$isPuO%ojfyraOauDwLg33HrK%0yWIotEM^DJ zY^?CUwcRr?(&B|4r@)V<4qW~FTMxbl-9G$y!=NwVFB$x|@F06`WX5-H3XFd9ioj@v zm*rf{_OE^2lX~QTJYLuiJ90}&>S$};{;QbRo6}X8`p%B8cm@yuHZa;c1>c``jUN8D zhQS^8Bkz}i(H-9h{3XCQ1O6%CZvy@_;2Qw{5bzPe`vETk{21WB2iyyICg5)Yz8`Qq z;B3HO0~`T-4d7SYgNMHaI0*O(z`p?e4}iA;o(}k%fFA(70dPLxuK@lO;MIUVfIkcP zcEHO3zZMwWaVOwg051ak9N=Amzl!S(xUNHfCG!6P@Jhhf0GaUI5WF0QZOIvdv)ah-+h3%E|l^)#-fxc&y$B3yrk>&3YK z0@r+8e}roeuK$5+Cazz{H4WFV;OfEki?~L?$7gZ<6R!8*`X;V_jq7W;-iqt1xbDLB zkGO8d^$)n-fa^NQ<7V_@Hm*Tje~f;u#3jCkws-OUE&4v;DAn_^JJzQA!)vDm!fU%8O?k(^YRLO! zX*loJZ>Eg4{w8u2TYlJk81ugW)S(&dsk8ytPL|YMR%XhTxetf@&zNZ~+`z*d6r0=Om+rEJBe^1|21V59*2XB4D zJKFV%7rYEf8J(KZ`qz-z#~=Hf)?u%d(+u!53p~vNPqV<&EKN?&VIAd^lQotb(B4{V z&wI4(27KQ_-#tg$w&D9$`tCm3whP~Hr0*^{maIR0^<8VPf$V>TPuD%(clI6hjxyJY ztGd{s;EvO|2X3lMAH4B$@8E5X_#E^OcHQqCy&GkBd=`C)cn4b>BUc3ik9k{{x!-9? zXaA>-vGr%LZcN9Tu`9A<@M@V?fjVBwqjo@Z1+E$C;kD&ckNiWXiw%~hGcgC>k;>PD zXFTqoRlpx!?Sh`Vn~$zwY}B8^j<&hi9QCKTa9xcn`s7}PtLLa*{@SA}ysG>)viwR} zenp(}%a5+`sq#&-{4!a7X`J#)j;_d3aFghD+s59{M7Jeo01rk3k4*mwNxms0)M5% zZ-|2r0v`Zg%E03L7sy{C>urlu?-k&eOMEa6{$}8>jYBtk@nD84cZ82+Bu>4Vz~3S9 z_s796qWTj>{lAZczk%vY{E;~Lo2h=DsQ=A4`1`58#D6~y{wq{JU(|m(4*rIQ(TqEQ zH}z&K@?8b7`Maomocvp<{6(?l|C-9j$-jrn7si(VER~Ow|3xZ)acudoQ299dU#Ieu zV$1&rm5-DEBPu^Rw)`)se4PAWQTZvc<$puvO@^SLtqVgrN<=<#t?se|<*w52xUkowx#%&>{jaZC zhBfU{tZkPJytREX*1U}a@7&ffaAx`syJ?>m-S5F#*oXXd;4@Gz6Xmn82A(i*2J5{c ztV#cHI~#oA3O4#0)}60sHVpm`)}e3g4~+hPH`WH9P8oO;>(h}x1qMg4Uj1EWM#WoL zzrOwI>VbE@)G#exdoD0~`o|4}&tc8={BsS1zsFka_ir+M zcOj3BzVKSZ;0v$022cOkHTVat+s+oD{U{s#@Aihl;d`k*8+{S$yFcbP4!-+;+2~7H z`@KB9aqyilxdvasI`P%Tje~E$nmzEl%#{_-eR}D@TR&vrGXVYrqoeyl^Y*~tTiaI- zjQoiWzP)<|WU+i;=wCboL-mb=XFQFAZ$8FGM_R5Kcw;{s{hwaf;2RsS9e6#{HTc@M z*y!tPRt>zm*o}5vXwyCTQjQCJxCdWMbBzw)OFX&;hc3h1`RTwY=D@+TMSw46qeECf z4$VUT*C2-+!2b^T-+{{k{58O{Am`as7VxJ5Uk13|J$U9ziwDkJ>>3^VDH}aAeMQBY zPovCpSO@L*P;7_CfqV>m5zOD|n73&>-(BZ~x?xyEr`g z_+buD8ad42Nyne$@WmrT=-1&{)fK+7B?B4AE2sR~mk#*KmJaxUn@#ZJ7ajMNE#r6} zuV27By(L1N$eA z99}#T^pPY$5*yypa9od{Co32Dy}p5_v;3{xK8Ka*ABRGh0dV-x&hLkI(&A_Tt6@o*Bbu) z`hgr=tNHg00~2tq;@>w8Wa3J4+%a<#^>efWpHn|B8aVx_DFe?xFnQpG2PX}@{!bSV z9NK@w$QRT0ReUAq69W%>J~r~@#VyDGAG zKJ;S-J;n8LT=zGR47DJC@pZ?C{*B=4Mus*Ly!QCemk3@vGPH)^b;sdX0IwSvY9e_3 z@uB?$uOAt@8nEa3ksczkFb@)zTJ4b|H?GBh3e zi#HudJ;0krhKdN@e0*p&!J9{h@(JE@9QlB^j0|NGoxd6xO2hT>zdAnT5qsYq&j`N& zTSpvmDQtfU=Agy!2aWIx4e-xD(0tMbe{sQIT<{kc{6#WUJ~F1dq$@aWQWumgh2gvKeA8>~?YAut6c*>^}sLvRG@CXztVu$>&WhHNY0I z6mT?!N>=yo=Wb=|vD!2H_MyJJ^Jh*j^}-Pc*Y>*ngdo4R&w+ik<`dn9~w zwe_Wa_s*WSK33h=8S1MRU2AFI)7RbB+uO3UzGY{qWqaq&n_DDiedn&;-N99DJ-xSd z-FjFC-?gj?uil{7eLsX;$Qzy1um(YvE9*xB*t z(~)3jZ)fmk^a!IA?Ck!t&^Gj=?a!hk9i3bEY;V~XY}?r>M#s!(tJZR0X6Jgq=2-Gf={orUIy3$Z z#!z)f*IAzXXJf)@YlNz=wPG3bdSMxIVHxsKTNcKuepyQk)}UC* zw{^GFv;b<~)6v$_yR)_?)Y8|!y>nO1{Fd!|+JYUywyy4$KFoD3?Yp~MwsvCm+S1X9 z`EPf7XGaT`+AYD(zCG9ybhK>U+rsmilX&g95#IwwK(eh5bkNYAox6I?#x5+KK6E<{ z&DGMv*33BfrPD`$@1-V|bvrwER`1@nt+%tUhPAZpx)~NCmjCNkEnU?jFfh8d?yk1p zI0cmX^|S^1y860y@8YWxiw050y6n$ttvS|&TLGj8Q2;Y7z)^(X5%5xAXlH(I@FB0u*bZbT{%ni2Il+ip1kKK+tze=+8d7 z=49Ow4wo}u^p0@3kAD?qs!SphRxiPVMSXqV!%jk0?BLa_J6m}mVIb?WMFBjfh>?cUwpTjhuM>_m*nUt3>2ugbrH zF4glFvc^4K-5vhky}f;%JN-NN^!E8XI=6N0>h!N)v*y}$EgP1uUAJP*>XxOO)-PSX zWa$z%XN|w!Kc_9&zQfmO#7SZ>KJ_gm{ ziM}h#qwiFPk~h!(elFoZ?=~-V5%n(fiuNuhju@+BuY1_piOehET6R{0Yl!2sQ!V^l zc1nY**|Qovhn>{mD)yuX&t@kyxRM>$;0kt3gUi`r4W7jgY4A+;ISrn{`Zc(W-KD|P zSy+Rov5*E&Wjz|~XFD{wl(lMb3EQN>m#}6HE@sO$xQGQbcnYfl++fLfGV^QjB$lhe z7n4hIlrLmw-A?!-c1nW_*hvk}XHNpQ$2*T5*YFeBVGYh@hqUrJtY5=tv#tVRzyKid?z#@wq9}7;i;#PZ$K|cfV!4M9>&GXIkGG=O^10*} z&EzeIM}kvrdNiUcVSQJSvavMG4U30T0p(waJhuhTMqjq z3~6wH#Z8{FX7f@ z0v7a&*(n^fwdobJ=2Zd~^orRD=)O&_nANNou%K7WPHq&i;;-hf1Z?qlgMbx(XWIm9 z@z*I}#UJYuu;Q<$Tfl-I_}wjF#b0x;fCWA9d$WKQf33F)Sn+q_69QKJ`R@?0;xGJ3 z0W1Dm?-KAP$S>99Wr2GI+$!PlKMS~A%J+nXYb4CRDDVp;9FTB8!eI$7m+(mm3wnjj z|0Pjh&?{s;5*G9d*$D{?dW9_aVNqVtgZ@ic@z*b5#otK@EB+$Kc(u+QRosW&TaL zO}!JJ?Nl0kAkp<*d3{G--Tq8M|dOgcVl5AXz4LQ?r^FS0~rTlo^S?xL&mD*sS| z@(O>xuaVnaQTAI<_2F|mhfUr~93f4`r#hQGz8CVO1R1lf*(?&mKt0XF9+dY!6w05F1~{53>Su*$aYlIT zA%2m=nkLh!21;QMdohPSa*H=g{+nf>{50whop6u|{6+R<3w}r<1cxxnej7iySM}Zt z{6zL(D8m~DTJ`V2P@$Iy+Hk5j?11}2sE7S`^mmZQlhnTyZzk23{67@(dWjzeNA?Mw zMHc4=?PuUS619)Rs(V?X0jE0QA7JjW(ynaw^*zx7FBzf>^tSp1OasJWCSWp5CtPE| zWT;O30s|()b>agCOpd{cUv9w7222X$EZ-{O4#pna8}%`QxqRlb`5{&&!28o^{F*BI zWcM-tFXgbgA#UEt0}TA6y_R$sdlu#IzSb@eeTjrp)qrz-jD#@1sw${o+vmkVV!UY3 z0J~A1zVeU~^!q?CC5z03+XL+7xjkad4(a{q_NSHT;|Y5Ag&4P|9{<@) z_6*6_=i}o;^2PZ6(Bc>MQNC#}mnH}j|Au}LeovNOA8R$4c>M<;4+*>D(Yxd}RUBMk zd`-A%@(4rskVN~MzWyqQt-4#)7kJa3Jsc{`qQFkn4gU*%Bm0_YJ41={7uh$_fEzTp zKa^J(hBYbw)gM|`$eGh8^W2`(_yU zvO?6f=tcHTH}KO6!y107hNt$IS?#+pU-}Dap`rMT>?_sE_lHUf!y5h)17BQ7GlpHi z$iPo23_I}DK93Ib{=3ls$wv7}h42IpdYJ}3qY##5;bDI{bJ3rk0PGL?=-wyK2g>>0 zz0ZwkE|P?){Q$wTee#cff;oOJ>U$ZsyH|iLGL;f#%BFz26fly8k_ZKzr7eB~hp9*D1d0?2r!x*oF zz=unzyukN|CS`}IK3~t3a{Z7iK<;1qLsPP;L8))<$i5;z6{3p77iWiUyeqQr5(8g? zNm12*0Q5@@yg%FaPiugmYT&V`v-}h07ucW9Us<-(f06xK_$P9>{AiCL^n=!`GxX&5O{!i3*>L2_|w(b8!eW$)+y`F9RJ5hh8MW5QoBF6G(qJFuB7xJ_H zov5Ew9+p2#AP)Z4BC+ycAZ(+ao z_zL^A#uMu&-5&^hw)HVZU*AAq_5BL$AvBdS?%*&8*gqWz7EPcXT-3+c1D^iSMFkAY zWm}*9d{2xbVto*r?x;`pGtIy+qR|zJ))%3x@%=2HGR1o1Vr@QnFqGvDql9V?`d+|J zsP-QK{V8g}#M^&3R8&Orv)3<^8N#cSr2kl0K-&{VFA{?FTKy$`ySy-r!V2FXT2R0O zHXPY^g~lKD4}KBIR`iPt*!Lau;hz?Q9+HsHz7jFNSp1Y0aepfJx8Hz16p)DesXW== z)B@TrDSjfMt2BNi`>xdF3wtbp@3G4FhbjtaJEiLPhbjvl{uA`;3T=Ox!_a=#B&&T7 z?T@DMK&-{@v_gk}hWyR-Nm&7V!fucJBQ%PV?Eg%o|FeqRK-&D3V|;BmvTwE~@BUD2 zfnV!SRRQ~fv%NX2f?BiMtJd0!gb-Y}#+&A!LPLIYi{Sn(erP-vSn{CpsKIW=W#J?H z>J9$p70~S{)t|_|`K*?+qxyfT#!r9fvLd*Dt31iS*lI6@R-tN<+47NFSqcSynK}f^YKPN)^3l=*I0NLFF((M!QUQ;uN>tk zqJPh4V}J8=-{127_G5oU_G!jLz`Z5^HL(9_yfpvA{Q>-k<_}7=@uBe&&O*sk_ZR)4 ztBbJQm-4*YL-xV_sihBBY5pLxZ;?G7PvwxjN}Tpg{VyOE$sV5<{l8Mdry(!ue~s+_ z!_@zW9sNiDV)}3BpVt4mkcZiST|XoHE@#WQuvGtbf7Bl;ElRcepM&{ZhT z)8EW0;_*A_KgaYpqQALLd*b#(EJ%L~I14*43CsQloc+Ca?EY&0h58G1Qi52W)QlHgR)po9Lmp%M--kjMU+na!+I)ch z4E#Z^=??=+J}*drs9>zW;13Edf5`3Ql<#iddGd#A;`jrnKO}u2W0UEe{9ztjB8p4- z0V%cF| z#R?YoRme1dx^HZGpg-i_k-$IM7rbAi$l(4{^EYH)sfIr_?F;HHs$24KuGjiQvx@lo z-;&4J>p9JSggjDBe{1@iGG#wX9)`ai+kVXc%h`td#N`8jr0IttkA2Gxf4jo)w`M$W zZqWo#wC!gMe;Z#vQceH6oLF|~N0ow4^Yv9BQ~sB)uZkV|@jx7ZV#>qtw;^-&UX-~9%rTwKXk1_nMn167*3HC4RbNf)^FV|P|6s+bSwZ7#3^sMi1 zP#FSC;7`ZuKlzX3{pa@+Eq|f=&j{>~=369gFSOpI_>bNHarjeZA1r}9l>ap4q5046 zj7uIr3d+?pc?E7CvNzNtuCInZ(EUrzesb~jG*6blaPDu(f0FtNd654Z!+$D$FxH>B z#FE1viWRK%LGzdQjko`#dr0y6AJ1RL_TSIztNCQ2w!bBRNgl`6hcW!6>c3-u$L&G& zSK33Ff>r-D|M;=-_Mf)5NbkM><${1U{uK_mKQy<9G|cY*82(Z9p8^z4f1vs+`(LGC z)qmv=xj(2HXaDPHyNyKs&w)Q6;xe6ve~6hc&GFaw`{w?h{2|>hv-_{__apmq80~0~ zd`$J<@P{>4f93kpg5`ch^M`Ny?%?Fk%^$-4X?u+HUi)wQLvy`ZtF1R9`)aiPcYmn9 zi2SQ9pE`U0&HaxX9=H3i z#{=W)hv6R=So$FM=N2sculdK%kF)>NFo9Bful?8e=jMK#{3C6z?f&cWz_|T4{G;FM zujs$+A6+6Iqx@qB#bbIL{U`r!#-sH3FP+!0688@z4(I-&QXQa?{?Ck078Q|w+x?qt ztS2Y2D$#(CiTOX_;*lN#~i;Yvk=Ys-MHELxLIp7yZ@F3jm(u#N&KR?D>II{XA4W`5q$< zkwC9MlwQybb5`Evf_J}oe8e&!hTsx^LSy+h`Vdlpjuh_8boRUh{!GMx7(BKv9$di5HA{UOBP zpKMe9(R>-!aPeBl|AZ_~Z9~`knk-o?+iV#`PVxA;wiJ;UC9uKGv!JB#w|_&dpGipD>+S7z0x`zf%uLyDg@fJ-tM z6iD@t?%zA*N$r&^C^`o`_$lxlws+P^HOeW!l(hpx!CA&vpseM~tiQl>TSQ$F>ANu=(s!#1Z{5zL_v(BB$W{{ht3{W-P2 z#2PQ+7ydn<%9nC^+WJZSU6F6cA4or-pD)Vti03~PiqS{@n6dD`#1H&qtD^q^r*GQ} zwO^WV$9rhJp?^b8`gNwhIs7H@2YqW+^w)r&MT<@O*#43DDayC^lQiD2kF%;i*C&TR zB>tx7+xu_g5Bl|rU7pTkSoXp7W7%}GeOup%U+7nls!#f<`$zb<82hmGi};_NZ}0br zf9Tt=qDT7T@R!m)?D3@jK)?D`dD72V|ETL{eZIZFqVb1*y{_t0dk+7|`ycBcb^XNn zhE;viPklamC{$wlcZWVneLbY=3w;e(`YiO--VaHAeM6Q{ps%()M`Y-g&9v^|X{h7eq`XlhRJ_x)$UIK5+Tj1^aLEtTU z#o7NfRkX78)L(ht+1%fm=OsRg^VJ1AuzscU-Zj}`zewv@>wfl#@BeUm@#85!3gUhg zPqaTKd8XUvk?8(1od=+J1<|X_#`(vr>GVVpKmQ2*ZrXF!rvFCl`F8Dj0={2L&GMr? zwVvq@rDc))6#P&qJ?aIVUV!IYP<|=x4`~7J#;OqTLF`XZIy!yietur_U?>gdE@8x< zn(lrCg(;!@OXMF@I6WUdFN6Aco`k~%lKw-X3=Zc?d?ZxH=-~>Uczgbo6St!UI`0WM z5}LsR0T!_FdFf94K|HS#b(Y6@T!@PjZ~tH@Bjc0_l5P!+lqcZ^96W$zQ24h zSjP2N$>(6G5X%*3`HQvk@$Vrgt&EZ zazY*zI1h1`$mH@W_LKbV{#;_zuM~CZFPG;=b5~IiJu-AQuVPvEu+nu>ObA@ z(D~uk$*T)`b8x@st{pZ#zn?a^Hoc58(l49dRE^#r_OetlpX8>~c{RXk z1e-KvW-6QR2}2)T3~*U70vv#Go}0tol1d2=aX7&@_prR$-hp(dc>AbKjIl>#IL+x z+8@fyBLAk~heAGKUzR;h;P3puh;p!SitK4xb-=^YFe8~z|gD{tE0 z!B8&zr>%dH&@2#k+G7ch!#eT0e#1XtdUE3PHGNHPkNzpTJ?i=ne^RXDFE;2Wv)8gI zPJ4~@kCTdZyy;((+3%DoPWv_e5yqof$Ls!v&iB^9Je0n|o-;@vlsyB^r1iCeVb3Lb zGc%m_>@O_W;Ic`T5=Oip<);)^*?8FZMG5U2>oe2-&dI)Wo%TJ?x0CZr^ZB{Vw|Uz6 z)&3BcuVLGM5l_t`=dI*}=Toz2y`tcULV4VN@+|w#=WxDd-?)9tS$-wBW3zV-R?AkMR&(&l4xzBlb#w+GmdVc)YMJ*R!sX%MIV zn(K#2ejT6OKJ@u^W{GaUvo!mq`CiAH^LaJ+m5I;CbB+1998Zip`L8hO7i;r%B&6G4 zB*a(Cq5)37)|j75N_6`*{hvO6(|La9{7?2xw}+Izz@Bpr`<`gb->~0Yr+v?yf&(g! z`MbQhTEfDemwt39#+YxKXh?c zzyUuLx`^BJMV5U};_xKPz6&8f+nx{Ne4!IQC;L8Wk57LH_T8h|_eHw>l}&no_FbgS z$7J75dIv)@lg!^@Da(g|&;NLS%g}$_zH7>Ke66ORn4f0qc+);&?N7JwI;?-4_E&Gv zFPW+1b^D2gV&_+Vy*&@>VJH3h27Uib9k1J8B&7SpNNBcU-(_XGeVhJoCRWqV_I3Z& zADUc1w?CEs!oKlB0ZP{T{9^t*k-a{zDK0N`+H+k=wFb}gS4mjdciGHJ8xQ-QeBS48 zztg@ue4U*CbG7d&S>!=g{}DgSTH=7o{#w=k_<7nNPk}g`_F6=5PZH~IzCV6~x32cb z+#X5(*7_dnZO8K-{h=)5{*za`pQZNKYCRo0A7^oYl4Y&G#s1rCt-n(=dyKW$tSsF> z_`SM4SK;=Itv?4tH7QPeI2f9j;?&=Rp^_A*zF~b~*tb64!+x`z^l5%`?hoSWzwRHZ z{d)a5nteq=a}8McKar5RzpgXpbKT!WLb`u2_an6b(Cz?1 zFQY>ue&IYHz~h^jTv7_|?e~=wQvi*UkBR*{y5sZ*J-#-MJ~w|r_Kvp=ig*~_chPkB zabBC-GtNJ7d+vAIGwj#8pXrEyN&Ab0su|e}PZ-Zwd)DpO)K}P>5&x<(>^WIHEGtX5 z=Nh;>+nyt#TFu@b3|4z-##H|5V90OS=TxnHBy_2i4=!9^FT)pYeGYZA+r#<*6Q&KD z`&qId=lU}esx#=<8}YF6DyKgpd)DjoXtAKd`CDk%XJwVse<2=L?M7+Uf0}P~dORBK zpf}&J&niTt?EXC%Dyb%Wv+3#niQ+Hix_zdK6}@GjX@1_C6(1{`d8r-m!v46ryiS9? zl?x;+?Xzm0jfZ{WfqU>1FFrQj{WH$nMfSz9KP_JNu0?Ptr4cvYnluhQdFDqd~Rm*tos?C~Rga?VEwLn$fxd^uB_FXOETb$jq?^Jyex z&WH1`;B&UG&li|qjrFeS|C8CvrIot9=<^Tk-B|DH^GUq@ras>-sC3#>B(zYopM#+? zW4)`dSIJ*miQDU3!@epJ{dLMmw@>(&B~5m@$?>6{;7U-L>F@h8BS`f^!e+8 zy?xZ|&5R!z_BJur-t2ggus02cy=n0y*qe?2U$HkWo>XxD?d@0gew^Fe3C-S&cuu7e z&w2m$c42;#GZ3GDYK`^chkt%E<2i=CT^ws~c6>(In+C()w0I8e&Bp()*qavLxz>06 zIOn&3ww`=K4sMvAXOA17ah#8&{j(mg!Te{Pzoh+~*uUBF8DW3+{m{ugJ~KICe8$|L znem#*llA$p#8_|Y`!U2@wDl#{bCdP;q`uxvzFw`??yn%;W9$d@^{KgkjTgVs{>wSOzH;w}Dh##{9L&B0Km5pU7&XV7_u zvCkiw@zhFfKX)*MaDzi1DZb#`Up^Qt_tN;=`&qAHZ>Igm?)UWlrnz6#;}5atePZJq zy1knF!`S$bzCP9Wdk==H%k}t*X&z^U8dqM!oF%|s(6dAuk#mQQTC<9Q;zvoasJPzecATJ*PCZ;dpj8V z;NvO!{B~h~r`ykmf4ym*|1s8^AKrM1vM()uGUQuvzU}L@VPBY`l)r*~ecJgCfkG`!L2!^!f8byu=x=ybv#uYYVzSf5&{DBayvn@sn@(@J4i`^B6x-&x?aU zzWxl@_V>RLKhfvI3-OZ+@skhJd6buY3(vQ`Io_AS?X6j}w+}p?qR&?s?CqmwZy(us zN}~Bqi?5WOe|vMh4}{xWt7dN>czor;{Kg&VM>c->k&Uk;vNtW>@-g51an5h!j<=k( z?JaJ+rN_p95b+j$zBA)3#m0G*Qtf_r?0J-uBHc9 zn9uL)=N-)FiRt}U=6Mu)ow(zExp^L?u1Y_DGOt>HUg**qb^b)$FZUPg&kOP2S3WP# zBdB<&7GJr`SIhZ7kNdr_ZErlj;&?w*Y<#7djqmv%{rOd#XE)BH_{aVHk1>C0&ktaK zYMe)zrP(jNABUa~Q}JBbvte)Md6d}m`%HTtNPj<0?DIP2^YgLKx9j$5z7Hq%JdD0R zHQ#?zndJR9vFBm*^Cn=)fPsGnav z81iZB#|VB8MYrGB^{sAS<;HqYf1VZZ`_k5rkx-^VKh{6!_GPXIb^FBo2etKMBs4*z zPxF@U}X{{wmILyf5YhnZLAnNzM7UFZI4m zWnagf_LXhKOLE8g{7yC=cX!$seuz9EO$P1J^QroN&9pBuTDE=R^%di?ue@Bnef@oD z$?Z$*ChYc0jrprMSD(L3{Y++GvHK(4zWjK9kyC#9eki$p#rg-`zH)P&&jZl!S32!K zxqZdHj|BscWX)gZ^E-K|Q|#w)U|+?)5)Jle;0G%C^QywWa1NFJ; zeT`>*??fr>=kIef-w*f6y&n7hu6V!WkAhx$S_lbJf4-DW4tb(3Knn`-7wzSRJRXAa zzS1=MJ&vq>%!3JqeorHl;r%eL?DaD04`cWhBK$PG*`5YJQ;PRXgn4Q8eu>F@rGDLo zY}C&S&2aPn%l67bGZy%Pn&B4h13%+xz=|K>X9g7h$x;S*MnB*(S$=-VJ);NWVGQ-* zFA765JnK^dQbPXdy~Vm$ho;Yn|oU{7_9p)=p&t+6{lvUFy_6o~)wJg&P2Zufw*CMgxeevJp}`~+ z)W0vpvPhqm{z9M8H3_5qBe&r9T5du{25M~0hCX??{&`5Bpzp=<_pp3thdm7MXCeF0 z_0i+$M|GutpWIvJVWfc6$l+}E$-OmzOFSGtolU<_1Mf`!;B~x@xYSE9>0-~;H$2qd zC-;_nSXkCCr}|}pW&N$`%onT@@4sg-!_#X3&q)#cQ-U#92CGwLghIk|)8>Lcl7#1b zYd9DN`~b?=ril0v!Bl@?s)#Ts`0})Rt9%Zd8mx*|aTtG4erhl^D%+QEbyT)5;ki-S zzJ#l)L_FE5pDNl{@WN_wKUKk(&%GR9dE)$)`{WK4{KCHsNJze~IdYpfO7DNfc=GWt z3#IY#j7ooi_*D+M1NDBLMY2EmRa+*;H!ZD|@Z^s$9&Wt9pZhzAhTuHBFPi+7djxpy zFVoV(@V?aq!(Yw_dDCb-z1$y_vOD*R-z!MHC^w5eKb18NI{3x=K6HLdpnu8uO$F=V zFCo8G=frQ-82qNKF{ zy3O00ke3_AZGv;3^!uctSit!ilqQ>$lwXe#1@cwf@V4hHZM#y&s>A#Qk|MXnY zo0&=UIM1>_KHyb+1g6;`o zeh7o!8g}Oj2LJ7MaekyeWb*RDetf}9{=QM6pL4;_glsXNP<`?@;{AV|B@U-VCBIT% zg@3g1Wo1I&kt6DRsXd-J>`tNaw&B#&2@cp7Ov@qrM&j)+S}W{D<+aWxKHQ&)I?=}I6t4iY?{w%N&k7`<=t+5{gYYa)IYMv zG8Wd_=j&6xK7@WX>i(Ut2Tw@(%k?BhIMn<^{+#G5eI$8h;0J3&|1!C~=JE2%-X>fm z^&R?9ivE^!f6cuz$p`vd!4Ap#a5}+kdSNvsVGq;Xn!ZmBPKY|~hxCtruSe-0$-5js zSfu3b3uYBif03jT$F-%0(-j>`T>IE(KOh#^`JlKiQ>)F1cMU`n>`Z!)u-;|YIvG1(gu_$~Do;`cU4 ze+nrF{B!#~sn!qK@E=0Im45ny7hf#=f#7%1+5+z16rRSjkkxo>`AwWG?FH?v;p-J@ zOWA{_4^n>h_<=%IpT>vwcT#?kckU!PDy=y!H4!I^YNNY^{GG5w<+wb)*j*1dKcqI zFStb_d6hz6yom4f1b8m@uansct3K@U5@|nDzf$o&5mjEScS)AE{bpSv{D~M3tOrVj zy$X1|>w(zu_ZsVgloaWYc>l4#)z$-rh&Rx9^B~$2Ij%*G=KSyQ*CaoTmxuYi!rs!@ zqP1eZ!i@my(M4PJ^#JK_25Xk($={~&-%o=`xc-y=P`t(I?=G?B<)!iRVk|^^!aq_( z+8(c5+^ygV^wxm>#HqIb5c-?Kj>-BS;Pa+A?GN@gUFyHZ53Nt7K6~%%V!l*%O4d)| z{@Ks*eE$S`?o#0&)p&aFgKU%tegJqjRp_t6d#4M3KyWGSIZazHO%0moQ`r_rkud2#x`bO);I!}~z z#~S}e5#O@>JM;zfTZ$OJlsNk0&^OXA=!=I1MERi(8-_nhVTWuOe7_ph}0@JXugiKdbQ(C?37J(~ypmcI{%_{vuuWa{RqGR9Aq5sSquKbpF8B?@ix8)#{OtB@Wei^|1|b*=^rByN%W8NLk@`e zLA;#k6PE8MC4GW9ecJz}2M9zG^$}kuc97tuXngSgQ=VM+bABf8m1xSO{s!gv*Moof zgYO1Yc)SAjdA#X~UfM0o)Q#Uyllz}&J>-e^vAg$q8TYqv6%>zpqF27|g{nN9&7RvE z6?p2==`@swO%smF!#^V)$_3ivM|>alMf*#>{viCbYuJ71(Wo2algh#wY$%Px9>|Bo zpx3llSg-28JLG2mf9CWVgM63l_4Cjt!yG`ruidm)^+(8y$~TWuUeg!IQ~$m12lV?T z>U>-|<~#a5jOXe1tX_A-i=&>Vy~V5t`+eLmiH%3A`ZcmXtT%S?Yt?K5^?viiQU z>i;}>{oJwYe~0xaYk$&M^&e)RNmk!KR{i_f=MvTD{xi<|=QVl8j_2pcOHY%pNpCqj zG+ug|yiIz!?BVg!)8uc`dv^TvwDBs0_<3_`kH*r z`abrJWc4+9n)Od6udm6^tp7Q7JW>1aNuPVzcg9Okldn0x3)qSA($nN^(nFYKy!15r zoAjO>KRsSeNDb*{gLGLHF=u#Pq3dm+W#2l z8xfyE6d(JM+CEQ|wnKcs3jJ)}d!M_Rmw%k^56tqkpX7-)xx9Q>E8n#DZT7Zm?}YJt zoqqwa5YC0a7bI{&0v9B3K>`;fa6tkWB=DCb0lHU-J1Hlgz)k9*+tXa^$yYz|_03;? zH~L$oG%O8&yYBig-x&VWpI&`uH2Npksp#*bXKx?5eI)w*Xil{B-K*c`z}cw!_jr{4 zjSNNq5PkFAci+AFmfMi<=a1ih`PVOh{FkodzqvB{;A>B(AAbAoKfU3q6&h#P5H*@t1oh9F{brSN~z}7d%_uy*;@5>F{So1ER!^(Kr8V z&&Pp^e*O=8vk$)=ef~F}cm17YL2&TyC-!{&m*KuYe(8bfGfsW;-Dvd9uiYMfEE>H% z`o>Rx9ep(#{qAG0{^p(NucL3i`OYiR-M73My?y_0qyO~s+c)3x=5KC$^_5rnpI5&7 z=5K!eYyA25fBzHRpKxV*Ge-DIzJV|3ZTJr6!Yvk8W@^S;9Pjc}Y4G3nGHPM zoAJ0V$A+LGyClP%m681xKfSi!?HX~rm^?TTMkz`Te2Nn>REPQzqJZ;g&s$Y`?Pg}IH>Muxw zr!Cr8^#e)pw8b5({_-UF<|Oz{N${;4|HZnuSm0s0gZI5Cj_qf#0DY%K_{OZp)P@e0 z;cDuZx+!oC?cNOfxL1F&ZiIET(hl-_(GyV?aKTH6VWzuiaq$6xr(2Q@O*dx{{e~s% zN{4>C=#HOjMHbo&pp;0iCI(IWR|PVYr|VpiCEOyUqG}MtPS38+doVj(58aI9-P$J{@)&a3syZ8*n7m zz#DMHXg}gfQr@VKa|yb94LF8hgMYW#Ujue$==9x2`!W49+K-Ws(SD44jrJo3eBRrK zyc=gU7T8`KPrAkP@~9?xmgdvjf>IlAV}-S|E)<@MF;T5Pcc5V&ugeieAjnc=VxrG&oKk zojjxm(isLPaT_>G02wI>a8K{7pApDl5hzz8`k8@D7RelgemI5tAkR@SJ`*_UX9coY zBr7rf34sYLGQpsK8lJr2%QS>8I=SJHk(f!HIuK4Hyf~GK3JG;UjiR9gIS@@G2kkrX zMTy%NGqKjb*8jvh;-K%8kEJ6H{21+vncUevmQfgA5jDkgx@P|^{ZS)i^xx7S2Y!t9 z14Li95v~6*`sD>{H2FlV5z=kY(k};ojP_|I!EHq2L6<>{4(0}O8BJh`c(8OZArD#~ z$E|~w3>;(Z)PdwWsI?zQ2gmkPR{Pd*F?|dy9dzLJG0@tN)j_PGVS{Q(9HWC)`wsa! z`)}!>13yOladc2?KSl@h0(mTg1&CW&iD3gu2c7sa+9w?}Y(Q&2M*ouaKe_%T^MP9X z%=7`t+PCyCq5QS>og~QXZyPH9Tevf!F47eXZBb{a7(y&+}v5Zfxjjqb90~`qLeF zVP{~%PHrb&_A?<~=t1K8Y8-9y@MDhl#W*Ieuf`!!ec=ZZ*H?C&sJ^h{#PyXOB&sju zleoULlZ5gU^GM?QY8{rSeIeh(^`)I8Y+uMXaee7$6V?~<4DLH=T|mv}$U z=ZG=pN?MEX*l2qP<}-VJ+gO>J%0J~qh2@`1nYg&<#vO27<`DsquTgLhLAFRkOS|O0vop!5_;V8Aq9sgbo9cOLR`E<3UTocDLC-&PYMY= z?fH;`!vn^ZLR`E<3UTocDLC-&PYMY=?E8@dw_92x#Py65+~isLJP{A?&Q9Wa)49bN zo>!AXT5`Xj$3_yaYo!$8t!W)=)l95sGsgC+o-AjXF}!M~=v_junIUFj-77kDER*|_ za3Zgad5!xU-RnA~cP?I4m)>8hSJkEPm+Dn@Dg32+Rb2{ysa{oIZC&uHc9h^;ys9p} zzf`ZPOW`lotLjqtOZBR{6wbd_#ZjuUqW}|llz>spB;HyTyF0#?O%w%i``mnYm^cc+ zvj3OYsu{^+uO3>@rmOYr7;9Asg;w&kH;oqoP_b9q+s4CVKf@zV3D>%r@z%N@wzX;m zfoDj8>qxvEK6*(~;t}My_)MWAi6Q`qt65UA>pP_o!0Gz&MWD~WS4HHRO93}+96NM- zUjkilldiy0;^Pr;COo2t@$oqLDW&MZJEd?gGpxQ!JO5ra;^k5Zr=Cc4?9d%jNOSXE zrkzNOkN0sYF!C364AoKJ%ca2gzBb+|g+SaGYeNT4>O1aPgnf7(gU-elouX6z@@yz> zUm((`?qO^~q|s&G=V@P?>T2JHI~`DHnM4`+UPI(Lo=!=pr%@W|Nr8n08T6s=+Gl}R z!0JkP`X1*)&x~+x2dH6isd3v1Zr#(eQ%6`M#|H4of-}_sbe^_Y2)Bk)`32&xfXn9^ zp&+ojM<8kndTRlATqZpt?_Q9NXB_7R4`uvNbw^+7*Z^z#Huctop|U-r(P+AC6Do(yAWlTP%M8i~mCcZ?tds zFCg$n`(f36y!MyVH~VLuE3Waxlm7|peV!-dwjZ$SGg04^uZ3@oOCRsTlc`_eP5vxA zUP(yuG5U{31Y|l9m;MQb#%o05)el?pZx(o?eP-EDSm2HR1uXnAfv?f^D{SFU34G%5 zNr}gw%|9ef{Wtlu`qv}y3HduD@CJW&|4ybjpJH<1WPi+aPYzEEV}t*jO{dUkJ?9#s ze!6D+@Klt9UChG)XwbbKs-;S2I}+>8Gx z$gkN)FK8+g4jOM8776bO<}Yf@$+{yPE@!^z9pQ2x|BAOmAOfF40JRPJwHjVQ^Z~vy$BQAI_PBhajVY++} z?VXbJb5lio0SSwXcdn1G6BhyWPV`x8BKoe@PxM`_HNAO`@7uhLy6U~m%kHG>7C*b^ zAE!P1um4!ElYOJ|wKq3;zC5dHYR!L^{P2OCtQFlEb5Z7Ev8tJEXxg+2=HP z2Hx%Oq+iDF(%|Xzw4$T^X$&tUbHGzsj|ThM4h=5FZxcAnm#|G5dEa4Lp-X!5u5^k07S^W3~^@sS+aAmV*{P+&R z#4o{z;GPL4{s=ycA7>z#_~GSC1x)P|e9A9iL9dXtP7|=8SIACI7qFmL$O1D2Ea(-o zV>1OT=oPZuasey;!W9Bm{GF;4u;MRJC1Azhp*aFp{4w}i;@_fIBVdbOt$;0h^#WG( zny(RXv+VCl2{R=R=poVfNw{^RfO93>^Kk)Bl5n_Pzp|#ox(q30U!W>N^5f{GI)tfE9o2hXPjo9ePT@n;^eb zmzOpFOu(%YZiVeS$NQv&Yoxpb!veoR!p9^Wkg)$XfnP4+LlQ3Ma3S-*F7Se0Av+{t zK@a%<5qLolcnJ%7(C;?{UeJU0mayXQtb`SRt#3N%WwVnKR{S+1ys7n9!irw=sK6_F zCnc=<8^G&NsD87OhlH7u$Dah=C*ghx=SrBpE%1{h+#_MXgwIM?(1Xu?$4L+RE@43r z_;;Q3fS0hKSIk<$Eb*)OJ1JqsUjWky;T3u3rmcA39?BB9hDC^#gn3=^55$f(-Qiib_O<;g*;yh#m;7!22g&YBS(DwS4`_3 z@#|X}?uW>yAHh)lZI)R4nljH<@(IZNEU@BXc^VA@uH^aR7m1w6O~}v7wfSk5yCV9r zxv7qVN~7i?-+yS510%jRBY%z~-!Dt_C7=^SV4N8=ybwW53do#x*o`wub+EEXcjMLn zp{#ekbtqp{G3)0e8$%P5)Q?i_kQJy*4nB$f%F87a&OD}{qklFjWPKbv;Dh}0TMC%-&Z0C`o+c45mjG};{zZ} zc>!5YU5?563Rdl_dTRV{M&+^Uha>^2Pvy+|rl-)lXnEVJxWG-Lmj_R8ed*&gw#?82^~e`}Qr z;;hHBPU__pr0h#wRlV&E>ozoPZ*LH3@v6qU?mg}Gjoq8;*Ij>OL-+c&jU9V7Z(3X1 zzH{UCH_g9pcU!~aRf{(??r7h+tZ&PvwcYJK>(+O#xTUdU{@UFQD{fh{sn&tJsjp}2 z&UL7__S%Nc8?V2D<(aHaby$ z%k^6}&cAhQ{Wa8H?dF}^m)_X0y1u=x?fSNcomRVT4Yj+M)NQ}6t#P$1vus{>)Ap4+ zde*M%US42!>y|iij`i_k&cQ>xMWl87q)jN}vY3$sxcK71VZRq3Lh24vH z>|U~QdvjZ3rZCZ9~``-Con>VfQ-c+~t)~$=3a@(+S>#pl=T}9)#T=Kec+1^d{ z9r0;4ZNKHl241Ib^-8{?yZ zzH`Hlt#M_$c-69|)wNrimhRcSvwP2$CVZe5Tb8fv>DakpFLZLnb+sFA?O49Dd-KLR zhfZv`scqx@ZEbOB+T*o&yzK6_jcae(vhmU+<15-Hdx_U~ zOJ79$$@Fd|^?OtOmYX-%E$fP_15MlaO1UrNW$QYV@YliR1Gy&}e`Obd$NA-T6uGO(}twu(^xOcZQivmNq@HN-3ouUslF*mxo%mm@#d{{dy|yy zk!4rkxOro!JAvFSImy05uC>Wzy0m>cY$Jibtx@*5sS&o;9ZJxyva@AGt8L?+_&2dij+Uw4ZzxKMM{H@@&*0gNbmQ9Itb*0lry4w@Y-Ak}; zs@VdaUski~R=NIwU)r%ntSN46r1@?`O~`bq05JcE{_3#E7n} zYU}B}rR&yPyE`p-oq&65>lSQnsc-4%+uPIG&IKU_vb7C-bo6!u(Y0&)^nv&uZMferZcx%huhYmX5BSM7V`MR!M_{mYQjo`)~WiC(xJL z>RR3-wYhZNx)$d$N?gTRr98@~_EA)*jF%LDs&GNL*GiwG4Gg9AS(gMp?f#7S?@~YO zdob$@+5adgQ~3^FK>WEcNqO3re#TKgR|Ro!Iww&;5tI&lh~5@E<3o zET59HqL|kr-hS?={~fh``dIcUSEBkTPqnjDS$tojXe`Oj{j~cFUcx)KVwOxC_-h^Y z%Z>WPSGN=Il?7VI=ogAmy@%gJJxM+0assrx?7w+z-9+M4teGXpJ~p3NayjG5)q=^# z&@Y!G>ZA5;MA*Z3rg{^1anec{Ixb$^T9l`qYHad2+ErLlp6^gq7Qc#oM-WG1MLtDj z^mH%4JcoQ#zZS@RWk*Uc{tNw5c}_|nLq5?7$V_EF*$K*bL9FrUP#7p){#6O`*Coh* z7Wu@_DYQ>X+24&SMKv&>(@v5akybPmaUH7?J{{;D{%veTc-elZ!JE%ypSFG@N!7YkVB-;4Z- zJVGt<@0Kdl5BikfFX0y@Kab%``C%FN{ejGXS>`X7eoi)P^)Davl&(>g0$J}OBfm!G zPeHzt4P|{y<`*Gfih}3tbL7h|^PIynpW2nI@*EYfr*@tB>mdh9L|^enFzGoZlgoVW$@7oSKPX3FfT@ig}e-}9NopW<@f_$YkL}!ZVI{&*sDih`7NR5>aJMx|7PdW0P zHk-uCjy($0oGT)T{CYev|(%i1n z0%;^R@Mg zTXkP&sE?iNj)e%dbnWWuWBgl3S6g>ydtVFoEuF!>mfk(>?VY{7ta>Z{jEm&1Jv#-n zU7fuw`Nm0ZD!O)U+kFnDZ|)5CcJ1CJ#n81i2$?i>_6bq2>Rr40I;)#jUq7d}udV$i zQEC}vwXUyPM z-`{Ux-jPc#jnIh};rRbXu5%PpP6rBP6lb73=URb~0s4cq$+;K-?3|McKMNmPk_GT_ zrq68=EiI7iHCm?A6sgt#YAyD!z*9*%jawuR$#%{56s)dvj*99NZD%^B_?G)?g;VV- zIE*ipXUg!{3ct-GpItBN y(>~RiO#bfxKHm79yg^hvD=SksXep}dtMPjb4ac;Na?frM^-oB_5R@IK{{IJr)Hl`u literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k81128243050e8617afaf6db477a16040.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k81128243050e8617afaf6db477a16040.o new file mode 100644 index 0000000000000000000000000000000000000000..fdae94eed81ed8838f0b1b5422bc2161a482645a GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10=3yQZAPh6{Z0i9+G9V$45Nu{eOKOlnk3KEQ zvrc@U{q*2Bbx%zQXB zvg5$uQ%^rVm^y!HXim#_!WZTp5AS>LXW^%uZ-$#sTnhKCeh+j7q0qsDgWFFW7~D7i zc)0xZiEwk<>2TkHH^U|hvr`jKed*4UKR?S zI50T(@<3+10bflMW9xXot`K zt$uBIKkKJ?#I%7&RskP*mi0&ene|_KMZfw^G3!5fAG`W5i&_8qG_XBV^nA=r=XpnUrM z^{hW!%=(!=@Ceyd+5hslU@Q3YyYS=pV6%UN{bK&W4qrY5yBh4+G+pR}{Pg3B!qBy( zqAO&-d-^cz<2g7F{FuM!?N;lR?^geV#$4=j13E#`nWj;I56U{hR^jD9=R0uq;FW|| zI&;ymqsk6npv~9P!772tC9IB_r>EkoYs@y@rs+1u?BlTH$zpRdJ8r)$^4da$(AobK zGD)&6_2TC2e-u}J2TSy@Wh?<4O(Dt!9ZembflhBd?$ourRn+;luA`}~F~}CQwRid! zv^}<f72 zREK|ymgGI0zjRq$T^{5Va@_@DbToMbzQ(&pkbtkl7kCU3!Ab#N%U$7Zki&ad2-4`Q z@7h+kHQ;Ua3F}w^-UTKIrdb0m4FRveiBCFzpuMxbv)hm6glli>==8RA=G_JE+u-%L z`T`Aig_&U*E{oNc94MDgZa^d8F0i2z$LzBlde=;0Wm_)L^$aY7ig7Jg?K(srAv2L1 zU+$$L_tKC%EiH`W`uEb1&sp|vTDHhKR=~}tA$R^hMlI6ovB6rhiEvYxZ{V=pDsVOXM$iRZe!J$TSB?8+OaqvYf*ctzh=&96o*&ycU~v z8{e^;QBQI>P6{pMMZhy$v3`1T1VKGmnSoo6qz4%|rFd;6kiRnsUo7rw^t$PEp}ED|+1%dd?rd*w>6qun z%=URZeD3_k3zp1t*U)7_!E#pA)!fqP?&$95^tHNMyE;1EjlQkTZ9eygii*|i>uOe1 ztuL!sTeqTi!-};fD@xe>3io37d~cwk$=%V4^MHrFZVXMl()kKL|FH17gFOMbeW8Pq zC*;w6y12(JvuDKaZ{|41)|ATZTkWt5+CFR-ecwl;Gk73QhAyWwyR6&E(w7UPbsLRt zsO!vas?%AO5l?+m-G**$QzJ}Yjbo-dn`WPr%56$^rDC6r@nIBKGHjKB+j8(WEtUDv zy$g<5iI?348W;@!`O7|Cc$qD01GA?%7OX)LQ-=!;L1949-zGcm#s;8H+Q?_>+;kWLlQrv_Lm z4eqbJY-7L7X1)9E5q22>V!F}E_Uucv^Kmz2r?TU2_Tz3L1A7v9^w%52M3NvHe?_;> zwh%9Rk4^A%A1Wv_@jB;ZhXZyyTCm8#!8A6skmUw#_YHivghlj&N$mAQY*sMA_S-3M z?BMmGY+IAhdyj1j=%B)&VMfCGth&dG-)sJgZ_w~>o22IBYsfVNj`sN z*hc&&1t-CZ+`b74FX+RJ@UDP!67idQHj=~!(^$b@pnuf4yl?RSoN1`}>(L+je{vs- zkUi`U*mJ0Y-M@=ysX)FgsAP?;F{Wcd|~t6mdX5=s!0& zc{2Iic&>j@XJ=m0m3~UIzAnuRd!m2X!^xeT>;ZlX0&Ld}ILp z52e{t$)B-M`nNzoz-I(=XVSU}|Kja)gD}T%wKkdL8_Z^<&mRGn3dX-c^bgLw5{YOr z`Z-S0Uqs5cq8rNuO>F7UCVt>w1%U@26%$-ja& z8}}vUZ^1cpNSqjc)~sm#H9lXsJ^1`Xd(xL3{d0caw&|r=&oNq8Bayw_{+Y0Ew1B=) z)ydxQ56tIWK3{1*|1ymgJ+BFJ@?YRfpZ~dRF6yKEGtmk*-oyae`)$%U-J?i-(|C5}pF_paPp81k`8-JR>(5!H8nHr5N5Mc*!0 zOy3oxFIQ}S^Z9}HQs3x5BM5ClfXG0vSbu4La05zxWtrur{-W!_P$4VLnr)JAEPeIv zLOyU7O=$hf`7_9GGx!_H@38oroySi*$$yI0GrPziXnm-s^-{D~@oi?%!RZ|_e{*ns zkiVVN^wInYZ{_PT=HDvVPZTl#tnrZSdxi+)`dttlKc4K*$49&wgfVLfwVt03}ezMdu9DB5uUfV_m~ z>s^MQk-P=H!1Dow|D^A>CG-5)!SiE#ZqSiD`yV6OPVy(Thx~jzK3{m)1igm#P`9S?nZ%4>GU{Mxfc%sCOTlnaygeg04s|1*rmqp7fVvT%h;++{PeMJ0 zx2$J;eY3k{JQ>r!D;U;m?jQawv_FA=?4!v^MfnHj=UG(AzOX;?{Y-L45f91!Q)2dq z|C{IMLQc=^Npk}FEC^21X}M*jzcc@p)~{)a>~e(XH$&6mU&)zK{~B6|{XkA8#76=B zFzq)S+<%gz`;)AkDE)cbZ#a^07cvxpw}++eaJ>QBgT(*y1-H+qi{J^9ma z2VWmCUK-KEpE8a9mK)4+8`~G`cDrf5aeD5LHV3RD{1M|*`=~#Dn8uz7b0mwLjieEO z1-+mb?UN|qK|XC8cmllg^)1bX{LCHg5Avr6rt4EcA8}zm4_Bn72eS34RG*9qgz+(6 zP8RBXzlr`}-%N5BKg|coTM*35jK(u7Kg548#=Aa0w0 z+^Ca&&n-k9=>l~~@?4<(ni`FrRbB$jABQ67l6i zg*l$Ri2QLkOC|q?KaFE=RFFTVQVV~H-!9_G7g3{PSTBZAMI|Jk6!@qjLk9Pclk9a4%p1?Ri1b7|u zWw!%I1t>PG7pE}(Zeb69{rltCuPENW#?SYZer{PF?AQ7FLH@?q=QAdITJtkgJ_7q4 z2rwil(tg(SjLYD!>$rpBA^007_hY7fhw>Re2~7D1CqnRQEb08q?y-i z%vZuk*XuF!6+XW$`3h~usR)1M`w3-#*c0cEnUjHyxG>*Oe~S4E`9s`%WyTo!ihIV5 z^OY=~uejsqE08D4P4PXNuLKSG#^`(n@zk9a%~yt@zfYczWYPM{<0bmD3uEoKcXM&n`o!Zi9jVdhML85t?&I@)@YvM#54!xLj^%Qh3f>K z?_&KE=S4?1vUu?f@gn`-dH-#SU&yEQ0RQwM`uPLlqwx&`{FLY0Z!84;WW#>^5I%3PPv-k^?jQe&_T#bh9*l?m zQ|$Z->;LHe_|$1Pj&D=AzfX-H-w;2hOy&D8wI3fH-%RH}xxuMZEb%RN{$t(WV*cL5 zetU{Bz7+%~=nNlViTxqXkIV@ZVIi^q&c^;bYXW^fXV`yFqy3^G-Z^PQ65D?}on!33 zaiBuwcH&*dZS23(v08#bv0p`d7wXadDDt%l#`7lNGmPg^&- zrZmg`I!(uZpM4Dk+@G=kydi)1JRJTY&eyZ(^HctLq)s20CsG~!uCeU5`F<@bpOcY% z{Ctsw6zi*<9E7(YPN)9q;}+SU7CZmXyE%Ey=Lc!*i{MA>XVC%9e@eSQisdKx{J@l- zd7J4Km4M6 z+}|{Qbm6;^Lj`Ph-&4J=YIon&pT0P%Y4S^pypC$@!8yQRA%GH88)r|s z!XtDc57e=`hQC<`@8g^)zsL5LvAx=I1ahsOfj(CB9A~1&ssONQ9+7xQksjP>;}X%H zG5}4R$NN*`y_#T0y(qOPkzR$V9=le$d9vUI)M%mLG{zNH1W4hiF)FF&hlnRW#+6sW z2=zrT(T?*r&dM8_u^4`DiWN`kj=s#f=nC6{&y(;y)f6O%y;>qzj+S`%sA665+fy#j zi#t9Z$-gLIAYT7G*Cz^3e7wlDfQMq@wU6?@C8m32gLw2&dHjs6)3jtAnvTLagzQO0 z^lZ>+lE>l?oQm6$Y_fNhEg>@Fb~ET*PZD~M%1tB1!AIqG#m7hGb_==V^%0c|jF7BK zG7O1gkj(z6NM;}HQDg}dXObq$8Kk=rXQSt!nRA#tqL)n54CSn=&f(SRIBddUx%U7wCB52(=U z8Pu~4b;47fD^nYniVTl-05)v~*jT6OIWELz&mPjThQN%K9&fQ-0p)PASGVc~b?S8n z>eLnKnnul0_1Hecrqd{esT=HtuE$btZZyh8#?no^T!lEjm0Z9Lb`3h#5y}9G719Xl zcvKb}C)YE1uO-i;evH-^BN#?z){)&HG2rjTdguc_W?G{^i#77o81u7>n(?^*?6|So zbF8{?hTV(dVhtHfXP*>r)cL&_ZeGPjtWfb`bljmmX8e;h%j&LMSAU8ZhMkZ`Ov8L(LE=hg9*_Ugq9x^EB&2TH|ZzN%4d})9=+P!(TWTD zJ9>Y~`8S{CecsOYKJWT9%_rlIO~m?G)Z061$L4_t#bWkeW{i4j*1o0dYfG@#X8i11 zy8`c{!*J}jRT*;x{#l7fe1=6wvX*mn;o~O!fcSo#*#5`*AAK^;B{1}_YrOv+#T$L^ z%-FxgD>VNzw{^N!=EC^`%_mXL*l}PJ=NU%Y4WD7OL^ZcudsYbmKC*#8XK~^oA7oTn z5~aME=l{fgw$Elad+LRp7r*zy#8&qGxxf8gt?h4S&6|<;)9F7sl$Ki7lCls(eh!}h zQwgcv_@lbKT-c)=JmL92FRx^5ODX=g9Ixv=|A!PrwxDagXZsNRSM-Kr!%0x|t97U= z?LGoHKbYrK*MFPC``|}ICZgpWKDr0-oG$LxCVKIlt_%~5Do5>pdA0@cW8=C3-n9kDPBkK*(50nKUwlau1Z35!8 z|A}n(C5vjU?eWKumNPEV?<)EuO%#W0KfC!9S@;n zf_VsOC}%BGOpUMFDf>T$#y4uO=%_u>S=-Y*Cly%SlreHll%5=MrS?SABdXDN{K)|C zvp5QoH_POe29<^Xp{t=z;P1ifh$rgbRAhUP321RL;Anjz{);Shc!WPMKR0w7&whS5 zCjP#T>?b{5v^)E!;=1I1am~VoKYFq1e?8j&=Q#4K_LpT# zxGA^4b$s#((&?6ZNocub$^q3?wI}fHwxhLF%}>j4(fO+8t#Lf!Rm}@)J5~2c2Qjwe zAUU5Dow};|rLL;ow!-x_9fd0jMJZlWw5X-4VR2E*rp4>4w->f-@NR7E+EiPW-_W|T zdPl*!c5h+vn&O(GriRv)ottZ`S{nT8H?)-PT-mUw39$9mgfD4XzHU=(c~kwyn!L@m z<*U6L3wE@WS5&XxS-P@i<)hV2{;KsYs}@y1TDG&Sc2QN!=GK+@^{YhNqL$??^|eKL zzRkSf)wPROcGb5o&jSzs`bA5tw->G5-dJ1Jv8i_L)u_l}E=d-L}?SxV(Ku>9!4$rgT$VO>6Dqn(a45 zw0)_IH8=5K1RrjWi1%bF@mhG(F z1itG_S3cg*UC<19TcE4Q>x&EWe6>aX`c@z8Q&V2ww(jvYWP?>T&ECTGYl_;|<~6j| zG}UjeYA@d8t*-W0EpI7qYA@NitE4ntdwsdVS#cT1JuFb2;{f(_P z-Oyjzy8N2Q8&{RLY}&X8_O8m`)V995X|u4`_M-Bw=sT~mw!F(~7jLou@lDN39#Q_L zu63;~&4Icd%kuLVEnBo?aUQr|wlx1??^f^DrH%DV9)8%Hzcg=2UY($8Xx{2=@LlgG zRyOe8R-s>2!LOa%P6a=Wa^0Uu$zQx=3HZxf#%Wf1TRMDo;=Q;UKPcn#w&n%%7~7R| zM?XF}=TF?bzVsy+Cx1acH<4P5wa2TNwtKoZEj1eta>4IMDg)x0Yq+0>d*dGSNm_8J zY)HM|^Q?Zv{$CRR(($#_Z#Z8NJrjK+ClGJD$DpUNju_})Q}?T!N5_3bf59&6ME665 z!}XN9x6n-k{G7t!TB7btoUUi}Z`cXX|L|ABsb5|U_*_FUnr*=oeTe}d{Z*vZ&X>p0 z056D@mgrh;f=+dBy}pRHcJ{tHk}#p=gX2svA%{|S3`Hzx;;RP&*eEX5c7pTB!vQKj zP$k~;o;UEW(I`%d_nc`#&m-cf7B<7U3MuGg{Fvc#C!~P53VP$tTOiwU$uR8Plw1@h z39kHK=!>y$LM{apJ&1SuQaRCkBvAQ%7z&L?zvepdKcIa)dfMN_gFk#7_zwXme$H?^ zFs9~}b{+noN8&~FmC^*SNjYfDc;mhy<#kIFsQJ|ldg|x0gsZlTBwTG$OJx5!vY$$6 zGNtFgm;B@DQoN{n^{nLcWeHdF={4EUfHa_jZv;GD4T_%Dp-u%)@#XL>vi^?(%)W&y z(dS_tDksH-y@e~muSmFE!j*gz(4X=hLQRzTNfvmXgiirnwI$FOC44I2#x3J+16*$9 zIOLdw)3`WIjF#i_j>a{@OQf9Czv7KZS4+6HpIQl@Zs|wqi~2F167?ARr{|og@|*!~ zq(6Qg_!R@(C~vORyOqz`5^j}uKpMy@@9Pq7?dRtbZk6{}hJK9ve`J6g<;|4yhsHI= z$h_;ok4QN2V_eOilkkTu@Bst8(Y~)qxRrl;iVhXxXNtJx?@J_HjjMdINy62*3Vy^u zZyfhI3Ac{>D+#xbn=0omjcc4QE(sqsUnHFRQM`}F1-f)SKxun_;pAA8&C9e zjqGN39@J4?b6Ych=CQt``ji`I`E&D9Gm(vU!Vg&4IxEst`9)& zQeUSK7e8D=zhhatwt7B(Pp)BypjruitoL;mcH;LZ>$^IA1QqmXXo5C+OMH(tH~36& zp*>%d{JFcaS=@^8=;tfy+UV!;RaUm!SFY^`{OofY4j%}U{CNVszMf{ge%(vir`kqV!4esVKZ-FTSr}o-`6nucSQs3jowZ#TfpCiLkcUc zh9o5%B}f)Dbadt~y=iUSA$#GQH*VY?Oud~Ays6D6-_48p*r9&A^$z`ISFyl;!uP!h zA~Mx7^>jrhs8ePBp^$Q!R#is(e1gfeAPV=$yE8J5DTInwsy+)hlyZ3#KwkYj#m#dv zAEVFKDBj6%9L>T09zYu3m|Igzmkhohsu+qIUtOP-?Wu2LsZ_zc(Fl!=ub_k`o~$~o zY!eiUUqx@N|CJp7dhI{5AQa!KzjO$1wF%)?2NZ?*YNg#S+mDi8wtvRbz{1poj5 literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8686d739fe1cf565bc7e78a1ab51c945.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k8686d739fe1cf565bc7e78a1ab51c945.o new file mode 100644 index 0000000000000000000000000000000000000000..33225c7267df81bce04721d932ac9e48f5727d87 GIT binary patch literal 71288 zcmeIb3v^V+l`dTUkZ3)Sga9EPMvV|41PDnyEHH+act~I#GT0!vN%RKENJv)?`(P{A z?Iz>FkFf=rKelHkx!fd^AYvRcla-)MGFIX&+mnYw_%jo)Wc~?bJB}cZ!^j?6a%}7W zc2%8os;avs96Px)SuR_g{%Tk4+O=!%s(tEIpT2wZ_AOqIhuO))zRp}^=Q3tZb#p^J zJH%2^GRVTa2YogFk`wg(GyS1d-#o@nyx_|@^CJFU^5qo#64#gU_bYrx)oFHFy1sMQ zH~ZgAVWV&C^Y*{K%h&&okDd5WA3JlZi=9B(zhqO{s;}mR2Hx1ne*6nBJ25Xcy!(&y$jX~gR`fbJov-m%U}A<@Rh@_pv=nt`yRUg;+bU+UhGHPSHA!Fa0}YJvg5ho zt!V4aGAeUJ|K-Q;zj%M^Ll+nS@a*vYfAPfdeQD1O7oIJx2n z!!3~c%Hfw$rnH}Qwe5+&xOiXwI>ev&6=Wa|_6?Oy!KmpYT~$vK~t7~Y-q?Hr57 zHRNNf0vI!oH@v$ZHa6W88olJ3euikZUbf1y2hfB)OozP*o2d1$RnNjEJRYJ=Kl6e& za1+%##n`HEBQKZN5xky+-MqeQWB(ib%3+70(Rc3p?rGJYW7+5HK2L#dqhyb^Erwu= zUx%%JXp4VZwiq7A7EeuJi>C}*BpbEuanP_w+kVHh$;LGym!H!*|%3JLe{je(i?PXzi@TcUrO%-)Vl@H*oqFq0yGV^NjX? zu56(B-#nwI|GsRXWv6eT|3{(G<|lju0d~*smd|=dL%W9jr$31@g{bokXeUwjSlK{r z=CK=_vyk^B^6o`m7s|Zr9XS1U;5&gEfL{jwGVl@Ln}GiX_;bJqfo}kQ8u+h)KL$J> z_-nuq1MdZ%3j77&KL_3k>;patd^hkG;MYO}&G!P|0lXggMc_{0FX4I+*FDHDME+j` z?*_gJcp>luz^j2b0%rl=RSvm*1CVF5`7PktEseTU;N)_1CyYaQzjox8u48`uG(3u>{xKas58}wHMb0&~rhr!F4XK z?YOS?jE)^+KfVj|W39LFM5DL(MAHjJCk}q6=)|GV7M&WR4;G} zbAB+qduAxSyXA?5cY-^H{LkiuGmm~PVYK$&A~&+i)Be*~`-7KGEnpvg@|U%Lm+%hB zSnFMKqVYS+PBeXX*@=UZWhV~RFFP?0^4T(8kI!62$VYO%DAx`y=hq;^-;oTdXY2kE zpT9w$Q_j{sjnChq&&g-&p2O$w(dQ&C=g{PE=A8!;My-z}jJE7dc!%U-7fvm>b1&rF zZOfDonF=6N0c0wGOa+e4NEXr+$wK;ruk*Ex%X2Th^f^JuGdq0p=o|jgmRJ3d1-{31bZJ5D|ANjw`sAP04*R8^7C@#&kZBQQ zS_GLEX?l7Qv6QMOXDs)kz1`HF|7_jI@c96J_MNReh|l%(*?YF`2tMCNpFLtMv0f)V z`oQj+p!@IPujPLKJH03Uqs%jSV+%Xg+5AGriNhsH1Gla958P3XzmxufmWTYK_n~a_ zr_h&(f1tKJa$_j;q`!8H_nn#~_CM?1f_IZE27<1wV5A4)ib#^Obqrn7rw$8ijY%sxt>rPzJ zC+`kieP{LZ+s|(G+vRVP<+sW5TVs^3Ji9eum#>iJx5)CFW0c=?c5A9#exod3F3Xq6 z^6W%G8T)Y!2Oxr=n}+Hmoe8n^xdFOlk|l#=m$a1mh{4CdZ{Pp zJn`E$hmATs=a!9b3!;wMRxa}MV&v~d8~Hq+^K$rfB7doDdv$c%hwlZwNYdBGpq~VN zrKE3(LH`2irINlg2K{TGuaoq>G3d909s*tJz>)iDVNCdM#R14{qNjW)_-~aGp*{q8to^H z9rGbJ4xmgD$|j>u3hJgJCZ61X8L{6GV$@&V%?4h+fsMX~81(g&vVs3ZO!~&L(C9DQ z5hr{yq5n<9sw2M(4U8gw{r8mQ#cv^=efzgN```Io*}%}Eq{Wwi6dD~u{Pyn8L!TWzVd-LpJbyW zH8=IYag2@rXP0N-jeXnuUr+H2y!LfA`ueUN{lDGlMLQm}=^c0_-2*wi16L+_Mu#6D z89f6-*JB<1WM~xY;lS0|z%$tB5Ms%pMacg$^pFnx55WHbS~~ETffqr~OQ&bq~ z!@bOz3HO#f!e*KiaYg-icYRyCmnZvB`LSy)pBL(<_Nn|lPl(66&B?Lbc;~Kf!1g?_ zJ=~jN4IQcR!T$WPzl8oVU)qlY;FY|ztUq;Sd4JlvjhyfQ`d(->1HAvu7aBNsZ0?As ztZZ>0DTMo{>qhdEkT+@F;?o;*&gY}dY2TcYN`Ad=Wb3@L#r;psIe+@8+2^;Sd^6>L zaczGy>iiS_PH&ub{y60wn>liv5Ws*Cj~3|3s8PB%Jgr{ zJfHM~(CC?CnIl0@=vzt1r@H;eri`54I0gJFIlnWpi3SC^vM-I}pg*#$_975cHwtlD3a%H@WpIWc@{`4_oK;uctEGoigjd{|4f_ zi1K&zUxA&@Lf%Z|mEw9mu7SLr{aN_D3V(sTUH#K=UBo|E^(W%GfPcQZ-;e8j{&{!5 z7gyK}%J1nX`zgU+^TJ#Dr{G$|Ki}G)j%y+RytjWcu1opnef=r8k{+8ER#QJm7vt~J z_hC!h4hd2f8O_zk-yzobN-7@edPSR zv%Leq{!&i=Z~lI6|8M`{y8i!oYEJ(fKh5fY6Mj7cKONoU8F=f9Z1i^j9t~x*TeFi(l^(kDBRgDbQAb;b{=ZF5D@XaGb`v~to zKlC}myGMq05#Dni^9t~uk)aC0x11k3M);PIp$)*kTSta4zn^~U*7Im*;Iwb=$j}Pl zW4JCS`o8l+dyu~o*PE!`zLBB%$lq9f9`%5$M}}q--hX~*3E}-CLsJPKIFEeb10zEz z#OHsE3{ArIssC|)$S3ZDn=gua0ltoX#m(^jO<03AVm>IxyikVu`DtxV`f$GZaK89( zzW8vym}tI0e8f{N{zV+b|LC8p-`x5Q56fVIf8}K6WgZVbK2Oo$k*3xzRs~8PQN4)7ys5Z)z05b+=kfYH%%6f#T_#@|!#)hVTRcZB zvk6}Ik3vRS%=(%5F!KM3tM@QV46#y{0EuQ%$-=JYuI|q6x_W%FuSeSi=GU68<|B<3 zD?HNP-Bftw_QLMNEk_QvySSl8;@bY3S*0t?TNlXOYP-EgQf_HF($(I%qpqXtj+Q&`Y;AJz9)Y`xOG@i&R@5|hAMI#r;7la5`ns;B zn#QhH5L%8Ly0zsFEt_jfYUT2q|9X-M*bvCvBNmv{DQTHd&k;bO_oF3qdAO=M!U)NewR0Gt|(^yy2)mB_&)pR!;YC2N1s^(BnU1wuw zT}x|CH`cnEhW6H)`X)rLHH}SJ|Joax8fy@0*K{^@_u!VGv8MiL4bNvP^Wtk`-UC6v zvaTC^&`?j?kuI~b4@0N_a(5isR7-)anQ`r*(|`YaFMDCx)7I2h*naR}S5tQptEoA1 zJ3K^$|9f_9-cchcaJstImb$JO1#I{0sO#)*>27I1!lM$02XRMS_9qoH9hs7a5$d@#_kE{2lz)(W~s?UVq0hab;uBP{Xdx_h{TVw zDbZ(2)J`}O&NrJ0M9^#&Q8J))kxo(`%jIUbv85yyV`1si>|K=qG^(q_TBId!jSba zyBGQXYAz^yTc5h6~yA8eYaOXm}}mR>Mo!poR`@IbV0{|SW5+c-pM^D?%PbAgV;vd}vStnEuv!hzWz`zKj#X)R4y)AgY!=e+ zELH?u=ID1O3u-u|s|uUfbR=S7_M7!c3)eC!zAY{M;u!Wf*`eRJs}0*<;MR9b4`-K99dRm0iW= zeNii8wfOTdp5x59G%uRTpFf8@i50=QGvMqU`0Kk3tr3ql2)!-L{A)V==p7p)Ay%TW zCqTXDxJqJZi{pUA&@RW-64y$cA#sPqwGxLV&X%}O;%13YNgR~;oWvax4@#Ub@db%3 zi7!c9B=J>=!xF=qxxA$k!`eAME^$!eki_s7PVbXABypw0RT4icajnFsB<_&-w8UYF z&q*AbD)?6cCwu&?YMQ{+Hl8kUt&Ot;ZniNTl*X&W#@PasK8PQiFEGi2GkvT|VkR+L zBPuWJaqx&J-z;&h#P)cclGu{;s}i5$IFp6iW&0BMNi6!C z$u3E3k4I65C@=b($-)wg{${cZ5{v$3vZDVf%G=}7CvhonFPB}FI3#gZrzl@3@hOQJ z$9e3k#6mtl3w4R|LOwt1lUT?HeMv0j!+0UKcI^c*Lg@_VpOeQ z7^gn`oufa@Qz7WxpMe+xK^DbX^yi-o`$qE39P#;+11t@n!#;doE%OiKF=ZS64pC{e zi$vFV<@FtTeOq2vied~?29JB}81TDP3CYg4W6@0_+sT*Ed5UV6xARZMDR0w1=xgLI z&lvj!xKbeaLOPpy6q5<1dH5?#V^1BG(cN+2`ZN}?P*A9jvFFl3CygR4z~5)n*on?S z60sT^d?=kgyPWk|34tou#n`2>bmHG~$YGCJDPyGa zv@9$%&+cNWoopcuP)F6U5B&U(SvyiFo*qj?55PawVhc&99aXRU*uSK+ikV*pqJ&xX zN;-S|j?^*M0+@{QxoK?kQP>Vu@Pdx|%Pl_qqCD^;Rz_+Vh|B*{4jVYi!)E08(VhqN zY4mkgLH|KI(KA4lbRX!csbQNg>SOkn^jC4%(BM~bUs#Z(yb+7D7kSV}q}R`f3K_h= z5o-!2Vp-)S(swExAcKwlJM{f*I`lU=H3ei_KaYV<`m}MRH^UxI&hKgXqLWspvA^pX z3#5``dq`i?1f~Jy*e@_Sx{9j}OpdS8YYj{jfl6;SFii|9y~Duy2DS`bWZlro@t^!3#YJG(lMX-we~-X{xwp4vi~;(KR(^W z<@ZW}=IF7pW1Qd2PA}<>?3?5Z%TUCxBe;y9G1c}ZbnndzHcyw&c3y;JclEQ{O z9QzO-+4%TT6g|Jv7ohfE@~}!<|I$BAeHV8NRSA8;Uk!b3NBd#4&(q7KKfkNLi|Oom z657MY2KpReFMBXw`NzUo*MI^v|7zoj{`6VXFx|TI>w^~a)A)1*ucokz%c1W;fR7jH z8}r{Yj{M+<`V~jHc2PsvkIBEHCsnWiaV|Oi!|Ojx`Vq++i{Ev3Nh*yx#@oadGtbHC zD~A34ES>GR&#o`%kKd8#&fn0-V^(sKI=@AFXRsZj3YRa^JJrBbG~8!xPo~WbcYVrA z4r@3wnQpRNdPZ_s!I9p{2EWu~G9J5pq&LB!`;)^89pja@9P)OAFka+e@E3&dK92cv!f^t} zSWw7l(f%ZkIX{k*U|(N#{TcT4n3a&GjECohiN1!ID~9Q3IM zJu5Zr&}qG$VbHTv;cl`%$v4xW&q{@*D)iYd{kuZ{u0E+hb5d!5ggnIWI=4LU@7&a| zM$ggcG@kBwi1FkR86O`X`1@12eRF-}7~|{V`sDV`>B~7?wTF|4f4KcC^;P>hX=SEz ze^BaU`jqyP0{&U4+#h6pjDIS(e~yJdW;x}B{Fwe^ePNGt9W3n8?XR#$x4%h7|5A+p z2>!18g1~#a?V}!q0z8(L!=4s_4 zy+ImoO5C0!R)_=s581m_hL2bI5D5EE(dLgxZ{(E2tdkYV=^t z0p@q)n3bpnlz03Hmn9ZO1pBe({%1JtCt&?4$ z!d{g2Ed;iXf7ARg&MDjcssD3AKvMtQUTJ`x{;vrXDf*4{-eB}+U4W&!{fYFh*W`^@ zvz7Ljqy1T$KT`jLPWwL4Z)9uGIFj_g4Kpb8b^BYM9EL~PIAURScKS#5w<$o|`7V8` zLf-}cnOggi-YuHEeOAQ`Eaz_h-i=0oD+4T3;orN>sDIN8K7Tsnu^&nxPoV?PEmAP-BaLr-D&_MPI+b6kV3fy(oX)@0p?hfl znY+%x-2eQn)Gd$rYYgF(GynHlYm<3A!u6Yj{=(h|U3>4d)&!_JAAj_h)(735*7F-g zVIRwO4Kol<3;wu&BK@fCL!Zfv1XT2|>W|8A()F3f$Uv0xA%#xmD;4}?58R^)<)=XY z&!-_C?hWimWf%yPeatGg@e2uz>!Vug z<0aBZrJ|3AW9UPjk8;@eI(amQ25{y2Q1wIl)a@@~nflV>5i|bivu4lW_a}0`G2;>J z=ehmIejc|UWqn6~6I1Q}Vm|F>I;`oC){?z|mY75C8f3^P`uA%>WykW*q6fcp+T>IDKt(g5+)^{5Jpxs~D z|D`s5LH55x?f?1r*#CuCtttJ!{nz8YNUy#=Qanc<=JsEY_hR-xsE>c1-Cx=NA{*QN zx77Y0pP>Kx{nYn53n*Mevir|>6GaxE56F8}dpG0788c{W((T`LEk4A4yldaMUyJlk zV=D!BzTU&%{myzV*Ry;QfzNk*eMy~WWAuNAv!8}l_DT+a{y_IL9aUHP^8|jsgn-wU zr_aLu$uzm1(esIYxkRvSyRP+rF`B(@uA!RlcrbLA7j77Y(L+rPx8Yb2JQAq{#<@TZR;=6J3p%ZrA~d4AO3(tx32t)rqX=r z)?XOaevwn3$@fNuRIQSI~ffR7)^4}Vx_w@>mfpQ_%kMtTdQ+Anq5 zC;4Ij7wq>%{R6_h`(!pBoFMb&@TTl@cgNI{HM>q)Shd9 zBp>YWsw&^YG^RYiiS+98FSYO5AIS^*E3(@sd6!Jp<1c;wrS@I>OGkOw--z8l$zPZj z*4kI*U-%=n@7f>Lhy9hENt?pI$4Y$f~T_bba$UcZ0;UMJ0dlrV}<)Ba!#eoXSiRCm%!nnWjx#Pcr1 zdp$!g|2J}?;yd&CYoC>sMC+Mt- z00-lKw&K2XC`<{f0G`L<`6+F$WT42=Y-XVB!{Viib_K8ZH z-$`rXEWc9!5o=B|!Xu^siwu5+f*1YE`7LKf>=VsAlgl$}76sI{JlKCtCVO!8iPPfB z`2FfA%cA)VR+vQRp@3)b`9k6=A7{bb{46%f7lwU!7;ti40mpdW&GDRtOC`qn3()7z zDs<_WyI2C#xt6nMCuKjcxx+cKl2DUy8)ASEy6<=b1BAO#V;t zC!_N~L*(CI1)v1{D#;4h3CiGvNP*vsE{T1f}{5oBqKO@#e{ycZC>dz4?N9#Z4Q+@+&`+Ee>t5yB{~;-)V4S}c>#O7M(!U&k zpM~{B_4nC$An4+W{XN~-e}KO$`x$h8Mqf{Jwe|HeE1k=i9@pPZe}p2bgOvFCk&&+R zH~n2-k4^u8zardG{b3mxtLyK=T%8{6FFJi?D{9N6i7i<22`O)xq-Jc>Bk6yVt z#QIuntk(s(x<4;5_y-Mt*ZnbKl?dMSFJHe`80&X#uI}&Ve3)nWyB=?lzt`&ijvF#c zj=#^Qc+|%5U-$>YyJh8vy3z*t|AN*a} zKZHM}>Fe)8%|9Qrrf~VD#PxU6A9D$N?$0Zt{6(jm z{xVVgy%OISQ0JRf2LD{c-*tbQ*xy&QmAM5PUY%DYvGDhW`Gqe1kK*q^)!#o7 z5a$c7)!*@aMmY~f=g+u($^EE4Yi?2y+OYM7{U`hJ7KzrgT!ef@v`>-$o}-hIlwHC+B{^>_G_a=wuC!$Pnx$KQiV zqz@O*OWLkr$DbU3pU(Yjy5sNUPpZGqo2TN5{e8aiybAt)TJ`tbOnv>$o38rjW7d2w z&-}Rlo@@BqBF%rXzcgRxZ?5n2weqI@qJGr+t^0c^)T{b)v_I-}(;uVbZQbA3=IQ<( z?T?9_V07~c}XX}KOR`ci(i}nlmB6-OzfZG`(_pQ{e;)%xj2u} zXDv{kzY<=ko!{mAY1;GPJncO3W7Y!he+wM{TgdT32aEl6dq3?lE05FjBpuI7lky>v zJ$@&x1q)RFCjU|~o^K^pY2|f)J!$3VspU^v*WS~P}lnsZ9P9}tweX#`9NR)BbFZjoA(nDYnidXzB*s^?~_)kvA*l^c*N4@ zpIGrkP}_fBo3Hx+No$?q@A`ZbvGn=DjE^E#vEl#u`Kte)v`>e%Dw`lYwOH{fUpP#hW7U=$;4}VkrzX%Je>VL(WKcBP~V*_3-U${gs zzg#OHu_p5W0>l58uGRUO>!ohr5lgIQG~!}@nQ!=i$y(k2R~r6r&L`%4BfP`qzrcw1 zSFhFmztr&miRKsKjV}L1hX1c!tNZ^t!~e&hU&#ND>;9iFA|_}3Uo4&=IQREU3)bg5 z{!I7#D|0oxx-?H>;s0ybEp+MsSNuPu`v3U=?I3+{{$HqlKc>%GrkqET{y*r(CyhbZmCW$zy_J> z&n4JkR{ej4=FcasCGdB(d=b>Gme=QtSpKQ6|4R-3FT~?fm7gBZ()u1!o|6gtRzv=I$zef9~zW$f4)BS&~;s0hlVfrgYcQWz#XtCk{>(=T1zuxeFGoIL> z>4*HkPxt=<@dUx~|0U{t0DmuBnp>dy|H{>Q8ZKSCP-5Z#>(=MH^#3dVU#a^4g#bMv z{^0x{@xSu@HGUqs;DTD zT)G=rp0}ZR|05bbFHfaM`~TFby8oN;|3WSPk67c!``FKe1FHUN#{bdJt91WgY54zY z-0rFTOn)@}Rah^LxLE&-jrG5Ds_y@54ga6WKck;AtE=b2r3Pg?5@{~zB!W8J?L zY3FhBao$(u|K8*OM|J<7DE?o)vc!$&X+L1C65j)_TW`ky%KgFn_J7*p0`}^&Y-aG~aq7-0aKLGZq&Id0AV9-d{z8~P6r{ntp=Xq^=Kj1MdiTmd? z$3Nlk&iPz9A2{cIN#0yuJ|N5E{yRzaKbxQJ@8*0!_vh;QIy$dH_RJGsFN}B?`z3V$ zg~ZSA;ryZ2Ui5sh76G(sUvxj<;zv4ju^PDZPsCbm!~=y%YWav|#{XE(?ZoA!r!ngN zzqy}pu8-#aeyPn(`uiHg--`_YF4oq=iTCeIH2;0XD#-PsMSK2@-hY^A|4v`8AF-C= z`GCq#U$1dLo+TmzXMLVw#OG@lu5-`F!5?#rN;O08Cx1&&&bQNfHotM6Ez3C1Cj2+a ziN6!LePKZsi1+&_XpNBN8H4;VbHeSQ7IejK&~ zT>qr?S&xUK?NN`1OS9DV(A@9B{Ipc3oBK=A>$$!juFX={!-%E(Gxo0#@T>A0^H+@X zGx~a%g;%7j^=D}P$A0@#ovz<6P<&RcuZP)U{c!wkCMHBA=YDwgLgjoKt%vz5-ScX| zYYR%W^K4m5i(ESVZR!WR9~Q&kwDW9L0X$$ry7u*Me1C&~Ip4RC>)RzazJ)y+=gIPo z^JKbzMXzt>^C2_dOwjxb{-d84k8!?Q_a}dX?k~E3p0rA|_0F8%alTz!AL;!Wy1z{n zf3DQ_pCgtYzv8@puI_KTJ)83ft*^Sjt=9IRBbGT|WE;<;XBzU;?xIY5eO_kVf9m_s zwEnC8r}#woH~oAJy$`8Y_qSQb{_||r|KLw0$p!BDFyNJ0_W3b!UVL@7eZEZYKj$jv z&){!Ke{g@xR@b-V0UX#u`T+gS`M#3$w;|QvW*O(Xa*gv`6ZspRFIDHy+1h$_(n?6s z*SBoL-=g;`-kZNIH2g_lzu<2;KC8-S`dhTU>gyZDYifOceZhJ6Y@I$?v*%dPV|0Hj zH2g{5pMpPQJyF{?{W*I5)YrH1{fWL0p!=Kd&*X2-y1&iAfFn8U+jWM&ElkFH-|go& z@VA0g{d`wpmVTZqH@n0w|3~q+pz3df0ek@h=>zmP=lglm-<oj|ZzZvnS>2J~guKU|U z!@{-hc4X1Yd?wSMaUrhngw-oK^uA3xqCe>2Zh zVZf0bf6FoaZBBBodp-*OwlFnc!v$$ewewW7vXt{ye-wYqSN&}yAl}FH0s5QseN^dh zBkKB=Wt?ZqHqJ9mx4FjprGNhb>s!Y7>)S-nGxCh}%RFzChA&j9`ZfJ6dcM&8ZT$6%zW<@GZ~FJO z$lok|eG6j1ksN=UXZYK!v7B>3Ch)Laecr7hIXGi7JwyY%<#Z`$_<7_)U7YvHP=7k=uliVw`r7;6o;SYlDE34B>@*aX0sj2^jjy}!%f|Pe%6Hy7RihEXXE`~nHFQUga?7~{mPk^m%nP8(fj-HePp8NfIfW$bokS& zc;B?(n8)A0Z?>P>i(igMdAvV|zYkfb=de|H-|-I68ROqK_7ndNK)n5*(D|+HoNDja z&OGW9_&6^3KDHO{n+Z@3zJ^5dpg+D4VK2$g`^SI7emY^$seC5UsXY3(3O~&>2z8=; zPS3KYdv8fZ|ELV;S=Ka;d8<^v(Ysw~0$vSg6Ms&}{Zz%#pwNf&efp{NdJ0PR zemd_yw;$*)>V4lmS2cSw^&7b>Z;aK#7Lvgc_3^&zBp@-q=r6{5OdW4}-#Pu>5X$rS zo#$CzcRWyjj>Sih1+EVB_bWy2f+@CwBW2@#W1l!0^zs3sP7e{D16=8&cTltZbl_=T zb^!xOxS24f(+iSbpT-tgc|I>ld3^l&`<$@fPX6}>+S}z%g}u?(cGQnh`yBKB)W7av z>S{;*TMka)_-(>0&^a$*9x#%hz3?z_&>!TGFwt`p1Umf* z;{KCkHm`HunB>p*#{-=SiQ@YkHa!HlDJ3aW2)f<>d7at(eUQi!^~(ceeEyK*HLo*wzG^R7)+|0ATxRrdUZ*dM{Ff)-ENgav z#*6m>nETsRIbOh|Z?T@iU+_Hd6GsF1Ei`El*yo_}nZ*Vj`2wADQe{L2{o%W3o!3nk z_6|0DyyvC~`{tOB&un(psZZ@uf8_Z3*W+;-^~crEoG~Zjlk)NTNccC}o0%%sD;keo z!2Zd?{t3e#pwC>q9M0A!wKs{khyH;M|4SqLU~)X?bzV0n<(K0(cTCDJ@yt2GUQvbD z&q){lB>Fe6(=YGe1s(o4SM|sBCD~)yAo2cvHy!>pbBy@8{mtR;M?sbt@7?13flVj< zXR|7g-JkWjialZeU0%FJ$rLXWxC8-MgRV`Pn&H9$jAK=+H^6- zF9BazbLbckKVFcx2G5uZ- z$KY3S)Q%oR|GkJGSsNIVzM;?MN-~air{=OOe z{;LY>8+|_o{jcaz@4xW-O-H-g=f&?Aae1`&)!6mvfhE;{$*ezy`llzZPY+y+fzNcvRoUzS6kjD0m;f8z0D zl3$+555ISX``!1!?EB?ks_Mh!U&(&(e))&S<6p!sykGv6V zyBppe8+-TNmtRFX^7B_;O}q5Q%WrS^#vMCD-~Z8*Uw-1(vn~w3_S-N0`WuN~ef#a- zUHQfno@E=x{_z`69CpaR9LxFiua2e-4Ug@5?zMH3x<|(r{p#pveBXF`?31sp`;q7D z+hafeR+sm{yC3iQ=ug9+y1VA1KV5a|m%rHe)1JxAqi_Cm&qsZd)cv2IyfN+hPf;NF z|BbzK$CG1U|M@#(?N@&{_V=Iu1!ViyoBwv#FW&t4Z*kqgJSqO<5gwJ6@!(X!2_9@0 zc~TOS-x72$4#O+>9v|(dc~a6qr(G(a4`2RfWsNM^o0^>V7Jta;^Ls|{LSVV68%8Ng zWodKOG6`xKu^|{$#;2AE^G!ohhIswh$8Pp`XMZg5EmlT?dGRBV z&bBeNPg@k@(P@inJUZPnjYqGZfKInaMVlc}82@@I7<~0B3f5(UCqi5_1%4Bge$NXU;ry%tP~#CoQn)EhhTlof`3|IFevszmB~o_8D|f#8aN1ni%g- zZOePD`Nji=g*c(Gmt-hfvl24_Z zy5{;=P(DprBa%EL5s$e>cxP+&?IHViY?;aT&|WFg*=mqol0kC&cSilosK zLilmq1aFc`*ZJeUcPwN-{Rn`6SxR2A#={rF4;W#PR354fiJ?RmNgSt*Ftssgw~=A? z!4LN(<;CMaDKv>iCXK@%uA%6NuMw(z0dy}C(D^&-^sz}X$3U&05=vo_6qtwI1E^io zVK|ZcC!X&!{BkNDdQdA_N?Qe?};SMKa>+C^MAFBAEvN7qE_%{VgdS{;kv*5?1iEhz@LJ`wH2HkjFuuBf#h*j%8N?W0xsLiGF5Oel5Z$?{jH1AL zZ?bz|DB>}Ft2b{v$*n!wK9Hvd!35czF24C z*BAPY#BE>Bb8+c%o{38r`i7ItAxNrV5FUcPA&`z4UkD#yUx|S)TxfU(j znu~kYwL!fNc>Piv-ozN%@Ft1gKpQdXktFXVsg2n5NnX-6X+xo_+6epKt2QWp`U7dh z?M)=6Hj*RMTRyh2=#gaJTZV}$W_@Uz+g9qJN`HUaplIz6q>WV38(6w`QY_n;%sq)b zB_=&GnfI2o5u1*kG}|^@{_js46dV3wwBemB{GP_vmDZg%ylGBvV$yMw%X>R{5O>Ql zv(gwql zuFzF&gwwA<8v(knws%&F#?ywq??~&C{W;z>Y8HiUq|(@>52j-uCWba9d()^lv}VMl zr+L$8dtZ#(UL9=|tMN7EO?bqP3ACEFIdwGol#UNcB#p-UbE( zvFTD<2q$CHrM8j=Rr>qWM(7WwjYx{Hbo4e7OB=8q(gs6sW7E?jWIJK>Ha0yyLbk)> z8JGY2(?;bVP8*ZO*wZZMTCzHe^4=PjuFyqqC-fBE(=6R|$=^&@7aCRbxxzhcr8xYzRa%cZMY z7`%qKRo6n?xYg`!JRGfs)QA^j&F@uQDHSbX2+0=W#-o$H&b>;^`pG=P8^>Y5`HXkF*dkj>Y2_TT8L)2ZWZ$R$Tt?PYZG5 zRI|77aFiBsU&?KP))`kyMGHwn3v}Nblb#&0wG^A4$hE|0Ubp`H(?Z<%G+wXG{3tEp zIV@>`VJpiUYkk3^KI$cpJz~)jcXB$#7BT6G?)|b$pI8fJd=JFk0|;XeK&&~J=)6J9 z=QO9W=0xqAH0)0FG#tcUiXGMB;SS$hv7Zn!R+hTS@18gMHly=J<*DU~WoxD-(;t0y z{!0Cc<;owO@Zl2m(=H7Z<);YRUX3Wvo12z6!fe&qljX7VcnKW}Lo-hCP58_?b)e9zVE8FW_VB0&%8^pK;QT0zgy z>5A@R^2Z}rnFeF>cb>-LOW<5SgMZj*zb__#=J2ob@cO3w4m~XB27gE;Q&7zvMLIp?&@aX0AGWLGOSv)i@6bbn9@P0eWPB-?*Ei&I+dn1fhJ4JC?*iU4>zvC8N1w}yq7RC0 zWfMpp*K0y>+3PO+cff1N)?MdQ&a zUKo!~E9rQ2T4M}))ULpI^wJ6HhbEv`M$uW+orT`MZa>lb)9KOr)9KOr)9KOr)9IwY zszf^UnkO6p-&Ge0t5Yw1N>9Bf9L{Hfv3tV#0sfJ`&ptM`QkFL_m(GbCe+I7pS~#g- z;4k{JQ1l23s|ByUS?&AVMN8)u{nOlM9!^i)+M2u^WoFPDFBmIfulv~5DGn}X zcxQluix^(0aCM$TU(PORxR5=s;brWChL^HuHN1q57AgD+*m(^vX6H1V&rWN25j&;f zh3rucFJOHd&SS?lJfGfCptPUM@MS**&tn}L4zgwq=dfB0&t=sbzK&ICcn+)7Fy73k z;aRK*xXjV-OcvB|Hp|d3-s7R+EOr&|vT^HAXO}cQ4ez;7=~LOW!0vcw(htcg^{3De zy(&0^e&|^#kGCgk^fVUM@MK!zmHMfyR>LW*O2f&l64;d|iG_e&dke5q4Nqc4TKPm4 z1a|wMKp~2PvCeDQhxf*?s~)dy{}dpom?8=lb0?wlyZqcIJ!KdGOcWj*i1f`KW8Upv z=Bx2}{Kctk@N+2rdB%$H=U+U>nHR-urT^>4`){wpu?Oi9C>*&1e;57_uAgPB7SX?V zTbTLRbokFfK>p=b%(~1IpwZ-*q6orQC8kw@Fsy~st0jhYaZIZM(W@k`mAFGF$)S@D{;p> zftw}1lq+zD#8r6$lRk*w)l~wMJjtF+R=ZkYCh^r91P(}?Q7&+X#6^_?XG>hQQ{bS) z9k&RaFY&1Z0vBmcMit-&24@zuF96BxNr#Q}JgA$AWX0lMfpo{*3F0tq@=w}37^cQrAMSns6 zlAw$JX0i(s7xDITS?%M3UMlfbi9-^H&kA~_#Ow)yg?xD|EU}Oe`a#G<<0s_vv#`WM zKG4qzx{wcaiG_TKBT8p5|C~H_4)f8u1(=T(;tw`kgmJ3EUl@Pvv&h4^`0+YB%%1c= z|MSlUf0<~uqmTHi7QenDuk^k7b*M`zfLdsV&v%4h`8@sW=4Z%qbI=|o#C!IZImf91 zeu>J*kJs^Jmbc?M%BSz3*=Zeh0O>|4j~$==Kx8}lcAU7$$d{018+`11m=sSoe>?y8 zy)H&YpA!|jT-NY>*?BjceU#_(4HG4ov9EZE58c1&=^5mE@$wtHzjj=lhbf-&ZT*B~ zwbTUSlc_TSTF3L9Hyb+iYUF2Txbk$!co;vj!sa=pU|;g{a^we3Max-*{AEghP?qnG z!$)ELdwrzgg$kllNai$1d$bEkbFd+|-B|UXk@aq+SU^c;ef&rr^dFsq86vM)R-iKJ z__M{aHCGLVnmnoIaXA!~2GM|w>`scPiE%*=J*F-#+2dztl6_MBkRe7dbqp!U{}u1X zB-zdSTLryC7IQ9tBkQ~VEl6hl-N?QdQP2R5*<> z%`ayC5$y*R~P6zCDf)gFty9SVW-d`adq~2L&7FUt>7f9@PJ{82vvj>)Yc8Gv(uN%1?enI<B&+QgW<49%N&`{r@hsb)?Sk@Z`)FK) zmF?MEaj2n8q>VeuOImvxR+P8yU$N)b+sayRsoU4sv%h+GaYNg_TMw_gxxKDz0y}fMzzFTkY*i+HE z<)&K?9=N&vwzB4f2P*cq?XTY3)wqq9+ucyPdsXYUU1+=e;QoE9dg@o~Zf>k- zy}iC;&yLL{JBu4DtmcLlJ9_r7*jsd9&#I!P>Tbz2X?n_++T-2?q1WnvAKQIzC%@Y<>l?0SL|)6E8DZ9 zd`0uY#*#Hh4^-}&K%N6db^D528;(p=uA`@MU-7}liZz|o@0jv!y`!4+R9~^>&W59_ zTJ~4(Y^^TYeP{hfRd0K@)gQU}&K)$4l~UGyTaH$*XpGIX;?NzpmGL?yJGZ%YHddfc z$$_r@`{LGHDfL;ozk1t2*iTO#Y$le@b{vs%9NAga(6+a^KBjIr?$}bXv-m*8=AQj+ ztvv@S@CUm%P`Rz6v2E{B*yPrmi}&8ySh=lr|GpB%CiWh#+qddqT}+0 zUm?ovKeA_n{v0@3kNK>6Ma2Z=T4lNN+v`h?PEfW(mfd;V{(V+!9KAbwl6{9>izm|Q z=7vi6MjU(FW&3A!Iee|viqo#`XIqF@-M*gKec9A3=f4AYG_D2+W1v8Ut77#Ym*N#$2IY5 zD7iNI8cHUVZ!7n;iY-SDRL8T`ZK{v7HpE-IHz95+IslvBQncev8Gm42YCa%hirdO* zz1v&VSiP;Mx@1dFeH+%5YOEP0SSvPdyK{fb${X!?tfsBCrL*R6>FUzejn}VObFitn z;oz#(tLhuBZ@Rv;uDGs#RdK_bm8)t*xrUa5bq!554L#khbzNOGZ7XWpteQhjZMWCl z(sZP&z4K68Q(Jq>>XpSR*1)(#t)u~pSJ$jAUb(VnWzm||H6=C0MK!l{_B4rqw$!zD zHPsZ?h%3(*AB$?lf5kPWvcq@f9&YM9($pHO50WCfvZJn}>yDN??`&;y@E(D?ic3oC zYgW`Wb|39%YT$~H3aPJy7>!-6AhaAgbZg6z?$Vm>wz`^w?VW9P-435kb=`GacvkuT zTQ;LJ_3c(oV@n$`uAx7tq#;2~QSRE{T_68A`chn2%zI?t z_fu3TkCznxEX8G|f*++D8A`XLUKjeL_b>hbPvWNnkEDJk?bCuYm2cz)#6J&CP@e8j z)mwvUDY9}$>UFga|KvwgA4~hIjK9wOn`xiP`gC?e<*bCQVuOQZ`-!YVmv7oPXdg+t zBr)Sl~zfTfv|HYmP&#$6Q8zj4K2|iQ#TA9CzQyJSU`J9va z_PAWM`z!NpKmCd9SBBiDwf*yoU=Emw8UFk}tc= zb56^AYFCQNbL@S4YFEv_1$v-F{B79?lbusC`83P?xdxvOg^&6)s87ikzb_;Bo>ubJ z^52Y;AHeuhB01Im;*lv&rktfR-|X)tIY4HAhh)CV=XIHH_V;atk1GEh=~u*G?Qf}) zuddB>R)G?gSM&YSXefV{D8m0;r{weB!jiPdm3+1QOG>_4ex0;$lYh68ua>`{G4i)dqc`m3U*s%bgW z(#`m%#+JI)riShs+_yA!cGq1uHz#-rZDKvGdktUEOsJhefF^(AA!%?y~Og&X)R~ z?k36=iR0*57XL!w7a!`dbF!g{EWyd0Fa+q1 z!_qS4^*B%FtFpjrw3M^etO`Wy5}lrpTXB#W@_aaZteK+z@%lXLhQ}0I7rav4W z-f>{ysi&VFNT0hnJj?%`$b~t_Bm3U_Y2+#An~~-dmm<9@-veDyIDGKn!1fad2KLQ8 z9;rNiBGTM?I?{XK&B*XWmm(K-TmsFaaE0^Wz=gR_4D4Hte)fMSGOGpSpL#P=asEg9K1dR9+H29c7KL;`v)oj+n0PYGW_8ANUD~5Z0yQUsKH9_-SJ!_lLOSmmMD~j14IAQ+@@Gs#5TQpYm zI($IW2sfd3P_})>A?~zX`|LeE-pYqom`QWu(KMY4pC%!fG zfPVPGKbKr}+gZlNUzJ=PerZ|Xuh+7^bj**lGuYLkAL>_MhhGoova4rHCmbGZ&<>yd zJN??we%43xh-v+gtN=dxEbEK@3+ub|ihlK-Qr36wK6dqA7qY(d8SLtX6m~TYw8fZf zLwl3@hW?rL4LzV8zW6`%s~6v4hll<)DLnKocKBig*3@yVZ}>+O0ZR@KJ-`nC<^jka7)!=V90@GOlRv;ejk(z626Td=GfkraACh%~t-#BH&UfJI!7Bx? zOy;6tN0c4DV5`sH&Z>dQC#;T{r>EnpYs@y*rs+1u>|?Ox$zpRdJ8r)$^4emB(AobK zGD)&6_2TC2e-u}J2TS&_B`gUXO(M#9?M>|+!47Xd?$oubMb!DVuDz+XF~sJzwsrXC zwLUhlV@GrA*0y-Pp;Oi$n-{xjXln_0JJ`GqU#MfA5UsAEv%~LgZ?E(E{oA}9O}=1V zgSWk-PGUQJt?g~WKu557;eu@~z80^)d0T7Y;&&FTw7~7W}oHIyJiY2+j4=fXJ8prjBBxK*Dmr1nTg!^ zaxV?JmxkPFX<-7_zn6x5&a!vYvPIUh0&YGHx%2liYLQ-x4c5H2ty|lD9R;keuJy6H z7-rS7RdoVWSAeZrv$vf`?--t1B7eE7a3M%n{wwJ=4K&86)@{YbUaoTG?Bha^P07_Y4a3Uq|vi=~~7UN@aCH2b|B&26pjjkT$Ex!YTC9`LZ&jiHHGI$y!(9~NGBuqObw&v!8L zgaW$H6!*Ag_N@5*%^c_0n$nqls~vVh+lTF<@B3(U1`ou^(B({KmvuW?`f_osZllo+ zb)C6Qbvmmu;;B!n+t96Ta+K*Sam-X_)9ka-xlPHgRO~Y`K8)f@g{`u1TM6E#q_Y6J zcfk>>@v^%>1B2l|f7z!BFSF&WWA-!$Bk%6E;eN4%@4(Sk3troRK%sHy`mwxTlh==M zEh7X9`NhcD##qlTEy{icBpGNhgXMGyeep?>=VZrsCP(=JTpB3qoopT*(y0Ra^dL*8 z!2{KoZR{7htY^PH$}R&yOgB2&o_)!7KJKR6bavd$e$*voU{3;%{(3@~ND@TjFYD6T z7UD(kwh4ajLj`3zUgwFd;Jib5lXWCcG4R= zczrO}UKGN?AMlJ39`c38M{`k2nh0%UycwYsJw|^rgJlNCCyl562M(pMb41TLeG1V7 zpC7W1r?XDcKhg7PCg{;W^kE}$0@WXI{_`Q9fgb#1NPB@F^hfkue_0(G@q@Zf@&&3R zHsUuWGyzuR_DxcFK_6j+cLkjjh~M|uAno`Y5F{#`^%1@c|WU|;DP8|CI8e7+O<(Z{Z@t*zzqzL5)gC+g%&Q3v#c{_{f< zCz8L7<@y(OcIG8r>8CvB>(aciC;EpyoZQLD9^j`a$i`3LT^hSg3~0hXRs{77sxs|k&W|f7eBJhI=mY)hK^t=to|HiL4Nd36NBhD5 zV1_-N{22>nU<>pEd{!ubI<1@VFWx>s1al14Y7p#FqYS;s^d!6nyYOZZA!YR}`Evi_1gpPiJ6$&WWH@zE_$j~_2>g&}KLvGx|0?j=z&-CwsykI{EaUPr{}}Ju-?b~i~MJ55{);N`_Cb==hT>g z4LY@=;FP48zhXU@9-1veaQ(Pv0a{%ua_B=L~;~l7r z_T-Ot;HAI8zoFmhG5sog69Z)Lw@Kf0k0SL=(KP#jHGMrb)ih^wqNq z`M_B;q4g{8&mh0e;BO?q!{Tpt9zX3Q|0!C}>>_`l^`V~DOVM7%x9K4Vr+38t&B66S z>)Sa^AIYEaR=yr%{;h!hL=p4P8V||7XNW+q-$kLZW6Az}e8h|L)O=o3@t|iHmVOn# z0wkXvmoLwWcqu+By^r&op%+p73D{sVTEA)j=saHPgvXZ~{zCEhGUyOLUs81%wIB6P zx{(F=gI!FgT~BiHxUrC*jg()~#@8pAA0U6j{FzJqsDk?Oov`2I08l;v|DpK%GC3r* z|A6E_8jtVLC*-ljcij-*a}&gKC^&u&3GoNdGote6}CoiF`xJS5Ntl!}+UgAC4FK4%sg)%Fbax?!RQ8 z8~Z2xo9BxTnO~ATyr}#mXmjxO`M<0DDb@BimNPXT)-xxz3L>B8>shLeq7C;C$V+&> z-em&TGz=HR+p|JrP&eXf`Wo>`s2lOgNVkmm6x8E* z%X-GwH@i#5lTrPU^@*537$4*1 z<)F^@o9GYr#pW*>4bXgmyhWkx>{vXr@&Ws<2dK?dVxxaFKKwhf5z=6<@4f8pFlzf~|E)icY zRGH)1i^v~$vvl%r_|q8nMiu!}I<*Lt1?(c8d=WJ&hV^0yRUQwflKs-3Lsi6c(swep zBgS~1oJ{?zc%Bpzvx(JU6yy(kx)>Uu(D?}VUx&J|)QY0h{v7*bafnI#!#B9UlXjwk z61kqGE|TPezs#VXPA4YOZnFO&ZFwwT5cr>lxHB+5KK!c({AkA?_=tDH>q(6BLx9&Y zUv@iiRDfc`dT|Ql?-uq5)W1K5{gUGCYy5mq>F1W^!G4{uALMU*eLiEdr!_w_dRiFpCTW@d~-R-ANlzT z{L`GT%m?1MKGS+-%~uTTFXBZG)@v*$BHlxPru7`_vuXWCJqPP`j2`W=9>w=tgPM80 z#(X7wY`q>eU*YrHlCRKaoQm*AzMoL`hdpurm^~5Ls0;J`^rx7wkUu2MSEh}Uuehh( zIA6)(`HDMnz5;o2+!Wtq`AW!;Z;Z@W5KrAXv3z9+`upVhNDi&9JYJ$dOFn{lY05_& z7;l;z{C;6u(C9zbU&wk^AohA!a8C&p&W}U*7rh zsPXEkDP9@Q*Wo|66R%?XZ>(?T{q}r(zA)N;dp8$Htxr5Y(~%l|UX(}i|1|=RBSvKg!%~=lK)%0~zdfU6yJt;{{0E#*=3WzSmPN!Pm0?g{AS`jssEtV5A28i z%tZP;hOf`(@Ok5|@$vongM5F^&rf;2{lOzSk58U*Qv30d@y&GplOLKq$r9h<=Rel{E#~h{ z?6)Tw<6BW^oX+q8me?QC{Ky_R9u^Y&?_BJ^bH>rN=B=P;X(>coi z8wV;>ZYSPV-Nybq6RRZ{6#G@QccC8Jk0M_iXFP8LKFfF>MgE^n`KH)^^7)V*+kf)? zH~AAvEZ$9=$lRz?zIuK>^y$(K`h@*6b&Ebt`;pP}*FWNXzm#OoUyKdTKbA;M{e9M7#|Ki#oKKxPit1tCd|8Dc&|Kraq z$NWv>M;HD!dZ>ua?0u@oRqO7Zop-Fam$5#3?X$!0|4Uu;YaErfCh4zx^Ll!F)7Y6G zzw(FQN8h%yWwmRj9n;s;CatN(ZUiCyr!y(87t$9;HBEkLk=Id;KR5^YD+Ex2YGdqa zS9pXj=7Bn1*YG#X;C+lU?f2N;GPYM+ia@T_GtkG2p65)~SPcL+%_9=;7}A3~ZA>!S zQwE@Eb9jGhyjPR#sF$RdB-5)n-DB6vH%}D2fEq0noW{7qN&qPyF-A4@{1EZP$GGw; z7@@xCCE9V`##nhnGZx41O|#-D-O-mg7hGXm@Ocuxr<#TYu}4b=%dwIOA5*L=aeK<; zd2uJkBl#Bv3?%BG=lVq9NsJe{7VuC^qV_TVx5Ra?Y>nTF-F}Z1^1o)WTuEhA5+-@OvqCR4Bff15b zNroX&43gPD70K))J&G)0;!M)SID>RI;%wv`G;?m#t(+&+xh^=1jfzf;wimiCE^kds zG-HLHO3+4Q*3qf~HRy2!p12_$tkLyV4`S}Qxd-*o=B!8a*nm~V$d7;(*XwC|o+J79 ze5KGvco&VWRut=s5$EZ88dUuzUV-B6_v%)eU50TJ**4~xpi6{TR-k@HN^KZBC@r@Q!{YlgnlSGd}b*jT? zBL1LGw|waX9&DKC-?r43XlA^}Qm?kuX*}wmFA?Z}qd3tf5Fh`Bb@GT?ZSzU#i_^>O zx7tUdHaElj8NQ;9HrmbfG+G~4e4;)-J8rDpjaE0##2y)gxR(jHB0s;OUnOd zUHsd^hPBH~|F)(55@gNB{ydg?wWW@1_4;d)*S3o(O?A5J$L7jS? zhB|dcx~5U{R6V}Wu<0~Par!!YvFow4n;VUCkkD1o!&ti@IG{*ewqGlrQKRa%$ z_8hHloMHE3xL89*)7dA*8+Cp!hMQM$5i3-D7#VkHkD2%+&9b`d*43Zlg<&VBX;+ZN zsGXdQxN2OJh5jgUiAIjcCR@^tx!;>~Zy)-{nX8Fb_gkI+w;v5YjnS>#ChGI{Ikfiv z$F7@ZYVyd_LMuOs=q>vpQ$94JeQeLk=U`H71)=2!#Y%st)lK?Iu<}{uNkp$UceLVy z{*K;Xa{kR{d7rnlz0bRTP4mgPV-vAH9`*K)*s*!wLGhTqml>m;nze7~`sy<5wHZJA z)~>+&=rA0+ZB<4cfqz!w5uahvk*wt$UHG^O-!Hx&C$|6b{zsmSa|sOn>l*LBTk%HU zJ2Un#@e0lV%x#UXRk(1zK=VnIGj<%>#$Ojo! zm&GV==J`JfpY5~R&7OK8@5S%FFusL-Z}xA0x6$@DGv-Vy_{r2CAIeCt@Tbkkke`F+ z|5QV2H~y%uARqRq1W$PW&&$gh+ft6dEywG6&;KC>kuB(&=-EC5|0TVl*l-dQ`)VEP zO1qB$&JX4})%D+I@jm#`kcnt1hmY(*Jg1AhwTWIlrz^_@qso(*QZ^=@^kPJc=XAyU zQU0Qy)1_$9oEPz&u51O4U1QJbk`P57Be=~Qn(6$Q&eGl^>kY9Flm#HZGKe~D0ur_V zv26Dxi)yXy@yC#sGcM8Z3i=~W6o+g;>m7}y*ijlHdszFVcGYsiH&Mk520g% zc?fAJXDw4qjj!4%`#*-pH)^lws6EkH+tWNJ6*|l?mTg&_C(Vysh&ln7*`)u=vb|(KcCqSzE!zL*1oEr)mt{-1 zDYw6MeDVp>>6Us)Xt`v|0o7HtC-CjI<567BPs?zz`KsovaXjKx%?oQgRrg2-F}C9% zIiD1rx~lo5uBzR(;Sfxok1VTpg!!nL*Ai~Z}o8yY(|ZLBG5XxUJ^ zqi9W=x43jw>H3nUhL+_Wn>W_@8v<+B`73rVZ&=U-*xFjcm-&~j*|f2;seZ%yg3TK% zS9&)T?eJGt)vn!HzTCh3(b}d!&07D81+|Y>?5xmR#2BsMqkMh& z`p$+0OS@>i#toHiB^5i%>KCpr@Rx6^jE!5qZMC;}Y1^{$ZR;dW`KH$OEgKiE-+ohc z3kpZ0TfVs^)UAG&(HzO<;ox3MHp-{OOP)>qcIu6cYF*ZgnWu2Q^{GFT2@mjd5bMuPIKx51L zF6ggfP2u{-8&_2NH*Ht|d)E|hYF%5~v{~3|dr9S1^j*-nv9i-@7jJ3c@lDN(9#Q_L zu5~T`=3w29C5syuEL!ASx^>agqJ;%Zwk|1L>VqIl7H?f#uxQc3g>{0ip?Ryf!FRo% zSXsw^TZMjA1;2K3JC%YPJe$05;UX|nP=KFL!Oy5H_xjs?^1ZYcKPcn#wiJX47~7S1 zM?XF}?@!#jzVsy+r*K{&H<4P5wa06iwtK2JB|R4pa>4IMDg)x0Z@8a>d*dGSNm_8J ztV_S&^Q?Zv{$G;+(($$QZ#Z8NJrjK+ClGJDN1>;&ju_})Q}-*JN5_0af59&6ME3)Q z!}XN9x6n-i{G7t!TBh#HoUUi}Z`cXX|L|ABsb5|U_o~ z056D@mgxL9L8rR6USC98J9}RpNtn>`!Eq*-kV7dujv^K`@znzXY!sJTJHdJ6;Q$pM zsFLV;&+B;CXq2ENdd{?<=Miy43!7nFg%tF0e#~&W6H-821-)_SEs$-vWEpmDN-heM z1Xunq^u^dWA(w)Q9>hC+shsFN5~%z>0)-}`Uv(Y$AJ9G#J?(E2!5_X3{0D#&KWDfd z7*q2~yAJ=)Bk>~oYH5Pkq#QJ6qH*7l^17u7)ck4&J@s>0!d2S^60SC>WwQS~*-y1J znbPw=NdED3DPGjPdRFrJvV^Po^qTCaUm8%sHvpcg21QS6P^W^Y_;UCbS^p;iX5Ye< z=nF6om6PJa-olmOS0vmn;Yz-7=ui0$p~g%61Pi=C!Y2W)+7jrC5e8SxW*V+ za2@y&2`7GxtJ!lB{*VRUZ=g5Y_caN(@=s6Ep+fvj61V()k%X&pl`l3)xEfc%j~M8U z<31V zYR0}`fE(p%l5nL5MemnztGq8txK-XV8FJLP1#;Y#2DowDkc6vo75#1rw~qTg3Ac_* zc?%WFU1&-v-BDblqTn80W2{MD6|PecjUkEL_gQa zZg%HE9o03rHsfa=>)YFJtyybli(shP*Um=YL|8g>Y(<T-t34bv4b~f;)HlKVqFXm&r`t8;`^p{=50{aQy_acbM zRLj)U6`7z;midQ5%4J$r8SV24Cewl#+#~PK$T+4DDqgAj4BSx4fC~ACleO9)ozKx|?1@A^9G(Nt95}J6j>ae<1 zP$+&Cy|w;Va{TMH|Hy(+e5?L4A-vTlgj*R@6y~dycDrmpLVnr)8A}5z|32Uo+0P;U m=dv80>_@Jk+N&ITKL)mrO-;_W(>s26T*^;u(38BZ?f)P2><;h% literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k877d7aa659e5e1d816e0442761e33bde.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k877d7aa659e5e1d816e0442761e33bde.o new file mode 100644 index 0000000000000000000000000000000000000000..4d8eac0a441950f4c52d7f47e1a4655657288ffe GIT binary patch literal 34072 zcmeHw3v^W1dG0=EM$$-#(MUWj0hR_?n1^{tfOxcV@8|(SG9V!kuclXwqyY)^m^G5j z$dDBwavftZn3^`uYMe_NHdk888!zHGm|`_I*DK9hAu=TrV2UTc4kX5N zzkfew&N&(oU^h0D4KwHL|Nr;i|7-96-+M-9zPN1Va$VDyDjIu*#cB%5SY}V`-otM_ zEESY))?M9itNmGy+xCCxg|^R5W$f^vEvNS#yw2HjX8jwk=kfYA?$LC-qNVHgUEdG= zb_~1r!3I6_ex)sR-Np|8KO5^k+RhGx_DFXIEBtPbr|*Lj_SP?Tc6e%PclBA@#NI*M zc#u_Es&%9zONVK*jk313NB1GqCdXGXvkyJ=Z|{EMA9ap8Qt0 z_r&q=W6p1fj~_S@9-ebD+&kw~c>n6t;Y<5ZhllrH4nIBT-S8tV=flTOy&F!Mwm3Ar z|2d~_J2D(vjyXydNW*p{!)1R zWg72~L!mhsZ}ma+b0R#e{g=Ti9T)x`6XQ0T;g zfmtV?90-Bu3v<2|-Z$q>@C7|U_FlY_XMQeJ4qaY&1T@fFO5(S|p?BX5r))hLo_YQv z__`P#GwspPFyWVj*BvLr!->B}yFWp@{R8EI?Mnpg;q&2SE$8Zw6Te(NeDS3~@}8XU zB`0=QC%>8#Bs+R+tk3})X?k~c9eiqn=DBv(HlderTXa?!wLfse{uAI|!Uwi!tnhXC zfTj^{LhqpNc#PT|WvuX3z;byT(bo^)C+}C5gg)3%48QYSyT0qSV@Cgt=a28(v{}k0 z$^Vq^^~3jGfp6aTy}R4@x^LopM@R9!qZZ#IpH%+WZ}C6nuQ&6->$`psij+)zYv>{U z@P&UVzUH>Gw2QwizBc^Qvc6xfWqqlbA7`htYePTKue}bx9?W6a&X!C#JXo(CKKpn2 z^`ZT&kLD56`X5~ZeB?RS7x`D#cj*=V+B+qz@7x3I+P^Jeedp8IwF^n?S_){3FxQ6m zCiD&c3+o$tNIQJ-f9cmQzQYa={cS>M=$q{D#d@r%<5=JD4<`ba7#ezr9sc!0p#2|7 zp|czIJW6v7JRiQk>s8ZS66?E==OXwgS%$zXjrlIg1Q{j|-Lk%6$X71q8V1k9;CUE4lZ+(aMesQW{d*yk^Y69LyO3k}m4whRGW=3fh-uw%<#I63wV4<{Sz~>)ZhTbvgAl=fgP1jH z>;q4vtHwlJ(f*^#k@_EmOh#2@2&TK3hCDmCt7u1wFhl-Zt8 z>cXOs^K-=^vJKgU>_q*&1HGPu{#=Nkfn-nL4)p>-V@Lf%|S;58!$Num6qv)42XS zjSqXh^ISs62D>W%vyElf-huvI)AW!F_PGeZB|H5JHXxf{fX&Y?V10wIAK5|q^m}Vr zU$}(zF}?p$vZ=EFm2bjU@a1>m$CqKVe~0~I{=g1jIRv{J?AVws^g(|5QDsr+dQ$OK zvfq7unDy~IoCki)U-Wjn^{RJk{!wEtcBLMjpy*7~D8L70onR~Qa-j1axO(tP!YiG* zXxI^DhcD3TYi?&%z~m8D$IR1Han&_u8*9^a8)Nn{Sn_1CIhh@|Ule(5kwWO~pM*@3 zY)hTEIs2#Ls_$Tl9=3!ffTKx7Ik&yBy(7@!t;3zVcD0B)zt*-lwl)OW+}5@Z-`v*6 z=XUI9YTeovtJinR`r~t>H}!2Tes2ex+u;j#%oU>5)^~O^d)wP_`r==m_^*8cK=MS`Xv~_g((VTE?t?eD&){gvpzv^)%$}-^Zv$dM!3sbKACVZTEHLv)bC$$7`dQ zRm)b@3QTQ2wrWk@b{@T>cxs9K<(|sPZWftl0%^l;*;}J$c9FCJhOL-CS3|Fk5?p#4o2Ue!y)+6ab#>+0i-AB`+a0zB#lk{q6MgJYZZ8*QM zmIz9o1kybf@V%R^X{mdh(aWp3e8R@~Xt+~98SYVYv1xLZ2gJKPPvtxc^y_qxi;m1}F)FRNZ# zUb(t<*~WFtR+la-WpgUs3*2+Of%-;wdkfA37JJbilTKJjc%yx z%x$XES(On_eNx?qZf%nzOkas(rdpe3pP9;SN_M4UpMmjV6jw5Am4Vv|@HQot`O&=# zj#!14-31yL4FCDdo-Mr0mbH%AQyh%EyW58QMH0RPM_Vm;Z36;@#-Zy+@_JofKg6|^ z5GdpqBWD|9J-f6B`z4TMpusej)g|=BCrPf89p9N4;RkRjpr~`QxpYXU3g}YTWFE~DugIdBwXdB~A4<_kR`jcraJup6DJoP_tD2bgTddBIKh#vU7 zpnW`@b&CFpo=-DDkN%+#8;RqudY|*32l)*2;3rMm3;duzqUZX{=+KBC)OC{2Ulq0y zze&Lfup+l_g2D^>FeAJx;G97Grk;%?v4J#J_~+;!buRB4ygz3eYW_O(hyI`1$0B48 zy94$dtYi=FB3det?@}83QrFlBHwWSKoY0Rxc707v4VU+g9LPISCtr#7@Wm#X7=7l}cKkVV;PEPg!KZOA{egYp~)O&U%MA-WPkPpEBG6J>; zd1PdPz%T34*cD+! z?5X6>SSbBlpda8ff_c+u-GqPf_IW{=W2i=(Nb(Kju(Gcn0hS8Jzd-a4PrDk4XfgVk zPSRgQ%D1cw%LGkq>CYy9;9rG-hacwl(!_X$f$1~3JkH_~|;4^{u2>gsR-hU?Q z0-vny>2d*&MU=*G6nGcv0-vA@dA3M=j^uxf#B0D4|9*k5n+N}T_+gTd`#uMyjm)k!R_Kgk$6X?`2@&F5+q9sA(>r4R=HU7u ze>Wbs}Dh&q^QQ{HE!J6o33Sn2gqMnm;;^mpb9`<%U03{Jji1#Lt&hoks0Py_0Ta z0sdeY(`na}STt@df^&-mv^j+6We?IAxOkIxrgcaigneC72@$S2m5 z6hW_{J=CqKd?qm?n2b6Y>L>rC{!%bp6mQQ6jzQgsr|E0NC!lV`CnDW4;*(I1;VtVK zU*GI58Ba#_?@EUCn)`=;3++$fANy!>Qc?ba`FR#qvM=n9d_R-iQN%;C|D>4x;s56O zIgitGd(xbMJ`00WbXsm1>F>;cr}b+}BD)gd`ORQ9{3|&#>R*HNuph|Hg!m|+AENz+ zgZoc%bbpeS8>K%_`wd4jw-@;j-=DZSeHPIJpBGGaMEwc*ZF<0+xvyn97udoO7 zqJ0wOJIJSP{ZE2dzP_coke|7u{XzcpK(;;!^br^4^H61KdLT!iO!bMFKo}q6OyoJHc%xFBb@X78QK>IgGLQKEpAIS+8{btfq<*yU*0Q^E5YJOH?ekY&~g-7RqVtNdp27D6g zM!XBB9;!Xz(@{6#Gscdcco%{GktdC%m4(I6nk<9rIK-mWMRF=aw6hA^k-Vnu|AvDZ`8A}UPtNC9_vwTzcr|V-mqR{z7jsV zUXPlu@cC`YS7Jsr|zt1zA^;;ef)eRi`G{jFVUYRA3?k{&&m;1qyEh*p|M^dukIXj3uaWz2$}Ujk{`)|X*$Kk)51ikZb-p}mygF)% zSBCR-_|KihtLXk4>zjGMJrAERjJDt2&Bamc6OYeyq(+|?n#&ZQ zuzs5M*IOw*CG!0_k7vk_(s$c={)GKN8hai2>+Xa^iswn=(;So!6$b4Tt`m5^i}g>O z7aiTe;>9z>i}e5C{kJK8A)nF%{L_c%=MRLB#xwArj(A4>@hx`#W8L3k{@%iVdy+A} z6$Z!Y3?E>L{UObd%yHvkA+i6?!TvjI9DP1#*ndx<{h}e>IcY-@+kZQqqwK$NphD$N z;$7t(?7!2oT7p5bUqyQt>e2lu^0jfs^CsXkjOS70|Cy9;iv1^_51G;ZC*OaQKcPh9 z-NcE^jXLG4=jTD6F5RF{*gsvj=+m?x89jgf1J3tL3FiD2`EA+0+lz0=j||@rNMoz$ zu!$E#K1)l2+)o#Dun5IN{)Y7jpBMA#Lm{gts3`r~c{V7TKQ`JO9tWHF?eF2Wjl{;79Cd(E-kX%DO&`$BH9H~ike)<(X17OoUBJqwQJ-E}xB%(cK0Gc+N z_ov2tHNlR0acXfQy^2yjcCBplM8ONF(L%v#jH|2|kfIS|R8h~55KnxJtFM9)>Wf~Y z9p`O~l{YkFG5p>XE1uFFeTj4aRkj76C*galDM%1|v_!BRE%ESC#k%6Rr(B*FcYHjO ze^J0dy#9HvPZXZ`c#&%X55>f5ALV~bO!vwL@#v%S_!(QHX~{Y?9fffS*^`Rs*`U)T zkHsH26}KhXWbX)DLS)A6X3)EyB=jDYn?{O*kILNM$SPq=QiESd0d_Ag0twT=)`Dyq3fcu)|7ZNR_Li1 zZ8T;btqM?s9!KDb8`8lVU0?Mu=AN5-P!Da+dNhv>SXGSt2v|{_o}%YE5`V{63T=dU z(b#H5v91^~o^GT;)o=V2DAsSv_XhLJ~T6Ag^oPK?Ky zo?7j^$~V8ZB<*SsV|dsJFEktq;sySwL&E{a7gZhKXff5FLR~RQ^cYm9I*cab59)Nw zmp+{p&#>(Ajb>mFzkuivSnQ$xe^PBory!L96(9{~Wy=h+R zNzr-f77SS1$J4fz=CfWm>P(LlgRK0=qgQ!=)Uh9Fsd?=(*PjAjnMJH!Hp|+@zb$N7 zyUg@&TiP!{)@d01aymlE^d*OSd#kZS919BYcCEsIqJsVXXP@&Z`sOK2! zgr_=JrZy%O86NEbY}!8Gax6C45^m1@-h})6&_B#vjkmhr?)<;=Xz)plZsj&!pLfoowf8@C-854Z zN1hg1`H4qw*$xgL#q7Pz81>YweM{F@mtwEY_}RC172ZdO z;n;1fGU^EYvl5T^42zCrE$8UM$4&Tt@%=cl{g3rO@?@M#VCY}hc>mpsH~QY0v44qI zX#QtzYjmyLh4TfPPokW$1Hf$f&9`N_i{K z|B3r-pUrOe^b5H!e&>bpE$q9qe)HRnw!fJ^duslVvww6bEw#KkWgdq73_Sm*3R1iA zM|Jsmutx=W!t;M#Ue4H-GW=~hUN?IF4=IRjLDzWC_96H$=nch&lc4BV>rhwPeH3th zFwd!O{5F&K!HP873H2p2C!}G4Z4qBT_u4E7p(l7xkPj zMT_RVi05==Dsc1~eNLBzDEcVD9p2DP=f`xG_8wWUkA9#m0I`)p)M*nCulDu!aECBKlLiHjvw(tc02 zKW=DCw^qHA|4@66g^dPjayM{P$n%JIYx{CS;_I2>5_|uWY@a7VO2^jrw4Zx2PWxk) z_Nr%V`>8@9evcxM*Hg0n`?5~s(|1j*?aA-a#R&fUiO$-d<~gar;--v|W1@8DiYv7znr=~zzT-~@c%LOvh`d=M zuQaGE{106Xbpn3}UPnAp|E40_drUx!lL1HT3-MoUp~EBmdHJcK<9PP-!!hyqb!0#3 z@uJ>eH9S6zzr0CRD z%`bIT?Y0%IUEf}`tVooSRmJn0JL?w|H*Z?7wq|=#^E&T_hR#hJs|)H|Hq`7WT+`+) zDp^&szPPczWqHTujn&Qd{SXh^lm8J z(Ogkkvvz0M^5*4_)inC6*EX-1U-MY`&hm}(tD85sEH9{AA=(xdSe)(5HdeQ9+PM0Ol`X3a8p>)q8&)*$#2BsMqilWI`p)|K zOS@>ih7A>M#pOFo>lUofZ!X(b5goT|+iGvo(zaz~+tx{%vQ4e)TQ)9Ozx|fz<`;}c zw|sL;uyON(O6a4djAY)rnRHjsRyu#%8gKFHf=w;kmTfPp?%K4WFc5dl@|_zuf$zGq z!L+s^Hg7?xd1`lV_6`ELga3;ez~p{Co<2MrFCTx!otVux=^dt7aNc>C3S5m+3d_nX~^bMRqyzL%^p2j+2pnpx>uW%k6^L70NyQ~x4_Z1G; zQ|jJAHwEx>3WsZ{x-WIQp3}c>Cp`bdUkRsvc`@Mg48drI1yA&)27L5akybii9zz4X zAXZwUYrX|K)xGulBHG&7d+JESgq9DEGr@!$O4%_Kv7m{s9tdEAxYXDQ&La;8sQ5sY zc+Y!Y$Gb+OI3?b5rUgBZh$C9q4C5-KppWrmhRdCh0^%y@jXQ6FY{MnPuya##QJ5sS z@_(T(#=Zf$6ioCW-swx_MDLM6<@aGIG#>q`8^C{$_VMUxe-jVB_y+Lr15W&$;dWq5 z%`5E&{J$ajuaYKsP0B%I#vAtl5;f}IElr^2R}<(7enrAn+xZf%HmRku|6Db$G?~)# zKS=)ZbSYlcyn0R2zbxTuK0PV<>6Zpn@C~3(SA(LbHK&{%0JCr6O7!{Y zkIG4LVQ=9|@T(GTmvANDIP|A{hfw1seu4#_FX58_S8WOOc?q8kxN*z4+WHu3v>carG_Da|D&?g96>mhkQo^nMY?Sb9OFv3q)Q|C$sN2v#J?BJ~=M8Wp{qY;X zuNvS+dGn;+t$faqaI3ui(m+;uUzc!eKR=altGvH3^kd}zLj&9>Z>F3-G_Em5=HCE* zM8b(5<7)Q2gg;_|_Z#Sq_I*vlt^Ctdbf^$Nlf*55Unt>fT;+?660XKo@FNC#HT;qIkN%)BQBH`4J;)Mj*Vt~sd9xm4r13dc2J-{rKgDH94oSL!E z8Q?~_8YNumLD4r$xK-X4CEO}+sSG)4+h`u7q31rM!g- z#`6}qiVyu3xQYWq7Pwk^)%mCDze?I^2Kb|*;N23w z!U9+8z25>?>pc@`cYAGJQ>!=7#jr*1ePcWGlMByHJcSgoBg;4QI)u-czQZA1I$g6*+xOraYWAxb? z#XA{}qdC|=07&B-b8BkplEK$Q6+=AJ2Xc=|5NG k@MJ%71=U{V(EBm4b!=*Kww>Pb!{btZVuPOKWo`fe0f;LaXaE2J literal 0 HcmV?d00001 diff --git a/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k87a26cbd91074d4c090e61c7442a511b.o b/tensorrt_llm/_torch/cute_aot_data/cutedsl-4.8.0-tvmffi-0.1.13.post2-sm107/k87a26cbd91074d4c090e61c7442a511b.o new file mode 100644 index 0000000000000000000000000000000000000000..9cceb2e1f4c425987edb68431d3205a39d2fdf29 GIT binary patch literal 34072 zcmeHw3v^V+nP%PFEvY3$t0f+m0Ha10PxFuj!Y~uhv>qTN0}=x9+RTcU)F6Q#eOi)f z8FC^-)-eWy8M7PvOq?MTPnMP;VH-Je^s|`e|g!;<+`RZRW$Z0i`5jAvCN*>y@%g= zSSl#pth>73R{Qg8x9xw^3vHjB#@OLOTXyfec%8Fl&-qte&*SxPxJT3Rik7a|cl{vr z+cE6gha2?J2bH$absIbUe{HPyXgfO$+9TZ=tnho;p1u!D*xSF-+2Lua-PLDp6MF}3 zd6lDcrmAGUy6Jp@Rnpwx2jKuy5Y+ zaK-5p;ilHp;obvpg@+%$6uz+I5@;5M%AE%XF3fv!VBc!=v;RBc*)16V)LY^5^OwTg zFVlE`916|Dc&iVhpA+FZE#D3g17C6eRCxBu^WdHO`J>SA{(}Qg&wFMdbm~O-0$xp* z-wMw;c`{t?yd1tT?>uOhhGw@sInWDTl%GG2zE6ZhCr*VcoTtN~d6&XZuO^2_M*^vBEdt z1DZy-3B7~5<1uP?l(E9s0L$TRL|;FIpL|eR68dmMG5pSR?fS0Qj~V?poz*s*AL%&6~1}j_wH`r>%NKa9UaB@j#_+=d{X&ezs3KQzuwFTukZR{C{i-}jt_}TAzxD?FdN7+^J6kg0@L;`m`0U^6 z*N67AKAJ~N>wk0w@R8?OU*um{-=$aeYwwn@zH<+-YyY~C^_@>+*DfTnYbl^D!dx5L zo6tA(&#Z6gA?@(Rf7GvCe3u;_`kRE%(6`v(i}hGj$FaWQA58=-F*NiLJN$1Cf%d;A zh0bo+^C-AX;ZI$T&Zy%b7gw2rOfq& zQWq74oL?vok!{E(WGCwHUFh{3^yfnS3?zH{cA)=n;<^T}3fy<&dH~lGc>OQjpT_my zX?)o0-RBZQHrQ49pKUC=_Ad1Any!aju+K&KE!pYUumRcp0&ISEA?q82{m2f=r{7=8 z`oblwkLmr7l1-KUuY3!(f-k=ZKfVl`{X6Uz^9OeL$|2a*V8_O(LLcO(A6FKIt|t{= zCHvjihgl!b!+GGx{6%lKTd#Vr<{vcXVpr_`r==m_^*8cK=MS`Xv~_g((VTE?t?eD&){eY;zv^)%$}-^Zv$dM!3s^V_y=ZTEHLvD(_!$7`dQ zRm)b@3QTPtwrWk@b{@T>cxs9K<(|sPZWftl18Ku<*;}J$c9FCJhOL++0i-AB`+a0zB#m-K3AMgLvEZ8*QM zmIz9&1kybf@V%R^X{mdhx@G`gX# zGqTh=;ePjN8v?rt0I7fbjK9BsAWwG9Xq8i%eQ%j{mdNfdc6?`Igdf1AfTGUH=F=gaDxgmduv8k{ zUv~^$Zk^X~eY;Xa~4cZ=9@sBfDL_e6s-Z;c&1`}+*ne^rk zULVZ17Y213;Ta=5THIOAm}s7*G8V97FY@=*KJX_%jLBkU~N(|pRCiu#-g zyCmy%m$m7^xwC1!xx^prv!}~3m(RBcr>3&Ma<);6L)=Hz%g>H=Q^yc>A83GYT-;FkeE1^7IH9}?}Spf2!V1wIpakHF7LH zlJd9U?Aatv3_o*bwEjAuFWeq{{-HhT%Z~mzKkwM|vaIJBt*ep9UT*(P*f&~0U!>|} zZ}P0RDybKI&iOKT{KEys6xO4v{^lM*VBh zsTBsMBt-oc>%sKklqp1?81uI&nN;WgwioSPXs`TjHR*SX%j9nk;N8NW$EI?;19j1! z{Lv1)^f&l7^gBJOUuAD%fb9Ja>6`9Rq`ql9yYkP$66mKx;N^TCseg%&M^e1|?Ek0rkAhWMTxC!Qk*CX-10>VeEyydGSjtq9mDavS2iu9JOvd`JERf5u10 ztnw7!-lD*=Q(&HhLFw;AKJ{rFDg8%n-9%6A;jUtRxjyvTRRekl=l4g+%kCHvgm zKjGgzUv$X)lH}n<h)+biWyB|;9>ZJK zGrqprT{51G>feoxZe{}$Sxz(4lUj;0u_|!h?j~}M7XTu!HB4;CM#9v_# z=tcV^%6E`Y+xnjbuY7$=b0I%-NBe{P>4B;GB+y4(n9oC%sp)}ieKOT2Vgg}&jF*#z zI^S=iKiD^u+{I7x0rD0GGc%*{%*qe(--GdP%nz+!2i?HC%BSF8C=?%G!j}+{e=#@e zq~CK3P)E8z9g;j3X#eI&i0PO7BRRpM-%MJn{B1Lw zmb^%k3;xmrdMcfmM7qiThqUFoAcDc_-d#!muM{z3VB^0Q1>Na*WM@|nC(HJ|Zw?v;Ld zJ`?r7SU$tgC(fnWZ<@a#zTGf?`7`(*;@fAPzl_XR9LcYEk$#GN1oO@1Ab;fNEAUTq zzOn##JoQQZ2{h8Kttk0(P8}%%#*HL=3$9fdoZw+eZ^&0b) z@X_^p)O>}{Z%e*Hn{g__ANhVl*&p`A`D5lpU?VQf_tT$ZzC!*GH(!}HO1|QrcJq8C zi{~ru`1uOt$#PSCkLD{uL%uOGUqL)|XGQasA?WXu=ObCPzVdj9{w(4+NQ=AUyxT`F&aEE2GA% zqo#OeIA4eV+)2EO?!U3VnfKcZ@cF`M`|aIa9JN02_)JG?^m$PZ#ghm4e4jWTQVyoM zOz{cpr)ht^mEuz(-=Fh%hWsdfx1Hxt*bk(!H;}*XPDrG9o-{tqLHST&&`#kxf#pr2^ij~~M44fctAKhFK*Khu6ZcHV>Wuz!l3 zUt#?pxgVcA<>v8i68HDX@#7of$E3-8|E2chBjcOt{3ka!d6Ff*#m;}M`&-Q4Ti9<; zGRC*U;5eP(11zyWr1_CKZagd`_TSmqe`k%O&*u#L?MvV{WlI& zsN6}stGt8#cRE%}Fevt`XzxNjx*tWpHqLn71bl|^Jc|53lk!cm|K#%_GrIre`)~3m zlxV!0IFY$gr+oGN0_fAF8}teLr|TAdn)V~3=dXXj`F<(EoWCN!E!%f{@eTQr;rjt; zY!w|g@nXnlX-Sa#>HH2Bp?JvOu>RolVt(E$&NB?>;YUGq3+Lg-a2^iI?!@J+ zl8|D3wUdMJ_Cx8^KYiRH`_p3Q|9Q72ulf8SjeQaPi2W=&!1+&E*GIAZ1fL(6@{`Y? z4yo>!`x>~ff%_V`uYvm-_|!E(hvx{j{rHvx>;A>{-+%a{$X8$Pt@`cezx#)uSB&}V zhL0}%P2^A^o7MYtkE_PrJ16H@Z!cqg_L}F0-~X4|$k#Y3Wlh3g_2%^S_NK5iKYsQ1 zzl*$MXUl5VOgpBpsYzH;8hdaK@K*_-1l7jaQ?Bv| zUBm-*tghjImcjcNXUgxey=82#wiJO}t7D*#6+Oq9sIh7QY??iJ>fiH~vhH84Vb z(Mz=Byp6H)hGr~=-AHOsfeBp zI!*Fe{DD((Tar!oj<6*}X54NDz3WLr?@_sFq&WDf+^+cesN8NLcf3BLa)A+&RY`^+ zQ4EsVKNZRBBRz^NVd6~EL^*?WH{)#N95i!o)2*Dx)wwP>i;jv;jJ6lLE-GtHi8o_~ zo{G^%W7g5C05#}w1fIAd9jwvyRWmU6+}wkDXmi%1d2GO{V&q4_it6+fJ;#ywJHAqA zBfN{oRx65i#fb5ABMqv4i9;Bss0q|ibNFnp&zA`FzfqiM6Nrz0(>i&??Y8-(^hK$q_S@|v zUYlFt{S04GM;q-{dK#?{D?VPIpB*<=?nbK{XJU_xLEOuPTalmN)UV>TSCfRM)~M}G z^HNWW&P%soz}h~ZwyiXu_qtJMdYl+!V}Bk?y~TiG(E?K*zDOuI@S=FvC`u$wkx0+v+C$#98^Q^2DQ8n>$)@L4Q~8 zFFF6_v%JsS+1}?}zoz+Q+_8yRAB%c>N9@=<@Ss@C-phKB{&kJ_->rC~@0}U@ zmw1Kd|K_$v*UDWuU!eIU${9NjY~nn_NW1AXj25ZpmTUJ40l-H#5a=vUJmiCns!F4j zxAOd-xX}F5DnDf$iUmV}UzBlJLzujp2>zQ+><^5#pj}N7#mN%y?z>uGV=l@he zYB&C=E-x4Mr~prR{?9AR8QW5Z|1HPsM$i8t1(7Z28t>UY1pg(yq1bQ|6#Z%)>Powh z0?rTSIn|BdX7fJy(U6H~DTj~jK|H66yS0g4Jf|ze1f$ARm{K+-p7dfwisy92`ceL( zp3|jh(VQ3YoUTj-j$WhB>5>pdA0@cM8=C3-n9kDPBkT3i50nKUwlau1Z35!8|FLZM zC5vjU?eWi$mNPEV?+W@SO%#W0KfC!9S@;nf_VsO zC}%BGOpUMFDf>T;#y4xP=%_u>S=-Y*Cly%SlreHll?b{5v^)F9;=1TTam~Vom6S9- z6^r_qPqz11#y=+8EB~eT_0AH7)hzZUKPOC0%C`zx{~+?3nj zIzIUX>2yn-B(z*I<$&s{+7tLr+wmwa=ci@3=zLZ4);J#Vs^*2YovM4JgBaU!kepA7 zPF>afQdiY(ThZF}?M2IqL@8NST+rNEzp%J@)55hi+l!jlc{en4ZrWI#U*EE!W=G+g zHg8eMs*?4^jrA?dJ2r2uZm##QUDsT`b9sG1BVcQ52w&Q~bj_xX6^(Tp*5_^BSh3Q( zp>RiYMP<#}on_0Lmp@k1=&xSeyrQ7yvGSee8w;wNH@7U$uUjG77B??#uG?6g=iAKt zUAeJfd1qbA(me3suPazwv%Prr_J)ns?VC2Peqv?I>imYXn$Ct5%{wthEBGi|U$(xp zzF=t=jn}ZDqOG`mXKCHS^?A)@+bW{tmTg<@En3>PtZdslNmI6|b$!dmh3mK95?w+5 zXmrarw*(tEFRX+&R8@5$>i%V85FI%0zxolbIrk3W;&1HBkT-CXGMTNhiWqlX) zSH32H{SyrD#Q(&mrbUk`e^b}m zmgc5F?T#gncncQS*EKA~vq%>;EUM32n&(@bU;oIWMFrl%{QSCFL08|j)m!ho(NC?JGqle-c6oOp1*L>qD2ey@|JL#<=*CYU#)mAslgA*_`EH7!92!x<=oYe zPtN%x_pUE}3C78vpU+LC7Gv#+YNqX;s!d7F#)Dk&`;p3kxaJz}=i=VD$9#$wTq^5Q zAM`w@AF=<}#Q)~_TIx5PFN&UtzJU{nx80-A(^y9g^slS?70#n$zM;Qpmvy51fx_W> zO5I!NrT~6U;czWg_oYtPbNV;zgy;YGE8)~HF9v+BAsEfF;EBG}fRFwv(n{wmV`zXE z#Y#(b&9^|Oy0>0mLR&j~UmZ!9(DK1?CYX>zDLaND7BunI0|9Ihml`|4dF0^$6(6V) z?|IMbc-LqYr^I{Cw4mn^aYPH7VO)h2^f7+SaJdswKwJgAapx_NZMb9@c5X^83X=p^ z{x9^!*f$}Uf{7l)JAJ8~=sgms{5}kY#-m?#1NiUJJ{~>oZ{oopxdHqKfD=DwxE&Z% z^Gdq`|1Ti%BKj(6g4d-SG-kYU-<0yYr3uvhY63m=b49{c+X4w!o77U-e~#>@N}5dR z`R^tFc)AoXYF<4j`Fusf)qHwg_R}v7sNfp_PgjGYr!}Zk!Bc!We2c9AlK``C;Y##* z7>CM9aba)cO7N=^ZkKQ+-#GNAe1}luC4Pbho+sgx09S1Z^hF7u47hR2xZ40Xri#ZT zoW{jzVzeBWcQmdMUMl6J{uOUTx>CZe{cM!*sg{0}zNjDLDN(nfe|pY|D$g6>M*8D7 zfL}Ghjq>J7y<7R5CE-?i`=x=b^1dPA)_#5_;Z}KnY3Rqu|3?P6QQk~Be`s7|jLf?M z{D_1TKgQMUc?o~m0`E7_8}0kLgj@Nir|3{2ekO@q{=P`U)ws$R8zo$gtKdfr^u}?Y zmvHO2zm#z6xT$j9(zwR?;*#(Y^F_j`AH@p^u*CqEM?74vBL;Z%pL>8=BnMOSx;Zst zUogOpay3f0(u1OJmT;@QFG;vn-clKI)VO(a+?58naonJUt8o?mZV9)J`#lM_j!St9 z70O*`N-5n@T%)4k9$sUtQC<~X#f|4Ja1|f=EpQbFhAeQk^s4hu)xTQrXMsN|3f?X4 zx55Hf>%HFsSL;0!X?J^VT~n(!(8aJvXz;G|cDB|xKE{7?av2`Tj9+);zwtyr*T`;l z=RqCSHnle4XCCX?+i$N~YiEmKsL9vPM&IPftpopg$+K#r`2F{oe z2&A`ygQ<72fj71JAKTS$x89|{>?#)6Px!tUK}4on zrk<|K1a-2^KNM0f)2hm7pHDEE7DVA5d3Q#}F@;d^O4VoLhEgt%0?4a>r?`1u=415P z8pS&qj-xr)-vdbF8*^)F>5{?MLlr|&q^}mqg->CgZ7KGwk^_LFetu`Ut%7CIUU#+y;W&08G%l6M&8d&-F0UytP4(UHv lvuyt%|a<-k`@x$X%eqw{3J~pM;zwkUs`UG?)_-AYgz10kK2%peSfjsHPoeq~9TsU{dnWor4XP z84t?zVv7~4-yP>(J40ufc?`vQnCJ0(D&ONBL&vYBE$txJdq2NBK3B?c9kkeDhTCF` zp6~a*YrSXh^CpqRvA3<)t>NtTTfeo|yWX|lcdh;IefD|a=C!wkLm?wa$b8HADM5`% zU-Zute3~%DV1m1^)|kgmM`{lKGycv*YUce9e7=sqf5ADZ6*ck5f04g;?|**q7e(fs zpZ|V%@WQ&t;JXp?*o8=F-v5l4gZul^4Hxo`19Yt5#UN}~N+fAq-7&+q-}$!qX;XzwE@zY$KngF5!%@3FzX z&kWCd>zUzip8MwTxov+t{6^7_hQI&T4~9SWx5tMM75`}X;X8gh{P}0j4S(g(Plp@V z{&cu-*0qD*d+RGF4|eT6Sv-H)U~Ivm;Wr;SG<@rVLb9e|i27hrieL`tX~_-va0N2Hz<9>dD()M?H6dbNrd%?=^mB_~6^4jy{sZt&sxkDUB|u z-$6ak97msjinghq<%74Poi~p^a&qvQL&NjF^EbnT^N$a|37tGV|E=LKLl?2RD+a&U z^`(<<{$S6^TT#cs$D|dFj^-V9ftLC{_ ze)`VX(V77o@BQa9OA9b&p>X=n%_e>4)KKD`Q<13$DR1a!nWaC!$-MMh7;^-9ry|o1 zo<`n3ySXy-)6CLi;lzt={`)3XQ)%AmeS0vuqR%niN-DZw#`$KbN|4lP4nKcKaw}E@`p_=E+vHi35 zKU96mtUMCkG|#;ILd@LTYD~*dU{5jF(ze?YM;>q(wHqaj+CkY*;j=iFzO$qD(0?op znImsP)|!|(vL+OIsU~C|`w7xrE6q!Pf%8om2THFERbLcrvtYY?>;b_-GqSuk!FKuB zR>4A7XL+5#-k>&iU48KywEF|7@XYkkAj;2sy(T>P##M&PzE_uc2mHs# zKgbwn{vq_zd+;}xdc%XiyeBevF&Tv~$%F67A57Mpd%rQ?v^1E+k(#Q|y}O|2hG^oQ zf>!ew>9uBtxi_@OVll{9I5qLkl2GAS9+;9CZbi9RS>nhIW#-6{^LA(v&SN-V zY2L0~Y+B}mcS9)r_R1xurKl|N&eX|?BcK24#9#;TzsNUlry=V$)bl_r!JK3${`RhB z(?VsAE;TJRd8qSR$X*)gTXAmRS|3C{FzMSe^zF6$#PCX#gPcch2u4&<2!d6J=upM-99a|Gjgl+J0695rwM_!gXlmkfp8 z4z0m_fnI6sYj3sl<@2*v4t4;`nJc?izP%FpNz9pDx4F6UDK}ScN52w*e%I&r`)CUJ zz@*>Tq2D)9zfn&5{f#?t?rh@dI-JW~`83X5ze9K8T;>YlGFRw4*l+OIHmKipF8#hs zYy__Dba40Nw2RtnJ^?--rC`1O02v?O&)D`M{)qE6~5c zm;^iDz;dxI9Q^~FyLqHk&As35Lb}gSKiG|Q8uHEv-P@YP`6|eZz8>3T{;vf5{Es73d5;u<@BAh9PVk+) z*9nm?#hxdO&l~Z%#aO?(345uKNtZXkubsj^i+pW-lUe#J?4urtOx^a4kZHNJ%?zH3 zTygNnSW{19ef{Ui^m%6@l?VR^(yvFVu%D}#x8l9bpTY;vyMpR*GMxIHSvsDw^HCP- z=e$#qt628g$P=1(srSag_ptYaFK6GL ziLk%t{z^Yd4_naRmCz0Lppe<_K^r2bLUg0K@bUY0qYTw|G&DZmOXK8i3y-h9YfvBU zc?z&LjwL6Qby*Jf!tY3bZk{0jx^eL5kAoAGp?-q=$_es|C&xLL;Jx2Gma+W*##!I(KqVzVxpeHv4bU4;<{c`u#T;hw+TP$oG5WtXKHbpR?mEml!8c*H3f(a{5{} z!8!No3G#2AApd_yU-#7s&NrDq8zwj}uz*cyUQ0XIkF%YBkNU@2BgJ2mUHyk~%!!Gv zspFY9LEkx^^%DEE%i2${&i~bG|AMvh$JkT-zaMaHzxx5pN!v%!_H!Swj;B9h zIhsEc@y{}k_oCe8{9$lBwLd+O$xQzs?e=^?c{iaP?d5BtW>Mlmrp4Npy()9w!gZU|i!=I8L4(53;xM^#6m_{|(G-#K;e>{#E8rN#92=`T9n? z)V5jw)@T0jkSR4Ck5I~(unC0*@Od7SO^!7gR#_UNcE7BXS%ro-WoiA;`!!nl%(7GWB@BQnLnx0(m+eS(#e5jKBMvH$Gx%2({^ zhCi{N;cX^AVV0XbNOUDJ7xZ`Zr@U0_W}M0Aj&7Ug&t!i`PuqZ5(9@f0U(j>kg4DLo zo~^xpddqf~zHdSHWJ_;%Uu()NNVN~77TB&Ow`@;!wf6TX+uFDEwzVhkY41t)wR){x zUF}`T{&qS|wI{vyw(VO|oxMFtSE?`NZC$kNp6>SU)~?QbdKN890_j|`I7yH@M^)`! z=N3whjPN9!t<=BD1Bt)2IDw70gkb#^aWygZreO|^C$m%e>|W&kPCnj-U{mz*Hsp6|TR3v+ z*C&&mJt=JEv3YDtBDH0ETWhkvdvR0J+tQcxdXm_DCi}a3zkuyAmbs62Ex%=({Twex zqoaZG?^=HMFMGtN--S!91-)Ch_P3{+OfuPXA1>4E&ik(0Z@xWgb1>g=h1S~7i+whq zN@Bb7k=nw&-!f+AA~%AoYkeS22^x}ktijUoKFuEeE92hlZ%SJ<-A;v&RTcB7go zNiykA=!^n0llEn?$LMSiu!4u_jAWd&H=FwioxKVy_9)IcMddB`BH@n%=Q@hftk2@q z7XJt%d^X&os&@MX<*nUn9)Sx zc+0wVYwt>a_U1e9x^>+h$(uK{+*7n|^IN?w9r6C|6b6>rgy3D7 ziMn*2X)xgnwb^vk#7#7ij+n}vbQm@e{+~WSgq{VZwi~ zj^KQm!?z)f*o}JbK?V+LgFfGLpYOWQOYXCcD#bzbg>)=)Y!!hx1h*;VE56|Wt3DsR zKa?@Ap*5OFcmT0#M~Q8E4n7u4F%RA!&Y06UC)YT0iaGR!Xodr_g($IkidjHGTRFJA zOU%OOUlM;(#@vVV0{nM+ikUSKE+Nf^%~vOx<27d2j&PVYU3GP_nOSS9P#=kcXXzl% zI+V}Pq;VsF%cH%px76?ekeJ+_o5{6EqrLJ0)OHbrF@*Y&fAbEECw#04rz6ZC$jj6q zX)2(2F3%}r$kA;sG2sEUP6B1?Zzi@ev+;-a%RDoQvNLhie{3Dnc~rZ}Bp#h)&Omt+zV+u> z-lYSji|rKQ{zR7!6fU;D2ig!wgP2qzUnD~PV|{V^MbbX>TVnG2^VufN zzBtdnrQYNWY&6aFpDrmV^P*w0S*{=YTa-f8l%`6pK8pzGTYn)!`dT^=jgq=uc~gM; zX{zxOTz@t4Bl)Bc&d2=B>yNO%Gv{k@V_}M^_a;x~@eM=%V>M=uH#tJP1D?-l5Bgki zo*JO>rv7@3lXxjVxfu=GS}CBGPx1>O=W5FWkHUX`WxF7n)f+n@dQ(X7kk& zH{Ma|Z{H?2KfHXpvyqnmpnM5U$V{oLe`h32fU*Pz&28x@Bzu4--{MkT0=TE8#70`TN>Y2%8|Dx|% zJEs(ry)K~fM}Gaz7_Tclo;F=f;}@gxMgLKMnC4SC$@85W)E6rW)>pQ(dGC= zJp<{s5>p9(n>{c;R!W-uFZ4->!)-)`iB1>sqa$O zcj?&rj+L>%cKl}TEHCx-T{D`#OLO&I%KENgeJ><^Uy-Zt3hc0;U)J|J=)1I>=A*Cg zspYx)E-NQ_e0?vh@bw*mzDr9ds_(v_z6aNHZ~gf_&DS^UuZZi6`c zXnz`y$21;~bu=DRbH>A)RxyqEG~Z!AWjl*1a`ivGf*ua=^*_JR*MA=LUoq|9r2ivY z|CPS}S)X=)5*?5JXOjQ5<7NHJI`luv`j3vO|H2Xa=M4<;S^tGm(m%$}`I|iDZ}Os~ z|2(&zlcLYPkPH z)_<&ic*y3{`f;wr#0JbFZqMwZ^&$V}9bvOSV2`ya>J8)}?DV72S}zZLlg1y*Q^fmg zY#RHwTCcD+Zu3$9x@eJS zD4jO6--JCD^v8Iq>{e1%Ss(4C$}Vi*XkJgZf9VS9bTy^9Acw z0r*+{p?^(YVFu#G%}TU)rlhpVtEY66OPjb?Tg!Eu1dU%o+>6BnX_U{;U_A|#pGoOJ z`MH@|RE;Y`9{(mUnwcW}7peU?rL+7xZ&r--HE~#kp8ff#Ht-sEzkJabo8_Exq zuZxjAOZhr)W{hUMpRSG3eDl-sSUe})5NpUu&y68z`J?JLMA_CJc|Q;``^7%b(tdzq zN_5!y5%bK#d7K`@e!S>HA~WUmqouXpJfgdFiSg|R`%#|Xa}3u<_QLiPuqU>kIdklI zqYAd4+0LGjw)Qh;Hj46L?T69<`^o!&_5=H7`*HS64Oshu`wZC6t-14mH2XPEwnKkx zKP_ez)%6>CzYzDy^}}D)*Hb>Pp~$cE>gp)XGIDz6OiJ?} ziqo~VlqR1*{-@51$0s>l)uN8s$Pxd-BkRFe{qI}jq$X-!Cn;y+xu|K!= zHio?-KTv+8y<-3VzV?dzK>7d8?Dfb9d%fl|_IeE_BPIimaoOwa%dD@nXXmc3bM*Q; zJL`|^`by~!VSWAZ>~%d|AZFLson{3q=eNaPmt}B$m7V`hUK6DQ^L=)3JuM)6#`RW~ zkNs+J{l$7)P5IgKwEw+Q_-vocV#~Py3Gx@bo+3ZXr}Z|l9wR@Rxk}1o{~Y(U{Tj*- zl+RsnnR;vUnw12Unw12Unw12Unw12 zUn$LVnCqWAm(qNUV|}Ij9ohRO*pDt9s9qf9!hC(n@qhn%dL7cy#rfI%M>)P$8V2C5 zcPU;=`5f=Z{*~gXjg&4wpN*fAf%ldB@m2B%!T9Q2P9wgG@m)Hwc=05pB|b^4W31sIP2iwI5%_^@GG?i~V?1Zag*^f5r7_+0Gf4i@#R$eBgKp#iKs#_-m=J z|I?)Z3B&^tFBlnrEf2&4YY^`a=pX$hebe?w=F_a5h2^B*V7y!DyIkoz5PzchmyO4k z2I5cF0oeEq;;)6YBlF`=GgSPktPs}|f%wzHfW9gI^kK(e3w?dV-syf_4cd3{ztZ=; zzc7>Tr&a{weG9`m`o(;d`sDsY*o*q}{fyN=_G37pAI$$*JEIk(Z^RF?_ZyV{E3*1W zd9=^*zCzOX>5>rQu~E9Tw(;(o=u+Gt&DDPeKWF0WzbH!idD;67Q==qLUiN-NF#Z~& z_^XYFPvCw-PW*dB{541aA$5NT`p+Bt{ton?qfh92*1%#O5C8toi20tk1pSlwo-;nM z2fDwL)i>#Pr}giF@nHY!-|w;UR;T~S<@H|~&5gfm{Z~aP{_5-hnn3)U?)OYQ{<=RH z-+6e%{&OA$Jk40~Q5Dar^%hY;)jqs|^~jI!AbtXU74!$coo)FvDK{gH_<|qL zt3g`!qb6*=zYy`l{6KsM^?PPSe22z|{C-#$hIq}Ch^l$w`g8GO3=#pSO75hWIA6@FlUtgqnPIR`NKU^OB z%c5X>v@Gr|nQiS&@+bT8RLmFb4`(lhY9v0>Dr^uf%+wKQ!`M6g!oA-h)J{0%z zXVd*rDNp-L+aEv}ueet-n}$mAslRsrmf(Df?+~A5 zeoE_UFrG>F5Qw*qVy`nO-^F|WaTI%<>c@B4UUTh%*4IG%mF+cXZ)C5?mv|EDj~3hO ziv;`sCT}{)8^|xqv1hEW6SUXq#T1ScdmhVPeft`HePy4*_BX!umD~I<>~&;3b{@rJ z#&56J80L>@uf=M8ooxA8dkw6or)hl+thaW3Ew=j`h-BASN`DCJ>xXBrr)WXV+G{1A zJg4I~y&q1y^{+H1{z~^7gYnTOuQC{~g}qD;uAf+63xn}j*jp^P{$jnJ;>T~~s`b{3iWBH-@(;@KZ%%%D5*Jo^fal~J6{Ruwae}Vuo zZ6MBsM%nLEWjNe^pP#SqMw*__$Mx}y`#Bf?z@ec1y?DNdTuWaZ{X4nPtlnPXzCM#K zqU(jP?Qs14_{8&0-;^}LLr8C@6L;Xjgzx(K9DiNCgRgWzu=>*1C#$y?=d}0TKa$EI zFXVf4hgAhoJnw{fvZ^nqKadac=Ni&oUrQhN@8%sL(?G^N^J^-D{^VSp(;)5n2apOOP>)}Tuqs3FWJZYKZDI{+& zo`UZe00=&BMhu=sqxu^j{H#4@zkjn3h0y;+sUm*9p8vu98_bU)fBum_x%~X2u5tgM z4?X{+zOm<<(-&33^KCL$#sBbpE4BnT-~8w0(#&u2X8PBUr%TFdKAxa1JO4@R6ZU3- z{a|G!>&hgkJUu@*J&?wF7Y|&o((`bo{`DuXe^s=e;^6Yg4?LfP_;#IlWsKGeigBX; z3OpYZi|3^4$T9=@_IWtRH!-h}CKQgW{Y=Grm}cm737$VIiU!j3{23+j{HC3sbpJDO zy^Hu*xqp2N{f@M+B75)A*;k!c6GM5C5B4?9*&8et_D1^)taVnORG&AMzyE?FA#-dY z`d`b>BT0Yoyy-N%KL^9wBc%iOc$K|3nzcuIoHre?$097nIrdm&?GgTh{E0Wc5chNK z0r~2@tJpq${a5kuavI}PFfg6mfp4!>)|~<4_!K?wMjnCZAI(SD?qT;t#PYph=hvyBZ z+VLlT4etLJPqhxpmPh<&YVQ7Y%GAJq7xS~O4%e?a_x~x)bB_0uwY8MyI^BMg(%c75 z*VItj->+VE6{WdN=qu*Un4arD%ye8or$Hh8u>T08$$wB1{=?c6`40@GwQsVQelBU( zUtW)M{YB3D%l;wo{4eCedhPr}8tpHn{k`*V3BDhWc${+&NP}PNO&)vwMn3%OmDFE< ze_mBZX+EgE{(5^sko6~pbU7LDCxzrMgU^>2=C1EkXVgo5c6}#z<@<|K;yZi+BJ&^q zqV)Gr0i|Kjt9IGG{RPSUA^Hmv1jj=Bx%HmR%dPiW`<{ru;2PCZTz?T*U%C9%)0MxNp6f44r(2Jip!p8}kv~1xUzAU$^;h;6o1%?ONhU4+k2$hUzCl#zi|G7`Wy5Yl;*?v3%+*u{RNWa@fX?shw>Mf-(PThWIt%1 z_7{m8a87CV7is$fQ|m8A-CuBd-(QftAELkD)jR7iXn(=$3ETVc;{Jm757z#%{*ByU z@b_y1{(|=p0e_LRzu^4}9CUzRsP+@}Q|xIpH{q5{GjsDeuZ*ulmxSk5`hv@n| zaJ{~e{VAqWR)2Ngw3(RsIsS~&>@%G|qqOs9v_F_0uwS|#OKE#O0ehnTMZlk8zc7`e zFZP|isE?l4;PpCd?;LIp$@4Psw|uFawz>(!k7 zMRD%_V)l&LQ~@2lzoAL;>tk0fvy+wD`!dlykDxU z{d4$>@+y1(fcr!Kg3?)kfp|qNRqysg{Cy@$^BAzdxW?|C zvi^d?BW{0y`G9x{`v>12T*cuCe?K;JCe_C}XMa)0{)l}T%H#blaBVsMBBtUS)z?r3 zbg;j;{QiRXdgLEaKkYB}uf}=SUv$`O%Q5#CG27ql{(_Rh_#@vN$caCaXzZUqVElr0 zPxAc^#4pHSVExGMCwzZlTs&izAAcCbUtlfC?k}3~z8R#m>tP(%x2#JyzjK~vbN&KT zF-~le=sEr9{RRBRmAU@n$}1=4FQ&Zj{$f1-g7*r(zd$k_^7j{` z`wODOUg3YNzhIpQ`~~~4_w^TYf1*nHiz@7A-hY3=@=}A&U!3?9&MD2;BOC3#s4?Rg zF}pYN{RQ>+Lwo*nwEGp4Aez6wC{Xt+qW=9)>JRRZu`cPKQR5XAT;7jw7Ehvc`+g)U zKhgLE?=-oK!9 zVE<5BOL_i!Ulyk{_Yvg_`sdo~`K&*fj_EbhANczhyMOTaBi0}Anh@{@{{DgY1HpLa z^xXZ!+(PSbAr$W)<{}yJ2cz8oU|R_IgH!8JmeTAGHW-%mefAGr-uDO8-w(|n;C&}f zettd}^CSEI47dJH3*1kz*Zb$gkUgB{`$hCTOmKb7eLqSG-j721$=U zq)9QN*!2kGU18r%LHk{v{{;nseAvr~`wx@Y_uKNg-VVGk0@vH>eh2Jn+9VtAPLREh za=#*f63-o1AKjnC&;|TU;UwOx+I;L^uegFl4 zi|%^ezW=VyE1pU1vk&3tH`vmFU_Rmb6uMu?4kTbNrFS7aYcG3|po6c^M~&xk`K-Or z*ONZHc;18eL*1v)JPkVLqi-*1-fzQy^82Zz|F?PNeE*lfAprk;n^(&BtNrwpn4KT+ z0?^;}UcnXQJ9$nZeVbPtv-4T=c%?ABx{E*Ja_qV9~?tP{9{dT1ie|X(~|BmK=9=(4P z`oZ%@d8A8zzbD?;xtiae1HjGKoce3gPWA`D0s+vFJ9pAd{qJ7=E$}r z^EJx%>tA4drTg3ABD&w_Umu2x^2v5=KGjdJWAp1@y<-mvhXeY+^HX>qR9*xNm^_aI zd$iAoC9Hg_u>jb^>aTp@YP|mR0mJW$V*9=tw@T7CWG>-(9;E60O?>Y}8KiTV*wC}iJo;%!=AqU8fcxS#0c z!F;bmebjgNzE-;4r~Vt35A$^rzW+)GC{g-;WVvU*FUtKv`K2D~6Hiy5KV07ou4)F2 zH}f+^RDbLuzh4pcXJ+tx8QRP5TcmWD+8;`@KB`51&-d~BS0~~9ki^FkC3^C^I; zdii!g<=!WX`NiqJ&Gvme+XAj zv_tR@y50#DO)2~(@AM*}>!|!@gZli#uP=nXIn`H?|4VZt2uDM@z9z0OAN7$uyFz)n z@`OSt@7H&968$cRpAJpVL;A+z8}sRJWpN@J{FHOT-c_ zRLe>GXxk6cOC+rG<3Od2PlrOWFf?NKHpmEQG?J%Fepod>^lPg*lKn=;j zzoFG#&i`8ob*`p5R$7x`Ms6Lx$6Xno57x@-Dhg4`6rvtjqoJSCNWg!tSbULxt{rWJ z!pO$Yq1Yz)U(&&dl8{(RPUB$Sq$qxrWt5V^o?C18r8*iJORAyB<)j)RQH0`gwv>}7 zf}gpVs6^2*C5ld1B6N4imHJpmc0!F9wU2c)!rp$}I?A#Ckk}0WB*LinPx1uH;KyyG z%tY;MbZ;71=|-%%G;3CtOeqZhg_lo(zuLsN6vH7HcsNAc-|P|eI2QMPZ75`YEqT_z z%a)k9XJw!0_NR2NduFRXF$tUaQGSyzw3G0$ev_{`D_8ExP|BwHXZ50ScwP6yeCe(% zk?LLHQgluSt{nbNzkS!|>XY`@bFkRSGQr=A8}VCke;7Ak{`$-VfA{r@Zu9KCfBnUV z$Y0NCtZn+<)!%)zr1;jZ!iBih`7Qjtxb>^>TXFbXk1+|~1{v{tarI+u$fX-Qa`E*peSC8+e%_@wel8bJ zyY!{s&&A^#Y2vNedEu84@?U_1b0dE^ccA0>(99WCjPso>=tvXvcGFuW<)-*-+za-H)5X*%;A5F zalV6@gWKOV_3JPFqRp1yH5D?7^I?D2)G1do@tOTyQ-=vZK>0wYG)&k8jyMty*;0_3 z<%Ts2(mkKi%753DJ482S0*5Y7)5oqi0TNCJ=)MJxxGO*$x}PfIh=w*m5xPz)_axmX z!7-pRRg#}ka5wIA#3(JdCzKe^yYg}5(LwH?WY^_yrA*wWv>*Pz zue$OSGonLutjphtyf2MX-o0GQKJU*p*nf2STC0%0M1Vu5P0p?_Ih@*`j6Yp|4**m# zLjFeQ2jYS9bSmX#93Mr+ge535aIAOb1(NoqoQ(f{DExlq*SHF(Jn?jSa_6K1HBH^a z493Xs5L0;?=QRF;$5|AnI&TVsbS@g9M%_XBV<7~<=ig!0QF*FA=Wyi{IiQ+k$UA`g zN&XuZPuhPrQ1NKv*N2ITXI`ka(qI)s{WjG_Mu(#cf7JgR64Ez7anwGJoUFz)W9F3o z$o}iN*)+Xl;h|C5FWzM-O^!+J(-BZ7r`uimgld0|74W@OL_ND2>WOnf89RW4e9iiuVkytIjM9F%BPek_=A?? zt@OyHmfID3mCs;%B&*m3lSla{K|T;`(2)FZq;mEj#+!Z}z-iPZhX%ps#&P zsy}{Rysy=3?QT!Cd(P4)_MZ5b)~>G3o_jtK-{iHoZQnw_be&AK_w@IA$=0^k zJ``%`-JbeHyoIrbyL*}%(4VDraS%}sY zI>(*t@9fzYC_-{@SD3cByL(ZxAG&G(grJRafQv_fZJi#poV<5Yva_deI}5SBudDs; z&&}PW#^d(R{_fV)mX7wecnU)nr|jOIc&m5M_HM|;={j*j$ZL$Z^9X`(l@IK*tzFw? zuu)%YSLZ!F+5C80YpRv20W+sOXE2#s`sAWz@y$C@?K!i-n1Fn?J-;elUr$?S_a~bf z_5TUT+tz+x=a%*)b@P*3@H^_dd>iMunM0ZDwx{B~TjQI1xA(Ngk;ar;(BIM8*N*eM zn?6TSP#>w@_~v#@EIYU2UTe=i?eV+kFJ9DqZS(Ra%V@S_&#$F*H_$h8_o6ZCLx=8O zZ1YBKZ_zSy_mcO^U%Z^kvFs_YvrQ>DpdTy$@&`0 zUX6EicDU`pKy7X9192(nJ_F?;A5Yad^~IK1z-59(0(_ni0a2J1$hYIoj=&gk6WIZF zmHN1^yt;sbg8oCw)-CZR3be=aEJc?1hh8b*)750>fH5zpgdbPm8rWHe{K>7^S5>`Sm}AKJK|e=y?BZoW{!MZ!Zztk+tD9Q`Y}^)V3MC z);nD8h#RN$Wt%O0viQ%Y{D;#2SpHPSpSuoFy=SCSe7fy&02%$%gPc+GHpjCxBQnYr z@l7Cw132}Y4m%A*Nfx7jNey@+AqxyVVo%>i2Gt`YB>JTp>KVB!x4EOmaV(#AfKmm2TAX)yqu}n@ z$w1C-y=zgpGP7O1BpP|xlFJ~e>v58${C z&mPZ>fv<83((M2Hg)!U)#ck*y+_^GN<7ZL$AUykhaUf}jUkjYS*llTM zkHbYC@f&*`E_xPxUx4qr$Td9?;0N(16<_?;Q;IM6eupntavpFv^^1JA9M1)C7wWcI zE@(}Bes10sSUv_m;Be9d&1dnqI~^|bS@5*t?{ofnx8e)FNAV9k{$9lw{D9)OJ!(7f zwBieXNb&m||FGf<{;c9RJN|QuFZlC{zsm846kqV;iZ61WP<+8(R{SAX|0{|w_^XO9 za-LFr!OtkZ$az-r1;3#9BIiZL7yOdqi=5&dW&Q{rbxu>}o5)$-=z40cPRd0$L~^n!TS_nDSAOFZe5pFa3H|@dZDn_|mU4iZA#%#g9K`=hJz` z7yN?ai<}n~U+_zcFLKHSh4?qYqwYdO=5v9Qr@-OD7aZT3%$ffioc=2mU+`+hPq_Ny ziZ6J*;!D386kqU0#g~3HDZb#%iZA_IuK0o{6kqIamEsG&R`F#Xu2+1)TNGdVwL$R( z->CS~uT6?Cc$?x&zd96O@IJ*CyGtp);5!w6lN;Z(;tRf8@ugpT6kqVYiZA`zr}%WYZPDPT&wtkuUCALvqkX*-=O#+=SIaByiM^%&JM*Fyi4&#&OXH#Jf-*| z=YZl1zEkl<&a~nSzE|;OU%gN91%E>ErC(1fzTi(O{tD;c_A9>N2Nb{IYj(YVTJZ%x ztoY*RpH+OppHqC1^LfP=d`R)d54@=Of*(`-gp>cc;tT$=;>*6}6~!0)RmGQaKc)DB zpHX}n_p^#G_yxrmJzP|L!7nMk_<8$MEj;7k;wpki?aMc^*RvvLfy0F_c(LNQIDM8X zzTnl0FLK5eUvRuqBWFJ;DPM27raIB#m{U|e8D#=zR0;r@da;Fe37$5@dZyQzQ{SC_=4|Le33J)_=4|Ne35gH z;tRf4@kP#kiZA$6iZ62RSA4+_D89(~wBieXNbyC^!-_BXvx+ZrKBxGCKd<;tPIC@x@NhD8Arl6@QbH|D56renIiY z4_{P#!7nMkjEgDB^(TTy9WMJl8J7aZ7ra98MGuvVFL<@$iyw|FzTowWFLE|0zTl0D zpK|(aQhdQzD89&TLxUa$CqwryKOoOl%X%dEPQ@2F)2e>KcPqZgxkvE@->dlI2lgqx;7=%i+RcY26<_cJ ziZ6P2TJZ%xr1+wT!-_BXvx+Zzcuw&Je_ruL4?~JC_;JM-IZr6Q;4dq_$oY!m3;wF& zi=3wvU+^=EFLIt$e8JBtzKr8}#TWd7;>X>7{-WXwZc44+6aRMD&4;MNWj+gDp!lM< zV#OES{hD8PKPi5oLh%K!RP{gMoHe8D#= zzUX0-;tSrU_@ajn#TUF!@kP#*;tM{Y_#)>{#TPuS_#)?S#TR^!;)|Sn6<_ct760(p z?EHR8@de+n_|mTfiZA%niZ6D0NbvDM*G0t_{F35JzvSU& z@o$1hugLX>(ys!C3t#Y3#g~0oh2jfdt@!3~+g@Dp1+Q0p=~si|3*M;s(l7ZQyT~ti zv*Js?maF;&PbmJ-<5vDviZA#Y#TPl(D!$YiKA`y0ubql7cv|tLU%M4w@I8vZ|8XnNUd0#u3B{Lr_@v?szF+ah z&m2&E!Jk%q>DM8}7yPi|OTV5~e8HbneCgNoiZA$!ihsuWpJR$I_;JOTb?1cQ3;wd= zZ+P75{}sg-{8h!5eZ?uo7yP{9i{36MzTg)XU-s3P6kl-nW3t)nQ|VXKeMv+7gWv@Y zm;K=@kK6thE56|J-W%adzbX`8@JdyG+U;+v6<_cM#TPxudp1OV!J8Cc=2f$*U-0FM zFZ=2hiZ6IV@kMW|6kqT)iZ6OwtN4P;`#-ckTNGdPxk1%GE<;tQTq zeA(9xD8Ar36<_p_R(!#CE57JqkKzlySMfy;`xIaBClvp@)6bKNFZcn)7dCR;cm2Le@da;Fe37$5@dfWv{K_Zn{OD7B!BdJa^J+lx z1>dRoaaVs@@de+j_%cuSDZb!OD1O4#|D@sz{*>a29`-A~;0F|6^zgLe3w}uPMGuD+ zU+`xYf0L8{ImH+JxWnoDo5AmUo^UvQUowcl>~Q-2V~rhJ{`Zu_#Sh4QJL7Qi1A?Db ze3@_O6kqW3iZA-Sp!kAcRQy$Lo?lXY!HeCOY_Q_ zZ&Q4cvqSL(?^1k`vrq8_Pbq%N$v>d@g6~#*k#mpY3%*zJMb3SSFZdIRFLFMq_<}#B z_#521wqNlDKcx5~=V8Sc{8_~pIiFK}!Jk)rk#k7#1%FZTWxYG5_<|o-eDQ}T6kqU{ z6@R_l*HC=HPbt3W;f&%7epc~C59btL@biiO|k#TUF;@r|p0x#9~h?zxZJQyoCeriEuo5w*dXPV(>i%%UvnKn_Rj>;ip_7clp6T z(nyfJXXUWs%lkjhDqQ3-+uFUJ_OAZq=FXm0Z--gnwRg2H*xK3CnQWzZ{g@=Z3T$ob z_MR;rpJM9G1Ezk!@lx#rDU$>x)s8o*;k9!;coE#T<;#~fFW%a`rFGeL%eS^IYFd8n z)@55-TbEtizIF4`YulG|mEd=#Ov`6)zvY%&`9*klwWscE?d)%FyT$A6zUi)%w;nH+ z>o*HF_xHaq&D@06SKYPt&X%ODEnCq|klemw&o$F?-|F@hYHn}scKzzy?4j2{ZQsm- zp59dZg4K6?WN^eQ7fqBh>OyFb8dGUBxd^%fYQm+dGn5>4kn~ z?b^@Yz6vkQ!wcwc?`_+Tmne~jlWi7awIsW&T*=;^Eog1R`b{R=Qg{_$PQBZE@XEio ze{0QLC0CT{P1<*iSsg(?w(B+t1q72;TR|P(((4If!rjAk>uT-qPmcBO!{jZk+uH5h z@NTj1Ic)2^Z&CB2Wq1)Gy}pfJ(T3AS0p6|jxkks&=n)3#CSKw=P+%N;@Q(yrsJX)<)9Y)Z4SQ z^PZ0O);7H15c*H1dQ;?wlHx?2@B1JhU3iZY{vUDd|5ppW=H8Pd1i1B_;ReaZ_S`(3*OZatJd z-TpU_L3L^!R35S=G+)}6`sgyN=~$a&#{2BTe*W_+sNQ^T4 H>GJr zS29V!agCt))p46&433j{vB{6NX%o^khQvSDC60q>ercNK_X`l>V&XVhbsfvGqyP7u znYs7QBEu!A{cn@pk5+fiJ!j6GIdkUB^Ul3{@wHcZJszPikNAqPA^lW{Bj52`K$=5o zQ4C7Z4%G><^MtQ8ow|DelkX_iqS;w5&Ge`#<{k=3QUU|FTGoy!u{m{*~pv z{F^?p^A(?`M>f{PKnQSKqq;@zNu2-u2|82YsF2 z|4ME8z$xIo9^Z4FNWb~I9Y=3{=}(SM!+kyOd%Wop@EF2x=hZv*4KMpre)t;Puf;vT z{@LM{*t5fr-FJBSBHWv=dVcuU(=QD__RP2ve> z`}Pg*{rSG(7e0A-c<+nf9?paA)_pI6t}UOx|Nf)5?z`{k_UVrgFMD?1@Vbr94)4A1 zJHszL_VeM>w-L?7`E`)z^mX?iy&7rn1zj`Jy!ZaY!`qu)9Nr6F(aD$Rx0ih4XiNP5 zqc42&OT#VS**Bb@{><>{#}5y`@a-1?6U}A$=JiPH@%xYFUwjPl_YH5SeBihJPhJ|n z^?FL{(tPAg_Z^Kcx$o#Y$b9;?FAO&WfBKWp46l3XXT$f-_yuU@<+snc@946oPagfp zQ~QSR-TUb9)nEF~@ab!w8g9PtCEQ;ejx=A9f1!BC(PhC;K%U1XpMBpMUiSRAhu6(` zez@iNpMmDe{Pw*_YvcV#|8c{YhELCcey)3Vc-ivr4(FR+9KIL&i8Qz8Bc9sVA{*~r z`NB(&Wg-K$UyTHFD+I9%f>p7ks=dEl>+a}#l z#jDum^M&^|`(MAsmw&C(4?7INZiD&xKfQC;zr+4c73GqV z+-Fw~9Di+cG&ivFt2@#oO@0wCPvll!jP!50%rhdEKI(t;6X}r`N_S^&p6wY~)8ZLf zUGEv`{(pm4c3k&@|Cc`B-q(HJy(2yk@+bCg{$^lA9N9g2ZN@Y5=soF?z7G@~So?6` z&Aw&0Z^W~^(<9wK9K5n?Z{W?fw*tN^J+k&@+&_Z*hjH)4{R6mf!u>|vZ^L~h?ziB6 zJ?=N*z6|$H+^@p@k8odr`+IS3$Nk;7&&7Q??lW+|2KOnrFTuSA_l3Ampu2cZeCUav zJ~|(81>nnZPvU+F?j^X-!99xmCvjip85zAtJa-3l(&e4DbM-f;?_4uDedo;!rtiGP zJAEgW|L8<<;5dB7=s0oU)gp0V)F%#5c_QArVM8&wY1B4|MGvyfSs&X@A!j z2Csw;KGwA$@FwN2%R6=F>cN(sYZkQZyxH5b^OkQmTX}Qli}Lvs-vqXG%HO)@7mchv zy=dgt;Np=xPar&{GQW0s(a5@&7LDBT>qR4Xc^8jtsezmyMgCs}{!ZLq$Ndi6qj=sj zKEJjEI;yY8uS){1z;8-7%ixrq3sf0VK1Nmf$b3{;`B-(;H?Qnk@Xae% zc_aH)U61fG3WxWtx)I@(6b|iM^L6Fc&NUA=?7VrfVdpKY8qOitcw4Rtq?x2N1N&ChAUuJ>{(Y;aAY4yj-@a8d5N@Hc zSIPz76xn_6%4?AC!}zU@`rphZ{UgG2@XB>!cV_L0*v?xo^Bw40;5%^J?C^oLPkG*4 zSK=SpPP7&Nk-lBN1M3ET2fDn2S9V6k?cGA$@B-S!2-?JTg9{JLSI?S3kJhzp9q#ME z=W5)ek=)9ry1jo<a!p-nO1g{EqjAZ{`XMW@JAI|rD z<3stYJz^yD)Ccq3gMXA?eg6mYH~-oD^YtGR`M%G-FF$XMCqLs#Z@zW0FF)Jk&({?P z@{Q%e{FF(dJn_{$H{8)N^3m_VH-GD2rAKc2XvfGMFM0D@YQ)I;J3B@$dzZ*hf4?uk z;S1@Jwq{R$+AMGW@_v8*SH9@pTYq?Ce*KL1r%2z|ixy=FKgZb?tF+%+5nNFkI zd-cAEC-GV;Aog{8m+ngiJh(5x-HZEGxcl}wW$N}W-WM6r^otbzLPfveEPBfGx$+zF zl)mQVzX|!=i2RYvx7MUNpKTi`f5CiYp!BtwmwNM?pqKSuTacgL9Lmp}<;%}NzF+W( zoh=>WxlWYDZS@@^QBU%_e?8HU@q8uGPduCcMxr0%`3H!;`fU0S6a5&^ZzlRlXVc$G^kY2VNc1&l(`ShO z?B`L~8TFxVTh%erfY|?Ej1|^gK8ED_{PFY3Y$u=$9XdZJ&sX{K@g2{0U#U<@s7~{)LIMJ^6J* zdIa=4ojgXr6++*s3TaG0S`(3GHPW7hK2uG;GY22|zr+nqB7LB?%5(bw{9%(nJreE~ zJIVjmP7=4*{lde~BFICuJUw!eC;G{Uiqpe&h!-hMADCY(Zujg*ID#j z|C)I#y45caeA%Yk^EZkvAYMN7x8jEK;`D*0ghT$ADS!8A$d6}Q{^^ka3CMpOaUsir z`JVFI4t<5wJ3z7?|0=>NFNY2w{1$ns`x?g|iKh0fY({?Od!j!(glFMWqz||j@^$|m z^qF$VR%Xg}=y}M7XIi!yknP8iZ4csU*#^Fju+sCMzeiZf=6M2P$VM`sqA>J5zfRmf zKw+i#Itn}VeH<|9yGY!#veQ%avw?$J=dqXf9Fnq=Eb~1pU+(xO;(~U*C-!ppw}3<0 zP@V>!M4nn5d8#(^)O{Rz!ZV$xX~@&v$kQRj(|OwSzY(T9LH0vWA*}MW=N}MOdD`|g z!YWU76jpgUL^Q}#bZEDw_io^wdiVSzWK26`Y&2yY_`i@5&$Ns)A>$}y>_9v%W8Gnd zm5d$VMp(&M_YA@$BW&IC9fV270de~tdah*bpfF@?CS8__mx<0P-yXnCD;)A&WXgBw zUmzczY5D3QUuO(9OX+EweHLLQ-|_DvtZnuP!b-k9&mpYj>!7fb?>NzD`A`pu&M9BV zQOMWjkndtsKF{|dAD(IXW|Mp+>EQvy)ADuy6T(Wqf$t%#)F2&*z)`hA3ze8=gz zl5c>*T0ZEn19VRLjstG$amaUtDPPA?$cJZIzE;Rr3;F8)nfY>^_xu21C12gYAgtu` z{19O!-zf?!`35MgL@*P4vt-n1# zLYU-(ogMnW2y6TMF~ZvZjv=h=kDhD$qcG%SKU)R+16?!MW$(*}5GGkE?fEGC?&E-g z8SzZZ*Z>*thKwDEr)8`=g|L#b;{}AZ zZcihuWDNfdVXa$wu60XcUG^K&FMB}ely48Hf6Bc5p)r$WY2lJRxsEoG#zmhlaQ zwTu*2I(+&~gq04vDXe9ru-0KW`a*!6I_&;6Q5*>EmbE_$1 z$NvHz&$Nt}LB>wVSm$HjtV538Bj>eae(?&liEmGcDg0knf=o{2}6L`SuhetmHdXg0PZrPaI(- z-#`LkC0`wdm3)Ul6NY@c&+92gdaVw5K4Z$${Rr@QrscU3@;m{3bs(OWr!I-GlBZ)F z!b+aHGK5u`hRYFFcDIL~D|tF7Ol6wB$Ev&Cpfz+>0U6Q`86Poa9M}y!o@p7|A>)rp zM#QsZ9FMS;u@Ye|V-><$#t8_MjOdr|q32pg3X_bDb|3aQo;hV4K-{Jk4jCUeW!&=_ z;PFh$csXRe8!{fEG*vsYXClJ7T&oe*A#{oBWIpq9uft;TP9?!I#b4i!f7prCdvJR&ZR&v(WA*|%|)FZ6qJVjw8 z=KzJ3oOKk2oKn8kFCPc&ly9H`>GwF~`?@LL@lOGdXIj2Fq(|zDA)ckjMufE~?3_oOLb-A>@~Ov`sM7cNZ?>Ny=Ij0A$`Z55#Q-^iaA;XYE#%D|!JN^WC zJkv5>0vSgkV>jYy89QbmtYqw-iLjEfqZMH#W78~zm5hhzxstJ)!fcCnd+Y(NQ^q}j zn|3>7{GKUe_jcg%Ov~5;89QknfYQ|cuGt7{J+xcQwI16LCK=J*bx>IAk!W-| z*NKQA2OA*#GUWTwP zYYJ-_i3T#BenN~ym)gFk8#GQ`dM<|?2OYXRSs>>C@OV}zX9wbGIqR-KSjpLOCBjP1 zx^{$>oZ?Vjb0ud7g-Oo8gPaZ@c?i#)a&{wr(-DW9FBZu8$H3!Rp`3dtT~+1- zX@r%WdpZzSat2c2mXg~`83u+UiTI6J?<-S`PHwAvA$5U_sxGV z!JdKm-Zy+jdq*Bh<6&F6=Mxh+OhI%sn2P6WmGjrRim!wl{tm%fGRsi`FFzaIAh z?#pq%ky&#^I7f2twK>7f5@OhL)5cj8W zAB6tSRn9uENtbk*e}&6(ndp)){ERC(e-S*tzG?mTLdsd^N$af5r7ZtD>`rFQ{~s@q zUd!{IsJnEYb;?k4u@yxunxN8}E|7+9h4eC0!%v z-fo@Ebt%hJ@U3p_LF?@B1=8!b|3;U-;=3;S*ZR({!=#J;Q@INoXJ7ia?Z+N*nZGWV zbh=ztyQE*|l5T@by4ykbcH8njE@c_#l1}Svyi59OmvkwYbnireX0A(Fo(f!G{dYJdmvr}n?(NpuXI#p1kxM#VH(OoO zx4ERd)FquxU$>4mh_KXyrf(k0y~mvk?J?(NpuuU*P=ze_q@ z?z>#lKj4z?FF@zUr_6y*dAoJ?mo8=Qa7m|iw%8^8GM9AMyQIU*3~#&6R=bqt74%0h zq;9_Ek{<6Zy5j3`Dd)Rf^ml*T@!EKovS|C!zVyl2Vul4iK??64h)1_=t(7Ex)Iu9i->BqUG z8}E|t-`;_GI^Cr#fA4kYgS5^Lxuk!}CEa0{bbtO1=2lxhlKy^|bh}*A{Tb-qZaaI_r7ZJY(rKMt;gY_?CEa3| zbUJ-qSHFyU^mgm;y)I>=HvU4!3_o*8f5s)zf z`=U#_$3XXX+u7f{l!e;(3(*;k5#7+==#uUOF6llDy0=?rJuYP_^c!DrX`hAhLfF5A zdcN}z_MtuS8}j(XZy2xtH;i{1;+?M?#{Y(Tyuk(UbB&d?K95763&D%_FxL9Ttb^b9 z^$oJ$A@^K9@*krR@%;PzE^?`(dY5#SF6l~K()nG|z3#F++_nwRyX5yHmvl#6(mm~x z?i()YzU-0?2e`QMdAnWG{h3R;`&`lux}?K_SFZB0(Iwq#mvnA@{0f)+u5w9tg-g2G zF6o+F($T&`H*F*aL$2teF6j!}mp$-CwY`0rzV1~B>3jbM3G=t9#c2v2Non?^(C8 zZ(e8LhVIUrH)S?-_gQ=vb@z2&#g7)e|Axh#vpZL9%66_^w}BXU&Y9ggtJ5MqTZGP5 zP38{d2&I%+v;KF&+K`X#--(Q@UbAZREuA-Kx;Lzm zrBnEzdxr4@)$FG3_3LlRboZ>~O4pOw)VGOhJwrEb+SuFIy|J(Lcffpa>h9UFCUev8 zL@}e-I4^}|$q$w56gTednq*=)rcbUW4{;(@kquccQD^nOW1f8AF2AovZpg`MGd1pXHAE zW(We7-F@JLgf?&3*jt#G!FINT?HuYR-_e=b2yt%iyR-9Vbe4P9Z~6#2_GqC0(BG27 zmj5!t6vIDL=@dS2-{1Q0KVZ|X(YrUS*)U_%%{TY1>1!38of~1!YRvHN>lRugVVrf*%}x=kCoD`W8>?&!?@PIXYv*-8u147?As2Zh(A{2Z%<1=ol8 zd2zv0A^s9CkNZUYtmprL=U#lVApN+UCP^!bMoeGO^IXC6yVUc8`Z-Oq;Oc;gKAs_w zu>OBLKgVoQ!SyG2o=iC8{IX2{8f2*sv}r|8b1rn@eG!q`9k-s#r$T&;>Fwt!g~%&D zS|rWq8+h)+JW}B`imOpQq%d8`&lbAJV*|NdlL(Ivf5wweeXpnIQli12TRSGGvnfm zQa93VM%$%g!RB@ASEqXWd;8XGNNw2M+m~9s=H_)9*Q9P(zWmyEciy;o<-4z5etqZS z_ua7g`bCQuiRsHzb5hg0GdHbG^={~#)p|*H3Q3c!RAg9BDv^9&k@o59D6hj6)UTibq3XwD5a*;LQGSOqekN56sP0G9{g}G9wn}_X-Fs#!dUf~DT7-|I z2(cT#{rDZk?+AX!@N#5sGZs3 zxc;Dl3()|4bl^gN{-(9w931KqRKPAc=pOuzq2W1#mlqBe!H&@tLia`tmEt!}h}KH< zfhGu%ngkqvtysWIgFNQr|h5J&#>+!q?c78kf^@1;cJ$>MF2kv*_*%qXc z!yUie0Mg%z@cp2}?+kva9pH`M3jA{T?Z$6Ee#h{m9SI5i()e}Z*Mr|N{8pe&@Bbg5 z|DWK2Ut%|&;kW-`!1#529(4G1eF6L*1s#4z@Jr=EhhGnVL-_5-Pdtu0eh0sd=YNCW z--7Ndh=bp5{PuqpIQ)hVfcD$K{{X*#LHHQ(Kf&*(;QbP0_ywN*60+jg3cu5XUsuQ@ zcH=h`_lP6(8xKd(>=7%bc|>|Pe(fI7h2Q?G5WW_E;d+l~T?w!EK94wvU*bb}-t7@R zYrtn6;%@*=FXG>YI9u@SZqVlt9smx%Gx#M2@eIGzJ&1?j5PtjdJBZ&A{Ep#w2EW8s z@Wy7D)F+YN4#3!Ov;WUQgJ1415dNPCKLVQ11BVSg$G(U#HazaePvjvteqt}uz)w7e zbnp}CmwT7xgg@OShM-R$9+7RYfSz=CH*}@LM_#qV0{YU=TcI-@?tow|V+u|y_>6*AD44n- zr0;`h*ANm31s?-U`DKcr8xOPhaC^q?*i+{*ZbAX~NZ32{up5??9E zer-t~ z@he5D+k{uZk1;>yUnz3%X-;@Q{F@U#f=zkTklubyuSO)`^9Uw+seE^T3=Ret<&WTF zAD8eEMStcK5+-?>{$2@_eh5x~R;H)xcaMUpep3D3uVC^Q1RqndU^pRC56Sqt{&p!? z*4MDutzd26#}rKVN%~Limhm;dOTijHq+nUz@DB!@5N8Y+`rae?NqYDn1BSj082eca z81|!JZ9j>JB|po46s+whr(kVA`xUJ1=a`1MJm9}RC;4gnNh?^}Pmcjtiroe9kpftt zeRA4MYXRH?*l9nzsm;Qr?ZY>3obXr2g!l+-2yNa$w0ZtSOmu;aDCy^~;TA^ts(#?} zN5edemGdDyS3SQCZ655_dyC{v?H#4@YjuB9-QU3dNl7BmEb-V_&lxN#c&_I%o+L)N zG<{mldo7ZWt>+5lF^Q92KTj3#*UxvWfab5Cztw&4E;#)k0~^KB%Khi$R8>R?D5t{fS@8zai;HB|Cu#wMUx?tL zZ1KEBD96!b7*PU-eC4Fto)xbbiD%+s$>#flQq|yhIxb=v;gkH%_(1P}{3F3pu?`U^ zzx5>|mKDAr#ZM0tUd-zAj#7O72iThxq7E<{9pJ&NpKX#Jp+hM?o+a*dp+JDQVQ@8I zXw8731_K5Qz@?w@`&fUKq~ExBAR7*nk;(kl0uBaq791CCpAG^_prSxOhhYz&4h4(A zYLqf{GVZVXbddGIF!;57nheKD9|@cE;V|c);)8zZX+H!{h|%NF_tSCl;L`!Wq7M`P zLEsfV!w33F5p-ep?*u;5*Dr{Z1#f#$_=9lXqon^+9zk#z&R@e%VeDdEt4qXR{mp2N zpBe)X=vSr0Uws}hH3kd|@mK!?Fgbq>e+)1=e+5_S`aw(~Zz2D`bmCXffc^9ZC_h3- z`V#)SvnHuyyvNy61XjfjS53~BDDQSBe%+gLkjVJr^qJIuJjvwmB z&P)JDf3UsB$)10**)Qu)82N835#B8URi6NZ|B}sR$SPfs_r=2WMdiMBAUn~SXA2Z zwtg`hc;5g5d2t>m~F`k9R^C4lBANZMY{4979`pfcy6wq%W{Z{0s2KfPZl+XR}Z_3@ED ztTLoOJnZ+fzZDN=1Eb_G{blgCz(0_Q`XPzsKbACqkYeHjKiI>!`jxyh@d*Bq>{ToP zJ*A2Cs_ncw>iSz;MECN524b)IyO#9<6QhDP6d>HK0CkD3!X-GyF`-j0TAsrp^ z)dDZ)5gGr3O=4zP`_n{lH%hZi#6J&*eLm4b_01)F1Fe z*E6Bk(CTa`ZFqP)S*EbpA6f2Yd($@7%=8^7n}U3r1!{nWY3 z8|=*TKKKroH+ra@ANxd-k2jiXZ^-2TEExj-FkuLec!pu8|=*T&b`Cs z{mgmF+y7suylK9u9R0zbw46^W?B5yvb&QWNKK9Z2%%~UR?>AJP2NxyN7V&r*k!F7)3Uvmt)(F+lt=-fQd&hQza&3niH5gR&tW4+hIHhydKS zB^adfprG+0`nT0t8u$s3sTpQ|hrR5+i2wtS+A;6})m_FIl;Vf2@gej7daoSbp-&&i zz7gd2q|qO4!~7NH59#xK0k_{eKj?oO({KiN80kP6+m*=C__+FOh<9;6B zpQAjLcHgo*dHNQ+F~=^?d(K^+gLh$m8TFSZ(^p+DW?(+Xx{#lNe!st+9!#u>iAQG8 zd@LeBe=A=0iKmv+e8~(+53Brst(d+|Li```4`%|>KzTj|`rFbd1)}+6%#RK9dmEX5 z75J}$y={Jx=R1X(zx!4{jV^H!7tDtbW=mM#{0#gb|471~f8qJ@m>v)E{Kw#yFt2Zb zUd~@+BPbYkA^szMCG%3>9vv;A`UIHfGtBvpfIU8b8uM3MilzL-5AzGVvi=D18;!~M zJ2TNR#UEAkzsCGE>idJ)B0tUVVE!#EFu(Q0W@tVQ2VM#o^QB8RZw<>OxgpHoojt!r z{^=kkNb?OepJ4c-PH*A-&rl{k2z`0PBdWf` z|FM2BKNlu{H`*g%%pXNTF8Oo*EPtiuuYy6EPnGR`!T22d#`v7)V~T#m`5{Z5r6f;* z{{erRZx?Ah|AW~gk%ByY0q=Ic=r@(`eEtaiarEA8`DlK_gagvP)OtxFzx%QR4F=Um ztnZ-x_(Ud%1rol(;#uU+8;B4r`(wP`;wOK_^sv7ro4djhCYST&R9{q$ll5gwgy)Mv zqTrY|7lxPgUm<8+4)e`CAGu`nwE-u;2eY-@ezCnn--k27i#R#*2dsb0KR&Jc2eLeB zXtGet`(U{sxvY-8i--N-+!XMrB|t%f&` zP61ZYeYU2lQd-7yNc*8-g@rs`(FYsW&l0U-aIMAG}(PL)jTYO5E0evr&FS*@gxz2UvL0FU|0ez9IFyE8Bwh$O&JB;<4eu>|~KnQ;o!h z^p|M z_JH-7C7Z9Xw@@C~3-!#b^52Ij(`z--D_lP*Os^1r zqJI>dV!%&yas4R7Q+siSDhRa~XQ1!xt*ZXY{wwK=npX6OdA)WhTbm$uT>j|qEZMvb zrAL?a4~H`;qyFy9gd$Uc9cBMZ`3**B&$_Pf)SlK;bjyDPjQa6lHp=xiWYu3s`@!~I zB&h!K1$bvB7HI%5nvnMPK*sMUfwa9o0RIPrv|y}9^NIx91LE(>Mj@U@6ul_p@5~g3 zQx1N+vZZJ)M@LhyNj$X|;WP;7Lj0jjEH(u}ogUV+i|0Au2eK0|+OhEuWUFa`*P?$Q zTR{u54!DxrKh5ufYzcSZZ8(Vr%%=aheo;#L#hPwJoWv5sAEW)88>3z4D1WTi>GGuZ z0P+G~J>g=)qx>X&WyNd*PL{MP81Y}|6}9Bw@WXnW59R-eQU1WswD2baVoCpJSQ^*| zuV-NWAbf92>f@%U@RPs#H@ zHgz8Rt1hs=WP$xnFzjz=TX797&ujjsJ$lIA{K2`{m*LAn-wao#E@l|^$8bgUYy(bC zXjL%i4f`Yb9#L%Do9C?cwL*Vqj`!}S{+-c(8p^8u5xTsx0qMULT*wdh;{ZhexA{sBor_AeYnYJ%4L z56=`+v;M$id-t~R{07fA_~iOC?FRz?(u~i~^B+Il!SfM~S-C^m&uRV?U*8E4--+mBSpPmmve+&&K z{E?g=nVfCq{I~M_$a0JyQ>{FHoP_b?qa}c=TiYoD!KVnWY;8{|xGC-(=r3z+Z?$0H z6NrxzBKjs8kH!$+35OBi3HuRW!Sz1O&s;uA{@q7b1p8>rPC;tu?~uGFmiv?0seqBe zg9H;kRcGVlz&8VS;_K=icyUK&?iAA3)L(LzQ^x}Td&qadiW-5sJ7{M5dXOlAl*dtEZ-#+Ha_l2W?Mk-j9-l%+g5rU9|EPPQGAwfI_>Co zzuX4_((^2NE<<@+ z^w4L;_yYQJj8M{-TKumBUeUw8?#PTU$A~3;nZj3PrT-`+{~__?kwE%Nm0oG4tc?B1 zY4{Vi&-rj)cKWHTZ~Ww|lb0h3*QaE*W1JJ;n7yKms;}}tue@8K0c5?KQhiwj_hrjN@qrdS$&8w>O+);gs^3(9 z%h{ie9@O=13EaCaPcoZEMsj!S`VW7o_D82FB>tI=H9@;7G3!yDS=PyB(s8&N#6zL#I|+~|C9Ir&qh zm&{%PdT0Cu%EykM%wApwfu+7c-iPk7D)LM;SBTxm@n)vgMP<3@X*f!9v`UkMEx2E*eOqJAp}l$DE_I; z)TwD^hW{tx;Z$bI6ccW4HsMstgqxa7xUSBGr%f~A`g#*?X))o31{0n>-Gr;FO}Mqy zgeOfh;aRgxxTeO0XU{g_+FBEyGslE0D^2*~i%qzy%7ok6OnAZs6P`EEgeOik;Y%+y z;kk27__E7P_>xOZ_{u9yI9=g@$)94n)Uprwi(V{8&CDn2r!V z%M-J=5BX{)U5J0PfM|NAag4{vs8BYPP!W#`nAjF9Aq@+n$*< z3;honALT*x5GcLGO0TXujqKJb|Jt6Zon5QxS$>-Sb*2|u3IWqsDScFBmvaBKj=Uh3 zM<@gV(^o5ed!}*@`X`qBQ?u7Bb@ngXGgTL(KO*sHpHkUtL(cZOJu{&V{SS!;|ElaV z9zR^8+W+>4 z;oTSHiCYhat;#TcB;4s5!#K494Xs>xbfZJ<~hr-#VlTdtj6Ax+6ld2K2 z_=S6y%JnR2WYAtFGs~8-J-wmYugs*&*